2024-04-17T07:00:20.378 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-17T07:00:20.378 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904 --verbose -- /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/orig.config.yaml 2024-04-17T07:00:20.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-17T07:00:20.502 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904 branch: wip-rishabh-testing-20240416.193735 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: null first_in_suite: false job_id: '7659904' kernel: client: branch: testing flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240416.193735 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: a8c84f5997950fd7d008204839253291d52e6eda ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a8c84f5997950fd7d008204839253291d52e6eda kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240416.193735 sha1: a8c84f5997950fd7d008204839253291d52e6eda owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3521 sha1: a8c84f5997950fd7d008204839253291d52e6eda sleep_before_teardown: 0 subset: 410/512 suite: fs suite_branch: wip-rishabh-testing-20240416.193735 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a8c84f5997950fd7d008204839253291d52e6eda syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjfw6h6JlM/VdJSY9YLTvtZbS8r4uvCOoXXUwb++K3eVRFZ3cvDITS7Ooghp+2zjwYm8uc+/FA5Tmz2lvvAQ9A= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMER5OGO3me8ZozXOURqAX21fPe/kP8GyM6fXYA7gpJD1upTcNvwuBB2LbxwQ9RM10QyUxYXSASm1L2lMQnlalE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-17_04:58:44 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-17T07:00:20.502 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa; will attempt to use it 2024-04-17T07:00:20.502 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks 2024-04-17T07:00:20.503 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-17T07:00:20.504 INFO:teuthology.task.internal:Checking packages... 2024-04-17T07:00:20.529 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a8c84f5997950fd7d008204839253291d52e6eda' 2024-04-17T07:00:20.529 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-17T07:00:20.529 INFO:teuthology.packaging:ref: None 2024-04-17T07:00:20.529 INFO:teuthology.packaging:tag: None 2024-04-17T07:00:20.529 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240416.193735 2024-04-17T07:00:20.530 INFO:teuthology.packaging:sha1: a8c84f5997950fd7d008204839253291d52e6eda 2024-04-17T07:00:20.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240416.193735 2024-04-17T07:00:20.671 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3133.ga8c84f59 2024-04-17T07:00:20.673 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-17T07:00:20.681 INFO:teuthology.task.internal:no buildpackages task found 2024-04-17T07:00:20.681 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-17T07:00:20.729 INFO:teuthology.task.internal:Saving configuration 2024-04-17T07:00:20.746 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-17T07:00:20.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-17T07:00:20.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 06:55:03.485080', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjfw6h6JlM/VdJSY9YLTvtZbS8r4uvCOoXXUwb++K3eVRFZ3cvDITS7Ooghp+2zjwYm8uc+/FA5Tmz2lvvAQ9A='} 2024-04-17T07:00:20.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 06:55:03.486577', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMER5OGO3me8ZozXOURqAX21fPe/kP8GyM6fXYA7gpJD1upTcNvwuBB2LbxwQ9RM10QyUxYXSASm1L2lMQnlalE='} 2024-04-17T07:00:20.825 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-17T07:00:20.832 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-17T07:00:20.832 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-17T07:00:20.832 INFO:teuthology.run_tasks:Running task console_log... 2024-04-17T07:00:20.922 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f05bb55d160>, signals=[15]) 2024-04-17T07:00:20.923 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-17T07:00:20.930 INFO:teuthology.task.internal:Opening connections... 2024-04-17T07:00:20.931 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:00:20.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:00:21.037 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:00:21.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:00:21.125 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-17T07:00:21.133 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-17T07:00:21.151 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-17T07:00:21.151 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-04-17T07:00:21.208 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-17T07:00:21.209 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-17T07:00:21.209 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-17T07:00:21.233 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-04-17T07:00:21.250 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-04-17T07:00:21.250 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-17T07:00:21.307 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-04-17T07:00:21.308 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-17T07:00:21.308 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-17T07:00:21.308 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-17T07:00:21.308 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-04-17T07:00:21.333 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-17T07:00:21.343 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-17T07:00:21.371 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-17T07:00:21.371 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-17T07:00:21.374 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-04-17T07:00:21.390 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-17T07:00:21.403 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-17T07:00:21.404 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-17T07:00:21.430 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-04-17T07:00:21.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-17T07:00:21.584 INFO:teuthology.run_tasks:Running task kernel... 2024-04-17T07:00:21.600 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-17T07:00:21.600 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-17T07:00:21.600 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-17T07:00:21.600 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T07:00:21.601 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T07:00:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:00:21.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-17T07:00:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:00:21.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-17T07:00:21.780 DEBUG:teuthology.task.kernel:sha1 for client.1 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:00:21.780 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-17T07:00:21.780 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-17T07:00:21.799 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-361.el9.x86_64 2024-04-17T07:00:21.799 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:00:21.799 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-17T07:00:21.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum upgrade -y linux-firmware 2024-04-17T07:00:21.853 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:00:21.853 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-17T07:00:21.853 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-17T07:00:21.871 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-04-17T07:00:21.871 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:00:21.871 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-17T07:00:21.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum upgrade -y linux-firmware 2024-04-17T07:00:23.998 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-17T07:00:24.022 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-04-17T07:00:25.430 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-04-17T07:00:25.682 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-04-17T07:00:26.555 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-17T07:00:27.431 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-04-17T07:00:28.514 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-17T07:00:31.481 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-04-17T07:00:32.890 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.2 MB 00:00 2024-04-17T07:00:34.683 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-17T07:00:35.582 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-17T07:00:36.517 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-17T07:00:37.207 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-17T07:00:41.243 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-04-17T07:00:41.650 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-17T07:00:44.154 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 1 Package 2024-04-17T07:00:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:00:44.164 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 375 M 2024-04-17T07:00:44.164 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-17T07:00:46.291 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-17T07:00:48.722 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-17T07:00:48.728 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 1 Package 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 375 M 2024-04-17T07:00:48.729 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-17T07:00:53.159 INFO:teuthology.orchestra.run.smithi067.stdout:linux-firmware-20240219-143.el9.noarch.rpm 42 MB/s | 375 MB 00:08 2024-04-17T07:00:53.160 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-17T07:00:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:Total 42 MB/s | 375 MB 00:08 2024-04-17T07:00:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-17T07:00:53.271 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-17T07:00:53.271 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-17T07:00:55.753 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-17T07:00:55.753 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-17T07:00:57.821 INFO:teuthology.orchestra.run.smithi187.stdout:linux-firmware-20240219-143.el9.noarch.rpm 41 MB/s | 375 MB 00:09 2024-04-17T07:00:57.822 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-17T07:00:57.822 INFO:teuthology.orchestra.run.smithi187.stdout:Total 41 MB/s | 375 MB 00:09 2024-04-17T07:00:57.823 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-17T07:00:57.918 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-17T07:00:57.918 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-17T07:01:00.383 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-17T07:01:00.384 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-17T07:01:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-17T07:01:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-17T07:01:02.695 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:01:04.413 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-17T07:01:04.711 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T07:01:04.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:01:04.935 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-17T07:01:04.935 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-17T07:01:04.938 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi067.front.sepia.ceph.com to complete... 2024-04-17T07:01:06.739 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-17T07:01:07.018 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-17T07:01:07.099 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:08.479 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:08.479 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:01:08.846 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-17T07:01:09.132 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-17T07:01:09.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T07:01:09.361 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-17T07:01:09.361 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-17T07:01:09.364 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi187.front.sepia.ceph.com to complete... 2024-04-17T07:01:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17 07:01:18 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-17T07:01:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:FINISHED --2024-04-17 07:01:18-- 2024-04-17T07:01:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:Total wall clock time: 14s 2024-04-17T07:01:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:Downloaded: 1 files, 341M in 14s (25.0 MB/s) 2024-04-17T07:01:18.898 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T07:01:18.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-17T07:01:21.058 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying... ######################################## 2024-04-17T07:01:21.112 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing... ######################################## 2024-04-17T07:01:21.197 INFO:teuthology.orchestra.run.smithi067.stdout:Updating / installing... 2024-04-17T07:01:29.475 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-17T07:01:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17 07:01:29 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-17T07:01:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:FINISHED --2024-04-17 07:01:29-- 2024-04-17T07:01:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:Total wall clock time: 21s 2024-04-17T07:01:29.981 INFO:teuthology.orchestra.run.smithi187.stderr:Downloaded: 1 files, 341M in 20s (17.0 MB/s) 2024-04-17T07:01:29.984 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-17T07:01:29.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-17T07:01:32.040 INFO:teuthology.orchestra.run.smithi187.stdout:Verifying... ######################################## 2024-04-17T07:01:32.121 INFO:teuthology.orchestra.run.smithi187.stdout:Preparing... ######################################## 2024-04-17T07:01:32.201 INFO:teuthology.orchestra.run.smithi187.stdout:Updating / installing... 2024-04-17T07:01:40.167 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-17T07:01:46.012 INFO:teuthology.orchestra.run.smithi067.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-17T07:01:57.043 INFO:teuthology.orchestra.run.smithi187.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-17T07:02:09.724 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-17T07:02:09.724 DEBUG:teuthology.orchestra.run.smithi067:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-17T07:02:09.805 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.805 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.805 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.805 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-17T07:02:09.806 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-17T07:02:09.807 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-17T07:02:09.808 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-17T07:02:09.809 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-17T07:02:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-17T07:02:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-17T07:02:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-17T07:02:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-17T07:02:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-17T07:02:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-17T07:02:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-17T07:02:09.817 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-17T07:02:09.818 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-17T07:02:09.819 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-17T07:02:09.820 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-17T07:02:09.821 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-17T07:02:09.822 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-17T07:02:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-17T07:02:09.824 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-17T07:02:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-17T07:02:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-17T07:02:09.827 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-17T07:02:09.828 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-17T07:02:09.829 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-17T07:02:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-17T07:02:09.831 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-17T07:02:09.832 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-17T07:02:09.833 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-17T07:02:09.834 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-17T07:02:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-17T07:02:09.836 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-17T07:02:09.837 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-17T07:02:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-17T07:02:09.839 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-17T07:02:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-17T07:02:09.841 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-17T07:02:09.842 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-17T07:02:09.843 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-17T07:02:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-17T07:02:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-17T07:02:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-17T07:02:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-17T07:02:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-17T07:02:09.849 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-17T07:02:09.850 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-17T07:02:09.851 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-17T07:02:09.852 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-17T07:02:09.853 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-17T07:02:09.854 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-17T07:02:09.855 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-17T07:02:09.856 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-17T07:02:09.857 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-17T07:02:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-17T07:02:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-17T07:02:09.860 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-17T07:02:09.861 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-17T07:02:09.862 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-17T07:02:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-17T07:02:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-17T07:02:09.865 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-17T07:02:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-17T07:02:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-17T07:02:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-17T07:02:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-17T07:02:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-17T07:02:09.871 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-17T07:02:09.872 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-17T07:02:09.873 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-17T07:02:09.874 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-17T07:02:09.875 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-17T07:02:09.876 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-17T07:02:09.877 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-17T07:02:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-17T07:02:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-17T07:02:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-17T07:02:09.881 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-17T07:02:09.882 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-17T07:02:09.883 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-17T07:02:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-17T07:02:09.885 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-17T07:02:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-17T07:02:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-17T07:02:09.888 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-17T07:02:09.889 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-17T07:02:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-17T07:02:09.891 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-17T07:02:09.892 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-17T07:02:09.893 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-17T07:02:09.894 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-17T07:02:09.895 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-17T07:02:09.896 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-17T07:02:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-17T07:02:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-17T07:02:09.899 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-17T07:02:09.900 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-17T07:02:09.901 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-17T07:02:09.902 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-17T07:02:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-17T07:02:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-17T07:02:09.905 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-17T07:02:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-17T07:02:09.907 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-17T07:02:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-17T07:02:09.909 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-17T07:02:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-17T07:02:09.911 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-17T07:02:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-17T07:02:09.913 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-17T07:02:09.914 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-17T07:02:09.915 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-17T07:02:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-17T07:02:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-17T07:02:09.918 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-17T07:02:09.919 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-17T07:02:09.920 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-17T07:02:09.921 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-17T07:02:09.922 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-17T07:02:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-17T07:02:09.924 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-17T07:02:09.925 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-17T07:02:09.926 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-17T07:02:09.927 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-17T07:02:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-17T07:02:09.929 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-17T07:02:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-17T07:02:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-17T07:02:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-17T07:02:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-17T07:02:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-17T07:02:09.935 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-17T07:02:09.936 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-17T07:02:09.937 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-17T07:02:09.938 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-17T07:02:09.939 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-17T07:02:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-17T07:02:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-17T07:02:09.942 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-17T07:02:09.943 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-17T07:02:09.944 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-17T07:02:09.945 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-17T07:02:09.946 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-17T07:02:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-17T07:02:09.948 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-17T07:02:09.949 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-17T07:02:09.950 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-17T07:02:09.951 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-17T07:02:09.952 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-17T07:02:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-17T07:02:09.954 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-17T07:02:09.955 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-17T07:02:09.956 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-17T07:02:09.957 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-17T07:02:09.958 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-17T07:02:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-17T07:02:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-17T07:02:09.961 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-17T07:02:09.962 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-17T07:02:09.963 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-17T07:02:09.964 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-17T07:02:09.965 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-17T07:02:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-17T07:02:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-17T07:02:09.968 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-17T07:02:09.972 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.972 DEBUG:teuthology.orchestra.run.smithi067:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-17T07:02:09.996 INFO:teuthology.orchestra.run.smithi067.stdout:postinstall scriptlet (using /bin/sh): 2024-04-17T07:02:09.996 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:for file in vmlinuz System.map config; do 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: fi 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-04-17T07:02:09.997 INFO:teuthology.orchestra.run.smithi067.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:09.998 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-04-17T07:02:09.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-04-17T07:02:10.080 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:02:10.081 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-04-17T07:02:10.082 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-17T07:02:10.082 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:10.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-17T07:02:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-17T07:02:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-17T07:02:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-17T07:02:12.113 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-17T07:02:12.127 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.AO64b631fq 2024-04-17T07:02:12.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AO64b631fq 2024-04-17T07:02:12.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.AO64b631fq 2024-04-17T07:02:12.360 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.AO64b631fq is 6KB 2024-04-17T07:02:12.408 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.AO64b631fq 2024-04-17T07:02:12.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-04-17T07:02:12.491 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-17T07:02:12.491 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-17T07:02:12.491 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-17T07:02:12.491 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-17T07:02:12.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:12.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-04-17T07:02:19.971 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-17T07:02:19.971 DEBUG:teuthology.orchestra.run.smithi187:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-17T07:02:20.045 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-17T07:02:20.046 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-17T07:02:20.047 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-17T07:02:20.048 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-17T07:02:20.049 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-17T07:02:20.050 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-17T07:02:20.051 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-17T07:02:20.052 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-17T07:02:20.053 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-17T07:02:20.054 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-17T07:02:20.055 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-17T07:02:20.056 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-17T07:02:20.057 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-17T07:02:20.058 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-17T07:02:20.059 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-17T07:02:20.060 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-17T07:02:20.061 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-17T07:02:20.062 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-17T07:02:20.063 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-17T07:02:20.064 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-17T07:02:20.065 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-17T07:02:20.066 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-17T07:02:20.067 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-17T07:02:20.068 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-17T07:02:20.069 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-17T07:02:20.070 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-17T07:02:20.071 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-17T07:02:20.072 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-17T07:02:20.073 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-17T07:02:20.074 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-17T07:02:20.075 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-17T07:02:20.076 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-17T07:02:20.077 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-17T07:02:20.078 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-17T07:02:20.079 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-17T07:02:20.080 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-17T07:02:20.081 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-17T07:02:20.082 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-17T07:02:20.083 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-17T07:02:20.084 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-17T07:02:20.085 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-17T07:02:20.086 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-17T07:02:20.087 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-17T07:02:20.088 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-17T07:02:20.089 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-17T07:02:20.090 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-17T07:02:20.091 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-17T07:02:20.092 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-17T07:02:20.093 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-17T07:02:20.094 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-17T07:02:20.095 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-17T07:02:20.096 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-17T07:02:20.097 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-17T07:02:20.098 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-17T07:02:20.099 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-17T07:02:20.100 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-17T07:02:20.101 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-17T07:02:20.102 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-17T07:02:20.103 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-17T07:02:20.104 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-17T07:02:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-17T07:02:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-17T07:02:20.107 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-17T07:02:20.108 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-17T07:02:20.109 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-17T07:02:20.110 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-17T07:02:20.111 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-17T07:02:20.112 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-17T07:02:20.113 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-17T07:02:20.114 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-17T07:02:20.115 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-17T07:02:20.116 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-17T07:02:20.117 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-17T07:02:20.118 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-17T07:02:20.119 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-17T07:02:20.120 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-17T07:02:20.121 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-17T07:02:20.122 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-17T07:02:20.123 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-17T07:02:20.124 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-17T07:02:20.125 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-17T07:02:20.126 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-17T07:02:20.127 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-17T07:02:20.128 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-17T07:02:20.129 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-17T07:02:20.130 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-17T07:02:20.131 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-17T07:02:20.132 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-17T07:02:20.133 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-17T07:02:20.134 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-17T07:02:20.135 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-17T07:02:20.136 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-17T07:02:20.137 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-17T07:02:20.138 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-17T07:02:20.139 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-17T07:02:20.140 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-17T07:02:20.141 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-17T07:02:20.142 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-17T07:02:20.143 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-17T07:02:20.144 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-17T07:02:20.145 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-17T07:02:20.146 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-17T07:02:20.147 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-17T07:02:20.148 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-17T07:02:20.149 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-17T07:02:20.150 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-17T07:02:20.151 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-17T07:02:20.152 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-17T07:02:20.153 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-17T07:02:20.154 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-17T07:02:20.155 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-17T07:02:20.156 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-17T07:02:20.157 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-17T07:02:20.158 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-17T07:02:20.159 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-17T07:02:20.160 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-17T07:02:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-17T07:02:20.162 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-17T07:02:20.163 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-17T07:02:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-17T07:02:20.165 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-17T07:02:20.166 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-17T07:02:20.167 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-17T07:02:20.168 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-17T07:02:20.169 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-17T07:02:20.170 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-17T07:02:20.171 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-17T07:02:20.172 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-17T07:02:20.173 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-17T07:02:20.174 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-17T07:02:20.175 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-17T07:02:20.176 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-17T07:02:20.177 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-17T07:02:20.178 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-17T07:02:20.179 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-17T07:02:20.180 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-17T07:02:20.181 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-17T07:02:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-17T07:02:20.183 INFO:teuthology.orchestra.run.smithi187.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-17T07:02:20.187 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.187 DEBUG:teuthology.orchestra.run.smithi187:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout:postinstall scriptlet (using /bin/sh): 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout:for file in vmlinuz System.map config; do 2024-04-17T07:02:20.208 INFO:teuthology.orchestra.run.smithi187.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: fi 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:done 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-17T07:02:20.209 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.210 INFO:teuthology.orchestra.run.smithi187.stdout:fi 2024-04-17T07:02:20.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-04-17T07:02:20.284 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-04-17T07:02:20.284 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 68.el9 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8264254 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-04-17T07:02:20.285 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:02:20.286 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-04-17T07:02:20.287 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-17T07:02:20.287 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:20.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-17T07:02:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-04-17T07:02:22.284 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-17T07:02:22.312 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-04-17T07:02:22.314 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-17T07:02:22.329 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.n7WnU45zRo 2024-04-17T07:02:22.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n7WnU45zRo 2024-04-17T07:02:22.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.n7WnU45zRo 2024-04-17T07:02:22.540 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.n7WnU45zRo is 6KB 2024-04-17T07:02:22.559 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.n7WnU45zRo 2024-04-17T07:02:22.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-04-17T07:02:22.642 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-17T07:02:22.642 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-17T07:02:22.642 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-17T07:02:22.642 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-17T07:02:22.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:02:22.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-04-17T07:02:42.664 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-17T07:02:42.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:02:42.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:02:52.813 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-17T07:02:52.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:02:52.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:03:17.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-17T07:03:26.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:03:26.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:03:27.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-17T07:03:36.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:03:36.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:03:44.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-17T07:03:54.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-04-17T07:03:56.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:03:56.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:03:59.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-17T07:04:06.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:04:06.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:04:10.384 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-17T07:04:11.019 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:04:11.020 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-17T07:04:11.020 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-17T07:04:11.058 INFO:teuthology.orchestra.run.smithi187.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:04:11.059 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:04:11.059 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-17T07:04:11.059 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-17T07:04:12.060 DEBUG:teuthology.parallel:result is None 2024-04-17T07:04:14.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:04:14.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:04:15.201 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-17T07:04:15.852 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:04:15.852 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-17T07:04:15.853 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-17T07:04:15.889 INFO:teuthology.orchestra.run.smithi067.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:04:15.889 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T07:04:15.889 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-17T07:04:15.889 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-17T07:04:16.890 DEBUG:teuthology.parallel:result is None 2024-04-17T07:04:16.891 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-17T07:04:16.904 INFO:teuthology.task.internal:Creating test directory... 2024-04-17T07:04:16.904 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T07:04:16.909 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T07:04:16.963 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-17T07:04:16.969 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-17T07:04:17.002 INFO:teuthology.task.internal:Creating archive directory... 2024-04-17T07:04:17.003 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T07:04:17.006 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T07:04:17.068 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-17T07:04:17.082 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-17T07:04:17.082 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T07:04:17.094 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T07:04:17.237 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T07:04:17.242 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T07:04:17.298 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T07:04:17.379 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T07:04:17.384 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-17T07:04:17.391 INFO:teuthology.task.internal:Configuring sudo... 2024-04-17T07:04:17.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T07:04:17.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T07:04:17.488 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-17T07:04:17.498 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-17T07:04:17.498 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T07:04:17.502 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T07:04:17.543 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T07:04:17.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T07:04:17.751 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T07:04:17.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T07:04:17.924 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:04:17.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T07:04:18.013 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T07:04:18.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T07:04:18.162 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T07:04:18.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T07:04:18.317 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:04:18.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T07:04:18.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-17T07:04:18.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-04-17T07:04:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-17T07:04:18.522 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-17T07:04:18.878 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-17T07:04:18.886 INFO:teuthology.task.internal:Starting timer... 2024-04-17T07:04:18.886 INFO:teuthology.run_tasks:Running task pcp... 2024-04-17T07:04:18.940 INFO:teuthology.run_tasks:Running task selinux... 2024-04-17T07:04:18.988 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-17T07:04:18.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-04-17T07:04:19.107 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-04-17T07:04:19.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-04-17T07:04:19.215 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-04-17T07:04:19.219 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-17T07:04:19.219 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-04-17T07:04:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-04-17T07:04:19.277 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-04-17T07:04:19.321 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-04-17T07:04:19.321 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-04-17T07:04:19.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-17T07:04:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:04:19.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-17T07:04:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:04:19.572 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-17T07:04:19.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-04-17T07:04:19.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-04-17T07:04:19.800 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-17T07:04:19.923 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-17T07:04:19.931 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-17T07:04:20.114 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-17T07:04:20.135 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-17T07:04:20.136 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-17T07:10:32.664 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-04-17T07:10:32.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:10:32.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:10:32.819 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-17T07:10:32.995 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T07:10:32.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:10:32.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T07:10:33.139 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-17T07:10:33.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-17T07:10:33.299 INFO:teuthology.run_tasks:Running task clock... 2024-04-17T07:10:33.311 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-17T07:10:33.311 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T07:10:33.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T07:10:33.317 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T07:10:33.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T07:10:33.416 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-17T07:10:33.423 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-17T07:10:33.468 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-17T07:10:33.476 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-17T07:10:33.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-04-17T07:10:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-04-17T07:10:33.596 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-04-17T07:10:33.615 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-04-17T07:10:33.647 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-17T07:10:33.672 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-17T07:10:33.699 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-17T07:10:33.728 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-17T07:10:33.839 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-17T07:10:33.847 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-17T07:10:33.847 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-17T07:10:33.847 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.848 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.848 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.848 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-17T07:10:33.892 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-17T07:10:33.892 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-17T07:10:33.893 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.893 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.893 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.893 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T07:10:33.894 INFO:teuthology.run_tasks:Running task install... 2024-04-17T07:10:33.904 DEBUG:teuthology.task.install:project ceph 2024-04-17T07:10:33.904 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}} 2024-04-17T07:10:33.904 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'} 2024-04-17T07:10:33.904 INFO:teuthology.task.install:Using flavor: default 2024-04-17T07:10:33.910 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-17T07:10:33.910 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-17T07:10:33.911 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'tag': None, 'wait_for_package': False} 2024-04-17T07:10:33.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a8c84f5997950fd7d008204839253291d52e6eda 2024-04-17T07:10:33.914 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'tag': None, 'wait_for_package': False} 2024-04-17T07:10:33.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a8c84f5997950fd7d008204839253291d52e6eda 2024-04-17T07:10:34.067 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/ 2024-04-17T07:10:34.067 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3133.ga8c84f59 2024-04-17T07:10:34.073 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/ 2024-04-17T07:10:34.074 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3133.ga8c84f59 2024-04-17T07:10:34.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-17T07:10:34.246 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:10:34.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-17T07:10:34.250 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735/a8c84f5997950fd7d008204839253291d52e6eda/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-17T07:10:34.250 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:10:34.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-17T07:10:34.347 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-17T07:10:34.348 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8c84f5997950fd7d008204839253291d52e6eda/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-17T07:10:34.358 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-17T07:10:34.358 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8c84f5997950fd7d008204839253291d52e6eda/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-17T07:10:34.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-17T07:10:34.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-17T07:10:34.620 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-17T07:10:34.647 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-17T07:10:34.709 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-04-17T07:10:34.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-04-17T07:10:34.742 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-04-17T07:10:34.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-04-17T07:10:35.057 INFO:teuthology.orchestra.run.smithi187.stdout:74 files removed 2024-04-17T07:10:35.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-17T07:10:35.118 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-04-17T07:10:35.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-17T07:10:36.095 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 170 kB/s | 84 kB 00:00 2024-04-17T07:10:36.339 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 125 kB/s | 84 kB 00:00 2024-04-17T07:10:36.646 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-17T07:10:36.980 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-04-17T07:10:37.213 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-17T07:10:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-04-17T07:10:38.057 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-04-17T07:10:38.377 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 8.8 MB/s | 6.7 MB 00:00 2024-04-17T07:10:40.117 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-04-17T07:10:40.402 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-04-17T07:10:42.471 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-17T07:10:44.006 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-17T07:10:47.082 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-17T07:10:58.498 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-04-17T07:11:00.974 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 642 kB/s | 6.2 MB 00:09 2024-04-17T07:11:02.641 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-04-17T07:11:03.096 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-04-17T07:11:04.165 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-17T07:11:04.889 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-17T07:11:06.377 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-17T07:11:10.210 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-17T07:11:12.307 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-17T07:11:13.228 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-17T07:11:13.229 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-17T07:11:13.249 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-17T07:11:13.250 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-17T07:11:13.252 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-17T07:11:13.253 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-17T07:11:13.256 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-17T07:11:13.259 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-17T07:11:13.260 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-17T07:11:13.357 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 6.6 k 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 5.4 M 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 893 k 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 145 k 2024-04-17T07:11:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 1.5 M 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 142 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 3.5 M 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 7.4 M 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 49 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 11 M 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 49 M 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 273 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 763 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 225 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 27 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 18 k 2024-04-17T07:11:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 32 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 748 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 126 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-17T07:11:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 162 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 321 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 299 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 99 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 85 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.1 M 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 170 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-17T07:11:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.4 M 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.1 M 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-17T07:11:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 21 M 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 34 k 2024-04-17T07:11:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 2.4 M 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 249 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 4.7 M 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 17 M 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 16 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 25 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-17T07:11:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 163 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 501 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 5.4 M 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-17T07:11:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-17T07:11:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 45 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 135 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-17T07:11:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-17T07:11:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-17T07:11:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-17T07:11:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:11:13.397 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout:Install 150 Packages 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 24 Packages 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 229 M 2024-04-17T07:11:13.398 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-17T07:11:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:(1/174): ceph-19.0.0-3133.ga8c84f59.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-17T07:11:14.944 INFO:teuthology.orchestra.run.smithi067.stdout:(2/174): ceph-base-19.0.0-3133.ga8c84f59.el9.x8 6.7 MB/s | 5.4 MB 00:00 2024-04-17T07:11:15.011 INFO:teuthology.orchestra.run.smithi067.stdout:(3/174): ceph-immutable-object-cache-19.0.0-313 2.2 MB/s | 145 kB 00:00 2024-04-17T07:11:15.290 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-17T07:11:15.291 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-17T07:11:15.312 INFO:teuthology.orchestra.run.smithi187.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-17T07:11:15.313 INFO:teuthology.orchestra.run.smithi187.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-17T07:11:15.315 INFO:teuthology.orchestra.run.smithi187.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-17T07:11:15.316 INFO:teuthology.orchestra.run.smithi187.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-17T07:11:15.318 INFO:teuthology.orchestra.run.smithi187.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-17T07:11:15.322 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-17T07:11:15.323 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-17T07:11:15.337 INFO:teuthology.orchestra.run.smithi067.stdout:(4/174): ceph-mds-19.0.0-3133.ga8c84f59.el9.x86 7.3 MB/s | 2.4 MB 00:00 2024-04-17T07:11:15.410 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-17T07:11:15.435 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-17T07:11:15.435 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-17T07:11:15.435 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-17T07:11:15.435 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 6.6 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 5.4 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 893 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 145 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 1.5 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 142 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 3.5 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 7.4 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 49 k 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 11 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 49 M 2024-04-17T07:11:15.436 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 273 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 763 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 225 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-shell x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 27 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 18 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 32 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 748 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 126 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-17T07:11:15.437 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 162 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 321 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 299 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 99 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 85 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.1 M 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 170 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-17T07:11:15.438 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.4 M 2024-04-17T07:11:15.439 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 3.1 M 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-17T07:11:15.440 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 21 M 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 34 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 2.4 M 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 249 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 4.7 M 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 17 M 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3133.ga8c84f59.el9 ceph-noarch 16 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 25 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-17T07:11:15.441 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 163 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 501 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 5.4 M 2024-04-17T07:11:15.442 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-17T07:11:15.443 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 45 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.0.0-3133.ga8c84f59.el9 ceph 135 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-17T07:11:15.444 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-17T07:11:15.445 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-17T07:11:15.446 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-17T07:11:15.447 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-17T07:11:15.448 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================ 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout:Install 150 Packages 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 24 Packages 2024-04-17T07:11:15.449 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:11:15.450 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 229 M 2024-04-17T07:11:15.450 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-17T07:11:15.487 INFO:teuthology.orchestra.run.smithi067.stdout:(5/174): ceph-mgr-19.0.0-3133.ga8c84f59.el9.x86 10 MB/s | 1.5 MB 00:00 2024-04-17T07:11:15.737 INFO:teuthology.orchestra.run.smithi067.stdout:(6/174): ceph-fuse-19.0.0-3133.ga8c84f59.el9.x8 705 kB/s | 893 kB 00:01 2024-04-17T07:11:16.029 INFO:teuthology.orchestra.run.smithi067.stdout:(7/174): ceph-common-19.0.0-3133.ga8c84f59.el9. 11 MB/s | 21 MB 00:01 2024-04-17T07:11:16.146 INFO:teuthology.orchestra.run.smithi067.stdout:(8/174): ceph-mon-19.0.0-3133.ga8c84f59.el9.x86 7.1 MB/s | 4.7 MB 00:00 2024-04-17T07:11:16.221 INFO:teuthology.orchestra.run.smithi067.stdout:(9/174): ceph-selinux-19.0.0-3133.ga8c84f59.el9 341 kB/s | 25 kB 00:00 2024-04-17T07:11:16.525 INFO:teuthology.orchestra.run.smithi187.stdout:(1/174): ceph-19.0.0-3133.ga8c84f59.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-17T07:11:16.922 INFO:teuthology.orchestra.run.smithi067.stdout:(10/174): ceph-radosgw-19.0.0-3133.ga8c84f59.el 12 MB/s | 11 MB 00:00 2024-04-17T07:11:16.980 INFO:teuthology.orchestra.run.smithi067.stdout:(11/174): cephfs-mirror-19.0.0-3133.ga8c84f59.e 3.8 MB/s | 225 kB 00:00 2024-04-17T07:11:17.009 INFO:teuthology.orchestra.run.smithi187.stdout:(2/174): ceph-fuse-19.0.0-3133.ga8c84f59.el9.x8 1.8 MB/s | 893 kB 00:00 2024-04-17T07:11:17.072 INFO:teuthology.orchestra.run.smithi067.stdout:(12/174): cephfs-shell-19.0.0-3133.ga8c84f59.el 298 kB/s | 27 kB 00:00 2024-04-17T07:11:17.101 INFO:teuthology.orchestra.run.smithi187.stdout:(3/174): ceph-immutable-object-cache-19.0.0-313 1.6 MB/s | 145 kB 00:00 2024-04-17T07:11:17.172 INFO:teuthology.orchestra.run.smithi067.stdout:(13/174): libcephfs-devel-19.0.0-3133.ga8c84f59 320 kB/s | 32 kB 00:00 2024-04-17T07:11:17.826 INFO:teuthology.orchestra.run.smithi187.stdout:(4/174): ceph-mds-19.0.0-3133.ga8c84f59.el9.x86 3.3 MB/s | 2.4 MB 00:00 2024-04-17T07:11:17.847 INFO:teuthology.orchestra.run.smithi067.stdout:(14/174): libcephfs2-19.0.0-3133.ga8c84f59.el9. 1.1 MB/s | 748 kB 00:00 2024-04-17T07:11:17.931 INFO:teuthology.orchestra.run.smithi067.stdout:(15/174): libcephsqlite-19.0.0-3133.ga8c84f59.e 1.9 MB/s | 163 kB 00:00 2024-04-17T07:11:18.015 INFO:teuthology.orchestra.run.smithi067.stdout:(16/174): librados-devel-19.0.0-3133.ga8c84f59. 1.5 MB/s | 126 kB 00:00 2024-04-17T07:11:18.140 INFO:teuthology.orchestra.run.smithi067.stdout:(17/174): libradosstriper1-19.0.0-3133.ga8c84f5 3.9 MB/s | 501 kB 00:00 2024-04-17T07:11:18.326 INFO:teuthology.orchestra.run.smithi187.stdout:(5/174): ceph-mgr-19.0.0-3133.ga8c84f59.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-04-17T07:11:18.807 INFO:teuthology.orchestra.run.smithi067.stdout:(18/174): librgw2-19.0.0-3133.ga8c84f59.el9.x86 8.1 MB/s | 5.4 MB 00:00 2024-04-17T07:11:18.866 INFO:teuthology.orchestra.run.smithi067.stdout:(19/174): python3-ceph-argparse-19.0.0-3133.ga8 787 kB/s | 45 kB 00:00 2024-04-17T07:11:18.941 INFO:teuthology.orchestra.run.smithi067.stdout:(20/174): python3-ceph-common-19.0.0-3133.ga8c8 1.8 MB/s | 135 kB 00:00 2024-04-17T07:11:19.024 INFO:teuthology.orchestra.run.smithi067.stdout:(21/174): python3-cephfs-19.0.0-3133.ga8c84f59. 1.9 MB/s | 162 kB 00:00 2024-04-17T07:11:19.100 INFO:teuthology.orchestra.run.smithi067.stdout:(22/174): python3-rados-19.0.0-3133.ga8c84f59.e 4.2 MB/s | 321 kB 00:00 2024-04-17T07:11:19.167 INFO:teuthology.orchestra.run.smithi067.stdout:(23/174): python3-rbd-19.0.0-3133.ga8c84f59.el9 4.4 MB/s | 299 kB 00:00 2024-04-17T07:11:19.225 INFO:teuthology.orchestra.run.smithi067.stdout:(24/174): python3-rgw-19.0.0-3133.ga8c84f59.el9 1.7 MB/s | 99 kB 00:00 2024-04-17T07:11:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:(25/174): rbd-fuse-19.0.0-3133.ga8c84f59.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-17T07:11:19.351 INFO:teuthology.orchestra.run.smithi187.stdout:(6/174): ceph-common-19.0.0-3133.ga8c84f59.el9. 6.5 MB/s | 21 MB 00:03 2024-04-17T07:11:19.668 INFO:teuthology.orchestra.run.smithi067.stdout:(26/174): rbd-mirror-19.0.0-3133.ga8c84f59.el9. 8.1 MB/s | 3.1 MB 00:00 2024-04-17T07:11:19.743 INFO:teuthology.orchestra.run.smithi067.stdout:(27/174): rbd-nbd-19.0.0-3133.ga8c84f59.el9.x86 2.2 MB/s | 170 kB 00:00 2024-04-17T07:11:19.768 INFO:teuthology.orchestra.run.smithi187.stdout:(7/174): ceph-mon-19.0.0-3133.ga8c84f59.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-04-17T07:11:19.818 INFO:teuthology.orchestra.run.smithi067.stdout:(28/174): ceph-grafana-dashboards-19.0.0-3133.g 461 kB/s | 34 kB 00:00 2024-04-17T07:11:19.885 INFO:teuthology.orchestra.run.smithi067.stdout:(29/174): ceph-mgr-cephadm-19.0.0-3133.ga8c84f5 2.1 MB/s | 142 kB 00:00 2024-04-17T07:11:20.427 INFO:teuthology.orchestra.run.smithi067.stdout:(30/174): ceph-mgr-dashboard-19.0.0-3133.ga8c84 6.5 MB/s | 3.5 MB 00:00 2024-04-17T07:11:20.969 INFO:teuthology.orchestra.run.smithi067.stdout:(31/174): ceph-test-19.0.0-3133.ga8c84f59.el9.x 10 MB/s | 49 MB 00:04 2024-04-17T07:11:21.069 INFO:teuthology.orchestra.run.smithi067.stdout:(32/174): ceph-mgr-modules-core-19.0.0-3133.ga8 2.5 MB/s | 249 kB 00:00 2024-04-17T07:11:21.128 INFO:teuthology.orchestra.run.smithi067.stdout:(33/174): ceph-mgr-rook-19.0.0-3133.ga8c84f59.e 850 kB/s | 49 kB 00:00 2024-04-17T07:11:21.151 INFO:teuthology.orchestra.run.smithi187.stdout:(8/174): ceph-osd-19.0.0-3133.ga8c84f59.el9.x86 9.4 MB/s | 17 MB 00:01 2024-04-17T07:11:21.186 INFO:teuthology.orchestra.run.smithi067.stdout:(34/174): ceph-prometheus-alerts-19.0.0-3133.ga 286 kB/s | 16 kB 00:00 2024-04-17T07:11:21.252 INFO:teuthology.orchestra.run.smithi187.stdout:(9/174): ceph-selinux-19.0.0-3133.ga8c84f59.el9 254 kB/s | 25 kB 00:00 2024-04-17T07:11:21.278 INFO:teuthology.orchestra.run.smithi067.stdout:(35/174): ceph-volume-19.0.0-3133.ga8c84f59.el9 2.9 MB/s | 273 kB 00:00 2024-04-17T07:11:21.454 INFO:teuthology.orchestra.run.smithi067.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 7.2 MB/s | 7.4 MB 00:01 2024-04-17T07:11:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:(37/174): cephadm-19.0.0-3133.ga8c84f59.el9.noa 3.0 MB/s | 763 kB 00:00 2024-04-17T07:11:21.554 INFO:teuthology.orchestra.run.smithi067.stdout:(38/174): cephfs-top-19.0.0-3133.ga8c84f59.el9. 184 kB/s | 18 kB 00:00 2024-04-17T07:11:21.960 INFO:teuthology.orchestra.run.smithi187.stdout:(10/174): ceph-radosgw-19.0.0-3133.ga8c84f59.el 4.8 MB/s | 11 MB 00:02 2024-04-17T07:11:22.000 INFO:teuthology.orchestra.run.smithi067.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.4 MB/s | 662 kB 00:00 2024-04-17T07:11:22.054 INFO:teuthology.orchestra.run.smithi187.stdout:(11/174): cephfs-mirror-19.0.0-3133.ga8c84f59.e 2.4 MB/s | 225 kB 00:00 2024-04-17T07:11:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 1.7 MB/s | 920 kB 00:00 2024-04-17T07:11:22.113 INFO:teuthology.orchestra.run.smithi067.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 968 kB/s | 106 kB 00:00 2024-04-17T07:11:22.272 INFO:teuthology.orchestra.run.smithi067.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 307 kB 00:00 2024-04-17T07:11:22.297 INFO:teuthology.orchestra.run.smithi067.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 173 kB/s | 32 kB 00:00 2024-04-17T07:11:22.339 INFO:teuthology.orchestra.run.smithi067.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 426 kB/s | 17 kB 00:00 2024-04-17T07:11:22.428 INFO:teuthology.orchestra.run.smithi187.stdout:(12/174): ceph-base-19.0.0-3133.ga8c84f59.el9.x 871 kB/s | 5.4 MB 00:06 2024-04-17T07:11:22.536 INFO:teuthology.orchestra.run.smithi187.stdout:(13/174): cephfs-shell-19.0.0-3133.ga8c84f59.el 56 kB/s | 27 kB 00:00 2024-04-17T07:11:22.589 INFO:teuthology.orchestra.run.smithi067.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 41 kB/s | 10 kB 00:00 2024-04-17T07:11:22.645 INFO:teuthology.orchestra.run.smithi187.stdout:(14/174): libcephfs-devel-19.0.0-3133.ga8c84f59 146 kB/s | 32 kB 00:00 2024-04-17T07:11:22.698 INFO:teuthology.orchestra.run.smithi067.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.5 MB/s | 3.0 MB 00:00 2024-04-17T07:11:22.723 INFO:teuthology.orchestra.run.smithi067.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-04-17T07:11:22.749 INFO:teuthology.orchestra.run.smithi067.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 898 kB/s | 44 kB 00:00 2024-04-17T07:11:22.753 INFO:teuthology.orchestra.run.smithi187.stdout:(15/174): libcephfs2-19.0.0-3133.ga8c84f59.el9. 3.4 MB/s | 748 kB 00:00 2024-04-17T07:11:22.791 INFO:teuthology.orchestra.run.smithi067.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-04-17T07:11:22.829 INFO:teuthology.orchestra.run.smithi187.stdout:(16/174): libcephsqlite-19.0.0-3133.ga8c84f59.e 893 kB/s | 163 kB 00:00 2024-04-17T07:11:22.833 INFO:teuthology.orchestra.run.smithi067.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-04-17T07:11:22.858 INFO:teuthology.orchestra.run.smithi067.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 764 kB/s | 46 kB 00:00 2024-04-17T07:11:22.887 INFO:teuthology.orchestra.run.smithi187.stdout:(17/174): librados-devel-19.0.0-3133.ga8c84f59. 953 kB/s | 126 kB 00:00 2024-04-17T07:11:22.891 INFO:teuthology.orchestra.run.smithi067.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-04-17T07:11:22.917 INFO:teuthology.orchestra.run.smithi067.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.3 MB/s | 187 kB 00:00 2024-04-17T07:11:22.950 INFO:teuthology.orchestra.run.smithi067.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.8 MB/s | 217 kB 00:00 2024-04-17T07:11:22.992 INFO:teuthology.orchestra.run.smithi067.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 472 kB/s | 34 kB 00:00 2024-04-17T07:11:23.193 INFO:teuthology.orchestra.run.smithi067.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-17T07:11:23.251 INFO:teuthology.orchestra.run.smithi067.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 6.2 MB/s | 1.5 MB 00:00 2024-04-17T07:11:23.271 INFO:teuthology.orchestra.run.smithi187.stdout:(18/174): libradosstriper1-19.0.0-3133.ga8c84f5 1.1 MB/s | 501 kB 00:00 2024-04-17T07:11:23.301 INFO:teuthology.orchestra.run.smithi067.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 8.5 MB/s | 817 kB 00:00 2024-04-17T07:11:23.327 INFO:teuthology.orchestra.run.smithi067.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 132 kB 00:00 2024-04-17T07:11:23.329 INFO:teuthology.orchestra.run.smithi187.stdout:(19/174): python3-ceph-argparse-19.0.0-3133.ga8 786 kB/s | 45 kB 00:00 2024-04-17T07:11:23.597 INFO:teuthology.orchestra.run.smithi187.stdout:(20/174): python3-ceph-common-19.0.0-3133.ga8c8 506 kB/s | 135 kB 00:00 2024-04-17T07:11:23.627 INFO:teuthology.orchestra.run.smithi067.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.7 MB 00:00 2024-04-17T07:11:23.669 INFO:teuthology.orchestra.run.smithi067.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-04-17T07:11:23.694 INFO:teuthology.orchestra.run.smithi067.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-17T07:11:23.830 INFO:teuthology.orchestra.run.smithi187.stdout:(21/174): python3-cephfs-19.0.0-3133.ga8c84f59. 698 kB/s | 162 kB 00:00 2024-04-17T07:11:23.946 INFO:teuthology.orchestra.run.smithi067.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-17T07:11:23.970 INFO:teuthology.orchestra.run.smithi067.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 90 kB/s | 24 kB 00:00 2024-04-17T07:11:24.012 INFO:teuthology.orchestra.run.smithi067.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 448 kB/s | 18 kB 00:00 2024-04-17T07:11:24.037 INFO:teuthology.orchestra.run.smithi067.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-04-17T07:11:24.062 INFO:teuthology.orchestra.run.smithi067.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 651 kB/s | 32 kB 00:00 2024-04-17T07:11:24.130 INFO:teuthology.orchestra.run.smithi187.stdout:(22/174): python3-rados-19.0.0-3133.ga8c84f59.e 1.0 MB/s | 321 kB 00:00 2024-04-17T07:11:24.246 INFO:teuthology.orchestra.run.smithi067.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-04-17T07:11:24.279 INFO:teuthology.orchestra.run.smithi067.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-04-17T07:11:24.313 INFO:teuthology.orchestra.run.smithi067.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.7 MB/s | 174 kB 00:00 2024-04-17T07:11:24.338 INFO:teuthology.orchestra.run.smithi067.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-04-17T07:11:24.364 INFO:teuthology.orchestra.run.smithi067.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 41 kB 00:00 2024-04-17T07:11:24.397 INFO:teuthology.orchestra.run.smithi067.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-04-17T07:11:24.414 INFO:teuthology.orchestra.run.smithi067.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 132 kB 00:00 2024-04-17T07:11:24.422 INFO:teuthology.orchestra.run.smithi187.stdout:(23/174): librgw2-19.0.0-3133.ga8c84f59.el9.x86 3.5 MB/s | 5.4 MB 00:01 2024-04-17T07:11:24.456 INFO:teuthology.orchestra.run.smithi187.stdout:(24/174): python3-rbd-19.0.0-3133.ga8c84f59.el9 921 kB/s | 299 kB 00:00 2024-04-17T07:11:24.490 INFO:teuthology.orchestra.run.smithi187.stdout:(25/174): python3-rgw-19.0.0-3133.ga8c84f59.el9 1.5 MB/s | 99 kB 00:00 2024-04-17T07:11:24.548 INFO:teuthology.orchestra.run.smithi067.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 211 kB 00:00 2024-04-17T07:11:24.557 INFO:teuthology.orchestra.run.smithi187.stdout:(26/174): rbd-fuse-19.0.0-3133.ga8c84f59.el9.x8 853 kB/s | 85 kB 00:00 2024-04-17T07:11:24.573 INFO:teuthology.orchestra.run.smithi067.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 304 kB/s | 47 kB 00:00 2024-04-17T07:11:24.599 INFO:teuthology.orchestra.run.smithi067.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 870 kB/s | 43 kB 00:00 2024-04-17T07:11:24.715 INFO:teuthology.orchestra.run.smithi187.stdout:(27/174): rbd-nbd-19.0.0-3133.ga8c84f59.el9.x86 1.1 MB/s | 170 kB 00:00 2024-04-17T07:11:24.807 INFO:teuthology.orchestra.run.smithi187.stdout:(28/174): ceph-grafana-dashboards-19.0.0-3133.g 377 kB/s | 34 kB 00:00 2024-04-17T07:11:24.882 INFO:teuthology.orchestra.run.smithi067.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 140 kB/s | 39 kB 00:00 2024-04-17T07:11:24.983 INFO:teuthology.orchestra.run.smithi187.stdout:(29/174): ceph-mgr-cephadm-19.0.0-3133.ga8c84f5 816 kB/s | 142 kB 00:00 2024-04-17T07:11:25.125 INFO:teuthology.orchestra.run.smithi067.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-17T07:11:25.141 INFO:teuthology.orchestra.run.smithi067.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 168 kB/s | 41 kB 00:00 2024-04-17T07:11:25.175 INFO:teuthology.orchestra.run.smithi067.stdout:(81/174): qatlib-service-24.02.0-1.el9.x86_64.r 1.1 MB/s | 35 kB 00:00 2024-04-17T07:11:25.200 INFO:teuthology.orchestra.run.smithi067.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-04-17T07:11:25.217 INFO:teuthology.orchestra.run.smithi067.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.2 MB/s | 46 kB 00:00 2024-04-17T07:11:25.251 INFO:teuthology.orchestra.run.smithi067.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-04-17T07:11:25.268 INFO:teuthology.orchestra.run.smithi067.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.2 MB/s | 56 kB 00:00 2024-04-17T07:11:25.325 INFO:teuthology.orchestra.run.smithi187.stdout:(30/174): rbd-mirror-19.0.0-3133.ga8c84f59.el9. 3.7 MB/s | 3.1 MB 00:00 2024-04-17T07:11:25.401 INFO:teuthology.orchestra.run.smithi067.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 421 kB/s | 63 kB 00:00 2024-04-17T07:11:25.418 INFO:teuthology.orchestra.run.smithi067.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 296 kB/s | 44 kB 00:00 2024-04-17T07:11:25.443 INFO:teuthology.orchestra.run.smithi067.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-04-17T07:11:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 241 kB 00:00 2024-04-17T07:11:25.636 INFO:teuthology.orchestra.run.smithi067.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.1 MB 00:00 2024-04-17T07:11:25.661 INFO:teuthology.orchestra.run.smithi067.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-04-17T07:11:25.753 INFO:teuthology.orchestra.run.smithi067.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-04-17T07:11:25.862 INFO:teuthology.orchestra.run.smithi067.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 940 kB/s | 187 kB 00:00 2024-04-17T07:11:25.925 INFO:teuthology.orchestra.run.smithi067.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-17T07:11:26.004 INFO:teuthology.orchestra.run.smithi067.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-17T07:11:26.054 INFO:teuthology.orchestra.run.smithi067.stdout:(96/174): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-17T07:11:26.121 INFO:teuthology.orchestra.run.smithi067.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-04-17T07:11:26.146 INFO:teuthology.orchestra.run.smithi067.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 236 kB/s | 21 kB 00:00 2024-04-17T07:11:26.222 INFO:teuthology.orchestra.run.smithi067.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 256 kB/s | 25 kB 00:00 2024-04-17T07:11:26.339 INFO:teuthology.orchestra.run.smithi067.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.1 MB/s | 245 kB 00:00 2024-04-17T07:11:26.397 INFO:teuthology.orchestra.run.smithi067.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 4.2 MB/s | 248 kB 00:00 2024-04-17T07:11:26.506 INFO:teuthology.orchestra.run.smithi067.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-04-17T07:11:26.531 INFO:teuthology.orchestra.run.smithi067.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 706 kB/s | 94 kB 00:00 2024-04-17T07:11:26.773 INFO:teuthology.orchestra.run.smithi067.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-04-17T07:11:27.298 INFO:teuthology.orchestra.run.smithi067.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-04-17T07:11:27.341 INFO:teuthology.orchestra.run.smithi067.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 967 kB/s | 548 kB 00:00 2024-04-17T07:11:27.491 INFO:teuthology.orchestra.run.smithi067.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 131 kB/s | 19 kB 00:00 2024-04-17T07:11:27.783 INFO:teuthology.orchestra.run.smithi067.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 639 kB/s | 309 kB 00:00 2024-04-17T07:11:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:(31/174): ceph-test-19.0.0-3133.ga8c84f59.el9.x 7.4 MB/s | 49 MB 00:06 2024-04-17T07:11:27.928 INFO:teuthology.orchestra.run.smithi067.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-17T07:11:27.958 INFO:teuthology.orchestra.run.smithi067.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 149 kB/s | 26 kB 00:00 2024-04-17T07:11:27.983 INFO:teuthology.orchestra.run.smithi187.stdout:(32/174): ceph-mgr-dashboard-19.0.0-3133.ga8c84 1.2 MB/s | 3.5 MB 00:02 2024-04-17T07:11:28.109 INFO:teuthology.orchestra.run.smithi187.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-04-17T07:11:28.142 INFO:teuthology.orchestra.run.smithi187.stdout:(34/174): ceph-mgr-modules-core-19.0.0-3133.ga8 1.0 MB/s | 249 kB 00:00 2024-04-17T07:11:28.158 INFO:teuthology.orchestra.run.smithi067.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 212 kB/s | 49 kB 00:00 2024-04-17T07:11:28.167 INFO:teuthology.orchestra.run.smithi187.stdout:(35/174): ceph-mgr-rook-19.0.0-3133.ga8c84f59.e 270 kB/s | 49 kB 00:00 2024-04-17T07:11:28.184 INFO:teuthology.orchestra.run.smithi067.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 300 kB/s | 67 kB 00:00 2024-04-17T07:11:28.218 INFO:teuthology.orchestra.run.smithi187.stdout:(36/174): ceph-prometheus-alerts-19.0.0-3133.ga 152 kB/s | 16 kB 00:00 2024-04-17T07:11:28.251 INFO:teuthology.orchestra.run.smithi187.stdout:(37/174): ceph-volume-19.0.0-3133.ga8c84f59.el9 2.5 MB/s | 273 kB 00:00 2024-04-17T07:11:28.284 INFO:teuthology.orchestra.run.smithi067.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-17T07:11:28.360 INFO:teuthology.orchestra.run.smithi187.stdout:(38/174): cephfs-top-19.0.0-3133.ga8c84f59.el9. 129 kB/s | 18 kB 00:00 2024-04-17T07:11:28.443 INFO:teuthology.orchestra.run.smithi067.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-04-17T07:11:28.485 INFO:teuthology.orchestra.run.smithi067.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-04-17T07:11:28.485 INFO:teuthology.orchestra.run.smithi187.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 920 kB 00:00 2024-04-17T07:11:28.519 INFO:teuthology.orchestra.run.smithi187.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-04-17T07:11:28.543 INFO:teuthology.orchestra.run.smithi067.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 438 kB/s | 43 kB 00:00 2024-04-17T07:11:28.544 INFO:teuthology.orchestra.run.smithi187.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-04-17T07:11:28.578 INFO:teuthology.orchestra.run.smithi187.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-04-17T07:11:28.585 INFO:teuthology.orchestra.run.smithi067.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2024-04-17T07:11:28.595 INFO:teuthology.orchestra.run.smithi187.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 676 kB/s | 32 kB 00:00 2024-04-17T07:11:28.644 INFO:teuthology.orchestra.run.smithi067.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-17T07:11:28.687 INFO:teuthology.orchestra.run.smithi187.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-17T07:11:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-17T07:11:28.712 INFO:teuthology.orchestra.run.smithi187.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 170 kB/s | 17 kB 00:00 2024-04-17T07:11:28.762 INFO:teuthology.orchestra.run.smithi187.stdout:(46/174): cephadm-19.0.0-3133.ga8c84f59.el9.noa 1.3 MB/s | 763 kB 00:00 2024-04-17T07:11:28.788 INFO:teuthology.orchestra.run.smithi187.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 136 kB/s | 10 kB 00:00 2024-04-17T07:11:28.813 INFO:teuthology.orchestra.run.smithi187.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-04-17T07:11:28.828 INFO:teuthology.orchestra.run.smithi067.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-04-17T07:11:28.838 INFO:teuthology.orchestra.run.smithi187.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 604 kB/s | 44 kB 00:00 2024-04-17T07:11:28.880 INFO:teuthology.orchestra.run.smithi187.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2024-04-17T07:11:28.886 INFO:teuthology.orchestra.run.smithi067.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-04-17T07:11:28.930 INFO:teuthology.orchestra.run.smithi187.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 578 kB 00:00 2024-04-17T07:11:28.945 INFO:teuthology.orchestra.run.smithi067.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 304 kB/s | 35 kB 00:00 2024-04-17T07:11:28.955 INFO:teuthology.orchestra.run.smithi187.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 400 kB/s | 46 kB 00:00 2024-04-17T07:11:29.006 INFO:teuthology.orchestra.run.smithi187.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-17T07:11:29.012 INFO:teuthology.orchestra.run.smithi067.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-04-17T07:11:29.040 INFO:teuthology.orchestra.run.smithi187.stdout:(54/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-04-17T07:11:29.065 INFO:teuthology.orchestra.run.smithi187.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2024-04-17T07:11:29.090 INFO:teuthology.orchestra.run.smithi187.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 432 kB/s | 34 kB 00:00 2024-04-17T07:11:29.170 INFO:teuthology.orchestra.run.smithi067.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 918 kB/s | 144 kB 00:00 2024-04-17T07:11:29.174 INFO:teuthology.orchestra.run.smithi187.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-04-17T07:11:29.238 INFO:teuthology.orchestra.run.smithi067.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.0 MB/s | 2.0 MB 00:00 2024-04-17T07:11:29.271 INFO:teuthology.orchestra.run.smithi067.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 108 kB/s | 11 kB 00:00 2024-04-17T07:11:29.291 INFO:teuthology.orchestra.run.smithi187.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-04-17T07:11:29.324 INFO:teuthology.orchestra.run.smithi187.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 817 kB 00:00 2024-04-17T07:11:29.338 INFO:teuthology.orchestra.run.smithi067.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-17T07:11:29.349 INFO:teuthology.orchestra.run.smithi187.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 820 kB/s | 132 kB 00:00 2024-04-17T07:11:29.372 INFO:teuthology.orchestra.run.smithi067.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 233 kB/s | 23 kB 00:00 2024-04-17T07:11:29.391 INFO:teuthology.orchestra.run.smithi187.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-04-17T07:11:29.455 INFO:teuthology.orchestra.run.smithi067.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 167 kB/s | 19 kB 00:00 2024-04-17T07:11:29.489 INFO:teuthology.orchestra.run.smithi067.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-04-17T07:11:29.558 INFO:teuthology.orchestra.run.smithi187.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.9 MB/s | 1.0 MB 00:00 2024-04-17T07:11:29.631 INFO:teuthology.orchestra.run.smithi067.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-17T07:11:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 279 kB/s | 46 kB 00:00 2024-04-17T07:11:29.700 INFO:teuthology.orchestra.run.smithi187.stdout:(63/174): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.7 MB 00:00 2024-04-17T07:11:29.723 INFO:teuthology.orchestra.run.smithi067.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 867 kB/s | 79 kB 00:00 2024-04-17T07:11:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-04-17T07:11:29.834 INFO:teuthology.orchestra.run.smithi187.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-04-17T07:11:29.859 INFO:teuthology.orchestra.run.smithi187.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 85 kB/s | 24 kB 00:00 2024-04-17T07:11:29.865 INFO:teuthology.orchestra.run.smithi067.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-04-17T07:11:29.884 INFO:teuthology.orchestra.run.smithi187.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 107 kB/s | 18 kB 00:00 2024-04-17T07:11:29.907 INFO:teuthology.orchestra.run.smithi067.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-04-17T07:11:29.951 INFO:teuthology.orchestra.run.smithi187.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-04-17T07:11:29.974 INFO:teuthology.orchestra.run.smithi067.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 836 kB/s | 90 kB 00:00 2024-04-17T07:11:29.976 INFO:teuthology.orchestra.run.smithi187.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 350 kB/s | 32 kB 00:00 2024-04-17T07:11:30.016 INFO:teuthology.orchestra.run.smithi067.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 262 kB/s | 28 kB 00:00 2024-04-17T07:11:30.093 INFO:teuthology.orchestra.run.smithi187.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-04-17T07:11:30.127 INFO:teuthology.orchestra.run.smithi187.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-04-17T07:11:30.133 INFO:teuthology.orchestra.run.smithi067.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 195 kB/s | 31 kB 00:00 2024-04-17T07:11:30.152 INFO:teuthology.orchestra.run.smithi187.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-04-17T07:11:30.177 INFO:teuthology.orchestra.run.smithi187.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 860 kB/s | 69 kB 00:00 2024-04-17T07:11:30.202 INFO:teuthology.orchestra.run.smithi187.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 857 kB/s | 41 kB 00:00 2024-04-17T07:11:30.228 INFO:teuthology.orchestra.run.smithi187.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 238 kB 00:00 2024-04-17T07:11:30.253 INFO:teuthology.orchestra.run.smithi187.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-17T07:11:30.287 INFO:teuthology.orchestra.run.smithi187.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 844 kB/s | 47 kB 00:00 2024-04-17T07:11:30.296 INFO:teuthology.orchestra.run.smithi067.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 676 kB/s | 188 kB 00:00 2024-04-17T07:11:30.320 INFO:teuthology.orchestra.run.smithi187.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-04-17T07:11:30.322 INFO:teuthology.orchestra.run.smithi067.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 316 kB/s | 59 kB 00:00 2024-04-17T07:11:30.346 INFO:teuthology.orchestra.run.smithi187.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 469 kB/s | 43 kB 00:00 2024-04-17T07:11:30.430 INFO:teuthology.orchestra.run.smithi067.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 270 kB/s | 36 kB 00:00 2024-04-17T07:11:30.456 INFO:teuthology.orchestra.run.smithi067.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 384 kB/s | 51 kB 00:00 2024-04-17T07:11:30.573 INFO:teuthology.orchestra.run.smithi067.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-04-17T07:11:30.598 INFO:teuthology.orchestra.run.smithi067.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 634 kB/s | 90 kB 00:00 2024-04-17T07:11:30.707 INFO:teuthology.orchestra.run.smithi067.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-17T07:11:30.721 INFO:teuthology.orchestra.run.smithi187.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-04-17T07:11:30.732 INFO:teuthology.orchestra.run.smithi067.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-04-17T07:11:30.746 INFO:teuthology.orchestra.run.smithi187.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 100 kB/s | 39 kB 00:00 2024-04-17T07:11:30.771 INFO:teuthology.orchestra.run.smithi187.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 100 kB/s | 41 kB 00:00 2024-04-17T07:11:30.805 INFO:teuthology.orchestra.run.smithi187.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-04-17T07:11:30.830 INFO:teuthology.orchestra.run.smithi187.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 836 kB/s | 46 kB 00:00 2024-04-17T07:11:30.855 INFO:teuthology.orchestra.run.smithi187.stdout:(84/174): qatlib-service-24.02.0-1.el9.x86_64.r 428 kB/s | 35 kB 00:00 2024-04-17T07:11:30.860 INFO:teuthology.orchestra.run.smithi067.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-17T07:11:30.881 INFO:teuthology.orchestra.run.smithi187.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-04-17T07:11:30.906 INFO:teuthology.orchestra.run.smithi187.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 784 kB/s | 56 kB 00:00 2024-04-17T07:11:30.941 INFO:teuthology.orchestra.run.smithi067.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-17T07:11:30.948 INFO:teuthology.orchestra.run.smithi187.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 677 kB/s | 44 kB 00:00 2024-04-17T07:11:30.973 INFO:teuthology.orchestra.run.smithi187.stdout:(88/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 691 kB/s | 63 kB 00:00 2024-04-17T07:11:30.998 INFO:teuthology.orchestra.run.smithi187.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 866 kB/s | 79 kB 00:00 2024-04-17T07:11:31.032 INFO:teuthology.orchestra.run.smithi187.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 241 kB 00:00 2024-04-17T07:11:31.057 INFO:teuthology.orchestra.run.smithi187.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-04-17T07:11:31.108 INFO:teuthology.orchestra.run.smithi187.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-04-17T07:11:31.133 INFO:teuthology.orchestra.run.smithi187.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-04-17T07:11:31.167 INFO:teuthology.orchestra.run.smithi187.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 187 kB 00:00 2024-04-17T07:11:31.300 INFO:teuthology.orchestra.run.smithi067.stdout:(150/174): ceph-osd-19.0.0-3133.ga8c84f59.el9.x 1.1 MB/s | 17 MB 00:15 2024-04-17T07:11:31.318 INFO:teuthology.orchestra.run.smithi187.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-17T07:11:31.343 INFO:teuthology.orchestra.run.smithi187.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-17T07:11:31.376 INFO:teuthology.orchestra.run.smithi187.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-17T07:11:31.440 INFO:teuthology.orchestra.run.smithi187.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-17T07:11:31.469 INFO:teuthology.orchestra.run.smithi187.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 172 kB/s | 21 kB 00:00 2024-04-17T07:11:31.494 INFO:teuthology.orchestra.run.smithi187.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 218 kB/s | 25 kB 00:00 2024-04-17T07:11:31.600 INFO:teuthology.orchestra.run.smithi067.stdout:(151/174): librados2-19.0.0-3133.ga8c84f59.el9. 4.6 MB/s | 3.4 MB 00:00 2024-04-17T07:11:31.653 INFO:teuthology.orchestra.run.smithi187.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.3 MB/s | 245 kB 00:00 2024-04-17T07:11:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:(152/174): NetworkManager-1.47.4-1.el9.x86_64.r 6.2 MB/s | 2.3 MB 00:00 2024-04-17T07:11:31.678 INFO:teuthology.orchestra.run.smithi187.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.3 MB/s | 248 kB 00:00 2024-04-17T07:11:31.712 INFO:teuthology.orchestra.run.smithi187.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-04-17T07:11:31.750 INFO:teuthology.orchestra.run.smithi067.stdout:(153/174): NetworkManager-libnm-1.47.4-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-04-17T07:11:31.776 INFO:teuthology.orchestra.run.smithi067.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 481 kB/s | 41 kB 00:00 2024-04-17T07:11:31.809 INFO:teuthology.orchestra.run.smithi067.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-04-17T07:11:31.843 INFO:teuthology.orchestra.run.smithi067.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.9 MB/s | 321 kB 00:00 2024-04-17T07:11:31.918 INFO:teuthology.orchestra.run.smithi067.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 488 kB/s | 36 kB 00:00 2024-04-17T07:11:31.962 INFO:teuthology.orchestra.run.smithi187.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-04-17T07:11:31.985 INFO:teuthology.orchestra.run.smithi067.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 416 kB/s | 71 kB 00:00 2024-04-17T07:11:32.019 INFO:teuthology.orchestra.run.smithi067.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-04-17T07:11:32.069 INFO:teuthology.orchestra.run.smithi067.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 470 kB/s | 23 kB 00:00 2024-04-17T07:11:32.094 INFO:teuthology.orchestra.run.smithi067.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-04-17T07:11:32.128 INFO:teuthology.orchestra.run.smithi067.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-04-17T07:11:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-04-17T07:11:32.195 INFO:teuthology.orchestra.run.smithi067.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-04-17T07:11:32.220 INFO:teuthology.orchestra.run.smithi067.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-04-17T07:11:32.262 INFO:teuthology.orchestra.run.smithi187.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-04-17T07:11:32.304 INFO:teuthology.orchestra.run.smithi067.stdout:(166/174): librbd1-19.0.0-3133.ga8c84f59.el9.x8 2.3 MB/s | 3.1 MB 00:01 2024-04-17T07:11:32.321 INFO:teuthology.orchestra.run.smithi067.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 230 kB/s | 29 kB 00:00 2024-04-17T07:11:32.354 INFO:teuthology.orchestra.run.smithi187.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-04-17T07:11:32.388 INFO:teuthology.orchestra.run.smithi067.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.3 MB/s | 1.2 MB 00:00 2024-04-17T07:11:32.421 INFO:teuthology.orchestra.run.smithi187.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 296 kB/s | 19 kB 00:00 2024-04-17T07:11:32.463 INFO:teuthology.orchestra.run.smithi067.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.1 MB/s | 465 kB 00:00 2024-04-17T07:11:32.505 INFO:teuthology.orchestra.run.smithi067.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 766 kB/s | 31 kB 00:00 2024-04-17T07:11:32.547 INFO:teuthology.orchestra.run.smithi187.stdout:(108/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.7 MB/s | 4.1 MB 00:01 2024-04-17T07:11:32.588 INFO:teuthology.orchestra.run.smithi187.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 986 kB/s | 863 kB 00:00 2024-04-17T07:11:32.589 INFO:teuthology.orchestra.run.smithi067.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.6 MB/s | 2.3 MB 00:00 2024-04-17T07:11:32.622 INFO:teuthology.orchestra.run.smithi187.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 365 kB/s | 26 kB 00:00 2024-04-17T07:11:32.672 INFO:teuthology.orchestra.run.smithi067.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.9 MB/s | 2.2 MB 00:00 2024-04-17T07:11:32.673 INFO:teuthology.orchestra.run.smithi187.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 611 kB/s | 49 kB 00:00 2024-04-17T07:11:32.714 INFO:teuthology.orchestra.run.smithi067.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.2 MB/s | 1.1 MB 00:00 2024-04-17T07:11:32.739 INFO:teuthology.orchestra.run.smithi067.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 499 kB/s | 74 kB 00:00 2024-04-17T07:11:32.748 INFO:teuthology.orchestra.run.smithi187.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 555 kB/s | 67 kB 00:00 2024-04-17T07:11:32.749 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-17T07:11:32.750 INFO:teuthology.orchestra.run.smithi067.stdout:Total 12 MB/s | 229 MB 00:19 2024-04-17T07:11:32.773 INFO:teuthology.orchestra.run.smithi187.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-17T07:11:32.990 INFO:teuthology.orchestra.run.smithi187.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-04-17T07:11:33.057 INFO:teuthology.orchestra.run.smithi187.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 671 kB/s | 43 kB 00:00 2024-04-17T07:11:33.099 INFO:teuthology.orchestra.run.smithi187.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-04-17T07:11:33.191 INFO:teuthology.orchestra.run.smithi187.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-04-17T07:11:33.216 INFO:teuthology.orchestra.run.smithi187.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 205 kB/s | 32 kB 00:00 2024-04-17T07:11:33.241 INFO:teuthology.orchestra.run.smithi187.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 108 kB/s | 15 kB 00:00 2024-04-17T07:11:33.283 INFO:teuthology.orchestra.run.smithi187.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-17T07:11:33.334 INFO:teuthology.orchestra.run.smithi187.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-17T07:11:33.359 INFO:teuthology.orchestra.run.smithi187.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-17T07:11:33.384 INFO:teuthology.orchestra.run.smithi187.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 365 kB/s | 35 kB 00:00 2024-04-17T07:11:33.426 INFO:teuthology.orchestra.run.smithi187.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-04-17T07:11:33.493 INFO:teuthology.orchestra.run.smithi187.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-17T07:11:33.614 INFO:teuthology.orchestra.run.smithi187.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-17T07:11:33.761 INFO:teuthology.orchestra.run.smithi187.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.1 MB/s | 2.0 MB 00:00 2024-04-17T07:11:33.786 INFO:teuthology.orchestra.run.smithi187.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 108 kB/s | 18 kB 00:00 2024-04-17T07:11:33.811 INFO:teuthology.orchestra.run.smithi187.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 122 kB/s | 23 kB 00:00 2024-04-17T07:11:33.836 INFO:teuthology.orchestra.run.smithi187.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 260 kB/s | 19 kB 00:00 2024-04-17T07:11:33.861 INFO:teuthology.orchestra.run.smithi187.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-04-17T07:11:33.876 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-17T07:11:33.887 INFO:teuthology.orchestra.run.smithi187.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 926 kB/s | 46 kB 00:00 2024-04-17T07:11:33.966 INFO:teuthology.orchestra.run.smithi187.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 6.6 MB/s | 1.0 MB 00:00 2024-04-17T07:11:33.996 INFO:teuthology.orchestra.run.smithi187.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 600 kB/s | 79 kB 00:00 2024-04-17T07:11:34.021 INFO:teuthology.orchestra.run.smithi187.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 441 kB/s | 58 kB 00:00 2024-04-17T07:11:34.046 INFO:teuthology.orchestra.run.smithi187.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.4 MB/s | 272 kB 00:00 2024-04-17T07:11:34.071 INFO:teuthology.orchestra.run.smithi187.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-17T07:11:34.097 INFO:teuthology.orchestra.run.smithi187.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-17T07:11:34.122 INFO:teuthology.orchestra.run.smithi187.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-04-17T07:11:34.147 INFO:teuthology.orchestra.run.smithi187.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-04-17T07:11:34.173 INFO:teuthology.orchestra.run.smithi187.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-04-17T07:11:34.198 INFO:teuthology.orchestra.run.smithi187.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-17T07:11:34.223 INFO:teuthology.orchestra.run.smithi187.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 483 kB/s | 36 kB 00:00 2024-04-17T07:11:34.240 INFO:teuthology.orchestra.run.smithi187.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 765 kB/s | 51 kB 00:00 2024-04-17T07:11:34.249 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-17T07:11:34.249 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-17T07:11:34.274 INFO:teuthology.orchestra.run.smithi187.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-17T07:11:34.307 INFO:teuthology.orchestra.run.smithi187.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-17T07:11:34.344 INFO:teuthology.orchestra.run.smithi187.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-17T07:11:34.369 INFO:teuthology.orchestra.run.smithi187.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-04-17T07:11:34.394 INFO:teuthology.orchestra.run.smithi187.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-17T07:11:34.536 INFO:teuthology.orchestra.run.smithi187.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.6 MB/s | 1.6 MB 00:00 2024-04-17T07:11:34.662 INFO:teuthology.orchestra.run.smithi187.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 18 MB/s | 2.3 MB 00:00 2024-04-17T07:11:34.754 INFO:teuthology.orchestra.run.smithi187.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-04-17T07:11:34.796 INFO:teuthology.orchestra.run.smithi187.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 993 kB/s | 41 kB 00:00 2024-04-17T07:11:34.838 INFO:teuthology.orchestra.run.smithi187.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 6.0 MB/s | 248 kB 00:00 2024-04-17T07:11:34.896 INFO:teuthology.orchestra.run.smithi187.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-04-17T07:11:34.988 INFO:teuthology.orchestra.run.smithi187.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 786 kB/s | 71 kB 00:00 2024-04-17T07:11:35.055 INFO:teuthology.orchestra.run.smithi187.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 736 kB/s | 36 kB 00:00 2024-04-17T07:11:35.122 INFO:teuthology.orchestra.run.smithi187.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-04-17T07:11:35.181 INFO:teuthology.orchestra.run.smithi187.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-04-17T07:11:35.223 INFO:teuthology.orchestra.run.smithi187.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 565 kB/s | 23 kB 00:00 2024-04-17T07:11:35.273 INFO:teuthology.orchestra.run.smithi187.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.2 MB/s | 108 kB 00:00 2024-04-17T07:11:35.332 INFO:teuthology.orchestra.run.smithi187.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-04-17T07:11:35.382 INFO:teuthology.orchestra.run.smithi187.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-04-17T07:11:35.424 INFO:teuthology.orchestra.run.smithi187.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-17T07:11:35.516 INFO:teuthology.orchestra.run.smithi187.stdout:(165/174): librados2-19.0.0-3133.ga8c84f59.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-17T07:11:35.533 INFO:teuthology.orchestra.run.smithi187.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 265 kB/s | 29 kB 00:00 2024-04-17T07:11:35.692 INFO:teuthology.orchestra.run.smithi187.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-04-17T07:11:35.859 INFO:teuthology.orchestra.run.smithi187.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-17T07:11:35.917 INFO:teuthology.orchestra.run.smithi187.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 3.0 MB/s | 1.2 MB 00:00 2024-04-17T07:11:35.993 INFO:teuthology.orchestra.run.smithi187.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-04-17T07:11:36.201 INFO:teuthology.orchestra.run.smithi187.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.2 MB/s | 1.1 MB 00:00 2024-04-17T07:11:36.218 INFO:teuthology.orchestra.run.smithi187.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 104 kB/s | 31 kB 00:00 2024-04-17T07:11:36.635 INFO:teuthology.orchestra.run.smithi187.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 172 kB/s | 74 kB 00:00 2024-04-17T07:11:36.722 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-17T07:11:36.723 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-17T07:11:36.735 INFO:teuthology.orchestra.run.smithi187.stdout:(174/174): librbd1-19.0.0-3133.ga8c84f59.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-04-17T07:11:36.746 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-17T07:11:36.746 INFO:teuthology.orchestra.run.smithi187.stdout:Total 11 MB/s | 229 MB 00:21 2024-04-17T07:11:37.871 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-17T07:11:38.252 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-17T07:11:38.252 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-17T07:11:40.561 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-17T07:11:40.561 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-17T07:11:44.251 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-17T07:11:44.364 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-17T07:11:44.438 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-17T07:11:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-17T07:11:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-17T07:11:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-17T07:11:44.734 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-17T07:11:44.796 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-17T07:11:44.834 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-17T07:11:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-17T07:11:45.346 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-17T07:11:45.415 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:11:45.583 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:11:45.608 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:11:45.732 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:11:45.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 12/198 2024-04-17T07:11:45.829 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-17T07:11:46.142 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-17T07:11:46.196 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-17T07:11:46.221 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:11:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:11:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 17/198 2024-04-17T07:11:46.480 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 17/198 2024-04-17T07:11:46.533 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-17T07:11:46.558 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-17T07:11:46.622 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-17T07:11:46.921 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-17T07:11:46.979 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-17T07:11:47.024 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-17T07:11:47.373 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-17T07:11:47.548 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-17T07:11:47.637 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-17T07:11:47.837 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:48.242 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-17T07:11:48.250 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:48.346 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-17T07:11:48.391 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-17T07:11:48.429 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-17T07:11:48.471 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-17T07:11:48.473 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-17T07:11:48.569 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-17T07:11:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-17T07:11:48.666 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-17T07:11:48.703 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-17T07:11:48.717 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-17T07:11:48.750 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-17T07:11:48.773 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-17T07:11:48.814 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-17T07:11:48.865 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-17T07:11:48.909 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-17T07:11:48.934 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-17T07:11:48.992 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-17T07:11:49.116 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-17T07:11:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-17T07:11:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-17T07:11:49.278 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-17T07:11:49.317 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-17T07:11:49.324 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-17T07:11:49.376 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:11:49.439 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-17T07:11:49.482 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9. 39/198 2024-04-17T07:11:49.534 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:11:49.540 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el 40/198 2024-04-17T07:11:49.560 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:11:49.571 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_6 41/198 2024-04-17T07:11:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-17T07:11:49.699 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:11:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 12/198 2024-04-17T07:11:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-17T07:11:49.784 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-17T07:11:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-17T07:11:50.046 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-17T07:11:50.060 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-17T07:11:50.112 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-17T07:11:50.123 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-17T07:11:50.166 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:11:50.326 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:11:50.350 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 17/198 2024-04-17T07:11:50.450 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 17/198 2024-04-17T07:11:50.493 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-17T07:11:50.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-17T07:11:50.584 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-17T07:11:50.842 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-17T07:11:50.933 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-17T07:11:50.944 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-17T07:11:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-17T07:11:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-17T07:11:51.062 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-17T07:11:51.116 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-17T07:11:51.144 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-17T07:11:51.232 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-17T07:11:51.305 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-17T07:11:51.347 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-17T07:11:51.362 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_6 53/198 2024-04-17T07:11:51.409 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-17T07:11:51.526 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-17T07:11:51.598 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-17T07:11:51.800 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:51.854 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:51.860 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-17T07:11:52.122 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-17T07:11:52.199 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-17T07:11:52.402 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-17T07:11:52.522 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-17T07:11:52.547 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-17T07:11:52.560 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-17T07:11:52.597 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-17T07:11:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-17T07:11:52.658 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-17T07:11:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-17T07:11:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-17T07:11:52.878 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-17T07:11:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-17T07:11:52.933 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-17T07:11:53.019 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-17T07:11:53.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-17T07:11:53.074 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-17T07:11:53.139 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-17T07:11:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-17T07:11:53.189 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-17T07:11:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-17T07:11:53.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-17T07:11:53.236 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-17T07:11:53.306 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-17T07:11:53.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-17T07:11:53.355 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-17T07:11:53.390 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9. 39/198 2024-04-17T07:11:53.413 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-17T07:11:53.445 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el 40/198 2024-04-17T07:11:53.480 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-17T07:11:53.490 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_6 41/198 2024-04-17T07:11:53.590 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-17T07:11:53.650 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-17T07:11:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-17T07:11:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-17T07:11:53.750 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-17T07:11:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-17T07:11:53.843 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-17T07:11:53.930 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-17T07:11:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-17T07:11:53.997 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-17T07:11:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-17T07:11:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-17T07:11:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-17T07:11:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-17T07:11:54.306 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-17T07:11:54.488 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-17T07:11:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-17T07:11:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-17T07:11:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-17T07:11:54.755 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-17T07:11:54.801 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-17T07:11:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-17T07:11:54.905 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-17T07:11:54.961 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-17T07:11:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-17T07:11:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-17T07:11:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-17T07:11:55.100 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-17T07:11:55.144 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_6 53/198 2024-04-17T07:11:55.161 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-17T07:11:55.184 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-17T07:11:55.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-17T07:11:55.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-17T07:11:55.438 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-17T07:11:55.477 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-17T07:11:55.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-17T07:11:55.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-17T07:11:55.617 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-17T07:11:55.632 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-17T07:11:55.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-17T07:11:55.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-17T07:11:55.834 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-17T07:11:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-17T07:11:55.915 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-17T07:11:55.917 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-17T07:11:55.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-17T07:11:55.979 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-17T07:11:56.024 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-17T07:11:56.085 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-17T07:11:56.180 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-17T07:11:56.241 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-17T07:11:56.254 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-17T07:11:56.330 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-17T07:11:56.368 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-17T07:11:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-17T07:11:56.427 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-17T07:11:56.463 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-17T07:11:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-17T07:11:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-17T07:11:56.522 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-17T07:11:56.586 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-17T07:11:56.626 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-17T07:11:56.677 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-17T07:11:56.688 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-17T07:11:56.745 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-17T07:11:56.754 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-17T07:11:56.957 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-17T07:11:57.016 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-17T07:11:57.053 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-17T07:11:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-17T07:11:57.240 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-17T07:11:57.293 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-17T07:11:57.347 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-17T07:11:57.392 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-17T07:11:57.416 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-17T07:11:57.523 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-17T07:11:57.694 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-17T07:11:57.761 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-17T07:11:57.819 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-17T07:11:57.826 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-17T07:11:57.852 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-17T07:11:57.877 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-17T07:11:57.911 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-17T07:11:57.938 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-17T07:11:58.012 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-17T07:11:58.027 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-17T07:11:58.069 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-17T07:11:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-17T07:11:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-17T07:11:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-17T07:11:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-17T07:11:58.449 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-17T07:11:58.470 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-17T07:11:58.527 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-17T07:11:58.568 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-17T07:11:58.615 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-17T07:11:58.656 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-17T07:11:58.702 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-17T07:11:58.762 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-17T07:11:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-17T07:11:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-17T07:11:58.997 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-17T07:11:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-17T07:11:59.074 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-17T07:11:59.127 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-17T07:11:59.169 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-17T07:11:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-17T07:11:59.312 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-17T07:11:59.433 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-17T07:11:59.434 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-17T07:11:59.482 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-17T07:11:59.518 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-17T07:11:59.573 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-17T07:11:59.712 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-17T07:11:59.817 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-17T07:12:00.789 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-17T07:12:00.842 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-17T07:12:00.875 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-17T07:12:00.948 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-17T07:12:01.048 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-17T07:12:01.054 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-17T07:12:01.083 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-17T07:12:01.144 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-17T07:12:01.182 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-17T07:12:01.211 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-17T07:12:01.239 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-17T07:12:01.296 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-17T07:12:01.322 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-17T07:12:01.420 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-17T07:12:01.623 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-17T07:12:01.651 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 124/198 2024-04-17T07:12:01.773 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 124/198 2024-04-17T07:12:01.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 125/198 2024-04-17T07:12:01.881 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-17T07:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-17T07:12:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-17T07:12:02.453 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-17T07:12:02.487 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 128/198 2024-04-17T07:12:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 128/198 2024-04-17T07:12:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 129/198 2024-04-17T07:12:03.922 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-17T07:12:04.009 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-17T07:12:04.039 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-17T07:12:04.070 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-17T07:12:04.104 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-17T07:12:04.468 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-17T07:12:04.500 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 124/198 2024-04-17T07:12:04.619 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 124/198 2024-04-17T07:12:04.647 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 125/198 2024-04-17T07:12:04.701 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-17T07:12:05.231 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-17T07:12:05.269 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 128/198 2024-04-17T07:12:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 128/198 2024-04-17T07:12:05.371 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 129/198 2024-04-17T07:12:05.421 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:06.088 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:06.132 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 131/198 2024-04-17T07:12:06.295 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 131/198 2024-04-17T07:12:06.295 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-17T07:12:06.295 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-17T07:12:06.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:06.341 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 132/198 2024-04-17T07:12:08.176 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:08.211 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:08.838 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 130/198 2024-04-17T07:12:08.881 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 131/198 2024-04-17T07:12:09.034 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 131/198 2024-04-17T07:12:09.034 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-17T07:12:09.034 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-17T07:12:09.034 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:09.082 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 132/198 2024-04-17T07:12:19.727 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 132/198 2024-04-17T07:12:19.727 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-04-17T07:12:19.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:19.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 133/198 2024-04-17T07:12:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 133/198 2024-04-17T07:12:20.013 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:20.013 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-17T07:12:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:21.680 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 134/198 2024-04-17T07:12:21.764 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 134/198 2024-04-17T07:12:21.764 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:21.765 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-17T07:12:21.765 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-17T07:12:21.765 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-17T07:12:21.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:22.030 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 135/198 2024-04-17T07:12:22.103 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 135/198 2024-04-17T07:12:22.103 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:22.103 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-17T07:12:22.103 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-17T07:12:22.104 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-17T07:12:22.104 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 132/198 2024-04-17T07:12:22.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-04-17T07:12:22.167 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-04-17T07:12:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:22.402 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 133/198 2024-04-17T07:12:22.449 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 133/198 2024-04-17T07:12:22.449 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:22.449 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-17T07:12:22.449 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:22.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 136/198 2024-04-17T07:12:22.686 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 136/198 2024-04-17T07:12:22.686 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:22.687 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-17T07:12:22.687 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-17T07:12:22.687 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-17T07:12:22.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-17T07:12:23.369 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 138/198 2024-04-17T07:12:23.409 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 138/198 2024-04-17T07:12:23.461 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.e 139/198 2024-04-17T07:12:23.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59. 140/198 2024-04-17T07:12:23.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 141/198 2024-04-17T07:12:24.096 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 134/198 2024-04-17T07:12:24.177 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 134/198 2024-04-17T07:12:24.178 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:24.178 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-17T07:12:24.178 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-17T07:12:24.178 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-17T07:12:24.178 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:24.422 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 135/198 2024-04-17T07:12:24.489 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 135/198 2024-04-17T07:12:24.489 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:24.489 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-17T07:12:24.490 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-17T07:12:24.490 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-17T07:12:24.490 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:24.926 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 141/198 2024-04-17T07:12:24.964 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 142/198 2024-04-17T07:12:24.989 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 136/198 2024-04-17T07:12:25.059 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 136/198 2024-04-17T07:12:25.059 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:25.060 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-17T07:12:25.060 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-17T07:12:25.060 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-17T07:12:25.060 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:25.174 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-17T07:12:25.813 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 138/198 2024-04-17T07:12:25.840 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 138/198 2024-04-17T07:12:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.e 139/198 2024-04-17T07:12:25.993 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59. 140/198 2024-04-17T07:12:26.029 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 141/198 2024-04-17T07:12:26.124 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 142/198 2024-04-17T07:12:26.147 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 143/198 2024-04-17T07:12:26.299 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 143/198 2024-04-17T07:12:26.486 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el 144/198 2024-04-17T07:12:26.524 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 145/198 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 145/198 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-17T07:12:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 146/198 2024-04-17T07:12:26.711 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 146/198 2024-04-17T07:12:26.821 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 147/198 2024-04-17T07:12:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 141/198 2024-04-17T07:12:27.347 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 142/198 2024-04-17T07:12:27.930 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-17T07:12:27.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 149/198 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 149/198 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-17T07:12:28.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:28.111 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 150/198 2024-04-17T07:12:28.196 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 150/198 2024-04-17T07:12:28.196 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:28.196 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-17T07:12:28.196 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:28.270 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-17T07:12:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-17T07:12:28.381 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-17T07:12:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:28.436 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 142/198 2024-04-17T07:12:28.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 143/198 2024-04-17T07:12:28.607 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 143/198 2024-04-17T07:12:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 152/198 2024-04-17T07:12:28.798 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el 144/198 2024-04-17T07:12:28.824 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 145/198 2024-04-17T07:12:28.892 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 145/198 2024-04-17T07:12:28.892 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:28.892 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-17T07:12:28.892 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-17T07:12:28.893 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-17T07:12:28.893 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 152/198 2024-04-17T07:12:28.898 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:28.898 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-17T07:12:28.898 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-17T07:12:28.899 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-17T07:12:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:28.950 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 146/198 2024-04-17T07:12:28.986 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 146/198 2024-04-17T07:12:29.095 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 147/198 2024-04-17T07:12:30.120 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-17T07:12:30.152 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 149/198 2024-04-17T07:12:30.223 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 149/198 2024-04-17T07:12:30.224 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:30.224 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-17T07:12:30.224 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-17T07:12:30.224 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-17T07:12:30.224 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:30.278 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 150/198 2024-04-17T07:12:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 150/198 2024-04-17T07:12:30.347 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:30.347 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-17T07:12:30.347 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:30.429 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-17T07:12:30.503 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:30.863 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 152/198 2024-04-17T07:12:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 152/198 2024-04-17T07:12:30.932 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-17T07:12:30.933 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-17T07:12:30.933 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-17T07:12:30.933 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-17T07:12:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 153/198 2024-04-17T07:12:34.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 154/198 2024-04-17T07:12:34.275 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 155/198 2024-04-17T07:12:34.314 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-17T07:12:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 157/198 2024-04-17T07:12:34.718 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-17T07:12:34.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 159/198 2024-04-17T07:12:34.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-17T07:12:34.893 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-17T07:12:34.938 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-17T07:12:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_ 163/198 2024-04-17T07:12:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-17T07:12:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-17T07:12:35.150 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-17T07:12:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-17T07:12:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-17T07:12:35.886 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 153/198 2024-04-17T07:12:35.940 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 154/198 2024-04-17T07:12:35.973 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 155/198 2024-04-17T07:12:36.013 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-17T07:12:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:36.298 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:36.310 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 157/198 2024-04-17T07:12:36.438 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-17T07:12:36.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 159/198 2024-04-17T07:12:36.534 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-17T07:12:36.625 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-17T07:12:36.670 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-17T07:12:36.705 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_ 163/198 2024-04-17T07:12:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-17T07:12:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-17T07:12:36.866 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-17T07:12:36.965 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-17T07:12:37.000 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-17T07:12:37.748 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 170/198 2024-04-17T07:12:37.864 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-17T07:12:37.871 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:37.938 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-17T07:12:37.938 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:37.971 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-17T07:12:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-17T07:12:38.107 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-17T07:12:38.140 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-17T07:12:38.175 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-17T07:12:38.218 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-17T07:12:38.218 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-17T07:12:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-17T07:12:38.420 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-17T07:12:38.421 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-17T07:12:38.559 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-17T07:12:38.604 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-17T07:12:38.657 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-17T07:12:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-17T07:12:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-17T07:12:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-17T07:12:38.897 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-17T07:12:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-17T07:12:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-17T07:12:39.011 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-17T07:12:39.044 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-17T07:12:39.077 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-17T07:12:39.117 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:39.117 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:39.279 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 170/198 2024-04-17T07:12:39.372 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-17T07:12:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-17T07:12:39.462 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:39.462 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-17T07:12:39.555 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-17T07:12:39.601 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-17T07:12:39.616 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-17T07:12:39.649 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-17T07:12:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-17T07:12:39.684 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-17T07:12:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-17T07:12:39.725 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-17T07:12:39.725 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-17T07:12:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-17T07:12:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-17T07:12:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-17T07:12:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-17T07:12:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-17T07:12:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-17T07:12:40.067 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-17T07:12:40.129 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-17T07:12:40.169 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-17T07:12:40.169 INFO:teuthology.orchestra.run.smithi067.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-17T07:12:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:40.174 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-17T07:12:40.207 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-17T07:12:40.239 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-17T07:12:40.309 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-17T07:12:40.378 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-17T07:12:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-17T07:12:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-17T07:12:40.493 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-17T07:12:40.520 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-17T07:12:40.553 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-17T07:12:40.590 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:40.591 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-17T07:12:40.907 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-17T07:12:41.042 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-17T07:12:41.094 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-17T07:12:41.131 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-17T07:12:41.168 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-17T07:12:41.207 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-17T07:12:41.326 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-17T07:12:41.588 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-17T07:12:41.588 INFO:teuthology.orchestra.run.smithi187.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-17T07:12:41.588 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:45.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 1/198 2024-04-17T07:12:45.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 2/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 3/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 4/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 5/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 6/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 7/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 8/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 9/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 12/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 13/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 14/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_ 15/198 2024-04-17T07:12:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 17/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_6 18/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 19/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 20/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el 21/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9. 22/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_6 23/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 24/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 25/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 26/198 2024-04-17T07:12:45.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 27/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 28/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 29/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59. 30/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 31/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 32/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 33/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el 34/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 35/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.e 36/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 37/198 2024-04-17T07:12:45.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 38/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 39/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-17T07:12:45.550 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-17T07:12:45.551 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-17T07:12:45.552 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-17T07:12:45.553 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-17T07:12:45.554 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-17T07:12:45.555 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-17T07:12:45.556 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:45.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 153/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-17T07:12:45.558 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-17T07:12:45.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-17T07:12:45.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-17T07:12:45.561 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-17T07:12:45.561 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-17T07:12:45.561 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-17T07:12:45.561 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-17T07:12:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 1/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 2/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 3/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 4/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3133.ga8c84 5/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 6/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 7/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 8/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 9/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 10/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 11/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 12/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 13/198 2024-04-17T07:12:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 14/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_ 15/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 16/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 17/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_6 18/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86 19/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 20/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el 21/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9. 22/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_6 23/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 24/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 25/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 26/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 27/198 2024-04-17T07:12:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 28/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 29/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59. 30/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noa 31/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.n 32/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c 33/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el 34/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 35/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.e 36/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 37/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 38/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 39/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-17T07:12:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-17T07:12:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-17T07:12:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-17T07:12:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-17T07:12:47.102 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-17T07:12:47.103 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-17T07:12:47.104 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-17T07:12:47.105 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 151/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 153/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-17T07:12:47.106 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-17T07:12:47.107 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-17T07:12:47.108 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-17T07:12:47.109 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-17T07:12:48.334 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: acl-2.3.1-4.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-17T07:12:48.335 INFO:teuthology.orchestra.run.smithi067.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-17T07:12:48.336 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.337 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-17T07:12:48.338 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-17T07:12:48.339 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-04-17T07:12:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-17T07:12:48.341 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-17T07:12:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-17T07:12:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-17T07:12:48.344 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-17T07:12:48.345 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-17T07:12:48.346 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-17T07:12:48.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:48.348 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-17T07:12:49.005 DEBUG:teuthology.parallel:result is None 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:49.722 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: acl-2.3.1-4.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.723 INFO:teuthology.orchestra.run.smithi187.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: automake-1.16.2-8.el9.noarch 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: bison-3.7.4-5.el9.x86_64 2024-04-17T07:12:49.724 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-shell-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top-2:19.0.0-3133.ga8c84f59.el9.noarch 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: flex-2.6.4-9.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.6-16.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-17T07:12:49.727 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-17T07:12:49.728 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-17T07:12:49.729 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-13.14-1.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-17T07:12:49.730 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-17T07:12:49.731 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-17T07:12:49.732 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-17T07:12:49.733 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-17T07:12:49.734 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-17T07:12:49.735 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.0.0-3133.ga8c84f59.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-3.el9.x86_64 2024-04-17T07:12:49.736 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:49.737 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-17T07:12:50.433 DEBUG:teuthology.parallel:result is None 2024-04-17T07:12:50.434 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a8c84f5997950fd7d008204839253291d52e6eda 2024-04-17T07:12:50.591 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-17T07:12:50.652 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-3133.ga8c84f59.el9 2024-04-17T07:12:50.652 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3133.ga8c84f59.el9 2024-04-17T07:12:50.653 INFO:teuthology.task.install:The correct ceph version 19.0.0-3133.ga8c84f59 is installed. 2024-04-17T07:12:50.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a8c84f5997950fd7d008204839253291d52e6eda 2024-04-17T07:12:50.879 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-17T07:12:50.937 INFO:teuthology.orchestra.run.smithi187.stdout:19.0.0-3133.ga8c84f59.el9 2024-04-17T07:12:50.937 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3133.ga8c84f59.el9 2024-04-17T07:12:50.937 INFO:teuthology.task.install:The correct ceph version 19.0.0-3133.ga8c84f59 is installed. 2024-04-17T07:12:50.939 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-17T07:12:50.939 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:50.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T07:12:51.030 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:51.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T07:12:51.113 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-17T07:12:51.113 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:51.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T07:12:51.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T07:12:51.315 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:51.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T07:12:51.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T07:12:51.505 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-17T07:12:51.505 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:51.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T07:12:51.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T07:12:51.701 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:51.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T07:12:51.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T07:12:51.893 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-17T07:12:51.893 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:51.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T07:12:51.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T07:12:52.106 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:52.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T07:12:52.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T07:12:52.293 INFO:teuthology.run_tasks:Running task ceph... 2024-04-17T07:12:52.379 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-17T07:12:52.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-04-17T07:12:52.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2024-04-17T07:12:52.459 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-17T07:12:52.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T07:12:52.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T07:12:52.534 INFO:tasks.ceph:Creating extra log directories... 2024-04-17T07:12:52.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T07:12:52.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T07:12:52.616 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-17T07:12:52.617 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-17T07:12:52.617 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-17T07:12:52.617 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:52.617 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T07:12:52.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T07:12:52.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-17T07:12:52.928 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:52.928 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T07:12:52.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T07:12:53.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-17T07:12:53.282 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-17T07:12:53.282 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-17T07:12:53.282 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904', 'branch': 'wip-rishabh-testing-20240416.193735', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': None, 'first_in_suite': False, 'job_id': '7659904', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240416.193735'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240416.193735', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3521, 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'sleep_before_teardown': 0, 'subset': '410/512', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240416.193735', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjfw6h6JlM/VdJSY9YLTvtZbS8r4uvCOoXXUwb++K3eVRFZ3cvDITS7Ooghp+2zjwYm8uc+/FA5Tmz2lvvAQ9A=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMER5OGO3me8ZozXOURqAX21fPe/kP8GyM6fXYA7gpJD1upTcNvwuBB2LbxwQ9RM10QyUxYXSASm1L2lMQnlalE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a8c84f5997950fd7d008204839253291d52e6eda', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-17_04:58:44', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-17T07:12:53.283 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T07:12:53.290 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T07:12:53.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T07:12:53.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T07:12:53.441 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:53.441 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T07:12:53.494 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T07:12:53.495 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-17T07:12:53.544 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T07:12:53.544 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:53.544 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-17T07:12:53.544 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:53.544 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:53.545 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 07:12:40.651542736 +0000 2024-04-17T07:12:53.545 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 07:10:00.231730335 +0000 2024-04-17T07:12:53.545 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 07:10:00.231730335 +0000 2024-04-17T07:12:53.545 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 07:10:00.231730335 +0000 2024-04-17T07:12:53.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T07:12:53.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T07:12:53.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T07:12:53.629 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000475767 s, 1.1 MB/s 2024-04-17T07:12:53.632 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T07:12:53.691 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-17T07:12:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T07:12:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 07:12:40.653542692 +0000 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 07:10:00.858716399 +0000 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 07:10:00.858716399 +0000 2024-04-17T07:12:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 07:10:00.858716399 +0000 2024-04-17T07:12:53.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T07:12:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T07:12:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T07:12:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000370578 s, 1.4 MB/s 2024-04-17T07:12:53.837 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T07:12:53.897 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-17T07:12:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T07:12:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:53.950 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-17T07:12:53.950 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:53.950 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:53.951 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 07:12:40.653542692 +0000 2024-04-17T07:12:53.951 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 07:10:01.475702686 +0000 2024-04-17T07:12:53.951 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 07:10:01.475702686 +0000 2024-04-17T07:12:53.951 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 07:10:01.475702686 +0000 2024-04-17T07:12:53.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T07:12:54.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T07:12:54.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T07:12:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000337384 s, 1.5 MB/s 2024-04-17T07:12:54.049 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T07:12:54.106 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-17T07:12:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T07:12:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:54.158 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-17T07:12:54.158 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:54.159 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:54.159 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 07:12:40.654542670 +0000 2024-04-17T07:12:54.159 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 07:10:02.102688751 +0000 2024-04-17T07:12:54.159 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 07:10:02.102688751 +0000 2024-04-17T07:12:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 07:10:02.102688751 +0000 2024-04-17T07:12:54.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T07:12:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T07:12:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T07:12:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000309424 s, 1.7 MB/s 2024-04-17T07:12:54.253 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T07:12:54.311 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T07:12:54.312 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:54.312 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T07:12:54.369 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T07:12:54.369 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-04-17T07:12:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-17 07:12:42.017812351 +0000 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-17 07:09:58.607890062 +0000 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-17 07:09:58.607890062 +0000 2024-04-17T07:12:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-17 07:09:58.607890062 +0000 2024-04-17T07:12:54.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T07:12:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-17T07:12:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-17T07:12:54.501 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000516098 s, 992 kB/s 2024-04-17T07:12:54.504 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T07:12:54.556 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-17 07:12:42.018812335 +0000 2024-04-17T07:12:54.606 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-17 07:09:59.204880377 +0000 2024-04-17T07:12:54.607 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-17 07:09:59.204880377 +0000 2024-04-17T07:12:54.607 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-17 07:09:59.204880377 +0000 2024-04-17T07:12:54.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T07:12:54.688 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-17T07:12:54.688 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-17T07:12:54.689 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.00033415 s, 1.5 MB/s 2024-04-17T07:12:54.692 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T07:12:54.737 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-17 07:12:42.019812319 +0000 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-17 07:09:59.819870399 +0000 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-17 07:09:59.819870399 +0000 2024-04-17T07:12:54.825 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-17 07:09:59.819870399 +0000 2024-04-17T07:12:54.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T07:12:54.896 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-17T07:12:54.896 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-17T07:12:54.896 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000340536 s, 1.5 MB/s 2024-04-17T07:12:54.899 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T07:12:54.951 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-17 07:12:42.019812319 +0000 2024-04-17T07:12:54.997 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-17 07:10:00.419860665 +0000 2024-04-17T07:12:54.998 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-17 07:10:00.419860665 +0000 2024-04-17T07:12:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-17 07:10:00.419860665 +0000 2024-04-17T07:12:54.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T07:12:55.130 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-17T07:12:55.130 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-17T07:12:55.131 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.00031153 s, 1.6 MB/s 2024-04-17T07:12:55.135 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T07:12:55.192 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T07:12:55.192 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T07:12:55.192 INFO:tasks.ceph:Generating config... 2024-04-17T07:12:55.193 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-17T07:12:55.193 INFO:tasks.ceph:[client] debug client = 20 2024-04-17T07:12:55.194 INFO:tasks.ceph:[client] debug ms = 1 2024-04-17T07:12:55.194 INFO:tasks.ceph:[client] pid file = 2024-04-17T07:12:55.194 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-17T07:12:55.194 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-17T07:12:55.194 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-17T07:12:55.194 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-17T07:12:55.195 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-17T07:12:55.195 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-17T07:12:55.195 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-17T07:12:55.195 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-17T07:12:55.195 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-17T07:12:55.196 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-17T07:12:55.196 INFO:tasks.ceph:Setting up mon.a... 2024-04-17T07:12:55.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-17T07:12:55.358 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-04-17T07:12:55.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-17T07:12:55.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T07:12:55.610 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.187'), ('mon.c', '[v2:172.21.15.187:3301,v1:172.21.15.187:6790]')] 2024-04-17T07:12:55.611 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-17T07:12:55.612 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:55.612 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T07:12:55.671 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.187 --addv c '[v2:172.21.15.187:3301,v1:172.21.15.187:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid c9dd6e72-8918-4f7c-8224-a4ea40cec0ec 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:fsid c9dd6e72-8918-4f7c-8224-a4ea40cec0ec 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-04-17T07:12:55.833428+0000 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-17T07:12:55.833428+0000 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-04-17T07:12:55.832 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-04-17T07:12:55.833 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] mon.c 2024-04-17T07:12:55.833 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-17T07:12:55.835 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T07:12:55.887 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c9dd6e72-8918-4f7c-8224-a4ea40cec0ec... 2024-04-17T07:12:55.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T07:12:55.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-17T07:12:56.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.062 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-17T07:12:56.063 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T07:12:56.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-04-17T07:12:56.065 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = c9dd6e72-8918-4f7c-8224-a4ea40cec0ec 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-17T07:12:56.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-17T07:12:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-17T07:12:56.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-04-17T07:12:56.069 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.070 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: debug finisher = 20 2024-04-17T07:12:56.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-17T07:12:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T07:12:56.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon down mkfs grace = 300 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-17T07:12:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-04-17T07:12:56.075 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-04-17T07:12:56.076 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-04-17T07:12:56.077 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-04-17T07:12:56.077 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-04-17T07:12:56.077 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-17T07:12:56.077 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-17T07:12:56.077 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-17T07:12:56.078 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-17T07:12:56.079 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.080 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.67,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = c9dd6e72-8918-4f7c-8224-a4ea40cec0ec 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-17T07:12:56.081 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.082 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bdev async discard = True 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bdev enable discard = True 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore allocator = bitmap 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 1/20 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 1/20 2024-04-17T07:12:56.083 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 4/10 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-17T07:12:56.084 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug finisher = 20 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-17T07:12:56.085 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-17T07:12:56.086 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-17T07:12:56.087 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-17T07:12:56.088 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-17T07:12:56.089 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-17T07:12:56.089 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-17T07:12:56.089 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-17T07:12:56.089 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-17T07:12:56.109 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-17T07:12:56.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-17T07:12:56.261 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-17T07:12:56.261 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:56.261 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-17T07:12:56.318 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:56.318 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-17T07:12:56.375 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:12:56.375 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:56.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T07:12:56.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T07:12:56.531 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:12:56.532 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:12:56.631 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:12:56.631 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:56.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T07:12:56.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T07:12:56.784 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:12:56.785 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:12:56.881 INFO:tasks.ceph:Setting up mon nodes... 2024-04-17T07:12:56.881 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-17T07:12:56.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T07:12:57.062 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T07:12:57.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T07:12:57.269 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T07:12:57.281 INFO:tasks.ceph:Setting up mds nodes... 2024-04-17T07:12:57.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T07:12:57.465 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T07:12:57.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-17T07:12:57.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T07:12:57.763 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T07:12:57.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-17T07:12:57.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T07:12:58.041 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T07:12:58.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-17T07:12:58.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T07:12:58.327 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T07:12:58.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-17T07:12:58.432 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-17T07:12:58.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-17T07:12:58.572 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-17T07:12:58.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-17T07:12:58.757 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-17T07:12:58.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-17T07:12:58.955 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-17T07:12:59.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-17T07:12:59.119 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-17T07:12:59.181 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-17T07:12:59.181 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T07:12:59.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-17T07:12:59.286 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T07:12:59.287 INFO:tasks.ceph:role: osd.0 2024-04-17T07:12:59.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:12:59.287 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T07:12:59.389 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:12:59.390 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:12:59.800 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T07:12:59.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T07:12:59.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-17T07:12:59.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-17T07:13:00.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-17T07:13:00.176 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:00.177 INFO:tasks.ceph:role: osd.1 2024-04-17T07:13:00.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:13:00.177 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T07:13:00.266 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:00.266 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:00.267 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:00.626 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T07:13:00.635 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T07:13:00.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-17T07:13:00.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-17T07:13:00.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-17T07:13:00.935 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:00.935 INFO:tasks.ceph:role: osd.2 2024-04-17T07:13:00.935 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:13:00.935 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T07:13:01.027 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:01.028 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:01.422 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T07:13:01.429 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T07:13:01.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-17T07:13:01.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-17T07:13:01.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-17T07:13:01.749 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:01.749 INFO:tasks.ceph:role: osd.3 2024-04-17T07:13:01.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:13:01.750 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T07:13:01.849 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:01.850 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:01.851 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:01.851 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:02.258 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T07:13:02.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T07:13:02.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-17T07:13:02.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-17T07:13:02.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.617+0000 7fe499611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-17T07:13:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.618+0000 7fe499611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-17T07:13:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.618+0000 7fe499611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T07:13:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.618+0000 7fe499611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T07:13:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.618+0000 7fe499611640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T07:13:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:02.622+0000 7fe499611640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-17T07:13:05.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-17T07:13:05.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.831+0000 7f1340011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-17T07:13:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.831+0000 7f1340011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-17T07:13:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.831+0000 7f1340011640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T07:13:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.832+0000 7f1340011640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T07:13:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.832+0000 7f1340011640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T07:13:05.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:05.834+0000 7f1340011640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-17T07:13:09.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-17T07:13:09.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.316+0000 7f1441c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-17T07:13:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.317+0000 7f1441c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-17T07:13:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.317+0000 7f1441c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T07:13:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.317+0000 7f1441c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T07:13:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.317+0000 7f1441c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T07:13:09.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:09.319+0000 7f1441c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-17T07:13:12.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-17T07:13:12.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:12.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.782+0000 7f3abbf38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-17T07:13:12.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.783+0000 7f3abbf38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-17T07:13:12.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.783+0000 7f3abbf38640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T07:13:12.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.783+0000 7f3abbf38640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T07:13:12.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.783+0000 7f3abbf38640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T07:13:12.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:12.786+0000 7f3abbf38640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-17T07:13:15.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-17T07:13:15.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-17T07:13:15.980 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:15.980 INFO:tasks.ceph:role: osd.4 2024-04-17T07:13:15.980 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:13:15.980 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:16.080 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:16.081 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:16.081 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:16.081 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:16.486 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-17T07:13:16.493 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-17T07:13:16.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-17T07:13:16.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-17T07:13:16.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-17T07:13:16.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:16.807 INFO:tasks.ceph:role: osd.5 2024-04-17T07:13:16.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:13:16.807 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:16.898 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:17.252 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-17T07:13:17.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-17T07:13:17.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-17T07:13:17.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-17T07:13:17.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-17T07:13:17.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:17.513 INFO:tasks.ceph:role: osd.6 2024-04-17T07:13:17.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:13:17.513 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:17.592 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:17.593 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:17.593 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:17.593 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:17.593 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:17.973 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-17T07:13:17.980 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-17T07:13:17.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-17T07:13:18.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-17T07:13:18.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-17T07:13:18.269 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T07:13:18.269 INFO:tasks.ceph:role: osd.7 2024-04-17T07:13:18.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:13:18.270 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-17T07:13:18.359 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T07:13:18.360 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T07:13:18.764 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-17T07:13:18.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-17T07:13:18.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-17T07:13:18.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-17T07:13:19.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:19.192 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.192+0000 7fac912b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-17T07:13:19.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.193+0000 7fac912b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-17T07:13:19.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.193+0000 7fac912b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T07:13:19.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.193+0000 7fac912b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T07:13:19.193 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.193+0000 7fac912b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T07:13:19.196 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:19.196+0000 7fac912b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-17T07:13:22.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-17T07:13:22.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:22.387 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.388+0000 7f646ac0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-17T07:13:22.388 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.388+0000 7f646ac0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-17T07:13:22.388 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.388+0000 7f646ac0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T07:13:22.388 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.389+0000 7f646ac0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T07:13:22.388 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.389+0000 7f646ac0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T07:13:22.390 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:22.391+0000 7f646ac0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-17T07:13:23.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:13:23.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:13:25.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-17T07:13:25.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:25.571 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.573+0000 7f12e1421640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-17T07:13:25.572 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.573+0000 7f12e1421640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-17T07:13:25.572 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.573+0000 7f12e1421640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T07:13:25.572 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.573+0000 7f12e1421640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T07:13:25.572 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.573+0000 7f12e1421640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T07:13:25.574 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:25.576+0000 7f12e1421640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-17T07:13:29.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-17T07:13:29.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:29.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.265+0000 7f2cae91d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-17T07:13:29.265 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.266+0000 7f2cae91d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-17T07:13:29.265 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.266+0000 7f2cae91d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T07:13:29.265 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.266+0000 7f2cae91d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T07:13:29.265 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.266+0000 7f2cae91d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T07:13:29.267 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:29.268+0000 7f2cae91d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-17T07:13:32.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-17T07:13:32.303 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-17T07:13:32.304 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-17T07:13:32.409 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-17T07:13:32.510 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-17T07:13:32.605 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-17T07:13:32.705 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-17T07:13:32.810 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-17T07:13:32.913 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:32.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-17T07:13:33.009 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-17T07:13:33.110 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-17T07:13:33.205 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-17T07:13:33.298 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-17T07:13:33.388 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-17T07:13:33.474 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-17T07:13:33.614 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-17T07:13:33.706 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:33.706 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-17T07:13:33.766 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:33.766 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-17T07:13:33.820 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.820 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-17T07:13:33.876 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:33.876 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-17T07:13:33.925 INFO:tasks.ceph:Adding keys to all mons... 2024-04-17T07:13:33.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T07:13:33.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w== 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ== 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig== 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA== 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ== 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-04-17T07:13:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow== 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-04-17T07:13:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w== 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ== 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg== 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g== 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2024-04-17T07:13:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.y] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ== 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-04-17T07:13:34.057 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.4] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.5] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow== 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.6] 2024-04-17T07:13:34.058 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w== 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.7] 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ== 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout:[client.2] 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg== 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g== 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout:[client.3] 2024-04-17T07:13:34.059 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA== 2024-04-17T07:13:34.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T07:13:34.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T07:13:34.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:34.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:34.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:34.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:34.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:34.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T07:13:35.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T07:13:35.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:35.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:35.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:35.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T07:13:35.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:35.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:36.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:36.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T07:13:36.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T07:13:36.783 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-17T07:13:36.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-17T07:13:36.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T07:13:37.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-17T07:13:37.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-17T07:13:37.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T07:13:37.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-17T07:13:37.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-17T07:13:37.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T07:13:37.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-17T07:13:37.826 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:37.833 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T07:13:37.887 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-17T07:13:37.887 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-17T07:13:37.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-17T07:13:37.895 INFO:tasks.ceph.mon.a:Started 2024-04-17T07:13:37.896 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-17T07:13:37.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-17T07:13:37.901 INFO:tasks.ceph.mon.b:Started 2024-04-17T07:13:37.901 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-17T07:13:37.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-17T07:13:37.905 INFO:tasks.ceph.mon.c:Started 2024-04-17T07:13:37.905 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-17T07:13:37.905 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-17T07:13:37.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-17T07:13:37.912 INFO:tasks.ceph.mgr.y:Started 2024-04-17T07:13:37.913 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-17T07:13:37.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-17T07:13:37.917 INFO:tasks.ceph.mgr.x:Started 2024-04-17T07:13:37.917 INFO:tasks.ceph:Setting crush tunables to default 2024-04-17T07:13:37.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-04-17T07:13:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.109+0000 7fa783600640 1 Processor -- start 2024-04-17T07:13:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.109+0000 7fa783600640 1 -- start start 2024-04-17T07:13:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.110+0000 7fa783600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.110+0000 7fa783600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa77c163860 con 0x7fa77c0d17a0 2024-04-17T07:13:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.110+0000 7fa783600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa77c0d1d40 con 0x7fa77c0cfa10 2024-04-17T07:13:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.110+0000 7fa783600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa77c0d2c30 con 0x7fa77c15eca0 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:32946/0 (socket says 172.21.15.67:32946) 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 learned_addr learned my addr 172.21.15.67:0/3538279268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa781600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa77c0d17a0 0x7fa77c0b7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43220/0 (socket says 172.21.15.67:43220) 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 msgr2=0x7fa77c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 msgr2=0x7fa77c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T07:13:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.111+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.114+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865132723 0 0) 0x7fa77c163860 con 0x7fa77c0d17a0 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa764003520 con 0x7fa77c0d17a0 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 990332465 0 0) 0x7fa764003520 con 0x7fa77c0d17a0 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 msgr2=0x7fa77c0d39f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa77c0d39f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 >> v1:172.21.15.187:6789/0 conn(0x7fa77c0cfa10 legacy=0x7fa77c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c0d2f10 con 0x7fa77c0d17a0 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa7740019f0 con 0x7fa77c0d17a0 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 >> v1:172.21.15.67:6789/0 conn(0x7fa77c0d17a0 legacy=0x7fa77c0b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa7640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa764004060 0x7fa764004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa764004d40 0x7fa7640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c0d2c30 con 0x7fa764004060 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa76400bc00 con 0x7fa764004d40 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa76400be00 con 0x7fa77c15eca0 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa7640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.115+0000 7fa781600640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa764004d40 0x7fa7640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa77be00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa764004060 0x7fa764004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa764004d40 msgr2=0x7fa7640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa764004d40 0x7fa7640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa764004060 msgr2=0x7fa764004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa764004060 0x7fa764004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.116+0000 7fa780c00640 1 -- 172.21.15.67:0/3538279268 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa764003520 con 0x7fa77c15eca0 2024-04-17T07:13:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.117+0000 7fa780c00640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa7640039e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa770009890 tx=0x7fa764005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbf9478be5ae2079 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.117+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa770034070 con 0x7fa77c15eca0 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.117+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa770030680 con 0x7fa77c15eca0 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.117+0000 7fa77b400640 1 -- 172.21.15.67:0/3538279268 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77000f040 con 0x7fa77c15eca0 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 -- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 msgr2=0x7fa7640039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 --2- 172.21.15.67:0/3538279268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c15eca0 0x7fa7640039e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa770009890 tx=0x7fa764005830 comp rx=0 tx=0).stop 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa781600640 1 -- 172.21.15.67:0/3538279268 reap_dead start 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 -- 172.21.15.67:0/3538279268 shutdown_connections 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 -- 172.21.15.67:0/3538279268 >> 172.21.15.67:0/3538279268 conn(0x7fa77c067600 msgr2=0x7fa77c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 -- 172.21.15.67:0/3538279268 shutdown_connections 2024-04-17T07:13:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.118+0000 7fa783600640 1 -- 172.21.15.67:0/3538279268 wait complete. 2024-04-17T07:13:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.120+0000 7fa783600640 1 Processor -- start 2024-04-17T07:13:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.120+0000 7fa783600640 1 -- start start 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa77c15eca0 0x7fa77c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c0b6630 0x7fa77c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c15f080 con 0x7fa77c0ba510 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c0d34d0 con 0x7fa77c15eca0 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa783600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa77c15f200 con 0x7fa77c0b6630 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40786/0 (socket says 172.21.15.67:40786) 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa77be00640 1 -- 172.21.15.67:0/99493691 learned_addr learned my addr 172.21.15.67:0/99493691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa780c00640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa77c15eca0 0x7fa77c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.121+0000 7fa781600640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c0b6630 0x7fa77c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 -- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c0b6630 msgr2=0x7fa77c0b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c0b6630 0x7fa77c0b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 -- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa77c15eca0 msgr2=0x7fa77c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa77c15eca0 0x7fa77c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 -- 172.21.15.67:0/99493691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c0b6ff0 con 0x7fa77c0ba510 2024-04-17T07:13:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa77be00640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa76800f620 tx=0x7fa76800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7680052e0 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa768018070 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768003810 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.122+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77c14a0f0 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.123+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77c14cdc0 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.123+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa7680039b0 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.123+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa76801dcc0 con 0x7fa77c0ba510 2024-04-17T07:13:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.124+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa77c068ca0 con 0x7fa77c0ba510 2024-04-17T07:13:38.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.128+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7fa76801d460 con 0x7fa77c0ba510 2024-04-17T07:13:38.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.200+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa77c0b9cf0 con 0x7fa77c0ba510 2024-04-17T07:13:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.203+0000 7fa778c00640 1 -- 172.21.15.67:0/99493691 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa76801d640 con 0x7fa77c0ba510 2024-04-17T07:13:38.203 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 msgr2=0x7fa77c0c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa76800f620 tx=0x7fa76800fae0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 shutdown_connections 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa77c0b6630 0x7fa77c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77c0ba510 0x7fa77c0c9810 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 --2- 172.21.15.67:0/99493691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa77c15eca0 0x7fa77c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:38.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 >> 172.21.15.67:0/99493691 conn(0x7fa77c067600 msgr2=0x7fa77c0ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:38.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 shutdown_connections 2024-04-17T07:13:38.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:38.205+0000 7fa783600640 1 -- 172.21.15.67:0/99493691 wait complete. 2024-04-17T07:13:38.224 INFO:tasks.ceph:check_enable_crimson: False 2024-04-17T07:13:38.224 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-17T07:13:38.224 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:38.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-17T07:13:38.312 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:38.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-17T07:13:38.441 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:38.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-17T07:13:38.573 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:13:38.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-17T07:13:38.657 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:38.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-17T07:13:38.744 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:38.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-17T07:13:38.821 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:38.821+0000 7fb0b8638040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:13:38.827 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:38.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-17T07:13:38.879 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:38.880+0000 7f6ef53bc040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:13:38.885 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:38.885+0000 7fb0b8638040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:13:38.949 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:38.950+0000 7f6ef53bc040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:13:38.954 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:13:38.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-17T07:13:39.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 03fdbf1a-d02e-4746-9057-f28a2d7c5535 0 2024-04-17T07:13:39.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.274+0000 7f20b1a00640 1 Processor -- start 2024-04-17T07:13:39.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.274+0000 7f20b1a00640 1 -- start start 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20ac0d33e0 0x7f20ac0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ac0d00b0 con 0x7f20ac15ecb0 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f20ac0d0c30 con 0x7f20ac0d33e0 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20ac0d0390 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:37298/0 (socket says 172.21.15.187:37298) 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.275+0000 7f20b1000640 1 -- 172.21.15.187:0/167330659 learned_addr learned my addr 172.21.15.187:0/167330659 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.276+0000 7f20a3e00640 1 --2- 172.21.15.187:0/167330659 >> v2:172.21.15.67:3300/0 conn(0x7f20ac0d33e0 0x7f20ac0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.276 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.276+0000 7f20b1000640 1 -- 172.21.15.187:0/167330659 >> v2:172.21.15.67:3300/0 conn(0x7f20ac0d33e0 msgr2=0x7f20ac0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:39.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.276+0000 7f20b1000640 1 --2- 172.21.15.187:0/167330659 >> v2:172.21.15.67:3300/0 conn(0x7f20ac0d33e0 0x7f20ac0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.276+0000 7f20b1000640 1 -- 172.21.15.187:0/167330659 >> v1:172.21.15.67:6789/0 conn(0x7f20ac15ecb0 legacy=0x7f20ac0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:39.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.276+0000 7f20b1000640 1 -- 172.21.15.187:0/167330659 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ac0d0670 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.277+0000 7f20b1000640 1 --2- 172.21.15.187:0/167330659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0d2a50 tx=0x7f20a402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec2c375ee5546af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.277+0000 7f20ab400640 1 -- 172.21.15.187:0/167330659 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a402c9f0 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.277+0000 7f20ab400640 1 -- 172.21.15.187:0/167330659 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a402cb90 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.277+0000 7f20ab400640 1 -- 172.21.15.187:0/167330659 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a40343f0 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 -- 172.21.15.187:0/167330659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac0cf8e0 msgr2=0x7f20ac0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 --2- 172.21.15.187:0/167330659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0d2a50 tx=0x7f20a402be50 comp rx=0 tx=0).stop 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 -- 172.21.15.187:0/167330659 shutdown_connections 2024-04-17T07:13:39.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 --2- 172.21.15.187:0/167330659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac0cf8e0 0x7f20ac0b3fc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 --2- 172.21.15.187:0/167330659 >> v2:172.21.15.67:3300/0 conn(0x7f20ac0d33e0 0x7f20ac0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 -- 172.21.15.187:0/167330659 >> 172.21.15.187:0/167330659 conn(0x7f20ac08bff0 msgr2=0x7f20ac0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 -- 172.21.15.187:0/167330659 shutdown_connections 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.278+0000 7f20b1a00640 1 -- 172.21.15.187:0/167330659 wait complete. 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.279+0000 7f20b1a00640 1 Processor -- start 2024-04-17T07:13:39.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.280+0000 7f20b1a00640 1 -- start start 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.280+0000 7f20b1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 0x7f20ac0b9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20ac0d33e0 0x7f20ac150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 0x7f20ac151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20ac15f090 con 0x7f20ac0d33e0 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20ac0cfd30 con 0x7f20ac15ecb0 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20ac15f210 con 0x7f20ac0cf8e0 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20abe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 0x7f20ac0b9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 0x7f20ac151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.282 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20abe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 0x7f20ac0b9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:46166/0 (socket says 172.21.15.187:46166) 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20abe00640 1 -- 172.21.15.187:0/3000078674 learned_addr learned my addr 172.21.15.187:0/3000078674 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20a3e00640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20ac0d33e0 0x7f20ac150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 -- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 msgr2=0x7f20ac0b9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 0x7f20ac0b9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 -- 172.21.15.187:0/3000078674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20ac0d33e0 msgr2=0x7f20ac150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20ac0d33e0 0x7f20ac150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.281+0000 7f20b1000640 1 -- 172.21.15.187:0/3000078674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ac0c90d0 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20b1000640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 0x7f20ac151020 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0d0db0 tx=0x7f20a4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4043020 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a403e450 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4009510 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20ac0c93b0 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20ac0c9970 con 0x7f20ac15ecb0 2024-04-17T07:13:39.283 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f20a4046020 con 0x7f20ac15ecb0 2024-04-17T07:13:39.284 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.282+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f20a4048400 con 0x7f20ac15ecb0 2024-04-17T07:13:39.284 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.283+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20ac068a10 con 0x7f20ac15ecb0 2024-04-17T07:13:39.285 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.286+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f20a404d030 con 0x7f20ac15ecb0 2024-04-17T07:13:39.364 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.364+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "03fdbf1a-d02e-4746-9057-f28a2d7c5535", "id": 0} v 0) -- 0x7f20ac0d2780 con 0x7f20ac15ecb0 2024-04-17T07:13:39.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.372+0000 7f20a8c00640 1 -- 172.21.15.187:0/3000078674 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03fdbf1a-d02e-4746-9057-f28a2d7c5535", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f20a402ca00 con 0x7f20ac15ecb0 2024-04-17T07:13:39.463 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 msgr2=0x7f20ac151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 0x7f20ac151020 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0d0db0 tx=0x7f20a4004a40 comp rx=0 tx=0).stop 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 shutdown_connections 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20ac15ecb0 0x7f20ac151020 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20ac0d33e0 0x7f20ac150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 --2- 172.21.15.187:0/3000078674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20ac0cf8e0 0x7f20ac0b9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 >> 172.21.15.187:0/3000078674 conn(0x7f20ac08bff0 msgr2=0x7f20ac0d4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 shutdown_connections 2024-04-17T07:13:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.374+0000 7f20b1a00640 1 -- 172.21.15.187:0/3000078674 wait complete. 2024-04-17T07:13:39.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new a364ec30-0f7e-4847-99b5-945415b1a816 1 2024-04-17T07:13:39.537 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.537+0000 7fb0b8638040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:13:39.597 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.598+0000 7f6ef53bc040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:13:39.609 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.609+0000 7fb0b8638040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:13:39.664 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.664+0000 7fd906400640 1 Processor -- start 2024-04-17T07:13:39.664 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.664+0000 7fd906400640 1 -- start start 2024-04-17T07:13:39.664 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.665+0000 7f6ef53bc040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.665+0000 7fd906400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9000cfa10 0x7fd9000cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.665+0000 7fd906400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 0x7fd9000d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd906400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9000d4100 con 0x7fd90015eca0 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd906400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd9000d03b0 con 0x7fd9000d1090 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd906400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd900163860 con 0x7fd9000cfa10 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9000cfa10 0x7fd9000cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9000cfa10 0x7fd9000cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:37324/0 (socket says 172.21.15.187:37324) 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 -- 172.21.15.187:0/319827142 learned_addr learned my addr 172.21.15.187:0/319827142 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:39.666 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8ffe00640 1 --2- 172.21.15.187:0/319827142 >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 0x7fd9000d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 -- 172.21.15.187:0/319827142 >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 msgr2=0x7fd9000d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:39.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 --2- 172.21.15.187:0/319827142 >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 0x7fd9000d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.666+0000 7fd8fea00640 1 -- 172.21.15.187:0/319827142 >> v1:172.21.15.67:6789/0 conn(0x7fd90015eca0 legacy=0x7fd9000d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.667+0000 7fd8fea00640 1 -- 172.21.15.187:0/319827142 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9000d1d40 con 0x7fd9000cfa10 2024-04-17T07:13:39.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.667+0000 7fd8ffe00640 1 --2- 172.21.15.187:0/319827142 >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 0x7fd9000d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:39.668 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.667+0000 7fb0b8638040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:13:39.668 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.668+0000 7fd8fea00640 1 --2- 172.21.15.187:0/319827142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9000cfa10 0x7fd9000cfe70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd9000b7500 tx=0x7fd8ec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f79ebce94e91e821 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.668 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd8fe000640 1 -- 172.21.15.187:0/319827142 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec034070 con 0x7fd9000cfa10 2024-04-17T07:13:39.668 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd8fe000640 1 -- 172.21.15.187:0/319827142 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ec030860 con 0x7fd9000cfa10 2024-04-17T07:13:39.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd8fe000640 1 -- 172.21.15.187:0/319827142 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec030c30 con 0x7fd9000cfa10 2024-04-17T07:13:39.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 -- 172.21.15.187:0/319827142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9000cfa10 msgr2=0x7fd9000cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 --2- 172.21.15.187:0/319827142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9000cfa10 0x7fd9000cfe70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd9000b7500 tx=0x7fd8ec02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 -- 172.21.15.187:0/319827142 shutdown_connections 2024-04-17T07:13:39.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 --2- 172.21.15.187:0/319827142 >> v2:172.21.15.67:3300/0 conn(0x7fd9000d1090 0x7fd9000d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 --2- 172.21.15.187:0/319827142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9000cfa10 0x7fd9000cfe70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 -- 172.21.15.187:0/319827142 >> 172.21.15.187:0/319827142 conn(0x7fd900067600 msgr2=0x7fd9000679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:39.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 -- 172.21.15.187:0/319827142 shutdown_connections 2024-04-17T07:13:39.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.669+0000 7fd906400640 1 -- 172.21.15.187:0/319827142 wait complete. 2024-04-17T07:13:39.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.671+0000 7fd906400640 1 Processor -- start 2024-04-17T07:13:39.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.671+0000 7fd906400640 1 -- start start 2024-04-17T07:13:39.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 0x7fd9000caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9000d1090 0x7fd9001590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9000d34d0 con 0x7fd9000cfa10 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9000d31f0 con 0x7fd90015eca0 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.672+0000 7fd906400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9000d19f0 con 0x7fd9000d1090 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8fea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9000d1090 0x7fd9001590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:37334/0 (socket says 172.21.15.187:37334) 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 -- 172.21.15.187:0/932068537 learned_addr learned my addr 172.21.15.187:0/932068537 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:39.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ff400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 0x7fd9000caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 -- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9000d1090 msgr2=0x7fd9001590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9000d1090 0x7fd9001590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 -- 172.21.15.187:0/932068537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 msgr2=0x7fd9000caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 0x7fd9000caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ffe00640 1 -- 172.21.15.187:0/932068537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9000b9770 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.673+0000 7fd8ff400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 0x7fd9000caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8ffe00640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009fa0 tx=0x7fd8f800d560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f801a070 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f8003bb0 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8004030 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9001576a0 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd900157c30 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd8f801d070 con 0x7fd90015eca0 2024-04-17T07:13:39.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.674+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fd8f8015d20 con 0x7fd90015eca0 2024-04-17T07:13:39.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.675+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd90015e500 con 0x7fd90015eca0 2024-04-17T07:13:39.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.678+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7fd90015e500 con 0x7fd90015eca0 2024-04-17T07:13:39.718 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.719+0000 7f6ef53bc040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:13:39.738 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.738+0000 7fb0b8638040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:13:39.757 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.757+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a364ec30-0f7e-4847-99b5-945415b1a816", "id": 1} v 0) -- 0x7fd9000c8f60 con 0x7fd90015eca0 2024-04-17T07:13:39.765 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.765+0000 7fd8f7400640 1 -- 172.21.15.187:0/932068537 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a364ec30-0f7e-4847-99b5-945415b1a816", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fd8f8010070 con 0x7fd90015eca0 2024-04-17T07:13:39.765 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-17T07:13:39.766 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 msgr2=0x7fd900159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.766 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009fa0 tx=0x7fd8f800d560 comp rx=0 tx=0).stop 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 shutdown_connections 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd90015eca0 0x7fd900159630 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9000d1090 0x7fd9001590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 --2- 172.21.15.187:0/932068537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9000cfa10 0x7fd9000caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.766+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 >> 172.21.15.187:0/932068537 conn(0x7fd900067600 msgr2=0x7fd900154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.767+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 shutdown_connections 2024-04-17T07:13:39.767 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.767+0000 7fd906400640 1 -- 172.21.15.187:0/932068537 wait complete. 2024-04-17T07:13:39.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 18b200d0-3b8a-4c95-9ead-ff065c9bb590 2 2024-04-17T07:13:39.785 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.785+0000 7f6ef53bc040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:13:39.794 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.794+0000 7fb0b8638040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:13:39.837 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.838+0000 7f6ef53bc040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:13:39.932 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:39.932+0000 7fb0b8638040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:13:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.966+0000 7f3e5b200640 1 Processor -- start 2024-04-17T07:13:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.966+0000 7f3e5b200640 1 -- start start 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e5c15ecb0 0x7f3e5c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5c0d00b0 con 0x7f3e5c0d3210 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5c0d0c30 con 0x7f3e5c15ecb0 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5c0d0390 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e53e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3e5c0d3210 0x7f3e5c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:58688/0 (socket says 172.21.15.187:58688) 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e53e00640 1 -- 172.21.15.187:0/2939827990 learned_addr learned my addr 172.21.15.187:0/2939827990 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-17T07:13:39.968 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.968+0000 7f3e5a800640 1 --2- 172.21.15.187:0/2939827990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.969 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e59e00640 1 --2- 172.21.15.187:0/2939827990 >> v2:172.21.15.67:3300/0 conn(0x7f3e5c15ecb0 0x7f3e5c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.969 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e59400640 1 -- 172.21.15.187:0/2939827990 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2115255529 0 0) 0x7f3e5c0d00b0 con 0x7f3e5c0d3210 2024-04-17T07:13:39.969 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e59400640 1 -- 172.21.15.187:0/2939827990 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e40003520 con 0x7f3e5c0d3210 2024-04-17T07:13:39.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e5a800640 1 -- 172.21.15.187:0/2939827990 >> v2:172.21.15.67:3300/0 conn(0x7f3e5c15ecb0 msgr2=0x7f3e5c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:39.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e5a800640 1 --2- 172.21.15.187:0/2939827990 >> v2:172.21.15.67:3300/0 conn(0x7f3e5c15ecb0 0x7f3e5c0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e5a800640 1 -- 172.21.15.187:0/2939827990 >> v1:172.21.15.187:6789/0 conn(0x7f3e5c0d3210 legacy=0x7f3e5c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.969+0000 7f3e5a800640 1 -- 172.21.15.187:0/2939827990 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c0d0670 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.970+0000 7f3e5a800640 1 --2- 172.21.15.187:0/2939827990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b3fc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009d90 tx=0x7f3e5402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=461047ba156a383b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.970+0000 7f3e59400640 1 -- 172.21.15.187:0/2939827990 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54039070 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.970+0000 7f3e59400640 1 -- 172.21.15.187:0/2939827990 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5402cbc0 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.970+0000 7f3e59400640 1 -- 172.21.15.187:0/2939827990 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54034440 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 -- 172.21.15.187:0/2939827990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 msgr2=0x7f3e5c0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2939827990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b3fc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009d90 tx=0x7f3e5402be10 comp rx=0 tx=0).stop 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 -- 172.21.15.187:0/2939827990 shutdown_connections 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2939827990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b3fc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2939827990 >> v2:172.21.15.67:3300/0 conn(0x7f3e5c15ecb0 0x7f3e5c0d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 -- 172.21.15.187:0/2939827990 >> 172.21.15.187:0/2939827990 conn(0x7f3e5c08bff0 msgr2=0x7f3e5c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:39.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 -- 172.21.15.187:0/2939827990 shutdown_connections 2024-04-17T07:13:39.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.971+0000 7f3e5b200640 1 -- 172.21.15.187:0/2939827990 wait complete. 2024-04-17T07:13:39.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.972+0000 7f3e5b200640 1 Processor -- start 2024-04-17T07:13:39.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.973+0000 7f3e5b200640 1 -- start start 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.973+0000 7f3e5b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.973+0000 7f3e5b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e5b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c15ecb0 0x7f3e5c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e5b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c15f210 con 0x7f3e5c0cfab0 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e5b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c15f090 con 0x7f3e5c0d3210 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e5b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5c0b4500 con 0x7f3e5c15ecb0 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49206/0 (socket says 172.21.15.187:49206) 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 -- 172.21.15.187:0/2824179279 learned_addr learned my addr 172.21.15.187:0/2824179279 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e5a800640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c15ecb0 0x7f3e5c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e59e00640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 -- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c15ecb0 msgr2=0x7f3e5c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c15ecb0 0x7f3e5c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 -- 172.21.15.187:0/2824179279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e5c0cfab0 msgr2=0x7f3e5c0b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.974+0000 7f3e53e00640 1 -- 172.21.15.187:0/2824179279 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c0b9f20 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e53e00640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c460 tx=0x7f3e4400c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44017070 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e44005ce0 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e44007290 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c0b8940 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c0b8f00 con 0x7f3e5c0d3210 2024-04-17T07:13:39.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3e4401a070 con 0x7f3e5c0d3210 2024-04-17T07:13:39.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.975+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3e4401c440 con 0x7f3e5c0d3210 2024-04-17T07:13:39.977 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:39.977+0000 7f6ef53bc040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:13:39.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.976+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e5c068a10 con 0x7f3e5c0d3210 2024-04-17T07:13:39.978 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:39.978+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f3e44021070 con 0x7f3e5c0d3210 2024-04-17T07:13:40.017 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.017+0000 7fb0b8638040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:13:40.054 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.054+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "18b200d0-3b8a-4c95-9ead-ff065c9bb590", "id": 2} v 0) -- 0x7f3e5c0b6bf0 con 0x7f3e5c0d3210 2024-04-17T07:13:40.061 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.062+0000 7f6ef53bc040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:13:40.062 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.062+0000 7f3e52000640 1 -- 172.21.15.187:0/2824179279 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18b200d0-3b8a-4c95-9ead-ff065c9bb590", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3e4401c7f0 con 0x7f3e5c0d3210 2024-04-17T07:13:40.062 INFO:teuthology.orchestra.run.smithi187.stdout:2 2024-04-17T07:13:40.063 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 msgr2=0x7f3e5c0b7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.063 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c460 tx=0x7f3e4400c920 comp rx=0 tx=0).stop 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 shutdown_connections 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e5c15ecb0 0x7f3e5c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5c0d3210 0x7f3e5c0b7ce0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 --2- 172.21.15.187:0/2824179279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e5c0cfab0 0x7f3e5c0b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.063+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 >> 172.21.15.187:0/2824179279 conn(0x7f3e5c08bff0 msgr2=0x7f3e5c15cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.064+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 shutdown_connections 2024-04-17T07:13:40.064 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.064+0000 7f3e5b200640 1 -- 172.21.15.187:0/2824179279 wait complete. 2024-04-17T07:13:40.074 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.075+0000 7fb0b8638040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:13:40.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 77a9ac54-619f-409f-8a76-2438f56f0ec9 3 2024-04-17T07:13:40.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.119+0000 7f6ef53bc040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:13:40.138 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.139+0000 7fb0b8638040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:13:40.182 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.183+0000 7f6ef53bc040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:13:40.197 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.197+0000 7fb0b8638040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:13:40.240 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.241+0000 7f6ef53bc040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:13:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.311+0000 7f7d7fe00640 1 Processor -- start 2024-04-17T07:13:40.311 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.311+0000 7f7d7fe00640 1 -- start start 2024-04-17T07:13:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d8015ecb0 0x7f7d800d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d800d00b0 con 0x7f7d800d3210 2024-04-17T07:13:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d800d0390 con 0x7f7d800cfab0 2024-04-17T07:13:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7d800d0c30 con 0x7f7d8015ecb0 2024-04-17T07:13:40.313 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7ea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d8015ecb0 0x7f7d800d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7f400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7d800cfab0 0x7f7d800b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:58690/0 (socket says 172.21.15.187:58690) 2024-04-17T07:13:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7ea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d8015ecb0 0x7f7d800d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49208/0 (socket says 172.21.15.187:49208) 2024-04-17T07:13:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.313+0000 7f7d7f400640 1 -- 172.21.15.187:0/3172389814 learned_addr learned my addr 172.21.15.187:0/3172389814 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-17T07:13:40.314 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.314+0000 7f7d7ea00640 1 -- 172.21.15.187:0/3172389814 >> v1:172.21.15.187:6789/0 conn(0x7f7d800cfab0 legacy=0x7f7d800b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.314+0000 7f7d7ea00640 1 -- 172.21.15.187:0/3172389814 >> v1:172.21.15.67:6789/0 conn(0x7f7d800d3210 legacy=0x7f7d800cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.314+0000 7f7d7ea00640 1 -- 172.21.15.187:0/3172389814 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d800d0670 con 0x7f7d8015ecb0 2024-04-17T07:13:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.315+0000 7f7d7ea00640 1 --2- 172.21.15.187:0/3172389814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d8015ecb0 0x7f7d800d2cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7d700097a0 tx=0x7f7d7002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a13f82361bfff833 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.315+0000 7f7d7e000640 1 -- 172.21.15.187:0/3172389814 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70035020 con 0x7f7d8015ecb0 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.315+0000 7f7d7e000640 1 -- 172.21.15.187:0/3172389814 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d70031830 con 0x7f7d8015ecb0 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.315+0000 7f7d7e000640 1 -- 172.21.15.187:0/3172389814 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70031c00 con 0x7f7d8015ecb0 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 -- 172.21.15.187:0/3172389814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d8015ecb0 msgr2=0x7f7d800d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/3172389814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d8015ecb0 0x7f7d800d2cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7d700097a0 tx=0x7f7d7002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 -- 172.21.15.187:0/3172389814 shutdown_connections 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/3172389814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d8015ecb0 0x7f7d800d2cd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 -- 172.21.15.187:0/3172389814 >> 172.21.15.187:0/3172389814 conn(0x7f7d8008bff0 msgr2=0x7f7d800d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 -- 172.21.15.187:0/3172389814 shutdown_connections 2024-04-17T07:13:40.316 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.316+0000 7f7d7fe00640 1 -- 172.21.15.187:0/3172389814 wait complete. 2024-04-17T07:13:40.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.318+0000 7f7d7fe00640 1 Processor -- start 2024-04-17T07:13:40.317 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.318+0000 7f7d7fe00640 1 -- start start 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d800d3210 0x7f7d800a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d8015ecb0 0x7f7d800a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8015f210 con 0x7f7d800d3210 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8015f090 con 0x7f7d800cfab0 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d800b46d0 con 0x7f7d8015ecb0 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49210/0 (socket says 172.21.15.187:49210) 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 -- 172.21.15.187:0/446197280 learned_addr learned my addr 172.21.15.187:0/446197280 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7f400640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d8015ecb0 0x7f7d800a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d77e00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d800d3210 0x7f7d800a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 -- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d8015ecb0 msgr2=0x7f7d800a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d8015ecb0 0x7f7d800a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 -- 172.21.15.187:0/446197280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d800d3210 msgr2=0x7f7d800a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.319+0000 7f7d7ea00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d800d3210 0x7f7d800a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d7ea00640 1 -- 172.21.15.187:0/446197280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8015f6e0 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d7ea00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d70034040 tx=0x7f7d700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70040070 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d70039a20 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70035040 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8015f9c0 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.320+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d80160260 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.321+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7d70038020 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.321+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7d7003a8b0 con 0x7f7d800cfab0 2024-04-17T07:13:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.322+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8015fb90 con 0x7f7d800cfab0 2024-04-17T07:13:40.325 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.324+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f7d70052020 con 0x7f7d800cfab0 2024-04-17T07:13:40.402 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.402+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "77a9ac54-619f-409f-8a76-2438f56f0ec9", "id": 3} v 0) -- 0x7f7d80068a10 con 0x7f7d800cfab0 2024-04-17T07:13:40.411 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.412+0000 7f7d76a00640 1 -- 172.21.15.187:0/446197280 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77a9ac54-619f-409f-8a76-2438f56f0ec9", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7d7003a420 con 0x7f7d800cfab0 2024-04-17T07:13:40.412 INFO:teuthology.orchestra.run.smithi187.stdout:3 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 msgr2=0x7f7d801f8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d70034040 tx=0x7f7d700046c0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 shutdown_connections 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d8015ecb0 0x7f7d800a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d800d3210 0x7f7d800a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 --2- 172.21.15.187:0/446197280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d800cfab0 0x7f7d801f8290 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 >> 172.21.15.187:0/446197280 conn(0x7f7d8008bff0 msgr2=0x7f7d8006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.414 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 shutdown_connections 2024-04-17T07:13:40.414 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.413+0000 7f7d7fe00640 1 -- 172.21.15.187:0/446197280 wait complete. 2024-04-17T07:13:40.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 24e32618-be8c-48e7-a978-94e71d7fc380 4 2024-04-17T07:13:40.665 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.665+0000 7f4034200640 1 Processor -- start 2024-04-17T07:13:40.665 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.665+0000 7f4034200640 1 -- start start 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4034200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f402c0d17a0 0x7f402c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4034200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402c0d4100 con 0x7f402c0cfa10 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4034200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402c163860 con 0x7f402c15eca0 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4034200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f402c0d03b0 con 0x7f402c0d17a0 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4031800640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f402c15eca0 0x7f402c0d39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:58702/0 (socket says 172.21.15.187:58702) 2024-04-17T07:13:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f4031800640 1 -- 172.21.15.187:0/1549583057 learned_addr learned my addr 172.21.15.187:0/1549583057 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899945907 0 0) 0x7f402c163860 con 0x7f402c15eca0 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.667+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4014003520 con 0x7f402c15eca0 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 39109193 0 0) 0x7f4014003520 con 0x7f402c15eca0 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f4032200640 1 --2- 172.21.15.187:0/1549583057 >> v2:172.21.15.67:3300/0 conn(0x7f402c0d17a0 0x7f402c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 >> v2:172.21.15.67:3300/0 conn(0x7f402c0d17a0 msgr2=0x7f402c0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 --2- 172.21.15.187:0/1549583057 >> v2:172.21.15.67:3300/0 conn(0x7f402c0d17a0 0x7f402c0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 >> v1:172.21.15.67:6789/0 conn(0x7f402c0cfa10 legacy=0x7f402c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c0d1d40 con 0x7f402c15eca0 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4020001670 con 0x7f402c15eca0 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 >> v1:172.21.15.187:6789/0 conn(0x7f402c15eca0 legacy=0x7f402c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f4014003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4014004800 0x7f4014004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40140052e0 0x7f401400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f401400c200 con 0x7f40140052e0 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f401400c3b0 con 0x7f4014003a00 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f401400c5b0 con 0x7f4014004800 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f4032200640 1 --2- 172.21.15.187:0/1549583057 >> v2:172.21.15.67:3300/0 conn(0x7f402c0d17a0 0x7f402c0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f4031800640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f4014003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.668+0000 7f4030e00640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4014004800 0x7f4014004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4032200640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40140052e0 0x7f401400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4031800640 1 -- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4014004800 msgr2=0x7f4014004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4031800640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4014004800 0x7f4014004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4031800640 1 -- 172.21.15.187:0/1549583057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40140052e0 msgr2=0x7f401400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4031800640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40140052e0 0x7f401400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4031800640 1 -- 172.21.15.187:0/1549583057 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4014003520 con 0x7f4014003a00 2024-04-17T07:13:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.669+0000 7f4032200640 1 -- 172.21.15.187:0/1549583057 reap_dead start 2024-04-17T07:13:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.670+0000 7f4031800640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f4014003e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f402000c160 tx=0x7f4014005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bddb85cfb6f0ee10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.670+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020007c60 con 0x7f4014003a00 2024-04-17T07:13:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.670+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4020030070 con 0x7f4014003a00 2024-04-17T07:13:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.670+0000 7f402be00640 1 -- 172.21.15.187:0/1549583057 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020005d50 con 0x7f4014003a00 2024-04-17T07:13:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 -- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 msgr2=0x7f4014003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f4014003e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f402000c160 tx=0x7f4014005830 comp rx=0 tx=0).stop 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 -- 172.21.15.187:0/1549583057 shutdown_connections 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 --2- 172.21.15.187:0/1549583057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f4014003e00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 -- 172.21.15.187:0/1549583057 >> 172.21.15.187:0/1549583057 conn(0x7f402c067600 msgr2=0x7f402c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 -- 172.21.15.187:0/1549583057 shutdown_connections 2024-04-17T07:13:40.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.671+0000 7f4034200640 1 -- 172.21.15.187:0/1549583057 wait complete. 2024-04-17T07:13:40.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.673+0000 7f4034200640 1 Processor -- start 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.673+0000 7f4034200640 1 -- start start 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402c0b6da0 0x7f402c14fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f402c0b6000 0x7f402c0b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f402c0d31f0 con 0x7f402c0b6da0 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f402c0d2f10 con 0x7f4014003a00 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4034200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f402c0d34d0 con 0x7f402c0b6000 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4031800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4031800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49238/0 (socket says 172.21.15.187:49238) 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4031800640 1 -- 172.21.15.187:0/2688653056 learned_addr learned my addr 172.21.15.187:0/2688653056 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.674+0000 7f4032200640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f402c0b6000 0x7f402c0b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4030e00640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402c0b6da0 0x7f402c14fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 -- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f402c0b6000 msgr2=0x7f402c0b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f402c0b6000 0x7f402c0b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 -- 172.21.15.187:0/2688653056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402c0b6da0 msgr2=0x7f402c14fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402c0b6da0 0x7f402c14fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 -- 172.21.15.187:0/2688653056 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c0ce690 con 0x7f4014003a00 2024-04-17T07:13:40.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4031800640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f402000c250 tx=0x7f402002e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020007c60 con 0x7f4014003a00 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4020004bf0 con 0x7f4014003a00 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020007070 con 0x7f4014003a00 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.675+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402c0cc6e0 con 0x7f4014003a00 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.676+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402c0ccca0 con 0x7f4014003a00 2024-04-17T07:13:40.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.676+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f40200075a0 con 0x7f4014003a00 2024-04-17T07:13:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.676+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4020035980 con 0x7f4014003a00 2024-04-17T07:13:40.678 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.676+0000 7fb0b8638040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:13:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.677+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f402c15c290 con 0x7f4014003a00 2024-04-17T07:13:40.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.680+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f402c15c290 con 0x7f4014003a00 2024-04-17T07:13:40.717 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.718+0000 7f6ef53bc040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:13:40.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.761+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "24e32618-be8c-48e7-a978-94e71d7fc380", "id": 4} v 0) -- 0x7f402c068ca0 con 0x7f4014003a00 2024-04-17T07:13:40.768 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.768+0000 7f4029600640 1 -- 172.21.15.187:0/2688653056 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24e32618-be8c-48e7-a978-94e71d7fc380", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4020035470 con 0x7f4014003a00 2024-04-17T07:13:40.768 INFO:teuthology.orchestra.run.smithi187.stdout:4 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.769+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 msgr2=0x7f402c0ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.769+0000 7f4034200640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f402000c250 tx=0x7f402002e070 comp rx=0 tx=0).stop 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 shutdown_connections 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f402c0b6000 0x7f402c0b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402c0b6da0 0x7f402c14fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 --2- 172.21.15.187:0/2688653056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4014003a00 0x7f402c0ce150 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 >> 172.21.15.187:0/2688653056 conn(0x7f402c067600 msgr2=0x7f402c15d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 shutdown_connections 2024-04-17T07:13:40.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.770+0000 7f4034200640 1 -- 172.21.15.187:0/2688653056 wait complete. 2024-04-17T07:13:40.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new faa69671-29c7-4e2f-8f96-d7a55545d56d 5 2024-04-17T07:13:40.847 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.847+0000 7fb0b8638040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:13:40.887 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:40.888+0000 7f6ef53bc040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:13:40.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.971+0000 7f5d47000640 1 Processor -- start 2024-04-17T07:13:40.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.971+0000 7f5d47000640 1 -- start start 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d47000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 0x7f5d4009b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d47000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d4009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d47000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d400ce550 con 0x7f5d4009afa0 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d47000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5d400cdcb0 con 0x7f5d4015fe60 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d47000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d400cdf90 con 0x7f5d400d1270 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d45200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d4009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d46600640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5d4009afa0 0x7f5d400d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:58718/0 (socket says 172.21.15.187:58718) 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d46600640 1 -- 172.21.15.187:0/4003480449 learned_addr learned my addr 172.21.15.187:0/4003480449 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-17T07:13:40.973 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.973+0000 7f5d45c00640 1 --2- 172.21.15.187:0/4003480449 >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 0x7f5d4009b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.974+0000 7f5d45200640 1 -- 172.21.15.187:0/4003480449 >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 msgr2=0x7f5d4009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.974+0000 7f5d45200640 1 --2- 172.21.15.187:0/4003480449 >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 0x7f5d4009b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.974+0000 7f5d45200640 1 -- 172.21.15.187:0/4003480449 >> v1:172.21.15.187:6789/0 conn(0x7f5d4009afa0 legacy=0x7f5d400d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.974+0000 7f5d45200640 1 -- 172.21.15.187:0/4003480449 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d400ce270 con 0x7f5d400d1270 2024-04-17T07:13:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.974+0000 7f5d45c00640 1 --2- 172.21.15.187:0/4003480449 >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 0x7f5d4009b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:40.975 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.975+0000 7f5d45200640 1 --2- 172.21.15.187:0/4003480449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d4009aa60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097a0 tx=0x7f5d2c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f454d91c0ba3e2da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.975+0000 7f5d3fe00640 1 -- 172.21.15.187:0/4003480449 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c038070 con 0x7f5d400d1270 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.975+0000 7f5d3fe00640 1 -- 172.21.15.187:0/4003480449 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d2c031700 con 0x7f5d400d1270 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.975+0000 7f5d3fe00640 1 -- 172.21.15.187:0/4003480449 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c031ad0 con 0x7f5d400d1270 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 -- 172.21.15.187:0/4003480449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 msgr2=0x7f5d4009aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 --2- 172.21.15.187:0/4003480449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d4009aa60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097a0 tx=0x7f5d2c02b7a0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 -- 172.21.15.187:0/4003480449 shutdown_connections 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 --2- 172.21.15.187:0/4003480449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d4009aa60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 --2- 172.21.15.187:0/4003480449 >> v2:172.21.15.67:3300/0 conn(0x7f5d4015fe60 0x7f5d4009b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.976 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 -- 172.21.15.187:0/4003480449 >> 172.21.15.187:0/4003480449 conn(0x7f5d400b3fc0 msgr2=0x7f5d400b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 -- 172.21.15.187:0/4003480449 shutdown_connections 2024-04-17T07:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.976+0000 7f5d47000640 1 -- 172.21.15.187:0/4003480449 wait complete. 2024-04-17T07:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.978+0000 7f5d47000640 1 Processor -- start 2024-04-17T07:13:40.977 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.978+0000 7f5d47000640 1 -- start start 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d400cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4015fe60 0x7f5d400ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4009bce0 con 0x7f5d4015fe60 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d40160240 con 0x7f5d4009afa0 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d47000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d400c9910 con 0x7f5d400d1270 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d400cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49248/0 (socket says 172.21.15.187:49248) 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 -- 172.21.15.187:0/502686102 learned_addr learned my addr 172.21.15.187:0/502686102 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:40.980 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d46600640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4015fe60 0x7f5d400ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 -- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 msgr2=0x7f5d400cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d400cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 -- 172.21.15.187:0/502686102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4015fe60 msgr2=0x7f5d400ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4015fe60 0x7f5d400ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.979+0000 7f5d45c00640 1 -- 172.21.15.187:0/502686102 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d400c9bf0 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d45c00640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400e7e0 tx=0x7f5d3400eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3400f040 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d34004030 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34004400 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d400c9ed0 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d40154da0 con 0x7f5d4009afa0 2024-04-17T07:13:40.981 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5d3400ca60 con 0x7f5d4009afa0 2024-04-17T07:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.980+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f5d3400cd50 con 0x7f5d4009afa0 2024-04-17T07:13:40.982 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.981+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d400755d0 con 0x7f5d4009afa0 2024-04-17T07:13:40.983 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:40.983+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f5d34017020 con 0x7f5d4009afa0 2024-04-17T07:13:40.986 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:40.986+0000 7fb0b8638040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:13:41.033 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.034+0000 7f6ef53bc040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:13:41.042 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.042+0000 7fb0b8638040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:13:41.062 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.062+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "faa69671-29c7-4e2f-8f96-d7a55545d56d", "id": 5} v 0) -- 0x7f5d40071980 con 0x7f5d4009afa0 2024-04-17T07:13:41.069 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.069+0000 7f5d3d600640 1 -- 172.21.15.187:0/502686102 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "faa69671-29c7-4e2f-8f96-d7a55545d56d", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f5d340136d0 con 0x7f5d4009afa0 2024-04-17T07:13:41.069 INFO:teuthology.orchestra.run.smithi187.stdout:5 2024-04-17T07:13:41.070 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 msgr2=0x7f5d40154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.070 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400e7e0 tx=0x7f5d3400eca0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 shutdown_connections 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4015fe60 0x7f5d400ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d400d1270 0x7f5d400cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 --2- 172.21.15.187:0/502686102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d4009afa0 0x7f5d40154860 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.070+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 >> 172.21.15.187:0/502686102 conn(0x7f5d400b3fc0 msgr2=0x7f5d40070d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.071+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 shutdown_connections 2024-04-17T07:13:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.071+0000 7f5d47000640 1 -- 172.21.15.187:0/502686102 wait complete. 2024-04-17T07:13:41.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 0d9b56be-9793-41b6-a062-a63e3a0b83e5 6 2024-04-17T07:13:41.090 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.091+0000 7f6ef53bc040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:13:41.179 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.179+0000 7fb0b8638040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:13:41.224 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.226+0000 7f6ef53bc040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:13:41.269 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.269+0000 7f8cc7e00640 1 Processor -- start 2024-04-17T07:13:41.270 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.270+0000 7f8cc7e00640 1 -- start start 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cc80d33e0 0x7f8cc80cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc80d00b0 con 0x7f8cc815ecb0 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8cc80d0c30 con 0x7f8cc80d33e0 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc80d0390 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cbfe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cc80d33e0 0x7f8cc80cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:38646/0 (socket says 172.21.15.187:38646) 2024-04-17T07:13:41.271 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.271+0000 7f8cc7400640 1 -- 172.21.15.187:0/1213533470 learned_addr learned my addr 172.21.15.187:0/1213533470 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:41.272 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.272+0000 7f8cc7400640 1 -- 172.21.15.187:0/1213533470 >> v2:172.21.15.187:3300/0 conn(0x7f8cc80d33e0 msgr2=0x7f8cc80cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:41.272 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.272+0000 7f8cc7400640 1 --2- 172.21.15.187:0/1213533470 >> v2:172.21.15.187:3300/0 conn(0x7f8cc80d33e0 0x7f8cc80cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.272 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.272+0000 7f8cc7400640 1 -- 172.21.15.187:0/1213533470 >> v1:172.21.15.67:6789/0 conn(0x7f8cc815ecb0 legacy=0x7f8cc80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.272 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.272+0000 7f8cc7400640 1 -- 172.21.15.187:0/1213533470 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc80d0670 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.273 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.273+0000 7f8cc7400640 1 --2- 172.21.15.187:0/1213533470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009e50 tx=0x7f8cc002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=743096f9734f100c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.273 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.273+0000 7f8cc6000640 1 -- 172.21.15.187:0/1213533470 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc002ca80 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.273 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.273+0000 7f8cc6000640 1 -- 172.21.15.187:0/1213533470 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc002cc20 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.273 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.273+0000 7f8cc6000640 1 -- 172.21.15.187:0/1213533470 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00343f0 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 -- 172.21.15.187:0/1213533470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 msgr2=0x7f8cc80b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/1213533470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009e50 tx=0x7f8cc002bee0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 -- 172.21.15.187:0/1213533470 shutdown_connections 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/1213533470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80b3fc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/1213533470 >> v2:172.21.15.187:3300/0 conn(0x7f8cc80d33e0 0x7f8cc80cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 -- 172.21.15.187:0/1213533470 >> 172.21.15.187:0/1213533470 conn(0x7f8cc808bff0 msgr2=0x7f8cc80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 -- 172.21.15.187:0/1213533470 shutdown_connections 2024-04-17T07:13:41.274 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.274+0000 7f8cc7e00640 1 -- 172.21.15.187:0/1213533470 wait complete. 2024-04-17T07:13:41.275 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.276+0000 7f8cc7e00640 1 Processor -- start 2024-04-17T07:13:41.275 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.276+0000 7f8cc7e00640 1 -- start start 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc80d33e0 0x7f8cc8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc80cfd00 con 0x7f8cc80d33e0 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc815f090 con 0x7f8cc815ecb0 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc815cff0 con 0x7f8cc80cf8e0 2024-04-17T07:13:41.277 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc6a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49268/0 (socket says 172.21.15.187:49268) 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 -- 172.21.15.187:0/2753392176 learned_addr learned my addr 172.21.15.187:0/2753392176 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cbfe00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc80d33e0 0x7f8cc8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 -- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 msgr2=0x7f8cc80a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 -- 172.21.15.187:0/2753392176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc80d33e0 msgr2=0x7f8cc8156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc80d33e0 0x7f8cc8156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc7400640 1 -- 172.21.15.187:0/2753392176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc80a2410 con 0x7f8cc815ecb0 2024-04-17T07:13:41.278 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.277+0000 7f8cc6a00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:13:41.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cc7400640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0036040 tx=0x7f8cc0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.279 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0043020 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc003e450 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc80ce0a0 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0009510 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc80ce660 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8cc0046020 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.278+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f8cc00489f0 con 0x7f8cc815ecb0 2024-04-17T07:13:41.280 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.279+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc8068a10 con 0x7f8cc815ecb0 2024-04-17T07:13:41.281 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.282+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f8cc004d030 con 0x7f8cc815ecb0 2024-04-17T07:13:41.297 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.298+0000 7fb0b8638040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:13:41.342 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.343+0000 7f6ef53bc040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:13:41.360 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.360+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0d9b56be-9793-41b6-a062-a63e3a0b83e5", "id": 6} v 0) -- 0x7f8cc80cb440 con 0x7f8cc815ecb0 2024-04-17T07:13:41.374 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.374+0000 7f8cbf400640 1 -- 172.21.15.187:0/2753392176 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d9b56be-9793-41b6-a062-a63e3a0b83e5", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f8cc002ca90 con 0x7f8cc815ecb0 2024-04-17T07:13:41.374 INFO:teuthology.orchestra.run.smithi187.stdout:6 2024-04-17T07:13:41.375 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 msgr2=0x7f8cc81f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.375 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0036040 tx=0x7f8cc0004a40 comp rx=0 tx=0).stop 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 shutdown_connections 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc815ecb0 0x7f8cc81f46a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc80d33e0 0x7f8cc8156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 --2- 172.21.15.187:0/2753392176 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc80cf8e0 0x7f8cc80a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.375+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 >> 172.21.15.187:0/2753392176 conn(0x7f8cc808bff0 msgr2=0x7f8cc80b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.376+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 shutdown_connections 2024-04-17T07:13:41.376 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.376+0000 7f8cc7e00640 1 -- 172.21.15.187:0/2753392176 wait complete. 2024-04-17T07:13:41.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 161cf315-6bb4-4c45-936c-59ec7d1e4e9c 7 2024-04-17T07:13:41.484 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.483+0000 7fb0b8638040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:13:41.529 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.530+0000 7f6ef53bc040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:13:41.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.535+0000 7fb0b8638040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:13:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.581+0000 7f0ce9200640 1 Processor -- start 2024-04-17T07:13:41.581 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.581+0000 7f0ce9200640 1 -- start start 2024-04-17T07:13:41.581 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.582+0000 7f6ef53bc040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.582+0000 7f0ce9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ce415ecb0 0x7f0ce40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce9200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce9200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ce40d00b0 con 0x7f0ce40d3210 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce9200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0ce40d0c30 con 0x7f0ce415ecb0 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce9200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce40d0390 con 0x7f0ce40cfab0 2024-04-17T07:13:41.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:38662/0 (socket says 172.21.15.187:38662) 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 -- 172.21.15.187:0/3441996880 learned_addr learned my addr 172.21.15.187:0/3441996880 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3400640 1 --2- 172.21.15.187:0/3441996880 >> v2:172.21.15.67:3300/0 conn(0x7f0ce415ecb0 0x7f0ce40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 -- 172.21.15.187:0/3441996880 >> v2:172.21.15.67:3300/0 conn(0x7f0ce415ecb0 msgr2=0x7f0ce40d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 --2- 172.21.15.187:0/3441996880 >> v2:172.21.15.67:3300/0 conn(0x7f0ce415ecb0 0x7f0ce40d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 -- 172.21.15.187:0/3441996880 >> v1:172.21.15.67:6789/0 conn(0x7f0ce40d3210 legacy=0x7f0ce40cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.583+0000 7f0ce3e00640 1 -- 172.21.15.187:0/3441996880 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce40d0670 con 0x7f0ce40cfab0 2024-04-17T07:13:41.584 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.584+0000 7f0ce3e00640 1 --2- 172.21.15.187:0/3441996880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009e50 tx=0x7f0cdc02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb11ab1e9db81ac server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.585 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce2a00640 1 -- 172.21.15.187:0/3441996880 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc02c9e0 con 0x7f0ce40cfab0 2024-04-17T07:13:41.585 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce2a00640 1 -- 172.21.15.187:0/3441996880 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cdc02cb80 con 0x7f0ce40cfab0 2024-04-17T07:13:41.585 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce2a00640 1 -- 172.21.15.187:0/3441996880 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0343f0 con 0x7f0ce40cfab0 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce9200640 1 -- 172.21.15.187:0/3441996880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 msgr2=0x7f0ce40b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce9200640 1 --2- 172.21.15.187:0/3441996880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009e50 tx=0x7f0cdc02be40 comp rx=0 tx=0).stop 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce9200640 1 -- 172.21.15.187:0/3441996880 shutdown_connections 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce9200640 1 --2- 172.21.15.187:0/3441996880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40b3fc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.585+0000 7f0ce9200640 1 --2- 172.21.15.187:0/3441996880 >> v2:172.21.15.67:3300/0 conn(0x7f0ce415ecb0 0x7f0ce40d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.586+0000 7f0ce9200640 1 -- 172.21.15.187:0/3441996880 >> 172.21.15.187:0/3441996880 conn(0x7f0ce408bff0 msgr2=0x7f0ce40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.586+0000 7f0ce9200640 1 -- 172.21.15.187:0/3441996880 shutdown_connections 2024-04-17T07:13:41.586 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.586+0000 7f0ce9200640 1 -- 172.21.15.187:0/3441996880 wait complete. 2024-04-17T07:13:41.587 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.587+0000 7f0ce9200640 1 Processor -- start 2024-04-17T07:13:41.587 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.587+0000 7f0ce9200640 1 -- start start 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ce40d3210 0x7f0ce41f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415ecb0 0x7f0ce4161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce415f210 con 0x7f0ce415ecb0 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce415f090 con 0x7f0ce40d3210 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.588+0000 7f0ce9200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ce40b4500 con 0x7f0ce40cfab0 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0cdbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ce40d3210 0x7f0ce41f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:38674/0 (socket says 172.21.15.187:38674) 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 -- 172.21.15.187:0/895530678 learned_addr learned my addr 172.21.15.187:0/895530678 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3e00640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415ecb0 0x7f0ce4161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 -- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ce40d3210 msgr2=0x7f0ce41f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.589 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ce40d3210 0x7f0ce41f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.590 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 -- 172.21.15.187:0/895530678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415ecb0 msgr2=0x7f0ce4161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.590 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415ecb0 0x7f0ce4161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.590 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 -- 172.21.15.187:0/895530678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce4161120 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0ce3400640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400d620 tx=0x7f0cd400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.589+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400b620 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd4019070 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400ba00 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce4161400 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce41619c0 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0cd400bba0 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.590+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0cd401eac0 con 0x7f0ce40cfab0 2024-04-17T07:13:41.591 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.591+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce4068a10 con 0x7f0ce40cfab0 2024-04-17T07:13:41.593 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.594+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f0cd401e460 con 0x7f0ce40cfab0 2024-04-17T07:13:41.672 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.672+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd new", "uuid": "161cf315-6bb4-4c45-936c-59ec7d1e4e9c", "id": 7} v 0) -- 0x7f0ce40b5a70 con 0x7f0ce40cfab0 2024-04-17T07:13:41.685 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.685+0000 7f0cdb400640 1 -- 172.21.15.187:0/895530678 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "161cf315-6bb4-4c45-936c-59ec7d1e4e9c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0cd401e640 con 0x7f0ce40cfab0 2024-04-17T07:13:41.685 INFO:teuthology.orchestra.run.smithi187.stdout:7 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 msgr2=0x7f0ce40ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400d620 tx=0x7f0cd400dae0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 shutdown_connections 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415ecb0 0x7f0ce4161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ce40d3210 0x7f0ce41f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 --2- 172.21.15.187:0/895530678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ce40cfab0 0x7f0ce40ce260 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 >> 172.21.15.187:0/895530678 conn(0x7f0ce408bff0 msgr2=0x7f0ce406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 shutdown_connections 2024-04-17T07:13:41.687 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:13:41.687+0000 7f0ce9200640 1 -- 172.21.15.187:0/895530678 wait complete. 2024-04-17T07:13:41.703 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-17T07:13:41.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-17T07:13:41.711 INFO:tasks.ceph.osd.0:Started 2024-04-17T07:13:41.711 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-17T07:13:41.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-17T07:13:41.718 INFO:tasks.ceph.osd.1:Started 2024-04-17T07:13:41.718 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-17T07:13:41.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-17T07:13:41.720 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:41.720+0000 7fb0b8638040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:13:41.724 INFO:tasks.ceph.osd.2:Started 2024-04-17T07:13:41.724 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-17T07:13:41.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-17T07:13:41.729 INFO:tasks.ceph.osd.3:Started 2024-04-17T07:13:41.729 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-17T07:13:41.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-17T07:13:41.734 INFO:tasks.ceph.osd.4:Started 2024-04-17T07:13:41.734 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-17T07:13:41.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-17T07:13:41.738 INFO:tasks.ceph.osd.5:Started 2024-04-17T07:13:41.738 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-17T07:13:41.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-17T07:13:41.743 INFO:tasks.ceph.osd.6:Started 2024-04-17T07:13:41.743 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-17T07:13:41.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-17T07:13:41.750 INFO:tasks.ceph.osd.7:Started 2024-04-17T07:13:41.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:41.782 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:41.783+0000 7f6ef53bc040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:13:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.974+0000 7f4b04600640 1 Processor -- start 2024-04-17T07:13:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.974+0000 7f4b04600640 1 -- start start 2024-04-17T07:13:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b04600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 0x7f4afc0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b04600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 0x7f4afc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b04600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4afc072b20 con 0x7f4afc0c6f90 2024-04-17T07:13:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b04600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4afc0ce960 con 0x7f4afc166440 2024-04-17T07:13:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b04600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4afc0c5eb0 con 0x7f4afc072420 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b01200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4afc0c6f90 0x7f4afc071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43252/0 (socket says 172.21.15.67:43252) 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b01200640 1 -- 172.21.15.67:0/2247693806 learned_addr learned my addr 172.21.15.67:0/2247693806 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b01c00640 1 --2- 172.21.15.67:0/2247693806 >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 0x7f4afc0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4b02600640 1 --2- 172.21.15.67:0/2247693806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 0x7f4afc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.976+0000 7f4afbe00640 1 -- 172.21.15.67:0/2247693806 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114562678 0 0) 0x7f4afc072b20 con 0x7f4afc0c6f90 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4afbe00640 1 -- 172.21.15.67:0/2247693806 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae4003520 con 0x7f4afc0c6f90 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4b02600640 1 -- 172.21.15.67:0/2247693806 >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 msgr2=0x7f4afc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4b02600640 1 --2- 172.21.15.67:0/2247693806 >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 0x7f4afc0ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4b02600640 1 -- 172.21.15.67:0/2247693806 >> v1:172.21.15.67:6789/0 conn(0x7f4afc0c6f90 legacy=0x7f4afc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4b02600640 1 -- 172.21.15.67:0/2247693806 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc0c6190 con 0x7f4afc072420 2024-04-17T07:13:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.977+0000 7f4b01c00640 1 --2- 172.21.15.67:0/2247693806 >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 0x7f4afc0ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:13:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.978+0000 7f4b02600640 1 --2- 172.21.15.67:0/2247693806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 0x7f4afc06dd50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4afc072cf0 tx=0x7f4af402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed18586afe4c1544 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.979+0000 7f4afbe00640 1 -- 172.21.15.67:0/2247693806 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4039070 con 0x7f4afc072420 2024-04-17T07:13:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.979+0000 7f4afbe00640 1 -- 172.21.15.67:0/2247693806 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af402cbb0 con 0x7f4afc072420 2024-04-17T07:13:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.979+0000 7f4afbe00640 1 -- 172.21.15.67:0/2247693806 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4034440 con 0x7f4afc072420 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.979+0000 7f4b04600640 1 -- 172.21.15.67:0/2247693806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 msgr2=0x7f4afc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.979+0000 7f4b04600640 1 --2- 172.21.15.67:0/2247693806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 0x7f4afc06dd50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4afc072cf0 tx=0x7f4af402be00 comp rx=0 tx=0).stop 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 -- 172.21.15.67:0/2247693806 shutdown_connections 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 --2- 172.21.15.67:0/2247693806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc072420 0x7f4afc06dd50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 --2- 172.21.15.67:0/2247693806 >> v2:172.21.15.187:3300/0 conn(0x7f4afc166440 0x7f4afc0ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 -- 172.21.15.67:0/2247693806 >> 172.21.15.67:0/2247693806 conn(0x7f4afc06bf50 msgr2=0x7f4afc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 -- 172.21.15.67:0/2247693806 shutdown_connections 2024-04-17T07:13:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.980+0000 7f4b04600640 1 -- 172.21.15.67:0/2247693806 wait complete. 2024-04-17T07:13:41.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.981+0000 7f4b04600640 1 Processor -- start 2024-04-17T07:13:41.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.982+0000 7f4b04600640 1 -- start start 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.982+0000 7f4b04600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.982+0000 7f4b04600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc0c6f90 0x7f4afc0b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.982+0000 7f4b04600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc166440 0x7f4afc15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.982+0000 7f4b04600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc0cf0d0 con 0x7f4afc072420 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b04600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc06e290 con 0x7f4afc166440 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b04600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4afc0b6020 con 0x7f4afc0c6f90 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc0c6f90 0x7f4afc0b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40838/0 (socket says 172.21.15.67:40838) 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 -- 172.21.15.67:0/2601879481 learned_addr learned my addr 172.21.15.67:0/2601879481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b02600640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc166440 0x7f4afc15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 -- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc0c6f90 msgr2=0x7f4afc0b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc0c6f90 0x7f4afc0b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 -- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc166440 msgr2=0x7f4afc15b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc166440 0x7f4afc15b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.983+0000 7f4b01c00640 1 -- 172.21.15.67:0/2601879481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc0b5da0 con 0x7f4afc072420 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4b01c00640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4af000e960 tx=0x7f4af000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0017070 con 0x7f4afc072420 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af000ca00 con 0x7f4afc072420 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af000cdd0 con 0x7f4afc072420 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc0b32d0 con 0x7f4afc072420 2024-04-17T07:13:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.984+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc0b3890 con 0x7f4afc072420 2024-04-17T07:13:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.985+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4af001a070 con 0x7f4afc072420 2024-04-17T07:13:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.985+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4afc0b3cc0 con 0x7f4afc072420 2024-04-17T07:13:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.985+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4af0009400 con 0x7f4afc072420 2024-04-17T07:13:41.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:41.990+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7f4afc0b5da0 con 0x7f4afc072420 2024-04-17T07:13:42.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.057+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4afc068b90 con 0x7f4afc072420 2024-04-17T07:13:42.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.058+0000 7f4af9600640 1 -- 172.21.15.67:0/2601879481 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4af00064d0 con 0x7f4afc072420 2024-04-17T07:13:42.057 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:42.057 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:41.675397+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:42.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 msgr2=0x7f4afc1f92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4af000e960 tx=0x7f4af000ee20 comp rx=0 tx=0).stop 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 shutdown_connections 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4afc166440 0x7f4afc15b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4afc0c6f90 0x7f4afc0b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 --2- 172.21.15.67:0/2601879481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4afc072420 0x7f4afc1f92d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.059+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 >> 172.21.15.67:0/2601879481 conn(0x7f4afc06bf50 msgr2=0x7f4afc165aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.060+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 shutdown_connections 2024-04-17T07:13:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:42.060+0000 7f4b04600640 1 -- 172.21.15.67:0/2601879481 wait complete. 2024-04-17T07:13:42.074 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-17T07:13:42.074 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-17T07:13:42.417 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-17T07:13:42.417+0000 7f4a58c11640 -1 Falling back to public interface 2024-04-17T07:13:42.440 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-17T07:13:42.441+0000 7efc84e68640 -1 Falling back to public interface 2024-04-17T07:13:42.456 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T07:13:42.457+0000 7f3a5bc14640 -1 Falling back to public interface 2024-04-17T07:13:42.457 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T07:13:42.458+0000 7fb9d7138640 -1 Falling back to public interface 2024-04-17T07:13:42.462 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T07:13:42.463+0000 7fe019819640 -1 Falling back to public interface 2024-04-17T07:13:42.467 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-17T07:13:42.467+0000 7fa5aba0f640 -1 Falling back to public interface 2024-04-17T07:13:42.467 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-17T07:13:42.467+0000 7f7e8e324640 -1 Falling back to public interface 2024-04-17T07:13:42.473 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T07:13:42.475+0000 7fa5b060e640 -1 Falling back to public interface 2024-04-17T07:13:42.819 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:42.819+0000 7fb0b8638040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:13:42.880 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:42.912 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:42.913+0000 7f6ef53bc040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:13:43.044 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.045+0000 7fe32cc00640 1 Processor -- start 2024-04-17T07:13:43.044 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.045+0000 7fe32cc00640 1 -- start start 2024-04-17T07:13:43.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.046+0000 7fe32cc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe328161fb0 0x7fe32806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:43.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.046+0000 7fe32cc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe32809b730 con 0x7fe3280711d0 2024-04-17T07:13:43.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.046+0000 7fe32cc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe32809ba10 con 0x7fe32806e7d0 2024-04-17T07:13:43.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.046+0000 7fe32cc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe32809bcf0 con 0x7fe328161fb0 2024-04-17T07:13:43.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3280711d0 0x7fe3280d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43264/0 (socket says 172.21.15.67:43264) 2024-04-17T07:13:43.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327e00640 1 -- 172.21.15.67:0/4286063574 learned_addr learned my addr 172.21.15.67:0/4286063574 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:43.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327400640 1 --2- 172.21.15.67:0/4286063574 >> v2:172.21.15.187:3300/0 conn(0x7fe328161fb0 0x7fe32806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe326000640 1 -- 172.21.15.67:0/4286063574 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1240286135 0 0) 0x7fe32809b730 con 0x7fe3280711d0 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe326000640 1 -- 172.21.15.67:0/4286063574 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe30c003520 con 0x7fe3280711d0 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327400640 1 -- 172.21.15.67:0/4286063574 >> v1:172.21.15.187:6789/0 conn(0x7fe32806e7d0 legacy=0x7fe328070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327400640 1 -- 172.21.15.67:0/4286063574 >> v1:172.21.15.67:6789/0 conn(0x7fe3280711d0 legacy=0x7fe3280d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.047+0000 7fe327400640 1 -- 172.21.15.67:0/4286063574 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32809bfd0 con 0x7fe328161fb0 2024-04-17T07:13:43.047 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe327400640 1 --2- 172.21.15.67:0/4286063574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe328161fb0 0x7fe32806dd30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3180097a0 tx=0x7fe31802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e25c606ab47ad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:43.048 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe326000640 1 -- 172.21.15.67:0/4286063574 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31802f810 con 0x7fe328161fb0 2024-04-17T07:13:43.048 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe326000640 1 -- 172.21.15.67:0/4286063574 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31802f9b0 con 0x7fe328161fb0 2024-04-17T07:13:43.048 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe326000640 1 -- 172.21.15.67:0/4286063574 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31802fd80 con 0x7fe328161fb0 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe32cc00640 1 -- 172.21.15.67:0/4286063574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe328161fb0 msgr2=0x7fe32806dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.049+0000 7fe32cc00640 1 --2- 172.21.15.67:0/4286063574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe328161fb0 0x7fe32806dd30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3180097a0 tx=0x7fe31802b9f0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.050+0000 7fe32cc00640 1 -- 172.21.15.67:0/4286063574 shutdown_connections 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.050+0000 7fe32cc00640 1 --2- 172.21.15.67:0/4286063574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe328161fb0 0x7fe32806dd30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.050+0000 7fe32cc00640 1 -- 172.21.15.67:0/4286063574 >> 172.21.15.67:0/4286063574 conn(0x7fe3280d2600 msgr2=0x7fe3280d2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.050+0000 7fe32cc00640 1 -- 172.21.15.67:0/4286063574 shutdown_connections 2024-04-17T07:13:43.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.050+0000 7fe32cc00640 1 -- 172.21.15.67:0/4286063574 wait complete. 2024-04-17T07:13:43.050 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.051+0000 7fe32cc00640 1 Processor -- start 2024-04-17T07:13:43.050 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.051+0000 7fe32cc00640 1 -- start start 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328161fb0 0x7fe3280c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 0x7fe3280cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe32806e270 con 0x7fe3280711d0 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe328162390 con 0x7fe3280c8930 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.052+0000 7fe32cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe32806e3f0 con 0x7fe328161fb0 2024-04-17T07:13:43.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40848/0 (socket says 172.21.15.67:40848) 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 -- 172.21.15.67:0/3106519365 learned_addr learned my addr 172.21.15.67:0/3106519365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327e00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 0x7fe3280cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe326a00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328161fb0 0x7fe3280c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 -- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328161fb0 msgr2=0x7fe3280c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328161fb0 0x7fe3280c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 -- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 msgr2=0x7fe3280cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 0x7fe3280cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 -- 172.21.15.67:0/3106519365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3280c96d0 con 0x7fe3280711d0 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327e00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 0x7fe3280cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:43.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.053+0000 7fe327400640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe31802c000 tx=0x7fe318004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:43.054 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe318047070 con 0x7fe3280711d0 2024-04-17T07:13:43.054 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3180378f0 con 0x7fe3280711d0 2024-04-17T07:13:43.054 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe318037ce0 con 0x7fe3280711d0 2024-04-17T07:13:43.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3280c6e10 con 0x7fe3280711d0 2024-04-17T07:13:43.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3280c73d0 con 0x7fe3280711d0 2024-04-17T07:13:43.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe31804b3f0 con 0x7fe3280711d0 2024-04-17T07:13:43.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.054+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe3180423f0 con 0x7fe3280711d0 2024-04-17T07:13:43.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.055+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32806d5c0 con 0x7fe3280711d0 2024-04-17T07:13:43.056 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.057+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73276 (secure 0 0 0) 0x7fe32806d5c0 con 0x7fe3280711d0 2024-04-17T07:13:43.124 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.125+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe328075df0 con 0x7fe3280711d0 2024-04-17T07:13:43.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.126+0000 7fe31f400640 1 -- 172.21.15.67:0/3106519365 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe318050080 con 0x7fe3280711d0 2024-04-17T07:13:43.125 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-17T07:13:43.125 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:41.675397+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:43.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 msgr2=0x7fe3280c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe31802c000 tx=0x7fe318004730 comp rx=0 tx=0).stop 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 shutdown_connections 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280c8930 0x7fe3280cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328161fb0 0x7fe3280c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 --2- 172.21.15.67:0/3106519365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280711d0 0x7fe3280c7eb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 >> 172.21.15.67:0/3106519365 conn(0x7fe3280d2600 msgr2=0x7fe32806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 shutdown_connections 2024-04-17T07:13:43.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:43.127+0000 7fe32cc00640 1 -- 172.21.15.67:0/3106519365 wait complete. 2024-04-17T07:13:43.134 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:43.135+0000 7fb0b8638040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:13:43.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-17T07:13:43.196 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:43.196+0000 7fb0b8638040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:13:43.264 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:43.265+0000 7f6ef53bc040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:13:43.328 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:43.329+0000 7f6ef53bc040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:13:43.331 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:43.331+0000 7fb0b8638040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:13:43.470 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:13:43.471+0000 7f6ef53bc040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:13:44.377 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:44.377+0000 7fb059000640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-17T07:13:44.377 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:13:44.377+0000 7fb059000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-17T07:13:46.028 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T07:13:46.029+0000 7fb9d7138640 -1 osd.3 0 log_to_monitors true 2024-04-17T07:13:46.057 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-17T07:13:46.058+0000 7efc84e68640 -1 osd.7 0 log_to_monitors true 2024-04-17T07:13:46.088 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-17T07:13:46.089+0000 7fa5aba0f640 -1 osd.5 0 log_to_monitors true 2024-04-17T07:13:46.113 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T07:13:46.114+0000 7f3a5bc14640 -1 osd.1 0 log_to_monitors true 2024-04-17T07:13:46.129 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-17T07:13:46.130+0000 7f7e8e324640 -1 osd.6 0 log_to_monitors true 2024-04-17T07:13:46.228 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-17T07:13:46.228+0000 7f4a58c11640 -1 osd.4 0 log_to_monitors true 2024-04-17T07:13:46.382 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T07:13:46.383+0000 7fe019819640 -1 osd.0 0 log_to_monitors true 2024-04-17T07:13:46.392 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T07:13:46.393+0000 7fa5b060e640 -1 osd.2 0 log_to_monitors true 2024-04-17T07:13:49.070 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T07:13:49.071+0000 7fb9d2000640 -1 osd.3 0 waiting for initial osdmap 2024-04-17T07:13:49.085 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T07:13:49.087+0000 7f3a56400640 -1 osd.1 0 waiting for initial osdmap 2024-04-17T07:13:49.097 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T07:13:49.098+0000 7fb9c8000640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.111 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T07:13:49.112+0000 7f3a4c400640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.696 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-17T07:13:49.696+0000 7f4a53600640 -1 osd.4 0 waiting for initial osdmap 2024-04-17T07:13:49.701 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-17T07:13:49.702+0000 7f7e89c00640 -1 osd.6 0 waiting for initial osdmap 2024-04-17T07:13:49.711 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-17T07:13:49.711+0000 7f4a49600640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.711 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-17T07:13:49.711+0000 7f7e7f200640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.714 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-17T07:13:49.714+0000 7efc7fe00640 -1 osd.7 0 waiting for initial osdmap 2024-04-17T07:13:49.723 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-17T07:13:49.724+0000 7efc75e00640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.727 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-17T07:13:49.728+0000 7fa5a7200640 -1 osd.5 0 waiting for initial osdmap 2024-04-17T07:13:49.731 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-17T07:13:49.732+0000 7fa59c800640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.946 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:49.974 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T07:13:49.975+0000 7fe014a00640 -1 osd.0 0 waiting for initial osdmap 2024-04-17T07:13:49.984 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T07:13:49.985+0000 7fe00a000640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:49.990 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T07:13:49.991+0000 7fa5aae00640 -1 osd.2 0 waiting for initial osdmap 2024-04-17T07:13:50.002 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T07:13:50.003+0000 7fa5a0e00640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.113+0000 7f6934c00640 1 Processor -- start 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.113+0000 7f6934c00640 1 -- start start 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.115+0000 7f6934c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 0x7f69300d3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.115+0000 7f6934c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69300cf1d0 0x7f69300b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.115+0000 7f6934c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69300cff00 con 0x7f69300d16e0 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.115+0000 7f6934c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f69300cfc20 con 0x7f69300cf1d0 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.115+0000 7f6934c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69300d2c40 con 0x7f693015ed10 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.116+0000 7f692de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69300d16e0 0x7f69300d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42348/0 (socket says 172.21.15.67:42348) 2024-04-17T07:13:50.115 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.116+0000 7f692de00640 1 -- 172.21.15.67:0/2773108415 learned_addr learned my addr 172.21.15.67:0/2773108415 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.116+0000 7f692d400640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 0x7f69300d3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.116+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916536703 0 0) 0x7f69300cff00 con 0x7f69300d16e0 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.116+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6914003520 con 0x7f69300d16e0 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1240492390 0 0) 0x7f6914003520 con 0x7f69300d16e0 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 msgr2=0x7f69300d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 0x7f69300d3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 >> v2:172.21.15.187:3300/0 conn(0x7f69300cf1d0 msgr2=0x7f69300b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:50.117 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 --2- 172.21.15.67:0/2773108415 >> v2:172.21.15.187:3300/0 conn(0x7f69300cf1d0 0x7f69300b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69300d2f20 con 0x7f69300d16e0 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f69280019f0 con 0x7f69300d16e0 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 >> v1:172.21.15.67:6789/0 conn(0x7f69300d16e0 legacy=0x7f69300d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69140039b0 0x7f6914003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 0x7f69140051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69300cfc20 con 0x7f6914004800 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69300d2c40 con 0x7f69140039b0 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f691400bbe0 con 0x7f693015ed10 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.117+0000 7f692d400640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.118+0000 7f692d400640 1 -- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 msgr2=0x7f69140051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.118+0000 7f692d400640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f693015ed10 0x7f69140051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.118+0000 7f692d400640 1 -- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69140039b0 msgr2=0x7f6914003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:50.118 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.118+0000 7f692d400640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69140039b0 0x7f6914003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.119 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.118+0000 7f692d400640 1 -- 172.21.15.67:0/2773108415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6914003520 con 0x7f6914004800 2024-04-17T07:13:50.119 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.119+0000 7f692d400640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6914004c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f691800ba00 tx=0x7f6914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e024bdf997e51a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.119+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918036020 con 0x7f6914004800 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.119+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6918032640 con 0x7f6914004800 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.119+0000 7f692ca00640 1 -- 172.21.15.67:0/2773108415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691800b600 con 0x7f6914004800 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.120+0000 7f6934c00640 1 -- 172.21.15.67:0/2773108415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 msgr2=0x7f6914004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.120+0000 7f6934c00640 1 --2- 172.21.15.67:0/2773108415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6914004c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f691800ba00 tx=0x7f6914005830 comp rx=0 tx=0).stop 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.120+0000 7f692e800640 1 -- 172.21.15.67:0/2773108415 reap_dead start 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.121+0000 7f6934c00640 1 -- 172.21.15.67:0/2773108415 shutdown_connections 2024-04-17T07:13:50.120 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.121+0000 7f6934c00640 1 -- 172.21.15.67:0/2773108415 >> 172.21.15.67:0/2773108415 conn(0x7f693006d180 msgr2=0x7f693006d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:50.121 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.121+0000 7f6934c00640 1 -- 172.21.15.67:0/2773108415 shutdown_connections 2024-04-17T07:13:50.121 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.121+0000 7f6934c00640 1 -- 172.21.15.67:0/2773108415 wait complete. 2024-04-17T07:13:50.122 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.123+0000 7f6934c00640 1 Processor -- start 2024-04-17T07:13:50.122 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.123+0000 7f6934c00640 1 -- start start 2024-04-17T07:13:50.124 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.124 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69300cd920 0x7f69300cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 0x7f69300b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69300b4190 con 0x7f6914004800 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69300b4010 con 0x7f69300cd920 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.124+0000 7f6934c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69300cf660 con 0x7f69300b59b0 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37452/0 (socket says 172.21.15.67:37452) 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 -- 172.21.15.67:0/230609398 learned_addr learned my addr 172.21.15.67:0/230609398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692d400640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69300cd920 0x7f69300cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692e800640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 0x7f69300b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 -- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 msgr2=0x7f69300b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 0x7f69300b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 -- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69300cd920 msgr2=0x7f69300cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69300cd920 0x7f69300cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 -- 172.21.15.67:0/230609398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69300b4910 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692e800640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 0x7f69300b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.125+0000 7f692de00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6928007fd0 tx=0x7f692800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.126+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928018070 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.126+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6928004d60 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.126+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692800ad00 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.126+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69300b4bf0 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.126+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69300b51b0 con 0x7f6914004800 2024-04-17T07:13:50.126 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.127+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6930069340 con 0x7f6914004800 2024-04-17T07:13:50.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.128+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f692801b070 con 0x7f6914004800 2024-04-17T07:13:50.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.128+0000 7f6926000640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 0x7f690c06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.129+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7f692808ae70 con 0x7f6914004800 2024-04-17T07:13:50.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.129+0000 7f692d400640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 0x7f690c06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.129+0000 7f692d400640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 0x7f690c06ed50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f693015f110 tx=0x7f691803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.134+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6928059160 con 0x7f6914004800 2024-04-17T07:13:50.328 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.329+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69301601d0 con 0x7f6914004800 2024-04-17T07:13:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.330+0000 7f6926000640 1 -- 172.21.15.67:0/230609398 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10861 (secure 0 0 0) 0x7f692805d460 con 0x7f6914004800 2024-04-17T07:13:50.329 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-17T07:13:50.329 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":14,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:50.093373+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 msgr2=0x7f690c06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 0x7f690c06ed50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f693015f110 tx=0x7f691803d040 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 msgr2=0x7f6930159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6928007fd0 tx=0x7f692800a430 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 shutdown_connections 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f690c06c890 0x7f690c06ed50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69300b59b0 0x7f69300b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69300cd920 0x7f69300cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 --2- 172.21.15.67:0/230609398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6914004800 0x7f6930159ed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.332+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 >> 172.21.15.67:0/230609398 conn(0x7f693006d180 msgr2=0x7f69300b9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.333+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 shutdown_connections 2024-04-17T07:13:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T07:13:50.333+0000 7f6934c00640 1 -- 172.21.15.67:0/230609398 wait complete. 2024-04-17T07:13:50.345 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-17T07:13:50.345 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-17T07:13:50.345 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:13:50.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:13:50.389 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:13:50.390 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:13:50.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:13:50.394 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:13:50.394 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:13:50.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:13:50.404 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:13:50.405 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:13:50.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:13:50.413 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:13:50.414 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-17T07:13:50.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:50.543 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:13:50.543 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:13:50.568 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:13:50.568 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:13:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.591+0000 7f007e600640 1 Processor -- start 2024-04-17T07:13:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.591+0000 7f007e600640 1 -- start start 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f007e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f007815ed10 0x7f00780d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f007e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00780d3020 con 0x7f00780cf640 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f007e600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00780d3300 con 0x7f00780d2ab0 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f007e600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f00780d3c30 con 0x7f007815ed10 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f0076a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f007815ed10 0x7f00780d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f0077400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00780cf640 0x7f00780cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42372/0 (socket says 172.21.15.67:42372) 2024-04-17T07:13:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.592+0000 7f0077400640 1 -- 172.21.15.67:0/1101342822 learned_addr learned my addr 172.21.15.67:0/1101342822 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.593+0000 7f0076a00640 1 -- 172.21.15.67:0/1101342822 >> v1:172.21.15.187:6789/0 conn(0x7f00780d2ab0 legacy=0x7f00780b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.593+0000 7f0076a00640 1 -- 172.21.15.67:0/1101342822 >> v1:172.21.15.67:6789/0 conn(0x7f00780cf640 legacy=0x7f00780cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.593+0000 7f0076a00640 1 -- 172.21.15.67:0/1101342822 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00780d35e0 con 0x7f007815ed10 2024-04-17T07:13:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.594+0000 7f0076a00640 1 --2- 172.21.15.67:0/1101342822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780d40c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00640097a0 tx=0x7f006402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92fef3c0f4aad15 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.594+0000 7f0076000640 1 -- 172.21.15.67:0/1101342822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006402f810 con 0x7f007815ed10 2024-04-17T07:13:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.594+0000 7f0076000640 1 -- 172.21.15.67:0/1101342822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006402f9b0 con 0x7f007815ed10 2024-04-17T07:13:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.594+0000 7f0076000640 1 -- 172.21.15.67:0/1101342822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006402fd80 con 0x7f007815ed10 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 -- 172.21.15.67:0/1101342822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 msgr2=0x7f00780d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 --2- 172.21.15.67:0/1101342822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780d40c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00640097a0 tx=0x7f006402b9f0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 -- 172.21.15.67:0/1101342822 shutdown_connections 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 --2- 172.21.15.67:0/1101342822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780d40c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 -- 172.21.15.67:0/1101342822 >> 172.21.15.67:0/1101342822 conn(0x7f007806cbd0 msgr2=0x7f007806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 -- 172.21.15.67:0/1101342822 shutdown_connections 2024-04-17T07:13:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.595+0000 7f007e600640 1 -- 172.21.15.67:0/1101342822 wait complete. 2024-04-17T07:13:50.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.596+0000 7f007e600640 1 Processor -- start 2024-04-17T07:13:50.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.596+0000 7f007e600640 1 -- start start 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 0x7f0078159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00780d2ab0 0x7f00780cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00780d4600 con 0x7f007815ed10 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00780d25b0 con 0x7f00780cf640 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.597+0000 7f007e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00780d4780 con 0x7f00780d2ab0 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 0x7f0078159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37496/0 (socket says 172.21.15.67:37496) 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0076a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00780d2ab0 0x7f00780cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 -- 172.21.15.67:0/758676950 learned_addr learned my addr 172.21.15.67:0/758676950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 -- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00780d2ab0 msgr2=0x7f00780cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00780d2ab0 0x7f00780cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 -- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 msgr2=0x7f0078159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 0x7f0078159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 -- 172.21.15.67:0/758676950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007815c070 con 0x7f007815ed10 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077400640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 0x7f0078159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.598+0000 7f0077e00640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f007000c7d0 tx=0x7f007000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.599+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070003b70 con 0x7f007815ed10 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.599+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0070003d30 con 0x7f007815ed10 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.599+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007000aa90 con 0x7f007815ed10 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.599+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007815c350 con 0x7f007815ed10 2024-04-17T07:13:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.599+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f007815c910 con 0x7f007815ed10 2024-04-17T07:13:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.600+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00780ba300 con 0x7f007815ed10 2024-04-17T07:13:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.601+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f007000ac30 con 0x7f007815ed10 2024-04-17T07:13:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.602+0000 7f006f400640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 0x7f005806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.602+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7f007008b3a0 con 0x7f007815ed10 2024-04-17T07:13:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.602+0000 7f0077400640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 0x7f005806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.603+0000 7f0077400640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 0x7f005806ed20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0068005880 tx=0x7f0068009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.606+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f00780ba300 con 0x7f007815ed10 2024-04-17T07:13:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.804+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0078069120 con 0x7f007815ed10 2024-04-17T07:13:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.806+0000 7f006f400640 1 -- 172.21.15.67:0/758676950 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10861 (secure 0 0 0) 0x7f0078069120 con 0x7f007815ed10 2024-04-17T07:13:50.805 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:50.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":14,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:50.093373+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.808+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 msgr2=0x7f005806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.808+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 0x7f005806ed20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0068005880 tx=0x7f0068009040 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.808+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 msgr2=0x7f00780b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.808+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f007000c7d0 tx=0x7f007000cc90 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 shutdown_connections 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f007815ed10 0x7f00780b6d60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f005806c860 0x7f005806ed20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00780d2ab0 0x7f00780cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 --2- 172.21.15.67:0/758676950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00780cf640 0x7f0078159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 >> 172.21.15.67:0/758676950 conn(0x7f007806cbd0 msgr2=0x7f0078160b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 shutdown_connections 2024-04-17T07:13:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:50.809+0000 7f007e600640 1 -- 172.21.15.67:0/758676950 wait complete. 2024-04-17T07:13:50.825 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-17T07:13:50.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:13:51.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.028+0000 7f3264a00640 1 Processor -- start 2024-04-17T07:13:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.029+0000 7f3264a00640 1 -- start start 2024-04-17T07:13:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f3264a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32600b6d00 0x7f32600cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f3264a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f3264a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600d2d30 con 0x7f326015ec60 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f3264a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f32600d49c0 con 0x7f32600b6d00 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f3264a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32600d3660 con 0x7f32600cf910 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f325fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f325ea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32600b6d00 0x7f32600cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f325f400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f326015ec60 0x7f32600b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:53456/0 (socket says 172.21.15.67:53456) 2024-04-17T07:13:51.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.030+0000 7f325f400640 1 -- 172.21.15.67:0/1523938601 learned_addr learned my addr 172.21.15.67:0/1523938601 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:51.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.031+0000 7f325e000640 1 -- 172.21.15.67:0/1523938601 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1146936243 0 0) 0x7f32600d2d30 con 0x7f326015ec60 2024-04-17T07:13:51.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.031+0000 7f325ea00640 1 -- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 msgr2=0x7f32600b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.031+0000 7f325ea00640 1 --2- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.031+0000 7f325ea00640 1 -- 172.21.15.67:0/1523938601 >> v1:172.21.15.187:6789/0 conn(0x7f326015ec60 legacy=0x7f32600b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.031+0000 7f325ea00640 1 -- 172.21.15.67:0/1523938601 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600d3010 con 0x7f32600b6d00 2024-04-17T07:13:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.033+0000 7f325ea00640 1 --2- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600b6d00 0x7f32600cf3d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f325002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d261ba86d2b66850 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.033+0000 7f325e000640 1 -- 172.21.15.67:0/1523938601 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250039070 con 0x7f32600b6d00 2024-04-17T07:13:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.033+0000 7f325e000640 1 -- 172.21.15.67:0/1523938601 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325002cc50 con 0x7f32600b6d00 2024-04-17T07:13:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.033+0000 7f325e000640 1 -- 172.21.15.67:0/1523938601 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250034440 con 0x7f32600b6d00 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 -- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600b6d00 msgr2=0x7f32600cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 --2- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600b6d00 0x7f32600cf3d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3250009d90 tx=0x7f325002bea0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 -- 172.21.15.67:0/1523938601 shutdown_connections 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 --2- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 --2- 172.21.15.67:0/1523938601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32600b6d00 0x7f32600cf3d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 -- 172.21.15.67:0/1523938601 >> 172.21.15.67:0/1523938601 conn(0x7f32600d15f0 msgr2=0x7f32600d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 -- 172.21.15.67:0/1523938601 shutdown_connections 2024-04-17T07:13:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.034+0000 7f3264a00640 1 -- 172.21.15.67:0/1523938601 wait complete. 2024-04-17T07:13:51.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.035+0000 7f3264a00640 1 Processor -- start 2024-04-17T07:13:51.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.036+0000 7f3264a00640 1 -- start start 2024-04-17T07:13:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.036+0000 7f3264a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.036+0000 7f3264a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f3264a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f326015ec60 0x7f32601e8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f3264a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f326015f040 con 0x7f32600b6d00 2024-04-17T07:13:51.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f3264a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f32600cfdc0 con 0x7f326015ec60 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f3264a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32600d00c0 con 0x7f32600cf910 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37512/0 (socket says 172.21.15.67:37512) 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 -- 172.21.15.67:0/2368336325 learned_addr learned my addr 172.21.15.67:0/2368336325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325ea00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325fe00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f326015ec60 0x7f32601e8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 -- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 msgr2=0x7f32600ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 -- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f326015ec60 msgr2=0x7f32601e8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f326015ec60 0x7f32601e8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.037+0000 7f325f400640 1 -- 172.21.15.67:0/2368336325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601f8920 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f325f400640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f324800c430 tx=0x7f324800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32480074e0 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32480047e0 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3248004bd0 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601f8080 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.038+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601f8640 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.039+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3260068a10 con 0x7f32600b6d00 2024-04-17T07:13:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.040+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f3248011070 con 0x7f32600b6d00 2024-04-17T07:13:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.041+0000 7f3257400640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 0x7f323c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.041+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7f324808b260 con 0x7f32600b6d00 2024-04-17T07:13:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.042+0000 7f325ea00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 0x7f323c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.043+0000 7f325ea00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 0x7f323c06ed20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3250036000 tx=0x7f3250030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.046+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3248059c30 con 0x7f32600b6d00 2024-04-17T07:13:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.281+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32600b9b00 con 0x7f32600b6d00 2024-04-17T07:13:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.282+0000 7f3257400640 1 -- 172.21.15.67:0/2368336325 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f3248059530 con 0x7f32600b6d00 2024-04-17T07:13:51.282 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:51.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-17T07:13:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 msgr2=0x7f323c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 0x7f323c06ed20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3250036000 tx=0x7f3250030000 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 msgr2=0x7f32601f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f324800c430 tx=0x7f324800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 shutdown_connections 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f326015ec60 0x7f32601e8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f323c06c860 0x7f323c06ed20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32600cf910 0x7f32600ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 --2- 172.21.15.67:0/2368336325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32600b6d00 0x7f32601f7aa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 >> 172.21.15.67:0/2368336325 conn(0x7f32600d15f0 msgr2=0x7f32600cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 shutdown_connections 2024-04-17T07:13:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:51.285+0000 7f3264a00640 1 -- 172.21.15.67:0/2368336325 wait complete. 2024-04-17T07:13:53.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:13:53.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:13:53.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:13:53.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.542+0000 7f3505e00640 1 Processor -- start 2024-04-17T07:13:53.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.542+0000 7f3505e00640 1 -- start start 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f350015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f35000b3c50 0x7f350006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35000705f0 0x7f3500070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f350015f620 con 0x7f35000705f0 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f35000b39d0 con 0x7f35000b3c50 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f350006bfa0 con 0x7f35000d1080 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35000705f0 0x7f3500070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.543+0000 7f3505400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35000705f0 0x7f3500070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56740/0 (socket says 172.21.15.67:56740) 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 -- 172.21.15.67:0/3797724350 learned_addr learned my addr 172.21.15.67:0/3797724350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3504a00640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f350015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f34ffe00640 1 --2- 172.21.15.67:0/3797724350 >> v2:172.21.15.187:3300/0 conn(0x7f35000b3c50 0x7f350006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 -- 172.21.15.67:0/3797724350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 msgr2=0x7f350015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f350015fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 -- 172.21.15.67:0/3797724350 >> v2:172.21.15.187:3300/0 conn(0x7f35000b3c50 msgr2=0x7f350006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 --2- 172.21.15.67:0/3797724350 >> v2:172.21.15.187:3300/0 conn(0x7f35000b3c50 0x7f350006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.544+0000 7f3505400640 1 -- 172.21.15.67:0/3797724350 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350006c280 con 0x7f35000705f0 2024-04-17T07:13:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.545+0000 7f3505400640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f3500070aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3500071900 tx=0x7f34f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ada524e9b71864 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.545+0000 7f34ff400640 1 -- 172.21.15.67:0/3797724350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f802ca80 con 0x7f35000705f0 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f34ff400640 1 -- 172.21.15.67:0/3797724350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f802cc20 con 0x7f35000705f0 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f34ff400640 1 -- 172.21.15.67:0/3797724350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f80343f0 con 0x7f35000705f0 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 -- 172.21.15.67:0/3797724350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 msgr2=0x7f3500070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f3500070aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3500071900 tx=0x7f34f802bee0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 -- 172.21.15.67:0/3797724350 shutdown_connections 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f3500070aa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 --2- 172.21.15.67:0/3797724350 >> v2:172.21.15.187:3300/0 conn(0x7f35000b3c50 0x7f350006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 --2- 172.21.15.67:0/3797724350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f350015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 -- 172.21.15.67:0/3797724350 >> 172.21.15.67:0/3797724350 conn(0x7f350009aa60 msgr2=0x7f35000b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 -- 172.21.15.67:0/3797724350 shutdown_connections 2024-04-17T07:13:53.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.546+0000 7f3505e00640 1 -- 172.21.15.67:0/3797724350 wait complete. 2024-04-17T07:13:53.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.548+0000 7f3505e00640 1 Processor -- start 2024-04-17T07:13:53.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.548+0000 7f3505e00640 1 -- start start 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.549+0000 7f3505e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.549+0000 7f3505e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35000b3c50 0x7f350014da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3505e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f35001e9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3505e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3500162550 con 0x7f35000705f0 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3505e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f350015ef60 con 0x7f35000b3c50 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3505e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f35001626d0 con 0x7f35000d1080 2024-04-17T07:13:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56752/0 (socket says 172.21.15.67:56752) 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 -- 172.21.15.67:0/4294941487 learned_addr learned my addr 172.21.15.67:0/4294941487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3505400640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f35001e9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f34ffe00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35000b3c50 0x7f350014da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 -- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 msgr2=0x7f35001e9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f35001e9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 -- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35000b3c50 msgr2=0x7f350014da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35000b3c50 0x7f350014da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.550+0000 7f3504a00640 1 -- 172.21.15.67:0/4294941487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350014dfa0 con 0x7f35000705f0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f3504a00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34f400e8a0 tx=0x7f34f400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4017070 con 0x7f35000705f0 2024-04-17T07:13:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f400c940 con 0x7f35000705f0 2024-04-17T07:13:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f400cd30 con 0x7f35000705f0 2024-04-17T07:13:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35000c9fa0 con 0x7f35000705f0 2024-04-17T07:13:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.551+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35000ca560 con 0x7f35000705f0 2024-04-17T07:13:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.552+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35000755d0 con 0x7f35000705f0 2024-04-17T07:13:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.554+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f34f401a070 con 0x7f35000705f0 2024-04-17T07:13:53.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.554+0000 7f34fcc00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 0x7f34e006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:53.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.555+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f34f408b540 con 0x7f35000705f0 2024-04-17T07:13:53.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.555+0000 7f34ffe00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 0x7f34e006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:53.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.556+0000 7f34ffe00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 0x7f34e006ed20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f350014e410 tx=0x7f34ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:53.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.560+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f350014dfa0 con 0x7f35000705f0 2024-04-17T07:13:53.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.819+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35000b7480 con 0x7f35000705f0 2024-04-17T07:13:53.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.820+0000 7f34fcc00640 1 -- 172.21.15.67:0/4294941487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7f34f4059da0 con 0x7f35000705f0 2024-04-17T07:13:53.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:53.820 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-17T07:13:53.820 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-04-17T07:13:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 msgr2=0x7f34e006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 0x7f34e006ed20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f350014e410 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-04-17T07:13:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 msgr2=0x7f35000cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34f400e8a0 tx=0x7f34f400ed60 comp rx=0 tx=0).stop 2024-04-17T07:13:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 shutdown_connections 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f34e006c860 0x7f34e006ed20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35000d1080 0x7f35001e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35000b3c50 0x7f350014da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 --2- 172.21.15.67:0/4294941487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35000705f0 0x7f35000cb040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 >> 172.21.15.67:0/4294941487 conn(0x7f350009aa60 msgr2=0x7f3500158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 shutdown_connections 2024-04-17T07:13:53.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:53.823+0000 7f3505e00640 1 -- 172.21.15.67:0/4294941487 wait complete. 2024-04-17T07:13:53.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.052+0000 7f240fe00640 1 Processor -- start 2024-04-17T07:13:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.052+0000 7f240fe00640 1 -- start start 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24080cf640 0x7f24080cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24080d2ab0 0x7f24080b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24080cff60 con 0x7f240815ed10 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f24080d3c30 con 0x7f24080d2ab0 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24080d3020 con 0x7f24080cf640 2024-04-17T07:13:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f240815ed10 0x7f24080d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49378/0 (socket says 172.21.15.67:49378) 2024-04-17T07:13:54.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.053+0000 7f240ca00640 1 -- 172.21.15.67:0/143398044 learned_addr learned my addr 172.21.15.67:0/143398044 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:54.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f240d400640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24080cf640 0x7f24080cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974705881 0 0) 0x7f24080cff60 con 0x7f240815ed10 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> v2:172.21.15.187:3300/0 conn(0x7f24080d2ab0 0x7f24080b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f0003520 con 0x7f240815ed10 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1632360220 0 0) 0x7f23f0003520 con 0x7f240815ed10 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24080cf640 msgr2=0x7f24080cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24080cf640 0x7f24080cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 >> v2:172.21.15.187:3300/0 conn(0x7f24080d2ab0 msgr2=0x7f24080b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 --2- 172.21.15.67:0/143398044 >> v2:172.21.15.187:3300/0 conn(0x7f24080d2ab0 0x7f24080b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24080d3300 con 0x7f240815ed10 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f23f4001670 con 0x7f240815ed10 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 >> v1:172.21.15.67:6789/0 conn(0x7f240815ed10 legacy=0x7f24080d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00039b0 0x7f23f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23f0004800 0x7f23f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f00051c0 0x7f23f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24080d3c30 con 0x7f23f00051c0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f24080d3020 con 0x7f23f00039b0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23f000c040 con 0x7f23f0004800 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> v2:172.21.15.187:3300/0 conn(0x7f24080d2ab0 0x7f24080b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.054+0000 7f240d400640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24080cf640 0x7f24080cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f00051c0 0x7f23f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240ca00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23f0004800 0x7f23f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240d400640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00039b0 0x7f23f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 -- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23f0004800 msgr2=0x7f23f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23f0004800 0x7f23f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 -- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00039b0 msgr2=0x7f23f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00039b0 0x7f23f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240de00640 1 -- 172.21.15.67:0/143398044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f0003520 con 0x7f23f00051c0 2024-04-17T07:13:54.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.055+0000 7f240d400640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00039b0 0x7f23f0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.056+0000 7f240de00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f00051c0 0x7f23f000b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24000073e0 tx=0x7f23f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce24fb76838c07b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400039070 con 0x7f23f00051c0 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f240002cdc0 con 0x7f23f00051c0 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f2407e00640 1 -- 172.21.15.67:0/143398044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400009c70 con 0x7f23f00051c0 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 -- 172.21.15.67:0/143398044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f00051c0 msgr2=0x7f23f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 --2- 172.21.15.67:0/143398044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f00051c0 0x7f23f000b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24000073e0 tx=0x7f23f0005830 comp rx=0 tx=0).stop 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240de00640 1 -- 172.21.15.67:0/143398044 reap_dead start 2024-04-17T07:13:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 -- 172.21.15.67:0/143398044 shutdown_connections 2024-04-17T07:13:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 -- 172.21.15.67:0/143398044 >> 172.21.15.67:0/143398044 conn(0x7f240806cbd0 msgr2=0x7f240806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 -- 172.21.15.67:0/143398044 shutdown_connections 2024-04-17T07:13:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.057+0000 7f240fe00640 1 -- 172.21.15.67:0/143398044 wait complete. 2024-04-17T07:13:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.059+0000 7f240fe00640 1 Processor -- start 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.059+0000 7f240fe00640 1 -- start start 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00051c0 0x7f240815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24081606d0 0x7f2408160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24080d25b0 con 0x7f24080cba40 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f24080d38c0 con 0x7f23f00051c0 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.060+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24080ba300 con 0x7f24081606d0 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56778/0 (socket says 172.21.15.67:56778) 2024-04-17T07:13:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 -- 172.21.15.67:0/3661729395 learned_addr learned my addr 172.21.15.67:0/3661729395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 -- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24081606d0 msgr2=0x7f2408160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240ca00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24081606d0 0x7f2408160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240d400640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00051c0 0x7f240815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24081606d0 0x7f2408160b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 -- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00051c0 msgr2=0x7f240815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00051c0 0x7f240815af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 -- 172.21.15.67:0/3661729395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24080cc430 con 0x7f24080cba40 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.061+0000 7f240de00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24000073e0 tx=0x7f2400004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.062+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400039030 con 0x7f24080cba40 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.062+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2400043070 con 0x7f24080cba40 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.062+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400009c70 con 0x7f24080cba40 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.062+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24080cc700 con 0x7f24080cba40 2024-04-17T07:13:54.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.062+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24080b44b0 con 0x7f24080cba40 2024-04-17T07:13:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.063+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24080ccc20 con 0x7f24080cba40 2024-04-17T07:13:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.064+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f240003d7d0 con 0x7f24080cba40 2024-04-17T07:13:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.065+0000 7f2405600640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 0x7f23e806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.065+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f24000ada50 con 0x7f24080cba40 2024-04-17T07:13:54.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.069+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f240007c210 con 0x7f24080cba40 2024-04-17T07:13:54.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.077+0000 7f240d400640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 0x7f23e806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.078+0000 7f240d400640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 0x7f23e806ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006fd0 tx=0x7f23fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.288+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2408068b90 con 0x7f24080cba40 2024-04-17T07:13:54.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.290+0000 7f2405600640 1 -- 172.21.15.67:0/3661729395 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12836 (secure 0 0 0) 0x7f240007bb10 con 0x7f24080cba40 2024-04-17T07:13:54.291 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:54.291 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:54.123341+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.292+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 msgr2=0x7f23e806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.292+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 0x7f23e806ed50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006fd0 tx=0x7f23fc008040 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.292+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 msgr2=0x7f24080cbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.292+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24000073e0 tx=0x7f2400004ab0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 shutdown_connections 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24080cba40 0x7f24080cbef0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f23e806c890 0x7f23e806ed50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24081606d0 0x7f2408160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 --2- 172.21.15.67:0/3661729395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f00051c0 0x7f240815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 >> 172.21.15.67:0/3661729395 conn(0x7f240806cbd0 msgr2=0x7f24080c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:54.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 shutdown_connections 2024-04-17T07:13:54.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.293+0000 7f240fe00640 1 -- 172.21.15.67:0/3661729395 wait complete. 2024-04-17T07:13:54.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:13:54.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:13:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.517+0000 7f7a8ae00640 1 Processor -- start 2024-04-17T07:13:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.517+0000 7f7a8ae00640 1 -- start start 2024-04-17T07:13:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.518+0000 7f7a8ae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a84142e20 0x7f7a840cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.518+0000 7f7a8ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a840cf160 0x7f7a840cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a8ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a841607a0 con 0x7f7a84143200 2024-04-17T07:13:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a8ae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7a84160a60 con 0x7f7a84142e20 2024-04-17T07:13:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a8ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a84160d40 con 0x7f7a840cf160 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a83e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a84142e20 0x7f7a840cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a88e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7a84143200 0x7f7a84071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49394/0 (socket says 172.21.15.67:49394) 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a88e00640 1 -- 172.21.15.67:0/1684524957 learned_addr learned my addr 172.21.15.67:0/1684524957 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a7b400640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a840cf160 0x7f7a840cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a83e00640 1 -- 172.21.15.67:0/1684524957 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a840cf160 msgr2=0x7f7a840cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a83e00640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a840cf160 0x7f7a840cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a83e00640 1 -- 172.21.15.67:0/1684524957 >> v1:172.21.15.67:6789/0 conn(0x7f7a84143200 legacy=0x7f7a84071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.519+0000 7f7a83e00640 1 -- 172.21.15.67:0/1684524957 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a84161020 con 0x7f7a84142e20 2024-04-17T07:13:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.520+0000 7f7a83e00640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a84142e20 0x7f7a840cec20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009d90 tx=0x7f7a7402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c08b07f6cd69e8a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.521+0000 7f7a83400640 1 -- 172.21.15.67:0/1684524957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a74039070 con 0x7f7a84142e20 2024-04-17T07:13:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.521+0000 7f7a83400640 1 -- 172.21.15.67:0/1684524957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7402cbd0 con 0x7f7a84142e20 2024-04-17T07:13:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.521+0000 7f7a83400640 1 -- 172.21.15.67:0/1684524957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a74034440 con 0x7f7a84142e20 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.521+0000 7f7a8ae00640 1 -- 172.21.15.67:0/1684524957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a84142e20 msgr2=0x7f7a840cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.521+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a84142e20 0x7f7a840cec20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009d90 tx=0x7f7a7402be20 comp rx=0 tx=0).stop 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 -- 172.21.15.67:0/1684524957 shutdown_connections 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a840cf160 0x7f7a840cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/1684524957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a84142e20 0x7f7a840cec20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 -- 172.21.15.67:0/1684524957 >> 172.21.15.67:0/1684524957 conn(0x7f7a8406b8c0 msgr2=0x7f7a8406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 -- 172.21.15.67:0/1684524957 shutdown_connections 2024-04-17T07:13:54.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.522+0000 7f7a8ae00640 1 -- 172.21.15.67:0/1684524957 wait complete. 2024-04-17T07:13:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.524+0000 7f7a8ae00640 1 Processor -- start 2024-04-17T07:13:54.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.524+0000 7f7a8ae00640 1 -- start start 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 0x7f7a8415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 0x7f7a8415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a840b18f0 con 0x7f7a840cf160 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a840ce730 con 0x7f7a84143200 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a8ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a840b1a70 con 0x7f7a84142e20 2024-04-17T07:13:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56802/0 (socket says 172.21.15.67:56802) 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 -- 172.21.15.67:0/2842208087 learned_addr learned my addr 172.21.15.67:0/2842208087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a7b400640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 0x7f7a8415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a88e00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 0x7f7a8415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 -- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 msgr2=0x7f7a8415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 0x7f7a8415dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 -- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 msgr2=0x7f7a8415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 0x7f7a8415e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a83e00640 1 -- 172.21.15.67:0/2842208087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a840b48b0 con 0x7f7a840cf160 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a7b400640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 0x7f7a8415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.525+0000 7f7a88e00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 0x7f7a8415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a83e00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a74037040 tx=0x7f7a74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a74039050 con 0x7f7a840cf160 2024-04-17T07:13:54.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a74045070 con 0x7f7a840cf160 2024-04-17T07:13:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7403e830 con 0x7f7a840cf160 2024-04-17T07:13:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a840b4b90 con 0x7f7a840cf160 2024-04-17T07:13:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.526+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a840b5150 con 0x7f7a840cf160 2024-04-17T07:13:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.527+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a84068b90 con 0x7f7a840cf160 2024-04-17T07:13:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.528+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f7a7403e9d0 con 0x7f7a840cf160 2024-04-17T07:13:54.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.529+0000 7f7a80c00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 0x7f7a6006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.529+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f7a740ae580 con 0x7f7a840cf160 2024-04-17T07:13:54.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.530+0000 7f7a7b400640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 0x7f7a6006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.531+0000 7f7a7b400640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 0x7f7a6006ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7a8413e630 tx=0x7f7a6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.534+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7a7407cd40 con 0x7f7a840cf160 2024-04-17T07:13:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.745+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7a840b3000 con 0x7f7a840cf160 2024-04-17T07:13:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.746+0000 7f7a80c00640 1 -- 172.21.15.67:0/2842208087 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f7a7407c640 con 0x7f7a840cf160 2024-04-17T07:13:54.746 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:13:54.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 msgr2=0x7f7a6006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 0x7f7a6006ed20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7a8413e630 tx=0x7f7a6c008040 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 msgr2=0x7f7a841f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a74037040 tx=0x7f7a74004a40 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 shutdown_connections 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a84143200 0x7f7a8415e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a6006c860 0x7f7a6006ed20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a84142e20 0x7f7a8415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 --2- 172.21.15.67:0/2842208087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a840cf160 0x7f7a841f7800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 >> 172.21.15.67:0/2842208087 conn(0x7f7a8406b8c0 msgr2=0x7f7a8415c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 shutdown_connections 2024-04-17T07:13:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.749+0000 7f7a8ae00640 1 -- 172.21.15.67:0/2842208087 wait complete. 2024-04-17T07:13:54.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:13:54.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.987+0000 7f003d200640 1 Processor -- start 2024-04-17T07:13:54.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.987+0000 7f003d200640 1 -- start start 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f003d200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f003806b8c0 0x7f00380cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f003d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0038160280 0x7f003806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f003d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00380cd4a0 con 0x7f0038141ed0 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f003d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f003815ce30 con 0x7f0038160280 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f003d200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00381607f0 con 0x7f003806b8c0 2024-04-17T07:13:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f0036a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0038160280 0x7f003806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f0037e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0038141ed0 0x7f003815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49406/0 (socket says 172.21.15.67:49406) 2024-04-17T07:13:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f0037400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f003806b8c0 0x7f00380cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.989+0000 7f0037e00640 1 -- 172.21.15.67:0/4024860280 learned_addr learned my addr 172.21.15.67:0/4024860280 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036000640 1 -- 172.21.15.67:0/4024860280 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493417697 0 0) 0x7f00380cd4a0 con 0x7f0038141ed0 2024-04-17T07:13:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036000640 1 -- 172.21.15.67:0/4024860280 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f001c003520 con 0x7f0038141ed0 2024-04-17T07:13:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036a00640 1 -- 172.21.15.67:0/4024860280 >> v2:172.21.15.187:3300/0 conn(0x7f003806b8c0 msgr2=0x7f00380cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036a00640 1 --2- 172.21.15.67:0/4024860280 >> v2:172.21.15.187:3300/0 conn(0x7f003806b8c0 0x7f00380cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036a00640 1 -- 172.21.15.67:0/4024860280 >> v1:172.21.15.67:6789/0 conn(0x7f0038141ed0 legacy=0x7f003815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.990+0000 7f0036a00640 1 -- 172.21.15.67:0/4024860280 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0038160ad0 con 0x7f0038160280 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.991+0000 7f0036a00640 1 --2- 172.21.15.67:0/4024860280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0038160280 0x7f003806bca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00380b3210 tx=0x7f002402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0806915bae722f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.991+0000 7f0036000640 1 -- 172.21.15.67:0/4024860280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024035020 con 0x7f0038160280 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.991+0000 7f0036000640 1 -- 172.21.15.67:0/4024860280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0024031830 con 0x7f0038160280 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.991+0000 7f0036000640 1 -- 172.21.15.67:0/4024860280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024031c00 con 0x7f0038160280 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 -- 172.21.15.67:0/4024860280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0038160280 msgr2=0x7f003806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 --2- 172.21.15.67:0/4024860280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0038160280 0x7f003806bca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00380b3210 tx=0x7f002402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 -- 172.21.15.67:0/4024860280 shutdown_connections 2024-04-17T07:13:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 --2- 172.21.15.67:0/4024860280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0038160280 0x7f003806bca0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 --2- 172.21.15.67:0/4024860280 >> v2:172.21.15.187:3300/0 conn(0x7f003806b8c0 0x7f00380cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 -- 172.21.15.67:0/4024860280 >> 172.21.15.67:0/4024860280 conn(0x7f0038067d10 msgr2=0x7f00380b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:54.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 -- 172.21.15.67:0/4024860280 shutdown_connections 2024-04-17T07:13:54.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.992+0000 7f003d200640 1 -- 172.21.15.67:0/4024860280 wait complete. 2024-04-17T07:13:54.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.994+0000 7f003d200640 1 Processor -- start 2024-04-17T07:13:54.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.994+0000 7f003d200640 1 -- start start 2024-04-17T07:13:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0038141ed0 0x7f00381f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 0x7f00380b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f003806c1e0 con 0x7f003806b8c0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00380cddf0 con 0x7f0038160280 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f003d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0038161090 con 0x7f0038141ed0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f0037400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f0037400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56824/0 (socket says 172.21.15.67:56824) 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f0037400640 1 -- 172.21.15.67:0/3419068381 learned_addr learned my addr 172.21.15.67:0/3419068381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.995+0000 7f0036a00640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0038141ed0 0x7f00381f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037e00640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 0x7f00380b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 -- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0038141ed0 msgr2=0x7f00381f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0038141ed0 0x7f00381f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 -- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 msgr2=0x7f00380b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 0x7f00380b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 -- 172.21.15.67:0/3419068381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003812dca0 con 0x7f003806b8c0 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037e00640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 0x7f00380b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f0037400640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f002800e9b0 tx=0x7f002800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028017070 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002800c9a0 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800cd90 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003812df10 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.996+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003812e4d0 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.998+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0038068a10 con 0x7f003806b8c0 2024-04-17T07:13:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.999+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f002801a070 con 0x7f003806b8c0 2024-04-17T07:13:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:54.999+0000 7f002f400640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 0x7f001806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.000+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f002808b710 con 0x7f003806b8c0 2024-04-17T07:13:55.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.000+0000 7f0036a00640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 0x7f001806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.001+0000 7f0036a00640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 0x7f001806ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0024036000 tx=0x7f002403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.004+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0028059880 con 0x7f003806b8c0 2024-04-17T07:13:55.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.208+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0038141b10 con 0x7f003806b8c0 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.209+0000 7f002f400640 1 -- 172.21.15.67:0/3419068381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f002805db80 con 0x7f003806b8c0 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:13:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:13:55.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:13:55.210 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.211+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 msgr2=0x7f001806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.211+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 0x7f001806ed20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0024036000 tx=0x7f002403c040 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.211+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 msgr2=0x7f00381f5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.211+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f002800e9b0 tx=0x7f002800ee70 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 shutdown_connections 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0038160280 0x7f00380b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f001806c860 0x7f001806ed20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0038141ed0 0x7f00381f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 --2- 172.21.15.67:0/3419068381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003806b8c0 0x7f00381f5630 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 >> 172.21.15.67:0/3419068381 conn(0x7f0038067d10 msgr2=0x7f00380cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 shutdown_connections 2024-04-17T07:13:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.212+0000 7f003d200640 1 -- 172.21.15.67:0/3419068381 wait complete. 2024-04-17T07:13:55.230 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:13:55.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:13:55.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.456+0000 7fc870200640 1 Processor -- start 2024-04-17T07:13:55.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.457+0000 7fc870200640 1 -- start start 2024-04-17T07:13:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.458+0000 7fc870200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc8680b6bd0 0x7fc8681632f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.458+0000 7fc870200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8680d0180 con 0x7fc86815ea10 2024-04-17T07:13:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.458+0000 7fc870200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8680d2cb0 con 0x7fc86815edf0 2024-04-17T07:13:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.458+0000 7fc870200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc8680d35e0 con 0x7fc8680b6bd0 2024-04-17T07:13:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.459+0000 7fc86d800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc86815ea10 0x7fc8680cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49410/0 (socket says 172.21.15.67:49410) 2024-04-17T07:13:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.459+0000 7fc86d800640 1 -- 172.21.15.67:0/3993505340 learned_addr learned my addr 172.21.15.67:0/3993505340 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.459+0000 7fc86e200640 1 --2- 172.21.15.67:0/3993505340 >> v2:172.21.15.187:3300/0 conn(0x7fc8680b6bd0 0x7fc8681632f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.459+0000 7fc867e00640 1 -- 172.21.15.67:0/3993505340 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393975334 0 0) 0x7fc8680d2cb0 con 0x7fc86815edf0 2024-04-17T07:13:55.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.459+0000 7fc867e00640 1 -- 172.21.15.67:0/3993505340 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc850003520 con 0x7fc86815edf0 2024-04-17T07:13:55.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.460+0000 7fc86e200640 1 -- 172.21.15.67:0/3993505340 >> v1:172.21.15.187:6789/0 conn(0x7fc86815edf0 legacy=0x7fc8680b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.460+0000 7fc86e200640 1 -- 172.21.15.67:0/3993505340 >> v1:172.21.15.67:6789/0 conn(0x7fc86815ea10 legacy=0x7fc8680cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.460+0000 7fc86e200640 1 -- 172.21.15.67:0/3993505340 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680d2f90 con 0x7fc8680b6bd0 2024-04-17T07:13:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.461+0000 7fc86e200640 1 --2- 172.21.15.67:0/3993505340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8680b6bd0 0x7fc8681632f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc860009d90 tx=0x7fc86002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d50deb0db575f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.461+0000 7fc867e00640 1 -- 172.21.15.67:0/3993505340 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860039070 con 0x7fc8680b6bd0 2024-04-17T07:13:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc867e00640 1 -- 172.21.15.67:0/3993505340 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86002cd00 con 0x7fc8680b6bd0 2024-04-17T07:13:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc867e00640 1 -- 172.21.15.67:0/3993505340 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860034640 con 0x7fc8680b6bd0 2024-04-17T07:13:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 -- 172.21.15.67:0/3993505340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8680b6bd0 msgr2=0x7fc8681632f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 --2- 172.21.15.67:0/3993505340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8680b6bd0 0x7fc8681632f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc860009d90 tx=0x7fc86002bf50 comp rx=0 tx=0).stop 2024-04-17T07:13:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 -- 172.21.15.67:0/3993505340 shutdown_connections 2024-04-17T07:13:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 --2- 172.21.15.67:0/3993505340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8680b6bd0 0x7fc8681632f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 -- 172.21.15.67:0/3993505340 >> 172.21.15.67:0/3993505340 conn(0x7fc8680d16a0 msgr2=0x7fc8680d1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 -- 172.21.15.67:0/3993505340 shutdown_connections 2024-04-17T07:13:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.462+0000 7fc870200640 1 -- 172.21.15.67:0/3993505340 wait complete. 2024-04-17T07:13:55.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.464+0000 7fc870200640 1 Processor -- start 2024-04-17T07:13:55.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.464+0000 7fc870200640 1 -- start start 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 0x7fc868156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc86815ea10 0x7fc8680caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 0x7fc8680ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680b7dd0 con 0x7fc8680b6bd0 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680b7c50 con 0x7fc86815edf0 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.465+0000 7fc870200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680b7f50 con 0x7fc86815ea10 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.466+0000 7fc86d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 0x7fc868156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.466+0000 7fc86d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 0x7fc868156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56840/0 (socket says 172.21.15.67:56840) 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.466+0000 7fc86d800640 1 -- 172.21.15.67:0/2113647578 learned_addr learned my addr 172.21.15.67:0/2113647578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.466+0000 7fc86e200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 0x7fc8680ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.466+0000 7fc86ce00640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc86815ea10 0x7fc8680caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 -- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc86815ea10 msgr2=0x7fc8680caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc86815ea10 0x7fc8680caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 -- 172.21.15.67:0/2113647578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 msgr2=0x7fc868156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 0x7fc868156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 -- 172.21.15.67:0/2113647578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8681554f0 con 0x7fc86815edf0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.467+0000 7fc86e200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 0x7fc8680ceb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc860033000 tx=0x7fc860004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.468+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860042030 con 0x7fc86815edf0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.468+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86003a040 con 0x7fc86815edf0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.468+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860039070 con 0x7fc86815edf0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.468+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8681557d0 con 0x7fc86815edf0 2024-04-17T07:13:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.468+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc868155d90 con 0x7fc86815edf0 2024-04-17T07:13:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.469+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86806e170 con 0x7fc86815edf0 2024-04-17T07:13:55.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.471+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7fc860045070 con 0x7fc86815edf0 2024-04-17T07:13:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.472+0000 7fc865600640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 0x7fc84806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.472+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fc8600adc40 con 0x7fc86815edf0 2024-04-17T07:13:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.473+0000 7fc86d800640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 0x7fc84806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:55.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.474+0000 7fc86d800640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 0x7fc84806ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc85c005880 tx=0x7fc85c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.488+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc86004c070 con 0x7fc86815edf0 2024-04-17T07:13:55.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:55.699+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc868068d60 con 0x7fc86815edf0 2024-04-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.181+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc860078810 con 0x7fc86815edf0 2024-04-17T07:13:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.182+0000 7fc865600640 1 -- 172.21.15.67:0/2113647578 <== mon.1 v2:172.21.15.187:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7fc8600af400 con 0x7fc86815edf0 2024-04-17T07:13:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.189+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 msgr2=0x7fc84806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.189+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 0x7fc84806ed20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc85c005880 tx=0x7fc85c009040 comp rx=0 tx=0).stop 2024-04-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.189+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 msgr2=0x7fc8680ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.189+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 0x7fc8680ceb60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc860033000 tx=0x7fc860004a40 comp rx=0 tx=0).stop 2024-04-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 shutdown_connections 2024-04-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc86815edf0 0x7fc8680ceb60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc84806c860 0x7fc84806ed20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc86815ea10 0x7fc8680caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 --2- 172.21.15.67:0/2113647578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8680b6bd0 0x7fc868156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 >> 172.21.15.67:0/2113647578 conn(0x7fc8680d16a0 msgr2=0x7fc8681590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 shutdown_connections 2024-04-17T07:13:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.190+0000 7fc870200640 1 -- 172.21.15.67:0/2113647578 wait complete. 2024-04-17T07:13:56.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:13:56.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.423+0000 7fd54b200640 1 Processor -- start 2024-04-17T07:13:56.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.423+0000 7fd54b200640 1 -- start start 2024-04-17T07:13:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd54b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 0x7fd54c071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd54b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd54c0cd5d0 0x7fd54c0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd54b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd54c15cbe0 con 0x7fd54c0b20b0 2024-04-17T07:13:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd54b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd54c15cea0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd54b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd54c15d180 con 0x7fd54c13f6d0 2024-04-17T07:13:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.425+0000 7fd549e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 0x7fd54c071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 0x7fd54c071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:58796/0 (socket says 172.21.15.67:58796) 2024-04-17T07:13:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549e00640 1 -- 172.21.15.67:0/2552792166 learned_addr learned my addr 172.21.15.67:0/2552792166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd54a800640 1 --1- 172.21.15.67:0/2552792166 >> v1:172.21.15.187:6789/0 conn(0x7fd54c0b20b0 0x7fd54c142020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:42466/0 (socket says 172.21.15.67:42466) 2024-04-17T07:13:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549400640 1 --2- 172.21.15.67:0/2552792166 >> v2:172.21.15.187:3300/0 conn(0x7fd54c0cd5d0 0x7fd54c0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549400640 1 -- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 msgr2=0x7fd54c071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549400640 1 --2- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 0x7fd54c071d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549400640 1 -- 172.21.15.67:0/2552792166 >> v1:172.21.15.187:6789/0 conn(0x7fd54c0b20b0 legacy=0x7fd54c142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.426+0000 7fd549400640 1 -- 172.21.15.67:0/2552792166 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c15d460 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.428+0000 7fd549400640 1 --2- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c0cda30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5340097a0 tx=0x7fd53402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8f1f93aa89a2be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.428+0000 7fd548a00640 1 -- 172.21.15.67:0/2552792166 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534031780 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.428+0000 7fd548a00640 1 -- 172.21.15.67:0/2552792166 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd534037070 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.428+0000 7fd548a00640 1 -- 172.21.15.67:0/2552792166 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534031b40 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 -- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c0cd5d0 msgr2=0x7fd54c0cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 --2- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c0cda30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5340097a0 tx=0x7fd53402b820 comp rx=0 tx=0).stop 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 -- 172.21.15.67:0/2552792166 shutdown_connections 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 --2- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c0cda30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 --2- 172.21.15.67:0/2552792166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c13f6d0 0x7fd54c071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 -- 172.21.15.67:0/2552792166 >> 172.21.15.67:0/2552792166 conn(0x7fd54c067d10 msgr2=0x7fd54c0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 -- 172.21.15.67:0/2552792166 shutdown_connections 2024-04-17T07:13:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.429+0000 7fd54b200640 1 -- 172.21.15.67:0/2552792166 wait complete. 2024-04-17T07:13:56.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.431+0000 7fd54b200640 1 Processor -- start 2024-04-17T07:13:56.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.432+0000 7fd54b200640 1 -- start start 2024-04-17T07:13:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c0b20b0 0x7fd54c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c13f6d0 0x7fd54c1f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c15da20 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c0d1120 con 0x7fd54c13f6d0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd54c0b2540 con 0x7fd54c0b20b0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd549400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd549400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56846/0 (socket says 172.21.15.67:56846) 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd549400640 1 -- 172.21.15.67:0/3336663061 learned_addr learned my addr 172.21.15.67:0/3336663061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd549e00640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c0b20b0 0x7fd54c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.433+0000 7fd54a800640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c13f6d0 0x7fd54c1f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 -- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c0b20b0 msgr2=0x7fd54c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c0b20b0 0x7fd54c1568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 -- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c13f6d0 msgr2=0x7fd54c1f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c13f6d0 0x7fd54c1f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 -- 172.21.15.67:0/3336663061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c1f28d0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd549400640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd534034040 tx=0x7fd5340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534032610 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5340327b0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54c1f2ba0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.434+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534032ba0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.435+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd54c1f3440 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.436+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd54c068a10 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.437+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd53404e430 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.438+0000 7fd542000640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 0x7fd52806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.438+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7fd5340ad8e0 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.438+0000 7fd549e00640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 0x7fd52806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:56.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.439+0000 7fd549e00640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 0x7fd52806edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd544009ce0 tx=0x7fd54400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.443+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd534040070 con 0x7fd54c0cd5d0 2024-04-17T07:13:56.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:56.650+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd54c146100 con 0x7fd54c0cd5d0 2024-04-17T07:13:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.159+0000 7fd542000640 1 -- 172.21.15.67:0/3336663061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) ==== 170+0+0 (secure 0 0 0) 0x7fd53402f050 con 0x7fd54c0cd5d0 2024-04-17T07:13:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 msgr2=0x7fd52806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 0x7fd52806edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd544009ce0 tx=0x7fd54400b040 comp rx=0 tx=0).stop 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 msgr2=0x7fd54c13cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd534034040 tx=0x7fd5340046c0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 shutdown_connections 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd54c13f6d0 0x7fd54c1f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd54c0cd5d0 0x7fd54c13cde0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd52806c8f0 0x7fd52806edb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 --2- 172.21.15.67:0/3336663061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd54c0b20b0 0x7fd54c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 >> 172.21.15.67:0/3336663061 conn(0x7fd54c067d10 msgr2=0x7fd54c06ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:57.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 shutdown_connections 2024-04-17T07:13:57.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.163+0000 7fd54b200640 1 -- 172.21.15.67:0/3336663061 wait complete. 2024-04-17T07:13:57.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:13:57.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.385+0000 7f4f38e00640 1 Processor -- start 2024-04-17T07:13:57.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.385+0000 7f4f38e00640 1 -- start start 2024-04-17T07:13:57.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.387+0000 7f4f38e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f3409aa60 0x7f4f3409ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:57.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.387+0000 7f4f38e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f341613c0 con 0x7f4f340d0280 2024-04-17T07:13:57.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.387+0000 7f4f38e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3409b380 con 0x7f4f340cead0 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.387+0000 7f4f38e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3415d350 con 0x7f4f3409aa60 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f3409aa60 0x7f4f3409ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4f340d0280 0x7f4f341605a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49420/0 (socket says 172.21.15.67:49420) 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f3409aa60 0x7f4f3409ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56848/0 (socket says 172.21.15.67:56848) 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31400640 1 -- 172.21.15.67:0/909237147 learned_addr learned my addr 172.21.15.67:0/909237147 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31e00640 1 -- 172.21.15.67:0/909237147 >> v1:172.21.15.187:6789/0 conn(0x7f4f340cead0 legacy=0x7f4f34160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31e00640 1 -- 172.21.15.67:0/909237147 >> v1:172.21.15.67:6789/0 conn(0x7f4f340d0280 legacy=0x7f4f341605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.388+0000 7f4f31e00640 1 -- 172.21.15.67:0/909237147 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f340d0e60 con 0x7f4f3409aa60 2024-04-17T07:13:57.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.389+0000 7f4f31e00640 1 --2- 172.21.15.67:0/909237147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f3409aa60 0x7f4f3409ae40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4f240097d0 tx=0x7f4f2402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2f7303e193a1f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:57.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.389+0000 7f4f30a00640 1 -- 172.21.15.67:0/909237147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24034070 con 0x7f4f3409aa60 2024-04-17T07:13:57.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.389+0000 7f4f30a00640 1 -- 172.21.15.67:0/909237147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24030800 con 0x7f4f3409aa60 2024-04-17T07:13:57.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.389+0000 7f4f30a00640 1 -- 172.21.15.67:0/909237147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24030bd0 con 0x7f4f3409aa60 2024-04-17T07:13:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 -- 172.21.15.67:0/909237147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f3409aa60 msgr2=0x7f4f3409ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 --2- 172.21.15.67:0/909237147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f3409aa60 0x7f4f3409ae40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4f240097d0 tx=0x7f4f2402b940 comp rx=0 tx=0).stop 2024-04-17T07:13:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 -- 172.21.15.67:0/909237147 shutdown_connections 2024-04-17T07:13:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 --2- 172.21.15.67:0/909237147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f3409aa60 0x7f4f3409ae40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 -- 172.21.15.67:0/909237147 >> 172.21.15.67:0/909237147 conn(0x7f4f340b41a0 msgr2=0x7f4f340b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:57.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.390+0000 7f4f38e00640 1 -- 172.21.15.67:0/909237147 shutdown_connections 2024-04-17T07:13:57.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.391+0000 7f4f38e00640 1 -- 172.21.15.67:0/909237147 wait complete. 2024-04-17T07:13:57.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.393+0000 7f4f38e00640 1 Processor -- start 2024-04-17T07:13:57.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.393+0000 7f4f38e00640 1 -- start start 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f3409aa60 0x7f4f3414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f340cead0 0x7f4f341f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f340d1420 con 0x7f4f340d0280 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4f340cd630 con 0x7f4f3409aa60 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.394+0000 7f4f38e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f340d1700 con 0x7f4f340cead0 2024-04-17T07:13:57.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56854/0 (socket says 172.21.15.67:56854) 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 -- 172.21.15.67:0/1883334272 learned_addr learned my addr 172.21.15.67:0/1883334272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f31400640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f340cead0 0x7f4f341f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f31e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f3409aa60 0x7f4f3414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 -- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f340cead0 msgr2=0x7f4f341f5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f340cead0 0x7f4f341f5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 -- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f3409aa60 msgr2=0x7f4f3414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f3409aa60 0x7f4f3414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 -- 172.21.15.67:0/1883334272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f341f5a80 con 0x7f4f340d0280 2024-04-17T07:13:57.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.395+0000 7f4f32800640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c730 tx=0x7f4f2c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.396+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c003b70 con 0x7f4f340d0280 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.396+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2c003d10 con 0x7f4f340d0280 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.396+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00aa90 con 0x7f4f340d0280 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.396+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f340b7ea0 con 0x7f4f340d0280 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.396+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f341506c0 con 0x7f4f340d0280 2024-04-17T07:13:57.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.397+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f34068ca0 con 0x7f4f340d0280 2024-04-17T07:13:57.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.398+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f2c00ac30 con 0x7f4f340d0280 2024-04-17T07:13:57.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.399+0000 7f4f2a000640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 0x7f4f1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:57.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.399+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7f4f2c08ba80 con 0x7f4f340d0280 2024-04-17T07:13:57.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.400+0000 7f4f31e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 0x7f4f1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:57.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.401+0000 7f4f31e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 0x7f4f1406edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f24036000 tx=0x7f4f2403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:57.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.405+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4f2c08de20 con 0x7f4f340d0280 2024-04-17T07:13:57.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:57.649+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4f3415c150 con 0x7f4f340d0280 2024-04-17T07:13:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.206+0000 7f4f2a000640 1 -- 172.21.15.67:0/1883334272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f4f341f5a80 con 0x7f4f340d0280 2024-04-17T07:13:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-04-17T07:13:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.219+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 msgr2=0x7f4f1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.219+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 0x7f4f1406edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f24036000 tx=0x7f4f2403b040 comp rx=0 tx=0).stop 2024-04-17T07:13:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 msgr2=0x7f4f340cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00c730 tx=0x7f4f2c00cbf0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 shutdown_connections 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340d0280 0x7f4f340cf050 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4f1406c8f0 0x7f4f1406edb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f340cead0 0x7f4f341f5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 --2- 172.21.15.67:0/1883334272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f3409aa60 0x7f4f3414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 >> 172.21.15.67:0/1883334272 conn(0x7f4f340b41a0 msgr2=0x7f4f340b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.220+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 shutdown_connections 2024-04-17T07:13:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.221+0000 7f4f38e00640 1 -- 172.21.15.67:0/1883334272 wait complete. 2024-04-17T07:13:58.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:58.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.428+0000 7fd3c5c00640 1 Processor -- start 2024-04-17T07:13:58.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.428+0000 7fd3c5c00640 1 -- start start 2024-04-17T07:13:58.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.429+0000 7fd3c5c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3c0141ed0 0x7fd3c0142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.429+0000 7fd3c5c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c00cd4a0 con 0x7fd3c006b8c0 2024-04-17T07:13:58.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.429+0000 7fd3c5c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c01607f0 con 0x7fd3c0160280 2024-04-17T07:13:58.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.429+0000 7fd3c5c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd3c015ce30 con 0x7fd3c0141ed0 2024-04-17T07:13:58.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd3c006b8c0 0x7fd3c00cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49432/0 (socket says 172.21.15.67:49432) 2024-04-17T07:13:58.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bfe00640 1 -- 172.21.15.67:0/2585839952 learned_addr learned my addr 172.21.15.67:0/2585839952 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:58.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704988512 0 0) 0x7fd3c00cd4a0 con 0x7fd3c006b8c0 2024-04-17T07:13:58.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3a4003520 con 0x7fd3c006b8c0 2024-04-17T07:13:58.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3c5200640 1 --2- 172.21.15.67:0/2585839952 >> v2:172.21.15.187:3300/0 conn(0x7fd3c0141ed0 0x7fd3c0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3706738112 0 0) 0x7fd3a4003520 con 0x7fd3c006b8c0 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 >> v2:172.21.15.187:3300/0 conn(0x7fd3c0141ed0 msgr2=0x7fd3c0142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 --2- 172.21.15.67:0/2585839952 >> v2:172.21.15.187:3300/0 conn(0x7fd3c0141ed0 0x7fd3c0142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 >> v1:172.21.15.187:6789/0 conn(0x7fd3c0160280 legacy=0x7fd3c006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c0160ad0 con 0x7fd3c006b8c0 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd3b0001670 con 0x7fd3c006b8c0 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3c5200640 1 --2- 172.21.15.67:0/2585839952 >> v2:172.21.15.187:3300/0 conn(0x7fd3c0141ed0 0x7fd3c0142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.430+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 >> v1:172.21.15.67:6789/0 conn(0x7fd3c006b8c0 legacy=0x7fd3c00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 0x7fd3a4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4004800 0x7fd3a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3a40052e0 0x7fd3a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c015ce30 con 0x7fd3a4003a70 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a400c1b0 con 0x7fd3a4004800 2024-04-17T07:13:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a400c3b0 con 0x7fd3a40052e0 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 0x7fd3a4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bf400640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4004800 0x7fd3a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 -- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3a40052e0 msgr2=0x7fd3a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3a40052e0 0x7fd3a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 -- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4004800 msgr2=0x7fd3a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4004800 0x7fd3a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bfe00640 1 -- 172.21.15.67:0/2585839952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a4003520 con 0x7fd3a4003a70 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3bf400640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4004800 0x7fd3a4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.431+0000 7fd3c5200640 1 -- 172.21.15.67:0/2585839952 reap_dead start 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.432+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 0x7fd3a4003e70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0001eb0 tx=0x7fd3a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb826d6071d7d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.432+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0007c80 con 0x7fd3a4003a70 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.432+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b0030070 con 0x7fd3a4003a70 2024-04-17T07:13:58.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.432+0000 7fd3bea00640 1 -- 172.21.15.67:0/2585839952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0005d70 con 0x7fd3a4003a70 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 -- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 msgr2=0x7fd3a4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 0x7fd3a4003e70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0001eb0 tx=0x7fd3a4005830 comp rx=0 tx=0).stop 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 -- 172.21.15.67:0/2585839952 shutdown_connections 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/2585839952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a4003a70 0x7fd3a4003e70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 -- 172.21.15.67:0/2585839952 >> 172.21.15.67:0/2585839952 conn(0x7fd3c0067d10 msgr2=0x7fd3c00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 -- 172.21.15.67:0/2585839952 shutdown_connections 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.433+0000 7fd3c5c00640 1 -- 172.21.15.67:0/2585839952 wait complete. 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.435+0000 7fd3c5c00640 1 Processor -- start 2024-04-17T07:13:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.435+0000 7fd3c5c00640 1 -- start start 2024-04-17T07:13:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4003a70 0x7fd3c006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3c01456b0 0x7fd3c013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c013f9b0 con 0x7fd3c006f590 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0161090 con 0x7fd3a4003a70 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c013fb30 con 0x7fd3c01456b0 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.436+0000 7fd3c5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56876/0 (socket says 172.21.15.67:56876) 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 -- 172.21.15.67:0/3836505316 learned_addr learned my addr 172.21.15.67:0/3836505316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4003a70 0x7fd3c006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 -- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3c01456b0 msgr2=0x7fd3c013c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3bf400640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3c01456b0 0x7fd3c013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3c01456b0 0x7fd3c013c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 -- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4003a70 msgr2=0x7fd3c006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4003a70 0x7fd3c006ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 -- 172.21.15.67:0/3836505316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c013c5b0 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.437+0000 7fd3c5200640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0071d80 tx=0x7fd3b800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.438+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b801a070 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.438+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b8003990 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.438+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8003d80 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.438+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c006e4f0 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.438+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c006eab0 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.439+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c0068a10 con 0x7fd3c006f590 2024-04-17T07:13:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.440+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd3b801d070 con 0x7fd3c006f590 2024-04-17T07:13:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.440+0000 7fd3b7e00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 0x7fd3a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.441+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd3b808b920 con 0x7fd3c006f590 2024-04-17T07:13:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.441+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 0x7fd3a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.442+0000 7fd3bfe00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 0x7fd3a006ede0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0034000 tx=0x7fd3b0009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.445+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3b8010070 con 0x7fd3c006f590 2024-04-17T07:13:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.643+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3c00af860 con 0x7fd3c006f590 2024-04-17T07:13:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.644+0000 7fd3b7e00640 1 -- 172.21.15.67:0/3836505316 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16570 (secure 0 0 0) 0x7fd3b8059de0 con 0x7fd3c006f590 2024-04-17T07:13:58.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:58.645 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:57.651467+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 msgr2=0x7fd3a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 0x7fd3a006ede0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0034000 tx=0x7fd3b0009030 comp rx=0 tx=0).stop 2024-04-17T07:13:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 msgr2=0x7fd3c015b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0071d80 tx=0x7fd3b800d4a0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 shutdown_connections 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3c006f590 0x7fd3c015b680 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd3a006c920 0x7fd3a006ede0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3c01456b0 0x7fd3c013c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 --2- 172.21.15.67:0/3836505316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a4003a70 0x7fd3c006ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 >> 172.21.15.67:0/3836505316 conn(0x7fd3c0067d10 msgr2=0x7fd3c0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 shutdown_connections 2024-04-17T07:13:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.647+0000 7fd3c5c00640 1 -- 172.21.15.67:0/3836505316 wait complete. 2024-04-17T07:13:58.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:13:58.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.882+0000 7f3137200640 1 Processor -- start 2024-04-17T07:13:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.883+0000 7f3137200640 1 -- start start 2024-04-17T07:13:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.884+0000 7f3137200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 0x7f31380cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.884+0000 7f3137200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3138160280 0x7f313806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.884+0000 7f3137200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31380cd4a0 con 0x7f3138141ed0 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.884+0000 7f3137200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f313815ce30 con 0x7f3138160280 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.884+0000 7f3137200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31381607f0 con 0x7f313806b8c0 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3138160280 0x7f313806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3138160280 0x7f313806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56882/0 (socket says 172.21.15.67:56882) 2024-04-17T07:13:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 -- 172.21.15.67:0/3478979285 learned_addr learned my addr 172.21.15.67:0/3478979285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f3135e00640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 0x7f31380cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 -- 172.21.15.67:0/3478979285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 msgr2=0x7f31380cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 0x7f31380cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 -- 172.21.15.67:0/3478979285 >> v1:172.21.15.187:6789/0 conn(0x7f3138141ed0 legacy=0x7f313815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f312d400640 1 -- 172.21.15.67:0/3478979285 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3138160ad0 con 0x7f3138160280 2024-04-17T07:13:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.885+0000 7f3135e00640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 0x7f31380cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:58.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.886+0000 7f312d400640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138160280 0x7f313806bca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f312002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b5e3dfe230d0f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3135400640 1 -- 172.21.15.67:0/3478979285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120034070 con 0x7f3138160280 2024-04-17T07:13:58.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3135400640 1 -- 172.21.15.67:0/3478979285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3120030840 con 0x7f3138160280 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3135400640 1 -- 172.21.15.67:0/3478979285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120030c10 con 0x7f3138160280 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 -- 172.21.15.67:0/3478979285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138160280 msgr2=0x7f313806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138160280 0x7f313806bca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f312002b980 comp rx=0 tx=0).stop 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 -- 172.21.15.67:0/3478979285 shutdown_connections 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138160280 0x7f313806bca0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 --2- 172.21.15.67:0/3478979285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f313806b8c0 0x7f31380cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.887+0000 7f3137200640 1 -- 172.21.15.67:0/3478979285 >> 172.21.15.67:0/3478979285 conn(0x7f3138067d10 msgr2=0x7f31380b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.888+0000 7f3137200640 1 -- 172.21.15.67:0/3478979285 shutdown_connections 2024-04-17T07:13:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.888+0000 7f3137200640 1 -- 172.21.15.67:0/3478979285 wait complete. 2024-04-17T07:13:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.889+0000 7f3137200640 1 Processor -- start 2024-04-17T07:13:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.889+0000 7f3137200640 1 -- start start 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313806b8c0 0x7f313813ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3138160280 0x7f3138143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f313806c1e0 con 0x7f3138141ed0 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f31380cddf0 con 0x7f313806b8c0 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.890+0000 7f3137200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3138161090 con 0x7f3138160280 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56890/0 (socket says 172.21.15.67:56890) 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 -- 172.21.15.67:0/3189029241 learned_addr learned my addr 172.21.15.67:0/3189029241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f3135e00640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313806b8c0 0x7f313813ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f3136800640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3138160280 0x7f3138143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 -- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3138160280 msgr2=0x7f3138143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3138160280 0x7f3138143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 -- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313806b8c0 msgr2=0x7f313813ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313806b8c0 0x7f313813ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.891+0000 7f312d400640 1 -- 172.21.15.67:0/3189029241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313815dcc0 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f312d400640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f31200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120034040 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3120039d90 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120043610 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313815dfa0 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.892+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f313815e560 con 0x7f3138141ed0 2024-04-17T07:13:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.893+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3138068a10 con 0x7f3138141ed0 2024-04-17T07:13:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.894+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f312004a060 con 0x7f3138141ed0 2024-04-17T07:13:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.895+0000 7f312ea00640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 0x7f311406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.895+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f31200addc0 con 0x7f3138141ed0 2024-04-17T07:13:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.896+0000 7f3135e00640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 0x7f311406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:58.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.897+0000 7f3135e00640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 0x7f311406edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f312800a990 tx=0x7f3128008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:58.900+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f312003e070 con 0x7f3138141ed0 2024-04-17T07:13:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.103+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f313813f9b0 con 0x7f3138141ed0 2024-04-17T07:13:59.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.107+0000 7f312ea00640 1 -- 172.21.15.67:0/3189029241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16570 (secure 0 0 0) 0x7f312002d330 con 0x7f3138141ed0 2024-04-17T07:13:59.107 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:13:59.107 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:57.651467+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.109+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 msgr2=0x7f311406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.109+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 0x7f311406edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f312800a990 tx=0x7f3128008040 comp rx=0 tx=0).stop 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 msgr2=0x7f3138143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3120009870 tx=0x7f31200046c0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 shutdown_connections 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f311406c8f0 0x7f311406edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3138160280 0x7f3138143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3138141ed0 0x7f3138143300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 --2- 172.21.15.67:0/3189029241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f313806b8c0 0x7f313813ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 >> 172.21.15.67:0/3189029241 conn(0x7f3138067d10 msgr2=0x7f31380d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 shutdown_connections 2024-04-17T07:13:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.110+0000 7f3137200640 1 -- 172.21.15.67:0/3189029241 wait complete. 2024-04-17T07:13:59.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:13:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.358+0000 7f14c7e00640 1 Processor -- start 2024-04-17T07:13:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.358+0000 7f14c7e00640 1 -- start start 2024-04-17T07:13:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.359+0000 7f14c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14c8145f80 0x7f14c80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.359+0000 7f14c7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 0x7f14c806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.359+0000 7f14c7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c8146360 con 0x7f14c80cd750 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.359+0000 7f14c7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f14c80727a0 con 0x7f14c8145f80 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.359+0000 7f14c7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f14c8071e70 con 0x7f14c806c270 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14bfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f14c80cd750 0x7f14c806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49448/0 (socket says 172.21.15.67:49448) 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14c8145f80 0x7f14c80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14bfe00640 1 -- 172.21.15.67:0/1453159599 learned_addr learned my addr 172.21.15.67:0/1453159599 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6000640 1 -- 172.21.15.67:0/1453159599 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011164560 0 0) 0x7f14c8146360 con 0x7f14c80cd750 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6000640 1 -- 172.21.15.67:0/1453159599 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14ac003520 con 0x7f14c80cd750 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c7400640 1 --2- 172.21.15.67:0/1453159599 >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 0x7f14c806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6a00640 1 -- 172.21.15.67:0/1453159599 >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 msgr2=0x7f14c806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1453159599 >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 0x7f14c806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6a00640 1 -- 172.21.15.67:0/1453159599 >> v1:172.21.15.67:6789/0 conn(0x7f14c80cd750 legacy=0x7f14c806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c6a00640 1 -- 172.21.15.67:0/1453159599 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c8072150 con 0x7f14c8145f80 2024-04-17T07:13:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.360+0000 7f14c7400640 1 --2- 172.21.15.67:0/1453159599 >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 0x7f14c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:13:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1453159599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c8145f80 0x7f14c80cd210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14c80ceed0 tx=0x7f14b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bd68bd1f5cbc73 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c6000640 1 -- 172.21.15.67:0/1453159599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b80315a0 con 0x7f14c8145f80 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c6000640 1 -- 172.21.15.67:0/1453159599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b8037070 con 0x7f14c8145f80 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c6000640 1 -- 172.21.15.67:0/1453159599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8031960 con 0x7f14c8145f80 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 -- 172.21.15.67:0/1453159599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c8145f80 msgr2=0x7f14c80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1453159599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c8145f80 0x7f14c80cd210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14c80ceed0 tx=0x7f14b802b840 comp rx=0 tx=0).stop 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 -- 172.21.15.67:0/1453159599 shutdown_connections 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1453159599 >> v2:172.21.15.187:3300/0 conn(0x7f14c806c270 0x7f14c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1453159599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c8145f80 0x7f14c80cd210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.362+0000 7f14c7e00640 1 -- 172.21.15.67:0/1453159599 >> 172.21.15.67:0/1453159599 conn(0x7f14c80b2190 msgr2=0x7f14c80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:13:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.363+0000 7f14c7e00640 1 -- 172.21.15.67:0/1453159599 shutdown_connections 2024-04-17T07:13:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.363+0000 7f14c7e00640 1 -- 172.21.15.67:0/1453159599 wait complete. 2024-04-17T07:13:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.364+0000 7f14c7e00640 1 Processor -- start 2024-04-17T07:13:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.364+0000 7f14c7e00640 1 -- start start 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14c80cd750 0x7f14c80c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14c8145f80 0x7f14c80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8072db0 con 0x7f14c806c270 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14c8072c30 con 0x7f14c80cd750 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.365+0000 7f14c7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14c806e700 con 0x7f14c8145f80 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56912/0 (socket says 172.21.15.67:56912) 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 -- 172.21.15.67:0/1008390358 learned_addr learned my addr 172.21.15.67:0/1008390358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14bfe00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14c80cd750 0x7f14c80c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 -- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14c8145f80 msgr2=0x7f14c80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14c8145f80 0x7f14c80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 -- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14c80cd750 msgr2=0x7f14c80c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14c80cd750 0x7f14c80c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 -- 172.21.15.67:0/1008390358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c80c54a0 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.366+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14b802be50 tx=0x7f14b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.367+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8032630 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.367+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b80327d0 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.367+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c80c5780 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.367+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8032bc0 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.367+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c80c5d40 con 0x7f14c806c270 2024-04-17T07:13:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.368+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c8068a10 con 0x7f14c806c270 2024-04-17T07:13:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.369+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14b804c020 con 0x7f14c806c270 2024-04-17T07:13:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.370+0000 7f14bf400640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 0x7f14a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:13:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.370+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f14b802f020 con 0x7f14c806c270 2024-04-17T07:13:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.371+0000 7f14bfe00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 0x7f14a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:13:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.372+0000 7f14bfe00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 0x7f14a406edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14c80cd720 tx=0x7f14b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:13:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.375+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f14b8040070 con 0x7f14c806c270 2024-04-17T07:13:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:13:59.624+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f14c806ccd0 con 0x7f14c806c270 2024-04-17T07:14:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.219+0000 7f14bf400640 1 -- 172.21.15.67:0/1008390358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f14b807bb10 con 0x7f14c806c270 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 msgr2=0x7f14a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 0x7f14a406edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f14c80cd720 tx=0x7f14b0009040 comp rx=0 tx=0).stop 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 msgr2=0x7f14c813acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14b802be50 tx=0x7f14b80046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 shutdown_connections 2024-04-17T07:14:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f14a406c8f0 0x7f14a406edb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14c8145f80 0x7f14c80c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14c80cd750 0x7f14c80c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 --2- 172.21.15.67:0/1008390358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c806c270 0x7f14c813acb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 >> 172.21.15.67:0/1008390358 conn(0x7f14c80b2190 msgr2=0x7f14c815cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 shutdown_connections 2024-04-17T07:14:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.223+0000 7f14c7e00640 1 -- 172.21.15.67:0/1008390358 wait complete. 2024-04-17T07:14:00.245 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:14:00.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:14:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.472+0000 7fba87e00640 1 Processor -- start 2024-04-17T07:14:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.473+0000 7fba87e00640 1 -- start start 2024-04-17T07:14:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba87e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba8806c270 0x7fba88167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba87e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba8806f1d0 con 0x7fba880cd920 2024-04-17T07:14:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba87e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba88146360 con 0x7fba88145f80 2024-04-17T07:14:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba87e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fba880727a0 con 0x7fba8806c270 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba87400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba8806c270 0x7fba88167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba7fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba880cd920 0x7fba8806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49454/0 (socket says 172.21.15.67:49454) 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.474+0000 7fba7fe00640 1 -- 172.21.15.67:0/282346179 learned_addr learned my addr 172.21.15.67:0/282346179 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.475+0000 7fba87400640 1 -- 172.21.15.67:0/282346179 >> v1:172.21.15.187:6789/0 conn(0x7fba88145f80 legacy=0x7fba880cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.475+0000 7fba87400640 1 -- 172.21.15.67:0/282346179 >> v1:172.21.15.67:6789/0 conn(0x7fba880cd920 legacy=0x7fba8806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.475+0000 7fba87400640 1 -- 172.21.15.67:0/282346179 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba88071e70 con 0x7fba8806c270 2024-04-17T07:14:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.476+0000 7fba87400640 1 --2- 172.21.15.67:0/282346179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba8806c270 0x7fba88167380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba80009d90 tx=0x7fba8002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdbe80e5d523aad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.476+0000 7fba86000640 1 -- 172.21.15.67:0/282346179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80039070 con 0x7fba8806c270 2024-04-17T07:14:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.476+0000 7fba86000640 1 -- 172.21.15.67:0/282346179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba8002cbd0 con 0x7fba8806c270 2024-04-17T07:14:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.476+0000 7fba86000640 1 -- 172.21.15.67:0/282346179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80034440 con 0x7fba8806c270 2024-04-17T07:14:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 -- 172.21.15.67:0/282346179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba8806c270 msgr2=0x7fba88167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 --2- 172.21.15.67:0/282346179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba8806c270 0x7fba88167380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba80009d90 tx=0x7fba8002be20 comp rx=0 tx=0).stop 2024-04-17T07:14:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 -- 172.21.15.67:0/282346179 shutdown_connections 2024-04-17T07:14:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 --2- 172.21.15.67:0/282346179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba8806c270 0x7fba88167380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 -- 172.21.15.67:0/282346179 >> 172.21.15.67:0/282346179 conn(0x7fba880b2190 msgr2=0x7fba880b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 -- 172.21.15.67:0/282346179 shutdown_connections 2024-04-17T07:14:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.477+0000 7fba87e00640 1 -- 172.21.15.67:0/282346179 wait complete. 2024-04-17T07:14:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.479+0000 7fba87e00640 1 Processor -- start 2024-04-17T07:14:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.479+0000 7fba87e00640 1 -- start start 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 0x7fba8815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba88145f80 0x7fba880af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba88072c30 con 0x7fba880cd920 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba88072430 con 0x7fba8806c270 2024-04-17T07:14:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba87e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba88072db0 con 0x7fba88145f80 2024-04-17T07:14:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56934/0 (socket says 172.21.15.67:56934) 2024-04-17T07:14:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.480+0000 7fba7fe00640 1 -- 172.21.15.67:0/2763844880 learned_addr learned my addr 172.21.15.67:0/2763844880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba87400640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba88145f80 0x7fba880af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba86a00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 0x7fba8815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 -- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba88145f80 msgr2=0x7fba880af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba88145f80 0x7fba880af090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 -- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 msgr2=0x7fba8815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 0x7fba8815bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 -- 172.21.15.67:0/2763844880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba881e5710 con 0x7fba880cd920 2024-04-17T07:14:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba86a00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 0x7fba8815bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7fe00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba7000c440 tx=0x7fba70001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.481+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70017070 con 0x7fba880cd920 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.482+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba70005ce0 con 0x7fba880cd920 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.482+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700072f0 con 0x7fba880cd920 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.482+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba881e59f0 con 0x7fba880cd920 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.482+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba881e5fb0 con 0x7fba880cd920 2024-04-17T07:14:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.483+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba88068a10 con 0x7fba880cd920 2024-04-17T07:14:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.484+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba7001a070 con 0x7fba880cd920 2024-04-17T07:14:00.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.485+0000 7fba7ea00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 0x7fba6406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:00.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.485+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fba7008ae90 con 0x7fba880cd920 2024-04-17T07:14:00.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.485+0000 7fba86a00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 0x7fba6406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.487+0000 7fba86a00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 0x7fba6406ee00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba7800a830 tx=0x7fba78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.489+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba8806f1d0 con 0x7fba880cd920 2024-04-17T07:14:00.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:00.732+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fba880d0ab0 con 0x7fba880cd920 2024-04-17T07:14:01.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.229+0000 7fba7ea00640 1 -- 172.21.15.67:0/2763844880 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fba700593a0 con 0x7fba880cd920 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 msgr2=0x7fba6406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 0x7fba6406ee00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba7800a830 tx=0x7fba78008040 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 msgr2=0x7fba881e4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba7000c440 tx=0x7fba70001e70 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 shutdown_connections 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fba6406c940 0x7fba6406ee00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba88145f80 0x7fba880af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba880cd920 0x7fba881e4c50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 --2- 172.21.15.67:0/2763844880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba8806c270 0x7fba8815bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.233+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 >> 172.21.15.67:0/2763844880 conn(0x7fba880b2190 msgr2=0x7fba8806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:01.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.234+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 shutdown_connections 2024-04-17T07:14:01.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.234+0000 7fba87e00640 1 -- 172.21.15.67:0/2763844880 wait complete. 2024-04-17T07:14:01.255 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:14:01.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.474+0000 7fa848c00640 1 Processor -- start 2024-04-17T07:14:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.474+0000 7fa848c00640 1 -- start start 2024-04-17T07:14:01.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.475+0000 7fa848c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa844071bd0 0x7fa84415fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa848c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84409b170 0x7fa84415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa848c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa84409ba70 con 0x7fa84406cbe0 2024-04-17T07:14:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa848c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa84406ff40 con 0x7fa844071bd0 2024-04-17T07:14:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa848c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa844160040 con 0x7fa84409b170 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa842a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa84406cbe0 0x7fa84409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49468/0 (socket says 172.21.15.67:49468) 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa843400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa844071bd0 0x7fa84415fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa842a00640 1 -- 172.21.15.67:0/3431761022 learned_addr learned my addr 172.21.15.67:0/3431761022 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.476+0000 7fa843e00640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84409b170 0x7fa84415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.477+0000 7fa843400640 1 -- 172.21.15.67:0/3431761022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84409b170 msgr2=0x7fa84415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.477+0000 7fa843400640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84409b170 0x7fa84415e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.477+0000 7fa843400640 1 -- 172.21.15.67:0/3431761022 >> v1:172.21.15.67:6789/0 conn(0x7fa84406cbe0 legacy=0x7fa84409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.477+0000 7fa843400640 1 -- 172.21.15.67:0/3431761022 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84406d0e0 con 0x7fa844071bd0 2024-04-17T07:14:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.477+0000 7fa842000640 1 -- 172.21.15.67:0/3431761022 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646205059 0 0) 0x7fa830001110 con 0x7fa84406cbe0 2024-04-17T07:14:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.478+0000 7fa843400640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844071bd0 0x7fa84415fb00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8340097a0 tx=0x7fa83402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261048d1d5f8eb66 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.478+0000 7fa842000640 1 -- 172.21.15.67:0/3431761022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834035020 con 0x7fa844071bd0 2024-04-17T07:14:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.478+0000 7fa842000640 1 -- 172.21.15.67:0/3431761022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa834031810 con 0x7fa844071bd0 2024-04-17T07:14:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.478+0000 7fa842000640 1 -- 172.21.15.67:0/3431761022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834031be0 con 0x7fa844071bd0 2024-04-17T07:14:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 -- 172.21.15.67:0/3431761022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844071bd0 msgr2=0x7fa84415fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844071bd0 0x7fa84415fb00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8340097a0 tx=0x7fa83402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 -- 172.21.15.67:0/3431761022 shutdown_connections 2024-04-17T07:14:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84409b170 0x7fa84415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 --2- 172.21.15.67:0/3431761022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844071bd0 0x7fa84415fb00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 -- 172.21.15.67:0/3431761022 >> 172.21.15.67:0/3431761022 conn(0x7fa8440b40e0 msgr2=0x7fa8440b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:01.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 -- 172.21.15.67:0/3431761022 shutdown_connections 2024-04-17T07:14:01.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.479+0000 7fa848c00640 1 -- 172.21.15.67:0/3431761022 wait complete. 2024-04-17T07:14:01.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.481+0000 7fa848c00640 1 Processor -- start 2024-04-17T07:14:01.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.481+0000 7fa848c00640 1 -- start start 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84406cbe0 0x7fa8440b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 0x7fa8440c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa84406d980 con 0x7fa84409b170 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa84406d6a0 con 0x7fa844071bd0 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.482+0000 7fa848c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa84406dc60 con 0x7fa84406cbe0 2024-04-17T07:14:01.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56958/0 (socket says 172.21.15.67:56958) 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 -- 172.21.15.67:0/3150592204 learned_addr learned my addr 172.21.15.67:0/3150592204 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 -- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84406cbe0 msgr2=0x7fa8440b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84406cbe0 0x7fa8440b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 -- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 msgr2=0x7fa8440c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa842a00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 0x7fa8440c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 0x7fa8440c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 -- 172.21.15.67:0/3150592204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8440c75c0 con 0x7fa84409b170 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa842a00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 0x7fa8440c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.483+0000 7fa843e00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00ebe0 tx=0x7fa83c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.484+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c018020 con 0x7fa84409b170 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.484+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83c00cd40 con 0x7fa84409b170 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.484+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0146c0 con 0x7fa84409b170 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.484+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8440c78a0 con 0x7fa84409b170 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.484+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8440c7e60 con 0x7fa84409b170 2024-04-17T07:14:01.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.485+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8440b2fe0 con 0x7fa84409b170 2024-04-17T07:14:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.486+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa83c005d10 con 0x7fa84409b170 2024-04-17T07:14:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.487+0000 7fa83b400640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 0x7fa82406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.487+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa83c08c8e0 con 0x7fa84409b170 2024-04-17T07:14:01.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.487+0000 7fa843400640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 0x7fa82406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.489+0000 7fa843400640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 0x7fa82406ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa834036000 tx=0x7fa834030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:01.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.491+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa8440b2fe0 con 0x7fa84409b170 2024-04-17T07:14:01.573 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T07:14:01.572+0000 7f0e09e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:14:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.737+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8440755d0 con 0x7fa84409b170 2024-04-17T07:14:01.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.738+0000 7fa83b400640 1 -- 172.21.15.67:0/3150592204 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7fa83c022030 con 0x7fa84409b170 2024-04-17T07:14:01.738 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:01.738 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:01.738 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.740+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 msgr2=0x7fa82406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.740+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 0x7fa82406ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa834036000 tx=0x7fa834030000 comp rx=0 tx=0).stop 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.740+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 msgr2=0x7fa8440b7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.740+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00ebe0 tx=0x7fa83c00c490 comp rx=0 tx=0).stop 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 shutdown_connections 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84409b170 0x7fa8440b7020 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa844071bd0 0x7fa8440c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa82406c920 0x7fa82406ede0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 --2- 172.21.15.67:0/3150592204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa84406cbe0 0x7fa8440b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 >> 172.21.15.67:0/3150592204 conn(0x7fa8440b40e0 msgr2=0x7fa84415f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:01.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 shutdown_connections 2024-04-17T07:14:01.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.741+0000 7fa848c00640 1 -- 172.21.15.67:0/3150592204 wait complete. 2024-04-17T07:14:01.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:01.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.986+0000 7f9b93200640 1 Processor -- start 2024-04-17T07:14:01.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.986+0000 7f9b93200640 1 -- start start 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b93200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b9415ca20 0x7f9b940d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b93200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b940d0720 0x7f9b9409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b93200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b9415ce00 con 0x7f9b9409afa0 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b93200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9415d090 con 0x7f9b9415ca20 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b93200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9415d330 con 0x7f9b940d0720 2024-04-17T07:14:01.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b92800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b9409afa0 0x7f9b941604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49470/0 (socket says 172.21.15.67:49470) 2024-04-17T07:14:01.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b92800640 1 -- 172.21.15.67:0/227308538 learned_addr learned my addr 172.21.15.67:0/227308538 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:01.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b91e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b9415ca20 0x7f9b940d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.988+0000 7f9b89400640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b940d0720 0x7f9b9409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.989+0000 7f9b91e00640 1 -- 172.21.15.67:0/227308538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b940d0720 msgr2=0x7f9b9409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:01.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.989+0000 7f9b91e00640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b940d0720 0x7f9b9409aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.989+0000 7f9b91e00640 1 -- 172.21.15.67:0/227308538 >> v1:172.21.15.67:6789/0 conn(0x7f9b9409afa0 legacy=0x7f9b941604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.989+0000 7f9b91e00640 1 -- 172.21.15.67:0/227308538 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9415d610 con 0x7f9b9415ca20 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.990+0000 7f9b91e00640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b9415ca20 0x7f9b940d01e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9b840097a0 tx=0x7f9b8402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4118012e1b2ca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.990+0000 7f9b91400640 1 -- 172.21.15.67:0/227308538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84035020 con 0x7f9b9415ca20 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.990+0000 7f9b91400640 1 -- 172.21.15.67:0/227308538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b84031810 con 0x7f9b9415ca20 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.990+0000 7f9b91400640 1 -- 172.21.15.67:0/227308538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84031be0 con 0x7f9b9415ca20 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 -- 172.21.15.67:0/227308538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b9415ca20 msgr2=0x7f9b940d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b9415ca20 0x7f9b940d01e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9b840097a0 tx=0x7f9b8402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 -- 172.21.15.67:0/227308538 shutdown_connections 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b940d0720 0x7f9b9409aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 --2- 172.21.15.67:0/227308538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b9415ca20 0x7f9b940d01e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 -- 172.21.15.67:0/227308538 >> 172.21.15.67:0/227308538 conn(0x7f9b940b4100 msgr2=0x7f9b940b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 -- 172.21.15.67:0/227308538 shutdown_connections 2024-04-17T07:14:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.991+0000 7f9b93200640 1 -- 172.21.15.67:0/227308538 wait complete. 2024-04-17T07:14:01.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.993+0000 7f9b93200640 1 Processor -- start 2024-04-17T07:14:01.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.993+0000 7f9b93200640 1 -- start start 2024-04-17T07:14:01.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b9409afa0 0x7f9b940b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b9415ca20 0x7f9b940c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9409bea0 con 0x7f9b940d0720 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9409bd20 con 0x7f9b9409afa0 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b93200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b940ce1a0 con 0x7f9b9415ca20 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56978/0 (socket says 172.21.15.67:56978) 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b89400640 1 -- 172.21.15.67:0/3252788000 learned_addr learned my addr 172.21.15.67:0/3252788000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.994+0000 7f9b91e00640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b9409afa0 0x7f9b940b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 -- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b9415ca20 msgr2=0x7f9b940c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b9415ca20 0x7f9b940c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 -- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b9409afa0 msgr2=0x7f9b940b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:01.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b9409afa0 0x7f9b940b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:01.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 -- 172.21.15.67:0/3252788000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b940ca1d0 con 0x7f9b940d0720 2024-04-17T07:14:01.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b89400640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00d7d0 tx=0x7f9b7c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:01.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c004020 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c0041e0 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b940ca4b0 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c0043d0 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.995+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b940b5cd0 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.996+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b940d2150 con 0x7f9b940d0720 2024-04-17T07:14:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.997+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b7c01b020 con 0x7f9b940d0720 2024-04-17T07:14:01.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.998+0000 7f9b8aa00640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 0x7f9b7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:01.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.998+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9b7c08b950 con 0x7f9b940d0720 2024-04-17T07:14:01.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:01.999+0000 7f9b91e00640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 0x7f9b7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.000+0000 7f9b91e00640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 0x7f9b7006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b84036000 tx=0x7f9b84008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:02.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.003+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9b940d2150 con 0x7f9b940d0720 2024-04-17T07:14:02.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.214+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b940755d0 con 0x7f9b940d0720 2024-04-17T07:14:02.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.216+0000 7f9b8aa00640 1 -- 172.21.15.67:0/3252788000 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f9b7c059d80 con 0x7f9b940d0720 2024-04-17T07:14:02.217 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:02.217 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 msgr2=0x7f9b7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 0x7f9b7006edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b84036000 tx=0x7f9b84008030 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 msgr2=0x7f9b940c4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00d7d0 tx=0x7f9b7c00dc90 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 shutdown_connections 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9b7006c8f0 0x7f9b7006edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b9415ca20 0x7f9b940c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b940d0720 0x7f9b940c4dd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 --2- 172.21.15.67:0/3252788000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b9409afa0 0x7f9b940b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 >> 172.21.15.67:0/3252788000 conn(0x7f9b940b4100 msgr2=0x7f9b9415f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 shutdown_connections 2024-04-17T07:14:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.219+0000 7f9b93200640 1 -- 172.21.15.67:0/3252788000 wait complete. 2024-04-17T07:14:02.241 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:14:02.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:14:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.466+0000 7f3850e00640 1 Processor -- start 2024-04-17T07:14:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.466+0000 7f3850e00640 1 -- start start 2024-04-17T07:14:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f3850e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f384c141ed0 0x7f384c142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f3850e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f384c0cd4a0 con 0x7f384c160280 2024-04-17T07:14:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f3850e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f384c1607f0 con 0x7f384c06b8c0 2024-04-17T07:14:02.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f3850e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f384c15ce30 con 0x7f384c141ed0 2024-04-17T07:14:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f384aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f384c160280 0x7f384c06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49478/0 (socket says 172.21.15.67:49478) 2024-04-17T07:14:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.468+0000 7f384aa00640 1 -- 172.21.15.67:0/691011047 learned_addr learned my addr 172.21.15.67:0/691011047 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200720376 0 0) 0x7f384c0cd4a0 con 0x7f384c160280 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384be00640 1 --2- 172.21.15.67:0/691011047 >> v2:172.21.15.187:3300/0 conn(0x7f384c141ed0 0x7f384c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3830003520 con 0x7f384c160280 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2420925365 0 0) 0x7f3830003520 con 0x7f384c160280 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 >> v2:172.21.15.187:3300/0 conn(0x7f384c141ed0 msgr2=0x7f384c142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 --2- 172.21.15.67:0/691011047 >> v2:172.21.15.187:3300/0 conn(0x7f384c141ed0 0x7f384c142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 >> v1:172.21.15.187:6789/0 conn(0x7f384c06b8c0 legacy=0x7f384c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c160ad0 con 0x7f384c160280 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3838001670 con 0x7f384c160280 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 >> v1:172.21.15.67:6789/0 conn(0x7f384c160280 legacy=0x7f384c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3830003a20 0x7f3830003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 0x7f3830004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38300051c0 0x7f383000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f384c15ce30 con 0x7f3830004800 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f383000c0f0 con 0x7f38300051c0 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.469+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f383000c2f0 con 0x7f3830003a20 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 0x7f3830004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384be00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38300051c0 0x7f383000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384b400640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3830003a20 0x7f3830003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 -- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3830003a20 msgr2=0x7f3830003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3830003a20 0x7f3830003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 -- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38300051c0 msgr2=0x7f383000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38300051c0 0x7f383000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384be00640 1 -- 172.21.15.67:0/691011047 reap_dead start 2024-04-17T07:14:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.470+0000 7f384aa00640 1 -- 172.21.15.67:0/691011047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3830003520 con 0x7f3830004800 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.472+0000 7f384aa00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 0x7f3830004c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f383800c210 tx=0x7f3830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db09d428d86bc005 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.472+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838007d30 con 0x7f3830004800 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.472+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3838030070 con 0x7f3830004800 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.472+0000 7f384a000640 1 -- 172.21.15.67:0/691011047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838005d70 con 0x7f3830004800 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 -- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 msgr2=0x7f3830004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 0x7f3830004c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f383800c210 tx=0x7f3830005830 comp rx=0 tx=0).stop 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 -- 172.21.15.67:0/691011047 shutdown_connections 2024-04-17T07:14:02.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 --2- 172.21.15.67:0/691011047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3830004800 0x7f3830004c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 -- 172.21.15.67:0/691011047 >> 172.21.15.67:0/691011047 conn(0x7f384c067d10 msgr2=0x7f384c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 -- 172.21.15.67:0/691011047 shutdown_connections 2024-04-17T07:14:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.473+0000 7f3850e00640 1 -- 172.21.15.67:0/691011047 wait complete. 2024-04-17T07:14:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.475+0000 7f3850e00640 1 Processor -- start 2024-04-17T07:14:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.476+0000 7f3850e00640 1 -- start start 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830004800 0x7f384c13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f384c1579d0 0x7f384c157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f384c13f9b0 con 0x7f384c0b41f0 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f384c161090 con 0x7f3830004800 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.477+0000 7f3850e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f384c13fb30 con 0x7f384c1579d0 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384b400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830004800 0x7f384c13ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57004/0 (socket says 172.21.15.67:57004) 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 -- 172.21.15.67:0/437674897 learned_addr learned my addr 172.21.15.67:0/437674897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:02.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384be00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f384c1579d0 0x7f384c157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 -- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f384c1579d0 msgr2=0x7f384c157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f384c1579d0 0x7f384c157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 -- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830004800 msgr2=0x7f384c13ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830004800 0x7f384c13ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.478+0000 7f384aa00640 1 -- 172.21.15.67:0/437674897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c158390 con 0x7f384c0b41f0 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f384aa00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f383802c4e0 tx=0x7f383802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:02.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838046070 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3838004bf0 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3838007d10 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384c13f3a0 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.479+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384c144d50 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.480+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384c068a10 con 0x7f384c0b41f0 2024-04-17T07:14:02.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.481+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3838007070 con 0x7f384c0b41f0 2024-04-17T07:14:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.483+0000 7f3843400640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 0x7f382c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.483+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f38380acec0 con 0x7f384c0b41f0 2024-04-17T07:14:02.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.483+0000 7f384b400640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 0x7f382c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:02.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.484+0000 7f384b400640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 0x7f382c06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f383c007aa0 tx=0x7f383c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.487+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f384c13f3a0 con 0x7f384c0b41f0 2024-04-17T07:14:02.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.688+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f384c144b00 con 0x7f382c06c8f0 2024-04-17T07:14:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.711+0000 7f3843400640 1 -- 172.21.15.67:0/437674897 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7f384c144b00 con 0x7f382c06c8f0 2024-04-17T07:14:02.715 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:02.716 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 msgr2=0x7f382c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 0x7f382c06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f383c007aa0 tx=0x7f383c009040 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 msgr2=0x7f384c0b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f383802c4e0 tx=0x7f383802e070 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 shutdown_connections 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f382c06c8f0 0x7f382c06edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f384c1579d0 0x7f384c157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f384c0b41f0 0x7f384c0b4670 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 --2- 172.21.15.67:0/437674897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3830004800 0x7f384c13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 >> 172.21.15.67:0/437674897 conn(0x7f384c067d10 msgr2=0x7f384c141640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.717+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 shutdown_connections 2024-04-17T07:14:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:02.718+0000 7f3850e00640 1 -- 172.21.15.67:0/437674897 wait complete. 2024-04-17T07:14:02.739 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-17T07:14:01.353848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223320,"kb_used_data":2960,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751400,"statfs":{"total":773094113280,"available":772865433600,"internally_reserved":0,"allocated":3031040,"data_stored":1803628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.194087"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228829+0000","last_change":"2024-04-17T07:13:58.227207+0000","last_active":"2024-04-17T07:13:59.228829+0000","last_peered":"2024-04-17T07:13:59.228829+0000","last_clean":"2024-04-17T07:13:59.228829+0000","last_became_active":"2024-04-17T07:13:58.223412+0000","last_became_peered":"2024-04-17T07:13:58.223412+0000","last_unstale":"2024-04-17T07:13:59.228829+0000","last_undegraded":"2024-04-17T07:13:59.228829+0000","last_fullsized":"2024-04-17T07:13:59.228829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217686+0000","last_change":"2024-04-17T07:13:58.238529+0000","last_active":"2024-04-17T07:13:59.217686+0000","last_peered":"2024-04-17T07:13:59.217686+0000","last_clean":"2024-04-17T07:13:59.217686+0000","last_became_active":"2024-04-17T07:13:58.238155+0000","last_became_peered":"2024-04-17T07:13:58.238155+0000","last_unstale":"2024-04-17T07:13:59.217686+0000","last_undegraded":"2024-04-17T07:13:59.217686+0000","last_fullsized":"2024-04-17T07:13:59.217686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148756+0000","last_change":"2024-04-17T07:13:58.242232+0000","last_active":"2024-04-17T07:14:00.148756+0000","last_peered":"2024-04-17T07:14:00.148756+0000","last_clean":"2024-04-17T07:14:00.148756+0000","last_became_active":"2024-04-17T07:13:58.241891+0000","last_became_peered":"2024-04-17T07:13:58.241891+0000","last_unstale":"2024-04-17T07:14:00.148756+0000","last_undegraded":"2024-04-17T07:14:00.148756+0000","last_fullsized":"2024-04-17T07:14:00.148756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:17.659180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229570+0000","last_change":"2024-04-17T07:13:58.234270+0000","last_active":"2024-04-17T07:13:59.229570+0000","last_peered":"2024-04-17T07:13:59.229570+0000","last_clean":"2024-04-17T07:13:59.229570+0000","last_became_active":"2024-04-17T07:13:58.233725+0000","last_became_peered":"2024-04-17T07:13:58.233725+0000","last_unstale":"2024-04-17T07:13:59.229570+0000","last_undegraded":"2024-04-17T07:13:59.229570+0000","last_fullsized":"2024-04-17T07:13:59.229570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226745+0000","last_change":"2024-04-17T07:13:58.215673+0000","last_active":"2024-04-17T07:13:59.226745+0000","last_peered":"2024-04-17T07:13:59.226745+0000","last_clean":"2024-04-17T07:13:59.226745+0000","last_became_active":"2024-04-17T07:13:58.215371+0000","last_became_peered":"2024-04-17T07:13:58.215371+0000","last_unstale":"2024-04-17T07:13:59.226745+0000","last_undegraded":"2024-04-17T07:13:59.226745+0000","last_fullsized":"2024-04-17T07:13:59.226745+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228445+0000","last_change":"2024-04-17T07:13:58.240332+0000","last_active":"2024-04-17T07:13:59.228445+0000","last_peered":"2024-04-17T07:13:59.228445+0000","last_clean":"2024-04-17T07:13:59.228445+0000","last_became_active":"2024-04-17T07:13:58.239852+0000","last_became_peered":"2024-04-17T07:13:58.239852+0000","last_unstale":"2024-04-17T07:13:59.228445+0000","last_undegraded":"2024-04-17T07:13:59.228445+0000","last_fullsized":"2024-04-17T07:13:59.228445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226714+0000","last_change":"2024-04-17T07:13:58.238930+0000","last_active":"2024-04-17T07:13:59.226714+0000","last_peered":"2024-04-17T07:13:59.226714+0000","last_clean":"2024-04-17T07:13:59.226714+0000","last_became_active":"2024-04-17T07:13:58.238663+0000","last_became_peered":"2024-04-17T07:13:58.238663+0000","last_unstale":"2024-04-17T07:13:59.226714+0000","last_undegraded":"2024-04-17T07:13:59.226714+0000","last_fullsized":"2024-04-17T07:13:59.226714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144325+0000","last_change":"2024-04-17T07:13:58.227713+0000","last_active":"2024-04-17T07:14:00.144325+0000","last_peered":"2024-04-17T07:14:00.144325+0000","last_clean":"2024-04-17T07:14:00.144325+0000","last_became_active":"2024-04-17T07:13:58.227352+0000","last_became_peered":"2024-04-17T07:13:58.227352+0000","last_unstale":"2024-04-17T07:14:00.144325+0000","last_undegraded":"2024-04-17T07:14:00.144325+0000","last_fullsized":"2024-04-17T07:14:00.144325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228952+0000","last_change":"2024-04-17T07:13:58.233019+0000","last_active":"2024-04-17T07:13:59.228952+0000","last_peered":"2024-04-17T07:13:59.228952+0000","last_clean":"2024-04-17T07:13:59.228952+0000","last_became_active":"2024-04-17T07:13:58.230661+0000","last_became_peered":"2024-04-17T07:13:58.230661+0000","last_unstale":"2024-04-17T07:13:59.228952+0000","last_undegraded":"2024-04-17T07:13:59.228952+0000","last_fullsized":"2024-04-17T07:13:59.228952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:33.970758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226560+0000","last_change":"2024-04-17T07:13:58.225623+0000","last_active":"2024-04-17T07:13:59.226560+0000","last_peered":"2024-04-17T07:13:59.226560+0000","last_clean":"2024-04-17T07:13:59.226560+0000","last_became_active":"2024-04-17T07:13:58.225345+0000","last_became_peered":"2024-04-17T07:13:58.225345+0000","last_unstale":"2024-04-17T07:13:59.226560+0000","last_undegraded":"2024-04-17T07:13:59.226560+0000","last_fullsized":"2024-04-17T07:13:59.226560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226194+0000","last_change":"2024-04-17T07:13:58.226592+0000","last_active":"2024-04-17T07:13:59.226194+0000","last_peered":"2024-04-17T07:13:59.226194+0000","last_clean":"2024-04-17T07:13:59.226194+0000","last_became_active":"2024-04-17T07:13:58.226241+0000","last_became_peered":"2024-04-17T07:13:58.226241+0000","last_unstale":"2024-04-17T07:13:59.226194+0000","last_undegraded":"2024-04-17T07:13:59.226194+0000","last_fullsized":"2024-04-17T07:13:59.226194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144301+0000","last_change":"2024-04-17T07:13:58.216920+0000","last_active":"2024-04-17T07:14:00.144301+0000","last_peered":"2024-04-17T07:14:00.144301+0000","last_clean":"2024-04-17T07:14:00.144301+0000","last_became_active":"2024-04-17T07:13:58.216635+0000","last_became_peered":"2024-04-17T07:13:58.216635+0000","last_unstale":"2024-04-17T07:14:00.144301+0000","last_undegraded":"2024-04-17T07:14:00.144301+0000","last_fullsized":"2024-04-17T07:14:00.144301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:51.668433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146122+0000","last_change":"2024-04-17T07:13:58.225422+0000","last_active":"2024-04-17T07:14:00.146122+0000","last_peered":"2024-04-17T07:14:00.146122+0000","last_clean":"2024-04-17T07:14:00.146122+0000","last_became_active":"2024-04-17T07:13:58.224906+0000","last_became_peered":"2024-04-17T07:13:58.224906+0000","last_unstale":"2024-04-17T07:14:00.146122+0000","last_undegraded":"2024-04-17T07:14:00.146122+0000","last_fullsized":"2024-04-17T07:14:00.146122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147718+0000","last_change":"2024-04-17T07:13:58.233409+0000","last_active":"2024-04-17T07:14:00.147718+0000","last_peered":"2024-04-17T07:14:00.147718+0000","last_clean":"2024-04-17T07:14:00.147718+0000","last_became_active":"2024-04-17T07:13:58.233182+0000","last_became_peered":"2024-04-17T07:13:58.233182+0000","last_unstale":"2024-04-17T07:14:00.147718+0000","last_undegraded":"2024-04-17T07:14:00.147718+0000","last_fullsized":"2024-04-17T07:14:00.147718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:18.125280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217646+0000","last_change":"2024-04-17T07:13:58.215689+0000","last_active":"2024-04-17T07:13:59.217646+0000","last_peered":"2024-04-17T07:13:59.217646+0000","last_clean":"2024-04-17T07:13:59.217646+0000","last_became_active":"2024-04-17T07:13:58.215054+0000","last_became_peered":"2024-04-17T07:13:58.215054+0000","last_unstale":"2024-04-17T07:13:59.217646+0000","last_undegraded":"2024-04-17T07:13:59.217646+0000","last_fullsized":"2024-04-17T07:13:59.217646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227948+0000","last_change":"2024-04-17T07:13:58.220483+0000","last_active":"2024-04-17T07:13:59.227948+0000","last_peered":"2024-04-17T07:13:59.227948+0000","last_clean":"2024-04-17T07:13:59.227948+0000","last_became_active":"2024-04-17T07:13:58.220205+0000","last_became_peered":"2024-04-17T07:13:58.220205+0000","last_unstale":"2024-04-17T07:13:59.227948+0000","last_undegraded":"2024-04-17T07:13:59.227948+0000","last_fullsized":"2024-04-17T07:13:59.227948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231680+0000","last_change":"2024-04-17T07:13:58.223188+0000","last_active":"2024-04-17T07:13:59.231680+0000","last_peered":"2024-04-17T07:13:59.231680+0000","last_clean":"2024-04-17T07:13:59.231680+0000","last_became_active":"2024-04-17T07:13:58.221699+0000","last_became_peered":"2024-04-17T07:13:58.221699+0000","last_unstale":"2024-04-17T07:13:59.231680+0000","last_undegraded":"2024-04-17T07:13:59.231680+0000","last_fullsized":"2024-04-17T07:13:59.231680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144144+0000","last_change":"2024-04-17T07:13:58.220308+0000","last_active":"2024-04-17T07:14:00.144144+0000","last_peered":"2024-04-17T07:14:00.144144+0000","last_clean":"2024-04-17T07:14:00.144144+0000","last_became_active":"2024-04-17T07:13:58.219976+0000","last_became_peered":"2024-04-17T07:13:58.219976+0000","last_unstale":"2024-04-17T07:14:00.144144+0000","last_undegraded":"2024-04-17T07:14:00.144144+0000","last_fullsized":"2024-04-17T07:14:00.144144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:11.629034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228541+0000","last_change":"2024-04-17T07:13:58.223620+0000","last_active":"2024-04-17T07:13:59.228541+0000","last_peered":"2024-04-17T07:13:59.228541+0000","last_clean":"2024-04-17T07:13:59.228541+0000","last_became_active":"2024-04-17T07:13:58.222125+0000","last_became_peered":"2024-04-17T07:13:58.222125+0000","last_unstale":"2024-04-17T07:13:59.228541+0000","last_undegraded":"2024-04-17T07:13:59.228541+0000","last_fullsized":"2024-04-17T07:13:59.228541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227555+0000","last_change":"2024-04-17T07:13:58.225907+0000","last_active":"2024-04-17T07:13:59.227555+0000","last_peered":"2024-04-17T07:13:59.227555+0000","last_clean":"2024-04-17T07:13:59.227555+0000","last_became_active":"2024-04-17T07:13:58.224993+0000","last_became_peered":"2024-04-17T07:13:58.224993+0000","last_unstale":"2024-04-17T07:13:59.227555+0000","last_undegraded":"2024-04-17T07:13:59.227555+0000","last_fullsized":"2024-04-17T07:13:59.227555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148489+0000","last_change":"2024-04-17T07:13:58.236523+0000","last_active":"2024-04-17T07:14:00.148489+0000","last_peered":"2024-04-17T07:14:00.148489+0000","last_clean":"2024-04-17T07:14:00.148489+0000","last_became_active":"2024-04-17T07:13:58.236164+0000","last_became_peered":"2024-04-17T07:13:58.236164+0000","last_unstale":"2024-04-17T07:14:00.148489+0000","last_undegraded":"2024-04-17T07:14:00.148489+0000","last_fullsized":"2024-04-17T07:14:00.148489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:33.648495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228822+0000","last_change":"2024-04-17T07:13:58.222632+0000","last_active":"2024-04-17T07:13:59.228822+0000","last_peered":"2024-04-17T07:13:59.228822+0000","last_clean":"2024-04-17T07:13:59.228822+0000","last_became_active":"2024-04-17T07:13:58.221666+0000","last_became_peered":"2024-04-17T07:13:58.221666+0000","last_unstale":"2024-04-17T07:13:59.228822+0000","last_undegraded":"2024-04-17T07:13:59.228822+0000","last_fullsized":"2024-04-17T07:13:59.228822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145740+0000","last_change":"2024-04-17T07:13:58.218642+0000","last_active":"2024-04-17T07:14:00.145740+0000","last_peered":"2024-04-17T07:14:00.145740+0000","last_clean":"2024-04-17T07:14:00.145740+0000","last_became_active":"2024-04-17T07:13:58.218235+0000","last_became_peered":"2024-04-17T07:13:58.218235+0000","last_unstale":"2024-04-17T07:14:00.145740+0000","last_undegraded":"2024-04-17T07:14:00.145740+0000","last_fullsized":"2024-04-17T07:14:00.145740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228851+0000","last_change":"2024-04-17T07:13:58.232074+0000","last_active":"2024-04-17T07:13:59.228851+0000","last_peered":"2024-04-17T07:13:59.228851+0000","last_clean":"2024-04-17T07:13:59.228851+0000","last_became_active":"2024-04-17T07:13:58.231716+0000","last_became_peered":"2024-04-17T07:13:58.231716+0000","last_unstale":"2024-04-17T07:13:59.228851+0000","last_undegraded":"2024-04-17T07:13:59.228851+0000","last_fullsized":"2024-04-17T07:13:59.228851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147203+0000","last_change":"2024-04-17T07:13:58.230039+0000","last_active":"2024-04-17T07:14:00.147203+0000","last_peered":"2024-04-17T07:14:00.147203+0000","last_clean":"2024-04-17T07:14:00.147203+0000","last_became_active":"2024-04-17T07:13:58.224374+0000","last_became_peered":"2024-04-17T07:13:58.224374+0000","last_unstale":"2024-04-17T07:14:00.147203+0000","last_undegraded":"2024-04-17T07:14:00.147203+0000","last_fullsized":"2024-04-17T07:14:00.147203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226940+0000","last_change":"2024-04-17T07:13:58.227485+0000","last_active":"2024-04-17T07:13:59.226940+0000","last_peered":"2024-04-17T07:13:59.226940+0000","last_clean":"2024-04-17T07:13:59.226940+0000","last_became_active":"2024-04-17T07:13:58.227182+0000","last_became_peered":"2024-04-17T07:13:58.227182+0000","last_unstale":"2024-04-17T07:13:59.226940+0000","last_undegraded":"2024-04-17T07:13:59.226940+0000","last_fullsized":"2024-04-17T07:13:59.226940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217943+0000","last_change":"2024-04-17T07:13:58.222243+0000","last_active":"2024-04-17T07:13:59.217943+0000","last_peered":"2024-04-17T07:13:59.217943+0000","last_clean":"2024-04-17T07:13:59.217943+0000","last_became_active":"2024-04-17T07:13:58.221925+0000","last_became_peered":"2024-04-17T07:13:58.221925+0000","last_unstale":"2024-04-17T07:13:59.217943+0000","last_undegraded":"2024-04-17T07:13:59.217943+0000","last_fullsized":"2024-04-17T07:13:59.217943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146781+0000","last_change":"2024-04-17T07:13:58.226680+0000","last_active":"2024-04-17T07:14:00.146781+0000","last_peered":"2024-04-17T07:14:00.146781+0000","last_clean":"2024-04-17T07:14:00.146781+0000","last_became_active":"2024-04-17T07:13:58.226283+0000","last_became_peered":"2024-04-17T07:13:58.226283+0000","last_unstale":"2024-04-17T07:14:00.146781+0000","last_undegraded":"2024-04-17T07:14:00.146781+0000","last_fullsized":"2024-04-17T07:14:00.146781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148779+0000","last_change":"2024-04-17T07:13:58.242478+0000","last_active":"2024-04-17T07:14:00.148779+0000","last_peered":"2024-04-17T07:14:00.148779+0000","last_clean":"2024-04-17T07:14:00.148779+0000","last_became_active":"2024-04-17T07:13:58.241968+0000","last_became_peered":"2024-04-17T07:13:58.241968+0000","last_unstale":"2024-04-17T07:14:00.148779+0000","last_undegraded":"2024-04-17T07:14:00.148779+0000","last_fullsized":"2024-04-17T07:14:00.148779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:29:46.240069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229245+0000","last_change":"2024-04-17T07:13:58.230046+0000","last_active":"2024-04-17T07:13:59.229245+0000","last_peered":"2024-04-17T07:13:59.229245+0000","last_clean":"2024-04-17T07:13:59.229245+0000","last_became_active":"2024-04-17T07:13:58.229744+0000","last_became_peered":"2024-04-17T07:13:58.229744+0000","last_unstale":"2024-04-17T07:13:59.229245+0000","last_undegraded":"2024-04-17T07:13:59.229245+0000","last_fullsized":"2024-04-17T07:13:59.229245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143075+0000","last_change":"2024-04-17T07:13:58.219601+0000","last_active":"2024-04-17T07:14:00.143075+0000","last_peered":"2024-04-17T07:14:00.143075+0000","last_clean":"2024-04-17T07:14:00.143075+0000","last_became_active":"2024-04-17T07:13:58.219208+0000","last_became_peered":"2024-04-17T07:13:58.219208+0000","last_unstale":"2024-04-17T07:14:00.143075+0000","last_undegraded":"2024-04-17T07:14:00.143075+0000","last_fullsized":"2024-04-17T07:14:00.143075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:06.342021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225081+0000","last_change":"2024-04-17T07:13:58.230063+0000","last_active":"2024-04-17T07:13:59.225081+0000","last_peered":"2024-04-17T07:13:59.225081+0000","last_clean":"2024-04-17T07:13:59.225081+0000","last_became_active":"2024-04-17T07:13:58.229129+0000","last_became_peered":"2024-04-17T07:13:58.229129+0000","last_unstale":"2024-04-17T07:13:59.225081+0000","last_undegraded":"2024-04-17T07:13:59.225081+0000","last_fullsized":"2024-04-17T07:13:59.225081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:59.849640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143839+0000","last_change":"2024-04-17T07:13:58.208959+0000","last_active":"2024-04-17T07:14:00.143839+0000","last_peered":"2024-04-17T07:14:00.143839+0000","last_clean":"2024-04-17T07:14:00.143839+0000","last_became_active":"2024-04-17T07:13:58.208200+0000","last_became_peered":"2024-04-17T07:13:58.208200+0000","last_unstale":"2024-04-17T07:14:00.143839+0000","last_undegraded":"2024-04-17T07:14:00.143839+0000","last_fullsized":"2024-04-17T07:14:00.143839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:14.893023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147150+0000","last_change":"2024-04-17T07:13:57.193193+0000","last_active":"2024-04-17T07:14:00.147150+0000","last_peered":"2024-04-17T07:14:00.147150+0000","last_clean":"2024-04-17T07:14:00.147150+0000","last_became_active":"2024-04-17T07:13:57.192931+0000","last_became_peered":"2024-04-17T07:13:57.192931+0000","last_unstale":"2024-04-17T07:14:00.147150+0000","last_undegraded":"2024-04-17T07:14:00.147150+0000","last_fullsized":"2024-04-17T07:14:00.147150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148893+0000","last_change":"2024-04-17T07:13:58.244126+0000","last_active":"2024-04-17T07:14:00.148893+0000","last_peered":"2024-04-17T07:14:00.148893+0000","last_clean":"2024-04-17T07:14:00.148893+0000","last_became_active":"2024-04-17T07:13:58.243846+0000","last_became_peered":"2024-04-17T07:13:58.243846+0000","last_unstale":"2024-04-17T07:14:00.148893+0000","last_undegraded":"2024-04-17T07:14:00.148893+0000","last_fullsized":"2024-04-17T07:14:00.148893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:20.038754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225363+0000","last_change":"2024-04-17T07:13:57.198746+0000","last_active":"2024-04-17T07:13:59.225363+0000","last_peered":"2024-04-17T07:13:59.225363+0000","last_clean":"2024-04-17T07:13:59.225363+0000","last_became_active":"2024-04-17T07:13:57.195870+0000","last_became_peered":"2024-04-17T07:13:57.195870+0000","last_unstale":"2024-04-17T07:13:59.225363+0000","last_undegraded":"2024-04-17T07:13:59.225363+0000","last_fullsized":"2024-04-17T07:13:59.225363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145653+0000","last_change":"2024-04-17T07:13:57.201813+0000","last_active":"2024-04-17T07:14:00.145653+0000","last_peered":"2024-04-17T07:14:00.145653+0000","last_clean":"2024-04-17T07:14:00.145653+0000","last_became_active":"2024-04-17T07:13:57.201530+0000","last_became_peered":"2024-04-17T07:13:57.201530+0000","last_unstale":"2024-04-17T07:14:00.145653+0000","last_undegraded":"2024-04-17T07:14:00.145653+0000","last_fullsized":"2024-04-17T07:14:00.145653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144391+0000","last_change":"2024-04-17T07:13:58.235592+0000","last_active":"2024-04-17T07:14:00.144391+0000","last_peered":"2024-04-17T07:14:00.144391+0000","last_clean":"2024-04-17T07:14:00.144391+0000","last_became_active":"2024-04-17T07:13:58.235287+0000","last_became_peered":"2024-04-17T07:13:58.235287+0000","last_unstale":"2024-04-17T07:14:00.144391+0000","last_undegraded":"2024-04-17T07:14:00.144391+0000","last_fullsized":"2024-04-17T07:14:00.144391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228264+0000","last_change":"2024-04-17T07:13:57.183240+0000","last_active":"2024-04-17T07:13:59.228264+0000","last_peered":"2024-04-17T07:13:59.228264+0000","last_clean":"2024-04-17T07:13:59.228264+0000","last_became_active":"2024-04-17T07:13:57.182859+0000","last_became_peered":"2024-04-17T07:13:57.182859+0000","last_unstale":"2024-04-17T07:13:59.228264+0000","last_undegraded":"2024-04-17T07:13:59.228264+0000","last_fullsized":"2024-04-17T07:13:59.228264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217429+0000","last_change":"2024-04-17T07:13:58.221113+0000","last_active":"2024-04-17T07:13:59.217429+0000","last_peered":"2024-04-17T07:13:59.217429+0000","last_clean":"2024-04-17T07:13:59.217429+0000","last_became_active":"2024-04-17T07:13:58.220807+0000","last_became_peered":"2024-04-17T07:13:58.220807+0000","last_unstale":"2024-04-17T07:13:59.217429+0000","last_undegraded":"2024-04-17T07:13:59.217429+0000","last_fullsized":"2024-04-17T07:13:59.217429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226316+0000","last_change":"2024-04-17T07:13:57.197641+0000","last_active":"2024-04-17T07:13:59.226316+0000","last_peered":"2024-04-17T07:13:59.226316+0000","last_clean":"2024-04-17T07:13:59.226316+0000","last_became_active":"2024-04-17T07:13:57.197394+0000","last_became_peered":"2024-04-17T07:13:57.197394+0000","last_unstale":"2024-04-17T07:13:59.226316+0000","last_undegraded":"2024-04-17T07:13:59.226316+0000","last_fullsized":"2024-04-17T07:13:59.226316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231265+0000","last_change":"2024-04-17T07:13:58.225914+0000","last_active":"2024-04-17T07:13:59.231265+0000","last_peered":"2024-04-17T07:13:59.231265+0000","last_clean":"2024-04-17T07:13:59.231265+0000","last_became_active":"2024-04-17T07:13:58.225649+0000","last_became_peered":"2024-04-17T07:13:58.225649+0000","last_unstale":"2024-04-17T07:13:59.231265+0000","last_undegraded":"2024-04-17T07:13:59.231265+0000","last_fullsized":"2024-04-17T07:13:59.231265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231330+0000","last_change":"2024-04-17T07:13:57.198101+0000","last_active":"2024-04-17T07:13:59.231330+0000","last_peered":"2024-04-17T07:13:59.231330+0000","last_clean":"2024-04-17T07:13:59.231330+0000","last_became_active":"2024-04-17T07:13:57.196771+0000","last_became_peered":"2024-04-17T07:13:57.196771+0000","last_unstale":"2024-04-17T07:13:59.231330+0000","last_undegraded":"2024-04-17T07:13:59.231330+0000","last_fullsized":"2024-04-17T07:13:59.231330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149355+0000","last_change":"2024-04-17T07:13:58.244824+0000","last_active":"2024-04-17T07:14:00.149355+0000","last_peered":"2024-04-17T07:14:00.149355+0000","last_clean":"2024-04-17T07:14:00.149355+0000","last_became_active":"2024-04-17T07:13:58.243368+0000","last_became_peered":"2024-04-17T07:13:58.243368+0000","last_unstale":"2024-04-17T07:14:00.149355+0000","last_undegraded":"2024-04-17T07:14:00.149355+0000","last_fullsized":"2024-04-17T07:14:00.149355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:43.836722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218062+0000","last_change":"2024-04-17T07:13:57.198519+0000","last_active":"2024-04-17T07:13:59.218062+0000","last_peered":"2024-04-17T07:13:59.218062+0000","last_clean":"2024-04-17T07:13:59.218062+0000","last_became_active":"2024-04-17T07:13:57.197225+0000","last_became_peered":"2024-04-17T07:13:57.197225+0000","last_unstale":"2024-04-17T07:13:59.218062+0000","last_undegraded":"2024-04-17T07:13:59.218062+0000","last_fullsized":"2024-04-17T07:13:59.218062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144417+0000","last_change":"2024-04-17T07:13:58.221225+0000","last_active":"2024-04-17T07:14:00.144417+0000","last_peered":"2024-04-17T07:14:00.144417+0000","last_clean":"2024-04-17T07:14:00.144417+0000","last_became_active":"2024-04-17T07:13:58.220917+0000","last_became_peered":"2024-04-17T07:13:58.220917+0000","last_unstale":"2024-04-17T07:14:00.144417+0000","last_undegraded":"2024-04-17T07:14:00.144417+0000","last_fullsized":"2024-04-17T07:14:00.144417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:04.188468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226310+0000","last_change":"2024-04-17T07:13:57.200719+0000","last_active":"2024-04-17T07:13:59.226310+0000","last_peered":"2024-04-17T07:13:59.226310+0000","last_clean":"2024-04-17T07:13:59.226310+0000","last_became_active":"2024-04-17T07:13:57.200404+0000","last_became_peered":"2024-04-17T07:13:57.200404+0000","last_unstale":"2024-04-17T07:13:59.226310+0000","last_undegraded":"2024-04-17T07:13:59.226310+0000","last_fullsized":"2024-04-17T07:13:59.226310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226654+0000","last_change":"2024-04-17T07:13:58.231037+0000","last_active":"2024-04-17T07:13:59.226654+0000","last_peered":"2024-04-17T07:13:59.226654+0000","last_clean":"2024-04-17T07:13:59.226654+0000","last_became_active":"2024-04-17T07:13:58.229754+0000","last_became_peered":"2024-04-17T07:13:58.229754+0000","last_unstale":"2024-04-17T07:13:59.226654+0000","last_undegraded":"2024-04-17T07:13:59.226654+0000","last_fullsized":"2024-04-17T07:13:59.226654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149082+0000","last_change":"2024-04-17T07:13:57.196583+0000","last_active":"2024-04-17T07:14:00.149082+0000","last_peered":"2024-04-17T07:14:00.149082+0000","last_clean":"2024-04-17T07:14:00.149082+0000","last_became_active":"2024-04-17T07:13:57.195074+0000","last_became_peered":"2024-04-17T07:13:57.195074+0000","last_unstale":"2024-04-17T07:14:00.149082+0000","last_undegraded":"2024-04-17T07:14:00.149082+0000","last_fullsized":"2024-04-17T07:14:00.149082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:53.288607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148521+0000","last_change":"2024-04-17T07:13:58.243294+0000","last_active":"2024-04-17T07:14:00.148521+0000","last_peered":"2024-04-17T07:14:00.148521+0000","last_clean":"2024-04-17T07:14:00.148521+0000","last_became_active":"2024-04-17T07:13:58.241931+0000","last_became_peered":"2024-04-17T07:13:58.241931+0000","last_unstale":"2024-04-17T07:14:00.148521+0000","last_undegraded":"2024-04-17T07:14:00.148521+0000","last_fullsized":"2024-04-17T07:14:00.148521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:49.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229051+0000","last_change":"2024-04-17T07:13:57.190224+0000","last_active":"2024-04-17T07:13:59.229051+0000","last_peered":"2024-04-17T07:13:59.229051+0000","last_clean":"2024-04-17T07:13:59.229051+0000","last_became_active":"2024-04-17T07:13:57.189879+0000","last_became_peered":"2024-04-17T07:13:57.189879+0000","last_unstale":"2024-04-17T07:13:59.229051+0000","last_undegraded":"2024-04-17T07:13:59.229051+0000","last_fullsized":"2024-04-17T07:13:59.229051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226963+0000","last_change":"2024-04-17T07:13:58.235859+0000","last_active":"2024-04-17T07:13:59.226963+0000","last_peered":"2024-04-17T07:13:59.226963+0000","last_clean":"2024-04-17T07:13:59.226963+0000","last_became_active":"2024-04-17T07:13:58.235565+0000","last_became_peered":"2024-04-17T07:13:58.235565+0000","last_unstale":"2024-04-17T07:13:59.226963+0000","last_undegraded":"2024-04-17T07:13:59.226963+0000","last_fullsized":"2024-04-17T07:13:59.226963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226241+0000","last_change":"2024-04-17T07:13:57.202465+0000","last_active":"2024-04-17T07:13:59.226241+0000","last_peered":"2024-04-17T07:13:59.226241+0000","last_clean":"2024-04-17T07:13:59.226241+0000","last_became_active":"2024-04-17T07:13:57.202105+0000","last_became_peered":"2024-04-17T07:13:57.202105+0000","last_unstale":"2024-04-17T07:13:59.226241+0000","last_undegraded":"2024-04-17T07:13:59.226241+0000","last_fullsized":"2024-04-17T07:13:59.226241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225347+0000","last_change":"2024-04-17T07:13:58.228208+0000","last_active":"2024-04-17T07:13:59.225347+0000","last_peered":"2024-04-17T07:13:59.225347+0000","last_clean":"2024-04-17T07:13:59.225347+0000","last_became_active":"2024-04-17T07:13:58.227889+0000","last_became_peered":"2024-04-17T07:13:58.227889+0000","last_unstale":"2024-04-17T07:13:59.225347+0000","last_undegraded":"2024-04-17T07:13:59.225347+0000","last_fullsized":"2024-04-17T07:13:59.225347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.142947+0000","last_change":"2024-04-17T07:13:57.204927+0000","last_active":"2024-04-17T07:14:00.142947+0000","last_peered":"2024-04-17T07:14:00.142947+0000","last_clean":"2024-04-17T07:14:00.142947+0000","last_became_active":"2024-04-17T07:13:57.204546+0000","last_became_peered":"2024-04-17T07:13:57.204546+0000","last_unstale":"2024-04-17T07:14:00.142947+0000","last_undegraded":"2024-04-17T07:14:00.142947+0000","last_fullsized":"2024-04-17T07:14:00.142947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:22:01.161179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227143+0000","last_change":"2024-04-17T07:13:58.235736+0000","last_active":"2024-04-17T07:13:59.227143+0000","last_peered":"2024-04-17T07:13:59.227143+0000","last_clean":"2024-04-17T07:13:59.227143+0000","last_became_active":"2024-04-17T07:13:58.235432+0000","last_became_peered":"2024-04-17T07:13:58.235432+0000","last_unstale":"2024-04-17T07:13:59.227143+0000","last_undegraded":"2024-04-17T07:13:59.227143+0000","last_fullsized":"2024-04-17T07:13:59.227143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144456+0000","last_change":"2024-04-17T07:13:57.195110+0000","last_active":"2024-04-17T07:14:00.144456+0000","last_peered":"2024-04-17T07:14:00.144456+0000","last_clean":"2024-04-17T07:14:00.144456+0000","last_became_active":"2024-04-17T07:13:57.194788+0000","last_became_peered":"2024-04-17T07:13:57.194788+0000","last_unstale":"2024-04-17T07:14:00.144456+0000","last_undegraded":"2024-04-17T07:14:00.144456+0000","last_fullsized":"2024-04-17T07:14:00.144456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:01.392379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228617+0000","last_change":"2024-04-17T07:13:58.224809+0000","last_active":"2024-04-17T07:13:59.228617+0000","last_peered":"2024-04-17T07:13:59.228617+0000","last_clean":"2024-04-17T07:13:59.228617+0000","last_became_active":"2024-04-17T07:13:58.224516+0000","last_became_peered":"2024-04-17T07:13:58.224516+0000","last_unstale":"2024-04-17T07:13:59.228617+0000","last_undegraded":"2024-04-17T07:13:59.228617+0000","last_fullsized":"2024-04-17T07:13:59.228617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144137+0000","last_change":"2024-04-17T07:13:57.203108+0000","last_active":"2024-04-17T07:14:00.144137+0000","last_peered":"2024-04-17T07:14:00.144137+0000","last_clean":"2024-04-17T07:14:00.144137+0000","last_became_active":"2024-04-17T07:13:57.202802+0000","last_became_peered":"2024-04-17T07:13:57.202802+0000","last_unstale":"2024-04-17T07:14:00.144137+0000","last_undegraded":"2024-04-17T07:14:00.144137+0000","last_fullsized":"2024-04-17T07:14:00.144137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:54.647968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229091+0000","last_change":"2024-04-17T07:13:58.230644+0000","last_active":"2024-04-17T07:13:59.229091+0000","last_peered":"2024-04-17T07:13:59.229091+0000","last_clean":"2024-04-17T07:13:59.229091+0000","last_became_active":"2024-04-17T07:13:58.229593+0000","last_became_peered":"2024-04-17T07:13:58.229593+0000","last_unstale":"2024-04-17T07:13:59.229091+0000","last_undegraded":"2024-04-17T07:13:59.229091+0000","last_fullsized":"2024-04-17T07:13:59.229091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:57.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218173+0000","last_change":"2024-04-17T07:13:57.200619+0000","last_active":"2024-04-17T07:13:59.218173+0000","last_peered":"2024-04-17T07:13:59.218173+0000","last_clean":"2024-04-17T07:13:59.218173+0000","last_became_active":"2024-04-17T07:13:57.200293+0000","last_became_peered":"2024-04-17T07:13:57.200293+0000","last_unstale":"2024-04-17T07:13:59.218173+0000","last_undegraded":"2024-04-17T07:13:59.218173+0000","last_fullsized":"2024-04-17T07:13:59.218173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229770+0000","last_change":"2024-04-17T07:13:58.233982+0000","last_active":"2024-04-17T07:13:59.229770+0000","last_peered":"2024-04-17T07:13:59.229770+0000","last_clean":"2024-04-17T07:13:59.229770+0000","last_became_active":"2024-04-17T07:13:58.233394+0000","last_became_peered":"2024-04-17T07:13:58.233394+0000","last_unstale":"2024-04-17T07:13:59.229770+0000","last_undegraded":"2024-04-17T07:13:59.229770+0000","last_fullsized":"2024-04-17T07:13:59.229770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143798+0000","last_change":"2024-04-17T07:13:57.198237+0000","last_active":"2024-04-17T07:14:00.143798+0000","last_peered":"2024-04-17T07:14:00.143798+0000","last_clean":"2024-04-17T07:14:00.143798+0000","last_became_active":"2024-04-17T07:13:57.197870+0000","last_became_peered":"2024-04-17T07:13:57.197870+0000","last_unstale":"2024-04-17T07:14:00.143798+0000","last_undegraded":"2024-04-17T07:14:00.143798+0000","last_fullsized":"2024-04-17T07:14:00.143798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:48.074499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227126+0000","last_change":"2024-04-17T07:13:58.232896+0000","last_active":"2024-04-17T07:13:59.227126+0000","last_peered":"2024-04-17T07:13:59.227126+0000","last_clean":"2024-04-17T07:13:59.227126+0000","last_became_active":"2024-04-17T07:13:58.232155+0000","last_became_peered":"2024-04-17T07:13:58.232155+0000","last_unstale":"2024-04-17T07:13:59.227126+0000","last_undegraded":"2024-04-17T07:13:59.227126+0000","last_fullsized":"2024-04-17T07:13:59.227126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225865+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:13:59.225865+0000","last_peered":"2024-04-17T07:13:59.225865+0000","last_clean":"2024-04-17T07:13:59.225865+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:13:59.225865+0000","last_undegraded":"2024-04-17T07:13:59.225865+0000","last_fullsized":"2024-04-17T07:13:59.225865+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228095+0000","last_change":"2024-04-17T07:13:57.187149+0000","last_active":"2024-04-17T07:13:59.228095+0000","last_peered":"2024-04-17T07:13:59.228095+0000","last_clean":"2024-04-17T07:13:59.228095+0000","last_became_active":"2024-04-17T07:13:57.186832+0000","last_became_peered":"2024-04-17T07:13:57.186832+0000","last_unstale":"2024-04-17T07:13:59.228095+0000","last_undegraded":"2024-04-17T07:13:59.228095+0000","last_fullsized":"2024-04-17T07:13:59.228095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146188+0000","last_change":"2024-04-17T07:13:58.228567+0000","last_active":"2024-04-17T07:14:00.146188+0000","last_peered":"2024-04-17T07:14:00.146188+0000","last_clean":"2024-04-17T07:14:00.146188+0000","last_became_active":"2024-04-17T07:13:58.228064+0000","last_became_peered":"2024-04-17T07:13:58.228064+0000","last_unstale":"2024-04-17T07:14:00.146188+0000","last_undegraded":"2024-04-17T07:14:00.146188+0000","last_fullsized":"2024-04-17T07:14:00.146188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224678+0000","last_change":"2024-04-17T07:13:57.204847+0000","last_active":"2024-04-17T07:13:59.224678+0000","last_peered":"2024-04-17T07:13:59.224678+0000","last_clean":"2024-04-17T07:13:59.224678+0000","last_became_active":"2024-04-17T07:13:57.204534+0000","last_became_peered":"2024-04-17T07:13:57.204534+0000","last_unstale":"2024-04-17T07:13:59.224678+0000","last_undegraded":"2024-04-17T07:13:59.224678+0000","last_fullsized":"2024-04-17T07:13:59.224678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148252+0000","last_change":"2024-04-17T07:13:58.230682+0000","last_active":"2024-04-17T07:14:00.148252+0000","last_peered":"2024-04-17T07:14:00.148252+0000","last_clean":"2024-04-17T07:14:00.148252+0000","last_became_active":"2024-04-17T07:13:58.230315+0000","last_became_peered":"2024-04-17T07:13:58.230315+0000","last_unstale":"2024-04-17T07:14:00.148252+0000","last_undegraded":"2024-04-17T07:14:00.148252+0000","last_fullsized":"2024-04-17T07:14:00.148252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:03.539255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143397+0000","last_change":"2024-04-17T07:13:57.204481+0000","last_active":"2024-04-17T07:14:00.143397+0000","last_peered":"2024-04-17T07:14:00.143397+0000","last_clean":"2024-04-17T07:14:00.143397+0000","last_became_active":"2024-04-17T07:13:57.203848+0000","last_became_peered":"2024-04-17T07:13:57.203848+0000","last_unstale":"2024-04-17T07:14:00.143397+0000","last_undegraded":"2024-04-17T07:14:00.143397+0000","last_fullsized":"2024-04-17T07:14:00.143397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:11.312213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217520+0000","last_change":"2024-04-17T07:13:58.208486+0000","last_active":"2024-04-17T07:13:59.217520+0000","last_peered":"2024-04-17T07:13:59.217520+0000","last_clean":"2024-04-17T07:13:59.217520+0000","last_became_active":"2024-04-17T07:13:58.208043+0000","last_became_peered":"2024-04-17T07:13:58.208043+0000","last_unstale":"2024-04-17T07:13:59.217520+0000","last_undegraded":"2024-04-17T07:13:59.217520+0000","last_fullsized":"2024-04-17T07:13:59.217520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229248+0000","last_change":"2024-04-17T07:13:57.196519+0000","last_active":"2024-04-17T07:13:59.229248+0000","last_peered":"2024-04-17T07:13:59.229248+0000","last_clean":"2024-04-17T07:13:59.229248+0000","last_became_active":"2024-04-17T07:13:57.196148+0000","last_became_peered":"2024-04-17T07:13:57.196148+0000","last_unstale":"2024-04-17T07:13:59.229248+0000","last_undegraded":"2024-04-17T07:13:59.229248+0000","last_fullsized":"2024-04-17T07:13:59.229248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144037+0000","last_change":"2024-04-17T07:13:58.241381+0000","last_active":"2024-04-17T07:14:00.144037+0000","last_peered":"2024-04-17T07:14:00.144037+0000","last_clean":"2024-04-17T07:14:00.144037+0000","last_became_active":"2024-04-17T07:13:58.240707+0000","last_became_peered":"2024-04-17T07:13:58.240707+0000","last_unstale":"2024-04-17T07:14:00.144037+0000","last_undegraded":"2024-04-17T07:14:00.144037+0000","last_fullsized":"2024-04-17T07:14:00.144037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:39.705592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217512+0000","last_change":"2024-04-17T07:13:57.196549+0000","last_active":"2024-04-17T07:13:59.217512+0000","last_peered":"2024-04-17T07:13:59.217512+0000","last_clean":"2024-04-17T07:13:59.217512+0000","last_became_active":"2024-04-17T07:13:57.196096+0000","last_became_peered":"2024-04-17T07:13:57.196096+0000","last_unstale":"2024-04-17T07:13:59.217512+0000","last_undegraded":"2024-04-17T07:13:59.217512+0000","last_fullsized":"2024-04-17T07:13:59.217512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231381+0000","last_change":"2024-04-17T07:13:58.218704+0000","last_active":"2024-04-17T07:13:59.231381+0000","last_peered":"2024-04-17T07:13:59.231381+0000","last_clean":"2024-04-17T07:13:59.231381+0000","last_became_active":"2024-04-17T07:13:58.218383+0000","last_became_peered":"2024-04-17T07:13:58.218383+0000","last_unstale":"2024-04-17T07:13:59.231381+0000","last_undegraded":"2024-04-17T07:13:59.231381+0000","last_fullsized":"2024-04-17T07:13:59.231381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228625+0000","last_change":"2024-04-17T07:13:57.199856+0000","last_active":"2024-04-17T07:13:59.228625+0000","last_peered":"2024-04-17T07:13:59.228625+0000","last_clean":"2024-04-17T07:13:59.228625+0000","last_became_active":"2024-04-17T07:13:57.199550+0000","last_became_peered":"2024-04-17T07:13:57.199550+0000","last_unstale":"2024-04-17T07:13:59.228625+0000","last_undegraded":"2024-04-17T07:13:59.228625+0000","last_fullsized":"2024-04-17T07:13:59.228625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:36.294837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217992+0000","last_change":"2024-04-17T07:13:58.213558+0000","last_active":"2024-04-17T07:13:59.217992+0000","last_peered":"2024-04-17T07:13:59.217992+0000","last_clean":"2024-04-17T07:13:59.217992+0000","last_became_active":"2024-04-17T07:13:58.213237+0000","last_became_peered":"2024-04-17T07:13:58.213237+0000","last_unstale":"2024-04-17T07:13:59.217992+0000","last_undegraded":"2024-04-17T07:13:59.217992+0000","last_fullsized":"2024-04-17T07:13:59.217992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.222842+0000","last_change":"2024-04-17T07:13:57.198259+0000","last_active":"2024-04-17T07:13:59.222842+0000","last_peered":"2024-04-17T07:13:59.222842+0000","last_clean":"2024-04-17T07:13:59.222842+0000","last_became_active":"2024-04-17T07:13:57.197982+0000","last_became_peered":"2024-04-17T07:13:57.197982+0000","last_unstale":"2024-04-17T07:13:59.222842+0000","last_undegraded":"2024-04-17T07:13:59.222842+0000","last_fullsized":"2024-04-17T07:13:59.222842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147642+0000","last_change":"2024-04-17T07:13:58.224013+0000","last_active":"2024-04-17T07:14:00.147642+0000","last_peered":"2024-04-17T07:14:00.147642+0000","last_clean":"2024-04-17T07:14:00.147642+0000","last_became_active":"2024-04-17T07:13:58.223705+0000","last_became_peered":"2024-04-17T07:13:58.223705+0000","last_unstale":"2024-04-17T07:14:00.147642+0000","last_undegraded":"2024-04-17T07:14:00.147642+0000","last_fullsized":"2024-04-17T07:14:00.147642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226587+0000","last_change":"2024-04-17T07:13:57.200987+0000","last_active":"2024-04-17T07:13:59.226587+0000","last_peered":"2024-04-17T07:13:59.226587+0000","last_clean":"2024-04-17T07:13:59.226587+0000","last_became_active":"2024-04-17T07:13:57.200681+0000","last_became_peered":"2024-04-17T07:13:57.200681+0000","last_unstale":"2024-04-17T07:13:59.226587+0000","last_undegraded":"2024-04-17T07:13:59.226587+0000","last_fullsized":"2024-04-17T07:13:59.226587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149119+0000","last_change":"2024-04-17T07:13:58.234490+0000","last_active":"2024-04-17T07:14:00.149119+0000","last_peered":"2024-04-17T07:14:00.149119+0000","last_clean":"2024-04-17T07:14:00.149119+0000","last_became_active":"2024-04-17T07:13:58.234250+0000","last_became_peered":"2024-04-17T07:13:58.234250+0000","last_unstale":"2024-04-17T07:14:00.149119+0000","last_undegraded":"2024-04-17T07:14:00.149119+0000","last_fullsized":"2024-04-17T07:14:00.149119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:17.137422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224627+0000","last_change":"2024-04-17T07:13:57.199555+0000","last_active":"2024-04-17T07:13:59.224627+0000","last_peered":"2024-04-17T07:13:59.224627+0000","last_clean":"2024-04-17T07:13:59.224627+0000","last_became_active":"2024-04-17T07:13:57.199295+0000","last_became_peered":"2024-04-17T07:13:57.199295+0000","last_unstale":"2024-04-17T07:13:59.224627+0000","last_undegraded":"2024-04-17T07:13:59.224627+0000","last_fullsized":"2024-04-17T07:13:59.224627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143813+0000","last_change":"2024-04-17T07:13:58.222076+0000","last_active":"2024-04-17T07:14:00.143813+0000","last_peered":"2024-04-17T07:14:00.143813+0000","last_clean":"2024-04-17T07:14:00.143813+0000","last_became_active":"2024-04-17T07:13:58.219377+0000","last_became_peered":"2024-04-17T07:13:58.219377+0000","last_unstale":"2024-04-17T07:14:00.143813+0000","last_undegraded":"2024-04-17T07:14:00.143813+0000","last_fullsized":"2024-04-17T07:14:00.143813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:37.495822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217874+0000","last_change":"2024-04-17T07:13:57.203197+0000","last_active":"2024-04-17T07:13:59.217874+0000","last_peered":"2024-04-17T07:13:59.217874+0000","last_clean":"2024-04-17T07:13:59.217874+0000","last_became_active":"2024-04-17T07:13:57.201296+0000","last_became_peered":"2024-04-17T07:13:57.201296+0000","last_unstale":"2024-04-17T07:13:59.217874+0000","last_undegraded":"2024-04-17T07:13:59.217874+0000","last_fullsized":"2024-04-17T07:13:59.217874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144614+0000","last_change":"2024-04-17T07:13:58.220538+0000","last_active":"2024-04-17T07:14:00.144614+0000","last_peered":"2024-04-17T07:14:00.144614+0000","last_clean":"2024-04-17T07:14:00.144614+0000","last_became_active":"2024-04-17T07:13:58.220281+0000","last_became_peered":"2024-04-17T07:13:58.220281+0000","last_unstale":"2024-04-17T07:14:00.144614+0000","last_undegraded":"2024-04-17T07:14:00.144614+0000","last_fullsized":"2024-04-17T07:14:00.144614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217905+0000","last_change":"2024-04-17T07:13:57.202287+0000","last_active":"2024-04-17T07:13:59.217905+0000","last_peered":"2024-04-17T07:13:59.217905+0000","last_clean":"2024-04-17T07:13:59.217905+0000","last_became_active":"2024-04-17T07:13:57.202007+0000","last_became_peered":"2024-04-17T07:13:57.202007+0000","last_unstale":"2024-04-17T07:13:59.217905+0000","last_undegraded":"2024-04-17T07:13:59.217905+0000","last_fullsized":"2024-04-17T07:13:59.217905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146417+0000","last_change":"2024-04-17T07:13:58.243714+0000","last_active":"2024-04-17T07:14:00.146417+0000","last_peered":"2024-04-17T07:14:00.146417+0000","last_clean":"2024-04-17T07:14:00.146417+0000","last_became_active":"2024-04-17T07:13:58.243199+0000","last_became_peered":"2024-04-17T07:13:58.243199+0000","last_unstale":"2024-04-17T07:14:00.146417+0000","last_undegraded":"2024-04-17T07:14:00.146417+0000","last_fullsized":"2024-04-17T07:14:00.146417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227303+0000","last_change":"2024-04-17T07:13:57.194783+0000","last_active":"2024-04-17T07:13:59.227303+0000","last_peered":"2024-04-17T07:13:59.227303+0000","last_clean":"2024-04-17T07:13:59.227303+0000","last_became_active":"2024-04-17T07:13:57.194418+0000","last_became_peered":"2024-04-17T07:13:57.194418+0000","last_unstale":"2024-04-17T07:13:59.227303+0000","last_undegraded":"2024-04-17T07:13:59.227303+0000","last_fullsized":"2024-04-17T07:13:59.227303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.173128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225654+0000","last_change":"2024-04-17T07:13:58.224933+0000","last_active":"2024-04-17T07:13:59.225654+0000","last_peered":"2024-04-17T07:13:59.225654+0000","last_clean":"2024-04-17T07:13:59.225654+0000","last_became_active":"2024-04-17T07:13:58.224635+0000","last_became_peered":"2024-04-17T07:13:58.224635+0000","last_unstale":"2024-04-17T07:13:59.225654+0000","last_undegraded":"2024-04-17T07:13:59.225654+0000","last_fullsized":"2024-04-17T07:13:59.225654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231353+0000","last_change":"2024-04-17T07:13:58.224558+0000","last_active":"2024-04-17T07:13:59.231353+0000","last_peered":"2024-04-17T07:13:59.231353+0000","last_clean":"2024-04-17T07:13:59.231353+0000","last_became_active":"2024-04-17T07:13:58.224291+0000","last_became_peered":"2024-04-17T07:13:58.224291+0000","last_unstale":"2024-04-17T07:13:59.231353+0000","last_undegraded":"2024-04-17T07:13:59.231353+0000","last_fullsized":"2024-04-17T07:13:59.231353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226442+0000","last_change":"2024-04-17T07:13:57.197610+0000","last_active":"2024-04-17T07:13:59.226442+0000","last_peered":"2024-04-17T07:13:59.226442+0000","last_clean":"2024-04-17T07:13:59.226442+0000","last_became_active":"2024-04-17T07:13:57.197297+0000","last_became_peered":"2024-04-17T07:13:57.197297+0000","last_unstale":"2024-04-17T07:13:59.226442+0000","last_undegraded":"2024-04-17T07:13:59.226442+0000","last_fullsized":"2024-04-17T07:13:59.226442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229831+0000","last_change":"2024-04-17T07:13:58.227513+0000","last_active":"2024-04-17T07:13:59.229831+0000","last_peered":"2024-04-17T07:13:59.229831+0000","last_clean":"2024-04-17T07:13:59.229831+0000","last_became_active":"2024-04-17T07:13:58.227195+0000","last_became_peered":"2024-04-17T07:13:58.227195+0000","last_unstale":"2024-04-17T07:13:59.229831+0000","last_undegraded":"2024-04-17T07:13:59.229831+0000","last_fullsized":"2024-04-17T07:13:59.229831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228330+0000","last_change":"2024-04-17T07:13:57.203401+0000","last_active":"2024-04-17T07:13:59.228330+0000","last_peered":"2024-04-17T07:13:59.228330+0000","last_clean":"2024-04-17T07:13:59.228330+0000","last_became_active":"2024-04-17T07:13:57.202900+0000","last_became_peered":"2024-04-17T07:13:57.202900+0000","last_unstale":"2024-04-17T07:13:59.228330+0000","last_undegraded":"2024-04-17T07:13:59.228330+0000","last_fullsized":"2024-04-17T07:13:59.228330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:16.876460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148975+0000","last_change":"2024-04-17T07:13:58.245486+0000","last_active":"2024-04-17T07:14:00.148975+0000","last_peered":"2024-04-17T07:14:00.148975+0000","last_clean":"2024-04-17T07:14:00.148975+0000","last_became_active":"2024-04-17T07:13:58.245180+0000","last_became_peered":"2024-04-17T07:13:58.245180+0000","last_unstale":"2024-04-17T07:14:00.148975+0000","last_undegraded":"2024-04-17T07:14:00.148975+0000","last_fullsized":"2024-04-17T07:14:00.148975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:42.645551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147085+0000","last_change":"2024-04-17T07:13:57.197244+0000","last_active":"2024-04-17T07:14:00.147085+0000","last_peered":"2024-04-17T07:14:00.147085+0000","last_clean":"2024-04-17T07:14:00.147085+0000","last_became_active":"2024-04-17T07:13:57.196651+0000","last_became_peered":"2024-04-17T07:13:57.196651+0000","last_unstale":"2024-04-17T07:14:00.147085+0000","last_undegraded":"2024-04-17T07:14:00.147085+0000","last_fullsized":"2024-04-17T07:14:00.147085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146961+0000","last_change":"2024-04-17T07:13:58.221178+0000","last_active":"2024-04-17T07:14:00.146961+0000","last_peered":"2024-04-17T07:14:00.146961+0000","last_clean":"2024-04-17T07:14:00.146961+0000","last_became_active":"2024-04-17T07:13:58.220895+0000","last_became_peered":"2024-04-17T07:13:58.220895+0000","last_unstale":"2024-04-17T07:14:00.146961+0000","last_undegraded":"2024-04-17T07:14:00.146961+0000","last_fullsized":"2024-04-17T07:14:00.146961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149303+0000","last_change":"2024-04-17T07:13:57.193073+0000","last_active":"2024-04-17T07:14:00.149303+0000","last_peered":"2024-04-17T07:14:00.149303+0000","last_clean":"2024-04-17T07:14:00.149303+0000","last_became_active":"2024-04-17T07:13:57.192203+0000","last_became_peered":"2024-04-17T07:13:57.192203+0000","last_unstale":"2024-04-17T07:14:00.149303+0000","last_undegraded":"2024-04-17T07:14:00.149303+0000","last_fullsized":"2024-04-17T07:14:00.149303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:24.305808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":669784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":78960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":74146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":669840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:14:02.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:14:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.005+0000 7f2110000640 1 Processor -- start 2024-04-17T07:14:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.005+0000 7f2110000640 1 -- start start 2024-04-17T07:14:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.006+0000 7f2110000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21080cd020 0x7f21080cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2110000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806e020 0x7f210806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2110000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21080ceb00 con 0x7f210806f670 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2110000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21080cede0 con 0x7f21080cd020 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2110000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21080cf0c0 con 0x7f210806e020 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f210d600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f210806f670 0x7f210806fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49486/0 (socket says 172.21.15.67:49486) 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f210d600640 1 -- 172.21.15.67:0/2586675182 learned_addr learned my addr 172.21.15.67:0/2586675182 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f210cc00640 1 --2- 172.21.15.67:0/2586675182 >> v2:172.21.15.67:3300/0 conn(0x7f21080cd020 0x7f21080cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f210e000640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806e020 0x7f210806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2107e00640 1 -- 172.21.15.67:0/2586675182 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995153303 0 0) 0x7f21080ceb00 con 0x7f210806f670 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.007+0000 7f2107e00640 1 -- 172.21.15.67:0/2586675182 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20f0003520 con 0x7f210806f670 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.008+0000 7f210cc00640 1 -- 172.21.15.67:0/2586675182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806e020 msgr2=0x7f210806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.008+0000 7f210cc00640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806e020 0x7f210806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.008+0000 7f210cc00640 1 -- 172.21.15.67:0/2586675182 >> v1:172.21.15.67:6789/0 conn(0x7f210806f670 legacy=0x7f210806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.008+0000 7f210cc00640 1 -- 172.21.15.67:0/2586675182 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21080cf3a0 con 0x7f21080cd020 2024-04-17T07:14:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.009+0000 7f210cc00640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21080cd020 0x7f21080cd480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20f40097a0 tx=0x7f20f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccc61e7e92f6ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.009+0000 7f2107e00640 1 -- 172.21.15.67:0/2586675182 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4035020 con 0x7f21080cd020 2024-04-17T07:14:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.009+0000 7f2107e00640 1 -- 172.21.15.67:0/2586675182 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20f4031830 con 0x7f21080cd020 2024-04-17T07:14:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.009+0000 7f2107e00640 1 -- 172.21.15.67:0/2586675182 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4031c00 con 0x7f21080cd020 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 -- 172.21.15.67:0/2586675182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21080cd020 msgr2=0x7f21080cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21080cd020 0x7f21080cd480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f20f40097a0 tx=0x7f20f402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 -- 172.21.15.67:0/2586675182 shutdown_connections 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806e020 0x7f210806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 --2- 172.21.15.67:0/2586675182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21080cd020 0x7f21080cd480 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 -- 172.21.15.67:0/2586675182 >> 172.21.15.67:0/2586675182 conn(0x7f210806bf50 msgr2=0x7f21080b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 -- 172.21.15.67:0/2586675182 shutdown_connections 2024-04-17T07:14:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.010+0000 7f2110000640 1 -- 172.21.15.67:0/2586675182 wait complete. 2024-04-17T07:14:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.011+0000 7f2110000640 1 Processor -- start 2024-04-17T07:14:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.012+0000 7f2110000640 1 -- start start 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.012+0000 7f2110000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f2110000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 0x7f21080b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f2110000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 0x7f21080b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f2110000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21080ce8a0 con 0x7f210806e020 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f2110000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f21080ce720 con 0x7f21080cd020 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f2110000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f210815cb00 con 0x7f210806f670 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57028/0 (socket says 172.21.15.67:57028) 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 -- 172.21.15.67:0/2666621397 learned_addr learned my addr 172.21.15.67:0/2666621397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210cc00640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 0x7f21080b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210e000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 0x7f21080b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 -- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 msgr2=0x7f21080b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 0x7f21080b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 -- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 msgr2=0x7f21080b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 0x7f21080b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210d600640 1 -- 172.21.15.67:0/2666621397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2108156860 con 0x7f210806e020 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210e000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 0x7f21080b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.013+0000 7f210cc00640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 0x7f21080b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f210d600640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c390 tx=0x7f20fc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc017070 con 0x7f210806e020 2024-04-17T07:14:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20fc005ce0 con 0x7f210806e020 2024-04-17T07:14:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0072f0 con 0x7f210806e020 2024-04-17T07:14:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21080c8640 con 0x7f210806e020 2024-04-17T07:14:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.014+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21080c8c00 con 0x7f210806e020 2024-04-17T07:14:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.015+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2108068b90 con 0x7f210806e020 2024-04-17T07:14:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.016+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20fc01a070 con 0x7f210806e020 2024-04-17T07:14:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.017+0000 7f2105600640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 0x7f20e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.017+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f20fc08aed0 con 0x7f210806e020 2024-04-17T07:14:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.018+0000 7f210cc00640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 0x7f20e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.019+0000 7f210cc00640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 0x7f20e806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20f4036000 tx=0x7f20f4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.022+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f20fc059370 con 0x7f210806e020 2024-04-17T07:14:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.229+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21080b09d0 con 0x7f20e806c8f0 2024-04-17T07:14:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.251+0000 7f2105600640 1 -- 172.21.15.67:0/2666621397 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7f21080b09d0 con 0x7f20e806c8f0 2024-04-17T07:14:03.253 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:03.256 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:14:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 msgr2=0x7f20e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 0x7f20e806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f20f4036000 tx=0x7f20f4008030 comp rx=0 tx=0).stop 2024-04-17T07:14:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 msgr2=0x7f2108156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c390 tx=0x7f20fc00c850 comp rx=0 tx=0).stop 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 shutdown_connections 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21080cd020 0x7f21080b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f20e806c8f0 0x7f20e806edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f210806f670 0x7f21080b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 --2- 172.21.15.67:0/2666621397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e020 0x7f2108156320 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 >> 172.21.15.67:0/2666621397 conn(0x7f210806bf50 msgr2=0x7f21080aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 shutdown_connections 2024-04-17T07:14:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.258+0000 7f2110000640 1 -- 172.21.15.67:0/2666621397 wait complete. 2024-04-17T07:14:03.280 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-17T07:14:01.353848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223320,"kb_used_data":2960,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751400,"statfs":{"total":773094113280,"available":772865433600,"internally_reserved":0,"allocated":3031040,"data_stored":1803628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.194087"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228829+0000","last_change":"2024-04-17T07:13:58.227207+0000","last_active":"2024-04-17T07:13:59.228829+0000","last_peered":"2024-04-17T07:13:59.228829+0000","last_clean":"2024-04-17T07:13:59.228829+0000","last_became_active":"2024-04-17T07:13:58.223412+0000","last_became_peered":"2024-04-17T07:13:58.223412+0000","last_unstale":"2024-04-17T07:13:59.228829+0000","last_undegraded":"2024-04-17T07:13:59.228829+0000","last_fullsized":"2024-04-17T07:13:59.228829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217686+0000","last_change":"2024-04-17T07:13:58.238529+0000","last_active":"2024-04-17T07:13:59.217686+0000","last_peered":"2024-04-17T07:13:59.217686+0000","last_clean":"2024-04-17T07:13:59.217686+0000","last_became_active":"2024-04-17T07:13:58.238155+0000","last_became_peered":"2024-04-17T07:13:58.238155+0000","last_unstale":"2024-04-17T07:13:59.217686+0000","last_undegraded":"2024-04-17T07:13:59.217686+0000","last_fullsized":"2024-04-17T07:13:59.217686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148756+0000","last_change":"2024-04-17T07:13:58.242232+0000","last_active":"2024-04-17T07:14:00.148756+0000","last_peered":"2024-04-17T07:14:00.148756+0000","last_clean":"2024-04-17T07:14:00.148756+0000","last_became_active":"2024-04-17T07:13:58.241891+0000","last_became_peered":"2024-04-17T07:13:58.241891+0000","last_unstale":"2024-04-17T07:14:00.148756+0000","last_undegraded":"2024-04-17T07:14:00.148756+0000","last_fullsized":"2024-04-17T07:14:00.148756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:17.659180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229570+0000","last_change":"2024-04-17T07:13:58.234270+0000","last_active":"2024-04-17T07:13:59.229570+0000","last_peered":"2024-04-17T07:13:59.229570+0000","last_clean":"2024-04-17T07:13:59.229570+0000","last_became_active":"2024-04-17T07:13:58.233725+0000","last_became_peered":"2024-04-17T07:13:58.233725+0000","last_unstale":"2024-04-17T07:13:59.229570+0000","last_undegraded":"2024-04-17T07:13:59.229570+0000","last_fullsized":"2024-04-17T07:13:59.229570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226745+0000","last_change":"2024-04-17T07:13:58.215673+0000","last_active":"2024-04-17T07:13:59.226745+0000","last_peered":"2024-04-17T07:13:59.226745+0000","last_clean":"2024-04-17T07:13:59.226745+0000","last_became_active":"2024-04-17T07:13:58.215371+0000","last_became_peered":"2024-04-17T07:13:58.215371+0000","last_unstale":"2024-04-17T07:13:59.226745+0000","last_undegraded":"2024-04-17T07:13:59.226745+0000","last_fullsized":"2024-04-17T07:13:59.226745+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228445+0000","last_change":"2024-04-17T07:13:58.240332+0000","last_active":"2024-04-17T07:13:59.228445+0000","last_peered":"2024-04-17T07:13:59.228445+0000","last_clean":"2024-04-17T07:13:59.228445+0000","last_became_active":"2024-04-17T07:13:58.239852+0000","last_became_peered":"2024-04-17T07:13:58.239852+0000","last_unstale":"2024-04-17T07:13:59.228445+0000","last_undegraded":"2024-04-17T07:13:59.228445+0000","last_fullsized":"2024-04-17T07:13:59.228445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226714+0000","last_change":"2024-04-17T07:13:58.238930+0000","last_active":"2024-04-17T07:13:59.226714+0000","last_peered":"2024-04-17T07:13:59.226714+0000","last_clean":"2024-04-17T07:13:59.226714+0000","last_became_active":"2024-04-17T07:13:58.238663+0000","last_became_peered":"2024-04-17T07:13:58.238663+0000","last_unstale":"2024-04-17T07:13:59.226714+0000","last_undegraded":"2024-04-17T07:13:59.226714+0000","last_fullsized":"2024-04-17T07:13:59.226714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144325+0000","last_change":"2024-04-17T07:13:58.227713+0000","last_active":"2024-04-17T07:14:00.144325+0000","last_peered":"2024-04-17T07:14:00.144325+0000","last_clean":"2024-04-17T07:14:00.144325+0000","last_became_active":"2024-04-17T07:13:58.227352+0000","last_became_peered":"2024-04-17T07:13:58.227352+0000","last_unstale":"2024-04-17T07:14:00.144325+0000","last_undegraded":"2024-04-17T07:14:00.144325+0000","last_fullsized":"2024-04-17T07:14:00.144325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228952+0000","last_change":"2024-04-17T07:13:58.233019+0000","last_active":"2024-04-17T07:13:59.228952+0000","last_peered":"2024-04-17T07:13:59.228952+0000","last_clean":"2024-04-17T07:13:59.228952+0000","last_became_active":"2024-04-17T07:13:58.230661+0000","last_became_peered":"2024-04-17T07:13:58.230661+0000","last_unstale":"2024-04-17T07:13:59.228952+0000","last_undegraded":"2024-04-17T07:13:59.228952+0000","last_fullsized":"2024-04-17T07:13:59.228952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:33.970758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226560+0000","last_change":"2024-04-17T07:13:58.225623+0000","last_active":"2024-04-17T07:13:59.226560+0000","last_peered":"2024-04-17T07:13:59.226560+0000","last_clean":"2024-04-17T07:13:59.226560+0000","last_became_active":"2024-04-17T07:13:58.225345+0000","last_became_peered":"2024-04-17T07:13:58.225345+0000","last_unstale":"2024-04-17T07:13:59.226560+0000","last_undegraded":"2024-04-17T07:13:59.226560+0000","last_fullsized":"2024-04-17T07:13:59.226560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226194+0000","last_change":"2024-04-17T07:13:58.226592+0000","last_active":"2024-04-17T07:13:59.226194+0000","last_peered":"2024-04-17T07:13:59.226194+0000","last_clean":"2024-04-17T07:13:59.226194+0000","last_became_active":"2024-04-17T07:13:58.226241+0000","last_became_peered":"2024-04-17T07:13:58.226241+0000","last_unstale":"2024-04-17T07:13:59.226194+0000","last_undegraded":"2024-04-17T07:13:59.226194+0000","last_fullsized":"2024-04-17T07:13:59.226194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144301+0000","last_change":"2024-04-17T07:13:58.216920+0000","last_active":"2024-04-17T07:14:00.144301+0000","last_peered":"2024-04-17T07:14:00.144301+0000","last_clean":"2024-04-17T07:14:00.144301+0000","last_became_active":"2024-04-17T07:13:58.216635+0000","last_became_peered":"2024-04-17T07:13:58.216635+0000","last_unstale":"2024-04-17T07:14:00.144301+0000","last_undegraded":"2024-04-17T07:14:00.144301+0000","last_fullsized":"2024-04-17T07:14:00.144301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:51.668433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146122+0000","last_change":"2024-04-17T07:13:58.225422+0000","last_active":"2024-04-17T07:14:00.146122+0000","last_peered":"2024-04-17T07:14:00.146122+0000","last_clean":"2024-04-17T07:14:00.146122+0000","last_became_active":"2024-04-17T07:13:58.224906+0000","last_became_peered":"2024-04-17T07:13:58.224906+0000","last_unstale":"2024-04-17T07:14:00.146122+0000","last_undegraded":"2024-04-17T07:14:00.146122+0000","last_fullsized":"2024-04-17T07:14:00.146122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147718+0000","last_change":"2024-04-17T07:13:58.233409+0000","last_active":"2024-04-17T07:14:00.147718+0000","last_peered":"2024-04-17T07:14:00.147718+0000","last_clean":"2024-04-17T07:14:00.147718+0000","last_became_active":"2024-04-17T07:13:58.233182+0000","last_became_peered":"2024-04-17T07:13:58.233182+0000","last_unstale":"2024-04-17T07:14:00.147718+0000","last_undegraded":"2024-04-17T07:14:00.147718+0000","last_fullsized":"2024-04-17T07:14:00.147718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:18.125280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217646+0000","last_change":"2024-04-17T07:13:58.215689+0000","last_active":"2024-04-17T07:13:59.217646+0000","last_peered":"2024-04-17T07:13:59.217646+0000","last_clean":"2024-04-17T07:13:59.217646+0000","last_became_active":"2024-04-17T07:13:58.215054+0000","last_became_peered":"2024-04-17T07:13:58.215054+0000","last_unstale":"2024-04-17T07:13:59.217646+0000","last_undegraded":"2024-04-17T07:13:59.217646+0000","last_fullsized":"2024-04-17T07:13:59.217646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227948+0000","last_change":"2024-04-17T07:13:58.220483+0000","last_active":"2024-04-17T07:13:59.227948+0000","last_peered":"2024-04-17T07:13:59.227948+0000","last_clean":"2024-04-17T07:13:59.227948+0000","last_became_active":"2024-04-17T07:13:58.220205+0000","last_became_peered":"2024-04-17T07:13:58.220205+0000","last_unstale":"2024-04-17T07:13:59.227948+0000","last_undegraded":"2024-04-17T07:13:59.227948+0000","last_fullsized":"2024-04-17T07:13:59.227948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231680+0000","last_change":"2024-04-17T07:13:58.223188+0000","last_active":"2024-04-17T07:13:59.231680+0000","last_peered":"2024-04-17T07:13:59.231680+0000","last_clean":"2024-04-17T07:13:59.231680+0000","last_became_active":"2024-04-17T07:13:58.221699+0000","last_became_peered":"2024-04-17T07:13:58.221699+0000","last_unstale":"2024-04-17T07:13:59.231680+0000","last_undegraded":"2024-04-17T07:13:59.231680+0000","last_fullsized":"2024-04-17T07:13:59.231680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144144+0000","last_change":"2024-04-17T07:13:58.220308+0000","last_active":"2024-04-17T07:14:00.144144+0000","last_peered":"2024-04-17T07:14:00.144144+0000","last_clean":"2024-04-17T07:14:00.144144+0000","last_became_active":"2024-04-17T07:13:58.219976+0000","last_became_peered":"2024-04-17T07:13:58.219976+0000","last_unstale":"2024-04-17T07:14:00.144144+0000","last_undegraded":"2024-04-17T07:14:00.144144+0000","last_fullsized":"2024-04-17T07:14:00.144144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:11.629034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228541+0000","last_change":"2024-04-17T07:13:58.223620+0000","last_active":"2024-04-17T07:13:59.228541+0000","last_peered":"2024-04-17T07:13:59.228541+0000","last_clean":"2024-04-17T07:13:59.228541+0000","last_became_active":"2024-04-17T07:13:58.222125+0000","last_became_peered":"2024-04-17T07:13:58.222125+0000","last_unstale":"2024-04-17T07:13:59.228541+0000","last_undegraded":"2024-04-17T07:13:59.228541+0000","last_fullsized":"2024-04-17T07:13:59.228541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227555+0000","last_change":"2024-04-17T07:13:58.225907+0000","last_active":"2024-04-17T07:13:59.227555+0000","last_peered":"2024-04-17T07:13:59.227555+0000","last_clean":"2024-04-17T07:13:59.227555+0000","last_became_active":"2024-04-17T07:13:58.224993+0000","last_became_peered":"2024-04-17T07:13:58.224993+0000","last_unstale":"2024-04-17T07:13:59.227555+0000","last_undegraded":"2024-04-17T07:13:59.227555+0000","last_fullsized":"2024-04-17T07:13:59.227555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148489+0000","last_change":"2024-04-17T07:13:58.236523+0000","last_active":"2024-04-17T07:14:00.148489+0000","last_peered":"2024-04-17T07:14:00.148489+0000","last_clean":"2024-04-17T07:14:00.148489+0000","last_became_active":"2024-04-17T07:13:58.236164+0000","last_became_peered":"2024-04-17T07:13:58.236164+0000","last_unstale":"2024-04-17T07:14:00.148489+0000","last_undegraded":"2024-04-17T07:14:00.148489+0000","last_fullsized":"2024-04-17T07:14:00.148489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:33.648495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228822+0000","last_change":"2024-04-17T07:13:58.222632+0000","last_active":"2024-04-17T07:13:59.228822+0000","last_peered":"2024-04-17T07:13:59.228822+0000","last_clean":"2024-04-17T07:13:59.228822+0000","last_became_active":"2024-04-17T07:13:58.221666+0000","last_became_peered":"2024-04-17T07:13:58.221666+0000","last_unstale":"2024-04-17T07:13:59.228822+0000","last_undegraded":"2024-04-17T07:13:59.228822+0000","last_fullsized":"2024-04-17T07:13:59.228822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145740+0000","last_change":"2024-04-17T07:13:58.218642+0000","last_active":"2024-04-17T07:14:00.145740+0000","last_peered":"2024-04-17T07:14:00.145740+0000","last_clean":"2024-04-17T07:14:00.145740+0000","last_became_active":"2024-04-17T07:13:58.218235+0000","last_became_peered":"2024-04-17T07:13:58.218235+0000","last_unstale":"2024-04-17T07:14:00.145740+0000","last_undegraded":"2024-04-17T07:14:00.145740+0000","last_fullsized":"2024-04-17T07:14:00.145740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228851+0000","last_change":"2024-04-17T07:13:58.232074+0000","last_active":"2024-04-17T07:13:59.228851+0000","last_peered":"2024-04-17T07:13:59.228851+0000","last_clean":"2024-04-17T07:13:59.228851+0000","last_became_active":"2024-04-17T07:13:58.231716+0000","last_became_peered":"2024-04-17T07:13:58.231716+0000","last_unstale":"2024-04-17T07:13:59.228851+0000","last_undegraded":"2024-04-17T07:13:59.228851+0000","last_fullsized":"2024-04-17T07:13:59.228851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147203+0000","last_change":"2024-04-17T07:13:58.230039+0000","last_active":"2024-04-17T07:14:00.147203+0000","last_peered":"2024-04-17T07:14:00.147203+0000","last_clean":"2024-04-17T07:14:00.147203+0000","last_became_active":"2024-04-17T07:13:58.224374+0000","last_became_peered":"2024-04-17T07:13:58.224374+0000","last_unstale":"2024-04-17T07:14:00.147203+0000","last_undegraded":"2024-04-17T07:14:00.147203+0000","last_fullsized":"2024-04-17T07:14:00.147203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226940+0000","last_change":"2024-04-17T07:13:58.227485+0000","last_active":"2024-04-17T07:13:59.226940+0000","last_peered":"2024-04-17T07:13:59.226940+0000","last_clean":"2024-04-17T07:13:59.226940+0000","last_became_active":"2024-04-17T07:13:58.227182+0000","last_became_peered":"2024-04-17T07:13:58.227182+0000","last_unstale":"2024-04-17T07:13:59.226940+0000","last_undegraded":"2024-04-17T07:13:59.226940+0000","last_fullsized":"2024-04-17T07:13:59.226940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217943+0000","last_change":"2024-04-17T07:13:58.222243+0000","last_active":"2024-04-17T07:13:59.217943+0000","last_peered":"2024-04-17T07:13:59.217943+0000","last_clean":"2024-04-17T07:13:59.217943+0000","last_became_active":"2024-04-17T07:13:58.221925+0000","last_became_peered":"2024-04-17T07:13:58.221925+0000","last_unstale":"2024-04-17T07:13:59.217943+0000","last_undegraded":"2024-04-17T07:13:59.217943+0000","last_fullsized":"2024-04-17T07:13:59.217943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146781+0000","last_change":"2024-04-17T07:13:58.226680+0000","last_active":"2024-04-17T07:14:00.146781+0000","last_peered":"2024-04-17T07:14:00.146781+0000","last_clean":"2024-04-17T07:14:00.146781+0000","last_became_active":"2024-04-17T07:13:58.226283+0000","last_became_peered":"2024-04-17T07:13:58.226283+0000","last_unstale":"2024-04-17T07:14:00.146781+0000","last_undegraded":"2024-04-17T07:14:00.146781+0000","last_fullsized":"2024-04-17T07:14:00.146781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148779+0000","last_change":"2024-04-17T07:13:58.242478+0000","last_active":"2024-04-17T07:14:00.148779+0000","last_peered":"2024-04-17T07:14:00.148779+0000","last_clean":"2024-04-17T07:14:00.148779+0000","last_became_active":"2024-04-17T07:13:58.241968+0000","last_became_peered":"2024-04-17T07:13:58.241968+0000","last_unstale":"2024-04-17T07:14:00.148779+0000","last_undegraded":"2024-04-17T07:14:00.148779+0000","last_fullsized":"2024-04-17T07:14:00.148779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:29:46.240069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229245+0000","last_change":"2024-04-17T07:13:58.230046+0000","last_active":"2024-04-17T07:13:59.229245+0000","last_peered":"2024-04-17T07:13:59.229245+0000","last_clean":"2024-04-17T07:13:59.229245+0000","last_became_active":"2024-04-17T07:13:58.229744+0000","last_became_peered":"2024-04-17T07:13:58.229744+0000","last_unstale":"2024-04-17T07:13:59.229245+0000","last_undegraded":"2024-04-17T07:13:59.229245+0000","last_fullsized":"2024-04-17T07:13:59.229245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143075+0000","last_change":"2024-04-17T07:13:58.219601+0000","last_active":"2024-04-17T07:14:00.143075+0000","last_peered":"2024-04-17T07:14:00.143075+0000","last_clean":"2024-04-17T07:14:00.143075+0000","last_became_active":"2024-04-17T07:13:58.219208+0000","last_became_peered":"2024-04-17T07:13:58.219208+0000","last_unstale":"2024-04-17T07:14:00.143075+0000","last_undegraded":"2024-04-17T07:14:00.143075+0000","last_fullsized":"2024-04-17T07:14:00.143075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:06.342021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225081+0000","last_change":"2024-04-17T07:13:58.230063+0000","last_active":"2024-04-17T07:13:59.225081+0000","last_peered":"2024-04-17T07:13:59.225081+0000","last_clean":"2024-04-17T07:13:59.225081+0000","last_became_active":"2024-04-17T07:13:58.229129+0000","last_became_peered":"2024-04-17T07:13:58.229129+0000","last_unstale":"2024-04-17T07:13:59.225081+0000","last_undegraded":"2024-04-17T07:13:59.225081+0000","last_fullsized":"2024-04-17T07:13:59.225081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:59.849640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143839+0000","last_change":"2024-04-17T07:13:58.208959+0000","last_active":"2024-04-17T07:14:00.143839+0000","last_peered":"2024-04-17T07:14:00.143839+0000","last_clean":"2024-04-17T07:14:00.143839+0000","last_became_active":"2024-04-17T07:13:58.208200+0000","last_became_peered":"2024-04-17T07:13:58.208200+0000","last_unstale":"2024-04-17T07:14:00.143839+0000","last_undegraded":"2024-04-17T07:14:00.143839+0000","last_fullsized":"2024-04-17T07:14:00.143839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:14.893023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147150+0000","last_change":"2024-04-17T07:13:57.193193+0000","last_active":"2024-04-17T07:14:00.147150+0000","last_peered":"2024-04-17T07:14:00.147150+0000","last_clean":"2024-04-17T07:14:00.147150+0000","last_became_active":"2024-04-17T07:13:57.192931+0000","last_became_peered":"2024-04-17T07:13:57.192931+0000","last_unstale":"2024-04-17T07:14:00.147150+0000","last_undegraded":"2024-04-17T07:14:00.147150+0000","last_fullsized":"2024-04-17T07:14:00.147150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148893+0000","last_change":"2024-04-17T07:13:58.244126+0000","last_active":"2024-04-17T07:14:00.148893+0000","last_peered":"2024-04-17T07:14:00.148893+0000","last_clean":"2024-04-17T07:14:00.148893+0000","last_became_active":"2024-04-17T07:13:58.243846+0000","last_became_peered":"2024-04-17T07:13:58.243846+0000","last_unstale":"2024-04-17T07:14:00.148893+0000","last_undegraded":"2024-04-17T07:14:00.148893+0000","last_fullsized":"2024-04-17T07:14:00.148893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:20.038754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225363+0000","last_change":"2024-04-17T07:13:57.198746+0000","last_active":"2024-04-17T07:13:59.225363+0000","last_peered":"2024-04-17T07:13:59.225363+0000","last_clean":"2024-04-17T07:13:59.225363+0000","last_became_active":"2024-04-17T07:13:57.195870+0000","last_became_peered":"2024-04-17T07:13:57.195870+0000","last_unstale":"2024-04-17T07:13:59.225363+0000","last_undegraded":"2024-04-17T07:13:59.225363+0000","last_fullsized":"2024-04-17T07:13:59.225363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145653+0000","last_change":"2024-04-17T07:13:57.201813+0000","last_active":"2024-04-17T07:14:00.145653+0000","last_peered":"2024-04-17T07:14:00.145653+0000","last_clean":"2024-04-17T07:14:00.145653+0000","last_became_active":"2024-04-17T07:13:57.201530+0000","last_became_peered":"2024-04-17T07:13:57.201530+0000","last_unstale":"2024-04-17T07:14:00.145653+0000","last_undegraded":"2024-04-17T07:14:00.145653+0000","last_fullsized":"2024-04-17T07:14:00.145653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144391+0000","last_change":"2024-04-17T07:13:58.235592+0000","last_active":"2024-04-17T07:14:00.144391+0000","last_peered":"2024-04-17T07:14:00.144391+0000","last_clean":"2024-04-17T07:14:00.144391+0000","last_became_active":"2024-04-17T07:13:58.235287+0000","last_became_peered":"2024-04-17T07:13:58.235287+0000","last_unstale":"2024-04-17T07:14:00.144391+0000","last_undegraded":"2024-04-17T07:14:00.144391+0000","last_fullsized":"2024-04-17T07:14:00.144391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228264+0000","last_change":"2024-04-17T07:13:57.183240+0000","last_active":"2024-04-17T07:13:59.228264+0000","last_peered":"2024-04-17T07:13:59.228264+0000","last_clean":"2024-04-17T07:13:59.228264+0000","last_became_active":"2024-04-17T07:13:57.182859+0000","last_became_peered":"2024-04-17T07:13:57.182859+0000","last_unstale":"2024-04-17T07:13:59.228264+0000","last_undegraded":"2024-04-17T07:13:59.228264+0000","last_fullsized":"2024-04-17T07:13:59.228264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217429+0000","last_change":"2024-04-17T07:13:58.221113+0000","last_active":"2024-04-17T07:13:59.217429+0000","last_peered":"2024-04-17T07:13:59.217429+0000","last_clean":"2024-04-17T07:13:59.217429+0000","last_became_active":"2024-04-17T07:13:58.220807+0000","last_became_peered":"2024-04-17T07:13:58.220807+0000","last_unstale":"2024-04-17T07:13:59.217429+0000","last_undegraded":"2024-04-17T07:13:59.217429+0000","last_fullsized":"2024-04-17T07:13:59.217429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226316+0000","last_change":"2024-04-17T07:13:57.197641+0000","last_active":"2024-04-17T07:13:59.226316+0000","last_peered":"2024-04-17T07:13:59.226316+0000","last_clean":"2024-04-17T07:13:59.226316+0000","last_became_active":"2024-04-17T07:13:57.197394+0000","last_became_peered":"2024-04-17T07:13:57.197394+0000","last_unstale":"2024-04-17T07:13:59.226316+0000","last_undegraded":"2024-04-17T07:13:59.226316+0000","last_fullsized":"2024-04-17T07:13:59.226316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231265+0000","last_change":"2024-04-17T07:13:58.225914+0000","last_active":"2024-04-17T07:13:59.231265+0000","last_peered":"2024-04-17T07:13:59.231265+0000","last_clean":"2024-04-17T07:13:59.231265+0000","last_became_active":"2024-04-17T07:13:58.225649+0000","last_became_peered":"2024-04-17T07:13:58.225649+0000","last_unstale":"2024-04-17T07:13:59.231265+0000","last_undegraded":"2024-04-17T07:13:59.231265+0000","last_fullsized":"2024-04-17T07:13:59.231265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231330+0000","last_change":"2024-04-17T07:13:57.198101+0000","last_active":"2024-04-17T07:13:59.231330+0000","last_peered":"2024-04-17T07:13:59.231330+0000","last_clean":"2024-04-17T07:13:59.231330+0000","last_became_active":"2024-04-17T07:13:57.196771+0000","last_became_peered":"2024-04-17T07:13:57.196771+0000","last_unstale":"2024-04-17T07:13:59.231330+0000","last_undegraded":"2024-04-17T07:13:59.231330+0000","last_fullsized":"2024-04-17T07:13:59.231330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149355+0000","last_change":"2024-04-17T07:13:58.244824+0000","last_active":"2024-04-17T07:14:00.149355+0000","last_peered":"2024-04-17T07:14:00.149355+0000","last_clean":"2024-04-17T07:14:00.149355+0000","last_became_active":"2024-04-17T07:13:58.243368+0000","last_became_peered":"2024-04-17T07:13:58.243368+0000","last_unstale":"2024-04-17T07:14:00.149355+0000","last_undegraded":"2024-04-17T07:14:00.149355+0000","last_fullsized":"2024-04-17T07:14:00.149355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:43.836722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218062+0000","last_change":"2024-04-17T07:13:57.198519+0000","last_active":"2024-04-17T07:13:59.218062+0000","last_peered":"2024-04-17T07:13:59.218062+0000","last_clean":"2024-04-17T07:13:59.218062+0000","last_became_active":"2024-04-17T07:13:57.197225+0000","last_became_peered":"2024-04-17T07:13:57.197225+0000","last_unstale":"2024-04-17T07:13:59.218062+0000","last_undegraded":"2024-04-17T07:13:59.218062+0000","last_fullsized":"2024-04-17T07:13:59.218062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144417+0000","last_change":"2024-04-17T07:13:58.221225+0000","last_active":"2024-04-17T07:14:00.144417+0000","last_peered":"2024-04-17T07:14:00.144417+0000","last_clean":"2024-04-17T07:14:00.144417+0000","last_became_active":"2024-04-17T07:13:58.220917+0000","last_became_peered":"2024-04-17T07:13:58.220917+0000","last_unstale":"2024-04-17T07:14:00.144417+0000","last_undegraded":"2024-04-17T07:14:00.144417+0000","last_fullsized":"2024-04-17T07:14:00.144417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:04.188468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226310+0000","last_change":"2024-04-17T07:13:57.200719+0000","last_active":"2024-04-17T07:13:59.226310+0000","last_peered":"2024-04-17T07:13:59.226310+0000","last_clean":"2024-04-17T07:13:59.226310+0000","last_became_active":"2024-04-17T07:13:57.200404+0000","last_became_peered":"2024-04-17T07:13:57.200404+0000","last_unstale":"2024-04-17T07:13:59.226310+0000","last_undegraded":"2024-04-17T07:13:59.226310+0000","last_fullsized":"2024-04-17T07:13:59.226310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226654+0000","last_change":"2024-04-17T07:13:58.231037+0000","last_active":"2024-04-17T07:13:59.226654+0000","last_peered":"2024-04-17T07:13:59.226654+0000","last_clean":"2024-04-17T07:13:59.226654+0000","last_became_active":"2024-04-17T07:13:58.229754+0000","last_became_peered":"2024-04-17T07:13:58.229754+0000","last_unstale":"2024-04-17T07:13:59.226654+0000","last_undegraded":"2024-04-17T07:13:59.226654+0000","last_fullsized":"2024-04-17T07:13:59.226654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149082+0000","last_change":"2024-04-17T07:13:57.196583+0000","last_active":"2024-04-17T07:14:00.149082+0000","last_peered":"2024-04-17T07:14:00.149082+0000","last_clean":"2024-04-17T07:14:00.149082+0000","last_became_active":"2024-04-17T07:13:57.195074+0000","last_became_peered":"2024-04-17T07:13:57.195074+0000","last_unstale":"2024-04-17T07:14:00.149082+0000","last_undegraded":"2024-04-17T07:14:00.149082+0000","last_fullsized":"2024-04-17T07:14:00.149082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:53.288607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148521+0000","last_change":"2024-04-17T07:13:58.243294+0000","last_active":"2024-04-17T07:14:00.148521+0000","last_peered":"2024-04-17T07:14:00.148521+0000","last_clean":"2024-04-17T07:14:00.148521+0000","last_became_active":"2024-04-17T07:13:58.241931+0000","last_became_peered":"2024-04-17T07:13:58.241931+0000","last_unstale":"2024-04-17T07:14:00.148521+0000","last_undegraded":"2024-04-17T07:14:00.148521+0000","last_fullsized":"2024-04-17T07:14:00.148521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:49.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229051+0000","last_change":"2024-04-17T07:13:57.190224+0000","last_active":"2024-04-17T07:13:59.229051+0000","last_peered":"2024-04-17T07:13:59.229051+0000","last_clean":"2024-04-17T07:13:59.229051+0000","last_became_active":"2024-04-17T07:13:57.189879+0000","last_became_peered":"2024-04-17T07:13:57.189879+0000","last_unstale":"2024-04-17T07:13:59.229051+0000","last_undegraded":"2024-04-17T07:13:59.229051+0000","last_fullsized":"2024-04-17T07:13:59.229051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226963+0000","last_change":"2024-04-17T07:13:58.235859+0000","last_active":"2024-04-17T07:13:59.226963+0000","last_peered":"2024-04-17T07:13:59.226963+0000","last_clean":"2024-04-17T07:13:59.226963+0000","last_became_active":"2024-04-17T07:13:58.235565+0000","last_became_peered":"2024-04-17T07:13:58.235565+0000","last_unstale":"2024-04-17T07:13:59.226963+0000","last_undegraded":"2024-04-17T07:13:59.226963+0000","last_fullsized":"2024-04-17T07:13:59.226963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226241+0000","last_change":"2024-04-17T07:13:57.202465+0000","last_active":"2024-04-17T07:13:59.226241+0000","last_peered":"2024-04-17T07:13:59.226241+0000","last_clean":"2024-04-17T07:13:59.226241+0000","last_became_active":"2024-04-17T07:13:57.202105+0000","last_became_peered":"2024-04-17T07:13:57.202105+0000","last_unstale":"2024-04-17T07:13:59.226241+0000","last_undegraded":"2024-04-17T07:13:59.226241+0000","last_fullsized":"2024-04-17T07:13:59.226241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225347+0000","last_change":"2024-04-17T07:13:58.228208+0000","last_active":"2024-04-17T07:13:59.225347+0000","last_peered":"2024-04-17T07:13:59.225347+0000","last_clean":"2024-04-17T07:13:59.225347+0000","last_became_active":"2024-04-17T07:13:58.227889+0000","last_became_peered":"2024-04-17T07:13:58.227889+0000","last_unstale":"2024-04-17T07:13:59.225347+0000","last_undegraded":"2024-04-17T07:13:59.225347+0000","last_fullsized":"2024-04-17T07:13:59.225347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.142947+0000","last_change":"2024-04-17T07:13:57.204927+0000","last_active":"2024-04-17T07:14:00.142947+0000","last_peered":"2024-04-17T07:14:00.142947+0000","last_clean":"2024-04-17T07:14:00.142947+0000","last_became_active":"2024-04-17T07:13:57.204546+0000","last_became_peered":"2024-04-17T07:13:57.204546+0000","last_unstale":"2024-04-17T07:14:00.142947+0000","last_undegraded":"2024-04-17T07:14:00.142947+0000","last_fullsized":"2024-04-17T07:14:00.142947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:22:01.161179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227143+0000","last_change":"2024-04-17T07:13:58.235736+0000","last_active":"2024-04-17T07:13:59.227143+0000","last_peered":"2024-04-17T07:13:59.227143+0000","last_clean":"2024-04-17T07:13:59.227143+0000","last_became_active":"2024-04-17T07:13:58.235432+0000","last_became_peered":"2024-04-17T07:13:58.235432+0000","last_unstale":"2024-04-17T07:13:59.227143+0000","last_undegraded":"2024-04-17T07:13:59.227143+0000","last_fullsized":"2024-04-17T07:13:59.227143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144456+0000","last_change":"2024-04-17T07:13:57.195110+0000","last_active":"2024-04-17T07:14:00.144456+0000","last_peered":"2024-04-17T07:14:00.144456+0000","last_clean":"2024-04-17T07:14:00.144456+0000","last_became_active":"2024-04-17T07:13:57.194788+0000","last_became_peered":"2024-04-17T07:13:57.194788+0000","last_unstale":"2024-04-17T07:14:00.144456+0000","last_undegraded":"2024-04-17T07:14:00.144456+0000","last_fullsized":"2024-04-17T07:14:00.144456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:01.392379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228617+0000","last_change":"2024-04-17T07:13:58.224809+0000","last_active":"2024-04-17T07:13:59.228617+0000","last_peered":"2024-04-17T07:13:59.228617+0000","last_clean":"2024-04-17T07:13:59.228617+0000","last_became_active":"2024-04-17T07:13:58.224516+0000","last_became_peered":"2024-04-17T07:13:58.224516+0000","last_unstale":"2024-04-17T07:13:59.228617+0000","last_undegraded":"2024-04-17T07:13:59.228617+0000","last_fullsized":"2024-04-17T07:13:59.228617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144137+0000","last_change":"2024-04-17T07:13:57.203108+0000","last_active":"2024-04-17T07:14:00.144137+0000","last_peered":"2024-04-17T07:14:00.144137+0000","last_clean":"2024-04-17T07:14:00.144137+0000","last_became_active":"2024-04-17T07:13:57.202802+0000","last_became_peered":"2024-04-17T07:13:57.202802+0000","last_unstale":"2024-04-17T07:14:00.144137+0000","last_undegraded":"2024-04-17T07:14:00.144137+0000","last_fullsized":"2024-04-17T07:14:00.144137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:54.647968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229091+0000","last_change":"2024-04-17T07:13:58.230644+0000","last_active":"2024-04-17T07:13:59.229091+0000","last_peered":"2024-04-17T07:13:59.229091+0000","last_clean":"2024-04-17T07:13:59.229091+0000","last_became_active":"2024-04-17T07:13:58.229593+0000","last_became_peered":"2024-04-17T07:13:58.229593+0000","last_unstale":"2024-04-17T07:13:59.229091+0000","last_undegraded":"2024-04-17T07:13:59.229091+0000","last_fullsized":"2024-04-17T07:13:59.229091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:57.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218173+0000","last_change":"2024-04-17T07:13:57.200619+0000","last_active":"2024-04-17T07:13:59.218173+0000","last_peered":"2024-04-17T07:13:59.218173+0000","last_clean":"2024-04-17T07:13:59.218173+0000","last_became_active":"2024-04-17T07:13:57.200293+0000","last_became_peered":"2024-04-17T07:13:57.200293+0000","last_unstale":"2024-04-17T07:13:59.218173+0000","last_undegraded":"2024-04-17T07:13:59.218173+0000","last_fullsized":"2024-04-17T07:13:59.218173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229770+0000","last_change":"2024-04-17T07:13:58.233982+0000","last_active":"2024-04-17T07:13:59.229770+0000","last_peered":"2024-04-17T07:13:59.229770+0000","last_clean":"2024-04-17T07:13:59.229770+0000","last_became_active":"2024-04-17T07:13:58.233394+0000","last_became_peered":"2024-04-17T07:13:58.233394+0000","last_unstale":"2024-04-17T07:13:59.229770+0000","last_undegraded":"2024-04-17T07:13:59.229770+0000","last_fullsized":"2024-04-17T07:13:59.229770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143798+0000","last_change":"2024-04-17T07:13:57.198237+0000","last_active":"2024-04-17T07:14:00.143798+0000","last_peered":"2024-04-17T07:14:00.143798+0000","last_clean":"2024-04-17T07:14:00.143798+0000","last_became_active":"2024-04-17T07:13:57.197870+0000","last_became_peered":"2024-04-17T07:13:57.197870+0000","last_unstale":"2024-04-17T07:14:00.143798+0000","last_undegraded":"2024-04-17T07:14:00.143798+0000","last_fullsized":"2024-04-17T07:14:00.143798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:48.074499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227126+0000","last_change":"2024-04-17T07:13:58.232896+0000","last_active":"2024-04-17T07:13:59.227126+0000","last_peered":"2024-04-17T07:13:59.227126+0000","last_clean":"2024-04-17T07:13:59.227126+0000","last_became_active":"2024-04-17T07:13:58.232155+0000","last_became_peered":"2024-04-17T07:13:58.232155+0000","last_unstale":"2024-04-17T07:13:59.227126+0000","last_undegraded":"2024-04-17T07:13:59.227126+0000","last_fullsized":"2024-04-17T07:13:59.227126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225865+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:13:59.225865+0000","last_peered":"2024-04-17T07:13:59.225865+0000","last_clean":"2024-04-17T07:13:59.225865+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:13:59.225865+0000","last_undegraded":"2024-04-17T07:13:59.225865+0000","last_fullsized":"2024-04-17T07:13:59.225865+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228095+0000","last_change":"2024-04-17T07:13:57.187149+0000","last_active":"2024-04-17T07:13:59.228095+0000","last_peered":"2024-04-17T07:13:59.228095+0000","last_clean":"2024-04-17T07:13:59.228095+0000","last_became_active":"2024-04-17T07:13:57.186832+0000","last_became_peered":"2024-04-17T07:13:57.186832+0000","last_unstale":"2024-04-17T07:13:59.228095+0000","last_undegraded":"2024-04-17T07:13:59.228095+0000","last_fullsized":"2024-04-17T07:13:59.228095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146188+0000","last_change":"2024-04-17T07:13:58.228567+0000","last_active":"2024-04-17T07:14:00.146188+0000","last_peered":"2024-04-17T07:14:00.146188+0000","last_clean":"2024-04-17T07:14:00.146188+0000","last_became_active":"2024-04-17T07:13:58.228064+0000","last_became_peered":"2024-04-17T07:13:58.228064+0000","last_unstale":"2024-04-17T07:14:00.146188+0000","last_undegraded":"2024-04-17T07:14:00.146188+0000","last_fullsized":"2024-04-17T07:14:00.146188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224678+0000","last_change":"2024-04-17T07:13:57.204847+0000","last_active":"2024-04-17T07:13:59.224678+0000","last_peered":"2024-04-17T07:13:59.224678+0000","last_clean":"2024-04-17T07:13:59.224678+0000","last_became_active":"2024-04-17T07:13:57.204534+0000","last_became_peered":"2024-04-17T07:13:57.204534+0000","last_unstale":"2024-04-17T07:13:59.224678+0000","last_undegraded":"2024-04-17T07:13:59.224678+0000","last_fullsized":"2024-04-17T07:13:59.224678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148252+0000","last_change":"2024-04-17T07:13:58.230682+0000","last_active":"2024-04-17T07:14:00.148252+0000","last_peered":"2024-04-17T07:14:00.148252+0000","last_clean":"2024-04-17T07:14:00.148252+0000","last_became_active":"2024-04-17T07:13:58.230315+0000","last_became_peered":"2024-04-17T07:13:58.230315+0000","last_unstale":"2024-04-17T07:14:00.148252+0000","last_undegraded":"2024-04-17T07:14:00.148252+0000","last_fullsized":"2024-04-17T07:14:00.148252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:03.539255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143397+0000","last_change":"2024-04-17T07:13:57.204481+0000","last_active":"2024-04-17T07:14:00.143397+0000","last_peered":"2024-04-17T07:14:00.143397+0000","last_clean":"2024-04-17T07:14:00.143397+0000","last_became_active":"2024-04-17T07:13:57.203848+0000","last_became_peered":"2024-04-17T07:13:57.203848+0000","last_unstale":"2024-04-17T07:14:00.143397+0000","last_undegraded":"2024-04-17T07:14:00.143397+0000","last_fullsized":"2024-04-17T07:14:00.143397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:11.312213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217520+0000","last_change":"2024-04-17T07:13:58.208486+0000","last_active":"2024-04-17T07:13:59.217520+0000","last_peered":"2024-04-17T07:13:59.217520+0000","last_clean":"2024-04-17T07:13:59.217520+0000","last_became_active":"2024-04-17T07:13:58.208043+0000","last_became_peered":"2024-04-17T07:13:58.208043+0000","last_unstale":"2024-04-17T07:13:59.217520+0000","last_undegraded":"2024-04-17T07:13:59.217520+0000","last_fullsized":"2024-04-17T07:13:59.217520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229248+0000","last_change":"2024-04-17T07:13:57.196519+0000","last_active":"2024-04-17T07:13:59.229248+0000","last_peered":"2024-04-17T07:13:59.229248+0000","last_clean":"2024-04-17T07:13:59.229248+0000","last_became_active":"2024-04-17T07:13:57.196148+0000","last_became_peered":"2024-04-17T07:13:57.196148+0000","last_unstale":"2024-04-17T07:13:59.229248+0000","last_undegraded":"2024-04-17T07:13:59.229248+0000","last_fullsized":"2024-04-17T07:13:59.229248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144037+0000","last_change":"2024-04-17T07:13:58.241381+0000","last_active":"2024-04-17T07:14:00.144037+0000","last_peered":"2024-04-17T07:14:00.144037+0000","last_clean":"2024-04-17T07:14:00.144037+0000","last_became_active":"2024-04-17T07:13:58.240707+0000","last_became_peered":"2024-04-17T07:13:58.240707+0000","last_unstale":"2024-04-17T07:14:00.144037+0000","last_undegraded":"2024-04-17T07:14:00.144037+0000","last_fullsized":"2024-04-17T07:14:00.144037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:39.705592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217512+0000","last_change":"2024-04-17T07:13:57.196549+0000","last_active":"2024-04-17T07:13:59.217512+0000","last_peered":"2024-04-17T07:13:59.217512+0000","last_clean":"2024-04-17T07:13:59.217512+0000","last_became_active":"2024-04-17T07:13:57.196096+0000","last_became_peered":"2024-04-17T07:13:57.196096+0000","last_unstale":"2024-04-17T07:13:59.217512+0000","last_undegraded":"2024-04-17T07:13:59.217512+0000","last_fullsized":"2024-04-17T07:13:59.217512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231381+0000","last_change":"2024-04-17T07:13:58.218704+0000","last_active":"2024-04-17T07:13:59.231381+0000","last_peered":"2024-04-17T07:13:59.231381+0000","last_clean":"2024-04-17T07:13:59.231381+0000","last_became_active":"2024-04-17T07:13:58.218383+0000","last_became_peered":"2024-04-17T07:13:58.218383+0000","last_unstale":"2024-04-17T07:13:59.231381+0000","last_undegraded":"2024-04-17T07:13:59.231381+0000","last_fullsized":"2024-04-17T07:13:59.231381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228625+0000","last_change":"2024-04-17T07:13:57.199856+0000","last_active":"2024-04-17T07:13:59.228625+0000","last_peered":"2024-04-17T07:13:59.228625+0000","last_clean":"2024-04-17T07:13:59.228625+0000","last_became_active":"2024-04-17T07:13:57.199550+0000","last_became_peered":"2024-04-17T07:13:57.199550+0000","last_unstale":"2024-04-17T07:13:59.228625+0000","last_undegraded":"2024-04-17T07:13:59.228625+0000","last_fullsized":"2024-04-17T07:13:59.228625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:36.294837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217992+0000","last_change":"2024-04-17T07:13:58.213558+0000","last_active":"2024-04-17T07:13:59.217992+0000","last_peered":"2024-04-17T07:13:59.217992+0000","last_clean":"2024-04-17T07:13:59.217992+0000","last_became_active":"2024-04-17T07:13:58.213237+0000","last_became_peered":"2024-04-17T07:13:58.213237+0000","last_unstale":"2024-04-17T07:13:59.217992+0000","last_undegraded":"2024-04-17T07:13:59.217992+0000","last_fullsized":"2024-04-17T07:13:59.217992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.222842+0000","last_change":"2024-04-17T07:13:57.198259+0000","last_active":"2024-04-17T07:13:59.222842+0000","last_peered":"2024-04-17T07:13:59.222842+0000","last_clean":"2024-04-17T07:13:59.222842+0000","last_became_active":"2024-04-17T07:13:57.197982+0000","last_became_peered":"2024-04-17T07:13:57.197982+0000","last_unstale":"2024-04-17T07:13:59.222842+0000","last_undegraded":"2024-04-17T07:13:59.222842+0000","last_fullsized":"2024-04-17T07:13:59.222842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147642+0000","last_change":"2024-04-17T07:13:58.224013+0000","last_active":"2024-04-17T07:14:00.147642+0000","last_peered":"2024-04-17T07:14:00.147642+0000","last_clean":"2024-04-17T07:14:00.147642+0000","last_became_active":"2024-04-17T07:13:58.223705+0000","last_became_peered":"2024-04-17T07:13:58.223705+0000","last_unstale":"2024-04-17T07:14:00.147642+0000","last_undegraded":"2024-04-17T07:14:00.147642+0000","last_fullsized":"2024-04-17T07:14:00.147642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226587+0000","last_change":"2024-04-17T07:13:57.200987+0000","last_active":"2024-04-17T07:13:59.226587+0000","last_peered":"2024-04-17T07:13:59.226587+0000","last_clean":"2024-04-17T07:13:59.226587+0000","last_became_active":"2024-04-17T07:13:57.200681+0000","last_became_peered":"2024-04-17T07:13:57.200681+0000","last_unstale":"2024-04-17T07:13:59.226587+0000","last_undegraded":"2024-04-17T07:13:59.226587+0000","last_fullsized":"2024-04-17T07:13:59.226587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149119+0000","last_change":"2024-04-17T07:13:58.234490+0000","last_active":"2024-04-17T07:14:00.149119+0000","last_peered":"2024-04-17T07:14:00.149119+0000","last_clean":"2024-04-17T07:14:00.149119+0000","last_became_active":"2024-04-17T07:13:58.234250+0000","last_became_peered":"2024-04-17T07:13:58.234250+0000","last_unstale":"2024-04-17T07:14:00.149119+0000","last_undegraded":"2024-04-17T07:14:00.149119+0000","last_fullsized":"2024-04-17T07:14:00.149119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:17.137422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224627+0000","last_change":"2024-04-17T07:13:57.199555+0000","last_active":"2024-04-17T07:13:59.224627+0000","last_peered":"2024-04-17T07:13:59.224627+0000","last_clean":"2024-04-17T07:13:59.224627+0000","last_became_active":"2024-04-17T07:13:57.199295+0000","last_became_peered":"2024-04-17T07:13:57.199295+0000","last_unstale":"2024-04-17T07:13:59.224627+0000","last_undegraded":"2024-04-17T07:13:59.224627+0000","last_fullsized":"2024-04-17T07:13:59.224627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143813+0000","last_change":"2024-04-17T07:13:58.222076+0000","last_active":"2024-04-17T07:14:00.143813+0000","last_peered":"2024-04-17T07:14:00.143813+0000","last_clean":"2024-04-17T07:14:00.143813+0000","last_became_active":"2024-04-17T07:13:58.219377+0000","last_became_peered":"2024-04-17T07:13:58.219377+0000","last_unstale":"2024-04-17T07:14:00.143813+0000","last_undegraded":"2024-04-17T07:14:00.143813+0000","last_fullsized":"2024-04-17T07:14:00.143813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:37.495822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217874+0000","last_change":"2024-04-17T07:13:57.203197+0000","last_active":"2024-04-17T07:13:59.217874+0000","last_peered":"2024-04-17T07:13:59.217874+0000","last_clean":"2024-04-17T07:13:59.217874+0000","last_became_active":"2024-04-17T07:13:57.201296+0000","last_became_peered":"2024-04-17T07:13:57.201296+0000","last_unstale":"2024-04-17T07:13:59.217874+0000","last_undegraded":"2024-04-17T07:13:59.217874+0000","last_fullsized":"2024-04-17T07:13:59.217874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144614+0000","last_change":"2024-04-17T07:13:58.220538+0000","last_active":"2024-04-17T07:14:00.144614+0000","last_peered":"2024-04-17T07:14:00.144614+0000","last_clean":"2024-04-17T07:14:00.144614+0000","last_became_active":"2024-04-17T07:13:58.220281+0000","last_became_peered":"2024-04-17T07:13:58.220281+0000","last_unstale":"2024-04-17T07:14:00.144614+0000","last_undegraded":"2024-04-17T07:14:00.144614+0000","last_fullsized":"2024-04-17T07:14:00.144614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217905+0000","last_change":"2024-04-17T07:13:57.202287+0000","last_active":"2024-04-17T07:13:59.217905+0000","last_peered":"2024-04-17T07:13:59.217905+0000","last_clean":"2024-04-17T07:13:59.217905+0000","last_became_active":"2024-04-17T07:13:57.202007+0000","last_became_peered":"2024-04-17T07:13:57.202007+0000","last_unstale":"2024-04-17T07:13:59.217905+0000","last_undegraded":"2024-04-17T07:13:59.217905+0000","last_fullsized":"2024-04-17T07:13:59.217905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146417+0000","last_change":"2024-04-17T07:13:58.243714+0000","last_active":"2024-04-17T07:14:00.146417+0000","last_peered":"2024-04-17T07:14:00.146417+0000","last_clean":"2024-04-17T07:14:00.146417+0000","last_became_active":"2024-04-17T07:13:58.243199+0000","last_became_peered":"2024-04-17T07:13:58.243199+0000","last_unstale":"2024-04-17T07:14:00.146417+0000","last_undegraded":"2024-04-17T07:14:00.146417+0000","last_fullsized":"2024-04-17T07:14:00.146417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227303+0000","last_change":"2024-04-17T07:13:57.194783+0000","last_active":"2024-04-17T07:13:59.227303+0000","last_peered":"2024-04-17T07:13:59.227303+0000","last_clean":"2024-04-17T07:13:59.227303+0000","last_became_active":"2024-04-17T07:13:57.194418+0000","last_became_peered":"2024-04-17T07:13:57.194418+0000","last_unstale":"2024-04-17T07:13:59.227303+0000","last_undegraded":"2024-04-17T07:13:59.227303+0000","last_fullsized":"2024-04-17T07:13:59.227303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.173128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225654+0000","last_change":"2024-04-17T07:13:58.224933+0000","last_active":"2024-04-17T07:13:59.225654+0000","last_peered":"2024-04-17T07:13:59.225654+0000","last_clean":"2024-04-17T07:13:59.225654+0000","last_became_active":"2024-04-17T07:13:58.224635+0000","last_became_peered":"2024-04-17T07:13:58.224635+0000","last_unstale":"2024-04-17T07:13:59.225654+0000","last_undegraded":"2024-04-17T07:13:59.225654+0000","last_fullsized":"2024-04-17T07:13:59.225654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231353+0000","last_change":"2024-04-17T07:13:58.224558+0000","last_active":"2024-04-17T07:13:59.231353+0000","last_peered":"2024-04-17T07:13:59.231353+0000","last_clean":"2024-04-17T07:13:59.231353+0000","last_became_active":"2024-04-17T07:13:58.224291+0000","last_became_peered":"2024-04-17T07:13:58.224291+0000","last_unstale":"2024-04-17T07:13:59.231353+0000","last_undegraded":"2024-04-17T07:13:59.231353+0000","last_fullsized":"2024-04-17T07:13:59.231353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226442+0000","last_change":"2024-04-17T07:13:57.197610+0000","last_active":"2024-04-17T07:13:59.226442+0000","last_peered":"2024-04-17T07:13:59.226442+0000","last_clean":"2024-04-17T07:13:59.226442+0000","last_became_active":"2024-04-17T07:13:57.197297+0000","last_became_peered":"2024-04-17T07:13:57.197297+0000","last_unstale":"2024-04-17T07:13:59.226442+0000","last_undegraded":"2024-04-17T07:13:59.226442+0000","last_fullsized":"2024-04-17T07:13:59.226442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229831+0000","last_change":"2024-04-17T07:13:58.227513+0000","last_active":"2024-04-17T07:13:59.229831+0000","last_peered":"2024-04-17T07:13:59.229831+0000","last_clean":"2024-04-17T07:13:59.229831+0000","last_became_active":"2024-04-17T07:13:58.227195+0000","last_became_peered":"2024-04-17T07:13:58.227195+0000","last_unstale":"2024-04-17T07:13:59.229831+0000","last_undegraded":"2024-04-17T07:13:59.229831+0000","last_fullsized":"2024-04-17T07:13:59.229831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228330+0000","last_change":"2024-04-17T07:13:57.203401+0000","last_active":"2024-04-17T07:13:59.228330+0000","last_peered":"2024-04-17T07:13:59.228330+0000","last_clean":"2024-04-17T07:13:59.228330+0000","last_became_active":"2024-04-17T07:13:57.202900+0000","last_became_peered":"2024-04-17T07:13:57.202900+0000","last_unstale":"2024-04-17T07:13:59.228330+0000","last_undegraded":"2024-04-17T07:13:59.228330+0000","last_fullsized":"2024-04-17T07:13:59.228330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:16.876460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148975+0000","last_change":"2024-04-17T07:13:58.245486+0000","last_active":"2024-04-17T07:14:00.148975+0000","last_peered":"2024-04-17T07:14:00.148975+0000","last_clean":"2024-04-17T07:14:00.148975+0000","last_became_active":"2024-04-17T07:13:58.245180+0000","last_became_peered":"2024-04-17T07:13:58.245180+0000","last_unstale":"2024-04-17T07:14:00.148975+0000","last_undegraded":"2024-04-17T07:14:00.148975+0000","last_fullsized":"2024-04-17T07:14:00.148975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:42.645551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147085+0000","last_change":"2024-04-17T07:13:57.197244+0000","last_active":"2024-04-17T07:14:00.147085+0000","last_peered":"2024-04-17T07:14:00.147085+0000","last_clean":"2024-04-17T07:14:00.147085+0000","last_became_active":"2024-04-17T07:13:57.196651+0000","last_became_peered":"2024-04-17T07:13:57.196651+0000","last_unstale":"2024-04-17T07:14:00.147085+0000","last_undegraded":"2024-04-17T07:14:00.147085+0000","last_fullsized":"2024-04-17T07:14:00.147085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146961+0000","last_change":"2024-04-17T07:13:58.221178+0000","last_active":"2024-04-17T07:14:00.146961+0000","last_peered":"2024-04-17T07:14:00.146961+0000","last_clean":"2024-04-17T07:14:00.146961+0000","last_became_active":"2024-04-17T07:13:58.220895+0000","last_became_peered":"2024-04-17T07:13:58.220895+0000","last_unstale":"2024-04-17T07:14:00.146961+0000","last_undegraded":"2024-04-17T07:14:00.146961+0000","last_fullsized":"2024-04-17T07:14:00.146961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149303+0000","last_change":"2024-04-17T07:13:57.193073+0000","last_active":"2024-04-17T07:14:00.149303+0000","last_peered":"2024-04-17T07:14:00.149303+0000","last_clean":"2024-04-17T07:14:00.149303+0000","last_became_active":"2024-04-17T07:13:57.192203+0000","last_became_peered":"2024-04-17T07:13:57.192203+0000","last_unstale":"2024-04-17T07:14:00.149303+0000","last_undegraded":"2024-04-17T07:14:00.149303+0000","last_fullsized":"2024-04-17T07:14:00.149303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:24.305808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":669784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542147,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":78960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":74146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":669840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:14:03.284 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:14:03.284 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-17T07:14:03.284 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-17T07:14:03.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-17T07:14:03.285 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-17T07:14:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.550+0000 7ff53be00640 1 Processor -- start 2024-04-17T07:14:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.550+0000 7ff53be00640 1 -- start start 2024-04-17T07:14:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.551+0000 7ff53be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.551+0000 7ff53be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53c0d2d30 con 0x7ff53c0b6d00 2024-04-17T07:14:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.551+0000 7ff53be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53c0d3010 con 0x7ff53c15ec60 2024-04-17T07:14:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.551+0000 7ff53be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c0d49c0 con 0x7ff53c0cfae0 2024-04-17T07:14:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.552+0000 7ff53aa00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff53c15ec60 0x7ff53c0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39318/0 (socket says 172.21.15.67:39318) 2024-04-17T07:14:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.552+0000 7ff53aa00640 1 -- 172.21.15.67:0/3912645661 learned_addr learned my addr 172.21.15.67:0/3912645661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.552+0000 7ff53b400640 1 --2- 172.21.15.67:0/3912645661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.552+0000 7ff53a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff53c0b6d00 0x7ff53c0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50172/0 (socket says 172.21.15.67:50172) 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218727426 0 0) 0x7ff53c0d3010 con 0x7ff53c15ec60 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff520003520 con 0x7ff53c15ec60 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1775130341 0 0) 0x7ff53c0d2d30 con 0x7ff53c0b6d00 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff53c0d3010 con 0x7ff53c0b6d00 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff53b400640 1 -- 172.21.15.67:0/3912645661 >> v1:172.21.15.187:6789/0 conn(0x7ff53c15ec60 legacy=0x7ff53c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff53b400640 1 -- 172.21.15.67:0/3912645661 >> v1:172.21.15.67:6789/0 conn(0x7ff53c0b6d00 legacy=0x7ff53c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.553+0000 7ff53b400640 1 -- 172.21.15.67:0/3912645661 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c0d32f0 con 0x7ff53c0cfae0 2024-04-17T07:14:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.554+0000 7ff53b400640 1 --2- 172.21.15.67:0/3912645661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c169810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0d4b40 tx=0x7ff53402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8d0b9f2bb91128 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534034070 con 0x7ff53c0cfae0 2024-04-17T07:14:03.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff534030800 con 0x7ff53c0cfae0 2024-04-17T07:14:03.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff539600640 1 -- 172.21.15.67:0/3912645661 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534030bb0 con 0x7ff53c0cfae0 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff53be00640 1 -- 172.21.15.67:0/3912645661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 msgr2=0x7ff53c169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff53be00640 1 --2- 172.21.15.67:0/3912645661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c169810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0d4b40 tx=0x7ff53402b940 comp rx=0 tx=0).stop 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff53be00640 1 -- 172.21.15.67:0/3912645661 shutdown_connections 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.555+0000 7ff53be00640 1 --2- 172.21.15.67:0/3912645661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c169810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.556+0000 7ff53be00640 1 -- 172.21.15.67:0/3912645661 >> 172.21.15.67:0/3912645661 conn(0x7ff53c0d15f0 msgr2=0x7ff53c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.556+0000 7ff53be00640 1 -- 172.21.15.67:0/3912645661 shutdown_connections 2024-04-17T07:14:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.556+0000 7ff53be00640 1 -- 172.21.15.67:0/3912645661 wait complete. 2024-04-17T07:14:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.557+0000 7ff53be00640 1 Processor -- start 2024-04-17T07:14:03.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.558+0000 7ff53be00640 1 -- start start 2024-04-17T07:14:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53c0b6d00 0x7ff53c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c0b8b80 con 0x7ff53c15ec60 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c0d3660 con 0x7ff53c0b6d00 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff53c0d00c0 con 0x7ff53c0cfae0 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41788/0 (socket says 172.21.15.67:41788) 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53b400640 1 -- 172.21.15.67:0/2631136908 learned_addr learned my addr 172.21.15.67:0/2631136908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53aa00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53c0b6d00 0x7ff53c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.559+0000 7ff53a000640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 -- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 msgr2=0x7ff53c0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c0b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 -- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53c0b6d00 msgr2=0x7ff53c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53c0b6d00 0x7ff53c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 -- 172.21.15.67:0/2631136908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c1f4370 con 0x7ff53c15ec60 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53b400640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5340098c0 tx=0x7ff5340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534047070 con 0x7ff53c15ec60 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5340044c0 con 0x7ff53c15ec60 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534034050 con 0x7ff53c15ec60 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.560+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c0aba40 con 0x7ff53c15ec60 2024-04-17T07:14:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.561+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c0ac000 con 0x7ff53c15ec60 2024-04-17T07:14:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.562+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c068a10 con 0x7ff53c15ec60 2024-04-17T07:14:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.563+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff53404a070 con 0x7ff53c15ec60 2024-04-17T07:14:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.564+0000 7ff532a00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 0x7ff51c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.564+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff5340ad4a0 con 0x7ff53c15ec60 2024-04-17T07:14:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.564+0000 7ff53aa00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 0x7ff51c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.565+0000 7ff53aa00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 0x7ff51c06edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0012e0 tx=0x7ff52c001a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.569+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff53c1f4370 con 0x7ff53c15ec60 2024-04-17T07:14:03.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.828+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff53c0d56f0 con 0x7ff53c15ec60 2024-04-17T07:14:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.832+0000 7ff532a00640 1 -- 172.21.15.67:0/2631136908 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174143 (secure 0 0 0) 0x7ff53407b270 con 0x7ff53c15ec60 2024-04-17T07:14:03.835 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.836+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 msgr2=0x7ff51c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.836+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 0x7ff51c06edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff52c0012e0 tx=0x7ff52c001a30 comp rx=0 tx=0).stop 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 msgr2=0x7ff53c0b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5340098c0 tx=0x7ff5340046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 shutdown_connections 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff53c15ec60 0x7ff53c0b2ea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff51c06c8f0 0x7ff51c06edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff53c0cfae0 0x7ff53c0b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 --2- 172.21.15.67:0/2631136908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff53c0b6d00 0x7ff53c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 >> 172.21.15.67:0/2631136908 conn(0x7ff53c0d15f0 msgr2=0x7ff53c0d4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 shutdown_connections 2024-04-17T07:14:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:03.837+0000 7ff53be00640 1 -- 172.21.15.67:0/2631136908 wait complete. 2024-04-17T07:14:03.854 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":3291261531},{"type":"v1","addr":"172.21.15.187:6833","nonce":3291261531}]},"active_addr":"172.21.15.187:6833/3291261531","active_change":"2024-04-17T07:13:43.335893+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":2778463946}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":618455126}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":2322153279}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3485602680}]}]} 2024-04-17T07:14:03.857 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-17T07:14:03.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-17T07:14:03.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.071+0000 7f3797e00640 1 Processor -- start 2024-04-17T07:14:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.072+0000 7f3797e00640 1 -- start start 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.073+0000 7f3797e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f37900d2ab0 0x7f37900b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.073+0000 7f3797e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37900d3020 con 0x7f37900cf640 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.073+0000 7f3797e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37900d3300 con 0x7f379015ed10 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.073+0000 7f3797e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f37900d3c30 con 0x7f37900d2ab0 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.074+0000 7f3795400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f37900cf640 0x7f37900cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50178/0 (socket says 172.21.15.67:50178) 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.074+0000 7f3795400640 1 -- 172.21.15.67:0/778470068 learned_addr learned my addr 172.21.15.67:0/778470068 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.074+0000 7f3795e00640 1 --2- 172.21.15.67:0/778470068 >> v2:172.21.15.187:3300/0 conn(0x7f37900d2ab0 0x7f37900b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.074+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096486899 0 0) 0x7f37900d3020 con 0x7f37900cf640 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.074+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3778003520 con 0x7f37900cf640 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.075+0000 7f3795e00640 1 -- 172.21.15.67:0/778470068 >> v1:172.21.15.187:6789/0 conn(0x7f379015ed10 legacy=0x7f37900d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.075+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2792946341 0 0) 0x7f3778003520 con 0x7f37900cf640 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.075+0000 7f3795e00640 1 -- 172.21.15.67:0/778470068 >> v1:172.21.15.67:6789/0 conn(0x7f37900cf640 legacy=0x7f37900cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.075+0000 7f3795e00640 1 -- 172.21.15.67:0/778470068 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37900d35e0 con 0x7f37900d2ab0 2024-04-17T07:14:04.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.076+0000 7f3795e00640 1 --2- 172.21.15.67:0/778470068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b7b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37880097a0 tx=0x7f378802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d80ea20ff13f70c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.076+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37840019f0 con 0x7f37900d2ab0 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.076+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3788037070 con 0x7f37900d2ab0 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.076+0000 7f378fe00640 1 -- 172.21.15.67:0/778470068 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788031790 con 0x7f37900d2ab0 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 -- 172.21.15.67:0/778470068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 msgr2=0x7f37900b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 --2- 172.21.15.67:0/778470068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b7b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37880097a0 tx=0x7f378802b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 -- 172.21.15.67:0/778470068 shutdown_connections 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 --2- 172.21.15.67:0/778470068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b7b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 -- 172.21.15.67:0/778470068 >> 172.21.15.67:0/778470068 conn(0x7f379006cbd0 msgr2=0x7f379006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 -- 172.21.15.67:0/778470068 shutdown_connections 2024-04-17T07:14:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.077+0000 7f3797e00640 1 -- 172.21.15.67:0/778470068 wait complete. 2024-04-17T07:14:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.079+0000 7f3797e00640 1 Processor -- start 2024-04-17T07:14:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.079+0000 7f3797e00640 1 -- start start 2024-04-17T07:14:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f379015ed10 0x7f37900b9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f37900ba300 con 0x7f37900cf640 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f37900d25b0 con 0x7f37900d2ab0 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3797e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f37900ba480 con 0x7f379015ed10 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3795400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3795400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41796/0 (socket says 172.21.15.67:41796) 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3795400640 1 -- 172.21.15.67:0/3317679615 learned_addr learned my addr 172.21.15.67:0/3317679615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.080+0000 7f3794a00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 -- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f379015ed10 msgr2=0x7f37900b9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f379015ed10 0x7f37900b9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 -- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 msgr2=0x7f37900b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 -- 172.21.15.67:0/3317679615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379015a7a0 con 0x7f37900cf640 2024-04-17T07:14:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3794a00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3795400640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3784002000 tx=0x7f378400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784012400 con 0x7f37900cf640 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37840125a0 con 0x7f37900cf640 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784012970 con 0x7f37900cf640 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f379015aa10 con 0x7f37900cf640 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.081+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f379015afd0 con 0x7f37900cf640 2024-04-17T07:14:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.082+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3790068b90 con 0x7f37900cf640 2024-04-17T07:14:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.083+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3784012b10 con 0x7f37900cf640 2024-04-17T07:14:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.084+0000 7f378d600640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.084+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f378408c460 con 0x7f37900cf640 2024-04-17T07:14:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.085+0000 7f3794a00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.086+0000 7f3794a00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 0x7f377006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f37900b6c20 tx=0x7f377c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.089+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f378405a890 con 0x7f37900cf640 2024-04-17T07:14:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.293+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37900b40b0 con 0x7f37900cf640 2024-04-17T07:14:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.296+0000 7f378d600640 1 -- 172.21.15.67:0/3317679615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f378405a190 con 0x7f37900cf640 2024-04-17T07:14:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:04.296 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 msgr2=0x7f377006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 0x7f377006edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f37900b6c20 tx=0x7f377c009040 comp rx=0 tx=0).stop 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 msgr2=0x7f3790158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3784002000 tx=0x7f378400a430 comp rx=0 tx=0).stop 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 shutdown_connections 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f377006c8f0 0x7f377006edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f379015ed10 0x7f37900b9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37900d2ab0 0x7f37900b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 --2- 172.21.15.67:0/3317679615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37900cf640 0x7f3790158dc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 >> 172.21.15.67:0/3317679615 conn(0x7f379006cbd0 msgr2=0x7f37900ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 shutdown_connections 2024-04-17T07:14:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.299+0000 7f3797e00640 1 -- 172.21.15.67:0/3317679615 wait complete. 2024-04-17T07:14:04.320 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-17T07:14:04.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.535+0000 7ff621e00640 1 Processor -- start 2024-04-17T07:14:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.536+0000 7ff621e00640 1 -- start start 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c0b6d00 0x7ff61c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff61c0cf910 0x7ff61c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff61c0d2d30 con 0x7ff61c15ec60 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff61c0d49c0 con 0x7ff61c0cf910 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff61c0d3660 con 0x7ff61c0b6d00 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff621400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff61c0cf910 0x7ff61c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff620a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff61c15ec60 0x7ff61c0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50194/0 (socket says 172.21.15.67:50194) 2024-04-17T07:14:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff620a00640 1 -- 172.21.15.67:0/2734125963 learned_addr learned my addr 172.21.15.67:0/2734125963 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.537+0000 7ff61be00640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c0b6d00 0x7ff61c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.538+0000 7ff621400640 1 -- 172.21.15.67:0/2734125963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c0b6d00 msgr2=0x7ff61c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.538+0000 7ff621400640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c0b6d00 0x7ff61c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.538+0000 7ff621400640 1 -- 172.21.15.67:0/2734125963 >> v1:172.21.15.67:6789/0 conn(0x7ff61c15ec60 legacy=0x7ff61c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.538+0000 7ff621400640 1 -- 172.21.15.67:0/2734125963 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c0d3010 con 0x7ff61c0cf910 2024-04-17T07:14:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.539+0000 7ff621400640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c0b7b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786339a71330e301 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.539+0000 7ff61b400640 1 -- 172.21.15.67:0/2734125963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614035020 con 0x7ff61c0cf910 2024-04-17T07:14:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.539+0000 7ff61b400640 1 -- 172.21.15.67:0/2734125963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff614031810 con 0x7ff61c0cf910 2024-04-17T07:14:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.539+0000 7ff61b400640 1 -- 172.21.15.67:0/2734125963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614031be0 con 0x7ff61c0cf910 2024-04-17T07:14:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 -- 172.21.15.67:0/2734125963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 msgr2=0x7ff61c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c0b7b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 -- 172.21.15.67:0/2734125963 shutdown_connections 2024-04-17T07:14:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c0b7b70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 --2- 172.21.15.67:0/2734125963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c0b6d00 0x7ff61c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 -- 172.21.15.67:0/2734125963 >> 172.21.15.67:0/2734125963 conn(0x7ff61c0d15f0 msgr2=0x7ff61c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 -- 172.21.15.67:0/2734125963 shutdown_connections 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.540+0000 7ff621e00640 1 -- 172.21.15.67:0/2734125963 wait complete. 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.542+0000 7ff621e00640 1 Processor -- start 2024-04-17T07:14:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.542+0000 7ff621e00640 1 -- start start 2024-04-17T07:14:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 0x7ff61c1537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 0x7ff61c1e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c15f040 con 0x7ff61c0cf910 2024-04-17T07:14:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff61c0cfdc0 con 0x7ff61c0b6d00 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff61c0d00c0 con 0x7ff61c15ec60 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff61be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff61be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41816/0 (socket says 172.21.15.67:41816) 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff61be00640 1 -- 172.21.15.67:0/1419032635 learned_addr learned my addr 172.21.15.67:0/1419032635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff621400640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 0x7ff61c1e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.543+0000 7ff620a00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 0x7ff61c1537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 -- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 msgr2=0x7ff61c1e9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 0x7ff61c1e9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 -- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 msgr2=0x7ff61c1537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 0x7ff61c1537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 -- 172.21.15.67:0/1419032635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61c1593d0 con 0x7ff61c0cf910 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff621400640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 0x7ff61c1e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff620a00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 0x7ff61c1537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff61be00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff60800d800 tx=0x7ff60800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60800b6c0 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff608019070 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61c158330 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60800ba80 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.544+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61c1588f0 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.545+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61c068a10 con 0x7ff61c0cf910 2024-04-17T07:14:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.546+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff60800bc20 con 0x7ff61c0cf910 2024-04-17T07:14:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.547+0000 7ff618c00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 0x7ff5fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.547+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff60808c0a0 con 0x7ff61c0cf910 2024-04-17T07:14:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.548+0000 7ff620a00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 0x7ff5fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.549+0000 7ff620a00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 0x7ff5fc06edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0012e0 tx=0x7ff60c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.552+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff608011340 con 0x7ff61c0cf910 2024-04-17T07:14:04.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.749+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff61c15e310 con 0x7ff61c0cf910 2024-04-17T07:14:04.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.752+0000 7ff618c00640 1 -- 172.21.15.67:0/1419032635 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7ff60805a430 con 0x7ff61c0cf910 2024-04-17T07:14:04.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:04.753 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 msgr2=0x7ff5fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 0x7ff5fc06edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0012e0 tx=0x7ff60c009040 comp rx=0 tx=0).stop 2024-04-17T07:14:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 msgr2=0x7ff61c1e9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff60800d800 tx=0x7ff60800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 shutdown_connections 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff5fc06c8f0 0x7ff5fc06edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff61c15ec60 0x7ff61c1e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff61c0cf910 0x7ff61c1e9000 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 --2- 172.21.15.67:0/1419032635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff61c0b6d00 0x7ff61c1537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 >> 172.21.15.67:0/1419032635 conn(0x7ff61c0d15f0 msgr2=0x7ff61c1e9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 shutdown_connections 2024-04-17T07:14:04.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:04.755+0000 7ff621e00640 1 -- 172.21.15.67:0/1419032635 wait complete. 2024-04-17T07:14:04.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-17T07:14:04.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-17T07:14:04.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-17T07:14:04.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-17T07:14:04.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-17T07:14:04.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-17T07:14:04.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-17T07:14:04.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-17T07:14:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.074+0000 7f7d75c00640 1 Processor -- start 2024-04-17T07:14:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.075+0000 7f7d75c00640 1 -- start start 2024-04-17T07:14:05.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.076+0000 7f7d75c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d70158850 0x7f7d700af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.077+0000 7f7d75c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d700afd70 0x7f7d70071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.077+0000 7f7d75c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d7006eeb0 0x7f7d70072340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.077+0000 7f7d75c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7015b2d0 con 0x7f7d7006eeb0 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.077+0000 7f7d75c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7015b5b0 con 0x7f7d70158850 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.077+0000 7f7d75c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7015b890 con 0x7f7d700afd70 2024-04-17T07:14:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.078+0000 7f7d6f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d7006eeb0 0x7f7d70072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.078+0000 7f7d6e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d700afd70 0x7f7d70071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.078+0000 7f7d6f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d7006eeb0 0x7f7d70072340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41818/0 (socket says 172.21.15.67:41818) 2024-04-17T07:14:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.078+0000 7f7d6f400640 1 -- 172.21.15.67:0/2554178084 learned_addr learned my addr 172.21.15.67:0/2554178084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.078+0000 7f7d6ea00640 1 --2- 172.21.15.67:0/2554178084 >> v2:172.21.15.187:3300/0 conn(0x7f7d70158850 0x7f7d700af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.079+0000 7f7d6f400640 1 -- 172.21.15.67:0/2554178084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d700afd70 msgr2=0x7f7d70071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.079+0000 7f7d6f400640 1 --2- 172.21.15.67:0/2554178084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d700afd70 0x7f7d70071e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.079+0000 7f7d6f400640 1 -- 172.21.15.67:0/2554178084 >> v2:172.21.15.187:3300/0 conn(0x7f7d70158850 msgr2=0x7f7d700af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.079+0000 7f7d6f400640 1 --2- 172.21.15.67:0/2554178084 >> v2:172.21.15.187:3300/0 conn(0x7f7d70158850 0x7f7d700af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.079+0000 7f7d6f400640 1 -- 172.21.15.67:0/2554178084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7015bb70 con 0x7f7d7006eeb0 2024-04-17T07:14:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.081+0000 7f7d6f400640 1 --2- 172.21.15.67:0/2554178084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7006eeb0 0x7f7d70072340 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d70071ad0 tx=0x7f7d6802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e402e94967c736 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.081+0000 7f7d6d600640 1 -- 172.21.15.67:0/2554178084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6802cac0 con 0x7f7d7006eeb0 2024-04-17T07:14:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.081+0000 7f7d6d600640 1 -- 172.21.15.67:0/2554178084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d6802cc60 con 0x7f7d7006eeb0 2024-04-17T07:14:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.081+0000 7f7d6d600640 1 -- 172.21.15.67:0/2554178084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68034610 con 0x7f7d7006eeb0 2024-04-17T07:14:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.082+0000 7f7d75c00640 1 -- 172.21.15.67:0/2554178084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7006eeb0 msgr2=0x7f7d70072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.082+0000 7f7d75c00640 1 --2- 172.21.15.67:0/2554178084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7006eeb0 0x7f7d70072340 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d70071ad0 tx=0x7f7d6802bf20 comp rx=0 tx=0).stop 2024-04-17T07:14:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 -- 172.21.15.67:0/2554178084 shutdown_connections 2024-04-17T07:14:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 --2- 172.21.15.67:0/2554178084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7006eeb0 0x7f7d70072340 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 --2- 172.21.15.67:0/2554178084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d700afd70 0x7f7d70071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 --2- 172.21.15.67:0/2554178084 >> v2:172.21.15.187:3300/0 conn(0x7f7d70158850 0x7f7d700af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 -- 172.21.15.67:0/2554178084 >> 172.21.15.67:0/2554178084 conn(0x7f7d700b23c0 msgr2=0x7f7d700b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 -- 172.21.15.67:0/2554178084 shutdown_connections 2024-04-17T07:14:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.083+0000 7f7d75c00640 1 -- 172.21.15.67:0/2554178084 wait complete. 2024-04-17T07:14:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.085+0000 7fdb20800640 1 Processor -- start 2024-04-17T07:14:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.085+0000 7fdb20800640 1 -- start start 2024-04-17T07:14:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7f7d75c00640 1 Processor -- start 2024-04-17T07:14:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7fdb20800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 0x7fdb180cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7fdb20800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb1815ed10 0x7fdb180d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7f75fee00640 1 Processor -- start 2024-04-17T07:14:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7f7d75c00640 1 -- start start 2024-04-17T07:14:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7fdb20800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb180cff60 con 0x7fdb180d2ab0 2024-04-17T07:14:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7fdb20800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdb180d3c30 con 0x7fdb1815ed10 2024-04-17T07:14:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7fdb20800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb180d3020 con 0x7fdb180cf640 2024-04-17T07:14:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.087+0000 7f75fee00640 1 -- start start 2024-04-17T07:14:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7f3428a00640 1 Processor -- start 2024-04-17T07:14:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7fdb1de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 0x7fdb180cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7f7d75c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 0x7f7d7015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7f7d75c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d700afd70 0x7f7d70158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7fdb1e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdb180d2ab0 0x7fdb180b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50196/0 (socket says 172.21.15.67:50196) 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75fee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f8166440 0x7f75f80ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75fee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75fee00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f8072b20 con 0x7f75f80c6f90 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75fee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75f80ce960 con 0x7f75f8166440 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75fee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80c5eb0 con 0x7f75f8072420 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d75c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d70158850 0x7f7d701567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d6ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 0x7f7d7015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d75c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d700b2ee0 con 0x7f7d700afd70 2024-04-17T07:14:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d75c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7d700cfa50 con 0x7f7d70158850 2024-04-17T07:14:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7fdb1e800640 1 -- 172.21.15.67:0/3900019135 learned_addr learned my addr 172.21.15.67:0/3900019135 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d75c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d700b3060 con 0x7f7d7006eeb0 2024-04-17T07:14:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d6ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 0x7f7d7015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41108/0 (socket says 172.21.15.67:41108) 2024-04-17T07:14:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f7d6ea00640 1 -- 172.21.15.67:0/1702211721 learned_addr learned my addr 172.21.15.67:0/1702211721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75f7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f8166440 0x7f75f80ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75f7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f8166440 0x7f75f80ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41838/0 (socket says 172.21.15.67:41838) 2024-04-17T07:14:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.090+0000 7f75f7e00640 1 -- 172.21.15.67:0/2290327113 learned_addr learned my addr 172.21.15.67:0/2290327113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7fdb1d400640 1 --2- 172.21.15.67:0/3900019135 >> v2:172.21.15.67:3300/0 conn(0x7fdb1815ed10 0x7fdb180d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f7d6e000640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d700afd70 0x7f7d70158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.089+0000 7f3428a00640 1 -- start start 2024-04-17T07:14:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f7d6f400640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d70158850 0x7f7d701567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f3428a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f3428a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c00a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f3428a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c054d10 con 0x7f341c0e1310 2024-04-17T07:14:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f75fce00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f3428a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f341c055920 con 0x7f341c00ab40 2024-04-17T07:14:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.091+0000 7f3428a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f341c054ff0 con 0x7f341c0516e0 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7fdb1de00640 1 -- 172.21.15.67:0/3900019135 >> v2:172.21.15.67:3300/0 conn(0x7fdb1815ed10 msgr2=0x7fdb180d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7fdb1de00640 1 --2- 172.21.15.67:0/3900019135 >> v2:172.21.15.67:3300/0 conn(0x7fdb1815ed10 0x7fdb180d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7fdb1de00640 1 -- 172.21.15.67:0/3900019135 >> v1:172.21.15.67:6789/0 conn(0x7fdb180d2ab0 legacy=0x7fdb180b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7fdb1de00640 1 -- 172.21.15.67:0/3900019135 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb180d3300 con 0x7fdb180cf640 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f3422a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f3423400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f341c0e1310 0x7f341c03a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50198/0 (socket says 172.21.15.67:50198) 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f3422a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41842/0 (socket says 172.21.15.67:41842) 2024-04-17T07:14:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f3423400640 1 -- 172.21.15.67:0/2515399827 learned_addr learned my addr 172.21.15.67:0/2515399827 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f7d6ea00640 1 -- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d70158850 msgr2=0x7f7d701567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f7d6ea00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d70158850 0x7f7d701567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f7d6ea00640 1 -- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d700afd70 msgr2=0x7f7d70158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f7d6ea00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d700afd70 0x7f7d70158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.092+0000 7f7d6ea00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d70159530 con 0x7f7d7006eeb0 2024-04-17T07:14:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f3423e00640 1 --2- 172.21.15.67:0/2515399827 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c00a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f75f7e00640 1 -- 172.21.15.67:0/2290327113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 msgr2=0x7f75f806dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f75f7e00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f806dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f75f7e00640 1 -- 172.21.15.67:0/2290327113 >> v1:172.21.15.187:6789/0 conn(0x7f75f80c6f90 legacy=0x7f75f8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.093+0000 7f75f7e00640 1 -- 172.21.15.67:0/2290327113 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80c6190 con 0x7f75f8166440 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f7d6ea00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 0x7f7d7015a5d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00e860 tx=0x7f7d5c00ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3422000640 1 -- 172.21.15.67:0/2515399827 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130760246 0 0) 0x7f341c054d10 con 0x7f341c0e1310 2024-04-17T07:14:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3422000640 1 -- 172.21.15.67:0/2515399827 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f340c003520 con 0x7f341c0e1310 2024-04-17T07:14:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c017070 con 0x7f7d7006eeb0 2024-04-17T07:14:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c00cb40 con 0x7f7d7006eeb0 2024-04-17T07:14:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c0113f0 con 0x7f7d7006eeb0 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3423e00640 1 -- 172.21.15.67:0/2515399827 >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 msgr2=0x7f341c009d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3423e00640 1 --2- 172.21.15.67:0/2515399827 >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3423e00640 1 -- 172.21.15.67:0/2515399827 >> v1:172.21.15.67:6789/0 conn(0x7f341c0e1310 legacy=0x7f341c03a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3423e00640 1 -- 172.21.15.67:0/2515399827 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0552d0 con 0x7f341c0516e0 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f3422a00640 1 --2- 172.21.15.67:0/2515399827 >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.094+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d70159810 con 0x7f7d7006eeb0 2024-04-17T07:14:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.095+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d70159dd0 con 0x7f7d7006eeb0 2024-04-17T07:14:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.095+0000 7fdb1de00640 1 --2- 172.21.15.67:0/3900019135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 0x7fdb180cfa20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb040097a0 tx=0x7fdb0402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19060364cf7a0584 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7f3423e00640 1 --2- 172.21.15.67:0/2515399827 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c00a290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f34100097a0 tx=0x7f341002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cf7112e8d37cd04 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d7015a1d0 con 0x7f7d7006eeb0 2024-04-17T07:14:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7f75f7e00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f80ce050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f75f8072cf0 tx=0x7f75e802b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd387641d6d9687c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7fdb1ca00640 1 -- 172.21.15.67:0/3900019135 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04034070 con 0x7fdb180cf640 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7fdb1ca00640 1 -- 172.21.15.67:0/3900019135 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb04030800 con 0x7fdb180cf640 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.096+0000 7fdb1ca00640 1 -- 172.21.15.67:0/3900019135 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04030bd0 con 0x7fdb180cf640 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.097+0000 7f75f6a00640 1 -- 172.21.15.67:0/2290327113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8037070 con 0x7f75f8166440 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.097+0000 7f75f6a00640 1 -- 172.21.15.67:0/2290327113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e8030710 con 0x7f75f8166440 2024-04-17T07:14:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.097+0000 7f75f6a00640 1 -- 172.21.15.67:0/2290327113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8030ae0 con 0x7f75f8166440 2024-04-17T07:14:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.097+0000 7fdb20800640 1 -- 172.21.15.67:0/3900019135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 msgr2=0x7fdb180cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.097+0000 7fdb20800640 1 --2- 172.21.15.67:0/3900019135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 0x7fdb180cfa20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdb040097a0 tx=0x7fdb0402b940 comp rx=0 tx=0).stop 2024-04-17T07:14:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f3422000640 1 -- 172.21.15.67:0/2515399827 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410035020 con 0x7f341c0516e0 2024-04-17T07:14:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f3422000640 1 -- 172.21.15.67:0/2515399827 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3410031810 con 0x7f341c0516e0 2024-04-17T07:14:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f3422000640 1 -- 172.21.15.67:0/2515399827 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410031be0 con 0x7f341c0516e0 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7fdb20800640 1 -- 172.21.15.67:0/3900019135 shutdown_connections 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7fdb20800640 1 --2- 172.21.15.67:0/3900019135 >> v2:172.21.15.67:3300/0 conn(0x7fdb1815ed10 0x7fdb180d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7fdb20800640 1 --2- 172.21.15.67:0/3900019135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb180cf640 0x7fdb180cfa20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7fdb20800640 1 -- 172.21.15.67:0/3900019135 >> 172.21.15.67:0/3900019135 conn(0x7fdb1806cbd0 msgr2=0x7fdb1806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f6c70200640 1 Processor -- start 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f75fee00640 1 -- 172.21.15.67:0/2290327113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 msgr2=0x7f75f80ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f75fee00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f80ce050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f75f8072cf0 tx=0x7f75e802b720 comp rx=0 tx=0).stop 2024-04-17T07:14:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f3428a00640 1 -- 172.21.15.67:0/2515399827 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 msgr2=0x7f341c00a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f3428a00640 1 --2- 172.21.15.67:0/2515399827 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c00a290 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f34100097a0 tx=0x7f341002b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.098+0000 7f6c70200640 1 -- start start 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f75fee00640 1 -- 172.21.15.67:0/2290327113 shutdown_connections 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f75fee00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f806dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f75fee00640 1 --2- 172.21.15.67:0/2290327113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f80ce050 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f75fee00640 1 -- 172.21.15.67:0/2290327113 >> 172.21.15.67:0/2290327113 conn(0x7f75f806bf50 msgr2=0x7f75f80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7f3428a00640 1 -- 172.21.15.67:0/2515399827 shutdown_connections 2024-04-17T07:14:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f3428a00640 1 --2- 172.21.15.67:0/2515399827 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c00a290 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d5c01a070 con 0x7f7d7006eeb0 2024-04-17T07:14:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f75fee00640 1 -- 172.21.15.67:0/2290327113 shutdown_connections 2024-04-17T07:14:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f3428a00640 1 --2- 172.21.15.67:0/2515399827 >> v2:172.21.15.67:3300/0 conn(0x7f341c00ab40 0x7f341c009d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f3428a00640 1 -- 172.21.15.67:0/2515399827 >> 172.21.15.67:0/2515399827 conn(0x7f341c056cd0 msgr2=0x7f341c0570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f3428a00640 1 -- 172.21.15.67:0/2515399827 shutdown_connections 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f75fee00640 1 -- 172.21.15.67:0/2290327113 wait complete. 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7fdb20800640 1 -- 172.21.15.67:0/3900019135 shutdown_connections 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.099+0000 7fdb20800640 1 -- 172.21.15.67:0/3900019135 wait complete. 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f6c70200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 0x7f6c680cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f6c70200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6c6815ed10 0x7f6c680d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f6c70200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c680cff60 con 0x7f6c680d2ab0 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f6c70200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6c680d3c30 con 0x7f6c680cf640 2024-04-17T07:14:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f6c70200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6c680d3020 con 0x7f6c6815ed10 2024-04-17T07:14:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.100+0000 7f3428a00640 1 -- 172.21.15.67:0/2515399827 wait complete. 2024-04-17T07:14:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f6c6d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 0x7f6c680cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7fdb20800640 1 Processor -- start 2024-04-17T07:14:05.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f6c6d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 0x7f6c680cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41850/0 (socket says 172.21.15.67:41850) 2024-04-17T07:14:05.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f38dcc00640 1 Processor -- start 2024-04-17T07:14:05.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f7d66a00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d5006c8f0 0x7f7d5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7d5c08bb70 con 0x7f7d7006eeb0 2024-04-17T07:14:05.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f6c6d800640 1 -- 172.21.15.67:0/1952499490 learned_addr learned my addr 172.21.15.67:0/1952499490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f7d66a00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f7d50074b10 0x7f7d50076f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7fdb20800640 1 -- start start 2024-04-17T07:14:05.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f38dcc00640 1 -- start start 2024-04-17T07:14:05.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d500775e0 con 0x7f7d50074b10 2024-04-17T07:14:05.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.102+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c08bf60 con 0x7f7d7006eeb0 2024-04-17T07:14:05.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f3428a00640 1 Processor -- start 2024-04-17T07:14:05.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f6dad600640 1 Processor -- start 2024-04-17T07:14:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264754126 0 0) 0x7f6c680cff60 con 0x7f6c680d2ab0 2024-04-17T07:14:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/1952499490 >> v2:172.21.15.187:3300/0 conn(0x7f6c6815ed10 0x7f6c680d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c50003520 con 0x7f6c680d2ab0 2024-04-17T07:14:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f6dad600640 1 -- start start 2024-04-17T07:14:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f7d6e000640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d5006c8f0 0x7f7d5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2754472225 0 0) 0x7f6c50003520 con 0x7f6c680d2ab0 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 >> v2:172.21.15.187:3300/0 conn(0x7f6c6815ed10 msgr2=0x7f6c680d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 --2- 172.21.15.67:0/1952499490 >> v2:172.21.15.187:3300/0 conn(0x7f6c6815ed10 0x7f6c680d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 msgr2=0x7f6c680cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 --2- 172.21.15.67:0/1952499490 >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 0x7f6c680cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c680d3300 con 0x7f6c680d2ab0 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6c54001670 con 0x7f6c680d2ab0 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 >> v1:172.21.15.187:6789/0 conn(0x7f6c680d2ab0 legacy=0x7f6c680b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c500039b0 0x7f6c50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c50004800 0x7f6c50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 0x7f6c5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680d3c30 con 0x7f6c50004800 2024-04-17T07:14:05.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680d3020 con 0x7f6c500051c0 2024-04-17T07:14:05.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6c5000bf30 con 0x7f6c500039b0 2024-04-17T07:14:05.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f7d6f400640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f7d50074b10 0x7f7d50076f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.103+0000 7f3428a00640 1 -- start start 2024-04-17T07:14:05.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f75fee00640 1 Processor -- start 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 0x7f341c0cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c0c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c0e1310 0x7f341c049870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f341c054a20 con 0x7f341c00ab40 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0548a0 con 0x7f341c0e1310 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f3428a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f341c055db0 con 0x7f341c0516e0 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 0x7fdb18202330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb180d2ab0 0x7fdb180c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb1815ed10 0x7fdb18160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180d25b0 con 0x7fdb180d2ab0 2024-04-17T07:14:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f75fee00640 1 -- start start 2024-04-17T07:14:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180d38c0 con 0x7fdb180cf640 2024-04-17T07:14:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7fdb20800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb180d4600 con 0x7fdb1815ed10 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f3423400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 0x7f341c0cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f3423400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 0x7f341c0cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41872/0 (socket says 172.21.15.67:41872) 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f3422a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c0c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f3423400640 1 -- 172.21.15.67:0/3141446188 learned_addr learned my addr 172.21.15.67:0/3141446188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f6c6e200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 0x7f6c5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f6c6d800640 1 --2- 172.21.15.67:0/1952499490 >> v2:172.21.15.67:3300/0 conn(0x7f6c680cf640 0x7f6c680cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f6dad600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 0x7f6da80cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6dad600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6dad600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6da80d2ab0 0x7f6da80b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6dad600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6da80d3c30 con 0x7f6da80cf640 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6dad600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6da80cff60 con 0x7f6da80d2ab0 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6dad600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6da80d3020 con 0x7f6da815ed10 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7fdb1e800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb1815ed10 0x7fdb18160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7fdb1de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 0x7fdb18202330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7fdb1de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 0x7fdb18202330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53746/0 (socket says 172.21.15.67:53746) 2024-04-17T07:14:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7fdb1de00640 1 -- 172.21.15.67:0/2589703 learned_addr learned my addr 172.21.15.67:0/2589703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f38dcc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 0x7f38d80d4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f38dcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38d80cf560 0x7f38d80cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.104+0000 7f38dcc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d80d30b0 con 0x7f38d815eba0 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f38dcc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f38d80d3390 con 0x7f38d80cf560 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f38dcc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f38d80d3670 con 0x7f38d80ba400 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f38d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38d80cf560 0x7f38d80cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f38d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38d80cf560 0x7f38d80cf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41874/0 (socket says 172.21.15.67:41874) 2024-04-17T07:14:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.105+0000 7f38d7e00640 1 -- 172.21.15.67:0/368911856 learned_addr learned my addr 172.21.15.67:0/368911856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f38d6a00640 1 --2- 172.21.15.67:0/368911856 >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 0x7f38d80d4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f3423e00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c0e1310 0x7f341c049870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f80c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75f80c6f90 0x7f75f80c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80cf0d0 con 0x7f75f8166440 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75f806e290 con 0x7f75f80c6f90 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f75fee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75f814fce0 con 0x7f75f8072420 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c50004800 0x7f6c50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6e200640 1 -- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c500039b0 msgr2=0x7f6c50003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-17T07:14:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6e200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c500039b0 0x7f6c50003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6e200640 1 -- 172.21.15.67:0/1952499490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c50004800 msgr2=0x7f6c50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6e200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c50004800 0x7f6c50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6e200640 1 -- 172.21.15.67:0/1952499490 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c680d3300 con 0x7f6c500051c0 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.106+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c50004800 0x7f6c50004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f7d6e000640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d5006c8f0 0x7f7d5006edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d701fbd70 tx=0x7f7d60008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f7d6f400640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f7d50074b10 0x7f7d50076f30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 -- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 msgr2=0x7f341c0c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c0c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 -- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c0e1310 msgr2=0x7f341c049870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f341c0e1310 0x7f341c049870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0cce00 con 0x7f341c00ab40 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3422a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f341c0516e0 0x7f341c0c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f3423400640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 0x7f341c0cc8c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3424051240 tx=0x7f342406dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7fdb1d400640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb180d2ab0 0x7fdb180c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7fdb1de00640 1 -- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb1815ed10 msgr2=0x7fdb18160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7fdb1de00640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb1815ed10 0x7fdb18160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7fdb1de00640 1 -- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb180d2ab0 msgr2=0x7fdb180c7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f6da6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f6da6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41176/0 (socket says 172.21.15.67:41176) 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f6da6000640 1 -- 172.21.15.67:0/547261836 learned_addr learned my addr 172.21.15.67:0/547261836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f38d7e00640 1 -- 172.21.15.67:0/368911856 >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 msgr2=0x7f38d80d4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f38d7e00640 1 --2- 172.21.15.67:0/368911856 >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 0x7f38d80d4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f38d7e00640 1 -- 172.21.15.67:0/368911856 >> v1:172.21.15.187:6789/0 conn(0x7f38d815eba0 legacy=0x7f38d80d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f38d7e00640 1 -- 172.21.15.67:0/368911856 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d80d3950 con 0x7f38d80cf560 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.107+0000 7f38d6a00640 1 --2- 172.21.15.67:0/368911856 >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 0x7f38d80d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f6da7400640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.187:3300/0 conn(0x7f6da80d2ab0 0x7f6da80b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f80c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f80c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41186/0 (socket says 172.21.15.67:41186) 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f75f7e00640 1 -- 172.21.15.67:0/3375652665 learned_addr learned my addr 172.21.15.67:0/3375652665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f75fce00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342406b8c0 con 0x7f341c00ab40 2024-04-17T07:14:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f342406ba60 con 0x7f341c00ab40 2024-04-17T07:14:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342406d110 con 0x7f341c00ab40 2024-04-17T07:14:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f75f7400640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75f80c6f90 0x7f75f80c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7fdb1de00640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb180d2ab0 0x7fdb180c7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7fdb1de00640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb18202870 con 0x7fdb180cf640 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7fdb1d400640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb180d2ab0 0x7fdb180c7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f6da6a00640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 0x7f6da80cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f75f7e00640 1 -- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75f80c6f90 msgr2=0x7f75f80c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f75f7e00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75f80c6f90 0x7f75f80c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f75f7e00640 1 -- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 msgr2=0x7f75f814d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.108+0000 7f6c6e200640 1 -- 172.21.15.67:0/1952499490 reap_dead start 2024-04-17T07:14:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f75f7e00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f8166440 0x7f75f814d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f75f7e00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80c7c10 con 0x7f75f8072420 2024-04-17T07:14:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c050d50 con 0x7f341c00ab40 2024-04-17T07:14:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c050a70 con 0x7f341c00ab40 2024-04-17T07:14:05.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== osd.0 v2:172.21.15.67:6815/2121978331 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d500775e0 con 0x7f7d50074b10 2024-04-17T07:14:05.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f6c6e200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 0x7f6c5000b9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400c190 tx=0x7f6c50005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b22bcfcb6c01f746 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f6da6000640 1 -- 172.21.15.67:0/547261836 >> v2:172.21.15.187:3300/0 conn(0x7f6da80d2ab0 msgr2=0x7f6da80b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f6da6000640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.187:3300/0 conn(0x7f6da80d2ab0 0x7f6da80b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7fdb1de00640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 0x7fdb18202330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb0402bf50 tx=0x7fdb040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f41b2600640 1 Processor -- start 2024-04-17T07:14:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04034040 con 0x7fdb180cf640 2024-04-17T07:14:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75f7e00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f80c7550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009960 tx=0x7f75e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb04039c90 con 0x7fdb180cf640 2024-04-17T07:14:05.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04042610 con 0x7fdb180cf640 2024-04-17T07:14:05.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb180c7650 con 0x7fdb180cf640 2024-04-17T07:14:05.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f41b2600640 1 -- start start 2024-04-17T07:14:05.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f38d7e00640 1 --2- 172.21.15.67:0/368911856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80cf560 0x7f38d80cf9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f38d80d2050 tx=0x7f38c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588e874b88701fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8037040 con 0x7f75f8072420 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e8042070 con 0x7f75f8072420 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e803ba70 con 0x7f75f8072420 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f814f3b0 con 0x7f75f8072420 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f814f970 con 0x7f75f8072420 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54039070 con 0x7f6c500051c0 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c5402ca90 con 0x7f6c500051c0 2024-04-17T07:14:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.110+0000 7f6c67e00640 1 -- 172.21.15.67:0/1952499490 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54007c80 con 0x7f6c500051c0 2024-04-17T07:14:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.109+0000 7f6da6000640 1 -- 172.21.15.67:0/547261836 >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 msgr2=0x7f6da80cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7f6da6000640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 0x7f6da80cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7f6da6000640 1 -- 172.21.15.67:0/547261836 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da80d3300 con 0x7f6da815ed10 2024-04-17T07:14:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f341c04bec0 con 0x7f341c00ab40 2024-04-17T07:14:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb1815ae20 con 0x7fdb180cf640 2024-04-17T07:14:05.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7f6c70200640 1 -- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 msgr2=0x7f6c5000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.111+0000 7f6c70200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 0x7f6c5000b9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400c190 tx=0x7f6c50005830 comp rx=0 tx=0).stop 2024-04-17T07:14:05.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6da6a00640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 0x7f6da80cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:05.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41b2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41b2600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41ac0c6f90 0x7f41ac06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6c70200640 1 -- 172.21.15.67:0/1952499490 shutdown_connections 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6c70200640 1 --2- 172.21.15.67:0/1952499490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c500051c0 0x7f6c5000b9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6c70200640 1 -- 172.21.15.67:0/1952499490 >> 172.21.15.67:0/1952499490 conn(0x7f6c6806cbd0 msgr2=0x7f6c6806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41b2600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ac072b20 con 0x7f41ac166440 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41b2600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f41ac0ce960 con 0x7f41ac0c6f90 2024-04-17T07:14:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f38d6000640 1 -- 172.21.15.67:0/368911856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c402e070 con 0x7f38d80cf560 2024-04-17T07:14:05.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41b2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac0c5eb0 con 0x7f41ac072420 2024-04-17T07:14:05.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f38d6000640 1 -- 172.21.15.67:0/368911856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c4037070 con 0x7f38d80cf560 2024-04-17T07:14:05.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f38d6000640 1 -- 172.21.15.67:0/368911856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4031a00 con 0x7f38d80cf560 2024-04-17T07:14:05.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6c70200640 1 -- 172.21.15.67:0/1952499490 shutdown_connections 2024-04-17T07:14:05.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f41aaa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.113+0000 7f41aaa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41188/0 (socket says 172.21.15.67:41188) 2024-04-17T07:14:05.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.113+0000 7f41aaa00640 1 -- 172.21.15.67:0/420061966 learned_addr learned my addr 172.21.15.67:0/420061966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.112+0000 7f6c70200640 1 -- 172.21.15.67:0/1952499490 wait complete. 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.113+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f342406d2b0 con 0x7f341c00ab40 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f41aaa00640 1 -- 172.21.15.67:0/420061966 >> v2:172.21.15.187:3300/0 conn(0x7f41ac0c6f90 msgr2=0x7f41ac06dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f41aaa00640 1 --2- 172.21.15.67:0/420061966 >> v2:172.21.15.187:3300/0 conn(0x7f41ac0c6f90 0x7f41ac06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f41aaa00640 1 -- 172.21.15.67:0/420061966 >> v1:172.21.15.67:6789/0 conn(0x7f41ac166440 legacy=0x7f41ac071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75e803a070 con 0x7f75f8072420 2024-04-17T07:14:05.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f75f80ca3c0 con 0x7f75f8072420 2024-04-17T07:14:05.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f41aaa00640 1 -- 172.21.15.67:0/420061966 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac0c6190 con 0x7f41ac072420 2024-04-17T07:14:05.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.114+0000 7f3417400640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f340406c940 0x7f340406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75efe00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f75d806c8f0 0x7f75d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 -- 172.21.15.67:0/368911856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80cf560 msgr2=0x7f38d80cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 --2- 172.21.15.67:0/368911856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80cf560 0x7f38d80cf9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f38d80d2050 tx=0x7f38c402b880 comp rx=0 tx=0).stop 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f75e80acda0 con 0x7f75f8072420 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75efe00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f75d8074af0 0x7f75d8076f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75d80775c0 con 0x7f75d8074af0 2024-04-17T07:14:05.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f75e80af4a0 con 0x7f75f8072420 2024-04-17T07:14:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 -- 172.21.15.67:0/368911856 shutdown_connections 2024-04-17T07:14:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 --2- 172.21.15.67:0/368911856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80cf560 0x7f38d80cf9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 --2- 172.21.15.67:0/368911856 >> v2:172.21.15.187:3300/0 conn(0x7f38d80ba400 0x7f38d80d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f38dcc00640 1 -- 172.21.15.67:0/368911856 >> 172.21.15.67:0/368911856 conn(0x7f38d806cac0 msgr2=0x7f38d806cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3424107080 con 0x7f341c00ab40 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f6da6000640 1 --2- 172.21.15.67:0/547261836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6da806d940 tx=0x7f6d9402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59c0dceba32327 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f3417400640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7f3404074bc0 0x7f3404076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3404077690 con 0x7f3404074bc0 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f342410e050 con 0x7f341c00ab40 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f3423e00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7f3404074bc0 0x7f3404076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb180d0610 con 0x7fdb180cf640 2024-04-17T07:14:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f38dcc00640 1 -- 172.21.15.67:0/368911856 shutdown_connections 2024-04-17T07:14:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.115+0000 7f75fce00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f75d8074af0 0x7f75d8076f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f75f7400640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f75d806c8f0 0x7f75d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f6da5600640 1 -- 172.21.15.67:0/547261836 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94034070 con 0x7f6da815ed10 2024-04-17T07:14:05.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f3422a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f340406c940 0x7f340406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f6c70200640 1 Processor -- start 2024-04-17T07:14:05.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f75f7400640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f75d806c8f0 0x7f75d806edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f75f815c590 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f6da5600640 1 -- 172.21.15.67:0/547261836 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d94030860 con 0x7f6da815ed10 2024-04-17T07:14:05.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f6da5600640 1 -- 172.21.15.67:0/547261836 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94030c30 con 0x7f6da815ed10 2024-04-17T07:14:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f6c70200640 1 -- start start 2024-04-17T07:14:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f41aaa00640 1 --2- 172.21.15.67:0/420061966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4198009870 tx=0x7f419802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cc50031200bc1f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.117+0000 7f3423e00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7f3404074bc0 0x7f3404076fe0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 -- 172.21.15.67:0/547261836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 msgr2=0x7f6da80d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 --2- 172.21.15.67:0/547261836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6da806d940 tx=0x7f6d9402b9a0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 -- 172.21.15.67:0/547261836 shutdown_connections 2024-04-17T07:14:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.187:3300/0 conn(0x7f6da80d2ab0 0x7f6da80b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 --2- 172.21.15.67:0/547261836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da815ed10 0x7f6da80d40c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6da806d940 tx=0x7f6d9402b9a0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 --2- 172.21.15.67:0/547261836 >> v2:172.21.15.67:3300/0 conn(0x7f6da80cf640 0x7f6da80cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f6dad600640 1 -- 172.21.15.67:0/547261836 >> 172.21.15.67:0/547261836 conn(0x7f6da806cbd0 msgr2=0x7f6da806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.118+0000 7f75fce00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f75d8074af0 0x7f75d8076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== osd.2 v2:172.21.15.67:6824/873912528 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3404077690 con 0x7f3404074bc0 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f6dad600640 1 -- 172.21.15.67:0/547261836 shutdown_connections 2024-04-17T07:14:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f6dad600640 1 -- 172.21.15.67:0/547261836 wait complete. 2024-04-17T07:14:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.116+0000 7f38dcc00640 1 -- 172.21.15.67:0/368911856 wait complete. 2024-04-17T07:14:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f38dcc00640 1 Processor -- start 2024-04-17T07:14:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== osd.1 v2:172.21.15.67:6800/2081347185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75d80775c0 con 0x7f75d8074af0 2024-04-17T07:14:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7f3422a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f340406c940 0x7f340406ee00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f341c180d20 tx=0x7f3418008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.120+0000 7f38dcc00640 1 -- start start 2024-04-17T07:14:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.121+0000 7f6c70200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 0x7f6c680b97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.121+0000 7f6dad600640 1 Processor -- start 2024-04-17T07:14:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f38dcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 0x7f38d8201a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c70200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c6815d020 0x7f6c680cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f38dcc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38d80cf560 0x7f38d80cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c70200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c6815d4a0 0x7f6c680cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6dad600640 1 -- start start 2024-04-17T07:14:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f38d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 0x7f38d8201a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c70200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680d25b0 con 0x7f6c500051c0 2024-04-17T07:14:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c70200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6c680d38c0 con 0x7f6c6815d020 2024-04-17T07:14:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f38dcc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38d815eba0 0x7f38d80ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f38d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 0x7f38d8201a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41914/0 (socket says 172.21.15.67:41914) 2024-04-17T07:14:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c70200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6c680ba300 con 0x7f6c6815d4a0 2024-04-17T07:14:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.122+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 0x7f6c680b97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 0x7f6c680b97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41904/0 (socket says 172.21.15.67:41904) 2024-04-17T07:14:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.119+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb04037070 con 0x7fdb180cf640 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.120+0000 7fdb12000640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdaf806c8f0 0x7fdaf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.120+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdb040add70 con 0x7fdb180cf640 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.121+0000 7fdb1d400640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdaf806c8f0 0x7fdaf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f41aa000640 1 -- 172.21.15.67:0/420061966 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198035020 con 0x7f41ac072420 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f41aa000640 1 -- 172.21.15.67:0/420061966 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4198031640 con 0x7f41ac072420 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f41aa000640 1 -- 172.21.15.67:0/420061966 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419802d040 con 0x7f41ac072420 2024-04-17T07:14:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38d7400640 1 -- 172.21.15.67:0/126363050 learned_addr learned my addr 172.21.15.67:0/126363050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7fdb12000640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fdaf8074a90 0x7fdaf8076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb04001d30 con 0x7fdaf8074a90 2024-04-17T07:14:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdb04077ca0 con 0x7fdb180cf640 2024-04-17T07:14:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38d80b7ad0 con 0x7f38d80ba400 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38d7e00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38d815eba0 0x7f38d80ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38d80b7950 con 0x7f38d815eba0 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38d80b7c50 con 0x7f38d80cf560 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6c6d800640 1 -- 172.21.15.67:0/4161502917 learned_addr learned my addr 172.21.15.67:0/4161502917 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6c6e200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c6815d4a0 0x7f6c680cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.123+0000 7f38d6a00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38d80cf560 0x7f38d80cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6dad600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 0x7f6da80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 -- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38d80cf560 msgr2=0x7f38d80cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38d80cf560 0x7f38d80cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7fdb1d400640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdaf806c8f0 0x7fdaf806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb1815df00 tx=0x7fdb0c007400 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c6815d020 0x7f6c680cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6dad600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 0x7f6da80b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6dad600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da80b6f50 0x7f6da81f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 -- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38d815eba0 msgr2=0x7f38d80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38d815eba0 0x7f38d80ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d815f190 con 0x7f38d80ba400 2024-04-17T07:14:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f38d7400640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 0x7f38d8201a70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b3f0 tx=0x7f38c800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7fdb1e800640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fdaf8074a90 0x7fdaf8076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6dad600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80d25b0 con 0x7f6da80cf640 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6dad600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6da80d38c0 con 0x7f6da80b6f50 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6dad600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6da80d4600 con 0x7f6da80d2ab0 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.124+0000 7f6da6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 0x7f6da80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6da6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 0x7f6da80b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8015020 con 0x7f38d80ba400 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6da6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 0x7f6da80b5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41206/0 (socket says 172.21.15.67:41206) 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6da6000640 1 -- 172.21.15.67:0/2491553719 learned_addr learned my addr 172.21.15.67:0/2491553719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6da7400640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da80b6f50 0x7f6da81f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c8011830 con 0x7f38d80ba400 2024-04-17T07:14:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d815f470 con 0x7f38d80ba400 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6c6e200640 1 -- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c6815d020 msgr2=0x7f6c680cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6c6e200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c6815d020 0x7f6c680cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6c6e200640 1 -- 172.21.15.67:0/4161502917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 msgr2=0x7f6c680b97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6c6e200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 0x7f6c680b97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6c6e200640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c680b5eb0 con 0x7f6c6815d4a0 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f6da6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 0x7f6da80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41916/0 (socket says 172.21.15.67:41916) 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6000640 1 -- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da80b6f50 msgr2=0x7f6da81f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6000640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da80b6f50 0x7f6da81f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6000640 1 -- 172.21.15.67:0/2491553719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 msgr2=0x7f6da80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6000640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 0x7f6da80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6000640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da80b5eb0 con 0x7f6da80d2ab0 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6da6a00640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da80cf640 0x7f6da80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.125+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d815fa30 con 0x7f38d80ba400 2024-04-17T07:14:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8011c20 con 0x7f38d80ba400 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f41b2600640 1 -- 172.21.15.67:0/420061966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 msgr2=0x7f41ac0ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f41b2600640 1 --2- 172.21.15.67:0/420061966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4198009870 tx=0x7f419802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 -- 172.21.15.67:0/420061966 shutdown_connections 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 --2- 172.21.15.67:0/420061966 >> v2:172.21.15.187:3300/0 conn(0x7f41ac0c6f90 0x7f41ac06dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 --2- 172.21.15.67:0/420061966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac0ce050 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 -- 172.21.15.67:0/420061966 >> 172.21.15.67:0/420061966 conn(0x7f41ac06bf50 msgr2=0x7f41ac0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.126+0000 7f6c6d800640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c500051c0 0x7f6c680b97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f6c6e200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c6815d4a0 0x7f6c680cde90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c680c82c0 tx=0x7f6c54001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f6da6000640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 0x7f6da80b5760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009960 tx=0x7f6d94004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38d80755d0 con 0x7f38d80ba400 2024-04-17T07:14:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 -- 172.21.15.67:0/420061966 shutdown_connections 2024-04-17T07:14:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f41b2600640 1 -- 172.21.15.67:0/420061966 wait complete. 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.128+0000 7fdb1e800640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fdaf8074a90 0x7fdaf8076eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.127+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54039040 con 0x7f6c6815d4a0 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.128+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c54042070 con 0x7f6c6815d4a0 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.128+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c54007230 con 0x7f6c6815d4a0 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.128+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c680b6190 con 0x7f6c6815d4a0 2024-04-17T07:14:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.128+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c680b6750 con 0x7f6c6815d4a0 2024-04-17T07:14:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94034040 con 0x7f6da80d2ab0 2024-04-17T07:14:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d94030660 con 0x7f6da80d2ab0 2024-04-17T07:14:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94044420 con 0x7f6da80d2ab0 2024-04-17T07:14:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da80b6190 con 0x7f6da80d2ab0 2024-04-17T07:14:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da80b6750 con 0x7f6da80d2ab0 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== osd.3 v2:172.21.15.67:6804/100610741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb04001d30 con 0x7fdaf8074a90 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.129+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f38c8018020 con 0x7f38d80ba400 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f38cf400640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38b806c940 0x7f38b806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f38c808b560 con 0x7f38d80ba400 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f38cf400640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7f38b8074bc0 0x7f38b8076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38b8077690 con 0x7f38b8074bc0 2024-04-17T07:14:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f38c8092050 con 0x7f38d80ba400 2024-04-17T07:14:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.130+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c44001030 con 0x7f6c6815d4a0 2024-04-17T07:14:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.131+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6da806d4e0 con 0x7f6da80d2ab0 2024-04-17T07:14:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.131+0000 7f41b2600640 1 Processor -- start 2024-04-17T07:14:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c540073d0 con 0x7f6c6815d4a0 2024-04-17T07:14:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.131+0000 7f38d7e00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7f38b8074bc0 0x7f38b8076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.131+0000 7f38d6a00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38b806c940 0x7f38b806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.132+0000 7f41b2600640 1 -- start start 2024-04-17T07:14:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41ac0c6f90 0x7f41ac0c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41ac166440 0x7f41ac0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac0cf0d0 con 0x7f41ac0c6f90 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac06e290 con 0x7f41ac166440 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.133+0000 7f41b2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac15f590 con 0x7f41ac072420 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f41ab400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f41aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41ac0c6f90 0x7f41ac0c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f41aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41ac0c6f90 0x7f41ac0c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41924/0 (socket says 172.21.15.67:41924) 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f41aaa00640 1 -- 172.21.15.67:0/1299777090 learned_addr learned my addr 172.21.15.67:0/1299777090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.134+0000 7f41abe00640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41ac166440 0x7f41ac0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f41ab400640 1 -- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41ac166440 msgr2=0x7f41ac0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f41ab400640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41ac166440 0x7f41ac0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f41ab400640 1 -- 172.21.15.67:0/1299777090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41ac0c6f90 msgr2=0x7f41ac0c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f41ab400640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41ac0c6f90 0x7f41ac0c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f41ab400640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac15e980 con 0x7f41ac072420 2024-04-17T07:14:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.135+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d94037070 con 0x7f6da80d2ab0 2024-04-17T07:14:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f41ab400640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac15aaf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c390 tx=0x7f419c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f6c65600640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6c4806c9c0 0x7f6c4806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6c540adbf0 con 0x7f6c6815d4a0 2024-04-17T07:14:05.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c017070 con 0x7f41ac072420 2024-04-17T07:14:05.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c0052d0 con 0x7f41ac072420 2024-04-17T07:14:05.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c003830 con 0x7f41ac072420 2024-04-17T07:14:05.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.136+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac15ec60 con 0x7f41ac072420 2024-04-17T07:14:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.137+0000 7f6c65600640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f6c48074be0 0x7f6c48077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.137+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac15f220 con 0x7f41ac072420 2024-04-17T07:14:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.137+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c54004ea0 con 0x7f6c48074be0 2024-04-17T07:14:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.137+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6c540adfe0 con 0x7f6c6815d4a0 2024-04-17T07:14:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.137+0000 7f38d6a00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38b806c940 0x7f38b806ee00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f38d0003fd0 tx=0x7f38d00039f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.138+0000 7f6c6d800640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6c4806c9c0 0x7f6c4806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.138+0000 7f38d7e00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7f38b8074bc0 0x7f38b8076fe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.138+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41ac0ce380 con 0x7f41ac072420 2024-04-17T07:14:05.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.138+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f6c48074be0 0x7f6c48077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.139+0000 7f6d9ea00640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6d8806c9c0 0x7f6d8806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.139+0000 7f6c6ce00640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f6c48074be0 0x7f6c48077000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6d940adfb0 con 0x7f6da80d2ab0 2024-04-17T07:14:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6c6d800640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6c4806c9c0 0x7f6c4806ee80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c60003e40 tx=0x7f6c6000d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6d9ea00640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f6d88074be0 0x7f6d88077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6da6a00640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6d8806c9c0 0x7f6d8806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d94046e70 con 0x7f6d88074be0 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.140+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6d940ae3a0 con 0x7f6da80d2ab0 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.141+0000 7f6da7400640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f6d88074be0 0x7f6d88077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.141+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f419c01a070 con 0x7f41ac072420 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.141+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== osd.6 v2:172.21.15.187:6817/3359222512 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38b8077690 con 0x7f38b8074bc0 2024-04-17T07:14:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 --> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7d70068a10 con 0x7f7d50074b10 2024-04-17T07:14:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f6da7400640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f6d88074be0 0x7f6d88077000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f6da6a00640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6d8806c9c0 0x7f6d8806ee80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d98006fd0 tx=0x7f6d98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== osd.4 v2:172.21.15.187:6800/27895010 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c54004ea0 con 0x7f6c48074be0 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f41a3400640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f418c06c8f0 0x7f418c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.142+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f419c08b190 con 0x7f41ac072420 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.143+0000 7f41a3400640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f418c074b10 0x7f418c076f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.143+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f418c0775e0 con 0x7f418c074b10 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.143+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f419c08b580 con 0x7f41ac072420 2024-04-17T07:14:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.143+0000 7f41aaa00640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f418c06c8f0 0x7f418c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.143+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 --> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f341c0057e0 con 0x7f3404074bc0 2024-04-17T07:14:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.144+0000 7f3417400640 1 -- 172.21.15.67:0/3141446188 <== osd.2 v2:172.21.15.67:6824/873912528 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f341c0057e0 con 0x7f3404074bc0 2024-04-17T07:14:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.144+0000 7f41aaa00640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f418c06c8f0 0x7f418c06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0caac0 tx=0x7f419803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.144+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7f3404074bc0 msgr2=0x7f3404076fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.144+0000 7f3428a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7f3404074bc0 0x7f3404076fe0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f340406c940 msgr2=0x7f340406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f3428a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f340406c940 0x7f340406ee00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f341c180d20 tx=0x7f3418008040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 msgr2=0x7f341c0cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f3428a00640 1 --2- 172.21.15.67:0/3141446188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f341c00ab40 0x7f341c0cc8c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3424051240 tx=0x7f342406dcb0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f3423e00640 1 -- 172.21.15.67:0/3141446188 reap_dead start 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.145+0000 7f41abe00640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f418c074b10 0x7f418c076f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 shutdown_connections 2024-04-17T07:14:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 >> 172.21.15.67:0/3141446188 conn(0x7f341c056cd0 msgr2=0x7f341c03b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f7d66a00640 1 -- 172.21.15.67:0/1702211721 <== osd.0 v2:172.21.15.67:6815/2121978331 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7d70068a10 con 0x7f7d50074b10 2024-04-17T07:14:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== osd.5 v2:172.21.15.187:6816/656328999 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d94046e70 con 0x7f6d88074be0 2024-04-17T07:14:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 shutdown_connections 2024-04-17T07:14:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f3428a00640 1 -- 172.21.15.67:0/3141446188 wait complete. 2024-04-17T07:14:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f41abe00640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f418c074b10 0x7f418c076f30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f7d50074b10 msgr2=0x7f7d50076f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.146+0000 7f7d75c00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f7d50074b10 0x7f7d50076f30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d5006c8f0 msgr2=0x7f7d5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d5006c8f0 0x7f7d5006edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d701fbd70 tx=0x7f7d60008040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 msgr2=0x7f7d7015a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 --2- 172.21.15.67:0/1702211721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d7006eeb0 0x7f7d7015a5d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00e860 tx=0x7f7d5c00ed20 comp rx=0 tx=0).stop 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 --> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f75f8068b90 con 0x7f75d8074af0 2024-04-17T07:14:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d6f400640 1 -- 172.21.15.67:0/1702211721 reap_dead start 2024-04-17T07:14:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 shutdown_connections 2024-04-17T07:14:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.147+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 >> 172.21.15.67:0/1702211721 conn(0x7f7d700b23c0 msgr2=0x7f7d700b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.148+0000 7f75efe00640 1 -- 172.21.15.67:0/3375652665 <== osd.1 v2:172.21.15.67:6800/2081347185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f75f8068b90 con 0x7f75d8074af0 2024-04-17T07:14:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.148+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 shutdown_connections 2024-04-17T07:14:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.148+0000 7f7d75c00640 1 -- 172.21.15.67:0/1702211721 wait complete. 2024-04-17T07:14:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== osd.7 v2:172.21.15.187:6808/1997375062 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f418c0775e0 con 0x7f418c074b10 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f75d8074af0 msgr2=0x7f75d8076f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f75d8074af0 0x7f75d8076f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f75d806c8f0 msgr2=0x7f75d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f75d806c8f0 0x7f75d806edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f75f815c590 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 msgr2=0x7f75f80c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.149+0000 7f75fee00640 1 --2- 172.21.15.67:0/3375652665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75f8072420 0x7f75f80c7550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009960 tx=0x7f75e80046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.150+0000 7f75fce00640 1 -- 172.21.15.67:0/3375652665 reap_dead start 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.153+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 shutdown_connections 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.153+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 >> 172.21.15.67:0/3375652665 conn(0x7f75f806bf50 msgr2=0x7f75f80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.153+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 shutdown_connections 2024-04-17T07:14:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.154+0000 7f75fee00640 1 -- 172.21.15.67:0/3375652665 wait complete. 2024-04-17T07:14:05.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.163+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 --> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb18068b90 con 0x7fdaf8074a90 2024-04-17T07:14:05.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.164+0000 7fdb12000640 1 -- 172.21.15.67:0/2589703 <== osd.3 v2:172.21.15.67:6804/100610741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb18068b90 con 0x7fdaf8074a90 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fdaf8074a90 msgr2=0x7fdaf8076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fdaf8074a90 0x7fdaf8076eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdaf806c8f0 msgr2=0x7fdaf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdaf806c8f0 0x7fdaf806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb1815df00 tx=0x7fdb0c007400 comp rx=0 tx=0).stop 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 msgr2=0x7fdb18202330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.166+0000 7fdb20800640 1 --2- 172.21.15.67:0/2589703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb180cf640 0x7fdb18202330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb0402bf50 tx=0x7fdb040046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.167+0000 7fdb1e800640 1 -- 172.21.15.67:0/2589703 reap_dead start 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.167+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 shutdown_connections 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.167+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 >> 172.21.15.67:0/2589703 conn(0x7fdb1806cbd0 msgr2=0x7fdb180ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.167+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 shutdown_connections 2024-04-17T07:14:05.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.169+0000 7fdb20800640 1 -- 172.21.15.67:0/2589703 wait complete. 2024-04-17T07:14:05.192 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:05.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T07:14:05.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.173+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 --> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c44002dd0 con 0x7f6c48074be0 2024-04-17T07:14:05.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.173+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 --> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f38d80b9580 con 0x7f38b8074bc0 2024-04-17T07:14:05.194 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:05.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T07:14:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.174+0000 7f6c65600640 1 -- 172.21.15.67:0/4161502917 <== osd.4 v2:172.21.15.187:6800/27895010 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6c44002dd0 con 0x7f6c48074be0 2024-04-17T07:14:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.174+0000 7f38cf400640 1 -- 172.21.15.67:0/126363050 <== osd.6 v2:172.21.15.187:6817/3359222512 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f38d80b9580 con 0x7f38b8074bc0 2024-04-17T07:14:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.174+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f6c48074be0 msgr2=0x7f6c48077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.174+0000 7f6c70200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f6c48074be0 0x7f6c48077000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6c4806c9c0 msgr2=0x7f6c4806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6c4806c9c0 0x7f6c4806ee80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c60003e40 tx=0x7f6c6000d040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c6815d4a0 msgr2=0x7f6c680cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 --2- 172.21.15.67:0/4161502917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c6815d4a0 0x7f6c680cde90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c680c82c0 tx=0x7f6c54001970 comp rx=0 tx=0).stop 2024-04-17T07:14:05.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c6e200640 1 -- 172.21.15.67:0/4161502917 reap_dead start 2024-04-17T07:14:05.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 --> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6da815ddf0 con 0x7f6d88074be0 2024-04-17T07:14:05.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 shutdown_connections 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 >> 172.21.15.67:0/4161502917 conn(0x7f6c6806cbd0 msgr2=0x7f6c680cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7f38b8074bc0 msgr2=0x7f38b8076fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7f38b8074bc0 0x7f38b8076fe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38b806c940 msgr2=0x7f38b806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38b806c940 0x7f38b806ee00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f38d0003fd0 tx=0x7f38d00039f0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 msgr2=0x7f38d8201a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 --2- 172.21.15.67:0/126363050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d80ba400 0x7f38d8201a70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b3f0 tx=0x7f38c800b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38d7e00640 1 -- 172.21.15.67:0/126363050 reap_dead start 2024-04-17T07:14:05.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 shutdown_connections 2024-04-17T07:14:05.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 shutdown_connections 2024-04-17T07:14:05.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f6c70200640 1 -- 172.21.15.67:0/4161502917 wait complete. 2024-04-17T07:14:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 >> 172.21.15.67:0/126363050 conn(0x7f38d806cac0 msgr2=0x7f38d80c8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.175+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 shutdown_connections 2024-04-17T07:14:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.176+0000 7f38dcc00640 1 -- 172.21.15.67:0/126363050 wait complete. 2024-04-17T07:14:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.176+0000 7f6d9ea00640 1 -- 172.21.15.67:0/2491553719 <== osd.5 v2:172.21.15.187:6816/656328999 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6da815ddf0 con 0x7f6d88074be0 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.176+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f6d88074be0 msgr2=0x7f6d88077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.176+0000 7f6dad600640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f6d88074be0 0x7f6d88077000 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6d8806c9c0 msgr2=0x7f6d8806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6d8806c9c0 0x7f6d8806ee80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d98006fd0 tx=0x7f6d98008040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 msgr2=0x7f6da80b5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 --2- 172.21.15.67:0/2491553719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da80d2ab0 0x7f6da80b5760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009960 tx=0x7f6d94004730 comp rx=0 tx=0).stop 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6da7400640 1 -- 172.21.15.67:0/2491553719 reap_dead start 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 shutdown_connections 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.177+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 >> 172.21.15.67:0/2491553719 conn(0x7f6da806cbd0 msgr2=0x7f6da80c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.178+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 shutdown_connections 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.178+0000 7f6dad600640 1 -- 172.21.15.67:0/2491553719 wait complete. 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.180+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 --> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41ac06e900 con 0x7f418c074b10 2024-04-17T07:14:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.181+0000 7f41a3400640 1 -- 172.21.15.67:0/1299777090 <== osd.7 v2:172.21.15.187:6808/1997375062 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41ac06e900 con 0x7f418c074b10 2024-04-17T07:14:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f418c074b10 msgr2=0x7f418c076f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f418c074b10 0x7f418c076f30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f418c06c8f0 msgr2=0x7f418c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f418c06c8f0 0x7f418c06edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41ac0caac0 tx=0x7f419803c040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 msgr2=0x7f41ac15aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 --2- 172.21.15.67:0/1299777090 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41ac072420 0x7f41ac15aaf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c390 tx=0x7f419c00c850 comp rx=0 tx=0).stop 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41abe00640 1 -- 172.21.15.67:0/1299777090 reap_dead start 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 shutdown_connections 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.182+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 >> 172.21.15.67:0/1299777090 conn(0x7f41ac06bf50 msgr2=0x7f41ac0c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.183+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 shutdown_connections 2024-04-17T07:14:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.183+0000 7f41b2600640 1 -- 172.21.15.67:0/1299777090 wait complete. 2024-04-17T07:14:05.205 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:05.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T07:14:05.206 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:05.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T07:14:05.207 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:05.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T07:14:05.208 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:05.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T07:14:05.209 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:05.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T07:14:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.495+0000 7fe0d1400640 1 Processor -- start 2024-04-17T07:14:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.495+0000 7fe0d1400640 1 -- start start 2024-04-17T07:14:05.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.497+0000 7fe0d1400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0cc071d60 0x7fe0cc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.497+0000 7fe0d1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0cc0b4a70 0x7fe0cc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.501+0000 7f7d56a00640 1 Processor -- start 2024-04-17T07:14:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.501+0000 7f7d56a00640 1 -- start start 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0d1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0cc0b53a0 con 0x7fe0cc16b9d0 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0d1400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe0cc08bdd0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0d1400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe0cc0ce0f0 con 0x7fe0cc071d60 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0cb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0cc0b4a70 0x7fe0cc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0c9e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe0cc16b9d0 0x7fe0cc072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39358/0 (socket says 172.21.15.67:39358) 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0c9e00640 1 -- 172.21.15.67:0/1997396090 learned_addr learned my addr 172.21.15.67:0/1997396090 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.498+0000 7fe0ca800640 1 --2- 172.21.15.67:0/1997396090 >> v2:172.21.15.187:3300/0 conn(0x7fe0cc071d60 0x7fe0cc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.499+0000 7fe0cb200640 1 -- 172.21.15.67:0/1997396090 >> v2:172.21.15.187:3300/0 conn(0x7fe0cc071d60 msgr2=0x7fe0cc0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.499+0000 7fe0cb200640 1 --2- 172.21.15.67:0/1997396090 >> v2:172.21.15.187:3300/0 conn(0x7fe0cc071d60 0x7fe0cc0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.499+0000 7fe0cb200640 1 -- 172.21.15.67:0/1997396090 >> v1:172.21.15.187:6789/0 conn(0x7fe0cc16b9d0 legacy=0x7fe0cc072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.499+0000 7fe0cb200640 1 -- 172.21.15.67:0/1997396090 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc0ce320 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.500+0000 7fe0cb200640 1 --2- 172.21.15.67:0/1997396090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc0cf690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc06d6d0 tx=0x7fe0c402c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed68f791d70b0bdd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.501+0000 7fe0c9400640 1 -- 172.21.15.67:0/1997396090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4039070 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.501+0000 7fe0c9400640 1 -- 172.21.15.67:0/1997396090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c402cc30 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.502+0000 7fe0d1400640 1 -- 172.21.15.67:0/1997396090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 msgr2=0x7fe0cc0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.502+0000 7fe0c9400640 1 -- 172.21.15.67:0/1997396090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4039070 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.502+0000 7fe0d1400640 1 --2- 172.21.15.67:0/1997396090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc0cf690 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc06d6d0 tx=0x7fe0c402c090 comp rx=0 tx=0).stop 2024-04-17T07:14:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.502+0000 7f7d56a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d480e32d0 0x7f7d48052c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7f7d56a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d48053190 0x7f7d4804f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7f7d56a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d480536a0 con 0x7f7d4804fdb0 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7f7d56a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7d480e36b0 con 0x7f7d480e32d0 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7f7d56a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d48053980 con 0x7f7d48053190 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7fe0d1400640 1 -- 172.21.15.67:0/1997396090 shutdown_connections 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7fe0d1400640 1 --2- 172.21.15.67:0/1997396090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc0cf690 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7fe0d1400640 1 --2- 172.21.15.67:0/1997396090 >> v2:172.21.15.187:3300/0 conn(0x7fe0cc071d60 0x7fe0cc0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.503+0000 7fe0d1400640 1 -- 172.21.15.67:0/1997396090 >> 172.21.15.67:0/1997396090 conn(0x7fe0cc0b2120 msgr2=0x7fe0cc0b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d56000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7d4804fdb0 0x7f7d48005e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50204/0 (socket says 172.21.15.67:50204) 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d56000640 1 -- 172.21.15.67:0/1856970498 learned_addr learned my addr 172.21.15.67:0/1856970498 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d55600640 1 --2- 172.21.15.67:0/1856970498 >> v2:172.21.15.187:3300/0 conn(0x7f7d480e32d0 0x7f7d48052c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d55600640 1 -- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d48053190 msgr2=0x7f7d4804f870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d55600640 1 --2- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d48053190 0x7f7d4804f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.504+0000 7f7d55600640 1 -- 172.21.15.67:0/1856970498 >> v1:172.21.15.67:6789/0 conn(0x7f7d4804fdb0 legacy=0x7f7d48005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.505+0000 7f7d55600640 1 -- 172.21.15.67:0/1856970498 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d48053c60 con 0x7f7d480e32d0 2024-04-17T07:14:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.506+0000 7fe0d1400640 1 -- 172.21.15.67:0/1997396090 shutdown_connections 2024-04-17T07:14:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.506+0000 7fe0d1400640 1 -- 172.21.15.67:0/1997396090 wait complete. 2024-04-17T07:14:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.506+0000 7f1c3ae00640 1 Processor -- start 2024-04-17T07:14:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.506+0000 7f7d55600640 1 --2- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d480e32d0 0x7f7d48052c50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d440097a0 tx=0x7f7d4402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c5785821ad3e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.506+0000 7f1c3ae00640 1 -- start start 2024-04-17T07:14:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.507+0000 7f7d4fe00640 1 -- 172.21.15.67:0/1856970498 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034070 con 0x7f7d480e32d0 2024-04-17T07:14:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f7d4fe00640 1 -- 172.21.15.67:0/1856970498 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d44030610 con 0x7f7d480e32d0 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f1c3ae00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c2c0e30b0 0x7f1c2c057d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f1c3ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 0x7f1c2c053be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f1c3ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c2c0582c0 con 0x7f1c2c054120 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f1c3ae00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1c2c058560 con 0x7f1c2c0e30b0 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f1c3ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2c058840 con 0x7f1c2c03e830 2024-04-17T07:14:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7f7d4fe00640 1 -- 172.21.15.67:0/1856970498 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034070 con 0x7f7d480e32d0 2024-04-17T07:14:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 -- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d480e32d0 msgr2=0x7f7d48052c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.508+0000 7fe0d1400640 1 Processor -- start 2024-04-17T07:14:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7fe0d1400640 1 -- start start 2024-04-17T07:14:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 --2- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d480e32d0 0x7f7d48052c50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d440097a0 tx=0x7f7d4402b960 comp rx=0 tx=0).stop 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f1c33e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1c2c0e30b0 0x7f1c2c057d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f1c38e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c2c054120 0x7f1c2c0e6c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50206/0 (socket says 172.21.15.67:50206) 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f1c38e00640 1 -- 172.21.15.67:0/1790859549 learned_addr learned my addr 172.21.15.67:0/1790859549 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 -- 172.21.15.67:0/1856970498 shutdown_connections 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 --2- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d48053190 0x7f7d4804f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 --2- 172.21.15.67:0/1856970498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d480e32d0 0x7f7d48052c50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 -- 172.21.15.67:0/1856970498 >> 172.21.15.67:0/1856970498 conn(0x7f7d48009fb0 msgr2=0x7f7d4800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f1c33400640 1 --2- 172.21.15.67:0/1790859549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 0x7f1c2c053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 -- 172.21.15.67:0/1856970498 shutdown_connections 2024-04-17T07:14:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.509+0000 7f7d56a00640 1 -- 172.21.15.67:0/1856970498 wait complete. 2024-04-17T07:14:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7f1c33400640 1 -- 172.21.15.67:0/1790859549 >> v2:172.21.15.187:3300/0 conn(0x7f1c2c0e30b0 msgr2=0x7f1c2c057d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7f1c33400640 1 --2- 172.21.15.67:0/1790859549 >> v2:172.21.15.187:3300/0 conn(0x7f1c2c0e30b0 0x7f1c2c057d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7f1c33400640 1 -- 172.21.15.67:0/1790859549 >> v1:172.21.15.67:6789/0 conn(0x7f1c2c054120 legacy=0x7f1c2c0e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 0x7fe0cc165630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0cc16b9d0 0x7fe0cc15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0b5680 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc0b4370 con 0x7fe0cc16b9d0 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7fe0d1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0cc0b4e80 con 0x7fe0cc071d60 2024-04-17T07:14:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.510+0000 7f1c33400640 1 -- 172.21.15.67:0/1790859549 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2c058b20 con 0x7f1c2c03e830 2024-04-17T07:14:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.511+0000 7fe0c9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.511+0000 7fe0c9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41934/0 (socket says 172.21.15.67:41934) 2024-04-17T07:14:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.511+0000 7fe0c9e00640 1 -- 172.21.15.67:0/3440091965 learned_addr learned my addr 172.21.15.67:0/3440091965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0ca800640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 0x7fe0cc165630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 -- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 msgr2=0x7fe0cc165630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 0x7fe0cc165630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 -- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0cc16b9d0 msgr2=0x7fe0cc15cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0cc16b9d0 0x7fe0cc15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 -- 172.21.15.67:0/3440091965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc161be0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0ca800640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 0x7fe0cc165630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7f7d56a00640 1 Processor -- start 2024-04-17T07:14:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7fe0c9e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00c4b0 tx=0x7fe0bc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7f7d56a00640 1 -- start start 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc017070 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.512+0000 7f1c33400640 1 --2- 172.21.15.67:0/1790859549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 0x7f1c2c053be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c200097a0 tx=0x7f1c2002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5723c14c7c798ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7f1c32a00640 1 -- 172.21.15.67:0/1790859549 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20034070 con 0x7f1c2c03e830 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7f1c32a00640 1 -- 172.21.15.67:0/1790859549 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c20030800 con 0x7f1c2c03e830 2024-04-17T07:14:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7f1c32a00640 1 -- 172.21.15.67:0/1790859549 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20030bd0 con 0x7f1c2c03e830 2024-04-17T07:14:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc005ce0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.513+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc007270 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7fe0d1400640 1 -- 172.21.15.67:0/3440091965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc161ec0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7fe0d1400640 1 -- 172.21.15.67:0/3440091965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc162480 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d4804fdb0 0x7f7d4804ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d480e32d0 0x7f7d4804dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d48054740 con 0x7f7d48053190 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7d48054220 con 0x7f7d4804fdb0 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f7d56a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d4803a900 con 0x7f7d480e32d0 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.515+0000 7f1a87a00640 1 Processor -- start 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 -- 172.21.15.67:0/1790859549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 msgr2=0x7f1c2c053be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/1790859549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 0x7f1c2c053be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c200097a0 tx=0x7f1c2002b940 comp rx=0 tx=0).stop 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 -- 172.21.15.67:0/1790859549 shutdown_connections 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/1790859549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c03e830 0x7f1c2c053be0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/1790859549 >> v2:172.21.15.187:3300/0 conn(0x7f1c2c0e30b0 0x7f1c2c057d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 -- 172.21.15.67:0/1790859549 >> 172.21.15.67:0/1790859549 conn(0x7f1c2c0559d0 msgr2=0x7f1c2c055de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 -- 172.21.15.67:0/1790859549 shutdown_connections 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.514+0000 7f1c3ae00640 1 -- 172.21.15.67:0/1790859549 wait complete. 2024-04-17T07:14:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.515+0000 7f1a87a00640 1 -- start start 2024-04-17T07:14:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.515+0000 7f7d54c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41938/0 (socket says 172.21.15.67:41938) 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 -- 172.21.15.67:0/2392042253 learned_addr learned my addr 172.21.15.67:0/2392042253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d55600640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d4804fdb0 0x7f7d4804ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d56000640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d480e32d0 0x7f7d4804dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 -- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d480e32d0 msgr2=0x7f7d4804dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d480e32d0 0x7f7d4804dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 -- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d4804fdb0 msgr2=0x7f7d4804ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d4804fdb0 0x7f7d4804ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.516+0000 7f7d54c00640 1 -- 172.21.15.67:0/2392042253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d48049df0 con 0x7f7d48053190 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f7d54c00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00b4a0 tx=0x7f7d3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1c3ae00640 1 Processor -- start 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a80155340 0x7f1a8015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a800b3420 0x7f1a80071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80072250 0x7f1a8015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1a800726a0 con 0x7f1a80155340 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1a800ce920 con 0x7f1a800b3420 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1a87a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1a800b3c70 con 0x7f1a80072250 2024-04-17T07:14:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.517+0000 7f1c3ae00640 1 -- start start 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014070 con 0x7f7d48053190 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d3c010610 con 0x7f7d48053190 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0109c0 con 0x7f7d48053190 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7f1a85000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a80155340 0x7f1a8015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7f1a85000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a80155340 0x7f1a8015b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41946/0 (socket says 172.21.15.67:41946) 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 -- 172.21.15.67:0/4107132447 learned_addr learned my addr 172.21.15.67:0/4107132447 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a7fe00640 1 --2- 172.21.15.67:0/4107132447 >> v2:172.21.15.187:3300/0 conn(0x7f1a800b3420 0x7f1a80071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.518+0000 7fe0d1400640 1 -- 172.21.15.67:0/3440091965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc068a10 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85a00640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80072250 0x7f1a8015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1c3ae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c2c03e830 0x7f1c2c0de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c3ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c3ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c0e30b0 0x7f1c2c0d8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f4b83400640 1 Processor -- start 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c3ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c0e3490 con 0x7f1c2c054120 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c3ae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c03bfb0 con 0x7f1c2c03e830 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c3ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2c0078d0 con 0x7f1c2c0e30b0 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f4b83400640 1 -- start start 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c33400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c33400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41962/0 (socket says 172.21.15.67:41962) 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f1c33400640 1 -- 172.21.15.67:0/399992297 learned_addr learned my addr 172.21.15.67:0/399992297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.521+0000 7f1c33e00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c2c03e830 0x7f1c2c0de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.521+0000 7f1c33400640 1 -- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c0e30b0 msgr2=0x7f1c2c0d8840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1c33400640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c0e30b0 0x7f1c2c0d8840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1c33400640 1 -- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c2c03e830 msgr2=0x7f1c2c0de3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1c33400640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c2c03e830 0x7f1c2c0de3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1c33400640 1 -- 172.21.15.67:0/399992297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2c0d9d40 con 0x7f1c2c054120 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f7d56a00640 1 -- 172.21.15.67:0/2392042253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4804a0d0 con 0x7f7d48053190 2024-04-17T07:14:05.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f7d56a00640 1 -- 172.21.15.67:0/2392042253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4804a690 con 0x7f7d48053190 2024-04-17T07:14:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.520+0000 7f9401e00640 1 Processor -- start 2024-04-17T07:14:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.521+0000 7f9401e00640 1 -- start start 2024-04-17T07:14:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f4b83400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 0x7f4b7c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f4b83400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7c163860 con 0x7f4b7c0d17a0 2024-04-17T07:14:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f4b83400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7c0d1d40 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f4b83400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c0d2c30 con 0x7f4b7c15eca0 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 -- 172.21.15.67:0/4107132447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80072250 msgr2=0x7f1a8015e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80072250 0x7f1a8015e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 -- 172.21.15.67:0/4107132447 >> v2:172.21.15.187:3300/0 conn(0x7f1a800b3420 msgr2=0x7f1a80071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 --2- 172.21.15.67:0/4107132447 >> v2:172.21.15.187:3300/0 conn(0x7f1a800b3420 0x7f1a80071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.519+0000 7f1a85000640 1 -- 172.21.15.67:0/4107132447 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a800b3f50 con 0x7f1a80155340 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.521+0000 7f1a85000640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80155340 0x7f1a8015b0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a80068290 tx=0x7f1a7802fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e608bac0f61b41e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1a7f400640 1 -- 172.21.15.67:0/4107132447 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7803d070 con 0x7f1a80155340 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1a7f400640 1 -- 172.21.15.67:0/4107132447 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a78030bc0 con 0x7f1a80155340 2024-04-17T07:14:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1a7f400640 1 -- 172.21.15.67:0/4107132447 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78038440 con 0x7f1a80155340 2024-04-17T07:14:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f30e2000640 1 Processor -- start 2024-04-17T07:14:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.522+0000 7f1c33400640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1c2002bf50 tx=0x7f1c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f4b81400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4b7c0d17a0 0x7f4b7c0b7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50218/0 (socket says 172.21.15.67:50218) 2024-04-17T07:14:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f4b81400640 1 -- 172.21.15.67:0/1128533645 learned_addr learned my addr 172.21.15.67:0/1128533645 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f9401e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 0x7f93fc06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f9401e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc163930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f7d56a00640 1 -- 172.21.15.67:0/2392042253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d48005390 con 0x7f7d48053190 2024-04-17T07:14:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f9401e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93fc166c30 con 0x7f93fc0cffe0 2024-04-17T07:14:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f9401e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f93fc06fde0 con 0x7f93fc071f30 2024-04-17T07:14:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f30e2000640 1 -- start start 2024-04-17T07:14:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f4b80a00640 1 --2- 172.21.15.67:0/1128533645 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 0x7f4b7c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f9401e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93fc163e70 con 0x7f93fc1634a0 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f1a87a00640 1 -- 172.21.15.67:0/4107132447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80155340 msgr2=0x7f1a8015b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.523+0000 7f1a87a00640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80155340 0x7f1a8015b0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a80068290 tx=0x7f1a7802fe10 comp rx=0 tx=0).stop 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 -- 172.21.15.67:0/4107132447 shutdown_connections 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80072250 0x7f1a8015e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 --2- 172.21.15.67:0/4107132447 >> v2:172.21.15.187:3300/0 conn(0x7f1a800b3420 0x7f1a80071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 --2- 172.21.15.67:0/4107132447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80155340 0x7f1a8015b0b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f93fa000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 0x7f93fc06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f93fa000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 0x7f93fc06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53900/0 (socket says 172.21.15.67:53900) 2024-04-17T07:14:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 -- 172.21.15.67:0/4107132447 >> 172.21.15.67:0/4107132447 conn(0x7f1a8006bce0 msgr2=0x7f1a8006c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f93fa000640 1 -- 172.21.15.67:0/1222936700 learned_addr learned my addr 172.21.15.67:0/1222936700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f4b80a00640 1 -- 172.21.15.67:0/1128533645 >> v1:172.21.15.187:6789/0 conn(0x7f4b7c0cfa10 legacy=0x7f4b7c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f4b80a00640 1 -- 172.21.15.67:0/1128533645 >> v1:172.21.15.67:6789/0 conn(0x7f4b7c0d17a0 legacy=0x7f4b7c0b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f4b80a00640 1 -- 172.21.15.67:0/1128533645 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c0d2f10 con 0x7f4b7c15eca0 2024-04-17T07:14:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f93fb400640 1 --2- 172.21.15.67:0/1222936700 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc163930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0bc01a070 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 -- 172.21.15.67:0/4107132447 shutdown_connections 2024-04-17T07:14:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.524+0000 7f1a87a00640 1 -- 172.21.15.67:0/4107132447 wait complete. 2024-04-17T07:14:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20034040 con 0x7f1c2c054120 2024-04-17T07:14:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c200443f0 con 0x7f1c2c054120 2024-04-17T07:14:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200447e0 con 0x7f1c2c054120 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7fe0c2a00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 0x7fe0ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe0bc08adb0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d3c017070 con 0x7f7d48053190 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.525+0000 7f93fb400640 1 -- 172.21.15.67:0/1222936700 >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 msgr2=0x7f93fc06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.525+0000 7f93fb400640 1 --2- 172.21.15.67:0/1222936700 >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 0x7f93fc06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.525+0000 7f93fb400640 1 -- 172.21.15.67:0/1222936700 >> v1:172.21.15.187:6789/0 conn(0x7f93fc0cffe0 legacy=0x7f93fc167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.525+0000 7f93fb400640 1 -- 172.21.15.67:0/1222936700 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93fc06f820 con 0x7f93fc1634a0 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f30e2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f30e2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 0x7f30d4006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f4b80a00640 1 --2- 172.21.15.67:0/1128533645 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 0x7f4b7c0d39f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097d0 tx=0x7f4b6802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9cdd458492c1afc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f4b7b400640 1 -- 172.21.15.67:0/1128533645 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68035020 con 0x7f4b7c15eca0 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f4b7b400640 1 -- 172.21.15.67:0/1128533645 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b68031830 con 0x7f4b7c15eca0 2024-04-17T07:14:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.526+0000 7f4b7b400640 1 -- 172.21.15.67:0/1128533645 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68031c40 con 0x7f4b7c15eca0 2024-04-17T07:14:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2c0da020 con 0x7f1c2c054120 2024-04-17T07:14:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2c0da5e0 con 0x7f1c2c054120 2024-04-17T07:14:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30e2000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d40544b0 con 0x7f30d40df210 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30e2000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f30d40541d0 con 0x7f30d4053c60 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30e2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f30d4005f20 con 0x7f30d4050190 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30db400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f30d40df210 0x7f30d404fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39392/0 (socket says 172.21.15.67:39392) 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30daa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30daa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41274/0 (socket says 172.21.15.67:41274) 2024-04-17T07:14:05.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30daa00640 1 -- 172.21.15.67:0/999371067 learned_addr learned my addr 172.21.15.67:0/999371067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f1a87a00640 1 Processor -- start 2024-04-17T07:14:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.527+0000 7f30dbe00640 1 --2- 172.21.15.67:0/999371067 >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 0x7f30d4006a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f1a87a00640 1 -- start start 2024-04-17T07:14:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f4b83400640 1 -- 172.21.15.67:0/1128533645 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 msgr2=0x7f4b7c0d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f4b83400640 1 --2- 172.21.15.67:0/1128533645 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 0x7f4b7c0d39f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097d0 tx=0x7f4b6802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7fe0ca800640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 0x7fe0ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f30daa00640 1 -- 172.21.15.67:0/999371067 >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 msgr2=0x7f30d4006a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f4b83400640 1 -- 172.21.15.67:0/1128533645 shutdown_connections 2024-04-17T07:14:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f4b83400640 1 --2- 172.21.15.67:0/1128533645 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c15eca0 0x7f4b7c0d39f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f4b83400640 1 -- 172.21.15.67:0/1128533645 >> 172.21.15.67:0/1128533645 conn(0x7f4b7c067600 msgr2=0x7f4b7c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f93fb400640 1 --2- 172.21.15.67:0/1222936700 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc163930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009d90 tx=0x7f93f402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=716932dc1d71042b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f7d4d600640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 0x7f7d3006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f30daa00640 1 --2- 172.21.15.67:0/999371067 >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 0x7f30d4006a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f4b83400640 1 -- 172.21.15.67:0/1128533645 shutdown_connections 2024-04-17T07:14:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7d3c08b550 con 0x7f7d48053190 2024-04-17T07:14:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c2c0057d0 con 0x7f1c2c054120 2024-04-17T07:14:05.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f30daa00640 1 -- 172.21.15.67:0/999371067 >> v1:172.21.15.187:6789/0 conn(0x7f30d40df210 legacy=0x7f30d404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f30da000640 1 -- 172.21.15.67:0/999371067 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 294619801 0 0) 0x7f30d40544b0 con 0x7f30d40df210 2024-04-17T07:14:05.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.528+0000 7f30daa00640 1 -- 172.21.15.67:0/999371067 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d4006200 con 0x7f30d4050190 2024-04-17T07:14:05.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f30dbe00640 1 --2- 172.21.15.67:0/999371067 >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 0x7f30d4006a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 0x7f1a801486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80155340 0x7f1a80148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800b4690 con 0x7f1a80072250 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1a800b4510 con 0x7f1a800b3420 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f1a87a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1a800b38d0 con 0x7f1a80155340 2024-04-17T07:14:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.529+0000 7f4b83400640 1 -- 172.21.15.67:0/1128533645 wait complete. 2024-04-17T07:14:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f93f9600640 1 -- 172.21.15.67:0/1222936700 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4039070 con 0x7f93fc1634a0 2024-04-17T07:14:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7fe0ca800640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 0x7fe0ac06edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0c9f30 tx=0x7fe0b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f1a85000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f93f9600640 1 -- 172.21.15.67:0/1222936700 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f402cc50 con 0x7f93fc1634a0 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f93f9600640 1 -- 172.21.15.67:0/1222936700 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4034440 con 0x7f93fc1634a0 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f1a85000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41982/0 (socket says 172.21.15.67:41982) 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f1a85000640 1 -- 172.21.15.67:0/1864590590 learned_addr learned my addr 172.21.15.67:0/1864590590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f1a85a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80155340 0x7f1a80148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.530+0000 7f1a7fe00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 0x7f1a801486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 -- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80155340 msgr2=0x7f1a80148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f7d55600640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 0x7f7d3006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 -- 172.21.15.67:0/1222936700 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 msgr2=0x7f93fc163930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 --2- 172.21.15.67:0/1222936700 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc163930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009d90 tx=0x7f93f402bea0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80155340 0x7f1a80148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 -- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 msgr2=0x7f1a801486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 0x7f1a801486a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 -- 172.21.15.67:0/1864590590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a80151490 con 0x7f1a80072250 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a7fe00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 0x7f1a801486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 -- 172.21.15.67:0/1222936700 shutdown_connections 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 --2- 172.21.15.67:0/1222936700 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc163930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 --2- 172.21.15.67:0/1222936700 >> v2:172.21.15.187:3300/0 conn(0x7f93fc071f30 0x7f93fc06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f1a85000640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 -- 172.21.15.67:0/1222936700 >> 172.21.15.67:0/1222936700 conn(0x7f93fc067ee0 msgr2=0x7f93fc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 -- 172.21.15.67:0/1222936700 shutdown_connections 2024-04-17T07:14:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.531+0000 7f9401e00640 1 -- 172.21.15.67:0/1222936700 wait complete. 2024-04-17T07:14:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f30daa00640 1 --2- 172.21.15.67:0/999371067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a7873dc8a1d5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f891b200640 1 Processor -- start 2024-04-17T07:14:05.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f4b83400640 1 Processor -- start 2024-04-17T07:14:05.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f7d55600640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 0x7f7d3006eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d44036000 tx=0x7f7d44008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f891b200640 1 -- start start 2024-04-17T07:14:05.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f4b83400640 1 -- start start 2024-04-17T07:14:05.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f30da000640 1 -- 172.21.15.67:0/999371067 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8035020 con 0x7f30d4050190 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f30da000640 1 -- 172.21.15.67:0/999371067 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c8031640 con 0x7f30d4050190 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.532+0000 7f30da000640 1 -- 172.21.15.67:0/999371067 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c802d040 con 0x7f30d4050190 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 -- 172.21.15.67:0/999371067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 msgr2=0x7f30d4038070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 --2- 172.21.15.67:0/999371067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f9401e00640 1 Processor -- start 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 -- 172.21.15.67:0/999371067 shutdown_connections 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 --2- 172.21.15.67:0/999371067 >> v2:172.21.15.67:3300/0 conn(0x7f30d4053c60 0x7f30d4006a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 --2- 172.21.15.67:0/999371067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4050190 0x7f30d4038070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f30e2000640 1 -- 172.21.15.67:0/999371067 >> 172.21.15.67:0/999371067 conn(0x7f30d4009fa0 msgr2=0x7f30d400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f9401e00640 1 -- start start 2024-04-17T07:14:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f4b83400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 0x7f4b7c0b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.533+0000 7f4b83400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 0x7f4b7c1f91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f4b83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b7c15eca0 0x7f4b7c0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f4b83400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c15f080 con 0x7f4b7c15eca0 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f4b83400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c0d34d0 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f4b83400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c15f200 con 0x7f4b7c0d17a0 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc071f30 0x7f93fc1507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f891b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c0ae860 0x7f891c0aed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f891b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891c06ffe0 con 0x7f891c0ce5a0 2024-04-17T07:14:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f891b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891c06d270 con 0x7f891c15c9f0 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f891b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f891c02d490 con 0x7f891c0ae860 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f8919e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f891c0ce5a0 0x7f891c0ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50230/0 (socket says 172.21.15.67:50230) 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f8919e00640 1 -- 172.21.15.67:0/1529765860 learned_addr learned my addr 172.21.15.67:0/1529765860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc1fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc167820 con 0x7f93fc0cffe0 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc06f4f0 con 0x7f93fc071f30 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f9401e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93fc0ae940 con 0x7f93fc1634a0 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f30e2000640 1 -- 172.21.15.67:0/999371067 shutdown_connections 2024-04-17T07:14:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.534+0000 7f30e2000640 1 -- 172.21.15.67:0/999371067 wait complete. 2024-04-17T07:14:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41992/0 (socket says 172.21.15.67:41992) 2024-04-17T07:14:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 -- 172.21.15.67:0/2063333923 learned_addr learned my addr 172.21.15.67:0/2063333923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe0bc058b80 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fb400640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc1fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f4b7be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 0x7f4b7c1f91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f4b7be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 0x7f4b7c1f91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41298/0 (socket says 172.21.15.67:41298) 2024-04-17T07:14:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f4b7be00640 1 -- 172.21.15.67:0/2967451194 learned_addr learned my addr 172.21.15.67:0/2967451194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f4b80a00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 0x7f4b7c0b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 -- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 msgr2=0x7f93fc1fbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc1fbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 -- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc071f30 msgr2=0x7f93fc1507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc071f30 0x7f93fc1507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f93fa000640 1 -- 172.21.15.67:0/2063333923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93fc1fc8b0 con 0x7f93fc0cffe0 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1987192896 0 0) 0x7f891c06d270 con 0x7f891c15c9f0 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8900003520 con 0x7f891c15c9f0 2024-04-17T07:14:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.535+0000 7f4b81400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b7c15eca0 0x7f4b7c0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f93fa000640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f93e800e930 tx=0x7f93e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8920e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c0ae860 0x7f891c0aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f4b80a00640 1 -- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 msgr2=0x7f4b7c1f91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f4b80a00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 0x7f4b7c1f91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f4b80a00640 1 -- 172.21.15.67:0/2967451194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b7c15eca0 msgr2=0x7f4b7c0c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41801584 0 0) 0x7f8900003520 con 0x7f891c15c9f0 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c0ae860 msgr2=0x7f891c0aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c0ae860 0x7f891c0aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 >> v1:172.21.15.67:6789/0 conn(0x7f891c0ce5a0 legacy=0x7f891c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 --> v1:172.21.15.187:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c02d770 con 0x7f891c15c9f0 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.1 v1:172.21.15.187:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8904001670 con 0x7f891c15c9f0 2024-04-17T07:14:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 >> v1:172.21.15.187:6789/0 conn(0x7f891c15c9f0 legacy=0x7f891c070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89000039b0 0x7f8900003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f4b80a00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b7c15eca0 0x7f4b7c0c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f4b80a00640 1 -- 172.21.15.67:0/2967451194 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c0cabc0 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89000043a0 0x7f8900004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8900005120 0x7f890000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f891c02d490 con 0x7f89000043a0 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f890000c060 con 0x7f8900005120 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.536+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f890000c260 con 0x7f89000039b0 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89000043a0 0x7f8900004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8920e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8900005120 0x7f890000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8919e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89000039b0 0x7f8900003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f30e2000640 1 Processor -- start 2024-04-17T07:14:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 -- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89000039b0 msgr2=0x7f8900003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89000039b0 0x7f8900003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 -- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8900005120 msgr2=0x7f890000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8900005120 0x7f890000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f8913e00640 1 -- 172.21.15.67:0/1529765860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c02d770 con 0x7f89000043a0 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.537+0000 7f4b80a00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 0x7f4b7c0b56c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b68033000 tx=0x7f4b680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68040070 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b68039980 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68035040 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f30e2000640 1 -- start start 2024-04-17T07:14:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f8913e00640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89000043a0 0x7f8900004880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f890400c160 tx=0x7f8900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7ecd3af5b047c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 0x7f30d41710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40df210 0x7f30d417d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30d4037a80 con 0x7f30d4050190 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f30d4037900 con 0x7f30d40df210 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30e2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f30d40df5f0 con 0x7f30d4053c60 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c0c9b20 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.538+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7c0ca0e0 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.539+0000 7f30db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42002/0 (socket says 172.21.15.67:42002) 2024-04-17T07:14:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 -- 172.21.15.67:0/1406177163 learned_addr learned my addr 172.21.15.67:0/1406177163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30daa00640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 0x7f30d41710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 -- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 msgr2=0x7f30d41710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 0x7f30d41710a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 -- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40df210 msgr2=0x7f30d417d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40df210 0x7f30d417d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 -- 172.21.15.67:0/1406177163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d404d780 con 0x7f30d4050190 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30daa00640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 0x7f30d41710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.540+0000 7f30db400640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c4d0 tx=0x7f30d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.541+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b7c068ca0 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7d3c01e030 con 0x7f7d48053190 2024-04-17T07:14:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0018070 con 0x7f30d4050190 2024-04-17T07:14:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b6804b060 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.541+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78046070 con 0x7f1a80072250 2024-04-17T07:14:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904039070 con 0x7f89000043a0 2024-04-17T07:14:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a80151770 con 0x7f1a80072250 2024-04-17T07:14:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d0007500 con 0x7f30d4050190 2024-04-17T07:14:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0004a20 con 0x7f30d4050190 2024-04-17T07:14:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a78011480 con 0x7f1a80072250 2024-04-17T07:14:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.543+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a80151d30 con 0x7f1a80072250 2024-04-17T07:14:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d404da60 con 0x7f30d4050190 2024-04-17T07:14:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d41715e0 con 0x7f30d4050190 2024-04-17T07:14:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.543+0000 7f4b78c00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 0x7f4b5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.543+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890402ca50 con 0x7f89000043a0 2024-04-17T07:14:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.543+0000 7f8919400640 1 -- 172.21.15.67:0/1529765860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904007c20 con 0x7f89000043a0 2024-04-17T07:14:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8017070 con 0x7f93fc0cffe0 2024-04-17T07:14:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e800ca00 con 0x7f93fc0cffe0 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e800cdd0 con 0x7f93fc0cffe0 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93fc1fcb90 con 0x7f93fc0cffe0 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.542+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93fc0af8b0 con 0x7f93fc0cffe0 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4b68079360 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93fc068ca0 con 0x7f93fc0cffe0 2024-04-17T07:14:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d4005380 con 0x7f30d4050190 2024-04-17T07:14:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f891b200640 1 -- 172.21.15.67:0/1529765860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89000043a0 msgr2=0x7f8900004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f891b200640 1 --2- 172.21.15.67:0/1529765860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89000043a0 0x7f8900004880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f890400c160 tx=0x7f8900005830 comp rx=0 tx=0).stop 2024-04-17T07:14:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.544+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a78046070 con 0x7f1a80072250 2024-04-17T07:14:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.545+0000 7f4b7be00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 0x7f4b5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.545+0000 7f8920e00640 1 -- 172.21.15.67:0/1529765860 reap_dead start 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f891b200640 1 -- 172.21.15.67:0/1529765860 shutdown_connections 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f891b200640 1 -- 172.21.15.67:0/1529765860 >> 172.21.15.67:0/1529765860 conn(0x7f891c0d1bb0 msgr2=0x7f891c0d1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.545+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a58005290 con 0x7f1a80072250 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f891b200640 1 -- 172.21.15.67:0/1529765860 shutdown_connections 2024-04-17T07:14:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f891b200640 1 -- 172.21.15.67:0/1529765860 wait complete. 2024-04-17T07:14:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f4b7be00640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 0x7f4b5c06edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c14dca0 tx=0x7f4b6c007ef0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.545+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1c2004b060 con 0x7f1c2c054120 2024-04-17T07:14:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.546+0000 7f1c27e00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 0x7f1c1406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.547+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1c20079d40 con 0x7f1c2c054120 2024-04-17T07:14:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.547+0000 7f1c33e00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 0x7f1c1406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.548+0000 7f1c33e00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 0x7f1c1406ee30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006fd0 tx=0x7f1c28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.548+0000 7f891b200640 1 Processor -- start 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.549+0000 7f891b200640 1 -- start start 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89000043a0 0x7f891c0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c1583d0 0x7f891c1f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f891c071190 con 0x7f891c1effd0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f891c070560 con 0x7f89000043a0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.550+0000 7f891b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f891c071310 con 0x7f891c1583d0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42018/0 (socket says 172.21.15.67:42018) 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 -- 172.21.15.67:0/4187136962 learned_addr learned my addr 172.21.15.67:0/4187136962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8919e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89000043a0 0x7f891c0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8920e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c1583d0 0x7f891c1f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 -- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c1583d0 msgr2=0x7f891c1f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c1583d0 0x7f891c1f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 -- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89000043a0 msgr2=0x7f891c0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89000043a0 0x7f891c0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.551+0000 7f8913e00640 1 -- 172.21.15.67:0/4187136962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c158850 con 0x7f891c1effd0 2024-04-17T07:14:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.552+0000 7f8913e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f890400c280 tx=0x7f8904000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.553+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1c2003f070 con 0x7f1c2c054120 2024-04-17T07:14:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.556+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a78049070 con 0x7f1a80072250 2024-04-17T07:14:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.556+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f93e801a070 con 0x7f93fc0cffe0 2024-04-17T07:14:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f93f3400640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 0x7f93dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f93e808c320 con 0x7f93fc0cffe0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904042020 con 0x7f891c1effd0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89040052e0 con 0x7f891c1effd0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904039040 con 0x7f891c1effd0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891c1f42f0 con 0x7f891c1effd0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c0cbba0 con 0x7f891c1effd0 2024-04-17T07:14:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f93faa00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 0x7f93dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.559+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f30d001b070 con 0x7f30d4050190 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.559+0000 7f93faa00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 0x7f93dc06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec0079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.557+0000 7f1a7cc00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 0x7f1a5c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.558+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1a780ad770 con 0x7f1a80072250 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.560+0000 7f1a7fe00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 0x7f1a5c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.560+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f891c068ca0 con 0x7f891c1effd0 2024-04-17T07:14:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.561+0000 7f30cf400640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 0x7f30bc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.561+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f30d008b4d0 con 0x7f30d4050190 2024-04-17T07:14:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.561+0000 7f30daa00640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 0x7f30bc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.561+0000 7f1a7fe00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 0x7f1a5c06efb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1a80155e50 tx=0x7f1a7000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.562+0000 7f30daa00640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 0x7f30bc06ee80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f30c8036000 tx=0x7f30c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.564+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4b680abe20 con 0x7f4b7c0cfa10 2024-04-17T07:14:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.564+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1a78050020 con 0x7f1a80072250 2024-04-17T07:14:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.570+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f93e805a7c0 con 0x7f93fc0cffe0 2024-04-17T07:14:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.583+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f30d0022020 con 0x7f30d4050190 2024-04-17T07:14:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.583+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8904045020 con 0x7f891c1effd0 2024-04-17T07:14:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.584+0000 7f8912000640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 0x7f88f806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.585+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8904079940 con 0x7f891c1effd0 2024-04-17T07:14:05.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.585+0000 7f8919e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 0x7f88f806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:05.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.586+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f890404c030 con 0x7f891c1effd0 2024-04-17T07:14:05.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.587+0000 7f8919e00640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 0x7f88f806ee30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f890c007fd0 tx=0x7f890c007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.971+0000 7fe0d1400640 1 -- 172.21.15.67:0/3440091965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe0cc0b1ad0 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.975 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.972+0000 7fe0c2a00640 1 -- 172.21.15.67:0/3440091965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe0bc05ce80 con 0x7fe0cc0b4a70 2024-04-17T07:14:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.976+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 msgr2=0x7fe0ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.976+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 0x7fe0ac06edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0c9f30 tx=0x7fe0b8008040 comp rx=0 tx=0).stop 2024-04-17T07:14:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.976+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 msgr2=0x7fe0cc1614e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.976+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00c4b0 tx=0x7fe0bc00c970 comp rx=0 tx=0).stop 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 shutdown_connections 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0cc16b9d0 0x7fe0cc15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0cc0b4a70 0x7fe0cc1614e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0ac06c8f0 0x7fe0ac06edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/3440091965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0cc071d60 0x7fe0cc165630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.977+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 >> 172.21.15.67:0/3440091965 conn(0x7fe0cc0b2120 msgr2=0x7fe0cc0cfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.978+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 shutdown_connections 2024-04-17T07:14:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.978+0000 7fe0b7e00640 1 -- 172.21.15.67:0/3440091965 wait complete. 2024-04-17T07:14:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.978+0000 7f7d56a00640 1 -- 172.21.15.67:0/2392042253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7d480e5200 con 0x7f7d48053190 2024-04-17T07:14:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.979+0000 7f7d4d600640 1 -- 172.21.15.67:0/2392042253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7d3c059960 con 0x7f7d48053190 2024-04-17T07:14:05.981 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.982+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 msgr2=0x7f7d3006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.982+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 0x7f7d3006eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d44036000 tx=0x7f7d44008030 comp rx=0 tx=0).stop 2024-04-17T07:14:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.983+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 msgr2=0x7f7d480d1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.983+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00b4a0 tx=0x7f7d3c00b960 comp rx=0 tx=0).stop 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.983+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 shutdown_connections 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.983+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7d3006c9f0 0x7f7d3006eeb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.984+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d480e32d0 0x7f7d4804dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.984+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d48053190 0x7f7d480d1d00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.984+0000 7f7d42a00640 1 --2- 172.21.15.67:0/2392042253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d4804fdb0 0x7f7d4804ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.984+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 >> 172.21.15.67:0/2392042253 conn(0x7f7d48009fb0 msgr2=0x7f7d480e5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.984+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 shutdown_connections 2024-04-17T07:14:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:05.985+0000 7f7d42a00640 1 -- 172.21.15.67:0/2392042253 wait complete. 2024-04-17T07:14:06.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.006+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1c2c0df980 con 0x7f1c2c054120 2024-04-17T07:14:06.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.1 2024-04-17T07:14:06.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.006+0000 7f1c27e00640 1 -- 172.21.15.67:0/399992297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1c2007c3a0 con 0x7f1c2c054120 2024-04-17T07:14:06.007 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-04-17T07:14:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.010+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 msgr2=0x7f1c1406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.010+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f93fc1601f0 con 0x7f93fc0cffe0 2024-04-17T07:14:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.010+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 0x7f1c1406ee30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006fd0 tx=0x7f1c28008040 comp rx=0 tx=0).stop 2024-04-17T07:14:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.010+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 msgr2=0x7f1c2c0d8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.010+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1c2002bf50 tx=0x7f1c200046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.011+0000 7f93f3400640 1 -- 172.21.15.67:0/2063333923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f93e805a0c0 con 0x7f93fc0cffe0 2024-04-17T07:14:06.012 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.2 2024-04-17T07:14:06.012 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 shutdown_connections 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1c1406c970 0x7f1c1406ee30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c2c0e30b0 0x7f1c2c0d8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2c054120 0x7f1c2c0d8300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 --2- 172.21.15.67:0/399992297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c2c03e830 0x7f1c2c0de3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 >> 172.21.15.67:0/399992297 conn(0x7f1c2c0559d0 msgr2=0x7f1c2c039780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 shutdown_connections 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.012+0000 7f1c3ae00640 1 -- 172.21.15.67:0/399992297 wait complete. 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 msgr2=0x7f93dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 0x7f93dc06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec0079b0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 msgr2=0x7f93fc0af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f93e800e930 tx=0x7f93e800edf0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 shutdown_connections 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93dc06c8f0 0x7f93dc06edb0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec0079b0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc1634a0 0x7f93fc1fbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93fc0cffe0 0x7f93fc0af370 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 --2- 172.21.15.67:0/2063333923 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc071f30 0x7f93fc1507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.014+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 >> 172.21.15.67:0/2063333923 conn(0x7f93fc067ee0 msgr2=0x7f93fc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.015+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 shutdown_connections 2024-04-17T07:14:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.015+0000 7f9401e00640 1 -- 172.21.15.67:0/2063333923 wait complete. 2024-04-17T07:14:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.016+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1a58005850 con 0x7f1a80072250 2024-04-17T07:14:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.017+0000 7f1a7cc00640 1 -- 172.21.15.67:0/1864590590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a7807b750 con 0x7f1a80072250 2024-04-17T07:14:06.017 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:06.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.018+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4b7c072dd0 con 0x7f4b7c0cfa10 2024-04-17T07:14:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.019+0000 7f4b78c00640 1 -- 172.21.15.67:0/2967451194 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4b6807b2c0 con 0x7f4b7c0cfa10 2024-04-17T07:14:06.019 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.019+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 msgr2=0x7f1a5c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.019+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 0x7f1a5c06efb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1a80155e50 tx=0x7f1a7000b040 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 msgr2=0x7f1a801538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1a78000c00 tx=0x7f1a780046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 shutdown_connections 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1a5c06caf0 0x7f1a5c06efb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a80155340 0x7f1a80148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a800b3420 0x7f1a801486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 --2- 172.21.15.67:0/1864590590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a80072250 0x7f1a801538c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 >> 172.21.15.67:0/1864590590 conn(0x7f1a8006bce0 msgr2=0x7f1a8015d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 shutdown_connections 2024-04-17T07:14:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.020+0000 7f1a87a00640 1 -- 172.21.15.67:0/1864590590 wait complete. 2024-04-17T07:14:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.022+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 msgr2=0x7f4b5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.022+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 0x7f4b5c06edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c14dca0 tx=0x7f4b6c007ef0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 msgr2=0x7f4b7c0b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 0x7f4b7c0b56c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b68033000 tx=0x7f4b680046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 shutdown_connections 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b7c15eca0 0x7f4b7c0c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4b5c06c8f0 0x7f4b5c06edb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b7c0d17a0 0x7f4b7c1f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 --2- 172.21.15.67:0/2967451194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b7c0cfa10 0x7f4b7c0b56c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 >> 172.21.15.67:0/2967451194 conn(0x7f4b7c067600 msgr2=0x7f4b7c15dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 shutdown_connections 2024-04-17T07:14:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.023+0000 7f4b83400640 1 -- 172.21.15.67:0/2967451194 wait complete. 2024-04-17T07:14:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.024+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f891c0b7a10 con 0x7f891c1effd0 2024-04-17T07:14:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.025+0000 7f8912000640 1 -- 172.21.15.67:0/4187136962 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f890407b8a0 con 0x7f891c1effd0 2024-04-17T07:14:06.025 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.027+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 msgr2=0x7f88f806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.027+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 0x7f88f806ee30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f890c007fd0 tx=0x7f890c007e40 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.027+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 msgr2=0x7f891c157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.027+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f890400c280 tx=0x7f8904000fe0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 shutdown_connections 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88f806c970 0x7f88f806ee30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f891c1583d0 0x7f891c1f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f891c1effd0 0x7f891c157e90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 --2- 172.21.15.67:0/4187136962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89000043a0 0x7f891c0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 >> 172.21.15.67:0/4187136962 conn(0x7f891c0d1bb0 msgr2=0x7f891c02c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 shutdown_connections 2024-04-17T07:14:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.028+0000 7f891b200640 1 -- 172.21.15.67:0/4187136962 wait complete. 2024-04-17T07:14:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.037+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f30d40db5b0 con 0x7f30d4050190 2024-04-17T07:14:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.038+0000 7f30cf400640 1 -- 172.21.15.67:0/1406177163 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30d00598e0 con 0x7f30d4050190 2024-04-17T07:14:06.038 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-04-17T07:14:06.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.0 2024-04-17T07:14:06.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.3 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.041+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 msgr2=0x7f30bc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.041+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 0x7f30bc06ee80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f30c8036000 tx=0x7f30c8008030 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 msgr2=0x7f30d4170010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c4d0 tx=0x7f30d000c990 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 shutdown_connections 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40df210 0x7f30d417d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f30bc06c9c0 0x7f30bc06ee80 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f30c8036000 tx=0x7f30c8008030 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d4053c60 0x7f30d41710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 --2- 172.21.15.67:0/1406177163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d4050190 0x7f30d4170010 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 >> 172.21.15.67:0/1406177163 conn(0x7f30d4009fa0 msgr2=0x7f30d4036580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:06.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.6 2024-04-17T07:14:06.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 shutdown_connections 2024-04-17T07:14:06.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:06.042+0000 7f30e2000640 1 -- 172.21.15.67:0/1406177163 wait complete. 2024-04-17T07:14:06.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-04-17T07:14:06.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.5 2024-04-17T07:14:06.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.4 2024-04-17T07:14:07.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T07:14:07.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T07:14:07.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T07:14:07.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T07:14:07.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T07:14:07.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T07:14:07.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T07:14:07.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T07:14:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.311+0000 7f3b8a800640 1 Processor -- start 2024-04-17T07:14:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.311+0000 7f3b8a800640 1 -- start start 2024-04-17T07:14:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.313+0000 7f3b8a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b8406fb00 0x7f3b840cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.313+0000 7f3b8a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.313+0000 7f3b8a800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8406ff90 con 0x7f3b8416b920 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.313+0000 7f3b8a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3b840727f0 con 0x7f3b84071e60 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.313+0000 7f3b8a800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3b84070810 con 0x7f3b8406fb00 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.314+0000 7f3b89e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.314+0000 7f3b89e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42030/0 (socket says 172.21.15.67:42030) 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.314+0000 7f3b89e00640 1 -- 172.21.15.67:0/4087955530 learned_addr learned my addr 172.21.15.67:0/4087955530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.315+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253229401 0 0) 0x7f3b8406ff90 con 0x7f3b8416b920 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.315+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b68003500 con 0x7f3b8416b920 2024-04-17T07:14:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.314+0000 7f3b88a00640 1 --2- 172.21.15.67:0/4087955530 >> v2:172.21.15.187:3300/0 conn(0x7f3b8406fb00 0x7f3b840cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.316+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1203760229 0 0) 0x7f3b68003500 con 0x7f3b8416b920 2024-04-17T07:14:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.316+0000 7f3b88a00640 1 -- 172.21.15.67:0/4087955530 >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 msgr2=0x7f3b840722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.316+0000 7f3b88a00640 1 --2- 172.21.15.67:0/4087955530 >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.316+0000 7f3b88a00640 1 -- 172.21.15.67:0/4087955530 >> v1:172.21.15.67:6789/0 conn(0x7f3b8416b920 legacy=0x7f3b8406e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.317+0000 7f3b88a00640 1 -- 172.21.15.67:0/4087955530 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b84070250 con 0x7f3b8406fb00 2024-04-17T07:14:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.318+0000 7f3b89e00640 1 --2- 172.21.15.67:0/4087955530 >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.320+0000 7f3b88a00640 1 --2- 172.21.15.67:0/4087955530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b8406fb00 0x7f3b840cee40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b8416f300 tx=0x7f3b7402b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ae0e47339eb3c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.320+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74037070 con 0x7f3b8406fb00 2024-04-17T07:14:07.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.321+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b74030510 con 0x7f3b8406fb00 2024-04-17T07:14:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.321+0000 7f3b83e00640 1 -- 172.21.15.67:0/4087955530 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74037070 con 0x7f3b8406fb00 2024-04-17T07:14:07.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.322+0000 7f3b8a800640 1 -- 172.21.15.67:0/4087955530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b8406fb00 msgr2=0x7f3b840cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.322+0000 7f3b8a800640 1 --2- 172.21.15.67:0/4087955530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b8406fb00 0x7f3b840cee40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3b8416f300 tx=0x7f3b7402b730 comp rx=0 tx=0).stop 2024-04-17T07:14:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.323+0000 7f3b8a800640 1 -- 172.21.15.67:0/4087955530 shutdown_connections 2024-04-17T07:14:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.323+0000 7f3b8a800640 1 --2- 172.21.15.67:0/4087955530 >> v2:172.21.15.67:3300/0 conn(0x7f3b84071e60 0x7f3b840722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.323+0000 7f3b8a800640 1 --2- 172.21.15.67:0/4087955530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b8406fb00 0x7f3b840cee40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.323+0000 7f3b8a800640 1 -- 172.21.15.67:0/4087955530 >> 172.21.15.67:0/4087955530 conn(0x7f3b840b23c0 msgr2=0x7f3b840b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.323+0000 7f3b8a800640 1 -- 172.21.15.67:0/4087955530 shutdown_connections 2024-04-17T07:14:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.324+0000 7f3b8a800640 1 -- 172.21.15.67:0/4087955530 wait complete. 2024-04-17T07:14:07.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.327+0000 7f3b8a800640 1 Processor -- start 2024-04-17T07:14:07.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.328+0000 7f3b8a800640 1 -- start start 2024-04-17T07:14:07.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.330+0000 7f3b8a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.330+0000 7f3b8a800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 0x7f3b840ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.331+0000 7f3b8a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.331+0000 7f3b89400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.331+0000 7f3b89400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41332/0 (socket says 172.21.15.67:41332) 2024-04-17T07:14:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.331+0000 7f3b88a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 0x7f3b840ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.331+0000 7f3b88a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 0x7f3b840ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53962/0 (socket says 172.21.15.67:53962) 2024-04-17T07:14:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.332+0000 7f3b89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.332+0000 7f3b89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42044/0 (socket says 172.21.15.67:42044) 2024-04-17T07:14:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.332+0000 7f3b89400640 1 -- 172.21.15.67:0/210027881 learned_addr learned my addr 172.21.15.67:0/210027881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.333+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840afe50 con 0x7f3b8416b920 2024-04-17T07:14:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.333+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840ce9b0 con 0x7f3b84071e60 2024-04-17T07:14:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.333+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b840ce640 con 0x7f3b8406fb00 2024-04-17T07:14:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.334+0000 7f3b89e00640 1 -- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 msgr2=0x7f3b840cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.334+0000 7f3b89e00640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.334+0000 7f3b89e00640 1 -- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 msgr2=0x7f3b840ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.334+0000 7f3b89e00640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 0x7f3b840ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.334+0000 7f3b89e00640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8415bf10 con 0x7f3b8416b920 2024-04-17T07:14:07.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.335+0000 7f3b89400640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:07.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.335+0000 7f3b89e00640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c004870 tx=0x7f3b7c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.336+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c016b50 con 0x7f3b8416b920 2024-04-17T07:14:07.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.336+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7c016cf0 con 0x7f3b8416b920 2024-04-17T07:14:07.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.336+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c021680 con 0x7f3b8416b920 2024-04-17T07:14:07.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.336+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8415b390 con 0x7f3b8416b920 2024-04-17T07:14:07.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.337+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8415b950 con 0x7f3b8416b920 2024-04-17T07:14:07.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.340+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5c005290 con 0x7f3b8416b920 2024-04-17T07:14:07.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.342+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b7c021820 con 0x7f3b8416b920 2024-04-17T07:14:07.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.344+0000 7f3b81600640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.345+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3b7c0948a0 con 0x7f3b8416b920 2024-04-17T07:14:07.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.345+0000 7f3b89400640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.351+0000 7f3b89400640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 0x7f3b6406edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b70001640 tx=0x7f3b70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.353+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3b7c062c40 con 0x7f3b8416b920 2024-04-17T07:14:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.364+0000 7f8102400640 1 Processor -- start 2024-04-17T07:14:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.364+0000 7f8102400640 1 -- start start 2024-04-17T07:14:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.366+0000 7f8102400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80fc0cb010 0x7f80fc0cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.367+0000 7f8102400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80fc06ebd0 con 0x7f80fc0703f0 2024-04-17T07:14:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.367+0000 7f8102400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80fc15bc60 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.367+0000 7f8102400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f80fc0708a0 con 0x7f80fc0cb010 2024-04-17T07:14:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.367+0000 7f8101000640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f80fc0b2eb0 0x7f80fc15b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39414/0 (socket says 172.21.15.67:39414) 2024-04-17T07:14:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.367+0000 7f8101000640 1 -- 172.21.15.67:0/2070235815 learned_addr learned my addr 172.21.15.67:0/2070235815 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.368+0000 7f8101a00640 1 --2- 172.21.15.67:0/2070235815 >> v2:172.21.15.67:3300/0 conn(0x7f80fc0cb010 0x7f80fc0cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.368+0000 7ff18fa00640 1 Processor -- start 2024-04-17T07:14:07.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.369+0000 7f8101a00640 1 -- 172.21.15.67:0/2070235815 >> v1:172.21.15.187:6789/0 conn(0x7f80fc0b2eb0 legacy=0x7f80fc15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.369+0000 7f8101a00640 1 -- 172.21.15.67:0/2070235815 >> v1:172.21.15.67:6789/0 conn(0x7f80fc0703f0 legacy=0x7f80fc06e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.369+0000 7f8101a00640 1 -- 172.21.15.67:0/2070235815 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc15bf40 con 0x7f80fc0cb010 2024-04-17T07:14:07.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.369+0000 7ff18fa00640 1 -- start start 2024-04-17T07:14:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.371+0000 7f8101a00640 1 --2- 172.21.15.67:0/2070235815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc0cb420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009d90 tx=0x7f80f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7b93f12c6ed03c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.371+0000 7ff18fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff188155340 0x7ff18815b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.371+0000 7ff18fa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1880b3420 0x7ff188071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff18fa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1880ce920 con 0x7ff188072250 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff18fa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff1880726a0 con 0x7ff1880b3420 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff18fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1880b3c70 con 0x7ff188155340 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff17fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1880b3420 0x7ff188071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff17fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1880b3420 0x7ff188071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42066/0 (socket says 172.21.15.67:42066) 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7ff17fe00640 1 -- 172.21.15.67:0/3260829706 learned_addr learned my addr 172.21.15.67:0/3260829706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.372+0000 7f2c17a00640 1 Processor -- start 2024-04-17T07:14:07.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7f80fb400640 1 -- 172.21.15.67:0/2070235815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4039070 con 0x7f80fc0cb010 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7ff17fe00640 1 -- 172.21.15.67:0/3260829706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff188155340 msgr2=0x7ff18815b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7ff17fe00640 1 --2- 172.21.15.67:0/3260829706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff188155340 0x7ff18815b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7ff17fe00640 1 -- 172.21.15.67:0/3260829706 >> v1:172.21.15.187:6789/0 conn(0x7ff188072250 legacy=0x7ff18815e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7ff17fe00640 1 -- 172.21.15.67:0/3260829706 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1880b3f50 con 0x7ff1880b3420 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.374+0000 7f80fb400640 1 -- 172.21.15.67:0/2070235815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f402cb70 con 0x7f80fc0cb010 2024-04-17T07:14:07.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.374+0000 7f80fb400640 1 -- 172.21.15.67:0/2070235815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4034440 con 0x7f80fc0cb010 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.373+0000 7f2c17a00640 1 -- start start 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f8102400640 1 -- 172.21.15.67:0/2070235815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 msgr2=0x7f80fc0cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f8102400640 1 --2- 172.21.15.67:0/2070235815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc0cb420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009d90 tx=0x7f80f402be40 comp rx=0 tx=0).stop 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c100d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 0x7f2c100cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c100d1090 0x7f2c100d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c100d03b0 con 0x7f2c100d1090 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2c10163860 con 0x7f2c100cfa10 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c17a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c100d1d40 con 0x7f2c1015eca0 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7ff17fe00640 1 --2- 172.21.15.67:0/3260829706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880b3420 0x7ff188071d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff170008870 tx=0x7ff17002de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27da36422f0cf4e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f8102400640 1 -- 172.21.15.67:0/2070235815 shutdown_connections 2024-04-17T07:14:07.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f8102400640 1 --2- 172.21.15.67:0/2070235815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc0cb420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f8102400640 1 -- 172.21.15.67:0/2070235815 >> 172.21.15.67:0/2070235815 conn(0x7f80fc0b23c0 msgr2=0x7f80fc0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7ff187e00640 1 -- 172.21.15.67:0/3260829706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170034070 con 0x7ff1880b3420 2024-04-17T07:14:07.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7ff187e00640 1 -- 172.21.15.67:0/3260829706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17002ec20 con 0x7ff1880b3420 2024-04-17T07:14:07.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7ff187e00640 1 -- 172.21.15.67:0/3260829706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170038400 con 0x7ff1880b3420 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.375+0000 7f2c0fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 0x7f2c100cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c0fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 0x7f2c100cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53972/0 (socket says 172.21.15.67:53972) 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c0fe00640 1 -- 172.21.15.67:0/2635966291 learned_addr learned my addr 172.21.15.67:0/2635966291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 --2- 172.21.15.67:0/2635966291 >> v2:172.21.15.67:3300/0 conn(0x7f2c100d1090 0x7f2c100d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15000640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c100d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 -- 172.21.15.67:0/2635966291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 msgr2=0x7f2c100d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c100d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 -- 172.21.15.67:0/2635966291 >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 msgr2=0x7f2c100cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 --2- 172.21.15.67:0/2635966291 >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 0x7f2c100cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.376+0000 7f2c15a00640 1 -- 172.21.15.67:0/2635966291 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c100d2c30 con 0x7f2c100d1090 2024-04-17T07:14:07.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.377+0000 7ff18fa00640 1 -- 172.21.15.67:0/3260829706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880b3420 msgr2=0x7ff188071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.377+0000 7ff18fa00640 1 --2- 172.21.15.67:0/3260829706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880b3420 0x7ff188071d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff170008870 tx=0x7ff17002de70 comp rx=0 tx=0).stop 2024-04-17T07:14:07.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 -- 172.21.15.67:0/3260829706 shutdown_connections 2024-04-17T07:14:07.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 --2- 172.21.15.67:0/3260829706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1880b3420 0x7ff188071d10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 --2- 172.21.15.67:0/3260829706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff188155340 0x7ff18815b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7f48da600640 1 Processor -- start 2024-04-17T07:14:07.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 -- 172.21.15.67:0/3260829706 >> 172.21.15.67:0/3260829706 conn(0x7ff18806bce0 msgr2=0x7ff18806c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7f8102400640 1 -- 172.21.15.67:0/2070235815 shutdown_connections 2024-04-17T07:14:07.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7f2c15a00640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100d1090 0x7f2c100d1520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2c100b7530 tx=0x7f2c0802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438d8371712c1d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7f48da600640 1 -- start start 2024-04-17T07:14:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.379+0000 7f2c0f400640 1 -- 172.21.15.67:0/2635966291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08039070 con 0x7f2c100d1090 2024-04-17T07:14:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.379+0000 7f2c0f400640 1 -- 172.21.15.67:0/2635966291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0802cbe0 con 0x7f2c100d1090 2024-04-17T07:14:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.379+0000 7f2c0f400640 1 -- 172.21.15.67:0/2635966291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08034440 con 0x7f2c100d1090 2024-04-17T07:14:07.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7f8102400640 1 -- 172.21.15.67:0/2070235815 wait complete. 2024-04-17T07:14:07.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 -- 172.21.15.67:0/3260829706 shutdown_connections 2024-04-17T07:14:07.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.378+0000 7ff18fa00640 1 -- 172.21.15.67:0/3260829706 wait complete. 2024-04-17T07:14:07.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f48da600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48d409aa60 0x7f48d409ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f48da600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 0x7f48d40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f48da600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d406e9e0 con 0x7f48d41605a0 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f48da600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f48d415d350 con 0x7f48d40d0280 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f48da600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f48d40d0c20 con 0x7f48d409aa60 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7fbe6c800640 1 Processor -- start 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7fbe6c800640 1 -- start start 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f2c17a00640 1 -- 172.21.15.67:0/2635966291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100d1090 msgr2=0x7f2c100d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f2c17a00640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100d1090 0x7f2c100d1520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2c100b7530 tx=0x7f2c0802be30 comp rx=0 tx=0).stop 2024-04-17T07:14:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f48d2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 0x7f48d40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f49f3a00640 1 Processor -- start 2024-04-17T07:14:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.380+0000 7f49f3a00640 1 -- start start 2024-04-17T07:14:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f2c17a00640 1 -- 172.21.15.67:0/2635966291 shutdown_connections 2024-04-17T07:14:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f2c17a00640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100d1090 0x7f2c100d1520 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f2c17a00640 1 --2- 172.21.15.67:0/2635966291 >> v2:172.21.15.187:3300/0 conn(0x7f2c100cfa10 0x7f2c100cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f2c17a00640 1 --2- 172.21.15.67:0/2635966291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c100d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f2c17a00640 1 -- 172.21.15.67:0/2635966291 >> 172.21.15.67:0/2635966291 conn(0x7f2c10067600 msgr2=0x7f2c100679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f9dc2e00640 1 Processor -- start 2024-04-17T07:14:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f48d3400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48d409aa60 0x7f48d409ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f48d3400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48d409aa60 0x7f48d409ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53976/0 (socket says 172.21.15.67:53976) 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f48d3400640 1 -- 172.21.15.67:0/64788243 learned_addr learned my addr 172.21.15.67:0/64788243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f48d2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 0x7f48d40d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42094/0 (socket says 172.21.15.67:42094) 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f48d3e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f48d41605a0 0x7f48d4160a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39430/0 (socket says 172.21.15.67:39430) 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f49f3a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49ec1552d0 0x7f49ec15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f49f3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49ec0d0140 0x7f49ec0d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f49f3a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49ec06de90 con 0x7f49ec0d0b10 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f49f3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec06e390 con 0x7f49ec0d0140 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f49f3a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec072ba0 con 0x7f49ec1552d0 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f48d3400640 1 -- 172.21.15.67:0/64788243 >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 msgr2=0x7f48d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f48d3400640 1 --2- 172.21.15.67:0/64788243 >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 0x7f48d40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f48d3400640 1 -- 172.21.15.67:0/64788243 >> v1:172.21.15.187:6789/0 conn(0x7f48d41605a0 legacy=0x7f48d4160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7fbe6c800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe640ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7f8102400640 1 Processor -- start 2024-04-17T07:14:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49ec1552d0 0x7f49ec15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49ec0d0140 0x7f49ec0d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7ff18fa00640 1 Processor -- start 2024-04-17T07:14:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7ff18fa00640 1 -- start start 2024-04-17T07:14:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f48d3400640 1 -- 172.21.15.67:0/64788243 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d409b380 con 0x7f48d409aa60 2024-04-17T07:14:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f2c17a00640 1 -- 172.21.15.67:0/2635966291 shutdown_connections 2024-04-17T07:14:07.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.382+0000 7fbe6c800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 0x7fbe640cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f2c17a00640 1 -- 172.21.15.67:0/2635966291 wait complete. 2024-04-17T07:14:07.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7fbe6c800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe64166c90 con 0x7fbe640b3e30 2024-04-17T07:14:07.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7fbe6c800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbe64166a40 con 0x7fbe640cfbf0 2024-04-17T07:14:07.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7fbe6c800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe64166f70 con 0x7fbe64162dd0 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.381+0000 7f9dc2e00640 1 -- start start 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49ec1552d0 0x7f49ec15bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53980/0 (socket says 172.21.15.67:53980) 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 -- 172.21.15.67:0/1949068269 learned_addr learned my addr 172.21.15.67:0/1949068269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 -- 172.21.15.67:0/1949068269 >> v2:172.21.15.67:3300/0 conn(0x7f49ec0d0140 msgr2=0x7f49ec0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 --2- 172.21.15.67:0/1949068269 >> v2:172.21.15.67:3300/0 conn(0x7f49ec0d0140 0x7f49ec0d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 -- 172.21.15.67:0/1949068269 >> v1:172.21.15.187:6789/0 conn(0x7f49ec0d0b10 legacy=0x7f49ec15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f49f1000640 1 -- 172.21.15.67:0/1949068269 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec072e80 con 0x7f49ec1552d0 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1880b3420 0x7ff188151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff188155340 0x7ff188151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880b4690 con 0x7ff188072250 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff1880b4510 con 0x7ff188155340 2024-04-17T07:14:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1880b38a0 con 0x7ff1880b3420 2024-04-17T07:14:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.383+0000 7f8102400640 1 -- start start 2024-04-17T07:14:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7f9dc2e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9dbc15b0b0 0x7f9dbc071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7f9dc2e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9dbc072250 0x7f9dbc0cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7ff18d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7fbe69400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 0x7fbe640cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7fbe69400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 0x7fbe640cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53990/0 (socket says 172.21.15.67:53990) 2024-04-17T07:14:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7fbe69400640 1 -- 172.21.15.67:0/3773838678 learned_addr learned my addr 172.21.15.67:0/3773838678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff17fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1880b3420 0x7ff188151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7fbe69e00640 1 --2- 172.21.15.67:0/3773838678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe640ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42110/0 (socket says 172.21.15.67:42110) 2024-04-17T07:14:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 -- 172.21.15.67:0/721615103 learned_addr learned my addr 172.21.15.67:0/721615103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7f9dc2e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dbc072650 con 0x7f9dbc155340 2024-04-17T07:14:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7f9dc2e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9dbc0ce920 con 0x7f9dbc072250 2024-04-17T07:14:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.384+0000 7f9dc2e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9dbc0b3c70 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7f49f1000640 1 --2- 172.21.15.67:0/1949068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec1552d0 0x7f49ec15bbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06c670 tx=0x7f49d80341b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9323652b2502a1e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe68a00640 1 -- 172.21.15.67:0/3773838678 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130225079 0 0) 0x7fbe64166c90 con 0x7fbe640b3e30 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe68a00640 1 -- 172.21.15.67:0/3773838678 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe4c003520 con 0x7fbe640b3e30 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 -- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1880b3420 msgr2=0x7ff188151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1880b3420 0x7ff188151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 -- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff188155340 msgr2=0x7ff188151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff188155340 0x7ff188151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7ff18d000640 1 -- 172.21.15.67:0/721615103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18814c4b0 con 0x7ff188072250 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe69e00640 1 -- 172.21.15.67:0/3773838678 >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 msgr2=0x7fbe640cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe69e00640 1 --2- 172.21.15.67:0/3773838678 >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 0x7fbe640cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe69e00640 1 -- 172.21.15.67:0/3773838678 >> v1:172.21.15.187:6789/0 conn(0x7fbe640b3e30 legacy=0x7fbe64071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7fbe69e00640 1 -- 172.21.15.67:0/3773838678 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe64167250 con 0x7fbe64162dd0 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.385+0000 7f9dbb400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9dbc15b0b0 0x7f9dbc071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9dbc15b0b0 0x7f9dbc071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:54018/0 (socket says 172.21.15.67:54018) 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 -- 172.21.15.67:0/3528532893 learned_addr learned my addr 172.21.15.67:0/3528532893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80fc0703f0 0x7f80fc0c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f2c17a00640 1 Processor -- start 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc13b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc0ced50 con 0x7f80fc0cb010 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc0cebd0 con 0x7f80fc0703f0 2024-04-17T07:14:07.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f8102400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f80fc0ceed0 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f49ebe00640 1 -- 172.21.15.67:0/1949068269 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8041070 con 0x7f49ec1552d0 2024-04-17T07:14:07.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f2c17a00640 1 -- start start 2024-04-17T07:14:07.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7ff18d000640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff17800ba50 tx=0x7ff17800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41394/0 (socket says 172.21.15.67:41394) 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 -- 172.21.15.67:0/1897830446 learned_addr learned my addr 172.21.15.67:0/1897830446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f48d3400640 1 --2- 172.21.15.67:0/64788243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d409ae40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f48d40cf100 tx=0x7f48bc02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bc945b4a6db7c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 -- 172.21.15.67:0/3528532893 >> v2:172.21.15.67:3300/0 conn(0x7f9dbc072250 msgr2=0x7f9dbc0cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 --2- 172.21.15.67:0/3528532893 >> v2:172.21.15.67:3300/0 conn(0x7f9dbc072250 0x7f9dbc0cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 -- 172.21.15.67:0/3528532893 >> v1:172.21.15.187:6789/0 conn(0x7f9dbc155340 legacy=0x7f9dbc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.386+0000 7f9dbb400640 1 -- 172.21.15.67:0/3528532893 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dbc0b3f50 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7fbe69e00640 1 --2- 172.21.15.67:0/3773838678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe640ccda0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe6002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec15d9fbece0bb25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178019070 con 0x7ff188072250 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17800cc00 con 0x7ff188072250 2024-04-17T07:14:07.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178014440 con 0x7ff188072250 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f49ebe00640 1 -- 172.21.15.67:0/1949068269 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d803c440 con 0x7f49ec1552d0 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f49ebe00640 1 -- 172.21.15.67:0/1949068269 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8041070 con 0x7f49ec1552d0 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f8101a00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f8101000640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80fc0703f0 0x7f80fc0c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 -- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80fc0703f0 msgr2=0x7f80fc0c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80fc0703f0 0x7f80fc0c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 -- 172.21.15.67:0/1897830446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 msgr2=0x7f80fc13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7f80fbe00640 1 -- 172.21.15.67:0/1897830446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc13a700 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f8101a00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.387+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18814c790 con 0x7ff188072250 2024-04-17T07:14:07.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7fbe68a00640 1 -- 172.21.15.67:0/3773838678 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60039070 con 0x7fbe64162dd0 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 -- 172.21.15.67:0/1949068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec1552d0 msgr2=0x7f49ec15bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1949068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec1552d0 0x7f49ec15bbf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06c670 tx=0x7f49d80341b0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f48d2000640 1 -- 172.21.15.67:0/64788243 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc034070 con 0x7f48d409aa60 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f48d2000640 1 -- 172.21.15.67:0/64788243 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48bc030860 con 0x7f48d409aa60 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f48d2000640 1 -- 172.21.15.67:0/64788243 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc030c30 con 0x7f48d409aa60 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7fbe68a00640 1 -- 172.21.15.67:0/3773838678 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe6002cc50 con 0x7fbe64162dd0 2024-04-17T07:14:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7fbe68a00640 1 -- 172.21.15.67:0/3773838678 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60034440 con 0x7fbe64162dd0 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 -- 172.21.15.67:0/1949068269 shutdown_connections 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1949068269 >> v2:172.21.15.67:3300/0 conn(0x7f49ec0d0140 0x7f49ec0d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1949068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec1552d0 0x7f49ec15bbf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 -- 172.21.15.67:0/1949068269 >> 172.21.15.67:0/1949068269 conn(0x7f49ec067ee0 msgr2=0x7f49ec06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f49f3a00640 1 -- 172.21.15.67:0/1949068269 shutdown_connections 2024-04-17T07:14:07.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f9dbb400640 1 --2- 172.21.15.67:0/3528532893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc071d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc068290 tx=0x7f9da802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fac64169f0e7edb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f49f3a00640 1 -- 172.21.15.67:0/1949068269 wait complete. 2024-04-17T07:14:07.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f80fbe00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c400 tx=0x7f80e800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18814d270 con 0x7ff188072250 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 0x7f2c10159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c100d34d0 con 0x7f2c100cfa10 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c100d31f0 con 0x7f2c100d1090 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.388+0000 7f2c17a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c100d1a60 con 0x7f2c1015eca0 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7fbe6c800640 1 -- 172.21.15.67:0/3773838678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 msgr2=0x7fbe640ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7fbe6c800640 1 --2- 172.21.15.67:0/3773838678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe640ccda0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009d90 tx=0x7fbe6002bea0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f9dbaa00640 1 -- 172.21.15.67:0/3528532893 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8038070 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f9dbaa00640 1 -- 172.21.15.67:0/3528532893 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9da802f050 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f9dbaa00640 1 -- 172.21.15.67:0/3528532893 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8031970 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8017070 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e80052f0 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e8003830 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f2c15a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f2c0fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f2c0fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:54032/0 (socket says 172.21.15.67:54032) 2024-04-17T07:14:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f2c0fe00640 1 -- 172.21.15.67:0/204206524 learned_addr learned my addr 172.21.15.67:0/204206524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.389+0000 7f2c15a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:41406/0 (socket says 172.21.15.67:41406) 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 -- 172.21.15.67:0/3773838678 shutdown_connections 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 --2- 172.21.15.67:0/3773838678 >> v2:172.21.15.187:3300/0 conn(0x7fbe640cfbf0 0x7fbe640cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 --2- 172.21.15.67:0/3773838678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe640ccda0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 -- 172.21.15.67:0/3773838678 >> 172.21.15.67:0/3773838678 conn(0x7fbe6406bce0 msgr2=0x7fbe6406c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 -- 172.21.15.67:0/64788243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 msgr2=0x7f48d409ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 --2- 172.21.15.67:0/64788243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d409ae40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f48d40cf100 tx=0x7f48bc02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 -- 172.21.15.67:0/3773838678 shutdown_connections 2024-04-17T07:14:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 -- 172.21.15.67:0/64788243 shutdown_connections 2024-04-17T07:14:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7fbe6c800640 1 -- 172.21.15.67:0/3773838678 wait complete. 2024-04-17T07:14:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 --2- 172.21.15.67:0/64788243 >> v2:172.21.15.67:3300/0 conn(0x7f48d40d0280 0x7f48d40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 --2- 172.21.15.67:0/64788243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d409ae40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f48da600640 1 -- 172.21.15.67:0/64788243 >> 172.21.15.67:0/64788243 conn(0x7f48d40b41a0 msgr2=0x7f48d40b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc0c5b70 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3528532893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 msgr2=0x7f9dbc071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3528532893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc071d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc068290 tx=0x7f9da802b7f0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff188068ca0 con 0x7ff188072250 2024-04-17T07:14:07.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.390+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc13ac80 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15a00640 1 -- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 msgr2=0x7f2c1015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15000640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 0x7f2c10159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15a00640 1 -- 172.21.15.67:0/204206524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 msgr2=0x7f2c10159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 0x7f2c10159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.391+0000 7f2c15a00640 1 -- 172.21.15.67:0/204206524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1015ac30 con 0x7f2c1015eca0 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f2c0fe00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f49f3a00640 1 Processor -- start 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f2c15000640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 0x7f2c10159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff17801c070 con 0x7ff188072250 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f48da600640 1 -- 172.21.15.67:0/64788243 shutdown_connections 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3528532893 shutdown_connections 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3528532893 >> v2:172.21.15.67:3300/0 conn(0x7f9dbc072250 0x7f9dbc0cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3528532893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc071d10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3528532893 >> 172.21.15.67:0/3528532893 conn(0x7f9dbc06bce0 msgr2=0x7f9dbc06c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80fc068a10 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f2c15a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c08033000 tx=0x7f2c08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f49f3a00640 1 -- start start 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec0d0140 0x7f49ec14a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec1552d0 0x7f49ec1efe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0cfee0 con 0x7f49ec1552d0 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0cfd60 con 0x7f49ec0d0b10 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f49f3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec0cf3c0 con 0x7f49ec0d0140 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7f49f1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec0d0140 0x7f49ec14a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7f49e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7f49e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:54042/0 (socket says 172.21.15.67:54042) 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7f49e3e00640 1 -- 172.21.15.67:0/1716668755 learned_addr learned my addr 172.21.15.67:0/1716668755 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 -- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec0d0140 msgr2=0x7f49ec14a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec0d0140 0x7f49ec14a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 -- 172.21.15.67:0/1716668755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec1552d0 msgr2=0x7f49ec1efe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec1552d0 0x7f49ec1efe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 -- 172.21.15.67:0/1716668755 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0b0eb0 con 0x7f49ec0d0b10 2024-04-17T07:14:07.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.392+0000 7f48da600640 1 -- 172.21.15.67:0/64788243 wait complete. 2024-04-17T07:14:07.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7fbe6c800640 1 Processor -- start 2024-04-17T07:14:07.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7fbe6c800640 1 -- start start 2024-04-17T07:14:07.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.393+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3528532893 shutdown_connections 2024-04-17T07:14:07.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3528532893 wait complete. 2024-04-17T07:14:07.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 0x7fbe640aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7ff185600640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 0x7ff16406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.394+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff17808b930 con 0x7ff188072250 2024-04-17T07:14:07.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7ff17fe00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 0x7ff16406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f49e3e00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00bdb0 tx=0x7f49dc00a650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc0175f0 con 0x7f49ec0d0b10 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc017790 con 0x7f49ec0d0b10 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc017b60 con 0x7f49ec0d0b10 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec0b1150 con 0x7f49ec0d0b10 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec0b1710 con 0x7f49ec0d0b10 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f48da600640 1 Processor -- start 2024-04-17T07:14:07.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f48da600640 1 -- start start 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe641f9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe641631b0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640cf5d0 con 0x7fbe640b3e30 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7fbe6c800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6414d290 con 0x7fbe64162dd0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42168/0 (socket says 172.21.15.67:42168) 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 -- 172.21.15.67:0/2437403372 learned_addr learned my addr 172.21.15.67:0/2437403372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69e00640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 0x7fbe640aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe6a800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe641f9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 -- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 msgr2=0x7fbe641f9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe641f9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 -- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 msgr2=0x7fbe640aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 0x7fbe640aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7fbe69400640 1 -- 172.21.15.67:0/2437403372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe640af380 con 0x7fbe640cfbf0 2024-04-17T07:14:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.397+0000 7fbe69e00640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 0x7fbe640aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.397+0000 7fbe69400640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e840 tx=0x7fbe5800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f9dc2e00640 1 Processor -- start 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.396+0000 7f9dc2e00640 1 -- start start 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.395+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80e801a070 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.397+0000 7f80f8c00640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 0x7f80dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.397+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f80e808b020 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7ff17fe00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 0x7ff16406ee80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff170039000 tx=0x7ff170034000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d414ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48d40d0280 0x7f48d40b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dbc155340 0x7f9dbc1450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc1455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc0b4690 con 0x7f9dbc072250 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc0b4510 con 0x7f9dbc15b0b0 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dc2e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9dbc0d04b0 con 0x7f9dbc155340 2024-04-17T07:14:07.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f48d40d1140 con 0x7f48d41605a0 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f48d40cd630 con 0x7f48d409aa60 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48da600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f48d40d1420 con 0x7f48d40d0280 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f8101000640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 0x7f80dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dbb400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dbc155340 0x7f9dbc1450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42176/0 (socket says 172.21.15.67:42176) 2024-04-17T07:14:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f9dbbe00640 1 -- 172.21.15.67:0/3057531038 learned_addr learned my addr 172.21.15.67:0/3057531038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f48d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f48d2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48d40d0280 0x7f48d40b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.398+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec1f03d0 con 0x7f49ec0d0b10 2024-04-17T07:14:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f48d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42186/0 (socket says 172.21.15.67:42186) 2024-04-17T07:14:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f48d3e00640 1 -- 172.21.15.67:0/3110497988 learned_addr learned my addr 172.21.15.67:0/3110497988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dc0e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc1455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f48d3400640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d414ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dbbe00640 1 -- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dbc155340 msgr2=0x7f9dbc1450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dbbe00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dbc155340 0x7f9dbc1450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dbbe00640 1 -- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 msgr2=0x7f9dbc1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dbbe00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc1455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.399+0000 7f9dbbe00640 1 -- 172.21.15.67:0/3057531038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dbc144b80 con 0x7f9dbc072250 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 -- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48d40d0280 msgr2=0x7f48d40b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48d40d0280 0x7f48d40b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 -- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 msgr2=0x7f48d414ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d414ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 -- 172.21.15.67:0/3110497988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d414f3c0 con 0x7f48d41605a0 2024-04-17T07:14:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f9dbbe00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c430 tx=0x7f9dac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f48d3e00640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f48c400c730 tx=0x7f48c400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.400+0000 7f8101000640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 0x7f80dc06ede0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000f60 tx=0x7f80f0007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.401+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f49dc01e020 con 0x7f49ec0d0b10 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.401+0000 7f49e9600640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 0x7f49cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.401+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f49dc08c330 con 0x7f49ec0d0b10 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58017070 con 0x7fbe640cfbf0 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5800c8e0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ccd0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac011790 con 0x7f9dbc072250 2024-04-17T07:14:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dac017070 con 0x7f9dbc072250 2024-04-17T07:14:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac011bb0 con 0x7f9dbc072250 2024-04-17T07:14:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f49f1000640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 0x7f49cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff178059d40 con 0x7ff188072250 2024-04-17T07:14:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4018070 con 0x7f48d41605a0 2024-04-17T07:14:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c4004d60 con 0x7f48d41605a0 2024-04-17T07:14:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c400a820 con 0x7f48d41605a0 2024-04-17T07:14:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6414fe00 con 0x7fbe640cfbf0 2024-04-17T07:14:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.403+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d414f6a0 con 0x7f48d41605a0 2024-04-17T07:14:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dbc145c20 con 0x7f9dbc072250 2024-04-17T07:14:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.402+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dbc1461e0 con 0x7f9dbc072250 2024-04-17T07:14:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.403+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d414fc60 con 0x7f48d41605a0 2024-04-17T07:14:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.404+0000 7f49f1000640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 0x7f49cc06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49d803e000 tx=0x7f49d8038000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.403+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe641503c0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.405+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe64068ca0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.405+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dbc068ca0 con 0x7f9dbc072250 2024-04-17T07:14:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.405+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d4068ca0 con 0x7f48d41605a0 2024-04-17T07:14:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.406+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48c401b070 con 0x7f48d41605a0 2024-04-17T07:14:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.406+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9dac01c460 con 0x7f9dbc072250 2024-04-17T07:14:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.407+0000 7f48cb400640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 0x7f48b006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.407+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe5801a070 con 0x7fbe640cfbf0 2024-04-17T07:14:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.407+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f48c408b0a0 con 0x7f48d41605a0 2024-04-17T07:14:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.408+0000 7f48d3400640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 0x7f48b006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.408+0000 7fbe5e000640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 0x7fbe4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.408+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fbe5808c200 con 0x7fbe640cfbf0 2024-04-17T07:14:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.409+0000 7fbe69e00640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 0x7fbe4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.409+0000 7f9db3e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 0x7f9d9c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.409+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9dac08b9e0 con 0x7f9dbc072250 2024-04-17T07:14:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.409+0000 7f9dbb400640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 0x7f9d9c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.410+0000 7f48d3400640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 0x7f48b006ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036000 tx=0x7f48bc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.410+0000 7f9dbb400640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 0x7f9d9c06ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc068290 tx=0x7f9da8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.411+0000 7fbe69e00640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 0x7fbe4806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe60036000 tx=0x7fbe60030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.412+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbe5805a6a0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.414+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08044020 con 0x7f2c1015eca0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.414+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c08035e50 con 0x7f2c1015eca0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.414+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08039040 con 0x7f2c1015eca0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.414+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c10151340 con 0x7f2c1015eca0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.414+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1015b180 con 0x7f2c1015eca0 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.415+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f49dc012070 con 0x7f49ec0d0b10 2024-04-17T07:14:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.415+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f80e8059440 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.416+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c10068ca0 con 0x7f2c1015eca0 2024-04-17T07:14:07.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.419+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f48c4058e60 con 0x7f48d41605a0 2024-04-17T07:14:07.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.418+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c08047020 con 0x7f2c1015eca0 2024-04-17T07:14:07.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.419+0000 7f2c0cc00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 0x7f2bf406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.419+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2c080ae410 con 0x7f2c1015eca0 2024-04-17T07:14:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.420+0000 7f2c15000640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 0x7f2bf406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.424+0000 7f2c15000640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 0x7f2bf406ee00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009a40 tx=0x7f2c04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.427+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9dac059e00 con 0x7f9dbc072250 2024-04-17T07:14:07.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.433+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c0804e030 con 0x7f2c1015eca0 2024-04-17T07:14:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.807+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3b5c005580 con 0x7f3b8416b920 2024-04-17T07:14:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.807+0000 7f3b81600640 1 -- 172.21.15.67:0/210027881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b7c062540 con 0x7f3b8416b920 2024-04-17T07:14:07.808 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 msgr2=0x7f3b6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 0x7f3b6406edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b70001640 tx=0x7f3b70009040 comp rx=0 tx=0).stop 2024-04-17T07:14:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 msgr2=0x7f3b840c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c004870 tx=0x7f3b7c00d720 comp rx=0 tx=0).stop 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 shutdown_connections 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8416b920 0x7f3b840c8260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b84071e60 0x7f3b840ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3b6406c8f0 0x7f3b6406edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 --2- 172.21.15.67:0/210027881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b8406fb00 0x7f3b840cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.812+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 >> 172.21.15.67:0/210027881 conn(0x7f3b840b23c0 msgr2=0x7f3b840cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.813+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 shutdown_connections 2024-04-17T07:14:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.814+0000 7f3b8a800640 1 -- 172.21.15.67:0/210027881 wait complete. 2024-04-17T07:14:07.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2024-04-17T07:14:07.841 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.845+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff1880cd8d0 con 0x7ff188072250 2024-04-17T07:14:07.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.846+0000 7ff185600640 1 -- 172.21.15.67:0/721615103 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff178059640 con 0x7ff188072250 2024-04-17T07:14:07.846 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 msgr2=0x7ff16406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 0x7ff16406ee80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff170039000 tx=0x7ff170034000 comp rx=0 tx=0).stop 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 msgr2=0x7ff18814bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff17800ba50 tx=0x7ff17800bf10 comp rx=0 tx=0).stop 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 shutdown_connections 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff188155340 0x7ff188151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff16406c9c0 0x7ff16406ee80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1880b3420 0x7ff188151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 --2- 172.21.15.67:0/721615103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff188072250 0x7ff18814bb40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 >> 172.21.15.67:0/721615103 conn(0x7ff18806bce0 msgr2=0x7ff18815d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 shutdown_connections 2024-04-17T07:14:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.849+0000 7ff18fa00640 1 -- 172.21.15.67:0/721615103 wait complete. 2024-04-17T07:14:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.860+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbe640cf2f0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.861+0000 7fbe5e000640 1 -- 172.21.15.67:0/2437403372 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbe58059fa0 con 0x7fbe640cfbf0 2024-04-17T07:14:07.862 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:07.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 msgr2=0x7fbe4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 0x7fbe4806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe60036000 tx=0x7fbe60030000 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 msgr2=0x7fbe6414f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e840 tx=0x7fbe5800ed00 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 shutdown_connections 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe4806c8f0 0x7fbe4806edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe64162dd0 0x7fbe641f9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640cfbf0 0x7fbe6414f5b0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e840 tx=0x7fbe5800ed00 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 --2- 172.21.15.67:0/2437403372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe640b3e30 0x7fbe640aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.864+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 >> 172.21.15.67:0/2437403372 conn(0x7fbe6406bce0 msgr2=0x7fbe640ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.865+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 shutdown_connections 2024-04-17T07:14:07.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.865+0000 7fbe6c800640 1 -- 172.21.15.67:0/2437403372 wait complete. 2024-04-17T07:14:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.867+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f48d40b5040 con 0x7f48d41605a0 2024-04-17T07:14:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.868+0000 7f48cb400640 1 -- 172.21.15.67:0/3110497988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f48c405d160 con 0x7f48d41605a0 2024-04-17T07:14:07.871 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:07.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.0 2024-04-17T07:14:07.871 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.871+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 msgr2=0x7f48b006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.871+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 0x7f48b006ee80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036000 tx=0x7f48bc008030 comp rx=0 tx=0).stop 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.871+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 msgr2=0x7f48d4147a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.871+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f48c400c730 tx=0x7f48c400cb30 comp rx=0 tx=0).stop 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 shutdown_connections 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d41605a0 0x7f48d4147a10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f48b006c9c0 0x7f48b006ee80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48d40d0280 0x7f48d40b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 --2- 172.21.15.67:0/3110497988 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48d409aa60 0x7f48d414ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 >> 172.21.15.67:0/3110497988 conn(0x7f48d40b41a0 msgr2=0x7f48d406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 shutdown_connections 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f48da600640 1 -- 172.21.15.67:0/3110497988 wait complete. 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.872+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f80fc0cfde0 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.873+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f49ec0b3440 con 0x7f49ec0d0b10 2024-04-17T07:14:07.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.874+0000 7f80f8c00640 1 -- 172.21.15.67:0/1897830446 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f80e8058d40 con 0x7f80fc0b2eb0 2024-04-17T07:14:07.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.874+0000 7f49e9600640 1 -- 172.21.15.67:0/1716668755 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f49dc05a740 con 0x7f49ec0d0b10 2024-04-17T07:14:07.876 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:07.876 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 msgr2=0x7f49cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 0x7f49cc06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49d803e000 tx=0x7f49d8038000 comp rx=0 tx=0).stop 2024-04-17T07:14:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 msgr2=0x7f49ec0b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00bdb0 tx=0x7f49dc00a650 comp rx=0 tx=0).stop 2024-04-17T07:14:07.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 msgr2=0x7f80dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 0x7f80dc06ede0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000f60 tx=0x7f80f0007c00 comp rx=0 tx=0).stop 2024-04-17T07:14:07.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 msgr2=0x7f80fc13ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c400 tx=0x7f80e800c8c0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 shutdown_connections 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80fc0cb010 0x7f80fc13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f80dc06c920 0x7f80dc06ede0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80fc0b2eb0 0x7f80fc13ce00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 shutdown_connections 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec1552d0 0x7f49ec1efe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49ec0d0b10 0x7f49ec0b0970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f49cc06c8f0 0x7f49cc06edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 --2- 172.21.15.67:0/1716668755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec0d0140 0x7f49ec14a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 >> 172.21.15.67:0/1716668755 conn(0x7f49ec067ee0 msgr2=0x7f49ec14f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 shutdown_connections 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.877+0000 7f49f3a00640 1 -- 172.21.15.67:0/1716668755 wait complete. 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 --2- 172.21.15.67:0/1897830446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80fc0703f0 0x7f80fc0c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 >> 172.21.15.67:0/1897830446 conn(0x7f80fc0b23c0 msgr2=0x7f80fc143f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 shutdown_connections 2024-04-17T07:14:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.878+0000 7f8102400640 1 -- 172.21.15.67:0/1897830446 wait complete. 2024-04-17T07:14:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.882+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9dbc15de10 con 0x7f9dbc072250 2024-04-17T07:14:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.882+0000 7f9db3e00640 1 -- 172.21.15.67:0/3057531038 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9dac059700 con 0x7f9dbc072250 2024-04-17T07:14:07.882 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 msgr2=0x7f9d9c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 0x7f9d9c06ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc068290 tx=0x7f9da8008030 comp rx=0 tx=0).stop 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 msgr2=0x7f9dbc143f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c430 tx=0x7f9dac00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 shutdown_connections 2024-04-17T07:14:07.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dbc15b0b0 0x7f9dbc1455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9d9c06c920 0x7f9d9c06ede0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dbc155340 0x7f9dbc1450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 --2- 172.21.15.67:0/3057531038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dbc072250 0x7f9dbc143f10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 >> 172.21.15.67:0/3057531038 conn(0x7f9dbc06bce0 msgr2=0x7f9dbc15e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 shutdown_connections 2024-04-17T07:14:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.885+0000 7f9dc2e00640 1 -- 172.21.15.67:0/3057531038 wait complete. 2024-04-17T07:14:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.888+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2c100d2970 con 0x7f2c1015eca0 2024-04-17T07:14:07.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.888+0000 7f2c0cc00640 1 -- 172.21.15.67:0/204206524 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2c0807c8b0 con 0x7f2c1015eca0 2024-04-17T07:14:07.889 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 msgr2=0x7f2bf406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 0x7f2bf406ee00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c04009a40 tx=0x7f2c04008040 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 msgr2=0x7f2c10157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c08033000 tx=0x7f2c08004a40 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 shutdown_connections 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2bf406c940 0x7f2bf406ee00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c1015eca0 0x7f2c10157f20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c100d1090 0x7f2c1015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 --2- 172.21.15.67:0/204206524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c100cfa10 0x7f2c10159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.891+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 >> 172.21.15.67:0/204206524 conn(0x7f2c10067600 msgr2=0x7f2c100b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.892+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 shutdown_connections 2024-04-17T07:14:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:07.892+0000 7f2c17a00640 1 -- 172.21.15.67:0/204206524 wait complete. 2024-04-17T07:14:07.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.5 2024-04-17T07:14:07.894 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.7 2024-04-17T07:14:07.897 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.6 2024-04-17T07:14:07.907 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.907 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.2 2024-04-17T07:14:07.907 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.910 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.4 2024-04-17T07:14:07.910 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.3 2024-04-17T07:14:07.913 DEBUG:teuthology.parallel:result is None 2024-04-17T07:14:07.913 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-17T07:14:07.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:14:08.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.162+0000 7f3907200640 1 Processor -- start 2024-04-17T07:14:08.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.162+0000 7f3907200640 1 -- start start 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39080cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3908160280 0x7f390806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 0x7f3908142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f390815ce30 con 0x7f3908160280 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f39080cd4a0 con 0x7f3908141ed0 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3907200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39081607f0 con 0x7f390806b8c0 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3905400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3908160280 0x7f390806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3905400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3908160280 0x7f390806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42190/0 (socket says 172.21.15.67:42190) 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.164+0000 7f3905400640 1 -- 172.21.15.67:0/1943719322 learned_addr learned my addr 172.21.15.67:0/1943719322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3905400640 1 -- 172.21.15.67:0/1943719322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 msgr2=0x7f39080cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3905400640 1 --2- 172.21.15.67:0/1943719322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39080cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3906800640 1 --2- 172.21.15.67:0/1943719322 >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 0x7f3908142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3905400640 1 -- 172.21.15.67:0/1943719322 >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 msgr2=0x7f3908142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3905400640 1 --2- 172.21.15.67:0/1943719322 >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 0x7f3908142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3905400640 1 -- 172.21.15.67:0/1943719322 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3908160ad0 con 0x7f3908160280 2024-04-17T07:14:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.165+0000 7f3906800640 1 --2- 172.21.15.67:0/1943719322 >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 0x7f3908142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.166+0000 7f3905400640 1 --2- 172.21.15.67:0/1943719322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f390806bca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3908071d80 tx=0x7f38f402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b719aa991054c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.166+0000 7f3904a00640 1 -- 172.21.15.67:0/1943719322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4035020 con 0x7f3908160280 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.166+0000 7f3904a00640 1 -- 172.21.15.67:0/1943719322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f402d040 con 0x7f3908160280 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.166+0000 7f3904a00640 1 -- 172.21.15.67:0/1943719322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4031a80 con 0x7f3908160280 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 -- 172.21.15.67:0/1943719322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 msgr2=0x7f390806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 --2- 172.21.15.67:0/1943719322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f390806bca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3908071d80 tx=0x7f38f402b900 comp rx=0 tx=0).stop 2024-04-17T07:14:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 -- 172.21.15.67:0/1943719322 shutdown_connections 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 --2- 172.21.15.67:0/1943719322 >> v2:172.21.15.187:3300/0 conn(0x7f3908141ed0 0x7f3908142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 --2- 172.21.15.67:0/1943719322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f390806bca0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 --2- 172.21.15.67:0/1943719322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39080cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 -- 172.21.15.67:0/1943719322 >> 172.21.15.67:0/1943719322 conn(0x7f3908067d10 msgr2=0x7f39080b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 -- 172.21.15.67:0/1943719322 shutdown_connections 2024-04-17T07:14:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.167+0000 7f3907200640 1 -- 172.21.15.67:0/1943719322 wait complete. 2024-04-17T07:14:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.169+0000 7f3907200640 1 Processor -- start 2024-04-17T07:14:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.169+0000 7f3907200640 1 -- start start 2024-04-17T07:14:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39081f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3908141ed0 0x7f390813f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f390806c1e0 con 0x7f3908160280 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39080cddf0 con 0x7f3908141ed0 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3907200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3908161090 con 0x7f390806b8c0 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3906800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3906800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42196/0 (socket says 172.21.15.67:42196) 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3906800640 1 -- 172.21.15.67:0/97304405 learned_addr learned my addr 172.21.15.67:0/97304405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3905400640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3908141ed0 0x7f390813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.170+0000 7f3905e00640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39081f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 -- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 msgr2=0x7f39081f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39081f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 -- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3908141ed0 msgr2=0x7f390813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3908141ed0 0x7f390813f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 -- 172.21.15.67:0/97304405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390813f550 con 0x7f3908160280 2024-04-17T07:14:08.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3906800640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f390000cef0 tx=0x7f390000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390001a070 con 0x7f3908160280 2024-04-17T07:14:08.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f390000d860 con 0x7f3908160280 2024-04-17T07:14:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390000dc50 con 0x7f3908160280 2024-04-17T07:14:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3908143450 con 0x7f3908160280 2024-04-17T07:14:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.171+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3908143a10 con 0x7f3908160280 2024-04-17T07:14:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.172+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3908068a10 con 0x7f3908160280 2024-04-17T07:14:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.173+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f390001d070 con 0x7f3908160280 2024-04-17T07:14:08.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.174+0000 7f38fe000640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 0x7f38e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.174+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f390008c390 con 0x7f3908160280 2024-04-17T07:14:08.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.175+0000 7f3905e00640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 0x7f38e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.176+0000 7f3905e00640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 0x7f38e806edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38f800a810 tx=0x7f38f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.179+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f390005a910 con 0x7f3908160280 2024-04-17T07:14:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.377+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39080cf2b0 con 0x7f38e806c8f0 2024-04-17T07:14:08.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.396+0000 7f38fe000640 1 -- 172.21.15.67:0/97304405 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7f39080cf2b0 con 0x7f38e806c8f0 2024-04-17T07:14:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:08.401 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:14:08.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 msgr2=0x7f38e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 0x7f38e806edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38f800a810 tx=0x7f38f8008040 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 msgr2=0x7f39080b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f390000cef0 tx=0x7f390000a710 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 shutdown_connections 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908160280 0x7f39080b5290 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3908141ed0 0x7f390813f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f38e806c8f0 0x7f38e806edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 --2- 172.21.15.67:0/97304405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f390806b8c0 0x7f39081f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 >> 172.21.15.67:0/97304405 conn(0x7f3908067d10 msgr2=0x7f390813fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 shutdown_connections 2024-04-17T07:14:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.402+0000 7f3907200640 1 -- 172.21.15.67:0/97304405 wait complete. 2024-04-17T07:14:08.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-17T07:14:07.356373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223332,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751388,"statfs":{"total":773094113280,"available":772865421312,"internally_reserved":0,"allocated":3055616,"data_stored":1816228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.169515"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228829+0000","last_change":"2024-04-17T07:13:58.227207+0000","last_active":"2024-04-17T07:13:59.228829+0000","last_peered":"2024-04-17T07:13:59.228829+0000","last_clean":"2024-04-17T07:13:59.228829+0000","last_became_active":"2024-04-17T07:13:58.223412+0000","last_became_peered":"2024-04-17T07:13:58.223412+0000","last_unstale":"2024-04-17T07:13:59.228829+0000","last_undegraded":"2024-04-17T07:13:59.228829+0000","last_fullsized":"2024-04-17T07:13:59.228829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217686+0000","last_change":"2024-04-17T07:13:58.238529+0000","last_active":"2024-04-17T07:13:59.217686+0000","last_peered":"2024-04-17T07:13:59.217686+0000","last_clean":"2024-04-17T07:13:59.217686+0000","last_became_active":"2024-04-17T07:13:58.238155+0000","last_became_peered":"2024-04-17T07:13:58.238155+0000","last_unstale":"2024-04-17T07:13:59.217686+0000","last_undegraded":"2024-04-17T07:13:59.217686+0000","last_fullsized":"2024-04-17T07:13:59.217686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148756+0000","last_change":"2024-04-17T07:13:58.242232+0000","last_active":"2024-04-17T07:14:00.148756+0000","last_peered":"2024-04-17T07:14:00.148756+0000","last_clean":"2024-04-17T07:14:00.148756+0000","last_became_active":"2024-04-17T07:13:58.241891+0000","last_became_peered":"2024-04-17T07:13:58.241891+0000","last_unstale":"2024-04-17T07:14:00.148756+0000","last_undegraded":"2024-04-17T07:14:00.148756+0000","last_fullsized":"2024-04-17T07:14:00.148756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:17.659180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229570+0000","last_change":"2024-04-17T07:13:58.234270+0000","last_active":"2024-04-17T07:13:59.229570+0000","last_peered":"2024-04-17T07:13:59.229570+0000","last_clean":"2024-04-17T07:13:59.229570+0000","last_became_active":"2024-04-17T07:13:58.233725+0000","last_became_peered":"2024-04-17T07:13:58.233725+0000","last_unstale":"2024-04-17T07:13:59.229570+0000","last_undegraded":"2024-04-17T07:13:59.229570+0000","last_fullsized":"2024-04-17T07:13:59.229570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226745+0000","last_change":"2024-04-17T07:13:58.215673+0000","last_active":"2024-04-17T07:13:59.226745+0000","last_peered":"2024-04-17T07:13:59.226745+0000","last_clean":"2024-04-17T07:13:59.226745+0000","last_became_active":"2024-04-17T07:13:58.215371+0000","last_became_peered":"2024-04-17T07:13:58.215371+0000","last_unstale":"2024-04-17T07:13:59.226745+0000","last_undegraded":"2024-04-17T07:13:59.226745+0000","last_fullsized":"2024-04-17T07:13:59.226745+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228445+0000","last_change":"2024-04-17T07:13:58.240332+0000","last_active":"2024-04-17T07:13:59.228445+0000","last_peered":"2024-04-17T07:13:59.228445+0000","last_clean":"2024-04-17T07:13:59.228445+0000","last_became_active":"2024-04-17T07:13:58.239852+0000","last_became_peered":"2024-04-17T07:13:58.239852+0000","last_unstale":"2024-04-17T07:13:59.228445+0000","last_undegraded":"2024-04-17T07:13:59.228445+0000","last_fullsized":"2024-04-17T07:13:59.228445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226714+0000","last_change":"2024-04-17T07:13:58.238930+0000","last_active":"2024-04-17T07:13:59.226714+0000","last_peered":"2024-04-17T07:13:59.226714+0000","last_clean":"2024-04-17T07:13:59.226714+0000","last_became_active":"2024-04-17T07:13:58.238663+0000","last_became_peered":"2024-04-17T07:13:58.238663+0000","last_unstale":"2024-04-17T07:13:59.226714+0000","last_undegraded":"2024-04-17T07:13:59.226714+0000","last_fullsized":"2024-04-17T07:13:59.226714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144325+0000","last_change":"2024-04-17T07:13:58.227713+0000","last_active":"2024-04-17T07:14:00.144325+0000","last_peered":"2024-04-17T07:14:00.144325+0000","last_clean":"2024-04-17T07:14:00.144325+0000","last_became_active":"2024-04-17T07:13:58.227352+0000","last_became_peered":"2024-04-17T07:13:58.227352+0000","last_unstale":"2024-04-17T07:14:00.144325+0000","last_undegraded":"2024-04-17T07:14:00.144325+0000","last_fullsized":"2024-04-17T07:14:00.144325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228952+0000","last_change":"2024-04-17T07:13:58.233019+0000","last_active":"2024-04-17T07:13:59.228952+0000","last_peered":"2024-04-17T07:13:59.228952+0000","last_clean":"2024-04-17T07:13:59.228952+0000","last_became_active":"2024-04-17T07:13:58.230661+0000","last_became_peered":"2024-04-17T07:13:58.230661+0000","last_unstale":"2024-04-17T07:13:59.228952+0000","last_undegraded":"2024-04-17T07:13:59.228952+0000","last_fullsized":"2024-04-17T07:13:59.228952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:33.970758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226560+0000","last_change":"2024-04-17T07:13:58.225623+0000","last_active":"2024-04-17T07:13:59.226560+0000","last_peered":"2024-04-17T07:13:59.226560+0000","last_clean":"2024-04-17T07:13:59.226560+0000","last_became_active":"2024-04-17T07:13:58.225345+0000","last_became_peered":"2024-04-17T07:13:58.225345+0000","last_unstale":"2024-04-17T07:13:59.226560+0000","last_undegraded":"2024-04-17T07:13:59.226560+0000","last_fullsized":"2024-04-17T07:13:59.226560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226194+0000","last_change":"2024-04-17T07:13:58.226592+0000","last_active":"2024-04-17T07:13:59.226194+0000","last_peered":"2024-04-17T07:13:59.226194+0000","last_clean":"2024-04-17T07:13:59.226194+0000","last_became_active":"2024-04-17T07:13:58.226241+0000","last_became_peered":"2024-04-17T07:13:58.226241+0000","last_unstale":"2024-04-17T07:13:59.226194+0000","last_undegraded":"2024-04-17T07:13:59.226194+0000","last_fullsized":"2024-04-17T07:13:59.226194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144301+0000","last_change":"2024-04-17T07:13:58.216920+0000","last_active":"2024-04-17T07:14:00.144301+0000","last_peered":"2024-04-17T07:14:00.144301+0000","last_clean":"2024-04-17T07:14:00.144301+0000","last_became_active":"2024-04-17T07:13:58.216635+0000","last_became_peered":"2024-04-17T07:13:58.216635+0000","last_unstale":"2024-04-17T07:14:00.144301+0000","last_undegraded":"2024-04-17T07:14:00.144301+0000","last_fullsized":"2024-04-17T07:14:00.144301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:51.668433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146122+0000","last_change":"2024-04-17T07:13:58.225422+0000","last_active":"2024-04-17T07:14:00.146122+0000","last_peered":"2024-04-17T07:14:00.146122+0000","last_clean":"2024-04-17T07:14:00.146122+0000","last_became_active":"2024-04-17T07:13:58.224906+0000","last_became_peered":"2024-04-17T07:13:58.224906+0000","last_unstale":"2024-04-17T07:14:00.146122+0000","last_undegraded":"2024-04-17T07:14:00.146122+0000","last_fullsized":"2024-04-17T07:14:00.146122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147718+0000","last_change":"2024-04-17T07:13:58.233409+0000","last_active":"2024-04-17T07:14:00.147718+0000","last_peered":"2024-04-17T07:14:00.147718+0000","last_clean":"2024-04-17T07:14:00.147718+0000","last_became_active":"2024-04-17T07:13:58.233182+0000","last_became_peered":"2024-04-17T07:13:58.233182+0000","last_unstale":"2024-04-17T07:14:00.147718+0000","last_undegraded":"2024-04-17T07:14:00.147718+0000","last_fullsized":"2024-04-17T07:14:00.147718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:18.125280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217646+0000","last_change":"2024-04-17T07:13:58.215689+0000","last_active":"2024-04-17T07:13:59.217646+0000","last_peered":"2024-04-17T07:13:59.217646+0000","last_clean":"2024-04-17T07:13:59.217646+0000","last_became_active":"2024-04-17T07:13:58.215054+0000","last_became_peered":"2024-04-17T07:13:58.215054+0000","last_unstale":"2024-04-17T07:13:59.217646+0000","last_undegraded":"2024-04-17T07:13:59.217646+0000","last_fullsized":"2024-04-17T07:13:59.217646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227948+0000","last_change":"2024-04-17T07:13:58.220483+0000","last_active":"2024-04-17T07:13:59.227948+0000","last_peered":"2024-04-17T07:13:59.227948+0000","last_clean":"2024-04-17T07:13:59.227948+0000","last_became_active":"2024-04-17T07:13:58.220205+0000","last_became_peered":"2024-04-17T07:13:58.220205+0000","last_unstale":"2024-04-17T07:13:59.227948+0000","last_undegraded":"2024-04-17T07:13:59.227948+0000","last_fullsized":"2024-04-17T07:13:59.227948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231680+0000","last_change":"2024-04-17T07:13:58.223188+0000","last_active":"2024-04-17T07:13:59.231680+0000","last_peered":"2024-04-17T07:13:59.231680+0000","last_clean":"2024-04-17T07:13:59.231680+0000","last_became_active":"2024-04-17T07:13:58.221699+0000","last_became_peered":"2024-04-17T07:13:58.221699+0000","last_unstale":"2024-04-17T07:13:59.231680+0000","last_undegraded":"2024-04-17T07:13:59.231680+0000","last_fullsized":"2024-04-17T07:13:59.231680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144144+0000","last_change":"2024-04-17T07:13:58.220308+0000","last_active":"2024-04-17T07:14:00.144144+0000","last_peered":"2024-04-17T07:14:00.144144+0000","last_clean":"2024-04-17T07:14:00.144144+0000","last_became_active":"2024-04-17T07:13:58.219976+0000","last_became_peered":"2024-04-17T07:13:58.219976+0000","last_unstale":"2024-04-17T07:14:00.144144+0000","last_undegraded":"2024-04-17T07:14:00.144144+0000","last_fullsized":"2024-04-17T07:14:00.144144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:11.629034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228541+0000","last_change":"2024-04-17T07:13:58.223620+0000","last_active":"2024-04-17T07:13:59.228541+0000","last_peered":"2024-04-17T07:13:59.228541+0000","last_clean":"2024-04-17T07:13:59.228541+0000","last_became_active":"2024-04-17T07:13:58.222125+0000","last_became_peered":"2024-04-17T07:13:58.222125+0000","last_unstale":"2024-04-17T07:13:59.228541+0000","last_undegraded":"2024-04-17T07:13:59.228541+0000","last_fullsized":"2024-04-17T07:13:59.228541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227555+0000","last_change":"2024-04-17T07:13:58.225907+0000","last_active":"2024-04-17T07:13:59.227555+0000","last_peered":"2024-04-17T07:13:59.227555+0000","last_clean":"2024-04-17T07:13:59.227555+0000","last_became_active":"2024-04-17T07:13:58.224993+0000","last_became_peered":"2024-04-17T07:13:58.224993+0000","last_unstale":"2024-04-17T07:13:59.227555+0000","last_undegraded":"2024-04-17T07:13:59.227555+0000","last_fullsized":"2024-04-17T07:13:59.227555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148489+0000","last_change":"2024-04-17T07:13:58.236523+0000","last_active":"2024-04-17T07:14:00.148489+0000","last_peered":"2024-04-17T07:14:00.148489+0000","last_clean":"2024-04-17T07:14:00.148489+0000","last_became_active":"2024-04-17T07:13:58.236164+0000","last_became_peered":"2024-04-17T07:13:58.236164+0000","last_unstale":"2024-04-17T07:14:00.148489+0000","last_undegraded":"2024-04-17T07:14:00.148489+0000","last_fullsized":"2024-04-17T07:14:00.148489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:33.648495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228822+0000","last_change":"2024-04-17T07:13:58.222632+0000","last_active":"2024-04-17T07:13:59.228822+0000","last_peered":"2024-04-17T07:13:59.228822+0000","last_clean":"2024-04-17T07:13:59.228822+0000","last_became_active":"2024-04-17T07:13:58.221666+0000","last_became_peered":"2024-04-17T07:13:58.221666+0000","last_unstale":"2024-04-17T07:13:59.228822+0000","last_undegraded":"2024-04-17T07:13:59.228822+0000","last_fullsized":"2024-04-17T07:13:59.228822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145740+0000","last_change":"2024-04-17T07:13:58.218642+0000","last_active":"2024-04-17T07:14:00.145740+0000","last_peered":"2024-04-17T07:14:00.145740+0000","last_clean":"2024-04-17T07:14:00.145740+0000","last_became_active":"2024-04-17T07:13:58.218235+0000","last_became_peered":"2024-04-17T07:13:58.218235+0000","last_unstale":"2024-04-17T07:14:00.145740+0000","last_undegraded":"2024-04-17T07:14:00.145740+0000","last_fullsized":"2024-04-17T07:14:00.145740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228851+0000","last_change":"2024-04-17T07:13:58.232074+0000","last_active":"2024-04-17T07:13:59.228851+0000","last_peered":"2024-04-17T07:13:59.228851+0000","last_clean":"2024-04-17T07:13:59.228851+0000","last_became_active":"2024-04-17T07:13:58.231716+0000","last_became_peered":"2024-04-17T07:13:58.231716+0000","last_unstale":"2024-04-17T07:13:59.228851+0000","last_undegraded":"2024-04-17T07:13:59.228851+0000","last_fullsized":"2024-04-17T07:13:59.228851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147203+0000","last_change":"2024-04-17T07:13:58.230039+0000","last_active":"2024-04-17T07:14:00.147203+0000","last_peered":"2024-04-17T07:14:00.147203+0000","last_clean":"2024-04-17T07:14:00.147203+0000","last_became_active":"2024-04-17T07:13:58.224374+0000","last_became_peered":"2024-04-17T07:13:58.224374+0000","last_unstale":"2024-04-17T07:14:00.147203+0000","last_undegraded":"2024-04-17T07:14:00.147203+0000","last_fullsized":"2024-04-17T07:14:00.147203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226940+0000","last_change":"2024-04-17T07:13:58.227485+0000","last_active":"2024-04-17T07:13:59.226940+0000","last_peered":"2024-04-17T07:13:59.226940+0000","last_clean":"2024-04-17T07:13:59.226940+0000","last_became_active":"2024-04-17T07:13:58.227182+0000","last_became_peered":"2024-04-17T07:13:58.227182+0000","last_unstale":"2024-04-17T07:13:59.226940+0000","last_undegraded":"2024-04-17T07:13:59.226940+0000","last_fullsized":"2024-04-17T07:13:59.226940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217943+0000","last_change":"2024-04-17T07:13:58.222243+0000","last_active":"2024-04-17T07:13:59.217943+0000","last_peered":"2024-04-17T07:13:59.217943+0000","last_clean":"2024-04-17T07:13:59.217943+0000","last_became_active":"2024-04-17T07:13:58.221925+0000","last_became_peered":"2024-04-17T07:13:58.221925+0000","last_unstale":"2024-04-17T07:13:59.217943+0000","last_undegraded":"2024-04-17T07:13:59.217943+0000","last_fullsized":"2024-04-17T07:13:59.217943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146781+0000","last_change":"2024-04-17T07:13:58.226680+0000","last_active":"2024-04-17T07:14:00.146781+0000","last_peered":"2024-04-17T07:14:00.146781+0000","last_clean":"2024-04-17T07:14:00.146781+0000","last_became_active":"2024-04-17T07:13:58.226283+0000","last_became_peered":"2024-04-17T07:13:58.226283+0000","last_unstale":"2024-04-17T07:14:00.146781+0000","last_undegraded":"2024-04-17T07:14:00.146781+0000","last_fullsized":"2024-04-17T07:14:00.146781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148779+0000","last_change":"2024-04-17T07:13:58.242478+0000","last_active":"2024-04-17T07:14:00.148779+0000","last_peered":"2024-04-17T07:14:00.148779+0000","last_clean":"2024-04-17T07:14:00.148779+0000","last_became_active":"2024-04-17T07:13:58.241968+0000","last_became_peered":"2024-04-17T07:13:58.241968+0000","last_unstale":"2024-04-17T07:14:00.148779+0000","last_undegraded":"2024-04-17T07:14:00.148779+0000","last_fullsized":"2024-04-17T07:14:00.148779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:29:46.240069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229245+0000","last_change":"2024-04-17T07:13:58.230046+0000","last_active":"2024-04-17T07:13:59.229245+0000","last_peered":"2024-04-17T07:13:59.229245+0000","last_clean":"2024-04-17T07:13:59.229245+0000","last_became_active":"2024-04-17T07:13:58.229744+0000","last_became_peered":"2024-04-17T07:13:58.229744+0000","last_unstale":"2024-04-17T07:13:59.229245+0000","last_undegraded":"2024-04-17T07:13:59.229245+0000","last_fullsized":"2024-04-17T07:13:59.229245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143075+0000","last_change":"2024-04-17T07:13:58.219601+0000","last_active":"2024-04-17T07:14:00.143075+0000","last_peered":"2024-04-17T07:14:00.143075+0000","last_clean":"2024-04-17T07:14:00.143075+0000","last_became_active":"2024-04-17T07:13:58.219208+0000","last_became_peered":"2024-04-17T07:13:58.219208+0000","last_unstale":"2024-04-17T07:14:00.143075+0000","last_undegraded":"2024-04-17T07:14:00.143075+0000","last_fullsized":"2024-04-17T07:14:00.143075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:06.342021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225081+0000","last_change":"2024-04-17T07:13:58.230063+0000","last_active":"2024-04-17T07:13:59.225081+0000","last_peered":"2024-04-17T07:13:59.225081+0000","last_clean":"2024-04-17T07:13:59.225081+0000","last_became_active":"2024-04-17T07:13:58.229129+0000","last_became_peered":"2024-04-17T07:13:58.229129+0000","last_unstale":"2024-04-17T07:13:59.225081+0000","last_undegraded":"2024-04-17T07:13:59.225081+0000","last_fullsized":"2024-04-17T07:13:59.225081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:59.849640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143839+0000","last_change":"2024-04-17T07:13:58.208959+0000","last_active":"2024-04-17T07:14:00.143839+0000","last_peered":"2024-04-17T07:14:00.143839+0000","last_clean":"2024-04-17T07:14:00.143839+0000","last_became_active":"2024-04-17T07:13:58.208200+0000","last_became_peered":"2024-04-17T07:13:58.208200+0000","last_unstale":"2024-04-17T07:14:00.143839+0000","last_undegraded":"2024-04-17T07:14:00.143839+0000","last_fullsized":"2024-04-17T07:14:00.143839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:14.893023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147150+0000","last_change":"2024-04-17T07:13:57.193193+0000","last_active":"2024-04-17T07:14:00.147150+0000","last_peered":"2024-04-17T07:14:00.147150+0000","last_clean":"2024-04-17T07:14:00.147150+0000","last_became_active":"2024-04-17T07:13:57.192931+0000","last_became_peered":"2024-04-17T07:13:57.192931+0000","last_unstale":"2024-04-17T07:14:00.147150+0000","last_undegraded":"2024-04-17T07:14:00.147150+0000","last_fullsized":"2024-04-17T07:14:00.147150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148893+0000","last_change":"2024-04-17T07:13:58.244126+0000","last_active":"2024-04-17T07:14:00.148893+0000","last_peered":"2024-04-17T07:14:00.148893+0000","last_clean":"2024-04-17T07:14:00.148893+0000","last_became_active":"2024-04-17T07:13:58.243846+0000","last_became_peered":"2024-04-17T07:13:58.243846+0000","last_unstale":"2024-04-17T07:14:00.148893+0000","last_undegraded":"2024-04-17T07:14:00.148893+0000","last_fullsized":"2024-04-17T07:14:00.148893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:20.038754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225363+0000","last_change":"2024-04-17T07:13:57.198746+0000","last_active":"2024-04-17T07:13:59.225363+0000","last_peered":"2024-04-17T07:13:59.225363+0000","last_clean":"2024-04-17T07:13:59.225363+0000","last_became_active":"2024-04-17T07:13:57.195870+0000","last_became_peered":"2024-04-17T07:13:57.195870+0000","last_unstale":"2024-04-17T07:13:59.225363+0000","last_undegraded":"2024-04-17T07:13:59.225363+0000","last_fullsized":"2024-04-17T07:13:59.225363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145653+0000","last_change":"2024-04-17T07:13:57.201813+0000","last_active":"2024-04-17T07:14:00.145653+0000","last_peered":"2024-04-17T07:14:00.145653+0000","last_clean":"2024-04-17T07:14:00.145653+0000","last_became_active":"2024-04-17T07:13:57.201530+0000","last_became_peered":"2024-04-17T07:13:57.201530+0000","last_unstale":"2024-04-17T07:14:00.145653+0000","last_undegraded":"2024-04-17T07:14:00.145653+0000","last_fullsized":"2024-04-17T07:14:00.145653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144391+0000","last_change":"2024-04-17T07:13:58.235592+0000","last_active":"2024-04-17T07:14:00.144391+0000","last_peered":"2024-04-17T07:14:00.144391+0000","last_clean":"2024-04-17T07:14:00.144391+0000","last_became_active":"2024-04-17T07:13:58.235287+0000","last_became_peered":"2024-04-17T07:13:58.235287+0000","last_unstale":"2024-04-17T07:14:00.144391+0000","last_undegraded":"2024-04-17T07:14:00.144391+0000","last_fullsized":"2024-04-17T07:14:00.144391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228264+0000","last_change":"2024-04-17T07:13:57.183240+0000","last_active":"2024-04-17T07:13:59.228264+0000","last_peered":"2024-04-17T07:13:59.228264+0000","last_clean":"2024-04-17T07:13:59.228264+0000","last_became_active":"2024-04-17T07:13:57.182859+0000","last_became_peered":"2024-04-17T07:13:57.182859+0000","last_unstale":"2024-04-17T07:13:59.228264+0000","last_undegraded":"2024-04-17T07:13:59.228264+0000","last_fullsized":"2024-04-17T07:13:59.228264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217429+0000","last_change":"2024-04-17T07:13:58.221113+0000","last_active":"2024-04-17T07:13:59.217429+0000","last_peered":"2024-04-17T07:13:59.217429+0000","last_clean":"2024-04-17T07:13:59.217429+0000","last_became_active":"2024-04-17T07:13:58.220807+0000","last_became_peered":"2024-04-17T07:13:58.220807+0000","last_unstale":"2024-04-17T07:13:59.217429+0000","last_undegraded":"2024-04-17T07:13:59.217429+0000","last_fullsized":"2024-04-17T07:13:59.217429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226316+0000","last_change":"2024-04-17T07:13:57.197641+0000","last_active":"2024-04-17T07:13:59.226316+0000","last_peered":"2024-04-17T07:13:59.226316+0000","last_clean":"2024-04-17T07:13:59.226316+0000","last_became_active":"2024-04-17T07:13:57.197394+0000","last_became_peered":"2024-04-17T07:13:57.197394+0000","last_unstale":"2024-04-17T07:13:59.226316+0000","last_undegraded":"2024-04-17T07:13:59.226316+0000","last_fullsized":"2024-04-17T07:13:59.226316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231265+0000","last_change":"2024-04-17T07:13:58.225914+0000","last_active":"2024-04-17T07:13:59.231265+0000","last_peered":"2024-04-17T07:13:59.231265+0000","last_clean":"2024-04-17T07:13:59.231265+0000","last_became_active":"2024-04-17T07:13:58.225649+0000","last_became_peered":"2024-04-17T07:13:58.225649+0000","last_unstale":"2024-04-17T07:13:59.231265+0000","last_undegraded":"2024-04-17T07:13:59.231265+0000","last_fullsized":"2024-04-17T07:13:59.231265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231330+0000","last_change":"2024-04-17T07:13:57.198101+0000","last_active":"2024-04-17T07:13:59.231330+0000","last_peered":"2024-04-17T07:13:59.231330+0000","last_clean":"2024-04-17T07:13:59.231330+0000","last_became_active":"2024-04-17T07:13:57.196771+0000","last_became_peered":"2024-04-17T07:13:57.196771+0000","last_unstale":"2024-04-17T07:13:59.231330+0000","last_undegraded":"2024-04-17T07:13:59.231330+0000","last_fullsized":"2024-04-17T07:13:59.231330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149355+0000","last_change":"2024-04-17T07:13:58.244824+0000","last_active":"2024-04-17T07:14:00.149355+0000","last_peered":"2024-04-17T07:14:00.149355+0000","last_clean":"2024-04-17T07:14:00.149355+0000","last_became_active":"2024-04-17T07:13:58.243368+0000","last_became_peered":"2024-04-17T07:13:58.243368+0000","last_unstale":"2024-04-17T07:14:00.149355+0000","last_undegraded":"2024-04-17T07:14:00.149355+0000","last_fullsized":"2024-04-17T07:14:00.149355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:43.836722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218062+0000","last_change":"2024-04-17T07:13:57.198519+0000","last_active":"2024-04-17T07:13:59.218062+0000","last_peered":"2024-04-17T07:13:59.218062+0000","last_clean":"2024-04-17T07:13:59.218062+0000","last_became_active":"2024-04-17T07:13:57.197225+0000","last_became_peered":"2024-04-17T07:13:57.197225+0000","last_unstale":"2024-04-17T07:13:59.218062+0000","last_undegraded":"2024-04-17T07:13:59.218062+0000","last_fullsized":"2024-04-17T07:13:59.218062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144417+0000","last_change":"2024-04-17T07:13:58.221225+0000","last_active":"2024-04-17T07:14:00.144417+0000","last_peered":"2024-04-17T07:14:00.144417+0000","last_clean":"2024-04-17T07:14:00.144417+0000","last_became_active":"2024-04-17T07:13:58.220917+0000","last_became_peered":"2024-04-17T07:13:58.220917+0000","last_unstale":"2024-04-17T07:14:00.144417+0000","last_undegraded":"2024-04-17T07:14:00.144417+0000","last_fullsized":"2024-04-17T07:14:00.144417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:04.188468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226310+0000","last_change":"2024-04-17T07:13:57.200719+0000","last_active":"2024-04-17T07:13:59.226310+0000","last_peered":"2024-04-17T07:13:59.226310+0000","last_clean":"2024-04-17T07:13:59.226310+0000","last_became_active":"2024-04-17T07:13:57.200404+0000","last_became_peered":"2024-04-17T07:13:57.200404+0000","last_unstale":"2024-04-17T07:13:59.226310+0000","last_undegraded":"2024-04-17T07:13:59.226310+0000","last_fullsized":"2024-04-17T07:13:59.226310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226654+0000","last_change":"2024-04-17T07:13:58.231037+0000","last_active":"2024-04-17T07:13:59.226654+0000","last_peered":"2024-04-17T07:13:59.226654+0000","last_clean":"2024-04-17T07:13:59.226654+0000","last_became_active":"2024-04-17T07:13:58.229754+0000","last_became_peered":"2024-04-17T07:13:58.229754+0000","last_unstale":"2024-04-17T07:13:59.226654+0000","last_undegraded":"2024-04-17T07:13:59.226654+0000","last_fullsized":"2024-04-17T07:13:59.226654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149082+0000","last_change":"2024-04-17T07:13:57.196583+0000","last_active":"2024-04-17T07:14:00.149082+0000","last_peered":"2024-04-17T07:14:00.149082+0000","last_clean":"2024-04-17T07:14:00.149082+0000","last_became_active":"2024-04-17T07:13:57.195074+0000","last_became_peered":"2024-04-17T07:13:57.195074+0000","last_unstale":"2024-04-17T07:14:00.149082+0000","last_undegraded":"2024-04-17T07:14:00.149082+0000","last_fullsized":"2024-04-17T07:14:00.149082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:53.288607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148521+0000","last_change":"2024-04-17T07:13:58.243294+0000","last_active":"2024-04-17T07:14:00.148521+0000","last_peered":"2024-04-17T07:14:00.148521+0000","last_clean":"2024-04-17T07:14:00.148521+0000","last_became_active":"2024-04-17T07:13:58.241931+0000","last_became_peered":"2024-04-17T07:13:58.241931+0000","last_unstale":"2024-04-17T07:14:00.148521+0000","last_undegraded":"2024-04-17T07:14:00.148521+0000","last_fullsized":"2024-04-17T07:14:00.148521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:49.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229051+0000","last_change":"2024-04-17T07:13:57.190224+0000","last_active":"2024-04-17T07:13:59.229051+0000","last_peered":"2024-04-17T07:13:59.229051+0000","last_clean":"2024-04-17T07:13:59.229051+0000","last_became_active":"2024-04-17T07:13:57.189879+0000","last_became_peered":"2024-04-17T07:13:57.189879+0000","last_unstale":"2024-04-17T07:13:59.229051+0000","last_undegraded":"2024-04-17T07:13:59.229051+0000","last_fullsized":"2024-04-17T07:13:59.229051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226963+0000","last_change":"2024-04-17T07:13:58.235859+0000","last_active":"2024-04-17T07:13:59.226963+0000","last_peered":"2024-04-17T07:13:59.226963+0000","last_clean":"2024-04-17T07:13:59.226963+0000","last_became_active":"2024-04-17T07:13:58.235565+0000","last_became_peered":"2024-04-17T07:13:58.235565+0000","last_unstale":"2024-04-17T07:13:59.226963+0000","last_undegraded":"2024-04-17T07:13:59.226963+0000","last_fullsized":"2024-04-17T07:13:59.226963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226241+0000","last_change":"2024-04-17T07:13:57.202465+0000","last_active":"2024-04-17T07:13:59.226241+0000","last_peered":"2024-04-17T07:13:59.226241+0000","last_clean":"2024-04-17T07:13:59.226241+0000","last_became_active":"2024-04-17T07:13:57.202105+0000","last_became_peered":"2024-04-17T07:13:57.202105+0000","last_unstale":"2024-04-17T07:13:59.226241+0000","last_undegraded":"2024-04-17T07:13:59.226241+0000","last_fullsized":"2024-04-17T07:13:59.226241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225347+0000","last_change":"2024-04-17T07:13:58.228208+0000","last_active":"2024-04-17T07:13:59.225347+0000","last_peered":"2024-04-17T07:13:59.225347+0000","last_clean":"2024-04-17T07:13:59.225347+0000","last_became_active":"2024-04-17T07:13:58.227889+0000","last_became_peered":"2024-04-17T07:13:58.227889+0000","last_unstale":"2024-04-17T07:13:59.225347+0000","last_undegraded":"2024-04-17T07:13:59.225347+0000","last_fullsized":"2024-04-17T07:13:59.225347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.142947+0000","last_change":"2024-04-17T07:13:57.204927+0000","last_active":"2024-04-17T07:14:00.142947+0000","last_peered":"2024-04-17T07:14:00.142947+0000","last_clean":"2024-04-17T07:14:00.142947+0000","last_became_active":"2024-04-17T07:13:57.204546+0000","last_became_peered":"2024-04-17T07:13:57.204546+0000","last_unstale":"2024-04-17T07:14:00.142947+0000","last_undegraded":"2024-04-17T07:14:00.142947+0000","last_fullsized":"2024-04-17T07:14:00.142947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:22:01.161179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227143+0000","last_change":"2024-04-17T07:13:58.235736+0000","last_active":"2024-04-17T07:13:59.227143+0000","last_peered":"2024-04-17T07:13:59.227143+0000","last_clean":"2024-04-17T07:13:59.227143+0000","last_became_active":"2024-04-17T07:13:58.235432+0000","last_became_peered":"2024-04-17T07:13:58.235432+0000","last_unstale":"2024-04-17T07:13:59.227143+0000","last_undegraded":"2024-04-17T07:13:59.227143+0000","last_fullsized":"2024-04-17T07:13:59.227143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144456+0000","last_change":"2024-04-17T07:13:57.195110+0000","last_active":"2024-04-17T07:14:00.144456+0000","last_peered":"2024-04-17T07:14:00.144456+0000","last_clean":"2024-04-17T07:14:00.144456+0000","last_became_active":"2024-04-17T07:13:57.194788+0000","last_became_peered":"2024-04-17T07:13:57.194788+0000","last_unstale":"2024-04-17T07:14:00.144456+0000","last_undegraded":"2024-04-17T07:14:00.144456+0000","last_fullsized":"2024-04-17T07:14:00.144456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:01.392379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228617+0000","last_change":"2024-04-17T07:13:58.224809+0000","last_active":"2024-04-17T07:13:59.228617+0000","last_peered":"2024-04-17T07:13:59.228617+0000","last_clean":"2024-04-17T07:13:59.228617+0000","last_became_active":"2024-04-17T07:13:58.224516+0000","last_became_peered":"2024-04-17T07:13:58.224516+0000","last_unstale":"2024-04-17T07:13:59.228617+0000","last_undegraded":"2024-04-17T07:13:59.228617+0000","last_fullsized":"2024-04-17T07:13:59.228617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144137+0000","last_change":"2024-04-17T07:13:57.203108+0000","last_active":"2024-04-17T07:14:00.144137+0000","last_peered":"2024-04-17T07:14:00.144137+0000","last_clean":"2024-04-17T07:14:00.144137+0000","last_became_active":"2024-04-17T07:13:57.202802+0000","last_became_peered":"2024-04-17T07:13:57.202802+0000","last_unstale":"2024-04-17T07:14:00.144137+0000","last_undegraded":"2024-04-17T07:14:00.144137+0000","last_fullsized":"2024-04-17T07:14:00.144137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:54.647968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229091+0000","last_change":"2024-04-17T07:13:58.230644+0000","last_active":"2024-04-17T07:13:59.229091+0000","last_peered":"2024-04-17T07:13:59.229091+0000","last_clean":"2024-04-17T07:13:59.229091+0000","last_became_active":"2024-04-17T07:13:58.229593+0000","last_became_peered":"2024-04-17T07:13:58.229593+0000","last_unstale":"2024-04-17T07:13:59.229091+0000","last_undegraded":"2024-04-17T07:13:59.229091+0000","last_fullsized":"2024-04-17T07:13:59.229091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:57.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218173+0000","last_change":"2024-04-17T07:13:57.200619+0000","last_active":"2024-04-17T07:13:59.218173+0000","last_peered":"2024-04-17T07:13:59.218173+0000","last_clean":"2024-04-17T07:13:59.218173+0000","last_became_active":"2024-04-17T07:13:57.200293+0000","last_became_peered":"2024-04-17T07:13:57.200293+0000","last_unstale":"2024-04-17T07:13:59.218173+0000","last_undegraded":"2024-04-17T07:13:59.218173+0000","last_fullsized":"2024-04-17T07:13:59.218173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229770+0000","last_change":"2024-04-17T07:13:58.233982+0000","last_active":"2024-04-17T07:13:59.229770+0000","last_peered":"2024-04-17T07:13:59.229770+0000","last_clean":"2024-04-17T07:13:59.229770+0000","last_became_active":"2024-04-17T07:13:58.233394+0000","last_became_peered":"2024-04-17T07:13:58.233394+0000","last_unstale":"2024-04-17T07:13:59.229770+0000","last_undegraded":"2024-04-17T07:13:59.229770+0000","last_fullsized":"2024-04-17T07:13:59.229770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143798+0000","last_change":"2024-04-17T07:13:57.198237+0000","last_active":"2024-04-17T07:14:00.143798+0000","last_peered":"2024-04-17T07:14:00.143798+0000","last_clean":"2024-04-17T07:14:00.143798+0000","last_became_active":"2024-04-17T07:13:57.197870+0000","last_became_peered":"2024-04-17T07:13:57.197870+0000","last_unstale":"2024-04-17T07:14:00.143798+0000","last_undegraded":"2024-04-17T07:14:00.143798+0000","last_fullsized":"2024-04-17T07:14:00.143798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:48.074499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227126+0000","last_change":"2024-04-17T07:13:58.232896+0000","last_active":"2024-04-17T07:13:59.227126+0000","last_peered":"2024-04-17T07:13:59.227126+0000","last_clean":"2024-04-17T07:13:59.227126+0000","last_became_active":"2024-04-17T07:13:58.232155+0000","last_became_peered":"2024-04-17T07:13:58.232155+0000","last_unstale":"2024-04-17T07:13:59.227126+0000","last_undegraded":"2024-04-17T07:13:59.227126+0000","last_fullsized":"2024-04-17T07:13:59.227126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225865+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:13:59.225865+0000","last_peered":"2024-04-17T07:13:59.225865+0000","last_clean":"2024-04-17T07:13:59.225865+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:13:59.225865+0000","last_undegraded":"2024-04-17T07:13:59.225865+0000","last_fullsized":"2024-04-17T07:13:59.225865+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228095+0000","last_change":"2024-04-17T07:13:57.187149+0000","last_active":"2024-04-17T07:13:59.228095+0000","last_peered":"2024-04-17T07:13:59.228095+0000","last_clean":"2024-04-17T07:13:59.228095+0000","last_became_active":"2024-04-17T07:13:57.186832+0000","last_became_peered":"2024-04-17T07:13:57.186832+0000","last_unstale":"2024-04-17T07:13:59.228095+0000","last_undegraded":"2024-04-17T07:13:59.228095+0000","last_fullsized":"2024-04-17T07:13:59.228095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146188+0000","last_change":"2024-04-17T07:13:58.228567+0000","last_active":"2024-04-17T07:14:00.146188+0000","last_peered":"2024-04-17T07:14:00.146188+0000","last_clean":"2024-04-17T07:14:00.146188+0000","last_became_active":"2024-04-17T07:13:58.228064+0000","last_became_peered":"2024-04-17T07:13:58.228064+0000","last_unstale":"2024-04-17T07:14:00.146188+0000","last_undegraded":"2024-04-17T07:14:00.146188+0000","last_fullsized":"2024-04-17T07:14:00.146188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224678+0000","last_change":"2024-04-17T07:13:57.204847+0000","last_active":"2024-04-17T07:13:59.224678+0000","last_peered":"2024-04-17T07:13:59.224678+0000","last_clean":"2024-04-17T07:13:59.224678+0000","last_became_active":"2024-04-17T07:13:57.204534+0000","last_became_peered":"2024-04-17T07:13:57.204534+0000","last_unstale":"2024-04-17T07:13:59.224678+0000","last_undegraded":"2024-04-17T07:13:59.224678+0000","last_fullsized":"2024-04-17T07:13:59.224678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148252+0000","last_change":"2024-04-17T07:13:58.230682+0000","last_active":"2024-04-17T07:14:00.148252+0000","last_peered":"2024-04-17T07:14:00.148252+0000","last_clean":"2024-04-17T07:14:00.148252+0000","last_became_active":"2024-04-17T07:13:58.230315+0000","last_became_peered":"2024-04-17T07:13:58.230315+0000","last_unstale":"2024-04-17T07:14:00.148252+0000","last_undegraded":"2024-04-17T07:14:00.148252+0000","last_fullsized":"2024-04-17T07:14:00.148252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:03.539255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143397+0000","last_change":"2024-04-17T07:13:57.204481+0000","last_active":"2024-04-17T07:14:00.143397+0000","last_peered":"2024-04-17T07:14:00.143397+0000","last_clean":"2024-04-17T07:14:00.143397+0000","last_became_active":"2024-04-17T07:13:57.203848+0000","last_became_peered":"2024-04-17T07:13:57.203848+0000","last_unstale":"2024-04-17T07:14:00.143397+0000","last_undegraded":"2024-04-17T07:14:00.143397+0000","last_fullsized":"2024-04-17T07:14:00.143397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:11.312213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217520+0000","last_change":"2024-04-17T07:13:58.208486+0000","last_active":"2024-04-17T07:13:59.217520+0000","last_peered":"2024-04-17T07:13:59.217520+0000","last_clean":"2024-04-17T07:13:59.217520+0000","last_became_active":"2024-04-17T07:13:58.208043+0000","last_became_peered":"2024-04-17T07:13:58.208043+0000","last_unstale":"2024-04-17T07:13:59.217520+0000","last_undegraded":"2024-04-17T07:13:59.217520+0000","last_fullsized":"2024-04-17T07:13:59.217520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229248+0000","last_change":"2024-04-17T07:13:57.196519+0000","last_active":"2024-04-17T07:13:59.229248+0000","last_peered":"2024-04-17T07:13:59.229248+0000","last_clean":"2024-04-17T07:13:59.229248+0000","last_became_active":"2024-04-17T07:13:57.196148+0000","last_became_peered":"2024-04-17T07:13:57.196148+0000","last_unstale":"2024-04-17T07:13:59.229248+0000","last_undegraded":"2024-04-17T07:13:59.229248+0000","last_fullsized":"2024-04-17T07:13:59.229248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144037+0000","last_change":"2024-04-17T07:13:58.241381+0000","last_active":"2024-04-17T07:14:00.144037+0000","last_peered":"2024-04-17T07:14:00.144037+0000","last_clean":"2024-04-17T07:14:00.144037+0000","last_became_active":"2024-04-17T07:13:58.240707+0000","last_became_peered":"2024-04-17T07:13:58.240707+0000","last_unstale":"2024-04-17T07:14:00.144037+0000","last_undegraded":"2024-04-17T07:14:00.144037+0000","last_fullsized":"2024-04-17T07:14:00.144037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:39.705592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217512+0000","last_change":"2024-04-17T07:13:57.196549+0000","last_active":"2024-04-17T07:13:59.217512+0000","last_peered":"2024-04-17T07:13:59.217512+0000","last_clean":"2024-04-17T07:13:59.217512+0000","last_became_active":"2024-04-17T07:13:57.196096+0000","last_became_peered":"2024-04-17T07:13:57.196096+0000","last_unstale":"2024-04-17T07:13:59.217512+0000","last_undegraded":"2024-04-17T07:13:59.217512+0000","last_fullsized":"2024-04-17T07:13:59.217512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231381+0000","last_change":"2024-04-17T07:13:58.218704+0000","last_active":"2024-04-17T07:13:59.231381+0000","last_peered":"2024-04-17T07:13:59.231381+0000","last_clean":"2024-04-17T07:13:59.231381+0000","last_became_active":"2024-04-17T07:13:58.218383+0000","last_became_peered":"2024-04-17T07:13:58.218383+0000","last_unstale":"2024-04-17T07:13:59.231381+0000","last_undegraded":"2024-04-17T07:13:59.231381+0000","last_fullsized":"2024-04-17T07:13:59.231381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228625+0000","last_change":"2024-04-17T07:13:57.199856+0000","last_active":"2024-04-17T07:13:59.228625+0000","last_peered":"2024-04-17T07:13:59.228625+0000","last_clean":"2024-04-17T07:13:59.228625+0000","last_became_active":"2024-04-17T07:13:57.199550+0000","last_became_peered":"2024-04-17T07:13:57.199550+0000","last_unstale":"2024-04-17T07:13:59.228625+0000","last_undegraded":"2024-04-17T07:13:59.228625+0000","last_fullsized":"2024-04-17T07:13:59.228625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:36.294837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217992+0000","last_change":"2024-04-17T07:13:58.213558+0000","last_active":"2024-04-17T07:13:59.217992+0000","last_peered":"2024-04-17T07:13:59.217992+0000","last_clean":"2024-04-17T07:13:59.217992+0000","last_became_active":"2024-04-17T07:13:58.213237+0000","last_became_peered":"2024-04-17T07:13:58.213237+0000","last_unstale":"2024-04-17T07:13:59.217992+0000","last_undegraded":"2024-04-17T07:13:59.217992+0000","last_fullsized":"2024-04-17T07:13:59.217992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.222842+0000","last_change":"2024-04-17T07:13:57.198259+0000","last_active":"2024-04-17T07:13:59.222842+0000","last_peered":"2024-04-17T07:13:59.222842+0000","last_clean":"2024-04-17T07:13:59.222842+0000","last_became_active":"2024-04-17T07:13:57.197982+0000","last_became_peered":"2024-04-17T07:13:57.197982+0000","last_unstale":"2024-04-17T07:13:59.222842+0000","last_undegraded":"2024-04-17T07:13:59.222842+0000","last_fullsized":"2024-04-17T07:13:59.222842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147642+0000","last_change":"2024-04-17T07:13:58.224013+0000","last_active":"2024-04-17T07:14:00.147642+0000","last_peered":"2024-04-17T07:14:00.147642+0000","last_clean":"2024-04-17T07:14:00.147642+0000","last_became_active":"2024-04-17T07:13:58.223705+0000","last_became_peered":"2024-04-17T07:13:58.223705+0000","last_unstale":"2024-04-17T07:14:00.147642+0000","last_undegraded":"2024-04-17T07:14:00.147642+0000","last_fullsized":"2024-04-17T07:14:00.147642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226587+0000","last_change":"2024-04-17T07:13:57.200987+0000","last_active":"2024-04-17T07:13:59.226587+0000","last_peered":"2024-04-17T07:13:59.226587+0000","last_clean":"2024-04-17T07:13:59.226587+0000","last_became_active":"2024-04-17T07:13:57.200681+0000","last_became_peered":"2024-04-17T07:13:57.200681+0000","last_unstale":"2024-04-17T07:13:59.226587+0000","last_undegraded":"2024-04-17T07:13:59.226587+0000","last_fullsized":"2024-04-17T07:13:59.226587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149119+0000","last_change":"2024-04-17T07:13:58.234490+0000","last_active":"2024-04-17T07:14:00.149119+0000","last_peered":"2024-04-17T07:14:00.149119+0000","last_clean":"2024-04-17T07:14:00.149119+0000","last_became_active":"2024-04-17T07:13:58.234250+0000","last_became_peered":"2024-04-17T07:13:58.234250+0000","last_unstale":"2024-04-17T07:14:00.149119+0000","last_undegraded":"2024-04-17T07:14:00.149119+0000","last_fullsized":"2024-04-17T07:14:00.149119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:17.137422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224627+0000","last_change":"2024-04-17T07:13:57.199555+0000","last_active":"2024-04-17T07:13:59.224627+0000","last_peered":"2024-04-17T07:13:59.224627+0000","last_clean":"2024-04-17T07:13:59.224627+0000","last_became_active":"2024-04-17T07:13:57.199295+0000","last_became_peered":"2024-04-17T07:13:57.199295+0000","last_unstale":"2024-04-17T07:13:59.224627+0000","last_undegraded":"2024-04-17T07:13:59.224627+0000","last_fullsized":"2024-04-17T07:13:59.224627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143813+0000","last_change":"2024-04-17T07:13:58.222076+0000","last_active":"2024-04-17T07:14:00.143813+0000","last_peered":"2024-04-17T07:14:00.143813+0000","last_clean":"2024-04-17T07:14:00.143813+0000","last_became_active":"2024-04-17T07:13:58.219377+0000","last_became_peered":"2024-04-17T07:13:58.219377+0000","last_unstale":"2024-04-17T07:14:00.143813+0000","last_undegraded":"2024-04-17T07:14:00.143813+0000","last_fullsized":"2024-04-17T07:14:00.143813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:37.495822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217874+0000","last_change":"2024-04-17T07:13:57.203197+0000","last_active":"2024-04-17T07:13:59.217874+0000","last_peered":"2024-04-17T07:13:59.217874+0000","last_clean":"2024-04-17T07:13:59.217874+0000","last_became_active":"2024-04-17T07:13:57.201296+0000","last_became_peered":"2024-04-17T07:13:57.201296+0000","last_unstale":"2024-04-17T07:13:59.217874+0000","last_undegraded":"2024-04-17T07:13:59.217874+0000","last_fullsized":"2024-04-17T07:13:59.217874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144614+0000","last_change":"2024-04-17T07:13:58.220538+0000","last_active":"2024-04-17T07:14:00.144614+0000","last_peered":"2024-04-17T07:14:00.144614+0000","last_clean":"2024-04-17T07:14:00.144614+0000","last_became_active":"2024-04-17T07:13:58.220281+0000","last_became_peered":"2024-04-17T07:13:58.220281+0000","last_unstale":"2024-04-17T07:14:00.144614+0000","last_undegraded":"2024-04-17T07:14:00.144614+0000","last_fullsized":"2024-04-17T07:14:00.144614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217905+0000","last_change":"2024-04-17T07:13:57.202287+0000","last_active":"2024-04-17T07:13:59.217905+0000","last_peered":"2024-04-17T07:13:59.217905+0000","last_clean":"2024-04-17T07:13:59.217905+0000","last_became_active":"2024-04-17T07:13:57.202007+0000","last_became_peered":"2024-04-17T07:13:57.202007+0000","last_unstale":"2024-04-17T07:13:59.217905+0000","last_undegraded":"2024-04-17T07:13:59.217905+0000","last_fullsized":"2024-04-17T07:13:59.217905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146417+0000","last_change":"2024-04-17T07:13:58.243714+0000","last_active":"2024-04-17T07:14:00.146417+0000","last_peered":"2024-04-17T07:14:00.146417+0000","last_clean":"2024-04-17T07:14:00.146417+0000","last_became_active":"2024-04-17T07:13:58.243199+0000","last_became_peered":"2024-04-17T07:13:58.243199+0000","last_unstale":"2024-04-17T07:14:00.146417+0000","last_undegraded":"2024-04-17T07:14:00.146417+0000","last_fullsized":"2024-04-17T07:14:00.146417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227303+0000","last_change":"2024-04-17T07:13:57.194783+0000","last_active":"2024-04-17T07:13:59.227303+0000","last_peered":"2024-04-17T07:13:59.227303+0000","last_clean":"2024-04-17T07:13:59.227303+0000","last_became_active":"2024-04-17T07:13:57.194418+0000","last_became_peered":"2024-04-17T07:13:57.194418+0000","last_unstale":"2024-04-17T07:13:59.227303+0000","last_undegraded":"2024-04-17T07:13:59.227303+0000","last_fullsized":"2024-04-17T07:13:59.227303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.173128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225654+0000","last_change":"2024-04-17T07:13:58.224933+0000","last_active":"2024-04-17T07:13:59.225654+0000","last_peered":"2024-04-17T07:13:59.225654+0000","last_clean":"2024-04-17T07:13:59.225654+0000","last_became_active":"2024-04-17T07:13:58.224635+0000","last_became_peered":"2024-04-17T07:13:58.224635+0000","last_unstale":"2024-04-17T07:13:59.225654+0000","last_undegraded":"2024-04-17T07:13:59.225654+0000","last_fullsized":"2024-04-17T07:13:59.225654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231353+0000","last_change":"2024-04-17T07:13:58.224558+0000","last_active":"2024-04-17T07:13:59.231353+0000","last_peered":"2024-04-17T07:13:59.231353+0000","last_clean":"2024-04-17T07:13:59.231353+0000","last_became_active":"2024-04-17T07:13:58.224291+0000","last_became_peered":"2024-04-17T07:13:58.224291+0000","last_unstale":"2024-04-17T07:13:59.231353+0000","last_undegraded":"2024-04-17T07:13:59.231353+0000","last_fullsized":"2024-04-17T07:13:59.231353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226442+0000","last_change":"2024-04-17T07:13:57.197610+0000","last_active":"2024-04-17T07:13:59.226442+0000","last_peered":"2024-04-17T07:13:59.226442+0000","last_clean":"2024-04-17T07:13:59.226442+0000","last_became_active":"2024-04-17T07:13:57.197297+0000","last_became_peered":"2024-04-17T07:13:57.197297+0000","last_unstale":"2024-04-17T07:13:59.226442+0000","last_undegraded":"2024-04-17T07:13:59.226442+0000","last_fullsized":"2024-04-17T07:13:59.226442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229831+0000","last_change":"2024-04-17T07:13:58.227513+0000","last_active":"2024-04-17T07:13:59.229831+0000","last_peered":"2024-04-17T07:13:59.229831+0000","last_clean":"2024-04-17T07:13:59.229831+0000","last_became_active":"2024-04-17T07:13:58.227195+0000","last_became_peered":"2024-04-17T07:13:58.227195+0000","last_unstale":"2024-04-17T07:13:59.229831+0000","last_undegraded":"2024-04-17T07:13:59.229831+0000","last_fullsized":"2024-04-17T07:13:59.229831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228330+0000","last_change":"2024-04-17T07:13:57.203401+0000","last_active":"2024-04-17T07:13:59.228330+0000","last_peered":"2024-04-17T07:13:59.228330+0000","last_clean":"2024-04-17T07:13:59.228330+0000","last_became_active":"2024-04-17T07:13:57.202900+0000","last_became_peered":"2024-04-17T07:13:57.202900+0000","last_unstale":"2024-04-17T07:13:59.228330+0000","last_undegraded":"2024-04-17T07:13:59.228330+0000","last_fullsized":"2024-04-17T07:13:59.228330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:16.876460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148975+0000","last_change":"2024-04-17T07:13:58.245486+0000","last_active":"2024-04-17T07:14:00.148975+0000","last_peered":"2024-04-17T07:14:00.148975+0000","last_clean":"2024-04-17T07:14:00.148975+0000","last_became_active":"2024-04-17T07:13:58.245180+0000","last_became_peered":"2024-04-17T07:13:58.245180+0000","last_unstale":"2024-04-17T07:14:00.148975+0000","last_undegraded":"2024-04-17T07:14:00.148975+0000","last_fullsized":"2024-04-17T07:14:00.148975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:42.645551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147085+0000","last_change":"2024-04-17T07:13:57.197244+0000","last_active":"2024-04-17T07:14:00.147085+0000","last_peered":"2024-04-17T07:14:00.147085+0000","last_clean":"2024-04-17T07:14:00.147085+0000","last_became_active":"2024-04-17T07:13:57.196651+0000","last_became_peered":"2024-04-17T07:13:57.196651+0000","last_unstale":"2024-04-17T07:14:00.147085+0000","last_undegraded":"2024-04-17T07:14:00.147085+0000","last_fullsized":"2024-04-17T07:14:00.147085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146961+0000","last_change":"2024-04-17T07:13:58.221178+0000","last_active":"2024-04-17T07:14:00.146961+0000","last_peered":"2024-04-17T07:14:00.146961+0000","last_clean":"2024-04-17T07:14:00.146961+0000","last_became_active":"2024-04-17T07:13:58.220895+0000","last_became_peered":"2024-04-17T07:13:58.220895+0000","last_unstale":"2024-04-17T07:14:00.146961+0000","last_undegraded":"2024-04-17T07:14:00.146961+0000","last_fullsized":"2024-04-17T07:14:00.146961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149303+0000","last_change":"2024-04-17T07:13:57.193073+0000","last_active":"2024-04-17T07:14:00.149303+0000","last_peered":"2024-04-17T07:14:00.149303+0000","last_clean":"2024-04-17T07:14:00.149303+0000","last_became_active":"2024-04-17T07:13:57.192203+0000","last_became_peered":"2024-04-17T07:13:57.192203+0000","last_unstale":"2024-04-17T07:14:00.149303+0000","last_undegraded":"2024-04-17T07:14:00.149303+0000","last_fullsized":"2024-04-17T07:14:00.149303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:24.305808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":669784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":78960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":669840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:14:08.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:14:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.671+0000 7f9f69c00640 1 Processor -- start 2024-04-17T07:14:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.671+0000 7f9f69c00640 1 -- start start 2024-04-17T07:14:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.674+0000 7f9f69c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 0x7f9f640cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.674+0000 7f9f69c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f64160280 0x7f9f6406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.674+0000 7f9f69c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f640cd4a0 con 0x7f9f64141ed0 2024-04-17T07:14:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.674+0000 7f9f69c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f6415ce30 con 0x7f9f64160280 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.674+0000 7f9f69c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9f641607f0 con 0x7f9f6406b8c0 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f64160280 0x7f9f6406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f64160280 0x7f9f6406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42210/0 (socket says 172.21.15.67:42210) 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 -- 172.21.15.67:0/1657601524 learned_addr learned my addr 172.21.15.67:0/1657601524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63e00640 1 --2- 172.21.15.67:0/1657601524 >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 0x7f9f640cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 -- 172.21.15.67:0/1657601524 >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 msgr2=0x7f9f640cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 --2- 172.21.15.67:0/1657601524 >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 0x7f9f640cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 -- 172.21.15.67:0/1657601524 >> v1:172.21.15.187:6789/0 conn(0x7f9f64141ed0 legacy=0x7f9f6415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63400640 1 -- 172.21.15.67:0/1657601524 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f64160ad0 con 0x7f9f64160280 2024-04-17T07:14:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.675+0000 7f9f63e00640 1 --2- 172.21.15.67:0/1657601524 >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 0x7f9f640cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.676+0000 7f9f63400640 1 --2- 172.21.15.67:0/1657601524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6406bca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f640b3210 tx=0x7f9f4c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efb11bfeea8c2ee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.676+0000 7f9f62a00640 1 -- 172.21.15.67:0/1657601524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c035020 con 0x7f9f64160280 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f62a00640 1 -- 172.21.15.67:0/1657601524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4c031890 con 0x7f9f64160280 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f62a00640 1 -- 172.21.15.67:0/1657601524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c031c60 con 0x7f9f64160280 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 -- 172.21.15.67:0/1657601524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 msgr2=0x7f9f6406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 --2- 172.21.15.67:0/1657601524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6406bca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f640b3210 tx=0x7f9f4c02b930 comp rx=0 tx=0).stop 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 -- 172.21.15.67:0/1657601524 shutdown_connections 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 --2- 172.21.15.67:0/1657601524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6406bca0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 --2- 172.21.15.67:0/1657601524 >> v2:172.21.15.187:3300/0 conn(0x7f9f6406b8c0 0x7f9f640cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 -- 172.21.15.67:0/1657601524 >> 172.21.15.67:0/1657601524 conn(0x7f9f64067d10 msgr2=0x7f9f640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 -- 172.21.15.67:0/1657601524 shutdown_connections 2024-04-17T07:14:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.677+0000 7f9f69c00640 1 -- 172.21.15.67:0/1657601524 wait complete. 2024-04-17T07:14:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.680+0000 7f9f69c00640 1 Processor -- start 2024-04-17T07:14:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.680+0000 7f9f69c00640 1 -- start start 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 0x7f9f64142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f64141ed0 0x7f9f6413c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6406c1e0 con 0x7f9f64160280 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640cddf0 con 0x7f9f6406b8c0 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.681+0000 7f9f69c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f64161090 con 0x7f9f64141ed0 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42216/0 (socket says 172.21.15.67:42216) 2024-04-17T07:14:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 -- 172.21.15.67:0/218586912 learned_addr learned my addr 172.21.15.67:0/218586912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f63e00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 0x7f9f64142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f63400640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f64141ed0 0x7f9f6413c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 -- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f64141ed0 msgr2=0x7f9f6413c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f64141ed0 0x7f9f6413c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 -- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 msgr2=0x7f9f64142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 0x7f9f64142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 -- 172.21.15.67:0/218586912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f640b49f0 con 0x7f9f64160280 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f63e00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 0x7f9f64142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.682+0000 7f9f69200640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00c780 tx=0x7f9f5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.683+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c004b60 con 0x7f9f64160280 2024-04-17T07:14:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.683+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f5c004d00 con 0x7f9f64160280 2024-04-17T07:14:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.683+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c00aa90 con 0x7f9f64160280 2024-04-17T07:14:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.683+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f640b4cd0 con 0x7f9f64160280 2024-04-17T07:14:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.683+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6415e510 con 0x7f9f64160280 2024-04-17T07:14:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.684+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f640692b0 con 0x7f9f64160280 2024-04-17T07:14:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.685+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f5c00ac30 con 0x7f9f64160280 2024-04-17T07:14:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.686+0000 7f9f5be00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 0x7f9f4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.686+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9f5c08be90 con 0x7f9f64160280 2024-04-17T07:14:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.686+0000 7f9f63e00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 0x7f9f4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.687+0000 7f9f63e00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 0x7f9f4006edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9f6415e990 tx=0x7f9f54008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.692+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9f640692b0 con 0x7f9f64160280 2024-04-17T07:14:08.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.886+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f641435a0 con 0x7f9f4006c8f0 2024-04-17T07:14:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.906+0000 7f9f5be00640 1 -- 172.21.15.67:0/218586912 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7f9f54008630 con 0x7f9f4006c8f0 2024-04-17T07:14:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:14:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 msgr2=0x7f9f4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 0x7f9f4006edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9f6415e990 tx=0x7f9f54008040 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 msgr2=0x7f9f6413bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00c780 tx=0x7f9f5c00cc40 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 shutdown_connections 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f64160280 0x7f9f6413bb70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f9f4006c8f0 0x7f9f4006edb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f64141ed0 0x7f9f6413c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 --2- 172.21.15.67:0/218586912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f6406b8c0 0x7f9f64142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 >> 172.21.15.67:0/218586912 conn(0x7f9f64067d10 msgr2=0x7f9f641408f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 shutdown_connections 2024-04-17T07:14:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:08.912+0000 7f9f69c00640 1 -- 172.21.15.67:0/218586912 wait complete. 2024-04-17T07:14:08.930 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-17T07:14:07.356373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223332,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751388,"statfs":{"total":773094113280,"available":772865421312,"internally_reserved":0,"allocated":3055616,"data_stored":1816228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.169515"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228829+0000","last_change":"2024-04-17T07:13:58.227207+0000","last_active":"2024-04-17T07:13:59.228829+0000","last_peered":"2024-04-17T07:13:59.228829+0000","last_clean":"2024-04-17T07:13:59.228829+0000","last_became_active":"2024-04-17T07:13:58.223412+0000","last_became_peered":"2024-04-17T07:13:58.223412+0000","last_unstale":"2024-04-17T07:13:59.228829+0000","last_undegraded":"2024-04-17T07:13:59.228829+0000","last_fullsized":"2024-04-17T07:13:59.228829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217686+0000","last_change":"2024-04-17T07:13:58.238529+0000","last_active":"2024-04-17T07:13:59.217686+0000","last_peered":"2024-04-17T07:13:59.217686+0000","last_clean":"2024-04-17T07:13:59.217686+0000","last_became_active":"2024-04-17T07:13:58.238155+0000","last_became_peered":"2024-04-17T07:13:58.238155+0000","last_unstale":"2024-04-17T07:13:59.217686+0000","last_undegraded":"2024-04-17T07:13:59.217686+0000","last_fullsized":"2024-04-17T07:13:59.217686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148756+0000","last_change":"2024-04-17T07:13:58.242232+0000","last_active":"2024-04-17T07:14:00.148756+0000","last_peered":"2024-04-17T07:14:00.148756+0000","last_clean":"2024-04-17T07:14:00.148756+0000","last_became_active":"2024-04-17T07:13:58.241891+0000","last_became_peered":"2024-04-17T07:13:58.241891+0000","last_unstale":"2024-04-17T07:14:00.148756+0000","last_undegraded":"2024-04-17T07:14:00.148756+0000","last_fullsized":"2024-04-17T07:14:00.148756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:17.659180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229570+0000","last_change":"2024-04-17T07:13:58.234270+0000","last_active":"2024-04-17T07:13:59.229570+0000","last_peered":"2024-04-17T07:13:59.229570+0000","last_clean":"2024-04-17T07:13:59.229570+0000","last_became_active":"2024-04-17T07:13:58.233725+0000","last_became_peered":"2024-04-17T07:13:58.233725+0000","last_unstale":"2024-04-17T07:13:59.229570+0000","last_undegraded":"2024-04-17T07:13:59.229570+0000","last_fullsized":"2024-04-17T07:13:59.229570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226745+0000","last_change":"2024-04-17T07:13:58.215673+0000","last_active":"2024-04-17T07:13:59.226745+0000","last_peered":"2024-04-17T07:13:59.226745+0000","last_clean":"2024-04-17T07:13:59.226745+0000","last_became_active":"2024-04-17T07:13:58.215371+0000","last_became_peered":"2024-04-17T07:13:58.215371+0000","last_unstale":"2024-04-17T07:13:59.226745+0000","last_undegraded":"2024-04-17T07:13:59.226745+0000","last_fullsized":"2024-04-17T07:13:59.226745+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228445+0000","last_change":"2024-04-17T07:13:58.240332+0000","last_active":"2024-04-17T07:13:59.228445+0000","last_peered":"2024-04-17T07:13:59.228445+0000","last_clean":"2024-04-17T07:13:59.228445+0000","last_became_active":"2024-04-17T07:13:58.239852+0000","last_became_peered":"2024-04-17T07:13:58.239852+0000","last_unstale":"2024-04-17T07:13:59.228445+0000","last_undegraded":"2024-04-17T07:13:59.228445+0000","last_fullsized":"2024-04-17T07:13:59.228445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226714+0000","last_change":"2024-04-17T07:13:58.238930+0000","last_active":"2024-04-17T07:13:59.226714+0000","last_peered":"2024-04-17T07:13:59.226714+0000","last_clean":"2024-04-17T07:13:59.226714+0000","last_became_active":"2024-04-17T07:13:58.238663+0000","last_became_peered":"2024-04-17T07:13:58.238663+0000","last_unstale":"2024-04-17T07:13:59.226714+0000","last_undegraded":"2024-04-17T07:13:59.226714+0000","last_fullsized":"2024-04-17T07:13:59.226714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144325+0000","last_change":"2024-04-17T07:13:58.227713+0000","last_active":"2024-04-17T07:14:00.144325+0000","last_peered":"2024-04-17T07:14:00.144325+0000","last_clean":"2024-04-17T07:14:00.144325+0000","last_became_active":"2024-04-17T07:13:58.227352+0000","last_became_peered":"2024-04-17T07:13:58.227352+0000","last_unstale":"2024-04-17T07:14:00.144325+0000","last_undegraded":"2024-04-17T07:14:00.144325+0000","last_fullsized":"2024-04-17T07:14:00.144325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228952+0000","last_change":"2024-04-17T07:13:58.233019+0000","last_active":"2024-04-17T07:13:59.228952+0000","last_peered":"2024-04-17T07:13:59.228952+0000","last_clean":"2024-04-17T07:13:59.228952+0000","last_became_active":"2024-04-17T07:13:58.230661+0000","last_became_peered":"2024-04-17T07:13:58.230661+0000","last_unstale":"2024-04-17T07:13:59.228952+0000","last_undegraded":"2024-04-17T07:13:59.228952+0000","last_fullsized":"2024-04-17T07:13:59.228952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:33.970758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226560+0000","last_change":"2024-04-17T07:13:58.225623+0000","last_active":"2024-04-17T07:13:59.226560+0000","last_peered":"2024-04-17T07:13:59.226560+0000","last_clean":"2024-04-17T07:13:59.226560+0000","last_became_active":"2024-04-17T07:13:58.225345+0000","last_became_peered":"2024-04-17T07:13:58.225345+0000","last_unstale":"2024-04-17T07:13:59.226560+0000","last_undegraded":"2024-04-17T07:13:59.226560+0000","last_fullsized":"2024-04-17T07:13:59.226560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226194+0000","last_change":"2024-04-17T07:13:58.226592+0000","last_active":"2024-04-17T07:13:59.226194+0000","last_peered":"2024-04-17T07:13:59.226194+0000","last_clean":"2024-04-17T07:13:59.226194+0000","last_became_active":"2024-04-17T07:13:58.226241+0000","last_became_peered":"2024-04-17T07:13:58.226241+0000","last_unstale":"2024-04-17T07:13:59.226194+0000","last_undegraded":"2024-04-17T07:13:59.226194+0000","last_fullsized":"2024-04-17T07:13:59.226194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144301+0000","last_change":"2024-04-17T07:13:58.216920+0000","last_active":"2024-04-17T07:14:00.144301+0000","last_peered":"2024-04-17T07:14:00.144301+0000","last_clean":"2024-04-17T07:14:00.144301+0000","last_became_active":"2024-04-17T07:13:58.216635+0000","last_became_peered":"2024-04-17T07:13:58.216635+0000","last_unstale":"2024-04-17T07:14:00.144301+0000","last_undegraded":"2024-04-17T07:14:00.144301+0000","last_fullsized":"2024-04-17T07:14:00.144301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:51.668433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146122+0000","last_change":"2024-04-17T07:13:58.225422+0000","last_active":"2024-04-17T07:14:00.146122+0000","last_peered":"2024-04-17T07:14:00.146122+0000","last_clean":"2024-04-17T07:14:00.146122+0000","last_became_active":"2024-04-17T07:13:58.224906+0000","last_became_peered":"2024-04-17T07:13:58.224906+0000","last_unstale":"2024-04-17T07:14:00.146122+0000","last_undegraded":"2024-04-17T07:14:00.146122+0000","last_fullsized":"2024-04-17T07:14:00.146122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147718+0000","last_change":"2024-04-17T07:13:58.233409+0000","last_active":"2024-04-17T07:14:00.147718+0000","last_peered":"2024-04-17T07:14:00.147718+0000","last_clean":"2024-04-17T07:14:00.147718+0000","last_became_active":"2024-04-17T07:13:58.233182+0000","last_became_peered":"2024-04-17T07:13:58.233182+0000","last_unstale":"2024-04-17T07:14:00.147718+0000","last_undegraded":"2024-04-17T07:14:00.147718+0000","last_fullsized":"2024-04-17T07:14:00.147718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:18.125280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217646+0000","last_change":"2024-04-17T07:13:58.215689+0000","last_active":"2024-04-17T07:13:59.217646+0000","last_peered":"2024-04-17T07:13:59.217646+0000","last_clean":"2024-04-17T07:13:59.217646+0000","last_became_active":"2024-04-17T07:13:58.215054+0000","last_became_peered":"2024-04-17T07:13:58.215054+0000","last_unstale":"2024-04-17T07:13:59.217646+0000","last_undegraded":"2024-04-17T07:13:59.217646+0000","last_fullsized":"2024-04-17T07:13:59.217646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227948+0000","last_change":"2024-04-17T07:13:58.220483+0000","last_active":"2024-04-17T07:13:59.227948+0000","last_peered":"2024-04-17T07:13:59.227948+0000","last_clean":"2024-04-17T07:13:59.227948+0000","last_became_active":"2024-04-17T07:13:58.220205+0000","last_became_peered":"2024-04-17T07:13:58.220205+0000","last_unstale":"2024-04-17T07:13:59.227948+0000","last_undegraded":"2024-04-17T07:13:59.227948+0000","last_fullsized":"2024-04-17T07:13:59.227948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231680+0000","last_change":"2024-04-17T07:13:58.223188+0000","last_active":"2024-04-17T07:13:59.231680+0000","last_peered":"2024-04-17T07:13:59.231680+0000","last_clean":"2024-04-17T07:13:59.231680+0000","last_became_active":"2024-04-17T07:13:58.221699+0000","last_became_peered":"2024-04-17T07:13:58.221699+0000","last_unstale":"2024-04-17T07:13:59.231680+0000","last_undegraded":"2024-04-17T07:13:59.231680+0000","last_fullsized":"2024-04-17T07:13:59.231680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:34.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144144+0000","last_change":"2024-04-17T07:13:58.220308+0000","last_active":"2024-04-17T07:14:00.144144+0000","last_peered":"2024-04-17T07:14:00.144144+0000","last_clean":"2024-04-17T07:14:00.144144+0000","last_became_active":"2024-04-17T07:13:58.219976+0000","last_became_peered":"2024-04-17T07:13:58.219976+0000","last_unstale":"2024-04-17T07:14:00.144144+0000","last_undegraded":"2024-04-17T07:14:00.144144+0000","last_fullsized":"2024-04-17T07:14:00.144144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:11.629034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228541+0000","last_change":"2024-04-17T07:13:58.223620+0000","last_active":"2024-04-17T07:13:59.228541+0000","last_peered":"2024-04-17T07:13:59.228541+0000","last_clean":"2024-04-17T07:13:59.228541+0000","last_became_active":"2024-04-17T07:13:58.222125+0000","last_became_peered":"2024-04-17T07:13:58.222125+0000","last_unstale":"2024-04-17T07:13:59.228541+0000","last_undegraded":"2024-04-17T07:13:59.228541+0000","last_fullsized":"2024-04-17T07:13:59.228541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227555+0000","last_change":"2024-04-17T07:13:58.225907+0000","last_active":"2024-04-17T07:13:59.227555+0000","last_peered":"2024-04-17T07:13:59.227555+0000","last_clean":"2024-04-17T07:13:59.227555+0000","last_became_active":"2024-04-17T07:13:58.224993+0000","last_became_peered":"2024-04-17T07:13:58.224993+0000","last_unstale":"2024-04-17T07:13:59.227555+0000","last_undegraded":"2024-04-17T07:13:59.227555+0000","last_fullsized":"2024-04-17T07:13:59.227555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148489+0000","last_change":"2024-04-17T07:13:58.236523+0000","last_active":"2024-04-17T07:14:00.148489+0000","last_peered":"2024-04-17T07:14:00.148489+0000","last_clean":"2024-04-17T07:14:00.148489+0000","last_became_active":"2024-04-17T07:13:58.236164+0000","last_became_peered":"2024-04-17T07:13:58.236164+0000","last_unstale":"2024-04-17T07:14:00.148489+0000","last_undegraded":"2024-04-17T07:14:00.148489+0000","last_fullsized":"2024-04-17T07:14:00.148489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:33.648495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228822+0000","last_change":"2024-04-17T07:13:58.222632+0000","last_active":"2024-04-17T07:13:59.228822+0000","last_peered":"2024-04-17T07:13:59.228822+0000","last_clean":"2024-04-17T07:13:59.228822+0000","last_became_active":"2024-04-17T07:13:58.221666+0000","last_became_peered":"2024-04-17T07:13:58.221666+0000","last_unstale":"2024-04-17T07:13:59.228822+0000","last_undegraded":"2024-04-17T07:13:59.228822+0000","last_fullsized":"2024-04-17T07:13:59.228822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145740+0000","last_change":"2024-04-17T07:13:58.218642+0000","last_active":"2024-04-17T07:14:00.145740+0000","last_peered":"2024-04-17T07:14:00.145740+0000","last_clean":"2024-04-17T07:14:00.145740+0000","last_became_active":"2024-04-17T07:13:58.218235+0000","last_became_peered":"2024-04-17T07:13:58.218235+0000","last_unstale":"2024-04-17T07:14:00.145740+0000","last_undegraded":"2024-04-17T07:14:00.145740+0000","last_fullsized":"2024-04-17T07:14:00.145740+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228851+0000","last_change":"2024-04-17T07:13:58.232074+0000","last_active":"2024-04-17T07:13:59.228851+0000","last_peered":"2024-04-17T07:13:59.228851+0000","last_clean":"2024-04-17T07:13:59.228851+0000","last_became_active":"2024-04-17T07:13:58.231716+0000","last_became_peered":"2024-04-17T07:13:58.231716+0000","last_unstale":"2024-04-17T07:13:59.228851+0000","last_undegraded":"2024-04-17T07:13:59.228851+0000","last_fullsized":"2024-04-17T07:13:59.228851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147203+0000","last_change":"2024-04-17T07:13:58.230039+0000","last_active":"2024-04-17T07:14:00.147203+0000","last_peered":"2024-04-17T07:14:00.147203+0000","last_clean":"2024-04-17T07:14:00.147203+0000","last_became_active":"2024-04-17T07:13:58.224374+0000","last_became_peered":"2024-04-17T07:13:58.224374+0000","last_unstale":"2024-04-17T07:14:00.147203+0000","last_undegraded":"2024-04-17T07:14:00.147203+0000","last_fullsized":"2024-04-17T07:14:00.147203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226940+0000","last_change":"2024-04-17T07:13:58.227485+0000","last_active":"2024-04-17T07:13:59.226940+0000","last_peered":"2024-04-17T07:13:59.226940+0000","last_clean":"2024-04-17T07:13:59.226940+0000","last_became_active":"2024-04-17T07:13:58.227182+0000","last_became_peered":"2024-04-17T07:13:58.227182+0000","last_unstale":"2024-04-17T07:13:59.226940+0000","last_undegraded":"2024-04-17T07:13:59.226940+0000","last_fullsized":"2024-04-17T07:13:59.226940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:09.808127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217943+0000","last_change":"2024-04-17T07:13:58.222243+0000","last_active":"2024-04-17T07:13:59.217943+0000","last_peered":"2024-04-17T07:13:59.217943+0000","last_clean":"2024-04-17T07:13:59.217943+0000","last_became_active":"2024-04-17T07:13:58.221925+0000","last_became_peered":"2024-04-17T07:13:58.221925+0000","last_unstale":"2024-04-17T07:13:59.217943+0000","last_undegraded":"2024-04-17T07:13:59.217943+0000","last_fullsized":"2024-04-17T07:13:59.217943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146781+0000","last_change":"2024-04-17T07:13:58.226680+0000","last_active":"2024-04-17T07:14:00.146781+0000","last_peered":"2024-04-17T07:14:00.146781+0000","last_clean":"2024-04-17T07:14:00.146781+0000","last_became_active":"2024-04-17T07:13:58.226283+0000","last_became_peered":"2024-04-17T07:13:58.226283+0000","last_unstale":"2024-04-17T07:14:00.146781+0000","last_undegraded":"2024-04-17T07:14:00.146781+0000","last_fullsized":"2024-04-17T07:14:00.146781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148779+0000","last_change":"2024-04-17T07:13:58.242478+0000","last_active":"2024-04-17T07:14:00.148779+0000","last_peered":"2024-04-17T07:14:00.148779+0000","last_clean":"2024-04-17T07:14:00.148779+0000","last_became_active":"2024-04-17T07:13:58.241968+0000","last_became_peered":"2024-04-17T07:13:58.241968+0000","last_unstale":"2024-04-17T07:14:00.148779+0000","last_undegraded":"2024-04-17T07:14:00.148779+0000","last_fullsized":"2024-04-17T07:14:00.148779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:29:46.240069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229245+0000","last_change":"2024-04-17T07:13:58.230046+0000","last_active":"2024-04-17T07:13:59.229245+0000","last_peered":"2024-04-17T07:13:59.229245+0000","last_clean":"2024-04-17T07:13:59.229245+0000","last_became_active":"2024-04-17T07:13:58.229744+0000","last_became_peered":"2024-04-17T07:13:58.229744+0000","last_unstale":"2024-04-17T07:13:59.229245+0000","last_undegraded":"2024-04-17T07:13:59.229245+0000","last_fullsized":"2024-04-17T07:13:59.229245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:24.313355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143075+0000","last_change":"2024-04-17T07:13:58.219601+0000","last_active":"2024-04-17T07:14:00.143075+0000","last_peered":"2024-04-17T07:14:00.143075+0000","last_clean":"2024-04-17T07:14:00.143075+0000","last_became_active":"2024-04-17T07:13:58.219208+0000","last_became_peered":"2024-04-17T07:13:58.219208+0000","last_unstale":"2024-04-17T07:14:00.143075+0000","last_undegraded":"2024-04-17T07:14:00.143075+0000","last_fullsized":"2024-04-17T07:14:00.143075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:06.342021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225081+0000","last_change":"2024-04-17T07:13:58.230063+0000","last_active":"2024-04-17T07:13:59.225081+0000","last_peered":"2024-04-17T07:13:59.225081+0000","last_clean":"2024-04-17T07:13:59.225081+0000","last_became_active":"2024-04-17T07:13:58.229129+0000","last_became_peered":"2024-04-17T07:13:58.229129+0000","last_unstale":"2024-04-17T07:13:59.225081+0000","last_undegraded":"2024-04-17T07:13:59.225081+0000","last_fullsized":"2024-04-17T07:13:59.225081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:59.849640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143839+0000","last_change":"2024-04-17T07:13:58.208959+0000","last_active":"2024-04-17T07:14:00.143839+0000","last_peered":"2024-04-17T07:14:00.143839+0000","last_clean":"2024-04-17T07:14:00.143839+0000","last_became_active":"2024-04-17T07:13:58.208200+0000","last_became_peered":"2024-04-17T07:13:58.208200+0000","last_unstale":"2024-04-17T07:14:00.143839+0000","last_undegraded":"2024-04-17T07:14:00.143839+0000","last_fullsized":"2024-04-17T07:14:00.143839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:14.893023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147150+0000","last_change":"2024-04-17T07:13:57.193193+0000","last_active":"2024-04-17T07:14:00.147150+0000","last_peered":"2024-04-17T07:14:00.147150+0000","last_clean":"2024-04-17T07:14:00.147150+0000","last_became_active":"2024-04-17T07:13:57.192931+0000","last_became_peered":"2024-04-17T07:13:57.192931+0000","last_unstale":"2024-04-17T07:14:00.147150+0000","last_undegraded":"2024-04-17T07:14:00.147150+0000","last_fullsized":"2024-04-17T07:14:00.147150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148893+0000","last_change":"2024-04-17T07:13:58.244126+0000","last_active":"2024-04-17T07:14:00.148893+0000","last_peered":"2024-04-17T07:14:00.148893+0000","last_clean":"2024-04-17T07:14:00.148893+0000","last_became_active":"2024-04-17T07:13:58.243846+0000","last_became_peered":"2024-04-17T07:13:58.243846+0000","last_unstale":"2024-04-17T07:14:00.148893+0000","last_undegraded":"2024-04-17T07:14:00.148893+0000","last_fullsized":"2024-04-17T07:14:00.148893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:20.038754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225363+0000","last_change":"2024-04-17T07:13:57.198746+0000","last_active":"2024-04-17T07:13:59.225363+0000","last_peered":"2024-04-17T07:13:59.225363+0000","last_clean":"2024-04-17T07:13:59.225363+0000","last_became_active":"2024-04-17T07:13:57.195870+0000","last_became_peered":"2024-04-17T07:13:57.195870+0000","last_unstale":"2024-04-17T07:13:59.225363+0000","last_undegraded":"2024-04-17T07:13:59.225363+0000","last_fullsized":"2024-04-17T07:13:59.225363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:44.168192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.145653+0000","last_change":"2024-04-17T07:13:57.201813+0000","last_active":"2024-04-17T07:14:00.145653+0000","last_peered":"2024-04-17T07:14:00.145653+0000","last_clean":"2024-04-17T07:14:00.145653+0000","last_became_active":"2024-04-17T07:13:57.201530+0000","last_became_peered":"2024-04-17T07:13:57.201530+0000","last_unstale":"2024-04-17T07:14:00.145653+0000","last_undegraded":"2024-04-17T07:14:00.145653+0000","last_fullsized":"2024-04-17T07:14:00.145653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144391+0000","last_change":"2024-04-17T07:13:58.235592+0000","last_active":"2024-04-17T07:14:00.144391+0000","last_peered":"2024-04-17T07:14:00.144391+0000","last_clean":"2024-04-17T07:14:00.144391+0000","last_became_active":"2024-04-17T07:13:58.235287+0000","last_became_peered":"2024-04-17T07:13:58.235287+0000","last_unstale":"2024-04-17T07:14:00.144391+0000","last_undegraded":"2024-04-17T07:14:00.144391+0000","last_fullsized":"2024-04-17T07:14:00.144391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228264+0000","last_change":"2024-04-17T07:13:57.183240+0000","last_active":"2024-04-17T07:13:59.228264+0000","last_peered":"2024-04-17T07:13:59.228264+0000","last_clean":"2024-04-17T07:13:59.228264+0000","last_became_active":"2024-04-17T07:13:57.182859+0000","last_became_peered":"2024-04-17T07:13:57.182859+0000","last_unstale":"2024-04-17T07:13:59.228264+0000","last_undegraded":"2024-04-17T07:13:59.228264+0000","last_fullsized":"2024-04-17T07:13:59.228264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217429+0000","last_change":"2024-04-17T07:13:58.221113+0000","last_active":"2024-04-17T07:13:59.217429+0000","last_peered":"2024-04-17T07:13:59.217429+0000","last_clean":"2024-04-17T07:13:59.217429+0000","last_became_active":"2024-04-17T07:13:58.220807+0000","last_became_peered":"2024-04-17T07:13:58.220807+0000","last_unstale":"2024-04-17T07:13:59.217429+0000","last_undegraded":"2024-04-17T07:13:59.217429+0000","last_fullsized":"2024-04-17T07:13:59.217429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226316+0000","last_change":"2024-04-17T07:13:57.197641+0000","last_active":"2024-04-17T07:13:59.226316+0000","last_peered":"2024-04-17T07:13:59.226316+0000","last_clean":"2024-04-17T07:13:59.226316+0000","last_became_active":"2024-04-17T07:13:57.197394+0000","last_became_peered":"2024-04-17T07:13:57.197394+0000","last_unstale":"2024-04-17T07:13:59.226316+0000","last_undegraded":"2024-04-17T07:13:59.226316+0000","last_fullsized":"2024-04-17T07:13:59.226316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231265+0000","last_change":"2024-04-17T07:13:58.225914+0000","last_active":"2024-04-17T07:13:59.231265+0000","last_peered":"2024-04-17T07:13:59.231265+0000","last_clean":"2024-04-17T07:13:59.231265+0000","last_became_active":"2024-04-17T07:13:58.225649+0000","last_became_peered":"2024-04-17T07:13:58.225649+0000","last_unstale":"2024-04-17T07:13:59.231265+0000","last_undegraded":"2024-04-17T07:13:59.231265+0000","last_fullsized":"2024-04-17T07:13:59.231265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231330+0000","last_change":"2024-04-17T07:13:57.198101+0000","last_active":"2024-04-17T07:13:59.231330+0000","last_peered":"2024-04-17T07:13:59.231330+0000","last_clean":"2024-04-17T07:13:59.231330+0000","last_became_active":"2024-04-17T07:13:57.196771+0000","last_became_peered":"2024-04-17T07:13:57.196771+0000","last_unstale":"2024-04-17T07:13:59.231330+0000","last_undegraded":"2024-04-17T07:13:59.231330+0000","last_fullsized":"2024-04-17T07:13:59.231330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149355+0000","last_change":"2024-04-17T07:13:58.244824+0000","last_active":"2024-04-17T07:14:00.149355+0000","last_peered":"2024-04-17T07:14:00.149355+0000","last_clean":"2024-04-17T07:14:00.149355+0000","last_became_active":"2024-04-17T07:13:58.243368+0000","last_became_peered":"2024-04-17T07:13:58.243368+0000","last_unstale":"2024-04-17T07:14:00.149355+0000","last_undegraded":"2024-04-17T07:14:00.149355+0000","last_fullsized":"2024-04-17T07:14:00.149355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:43.836722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218062+0000","last_change":"2024-04-17T07:13:57.198519+0000","last_active":"2024-04-17T07:13:59.218062+0000","last_peered":"2024-04-17T07:13:59.218062+0000","last_clean":"2024-04-17T07:13:59.218062+0000","last_became_active":"2024-04-17T07:13:57.197225+0000","last_became_peered":"2024-04-17T07:13:57.197225+0000","last_unstale":"2024-04-17T07:13:59.218062+0000","last_undegraded":"2024-04-17T07:13:59.218062+0000","last_fullsized":"2024-04-17T07:13:59.218062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144417+0000","last_change":"2024-04-17T07:13:58.221225+0000","last_active":"2024-04-17T07:14:00.144417+0000","last_peered":"2024-04-17T07:14:00.144417+0000","last_clean":"2024-04-17T07:14:00.144417+0000","last_became_active":"2024-04-17T07:13:58.220917+0000","last_became_peered":"2024-04-17T07:13:58.220917+0000","last_unstale":"2024-04-17T07:14:00.144417+0000","last_undegraded":"2024-04-17T07:14:00.144417+0000","last_fullsized":"2024-04-17T07:14:00.144417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:04.188468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226310+0000","last_change":"2024-04-17T07:13:57.200719+0000","last_active":"2024-04-17T07:13:59.226310+0000","last_peered":"2024-04-17T07:13:59.226310+0000","last_clean":"2024-04-17T07:13:59.226310+0000","last_became_active":"2024-04-17T07:13:57.200404+0000","last_became_peered":"2024-04-17T07:13:57.200404+0000","last_unstale":"2024-04-17T07:13:59.226310+0000","last_undegraded":"2024-04-17T07:13:59.226310+0000","last_fullsized":"2024-04-17T07:13:59.226310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:39.677948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226654+0000","last_change":"2024-04-17T07:13:58.231037+0000","last_active":"2024-04-17T07:13:59.226654+0000","last_peered":"2024-04-17T07:13:59.226654+0000","last_clean":"2024-04-17T07:13:59.226654+0000","last_became_active":"2024-04-17T07:13:58.229754+0000","last_became_peered":"2024-04-17T07:13:58.229754+0000","last_unstale":"2024-04-17T07:13:59.226654+0000","last_undegraded":"2024-04-17T07:13:59.226654+0000","last_fullsized":"2024-04-17T07:13:59.226654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149082+0000","last_change":"2024-04-17T07:13:57.196583+0000","last_active":"2024-04-17T07:14:00.149082+0000","last_peered":"2024-04-17T07:14:00.149082+0000","last_clean":"2024-04-17T07:14:00.149082+0000","last_became_active":"2024-04-17T07:13:57.195074+0000","last_became_peered":"2024-04-17T07:13:57.195074+0000","last_unstale":"2024-04-17T07:14:00.149082+0000","last_undegraded":"2024-04-17T07:14:00.149082+0000","last_fullsized":"2024-04-17T07:14:00.149082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:53.288607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148521+0000","last_change":"2024-04-17T07:13:58.243294+0000","last_active":"2024-04-17T07:14:00.148521+0000","last_peered":"2024-04-17T07:14:00.148521+0000","last_clean":"2024-04-17T07:14:00.148521+0000","last_became_active":"2024-04-17T07:13:58.241931+0000","last_became_peered":"2024-04-17T07:13:58.241931+0000","last_unstale":"2024-04-17T07:14:00.148521+0000","last_undegraded":"2024-04-17T07:14:00.148521+0000","last_fullsized":"2024-04-17T07:14:00.148521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:49.353882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229051+0000","last_change":"2024-04-17T07:13:57.190224+0000","last_active":"2024-04-17T07:13:59.229051+0000","last_peered":"2024-04-17T07:13:59.229051+0000","last_clean":"2024-04-17T07:13:59.229051+0000","last_became_active":"2024-04-17T07:13:57.189879+0000","last_became_peered":"2024-04-17T07:13:57.189879+0000","last_unstale":"2024-04-17T07:13:59.229051+0000","last_undegraded":"2024-04-17T07:13:59.229051+0000","last_fullsized":"2024-04-17T07:13:59.229051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226963+0000","last_change":"2024-04-17T07:13:58.235859+0000","last_active":"2024-04-17T07:13:59.226963+0000","last_peered":"2024-04-17T07:13:59.226963+0000","last_clean":"2024-04-17T07:13:59.226963+0000","last_became_active":"2024-04-17T07:13:58.235565+0000","last_became_peered":"2024-04-17T07:13:58.235565+0000","last_unstale":"2024-04-17T07:13:59.226963+0000","last_undegraded":"2024-04-17T07:13:59.226963+0000","last_fullsized":"2024-04-17T07:13:59.226963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226241+0000","last_change":"2024-04-17T07:13:57.202465+0000","last_active":"2024-04-17T07:13:59.226241+0000","last_peered":"2024-04-17T07:13:59.226241+0000","last_clean":"2024-04-17T07:13:59.226241+0000","last_became_active":"2024-04-17T07:13:57.202105+0000","last_became_peered":"2024-04-17T07:13:57.202105+0000","last_unstale":"2024-04-17T07:13:59.226241+0000","last_undegraded":"2024-04-17T07:13:59.226241+0000","last_fullsized":"2024-04-17T07:13:59.226241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225347+0000","last_change":"2024-04-17T07:13:58.228208+0000","last_active":"2024-04-17T07:13:59.225347+0000","last_peered":"2024-04-17T07:13:59.225347+0000","last_clean":"2024-04-17T07:13:59.225347+0000","last_became_active":"2024-04-17T07:13:58.227889+0000","last_became_peered":"2024-04-17T07:13:58.227889+0000","last_unstale":"2024-04-17T07:13:59.225347+0000","last_undegraded":"2024-04-17T07:13:59.225347+0000","last_fullsized":"2024-04-17T07:13:59.225347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:28.651988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.142947+0000","last_change":"2024-04-17T07:13:57.204927+0000","last_active":"2024-04-17T07:14:00.142947+0000","last_peered":"2024-04-17T07:14:00.142947+0000","last_clean":"2024-04-17T07:14:00.142947+0000","last_became_active":"2024-04-17T07:13:57.204546+0000","last_became_peered":"2024-04-17T07:13:57.204546+0000","last_unstale":"2024-04-17T07:14:00.142947+0000","last_undegraded":"2024-04-17T07:14:00.142947+0000","last_fullsized":"2024-04-17T07:14:00.142947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:22:01.161179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227143+0000","last_change":"2024-04-17T07:13:58.235736+0000","last_active":"2024-04-17T07:13:59.227143+0000","last_peered":"2024-04-17T07:13:59.227143+0000","last_clean":"2024-04-17T07:13:59.227143+0000","last_became_active":"2024-04-17T07:13:58.235432+0000","last_became_peered":"2024-04-17T07:13:58.235432+0000","last_unstale":"2024-04-17T07:13:59.227143+0000","last_undegraded":"2024-04-17T07:13:59.227143+0000","last_fullsized":"2024-04-17T07:13:59.227143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:14.317722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144456+0000","last_change":"2024-04-17T07:13:57.195110+0000","last_active":"2024-04-17T07:14:00.144456+0000","last_peered":"2024-04-17T07:14:00.144456+0000","last_clean":"2024-04-17T07:14:00.144456+0000","last_became_active":"2024-04-17T07:13:57.194788+0000","last_became_peered":"2024-04-17T07:13:57.194788+0000","last_unstale":"2024-04-17T07:14:00.144456+0000","last_undegraded":"2024-04-17T07:14:00.144456+0000","last_fullsized":"2024-04-17T07:14:00.144456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:01.392379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228617+0000","last_change":"2024-04-17T07:13:58.224809+0000","last_active":"2024-04-17T07:13:59.228617+0000","last_peered":"2024-04-17T07:13:59.228617+0000","last_clean":"2024-04-17T07:13:59.228617+0000","last_became_active":"2024-04-17T07:13:58.224516+0000","last_became_peered":"2024-04-17T07:13:58.224516+0000","last_unstale":"2024-04-17T07:13:59.228617+0000","last_undegraded":"2024-04-17T07:13:59.228617+0000","last_fullsized":"2024-04-17T07:13:59.228617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144137+0000","last_change":"2024-04-17T07:13:57.203108+0000","last_active":"2024-04-17T07:14:00.144137+0000","last_peered":"2024-04-17T07:14:00.144137+0000","last_clean":"2024-04-17T07:14:00.144137+0000","last_became_active":"2024-04-17T07:13:57.202802+0000","last_became_peered":"2024-04-17T07:13:57.202802+0000","last_unstale":"2024-04-17T07:14:00.144137+0000","last_undegraded":"2024-04-17T07:14:00.144137+0000","last_fullsized":"2024-04-17T07:14:00.144137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:54.647968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229091+0000","last_change":"2024-04-17T07:13:58.230644+0000","last_active":"2024-04-17T07:13:59.229091+0000","last_peered":"2024-04-17T07:13:59.229091+0000","last_clean":"2024-04-17T07:13:59.229091+0000","last_became_active":"2024-04-17T07:13:58.229593+0000","last_became_peered":"2024-04-17T07:13:58.229593+0000","last_unstale":"2024-04-17T07:13:59.229091+0000","last_undegraded":"2024-04-17T07:13:59.229091+0000","last_fullsized":"2024-04-17T07:13:59.229091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:57.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.218173+0000","last_change":"2024-04-17T07:13:57.200619+0000","last_active":"2024-04-17T07:13:59.218173+0000","last_peered":"2024-04-17T07:13:59.218173+0000","last_clean":"2024-04-17T07:13:59.218173+0000","last_became_active":"2024-04-17T07:13:57.200293+0000","last_became_peered":"2024-04-17T07:13:57.200293+0000","last_unstale":"2024-04-17T07:13:59.218173+0000","last_undegraded":"2024-04-17T07:13:59.218173+0000","last_fullsized":"2024-04-17T07:13:59.218173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229770+0000","last_change":"2024-04-17T07:13:58.233982+0000","last_active":"2024-04-17T07:13:59.229770+0000","last_peered":"2024-04-17T07:13:59.229770+0000","last_clean":"2024-04-17T07:13:59.229770+0000","last_became_active":"2024-04-17T07:13:58.233394+0000","last_became_peered":"2024-04-17T07:13:58.233394+0000","last_unstale":"2024-04-17T07:13:59.229770+0000","last_undegraded":"2024-04-17T07:13:59.229770+0000","last_fullsized":"2024-04-17T07:13:59.229770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:23.962161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143798+0000","last_change":"2024-04-17T07:13:57.198237+0000","last_active":"2024-04-17T07:14:00.143798+0000","last_peered":"2024-04-17T07:14:00.143798+0000","last_clean":"2024-04-17T07:14:00.143798+0000","last_became_active":"2024-04-17T07:13:57.197870+0000","last_became_peered":"2024-04-17T07:13:57.197870+0000","last_unstale":"2024-04-17T07:14:00.143798+0000","last_undegraded":"2024-04-17T07:14:00.143798+0000","last_fullsized":"2024-04-17T07:14:00.143798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:48.074499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227126+0000","last_change":"2024-04-17T07:13:58.232896+0000","last_active":"2024-04-17T07:13:59.227126+0000","last_peered":"2024-04-17T07:13:59.227126+0000","last_clean":"2024-04-17T07:13:59.227126+0000","last_became_active":"2024-04-17T07:13:58.232155+0000","last_became_peered":"2024-04-17T07:13:58.232155+0000","last_unstale":"2024-04-17T07:13:59.227126+0000","last_undegraded":"2024-04-17T07:13:59.227126+0000","last_fullsized":"2024-04-17T07:13:59.227126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225865+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:13:59.225865+0000","last_peered":"2024-04-17T07:13:59.225865+0000","last_clean":"2024-04-17T07:13:59.225865+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:13:59.225865+0000","last_undegraded":"2024-04-17T07:13:59.225865+0000","last_fullsized":"2024-04-17T07:13:59.225865+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228095+0000","last_change":"2024-04-17T07:13:57.187149+0000","last_active":"2024-04-17T07:13:59.228095+0000","last_peered":"2024-04-17T07:13:59.228095+0000","last_clean":"2024-04-17T07:13:59.228095+0000","last_became_active":"2024-04-17T07:13:57.186832+0000","last_became_peered":"2024-04-17T07:13:57.186832+0000","last_unstale":"2024-04-17T07:13:59.228095+0000","last_undegraded":"2024-04-17T07:13:59.228095+0000","last_fullsized":"2024-04-17T07:13:59.228095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146188+0000","last_change":"2024-04-17T07:13:58.228567+0000","last_active":"2024-04-17T07:14:00.146188+0000","last_peered":"2024-04-17T07:14:00.146188+0000","last_clean":"2024-04-17T07:14:00.146188+0000","last_became_active":"2024-04-17T07:13:58.228064+0000","last_became_peered":"2024-04-17T07:13:58.228064+0000","last_unstale":"2024-04-17T07:14:00.146188+0000","last_undegraded":"2024-04-17T07:14:00.146188+0000","last_fullsized":"2024-04-17T07:14:00.146188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:43.464852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224678+0000","last_change":"2024-04-17T07:13:57.204847+0000","last_active":"2024-04-17T07:13:59.224678+0000","last_peered":"2024-04-17T07:13:59.224678+0000","last_clean":"2024-04-17T07:13:59.224678+0000","last_became_active":"2024-04-17T07:13:57.204534+0000","last_became_peered":"2024-04-17T07:13:57.204534+0000","last_unstale":"2024-04-17T07:13:59.224678+0000","last_undegraded":"2024-04-17T07:13:59.224678+0000","last_fullsized":"2024-04-17T07:13:59.224678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:38.690252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148252+0000","last_change":"2024-04-17T07:13:58.230682+0000","last_active":"2024-04-17T07:14:00.148252+0000","last_peered":"2024-04-17T07:14:00.148252+0000","last_clean":"2024-04-17T07:14:00.148252+0000","last_became_active":"2024-04-17T07:13:58.230315+0000","last_became_peered":"2024-04-17T07:13:58.230315+0000","last_unstale":"2024-04-17T07:14:00.148252+0000","last_undegraded":"2024-04-17T07:14:00.148252+0000","last_fullsized":"2024-04-17T07:14:00.148252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:03.539255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143397+0000","last_change":"2024-04-17T07:13:57.204481+0000","last_active":"2024-04-17T07:14:00.143397+0000","last_peered":"2024-04-17T07:14:00.143397+0000","last_clean":"2024-04-17T07:14:00.143397+0000","last_became_active":"2024-04-17T07:13:57.203848+0000","last_became_peered":"2024-04-17T07:13:57.203848+0000","last_unstale":"2024-04-17T07:14:00.143397+0000","last_undegraded":"2024-04-17T07:14:00.143397+0000","last_fullsized":"2024-04-17T07:14:00.143397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:11.312213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217520+0000","last_change":"2024-04-17T07:13:58.208486+0000","last_active":"2024-04-17T07:13:59.217520+0000","last_peered":"2024-04-17T07:13:59.217520+0000","last_clean":"2024-04-17T07:13:59.217520+0000","last_became_active":"2024-04-17T07:13:58.208043+0000","last_became_peered":"2024-04-17T07:13:58.208043+0000","last_unstale":"2024-04-17T07:13:59.217520+0000","last_undegraded":"2024-04-17T07:13:59.217520+0000","last_fullsized":"2024-04-17T07:13:59.217520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:37.915718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229248+0000","last_change":"2024-04-17T07:13:57.196519+0000","last_active":"2024-04-17T07:13:59.229248+0000","last_peered":"2024-04-17T07:13:59.229248+0000","last_clean":"2024-04-17T07:13:59.229248+0000","last_became_active":"2024-04-17T07:13:57.196148+0000","last_became_peered":"2024-04-17T07:13:57.196148+0000","last_unstale":"2024-04-17T07:13:59.229248+0000","last_undegraded":"2024-04-17T07:13:59.229248+0000","last_fullsized":"2024-04-17T07:13:59.229248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144037+0000","last_change":"2024-04-17T07:13:58.241381+0000","last_active":"2024-04-17T07:14:00.144037+0000","last_peered":"2024-04-17T07:14:00.144037+0000","last_clean":"2024-04-17T07:14:00.144037+0000","last_became_active":"2024-04-17T07:13:58.240707+0000","last_became_peered":"2024-04-17T07:13:58.240707+0000","last_unstale":"2024-04-17T07:14:00.144037+0000","last_undegraded":"2024-04-17T07:14:00.144037+0000","last_fullsized":"2024-04-17T07:14:00.144037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:39.705592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217512+0000","last_change":"2024-04-17T07:13:57.196549+0000","last_active":"2024-04-17T07:13:59.217512+0000","last_peered":"2024-04-17T07:13:59.217512+0000","last_clean":"2024-04-17T07:13:59.217512+0000","last_became_active":"2024-04-17T07:13:57.196096+0000","last_became_peered":"2024-04-17T07:13:57.196096+0000","last_unstale":"2024-04-17T07:13:59.217512+0000","last_undegraded":"2024-04-17T07:13:59.217512+0000","last_fullsized":"2024-04-17T07:13:59.217512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.784270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231381+0000","last_change":"2024-04-17T07:13:58.218704+0000","last_active":"2024-04-17T07:13:59.231381+0000","last_peered":"2024-04-17T07:13:59.231381+0000","last_clean":"2024-04-17T07:13:59.231381+0000","last_became_active":"2024-04-17T07:13:58.218383+0000","last_became_peered":"2024-04-17T07:13:58.218383+0000","last_unstale":"2024-04-17T07:13:59.231381+0000","last_undegraded":"2024-04-17T07:13:59.231381+0000","last_fullsized":"2024-04-17T07:13:59.231381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:14.310829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228625+0000","last_change":"2024-04-17T07:13:57.199856+0000","last_active":"2024-04-17T07:13:59.228625+0000","last_peered":"2024-04-17T07:13:59.228625+0000","last_clean":"2024-04-17T07:13:59.228625+0000","last_became_active":"2024-04-17T07:13:57.199550+0000","last_became_peered":"2024-04-17T07:13:57.199550+0000","last_unstale":"2024-04-17T07:13:59.228625+0000","last_undegraded":"2024-04-17T07:13:59.228625+0000","last_fullsized":"2024-04-17T07:13:59.228625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:36.294837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217992+0000","last_change":"2024-04-17T07:13:58.213558+0000","last_active":"2024-04-17T07:13:59.217992+0000","last_peered":"2024-04-17T07:13:59.217992+0000","last_clean":"2024-04-17T07:13:59.217992+0000","last_became_active":"2024-04-17T07:13:58.213237+0000","last_became_peered":"2024-04-17T07:13:58.213237+0000","last_unstale":"2024-04-17T07:13:59.217992+0000","last_undegraded":"2024-04-17T07:13:59.217992+0000","last_fullsized":"2024-04-17T07:13:59.217992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:15.796411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.222842+0000","last_change":"2024-04-17T07:13:57.198259+0000","last_active":"2024-04-17T07:13:59.222842+0000","last_peered":"2024-04-17T07:13:59.222842+0000","last_clean":"2024-04-17T07:13:59.222842+0000","last_became_active":"2024-04-17T07:13:57.197982+0000","last_became_peered":"2024-04-17T07:13:57.197982+0000","last_unstale":"2024-04-17T07:13:59.222842+0000","last_undegraded":"2024-04-17T07:13:59.222842+0000","last_fullsized":"2024-04-17T07:13:59.222842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147642+0000","last_change":"2024-04-17T07:13:58.224013+0000","last_active":"2024-04-17T07:14:00.147642+0000","last_peered":"2024-04-17T07:14:00.147642+0000","last_clean":"2024-04-17T07:14:00.147642+0000","last_became_active":"2024-04-17T07:13:58.223705+0000","last_became_peered":"2024-04-17T07:13:58.223705+0000","last_unstale":"2024-04-17T07:14:00.147642+0000","last_undegraded":"2024-04-17T07:14:00.147642+0000","last_fullsized":"2024-04-17T07:14:00.147642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:54.742720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226587+0000","last_change":"2024-04-17T07:13:57.200987+0000","last_active":"2024-04-17T07:13:59.226587+0000","last_peered":"2024-04-17T07:13:59.226587+0000","last_clean":"2024-04-17T07:13:59.226587+0000","last_became_active":"2024-04-17T07:13:57.200681+0000","last_became_peered":"2024-04-17T07:13:57.200681+0000","last_unstale":"2024-04-17T07:13:59.226587+0000","last_undegraded":"2024-04-17T07:13:59.226587+0000","last_fullsized":"2024-04-17T07:13:59.226587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149119+0000","last_change":"2024-04-17T07:13:58.234490+0000","last_active":"2024-04-17T07:14:00.149119+0000","last_peered":"2024-04-17T07:14:00.149119+0000","last_clean":"2024-04-17T07:14:00.149119+0000","last_became_active":"2024-04-17T07:13:58.234250+0000","last_became_peered":"2024-04-17T07:13:58.234250+0000","last_unstale":"2024-04-17T07:14:00.149119+0000","last_undegraded":"2024-04-17T07:14:00.149119+0000","last_fullsized":"2024-04-17T07:14:00.149119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:17.137422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.224627+0000","last_change":"2024-04-17T07:13:57.199555+0000","last_active":"2024-04-17T07:13:59.224627+0000","last_peered":"2024-04-17T07:13:59.224627+0000","last_clean":"2024-04-17T07:13:59.224627+0000","last_became_active":"2024-04-17T07:13:57.199295+0000","last_became_peered":"2024-04-17T07:13:57.199295+0000","last_unstale":"2024-04-17T07:13:59.224627+0000","last_undegraded":"2024-04-17T07:13:59.224627+0000","last_fullsized":"2024-04-17T07:13:59.224627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:36.480482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.143813+0000","last_change":"2024-04-17T07:13:58.222076+0000","last_active":"2024-04-17T07:14:00.143813+0000","last_peered":"2024-04-17T07:14:00.143813+0000","last_clean":"2024-04-17T07:14:00.143813+0000","last_became_active":"2024-04-17T07:13:58.219377+0000","last_became_peered":"2024-04-17T07:13:58.219377+0000","last_unstale":"2024-04-17T07:14:00.143813+0000","last_undegraded":"2024-04-17T07:14:00.143813+0000","last_fullsized":"2024-04-17T07:14:00.143813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:37.495822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217874+0000","last_change":"2024-04-17T07:13:57.203197+0000","last_active":"2024-04-17T07:13:59.217874+0000","last_peered":"2024-04-17T07:13:59.217874+0000","last_clean":"2024-04-17T07:13:59.217874+0000","last_became_active":"2024-04-17T07:13:57.201296+0000","last_became_peered":"2024-04-17T07:13:57.201296+0000","last_unstale":"2024-04-17T07:13:59.217874+0000","last_undegraded":"2024-04-17T07:13:59.217874+0000","last_fullsized":"2024-04-17T07:13:59.217874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:32:13.877683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.144614+0000","last_change":"2024-04-17T07:13:58.220538+0000","last_active":"2024-04-17T07:14:00.144614+0000","last_peered":"2024-04-17T07:14:00.144614+0000","last_clean":"2024-04-17T07:14:00.144614+0000","last_became_active":"2024-04-17T07:13:58.220281+0000","last_became_peered":"2024-04-17T07:13:58.220281+0000","last_unstale":"2024-04-17T07:14:00.144614+0000","last_undegraded":"2024-04-17T07:14:00.144614+0000","last_fullsized":"2024-04-17T07:14:00.144614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:13.826729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.217905+0000","last_change":"2024-04-17T07:13:57.202287+0000","last_active":"2024-04-17T07:13:59.217905+0000","last_peered":"2024-04-17T07:13:59.217905+0000","last_clean":"2024-04-17T07:13:59.217905+0000","last_became_active":"2024-04-17T07:13:57.202007+0000","last_became_peered":"2024-04-17T07:13:57.202007+0000","last_unstale":"2024-04-17T07:13:59.217905+0000","last_undegraded":"2024-04-17T07:13:59.217905+0000","last_fullsized":"2024-04-17T07:13:59.217905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146417+0000","last_change":"2024-04-17T07:13:58.243714+0000","last_active":"2024-04-17T07:14:00.146417+0000","last_peered":"2024-04-17T07:14:00.146417+0000","last_clean":"2024-04-17T07:14:00.146417+0000","last_became_active":"2024-04-17T07:13:58.243199+0000","last_became_peered":"2024-04-17T07:13:58.243199+0000","last_unstale":"2024-04-17T07:14:00.146417+0000","last_undegraded":"2024-04-17T07:14:00.146417+0000","last_fullsized":"2024-04-17T07:14:00.146417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:52.399662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.227303+0000","last_change":"2024-04-17T07:13:57.194783+0000","last_active":"2024-04-17T07:13:59.227303+0000","last_peered":"2024-04-17T07:13:59.227303+0000","last_clean":"2024-04-17T07:13:59.227303+0000","last_became_active":"2024-04-17T07:13:57.194418+0000","last_became_peered":"2024-04-17T07:13:57.194418+0000","last_unstale":"2024-04-17T07:13:59.227303+0000","last_undegraded":"2024-04-17T07:13:59.227303+0000","last_fullsized":"2024-04-17T07:13:59.227303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.173128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.225654+0000","last_change":"2024-04-17T07:13:58.224933+0000","last_active":"2024-04-17T07:13:59.225654+0000","last_peered":"2024-04-17T07:13:59.225654+0000","last_clean":"2024-04-17T07:13:59.225654+0000","last_became_active":"2024-04-17T07:13:58.224635+0000","last_became_peered":"2024-04-17T07:13:58.224635+0000","last_unstale":"2024-04-17T07:13:59.225654+0000","last_undegraded":"2024-04-17T07:13:59.225654+0000","last_fullsized":"2024-04-17T07:13:59.225654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:41.569163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.231353+0000","last_change":"2024-04-17T07:13:58.224558+0000","last_active":"2024-04-17T07:13:59.231353+0000","last_peered":"2024-04-17T07:13:59.231353+0000","last_clean":"2024-04-17T07:13:59.231353+0000","last_became_active":"2024-04-17T07:13:58.224291+0000","last_became_peered":"2024-04-17T07:13:58.224291+0000","last_unstale":"2024-04-17T07:13:59.231353+0000","last_undegraded":"2024-04-17T07:13:59.231353+0000","last_fullsized":"2024-04-17T07:13:59.231353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:30:45.940920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.226442+0000","last_change":"2024-04-17T07:13:57.197610+0000","last_active":"2024-04-17T07:13:59.226442+0000","last_peered":"2024-04-17T07:13:59.226442+0000","last_clean":"2024-04-17T07:13:59.226442+0000","last_became_active":"2024-04-17T07:13:57.197297+0000","last_became_peered":"2024-04-17T07:13:57.197297+0000","last_unstale":"2024-04-17T07:13:59.226442+0000","last_undegraded":"2024-04-17T07:13:59.226442+0000","last_fullsized":"2024-04-17T07:13:59.226442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:50.653092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.229831+0000","last_change":"2024-04-17T07:13:58.227513+0000","last_active":"2024-04-17T07:13:59.229831+0000","last_peered":"2024-04-17T07:13:59.229831+0000","last_clean":"2024-04-17T07:13:59.229831+0000","last_became_active":"2024-04-17T07:13:58.227195+0000","last_became_peered":"2024-04-17T07:13:58.227195+0000","last_unstale":"2024-04-17T07:13:59.229831+0000","last_undegraded":"2024-04-17T07:13:59.229831+0000","last_fullsized":"2024-04-17T07:13:59.229831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:15:09.056824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:13:59.228330+0000","last_change":"2024-04-17T07:13:57.203401+0000","last_active":"2024-04-17T07:13:59.228330+0000","last_peered":"2024-04-17T07:13:59.228330+0000","last_clean":"2024-04-17T07:13:59.228330+0000","last_became_active":"2024-04-17T07:13:57.202900+0000","last_became_peered":"2024-04-17T07:13:57.202900+0000","last_unstale":"2024-04-17T07:13:59.228330+0000","last_undegraded":"2024-04-17T07:13:59.228330+0000","last_fullsized":"2024-04-17T07:13:59.228330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:16.876460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.148975+0000","last_change":"2024-04-17T07:13:58.245486+0000","last_active":"2024-04-17T07:14:00.148975+0000","last_peered":"2024-04-17T07:14:00.148975+0000","last_clean":"2024-04-17T07:14:00.148975+0000","last_became_active":"2024-04-17T07:13:58.245180+0000","last_became_peered":"2024-04-17T07:13:58.245180+0000","last_unstale":"2024-04-17T07:14:00.148975+0000","last_undegraded":"2024-04-17T07:14:00.148975+0000","last_fullsized":"2024-04-17T07:14:00.148975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:42.645551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.147085+0000","last_change":"2024-04-17T07:13:57.197244+0000","last_active":"2024-04-17T07:14:00.147085+0000","last_peered":"2024-04-17T07:14:00.147085+0000","last_clean":"2024-04-17T07:14:00.147085+0000","last_became_active":"2024-04-17T07:13:57.196651+0000","last_became_peered":"2024-04-17T07:13:57.196651+0000","last_unstale":"2024-04-17T07:14:00.147085+0000","last_undegraded":"2024-04-17T07:14:00.147085+0000","last_fullsized":"2024-04-17T07:14:00.147085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:41.544283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.146961+0000","last_change":"2024-04-17T07:13:58.221178+0000","last_active":"2024-04-17T07:14:00.146961+0000","last_peered":"2024-04-17T07:14:00.146961+0000","last_clean":"2024-04-17T07:14:00.146961+0000","last_became_active":"2024-04-17T07:13:58.220895+0000","last_became_peered":"2024-04-17T07:13:58.220895+0000","last_unstale":"2024-04-17T07:14:00.146961+0000","last_undegraded":"2024-04-17T07:14:00.146961+0000","last_fullsized":"2024-04-17T07:14:00.146961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:57.153714+0000","last_clean_scrub_stamp":"2024-04-17T07:13:57.153714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:43.534260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-17T07:14:00.149303+0000","last_change":"2024-04-17T07:13:57.193073+0000","last_active":"2024-04-17T07:14:00.149303+0000","last_peered":"2024-04-17T07:14:00.149303+0000","last_clean":"2024-04-17T07:14:00.149303+0000","last_became_active":"2024-04-17T07:13:57.192203+0000","last_became_peered":"2024-04-17T07:13:57.192203+0000","last_unstale":"2024-04-17T07:14:00.149303+0000","last_undegraded":"2024-04-17T07:14:00.149303+0000","last_fullsized":"2024-04-17T07:14:00.149303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:56.138374+0000","last_clean_scrub_stamp":"2024-04-17T07:13:56.138374+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:24.305808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":669784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":78924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":78960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":80446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":669840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:14:08.934 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-17T07:14:08.934 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-17T07:14:08.934 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-17T07:14:08.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T07:14:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.189+0000 7f7f49e00640 1 Processor -- start 2024-04-17T07:14:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.190+0000 7f7f49e00640 1 -- start start 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.191+0000 7f7f49e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f4415ec60 0x7f7f440b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.191+0000 7f7f49e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f440b6b30 0x7f7f440cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.191+0000 7f7f49e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f440d2d30 con 0x7f7f440cf910 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.191+0000 7f7f49e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7f440d49c0 con 0x7f7f4415ec60 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.191+0000 7f7f49e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7f440d3660 con 0x7f7f440b6b30 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f4415ec60 0x7f7f440b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f4415ec60 0x7f7f440b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42226/0 (socket says 172.21.15.67:42226) 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 -- 172.21.15.67:0/263662489 learned_addr learned my addr 172.21.15.67:0/263662489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 -- 172.21.15.67:0/263662489 >> v2:172.21.15.187:3300/0 conn(0x7f7f440b6b30 msgr2=0x7f7f440cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 --2- 172.21.15.67:0/263662489 >> v2:172.21.15.187:3300/0 conn(0x7f7f440b6b30 0x7f7f440cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 -- 172.21.15.67:0/263662489 >> v1:172.21.15.187:6789/0 conn(0x7f7f440cf910 legacy=0x7f7f4406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.192+0000 7f7f48a00640 1 -- 172.21.15.67:0/263662489 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f440d3010 con 0x7f7f4415ec60 2024-04-17T07:14:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.193+0000 7f7f48a00640 1 --2- 172.21.15.67:0/263662489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440b65f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f440ba590 tx=0x7f7f3402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b792a97a9d217666 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.193+0000 7f7f43400640 1 -- 172.21.15.67:0/263662489 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3402ca80 con 0x7f7f4415ec60 2024-04-17T07:14:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.193+0000 7f7f43400640 1 -- 172.21.15.67:0/263662489 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f3402cc20 con 0x7f7f4415ec60 2024-04-17T07:14:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.193+0000 7f7f43400640 1 -- 172.21.15.67:0/263662489 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f340343f0 con 0x7f7f4415ec60 2024-04-17T07:14:09.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 -- 172.21.15.67:0/263662489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 msgr2=0x7f7f440b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 --2- 172.21.15.67:0/263662489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440b65f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f440ba590 tx=0x7f7f3402bee0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 -- 172.21.15.67:0/263662489 shutdown_connections 2024-04-17T07:14:09.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 --2- 172.21.15.67:0/263662489 >> v2:172.21.15.187:3300/0 conn(0x7f7f440b6b30 0x7f7f440cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 --2- 172.21.15.67:0/263662489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440b65f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 -- 172.21.15.67:0/263662489 >> 172.21.15.67:0/263662489 conn(0x7f7f440d15f0 msgr2=0x7f7f440d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 -- 172.21.15.67:0/263662489 shutdown_connections 2024-04-17T07:14:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.194+0000 7f7f49e00640 1 -- 172.21.15.67:0/263662489 wait complete. 2024-04-17T07:14:09.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.196+0000 7f7f49e00640 1 Processor -- start 2024-04-17T07:14:09.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.196+0000 7f7f49e00640 1 -- start start 2024-04-17T07:14:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 0x7f7f440ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f440cf910 0x7f7f44151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f440cfda0 con 0x7f7f4415ec60 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7f440b6f70 con 0x7f7f440b6b30 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f4415f040 con 0x7f7f440cf910 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42238/0 (socket says 172.21.15.67:42238) 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.197+0000 7f7f49400640 1 -- 172.21.15.67:0/2472057563 learned_addr learned my addr 172.21.15.67:0/2472057563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 -- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f440cf910 msgr2=0x7f7f44151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f440cf910 0x7f7f44151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f48a00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 0x7f7f440ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 -- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 msgr2=0x7f7f440ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 0x7f7f440ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 -- 172.21.15.67:0/2472057563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f440cd7a0 con 0x7f7f4415ec60 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f48a00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 0x7f7f440ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49400640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00c390 tx=0x7f7f3c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c017070 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f3c0035f0 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f440cda80 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.198+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0039e0 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.199+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f440ce040 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.200+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f44068a10 con 0x7f7f4415ec60 2024-04-17T07:14:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.200+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f3c01a070 con 0x7f7f4415ec60 2024-04-17T07:14:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.201+0000 7f7f40c00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 0x7f7f2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.202+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7f3c08afe0 con 0x7f7f4415ec60 2024-04-17T07:14:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.202+0000 7f7f48a00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 0x7f7f2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.203+0000 7f7f48a00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 0x7f7f2006edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7f34035000 tx=0x7f7f34030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.206+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7f3c059390 con 0x7f7f4415ec60 2024-04-17T07:14:09.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.456+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7f440d4290 con 0x7f7f4415ec60 2024-04-17T07:14:09.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.457+0000 7f7f40c00640 1 -- 172.21.15.67:0/2472057563 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f7f3c058c90 con 0x7f7f4415ec60 2024-04-17T07:14:09.457 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:09.457 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-17T07:14:09.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.460+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 msgr2=0x7f7f2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.460+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 0x7f7f2006edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7f34035000 tx=0x7f7f34030000 comp rx=0 tx=0).stop 2024-04-17T07:14:09.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 msgr2=0x7f7f440ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00c390 tx=0x7f7f3c00c850 comp rx=0 tx=0).stop 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 shutdown_connections 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f4415ec60 0x7f7f440ccd70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7f2006c8f0 0x7f7f2006edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f440cf910 0x7f7f44151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 --2- 172.21.15.67:0/2472057563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f440b6b30 0x7f7f440ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 >> 172.21.15.67:0/2472057563 conn(0x7f7f440d15f0 msgr2=0x7f7f44074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 shutdown_connections 2024-04-17T07:14:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.461+0000 7f7f49e00640 1 -- 172.21.15.67:0/2472057563 wait complete. 2024-04-17T07:14:09.480 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-17T07:14:09.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:09.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.706+0000 7fab69000640 1 Processor -- start 2024-04-17T07:14:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.707+0000 7fab69000640 1 -- start start 2024-04-17T07:14:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab69000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab6406b8c0 0x7fab640cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab69000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab64160280 0x7fab6406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab69000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab640cd4a0 con 0x7fab64141ed0 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab69000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fab6415ce30 con 0x7fab6406b8c0 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab69000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fab641607f0 con 0x7fab64160280 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab63400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab6406b8c0 0x7fab640cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab63400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab6406b8c0 0x7fab640cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42250/0 (socket says 172.21.15.67:42250) 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.708+0000 7fab63400640 1 -- 172.21.15.67:0/3788741295 learned_addr learned my addr 172.21.15.67:0/3788741295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.709+0000 7fab63400640 1 -- 172.21.15.67:0/3788741295 >> v2:172.21.15.187:3300/0 conn(0x7fab64160280 msgr2=0x7fab6406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.709+0000 7fab63400640 1 --2- 172.21.15.67:0/3788741295 >> v2:172.21.15.187:3300/0 conn(0x7fab64160280 0x7fab6406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.709+0000 7fab63400640 1 -- 172.21.15.67:0/3788741295 >> v1:172.21.15.187:6789/0 conn(0x7fab64141ed0 legacy=0x7fab6415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.709+0000 7fab63400640 1 -- 172.21.15.67:0/3788741295 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab64160ad0 con 0x7fab6406b8c0 2024-04-17T07:14:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.710+0000 7fab63400640 1 --2- 172.21.15.67:0/3788741295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab640cd8b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fab640aff50 tx=0x7fab5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3d10b44735a08f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.710+0000 7fab62a00640 1 -- 172.21.15.67:0/3788741295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54035020 con 0x7fab6406b8c0 2024-04-17T07:14:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.710+0000 7fab62a00640 1 -- 172.21.15.67:0/3788741295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab54031870 con 0x7fab6406b8c0 2024-04-17T07:14:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.710+0000 7fab62a00640 1 -- 172.21.15.67:0/3788741295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54031c40 con 0x7fab6406b8c0 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 -- 172.21.15.67:0/3788741295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 msgr2=0x7fab640cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 --2- 172.21.15.67:0/3788741295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab640cd8b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fab640aff50 tx=0x7fab5402b910 comp rx=0 tx=0).stop 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 -- 172.21.15.67:0/3788741295 shutdown_connections 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 --2- 172.21.15.67:0/3788741295 >> v2:172.21.15.187:3300/0 conn(0x7fab64160280 0x7fab6406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 --2- 172.21.15.67:0/3788741295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab640cd8b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 -- 172.21.15.67:0/3788741295 >> 172.21.15.67:0/3788741295 conn(0x7fab64067d10 msgr2=0x7fab640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 -- 172.21.15.67:0/3788741295 shutdown_connections 2024-04-17T07:14:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.711+0000 7fab69000640 1 -- 172.21.15.67:0/3788741295 wait complete. 2024-04-17T07:14:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.713+0000 7fab69000640 1 Processor -- start 2024-04-17T07:14:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.713+0000 7fab69000640 1 -- start start 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab6415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 0x7fab6415d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab6406c1e0 con 0x7fab6406b8c0 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab640cddf0 con 0x7fab64160280 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab69000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab64161090 con 0x7fab64141ed0 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab63400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab63400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42264/0 (socket says 172.21.15.67:42264) 2024-04-17T07:14:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.714+0000 7fab63400640 1 -- 172.21.15.67:0/1310918268 learned_addr learned my addr 172.21.15.67:0/1310918268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 -- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 msgr2=0x7fab6415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab6415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 -- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 msgr2=0x7fab6415d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63e00640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 0x7fab6415d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 0x7fab6415d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 -- 172.21.15.67:0/1310918268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6406e930 con 0x7fab6406b8c0 2024-04-17T07:14:09.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63e00640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 0x7fab6415d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab63400640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab54009960 tx=0x7fab540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540443f0 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab540445b0 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab540449a0 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.715+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6406ec10 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.716+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6406f6f0 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.717+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab64068a10 con 0x7fab6406b8c0 2024-04-17T07:14:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.718+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab54038020 con 0x7fab6406b8c0 2024-04-17T07:14:09.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.719+0000 7fab5be00640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 0x7fab4006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:09.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.719+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fab540add30 con 0x7fab6406b8c0 2024-04-17T07:14:09.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.719+0000 7fab5aa00640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 0x7fab4006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.720+0000 7fab5aa00640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 0x7fab4006ee30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab6415f2d0 tx=0x7fab4c009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:09.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.723+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fab5403f070 con 0x7fab6406b8c0 2024-04-17T07:14:09.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.936+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab641565c0 con 0x7fab6406b8c0 2024-04-17T07:14:09.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.939+0000 7fab5be00640 1 -- 172.21.15.67:0/1310918268 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7fab5407c1d0 con 0x7fab6406b8c0 2024-04-17T07:14:09.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 msgr2=0x7fab4006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 0x7fab4006ee30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fab6415f2d0 tx=0x7fab4c009250 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 msgr2=0x7fab6413c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab54009960 tx=0x7fab540046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 shutdown_connections 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64160280 0x7fab6415d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fab4006c970 0x7fab4006ee30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab6415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 --2- 172.21.15.67:0/1310918268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406b8c0 0x7fab6413c9f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 >> 172.21.15.67:0/1310918268 conn(0x7fab64067d10 msgr2=0x7fab6413e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.942+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 shutdown_connections 2024-04-17T07:14:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:09.943+0000 7fab69000640 1 -- 172.21.15.67:0/1310918268 wait complete. 2024-04-17T07:14:09.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:09.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.188+0000 7fd2cd200640 1 Processor -- start 2024-04-17T07:14:10.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.189+0000 7fd2cd200640 1 -- start start 2024-04-17T07:14:10.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2c8142e20 0x7fd2c80cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c80cf160 0x7fd2c80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2c8143200 0x7fd2c80cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c81607a0 con 0x7fd2c8142e20 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c8160a60 con 0x7fd2c8143200 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2cd200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c8160d40 con 0x7fd2c80cf160 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.190+0000 7fd2c5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2c8142e20 0x7fd2c80cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2c8142e20 0x7fd2c80cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42268/0 (socket says 172.21.15.67:42268) 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 -- 172.21.15.67:0/3125530036 learned_addr learned my addr 172.21.15.67:0/3125530036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c6800640 1 --2- 172.21.15.67:0/3125530036 >> v2:172.21.15.187:3300/0 conn(0x7fd2c8143200 0x7fd2c80cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2bd400640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c80cf160 0x7fd2c80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 -- 172.21.15.67:0/3125530036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c80cf160 msgr2=0x7fd2c80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c80cf160 0x7fd2c80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 -- 172.21.15.67:0/3125530036 >> v2:172.21.15.187:3300/0 conn(0x7fd2c8143200 msgr2=0x7fd2c80cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/3125530036 >> v2:172.21.15.187:3300/0 conn(0x7fd2c8143200 0x7fd2c80cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.191+0000 7fd2c5e00640 1 -- 172.21.15.67:0/3125530036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c8161020 con 0x7fd2c8142e20 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.192+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8142e20 0x7fd2c80cec20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80cfad0 tx=0x7fd2b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64f6d2db7d68f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2c5400640 1 -- 172.21.15.67:0/3125530036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b802e070 con 0x7fd2c8142e20 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2c5400640 1 -- 172.21.15.67:0/3125530036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8037070 con 0x7fd2c8142e20 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2c5400640 1 -- 172.21.15.67:0/3125530036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8031a00 con 0x7fd2c8142e20 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 -- 172.21.15.67:0/3125530036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8142e20 msgr2=0x7fd2c80cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8142e20 0x7fd2c80cec20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80cfad0 tx=0x7fd2b802b880 comp rx=0 tx=0).stop 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 -- 172.21.15.67:0/3125530036 shutdown_connections 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 --2- 172.21.15.67:0/3125530036 >> v2:172.21.15.187:3300/0 conn(0x7fd2c8143200 0x7fd2c80cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c80cf160 0x7fd2c80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 --2- 172.21.15.67:0/3125530036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c8142e20 0x7fd2c80cec20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.193+0000 7fd2cd200640 1 -- 172.21.15.67:0/3125530036 >> 172.21.15.67:0/3125530036 conn(0x7fd2c806b8c0 msgr2=0x7fd2c806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.194+0000 7fd2cd200640 1 -- 172.21.15.67:0/3125530036 shutdown_connections 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.194+0000 7fd2cd200640 1 -- 172.21.15.67:0/3125530036 wait complete. 2024-04-17T07:14:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.195+0000 7fd2cd200640 1 Processor -- start 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.195+0000 7fd2cd200640 1 -- start start 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 0x7fd2c8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2c8143200 0x7fd2c8158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c8071df0 con 0x7fd2c80cf160 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80ce730 con 0x7fd2c8143200 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.196+0000 7fd2cd200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c8071f70 con 0x7fd2c8142e20 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42284/0 (socket says 172.21.15.67:42284) 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 -- 172.21.15.67:0/2895670972 learned_addr learned my addr 172.21.15.67:0/2895670972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2bd400640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 0x7fd2c8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c6800640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2c8143200 0x7fd2c8158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 -- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 msgr2=0x7fd2c8158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 0x7fd2c8158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 -- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2c8143200 msgr2=0x7fd2c8158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2c8143200 0x7fd2c8158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 -- 172.21.15.67:0/2895670972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c8159310 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2bd400640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 0x7fd2c8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.197+0000 7fd2c5e00640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8000c00 tx=0x7fd2b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.198+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8032880 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.198+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8032a20 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.198+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8032e10 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.198+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c81595f0 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.198+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c8159bb0 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.199+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c8068b90 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.200+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2b80453f0 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.201+0000 7fd2bea00640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 0x7fd2a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.201+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd2b80adb40 con 0x7fd2c80cf160 2024-04-17T07:14:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.202+0000 7fd2bd400640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 0x7fd2a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.203+0000 7fd2bd400640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 0x7fd2a406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80b0900 tx=0x7fd2b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.206+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd2b8040070 con 0x7fd2c80cf160 2024-04-17T07:14:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.414+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2c80cde80 con 0x7fd2c80cf160 2024-04-17T07:14:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.415+0000 7fd2bea00640 1 -- 172.21.15.67:0/2895670972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fd2b807b920 con 0x7fd2c80cf160 2024-04-17T07:14:10.415 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:10.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 msgr2=0x7fd2a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 0x7fd2a406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80b0900 tx=0x7fd2b0008040 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 msgr2=0x7fd2c8153be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8000c00 tx=0x7fd2b80046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 shutdown_connections 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2c8143200 0x7fd2c8158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd2a406c8f0 0x7fd2a406edb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2c8142e20 0x7fd2c8158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 --2- 172.21.15.67:0/2895670972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2c80cf160 0x7fd2c8153be0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.418+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 >> 172.21.15.67:0/2895670972 conn(0x7fd2c806b8c0 msgr2=0x7fd2c815ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.419+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 shutdown_connections 2024-04-17T07:14:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.419+0000 7fd2cd200640 1 -- 172.21.15.67:0/2895670972 wait complete. 2024-04-17T07:14:10.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.653+0000 7fc609800640 1 Processor -- start 2024-04-17T07:14:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.653+0000 7fc609800640 1 -- start start 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc609800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc60406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc609800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc604146360 con 0x7fc604145f80 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc609800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc604071e70 con 0x7fc60406c0a0 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc609800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc6040727a0 con 0x7fc6040cd920 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc603e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc604145f80 0x7fc6040cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50258/0 (socket says 172.21.15.67:50258) 2024-04-17T07:14:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.655+0000 7fc603e00640 1 -- 172.21.15.67:0/1416878914 learned_addr learned my addr 172.21.15.67:0/1416878914 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700063777 0 0) 0x7fc604146360 con 0x7fc604145f80 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5e8003520 con 0x7fc604145f80 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc5fb400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc60406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2878609484 0 0) 0x7fc5e8003520 con 0x7fc604145f80 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 msgr2=0x7fc60406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc60406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 >> v1:172.21.15.187:6789/0 conn(0x7fc60406c0a0 legacy=0x7fc60406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc604072150 con 0x7fc604145f80 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc5f40019f0 con 0x7fc604145f80 2024-04-17T07:14:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 >> v1:172.21.15.67:6789/0 conn(0x7fc604145f80 legacy=0x7fc6040cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e80039e0 0x7fc5e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e8004800 0x7fc5e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc5e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6040727a0 con 0x7fc5e80039e0 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e8005550 con 0x7fc5e8004800 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e800bf00 con 0x7fc6040cd920 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.656+0000 7fc603e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e80039e0 0x7fc5e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc5fb400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e8004800 0x7fc5e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc608e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc5e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc603e00640 1 -- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 msgr2=0x7fc5e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc603e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc5e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc603e00640 1 -- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e8004800 msgr2=0x7fc5e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc603e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e8004800 0x7fc5e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc603e00640 1 -- 172.21.15.67:0/1416878914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e8003520 con 0x7fc5e80039e0 2024-04-17T07:14:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc608e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6040cd920 0x7fc5e800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.657+0000 7fc5fb400640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e8004800 0x7fc5e8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.658+0000 7fc603e00640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e80039e0 0x7fc5e8003e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc5f400a850 tx=0x7fc5e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19ca01a3d373ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.658+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400a4a0 con 0x7fc5e80039e0 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.658+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f4034650 con 0x7fc5e80039e0 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.658+0000 7fc603400640 1 -- 172.21.15.67:0/1416878914 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4034a40 con 0x7fc5e80039e0 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 -- 172.21.15.67:0/1416878914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e80039e0 msgr2=0x7fc5e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 --2- 172.21.15.67:0/1416878914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e80039e0 0x7fc5e8003e00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc5f400a850 tx=0x7fc5e8005830 comp rx=0 tx=0).stop 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc608e00640 1 -- 172.21.15.67:0/1416878914 reap_dead start 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 -- 172.21.15.67:0/1416878914 shutdown_connections 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 -- 172.21.15.67:0/1416878914 >> 172.21.15.67:0/1416878914 conn(0x7fc6040b2190 msgr2=0x7fc6040b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 -- 172.21.15.67:0/1416878914 shutdown_connections 2024-04-17T07:14:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.659+0000 7fc609800640 1 -- 172.21.15.67:0/1416878914 wait complete. 2024-04-17T07:14:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.661+0000 7fc609800640 1 Processor -- start 2024-04-17T07:14:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.661+0000 7fc609800640 1 -- start start 2024-04-17T07:14:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 0x7fc6041e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6041e85d0 0x7fc6041e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc604072db0 con 0x7fc6041e9730 2024-04-17T07:14:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc604072c30 con 0x7fc5e80039e0 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc609800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc60406c4a0 con 0x7fc6041e85d0 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.662+0000 7fc5fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42302/0 (socket says 172.21.15.67:42302) 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 -- 172.21.15.67:0/3893620072 learned_addr learned my addr 172.21.15.67:0/3893620072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc603e00640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 0x7fc6041e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 -- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6041e85d0 msgr2=0x7fc6041e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6041e85d0 0x7fc6041e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 -- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 msgr2=0x7fc6041e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 0x7fc6041e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 -- 172.21.15.67:0/3893620072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6041e5f70 con 0x7fc6041e9730 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc603e00640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 0x7fc6041e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.663+0000 7fc5fb400640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00e840 tx=0x7fc5ec00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.664+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec017070 con 0x7fc6041e9730 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.664+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec00c6d0 con 0x7fc6041e9730 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.664+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec00f040 con 0x7fc6041e9730 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.664+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6041e6250 con 0x7fc6041e9730 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.664+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6041e6810 con 0x7fc6041e9730 2024-04-17T07:14:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.665+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc604069120 con 0x7fc6041e9730 2024-04-17T07:14:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.666+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5ec01a070 con 0x7fc6041e9730 2024-04-17T07:14:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.667+0000 7fc600c00640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 0x7fc5e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.667+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc5ec08b5f0 con 0x7fc6041e9730 2024-04-17T07:14:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.668+0000 7fc603e00640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 0x7fc5e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.669+0000 7fc603e00640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 0x7fc5e006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4034000 tx=0x7fc5f402f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.671+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc604069120 con 0x7fc6041e9730 2024-04-17T07:14:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.878+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc604068a10 con 0x7fc6041e9730 2024-04-17T07:14:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.879+0000 7fc600c00640 1 -- 172.21.15.67:0/3893620072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fc5ec059a20 con 0x7fc6041e9730 2024-04-17T07:14:10.879 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 msgr2=0x7fc5e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 0x7fc5e006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4034000 tx=0x7fc5f402f000 comp rx=0 tx=0).stop 2024-04-17T07:14:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 msgr2=0x7fc6041e8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00e840 tx=0x7fc5ec00ed00 comp rx=0 tx=0).stop 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 shutdown_connections 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5e006c920 0x7fc5e006ede0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6041e85d0 0x7fc6041e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6041e9730 0x7fc6041e8090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 --2- 172.21.15.67:0/3893620072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80039e0 0x7fc6041e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 >> 172.21.15.67:0/3893620072 conn(0x7fc6040b2190 msgr2=0x7fc60406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 shutdown_connections 2024-04-17T07:14:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:10.882+0000 7fc609800640 1 -- 172.21.15.67:0/3893620072 wait complete. 2024-04-17T07:14:10.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.130+0000 7f19b7e00640 1 Processor -- start 2024-04-17T07:14:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.130+0000 7f19b7e00640 1 -- start start 2024-04-17T07:14:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.131+0000 7f19b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19b8160950 0x7f19b809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.131+0000 7f19b7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19b80d0ee0 con 0x7f19b80ce160 2024-04-17T07:14:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.131+0000 7f19b7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19b80d11c0 con 0x7f19b8160d30 2024-04-17T07:14:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f19b809beb0 con 0x7f19b8160950 2024-04-17T07:14:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19b80ce160 0x7f19b8166a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50274/0 (socket says 172.21.15.67:50274) 2024-04-17T07:14:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b7400640 1 -- 172.21.15.67:0/922933077 learned_addr learned my addr 172.21.15.67:0/922933077 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b6a00640 1 --2- 172.21.15.67:0/922933077 >> v2:172.21.15.67:3300/0 conn(0x7f19b8160950 0x7f19b809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b5600640 1 -- 172.21.15.67:0/922933077 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143205687 0 0) 0x7f19b80d0ee0 con 0x7f19b80ce160 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b5600640 1 -- 172.21.15.67:0/922933077 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1998003520 con 0x7f19b80ce160 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b6a00640 1 -- 172.21.15.67:0/922933077 >> v1:172.21.15.187:6789/0 conn(0x7f19b8160d30 legacy=0x7f19b80cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b6a00640 1 -- 172.21.15.67:0/922933077 >> v1:172.21.15.67:6789/0 conn(0x7f19b80ce160 legacy=0x7f19b8166a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.132+0000 7f19b6a00640 1 -- 172.21.15.67:0/922933077 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b80cd250 con 0x7f19b8160950 2024-04-17T07:14:11.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b6a00640 1 --2- 172.21.15.67:0/922933077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b8160950 0x7f19b809aa60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f19a80097a0 tx=0x7f19a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5888b3138b281f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b5600640 1 -- 172.21.15.67:0/922933077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8034070 con 0x7f19b8160950 2024-04-17T07:14:11.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b5600640 1 -- 172.21.15.67:0/922933077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a8030610 con 0x7f19b8160950 2024-04-17T07:14:11.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b5600640 1 -- 172.21.15.67:0/922933077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80309e0 con 0x7f19b8160950 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b7e00640 1 -- 172.21.15.67:0/922933077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b8160950 msgr2=0x7f19b809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.134+0000 7f19b7e00640 1 --2- 172.21.15.67:0/922933077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b8160950 0x7f19b809aa60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f19a80097a0 tx=0x7f19a802b960 comp rx=0 tx=0).stop 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.135+0000 7f19b7e00640 1 -- 172.21.15.67:0/922933077 shutdown_connections 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.135+0000 7f19b7e00640 1 --2- 172.21.15.67:0/922933077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b8160950 0x7f19b809aa60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.135+0000 7f19b7e00640 1 -- 172.21.15.67:0/922933077 >> 172.21.15.67:0/922933077 conn(0x7f19b80b3f90 msgr2=0x7f19b80b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.135+0000 7f19b7e00640 1 -- 172.21.15.67:0/922933077 shutdown_connections 2024-04-17T07:14:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.135+0000 7f19b7e00640 1 -- 172.21.15.67:0/922933077 wait complete. 2024-04-17T07:14:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.136+0000 7f19b7e00640 1 Processor -- start 2024-04-17T07:14:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.137+0000 7f19b7e00640 1 -- start start 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.137+0000 7f19b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 0x7f19b81552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19b8160d30 0x7f19b81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19b80d14a0 con 0x7f19b80ce160 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19b80d0c90 con 0x7f19b8160950 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19b80b3330 con 0x7f19b8160d30 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42324/0 (socket says 172.21.15.67:42324) 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 -- 172.21.15.67:0/3220855616 learned_addr learned my addr 172.21.15.67:0/3220855616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6000640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 0x7f19b81552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b7400640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19b8160d30 0x7f19b81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 -- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19b8160d30 msgr2=0x7f19b81585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19b8160d30 0x7f19b81585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 -- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 msgr2=0x7f19b81552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 0x7f19b81552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6a00640 1 -- 172.21.15.67:0/3220855616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b80b5260 con 0x7f19b80ce160 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.138+0000 7f19b6000640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 0x7f19b81552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19b6a00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19a802bf70 tx=0x7f19a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8034040 con 0x7f19b80ce160 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a803aaa0 con 0x7f19b80ce160 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80443f0 con 0x7f19b80ce160 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b80b5540 con 0x7f19b80ce160 2024-04-17T07:14:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.139+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b80b5b00 con 0x7f19b80ce160 2024-04-17T07:14:11.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.140+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b80b4b80 con 0x7f19b80ce160 2024-04-17T07:14:11.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.141+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19a804b060 con 0x7f19b80ce160 2024-04-17T07:14:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.142+0000 7f19aea00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 0x7f199406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.142+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f19a80ad970 con 0x7f19b80ce160 2024-04-17T07:14:11.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.143+0000 7f19b6000640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 0x7f199406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.144+0000 7f19b6000640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 0x7f199406ede0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f19a00012e0 tx=0x7f19a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.147+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f19a803f070 con 0x7f19b80ce160 2024-04-17T07:14:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.352+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19b80c8ec0 con 0x7f19b80ce160 2024-04-17T07:14:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.353+0000 7f19aea00640 1 -- 172.21.15.67:0/3220855616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f19a804b350 con 0x7f19b80ce160 2024-04-17T07:14:11.353 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 msgr2=0x7f199406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 0x7f199406ede0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f19a00012e0 tx=0x7f19a0009040 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 msgr2=0x7f19b814b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19a802bf70 tx=0x7f19a80046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 shutdown_connections 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f199406c920 0x7f199406ede0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19b8160d30 0x7f19b81585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19b8160950 0x7f19b81552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3220855616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19b80ce160 0x7f19b814b940 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.355+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 >> 172.21.15.67:0/3220855616 conn(0x7f19b80b3f90 msgr2=0x7f19b815c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.356+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 shutdown_connections 2024-04-17T07:14:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.356+0000 7f19b7e00640 1 -- 172.21.15.67:0/3220855616 wait complete. 2024-04-17T07:14:11.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.588+0000 7f9524a00640 1 Processor -- start 2024-04-17T07:14:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.589+0000 7f9524a00640 1 -- start start 2024-04-17T07:14:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.590+0000 7f9524a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f952015c5a0 0x7f952015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.590+0000 7f9524a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 0x7f95200ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.590+0000 7f9524a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952015f440 con 0x7f95200b4210 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.590+0000 7f9524a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f952015f720 con 0x7f952015c5a0 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.590+0000 7f9524a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f952015fa00 con 0x7f95200b46c0 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95200b4210 0x7f95200af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50280/0 (socket says 172.21.15.67:50280) 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f952015c5a0 0x7f952015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951ea00640 1 -- 172.21.15.67:0/2211542145 learned_addr learned my addr 172.21.15.67:0/2211542145 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951fe00640 1 --2- 172.21.15.67:0/2211542145 >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 0x7f95200ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951f400640 1 -- 172.21.15.67:0/2211542145 >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 msgr2=0x7f95200ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951f400640 1 --2- 172.21.15.67:0/2211542145 >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 0x7f95200ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951f400640 1 -- 172.21.15.67:0/2211542145 >> v1:172.21.15.67:6789/0 conn(0x7f95200b4210 legacy=0x7f95200af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951e000640 1 -- 172.21.15.67:0/2211542145 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868991189 0 0) 0x7f952015f440 con 0x7f95200b4210 2024-04-17T07:14:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951f400640 1 -- 172.21.15.67:0/2211542145 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952015fce0 con 0x7f952015c5a0 2024-04-17T07:14:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.591+0000 7f951fe00640 1 --2- 172.21.15.67:0/2211542145 >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 0x7f95200ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f951f400640 1 --2- 172.21.15.67:0/2211542145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952015c5a0 0x7f952015ee60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95200b5b60 tx=0x7f951002bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b146be8df1f88214 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f951e000640 1 -- 172.21.15.67:0/2211542145 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510039070 con 0x7f952015c5a0 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f951e000640 1 -- 172.21.15.67:0/2211542145 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f951002cb50 con 0x7f952015c5a0 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f951e000640 1 -- 172.21.15.67:0/2211542145 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510034440 con 0x7f952015c5a0 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f9524a00640 1 -- 172.21.15.67:0/2211542145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952015c5a0 msgr2=0x7f952015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.593+0000 7f9524a00640 1 --2- 172.21.15.67:0/2211542145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952015c5a0 0x7f952015ee60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95200b5b60 tx=0x7f951002bda0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 -- 172.21.15.67:0/2211542145 shutdown_connections 2024-04-17T07:14:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 --2- 172.21.15.67:0/2211542145 >> v2:172.21.15.187:3300/0 conn(0x7f95200b46c0 0x7f95200ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 --2- 172.21.15.67:0/2211542145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952015c5a0 0x7f952015ee60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 -- 172.21.15.67:0/2211542145 >> 172.21.15.67:0/2211542145 conn(0x7f9520067d10 msgr2=0x7f95200b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 -- 172.21.15.67:0/2211542145 shutdown_connections 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.594+0000 7f9524a00640 1 -- 172.21.15.67:0/2211542145 wait complete. 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.596+0000 7f9524a00640 1 Processor -- start 2024-04-17T07:14:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.596+0000 7f9524a00640 1 -- start start 2024-04-17T07:14:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95200b4210 0x7f95201e6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952015c5a0 0x7f9520145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f952006e7f0 con 0x7f95201f7f60 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f952006e670 con 0x7f95200b4210 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f9524a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95200afb50 con 0x7f952015c5a0 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42334/0 (socket says 172.21.15.67:42334) 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 -- 172.21.15.67:0/1093948042 learned_addr learned my addr 172.21.15.67:0/1093948042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951f400640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95200b4210 0x7f95201e6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951ea00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952015c5a0 0x7f9520145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 -- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952015c5a0 msgr2=0x7f9520145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952015c5a0 0x7f9520145330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 -- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95200b4210 msgr2=0x7f95201e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95200b4210 0x7f95201e6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.597+0000 7f951fe00640 1 -- 172.21.15.67:0/1093948042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95201e69b0 con 0x7f95201f7f60 2024-04-17T07:14:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f951fe00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f951800c850 tx=0x7f951800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518015070 con 0x7f95201f7f60 2024-04-17T07:14:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9518004020 con 0x7f95201f7f60 2024-04-17T07:14:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95201f8460 con 0x7f95201f7f60 2024-04-17T07:14:11.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95180043f0 con 0x7f95201f7f60 2024-04-17T07:14:11.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.598+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95201f8a20 con 0x7f95201f7f60 2024-04-17T07:14:11.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.599+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9520068a10 con 0x7f95201f7f60 2024-04-17T07:14:11.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.600+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f95180077d0 con 0x7f95201f7f60 2024-04-17T07:14:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.601+0000 7f9517400640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 0x7f94fc06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.601+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f951808b990 con 0x7f95201f7f60 2024-04-17T07:14:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.602+0000 7f951f400640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 0x7f94fc06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:11.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.603+0000 7f951f400640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 0x7f94fc06ee10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9510036000 tx=0x7f9510030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.606+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9518059db0 con 0x7f95201f7f60 2024-04-17T07:14:11.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.856+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95200b3ff0 con 0x7f95201f7f60 2024-04-17T07:14:11.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.857+0000 7f9517400640 1 -- 172.21.15.67:0/1093948042 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7f95180596b0 con 0x7f95201f7f60 2024-04-17T07:14:11.857 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:11.857 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:11.857 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:11.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.859+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 msgr2=0x7f94fc06ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.859+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 0x7f94fc06ee10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9510036000 tx=0x7f9510030000 comp rx=0 tx=0).stop 2024-04-17T07:14:11.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.859+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 msgr2=0x7f9520159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:11.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.859+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f951800c850 tx=0x7f951800cd10 comp rx=0 tx=0).stop 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 shutdown_connections 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95201f7f60 0x7f9520159340 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f94fc06c950 0x7f94fc06ee10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952015c5a0 0x7f9520145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 --2- 172.21.15.67:0/1093948042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95200b4210 0x7f95201e6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 >> 172.21.15.67:0/1093948042 conn(0x7f9520067d10 msgr2=0x7f95200b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 shutdown_connections 2024-04-17T07:14:11.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:11.860+0000 7f9524a00640 1 -- 172.21.15.67:0/1093948042 wait complete. 2024-04-17T07:14:11.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.099+0000 7f22c6400640 1 Processor -- start 2024-04-17T07:14:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.099+0000 7f22c6400640 1 -- start start 2024-04-17T07:14:12.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.100+0000 7f22c6400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22c00cfae0 0x7f22c0169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.100+0000 7f22c6400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c00d2d30 con 0x7f22c00b6d00 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.100+0000 7f22c6400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c00d3010 con 0x7f22c015ec60 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.100+0000 7f22c6400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f22c00d49c0 con 0x7f22c00cfae0 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22c5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22c00cfae0 0x7f22c0169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22bfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f22c00b6d00 0x7f22c00cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50286/0 (socket says 172.21.15.67:50286) 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22bfe00640 1 -- 172.21.15.67:0/1737314094 learned_addr learned my addr 172.21.15.67:0/1737314094 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22c5a00640 1 -- 172.21.15.67:0/1737314094 >> v1:172.21.15.187:6789/0 conn(0x7f22c015ec60 legacy=0x7f22c00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22c5a00640 1 -- 172.21.15.67:0/1737314094 >> v1:172.21.15.67:6789/0 conn(0x7f22c00b6d00 legacy=0x7f22c00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.101+0000 7f22c5a00640 1 -- 172.21.15.67:0/1737314094 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c00d32f0 con 0x7f22c00cfae0 2024-04-17T07:14:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.102+0000 7f22c5a00640 1 --2- 172.21.15.67:0/1737314094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c0169810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22b80097a0 tx=0x7f22b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689a92d2d5d1dbe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.102+0000 7f22bf400640 1 -- 172.21.15.67:0/1737314094 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8034070 con 0x7f22c00cfae0 2024-04-17T07:14:12.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22bf400640 1 -- 172.21.15.67:0/1737314094 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b8030610 con 0x7f22c00cfae0 2024-04-17T07:14:12.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22bf400640 1 -- 172.21.15.67:0/1737314094 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b80309e0 con 0x7f22c00cfae0 2024-04-17T07:14:12.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 -- 172.21.15.67:0/1737314094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 msgr2=0x7f22c0169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 --2- 172.21.15.67:0/1737314094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c0169810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f22b80097a0 tx=0x7f22b802b960 comp rx=0 tx=0).stop 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 -- 172.21.15.67:0/1737314094 shutdown_connections 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 --2- 172.21.15.67:0/1737314094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c0169810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 -- 172.21.15.67:0/1737314094 >> 172.21.15.67:0/1737314094 conn(0x7f22c00d15f0 msgr2=0x7f22c00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 -- 172.21.15.67:0/1737314094 shutdown_connections 2024-04-17T07:14:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.103+0000 7f22c6400640 1 -- 172.21.15.67:0/1737314094 wait complete. 2024-04-17T07:14:12.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.105+0000 7f22c6400640 1 Processor -- start 2024-04-17T07:14:12.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.105+0000 7f22c6400640 1 -- start start 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22c00b6d00 0x7f22c014d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c015ec60 0x7f22c00cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00b8b80 con 0x7f22c00cfae0 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00d3660 con 0x7f22c015ec60 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.106+0000 7f22c6400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f22c00d00c0 con 0x7f22c00b6d00 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42354/0 (socket says 172.21.15.67:42354) 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 -- 172.21.15.67:0/2662088596 learned_addr learned my addr 172.21.15.67:0/2662088596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22c5000640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22c00b6d00 0x7f22c014d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22c5a00640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c015ec60 0x7f22c00cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 -- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22c00b6d00 msgr2=0x7f22c014d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22c00b6d00 0x7f22c014d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 -- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c015ec60 msgr2=0x7f22c00cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c015ec60 0x7f22c00cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 -- 172.21.15.67:0/2662088596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c00cce60 con 0x7f22c00cfae0 2024-04-17T07:14:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.107+0000 7f22bfe00640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22b000cba0 tx=0x7f22b000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.108+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0017070 con 0x7f22c00cfae0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.108+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b000ad40 con 0x7f22c00cfae0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.108+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0012600 con 0x7f22c00cfae0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.108+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c00cd100 con 0x7f22c00cfae0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.108+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c00cd6c0 con 0x7f22c00cfae0 2024-04-17T07:14:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.109+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c0068a10 con 0x7f22c00cfae0 2024-04-17T07:14:12.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.110+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22b001a070 con 0x7f22c00cfae0 2024-04-17T07:14:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.111+0000 7f22bcc00640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 0x7f229c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.112+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f22b008b830 con 0x7f22c00cfae0 2024-04-17T07:14:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.112+0000 7f22c5000640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 0x7f229c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.113+0000 7f22c5000640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 0x7f229c06edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f22a8007fd0 tx=0x7f22a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.116+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f22b0059640 con 0x7f22c00cfae0 2024-04-17T07:14:12.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.326+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22c00cdae0 con 0x7f22c00cfae0 2024-04-17T07:14:12.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.329+0000 7f22bcc00640 1 -- 172.21.15.67:0/2662088596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f22b005d940 con 0x7f22c00cfae0 2024-04-17T07:14:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:12.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 msgr2=0x7f229c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 0x7f229c06edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f22a8007fd0 tx=0x7f22a8009040 comp rx=0 tx=0).stop 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 msgr2=0x7f22c01f8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22b000cba0 tx=0x7f22b000a430 comp rx=0 tx=0).stop 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 shutdown_connections 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c015ec60 0x7f22c00cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22c00cfae0 0x7f22c01f8300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f229c06c8f0 0x7f229c06edb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 --2- 172.21.15.67:0/2662088596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22c00b6d00 0x7f22c014d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 >> 172.21.15.67:0/2662088596 conn(0x7f22c00d15f0 msgr2=0x7f22c015e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:12.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 shutdown_connections 2024-04-17T07:14:12.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.332+0000 7f22c6400640 1 -- 172.21.15.67:0/2662088596 wait complete. 2024-04-17T07:14:12.350 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:12.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:12.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.569+0000 7f5d4a000640 1 Processor -- start 2024-04-17T07:14:12.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.569+0000 7f5d4a000640 1 -- start start 2024-04-17T07:14:12.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.570+0000 7f5d4a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 0x7f5d440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.570+0000 7f5d4a000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5d440cfcd0 0x7f5d440d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.570+0000 7f5d4a000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d440d4f70 con 0x7f5d440b3b50 2024-04-17T07:14:12.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.570+0000 7f5d4a000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5d440d5250 con 0x7f5d440cfcd0 2024-04-17T07:14:12.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.570+0000 7f5d4a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d440cf4d0 con 0x7f5d4415ec20 2024-04-17T07:14:12.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.571+0000 7f5d48c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 0x7f5d440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.571+0000 7f5d43e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5d440b3b50 0x7f5d4406ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39532/0 (socket says 172.21.15.67:39532) 2024-04-17T07:14:12.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.571+0000 7f5d43e00640 1 -- 172.21.15.67:0/3811956454 learned_addr learned my addr 172.21.15.67:0/3811956454 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:12.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.571+0000 7f5d49600640 1 --2- 172.21.15.67:0/3811956454 >> v2:172.21.15.187:3300/0 conn(0x7f5d440cfcd0 0x7f5d440d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.572+0000 7f5d48c00640 1 -- 172.21.15.67:0/3811956454 >> v2:172.21.15.187:3300/0 conn(0x7f5d440cfcd0 msgr2=0x7f5d440d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:12.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.572+0000 7f5d48c00640 1 --2- 172.21.15.67:0/3811956454 >> v2:172.21.15.187:3300/0 conn(0x7f5d440cfcd0 0x7f5d440d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.572+0000 7f5d48c00640 1 -- 172.21.15.67:0/3811956454 >> v1:172.21.15.187:6789/0 conn(0x7f5d440b3b50 legacy=0x7f5d4406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.572+0000 7f5d48c00640 1 -- 172.21.15.67:0/3811956454 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d440cf7b0 con 0x7f5d4415ec20 2024-04-17T07:14:12.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.572+0000 7f5d43400640 1 -- 172.21.15.67:0/3811956454 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996513313 0 0) 0x7f5d440d4f70 con 0x7f5d440b3b50 2024-04-17T07:14:12.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.574+0000 7f5d48c00640 1 --2- 172.21.15.67:0/3811956454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 0x7f5d440d22b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d380097a0 tx=0x7f5d3802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b476c15aef483b9b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.574+0000 7f5d43400640 1 -- 172.21.15.67:0/3811956454 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38038070 con 0x7f5d4415ec20 2024-04-17T07:14:12.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.574+0000 7f5d43400640 1 -- 172.21.15.67:0/3811956454 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d38031700 con 0x7f5d4415ec20 2024-04-17T07:14:12.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.574+0000 7f5d43400640 1 -- 172.21.15.67:0/3811956454 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38031ad0 con 0x7f5d4415ec20 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 -- 172.21.15.67:0/3811956454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 msgr2=0x7f5d440d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 --2- 172.21.15.67:0/3811956454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 0x7f5d440d22b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d380097a0 tx=0x7f5d3802b7a0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 -- 172.21.15.67:0/3811956454 shutdown_connections 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 --2- 172.21.15.67:0/3811956454 >> v2:172.21.15.187:3300/0 conn(0x7f5d440cfcd0 0x7f5d440d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 --2- 172.21.15.67:0/3811956454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d4415ec20 0x7f5d440d22b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 -- 172.21.15.67:0/3811956454 >> 172.21.15.67:0/3811956454 conn(0x7f5d4408bff0 msgr2=0x7f5d440d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 -- 172.21.15.67:0/3811956454 shutdown_connections 2024-04-17T07:14:12.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.575+0000 7f5d4a000640 1 -- 172.21.15.67:0/3811956454 wait complete. 2024-04-17T07:14:12.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.577+0000 7f5d4a000640 1 Processor -- start 2024-04-17T07:14:12.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.577+0000 7f5d4a000640 1 -- start start 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d440b3b50 0x7f5d4414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d440cfcd0 0x7f5d440b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4406d2d0 con 0x7f5d4415ec20 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4406d150 con 0x7f5d440cfcd0 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.578+0000 7f5d4a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d440d10d0 con 0x7f5d440b3b50 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42358/0 (socket says 172.21.15.67:42358) 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 -- 172.21.15.67:0/658091676 learned_addr learned my addr 172.21.15.67:0/658091676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d43e00640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d440cfcd0 0x7f5d440b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d48c00640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d440b3b50 0x7f5d4414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 -- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d440b3b50 msgr2=0x7f5d4414cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d440b3b50 0x7f5d4414cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 -- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d440cfcd0 msgr2=0x7f5d440b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d440cfcd0 0x7f5d440b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.579+0000 7f5d49600640 1 -- 172.21.15.67:0/658091676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d440b50e0 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d49600640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009d20 tx=0x7f5d3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c018020 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c009b10 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d440b4b20 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0146c0 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.580+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d440b5600 con 0x7f5d4415ec20 2024-04-17T07:14:12.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.581+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d44068a10 con 0x7f5d4415ec20 2024-04-17T07:14:12.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.582+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d3c005d10 con 0x7f5d4415ec20 2024-04-17T07:14:12.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.583+0000 7f5d40c00640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 0x7f5d2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:12.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.583+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5d3c08c4d0 con 0x7f5d4415ec20 2024-04-17T07:14:12.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.584+0000 7f5d48c00640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 0x7f5d2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:12.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.585+0000 7f5d48c00640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 0x7f5d2406edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d38035000 tx=0x7f5d38008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:12.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.588+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5d3c022070 con 0x7f5d4415ec20 2024-04-17T07:14:12.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.790+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d4406b9a0 con 0x7f5d4415ec20 2024-04-17T07:14:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.792+0000 7f5d40c00640 1 -- 172.21.15.67:0/658091676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f5d3c08ae20 con 0x7f5d4415ec20 2024-04-17T07:14:12.792 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 msgr2=0x7f5d2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 0x7f5d2406edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d38035000 tx=0x7f5d38008030 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 msgr2=0x7f5d44158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009d20 tx=0x7f5d3c00c490 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 shutdown_connections 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d4415ec20 0x7f5d44158b10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d440cfcd0 0x7f5d440b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f5d2406c8f0 0x7f5d2406edb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 --2- 172.21.15.67:0/658091676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d440b3b50 0x7f5d4414cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 >> 172.21.15.67:0/658091676 conn(0x7f5d4408bff0 msgr2=0x7f5d440d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.794+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 shutdown_connections 2024-04-17T07:14:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:12.795+0000 7f5d4a000640 1 -- 172.21.15.67:0/658091676 wait complete. 2024-04-17T07:14:12.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.045+0000 7ff0d8e00640 1 Processor -- start 2024-04-17T07:14:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.046+0000 7ff0d8e00640 1 -- start start 2024-04-17T07:14:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.047+0000 7ff0d8e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.047+0000 7ff0d8e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 0x7ff0d40b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.047+0000 7ff0d8e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d40d00b0 con 0x7ff0d415ecb0 2024-04-17T07:14:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.047+0000 7ff0d8e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d40d0c30 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.047+0000 7ff0d8e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d40d0390 con 0x7ff0d40d33e0 2024-04-17T07:14:13.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.048+0000 7ff0d3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 0x7ff0d40b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.048+0000 7ff0d2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.048+0000 7ff0d3400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff0d415ecb0 0x7ff0d40d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39538/0 (socket says 172.21.15.67:39538) 2024-04-17T07:14:13.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.048+0000 7ff0d3400640 1 -- 172.21.15.67:0/1810234147 learned_addr learned my addr 172.21.15.67:0/1810234147 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2000640 1 -- 172.21.15.67:0/1810234147 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2950766009 0 0) 0x7ff0d40d00b0 con 0x7ff0d415ecb0 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2000640 1 -- 172.21.15.67:0/1810234147 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b4003520 con 0x7ff0d415ecb0 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2a00640 1 -- 172.21.15.67:0/1810234147 >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 msgr2=0x7ff0d40b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2a00640 1 --2- 172.21.15.67:0/1810234147 >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 0x7ff0d40b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2a00640 1 -- 172.21.15.67:0/1810234147 >> v1:172.21.15.187:6789/0 conn(0x7ff0d415ecb0 legacy=0x7ff0d40d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d2a00640 1 -- 172.21.15.67:0/1810234147 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d40d0670 con 0x7ff0d40d33e0 2024-04-17T07:14:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.049+0000 7ff0d3e00640 1 --2- 172.21.15.67:0/1810234147 >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 0x7ff0d40b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:14:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.050+0000 7ff0d2a00640 1 --2- 172.21.15.67:0/1810234147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cf3a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009d90 tx=0x7ff0bc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f48f0d47e59b9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.051+0000 7ff0d2000640 1 -- 172.21.15.67:0/1810234147 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc039070 con 0x7ff0d40d33e0 2024-04-17T07:14:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.051+0000 7ff0d2000640 1 -- 172.21.15.67:0/1810234147 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0bc02cbc0 con 0x7ff0d40d33e0 2024-04-17T07:14:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.051+0000 7ff0d2000640 1 -- 172.21.15.67:0/1810234147 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc034440 con 0x7ff0d40d33e0 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 -- 172.21.15.67:0/1810234147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 msgr2=0x7ff0d40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/1810234147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cf3a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009d90 tx=0x7ff0bc02be10 comp rx=0 tx=0).stop 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 -- 172.21.15.67:0/1810234147 shutdown_connections 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/1810234147 >> v2:172.21.15.187:3300/0 conn(0x7ff0d40cf8e0 0x7ff0d40b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/1810234147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cf3a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 -- 172.21.15.67:0/1810234147 >> 172.21.15.67:0/1810234147 conn(0x7ff0d408bff0 msgr2=0x7ff0d40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 -- 172.21.15.67:0/1810234147 shutdown_connections 2024-04-17T07:14:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.052+0000 7ff0d8e00640 1 -- 172.21.15.67:0/1810234147 wait complete. 2024-04-17T07:14:13.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.054+0000 7ff0d8e00640 1 Processor -- start 2024-04-17T07:14:13.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.054+0000 7ff0d8e00640 1 -- start start 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0d415ecb0 0x7ff0d40cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d415f090 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d40cfd30 con 0x7ff0d415ecb0 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d8e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d415f210 con 0x7ff0d40d33e0 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42366/0 (socket says 172.21.15.67:42366) 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d3400640 1 -- 172.21.15.67:0/770418802 learned_addr learned my addr 172.21.15.67:0/770418802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d3e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0d415ecb0 0x7ff0d40cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.056+0000 7ff0d2a00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 -- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 msgr2=0x7ff0d40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 -- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0d415ecb0 msgr2=0x7ff0d40cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0d415ecb0 0x7ff0d40cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 -- 172.21.15.67:0/770418802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d40cc320 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d3400640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c410 tx=0x7ff0c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4017070 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c4005ce0 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40072f0 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d40cc600 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.057+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d40ccbc0 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.058+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d415e7f0 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.059+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0c401a070 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.061+0000 7ff0cb400640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.061+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff0c408afa0 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.061+0000 7ff0d2a00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.063+0000 7ff0d2a00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 0x7ff0b006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc036000 tx=0x7ff0bc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.065+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff0c4059490 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.272+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff0d4068a10 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.273+0000 7ff0cb400640 1 -- 172.21.15.67:0/770418802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7ff0c4058d90 con 0x7ff0d40cf8e0 2024-04-17T07:14:13.273 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 msgr2=0x7ff0b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 0x7ff0b006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc036000 tx=0x7ff0bc030000 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 msgr2=0x7ff0d40ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400c410 tx=0x7ff0c400c8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 shutdown_connections 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0d415ecb0 0x7ff0d40cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0d40d33e0 0x7ff0d40cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 --2- 172.21.15.67:0/770418802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0d40cf8e0 0x7ff0d40ca560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.276+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 >> 172.21.15.67:0/770418802 conn(0x7ff0d408bff0 msgr2=0x7ff0d406ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.277+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 shutdown_connections 2024-04-17T07:14:13.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.277+0000 7ff0d8e00640 1 -- 172.21.15.67:0/770418802 wait complete. 2024-04-17T07:14:13.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:13.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.515+0000 7ff573200640 1 Processor -- start 2024-04-17T07:14:13.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.516+0000 7ff573200640 1 -- start start 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.517+0000 7ff573200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff574144fb0 0x7ff5740b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.517+0000 7ff573200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 0x7ff5740cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.517+0000 7ff573200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57415f240 con 0x7ff57415fcb0 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.517+0000 7ff573200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff574160140 con 0x7ff574144fb0 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.517+0000 7ff573200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff5740cffb0 con 0x7ff5740cf620 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff571400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff57415fcb0 0x7ff5740b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44902/0 (socket says 172.21.15.67:44902) 2024-04-17T07:14:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff571400640 1 -- 172.21.15.67:0/255755843 learned_addr learned my addr 172.21.15.67:0/255755843 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:13.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff571e00640 1 --2- 172.21.15.67:0/255755843 >> v2:172.21.15.187:3300/0 conn(0x7ff574144fb0 0x7ff5740b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff572800640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 0x7ff5740cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309594574 0 0) 0x7ff57415f240 con 0x7ff57415fcb0 2024-04-17T07:14:13.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.518+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff558003520 con 0x7ff57415fcb0 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 345714842 0 0) 0x7ff558003520 con 0x7ff57415fcb0 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 msgr2=0x7ff5740cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 0x7ff5740cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 >> v2:172.21.15.187:3300/0 conn(0x7ff574144fb0 msgr2=0x7ff5740b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 --2- 172.21.15.67:0/255755843 >> v2:172.21.15.187:3300/0 conn(0x7ff574144fb0 0x7ff5740b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5740b6420 con 0x7ff57415fcb0 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff55c001670 con 0x7ff57415fcb0 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 >> v1:172.21.15.67:6789/0 conn(0x7ff57415fcb0 legacy=0x7ff5740b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 0x7ff5580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff558003f30 0x7ff558004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff558004d40 0x7ff5580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740cffb0 con 0x7ff558004d40 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff55800b9c0 con 0x7ff558003f30 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff55800bb90 con 0x7ff5740cf620 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff572800640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff558004d40 0x7ff5580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff571400640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff558003f30 0x7ff558004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff572800640 1 -- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 msgr2=0x7ff5580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff572800640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5740cf620 0x7ff5580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff572800640 1 -- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff558003f30 msgr2=0x7ff558004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.519+0000 7ff572800640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff558003f30 0x7ff558004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.520+0000 7ff572800640 1 -- 172.21.15.67:0/255755843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff558003520 con 0x7ff558004d40 2024-04-17T07:14:13.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.521+0000 7ff572800640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff558004d40 0x7ff5580051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00e170 tx=0x7ff558005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6454da7fec581683 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.521+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c039070 con 0x7ff558004d40 2024-04-17T07:14:13.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.521+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56c02cc20 con 0x7ff558004d40 2024-04-17T07:14:13.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.521+0000 7ff570a00640 1 -- 172.21.15.67:0/255755843 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c009c10 con 0x7ff558004d40 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 -- 172.21.15.67:0/255755843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff558004d40 msgr2=0x7ff5580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 --2- 172.21.15.67:0/255755843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff558004d40 0x7ff5580051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00e170 tx=0x7ff558005830 comp rx=0 tx=0).stop 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff572800640 1 -- 172.21.15.67:0/255755843 reap_dead start 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 -- 172.21.15.67:0/255755843 shutdown_connections 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 -- 172.21.15.67:0/255755843 >> 172.21.15.67:0/255755843 conn(0x7ff5740b23c0 msgr2=0x7ff5740b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 -- 172.21.15.67:0/255755843 shutdown_connections 2024-04-17T07:14:13.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.522+0000 7ff573200640 1 -- 172.21.15.67:0/255755843 wait complete. 2024-04-17T07:14:13.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.524+0000 7ff573200640 1 Processor -- start 2024-04-17T07:14:13.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.525+0000 7ff573200640 1 -- start start 2024-04-17T07:14:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff574134380 0x7ff574132160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff574134800 0x7ff5741326a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff57415f790 con 0x7ff57415c090 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff5740b61c0 con 0x7ff574134800 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff573200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff5740b6700 con 0x7ff574134380 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff571e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff571e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47942/0 (socket says 172.21.15.67:47942) 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.526+0000 7ff571e00640 1 -- 172.21.15.67:0/2905179093 learned_addr learned my addr 172.21.15.67:0/2905179093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 -- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff574134380 msgr2=0x7ff574132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff574134380 0x7ff574132160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 -- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff574134800 msgr2=0x7ff5741326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff574134800 0x7ff5741326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 -- 172.21.15.67:0/2905179093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57415cb00 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff571e00640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff56400e7f0 tx=0x7ff56400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564017070 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56400c860 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56400cc50 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.527+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57415cde0 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.528+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57415d3a0 con 0x7ff57415c090 2024-04-17T07:14:13.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.529+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff574068a10 con 0x7ff57415c090 2024-04-17T07:14:13.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.530+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff56401a070 con 0x7ff57415c090 2024-04-17T07:14:13.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.531+0000 7ff56a000640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 0x7ff55006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.531+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff56408b960 con 0x7ff57415c090 2024-04-17T07:14:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.531+0000 7ff571400640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 0x7ff55006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.532+0000 7ff571400640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 0x7ff55006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff55c001050 tx=0x7ff55c0013b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.536+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff564059d80 con 0x7ff57415c090 2024-04-17T07:14:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.748+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff574068610 con 0x7ff57415c090 2024-04-17T07:14:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.749+0000 7ff56a000640 1 -- 172.21.15.67:0/2905179093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7ff564059680 con 0x7ff57415c090 2024-04-17T07:14:13.750 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.752+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 msgr2=0x7ff55006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.752+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 0x7ff55006edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff55c001050 tx=0x7ff55c0013b0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.752+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 msgr2=0x7ff57415c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.752+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff56400e7f0 tx=0x7ff56400ecb0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 shutdown_connections 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff574134800 0x7ff5741326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55006c8f0 0x7ff55006edb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff574134380 0x7ff574132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 --2- 172.21.15.67:0/2905179093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57415c090 0x7ff57415c470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 >> 172.21.15.67:0/2905179093 conn(0x7ff5740b23c0 msgr2=0x7ff5740b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 shutdown_connections 2024-04-17T07:14:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.753+0000 7ff573200640 1 -- 172.21.15.67:0/2905179093 wait complete. 2024-04-17T07:14:13.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:13.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.980+0000 7fb197e00640 1 Processor -- start 2024-04-17T07:14:13.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.980+0000 7fb197e00640 1 -- start start 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.982+0000 7fb197e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb198072eb0 0x7fb19815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.982+0000 7fb197e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb19806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.982+0000 7fb197e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1980cd920 con 0x7fb19806bd20 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.982+0000 7fb197e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb1980cdc30 con 0x7fb198072eb0 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.982+0000 7fb197e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb19815b1c0 con 0x7fb19806b8c0 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb198072eb0 0x7fb19815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb198072eb0 0x7fb19815ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47948/0 (socket says 172.21.15.67:47948) 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 -- 172.21.15.67:0/1355695089 learned_addr learned my addr 172.21.15.67:0/1355695089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:13.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196000640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb19806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 -- 172.21.15.67:0/1355695089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 msgr2=0x7fb19806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb19806f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 -- 172.21.15.67:0/1355695089 >> v1:172.21.15.187:6789/0 conn(0x7fb19806bd20 legacy=0x7fb198071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.983+0000 7fb196a00640 1 -- 172.21.15.67:0/1355695089 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19806f980 con 0x7fb198072eb0 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.984+0000 7fb196a00640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198072eb0 0x7fb19815ac80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb188009870 tx=0x7fb18802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0dbeef98b12394 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb195600640 1 -- 172.21.15.67:0/1355695089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188035020 con 0x7fb198072eb0 2024-04-17T07:14:13.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb195600640 1 -- 172.21.15.67:0/1355695089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb188031640 con 0x7fb198072eb0 2024-04-17T07:14:13.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb195600640 1 -- 172.21.15.67:0/1355695089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18802d040 con 0x7fb198072eb0 2024-04-17T07:14:13.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 -- 172.21.15.67:0/1355695089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198072eb0 msgr2=0x7fb19815ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198072eb0 0x7fb19815ac80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb188009870 tx=0x7fb18802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 -- 172.21.15.67:0/1355695089 shutdown_connections 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb19806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 --2- 172.21.15.67:0/1355695089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb198072eb0 0x7fb19815ac80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 -- 172.21.15.67:0/1355695089 >> 172.21.15.67:0/1355695089 conn(0x7fb198067d10 msgr2=0x7fb1980b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 -- 172.21.15.67:0/1355695089 shutdown_connections 2024-04-17T07:14:13.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.985+0000 7fb197e00640 1 -- 172.21.15.67:0/1355695089 wait complete. 2024-04-17T07:14:13.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.987+0000 7fb197e00640 1 Processor -- start 2024-04-17T07:14:13.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.987+0000 7fb197e00640 1 -- start start 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb1980b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb198072eb0 0x7fb1981f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb19806f550 con 0x7fb19806bd20 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb19806ff40 con 0x7fb198072eb0 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.988+0000 7fb197e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb19806f6d0 con 0x7fb19806b8c0 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47964/0 (socket says 172.21.15.67:47964) 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 -- 172.21.15.67:0/988270633 learned_addr learned my addr 172.21.15.67:0/988270633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:13.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196a00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb1980b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb197400640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb198072eb0 0x7fb1981f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 -- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 msgr2=0x7fb1980b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb1980b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 -- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb198072eb0 msgr2=0x7fb1981f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb198072eb0 0x7fb1981f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 -- 172.21.15.67:0/988270633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1981f82d0 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.989+0000 7fb196000640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb18400d800 tx=0x7fb18400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.990+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18400b660 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.990+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb184019070 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.990+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18400ba60 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.990+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1981f85b0 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.990+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb198144660 con 0x7fb19806bd20 2024-04-17T07:14:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.991+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1980afe90 con 0x7fb19806bd20 2024-04-17T07:14:13.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.992+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb18400bc00 con 0x7fb19806bd20 2024-04-17T07:14:13.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.993+0000 7fb18ea00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 0x7fb17806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:13.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.993+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb18408bed0 con 0x7fb19806bd20 2024-04-17T07:14:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.994+0000 7fb196a00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 0x7fb17806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.995+0000 7fb196a00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 0x7fb17806ede0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb188036000 tx=0x7fb18803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:13.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:13.998+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb1980afe90 con 0x7fb19806bd20 2024-04-17T07:14:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.251+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb198068a10 con 0x7fb19806bd20 2024-04-17T07:14:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.252+0000 7fb18ea00640 1 -- 172.21.15.67:0/988270633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7fb18405a390 con 0x7fb19806bd20 2024-04-17T07:14:14.252 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:14.252 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 msgr2=0x7fb17806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 0x7fb17806ede0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb188036000 tx=0x7fb18803c040 comp rx=0 tx=0).stop 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 msgr2=0x7fb1981f7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb18400d800 tx=0x7fb18400dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 shutdown_connections 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb198072eb0 0x7fb1981f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19806bd20 0x7fb1981f7bf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb17806c920 0x7fb17806ede0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 --2- 172.21.15.67:0/988270633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19806b8c0 0x7fb1980b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 >> 172.21.15.67:0/988270633 conn(0x7fb198067d10 msgr2=0x7fb1980d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 shutdown_connections 2024-04-17T07:14:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.255+0000 7fb197e00640 1 -- 172.21.15.67:0/988270633 wait complete. 2024-04-17T07:14:14.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.498+0000 7f70bb200640 1 Processor -- start 2024-04-17T07:14:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.498+0000 7f70bb200640 1 -- start start 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.499+0000 7f70bb200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70b40cd020 0x7f70b40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.499+0000 7f70bb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f70b406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70bb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b40ceb00 con 0x7f70b406f670 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70bb200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f70b40cede0 con 0x7f70b40cd020 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70bb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70b40cf0c0 con 0x7f70b406e020 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f70b406f670 0x7f70b406fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44912/0 (socket says 172.21.15.67:44912) 2024-04-17T07:14:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b3e00640 1 -- 172.21.15.67:0/3769091336 learned_addr learned my addr 172.21.15.67:0/3769091336 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b9200640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f70b406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b3400640 1 --2- 172.21.15.67:0/3769091336 >> v2:172.21.15.187:3300/0 conn(0x7f70b40cd020 0x7f70b40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363857455 0 0) 0x7f70b40ceb00 con 0x7f70b406f670 2024-04-17T07:14:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.500+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7098003520 con 0x7f70b406f670 2024-04-17T07:14:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 829094147 0 0) 0x7f7098003520 con 0x7f70b406f670 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 msgr2=0x7f70b406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f70b406e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 >> v2:172.21.15.187:3300/0 conn(0x7f70b40cd020 msgr2=0x7f70b40cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 --2- 172.21.15.67:0/3769091336 >> v2:172.21.15.187:3300/0 conn(0x7f70b40cd020 0x7f70b40cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b40cf3a0 con 0x7f70b406f670 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f70a4001670 con 0x7f70b406f670 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 >> v1:172.21.15.67:6789/0 conn(0x7f70b406f670 legacy=0x7f70b406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70980039b0 0x7f7098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f7098004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7098004d40 0x7f70980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70b40cede0 con 0x7f70980039b0 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70b40cf0c0 con 0x7f7098004d40 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f709800b9c0 con 0x7f70b406e020 2024-04-17T07:14:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.501+0000 7f70b3e00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70980039b0 0x7f7098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3400640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f7098004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3e00640 1 -- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 msgr2=0x7f7098004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3e00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70b406e020 0x7f7098004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3e00640 1 -- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7098004d40 msgr2=0x7f70980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b9200640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7098004d40 0x7f70980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3e00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7098004d40 0x7f70980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.502+0000 7f70b3e00640 1 -- 172.21.15.67:0/3769091336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7098003520 con 0x7f70980039b0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.503+0000 7f70b3e00640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70980039b0 0x7f7098003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c190 tx=0x7f7098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435de5298569c632 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.503+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4007c80 con 0x7f70980039b0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.503+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a4030070 con 0x7f70980039b0 2024-04-17T07:14:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.503+0000 7f70b2a00640 1 -- 172.21.15.67:0/3769091336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4005db0 con 0x7f70980039b0 2024-04-17T07:14:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 -- 172.21.15.67:0/3769091336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70980039b0 msgr2=0x7f7098003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 --2- 172.21.15.67:0/3769091336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70980039b0 0x7f7098003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c190 tx=0x7f7098005830 comp rx=0 tx=0).stop 2024-04-17T07:14:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70b9200640 1 -- 172.21.15.67:0/3769091336 reap_dead start 2024-04-17T07:14:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 -- 172.21.15.67:0/3769091336 shutdown_connections 2024-04-17T07:14:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 -- 172.21.15.67:0/3769091336 >> 172.21.15.67:0/3769091336 conn(0x7f70b406bf50 msgr2=0x7f70b40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 -- 172.21.15.67:0/3769091336 shutdown_connections 2024-04-17T07:14:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.504+0000 7f70bb200640 1 -- 172.21.15.67:0/3769091336 wait complete. 2024-04-17T07:14:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.506+0000 7f70bb200640 1 Processor -- start 2024-04-17T07:14:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.506+0000 7f70bb200640 1 -- start start 2024-04-17T07:14:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70980039b0 0x7f70b414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b40afa60 0x7f70b40b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70b40ce8a0 con 0x7f70b40b4fe0 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70b40ce720 con 0x7f70b40afa60 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70bb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70b415cb00 con 0x7f70980039b0 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70b9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70b9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47990/0 (socket says 172.21.15.67:47990) 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.507+0000 7f70b9200640 1 -- 172.21.15.67:0/19605871 learned_addr learned my addr 172.21.15.67:0/19605871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b3e00640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70980039b0 0x7f70b414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b3400640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b40afa60 0x7f70b40b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 -- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70980039b0 msgr2=0x7f70b414ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70980039b0 0x7f70b414ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 -- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b40afa60 msgr2=0x7f70b40b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b40afa60 0x7f70b40b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 -- 172.21.15.67:0/19605871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b40b5580 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70b9200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70b40ce670 tx=0x7f70ac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac01c070 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ac017400 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0177f0 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b40b5860 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.508+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b40b5e20 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.510+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b40ad990 con 0x7f70b40b4fe0 2024-04-17T07:14:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.511+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f70ac01f070 con 0x7f70b40b4fe0 2024-04-17T07:14:14.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.511+0000 7f70abe00640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 0x7f709006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.512+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f70ac08bc60 con 0x7f70b40b4fe0 2024-04-17T07:14:14.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.512+0000 7f70b3e00640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 0x7f709006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.513+0000 7f70b3e00640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 0x7f709006edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70a4034000 tx=0x7f70a4007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.516+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f70b40ad990 con 0x7f70b40b4fe0 2024-04-17T07:14:14.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.727+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70b415ada0 con 0x7f70b40b4fe0 2024-04-17T07:14:14.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.729+0000 7f70abe00640 1 -- 172.21.15.67:0/19605871 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f70ac012070 con 0x7f70b40b4fe0 2024-04-17T07:14:14.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:14.730 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 msgr2=0x7f709006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 0x7f709006edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70a4034000 tx=0x7f70a4007000 comp rx=0 tx=0).stop 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 msgr2=0x7f70b40c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70b40ce670 tx=0x7f70ac00f4a0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 shutdown_connections 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b40b4fe0 0x7f70b40c9560 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70b40afa60 0x7f70b40b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f709006c8f0 0x7f709006edb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 --2- 172.21.15.67:0/19605871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70980039b0 0x7f70b414ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 >> 172.21.15.67:0/19605871 conn(0x7f70b406bf50 msgr2=0x7f70b415b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:14.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 shutdown_connections 2024-04-17T07:14:14.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.732+0000 7f70bb200640 1 -- 172.21.15.67:0/19605871 wait complete. 2024-04-17T07:14:14.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.964+0000 7fa0cfe00640 1 Processor -- start 2024-04-17T07:14:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.964+0000 7fa0cfe00640 1 -- start start 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0d0160280 0x7fa0d006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0141ed0 0x7fa0d0142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cfe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00cd4a0 con 0x7fa0d006b8c0 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d015ce30 con 0x7fa0d0160280 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d01607f0 con 0x7fa0d0141ed0 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0ce000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0d0160280 0x7fa0d006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0d006b8c0 0x7fa0d00cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44926/0 (socket says 172.21.15.67:44926) 2024-04-17T07:14:14.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.966+0000 7fa0cea00640 1 -- 172.21.15.67:0/3811747603 learned_addr learned my addr 172.21.15.67:0/3811747603 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:14.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0cf400640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0141ed0 0x7fa0d0142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0ce000640 1 -- 172.21.15.67:0/3811747603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0141ed0 msgr2=0x7fa0d0142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:14.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0ce000640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0141ed0 0x7fa0d0142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0ce000640 1 -- 172.21.15.67:0/3811747603 >> v1:172.21.15.67:6789/0 conn(0x7fa0d006b8c0 legacy=0x7fa0d00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0ce000640 1 -- 172.21.15.67:0/3811747603 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d0160ad0 con 0x7fa0d0160280 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.967+0000 7fa0cd600640 1 -- 172.21.15.67:0/3811747603 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212645895 0 0) 0x7fa0d00cd4a0 con 0x7fa0d006b8c0 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.968+0000 7fa0ce000640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d0160280 0x7fa0d006bca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0097a0 tx=0x7fa0bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13ce610c557a4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.968+0000 7fa0cd600640 1 -- 172.21.15.67:0/3811747603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc035020 con 0x7fa0d0160280 2024-04-17T07:14:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.968+0000 7fa0cd600640 1 -- 172.21.15.67:0/3811747603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc031830 con 0x7fa0d0160280 2024-04-17T07:14:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.968+0000 7fa0cd600640 1 -- 172.21.15.67:0/3811747603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc031c00 con 0x7fa0d0160280 2024-04-17T07:14:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 -- 172.21.15.67:0/3811747603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d0160280 msgr2=0x7fa0d006bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d0160280 0x7fa0d006bca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0097a0 tx=0x7fa0bc02b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 -- 172.21.15.67:0/3811747603 shutdown_connections 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0141ed0 0x7fa0d0142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/3811747603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d0160280 0x7fa0d006bca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 -- 172.21.15.67:0/3811747603 >> 172.21.15.67:0/3811747603 conn(0x7fa0d0067d10 msgr2=0x7fa0d00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 -- 172.21.15.67:0/3811747603 shutdown_connections 2024-04-17T07:14:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.969+0000 7fa0cfe00640 1 -- 172.21.15.67:0/3811747603 wait complete. 2024-04-17T07:14:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.971+0000 7fa0cfe00640 1 Processor -- start 2024-04-17T07:14:14.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.971+0000 7fa0cfe00640 1 -- start start 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0d0141ed0 0x7fa0d00b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0160280 0x7fa0d0155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d0161090 con 0x7fa0d006b8c0 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d006c1e0 con 0x7fa0d0141ed0 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d013f9b0 con 0x7fa0d0160280 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48006/0 (socket says 172.21.15.67:48006) 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.972+0000 7fa0cea00640 1 -- 172.21.15.67:0/2913879270 learned_addr learned my addr 172.21.15.67:0/2913879270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cf400640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0160280 0x7fa0d0155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 -- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0160280 msgr2=0x7fa0d0155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0160280 0x7fa0d0155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 -- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0d0141ed0 msgr2=0x7fa0d00b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0d0141ed0 0x7fa0d00b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 -- 172.21.15.67:0/2913879270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00b3ed0 con 0x7fa0d006b8c0 2024-04-17T07:14:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cea00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c4d0 tx=0x7fa0c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0018070 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c0007500 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0004a20 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.973+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d00b4170 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.974+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d00b4730 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.975+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d00b4e40 con 0x7fa0d006b8c0 2024-04-17T07:14:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.976+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0c001b070 con 0x7fa0d006b8c0 2024-04-17T07:14:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.977+0000 7fa0c6a00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.977+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa0c008aef0 con 0x7fa0d006b8c0 2024-04-17T07:14:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.977+0000 7fa0ce000640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.979+0000 7fa0ce000640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 0x7fa0b006edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc036000 tx=0x7fa0bc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:14.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:14.981+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa0c0059280 con 0x7fa0d006b8c0 2024-04-17T07:14:15.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.230+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0d00b0910 con 0x7fa0d006b8c0 2024-04-17T07:14:15.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.231+0000 7fa0c6a00640 1 -- 172.21.15.67:0/2913879270 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7fa0c0058b80 con 0x7fa0d006b8c0 2024-04-17T07:14:15.231 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:15.232 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:15.232 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 msgr2=0x7fa0b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 0x7fa0b006edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc036000 tx=0x7fa0bc030000 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 msgr2=0x7fa0d015e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c4d0 tx=0x7fa0c000c990 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 shutdown_connections 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa0b006c8f0 0x7fa0b006edb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0d0160280 0x7fa0d0155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0d0141ed0 0x7fa0d00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 --2- 172.21.15.67:0/2913879270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0d006b8c0 0x7fa0d015e790 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.234+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 >> 172.21.15.67:0/2913879270 conn(0x7fa0d0067d10 msgr2=0x7fa0d00d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.235+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 shutdown_connections 2024-04-17T07:14:15.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.235+0000 7fa0cfe00640 1 -- 172.21.15.67:0/2913879270 wait complete. 2024-04-17T07:14:15.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:13:57.651408+0000', 'modified': '2024-04-17T07:14:00.734636+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/4066505698', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 4066505698}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 4066505698}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4254, 'qdb_cluster': [4254]} 2024-04-17T07:14:15.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:14:15.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T07:14:15.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.886+0000 7fcf85000640 1 Processor -- start 2024-04-17T07:14:15.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.886+0000 7fcf85000640 1 -- start start 2024-04-17T07:14:15.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf85000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcf800cbc00 0x7fcf800cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf85000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8015d060 con 0x7fcf80072570 2024-04-17T07:14:15.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf85000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8015d340 con 0x7fcf800b63c0 2024-04-17T07:14:15.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf85000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcf800cfe90 con 0x7fcf800cbc00 2024-04-17T07:14:15.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf7fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf80072570 0x7fcf800ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44930/0 (socket says 172.21.15.67:44930) 2024-04-17T07:14:15.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.888+0000 7fcf7fe00640 1 -- 172.21.15.67:0/2758149465 learned_addr learned my addr 172.21.15.67:0/2758149465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:15.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2758149465 >> v2:172.21.15.187:3300/0 conn(0x7fcf800cbc00 0x7fcf800cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375558525 0 0) 0x7fcf8015d060 con 0x7fcf80072570 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf64003520 con 0x7fcf80072570 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645111069 0 0) 0x7fcf64003520 con 0x7fcf80072570 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 >> v2:172.21.15.187:3300/0 conn(0x7fcf800cbc00 msgr2=0x7fcf800cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 --2- 172.21.15.67:0/2758149465 >> v2:172.21.15.187:3300/0 conn(0x7fcf800cbc00 0x7fcf800cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 >> v1:172.21.15.187:6789/0 conn(0x7fcf800b63c0 legacy=0x7fcf80071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8015d620 con 0x7fcf80072570 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcf780019f0 con 0x7fcf80072570 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.889+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 >> v1:172.21.15.67:6789/0 conn(0x7fcf80072570 legacy=0x7fcf800ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 0x7fcf64003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64004800 0x7fcf64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf64005210 0x7fcf6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf800cfe90 con 0x7fcf64003a70 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6400c140 con 0x7fcf64004800 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf6400c340 con 0x7fcf64005210 2024-04-17T07:14:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 0x7fcf64003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 -- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf64005210 msgr2=0x7fcf6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf64005210 0x7fcf6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 -- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64004800 msgr2=0x7fcf64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64004800 0x7fcf64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7f400640 1 -- 172.21.15.67:0/2758149465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf64003520 con 0x7fcf64003a70 2024-04-17T07:14:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.890+0000 7fcf7fe00640 1 -- 172.21.15.67:0/2758149465 reap_dead start 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.891+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 0x7fcf64003e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ac0 tx=0x7fcf64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cb0198388141ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70039070 con 0x7fcf64003a70 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf7002c6d0 con 0x7fcf64003a70 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf7ea00640 1 -- 172.21.15.67:0/2758149465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70009690 con 0x7fcf64003a70 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 -- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 msgr2=0x7fcf64003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 0x7fcf64003e90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ac0 tx=0x7fcf64005830 comp rx=0 tx=0).stop 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 -- 172.21.15.67:0/2758149465 shutdown_connections 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 --2- 172.21.15.67:0/2758149465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf64003a70 0x7fcf64003e90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 -- 172.21.15.67:0/2758149465 >> 172.21.15.67:0/2758149465 conn(0x7fcf8006bef0 msgr2=0x7fcf800b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 -- 172.21.15.67:0/2758149465 shutdown_connections 2024-04-17T07:14:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.892+0000 7fcf85000640 1 -- 172.21.15.67:0/2758149465 wait complete. 2024-04-17T07:14:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.894+0000 7fcf85000640 1 Processor -- start 2024-04-17T07:14:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.894+0000 7fcf85000640 1 -- start start 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 0x7fcf8014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf80158c40 0x7fcf801590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf800d02d0 con 0x7fcf801e6d50 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8015dc70 con 0x7fcf64003a70 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.895+0000 7fcf85000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf800cbfe0 con 0x7fcf80158c40 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48030/0 (socket says 172.21.15.67:48030) 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 -- 172.21.15.67:0/2450197751 learned_addr learned my addr 172.21.15.67:0/2450197751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 0x7fcf8014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf80158c40 0x7fcf801590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 -- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf80158c40 msgr2=0x7fcf801590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf80158c40 0x7fcf801590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 -- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 msgr2=0x7fcf8014ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 0x7fcf8014ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf77e00640 1 -- 172.21.15.67:0/2450197751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8015b8d0 con 0x7fcf801e6d50 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.896+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 0x7fcf8014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf77e00640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c3f0 tx=0x7fcf6800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf77400640 1 -- 172.21.15.67:0/2450197751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68017070 con 0x7fcf801e6d50 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf77400640 1 -- 172.21.15.67:0/2450197751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf680052f0 con 0x7fcf801e6d50 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf77400640 1 -- 172.21.15.67:0/2450197751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68003830 con 0x7fcf801e6d50 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8015bbb0 con 0x7fcf801e6d50 2024-04-17T07:14:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.897+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8015c170 con 0x7fcf801e6d50 2024-04-17T07:14:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.900+0000 7fcf77400640 1 -- 172.21.15.67:0/2450197751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcf6801a070 con 0x7fcf801e6d50 2024-04-17T07:14:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.901+0000 7fcf77400640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 0x7fcf5c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.901+0000 7fcf77400640 1 -- 172.21.15.67:0/2450197751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcf6808af40 con 0x7fcf801e6d50 2024-04-17T07:14:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.902+0000 7fcf89432740 1 Processor -- start 2024-04-17T07:14:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.902+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 0x7fcf5c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.902+0000 7fcf89432740 1 -- start start 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b3c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b66db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5572a2aa2980 con 0x5572a2b64180 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5572a2a49220 con 0x5572a2b66930 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5572a2a9e940 con 0x5572a2b82370 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf7f400640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 0x7fcf5c06ede0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf70036000 tx=0x7fcf70030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf7f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf7f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48038/0 (socket says 172.21.15.67:48038) 2024-04-17T07:14:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf7f400640 1 -- 172.21.15.67:0/3146343696 learned_addr learned my addr 172.21.15.67:0/3146343696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.903+0000 7fcf77e00640 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b3c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b66db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 -- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 msgr2=0x5572a2b3c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b3c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 -- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 msgr2=0x5572a2b66db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b66db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 -- 172.21.15.67:0/3146343696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a27b52d0 con 0x5572a2b64180 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009bb0 tx=0x7fcf7002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59c03faf525f2be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf6fe00640 1 -- 172.21.15.67:0/3146343696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70045030 con 0x5572a2b64180 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf6fe00640 1 -- 172.21.15.67:0/3146343696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf7003a040 con 0x5572a2b64180 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.904+0000 7fcf6fe00640 1 -- 172.21.15.67:0/3146343696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70039050 con 0x5572a2b64180 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 -- 172.21.15.67:0/3146343696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 msgr2=0x5572a2b82c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009bb0 tx=0x7fcf7002e000 comp rx=0 tx=0).stop 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 -- 172.21.15.67:0/3146343696 shutdown_connections 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b66db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b3c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 --2- 172.21.15.67:0/3146343696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b64180 0x5572a2b82c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 -- 172.21.15.67:0/3146343696 >> 172.21.15.67:0/3146343696 conn(0x5572a2b38db0 msgr2=0x5572a2b81930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 -- 172.21.15.67:0/3146343696 shutdown_connections 2024-04-17T07:14:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.905+0000 7fcf89432740 1 -- 172.21.15.67:0/3146343696 wait complete. 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.906+0000 7fcf89432740 1 -- start start 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.906+0000 7fcf89432740 1 Processor -- start 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b8def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b99890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5572a2b64560 con 0x5572a2b9a0e0 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5572a2a5c590 con 0x5572a2b66930 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.907+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5572a2b36b40 con 0x5572a2b82370 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf89432740 10 client.0 _ll_register_callbacks cb 0x5572a27b4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48054/0 (socket says 172.21.15.67:48054) 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 -- 172.21.15.67:0/230959945 learned_addr learned my addr 172.21.15.67:0/230959945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7f400640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b8def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf77e00640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b99890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 -- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 msgr2=0x5572a2b99890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b99890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 -- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 msgr2=0x5572a2b8def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b8def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 -- 172.21.15.67:0/230959945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a287f6b0 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.908+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015dbe0 tx=0x7fcf7800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cea2b420fd5a32a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78019060 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcf78010510 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf7003a040 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcf7801f050 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf89432740 10 client.4647 fetch_fsmap learned FSMap version 8 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5572a2b84390 con 0x5572a2b9a0e0 2024-04-17T07:14:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.909+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf7003a040 con 0x5572a2b9a0e0 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 8) ==== 3495+0+0 (secure 0 0 0) 0x7fcf78022020 con 0x5572a2b9a0e0 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 10 client.4647 fetch_fsmap finished waiting for FSMap version 8 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 10 client.4647 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 20 client.4647 populate_metadata read hostname 'smithi067' 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b872f0 0x5572a2b85520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 4 client.4647 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 --> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572a2ab03a0 con 0x5572a2b872f0 2024-04-17T07:14:15.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.910+0000 7fcf7f400640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b872f0 0x5572a2b85520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.911+0000 7fcf7f400640 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b872f0 0x5572a2b85520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=453ff6e53fd5dad3 server_cookie=fd7d6386f030b02d in_seq=0 out_seq=0 2024-04-17T07:14:15.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.911+0000 7fcf6cc00640 10 client.4647 ms_handle_connect on v2:172.21.15.67:6833/4066505698 2024-04-17T07:14:15.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.911+0000 7fcf6cc00640 1 -- 172.21.15.67:0/230959945 <== mds.0 v2:172.21.15.67:6833/4066505698 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572a2ab03a0 con 0x5572a2b872f0 2024-04-17T07:14:15.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.911+0000 7fcf6cc00640 10 client.4647 handle_command_reply: tid=0 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.911+0000 7fcf89432740 1 client.4647 shutdown 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 msgr2=0x5572a2b61ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015dbe0 tx=0x7fcf7800e820 comp rx=0 tx=0).stop 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 shutdown_connections 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b872f0 0x5572a2b85520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b61ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b82370 0x5572a2b99890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 --2- 172.21.15.67:0/230959945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b8def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf6cc00640 0 client.4647 ms_handle_reset on v2:172.21.15.67:6833/4066505698 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 >> 172.21.15.67:0/230959945 conn(0x5572a2b64180 msgr2=0x5572a2b65d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 shutdown_connections 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 1 -- 172.21.15.67:0/230959945 wait complete. 2024-04-17T07:14:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.912+0000 7fcf89432740 20 client.4647 trim_cache size 0 max 16384 2024-04-17T07:14:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.932+0000 7fcf89432740 1 Processor -- start 2024-04-17T07:14:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.932+0000 7fcf89432740 1 -- start start 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b63060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b9a0e0 0x5572a2b66140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8006f5b0 con 0x5572a2b82370 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf80150bd0 con 0x5572a2b66930 2024-04-17T07:14:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.933+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf80151220 con 0x5572a2b9a0e0 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48070/0 (socket says 172.21.15.67:48070) 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 -- 172.21.15.67:0/1985439822 learned_addr learned my addr 172.21.15.67:0/1985439822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf7f400640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b63060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b9a0e0 0x5572a2b66140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 -- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b9a0e0 msgr2=0x5572a2b66140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b9a0e0 0x5572a2b66140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 -- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 msgr2=0x5572a2b63060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b63060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 -- 172.21.15.67:0/1985439822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a2b3da40 con 0x5572a2b82370 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf7f400640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b63060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.934+0000 7fcf77e00640 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcf68061340 tx=0x7fcf68058de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b200bf905641150c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf6f400640 1 -- 172.21.15.67:0/1985439822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68086070 con 0x5572a2b82370 2024-04-17T07:14:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf6f400640 1 -- 172.21.15.67:0/1985439822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf68056f90 con 0x5572a2b82370 2024-04-17T07:14:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf6f400640 1 -- 172.21.15.67:0/1985439822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf680580e0 con 0x5572a2b82370 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 -- 172.21.15.67:0/1985439822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 msgr2=0x5572a2b65c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcf68061340 tx=0x7fcf68058de0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 -- 172.21.15.67:0/1985439822 shutdown_connections 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b9a0e0 0x5572a2b66140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b82370 0x5572a2b65c00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 --2- 172.21.15.67:0/1985439822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b66930 0x5572a2b63060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 -- 172.21.15.67:0/1985439822 >> 172.21.15.67:0/1985439822 conn(0x5572a2b64180 msgr2=0x5572a2b854f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.935+0000 7fcf89432740 1 -- 172.21.15.67:0/1985439822 shutdown_connections 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.936+0000 7fcf89432740 1 -- 172.21.15.67:0/1985439822 wait complete. 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.936+0000 7fcf89432740 1 -- start start 2024-04-17T07:14:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.936+0000 7fcf89432740 1 Processor -- start 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b66930 0x5572a2b69980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b82370 0x5572a2b981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5572a2b87170 con 0x5572a2b9a0e0 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5572a2b606c0 con 0x5572a2b82370 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5572a2b877c0 con 0x5572a2b66930 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.937+0000 7fcf89432740 10 client.0 _ll_register_callbacks cb 0x7fcf78001ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48080/0 (socket says 172.21.15.67:48080) 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 -- 172.21.15.67:0/3623613718 learned_addr learned my addr 172.21.15.67:0/3623613718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b66930 0x5572a2b69980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf77e00640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b82370 0x5572a2b981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 -- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b66930 msgr2=0x5572a2b69980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b66930 0x5572a2b69980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 -- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b82370 msgr2=0x5572a2b981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b82370 0x5572a2b981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 -- 172.21.15.67:0/3623613718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a2b61d60 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf7fe00640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015dbe0 tx=0x7fcf8015d230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b15b786f13d8cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.938+0000 7fcf6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78004b60 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcf78004d00 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf68056f90 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fcf78031050 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf89432740 10 client.4653 fetch_fsmap learned FSMap version 8 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5572a2b656f0 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf68056f90 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 8) ==== 3495+0+0 (secure 0 0 0) 0x7fcf78034020 con 0x5572a2b9a0e0 2024-04-17T07:14:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.939+0000 7fcf89432740 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf89432740 10 client.4653 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf89432740 20 client.4653 populate_metadata read hostname 'smithi067' 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b81140 0x5572a2b81520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf89432740 4 client.4653 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 --> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] -- command(tid 0: {"prefix": "status"}) -- 0x5572a2b88110 con 0x5572a2b81140 2024-04-17T07:14:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.940+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b81140 0x5572a2b81520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf7f400640 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b81140 0x5572a2b81520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=408fd5f0c40f31d7 server_cookie=f725124e691945d6 in_seq=0 out_seq=0 2024-04-17T07:14:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf6cc00640 10 client.4653 ms_handle_connect on v2:172.21.15.67:6833/4066505698 2024-04-17T07:14:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf6cc00640 1 -- 172.21.15.67:0/3623613718 <== mds.0 v2:172.21.15.67:6833/4066505698 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5572a2b88110 con 0x5572a2b81140 2024-04-17T07:14:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf6cc00640 10 client.4653 handle_command_reply: tid=0 2024-04-17T07:14:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 client.4653 shutdown 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 msgr2=0x5572a2b65c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcf8015dbe0 tx=0x7fcf8015d230 comp rx=0 tx=0).stop 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 shutdown_connections 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:6833/4066505698,v1:172.21.15.67:6835/4066505698] conn(0x5572a2b81140 0x5572a2b81520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572a2b9a0e0 0x5572a2b65c00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5572a2b82370 0x5572a2b981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 --2- 172.21.15.67:0/3623613718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5572a2b66930 0x5572a2b69980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 >> 172.21.15.67:0/3623613718 conn(0x5572a2b64180 msgr2=0x5572a2b62d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf6cc00640 0 client.4653 ms_handle_reset on v2:172.21.15.67:6833/4066505698 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 shutdown_connections 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 1 -- 172.21.15.67:0/3623613718 wait complete. 2024-04-17T07:14:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.941+0000 7fcf89432740 20 client.4653 trim_cache size 0 max 16384 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4254, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 17.734342143999999, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 8, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 22, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 22, 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 25.393148524000001 2024-04-17T07:14:15.945 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:14:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 msgr2=0x7fcf5c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 0x7fcf5c06ede0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf70036000 tx=0x7fcf70030000 comp rx=0 tx=0).stop 2024-04-17T07:14:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 msgr2=0x7fcf8015af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c3f0 tx=0x7fcf6800c8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 shutdown_connections 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcf5c06c920 0x7fcf5c06ede0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf80158c40 0x7fcf801590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf801e6d50 0x7fcf8015af80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 --2- 172.21.15.67:0/2450197751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf64003a70 0x7fcf8014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 >> 172.21.15.67:0/2450197751 conn(0x7fcf8006bef0 msgr2=0x7fcf800b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 shutdown_connections 2024-04-17T07:14:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:15.942+0000 7fcf85000640 1 -- 172.21.15.67:0/2450197751 wait complete. 2024-04-17T07:14:15.960 INFO:teuthology.run_tasks:Running task sequential... 2024-04-17T07:14:15.969 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-17T07:14:15.969 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-17T07:14:15.970 INFO:teuthology.task.print:Enabling mgr modules 2024-04-17T07:14:15.970 INFO:teuthology.run_tasks:Running task kclient... 2024-04-17T07:14:16.046 INFO:tasks.kclient:Mounting kernel clients... 2024-04-17T07:14:16.046 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-17T07:14:16.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.047 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:14:16.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T07:14:16.208 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-17T07:14:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:14:16.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.212 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:14:16.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T07:14:16.368 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-17T07:14:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:14:16.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.373 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T07:14:16.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-17T07:14:16.531 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-17T07:14:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:14:16.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.537 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T07:14:16.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:16.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-17T07:14:16.679 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-17T07:14:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T07:14:16.683 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-17T07:14:16.683 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T07:14:16.683 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:14:16.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:14:16.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.916+0000 7f1fece00640 1 Processor -- start 2024-04-17T07:14:16.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.917+0000 7f1fece00640 1 -- start start 2024-04-17T07:14:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.918+0000 7f1fece00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.918+0000 7f1fece00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe8162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.918+0000 7f1fece00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe80cc500 con 0x7f1fe8161e70 2024-04-17T07:14:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.918+0000 7f1fece00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe80c71e0 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.918+0000 7f1fece00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe80cc7a0 con 0x7f1fe80ce4a0 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48096/0 (socket says 172.21.15.67:48096) 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 -- 172.21.15.67:0/1554492705 learned_addr learned my addr 172.21.15.67:0/1554492705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe6800640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe8162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 -- 172.21.15.67:0/1554492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 msgr2=0x7f1fe8162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe8162fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 -- 172.21.15.67:0/1554492705 >> v1:172.21.15.187:6789/0 conn(0x7f1fe8161e70 legacy=0x7f1fe80cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.919+0000 7f1fe5400640 1 -- 172.21.15.67:0/1554492705 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe80cca80 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.921+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80c74c0 tx=0x7f1fd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e302eb2e1d34404a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:16.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.921+0000 7f1fe4a00640 1 -- 172.21.15.67:0/1554492705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0035020 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.921+0000 7f1fe4a00640 1 -- 172.21.15.67:0/1554492705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd0031830 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.921+0000 7f1fe4a00640 1 -- 172.21.15.67:0/1554492705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0031c00 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 -- 172.21.15.67:0/1554492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 msgr2=0x7f1fe80c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80c74c0 tx=0x7f1fd002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 -- 172.21.15.67:0/1554492705 shutdown_connections 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe8162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 --2- 172.21.15.67:0/1554492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c6c00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 -- 172.21.15.67:0/1554492705 >> 172.21.15.67:0/1554492705 conn(0x7f1fe806bf50 msgr2=0x7f1fe80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 -- 172.21.15.67:0/1554492705 shutdown_connections 2024-04-17T07:14:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.922+0000 7f1fece00640 1 -- 172.21.15.67:0/1554492705 wait complete. 2024-04-17T07:14:16.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.924+0000 7f1fece00640 1 Processor -- start 2024-04-17T07:14:16.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.924+0000 7f1fece00640 1 -- start start 2024-04-17T07:14:16.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe814ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe8161e70 0x7f1fe80c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe80c5630 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe80c54b0 con 0x7f1fe8161e70 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fece00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe80c57b0 con 0x7f1fe80ce4a0 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48100/0 (socket says 172.21.15.67:48100) 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 -- 172.21.15.67:0/1932080729 learned_addr learned my addr 172.21.15.67:0/1932080729 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe814ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 -- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 msgr2=0x7f1fe814ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe814ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe6800640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe8161e70 0x7f1fe80c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 -- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe8161e70 msgr2=0x7f1fe80c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:16.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.925+0000 7f1fe5e00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe8161e70 0x7f1fe80c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:16.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fe5e00640 1 -- 172.21.15.67:0/1932080729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe8156780 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe814ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:16.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fe5e00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800b420 tx=0x7f1fd800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:16.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8011830 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd8016070 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe8156a60 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8011c10 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.926+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe8157020 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.927+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe8068b90 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.929+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fd801b460 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.929+0000 7f1fde000640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 0x7f1fc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.930+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1fd808b680 con 0x7f1fe80cd5a0 2024-04-17T07:14:16.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.930+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 0x7f1fc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:16.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.931+0000 7f1fe5400640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 0x7f1fc406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0036000 tx=0x7f1fd0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:16.934+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1fd8020020 con 0x7f1fe80cd5a0 2024-04-17T07:14:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.180+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1fe8162e30 con 0x7f1fe80cd5a0 2024-04-17T07:14:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.181+0000 7f1fde000640 1 -- 172.21.15.67:0/1932080729 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f1fd8059ab0 con 0x7f1fe80cd5a0 2024-04-17T07:14:17.181 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.183+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 msgr2=0x7f1fc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.183+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 0x7f1fc406edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0036000 tx=0x7f1fd0008030 comp rx=0 tx=0).stop 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.183+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 msgr2=0x7f1fe80c9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.183+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800b420 tx=0x7f1fd800b8e0 comp rx=0 tx=0).stop 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 shutdown_connections 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe8161e70 0x7f1fe80c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1fc406c8f0 0x7f1fc406edb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe80ce4a0 0x7f1fe814ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 --2- 172.21.15.67:0/1932080729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fe80cd5a0 0x7f1fe80c9fd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 >> 172.21.15.67:0/1932080729 conn(0x7f1fe806bf50 msgr2=0x7f1fe80d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 shutdown_connections 2024-04-17T07:14:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17.184+0000 7f1fece00640 1 -- 172.21.15.67:0/1932080729 wait complete. 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:14:17.204 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:14:17.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:17.205 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:14:17.258 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:17.259 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42587sec preferred_lft 42587sec 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:14:17.260 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:17.260 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T07:14:17.261 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:17.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:17.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:17.796 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:14:17.878 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:14:17.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:17.882 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:14:17.947 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:14:17.947 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:14:17.947 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:14:17.947 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:14:17.947 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:14:17.948 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:18.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:18.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:18.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:18.764 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:14:18.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:18.812 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-17T07:14:18.857 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:18.857 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:18.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:14:18.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-17T07:14:18.857 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:19.101 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-17T07:14:19.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-17T07:14:19.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-17T07:14:19.102 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:19.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:19.434 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:19.434 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:19.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:14:19.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T07:14:19.434 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:19.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:19.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:19.666 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:19.666 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:14:19.708 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:14:19.709 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:19.709 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:14:19.792 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:14:19.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:19.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:19.974 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:19.974 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:20.759 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:20.759 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:20.759 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:14:20.759 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:14:20.759 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:14:20.764 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:14:20.764 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:14:20.764 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:14:20.764 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:14:20.764 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:14:20.770 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:14:20.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:20.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:14:20.875 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:21.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:21.028 INFO:teuthology.orchestra.run.smithi067.stdout:client4463 2024-04-17T07:14:21.028 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:21.029 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:21.082 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:21.210 INFO:teuthology.orchestra.run.smithi067.stdout:client4463 2024-04-17T07:14:21.211 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:21.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:client4463 2024-04-17T07:14:21.355 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.client4463/status') 2024-04-17T07:14:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4463 (3)172.21.15.67:0/718693124 2024-04-17T07:14:21.517 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-17T07:14:21.517 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-17T07:14:21.517 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-17T07:14:21.517 INFO:teuthology.orchestra.run.smithi067.stderr:69 bytes copied, 7.9014e-05 s, 873 kB/s 2024-04-17T07:14:21.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:21.531 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-17T07:14:21.531 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T07:14:21.531 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-17T07:14:21.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:14:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.749+0000 7f4950a00640 1 Processor -- start 2024-04-17T07:14:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.749+0000 7f4950a00640 1 -- start start 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.750+0000 7f4950a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f494c15b570 0x7f494c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.750+0000 7f4950a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f494c06c040 0x7f494c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.750+0000 7f4950a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f494c15bc80 con 0x7f494c06fce0 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.750+0000 7f4950a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f494c06c470 con 0x7f494c06c040 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.750+0000 7f4950a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f494c15bf40 con 0x7f494c15b570 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f494c06c040 0x7f494c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f494c06c040 0x7f494c15f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48106/0 (socket says 172.21.15.67:48106) 2024-04-17T07:14:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 -- 172.21.15.67:0/2756469034 learned_addr learned my addr 172.21.15.67:0/2756469034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494a800640 1 --2- 172.21.15.67:0/2756469034 >> v2:172.21.15.187:3300/0 conn(0x7f494c15b570 0x7f494c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 -- 172.21.15.67:0/2756469034 >> v2:172.21.15.187:3300/0 conn(0x7f494c15b570 msgr2=0x7f494c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 --2- 172.21.15.67:0/2756469034 >> v2:172.21.15.187:3300/0 conn(0x7f494c15b570 0x7f494c06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 -- 172.21.15.67:0/2756469034 >> v1:172.21.15.67:6789/0 conn(0x7f494c06fce0 legacy=0x7f494c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.751+0000 7f494b200640 1 -- 172.21.15.67:0/2756469034 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c15c220 con 0x7f494c06c040 2024-04-17T07:14:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.753+0000 7f494b200640 1 --2- 172.21.15.67:0/2756469034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c15f140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f494c06c5f0 tx=0x7f494402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee6b6d7be453a0a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.753+0000 7f4949400640 1 -- 172.21.15.67:0/2756469034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494402ea80 con 0x7f494c06c040 2024-04-17T07:14:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.753+0000 7f4949400640 1 -- 172.21.15.67:0/2756469034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494402ec20 con 0x7f494c06c040 2024-04-17T07:14:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.753+0000 7f4949400640 1 -- 172.21.15.67:0/2756469034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944009600 con 0x7f494c06c040 2024-04-17T07:14:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 -- 172.21.15.67:0/2756469034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 msgr2=0x7f494c15f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 --2- 172.21.15.67:0/2756469034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c15f140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f494c06c5f0 tx=0x7f494402e0e0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 -- 172.21.15.67:0/2756469034 shutdown_connections 2024-04-17T07:14:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 --2- 172.21.15.67:0/2756469034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c15f140 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 --2- 172.21.15.67:0/2756469034 >> v2:172.21.15.187:3300/0 conn(0x7f494c15b570 0x7f494c06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 -- 172.21.15.67:0/2756469034 >> 172.21.15.67:0/2756469034 conn(0x7f494c067d10 msgr2=0x7f494c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 -- 172.21.15.67:0/2756469034 shutdown_connections 2024-04-17T07:14:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.754+0000 7f4950a00640 1 -- 172.21.15.67:0/2756469034 wait complete. 2024-04-17T07:14:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.756+0000 7f4950a00640 1 Processor -- start 2024-04-17T07:14:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.756+0000 7f4950a00640 1 -- start start 2024-04-17T07:14:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 0x7f494c146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 0x7f494c0aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f494c0b47b0 con 0x7f494c06c040 2024-04-17T07:14:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f494c15b950 con 0x7f494c15b570 2024-04-17T07:14:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f4950a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f494c0b4930 con 0x7f494c06fce0 2024-04-17T07:14:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f494a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.757+0000 7f494a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48112/0 (socket says 172.21.15.67:48112) 2024-04-17T07:14:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 -- 172.21.15.67:0/1001837304 learned_addr learned my addr 172.21.15.67:0/1001837304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f4949e00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 0x7f494c146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494b200640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 0x7f494c0aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 -- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 msgr2=0x7f494c146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 0x7f494c146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 -- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 msgr2=0x7f494c0aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 0x7f494c0aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 -- 172.21.15.67:0/1001837304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c0b0c10 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494b200640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 0x7f494c0aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f4949e00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 0x7f494c146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.758+0000 7f494a800640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f493c00d620 tx=0x7f493c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.759+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c00b620 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.759+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f493c019070 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.759+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c00ba00 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.759+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494c0b0ef0 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.759+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f494c0b14b0 con 0x7f494c06c040 2024-04-17T07:14:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.760+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494c068a10 con 0x7f494c06c040 2024-04-17T07:14:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.761+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f493c00bba0 con 0x7f494c06c040 2024-04-17T07:14:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.762+0000 7f4942a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.762+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f493c08bf90 con 0x7f494c06c040 2024-04-17T07:14:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.763+0000 7f4949e00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.764+0000 7f4949e00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 0x7f492806edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f494c156d60 tx=0x7f4934009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:21.767+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f493c05a3b0 con 0x7f494c06c040 2024-04-17T07:14:22.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.016+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f494c0ce580 con 0x7f494c06c040 2024-04-17T07:14:22.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.017+0000 7f4942a00640 1 -- 172.21.15.67:0/1001837304 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f493c059cb0 con 0x7f494c06c040 2024-04-17T07:14:22.017 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:14:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 msgr2=0x7f492806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 0x7f492806edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f494c156d60 tx=0x7f4934009040 comp rx=0 tx=0).stop 2024-04-17T07:14:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 msgr2=0x7f494c151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f493c00d620 tx=0x7f493c00dae0 comp rx=0 tx=0).stop 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 shutdown_connections 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f494c15b570 0x7f494c0aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f492806c8f0 0x7f492806edb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f494c06fce0 0x7f494c146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 --2- 172.21.15.67:0/1001837304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f494c06c040 0x7f494c151380 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 >> 172.21.15.67:0/1001837304 conn(0x7f494c067d10 msgr2=0x7f494c0ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 shutdown_connections 2024-04-17T07:14:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22.019+0000 7f4950a00640 1 -- 172.21.15.67:0/1001837304 wait complete. 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:14:22.040 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-17T07:14:22.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.041 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:14:22.097 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:22.097 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42582sec preferred_lft 42582sec 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:14:22.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.100 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:14:22.192 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:14:22.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.197 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:14:22.248 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:14:22.249 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:14:22.249 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:14:22.249 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:14:22.249 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:14:22.250 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:22.250 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:22.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:14:22.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:14:22.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:14:22.251 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:22.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:22.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.511 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:14:22.564 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:14:22.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.565 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-17T07:14:22.620 INFO:teuthology.orchestra.run.smithi067.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-17T07:14:22.621 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:22.621 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:22.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:14:22.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-17T07:14:22.622 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:22.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:22.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.927 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:22.928 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:23:ad:6a:82:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T07:14:22.928 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T07:14:22.928 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.928 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b823:adff:fe6a:8203/64 scope link 2024-04-17T07:14:22.928 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:22.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:22.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-17T07:14:23.030 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:23:ad:6a:82:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b823:adff:fe6a:8203/64 scope link 2024-04-17T07:14:23.031 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:23.035 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-17T07:14:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-17T07:14:23.036 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:23.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:23.384 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:23.384 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:14:23.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.1 up 2024-04-17T07:14:23.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.1 master ceph-brx 2024-04-17T07:14:23.385 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:14:23.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:23.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:14:23.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:14:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:23.628 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:23.628 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-17T07:14:23.723 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-17T07:14:23.724 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:23.724 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-17T07:14:23.821 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-17T07:14:23.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:23.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:23.970 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:23.970 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:24.004 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:24.004 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:24.004 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:14:24.005 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 2@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:14:24.005 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:14:24.006 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-17T07:14:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:14:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:14:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:14:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:14:24.010 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:14:24.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:24.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-17T07:14:24.100 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:24.223 INFO:teuthology.orchestra.run.smithi067.stdout:client4466 2024-04-17T07:14:24.223 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:24.223 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:24.275 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:24.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:24.450 INFO:teuthology.orchestra.run.smithi067.stdout:client4466 2024-04-17T07:14:24.450 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:24.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:24.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:client4466 2024-04-17T07:14:24.575 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.client4466/status') 2024-04-17T07:14:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:24.727 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4466 (3)172.21.15.67:0/3736550727 2024-04-17T07:14:24.727 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-17T07:14:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-17T07:14:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-17T07:14:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 7.2363e-05 s, 967 kB/s 2024-04-17T07:14:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:24.740 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-17T07:14:24.741 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T07:14:24.741 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-17T07:14:24.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:14:24.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.962+0000 7f74e3e00640 1 Processor -- start 2024-04-17T07:14:24.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.962+0000 7f74e3e00640 1 -- start start 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.963+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74e40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.963+0000 7f74e3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f74e40d0700 0x7f74e40d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e409b8a0 con 0x7f74e40c6710 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f74e40cd7f0 con 0x7f74e40d0700 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74e409bb80 con 0x7f74e415fe10 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74e40c6710 0x7f74e40cdc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34454/0 (socket says 172.21.15.67:34454) 2024-04-17T07:14:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e2000640 1 -- 172.21.15.67:0/417989901 learned_addr learned my addr 172.21.15.67:0/417989901 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e2a00640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74e40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e3400640 1 --2- 172.21.15.67:0/417989901 >> v2:172.21.15.187:3300/0 conn(0x7f74e40d0700 0x7f74e40d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.964+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526733262 0 0) 0x7f74e409b8a0 con 0x7f74e40c6710 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74c8003520 con 0x7f74e40c6710 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3252107378 0 0) 0x7f74c8003520 con 0x7f74e40c6710 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 msgr2=0x7f74e40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74e40c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 >> v2:172.21.15.187:3300/0 conn(0x7f74e40d0700 msgr2=0x7f74e40d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 --2- 172.21.15.67:0/417989901 >> v2:172.21.15.187:3300/0 conn(0x7f74e40d0700 0x7f74e40d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e40cd480 con 0x7f74e40c6710 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f74d0001670 con 0x7f74e40c6710 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 >> v1:172.21.15.67:6789/0 conn(0x7f74e40c6710 legacy=0x7f74e40cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74c8003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c8003fa0 0x7f74c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74c8004d40 0x7f74c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74e40cd7f0 con 0x7f74c8004d40 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f74e409bb80 con 0x7f74c8003fa0 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74c800b9f0 con 0x7f74e415fe10 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.965+0000 7f74e3400640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74c8004d40 0x7f74c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e2a00640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74c8003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e2000640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c8003fa0 0x7f74c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e3400640 1 -- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 msgr2=0x7f74c8003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e3400640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74e415fe10 0x7f74c8003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e3400640 1 -- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c8003fa0 msgr2=0x7f74c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e3400640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c8003fa0 0x7f74c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.966+0000 7f74e3400640 1 -- 172.21.15.67:0/417989901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c8003520 con 0x7f74c8004d40 2024-04-17T07:14:24.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.967+0000 7f74e3400640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74c8004d40 0x7f74c80051c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009fc0 tx=0x7f74c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48bb32d2d159efc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:24.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.967+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc039070 con 0x7f74c8004d40 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.967+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74dc02cc60 con 0x7f74c8004d40 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.967+0000 7f74e1600640 1 -- 172.21.15.67:0/417989901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc009c10 con 0x7f74c8004d40 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 -- 172.21.15.67:0/417989901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74c8004d40 msgr2=0x7f74c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 --2- 172.21.15.67:0/417989901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74c8004d40 0x7f74c80051c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009fc0 tx=0x7f74c8005830 comp rx=0 tx=0).stop 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3400640 1 -- 172.21.15.67:0/417989901 reap_dead start 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 -- 172.21.15.67:0/417989901 shutdown_connections 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 -- 172.21.15.67:0/417989901 >> 172.21.15.67:0/417989901 conn(0x7f74e40b4170 msgr2=0x7f74e40b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 -- 172.21.15.67:0/417989901 shutdown_connections 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.968+0000 7f74e3e00640 1 -- 172.21.15.67:0/417989901 wait complete. 2024-04-17T07:14:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.971+0000 7f74e3e00640 1 Processor -- start 2024-04-17T07:14:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.971+0000 7f74e3e00640 1 -- start start 2024-04-17T07:14:24.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74c8004d40 0x7f74e40b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 0x7f74e4156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74e4160370 con 0x7f74e4158150 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f74e41601f0 con 0x7f74e41562e0 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.972+0000 7f74e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74e409b470 con 0x7f74c8004d40 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33062/0 (socket says 172.21.15.67:33062) 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 -- 172.21.15.67:0/2664813612 learned_addr learned my addr 172.21.15.67:0/2664813612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2a00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74c8004d40 0x7f74e40b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e3400640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 0x7f74e4156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 -- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74c8004d40 msgr2=0x7f74e40b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74c8004d40 0x7f74e40b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 -- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 msgr2=0x7f74e4156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 0x7f74e4156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e2000640 1 -- 172.21.15.67:0/2664813612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e41f2670 con 0x7f74e4158150 2024-04-17T07:14:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.973+0000 7f74e3400640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 0x7f74e4156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74e2000640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74d000e840 tx=0x7f74d000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d000c7f0 con 0x7f74e4158150 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d0018070 con 0x7f74e4158150 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d000cbb0 con 0x7f74e4158150 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e40b6630 con 0x7f74e4158150 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.974+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e41f2bf0 con 0x7f74e4158150 2024-04-17T07:14:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.975+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e40755d0 con 0x7f74e4158150 2024-04-17T07:14:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.976+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74d001d460 con 0x7f74e4158150 2024-04-17T07:14:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.977+0000 7f74daa00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.977+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f74d008bb20 con 0x7f74e4158150 2024-04-17T07:14:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.978+0000 7f74e2a00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.979+0000 7f74e2a00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 0x7f74c406edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74e415b860 tx=0x7f74d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:24.983+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f74d0059860 con 0x7f74e4158150 2024-04-17T07:14:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.230+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f74e40b4ed0 con 0x7f74e4158150 2024-04-17T07:14:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.231+0000 7f74daa00640 1 -- 172.21.15.67:0/2664813612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f74d0060080 con 0x7f74e4158150 2024-04-17T07:14:25.230 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 msgr2=0x7f74c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 0x7f74c406edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74e415b860 tx=0x7f74d4008040 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 msgr2=0x7f74e41585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74d000e840 tx=0x7f74d000ed00 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 shutdown_connections 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74e41562e0 0x7f74e4156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74e4158150 0x7f74e41585d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f74c406c8f0 0x7f74c406edb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 --2- 172.21.15.67:0/2664813612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74c8004d40 0x7f74e40b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.233+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 >> 172.21.15.67:0/2664813612 conn(0x7f74e40b4170 msgr2=0x7f74e40cb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.234+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 shutdown_connections 2024-04-17T07:14:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:25.234+0000 7f74e3e00640 1 -- 172.21.15.67:0/2664813612 wait complete. 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-17T07:14:25.251 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T07:14:25.252 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T07:14:25.252 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:14:25.252 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-17T07:14:25.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:25.252 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2024-04-17T07:14:25.304 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f0 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f1 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:14:25.305 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:25.306 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:14:25.306 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42585sec preferred_lft 42585sec 2024-04-17T07:14:25.306 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2024-04-17T07:14:25.306 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:25.306 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T07:14:25.306 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:25.306 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:25.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add name ceph-brx type bridge 2024-04-17T07:14:25.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr flush dev ceph-brx 2024-04-17T07:14:25.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set ceph-brx up 2024-04-17T07:14:25.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T07:14:25.307 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:25.401 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:25.811 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:25.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:25.821 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:14:25.897 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-17T07:14:25.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:25.901 DEBUG:teuthology.orchestra.run.smithi187:> route 2024-04-17T07:14:25.957 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2024-04-17T07:14:25.958 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:14:25.958 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:14:25.958 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:14:25.958 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:14:25.958 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:25.958 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:25.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:14:25.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:14:25.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:14:25.959 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:26.062 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:26.722 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:26.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:26.733 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T07:14:26.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:26.787 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2024-04-17T07:14:26.838 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:26.838 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:26.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-17T07:14:26.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-17T07:14:26.838 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:26.937 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:27.023 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:27.032 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-17T07:14:27.032 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:27.032 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:27.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-17T07:14:27.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T07:14:27.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-17T07:14:27.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-17T07:14:27.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-17T07:14:27.033 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:27.132 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:27.357 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:27.367 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:27.367 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:27.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2024-04-17T07:14:27.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T07:14:27.367 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:27.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:27.578 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:27.587 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:27.588 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-17T07:14:27.631 INFO:teuthology.orchestra.run.smithi187.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-17T07:14:27.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:27.632 DEBUG:teuthology.orchestra.run.smithi187:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-17T07:14:27.723 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-17T07:14:27.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:27.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:27.840 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:28.560 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:28.560 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:28.560 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:14:28.560 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: trying mount with new device syntax: 1@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:14:28.561 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:14:28.563 INFO:teuthology.orchestra.run.smithi187.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-17T07:14:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:14:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:14:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:14:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:14:28.569 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:14:28.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:28.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-17T07:14:28.658 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:28.763 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:28.770 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:28.779 INFO:teuthology.orchestra.run.smithi187.stdout:client4579 2024-04-17T07:14:28.779 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:28.779 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:28.830 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:28.983 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:28.989 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:28.999 INFO:teuthology.orchestra.run.smithi187.stdout:client4579 2024-04-17T07:14:28.999 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:29.103 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:29.108 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:29.117 INFO:teuthology.orchestra.run.smithi187.stdout:client4579 2024-04-17T07:14:29.118 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.client4579/status') 2024-04-17T07:14:29.220 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:29.254 INFO:teuthology.orchestra.run.smithi187.stdout:instance: client.4579 (3)192.168.0.1:0/2186052212 2024-04-17T07:14:29.254 INFO:teuthology.orchestra.run.smithi187.stdout:blocklisted: false 2024-04-17T07:14:29.254 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2024-04-17T07:14:29.255 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2024-04-17T07:14:29.255 INFO:teuthology.orchestra.run.smithi187.stderr:69 bytes copied, 6.1123e-05 s, 1.1 MB/s 2024-04-17T07:14:29.257 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:29.267 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-17T07:14:29.267 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T07:14:29.267 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-17T07:14:29.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:14:29.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.498+0000 7f395c000640 1 Processor -- start 2024-04-17T07:14:29.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.498+0000 7f395c000640 1 -- start start 2024-04-17T07:14:29.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395c000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f395406e020 0x7f395406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395c000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39540ceb00 con 0x7f395406f670 2024-04-17T07:14:29.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f39540cede0 con 0x7f395406e020 2024-04-17T07:14:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395c000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39540cf0c0 con 0x7f39540cd020 2024-04-17T07:14:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f395406e020 0x7f395406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f395406e020 0x7f395406e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33078/0 (socket says 172.21.15.67:33078) 2024-04-17T07:14:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.500+0000 7f395a000640 1 -- 172.21.15.67:0/2918876929 learned_addr learned my addr 172.21.15.67:0/2918876929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.501+0000 7f3958c00640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.501+0000 7f395a000640 1 -- 172.21.15.67:0/2918876929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 msgr2=0x7f39540cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.501+0000 7f395a000640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.501+0000 7f395a000640 1 -- 172.21.15.67:0/2918876929 >> v1:172.21.15.187:6789/0 conn(0x7f395406f670 legacy=0x7f395406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.501+0000 7f395a000640 1 -- 172.21.15.67:0/2918876929 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39540cf3a0 con 0x7f395406e020 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.502+0000 7f395a000640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406e020 0x7f395406e4d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f394c009e50 tx=0x7f394c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1571355722e17d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.502+0000 7f3953e00640 1 -- 172.21.15.67:0/2918876929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c02ca70 con 0x7f395406e020 2024-04-17T07:14:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f3953e00640 1 -- 172.21.15.67:0/2918876929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c02cc10 con 0x7f395406e020 2024-04-17T07:14:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f3953e00640 1 -- 172.21.15.67:0/2918876929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c0343f0 con 0x7f395406e020 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 -- 172.21.15.67:0/2918876929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406e020 msgr2=0x7f395406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406e020 0x7f395406e4d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f394c009e50 tx=0x7f394c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 -- 172.21.15.67:0/2918876929 shutdown_connections 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406e020 0x7f395406e4d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 --2- 172.21.15.67:0/2918876929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 -- 172.21.15.67:0/2918876929 >> 172.21.15.67:0/2918876929 conn(0x7f395406bf50 msgr2=0x7f39540b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 -- 172.21.15.67:0/2918876929 shutdown_connections 2024-04-17T07:14:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.503+0000 7f395c000640 1 -- 172.21.15.67:0/2918876929 wait complete. 2024-04-17T07:14:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.505+0000 7f395c000640 1 Processor -- start 2024-04-17T07:14:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.505+0000 7f395c000640 1 -- start start 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 0x7f39540c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39540ce8a0 con 0x7f395406f670 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39540ce720 con 0x7f395406e020 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.506+0000 7f395c000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f395415cb00 con 0x7f39540cd020 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33086/0 (socket says 172.21.15.67:33086) 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 -- 172.21.15.67:0/1731343440 learned_addr learned my addr 172.21.15.67:0/1731343440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3959600640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 0x7f39540c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f395a000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 -- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 msgr2=0x7f39540c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 -- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 msgr2=0x7f39540c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 0x7f39540c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 -- 172.21.15.67:0/1731343440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395414e6a0 con 0x7f395406f670 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f395a000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3959600640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 0x7f39540c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.507+0000 7f3958c00640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f394000d800 tx=0x7f394000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.508+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394000b6c0 con 0x7f395406f670 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.508+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3940019070 con 0x7f395406f670 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.508+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394000ba80 con 0x7f395406f670 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.508+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395414e980 con 0x7f395406f670 2024-04-17T07:14:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.508+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395414ef40 con 0x7f395406f670 2024-04-17T07:14:29.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.509+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3954068b90 con 0x7f395406f670 2024-04-17T07:14:29.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.510+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f394000bc20 con 0x7f395406f670 2024-04-17T07:14:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.511+0000 7f3951600640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 0x7f393406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.511+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f394008c050 con 0x7f395406f670 2024-04-17T07:14:29.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.512+0000 7f3959600640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 0x7f393406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:29.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.513+0000 7f3959600640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 0x7f393406edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39540755a0 tx=0x7f3948008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:29.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.516+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f394001c360 con 0x7f395406f670 2024-04-17T07:14:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.777+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f395406ccd0 con 0x7f395406f670 2024-04-17T07:14:29.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.778+0000 7f3951600640 1 -- 172.21.15.67:0/1731343440 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f394005a470 con 0x7f395406f670 2024-04-17T07:14:29.777 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:14:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.780+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 msgr2=0x7f393406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.780+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 0x7f393406edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39540755a0 tx=0x7f3948008040 comp rx=0 tx=0).stop 2024-04-17T07:14:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 msgr2=0x7f39540c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f394000d800 tx=0x7f394000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 shutdown_connections 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f393406c8f0 0x7f393406edb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39540cd020 0x7f39540c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395406f670 0x7f39540c68b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 --2- 172.21.15.67:0/1731343440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f395406e020 0x7f39540c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 >> 172.21.15.67:0/1731343440 conn(0x7f395406bf50 msgr2=0x7f395406eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 shutdown_connections 2024-04-17T07:14:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:29.781+0000 7f395c000640 1 -- 172.21.15.67:0/1731343440 wait complete. 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T07:14:29.799 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:14:29.800 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-17T07:14:29.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:29.800 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2024-04-17T07:14:29.860 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:29.860 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:29.860 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f0 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f1 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:14:29.861 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42581sec preferred_lft 42581sec 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 2e:c3:cd:1b:9d:c4 brd ff:ff:ff:ff:ff:ff 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::f437:33ff:fe51:8b5e/64 scope link 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 2e:c3:cd:1b:9d:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::2cc3:cdff:fe1b:9dc4/64 scope link 2024-04-17T07:14:29.862 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:29.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:29.863 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:14:29.947 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-17T07:14:29.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:29.951 DEBUG:teuthology.orchestra.run.smithi187:> route 2024-04-17T07:14:30.002 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2024-04-17T07:14:30.002 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:14:30.002 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:14:30.002 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:14:30.002 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:14:30.003 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:30.004 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:30.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:14:30.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:14:30.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:14:30.004 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:30.108 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:30.244 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:30.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:30.254 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T07:14:30.302 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-17T07:14:30.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:30.304 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2024-04-17T07:14:30.396 INFO:teuthology.orchestra.run.smithi187.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-17T07:14:30.397 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:30.397 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:30.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-17T07:14:30.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-17T07:14:30.398 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:30.498 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:30.583 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:30.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:30.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 72:b5:4c:50:bd:8f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T07:14:30.680 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.681 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::70b5:4cff:fe50:bd8f/64 scope link 2024-04-17T07:14:30.681 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:30.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-17T07:14:30.774 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:14:30.774 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:14:30.774 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 72:b5:4c:50:bd:8f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::70b5:4cff:fe50:bd8f/64 scope link 2024-04-17T07:14:30.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:14:30.779 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-17T07:14:30.779 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-17T07:14:30.780 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:30.887 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:31.097 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:31.107 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:14:31.107 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-17T07:14:31.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.1 up 2024-04-17T07:14:31.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.1 master ceph-brx 2024-04-17T07:14:31.107 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-17T07:14:31.207 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:31.323 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:31.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:31.332 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-17T07:14:31.379 INFO:teuthology.orchestra.run.smithi187.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-17T07:14:31.379 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:14:31.379 DEBUG:teuthology.orchestra.run.smithi187:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-17T07:14:31.476 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-17T07:14:31.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:31.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:31.606 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:31.607 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:31.640 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:14:31.641 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:14:31.641 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:14:31.641 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: trying mount with new device syntax: 3@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:14:31.641 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:14:31.642 INFO:teuthology.orchestra.run.smithi187.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-17T07:14:31.643 INFO:teuthology.orchestra.run.smithi187.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:14:31.643 INFO:teuthology.orchestra.run.smithi187.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:14:31.643 INFO:teuthology.orchestra.run.smithi187.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:14:31.643 INFO:teuthology.orchestra.run.smithi187.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:14:31.646 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:14:31.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:31.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-17T07:14:31.726 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:31.824 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:31.829 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:31.838 INFO:teuthology.orchestra.run.smithi187.stdout:client4478 2024-04-17T07:14:31.839 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:31.839 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:31.889 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:32.038 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:32.044 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:32.053 INFO:teuthology.orchestra.run.smithi187.stdout:client4478 2024-04-17T07:14:32.053 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:14:32.151 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:14:32.157 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:32.166 INFO:teuthology.orchestra.run.smithi187.stdout:client4478 2024-04-17T07:14:32.166 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.client4478/status') 2024-04-17T07:14:32.266 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:14:32.306 INFO:teuthology.orchestra.run.smithi187.stdout:instance: client.4478 (3)192.168.0.2:0/3091277572 2024-04-17T07:14:32.306 INFO:teuthology.orchestra.run.smithi187.stdout:blocklisted: false 2024-04-17T07:14:32.307 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2024-04-17T07:14:32.307 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2024-04-17T07:14:32.307 INFO:teuthology.orchestra.run.smithi187.stderr:69 bytes copied, 8.3502e-05 s, 826 kB/s 2024-04-17T07:14:32.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-17T07:14:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:14:32.318 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-17T07:14:32.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:32.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.564+0000 7f93f1400640 1 Processor -- start 2024-04-17T07:14:32.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.565+0000 7f93f1400640 1 -- start start 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.566+0000 7f93f1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93ec06f670 0x7f93ec06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.566+0000 7f93f1400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ec072b30 con 0x7f93ec06f030 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.566+0000 7f93f1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ec06ff90 con 0x7f93ec0cd020 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.566+0000 7f93f1400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f93ec0ceb00 con 0x7f93ec06f670 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93ea800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93ec06f670 0x7f93ec06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93eb200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f93ec06f030 0x7f93ec072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34458/0 (socket says 172.21.15.67:34458) 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93eb200640 1 -- 172.21.15.67:0/2020130989 learned_addr learned my addr 172.21.15.67:0/2020130989 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93ea800640 1 -- 172.21.15.67:0/2020130989 >> v1:172.21.15.187:6789/0 conn(0x7f93ec0cd020 legacy=0x7f93ec071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93ea800640 1 -- 172.21.15.67:0/2020130989 >> v1:172.21.15.67:6789/0 conn(0x7f93ec06f030 legacy=0x7f93ec072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.567+0000 7f93ea800640 1 -- 172.21.15.67:0/2020130989 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ec0cede0 con 0x7f93ec06f670 2024-04-17T07:14:32.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.569+0000 7f93ea800640 1 --2- 172.21.15.67:0/2020130989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f670 0x7f93ec06fa50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93d40097a0 tx=0x7f93d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d170e1e1ce3b58a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:32.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.569+0000 7f93e9400640 1 -- 172.21.15.67:0/2020130989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4035020 con 0x7f93ec06f670 2024-04-17T07:14:32.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.569+0000 7f93e9400640 1 -- 172.21.15.67:0/2020130989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d4031830 con 0x7f93ec06f670 2024-04-17T07:14:32.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.569+0000 7f93e9400640 1 -- 172.21.15.67:0/2020130989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4031c00 con 0x7f93ec06f670 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 -- 172.21.15.67:0/2020130989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f670 msgr2=0x7f93ec06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 --2- 172.21.15.67:0/2020130989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f670 0x7f93ec06fa50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93d40097a0 tx=0x7f93d402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 -- 172.21.15.67:0/2020130989 shutdown_connections 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 --2- 172.21.15.67:0/2020130989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f670 0x7f93ec06fa50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 -- 172.21.15.67:0/2020130989 >> 172.21.15.67:0/2020130989 conn(0x7f93ec06bf50 msgr2=0x7f93ec0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 -- 172.21.15.67:0/2020130989 shutdown_connections 2024-04-17T07:14:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.570+0000 7f93f1400640 1 -- 172.21.15.67:0/2020130989 wait complete. 2024-04-17T07:14:32.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.572+0000 7f93f1400640 1 Processor -- start 2024-04-17T07:14:32.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.572+0000 7f93f1400640 1 -- start start 2024-04-17T07:14:32.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:32.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93ec06f670 0x7f93ec15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93ec0cd020 0x7f93ec158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec0cf680 con 0x7f93ec06f030 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec0cf3a0 con 0x7f93ec06f670 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93f1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93ec0ce720 con 0x7f93ec0cd020 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33100/0 (socket says 172.21.15.67:33100) 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 -- 172.21.15.67:0/2713265933 learned_addr learned my addr 172.21.15.67:0/2713265933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93e9e00640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93ec06f670 0x7f93ec15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 -- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93ec0cd020 msgr2=0x7f93ec158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93ec0cd020 0x7f93ec158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 -- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93ec06f670 msgr2=0x7f93ec15b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.573+0000 7f93ea800640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93ec06f670 0x7f93ec15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93ea800640 1 -- 172.21.15.67:0/2713265933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ec1f4860 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93ea800640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f93d402bee0 tx=0x7f93d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4041070 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d403aa30 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ec15c190 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4035040 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.574+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ec1f4dc0 con 0x7f93ec06f030 2024-04-17T07:14:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.576+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ec068b90 con 0x7f93ec06f030 2024-04-17T07:14:32.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.576+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f93d4038020 con 0x7f93ec06f030 2024-04-17T07:14:32.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.577+0000 7f93e2a00640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:32.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.578+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f93d40ad6d0 con 0x7f93ec06f030 2024-04-17T07:14:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.578+0000 7f93e9e00640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.579+0000 7f93e9e00640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 0x7f93c806edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0b52a0 tx=0x7f93dc007240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:32.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.582+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f93ec1f4860 con 0x7f93ec06f030 2024-04-17T07:14:32.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.787+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93ec0b1d60 con 0x7f93ec06f030 2024-04-17T07:14:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.790+0000 7f93e2a00640 1 -- 172.21.15.67:0/2713265933 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f93d407baf0 con 0x7f93ec06f030 2024-04-17T07:14:32.790 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:32.790 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 msgr2=0x7f93c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 0x7f93c806edb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93ec0b52a0 tx=0x7f93dc007240 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 msgr2=0x7f93ec1531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f93d402bee0 tx=0x7f93d40046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 shutdown_connections 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93ec0cd020 0x7f93ec158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93ec06f670 0x7f93ec15b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 --2- 172.21.15.67:0/2713265933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93ec06f030 0x7f93ec1531f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 >> 172.21.15.67:0/2713265933 conn(0x7f93ec06bf50 msgr2=0x7f93ec0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 shutdown_connections 2024-04-17T07:14:32.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:32.793+0000 7f93f1400640 1 -- 172.21.15.67:0/2713265933 wait complete. 2024-04-17T07:14:32.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:32.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.044+0000 7fb47e400640 1 Processor -- start 2024-04-17T07:14:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.045+0000 7fb47e400640 1 -- start start 2024-04-17T07:14:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.046+0000 7fb47e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb478142e20 0x7fb4780cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.046+0000 7fb47e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4780b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.046+0000 7fb47e400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4781607a0 con 0x7fb4780cf160 2024-04-17T07:14:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.046+0000 7fb47e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb478160a60 con 0x7fb478142e20 2024-04-17T07:14:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.046+0000 7fb47e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb478160d40 con 0x7fb4780cd750 2024-04-17T07:14:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb478142e20 0x7fb4780cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb478142e20 0x7fb4780cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33116/0 (socket says 172.21.15.67:33116) 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 -- 172.21.15.67:0/423370353 learned_addr learned my addr 172.21.15.67:0/423370353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477e00640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4780b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 -- 172.21.15.67:0/423370353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 msgr2=0x7fb4780b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4780b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 -- 172.21.15.67:0/423370353 >> v1:172.21.15.187:6789/0 conn(0x7fb4780cf160 legacy=0x7fb4780cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477400640 1 -- 172.21.15.67:0/423370353 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb478161020 con 0x7fb478142e20 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.047+0000 7fb477e00640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4780b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.049+0000 7fb477400640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb4780cec20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb468009870 tx=0x7fb46802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760bbbba2077feb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.049+0000 7fb476a00640 1 -- 172.21.15.67:0/423370353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb468035020 con 0x7fb478142e20 2024-04-17T07:14:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.049+0000 7fb476a00640 1 -- 172.21.15.67:0/423370353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb468031640 con 0x7fb478142e20 2024-04-17T07:14:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.049+0000 7fb476a00640 1 -- 172.21.15.67:0/423370353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46802d040 con 0x7fb478142e20 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 -- 172.21.15.67:0/423370353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 msgr2=0x7fb4780cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb4780cec20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb468009870 tx=0x7fb46802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 -- 172.21.15.67:0/423370353 shutdown_connections 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4780b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 --2- 172.21.15.67:0/423370353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb4780cec20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 -- 172.21.15.67:0/423370353 >> 172.21.15.67:0/423370353 conn(0x7fb47806b8c0 msgr2=0x7fb47806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 -- 172.21.15.67:0/423370353 shutdown_connections 2024-04-17T07:14:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.050+0000 7fb47e400640 1 -- 172.21.15.67:0/423370353 wait complete. 2024-04-17T07:14:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.052+0000 7fb47e400640 1 Processor -- start 2024-04-17T07:14:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.052+0000 7fb47e400640 1 -- start start 2024-04-17T07:14:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4781515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4780cf160 0x7fb47813c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb478143200 con 0x7fb478142e20 2024-04-17T07:14:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb4780ce730 con 0x7fb4780cf160 2024-04-17T07:14:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb47e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb478143380 con 0x7fb4780cd750 2024-04-17T07:14:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb477e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb477e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33120/0 (socket says 172.21.15.67:33120) 2024-04-17T07:14:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.053+0000 7fb477e00640 1 -- 172.21.15.67:0/972290444 learned_addr learned my addr 172.21.15.67:0/972290444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4781515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb46fe00640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4780cf160 0x7fb47813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 -- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 msgr2=0x7fb4781515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4781515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 -- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4780cf160 msgr2=0x7fb47813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4780cf160 0x7fb47813c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 -- 172.21.15.67:0/972290444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47815e490 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb477e00640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb47000ce80 tx=0x7fb47000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.054+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47001a070 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.055+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb47000d880 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.055+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47000dc70 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.055+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47814ea20 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.055+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb47813ce70 con 0x7fb478142e20 2024-04-17T07:14:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.056+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb478068b90 con 0x7fb478142e20 2024-04-17T07:14:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.057+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb47001d070 con 0x7fb478142e20 2024-04-17T07:14:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.058+0000 7fb46f400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 0x7fb45406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.058+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb47008c2b0 con 0x7fb478142e20 2024-04-17T07:14:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.058+0000 7fb477400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 0x7fb45406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.060+0000 7fb477400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 0x7fb45406ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb468036000 tx=0x7fb46803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.062+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb470059ff0 con 0x7fb478142e20 2024-04-17T07:14:33.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.267+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb4780cde80 con 0x7fb478142e20 2024-04-17T07:14:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.268+0000 7fb46f400640 1 -- 172.21.15.67:0/972290444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fb47005e2f0 con 0x7fb478142e20 2024-04-17T07:14:33.268 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 msgr2=0x7fb45406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 0x7fb45406ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb468036000 tx=0x7fb46803c040 comp rx=0 tx=0).stop 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 msgr2=0x7fb47815ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb47000ce80 tx=0x7fb47000a710 comp rx=0 tx=0).stop 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 shutdown_connections 2024-04-17T07:14:33.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb478142e20 0x7fb47815ae50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4780cf160 0x7fb47813c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb45406c920 0x7fb45406ede0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 --2- 172.21.15.67:0/972290444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4780cd750 0x7fb4781515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 >> 172.21.15.67:0/972290444 conn(0x7fb47806b8c0 msgr2=0x7fb4781413e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 shutdown_connections 2024-04-17T07:14:33.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.271+0000 7fb47e400640 1 -- 172.21.15.67:0/972290444 wait complete. 2024-04-17T07:14:33.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.514+0000 7f8665200640 1 Processor -- start 2024-04-17T07:14:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.514+0000 7f8665200640 1 -- start start 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f8665200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f866015f110 0x7f8660072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f8665200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866006f990 con 0x7f866015fc00 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f8665200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866006fc70 con 0x7f866006f4f0 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f8665200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f86600d0af0 con 0x7f866015f110 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f865e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f866015fc00 0x7f86601686c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57374/0 (socket says 172.21.15.67:57374) 2024-04-17T07:14:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.516+0000 7f865e800640 1 -- 172.21.15.67:0/615255038 learned_addr learned my addr 172.21.15.67:0/615255038 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865de00640 1 --2- 172.21.15.67:0/615255038 >> v2:172.21.15.187:3300/0 conn(0x7f866015f110 0x7f8660072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803705750 0 0) 0x7f866006f990 con 0x7f866015fc00 2024-04-17T07:14:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8644003520 con 0x7f866015fc00 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3712956437 0 0) 0x7f8644003520 con 0x7f866015fc00 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 >> v2:172.21.15.187:3300/0 conn(0x7f866015f110 msgr2=0x7f8660072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 --2- 172.21.15.67:0/615255038 >> v2:172.21.15.187:3300/0 conn(0x7f866015f110 0x7f8660072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 >> v1:172.21.15.187:6789/0 conn(0x7f866006f4f0 legacy=0x7f866015f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600d04a0 con 0x7f866015fc00 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f86580019f0 con 0x7f866015fc00 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 >> v1:172.21.15.67:6789/0 conn(0x7f866015fc00 legacy=0x7f86601686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.517+0000 7f865ca00640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86440039b0 0x7f8644003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865ca00640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865ca00640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86440051c0 0x7f864400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f864400bfe0 con 0x7f8644004800 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f864400c190 con 0x7f86440039b0 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f864400c390 con 0x7f86440051c0 2024-04-17T07:14:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865de00640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86440039b0 0x7f8644003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 -- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86440051c0 msgr2=0x7f864400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86440051c0 0x7f864400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 -- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86440039b0 msgr2=0x7f8644003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86440039b0 0x7f8644003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865d400640 1 -- 172.21.15.67:0/615255038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600d04a0 con 0x7f8644004800 2024-04-17T07:14:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.518+0000 7f865e800640 1 -- 172.21.15.67:0/615255038 reap_dead start 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.519+0000 7f865d400640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8644004c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f864c001750 tx=0x7f8644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b616b63641eaba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c001370 con 0x7f8644004800 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864c0052d0 con 0x7f8644004800 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f865ca00640 1 -- 172.21.15.67:0/615255038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c003830 con 0x7f8644004800 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 -- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 msgr2=0x7f8644004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8644004c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f864c001750 tx=0x7f8644005830 comp rx=0 tx=0).stop 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 -- 172.21.15.67:0/615255038 shutdown_connections 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 --2- 172.21.15.67:0/615255038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8644004c80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 -- 172.21.15.67:0/615255038 >> 172.21.15.67:0/615255038 conn(0x7f8660067ee0 msgr2=0x7f866006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:33.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.520+0000 7f8665200640 1 -- 172.21.15.67:0/615255038 shutdown_connections 2024-04-17T07:14:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.521+0000 7f8665200640 1 -- 172.21.15.67:0/615255038 wait complete. 2024-04-17T07:14:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.522+0000 7f8665200640 1 Processor -- start 2024-04-17T07:14:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.522+0000 7f8665200640 1 -- start start 2024-04-17T07:14:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f866015e150 0x7f86600b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 0x7f866015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f866015f4f0 con 0x7f8644004800 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f86600afdc0 con 0x7f86601f51f0 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.523+0000 7f8665200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f866015f670 con 0x7f866015e150 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55244/0 (socket says 172.21.15.67:55244) 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 -- 172.21.15.67:0/945833140 learned_addr learned my addr 172.21.15.67:0/945833140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865d400640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f866015e150 0x7f86600b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865e800640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 0x7f866015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 -- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f866015e150 msgr2=0x7f86600b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f866015e150 0x7f86600b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 -- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 msgr2=0x7f866015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 0x7f866015cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865de00640 1 -- 172.21.15.67:0/945833140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600b4ad0 con 0x7f8644004800 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.524+0000 7f865e800640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 0x7f866015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f865de00640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f86500116d0 tx=0x7f8650011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865000f9c0 con 0x7f8644004800 2024-04-17T07:14:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865000fb60 con 0x7f8644004800 2024-04-17T07:14:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650015460 con 0x7f8644004800 2024-04-17T07:14:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866015eba0 con 0x7f8644004800 2024-04-17T07:14:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.525+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86601f4710 con 0x7f8644004800 2024-04-17T07:14:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.526+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660068ca0 con 0x7f8644004800 2024-04-17T07:14:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.527+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8650015600 con 0x7f8644004800 2024-04-17T07:14:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.528+0000 7f8656000640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.528+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f865008c200 con 0x7f8644004800 2024-04-17T07:14:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.529+0000 7f865d400640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.530+0000 7f865d400640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 0x7f864006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f864c033000 tx=0x7f864c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.534+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8650019030 con 0x7f8644004800 2024-04-17T07:14:33.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.739+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86600685d0 con 0x7f8644004800 2024-04-17T07:14:33.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.740+0000 7f8656000640 1 -- 172.21.15.67:0/945833140 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f865005a590 con 0x7f8644004800 2024-04-17T07:14:33.739 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:33.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 msgr2=0x7f864006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 0x7f864006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f864c033000 tx=0x7f864c009030 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 msgr2=0x7f8660161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f86500116d0 tx=0x7f8650011b90 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 shutdown_connections 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86601f51f0 0x7f866015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f864006c8f0 0x7f864006edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f866015e150 0x7f86600b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 --2- 172.21.15.67:0/945833140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8644004800 0x7f8660161d30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.743+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 >> 172.21.15.67:0/945833140 conn(0x7f8660067ee0 msgr2=0x7f866006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.744+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 shutdown_connections 2024-04-17T07:14:33.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.744+0000 7f8665200640 1 -- 172.21.15.67:0/945833140 wait complete. 2024-04-17T07:14:33.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.992+0000 7fc8dfe00640 1 Processor -- start 2024-04-17T07:14:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.992+0000 7fc8dfe00640 1 -- start start 2024-04-17T07:14:33.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8e0145040 0x7fc8e00aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:33.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dfe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e00b0150 con 0x7fc8e0072380 2024-04-17T07:14:33.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e0145420 con 0x7fc8e0145040 2024-04-17T07:14:33.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00af0b0 con 0x7fc8e0162fe0 2024-04-17T07:14:33.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.994+0000 7fc8dea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8e0145040 0x7fc8e00aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8e0145040 0x7fc8e00aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55250/0 (socket says 172.21.15.67:55250) 2024-04-17T07:14:33.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 -- 172.21.15.67:0/2962024200 learned_addr learned my addr 172.21.15.67:0/2962024200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:33.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8de000640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 -- 172.21.15.67:0/2962024200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 msgr2=0x7fc8e0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 -- 172.21.15.67:0/2962024200 >> v1:172.21.15.187:6789/0 conn(0x7fc8e0072380 legacy=0x7fc8e006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8dea00640 1 -- 172.21.15.67:0/2962024200 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00af390 con 0x7fc8e0145040 2024-04-17T07:14:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.995+0000 7fc8de000640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.997+0000 7fc8dea00640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e00aea80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009870 tx=0x7fc8d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bb00874355e3d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.997+0000 7fc8dd600640 1 -- 172.21.15.67:0/2962024200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0035020 con 0x7fc8e0145040 2024-04-17T07:14:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.997+0000 7fc8dd600640 1 -- 172.21.15.67:0/2962024200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0031640 con 0x7fc8e0145040 2024-04-17T07:14:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.997+0000 7fc8dd600640 1 -- 172.21.15.67:0/2962024200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d002d040 con 0x7fc8e0145040 2024-04-17T07:14:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 -- 172.21.15.67:0/2962024200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 msgr2=0x7fc8e00aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e00aea80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009870 tx=0x7fc8d002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 -- 172.21.15.67:0/2962024200 shutdown_connections 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/2962024200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e00aea80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.998+0000 7fc8dfe00640 1 -- 172.21.15.67:0/2962024200 >> 172.21.15.67:0/2962024200 conn(0x7fc8e00b23c0 msgr2=0x7fc8e00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.999+0000 7fc8dfe00640 1 -- 172.21.15.67:0/2962024200 shutdown_connections 2024-04-17T07:14:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:33.999+0000 7fc8dfe00640 1 -- 172.21.15.67:0/2962024200 wait complete. 2024-04-17T07:14:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.001+0000 7fc8dfe00640 1 Processor -- start 2024-04-17T07:14:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.001+0000 7fc8dfe00640 1 -- start start 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8e0072380 0x7fc8e0140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e01f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00afc30 con 0x7fc8e0145040 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00af950 con 0x7fc8e0072380 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e0163440 con 0x7fc8e0162fe0 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55264/0 (socket says 172.21.15.67:55264) 2024-04-17T07:14:34.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8de000640 1 -- 172.21.15.67:0/932936305 learned_addr learned my addr 172.21.15.67:0/932936305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:34.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.002+0000 7fc8dea00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8e0072380 0x7fc8e0140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8df400640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e01f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 -- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 msgr2=0x7fc8e01f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e01f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 -- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8e0072380 msgr2=0x7fc8e0140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8e0072380 0x7fc8e0140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 -- 172.21.15.67:0/932936305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e013f670 con 0x7fc8e0145040 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8df400640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e01f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8de000640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800d7d0 tx=0x7fc8c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8004030 con 0x7fc8e0145040 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c80041d0 con 0x7fc8e0145040 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800b6c0 con 0x7fc8e0145040 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.003+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e013dc50 con 0x7fc8e0145040 2024-04-17T07:14:34.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.004+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e013fbf0 con 0x7fc8e0145040 2024-04-17T07:14:34.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.005+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e0068a10 con 0x7fc8e0145040 2024-04-17T07:14:34.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.005+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8c800b860 con 0x7fc8e0145040 2024-04-17T07:14:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.006+0000 7fc8d6a00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 0x7fc8bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.007+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc8c808bde0 con 0x7fc8e0145040 2024-04-17T07:14:34.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.007+0000 7fc8dea00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 0x7fc8bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.008+0000 7fc8dea00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 0x7fc8bc06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0036000 tx=0x7fc8d003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.011+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc8c805a200 con 0x7fc8e0145040 2024-04-17T07:14:34.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.215+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8e0068610 con 0x7fc8e0145040 2024-04-17T07:14:34.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.216+0000 7fc8d6a00640 1 -- 172.21.15.67:0/932936305 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fc8c8059b00 con 0x7fc8e0145040 2024-04-17T07:14:34.215 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 msgr2=0x7fc8bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 0x7fc8bc06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0036000 tx=0x7fc8d003c040 comp rx=0 tx=0).stop 2024-04-17T07:14:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 msgr2=0x7fc8e013d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800d7d0 tx=0x7fc8c800dc90 comp rx=0 tx=0).stop 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 shutdown_connections 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc8bc06c8f0 0x7fc8bc06edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8e0162fe0 0x7fc8e01f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8e0145040 0x7fc8e013d220 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 --2- 172.21.15.67:0/932936305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8e0072380 0x7fc8e0140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 >> 172.21.15.67:0/932936305 conn(0x7fc8e00b23c0 msgr2=0x7fc8e00b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 shutdown_connections 2024-04-17T07:14:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.219+0000 7fc8dfe00640 1 -- 172.21.15.67:0/932936305 wait complete. 2024-04-17T07:14:34.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.466+0000 7fd33b200640 1 Processor -- start 2024-04-17T07:14:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.466+0000 7fd33b200640 1 -- start start 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd33b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd33c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd33b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33c0cd4a0 con 0x7fd33c06b8c0 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd33b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33c1607f0 con 0x7fd33c141ed0 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd33b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd33c15ce30 con 0x7fd33c160280 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd339e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd33c06b8c0 0x7fd33c0cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57390/0 (socket says 172.21.15.67:57390) 2024-04-17T07:14:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd339e00640 1 -- 172.21.15.67:0/196795963 learned_addr learned my addr 172.21.15.67:0/196795963 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.468+0000 7fd331400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd33c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4180293311 0 0) 0x7fd33c0cd4a0 con 0x7fd33c06b8c0 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd320003520 con 0x7fd33c06b8c0 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1273486320 0 0) 0x7fd320003520 con 0x7fd33c06b8c0 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 msgr2=0x7fd33c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd33c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 >> v1:172.21.15.187:6789/0 conn(0x7fd33c141ed0 legacy=0x7fd33c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c160ad0 con 0x7fd33c06b8c0 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd32c0019f0 con 0x7fd33c06b8c0 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 >> v1:172.21.15.67:6789/0 conn(0x7fd33c06b8c0 legacy=0x7fd33c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd320003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd320004d40 0x7fd3200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c15ce30 con 0x7fd320003fd0 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd32000bb20 con 0x7fd320004d40 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd32000bd20 con 0x7fd33c160280 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.469+0000 7fd331400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd339e00640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd320003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd331400640 1 -- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 msgr2=0x7fd320003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd331400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c160280 0x7fd320003a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd331400640 1 -- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd320004d40 msgr2=0x7fd3200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd33a800640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd320004d40 0x7fd3200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd331400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd320004d40 0x7fd3200051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.470+0000 7fd331400640 1 -- 172.21.15.67:0/196795963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd320003520 con 0x7fd320003fd0 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.471+0000 7fd331400640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd320004800 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd324009ac0 tx=0x7fd320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6da57c3362a56e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.471+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324039070 con 0x7fd320003fd0 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.471+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd32402c600 con 0x7fd320003fd0 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.471+0000 7fd339400640 1 -- 172.21.15.67:0/196795963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324009690 con 0x7fd320003fd0 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 -- 172.21.15.67:0/196795963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 msgr2=0x7fd320004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 --2- 172.21.15.67:0/196795963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd320004800 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd324009ac0 tx=0x7fd320005830 comp rx=0 tx=0).stop 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33a800640 1 -- 172.21.15.67:0/196795963 reap_dead start 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 -- 172.21.15.67:0/196795963 shutdown_connections 2024-04-17T07:14:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 -- 172.21.15.67:0/196795963 >> 172.21.15.67:0/196795963 conn(0x7fd33c067d10 msgr2=0x7fd33c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 -- 172.21.15.67:0/196795963 shutdown_connections 2024-04-17T07:14:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.472+0000 7fd33b200640 1 -- 172.21.15.67:0/196795963 wait complete. 2024-04-17T07:14:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.474+0000 7fd33b200640 1 Processor -- start 2024-04-17T07:14:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.474+0000 7fd33b200640 1 -- start start 2024-04-17T07:14:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c15e0f0 0x7fd33c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd33c1411b0 0x7fd33c141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c13f9b0 con 0x7fd320003fd0 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c161090 con 0x7fd33c1411b0 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd33b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd33c13fb30 con 0x7fd33c15e0f0 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.475+0000 7fd339e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55294/0 (socket says 172.21.15.67:55294) 2024-04-17T07:14:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 -- 172.21.15.67:0/3163766579 learned_addr learned my addr 172.21.15.67:0/3163766579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd331400640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c15e0f0 0x7fd33c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 -- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c15e0f0 msgr2=0x7fd33c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c15e0f0 0x7fd33c155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 -- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd33c1411b0 msgr2=0x7fd33c141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd33c1411b0 0x7fd33c141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 -- 172.21.15.67:0/3163766579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c15e900 con 0x7fd320003fd0 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.476+0000 7fd339e00640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd32c001580 tx=0x7fd32c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.477+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c019070 con 0x7fd320003fd0 2024-04-17T07:14:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.477+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd32c004d60 con 0x7fd320003fd0 2024-04-17T07:14:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.477+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0072d0 con 0x7fd320003fd0 2024-04-17T07:14:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.477+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33c15ebe0 con 0x7fd320003fd0 2024-04-17T07:14:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.477+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33c15f1a0 con 0x7fd320003fd0 2024-04-17T07:14:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.478+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd33c142360 con 0x7fd320003fd0 2024-04-17T07:14:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.479+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd32c007470 con 0x7fd320003fd0 2024-04-17T07:14:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.480+0000 7fd332a00640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 0x7fd31806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.480+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd32c08bd50 con 0x7fd320003fd0 2024-04-17T07:14:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.481+0000 7fd331400640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 0x7fd31806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.482+0000 7fd331400640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 0x7fd31806ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd324036000 tx=0x7fd324030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.485+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd33c142360 con 0x7fd320003fd0 2024-04-17T07:14:34.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.690+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd33c068a10 con 0x7fd320003fd0 2024-04-17T07:14:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.693+0000 7fd332a00640 1 -- 172.21.15.67:0/3163766579 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7fd32c05a0e0 con 0x7fd320003fd0 2024-04-17T07:14:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:34.693 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:34.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 msgr2=0x7fd31806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 0x7fd31806ede0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd324036000 tx=0x7fd324030000 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 msgr2=0x7fd33c143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd32c001580 tx=0x7fd32c00b4a0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 shutdown_connections 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd33c1411b0 0x7fd33c141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd31806c920 0x7fd31806ede0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd33c15e0f0 0x7fd33c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 --2- 172.21.15.67:0/3163766579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd320003fd0 0x7fd33c143570 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 >> 172.21.15.67:0/3163766579 conn(0x7fd33c067d10 msgr2=0x7fd33c0b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 shutdown_connections 2024-04-17T07:14:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.696+0000 7fd33b200640 1 -- 172.21.15.67:0/3163766579 wait complete. 2024-04-17T07:14:34.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:34.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.944+0000 7f1867e00640 1 Processor -- start 2024-04-17T07:14:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.945+0000 7f1867e00640 1 -- start start 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.946+0000 7f1867e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f186815ec20 0x7f18680d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.946+0000 7f1867e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18680b3b50 0x7f186806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.946+0000 7f1867e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186806cf80 0x7f18680cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.946+0000 7f1867e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f18680d4f70 con 0x7f186806cf80 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f18680d5250 con 0x7f186815ec20 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f18680cf4d0 con 0x7f18680b3b50 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186806cf80 0x7f18680cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186806cf80 0x7f18680cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55300/0 (socket says 172.21.15.67:55300) 2024-04-17T07:14:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 -- 172.21.15.67:0/1579106070 learned_addr learned my addr 172.21.15.67:0/1579106070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 -- 172.21.15.67:0/1579106070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18680b3b50 msgr2=0x7f186806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 --2- 172.21.15.67:0/1579106070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18680b3b50 0x7f186806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 -- 172.21.15.67:0/1579106070 >> v2:172.21.15.187:3300/0 conn(0x7f186815ec20 msgr2=0x7f18680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 --2- 172.21.15.67:0/1579106070 >> v2:172.21.15.187:3300/0 conn(0x7f186815ec20 0x7f18680d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.947+0000 7f1867400640 1 -- 172.21.15.67:0/1579106070 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18680cf7b0 con 0x7f186806cf80 2024-04-17T07:14:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.948+0000 7f1867400640 1 --2- 172.21.15.67:0/1579106070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186806cf80 0x7f18680cfcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f186808ca90 tx=0x7f186002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1990936a5fa296 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.949+0000 7f1865600640 1 -- 172.21.15.67:0/1579106070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186002c950 con 0x7f186806cf80 2024-04-17T07:14:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.949+0000 7f1865600640 1 -- 172.21.15.67:0/1579106070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186002caf0 con 0x7f186806cf80 2024-04-17T07:14:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.949+0000 7f1865600640 1 -- 172.21.15.67:0/1579106070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18600343f0 con 0x7f186806cf80 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.949+0000 7f1867e00640 1 -- 172.21.15.67:0/1579106070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186806cf80 msgr2=0x7f18680cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.949+0000 7f1867e00640 1 --2- 172.21.15.67:0/1579106070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186806cf80 0x7f18680cfcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f186808ca90 tx=0x7f186002bdb0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 -- 172.21.15.67:0/1579106070 shutdown_connections 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 --2- 172.21.15.67:0/1579106070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186806cf80 0x7f18680cfcd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 --2- 172.21.15.67:0/1579106070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18680b3b50 0x7f186806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 --2- 172.21.15.67:0/1579106070 >> v2:172.21.15.187:3300/0 conn(0x7f186815ec20 0x7f18680d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 -- 172.21.15.67:0/1579106070 >> 172.21.15.67:0/1579106070 conn(0x7f186808bff0 msgr2=0x7f18680d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 -- 172.21.15.67:0/1579106070 shutdown_connections 2024-04-17T07:14:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.950+0000 7f1867e00640 1 -- 172.21.15.67:0/1579106070 wait complete. 2024-04-17T07:14:34.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.951+0000 7f1867e00640 1 Processor -- start 2024-04-17T07:14:34.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.952+0000 7f1867e00640 1 -- start start 2024-04-17T07:14:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186806cf80 0x7f186814dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186815ec20 0x7f18680b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18680d0390 con 0x7f18680b3b50 2024-04-17T07:14:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f18680d0210 con 0x7f186806cf80 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f18680d0510 con 0x7f186815ec20 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55306/0 (socket says 172.21.15.67:55306) 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 -- 172.21.15.67:0/3742737080 learned_addr learned my addr 172.21.15.67:0/3742737080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1867400640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186815ec20 0x7f18680b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 -- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186815ec20 msgr2=0x7f18680b8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186815ec20 0x7f18680b8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 -- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186806cf80 msgr2=0x7f186814dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186806cf80 0x7f186814dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.953+0000 7f1866000640 1 -- 172.21.15.67:0/3742737080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18681e8b30 con 0x7f18680b3b50 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f1866000640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f185000b3f0 tx=0x7f185000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850015030 con 0x7f18680b3b50 2024-04-17T07:14:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1850005970 con 0x7f18680b3b50 2024-04-17T07:14:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850005d60 con 0x7f18680b3b50 2024-04-17T07:14:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18681e8e10 con 0x7f18680b3b50 2024-04-17T07:14:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.954+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18680cb9c0 con 0x7f18680b3b50 2024-04-17T07:14:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.956+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186815a330 con 0x7f18680b3b50 2024-04-17T07:14:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.957+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1850018070 con 0x7f18680b3b50 2024-04-17T07:14:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.958+0000 7f185ea00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 0x7f184406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.958+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f185008b630 con 0x7f18680b3b50 2024-04-17T07:14:34.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.958+0000 7f1866a00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 0x7f184406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:34.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.959+0000 7f1866a00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 0x7f184406edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1858009710 tx=0x7f1858006c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:34.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:34.963+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f186815a330 con 0x7f18680b3b50 2024-04-17T07:14:35.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.167+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18680cbbc0 con 0x7f18680b3b50 2024-04-17T07:14:35.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.168+0000 7f185ea00640 1 -- 172.21.15.67:0/3742737080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f1850059a60 con 0x7f18680b3b50 2024-04-17T07:14:35.167 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:35.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.171+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 msgr2=0x7f184406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.171+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 0x7f184406edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1858009710 tx=0x7f1858006c90 comp rx=0 tx=0).stop 2024-04-17T07:14:35.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.171+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 msgr2=0x7f18681e83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.171+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f185000b3f0 tx=0x7f185000b8b0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 shutdown_connections 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f184406c8f0 0x7f184406edb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186815ec20 0x7f18680b8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18680b3b50 0x7f18681e83c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 --2- 172.21.15.67:0/3742737080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186806cf80 0x7f186814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 >> 172.21.15.67:0/3742737080 conn(0x7f186808bff0 msgr2=0x7f186806e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 shutdown_connections 2024-04-17T07:14:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.172+0000 7f1867e00640 1 -- 172.21.15.67:0/3742737080 wait complete. 2024-04-17T07:14:35.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:35.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.408+0000 7fbe73e00640 1 Processor -- start 2024-04-17T07:14:35.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.409+0000 7fbe73e00640 1 -- start start 2024-04-17T07:14:35.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 0x7fbe740cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe740cf8e0 0x7fbe740d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe740d3670 con 0x7fbe740cf8e0 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbe740d0c30 con 0x7fbe740d3210 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.410+0000 7fbe73e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe740d00b0 con 0x7fbe7415ecb0 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.411+0000 7fbe73400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe740cf8e0 0x7fbe740d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.411+0000 7fbe73400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe740cf8e0 0x7fbe740d10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55320/0 (socket says 172.21.15.67:55320) 2024-04-17T07:14:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.411+0000 7fbe73400640 1 -- 172.21.15.67:0/87143629 learned_addr learned my addr 172.21.15.67:0/87143629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.411+0000 7fbe6a000640 1 --2- 172.21.15.67:0/87143629 >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 0x7fbe740cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.411+0000 7fbe72a00640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe73400640 1 -- 172.21.15.67:0/87143629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 msgr2=0x7fbe740d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe73400640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe73400640 1 -- 172.21.15.67:0/87143629 >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 msgr2=0x7fbe740cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe73400640 1 --2- 172.21.15.67:0/87143629 >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 0x7fbe740cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe73400640 1 -- 172.21.15.67:0/87143629 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe740d0390 con 0x7fbe740cf8e0 2024-04-17T07:14:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.412+0000 7fbe6a000640 1 --2- 172.21.15.67:0/87143629 >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 0x7fbe740cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.413+0000 7fbe73400640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe740d10f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbe740696e0 tx=0x7fbe6c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db8d6caed58ba4a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe72000640 1 -- 172.21.15.67:0/87143629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c02e070 con 0x7fbe740cf8e0 2024-04-17T07:14:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe72000640 1 -- 172.21.15.67:0/87143629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe6c037070 con 0x7fbe740cf8e0 2024-04-17T07:14:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe72000640 1 -- 172.21.15.67:0/87143629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c031a00 con 0x7fbe740cf8e0 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 -- 172.21.15.67:0/87143629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 msgr2=0x7fbe740d10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe740d10f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbe740696e0 tx=0x7fbe6c02b880 comp rx=0 tx=0).stop 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 -- 172.21.15.67:0/87143629 shutdown_connections 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe740d10f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 --2- 172.21.15.67:0/87143629 >> v2:172.21.15.187:3300/0 conn(0x7fbe740d3210 0x7fbe740cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 --2- 172.21.15.67:0/87143629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.414+0000 7fbe73e00640 1 -- 172.21.15.67:0/87143629 >> 172.21.15.67:0/87143629 conn(0x7fbe7408bff0 msgr2=0x7fbe740d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.415+0000 7fbe73e00640 1 -- 172.21.15.67:0/87143629 shutdown_connections 2024-04-17T07:14:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.415+0000 7fbe73e00640 1 -- 172.21.15.67:0/87143629 wait complete. 2024-04-17T07:14:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.416+0000 7fbe73e00640 1 Processor -- start 2024-04-17T07:14:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.417+0000 7fbe73e00640 1 -- start start 2024-04-17T07:14:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 0x7fbe740b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe740cfd50 con 0x7fbe740cf8e0 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbe740d0950 con 0x7fbe740d3210 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7415f090 con 0x7fbe7415ecb0 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55332/0 (socket says 172.21.15.67:55332) 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 -- 172.21.15.67:0/96465513 learned_addr learned my addr 172.21.15.67:0/96465513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73400640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe6a000640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 0x7fbe740b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 -- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 msgr2=0x7fbe740b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 -- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 msgr2=0x7fbe740b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 0x7fbe740b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe72a00640 1 -- 172.21.15.67:0/96465513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe740b7480 con 0x7fbe740cf8e0 2024-04-17T07:14:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe73400640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.418+0000 7fbe6a000640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 0x7fbe740b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe72a00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009db0 tx=0x7fbe6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64018020 con 0x7fbe740cf8e0 2024-04-17T07:14:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe64014480 con 0x7fbe740cf8e0 2024-04-17T07:14:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64014850 con 0x7fbe740cf8e0 2024-04-17T07:14:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe740b7760 con 0x7fbe740cf8e0 2024-04-17T07:14:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.419+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe740b7d20 con 0x7fbe740cf8e0 2024-04-17T07:14:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.420+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe740b8100 con 0x7fbe740cf8e0 2024-04-17T07:14:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.421+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe64005d10 con 0x7fbe740cf8e0 2024-04-17T07:14:35.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.422+0000 7fbe6b400640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 0x7fbe5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.422+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fbe6408c700 con 0x7fbe740cf8e0 2024-04-17T07:14:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.423+0000 7fbe6a000640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 0x7fbe5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.424+0000 7fbe6a000640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 0x7fbe5006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.427+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbe740b8100 con 0x7fbe740cf8e0 2024-04-17T07:14:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.628+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe74068a10 con 0x7fbe740cf8e0 2024-04-17T07:14:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.629+0000 7fbe6b400640 1 -- 172.21.15.67:0/96465513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fbe64022070 con 0x7fbe740cf8e0 2024-04-17T07:14:35.628 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.631+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 msgr2=0x7fbe5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.631+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 0x7fbe5006edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c008040 comp rx=0 tx=0).stop 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.631+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 msgr2=0x7fbe7415cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.631+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009db0 tx=0x7fbe6400c490 comp rx=0 tx=0).stop 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 shutdown_connections 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbe5006c8f0 0x7fbe5006edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7415ecb0 0x7fbe740b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe740d3210 0x7fbe740b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 --2- 172.21.15.67:0/96465513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe740cf8e0 0x7fbe7415cbe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 >> 172.21.15.67:0/96465513 conn(0x7fbe7408bff0 msgr2=0x7fbe7406d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 shutdown_connections 2024-04-17T07:14:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.632+0000 7fbe73e00640 1 -- 172.21.15.67:0/96465513 wait complete. 2024-04-17T07:14:35.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.867+0000 7f575ee00640 1 Processor -- start 2024-04-17T07:14:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.867+0000 7f575ee00640 1 -- start start 2024-04-17T07:14:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.869+0000 7f575ee00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5758145b00 0x7f5758160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.869+0000 7f575ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5758146060 0x7f575815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.869+0000 7f575ee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5758160920 con 0x7f57580b4ac0 2024-04-17T07:14:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.869+0000 7f575ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5758160c00 con 0x7f5758146060 2024-04-17T07:14:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.869+0000 7f575ee00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5758160ee0 con 0x7f5758145b00 2024-04-17T07:14:35.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575e400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57580b4ac0 0x7f57580b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57394/0 (socket says 172.21.15.67:57394) 2024-04-17T07:14:35.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575e400640 1 -- 172.21.15.67:0/59899858 learned_addr learned my addr 172.21.15.67:0/59899858 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:35.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575d000640 1 --2- 172.21.15.67:0/59899858 >> v2:172.21.15.67:3300/0 conn(0x7f5758146060 0x7f575815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575da00640 1 --2- 172.21.15.67:0/59899858 >> v2:172.21.15.187:3300/0 conn(0x7f5758145b00 0x7f5758160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f5757e00640 1 -- 172.21.15.67:0/59899858 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468075515 0 0) 0x7f5758160920 con 0x7f57580b4ac0 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f5757e00640 1 -- 172.21.15.67:0/59899858 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5740003520 con 0x7f57580b4ac0 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575d000640 1 -- 172.21.15.67:0/59899858 >> v2:172.21.15.187:3300/0 conn(0x7f5758145b00 msgr2=0x7f5758160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575d000640 1 --2- 172.21.15.67:0/59899858 >> v2:172.21.15.187:3300/0 conn(0x7f5758145b00 0x7f5758160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575d000640 1 -- 172.21.15.67:0/59899858 >> v1:172.21.15.67:6789/0 conn(0x7f57580b4ac0 legacy=0x7f57580b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.870+0000 7f575d000640 1 -- 172.21.15.67:0/59899858 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57581611c0 con 0x7f5758146060 2024-04-17T07:14:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.871+0000 7f575d000640 1 --2- 172.21.15.67:0/59899858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5758146060 0x7f575815cd80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5758073580 tx=0x7f574402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175be933c8e75ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f5757e00640 1 -- 172.21.15.67:0/59899858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440315a0 con 0x7f5758146060 2024-04-17T07:14:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f5757e00640 1 -- 172.21.15.67:0/59899858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5744037070 con 0x7f5758146060 2024-04-17T07:14:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f5757e00640 1 -- 172.21.15.67:0/59899858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744031960 con 0x7f5758146060 2024-04-17T07:14:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 -- 172.21.15.67:0/59899858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5758146060 msgr2=0x7f575815cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 --2- 172.21.15.67:0/59899858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5758146060 0x7f575815cd80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5758073580 tx=0x7f574402b840 comp rx=0 tx=0).stop 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 -- 172.21.15.67:0/59899858 shutdown_connections 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 --2- 172.21.15.67:0/59899858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5758146060 0x7f575815cd80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 --2- 172.21.15.67:0/59899858 >> v2:172.21.15.187:3300/0 conn(0x7f5758145b00 0x7f5758160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 -- 172.21.15.67:0/59899858 >> 172.21.15.67:0/59899858 conn(0x7f57580b23c0 msgr2=0x7f57580b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 -- 172.21.15.67:0/59899858 shutdown_connections 2024-04-17T07:14:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.872+0000 7f575ee00640 1 -- 172.21.15.67:0/59899858 wait complete. 2024-04-17T07:14:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.874+0000 7f575ee00640 1 Processor -- start 2024-04-17T07:14:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.874+0000 7f575ee00640 1 -- start start 2024-04-17T07:14:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758145b00 0x7f5758144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758146060 0x7f575815bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f575815d2c0 con 0x7f57580b4ac0 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f57580cfba0 con 0x7f5758146060 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57580b4390 con 0x7f5758145b00 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55352/0 (socket says 172.21.15.67:55352) 2024-04-17T07:14:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.875+0000 7f575da00640 1 -- 172.21.15.67:0/1606149014 learned_addr learned my addr 172.21.15.67:0/1606149014 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 -- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758145b00 msgr2=0x7f5758144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758145b00 0x7f5758144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 -- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758146060 msgr2=0x7f575815bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758146060 0x7f575815bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 -- 172.21.15.67:0/1606149014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575813f300 con 0x7f57580b4ac0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f575da00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f574c00d800 tx=0x7f574c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00b6c0 con 0x7f57580b4ac0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574c019070 con 0x7f57580b4ac0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.876+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00ba80 con 0x7f57580b4ac0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.877+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575813eb00 con 0x7f57580b4ac0 2024-04-17T07:14:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.877+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5758141ee0 con 0x7f57580b4ac0 2024-04-17T07:14:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.878+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5758068a10 con 0x7f57580b4ac0 2024-04-17T07:14:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.879+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f574c00bc20 con 0x7f57580b4ac0 2024-04-17T07:14:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.880+0000 7f5755600640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 0x7f573806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.880+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f574c08bfc0 con 0x7f57580b4ac0 2024-04-17T07:14:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.880+0000 7f575d000640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 0x7f573806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.881+0000 7f575d000640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 0x7f573806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5744036000 tx=0x7f5744030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:35.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:35.884+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f574c05a480 con 0x7f57580b4ac0 2024-04-17T07:14:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.083+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5758068610 con 0x7f57580b4ac0 2024-04-17T07:14:36.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.084+0000 7f5755600640 1 -- 172.21.15.67:0/1606149014 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f574c059d80 con 0x7f57580b4ac0 2024-04-17T07:14:36.083 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 msgr2=0x7f573806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 0x7f573806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5744036000 tx=0x7f5744030000 comp rx=0 tx=0).stop 2024-04-17T07:14:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 msgr2=0x7f575813e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f574c00d800 tx=0x7f574c00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 shutdown_connections 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758146060 0x7f575815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f573806c8f0 0x7f573806edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758145b00 0x7f5758144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 --2- 172.21.15.67:0/1606149014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57580b4ac0 0x7f575813e520 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 >> 172.21.15.67:0/1606149014 conn(0x7f57580b23c0 msgr2=0x7f57580b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 shutdown_connections 2024-04-17T07:14:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.087+0000 7f575ee00640 1 -- 172.21.15.67:0/1606149014 wait complete. 2024-04-17T07:14:36.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.322+0000 7f4d42c00640 1 Processor -- start 2024-04-17T07:14:36.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.322+0000 7f4d42c00640 1 -- start start 2024-04-17T07:14:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.324+0000 7f4d42c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d3c0d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.324+0000 7f4d42c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0d3170 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.324+0000 7f4d42c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0d3450 con 0x7f4d3c0d18d0 2024-04-17T07:14:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.324+0000 7f4d42c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d3730 con 0x7f4d3c15ebb0 2024-04-17T07:14:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d41800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d3c0d14f0 0x7f4d3c0d2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57410/0 (socket says 172.21.15.67:57410) 2024-04-17T07:14:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d41800640 1 -- 172.21.15.67:0/4056690950 learned_addr learned my addr 172.21.15.67:0/4056690950 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952880761 0 0) 0x7f4d3c0d3170 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d24003520 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d42200640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d3c0d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3615830674 0 0) 0x7f4d24003520 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 msgr2=0x7f4d3c0d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.325+0000 7f4d3be00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d3c0d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 >> v1:172.21.15.187:6789/0 conn(0x7f4d3c0d18d0 legacy=0x7f4d3c15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c0d3a10 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4d30001670 con 0x7f4d3c0d14f0 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 >> v1:172.21.15.67:6789/0 conn(0x7f4d3c0d14f0 legacy=0x7f4d3c0d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d24003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24004040 0x7f4d24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d3730 con 0x7f4d24004040 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2400bbb0 con 0x7f4d24004d40 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2400bdb0 con 0x7f4d3c15ebb0 2024-04-17T07:14:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d40e00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24004040 0x7f4d24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d42200640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.326+0000 7f4d41800640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d24003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.327+0000 7f4d40e00640 1 -- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 msgr2=0x7f4d24003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.327+0000 7f4d40e00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d3c15ebb0 0x7f4d24003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.327+0000 7f4d40e00640 1 -- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d24004d40 msgr2=0x7f4d240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.327+0000 7f4d40e00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.327+0000 7f4d40e00640 1 -- 172.21.15.67:0/4056690950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d24003520 con 0x7f4d24004040 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.328+0000 7f4d40e00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24004040 0x7f4d24004800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4d28001860 tx=0x7f4d24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3315dbaef3bcc549 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.328+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28038070 con 0x7f4d24004040 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.328+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d28033410 con 0x7f4d24004040 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.328+0000 7f4d3be00640 1 -- 172.21.15.67:0/4056690950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28001430 con 0x7f4d24004040 2024-04-17T07:14:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 -- 172.21.15.67:0/4056690950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24004040 msgr2=0x7f4d24004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 --2- 172.21.15.67:0/4056690950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24004040 0x7f4d24004800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4d28001860 tx=0x7f4d24005830 comp rx=0 tx=0).stop 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42200640 1 -- 172.21.15.67:0/4056690950 reap_dead start 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 -- 172.21.15.67:0/4056690950 shutdown_connections 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 -- 172.21.15.67:0/4056690950 >> 172.21.15.67:0/4056690950 conn(0x7f4d3c06cbd0 msgr2=0x7f4d3c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 -- 172.21.15.67:0/4056690950 shutdown_connections 2024-04-17T07:14:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.329+0000 7f4d42c00640 1 -- 172.21.15.67:0/4056690950 wait complete. 2024-04-17T07:14:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.330+0000 7f4d42c00640 1 Processor -- start 2024-04-17T07:14:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.330+0000 7f4d42c00640 1 -- start start 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d24004040 0x7f4d3c1605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 0x7f4d3c15efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0ba5f0 con 0x7f4d3c153d20 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0ba470 con 0x7f4d3c0ca360 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d42c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0ba770 con 0x7f4d24004040 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d40e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.331+0000 7f4d40e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55366/0 (socket says 172.21.15.67:55366) 2024-04-17T07:14:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 -- 172.21.15.67:0/438753248 learned_addr learned my addr 172.21.15.67:0/438753248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d42200640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 0x7f4d3c15efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 -- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d24004040 msgr2=0x7f4d3c1605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d24004040 0x7f4d3c1605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 -- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 msgr2=0x7f4d3c15efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 0x7f4d3c15efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 -- 172.21.15.67:0/438753248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c160c30 con 0x7f4d3c153d20 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d42200640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 0x7f4d3c15efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.332+0000 7f4d40e00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4d28001980 tx=0x7f4d28001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.333+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28041020 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.333+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d28006e00 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.333+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28038040 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.333+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c160f10 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.333+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c1614d0 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.334+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3c068a10 con 0x7f4d3c153d20 2024-04-17T07:14:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.335+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d28044020 con 0x7f4d3c153d20 2024-04-17T07:14:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.336+0000 7f4d39600640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 0x7f4d1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.336+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4d280ad080 con 0x7f4d3c153d20 2024-04-17T07:14:36.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.337+0000 7f4d41800640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 0x7f4d1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.338+0000 7f4d41800640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 0x7f4d1c06ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d30001050 tx=0x7f4d30000f40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.340+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4d2804b020 con 0x7f4d3c153d20 2024-04-17T07:14:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.538+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d3c15d560 con 0x7f4d3c153d20 2024-04-17T07:14:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.540+0000 7f4d39600640 1 -- 172.21.15.67:0/438753248 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f4d2807b530 con 0x7f4d3c153d20 2024-04-17T07:14:36.541 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:36.541 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 msgr2=0x7f4d1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 0x7f4d1c06ede0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d30001050 tx=0x7f4d30000f40 comp rx=0 tx=0).stop 2024-04-17T07:14:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 msgr2=0x7f4d3c0c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4d28001980 tx=0x7f4d28001230 comp rx=0 tx=0).stop 2024-04-17T07:14:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 shutdown_connections 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d3c0ca360 0x7f4d3c15efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d3c153d20 0x7f4d3c0c9e20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f4d1c06c920 0x7f4d1c06ede0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 --2- 172.21.15.67:0/438753248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d24004040 0x7f4d3c1605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 >> 172.21.15.67:0/438753248 conn(0x7f4d3c06cbd0 msgr2=0x7f4d3c0b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 shutdown_connections 2024-04-17T07:14:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.543+0000 7f4d42c00640 1 -- 172.21.15.67:0/438753248 wait complete. 2024-04-17T07:14:36.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:36.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.763+0000 7fc3ada00640 1 Processor -- start 2024-04-17T07:14:36.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.764+0000 7fc3ada00640 1 -- start start 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3ada00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80b7b60 0x7fc3a80d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3ada00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3a80d18b0 0x7fc3a80b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3ada00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a80d4740 con 0x7fc3a815ec00 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3ada00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a80d3610 con 0x7fc3a80d18b0 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3ada00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a80d2a00 con 0x7fc3a80b7b60 2024-04-17T07:14:36.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3a6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3a815ec00 0x7fc3a80cf870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57412/0 (socket says 172.21.15.67:57412) 2024-04-17T07:14:36.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3a6a00640 1 -- 172.21.15.67:0/2004488366 learned_addr learned my addr 172.21.15.67:0/2004488366 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:36.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.765+0000 7fc3a7400640 1 --2- 172.21.15.67:0/2004488366 >> v2:172.21.15.67:3300/0 conn(0x7fc3a80d18b0 0x7fc3a80b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a5600640 1 -- 172.21.15.67:0/2004488366 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589289850 0 0) 0x7fc3a80d4740 con 0x7fc3a815ec00 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a5600640 1 -- 172.21.15.67:0/2004488366 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc38c003520 con 0x7fc3a815ec00 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a7400640 1 -- 172.21.15.67:0/2004488366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80b7b60 msgr2=0x7fc3a80d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a6000640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80b7b60 0x7fc3a80d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a7400640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80b7b60 0x7fc3a80d1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a7400640 1 -- 172.21.15.67:0/2004488366 >> v1:172.21.15.67:6789/0 conn(0x7fc3a815ec00 legacy=0x7fc3a80cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.766+0000 7fc3a7400640 1 -- 172.21.15.67:0/2004488366 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a80d2ce0 con 0x7fc3a80d18b0 2024-04-17T07:14:36.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.767+0000 7fc3a7400640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80d18b0 0x7fc3a80b6420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00097a0 tx=0x7fc3a002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4c05e428ffcde server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.767+0000 7fc3a5600640 1 -- 172.21.15.67:0/2004488366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031780 con 0x7fc3a80d18b0 2024-04-17T07:14:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.767+0000 7fc3a5600640 1 -- 172.21.15.67:0/2004488366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a0037070 con 0x7fc3a80d18b0 2024-04-17T07:14:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.767+0000 7fc3a5600640 1 -- 172.21.15.67:0/2004488366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031b40 con 0x7fc3a80d18b0 2024-04-17T07:14:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 -- 172.21.15.67:0/2004488366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80d18b0 msgr2=0x7fc3a80b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80d18b0 0x7fc3a80b6420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00097a0 tx=0x7fc3a002b820 comp rx=0 tx=0).stop 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 -- 172.21.15.67:0/2004488366 shutdown_connections 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80d18b0 0x7fc3a80b6420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2004488366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80b7b60 0x7fc3a80d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 -- 172.21.15.67:0/2004488366 >> 172.21.15.67:0/2004488366 conn(0x7fc3a806ccb0 msgr2=0x7fc3a806d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 -- 172.21.15.67:0/2004488366 shutdown_connections 2024-04-17T07:14:36.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.768+0000 7fc3ada00640 1 -- 172.21.15.67:0/2004488366 wait complete. 2024-04-17T07:14:36.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.770+0000 7fc3ada00640 1 Processor -- start 2024-04-17T07:14:36.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.770+0000 7fc3ada00640 1 -- start start 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80d18b0 0x7fc3a8152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a815ec00 0x7fc3a81f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a815efe0 con 0x7fc3a80b7b60 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a80d32a0 con 0x7fc3a815ec00 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3ada00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a815f160 con 0x7fc3a80d18b0 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55388/0 (socket says 172.21.15.67:55388) 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.771+0000 7fc3a6a00640 1 -- 172.21.15.67:0/2922316632 learned_addr learned my addr 172.21.15.67:0/2922316632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 -- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80d18b0 msgr2=0x7fc3a8152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80d18b0 0x7fc3a8152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 -- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a815ec00 msgr2=0x7fc3a81f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a7400640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a815ec00 0x7fc3a81f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a815ec00 0x7fc3a81f4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 -- 172.21.15.67:0/2922316632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a81f4da0 con 0x7fc3a80b7b60 2024-04-17T07:14:36.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3a6a00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc39800caa0 tx=0x7fc39800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398017070 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39800ac00 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398012400 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.772+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a81f5070 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.773+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a81f5630 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.774+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a81f5800 con 0x7fc3a80b7b60 2024-04-17T07:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.775+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc39801a070 con 0x7fc3a80b7b60 2024-04-17T07:14:36.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.776+0000 7fc39ea00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 0x7fc38406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:36.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.776+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc39808b910 con 0x7fc3a80b7b60 2024-04-17T07:14:36.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.776+0000 7fc3a6000640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 0x7fc38406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.777+0000 7fc3a6000640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 0x7fc38406ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3a81f8ef0 tx=0x7fc390008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.780+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc3a81f5800 con 0x7fc3a80b7b60 2024-04-17T07:14:36.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.973+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc3a80d26b0 con 0x7fc3a80b7b60 2024-04-17T07:14:36.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.974+0000 7fc39ea00640 1 -- 172.21.15.67:0/2922316632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fc398059d30 con 0x7fc3a80b7b60 2024-04-17T07:14:36.974 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.976+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 msgr2=0x7fc38406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.976+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 0x7fc38406ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3a81f8ef0 tx=0x7fc390008040 comp rx=0 tx=0).stop 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.976+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 msgr2=0x7fc3a8153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.976+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc39800caa0 tx=0x7fc39800cf60 comp rx=0 tx=0).stop 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 shutdown_connections 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3a815ec00 0x7fc3a81f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc38406c920 0x7fc38406ede0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3a80d18b0 0x7fc3a8152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 --2- 172.21.15.67:0/2922316632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3a80b7b60 0x7fc3a8153c00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:36.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 >> 172.21.15.67:0/2922316632 conn(0x7fc3a806ccb0 msgr2=0x7fc3a80b9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:36.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 shutdown_connections 2024-04-17T07:14:36.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:36.977+0000 7fc3ada00640 1 -- 172.21.15.67:0/2922316632 wait complete. 2024-04-17T07:14:36.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.220+0000 7fbc10e00640 1 Processor -- start 2024-04-17T07:14:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.220+0000 7fbc10e00640 1 -- start start 2024-04-17T07:14:37.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc10e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbc0c145f80 0x7fbc0c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc10e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc0c06c270 0x7fbc0c06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc10e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0c146360 con 0x7fbc0c0cd750 2024-04-17T07:14:37.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc10e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0c0727a0 con 0x7fbc0c06c270 2024-04-17T07:14:37.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc10e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0c071e70 con 0x7fbc0c145f80 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc0be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc0c06c270 0x7fbc0c06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc0be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc0c06c270 0x7fbc0c06e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55400/0 (socket says 172.21.15.67:55400) 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc0be00640 1 -- 172.21.15.67:0/4280541975 learned_addr learned my addr 172.21.15.67:0/4280541975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.222+0000 7fbc0b400640 1 --2- 172.21.15.67:0/4280541975 >> v2:172.21.15.187:3300/0 conn(0x7fbc0c145f80 0x7fbc0c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.223+0000 7fbc0be00640 1 -- 172.21.15.67:0/4280541975 >> v2:172.21.15.187:3300/0 conn(0x7fbc0c145f80 msgr2=0x7fbc0c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.223+0000 7fbc0be00640 1 --2- 172.21.15.67:0/4280541975 >> v2:172.21.15.187:3300/0 conn(0x7fbc0c145f80 0x7fbc0c0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.223+0000 7fbc0be00640 1 -- 172.21.15.67:0/4280541975 >> v1:172.21.15.187:6789/0 conn(0x7fbc0c0cd750 legacy=0x7fbc0c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:37.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.223+0000 7fbc0be00640 1 -- 172.21.15.67:0/4280541975 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c072150 con 0x7fbc0c06c270 2024-04-17T07:14:37.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.224+0000 7fbc0be00640 1 --2- 172.21.15.67:0/4280541975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c06c270 0x7fbc0c06e1c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c06c9c0 tx=0x7fbc0402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c1c6c640c09d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.224+0000 7fbc0a000640 1 -- 172.21.15.67:0/4280541975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0402c9f0 con 0x7fbc0c06c270 2024-04-17T07:14:37.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.224+0000 7fbc0a000640 1 -- 172.21.15.67:0/4280541975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc0402cb90 con 0x7fbc0c06c270 2024-04-17T07:14:37.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.224+0000 7fbc0a000640 1 -- 172.21.15.67:0/4280541975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040343f0 con 0x7fbc0c06c270 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 -- 172.21.15.67:0/4280541975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c06c270 msgr2=0x7fbc0c06e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4280541975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c06c270 0x7fbc0c06e1c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c06c9c0 tx=0x7fbc0402be50 comp rx=0 tx=0).stop 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 -- 172.21.15.67:0/4280541975 shutdown_connections 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4280541975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c06c270 0x7fbc0c06e1c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4280541975 >> v2:172.21.15.187:3300/0 conn(0x7fbc0c145f80 0x7fbc0c0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 -- 172.21.15.67:0/4280541975 >> 172.21.15.67:0/4280541975 conn(0x7fbc0c0b2190 msgr2=0x7fbc0c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 -- 172.21.15.67:0/4280541975 shutdown_connections 2024-04-17T07:14:37.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.225+0000 7fbc10e00640 1 -- 172.21.15.67:0/4280541975 wait complete. 2024-04-17T07:14:37.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.227+0000 7fbc10e00640 1 Processor -- start 2024-04-17T07:14:37.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.227+0000 7fbc10e00640 1 -- start start 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc0c06c270 0x7fbc0c1409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c145f80 0x7fbc0c0aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c072db0 con 0x7fbc0c0cd750 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c072c30 con 0x7fbc0c145f80 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc10e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0c06e700 con 0x7fbc0c06c270 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55404/0 (socket says 172.21.15.67:55404) 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.228+0000 7fbc0aa00640 1 -- 172.21.15.67:0/4119324748 learned_addr learned my addr 172.21.15.67:0/4119324748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 -- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc0c06c270 msgr2=0x7fbc0c1409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc0c06c270 0x7fbc0c1409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 -- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c145f80 msgr2=0x7fbc0c0aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0be00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c145f80 0x7fbc0c0aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c145f80 0x7fbc0c0aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 -- 172.21.15.67:0/4119324748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c140f00 con 0x7fbc0c0cd750 2024-04-17T07:14:37.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc0aa00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c430 tx=0x7fbbf800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011790 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8017070 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8011b70 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c1e5880 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.229+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c1e5e40 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.230+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c068a10 con 0x7fbc0c0cd750 2024-04-17T07:14:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.231+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbbf8011d10 con 0x7fbc0c0cd750 2024-04-17T07:14:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.232+0000 7fbc03400640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 0x7fbbec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.232+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fbbf808b970 con 0x7fbc0c0cd750 2024-04-17T07:14:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.233+0000 7fbc0b400640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 0x7fbbec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.234+0000 7fbc0b400640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 0x7fbbec06edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.237+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbbf8059d90 con 0x7fbc0c0cd750 2024-04-17T07:14:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.439+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc0c06f6e0 con 0x7fbc0c0cd750 2024-04-17T07:14:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.440+0000 7fbc03400640 1 -- 172.21.15.67:0/4119324748 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fbbf8059690 con 0x7fbc0c0cd750 2024-04-17T07:14:37.441 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 msgr2=0x7fbbec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 0x7fbbec06edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc008040 comp rx=0 tx=0).stop 2024-04-17T07:14:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 msgr2=0x7fbc0c0ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c430 tx=0x7fbbf800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 shutdown_connections 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc0c145f80 0x7fbc0c0aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc0c0cd750 0x7fbc0c0ae920 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbbec06c8f0 0x7fbbec06edb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 --2- 172.21.15.67:0/4119324748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc0c06c270 0x7fbc0c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 >> 172.21.15.67:0/4119324748 conn(0x7fbc0c0b2190 msgr2=0x7fbc0c0b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 shutdown_connections 2024-04-17T07:14:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.443+0000 7fbc10e00640 1 -- 172.21.15.67:0/4119324748 wait complete. 2024-04-17T07:14:37.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:37.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.685+0000 7f8d25200640 1 Processor -- start 2024-04-17T07:14:37.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.685+0000 7f8d25200640 1 -- start start 2024-04-17T07:14:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d25200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d20146060 0x7f8d20160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d25200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d20160920 con 0x7f8d20145b00 2024-04-17T07:14:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d25200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d20160c00 con 0x7f8d200b4ac0 2024-04-17T07:14:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d25200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8d20160ee0 con 0x7f8d20146060 2024-04-17T07:14:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d1f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d20145b00 0x7f8d2015cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57418/0 (socket says 172.21.15.67:57418) 2024-04-17T07:14:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.687+0000 7f8d1f400640 1 -- 172.21.15.67:0/3980306633 learned_addr learned my addr 172.21.15.67:0/3980306633 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:37.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d17e00640 1 --2- 172.21.15.67:0/3980306633 >> v2:172.21.15.187:3300/0 conn(0x7f8d20146060 0x7f8d20160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743416536 0 0) 0x7f8d20160920 con 0x7f8d20145b00 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d04003520 con 0x7f8d20145b00 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4050136583 0 0) 0x7f8d04003520 con 0x7f8d20145b00 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 >> v2:172.21.15.187:3300/0 conn(0x7f8d20146060 msgr2=0x7f8d20160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 --2- 172.21.15.67:0/3980306633 >> v2:172.21.15.187:3300/0 conn(0x7f8d20146060 0x7f8d20160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 >> v1:172.21.15.187:6789/0 conn(0x7f8d200b4ac0 legacy=0x7f8d200b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d201611c0 con 0x7f8d20145b00 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8d10001670 con 0x7f8d20145b00 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.688+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 >> v1:172.21.15.67:6789/0 conn(0x7f8d20145b00 legacy=0x7f8d2015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040051c0 0x7f8d0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d20160ee0 con 0x7f8d04004800 2024-04-17T07:14:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0400c160 con 0x7f8d040039b0 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0400c360 con 0x7f8d040051c0 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1f400640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 -- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040051c0 msgr2=0x7f8d0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040051c0 0x7f8d0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 -- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040039b0 msgr2=0x7f8d04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d17e00640 1 -- 172.21.15.67:0/3980306633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d04003520 con 0x7f8d04004800 2024-04-17T07:14:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.689+0000 7f8d1fe00640 1 -- 172.21.15.67:0/3980306633 reap_dead start 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d17e00640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009ac0 tx=0x7f8d04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee092b95fff821a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08039070 con 0x7f8d04004800 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d0802c710 con 0x7f8d04004800 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d1ea00640 1 -- 172.21.15.67:0/3980306633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08009690 con 0x7f8d04004800 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d25200640 1 -- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 msgr2=0x7f8d04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.691+0000 7f8d25200640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009ac0 tx=0x7f8d04005830 comp rx=0 tx=0).stop 2024-04-17T07:14:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.692+0000 7f8d25200640 1 -- 172.21.15.67:0/3980306633 shutdown_connections 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.692+0000 7f8d25200640 1 --2- 172.21.15.67:0/3980306633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.692+0000 7f8d25200640 1 -- 172.21.15.67:0/3980306633 >> 172.21.15.67:0/3980306633 conn(0x7f8d200b23c0 msgr2=0x7f8d200b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.692+0000 7f8d25200640 1 -- 172.21.15.67:0/3980306633 shutdown_connections 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.692+0000 7f8d25200640 1 -- 172.21.15.67:0/3980306633 wait complete. 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.694+0000 7f8d25200640 1 Processor -- start 2024-04-17T07:14:37.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.694+0000 7f8d25200640 1 -- start start 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d04004800 0x7f8d201ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 0x7f8d20144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d20160340 con 0x7f8d20135c90 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8d200cfba0 con 0x7f8d04004800 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d25200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d201604c0 con 0x7f8d20136110 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55410/0 (socket says 172.21.15.67:55410) 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 -- 172.21.15.67:0/1708815040 learned_addr learned my addr 172.21.15.67:0/1708815040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d1fe00640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 0x7f8d20144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d1f400640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d04004800 0x7f8d201ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 -- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 msgr2=0x7f8d20144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 0x7f8d20144610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 -- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d04004800 msgr2=0x7f8d201ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d04004800 0x7f8d201ea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.695+0000 7f8d17e00640 1 -- 172.21.15.67:0/1708815040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d201eac90 con 0x7f8d20135c90 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d1fe00640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 0x7f8d20144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d17e00640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d08037040 tx=0x7f8d08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08042020 con 0x7f8d20135c90 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d0803a040 con 0x7f8d20135c90 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08039070 con 0x7f8d20135c90 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d201e6410 con 0x7f8d20135c90 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.696+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d201e69d0 con 0x7f8d20135c90 2024-04-17T07:14:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.698+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d20068a10 con 0x7f8d20135c90 2024-04-17T07:14:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.699+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d08045020 con 0x7f8d20135c90 2024-04-17T07:14:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.700+0000 7f8d17400640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 0x7f8cfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.700+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8d080acdf0 con 0x7f8d20135c90 2024-04-17T07:14:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.700+0000 7f8d1f400640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 0x7f8cfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.701+0000 7f8d1f400640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 0x7f8cfc06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d10001200 tx=0x7f8d10001150 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:37.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.704+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8d0804c020 con 0x7f8d20135c90 2024-04-17T07:14:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.915+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d200d01e0 con 0x7f8d20135c90 2024-04-17T07:14:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.916+0000 7f8d17400640 1 -- 172.21.15.67:0/1708815040 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f8d0807b180 con 0x7f8d20135c90 2024-04-17T07:14:37.916 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 msgr2=0x7f8cfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 0x7f8cfc06edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d10001200 tx=0x7f8d10001150 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 msgr2=0x7f8d201440d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d08037040 tx=0x7f8d08004730 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 shutdown_connections 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8cfc06c8f0 0x7f8cfc06edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d20136110 0x7f8d20144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d20135c90 0x7f8d201440d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 --2- 172.21.15.67:0/1708815040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d04004800 0x7f8d201ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 >> 172.21.15.67:0/1708815040 conn(0x7f8d200b23c0 msgr2=0x7f8d200cd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 shutdown_connections 2024-04-17T07:14:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:37.919+0000 7f8d25200640 1 -- 172.21.15.67:0/1708815040 wait complete. 2024-04-17T07:14:37.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.159+0000 7fec37e00640 1 Processor -- start 2024-04-17T07:14:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.160+0000 7fec37e00640 1 -- start start 2024-04-17T07:14:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.161+0000 7fec37e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec3006f670 0x7fec3006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.161+0000 7fec37e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec300cd020 0x7fec300cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.161+0000 7fec37e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3006ff90 con 0x7fec30071d10 2024-04-17T07:14:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.161+0000 7fec37e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fec300ceb00 con 0x7fec300cd020 2024-04-17T07:14:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.161+0000 7fec37e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec300cede0 con 0x7fec3006f670 2024-04-17T07:14:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.162+0000 7fec34a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec300cd020 0x7fec300cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.162+0000 7fec35400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec3006f670 0x7fec3006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.162+0000 7fec34a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec300cd020 0x7fec300cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:60448/0 (socket says 172.21.15.67:60448) 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.162+0000 7fec34a00640 1 -- 172.21.15.67:0/1307583837 learned_addr learned my addr 172.21.15.67:0/1307583837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.163+0000 7fec34a00640 1 -- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec3006f670 msgr2=0x7fec3006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.163+0000 7fec34a00640 1 --2- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec3006f670 0x7fec3006fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.163+0000 7fec34a00640 1 -- 172.21.15.67:0/1307583837 >> v1:172.21.15.187:6789/0 conn(0x7fec30071d10 legacy=0x7fec300721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.163+0000 7fec34a00640 1 -- 172.21.15.67:0/1307583837 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec300cf0c0 con 0x7fec300cd020 2024-04-17T07:14:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.164+0000 7fec34a00640 1 --2- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec300cd020 0x7fec300cd480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009870 tx=0x7fec1c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e40430f20e5b842 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.165+0000 7fec2fe00640 1 -- 172.21.15.67:0/1307583837 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c034070 con 0x7fec300cd020 2024-04-17T07:14:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.165+0000 7fec2fe00640 1 -- 172.21.15.67:0/1307583837 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec1c030840 con 0x7fec300cd020 2024-04-17T07:14:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.165+0000 7fec2fe00640 1 -- 172.21.15.67:0/1307583837 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c030c10 con 0x7fec300cd020 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 -- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec300cd020 msgr2=0x7fec300cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 --2- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec300cd020 0x7fec300cd480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009870 tx=0x7fec1c02b980 comp rx=0 tx=0).stop 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 -- 172.21.15.67:0/1307583837 shutdown_connections 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 --2- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec300cd020 0x7fec300cd480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 --2- 172.21.15.67:0/1307583837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec3006f670 0x7fec3006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 -- 172.21.15.67:0/1307583837 >> 172.21.15.67:0/1307583837 conn(0x7fec3006bf50 msgr2=0x7fec300b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 -- 172.21.15.67:0/1307583837 shutdown_connections 2024-04-17T07:14:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.166+0000 7fec37e00640 1 -- 172.21.15.67:0/1307583837 wait complete. 2024-04-17T07:14:38.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.168+0000 7fec37e00640 1 Processor -- start 2024-04-17T07:14:38.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.168+0000 7fec37e00640 1 -- start start 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec3006f670 0x7fec30159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec300cd020 0x7fec3015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec300ce720 con 0x7fec30071d10 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec300cf680 con 0x7fec3006f670 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec37e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec300ce8a0 con 0x7fec300cd020 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec34a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec34a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55422/0 (socket says 172.21.15.67:55422) 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec34a00640 1 -- 172.21.15.67:0/1927043571 learned_addr learned my addr 172.21.15.67:0/1927043571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.170+0000 7fec35400640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec3006f670 0x7fec30159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec35e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec300cd020 0x7fec3015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 -- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec300cd020 msgr2=0x7fec3015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec300cd020 0x7fec3015d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 -- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec3006f670 msgr2=0x7fec30159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec3006f670 0x7fec30159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 -- 172.21.15.67:0/1927043571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec301f2a40 con 0x7fec30071d10 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec34a00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec1c000c00 tx=0x7fec1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c048070 con 0x7fec30071d10 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec1c030640 con 0x7fec30071d10 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec301f2d20 con 0x7fec30071d10 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.171+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1c034040 con 0x7fec30071d10 2024-04-17T07:14:38.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.172+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec301f3610 con 0x7fec30071d10 2024-04-17T07:14:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.173+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec30068b90 con 0x7fec30071d10 2024-04-17T07:14:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.174+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec1c04b070 con 0x7fec30071d10 2024-04-17T07:14:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.175+0000 7fec2d600640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 0x7fec1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.175+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fec1c0adae0 con 0x7fec30071d10 2024-04-17T07:14:38.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.175+0000 7fec35400640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 0x7fec1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.176+0000 7fec35400640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 0x7fec1006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec24009a30 tx=0x7fec24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.180+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fec1c07be70 con 0x7fec30071d10 2024-04-17T07:14:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.397+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec301f3850 con 0x7fec30071d10 2024-04-17T07:14:38.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.399+0000 7fec2d600640 1 -- 172.21.15.67:0/1927043571 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7fec1c030810 con 0x7fec30071d10 2024-04-17T07:14:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:38.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 msgr2=0x7fec1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 0x7fec1006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec24009a30 tx=0x7fec24008040 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 msgr2=0x7fec301f2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec1c000c00 tx=0x7fec1c004730 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 shutdown_connections 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fec1006c8f0 0x7fec1006edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec300cd020 0x7fec3015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30071d10 0x7fec301f2060 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 --2- 172.21.15.67:0/1927043571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec3006f670 0x7fec30159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 >> 172.21.15.67:0/1927043571 conn(0x7fec3006bf50 msgr2=0x7fec3006e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 shutdown_connections 2024-04-17T07:14:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.402+0000 7fec37e00640 1 -- 172.21.15.67:0/1927043571 wait complete. 2024-04-17T07:14:38.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:38.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:38.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.634+0000 7f15e5600640 1 Processor -- start 2024-04-17T07:14:38.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.634+0000 7f15e5600640 1 -- start start 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.635+0000 7f15e5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15e009aa60 0x7f15e009aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15e5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15e5600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e00b3cc0 con 0x7f15e00b38e0 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15e5600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f15e009b400 con 0x7f15e009aa60 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15e5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15e00d1790 con 0x7f15e00ccb20 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15e009aa60 0x7f15e009aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15e009aa60 0x7f15e009aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55428/0 (socket says 172.21.15.67:55428) 2024-04-17T07:14:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15df400640 1 -- 172.21.15.67:0/687318511 learned_addr learned my addr 172.21.15.67:0/687318511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.636+0000 7f15e4c00640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.637+0000 7f15df400640 1 -- 172.21.15.67:0/687318511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 msgr2=0x7f15e00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.637+0000 7f15df400640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.637+0000 7f15df400640 1 -- 172.21.15.67:0/687318511 >> v1:172.21.15.187:6789/0 conn(0x7f15e00b38e0 legacy=0x7f15e006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.637+0000 7f15df400640 1 -- 172.21.15.67:0/687318511 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e00d0b80 con 0x7f15e009aa60 2024-04-17T07:14:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.638+0000 7f15df400640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e009aec0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009870 tx=0x7f15cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c21dc2cb581bc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.638+0000 7f15dea00640 1 -- 172.21.15.67:0/687318511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc035020 con 0x7f15e009aa60 2024-04-17T07:14:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.638+0000 7f15dea00640 1 -- 172.21.15.67:0/687318511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15cc031640 con 0x7f15e009aa60 2024-04-17T07:14:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.638+0000 7f15dea00640 1 -- 172.21.15.67:0/687318511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc02d040 con 0x7f15e009aa60 2024-04-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 -- 172.21.15.67:0/687318511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 msgr2=0x7f15e009aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e009aec0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009870 tx=0x7f15cc02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 -- 172.21.15.67:0/687318511 shutdown_connections 2024-04-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 --2- 172.21.15.67:0/687318511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e009aec0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 -- 172.21.15.67:0/687318511 >> 172.21.15.67:0/687318511 conn(0x7f15e00b3fc0 msgr2=0x7f15e00b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 -- 172.21.15.67:0/687318511 shutdown_connections 2024-04-17T07:14:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.639+0000 7f15e5600640 1 -- 172.21.15.67:0/687318511 wait complete. 2024-04-17T07:14:38.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.641+0000 7f15e5600640 1 Processor -- start 2024-04-17T07:14:38.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.641+0000 7f15e5600640 1 -- start start 2024-04-17T07:14:38.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 0x7f15e015f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e015c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15e00d1420 con 0x7f15e009aa60 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15e00d1140 con 0x7f15e00b38e0 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15e009b9a0 con 0x7f15e00ccb20 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55432/0 (socket says 172.21.15.67:55432) 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 -- 172.21.15.67:0/1950951181 learned_addr learned my addr 172.21.15.67:0/1950951181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15df400640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 0x7f15e015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e4c00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 -- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 msgr2=0x7f15e015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e015c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 -- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 msgr2=0x7f15e015f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 0x7f15e015f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15dfe00640 1 -- 172.21.15.67:0/1950951181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e015fc90 con 0x7f15e009aa60 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15df400640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 0x7f15e015f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.643+0000 7f15e4c00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e015c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15dfe00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f15d000c460 tx=0x7f15d000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0016020 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d0012870 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0012c60 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e00cab60 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.644+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e00cb120 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.646+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e00755d0 con 0x7f15e009aa60 2024-04-17T07:14:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.647+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f15d0019020 con 0x7f15e009aa60 2024-04-17T07:14:38.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.647+0000 7f15d7e00640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:38.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.648+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f15d008b840 con 0x7f15e009aa60 2024-04-17T07:14:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.648+0000 7f15df400640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.649+0000 7f15df400640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 0x7f15c006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f15cc036000 tx=0x7f15cc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.652+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f15d0020030 con 0x7f15e009aa60 2024-04-17T07:14:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.857+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15e00cff30 con 0x7f15e009aa60 2024-04-17T07:14:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.858+0000 7f15d7e00640 1 -- 172.21.15.67:0/1950951181 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f15d0059c60 con 0x7f15e009aa60 2024-04-17T07:14:38.858 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.860+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 msgr2=0x7f15c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.860+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 0x7f15c006edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f15cc036000 tx=0x7f15cc008030 comp rx=0 tx=0).stop 2024-04-17T07:14:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 msgr2=0x7f15e0149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f15d000c460 tx=0x7f15d000c920 comp rx=0 tx=0).stop 2024-04-17T07:14:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 shutdown_connections 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f15c006c8f0 0x7f15c006edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15e00ccb20 0x7f15e015c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15e00b38e0 0x7f15e015f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 --2- 172.21.15.67:0/1950951181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e009aa60 0x7f15e0149490 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 >> 172.21.15.67:0/1950951181 conn(0x7f15e00b3fc0 msgr2=0x7f15e00b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 shutdown_connections 2024-04-17T07:14:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:38.861+0000 7f15e5600640 1 -- 172.21.15.67:0/1950951181 wait complete. 2024-04-17T07:14:38.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.111+0000 7faef3e00640 1 Processor -- start 2024-04-17T07:14:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.111+0000 7faef3e00640 1 -- start start 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faef415b6b0 0x7faef406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 0x7faef406be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef406df40 con 0x7faef406f800 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faef406ed00 con 0x7faef415b6b0 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faef406e870 con 0x7faef40cfc10 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faef406f800 0x7faef415f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57420/0 (socket says 172.21.15.67:57420) 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef2000640 1 -- 172.21.15.67:0/3665047792 learned_addr learned my addr 172.21.15.67:0/3665047792 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef2a00640 1 --2- 172.21.15.67:0/3665047792 >> v2:172.21.15.67:3300/0 conn(0x7faef415b6b0 0x7faef406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.113+0000 7faef3400640 1 --2- 172.21.15.67:0/3665047792 >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 0x7faef406be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef1600640 1 -- 172.21.15.67:0/3665047792 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2687351404 0 0) 0x7faef406df40 con 0x7faef406f800 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef1600640 1 -- 172.21.15.67:0/3665047792 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faed4005730 con 0x7faef406f800 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef2a00640 1 -- 172.21.15.67:0/3665047792 >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 msgr2=0x7faef406be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef2a00640 1 --2- 172.21.15.67:0/3665047792 >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 0x7faef406be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef2a00640 1 -- 172.21.15.67:0/3665047792 >> v1:172.21.15.67:6789/0 conn(0x7faef406f800 legacy=0x7faef415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef2a00640 1 -- 172.21.15.67:0/3665047792 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef406e220 con 0x7faef415b6b0 2024-04-17T07:14:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.114+0000 7faef3400640 1 --2- 172.21.15.67:0/3665047792 >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 0x7faef406be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.115+0000 7faef2a00640 1 --2- 172.21.15.67:0/3665047792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef415b6b0 0x7faef406b8c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faef415f050 tx=0x7faee402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674932e5d035d907 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.115+0000 7faef1600640 1 -- 172.21.15.67:0/3665047792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee40315a0 con 0x7faef415b6b0 2024-04-17T07:14:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.115+0000 7faef1600640 1 -- 172.21.15.67:0/3665047792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee4037070 con 0x7faef415b6b0 2024-04-17T07:14:39.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.115+0000 7faef1600640 1 -- 172.21.15.67:0/3665047792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4031960 con 0x7faef415b6b0 2024-04-17T07:14:39.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 -- 172.21.15.67:0/3665047792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef415b6b0 msgr2=0x7faef406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 --2- 172.21.15.67:0/3665047792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef415b6b0 0x7faef406b8c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faef415f050 tx=0x7faee402b840 comp rx=0 tx=0).stop 2024-04-17T07:14:39.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 -- 172.21.15.67:0/3665047792 shutdown_connections 2024-04-17T07:14:39.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 --2- 172.21.15.67:0/3665047792 >> v2:172.21.15.187:3300/0 conn(0x7faef40cfc10 0x7faef406be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 --2- 172.21.15.67:0/3665047792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef415b6b0 0x7faef406b8c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 -- 172.21.15.67:0/3665047792 >> 172.21.15.67:0/3665047792 conn(0x7faef40b2190 msgr2=0x7faef40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:39.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 -- 172.21.15.67:0/3665047792 shutdown_connections 2024-04-17T07:14:39.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.116+0000 7faef3e00640 1 -- 172.21.15.67:0/3665047792 wait complete. 2024-04-17T07:14:39.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.118+0000 7faef3e00640 1 Processor -- start 2024-04-17T07:14:39.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.118+0000 7faef3e00640 1 -- start start 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faef40cfc10 0x7faef40d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faef415b6b0 0x7faef40d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faef40afee0 con 0x7faef406f800 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faef40afd60 con 0x7faef40cfc10 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faef40b44a0 con 0x7faef415b6b0 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.119+0000 7faef2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55444/0 (socket says 172.21.15.67:55444) 2024-04-17T07:14:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 -- 172.21.15.67:0/1480716922 learned_addr learned my addr 172.21.15.67:0/1480716922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef3400640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faef415b6b0 0x7faef40d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2000640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faef40cfc10 0x7faef40d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 -- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faef415b6b0 msgr2=0x7faef40d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faef415b6b0 0x7faef40d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 -- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faef40cfc10 msgr2=0x7faef40d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faef40cfc10 0x7faef40d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 -- 172.21.15.67:0/1480716922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef41452e0 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.120+0000 7faef2a00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faee4009890 tx=0x7faee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.121+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4032630 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.121+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee40327f0 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.121+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4032be0 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.121+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef415d7a0 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.121+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef41455a0 con 0x7faef406f800 2024-04-17T07:14:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.122+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef4068a10 con 0x7faef406f800 2024-04-17T07:14:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.123+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faee404c020 con 0x7faef406f800 2024-04-17T07:14:39.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.124+0000 7faeeb400640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 0x7faed006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.124+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7faee402f020 con 0x7faef406f800 2024-04-17T07:14:39.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.125+0000 7faef2000640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 0x7faed006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.126+0000 7faef2000640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 0x7faed006edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faef4158680 tx=0x7faedc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.129+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faee4040070 con 0x7faef406f800 2024-04-17T07:14:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.333+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faef40cc910 con 0x7faef406f800 2024-04-17T07:14:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.334+0000 7faeeb400640 1 -- 172.21.15.67:0/1480716922 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7faee407bce0 con 0x7faef406f800 2024-04-17T07:14:39.335 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 msgr2=0x7faed006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 0x7faed006edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faef4158680 tx=0x7faedc009040 comp rx=0 tx=0).stop 2024-04-17T07:14:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 msgr2=0x7faef4144b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faee4009890 tx=0x7faee40046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 shutdown_connections 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faed006c8f0 0x7faed006edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faef415b6b0 0x7faef40d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faef40cfc10 0x7faef40d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 --2- 172.21.15.67:0/1480716922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef406f800 0x7faef4144b40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 >> 172.21.15.67:0/1480716922 conn(0x7faef40b2190 msgr2=0x7faef40ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 shutdown_connections 2024-04-17T07:14:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.337+0000 7faef3e00640 1 -- 172.21.15.67:0/1480716922 wait complete. 2024-04-17T07:14:39.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:39.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.583+0000 7f222fe00640 1 Processor -- start 2024-04-17T07:14:39.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.583+0000 7f222fe00640 1 -- start start 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2230145b00 0x7f2230160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 0x7f223015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2230160920 con 0x7f22300b4ac0 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2230160c00 con 0x7f2230145b00 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2230160ee0 con 0x7f2230146060 2024-04-17T07:14:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2230145b00 0x7f2230160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f22300b4ac0 0x7f22300b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57432/0 (socket says 172.21.15.67:57432) 2024-04-17T07:14:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.585+0000 7f222f400640 1 -- 172.21.15.67:0/3209716869 learned_addr learned my addr 172.21.15.67:0/3209716869 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222e000640 1 --2- 172.21.15.67:0/3209716869 >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 0x7f223015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222ea00640 1 -- 172.21.15.67:0/3209716869 >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 msgr2=0x7f223015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222ea00640 1 --2- 172.21.15.67:0/3209716869 >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 0x7f223015cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222ea00640 1 -- 172.21.15.67:0/3209716869 >> v1:172.21.15.67:6789/0 conn(0x7f22300b4ac0 legacy=0x7f22300b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222d600640 1 -- 172.21.15.67:0/3209716869 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894887784 0 0) 0x7f2230160920 con 0x7f22300b4ac0 2024-04-17T07:14:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222ea00640 1 -- 172.21.15.67:0/3209716869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22301611c0 con 0x7f2230145b00 2024-04-17T07:14:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.586+0000 7f222e000640 1 --2- 172.21.15.67:0/3209716869 >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 0x7f223015cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.587+0000 7f222ea00640 1 --2- 172.21.15.67:0/3209716869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2230145b00 0x7f2230160340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2230146520 tx=0x7f222002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eabdbf66f3d6bed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.587+0000 7f222d600640 1 -- 172.21.15.67:0/3209716869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22200315a0 con 0x7f2230145b00 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.587+0000 7f222d600640 1 -- 172.21.15.67:0/3209716869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2220037070 con 0x7f2230145b00 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.587+0000 7f222d600640 1 -- 172.21.15.67:0/3209716869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2220031960 con 0x7f2230145b00 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 -- 172.21.15.67:0/3209716869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2230145b00 msgr2=0x7f2230160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 --2- 172.21.15.67:0/3209716869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2230145b00 0x7f2230160340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2230146520 tx=0x7f222002b840 comp rx=0 tx=0).stop 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 -- 172.21.15.67:0/3209716869 shutdown_connections 2024-04-17T07:14:39.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 --2- 172.21.15.67:0/3209716869 >> v2:172.21.15.187:3300/0 conn(0x7f2230146060 0x7f223015cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 --2- 172.21.15.67:0/3209716869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2230145b00 0x7f2230160340 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 -- 172.21.15.67:0/3209716869 >> 172.21.15.67:0/3209716869 conn(0x7f22300b23c0 msgr2=0x7f22300b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:39.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 -- 172.21.15.67:0/3209716869 shutdown_connections 2024-04-17T07:14:39.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.588+0000 7f222fe00640 1 -- 172.21.15.67:0/3209716869 wait complete. 2024-04-17T07:14:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.590+0000 7f222fe00640 1 Processor -- start 2024-04-17T07:14:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.590+0000 7f222fe00640 1 -- start start 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2230145b00 0x7f223013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2230146060 0x7f223013a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f223015d2c0 con 0x7f22300b4ac0 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f22300cfba0 con 0x7f2230145b00 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f22300b4390 con 0x7f2230146060 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55454/0 (socket says 172.21.15.67:55454) 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.591+0000 7f222ea00640 1 -- 172.21.15.67:0/2957307787 learned_addr learned my addr 172.21.15.67:0/2957307787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:39.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 -- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2230146060 msgr2=0x7f223013a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2230146060 0x7f223013a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222e000640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2230145b00 0x7f223013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 -- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2230145b00 msgr2=0x7f223013c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2230145b00 0x7f223013c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 -- 172.21.15.67:0/2957307787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2230133270 con 0x7f22300b4ac0 2024-04-17T07:14:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.592+0000 7f222ea00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2220009890 tx=0x7f22200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.593+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2220032840 con 0x7f22300b4ac0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.593+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2220032a00 con 0x7f22300b4ac0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.593+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223013d870 con 0x7f22300b4ac0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.593+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2220032df0 con 0x7f22300b4ac0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.593+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223013d590 con 0x7f22300b4ac0 2024-04-17T07:14:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.594+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22300cfd90 con 0x7f22300b4ac0 2024-04-17T07:14:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.595+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f222004c020 con 0x7f22300b4ac0 2024-04-17T07:14:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.596+0000 7f2226a00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 0x7f221006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.596+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f222002f020 con 0x7f22300b4ac0 2024-04-17T07:14:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.597+0000 7f222e000640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 0x7f221006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.598+0000 7f222e000640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 0x7f221006edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f223013df00 tx=0x7f221c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:39.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.601+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f22300cfd90 con 0x7f22300b4ac0 2024-04-17T07:14:39.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.805+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2230068a10 con 0x7f22300b4ac0 2024-04-17T07:14:39.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.806+0000 7f2226a00640 1 -- 172.21.15.67:0/2957307787 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f2220040070 con 0x7f22300b4ac0 2024-04-17T07:14:39.806 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 msgr2=0x7f221006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 0x7f221006edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f223013df00 tx=0x7f221c008040 comp rx=0 tx=0).stop 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 msgr2=0x7f2230132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2220009890 tx=0x7f22200046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 shutdown_connections 2024-04-17T07:14:39.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f221006c8f0 0x7f221006edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2230146060 0x7f223013a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2230145b00 0x7f223013c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 --2- 172.21.15.67:0/2957307787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22300b4ac0 0x7f2230132d30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 >> 172.21.15.67:0/2957307787 conn(0x7f22300b23c0 msgr2=0x7f22300cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 shutdown_connections 2024-04-17T07:14:39.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:39.809+0000 7f222fe00640 1 -- 172.21.15.67:0/2957307787 wait complete. 2024-04-17T07:14:39.869 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-04-17T07:14:39.869 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:14:39.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.093+0000 7fcefda00640 1 Processor -- start 2024-04-17T07:14:40.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.094+0000 7fcefda00640 1 -- start start 2024-04-17T07:14:40.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.095+0000 7fcefda00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcef806f670 0x7fcef806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.095+0000 7fcefda00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef8072b30 con 0x7fcef806f030 2024-04-17T07:14:40.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.095+0000 7fcefda00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef806ff90 con 0x7fcef80cd020 2024-04-17T07:14:40.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.095+0000 7fcefda00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcef80ceb00 con 0x7fcef806f670 2024-04-17T07:14:40.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcef806f030 0x7fcef8072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57436/0 (socket says 172.21.15.67:57436) 2024-04-17T07:14:40.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef7400640 1 -- 172.21.15.67:0/2907949785 learned_addr learned my addr 172.21.15.67:0/2907949785 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef6a00640 1 --2- 172.21.15.67:0/2907949785 >> v2:172.21.15.187:3300/0 conn(0x7fcef806f670 0x7fcef806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592115081 0 0) 0x7fcef8072b30 con 0x7fcef806f030 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcedc003520 con 0x7fcef806f030 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.096+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 518443613 0 0) 0x7fcedc003520 con 0x7fcef806f030 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 >> v2:172.21.15.187:3300/0 conn(0x7fcef806f670 msgr2=0x7fcef806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 --2- 172.21.15.67:0/2907949785 >> v2:172.21.15.187:3300/0 conn(0x7fcef806f670 0x7fcef806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 >> v1:172.21.15.187:6789/0 conn(0x7fcef80cd020 legacy=0x7fcef8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef80cede0 con 0x7fcef806f030 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcef0001670 con 0x7fcef806f030 2024-04-17T07:14:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 >> v1:172.21.15.67:6789/0 conn(0x7fcef806f030 legacy=0x7fcef8072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc003a00 0x7fcedc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 0x7fcedc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcedc0051c0 0x7fcedc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcef80ceb00 con 0x7fcedc004800 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc00c0c0 con 0x7fcedc0051c0 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcedc00c2c0 con 0x7fcedc003a00 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef6000640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 0x7fcedc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef6a00640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc003a00 0x7fcedc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.097+0000 7fcef7400640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcedc0051c0 0x7fcedc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef6000640 1 -- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc003a00 msgr2=0x7fcedc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef6000640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc003a00 0x7fcedc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef6000640 1 -- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcedc0051c0 msgr2=0x7fcedc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:40.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef6000640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcedc0051c0 0x7fcedc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef6000640 1 -- 172.21.15.67:0/2907949785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcedc003520 con 0x7fcedc004800 2024-04-17T07:14:40.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.098+0000 7fcef7400640 1 -- 172.21.15.67:0/2907949785 reap_dead start 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcef6000640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 0x7fcedc004c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcee0001750 tx=0x7fcedc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2473c7c9c8e8942 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0001370 con 0x7fcedc004800 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00052d0 con 0x7fcedc004800 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcef5600640 1 -- 172.21.15.67:0/2907949785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0003830 con 0x7fcedc004800 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcefda00640 1 -- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 msgr2=0x7fcedc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.099+0000 7fcefda00640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 0x7fcedc004c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcee0001750 tx=0x7fcedc005830 comp rx=0 tx=0).stop 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.100+0000 7fcefda00640 1 -- 172.21.15.67:0/2907949785 shutdown_connections 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.100+0000 7fcefda00640 1 --2- 172.21.15.67:0/2907949785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcedc004800 0x7fcedc004c80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.100+0000 7fcefda00640 1 -- 172.21.15.67:0/2907949785 >> 172.21.15.67:0/2907949785 conn(0x7fcef806bf50 msgr2=0x7fcef80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:40.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.100+0000 7fcefda00640 1 -- 172.21.15.67:0/2907949785 shutdown_connections 2024-04-17T07:14:40.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.100+0000 7fcefda00640 1 -- 172.21.15.67:0/2907949785 wait complete. 2024-04-17T07:14:40.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.102+0000 7fcefda00640 1 Processor -- start 2024-04-17T07:14:40.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.102+0000 7fcefda00640 1 -- start start 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc004800 0x7fcef8151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcef815b120 0x7fcef815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcef80cf680 con 0x7fcef815cb00 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcef80cf3a0 con 0x7fcef815b120 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcefda00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcef80ce720 con 0x7fcedc004800 2024-04-17T07:14:40.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55472/0 (socket says 172.21.15.67:55472) 2024-04-17T07:14:40.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.103+0000 7fcef7400640 1 -- 172.21.15.67:0/2491272303 learned_addr learned my addr 172.21.15.67:0/2491272303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef6000640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcef815b120 0x7fcef815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef6a00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc004800 0x7fcef8151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 -- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc004800 msgr2=0x7fcef8151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc004800 0x7fcef8151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 -- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcef815b120 msgr2=0x7fcef815b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcef815b120 0x7fcef815b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 -- 172.21.15.67:0/2491272303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef815d7c0 con 0x7fcef815cb00 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcef7400640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcef8072f40 tx=0x7fcef000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0019070 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef0010510 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0010900 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.104+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef815daa0 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.105+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef815e060 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.106+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8068b90 con 0x7fcef815cb00 2024-04-17T07:14:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.107+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcef0010aa0 con 0x7fcef815cb00 2024-04-17T07:14:40.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.108+0000 7fceeea00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 0x7fced406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.108+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcef008ad70 con 0x7fcef815cb00 2024-04-17T07:14:40.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.108+0000 7fcef6a00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 0x7fced406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.110+0000 7fcef6a00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 0x7fced406ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcef80516a0 tx=0x7fcee800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.112+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcef00591a0 con 0x7fcef815cb00 2024-04-17T07:14:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.316+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcef8158c80 con 0x7fcef815cb00 2024-04-17T07:14:40.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.319+0000 7fceeea00640 1 -- 172.21.15.67:0/2491272303 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7fcef0058aa0 con 0x7fcef815cb00 2024-04-17T07:14:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:40.320 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:40.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 msgr2=0x7fced406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 0x7fced406ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcef80516a0 tx=0x7fcee800b040 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 msgr2=0x7fcef815cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcef8072f40 tx=0x7fcef000e960 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 shutdown_connections 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcef815cb00 0x7fcef815cf80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcef815b120 0x7fcef815b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fced406c920 0x7fced406ede0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 --2- 172.21.15.67:0/2491272303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcedc004800 0x7fcef8151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 >> 172.21.15.67:0/2491272303 conn(0x7fcef806bf50 msgr2=0x7fcef8071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 shutdown_connections 2024-04-17T07:14:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.322+0000 7fcefda00640 1 -- 172.21.15.67:0/2491272303 wait complete. 2024-04-17T07:14:40.344 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:40.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:40.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.560+0000 7fa9f6200640 1 Processor -- start 2024-04-17T07:14:40.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.560+0000 7fa9f6200640 1 -- start start 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.561+0000 7fa9f6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9f006f0d0 0x7fa9f00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.561+0000 7fa9f6200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa9f015ac50 0x7fa9f00b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.561+0000 7fa9f6200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f006e100 con 0x7fa9f0072eb0 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.561+0000 7fa9f6200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f006deb0 con 0x7fa9f006f0d0 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.561+0000 7fa9f6200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f006ea30 con 0x7fa9f015ac50 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9f006f0d0 0x7fa9f00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9f006f0d0 0x7fa9f00ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55484/0 (socket says 172.21.15.67:55484) 2024-04-17T07:14:40.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 -- 172.21.15.67:0/1037222950 learned_addr learned my addr 172.21.15.67:0/1037222950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f5800640 1 --2- 172.21.15.67:0/1037222950 >> v2:172.21.15.187:3300/0 conn(0x7fa9f015ac50 0x7fa9f00b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 -- 172.21.15.67:0/1037222950 >> v2:172.21.15.187:3300/0 conn(0x7fa9f015ac50 msgr2=0x7fa9f00b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 --2- 172.21.15.67:0/1037222950 >> v2:172.21.15.187:3300/0 conn(0x7fa9f015ac50 0x7fa9f00b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 -- 172.21.15.67:0/1037222950 >> v1:172.21.15.187:6789/0 conn(0x7fa9f0072eb0 legacy=0x7fa9f00cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.562+0000 7fa9f4e00640 1 -- 172.21.15.67:0/1037222950 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f006e3e0 con 0x7fa9f006f0d0 2024-04-17T07:14:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.563+0000 7fa9f4e00640 1 --2- 172.21.15.67:0/1037222950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f006f0d0 0x7fa9f00ce510 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00b02a0 tx=0x7fa9e002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62126519c3496a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9ef400640 1 -- 172.21.15.67:0/1037222950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0034070 con 0x7fa9f006f0d0 2024-04-17T07:14:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9ef400640 1 -- 172.21.15.67:0/1037222950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e0038400 con 0x7fa9f006f0d0 2024-04-17T07:14:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9ef400640 1 -- 172.21.15.67:0/1037222950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00387d0 con 0x7fa9f006f0d0 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9f6200640 1 -- 172.21.15.67:0/1037222950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f006f0d0 msgr2=0x7fa9f00ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1037222950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f006f0d0 0x7fa9f00ce510 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00b02a0 tx=0x7fa9e002e0f0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9f6200640 1 -- 172.21.15.67:0/1037222950 shutdown_connections 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1037222950 >> v2:172.21.15.187:3300/0 conn(0x7fa9f015ac50 0x7fa9f00b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.564+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1037222950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f006f0d0 0x7fa9f00ce510 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.565+0000 7fa9f6200640 1 -- 172.21.15.67:0/1037222950 >> 172.21.15.67:0/1037222950 conn(0x7fa9f00b23c0 msgr2=0x7fa9f00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.565+0000 7fa9f6200640 1 -- 172.21.15.67:0/1037222950 shutdown_connections 2024-04-17T07:14:40.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.565+0000 7fa9f6200640 1 -- 172.21.15.67:0/1037222950 wait complete. 2024-04-17T07:14:40.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.566+0000 7fa9f6200640 1 Processor -- start 2024-04-17T07:14:40.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.566+0000 7fa9f6200640 1 -- start start 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9f0072eb0 0x7fa9f0148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9f00b12e0 0x7fa9f0151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f00cf2e0 con 0x7fa9f015ac50 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f00cf160 con 0x7fa9f00b12e0 2024-04-17T07:14:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.567+0000 7fa9f6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f006ff30 con 0x7fa9f0072eb0 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55486/0 (socket says 172.21.15.67:55486) 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 -- 172.21.15.67:0/1218111820 learned_addr learned my addr 172.21.15.67:0/1218111820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9f4e00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9f0072eb0 0x7fa9f0148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9f5800640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9f00b12e0 0x7fa9f0151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 -- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9f0072eb0 msgr2=0x7fa9f0148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9f0072eb0 0x7fa9f0148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 -- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9f00b12e0 msgr2=0x7fa9f0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9f00b12e0 0x7fa9f0151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.568+0000 7fa9efe00640 1 -- 172.21.15.67:0/1218111820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f01494c0 con 0x7fa9f015ac50 2024-04-17T07:14:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9efe00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c430 tx=0x7fa9dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc017070 con 0x7fa9f015ac50 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9dc0035f0 con 0x7fa9f015ac50 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0039e0 con 0x7fa9f015ac50 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f01f8ac0 con 0x7fa9f015ac50 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.569+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f01f7d00 con 0x7fa9f015ac50 2024-04-17T07:14:40.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.570+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f0068a10 con 0x7fa9f015ac50 2024-04-17T07:14:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.571+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9dc01a070 con 0x7fa9f015ac50 2024-04-17T07:14:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.572+0000 7fa9ecc00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 0x7fa9d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.572+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa9dc08ae60 con 0x7fa9f015ac50 2024-04-17T07:14:40.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.573+0000 7fa9f4e00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 0x7fa9d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.574+0000 7fa9f4e00640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 0x7fa9d006edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0039000 tx=0x7fa9e000a040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.577+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa9f01494c0 con 0x7fa9f015ac50 2024-04-17T07:14:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.781+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa9f0068610 con 0x7fa9f015ac50 2024-04-17T07:14:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.782+0000 7fa9ecc00640 1 -- 172.21.15.67:0/1218111820 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa9dc0010c0 con 0x7fa9f015ac50 2024-04-17T07:14:40.782 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 msgr2=0x7fa9d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 0x7fa9d006edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0039000 tx=0x7fa9e000a040 comp rx=0 tx=0).stop 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 msgr2=0x7fa9f00b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c430 tx=0x7fa9dc00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 shutdown_connections 2024-04-17T07:14:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9f00b12e0 0x7fa9f0151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f015ac50 0x7fa9f00b0da0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa9d006c8f0 0x7fa9d006edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 --2- 172.21.15.67:0/1218111820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9f0072eb0 0x7fa9f0148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 >> 172.21.15.67:0/1218111820 conn(0x7fa9f00b23c0 msgr2=0x7fa9f00b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 shutdown_connections 2024-04-17T07:14:40.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:40.785+0000 7fa9f6200640 1 -- 172.21.15.67:0/1218111820 wait complete. 2024-04-17T07:14:40.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:41.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.094+0000 7f8b77200640 1 Processor -- start 2024-04-17T07:14:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.095+0000 7f8b77200640 1 -- start start 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.096+0000 7f8b77200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b7806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.096+0000 7f8b77200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b7806c0a0 0x7f8b7806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.096+0000 7f8b77200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b78146360 con 0x7f8b78145f80 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.096+0000 7f8b77200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8b780727a0 con 0x7f8b7806c0a0 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b77200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b78071e70 con 0x7f8b780cd920 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b75e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b78145f80 0x7f8b780cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57438/0 (socket says 172.21.15.67:57438) 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b75e00640 1 -- 172.21.15.67:0/2369183403 learned_addr learned my addr 172.21.15.67:0/2369183403 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b76800640 1 --2- 172.21.15.67:0/2369183403 >> v2:172.21.15.67:3300/0 conn(0x7f8b7806c0a0 0x7f8b7806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b75400640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b7806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.097+0000 7f8b74a00640 1 -- 172.21.15.67:0/2369183403 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 39927058 0 0) 0x7f8b78146360 con 0x7f8b78145f80 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.098+0000 7f8b74a00640 1 -- 172.21.15.67:0/2369183403 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b58003520 con 0x7f8b78145f80 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.098+0000 7f8b76800640 1 -- 172.21.15.67:0/2369183403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 msgr2=0x7f8b7806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.098+0000 7f8b76800640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b7806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.098+0000 7f8b76800640 1 -- 172.21.15.67:0/2369183403 >> v1:172.21.15.67:6789/0 conn(0x7f8b78145f80 legacy=0x7f8b780cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.098+0000 7f8b76800640 1 -- 172.21.15.67:0/2369183403 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b78072150 con 0x7f8b7806c0a0 2024-04-17T07:14:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.099+0000 7f8b76800640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b7806c0a0 0x7f8b7806e1c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009d90 tx=0x7f8b7002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9767c991e2287d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.099+0000 7f8b74a00640 1 -- 172.21.15.67:0/2369183403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70039070 con 0x7f8b7806c0a0 2024-04-17T07:14:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.099+0000 7f8b74a00640 1 -- 172.21.15.67:0/2369183403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b7002cbc0 con 0x7f8b7806c0a0 2024-04-17T07:14:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.099+0000 7f8b74a00640 1 -- 172.21.15.67:0/2369183403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70034440 con 0x7f8b7806c0a0 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 -- 172.21.15.67:0/2369183403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b7806c0a0 msgr2=0x7f8b7806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b7806c0a0 0x7f8b7806e1c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009d90 tx=0x7f8b7002be10 comp rx=0 tx=0).stop 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 -- 172.21.15.67:0/2369183403 shutdown_connections 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b7806c0a0 0x7f8b7806e1c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 --2- 172.21.15.67:0/2369183403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b7806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 -- 172.21.15.67:0/2369183403 >> 172.21.15.67:0/2369183403 conn(0x7f8b780b2190 msgr2=0x7f8b780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 -- 172.21.15.67:0/2369183403 shutdown_connections 2024-04-17T07:14:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.100+0000 7f8b77200640 1 -- 172.21.15.67:0/2369183403 wait complete. 2024-04-17T07:14:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.102+0000 7f8b77200640 1 Processor -- start 2024-04-17T07:14:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.102+0000 7f8b77200640 1 -- start start 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 0x7f8b7815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b781e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b78072db0 con 0x7f8b78145f80 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b78072c30 con 0x7f8b7806c0a0 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b77200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7806c4a0 con 0x7f8b780cd920 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b76800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b76800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55506/0 (socket says 172.21.15.67:55506) 2024-04-17T07:14:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.103+0000 7f8b76800640 1 -- 172.21.15.67:0/2687107010 learned_addr learned my addr 172.21.15.67:0/2687107010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b75e00640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 0x7f8b7815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b75400640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b781e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 -- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 msgr2=0x7f8b781e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b781e85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 -- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 msgr2=0x7f8b7815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 0x7f8b7815c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 -- 172.21.15.67:0/2687107010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b781e69a0 con 0x7f8b78145f80 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b75e00640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 0x7f8b7815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b75400640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b781e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.104+0000 7f8b76800640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009eb0 tx=0x7f8b70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.105+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70044020 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.105+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b70035e50 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.105+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70039040 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.105+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b781e6c80 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.105+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b781e7240 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.106+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b78068a10 con 0x7f8b78145f80 2024-04-17T07:14:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.107+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b70047020 con 0x7f8b78145f80 2024-04-17T07:14:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.108+0000 7f8b6e000640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 0x7f8b5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.108+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8b700ae8a0 con 0x7f8b78145f80 2024-04-17T07:14:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.109+0000 7f8b75e00640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 0x7f8b5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.110+0000 7f8b75e00640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 0x7f8b5406edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8b68005880 tx=0x7f8b68009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.112+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8b7004e020 con 0x7f8b78145f80 2024-04-17T07:14:41.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.316+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b780b0d20 con 0x7f8b78145f80 2024-04-17T07:14:41.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.317+0000 7f8b6e000640 1 -- 172.21.15.67:0/2687107010 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f8b7007cc30 con 0x7f8b78145f80 2024-04-17T07:14:41.317 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 msgr2=0x7f8b5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 0x7f8b5406edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8b68005880 tx=0x7f8b68009040 comp rx=0 tx=0).stop 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 msgr2=0x7f8b781e8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009eb0 tx=0x7f8b70004a40 comp rx=0 tx=0).stop 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 shutdown_connections 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78145f80 0x7f8b781e8b00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f8b5406c8f0 0x7f8b5406edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b780cd920 0x7f8b781e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 --2- 172.21.15.67:0/2687107010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b7806c0a0 0x7f8b7815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 >> 172.21.15.67:0/2687107010 conn(0x7f8b780b2190 msgr2=0x7f8b780b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 shutdown_connections 2024-04-17T07:14:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.320+0000 7f8b77200640 1 -- 172.21.15.67:0/2687107010 wait complete. 2024-04-17T07:14:41.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.568+0000 7fd18ce00640 1 Processor -- start 2024-04-17T07:14:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.568+0000 7fd18ce00640 1 -- start start 2024-04-17T07:14:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd18ce00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1880d1970 0x7fd18815be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd18ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1880d03f0 con 0x7fd188071650 2024-04-17T07:14:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd18ce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1880b5030 con 0x7fd18815ba70 2024-04-17T07:14:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd18ce00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd18815c390 con 0x7fd1880d1970 2024-04-17T07:14:41.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd186800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd188071650 0x7fd1880b5b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57448/0 (socket says 172.21.15.67:57448) 2024-04-17T07:14:41.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd186800640 1 -- 172.21.15.67:0/2154001116 learned_addr learned my addr 172.21.15.67:0/2154001116 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.570+0000 7fd185400640 1 --2- 172.21.15.67:0/2154001116 >> v2:172.21.15.187:3300/0 conn(0x7fd1880d1970 0x7fd18815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229121733 0 0) 0x7fd1880d03f0 con 0x7fd188071650 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003520 con 0x7fd188071650 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2857492247 0 0) 0x7fd168003520 con 0x7fd188071650 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 >> v2:172.21.15.187:3300/0 conn(0x7fd1880d1970 msgr2=0x7fd18815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 --2- 172.21.15.67:0/2154001116 >> v2:172.21.15.187:3300/0 conn(0x7fd1880d1970 0x7fd18815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 >> v1:172.21.15.187:6789/0 conn(0x7fd18815ba70 legacy=0x7fd1880d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1880b5310 con 0x7fd188071650 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd1800019f0 con 0x7fd188071650 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 >> v1:172.21.15.67:6789/0 conn(0x7fd188071650 legacy=0x7fd1880b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1680039b0 0x7fd168003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd168004800 0x7fd168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 0x7fd16800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd18815c390 con 0x7fd1680051c0 2024-04-17T07:14:41.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd16800c160 con 0x7fd1680039b0 2024-04-17T07:14:41.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.571+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd16800c360 con 0x7fd168004800 2024-04-17T07:14:41.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 0x7fd16800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd185400640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd168004800 0x7fd168004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd185e00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1680039b0 0x7fd168003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 -- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd168004800 msgr2=0x7fd168004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd168004800 0x7fd168004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 -- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1680039b0 msgr2=0x7fd168003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1680039b0 0x7fd168003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.572+0000 7fd186800640 1 -- 172.21.15.67:0/2154001116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd168003520 con 0x7fd1680051c0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.573+0000 7fd186800640 1 -- 172.21.15.67:0/2154001116 reap_dead start 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.573+0000 7fd186800640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 0x7fd16800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd18000a840 tx=0x7fd168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b63724f0e53c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000a460 con 0x7fd1680051c0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd180034650 con 0x7fd1680051c0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd184a00640 1 -- 172.21.15.67:0/2154001116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180034a40 con 0x7fd1680051c0 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 -- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 msgr2=0x7fd16800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 0x7fd16800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd18000a840 tx=0x7fd168005830 comp rx=0 tx=0).stop 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 -- 172.21.15.67:0/2154001116 shutdown_connections 2024-04-17T07:14:41.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 --2- 172.21.15.67:0/2154001116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1680051c0 0x7fd16800b9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 -- 172.21.15.67:0/2154001116 >> 172.21.15.67:0/2154001116 conn(0x7fd1880b3390 msgr2=0x7fd1880b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 -- 172.21.15.67:0/2154001116 shutdown_connections 2024-04-17T07:14:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.574+0000 7fd18ce00640 1 -- 172.21.15.67:0/2154001116 wait complete. 2024-04-17T07:14:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.576+0000 7fd18ce00640 1 Processor -- start 2024-04-17T07:14:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.576+0000 7fd18ce00640 1 -- start start 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.577+0000 7fd18ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1680051c0 0x7fd1881e5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.577+0000 7fd18ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd18ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd188153ef0 0x7fd188159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd18ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1880b4c50 con 0x7fd1881e6d50 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd18ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1880b5680 con 0x7fd188153ef0 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd18ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1880b4dd0 con 0x7fd1680051c0 2024-04-17T07:14:41.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55528/0 (socket says 172.21.15.67:55528) 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 -- 172.21.15.67:0/1615832196 learned_addr learned my addr 172.21.15.67:0/1615832196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd186800640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd188153ef0 0x7fd188159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185e00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1680051c0 0x7fd1881e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 -- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1680051c0 msgr2=0x7fd1881e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1680051c0 0x7fd1881e5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 -- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd188153ef0 msgr2=0x7fd188159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd188153ef0 0x7fd188159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.578+0000 7fd185400640 1 -- 172.21.15.67:0/1615832196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1881e5cb0 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd185400640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd170011590 tx=0x7fd170011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170004220 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1700043c0 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170015930 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1881e5f90 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.579+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1881e6550 con 0x7fd1881e6d50 2024-04-17T07:14:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.580+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd188068b90 con 0x7fd1881e6d50 2024-04-17T07:14:41.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.581+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd170015ad0 con 0x7fd1881e6d50 2024-04-17T07:14:41.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.582+0000 7fd17e000640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:41.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.582+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd17008c150 con 0x7fd1881e6d50 2024-04-17T07:14:41.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.583+0000 7fd185e00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:41.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.584+0000 7fd185e00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 0x7fd16406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd178007fd0 tx=0x7fd178009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:41.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.587+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd170019030 con 0x7fd1881e6d50 2024-04-17T07:14:41.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.793+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd18815bff0 con 0x7fd1881e6d50 2024-04-17T07:14:41.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.794+0000 7fd17e000640 1 -- 172.21.15.67:0/1615832196 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fd17005a580 con 0x7fd1881e6d50 2024-04-17T07:14:41.794 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:41.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 msgr2=0x7fd16406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 0x7fd16406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd178007fd0 tx=0x7fd178009040 comp rx=0 tx=0).stop 2024-04-17T07:14:41.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 msgr2=0x7fd1880cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd170011590 tx=0x7fd170011a50 comp rx=0 tx=0).stop 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 shutdown_connections 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd188153ef0 0x7fd188159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1881e6d50 0x7fd1880cc8a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fd16406c8f0 0x7fd16406edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 --2- 172.21.15.67:0/1615832196 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1680051c0 0x7fd1881e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 >> 172.21.15.67:0/1615832196 conn(0x7fd1880b3390 msgr2=0x7fd188158530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 shutdown_connections 2024-04-17T07:14:41.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:41.797+0000 7fd18ce00640 1 -- 172.21.15.67:0/1615832196 wait complete. 2024-04-17T07:14:41.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-17T07:14:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.040+0000 7f01d7e00640 1 Processor -- start 2024-04-17T07:14:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.040+0000 7f01d7e00640 1 -- start start 2024-04-17T07:14:42.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.041+0000 7f01d7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.041+0000 7f01d7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01d80cf910 0x7f01d80b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.041+0000 7f01d7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d80d2d30 con 0x7f01d815ec60 2024-04-17T07:14:42.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.041+0000 7f01d7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f01d80d49c0 con 0x7f01d80cf910 2024-04-17T07:14:42.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.041+0000 7f01d7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01d80d3660 con 0x7f01d80b6d00 2024-04-17T07:14:42.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.042+0000 7f01d7400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01d80cf910 0x7f01d80b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.042+0000 7f01d6a00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f01d815ec60 0x7f01d80b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:51934/0 (socket says 172.21.15.67:51934) 2024-04-17T07:14:42.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.042+0000 7f01d6a00640 1 -- 172.21.15.67:0/3493333054 learned_addr learned my addr 172.21.15.67:0/3493333054 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:42.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.042+0000 7f01d6000640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.043+0000 7f01d7400640 1 -- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 msgr2=0x7f01d80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.043+0000 7f01d7400640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.043+0000 7f01d7400640 1 -- 172.21.15.67:0/3493333054 >> v1:172.21.15.187:6789/0 conn(0x7f01d815ec60 legacy=0x7f01d80b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.043+0000 7f01d7400640 1 -- 172.21.15.67:0/3493333054 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d80d3010 con 0x7f01d80cf910 2024-04-17T07:14:42.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.044+0000 7f01d7400640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d80cf910 0x7f01d80b7b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f9a7dceb10bba18 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.045+0000 7f01d5600640 1 -- 172.21.15.67:0/3493333054 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0039070 con 0x7f01d80cf910 2024-04-17T07:14:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.045+0000 7f01d5600640 1 -- 172.21.15.67:0/3493333054 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d002cc50 con 0x7f01d80cf910 2024-04-17T07:14:42.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.045+0000 7f01d5600640 1 -- 172.21.15.67:0/3493333054 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0034440 con 0x7f01d80cf910 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 -- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d80cf910 msgr2=0x7f01d80b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d80cf910 0x7f01d80b7b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009d90 tx=0x7f01d002bea0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 -- 172.21.15.67:0/3493333054 shutdown_connections 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d80cf910 0x7f01d80b7b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3493333054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d80cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 -- 172.21.15.67:0/3493333054 >> 172.21.15.67:0/3493333054 conn(0x7f01d80d15f0 msgr2=0x7f01d80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 -- 172.21.15.67:0/3493333054 shutdown_connections 2024-04-17T07:14:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.046+0000 7f01d7e00640 1 -- 172.21.15.67:0/3493333054 wait complete. 2024-04-17T07:14:42.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.047+0000 7f01d7e00640 1 Processor -- start 2024-04-17T07:14:42.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.048+0000 7f01d7e00640 1 -- start start 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d815ec60 0x7f01d80b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01d815f040 con 0x7f01d80cf910 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f01d80cfdc0 con 0x7f01d815ec60 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01d80d00c0 con 0x7f01d80b6d00 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55532/0 (socket says 172.21.15.67:55532) 2024-04-17T07:14:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 -- 172.21.15.67:0/3297275261 learned_addr learned my addr 172.21.15.67:0/3297275261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6a00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d7400640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d815ec60 0x7f01d80b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 -- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 msgr2=0x7f01d814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 -- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d815ec60 msgr2=0x7f01d80b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d815ec60 0x7f01d80b8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6000640 1 -- 172.21.15.67:0/3297275261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d814d900 con 0x7f01d80cf910 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.049+0000 7f01d6a00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01d6000640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e9e0 tx=0x7f01c400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4017070 con 0x7f01d80cf910 2024-04-17T07:14:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c400c9d0 con 0x7f01d80cf910 2024-04-17T07:14:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c400cdc0 con 0x7f01d80cf910 2024-04-17T07:14:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d80cce10 con 0x7f01d80cf910 2024-04-17T07:14:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.050+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d80cd0d0 con 0x7f01d80cf910 2024-04-17T07:14:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.051+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d8068a10 con 0x7f01d80cf910 2024-04-17T07:14:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.052+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01c401a070 con 0x7f01d80cf910 2024-04-17T07:14:42.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.054+0000 7f01cea00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 0x7f01b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.054+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f01c408bac0 con 0x7f01d80cf910 2024-04-17T07:14:42.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.054+0000 7f01d6a00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 0x7f01b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.056+0000 7f01d6a00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 0x7f01b806edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007fd0 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.058+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f01c4059f80 con 0x7f01d80cf910 2024-04-17T07:14:42.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.333+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7f01d80b98d0 con 0x7f01d80cf910 2024-04-17T07:14:42.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.534+0000 7f01cea00640 1 -- 172.21.15.67:0/3297275261 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v59) ==== 161+0+0 (secure 0 0 0) 0x7f01c4059880 con 0x7f01d80cf910 2024-04-17T07:14:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 msgr2=0x7f01b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 0x7f01b806edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007fd0 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2024-04-17T07:14:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 msgr2=0x7f01d80b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e9e0 tx=0x7f01c400eea0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 shutdown_connections 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01d815ec60 0x7f01d80b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01d80cf910 0x7f01d80b8620 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f01b806c8f0 0x7f01b806edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 --2- 172.21.15.67:0/3297275261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01d80b6d00 0x7f01d814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 >> 172.21.15.67:0/3297275261 conn(0x7f01d80d15f0 msgr2=0x7f01d80d3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 shutdown_connections 2024-04-17T07:14:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.537+0000 7f01d7e00640 1 -- 172.21.15.67:0/3297275261 wait complete. 2024-04-17T07:14:42.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:14:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.786+0000 7f1033e00640 1 Processor -- start 2024-04-17T07:14:42.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.786+0000 7f1033e00640 1 -- start start 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1034145f80 0x7f10340cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340cd750 0x7f103406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f1034160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10340727a0 con 0x7f10340cd750 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1034146360 con 0x7f1034145f80 2024-04-17T07:14:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.788+0000 7f1033e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1034071e70 con 0x7f103406c0a0 2024-04-17T07:14:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340cd750 0x7f103406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340cd750 0x7f103406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55542/0 (socket says 172.21.15.67:55542) 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 -- 172.21.15.67:0/3001446334 learned_addr learned my addr 172.21.15.67:0/3001446334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f1032a00640 1 --2- 172.21.15.67:0/3001446334 >> v2:172.21.15.187:3300/0 conn(0x7f1034145f80 0x7f10340cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f1033400640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f1034160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 -- 172.21.15.67:0/3001446334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 msgr2=0x7f1034160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f1034160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 -- 172.21.15.67:0/3001446334 >> v2:172.21.15.187:3300/0 conn(0x7f1034145f80 msgr2=0x7f10340cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 --2- 172.21.15.67:0/3001446334 >> v2:172.21.15.187:3300/0 conn(0x7f1034145f80 0x7f10340cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.789+0000 7f102be00640 1 -- 172.21.15.67:0/3001446334 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1034072150 con 0x7f10340cd750 2024-04-17T07:14:42.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.790+0000 7f102be00640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103406bb60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f103406c9c0 tx=0x7f101c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7eb7fd3cc7f8b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1032000640 1 -- 172.21.15.67:0/3001446334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c0317d0 con 0x7f10340cd750 2024-04-17T07:14:42.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1032000640 1 -- 172.21.15.67:0/3001446334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f101c037070 con 0x7f10340cd750 2024-04-17T07:14:42.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1032000640 1 -- 172.21.15.67:0/3001446334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c031b90 con 0x7f10340cd750 2024-04-17T07:14:42.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 -- 172.21.15.67:0/3001446334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 msgr2=0x7f103406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103406bb60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f103406c9c0 tx=0x7f101c02b870 comp rx=0 tx=0).stop 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 -- 172.21.15.67:0/3001446334 shutdown_connections 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f1034160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 --2- 172.21.15.67:0/3001446334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103406bb60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 --2- 172.21.15.67:0/3001446334 >> v2:172.21.15.187:3300/0 conn(0x7f1034145f80 0x7f10340cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 -- 172.21.15.67:0/3001446334 >> 172.21.15.67:0/3001446334 conn(0x7f10340b2190 msgr2=0x7f10340b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 -- 172.21.15.67:0/3001446334 shutdown_connections 2024-04-17T07:14:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.791+0000 7f1033e00640 1 -- 172.21.15.67:0/3001446334 wait complete. 2024-04-17T07:14:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.793+0000 7f1033e00640 1 Processor -- start 2024-04-17T07:14:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.793+0000 7f1033e00640 1 -- start start 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f103415bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1034145f80 0x7f103415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1034072db0 con 0x7f10340cd750 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1034072c30 con 0x7f1034145f80 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f1033e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f103406c4d0 con 0x7f103406c0a0 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.794+0000 7f102be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55546/0 (socket says 172.21.15.67:55546) 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 -- 172.21.15.67:0/4121217580 learned_addr learned my addr 172.21.15.67:0/4121217580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:42.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f1033400640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1034145f80 0x7f103415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f1032a00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f103415bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 -- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 msgr2=0x7f103415bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f103415bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 -- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1034145f80 msgr2=0x7f103415d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1034145f80 0x7f103415d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 -- 172.21.15.67:0/4121217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103413e1e0 con 0x7f10340cd750 2024-04-17T07:14:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f1032a00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f103415bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.795+0000 7f102be00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f101c000c00 tx=0x7f101c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.796+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c032660 con 0x7f10340cd750 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.796+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f101c032840 con 0x7f10340cd750 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.796+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c032c30 con 0x7f10340cd750 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.796+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103413e4c0 con 0x7f10340cd750 2024-04-17T07:14:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.796+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103413ea80 con 0x7f10340cd750 2024-04-17T07:14:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.797+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1034068a10 con 0x7f10340cd750 2024-04-17T07:14:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.798+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f101c04c060 con 0x7f10340cd750 2024-04-17T07:14:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.799+0000 7f102aa00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 0x7f101006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:42.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.799+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f101c0ad690 con 0x7f10340cd750 2024-04-17T07:14:42.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.799+0000 7f1032a00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 0x7f101006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:42.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.800+0000 7f1032a00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 0x7f101006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:42.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.803+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f101c040070 con 0x7f10340cd750 2024-04-17T07:14:42.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:42.997+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f10340b0490 con 0x7f10340cd750 2024-04-17T07:14:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.005+0000 7f102aa00640 1 -- 172.21.15.67:0/4121217580 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f101c07ba20 con 0x7f10340cd750 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 msgr2=0x7f101006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 0x7f101006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024008040 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 msgr2=0x7f103415cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f101c000c00 tx=0x7f101c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 shutdown_connections 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1034145f80 0x7f103415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340cd750 0x7f103415cd00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f101006c8f0 0x7f101006edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 --2- 172.21.15.67:0/4121217580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103406c0a0 0x7f103415bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 >> 172.21.15.67:0/4121217580 conn(0x7f10340b2190 msgr2=0x7f103413d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 shutdown_connections 2024-04-17T07:14:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.008+0000 7f1033e00640 1 -- 172.21.15.67:0/4121217580 wait complete. 2024-04-17T07:14:43.026 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:14:43.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:14:43.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:14:43.270 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:14:43.270 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:43.270 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:43.331 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:43.331 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:43.384 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:43.384 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:43.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.664+0000 7f6c00600640 1 Processor -- start 2024-04-17T07:14:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.664+0000 7f6c00600640 1 -- start start 2024-04-17T07:14:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6c00600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6bf806f670 0x7f6bf806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6c00600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf806f030 0x7f6bf80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6c00600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf806ff90 con 0x7f6bf80cd020 2024-04-17T07:14:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6c00600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf80ceb00 con 0x7f6bf806f030 2024-04-17T07:14:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6c00600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf80cede0 con 0x7f6bf806f670 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.666+0000 7f6bfe600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf806f030 0x7f6bf80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf806f030 0x7f6bf80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40754/0 (socket says 172.21.15.67:40754) 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 -- 172.21.15.67:0/1739836017 learned_addr learned my addr 172.21.15.67:0/1739836017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfdc00640 1 --2- 172.21.15.67:0/1739836017 >> v2:172.21.15.187:3300/0 conn(0x7f6bf806f670 0x7f6bf806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 -- 172.21.15.67:0/1739836017 >> v2:172.21.15.187:3300/0 conn(0x7f6bf806f670 msgr2=0x7f6bf806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 --2- 172.21.15.67:0/1739836017 >> v2:172.21.15.187:3300/0 conn(0x7f6bf806f670 0x7f6bf806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 -- 172.21.15.67:0/1739836017 >> v1:172.21.15.187:6789/0 conn(0x7f6bf80cd020 legacy=0x7f6bf8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:43.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.667+0000 7f6bfe600640 1 -- 172.21.15.67:0/1739836017 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf80cf0c0 con 0x7f6bf806f030 2024-04-17T07:14:43.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.669+0000 7f6bfe600640 1 --2- 172.21.15.67:0/1739836017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf806f030 0x7f6bf80cd480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6bf806c910 tx=0x7f6bf002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23db7ea3af40edd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.669+0000 7f6bf7e00640 1 -- 172.21.15.67:0/1739836017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf002ca80 con 0x7f6bf806f030 2024-04-17T07:14:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.669+0000 7f6bf7e00640 1 -- 172.21.15.67:0/1739836017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf002cc20 con 0x7f6bf806f030 2024-04-17T07:14:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.669+0000 7f6bf7e00640 1 -- 172.21.15.67:0/1739836017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf00343f0 con 0x7f6bf806f030 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 -- 172.21.15.67:0/1739836017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf806f030 msgr2=0x7f6bf80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 --2- 172.21.15.67:0/1739836017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf806f030 0x7f6bf80cd480 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6bf806c910 tx=0x7f6bf002bee0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 -- 172.21.15.67:0/1739836017 shutdown_connections 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 --2- 172.21.15.67:0/1739836017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf806f030 0x7f6bf80cd480 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 --2- 172.21.15.67:0/1739836017 >> v2:172.21.15.187:3300/0 conn(0x7f6bf806f670 0x7f6bf806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 -- 172.21.15.67:0/1739836017 >> 172.21.15.67:0/1739836017 conn(0x7f6bf806bf50 msgr2=0x7f6bf80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 -- 172.21.15.67:0/1739836017 shutdown_connections 2024-04-17T07:14:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.670+0000 7f6c00600640 1 -- 172.21.15.67:0/1739836017 wait complete. 2024-04-17T07:14:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.672+0000 7f6c00600640 1 Processor -- start 2024-04-17T07:14:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.672+0000 7f6c00600640 1 -- start start 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 0x7f6bf81498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 0x7f6bf814a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf80ce720 con 0x7f6bf80cd020 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf80cf680 con 0x7f6bf806f670 2024-04-17T07:14:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6c00600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf80ce8a0 con 0x7f6bf806f030 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6bfe600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6bfe600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40756/0 (socket says 172.21.15.67:40756) 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.673+0000 7f6bfe600640 1 -- 172.21.15.67:0/3338845979 learned_addr learned my addr 172.21.15.67:0/3338845979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfdc00640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 0x7f6bf81498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfd200640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 0x7f6bf814a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 -- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 msgr2=0x7f6bf81498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 0x7f6bf81498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 -- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 msgr2=0x7f6bf814a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 0x7f6bf814a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 -- 172.21.15.67:0/3338845979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf814a5c0 con 0x7f6bf80cd020 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfdc00640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 0x7f6bf81498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfd200640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 0x7f6bf814a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bfe600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0036040 tx=0x7f6bf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.674+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0043020 con 0x7f6bf80cd020 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.675+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf003e450 con 0x7f6bf80cd020 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.675+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0009510 con 0x7f6bf80cd020 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.675+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf815f8e0 con 0x7f6bf80cd020 2024-04-17T07:14:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.675+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf814cd30 con 0x7f6bf80cd020 2024-04-17T07:14:43.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.676+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf8068b90 con 0x7f6bf80cd020 2024-04-17T07:14:43.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.677+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6bf0005d10 con 0x7f6bf80cd020 2024-04-17T07:14:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.678+0000 7f6bf5600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 0x7f6bd806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.678+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6bf00ae6e0 con 0x7f6bf80cd020 2024-04-17T07:14:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.679+0000 7f6bfdc00640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 0x7f6bd806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.680+0000 7f6bfdc00640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 0x7f6bd806ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6bec006fd0 tx=0x7f6bec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:43.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.683+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6bf007ca70 con 0x7f6bf80cd020 2024-04-17T07:14:43.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.892+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bf80b1010 con 0x7f6bf80cd020 2024-04-17T07:14:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.893+0000 7f6bf5600640 1 -- 172.21.15.67:0/3338845979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16557 (secure 0 0 0) 0x7f6bf007c370 con 0x7f6bf80cd020 2024-04-17T07:14:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:43.894 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:13:59.205687+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 msgr2=0x7f6bd806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 0x7f6bd806ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6bec006fd0 tx=0x7f6bec008040 comp rx=0 tx=0).stop 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 msgr2=0x7f6bf815f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0036040 tx=0x7f6bf0004a40 comp rx=0 tx=0).stop 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 shutdown_connections 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf80cd020 0x7f6bf815f3a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf806f670 0x7f6bf814a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f6bd806c920 0x7f6bd806ede0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 --2- 172.21.15.67:0/3338845979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf806f030 0x7f6bf81498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 >> 172.21.15.67:0/3338845979 conn(0x7f6bf806bf50 msgr2=0x7f6bf80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 shutdown_connections 2024-04-17T07:14:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:43.897+0000 7f6c00600640 1 -- 172.21.15.67:0/3338845979 wait complete. 2024-04-17T07:14:43.916 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:43.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:44.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.141+0000 7f7aaa800640 1 Processor -- start 2024-04-17T07:14:44.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.142+0000 7f7aaa800640 1 -- start start 2024-04-17T07:14:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aaa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7aa415ec20 0x7f7aa40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aaa800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa40d4f70 con 0x7f7aa40cfcd0 2024-04-17T07:14:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aaa800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa40d5250 con 0x7f7aa40b3b50 2024-04-17T07:14:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aaa800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7aa40cf4d0 con 0x7f7aa415ec20 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aa9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7aa415ec20 0x7f7aa40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aa9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7aa415ec20 0x7f7aa40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40766/0 (socket says 172.21.15.67:40766) 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.143+0000 7f7aa9400640 1 -- 172.21.15.67:0/3442935603 learned_addr learned my addr 172.21.15.67:0/3442935603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.144+0000 7f7aa9400640 1 -- 172.21.15.67:0/3442935603 >> v1:172.21.15.187:6789/0 conn(0x7f7aa40b3b50 legacy=0x7f7aa406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.144+0000 7f7aa9400640 1 -- 172.21.15.67:0/3442935603 >> v1:172.21.15.67:6789/0 conn(0x7f7aa40cfcd0 legacy=0x7f7aa40d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.144+0000 7f7aa9400640 1 -- 172.21.15.67:0/3442935603 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa40cf7b0 con 0x7f7aa415ec20 2024-04-17T07:14:44.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.145+0000 7f7aa9400640 1 --2- 172.21.15.67:0/3442935603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa415ec20 0x7f7aa40d22b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a9402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60df9d75ee385013 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.145+0000 7f7aa3e00640 1 -- 172.21.15.67:0/3442935603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9402e070 con 0x7f7aa415ec20 2024-04-17T07:14:44.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aa3e00640 1 -- 172.21.15.67:0/3442935603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a94037070 con 0x7f7aa415ec20 2024-04-17T07:14:44.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aa3e00640 1 -- 172.21.15.67:0/3442935603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94031a00 con 0x7f7aa415ec20 2024-04-17T07:14:44.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 -- 172.21.15.67:0/3442935603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa415ec20 msgr2=0x7f7aa40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 --2- 172.21.15.67:0/3442935603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa415ec20 0x7f7aa40d22b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a9402b880 comp rx=0 tx=0).stop 2024-04-17T07:14:44.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 -- 172.21.15.67:0/3442935603 shutdown_connections 2024-04-17T07:14:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 --2- 172.21.15.67:0/3442935603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa415ec20 0x7f7aa40d22b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 -- 172.21.15.67:0/3442935603 >> 172.21.15.67:0/3442935603 conn(0x7f7aa408bff0 msgr2=0x7f7aa40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 -- 172.21.15.67:0/3442935603 shutdown_connections 2024-04-17T07:14:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.146+0000 7f7aaa800640 1 -- 172.21.15.67:0/3442935603 wait complete. 2024-04-17T07:14:44.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.148+0000 7f7aaa800640 1 Processor -- start 2024-04-17T07:14:44.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.148+0000 7f7aaa800640 1 -- start start 2024-04-17T07:14:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 0x7f7aa40b9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 0x7f7aa41f9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa406d2d0 con 0x7f7aa40b3b50 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa406d150 con 0x7f7aa415ec20 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.149+0000 7f7aaa800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa40d17e0 con 0x7f7aa40cfcd0 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40782/0 (socket says 172.21.15.67:40782) 2024-04-17T07:14:44.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 -- 172.21.15.67:0/2770055192 learned_addr learned my addr 172.21.15.67:0/2770055192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 -- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 msgr2=0x7f7aa40b9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa8a00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 0x7f7aa40b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9e00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 0x7f7aa41f9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 0x7f7aa40b9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 -- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 msgr2=0x7f7aa41f9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 0x7f7aa41f9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9400640 1 -- 172.21.15.67:0/2770055192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa41fa1a0 con 0x7f7aa40b3b50 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa8a00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 0x7f7aa40b9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.150+0000 7f7aa9e00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 0x7f7aa41f9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aa9400640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94032880 con 0x7f7aa40b3b50 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a94032a20 con 0x7f7aa40b3b50 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94032e10 con 0x7f7aa40b3b50 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa40ba290 con 0x7f7aa40b3b50 2024-04-17T07:14:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.151+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa40b8870 con 0x7f7aa40b3b50 2024-04-17T07:14:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.152+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa40d4450 con 0x7f7aa40b3b50 2024-04-17T07:14:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.153+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a9404c060 con 0x7f7aa40b3b50 2024-04-17T07:14:44.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.154+0000 7f7aa1600640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 0x7f7a8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.155+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7a940ad560 con 0x7f7aa40b3b50 2024-04-17T07:14:44.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.155+0000 7f7aa8a00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 0x7f7a8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.156+0000 7f7aa8a00640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 0x7f7a8806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7a98007fd0 tx=0x7f7a98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.159+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7aa40d4450 con 0x7f7aa40b3b50 2024-04-17T07:14:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.363+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7aa4068a10 con 0x7f7aa40b3b50 2024-04-17T07:14:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.363+0000 7f7aa1600640 1 -- 172.21.15.67:0/2770055192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f7aa41fa1a0 con 0x7f7aa40b3b50 2024-04-17T07:14:44.364 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 msgr2=0x7f7a8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 0x7f7a8806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7a98007fd0 tx=0x7f7a98009040 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 msgr2=0x7f7aa40b9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009870 tx=0x7f7a940046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 shutdown_connections 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7aa415ec20 0x7f7aa41f9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7a8806c8f0 0x7f7a8806edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7aa40cfcd0 0x7f7aa40b9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 --2- 172.21.15.67:0/2770055192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7aa40b3b50 0x7f7aa40b9cb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 >> 172.21.15.67:0/2770055192 conn(0x7f7aa408bff0 msgr2=0x7f7aa406d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 shutdown_connections 2024-04-17T07:14:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.366+0000 7f7aaa800640 1 -- 172.21.15.67:0/2770055192 wait complete. 2024-04-17T07:14:44.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.614+0000 7fe480e00640 1 Processor -- start 2024-04-17T07:14:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.614+0000 7fe480e00640 1 -- start start 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe480e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe47c0b4fd0 0x7fe47c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe480e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe47c0d00b0 con 0x7fe47c0d33e0 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe480e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe47c0d0390 con 0x7fe47c15ecb0 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe480e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe47c0d0c30 con 0x7fe47c0b4fd0 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe47aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe47c0d33e0 0x7fe47c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45364/0 (socket says 172.21.15.67:45364) 2024-04-17T07:14:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe47aa00640 1 -- 172.21.15.67:0/3380777468 learned_addr learned my addr 172.21.15.67:0/3380777468 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.616+0000 7fe47be00640 1 --2- 172.21.15.67:0/3380777468 >> v2:172.21.15.187:3300/0 conn(0x7fe47c0b4fd0 0x7fe47c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223426099 0 0) 0x7fe47c0d00b0 con 0x7fe47c0d33e0 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe45c003520 con 0x7fe47c0d33e0 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2196186262 0 0) 0x7fe45c003520 con 0x7fe47c0d33e0 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 >> v2:172.21.15.187:3300/0 conn(0x7fe47c0b4fd0 msgr2=0x7fe47c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:44.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 --2- 172.21.15.67:0/3380777468 >> v2:172.21.15.187:3300/0 conn(0x7fe47c0b4fd0 0x7fe47c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 >> v1:172.21.15.187:6789/0 conn(0x7fe47c15ecb0 legacy=0x7fe47c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c0d0670 con 0x7fe47c0d33e0 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe4640019f0 con 0x7fe47c0d33e0 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 >> v1:172.21.15.67:6789/0 conn(0x7fe47c0d33e0 legacy=0x7fe47c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c003a00 0x7fe45c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c004800 0x7fe45c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 0x7fe45c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c0d0c30 con 0x7fe45c005230 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c00c190 con 0x7fe45c003a00 2024-04-17T07:14:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.617+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe45c00c390 con 0x7fe45c004800 2024-04-17T07:14:44.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 0x7fe45c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47b400640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c003a00 0x7fe45c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47aa00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c004800 0x7fe45c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 -- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c004800 msgr2=0x7fe45c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c004800 0x7fe45c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 -- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c003a00 msgr2=0x7fe45c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c003a00 0x7fe45c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.618+0000 7fe47be00640 1 -- 172.21.15.67:0/3380777468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c003520 con 0x7fe45c005230 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.619+0000 7fe47be00640 1 -- 172.21.15.67:0/3380777468 reap_dead start 2024-04-17T07:14:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.619+0000 7fe47be00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 0x7fe45c00b9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe474009ac0 tx=0x7fe45c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8706691b4fcd486 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.620+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474009690 con 0x7fe45c005230 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.620+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe474004020 con 0x7fe45c005230 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.620+0000 7fe47a000640 1 -- 172.21.15.67:0/3380777468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474004410 con 0x7fe45c005230 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.620+0000 7fe480e00640 1 -- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 msgr2=0x7fe45c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.620+0000 7fe480e00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 0x7fe45c00b9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe474009ac0 tx=0x7fe45c005830 comp rx=0 tx=0).stop 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.621+0000 7fe480e00640 1 -- 172.21.15.67:0/3380777468 shutdown_connections 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.621+0000 7fe480e00640 1 --2- 172.21.15.67:0/3380777468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe45c005230 0x7fe45c00b9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.621+0000 7fe480e00640 1 -- 172.21.15.67:0/3380777468 >> 172.21.15.67:0/3380777468 conn(0x7fe47c08bff0 msgr2=0x7fe47c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.621+0000 7fe480e00640 1 -- 172.21.15.67:0/3380777468 shutdown_connections 2024-04-17T07:14:44.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.621+0000 7fe480e00640 1 -- 172.21.15.67:0/3380777468 wait complete. 2024-04-17T07:14:44.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.623+0000 7fe480e00640 1 Processor -- start 2024-04-17T07:14:44.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.623+0000 7fe480e00640 1 -- start start 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c005230 0x7fe47c0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 0x7fe47c0b94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c15f210 con 0x7fe47c0c9a80 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c15f090 con 0x7fe47c0b9030 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe480e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe47c0d2cd0 con 0x7fe45c005230 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40800/0 (socket says 172.21.15.67:40800) 2024-04-17T07:14:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 -- 172.21.15.67:0/1955465913 learned_addr learned my addr 172.21.15.67:0/1955465913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:44.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47be00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 0x7fe47c0b94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 -- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c005230 msgr2=0x7fe47c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c005230 0x7fe47c0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 -- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 msgr2=0x7fe47c0b94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 0x7fe47c0b94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47aa00640 1 -- 172.21.15.67:0/1955465913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c0b8a50 con 0x7fe47c0c9a80 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.624+0000 7fe47be00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 0x7fe47c0b94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe47aa00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe4640013d0 tx=0x7fe46400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46400ee50 con 0x7fe47c0c9a80 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe464018070 con 0x7fe47c0c9a80 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe464013660 con 0x7fe47c0c9a80 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47c0c8cc0 con 0x7fe47c0c9a80 2024-04-17T07:14:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.625+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47c0c9280 con 0x7fe47c0c9a80 2024-04-17T07:14:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.626+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47c068a10 con 0x7fe47c0c9a80 2024-04-17T07:14:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.627+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe464013800 con 0x7fe47c0c9a80 2024-04-17T07:14:44.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.628+0000 7fe473400640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 0x7fe45806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:44.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.628+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe46408c1f0 con 0x7fe47c0c9a80 2024-04-17T07:14:44.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.629+0000 7fe47b400640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 0x7fe45806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:44.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.630+0000 7fe47b400640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 0x7fe45806ef50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0016d0 tx=0x7fe46c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:44.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.633+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe46405a5a0 con 0x7fe47c0c9a80 2024-04-17T07:14:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.835+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe47c0b5a70 con 0x7fe47c0c9a80 2024-04-17T07:14:44.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.836+0000 7fe473400640 1 -- 172.21.15.67:0/1955465913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7fe464059ea0 con 0x7fe47c0c9a80 2024-04-17T07:14:44.836 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:44.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 msgr2=0x7fe45806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 0x7fe45806ef50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0016d0 tx=0x7fe46c009040 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 msgr2=0x7fe47c153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe4640013d0 tx=0x7fe46400e530 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 shutdown_connections 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe47c0b9030 0x7fe47c0b94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47c0c9a80 0x7fe47c153a40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe45806ca90 0x7fe45806ef50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 --2- 172.21.15.67:0/1955465913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c005230 0x7fe47c0cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 >> 172.21.15.67:0/1955465913 conn(0x7fe47c08bff0 msgr2=0x7fe47c06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 shutdown_connections 2024-04-17T07:14:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:44.839+0000 7fe480e00640 1 -- 172.21.15.67:0/1955465913 wait complete. 2024-04-17T07:14:44.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.100+0000 7fe8f7e00640 1 Processor -- start 2024-04-17T07:14:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.101+0000 7fe8f7e00640 1 -- start start 2024-04-17T07:14:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.102+0000 7fe8f7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8f815ec20 0x7fe8f80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.102+0000 7fe8f7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8f80d4f70 con 0x7fe8f80b3b50 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.102+0000 7fe8f7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8f80d5250 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.102+0000 7fe8f7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8f80cf4d0 con 0x7fe8f815ec20 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8efe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8f80b3b50 0x7fe8f806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45376/0 (socket says 172.21.15.67:45376) 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8efe00640 1 -- 172.21.15.67:0/2820138856 learned_addr learned my addr 172.21.15.67:0/2820138856 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8f6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8f815ec20 0x7fe8f80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8f6a00640 1 -- 172.21.15.67:0/2820138856 >> v1:172.21.15.187:6789/0 conn(0x7fe8f80cfcd0 legacy=0x7fe8f80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:45.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8f6a00640 1 -- 172.21.15.67:0/2820138856 >> v1:172.21.15.67:6789/0 conn(0x7fe8f80b3b50 legacy=0x7fe8f806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8f6a00640 1 -- 172.21.15.67:0/2820138856 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f80cf7b0 con 0x7fe8f815ec20 2024-04-17T07:14:45.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.103+0000 7fe8f6000640 1 -- 172.21.15.67:0/2820138856 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3754845413 0 0) 0x7fe8f80d4f70 con 0x7fe8f80b3b50 2024-04-17T07:14:45.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.104+0000 7fe8f6a00640 1 --2- 172.21.15.67:0/2820138856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f815ec20 0x7fe8f80d22b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80097a0 tx=0x7fe8e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c073ce12e59f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.105+0000 7fe8f6000640 1 -- 172.21.15.67:0/2820138856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80315a0 con 0x7fe8f815ec20 2024-04-17T07:14:45.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.105+0000 7fe8f6000640 1 -- 172.21.15.67:0/2820138856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e8037070 con 0x7fe8f815ec20 2024-04-17T07:14:45.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.105+0000 7fe8f6000640 1 -- 172.21.15.67:0/2820138856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8031960 con 0x7fe8f815ec20 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.105+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2820138856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f815ec20 msgr2=0x7fe8f80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.105+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2820138856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f815ec20 0x7fe8f80d22b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80097a0 tx=0x7fe8e802b840 comp rx=0 tx=0).stop 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.106+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2820138856 shutdown_connections 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.106+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2820138856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f815ec20 0x7fe8f80d22b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.106+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2820138856 >> 172.21.15.67:0/2820138856 conn(0x7fe8f808bff0 msgr2=0x7fe8f80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.106+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2820138856 shutdown_connections 2024-04-17T07:14:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.106+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2820138856 wait complete. 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.107+0000 7fe8f7e00640 1 Processor -- start 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.108+0000 7fe8f7e00640 1 -- start start 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 0x7fe8f8150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8f815ec20 0x7fe8f81598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f806d2d0 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f806d150 con 0x7fe8f815ec20 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f80d17e0 con 0x7fe8f80b3b50 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40826/0 (socket says 172.21.15.67:40826) 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 -- 172.21.15.67:0/2886514170 learned_addr learned my addr 172.21.15.67:0/2886514170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f6a00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 0x7fe8f8150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 -- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 msgr2=0x7fe8f8150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 0x7fe8f8150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8efe00640 1 -- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8f815ec20 msgr2=0x7fe8f81598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.109+0000 7fe8f7400640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8f815ec20 0x7fe8f81598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8efe00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8f815ec20 0x7fe8f81598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8efe00640 1 -- 172.21.15.67:0/2886514170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f8151110 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8f6a00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 0x7fe8f8150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8efe00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000c760 tx=0x7fe8e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0007270 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e0007430 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00059e0 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f80b7380 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.110+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f80b7940 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.111+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f8068a10 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.112+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8e0005b80 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.113+0000 7fe8ef400640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 0x7fe8d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.113+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fe8e008b830 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.114+0000 7fe8f6a00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 0x7fe8d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.115+0000 7fe8f6a00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 0x7fe8d406ede0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8036000 tx=0x7fe8e8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.118+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe8e0010300 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.323+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe8f80d55f0 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.324+0000 7fe8ef400640 1 -- 172.21.15.67:0/2886514170 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fe8e00075e0 con 0x7fe8f80cfcd0 2024-04-17T07:14:45.324 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 msgr2=0x7fe8d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 0x7fe8d406ede0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8036000 tx=0x7fe8e8008030 comp rx=0 tx=0).stop 2024-04-17T07:14:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 msgr2=0x7fe8f81593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000c760 tx=0x7fe8e000cc20 comp rx=0 tx=0).stop 2024-04-17T07:14:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 shutdown_connections 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8f815ec20 0x7fe8f81598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f80cfcd0 0x7fe8f81593a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8d406c920 0x7fe8d406ede0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 --2- 172.21.15.67:0/2886514170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8f80b3b50 0x7fe8f8150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 >> 172.21.15.67:0/2886514170 conn(0x7fe8f808bff0 msgr2=0x7fe8f8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 shutdown_connections 2024-04-17T07:14:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.327+0000 7fe8f7e00640 1 -- 172.21.15.67:0/2886514170 wait complete. 2024-04-17T07:14:45.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.571+0000 7efcc3200640 1 Processor -- start 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.572+0000 7efcc3200640 1 -- start start 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.573+0000 7efcc3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efcbc15ee20 0x7efcbc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.573+0000 7efcc3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcbc0d3520 0x7efcbc0b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.573+0000 7efcc3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc0d4140 con 0x7efcbc0d18e0 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.573+0000 7efcc3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0b83b0 con 0x7efcbc0d3520 2024-04-17T07:14:45.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.573+0000 7efcc3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0d1d20 con 0x7efcbc15ee20 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcbc0d3520 0x7efcbc0b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcbc0d18e0 0x7efcbc0d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45388/0 (socket says 172.21.15.67:45388) 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc1400640 1 -- 172.21.15.67:0/1508032497 learned_addr learned my addr 172.21.15.67:0/1508032497 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc1e00640 1 --2- 172.21.15.67:0/1508032497 >> v2:172.21.15.187:3300/0 conn(0x7efcbc15ee20 0x7efcbc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc2800640 1 -- 172.21.15.67:0/1508032497 >> v2:172.21.15.187:3300/0 conn(0x7efcbc15ee20 msgr2=0x7efcbc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc2800640 1 --2- 172.21.15.67:0/1508032497 >> v2:172.21.15.187:3300/0 conn(0x7efcbc15ee20 0x7efcbc0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc2800640 1 -- 172.21.15.67:0/1508032497 >> v1:172.21.15.67:6789/0 conn(0x7efcbc0d18e0 legacy=0x7efcbc0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc2800640 1 -- 172.21.15.67:0/1508032497 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0d3b80 con 0x7efcbc0d3520 2024-04-17T07:14:45.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.574+0000 7efcc0a00640 1 -- 172.21.15.67:0/1508032497 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658224451 0 0) 0x7efcbc0d4140 con 0x7efcbc0d18e0 2024-04-17T07:14:45.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc2800640 1 --2- 172.21.15.67:0/1508032497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d3520 0x7efcbc0b8840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0d1370 tx=0x7efcb802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5b81e183ecb0bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc0a00640 1 -- 172.21.15.67:0/1508032497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8039070 con 0x7efcbc0d3520 2024-04-17T07:14:45.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc0a00640 1 -- 172.21.15.67:0/1508032497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb802cc80 con 0x7efcbc0d3520 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc0a00640 1 -- 172.21.15.67:0/1508032497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8034660 con 0x7efcbc0d3520 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc3200640 1 -- 172.21.15.67:0/1508032497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d3520 msgr2=0x7efcbc0b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.576+0000 7efcc3200640 1 --2- 172.21.15.67:0/1508032497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d3520 0x7efcbc0b8840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0d1370 tx=0x7efcb802bed0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 -- 172.21.15.67:0/1508032497 shutdown_connections 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 --2- 172.21.15.67:0/1508032497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d3520 0x7efcbc0b8840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 --2- 172.21.15.67:0/1508032497 >> v2:172.21.15.187:3300/0 conn(0x7efcbc15ee20 0x7efcbc0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 -- 172.21.15.67:0/1508032497 >> 172.21.15.67:0/1508032497 conn(0x7efcbc06d3c0 msgr2=0x7efcbc0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 -- 172.21.15.67:0/1508032497 shutdown_connections 2024-04-17T07:14:45.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.577+0000 7efcc3200640 1 -- 172.21.15.67:0/1508032497 wait complete. 2024-04-17T07:14:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.578+0000 7efcc3200640 1 Processor -- start 2024-04-17T07:14:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.579+0000 7efcc3200640 1 -- start start 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.579+0000 7efcc3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.579+0000 7efcc3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc0d3520 0x7efcbc160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.579+0000 7efcc3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc15ee20 0x7efcbc157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.579+0000 7efcc3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc15f200 con 0x7efcbc0d18e0 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc0b8040 con 0x7efcbc0d3520 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc0b8d80 con 0x7efcbc15ee20 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40836/0 (socket says 172.21.15.67:40836) 2024-04-17T07:14:45.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 -- 172.21.15.67:0/521051585 learned_addr learned my addr 172.21.15.67:0/521051585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:45.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc2800640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc15ee20 0x7efcbc157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 -- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc15ee20 msgr2=0x7efcbc157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc15ee20 0x7efcbc157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 -- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc0d3520 msgr2=0x7efcbc160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1400640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc0d3520 0x7efcbc160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc0d3520 0x7efcbc160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.580+0000 7efcc1e00640 1 -- 172.21.15.67:0/521051585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc1582e0 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcc1e00640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcb000d620 tx=0x7efcb000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb000b620 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb0019070 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb000ba00 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc161310 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.581+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc15f4b0 con 0x7efcbc0d18e0 2024-04-17T07:14:45.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.582+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc068a10 con 0x7efcbc0d18e0 2024-04-17T07:14:45.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.583+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcb000bba0 con 0x7efcbc0d18e0 2024-04-17T07:14:45.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.584+0000 7efcb6000640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 0x7efca006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:45.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.584+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7efcb008bfe0 con 0x7efcbc0d18e0 2024-04-17T07:14:45.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.585+0000 7efcc1400640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 0x7efca006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:45.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.586+0000 7efcc1400640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 0x7efca006ede0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efcac007fd0 tx=0x7efcac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.589+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efcb005a380 con 0x7efcbc0d18e0 2024-04-17T07:14:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.811+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efcbc162440 con 0x7efcbc0d18e0 2024-04-17T07:14:45.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.812+0000 7efcb6000640 1 -- 172.21.15.67:0/521051585 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7efcb0059c80 con 0x7efcbc0d18e0 2024-04-17T07:14:45.812 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 msgr2=0x7efca006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 0x7efca006ede0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efcac007fd0 tx=0x7efcac009040 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 msgr2=0x7efcbc157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efcb000d620 tx=0x7efcb000dae0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 shutdown_connections 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7efca006c920 0x7efca006ede0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc15ee20 0x7efcbc157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc0d3520 0x7efcbc160c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 --2- 172.21.15.67:0/521051585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d18e0 0x7efcbc157a80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.815+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 >> 172.21.15.67:0/521051585 conn(0x7efcbc06d3c0 msgr2=0x7efcbc0cf290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.816+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 shutdown_connections 2024-04-17T07:14:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:45.816+0000 7efcc3200640 1 -- 172.21.15.67:0/521051585 wait complete. 2024-04-17T07:14:45.835 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:14:45.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:45.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-17T07:14:45.944 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:14:45.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:45.944 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:14:45.999 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:45.999 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:46.057 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:46.057 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:46.110 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:46.110 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:46.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:46.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.394+0000 7f53a5600640 1 Processor -- start 2024-04-17T07:14:46.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.394+0000 7f53a5600640 1 -- start start 2024-04-17T07:14:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.396+0000 7f53a5600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53a015b570 0x7f53a006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.396+0000 7f53a5600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a015bc80 con 0x7f53a006c040 2024-04-17T07:14:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.396+0000 7f53a5600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a015bf40 con 0x7f53a006fce0 2024-04-17T07:14:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.396+0000 7f53a5600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f53a006c470 con 0x7f53a015b570 2024-04-17T07:14:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.397+0000 7f53a4c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53a006c040 0x7f53a015f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45392/0 (socket says 172.21.15.67:45392) 2024-04-17T07:14:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.397+0000 7f53a4c00640 1 -- 172.21.15.67:0/2134128408 learned_addr learned my addr 172.21.15.67:0/2134128408 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.397+0000 7f539fe00640 1 --2- 172.21.15.67:0/2134128408 >> v2:172.21.15.187:3300/0 conn(0x7f53a015b570 0x7f53a006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.397+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475733138 0 0) 0x7f53a015bc80 con 0x7f53a006c040 2024-04-17T07:14:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.397+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5384003520 con 0x7f53a006c040 2024-04-17T07:14:46.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4245193192 0 0) 0x7f5384003520 con 0x7f53a006c040 2024-04-17T07:14:46.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 >> v2:172.21.15.187:3300/0 conn(0x7f53a015b570 msgr2=0x7f53a006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 --2- 172.21.15.67:0/2134128408 >> v2:172.21.15.187:3300/0 conn(0x7f53a015b570 0x7f53a006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 >> v1:172.21.15.187:6789/0 conn(0x7f53a006fce0 legacy=0x7f53a006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a015c220 con 0x7f53a006c040 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f53980019f0 con 0x7f53a006c040 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 >> v1:172.21.15.67:6789/0 conn(0x7f53a006c040 legacy=0x7f53a015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 0x7f5384003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384004800 0x7f5384004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53840052e0 0x7f538400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53a006c470 con 0x7f5384003a20 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f538400c190 con 0x7f53840052e0 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f538400c390 con 0x7f5384004800 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.398+0000 7f539fe00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 0x7f5384003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539f400640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384004800 0x7f5384004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539fe00640 1 -- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384004800 msgr2=0x7f5384004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539fe00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384004800 0x7f5384004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f53a4c00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53840052e0 0x7f538400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539fe00640 1 -- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53840052e0 msgr2=0x7f538400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539fe00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53840052e0 0x7f538400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539fe00640 1 -- 172.21.15.67:0/2134128408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5384003520 con 0x7f5384003a20 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f53a4c00640 1 -- 172.21.15.67:0/2134128408 reap_dead start 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.399+0000 7f539f400640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384004800 0x7f5384004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.400+0000 7f539fe00640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 0x7f5384003e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5390009a90 tx=0x7f5384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bf14f2c6f6cb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.400+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390039070 con 0x7f5384003a20 2024-04-17T07:14:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.400+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539002c680 con 0x7f5384003a20 2024-04-17T07:14:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.400+0000 7f539ea00640 1 -- 172.21.15.67:0/2134128408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390009690 con 0x7f5384003a20 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 -- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 msgr2=0x7f5384003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 0x7f5384003e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5390009a90 tx=0x7f5384005830 comp rx=0 tx=0).stop 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 -- 172.21.15.67:0/2134128408 shutdown_connections 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 --2- 172.21.15.67:0/2134128408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5384003a20 0x7f5384003e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 -- 172.21.15.67:0/2134128408 >> 172.21.15.67:0/2134128408 conn(0x7f53a0067d10 msgr2=0x7f53a00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:46.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 -- 172.21.15.67:0/2134128408 shutdown_connections 2024-04-17T07:14:46.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.401+0000 7f53a5600640 1 -- 172.21.15.67:0/2134128408 wait complete. 2024-04-17T07:14:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.403+0000 7f53a5600640 1 Processor -- start 2024-04-17T07:14:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.403+0000 7f53a5600640 1 -- start start 2024-04-17T07:14:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 0x7f53a0157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53a01f7e30 0x7f53a01f82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53a00b47b0 con 0x7f53a01485a0 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53a015b950 con 0x7f53a01f7e30 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53a00b4930 con 0x7f5384003a20 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40858/0 (socket says 172.21.15.67:40858) 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.404+0000 7f53a4c00640 1 -- 172.21.15.67:0/2182675150 learned_addr learned my addr 172.21.15.67:0/2182675150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f539f400640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53a01f7e30 0x7f53a01f82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f539fe00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 0x7f53a0157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 -- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 msgr2=0x7f53a0157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 0x7f53a0157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 -- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53a01f7e30 msgr2=0x7f53a01f82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53a01f7e30 0x7f53a01f82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 -- 172.21.15.67:0/2182675150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a0149670 con 0x7f53a01485a0 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f539fe00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 0x7f53a0157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a4c00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53a006f040 tx=0x7f539800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398004d60 con 0x7f53a01485a0 2024-04-17T07:14:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539800bb80 con 0x7f53a01485a0 2024-04-17T07:14:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a01498e0 con 0x7f53a01485a0 2024-04-17T07:14:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398007070 con 0x7f53a01485a0 2024-04-17T07:14:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.405+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a0149ea0 con 0x7f53a01485a0 2024-04-17T07:14:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.407+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a0068a10 con 0x7f53a01485a0 2024-04-17T07:14:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.408+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f539801b020 con 0x7f53a01485a0 2024-04-17T07:14:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.408+0000 7f5397e00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 0x7f538006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.409+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f539808bdf0 con 0x7f53a01485a0 2024-04-17T07:14:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.409+0000 7f539fe00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 0x7f538006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.410+0000 7f539fe00640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 0x7f538006ede0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5390036000 tx=0x7f5390030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.413+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f539805a100 con 0x7f53a01485a0 2024-04-17T07:14:46.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.620+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53a006f350 con 0x7f53a01485a0 2024-04-17T07:14:46.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.622+0000 7f5397e00640 1 -- 172.21.15.67:0/2182675150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7f5398059a00 con 0x7f53a01485a0 2024-04-17T07:14:46.623 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:46.623 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 msgr2=0x7f538006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 0x7f538006ede0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5390036000 tx=0x7f5390030000 comp rx=0 tx=0).stop 2024-04-17T07:14:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 msgr2=0x7f53a0148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53a006f040 tx=0x7f539800b4a0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 shutdown_connections 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a01485a0 0x7f53a0148a20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53a01f7e30 0x7f53a01f82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f538006c920 0x7f538006ede0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 --2- 172.21.15.67:0/2182675150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5384003a20 0x7f53a0157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 >> 172.21.15.67:0/2182675150 conn(0x7f53a0067d10 msgr2=0x7f53a00cefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.625+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 shutdown_connections 2024-04-17T07:14:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.626+0000 7f53a5600640 1 -- 172.21.15.67:0/2182675150 wait complete. 2024-04-17T07:14:46.645 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:46.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.875+0000 7f9531000640 1 Processor -- start 2024-04-17T07:14:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.875+0000 7f9531000640 1 -- start start 2024-04-17T07:14:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f9531000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f952c146060 0x7f952c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f9531000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c160920 con 0x7f952c145b00 2024-04-17T07:14:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f9531000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c160c00 con 0x7f952c0b4ac0 2024-04-17T07:14:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f9531000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f952c160ee0 con 0x7f952c146060 2024-04-17T07:14:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f952b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f952c145b00 0x7f952c15cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45402/0 (socket says 172.21.15.67:45402) 2024-04-17T07:14:46.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.877+0000 7f952b400640 1 -- 172.21.15.67:0/2906367712 learned_addr learned my addr 172.21.15.67:0/2906367712 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:46.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952aa00640 1 --2- 172.21.15.67:0/2906367712 >> v2:172.21.15.187:3300/0 conn(0x7f952c146060 0x7f952c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056457420 0 0) 0x7f952c160920 con 0x7f952c145b00 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9510003520 con 0x7f952c145b00 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3230069832 0 0) 0x7f9510003520 con 0x7f952c145b00 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 >> v2:172.21.15.187:3300/0 conn(0x7f952c146060 msgr2=0x7f952c160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 --2- 172.21.15.67:0/2906367712 >> v2:172.21.15.187:3300/0 conn(0x7f952c146060 0x7f952c160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 >> v1:172.21.15.187:6789/0 conn(0x7f952c0b4ac0 legacy=0x7f952c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c1611c0 con 0x7f952c145b00 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f951c001670 con 0x7f952c145b00 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.878+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 >> v1:172.21.15.67:6789/0 conn(0x7f952c145b00 legacy=0x7f952c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 0x7f9510003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510004800 0x7f9510004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95100052e0 0x7f951000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f952c160ee0 con 0x7f9510003a00 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f951000c200 con 0x7f95100052e0 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f951000c400 con 0x7f9510004800 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952aa00640 1 --2- 172.21.15.67:0/2906367712 >> v2:172.21.15.187:3300/0 conn(0x7f952c146060 0x7f952c160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 0x7f9510003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 -- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510004800 msgr2=0x7f9510004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510004800 0x7f9510004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 -- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95100052e0 msgr2=0x7f951000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952be00640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95100052e0 0x7f951000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95100052e0 0x7f951000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.879+0000 7f952b400640 1 -- 172.21.15.67:0/2906367712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9510003520 con 0x7f9510003a00 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.880+0000 7f952be00640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95100052e0 0x7f951000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.880+0000 7f952be00640 1 -- 172.21.15.67:0/2906367712 reap_dead start 2024-04-17T07:14:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.881+0000 7f952b400640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 0x7f9510003e20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f951c00c110 tx=0x7f9510005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebf6075b1b16706 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.881+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c007c40 con 0x7f9510003a00 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.881+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f951c030070 con 0x7f9510003a00 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.881+0000 7f952a000640 1 -- 172.21.15.67:0/2906367712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951c005d70 con 0x7f9510003a00 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 -- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 msgr2=0x7f9510003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 0x7f9510003e20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f951c00c110 tx=0x7f9510005830 comp rx=0 tx=0).stop 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 -- 172.21.15.67:0/2906367712 shutdown_connections 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 --2- 172.21.15.67:0/2906367712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9510003a00 0x7f9510003e20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 -- 172.21.15.67:0/2906367712 >> 172.21.15.67:0/2906367712 conn(0x7f952c0b23c0 msgr2=0x7f952c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 -- 172.21.15.67:0/2906367712 shutdown_connections 2024-04-17T07:14:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.882+0000 7f9531000640 1 -- 172.21.15.67:0/2906367712 wait complete. 2024-04-17T07:14:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.884+0000 7f9531000640 1 Processor -- start 2024-04-17T07:14:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.884+0000 7f9531000640 1 -- start start 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510003a00 0x7f952c15ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f952c1f5300 0x7f952c140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f952c160340 con 0x7f952c1f6550 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0cfba0 con 0x7f952c1f5300 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f9531000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f952c1604c0 con 0x7f9510003a00 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40872/0 (socket says 172.21.15.67:40872) 2024-04-17T07:14:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 -- 172.21.15.67:0/1904707015 learned_addr learned my addr 172.21.15.67:0/1904707015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 -- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510003a00 msgr2=0x7f952c15ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510003a00 0x7f952c15ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952aa00640 1 -- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f952c1f5300 msgr2=0x7f952c140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.885+0000 7f952be00640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f952c1f5300 0x7f952c140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f952aa00640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f952c1f5300 0x7f952c140080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f952aa00640 1 -- 172.21.15.67:0/1904707015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c13bb20 con 0x7f952c1f6550 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f952aa00640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f951800e7e0 tx=0x7f951800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951800f040 con 0x7f952c1f6550 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9518004230 con 0x7f952c1f6550 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951800c6d0 con 0x7f952c1f6550 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952c13be00 con 0x7f952c1f6550 2024-04-17T07:14:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.886+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f952c13c3c0 con 0x7f952c1f6550 2024-04-17T07:14:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.888+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f952c068a10 con 0x7f952c1f6550 2024-04-17T07:14:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.889+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f951800c870 con 0x7f952c1f6550 2024-04-17T07:14:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.889+0000 7f9523400640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 0x7f950c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:46.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.890+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f951808b8e0 con 0x7f952c1f6550 2024-04-17T07:14:46.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.890+0000 7f952b400640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 0x7f950c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.891+0000 7f952b400640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 0x7f950c06ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f951c034000 tx=0x7f951c007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:46.894+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9518017020 con 0x7f952c1f6550 2024-04-17T07:14:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.095+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f952c068610 con 0x7f952c1f6550 2024-04-17T07:14:47.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.096+0000 7f9523400640 1 -- 172.21.15.67:0/1904707015 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f9518059bf0 con 0x7f952c1f6550 2024-04-17T07:14:47.096 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:47.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 msgr2=0x7f950c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 0x7f950c06ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f951c034000 tx=0x7f951c007000 comp rx=0 tx=0).stop 2024-04-17T07:14:47.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 msgr2=0x7f952c1f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f951800e7e0 tx=0x7f951800eca0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 shutdown_connections 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f952c1f5300 0x7f952c140080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c1f6550 0x7f952c1f25c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f950c06c920 0x7f950c06ede0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 --2- 172.21.15.67:0/1904707015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9510003a00 0x7f952c15ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 >> 172.21.15.67:0/1904707015 conn(0x7f952c0b23c0 msgr2=0x7f952c0b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 shutdown_connections 2024-04-17T07:14:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.099+0000 7f9531000640 1 -- 172.21.15.67:0/1904707015 wait complete. 2024-04-17T07:14:47.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:47.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.351+0000 7f965ca00640 1 Processor -- start 2024-04-17T07:14:47.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.351+0000 7f965ca00640 1 -- start start 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f965ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9658162da0 0x7f96580b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f965ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9658166a60 con 0x7f96580b41a0 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f965ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9658166d40 con 0x7f96580b4580 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f965ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9658166810 con 0x7f9658162da0 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f9656800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96580b41a0 0x7f9658162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45406/0 (socket says 172.21.15.67:45406) 2024-04-17T07:14:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.353+0000 7f9656800640 1 -- 172.21.15.67:0/2305901937 learned_addr learned my addr 172.21.15.67:0/2305901937 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9657200640 1 --2- 172.21.15.67:0/2305901937 >> v2:172.21.15.187:3300/0 conn(0x7f9658162da0 0x7f96580b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932742729 0 0) 0x7f9658166a60 con 0x7f96580b41a0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f963c003520 con 0x7f96580b41a0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1284129908 0 0) 0x7f963c003520 con 0x7f96580b41a0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 >> v2:172.21.15.187:3300/0 conn(0x7f9658162da0 msgr2=0x7f96580b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 --2- 172.21.15.67:0/2305901937 >> v2:172.21.15.187:3300/0 conn(0x7f9658162da0 0x7f96580b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 >> v1:172.21.15.187:6789/0 conn(0x7f96580b4580 legacy=0x7f965806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9658167020 con 0x7f96580b41a0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f96480019f0 con 0x7f96580b41a0 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 >> v1:172.21.15.67:6789/0 conn(0x7f96580b41a0 legacy=0x7f9658162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9658166810 con 0x7f963c004800 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f963c00c0c0 con 0x7f963c003a00 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.354+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f963c00c2c0 con 0x7f963c0051c0 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f9657200640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 -- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c0051c0 msgr2=0x7f963c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 -- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f963c003a00 msgr2=0x7f963c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f963c003a00 0x7f963c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f964de00640 1 -- 172.21.15.67:0/2305901937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c003520 con 0x7f963c004800 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f9657200640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c0051c0 0x7f963c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.355+0000 7f9657200640 1 -- 172.21.15.67:0/2305901937 reap_dead start 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.356+0000 7f964de00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 0x7f963c004c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9640001750 tx=0x7f963c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96583cdc1ca8915a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640001370 con 0x7f963c004800 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96400052d0 con 0x7f963c004800 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f9655e00640 1 -- 172.21.15.67:0/2305901937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640003830 con 0x7f963c004800 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 -- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 msgr2=0x7f963c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 0x7f963c004c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9640001750 tx=0x7f963c005830 comp rx=0 tx=0).stop 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 -- 172.21.15.67:0/2305901937 shutdown_connections 2024-04-17T07:14:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 --2- 172.21.15.67:0/2305901937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f963c004800 0x7f963c004c80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 -- 172.21.15.67:0/2305901937 >> 172.21.15.67:0/2305901937 conn(0x7f96580b2190 msgr2=0x7f96580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 -- 172.21.15.67:0/2305901937 shutdown_connections 2024-04-17T07:14:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.357+0000 7f965ca00640 1 -- 172.21.15.67:0/2305901937 wait complete. 2024-04-17T07:14:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.359+0000 7f965ca00640 1 Processor -- start 2024-04-17T07:14:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.359+0000 7f965ca00640 1 -- start start 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c004800 0x7f96581f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96581f9db0 0x7f9658164640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f965806f980 con 0x7f96581f9320 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f965806f800 con 0x7f96581f9db0 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.360+0000 7f965ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f965806fb00 con 0x7f963c004800 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40882/0 (socket says 172.21.15.67:40882) 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 -- 172.21.15.67:0/3271485780 learned_addr learned my addr 172.21.15.67:0/3271485780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:47.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f9656800640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c004800 0x7f96581f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f9657200640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96581f9db0 0x7f9658164640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 -- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c004800 msgr2=0x7f96581f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c004800 0x7f96581f8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 -- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96581f9db0 msgr2=0x7f9658164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96581f9db0 0x7f9658164640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.361+0000 7f964de00640 1 -- 172.21.15.67:0/3271485780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96581635a0 con 0x7f96581f9320 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f964de00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9640005040 tx=0x7f9640001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640041070 con 0x7f96581f9320 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9640005480 con 0x7f96581f9320 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640001370 con 0x7f96581f9320 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9658163880 con 0x7f96581f9320 2024-04-17T07:14:47.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.362+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9658163e40 con 0x7f96581f9320 2024-04-17T07:14:47.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.363+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9658068a10 con 0x7f96581f9320 2024-04-17T07:14:47.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.364+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9640044070 con 0x7f96581f9320 2024-04-17T07:14:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.365+0000 7f964f400640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 0x7f963406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.365+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f96400ad690 con 0x7f96581f9320 2024-04-17T07:14:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.366+0000 7f9656800640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 0x7f963406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.367+0000 7f9656800640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 0x7f963406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f96580af850 tx=0x7f9648009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.369+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f96400052a0 con 0x7f96581f9320 2024-04-17T07:14:47.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.576+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f965806c7b0 con 0x7f96581f9320 2024-04-17T07:14:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.577+0000 7f964f400640 1 -- 172.21.15.67:0/3271485780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f964007b9a0 con 0x7f96581f9320 2024-04-17T07:14:47.577 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 msgr2=0x7f963406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 0x7f963406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f96580af850 tx=0x7f9648009040 comp rx=0 tx=0).stop 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 msgr2=0x7f96581f9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9640005040 tx=0x7f9640001120 comp rx=0 tx=0).stop 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 shutdown_connections 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96581f9db0 0x7f9658164640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581f9320 0x7f96581f9870 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f963406c920 0x7f963406ede0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 --2- 172.21.15.67:0/3271485780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f963c004800 0x7f96581f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 >> 172.21.15.67:0/3271485780 conn(0x7f96580b2190 msgr2=0x7f9658074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.580+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 shutdown_connections 2024-04-17T07:14:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.581+0000 7f965ca00640 1 -- 172.21.15.67:0/3271485780 wait complete. 2024-04-17T07:14:47.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:47.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.829+0000 7f02af200640 1 Processor -- start 2024-04-17T07:14:47.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.829+0000 7f02af200640 1 -- start start 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f02a81675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02af200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02a806ec40 con 0x7f02a806c8d0 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02af200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02a806ef20 con 0x7f02a806e6a0 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02a80ceb10 con 0x7f02a80d0600 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02ade00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02a806c8d0 0x7f02a815d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45412/0 (socket says 172.21.15.67:45412) 2024-04-17T07:14:47.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.831+0000 7f02ade00640 1 -- 172.21.15.67:0/2987331624 learned_addr learned my addr 172.21.15.67:0/2987331624 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11004174 0 0) 0x7f02a806ec40 con 0x7f02a806c8d0 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0290003520 con 0x7f02a806c8d0 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02ae800640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f02a81675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2725118996 0 0) 0x7f0290003520 con 0x7f02a806c8d0 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 msgr2=0x7f02a81675d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f02a81675d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 >> v1:172.21.15.187:6789/0 conn(0x7f02a806e6a0 legacy=0x7f02a806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a80cedf0 con 0x7f02a806c8d0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f029c001670 con 0x7f02a806c8d0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 >> v1:172.21.15.67:6789/0 conn(0x7f02a806c8d0 legacy=0x7f02a815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02900039e0 0x7f0290003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f0290004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0290004d40 0x7f02900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02a80ceb10 con 0x7f0290004d40 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f029000ffb0 con 0x7f02900039e0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.832+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02900101b0 con 0x7f02a80d0600 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0290004d40 0x7f02900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ade00640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02900039e0 0x7f0290003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ad400640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f0290004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 -- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 msgr2=0x7f0290004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a80d0600 0x7f0290004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 -- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02900039e0 msgr2=0x7f0290003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02900039e0 0x7f0290003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.833+0000 7f02ae800640 1 -- 172.21.15.67:0/2987331624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0290003520 con 0x7f0290004d40 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.834+0000 7f02ae800640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0290004d40 0x7f02900051c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02a4007360 tx=0x7f0290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd2695236ba14ba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4039070 con 0x7f0290004d40 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a402ccd0 con 0x7f0290004d40 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02aca00640 1 -- 172.21.15.67:0/2987331624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4009be0 con 0x7f0290004d40 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 -- 172.21.15.67:0/2987331624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0290004d40 msgr2=0x7f02900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 --2- 172.21.15.67:0/2987331624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0290004d40 0x7f02900051c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02a4007360 tx=0x7f0290005830 comp rx=0 tx=0).stop 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02ae800640 1 -- 172.21.15.67:0/2987331624 reap_dead start 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 -- 172.21.15.67:0/2987331624 shutdown_connections 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 -- 172.21.15.67:0/2987331624 >> 172.21.15.67:0/2987331624 conn(0x7f02a80ce600 msgr2=0x7f02a8072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 -- 172.21.15.67:0/2987331624 shutdown_connections 2024-04-17T07:14:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.835+0000 7f02af200640 1 -- 172.21.15.67:0/2987331624 wait complete. 2024-04-17T07:14:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.837+0000 7f02af200640 1 Processor -- start 2024-04-17T07:14:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.838+0000 7f02af200640 1 -- start start 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 0x7f02a81412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a8160160 0x7f02a81605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02a815a6d0 con 0x7f02a8143350 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f02a815a550 con 0x7f02a815ce10 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02a80b3d70 con 0x7f02a8160160 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02ad400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02ad400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40894/0 (socket says 172.21.15.67:40894) 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.839+0000 7f02ad400640 1 -- 172.21.15.67:0/405161423 learned_addr learned my addr 172.21.15.67:0/405161423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ade00640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 0x7f02a81412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 -- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a8160160 msgr2=0x7f02a81605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ae800640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a8160160 0x7f02a81605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a8160160 0x7f02a81605e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 -- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 msgr2=0x7f02a81412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 0x7f02a81412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 -- 172.21.15.67:0/405161423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a8143e10 con 0x7f02a8143350 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ade00640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 0x7f02a81412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02ad400640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f029800e4b0 tx=0x7f029800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.840+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02980052e0 con 0x7f02a8143350 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.841+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0298018070 con 0x7f02a8143350 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.841+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298003830 con 0x7f02a8143350 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.841+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a81440f0 con 0x7f02a8143350 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.841+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a815ecd0 con 0x7f02a8143350 2024-04-17T07:14:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.842+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a8160b20 con 0x7f02a8143350 2024-04-17T07:14:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.843+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f02980039d0 con 0x7f02a8143350 2024-04-17T07:14:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.844+0000 7f02a2000640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 0x7f028c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.844+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f029808b850 con 0x7f02a8143350 2024-04-17T07:14:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.845+0000 7f02ade00640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 0x7f028c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:47.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.846+0000 7f02ade00640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 0x7f028c06edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f029c001050 tx=0x7f029c000fe0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:47.848+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0298059b60 con 0x7f02a8143350 2024-04-17T07:14:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.049+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f02a8068a10 con 0x7f02a8143350 2024-04-17T07:14:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.050+0000 7f02a2000640 1 -- 172.21.15.67:0/405161423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f0298059460 con 0x7f02a8143350 2024-04-17T07:14:48.050 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 msgr2=0x7f028c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 0x7f028c06edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f029c001050 tx=0x7f029c000fe0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 msgr2=0x7f02a81437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f029800e4b0 tx=0x7f029800e970 comp rx=0 tx=0).stop 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 shutdown_connections 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f028c06c8f0 0x7f028c06edb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02a8160160 0x7f02a81605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02a8143350 0x7f02a81437d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 --2- 172.21.15.67:0/405161423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02a815ce10 0x7f02a81412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 >> 172.21.15.67:0/405161423 conn(0x7f02a80ce600 msgr2=0x7f02a80b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 shutdown_connections 2024-04-17T07:14:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.052+0000 7f02af200640 1 -- 172.21.15.67:0/405161423 wait complete. 2024-04-17T07:14:48.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:14:48.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.279+0000 7f1bb7200640 1 Processor -- start 2024-04-17T07:14:48.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.280+0000 7f1bb7200640 1 -- start start 2024-04-17T07:14:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.281+0000 7f1bb7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bb80cbc00 0x7f1bb80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.281+0000 7f1bb7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb815d060 con 0x7f1bb80b63c0 2024-04-17T07:14:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.281+0000 7f1bb7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb815d340 con 0x7f1bb8072570 2024-04-17T07:14:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.281+0000 7f1bb7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1bb80cfe90 con 0x7f1bb80cbc00 2024-04-17T07:14:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.282+0000 7f1bafe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bb80b63c0 0x7f1bb8071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45422/0 (socket says 172.21.15.67:45422) 2024-04-17T07:14:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.282+0000 7f1bafe00640 1 -- 172.21.15.67:0/2198889023 learned_addr learned my addr 172.21.15.67:0/2198889023 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.282+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/2198889023 >> v2:172.21.15.187:3300/0 conn(0x7f1bb80cbc00 0x7f1bb80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.282+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543190322 0 0) 0x7f1bb815d060 con 0x7f1bb80b63c0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.282+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b9c003520 con 0x7f1bb80b63c0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1732077140 0 0) 0x7f1b9c003520 con 0x7f1bb80b63c0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 >> v2:172.21.15.187:3300/0 conn(0x7f1bb80cbc00 msgr2=0x7f1bb80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 --2- 172.21.15.67:0/2198889023 >> v2:172.21.15.187:3300/0 conn(0x7f1bb80cbc00 0x7f1bb80cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 >> v1:172.21.15.187:6789/0 conn(0x7f1bb8072570 legacy=0x7f1bb80ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb815d620 con 0x7f1bb80b63c0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1ba0001670 con 0x7f1bb80b63c0 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 >> v1:172.21.15.67:6789/0 conn(0x7f1bb80b63c0 legacy=0x7f1bb8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b9c003a70 0x7f1b9c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1b9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b9c0052e0 0x7f1b9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80cfe90 con 0x7f1b9c004800 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9c00c1e0 con 0x7f1b9c0052e0 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b9c00c3e0 con 0x7f1b9c003a70 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/2198889023 >> v2:172.21.15.187:3300/0 conn(0x7f1bb80cbc00 0x7f1bb80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bafe00640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1b9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.283+0000 7f1bb6800640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b9c0052e0 0x7f1b9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bafe00640 1 -- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b9c003a70 msgr2=0x7f1b9c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b9c003a70 0x7f1b9c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bafe00640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b9c003a70 0x7f1b9c003e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bafe00640 1 -- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b9c0052e0 msgr2=0x7f1b9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bafe00640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b9c0052e0 0x7f1b9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bafe00640 1 -- 172.21.15.67:0/2198889023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9c003520 con 0x7f1b9c004800 2024-04-17T07:14:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.284+0000 7f1bb6800640 1 -- 172.21.15.67:0/2198889023 reap_dead start 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.285+0000 7f1bafe00640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1b9c004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007f90 tx=0x7f1b9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748b69708742cbcf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.285+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0007be0 con 0x7f1b9c004800 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.285+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba0030070 con 0x7f1b9c004800 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.285+0000 7f1bb5400640 1 -- 172.21.15.67:0/2198889023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0005d70 con 0x7f1b9c004800 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 -- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 msgr2=0x7f1b9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1b9c004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007f90 tx=0x7f1b9c005830 comp rx=0 tx=0).stop 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 -- 172.21.15.67:0/2198889023 shutdown_connections 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 --2- 172.21.15.67:0/2198889023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1b9c004c80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 -- 172.21.15.67:0/2198889023 >> 172.21.15.67:0/2198889023 conn(0x7f1bb806bef0 msgr2=0x7f1bb80b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 -- 172.21.15.67:0/2198889023 shutdown_connections 2024-04-17T07:14:48.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.286+0000 7f1bb7200640 1 -- 172.21.15.67:0/2198889023 wait complete. 2024-04-17T07:14:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.288+0000 7f1bb7200640 1 Processor -- start 2024-04-17T07:14:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.288+0000 7f1bb7200640 1 -- start start 2024-04-17T07:14:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80b1340 0x7f1bb80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bb8153f90 0x7f1bb8154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80d02d0 con 0x7f1b9c004800 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb815dc70 con 0x7f1bb80b1340 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb80cbfe0 con 0x7f1bb8153f90 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40922/0 (socket says 172.21.15.67:40922) 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.289+0000 7f1bb5e00640 1 -- 172.21.15.67:0/394763725 learned_addr learned my addr 172.21.15.67:0/394763725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 -- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bb8153f90 msgr2=0x7f1bb8154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bb8153f90 0x7f1bb8154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 -- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80b1340 msgr2=0x7f1bb80b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80b1340 0x7f1bb80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 -- 172.21.15.67:0/394763725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb8154ce0 con 0x7f1b9c004800 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb5e00640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800f490 tx=0x7f1ba800f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8018020 con 0x7f1b9c004800 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba8014890 con 0x7f1b9c004800 2024-04-17T07:14:48.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb8145890 con 0x7f1b9c004800 2024-04-17T07:14:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8014c60 con 0x7f1b9c004800 2024-04-17T07:14:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.290+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb8145e50 con 0x7f1b9c004800 2024-04-17T07:14:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.291+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb8068a10 con 0x7f1b9c004800 2024-04-17T07:14:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.292+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ba801b020 con 0x7f1b9c004800 2024-04-17T07:14:48.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.293+0000 7f1bae000640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 0x7f1b9406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:48.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.293+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f1ba808c100 con 0x7f1b9c004800 2024-04-17T07:14:48.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.294+0000 7f1bafe00640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 0x7f1b9406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.295+0000 7f1bafe00640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 0x7f1b9406ede0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0034000 tx=0x7f1ba002e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:48.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.299+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1ba8022020 con 0x7f1b9c004800 2024-04-17T07:14:48.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.508+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1bb80b3020 con 0x7f1b9c004800 2024-04-17T07:14:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.509+0000 7f1bae000640 1 -- 172.21.15.67:0/394763725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f1ba8024300 con 0x7f1b9c004800 2024-04-17T07:14:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.511+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 msgr2=0x7f1b9406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.511+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 0x7f1b9406ede0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0034000 tx=0x7f1ba002e000 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.511+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 msgr2=0x7f1bb8145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.511+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800f490 tx=0x7f1ba800f950 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 shutdown_connections 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f1b9406c920 0x7f1b9406ede0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bb8153f90 0x7f1bb8154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80b1340 0x7f1bb80b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 --2- 172.21.15.67:0/394763725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c004800 0x7f1bb8145130 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 >> 172.21.15.67:0/394763725 conn(0x7f1bb806bef0 msgr2=0x7f1bb80ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 shutdown_connections 2024-04-17T07:14:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:48.512+0000 7f1bb7200640 1 -- 172.21.15.67:0/394763725 wait complete. 2024-04-17T07:14:48.530 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-17T07:14:48.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:48.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-17T07:14:48.634 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:14:48.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:48.634 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-17T07:14:48.697 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:48.697 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:48.755 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:48.755 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:48.809 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:14:48.810 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:48.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:49.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.088+0000 7f09b7e00640 1 Processor -- start 2024-04-17T07:14:49.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.088+0000 7f09b7e00640 1 -- start start 2024-04-17T07:14:49.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.089+0000 7f09b7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b80c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.089+0000 7f09b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b80c6710 0x7f09b80cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.089+0000 7f09b7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b809b8a0 con 0x7f09b80ce1c0 2024-04-17T07:14:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.089+0000 7f09b7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f09b80cd7f0 con 0x7f09b80c6710 2024-04-17T07:14:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.089+0000 7f09b7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09b809bb80 con 0x7f09b815fe10 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.090+0000 7f09b6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b80c6710 0x7f09b80cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.090+0000 7f09b6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b80c6710 0x7f09b80cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40936/0 (socket says 172.21.15.67:40936) 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.090+0000 7f09b6000640 1 -- 172.21.15.67:0/623202621 learned_addr learned my addr 172.21.15.67:0/623202621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.090+0000 7f09b6a00640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b80c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.090+0000 7f09b6000640 1 -- 172.21.15.67:0/623202621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 msgr2=0x7f09b80c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.091+0000 7f09b6000640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b80c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.091+0000 7f09b6000640 1 -- 172.21.15.67:0/623202621 >> v1:172.21.15.187:6789/0 conn(0x7f09b80ce1c0 legacy=0x7f09b8160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.091+0000 7f09b6000640 1 -- 172.21.15.67:0/623202621 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b80cd480 con 0x7f09b80c6710 2024-04-17T07:14:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.092+0000 7f09b6000640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80cdc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009e50 tx=0x7f09a402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669b3382943a5cb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.092+0000 7f09b5600640 1 -- 172.21.15.67:0/623202621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a402caf0 con 0x7f09b80c6710 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.092+0000 7f09b5600640 1 -- 172.21.15.67:0/623202621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a402cc90 con 0x7f09b80c6710 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.092+0000 7f09b5600640 1 -- 172.21.15.67:0/623202621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4034610 con 0x7f09b80c6710 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 -- 172.21.15.67:0/623202621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 msgr2=0x7f09b80cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80cdc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009e50 tx=0x7f09a402bf50 comp rx=0 tx=0).stop 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 -- 172.21.15.67:0/623202621 shutdown_connections 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80cdc80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 --2- 172.21.15.67:0/623202621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b80c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 -- 172.21.15.67:0/623202621 >> 172.21.15.67:0/623202621 conn(0x7f09b80b4170 msgr2=0x7f09b80b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 -- 172.21.15.67:0/623202621 shutdown_connections 2024-04-17T07:14:49.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.093+0000 7f09b7e00640 1 -- 172.21.15.67:0/623202621 wait complete. 2024-04-17T07:14:49.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.095+0000 7f09b7e00640 1 Processor -- start 2024-04-17T07:14:49.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.095+0000 7f09b7e00640 1 -- start start 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09b80ce1c0 0x7f09b806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b806e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09b8160370 con 0x7f09b80c6710 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f09b81601f0 con 0x7f09b80ce1c0 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09b809b470 con 0x7f09b815fe10 2024-04-17T07:14:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40950/0 (socket says 172.21.15.67:40950) 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 -- 172.21.15.67:0/881586589 learned_addr learned my addr 172.21.15.67:0/881586589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 -- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 msgr2=0x7f09b806e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 -- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09b80ce1c0 msgr2=0x7f09b806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09b80ce1c0 0x7f09b806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.096+0000 7f09b6a00640 1 -- 172.21.15.67:0/881586589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b806d260 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09b6a00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f09a800d5f0 tx=0x7f09a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80125f0 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a8019070 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80129d0 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b806d540 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.097+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b80cace0 con 0x7f09b80c6710 2024-04-17T07:14:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.098+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b80755d0 con 0x7f09b80c6710 2024-04-17T07:14:49.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.099+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09a8012b70 con 0x7f09b80c6710 2024-04-17T07:14:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.100+0000 7f09aea00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 0x7f099806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.100+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f09a808bfe0 con 0x7f09b80c6710 2024-04-17T07:14:49.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.101+0000 7f09b6000640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 0x7f099806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.102+0000 7f09b6000640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 0x7f099806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09a4035000 tx=0x7f09a402e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.104+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f09a805a380 con 0x7f09b80c6710 2024-04-17T07:14:49.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.310+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09b80b4ed0 con 0x7f09b80c6710 2024-04-17T07:14:49.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.311+0000 7f09aea00640 1 -- 172.21.15.67:0/881586589 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7f09a8059c80 con 0x7f09b80c6710 2024-04-17T07:14:49.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:49.312 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 msgr2=0x7f099806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 0x7f099806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f09a4035000 tx=0x7f09a402e000 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 msgr2=0x7f09b80ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f09a800d5f0 tx=0x7f09a800dab0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 shutdown_connections 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f099806c8f0 0x7f099806edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09b815fe10 0x7f09b806e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09b80ce1c0 0x7f09b806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 --2- 172.21.15.67:0/881586589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b80c6710 0x7f09b80ca7a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 >> 172.21.15.67:0/881586589 conn(0x7f09b80b4170 msgr2=0x7f09b809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 shutdown_connections 2024-04-17T07:14:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.314+0000 7f09b7e00640 1 -- 172.21.15.67:0/881586589 wait complete. 2024-04-17T07:14:49.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:49.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.560+0000 7f8627200640 1 Processor -- start 2024-04-17T07:14:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.561+0000 7f8627200640 1 -- start start 2024-04-17T07:14:49.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 0x7f86280d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f86280b3b50 0x7f862806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f862806cf80 0x7f86280cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f86280d4f70 con 0x7f862806cf80 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f86280d5250 con 0x7f86280b3b50 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.562+0000 7f8627200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f86280cf4d0 con 0x7f862815ec20 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f862806cf80 0x7f86280cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f862806cf80 0x7f86280cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40964/0 (socket says 172.21.15.67:40964) 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 -- 172.21.15.67:0/626263355 learned_addr learned my addr 172.21.15.67:0/626263355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f861fe00640 1 --2- 172.21.15.67:0/626263355 >> v2:172.21.15.187:3300/0 conn(0x7f86280b3b50 0x7f862806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8625e00640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 0x7f86280d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 -- 172.21.15.67:0/626263355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 msgr2=0x7f86280d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 0x7f86280d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 -- 172.21.15.67:0/626263355 >> v2:172.21.15.187:3300/0 conn(0x7f86280b3b50 msgr2=0x7f862806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.563+0000 7f8626800640 1 --2- 172.21.15.67:0/626263355 >> v2:172.21.15.187:3300/0 conn(0x7f86280b3b50 0x7f862806ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.564+0000 7f8626800640 1 -- 172.21.15.67:0/626263355 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280cf7b0 con 0x7f862806cf80 2024-04-17T07:14:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.564+0000 7f8625e00640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 0x7f86280d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:49.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.565+0000 7f8626800640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862806cf80 0x7f86280cfcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f86280694b0 tx=0x7f862002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e15a67b2674673f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.566+0000 7f8625400640 1 -- 172.21.15.67:0/626263355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862002c960 con 0x7f862806cf80 2024-04-17T07:14:49.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.566+0000 7f8625400640 1 -- 172.21.15.67:0/626263355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862002cb00 con 0x7f862806cf80 2024-04-17T07:14:49.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.566+0000 7f8625400640 1 -- 172.21.15.67:0/626263355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200343f0 con 0x7f862806cf80 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 -- 172.21.15.67:0/626263355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862806cf80 msgr2=0x7f86280cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862806cf80 0x7f86280cfcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f86280694b0 tx=0x7f862002bdc0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 -- 172.21.15.67:0/626263355 shutdown_connections 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862806cf80 0x7f86280cfcd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 --2- 172.21.15.67:0/626263355 >> v2:172.21.15.187:3300/0 conn(0x7f86280b3b50 0x7f862806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 --2- 172.21.15.67:0/626263355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f862815ec20 0x7f86280d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 -- 172.21.15.67:0/626263355 >> 172.21.15.67:0/626263355 conn(0x7f862808bff0 msgr2=0x7f86280d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 -- 172.21.15.67:0/626263355 shutdown_connections 2024-04-17T07:14:49.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.567+0000 7f8627200640 1 -- 172.21.15.67:0/626263355 wait complete. 2024-04-17T07:14:49.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.569+0000 7f8627200640 1 Processor -- start 2024-04-17T07:14:49.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.569+0000 7f8627200640 1 -- start start 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f862806cf80 0x7f8628152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86280b3b50 0x7f86280b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86280d0390 con 0x7f862815ec20 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f86280d0210 con 0x7f862806cf80 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8627200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f86280d0510 con 0x7f86280b3b50 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8626800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8626800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40974/0 (socket says 172.21.15.67:40974) 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8626800640 1 -- 172.21.15.67:0/1288673889 learned_addr learned my addr 172.21.15.67:0/1288673889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f8625e00640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f862806cf80 0x7f8628152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.570+0000 7f861fe00640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86280b3b50 0x7f86280b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 -- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86280b3b50 msgr2=0x7f86280b6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86280b3b50 0x7f86280b6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 -- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f862806cf80 msgr2=0x7f8628152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f862806cf80 0x7f8628152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 -- 172.21.15.67:0/1288673889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86280b58c0 con 0x7f862815ec20 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.571+0000 7f8626800640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8620036040 tx=0x7f8620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.572+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620043020 con 0x7f862815ec20 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.572+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862003e450 con 0x7f862815ec20 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.572+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862003e820 con 0x7f862815ec20 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.572+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86280b5ba0 con 0x7f862815ec20 2024-04-17T07:14:49.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.573+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86280b6160 con 0x7f862815ec20 2024-04-17T07:14:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.574+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8628068a10 con 0x7f862815ec20 2024-04-17T07:14:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.575+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8620046020 con 0x7f862815ec20 2024-04-17T07:14:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.576+0000 7f861e000640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 0x7f860406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.576+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f86200ae3f0 con 0x7f862815ec20 2024-04-17T07:14:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.577+0000 7f8625e00640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 0x7f860406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.578+0000 7f8625e00640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 0x7f860406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f861800a990 tx=0x7f8618008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:49.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.581+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f862004d030 con 0x7f862815ec20 2024-04-17T07:14:49.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.787+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86280d55f0 con 0x7f862815ec20 2024-04-17T07:14:49.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.788+0000 7f861e000640 1 -- 172.21.15.67:0/1288673889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f862007c670 con 0x7f862815ec20 2024-04-17T07:14:49.788 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:49.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 msgr2=0x7f860406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 0x7f860406edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f861800a990 tx=0x7f8618008040 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 msgr2=0x7f8628160170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8620036040 tx=0x7f8620004a40 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 shutdown_connections 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f862815ec20 0x7f8628160170 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f860406c8f0 0x7f860406edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86280b3b50 0x7f86280b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 --2- 172.21.15.67:0/1288673889 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f862806cf80 0x7f8628152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 >> 172.21.15.67:0/1288673889 conn(0x7f862808bff0 msgr2=0x7f86280d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 shutdown_connections 2024-04-17T07:14:49.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:49.791+0000 7f8627200640 1 -- 172.21.15.67:0/1288673889 wait complete. 2024-04-17T07:14:49.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:50.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.030+0000 7f51c7e00640 1 Processor -- start 2024-04-17T07:14:50.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.031+0000 7f51c7e00640 1 -- start start 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.032+0000 7f51c7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c80b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.032+0000 7f51c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51c8162da0 0x7f51c806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.032+0000 7f51c7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c8166a60 con 0x7f51c80b41a0 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.032+0000 7f51c7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f51c8166810 con 0x7f51c8162da0 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.032+0000 7f51c7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f51c8166d40 con 0x7f51c80b4580 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51c8162da0 0x7f51c806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51c8162da0 0x7f51c806dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40990/0 (socket says 172.21.15.67:40990) 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 -- 172.21.15.67:0/308460993 learned_addr learned my addr 172.21.15.67:0/308460993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 -- 172.21.15.67:0/308460993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 msgr2=0x7f51c80b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:50.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 --2- 172.21.15.67:0/308460993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c80b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 -- 172.21.15.67:0/308460993 >> v1:172.21.15.187:6789/0 conn(0x7f51c80b41a0 legacy=0x7f51c8162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:50.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.033+0000 7f51c7400640 1 -- 172.21.15.67:0/308460993 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c8167020 con 0x7f51c8162da0 2024-04-17T07:14:50.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.034+0000 7f51c7400640 1 --2- 172.21.15.67:0/308460993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c8162da0 0x7f51c806dd30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009870 tx=0x7f51c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa54164263b649a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.035+0000 7f51c6000640 1 -- 172.21.15.67:0/308460993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0035020 con 0x7f51c8162da0 2024-04-17T07:14:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.035+0000 7f51c6000640 1 -- 172.21.15.67:0/308460993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c0031640 con 0x7f51c8162da0 2024-04-17T07:14:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.035+0000 7f51c6000640 1 -- 172.21.15.67:0/308460993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c002d040 con 0x7f51c8162da0 2024-04-17T07:14:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.035+0000 7f51c7e00640 1 -- 172.21.15.67:0/308460993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c8162da0 msgr2=0x7f51c806dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.035+0000 7f51c7e00640 1 --2- 172.21.15.67:0/308460993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c8162da0 0x7f51c806dd30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009870 tx=0x7f51c002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 -- 172.21.15.67:0/308460993 shutdown_connections 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 --2- 172.21.15.67:0/308460993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c8162da0 0x7f51c806dd30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 --2- 172.21.15.67:0/308460993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c80b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 -- 172.21.15.67:0/308460993 >> 172.21.15.67:0/308460993 conn(0x7f51c80b2190 msgr2=0x7f51c80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 -- 172.21.15.67:0/308460993 shutdown_connections 2024-04-17T07:14:50.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.036+0000 7f51c7e00640 1 -- 172.21.15.67:0/308460993 wait complete. 2024-04-17T07:14:50.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.037+0000 7f51c7e00640 1 Processor -- start 2024-04-17T07:14:50.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 -- start start 2024-04-17T07:14:50.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c8165c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51c8162da0 0x7f51c8156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51c806f980 con 0x7f51c80b41a0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f51c806f800 con 0x7f51c8162da0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.038+0000 7f51c7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f51c806fb00 con 0x7f51c80b4580 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40998/0 (socket says 172.21.15.67:40998) 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 -- 172.21.15.67:0/3149785044 learned_addr learned my addr 172.21.15.67:0/3149785044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51be000640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c8165c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c7400640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51c8162da0 0x7f51c8156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 -- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 msgr2=0x7f51c8165c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c8165c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 -- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51c8162da0 msgr2=0x7f51c8156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51c8162da0 0x7f51c8156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.039+0000 7f51c6a00640 1 -- 172.21.15.67:0/3149785044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c81fca90 con 0x7f51c80b41a0 2024-04-17T07:14:50.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51c6a00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c7c0 tx=0x7f51b800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8016020 con 0x7f51c80b41a0 2024-04-17T07:14:50.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b800aa50 con 0x7f51c80b41a0 2024-04-17T07:14:50.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8012420 con 0x7f51c80b41a0 2024-04-17T07:14:50.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c81fcd70 con 0x7f51c80b41a0 2024-04-17T07:14:50.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.040+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c8157b90 con 0x7f51c80b41a0 2024-04-17T07:14:50.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.041+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c8068a10 con 0x7f51c80b41a0 2024-04-17T07:14:50.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.043+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f51b8019020 con 0x7f51c80b41a0 2024-04-17T07:14:50.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.043+0000 7f51bf400640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 0x7f51a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.043+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f51b808b8f0 con 0x7f51c80b41a0 2024-04-17T07:14:50.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.044+0000 7f51be000640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 0x7f51a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.045+0000 7f51be000640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 0x7f51a406edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51c8161440 tx=0x7f51b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.048+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f51b8020070 con 0x7f51c80b41a0 2024-04-17T07:14:50.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.254+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f51c806c7b0 con 0x7f51c80b41a0 2024-04-17T07:14:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.255+0000 7f51bf400640 1 -- 172.21.15.67:0/3149785044 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f51b8059d30 con 0x7f51c80b41a0 2024-04-17T07:14:50.255 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.257+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 msgr2=0x7f51a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.257+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 0x7f51a406edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f51c8161440 tx=0x7f51b0008040 comp rx=0 tx=0).stop 2024-04-17T07:14:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.257+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 msgr2=0x7f51c81fc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.257+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c7c0 tx=0x7f51b800cc80 comp rx=0 tx=0).stop 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 shutdown_connections 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51c8162da0 0x7f51c8156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f51a406c8f0 0x7f51a406edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51c80b4580 0x7f51c8165c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 --2- 172.21.15.67:0/3149785044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51c80b41a0 0x7f51c81fc3b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 >> 172.21.15.67:0/3149785044 conn(0x7f51c80b2190 msgr2=0x7f51c8071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 shutdown_connections 2024-04-17T07:14:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.258+0000 7f51c7e00640 1 -- 172.21.15.67:0/3149785044 wait complete. 2024-04-17T07:14:50.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:50.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.503+0000 7fc5ff200640 1 Processor -- start 2024-04-17T07:14:50.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.504+0000 7fc5ff200640 1 -- start start 2024-04-17T07:14:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.505+0000 7fc5ff200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5f8144fb0 0x7fc5f80b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.505+0000 7fc5ff200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f815f240 con 0x7fc5f80cf620 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.505+0000 7fc5ff200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f80b6420 con 0x7fc5f815fcb0 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.505+0000 7fc5ff200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc5f8160140 con 0x7fc5f8144fb0 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5f8144fb0 0x7fc5f80b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fe800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5f80cf620 0x7fc5f806e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45424/0 (socket says 172.21.15.67:45424) 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fe800640 1 -- 172.21.15.67:0/4015321144 learned_addr learned my addr 172.21.15.67:0/4015321144 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fde00640 1 -- 172.21.15.67:0/4015321144 >> v1:172.21.15.187:6789/0 conn(0x7fc5f815fcb0 legacy=0x7fc5f80b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fde00640 1 -- 172.21.15.67:0/4015321144 >> v1:172.21.15.67:6789/0 conn(0x7fc5f80cf620 legacy=0x7fc5f806e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fca00640 1 -- 172.21.15.67:0/4015321144 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312441876 0 0) 0x7fc5f815f240 con 0x7fc5f80cf620 2024-04-17T07:14:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.506+0000 7fc5fde00640 1 -- 172.21.15.67:0/4015321144 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f815f4b0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.507+0000 7fc5fde00640 1 --2- 172.21.15.67:0/4015321144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b4260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0097a0 tx=0x7fc5ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cf95f9e7c36203 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5fca00640 1 -- 172.21.15.67:0/4015321144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec0315a0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5fca00640 1 -- 172.21.15.67:0/4015321144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec037070 con 0x7fc5f8144fb0 2024-04-17T07:14:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5fca00640 1 -- 172.21.15.67:0/4015321144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec031960 con 0x7fc5f8144fb0 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 -- 172.21.15.67:0/4015321144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 msgr2=0x7fc5f80b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 --2- 172.21.15.67:0/4015321144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b4260 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0097a0 tx=0x7fc5ec02b840 comp rx=0 tx=0).stop 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 -- 172.21.15.67:0/4015321144 shutdown_connections 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 --2- 172.21.15.67:0/4015321144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b4260 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 -- 172.21.15.67:0/4015321144 >> 172.21.15.67:0/4015321144 conn(0x7fc5f80b23c0 msgr2=0x7fc5f80b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 -- 172.21.15.67:0/4015321144 shutdown_connections 2024-04-17T07:14:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.508+0000 7fc5ff200640 1 -- 172.21.15.67:0/4015321144 wait complete. 2024-04-17T07:14:50.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.510+0000 7fc5ff200640 1 Processor -- start 2024-04-17T07:14:50.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.510+0000 7fc5ff200640 1 -- start start 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5f80cf620 0x7fc5f8139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5f815fcb0 0x7fc5f815bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f80b6700 con 0x7fc5f8144fb0 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f80b6200 con 0x7fc5f815fcb0 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.511+0000 7fc5ff200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f8145390 con 0x7fc5f80cf620 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41010/0 (socket says 172.21.15.67:41010) 2024-04-17T07:14:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 -- 172.21.15.67:0/943630416 learned_addr learned my addr 172.21.15.67:0/943630416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fde00640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5f80cf620 0x7fc5f8139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fe800640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5f815fcb0 0x7fc5f815bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 -- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5f80cf620 msgr2=0x7fc5f8139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5f80cf620 0x7fc5f8139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 -- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5f815fcb0 msgr2=0x7fc5f815bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5f815fcb0 0x7fc5f815bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.512+0000 7fc5fd400640 1 -- 172.21.15.67:0/943630416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f815cbc0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5fd400640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800c430 tx=0x7fc5e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8011790 con 0x7fc5f8144fb0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e8017070 con 0x7fc5f8144fb0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8011bb0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f815c8e0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.513+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f81397e0 con 0x7fc5f8144fb0 2024-04-17T07:14:50.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.514+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f8068a10 con 0x7fc5f8144fb0 2024-04-17T07:14:50.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.515+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5e801c460 con 0x7fc5f8144fb0 2024-04-17T07:14:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.516+0000 7fc5f2000640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 0x7fc5dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.516+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fc5e808bb30 con 0x7fc5f8144fb0 2024-04-17T07:14:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.517+0000 7fc5fde00640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 0x7fc5dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.518+0000 7fc5fde00640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 0x7fc5dc06edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.521+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc5e8059e40 con 0x7fc5f8144fb0 2024-04-17T07:14:50.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.729+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc5f8068610 con 0x7fc5f8144fb0 2024-04-17T07:14:50.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.730+0000 7fc5f2000640 1 -- 172.21.15.67:0/943630416 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fc5e8059740 con 0x7fc5f8144fb0 2024-04-17T07:14:50.730 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.732+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 msgr2=0x7fc5dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.732+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 0x7fc5dc06edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03d040 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.732+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 msgr2=0x7fc5f80b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.732+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800c430 tx=0x7fc5e800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 shutdown_connections 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5f815fcb0 0x7fc5f815bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f8144fb0 0x7fc5f80b0f40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5dc06c8f0 0x7fc5dc06edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 --2- 172.21.15.67:0/943630416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5f80cf620 0x7fc5f8139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 >> 172.21.15.67:0/943630416 conn(0x7fc5f80b23c0 msgr2=0x7fc5f8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 shutdown_connections 2024-04-17T07:14:50.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.733+0000 7fc5ff200640 1 -- 172.21.15.67:0/943630416 wait complete. 2024-04-17T07:14:50.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:14:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.980+0000 7f0381e00640 1 Processor -- start 2024-04-17T07:14:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.980+0000 7f0381e00640 1 -- start start 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f037c072570 0x7f037c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f037c0b63c0 0x7f037c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037c15d060 con 0x7f037c0cbc00 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f037c0cfe90 con 0x7f037c0b63c0 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f037c15dc70 con 0x7f037c072570 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0380a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f037c0cbc00 0x7f037c071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45428/0 (socket says 172.21.15.67:45428) 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0380a00640 1 -- 172.21.15.67:0/778866931 learned_addr learned my addr 172.21.15.67:0/778866931 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f0381400640 1 --2- 172.21.15.67:0/778866931 >> v2:172.21.15.67:3300/0 conn(0x7f037c0b63c0 0x7f037c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.982+0000 7f037be00640 1 --2- 172.21.15.67:0/778866931 >> v2:172.21.15.187:3300/0 conn(0x7f037c072570 0x7f037c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.983+0000 7f037b400640 1 -- 172.21.15.67:0/778866931 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1779869067 0 0) 0x7f037c15d060 con 0x7f037c0cbc00 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.983+0000 7f0381400640 1 -- 172.21.15.67:0/778866931 >> v2:172.21.15.187:3300/0 conn(0x7f037c072570 msgr2=0x7f037c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.983+0000 7f0381400640 1 --2- 172.21.15.67:0/778866931 >> v2:172.21.15.187:3300/0 conn(0x7f037c072570 0x7f037c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.983+0000 7f0381400640 1 -- 172.21.15.67:0/778866931 >> v1:172.21.15.67:6789/0 conn(0x7f037c0cbc00 legacy=0x7f037c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.983+0000 7f0381400640 1 -- 172.21.15.67:0/778866931 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c15d340 con 0x7f037c0b63c0 2024-04-17T07:14:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.984+0000 7f0381400640 1 --2- 172.21.15.67:0/778866931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c0b63c0 0x7f037c0ce610 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f037c0caee0 tx=0x7f037402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d327d9526dbd726e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.985+0000 7f037b400640 1 -- 172.21.15.67:0/778866931 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374039070 con 0x7f037c0b63c0 2024-04-17T07:14:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.985+0000 7f037b400640 1 -- 172.21.15.67:0/778866931 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037402cc80 con 0x7f037c0b63c0 2024-04-17T07:14:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.985+0000 7f037b400640 1 -- 172.21.15.67:0/778866931 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374034660 con 0x7f037c0b63c0 2024-04-17T07:14:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.985+0000 7f0381e00640 1 -- 172.21.15.67:0/778866931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c0b63c0 msgr2=0x7f037c0ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.985+0000 7f0381e00640 1 --2- 172.21.15.67:0/778866931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c0b63c0 0x7f037c0ce610 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f037c0caee0 tx=0x7f037402bed0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 -- 172.21.15.67:0/778866931 shutdown_connections 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 --2- 172.21.15.67:0/778866931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c0b63c0 0x7f037c0ce610 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 --2- 172.21.15.67:0/778866931 >> v2:172.21.15.187:3300/0 conn(0x7f037c072570 0x7f037c0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 -- 172.21.15.67:0/778866931 >> 172.21.15.67:0/778866931 conn(0x7f037c06bef0 msgr2=0x7f037c0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 -- 172.21.15.67:0/778866931 shutdown_connections 2024-04-17T07:14:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.986+0000 7f0381e00640 1 -- 172.21.15.67:0/778866931 wait complete. 2024-04-17T07:14:50.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.987+0000 7f0381e00640 1 Processor -- start 2024-04-17T07:14:50.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.988+0000 7f0381e00640 1 -- start start 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.988+0000 7f0381e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.988+0000 7f0381e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f037c0b63c0 0x7f037c15b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0381e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037c0cbc00 0x7f037c148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0381e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f037c0d02d0 con 0x7f037c072570 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0381e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15d900 con 0x7f037c0cbc00 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0381e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f037c0cbfe0 con 0x7f037c0b63c0 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41020/0 (socket says 172.21.15.67:41020) 2024-04-17T07:14:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 -- 172.21.15.67:0/742298099 learned_addr learned my addr 172.21.15.67:0/742298099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 -- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f037c0b63c0 msgr2=0x7f037c15b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f037c0b63c0 0x7f037c15b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 -- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037c0cbc00 msgr2=0x7f037c148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037c0cbc00 0x7f037c148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.989+0000 7f0380a00640 1 -- 172.21.15.67:0/742298099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c15bed0 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0380a00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f036400c430 tx=0x7f036400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03640074e0 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03640047e0 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364004bd0 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037c15c170 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.990+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037c15c730 con 0x7f037c072570 2024-04-17T07:14:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.991+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037c068a10 con 0x7f037c072570 2024-04-17T07:14:50.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.992+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0364011070 con 0x7f037c072570 2024-04-17T07:14:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.993+0000 7f0378c00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 0x7f035806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.993+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f036408b820 con 0x7f037c072570 2024-04-17T07:14:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.994+0000 7f037be00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 0x7f035806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.995+0000 7f037be00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 0x7f035806edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f036c006a90 tx=0x7f036c006a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:50.998+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0364059c40 con 0x7f037c072570 2024-04-17T07:14:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.214+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f037c0b3020 con 0x7f037c072570 2024-04-17T07:14:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.215+0000 7f0378c00640 1 -- 172.21.15.67:0/742298099 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f0364059540 con 0x7f037c072570 2024-04-17T07:14:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T07:14:51.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 msgr2=0x7f035806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 0x7f035806edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f036c006a90 tx=0x7f036c006a20 comp rx=0 tx=0).stop 2024-04-17T07:14:51.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 msgr2=0x7f037c14bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f036400c430 tx=0x7f036400c8f0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 shutdown_connections 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f037c0cbc00 0x7f037c148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f035806c8f0 0x7f035806edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f037c0b63c0 0x7f037c15b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 --2- 172.21.15.67:0/742298099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c072570 0x7f037c14bc00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 >> 172.21.15.67:0/742298099 conn(0x7f037c06bef0 msgr2=0x7f037c0ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 shutdown_connections 2024-04-17T07:14:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.218+0000 7f0381e00640 1 -- 172.21.15.67:0/742298099 wait complete. 2024-04-17T07:14:51.240 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-17T07:14:51.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:51.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-17T07:14:51.334 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:14:51.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:51.334 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-17T07:14:51.385 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:51.386 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:51.443 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:51.443 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:51.493 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:14:51.493 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:14:51.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.774+0000 7fb753000640 1 Processor -- start 2024-04-17T07:14:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.775+0000 7fb753000640 1 -- start start 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.776+0000 7fb753000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.776+0000 7fb753000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb74c0cf910 0x7fb74c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.776+0000 7fb753000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb74c0d2d30 con 0x7fb74c15ec60 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.776+0000 7fb753000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb74c0d49c0 con 0x7fb74c0cf910 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.776+0000 7fb753000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c0d3660 con 0x7fb74c0b6d00 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb751c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb74c15ec60 0x7fb74c0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45442/0 (socket says 172.21.15.67:45442) 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb751c00640 1 -- 172.21.15.67:0/2858059142 learned_addr learned my addr 172.21.15.67:0/2858059142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb752600640 1 --2- 172.21.15.67:0/2858059142 >> v2:172.21.15.67:3300/0 conn(0x7fb74c0cf910 0x7fb74c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:51.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb751200640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb74be00640 1 -- 172.21.15.67:0/2858059142 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854754789 0 0) 0x7fb74c0d2d30 con 0x7fb74c15ec60 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb74be00640 1 -- 172.21.15.67:0/2858059142 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb734003520 con 0x7fb74c15ec60 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb752600640 1 -- 172.21.15.67:0/2858059142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 msgr2=0x7fb74c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb752600640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb752600640 1 -- 172.21.15.67:0/2858059142 >> v1:172.21.15.67:6789/0 conn(0x7fb74c15ec60 legacy=0x7fb74c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb752600640 1 -- 172.21.15.67:0/2858059142 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c0d3010 con 0x7fb74c0cf910 2024-04-17T07:14:51.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.777+0000 7fb751200640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.779+0000 7fb752600640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0b7b70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb744009d90 tx=0x7fb74402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f29b1c33805832a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.779+0000 7fb74be00640 1 -- 172.21.15.67:0/2858059142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744039070 con 0x7fb74c0cf910 2024-04-17T07:14:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.779+0000 7fb74be00640 1 -- 172.21.15.67:0/2858059142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb74402cc60 con 0x7fb74c0cf910 2024-04-17T07:14:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.779+0000 7fb74be00640 1 -- 172.21.15.67:0/2858059142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744034660 con 0x7fb74c0cf910 2024-04-17T07:14:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 -- 172.21.15.67:0/2858059142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 msgr2=0x7fb74c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0b7b70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb744009d90 tx=0x7fb74402beb0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 -- 172.21.15.67:0/2858059142 shutdown_connections 2024-04-17T07:14:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0b7b70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 --2- 172.21.15.67:0/2858059142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 -- 172.21.15.67:0/2858059142 >> 172.21.15.67:0/2858059142 conn(0x7fb74c0d15f0 msgr2=0x7fb74c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:51.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 -- 172.21.15.67:0/2858059142 shutdown_connections 2024-04-17T07:14:51.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.780+0000 7fb753000640 1 -- 172.21.15.67:0/2858059142 wait complete. 2024-04-17T07:14:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.782+0000 7fb753000640 1 Processor -- start 2024-04-17T07:14:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.782+0000 7fb753000640 1 -- start start 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c1f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb74c15ec60 0x7fb74c0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c15f040 con 0x7fb74c0cf910 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0cfdc0 con 0x7fb74c15ec60 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb753000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c0d00c0 con 0x7fb74c0b6d00 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb751200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb751200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41032/0 (socket says 172.21.15.67:41032) 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.783+0000 7fb751200640 1 -- 172.21.15.67:0/1238140850 learned_addr learned my addr 172.21.15.67:0/1238140850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751c00640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c1f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 -- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 msgr2=0x7fb74c1f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c1f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 -- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb74c15ec60 msgr2=0x7fb74c0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb74c15ec60 0x7fb74c0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 -- 172.21.15.67:0/1238140850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c0adbe0 con 0x7fb74c0cf910 2024-04-17T07:14:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751c00640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb751200640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00e7e0 tx=0x7fb73c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00f040 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb73c004030 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004400 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.784+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74c0adec0 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.785+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74c0ab520 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.786+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb74c068a10 con 0x7fb74c0cf910 2024-04-17T07:14:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.787+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb73c00c6d0 con 0x7fb74c0cf910 2024-04-17T07:14:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.788+0000 7fb749600640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 0x7fb73006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.788+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fb73c08b9a0 con 0x7fb74c0cf910 2024-04-17T07:14:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.789+0000 7fb751c00640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 0x7fb73006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.790+0000 7fb751c00640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 0x7fb73006ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb740005880 tx=0x7fb740009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:51.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:51.792+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb73c017020 con 0x7fb74c0cf910 2024-04-17T07:14:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.000+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb74c0b9790 con 0x7fb74c0cf910 2024-04-17T07:14:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.001+0000 7fb749600640 1 -- 172.21.15.67:0/1238140850 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7fb73c059d40 con 0x7fb74c0cf910 2024-04-17T07:14:52.002 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:52.002 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.004+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 msgr2=0x7fb73006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.004+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 0x7fb73006ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb740005880 tx=0x7fb740009040 comp rx=0 tx=0).stop 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.004+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 msgr2=0x7fb74c0ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.004+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00e7e0 tx=0x7fb73c00eca0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 shutdown_connections 2024-04-17T07:14:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb74c15ec60 0x7fb74c0b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb74c0cf910 0x7fb74c0ad290 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fb73006c920 0x7fb73006ede0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 --2- 172.21.15.67:0/1238140850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb74c0b6d00 0x7fb74c1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 >> 172.21.15.67:0/1238140850 conn(0x7fb74c0d15f0 msgr2=0x7fb74c15e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 shutdown_connections 2024-04-17T07:14:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.005+0000 7fb753000640 1 -- 172.21.15.67:0/1238140850 wait complete. 2024-04-17T07:14:52.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:52.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.248+0000 7f54a8a00640 1 Processor -- start 2024-04-17T07:14:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.248+0000 7f54a8a00640 1 -- start start 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a415ec20 0x7f54a40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a40b3b50 0x7f54a406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54a406cf80 0x7f54a40cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f54a40d4f70 con 0x7f54a415ec20 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f54a40d5250 con 0x7f54a406cf80 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54a40cf4d0 con 0x7f54a40b3b50 2024-04-17T07:14:52.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a415ec20 0x7f54a40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a415ec20 0x7f54a40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41042/0 (socket says 172.21.15.67:41042) 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.250+0000 7f54a3400640 1 -- 172.21.15.67:0/2356429666 learned_addr learned my addr 172.21.15.67:0/2356429666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f549be00640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a40b3b50 0x7f54a406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3e00640 1 --2- 172.21.15.67:0/2356429666 >> v2:172.21.15.187:3300/0 conn(0x7f54a406cf80 0x7f54a40cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3400640 1 -- 172.21.15.67:0/2356429666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a40b3b50 msgr2=0x7f54a406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3400640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a40b3b50 0x7f54a406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3400640 1 -- 172.21.15.67:0/2356429666 >> v2:172.21.15.187:3300/0 conn(0x7f54a406cf80 msgr2=0x7f54a40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3400640 1 --2- 172.21.15.67:0/2356429666 >> v2:172.21.15.187:3300/0 conn(0x7f54a406cf80 0x7f54a40cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.251+0000 7f54a3400640 1 -- 172.21.15.67:0/2356429666 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a40cf7b0 con 0x7f54a415ec20 2024-04-17T07:14:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.252+0000 7f54a3400640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a415ec20 0x7f54a40d22b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c460 tx=0x7f549402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119bed23e16c4c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.252+0000 7f54a2a00640 1 -- 172.21.15.67:0/2356429666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494038070 con 0x7f54a415ec20 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a2a00640 1 -- 172.21.15.67:0/2356429666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494031740 con 0x7f54a415ec20 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a2a00640 1 -- 172.21.15.67:0/2356429666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549402f050 con 0x7f54a415ec20 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 -- 172.21.15.67:0/2356429666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a415ec20 msgr2=0x7f54a40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a415ec20 0x7f54a40d22b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c460 tx=0x7f549402b7e0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 -- 172.21.15.67:0/2356429666 shutdown_connections 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 --2- 172.21.15.67:0/2356429666 >> v2:172.21.15.187:3300/0 conn(0x7f54a406cf80 0x7f54a40cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a40b3b50 0x7f54a406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 --2- 172.21.15.67:0/2356429666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a415ec20 0x7f54a40d22b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 -- 172.21.15.67:0/2356429666 >> 172.21.15.67:0/2356429666 conn(0x7f54a408bff0 msgr2=0x7f54a40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 -- 172.21.15.67:0/2356429666 shutdown_connections 2024-04-17T07:14:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.253+0000 7f54a8a00640 1 -- 172.21.15.67:0/2356429666 wait complete. 2024-04-17T07:14:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.255+0000 7f54a8a00640 1 Processor -- start 2024-04-17T07:14:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.256+0000 7f54a8a00640 1 -- start start 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.256+0000 7f54a8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.256+0000 7f54a8a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40b3b50 0x7f54a4161380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.256+0000 7f54a8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a415ec20 0x7f54a41618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.256+0000 7f54a8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54a40d0390 con 0x7f54a406cf80 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a8a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54a40d0210 con 0x7f54a40b3b50 2024-04-17T07:14:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54a40d0510 con 0x7f54a415ec20 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41044/0 (socket says 172.21.15.67:41044) 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 -- 172.21.15.67:0/1446941635 learned_addr learned my addr 172.21.15.67:0/1446941635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f549be00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40b3b50 0x7f54a4161380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3e00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a415ec20 0x7f54a41618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 -- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a415ec20 msgr2=0x7f54a41618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a415ec20 0x7f54a41618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 -- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40b3b50 msgr2=0x7f54a4161380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40b3b50 0x7f54a4161380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.257+0000 7f54a3400640 1 -- 172.21.15.67:0/1446941635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a40b7e20 con 0x7f54a406cf80 2024-04-17T07:14:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f54a3400640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f549402b820 tx=0x7f54940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494042070 con 0x7f54a406cf80 2024-04-17T07:14:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494031e80 con 0x7f54a406cf80 2024-04-17T07:14:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494038040 con 0x7f54a406cf80 2024-04-17T07:14:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a40b80c0 con 0x7f54a406cf80 2024-04-17T07:14:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.258+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a40b8680 con 0x7f54a406cf80 2024-04-17T07:14:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.259+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a4068a10 con 0x7f54a406cf80 2024-04-17T07:14:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.260+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5494045070 con 0x7f54a406cf80 2024-04-17T07:14:52.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.261+0000 7f549b400640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 0x7f548006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.261+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f54940ad8a0 con 0x7f54a406cf80 2024-04-17T07:14:52.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.262+0000 7f549be00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 0x7f548006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.263+0000 7f549be00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 0x7f548006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54a40ceca0 tx=0x7f548c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.266+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f549404c030 con 0x7f54a406cf80 2024-04-17T07:14:52.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.472+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54a4161160 con 0x7f54a406cf80 2024-04-17T07:14:52.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.473+0000 7f549b400640 1 -- 172.21.15.67:0/1446941635 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f5494045e40 con 0x7f54a406cf80 2024-04-17T07:14:52.473 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:52.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 msgr2=0x7f548006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 0x7f548006edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54a40ceca0 tx=0x7f548c008040 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 msgr2=0x7f54a4157760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f549402b820 tx=0x7f54940046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 shutdown_connections 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f548006c8f0 0x7f548006edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a415ec20 0x7f54a41618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40b3b50 0x7f54a4161380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 --2- 172.21.15.67:0/1446941635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406cf80 0x7f54a4157760 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.476+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 >> 172.21.15.67:0/1446941635 conn(0x7f54a408bff0 msgr2=0x7f54a40b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.477+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 shutdown_connections 2024-04-17T07:14:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.477+0000 7f54a8a00640 1 -- 172.21.15.67:0/1446941635 wait complete. 2024-04-17T07:14:52.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.719+0000 7f8900e00640 1 Processor -- start 2024-04-17T07:14:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.719+0000 7f8900e00640 1 -- start start 2024-04-17T07:14:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.720+0000 7f8900e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f88fc15eca0 0x7f88fc0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.720+0000 7f8900e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88fc0cfa10 0x7f88fc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.720+0000 7f8900e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88fc163860 con 0x7f88fc0d1090 2024-04-17T07:14:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.720+0000 7f8900e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f88fc0d03b0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.720+0000 7f8900e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f88fc0d1d40 con 0x7f88fc15eca0 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88fc0cfa10 0x7f88fc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88fc0cfa10 0x7f88fc0cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41056/0 (socket says 172.21.15.67:41056) 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 -- 172.21.15.67:0/773044173 learned_addr learned my addr 172.21.15.67:0/773044173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9e00640 1 --2- 172.21.15.67:0/773044173 >> v2:172.21.15.187:3300/0 conn(0x7f88fc15eca0 0x7f88fc0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 -- 172.21.15.67:0/773044173 >> v2:172.21.15.187:3300/0 conn(0x7f88fc15eca0 msgr2=0x7f88fc0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 --2- 172.21.15.67:0/773044173 >> v2:172.21.15.187:3300/0 conn(0x7f88fc15eca0 0x7f88fc0d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 -- 172.21.15.67:0/773044173 >> v1:172.21.15.187:6789/0 conn(0x7f88fc0d1090 legacy=0x7f88fc0d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.721+0000 7f88f9400640 1 -- 172.21.15.67:0/773044173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc0d2c30 con 0x7f88fc0cfa10 2024-04-17T07:14:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f88f9400640 1 --2- 172.21.15.67:0/773044173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0cfe70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0d26a0 tx=0x7f88e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42c78a9b69d59ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f88f8a00640 1 -- 172.21.15.67:0/773044173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4035020 con 0x7f88fc0cfa10 2024-04-17T07:14:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f88f8a00640 1 -- 172.21.15.67:0/773044173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e4031870 con 0x7f88fc0cfa10 2024-04-17T07:14:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f88f8a00640 1 -- 172.21.15.67:0/773044173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4031c40 con 0x7f88fc0cfa10 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f8900e00640 1 -- 172.21.15.67:0/773044173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 msgr2=0x7f88fc0cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.723+0000 7f8900e00640 1 --2- 172.21.15.67:0/773044173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0cfe70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0d26a0 tx=0x7f88e402b910 comp rx=0 tx=0).stop 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 -- 172.21.15.67:0/773044173 shutdown_connections 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 --2- 172.21.15.67:0/773044173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0cfe70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 --2- 172.21.15.67:0/773044173 >> v2:172.21.15.187:3300/0 conn(0x7f88fc15eca0 0x7f88fc0d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 -- 172.21.15.67:0/773044173 >> 172.21.15.67:0/773044173 conn(0x7f88fc067600 msgr2=0x7f88fc0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 -- 172.21.15.67:0/773044173 shutdown_connections 2024-04-17T07:14:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.724+0000 7f8900e00640 1 -- 172.21.15.67:0/773044173 wait complete. 2024-04-17T07:14:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.725+0000 7f8900e00640 1 Processor -- start 2024-04-17T07:14:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.726+0000 7f8900e00640 1 -- start start 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88fc0d1090 0x7f88fc15f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88fc15eca0 0x7f88fc161090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc0d34d0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc0d31f0 con 0x7f88fc15eca0 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f8900e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc15f080 con 0x7f88fc0d1090 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41070/0 (socket says 172.21.15.67:41070) 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f88f9e00640 1 -- 172.21.15.67:0/1787100460 learned_addr learned my addr 172.21.15.67:0/1787100460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.727+0000 7f88f9400640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88fc0d1090 0x7f88fc15f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 -- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88fc0d1090 msgr2=0x7f88fc15f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88fc0d1090 0x7f88fc15f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 -- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88fc15eca0 msgr2=0x7f88fc161090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88fc15eca0 0x7f88fc161090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 -- 172.21.15.67:0/1787100460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc0c75f0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f9e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c820 tx=0x7f88ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec015070 con 0x7f88fc0cfa10 2024-04-17T07:14:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88ec004240 con 0x7f88fc0cfa10 2024-04-17T07:14:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc0b4bf0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00ce10 con 0x7f88fc0cfa10 2024-04-17T07:14:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.728+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc0b51b0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.729+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88fc068ca0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.730+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f88ec007830 con 0x7f88fc0cfa10 2024-04-17T07:14:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.731+0000 7f88f2000640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 0x7f88d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.732+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f88ec08ba90 con 0x7f88fc0cfa10 2024-04-17T07:14:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.732+0000 7f88f9400640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 0x7f88d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:52.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.733+0000 7f88f9400640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 0x7f88d806ede0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f88e4036000 tx=0x7f88e403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.736+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f88fc0c75f0 con 0x7f88fc0cfa10 2024-04-17T07:14:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.937+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88fc0d2970 con 0x7f88fc0cfa10 2024-04-17T07:14:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.937+0000 7f88f2000640 1 -- 172.21.15.67:0/1787100460 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f88ec059e40 con 0x7f88fc0cfa10 2024-04-17T07:14:52.937 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 msgr2=0x7f88d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 0x7f88d806ede0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f88e4036000 tx=0x7f88e403c040 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 msgr2=0x7f88fc0c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c820 tx=0x7f88ec00cce0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 shutdown_connections 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88fc15eca0 0x7f88fc161090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f88d806c920 0x7f88d806ede0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88fc0d1090 0x7f88fc15f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 --2- 172.21.15.67:0/1787100460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88fc0cfa10 0x7f88fc0c70b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 >> 172.21.15.67:0/1787100460 conn(0x7f88fc067600 msgr2=0x7f88fc15d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.940+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 shutdown_connections 2024-04-17T07:14:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:52.941+0000 7f8900e00640 1 -- 172.21.15.67:0/1787100460 wait complete. 2024-04-17T07:14:52.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.178+0000 7f677f000640 1 Processor -- start 2024-04-17T07:14:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.179+0000 7f677f000640 1 -- start start 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 0x7f6778161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677f000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67780b4e40 con 0x7f67780d2ff0 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677f000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67780b5110 con 0x7f67781620e0 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6778162530 con 0x7f67780d33d0 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67780d2ff0 0x7f67780b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45450/0 (socket says 172.21.15.67:45450) 2024-04-17T07:14:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677dc00640 1 -- 172.21.15.67:0/2218942617 learned_addr learned my addr 172.21.15.67:0/2218942617 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f677d200640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 0x7f6778161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.181+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468524188 0 0) 0x7f67780b4e40 con 0x7f67780d2ff0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6760003520 con 0x7f67780d2ff0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617824469 0 0) 0x7f6760003520 con 0x7f67780d2ff0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 msgr2=0x7f6778161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 0x7f6778161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 >> v1:172.21.15.187:6789/0 conn(0x7f67781620e0 legacy=0x7f67780cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67780b53f0 con 0x7f67780d2ff0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f676c001670 con 0x7f67780d2ff0 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 >> v1:172.21.15.67:6789/0 conn(0x7f67780d2ff0 legacy=0x7f67780b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67600039e0 0x7f6760003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f6760004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 0x7f67600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6778162530 con 0x7f6760004800 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f676000ba70 con 0x7f67600039e0 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f676000bc70 con 0x7f67780d33d0 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.182+0000 7f677d200640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f6760004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677dc00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67600039e0 0x7f6760003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677d200640 1 -- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 msgr2=0x7f67600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677d200640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780d33d0 0x7f67600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677d200640 1 -- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67600039e0 msgr2=0x7f6760003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677d200640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67600039e0 0x7f6760003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677d200640 1 -- 172.21.15.67:0/2218942617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6760003520 con 0x7f6760004800 2024-04-17T07:14:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.183+0000 7f677dc00640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67600039e0 0x7f6760003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:53.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.184+0000 7f677d200640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f6760004c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6764009a90 tx=0x7f6760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4dadc54e615ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.184+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764039070 con 0x7f6760004800 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.184+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f676402c6b0 con 0x7f6760004800 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.184+0000 7f6777e00640 1 -- 172.21.15.67:0/2218942617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764009690 con 0x7f6760004800 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 -- 172.21.15.67:0/2218942617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 msgr2=0x7f6760004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 --2- 172.21.15.67:0/2218942617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f6760004c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6764009a90 tx=0x7f6760005830 comp rx=0 tx=0).stop 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677e600640 1 -- 172.21.15.67:0/2218942617 reap_dead start 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 -- 172.21.15.67:0/2218942617 shutdown_connections 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 -- 172.21.15.67:0/2218942617 >> 172.21.15.67:0/2218942617 conn(0x7f677809da80 msgr2=0x7f677808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 -- 172.21.15.67:0/2218942617 shutdown_connections 2024-04-17T07:14:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.185+0000 7f677f000640 1 -- 172.21.15.67:0/2218942617 wait complete. 2024-04-17T07:14:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.187+0000 7f677f000640 1 Processor -- start 2024-04-17T07:14:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.187+0000 7f677f000640 1 -- start start 2024-04-17T07:14:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 0x7f67780c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780c87d0 0x7f67780d1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67780cf4c0 con 0x7f6760004800 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f67780cf340 con 0x7f67780c9df0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67780d26d0 con 0x7f67780c87d0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41086/0 (socket says 172.21.15.67:41086) 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.188+0000 7f677dc00640 1 -- 172.21.15.67:0/3305760243 learned_addr learned my addr 172.21.15.67:0/3305760243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677e600640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780c87d0 0x7f67780d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677d200640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 0x7f67780c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 -- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780c87d0 msgr2=0x7f67780d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780c87d0 0x7f67780d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 -- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 msgr2=0x7f67780c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 0x7f67780c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 -- 172.21.15.67:0/3305760243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67780c8d50 con 0x7f6760004800 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677d200640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 0x7f67780c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677dc00640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f676c00d910 tx=0x7f676c00ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c019070 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f676c0074e0 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c0049e0 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67780c9030 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.189+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67780c95f0 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.191+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67780755d0 con 0x7f6760004800 2024-04-17T07:14:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.192+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f676c004b80 con 0x7f6760004800 2024-04-17T07:14:53.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.193+0000 7f6775600640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 0x7f675806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.193+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f676c08b740 con 0x7f6760004800 2024-04-17T07:14:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.193+0000 7f677d200640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 0x7f675806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.195+0000 7f677d200640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 0x7f675806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6764036000 tx=0x7f6764030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.198+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f676c0599c0 con 0x7f6760004800 2024-04-17T07:14:53.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.401+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6778153c80 con 0x7f6760004800 2024-04-17T07:14:53.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.401+0000 7f6775600640 1 -- 172.21.15.67:0/3305760243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f676c0592c0 con 0x7f6760004800 2024-04-17T07:14:53.401 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 msgr2=0x7f675806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 0x7f675806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6764036000 tx=0x7f6764030000 comp rx=0 tx=0).stop 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 msgr2=0x7f67780cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f676c00d910 tx=0x7f676c00ddd0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 shutdown_connections 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f675806c8f0 0x7f675806edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67780c87d0 0x7f67780d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67780c9df0 0x7f67780c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 --2- 172.21.15.67:0/3305760243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6760004800 0x7f67780cdc10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 >> 172.21.15.67:0/3305760243 conn(0x7f677809da80 msgr2=0x7f67780b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 shutdown_connections 2024-04-17T07:14:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.404+0000 7f677f000640 1 -- 172.21.15.67:0/3305760243 wait complete. 2024-04-17T07:14:53.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:14:53.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.638+0000 7f4993200640 1 Processor -- start 2024-04-17T07:14:53.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.639+0000 7f4993200640 1 -- start start 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.640+0000 7f4993200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49940b63c0 0x7f49940cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.640+0000 7f4993200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499415d060 con 0x7f49940cbc00 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4993200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499415d340 con 0x7f4994072570 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4993200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f49940cfe90 con 0x7f49940b63c0 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4991e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49940cbc00 0x7f4994071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45308/0 (socket says 172.21.15.67:45308) 2024-04-17T07:14:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4991e00640 1 -- 172.21.15.67:0/576399011 learned_addr learned my addr 172.21.15.67:0/576399011 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429532032 0 0) 0x7f499415d060 con 0x7f49940cbc00 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4992800640 1 --2- 172.21.15.67:0/576399011 >> v2:172.21.15.187:3300/0 conn(0x7f49940b63c0 0x7f49940cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.641+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4978003520 con 0x7f49940cbc00 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4196615531 0 0) 0x7f4978003520 con 0x7f49940cbc00 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 >> v2:172.21.15.187:3300/0 conn(0x7f49940b63c0 msgr2=0x7f49940cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 --2- 172.21.15.67:0/576399011 >> v2:172.21.15.187:3300/0 conn(0x7f49940b63c0 0x7f49940cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 >> v1:172.21.15.187:6789/0 conn(0x7f4994072570 legacy=0x7f49940ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499415d620 con 0x7f49940cbc00 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4984001670 con 0x7f49940cbc00 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 >> v1:172.21.15.67:6789/0 conn(0x7f49940cbc00 legacy=0x7f4994071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f4978003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4978004800 0x7f4978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4978005210 0x7f497800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49940cfe90 con 0x7f4978003a70 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f497800c140 con 0x7f4978005210 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f497800c340 con 0x7f4978004800 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.642+0000 7f4991e00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f4978003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4992800640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4978005210 0x7f497800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f498be00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4978004800 0x7f4978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4991e00640 1 -- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4978004800 msgr2=0x7f4978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4991e00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4978004800 0x7f4978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4991e00640 1 -- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4978005210 msgr2=0x7f497800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4991e00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4978005210 0x7f497800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4991e00640 1 -- 172.21.15.67:0/576399011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4978003520 con 0x7f4978003a70 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f4992800640 1 -- 172.21.15.67:0/576399011 reap_dead start 2024-04-17T07:14:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.643+0000 7f498be00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4978004800 0x7f4978004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.644+0000 7f4991e00640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f4978003e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f498400c0e0 tx=0x7f4978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37171acabf98abd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.644+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984007c20 con 0x7f4978003a70 2024-04-17T07:14:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.644+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4984030070 con 0x7f4978003a70 2024-04-17T07:14:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.644+0000 7f4991400640 1 -- 172.21.15.67:0/576399011 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984005d70 con 0x7f4978003a70 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 -- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 msgr2=0x7f4978003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f4978003e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f498400c0e0 tx=0x7f4978005830 comp rx=0 tx=0).stop 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 -- 172.21.15.67:0/576399011 shutdown_connections 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 --2- 172.21.15.67:0/576399011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f4978003e90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 -- 172.21.15.67:0/576399011 >> 172.21.15.67:0/576399011 conn(0x7f499406bef0 msgr2=0x7f49940b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 -- 172.21.15.67:0/576399011 shutdown_connections 2024-04-17T07:14:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.645+0000 7f4993200640 1 -- 172.21.15.67:0/576399011 wait complete. 2024-04-17T07:14:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.647+0000 7f4993200640 1 Processor -- start 2024-04-17T07:14:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.647+0000 7f4993200640 1 -- start start 2024-04-17T07:14:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49941f5940 0x7f49940c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49941f5dc0 0x7f49940c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49940d02d0 con 0x7f4978003a70 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f499415dc70 con 0x7f49941f5940 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.648+0000 7f4993200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49940cbfe0 con 0x7f49941f5dc0 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51574/0 (socket says 172.21.15.67:51574) 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 -- 172.21.15.67:0/2379186313 learned_addr learned my addr 172.21.15.67:0/2379186313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f498be00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49941f5940 0x7f49940c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4992800640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49941f5dc0 0x7f49940c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 -- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49941f5dc0 msgr2=0x7f49940c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49941f5dc0 0x7f49940c9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 -- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49941f5940 msgr2=0x7f49940c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49941f5940 0x7f49940c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.649+0000 7f4991e00640 1 -- 172.21.15.67:0/2379186313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499406e820 con 0x7f4978003a70 2024-04-17T07:14:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f4991e00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f498400c0e0 tx=0x7f499415d510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984001b60 con 0x7f4978003a70 2024-04-17T07:14:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4984000ed0 con 0x7f4978003a70 2024-04-17T07:14:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498402cdd0 con 0x7f4978003a70 2024-04-17T07:14:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499406eb00 con 0x7f4978003a70 2024-04-17T07:14:53.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.650+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499406f0c0 con 0x7f4978003a70 2024-04-17T07:14:53.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.651+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4994068a10 con 0x7f4978003a70 2024-04-17T07:14:53.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.652+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4984007070 con 0x7f4978003a70 2024-04-17T07:14:53.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.653+0000 7f498a000640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 0x7f497006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:53.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.653+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f49840ad080 con 0x7f4978003a70 2024-04-17T07:14:53.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.654+0000 7f498be00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 0x7f497006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:53.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.655+0000 7f498be00640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 0x7f497006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f49941f2370 tx=0x7f497c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.658+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f498407b420 con 0x7f4978003a70 2024-04-17T07:14:53.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:14:53.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:14:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.878+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f49940b3020 con 0x7f4978003a70 2024-04-17T07:14:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.879+0000 7f498a000640 1 -- 172.21.15.67:0/2379186313 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f498407ad20 con 0x7f4978003a70 2024-04-17T07:14:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T07:14:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 msgr2=0x7f497006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 0x7f497006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f49941f2370 tx=0x7f497c009040 comp rx=0 tx=0).stop 2024-04-17T07:14:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 msgr2=0x7f49941590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f498400c0e0 tx=0x7f499415d510 comp rx=0 tx=0).stop 2024-04-17T07:14:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 shutdown_connections 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f497006c920 0x7f497006ede0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49941f5dc0 0x7f49940c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49941f5940 0x7f49940c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 --2- 172.21.15.67:0/2379186313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978003a70 0x7f49941590f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 >> 172.21.15.67:0/2379186313 conn(0x7f499406bef0 msgr2=0x7f49940ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 shutdown_connections 2024-04-17T07:14:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:53.882+0000 7f4993200640 1 -- 172.21.15.67:0/2379186313 wait complete. 2024-04-17T07:14:53.902 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-17T07:14:53.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:53.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-17T07:14:54.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T07:14:54.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:14:54.007 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-17T07:14:54.057 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:14:54.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:54.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.296+0000 7f8149200640 1 Processor -- start 2024-04-17T07:14:54.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.296+0000 7f8149200640 1 -- start start 2024-04-17T07:14:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.297+0000 7f8149200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f814415ec60 0x7f81440b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.297+0000 7f8149200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81440b6b30 0x7f81440cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.297+0000 7f8149200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f81440cf910 0x7f814406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.297+0000 7f8149200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f81440d49c0 con 0x7f81440b6b30 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8149200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f81440d3660 con 0x7f81440cf910 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8149200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81440b6f70 con 0x7f814415ec60 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8142a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81440b6b30 0x7f81440cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8142a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81440b6b30 0x7f81440cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51576/0 (socket says 172.21.15.67:51576) 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8142a00640 1 -- 172.21.15.67:0/3412147895 learned_addr learned my addr 172.21.15.67:0/3412147895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8143400640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f814415ec60 0x7f81440b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8143e00640 1 --2- 172.21.15.67:0/3412147895 >> v2:172.21.15.187:3300/0 conn(0x7f81440cf910 0x7f814406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.298+0000 7f8142a00640 1 -- 172.21.15.67:0/3412147895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f814415ec60 msgr2=0x7f81440b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.299+0000 7f8142a00640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f814415ec60 0x7f81440b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.299+0000 7f8142a00640 1 -- 172.21.15.67:0/3412147895 >> v2:172.21.15.187:3300/0 conn(0x7f81440cf910 msgr2=0x7f814406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.299+0000 7f8142a00640 1 --2- 172.21.15.67:0/3412147895 >> v2:172.21.15.187:3300/0 conn(0x7f81440cf910 0x7f814406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.299+0000 7f8142a00640 1 -- 172.21.15.67:0/3412147895 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81440d2d30 con 0x7f81440b6b30 2024-04-17T07:14:54.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.300+0000 7f8142a00640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81440b6b30 0x7f81440cf3d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f81440c8670 tx=0x7f812c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752f1557f789f3e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.300+0000 7f8142000640 1 -- 172.21.15.67:0/3412147895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c035020 con 0x7f81440b6b30 2024-04-17T07:14:54.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.300+0000 7f8142000640 1 -- 172.21.15.67:0/3412147895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812c031870 con 0x7f81440b6b30 2024-04-17T07:14:54.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.300+0000 7f8142000640 1 -- 172.21.15.67:0/3412147895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c031c40 con 0x7f81440b6b30 2024-04-17T07:14:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 -- 172.21.15.67:0/3412147895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81440b6b30 msgr2=0x7f81440cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81440b6b30 0x7f81440cf3d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f81440c8670 tx=0x7f812c02b910 comp rx=0 tx=0).stop 2024-04-17T07:14:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 -- 172.21.15.67:0/3412147895 shutdown_connections 2024-04-17T07:14:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 --2- 172.21.15.67:0/3412147895 >> v2:172.21.15.187:3300/0 conn(0x7f81440cf910 0x7f814406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81440b6b30 0x7f81440cf3d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 --2- 172.21.15.67:0/3412147895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f814415ec60 0x7f81440b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 -- 172.21.15.67:0/3412147895 >> 172.21.15.67:0/3412147895 conn(0x7f81440d15f0 msgr2=0x7f81440d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 -- 172.21.15.67:0/3412147895 shutdown_connections 2024-04-17T07:14:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.301+0000 7f8149200640 1 -- 172.21.15.67:0/3412147895 wait complete. 2024-04-17T07:14:54.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.303+0000 7f8149200640 1 Processor -- start 2024-04-17T07:14:54.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.303+0000 7f8149200640 1 -- start start 2024-04-17T07:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81440b6b30 0x7f81440cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81440cf910 0x7f81441594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81440cfda0 con 0x7f814415ec60 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81440d32f0 con 0x7f81440b6b30 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.304+0000 7f8149200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f814415f040 con 0x7f81440cf910 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51580/0 (socket says 172.21.15.67:51580) 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 -- 172.21.15.67:0/2440878575 learned_addr learned my addr 172.21.15.67:0/2440878575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143400640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81440b6b30 0x7f81440cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8142a00640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81440cf910 0x7f81441594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 -- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81440cf910 msgr2=0x7f81441594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81440cf910 0x7f81441594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 -- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81440b6b30 msgr2=0x7f81440cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81440b6b30 0x7f81440cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.305+0000 7f8143e00640 1 -- 172.21.15.67:0/2440878575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81440ccc10 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f8143e00640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f813c00d800 tx=0x7f813c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c018020 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f813c0096c0 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81440ccef0 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c009a90 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.306+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81440cd7e0 con 0x7f814415ec60 2024-04-17T07:14:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.307+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8144068a10 con 0x7f814415ec60 2024-04-17T07:14:54.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.308+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f813c01b020 con 0x7f814415ec60 2024-04-17T07:14:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.309+0000 7f813b400640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 0x7f812006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.309+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f813c08be30 con 0x7f814415ec60 2024-04-17T07:14:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.310+0000 7f8143400640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 0x7f812006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.311+0000 7f8143400640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 0x7f812006edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f81340099c0 tx=0x7f8134003990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.313+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f813c022030 con 0x7f814415ec60 2024-04-17T07:14:54.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.561+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81440b4790 con 0x7f814415ec60 2024-04-17T07:14:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.562+0000 7f813b400640 1 -- 172.21.15.67:0/2440878575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7f813c0102e0 con 0x7f814415ec60 2024-04-17T07:14:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:54.563 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 msgr2=0x7f812006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 0x7f812006edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f81340099c0 tx=0x7f8134003990 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 msgr2=0x7f81440ac2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f813c00d800 tx=0x7f813c00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 shutdown_connections 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f814415ec60 0x7f81440ac2b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f812006c8f0 0x7f812006edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81440cf910 0x7f81441594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 --2- 172.21.15.67:0/2440878575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81440b6b30 0x7f81440cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.565+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 >> 172.21.15.67:0/2440878575 conn(0x7f81440d15f0 msgr2=0x7f814415f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.566+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 shutdown_connections 2024-04-17T07:14:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.566+0000 7f8149200640 1 -- 172.21.15.67:0/2440878575 wait complete. 2024-04-17T07:14:54.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:54.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.805+0000 7fcb16800640 1 Processor -- start 2024-04-17T07:14:54.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.805+0000 7fcb16800640 1 -- start start 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1006b8c0 0x7fcb100cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb10160280 0x7fcb1006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb10141ed0 0x7fcb10142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1015ce30 con 0x7fcb1006b8c0 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcb100cd4a0 con 0x7fcb10160280 2024-04-17T07:14:54.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.807+0000 7fcb16800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb101607f0 con 0x7fcb10141ed0 2024-04-17T07:14:54.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1006b8c0 0x7fcb100cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb1006b8c0 0x7fcb100cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51588/0 (socket says 172.21.15.67:51588) 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 -- 172.21.15.67:0/3087709374 learned_addr learned my addr 172.21.15.67:0/3087709374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3087709374 >> v2:172.21.15.187:3300/0 conn(0x7fcb10160280 0x7fcb1006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15e00640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb10141ed0 0x7fcb10142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 -- 172.21.15.67:0/3087709374 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb10141ed0 msgr2=0x7fcb10142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb10141ed0 0x7fcb10142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 -- 172.21.15.67:0/3087709374 >> v2:172.21.15.187:3300/0 conn(0x7fcb10160280 msgr2=0x7fcb1006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 --2- 172.21.15.67:0/3087709374 >> v2:172.21.15.187:3300/0 conn(0x7fcb10160280 0x7fcb1006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.808+0000 7fcb15400640 1 -- 172.21.15.67:0/3087709374 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb10160ad0 con 0x7fcb1006b8c0 2024-04-17T07:14:54.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.810+0000 7fcb15400640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1006b8c0 0x7fcb100cd8b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcb100aff50 tx=0x7fcb0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5f935e320b0715 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.810+0000 7fcb0fe00640 1 -- 172.21.15.67:0/3087709374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0002ca70 con 0x7fcb1006b8c0 2024-04-17T07:14:54.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.810+0000 7fcb0fe00640 1 -- 172.21.15.67:0/3087709374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0002cc10 con 0x7fcb1006b8c0 2024-04-17T07:14:54.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.810+0000 7fcb0fe00640 1 -- 172.21.15.67:0/3087709374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000343f0 con 0x7fcb1006b8c0 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 -- 172.21.15.67:0/3087709374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1006b8c0 msgr2=0x7fcb100cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1006b8c0 0x7fcb100cd8b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcb100aff50 tx=0x7fcb0002bed0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 -- 172.21.15.67:0/3087709374 shutdown_connections 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb10141ed0 0x7fcb10142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 --2- 172.21.15.67:0/3087709374 >> v2:172.21.15.187:3300/0 conn(0x7fcb10160280 0x7fcb1006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 --2- 172.21.15.67:0/3087709374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1006b8c0 0x7fcb100cd8b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 -- 172.21.15.67:0/3087709374 >> 172.21.15.67:0/3087709374 conn(0x7fcb10067d10 msgr2=0x7fcb100b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 -- 172.21.15.67:0/3087709374 shutdown_connections 2024-04-17T07:14:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.811+0000 7fcb16800640 1 -- 172.21.15.67:0/3087709374 wait complete. 2024-04-17T07:14:54.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.813+0000 7fcb16800640 1 Processor -- start 2024-04-17T07:14:54.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.813+0000 7fcb16800640 1 -- start start 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb1006b8c0 0x7fcb100b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 0x7fcb1013f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1006c1e0 con 0x7fcb10160280 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcb100cddf0 con 0x7fcb10141ed0 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb16800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb10161090 con 0x7fcb1006b8c0 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51592/0 (socket says 172.21.15.67:51592) 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.814+0000 7fcb15e00640 1 -- 172.21.15.67:0/3977703261 learned_addr learned my addr 172.21.15.67:0/3977703261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15400640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb1006b8c0 0x7fcb100b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 0x7fcb1013f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 -- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb1006b8c0 msgr2=0x7fcb100b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb1006b8c0 0x7fcb100b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 -- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 msgr2=0x7fcb1013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 0x7fcb1013f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 -- 172.21.15.67:0/3977703261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb100b4170 con 0x7fcb10160280 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 0x7fcb1013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb15e00640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800d620 tx=0x7fcb0800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.815+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08009600 con 0x7fcb10160280 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.816+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb08019070 con 0x7fcb10160280 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.816+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb101f7cb0 con 0x7fcb10160280 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.816+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08009a00 con 0x7fcb10160280 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.816+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb101f8270 con 0x7fcb10160280 2024-04-17T07:14:54.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.817+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb10068a10 con 0x7fcb10160280 2024-04-17T07:14:54.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.818+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb08009ba0 con 0x7fcb10160280 2024-04-17T07:14:54.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.819+0000 7fcb0d600640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:54.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.819+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fcb0808c080 con 0x7fcb10160280 2024-04-17T07:14:54.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.819+0000 7fcb15400640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:54.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.821+0000 7fcb15400640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 0x7fcaf006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb00035000 tx=0x7fcb00030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:54.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:54.823+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcb0805a390 con 0x7fcb10160280 2024-04-17T07:14:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.030+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb10146220 con 0x7fcb10160280 2024-04-17T07:14:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.033+0000 7fcb0d600640 1 -- 172.21.15.67:0/3977703261 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7fcb08059c90 con 0x7fcb10160280 2024-04-17T07:14:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:55.033 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.035+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 msgr2=0x7fcaf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.035+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 0x7fcaf006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb00035000 tx=0x7fcb00030000 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 msgr2=0x7fcb100b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800d620 tx=0x7fcb0800dae0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 shutdown_connections 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb10160280 0x7fcb100b3c30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb10141ed0 0x7fcb1013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fcaf006c8f0 0x7fcaf006edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 --2- 172.21.15.67:0/3977703261 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb1006b8c0 0x7fcb100b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 >> 172.21.15.67:0/3977703261 conn(0x7fcb10067d10 msgr2=0x7fcb100cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 shutdown_connections 2024-04-17T07:14:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.036+0000 7fcb16800640 1 -- 172.21.15.67:0/3977703261 wait complete. 2024-04-17T07:14:55.056 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:13:55.714444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:13:56.666401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:14:55.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:14:55.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.284+0000 7f45e3a00640 1 Processor -- start 2024-04-17T07:14:55.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.284+0000 7f45e3a00640 1 -- start start 2024-04-17T07:14:55.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.285+0000 7f45e3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45dc162250 0x7f45dc073a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.285+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc06bfd0 0x7f45dc0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.285+0000 7f45e3a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45dc072f70 con 0x7f45dc0b0860 2024-04-17T07:14:55.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.285+0000 7f45e3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f45dc073f70 con 0x7f45dc162250 2024-04-17T07:14:55.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.285+0000 7f45e3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45dc0735a0 con 0x7f45dc06bfd0 2024-04-17T07:14:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45dc162250 0x7f45dc073a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45dc162250 0x7f45dc073a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51598/0 (socket says 172.21.15.67:51598) 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 -- 172.21.15.67:0/3500445361 learned_addr learned my addr 172.21.15.67:0/3500445361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1a00640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc06bfd0 0x7f45dc0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 -- 172.21.15.67:0/3500445361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc06bfd0 msgr2=0x7f45dc0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc06bfd0 0x7f45dc0b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 -- 172.21.15.67:0/3500445361 >> v1:172.21.15.187:6789/0 conn(0x7f45dc0b0860 legacy=0x7f45dc06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.286+0000 7f45e1000640 1 -- 172.21.15.67:0/3500445361 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc162630 con 0x7f45dc162250 2024-04-17T07:14:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.288+0000 7f45e1000640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc162250 0x7f45dc073a30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009e50 tx=0x7f45cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c26d748e366f168 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.288+0000 7f45db400640 1 -- 172.21.15.67:0/3500445361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc02c9e0 con 0x7f45dc162250 2024-04-17T07:14:55.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.288+0000 7f45db400640 1 -- 172.21.15.67:0/3500445361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc02cb80 con 0x7f45dc162250 2024-04-17T07:14:55.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.288+0000 7f45db400640 1 -- 172.21.15.67:0/3500445361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc0343f0 con 0x7f45dc162250 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 -- 172.21.15.67:0/3500445361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc162250 msgr2=0x7f45dc073a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc162250 0x7f45dc073a30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009e50 tx=0x7f45cc02be40 comp rx=0 tx=0).stop 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 -- 172.21.15.67:0/3500445361 shutdown_connections 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc06bfd0 0x7f45dc0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 --2- 172.21.15.67:0/3500445361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc162250 0x7f45dc073a30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 -- 172.21.15.67:0/3500445361 >> 172.21.15.67:0/3500445361 conn(0x7f45dc0b3390 msgr2=0x7f45dc0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:55.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 -- 172.21.15.67:0/3500445361 shutdown_connections 2024-04-17T07:14:55.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.289+0000 7f45e3a00640 1 -- 172.21.15.67:0/3500445361 wait complete. 2024-04-17T07:14:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.291+0000 7f45e3a00640 1 Processor -- start 2024-04-17T07:14:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.291+0000 7f45e3a00640 1 -- start start 2024-04-17T07:14:55.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc0b0860 0x7f45dc14bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45dc162250 0x7f45dc1f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc073230 con 0x7f45dc06bfd0 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc071850 con 0x7f45dc162250 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45dc0b0f40 con 0x7f45dc0b0860 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51612/0 (socket says 172.21.15.67:51612) 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.292+0000 7f45e1000640 1 -- 172.21.15.67:0/1197648269 learned_addr learned my addr 172.21.15.67:0/1197648269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 -- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc0b0860 msgr2=0x7f45dc14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc0b0860 0x7f45dc14bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 -- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45dc162250 msgr2=0x7f45dc1f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45dc162250 0x7f45dc1f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 -- 172.21.15.67:0/1197648269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc155470 con 0x7f45dc06bfd0 2024-04-17T07:14:55.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e1000640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45cc036040 tx=0x7f45cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc043020 con 0x7f45dc06bfd0 2024-04-17T07:14:55.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc03e450 con 0x7f45dc06bfd0 2024-04-17T07:14:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc009510 con 0x7f45dc06bfd0 2024-04-17T07:14:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc0cae80 con 0x7f45dc06bfd0 2024-04-17T07:14:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.293+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc159c40 con 0x7f45dc06bfd0 2024-04-17T07:14:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.294+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc068b90 con 0x7f45dc06bfd0 2024-04-17T07:14:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.295+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45cc005d10 con 0x7f45dc06bfd0 2024-04-17T07:14:55.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.296+0000 7f45d8c00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 0x7f45bc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.296+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f45cc0ae780 con 0x7f45dc06bfd0 2024-04-17T07:14:55.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.297+0000 7f45dbe00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 0x7f45bc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.298+0000 7f45dbe00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 0x7f45bc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f45c8000f60 tx=0x7f45c80079d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.301+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f45cc07cb10 con 0x7f45dc06bfd0 2024-04-17T07:14:55.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.504+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f45dc0b3050 con 0x7f45dc06bfd0 2024-04-17T07:14:55.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.505+0000 7f45d8c00640 1 -- 172.21.15.67:0/1197648269 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f45cc07c410 con 0x7f45dc06bfd0 2024-04-17T07:14:55.505 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.507+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 msgr2=0x7f45bc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.507+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 0x7f45bc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f45c8000f60 tx=0x7f45c80079d0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.507+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 msgr2=0x7f45dc15bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.507+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f45cc036040 tx=0x7f45cc004a40 comp rx=0 tx=0).stop 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 shutdown_connections 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45dc162250 0x7f45dc1f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f45bc06c920 0x7f45bc06ede0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45dc0b0860 0x7f45dc14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 --2- 172.21.15.67:0/1197648269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45dc06bfd0 0x7f45dc15bec0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 >> 172.21.15.67:0/1197648269 conn(0x7f45dc0b3390 msgr2=0x7f45dc160aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 shutdown_connections 2024-04-17T07:14:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.508+0000 7f45e3a00640 1 -- 172.21.15.67:0/1197648269 wait complete. 2024-04-17T07:14:55.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:14:55.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.742+0000 7f1007e00640 1 Processor -- start 2024-04-17T07:14:55.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.742+0000 7f1007e00640 1 -- start start 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f100815ec40 0x7f10080d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 0x7f100815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10080d2ce0 0x7f10080d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10080d51d0 con 0x7f10080d2ce0 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10080d54b0 con 0x7f10080b3660 2024-04-17T07:14:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.743+0000 7f1007e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f100815f3a0 con 0x7f100815ec40 2024-04-17T07:14:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10080d2ce0 0x7f10080d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10080d2ce0 0x7f10080d3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51626/0 (socket says 172.21.15.67:51626) 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 -- 172.21.15.67:0/1745424976 learned_addr learned my addr 172.21.15.67:0/1745424976 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1006000640 1 --2- 172.21.15.67:0/1745424976 >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 0x7f100815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1006a00640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f100815ec40 0x7f10080d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 -- 172.21.15.67:0/1745424976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f100815ec40 msgr2=0x7f10080d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f100815ec40 0x7f10080d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 -- 172.21.15.67:0/1745424976 >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 msgr2=0x7f100815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 --2- 172.21.15.67:0/1745424976 >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 0x7f100815fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1007400640 1 -- 172.21.15.67:0/1745424976 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100815f680 con 0x7f10080d2ce0 2024-04-17T07:14:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.744+0000 7f1006000640 1 --2- 172.21.15.67:0/1745424976 >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 0x7f100815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.745+0000 7f1007400640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10080d2ce0 0x7f10080d3170 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1008073f90 tx=0x7f100002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8323cd5740da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.746+0000 7f1005600640 1 -- 172.21.15.67:0/1745424976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000035020 con 0x7f10080d2ce0 2024-04-17T07:14:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.746+0000 7f1005600640 1 -- 172.21.15.67:0/1745424976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1000031870 con 0x7f10080d2ce0 2024-04-17T07:14:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.746+0000 7f1005600640 1 -- 172.21.15.67:0/1745424976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000031c40 con 0x7f10080d2ce0 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.746+0000 7f1007e00640 1 -- 172.21.15.67:0/1745424976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10080d2ce0 msgr2=0x7f10080d3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.746+0000 7f1007e00640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10080d2ce0 0x7f10080d3170 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1008073f90 tx=0x7f100002b910 comp rx=0 tx=0).stop 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 -- 172.21.15.67:0/1745424976 shutdown_connections 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10080d2ce0 0x7f10080d3170 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 --2- 172.21.15.67:0/1745424976 >> v2:172.21.15.187:3300/0 conn(0x7f10080b3660 0x7f100815fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 --2- 172.21.15.67:0/1745424976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f100815ec40 0x7f10080d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 -- 172.21.15.67:0/1745424976 >> 172.21.15.67:0/1745424976 conn(0x7f100808bff0 msgr2=0x7f10080d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 -- 172.21.15.67:0/1745424976 shutdown_connections 2024-04-17T07:14:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.747+0000 7f1007e00640 1 -- 172.21.15.67:0/1745424976 wait complete. 2024-04-17T07:14:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.748+0000 7f1007e00640 1 Processor -- start 2024-04-17T07:14:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.749+0000 7f1007e00640 1 -- start start 2024-04-17T07:14:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10080b3660 0x7f10080c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 0x7f100814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10080cf400 con 0x7f100815ec40 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10080cf280 con 0x7f10080d2ce0 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10080cf580 con 0x7f10080b3660 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51640/0 (socket says 172.21.15.67:51640) 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 -- 172.21.15.67:0/3679245413 learned_addr learned my addr 172.21.15.67:0/3679245413 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1006a00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10080b3660 0x7f10080c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1006000640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 0x7f100814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 -- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10080b3660 msgr2=0x7f10080c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10080b3660 0x7f10080c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 -- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 msgr2=0x7f100814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 0x7f100814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1007400640 1 -- 172.21.15.67:0/3679245413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10080cb580 con 0x7f100815ec40 2024-04-17T07:14:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.750+0000 7f1006000640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 0x7f100814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f1007400640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1000009960 tx=0x7f10000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100003ace0 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10000443f0 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10080b8b30 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10000447a0 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.751+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10080b90f0 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.752+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1008068a10 con 0x7f100815ec40 2024-04-17T07:14:55.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.753+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1000038020 con 0x7f100815ec40 2024-04-17T07:14:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.754+0000 7f0ffea00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 0x7f0fe406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.754+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f10000add00 con 0x7f100815ec40 2024-04-17T07:14:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.755+0000 7f1006a00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 0x7f0fe406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:55.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.756+0000 7f1006a00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 0x7f0fe406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0006fd0 tx=0x7f0ff0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.758+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f100003f070 con 0x7f100815ec40 2024-04-17T07:14:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.961+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f10080d1d40 con 0x7f100815ec40 2024-04-17T07:14:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.962+0000 7f0ffea00640 1 -- 172.21.15.67:0/3679245413 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f100007c0a0 con 0x7f100815ec40 2024-04-17T07:14:55.962 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:14:55.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.964+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 msgr2=0x7f0fe406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.964+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 0x7f0fe406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0006fd0 tx=0x7f0ff0008040 comp rx=0 tx=0).stop 2024-04-17T07:14:55.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.964+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 msgr2=0x7f10080b9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:55.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.964+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1000009960 tx=0x7f10000046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 shutdown_connections 2024-04-17T07:14:55.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100815ec40 0x7f10080b9bd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10080d2ce0 0x7f100814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f0fe406c8f0 0x7f0fe406edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 --2- 172.21.15.67:0/3679245413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10080b3660 0x7f10080c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:55.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 >> 172.21.15.67:0/3679245413 conn(0x7f100808bff0 msgr2=0x7f10080b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:55.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 shutdown_connections 2024-04-17T07:14:55.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:55.965+0000 7f1007e00640 1 -- 172.21.15.67:0/3679245413 wait complete. 2024-04-17T07:14:55.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:14:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.211+0000 7fe9ae600640 1 Processor -- start 2024-04-17T07:14:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.211+0000 7fe9ae600640 1 -- start start 2024-04-17T07:14:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9ae600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9a8143200 0x7fe9a80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9ae600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a81607a0 con 0x7fe9a80cd580 2024-04-17T07:14:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9ae600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a8160a60 con 0x7fe9a8142e20 2024-04-17T07:14:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9ae600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a8160d40 con 0x7fe9a8143200 2024-04-17T07:14:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9a7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9a80cd580 0x7fe9a80b08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45310/0 (socket says 172.21.15.67:45310) 2024-04-17T07:14:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.213+0000 7fe9a7e00640 1 -- 172.21.15.67:0/2492979387 learned_addr learned my addr 172.21.15.67:0/2492979387 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6a00640 1 --2- 172.21.15.67:0/2492979387 >> v2:172.21.15.187:3300/0 conn(0x7fe9a8143200 0x7fe9a80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978453936 0 0) 0x7fe9a81607a0 con 0x7fe9a80cd580 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe98c003520 con 0x7fe9a80cd580 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1661956743 0 0) 0x7fe98c003520 con 0x7fe9a80cd580 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 >> v2:172.21.15.187:3300/0 conn(0x7fe9a8143200 msgr2=0x7fe9a80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 --2- 172.21.15.67:0/2492979387 >> v2:172.21.15.187:3300/0 conn(0x7fe9a8143200 0x7fe9a80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 >> v1:172.21.15.187:6789/0 conn(0x7fe9a8142e20 legacy=0x7fe9a80cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a8161020 con 0x7fe9a80cd580 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe9a0001670 con 0x7fe9a80cd580 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.214+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 >> v1:172.21.15.67:6789/0 conn(0x7fe9a80cd580 legacy=0x7fe9a80b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe98c003a00 0x7fe98c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c004800 0x7fe98c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 0x7fe98c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a8160d40 con 0x7fe98c005210 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c00c160 con 0x7fe98c004800 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe98c00c360 con 0x7fe98c003a00 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 0x7fe98c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a6a00640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c004800 0x7fe98c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 -- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe98c003a00 msgr2=0x7fe98c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7400640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe98c003a00 0x7fe98c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe98c003a00 0x7fe98c003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 -- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c004800 msgr2=0x7fe98c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c004800 0x7fe98c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.215+0000 7fe9a7e00640 1 -- 172.21.15.67:0/2492979387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c003520 con 0x7fe98c005210 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.216+0000 7fe9a7e00640 1 -- 172.21.15.67:0/2492979387 reap_dead start 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 0x7fe98c00b9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c0e0 tx=0x7fe98c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeafd65ef9b4282 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0007c40 con 0x7fe98c005210 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a0030070 con 0x7fe98c005210 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9a6000640 1 -- 172.21.15.67:0/2492979387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0005d10 con 0x7fe98c005210 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9ae600640 1 -- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 msgr2=0x7fe98c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.217+0000 7fe9ae600640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 0x7fe98c00b9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c0e0 tx=0x7fe98c005830 comp rx=0 tx=0).stop 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.218+0000 7fe9ae600640 1 -- 172.21.15.67:0/2492979387 shutdown_connections 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.218+0000 7fe9ae600640 1 --2- 172.21.15.67:0/2492979387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe98c005210 0x7fe98c00b9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.218+0000 7fe9ae600640 1 -- 172.21.15.67:0/2492979387 >> 172.21.15.67:0/2492979387 conn(0x7fe9a806b8c0 msgr2=0x7fe9a806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.218+0000 7fe9ae600640 1 -- 172.21.15.67:0/2492979387 shutdown_connections 2024-04-17T07:14:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.218+0000 7fe9ae600640 1 -- 172.21.15.67:0/2492979387 wait complete. 2024-04-17T07:14:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.220+0000 7fe9ae600640 1 Processor -- start 2024-04-17T07:14:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.220+0000 7fe9ae600640 1 -- start start 2024-04-17T07:14:56.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 0x7fe9a815f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 0x7fe9a8135c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a80cd040 con 0x7fe9a80aec80 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a80ce730 con 0x7fe98c005210 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9ae600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a80cd1c0 con 0x7fe9a813df00 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51656/0 (socket says 172.21.15.67:51656) 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9a7e00640 1 -- 172.21.15.67:0/775750385 learned_addr learned my addr 172.21.15.67:0/775750385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9a7400640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 0x7fe9a815f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.221+0000 7fe9a6a00640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 0x7fe9a8135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 -- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 msgr2=0x7fe9a8135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 0x7fe9a8135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 -- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 msgr2=0x7fe9a815f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 0x7fe9a815f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 -- 172.21.15.67:0/775750385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a80b5090 con 0x7fe9a80aec80 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7400640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 0x7fe9a815f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a6a00640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 0x7fe9a8135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9a7e00640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c0e0 tx=0x7fe9a8160c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0004bf0 con 0x7fe9a80aec80 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a0000ed0 con 0x7fe9a80aec80 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a80b5330 con 0x7fe9a80aec80 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.222+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a002de20 con 0x7fe9a80aec80 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.223+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a8136160 con 0x7fe9a80aec80 2024-04-17T07:14:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.224+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a8068b90 con 0x7fe9a80aec80 2024-04-17T07:14:56.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.225+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe9a0007070 con 0x7fe9a80aec80 2024-04-17T07:14:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.226+0000 7fe99f400640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 0x7fe98406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.226+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fe9a00acaa0 con 0x7fe9a80aec80 2024-04-17T07:14:56.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.227+0000 7fe9a7400640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 0x7fe98406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.228+0000 7fe9a7400640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 0x7fe98406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe998001d60 tx=0x7fe998009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.231+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe9a007aec0 con 0x7fe9a80aec80 2024-04-17T07:14:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.433+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe9a81607a0 con 0x7fe9a80aec80 2024-04-17T07:14:56.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.434+0000 7fe99f400640 1 -- 172.21.15.67:0/775750385 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fe9a007a7c0 con 0x7fe9a80aec80 2024-04-17T07:14:56.434 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 msgr2=0x7fe98406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 0x7fe98406edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe998001d60 tx=0x7fe998009040 comp rx=0 tx=0).stop 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 msgr2=0x7fe9a815cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c0e0 tx=0x7fe9a8160c30 comp rx=0 tx=0).stop 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 shutdown_connections 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9a80aec80 0x7fe9a815cde0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe98406c8f0 0x7fe98406edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9a813df00 0x7fe9a8135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 --2- 172.21.15.67:0/775750385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe98c005210 0x7fe9a815f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 >> 172.21.15.67:0/775750385 conn(0x7fe9a806b8c0 msgr2=0x7fe9a8142250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 shutdown_connections 2024-04-17T07:14:56.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.437+0000 7fe9ae600640 1 -- 172.21.15.67:0/775750385 wait complete. 2024-04-17T07:14:56.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.678+0000 7f6a16600640 1 Processor -- start 2024-04-17T07:14:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.678+0000 7f6a16600640 1 -- start start 2024-04-17T07:14:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.680+0000 7f6a16600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a1015ec60 0x7f6a100b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.680+0000 7f6a16600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.680+0000 7f6a16600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a100d2d30 con 0x7f6a100b6b30 2024-04-17T07:14:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.680+0000 7f6a16600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6a100d49c0 con 0x7f6a1015ec60 2024-04-17T07:14:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.680+0000 7f6a16600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6a100d3660 con 0x7f6a100cfae0 2024-04-17T07:14:56.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.681+0000 7f6a15c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.681+0000 7f6a15c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:38500/0 (socket says 172.21.15.67:38500) 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.681+0000 7f6a15c00640 1 -- 172.21.15.67:0/982070246 learned_addr learned my addr 172.21.15.67:0/982070246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.681+0000 7f6a15200640 1 --2- 172.21.15.67:0/982070246 >> v2:172.21.15.187:3300/0 conn(0x7f6a1015ec60 0x7f6a100b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.682+0000 7f6a15c00640 1 -- 172.21.15.67:0/982070246 >> v2:172.21.15.187:3300/0 conn(0x7f6a1015ec60 msgr2=0x7f6a100b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.682+0000 7f6a15c00640 1 --2- 172.21.15.67:0/982070246 >> v2:172.21.15.187:3300/0 conn(0x7f6a1015ec60 0x7f6a100b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.682+0000 7f6a15c00640 1 -- 172.21.15.67:0/982070246 >> v1:172.21.15.187:6789/0 conn(0x7f6a100b6b30 legacy=0x7f6a100cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.682+0000 7f6a15c00640 1 -- 172.21.15.67:0/982070246 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a100d3010 con 0x7f6a100cfae0 2024-04-17T07:14:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.682+0000 7f6a0f400640 1 -- 172.21.15.67:0/982070246 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869283762 0 0) 0x7f6a100d2d30 con 0x7f6a100b6b30 2024-04-17T07:14:56.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.683+0000 7f6a15c00640 1 --2- 172.21.15.67:0/982070246 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009e50 tx=0x7f6a0802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=112fd49d71a0614f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.683+0000 7f6a0f400640 1 -- 172.21.15.67:0/982070246 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0802cb00 con 0x7f6a100cfae0 2024-04-17T07:14:56.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.683+0000 7f6a0f400640 1 -- 172.21.15.67:0/982070246 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a0802cca0 con 0x7f6a100cfae0 2024-04-17T07:14:56.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.683+0000 7f6a0f400640 1 -- 172.21.15.67:0/982070246 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08034610 con 0x7f6a100cfae0 2024-04-17T07:14:56.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 -- 172.21.15.67:0/982070246 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 msgr2=0x7f6a100b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 --2- 172.21.15.67:0/982070246 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009e50 tx=0x7f6a0802bf60 comp rx=0 tx=0).stop 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 -- 172.21.15.67:0/982070246 shutdown_connections 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 --2- 172.21.15.67:0/982070246 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a100cfae0 0x7f6a100b7b70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 --2- 172.21.15.67:0/982070246 >> v2:172.21.15.187:3300/0 conn(0x7f6a1015ec60 0x7f6a100b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 -- 172.21.15.67:0/982070246 >> 172.21.15.67:0/982070246 conn(0x7f6a100d15f0 msgr2=0x7f6a100d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 -- 172.21.15.67:0/982070246 shutdown_connections 2024-04-17T07:14:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.684+0000 7f6a16600640 1 -- 172.21.15.67:0/982070246 wait complete. 2024-04-17T07:14:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.686+0000 7f6a16600640 1 Processor -- start 2024-04-17T07:14:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.686+0000 7f6a16600640 1 -- start start 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a100cfae0 0x7f6a101f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a1015ec60 0x7f6a1015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1015f040 con 0x7f6a100b6b30 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100b6f70 con 0x7f6a100cfae0 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a16600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6a100d00c0 con 0x7f6a1015ec60 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.687+0000 7f6a15200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51672/0 (socket says 172.21.15.67:51672) 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 -- 172.21.15.67:0/2342759330 learned_addr learned my addr 172.21.15.67:0/2342759330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15c00640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a1015ec60 0x7f6a1015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 -- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a1015ec60 msgr2=0x7f6a1015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a1015ec60 0x7f6a1015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 -- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a100cfae0 msgr2=0x7f6a101f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a100cfae0 0x7f6a101f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 -- 172.21.15.67:0/2342759330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a101f4400 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.688+0000 7f6a15200640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400e9e0 tx=0x7f6a0400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.689+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a04017070 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.689+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a0400c970 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.689+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a10151560 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.689+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0400cd60 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.689+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a10151b20 con 0x7f6a100b6b30 2024-04-17T07:14:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.690+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a10068a10 con 0x7f6a100b6b30 2024-04-17T07:14:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.691+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6a0401a070 con 0x7f6a100b6b30 2024-04-17T07:14:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.692+0000 7f6a0cc00640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 0x7f69f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.692+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f6a0408bb90 con 0x7f6a100b6b30 2024-04-17T07:14:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.693+0000 7f6a0fe00640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 0x7f69f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:56.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.694+0000 7f6a0fe00640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 0x7f69f006edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6a1015e1e0 tx=0x7f69fc007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.697+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6a04059f30 con 0x7f6a100b6b30 2024-04-17T07:14:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.943+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a1015de50 con 0x7f6a100b6b30 2024-04-17T07:14:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.944+0000 7f6a0cc00640 1 -- 172.21.15.67:0/2342759330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7f6a04059830 con 0x7f6a100b6b30 2024-04-17T07:14:56.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:56.944 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:56.945 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 msgr2=0x7f69f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 0x7f69f006edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6a1015e1e0 tx=0x7f69fc007930 comp rx=0 tx=0).stop 2024-04-17T07:14:56.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 msgr2=0x7f6a100cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400e9e0 tx=0x7f6a0400eea0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 shutdown_connections 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f69f006c8f0 0x7f69f006edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a1015ec60 0x7f6a1015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a100cfae0 0x7f6a101f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 --2- 172.21.15.67:0/2342759330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a100b6b30 0x7f6a100cc750 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 >> 172.21.15.67:0/2342759330 conn(0x7f6a100d15f0 msgr2=0x7f6a100b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 shutdown_connections 2024-04-17T07:14:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:56.947+0000 7f6a16600640 1 -- 172.21.15.67:0/2342759330 wait complete. 2024-04-17T07:14:56.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.184+0000 7fe104e00640 1 Processor -- start 2024-04-17T07:14:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.185+0000 7fe104e00640 1 -- start start 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.186+0000 7fe104e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10015fe10 0x7fe1000c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.186+0000 7fe104e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1000d0700 0x7fe1000d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.186+0000 7fe104e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10009b8a0 con 0x7fe1000c6710 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.186+0000 7fe104e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1000cd7f0 con 0x7fe10015fe10 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.186+0000 7fe104e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe10009bb80 con 0x7fe1000d0700 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10015fe10 0x7fe1000c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10015fe10 0x7fe1000c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51676/0 (socket says 172.21.15.67:51676) 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 -- 172.21.15.67:0/1693003188 learned_addr learned my addr 172.21.15.67:0/1693003188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:57.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/1693003188 >> v2:172.21.15.187:3300/0 conn(0x7fe1000d0700 0x7fe1000d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 -- 172.21.15.67:0/1693003188 >> v2:172.21.15.187:3300/0 conn(0x7fe1000d0700 msgr2=0x7fe1000d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:57.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 --2- 172.21.15.67:0/1693003188 >> v2:172.21.15.187:3300/0 conn(0x7fe1000d0700 0x7fe1000d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 -- 172.21.15.67:0/1693003188 >> v1:172.21.15.187:6789/0 conn(0x7fe1000c6710 legacy=0x7fe1000cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:57.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.187+0000 7fe0ff400640 1 -- 172.21.15.67:0/1693003188 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1000cd480 con 0x7fe10015fe10 2024-04-17T07:14:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.189+0000 7fe0ff400640 1 --2- 172.21.15.67:0/1693003188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10015fe10 0x7fe1000c61d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe100073940 tx=0x7fe0f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da98a1eacafd7fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.189+0000 7fe0fea00640 1 -- 172.21.15.67:0/1693003188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0035020 con 0x7fe10015fe10 2024-04-17T07:14:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.189+0000 7fe0fea00640 1 -- 172.21.15.67:0/1693003188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f0031870 con 0x7fe10015fe10 2024-04-17T07:14:57.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.189+0000 7fe0fea00640 1 -- 172.21.15.67:0/1693003188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0031c40 con 0x7fe10015fe10 2024-04-17T07:14:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 -- 172.21.15.67:0/1693003188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10015fe10 msgr2=0x7fe1000c61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 --2- 172.21.15.67:0/1693003188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10015fe10 0x7fe1000c61d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe100073940 tx=0x7fe0f002b910 comp rx=0 tx=0).stop 2024-04-17T07:14:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 -- 172.21.15.67:0/1693003188 shutdown_connections 2024-04-17T07:14:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 --2- 172.21.15.67:0/1693003188 >> v2:172.21.15.187:3300/0 conn(0x7fe1000d0700 0x7fe1000d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 --2- 172.21.15.67:0/1693003188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10015fe10 0x7fe1000c61d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 -- 172.21.15.67:0/1693003188 >> 172.21.15.67:0/1693003188 conn(0x7fe1000b4170 msgr2=0x7fe1000b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 -- 172.21.15.67:0/1693003188 shutdown_connections 2024-04-17T07:14:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.190+0000 7fe104e00640 1 -- 172.21.15.67:0/1693003188 wait complete. 2024-04-17T07:14:57.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.192+0000 7fe104e00640 1 Processor -- start 2024-04-17T07:14:57.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.192+0000 7fe104e00640 1 -- start start 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1000d0700 0x7fe10009f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe10015fe10 0x7fe1001f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe100160370 con 0x7fe1000c6710 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1001601f0 con 0x7fe10015fe10 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe104e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe10009b470 con 0x7fe1000d0700 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe0ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe0ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51684/0 (socket says 172.21.15.67:51684) 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.193+0000 7fe0ff400640 1 -- 172.21.15.67:0/1019585035 learned_addr learned my addr 172.21.15.67:0/1019585035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 -- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1000d0700 msgr2=0x7fe10009f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1000d0700 0x7fe10009f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 -- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe10015fe10 msgr2=0x7fe1001f8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe10015fe10 0x7fe1001f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 -- 172.21.15.67:0/1019585035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10009faa0 con 0x7fe1000c6710 2024-04-17T07:14:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0ff400640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0000c00 tx=0x7fe0f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0040070 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f0030d00 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0035040 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.194+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10009fd80 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.195+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1000a0340 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.196+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1000755d0 con 0x7fe1000c6710 2024-04-17T07:14:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.197+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0f004b060 con 0x7fe1000c6710 2024-04-17T07:14:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.198+0000 7fe0f7400640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 0x7fe0dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.198+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fe0f00adaf0 con 0x7fe1000c6710 2024-04-17T07:14:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.198+0000 7fe0f7e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 0x7fe0dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.199+0000 7fe0f7e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 0x7fe0dc06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe10015f770 tx=0x7fe0e80079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.202+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe0f007bd70 con 0x7fe1000c6710 2024-04-17T07:14:57.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.408+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe10006e9c0 con 0x7fe1000c6710 2024-04-17T07:14:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.409+0000 7fe0f7400640 1 -- 172.21.15.67:0/1019585035 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7fe0f007b670 con 0x7fe1000c6710 2024-04-17T07:14:57.410 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:57.410 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 msgr2=0x7fe0dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 0x7fe0dc06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe10015f770 tx=0x7fe0e80079b0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 msgr2=0x7fe10009cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0000c00 tx=0x7fe0f00046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 shutdown_connections 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe10015fe10 0x7fe1001f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe0dc06c8f0 0x7fe0dc06edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1000d0700 0x7fe10009f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 --2- 172.21.15.67:0/1019585035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1000c6710 0x7fe10009cea0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 >> 172.21.15.67:0/1019585035 conn(0x7fe1000b4170 msgr2=0x7fe100071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:57.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 shutdown_connections 2024-04-17T07:14:57.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.412+0000 7fe104e00640 1 -- 172.21.15.67:0/1019585035 wait complete. 2024-04-17T07:14:57.432 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:14:57.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:57.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.657+0000 7fdfd3200640 1 Processor -- start 2024-04-17T07:14:57.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.658+0000 7fdfd3200640 1 -- start start 2024-04-17T07:14:57.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.659+0000 7fdfd3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfd40cfae0 0x7fdfd4169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.659+0000 7fdfd3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd40d2d30 con 0x7fdfd40b6d00 2024-04-17T07:14:57.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.659+0000 7fdfd3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd40d3010 con 0x7fdfd415ec60 2024-04-17T07:14:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.659+0000 7fdfd3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd40d49c0 con 0x7fdfd40cfae0 2024-04-17T07:14:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.660+0000 7fdfcbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdfd40b6d00 0x7fdfd40cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45316/0 (socket says 172.21.15.67:45316) 2024-04-17T07:14:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.660+0000 7fdfcbe00640 1 -- 172.21.15.67:0/760496621 learned_addr learned my addr 172.21.15.67:0/760496621 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:57.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.660+0000 7fdfd2800640 1 --2- 172.21.15.67:0/760496621 >> v2:172.21.15.187:3300/0 conn(0x7fdfd40cfae0 0x7fdfd4169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.660+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968481555 0 0) 0x7fdfd40d2d30 con 0x7fdfd40b6d00 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.660+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfb8003520 con 0x7fdfd40b6d00 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2445709846 0 0) 0x7fdfb8003520 con 0x7fdfd40b6d00 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 >> v2:172.21.15.187:3300/0 conn(0x7fdfd40cfae0 msgr2=0x7fdfd4169810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 --2- 172.21.15.67:0/760496621 >> v2:172.21.15.187:3300/0 conn(0x7fdfd40cfae0 0x7fdfd4169810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 >> v1:172.21.15.187:6789/0 conn(0x7fdfd415ec60 legacy=0x7fdfd40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd40d32f0 con 0x7fdfd40b6d00 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fdfbc001670 con 0x7fdfd40b6d00 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 >> v1:172.21.15.67:6789/0 conn(0x7fdfd40b6d00 legacy=0x7fdfd40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8003a20 0x7fdfb8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb8004800 0x7fdfb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd40d49c0 con 0x7fdfb80051c0 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800c0f0 con 0x7fdfb8003a20 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb800c2f0 con 0x7fdfb8004800 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.661+0000 7fdfd2800640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd2800640 1 -- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb8004800 msgr2=0x7fdfb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd2800640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb8004800 0x7fdfb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd1e00640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8003a20 0x7fdfb8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd2800640 1 -- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8003a20 msgr2=0x7fdfb8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd2800640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8003a20 0x7fdfb8003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.662+0000 7fdfd2800640 1 -- 172.21.15.67:0/760496621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb8003520 con 0x7fdfb80051c0 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.663+0000 7fdfd2800640 1 -- 172.21.15.67:0/760496621 reap_dead start 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.663+0000 7fdfd2800640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfb800b9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009fc0 tx=0x7fdfb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bd3ce988dfc308 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.663+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc009c10 con 0x7fdfb80051c0 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.663+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc02cca0 con 0x7fdfb80051c0 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.663+0000 7fdfd1400640 1 -- 172.21.15.67:0/760496621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc003bd0 con 0x7fdfb80051c0 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 -- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 msgr2=0x7fdfb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfb800b9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009fc0 tx=0x7fdfb8005830 comp rx=0 tx=0).stop 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 -- 172.21.15.67:0/760496621 shutdown_connections 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 --2- 172.21.15.67:0/760496621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfb800b9c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 -- 172.21.15.67:0/760496621 >> 172.21.15.67:0/760496621 conn(0x7fdfd40d15f0 msgr2=0x7fdfd40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 -- 172.21.15.67:0/760496621 shutdown_connections 2024-04-17T07:14:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.664+0000 7fdfd3200640 1 -- 172.21.15.67:0/760496621 wait complete. 2024-04-17T07:14:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.666+0000 7fdfd3200640 1 Processor -- start 2024-04-17T07:14:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.666+0000 7fdfd3200640 1 -- start start 2024-04-17T07:14:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfd41f7610 0x7fdfd415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd415f2c0 0x7fdfd415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd40d00c0 con 0x7fdfb80051c0 2024-04-17T07:14:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd40d3660 con 0x7fdfd415f2c0 2024-04-17T07:14:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd40d3af0 con 0x7fdfd41f7610 2024-04-17T07:14:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51696/0 (socket says 172.21.15.67:51696) 2024-04-17T07:14:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.667+0000 7fdfd1e00640 1 -- 172.21.15.67:0/3925016921 learned_addr learned my addr 172.21.15.67:0/3925016921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfcbe00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfd41f7610 0x7fdfd415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 -- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfd41f7610 msgr2=0x7fdfd415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfd41f7610 0x7fdfd415ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 -- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd415f2c0 msgr2=0x7fdfd415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd2800640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd415f2c0 0x7fdfd415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd415f2c0 0x7fdfd415f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 -- 172.21.15.67:0/3925016921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd4160030 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.668+0000 7fdfd1e00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d520 tx=0x7fdfc400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.669+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4005480 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.669+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc4017070 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.669+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4003830 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.669+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd4160310 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.669+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd41608d0 con 0x7fdfb80051c0 2024-04-17T07:14:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.670+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd4068a10 con 0x7fdfb80051c0 2024-04-17T07:14:57.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.671+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdfc40039d0 con 0x7fdfb80051c0 2024-04-17T07:14:57.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.672+0000 7fdfca000640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 0x7fdfb006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:57.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.672+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fdfc408b9a0 con 0x7fdfb80051c0 2024-04-17T07:14:57.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.673+0000 7fdfcbe00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 0x7fdfb006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:57.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.674+0000 7fdfcbe00640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 0x7fdfb006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc001eb0 tx=0x7fdfbc001cc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:57.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.677+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdfc4059c20 con 0x7fdfb80051c0 2024-04-17T07:14:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.924+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd40b6500 con 0x7fdfb80051c0 2024-04-17T07:14:57.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.926+0000 7fdfca000640 1 -- 172.21.15.67:0/3925016921 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7fdfc4059520 con 0x7fdfb80051c0 2024-04-17T07:14:57.926 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:57.926 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.928+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 msgr2=0x7fdfb006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.928+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 0x7fdfb006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc001eb0 tx=0x7fdfbc001cc0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.928+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 msgr2=0x7fdfd415a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.928+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d520 tx=0x7fdfc400d9e0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 shutdown_connections 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd415f2c0 0x7fdfd415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdfb006c920 0x7fdfb006ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfd41f7610 0x7fdfd415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 --2- 172.21.15.67:0/3925016921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb80051c0 0x7fdfd415a6f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:57.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 >> 172.21.15.67:0/3925016921 conn(0x7fdfd40d15f0 msgr2=0x7fdfd40b8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:57.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 shutdown_connections 2024-04-17T07:14:57.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:57.929+0000 7fdfd3200640 1 -- 172.21.15.67:0/3925016921 wait complete. 2024-04-17T07:14:57.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:14:58.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.172+0000 7f54a4e00640 1 Processor -- start 2024-04-17T07:14:58.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.172+0000 7f54a4e00640 1 -- start start 2024-04-17T07:14:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f54a4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 0x7f54a006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f54a4e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54a0141ed0 0x7f54a0142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f54a4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a00cd4a0 con 0x7f54a006b8c0 2024-04-17T07:14:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f54a4e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f54a015ce30 con 0x7f54a0141ed0 2024-04-17T07:14:58.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f54a4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54a01607f0 con 0x7f54a0160280 2024-04-17T07:14:58.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f549f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f54a006b8c0 0x7f54a00cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45318/0 (socket says 172.21.15.67:45318) 2024-04-17T07:14:58.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.174+0000 7f549f400640 1 -- 172.21.15.67:0/1406084033 learned_addr learned my addr 172.21.15.67:0/1406084033 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f5496a00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 0x7f54a006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549fe00640 1 --2- 172.21.15.67:0/1406084033 >> v2:172.21.15.187:3300/0 conn(0x7f54a0141ed0 0x7f54a0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341465149 0 0) 0x7f54a00cd4a0 con 0x7f54a006b8c0 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5484003520 con 0x7f54a006b8c0 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2889119243 0 0) 0x7f5484003520 con 0x7f54a006b8c0 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 msgr2=0x7f54a006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 0x7f54a006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 >> v2:172.21.15.187:3300/0 conn(0x7f54a0141ed0 msgr2=0x7f54a0142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 --2- 172.21.15.67:0/1406084033 >> v2:172.21.15.187:3300/0 conn(0x7f54a0141ed0 0x7f54a0142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a0160ad0 con 0x7f54a006b8c0 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5490001670 con 0x7f54a006b8c0 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 >> v1:172.21.15.67:6789/0 conn(0x7f54a006b8c0 legacy=0x7f54a00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5484003a20 0x7f5484003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.175+0000 7f549ea00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549ea00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 0x7f54840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54a015ce30 con 0x7f5484003a20 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54a01607f0 con 0x7f5484004800 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f548400b9f0 con 0x7f54a0160280 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5484003a20 0x7f5484003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 -- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 msgr2=0x7f54840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a0160280 0x7f54840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 -- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5484004800 msgr2=0x7f5484004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.176+0000 7f549f400640 1 -- 172.21.15.67:0/1406084033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5484003520 con 0x7f5484003a20 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.177+0000 7f549f400640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5484003a20 0x7f5484003e40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5490001050 tx=0x7f5484005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08582e9d89857e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490007ca0 con 0x7f5484003a20 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5490030070 con 0x7f5484003a20 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f549ea00640 1 -- 172.21.15.67:0/1406084033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490005d10 con 0x7f5484003a20 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f54a4e00640 1 -- 172.21.15.67:0/1406084033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5484003a20 msgr2=0x7f5484003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f54a4e00640 1 --2- 172.21.15.67:0/1406084033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5484003a20 0x7f5484003e40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5490001050 tx=0x7f5484005830 comp rx=0 tx=0).stop 2024-04-17T07:14:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f549fe00640 1 -- 172.21.15.67:0/1406084033 reap_dead start 2024-04-17T07:14:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f54a4e00640 1 -- 172.21.15.67:0/1406084033 shutdown_connections 2024-04-17T07:14:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.178+0000 7f54a4e00640 1 -- 172.21.15.67:0/1406084033 >> 172.21.15.67:0/1406084033 conn(0x7f54a0067d10 msgr2=0x7f54a00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.179+0000 7f54a4e00640 1 -- 172.21.15.67:0/1406084033 shutdown_connections 2024-04-17T07:14:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.179+0000 7f54a4e00640 1 -- 172.21.15.67:0/1406084033 wait complete. 2024-04-17T07:14:58.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.180+0000 7f54a4e00640 1 Processor -- start 2024-04-17T07:14:58.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.181+0000 7f54a4e00640 1 -- start start 2024-04-17T07:14:58.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.181+0000 7f54a4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5484003a20 0x7f54a0142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f54a4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 0x7f54a0155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f54a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f54a4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54a013f9b0 con 0x7f54a00b3910 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f54a4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54a0161090 con 0x7f54a00b4f20 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f54a4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54a013fb30 con 0x7f5484003a20 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f549fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f549fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51714/0 (socket says 172.21.15.67:51714) 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f549fe00640 1 -- 172.21.15.67:0/3219119821 learned_addr learned my addr 172.21.15.67:0/3219119821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f5496a00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 0x7f54a0155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f549f400640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5484003a20 0x7f54a0142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.182+0000 7f549fe00640 1 -- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5484003a20 msgr2=0x7f54a0142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f549fe00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5484003a20 0x7f54a0142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f549fe00640 1 -- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 msgr2=0x7f54a0155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f549fe00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 0x7f54a0155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f549fe00640 1 -- 172.21.15.67:0/3219119821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a01564e0 con 0x7f54a00b3910 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f5496a00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 0x7f54a0155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f549fe00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f549800ede0 tx=0x7f549800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549800bd10 con 0x7f54a00b3910 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f549801a070 con 0x7f54a00b3910 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498014640 con 0x7f54a00b3910 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a00b3e80 con 0x7f54a00b3910 2024-04-17T07:14:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.183+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a00b4440 con 0x7f54a00b3910 2024-04-17T07:14:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.185+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a0160040 con 0x7f54a00b3910 2024-04-17T07:14:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.186+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5498005d10 con 0x7f54a00b3910 2024-04-17T07:14:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.186+0000 7f5497e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 0x7f547c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.187+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f549808bc00 con 0x7f54a00b3910 2024-04-17T07:14:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.187+0000 7f549f400640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 0x7f547c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.188+0000 7f549f400640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 0x7f547c06ede0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5490034000 tx=0x7f5490007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.191+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f54a0160040 con 0x7f54a00b3910 2024-04-17T07:14:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.436+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54a0068a10 con 0x7f54a00b3910 2024-04-17T07:14:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.437+0000 7f5497e00640 1 -- 172.21.15.67:0/3219119821 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5196 (secure 0 0 0) 0x7f5498092080 con 0x7f54a00b3910 2024-04-17T07:14:58.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:58.438 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/195262631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":195262631},{"type":"v1","addr":"172.21.15.187:6836","nonce":195262631}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1661364933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1661364933},{"type":"v1","addr":"172.21.15.187:6837","nonce":1661364933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1549983218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1549983218},{"type":"v1","addr":"172.21.15.67:6834","nonce":1549983218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:13:57.651408+0000","modified":"2024-04-17T07:14:00.734636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/4066505698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4066505698},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066505698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4254,"qdb_cluster":[4254]},"id":1}]} 2024-04-17T07:14:58.438 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 msgr2=0x7f547c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 0x7f547c06ede0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5490034000 tx=0x7f5490007000 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 msgr2=0x7f54a00b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f549800ede0 tx=0x7f549800b480 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 shutdown_connections 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a00b3910 0x7f54a00b1280 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a00b4f20 0x7f54a0155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f547c06c920 0x7f547c06ede0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 --2- 172.21.15.67:0/3219119821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5484003a20 0x7f54a0142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 >> 172.21.15.67:0/3219119821 conn(0x7f54a0067d10 msgr2=0x7f54a006f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 shutdown_connections 2024-04-17T07:14:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.440+0000 7f54a4e00640 1 -- 172.21.15.67:0/3219119821 wait complete. 2024-04-17T07:14:58.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:14:58.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.671+0000 7f2d3ee00640 1 Processor -- start 2024-04-17T07:14:58.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.671+0000 7f2d3ee00640 1 -- start start 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3815c5a0 0x7f2d3815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d380b46c0 0x7f2d380ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3ee00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3815f440 con 0x7f2d380b4210 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3815f720 con 0x7f2d3815c5a0 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3815fa00 con 0x7f2d380b46c0 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3815c5a0 0x7f2d3815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3815c5a0 0x7f2d3815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51730/0 (socket says 172.21.15.67:51730) 2024-04-17T07:14:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.673+0000 7f2d3da00640 1 -- 172.21.15.67:0/1237699665 learned_addr learned my addr 172.21.15.67:0/1237699665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.674+0000 7f2d3da00640 1 -- 172.21.15.67:0/1237699665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d380b46c0 msgr2=0x7f2d380ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.674+0000 7f2d3da00640 1 --2- 172.21.15.67:0/1237699665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d380b46c0 0x7f2d380ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.674+0000 7f2d3da00640 1 -- 172.21.15.67:0/1237699665 >> v1:172.21.15.187:6789/0 conn(0x7f2d380b4210 legacy=0x7f2d380af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.674+0000 7f2d3da00640 1 -- 172.21.15.67:0/1237699665 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3815fce0 con 0x7f2d3815c5a0 2024-04-17T07:14:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.675+0000 7f2d3da00640 1 --2- 172.21.15.67:0/1237699665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3815c5a0 0x7f2d3815ee60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009870 tx=0x7f2d2c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f195ee4f8e4d5b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.675+0000 7f2d37e00640 1 -- 172.21.15.67:0/1237699665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c038070 con 0x7f2d3815c5a0 2024-04-17T07:14:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.675+0000 7f2d37e00640 1 -- 172.21.15.67:0/1237699665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c031740 con 0x7f2d3815c5a0 2024-04-17T07:14:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.675+0000 7f2d37e00640 1 -- 172.21.15.67:0/1237699665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c02f050 con 0x7f2d3815c5a0 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 -- 172.21.15.67:0/1237699665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3815c5a0 msgr2=0x7f2d3815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/1237699665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3815c5a0 0x7f2d3815ee60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009870 tx=0x7f2d2c02b7e0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 -- 172.21.15.67:0/1237699665 shutdown_connections 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/1237699665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d380b46c0 0x7f2d380ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/1237699665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3815c5a0 0x7f2d3815ee60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 -- 172.21.15.67:0/1237699665 >> 172.21.15.67:0/1237699665 conn(0x7f2d38067d10 msgr2=0x7f2d380b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 -- 172.21.15.67:0/1237699665 shutdown_connections 2024-04-17T07:14:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.676+0000 7f2d3ee00640 1 -- 172.21.15.67:0/1237699665 wait complete. 2024-04-17T07:14:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.678+0000 7f2d3ee00640 1 Processor -- start 2024-04-17T07:14:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.678+0000 7f2d3ee00640 1 -- start start 2024-04-17T07:14:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3815c5a0 0x7f2d381ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d381eaa10 0x7f2d381e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3806e7f0 con 0x7f2d380b4210 2024-04-17T07:14:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3806e670 con 0x7f2d3815c5a0 2024-04-17T07:14:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d380afb50 con 0x7f2d381eaa10 2024-04-17T07:14:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51734/0 (socket says 172.21.15.67:51734) 2024-04-17T07:14:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.679+0000 7f2d3da00640 1 -- 172.21.15.67:0/4050570191 learned_addr learned my addr 172.21.15.67:0/4050570191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 -- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d381eaa10 msgr2=0x7f2d381e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3e400640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d381eaa10 0x7f2d381e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d381eaa10 0x7f2d381e4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 -- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3815c5a0 msgr2=0x7f2d381ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3815c5a0 0x7f2d381ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 -- 172.21.15.67:0/4050570191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d381e5950 con 0x7f2d380b4210 2024-04-17T07:14:58.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.680+0000 7f2d3da00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009870 tx=0x7f2d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.681+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c043070 con 0x7f2d380b4210 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.681+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c031e80 con 0x7f2d380b4210 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.681+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c038040 con 0x7f2d380b4210 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.681+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d381f5310 con 0x7f2d380b4210 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.681+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d381f58d0 con 0x7f2d380b4210 2024-04-17T07:14:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.682+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d381eae20 con 0x7f2d380b4210 2024-04-17T07:14:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.683+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d2c046070 con 0x7f2d380b4210 2024-04-17T07:14:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.684+0000 7f2d35600640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 0x7f2d1806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.684+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f2d2c0ad920 con 0x7f2d380b4210 2024-04-17T07:14:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.685+0000 7f2d3d000640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 0x7f2d1806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.686+0000 7f2d3d000640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 0x7f2d1806ee00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d3815ec90 tx=0x7f2d24007990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:58.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.689+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2d381eae20 con 0x7f2d380b4210 2024-04-17T07:14:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.889+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d38068a10 con 0x7f2d380b4210 2024-04-17T07:14:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.892+0000 7f2d35600640 1 -- 172.21.15.67:0/4050570191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16807 (secure 0 0 0) 0x7f2d2c04d030 con 0x7f2d380b4210 2024-04-17T07:14:58.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:14:58.893 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:14:45.013339+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:13:55.714444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T07:13:56.666401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 msgr2=0x7f2d1806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 0x7f2d1806ee00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d3815ec90 tx=0x7f2d24007990 comp rx=0 tx=0).stop 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 msgr2=0x7f2d38153250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009870 tx=0x7f2d2c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 shutdown_connections 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d1806c940 0x7f2d1806ee00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d381eaa10 0x7f2d381e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3815c5a0 0x7f2d381ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 --2- 172.21.15.67:0/4050570191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d380b4210 0x7f2d38153250 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 >> 172.21.15.67:0/4050570191 conn(0x7f2d38067d10 msgr2=0x7f2d380ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 shutdown_connections 2024-04-17T07:14:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:58.895+0000 7f2d3ee00640 1 -- 172.21.15.67:0/4050570191 wait complete. 2024-04-17T07:14:58.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:14:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.135+0000 7fc5cfe00640 1 Processor -- start 2024-04-17T07:14:59.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.136+0000 7fc5cfe00640 1 -- start start 2024-04-17T07:14:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.137+0000 7fc5cfe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc5d00d1ab0 0x7fc5d00d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.137+0000 7fc5cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5d00d3350 0x7fc5d00b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.137+0000 7fc5cfe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d00d4140 con 0x7fc5d015ee20 2024-04-17T07:14:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.137+0000 7fc5cfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d00b83b0 con 0x7fc5d00d3350 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.137+0000 7fc5cfe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d00d3b80 con 0x7fc5d00d1ab0 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5d015ee20 0x7fc5d00d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45334/0 (socket says 172.21.15.67:45334) 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cea00640 1 -- 172.21.15.67:0/703126406 learned_addr learned my addr 172.21.15.67:0/703126406 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cf400640 1 --2- 172.21.15.67:0/703126406 >> v2:172.21.15.67:3300/0 conn(0x7fc5d00d3350 0x7fc5d00b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5ce000640 1 --2- 172.21.15.67:0/703126406 >> v2:172.21.15.187:3300/0 conn(0x7fc5d00d1ab0 0x7fc5d00d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cd600640 1 -- 172.21.15.67:0/703126406 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428280294 0 0) 0x7fc5d00d4140 con 0x7fc5d015ee20 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cd600640 1 -- 172.21.15.67:0/703126406 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b4003520 con 0x7fc5d015ee20 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cf400640 1 -- 172.21.15.67:0/703126406 >> v2:172.21.15.187:3300/0 conn(0x7fc5d00d1ab0 msgr2=0x7fc5d00d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cf400640 1 --2- 172.21.15.67:0/703126406 >> v2:172.21.15.187:3300/0 conn(0x7fc5d00d1ab0 0x7fc5d00d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cf400640 1 -- 172.21.15.67:0/703126406 >> v1:172.21.15.67:6789/0 conn(0x7fc5d015ee20 legacy=0x7fc5d00d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.138+0000 7fc5cf400640 1 -- 172.21.15.67:0/703126406 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d00d3e60 con 0x7fc5d00d3350 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.139+0000 7fc5cf400640 1 --2- 172.21.15.67:0/703126406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d00b8840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00d5a90 tx=0x7fc5c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd742fe9781dc9a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cd600640 1 -- 172.21.15.67:0/703126406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8035020 con 0x7fc5d00d3350 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cd600640 1 -- 172.21.15.67:0/703126406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c8031830 con 0x7fc5d00d3350 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cd600640 1 -- 172.21.15.67:0/703126406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8031c00 con 0x7fc5d00d3350 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cfe00640 1 -- 172.21.15.67:0/703126406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 msgr2=0x7fc5d00b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/703126406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d00b8840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00d5a90 tx=0x7fc5c802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cfe00640 1 -- 172.21.15.67:0/703126406 shutdown_connections 2024-04-17T07:14:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/703126406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d00b8840 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.140+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/703126406 >> v2:172.21.15.187:3300/0 conn(0x7fc5d00d1ab0 0x7fc5d00d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.141+0000 7fc5cfe00640 1 -- 172.21.15.67:0/703126406 >> 172.21.15.67:0/703126406 conn(0x7fc5d006d3c0 msgr2=0x7fc5d00b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.141+0000 7fc5cfe00640 1 -- 172.21.15.67:0/703126406 shutdown_connections 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.141+0000 7fc5cfe00640 1 -- 172.21.15.67:0/703126406 wait complete. 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.142+0000 7fc5cfe00640 1 Processor -- start 2024-04-17T07:14:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 -- start start 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5d00d1ab0 0x7fc5d015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5d015ee20 0x7fc5d0150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d00d37e0 con 0x7fc5d00d3350 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d015f200 con 0x7fc5d015ee20 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.143+0000 7fc5cfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d00d3960 con 0x7fc5d00d1ab0 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51746/0 (socket says 172.21.15.67:51746) 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 -- 172.21.15.67:0/2610021379 learned_addr learned my addr 172.21.15.67:0/2610021379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 -- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5d00d1ab0 msgr2=0x7fc5d015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5d00d1ab0 0x7fc5d015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 -- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5d015ee20 msgr2=0x7fc5d0150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5d015ee20 0x7fc5d0150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.144+0000 7fc5ce000640 1 -- 172.21.15.67:0/2610021379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d015c1e0 con 0x7fc5d00d3350 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5ce000640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800d800 tx=0x7fc5b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800b6c0 con 0x7fc5d00d3350 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b8019070 con 0x7fc5d00d3350 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800ba80 con 0x7fc5d00d3350 2024-04-17T07:14:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d015e260 con 0x7fc5d00d3350 2024-04-17T07:14:59.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.145+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d015c480 con 0x7fc5d00d3350 2024-04-17T07:14:59.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.146+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d0068a10 con 0x7fc5d00d3350 2024-04-17T07:14:59.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.147+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5b800bc20 con 0x7fc5d00d3350 2024-04-17T07:14:59.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.148+0000 7fc5c7400640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 0x7fc5ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.148+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fc5b808c1f0 con 0x7fc5d00d3350 2024-04-17T07:14:59.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.149+0000 7fc5cea00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 0x7fc5ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.150+0000 7fc5cea00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 0x7fc5ac06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00012e0 tx=0x7fc5c00015b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.152+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc5b801c360 con 0x7fc5d00d3350 2024-04-17T07:14:59.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.401+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc5d00d0120 con 0x7fc5d00d3350 2024-04-17T07:14:59.402 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:14:59.402+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:14:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.406+0000 7fc5c7400640 1 -- 172.21.15.67:0/2610021379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fc5b805a470 con 0x7fc5d00d3350 2024-04-17T07:14:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:14:59.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.409+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 msgr2=0x7fc5ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.409+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 0x7fc5ac06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00012e0 tx=0x7fc5c00015b0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.409+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 msgr2=0x7fc5d015cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.409+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800d800 tx=0x7fc5b800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 shutdown_connections 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5d015ee20 0x7fc5d0150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d00d3350 0x7fc5d015cc80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc5ac06c8f0 0x7fc5ac06edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 --2- 172.21.15.67:0/2610021379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5d00d1ab0 0x7fc5d015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 >> 172.21.15.67:0/2610021379 conn(0x7fc5d006d3c0 msgr2=0x7fc5d01519f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 shutdown_connections 2024-04-17T07:14:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.410+0000 7fc5cfe00640 1 -- 172.21.15.67:0/2610021379 wait complete. 2024-04-17T07:14:59.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:14:59.431 INFO:tasks.ceph.mds.c.smithi067.stderr: -8656> 2024-04-17T07:14:01.572+0000 7f0e09e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:14:59.505 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:14:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.627+0000 7ff577200640 1 Processor -- start 2024-04-17T07:14:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.627+0000 7ff577200640 1 -- start start 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.628+0000 7ff577200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5780b41a0 0x7ff5780b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.628+0000 7ff577200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff5780b4580 0x7ff578162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff577200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff578166a60 con 0x7ff57806f800 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff577200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff578166810 con 0x7ff5780b41a0 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff577200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff578166d40 con 0x7ff5780b4580 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff576800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff57806f800 0x7ff578071cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45352/0 (socket says 172.21.15.67:45352) 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff576800640 1 -- 172.21.15.67:0/687503306 learned_addr learned my addr 172.21.15.67:0/687503306 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:14:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff575e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5780b41a0 0x7ff5780b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff575400640 1 --2- 172.21.15.67:0/687503306 >> v2:172.21.15.187:3300/0 conn(0x7ff5780b4580 0x7ff578162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff575e00640 1 -- 172.21.15.67:0/687503306 >> v2:172.21.15.187:3300/0 conn(0x7ff5780b4580 msgr2=0x7ff578162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:14:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.629+0000 7ff575e00640 1 --2- 172.21.15.67:0/687503306 >> v2:172.21.15.187:3300/0 conn(0x7ff5780b4580 0x7ff578162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.630+0000 7ff575e00640 1 -- 172.21.15.67:0/687503306 >> v1:172.21.15.67:6789/0 conn(0x7ff57806f800 legacy=0x7ff578071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.630+0000 7ff575e00640 1 -- 172.21.15.67:0/687503306 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff578167020 con 0x7ff5780b41a0 2024-04-17T07:14:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.631+0000 7ff575e00640 1 --2- 172.21.15.67:0/687503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5780b41a0 0x7ff5780b3490 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff5780b5980 tx=0x7ff56802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f234ad67f7fa22a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.632+0000 7ff574a00640 1 -- 172.21.15.67:0/687503306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568035020 con 0x7ff5780b41a0 2024-04-17T07:14:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.632+0000 7ff574a00640 1 -- 172.21.15.67:0/687503306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff568031830 con 0x7ff5780b41a0 2024-04-17T07:14:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.632+0000 7ff574a00640 1 -- 172.21.15.67:0/687503306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568031c00 con 0x7ff5780b41a0 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 -- 172.21.15.67:0/687503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5780b41a0 msgr2=0x7ff5780b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 --2- 172.21.15.67:0/687503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5780b41a0 0x7ff5780b3490 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff5780b5980 tx=0x7ff56802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 -- 172.21.15.67:0/687503306 shutdown_connections 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 --2- 172.21.15.67:0/687503306 >> v2:172.21.15.187:3300/0 conn(0x7ff5780b4580 0x7ff578162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 --2- 172.21.15.67:0/687503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5780b41a0 0x7ff5780b3490 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 -- 172.21.15.67:0/687503306 >> 172.21.15.67:0/687503306 conn(0x7ff5780b2190 msgr2=0x7ff5780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 -- 172.21.15.67:0/687503306 shutdown_connections 2024-04-17T07:14:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.633+0000 7ff577200640 1 -- 172.21.15.67:0/687503306 wait complete. 2024-04-17T07:14:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.635+0000 7ff577200640 1 Processor -- start 2024-04-17T07:14:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.635+0000 7ff577200640 1 -- start start 2024-04-17T07:14:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5780b41a0 0x7ff57815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5780b4580 0x7ff578156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff57806eec0 con 0x7ff57806f800 2024-04-17T07:14:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff57806ed40 con 0x7ff5780b4580 2024-04-17T07:14:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff577200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff578162bd0 con 0x7ff5780b41a0 2024-04-17T07:14:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff575e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff575e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51784/0 (socket says 172.21.15.67:51784) 2024-04-17T07:14:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.636+0000 7ff575e00640 1 -- 172.21.15.67:0/3256843611 learned_addr learned my addr 172.21.15.67:0/3256843611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 -- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5780b41a0 msgr2=0x7ff57815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff576800640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5780b4580 0x7ff578156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575400640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5780b41a0 0x7ff57815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5780b41a0 0x7ff57815a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 -- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5780b4580 msgr2=0x7ff578156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5780b4580 0x7ff578156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:14:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 -- 172.21.15.67:0/3256843611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57815a680 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.637+0000 7ff575e00640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff56802bee0 tx=0x7ff5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.638+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568041070 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.638+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff56803f070 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.638+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568035040 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.638+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5781646c0 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.638+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff578164c80 con 0x7ff57806f800 2024-04-17T07:14:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.639+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff578068a10 con 0x7ff57806f800 2024-04-17T07:14:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.640+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff568038020 con 0x7ff57806f800 2024-04-17T07:14:59.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.641+0000 7ff56e000640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:14:59.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.641+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7ff5680ad2a0 con 0x7ff57806f800 2024-04-17T07:14:59.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.642+0000 7ff575400640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:14:59.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.643+0000 7ff575400640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 0x7ff55406edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff560006fd0 tx=0x7ff560008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:14:59.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.646+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff56807b5f0 con 0x7ff57806f800 2024-04-17T07:14:59.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:14:59.894+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff5780cd870 con 0x7ff57806f800 2024-04-17T07:15:00.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.410+0000 7ff56e000640 1 -- 172.21.15.67:0/3256843611 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7ff56807aef0 con 0x7ff57806f800 2024-04-17T07:15:00.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 msgr2=0x7ff55406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 0x7ff55406edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff560006fd0 tx=0x7ff560008040 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 msgr2=0x7ff578164020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff56802bee0 tx=0x7ff5680046c0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 shutdown_connections 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5780b4580 0x7ff578156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff55406c8f0 0x7ff55406edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5780b41a0 0x7ff57815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 --2- 172.21.15.67:0/3256843611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff57806f800 0x7ff578164020 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 >> 172.21.15.67:0/3256843611 conn(0x7ff5780b2190 msgr2=0x7ff57806bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.414+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 shutdown_connections 2024-04-17T07:15:00.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.415+0000 7ff577200640 1 -- 172.21.15.67:0/3256843611 wait complete. 2024-04-17T07:15:00.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:15:00.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.661+0000 7f5928e00640 1 Processor -- start 2024-04-17T07:15:00.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.662+0000 7f5928e00640 1 -- start start 2024-04-17T07:15:00.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.663+0000 7f5928e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f592406ec10 0x7f59241635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:00.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.663+0000 7f5928e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59241640b0 con 0x7f5924163b30 2024-04-17T07:15:00.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.663+0000 7f5928e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5924164390 con 0x7f592406eff0 2024-04-17T07:15:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5928e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f59240b4270 con 0x7f592406ec10 2024-04-17T07:15:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5924163b30 0x7f59240b4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45354/0 (socket says 172.21.15.67:45354) 2024-04-17T07:15:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921400640 1 -- 172.21.15.67:0/3178971345 learned_addr learned my addr 172.21.15.67:0/3178971345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921e00640 1 --2- 172.21.15.67:0/3178971345 >> v2:172.21.15.67:3300/0 conn(0x7f592406ec10 0x7f59241635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5920a00640 1 -- 172.21.15.67:0/3178971345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176249733 0 0) 0x7f59241640b0 con 0x7f5924163b30 2024-04-17T07:15:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5920a00640 1 -- 172.21.15.67:0/3178971345 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5908003520 con 0x7f5924163b30 2024-04-17T07:15:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921e00640 1 -- 172.21.15.67:0/3178971345 >> v1:172.21.15.187:6789/0 conn(0x7f592406eff0 legacy=0x7f59241647b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921e00640 1 -- 172.21.15.67:0/3178971345 >> v1:172.21.15.67:6789/0 conn(0x7f5924163b30 legacy=0x7f59240b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.664+0000 7f5921e00640 1 -- 172.21.15.67:0/3178971345 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592406e200 con 0x7f592406ec10 2024-04-17T07:15:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.666+0000 7f5921e00640 1 --2- 172.21.15.67:0/3178971345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f59241635f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f59140097d0 tx=0x7f591402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa49354dcfa8b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.666+0000 7f5920a00640 1 -- 172.21.15.67:0/3178971345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914035020 con 0x7f592406ec10 2024-04-17T07:15:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.666+0000 7f5920a00640 1 -- 172.21.15.67:0/3178971345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5914031810 con 0x7f592406ec10 2024-04-17T07:15:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.666+0000 7f5920a00640 1 -- 172.21.15.67:0/3178971345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914031be0 con 0x7f592406ec10 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 -- 172.21.15.67:0/3178971345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 msgr2=0x7f59241635f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 --2- 172.21.15.67:0/3178971345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f59241635f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f59140097d0 tx=0x7f591402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 -- 172.21.15.67:0/3178971345 shutdown_connections 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 --2- 172.21.15.67:0/3178971345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f59241635f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 -- 172.21.15.67:0/3178971345 >> 172.21.15.67:0/3178971345 conn(0x7f59240b2370 msgr2=0x7f59240b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 -- 172.21.15.67:0/3178971345 shutdown_connections 2024-04-17T07:15:00.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.667+0000 7f5928e00640 1 -- 172.21.15.67:0/3178971345 wait complete. 2024-04-17T07:15:00.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.668+0000 7f5928e00640 1 Processor -- start 2024-04-17T07:15:00.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.669+0000 7f5928e00640 1 -- start start 2024-04-17T07:15:00.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f592406eff0 0x7f59241ec540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 0x7f59241eca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5924165040 con 0x7f592406ec10 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5924164ec0 con 0x7f5924163b30 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5928e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f592415d050 con 0x7f592406eff0 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51808/0 (socket says 172.21.15.67:51808) 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 -- 172.21.15.67:0/1785525093 learned_addr learned my addr 172.21.15.67:0/1785525093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 -- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f592406eff0 msgr2=0x7f59241ec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f592406eff0 0x7f59241ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 -- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 msgr2=0x7f59241eca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5922800640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 0x7f59241eca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 0x7f59241eca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5921e00640 1 -- 172.21.15.67:0/1785525093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5924151af0 con 0x7f592406ec10 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.670+0000 7f5922800640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 0x7f59241eca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:00.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f5921e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f591402bec0 tx=0x7f59140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:00.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914041070 con 0x7f592406ec10 2024-04-17T07:15:00.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591403ad00 con 0x7f592406ec10 2024-04-17T07:15:00.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914035040 con 0x7f592406ec10 2024-04-17T07:15:00.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59241ead40 con 0x7f592406ec10 2024-04-17T07:15:00.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.671+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5924156bd0 con 0x7f592406ec10 2024-04-17T07:15:00.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.672+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5924073980 con 0x7f592406ec10 2024-04-17T07:15:00.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.673+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5914038020 con 0x7f592406ec10 2024-04-17T07:15:00.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.675+0000 7f591a000640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:00.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.675+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f59140ad780 con 0x7f592406ec10 2024-04-17T07:15:00.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.675+0000 7f5921400640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.677+0000 7f5921400640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 0x7f590406edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5924152050 tx=0x7f5910007930 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.680+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f591407bb70 con 0x7f592406ec10 2024-04-17T07:15:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:00.884+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5924068d60 con 0x7f592406ec10 2024-04-17T07:15:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.652+0000 7f591a000640 1 -- 172.21.15.67:0/1785525093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) ==== 181+0+0 (secure 0 0 0) 0x7f591407b470 con 0x7f592406ec10 2024-04-17T07:15:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 msgr2=0x7f590406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 0x7f590406edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5924152050 tx=0x7f5910007930 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 msgr2=0x7f5924155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f591402bec0 tx=0x7f59140046c0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 shutdown_connections 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924163b30 0x7f59241eca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f590406c8f0 0x7f590406edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f592406eff0 0x7f59241ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 --2- 172.21.15.67:0/1785525093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406ec10 0x7f5924155070 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 >> 172.21.15.67:0/1785525093 conn(0x7f59240b2370 msgr2=0x7f59240cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 shutdown_connections 2024-04-17T07:15:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.656+0000 7f5928e00640 1 -- 172.21.15.67:0/1785525093 wait complete. 2024-04-17T07:15:01.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:15:01.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.895+0000 7fe1b2400640 1 Processor -- start 2024-04-17T07:15:01.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.895+0000 7fe1b2400640 1 -- start start 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.896+0000 7fe1b2400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe1ac145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1b2400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1ac146510 con 0x7fe1ac146080 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1b2400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1ac161000 con 0x7fe1ac0cd2f0 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1b2400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1ac0cfa60 con 0x7fe1ac0cd6d0 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1b1a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe1ac146080 0x7fe1ac160360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45362/0 (socket says 172.21.15.67:45362) 2024-04-17T07:15:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1b1a00640 1 -- 172.21.15.67:0/4089629104 learned_addr learned my addr 172.21.15.67:0/4089629104 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.897+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635881065 0 0) 0x7fe1ac146510 con 0x7fe1ac146080 2024-04-17T07:15:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe194003520 con 0x7fe1ac146080 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1abe00640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe1ac145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3449829442 0 0) 0x7fe194003520 con 0x7fe1ac146080 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 msgr2=0x7fe1ac145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe1ac145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 >> v1:172.21.15.187:6789/0 conn(0x7fe1ac0cd2f0 legacy=0x7fe1ac0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac0ce870 con 0x7fe1ac146080 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe1a40019f0 con 0x7fe1ac146080 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 >> v1:172.21.15.67:6789/0 conn(0x7fe1ac146080 legacy=0x7fe1ac160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940039e0 0x7fe194003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe194004800 0x7fe194004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe194005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac0cfa60 con 0x7fe1940039e0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe19400bb20 con 0x7fe194004800 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe19400bd20 con 0x7fe1ac0cd6d0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.898+0000 7fe1b1000640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940039e0 0x7fe194003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1abe00640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe194004800 0x7fe194004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1a00640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe194005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1000640 1 -- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 msgr2=0x7fe194005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1000640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe194005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1000640 1 -- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe194004800 msgr2=0x7fe194004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1000640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe194004800 0x7fe194004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1000640 1 -- 172.21.15.67:0/4089629104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe194003520 con 0x7fe1940039e0 2024-04-17T07:15:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.899+0000 7fe1b1a00640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0cd6d0 0x7fe194005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.900+0000 7fe1b1000640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940039e0 0x7fe194003e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0001750 tx=0x7fe194005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fb76460a374d8a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.900+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0001370 con 0x7fe1940039e0 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.900+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a00052d0 con 0x7fe1940039e0 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1ab400640 1 -- 172.21.15.67:0/4089629104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0003830 con 0x7fe1940039e0 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 -- 172.21.15.67:0/4089629104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940039e0 msgr2=0x7fe194003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 --2- 172.21.15.67:0/4089629104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940039e0 0x7fe194003e00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0001750 tx=0x7fe194005830 comp rx=0 tx=0).stop 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b1a00640 1 -- 172.21.15.67:0/4089629104 reap_dead start 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 -- 172.21.15.67:0/4089629104 shutdown_connections 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 -- 172.21.15.67:0/4089629104 >> 172.21.15.67:0/4089629104 conn(0x7fe1ac06b8c0 msgr2=0x7fe1ac06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 -- 172.21.15.67:0/4089629104 shutdown_connections 2024-04-17T07:15:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.901+0000 7fe1b2400640 1 -- 172.21.15.67:0/4089629104 wait complete. 2024-04-17T07:15:01.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.903+0000 7fe1b2400640 1 Processor -- start 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.903+0000 7fe1b2400640 1 -- start start 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0afab0 0x7fe1ac1310d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 0x7fe1ac0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac160a70 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ac0cee30 con 0x7fe1ac12fd50 2024-04-17T07:15:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.904+0000 7fe1b2400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1ac160bf0 con 0x7fe1ac0afab0 2024-04-17T07:15:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51834/0 (socket says 172.21.15.67:51834) 2024-04-17T07:15:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 -- 172.21.15.67:0/970716772 learned_addr learned my addr 172.21.15.67:0/970716772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1a00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 0x7fe1ac0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 -- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0afab0 msgr2=0x7fe1ac1310d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1abe00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0afab0 0x7fe1ac1310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0afab0 0x7fe1ac1310d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 -- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 msgr2=0x7fe1ac0b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 0x7fe1ac0b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1000640 1 -- 172.21.15.67:0/970716772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ac12ecb0 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.905+0000 7fe1b1a00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 0x7fe1ac0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1b1000640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0005040 tx=0x7fe1a0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0041020 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a0005480 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0001370 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ac12ef90 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.906+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ac12f550 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.908+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1ac068a10 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.909+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1a0044020 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.909+0000 7fe1a8c00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 0x7fe18c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.910+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5473+0+0 (secure 0 0 0) 0x7fe1a00ad3d0 con 0x7fe1ac0af6d0 2024-04-17T07:15:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.910+0000 7fe1abe00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 0x7fe18c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:01.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.912+0000 7fe1abe00640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 0x7fe18c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b0670 tx=0x7fe198008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:01.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:01.914+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1a004b030 con 0x7fe1ac0af6d0 2024-04-17T07:15:02.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.121+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe1ac068610 con 0x7fe1ac0af6d0 2024-04-17T07:15:02.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.655+0000 7fe1a8c00640 1 -- 172.21.15.67:0/970716772 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) ==== 169+0+0 (secure 0 0 0) 0x7fe1a00303c0 con 0x7fe1ac0af6d0 2024-04-17T07:15:02.656 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:15:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.660+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 msgr2=0x7fe18c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.660+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 0x7fe18c06edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b0670 tx=0x7fe198008040 comp rx=0 tx=0).stop 2024-04-17T07:15:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.660+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 msgr2=0x7fe1ac12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.660+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0005040 tx=0x7fe1a0001120 comp rx=0 tx=0).stop 2024-04-17T07:15:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 shutdown_connections 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1ac12fd50 0x7fe1ac0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe18c06c8f0 0x7fe18c06edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1ac0afab0 0x7fe1ac1310d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 --2- 172.21.15.67:0/970716772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1ac0af6d0 0x7fe1ac12e5f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.662+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 >> 172.21.15.67:0/970716772 conn(0x7fe1ac06b8c0 msgr2=0x7fe1ac074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.663+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 shutdown_connections 2024-04-17T07:15:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.663+0000 7fe1b2400640 1 -- 172.21.15.67:0/970716772 wait complete. 2024-04-17T07:15:02.687 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:15:02.687 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:15:02.687 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:15:02.687 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:15:02.687 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:15:02.687 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:15:02.688 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:15:02.688 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:15:02.688 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:15:02.688 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:15:02.688 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:15:02.688 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:15:02.688 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:15:02.689 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:15:02.689 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:15:02.689 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:15:02.709 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:15:02.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:15:02.711 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:15:02.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:15:02.712 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:15:02.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:15:02.713 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:15:02.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:15:02.756 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:15:02.756 DEBUG:teuthology.parallel:result is None 2024-04-17T07:15:02.759 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:15:02.759 DEBUG:teuthology.parallel:result is None 2024-04-17T07:15:02.760 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:15:02.760 DEBUG:teuthology.parallel:result is None 2024-04-17T07:15:02.765 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:15:02.765 DEBUG:teuthology.parallel:result is None 2024-04-17T07:15:02.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:15:02.920 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:15:02.921 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:15:02.931 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:15:02.932 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:15:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.971+0000 7faf52200640 1 Processor -- start 2024-04-17T07:15:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.971+0000 7faf52200640 1 -- start start 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf52200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 0x7faf4c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf52200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faf4c1634a0 0x7faf4c163930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf52200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf4c166c30 con 0x7faf4c0cffe0 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf52200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faf4c06fde0 con 0x7faf4c1634a0 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf52200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faf4c163e70 con 0x7faf4c071f30 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf4b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faf4c0cffe0 0x7faf4c167110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45390/0 (socket says 172.21.15.67:45390) 2024-04-17T07:15:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf4b400640 1 -- 172.21.15.67:0/2504939102 learned_addr learned my addr 172.21.15.67:0/2504939102 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf43e00640 1 --2- 172.21.15.67:0/2504939102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 0x7faf4c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf4be00640 1 --2- 172.21.15.67:0/2504939102 >> v2:172.21.15.187:3300/0 conn(0x7faf4c1634a0 0x7faf4c163930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035926972 0 0) 0x7faf4c166c30 con 0x7faf4c0cffe0 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.973+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf30003520 con 0x7faf4c0cffe0 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.974+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2636875328 0 0) 0x7faf30003520 con 0x7faf4c0cffe0 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.974+0000 7faf43e00640 1 -- 172.21.15.67:0/2504939102 >> v2:172.21.15.187:3300/0 conn(0x7faf4c1634a0 msgr2=0x7faf4c163930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.974+0000 7faf43e00640 1 --2- 172.21.15.67:0/2504939102 >> v2:172.21.15.187:3300/0 conn(0x7faf4c1634a0 0x7faf4c163930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.974+0000 7faf43e00640 1 -- 172.21.15.67:0/2504939102 >> v1:172.21.15.67:6789/0 conn(0x7faf4c0cffe0 legacy=0x7faf4c167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.974+0000 7faf43e00640 1 -- 172.21.15.67:0/2504939102 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4c06f820 con 0x7faf4c071f30 2024-04-17T07:15:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.975+0000 7faf43e00640 1 --2- 172.21.15.67:0/2504939102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 0x7faf4c06dd10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf340097a0 tx=0x7faf3402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebb8dde73d011db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.975+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34037070 con 0x7faf4c071f30 2024-04-17T07:15:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf34030510 con 0x7faf4c071f30 2024-04-17T07:15:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf4aa00640 1 -- 172.21.15.67:0/2504939102 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c001670 con 0x7faf4c071f30 2024-04-17T07:15:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 -- 172.21.15.67:0/2504939102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 msgr2=0x7faf4c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 --2- 172.21.15.67:0/2504939102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 0x7faf4c06dd10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf340097a0 tx=0x7faf3402b790 comp rx=0 tx=0).stop 2024-04-17T07:15:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 -- 172.21.15.67:0/2504939102 shutdown_connections 2024-04-17T07:15:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 --2- 172.21.15.67:0/2504939102 >> v2:172.21.15.187:3300/0 conn(0x7faf4c1634a0 0x7faf4c163930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 --2- 172.21.15.67:0/2504939102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c071f30 0x7faf4c06dd10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 -- 172.21.15.67:0/2504939102 >> 172.21.15.67:0/2504939102 conn(0x7faf4c067ee0 msgr2=0x7faf4c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 -- 172.21.15.67:0/2504939102 shutdown_connections 2024-04-17T07:15:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.976+0000 7faf52200640 1 -- 172.21.15.67:0/2504939102 wait complete. 2024-04-17T07:15:02.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.978+0000 7faf52200640 1 Processor -- start 2024-04-17T07:15:02.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.978+0000 7faf52200640 1 -- start start 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf4c071f30 0x7faf4c165fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c0cffe0 0x7faf4c0ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c167820 con 0x7faf4c1634a0 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c06f4f0 con 0x7faf4c071f30 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf52200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faf4c0aecf0 con 0x7faf4c0cffe0 2024-04-17T07:15:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.979+0000 7faf4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51886/0 (socket says 172.21.15.67:51886) 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 -- 172.21.15.67:0/3724674146 learned_addr learned my addr 172.21.15.67:0/3724674146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf43e00640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c0cffe0 0x7faf4c0ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4b400640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf4c071f30 0x7faf4c165fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 -- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c0cffe0 msgr2=0x7faf4c0ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c0cffe0 0x7faf4c0ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 -- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf4c071f30 msgr2=0x7faf4c165fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf4c071f30 0x7faf4c165fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 -- 172.21.15.67:0/3724674146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4c0af230 con 0x7faf4c1634a0 2024-04-17T07:15:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.980+0000 7faf4be00640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf4400efc0 tx=0x7faf4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.981+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44018020 con 0x7faf4c1634a0 2024-04-17T07:15:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.981+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf44014480 con 0x7faf4c1634a0 2024-04-17T07:15:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.981+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf44014870 con 0x7faf4c1634a0 2024-04-17T07:15:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.981+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4c0af510 con 0x7faf4c1634a0 2024-04-17T07:15:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.981+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4c0ae8c0 con 0x7faf4c1634a0 2024-04-17T07:15:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.982+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf4c0af030 con 0x7faf4c1634a0 2024-04-17T07:15:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.983+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf44005d10 con 0x7faf4c1634a0 2024-04-17T07:15:02.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.984+0000 7faf43400640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 0x7faf2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:02.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.984+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4943+0+0 (secure 0 0 0) 0x7faf4408c1e0 con 0x7faf4c1634a0 2024-04-17T07:15:02.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.985+0000 7faf4b400640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 0x7faf2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.986+0000 7faf4b400640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 0x7faf2806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faf3c001eb0 tx=0x7faf3c0016d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:02.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:02.989+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faf44022070 con 0x7faf4c1634a0 2024-04-17T07:15:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.211+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7faf4c0b35a0 con 0x7faf4c1634a0 2024-04-17T07:15:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.690+0000 7faf43400640 1 -- 172.21.15.67:0/3724674146 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) ==== 97+0+0 (secure 0 0 0) 0x7faf44010040 con 0x7faf4c1634a0 2024-04-17T07:15:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:15:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 msgr2=0x7faf2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 0x7faf2806edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faf3c001eb0 tx=0x7faf3c0016d0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 msgr2=0x7faf4c165310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf4400efc0 tx=0x7faf4400c490 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 shutdown_connections 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c1634a0 0x7faf4c165310 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7faf2806c8f0 0x7faf2806edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf4c0cffe0 0x7faf4c0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.695+0000 7faf52200640 1 --2- 172.21.15.67:0/3724674146 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf4c071f30 0x7faf4c165fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.696+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 >> 172.21.15.67:0/3724674146 conn(0x7faf4c067ee0 msgr2=0x7faf4c0b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.696+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 shutdown_connections 2024-04-17T07:15:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.696+0000 7faf52200640 1 -- 172.21.15.67:0/3724674146 wait complete. 2024-04-17T07:15:03.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:15:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.930+0000 7f304f200640 1 Processor -- start 2024-04-17T07:15:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.930+0000 7f304f200640 1 -- start start 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3050072470 0x7f30501675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f305015cd40 con 0x7f30501452f0 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f305015d020 con 0x7f3050160e50 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f305015d670 con 0x7f3050072470 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30501452f0 0x7f3050145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48098/0 (socket says 172.21.15.67:48098) 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3050072470 0x7f30501675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.932+0000 7f304de00640 1 -- 172.21.15.67:0/3934403353 learned_addr learned my addr 172.21.15.67:0/3934403353 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.933+0000 7f304e800640 1 -- 172.21.15.67:0/3934403353 >> v1:172.21.15.187:6789/0 conn(0x7f3050160e50 legacy=0x7f3050071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.933+0000 7f304e800640 1 -- 172.21.15.67:0/3934403353 >> v1:172.21.15.67:6789/0 conn(0x7f30501452f0 legacy=0x7f3050145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.933+0000 7f304e800640 1 -- 172.21.15.67:0/3934403353 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305015d300 con 0x7f3050072470 2024-04-17T07:15:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.934+0000 7f304e800640 1 --2- 172.21.15.67:0/3934403353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3050072470 0x7f30501675b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30480097a0 tx=0x7f304802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2392089604992f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.934+0000 7f304ca00640 1 -- 172.21.15.67:0/3934403353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30480315a0 con 0x7f3050072470 2024-04-17T07:15:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.934+0000 7f304ca00640 1 -- 172.21.15.67:0/3934403353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3048037070 con 0x7f3050072470 2024-04-17T07:15:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.934+0000 7f304ca00640 1 -- 172.21.15.67:0/3934403353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048031960 con 0x7f3050072470 2024-04-17T07:15:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 -- 172.21.15.67:0/3934403353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3050072470 msgr2=0x7f30501675b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 --2- 172.21.15.67:0/3934403353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3050072470 0x7f30501675b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30480097a0 tx=0x7f304802b840 comp rx=0 tx=0).stop 2024-04-17T07:15:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 -- 172.21.15.67:0/3934403353 shutdown_connections 2024-04-17T07:15:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 --2- 172.21.15.67:0/3934403353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3050072470 0x7f30501675b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 -- 172.21.15.67:0/3934403353 >> 172.21.15.67:0/3934403353 conn(0x7f30500b5f70 msgr2=0x7f305006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 -- 172.21.15.67:0/3934403353 shutdown_connections 2024-04-17T07:15:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.935+0000 7f304f200640 1 -- 172.21.15.67:0/3934403353 wait complete. 2024-04-17T07:15:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.938+0000 7f304f200640 1 Processor -- start 2024-04-17T07:15:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.938+0000 7f304f200640 1 -- start start 2024-04-17T07:15:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3050072470 0x7f30501f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3050160e50 0x7f30501e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f305015eb10 con 0x7f30501452f0 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f30501456d0 con 0x7f3050160e50 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.939+0000 7f304f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f305015ec90 con 0x7f3050072470 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42230/0 (socket says 172.21.15.67:42230) 2024-04-17T07:15:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 -- 172.21.15.67:0/1835463338 learned_addr learned my addr 172.21.15.67:0/1835463338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304de00640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3050072470 0x7f30501f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304e800640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3050160e50 0x7f30501e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 -- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3050072470 msgr2=0x7f30501f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3050072470 0x7f30501f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 -- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3050160e50 msgr2=0x7f30501e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3050160e50 0x7f30501e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.940+0000 7f304d400640 1 -- 172.21.15.67:0/1835463338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30500b3aa0 con 0x7f30501452f0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f304d400640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f303c00b3f0 tx=0x7f303c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c015020 con 0x7f30501452f0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c011830 con 0x7f30501452f0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30500b34e0 con 0x7f30501452f0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c011c20 con 0x7f30501452f0 2024-04-17T07:15:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.941+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30501e6270 con 0x7f30501452f0 2024-04-17T07:15:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.942+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3050068a10 con 0x7f30501452f0 2024-04-17T07:15:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.943+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f303c018020 con 0x7f30501452f0 2024-04-17T07:15:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.945+0000 7f3046000640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 0x7f303006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.945+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f303c08b540 con 0x7f30501452f0 2024-04-17T07:15:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.946+0000 7f304de00640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 0x7f303006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.947+0000 7f304de00640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 0x7f303006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f30400012e0 tx=0x7f3040009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:03.949+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f303c01f030 con 0x7f30501452f0 2024-04-17T07:15:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.227+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f305006f9a0 con 0x7f30501452f0 2024-04-17T07:15:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.229+0000 7f3046000640 1 -- 172.21.15.67:0/1835463338 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) ==== 80+0+6645 (secure 0 0 0) 0x7f303c0172c0 con 0x7f30501452f0 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:15:04.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.232 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.236 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:15:04.237 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:15:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:15:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:15:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mds", 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9mxObLBRAAXNiqkVyPbBRlMgbO5k5zyg==", 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mds" 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mgr", 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9mSvLLBRAAzpIhTuhhzIYP6O2PKKffsg==", 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-osd", 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9m7frLBRAAmAT28tUkZEWDiLwhr3YbBw==", 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-osd" 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd", 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9mSQPMBRAAl0oih5fL6vdNFuppSUckfQ==", 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-17T07:15:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9mWQzMBRAAj8y92bqdiYFu4ve6Cr97rQ==", 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rgw", 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCidh9mbhXMBRAADdE+2R8HMLd/C/KqMb85aQ==", 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:15:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:15:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:15:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:15:04.245 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:15:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 msgr2=0x7f303006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 0x7f303006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f30400012e0 tx=0x7f3040009040 comp rx=0 tx=0).stop 2024-04-17T07:15:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 msgr2=0x7f30500b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f303c00b3f0 tx=0x7f303c00b8b0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 shutdown_connections 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3050160e50 0x7f30501e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30501452f0 0x7f30500b2b70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f303006c8f0 0x7f303006edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 --2- 172.21.15.67:0/1835463338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3050072470 0x7f30501f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.231+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 >> 172.21.15.67:0/1835463338 conn(0x7f30500b5f70 msgr2=0x7f305006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.232+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 shutdown_connections 2024-04-17T07:15:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.232+0000 7f304f200640 1 -- 172.21.15.67:0/1835463338 wait complete. 2024-04-17T07:15:04.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-17T07:15:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.481+0000 7fc7f6600640 1 Processor -- start 2024-04-17T07:15:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.481+0000 7fc7f6600640 1 -- start start 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f00b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7f00b4dc0 0x7fc7f006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc7f006f3d0 0x7fc7f015bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc7f006f820 con 0x7fc7f00b4dc0 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc7f00b3b50 con 0x7fc7f006f3d0 2024-04-17T07:15:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7f6600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f00d0000 con 0x7fc7f01450c0 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7f00b4dc0 0x7fc7f006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7f00b4dc0 0x7fc7f006ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42236/0 (socket says 172.21.15.67:42236) 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.483+0000 7fc7efe00640 1 -- 172.21.15.67:0/2599399132 learned_addr learned my addr 172.21.15.67:0/2599399132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7f5200640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f00b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7f5c00640 1 --2- 172.21.15.67:0/2599399132 >> v2:172.21.15.187:3300/0 conn(0x7fc7f006f3d0 0x7fc7f015bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7efe00640 1 -- 172.21.15.67:0/2599399132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 msgr2=0x7fc7f00b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7efe00640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f00b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7efe00640 1 -- 172.21.15.67:0/2599399132 >> v2:172.21.15.187:3300/0 conn(0x7fc7f006f3d0 msgr2=0x7fc7f015bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7efe00640 1 --2- 172.21.15.67:0/2599399132 >> v2:172.21.15.187:3300/0 conn(0x7fc7f006f3d0 0x7fc7f015bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.484+0000 7fc7efe00640 1 -- 172.21.15.67:0/2599399132 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f00b3da0 con 0x7fc7f00b4dc0 2024-04-17T07:15:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.485+0000 7fc7efe00640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f00b4dc0 0x7fc7f006ee90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc7f006ff10 tx=0x7fc7dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6209a4c74651b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7ef400640 1 -- 172.21.15.67:0/2599399132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc02e070 con 0x7fc7f00b4dc0 2024-04-17T07:15:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7ef400640 1 -- 172.21.15.67:0/2599399132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7dc037070 con 0x7fc7f00b4dc0 2024-04-17T07:15:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7ef400640 1 -- 172.21.15.67:0/2599399132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031a00 con 0x7fc7f00b4dc0 2024-04-17T07:15:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 -- 172.21.15.67:0/2599399132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f00b4dc0 msgr2=0x7fc7f006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f00b4dc0 0x7fc7f006ee90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc7f006ff10 tx=0x7fc7dc02b880 comp rx=0 tx=0).stop 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 -- 172.21.15.67:0/2599399132 shutdown_connections 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 --2- 172.21.15.67:0/2599399132 >> v2:172.21.15.187:3300/0 conn(0x7fc7f006f3d0 0x7fc7f015bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f00b4dc0 0x7fc7f006ee90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 --2- 172.21.15.67:0/2599399132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.486+0000 7fc7f6600640 1 -- 172.21.15.67:0/2599399132 >> 172.21.15.67:0/2599399132 conn(0x7fc7f006b8c0 msgr2=0x7fc7f006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.487+0000 7fc7f6600640 1 -- 172.21.15.67:0/2599399132 shutdown_connections 2024-04-17T07:15:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.487+0000 7fc7f6600640 1 -- 172.21.15.67:0/2599399132 wait complete. 2024-04-17T07:15:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.488+0000 7fc7f6600640 1 Processor -- start 2024-04-17T07:15:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.489+0000 7fc7f6600640 1 -- start start 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.489+0000 7fc7f6600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7f006f3d0 0x7fc7f01f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.489+0000 7fc7f6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f0142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f015c080 con 0x7fc7f0143090 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f6600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f00b4360 con 0x7fc7f006f3d0 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f6600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f015c200 con 0x7fc7f01450c0 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42240/0 (socket says 172.21.15.67:42240) 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 -- 172.21.15.67:0/494241129 learned_addr learned my addr 172.21.15.67:0/494241129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5200640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7f006f3d0 0x7fc7f01f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7efe00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f0142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 -- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 msgr2=0x7fc7f0142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f0142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 -- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7f006f3d0 msgr2=0x7fc7f01f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7f006f3d0 0x7fc7f01f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7f5c00640 1 -- 172.21.15.67:0/494241129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f0135390 con 0x7fc7f0143090 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.490+0000 7fc7efe00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f0142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7f5c00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8007e60 tx=0x7fc7e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8014620 con 0x7fc7f0143090 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e8019070 con 0x7fc7f0143090 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8014a20 con 0x7fc7f0143090 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f0143c80 con 0x7fc7f0143090 2024-04-17T07:15:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.491+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f01439a0 con 0x7fc7f0143090 2024-04-17T07:15:04.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.492+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f0068a10 con 0x7fc7f0143090 2024-04-17T07:15:04.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.493+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc7e8014bc0 con 0x7fc7f0143090 2024-04-17T07:15:04.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.494+0000 7fc7ecc00640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 0x7fc7d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:04.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.494+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7fc7e808c080 con 0x7fc7f0143090 2024-04-17T07:15:04.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.495+0000 7fc7f5200640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 0x7fc7d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.496+0000 7fc7f5200640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 0x7fc7d006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4006fd0 tx=0x7fc7e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:04.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.499+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc7e8011260 con 0x7fc7f0143090 2024-04-17T07:15:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.784+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fc7f006ebf0 con 0x7fc7f0143090 2024-04-17T07:15:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.788+0000 7fc7ecc00640 1 -- 172.21.15.67:0/494241129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) ==== 90+0+0 (secure 0 0 0) 0x7fc7e805a760 con 0x7fc7f0143090 2024-04-17T07:15:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.791+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 msgr2=0x7fc7d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 0x7fc7d006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4006fd0 tx=0x7fc7e4008040 comp rx=0 tx=0).stop 2024-04-17T07:15:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 msgr2=0x7fc7f01e62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8007e60 tx=0x7fc7e800e4b0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 shutdown_connections 2024-04-17T07:15:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7f0143090 0x7fc7f01e62a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc7d006c920 0x7fc7d006ede0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7f01450c0 0x7fc7f0142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 --2- 172.21.15.67:0/494241129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7f006f3d0 0x7fc7f01f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 >> 172.21.15.67:0/494241129 conn(0x7fc7f006b8c0 msgr2=0x7fc7f00aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 shutdown_connections 2024-04-17T07:15:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:04.792+0000 7fc7f6600640 1 -- 172.21.15.67:0/494241129 wait complete. 2024-04-17T07:15:04.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-17T07:15:05.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.030+0000 7f88a4e00640 1 Processor -- start 2024-04-17T07:15:05.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.030+0000 7f88a4e00640 1 -- start start 2024-04-17T07:15:05.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f88a4e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f88a0145eb0 0x7f88a00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f88a4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88a0072470 0x7f88a015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f88a4e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a0146290 con 0x7f88a0160e80 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f88a4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f88a0160150 con 0x7f88a0072470 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f88a4e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00cec50 con 0x7f88a0145eb0 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.032+0000 7f889fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88a0072470 0x7f88a015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88a0072470 0x7f88a015dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42244/0 (socket says 172.21.15.67:42244) 2024-04-17T07:15:05.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 -- 172.21.15.67:0/1831781419 learned_addr learned my addr 172.21.15.67:0/1831781419 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889f400640 1 --2- 172.21.15.67:0/1831781419 >> v2:172.21.15.187:3300/0 conn(0x7f88a0145eb0 0x7f88a00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 -- 172.21.15.67:0/1831781419 >> v2:172.21.15.187:3300/0 conn(0x7f88a0145eb0 msgr2=0x7f88a00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 --2- 172.21.15.67:0/1831781419 >> v2:172.21.15.187:3300/0 conn(0x7f88a0145eb0 0x7f88a00ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 -- 172.21.15.67:0/1831781419 >> v1:172.21.15.187:6789/0 conn(0x7f88a0160e80 legacy=0x7f88a0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.033+0000 7f889fe00640 1 -- 172.21.15.67:0/1831781419 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a01603a0 con 0x7f88a0072470 2024-04-17T07:15:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.034+0000 7f889fe00640 1 --2- 172.21.15.67:0/1831781419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a015dd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f88a006d620 tx=0x7f889802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eeb4782559ad497 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.035+0000 7f889ea00640 1 -- 172.21.15.67:0/1831781419 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889802ca80 con 0x7f88a0072470 2024-04-17T07:15:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.035+0000 7f889ea00640 1 -- 172.21.15.67:0/1831781419 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889802cc20 con 0x7f88a0072470 2024-04-17T07:15:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.035+0000 7f889ea00640 1 -- 172.21.15.67:0/1831781419 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88980343f0 con 0x7f88a0072470 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.035+0000 7f88a4e00640 1 -- 172.21.15.67:0/1831781419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 msgr2=0x7f88a015dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.035+0000 7f88a4e00640 1 --2- 172.21.15.67:0/1831781419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a015dd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f88a006d620 tx=0x7f889802bee0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 -- 172.21.15.67:0/1831781419 shutdown_connections 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 --2- 172.21.15.67:0/1831781419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a015dd20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 --2- 172.21.15.67:0/1831781419 >> v2:172.21.15.187:3300/0 conn(0x7f88a0145eb0 0x7f88a00ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 -- 172.21.15.67:0/1831781419 >> 172.21.15.67:0/1831781419 conn(0x7f88a006b8c0 msgr2=0x7f88a006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 -- 172.21.15.67:0/1831781419 shutdown_connections 2024-04-17T07:15:05.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.036+0000 7f88a4e00640 1 -- 172.21.15.67:0/1831781419 wait complete. 2024-04-17T07:15:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.037+0000 7f88a4e00640 1 Processor -- start 2024-04-17T07:15:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.037+0000 7f88a4e00640 1 -- start start 2024-04-17T07:15:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a0145eb0 0x7f88a0142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 0x7f88a015f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88a015da00 con 0x7f88a0072470 2024-04-17T07:15:05.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f88a0160960 con 0x7f88a0160e80 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.038+0000 7f88a4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00ce6d0 con 0x7f88a0145eb0 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42254/0 (socket says 172.21.15.67:42254) 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 -- 172.21.15.67:0/3733007209 learned_addr learned my addr 172.21.15.67:0/3733007209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889fe00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 0x7f88a015f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f8896a00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a0145eb0 0x7f88a0142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 -- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a0145eb0 msgr2=0x7f88a0142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a0145eb0 0x7f88a0142230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 -- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 msgr2=0x7f88a015f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 0x7f88a015f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889f400640 1 -- 172.21.15.67:0/3733007209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a01e85d0 con 0x7f88a0072470 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.039+0000 7f889fe00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 0x7f88a015f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:05.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f889f400640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f889000d830 tx=0x7f889000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889000b660 con 0x7f88a0072470 2024-04-17T07:15:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8890019070 con 0x7f88a0072470 2024-04-17T07:15:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889000ba20 con 0x7f88a0072470 2024-04-17T07:15:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a01e88b0 con 0x7f88a0072470 2024-04-17T07:15:05.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.040+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a01e8e70 con 0x7f88a0072470 2024-04-17T07:15:05.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.041+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a0068a10 con 0x7f88a0072470 2024-04-17T07:15:05.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.042+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f889001e460 con 0x7f88a0072470 2024-04-17T07:15:05.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.043+0000 7f8897e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 0x7f887c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.043+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f889008be90 con 0x7f88a0072470 2024-04-17T07:15:05.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.044+0000 7f8896a00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 0x7f887c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.045+0000 7f8896a00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 0x7f887c06edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.048+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f889005a600 con 0x7f88a0072470 2024-04-17T07:15:05.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.328+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f88a015c650 con 0x7f88a0072470 2024-04-17T07:15:05.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.332+0000 7f8897e00640 1 -- 172.21.15.67:0/3733007209 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7f889008d0a0 con 0x7f88a0072470 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.334+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 msgr2=0x7f887c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 0x7f887c06edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).stop 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 msgr2=0x7f88a01f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f889000d830 tx=0x7f889000dcf0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 shutdown_connections 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88a0160e80 0x7f88a015f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f887c06c8f0 0x7f887c06edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a0145eb0 0x7f88a0142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 --2- 172.21.15.67:0/3733007209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0072470 0x7f88a01f4a20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 >> 172.21.15.67:0/3733007209 conn(0x7f88a006b8c0 msgr2=0x7f88a00b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:05.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 shutdown_connections 2024-04-17T07:15:05.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.335+0000 7f88a4e00640 1 -- 172.21.15.67:0/3733007209 wait complete. 2024-04-17T07:15:05.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-17T07:15:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.564+0000 7f1569a00640 1 Processor -- start 2024-04-17T07:15:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.564+0000 7f1569a00640 1 -- start start 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.565+0000 7f1569a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 0x7f15640d4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f1569a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15640d2ec0 con 0x7f15640d52e0 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f1569a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15640d31a0 con 0x7f15640d15e0 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f1569a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15640d0590 con 0x7f156415e980 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f155b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15640d52e0 0x7f15640d0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48102/0 (socket says 172.21.15.67:48102) 2024-04-17T07:15:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f155b400640 1 -- 172.21.15.67:0/3121501104 learned_addr learned my addr 172.21.15.67:0/3121501104 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.566+0000 7f1563e00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 0x7f15640d4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421426319 0 0) 0x7f15640d2ec0 con 0x7f15640d52e0 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1548003520 con 0x7f15640d52e0 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3071864981 0 0) 0x7f1548003520 con 0x7f15640d52e0 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 msgr2=0x7f15640d4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 0x7f15640d4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 >> v1:172.21.15.187:6789/0 conn(0x7f15640d15e0 legacy=0x7f156406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15640d3480 con 0x7f15640d52e0 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f154c001670 con 0x7f15640d52e0 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 >> v1:172.21.15.67:6789/0 conn(0x7f15640d52e0 legacy=0x7f15640d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15480039e0 0x7f1548003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004390 0x7f1548004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 0x7f15480050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f154800b9c0 con 0x7f15480039e0 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f154800bb40 con 0x7f1548004390 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f154800bd40 con 0x7f156415e980 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.567+0000 7f1563e00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15480039e0 0x7f1548003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.568+0000 7f1563e00640 1 -- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 msgr2=0x7f15480050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.568+0000 7f1563e00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156415e980 0x7f15480050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.568+0000 7f1563e00640 1 -- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004390 msgr2=0x7f1548004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.568+0000 7f1563e00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004390 0x7f1548004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.568+0000 7f1563e00640 1 -- 172.21.15.67:0/3121501104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1548003520 con 0x7f15480039e0 2024-04-17T07:15:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.569+0000 7f1563e00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15480039e0 0x7f1548003dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1554009fc0 tx=0x7f1548005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb64fe2539422949 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.569+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554039070 con 0x7f15480039e0 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.569+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155402cc60 con 0x7f15480039e0 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.569+0000 7f1563400640 1 -- 172.21.15.67:0/3121501104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554009c10 con 0x7f15480039e0 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 -- 172.21.15.67:0/3121501104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15480039e0 msgr2=0x7f1548003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 --2- 172.21.15.67:0/3121501104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15480039e0 0x7f1548003dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1554009fc0 tx=0x7f1548005830 comp rx=0 tx=0).stop 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569000640 1 -- 172.21.15.67:0/3121501104 reap_dead start 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 -- 172.21.15.67:0/3121501104 shutdown_connections 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 -- 172.21.15.67:0/3121501104 >> 172.21.15.67:0/3121501104 conn(0x7f156408bff0 msgr2=0x7f15640d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 -- 172.21.15.67:0/3121501104 shutdown_connections 2024-04-17T07:15:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.570+0000 7f1569a00640 1 -- 172.21.15.67:0/3121501104 wait complete. 2024-04-17T07:15:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.572+0000 7f1569a00640 1 Processor -- start 2024-04-17T07:15:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.572+0000 7f1569a00640 1 -- start start 2024-04-17T07:15:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 0x7f15641f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 0x7f15641f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15640d2c70 con 0x7f15641f4cb0 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f156415ed60 con 0x7f156415a870 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f156406cd60 con 0x7f15480039e0 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.573+0000 7f1569000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42270/0 (socket says 172.21.15.67:42270) 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 -- 172.21.15.67:0/1203692297 learned_addr learned my addr 172.21.15.67:0/1203692297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1563e00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 0x7f15641f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 -- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 msgr2=0x7f15641f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 0x7f15641f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 -- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 msgr2=0x7f15641f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f155b400640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 0x7f15641f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 0x7f15641f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 -- 172.21.15.67:0/1203692297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15641f1e40 con 0x7f15641f4cb0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1563e00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 0x7f15641f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f155b400640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 0x7f15641f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.574+0000 7f1569000640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f155c00c390 tx=0x7f155c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.575+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c007520 con 0x7f15641f4cb0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.575+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f155c0057c0 con 0x7f15641f4cb0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.575+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c005bb0 con 0x7f15641f4cb0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.575+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15641f8260 con 0x7f15641f4cb0 2024-04-17T07:15:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.575+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15641f8820 con 0x7f15641f4cb0 2024-04-17T07:15:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.576+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156404fd90 con 0x7f15641f4cb0 2024-04-17T07:15:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.577+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f155c011070 con 0x7f15641f4cb0 2024-04-17T07:15:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.578+0000 7f1560c00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 0x7f154006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.578+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f155c08a9c0 con 0x7f15641f4cb0 2024-04-17T07:15:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.579+0000 7f1563e00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 0x7f154006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.580+0000 7f1563e00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 0x7f154006ee30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f80 tx=0x7f155402e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:05.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.583+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f156404fd90 con 0x7f15641f4cb0 2024-04-17T07:15:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.867+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f15640b6f50 con 0x7f15641f4cb0 2024-04-17T07:15:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.872+0000 7f1560c00640 1 -- 172.21.15.67:0/1203692297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f155c019060 con 0x7f15641f4cb0 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 msgr2=0x7f154006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 0x7f154006ee30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1554000f80 tx=0x7f155402e000 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 msgr2=0x7f15640ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f155c00c390 tx=0x7f155c00c790 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 shutdown_connections 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15641f4cb0 0x7f15640ccbb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f156415a870 0x7f15641f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f154006c970 0x7f154006ee30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.874+0000 7f1569a00640 1 --2- 172.21.15.67:0/1203692297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15480039e0 0x7f15641f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.875+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 >> 172.21.15.67:0/1203692297 conn(0x7f156408bff0 msgr2=0x7f15640b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.875+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 shutdown_connections 2024-04-17T07:15:05.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:05.875+0000 7f1569a00640 1 -- 172.21.15.67:0/1203692297 wait complete. 2024-04-17T07:15:05.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-17T07:15:06.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.121+0000 7fc01e200640 1 Processor -- start 2024-04-17T07:15:06.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.121+0000 7fc01e200640 1 -- start start 2024-04-17T07:15:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01e200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc018145eb0 0x7fc0180ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc018072470 0x7fc01815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc018146290 con 0x7fc018160e80 2024-04-17T07:15:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc018160150 con 0x7fc018072470 2024-04-17T07:15:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01e200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc0180cec50 con 0x7fc018145eb0 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc01d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc018072470 0x7fc01815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc017e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc018160e80 0x7fc018071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48108/0 (socket says 172.21.15.67:48108) 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.123+0000 7fc017e00640 1 -- 172.21.15.67:0/3589665478 learned_addr learned my addr 172.21.15.67:0/3589665478 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc01ce00640 1 --2- 172.21.15.67:0/3589665478 >> v2:172.21.15.187:3300/0 conn(0x7fc018145eb0 0x7fc0180ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc017400640 1 -- 172.21.15.67:0/3589665478 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022566105 0 0) 0x7fc018146290 con 0x7fc018160e80 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc017400640 1 -- 172.21.15.67:0/3589665478 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc000003520 con 0x7fc018160e80 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc01d800640 1 -- 172.21.15.67:0/3589665478 >> v2:172.21.15.187:3300/0 conn(0x7fc018145eb0 msgr2=0x7fc0180ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc01d800640 1 --2- 172.21.15.67:0/3589665478 >> v2:172.21.15.187:3300/0 conn(0x7fc018145eb0 0x7fc0180ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc01d800640 1 -- 172.21.15.67:0/3589665478 >> v1:172.21.15.67:6789/0 conn(0x7fc018160e80 legacy=0x7fc018071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.124+0000 7fc01d800640 1 -- 172.21.15.67:0/3589665478 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0181603a0 con 0x7fc018072470 2024-04-17T07:15:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.125+0000 7fc01d800640 1 --2- 172.21.15.67:0/3589665478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01815dd20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc01806d620 tx=0x7fc01002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd3a3b69ff677cc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.125+0000 7fc017400640 1 -- 172.21.15.67:0/3589665478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010039070 con 0x7fc018072470 2024-04-17T07:15:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.125+0000 7fc017400640 1 -- 172.21.15.67:0/3589665478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc01002cc70 con 0x7fc018072470 2024-04-17T07:15:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.125+0000 7fc017400640 1 -- 172.21.15.67:0/3589665478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010034660 con 0x7fc018072470 2024-04-17T07:15:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 -- 172.21.15.67:0/3589665478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 msgr2=0x7fc01815dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 --2- 172.21.15.67:0/3589665478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01815dd20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc01806d620 tx=0x7fc01002bec0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 -- 172.21.15.67:0/3589665478 shutdown_connections 2024-04-17T07:15:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 --2- 172.21.15.67:0/3589665478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01815dd20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 --2- 172.21.15.67:0/3589665478 >> v2:172.21.15.187:3300/0 conn(0x7fc018145eb0 0x7fc0180ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 -- 172.21.15.67:0/3589665478 >> 172.21.15.67:0/3589665478 conn(0x7fc01806b8c0 msgr2=0x7fc01806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 -- 172.21.15.67:0/3589665478 shutdown_connections 2024-04-17T07:15:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.126+0000 7fc01e200640 1 -- 172.21.15.67:0/3589665478 wait complete. 2024-04-17T07:15:06.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.128+0000 7fc01e200640 1 Processor -- start 2024-04-17T07:15:06.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.128+0000 7fc01e200640 1 -- start start 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc018145eb0 0x7fc01815ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc018160e80 0x7fc0181445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc01815da00 con 0x7fc018072470 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc018160960 con 0x7fc018160e80 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0180ce6d0 con 0x7fc018145eb0 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42296/0 (socket says 172.21.15.67:42296) 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.129+0000 7fc01ce00640 1 -- 172.21.15.67:0/116300565 learned_addr learned my addr 172.21.15.67:0/116300565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01d800640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc018160e80 0x7fc0181445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 -- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc018145eb0 msgr2=0x7fc01815ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc018145eb0 0x7fc01815ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 -- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc018160e80 msgr2=0x7fc0181445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc018160e80 0x7fc0181445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 -- 172.21.15.67:0/116300565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0180a9950 con 0x7fc018072470 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01ce00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00d800 tx=0x7fc00c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00b6c0 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c019070 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00ba80 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.130+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0181426f0 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.131+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01815f340 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.132+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc018068a10 con 0x7fc018072470 2024-04-17T07:15:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.133+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc00c00bc20 con 0x7fc018072470 2024-04-17T07:15:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.134+0000 7fc014c00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 0x7fbff806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.134+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7fc00c08bc60 con 0x7fc018072470 2024-04-17T07:15:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.135+0000 7fc017e00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 0x7fbff806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.136+0000 7fc017e00640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 0x7fbff806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc004000f60 tx=0x7fc0040078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.138+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc00c01c360 con 0x7fc018072470 2024-04-17T07:15:06.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.421+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fc01815ea60 con 0x7fc018072470 2024-04-17T07:15:06.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.425+0000 7fc014c00640 1 -- 172.21.15.67:0/116300565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fc00c05a470 con 0x7fc018072470 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 msgr2=0x7fbff806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 0x7fbff806edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc004000f60 tx=0x7fc0040078f0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 msgr2=0x7fc01814fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00d800 tx=0x7fc00c00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 shutdown_connections 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc018160e80 0x7fc0181445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbff806c8f0 0x7fbff806edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc018145eb0 0x7fc01815ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 --2- 172.21.15.67:0/116300565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc018072470 0x7fc01814fa30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.427+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 >> 172.21.15.67:0/116300565 conn(0x7fc01806b8c0 msgr2=0x7fc0180b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.428+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 shutdown_connections 2024-04-17T07:15:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.428+0000 7fc01e200640 1 -- 172.21.15.67:0/116300565 wait complete. 2024-04-17T07:15:06.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-17T07:15:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.644+0000 7feb06200640 1 Processor -- start 2024-04-17T07:15:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.645+0000 7feb06200640 1 -- start start 2024-04-17T07:15:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.646+0000 7feb06200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.646+0000 7feb06200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb000d3350 0x7feb000b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.646+0000 7feb06200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb000d4140 con 0x7feb0015ee20 2024-04-17T07:15:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.646+0000 7feb06200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7feb000b83b0 con 0x7feb000d3350 2024-04-17T07:15:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.646+0000 7feb06200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb000d3b80 con 0x7feb000d1ab0 2024-04-17T07:15:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.647+0000 7feaffe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.647+0000 7feaffe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:53562/0 (socket says 172.21.15.67:53562) 2024-04-17T07:15:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.647+0000 7feaffe00640 1 -- 172.21.15.67:0/1423993354 learned_addr learned my addr 172.21.15.67:0/1423993354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.647+0000 7feb05800640 1 --2- 172.21.15.67:0/1423993354 >> v2:172.21.15.187:3300/0 conn(0x7feb000d3350 0x7feb000b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.648+0000 7feaffe00640 1 -- 172.21.15.67:0/1423993354 >> v2:172.21.15.187:3300/0 conn(0x7feb000d3350 msgr2=0x7feb000b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.648+0000 7feaffe00640 1 --2- 172.21.15.67:0/1423993354 >> v2:172.21.15.187:3300/0 conn(0x7feb000d3350 0x7feb000b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.648+0000 7feaffe00640 1 -- 172.21.15.67:0/1423993354 >> v1:172.21.15.187:6789/0 conn(0x7feb0015ee20 legacy=0x7feb000d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.648+0000 7feaffe00640 1 -- 172.21.15.67:0/1423993354 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb000d3e60 con 0x7feb000d1ab0 2024-04-17T07:15:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.649+0000 7feaffe00640 1 --2- 172.21.15.67:0/1423993354 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feaec009870 tx=0x7feaec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff36774c24ce7989 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feaff400640 1 -- 172.21.15.67:0/1423993354 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec034070 con 0x7feb000d1ab0 2024-04-17T07:15:06.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feaff400640 1 -- 172.21.15.67:0/1423993354 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaec030840 con 0x7feb000d1ab0 2024-04-17T07:15:06.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feaff400640 1 -- 172.21.15.67:0/1423993354 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec030c10 con 0x7feb000d1ab0 2024-04-17T07:15:06.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 -- 172.21.15.67:0/1423993354 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 msgr2=0x7feb000d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 --2- 172.21.15.67:0/1423993354 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feaec009870 tx=0x7feaec02b980 comp rx=0 tx=0).stop 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 -- 172.21.15.67:0/1423993354 shutdown_connections 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 --2- 172.21.15.67:0/1423993354 >> v2:172.21.15.187:3300/0 conn(0x7feb000d3350 0x7feb000b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 --2- 172.21.15.67:0/1423993354 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb000d2e10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 -- 172.21.15.67:0/1423993354 >> 172.21.15.67:0/1423993354 conn(0x7feb0006d3c0 msgr2=0x7feb000b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 -- 172.21.15.67:0/1423993354 shutdown_connections 2024-04-17T07:15:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.650+0000 7feb06200640 1 -- 172.21.15.67:0/1423993354 wait complete. 2024-04-17T07:15:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.652+0000 7feb06200640 1 Processor -- start 2024-04-17T07:15:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.652+0000 7feb06200640 1 -- start start 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb001f7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb000d3350 0x7feb001f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb000d3800 con 0x7feb0015ee20 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feb0015f200 con 0x7feb000d3350 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb06200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb000b8d80 con 0x7feb000d1ab0 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb05800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb05800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42310/0 (socket says 172.21.15.67:42310) 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.653+0000 7feb05800640 1 -- 172.21.15.67:0/1018511760 learned_addr learned my addr 172.21.15.67:0/1018511760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feaffe00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb000d3350 0x7feb001f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb04e00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb001f7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 -- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 msgr2=0x7feb001f7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb001f7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 -- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb000d3350 msgr2=0x7feb001f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb000d3350 0x7feb001f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 -- 172.21.15.67:0/1018511760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb00161ed0 con 0x7feb0015ee20 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb04e00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb001f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.654+0000 7feb05800640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feaf800efc0 tx=0x7feaf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.655+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8018020 con 0x7feb0015ee20 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.655+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf8014480 con 0x7feb0015ee20 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.655+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb001621b0 con 0x7feb0015ee20 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.655+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8014870 con 0x7feb0015ee20 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.655+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb00146a40 con 0x7feb0015ee20 2024-04-17T07:15:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.656+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb00068a10 con 0x7feb0015ee20 2024-04-17T07:15:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.657+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feaf8005d10 con 0x7feb0015ee20 2024-04-17T07:15:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.658+0000 7feafcc00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 0x7feae006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.658+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7feaf808c400 con 0x7feb0015ee20 2024-04-17T07:15:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.658+0000 7feb04e00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 0x7feae006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.659+0000 7feb04e00640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 0x7feae006edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feaf40015e0 tx=0x7feaf4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.662+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7feaf8022070 con 0x7feb0015ee20 2024-04-17T07:15:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.936+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7feb000b6830 con 0x7feb0015ee20 2024-04-17T07:15:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.939+0000 7feafcc00640 1 -- 172.21.15.67:0/1018511760 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) ==== 97+0+0 (secure 0 0 0) 0x7feaf8010040 con 0x7feb0015ee20 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 msgr2=0x7feae006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 0x7feae006edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feaf40015e0 tx=0x7feaf4009040 comp rx=0 tx=0).stop 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 msgr2=0x7feb000ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feaf800efc0 tx=0x7feaf800c490 comp rx=0 tx=0).stop 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 shutdown_connections 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0015ee20 0x7feb000ca980 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb000d3350 0x7feb001f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feae006c8f0 0x7feae006edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 --2- 172.21.15.67:0/1018511760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb000d1ab0 0x7feb001f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 >> 172.21.15.67:0/1018511760 conn(0x7feb0006d3c0 msgr2=0x7feb000b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 shutdown_connections 2024-04-17T07:15:06.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:06.942+0000 7feb06200640 1 -- 172.21.15.67:0/1018511760 wait complete. 2024-04-17T07:15:06.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-17T07:15:07.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.152+0000 7fc34de00640 1 Processor -- start 2024-04-17T07:15:07.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.152+0000 7fc34de00640 1 -- start start 2024-04-17T07:15:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc34de00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc34806f670 0x7fc34806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc34de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc3480cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc34de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34806ff90 con 0x7fc348071d10 2024-04-17T07:15:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc34de00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc3480ceb00 con 0x7fc34806f670 2024-04-17T07:15:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc34de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480cede0 con 0x7fc3480cd020 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc347400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc348071d10 0x7fc3480721c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48122/0 (socket says 172.21.15.67:48122) 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc347400640 1 -- 172.21.15.67:0/1718417281 learned_addr learned my addr 172.21.15.67:0/1718417281 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc346a00640 1 --2- 172.21.15.67:0/1718417281 >> v2:172.21.15.187:3300/0 conn(0x7fc34806f670 0x7fc34806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.154+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc3480cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472211266 0 0) 0x7fc34806ff90 con 0x7fc348071d10 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32c003520 con 0x7fc348071d10 2024-04-17T07:15:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 105749334 0 0) 0x7fc32c003520 con 0x7fc348071d10 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 msgr2=0x7fc3480cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc3480cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 >> v2:172.21.15.187:3300/0 conn(0x7fc34806f670 msgr2=0x7fc34806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 --2- 172.21.15.67:0/1718417281 >> v2:172.21.15.187:3300/0 conn(0x7fc34806f670 0x7fc34806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3480cf0c0 con 0x7fc348071d10 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc3400019f0 con 0x7fc348071d10 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc3480cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 >> v1:172.21.15.67:6789/0 conn(0x7fc348071d10 legacy=0x7fc3480721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc32c0039b0 0x7fc32c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32c004800 0x7fc32c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc32c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480ceb00 con 0x7fc32c004800 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480cede0 con 0x7fc32c0039b0 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.155+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc32c00b9c0 con 0x7fc3480cd020 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32c004800 0x7fc32c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc346a00640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc32c0039b0 0x7fc32c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 -- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 msgr2=0x7fc32c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3480cd020 0x7fc32c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 -- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc32c0039b0 msgr2=0x7fc32c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc32c0039b0 0x7fc32c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.156+0000 7fc33e000640 1 -- 172.21.15.67:0/1718417281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c003520 con 0x7fc32c004800 2024-04-17T07:15:07.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.157+0000 7fc33e000640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32c004800 0x7fc32c004c80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc3300098d0 tx=0x7fc32c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d752f63831fd8331 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.157+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33000f040 con 0x7fc32c004800 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.157+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc330004230 con 0x7fc32c004800 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.157+0000 7fc346000640 1 -- 172.21.15.67:0/1718417281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330003610 con 0x7fc32c004800 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 -- 172.21.15.67:0/1718417281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32c004800 msgr2=0x7fc32c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 --2- 172.21.15.67:0/1718417281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32c004800 0x7fc32c004c80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc3300098d0 tx=0x7fc32c005830 comp rx=0 tx=0).stop 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc347400640 1 -- 172.21.15.67:0/1718417281 reap_dead start 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 -- 172.21.15.67:0/1718417281 shutdown_connections 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 -- 172.21.15.67:0/1718417281 >> 172.21.15.67:0/1718417281 conn(0x7fc34806bf50 msgr2=0x7fc3480b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 -- 172.21.15.67:0/1718417281 shutdown_connections 2024-04-17T07:15:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.158+0000 7fc34de00640 1 -- 172.21.15.67:0/1718417281 wait complete. 2024-04-17T07:15:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.160+0000 7fc34de00640 1 Processor -- start 2024-04-17T07:15:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.160+0000 7fc34de00640 1 -- start start 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc32c004800 0x7fc34815b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 0x7fc3480aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480ce720 con 0x7fc34814f1b0 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480cf680 con 0x7fc34814f630 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc34de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480ce8a0 con 0x7fc32c004800 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc33e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc33e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42316/0 (socket says 172.21.15.67:42316) 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.161+0000 7fc33e000640 1 -- 172.21.15.67:0/3740671747 learned_addr learned my addr 172.21.15.67:0/3740671747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc347400640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 0x7fc3480aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc346a00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc32c004800 0x7fc34815b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 -- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc32c004800 msgr2=0x7fc34815b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc32c004800 0x7fc34815b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 -- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 msgr2=0x7fc3480aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 0x7fc3480aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 -- 172.21.15.67:0/3740671747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3481f7e10 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc347400640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 0x7fc3480aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33e000640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc3300099f0 tx=0x7fc330004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330048070 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.162+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc330030460 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.163+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3481f80f0 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.163+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330030850 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.163+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3481f86b0 con 0x7fc34814f1b0 2024-04-17T07:15:07.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.164+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc348068b90 con 0x7fc34814f1b0 2024-04-17T07:15:07.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.165+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc33004b070 con 0x7fc34814f1b0 2024-04-17T07:15:07.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.166+0000 7fc33fe00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 0x7fc32406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.166+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7fc3300ad240 con 0x7fc34814f1b0 2024-04-17T07:15:07.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.167+0000 7fc346a00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 0x7fc32406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.168+0000 7fc346a00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 0x7fc32406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc3380099d0 tx=0x7fc338008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.171+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc330034020 con 0x7fc34814f1b0 2024-04-17T07:15:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.451+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fc3480757e0 con 0x7fc34814f1b0 2024-04-17T07:15:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.455+0000 7fc33fe00640 1 -- 172.21.15.67:0/3740671747 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fc33007ba50 con 0x7fc34814f1b0 2024-04-17T07:15:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 msgr2=0x7fc32406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 0x7fc32406edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc3380099d0 tx=0x7fc338008040 comp rx=0 tx=0).stop 2024-04-17T07:15:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 msgr2=0x7fc3480c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc3300099f0 tx=0x7fc330004730 comp rx=0 tx=0).stop 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 shutdown_connections 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc34814f630 0x7fc3480aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc34814f1b0 0x7fc3480c9070 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc32406c8f0 0x7fc32406edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 --2- 172.21.15.67:0/3740671747 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc32c004800 0x7fc34815b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 >> 172.21.15.67:0/3740671747 conn(0x7fc34806bf50 msgr2=0x7fc3480caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 shutdown_connections 2024-04-17T07:15:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.458+0000 7fc34de00640 1 -- 172.21.15.67:0/3740671747 wait complete. 2024-04-17T07:15:07.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:15:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.676+0000 7f1a13e00640 1 Processor -- start 2024-04-17T07:15:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.677+0000 7f1a13e00640 1 -- start start 2024-04-17T07:15:07.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a1415ec60 0x7f1a140b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a140cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1a140cf910 0x7f1a1406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1a140d49c0 con 0x7f1a1415ec60 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1a140d3660 con 0x7f1a140cf910 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.678+0000 7f1a13e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1a140b6f70 con 0x7f1a140b6b30 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a1415ec60 0x7f1a140b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a1415ec60 0x7f1a140b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42320/0 (socket says 172.21.15.67:42320) 2024-04-17T07:15:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 -- 172.21.15.67:0/1363197232 learned_addr learned my addr 172.21.15.67:0/1363197232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12000640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a140cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a13400640 1 --2- 172.21.15.67:0/1363197232 >> v2:172.21.15.187:3300/0 conn(0x7f1a140cf910 0x7f1a1406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 -- 172.21.15.67:0/1363197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 msgr2=0x7f1a140cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a140cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 -- 172.21.15.67:0/1363197232 >> v2:172.21.15.187:3300/0 conn(0x7f1a140cf910 msgr2=0x7f1a1406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 --2- 172.21.15.67:0/1363197232 >> v2:172.21.15.187:3300/0 conn(0x7f1a140cf910 0x7f1a1406cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.679+0000 7f1a12a00640 1 -- 172.21.15.67:0/1363197232 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a140d2d30 con 0x7f1a1415ec60 2024-04-17T07:15:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.680+0000 7f1a12a00640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a1415ec60 0x7f1a140b65f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1a140c8670 tx=0x7f1a0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cabf31c07558b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a11600640 1 -- 172.21.15.67:0/1363197232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04035020 con 0x7f1a1415ec60 2024-04-17T07:15:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a11600640 1 -- 172.21.15.67:0/1363197232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a04031870 con 0x7f1a1415ec60 2024-04-17T07:15:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a11600640 1 -- 172.21.15.67:0/1363197232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04031c40 con 0x7f1a1415ec60 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a13e00640 1 -- 172.21.15.67:0/1363197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a1415ec60 msgr2=0x7f1a140b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a13e00640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a1415ec60 0x7f1a140b65f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1a140c8670 tx=0x7f1a0402b910 comp rx=0 tx=0).stop 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.681+0000 7f1a13e00640 1 -- 172.21.15.67:0/1363197232 shutdown_connections 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 --2- 172.21.15.67:0/1363197232 >> v2:172.21.15.187:3300/0 conn(0x7f1a140cf910 0x7f1a1406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a140cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 --2- 172.21.15.67:0/1363197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a1415ec60 0x7f1a140b65f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 -- 172.21.15.67:0/1363197232 >> 172.21.15.67:0/1363197232 conn(0x7f1a140d15f0 msgr2=0x7f1a140d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 -- 172.21.15.67:0/1363197232 shutdown_connections 2024-04-17T07:15:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.682+0000 7f1a13e00640 1 -- 172.21.15.67:0/1363197232 wait complete. 2024-04-17T07:15:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.683+0000 7f1a13e00640 1 Processor -- start 2024-04-17T07:15:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.684+0000 7f1a13e00640 1 -- start start 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a14154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 0x7f1a140ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140cfda0 con 0x7f1a140cf910 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140d32f0 con 0x7f1a1415ec60 2024-04-17T07:15:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1415f040 con 0x7f1a140b6b30 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42330/0 (socket says 172.21.15.67:42330) 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 -- 172.21.15.67:0/4032691418 learned_addr learned my addr 172.21.15.67:0/4032691418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12a00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a14154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13400640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 0x7f1a140ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 -- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 msgr2=0x7f1a14154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a14154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 -- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 msgr2=0x7f1a140ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 0x7f1a140ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12000640 1 -- 172.21.15.67:0/4032691418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a140b51a0 con 0x7f1a140cf910 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a13400640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 0x7f1a140ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.685+0000 7f1a12a00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a14154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a12000640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00ce70 tx=0x7f19fc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc003b90 con 0x7f1a140cf910 2024-04-17T07:15:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19fc003d30 con 0x7f1a140cf910 2024-04-17T07:15:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc004260 con 0x7f1a140cf910 2024-04-17T07:15:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a140b5480 con 0x7f1a140cf910 2024-04-17T07:15:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.686+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a14155440 con 0x7f1a140cf910 2024-04-17T07:15:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.687+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a14068a10 con 0x7f1a140cf910 2024-04-17T07:15:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.688+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19fc004400 con 0x7f1a140cf910 2024-04-17T07:15:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.689+0000 7f1a0aa00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 0x7f19f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.689+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f19fc08c330 con 0x7f1a140cf910 2024-04-17T07:15:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.690+0000 7f1a12a00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 0x7f19f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.691+0000 7f1a12a00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 0x7f19f006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1a04036000 tx=0x7f1a04008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:07.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.694+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f19fc05aa10 con 0x7f1a140cf910 2024-04-17T07:15:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.900+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a140d2050 con 0x7f1a140cf910 2024-04-17T07:15:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.901+0000 7f1a0aa00640 1 -- 172.21.15.67:0/4032691418 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+12843 (secure 0 0 0) 0x7f19fc014e10 con 0x7f1a140cf910 2024-04-17T07:15:07.901 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:07.902 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:15:03.703357+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.903+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 msgr2=0x7f19f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.903+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 0x7f19f006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1a04036000 tx=0x7f1a04008030 comp rx=0 tx=0).stop 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 msgr2=0x7f1a140cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00ce70 tx=0x7f19fc00a660 comp rx=0 tx=0).stop 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 shutdown_connections 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a1415ec60 0x7f1a140ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140cf910 0x7f1a140cdf00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f19f006c8f0 0x7f19f006edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 --2- 172.21.15.67:0/4032691418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1a140b6b30 0x7f1a14154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 >> 172.21.15.67:0/4032691418 conn(0x7f1a140d15f0 msgr2=0x7f1a140b8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 shutdown_connections 2024-04-17T07:15:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:07.904+0000 7f1a13e00640 1 -- 172.21.15.67:0/4032691418 wait complete. 2024-04-17T07:15:07.925 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:15:07.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:15:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.154+0000 7f27eee00640 1 Processor -- start 2024-04-17T07:15:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.154+0000 7f27eee00640 1 -- start start 2024-04-17T07:15:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.155+0000 7f27eee00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f27e8144fb0 0x7f27e80b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.155+0000 7f27eee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 0x7f27e80b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.155+0000 7f27eee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e815f240 con 0x7f27e80cf620 2024-04-17T07:15:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.155+0000 7f27eee00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f27e8160110 con 0x7f27e8144fb0 2024-04-17T07:15:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.155+0000 7f27eee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27e80b6420 con 0x7f27e815fcb0 2024-04-17T07:15:08.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27ee400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27e80cf620 0x7f27e80cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48124/0 (socket says 172.21.15.67:48124) 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27ee400640 1 -- 172.21.15.67:0/2790059549 learned_addr learned my addr 172.21.15.67:0/2790059549 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27ed000640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 0x7f27e80b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27eda00640 1 --2- 172.21.15.67:0/2790059549 >> v2:172.21.15.187:3300/0 conn(0x7f27e8144fb0 0x7f27e80b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342129827 0 0) 0x7f27e815f240 con 0x7f27e80cf620 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.156+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d0003520 con 0x7f27e80cf620 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2809088199 0 0) 0x7f27d0003520 con 0x7f27e80cf620 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 msgr2=0x7f27e80b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 0x7f27e80b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 >> v2:172.21.15.187:3300/0 conn(0x7f27e8144fb0 msgr2=0x7f27e80b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 --2- 172.21.15.67:0/2790059549 >> v2:172.21.15.187:3300/0 conn(0x7f27e8144fb0 0x7f27e80b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e815f4b0 con 0x7f27e80cf620 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f27e00019f0 con 0x7f27e80cf620 2024-04-17T07:15:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 >> v1:172.21.15.67:6789/0 conn(0x7f27e80cf620 legacy=0x7f27e80cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d0004800 0x7f27d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 0x7f27d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8160110 con 0x7f27d00039e0 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27e80b6420 con 0x7f27d0004800 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27d000ba10 con 0x7f27e815fcb0 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.157+0000 7f27eda00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27eda00640 1 -- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 msgr2=0x7f27d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27eda00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e815fcb0 0x7f27d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27eda00640 1 -- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d0004800 msgr2=0x7f27d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27ed000640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d0004800 0x7f27d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27eda00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d0004800 0x7f27d0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.158+0000 7f27eda00640 1 -- 172.21.15.67:0/2790059549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d0003520 con 0x7f27d00039e0 2024-04-17T07:15:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.159+0000 7f27eda00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27d0003e00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ac0 tx=0x7f27d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abac1768afd55e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.159+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc039070 con 0x7f27d00039e0 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.159+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27dc02c710 con 0x7f27d00039e0 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.159+0000 7f27e7e00640 1 -- 172.21.15.67:0/2790059549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc009690 con 0x7f27d00039e0 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 -- 172.21.15.67:0/2790059549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 msgr2=0x7f27d0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 --2- 172.21.15.67:0/2790059549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27d0003e00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ac0 tx=0x7f27d0005830 comp rx=0 tx=0).stop 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27ee400640 1 -- 172.21.15.67:0/2790059549 reap_dead start 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 -- 172.21.15.67:0/2790059549 shutdown_connections 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 -- 172.21.15.67:0/2790059549 >> 172.21.15.67:0/2790059549 conn(0x7f27e80b23c0 msgr2=0x7f27e80b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 -- 172.21.15.67:0/2790059549 shutdown_connections 2024-04-17T07:15:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.160+0000 7f27eee00640 1 -- 172.21.15.67:0/2790059549 wait complete. 2024-04-17T07:15:08.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.162+0000 7f27eee00640 1 Processor -- start 2024-04-17T07:15:08.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.162+0000 7f27eee00640 1 -- start start 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e80a9ed0 0x7f27e80b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27e80aa350 0x7f27e80b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8145390 con 0x7f27d00039e0 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27e80b6700 con 0x7f27e80aa350 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27e80b5ab0 con 0x7f27e80a9ed0 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42352/0 (socket says 172.21.15.67:42352) 2024-04-17T07:15:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 -- 172.21.15.67:0/2574098734 learned_addr learned my addr 172.21.15.67:0/2574098734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27ed000640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e80a9ed0 0x7f27e80b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 -- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e80a9ed0 msgr2=0x7f27e80b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e80a9ed0 0x7f27e80b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 -- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27e80aa350 msgr2=0x7f27e80b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27e80aa350 0x7f27e80b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.163+0000 7f27eda00640 1 -- 172.21.15.67:0/2574098734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e813cf40 con 0x7f27d00039e0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27eda00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f27dc037040 tx=0x7f27dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc042020 con 0x7f27d00039e0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27dc03a040 con 0x7f27d00039e0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc039070 con 0x7f27d00039e0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e813d220 con 0x7f27d00039e0 2024-04-17T07:15:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.164+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e813d7e0 con 0x7f27d00039e0 2024-04-17T07:15:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.165+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e8068a10 con 0x7f27d00039e0 2024-04-17T07:15:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.166+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27dc045020 con 0x7f27d00039e0 2024-04-17T07:15:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.168+0000 7f27e5600640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 0x7f27c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.168+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f27dc0acca0 con 0x7f27d00039e0 2024-04-17T07:15:08.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.169+0000 7f27ed000640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 0x7f27c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.170+0000 7f27ed000640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 0x7f27c806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f27e80ccf50 tx=0x7f27d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.174+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f27dc04c070 con 0x7f27d00039e0 2024-04-17T07:15:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.380+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f27e8068610 con 0x7f27d00039e0 2024-04-17T07:15:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.381+0000 7f27e5600640 1 -- 172.21.15.67:0/2574098734 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) ==== 93+0+10 (secure 0 0 0) 0x7f27dc07b410 con 0x7f27d00039e0 2024-04-17T07:15:08.381 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 msgr2=0x7f27c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 0x7f27c806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f27e80ccf50 tx=0x7f27d4008040 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 msgr2=0x7f27e814adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f27dc037040 tx=0x7f27dc004730 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 shutdown_connections 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27e80aa350 0x7f27e80b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f27c806c8f0 0x7f27c806edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27e80a9ed0 0x7f27e80b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 --2- 172.21.15.67:0/2574098734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d00039e0 0x7f27e814adc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 >> 172.21.15.67:0/2574098734 conn(0x7f27e80b23c0 msgr2=0x7f27e8071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.383+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 shutdown_connections 2024-04-17T07:15:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.384+0000 7f27eee00640 1 -- 172.21.15.67:0/2574098734 wait complete. 2024-04-17T07:15:08.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:15:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.630+0000 7ff6a6800640 1 Processor -- start 2024-04-17T07:15:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.630+0000 7ff6a6800640 1 -- start start 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a6800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff6a00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a6800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00d2d30 con 0x7ff6a015ec60 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a6800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00d3010 con 0x7ff6a00cf910 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a6800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a00d49c0 con 0x7ff6a00b6d00 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6a015ec60 0x7ff6a00b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48130/0 (socket says 172.21.15.67:48130) 2024-04-17T07:15:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a5400640 1 -- 172.21.15.67:0/1287879253 learned_addr learned my addr 172.21.15.67:0/1287879253 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff6a4a00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff6a00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.632+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623951250 0 0) 0x7ff6a00d2d30 con 0x7ff6a015ec60 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff688003520 con 0x7ff6a015ec60 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1760106189 0 0) 0x7ff688003520 con 0x7ff6a015ec60 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 msgr2=0x7ff6a00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff6a00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 >> v1:172.21.15.187:6789/0 conn(0x7ff6a00cf910 legacy=0x7ff6a00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a00d32f0 con 0x7ff6a015ec60 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff694001670 con 0x7ff6a015ec60 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 >> v1:172.21.15.67:6789/0 conn(0x7ff6a015ec60 legacy=0x7ff6a00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff688003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff688004800 0x7ff688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff688005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a00d49c0 con 0x7ff688003a50 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff68800bb80 con 0x7ff688004800 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.633+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff68800bd80 con 0x7ff6a00b6d00 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff688003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a4a00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff688004800 0x7ff688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5e00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff688005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 -- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 msgr2=0x7ff688005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff688005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 -- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff688004800 msgr2=0x7ff688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff688004800 0x7ff688004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5400640 1 -- 172.21.15.67:0/1287879253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff688003520 con 0x7ff688003a50 2024-04-17T07:15:08.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.634+0000 7ff6a5e00640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b6d00 0x7ff688005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.635+0000 7ff6a5400640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff688003e70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff69400c240 tx=0x7ff688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2af22ac656161e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694007d10 con 0x7ff688003a50 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff694030070 con 0x7ff688003a50 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff69fe00640 1 -- 172.21.15.67:0/1287879253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69402cd60 con 0x7ff688003a50 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 -- 172.21.15.67:0/1287879253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 msgr2=0x7ff688003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 --2- 172.21.15.67:0/1287879253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff688003e70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff69400c240 tx=0x7ff688005830 comp rx=0 tx=0).stop 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a5e00640 1 -- 172.21.15.67:0/1287879253 reap_dead start 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 -- 172.21.15.67:0/1287879253 shutdown_connections 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 -- 172.21.15.67:0/1287879253 >> 172.21.15.67:0/1287879253 conn(0x7ff6a00d15f0 msgr2=0x7ff6a00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 -- 172.21.15.67:0/1287879253 shutdown_connections 2024-04-17T07:15:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.636+0000 7ff6a6800640 1 -- 172.21.15.67:0/1287879253 wait complete. 2024-04-17T07:15:08.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.638+0000 7ff6a6800640 1 Processor -- start 2024-04-17T07:15:08.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.639+0000 7ff6a6800640 1 -- start start 2024-04-17T07:15:08.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b4d50 0x7ff6a00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6a00ce960 0x7ff6a01539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a015f040 con 0x7ff688003a50 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a00d3660 con 0x7ff6a00ce960 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a6800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a00d00c0 con 0x7ff6a00b4d50 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42370/0 (socket says 172.21.15.67:42370) 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 -- 172.21.15.67:0/2481972252 learned_addr learned my addr 172.21.15.67:0/2481972252 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a4a00640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b4d50 0x7ff6a00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 -- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b4d50 msgr2=0x7ff6a00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b4d50 0x7ff6a00ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.640+0000 7ff6a5400640 1 -- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6a00ce960 msgr2=0x7ff6a01539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff6a5400640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6a00ce960 0x7ff6a01539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff6a5400640 1 -- 172.21.15.67:0/2481972252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a00ab520 con 0x7ff688003a50 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff6a5400640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff694001970 tx=0x7ff6a00b7ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694007d10 con 0x7ff688003a50 2024-04-17T07:15:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff694004bf0 con 0x7ff688003a50 2024-04-17T07:15:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694007070 con 0x7ff688003a50 2024-04-17T07:15:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a00ab800 con 0x7ff688003a50 2024-04-17T07:15:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.641+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a00abdc0 con 0x7ff688003a50 2024-04-17T07:15:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.643+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a0068a10 con 0x7ff688003a50 2024-04-17T07:15:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.644+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff694007210 con 0x7ff688003a50 2024-04-17T07:15:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.645+0000 7ff69d600640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 0x7ff68006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.645+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7ff6940ad100 con 0x7ff688003a50 2024-04-17T07:15:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.646+0000 7ff6a4a00640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 0x7ff68006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.647+0000 7ff6a4a00640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 0x7ff68006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00af3d0 tx=0x7ff68c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.650+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff69407b880 con 0x7ff688003a50 2024-04-17T07:15:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.860+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff6a00b9900 con 0x7ff688003a50 2024-04-17T07:15:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.861+0000 7ff69d600640 1 -- 172.21.15.67:0/2481972252 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) ==== 84+0+65 (secure 0 0 0) 0x7ff69407b180 con 0x7ff688003a50 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:08.861 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:15:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 msgr2=0x7ff68006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 0x7ff68006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00af3d0 tx=0x7ff68c008040 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 msgr2=0x7ff6a00ac5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff694001970 tx=0x7ff6a00b7ac0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 shutdown_connections 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6a00ce960 0x7ff6a01539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff68006c8f0 0x7ff68006edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6a00b4d50 0x7ff6a00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 --2- 172.21.15.67:0/2481972252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff688003a50 0x7ff6a00ac5c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.863+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 >> 172.21.15.67:0/2481972252 conn(0x7ff6a00d15f0 msgr2=0x7ff6a00b9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.864+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 shutdown_connections 2024-04-17T07:15:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:08.864+0000 7ff6a6800640 1 -- 172.21.15.67:0/2481972252 wait complete. 2024-04-17T07:15:08.884 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:15:08.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:15:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.103+0000 7f83ab400640 1 Processor -- start 2024-04-17T07:15:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.104+0000 7f83ab400640 1 -- start start 2024-04-17T07:15:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.105+0000 7f83ab400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f83a40cfe30 0x7f83a41662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.105+0000 7f83ab400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a40b5860 con 0x7f83a40b4e20 2024-04-17T07:15:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.105+0000 7f83ab400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a40b4060 con 0x7f83a40cd7c0 2024-04-17T07:15:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.105+0000 7f83ab400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f83a40b4340 con 0x7f83a40cfe30 2024-04-17T07:15:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.106+0000 7f83a3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83a40b4e20 0x7f83a40cf720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48138/0 (socket says 172.21.15.67:48138) 2024-04-17T07:15:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.106+0000 7f83a3e00640 1 -- 172.21.15.67:0/2240718 learned_addr learned my addr 172.21.15.67:0/2240718 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.106+0000 7f83a9400640 1 --2- 172.21.15.67:0/2240718 >> v2:172.21.15.187:3300/0 conn(0x7f83a40cfe30 0x7f83a41662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996252060 0 0) 0x7f83a40b5860 con 0x7f83a40b4e20 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f838c005730 con 0x7f83a40b4e20 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468483451 0 0) 0x7f83a40b4060 con 0x7f83a40cd7c0 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83a40b5860 con 0x7f83a40cd7c0 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3182208099 0 0) 0x7f838c005730 con 0x7f83a40b4e20 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 >> v2:172.21.15.187:3300/0 conn(0x7f83a40cfe30 msgr2=0x7f83a41662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 --2- 172.21.15.67:0/2240718 >> v2:172.21.15.187:3300/0 conn(0x7f83a40cfe30 0x7f83a41662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 >> v1:172.21.15.187:6789/0 conn(0x7f83a40cd7c0 legacy=0x7f83a40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a40b4620 con 0x7f83a40b4e20 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8394001670 con 0x7f83a40b4e20 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 >> v1:172.21.15.67:6789/0 conn(0x7f83a40b4e20 legacy=0x7f83a40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c005da0 0x7f838c0061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 0x7f838c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f838c007420 0x7f838c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83a40b4340 con 0x7f838c006a10 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f838c00e350 con 0x7f838c005da0 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.107+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f838c00e550 con 0x7f838c007420 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 0x7f838c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a8a00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c005da0 0x7f838c0061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 -- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f838c007420 msgr2=0x7f838c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f838c007420 0x7f838c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 -- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c005da0 msgr2=0x7f838c0061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c005da0 0x7f838c0061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a3e00640 1 -- 172.21.15.67:0/2240718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c005730 con 0x7f838c006a10 2024-04-17T07:15:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a8a00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c005da0 0x7f838c0061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.108+0000 7f83a9400640 1 -- 172.21.15.67:0/2240718 reap_dead start 2024-04-17T07:15:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.109+0000 7f83a3e00640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 0x7f838c006e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f839400c0e0 tx=0x7f838c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf9452d5877074c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.110+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394007c40 con 0x7f838c006a10 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.110+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8394030070 con 0x7f838c006a10 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.110+0000 7f83a3400640 1 -- 172.21.15.67:0/2240718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394005d90 con 0x7f838c006a10 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.110+0000 7f83ab400640 1 -- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 msgr2=0x7f838c006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.110+0000 7f83ab400640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 0x7f838c006e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f839400c0e0 tx=0x7f838c007a40 comp rx=0 tx=0).stop 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.111+0000 7f83ab400640 1 -- 172.21.15.67:0/2240718 shutdown_connections 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.111+0000 7f83ab400640 1 --2- 172.21.15.67:0/2240718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838c006a10 0x7f838c006e90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.111+0000 7f83ab400640 1 -- 172.21.15.67:0/2240718 >> 172.21.15.67:0/2240718 conn(0x7f83a406b8c0 msgr2=0x7f83a406bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.111+0000 7f83ab400640 1 -- 172.21.15.67:0/2240718 shutdown_connections 2024-04-17T07:15:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.111+0000 7f83ab400640 1 -- 172.21.15.67:0/2240718 wait complete. 2024-04-17T07:15:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.112+0000 7f83ab400640 1 Processor -- start 2024-04-17T07:15:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.113+0000 7f83ab400640 1 -- start start 2024-04-17T07:15:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 0x7f83a40b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83a40a94c0 0x7f83a40a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4145510 con 0x7f83a4135b40 2024-04-17T07:15:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f83a4145390 con 0x7f838c006a10 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83ab400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f83a40cf720 con 0x7f83a40a94c0 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42392/0 (socket says 172.21.15.67:42392) 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 -- 172.21.15.67:0/4107505463 learned_addr learned my addr 172.21.15.67:0/4107505463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a8a00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 0x7f83a40b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a9400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83a40a94c0 0x7f83a40a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 -- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83a40a94c0 msgr2=0x7f83a40a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83a40a94c0 0x7f83a40a9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 -- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 msgr2=0x7f83a40b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 0x7f83a40b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a3e00640 1 -- 172.21.15.67:0/4107505463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a41e7340 con 0x7f83a4135b40 2024-04-17T07:15:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.114+0000 7f83a8a00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 0x7f83a40b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83a3e00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f839400c0e0 tx=0x7f839402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394045070 con 0x7f83a4135b40 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8394005e50 con 0x7f83a4135b40 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839402ce30 con 0x7f83a4135b40 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a41e62a0 con 0x7f83a4135b40 2024-04-17T07:15:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.115+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a41e6860 con 0x7f83a4135b40 2024-04-17T07:15:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.116+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a4073a50 con 0x7f83a4135b40 2024-04-17T07:15:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.117+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8394048070 con 0x7f83a4135b40 2024-04-17T07:15:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.119+0000 7f83a0c00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 0x7f838806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.119+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 4857+0+0 (secure 0 0 0) 0x7f83940ac790 con 0x7f83a4135b40 2024-04-17T07:15:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.120+0000 7f83a8a00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 0x7f838806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.121+0000 7f83a8a00640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 0x7f838806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f83a40516a0 tx=0x7f8398009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.124+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f839407af00 con 0x7f83a4135b40 2024-04-17T07:15:09.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.329+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f83a4068d60 con 0x7f83a4135b40 2024-04-17T07:15:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.713+0000 7f83a0c00640 1 -- 172.21.15.67:0/4107505463 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) ==== 138+0+0 (secure 0 0 0) 0x7f839407a800 con 0x7f83a4135b40 2024-04-17T07:15:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 msgr2=0x7f838806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 0x7f838806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f83a40516a0 tx=0x7f8398009040 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 msgr2=0x7f83a41e59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f839400c0e0 tx=0x7f839402e070 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 shutdown_connections 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f838806c8f0 0x7f838806edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83a40a94c0 0x7f83a40a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a4135b40 0x7f83a41e59a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 --2- 172.21.15.67:0/4107505463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f838c006a10 0x7f83a40b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.717+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 >> 172.21.15.67:0/4107505463 conn(0x7f83a406b8c0 msgr2=0x7f83a40cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.718+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 shutdown_connections 2024-04-17T07:15:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.718+0000 7f83ab400640 1 -- 172.21.15.67:0/4107505463 wait complete. 2024-04-17T07:15:09.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:15:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.961+0000 7f7e5da00640 1 Processor -- start 2024-04-17T07:15:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.961+0000 7f7e5da00640 1 -- start start 2024-04-17T07:15:09.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 0x7f7e5806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e580b3b60 con 0x7f7e580b27c0 2024-04-17T07:15:09.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e5806e7e0 con 0x7f7e580b36e0 2024-04-17T07:15:09.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5806eac0 con 0x7f7e58162cc0 2024-04-17T07:15:09.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e580b27c0 0x7f7e580cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48154/0 (socket says 172.21.15.67:48154) 2024-04-17T07:15:09.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.963+0000 7f7e5d000640 1 -- 172.21.15.67:0/3934966815 learned_addr learned my addr 172.21.15.67:0/3934966815 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:09.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e57e00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 0x7f7e5806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2380868325 0 0) 0x7f7e580b3b60 con 0x7f7e580b27c0 2024-04-17T07:15:09.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e3c003520 con 0x7f7e580b27c0 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2318935646 0 0) 0x7f7e3c003520 con 0x7f7e580b27c0 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 msgr2=0x7f7e5806e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 0x7f7e5806e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 >> v1:172.21.15.187:6789/0 conn(0x7f7e580b36e0 legacy=0x7f7e580b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5806eda0 con 0x7f7e580b27c0 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7e500019f0 con 0x7f7e580b27c0 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 >> v1:172.21.15.67:6789/0 conn(0x7f7e580b27c0 legacy=0x7f7e580cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0039e0 0x7f7e3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 0x7f7e3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c004d40 0x7f7e3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5806eac0 con 0x7f7e3c0039e0 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c00bb30 con 0x7f7e3c004d40 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.964+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c00bd30 con 0x7f7e58162cc0 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0039e0 0x7f7e3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e5d000640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c004d40 0x7f7e3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 -- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 msgr2=0x7f7e3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e58162cc0 0x7f7e3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 -- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c004d40 msgr2=0x7f7e3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c004d40 0x7f7e3c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.965+0000 7f7e57e00640 1 -- 172.21.15.67:0/3934966815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c003520 con 0x7f7e3c0039e0 2024-04-17T07:15:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.966+0000 7f7e57e00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0039e0 0x7f7e3c003e00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009a90 tx=0x7f7e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d888a1bd0245e63b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48039070 con 0x7f7e3c0039e0 2024-04-17T07:15:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4802c6b0 con 0x7f7e3c0039e0 2024-04-17T07:15:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e56a00640 1 -- 172.21.15.67:0/3934966815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48009690 con 0x7f7e3c0039e0 2024-04-17T07:15:09.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e5da00640 1 -- 172.21.15.67:0/3934966815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0039e0 msgr2=0x7f7e3c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e5da00640 1 --2- 172.21.15.67:0/3934966815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0039e0 0x7f7e3c003e00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009a90 tx=0x7f7e3c005830 comp rx=0 tx=0).stop 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e5d000640 1 -- 172.21.15.67:0/3934966815 reap_dead start 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e5da00640 1 -- 172.21.15.67:0/3934966815 shutdown_connections 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.967+0000 7f7e5da00640 1 -- 172.21.15.67:0/3934966815 >> 172.21.15.67:0/3934966815 conn(0x7f7e58067d10 msgr2=0x7f7e580d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.968+0000 7f7e5da00640 1 -- 172.21.15.67:0/3934966815 shutdown_connections 2024-04-17T07:15:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.968+0000 7f7e5da00640 1 -- 172.21.15.67:0/3934966815 wait complete. 2024-04-17T07:15:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.969+0000 7f7e5da00640 1 Processor -- start 2024-04-17T07:15:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 -- start start 2024-04-17T07:15:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0039e0 0x7f7e581f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e580ceee0 0x7f7e5815de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e580ccf80 con 0x7f7e581585f0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e581630a0 con 0x7f7e3c0039e0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.970+0000 7f7e5da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e580cd100 con 0x7f7e580ceee0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42416/0 (socket says 172.21.15.67:42416) 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 -- 172.21.15.67:0/1638916925 learned_addr learned my addr 172.21.15.67:0/1638916925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e5d000640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e580ceee0 0x7f7e5815de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57e00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0039e0 0x7f7e581f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 -- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e580ceee0 msgr2=0x7f7e5815de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e580ceee0 0x7f7e5815de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 -- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0039e0 msgr2=0x7f7e581f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0039e0 0x7f7e581f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.971+0000 7f7e57400640 1 -- 172.21.15.67:0/1638916925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e580a95f0 con 0x7f7e581585f0 2024-04-17T07:15:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e57400640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c500 tx=0x7f7e4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440052d0 con 0x7f7e581585f0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e44005470 con 0x7f7e581585f0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e580a98d0 con 0x7f7e581585f0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44012870 con 0x7f7e581585f0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.972+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e580a9e90 con 0x7f7e581585f0 2024-04-17T07:15:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.973+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e58068a10 con 0x7f7e581585f0 2024-04-17T07:15:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.974+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e44012a10 con 0x7f7e581585f0 2024-04-17T07:15:09.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.975+0000 7f7e4fe00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 0x7f7e3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:09.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.975+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5227+0+0 (secure 0 0 0) 0x7f7e4408b620 con 0x7f7e581585f0 2024-04-17T07:15:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.976+0000 7f7e57e00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 0x7f7e3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.977+0000 7f7e57e00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 0x7f7e3806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:09.981+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7e44059c20 con 0x7f7e581585f0 2024-04-17T07:15:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.185+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7e581601e0 con 0x7f7e581585f0 2024-04-17T07:15:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.720+0000 7f7e4fe00640 1 -- 172.21.15.67:0/1638916925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) ==== 170+0+0 (secure 0 0 0) 0x7f7e44059520 con 0x7f7e581585f0 2024-04-17T07:15:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:15:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 msgr2=0x7f7e3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 0x7f7e3806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 msgr2=0x7f7e580ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c500 tx=0x7f7e4400c9c0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 shutdown_connections 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f7e3806c920 0x7f7e3806ede0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e580ceee0 0x7f7e5815de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e581585f0 0x7f7e580ce9a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 --2- 172.21.15.67:0/1638916925 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0039e0 0x7f7e581f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.724+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 >> 172.21.15.67:0/1638916925 conn(0x7f7e58067d10 msgr2=0x7f7e58074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.725+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 shutdown_connections 2024-04-17T07:15:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.725+0000 7f7e5da00640 1 -- 172.21.15.67:0/1638916925 wait complete. 2024-04-17T07:15:10.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:15:10.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.945+0000 7f43a2200640 1 Processor -- start 2024-04-17T07:15:10.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.945+0000 7f43a2200640 1 -- start start 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.946+0000 7f43a2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f439c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.946+0000 7f43a2200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f439c15ff40 con 0x7f439c0cfc10 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.946+0000 7f43a2200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f439c06df40 con 0x7f439c06f800 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.946+0000 7f43a2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f439c06ed00 con 0x7f439c15b6b0 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.947+0000 7f43a1800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f439c0cfc10 0x7f439c15f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48168/0 (socket says 172.21.15.67:48168) 2024-04-17T07:15:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.947+0000 7f43a1800640 1 -- 172.21.15.67:0/2587166612 learned_addr learned my addr 172.21.15.67:0/2587166612 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.947+0000 7f43a0e00640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f439c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.947+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970293643 0 0) 0x7f439c15ff40 con 0x7f439c0cfc10 2024-04-17T07:15:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.947+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4384003520 con 0x7f439c0cfc10 2024-04-17T07:15:10.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1781691762 0 0) 0x7f4384003520 con 0x7f439c0cfc10 2024-04-17T07:15:10.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 msgr2=0x7f439c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:10.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f439c06b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 >> v1:172.21.15.187:6789/0 conn(0x7f439c06f800 legacy=0x7f439c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c06e220 con 0x7f439c0cfc10 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f43940019f0 con 0x7f439c0cfc10 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f43a0e00640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f439c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 >> v1:172.21.15.67:6789/0 conn(0x7f439c0cfc10 legacy=0x7f439c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840039e0 0x7f4384003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f4384004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384004d40 0x7f43840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f439c06ed00 con 0x7f4384004d40 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f438400dd60 con 0x7f43840039e0 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f438400df60 con 0x7f439c15b6b0 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.948+0000 7f43a1800640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384004d40 0x7f43840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a0e00640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840039e0 0x7f4384003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f439be00640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f4384004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a1800640 1 -- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 msgr2=0x7f4384004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a1800640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c15b6b0 0x7f4384004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a1800640 1 -- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840039e0 msgr2=0x7f4384003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a1800640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840039e0 0x7f4384003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.949+0000 7f43a1800640 1 -- 172.21.15.67:0/2587166612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4384003520 con 0x7f4384004d40 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.950+0000 7f43a1800640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384004d40 0x7f43840051c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f439400a860 tx=0x7f4384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b937181abef0a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.950+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439400a480 con 0x7f4384004d40 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.950+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4394034630 con 0x7f4384004d40 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.950+0000 7f439b400640 1 -- 172.21.15.67:0/2587166612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394034a40 con 0x7f4384004d40 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 -- 172.21.15.67:0/2587166612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384004d40 msgr2=0x7f43840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 --2- 172.21.15.67:0/2587166612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384004d40 0x7f43840051c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f439400a860 tx=0x7f4384005830 comp rx=0 tx=0).stop 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a1800640 1 -- 172.21.15.67:0/2587166612 reap_dead start 2024-04-17T07:15:10.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 -- 172.21.15.67:0/2587166612 shutdown_connections 2024-04-17T07:15:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 -- 172.21.15.67:0/2587166612 >> 172.21.15.67:0/2587166612 conn(0x7f439c0b2190 msgr2=0x7f439c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 -- 172.21.15.67:0/2587166612 shutdown_connections 2024-04-17T07:15:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.951+0000 7f43a2200640 1 -- 172.21.15.67:0/2587166612 wait complete. 2024-04-17T07:15:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.953+0000 7f43a2200640 1 Processor -- start 2024-04-17T07:15:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.953+0000 7f43a2200640 1 -- start start 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 0x7f439c0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c14cac0 0x7f439c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f439c0afd60 con 0x7f439c0c5630 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f439c06e870 con 0x7f4384004d40 2024-04-17T07:15:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f43a2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f439c0afee0 con 0x7f439c14cac0 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f439be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f439be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42438/0 (socket says 172.21.15.67:42438) 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.954+0000 7f439be00640 1 -- 172.21.15.67:0/3454727660 learned_addr learned my addr 172.21.15.67:0/3454727660 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f43a1800640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c14cac0 0x7f439c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f43a0e00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 0x7f439c0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 -- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c14cac0 msgr2=0x7f439c14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c14cac0 0x7f439c14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 -- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 msgr2=0x7f439c0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 0x7f439c0ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 -- 172.21.15.67:0/3454727660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c0af240 con 0x7f439c0c5630 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f43a0e00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 0x7f439c0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:10.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.955+0000 7f439be00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f438800e580 tx=0x7f438800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.956+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43880052e0 con 0x7f439c0c5630 2024-04-17T07:15:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.956+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4388018070 con 0x7f439c0c5630 2024-04-17T07:15:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.956+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388003810 con 0x7f439c0c5630 2024-04-17T07:15:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.956+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439c0c5ab0 con 0x7f439c0c5630 2024-04-17T07:15:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.956+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439c0aec80 con 0x7f439c0c5630 2024-04-17T07:15:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.957+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439c069120 con 0x7f439c0c5630 2024-04-17T07:15:10.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.958+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f43880039b0 con 0x7f439c0c5630 2024-04-17T07:15:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.959+0000 7f4398c00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 0x7f437c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:10.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.959+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5609+0+0 (secure 0 0 0) 0x7f438808b7d0 con 0x7f439c0c5630 2024-04-17T07:15:10.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.960+0000 7f43a0e00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 0x7f437c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:10.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.961+0000 7f43a0e00640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 0x7f437c06edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4390006fd0 tx=0x7f4390008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:10.964+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f439c069120 con 0x7f439c0c5630 2024-04-17T07:15:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.211+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f439c0d1330 con 0x7f439c0c5630 2024-04-17T07:15:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 4 and data pool 5 2024-04-17T07:15:11.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.769+0000 7f4398c00640 1 -- 172.21.15.67:0/3454727660 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f4388059c70 con 0x7f439c0c5630 2024-04-17T07:15:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.779+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 msgr2=0x7f437c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.779+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 0x7f437c06edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4390006fd0 tx=0x7f4390008040 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.779+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 msgr2=0x7f439c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.779+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f438800e580 tx=0x7f438800ea40 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 shutdown_connections 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f437c06c8f0 0x7f437c06edb0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4390006fd0 tx=0x7f4390008040 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f439c14cac0 0x7f439c14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f439c0c5630 0x7f439c0c6080 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 --2- 172.21.15.67:0/3454727660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4384004d40 0x7f439c0ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:11.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.780+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 >> 172.21.15.67:0/3454727660 conn(0x7f439c0b2190 msgr2=0x7f439c0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.785+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 shutdown_connections 2024-04-17T07:15:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.786+0000 7f43a2200640 1 -- 172.21.15.67:0/3454727660 wait complete. 2024-04-17T07:15:11.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:15:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.999+0000 7fac8fc00640 1 Processor -- start 2024-04-17T07:15:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:11.999+0000 7fac8fc00640 1 -- start start 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.000+0000 7fac8fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac8806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8fc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac8809b170 0x7fac8815e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8fc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac8806dc30 con 0x7fac880d09f0 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8fc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fac8806df10 con 0x7fac8809b170 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac8806e1f0 con 0x7fac880b3d10 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac87e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac880d09f0 0x7fac8809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48180/0 (socket says 172.21.15.67:48180) 2024-04-17T07:15:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac87e00640 1 -- 172.21.15.67:0/1364036505 learned_addr learned my addr 172.21.15.67:0/1364036505 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8d200640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac8806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac8dc00640 1 --2- 172.21.15.67:0/1364036505 >> v2:172.21.15.187:3300/0 conn(0x7fac8809b170 0x7fac8815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401489205 0 0) 0x7fac8806dc30 con 0x7fac880d09f0 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.001+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac70003520 con 0x7fac880d09f0 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794815560 0 0) 0x7fac70003520 con 0x7fac880d09f0 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 msgr2=0x7fac8806d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac8806d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 >> v2:172.21.15.187:3300/0 conn(0x7fac8809b170 msgr2=0x7fac8815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 --2- 172.21.15.67:0/1364036505 >> v2:172.21.15.187:3300/0 conn(0x7fac8809b170 0x7fac8815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8806e4d0 con 0x7fac880d09f0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fac74001670 con 0x7fac880d09f0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 >> v1:172.21.15.67:6789/0 conn(0x7fac880d09f0 legacy=0x7fac8809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700039b0 0x7fac70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac70004800 0x7fac70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac8806df10 con 0x7fac70004800 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac8806e1f0 con 0x7fac700039b0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac7000b9c0 con 0x7fac880b3d10 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.002+0000 7fac87e00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac70004800 0x7fac70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac8dc00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac8d200640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700039b0 0x7fac70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac87e00640 1 -- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 msgr2=0x7fac700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac87e00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac87e00640 1 -- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700039b0 msgr2=0x7fac70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac87e00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700039b0 0x7fac70003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac87e00640 1 -- 172.21.15.67:0/1364036505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac70003520 con 0x7fac70004800 2024-04-17T07:15:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.003+0000 7fac8dc00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac880b3d10 0x7fac700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.004+0000 7fac87e00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac70004800 0x7fac70004c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac74007fd0 tx=0x7fac70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc209af930bd98d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.004+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74007d50 con 0x7fac70004800 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.004+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac74030070 con 0x7fac70004800 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.004+0000 7fac87400640 1 -- 172.21.15.67:0/1364036505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7402cde0 con 0x7fac70004800 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 -- 172.21.15.67:0/1364036505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac70004800 msgr2=0x7fac70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 --2- 172.21.15.67:0/1364036505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac70004800 0x7fac70004c80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac74007fd0 tx=0x7fac70005830 comp rx=0 tx=0).stop 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8dc00640 1 -- 172.21.15.67:0/1364036505 reap_dead start 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 -- 172.21.15.67:0/1364036505 shutdown_connections 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 -- 172.21.15.67:0/1364036505 >> 172.21.15.67:0/1364036505 conn(0x7fac880b41b0 msgr2=0x7fac880b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 -- 172.21.15.67:0/1364036505 shutdown_connections 2024-04-17T07:15:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.005+0000 7fac8fc00640 1 -- 172.21.15.67:0/1364036505 wait complete. 2024-04-17T07:15:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.007+0000 7fac8fc00640 1 Processor -- start 2024-04-17T07:15:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.007+0000 7fac8fc00640 1 -- start start 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac70004800 0x7fac8815ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 0x7fac8814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac8806d340 con 0x7fac8815d1c0 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac8806d1c0 con 0x7fac881f8310 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac8806d4c0 con 0x7fac70004800 2024-04-17T07:15:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42470/0 (socket says 172.21.15.67:42470) 2024-04-17T07:15:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.008+0000 7fac8dc00640 1 -- 172.21.15.67:0/2722660401 learned_addr learned my addr 172.21.15.67:0/2722660401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8d200640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac70004800 0x7fac8815ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac87e00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 0x7fac8814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 -- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac70004800 msgr2=0x7fac8815ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac70004800 0x7fac8815ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 -- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 msgr2=0x7fac8814b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 0x7fac8814b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 -- 172.21.15.67:0/2722660401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8815dbc0 con 0x7fac8815d1c0 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac87e00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 0x7fac8814b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac8dc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac880733f0 tx=0x7fac8000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.009+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8001c070 con 0x7fac8815d1c0 2024-04-17T07:15:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.010+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac80017400 con 0x7fac8815d1c0 2024-04-17T07:15:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.010+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800177d0 con 0x7fac8815d1c0 2024-04-17T07:15:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.010+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8815dea0 con 0x7fac8815d1c0 2024-04-17T07:15:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.010+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8815e460 con 0x7fac8815d1c0 2024-04-17T07:15:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.011+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac880cd960 con 0x7fac8815d1c0 2024-04-17T07:15:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.012+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac8001f070 con 0x7fac8815d1c0 2024-04-17T07:15:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.013+0000 7fac84c00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 0x7fac6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.013+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7fac8008bca0 con 0x7fac8815d1c0 2024-04-17T07:15:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.013+0000 7fac8d200640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 0x7fac6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.014+0000 7fac8d200640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 0x7fac6806ede0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009930 tx=0x7fac7c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.017+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fac880cd960 con 0x7fac8815d1c0 2024-04-17T07:15:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.222+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac88071db0 con 0x7fac8815d1c0 2024-04-17T07:15:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.224+0000 7fac84c00640 1 -- 172.21.15.67:0/2722660401 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16634 (secure 0 0 0) 0x7fac88071db0 con 0x7fac8815d1c0 2024-04-17T07:15:12.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:12.224 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:15:11.212695+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:15:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 msgr2=0x7fac6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 0x7fac6806ede0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009930 tx=0x7fac7c008040 comp rx=0 tx=0).stop 2024-04-17T07:15:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 msgr2=0x7fac880b6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac880733f0 tx=0x7fac8000f4a0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 shutdown_connections 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac8815d1c0 0x7fac880b6f60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac881f8310 0x7fac8814b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac6806c920 0x7fac6806ede0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 --2- 172.21.15.67:0/2722660401 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac70004800 0x7fac8815ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 >> 172.21.15.67:0/2722660401 conn(0x7fac880b41b0 msgr2=0x7fac880ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 shutdown_connections 2024-04-17T07:15:12.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.227+0000 7fac8fc00640 1 -- 172.21.15.67:0/2722660401 wait complete. 2024-04-17T07:15:12.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:15:12.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.466+0000 7f8292600640 1 Processor -- start 2024-04-17T07:15:12.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.466+0000 7f8292600640 1 -- start start 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.467+0000 7f8292600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 0x7f828c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.467+0000 7f8292600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828c0ceb00 con 0x7f828c06f670 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.467+0000 7f8292600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828c0cede0 con 0x7f828c06e020 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.467+0000 7f8292600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f828c0cf0c0 con 0x7f828c0cd020 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.468+0000 7f828b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f828c06f670 0x7f828c06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48190/0 (socket says 172.21.15.67:48190) 2024-04-17T07:15:12.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.468+0000 7f828b400640 1 -- 172.21.15.67:0/3881174407 learned_addr learned my addr 172.21.15.67:0/3881174407 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.468+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724981060 0 0) 0x7f828c0ceb00 con 0x7f828c06f670 2024-04-17T07:15:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.468+0000 7f8283e00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 0x7f828c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.468+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8270003520 con 0x7f828c06f670 2024-04-17T07:15:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 900346032 0 0) 0x7f8270003520 con 0x7f828c06f670 2024-04-17T07:15:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 msgr2=0x7f828c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 0x7f828c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 >> v1:172.21.15.187:6789/0 conn(0x7f828c06e020 legacy=0x7f828c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c0cf3a0 con 0x7f828c06f670 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f827c001670 con 0x7f828c06f670 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 >> v1:172.21.15.67:6789/0 conn(0x7f828c06f670 legacy=0x7f828c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82700039e0 0x7f8270003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 0x7f82700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f828c0cf0c0 con 0x7f82700039e0 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f827000baa0 con 0x7f8270004800 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f827000bca0 con 0x7f828c0cd020 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.469+0000 7f828b400640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82700039e0 0x7f8270003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f828b400640 1 -- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 msgr2=0x7f82700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f828b400640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0cd020 0x7f82700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f828b400640 1 -- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8270004800 msgr2=0x7f8270004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f8283e00640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f828b400640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8270004800 0x7f8270004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.470+0000 7f828b400640 1 -- 172.21.15.67:0/3881174407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8270003520 con 0x7f82700039e0 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f828b400640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82700039e0 0x7f8270003e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f827c00c160 tx=0x7f8270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3c3edd5b96ecd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c007c60 con 0x7f82700039e0 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827c030070 con 0x7f82700039e0 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f828aa00640 1 -- 172.21.15.67:0/3881174407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c005d70 con 0x7f82700039e0 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f8292600640 1 -- 172.21.15.67:0/3881174407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82700039e0 msgr2=0x7f8270003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.471+0000 7f8292600640 1 --2- 172.21.15.67:0/3881174407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82700039e0 0x7f8270003e00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f827c00c160 tx=0x7f8270005830 comp rx=0 tx=0).stop 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.472+0000 7f828be00640 1 -- 172.21.15.67:0/3881174407 reap_dead start 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.472+0000 7f8292600640 1 -- 172.21.15.67:0/3881174407 shutdown_connections 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.472+0000 7f8292600640 1 -- 172.21.15.67:0/3881174407 >> 172.21.15.67:0/3881174407 conn(0x7f828c06bf50 msgr2=0x7f828c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.472+0000 7f8292600640 1 -- 172.21.15.67:0/3881174407 shutdown_connections 2024-04-17T07:15:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.472+0000 7f8292600640 1 -- 172.21.15.67:0/3881174407 wait complete. 2024-04-17T07:15:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.474+0000 7f8292600640 1 Processor -- start 2024-04-17T07:15:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.474+0000 7f8292600640 1 -- start start 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82700039e0 0x7f828c153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0b13b0 0x7f828c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f828c0ce8a0 con 0x7f828c157de0 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f828c0ce720 con 0x7f82700039e0 2024-04-17T07:15:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f8292600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f828c15cb00 con 0x7f828c0b13b0 2024-04-17T07:15:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f828be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f828be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42490/0 (socket says 172.21.15.67:42490) 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.475+0000 7f828be00640 1 -- 172.21.15.67:0/1637928929 learned_addr learned my addr 172.21.15.67:0/1637928929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f8283e00640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0b13b0 0x7f828c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 -- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0b13b0 msgr2=0x7f828c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0b13b0 0x7f828c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 -- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82700039e0 msgr2=0x7f828c153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82700039e0 0x7f828c153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 -- 172.21.15.67:0/1637928929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c153580 con 0x7f828c157de0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f828be00640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f828c0b28c0 tx=0x7f828400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.476+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284017070 con 0x7f828c157de0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.477+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284005d60 con 0x7f828c157de0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.477+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828400a820 con 0x7f828c157de0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.477+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828c1f79c0 con 0x7f828c157de0 2024-04-17T07:15:12.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.477+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828c1f7f80 con 0x7f828c157de0 2024-04-17T07:15:12.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.478+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828c068b70 con 0x7f828c157de0 2024-04-17T07:15:12.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.479+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f828401a070 con 0x7f828c157de0 2024-04-17T07:15:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.480+0000 7f8283400640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 0x7f826806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.480+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5709+0+0 (secure 0 0 0) 0x7f828408ab60 con 0x7f828c157de0 2024-04-17T07:15:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.481+0000 7f828b400640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 0x7f826806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.482+0000 7f828b400640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 0x7f826806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82700057e0 tx=0x7f827c039040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.485+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8284058ef0 con 0x7f828c157de0 2024-04-17T07:15:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.683+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f828c1f85d0 con 0x7f828c157de0 2024-04-17T07:15:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.686+0000 7f8283400640 1 -- 172.21.15.67:0/1637928929 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+16634 (secure 0 0 0) 0x7f82840587f0 con 0x7f828c157de0 2024-04-17T07:15:12.685 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:12.686 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:15:11.212695+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.688+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 msgr2=0x7f826806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.688+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 0x7f826806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82700057e0 tx=0x7f827c039040 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.688+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 msgr2=0x7f828c158260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.688+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f828c0b28c0 tx=0x7f828400cb90 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 shutdown_connections 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f828c157de0 0x7f828c158260 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f826806c920 0x7f826806ede0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828c0b13b0 0x7f828c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 --2- 172.21.15.67:0/1637928929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82700039e0 0x7f828c153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 >> 172.21.15.67:0/1637928929 conn(0x7f828c06bf50 msgr2=0x7f828c15ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 shutdown_connections 2024-04-17T07:15:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.689+0000 7f8292600640 1 -- 172.21.15.67:0/1637928929 wait complete. 2024-04-17T07:15:12.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:15:12.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.922+0000 7ff101200640 1 Processor -- start 2024-04-17T07:15:12.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.922+0000 7ff101200640 1 -- start start 2024-04-17T07:15:12.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.924+0000 7ff101200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0fc0cd920 0x7ff0fc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.924+0000 7ff101200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc146360 con 0x7ff0fc145f80 2024-04-17T07:15:12.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.924+0000 7ff101200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc071e70 con 0x7ff0fc06c0a0 2024-04-17T07:15:12.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.924+0000 7ff101200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0727a0 con 0x7ff0fc0cd920 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.924+0000 7ff0faa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0fc0cd920 0x7ff0fc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.925+0000 7ff0fb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff0fc145f80 0x7ff0fc0cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48204/0 (socket says 172.21.15.67:48204) 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.925+0000 7ff0fb400640 1 -- 172.21.15.67:0/299546506 learned_addr learned my addr 172.21.15.67:0/299546506 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.925+0000 7ff0faa00640 1 -- 172.21.15.67:0/299546506 >> v1:172.21.15.187:6789/0 conn(0x7ff0fc06c0a0 legacy=0x7ff0fc06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.925+0000 7ff0faa00640 1 -- 172.21.15.67:0/299546506 >> v1:172.21.15.67:6789/0 conn(0x7ff0fc145f80 legacy=0x7ff0fc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.925+0000 7ff0faa00640 1 -- 172.21.15.67:0/299546506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc072150 con 0x7ff0fc0cd920 2024-04-17T07:15:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.926+0000 7ff0faa00640 1 --2- 172.21.15.67:0/299546506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc06bb60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097a0 tx=0x7ff0e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37922c7d5d05d114 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.926+0000 7ff0fa000640 1 -- 172.21.15.67:0/299546506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80315a0 con 0x7ff0fc0cd920 2024-04-17T07:15:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff0fa000640 1 -- 172.21.15.67:0/299546506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8037070 con 0x7ff0fc0cd920 2024-04-17T07:15:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff0fa000640 1 -- 172.21.15.67:0/299546506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8031960 con 0x7ff0fc0cd920 2024-04-17T07:15:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 -- 172.21.15.67:0/299546506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 msgr2=0x7ff0fc06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 --2- 172.21.15.67:0/299546506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc06bb60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097a0 tx=0x7ff0e802b840 comp rx=0 tx=0).stop 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 -- 172.21.15.67:0/299546506 shutdown_connections 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 --2- 172.21.15.67:0/299546506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc06bb60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 -- 172.21.15.67:0/299546506 >> 172.21.15.67:0/299546506 conn(0x7ff0fc0b2190 msgr2=0x7ff0fc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 -- 172.21.15.67:0/299546506 shutdown_connections 2024-04-17T07:15:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.927+0000 7ff101200640 1 -- 172.21.15.67:0/299546506 wait complete. 2024-04-17T07:15:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.929+0000 7ff101200640 1 Processor -- start 2024-04-17T07:15:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.929+0000 7ff101200640 1 -- start start 2024-04-17T07:15:12.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc06c0a0 0x7ff0fc1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc145f80 0x7ff0fc0abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc072c30 con 0x7ff0fc0cd920 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc06c530 con 0x7ff0fc06c0a0 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff101200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc072db0 con 0x7ff0fc145f80 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff0faa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.930+0000 7ff0faa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42496/0 (socket says 172.21.15.67:42496) 2024-04-17T07:15:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 -- 172.21.15.67:0/860066877 learned_addr learned my addr 172.21.15.67:0/860066877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 -- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc145f80 msgr2=0x7ff0fc0abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc145f80 0x7ff0fc0abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 -- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc06c0a0 msgr2=0x7ff0fc1e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc06c0a0 0x7ff0fc1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 -- 172.21.15.67:0/860066877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0ac520 con 0x7ff0fc0cd920 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0faa00640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80098c0 tx=0x7ff0e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.931+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8032840 con 0x7ff0fc0cd920 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.932+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8032a00 con 0x7ff0fc0cd920 2024-04-17T07:15:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.932+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc0ac800 con 0x7ff0fc0cd920 2024-04-17T07:15:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.932+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8032df0 con 0x7ff0fc0cd920 2024-04-17T07:15:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.932+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc0ada10 con 0x7ff0fc0cd920 2024-04-17T07:15:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.933+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc069120 con 0x7ff0fc0cd920 2024-04-17T07:15:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.934+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0e803b3f0 con 0x7ff0fc0cd920 2024-04-17T07:15:12.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.935+0000 7ff0f3400640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:12.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.935+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7ff0e802f030 con 0x7ff0fc0cd920 2024-04-17T07:15:12.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.936+0000 7ff0fb400640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.937+0000 7ff0fb400640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f60 tx=0x7ff0ec007a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:12.940+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff0fc069120 con 0x7ff0fc0cd920 2024-04-17T07:15:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.188+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff0fc0af950 con 0x7ff0fc0cd920 2024-04-17T07:15:13.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.785+0000 7ff0f3400640 1 -- 172.21.15.67:0/860066877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7ff0e8040070 con 0x7ff0fc0cd920 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.788+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 msgr2=0x7ff0dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.788+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f60 tx=0x7ff0ec007a80 comp rx=0 tx=0).stop 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.788+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 msgr2=0x7ff0fc0ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.788+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80098c0 tx=0x7ff0e80046c0 comp rx=0 tx=0).stop 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 shutdown_connections 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff0dc06c8f0 0x7ff0dc06edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc145f80 0x7ff0fc0abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0cd920 0x7ff0fc0ab9b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:13.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 --2- 172.21.15.67:0/860066877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc06c0a0 0x7ff0fc1e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:13.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 >> 172.21.15.67:0/860066877 conn(0x7ff0fc0b2190 msgr2=0x7ff0fc15cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:13.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 shutdown_connections 2024-04-17T07:15:13.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:13.789+0000 7ff101200640 1 -- 172.21.15.67:0/860066877 wait complete. 2024-04-17T07:15:13.813 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:15:13.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:15:14.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.038+0000 7fac67200640 1 Processor -- start 2024-04-17T07:15:14.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.038+0000 7fac67200640 1 -- start start 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac67200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac680d0ec0 0x7fac681602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac67200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6806e600 con 0x7fac6806f0a0 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac67200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac68072b90 con 0x7fac680cde60 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac67200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fac68153c50 con 0x7fac680d0ec0 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac65400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac6806f0a0 0x7fac680d07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36142/0 (socket says 172.21.15.67:36142) 2024-04-17T07:15:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.040+0000 7fac65400640 1 -- 172.21.15.67:0/2399824671 learned_addr learned my addr 172.21.15.67:0/2399824671 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:14.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac66800640 1 --2- 172.21.15.67:0/2399824671 >> v2:172.21.15.187:3300/0 conn(0x7fac680d0ec0 0x7fac681602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2271329512 0 0) 0x7fac6806e600 con 0x7fac6806f0a0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c003520 con 0x7fac6806f0a0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2978945604 0 0) 0x7fac4c003520 con 0x7fac6806f0a0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 >> v2:172.21.15.187:3300/0 conn(0x7fac680d0ec0 msgr2=0x7fac681602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 --2- 172.21.15.67:0/2399824671 >> v2:172.21.15.187:3300/0 conn(0x7fac680d0ec0 0x7fac681602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 >> v1:172.21.15.187:6789/0 conn(0x7fac680cde60 legacy=0x7fac680720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac68072e70 con 0x7fac6806f0a0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fac54001690 con 0x7fac6806f0a0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 >> v1:172.21.15.67:6789/0 conn(0x7fac6806f0a0 legacy=0x7fac680d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac4c004800 0x7fac4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac4c0051c0 0x7fac4c00fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac68153c50 con 0x7fac4c0039b0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c010570 con 0x7fac4c0051c0 2024-04-17T07:15:14.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.041+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c010770 con 0x7fac4c004800 2024-04-17T07:15:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac66800640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac4c0051c0 0x7fac4c00fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65400640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac4c004800 0x7fac4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 -- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac4c004800 msgr2=0x7fac4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:14.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac4c004800 0x7fac4c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 -- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac4c0051c0 msgr2=0x7fac4c00fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac4c0051c0 0x7fac4c00fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac65e00640 1 -- 172.21.15.67:0/2399824671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c003520 con 0x7fac4c0039b0 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.042+0000 7fac66800640 1 -- 172.21.15.67:0/2399824671 reap_dead start 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.043+0000 7fac65e00640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fac58001750 tx=0x7fac4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfefc4f76d1d0c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58001370 con 0x7fac4c0039b0 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac580052d0 con 0x7fac4c0039b0 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac64a00640 1 -- 172.21.15.67:0/2399824671 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58003830 con 0x7fac4c0039b0 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 -- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 msgr2=0x7fac4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fac58001750 tx=0x7fac4c005830 comp rx=0 tx=0).stop 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 -- 172.21.15.67:0/2399824671 shutdown_connections 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 --2- 172.21.15.67:0/2399824671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac4c003dd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 -- 172.21.15.67:0/2399824671 >> 172.21.15.67:0/2399824671 conn(0x7fac6806b8c0 msgr2=0x7fac6806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.044+0000 7fac67200640 1 -- 172.21.15.67:0/2399824671 shutdown_connections 2024-04-17T07:15:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.045+0000 7fac67200640 1 -- 172.21.15.67:0/2399824671 wait complete. 2024-04-17T07:15:14.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.046+0000 7fac67200640 1 Processor -- start 2024-04-17T07:15:14.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.046+0000 7fac67200640 1 -- start start 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.047+0000 7fac67200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.047+0000 7fac67200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac68146250 0x7fac680b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac67200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 0x7fac680b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac67200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac6806eb80 con 0x7fac4c0039b0 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac67200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac6806f530 con 0x7fac680b1920 2024-04-17T07:15:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac67200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac6806c5b0 con 0x7fac68146250 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46484/0 (socket says 172.21.15.67:46484) 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 -- 172.21.15.67:0/3072804435 learned_addr learned my addr 172.21.15.67:0/3072804435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65400640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac68146250 0x7fac680b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac66800640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 0x7fac680b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 -- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac68146250 msgr2=0x7fac680b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac68146250 0x7fac680b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 -- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 msgr2=0x7fac680b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 0x7fac680b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac65e00640 1 -- 172.21.15.67:0/3072804435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac68153f70 con 0x7fac4c0039b0 2024-04-17T07:15:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.048+0000 7fac66800640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 0x7fac680b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac65e00640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac58005040 tx=0x7fac58001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58041070 con 0x7fac4c0039b0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac58005480 con 0x7fac4c0039b0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac68154250 con 0x7fac4c0039b0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58007a30 con 0x7fac4c0039b0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.049+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac68154810 con 0x7fac4c0039b0 2024-04-17T07:15:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.050+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac68068a10 con 0x7fac4c0039b0 2024-04-17T07:15:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.051+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac58044070 con 0x7fac4c0039b0 2024-04-17T07:15:14.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.052+0000 7fac5e000640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 0x7fac4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:14.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.052+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fac580ad690 con 0x7fac4c0039b0 2024-04-17T07:15:14.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.053+0000 7fac65400640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 0x7fac4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.054+0000 7fac65400640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 0x7fac4806edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fac54001800 tx=0x7fac54009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.057+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fac5807baa0 con 0x7fac4c0039b0 2024-04-17T07:15:14.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.301+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fac681514d0 con 0x7fac4c0039b0 2024-04-17T07:15:14.705 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T07:15:14.706+0000 7fd419600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:15:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.796+0000 7fac5e000640 1 -- 172.21.15.67:0/3072804435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7fac5807b3a0 con 0x7fac4c0039b0 2024-04-17T07:15:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 msgr2=0x7fac4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 0x7fac4806edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fac54001800 tx=0x7fac54009040 comp rx=0 tx=0).stop 2024-04-17T07:15:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 msgr2=0x7fac681459e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fac58005040 tx=0x7fac58001120 comp rx=0 tx=0).stop 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 shutdown_connections 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac680b1920 0x7fac680b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fac4806c8f0 0x7fac4806edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac68146250 0x7fac680b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 --2- 172.21.15.67:0/3072804435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4c0039b0 0x7fac681459e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 >> 172.21.15.67:0/3072804435 conn(0x7fac6806b8c0 msgr2=0x7fac68153070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.800+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 shutdown_connections 2024-04-17T07:15:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:14.801+0000 7fac67200640 1 -- 172.21.15.67:0/3072804435 wait complete. 2024-04-17T07:15:14.821 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:15:14.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:15:15.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.038+0000 7f24ce600640 1 Processor -- start 2024-04-17T07:15:15.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.038+0000 7f24ce600640 1 -- start start 2024-04-17T07:15:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.040+0000 7f24ce600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.040+0000 7f24ce600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24c8141ed0 0x7f24c8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.040+0000 7f24ce600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c80cd4a0 con 0x7f24c8160280 2024-04-17T07:15:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.040+0000 7f24ce600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f24c815ce30 con 0x7f24c8141ed0 2024-04-17T07:15:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.040+0000 7f24ce600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24c81607f0 con 0x7f24c806b8c0 2024-04-17T07:15:15.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.041+0000 7f24cd200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.041+0000 7f24cdc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24c8141ed0 0x7f24c8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.041+0000 7f24c7e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f24c8160280 0x7f24c806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:37470/0 (socket says 172.21.15.67:37470) 2024-04-17T07:15:15.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.041+0000 7f24c7e00640 1 -- 172.21.15.67:0/2068475525 learned_addr learned my addr 172.21.15.67:0/2068475525 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.042+0000 7f24cdc00640 1 -- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 msgr2=0x7f24c80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.042+0000 7f24cdc00640 1 --2- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c80cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.042+0000 7f24cdc00640 1 -- 172.21.15.67:0/2068475525 >> v1:172.21.15.187:6789/0 conn(0x7f24c8160280 legacy=0x7f24c806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.042+0000 7f24cdc00640 1 -- 172.21.15.67:0/2068475525 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c8160ad0 con 0x7f24c8141ed0 2024-04-17T07:15:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.044+0000 7f24cdc00640 1 --2- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c8142360 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009d90 tx=0x7f24c002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccfe1fa8cbabc2f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.044+0000 7f24c7400640 1 -- 172.21.15.67:0/2068475525 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0039070 con 0x7f24c8141ed0 2024-04-17T07:15:15.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.044+0000 7f24c7400640 1 -- 172.21.15.67:0/2068475525 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c002cc60 con 0x7f24c8141ed0 2024-04-17T07:15:15.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.044+0000 7f24c7400640 1 -- 172.21.15.67:0/2068475525 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0034660 con 0x7f24c8141ed0 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 -- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 msgr2=0x7f24c8142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 --2- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c8142360 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009d90 tx=0x7f24c002beb0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 -- 172.21.15.67:0/2068475525 shutdown_connections 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 --2- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c8142360 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 --2- 172.21.15.67:0/2068475525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c80cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 -- 172.21.15.67:0/2068475525 >> 172.21.15.67:0/2068475525 conn(0x7f24c8067d10 msgr2=0x7f24c80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 -- 172.21.15.67:0/2068475525 shutdown_connections 2024-04-17T07:15:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.045+0000 7f24ce600640 1 -- 172.21.15.67:0/2068475525 wait complete. 2024-04-17T07:15:15.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.047+0000 7f24ce600640 1 Processor -- start 2024-04-17T07:15:15.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.047+0000 7f24ce600640 1 -- start start 2024-04-17T07:15:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c813ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24c8161090 con 0x7f24c8160280 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f24c80cddf0 con 0x7f24c8141ed0 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24ce600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24c813f9b0 con 0x7f24c806b8c0 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24cdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46486/0 (socket says 172.21.15.67:46486) 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24cdc00640 1 -- 172.21.15.67:0/3825235317 learned_addr learned my addr 172.21.15.67:0/3825235317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24c7e00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.049+0000 7f24cd200640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c813ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 -- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 msgr2=0x7f24c813ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c813ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 -- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 msgr2=0x7f24c815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 -- 172.21.15.67:0/3825235317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c806f9e0 con 0x7f24c8160280 2024-04-17T07:15:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24c7e00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cd200640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c813ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24cdc00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f24c002c4c0 tx=0x7f24c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0044020 con 0x7f24c8160280 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c003a040 con 0x7f24c8160280 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0039070 con 0x7f24c8160280 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.050+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c8142fc0 con 0x7f24c8160280 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.051+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c8143580 con 0x7f24c8160280 2024-04-17T07:15:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.052+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c8068a10 con 0x7f24c8160280 2024-04-17T07:15:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.053+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24c0047020 con 0x7f24c8160280 2024-04-17T07:15:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.054+0000 7f24c4c00640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 0x7f24a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.054+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f24c00ae4e0 con 0x7f24c8160280 2024-04-17T07:15:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.055+0000 7f24cd200640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 0x7f24a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.056+0000 7f24cd200640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 0x7f24a806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009ac0 tx=0x7f24bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.060+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f24c004e030 con 0x7f24c8160280 2024-04-17T07:15:15.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.311+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24c8142560 con 0x7f24c8160280 2024-04-17T07:15:15.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.312+0000 7f24c4c00640 1 -- 172.21.15.67:0/3825235317 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7f24c007c900 con 0x7f24c8160280 2024-04-17T07:15:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:15.311 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:15:15.311 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 msgr2=0x7f24a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 0x7f24a806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009ac0 tx=0x7f24bc008040 comp rx=0 tx=0).stop 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 msgr2=0x7f24c815bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f24c002c4c0 tx=0x7f24c0004a40 comp rx=0 tx=0).stop 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 shutdown_connections 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8160280 0x7f24c815bb30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24c8141ed0 0x7f24c815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f24a806c8f0 0x7f24a806edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 --2- 172.21.15.67:0/3825235317 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24c806b8c0 0x7f24c813ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 >> 172.21.15.67:0/3825235317 conn(0x7f24c8067d10 msgr2=0x7f24c806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 shutdown_connections 2024-04-17T07:15:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.315+0000 7f24ce600640 1 -- 172.21.15.67:0/3825235317 wait complete. 2024-04-17T07:15:15.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:15:15.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.548+0000 7f9560a00640 1 Processor -- start 2024-04-17T07:15:15.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.548+0000 7f9560a00640 1 -- start start 2024-04-17T07:15:15.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f9560a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f955c15b570 0x7f955c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f9560a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f955c06fce0 0x7f955c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f9560a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955c15bc80 con 0x7f955c06be70 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f9560a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f955c15bf40 con 0x7f955c15b570 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f9560a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f955c15c220 con 0x7f955c06fce0 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f955b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f955c15b570 0x7f955c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f955b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f955c15b570 0x7f955c06f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46500/0 (socket says 172.21.15.67:46500) 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f955b400640 1 -- 172.21.15.67:0/1459886934 learned_addr learned my addr 172.21.15.67:0/1459886934 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.550+0000 7f955be00640 1 --1- 172.21.15.67:0/1459886934 >> v1:172.21.15.67:6789/0 conn(0x7f955c06be70 0x7f955c06ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36144/0 (socket says 172.21.15.67:36144) 2024-04-17T07:15:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.551+0000 7f955b400640 1 -- 172.21.15.67:0/1459886934 >> v2:172.21.15.187:3300/0 conn(0x7f955c06fce0 msgr2=0x7f955c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.551+0000 7f955b400640 1 --2- 172.21.15.67:0/1459886934 >> v2:172.21.15.187:3300/0 conn(0x7f955c06fce0 0x7f955c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.551+0000 7f955b400640 1 -- 172.21.15.67:0/1459886934 >> v1:172.21.15.67:6789/0 conn(0x7f955c06be70 legacy=0x7f955c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.551+0000 7f955b400640 1 -- 172.21.15.67:0/1459886934 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955c15c500 con 0x7f955c15b570 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.552+0000 7f955b400640 1 --2- 172.21.15.67:0/1459886934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c15b570 0x7f955c06f7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f955c073040 tx=0x7f954c02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7384e6dde69014e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.552+0000 7f955aa00640 1 -- 172.21.15.67:0/1459886934 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c02f8e0 con 0x7f955c15b570 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.552+0000 7f955aa00640 1 -- 172.21.15.67:0/1459886934 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954c02fa80 con 0x7f955c15b570 2024-04-17T07:15:15.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.552+0000 7f955aa00640 1 -- 172.21.15.67:0/1459886934 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c02fe50 con 0x7f955c15b570 2024-04-17T07:15:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 -- 172.21.15.67:0/1459886934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c15b570 msgr2=0x7f955c06f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 --2- 172.21.15.67:0/1459886934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c15b570 0x7f955c06f7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f955c073040 tx=0x7f954c02db20 comp rx=0 tx=0).stop 2024-04-17T07:15:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 -- 172.21.15.67:0/1459886934 shutdown_connections 2024-04-17T07:15:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 --2- 172.21.15.67:0/1459886934 >> v2:172.21.15.187:3300/0 conn(0x7f955c06fce0 0x7f955c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 --2- 172.21.15.67:0/1459886934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c15b570 0x7f955c06f7a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 -- 172.21.15.67:0/1459886934 >> 172.21.15.67:0/1459886934 conn(0x7f955c067d10 msgr2=0x7f955c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 -- 172.21.15.67:0/1459886934 shutdown_connections 2024-04-17T07:15:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.553+0000 7f9560a00640 1 -- 172.21.15.67:0/1459886934 wait complete. 2024-04-17T07:15:15.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.555+0000 7f9560a00640 1 Processor -- start 2024-04-17T07:15:15.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.555+0000 7f9560a00640 1 -- start start 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 0x7f955c14baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 0x7f955c0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f955c0b4930 con 0x7f955c06be70 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f955c0b47b0 con 0x7f955c15b570 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f9560a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f955c06c2a0 con 0x7f955c06fce0 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f955b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f955b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46516/0 (socket says 172.21.15.67:46516) 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.556+0000 7f955b400640 1 -- 172.21.15.67:0/2814320314 learned_addr learned my addr 172.21.15.67:0/2814320314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955be00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 0x7f955c0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 -- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 msgr2=0x7f955c14baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f9553e00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 0x7f955c14baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 0x7f955c14baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 -- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 msgr2=0x7f955c0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 0x7f955c0b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 -- 172.21.15.67:0/2814320314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955c1f84f0 con 0x7f955c06be70 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955be00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 0x7f955c0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f9553e00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 0x7f955c14baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f955b400640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f954c02db60 tx=0x7f954c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.557+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c046020 con 0x7f955c06be70 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.558+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954c037d30 con 0x7f955c06be70 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.558+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c0426b0 con 0x7f955c06be70 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.558+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955c1f87d0 con 0x7f955c06be70 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.558+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955c1f81e0 con 0x7f955c06be70 2024-04-17T07:15:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.559+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f955c068a10 con 0x7f955c06be70 2024-04-17T07:15:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.560+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f954c049020 con 0x7f955c06be70 2024-04-17T07:15:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.561+0000 7f9553400640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.561+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f954c0aea30 con 0x7f955c06be70 2024-04-17T07:15:15.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.562+0000 7f9553e00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:15.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.563+0000 7f9553e00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 0x7f953806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9544006fd0 tx=0x7f9544008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:15.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.566+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f954c07cdc0 con 0x7f955c06be70 2024-04-17T07:15:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.772+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f955c06e2e0 con 0x7f955c06be70 2024-04-17T07:15:15.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.774+0000 7f9553400640 1 -- 172.21.15.67:0/2814320314 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+16621 (secure 0 0 0) 0x7f954c07c6c0 con 0x7f955c06be70 2024-04-17T07:15:15.774 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:15.774 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:15:12.769426+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:15:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 msgr2=0x7f953806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 0x7f953806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9544006fd0 tx=0x7f9544008040 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 msgr2=0x7f955c1f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f954c02db60 tx=0x7f954c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 shutdown_connections 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955c15b570 0x7f955c0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f955c06fce0 0x7f955c14baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 --2- 172.21.15.67:0/2814320314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f955c06be70 0x7f955c1f7ca0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.777+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 >> 172.21.15.67:0/2814320314 conn(0x7f955c067d10 msgr2=0x7f955c06e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.778+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 shutdown_connections 2024-04-17T07:15:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:15.778+0000 7f9560a00640 1 -- 172.21.15.67:0/2814320314 wait complete. 2024-04-17T07:15:15.796 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:15:15.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:15:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.019+0000 7fdb57e00640 1 Processor -- start 2024-04-17T07:15:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.019+0000 7fdb57e00640 1 -- start start 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb5815cac0 0x7fdb580d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb58160460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdb580d10e0 0x7fdb581609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5809b650 con 0x7fdb5815cac0 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5809aa60 con 0x7fdb580d10e0 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.020+0000 7fdb57e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5809ad20 con 0x7fdb5809bae0 2024-04-17T07:15:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb5815cac0 0x7fdb580d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb5815cac0 0x7fdb580d0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46518/0 (socket says 172.21.15.67:46518) 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 -- 172.21.15.67:0/2330063331 learned_addr learned my addr 172.21.15.67:0/2330063331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56000640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb58160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb57400640 1 --2- 172.21.15.67:0/2330063331 >> v2:172.21.15.187:3300/0 conn(0x7fdb580d10e0 0x7fdb581609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 -- 172.21.15.67:0/2330063331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 msgr2=0x7fdb58160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb58160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 -- 172.21.15.67:0/2330063331 >> v2:172.21.15.187:3300/0 conn(0x7fdb580d10e0 msgr2=0x7fdb581609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2330063331 >> v2:172.21.15.187:3300/0 conn(0x7fdb580d10e0 0x7fdb581609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.021+0000 7fdb56a00640 1 -- 172.21.15.67:0/2330063331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5809b000 con 0x7fdb5815cac0 2024-04-17T07:15:16.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.022+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb580d0ba0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdb58073ce0 tx=0x7fdb4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a43e7c4b763e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.023+0000 7fdb55600640 1 -- 172.21.15.67:0/2330063331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4802ca80 con 0x7fdb5815cac0 2024-04-17T07:15:16.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.023+0000 7fdb55600640 1 -- 172.21.15.67:0/2330063331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4802cc20 con 0x7fdb5815cac0 2024-04-17T07:15:16.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.023+0000 7fdb55600640 1 -- 172.21.15.67:0/2330063331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb480343f0 con 0x7fdb5815cac0 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.023+0000 7fdb57e00640 1 -- 172.21.15.67:0/2330063331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 msgr2=0x7fdb580d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.023+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb580d0ba0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdb58073ce0 tx=0x7fdb4802bee0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 -- 172.21.15.67:0/2330063331 shutdown_connections 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2330063331 >> v2:172.21.15.187:3300/0 conn(0x7fdb580d10e0 0x7fdb581609a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb58160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2330063331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb580d0ba0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 -- 172.21.15.67:0/2330063331 >> 172.21.15.67:0/2330063331 conn(0x7fdb580b40e0 msgr2=0x7fdb580b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 -- 172.21.15.67:0/2330063331 shutdown_connections 2024-04-17T07:15:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.024+0000 7fdb57e00640 1 -- 172.21.15.67:0/2330063331 wait complete. 2024-04-17T07:15:16.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.025+0000 7fdb57e00640 1 Processor -- start 2024-04-17T07:15:16.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.026+0000 7fdb57e00640 1 -- start start 2024-04-17T07:15:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb581f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb580d10e0 0x7fdb580b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb580cd090 con 0x7fdb5815cac0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdb580b1bc0 con 0x7fdb580d10e0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb580cd210 con 0x7fdb5809bae0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46532/0 (socket says 172.21.15.67:46532) 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 -- 172.21.15.67:0/2129932748 learned_addr learned my addr 172.21.15.67:0/2129932748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb56000640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb580d10e0 0x7fdb580b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb581f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 -- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 msgr2=0x7fdb581f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb581f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 -- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb580d10e0 msgr2=0x7fdb580b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb580d10e0 0x7fdb580b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.027+0000 7fdb57400640 1 -- 172.21.15.67:0/2129932748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb581f8650 con 0x7fdb5815cac0 2024-04-17T07:15:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb57400640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000c7f0 tx=0x7fdb5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50015070 con 0x7fdb5815cac0 2024-04-17T07:15:16.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb50004220 con 0x7fdb5815cac0 2024-04-17T07:15:16.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb581f88d0 con 0x7fdb5815cac0 2024-04-17T07:15:16.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50007620 con 0x7fdb5815cac0 2024-04-17T07:15:16.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.028+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb581f8e90 con 0x7fdb5815cac0 2024-04-17T07:15:16.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.029+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb580b6fd0 con 0x7fdb5815cac0 2024-04-17T07:15:16.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.030+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb500079d0 con 0x7fdb5815cac0 2024-04-17T07:15:16.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.031+0000 7fdb4ea00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 0x7fdb3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.031+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fdb5008bc20 con 0x7fdb5815cac0 2024-04-17T07:15:16.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.032+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 0x7fdb3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.033+0000 7fdb56a00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 0x7fdb3406edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb48035000 tx=0x7fdb48030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.036+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdb581f8650 con 0x7fdb5815cac0 2024-04-17T07:15:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.237+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb580755d0 con 0x7fdb3406c8f0 2024-04-17T07:15:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.258+0000 7fdb4ea00640 1 -- 172.21.15.67:0/2129932748 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239854 (secure 0 0 0) 0x7fdb580755d0 con 0x7fdb3406c8f0 2024-04-17T07:15:16.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:16.262 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:15:16.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.263+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 msgr2=0x7fdb3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.263+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 0x7fdb3406edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb48035000 tx=0x7fdb48030000 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.263+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 msgr2=0x7fdb581f5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.263+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000c7f0 tx=0x7fdb5000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 shutdown_connections 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb5815cac0 0x7fdb581f5650 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb580d10e0 0x7fdb580b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fdb3406c8f0 0x7fdb3406edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 --2- 172.21.15.67:0/2129932748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb5809bae0 0x7fdb581f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 >> 172.21.15.67:0/2129932748 conn(0x7fdb580b40e0 msgr2=0x7fdb580b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 shutdown_connections 2024-04-17T07:15:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.264+0000 7fdb57e00640 1 -- 172.21.15.67:0/2129932748 wait complete. 2024-04-17T07:15:16.282 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-17T07:15:15.381033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224268,"kb_used_data":3920,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750452,"statfs":{"total":773094113280,"available":772864462848,"internally_reserved":0,"allocated":4014080,"data_stored":2250430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.664368"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736436+0000","last_change":"2024-04-17T07:15:09.736436+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736436+0000","last_undegraded":"2024-04-17T07:15:09.736436+0000","last_fullsized":"2024-04-17T07:15:09.736436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723359+0000","last_change":"2024-04-17T07:15:09.723359+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723359+0000","last_undegraded":"2024-04-17T07:15:09.723359+0000","last_fullsized":"2024-04-17T07:15:09.723359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727967+0000","last_change":"2024-04-17T07:15:09.727967+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727967+0000","last_undegraded":"2024-04-17T07:15:09.727967+0000","last_fullsized":"2024-04-17T07:15:09.727967+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.729246+0000","last_change":"2024-04-17T07:15:09.729246+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.729246+0000","last_undegraded":"2024-04-17T07:15:09.729246+0000","last_fullsized":"2024-04-17T07:15:09.729246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.732906+0000","last_change":"2024-04-17T07:15:09.732906+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.732906+0000","last_undegraded":"2024-04-17T07:15:09.732906+0000","last_fullsized":"2024-04-17T07:15:09.732906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724544+0000","last_change":"2024-04-17T07:15:09.724544+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724544+0000","last_undegraded":"2024-04-17T07:15:09.724544+0000","last_fullsized":"2024-04-17T07:15:09.724544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727100+0000","last_change":"2024-04-17T07:15:09.727100+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727100+0000","last_undegraded":"2024-04-17T07:15:09.727100+0000","last_fullsized":"2024-04-17T07:15:09.727100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722588+0000","last_change":"2024-04-17T07:15:09.722588+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722588+0000","last_undegraded":"2024-04-17T07:15:09.722588+0000","last_fullsized":"2024-04-17T07:15:09.722588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726615+0000","last_change":"2024-04-17T07:15:09.726615+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726615+0000","last_undegraded":"2024-04-17T07:15:09.726615+0000","last_fullsized":"2024-04-17T07:15:09.726615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722005+0000","last_change":"2024-04-17T07:15:09.722005+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722005+0000","last_undegraded":"2024-04-17T07:15:09.722005+0000","last_fullsized":"2024-04-17T07:15:09.722005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722334+0000","last_change":"2024-04-17T07:15:09.722334+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722334+0000","last_undegraded":"2024-04-17T07:15:09.722334+0000","last_fullsized":"2024-04-17T07:15:09.722334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.730625+0000","last_change":"2024-04-17T07:15:09.730625+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.730625+0000","last_undegraded":"2024-04-17T07:15:09.730625+0000","last_fullsized":"2024-04-17T07:15:09.730625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.731518+0000","last_change":"2024-04-17T07:15:09.731518+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.731518+0000","last_undegraded":"2024-04-17T07:15:09.731518+0000","last_fullsized":"2024-04-17T07:15:09.731518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725054+0000","last_change":"2024-04-17T07:15:09.725054+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725054+0000","last_undegraded":"2024-04-17T07:15:09.725054+0000","last_fullsized":"2024-04-17T07:15:09.725054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726493+0000","last_change":"2024-04-17T07:15:09.726493+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726493+0000","last_undegraded":"2024-04-17T07:15:09.726493+0000","last_fullsized":"2024-04-17T07:15:09.726493+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726854+0000","last_change":"2024-04-17T07:15:09.726854+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726854+0000","last_undegraded":"2024-04-17T07:15:09.726854+0000","last_fullsized":"2024-04-17T07:15:09.726854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725478+0000","last_change":"2024-04-17T07:15:09.725478+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725478+0000","last_undegraded":"2024-04-17T07:15:09.725478+0000","last_fullsized":"2024-04-17T07:15:09.725478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725562+0000","last_change":"2024-04-17T07:15:09.725562+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725562+0000","last_undegraded":"2024-04-17T07:15:09.725562+0000","last_fullsized":"2024-04-17T07:15:09.725562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725285+0000","last_change":"2024-04-17T07:15:09.725285+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725285+0000","last_undegraded":"2024-04-17T07:15:09.725285+0000","last_fullsized":"2024-04-17T07:15:09.725285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726693+0000","last_change":"2024-04-17T07:15:09.726693+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726693+0000","last_undegraded":"2024-04-17T07:15:09.726693+0000","last_fullsized":"2024-04-17T07:15:09.726693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.721447+0000","last_change":"2024-04-17T07:15:09.721447+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.721447+0000","last_undegraded":"2024-04-17T07:15:09.721447+0000","last_fullsized":"2024-04-17T07:15:09.721447+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736392+0000","last_change":"2024-04-17T07:15:09.736392+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736392+0000","last_undegraded":"2024-04-17T07:15:09.736392+0000","last_fullsized":"2024-04-17T07:15:09.736392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724546+0000","last_change":"2024-04-17T07:15:09.724546+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724546+0000","last_undegraded":"2024-04-17T07:15:09.724546+0000","last_fullsized":"2024-04-17T07:15:09.724546+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723432+0000","last_change":"2024-04-17T07:15:09.723432+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723432+0000","last_undegraded":"2024-04-17T07:15:09.723432+0000","last_fullsized":"2024-04-17T07:15:09.723432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724527+0000","last_change":"2024-04-17T07:15:09.724527+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724527+0000","last_undegraded":"2024-04-17T07:15:09.724527+0000","last_fullsized":"2024-04-17T07:15:09.724527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720472+0000","last_change":"2024-04-17T07:15:09.720472+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720472+0000","last_undegraded":"2024-04-17T07:15:09.720472+0000","last_fullsized":"2024-04-17T07:15:09.720472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724543+0000","last_change":"2024-04-17T07:15:09.724543+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724543+0000","last_undegraded":"2024-04-17T07:15:09.724543+0000","last_fullsized":"2024-04-17T07:15:09.724543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720830+0000","last_change":"2024-04-17T07:15:09.720830+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720830+0000","last_undegraded":"2024-04-17T07:15:09.720830+0000","last_fullsized":"2024-04-17T07:15:09.720830+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724410+0000","last_change":"2024-04-17T07:15:09.724410+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724410+0000","last_undegraded":"2024-04-17T07:15:09.724410+0000","last_fullsized":"2024-04-17T07:15:09.724410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722674+0000","last_change":"2024-04-17T07:15:09.722674+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722674+0000","last_undegraded":"2024-04-17T07:15:09.722674+0000","last_fullsized":"2024-04-17T07:15:09.722674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726192+0000","last_change":"2024-04-17T07:15:09.726192+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726192+0000","last_undegraded":"2024-04-17T07:15:09.726192+0000","last_fullsized":"2024-04-17T07:15:09.726192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736252+0000","last_change":"2024-04-17T07:15:09.736252+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736252+0000","last_undegraded":"2024-04-17T07:15:09.736252+0000","last_fullsized":"2024-04-17T07:15:09.736252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-17T07:15:09.730330+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:15:09.730330+0000","last_peered":"2024-04-17T07:15:09.730330+0000","last_clean":"2024-04-17T07:15:09.730330+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:15:09.730330+0000","last_undegraded":"2024-04-17T07:15:09.730330+0000","last_fullsized":"2024-04-17T07:15:09.730330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":344064,"data_stored":129349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":937984,"data_stored":719717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:15:16.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:15:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.536+0000 7f1160a00640 1 Processor -- start 2024-04-17T07:15:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.536+0000 7f1160a00640 1 -- start start 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.537+0000 7f1160a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f115c0cf640 0x7f115c0cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.537+0000 7f1160a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f115c0b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.537+0000 7f1160a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115c0cff60 con 0x7f115c15ed10 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.537+0000 7f1160a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f115c0d3c30 con 0x7f115c0cf640 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.537+0000 7f1160a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f115c0d3020 con 0x7f115c0d2ab0 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f1159400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f115c15ed10 0x7f115c0d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36152/0 (socket says 172.21.15.67:36152) 2024-04-17T07:15:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f1159400640 1 -- 172.21.15.67:0/920012135 learned_addr learned my addr 172.21.15.67:0/920012135 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f1159e00640 1 --2- 172.21.15.67:0/920012135 >> v2:172.21.15.187:3300/0 conn(0x7f115c0cf640 0x7f115c0cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f115a800640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f115c0b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417713088 0 0) 0x7f115c0cff60 con 0x7f115c15ed10 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.538+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1140003520 con 0x7f115c15ed10 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3538587938 0 0) 0x7f1140003520 con 0x7f115c15ed10 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 msgr2=0x7f115c0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f115c0b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 >> v2:172.21.15.187:3300/0 conn(0x7f115c0cf640 msgr2=0x7f115c0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 --2- 172.21.15.67:0/920012135 >> v2:172.21.15.187:3300/0 conn(0x7f115c0cf640 0x7f115c0cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c0d3300 con 0x7f115c15ed10 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1148001670 con 0x7f115c15ed10 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 >> v1:172.21.15.67:6789/0 conn(0x7f115c15ed10 legacy=0x7f115c0d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f1140003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f1140004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1140004d40 0x7f11400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0d3c30 con 0x7f11400039b0 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0d3020 con 0x7f1140004d40 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f114000b9f0 con 0x7f115c0d2ab0 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1159e00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f1140003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.539+0000 7f1159400640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f1140004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f115a800640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1140004d40 0x7f11400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f1159e00640 1 -- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 msgr2=0x7f1140004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f1159e00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0d2ab0 0x7f1140004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f1159e00640 1 -- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1140004d40 msgr2=0x7f11400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f1159e00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1140004d40 0x7f11400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.540+0000 7f1159e00640 1 -- 172.21.15.67:0/920012135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1140003520 con 0x7f11400039b0 2024-04-17T07:15:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.541+0000 7f1159e00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f1140003dd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f114c009ac0 tx=0x7f1140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439c687a14aded37 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.541+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c039070 con 0x7f11400039b0 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.541+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c02c710 con 0x7f11400039b0 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.541+0000 7f1158a00640 1 -- 172.21.15.67:0/920012135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c009690 con 0x7f11400039b0 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 -- 172.21.15.67:0/920012135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 msgr2=0x7f1140003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 --2- 172.21.15.67:0/920012135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f1140003dd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f114c009ac0 tx=0x7f1140005830 comp rx=0 tx=0).stop 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f115a800640 1 -- 172.21.15.67:0/920012135 reap_dead start 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 -- 172.21.15.67:0/920012135 shutdown_connections 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 -- 172.21.15.67:0/920012135 >> 172.21.15.67:0/920012135 conn(0x7f115c06cbd0 msgr2=0x7f115c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 -- 172.21.15.67:0/920012135 shutdown_connections 2024-04-17T07:15:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.542+0000 7f1160a00640 1 -- 172.21.15.67:0/920012135 wait complete. 2024-04-17T07:15:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.544+0000 7f1160a00640 1 Processor -- start 2024-04-17T07:15:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.544+0000 7f1160a00640 1 -- start start 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f115c1f4c30 0x7f115c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0b8cb0 0x7f115c0b9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0d25b0 con 0x7f11400039b0 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0d38c0 con 0x7f115c1f4c30 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1160a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f115c0ba300 con 0x7f115c0b8cb0 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46556/0 (socket says 172.21.15.67:46556) 2024-04-17T07:15:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 -- 172.21.15.67:0/2435165256 learned_addr learned my addr 172.21.15.67:0/2435165256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 -- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0b8cb0 msgr2=0x7f115c0b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0b8cb0 0x7f115c0b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 -- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f115c1f4c30 msgr2=0x7f115c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159400640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f115c1f4c30 0x7f115c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.545+0000 7f1159e00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f115c1f4c30 0x7f115c157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1159e00640 1 -- 172.21.15.67:0/2435165256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c1f9160 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1159e00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f114c009be0 tx=0x7f114c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c042020 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c03a040 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c039070 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115c1f51a0 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.546+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115c1f5760 con 0x7f11400039b0 2024-04-17T07:15:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.548+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c068b90 con 0x7f11400039b0 2024-04-17T07:15:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.549+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f114c045020 con 0x7f11400039b0 2024-04-17T07:15:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.550+0000 7f1152000640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 0x7f113c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.550+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f114c0acb50 con 0x7f11400039b0 2024-04-17T07:15:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.551+0000 7f1159400640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 0x7f113c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.552+0000 7f1159400640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 0x7f113c06ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1148001050 tx=0x7f1148001190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.554+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f114c04c020 con 0x7f11400039b0 2024-04-17T07:15:16.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.752+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f115c0b40b0 con 0x7f113c06c920 2024-04-17T07:15:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.770+0000 7f1152000640 1 -- 172.21.15.67:0/2435165256 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239854 (secure 0 0 0) 0x7f115c0b40b0 con 0x7f113c06c920 2024-04-17T07:15:16.773 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:16.774 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:15:16.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 msgr2=0x7f113c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 0x7f113c06ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1148001050 tx=0x7f1148001190 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 msgr2=0x7f115c1f8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f114c009be0 tx=0x7f114c004730 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 shutdown_connections 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f113c06c920 0x7f113c06ede0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f115c0b8cb0 0x7f115c0b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f115c1f4c30 0x7f115c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 --2- 172.21.15.67:0/2435165256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11400039b0 0x7f115c1f8c20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.776+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 >> 172.21.15.67:0/2435165256 conn(0x7f115c06cbd0 msgr2=0x7f115c0b7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.777+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 shutdown_connections 2024-04-17T07:15:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:16.777+0000 7f1160a00640 1 -- 172.21.15.67:0/2435165256 wait complete. 2024-04-17T07:15:16.795 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-17T07:15:15.381033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224268,"kb_used_data":3920,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750452,"statfs":{"total":773094113280,"available":772864462848,"internally_reserved":0,"allocated":4014080,"data_stored":2250430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.664368"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736436+0000","last_change":"2024-04-17T07:15:09.736436+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736436+0000","last_undegraded":"2024-04-17T07:15:09.736436+0000","last_fullsized":"2024-04-17T07:15:09.736436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723359+0000","last_change":"2024-04-17T07:15:09.723359+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723359+0000","last_undegraded":"2024-04-17T07:15:09.723359+0000","last_fullsized":"2024-04-17T07:15:09.723359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727967+0000","last_change":"2024-04-17T07:15:09.727967+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727967+0000","last_undegraded":"2024-04-17T07:15:09.727967+0000","last_fullsized":"2024-04-17T07:15:09.727967+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.729246+0000","last_change":"2024-04-17T07:15:09.729246+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.729246+0000","last_undegraded":"2024-04-17T07:15:09.729246+0000","last_fullsized":"2024-04-17T07:15:09.729246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.732906+0000","last_change":"2024-04-17T07:15:09.732906+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.732906+0000","last_undegraded":"2024-04-17T07:15:09.732906+0000","last_fullsized":"2024-04-17T07:15:09.732906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724544+0000","last_change":"2024-04-17T07:15:09.724544+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724544+0000","last_undegraded":"2024-04-17T07:15:09.724544+0000","last_fullsized":"2024-04-17T07:15:09.724544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727100+0000","last_change":"2024-04-17T07:15:09.727100+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727100+0000","last_undegraded":"2024-04-17T07:15:09.727100+0000","last_fullsized":"2024-04-17T07:15:09.727100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722588+0000","last_change":"2024-04-17T07:15:09.722588+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722588+0000","last_undegraded":"2024-04-17T07:15:09.722588+0000","last_fullsized":"2024-04-17T07:15:09.722588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726615+0000","last_change":"2024-04-17T07:15:09.726615+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726615+0000","last_undegraded":"2024-04-17T07:15:09.726615+0000","last_fullsized":"2024-04-17T07:15:09.726615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722005+0000","last_change":"2024-04-17T07:15:09.722005+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722005+0000","last_undegraded":"2024-04-17T07:15:09.722005+0000","last_fullsized":"2024-04-17T07:15:09.722005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722334+0000","last_change":"2024-04-17T07:15:09.722334+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722334+0000","last_undegraded":"2024-04-17T07:15:09.722334+0000","last_fullsized":"2024-04-17T07:15:09.722334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.730625+0000","last_change":"2024-04-17T07:15:09.730625+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.730625+0000","last_undegraded":"2024-04-17T07:15:09.730625+0000","last_fullsized":"2024-04-17T07:15:09.730625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.731518+0000","last_change":"2024-04-17T07:15:09.731518+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.731518+0000","last_undegraded":"2024-04-17T07:15:09.731518+0000","last_fullsized":"2024-04-17T07:15:09.731518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725054+0000","last_change":"2024-04-17T07:15:09.725054+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725054+0000","last_undegraded":"2024-04-17T07:15:09.725054+0000","last_fullsized":"2024-04-17T07:15:09.725054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726493+0000","last_change":"2024-04-17T07:15:09.726493+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726493+0000","last_undegraded":"2024-04-17T07:15:09.726493+0000","last_fullsized":"2024-04-17T07:15:09.726493+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726854+0000","last_change":"2024-04-17T07:15:09.726854+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726854+0000","last_undegraded":"2024-04-17T07:15:09.726854+0000","last_fullsized":"2024-04-17T07:15:09.726854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725478+0000","last_change":"2024-04-17T07:15:09.725478+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725478+0000","last_undegraded":"2024-04-17T07:15:09.725478+0000","last_fullsized":"2024-04-17T07:15:09.725478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725562+0000","last_change":"2024-04-17T07:15:09.725562+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725562+0000","last_undegraded":"2024-04-17T07:15:09.725562+0000","last_fullsized":"2024-04-17T07:15:09.725562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725285+0000","last_change":"2024-04-17T07:15:09.725285+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725285+0000","last_undegraded":"2024-04-17T07:15:09.725285+0000","last_fullsized":"2024-04-17T07:15:09.725285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726693+0000","last_change":"2024-04-17T07:15:09.726693+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726693+0000","last_undegraded":"2024-04-17T07:15:09.726693+0000","last_fullsized":"2024-04-17T07:15:09.726693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.721447+0000","last_change":"2024-04-17T07:15:09.721447+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.721447+0000","last_undegraded":"2024-04-17T07:15:09.721447+0000","last_fullsized":"2024-04-17T07:15:09.721447+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736392+0000","last_change":"2024-04-17T07:15:09.736392+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736392+0000","last_undegraded":"2024-04-17T07:15:09.736392+0000","last_fullsized":"2024-04-17T07:15:09.736392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724546+0000","last_change":"2024-04-17T07:15:09.724546+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724546+0000","last_undegraded":"2024-04-17T07:15:09.724546+0000","last_fullsized":"2024-04-17T07:15:09.724546+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723432+0000","last_change":"2024-04-17T07:15:09.723432+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723432+0000","last_undegraded":"2024-04-17T07:15:09.723432+0000","last_fullsized":"2024-04-17T07:15:09.723432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724527+0000","last_change":"2024-04-17T07:15:09.724527+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724527+0000","last_undegraded":"2024-04-17T07:15:09.724527+0000","last_fullsized":"2024-04-17T07:15:09.724527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720472+0000","last_change":"2024-04-17T07:15:09.720472+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720472+0000","last_undegraded":"2024-04-17T07:15:09.720472+0000","last_fullsized":"2024-04-17T07:15:09.720472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724543+0000","last_change":"2024-04-17T07:15:09.724543+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724543+0000","last_undegraded":"2024-04-17T07:15:09.724543+0000","last_fullsized":"2024-04-17T07:15:09.724543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720830+0000","last_change":"2024-04-17T07:15:09.720830+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720830+0000","last_undegraded":"2024-04-17T07:15:09.720830+0000","last_fullsized":"2024-04-17T07:15:09.720830+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724410+0000","last_change":"2024-04-17T07:15:09.724410+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724410+0000","last_undegraded":"2024-04-17T07:15:09.724410+0000","last_fullsized":"2024-04-17T07:15:09.724410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722674+0000","last_change":"2024-04-17T07:15:09.722674+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722674+0000","last_undegraded":"2024-04-17T07:15:09.722674+0000","last_fullsized":"2024-04-17T07:15:09.722674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726192+0000","last_change":"2024-04-17T07:15:09.726192+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726192+0000","last_undegraded":"2024-04-17T07:15:09.726192+0000","last_fullsized":"2024-04-17T07:15:09.726192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736252+0000","last_change":"2024-04-17T07:15:09.736252+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736252+0000","last_undegraded":"2024-04-17T07:15:09.736252+0000","last_fullsized":"2024-04-17T07:15:09.736252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-17T07:15:09.730330+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:15:09.730330+0000","last_peered":"2024-04-17T07:15:09.730330+0000","last_clean":"2024-04-17T07:15:09.730330+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:15:09.730330+0000","last_undegraded":"2024-04-17T07:15:09.730330+0000","last_fullsized":"2024-04-17T07:15:09.730330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":344064,"data_stored":129349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":937984,"data_stored":719717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:15:16.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:15:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.067+0000 7fe8dbe00640 1 Processor -- start 2024-04-17T07:15:17.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.067+0000 7fe8dbe00640 1 -- start start 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.068+0000 7fe8dbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8d406f670 0x7fe8d406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8dbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d4072b30 con 0x7fe8d40cd020 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8dbe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8d406ff90 con 0x7fe8d406f030 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8dbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8d40ceb00 con 0x7fe8d406f670 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8d406f670 0x7fe8d406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d8a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8d40cd020 0x7fe8d4071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36156/0 (socket says 172.21.15.67:36156) 2024-04-17T07:15:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d8a00640 1 -- 172.21.15.67:0/578194674 learned_addr learned my addr 172.21.15.67:0/578194674 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d9400640 1 -- 172.21.15.67:0/578194674 >> v1:172.21.15.187:6789/0 conn(0x7fe8d406f030 legacy=0x7fe8d4072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d9400640 1 -- 172.21.15.67:0/578194674 >> v1:172.21.15.67:6789/0 conn(0x7fe8d40cd020 legacy=0x7fe8d4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d9400640 1 -- 172.21.15.67:0/578194674 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d40cede0 con 0x7fe8d406f670 2024-04-17T07:15:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.069+0000 7fe8d3e00640 1 -- 172.21.15.67:0/578194674 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482862593 0 0) 0x7fe8d4072b30 con 0x7fe8d40cd020 2024-04-17T07:15:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8d9400640 1 --2- 172.21.15.67:0/578194674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d406f670 0x7fe8d406fa50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80097a0 tx=0x7fe8c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7cb52482c0545 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8d3e00640 1 -- 172.21.15.67:0/578194674 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8035020 con 0x7fe8d406f670 2024-04-17T07:15:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8d3e00640 1 -- 172.21.15.67:0/578194674 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8c8031830 con 0x7fe8d406f670 2024-04-17T07:15:17.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8d3e00640 1 -- 172.21.15.67:0/578194674 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8031c00 con 0x7fe8d406f670 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8dbe00640 1 -- 172.21.15.67:0/578194674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d406f670 msgr2=0x7fe8d406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/578194674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d406f670 0x7fe8d406fa50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80097a0 tx=0x7fe8c802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8dbe00640 1 -- 172.21.15.67:0/578194674 shutdown_connections 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/578194674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d406f670 0x7fe8d406fa50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.071+0000 7fe8dbe00640 1 -- 172.21.15.67:0/578194674 >> 172.21.15.67:0/578194674 conn(0x7fe8d406bf50 msgr2=0x7fe8d40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.072+0000 7fe8dbe00640 1 -- 172.21.15.67:0/578194674 shutdown_connections 2024-04-17T07:15:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.072+0000 7fe8dbe00640 1 -- 172.21.15.67:0/578194674 wait complete. 2024-04-17T07:15:17.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.073+0000 7fe8dbe00640 1 Processor -- start 2024-04-17T07:15:17.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.073+0000 7fe8dbe00640 1 -- start start 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 0x7fe8d4154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d406f670 0x7fe8d415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d40cf680 con 0x7fe8d40cd020 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d40cf3a0 con 0x7fe8d406f670 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.074+0000 7fe8dbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d40ce720 con 0x7fe8d406f030 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46584/0 (socket says 172.21.15.67:46584) 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 -- 172.21.15.67:0/2397349845 learned_addr learned my addr 172.21.15.67:0/2397349845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9400640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 0x7fe8d4154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d8a00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d406f670 0x7fe8d415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 -- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 msgr2=0x7fe8d4154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 0x7fe8d4154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 -- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d406f670 msgr2=0x7fe8d415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d406f670 0x7fe8d415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 -- 172.21.15.67:0/2397349845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d415cb00 con 0x7fe8d40cd020 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9400640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 0x7fe8d4154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.075+0000 7fe8d9e00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00c780 tx=0x7fe8cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.076+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc004b60 con 0x7fe8d40cd020 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.076+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8cc004d00 con 0x7fe8d40cd020 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.076+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc00aa90 con 0x7fe8d40cd020 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.076+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d4154b90 con 0x7fe8d40cd020 2024-04-17T07:15:17.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.076+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d40c8490 con 0x7fe8d40cd020 2024-04-17T07:15:17.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.077+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d4068b90 con 0x7fe8d40cd020 2024-04-17T07:15:17.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.078+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8cc00ac30 con 0x7fe8d40cd020 2024-04-17T07:15:17.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.079+0000 7fe8d1600640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 0x7fe8b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.079+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fe8cc08be70 con 0x7fe8d40cd020 2024-04-17T07:15:17.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.080+0000 7fe8d9400640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 0x7fe8b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.081+0000 7fe8d9400640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 0x7fe8b406ede0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8036000 tx=0x7fe8c8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.083+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe8cc020070 con 0x7fe8d40cd020 2024-04-17T07:15:17.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.287+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8d40d0ee0 con 0x7fe8b406c920 2024-04-17T07:15:17.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.306+0000 7fe8d1600640 1 -- 172.21.15.67:0/2397349845 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239854 (secure 0 0 0) 0x7fe8d40d0ee0 con 0x7fe8b406c920 2024-04-17T07:15:17.307 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:17.309 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:15:17.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 msgr2=0x7fe8b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 0x7fe8b406ede0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8036000 tx=0x7fe8c8008030 comp rx=0 tx=0).stop 2024-04-17T07:15:17.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 msgr2=0x7fe8d4154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc00c780 tx=0x7fe8cc00cc40 comp rx=0 tx=0).stop 2024-04-17T07:15:17.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 shutdown_connections 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d40cd020 0x7fe8d4154de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8d406f670 0x7fe8d415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe8b406c920 0x7fe8b406ede0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 --2- 172.21.15.67:0/2397349845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8d406f030 0x7fe8d4154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 >> 172.21.15.67:0/2397349845 conn(0x7fe8d406bf50 msgr2=0x7fe8d40caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 shutdown_connections 2024-04-17T07:15:17.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.312+0000 7fe8dbe00640 1 -- 172.21.15.67:0/2397349845 wait complete. 2024-04-17T07:15:17.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-17T07:15:15.381033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224268,"kb_used_data":3920,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750452,"statfs":{"total":773094113280,"available":772864462848,"internally_reserved":0,"allocated":4014080,"data_stored":2250430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.664368"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736436+0000","last_change":"2024-04-17T07:15:09.736436+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736436+0000","last_undegraded":"2024-04-17T07:15:09.736436+0000","last_fullsized":"2024-04-17T07:15:09.736436+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723359+0000","last_change":"2024-04-17T07:15:09.723359+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723359+0000","last_undegraded":"2024-04-17T07:15:09.723359+0000","last_fullsized":"2024-04-17T07:15:09.723359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727967+0000","last_change":"2024-04-17T07:15:09.727967+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727967+0000","last_undegraded":"2024-04-17T07:15:09.727967+0000","last_fullsized":"2024-04-17T07:15:09.727967+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.729246+0000","last_change":"2024-04-17T07:15:09.729246+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.729246+0000","last_undegraded":"2024-04-17T07:15:09.729246+0000","last_fullsized":"2024-04-17T07:15:09.729246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.732906+0000","last_change":"2024-04-17T07:15:09.732906+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.732906+0000","last_undegraded":"2024-04-17T07:15:09.732906+0000","last_fullsized":"2024-04-17T07:15:09.732906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724544+0000","last_change":"2024-04-17T07:15:09.724544+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724544+0000","last_undegraded":"2024-04-17T07:15:09.724544+0000","last_fullsized":"2024-04-17T07:15:09.724544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.727100+0000","last_change":"2024-04-17T07:15:09.727100+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.727100+0000","last_undegraded":"2024-04-17T07:15:09.727100+0000","last_fullsized":"2024-04-17T07:15:09.727100+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722588+0000","last_change":"2024-04-17T07:15:09.722588+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722588+0000","last_undegraded":"2024-04-17T07:15:09.722588+0000","last_fullsized":"2024-04-17T07:15:09.722588+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726615+0000","last_change":"2024-04-17T07:15:09.726615+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726615+0000","last_undegraded":"2024-04-17T07:15:09.726615+0000","last_fullsized":"2024-04-17T07:15:09.726615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722005+0000","last_change":"2024-04-17T07:15:09.722005+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722005+0000","last_undegraded":"2024-04-17T07:15:09.722005+0000","last_fullsized":"2024-04-17T07:15:09.722005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722334+0000","last_change":"2024-04-17T07:15:09.722334+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722334+0000","last_undegraded":"2024-04-17T07:15:09.722334+0000","last_fullsized":"2024-04-17T07:15:09.722334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.730625+0000","last_change":"2024-04-17T07:15:09.730625+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.730625+0000","last_undegraded":"2024-04-17T07:15:09.730625+0000","last_fullsized":"2024-04-17T07:15:09.730625+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.731518+0000","last_change":"2024-04-17T07:15:09.731518+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.731518+0000","last_undegraded":"2024-04-17T07:15:09.731518+0000","last_fullsized":"2024-04-17T07:15:09.731518+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725054+0000","last_change":"2024-04-17T07:15:09.725054+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725054+0000","last_undegraded":"2024-04-17T07:15:09.725054+0000","last_fullsized":"2024-04-17T07:15:09.725054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726493+0000","last_change":"2024-04-17T07:15:09.726493+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726493+0000","last_undegraded":"2024-04-17T07:15:09.726493+0000","last_fullsized":"2024-04-17T07:15:09.726493+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726854+0000","last_change":"2024-04-17T07:15:09.726854+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726854+0000","last_undegraded":"2024-04-17T07:15:09.726854+0000","last_fullsized":"2024-04-17T07:15:09.726854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725478+0000","last_change":"2024-04-17T07:15:09.725478+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725478+0000","last_undegraded":"2024-04-17T07:15:09.725478+0000","last_fullsized":"2024-04-17T07:15:09.725478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725562+0000","last_change":"2024-04-17T07:15:09.725562+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725562+0000","last_undegraded":"2024-04-17T07:15:09.725562+0000","last_fullsized":"2024-04-17T07:15:09.725562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.725285+0000","last_change":"2024-04-17T07:15:09.725285+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.725285+0000","last_undegraded":"2024-04-17T07:15:09.725285+0000","last_fullsized":"2024-04-17T07:15:09.725285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726693+0000","last_change":"2024-04-17T07:15:09.726693+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726693+0000","last_undegraded":"2024-04-17T07:15:09.726693+0000","last_fullsized":"2024-04-17T07:15:09.726693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.721447+0000","last_change":"2024-04-17T07:15:09.721447+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.721447+0000","last_undegraded":"2024-04-17T07:15:09.721447+0000","last_fullsized":"2024-04-17T07:15:09.721447+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736392+0000","last_change":"2024-04-17T07:15:09.736392+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736392+0000","last_undegraded":"2024-04-17T07:15:09.736392+0000","last_fullsized":"2024-04-17T07:15:09.736392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724546+0000","last_change":"2024-04-17T07:15:09.724546+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724546+0000","last_undegraded":"2024-04-17T07:15:09.724546+0000","last_fullsized":"2024-04-17T07:15:09.724546+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.723432+0000","last_change":"2024-04-17T07:15:09.723432+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.723432+0000","last_undegraded":"2024-04-17T07:15:09.723432+0000","last_fullsized":"2024-04-17T07:15:09.723432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724527+0000","last_change":"2024-04-17T07:15:09.724527+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724527+0000","last_undegraded":"2024-04-17T07:15:09.724527+0000","last_fullsized":"2024-04-17T07:15:09.724527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720472+0000","last_change":"2024-04-17T07:15:09.720472+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720472+0000","last_undegraded":"2024-04-17T07:15:09.720472+0000","last_fullsized":"2024-04-17T07:15:09.720472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724543+0000","last_change":"2024-04-17T07:15:09.724543+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724543+0000","last_undegraded":"2024-04-17T07:15:09.724543+0000","last_fullsized":"2024-04-17T07:15:09.724543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.720830+0000","last_change":"2024-04-17T07:15:09.720830+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.720830+0000","last_undegraded":"2024-04-17T07:15:09.720830+0000","last_fullsized":"2024-04-17T07:15:09.720830+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.724410+0000","last_change":"2024-04-17T07:15:09.724410+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.724410+0000","last_undegraded":"2024-04-17T07:15:09.724410+0000","last_fullsized":"2024-04-17T07:15:09.724410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.722674+0000","last_change":"2024-04-17T07:15:09.722674+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.722674+0000","last_undegraded":"2024-04-17T07:15:09.722674+0000","last_fullsized":"2024-04-17T07:15:09.722674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.726192+0000","last_change":"2024-04-17T07:15:09.726192+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.726192+0000","last_undegraded":"2024-04-17T07:15:09.726192+0000","last_fullsized":"2024-04-17T07:15:09.726192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T07:15:10.713413+0000","last_change":"2024-04-17T07:15:10.713413+0000","last_active":"2024-04-17T07:15:10.713413+0000","last_peered":"2024-04-17T07:15:10.713413+0000","last_clean":"2024-04-17T07:15:10.713413+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:10.713413+0000","last_undegraded":"2024-04-17T07:15:10.713413+0000","last_fullsized":"2024-04-17T07:15:10.713413+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-04-17T07:15:09.736252+0000","last_change":"2024-04-17T07:15:09.736252+0000","last_active":"2024-04-17T07:15:09.707694+0000","last_peered":"2024-04-17T07:15:09.707694+0000","last_clean":"2024-04-17T07:15:09.707694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T07:15:09.736252+0000","last_undegraded":"2024-04-17T07:15:09.736252+0000","last_fullsized":"2024-04-17T07:15:09.736252+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-17T07:15:09.730330+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:15:09.730330+0000","last_peered":"2024-04-17T07:15:09.730330+0000","last_clean":"2024-04-17T07:15:09.730330+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:15:09.730330+0000","last_undegraded":"2024-04-17T07:15:09.730330+0000","last_fullsized":"2024-04-17T07:15:09.730330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":344064,"data_stored":129349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":937984,"data_stored":719717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-04-17T07:15:17.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-04-17T07:15:17.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1f is not active+clean 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.736436+0000', 'last_change': '2024-04-17T07:15:09.736436+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.736436+0000', 'last_undegraded': '2024-04-17T07:15:09.736436+0000', 'last_fullsized': '2024-04-17T07:15:09.736436+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.723359+0000', 'last_change': '2024-04-17T07:15:09.723359+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.723359+0000', 'last_undegraded': '2024-04-17T07:15:09.723359+0000', 'last_fullsized': '2024-04-17T07:15:09.723359+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2024-04-17T07:15:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.727967+0000', 'last_change': '2024-04-17T07:15:09.727967+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.727967+0000', 'last_undegraded': '2024-04-17T07:15:09.727967+0000', 'last_fullsized': '2024-04-17T07:15:09.727967+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.729246+0000', 'last_change': '2024-04-17T07:15:09.729246+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.729246+0000', 'last_undegraded': '2024-04-17T07:15:09.729246+0000', 'last_fullsized': '2024-04-17T07:15:09.729246+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1b is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.732906+0000', 'last_change': '2024-04-17T07:15:09.732906+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.732906+0000', 'last_undegraded': '2024-04-17T07:15:09.732906+0000', 'last_fullsized': '2024-04-17T07:15:09.732906+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1a is not active+clean 2024-04-17T07:15:17.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.724544+0000', 'last_change': '2024-04-17T07:15:09.724544+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.724544+0000', 'last_undegraded': '2024-04-17T07:15:09.724544+0000', 'last_fullsized': '2024-04-17T07:15:09.724544+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.727100+0000', 'last_change': '2024-04-17T07:15:09.727100+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.727100+0000', 'last_undegraded': '2024-04-17T07:15:09.727100+0000', 'last_fullsized': '2024-04-17T07:15:09.727100+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.18 is not active+clean 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.722588+0000', 'last_change': '2024-04-17T07:15:09.722588+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.722588+0000', 'last_undegraded': '2024-04-17T07:15:09.722588+0000', 'last_fullsized': '2024-04-17T07:15:09.722588+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-04-17T07:15:17.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.726615+0000', 'last_change': '2024-04-17T07:15:09.726615+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.726615+0000', 'last_undegraded': '2024-04-17T07:15:09.726615+0000', 'last_fullsized': '2024-04-17T07:15:09.726615+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.722005+0000', 'last_change': '2024-04-17T07:15:09.722005+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.722005+0000', 'last_undegraded': '2024-04-17T07:15:09.722005+0000', 'last_fullsized': '2024-04-17T07:15:09.722005+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.15 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.722334+0000', 'last_change': '2024-04-17T07:15:09.722334+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.722334+0000', 'last_undegraded': '2024-04-17T07:15:09.722334+0000', 'last_fullsized': '2024-04-17T07:15:09.722334+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.730625+0000', 'last_change': '2024-04-17T07:15:09.730625+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.730625+0000', 'last_undegraded': '2024-04-17T07:15:09.730625+0000', 'last_fullsized': '2024-04-17T07:15:09.730625+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.13 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.731518+0000', 'last_change': '2024-04-17T07:15:09.731518+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.731518+0000', 'last_undegraded': '2024-04-17T07:15:09.731518+0000', 'last_fullsized': '2024-04-17T07:15:09.731518+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.725054+0000', 'last_change': '2024-04-17T07:15:09.725054+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.725054+0000', 'last_undegraded': '2024-04-17T07:15:09.725054+0000', 'last_fullsized': '2024-04-17T07:15:09.725054+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.726493+0000', 'last_change': '2024-04-17T07:15:09.726493+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.726493+0000', 'last_undegraded': '2024-04-17T07:15:09.726493+0000', 'last_fullsized': '2024-04-17T07:15:09.726493+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.726854+0000', 'last_change': '2024-04-17T07:15:09.726854+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.726854+0000', 'last_undegraded': '2024-04-17T07:15:09.726854+0000', 'last_fullsized': '2024-04-17T07:15:09.726854+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.725478+0000', 'last_change': '2024-04-17T07:15:09.725478+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.725478+0000', 'last_undegraded': '2024-04-17T07:15:09.725478+0000', 'last_fullsized': '2024-04-17T07:15:09.725478+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.345 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.e is not active+clean 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.725562+0000', 'last_change': '2024-04-17T07:15:09.725562+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.725562+0000', 'last_undegraded': '2024-04-17T07:15:09.725562+0000', 'last_fullsized': '2024-04-17T07:15:09.725562+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.d is not active+clean 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.725285+0000', 'last_change': '2024-04-17T07:15:09.725285+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.725285+0000', 'last_undegraded': '2024-04-17T07:15:09.725285+0000', 'last_fullsized': '2024-04-17T07:15:09.725285+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.c is not active+clean 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.726693+0000', 'last_change': '2024-04-17T07:15:09.726693+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.726693+0000', 'last_undegraded': '2024-04-17T07:15:09.726693+0000', 'last_fullsized': '2024-04-17T07:15:09.726693+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.346 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.721447+0000', 'last_change': '2024-04-17T07:15:09.721447+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.721447+0000', 'last_undegraded': '2024-04-17T07:15:09.721447+0000', 'last_fullsized': '2024-04-17T07:15:09.721447+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.a is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.736392+0000', 'last_change': '2024-04-17T07:15:09.736392+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.736392+0000', 'last_undegraded': '2024-04-17T07:15:09.736392+0000', 'last_fullsized': '2024-04-17T07:15:09.736392+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.9 is not active+clean 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.724546+0000', 'last_change': '2024-04-17T07:15:09.724546+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.724546+0000', 'last_undegraded': '2024-04-17T07:15:09.724546+0000', 'last_fullsized': '2024-04-17T07:15:09.724546+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.347 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.8 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.723432+0000', 'last_change': '2024-04-17T07:15:09.723432+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.723432+0000', 'last_undegraded': '2024-04-17T07:15:09.723432+0000', 'last_fullsized': '2024-04-17T07:15:09.723432+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.724527+0000', 'last_change': '2024-04-17T07:15:09.724527+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.724527+0000', 'last_undegraded': '2024-04-17T07:15:09.724527+0000', 'last_fullsized': '2024-04-17T07:15:09.724527+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2024-04-17T07:15:17.348 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.720472+0000', 'last_change': '2024-04-17T07:15:09.720472+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.720472+0000', 'last_undegraded': '2024-04-17T07:15:09.720472+0000', 'last_fullsized': '2024-04-17T07:15:09.720472+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.724543+0000', 'last_change': '2024-04-17T07:15:09.724543+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.724543+0000', 'last_undegraded': '2024-04-17T07:15:09.724543+0000', 'last_fullsized': '2024-04-17T07:15:09.724543+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.720830+0000', 'last_change': '2024-04-17T07:15:09.720830+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.720830+0000', 'last_undegraded': '2024-04-17T07:15:09.720830+0000', 'last_fullsized': '2024-04-17T07:15:09.720830+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.349 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.724410+0000', 'last_change': '2024-04-17T07:15:09.724410+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.724410+0000', 'last_undegraded': '2024-04-17T07:15:09.724410+0000', 'last_fullsized': '2024-04-17T07:15:09.724410+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.722674+0000', 'last_change': '2024-04-17T07:15:09.722674+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.722674+0000', 'last_undegraded': '2024-04-17T07:15:09.722674+0000', 'last_fullsized': '2024-04-17T07:15:09.722674+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.0 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.726192+0000', 'last_change': '2024-04-17T07:15:09.726192+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.726192+0000', 'last_undegraded': '2024-04-17T07:15:09.726192+0000', 'last_fullsized': '2024-04-17T07:15:09.726192+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-04-17T07:15:17.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T07:15:10.713413+0000', 'last_change': '2024-04-17T07:15:10.713413+0000', 'last_active': '2024-04-17T07:15:10.713413+0000', 'last_peered': '2024-04-17T07:15:10.713413+0000', 'last_clean': '2024-04-17T07:15:10.713413+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:10.713413+0000', 'last_undegraded': '2024-04-17T07:15:10.713413+0000', 'last_fullsized': '2024-04-17T07:15:10.713413+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:10.713413+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T07:15:17.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.5 is not active+clean 2024-04-17T07:15:17.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-04-17T07:15:09.736252+0000', 'last_change': '2024-04-17T07:15:09.736252+0000', 'last_active': '2024-04-17T07:15:09.707694+0000', 'last_peered': '2024-04-17T07:15:09.707694+0000', 'last_clean': '2024-04-17T07:15:09.707694+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T07:15:09.736252+0000', 'last_undegraded': '2024-04-17T07:15:09.736252+0000', 'last_fullsized': '2024-04-17T07:15:09.736252+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'last_clean_scrub_stamp': '2024-04-17T07:15:09.707694+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T07:15:17.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T07:15:17.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.597+0000 7f3d62200640 1 Processor -- start 2024-04-17T07:15:17.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.597+0000 7f3d62200640 1 -- start start 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d62200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c15b570 0x7f3d5c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d62200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06fce0 0x7f3d5c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d62200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5c15bc80 con 0x7f3d5c06be70 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d62200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5c15bf40 con 0x7f3d5c06fce0 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d62200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c15c220 con 0x7f3d5c15b570 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06fce0 0x7f3d5c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06fce0 0x7f3d5c06b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46600/0 (socket says 172.21.15.67:46600) 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.599+0000 7f3d5be00640 1 -- 172.21.15.67:0/1334064798 learned_addr learned my addr 172.21.15.67:0/1334064798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:17.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.600+0000 7f3d60e00640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c15b570 0x7f3d5c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.600+0000 7f3d5be00640 1 -- 172.21.15.67:0/1334064798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c15b570 msgr2=0x7f3d5c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.600+0000 7f3d5be00640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c15b570 0x7f3d5c06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.600+0000 7f3d5be00640 1 -- 172.21.15.67:0/1334064798 >> v1:172.21.15.67:6789/0 conn(0x7f3d5c06be70 legacy=0x7f3d5c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.600+0000 7f3d5be00640 1 -- 172.21.15.67:0/1334064798 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c15c500 con 0x7f3d5c06fce0 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.601+0000 7f3d5be00640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06fce0 0x7f3d5c06b930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009870 tx=0x7f3d4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa31e8d8bc96d3d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.601+0000 7f3d5b400640 1 -- 172.21.15.67:0/1334064798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48035020 con 0x7f3d5c06fce0 2024-04-17T07:15:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.601+0000 7f3d5b400640 1 -- 172.21.15.67:0/1334064798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d48031640 con 0x7f3d5c06fce0 2024-04-17T07:15:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.601+0000 7f3d5b400640 1 -- 172.21.15.67:0/1334064798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4802d040 con 0x7f3d5c06fce0 2024-04-17T07:15:17.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 -- 172.21.15.67:0/1334064798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06fce0 msgr2=0x7f3d5c06b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06fce0 0x7f3d5c06b930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009870 tx=0x7f3d4802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 -- 172.21.15.67:0/1334064798 shutdown_connections 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06fce0 0x7f3d5c06b930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 --2- 172.21.15.67:0/1334064798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c15b570 0x7f3d5c06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 -- 172.21.15.67:0/1334064798 >> 172.21.15.67:0/1334064798 conn(0x7f3d5c067d10 msgr2=0x7f3d5c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 -- 172.21.15.67:0/1334064798 shutdown_connections 2024-04-17T07:15:17.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.602+0000 7f3d62200640 1 -- 172.21.15.67:0/1334064798 wait complete. 2024-04-17T07:15:17.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.604+0000 7f3d62200640 1 Processor -- start 2024-04-17T07:15:17.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.604+0000 7f3d62200640 1 -- start start 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c06fce0 0x7f3d5c15a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d5c15b570 0x7f3d5c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c06c460 con 0x7f3d5c06be70 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c06c2e0 con 0x7f3d5c15b570 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.605+0000 7f3d62200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c0b47b0 con 0x7f3d5c06fce0 2024-04-17T07:15:17.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46612/0 (socket says 172.21.15.67:46612) 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 -- 172.21.15.67:0/4269648446 learned_addr learned my addr 172.21.15.67:0/4269648446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 -- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c06fce0 msgr2=0x7f3d5c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c06fce0 0x7f3d5c15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 -- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d5c15b570 msgr2=0x7f3d5c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d5c15b570 0x7f3d5c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 -- 172.21.15.67:0/4269648446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c15acd0 con 0x7f3d5c06be70 2024-04-17T07:15:17.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.606+0000 7f3d60e00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000d5f0 tx=0x7f3d5000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.607+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500125f0 con 0x7f3d5c06be70 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.607+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d50019070 con 0x7f3d5c06be70 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.607+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500129d0 con 0x7f3d5c06be70 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.607+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c14e9e0 con 0x7f3d5c06be70 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.607+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c14efa0 con 0x7f3d5c06be70 2024-04-17T07:15:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.608+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d5c068a10 con 0x7f3d5c06be70 2024-04-17T07:15:17.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.609+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d50012b70 con 0x7f3d5c06be70 2024-04-17T07:15:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.610+0000 7f3d58c00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 0x7f3d3c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.610+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f3d5008bff0 con 0x7f3d5c06be70 2024-04-17T07:15:17.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.611+0000 7f3d5be00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 0x7f3d3c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.612+0000 7f3d5be00640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 0x7f3d3c06ede0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d48036000 tx=0x7f3d48030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:17.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.614+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3d5c15acd0 con 0x7f3d5c06be70 2024-04-17T07:15:17.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.895+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f3d5c0b1f00 con 0x7f3d5c06be70 2024-04-17T07:15:17.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.896+0000 7f3d58c00640 1 -- 172.21.15.67:0/4269648446 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1173 (secure 0 0 0) 0x7f3d5005a380 con 0x7f3d5c06be70 2024-04-17T07:15:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:17.896 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":99,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":8,"num_up_osds":8,"osd_up_since":1713338030,"num_in_osds":8,"osd_in_since":1713338021,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":230023168,"bytes_avail":772864090112,"bytes_total":773094113280,"write_bytes_sec":2153,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5079}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-17T07:14:53.373764+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 msgr2=0x7f3d3c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 0x7f3d3c06ede0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d48036000 tx=0x7f3d48030000 comp rx=0 tx=0).stop 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 msgr2=0x7f3d5c151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000d5f0 tx=0x7f3d5000dab0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 shutdown_connections 2024-04-17T07:15:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d5c15b570 0x7f3d5c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f3d3c06c920 0x7f3d3c06ede0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d5c06fce0 0x7f3d5c15a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 --2- 172.21.15.67:0/4269648446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06be70 0x7f3d5c151c10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 >> 172.21.15.67:0/4269648446 conn(0x7f3d5c067d10 msgr2=0x7f3d5c15ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 shutdown_connections 2024-04-17T07:15:17.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:17.899+0000 7f3d62200640 1 -- 172.21.15.67:0/4269648446 wait complete. 2024-04-17T07:15:17.917 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-17T07:15:17.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:15:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.140+0000 7f39ea800640 1 Processor -- start 2024-04-17T07:15:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.141+0000 7f39ea800640 1 -- start start 2024-04-17T07:15:18.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.142+0000 7f39ea800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f39e40cfae0 0x7f39e4169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.142+0000 7f39ea800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39e40d2d30 con 0x7f39e40b6d00 2024-04-17T07:15:18.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.142+0000 7f39ea800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39e40d3010 con 0x7f39e415ec60 2024-04-17T07:15:18.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.142+0000 7f39ea800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f39e40d49c0 con 0x7f39e40cfae0 2024-04-17T07:15:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.143+0000 7f39e8a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f39e40b6d00 0x7f39e40cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36180/0 (socket says 172.21.15.67:36180) 2024-04-17T07:15:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.143+0000 7f39e8a00640 1 -- 172.21.15.67:0/496457495 learned_addr learned my addr 172.21.15.67:0/496457495 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.143+0000 7f39e9e00640 1 --2- 172.21.15.67:0/496457495 >> v2:172.21.15.187:3300/0 conn(0x7f39e40cfae0 0x7f39e4169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.143+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 709541906 0 0) 0x7f39e40d2d30 con 0x7f39e40b6d00 2024-04-17T07:15:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.143+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39cc003520 con 0x7f39e40b6d00 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3678142458 0 0) 0x7f39cc003520 con 0x7f39e40b6d00 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 >> v2:172.21.15.187:3300/0 conn(0x7f39e40cfae0 msgr2=0x7f39e4169810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 --2- 172.21.15.67:0/496457495 >> v2:172.21.15.187:3300/0 conn(0x7f39e40cfae0 0x7f39e4169810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 >> v1:172.21.15.187:6789/0 conn(0x7f39e415ec60 legacy=0x7f39e40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e40d32f0 con 0x7f39e40b6d00 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f39d0001670 con 0x7f39e40b6d00 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 >> v1:172.21.15.67:6789/0 conn(0x7f39e40b6d00 legacy=0x7f39e40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39cc003a70 0x7f39cc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39cc005210 0x7f39cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39e40d49c0 con 0x7f39cc004800 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc00c140 con 0x7f39cc003a70 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc00c340 con 0x7f39cc005210 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e8a00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.144+0000 7f39e9e00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39cc005210 0x7f39cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e9400640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39cc003a70 0x7f39cc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e8a00640 1 -- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39cc005210 msgr2=0x7f39cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e8a00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39cc005210 0x7f39cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e8a00640 1 -- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39cc003a70 msgr2=0x7f39cc003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e8a00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39cc003a70 0x7f39cc003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e8a00640 1 -- 172.21.15.67:0/496457495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc003520 con 0x7f39cc004800 2024-04-17T07:15:18.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.145+0000 7f39e9e00640 1 -- 172.21.15.67:0/496457495 reap_dead start 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.146+0000 7f39e8a00640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39cc004c80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c0e0 tx=0x7f39cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb260a32906b98c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.146+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0007c20 con 0x7f39cc004800 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.146+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d0030070 con 0x7f39cc004800 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.146+0000 7f39e3e00640 1 -- 172.21.15.67:0/496457495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d0005d70 con 0x7f39cc004800 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 -- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 msgr2=0x7f39cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39cc004c80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c0e0 tx=0x7f39cc005830 comp rx=0 tx=0).stop 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 -- 172.21.15.67:0/496457495 shutdown_connections 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 --2- 172.21.15.67:0/496457495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39cc004c80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 -- 172.21.15.67:0/496457495 >> 172.21.15.67:0/496457495 conn(0x7f39e40d15f0 msgr2=0x7f39e40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:18.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 -- 172.21.15.67:0/496457495 shutdown_connections 2024-04-17T07:15:18.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.147+0000 7f39ea800640 1 -- 172.21.15.67:0/496457495 wait complete. 2024-04-17T07:15:18.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.149+0000 7f39ea800640 1 Processor -- start 2024-04-17T07:15:18.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.149+0000 7f39ea800640 1 -- start start 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39e40c9110 0x7f39e40c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39e40caf20 0x7f39e40cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39e40d00c0 con 0x7f39cc004800 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39e40d3660 con 0x7f39e40c9110 2024-04-17T07:15:18.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.150+0000 7f39ea800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39e40d3af0 con 0x7f39e40caf20 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46624/0 (socket says 172.21.15.67:46624) 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 -- 172.21.15.67:0/1524732239 learned_addr learned my addr 172.21.15.67:0/1524732239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e8a00640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39e40c9110 0x7f39e40c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9e00640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39e40caf20 0x7f39e40cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 -- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39e40caf20 msgr2=0x7f39e40cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39e40caf20 0x7f39e40cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 -- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39e40c9110 msgr2=0x7f39e40c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39e40c9110 0x7f39e40c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 -- 172.21.15.67:0/1524732239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e40c9ad0 con 0x7f39cc004800 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.151+0000 7f39e9400640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39d8007fd0 tx=0x7f39d800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.152+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8019070 con 0x7f39cc004800 2024-04-17T07:15:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.152+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d80052e0 con 0x7f39cc004800 2024-04-17T07:15:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.152+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8003810 con 0x7f39cc004800 2024-04-17T07:15:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.152+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e41f4a00 con 0x7f39cc004800 2024-04-17T07:15:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.152+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e41f4fc0 con 0x7f39cc004800 2024-04-17T07:15:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.153+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e4068a10 con 0x7f39cc004800 2024-04-17T07:15:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.154+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39d801c070 con 0x7f39cc004800 2024-04-17T07:15:18.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.155+0000 7f39e1600640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 0x7f39c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:18.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.155+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f39d808b090 con 0x7f39cc004800 2024-04-17T07:15:18.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.156+0000 7f39e8a00640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 0x7f39c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.157+0000 7f39e8a00640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 0x7f39c406edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39d0034000 tx=0x7f39d0007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.160+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f39d8058e60 con 0x7f39cc004800 2024-04-17T07:15:18.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.360+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39e415e5d0 con 0x7f39c406c8f0 2024-04-17T07:15:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.380+0000 7f39e1600640 1 -- 172.21.15.67:0/1524732239 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7f39e415e5d0 con 0x7f39c406c8f0 2024-04-17T07:15:18.383 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 msgr2=0x7f39c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 0x7f39c406edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39d0034000 tx=0x7f39d0007000 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 msgr2=0x7f39e4158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39d8007fd0 tx=0x7f39d800e8a0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 shutdown_connections 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f39c406c8f0 0x7f39c406edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39e40caf20 0x7f39e40cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39e40c9110 0x7f39e40c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 --2- 172.21.15.67:0/1524732239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39cc004800 0x7f39e4158020 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 >> 172.21.15.67:0/1524732239 conn(0x7f39e40d15f0 msgr2=0x7f39e40b9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 shutdown_connections 2024-04-17T07:15:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:18.386+0000 7f39ea800640 1 -- 172.21.15.67:0/1524732239 wait complete. 2024-04-17T07:15:18.406 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-04-17T07:15:17.381648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224632,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750088,"statfs":{"total":773094113280,"available":772864090112,"internally_reserved":0,"allocated":4399104,"data_stored":2421620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.658338"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865661+0000","last_change":"2024-04-17T07:15:11.785362+0000","last_active":"2024-04-17T07:15:12.865661+0000","last_peered":"2024-04-17T07:15:12.865661+0000","last_clean":"2024-04-17T07:15:12.865661+0000","last_became_active":"2024-04-17T07:15:11.785026+0000","last_became_peered":"2024-04-17T07:15:11.785026+0000","last_unstale":"2024-04-17T07:15:12.865661+0000","last_undegraded":"2024-04-17T07:15:12.865661+0000","last_fullsized":"2024-04-17T07:15:12.865661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:34.187674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781692+0000","last_change":"2024-04-17T07:15:11.788735+0000","last_active":"2024-04-17T07:15:12.781692+0000","last_peered":"2024-04-17T07:15:12.781692+0000","last_clean":"2024-04-17T07:15:12.781692+0000","last_became_active":"2024-04-17T07:15:11.788305+0000","last_became_peered":"2024-04-17T07:15:11.788305+0000","last_unstale":"2024-04-17T07:15:12.781692+0000","last_undegraded":"2024-04-17T07:15:12.781692+0000","last_fullsized":"2024-04-17T07:15:12.781692+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:49.234619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787705+0000","last_change":"2024-04-17T07:15:11.781953+0000","last_active":"2024-04-17T07:15:12.787705+0000","last_peered":"2024-04-17T07:15:12.787705+0000","last_clean":"2024-04-17T07:15:12.787705+0000","last_became_active":"2024-04-17T07:15:11.781695+0000","last_became_peered":"2024-04-17T07:15:11.781695+0000","last_unstale":"2024-04-17T07:15:12.787705+0000","last_undegraded":"2024-04-17T07:15:12.787705+0000","last_fullsized":"2024-04-17T07:15:12.787705+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798770+0000","last_change":"2024-04-17T07:15:11.791526+0000","last_active":"2024-04-17T07:15:12.798770+0000","last_peered":"2024-04-17T07:15:12.798770+0000","last_clean":"2024-04-17T07:15:12.798770+0000","last_became_active":"2024-04-17T07:15:11.791197+0000","last_became_peered":"2024-04-17T07:15:11.791197+0000","last_unstale":"2024-04-17T07:15:12.798770+0000","last_undegraded":"2024-04-17T07:15:12.798770+0000","last_fullsized":"2024-04-17T07:15:12.798770+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:38.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826009+0000","last_change":"2024-04-17T07:15:11.788847+0000","last_active":"2024-04-17T07:15:12.826009+0000","last_peered":"2024-04-17T07:15:12.826009+0000","last_clean":"2024-04-17T07:15:12.826009+0000","last_became_active":"2024-04-17T07:15:11.788563+0000","last_became_peered":"2024-04-17T07:15:11.788563+0000","last_unstale":"2024-04-17T07:15:12.826009+0000","last_undegraded":"2024-04-17T07:15:12.826009+0000","last_fullsized":"2024-04-17T07:15:12.826009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:28.367949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780970+0000","last_change":"2024-04-17T07:15:11.788273+0000","last_active":"2024-04-17T07:15:12.780970+0000","last_peered":"2024-04-17T07:15:12.780970+0000","last_clean":"2024-04-17T07:15:12.780970+0000","last_became_active":"2024-04-17T07:15:11.787974+0000","last_became_peered":"2024-04-17T07:15:11.787974+0000","last_unstale":"2024-04-17T07:15:12.780970+0000","last_undegraded":"2024-04-17T07:15:12.780970+0000","last_fullsized":"2024-04-17T07:15:12.780970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789467+0000","last_change":"2024-04-17T07:15:11.788707+0000","last_active":"2024-04-17T07:15:12.789467+0000","last_peered":"2024-04-17T07:15:12.789467+0000","last_clean":"2024-04-17T07:15:12.789467+0000","last_became_active":"2024-04-17T07:15:11.788438+0000","last_became_peered":"2024-04-17T07:15:11.788438+0000","last_unstale":"2024-04-17T07:15:12.789467+0000","last_undegraded":"2024-04-17T07:15:12.789467+0000","last_fullsized":"2024-04-17T07:15:12.789467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:50.447380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787247+0000","last_change":"2024-04-17T07:15:11.791775+0000","last_active":"2024-04-17T07:15:12.787247+0000","last_peered":"2024-04-17T07:15:12.787247+0000","last_clean":"2024-04-17T07:15:12.787247+0000","last_became_active":"2024-04-17T07:15:11.791480+0000","last_became_peered":"2024-04-17T07:15:11.791480+0000","last_unstale":"2024-04-17T07:15:12.787247+0000","last_undegraded":"2024-04-17T07:15:12.787247+0000","last_fullsized":"2024-04-17T07:15:12.787247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.928000+0000","last_change":"2024-04-17T07:15:11.786623+0000","last_active":"2024-04-17T07:15:13.928000+0000","last_peered":"2024-04-17T07:15:13.928000+0000","last_clean":"2024-04-17T07:15:13.928000+0000","last_became_active":"2024-04-17T07:15:11.785422+0000","last_became_peered":"2024-04-17T07:15:11.785422+0000","last_unstale":"2024-04-17T07:15:13.928000+0000","last_undegraded":"2024-04-17T07:15:13.928000+0000","last_fullsized":"2024-04-17T07:15:13.928000+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:35.216294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798967+0000","last_change":"2024-04-17T07:15:11.780193+0000","last_active":"2024-04-17T07:15:12.798967+0000","last_peered":"2024-04-17T07:15:12.798967+0000","last_clean":"2024-04-17T07:15:12.798967+0000","last_became_active":"2024-04-17T07:15:11.779177+0000","last_became_peered":"2024-04-17T07:15:11.779177+0000","last_unstale":"2024-04-17T07:15:12.798967+0000","last_undegraded":"2024-04-17T07:15:12.798967+0000","last_fullsized":"2024-04-17T07:15:12.798967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789965+0000","last_change":"2024-04-17T07:15:11.790940+0000","last_active":"2024-04-17T07:15:12.789965+0000","last_peered":"2024-04-17T07:15:12.789965+0000","last_clean":"2024-04-17T07:15:12.789965+0000","last_became_active":"2024-04-17T07:15:11.790272+0000","last_became_peered":"2024-04-17T07:15:11.790272+0000","last_unstale":"2024-04-17T07:15:12.789965+0000","last_undegraded":"2024-04-17T07:15:12.789965+0000","last_fullsized":"2024-04-17T07:15:12.789965+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:40.742949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788688+0000","last_change":"2024-04-17T07:15:11.780496+0000","last_active":"2024-04-17T07:15:12.788688+0000","last_peered":"2024-04-17T07:15:12.788688+0000","last_clean":"2024-04-17T07:15:12.788688+0000","last_became_active":"2024-04-17T07:15:11.778226+0000","last_became_peered":"2024-04-17T07:15:11.778226+0000","last_unstale":"2024-04-17T07:15:12.788688+0000","last_undegraded":"2024-04-17T07:15:12.788688+0000","last_fullsized":"2024-04-17T07:15:12.788688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826192+0000","last_change":"2024-04-17T07:15:11.780581+0000","last_active":"2024-04-17T07:15:12.826192+0000","last_peered":"2024-04-17T07:15:12.826192+0000","last_clean":"2024-04-17T07:15:12.826192+0000","last_became_active":"2024-04-17T07:15:11.777868+0000","last_became_peered":"2024-04-17T07:15:11.777868+0000","last_unstale":"2024-04-17T07:15:12.826192+0000","last_undegraded":"2024-04-17T07:15:12.826192+0000","last_fullsized":"2024-04-17T07:15:12.826192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:02.045200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781070+0000","last_change":"2024-04-17T07:15:11.794690+0000","last_active":"2024-04-17T07:15:12.781070+0000","last_peered":"2024-04-17T07:15:12.781070+0000","last_clean":"2024-04-17T07:15:12.781070+0000","last_became_active":"2024-04-17T07:15:11.790136+0000","last_became_peered":"2024-04-17T07:15:11.790136+0000","last_unstale":"2024-04-17T07:15:12.781070+0000","last_undegraded":"2024-04-17T07:15:12.781070+0000","last_fullsized":"2024-04-17T07:15:12.781070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789910+0000","last_change":"2024-04-17T07:15:11.784337+0000","last_active":"2024-04-17T07:15:12.789910+0000","last_peered":"2024-04-17T07:15:12.789910+0000","last_clean":"2024-04-17T07:15:12.789910+0000","last_became_active":"2024-04-17T07:15:11.783994+0000","last_became_peered":"2024-04-17T07:15:11.783994+0000","last_unstale":"2024-04-17T07:15:12.789910+0000","last_undegraded":"2024-04-17T07:15:12.789910+0000","last_fullsized":"2024-04-17T07:15:12.789910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788860+0000","last_change":"2024-04-17T07:15:11.801844+0000","last_active":"2024-04-17T07:15:12.788860+0000","last_peered":"2024-04-17T07:15:12.788860+0000","last_clean":"2024-04-17T07:15:12.788860+0000","last_became_active":"2024-04-17T07:15:11.801434+0000","last_became_peered":"2024-04-17T07:15:11.801434+0000","last_unstale":"2024-04-17T07:15:12.788860+0000","last_undegraded":"2024-04-17T07:15:12.788860+0000","last_fullsized":"2024-04-17T07:15:12.788860+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788486+0000","last_change":"2024-04-17T07:15:11.787720+0000","last_active":"2024-04-17T07:15:12.788486+0000","last_peered":"2024-04-17T07:15:12.788486+0000","last_clean":"2024-04-17T07:15:12.788486+0000","last_became_active":"2024-04-17T07:15:11.787372+0000","last_became_peered":"2024-04-17T07:15:11.787372+0000","last_unstale":"2024-04-17T07:15:12.788486+0000","last_undegraded":"2024-04-17T07:15:12.788486+0000","last_fullsized":"2024-04-17T07:15:12.788486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781699+0000","last_change":"2024-04-17T07:15:11.784260+0000","last_active":"2024-04-17T07:15:12.781699+0000","last_peered":"2024-04-17T07:15:12.781699+0000","last_clean":"2024-04-17T07:15:12.781699+0000","last_became_active":"2024-04-17T07:15:11.783844+0000","last_became_peered":"2024-04-17T07:15:11.783844+0000","last_unstale":"2024-04-17T07:15:12.781699+0000","last_undegraded":"2024-04-17T07:15:12.781699+0000","last_fullsized":"2024-04-17T07:15:12.781699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:50.787356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781167+0000","last_change":"2024-04-17T07:15:11.785355+0000","last_active":"2024-04-17T07:15:12.781167+0000","last_peered":"2024-04-17T07:15:12.781167+0000","last_clean":"2024-04-17T07:15:12.781167+0000","last_became_active":"2024-04-17T07:15:11.785044+0000","last_became_peered":"2024-04-17T07:15:11.785044+0000","last_unstale":"2024-04-17T07:15:12.781167+0000","last_undegraded":"2024-04-17T07:15:12.781167+0000","last_fullsized":"2024-04-17T07:15:12.781167+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825968+0000","last_change":"2024-04-17T07:15:11.786281+0000","last_active":"2024-04-17T07:15:12.825968+0000","last_peered":"2024-04-17T07:15:12.825968+0000","last_clean":"2024-04-17T07:15:12.825968+0000","last_became_active":"2024-04-17T07:15:11.782295+0000","last_became_peered":"2024-04-17T07:15:11.782295+0000","last_unstale":"2024-04-17T07:15:12.825968+0000","last_undegraded":"2024-04-17T07:15:12.825968+0000","last_fullsized":"2024-04-17T07:15:12.825968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:26.827107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926990+0000","last_change":"2024-04-17T07:15:11.780594+0000","last_active":"2024-04-17T07:15:13.926990+0000","last_peered":"2024-04-17T07:15:13.926990+0000","last_clean":"2024-04-17T07:15:13.926990+0000","last_became_active":"2024-04-17T07:15:11.780233+0000","last_became_peered":"2024-04-17T07:15:11.780233+0000","last_unstale":"2024-04-17T07:15:13.926990+0000","last_undegraded":"2024-04-17T07:15:13.926990+0000","last_fullsized":"2024-04-17T07:15:13.926990+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:34.187674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927194+0000","last_change":"2024-04-17T07:15:11.783398+0000","last_active":"2024-04-17T07:15:13.927194+0000","last_peered":"2024-04-17T07:15:13.927194+0000","last_clean":"2024-04-17T07:15:13.927194+0000","last_became_active":"2024-04-17T07:15:11.783101+0000","last_became_peered":"2024-04-17T07:15:11.783101+0000","last_unstale":"2024-04-17T07:15:13.927194+0000","last_undegraded":"2024-04-17T07:15:13.927194+0000","last_fullsized":"2024-04-17T07:15:13.927194+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:16.977647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.793928+0000","last_change":"2024-04-17T07:15:11.794368+0000","last_active":"2024-04-17T07:15:12.793928+0000","last_peered":"2024-04-17T07:15:12.793928+0000","last_clean":"2024-04-17T07:15:12.793928+0000","last_became_active":"2024-04-17T07:15:11.793994+0000","last_became_peered":"2024-04-17T07:15:11.793994+0000","last_unstale":"2024-04-17T07:15:12.793928+0000","last_undegraded":"2024-04-17T07:15:12.793928+0000","last_fullsized":"2024-04-17T07:15:12.793928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:15.896666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787219+0000","last_change":"2024-04-17T07:15:11.783232+0000","last_active":"2024-04-17T07:15:12.787219+0000","last_peered":"2024-04-17T07:15:12.787219+0000","last_clean":"2024-04-17T07:15:12.787219+0000","last_became_active":"2024-04-17T07:15:11.782887+0000","last_became_peered":"2024-04-17T07:15:11.782887+0000","last_unstale":"2024-04-17T07:15:12.787219+0000","last_undegraded":"2024-04-17T07:15:12.787219+0000","last_fullsized":"2024-04-17T07:15:12.787219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:29.771615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799317+0000","last_change":"2024-04-17T07:15:11.791102+0000","last_active":"2024-04-17T07:15:12.799317+0000","last_peered":"2024-04-17T07:15:12.799317+0000","last_clean":"2024-04-17T07:15:12.799317+0000","last_became_active":"2024-04-17T07:15:11.789629+0000","last_became_peered":"2024-04-17T07:15:11.789629+0000","last_unstale":"2024-04-17T07:15:12.799317+0000","last_undegraded":"2024-04-17T07:15:12.799317+0000","last_fullsized":"2024-04-17T07:15:12.799317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781899+0000","last_change":"2024-04-17T07:15:11.798951+0000","last_active":"2024-04-17T07:15:12.781899+0000","last_peered":"2024-04-17T07:15:12.781899+0000","last_clean":"2024-04-17T07:15:12.781899+0000","last_became_active":"2024-04-17T07:15:11.796782+0000","last_became_peered":"2024-04-17T07:15:11.796782+0000","last_unstale":"2024-04-17T07:15:12.781899+0000","last_undegraded":"2024-04-17T07:15:12.781899+0000","last_fullsized":"2024-04-17T07:15:12.781899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:35:31.757126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799208+0000","last_change":"2024-04-17T07:15:11.791795+0000","last_active":"2024-04-17T07:15:12.799208+0000","last_peered":"2024-04-17T07:15:12.799208+0000","last_clean":"2024-04-17T07:15:12.799208+0000","last_became_active":"2024-04-17T07:15:11.791304+0000","last_became_peered":"2024-04-17T07:15:11.791304+0000","last_unstale":"2024-04-17T07:15:12.799208+0000","last_undegraded":"2024-04-17T07:15:12.799208+0000","last_fullsized":"2024-04-17T07:15:12.799208+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787262+0000","last_change":"2024-04-17T07:15:11.773030+0000","last_active":"2024-04-17T07:15:12.787262+0000","last_peered":"2024-04-17T07:15:12.787262+0000","last_clean":"2024-04-17T07:15:12.787262+0000","last_became_active":"2024-04-17T07:15:11.772331+0000","last_became_peered":"2024-04-17T07:15:11.772331+0000","last_unstale":"2024-04-17T07:15:12.787262+0000","last_undegraded":"2024-04-17T07:15:12.787262+0000","last_fullsized":"2024-04-17T07:15:12.787262+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927027+0000","last_change":"2024-04-17T07:15:11.781460+0000","last_active":"2024-04-17T07:15:13.927027+0000","last_peered":"2024-04-17T07:15:13.927027+0000","last_clean":"2024-04-17T07:15:13.927027+0000","last_became_active":"2024-04-17T07:15:11.781103+0000","last_became_peered":"2024-04-17T07:15:11.781103+0000","last_unstale":"2024-04-17T07:15:13.927027+0000","last_undegraded":"2024-04-17T07:15:13.927027+0000","last_fullsized":"2024-04-17T07:15:13.927027+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:11.476872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787980+0000","last_change":"2024-04-17T07:15:11.783097+0000","last_active":"2024-04-17T07:15:12.787980+0000","last_peered":"2024-04-17T07:15:12.787980+0000","last_clean":"2024-04-17T07:15:12.787980+0000","last_became_active":"2024-04-17T07:15:11.782801+0000","last_became_peered":"2024-04-17T07:15:11.782801+0000","last_unstale":"2024-04-17T07:15:12.787980+0000","last_undegraded":"2024-04-17T07:15:12.787980+0000","last_fullsized":"2024-04-17T07:15:12.787980+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865210+0000","last_change":"2024-04-17T07:15:11.783092+0000","last_active":"2024-04-17T07:15:12.865210+0000","last_peered":"2024-04-17T07:15:12.865210+0000","last_clean":"2024-04-17T07:15:12.865210+0000","last_became_active":"2024-04-17T07:15:11.782733+0000","last_became_peered":"2024-04-17T07:15:11.782733+0000","last_unstale":"2024-04-17T07:15:12.865210+0000","last_undegraded":"2024-04-17T07:15:12.865210+0000","last_fullsized":"2024-04-17T07:15:12.865210+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:17.023666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787669+0000","last_change":"2024-04-17T07:15:11.789442+0000","last_active":"2024-04-17T07:15:12.787669+0000","last_peered":"2024-04-17T07:15:12.787669+0000","last_clean":"2024-04-17T07:15:12.787669+0000","last_became_active":"2024-04-17T07:15:11.788735+0000","last_became_peered":"2024-04-17T07:15:11.788735+0000","last_unstale":"2024-04-17T07:15:12.787669+0000","last_undegraded":"2024-04-17T07:15:12.787669+0000","last_fullsized":"2024-04-17T07:15:12.787669+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794612+0000","last_change":"2024-04-17T07:15:11.792035+0000","last_active":"2024-04-17T07:15:12.794612+0000","last_peered":"2024-04-17T07:15:12.794612+0000","last_clean":"2024-04-17T07:15:12.794612+0000","last_became_active":"2024-04-17T07:15:11.791726+0000","last_became_peered":"2024-04-17T07:15:11.791726+0000","last_unstale":"2024-04-17T07:15:12.794612+0000","last_undegraded":"2024-04-17T07:15:12.794612+0000","last_fullsized":"2024-04-17T07:15:12.794612+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:01.580287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865739+0000","last_change":"2024-04-17T07:15:10.760324+0000","last_active":"2024-04-17T07:15:12.865739+0000","last_peered":"2024-04-17T07:15:12.865739+0000","last_clean":"2024-04-17T07:15:12.865739+0000","last_became_active":"2024-04-17T07:15:10.759991+0000","last_became_peered":"2024-04-17T07:15:10.759991+0000","last_unstale":"2024-04-17T07:15:12.865739+0000","last_undegraded":"2024-04-17T07:15:12.865739+0000","last_fullsized":"2024-04-17T07:15:12.865739+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:24.506173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865657+0000","last_change":"2024-04-17T07:15:11.787483+0000","last_active":"2024-04-17T07:15:12.865657+0000","last_peered":"2024-04-17T07:15:12.865657+0000","last_clean":"2024-04-17T07:15:12.865657+0000","last_became_active":"2024-04-17T07:15:11.787086+0000","last_became_peered":"2024-04-17T07:15:11.787086+0000","last_unstale":"2024-04-17T07:15:12.865657+0000","last_undegraded":"2024-04-17T07:15:12.865657+0000","last_fullsized":"2024-04-17T07:15:12.865657+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:11.476872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794555+0000","last_change":"2024-04-17T07:15:10.753951+0000","last_active":"2024-04-17T07:15:12.794555+0000","last_peered":"2024-04-17T07:15:12.794555+0000","last_clean":"2024-04-17T07:15:12.794555+0000","last_became_active":"2024-04-17T07:15:10.753762+0000","last_became_peered":"2024-04-17T07:15:10.753762+0000","last_unstale":"2024-04-17T07:15:12.794555+0000","last_undegraded":"2024-04-17T07:15:12.794555+0000","last_fullsized":"2024-04-17T07:15:12.794555+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:25.821388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927920+0000","last_change":"2024-04-17T07:15:11.785316+0000","last_active":"2024-04-17T07:15:13.927920+0000","last_peered":"2024-04-17T07:15:13.927920+0000","last_clean":"2024-04-17T07:15:13.927920+0000","last_became_active":"2024-04-17T07:15:11.782457+0000","last_became_peered":"2024-04-17T07:15:11.782457+0000","last_unstale":"2024-04-17T07:15:13.927920+0000","last_undegraded":"2024-04-17T07:15:13.927920+0000","last_fullsized":"2024-04-17T07:15:13.927920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:17.674336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781218+0000","last_change":"2024-04-17T07:15:10.753297+0000","last_active":"2024-04-17T07:15:12.781218+0000","last_peered":"2024-04-17T07:15:12.781218+0000","last_clean":"2024-04-17T07:15:12.781218+0000","last_became_active":"2024-04-17T07:15:10.753007+0000","last_became_peered":"2024-04-17T07:15:10.753007+0000","last_unstale":"2024-04-17T07:15:12.781218+0000","last_undegraded":"2024-04-17T07:15:12.781218+0000","last_fullsized":"2024-04-17T07:15:12.781218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:33.181954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799412+0000","last_change":"2024-04-17T07:15:11.791434+0000","last_active":"2024-04-17T07:15:12.799412+0000","last_peered":"2024-04-17T07:15:12.799412+0000","last_clean":"2024-04-17T07:15:12.799412+0000","last_became_active":"2024-04-17T07:15:11.791105+0000","last_became_peered":"2024-04-17T07:15:11.791105+0000","last_unstale":"2024-04-17T07:15:12.799412+0000","last_undegraded":"2024-04-17T07:15:12.799412+0000","last_fullsized":"2024-04-17T07:15:12.799412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825937+0000","last_change":"2024-04-17T07:15:10.759387+0000","last_active":"2024-04-17T07:15:12.825937+0000","last_peered":"2024-04-17T07:15:12.825937+0000","last_clean":"2024-04-17T07:15:12.825937+0000","last_became_active":"2024-04-17T07:15:10.758609+0000","last_became_peered":"2024-04-17T07:15:10.758609+0000","last_unstale":"2024-04-17T07:15:12.825937+0000","last_undegraded":"2024-04-17T07:15:12.825937+0000","last_fullsized":"2024-04-17T07:15:12.825937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:22.823436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788454+0000","last_change":"2024-04-17T07:15:11.786963+0000","last_active":"2024-04-17T07:15:12.788454+0000","last_peered":"2024-04-17T07:15:12.788454+0000","last_clean":"2024-04-17T07:15:12.788454+0000","last_became_active":"2024-04-17T07:15:11.786663+0000","last_became_peered":"2024-04-17T07:15:11.786663+0000","last_unstale":"2024-04-17T07:15:12.788454+0000","last_undegraded":"2024-04-17T07:15:12.788454+0000","last_fullsized":"2024-04-17T07:15:12.788454+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:51.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927884+0000","last_change":"2024-04-17T07:15:10.757342+0000","last_active":"2024-04-17T07:15:13.927884+0000","last_peered":"2024-04-17T07:15:13.927884+0000","last_clean":"2024-04-17T07:15:13.927884+0000","last_became_active":"2024-04-17T07:15:10.757059+0000","last_became_peered":"2024-04-17T07:15:10.757059+0000","last_unstale":"2024-04-17T07:15:13.927884+0000","last_undegraded":"2024-04-17T07:15:13.927884+0000","last_fullsized":"2024-04-17T07:15:13.927884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:24.506173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788121+0000","last_change":"2024-04-17T07:15:11.772728+0000","last_active":"2024-04-17T07:15:12.788121+0000","last_peered":"2024-04-17T07:15:12.788121+0000","last_clean":"2024-04-17T07:15:12.788121+0000","last_became_active":"2024-04-17T07:15:11.771937+0000","last_became_peered":"2024-04-17T07:15:11.771937+0000","last_unstale":"2024-04-17T07:15:12.788121+0000","last_undegraded":"2024-04-17T07:15:12.788121+0000","last_fullsized":"2024-04-17T07:15:12.788121+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926892+0000","last_change":"2024-04-17T07:15:10.759698+0000","last_active":"2024-04-17T07:15:13.926892+0000","last_peered":"2024-04-17T07:15:13.926892+0000","last_clean":"2024-04-17T07:15:13.926892+0000","last_became_active":"2024-04-17T07:15:10.758698+0000","last_became_peered":"2024-04-17T07:15:10.758698+0000","last_unstale":"2024-04-17T07:15:13.926892+0000","last_undegraded":"2024-04-17T07:15:13.926892+0000","last_fullsized":"2024-04-17T07:15:13.926892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:05.510781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927260+0000","last_change":"2024-04-17T07:15:11.786810+0000","last_active":"2024-04-17T07:15:13.927260+0000","last_peered":"2024-04-17T07:15:13.927260+0000","last_clean":"2024-04-17T07:15:13.927260+0000","last_became_active":"2024-04-17T07:15:11.786357+0000","last_became_peered":"2024-04-17T07:15:11.786357+0000","last_unstale":"2024-04-17T07:15:13.927260+0000","last_undegraded":"2024-04-17T07:15:13.927260+0000","last_fullsized":"2024-04-17T07:15:13.927260+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:11.999039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780769+0000","last_change":"2024-04-17T07:15:10.744944+0000","last_active":"2024-04-17T07:15:12.780769+0000","last_peered":"2024-04-17T07:15:12.780769+0000","last_clean":"2024-04-17T07:15:12.780769+0000","last_became_active":"2024-04-17T07:15:10.744632+0000","last_became_peered":"2024-04-17T07:15:10.744632+0000","last_unstale":"2024-04-17T07:15:12.780769+0000","last_undegraded":"2024-04-17T07:15:12.780769+0000","last_fullsized":"2024-04-17T07:15:12.780769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798369+0000","last_change":"2024-04-17T07:15:11.784231+0000","last_active":"2024-04-17T07:15:12.798369+0000","last_peered":"2024-04-17T07:15:12.798369+0000","last_clean":"2024-04-17T07:15:12.798369+0000","last_became_active":"2024-04-17T07:15:11.784070+0000","last_became_peered":"2024-04-17T07:15:11.784070+0000","last_unstale":"2024-04-17T07:15:12.798369+0000","last_undegraded":"2024-04-17T07:15:12.798369+0000","last_fullsized":"2024-04-17T07:15:12.798369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:36.110314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926716+0000","last_change":"2024-04-17T07:15:10.753739+0000","last_active":"2024-04-17T07:15:13.926716+0000","last_peered":"2024-04-17T07:15:13.926716+0000","last_clean":"2024-04-17T07:15:13.926716+0000","last_became_active":"2024-04-17T07:15:10.753411+0000","last_became_peered":"2024-04-17T07:15:10.753411+0000","last_unstale":"2024-04-17T07:15:13.926716+0000","last_undegraded":"2024-04-17T07:15:13.926716+0000","last_fullsized":"2024-04-17T07:15:13.926716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:22.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788318+0000","last_change":"2024-04-17T07:15:11.779401+0000","last_active":"2024-04-17T07:15:12.788318+0000","last_peered":"2024-04-17T07:15:12.788318+0000","last_clean":"2024-04-17T07:15:12.788318+0000","last_became_active":"2024-04-17T07:15:11.779116+0000","last_became_peered":"2024-04-17T07:15:11.779116+0000","last_unstale":"2024-04-17T07:15:12.788318+0000","last_undegraded":"2024-04-17T07:15:12.788318+0000","last_fullsized":"2024-04-17T07:15:12.788318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:51.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781584+0000","last_change":"2024-04-17T07:15:10.756984+0000","last_active":"2024-04-17T07:15:12.781584+0000","last_peered":"2024-04-17T07:15:12.781584+0000","last_clean":"2024-04-17T07:15:12.781584+0000","last_became_active":"2024-04-17T07:15:10.755966+0000","last_became_peered":"2024-04-17T07:15:10.755966+0000","last_unstale":"2024-04-17T07:15:12.781584+0000","last_undegraded":"2024-04-17T07:15:12.781584+0000","last_fullsized":"2024-04-17T07:15:12.781584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:47.288197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781661+0000","last_change":"2024-04-17T07:15:11.783941+0000","last_active":"2024-04-17T07:15:12.781661+0000","last_peered":"2024-04-17T07:15:12.781661+0000","last_clean":"2024-04-17T07:15:12.781661+0000","last_became_active":"2024-04-17T07:15:11.783578+0000","last_became_peered":"2024-04-17T07:15:11.783578+0000","last_unstale":"2024-04-17T07:15:12.781661+0000","last_undegraded":"2024-04-17T07:15:12.781661+0000","last_fullsized":"2024-04-17T07:15:12.781661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:23.035859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794614+0000","last_change":"2024-04-17T07:15:10.750003+0000","last_active":"2024-04-17T07:15:12.794614+0000","last_peered":"2024-04-17T07:15:12.794614+0000","last_clean":"2024-04-17T07:15:12.794614+0000","last_became_active":"2024-04-17T07:15:10.749706+0000","last_became_peered":"2024-04-17T07:15:10.749706+0000","last_unstale":"2024-04-17T07:15:12.794614+0000","last_undegraded":"2024-04-17T07:15:12.794614+0000","last_fullsized":"2024-04-17T07:15:12.794614+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:49.742557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781197+0000","last_change":"2024-04-17T07:15:11.790257+0000","last_active":"2024-04-17T07:15:12.781197+0000","last_peered":"2024-04-17T07:15:12.781197+0000","last_clean":"2024-04-17T07:15:12.781197+0000","last_became_active":"2024-04-17T07:15:11.789887+0000","last_became_peered":"2024-04-17T07:15:11.789887+0000","last_unstale":"2024-04-17T07:15:12.781197+0000","last_undegraded":"2024-04-17T07:15:12.781197+0000","last_fullsized":"2024-04-17T07:15:12.781197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:50.447380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787707+0000","last_change":"2024-04-17T07:15:10.754401+0000","last_active":"2024-04-17T07:15:12.787707+0000","last_peered":"2024-04-17T07:15:12.787707+0000","last_clean":"2024-04-17T07:15:12.787707+0000","last_became_active":"2024-04-17T07:15:10.754054+0000","last_became_peered":"2024-04-17T07:15:10.754054+0000","last_unstale":"2024-04-17T07:15:12.787707+0000","last_undegraded":"2024-04-17T07:15:12.787707+0000","last_fullsized":"2024-04-17T07:15:12.787707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:58.814832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787560+0000","last_change":"2024-04-17T07:15:11.781362+0000","last_active":"2024-04-17T07:15:12.787560+0000","last_peered":"2024-04-17T07:15:12.787560+0000","last_clean":"2024-04-17T07:15:12.787560+0000","last_became_active":"2024-04-17T07:15:11.780192+0000","last_became_peered":"2024-04-17T07:15:11.780192+0000","last_unstale":"2024-04-17T07:15:12.787560+0000","last_undegraded":"2024-04-17T07:15:12.787560+0000","last_fullsized":"2024-04-17T07:15:12.787560+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781217+0000","last_change":"2024-04-17T07:15:10.753307+0000","last_active":"2024-04-17T07:15:12.781217+0000","last_peered":"2024-04-17T07:15:12.781217+0000","last_clean":"2024-04-17T07:15:12.781217+0000","last_became_active":"2024-04-17T07:15:10.753049+0000","last_became_peered":"2024-04-17T07:15:10.753049+0000","last_unstale":"2024-04-17T07:15:12.781217+0000","last_undegraded":"2024-04-17T07:15:12.781217+0000","last_fullsized":"2024-04-17T07:15:12.781217+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:10.993320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798324+0000","last_change":"2024-04-17T07:15:11.784972+0000","last_active":"2024-04-17T07:15:12.798324+0000","last_peered":"2024-04-17T07:15:12.798324+0000","last_clean":"2024-04-17T07:15:12.798324+0000","last_became_active":"2024-04-17T07:15:11.784776+0000","last_became_peered":"2024-04-17T07:15:11.784776+0000","last_unstale":"2024-04-17T07:15:12.798324+0000","last_undegraded":"2024-04-17T07:15:12.798324+0000","last_fullsized":"2024-04-17T07:15:12.798324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:17.756837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927265+0000","last_change":"2024-04-17T07:15:10.752949+0000","last_active":"2024-04-17T07:15:13.927265+0000","last_peered":"2024-04-17T07:15:13.927265+0000","last_clean":"2024-04-17T07:15:13.927265+0000","last_became_active":"2024-04-17T07:15:10.752625+0000","last_became_peered":"2024-04-17T07:15:10.752625+0000","last_unstale":"2024-04-17T07:15:13.927265+0000","last_undegraded":"2024-04-17T07:15:13.927265+0000","last_fullsized":"2024-04-17T07:15:13.927265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:59.786245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781202+0000","last_change":"2024-04-17T07:15:11.787599+0000","last_active":"2024-04-17T07:15:12.781202+0000","last_peered":"2024-04-17T07:15:12.781202+0000","last_clean":"2024-04-17T07:15:12.781202+0000","last_became_active":"2024-04-17T07:15:11.787356+0000","last_became_peered":"2024-04-17T07:15:11.787356+0000","last_unstale":"2024-04-17T07:15:12.781202+0000","last_undegraded":"2024-04-17T07:15:12.781202+0000","last_fullsized":"2024-04-17T07:15:12.781202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:40.742949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798478+0000","last_change":"2024-04-17T07:15:10.754700+0000","last_active":"2024-04-17T07:15:12.798478+0000","last_peered":"2024-04-17T07:15:12.798478+0000","last_clean":"2024-04-17T07:15:12.798478+0000","last_became_active":"2024-04-17T07:15:10.754438+0000","last_became_peered":"2024-04-17T07:15:10.754438+0000","last_unstale":"2024-04-17T07:15:12.798478+0000","last_undegraded":"2024-04-17T07:15:12.798478+0000","last_fullsized":"2024-04-17T07:15:12.798478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:18.428942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788103+0000","last_change":"2024-04-17T07:15:11.790073+0000","last_active":"2024-04-17T07:15:12.788103+0000","last_peered":"2024-04-17T07:15:12.788103+0000","last_clean":"2024-04-17T07:15:12.788103+0000","last_became_active":"2024-04-17T07:15:11.789138+0000","last_became_peered":"2024-04-17T07:15:11.789138+0000","last_unstale":"2024-04-17T07:15:12.788103+0000","last_undegraded":"2024-04-17T07:15:12.788103+0000","last_fullsized":"2024-04-17T07:15:12.788103+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780841+0000","last_change":"2024-04-17T07:15:10.753648+0000","last_active":"2024-04-17T07:15:12.780841+0000","last_peered":"2024-04-17T07:15:12.780841+0000","last_clean":"2024-04-17T07:15:12.780841+0000","last_became_active":"2024-04-17T07:15:10.752763+0000","last_became_peered":"2024-04-17T07:15:10.752763+0000","last_unstale":"2024-04-17T07:15:12.780841+0000","last_undegraded":"2024-04-17T07:15:12.780841+0000","last_fullsized":"2024-04-17T07:15:12.780841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:58.814832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865402+0000","last_change":"2024-04-17T07:15:11.792573+0000","last_active":"2024-04-17T07:15:12.865402+0000","last_peered":"2024-04-17T07:15:12.865402+0000","last_clean":"2024-04-17T07:15:12.865402+0000","last_became_active":"2024-04-17T07:15:11.787417+0000","last_became_peered":"2024-04-17T07:15:11.787417+0000","last_unstale":"2024-04-17T07:15:12.865402+0000","last_undegraded":"2024-04-17T07:15:12.865402+0000","last_fullsized":"2024-04-17T07:15:12.865402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:59.820552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780352+0000","last_change":"2024-04-17T07:15:10.750662+0000","last_active":"2024-04-17T07:15:12.780352+0000","last_peered":"2024-04-17T07:15:12.780352+0000","last_clean":"2024-04-17T07:15:12.780352+0000","last_became_active":"2024-04-17T07:15:10.750363+0000","last_became_peered":"2024-04-17T07:15:10.750363+0000","last_unstale":"2024-04-17T07:15:12.780352+0000","last_undegraded":"2024-04-17T07:15:12.780352+0000","last_fullsized":"2024-04-17T07:15:12.780352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:10.471152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927533+0000","last_change":"2024-04-17T07:15:11.781408+0000","last_active":"2024-04-17T07:15:13.927533+0000","last_peered":"2024-04-17T07:15:13.927533+0000","last_clean":"2024-04-17T07:15:13.927533+0000","last_became_active":"2024-04-17T07:15:11.780292+0000","last_became_peered":"2024-04-17T07:15:11.780292+0000","last_unstale":"2024-04-17T07:15:13.927533+0000","last_undegraded":"2024-04-17T07:15:13.927533+0000","last_fullsized":"2024-04-17T07:15:13.927533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:48.293916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799370+0000","last_change":"2024-04-17T07:15:10.753995+0000","last_active":"2024-04-17T07:15:12.799370+0000","last_peered":"2024-04-17T07:15:12.799370+0000","last_clean":"2024-04-17T07:15:12.799370+0000","last_became_active":"2024-04-17T07:15:10.749692+0000","last_became_peered":"2024-04-17T07:15:10.749692+0000","last_unstale":"2024-04-17T07:15:12.799370+0000","last_undegraded":"2024-04-17T07:15:12.799370+0000","last_fullsized":"2024-04-17T07:15:12.799370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:16.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788628+0000","last_change":"2024-04-17T07:15:11.786775+0000","last_active":"2024-04-17T07:15:12.788628+0000","last_peered":"2024-04-17T07:15:12.788628+0000","last_clean":"2024-04-17T07:15:12.788628+0000","last_became_active":"2024-04-17T07:15:11.786284+0000","last_became_peered":"2024-04-17T07:15:11.786284+0000","last_unstale":"2024-04-17T07:15:12.788628+0000","last_undegraded":"2024-04-17T07:15:12.788628+0000","last_fullsized":"2024-04-17T07:15:12.788628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927329+0000","last_change":"2024-04-17T07:15:10.758511+0000","last_active":"2024-04-17T07:15:13.927329+0000","last_peered":"2024-04-17T07:15:13.927329+0000","last_clean":"2024-04-17T07:15:13.927329+0000","last_became_active":"2024-04-17T07:15:10.757894+0000","last_became_peered":"2024-04-17T07:15:10.757894+0000","last_unstale":"2024-04-17T07:15:13.927329+0000","last_undegraded":"2024-04-17T07:15:13.927329+0000","last_fullsized":"2024-04-17T07:15:13.927329+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:47.131852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798899+0000","last_change":"2024-04-17T07:15:11.781275+0000","last_active":"2024-04-17T07:15:12.798899+0000","last_peered":"2024-04-17T07:15:12.798899+0000","last_clean":"2024-04-17T07:15:12.798899+0000","last_became_active":"2024-04-17T07:15:11.780976+0000","last_became_peered":"2024-04-17T07:15:11.780976+0000","last_unstale":"2024-04-17T07:15:12.798899+0000","last_undegraded":"2024-04-17T07:15:12.798899+0000","last_fullsized":"2024-04-17T07:15:12.798899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:29.771615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927598+0000","last_change":"2024-04-17T07:15:10.756433+0000","last_active":"2024-04-17T07:15:13.927598+0000","last_peered":"2024-04-17T07:15:13.927598+0000","last_clean":"2024-04-17T07:15:13.927598+0000","last_became_active":"2024-04-17T07:15:10.756141+0000","last_became_peered":"2024-04-17T07:15:10.756141+0000","last_unstale":"2024-04-17T07:15:13.927598+0000","last_undegraded":"2024-04-17T07:15:13.927598+0000","last_fullsized":"2024-04-17T07:15:13.927598+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:43.744189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825883+0000","last_change":"2024-04-17T07:15:11.789536+0000","last_active":"2024-04-17T07:15:12.825883+0000","last_peered":"2024-04-17T07:15:12.825883+0000","last_clean":"2024-04-17T07:15:12.825883+0000","last_became_active":"2024-04-17T07:15:11.789254+0000","last_became_peered":"2024-04-17T07:15:11.789254+0000","last_unstale":"2024-04-17T07:15:12.825883+0000","last_undegraded":"2024-04-17T07:15:12.825883+0000","last_fullsized":"2024-04-17T07:15:12.825883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:50.687319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927850+0000","last_change":"2024-04-17T07:15:10.758159+0000","last_active":"2024-04-17T07:15:13.927850+0000","last_peered":"2024-04-17T07:15:13.927850+0000","last_clean":"2024-04-17T07:15:13.927850+0000","last_became_active":"2024-04-17T07:15:10.757799+0000","last_became_peered":"2024-04-17T07:15:10.757799+0000","last_unstale":"2024-04-17T07:15:13.927850+0000","last_undegraded":"2024-04-17T07:15:13.927850+0000","last_fullsized":"2024-04-17T07:15:13.927850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:02.814854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825758+0000","last_change":"2024-04-17T07:15:11.793490+0000","last_active":"2024-04-17T07:15:12.825758+0000","last_peered":"2024-04-17T07:15:12.825758+0000","last_clean":"2024-04-17T07:15:12.825758+0000","last_became_active":"2024-04-17T07:15:11.792875+0000","last_became_peered":"2024-04-17T07:15:11.792875+0000","last_unstale":"2024-04-17T07:15:12.825758+0000","last_undegraded":"2024-04-17T07:15:12.825758+0000","last_fullsized":"2024-04-17T07:15:12.825758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:41.684920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794219+0000","last_change":"2024-04-17T07:15:10.756878+0000","last_active":"2024-04-17T07:15:12.794219+0000","last_peered":"2024-04-17T07:15:12.794219+0000","last_clean":"2024-04-17T07:15:12.794219+0000","last_became_active":"2024-04-17T07:15:10.755581+0000","last_became_peered":"2024-04-17T07:15:10.755581+0000","last_unstale":"2024-04-17T07:15:12.794219+0000","last_undegraded":"2024-04-17T07:15:12.794219+0000","last_fullsized":"2024-04-17T07:15:12.794219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:49.681600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826140+0000","last_change":"2024-04-17T07:15:11.777553+0000","last_active":"2024-04-17T07:15:12.826140+0000","last_peered":"2024-04-17T07:15:12.826140+0000","last_clean":"2024-04-17T07:15:12.826140+0000","last_became_active":"2024-04-17T07:15:11.774880+0000","last_became_peered":"2024-04-17T07:15:11.774880+0000","last_unstale":"2024-04-17T07:15:12.826140+0000","last_undegraded":"2024-04-17T07:15:12.826140+0000","last_fullsized":"2024-04-17T07:15:12.826140+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:00.042145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865131+0000","last_change":"2024-04-17T07:15:10.756968+0000","last_active":"2024-04-17T07:15:12.865131+0000","last_peered":"2024-04-17T07:15:12.865131+0000","last_clean":"2024-04-17T07:15:12.865131+0000","last_became_active":"2024-04-17T07:15:10.756661+0000","last_became_peered":"2024-04-17T07:15:10.756661+0000","last_unstale":"2024-04-17T07:15:12.865131+0000","last_undegraded":"2024-04-17T07:15:12.865131+0000","last_fullsized":"2024-04-17T07:15:12.865131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:22.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825752+0000","last_change":"2024-04-17T07:15:11.776185+0000","last_active":"2024-04-17T07:15:12.825752+0000","last_peered":"2024-04-17T07:15:12.825752+0000","last_clean":"2024-04-17T07:15:12.825752+0000","last_became_active":"2024-04-17T07:15:11.775873+0000","last_became_peered":"2024-04-17T07:15:11.775873+0000","last_unstale":"2024-04-17T07:15:12.825752+0000","last_undegraded":"2024-04-17T07:15:12.825752+0000","last_fullsized":"2024-04-17T07:15:12.825752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:06.843379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926785+0000","last_change":"2024-04-17T07:15:10.754318+0000","last_active":"2024-04-17T07:15:13.926785+0000","last_peered":"2024-04-17T07:15:13.926785+0000","last_clean":"2024-04-17T07:15:13.926785+0000","last_became_active":"2024-04-17T07:15:10.752600+0000","last_became_peered":"2024-04-17T07:15:10.752600+0000","last_unstale":"2024-04-17T07:15:13.926785+0000","last_undegraded":"2024-04-17T07:15:13.926785+0000","last_fullsized":"2024-04-17T07:15:13.926785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:02.716404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789036+0000","last_change":"2024-04-17T07:15:11.789823+0000","last_active":"2024-04-17T07:15:12.789036+0000","last_peered":"2024-04-17T07:15:12.789036+0000","last_clean":"2024-04-17T07:15:12.789036+0000","last_became_active":"2024-04-17T07:15:11.789026+0000","last_became_peered":"2024-04-17T07:15:11.789026+0000","last_unstale":"2024-04-17T07:15:12.789036+0000","last_undegraded":"2024-04-17T07:15:12.789036+0000","last_fullsized":"2024-04-17T07:15:12.789036+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.786269+0000","last_change":"2024-04-17T07:15:10.754812+0000","last_active":"2024-04-17T07:15:12.786269+0000","last_peered":"2024-04-17T07:15:12.786269+0000","last_clean":"2024-04-17T07:15:12.786269+0000","last_became_active":"2024-04-17T07:15:10.753501+0000","last_became_peered":"2024-04-17T07:15:10.753501+0000","last_unstale":"2024-04-17T07:15:12.786269+0000","last_undegraded":"2024-04-17T07:15:12.786269+0000","last_fullsized":"2024-04-17T07:15:12.786269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826101+0000","last_change":"2024-04-17T07:15:11.782597+0000","last_active":"2024-04-17T07:15:12.826101+0000","last_peered":"2024-04-17T07:15:12.826101+0000","last_clean":"2024-04-17T07:15:12.826101+0000","last_became_active":"2024-04-17T07:15:11.782301+0000","last_became_peered":"2024-04-17T07:15:11.782301+0000","last_unstale":"2024-04-17T07:15:12.826101+0000","last_undegraded":"2024-04-17T07:15:12.826101+0000","last_fullsized":"2024-04-17T07:15:12.826101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:50.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799478+0000","last_change":"2024-04-17T07:15:10.759311+0000","last_active":"2024-04-17T07:15:12.799478+0000","last_peered":"2024-04-17T07:15:12.799478+0000","last_clean":"2024-04-17T07:15:12.799478+0000","last_became_active":"2024-04-17T07:15:10.759020+0000","last_became_peered":"2024-04-17T07:15:10.759020+0000","last_unstale":"2024-04-17T07:15:12.799478+0000","last_undegraded":"2024-04-17T07:15:12.799478+0000","last_fullsized":"2024-04-17T07:15:12.799478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789044+0000","last_change":"2024-04-17T07:15:11.776506+0000","last_active":"2024-04-17T07:15:12.789044+0000","last_peered":"2024-04-17T07:15:12.789044+0000","last_clean":"2024-04-17T07:15:12.789044+0000","last_became_active":"2024-04-17T07:15:11.776233+0000","last_became_peered":"2024-04-17T07:15:11.776233+0000","last_unstale":"2024-04-17T07:15:12.789044+0000","last_undegraded":"2024-04-17T07:15:12.789044+0000","last_fullsized":"2024-04-17T07:15:12.789044+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794701+0000","last_change":"2024-04-17T07:15:10.752962+0000","last_active":"2024-04-17T07:15:12.794701+0000","last_peered":"2024-04-17T07:15:12.794701+0000","last_clean":"2024-04-17T07:15:12.794701+0000","last_became_active":"2024-04-17T07:15:10.752256+0000","last_became_peered":"2024-04-17T07:15:10.752256+0000","last_unstale":"2024-04-17T07:15:12.794701+0000","last_undegraded":"2024-04-17T07:15:12.794701+0000","last_fullsized":"2024-04-17T07:15:12.794701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:53:59.036426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794379+0000","last_change":"2024-04-17T07:15:11.775823+0000","last_active":"2024-04-17T07:15:12.794379+0000","last_peered":"2024-04-17T07:15:12.794379+0000","last_clean":"2024-04-17T07:15:12.794379+0000","last_became_active":"2024-04-17T07:15:11.775511+0000","last_became_peered":"2024-04-17T07:15:11.775511+0000","last_unstale":"2024-04-17T07:15:12.794379+0000","last_undegraded":"2024-04-17T07:15:12.794379+0000","last_fullsized":"2024-04-17T07:15:12.794379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:38.306939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798907+0000","last_change":"2024-04-17T07:15:10.754618+0000","last_active":"2024-04-17T07:15:12.798907+0000","last_peered":"2024-04-17T07:15:12.798907+0000","last_clean":"2024-04-17T07:15:12.798907+0000","last_became_active":"2024-04-17T07:15:10.754339+0000","last_became_peered":"2024-04-17T07:15:10.754339+0000","last_unstale":"2024-04-17T07:15:12.798907+0000","last_undegraded":"2024-04-17T07:15:12.798907+0000","last_fullsized":"2024-04-17T07:15:12.798907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:46.911722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865371+0000","last_change":"2024-04-17T07:15:11.780155+0000","last_active":"2024-04-17T07:15:12.865371+0000","last_peered":"2024-04-17T07:15:12.865371+0000","last_clean":"2024-04-17T07:15:12.865371+0000","last_became_active":"2024-04-17T07:15:11.779020+0000","last_became_peered":"2024-04-17T07:15:11.779020+0000","last_unstale":"2024-04-17T07:15:12.865371+0000","last_undegraded":"2024-04-17T07:15:12.865371+0000","last_fullsized":"2024-04-17T07:15:12.865371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:22.689645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794308+0000","last_change":"2024-04-17T07:15:10.760063+0000","last_active":"2024-04-17T07:15:12.794308+0000","last_peered":"2024-04-17T07:15:12.794308+0000","last_clean":"2024-04-17T07:15:12.794308+0000","last_became_active":"2024-04-17T07:15:10.759760+0000","last_became_peered":"2024-04-17T07:15:10.759760+0000","last_unstale":"2024-04-17T07:15:12.794308+0000","last_undegraded":"2024-04-17T07:15:12.794308+0000","last_fullsized":"2024-04-17T07:15:12.794308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:27.362229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794156+0000","last_change":"2024-04-17T07:15:11.781895+0000","last_active":"2024-04-17T07:15:12.794156+0000","last_peered":"2024-04-17T07:15:12.794156+0000","last_clean":"2024-04-17T07:15:12.794156+0000","last_became_active":"2024-04-17T07:15:11.781491+0000","last_became_peered":"2024-04-17T07:15:11.781491+0000","last_unstale":"2024-04-17T07:15:12.794156+0000","last_undegraded":"2024-04-17T07:15:12.794156+0000","last_fullsized":"2024-04-17T07:15:12.794156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:22.125784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798532+0000","last_change":"2024-04-17T07:15:10.760218+0000","last_active":"2024-04-17T07:15:12.798532+0000","last_peered":"2024-04-17T07:15:12.798532+0000","last_clean":"2024-04-17T07:15:12.798532+0000","last_became_active":"2024-04-17T07:15:10.759943+0000","last_became_peered":"2024-04-17T07:15:10.759943+0000","last_unstale":"2024-04-17T07:15:12.798532+0000","last_undegraded":"2024-04-17T07:15:12.798532+0000","last_fullsized":"2024-04-17T07:15:12.798532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:23:54.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780428+0000","last_change":"2024-04-17T07:15:11.786519+0000","last_active":"2024-04-17T07:15:12.780428+0000","last_peered":"2024-04-17T07:15:12.780428+0000","last_clean":"2024-04-17T07:15:12.780428+0000","last_became_active":"2024-04-17T07:15:11.783412+0000","last_became_peered":"2024-04-17T07:15:11.783412+0000","last_unstale":"2024-04-17T07:15:12.780428+0000","last_undegraded":"2024-04-17T07:15:12.780428+0000","last_fullsized":"2024-04-17T07:15:12.780428+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927118+0000","last_change":"2024-04-17T07:15:10.758123+0000","last_active":"2024-04-17T07:15:13.927118+0000","last_peered":"2024-04-17T07:15:13.927118+0000","last_clean":"2024-04-17T07:15:13.927118+0000","last_became_active":"2024-04-17T07:15:10.757812+0000","last_became_peered":"2024-04-17T07:15:10.757812+0000","last_unstale":"2024-04-17T07:15:13.927118+0000","last_undegraded":"2024-04-17T07:15:13.927118+0000","last_fullsized":"2024-04-17T07:15:13.927118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:12.245915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.928069+0000","last_change":"2024-04-17T07:15:11.782972+0000","last_active":"2024-04-17T07:15:13.928069+0000","last_peered":"2024-04-17T07:15:13.928069+0000","last_clean":"2024-04-17T07:15:13.928069+0000","last_became_active":"2024-04-17T07:15:11.781277+0000","last_became_peered":"2024-04-17T07:15:11.781277+0000","last_unstale":"2024-04-17T07:15:13.928069+0000","last_undegraded":"2024-04-17T07:15:13.928069+0000","last_fullsized":"2024-04-17T07:15:13.928069+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:17.756837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780702+0000","last_change":"2024-04-17T07:15:10.754805+0000","last_active":"2024-04-17T07:15:12.780702+0000","last_peered":"2024-04-17T07:15:12.780702+0000","last_clean":"2024-04-17T07:15:12.780702+0000","last_became_active":"2024-04-17T07:15:10.754522+0000","last_became_peered":"2024-04-17T07:15:10.754522+0000","last_unstale":"2024-04-17T07:15:12.780702+0000","last_undegraded":"2024-04-17T07:15:12.780702+0000","last_fullsized":"2024-04-17T07:15:12.780702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:15.971927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789293+0000","last_change":"2024-04-17T07:15:11.791410+0000","last_active":"2024-04-17T07:15:12.789293+0000","last_peered":"2024-04-17T07:15:12.789293+0000","last_clean":"2024-04-17T07:15:12.789293+0000","last_became_active":"2024-04-17T07:15:11.790779+0000","last_became_peered":"2024-04-17T07:15:11.790779+0000","last_unstale":"2024-04-17T07:15:12.789293+0000","last_undegraded":"2024-04-17T07:15:12.789293+0000","last_fullsized":"2024-04-17T07:15:12.789293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865155+0000","last_change":"2024-04-17T07:15:10.760465+0000","last_active":"2024-04-17T07:15:12.865155+0000","last_peered":"2024-04-17T07:15:12.865155+0000","last_clean":"2024-04-17T07:15:12.865155+0000","last_became_active":"2024-04-17T07:15:10.760075+0000","last_became_peered":"2024-04-17T07:15:10.760075+0000","last_unstale":"2024-04-17T07:15:12.865155+0000","last_undegraded":"2024-04-17T07:15:12.865155+0000","last_fullsized":"2024-04-17T07:15:12.865155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:47.131852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788985+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:15:12.788985+0000","last_peered":"2024-04-17T07:15:12.788985+0000","last_clean":"2024-04-17T07:15:12.788985+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:15:12.788985+0000","last_undegraded":"2024-04-17T07:15:12.788985+0000","last_fullsized":"2024-04-17T07:15:12.788985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":991232,"data_stored":744966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":154612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":397312,"data_stored":154578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":744876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:15:21.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:15:21.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.640+0000 7fc397200640 1 Processor -- start 2024-04-17T07:15:21.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.641+0000 7fc397200640 1 -- start start 2024-04-17T07:15:21.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.642+0000 7fc397200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc3980b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.642+0000 7fc397200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3980d2d30 con 0x7fc3980cfae0 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.642+0000 7fc397200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3980d3010 con 0x7fc3980b6b30 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.642+0000 7fc397200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3980d49c0 con 0x7fc39815ec60 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.643+0000 7fc396800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3980cfae0 0x7fc3980b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36184/0 (socket says 172.21.15.67:36184) 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.643+0000 7fc396800640 1 -- 172.21.15.67:0/2486201322 learned_addr learned my addr 172.21.15.67:0/2486201322 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:21.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.643+0000 7fc395e00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc3980b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.643+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144929099 0 0) 0x7fc3980d2d30 con 0x7fc3980cfae0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.643+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc378003520 con 0x7fc3980cfae0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3084202102 0 0) 0x7fc378003520 con 0x7fc3980cfae0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 msgr2=0x7fc3980b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc3980b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 >> v1:172.21.15.187:6789/0 conn(0x7fc3980b6b30 legacy=0x7fc3980cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3980d32f0 con 0x7fc3980cfae0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc390001670 con 0x7fc3980cfae0 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 >> v1:172.21.15.67:6789/0 conn(0x7fc3980cfae0 legacy=0x7fc3980b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc378003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378003f90 0x7fc378004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc378004d40 0x7fc3780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3980d49c0 con 0x7fc378003f90 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc37800baa0 con 0x7fc378004d40 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc37800bca0 con 0x7fc39815ec60 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc395400640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378003f90 0x7fc378004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.644+0000 7fc395e00640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc378003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.645+0000 7fc395400640 1 -- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 msgr2=0x7fc378003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.645+0000 7fc395400640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc39815ec60 0x7fc378003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.645+0000 7fc395400640 1 -- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc378004d40 msgr2=0x7fc3780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.645+0000 7fc395400640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc378004d40 0x7fc3780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.645+0000 7fc395400640 1 -- 172.21.15.67:0/2486201322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc378003520 con 0x7fc378003f90 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.646+0000 7fc395400640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378003f90 0x7fc378004800 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc380001f60 tx=0x7fc378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4391fb0d3bab84aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.646+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380001b30 con 0x7fc378003f90 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.646+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38002d710 con 0x7fc378003f90 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.646+0000 7fc394a00640 1 -- 172.21.15.67:0/2486201322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38002db40 con 0x7fc378003f90 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 -- 172.21.15.67:0/2486201322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378003f90 msgr2=0x7fc378004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 --2- 172.21.15.67:0/2486201322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378003f90 0x7fc378004800 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc380001f60 tx=0x7fc378005830 comp rx=0 tx=0).stop 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc396800640 1 -- 172.21.15.67:0/2486201322 reap_dead start 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 -- 172.21.15.67:0/2486201322 shutdown_connections 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 -- 172.21.15.67:0/2486201322 >> 172.21.15.67:0/2486201322 conn(0x7fc3980d15f0 msgr2=0x7fc3980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 -- 172.21.15.67:0/2486201322 shutdown_connections 2024-04-17T07:15:21.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.647+0000 7fc397200640 1 -- 172.21.15.67:0/2486201322 wait complete. 2024-04-17T07:15:21.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.649+0000 7fc397200640 1 Processor -- start 2024-04-17T07:15:21.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.649+0000 7fc397200640 1 -- start start 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc378003f90 0x7fc398157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 0x7fc398159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3980b6f70 con 0x7fc3980cb0b0 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3980d3660 con 0x7fc398159080 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc397200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3980d00c0 con 0x7fc378003f90 2024-04-17T07:15:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46634/0 (socket says 172.21.15.67:46634) 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 -- 172.21.15.67:0/1947091157 learned_addr learned my addr 172.21.15.67:0/1947091157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395e00640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc378003f90 0x7fc398157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 -- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc378003f90 msgr2=0x7fc398157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc396800640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 0x7fc398159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc378003f90 0x7fc398157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 -- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 msgr2=0x7fc398159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 0x7fc398159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc395400640 1 -- 172.21.15.67:0/1947091157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc398157d50 con 0x7fc3980cb0b0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.650+0000 7fc396800640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 0x7fc398159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc395400640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc380002080 tx=0x7fc3800015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380043070 con 0x7fc3980cb0b0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38002dd90 con 0x7fc3980cb0b0 2024-04-17T07:15:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380006d90 con 0x7fc3980cb0b0 2024-04-17T07:15:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3980cbe90 con 0x7fc3980cb0b0 2024-04-17T07:15:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.651+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3980cc450 con 0x7fc3980cb0b0 2024-04-17T07:15:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.652+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3980cbc20 con 0x7fc3980cb0b0 2024-04-17T07:15:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.653+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3800050c0 con 0x7fc3980cb0b0 2024-04-17T07:15:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.654+0000 7fc38e000640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 0x7fc37406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.654+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fc3800adfb0 con 0x7fc3980cb0b0 2024-04-17T07:15:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.655+0000 7fc395e00640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 0x7fc37406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:21.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.656+0000 7fc395e00640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 0x7fc37406ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc398160fa0 tx=0x7fc388008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:21.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.659+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc38004c030 con 0x7fc3980cb0b0 2024-04-17T07:15:21.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.862+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc398159a70 con 0x7fc37406c920 2024-04-17T07:15:21.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.881+0000 7fc38e000640 1 -- 172.21.15.67:0/1947091157 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251839 (secure 0 0 0) 0x7fc388004030 con 0x7fc37406c920 2024-04-17T07:15:21.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:21.886 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.886+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 msgr2=0x7fc37406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.886+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 0x7fc37406ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc398160fa0 tx=0x7fc388008040 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 msgr2=0x7fc3980cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc380002080 tx=0x7fc3800015b0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 shutdown_connections 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc398159080 0x7fc398159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3980cb0b0 0x7fc3980cb530 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fc37406c920 0x7fc37406ede0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 --2- 172.21.15.67:0/1947091157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc378003f90 0x7fc398157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:21.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 >> 172.21.15.67:0/1947091157 conn(0x7fc3980d15f0 msgr2=0x7fc39815b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:21.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 shutdown_connections 2024-04-17T07:15:21.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:21.887+0000 7fc397200640 1 -- 172.21.15.67:0/1947091157 wait complete. 2024-04-17T07:15:21.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-04-17T07:15:21.383405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224632,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750088,"statfs":{"total":773094113280,"available":772864090112,"internally_reserved":0,"allocated":4399104,"data_stored":2421620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.632026"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865661+0000","last_change":"2024-04-17T07:15:11.785362+0000","last_active":"2024-04-17T07:15:12.865661+0000","last_peered":"2024-04-17T07:15:12.865661+0000","last_clean":"2024-04-17T07:15:12.865661+0000","last_became_active":"2024-04-17T07:15:11.785026+0000","last_became_peered":"2024-04-17T07:15:11.785026+0000","last_unstale":"2024-04-17T07:15:12.865661+0000","last_undegraded":"2024-04-17T07:15:12.865661+0000","last_fullsized":"2024-04-17T07:15:12.865661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:34.187674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781692+0000","last_change":"2024-04-17T07:15:11.788735+0000","last_active":"2024-04-17T07:15:12.781692+0000","last_peered":"2024-04-17T07:15:12.781692+0000","last_clean":"2024-04-17T07:15:12.781692+0000","last_became_active":"2024-04-17T07:15:11.788305+0000","last_became_peered":"2024-04-17T07:15:11.788305+0000","last_unstale":"2024-04-17T07:15:12.781692+0000","last_undegraded":"2024-04-17T07:15:12.781692+0000","last_fullsized":"2024-04-17T07:15:12.781692+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:49.234619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787705+0000","last_change":"2024-04-17T07:15:11.781953+0000","last_active":"2024-04-17T07:15:12.787705+0000","last_peered":"2024-04-17T07:15:12.787705+0000","last_clean":"2024-04-17T07:15:12.787705+0000","last_became_active":"2024-04-17T07:15:11.781695+0000","last_became_peered":"2024-04-17T07:15:11.781695+0000","last_unstale":"2024-04-17T07:15:12.787705+0000","last_undegraded":"2024-04-17T07:15:12.787705+0000","last_fullsized":"2024-04-17T07:15:12.787705+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798770+0000","last_change":"2024-04-17T07:15:11.791526+0000","last_active":"2024-04-17T07:15:12.798770+0000","last_peered":"2024-04-17T07:15:12.798770+0000","last_clean":"2024-04-17T07:15:12.798770+0000","last_became_active":"2024-04-17T07:15:11.791197+0000","last_became_peered":"2024-04-17T07:15:11.791197+0000","last_unstale":"2024-04-17T07:15:12.798770+0000","last_undegraded":"2024-04-17T07:15:12.798770+0000","last_fullsized":"2024-04-17T07:15:12.798770+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:38.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826009+0000","last_change":"2024-04-17T07:15:11.788847+0000","last_active":"2024-04-17T07:15:12.826009+0000","last_peered":"2024-04-17T07:15:12.826009+0000","last_clean":"2024-04-17T07:15:12.826009+0000","last_became_active":"2024-04-17T07:15:11.788563+0000","last_became_peered":"2024-04-17T07:15:11.788563+0000","last_unstale":"2024-04-17T07:15:12.826009+0000","last_undegraded":"2024-04-17T07:15:12.826009+0000","last_fullsized":"2024-04-17T07:15:12.826009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:28.367949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780970+0000","last_change":"2024-04-17T07:15:11.788273+0000","last_active":"2024-04-17T07:15:12.780970+0000","last_peered":"2024-04-17T07:15:12.780970+0000","last_clean":"2024-04-17T07:15:12.780970+0000","last_became_active":"2024-04-17T07:15:11.787974+0000","last_became_peered":"2024-04-17T07:15:11.787974+0000","last_unstale":"2024-04-17T07:15:12.780970+0000","last_undegraded":"2024-04-17T07:15:12.780970+0000","last_fullsized":"2024-04-17T07:15:12.780970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789467+0000","last_change":"2024-04-17T07:15:11.788707+0000","last_active":"2024-04-17T07:15:12.789467+0000","last_peered":"2024-04-17T07:15:12.789467+0000","last_clean":"2024-04-17T07:15:12.789467+0000","last_became_active":"2024-04-17T07:15:11.788438+0000","last_became_peered":"2024-04-17T07:15:11.788438+0000","last_unstale":"2024-04-17T07:15:12.789467+0000","last_undegraded":"2024-04-17T07:15:12.789467+0000","last_fullsized":"2024-04-17T07:15:12.789467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:50.447380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787247+0000","last_change":"2024-04-17T07:15:11.791775+0000","last_active":"2024-04-17T07:15:12.787247+0000","last_peered":"2024-04-17T07:15:12.787247+0000","last_clean":"2024-04-17T07:15:12.787247+0000","last_became_active":"2024-04-17T07:15:11.791480+0000","last_became_peered":"2024-04-17T07:15:11.791480+0000","last_unstale":"2024-04-17T07:15:12.787247+0000","last_undegraded":"2024-04-17T07:15:12.787247+0000","last_fullsized":"2024-04-17T07:15:12.787247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.928000+0000","last_change":"2024-04-17T07:15:11.786623+0000","last_active":"2024-04-17T07:15:13.928000+0000","last_peered":"2024-04-17T07:15:13.928000+0000","last_clean":"2024-04-17T07:15:13.928000+0000","last_became_active":"2024-04-17T07:15:11.785422+0000","last_became_peered":"2024-04-17T07:15:11.785422+0000","last_unstale":"2024-04-17T07:15:13.928000+0000","last_undegraded":"2024-04-17T07:15:13.928000+0000","last_fullsized":"2024-04-17T07:15:13.928000+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:35.216294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798967+0000","last_change":"2024-04-17T07:15:11.780193+0000","last_active":"2024-04-17T07:15:12.798967+0000","last_peered":"2024-04-17T07:15:12.798967+0000","last_clean":"2024-04-17T07:15:12.798967+0000","last_became_active":"2024-04-17T07:15:11.779177+0000","last_became_peered":"2024-04-17T07:15:11.779177+0000","last_unstale":"2024-04-17T07:15:12.798967+0000","last_undegraded":"2024-04-17T07:15:12.798967+0000","last_fullsized":"2024-04-17T07:15:12.798967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789965+0000","last_change":"2024-04-17T07:15:11.790940+0000","last_active":"2024-04-17T07:15:12.789965+0000","last_peered":"2024-04-17T07:15:12.789965+0000","last_clean":"2024-04-17T07:15:12.789965+0000","last_became_active":"2024-04-17T07:15:11.790272+0000","last_became_peered":"2024-04-17T07:15:11.790272+0000","last_unstale":"2024-04-17T07:15:12.789965+0000","last_undegraded":"2024-04-17T07:15:12.789965+0000","last_fullsized":"2024-04-17T07:15:12.789965+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:40.742949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788688+0000","last_change":"2024-04-17T07:15:11.780496+0000","last_active":"2024-04-17T07:15:12.788688+0000","last_peered":"2024-04-17T07:15:12.788688+0000","last_clean":"2024-04-17T07:15:12.788688+0000","last_became_active":"2024-04-17T07:15:11.778226+0000","last_became_peered":"2024-04-17T07:15:11.778226+0000","last_unstale":"2024-04-17T07:15:12.788688+0000","last_undegraded":"2024-04-17T07:15:12.788688+0000","last_fullsized":"2024-04-17T07:15:12.788688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826192+0000","last_change":"2024-04-17T07:15:11.780581+0000","last_active":"2024-04-17T07:15:12.826192+0000","last_peered":"2024-04-17T07:15:12.826192+0000","last_clean":"2024-04-17T07:15:12.826192+0000","last_became_active":"2024-04-17T07:15:11.777868+0000","last_became_peered":"2024-04-17T07:15:11.777868+0000","last_unstale":"2024-04-17T07:15:12.826192+0000","last_undegraded":"2024-04-17T07:15:12.826192+0000","last_fullsized":"2024-04-17T07:15:12.826192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:02.045200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781070+0000","last_change":"2024-04-17T07:15:11.794690+0000","last_active":"2024-04-17T07:15:12.781070+0000","last_peered":"2024-04-17T07:15:12.781070+0000","last_clean":"2024-04-17T07:15:12.781070+0000","last_became_active":"2024-04-17T07:15:11.790136+0000","last_became_peered":"2024-04-17T07:15:11.790136+0000","last_unstale":"2024-04-17T07:15:12.781070+0000","last_undegraded":"2024-04-17T07:15:12.781070+0000","last_fullsized":"2024-04-17T07:15:12.781070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789910+0000","last_change":"2024-04-17T07:15:11.784337+0000","last_active":"2024-04-17T07:15:12.789910+0000","last_peered":"2024-04-17T07:15:12.789910+0000","last_clean":"2024-04-17T07:15:12.789910+0000","last_became_active":"2024-04-17T07:15:11.783994+0000","last_became_peered":"2024-04-17T07:15:11.783994+0000","last_unstale":"2024-04-17T07:15:12.789910+0000","last_undegraded":"2024-04-17T07:15:12.789910+0000","last_fullsized":"2024-04-17T07:15:12.789910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788860+0000","last_change":"2024-04-17T07:15:11.801844+0000","last_active":"2024-04-17T07:15:12.788860+0000","last_peered":"2024-04-17T07:15:12.788860+0000","last_clean":"2024-04-17T07:15:12.788860+0000","last_became_active":"2024-04-17T07:15:11.801434+0000","last_became_peered":"2024-04-17T07:15:11.801434+0000","last_unstale":"2024-04-17T07:15:12.788860+0000","last_undegraded":"2024-04-17T07:15:12.788860+0000","last_fullsized":"2024-04-17T07:15:12.788860+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:37.971258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788486+0000","last_change":"2024-04-17T07:15:11.787720+0000","last_active":"2024-04-17T07:15:12.788486+0000","last_peered":"2024-04-17T07:15:12.788486+0000","last_clean":"2024-04-17T07:15:12.788486+0000","last_became_active":"2024-04-17T07:15:11.787372+0000","last_became_peered":"2024-04-17T07:15:11.787372+0000","last_unstale":"2024-04-17T07:15:12.788486+0000","last_undegraded":"2024-04-17T07:15:12.788486+0000","last_fullsized":"2024-04-17T07:15:12.788486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781699+0000","last_change":"2024-04-17T07:15:11.784260+0000","last_active":"2024-04-17T07:15:12.781699+0000","last_peered":"2024-04-17T07:15:12.781699+0000","last_clean":"2024-04-17T07:15:12.781699+0000","last_became_active":"2024-04-17T07:15:11.783844+0000","last_became_peered":"2024-04-17T07:15:11.783844+0000","last_unstale":"2024-04-17T07:15:12.781699+0000","last_undegraded":"2024-04-17T07:15:12.781699+0000","last_fullsized":"2024-04-17T07:15:12.781699+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:50.787356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781167+0000","last_change":"2024-04-17T07:15:11.785355+0000","last_active":"2024-04-17T07:15:12.781167+0000","last_peered":"2024-04-17T07:15:12.781167+0000","last_clean":"2024-04-17T07:15:12.781167+0000","last_became_active":"2024-04-17T07:15:11.785044+0000","last_became_peered":"2024-04-17T07:15:11.785044+0000","last_unstale":"2024-04-17T07:15:12.781167+0000","last_undegraded":"2024-04-17T07:15:12.781167+0000","last_fullsized":"2024-04-17T07:15:12.781167+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825968+0000","last_change":"2024-04-17T07:15:11.786281+0000","last_active":"2024-04-17T07:15:12.825968+0000","last_peered":"2024-04-17T07:15:12.825968+0000","last_clean":"2024-04-17T07:15:12.825968+0000","last_became_active":"2024-04-17T07:15:11.782295+0000","last_became_peered":"2024-04-17T07:15:11.782295+0000","last_unstale":"2024-04-17T07:15:12.825968+0000","last_undegraded":"2024-04-17T07:15:12.825968+0000","last_fullsized":"2024-04-17T07:15:12.825968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:26.827107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926990+0000","last_change":"2024-04-17T07:15:11.780594+0000","last_active":"2024-04-17T07:15:13.926990+0000","last_peered":"2024-04-17T07:15:13.926990+0000","last_clean":"2024-04-17T07:15:13.926990+0000","last_became_active":"2024-04-17T07:15:11.780233+0000","last_became_peered":"2024-04-17T07:15:11.780233+0000","last_unstale":"2024-04-17T07:15:13.926990+0000","last_undegraded":"2024-04-17T07:15:13.926990+0000","last_fullsized":"2024-04-17T07:15:13.926990+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:34.187674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927194+0000","last_change":"2024-04-17T07:15:11.783398+0000","last_active":"2024-04-17T07:15:13.927194+0000","last_peered":"2024-04-17T07:15:13.927194+0000","last_clean":"2024-04-17T07:15:13.927194+0000","last_became_active":"2024-04-17T07:15:11.783101+0000","last_became_peered":"2024-04-17T07:15:11.783101+0000","last_unstale":"2024-04-17T07:15:13.927194+0000","last_undegraded":"2024-04-17T07:15:13.927194+0000","last_fullsized":"2024-04-17T07:15:13.927194+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:16.977647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.793928+0000","last_change":"2024-04-17T07:15:11.794368+0000","last_active":"2024-04-17T07:15:12.793928+0000","last_peered":"2024-04-17T07:15:12.793928+0000","last_clean":"2024-04-17T07:15:12.793928+0000","last_became_active":"2024-04-17T07:15:11.793994+0000","last_became_peered":"2024-04-17T07:15:11.793994+0000","last_unstale":"2024-04-17T07:15:12.793928+0000","last_undegraded":"2024-04-17T07:15:12.793928+0000","last_fullsized":"2024-04-17T07:15:12.793928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:15.896666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787219+0000","last_change":"2024-04-17T07:15:11.783232+0000","last_active":"2024-04-17T07:15:12.787219+0000","last_peered":"2024-04-17T07:15:12.787219+0000","last_clean":"2024-04-17T07:15:12.787219+0000","last_became_active":"2024-04-17T07:15:11.782887+0000","last_became_peered":"2024-04-17T07:15:11.782887+0000","last_unstale":"2024-04-17T07:15:12.787219+0000","last_undegraded":"2024-04-17T07:15:12.787219+0000","last_fullsized":"2024-04-17T07:15:12.787219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:29.771615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799317+0000","last_change":"2024-04-17T07:15:11.791102+0000","last_active":"2024-04-17T07:15:12.799317+0000","last_peered":"2024-04-17T07:15:12.799317+0000","last_clean":"2024-04-17T07:15:12.799317+0000","last_became_active":"2024-04-17T07:15:11.789629+0000","last_became_peered":"2024-04-17T07:15:11.789629+0000","last_unstale":"2024-04-17T07:15:12.799317+0000","last_undegraded":"2024-04-17T07:15:12.799317+0000","last_fullsized":"2024-04-17T07:15:12.799317+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781899+0000","last_change":"2024-04-17T07:15:11.798951+0000","last_active":"2024-04-17T07:15:12.781899+0000","last_peered":"2024-04-17T07:15:12.781899+0000","last_clean":"2024-04-17T07:15:12.781899+0000","last_became_active":"2024-04-17T07:15:11.796782+0000","last_became_peered":"2024-04-17T07:15:11.796782+0000","last_unstale":"2024-04-17T07:15:12.781899+0000","last_undegraded":"2024-04-17T07:15:12.781899+0000","last_fullsized":"2024-04-17T07:15:12.781899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:35:31.757126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799208+0000","last_change":"2024-04-17T07:15:11.791795+0000","last_active":"2024-04-17T07:15:12.799208+0000","last_peered":"2024-04-17T07:15:12.799208+0000","last_clean":"2024-04-17T07:15:12.799208+0000","last_became_active":"2024-04-17T07:15:11.791304+0000","last_became_peered":"2024-04-17T07:15:11.791304+0000","last_unstale":"2024-04-17T07:15:12.799208+0000","last_undegraded":"2024-04-17T07:15:12.799208+0000","last_fullsized":"2024-04-17T07:15:12.799208+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787262+0000","last_change":"2024-04-17T07:15:11.773030+0000","last_active":"2024-04-17T07:15:12.787262+0000","last_peered":"2024-04-17T07:15:12.787262+0000","last_clean":"2024-04-17T07:15:12.787262+0000","last_became_active":"2024-04-17T07:15:11.772331+0000","last_became_peered":"2024-04-17T07:15:11.772331+0000","last_unstale":"2024-04-17T07:15:12.787262+0000","last_undegraded":"2024-04-17T07:15:12.787262+0000","last_fullsized":"2024-04-17T07:15:12.787262+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:49.061011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927027+0000","last_change":"2024-04-17T07:15:11.781460+0000","last_active":"2024-04-17T07:15:13.927027+0000","last_peered":"2024-04-17T07:15:13.927027+0000","last_clean":"2024-04-17T07:15:13.927027+0000","last_became_active":"2024-04-17T07:15:11.781103+0000","last_became_peered":"2024-04-17T07:15:11.781103+0000","last_unstale":"2024-04-17T07:15:13.927027+0000","last_undegraded":"2024-04-17T07:15:13.927027+0000","last_fullsized":"2024-04-17T07:15:13.927027+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:11.476872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787980+0000","last_change":"2024-04-17T07:15:11.783097+0000","last_active":"2024-04-17T07:15:12.787980+0000","last_peered":"2024-04-17T07:15:12.787980+0000","last_clean":"2024-04-17T07:15:12.787980+0000","last_became_active":"2024-04-17T07:15:11.782801+0000","last_became_peered":"2024-04-17T07:15:11.782801+0000","last_unstale":"2024-04-17T07:15:12.787980+0000","last_undegraded":"2024-04-17T07:15:12.787980+0000","last_fullsized":"2024-04-17T07:15:12.787980+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:02.524480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865210+0000","last_change":"2024-04-17T07:15:11.783092+0000","last_active":"2024-04-17T07:15:12.865210+0000","last_peered":"2024-04-17T07:15:12.865210+0000","last_clean":"2024-04-17T07:15:12.865210+0000","last_became_active":"2024-04-17T07:15:11.782733+0000","last_became_peered":"2024-04-17T07:15:11.782733+0000","last_unstale":"2024-04-17T07:15:12.865210+0000","last_undegraded":"2024-04-17T07:15:12.865210+0000","last_fullsized":"2024-04-17T07:15:12.865210+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:17.023666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787669+0000","last_change":"2024-04-17T07:15:11.789442+0000","last_active":"2024-04-17T07:15:12.787669+0000","last_peered":"2024-04-17T07:15:12.787669+0000","last_clean":"2024-04-17T07:15:12.787669+0000","last_became_active":"2024-04-17T07:15:11.788735+0000","last_became_peered":"2024-04-17T07:15:11.788735+0000","last_unstale":"2024-04-17T07:15:12.787669+0000","last_undegraded":"2024-04-17T07:15:12.787669+0000","last_fullsized":"2024-04-17T07:15:12.787669+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794612+0000","last_change":"2024-04-17T07:15:11.792035+0000","last_active":"2024-04-17T07:15:12.794612+0000","last_peered":"2024-04-17T07:15:12.794612+0000","last_clean":"2024-04-17T07:15:12.794612+0000","last_became_active":"2024-04-17T07:15:11.791726+0000","last_became_peered":"2024-04-17T07:15:11.791726+0000","last_unstale":"2024-04-17T07:15:12.794612+0000","last_undegraded":"2024-04-17T07:15:12.794612+0000","last_fullsized":"2024-04-17T07:15:12.794612+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:01:01.580287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865739+0000","last_change":"2024-04-17T07:15:10.760324+0000","last_active":"2024-04-17T07:15:12.865739+0000","last_peered":"2024-04-17T07:15:12.865739+0000","last_clean":"2024-04-17T07:15:12.865739+0000","last_became_active":"2024-04-17T07:15:10.759991+0000","last_became_peered":"2024-04-17T07:15:10.759991+0000","last_unstale":"2024-04-17T07:15:12.865739+0000","last_undegraded":"2024-04-17T07:15:12.865739+0000","last_fullsized":"2024-04-17T07:15:12.865739+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:24.506173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865657+0000","last_change":"2024-04-17T07:15:11.787483+0000","last_active":"2024-04-17T07:15:12.865657+0000","last_peered":"2024-04-17T07:15:12.865657+0000","last_clean":"2024-04-17T07:15:12.865657+0000","last_became_active":"2024-04-17T07:15:11.787086+0000","last_became_peered":"2024-04-17T07:15:11.787086+0000","last_unstale":"2024-04-17T07:15:12.865657+0000","last_undegraded":"2024-04-17T07:15:12.865657+0000","last_fullsized":"2024-04-17T07:15:12.865657+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:11.476872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794555+0000","last_change":"2024-04-17T07:15:10.753951+0000","last_active":"2024-04-17T07:15:12.794555+0000","last_peered":"2024-04-17T07:15:12.794555+0000","last_clean":"2024-04-17T07:15:12.794555+0000","last_became_active":"2024-04-17T07:15:10.753762+0000","last_became_peered":"2024-04-17T07:15:10.753762+0000","last_unstale":"2024-04-17T07:15:12.794555+0000","last_undegraded":"2024-04-17T07:15:12.794555+0000","last_fullsized":"2024-04-17T07:15:12.794555+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:25.821388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927920+0000","last_change":"2024-04-17T07:15:11.785316+0000","last_active":"2024-04-17T07:15:13.927920+0000","last_peered":"2024-04-17T07:15:13.927920+0000","last_clean":"2024-04-17T07:15:13.927920+0000","last_became_active":"2024-04-17T07:15:11.782457+0000","last_became_peered":"2024-04-17T07:15:11.782457+0000","last_unstale":"2024-04-17T07:15:13.927920+0000","last_undegraded":"2024-04-17T07:15:13.927920+0000","last_fullsized":"2024-04-17T07:15:13.927920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:17.674336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781218+0000","last_change":"2024-04-17T07:15:10.753297+0000","last_active":"2024-04-17T07:15:12.781218+0000","last_peered":"2024-04-17T07:15:12.781218+0000","last_clean":"2024-04-17T07:15:12.781218+0000","last_became_active":"2024-04-17T07:15:10.753007+0000","last_became_peered":"2024-04-17T07:15:10.753007+0000","last_unstale":"2024-04-17T07:15:12.781218+0000","last_undegraded":"2024-04-17T07:15:12.781218+0000","last_fullsized":"2024-04-17T07:15:12.781218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:33.181954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799412+0000","last_change":"2024-04-17T07:15:11.791434+0000","last_active":"2024-04-17T07:15:12.799412+0000","last_peered":"2024-04-17T07:15:12.799412+0000","last_clean":"2024-04-17T07:15:12.799412+0000","last_became_active":"2024-04-17T07:15:11.791105+0000","last_became_peered":"2024-04-17T07:15:11.791105+0000","last_unstale":"2024-04-17T07:15:12.799412+0000","last_undegraded":"2024-04-17T07:15:12.799412+0000","last_fullsized":"2024-04-17T07:15:12.799412+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825937+0000","last_change":"2024-04-17T07:15:10.759387+0000","last_active":"2024-04-17T07:15:12.825937+0000","last_peered":"2024-04-17T07:15:12.825937+0000","last_clean":"2024-04-17T07:15:12.825937+0000","last_became_active":"2024-04-17T07:15:10.758609+0000","last_became_peered":"2024-04-17T07:15:10.758609+0000","last_unstale":"2024-04-17T07:15:12.825937+0000","last_undegraded":"2024-04-17T07:15:12.825937+0000","last_fullsized":"2024-04-17T07:15:12.825937+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:22.823436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788454+0000","last_change":"2024-04-17T07:15:11.786963+0000","last_active":"2024-04-17T07:15:12.788454+0000","last_peered":"2024-04-17T07:15:12.788454+0000","last_clean":"2024-04-17T07:15:12.788454+0000","last_became_active":"2024-04-17T07:15:11.786663+0000","last_became_peered":"2024-04-17T07:15:11.786663+0000","last_unstale":"2024-04-17T07:15:12.788454+0000","last_undegraded":"2024-04-17T07:15:12.788454+0000","last_fullsized":"2024-04-17T07:15:12.788454+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:51.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927884+0000","last_change":"2024-04-17T07:15:10.757342+0000","last_active":"2024-04-17T07:15:13.927884+0000","last_peered":"2024-04-17T07:15:13.927884+0000","last_clean":"2024-04-17T07:15:13.927884+0000","last_became_active":"2024-04-17T07:15:10.757059+0000","last_became_peered":"2024-04-17T07:15:10.757059+0000","last_unstale":"2024-04-17T07:15:13.927884+0000","last_undegraded":"2024-04-17T07:15:13.927884+0000","last_fullsized":"2024-04-17T07:15:13.927884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:24.506173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788121+0000","last_change":"2024-04-17T07:15:11.772728+0000","last_active":"2024-04-17T07:15:12.788121+0000","last_peered":"2024-04-17T07:15:12.788121+0000","last_clean":"2024-04-17T07:15:12.788121+0000","last_became_active":"2024-04-17T07:15:11.771937+0000","last_became_peered":"2024-04-17T07:15:11.771937+0000","last_unstale":"2024-04-17T07:15:12.788121+0000","last_undegraded":"2024-04-17T07:15:12.788121+0000","last_fullsized":"2024-04-17T07:15:12.788121+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926892+0000","last_change":"2024-04-17T07:15:10.759698+0000","last_active":"2024-04-17T07:15:13.926892+0000","last_peered":"2024-04-17T07:15:13.926892+0000","last_clean":"2024-04-17T07:15:13.926892+0000","last_became_active":"2024-04-17T07:15:10.758698+0000","last_became_peered":"2024-04-17T07:15:10.758698+0000","last_unstale":"2024-04-17T07:15:13.926892+0000","last_undegraded":"2024-04-17T07:15:13.926892+0000","last_fullsized":"2024-04-17T07:15:13.926892+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:05.510781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927260+0000","last_change":"2024-04-17T07:15:11.786810+0000","last_active":"2024-04-17T07:15:13.927260+0000","last_peered":"2024-04-17T07:15:13.927260+0000","last_clean":"2024-04-17T07:15:13.927260+0000","last_became_active":"2024-04-17T07:15:11.786357+0000","last_became_peered":"2024-04-17T07:15:11.786357+0000","last_unstale":"2024-04-17T07:15:13.927260+0000","last_undegraded":"2024-04-17T07:15:13.927260+0000","last_fullsized":"2024-04-17T07:15:13.927260+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:11.999039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780769+0000","last_change":"2024-04-17T07:15:10.744944+0000","last_active":"2024-04-17T07:15:12.780769+0000","last_peered":"2024-04-17T07:15:12.780769+0000","last_clean":"2024-04-17T07:15:12.780769+0000","last_became_active":"2024-04-17T07:15:10.744632+0000","last_became_peered":"2024-04-17T07:15:10.744632+0000","last_unstale":"2024-04-17T07:15:12.780769+0000","last_undegraded":"2024-04-17T07:15:12.780769+0000","last_fullsized":"2024-04-17T07:15:12.780769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798369+0000","last_change":"2024-04-17T07:15:11.784231+0000","last_active":"2024-04-17T07:15:12.798369+0000","last_peered":"2024-04-17T07:15:12.798369+0000","last_clean":"2024-04-17T07:15:12.798369+0000","last_became_active":"2024-04-17T07:15:11.784070+0000","last_became_peered":"2024-04-17T07:15:11.784070+0000","last_unstale":"2024-04-17T07:15:12.798369+0000","last_undegraded":"2024-04-17T07:15:12.798369+0000","last_fullsized":"2024-04-17T07:15:12.798369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:36.110314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926716+0000","last_change":"2024-04-17T07:15:10.753739+0000","last_active":"2024-04-17T07:15:13.926716+0000","last_peered":"2024-04-17T07:15:13.926716+0000","last_clean":"2024-04-17T07:15:13.926716+0000","last_became_active":"2024-04-17T07:15:10.753411+0000","last_became_peered":"2024-04-17T07:15:10.753411+0000","last_unstale":"2024-04-17T07:15:13.926716+0000","last_undegraded":"2024-04-17T07:15:13.926716+0000","last_fullsized":"2024-04-17T07:15:13.926716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:22.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788318+0000","last_change":"2024-04-17T07:15:11.779401+0000","last_active":"2024-04-17T07:15:12.788318+0000","last_peered":"2024-04-17T07:15:12.788318+0000","last_clean":"2024-04-17T07:15:12.788318+0000","last_became_active":"2024-04-17T07:15:11.779116+0000","last_became_peered":"2024-04-17T07:15:11.779116+0000","last_unstale":"2024-04-17T07:15:12.788318+0000","last_undegraded":"2024-04-17T07:15:12.788318+0000","last_fullsized":"2024-04-17T07:15:12.788318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:51.907975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781584+0000","last_change":"2024-04-17T07:15:10.756984+0000","last_active":"2024-04-17T07:15:12.781584+0000","last_peered":"2024-04-17T07:15:12.781584+0000","last_clean":"2024-04-17T07:15:12.781584+0000","last_became_active":"2024-04-17T07:15:10.755966+0000","last_became_peered":"2024-04-17T07:15:10.755966+0000","last_unstale":"2024-04-17T07:15:12.781584+0000","last_undegraded":"2024-04-17T07:15:12.781584+0000","last_fullsized":"2024-04-17T07:15:12.781584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:47.288197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781661+0000","last_change":"2024-04-17T07:15:11.783941+0000","last_active":"2024-04-17T07:15:12.781661+0000","last_peered":"2024-04-17T07:15:12.781661+0000","last_clean":"2024-04-17T07:15:12.781661+0000","last_became_active":"2024-04-17T07:15:11.783578+0000","last_became_peered":"2024-04-17T07:15:11.783578+0000","last_unstale":"2024-04-17T07:15:12.781661+0000","last_undegraded":"2024-04-17T07:15:12.781661+0000","last_fullsized":"2024-04-17T07:15:12.781661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:23.035859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794614+0000","last_change":"2024-04-17T07:15:10.750003+0000","last_active":"2024-04-17T07:15:12.794614+0000","last_peered":"2024-04-17T07:15:12.794614+0000","last_clean":"2024-04-17T07:15:12.794614+0000","last_became_active":"2024-04-17T07:15:10.749706+0000","last_became_peered":"2024-04-17T07:15:10.749706+0000","last_unstale":"2024-04-17T07:15:12.794614+0000","last_undegraded":"2024-04-17T07:15:12.794614+0000","last_fullsized":"2024-04-17T07:15:12.794614+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:49.742557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781197+0000","last_change":"2024-04-17T07:15:11.790257+0000","last_active":"2024-04-17T07:15:12.781197+0000","last_peered":"2024-04-17T07:15:12.781197+0000","last_clean":"2024-04-17T07:15:12.781197+0000","last_became_active":"2024-04-17T07:15:11.789887+0000","last_became_peered":"2024-04-17T07:15:11.789887+0000","last_unstale":"2024-04-17T07:15:12.781197+0000","last_undegraded":"2024-04-17T07:15:12.781197+0000","last_fullsized":"2024-04-17T07:15:12.781197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:50.447380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":17,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787707+0000","last_change":"2024-04-17T07:15:10.754401+0000","last_active":"2024-04-17T07:15:12.787707+0000","last_peered":"2024-04-17T07:15:12.787707+0000","last_clean":"2024-04-17T07:15:12.787707+0000","last_became_active":"2024-04-17T07:15:10.754054+0000","last_became_peered":"2024-04-17T07:15:10.754054+0000","last_unstale":"2024-04-17T07:15:12.787707+0000","last_undegraded":"2024-04-17T07:15:12.787707+0000","last_fullsized":"2024-04-17T07:15:12.787707+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:58.814832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.787560+0000","last_change":"2024-04-17T07:15:11.781362+0000","last_active":"2024-04-17T07:15:12.787560+0000","last_peered":"2024-04-17T07:15:12.787560+0000","last_clean":"2024-04-17T07:15:12.787560+0000","last_became_active":"2024-04-17T07:15:11.780192+0000","last_became_peered":"2024-04-17T07:15:11.780192+0000","last_unstale":"2024-04-17T07:15:12.787560+0000","last_undegraded":"2024-04-17T07:15:12.787560+0000","last_fullsized":"2024-04-17T07:15:12.787560+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:49.369753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781217+0000","last_change":"2024-04-17T07:15:10.753307+0000","last_active":"2024-04-17T07:15:12.781217+0000","last_peered":"2024-04-17T07:15:12.781217+0000","last_clean":"2024-04-17T07:15:12.781217+0000","last_became_active":"2024-04-17T07:15:10.753049+0000","last_became_peered":"2024-04-17T07:15:10.753049+0000","last_unstale":"2024-04-17T07:15:12.781217+0000","last_undegraded":"2024-04-17T07:15:12.781217+0000","last_fullsized":"2024-04-17T07:15:12.781217+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:10.993320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798324+0000","last_change":"2024-04-17T07:15:11.784972+0000","last_active":"2024-04-17T07:15:12.798324+0000","last_peered":"2024-04-17T07:15:12.798324+0000","last_clean":"2024-04-17T07:15:12.798324+0000","last_became_active":"2024-04-17T07:15:11.784776+0000","last_became_peered":"2024-04-17T07:15:11.784776+0000","last_unstale":"2024-04-17T07:15:12.798324+0000","last_undegraded":"2024-04-17T07:15:12.798324+0000","last_fullsized":"2024-04-17T07:15:12.798324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:17.756837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927265+0000","last_change":"2024-04-17T07:15:10.752949+0000","last_active":"2024-04-17T07:15:13.927265+0000","last_peered":"2024-04-17T07:15:13.927265+0000","last_clean":"2024-04-17T07:15:13.927265+0000","last_became_active":"2024-04-17T07:15:10.752625+0000","last_became_peered":"2024-04-17T07:15:10.752625+0000","last_unstale":"2024-04-17T07:15:13.927265+0000","last_undegraded":"2024-04-17T07:15:13.927265+0000","last_fullsized":"2024-04-17T07:15:13.927265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:59.786245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.781202+0000","last_change":"2024-04-17T07:15:11.787599+0000","last_active":"2024-04-17T07:15:12.781202+0000","last_peered":"2024-04-17T07:15:12.781202+0000","last_clean":"2024-04-17T07:15:12.781202+0000","last_became_active":"2024-04-17T07:15:11.787356+0000","last_became_peered":"2024-04-17T07:15:11.787356+0000","last_unstale":"2024-04-17T07:15:12.781202+0000","last_undegraded":"2024-04-17T07:15:12.781202+0000","last_fullsized":"2024-04-17T07:15:12.781202+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:40.742949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798478+0000","last_change":"2024-04-17T07:15:10.754700+0000","last_active":"2024-04-17T07:15:12.798478+0000","last_peered":"2024-04-17T07:15:12.798478+0000","last_clean":"2024-04-17T07:15:12.798478+0000","last_became_active":"2024-04-17T07:15:10.754438+0000","last_became_peered":"2024-04-17T07:15:10.754438+0000","last_unstale":"2024-04-17T07:15:12.798478+0000","last_undegraded":"2024-04-17T07:15:12.798478+0000","last_fullsized":"2024-04-17T07:15:12.798478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:18.428942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788103+0000","last_change":"2024-04-17T07:15:11.790073+0000","last_active":"2024-04-17T07:15:12.788103+0000","last_peered":"2024-04-17T07:15:12.788103+0000","last_clean":"2024-04-17T07:15:12.788103+0000","last_became_active":"2024-04-17T07:15:11.789138+0000","last_became_peered":"2024-04-17T07:15:11.789138+0000","last_unstale":"2024-04-17T07:15:12.788103+0000","last_undegraded":"2024-04-17T07:15:12.788103+0000","last_fullsized":"2024-04-17T07:15:12.788103+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:42.168172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780841+0000","last_change":"2024-04-17T07:15:10.753648+0000","last_active":"2024-04-17T07:15:12.780841+0000","last_peered":"2024-04-17T07:15:12.780841+0000","last_clean":"2024-04-17T07:15:12.780841+0000","last_became_active":"2024-04-17T07:15:10.752763+0000","last_became_peered":"2024-04-17T07:15:10.752763+0000","last_unstale":"2024-04-17T07:15:12.780841+0000","last_undegraded":"2024-04-17T07:15:12.780841+0000","last_fullsized":"2024-04-17T07:15:12.780841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:58.814832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865402+0000","last_change":"2024-04-17T07:15:11.792573+0000","last_active":"2024-04-17T07:15:12.865402+0000","last_peered":"2024-04-17T07:15:12.865402+0000","last_clean":"2024-04-17T07:15:12.865402+0000","last_became_active":"2024-04-17T07:15:11.787417+0000","last_became_peered":"2024-04-17T07:15:11.787417+0000","last_unstale":"2024-04-17T07:15:12.865402+0000","last_undegraded":"2024-04-17T07:15:12.865402+0000","last_fullsized":"2024-04-17T07:15:12.865402+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:59.820552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":22,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780352+0000","last_change":"2024-04-17T07:15:10.750662+0000","last_active":"2024-04-17T07:15:12.780352+0000","last_peered":"2024-04-17T07:15:12.780352+0000","last_clean":"2024-04-17T07:15:12.780352+0000","last_became_active":"2024-04-17T07:15:10.750363+0000","last_became_peered":"2024-04-17T07:15:10.750363+0000","last_unstale":"2024-04-17T07:15:12.780352+0000","last_undegraded":"2024-04-17T07:15:12.780352+0000","last_fullsized":"2024-04-17T07:15:12.780352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:10.471152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927533+0000","last_change":"2024-04-17T07:15:11.781408+0000","last_active":"2024-04-17T07:15:13.927533+0000","last_peered":"2024-04-17T07:15:13.927533+0000","last_clean":"2024-04-17T07:15:13.927533+0000","last_became_active":"2024-04-17T07:15:11.780292+0000","last_became_peered":"2024-04-17T07:15:11.780292+0000","last_unstale":"2024-04-17T07:15:13.927533+0000","last_undegraded":"2024-04-17T07:15:13.927533+0000","last_fullsized":"2024-04-17T07:15:13.927533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:48.293916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799370+0000","last_change":"2024-04-17T07:15:10.753995+0000","last_active":"2024-04-17T07:15:12.799370+0000","last_peered":"2024-04-17T07:15:12.799370+0000","last_clean":"2024-04-17T07:15:12.799370+0000","last_became_active":"2024-04-17T07:15:10.749692+0000","last_became_peered":"2024-04-17T07:15:10.749692+0000","last_unstale":"2024-04-17T07:15:12.799370+0000","last_undegraded":"2024-04-17T07:15:12.799370+0000","last_fullsized":"2024-04-17T07:15:12.799370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:16.017947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788628+0000","last_change":"2024-04-17T07:15:11.786775+0000","last_active":"2024-04-17T07:15:12.788628+0000","last_peered":"2024-04-17T07:15:12.788628+0000","last_clean":"2024-04-17T07:15:12.788628+0000","last_became_active":"2024-04-17T07:15:11.786284+0000","last_became_peered":"2024-04-17T07:15:11.786284+0000","last_unstale":"2024-04-17T07:15:12.788628+0000","last_undegraded":"2024-04-17T07:15:12.788628+0000","last_fullsized":"2024-04-17T07:15:12.788628+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927329+0000","last_change":"2024-04-17T07:15:10.758511+0000","last_active":"2024-04-17T07:15:13.927329+0000","last_peered":"2024-04-17T07:15:13.927329+0000","last_clean":"2024-04-17T07:15:13.927329+0000","last_became_active":"2024-04-17T07:15:10.757894+0000","last_became_peered":"2024-04-17T07:15:10.757894+0000","last_unstale":"2024-04-17T07:15:13.927329+0000","last_undegraded":"2024-04-17T07:15:13.927329+0000","last_fullsized":"2024-04-17T07:15:13.927329+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:47.131852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798899+0000","last_change":"2024-04-17T07:15:11.781275+0000","last_active":"2024-04-17T07:15:12.798899+0000","last_peered":"2024-04-17T07:15:12.798899+0000","last_clean":"2024-04-17T07:15:12.798899+0000","last_became_active":"2024-04-17T07:15:11.780976+0000","last_became_peered":"2024-04-17T07:15:11.780976+0000","last_unstale":"2024-04-17T07:15:12.798899+0000","last_undegraded":"2024-04-17T07:15:12.798899+0000","last_fullsized":"2024-04-17T07:15:12.798899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:29.771615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927598+0000","last_change":"2024-04-17T07:15:10.756433+0000","last_active":"2024-04-17T07:15:13.927598+0000","last_peered":"2024-04-17T07:15:13.927598+0000","last_clean":"2024-04-17T07:15:13.927598+0000","last_became_active":"2024-04-17T07:15:10.756141+0000","last_became_peered":"2024-04-17T07:15:10.756141+0000","last_unstale":"2024-04-17T07:15:13.927598+0000","last_undegraded":"2024-04-17T07:15:13.927598+0000","last_fullsized":"2024-04-17T07:15:13.927598+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:43.744189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825883+0000","last_change":"2024-04-17T07:15:11.789536+0000","last_active":"2024-04-17T07:15:12.825883+0000","last_peered":"2024-04-17T07:15:12.825883+0000","last_clean":"2024-04-17T07:15:12.825883+0000","last_became_active":"2024-04-17T07:15:11.789254+0000","last_became_peered":"2024-04-17T07:15:11.789254+0000","last_unstale":"2024-04-17T07:15:12.825883+0000","last_undegraded":"2024-04-17T07:15:12.825883+0000","last_fullsized":"2024-04-17T07:15:12.825883+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:50.687319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927850+0000","last_change":"2024-04-17T07:15:10.758159+0000","last_active":"2024-04-17T07:15:13.927850+0000","last_peered":"2024-04-17T07:15:13.927850+0000","last_clean":"2024-04-17T07:15:13.927850+0000","last_became_active":"2024-04-17T07:15:10.757799+0000","last_became_peered":"2024-04-17T07:15:10.757799+0000","last_unstale":"2024-04-17T07:15:13.927850+0000","last_undegraded":"2024-04-17T07:15:13.927850+0000","last_fullsized":"2024-04-17T07:15:13.927850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:02.814854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825758+0000","last_change":"2024-04-17T07:15:11.793490+0000","last_active":"2024-04-17T07:15:12.825758+0000","last_peered":"2024-04-17T07:15:12.825758+0000","last_clean":"2024-04-17T07:15:12.825758+0000","last_became_active":"2024-04-17T07:15:11.792875+0000","last_became_peered":"2024-04-17T07:15:11.792875+0000","last_unstale":"2024-04-17T07:15:12.825758+0000","last_undegraded":"2024-04-17T07:15:12.825758+0000","last_fullsized":"2024-04-17T07:15:12.825758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:41.684920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794219+0000","last_change":"2024-04-17T07:15:10.756878+0000","last_active":"2024-04-17T07:15:12.794219+0000","last_peered":"2024-04-17T07:15:12.794219+0000","last_clean":"2024-04-17T07:15:12.794219+0000","last_became_active":"2024-04-17T07:15:10.755581+0000","last_became_peered":"2024-04-17T07:15:10.755581+0000","last_unstale":"2024-04-17T07:15:12.794219+0000","last_undegraded":"2024-04-17T07:15:12.794219+0000","last_fullsized":"2024-04-17T07:15:12.794219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:49.681600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826140+0000","last_change":"2024-04-17T07:15:11.777553+0000","last_active":"2024-04-17T07:15:12.826140+0000","last_peered":"2024-04-17T07:15:12.826140+0000","last_clean":"2024-04-17T07:15:12.826140+0000","last_became_active":"2024-04-17T07:15:11.774880+0000","last_became_peered":"2024-04-17T07:15:11.774880+0000","last_unstale":"2024-04-17T07:15:12.826140+0000","last_undegraded":"2024-04-17T07:15:12.826140+0000","last_fullsized":"2024-04-17T07:15:12.826140+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:00.042145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865131+0000","last_change":"2024-04-17T07:15:10.756968+0000","last_active":"2024-04-17T07:15:12.865131+0000","last_peered":"2024-04-17T07:15:12.865131+0000","last_clean":"2024-04-17T07:15:12.865131+0000","last_became_active":"2024-04-17T07:15:10.756661+0000","last_became_peered":"2024-04-17T07:15:10.756661+0000","last_unstale":"2024-04-17T07:15:12.865131+0000","last_undegraded":"2024-04-17T07:15:12.865131+0000","last_fullsized":"2024-04-17T07:15:12.865131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:22.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.825752+0000","last_change":"2024-04-17T07:15:11.776185+0000","last_active":"2024-04-17T07:15:12.825752+0000","last_peered":"2024-04-17T07:15:12.825752+0000","last_clean":"2024-04-17T07:15:12.825752+0000","last_became_active":"2024-04-17T07:15:11.775873+0000","last_became_peered":"2024-04-17T07:15:11.775873+0000","last_unstale":"2024-04-17T07:15:12.825752+0000","last_undegraded":"2024-04-17T07:15:12.825752+0000","last_fullsized":"2024-04-17T07:15:12.825752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:06.843379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.926785+0000","last_change":"2024-04-17T07:15:10.754318+0000","last_active":"2024-04-17T07:15:13.926785+0000","last_peered":"2024-04-17T07:15:13.926785+0000","last_clean":"2024-04-17T07:15:13.926785+0000","last_became_active":"2024-04-17T07:15:10.752600+0000","last_became_peered":"2024-04-17T07:15:10.752600+0000","last_unstale":"2024-04-17T07:15:13.926785+0000","last_undegraded":"2024-04-17T07:15:13.926785+0000","last_fullsized":"2024-04-17T07:15:13.926785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:02.716404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789036+0000","last_change":"2024-04-17T07:15:11.789823+0000","last_active":"2024-04-17T07:15:12.789036+0000","last_peered":"2024-04-17T07:15:12.789036+0000","last_clean":"2024-04-17T07:15:12.789036+0000","last_became_active":"2024-04-17T07:15:11.789026+0000","last_became_peered":"2024-04-17T07:15:11.789026+0000","last_unstale":"2024-04-17T07:15:12.789036+0000","last_undegraded":"2024-04-17T07:15:12.789036+0000","last_fullsized":"2024-04-17T07:15:12.789036+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.786269+0000","last_change":"2024-04-17T07:15:10.754812+0000","last_active":"2024-04-17T07:15:12.786269+0000","last_peered":"2024-04-17T07:15:12.786269+0000","last_clean":"2024-04-17T07:15:12.786269+0000","last_became_active":"2024-04-17T07:15:10.753501+0000","last_became_peered":"2024-04-17T07:15:10.753501+0000","last_unstale":"2024-04-17T07:15:12.786269+0000","last_undegraded":"2024-04-17T07:15:12.786269+0000","last_fullsized":"2024-04-17T07:15:12.786269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.826101+0000","last_change":"2024-04-17T07:15:11.782597+0000","last_active":"2024-04-17T07:15:12.826101+0000","last_peered":"2024-04-17T07:15:12.826101+0000","last_clean":"2024-04-17T07:15:12.826101+0000","last_became_active":"2024-04-17T07:15:11.782301+0000","last_became_peered":"2024-04-17T07:15:11.782301+0000","last_unstale":"2024-04-17T07:15:12.826101+0000","last_undegraded":"2024-04-17T07:15:12.826101+0000","last_fullsized":"2024-04-17T07:15:12.826101+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:50.748276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.799478+0000","last_change":"2024-04-17T07:15:10.759311+0000","last_active":"2024-04-17T07:15:12.799478+0000","last_peered":"2024-04-17T07:15:12.799478+0000","last_clean":"2024-04-17T07:15:12.799478+0000","last_became_active":"2024-04-17T07:15:10.759020+0000","last_became_peered":"2024-04-17T07:15:10.759020+0000","last_unstale":"2024-04-17T07:15:12.799478+0000","last_undegraded":"2024-04-17T07:15:12.799478+0000","last_fullsized":"2024-04-17T07:15:12.799478+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:21.683926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789044+0000","last_change":"2024-04-17T07:15:11.776506+0000","last_active":"2024-04-17T07:15:12.789044+0000","last_peered":"2024-04-17T07:15:12.789044+0000","last_clean":"2024-04-17T07:15:12.789044+0000","last_became_active":"2024-04-17T07:15:11.776233+0000","last_became_peered":"2024-04-17T07:15:11.776233+0000","last_unstale":"2024-04-17T07:15:12.789044+0000","last_undegraded":"2024-04-17T07:15:12.789044+0000","last_fullsized":"2024-04-17T07:15:12.789044+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.944469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794701+0000","last_change":"2024-04-17T07:15:10.752962+0000","last_active":"2024-04-17T07:15:12.794701+0000","last_peered":"2024-04-17T07:15:12.794701+0000","last_clean":"2024-04-17T07:15:12.794701+0000","last_became_active":"2024-04-17T07:15:10.752256+0000","last_became_peered":"2024-04-17T07:15:10.752256+0000","last_unstale":"2024-04-17T07:15:12.794701+0000","last_undegraded":"2024-04-17T07:15:12.794701+0000","last_fullsized":"2024-04-17T07:15:12.794701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:53:59.036426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794379+0000","last_change":"2024-04-17T07:15:11.775823+0000","last_active":"2024-04-17T07:15:12.794379+0000","last_peered":"2024-04-17T07:15:12.794379+0000","last_clean":"2024-04-17T07:15:12.794379+0000","last_became_active":"2024-04-17T07:15:11.775511+0000","last_became_peered":"2024-04-17T07:15:11.775511+0000","last_unstale":"2024-04-17T07:15:12.794379+0000","last_undegraded":"2024-04-17T07:15:12.794379+0000","last_fullsized":"2024-04-17T07:15:12.794379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:38.306939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798907+0000","last_change":"2024-04-17T07:15:10.754618+0000","last_active":"2024-04-17T07:15:12.798907+0000","last_peered":"2024-04-17T07:15:12.798907+0000","last_clean":"2024-04-17T07:15:12.798907+0000","last_became_active":"2024-04-17T07:15:10.754339+0000","last_became_peered":"2024-04-17T07:15:10.754339+0000","last_unstale":"2024-04-17T07:15:12.798907+0000","last_undegraded":"2024-04-17T07:15:12.798907+0000","last_fullsized":"2024-04-17T07:15:12.798907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:46.911722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865371+0000","last_change":"2024-04-17T07:15:11.780155+0000","last_active":"2024-04-17T07:15:12.865371+0000","last_peered":"2024-04-17T07:15:12.865371+0000","last_clean":"2024-04-17T07:15:12.865371+0000","last_became_active":"2024-04-17T07:15:11.779020+0000","last_became_peered":"2024-04-17T07:15:11.779020+0000","last_unstale":"2024-04-17T07:15:12.865371+0000","last_undegraded":"2024-04-17T07:15:12.865371+0000","last_fullsized":"2024-04-17T07:15:12.865371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:22.689645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794308+0000","last_change":"2024-04-17T07:15:10.760063+0000","last_active":"2024-04-17T07:15:12.794308+0000","last_peered":"2024-04-17T07:15:12.794308+0000","last_clean":"2024-04-17T07:15:12.794308+0000","last_became_active":"2024-04-17T07:15:10.759760+0000","last_became_peered":"2024-04-17T07:15:10.759760+0000","last_unstale":"2024-04-17T07:15:12.794308+0000","last_undegraded":"2024-04-17T07:15:12.794308+0000","last_fullsized":"2024-04-17T07:15:12.794308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:27.362229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.794156+0000","last_change":"2024-04-17T07:15:11.781895+0000","last_active":"2024-04-17T07:15:12.794156+0000","last_peered":"2024-04-17T07:15:12.794156+0000","last_clean":"2024-04-17T07:15:12.794156+0000","last_became_active":"2024-04-17T07:15:11.781491+0000","last_became_peered":"2024-04-17T07:15:11.781491+0000","last_unstale":"2024-04-17T07:15:12.794156+0000","last_undegraded":"2024-04-17T07:15:12.794156+0000","last_fullsized":"2024-04-17T07:15:12.794156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:22.125784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.798532+0000","last_change":"2024-04-17T07:15:10.760218+0000","last_active":"2024-04-17T07:15:12.798532+0000","last_peered":"2024-04-17T07:15:12.798532+0000","last_clean":"2024-04-17T07:15:12.798532+0000","last_became_active":"2024-04-17T07:15:10.759943+0000","last_became_peered":"2024-04-17T07:15:10.759943+0000","last_unstale":"2024-04-17T07:15:12.798532+0000","last_undegraded":"2024-04-17T07:15:12.798532+0000","last_fullsized":"2024-04-17T07:15:12.798532+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:23:54.875499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780428+0000","last_change":"2024-04-17T07:15:11.786519+0000","last_active":"2024-04-17T07:15:12.780428+0000","last_peered":"2024-04-17T07:15:12.780428+0000","last_clean":"2024-04-17T07:15:12.780428+0000","last_became_active":"2024-04-17T07:15:11.783412+0000","last_became_peered":"2024-04-17T07:15:11.783412+0000","last_unstale":"2024-04-17T07:15:12.780428+0000","last_undegraded":"2024-04-17T07:15:12.780428+0000","last_fullsized":"2024-04-17T07:15:12.780428+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:55.533190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.927118+0000","last_change":"2024-04-17T07:15:10.758123+0000","last_active":"2024-04-17T07:15:13.927118+0000","last_peered":"2024-04-17T07:15:13.927118+0000","last_clean":"2024-04-17T07:15:13.927118+0000","last_became_active":"2024-04-17T07:15:10.757812+0000","last_became_peered":"2024-04-17T07:15:10.757812+0000","last_unstale":"2024-04-17T07:15:13.927118+0000","last_undegraded":"2024-04-17T07:15:13.927118+0000","last_fullsized":"2024-04-17T07:15:13.927118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:12.245915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:13.928069+0000","last_change":"2024-04-17T07:15:11.782972+0000","last_active":"2024-04-17T07:15:13.928069+0000","last_peered":"2024-04-17T07:15:13.928069+0000","last_clean":"2024-04-17T07:15:13.928069+0000","last_became_active":"2024-04-17T07:15:11.781277+0000","last_became_peered":"2024-04-17T07:15:11.781277+0000","last_unstale":"2024-04-17T07:15:13.928069+0000","last_undegraded":"2024-04-17T07:15:13.928069+0000","last_fullsized":"2024-04-17T07:15:13.928069+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:17.756837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.780702+0000","last_change":"2024-04-17T07:15:10.754805+0000","last_active":"2024-04-17T07:15:12.780702+0000","last_peered":"2024-04-17T07:15:12.780702+0000","last_clean":"2024-04-17T07:15:12.780702+0000","last_became_active":"2024-04-17T07:15:10.754522+0000","last_became_peered":"2024-04-17T07:15:10.754522+0000","last_unstale":"2024-04-17T07:15:12.780702+0000","last_undegraded":"2024-04-17T07:15:12.780702+0000","last_fullsized":"2024-04-17T07:15:12.780702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:15.971927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.789293+0000","last_change":"2024-04-17T07:15:11.791410+0000","last_active":"2024-04-17T07:15:12.789293+0000","last_peered":"2024-04-17T07:15:12.789293+0000","last_clean":"2024-04-17T07:15:12.789293+0000","last_became_active":"2024-04-17T07:15:11.790779+0000","last_became_peered":"2024-04-17T07:15:11.790779+0000","last_unstale":"2024-04-17T07:15:12.789293+0000","last_undegraded":"2024-04-17T07:15:12.789293+0000","last_fullsized":"2024-04-17T07:15:12.789293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:10.713413+0000","last_clean_scrub_stamp":"2024-04-17T07:15:10.713413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:32.957328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.865155+0000","last_change":"2024-04-17T07:15:10.760465+0000","last_active":"2024-04-17T07:15:12.865155+0000","last_peered":"2024-04-17T07:15:12.865155+0000","last_clean":"2024-04-17T07:15:12.865155+0000","last_became_active":"2024-04-17T07:15:10.760075+0000","last_became_peered":"2024-04-17T07:15:10.760075+0000","last_unstale":"2024-04-17T07:15:12.865155+0000","last_undegraded":"2024-04-17T07:15:12.865155+0000","last_fullsized":"2024-04-17T07:15:12.865155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:15:09.707694+0000","last_clean_scrub_stamp":"2024-04-17T07:15:09.707694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:47.131852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-17T07:15:12.788985+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:15:12.788985+0000","last_peered":"2024-04-17T07:15:12.788985+0000","last_clean":"2024-04-17T07:15:12.788985+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:15:12.788985+0000","last_undegraded":"2024-04-17T07:15:12.788985+0000","last_fullsized":"2024-04-17T07:15:12.788985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":991232,"data_stored":744966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":154612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":397312,"data_stored":154578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":156142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":987136,"data_stored":744876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:15:21.910 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:15:21.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:15:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.186+0000 7ff806800640 1 Processor -- start 2024-04-17T07:15:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.186+0000 7ff806800640 1 -- start start 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff806800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff80013f6d0 0x7ff800071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff806800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 0x7ff8000cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff806800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff80015cbe0 con 0x7ff8000cd5d0 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff806800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff80015cea0 con 0x7ff80013f6d0 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff806800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff80015d180 con 0x7ff8000b27c0 2024-04-17T07:15:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff805400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff80013f6d0 0x7ff800071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff805400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff80013f6d0 0x7ff800071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46650/0 (socket says 172.21.15.67:46650) 2024-04-17T07:15:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.188+0000 7ff805400640 1 -- 172.21.15.67:0/1308835924 learned_addr learned my addr 172.21.15.67:0/1308835924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805e00640 1 --2- 172.21.15.67:0/1308835924 >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 0x7ff8000cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805400640 1 -- 172.21.15.67:0/1308835924 >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 msgr2=0x7ff8000cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805400640 1 --2- 172.21.15.67:0/1308835924 >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 0x7ff8000cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805400640 1 -- 172.21.15.67:0/1308835924 >> v1:172.21.15.187:6789/0 conn(0x7ff8000cd5d0 legacy=0x7ff8000b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805400640 1 -- 172.21.15.67:0/1308835924 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80015d460 con 0x7ff80013f6d0 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.189+0000 7ff805e00640 1 --2- 172.21.15.67:0/1308835924 >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 0x7ff8000cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.190+0000 7ff805400640 1 --2- 172.21.15.67:0/1308835924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80013f6d0 0x7ff800071d00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff8000d1410 tx=0x7ff7f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950183d658e943e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.190+0000 7ff7ffe00640 1 -- 172.21.15.67:0/1308835924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4035020 con 0x7ff80013f6d0 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.190+0000 7ff7ffe00640 1 -- 172.21.15.67:0/1308835924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4031870 con 0x7ff80013f6d0 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.190+0000 7ff7ffe00640 1 -- 172.21.15.67:0/1308835924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4031c40 con 0x7ff80013f6d0 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 -- 172.21.15.67:0/1308835924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80013f6d0 msgr2=0x7ff800071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 --2- 172.21.15.67:0/1308835924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80013f6d0 0x7ff800071d00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff8000d1410 tx=0x7ff7f402b910 comp rx=0 tx=0).stop 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 -- 172.21.15.67:0/1308835924 shutdown_connections 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 --2- 172.21.15.67:0/1308835924 >> v2:172.21.15.187:3300/0 conn(0x7ff8000b27c0 0x7ff8000cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 --2- 172.21.15.67:0/1308835924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80013f6d0 0x7ff800071d00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 -- 172.21.15.67:0/1308835924 >> 172.21.15.67:0/1308835924 conn(0x7ff800067d10 msgr2=0x7ff8000d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 -- 172.21.15.67:0/1308835924 shutdown_connections 2024-04-17T07:15:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.191+0000 7ff806800640 1 -- 172.21.15.67:0/1308835924 wait complete. 2024-04-17T07:15:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.193+0000 7ff806800640 1 Processor -- start 2024-04-17T07:15:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.193+0000 7ff806800640 1 -- start start 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000b27c0 0x7ff800142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff80013f6d0 0x7ff8000b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff80015da20 con 0x7ff8000cd5d0 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8000d1120 con 0x7ff80013f6d0 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.194+0000 7ff806800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff800142020 con 0x7ff8000b27c0 2024-04-17T07:15:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46660/0 (socket says 172.21.15.67:46660) 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 -- 172.21.15.67:0/641772703 learned_addr learned my addr 172.21.15.67:0/641772703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff805e00640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff80013f6d0 0x7ff8000b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff805400640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000b27c0 0x7ff800142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 -- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000b27c0 msgr2=0x7ff800142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000b27c0 0x7ff800142ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 -- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff80013f6d0 msgr2=0x7ff8000b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff80013f6d0 0x7ff8000b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 -- 172.21.15.67:0/641772703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8001f29a0 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.195+0000 7ff804a00640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c430 tx=0x7ff7ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.196+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec011790 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.196+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7ec017070 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.196+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec011bb0 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.196+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8001f2c80 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.196+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8001426d0 con 0x7ff8000cd5d0 2024-04-17T07:15:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.197+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff800068a10 con 0x7ff8000cd5d0 2024-04-17T07:15:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.198+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7ec01c460 con 0x7ff8000cd5d0 2024-04-17T07:15:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.199+0000 7ff7fd600640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 0x7ff7e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.199+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7ff7ec08ba80 con 0x7ff8000cd5d0 2024-04-17T07:15:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.200+0000 7ff805400640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 0x7ff7e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.201+0000 7ff805400640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 0x7ff7e006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4036000 tx=0x7ff7f403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.203+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff7ec08de20 con 0x7ff8000cd5d0 2024-04-17T07:15:22.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.482+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff80006cd40 con 0x7ff8000cd5d0 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.487+0000 7ff7fd600640 1 -- 172.21.15.67:0/641772703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) ==== 190+0+147 (secure 0 0 0) 0x7ff7ec059ea0 con 0x7ff8000cd5d0 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:15:22.487 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:15:22.488 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 msgr2=0x7ff7e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 0x7ff7e006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4036000 tx=0x7ff7f403c040 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 msgr2=0x7ff800158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c430 tx=0x7ff7ec00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 shutdown_connections 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff80013f6d0 0x7ff8000b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cd5d0 0x7ff800158530 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7ff7e006c8f0 0x7ff7e006edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.489+0000 7ff806800640 1 --2- 172.21.15.67:0/641772703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000b27c0 0x7ff800142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.490+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 >> 172.21.15.67:0/641772703 conn(0x7ff800067d10 msgr2=0x7ff80006f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.490+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 shutdown_connections 2024-04-17T07:15:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.490+0000 7ff806800640 1 -- 172.21.15.67:0/641772703 wait complete. 2024-04-17T07:15:22.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:15:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.715+0000 7fbb9ca00640 1 Processor -- start 2024-04-17T07:15:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.715+0000 7fbb9ca00640 1 -- start start 2024-04-17T07:15:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.717+0000 7fbb9ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 0x7fbb9806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.717+0000 7fbb9ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb98141ed0 0x7fbb98142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.717+0000 7fbb9ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb980cd4a0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.717+0000 7fbb9ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9815ce30 con 0x7fbb98141ed0 2024-04-17T07:15:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.717+0000 7fbb9ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb981607f0 con 0x7fbb98160280 2024-04-17T07:15:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.718+0000 7fbb96a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 0x7fbb9806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.718+0000 7fbb97400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbb9806b8c0 0x7fbb980cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:37502/0 (socket says 172.21.15.67:37502) 2024-04-17T07:15:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.718+0000 7fbb97400640 1 -- 172.21.15.67:0/2193344044 learned_addr learned my addr 172.21.15.67:0/2193344044 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.718+0000 7fbb97e00640 1 --2- 172.21.15.67:0/2193344044 >> v2:172.21.15.187:3300/0 conn(0x7fbb98141ed0 0x7fbb98142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.719+0000 7fbb96a00640 1 -- 172.21.15.67:0/2193344044 >> v2:172.21.15.187:3300/0 conn(0x7fbb98141ed0 msgr2=0x7fbb98142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.719+0000 7fbb96a00640 1 --2- 172.21.15.67:0/2193344044 >> v2:172.21.15.187:3300/0 conn(0x7fbb98141ed0 0x7fbb98142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.719+0000 7fbb96a00640 1 -- 172.21.15.67:0/2193344044 >> v1:172.21.15.187:6789/0 conn(0x7fbb9806b8c0 legacy=0x7fbb980cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.719+0000 7fbb96a00640 1 -- 172.21.15.67:0/2193344044 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb98160ad0 con 0x7fbb98160280 2024-04-17T07:15:22.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.720+0000 7fbb96a00640 1 --2- 172.21.15.67:0/2193344044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 0x7fbb9806bca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097a0 tx=0x7fbb8402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1bec55b6057104 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.721+0000 7fbb96000640 1 -- 172.21.15.67:0/2193344044 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84035020 con 0x7fbb98160280 2024-04-17T07:15:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.721+0000 7fbb96000640 1 -- 172.21.15.67:0/2193344044 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb84031810 con 0x7fbb98160280 2024-04-17T07:15:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.721+0000 7fbb96000640 1 -- 172.21.15.67:0/2193344044 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84031be0 con 0x7fbb98160280 2024-04-17T07:15:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 -- 172.21.15.67:0/2193344044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 msgr2=0x7fbb9806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/2193344044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 0x7fbb9806bca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097a0 tx=0x7fbb8402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 -- 172.21.15.67:0/2193344044 shutdown_connections 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/2193344044 >> v2:172.21.15.187:3300/0 conn(0x7fbb98141ed0 0x7fbb98142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/2193344044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98160280 0x7fbb9806bca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 -- 172.21.15.67:0/2193344044 >> 172.21.15.67:0/2193344044 conn(0x7fbb98067d10 msgr2=0x7fbb980b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 -- 172.21.15.67:0/2193344044 shutdown_connections 2024-04-17T07:15:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.722+0000 7fbb9ca00640 1 -- 172.21.15.67:0/2193344044 wait complete. 2024-04-17T07:15:22.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.725+0000 7fbb9ca00640 1 Processor -- start 2024-04-17T07:15:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.725+0000 7fbb9ca00640 1 -- start start 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98141ed0 0x7fbb980b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb98160280 0x7fbb98145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb98161090 con 0x7fbb9806b8c0 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9806c1e0 con 0x7fbb98160280 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb9ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9813f9b0 con 0x7fbb98141ed0 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.726+0000 7fbb97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46670/0 (socket says 172.21.15.67:46670) 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 -- 172.21.15.67:0/3210754161 learned_addr learned my addr 172.21.15.67:0/3210754161 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97e00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb98160280 0x7fbb98145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb96a00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98141ed0 0x7fbb980b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 -- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98141ed0 msgr2=0x7fbb980b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98141ed0 0x7fbb980b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 -- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb98160280 msgr2=0x7fbb98145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb98160280 0x7fbb98145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 -- 172.21.15.67:0/3210754161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb980b11a0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.727+0000 7fbb97400640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800c400 tx=0x7fbb8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.728+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb88017070 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.728+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb88005ce0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.728+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb880072f0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.728+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9806ef00 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.728+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9806f4c0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.729+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb98068a10 con 0x7fbb9806b8c0 2024-04-17T07:15:22.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.730+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb8801a070 con 0x7fbb9806b8c0 2024-04-17T07:15:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.731+0000 7fbb8f400640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 0x7fbb7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.731+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fbb8808aed0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.732+0000 7fbb96a00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 0x7fbb7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:22.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.733+0000 7fbb96a00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 0x7fbb7806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb8403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.736+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb88059390 con 0x7fbb9806b8c0 2024-04-17T07:15:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.979+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb9815f3f0 con 0x7fbb9806b8c0 2024-04-17T07:15:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.980+0000 7fbb8f400640 1 -- 172.21.15.67:0/3210754161 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7fbb88058c90 con 0x7fbb9806b8c0 2024-04-17T07:15:22.980 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:15:22.981 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:15:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 msgr2=0x7fbb7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 0x7fbb7806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb8403c040 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 msgr2=0x7fbb9806e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800c400 tx=0x7fbb8800c8c0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 shutdown_connections 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb98160280 0x7fbb98145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbb7806c8f0 0x7fbb7806edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb98141ed0 0x7fbb980b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 --2- 172.21.15.67:0/3210754161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9806b8c0 0x7fbb9806e800 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.983+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 >> 172.21.15.67:0/3210754161 conn(0x7fbb98067d10 msgr2=0x7fbb980ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.984+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 shutdown_connections 2024-04-17T07:15:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:22.984+0000 7fbb9ca00640 1 -- 172.21.15.67:0/3210754161 wait complete. 2024-04-17T07:15:23.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:15:11.212642+0000', 'modified': '2024-04-17T07:15:14.302921+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5079}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5079': {'gid': 5079, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6835/504171556', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 504171556}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 504171556}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5079, 'qdb_cluster': [5079]} 2024-04-17T07:15:23.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:15:23.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T07:15:23.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.637+0000 7fa953200640 1 Processor -- start 2024-04-17T07:15:23.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.638+0000 7fa953200640 1 -- start start 2024-04-17T07:15:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.639+0000 7fa953200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 0x7fa9540d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.639+0000 7fa953200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa9540d3350 0x7fa9540b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.639+0000 7fa953200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9540d4140 con 0x7fa95415ee20 2024-04-17T07:15:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.639+0000 7fa953200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa9540b83b0 con 0x7fa9540d3350 2024-04-17T07:15:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.639+0000 7fa953200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9540d3b80 con 0x7fa9540d1ab0 2024-04-17T07:15:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.640+0000 7fa951e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa95415ee20 0x7fa9540d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:35192/0 (socket says 172.21.15.67:35192) 2024-04-17T07:15:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.640+0000 7fa951e00640 1 -- 172.21.15.67:0/2617040137 learned_addr learned my addr 172.21.15.67:0/2617040137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.640+0000 7fa952800640 1 --2- 172.21.15.67:0/2617040137 >> v2:172.21.15.187:3300/0 conn(0x7fa9540d3350 0x7fa9540b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.640+0000 7fa951400640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 0x7fa9540d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa950a00640 1 -- 172.21.15.67:0/2617040137 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596917216 0 0) 0x7fa9540d4140 con 0x7fa95415ee20 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa950a00640 1 -- 172.21.15.67:0/2617040137 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa938003520 con 0x7fa95415ee20 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa952800640 1 -- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 msgr2=0x7fa9540d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa952800640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 0x7fa9540d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa952800640 1 -- 172.21.15.67:0/2617040137 >> v1:172.21.15.187:6789/0 conn(0x7fa95415ee20 legacy=0x7fa9540d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa952800640 1 -- 172.21.15.67:0/2617040137 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9540d3e60 con 0x7fa9540d3350 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.641+0000 7fa951400640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 0x7fa9540d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:15:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.643+0000 7fa952800640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa9540b8840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009d90 tx=0x7fa94c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3687cb4f404b2eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.643+0000 7fa950a00640 1 -- 172.21.15.67:0/2617040137 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c039070 con 0x7fa9540d3350 2024-04-17T07:15:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.643+0000 7fa950a00640 1 -- 172.21.15.67:0/2617040137 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c02cc60 con 0x7fa9540d3350 2024-04-17T07:15:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.643+0000 7fa950a00640 1 -- 172.21.15.67:0/2617040137 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c034660 con 0x7fa9540d3350 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 -- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 msgr2=0x7fa9540b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa9540b8840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009d90 tx=0x7fa94c02beb0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 -- 172.21.15.67:0/2617040137 shutdown_connections 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa9540b8840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 --2- 172.21.15.67:0/2617040137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9540d1ab0 0x7fa9540d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 -- 172.21.15.67:0/2617040137 >> 172.21.15.67:0/2617040137 conn(0x7fa95406d3c0 msgr2=0x7fa9540b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 -- 172.21.15.67:0/2617040137 shutdown_connections 2024-04-17T07:15:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.644+0000 7fa953200640 1 -- 172.21.15.67:0/2617040137 wait complete. 2024-04-17T07:15:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.646+0000 7fa953200640 1 Processor -- start 2024-04-17T07:15:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.646+0000 7fa953200640 1 -- start start 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa954161be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95415ee20 0x7fa9540b9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9540d3800 con 0x7fa9540d1ab0 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa95415f200 con 0x7fa9540d3350 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa953200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9540b8d80 con 0x7fa95415ee20 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.647+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43554/0 (socket says 172.21.15.67:43554) 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 -- 172.21.15.67:0/2136201425 learned_addr learned my addr 172.21.15.67:0/2136201425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951400640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa954161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 -- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95415ee20 msgr2=0x7fa9540b9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95415ee20 0x7fa9540b9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 -- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 msgr2=0x7fa954161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa954161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 -- 172.21.15.67:0/2136201425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9540cb3f0 con 0x7fa9540d1ab0 2024-04-17T07:15:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.648+0000 7fa951e00640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa94400c410 tx=0x7fa94400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.649+0000 7fa94a000640 1 -- 172.21.15.67:0/2136201425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944017070 con 0x7fa9540d1ab0 2024-04-17T07:15:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.649+0000 7fa94a000640 1 -- 172.21.15.67:0/2136201425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa944005ce0 con 0x7fa9540d1ab0 2024-04-17T07:15:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.649+0000 7fa94a000640 1 -- 172.21.15.67:0/2136201425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944007310 con 0x7fa9540d1ab0 2024-04-17T07:15:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.649+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9540cab50 con 0x7fa9540d1ab0 2024-04-17T07:15:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.649+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9540cb110 con 0x7fa9540d1ab0 2024-04-17T07:15:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.651+0000 7fa94a000640 1 -- 172.21.15.67:0/2136201425 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa94401a070 con 0x7fa9540d1ab0 2024-04-17T07:15:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.652+0000 7fa94a000640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 0x7fa93006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.652+0000 7fa94a000640 1 -- 172.21.15.67:0/2136201425 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fa94408af70 con 0x7fa9540d1ab0 2024-04-17T07:15:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.653+0000 7fa951400640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 0x7fa93006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.653+0000 7fa95ba1e740 1 Processor -- start 2024-04-17T07:15:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.653+0000 7fa95ba1e740 1 -- start start 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951400640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 0x7fa93006edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009560 tx=0x7fa93c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fca0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fc9ab40 0x55555fcca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55555fc06980 con 0x55555fce8620 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55555fbad220 con 0x55555fc9ab40 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55555fc02940 con 0x55555fc9e6a0 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43568/0 (socket says 172.21.15.67:43568) 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 -- 172.21.15.67:0/4037199382 learned_addr learned my addr 172.21.15.67:0/4037199382 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 -- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 msgr2=0x55555fca0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fca0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 -- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fc9ab40 msgr2=0x55555fcca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fc9ab40 0x55555fcca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.654+0000 7fa951e00640 1 -- 172.21.15.67:0/4037199382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55555f9192d0 con 0x55555fce8620 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.655+0000 7fa951e00640 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa944005fd0 tx=0x7fa944058ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2cf72a2b7a40542 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.655+0000 7fa942600640 1 -- 172.21.15.67:0/4037199382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944056e00 con 0x55555fce8620 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.655+0000 7fa942600640 1 -- 172.21.15.67:0/4037199382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa944057060 con 0x55555fce8620 2024-04-17T07:15:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.655+0000 7fa942600640 1 -- 172.21.15.67:0/4037199382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440074b0 con 0x55555fce8620 2024-04-17T07:15:23.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 -- 172.21.15.67:0/4037199382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 msgr2=0x55555fc9fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa944005fd0 tx=0x7fa944058ea0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 -- 172.21.15.67:0/4037199382 shutdown_connections 2024-04-17T07:15:23.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fc9ab40 0x55555fcca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fca0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/4037199382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fc9fc50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 -- 172.21.15.67:0/4037199382 >> 172.21.15.67:0/4037199382 conn(0x55555fc9cdb0 msgr2=0x55555fce5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 -- 172.21.15.67:0/4037199382 shutdown_connections 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.656+0000 7fa95ba1e740 1 -- 172.21.15.67:0/4037199382 wait complete. 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.657+0000 7fa95ba1e740 1 -- start start 2024-04-17T07:15:23.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.657+0000 7fa95ba1e740 1 Processor -- start 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fce8620 0x55555fced8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fcc3070 0x55555fccb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55555fc9ebb0 con 0x55555fc9e6a0 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55555fbc0590 con 0x55555fce8620 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55555fc9ee90 con 0x55555fcc3070 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa95ba1e740 10 client.0 _ll_register_callbacks cb 0x55555f918ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43572/0 (socket says 172.21.15.67:43572) 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 -- 172.21.15.67:0/3724671163 learned_addr learned my addr 172.21.15.67:0/3724671163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 -- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fcc3070 msgr2=0x55555fccb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fcc3070 0x55555fccb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 -- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fce8620 msgr2=0x55555fced8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fce8620 0x55555fced8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.658+0000 7fa951e00640 1 -- 172.21.15.67:0/3724671163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55555f9e36b0 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa951e00640 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa944058cf0 tx=0x7fa944001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cf9a890a182538 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944075020 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fa94405f340 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa944057060 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa94405f580 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa95ba1e740 10 client.5265 fetch_fsmap learned FSMap version 17 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55555fced3a0 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa944057060 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 3495+0+0 (secure 0 0 0) 0x7fa9440581b0 con 0x55555fc9e6a0 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.659+0000 7fa95ba1e740 10 client.5265 fetch_fsmap finished waiting for FSMap version 17 2024-04-17T07:15:23.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa95ba1e740 10 client.5265 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa95ba1e740 20 client.5265 populate_metadata read hostname 'smithi067' 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fcea080 0x55555fcea460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa95ba1e740 4 client.5265 mds_command: new command op to 5079 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 --> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55555fc143a0 con 0x55555fcea080 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa951400640 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fcea080 0x55555fcea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa951400640 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fcea080 0x55555fcea460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4320b453a3bc2617 server_cookie=463285ffa52dc8f5 in_seq=0 out_seq=0 2024-04-17T07:15:23.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.660+0000 7fa936a00640 10 client.5265 ms_handle_connect on v2:172.21.15.67:6833/504171556 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.661+0000 7fa936a00640 1 -- 172.21.15.67:0/3724671163 <== mds.0 v2:172.21.15.67:6833/504171556 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55555fc143a0 con 0x55555fcea080 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.661+0000 7fa936a00640 10 client.5265 handle_command_reply: tid=0 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.661+0000 7fa95ba1e740 1 client.5265 shutdown 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 msgr2=0x55555fcebfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa944058cf0 tx=0x7fa944001150 comp rx=0 tx=0).stop 2024-04-17T07:15:23.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 shutdown_connections 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fcea080 0x55555fcea460 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fcc3070 0x55555fccb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fce8620 0x55555fced8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3724671163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fcebfd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa936a00640 0 client.5265 ms_handle_reset on v2:172.21.15.67:6833/504171556 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 >> 172.21.15.67:0/3724671163 conn(0x55555fc9ab40 msgr2=0x55555fce5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 shutdown_connections 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3724671163 wait complete. 2024-04-17T07:15:23.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.662+0000 7fa95ba1e740 20 client.5265 trim_cache size 0 max 16384 2024-04-17T07:15:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.682+0000 7fa95ba1e740 1 Processor -- start 2024-04-17T07:15:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.682+0000 7fa95ba1e740 1 -- start start 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fce70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fce8620 0x55555fcea460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541ffce0 con 0x55555fc9e6a0 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9541600c0 con 0x55555fcc3070 2024-04-17T07:15:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9541606e0 con 0x55555fce8620 2024-04-17T07:15:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa951e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43582/0 (socket says 172.21.15.67:43582) 2024-04-17T07:15:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa951e00640 1 -- 172.21.15.67:0/213629471 learned_addr learned my addr 172.21.15.67:0/213629471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa952800640 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fce8620 0x55555fcea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.683+0000 7fa951400640 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fce70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 -- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fce8620 msgr2=0x55555fcea460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fce8620 0x55555fcea460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 -- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 msgr2=0x55555fce70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fce70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 -- 172.21.15.67:0/213629471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55555fcc9d70 con 0x55555fc9e6a0 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa951e00640 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa9440551c0 tx=0x7fa944088000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c16cf1c0a93f307 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa941c00640 1 -- 172.21.15.67:0/213629471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944075070 con 0x55555fc9e6a0 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa941c00640 1 -- 172.21.15.67:0/213629471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94405f340 con 0x55555fc9e6a0 2024-04-17T07:15:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.684+0000 7fa941c00640 1 -- 172.21.15.67:0/213629471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440588a0 con 0x55555fc9e6a0 2024-04-17T07:15:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 -- 172.21.15.67:0/213629471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 msgr2=0x55555fce6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa9440551c0 tx=0x7fa944088000 comp rx=0 tx=0).stop 2024-04-17T07:15:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 -- 172.21.15.67:0/213629471 shutdown_connections 2024-04-17T07:15:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fce8620 0x55555fcea460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fce70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/213629471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fc9e6a0 0x55555fce6ba0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 -- 172.21.15.67:0/213629471 >> 172.21.15.67:0/213629471 conn(0x55555fc9ab40 msgr2=0x55555fce63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 -- 172.21.15.67:0/213629471 shutdown_connections 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.685+0000 7fa95ba1e740 1 -- 172.21.15.67:0/213629471 wait complete. 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.686+0000 7fa95ba1e740 1 -- start start 2024-04-17T07:15:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.686+0000 7fa95ba1e740 1 Processor -- start 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fcce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fcc7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55555fc9ebb0 con 0x55555fce8620 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55555fc9ee90 con 0x55555fcc3070 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55555fbc0590 con 0x55555fc9e6a0 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa95ba1e740 10 client.0 _ll_register_callbacks cb 0x7fa9440594f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.687+0000 7fa952800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43598/0 (socket says 172.21.15.67:43598) 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 -- 172.21.15.67:0/3512750236 learned_addr learned my addr 172.21.15.67:0/3512750236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa951e00640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fcce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 -- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 msgr2=0x55555fcce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa951400640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fcc7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fcce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 -- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 msgr2=0x55555fcc7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fcc7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 -- 172.21.15.67:0/3512750236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55555fcc7ed0 con 0x55555fce8620 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa951400640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fcc7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa952800640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009eb0 tx=0x7fa94c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0d03e8fac6cd88 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.688+0000 7fa936a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c043030 con 0x55555fce8620 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fa94c0347e0 con 0x55555fce8620 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa94405f340 con 0x55555fce8620 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa94c039040 con 0x55555fce8620 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 10 client.5271 fetch_fsmap learned FSMap version 17 2024-04-17T07:15:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55555fcea460 con 0x55555fce8620 2024-04-17T07:15:23.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94405f340 con 0x55555fce8620 2024-04-17T07:15:23.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 3495+0+0 (secure 0 0 0) 0x7fa94c0349e0 con 0x55555fce8620 2024-04-17T07:15:23.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 10 client.5271 fetch_fsmap finished waiting for FSMap version 17 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 10 client.5271 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 20 client.5271 populate_metadata read hostname 'smithi067' 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fce9ab0 0x55555fcc5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 4 client.5271 mds_command: new command op to 5079 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.689+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 --> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] -- command(tid 0: {"prefix": "status"}) -- 0x55555fc0e210 con 0x55555fce9ab0 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa951e00640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fce9ab0 0x55555fcc5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa936a00640 10 client.5271 ms_handle_connect on v2:172.21.15.67:6833/504171556 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa951e00640 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fce9ab0 0x55555fcc5c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a335ca23b3ff8f1 server_cookie=c2f9b8c725c09702 in_seq=0 out_seq=0 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa936a00640 1 -- 172.21.15.67:0/3512750236 <== mds.0 v2:172.21.15.67:6833/504171556 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55555fc0e210 con 0x55555fce9ab0 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa936a00640 10 client.5271 handle_command_reply: tid=0 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.690+0000 7fa95ba1e740 1 client.5271 shutdown 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 msgr2=0x55555fd62d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009eb0 tx=0x7fa94c034670 comp rx=0 tx=0).stop 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 shutdown_connections 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:6833/504171556,v1:172.21.15.67:6835/504171556] conn(0x55555fce9ab0 0x55555fcc5c30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55555fce8620 0x55555fd62d40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55555fcc3070 0x55555fcc7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 --2- 172.21.15.67:0/3512750236 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55555fc9e6a0 0x55555fcce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa936a00640 0 client.5271 ms_handle_reset on v2:172.21.15.67:6833/504171556 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 >> 172.21.15.67:0/3512750236 conn(0x55555fc9ab40 msgr2=0x55555fd09c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 shutdown_connections 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 1 -- 172.21.15.67:0/3512750236 wait complete. 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.691+0000 7fa95ba1e740 20 client.5271 trim_cache size 0 max 16384 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:15:23.694 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5079, 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 11.921445884000001, 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 35, 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 35, 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 20.754398596000001 2024-04-17T07:15:23.695 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 msgr2=0x7fa93006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 0x7fa93006edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009560 tx=0x7fa93c008040 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 msgr2=0x7fa9540ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa94400c410 tx=0x7fa94400c8d0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 shutdown_connections 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fa93006c8f0 0x7fa93006edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95415ee20 0x7fa9540b9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9540d3350 0x7fa954161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 --2- 172.21.15.67:0/2136201425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9540d1ab0 0x7fa9540ca570 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 >> 172.21.15.67:0/2136201425 conn(0x7fa95406d3c0 msgr2=0x7fa9540b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 shutdown_connections 2024-04-17T07:15:23.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.692+0000 7fa953200640 1 -- 172.21.15.67:0/2136201425 wait complete. 2024-04-17T07:15:23.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:15:23.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:15:23.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:15:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.911+0000 7f2d8c000640 1 Processor -- start 2024-04-17T07:15:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.911+0000 7f2d8c000640 1 -- start start 2024-04-17T07:15:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.912+0000 7f2d8c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d840cf640 0x7f2d840cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.912+0000 7f2d8c000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d8415ed10 0x7f2d840d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.912+0000 7f2d8c000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d840cff60 con 0x7f2d840d2ab0 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.912+0000 7f2d8c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d840d3c30 con 0x7f2d840cf640 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.912+0000 7f2d8c000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2d840d3020 con 0x7f2d8415ed10 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d840cf640 0x7f2d840cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d840cf640 0x7f2d840cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43612/0 (socket says 172.21.15.67:43612) 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 -- 172.21.15.67:0/2821139910 learned_addr learned my addr 172.21.15.67:0/2821139910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d88c00640 1 --2- 172.21.15.67:0/2821139910 >> v2:172.21.15.187:3300/0 conn(0x7f2d8415ed10 0x7f2d840d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 -- 172.21.15.67:0/2821139910 >> v2:172.21.15.187:3300/0 conn(0x7f2d8415ed10 msgr2=0x7f2d840d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 --2- 172.21.15.67:0/2821139910 >> v2:172.21.15.187:3300/0 conn(0x7f2d8415ed10 0x7f2d840d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 -- 172.21.15.67:0/2821139910 >> v1:172.21.15.187:6789/0 conn(0x7f2d840d2ab0 legacy=0x7f2d840b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.913+0000 7f2d89600640 1 -- 172.21.15.67:0/2821139910 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d840d3300 con 0x7f2d840cf640 2024-04-17T07:15:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.914+0000 7f2d89600640 1 --2- 172.21.15.67:0/2821139910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d840cf640 0x7f2d840cfa20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2d8408ddd0 tx=0x7f2d7802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6190e68a59dcee36 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d83e00640 1 -- 172.21.15.67:0/2821139910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7802f850 con 0x7f2d840cf640 2024-04-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d83e00640 1 -- 172.21.15.67:0/2821139910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7802f9f0 con 0x7f2d840cf640 2024-04-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d83e00640 1 -- 172.21.15.67:0/2821139910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7802fdc0 con 0x7f2d840cf640 2024-04-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 -- 172.21.15.67:0/2821139910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d840cf640 msgr2=0x7f2d840cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2821139910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d840cf640 0x7f2d840cfa20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2d8408ddd0 tx=0x7f2d7802ba30 comp rx=0 tx=0).stop 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 -- 172.21.15.67:0/2821139910 shutdown_connections 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2821139910 >> v2:172.21.15.187:3300/0 conn(0x7f2d8415ed10 0x7f2d840d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2821139910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d840cf640 0x7f2d840cfa20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 -- 172.21.15.67:0/2821139910 >> 172.21.15.67:0/2821139910 conn(0x7f2d8406cbd0 msgr2=0x7f2d8406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 -- 172.21.15.67:0/2821139910 shutdown_connections 2024-04-17T07:15:23.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.915+0000 7f2d8c000640 1 -- 172.21.15.67:0/2821139910 wait complete. 2024-04-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.917+0000 7f2d8c000640 1 Processor -- start 2024-04-17T07:15:23.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.917+0000 7f2d8c000640 1 -- start start 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d840cf640 0x7f2d840ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 0x7f2d84155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840d25b0 con 0x7f2d8415ed10 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d840d38c0 con 0x7f2d840d2ab0 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.918+0000 7f2d8c000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d840d4600 con 0x7f2d840cf640 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43616/0 (socket says 172.21.15.67:43616) 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 -- 172.21.15.67:0/2976414657 learned_addr learned my addr 172.21.15.67:0/2976414657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d89600640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d840cf640 0x7f2d840ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d88c00640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 0x7f2d84155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 -- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d840cf640 msgr2=0x7f2d840ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d840cf640 0x7f2d840ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 -- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 msgr2=0x7f2d84155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 0x7f2d84155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 -- 172.21.15.67:0/2976414657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d840ccca0 con 0x7f2d8415ed10 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d88c00640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 0x7f2d84155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.919+0000 7f2d8a000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c7a0 tx=0x7f2d7c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.920+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c003b70 con 0x7f2d8415ed10 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.920+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7c003d30 con 0x7f2d8415ed10 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.920+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d840ccf80 con 0x7f2d8415ed10 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.920+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00aa90 con 0x7f2d8415ed10 2024-04-17T07:15:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.920+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d840cd540 con 0x7f2d8415ed10 2024-04-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.921+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d84068b90 con 0x7f2d8415ed10 2024-04-17T07:15:23.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.922+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d7c00ac30 con 0x7f2d8415ed10 2024-04-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.924+0000 7f2d81600640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 0x7f2d6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:23.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.924+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f2d7c08b9e0 con 0x7f2d8415ed10 2024-04-17T07:15:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.924+0000 7f2d89600640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 0x7f2d6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.925+0000 7f2d89600640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 0x7f2d6406edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d78036000 tx=0x7f2d78008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:23.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:23.928+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2d7c059e10 con 0x7f2d8415ed10 2024-04-17T07:15:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.173+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2d840b40b0 con 0x7f2d8415ed10 2024-04-17T07:15:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.174+0000 7f2d81600640 1 -- 172.21.15.67:0/2976414657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f2d7c059710 con 0x7f2d8415ed10 2024-04-17T07:15:24.174 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 msgr2=0x7f2d6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 0x7f2d6406edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d78036000 tx=0x7f2d78008030 comp rx=0 tx=0).stop 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 msgr2=0x7f2d840b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c7a0 tx=0x7f2d7c00cc60 comp rx=0 tx=0).stop 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 shutdown_connections 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d8415ed10 0x7f2d840b91f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d840d2ab0 0x7f2d84155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f2d6406c8f0 0x7f2d6406edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 --2- 172.21.15.67:0/2976414657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d840cf640 0x7f2d840ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 >> 172.21.15.67:0/2976414657 conn(0x7f2d8406cbd0 msgr2=0x7f2d840c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 shutdown_connections 2024-04-17T07:15:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24.177+0000 7f2d8c000640 1 -- 172.21.15.67:0/2976414657 wait complete. 2024-04-17T07:15:24.197 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:15:24.197 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:15:24.198 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:15:24.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.199 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:15:24.255 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:15:24.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42520sec preferred_lft 42520sec 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.257 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:15:24.258 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:15:24.258 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:15:24.258 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:15:24.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.259 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:15:24.354 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:15:24.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.359 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:15:24.412 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:15:24.412 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:15:24.412 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:15:24.413 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:15:24.413 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:15:24.414 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:15:24.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:15:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:15:24.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.678 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:15:24.727 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:15:24.728 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:15:24.729 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:15:24.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:15:24.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:15:24.831 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:15:24.888 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:15:24.888 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:15:24.888 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:15:24.942 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:15:24.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:24.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:15:25.090 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:15:25.091 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:15:25.124 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:15:25.124 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:15:25.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:15:25.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:15:25.125 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:15:25.128 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:15:25.128 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:15:25.128 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:15:25.128 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:15:25.129 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:15:25.134 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:15:25.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:25.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:15:25.222 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:15:25.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:15:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:15:25.361 INFO:teuthology.orchestra.run.smithi067.stdout:client4915 2024-04-17T07:15:25.361 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:15:25.361 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:15:25.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:15:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.633+0000 7fbc75600640 1 Processor -- start 2024-04-17T07:15:25.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.634+0000 7fbc75600640 1 -- start start 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.635+0000 7fbc75600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc700d0b30 0x7fbc700b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.635+0000 7fbc75600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 0x7fbc700b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.635+0000 7fbc75600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc70071f60 con 0x7fbc7015b860 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.635+0000 7fbc75600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbc70071d10 con 0x7fbc700d0b30 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.635+0000 7fbc75600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbc70072240 con 0x7fbc700ce4d0 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc700d0b30 0x7fbc700b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc700d0b30 0x7fbc700b3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43618/0 (socket says 172.21.15.67:43618) 2024-04-17T07:15:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 -- 172.21.15.67:0/2320406655 learned_addr learned my addr 172.21.15.67:0/2320406655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6f400640 1 --2- 172.21.15.67:0/2320406655 >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 0x7fbc700b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 -- 172.21.15.67:0/2320406655 >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 msgr2=0x7fbc700b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 --2- 172.21.15.67:0/2320406655 >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 0x7fbc700b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 -- 172.21.15.67:0/2320406655 >> v1:172.21.15.187:6789/0 conn(0x7fbc7015b860 legacy=0x7fbc700d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6e000640 1 -- 172.21.15.67:0/2320406655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc70072520 con 0x7fbc700d0b30 2024-04-17T07:15:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.636+0000 7fbc6f400640 1 --2- 172.21.15.67:0/2320406655 >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 0x7fbc700b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:25.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.637+0000 7fbc6e000640 1 --2- 172.21.15.67:0/2320406655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700d0b30 0x7fbc700b3f70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbc7006c670 tx=0x7fbc5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1493101333f76ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:25.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.638+0000 7fbc6d600640 1 -- 172.21.15.67:0/2320406655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c035020 con 0x7fbc700d0b30 2024-04-17T07:15:25.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.638+0000 7fbc6d600640 1 -- 172.21.15.67:0/2320406655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc5c031870 con 0x7fbc700d0b30 2024-04-17T07:15:25.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.638+0000 7fbc6d600640 1 -- 172.21.15.67:0/2320406655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c031c40 con 0x7fbc700d0b30 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.638+0000 7fbc75600640 1 -- 172.21.15.67:0/2320406655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700d0b30 msgr2=0x7fbc700b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.638+0000 7fbc75600640 1 --2- 172.21.15.67:0/2320406655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700d0b30 0x7fbc700b3f70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbc7006c670 tx=0x7fbc5c02b910 comp rx=0 tx=0).stop 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 -- 172.21.15.67:0/2320406655 shutdown_connections 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 --2- 172.21.15.67:0/2320406655 >> v2:172.21.15.187:3300/0 conn(0x7fbc700ce4d0 0x7fbc700b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 --2- 172.21.15.67:0/2320406655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700d0b30 0x7fbc700b3f70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 -- 172.21.15.67:0/2320406655 >> 172.21.15.67:0/2320406655 conn(0x7fbc70067ee0 msgr2=0x7fbc7006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 -- 172.21.15.67:0/2320406655 shutdown_connections 2024-04-17T07:15:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.639+0000 7fbc75600640 1 -- 172.21.15.67:0/2320406655 wait complete. 2024-04-17T07:15:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.640+0000 7fbc75600640 1 Processor -- start 2024-04-17T07:15:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.641+0000 7fbc75600640 1 -- start start 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 0x7fbc70137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc7015b860 0x7fbc70137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7015bc40 con 0x7fbc700ce4d0 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70072ae0 con 0x7fbc700d0b30 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc75600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7006f380 con 0x7fbc7015b860 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43634/0 (socket says 172.21.15.67:43634) 2024-04-17T07:15:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 -- 172.21.15.67:0/3476177069 learned_addr learned my addr 172.21.15.67:0/3476177069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6e000640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 0x7fbc70137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 -- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc7015b860 msgr2=0x7fbc70137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc7015b860 0x7fbc70137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 -- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 msgr2=0x7fbc70137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 0x7fbc70137a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6ea00640 1 -- 172.21.15.67:0/3476177069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7014cf30 con 0x7fbc700ce4d0 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.642+0000 7fbc6e000640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 0x7fbc70137a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc6ea00640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b410 tx=0x7fbc6000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60015020 con 0x7fbc700ce4d0 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc60011830 con 0x7fbc700ce4d0 2024-04-17T07:15:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc60011c20 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7014d210 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.643+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7014d7d0 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.644+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc70068ca0 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.645+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc60018020 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.646+0000 7fbc66a00640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 0x7fbc5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.646+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fbc6008bc30 con 0x7fbc700ce4d0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.647+0000 7fbc6e000640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 0x7fbc5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.648+0000 7fbc6e000640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 0x7fbc5006edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c036000 tx=0x7fbc5c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:25.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.651+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbc6001f020 con 0x7fbc700ce4d0 2024-04-17T07:15:25.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.881+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbc700cf800 con 0x7fbc5006c8f0 2024-04-17T07:15:25.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.885+0000 7fbc66a00640 1 -- 172.21.15.67:0/3476177069 <== mgr.4108 v2:172.21.15.187:6832/3291261531 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fbc700cf800 con 0x7fbc5006c8f0 2024-04-17T07:15:25.885 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:15:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:15:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:15:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:15:25.886 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:15:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 msgr2=0x7fbc5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 0x7fbc5006edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c036000 tx=0x7fbc5c03c040 comp rx=0 tx=0).stop 2024-04-17T07:15:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 msgr2=0x7fbc7014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b410 tx=0x7fbc6000b8d0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 shutdown_connections 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc5006c8f0 0x7fbc5006edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc7015b860 0x7fbc70137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc700d0b30 0x7fbc70137a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 --2- 172.21.15.67:0/3476177069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc700ce4d0 0x7fbc7014e800 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 >> 172.21.15.67:0/3476177069 conn(0x7fbc70067ee0 msgr2=0x7fbc700b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 shutdown_connections 2024-04-17T07:15:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:25.888+0000 7fbc75600640 1 -- 172.21.15.67:0/3476177069 wait complete. 2024-04-17T07:15:25.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:15:26.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.136+0000 7fe0c4a00640 1 Processor -- start 2024-04-17T07:15:26.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.136+0000 7fe0c4a00640 1 -- start start 2024-04-17T07:15:26.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.137+0000 7fe0c4a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0c0162da0 0x7fe0c00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.137+0000 7fe0c4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c0166a60 con 0x7fe0c00b41a0 2024-04-17T07:15:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.137+0000 7fe0c4a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c0166d40 con 0x7fe0c00b4580 2024-04-17T07:15:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.137+0000 7fe0c4a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c0166810 con 0x7fe0c0162da0 2024-04-17T07:15:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0c00b41a0 0x7fe0c0162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37938/0 (socket says 172.21.15.67:37938) 2024-04-17T07:15:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bf400640 1 -- 172.21.15.67:0/1763605833 learned_addr learned my addr 172.21.15.67:0/1763605833 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002895341 0 0) 0x7fe0c0166a60 con 0x7fe0c00b41a0 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/1763605833 >> v2:172.21.15.187:3300/0 conn(0x7fe0c0162da0 0x7fe0c00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a4003520 con 0x7fe0c00b41a0 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.138+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3992855507 0 0) 0x7fe0a4003520 con 0x7fe0c00b41a0 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 >> v2:172.21.15.187:3300/0 conn(0x7fe0c0162da0 msgr2=0x7fe0c00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 --2- 172.21.15.67:0/1763605833 >> v2:172.21.15.187:3300/0 conn(0x7fe0c0162da0 0x7fe0c00b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 >> v1:172.21.15.187:6789/0 conn(0x7fe0c00b4580 legacy=0x7fe0c006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c0167020 con 0x7fe0c00b41a0 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe0b0001670 con 0x7fe0c00b41a0 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/1763605833 >> v2:172.21.15.187:3300/0 conn(0x7fe0c0162da0 0x7fe0c00b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 >> v1:172.21.15.67:6789/0 conn(0x7fe0c00b41a0 legacy=0x7fe0c0162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a4003a00 0x7fe0a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a4005210 0x7fe0a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0166810 con 0x7fe0a4004800 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a400c140 con 0x7fe0a4005210 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a400c340 con 0x7fe0a4003a00 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bf400640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a4003a00 0x7fe0a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.139+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a4005210 0x7fe0a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0b6a00640 1 -- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a4003a00 msgr2=0x7fe0a4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a4003a00 0x7fe0a4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0b6a00640 1 -- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a4005210 msgr2=0x7fe0a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a4005210 0x7fe0a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0b6a00640 1 -- 172.21.15.67:0/1763605833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a4003520 con 0x7fe0a4004800 2024-04-17T07:15:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.140+0000 7fe0bfe00640 1 -- 172.21.15.67:0/1763605833 reap_dead start 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.141+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0a4004c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8004480 tx=0x7fe0a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b408e6be991b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.141+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035020 con 0x7fe0a4004800 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.141+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a8031670 con 0x7fe0a4004800 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.141+0000 7fe0bea00640 1 -- 172.21.15.67:0/1763605833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8004080 con 0x7fe0a4004800 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 -- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 msgr2=0x7fe0a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0a4004c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8004480 tx=0x7fe0a4005830 comp rx=0 tx=0).stop 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 -- 172.21.15.67:0/1763605833 shutdown_connections 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/1763605833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0a4004c80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 -- 172.21.15.67:0/1763605833 >> 172.21.15.67:0/1763605833 conn(0x7fe0c00b2190 msgr2=0x7fe0c00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 -- 172.21.15.67:0/1763605833 shutdown_connections 2024-04-17T07:15:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.142+0000 7fe0c4a00640 1 -- 172.21.15.67:0/1763605833 wait complete. 2024-04-17T07:15:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.144+0000 7fe0c4a00640 1 Processor -- start 2024-04-17T07:15:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.144+0000 7fe0c4a00640 1 -- start start 2024-04-17T07:15:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c0163270 0x7fe0c006e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c01636d0 0x7fe0c006e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c006f980 con 0x7fe0a4004800 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c006f800 con 0x7fe0c01636d0 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c006fb00 con 0x7fe0c0163270 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43650/0 (socket says 172.21.15.67:43650) 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.145+0000 7fe0bf400640 1 -- 172.21.15.67:0/2866729895 learned_addr learned my addr 172.21.15.67:0/2866729895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 -- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c0163270 msgr2=0x7fe0c006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c0163270 0x7fe0c006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 -- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c01636d0 msgr2=0x7fe0c006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c01636d0 0x7fe0c006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 -- 172.21.15.67:0/2866729895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c0163c30 con 0x7fe0a4004800 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0bf400640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0001050 tx=0x7fe0b000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000c690 con 0x7fe0a4004800 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b0018070 con 0x7fe0a4004800 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000ca90 con 0x7fe0a4004800 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c0163f10 con 0x7fe0a4004800 2024-04-17T07:15:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.146+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c015a2d0 con 0x7fe0a4004800 2024-04-17T07:15:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.148+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c0068a10 con 0x7fe0a4004800 2024-04-17T07:15:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.149+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0b001d460 con 0x7fe0a4004800 2024-04-17T07:15:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.150+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 0x7fe09c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.150+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fe0b008bbc0 con 0x7fe0a4004800 2024-04-17T07:15:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.151+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 0x7fe09c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.152+0000 7fe0b6a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 0x7fe09c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8036000 tx=0x7fe0a8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:26.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.155+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe0b005a080 con 0x7fe0a4004800 2024-04-17T07:15:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.419+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fe0c006c7b0 con 0x7fe0a4004800 2024-04-17T07:15:26.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.876+0000 7fe0b7e00640 1 -- 172.21.15.67:0/2866729895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) ==== 92+0+0 (secure 0 0 0) 0x7fe0b0059980 con 0x7fe0a4004800 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.882+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 msgr2=0x7fe09c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.882+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 0x7fe09c06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8036000 tx=0x7fe0a8030000 comp rx=0 tx=0).stop 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.882+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 msgr2=0x7fe0c014c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.882+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0001050 tx=0x7fe0b000eb90 comp rx=0 tx=0).stop 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 shutdown_connections 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c01636d0 0x7fe0c006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fe09c06c8f0 0x7fe09c06edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c0163270 0x7fe0c006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 --2- 172.21.15.67:0/2866729895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0a4004800 0x7fe0c014c6e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:26.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 >> 172.21.15.67:0/2866729895 conn(0x7fe0c00b2190 msgr2=0x7fe0c00b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:26.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 shutdown_connections 2024-04-17T07:15:26.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:26.884+0000 7fe0c4a00640 1 -- 172.21.15.67:0/2866729895 wait complete. 2024-04-17T07:15:26.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:15:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.111+0000 7fbcb1400640 1 Processor -- start 2024-04-17T07:15:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.112+0000 7fbcb1400640 1 -- start start 2024-04-17T07:15:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.113+0000 7fbcb1400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbcac0ce4d0 0x7fbcac0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.113+0000 7fbcb1400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac071f60 con 0x7fbcac15b860 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.113+0000 7fbcb1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac072240 con 0x7fbcac0d0b30 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.113+0000 7fbcb1400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbcac071d10 con 0x7fbcac0ce4d0 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbcaa800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcac15b860 0x7fbcac0d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37960/0 (socket says 172.21.15.67:37960) 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbcaa800640 1 -- 172.21.15.67:0/1889705158 learned_addr learned my addr 172.21.15.67:0/1889705158 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbcab200640 1 --2- 172.21.15.67:0/1889705158 >> v2:172.21.15.187:3300/0 conn(0x7fbcac0ce4d0 0x7fbcac0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107747509 0 0) 0x7fbcac071f60 con 0x7fbcac15b860 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc90003520 con 0x7fbcac15b860 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.114+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3919211496 0 0) 0x7fbc90003520 con 0x7fbcac15b860 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 >> v2:172.21.15.187:3300/0 conn(0x7fbcac0ce4d0 msgr2=0x7fbcac0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 --2- 172.21.15.67:0/1889705158 >> v2:172.21.15.187:3300/0 conn(0x7fbcac0ce4d0 0x7fbcac0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 >> v1:172.21.15.187:6789/0 conn(0x7fbcac0d0b30 legacy=0x7fbcac0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac072520 con 0x7fbcac15b860 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbc9c001670 con 0x7fbcac15b860 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 >> v1:172.21.15.67:6789/0 conn(0x7fbcac15b860 legacy=0x7fbcac0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc900039b0 0x7fbc90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbc90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc900051c0 0x7fbc9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac071d10 con 0x7fbc90004800 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9000c060 con 0x7fbc900051c0 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9000c260 con 0x7fbc900039b0 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca3e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbc90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbcaa800640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc900039b0 0x7fbc90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbcab200640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc900051c0 0x7fbc9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.115+0000 7fbca3e00640 1 -- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc900039b0 msgr2=0x7fbc90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.116+0000 7fbca3e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc900039b0 0x7fbc90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.116+0000 7fbca3e00640 1 -- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc900051c0 msgr2=0x7fbc9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.116+0000 7fbca3e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc900051c0 0x7fbc9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.116+0000 7fbca3e00640 1 -- 172.21.15.67:0/1889705158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc90003520 con 0x7fbc90004800 2024-04-17T07:15:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.116+0000 7fbcab200640 1 -- 172.21.15.67:0/1889705158 reap_dead start 2024-04-17T07:15:27.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.117+0000 7fbca3e00640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbc90004c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbc94001750 tx=0x7fbc90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4be046edb59356 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.117+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94001370 con 0x7fbc90004800 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.117+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc940052d0 con 0x7fbc90004800 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.117+0000 7fbca9e00640 1 -- 172.21.15.67:0/1889705158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94003830 con 0x7fbc90004800 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 -- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 msgr2=0x7fbc90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbc90004c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbc94001750 tx=0x7fbc90005830 comp rx=0 tx=0).stop 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 -- 172.21.15.67:0/1889705158 shutdown_connections 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 --2- 172.21.15.67:0/1889705158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbc90004c80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 -- 172.21.15.67:0/1889705158 >> 172.21.15.67:0/1889705158 conn(0x7fbcac067ee0 msgr2=0x7fbcac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 -- 172.21.15.67:0/1889705158 shutdown_connections 2024-04-17T07:15:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.118+0000 7fbcb1400640 1 -- 172.21.15.67:0/1889705158 wait complete. 2024-04-17T07:15:27.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.121+0000 7fbcb1400640 1 Processor -- start 2024-04-17T07:15:27.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.121+0000 7fbcb1400640 1 -- start start 2024-04-17T07:15:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 0x7fbcac1549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcac0b18e0 0x7fbcac154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac15bc40 con 0x7fbc90004800 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac072ae0 con 0x7fbcac0b18e0 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcb1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbcac06f380 con 0x7fbcac0b1480 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcaa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcaa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43680/0 (socket says 172.21.15.67:43680) 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.123+0000 7fbcaa800640 1 -- 172.21.15.67:0/923689291 learned_addr learned my addr 172.21.15.67:0/923689291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcab200640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcac0b18e0 0x7fbcac154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbca3e00640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 0x7fbcac1549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 msgr2=0x7fbcac1549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 0x7fbcac1549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcac0b18e0 msgr2=0x7fbcac154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcac0b18e0 0x7fbcac154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 -- 172.21.15.67:0/923689291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac0c54c0 con 0x7fbc90004800 2024-04-17T07:15:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbca3e00640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 0x7fbcac1549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbcaa800640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00e760 tx=0x7fbc9c00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.124+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c00c6c0 con 0x7fbc90004800 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.125+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c018070 con 0x7fbc90004800 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.125+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac0c57a0 con 0x7fbc90004800 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.125+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c00ca80 con 0x7fbc90004800 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.125+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac0c5d60 con 0x7fbc90004800 2024-04-17T07:15:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.126+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac068ca0 con 0x7fbc90004800 2024-04-17T07:15:27.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.127+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) ==== 91057+0+0 (secure 0 0 0) 0x7fbc9c00cc20 con 0x7fbc90004800 2024-04-17T07:15:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.128+0000 7fbca2a00640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 0x7fbc8806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.129+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7fbc9c08ba30 con 0x7fbc90004800 2024-04-17T07:15:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.129+0000 7fbca3e00640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 msgr2=0x7fbc8806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.129+0000 7fbca3e00640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 0x7fbc8806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:15:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.133+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbc9c059e50 con 0x7fbc90004800 2024-04-17T07:15:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.330+0000 7fbca3e00640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 msgr2=0x7fbc8806ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.330+0000 7fbca3e00640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 0x7fbc8806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:15:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.394+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fbcac15edf0 con 0x7fbc90004800 2024-04-17T07:15:27.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.399+0000 7fbca2a00640 1 -- 172.21.15.67:0/923689291 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) ==== 138+0+0 (secure 0 0 0) 0x7fbc9c059750 con 0x7fbc90004800 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 msgr2=0x7fbc8806ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 0x7fbc8806ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 msgr2=0x7fbcac0c6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00e760 tx=0x7fbc9c00eb60 comp rx=0 tx=0).stop 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 shutdown_connections 2024-04-17T07:15:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcac0b18e0 0x7fbcac154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7fbc8806c940 0x7fbc8806ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcac0b1480 0x7fbcac1549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 --2- 172.21.15.67:0/923689291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90004800 0x7fbcac0c6280 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 >> 172.21.15.67:0/923689291 conn(0x7fbcac067ee0 msgr2=0x7fbcac0b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 shutdown_connections 2024-04-17T07:15:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.402+0000 7fbcb1400640 1 -- 172.21.15.67:0/923689291 wait complete. 2024-04-17T07:15:27.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:15:27.437 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:27.438+0000 7fbddda13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:15:27.488 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:27.488+0000 7f4723c42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:15:27.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:27.494+0000 7fbddda13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:15:27.547 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:27.547+0000 7f4723c42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:15:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.618+0000 7feefbe00640 1 Processor -- start 2024-04-17T07:15:27.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.619+0000 7feefbe00640 1 -- start start 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feefbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feef4160c30 0x7feef4161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feefbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef40d0980 0x7feef40d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feefbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef415d800 con 0x7feef415cfc0 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feefbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7feef406b8c0 con 0x7feef4160c30 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feefbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feef406bba0 con 0x7feef40d0980 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.620+0000 7feef9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feef4160c30 0x7feef4161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feef4160c30 0x7feef4161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43696/0 (socket says 172.21.15.67:43696) 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 -- 172.21.15.67:0/1211110370 learned_addr learned my addr 172.21.15.67:0/1211110370 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef8a00640 1 --1- 172.21.15.67:0/1211110370 >> v1:172.21.15.67:6789/0 conn(0x7feef415cfc0 0x7feef406c030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37966/0 (socket says 172.21.15.67:37966) 2024-04-17T07:15:27.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9e00640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef40d0980 0x7feef40d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 -- 172.21.15.67:0/1211110370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef40d0980 msgr2=0x7feef40d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef40d0980 0x7feef40d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 -- 172.21.15.67:0/1211110370 >> v1:172.21.15.67:6789/0 conn(0x7feef415cfc0 legacy=0x7feef406c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.621+0000 7feef9400640 1 -- 172.21.15.67:0/1211110370 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef415dac0 con 0x7feef4160c30 2024-04-17T07:15:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.622+0000 7feef9400640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160c30 0x7feef4161010 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7feee8009870 tx=0x7feee802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b81c365faacfc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feef3e00640 1 -- 172.21.15.67:0/1211110370 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee802f830 con 0x7feef4160c30 2024-04-17T07:15:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feef3e00640 1 -- 172.21.15.67:0/1211110370 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee802f9d0 con 0x7feef4160c30 2024-04-17T07:15:27.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feef3e00640 1 -- 172.21.15.67:0/1211110370 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee802fda0 con 0x7feef4160c30 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 -- 172.21.15.67:0/1211110370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160c30 msgr2=0x7feef4161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160c30 0x7feef4161010 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7feee8009870 tx=0x7feee802ba10 comp rx=0 tx=0).stop 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 -- 172.21.15.67:0/1211110370 shutdown_connections 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef40d0980 0x7feef40d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 --2- 172.21.15.67:0/1211110370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160c30 0x7feef4161010 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.623+0000 7feefbe00640 1 -- 172.21.15.67:0/1211110370 >> 172.21.15.67:0/1211110370 conn(0x7feef4073360 msgr2=0x7feef409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.624+0000 7feefbe00640 1 -- 172.21.15.67:0/1211110370 shutdown_connections 2024-04-17T07:15:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.624+0000 7feefbe00640 1 -- 172.21.15.67:0/1211110370 wait complete. 2024-04-17T07:15:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.625+0000 7feefbe00640 1 Processor -- start 2024-04-17T07:15:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.625+0000 7feefbe00640 1 -- start start 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 0x7feef414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 0x7feef40b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feef415d3a0 con 0x7feef415cfc0 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feef406c740 con 0x7feef40d0980 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.626+0000 7feefbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feef415d520 con 0x7feef4160c30 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43704/0 (socket says 172.21.15.67:43704) 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 -- 172.21.15.67:0/1499926565 learned_addr learned my addr 172.21.15.67:0/1499926565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef9400640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 0x7feef414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef9e00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 0x7feef40b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 msgr2=0x7feef40b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 0x7feef40b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 msgr2=0x7feef414e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 0x7feef414e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 -- 172.21.15.67:0/1499926565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef414e920 con 0x7feef415cfc0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef9e00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 0x7feef40b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef9400640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 0x7feef414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.627+0000 7feef8a00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feee400c430 tx=0x7feee400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.628+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee40074e0 con 0x7feef415cfc0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.628+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee4007680 con 0x7feef415cfc0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.628+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4004a00 con 0x7feef415cfc0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.628+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef415f610 con 0x7feef415cfc0 2024-04-17T07:15:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.628+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef415fbd0 con 0x7feef415cfc0 2024-04-17T07:15:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.629+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef40b7570 con 0x7feef415cfc0 2024-04-17T07:15:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.630+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) ==== 91057+0+0 (secure 0 0 0) 0x7feee401a020 con 0x7feef415cfc0 2024-04-17T07:15:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.631+0000 7feef1600640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 0x7feed806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.631+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7feee408b530 con 0x7feef415cfc0 2024-04-17T07:15:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.631+0000 7feef9400640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 msgr2=0x7feed806ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.631+0000 7feef9400640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 0x7feed806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:15:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.639+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7feee4059830 con 0x7feef415cfc0 2024-04-17T07:15:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.832+0000 7feef9400640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 msgr2=0x7feed806ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.832+0000 7feef9400640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 0x7feed806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:15:27.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.850+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7feef40685d0 con 0x7feef415cfc0 2024-04-17T07:15:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.856+0000 7feef1600640 1 -- 172.21.15.67:0/1499926565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) ==== 133+0+0 (secure 0 0 0) 0x7feee4059130 con 0x7feef415cfc0 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.858+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 msgr2=0x7feed806ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.858+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 0x7feed806ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.858+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 msgr2=0x7feef40cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.858+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7feee400c430 tx=0x7feee400c8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 shutdown_connections 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7feed806c970 0x7feed806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feef4160c30 0x7feef40b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef415cfc0 0x7feef40cae30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 --2- 172.21.15.67:0/1499926565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feef40d0980 0x7feef414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 >> 172.21.15.67:0/1499926565 conn(0x7feef4073360 msgr2=0x7feef415ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 shutdown_connections 2024-04-17T07:15:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:27.859+0000 7feefbe00640 1 -- 172.21.15.67:0/1499926565 wait complete. 2024-04-17T07:15:27.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:27.879 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T07:15:27.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:27.936 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-04-17T07:15:27.958 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:27.959+0000 7fbddda13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:15:27.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-04-17T07:15:28.015 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.016+0000 7fbddda13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:15:28.044 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.044+0000 7f4723c42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:15:28.067 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.068+0000 7fbddda13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:15:28.100 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.100+0000 7f4723c42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:15:28.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.130+0000 7fbddda13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:15:28.155 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.155+0000 7f4723c42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:15:28.178 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.179+0000 7fbddda13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:15:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.186+0000 7f2906400640 1 Processor -- start 2024-04-17T07:15:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.187+0000 7f2906400640 1 -- start start 2024-04-17T07:15:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f290015eca0 0x7f29000d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 0x7f29000cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f29000d03b0 con 0x7f290015eca0 2024-04-17T07:15:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2900163860 con 0x7f29000cfa10 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f2906400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29000d1d40 con 0x7f29000d1090 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f28ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f290015eca0 0x7f29000d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f28ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f290015eca0 0x7f29000d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43706/0 (socket says 172.21.15.67:43706) 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.188+0000 7f28ff400640 1 -- 172.21.15.67:0/4129585614 learned_addr learned my addr 172.21.15.67:0/4129585614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28fea00640 1 --2- 172.21.15.67:0/4129585614 >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 0x7f29000cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ffe00640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ff400640 1 -- 172.21.15.67:0/4129585614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 msgr2=0x7f29000d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ff400640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ff400640 1 -- 172.21.15.67:0/4129585614 >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 msgr2=0x7f29000cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ff400640 1 --2- 172.21.15.67:0/4129585614 >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 0x7f29000cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ff400640 1 -- 172.21.15.67:0/4129585614 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29000d2c30 con 0x7f290015eca0 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28ffe00640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.189+0000 7f28fea00640 1 --2- 172.21.15.67:0/4129585614 >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 0x7f29000cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.190+0000 7f28ff400640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000d39f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f29000b7500 tx=0x7f28f002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67806bb3223d6d27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.190+0000 7f28fe000640 1 -- 172.21.15.67:0/4129585614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0035020 con 0x7f290015eca0 2024-04-17T07:15:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.190+0000 7f28fe000640 1 -- 172.21.15.67:0/4129585614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f002d040 con 0x7f290015eca0 2024-04-17T07:15:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.190+0000 7f28fe000640 1 -- 172.21.15.67:0/4129585614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0031a80 con 0x7f290015eca0 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 -- 172.21.15.67:0/4129585614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 msgr2=0x7f29000d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000d39f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f29000b7500 tx=0x7f28f002b900 comp rx=0 tx=0).stop 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 -- 172.21.15.67:0/4129585614 shutdown_connections 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 --2- 172.21.15.67:0/4129585614 >> v2:172.21.15.187:3300/0 conn(0x7f29000cfa10 0x7f29000cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 --2- 172.21.15.67:0/4129585614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000d39f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 -- 172.21.15.67:0/4129585614 >> 172.21.15.67:0/4129585614 conn(0x7f2900067600 msgr2=0x7f29000679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 -- 172.21.15.67:0/4129585614 shutdown_connections 2024-04-17T07:15:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.191+0000 7f2906400640 1 -- 172.21.15.67:0/4129585614 wait complete. 2024-04-17T07:15:28.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.193+0000 7f2906400640 1 Processor -- start 2024-04-17T07:15:28.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.193+0000 7f2906400640 1 -- start start 2024-04-17T07:15:28.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29000cfa10 0x7f290014e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29000d34d0 con 0x7f290015eca0 2024-04-17T07:15:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f29000d31f0 con 0x7f29000cfa10 2024-04-17T07:15:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f2906400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29000d1a40 con 0x7f29000d1090 2024-04-17T07:15:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.194+0000 7f28ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43712/0 (socket says 172.21.15.67:43712) 2024-04-17T07:15:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 -- 172.21.15.67:0/2906837178 learned_addr learned my addr 172.21.15.67:0/2906837178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29000cfa10 0x7f290014e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28fea00640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 msgr2=0x7f29000ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29000cfa10 msgr2=0x7f290014e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29000cfa10 0x7f290014e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290014e7d0 con 0x7f290015eca0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29000cfa10 0x7f290014e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28fea00640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29000d1090 0x7f29000ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.195+0000 7f28ffe00640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000cabf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f28f800cef0 tx=0x7f28f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.196+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f801a070 con 0x7f290015eca0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.196+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f800d860 con 0x7f290015eca0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.196+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f800dc50 con 0x7f290015eca0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.196+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290014b040 con 0x7f290015eca0 2024-04-17T07:15:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.196+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290014b600 con 0x7f290015eca0 2024-04-17T07:15:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.197+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2900068ca0 con 0x7f290015eca0 2024-04-17T07:15:28.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.198+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) ==== 91057+0+0 (secure 0 0 0) 0x7f28f801d070 con 0x7f290015eca0 2024-04-17T07:15:28.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.199+0000 7f28f7400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:28.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.199+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5709+0+0 (secure 0 0 0) 0x7f28f808c4d0 con 0x7f290015eca0 2024-04-17T07:15:28.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.199+0000 7f28ff400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 msgr2=0x7f28e006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:28.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.199+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:15:28.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.204+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f28f805a920 con 0x7f290015eca0 2024-04-17T07:15:28.223 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.223+0000 7f4723c42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:15:28.277 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.277+0000 7f4723c42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:15:28.308 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.309+0000 7fbddda13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:15:28.371 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.372+0000 7fbddda13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:15:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.400+0000 7f28ff400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 msgr2=0x7f28e006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.400+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:15:28.415 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.415+0000 7f4723c42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:15:28.421 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.422+0000 7fbddda13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:15:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.431+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f29000d2970 con 0x7f28e006c940 2024-04-17T07:15:28.478 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.479+0000 7fbddda13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:15:28.481 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.481+0000 7f4723c42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:15:28.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.531+0000 7fbddda13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:15:28.534 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.534+0000 7f4723c42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:15:28.594 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.593+0000 7f4723c42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:15:28.647 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:28.647+0000 7f4723c42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:15:28.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.801+0000 7f28ff400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 msgr2=0x7f28e006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:28.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:28.801+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:15:28.919 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:28.920+0000 7fbddda13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:15:29.054 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.053+0000 7f4723c42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:15:29.078 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.080+0000 7fbddda13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:15:29.188 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.189+0000 7fbddda13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:15:29.222 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.222+0000 7f4723c42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:15:29.238 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.239+0000 7fbddda13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:15:29.337 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.337+0000 7f4723c42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:15:29.365 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.366+0000 7fbddda13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:15:29.390 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.390+0000 7f4723c42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:15:29.473 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.474+0000 7fbddda13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:15:29.523 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.523+0000 7f4723c42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:15:29.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:29.602+0000 7f28ff400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 msgr2=0x7f28e006ee00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3291261531 2024-04-17T07:15:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:29.602+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:15:29.636 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.636+0000 7f4723c42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:15:29.647 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.648+0000 7fbddda13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:15:29.696 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.697+0000 7fbddda13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:15:29.820 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.820+0000 7f4723c42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:15:29.833 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:29.834+0000 7fbddda13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:15:29.872 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:29.872+0000 7f4723c42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:15:30.017 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:30.016+0000 7f4723c42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:15:30.633 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:30.634+0000 7fbddda13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:15:30.867 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:30.866+0000 7f4723c42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:15:30.937 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:30.938+0000 7fbddda13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:15:30.994 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:30.995+0000 7fbddda13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:15:31.125 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.126+0000 7fbddda13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:15:31.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:31.149+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) ==== 90768+0+0 (secure 0 0 0) 0x7f28f8056890 con 0x7f290015eca0 2024-04-17T07:15:31.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:31.150+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 msgr2=0x7f28e006ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:15:31.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:31.150+0000 7f28f7400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/3291261531,v1:172.21.15.187:6833/3291261531] conn(0x7f28e006c940 0x7f28e006ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:31.197 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:31.197+0000 7f4723c42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:15:31.265 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:31.265+0000 7f4723c42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:15:31.428 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:15:31.427+0000 7f4723c42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd54200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd54200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd54200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd54200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd54200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd50600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd50600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.503 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd50600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.504 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd50600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:31.504 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:15:31.504+0000 7fbd50600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:15:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.160+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 8) ==== 90895+0+0 (secure 0 0 0) 0x7f28f8005d10 con 0x7f290015eca0 2024-04-17T07:15:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.160+0000 7f28f7400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f28e00747f0 0x7f28e0076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.160+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f28f8019ec0 con 0x7f28e00747f0 2024-04-17T07:15:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.164+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f28e00747f0 0x7f28e0076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.175+0000 7f28ff400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f28e00747f0 0x7f28e0076be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28f0036000 tx=0x7f28f0008030 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:32.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.393+0000 7f28f7400640 1 -- 172.21.15.67:0/2906837178 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f28f8019ec0 con 0x7f28e00747f0 2024-04-17T07:15:32.394 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir0 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f28e00747f0 msgr2=0x7f28e0076be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f28e00747f0 0x7f28e0076be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28f0036000 tx=0x7f28f0008030 comp rx=0 tx=0).stop 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 msgr2=0x7f29000cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 --2- 172.21.15.67:0/2906837178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f290015eca0 0x7f29000cabf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f28f800cef0 tx=0x7f28f800a710 comp rx=0 tx=0).stop 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f28ffe00640 1 -- 172.21.15.67:0/2906837178 reap_dead start 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 shutdown_connections 2024-04-17T07:15:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 >> 172.21.15.67:0/2906837178 conn(0x7f2900067600 msgr2=0x7f29000cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:32.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 shutdown_connections 2024-04-17T07:15:32.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.397+0000 7f2906400640 1 -- 172.21.15.67:0/2906837178 wait complete. 2024-04-17T07:15:32.418 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-04-17T07:15:32.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:32.418 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-04-17T07:15:32.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-04-17T07:15:32.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.691+0000 7f1bd4800640 1 Processor -- start 2024-04-17T07:15:32.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.691+0000 7f1bd4800640 1 -- start start 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd4800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bcc0cfa10 0x7f1bcc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd4800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bcc0d1090 0x7f1bcc0d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd4800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc0d4100 con 0x7f1bcc15eca0 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd4800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1bcc0d03b0 con 0x7f1bcc0d1090 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd4800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1bcc163860 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bcc0d1090 0x7f1bcc0d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bcc15eca0 0x7f1bcc0d39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37978/0 (socket says 172.21.15.67:37978) 2024-04-17T07:15:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd1e00640 1 -- 172.21.15.67:0/3223436159 learned_addr learned my addr 172.21.15.67:0/3223436159 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:32.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.693+0000 7f1bd1400640 1 --2- 172.21.15.67:0/3223436159 >> v2:172.21.15.187:3300/0 conn(0x7f1bcc0cfa10 0x7f1bcc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:32.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.694+0000 7f1bd2800640 1 -- 172.21.15.67:0/3223436159 >> v2:172.21.15.187:3300/0 conn(0x7f1bcc0cfa10 msgr2=0x7f1bcc0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:32.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.694+0000 7f1bd2800640 1 --2- 172.21.15.67:0/3223436159 >> v2:172.21.15.187:3300/0 conn(0x7f1bcc0cfa10 0x7f1bcc0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.694+0000 7f1bd2800640 1 -- 172.21.15.67:0/3223436159 >> v1:172.21.15.67:6789/0 conn(0x7f1bcc15eca0 legacy=0x7f1bcc0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.694+0000 7f1bd2800640 1 -- 172.21.15.67:0/3223436159 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc0d1d40 con 0x7f1bcc0d1090 2024-04-17T07:15:32.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.694+0000 7f1bd0a00640 1 -- 172.21.15.67:0/3223436159 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719406325 0 0) 0x7f1bcc0d4100 con 0x7f1bcc15eca0 2024-04-17T07:15:32.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.695+0000 7f1bd2800640 1 --2- 172.21.15.67:0/3223436159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0d1090 0x7f1bcc0d14b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0b7500 tx=0x7f1bc802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf48373f3f6301c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:32.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd0a00640 1 -- 172.21.15.67:0/3223436159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8035020 con 0x7f1bcc0d1090 2024-04-17T07:15:32.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd0a00640 1 -- 172.21.15.67:0/3223436159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc8031830 con 0x7f1bcc0d1090 2024-04-17T07:15:32.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd0a00640 1 -- 172.21.15.67:0/3223436159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8031c00 con 0x7f1bcc0d1090 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd4800640 1 -- 172.21.15.67:0/3223436159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0d1090 msgr2=0x7f1bcc0d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd4800640 1 --2- 172.21.15.67:0/3223436159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0d1090 0x7f1bcc0d14b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0b7500 tx=0x7f1bc802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd4800640 1 -- 172.21.15.67:0/3223436159 shutdown_connections 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd4800640 1 --2- 172.21.15.67:0/3223436159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0d1090 0x7f1bcc0d14b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.696+0000 7f1bd4800640 1 --2- 172.21.15.67:0/3223436159 >> v2:172.21.15.187:3300/0 conn(0x7f1bcc0cfa10 0x7f1bcc0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.697+0000 7f1bd4800640 1 -- 172.21.15.67:0/3223436159 >> 172.21.15.67:0/3223436159 conn(0x7f1bcc067600 msgr2=0x7f1bcc0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.697+0000 7f1bd4800640 1 -- 172.21.15.67:0/3223436159 shutdown_connections 2024-04-17T07:15:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.697+0000 7f1bd4800640 1 -- 172.21.15.67:0/3223436159 wait complete. 2024-04-17T07:15:32.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.698+0000 7f1bd4800640 1 Processor -- start 2024-04-17T07:15:32.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.698+0000 7f1bd4800640 1 -- start start 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bcc0d1090 0x7f1bcc1f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15eca0 0x7f1bcc154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc0d34d0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc0d31f0 con 0x7f1bcc15eca0 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.699+0000 7f1bd4800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bcc0d19f0 con 0x7f1bcc0d1090 2024-04-17T07:15:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43732/0 (socket says 172.21.15.67:43732) 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 -- 172.21.15.67:0/921381064 learned_addr learned my addr 172.21.15.67:0/921381064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 -- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bcc0d1090 msgr2=0x7f1bcc1f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bcc0d1090 0x7f1bcc1f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 -- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15eca0 msgr2=0x7f1bcc154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15eca0 0x7f1bcc154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 -- 172.21.15.67:0/921381064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc1f6430 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.700+0000 7f1bd1e00640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c7c0 tx=0x7f1bc400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.701+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4018070 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.701+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc400a5e0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.701+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bcc1f6710 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.701+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc400a9d0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.701+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bcc1f6cd0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.702+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bcc068ca0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.703+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 90895+0+0 (secure 0 0 0) 0x7f1bc401b070 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.704+0000 7f1bc2000640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 0x7f1bb006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.704+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f1bc408afa0 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.705+0000 7f1bd1400640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 0x7f1bb006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:32.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.706+0000 7f1bd1400640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 0x7f1bb006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0d50c0 tx=0x7f1bbc008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.709+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1bc4059540 con 0x7f1bcc0cfa10 2024-04-17T07:15:32.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:32.939+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1bcc0d2970 con 0x7f1bb006c790 2024-04-17T07:15:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.141+0000 7f1bc2000640 1 -- 172.21.15.67:0/921381064 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1bcc0d2970 con 0x7f1bb006c790 2024-04-17T07:15:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir1 2024-04-17T07:15:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 msgr2=0x7f1bb006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 0x7f1bb006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0d50c0 tx=0x7f1bbc008040 comp rx=0 tx=0).stop 2024-04-17T07:15:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 msgr2=0x7f1bcc1602f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c7c0 tx=0x7f1bc400cc80 comp rx=0 tx=0).stop 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 shutdown_connections 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1bb006c790 0x7f1bb006ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bcc15eca0 0x7f1bcc154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bcc0d1090 0x7f1bcc1f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 --2- 172.21.15.67:0/921381064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bcc0cfa10 0x7f1bcc1602f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 >> 172.21.15.67:0/921381064 conn(0x7f1bcc067600 msgr2=0x7f1bcc15d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 shutdown_connections 2024-04-17T07:15:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.144+0000 7f1bd4800640 1 -- 172.21.15.67:0/921381064 wait complete. 2024-04-17T07:15:33.170 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-04-17T07:15:33.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:33.171 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-04-17T07:15:33.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-04-17T07:15:33.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.439+0000 7fdf57200640 1 Processor -- start 2024-04-17T07:15:33.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.439+0000 7fdf57200640 1 -- start start 2024-04-17T07:15:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.441+0000 7fdf57200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 0x7fdf5806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.441+0000 7fdf57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.441+0000 7fdf57200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5806df40 con 0x7fdf5806f800 2024-04-17T07:15:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.441+0000 7fdf57200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdf5806ed00 con 0x7fdf5815b6b0 2024-04-17T07:15:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.441+0000 7fdf57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5806e870 con 0x7fdf580cfc10 2024-04-17T07:15:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.442+0000 7fdf55e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 0x7fdf5806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.442+0000 7fdf56800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.442+0000 7fdf56800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:57450/0 (socket says 172.21.15.67:57450) 2024-04-17T07:15:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.442+0000 7fdf56800640 1 -- 172.21.15.67:0/3926128005 learned_addr learned my addr 172.21.15.67:0/3926128005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.443+0000 7fdf56800640 1 -- 172.21.15.67:0/3926128005 >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 msgr2=0x7fdf5806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.443+0000 7fdf56800640 1 --2- 172.21.15.67:0/3926128005 >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 0x7fdf5806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.443+0000 7fdf56800640 1 -- 172.21.15.67:0/3926128005 >> v1:172.21.15.187:6789/0 conn(0x7fdf5806f800 legacy=0x7fdf5815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.443+0000 7fdf56800640 1 -- 172.21.15.67:0/3926128005 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5806e220 con 0x7fdf580cfc10 2024-04-17T07:15:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.443+0000 7fdf55e00640 1 --2- 172.21.15.67:0/3926128005 >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 0x7fdf5806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:15:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.444+0000 7fdf56800640 1 --2- 172.21.15.67:0/3926128005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009e50 tx=0x7fdf5002e050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766259bb0c5a62bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.445+0000 7fdf54a00640 1 -- 172.21.15.67:0/3926128005 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5002ebf0 con 0x7fdf580cfc10 2024-04-17T07:15:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.445+0000 7fdf54a00640 1 -- 172.21.15.67:0/3926128005 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5002ed90 con 0x7fdf580cfc10 2024-04-17T07:15:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.445+0000 7fdf54a00640 1 -- 172.21.15.67:0/3926128005 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50036610 con 0x7fdf580cfc10 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.445+0000 7fdf57200640 1 -- 172.21.15.67:0/3926128005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 msgr2=0x7fdf5806be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.445+0000 7fdf57200640 1 --2- 172.21.15.67:0/3926128005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009e50 tx=0x7fdf5002e050 comp rx=0 tx=0).stop 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 -- 172.21.15.67:0/3926128005 shutdown_connections 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 --2- 172.21.15.67:0/3926128005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5806be00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 --2- 172.21.15.67:0/3926128005 >> v2:172.21.15.187:3300/0 conn(0x7fdf5815b6b0 0x7fdf5806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 -- 172.21.15.67:0/3926128005 >> 172.21.15.67:0/3926128005 conn(0x7fdf580b2190 msgr2=0x7fdf580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 -- 172.21.15.67:0/3926128005 shutdown_connections 2024-04-17T07:15:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.446+0000 7fdf57200640 1 -- 172.21.15.67:0/3926128005 wait complete. 2024-04-17T07:15:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.447+0000 7fdf57200640 1 Processor -- start 2024-04-17T07:15:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.447+0000 7fdf57200640 1 -- start start 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5806f800 0x7fdf5814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5814bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf580afee0 con 0x7fdf5815b6b0 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdf580afd60 con 0x7fdf5806f800 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.448+0000 7fdf57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdf580b44a0 con 0x7fdf580cfc10 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53698/0 (socket says 172.21.15.67:53698) 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 -- 172.21.15.67:0/4000521941 learned_addr learned my addr 172.21.15.67:0/4000521941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf55e00640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5806f800 0x7fdf5814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf55400640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5814bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 -- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 msgr2=0x7fdf5814bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5814bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 -- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5806f800 msgr2=0x7fdf5814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5806f800 0x7fdf5814e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.449+0000 7fdf56800640 1 -- 172.21.15.67:0/4000521941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5814d210 con 0x7fdf5815b6b0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf56800640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdf50038040 tx=0x7fdf50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50042030 con 0x7fdf5815b6b0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5002e9f0 con 0x7fdf5815b6b0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5003e630 con 0x7fdf5815b6b0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5814c970 con 0x7fdf5815b6b0 2024-04-17T07:15:33.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.450+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5814cf30 con 0x7fdf5815b6b0 2024-04-17T07:15:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.451+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf580cc910 con 0x7fdf5815b6b0 2024-04-17T07:15:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.452+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 91160+0+0 (secure 0 0 0) 0x7fdf50005d10 con 0x7fdf5815b6b0 2024-04-17T07:15:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.454+0000 7fdf4e000640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 0x7fdf3806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.454+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fdf500ad990 con 0x7fdf5815b6b0 2024-04-17T07:15:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.455+0000 7fdf55e00640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 0x7fdf3806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.456+0000 7fdf55e00640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 0x7fdf3806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009560 tx=0x7fdf48008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.459+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdf580cc910 con 0x7fdf5815b6b0 2024-04-17T07:15:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.688+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fdf580cd5f0 con 0x7fdf3806ca60 2024-04-17T07:15:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.887+0000 7fdf4e000640 1 -- 172.21.15.67:0/4000521941 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fdf580cd5f0 con 0x7fdf3806ca60 2024-04-17T07:15:33.887 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir2 2024-04-17T07:15:33.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 msgr2=0x7fdf3806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 0x7fdf3806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009560 tx=0x7fdf48008040 comp rx=0 tx=0).stop 2024-04-17T07:15:33.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 msgr2=0x7fdf58155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:33.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdf50038040 tx=0x7fdf50004a40 comp rx=0 tx=0).stop 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 shutdown_connections 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fdf3806ca60 0x7fdf3806ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5815b6b0 0x7fdf58155fa0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf580cfc10 0x7fdf5814bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 --2- 172.21.15.67:0/4000521941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5806f800 0x7fdf5814e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.891+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 >> 172.21.15.67:0/4000521941 conn(0x7fdf580b2190 msgr2=0x7fdf580ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.892+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 shutdown_connections 2024-04-17T07:15:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:33.892+0000 7fdf57200640 1 -- 172.21.15.67:0/4000521941 wait complete. 2024-04-17T07:15:33.916 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-04-17T07:15:33.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:33.916 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-04-17T07:15:33.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-04-17T07:15:34.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.194+0000 7ff782800640 1 Processor -- start 2024-04-17T07:15:34.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.194+0000 7ff782800640 1 -- start start 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff782800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff782800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c162da0 0x7ff77c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff782800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77c166a60 con 0x7ff77c0b4580 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff782800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff77c166810 con 0x7ff77c162da0 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff782800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c166d40 con 0x7ff77c0b41a0 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff781e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c162da0 0x7ff77c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff781e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c162da0 0x7ff77c06dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53700/0 (socket says 172.21.15.67:53700) 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff781e00640 1 -- 172.21.15.67:0/2733078727 learned_addr learned my addr 172.21.15.67:0/2733078727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.196+0000 7ff780a00640 1 --1- 172.21.15.67:0/2733078727 >> v1:172.21.15.67:6789/0 conn(0x7ff77c0b4580 0x7ff77c162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33694/0 (socket says 172.21.15.67:33694) 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781400640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781e00640 1 -- 172.21.15.67:0/2733078727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 msgr2=0x7ff77c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781e00640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c0b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781e00640 1 -- 172.21.15.67:0/2733078727 >> v1:172.21.15.67:6789/0 conn(0x7ff77c0b4580 legacy=0x7ff77c162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781e00640 1 -- 172.21.15.67:0/2733078727 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c167020 con 0x7ff77c162da0 2024-04-17T07:15:34.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.197+0000 7ff781400640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c0b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.198+0000 7ff781e00640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c162da0 0x7ff77c06dd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e50 tx=0x7ff77402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8a76e410dfd777 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.198+0000 7ff77be00640 1 -- 172.21.15.67:0/2733078727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77402cb00 con 0x7ff77c162da0 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.198+0000 7ff77be00640 1 -- 172.21.15.67:0/2733078727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77402cca0 con 0x7ff77c162da0 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.198+0000 7ff77be00640 1 -- 172.21.15.67:0/2733078727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774034610 con 0x7ff77c162da0 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 -- 172.21.15.67:0/2733078727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c162da0 msgr2=0x7ff77c06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c162da0 0x7ff77c06dd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff774009e50 tx=0x7ff77402bf60 comp rx=0 tx=0).stop 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 -- 172.21.15.67:0/2733078727 shutdown_connections 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c162da0 0x7ff77c06dd30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 --2- 172.21.15.67:0/2733078727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c0b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 -- 172.21.15.67:0/2733078727 >> 172.21.15.67:0/2733078727 conn(0x7ff77c0b2190 msgr2=0x7ff77c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 -- 172.21.15.67:0/2733078727 shutdown_connections 2024-04-17T07:15:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.199+0000 7ff782800640 1 -- 172.21.15.67:0/2733078727 wait complete. 2024-04-17T07:15:34.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.201+0000 7ff782800640 1 Processor -- start 2024-04-17T07:15:34.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.201+0000 7ff782800640 1 -- start start 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c162da0 0x7ff77c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c06f980 con 0x7ff77c0b4580 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c06f800 con 0x7ff77c162da0 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.202+0000 7ff782800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c06fb00 con 0x7ff77c0b41a0 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53708/0 (socket says 172.21.15.67:53708) 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 -- 172.21.15.67:0/1166455219 learned_addr learned my addr 172.21.15.67:0/1166455219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff781400640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 -- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 msgr2=0x7ff77c14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c14c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 -- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c162da0 msgr2=0x7ff77c0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff781e00640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c162da0 0x7ff77c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c162da0 0x7ff77c0ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 -- 172.21.15.67:0/1166455219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c15ba20 con 0x7ff77c0b4580 2024-04-17T07:15:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.203+0000 7ff780a00640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c400 tx=0x7ff76800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.204+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680074e0 con 0x7ff77c0b4580 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.204+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768007680 con 0x7ff77c0b4580 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.204+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77c15bd00 con 0x7ff77c0b4580 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.204+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768004a00 con 0x7ff77c0b4580 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.204+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77c14ca80 con 0x7ff77c0b4580 2024-04-17T07:15:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.205+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff77c068a10 con 0x7ff77c0b4580 2024-04-17T07:15:34.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.209+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 91160+0+0 (secure 0 0 0) 0x7ff76801a020 con 0x7ff77c0b4580 2024-04-17T07:15:34.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.210+0000 7ff779600640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 0x7ff75c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.210+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7ff76808bf20 con 0x7ff77c0b4580 2024-04-17T07:15:34.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.210+0000 7ff781400640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 0x7ff75c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.212+0000 7ff781400640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 0x7ff75c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff770008850 tx=0x7ff770008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.213+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff768021070 con 0x7ff77c0b4580 2024-04-17T07:15:34.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.446+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff77c06c7b0 con 0x7ff75c06ca60 2024-04-17T07:15:34.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.651+0000 7ff779600640 1 -- 172.21.15.67:0/1166455219 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ff77c06c7b0 con 0x7ff75c06ca60 2024-04-17T07:15:34.652 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir3 2024-04-17T07:15:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 msgr2=0x7ff75c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 0x7ff75c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff770008850 tx=0x7ff770008040 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 msgr2=0x7ff77c0ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff76800c400 tx=0x7ff76800c8c0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 shutdown_connections 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff75c06ca60 0x7ff75c06ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c162da0 0x7ff77c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c0b4580 0x7ff77c0ce230 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 --2- 172.21.15.67:0/1166455219 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c0b41a0 0x7ff77c14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 >> 172.21.15.67:0/1166455219 conn(0x7ff77c0b2190 msgr2=0x7ff77c072a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.655+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 shutdown_connections 2024-04-17T07:15:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.656+0000 7ff782800640 1 -- 172.21.15.67:0/1166455219 wait complete. 2024-04-17T07:15:34.679 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-04-17T07:15:34.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:34.680 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-04-17T07:15:34.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-04-17T07:15:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.967+0000 7f5e80c00640 1 Processor -- start 2024-04-17T07:15:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.967+0000 7f5e80c00640 1 -- start start 2024-04-17T07:15:34.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e80c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e80c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 0x7f5e7c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e80c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e7c166a60 con 0x7f5e7c0b4580 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e80c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5e7c166810 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e80c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7c166d40 con 0x7f5e7c162da0 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.968+0000 7f5e7b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53722/0 (socket says 172.21.15.67:53722) 2024-04-17T07:15:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 -- 172.21.15.67:0/2299095733 learned_addr learned my addr 172.21.15.67:0/2299095733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 -- 172.21.15.67:0/2299095733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 msgr2=0x7f5e7c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7be00640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 0x7f5e7c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 0x7f5e7c06dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 -- 172.21.15.67:0/2299095733 >> v1:172.21.15.187:6789/0 conn(0x7f5e7c0b4580 legacy=0x7f5e7c162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7b400640 1 -- 172.21.15.67:0/2299095733 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c167020 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.969+0000 7f5e7be00640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 0x7f5e7c06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:34.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.970+0000 7f5e7b400640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dde4b1d02b23fbb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e7aa00640 1 -- 172.21.15.67:0/2299095733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c035020 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e7aa00640 1 -- 172.21.15.67:0/2299095733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e6c031640 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e7aa00640 1 -- 172.21.15.67:0/2299095733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c02d040 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e80c00640 1 -- 172.21.15.67:0/2299095733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 msgr2=0x7f5e7c0b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e80c00640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.971+0000 7f5e80c00640 1 -- 172.21.15.67:0/2299095733 shutdown_connections 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.972+0000 7f5e80c00640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c162da0 0x7f5e7c06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.972+0000 7f5e80c00640 1 --2- 172.21.15.67:0/2299095733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c0b3490 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.972+0000 7f5e80c00640 1 -- 172.21.15.67:0/2299095733 >> 172.21.15.67:0/2299095733 conn(0x7f5e7c0b2190 msgr2=0x7f5e7c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.972+0000 7f5e80c00640 1 -- 172.21.15.67:0/2299095733 shutdown_connections 2024-04-17T07:15:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.972+0000 7f5e80c00640 1 -- 172.21.15.67:0/2299095733 wait complete. 2024-04-17T07:15:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.973+0000 7f5e80c00640 1 Processor -- start 2024-04-17T07:15:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.974+0000 7f5e80c00640 1 -- start start 2024-04-17T07:15:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.974+0000 7f5e80c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.974+0000 7f5e80c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c0b4580 0x7f5e7c1f9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.974+0000 7f5e80c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e7c162da0 0x7f5e7c165610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e80c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c06f980 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e80c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c06f800 con 0x7f5e7c162da0 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e80c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7c06fb00 con 0x7f5e7c0b4580 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53724/0 (socket says 172.21.15.67:53724) 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 -- 172.21.15.67:0/1672959647 learned_addr learned my addr 172.21.15.67:0/1672959647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e73e00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c0b4580 0x7f5e7c1f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7be00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e7c162da0 0x7f5e7c165610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 -- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c0b4580 msgr2=0x7f5e7c1f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c0b4580 0x7f5e7c1f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 -- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e7c162da0 msgr2=0x7f5e7c165610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e7c162da0 0x7f5e7c165610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.975+0000 7f5e7b400640 1 -- 172.21.15.67:0/1672959647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c159f70 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e7b400640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c02b930 tx=0x7f5e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c040070 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e6c039c10 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7c158ed0 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c035040 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.976+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7c159490 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.977+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e7c068a10 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.978+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 91160+0+0 (secure 0 0 0) 0x7f5e6c04b060 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.979+0000 7f5e73400640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 0x7f5e5806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.979+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f5e6c0ada10 con 0x7f5e7c0b41a0 2024-04-17T07:15:34.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.980+0000 7f5e73e00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 0x7f5e5806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.981+0000 7f5e73e00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 0x7f5e5806ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c156a20 tx=0x7f5e64009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:34.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:34.984+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5e6c02f300 con 0x7f5e7c0b41a0 2024-04-17T07:15:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.214+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5e7c06c7b0 con 0x7f5e5806ca60 2024-04-17T07:15:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.229+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f5e6c038020 con 0x7f5e7c0b41a0 2024-04-17T07:15:35.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.428+0000 7f5e73400640 1 -- 172.21.15.67:0/1672959647 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5e7c06c7b0 con 0x7f5e5806ca60 2024-04-17T07:15:35.428 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir4 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.432+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 msgr2=0x7f5e5806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.432+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 0x7f5e5806ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c156a20 tx=0x7f5e64009040 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.432+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 msgr2=0x7f5e7c1564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.432+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c02b930 tx=0x7f5e6c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.432+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 shutdown_connections 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f5e5806ca60 0x7f5e5806ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e7c162da0 0x7f5e7c165610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e7c0b4580 0x7f5e7c1f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 --2- 172.21.15.67:0/1672959647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e7c0b41a0 0x7f5e7c1564e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 >> 172.21.15.67:0/1672959647 conn(0x7f5e7c0b2190 msgr2=0x7f5e7c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:35.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 shutdown_connections 2024-04-17T07:15:35.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.433+0000 7f5e80c00640 1 -- 172.21.15.67:0/1672959647 wait complete. 2024-04-17T07:15:35.457 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-04-17T07:15:35.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:35.458 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-04-17T07:15:35.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-04-17T07:15:35.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.739+0000 7ff916000640 1 Processor -- start 2024-04-17T07:15:35.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.739+0000 7ff916000640 1 -- start start 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.741+0000 7ff916000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff9100cd920 0x7ff91006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.741+0000 7ff916000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff91006c0a0 0x7ff91006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff916000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff910146360 con 0x7ff910145f80 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff916000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff9100727a0 con 0x7ff91006c0a0 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff916000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff910071e70 con 0x7ff9100cd920 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff91006c0a0 0x7ff91006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff91006c0a0 0x7ff91006e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53728/0 (socket says 172.21.15.67:53728) 2024-04-17T07:15:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 -- 172.21.15.67:0/3455901680 learned_addr learned my addr 172.21.15.67:0/3455901680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff90fe00640 1 --2- 172.21.15.67:0/3455901680 >> v2:172.21.15.187:3300/0 conn(0x7ff9100cd920 0x7ff91006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 -- 172.21.15.67:0/3455901680 >> v2:172.21.15.187:3300/0 conn(0x7ff9100cd920 msgr2=0x7ff91006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 --2- 172.21.15.67:0/3455901680 >> v2:172.21.15.187:3300/0 conn(0x7ff9100cd920 0x7ff91006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 -- 172.21.15.67:0/3455901680 >> v1:172.21.15.187:6789/0 conn(0x7ff910145f80 legacy=0x7ff9100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.742+0000 7ff915600640 1 -- 172.21.15.67:0/3455901680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff910072150 con 0x7ff91006c0a0 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.744+0000 7ff915600640 1 --2- 172.21.15.67:0/3455901680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff91006e1c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff91006e170 tx=0x7ff90802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487972e198e5ea9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.744+0000 7ff90f400640 1 -- 172.21.15.67:0/3455901680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90802c9f0 con 0x7ff91006c0a0 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.744+0000 7ff90f400640 1 -- 172.21.15.67:0/3455901680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff90802cb90 con 0x7ff91006c0a0 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.744+0000 7ff90f400640 1 -- 172.21.15.67:0/3455901680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9080343f0 con 0x7ff91006c0a0 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 -- 172.21.15.67:0/3455901680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 msgr2=0x7ff91006e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:35.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 --2- 172.21.15.67:0/3455901680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff91006e1c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff91006e170 tx=0x7ff90802be50 comp rx=0 tx=0).stop 2024-04-17T07:15:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 -- 172.21.15.67:0/3455901680 shutdown_connections 2024-04-17T07:15:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 --2- 172.21.15.67:0/3455901680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff91006e1c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 --2- 172.21.15.67:0/3455901680 >> v2:172.21.15.187:3300/0 conn(0x7ff9100cd920 0x7ff91006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 -- 172.21.15.67:0/3455901680 >> 172.21.15.67:0/3455901680 conn(0x7ff9100b2190 msgr2=0x7ff9100b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 -- 172.21.15.67:0/3455901680 shutdown_connections 2024-04-17T07:15:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.745+0000 7ff916000640 1 -- 172.21.15.67:0/3455901680 wait complete. 2024-04-17T07:15:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.747+0000 7ff916000640 1 Processor -- start 2024-04-17T07:15:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.747+0000 7ff916000640 1 -- start start 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9100cd920 0x7ff9101f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff910145f80 0x7ff91015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff910072c30 con 0x7ff91006c0a0 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff91006c530 con 0x7ff910145f80 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff916000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff910072db0 con 0x7ff9100cd920 2024-04-17T07:15:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff914c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff914c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53732/0 (socket says 172.21.15.67:53732) 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.748+0000 7ff914c00640 1 -- 172.21.15.67:0/1744362551 learned_addr learned my addr 172.21.15.67:0/1744362551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff90fe00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9100cd920 0x7ff9101f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff915600640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff910145f80 0x7ff91015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 -- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9100cd920 msgr2=0x7ff9101f5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9100cd920 0x7ff9101f5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 -- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff910145f80 msgr2=0x7ff91015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff910145f80 0x7ff91015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 -- 172.21.15.67:0/1744362551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff910140200 con 0x7ff91006c0a0 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff914c00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff90400b3e0 tx=0x7ff90400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:35.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904015020 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff904011850 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904011c40 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.749+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff910141f10 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.750+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9101424d0 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.751+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff910142be0 con 0x7ff91006c0a0 2024-04-17T07:15:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.752+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7ff904018020 con 0x7ff91006c0a0 2024-04-17T07:15:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.753+0000 7ff90cc00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 0x7ff8f006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:35.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.753+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7ff90408bff0 con 0x7ff91006c0a0 2024-04-17T07:15:35.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.753+0000 7ff90fe00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 0x7ff8f006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:35.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.755+0000 7ff90fe00640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 0x7ff8f006efb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff910140b90 tx=0x7ff8fc008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.757+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff90401f020 con 0x7ff91006c0a0 2024-04-17T07:15:35.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:35.991+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff910068a10 con 0x7ff8f006caf0 2024-04-17T07:15:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.192+0000 7ff90cc00640 1 -- 172.21.15.67:0/1744362551 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ff910068a10 con 0x7ff8f006caf0 2024-04-17T07:15:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir5 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.195+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 msgr2=0x7ff8f006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.195+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 0x7ff8f006efb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff910140b90 tx=0x7ff8fc008040 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.195+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 msgr2=0x7ff9101f22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.195+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff90400b3e0 tx=0x7ff90400b8a0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 shutdown_connections 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff8f006caf0 0x7ff8f006efb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff910145f80 0x7ff91015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9100cd920 0x7ff9101f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 --2- 172.21.15.67:0/1744362551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff91006c0a0 0x7ff9101f22a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 >> 172.21.15.67:0/1744362551 conn(0x7ff9100b2190 msgr2=0x7ff9100b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:36.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 shutdown_connections 2024-04-17T07:15:36.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.196+0000 7ff916000640 1 -- 172.21.15.67:0/1744362551 wait complete. 2024-04-17T07:15:36.221 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-04-17T07:15:36.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:36.222 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-04-17T07:15:36.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-04-17T07:15:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.491+0000 7facfb200640 1 Processor -- start 2024-04-17T07:15:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.491+0000 7facfb200640 1 -- start start 2024-04-17T07:15:36.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfb200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 0x7facf406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf4146360 con 0x7facf4145f80 2024-04-17T07:15:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfb200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7facf40727a0 con 0x7facf406c0a0 2024-04-17T07:15:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facf4071e70 con 0x7facf40cd920 2024-04-17T07:15:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facf9400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facf9400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:57508/0 (socket says 172.21.15.67:57508) 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facf9400640 1 -- 172.21.15.67:0/3095488579 learned_addr learned my addr 172.21.15.67:0/3095488579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.493+0000 7facfa800640 1 --2- 172.21.15.67:0/3095488579 >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 0x7facf406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.494+0000 7facf9400640 1 -- 172.21.15.67:0/3095488579 >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 msgr2=0x7facf406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.494+0000 7facf9400640 1 --2- 172.21.15.67:0/3095488579 >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 0x7facf406e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.494+0000 7facf9400640 1 -- 172.21.15.67:0/3095488579 >> v1:172.21.15.187:6789/0 conn(0x7facf4145f80 legacy=0x7facf40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.494+0000 7facf9400640 1 -- 172.21.15.67:0/3095488579 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf4072150 con 0x7facf40cd920 2024-04-17T07:15:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.494+0000 7facfa800640 1 --2- 172.21.15.67:0/3095488579 >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 0x7facf406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:15:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.496+0000 7facf9400640 1 --2- 172.21.15.67:0/3095488579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7face4009870 tx=0x7face402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1065d5bfd9d15bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.496+0000 7facf8a00640 1 -- 172.21.15.67:0/3095488579 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4035020 con 0x7facf40cd920 2024-04-17T07:15:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.496+0000 7facf8a00640 1 -- 172.21.15.67:0/3095488579 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7face4031640 con 0x7facf40cd920 2024-04-17T07:15:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.496+0000 7facf8a00640 1 -- 172.21.15.67:0/3095488579 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face402d040 con 0x7facf40cd920 2024-04-17T07:15:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 -- 172.21.15.67:0/3095488579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 msgr2=0x7facf406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 --2- 172.21.15.67:0/3095488579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7face4009870 tx=0x7face402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 -- 172.21.15.67:0/3095488579 shutdown_connections 2024-04-17T07:15:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 --2- 172.21.15.67:0/3095488579 >> v2:172.21.15.187:3300/0 conn(0x7facf406c0a0 0x7facf406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 --2- 172.21.15.67:0/3095488579 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf406bb60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 -- 172.21.15.67:0/3095488579 >> 172.21.15.67:0/3095488579 conn(0x7facf40b2190 msgr2=0x7facf40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 -- 172.21.15.67:0/3095488579 shutdown_connections 2024-04-17T07:15:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.497+0000 7facfb200640 1 -- 172.21.15.67:0/3095488579 wait complete. 2024-04-17T07:15:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.500+0000 7facfb200640 1 Processor -- start 2024-04-17T07:15:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.500+0000 7facfb200640 1 -- start start 2024-04-17T07:15:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf41f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf4145f80 0x7facf41f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7facf4072db0 con 0x7facf406c0a0 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7facf4072c30 con 0x7facf4145f80 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facfb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facf406c4a0 con 0x7facf40cd920 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.501+0000 7facf9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53746/0 (socket says 172.21.15.67:53746) 2024-04-17T07:15:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 -- 172.21.15.67:0/3330148996 learned_addr learned my addr 172.21.15.67:0/3330148996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9400640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf41f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facfa800640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf4145f80 0x7facf41f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 -- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 msgr2=0x7facf41f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf41f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 -- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf4145f80 msgr2=0x7facf41f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf4145f80 0x7facf41f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 -- 172.21.15.67:0/3330148996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf41f4330 con 0x7facf406c0a0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.502+0000 7facf9e00640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7face800c3f0 tx=0x7face800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.503+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face80052d0 con 0x7facf406c0a0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.503+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7face8005490 con 0x7facf406c0a0 2024-04-17T07:15:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.503+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8007260 con 0x7facf406c0a0 2024-04-17T07:15:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.503+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf41f4610 con 0x7facf406c0a0 2024-04-17T07:15:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.503+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf41f4bd0 con 0x7facf406c0a0 2024-04-17T07:15:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.504+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf4068a10 con 0x7facf406c0a0 2024-04-17T07:15:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.505+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7face8007400 con 0x7facf406c0a0 2024-04-17T07:15:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.506+0000 7facee000640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 0x7facd806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.506+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7face808b490 con 0x7facf406c0a0 2024-04-17T07:15:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.507+0000 7facf9400640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 0x7facd806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:36.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.508+0000 7facf9400640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 0x7facd806efb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7face4036000 tx=0x7face403c040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.511+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7face8059540 con 0x7facf406c0a0 2024-04-17T07:15:36.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.740+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7facf40d1130 con 0x7facd806caf0 2024-04-17T07:15:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.936+0000 7facee000640 1 -- 172.21.15.67:0/3330148996 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7facf40d1130 con 0x7facd806caf0 2024-04-17T07:15:36.936 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir6 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.939+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 msgr2=0x7facd806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.939+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 0x7facd806efb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7face4036000 tx=0x7face403c040 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.939+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 msgr2=0x7facf413e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.939+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7face800c3f0 tx=0x7face800c8b0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 shutdown_connections 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7facd806caf0 0x7facd806efb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facf4145f80 0x7facf41f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facf40cd920 0x7facf41f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 --2- 172.21.15.67:0/3330148996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facf406c0a0 0x7facf413e600 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 >> 172.21.15.67:0/3330148996 conn(0x7facf40b2190 msgr2=0x7facf406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:36.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 shutdown_connections 2024-04-17T07:15:36.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:36.940+0000 7facfb200640 1 -- 172.21.15.67:0/3330148996 wait complete. 2024-04-17T07:15:36.963 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-04-17T07:15:36.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:36.963 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-04-17T07:15:37.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-04-17T07:15:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.243+0000 7f8ae3e00640 1 Processor -- start 2024-04-17T07:15:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.244+0000 7f8ae3e00640 1 -- start start 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.245+0000 7f8ae3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae40b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.245+0000 7f8ae3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae40b4580 0x7f8ae4162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.245+0000 7f8ae3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ae4166a60 con 0x7f8ae406f800 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.245+0000 7f8ae3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8ae4166810 con 0x7f8ae40b4580 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.245+0000 7f8ae3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae4166d40 con 0x7f8ae40b41a0 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae40b4580 0x7f8ae4162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae40b4580 0x7f8ae4162690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53752/0 (socket says 172.21.15.67:53752) 2024-04-17T07:15:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 -- 172.21.15.67:0/2895465065 learned_addr learned my addr 172.21.15.67:0/2895465065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2a00640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae40b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 -- 172.21.15.67:0/2895465065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 msgr2=0x7f8ae40b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae40b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 -- 172.21.15.67:0/2895465065 >> v1:172.21.15.187:6789/0 conn(0x7f8ae406f800 legacy=0x7f8ae4071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.246+0000 7f8ae2000640 1 -- 172.21.15.67:0/2895465065 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae4167020 con 0x7f8ae40b4580 2024-04-17T07:15:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.247+0000 7f8ae2000640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae40b4580 0x7f8ae4162690 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009870 tx=0x7f8ad402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c56cf419d270dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.248+0000 7f8ae1600640 1 -- 172.21.15.67:0/2895465065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4035020 con 0x7f8ae40b4580 2024-04-17T07:15:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.248+0000 7f8ae1600640 1 -- 172.21.15.67:0/2895465065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad4031640 con 0x7f8ae40b4580 2024-04-17T07:15:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.248+0000 7f8ae1600640 1 -- 172.21.15.67:0/2895465065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad402d040 con 0x7f8ae40b4580 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2895465065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae40b4580 msgr2=0x7f8ae4162690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae40b4580 0x7f8ae4162690 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009870 tx=0x7f8ad402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2895465065 shutdown_connections 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae40b4580 0x7f8ae4162690 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2895465065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae40b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2895465065 >> 172.21.15.67:0/2895465065 conn(0x7f8ae40b2190 msgr2=0x7f8ae40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2895465065 shutdown_connections 2024-04-17T07:15:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.249+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2895465065 wait complete. 2024-04-17T07:15:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.251+0000 7f8ae3e00640 1 Processor -- start 2024-04-17T07:15:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.251+0000 7f8ae3e00640 1 -- start start 2024-04-17T07:15:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae41ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 0x7f8ae414d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae406eec0 con 0x7f8ae406f800 2024-04-17T07:15:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae406ed40 con 0x7f8ae40b4580 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae4162bd0 con 0x7f8ae40b41a0 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53758/0 (socket says 172.21.15.67:53758) 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.252+0000 7f8ae2a00640 1 -- 172.21.15.67:0/2264630108 learned_addr learned my addr 172.21.15.67:0/2264630108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 -- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 msgr2=0x7f8ae41ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae41ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 -- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 msgr2=0x7f8ae414d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae3400640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 0x7f8ae414d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 0x7f8ae414d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 -- 172.21.15.67:0/2264630108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae41fc460 con 0x7f8ae406f800 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae3400640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 0x7f8ae414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae2a00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000d800 tx=0x7f8ad000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000b6c0 con 0x7f8ae406f800 2024-04-17T07:15:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad0019070 con 0x7f8ae406f800 2024-04-17T07:15:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae41fc740 con 0x7f8ae406f800 2024-04-17T07:15:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000ba80 con 0x7f8ae406f800 2024-04-17T07:15:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.253+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae41fcd00 con 0x7f8ae406f800 2024-04-17T07:15:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.255+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae4068a10 con 0x7f8ae406f800 2024-04-17T07:15:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.256+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f8ad000bc20 con 0x7f8ae406f800 2024-04-17T07:15:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.257+0000 7f8adaa00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 0x7f8ac406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.257+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f8ad008c560 con 0x7f8ae406f800 2024-04-17T07:15:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.258+0000 7f8ae2000640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 0x7f8ac406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:37.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.259+0000 7f8ae2000640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 0x7f8ac406efb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4036000 tx=0x7f8ad4030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.262+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8ad005a570 con 0x7f8ae406f800 2024-04-17T07:15:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.492+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8ae406c7b0 con 0x7f8ac406caf0 2024-04-17T07:15:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.701+0000 7f8adaa00640 1 -- 172.21.15.67:0/2264630108 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8ae406c7b0 con 0x7f8ac406caf0 2024-04-17T07:15:37.702 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir7 2024-04-17T07:15:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 msgr2=0x7f8ac406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 0x7f8ac406efb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4036000 tx=0x7f8ad4030000 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 msgr2=0x7f8ae4162110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000d800 tx=0x7f8ad000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 shutdown_connections 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f8ac406caf0 0x7f8ac406efb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ae40b4580 0x7f8ae414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ae40b41a0 0x7f8ae41ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 --2- 172.21.15.67:0/2264630108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae406f800 0x7f8ae4162110 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 >> 172.21.15.67:0/2264630108 conn(0x7f8ae40b2190 msgr2=0x7f8ae4071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 shutdown_connections 2024-04-17T07:15:37.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:37.705+0000 7f8ae3e00640 1 -- 172.21.15.67:0/2264630108 wait complete. 2024-04-17T07:15:37.728 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-04-17T07:15:37.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:37.728 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-04-17T07:15:37.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-04-17T07:15:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.014+0000 7fc29ac00640 1 Processor -- start 2024-04-17T07:15:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.014+0000 7fc29ac00640 1 -- start start 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc294145f80 0x7fc2940cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc29406c270 0x7fc29406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc294146360 con 0x7fc2940cd750 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc2940727a0 con 0x7fc294145f80 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc294071e70 con 0x7fc29406c270 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc299800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc294145f80 0x7fc2940cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc298e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc2940cd750 0x7fc29406bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33708/0 (socket says 172.21.15.67:33708) 2024-04-17T07:15:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc298e00640 1 -- 172.21.15.67:0/735940968 learned_addr learned my addr 172.21.15.67:0/735940968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.016+0000 7fc29a200640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc29406c270 0x7fc29406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.017+0000 7fc299800640 1 -- 172.21.15.67:0/735940968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc29406c270 msgr2=0x7fc29406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.017+0000 7fc299800640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc29406c270 0x7fc29406e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.017+0000 7fc293e00640 1 -- 172.21.15.67:0/735940968 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1928013174 0 0) 0x7fc294146360 con 0x7fc2940cd750 2024-04-17T07:15:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.017+0000 7fc299800640 1 -- 172.21.15.67:0/735940968 >> v1:172.21.15.67:6789/0 conn(0x7fc2940cd750 legacy=0x7fc29406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.017+0000 7fc299800640 1 -- 172.21.15.67:0/735940968 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc294072150 con 0x7fc294145f80 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.018+0000 7fc299800640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc294145f80 0x7fc2940cd210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc2880097a0 tx=0x7fc28802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b6d10493ce7332 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.018+0000 7fc293e00640 1 -- 172.21.15.67:0/735940968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288031780 con 0x7fc294145f80 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.018+0000 7fc293e00640 1 -- 172.21.15.67:0/735940968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc288037070 con 0x7fc294145f80 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.018+0000 7fc293e00640 1 -- 172.21.15.67:0/735940968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288031b40 con 0x7fc294145f80 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 -- 172.21.15.67:0/735940968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc294145f80 msgr2=0x7fc2940cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc294145f80 0x7fc2940cd210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc2880097a0 tx=0x7fc28802b820 comp rx=0 tx=0).stop 2024-04-17T07:15:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 -- 172.21.15.67:0/735940968 shutdown_connections 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc29406c270 0x7fc29406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 --2- 172.21.15.67:0/735940968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc294145f80 0x7fc2940cd210 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 -- 172.21.15.67:0/735940968 >> 172.21.15.67:0/735940968 conn(0x7fc2940b2190 msgr2=0x7fc2940b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 -- 172.21.15.67:0/735940968 shutdown_connections 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.019+0000 7fc29ac00640 1 -- 172.21.15.67:0/735940968 wait complete. 2024-04-17T07:15:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.021+0000 7fc29ac00640 1 Processor -- start 2024-04-17T07:15:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.021+0000 7fc29ac00640 1 -- start start 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 0x7fc2941f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 0x7fc2940a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc294072db0 con 0x7fc2940cd750 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc294072c30 con 0x7fc29406c270 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc29406e700 con 0x7fc294145f80 2024-04-17T07:15:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc298e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc298e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53780/0 (socket says 172.21.15.67:53780) 2024-04-17T07:15:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc298e00640 1 -- 172.21.15.67:0/948579483 learned_addr learned my addr 172.21.15.67:0/948579483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.022+0000 7fc29a200640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 0x7fc2940a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc299800640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 0x7fc2941f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 -- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 msgr2=0x7fc2940a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 0x7fc2940a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 -- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 msgr2=0x7fc2941f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 0x7fc2941f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 -- 172.21.15.67:0/948579483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2940cae50 con 0x7fc2940cd750 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc29a200640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 0x7fc2940a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc299800640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 0x7fc2941f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc298e00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc28000c4d0 tx=0x7fc28000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280018070 con 0x7fc2940cd750 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc280007500 con 0x7fc2940cd750 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc280004a20 con 0x7fc2940cd750 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.023+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2940cb0d0 con 0x7fc2940cd750 2024-04-17T07:15:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.024+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2940cb970 con 0x7fc2940cd750 2024-04-17T07:15:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.025+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc294068a10 con 0x7fc2940cd750 2024-04-17T07:15:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.026+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fc28001b070 con 0x7fc2940cd750 2024-04-17T07:15:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.026+0000 7fc291600640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 0x7fc27406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.027+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fc28008b2e0 con 0x7fc2940cd750 2024-04-17T07:15:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.027+0000 7fc299800640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 0x7fc27406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.028+0000 7fc299800640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 0x7fc27406efb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc288036000 tx=0x7fc288008030 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.031+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc2940cae50 con 0x7fc2940cd750 2024-04-17T07:15:38.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.264+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc2940ce230 con 0x7fc27406caf0 2024-04-17T07:15:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.469+0000 7fc291600640 1 -- 172.21.15.67:0/948579483 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc2940ce230 con 0x7fc27406caf0 2024-04-17T07:15:38.469 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir8 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 msgr2=0x7fc27406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 0x7fc27406efb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc288036000 tx=0x7fc288008030 comp rx=0 tx=0).stop 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 msgr2=0x7fc2940ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc28000c4d0 tx=0x7fc28000c990 comp rx=0 tx=0).stop 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 shutdown_connections 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc27406caf0 0x7fc27406efb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc294145f80 0x7fc2940a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2940cd750 0x7fc2940ca910 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 --2- 172.21.15.67:0/948579483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc29406c270 0x7fc2941f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 >> 172.21.15.67:0/948579483 conn(0x7fc2940b2190 msgr2=0x7fc29415db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:38.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 shutdown_connections 2024-04-17T07:15:38.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.473+0000 7fc29ac00640 1 -- 172.21.15.67:0/948579483 wait complete. 2024-04-17T07:15:38.495 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-04-17T07:15:38.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:15:38.495 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-04-17T07:15:38.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-04-17T07:15:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.772+0000 7fc6af200640 1 Processor -- start 2024-04-17T07:15:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.773+0000 7fc6af200640 1 -- start start 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.774+0000 7fc6af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6b00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6af200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc6b0162da0 0x7fc6b006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6af200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b0166a60 con 0x7fc6b00b41a0 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6af200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b0166810 con 0x7fc6b0162da0 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b0166d40 con 0x7fc6b00b4580 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6ade00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6b00b41a0 0x7fc6b0162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33710/0 (socket says 172.21.15.67:33710) 2024-04-17T07:15:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6ade00640 1 -- 172.21.15.67:0/669086181 learned_addr learned my addr 172.21.15.67:0/669086181 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6ae800640 1 --2- 172.21.15.67:0/669086181 >> v2:172.21.15.187:3300/0 conn(0x7fc6b0162da0 0x7fc6b006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6b00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001131553 0 0) 0x7fc6b0166a60 con 0x7fc6b00b41a0 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.775+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc694003520 con 0x7fc6b00b41a0 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2036974424 0 0) 0x7fc694003520 con 0x7fc6b00b41a0 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 msgr2=0x7fc6b00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6b00b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 >> v2:172.21.15.187:3300/0 conn(0x7fc6b0162da0 msgr2=0x7fc6b006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 --2- 172.21.15.67:0/669086181 >> v2:172.21.15.187:3300/0 conn(0x7fc6b0162da0 0x7fc6b006dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b0167020 con 0x7fc6b00b41a0 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc6a0001670 con 0x7fc6b00b41a0 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 >> v1:172.21.15.67:6789/0 conn(0x7fc6b00b41a0 legacy=0x7fc6b0162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6940039b0 0x7fc694003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc694004800 0x7fc694004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0166810 con 0x7fc6940039b0 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0166d40 con 0x7fc694004800 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc69400b9c0 con 0x7fc6b00b4580 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.776+0000 7fc6ade00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6940039b0 0x7fc694003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc694004800 0x7fc694004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ae800640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ade00640 1 -- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 msgr2=0x7fc6940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ade00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6b00b4580 0x7fc6940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ade00640 1 -- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc694004800 msgr2=0x7fc694004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ade00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc694004800 0x7fc694004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.777+0000 7fc6ade00640 1 -- 172.21.15.67:0/669086181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc694003520 con 0x7fc6940039b0 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.778+0000 7fc6ade00640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6940039b0 0x7fc694003dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c160 tx=0x7fc694005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233f6b7aa455685d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0007c80 con 0x7fc6940039b0 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a0030070 con 0x7fc6940039b0 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6ad400640 1 -- 172.21.15.67:0/669086181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0005cf0 con 0x7fc6940039b0 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 -- 172.21.15.67:0/669086181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6940039b0 msgr2=0x7fc694003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 --2- 172.21.15.67:0/669086181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6940039b0 0x7fc694003dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c160 tx=0x7fc694005830 comp rx=0 tx=0).stop 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6ae800640 1 -- 172.21.15.67:0/669086181 reap_dead start 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 -- 172.21.15.67:0/669086181 shutdown_connections 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 -- 172.21.15.67:0/669086181 >> 172.21.15.67:0/669086181 conn(0x7fc6b00b2190 msgr2=0x7fc6b00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 -- 172.21.15.67:0/669086181 shutdown_connections 2024-04-17T07:15:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.779+0000 7fc6af200640 1 -- 172.21.15.67:0/669086181 wait complete. 2024-04-17T07:15:38.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.781+0000 7fc6af200640 1 Processor -- start 2024-04-17T07:15:38.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.781+0000 7fc6af200640 1 -- start start 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6940039b0 0x7fc6b0159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6b0158150 0x7fc6b015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b006f980 con 0x7fc6b01568b0 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b006f800 con 0x7fc6b0158150 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b006fb00 con 0x7fc6940039b0 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.782+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53796/0 (socket says 172.21.15.67:53796) 2024-04-17T07:15:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 -- 172.21.15.67:0/3993328902 learned_addr learned my addr 172.21.15.67:0/3993328902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 -- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6940039b0 msgr2=0x7fc6b0159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6940039b0 0x7fc6b0159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6ae800640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6b0158150 0x7fc6b015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 -- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6b0158150 msgr2=0x7fc6b015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6b0158150 0x7fc6b015acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 -- 172.21.15.67:0/3993328902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b0158a30 con 0x7fc6b01568b0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.783+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc698011540 tx=0x7fc698011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.784+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698015930 con 0x7fc6b01568b0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.784+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc698015ad0 con 0x7fc6b01568b0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.784+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69800f460 con 0x7fc6b01568b0 2024-04-17T07:15:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.784+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b0158d10 con 0x7fc6b01568b0 2024-04-17T07:15:38.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.784+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b01592d0 con 0x7fc6b01568b0 2024-04-17T07:15:38.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.785+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b0068a10 con 0x7fc6b01568b0 2024-04-17T07:15:38.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.786+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fc69800f600 con 0x7fc6b01568b0 2024-04-17T07:15:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.787+0000 7fc6a6000640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 0x7fc68c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.787+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fc69808c7a0 con 0x7fc6b01568b0 2024-04-17T07:15:38.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.788+0000 7fc6ade00640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 0x7fc68c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:38.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.789+0000 7fc6ade00640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 0x7fc68c06efe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0034000 tx=0x7fc6a0039040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:38.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:38.792+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc698019030 con 0x7fc6b01568b0 2024-04-17T07:15:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.019+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc6b0071e50 con 0x7fc68c06cb20 2024-04-17T07:15:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.224+0000 7fc6a6000640 1 -- 172.21.15.67:0/3993328902 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc6b0071e50 con 0x7fc68c06cb20 2024-04-17T07:15:39.224 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/dir9 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.227+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 msgr2=0x7fc68c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.227+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 0x7fc68c06efe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0034000 tx=0x7fc6a0039040 comp rx=0 tx=0).stop 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.227+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 msgr2=0x7fc6b014c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.227+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc698011540 tx=0x7fc698011a00 comp rx=0 tx=0).stop 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.227+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 shutdown_connections 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc68c06cb20 0x7fc68c06efe0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6b0158150 0x7fc6b015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6b01568b0 0x7fc6b014c540 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 --2- 172.21.15.67:0/3993328902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6940039b0 0x7fc6b0159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 >> 172.21.15.67:0/3993328902 conn(0x7fc6b00b2190 msgr2=0x7fc6b0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 shutdown_connections 2024-04-17T07:15:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.228+0000 7fc6af200640 1 -- 172.21.15.67:0/3993328902 wait complete. 2024-04-17T07:15:39.251 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-04-17T07:15:39.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-04-17T07:15:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.479+0000 7f4807e00640 1 Processor -- start 2024-04-17T07:15:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.479+0000 7f4807e00640 1 -- start start 2024-04-17T07:15:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48080cd920 0x7f480806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f480806c0a0 0x7f480806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4808146360 con 0x7f4808145f80 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f48080727a0 con 0x7f480806c0a0 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4808071e70 con 0x7f48080cd920 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f480806c0a0 0x7f480806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4806a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4808145f80 0x7f48080cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33714/0 (socket says 172.21.15.67:33714) 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4806a00640 1 -- 172.21.15.67:0/3959048474 learned_addr learned my addr 172.21.15.67:0/3959048474 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.481+0000 7f4807400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f480806c0a0 0x7f480806e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53798/0 (socket says 172.21.15.67:53798) 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.482+0000 7f4806000640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48080cd920 0x7f480806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.482+0000 7f4807400640 1 -- 172.21.15.67:0/3959048474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48080cd920 msgr2=0x7f480806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.482+0000 7f4807400640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48080cd920 0x7f480806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.482+0000 7f4807400640 1 -- 172.21.15.67:0/3959048474 >> v1:172.21.15.67:6789/0 conn(0x7f4808145f80 legacy=0x7f48080cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.482+0000 7f4807400640 1 -- 172.21.15.67:0/3959048474 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4808072150 con 0x7f480806c0a0 2024-04-17T07:15:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.483+0000 7f4807400640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f480806e1c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f48000097a0 tx=0x7f480002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a8b27b941b8464 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.483+0000 7f4805600640 1 -- 172.21.15.67:0/3959048474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800031780 con 0x7f480806c0a0 2024-04-17T07:15:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.483+0000 7f4805600640 1 -- 172.21.15.67:0/3959048474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4800037070 con 0x7f480806c0a0 2024-04-17T07:15:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.483+0000 7f4805600640 1 -- 172.21.15.67:0/3959048474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800031b40 con 0x7f480806c0a0 2024-04-17T07:15:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 -- 172.21.15.67:0/3959048474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 msgr2=0x7f480806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f480806e1c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f48000097a0 tx=0x7f480002b820 comp rx=0 tx=0).stop 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 -- 172.21.15.67:0/3959048474 shutdown_connections 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f480806e1c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 --2- 172.21.15.67:0/3959048474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48080cd920 0x7f480806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 -- 172.21.15.67:0/3959048474 >> 172.21.15.67:0/3959048474 conn(0x7f48080b2190 msgr2=0x7f48080b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 -- 172.21.15.67:0/3959048474 shutdown_connections 2024-04-17T07:15:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.484+0000 7f4807e00640 1 -- 172.21.15.67:0/3959048474 wait complete. 2024-04-17T07:15:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.486+0000 7f4807e00640 1 Processor -- start 2024-04-17T07:15:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.486+0000 7f4807e00640 1 -- start start 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48080cd920 0x7f48080c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4808145f80 0x7f48080c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4808072db0 con 0x7f480806c0a0 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4808072c30 con 0x7f48080cd920 2024-04-17T07:15:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4807e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f480806c4a0 con 0x7f4808145f80 2024-04-17T07:15:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4806a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4806a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53800/0 (socket says 172.21.15.67:53800) 2024-04-17T07:15:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.487+0000 7f4806a00640 1 -- 172.21.15.67:0/1059326580 learned_addr learned my addr 172.21.15.67:0/1059326580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4807400640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4808145f80 0x7f48080c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806000640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48080cd920 0x7f48080c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 -- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4808145f80 msgr2=0x7f48080c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4808145f80 0x7f48080c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 -- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48080cd920 msgr2=0x7f48080c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48080cd920 0x7f48080c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 -- 172.21.15.67:0/1059326580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48080b4b80 con 0x7f480806c0a0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f4806a00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f47f800c3c0 tx=0x7f47f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.488+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8017070 con 0x7f480806c0a0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.489+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f8005ce0 con 0x7f480806c0a0 2024-04-17T07:15:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.489+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f80072f0 con 0x7f480806c0a0 2024-04-17T07:15:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.489+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48080b4e60 con 0x7f480806c0a0 2024-04-17T07:15:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.489+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48080adc90 con 0x7f480806c0a0 2024-04-17T07:15:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.490+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48080b3280 con 0x7f480806c0a0 2024-04-17T07:15:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.491+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f47f801a070 con 0x7f480806c0a0 2024-04-17T07:15:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.492+0000 7f47fea00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 0x7f47e806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.492+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f47f808b3d0 con 0x7f480806c0a0 2024-04-17T07:15:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.493+0000 7f4806000640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 0x7f47e806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.494+0000 7f4806000640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 0x7f47e806efb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f48081e8460 tx=0x7f47f400b040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.497+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f48080b3280 con 0x7f480806c0a0 2024-04-17T07:15:39.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.729+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f48080cc250 con 0x7f47e806caf0 2024-04-17T07:15:39.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.883+0000 7f47fea00640 1 -- 172.21.15.67:0/1059326580 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f48080cc250 con 0x7f47e806caf0 2024-04-17T07:15:39.884 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 msgr2=0x7f47e806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 0x7f47e806efb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f48081e8460 tx=0x7f47f400b040 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 msgr2=0x7f48081f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f47f800c3c0 tx=0x7f47f800c880 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 shutdown_connections 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f47e806caf0 0x7f47e806efb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4808145f80 0x7f48080c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48080cd920 0x7f48080c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 --2- 172.21.15.67:0/1059326580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f480806c0a0 0x7f48081f4190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 >> 172.21.15.67:0/1059326580 conn(0x7f48080b2190 msgr2=0x7f48080cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 shutdown_connections 2024-04-17T07:15:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:39.887+0000 7f4807e00640 1 -- 172.21.15.67:0/1059326580 wait complete. 2024-04-17T07:15:39.907 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-04-17T07:15:39.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-04-17T07:15:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.131+0000 7f53d9600640 1 Processor -- start 2024-04-17T07:15:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.132+0000 7f53d9600640 1 -- start start 2024-04-17T07:15:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.133+0000 7f53d9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53d40b41a0 0x7f53d40b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.133+0000 7f53d9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b4580 0x7f53d4162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.133+0000 7f53d9600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d4166a60 con 0x7f53d406f800 2024-04-17T07:15:40.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.133+0000 7f53d9600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f53d4166810 con 0x7f53d40b41a0 2024-04-17T07:15:40.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.133+0000 7f53d9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53d4166d40 con 0x7f53d40b4580 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53d40b41a0 0x7f53d40b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53d40b41a0 0x7f53d40b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53812/0 (socket says 172.21.15.67:53812) 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 -- 172.21.15.67:0/2078602578 learned_addr learned my addr 172.21.15.67:0/2078602578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53cbe00640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b4580 0x7f53d4162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 -- 172.21.15.67:0/2078602578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b4580 msgr2=0x7f53d4162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b4580 0x7f53d4162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 -- 172.21.15.67:0/2078602578 >> v1:172.21.15.187:6789/0 conn(0x7f53d406f800 legacy=0x7f53d4071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.134+0000 7f53d3e00640 1 -- 172.21.15.67:0/2078602578 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d4167020 con 0x7f53d40b41a0 2024-04-17T07:15:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.136+0000 7f53d3e00640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b41a0 0x7f53d40b3490 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a63b64e874f2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.136+0000 7f53d3400640 1 -- 172.21.15.67:0/2078602578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4035020 con 0x7f53d40b41a0 2024-04-17T07:15:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.136+0000 7f53d3400640 1 -- 172.21.15.67:0/2078602578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c4031640 con 0x7f53d40b41a0 2024-04-17T07:15:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.136+0000 7f53d3400640 1 -- 172.21.15.67:0/2078602578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c402d040 con 0x7f53d40b41a0 2024-04-17T07:15:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 -- 172.21.15.67:0/2078602578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b41a0 msgr2=0x7f53d40b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b41a0 0x7f53d40b3490 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 -- 172.21.15.67:0/2078602578 shutdown_connections 2024-04-17T07:15:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b4580 0x7f53d4162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 --2- 172.21.15.67:0/2078602578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b41a0 0x7f53d40b3490 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 -- 172.21.15.67:0/2078602578 >> 172.21.15.67:0/2078602578 conn(0x7f53d40b2190 msgr2=0x7f53d40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 -- 172.21.15.67:0/2078602578 shutdown_connections 2024-04-17T07:15:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.137+0000 7f53d9600640 1 -- 172.21.15.67:0/2078602578 wait complete. 2024-04-17T07:15:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.139+0000 7f53d9600640 1 Processor -- start 2024-04-17T07:15:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.140+0000 7f53d9600640 1 -- start start 2024-04-17T07:15:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d406f800 0x7f53d415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 0x7f53d4159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53d406eec0 con 0x7f53d40b4580 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53d406ed40 con 0x7f53d406f800 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53d4162bd0 con 0x7f53d40b41a0 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53824/0 (socket says 172.21.15.67:53824) 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.141+0000 7f53d8c00640 1 -- 172.21.15.67:0/2858379884 learned_addr learned my addr 172.21.15.67:0/2858379884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d3e00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d406f800 0x7f53d415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53cbe00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 0x7f53d4159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 -- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 msgr2=0x7f53d4159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 0x7f53d4159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 -- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d406f800 msgr2=0x7f53d415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d406f800 0x7f53d415dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 -- 172.21.15.67:0/2858379884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d41595f0 con 0x7f53d40b4580 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53cbe00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 0x7f53d4159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.142+0000 7f53d8c00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00c780 tx=0x7f53cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.143+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc004b60 con 0x7f53d40b4580 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.143+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53cc004d00 con 0x7f53d40b4580 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.143+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53cc00aa90 con 0x7f53d40b4580 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.143+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d4158a70 con 0x7f53d40b4580 2024-04-17T07:15:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.143+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d4159030 con 0x7f53d40b4580 2024-04-17T07:15:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.145+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d4068a10 con 0x7f53d40b4580 2024-04-17T07:15:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.145+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f53cc00ac30 con 0x7f53d40b4580 2024-04-17T07:15:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.146+0000 7f53d0c00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 0x7f53b006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.146+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f53cc08c310 con 0x7f53d40b4580 2024-04-17T07:15:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.147+0000 7f53d3e00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 0x7f53b006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.148+0000 7f53d3e00640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 0x7f53b006efb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53c4036000 tx=0x7f53c403c040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.151+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f53cc020070 con 0x7f53d40b4580 2024-04-17T07:15:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.380+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f53d415bf40 con 0x7f53b006caf0 2024-04-17T07:15:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.537+0000 7f53d0c00640 1 -- 172.21.15.67:0/2858379884 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f53d415bf40 con 0x7f53b006caf0 2024-04-17T07:15:40.538 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-04-17T07:15:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 msgr2=0x7f53b006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 0x7f53b006efb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53c4036000 tx=0x7f53c403c040 comp rx=0 tx=0).stop 2024-04-17T07:15:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 msgr2=0x7f53d415a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00c780 tx=0x7f53cc00cc40 comp rx=0 tx=0).stop 2024-04-17T07:15:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 shutdown_connections 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f53b006caf0 0x7f53b006efb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53d40b4580 0x7f53d415a2c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d40b41a0 0x7f53d4159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 --2- 172.21.15.67:0/2858379884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d406f800 0x7f53d415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 >> 172.21.15.67:0/2858379884 conn(0x7f53d40b2190 msgr2=0x7f53d4161220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 shutdown_connections 2024-04-17T07:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.541+0000 7f53d9600640 1 -- 172.21.15.67:0/2858379884 wait complete. 2024-04-17T07:15:40.564 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-04-17T07:15:40.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-04-17T07:15:40.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.794+0000 7f82d8000640 1 Processor -- start 2024-04-17T07:15:40.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.795+0000 7f82d8000640 1 -- start start 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.796+0000 7f82d8000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82d00cffe0 0x7f82d006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.796+0000 7f82d8000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d01634a0 0x7f82d0163930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.796+0000 7f82d8000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d0166c30 con 0x7f82d0071f30 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.796+0000 7f82d8000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f82d006fde0 con 0x7f82d00cffe0 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.796+0000 7f82d8000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82d0163e70 con 0x7f82d01634a0 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82d00cffe0 0x7f82d006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82d00cffe0 0x7f82d006dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53840/0 (socket says 172.21.15.67:53840) 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 -- 172.21.15.67:0/2207964569 learned_addr learned my addr 172.21.15.67:0/2207964569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d4c00640 1 --1- 172.21.15.67:0/2207964569 >> v1:172.21.15.67:6789/0 conn(0x7f82d0071f30 0x7f82d0167110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33722/0 (socket says 172.21.15.67:33722) 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d6000640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d01634a0 0x7f82d0163930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 -- 172.21.15.67:0/2207964569 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d01634a0 msgr2=0x7f82d0163930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d01634a0 0x7f82d0163930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 -- 172.21.15.67:0/2207964569 >> v1:172.21.15.67:6789/0 conn(0x7f82d0071f30 legacy=0x7f82d0167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82d5600640 1 -- 172.21.15.67:0/2207964569 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d006f820 con 0x7f82d00cffe0 2024-04-17T07:15:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.797+0000 7f82cfe00640 1 -- 172.21.15.67:0/2207964569 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193925332 0 0) 0x7f82d0166c30 con 0x7f82d0071f30 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82d5600640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cffe0 0x7f82d006dd10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009870 tx=0x7f82c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830e33dea6a82bca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82cfe00640 1 -- 172.21.15.67:0/2207964569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4034070 con 0x7f82d00cffe0 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82cfe00640 1 -- 172.21.15.67:0/2207964569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c4030840 con 0x7f82d00cffe0 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82cfe00640 1 -- 172.21.15.67:0/2207964569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4030c10 con 0x7f82d00cffe0 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82d8000640 1 -- 172.21.15.67:0/2207964569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cffe0 msgr2=0x7f82d006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.799+0000 7f82d8000640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cffe0 0x7f82d006dd10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009870 tx=0x7f82c402b980 comp rx=0 tx=0).stop 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 -- 172.21.15.67:0/2207964569 shutdown_connections 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d01634a0 0x7f82d0163930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 --2- 172.21.15.67:0/2207964569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cffe0 0x7f82d006dd10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 -- 172.21.15.67:0/2207964569 >> 172.21.15.67:0/2207964569 conn(0x7f82d0067ee0 msgr2=0x7f82d006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 -- 172.21.15.67:0/2207964569 shutdown_connections 2024-04-17T07:15:40.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.800+0000 7f82d8000640 1 -- 172.21.15.67:0/2207964569 wait complete. 2024-04-17T07:15:40.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.801+0000 7f82d8000640 1 Processor -- start 2024-04-17T07:15:40.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.802+0000 7f82d8000640 1 -- start start 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.802+0000 7f82d8000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.802+0000 7f82d8000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 0x7f82d015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d8000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82d01634a0 0x7f82d00b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d8000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f82d0167820 con 0x7f82d0071f30 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d8000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f82d006f4f0 con 0x7f82d01634a0 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d8000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82d0158e40 con 0x7f82d00cffe0 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53852/0 (socket says 172.21.15.67:53852) 2024-04-17T07:15:40.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 -- 172.21.15.67:0/569567249 learned_addr learned my addr 172.21.15.67:0/569567249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:40.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d4c00640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 0x7f82d015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d6000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82d01634a0 0x7f82d00b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 -- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 msgr2=0x7f82d015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 0x7f82d015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 -- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82d01634a0 msgr2=0x7f82d00b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82d01634a0 0x7f82d00b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d5600640 1 -- 172.21.15.67:0/569567249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d01590b0 con 0x7f82d0071f30 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.803+0000 7f82d4c00640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 0x7f82d015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82d5600640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f82c4000c00 tx=0x7f82c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4048070 con 0x7f82d0071f30 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c4030640 con 0x7f82d0071f30 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4034040 con 0x7f82d0071f30 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d0159350 con 0x7f82d0071f30 2024-04-17T07:15:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.804+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d015ccb0 con 0x7f82d0071f30 2024-04-17T07:15:40.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.805+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d0159870 con 0x7f82d0071f30 2024-04-17T07:15:40.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.806+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f82c404b070 con 0x7f82d0071f30 2024-04-17T07:15:40.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.807+0000 7f82cd600640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 0x7f82b006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:40.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.807+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f82c40ae140 con 0x7f82d0071f30 2024-04-17T07:15:40.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.808+0000 7f82d4c00640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 0x7f82b006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.809+0000 7f82d4c00640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 0x7f82b006efb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f82d00cf6f0 tx=0x7f82bc009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:40.812+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f82c407bfb0 con 0x7f82d0071f30 2024-04-17T07:15:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.047+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f82d0158360 con 0x7f82b006caf0 2024-04-17T07:15:41.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.205+0000 7f82cd600640 1 -- 172.21.15.67:0/569567249 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f82d0158360 con 0x7f82b006caf0 2024-04-17T07:15:41.205 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 msgr2=0x7f82b006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 0x7f82b006efb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f82d00cf6f0 tx=0x7f82bc009040 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 msgr2=0x7f82d0157e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f82c4000c00 tx=0x7f82c4004730 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 shutdown_connections 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f82b006caf0 0x7f82b006efb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82d01634a0 0x7f82d00b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82d00cffe0 0x7f82d015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 --2- 172.21.15.67:0/569567249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0071f30 0x7f82d0157e20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.208+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 >> 172.21.15.67:0/569567249 conn(0x7f82d0067ee0 msgr2=0x7f82d01607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.209+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 shutdown_connections 2024-04-17T07:15:41.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.209+0000 7f82d8000640 1 -- 172.21.15.67:0/569567249 wait complete. 2024-04-17T07:15:41.232 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-04-17T07:15:41.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-04-17T07:15:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.457+0000 7fb1e7200640 1 Processor -- start 2024-04-17T07:15:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.457+0000 7fb1e7200640 1 -- start start 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.458+0000 7fb1e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.458+0000 7fb1e7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e006c270 0x7fb1e006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.458+0000 7fb1e7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e0146360 con 0x7fb1e00cd750 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e00cdbb0 con 0x7fb1e006c270 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e00727a0 con 0x7fb1e0145f80 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e006c270 0x7fb1e006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1e006c270 0x7fb1e006e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53866/0 (socket says 172.21.15.67:53866) 2024-04-17T07:15:41.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e6800640 1 -- 172.21.15.67:0/391829812 learned_addr learned my addr 172.21.15.67:0/391829812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e5e00640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e6800640 1 -- 172.21.15.67:0/391829812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 msgr2=0x7fb1e00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.459+0000 7fb1e6800640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.460+0000 7fb1e6800640 1 -- 172.21.15.67:0/391829812 >> v1:172.21.15.187:6789/0 conn(0x7fb1e00cd750 legacy=0x7fb1e006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.460+0000 7fb1e6800640 1 -- 172.21.15.67:0/391829812 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e0071e70 con 0x7fb1e006c270 2024-04-17T07:15:41.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.461+0000 7fb1e6800640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e006c270 0x7fb1e006e1c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e50 tx=0x7fb1dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d521f915fe4bad78 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:41.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.461+0000 7fb1e4a00640 1 -- 172.21.15.67:0/391829812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc02c9e0 con 0x7fb1e006c270 2024-04-17T07:15:41.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.461+0000 7fb1e4a00640 1 -- 172.21.15.67:0/391829812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc02cb80 con 0x7fb1e006c270 2024-04-17T07:15:41.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.461+0000 7fb1e4a00640 1 -- 172.21.15.67:0/391829812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0343f0 con 0x7fb1e006c270 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 -- 172.21.15.67:0/391829812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e006c270 msgr2=0x7fb1e006e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e006c270 0x7fb1e006e1c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009e50 tx=0x7fb1dc02be40 comp rx=0 tx=0).stop 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 -- 172.21.15.67:0/391829812 shutdown_connections 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e006c270 0x7fb1e006e1c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 --2- 172.21.15.67:0/391829812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 -- 172.21.15.67:0/391829812 >> 172.21.15.67:0/391829812 conn(0x7fb1e00b2190 msgr2=0x7fb1e00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 -- 172.21.15.67:0/391829812 shutdown_connections 2024-04-17T07:15:41.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.462+0000 7fb1e7200640 1 -- 172.21.15.67:0/391829812 wait complete. 2024-04-17T07:15:41.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.464+0000 7fb1e7200640 1 Processor -- start 2024-04-17T07:15:41.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.464+0000 7fb1e7200640 1 -- start start 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1e006c270 0x7fb1e015c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e013e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e0072c30 con 0x7fb1e00cd750 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e0072430 con 0x7fb1e006c270 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e0072db0 con 0x7fb1e0145f80 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53876/0 (socket says 172.21.15.67:53876) 2024-04-17T07:15:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 -- 172.21.15.67:0/3358348079 learned_addr learned my addr 172.21.15.67:0/3358348079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:41.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5e00640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1e006c270 0x7fb1e015c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e6800640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e013e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 -- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 msgr2=0x7fb1e013e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e013e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 -- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1e006c270 msgr2=0x7fb1e015c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1e006c270 0x7fb1e015c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.465+0000 7fb1e5400640 1 -- 172.21.15.67:0/3358348079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e01f7e70 con 0x7fb1e00cd750 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1e5400640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000c430 tx=0x7fb1d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0011790 con 0x7fb1e00cd750 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d0017070 con 0x7fb1e00cd750 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0011bb0 con 0x7fb1e00cd750 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e01f8150 con 0x7fb1e00cd750 2024-04-17T07:15:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.466+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e01f8710 con 0x7fb1e00cd750 2024-04-17T07:15:41.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.467+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e0068a10 con 0x7fb1e00cd750 2024-04-17T07:15:41.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.468+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fb1d001c460 con 0x7fb1e00cd750 2024-04-17T07:15:41.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.469+0000 7fb1da000640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 0x7fb1c406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:41.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.470+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fb1d008be80 con 0x7fb1e00cd750 2024-04-17T07:15:41.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.470+0000 7fb1e5e00640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 0x7fb1c406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:41.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.471+0000 7fb1e5e00640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 0x7fb1c406efb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4006fd0 tx=0x7fb1d4008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:41.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.474+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb1d0059f20 con 0x7fb1e00cd750 2024-04-17T07:15:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.707+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb1e00cce30 con 0x7fb1c406caf0 2024-04-17T07:15:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.867+0000 7fb1da000640 1 -- 172.21.15.67:0/3358348079 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb1e00cce30 con 0x7fb1c406caf0 2024-04-17T07:15:41.867 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 msgr2=0x7fb1c406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 0x7fb1c406efb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4006fd0 tx=0x7fb1d4008040 comp rx=0 tx=0).stop 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 msgr2=0x7fb1e0141910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000c430 tx=0x7fb1d000c8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 shutdown_connections 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb1c406caf0 0x7fb1c406efb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1e0145f80 0x7fb1e013e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1e00cd750 0x7fb1e0141910 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 --2- 172.21.15.67:0/3358348079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1e006c270 0x7fb1e015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 >> 172.21.15.67:0/3358348079 conn(0x7fb1e00b2190 msgr2=0x7fb1e00d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.871+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 shutdown_connections 2024-04-17T07:15:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:41.872+0000 7fb1e7200640 1 -- 172.21.15.67:0/3358348079 wait complete. 2024-04-17T07:15:41.893 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-04-17T07:15:41.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-04-17T07:15:42.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.115+0000 7fcc82c00640 1 Processor -- start 2024-04-17T07:15:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.116+0000 7fcc82c00640 1 -- start start 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.117+0000 7fcc82c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcc7c071f30 0x7fcc7c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.117+0000 7fcc82c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc7c1634a0 0x7fcc7c163910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.117+0000 7fcc82c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc7c166c30 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.117+0000 7fcc82c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcc7c06fde0 con 0x7fcc7c1634a0 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.117+0000 7fcc82c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcc7c163e50 con 0x7fcc7c071f30 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc7be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc7c0cffe0 0x7fcc7c167110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33732/0 (socket says 172.21.15.67:33732) 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc7be00640 1 -- 172.21.15.67:0/3996991351 learned_addr learned my addr 172.21.15.67:0/3996991351 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc80c00640 1 --2- 172.21.15.67:0/3996991351 >> v2:172.21.15.67:3300/0 conn(0x7fcc7c1634a0 0x7fcc7c163910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc7b400640 1 --2- 172.21.15.67:0/3996991351 >> v2:172.21.15.187:3300/0 conn(0x7fcc7c071f30 0x7fcc7c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc7aa00640 1 -- 172.21.15.67:0/3996991351 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215865228 0 0) 0x7fcc7c166c30 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc7aa00640 1 -- 172.21.15.67:0/3996991351 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc60003520 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc80c00640 1 -- 172.21.15.67:0/3996991351 >> v2:172.21.15.187:3300/0 conn(0x7fcc7c071f30 msgr2=0x7fcc7c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.118+0000 7fcc80c00640 1 --2- 172.21.15.67:0/3996991351 >> v2:172.21.15.187:3300/0 conn(0x7fcc7c071f30 0x7fcc7c06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.119+0000 7fcc80c00640 1 -- 172.21.15.67:0/3996991351 >> v1:172.21.15.67:6789/0 conn(0x7fcc7c0cffe0 legacy=0x7fcc7c167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.119+0000 7fcc80c00640 1 -- 172.21.15.67:0/3996991351 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c06f820 con 0x7fcc7c1634a0 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.120+0000 7fcc80c00640 1 --2- 172.21.15.67:0/3996991351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c1634a0 0x7fcc7c163910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0727c0 tx=0x7fcc7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53a5b8d1fbb5b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.120+0000 7fcc7aa00640 1 -- 172.21.15.67:0/3996991351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74039070 con 0x7fcc7c1634a0 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.120+0000 7fcc7aa00640 1 -- 172.21.15.67:0/3996991351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc7402cc80 con 0x7fcc7c1634a0 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.120+0000 7fcc7aa00640 1 -- 172.21.15.67:0/3996991351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74034660 con 0x7fcc7c1634a0 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 -- 172.21.15.67:0/3996991351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c1634a0 msgr2=0x7fcc7c163910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 --2- 172.21.15.67:0/3996991351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c1634a0 0x7fcc7c163910 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0727c0 tx=0x7fcc7402bed0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 -- 172.21.15.67:0/3996991351 shutdown_connections 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 --2- 172.21.15.67:0/3996991351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c1634a0 0x7fcc7c163910 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 --2- 172.21.15.67:0/3996991351 >> v2:172.21.15.187:3300/0 conn(0x7fcc7c071f30 0x7fcc7c06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 -- 172.21.15.67:0/3996991351 >> 172.21.15.67:0/3996991351 conn(0x7fcc7c067ee0 msgr2=0x7fcc7c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:42.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 -- 172.21.15.67:0/3996991351 shutdown_connections 2024-04-17T07:15:42.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.121+0000 7fcc82c00640 1 -- 172.21.15.67:0/3996991351 wait complete. 2024-04-17T07:15:42.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.123+0000 7fcc82c00640 1 Processor -- start 2024-04-17T07:15:42.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.123+0000 7fcc82c00640 1 -- start start 2024-04-17T07:15:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c071f30 0x7fcc7c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc7c1634a0 0x7fcc7c165ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c167820 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c06f4f0 con 0x7fcc7c071f30 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc82c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7c1f9820 con 0x7fcc7c1634a0 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53902/0 (socket says 172.21.15.67:53902) 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 -- 172.21.15.67:0/1155933692 learned_addr learned my addr 172.21.15.67:0/1155933692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 -- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc7c1634a0 msgr2=0x7fcc7c165ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc7c1634a0 0x7fcc7c165ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 -- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c071f30 msgr2=0x7fcc7c1599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c071f30 0x7fcc7c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.124+0000 7fcc7b400640 1 -- 172.21.15.67:0/1155933692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1f8c10 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc7b400640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400e980 tx=0x7fcc6400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64017070 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc6400ca00 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400cdd0 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c1f8ef0 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.125+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c1f94b0 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.126+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7c068ca0 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.127+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fcc6401a070 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.129+0000 7fcc73e00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 0x7fcc5806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.129+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fcc6408c340 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.130+0000 7fcc7be00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 0x7fcc5806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.131+0000 7fcc7be00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 0x7fcc5806efe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000f60 tx=0x7fcc6c0078f0 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.134+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcc6405a3d0 con 0x7fcc7c0cffe0 2024-04-17T07:15:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.364+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fcc7c0b5150 con 0x7fcc5806cb20 2024-04-17T07:15:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.520+0000 7fcc73e00640 1 -- 172.21.15.67:0/1155933692 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fcc7c0b5150 con 0x7fcc5806cb20 2024-04-17T07:15:42.520 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-04-17T07:15:42.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.523+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 msgr2=0x7fcc5806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.523+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 0x7fcc5806efe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000f60 tx=0x7fcc6c0078f0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.523+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 msgr2=0x7fcc7c1657a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.523+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400e980 tx=0x7fcc6400ee40 comp rx=0 tx=0).stop 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 shutdown_connections 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcc5806cb20 0x7fcc5806efe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc7c1634a0 0x7fcc7c165ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c0cffe0 0x7fcc7c1657a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 --2- 172.21.15.67:0/1155933692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c071f30 0x7fcc7c1599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 >> 172.21.15.67:0/1155933692 conn(0x7fcc7c067ee0 msgr2=0x7fcc7c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 shutdown_connections 2024-04-17T07:15:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.524+0000 7fcc82c00640 1 -- 172.21.15.67:0/1155933692 wait complete. 2024-04-17T07:15:42.545 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-04-17T07:15:42.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-04-17T07:15:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.773+0000 7fd1d5a00640 1 Processor -- start 2024-04-17T07:15:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.774+0000 7fd1d5a00640 1 -- start start 2024-04-17T07:15:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.775+0000 7fd1d5a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1d015ecb0 0x7fd1d00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.775+0000 7fd1d5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d00d00b0 con 0x7fd1d00d3210 2024-04-17T07:15:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.775+0000 7fd1d5a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d00d0390 con 0x7fd1d00cfab0 2024-04-17T07:15:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.775+0000 7fd1d5a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d00d0c30 con 0x7fd1d015ecb0 2024-04-17T07:15:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1c7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1d00d3210 0x7fd1d00cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33738/0 (socket says 172.21.15.67:33738) 2024-04-17T07:15:42.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1c7400640 1 -- 172.21.15.67:0/2964513857 learned_addr learned my addr 172.21.15.67:0/2964513857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/2964513857 >> v2:172.21.15.187:3300/0 conn(0x7fd1d015ecb0 0x7fd1d00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984544789 0 0) 0x7fd1d00d00b0 con 0x7fd1d00d3210 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b4003520 con 0x7fd1d00d3210 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3723842235 0 0) 0x7fd1b4003520 con 0x7fd1d00d3210 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.776+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 >> v2:172.21.15.187:3300/0 conn(0x7fd1d015ecb0 msgr2=0x7fd1d00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 --2- 172.21.15.67:0/2964513857 >> v2:172.21.15.187:3300/0 conn(0x7fd1d015ecb0 0x7fd1d00d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 >> v1:172.21.15.187:6789/0 conn(0x7fd1d00cfab0 legacy=0x7fd1d00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d00d0670 con 0x7fd1d00d3210 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd1b8001670 con 0x7fd1d00d3210 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 >> v1:172.21.15.67:6789/0 conn(0x7fd1d00d3210 legacy=0x7fd1d00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 0x7fd1b4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1b4004800 0x7fd1b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40051c0 0x7fd1b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00d0c30 con 0x7fd1b40039b0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b400c0f0 con 0x7fd1b40051c0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b400c2f0 con 0x7fd1b4004800 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 0x7fd1b4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1d5000640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40051c0 0x7fd1b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1c7400640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1b4004800 0x7fd1b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cfe00640 1 -- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1b4004800 msgr2=0x7fd1b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.777+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1b4004800 0x7fd1b4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.778+0000 7fd1cfe00640 1 -- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40051c0 msgr2=0x7fd1b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.778+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40051c0 0x7fd1b400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.778+0000 7fd1cfe00640 1 -- 172.21.15.67:0/2964513857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b4003520 con 0x7fd1b40039b0 2024-04-17T07:15:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.778+0000 7fd1d5000640 1 -- 172.21.15.67:0/2964513857 reap_dead start 2024-04-17T07:15:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.779+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 0x7fd1b4003db0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009ac0 tx=0x7fd1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8cae58e9e2b015 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.779+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0039070 con 0x7fd1b40039b0 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.779+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c002c6b0 con 0x7fd1b40039b0 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.779+0000 7fd1cf400640 1 -- 172.21.15.67:0/2964513857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0009690 con 0x7fd1b40039b0 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 -- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 msgr2=0x7fd1b4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 0x7fd1b4003db0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009ac0 tx=0x7fd1b4005830 comp rx=0 tx=0).stop 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 -- 172.21.15.67:0/2964513857 shutdown_connections 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/2964513857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1b40039b0 0x7fd1b4003db0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 -- 172.21.15.67:0/2964513857 >> 172.21.15.67:0/2964513857 conn(0x7fd1d008bff0 msgr2=0x7fd1d00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 -- 172.21.15.67:0/2964513857 shutdown_connections 2024-04-17T07:15:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.780+0000 7fd1d5a00640 1 -- 172.21.15.67:0/2964513857 wait complete. 2024-04-17T07:15:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.782+0000 7fd1d5a00640 1 Processor -- start 2024-04-17T07:15:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.782+0000 7fd1d5a00640 1 -- start start 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 0x7fd1d00b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 0x7fd1d00cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d015f210 con 0x7fd1d01f7d00 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d015f090 con 0x7fd1b40039b0 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d00d2cd0 con 0x7fd1d00ce2e0 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53910/0 (socket says 172.21.15.67:53910) 2024-04-17T07:15:42.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1d5000640 1 -- 172.21.15.67:0/3728130033 learned_addr learned my addr 172.21.15.67:0/3728130033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1c7400640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 0x7fd1d00cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.783+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 0x7fd1d00b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 -- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 msgr2=0x7fd1d00cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 0x7fd1d00cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 -- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 msgr2=0x7fd1d00b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 0x7fd1d00b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 -- 172.21.15.67:0/3728130033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d00b5120 con 0x7fd1d01f7d00 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1c7400640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 0x7fd1d00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 0x7fd1d00b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5000640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0069630 tx=0x7fd1c800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8012800 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c8019070 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8012be0 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d00b5400 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.784+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d00b8330 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.786+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d0068a10 con 0x7fd1d01f7d00 2024-04-17T07:15:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.787+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fd1c801e460 con 0x7fd1d01f7d00 2024-04-17T07:15:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.788+0000 7fd1ccc00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 0x7fd1ac06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.788+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fd1c808bed0 con 0x7fd1d01f7d00 2024-04-17T07:15:42.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.788+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 0x7fd1ac06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.790+0000 7fd1cfe00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 0x7fd1ac06efb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0036000 tx=0x7fd1c0030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:42.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:42.792+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd1c8059de0 con 0x7fd1d01f7d00 2024-04-17T07:15:43.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.025+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd1d00d4e60 con 0x7fd1ac06caf0 2024-04-17T07:15:43.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.174+0000 7fd1ccc00640 1 -- 172.21.15.67:0/3728130033 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd1d00d4e60 con 0x7fd1ac06caf0 2024-04-17T07:15:43.175 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 msgr2=0x7fd1ac06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 0x7fd1ac06efb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0036000 tx=0x7fd1c0030000 comp rx=0 tx=0).stop 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 msgr2=0x7fd1d01f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0069630 tx=0x7fd1c800d9f0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 shutdown_connections 2024-04-17T07:15:43.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fd1ac06caf0 0x7fd1ac06efb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1d01f7d00 0x7fd1d01f81b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1d00ce2e0 0x7fd1d00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 --2- 172.21.15.67:0/3728130033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1b40039b0 0x7fd1d00b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 >> 172.21.15.67:0/3728130033 conn(0x7fd1d008bff0 msgr2=0x7fd1d006d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 shutdown_connections 2024-04-17T07:15:43.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.178+0000 7fd1d5a00640 1 -- 172.21.15.67:0/3728130033 wait complete. 2024-04-17T07:15:43.201 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-04-17T07:15:43.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-04-17T07:15:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.427+0000 7f0039200640 1 Processor -- start 2024-04-17T07:15:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.427+0000 7f0039200640 1 -- start start 2024-04-17T07:15:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.428+0000 7f0039200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 0x7f00340d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.428+0000 7f0039200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00340cfab0 0x7f00340b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.428+0000 7f0039200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00340d00b0 con 0x7f00340d3210 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.428+0000 7f0039200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f00340d0c30 con 0x7f00340cfab0 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.428+0000 7f0039200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00340d0390 con 0x7f003415ecb0 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00340cfab0 0x7f00340b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00340cfab0 0x7f00340b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42888/0 (socket says 172.21.15.67:42888) 2024-04-17T07:15:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 -- 172.21.15.67:0/1781959452 learned_addr learned my addr 172.21.15.67:0/1781959452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 -- 172.21.15.67:0/1781959452 >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 msgr2=0x7f00340d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033400640 1 --2- 172.21.15.67:0/1781959452 >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 0x7f00340d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 --2- 172.21.15.67:0/1781959452 >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 0x7f00340d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 -- 172.21.15.67:0/1781959452 >> v1:172.21.15.187:6789/0 conn(0x7f00340d3210 legacy=0x7f00340cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033e00640 1 -- 172.21.15.67:0/1781959452 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00340d0670 con 0x7f00340cfab0 2024-04-17T07:15:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.429+0000 7f0033400640 1 --2- 172.21.15.67:0/1781959452 >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 0x7f00340d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.430+0000 7f0033e00640 1 --2- 172.21.15.67:0/1781959452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340cfab0 0x7f00340b3fc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f003408ca90 tx=0x7f002c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4449e5200d9ce7e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.431+0000 7f0032000640 1 -- 172.21.15.67:0/1781959452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c02c9f0 con 0x7f00340cfab0 2024-04-17T07:15:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.431+0000 7f0032000640 1 -- 172.21.15.67:0/1781959452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c02cb90 con 0x7f00340cfab0 2024-04-17T07:15:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.431+0000 7f0032000640 1 -- 172.21.15.67:0/1781959452 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0343f0 con 0x7f00340cfab0 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.431+0000 7f0039200640 1 -- 172.21.15.67:0/1781959452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340cfab0 msgr2=0x7f00340b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.431+0000 7f0039200640 1 --2- 172.21.15.67:0/1781959452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340cfab0 0x7f00340b3fc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f003408ca90 tx=0x7f002c02be50 comp rx=0 tx=0).stop 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 -- 172.21.15.67:0/1781959452 shutdown_connections 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 --2- 172.21.15.67:0/1781959452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340cfab0 0x7f00340b3fc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 --2- 172.21.15.67:0/1781959452 >> v2:172.21.15.187:3300/0 conn(0x7f003415ecb0 0x7f00340d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 -- 172.21.15.67:0/1781959452 >> 172.21.15.67:0/1781959452 conn(0x7f003408bff0 msgr2=0x7f00340d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 -- 172.21.15.67:0/1781959452 shutdown_connections 2024-04-17T07:15:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.432+0000 7f0039200640 1 -- 172.21.15.67:0/1781959452 wait complete. 2024-04-17T07:15:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.433+0000 7f0039200640 1 Processor -- start 2024-04-17T07:15:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.434+0000 7f0039200640 1 -- start start 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00340cfab0 0x7f00340caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f003415ecb0 0x7f003414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f003415f210 con 0x7f00340d3210 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f003415f090 con 0x7f00340cfab0 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0039200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00340b4500 con 0x7f003415ecb0 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42892/0 (socket says 172.21.15.67:42892) 2024-04-17T07:15:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 -- 172.21.15.67:0/429075187 learned_addr learned my addr 172.21.15.67:0/429075187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 -- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f003415ecb0 msgr2=0x7f003414d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f003415ecb0 0x7f003414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 -- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00340cfab0 msgr2=0x7f00340caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00340cfab0 0x7f00340caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.435+0000 7f0032a00640 1 -- 172.21.15.67:0/429075187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034161260 con 0x7f00340d3210 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f0032a00640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f002000c460 tx=0x7f002000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020016020 con 0x7f00340d3210 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020012870 con 0x7f00340d3210 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0034161540 con 0x7f00340d3210 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020012c60 con 0x7f00340d3210 2024-04-17T07:15:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.436+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0034161b00 con 0x7f00340d3210 2024-04-17T07:15:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.437+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0034068a10 con 0x7f00340d3210 2024-04-17T07:15:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.438+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f0020019020 con 0x7f00340d3210 2024-04-17T07:15:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.439+0000 7f002b400640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 0x7f001406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.439+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f002008c0f0 con 0x7f00340d3210 2024-04-17T07:15:43.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.440+0000 7f0033400640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 0x7f001406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:43.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.441+0000 7f0033400640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 0x7f001406efb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0024006fd0 tx=0x7f0024008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.444+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0020020020 con 0x7f00340d3210 2024-04-17T07:15:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.673+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f00340b5a70 con 0x7f001406caf0 2024-04-17T07:15:43.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.833+0000 7f002b400640 1 -- 172.21.15.67:0/429075187 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f00340b5a70 con 0x7f001406caf0 2024-04-17T07:15:43.834 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 msgr2=0x7f001406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 0x7f001406efb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0024006fd0 tx=0x7f0024008040 comp rx=0 tx=0).stop 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 msgr2=0x7f003414d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f002000c460 tx=0x7f002000c920 comp rx=0 tx=0).stop 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 shutdown_connections 2024-04-17T07:15:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f001406caf0 0x7f001406efb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f003415ecb0 0x7f003414d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340d3210 0x7f003414d390 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 --2- 172.21.15.67:0/429075187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00340cfab0 0x7f00340caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 >> 172.21.15.67:0/429075187 conn(0x7f003408bff0 msgr2=0x7f003415de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 shutdown_connections 2024-04-17T07:15:43.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:43.837+0000 7f0039200640 1 -- 172.21.15.67:0/429075187 wait complete. 2024-04-17T07:15:43.861 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-04-17T07:15:43.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-04-17T07:15:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.086+0000 7fb630a00640 1 Processor -- start 2024-04-17T07:15:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.086+0000 7fb630a00640 1 -- start start 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.087+0000 7fb630a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.087+0000 7fb630a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 0x7fb62c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.087+0000 7fb630a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 0x7fb62c0d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb630a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb62c0d0c30 con 0x7fb62c15ecb0 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb630a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb62c0d00b0 con 0x7fb62c0cf8e0 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb630a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c0d0390 con 0x7fb62c0d3210 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42894/0 (socket says 172.21.15.67:42894) 2024-04-17T07:15:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 -- 172.21.15.67:0/299418007 learned_addr learned my addr 172.21.15.67:0/299418007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62b200640 1 --2- 172.21.15.67:0/299418007 >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 0x7fb62c0d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb629e00640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 0x7fb62c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 -- 172.21.15.67:0/299418007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 msgr2=0x7fb62c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 0x7fb62c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 -- 172.21.15.67:0/299418007 >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 msgr2=0x7fb62c0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 --2- 172.21.15.67:0/299418007 >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 0x7fb62c0d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62a800640 1 -- 172.21.15.67:0/299418007 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c0d0670 con 0x7fb62c15ecb0 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb62b200640 1 --2- 172.21.15.67:0/299418007 >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 0x7fb62c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.088+0000 7fb629e00640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 0x7fb62c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.090+0000 7fb62a800640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0b61a0 tx=0x7fb61c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c667bb946f125e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.090+0000 7fb629400640 1 -- 172.21.15.67:0/299418007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c02e070 con 0x7fb62c15ecb0 2024-04-17T07:15:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.090+0000 7fb629400640 1 -- 172.21.15.67:0/299418007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb61c037070 con 0x7fb62c15ecb0 2024-04-17T07:15:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.090+0000 7fb629400640 1 -- 172.21.15.67:0/299418007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c031a00 con 0x7fb62c15ecb0 2024-04-17T07:15:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 -- 172.21.15.67:0/299418007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c15ecb0 msgr2=0x7fb62c0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0b61a0 tx=0x7fb61c02b880 comp rx=0 tx=0).stop 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 -- 172.21.15.67:0/299418007 shutdown_connections 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 --2- 172.21.15.67:0/299418007 >> v2:172.21.15.187:3300/0 conn(0x7fb62c0cf8e0 0x7fb62c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c0d3210 0x7fb62c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 --2- 172.21.15.67:0/299418007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c15ecb0 0x7fb62c0d2cd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 -- 172.21.15.67:0/299418007 >> 172.21.15.67:0/299418007 conn(0x7fb62c08bff0 msgr2=0x7fb62c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 -- 172.21.15.67:0/299418007 shutdown_connections 2024-04-17T07:15:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.091+0000 7fb630a00640 1 -- 172.21.15.67:0/299418007 wait complete. 2024-04-17T07:15:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.093+0000 7fb630a00640 1 Processor -- start 2024-04-17T07:15:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.093+0000 7fb630a00640 1 -- start start 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb62c0cf8e0 0x7fb62c0c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c15ecb0 0x7fb62c15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c15f090 con 0x7fb62c0d3210 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c0cfd70 con 0x7fb62c0cf8e0 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb630a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c15f210 con 0x7fb62c15ecb0 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42910/0 (socket says 172.21.15.67:42910) 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 -- 172.21.15.67:0/1776743210 learned_addr learned my addr 172.21.15.67:0/1776743210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 -- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c15ecb0 msgr2=0x7fb62c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c15ecb0 0x7fb62c15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 -- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb62c0cf8e0 msgr2=0x7fb62c0c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb62c0cf8e0 0x7fb62c0c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.094+0000 7fb629e00640 1 -- 172.21.15.67:0/1776743210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c0c9ba0 con 0x7fb62c0d3210 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb629e00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cf90 tx=0x7fb61400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61401a070 con 0x7fb62c0d3210 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb61400d840 con 0x7fb62c0d3210 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61400dc10 con 0x7fb62c0d3210 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62c0c85c0 con 0x7fb62c0d3210 2024-04-17T07:15:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.095+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62c0c8e60 con 0x7fb62c0d3210 2024-04-17T07:15:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.096+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62c068a10 con 0x7fb62c0d3210 2024-04-17T07:15:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.098+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fb61401d070 con 0x7fb62c0d3210 2024-04-17T07:15:44.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.098+0000 7fb622a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 0x7fb60806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.099+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fb61408c850 con 0x7fb62c0d3210 2024-04-17T07:15:44.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.099+0000 7fb62a800640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 0x7fb60806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.101+0000 7fb62a800640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 0x7fb60806efb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb61c036000 tx=0x7fb61c030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.103+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb61405a210 con 0x7fb62c0d3210 2024-04-17T07:15:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.334+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb62c0cde70 con 0x7fb60806caf0 2024-04-17T07:15:44.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.491+0000 7fb622a00640 1 -- 172.21.15.67:0/1776743210 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb62c0cde70 con 0x7fb60806caf0 2024-04-17T07:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 msgr2=0x7fb60806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 0x7fb60806efb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb61c036000 tx=0x7fb61c030000 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 msgr2=0x7fb62c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cf90 tx=0x7fb61400a7d0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 shutdown_connections 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb60806caf0 0x7fb60806efb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb62c15ecb0 0x7fb62c15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62c0d3210 0x7fb62c0cd930 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 --2- 172.21.15.67:0/1776743210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb62c0cf8e0 0x7fb62c0c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.495+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 >> 172.21.15.67:0/1776743210 conn(0x7fb62c08bff0 msgr2=0x7fb62c06d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.496+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 shutdown_connections 2024-04-17T07:15:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.496+0000 7fb630a00640 1 -- 172.21.15.67:0/1776743210 wait complete. 2024-04-17T07:15:44.521 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-04-17T07:15:44.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-04-17T07:15:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.744+0000 7fc71fe00640 1 Processor -- start 2024-04-17T07:15:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.745+0000 7fc71fe00640 1 -- start start 2024-04-17T07:15:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc71815b860 0x7fc7180d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7180b4f80 0x7fc7180d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc718071f60 con 0x7fc7180ce4d0 2024-04-17T07:15:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc718071d10 con 0x7fc7180b4f80 2024-04-17T07:15:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc718072240 con 0x7fc71815b860 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7180b4f80 0x7fc7180d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7180b4f80 0x7fc7180d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42922/0 (socket says 172.21.15.67:42922) 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71de00640 1 -- 172.21.15.67:0/163767811 learned_addr learned my addr 172.21.15.67:0/163767811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.747+0000 7fc71d400640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc71815b860 0x7fc7180d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.748+0000 7fc71de00640 1 -- 172.21.15.67:0/163767811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc71815b860 msgr2=0x7fc7180d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.748+0000 7fc71de00640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc71815b860 0x7fc7180d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.748+0000 7fc71de00640 1 -- 172.21.15.67:0/163767811 >> v1:172.21.15.187:6789/0 conn(0x7fc7180ce4d0 legacy=0x7fc7180ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.748+0000 7fc71de00640 1 -- 172.21.15.67:0/163767811 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc718072520 con 0x7fc7180b4f80 2024-04-17T07:15:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.749+0000 7fc71de00640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7180b4f80 0x7fc7180d0960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e50 tx=0x7fc71002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49b9549f2615f8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.749+0000 7fc717e00640 1 -- 172.21.15.67:0/163767811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71002ca70 con 0x7fc7180b4f80 2024-04-17T07:15:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.749+0000 7fc717e00640 1 -- 172.21.15.67:0/163767811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71002cc10 con 0x7fc7180b4f80 2024-04-17T07:15:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.749+0000 7fc717e00640 1 -- 172.21.15.67:0/163767811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100343f0 con 0x7fc7180b4f80 2024-04-17T07:15:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 -- 172.21.15.67:0/163767811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7180b4f80 msgr2=0x7fc7180d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7180b4f80 0x7fc7180d0960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e50 tx=0x7fc71002bed0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 -- 172.21.15.67:0/163767811 shutdown_connections 2024-04-17T07:15:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7180b4f80 0x7fc7180d0960 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 --2- 172.21.15.67:0/163767811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc71815b860 0x7fc7180d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 -- 172.21.15.67:0/163767811 >> 172.21.15.67:0/163767811 conn(0x7fc718067ee0 msgr2=0x7fc71806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 -- 172.21.15.67:0/163767811 shutdown_connections 2024-04-17T07:15:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.750+0000 7fc71fe00640 1 -- 172.21.15.67:0/163767811 wait complete. 2024-04-17T07:15:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.752+0000 7fc71fe00640 1 Processor -- start 2024-04-17T07:15:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.752+0000 7fc71fe00640 1 -- start start 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 0x7fc7180af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7180ce4d0 0x7fc71814af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc71815bc40 con 0x7fc71815b860 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc718072ae0 con 0x7fc7180b4f80 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc71806f380 con 0x7fc7180ce4d0 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.753+0000 7fc71de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42936/0 (socket says 172.21.15.67:42936) 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 -- 172.21.15.67:0/2248211020 learned_addr learned my addr 172.21.15.67:0/2248211020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71ca00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7180ce4d0 0x7fc71814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71d400640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 0x7fc7180af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 -- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7180ce4d0 msgr2=0x7fc71814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7180ce4d0 0x7fc71814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 -- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 msgr2=0x7fc7180af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 0x7fc7180af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 -- 172.21.15.67:0/2248211020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71814b450 con 0x7fc71815b860 2024-04-17T07:15:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71d400640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 0x7fc7180af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.754+0000 7fc71de00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e50 tx=0x7fc710004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.755+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710043020 con 0x7fc71815b860 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.755+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71003e450 con 0x7fc71815b860 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.755+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710009510 con 0x7fc71815b860 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.755+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7180ae760 con 0x7fc71815b860 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.755+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7180aed20 con 0x7fc71815b860 2024-04-17T07:15:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.756+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71806deb0 con 0x7fc71815b860 2024-04-17T07:15:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.757+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7fc710046020 con 0x7fc71815b860 2024-04-17T07:15:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.758+0000 7fc715600640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 0x7fc6f806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.758+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fc7100ae8c0 con 0x7fc71815b860 2024-04-17T07:15:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.759+0000 7fc71d400640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 0x7fc6f806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.760+0000 7fc71d400640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 0x7fc6f806efb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.763+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc71004d030 con 0x7fc71815b860 2024-04-17T07:15:44.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:44.994+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc7181e56b0 con 0x7fc6f806caf0 2024-04-17T07:15:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.149+0000 7fc715600640 1 -- 172.21.15.67:0/2248211020 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc7181e56b0 con 0x7fc6f806caf0 2024-04-17T07:15:45.149 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-04-17T07:15:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 msgr2=0x7fc6f806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 0x7fc6f806efb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).stop 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 msgr2=0x7fc7181e5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc710009e50 tx=0x7fc710004a40 comp rx=0 tx=0).stop 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 shutdown_connections 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fc6f806caf0 0x7fc6f806efb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc71815b860 0x7fc7181e5170 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc7180ce4d0 0x7fc71814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.153+0000 7fc71fe00640 1 --2- 172.21.15.67:0/2248211020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc7180b4f80 0x7fc7180af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.154+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 >> 172.21.15.67:0/2248211020 conn(0x7fc718067ee0 msgr2=0x7fc7180b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.154+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 shutdown_connections 2024-04-17T07:15:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.154+0000 7fc71fe00640 1 -- 172.21.15.67:0/2248211020 wait complete. 2024-04-17T07:15:45.181 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-04-17T07:15:45.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-04-17T07:15:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.410+0000 7f6cc7400640 1 Processor -- start 2024-04-17T07:15:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.410+0000 7f6cc7400640 1 -- start start 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 0x7f6cc0071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cc7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cc00ce3b0 0x7f6cc0072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cc7400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cc00cea70 con 0x7f6cc0072bb0 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cc7400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6cc00ce810 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cc7400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc00ced50 con 0x7f6cc0145ac0 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cbfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cc00ce3b0 0x7f6cc0072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cbfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cc00ce3b0 0x7f6cc0072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42944/0 (socket says 172.21.15.67:42944) 2024-04-17T07:15:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.412+0000 7f6cbfe00640 1 -- 172.21.15.67:0/1488530971 learned_addr learned my addr 172.21.15.67:0/1488530971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cbfe00640 1 -- 172.21.15.67:0/1488530971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 msgr2=0x7f6cc0071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 0x7f6cc0071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cbfe00640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 0x7f6cc0071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cbfe00640 1 -- 172.21.15.67:0/1488530971 >> v1:172.21.15.187:6789/0 conn(0x7f6cc0072bb0 legacy=0x7f6cc00ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cbfe00640 1 -- 172.21.15.67:0/1488530971 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc00cf030 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.413+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 0x7f6cc0071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.414+0000 7f6cbfe00640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc0072670 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009870 tx=0x7f6cac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169348d5e4d8e8ab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.414+0000 7f6cbf400640 1 -- 172.21.15.67:0/1488530971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac035020 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.414+0000 7f6cbf400640 1 -- 172.21.15.67:0/1488530971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cac031640 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.414+0000 7f6cbf400640 1 -- 172.21.15.67:0/1488530971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac02d040 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 -- 172.21.15.67:0/1488530971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc00ce3b0 msgr2=0x7f6cc0072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc0072670 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009870 tx=0x7f6cac02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 -- 172.21.15.67:0/1488530971 shutdown_connections 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc0072670 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 --2- 172.21.15.67:0/1488530971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0145ac0 0x7f6cc0071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 -- 172.21.15.67:0/1488530971 >> 172.21.15.67:0/1488530971 conn(0x7f6cc0067ee0 msgr2=0x7f6cc006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 -- 172.21.15.67:0/1488530971 shutdown_connections 2024-04-17T07:15:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.415+0000 7f6cc7400640 1 -- 172.21.15.67:0/1488530971 wait complete. 2024-04-17T07:15:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.417+0000 7f6cc7400640 1 Processor -- start 2024-04-17T07:15:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.417+0000 7f6cc7400640 1 -- start start 2024-04-17T07:15:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 0x7f6cc0137280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc015bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0145ea0 con 0x7f6cc0145ac0 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc00cf5f0 con 0x7f6cc00ce3b0 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.418+0000 7f6cc7400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc00d0b20 con 0x7f6cc0072bb0 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42946/0 (socket says 172.21.15.67:42946) 2024-04-17T07:15:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 -- 172.21.15.67:0/872111855 learned_addr learned my addr 172.21.15.67:0/872111855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:15:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cbfe00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc015bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 0x7f6cc0137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 -- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 msgr2=0x7f6cc0137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 0x7f6cc0137280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 -- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 msgr2=0x7f6cc015bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc015bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc5400640 1 -- 172.21.15.67:0/872111855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc006fab0 con 0x7f6cc0145ac0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 0x7f6cc0137280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.419+0000 7f6cbfe00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc015bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cc5400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800c780 tx=0x7f6cb800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8004b60 con 0x7f6cc0145ac0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb8004d00 con 0x7f6cc0145ac0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800aa90 con 0x7f6cc0145ac0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc00c97a0 con 0x7f6cc0145ac0 2024-04-17T07:15:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.420+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc00c9d60 con 0x7f6cc0145ac0 2024-04-17T07:15:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.421+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc00cc6c0 con 0x7f6cc0145ac0 2024-04-17T07:15:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.422+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) ==== 91216+0+0 (secure 0 0 0) 0x7f6cb800ac30 con 0x7f6cc0145ac0 2024-04-17T07:15:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.423+0000 7f6cbcc00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 0x7f6ca006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:15:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.423+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f6cb808c310 con 0x7f6cc0145ac0 2024-04-17T07:15:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.424+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 0x7f6ca006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:15:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.425+0000 7f6cc4a00640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 0x7f6ca006efb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4006fd0 tx=0x7f6cb4008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:15:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.428+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6cb8020070 con 0x7f6cc0145ac0 2024-04-17T07:15:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.660+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6cc00b1aa0 con 0x7f6ca006caf0 2024-04-17T07:15:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.815+0000 7f6cbcc00640 1 -- 172.21.15.67:0/872111855 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f6cc00b1aa0 con 0x7f6ca006caf0 2024-04-17T07:15:45.816 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-04-17T07:15:45.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 msgr2=0x7f6ca006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 0x7f6ca006efb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4006fd0 tx=0x7f6cb4008040 comp rx=0 tx=0).stop 2024-04-17T07:15:45.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 msgr2=0x7f6cc015f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800c780 tx=0x7f6cb800cc40 comp rx=0 tx=0).stop 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 shutdown_connections 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6ca006caf0 0x7f6ca006efb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cc0145ac0 0x7f6cc015f6f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cc00ce3b0 0x7f6cc015bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 --2- 172.21.15.67:0/872111855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cc0072bb0 0x7f6cc0137280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 >> 172.21.15.67:0/872111855 conn(0x7f6cc0067ee0 msgr2=0x7f6cc013dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.819+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 shutdown_connections 2024-04-17T07:15:45.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:15:45.820+0000 7f6cc7400640 1 -- 172.21.15.67:0/872111855 wait complete. 2024-04-17T07:15:45.840 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-04-17T07:15:54.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:15:54.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:16:00.176 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:16:00.176+0000 7fbd36200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:16:00.177 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:16:00.176+0000 7fbd36200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:16:00.177 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:16:00.176+0000 7fbd36200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:16:00.177 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:16:00.176+0000 7fbd36200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:16:00.177 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:16:00.176+0000 7fbd36200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:16:24.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:16:24.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:16:54.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:16:54.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:17:24.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:17:24.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:17:54.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:17:54.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:18:24.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:18:24.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:18:54.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:18:54.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:19:25.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:19:25.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:19:55.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:19:55.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:20:00.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:20:00.380+0000 7fbd39400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:20:00.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:20:00.380+0000 7fbd39400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:20:00.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:20:00.380+0000 7fbd39400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:20:00.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:20:00.380+0000 7fbd39400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:20:00.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:20:00.380+0000 7fbd39400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:20:25.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:20:25.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:20:55.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:20:55.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:21:25.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:21:25.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:21:55.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:21:55.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:22:25.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:22:25.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:22:56.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:22:56.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:23:26.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:23:26.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:23:56.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:23:56.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:24:26.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:24:26.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:24:56.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:24:56.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:25:26.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:25:26.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:25:57.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:25:57.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:26:21.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-04-17T07:26:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.076+0000 7fe7a8600640 1 Processor -- start 2024-04-17T07:26:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.076+0000 7fe7a8600640 1 -- start start 2024-04-17T07:26:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.077+0000 7fe7a8600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe7a015b800 0x7fe7a015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.077+0000 7fe7a8600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a015c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.077+0000 7fe7a8600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a015c740 con 0x7fe7a015bbe0 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.077+0000 7fe7a8600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a01604b0 con 0x7fe7a015b800 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.077+0000 7fe7a8600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a015ca20 con 0x7fe7a00d02c0 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.078+0000 7fe7a5c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe7a015b800 0x7fe7a015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.078+0000 7fe7a5c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe7a015b800 0x7fe7a015ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:32836/0 (socket says 172.21.15.67:32836) 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.078+0000 7fe7a5c00640 1 -- 172.21.15.67:0/2087103826 learned_addr learned my addr 172.21.15.67:0/2087103826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.078+0000 7fe7a6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a015c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.079+0000 7fe7a5c00640 1 -- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 msgr2=0x7fe7a015c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.079+0000 7fe7a5c00640 1 --2- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a015c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.079+0000 7fe7a5c00640 1 -- 172.21.15.67:0/2087103826 >> v1:172.21.15.187:6789/0 conn(0x7fe7a015bbe0 legacy=0x7fe7a00cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.079+0000 7fe7a5c00640 1 -- 172.21.15.67:0/2087103826 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a015cd00 con 0x7fe7a015b800 2024-04-17T07:26:22.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.080+0000 7fe7a5c00640 1 --2- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015b800 0x7fe7a015ff70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe79000bb10 tx=0x7fe79002e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0eddcf2dcd4f9eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.081+0000 7fe79fe00640 1 -- 172.21.15.67:0/2087103826 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79003b070 con 0x7fe7a015b800 2024-04-17T07:26:22.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.081+0000 7fe79fe00640 1 -- 172.21.15.67:0/2087103826 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe790036440 con 0x7fe7a015b800 2024-04-17T07:26:22.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.081+0000 7fe79fe00640 1 -- 172.21.15.67:0/2087103826 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790036810 con 0x7fe7a015b800 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 -- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015b800 msgr2=0x7fe7a015ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015b800 0x7fe7a015ff70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe79000bb10 tx=0x7fe79002e160 comp rx=0 tx=0).stop 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 -- 172.21.15.67:0/2087103826 shutdown_connections 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2087103826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015b800 0x7fe7a015ff70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 -- 172.21.15.67:0/2087103826 >> 172.21.15.67:0/2087103826 conn(0x7fe7a0073360 msgr2=0x7fe7a009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 -- 172.21.15.67:0/2087103826 shutdown_connections 2024-04-17T07:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.082+0000 7fe7a8600640 1 -- 172.21.15.67:0/2087103826 wait complete. 2024-04-17T07:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.084+0000 7fe7a8600640 1 Processor -- start 2024-04-17T07:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.085+0000 7fe7a8600640 1 -- start start 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a00c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015bbe0 0x7fe7a00b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a00d1400 con 0x7fe7a015b800 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a00d1280 con 0x7fe7a015bbe0 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a8600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a00d1580 con 0x7fe7a00d02c0 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58376/0 (socket says 172.21.15.67:58376) 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.086+0000 7fe7a5200640 1 -- 172.21.15.67:0/2836271981 learned_addr learned my addr 172.21.15.67:0/2836271981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5c00640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a00c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a6600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015bbe0 0x7fe7a00b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 -- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 msgr2=0x7fe7a00c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a00c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 -- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015bbe0 msgr2=0x7fe7a00b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015bbe0 0x7fe7a00b6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 -- 172.21.15.67:0/2836271981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a00c67a0 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.087+0000 7fe7a5200640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c340 tx=0x7fe78c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.088+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c017070 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.088+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c0052d0 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.088+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a00c6a40 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.088+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003850 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.088+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a00c7000 con 0x7fe7a015b800 2024-04-17T07:26:22.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.089+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a00d2430 con 0x7fe7a015b800 2024-04-17T07:26:22.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.090+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fe78c01a070 con 0x7fe7a015b800 2024-04-17T07:26:22.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.091+0000 7fe79d600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 0x7fe78006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.091+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fe78c08b480 con 0x7fe7a015b800 2024-04-17T07:26:22.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.092+0000 7fe7a5c00640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 0x7fe78006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.093+0000 7fe7a5c00640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 0x7fe78006efb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe790038000 tx=0x7fe790008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.096+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe78c059400 con 0x7fe7a015b800 2024-04-17T07:26:22.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.332+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fe7a00cc1f0 con 0x7fe78006caf0 2024-04-17T07:26:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.556+0000 7fe79d600640 1 -- 172.21.15.67:0/2836271981 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe7a00cc1f0 con 0x7fe78006caf0 2024-04-17T07:26:22.556 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-04-17T07:26:22.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 msgr2=0x7fe78006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 0x7fe78006efb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe790038000 tx=0x7fe790008040 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 msgr2=0x7fe7a015dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c340 tx=0x7fe78c00c800 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 shutdown_connections 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe78006caf0 0x7fe78006efb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe7a015bbe0 0x7fe7a00b6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a015b800 0x7fe7a015dce0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 --2- 172.21.15.67:0/2836271981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7a00d02c0 0x7fe7a00c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.559+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 >> 172.21.15.67:0/2836271981 conn(0x7fe7a0073360 msgr2=0x7fe7a006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.560+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 shutdown_connections 2024-04-17T07:26:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.560+0000 7fe7a8600640 1 -- 172.21.15.67:0/2836271981 wait complete. 2024-04-17T07:26:22.585 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-04-17T07:26:22.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-04-17T07:26:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.814+0000 7f25aba00640 1 Processor -- start 2024-04-17T07:26:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.814+0000 7f25aba00640 1 -- start start 2024-04-17T07:26:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.816+0000 7f25aba00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25a40d0b30 0x7f25a40b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.816+0000 7f25aba00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40ce4d0 0x7f25a40b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.816+0000 7f25aba00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25a4071f60 con 0x7f25a415b860 2024-04-17T07:26:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.816+0000 7f25aba00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f25a4071d10 con 0x7f25a40d0b30 2024-04-17T07:26:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.816+0000 7f25aba00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f25a4072240 con 0x7f25a40ce4d0 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25a40d0b30 0x7f25a40b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40ce4d0 0x7f25a40b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25a40d0b30 0x7f25a40b3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:32854/0 (socket says 172.21.15.67:32854) 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 -- 172.21.15.67:0/4255875170 learned_addr learned my addr 172.21.15.67:0/4255875170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 -- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40ce4d0 msgr2=0x7f25a40b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 --2- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40ce4d0 0x7f25a40b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 -- 172.21.15.67:0/4255875170 >> v1:172.21.15.187:6789/0 conn(0x7f25a415b860 legacy=0x7f25a40d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.817+0000 7f25a3e00640 1 -- 172.21.15.67:0/4255875170 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a4072520 con 0x7f25a40d0b30 2024-04-17T07:26:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.819+0000 7f25a3e00640 1 --2- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40d0b30 0x7f25a40b3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f25a4071e90 tx=0x7f259402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17c628cf462a5485 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25a3400640 1 -- 172.21.15.67:0/4255875170 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259402cae0 con 0x7f25a40d0b30 2024-04-17T07:26:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25a3400640 1 -- 172.21.15.67:0/4255875170 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259402cc80 con 0x7f25a40d0b30 2024-04-17T07:26:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25a3400640 1 -- 172.21.15.67:0/4255875170 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594034610 con 0x7f25a40d0b30 2024-04-17T07:26:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 -- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40d0b30 msgr2=0x7f25a40b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 --2- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40d0b30 0x7f25a40b3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f25a4071e90 tx=0x7f259402bf40 comp rx=0 tx=0).stop 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 -- 172.21.15.67:0/4255875170 shutdown_connections 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 --2- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40ce4d0 0x7f25a40b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 --2- 172.21.15.67:0/4255875170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40d0b30 0x7f25a40b3f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.820+0000 7f25aba00640 1 -- 172.21.15.67:0/4255875170 >> 172.21.15.67:0/4255875170 conn(0x7f25a4067ee0 msgr2=0x7f25a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.821+0000 7f25aba00640 1 -- 172.21.15.67:0/4255875170 shutdown_connections 2024-04-17T07:26:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.821+0000 7f25aba00640 1 -- 172.21.15.67:0/4255875170 wait complete. 2024-04-17T07:26:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.822+0000 7f25aba00640 1 Processor -- start 2024-04-17T07:26:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.822+0000 7f25aba00640 1 -- start start 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40ce4d0 0x7f25a414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40d0b30 0x7f25a415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25a415bc40 con 0x7f25a415b860 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4072ae0 con 0x7f25a40ce4d0 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.823+0000 7f25aba00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f25a406f380 con 0x7f25a40d0b30 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58378/0 (socket says 172.21.15.67:58378) 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 -- 172.21.15.67:0/1536765026 learned_addr learned my addr 172.21.15.67:0/1536765026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a3e00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40d0b30 0x7f25a415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9000640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40ce4d0 0x7f25a414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 -- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40d0b30 msgr2=0x7f25a415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40d0b30 0x7f25a415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 -- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40ce4d0 msgr2=0x7f25a414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40ce4d0 0x7f25a414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.824+0000 7f25a9a00640 1 -- 172.21.15.67:0/1536765026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a4148570 con 0x7f25a415b860 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25a9a00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f259c00e930 tx=0x7f259c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c017070 con 0x7f25a415b860 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259c00c980 con 0x7f25a415b860 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c00cd70 con 0x7f25a415b860 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a4148850 con 0x7f25a415b860 2024-04-17T07:26:22.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.825+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a4148e10 con 0x7f25a415b860 2024-04-17T07:26:22.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.826+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a40cf810 con 0x7f25a415b860 2024-04-17T07:26:22.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.827+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f259c01a070 con 0x7f25a415b860 2024-04-17T07:26:22.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.828+0000 7f25a0c00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 0x7f258006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:22.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.829+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f259c08c430 con 0x7f25a415b860 2024-04-17T07:26:22.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.829+0000 7f25a9000640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 0x7f258006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:22.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.830+0000 7f25a9000640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 0x7f258006efb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:22.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:22.834+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f259c05a500 con 0x7f25a415b860 2024-04-17T07:26:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.067+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f25a4068ca0 con 0x7f258006caf0 2024-04-17T07:26:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.280+0000 7f25a0c00640 1 -- 172.21.15.67:0/1536765026 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f25a4068ca0 con 0x7f258006caf0 2024-04-17T07:26:23.280 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-04-17T07:26:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 msgr2=0x7f258006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 0x7f258006efb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c009040 comp rx=0 tx=0).stop 2024-04-17T07:26:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 msgr2=0x7f25a415c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f259c00e930 tx=0x7f259c00edf0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 shutdown_connections 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f258006caf0 0x7f258006efb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a415b860 0x7f25a415c420 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25a40d0b30 0x7f25a415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 --2- 172.21.15.67:0/1536765026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25a40ce4d0 0x7f25a414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 >> 172.21.15.67:0/1536765026 conn(0x7f25a4067ee0 msgr2=0x7f25a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 shutdown_connections 2024-04-17T07:26:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.284+0000 7f25aba00640 1 -- 172.21.15.67:0/1536765026 wait complete. 2024-04-17T07:26:23.308 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-04-17T07:26:23.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-04-17T07:26:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.532+0000 7fba10e00640 1 Processor -- start 2024-04-17T07:26:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.532+0000 7fba10e00640 1 -- start start 2024-04-17T07:26:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.533+0000 7fba10e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba0c0b4fd0 0x7fba0c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.533+0000 7fba10e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0c0d00b0 con 0x7fba0c15ecb0 2024-04-17T07:26:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.533+0000 7fba10e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0c0d0390 con 0x7fba0c0d33e0 2024-04-17T07:26:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.533+0000 7fba10e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fba0c0d0c30 con 0x7fba0c0b4fd0 2024-04-17T07:26:23.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.533+0000 7fba0b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba0c15ecb0 0x7fba0c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51360/0 (socket says 172.21.15.67:51360) 2024-04-17T07:26:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0b400640 1 -- 172.21.15.67:0/1190797704 learned_addr learned my addr 172.21.15.67:0/1190797704 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 41644850 0 0) 0x7fba0c0d00b0 con 0x7fba0c15ecb0 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0be00640 1 --2- 172.21.15.67:0/1190797704 >> v2:172.21.15.187:3300/0 conn(0x7fba0c0b4fd0 0x7fba0c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9f0003520 con 0x7fba0c15ecb0 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2190164082 0 0) 0x7fb9f0003520 con 0x7fba0c15ecb0 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 >> v2:172.21.15.187:3300/0 conn(0x7fba0c0b4fd0 msgr2=0x7fba0c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 --2- 172.21.15.67:0/1190797704 >> v2:172.21.15.187:3300/0 conn(0x7fba0c0b4fd0 0x7fba0c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 >> v1:172.21.15.187:6789/0 conn(0x7fba0c0d33e0 legacy=0x7fba0c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c0d0670 con 0x7fba0c15ecb0 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb9fc001670 con 0x7fba0c15ecb0 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.534+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 >> v1:172.21.15.67:6789/0 conn(0x7fba0c15ecb0 legacy=0x7fba0c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9f0003a00 0x7fb9f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 0x7fb9f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00052e0 0x7fb9f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c0d0c30 con 0x7fb9f0004800 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f000c200 con 0x7fb9f00052e0 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f000c400 con 0x7fb9f0003a00 2024-04-17T07:26:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 0x7fb9f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0b400640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9f0003a00 0x7fb9f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 -- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9f0003a00 msgr2=0x7fb9f0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9f0003a00 0x7fb9f0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 -- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00052e0 msgr2=0x7fb9f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0be00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00052e0 0x7fb9f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00052e0 0x7fb9f000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0aa00640 1 -- 172.21.15.67:0/1190797704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f0003520 con 0x7fb9f0004800 2024-04-17T07:26:23.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.535+0000 7fba0be00640 1 -- 172.21.15.67:0/1190797704 reap_dead start 2024-04-17T07:26:23.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba0aa00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 0x7fb9f0004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4003db0 tx=0x7fb9f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b8881e13001446 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:23.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4037070 con 0x7fb9f0004800 2024-04-17T07:26:23.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f402be50 con 0x7fb9f0004800 2024-04-17T07:26:23.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba0a000640 1 -- 172.21.15.67:0/1190797704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4003980 con 0x7fb9f0004800 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba10e00640 1 -- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 msgr2=0x7fb9f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.537+0000 7fba10e00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 0x7fb9f0004c80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4003db0 tx=0x7fb9f0005830 comp rx=0 tx=0).stop 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.538+0000 7fba10e00640 1 -- 172.21.15.67:0/1190797704 shutdown_connections 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.538+0000 7fba10e00640 1 --2- 172.21.15.67:0/1190797704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9f0004800 0x7fb9f0004c80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.538+0000 7fba10e00640 1 -- 172.21.15.67:0/1190797704 >> 172.21.15.67:0/1190797704 conn(0x7fba0c08bff0 msgr2=0x7fba0c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.538+0000 7fba10e00640 1 -- 172.21.15.67:0/1190797704 shutdown_connections 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.538+0000 7fba10e00640 1 -- 172.21.15.67:0/1190797704 wait complete. 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.539+0000 7fba10e00640 1 Processor -- start 2024-04-17T07:26:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.540+0000 7fba10e00640 1 -- start start 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f0004800 0x7fba0c0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 0x7fba0c15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c15f210 con 0x7fba0c0c8210 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c15f090 con 0x7fb9f0004800 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba10e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba0c0d2cd0 con 0x7fba0c159c60 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42388/0 (socket says 172.21.15.67:42388) 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 -- 172.21.15.67:0/2194726576 learned_addr learned my addr 172.21.15.67:0/2194726576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0b400640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f0004800 0x7fba0c0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0be00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 0x7fba0c15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 -- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 msgr2=0x7fba0c15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 0x7fba0c15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 -- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f0004800 msgr2=0x7fba0c0cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f0004800 0x7fba0c0cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0aa00640 1 -- 172.21.15.67:0/2194726576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c1f4420 con 0x7fba0c0c8210 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.541+0000 7fba0be00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 0x7fba0c15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba0aa00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4003db0 tx=0x7fb9f40036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4042030 con 0x7fba0c0c8210 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f4007470 con 0x7fba0c0c8210 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4037040 con 0x7fba0c0c8210 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c0c87d0 con 0x7fba0c0c8210 2024-04-17T07:26:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.542+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c0c8d90 con 0x7fba0c0c8210 2024-04-17T07:26:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.544+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba0c068a10 con 0x7fba0c0c8210 2024-04-17T07:26:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.545+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fb9f4045070 con 0x7fba0c0c8210 2024-04-17T07:26:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.546+0000 7fba03400640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 0x7fb9e806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.546+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fb9f40ad900 con 0x7fba0c0c8210 2024-04-17T07:26:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.546+0000 7fba0b400640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 0x7fb9e806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.547+0000 7fba0b400640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 0x7fb9e806efe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005a90 tx=0x7fb9fc009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:23.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.551+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb9f407b930 con 0x7fba0c0c8210 2024-04-17T07:26:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:23.783+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fba0c0c7fc0 con 0x7fb9e806cb20 2024-04-17T07:26:24.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.002+0000 7fba03400640 1 -- 172.21.15.67:0/2194726576 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fba0c0c7fc0 con 0x7fb9e806cb20 2024-04-17T07:26:24.003 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 msgr2=0x7fb9e806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 0x7fb9e806efe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc005a90 tx=0x7fb9fc009040 comp rx=0 tx=0).stop 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 msgr2=0x7fba0c1f3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4003db0 tx=0x7fb9f40036f0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 shutdown_connections 2024-04-17T07:26:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fb9e806cb20 0x7fb9e806efe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba0c159c60 0x7fba0c15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba0c0c8210 0x7fba0c1f3ee0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 --2- 172.21.15.67:0/2194726576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f0004800 0x7fba0c0cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 >> 172.21.15.67:0/2194726576 conn(0x7fba0c08bff0 msgr2=0x7fba0c0b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 shutdown_connections 2024-04-17T07:26:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.006+0000 7fba10e00640 1 -- 172.21.15.67:0/2194726576 wait complete. 2024-04-17T07:26:24.028 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-04-17T07:26:24.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-04-17T07:26:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.254+0000 7fae93e00640 1 Processor -- start 2024-04-17T07:26:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.254+0000 7fae93e00640 1 -- start start 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 0x7fae940cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae940cd750 0x7fae9406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae9406f1d0 con 0x7fae9406c0a0 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fae940cdbb0 con 0x7fae940cd750 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae940727a0 con 0x7fae94145f80 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae9406c0a0 0x7fae94160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51362/0 (socket says 172.21.15.67:51362) 2024-04-17T07:26:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.256+0000 7fae93400640 1 -- 172.21.15.67:0/2897548170 learned_addr learned my addr 172.21.15.67:0/2897548170 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae92a00640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 0x7fae940cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae92000640 1 --2- 172.21.15.67:0/2897548170 >> v2:172.21.15.187:3300/0 conn(0x7fae940cd750 0x7fae9406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864746992 0 0) 0x7fae8c001490 con 0x7fae9406c0a0 2024-04-17T07:26:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae78003520 con 0x7fae9406c0a0 2024-04-17T07:26:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 548405331 0 0) 0x7fae78003520 con 0x7fae9406c0a0 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 msgr2=0x7fae940cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 0x7fae940cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 >> v2:172.21.15.187:3300/0 conn(0x7fae940cd750 msgr2=0x7fae9406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 --2- 172.21.15.67:0/2897548170 >> v2:172.21.15.187:3300/0 conn(0x7fae940cd750 0x7fae9406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae94146360 con 0x7fae9406c0a0 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fae8c001bc0 con 0x7fae9406c0a0 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 >> v1:172.21.15.67:6789/0 conn(0x7fae9406c0a0 legacy=0x7fae94160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae780039b0 0x7fae78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 0x7fae780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae940727a0 con 0x7fae78004800 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae7800b9c0 con 0x7fae780039b0 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.257+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae7800bb90 con 0x7fae94145f80 2024-04-17T07:26:24.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92a00640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae780039b0 0x7fae78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 -- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 msgr2=0x7fae780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae94145f80 0x7fae780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 -- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae780039b0 msgr2=0x7fae78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae780039b0 0x7fae78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92000640 1 -- 172.21.15.67:0/2897548170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae78003520 con 0x7fae78004800 2024-04-17T07:26:24.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.258+0000 7fae92a00640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae780039b0 0x7fae78003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.259+0000 7fae92000640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae78004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fae80009af0 tx=0x7fae78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f2e1f1c1d59520 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80039070 con 0x7fae78004800 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8002c710 con 0x7fae78004800 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae91600640 1 -- 172.21.15.67:0/2897548170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80009690 con 0x7fae78004800 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 -- 172.21.15.67:0/2897548170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 msgr2=0x7fae78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 --2- 172.21.15.67:0/2897548170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae78004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fae80009af0 tx=0x7fae78005830 comp rx=0 tx=0).stop 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93400640 1 -- 172.21.15.67:0/2897548170 reap_dead start 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 -- 172.21.15.67:0/2897548170 shutdown_connections 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 -- 172.21.15.67:0/2897548170 >> 172.21.15.67:0/2897548170 conn(0x7fae940b2190 msgr2=0x7fae940b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 -- 172.21.15.67:0/2897548170 shutdown_connections 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.260+0000 7fae93e00640 1 -- 172.21.15.67:0/2897548170 wait complete. 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.262+0000 7fae93e00640 1 Processor -- start 2024-04-17T07:26:24.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.262+0000 7fae93e00640 1 -- start start 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae941e4c50 0x7fae940cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 0x7fae94134c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae94072430 con 0x7fae78004800 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae94072150 con 0x7fae941347b0 2024-04-17T07:26:24.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.263+0000 7fae93e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae94072c30 con 0x7fae941e4c50 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42402/0 (socket says 172.21.15.67:42402) 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 -- 172.21.15.67:0/503368188 learned_addr learned my addr 172.21.15.67:0/503368188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92000640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae941e4c50 0x7fae940cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae93400640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 0x7fae94134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 -- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae941e4c50 msgr2=0x7fae940cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae941e4c50 0x7fae940cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 -- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 msgr2=0x7fae94134c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 0x7fae94134c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae92a00640 1 -- 172.21.15.67:0/503368188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae941e5410 con 0x7fae78004800 2024-04-17T07:26:24.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.264+0000 7fae93400640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 0x7fae94134c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:24.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae92a00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fae840108b0 tx=0x7fae84010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:24.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8400f040 con 0x7fae78004800 2024-04-17T07:26:24.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8400eb60 con 0x7fae78004800 2024-04-17T07:26:24.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae941e56f0 con 0x7fae78004800 2024-04-17T07:26:24.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8401c400 con 0x7fae78004800 2024-04-17T07:26:24.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.265+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae940c9940 con 0x7fae78004800 2024-04-17T07:26:24.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.266+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae94068a10 con 0x7fae78004800 2024-04-17T07:26:24.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.268+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fae8401c5a0 con 0x7fae78004800 2024-04-17T07:26:24.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.269+0000 7fae8aa00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 0x7fae7406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.269+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fae8408c5c0 con 0x7fae78004800 2024-04-17T07:26:24.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.270+0000 7fae92000640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 0x7fae7406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.271+0000 7fae92000640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 0x7fae7406efb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae80036000 tx=0x7fae80030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:24.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.274+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fae84018030 con 0x7fae78004800 2024-04-17T07:26:24.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.506+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fae940b1f70 con 0x7fae7406caf0 2024-04-17T07:26:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.726+0000 7fae8aa00640 1 -- 172.21.15.67:0/503368188 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fae940b1f70 con 0x7fae7406caf0 2024-04-17T07:26:24.726 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 msgr2=0x7fae7406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 0x7fae7406efb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae80036000 tx=0x7fae80030000 comp rx=0 tx=0).stop 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 msgr2=0x7fae94140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fae840108b0 tx=0x7fae84010d70 comp rx=0 tx=0).stop 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 shutdown_connections 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fae7406caf0 0x7fae7406efb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae941347b0 0x7fae94134c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae941e4c50 0x7fae940cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 --2- 172.21.15.67:0/503368188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae78004800 0x7fae94140c10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 >> 172.21.15.67:0/503368188 conn(0x7fae940b2190 msgr2=0x7fae9415cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 shutdown_connections 2024-04-17T07:26:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.729+0000 7fae93e00640 1 -- 172.21.15.67:0/503368188 wait complete. 2024-04-17T07:26:24.753 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-04-17T07:26:24.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-04-17T07:26:24.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.988+0000 7f2cdbe00640 1 Processor -- start 2024-04-17T07:26:24.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.988+0000 7f2cdbe00640 1 -- start start 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.989+0000 7f2cdbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.989+0000 7f2cdbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 0x7f2cdc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdbe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cdc0d00b0 con 0x7f2cdc0cf8e0 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2cdc0d0c30 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc0d0390 con 0x7f2cdc0d3210 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42408/0 (socket says 172.21.15.67:42408) 2024-04-17T07:26:24.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 -- 172.21.15.67:0/1642756391 learned_addr learned my addr 172.21.15.67:0/1642756391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cda000640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 0x7f2cdc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 -- 172.21.15.67:0/1642756391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 msgr2=0x7f2cdc0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 0x7f2cdc0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 -- 172.21.15.67:0/1642756391 >> v1:172.21.15.187:6789/0 conn(0x7f2cdc0cf8e0 legacy=0x7f2cdc0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.990+0000 7f2cdaa00640 1 -- 172.21.15.67:0/1642756391 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0d0670 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.991+0000 7f2cda000640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 0x7f2cdc0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:24.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.992+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0d0db0 tx=0x7f2cc402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0307c7e7dbc1e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:24.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.992+0000 7f2cd9600640 1 -- 172.21.15.67:0/1642756391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40317c0 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.992+0000 7f2cd9600640 1 -- 172.21.15.67:0/1642756391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cc4037070 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.992+0000 7f2cd9600640 1 -- 172.21.15.67:0/1642756391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4031b80 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 -- 172.21.15.67:0/1642756391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc15ecb0 msgr2=0x7f2cdc0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0d0db0 tx=0x7f2cc402b860 comp rx=0 tx=0).stop 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 -- 172.21.15.67:0/1642756391 shutdown_connections 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc0d3210 0x7f2cdc0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/1642756391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc15ecb0 0x7f2cdc0d2cd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 -- 172.21.15.67:0/1642756391 >> 172.21.15.67:0/1642756391 conn(0x7f2cdc08bff0 msgr2=0x7f2cdc0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 -- 172.21.15.67:0/1642756391 shutdown_connections 2024-04-17T07:26:24.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.993+0000 7f2cdbe00640 1 -- 172.21.15.67:0/1642756391 wait complete. 2024-04-17T07:26:24.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.995+0000 7f2cdbe00640 1 Processor -- start 2024-04-17T07:26:24.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.995+0000 7f2cdbe00640 1 -- start start 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 0x7f2cdc158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc15ecb0 0x7f2cdc1f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc15f090 con 0x7f2cdc0d3210 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0cfd70 con 0x7f2cdc0cf8e0 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc15f210 con 0x7f2cdc15ecb0 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42414/0 (socket says 172.21.15.67:42414) 2024-04-17T07:26:24.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 -- 172.21.15.67:0/3860845833 learned_addr learned my addr 172.21.15.67:0/3860845833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 0x7f2cdc158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 -- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc15ecb0 msgr2=0x7f2cdc1f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc15ecb0 0x7f2cdc1f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 -- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 msgr2=0x7f2cdc158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 0x7f2cdc158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cda000640 1 -- 172.21.15.67:0/3860845833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc159190 con 0x7f2cdc0d3210 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.996+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 0x7f2cdc158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cda000640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00cf60 tx=0x7f2ccc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc01a070 con 0x7f2cdc0d3210 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ccc00d840 con 0x7f2cdc0d3210 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00dc10 con 0x7f2cdc0d3210 2024-04-17T07:26:24.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc0cb7b0 con 0x7f2cdc0d3210 2024-04-17T07:26:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.997+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc160440 con 0x7f2cdc0d3210 2024-04-17T07:26:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.998+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc068a10 con 0x7f2cdc0d3210 2024-04-17T07:26:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:24.999+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f2ccc01d070 con 0x7f2cdc0d3210 2024-04-17T07:26:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.000+0000 7f2cd2a00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 0x7f2cb806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.000+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f2ccc08c7d0 con 0x7f2cdc0d3210 2024-04-17T07:26:25.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.001+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 0x7f2cb806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.002+0000 7f2cdaa00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 0x7f2cb806efe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4036000 tx=0x7f2cc4008030 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.005+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2ccc05a190 con 0x7f2cdc0d3210 2024-04-17T07:26:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.234+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2cdc15e2b0 con 0x7f2cb806cb20 2024-04-17T07:26:25.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.447+0000 7f2cd2a00640 1 -- 172.21.15.67:0/3860845833 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2cdc15e2b0 con 0x7f2cb806cb20 2024-04-17T07:26:25.447 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 msgr2=0x7f2cb806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 0x7f2cb806efe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4036000 tx=0x7f2cc4008030 comp rx=0 tx=0).stop 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 msgr2=0x7f2cdc0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00cf60 tx=0x7f2ccc00cf90 comp rx=0 tx=0).stop 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 shutdown_connections 2024-04-17T07:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2cb806cb20 0x7f2cb806efe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2cdc15ecb0 0x7f2cdc1f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cdc0d3210 0x7f2cdc0b4ea0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 --2- 172.21.15.67:0/3860845833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2cdc0cf8e0 0x7f2cdc158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 >> 172.21.15.67:0/3860845833 conn(0x7f2cdc08bff0 msgr2=0x7f2cdc06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 shutdown_connections 2024-04-17T07:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.451+0000 7f2cdbe00640 1 -- 172.21.15.67:0/3860845833 wait complete. 2024-04-17T07:26:25.472 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-04-17T07:26:25.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-04-17T07:26:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.699+0000 7f9f92c00640 1 Processor -- start 2024-04-17T07:26:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.699+0000 7f9f92c00640 1 -- start start 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9f8c15f110 0x7f9f8c072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c06f4f0 0x7f9f8c15f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c0d0af0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8c06f990 con 0x7f9f8c15f110 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f92c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c06fc70 con 0x7f9f8c06f4f0 2024-04-17T07:26:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f90c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f90c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42416/0 (socket says 172.21.15.67:42416) 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.701+0000 7f9f90c00640 1 -- 172.21.15.67:0/557393417 learned_addr learned my addr 172.21.15.67:0/557393417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.702+0000 7f9f90c00640 1 -- 172.21.15.67:0/557393417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c06f4f0 msgr2=0x7f9f8c15f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.702+0000 7f9f90c00640 1 --2- 172.21.15.67:0/557393417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c06f4f0 0x7f9f8c15f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.702+0000 7f9f90c00640 1 -- 172.21.15.67:0/557393417 >> v2:172.21.15.187:3300/0 conn(0x7f9f8c15f110 msgr2=0x7f9f8c072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.702+0000 7f9f90c00640 1 --2- 172.21.15.67:0/557393417 >> v2:172.21.15.187:3300/0 conn(0x7f9f8c15f110 0x7f9f8c072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.702+0000 7f9f90c00640 1 -- 172.21.15.67:0/557393417 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c0d04a0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.703+0000 7f9f90c00640 1 --2- 172.21.15.67:0/557393417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c070130 tx=0x7f9f8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd3b1d809354c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.703+0000 7f9f8b400640 1 -- 172.21.15.67:0/557393417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8402ca80 con 0x7f9f8c15fa30 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.703+0000 7f9f8b400640 1 -- 172.21.15.67:0/557393417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8402cc20 con 0x7f9f8c15fa30 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.703+0000 7f9f8b400640 1 -- 172.21.15.67:0/557393417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f840343f0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 -- 172.21.15.67:0/557393417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 msgr2=0x7f9f8c0ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 --2- 172.21.15.67:0/557393417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c070130 tx=0x7f9f8402bee0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 -- 172.21.15.67:0/557393417 shutdown_connections 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 --2- 172.21.15.67:0/557393417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c0ce3b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 --2- 172.21.15.67:0/557393417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c06f4f0 0x7f9f8c15f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 --2- 172.21.15.67:0/557393417 >> v2:172.21.15.187:3300/0 conn(0x7f9f8c15f110 0x7f9f8c072870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 -- 172.21.15.67:0/557393417 >> 172.21.15.67:0/557393417 conn(0x7f9f8c067ee0 msgr2=0x7f9f8c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:25.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 -- 172.21.15.67:0/557393417 shutdown_connections 2024-04-17T07:26:25.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.704+0000 7f9f92c00640 1 -- 172.21.15.67:0/557393417 wait complete. 2024-04-17T07:26:25.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.706+0000 7f9f92c00640 1 Processor -- start 2024-04-17T07:26:25.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.706+0000 7f9f92c00640 1 -- start start 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f8c06f4f0 0x7f9f8c158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c15f110 0x7f9f8c1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c15fec0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c0afdc0 con 0x7f9f8c06f4f0 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f92c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c072db0 con 0x7f9f8c15f110 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f90c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f90c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42426/0 (socket says 172.21.15.67:42426) 2024-04-17T07:26:25.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f90c00640 1 -- 172.21.15.67:0/279209323 learned_addr learned my addr 172.21.15.67:0/279209323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:25.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f83e00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c15f110 0x7f9f8c1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.707+0000 7f9f90c00640 1 -- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c15f110 msgr2=0x7f9f8c1f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f90c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c15f110 0x7f9f8c1f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f90c00640 1 -- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f8c06f4f0 msgr2=0x7f9f8c158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f90c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f8c06f4f0 0x7f9f8c158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f90c00640 1 -- 172.21.15.67:0/279209323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c1e7340 con 0x7f9f8c15fa30 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f90c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009f40 tx=0x7f9f84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84043020 con 0x7f9f8c15fa30 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8403e450 con 0x7f9f8c15fa30 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84009510 con 0x7f9f8c15fa30 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c1e7620 con 0x7f9f8c15fa30 2024-04-17T07:26:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.708+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c1e7be0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.709+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c068ca0 con 0x7f9f8c15fa30 2024-04-17T07:26:25.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.711+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f9f84046020 con 0x7f9f8c15fa30 2024-04-17T07:26:25.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.712+0000 7f9f88c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 0x7f9f6806f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:25.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.712+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f9f840aea50 con 0x7f9f8c15fa30 2024-04-17T07:26:25.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.712+0000 7f9f8be00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 0x7f9f6806f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:25.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.714+0000 7f9f8be00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 0x7f9f6806f000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009710 tx=0x7f9f7c006c90 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.716+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9f8404d030 con 0x7f9f8c15fa30 2024-04-17T07:26:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:25.946+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9f8c0cde70 con 0x7f9f6806cb40 2024-04-17T07:26:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.155+0000 7f9f88c00640 1 -- 172.21.15.67:0/279209323 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9f8c0cde70 con 0x7f9f6806cb40 2024-04-17T07:26:26.154 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-04-17T07:26:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 msgr2=0x7f9f6806f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 0x7f9f6806f000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009710 tx=0x7f9f7c006c90 comp rx=0 tx=0).stop 2024-04-17T07:26:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 msgr2=0x7f9f8c1e8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009f40 tx=0x7f9f84004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 shutdown_connections 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9f6806cb40 0x7f9f6806f000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c15fa30 0x7f9f8c1e8100 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f8c15f110 0x7f9f8c1f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 --2- 172.21.15.67:0/279209323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f8c06f4f0 0x7f9f8c158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 >> 172.21.15.67:0/279209323 conn(0x7f9f8c067ee0 msgr2=0x7f9f8c0ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 shutdown_connections 2024-04-17T07:26:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.159+0000 7f9f92c00640 1 -- 172.21.15.67:0/279209323 wait complete. 2024-04-17T07:26:26.184 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-04-17T07:26:26.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-04-17T07:26:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.408+0000 7f3cfa000640 1 Processor -- start 2024-04-17T07:26:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.408+0000 7f3cfa000640 1 -- start start 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.409+0000 7f3cfa000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3cf415e9b0 0x7f3cf40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.409+0000 7f3cfa000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cf40cf910 0x7f3cf40b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf40d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cfa000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf40b5790 con 0x7f3cf40cf910 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cfa000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf40d2830 con 0x7f3cf415e9b0 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cfa000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf40b51d0 con 0x7f3cf40b4c30 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cf40cf910 0x7f3cf40b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cf40cf910 0x7f3cf40b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42440/0 (socket says 172.21.15.67:42440) 2024-04-17T07:26:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 -- 172.21.15.67:0/2274317607 learned_addr learned my addr 172.21.15.67:0/2274317607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf8c00640 1 --2- 172.21.15.67:0/2274317607 >> v2:172.21.15.187:3300/0 conn(0x7f3cf415e9b0 0x7f3cf40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf9600640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf40d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 -- 172.21.15.67:0/2274317607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 msgr2=0x7f3cf40d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf40d4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 -- 172.21.15.67:0/2274317607 >> v2:172.21.15.187:3300/0 conn(0x7f3cf415e9b0 msgr2=0x7f3cf40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 --2- 172.21.15.67:0/2274317607 >> v2:172.21.15.187:3300/0 conn(0x7f3cf415e9b0 0x7f3cf40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.410+0000 7f3cf3e00640 1 -- 172.21.15.67:0/2274317607 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf40b54b0 con 0x7f3cf40cf910 2024-04-17T07:26:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.412+0000 7f3cf3e00640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf40cf910 0x7f3cf40b46f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40d29b0 tx=0x7f3ce002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbf986d5a637de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.412+0000 7f3cf3400640 1 -- 172.21.15.67:0/2274317607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00317c0 con 0x7f3cf40cf910 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.412+0000 7f3cf3400640 1 -- 172.21.15.67:0/2274317607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce0037070 con 0x7f3cf40cf910 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.412+0000 7f3cf3400640 1 -- 172.21.15.67:0/2274317607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0031b80 con 0x7f3cf40cf910 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 -- 172.21.15.67:0/2274317607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf40cf910 msgr2=0x7f3cf40b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf40cf910 0x7f3cf40b46f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40d29b0 tx=0x7f3ce002b860 comp rx=0 tx=0).stop 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 -- 172.21.15.67:0/2274317607 shutdown_connections 2024-04-17T07:26:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf40d4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2274317607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf40cf910 0x7f3cf40b46f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2274317607 >> v2:172.21.15.187:3300/0 conn(0x7f3cf415e9b0 0x7f3cf40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 -- 172.21.15.67:0/2274317607 >> 172.21.15.67:0/2274317607 conn(0x7f3cf409daa0 msgr2=0x7f3cf408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 -- 172.21.15.67:0/2274317607 shutdown_connections 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.413+0000 7f3cfa000640 1 -- 172.21.15.67:0/2274317607 wait complete. 2024-04-17T07:26:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.415+0000 7f3cfa000640 1 Processor -- start 2024-04-17T07:26:26.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.415+0000 7f3cfa000640 1 -- start start 2024-04-17T07:26:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf41f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3cf40cf910 0x7f3cf40cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf415ed90 con 0x7f3cf415e9b0 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4162830 con 0x7f3cf40cf910 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cfa000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf415ef10 con 0x7f3cf40b4c30 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cf9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cf9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42456/0 (socket says 172.21.15.67:42456) 2024-04-17T07:26:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.416+0000 7f3cf9600640 1 -- 172.21.15.67:0/2027774307 learned_addr learned my addr 172.21.15.67:0/2027774307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 -- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 msgr2=0x7f3cf41f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf41f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 -- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3cf40cf910 msgr2=0x7f3cf40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3cf40cf910 0x7f3cf40cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 -- 172.21.15.67:0/2027774307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf4159e30 con 0x7f3cf415e9b0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf9600640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00ccf0 tx=0x7f3cec00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec003b90 con 0x7f3cf415e9b0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec003d30 con 0x7f3cf415e9b0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec004240 con 0x7f3cf415e9b0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf415a110 con 0x7f3cf415e9b0 2024-04-17T07:26:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.417+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf415a6d0 con 0x7f3cf415e9b0 2024-04-17T07:26:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.419+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf40755d0 con 0x7f3cf415e9b0 2024-04-17T07:26:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.419+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f3cec0043e0 con 0x7f3cf415e9b0 2024-04-17T07:26:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.420+0000 7f3cf0c00640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 0x7f3cd406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.420+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f3cec08cb70 con 0x7f3cf415e9b0 2024-04-17T07:26:26.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.421+0000 7f3cf8c00640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 0x7f3cd406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:26.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.422+0000 7f3cf8c00640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 0x7f3cd406efb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8006fd0 tx=0x7f3ce8008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.425+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3cec05ab10 con 0x7f3cf415e9b0 2024-04-17T07:26:26.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.654+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3cf41e8320 con 0x7f3cd406caf0 2024-04-17T07:26:26.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.874+0000 7f3cf0c00640 1 -- 172.21.15.67:0/2027774307 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f3cf41e8320 con 0x7f3cd406caf0 2024-04-17T07:26:26.874 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.877+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 msgr2=0x7f3cd406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.877+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 0x7f3cd406efb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8006fd0 tx=0x7f3ce8008040 comp rx=0 tx=0).stop 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 msgr2=0x7f3cf40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00ccf0 tx=0x7f3cec00a660 comp rx=0 tx=0).stop 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 shutdown_connections 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3cd406caf0 0x7f3cd406efb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf415e9b0 0x7f3cf40cd840 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3cf40cf910 0x7f3cf40cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 --2- 172.21.15.67:0/2027774307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3cf40b4c30 0x7f3cf41f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 >> 172.21.15.67:0/2027774307 conn(0x7f3cf409daa0 msgr2=0x7f3cf40b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 shutdown_connections 2024-04-17T07:26:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:26.878+0000 7f3cfa000640 1 -- 172.21.15.67:0/2027774307 wait complete. 2024-04-17T07:26:26.900 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-04-17T07:26:26.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-04-17T07:26:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.121+0000 7f6fae600640 1 Processor -- start 2024-04-17T07:26:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.121+0000 7f6fae600640 1 -- start start 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6fae600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6fa806c270 0x7f6fa8167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6fae600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa806f1d0 con 0x7f6fa80cd920 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6fae600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa8146360 con 0x7f6fa8145f80 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6fae600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa80727a0 con 0x7f6fa806c270 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6f9fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6fa80cd920 0x7f6fa806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51364/0 (socket says 172.21.15.67:51364) 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6f9fe00640 1 -- 172.21.15.67:0/857657994 learned_addr learned my addr 172.21.15.67:0/857657994 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:27.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.123+0000 7f6fadc00640 1 --2- 172.21.15.67:0/857657994 >> v2:172.21.15.67:3300/0 conn(0x7f6fa806c270 0x7f6fa8167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.124+0000 7f6fadc00640 1 -- 172.21.15.67:0/857657994 >> v1:172.21.15.187:6789/0 conn(0x7f6fa8145f80 legacy=0x7f6fa80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:27.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.124+0000 7f6fadc00640 1 -- 172.21.15.67:0/857657994 >> v1:172.21.15.67:6789/0 conn(0x7f6fa80cd920 legacy=0x7f6fa806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.124+0000 7f6fadc00640 1 -- 172.21.15.67:0/857657994 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa8071e70 con 0x7f6fa806c270 2024-04-17T07:26:27.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.124+0000 7f6fa7e00640 1 -- 172.21.15.67:0/857657994 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851341487 0 0) 0x7f6f90001110 con 0x7f6fa80cd920 2024-04-17T07:26:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.125+0000 7f6fadc00640 1 --2- 172.21.15.67:0/857657994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa806c270 0x7f6fa8167380 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009d90 tx=0x7f6fa002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65358547a4adcd3a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.125+0000 7f6fa7e00640 1 -- 172.21.15.67:0/857657994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0039070 con 0x7f6fa806c270 2024-04-17T07:26:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.125+0000 7f6fa7e00640 1 -- 172.21.15.67:0/857657994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa002cbd0 con 0x7f6fa806c270 2024-04-17T07:26:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.125+0000 7f6fa7e00640 1 -- 172.21.15.67:0/857657994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0034440 con 0x7f6fa806c270 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 -- 172.21.15.67:0/857657994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa806c270 msgr2=0x7f6fa8167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 --2- 172.21.15.67:0/857657994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa806c270 0x7f6fa8167380 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009d90 tx=0x7f6fa002be20 comp rx=0 tx=0).stop 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 -- 172.21.15.67:0/857657994 shutdown_connections 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 --2- 172.21.15.67:0/857657994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa806c270 0x7f6fa8167380 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 -- 172.21.15.67:0/857657994 >> 172.21.15.67:0/857657994 conn(0x7f6fa80b2190 msgr2=0x7f6fa80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 -- 172.21.15.67:0/857657994 shutdown_connections 2024-04-17T07:26:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.126+0000 7f6fae600640 1 -- 172.21.15.67:0/857657994 wait complete. 2024-04-17T07:26:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.128+0000 7f6fae600640 1 Processor -- start 2024-04-17T07:26:27.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.128+0000 7f6fae600640 1 -- start start 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa806c270 0x7f6fa81e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6fa8145f80 0x7f6fa815b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa8072c30 con 0x7f6fa80cd920 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa8072430 con 0x7f6fa806c270 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6fae600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa8072db0 con 0x7f6fa8145f80 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6f9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6f9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42474/0 (socket says 172.21.15.67:42474) 2024-04-17T07:26:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.129+0000 7f6f9fe00640 1 -- 172.21.15.67:0/239421383 learned_addr learned my addr 172.21.15.67:0/239421383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fad200640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa806c270 0x7f6fa81e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 -- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6fa8145f80 msgr2=0x7f6fa815b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6fa8145f80 0x7f6fa815b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 -- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa806c270 msgr2=0x7f6fa81e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa806c270 0x7f6fa81e5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 -- 172.21.15.67:0/239421383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa81e6550 con 0x7f6fa80cd920 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6f9fe00640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c750 tx=0x7f6f9000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90018070 con 0x7f6fa80cd920 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f900072d0 con 0x7f6fa80cd920 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f900047e0 con 0x7f6fa80cd920 2024-04-17T07:26:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa81e6830 con 0x7f6fa80cd920 2024-04-17T07:26:27.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.130+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa81e6df0 con 0x7f6fa80cd920 2024-04-17T07:26:27.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.132+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa8068a10 con 0x7f6fa80cd920 2024-04-17T07:26:27.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.133+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f6f90004980 con 0x7f6fa80cd920 2024-04-17T07:26:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.134+0000 7f6fa5600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 0x7f6f8406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.134+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f6f9008b310 con 0x7f6fa80cd920 2024-04-17T07:26:27.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.134+0000 7f6fad200640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 0x7f6f8406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.136+0000 7f6fad200640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 0x7f6f8406efb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800a730 tx=0x7f6f98008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.138+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6fa806f1d0 con 0x7f6fa80cd920 2024-04-17T07:26:27.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:26:27.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:26:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.373+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6fa815e380 con 0x7f6f8406caf0 2024-04-17T07:26:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.568+0000 7f6fa5600640 1 -- 172.21.15.67:0/239421383 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f6fa815e380 con 0x7f6f8406caf0 2024-04-17T07:26:27.568 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.571+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 msgr2=0x7f6f8406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.571+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 0x7f6f8406efb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800a730 tx=0x7f6f98008040 comp rx=0 tx=0).stop 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.571+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 msgr2=0x7f6fa81e5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.571+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c750 tx=0x7f6f9000cc10 comp rx=0 tx=0).stop 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 shutdown_connections 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6f8406caf0 0x7f6f8406efb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6fa8145f80 0x7f6fa815b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fa80cd920 0x7f6fa81e5740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 --2- 172.21.15.67:0/239421383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6fa806c270 0x7f6fa81e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 >> 172.21.15.67:0/239421383 conn(0x7f6fa80b2190 msgr2=0x7f6fa815d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:27.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 shutdown_connections 2024-04-17T07:26:27.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.572+0000 7f6fae600640 1 -- 172.21.15.67:0/239421383 wait complete. 2024-04-17T07:26:27.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-04-17T07:26:27.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-04-17T07:26:27.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.823+0000 7fcf73e00640 1 Processor -- start 2024-04-17T07:26:27.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.823+0000 7fcf73e00640 1 -- start start 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.825+0000 7fcf73e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcf7406bfd0 0x7fcf740b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.825+0000 7fcf73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7406f800 0x7fcf740b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.825+0000 7fcf73e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf7406df40 con 0x7fcf7415b6b0 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.825+0000 7fcf73e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcf7406ed00 con 0x7fcf7406f800 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.825+0000 7fcf73e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcf7406e870 con 0x7fcf7406bfd0 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7406f800 0x7fcf740b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7406f800 0x7fcf740b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42476/0 (socket says 172.21.15.67:42476) 2024-04-17T07:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 -- 172.21.15.67:0/2601698495 learned_addr learned my addr 172.21.15.67:0/2601698495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 -- 172.21.15.67:0/2601698495 >> v2:172.21.15.187:3300/0 conn(0x7fcf7406bfd0 msgr2=0x7fcf740b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 --2- 172.21.15.67:0/2601698495 >> v2:172.21.15.187:3300/0 conn(0x7fcf7406bfd0 0x7fcf740b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 -- 172.21.15.67:0/2601698495 >> v1:172.21.15.187:6789/0 conn(0x7fcf7415b6b0 legacy=0x7fcf7406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.826+0000 7fcf73400640 1 -- 172.21.15.67:0/2601698495 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7406e220 con 0x7fcf7406f800 2024-04-17T07:26:27.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.828+0000 7fcf73400640 1 --2- 172.21.15.67:0/2601698495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7406f800 0x7fcf740b39d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcf7406e9f0 tx=0x7fcf6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c018e34044a9c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.828+0000 7fcf71600640 1 -- 172.21.15.67:0/2601698495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c02ca70 con 0x7fcf7406f800 2024-04-17T07:26:27.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.828+0000 7fcf71600640 1 -- 172.21.15.67:0/2601698495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf6c02cc10 con 0x7fcf7406f800 2024-04-17T07:26:27.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.828+0000 7fcf71600640 1 -- 172.21.15.67:0/2601698495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c0343f0 con 0x7fcf7406f800 2024-04-17T07:26:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 -- 172.21.15.67:0/2601698495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7406f800 msgr2=0x7fcf740b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2601698495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7406f800 0x7fcf740b39d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcf7406e9f0 tx=0x7fcf6c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 -- 172.21.15.67:0/2601698495 shutdown_connections 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2601698495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7406f800 0x7fcf740b39d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2601698495 >> v2:172.21.15.187:3300/0 conn(0x7fcf7406bfd0 0x7fcf740b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 -- 172.21.15.67:0/2601698495 >> 172.21.15.67:0/2601698495 conn(0x7fcf740b2190 msgr2=0x7fcf740b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 -- 172.21.15.67:0/2601698495 shutdown_connections 2024-04-17T07:26:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.829+0000 7fcf73e00640 1 -- 172.21.15.67:0/2601698495 wait complete. 2024-04-17T07:26:27.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.831+0000 7fcf73e00640 1 Processor -- start 2024-04-17T07:26:27.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.831+0000 7fcf73e00640 1 -- start start 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf7406bfd0 0x7fcf7415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf7406f800 0x7fcf74149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf740afee0 con 0x7fcf7415b6b0 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf740afd60 con 0x7fcf7406f800 2024-04-17T07:26:27.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.832+0000 7fcf73e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf740cfc10 con 0x7fcf7406bfd0 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42480/0 (socket says 172.21.15.67:42480) 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 -- 172.21.15.67:0/2796006701 learned_addr learned my addr 172.21.15.67:0/2796006701 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf72000640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf7406f800 0x7fcf74149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 -- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf7406bfd0 msgr2=0x7fcf7415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf7406bfd0 0x7fcf7415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 -- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf7406f800 msgr2=0x7fcf74149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf7406f800 0x7fcf74149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.833+0000 7fcf73400640 1 -- 172.21.15.67:0/2796006701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf74153e10 con 0x7fcf7415b6b0 2024-04-17T07:26:27.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf73400640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009f10 tx=0x7fcf6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c043020 con 0x7fcf7415b6b0 2024-04-17T07:26:27.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf6c03e450 con 0x7fcf7415b6b0 2024-04-17T07:26:27.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c03e820 con 0x7fcf7415b6b0 2024-04-17T07:26:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf741540f0 con 0x7fcf7415b6b0 2024-04-17T07:26:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.834+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf741546b0 con 0x7fcf7415b6b0 2024-04-17T07:26:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.835+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf74068a10 con 0x7fcf7415b6b0 2024-04-17T07:26:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.836+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fcf6c046020 con 0x7fcf7415b6b0 2024-04-17T07:26:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.838+0000 7fcf6b400640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 0x7fcf5406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.838+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fcf6c0ae6f0 con 0x7fcf7415b6b0 2024-04-17T07:26:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.839+0000 7fcf72a00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 0x7fcf5406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:27.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.840+0000 7fcf72a00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 0x7fcf5406efe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf64000f60 tx=0x7fcf640079b0 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:27.842+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcf6c04d070 con 0x7fcf7415b6b0 2024-04-17T07:26:28.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.075+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fcf7406ee80 con 0x7fcf5406cb20 2024-04-17T07:26:28.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.296+0000 7fcf6b400640 1 -- 172.21.15.67:0/2796006701 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fcf7406ee80 con 0x7fcf5406cb20 2024-04-17T07:26:28.299 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-04-17T07:26:28.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 msgr2=0x7fcf5406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 0x7fcf5406efe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf64000f60 tx=0x7fcf640079b0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 msgr2=0x7fcf741495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009f10 tx=0x7fcf6c004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 shutdown_connections 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fcf5406cb20 0x7fcf5406efe0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7415b6b0 0x7fcf741495b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf7406f800 0x7fcf74149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 --2- 172.21.15.67:0/2796006701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf7406bfd0 0x7fcf7415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 >> 172.21.15.67:0/2796006701 conn(0x7fcf740b2190 msgr2=0x7fcf74071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:28.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 shutdown_connections 2024-04-17T07:26:28.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.300+0000 7fcf73e00640 1 -- 172.21.15.67:0/2796006701 wait complete. 2024-04-17T07:26:28.325 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-04-17T07:26:28.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-04-17T07:26:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.552+0000 7fbed5600640 1 Processor -- start 2024-04-17T07:26:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.553+0000 7fbed5600640 1 -- start start 2024-04-17T07:26:28.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.554+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006bfd0 0x7fbed00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.554+0000 7fbed5600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbed006f800 0x7fbed00b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.554+0000 7fbed5600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed006df40 con 0x7fbed015b6b0 2024-04-17T07:26:28.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.554+0000 7fbed5600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbed006ed00 con 0x7fbed006f800 2024-04-17T07:26:28.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.554+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbed006e870 con 0x7fbed006bfd0 2024-04-17T07:26:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.555+0000 7fbed4c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbed006f800 0x7fbed00b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.555+0000 7fbecfe00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbed015b6b0 0x7fbed006b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:38582/0 (socket says 172.21.15.67:38582) 2024-04-17T07:26:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.555+0000 7fbecfe00640 1 -- 172.21.15.67:0/1065508481 learned_addr learned my addr 172.21.15.67:0/1065508481 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.556+0000 7fbec7e00640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006bfd0 0x7fbed00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.556+0000 7fbed4c00640 1 -- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006bfd0 msgr2=0x7fbed00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.556+0000 7fbed4c00640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006bfd0 0x7fbed00b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.556+0000 7fbed4c00640 1 -- 172.21.15.67:0/1065508481 >> v1:172.21.15.187:6789/0 conn(0x7fbed015b6b0 legacy=0x7fbed006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.556+0000 7fbed4c00640 1 -- 172.21.15.67:0/1065508481 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed006e220 con 0x7fbed006f800 2024-04-17T07:26:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.558+0000 7fbed4c00640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f800 0x7fbed00b39d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbec800bfa0 tx=0x7fbec802e010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5f537c914724ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.558+0000 7fbecf400640 1 -- 172.21.15.67:0/1065508481 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec802e9b0 con 0x7fbed006f800 2024-04-17T07:26:28.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbecf400640 1 -- 172.21.15.67:0/1065508481 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec802eb50 con 0x7fbed006f800 2024-04-17T07:26:28.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbecf400640 1 -- 172.21.15.67:0/1065508481 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8009540 con 0x7fbed006f800 2024-04-17T07:26:28.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 -- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f800 msgr2=0x7fbed00b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f800 0x7fbed00b39d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbec800bfa0 tx=0x7fbec802e010 comp rx=0 tx=0).stop 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 -- 172.21.15.67:0/1065508481 shutdown_connections 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f800 0x7fbed00b39d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 --2- 172.21.15.67:0/1065508481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006bfd0 0x7fbed00b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 -- 172.21.15.67:0/1065508481 >> 172.21.15.67:0/1065508481 conn(0x7fbed00b2190 msgr2=0x7fbed00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.559+0000 7fbed5600640 1 -- 172.21.15.67:0/1065508481 shutdown_connections 2024-04-17T07:26:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.560+0000 7fbed5600640 1 -- 172.21.15.67:0/1065508481 wait complete. 2024-04-17T07:26:28.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.562+0000 7fbed5600640 1 Processor -- start 2024-04-17T07:26:28.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.562+0000 7fbed5600640 1 -- start start 2024-04-17T07:26:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006bfd0 0x7fbed014be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006f800 0x7fbed00aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00afee0 con 0x7fbed015b6b0 2024-04-17T07:26:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00afd60 con 0x7fbed006bfd0 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.563+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00cfc10 con 0x7fbed006f800 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42496/0 (socket says 172.21.15.67:42496) 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 -- 172.21.15.67:0/1618055080 learned_addr learned my addr 172.21.15.67:0/1618055080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbecfe00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006bfd0 0x7fbed014be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbec7e00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006f800 0x7fbed00aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 -- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006f800 msgr2=0x7fbed00aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006f800 0x7fbed00aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 -- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006bfd0 msgr2=0x7fbed014be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006bfd0 0x7fbed014be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:28.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 -- 172.21.15.67:0/1618055080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed00a9830 con 0x7fbed015b6b0 2024-04-17T07:26:28.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.564+0000 7fbed4c00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbec800bfa0 tx=0x7fbec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:28.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.565+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec800b660 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.565+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec80092b0 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.565+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8040620 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.565+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed00a9b10 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.565+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed00aa0d0 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.566+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed00b1f00 con 0x7fbed015b6b0 2024-04-17T07:26:28.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.567+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fbec8047020 con 0x7fbed015b6b0 2024-04-17T07:26:28.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.568+0000 7fbeccc00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 0x7fbeac06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:28.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.568+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fbec80ae9d0 con 0x7fbed015b6b0 2024-04-17T07:26:28.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.569+0000 7fbecfe00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 0x7fbeac06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:28.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.570+0000 7fbecfe00640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 0x7fbeac06efb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbec0000f60 tx=0x7fbec0009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:28.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.573+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbed00b1f00 con 0x7fbed015b6b0 2024-04-17T07:26:28.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:28.806+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbed0068a10 con 0x7fbeac06caf0 2024-04-17T07:26:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.026+0000 7fbeccc00640 1 -- 172.21.15.67:0/1618055080 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbed0068a10 con 0x7fbeac06caf0 2024-04-17T07:26:29.027 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 msgr2=0x7fbeac06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 0x7fbeac06efb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbec0000f60 tx=0x7fbec0009040 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 msgr2=0x7fbed0155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbec800bfa0 tx=0x7fbec8004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 shutdown_connections 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbeac06caf0 0x7fbeac06efb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed015b6b0 0x7fbed0155a20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006f800 0x7fbed00aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 --2- 172.21.15.67:0/1618055080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006bfd0 0x7fbed014be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.030+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 >> 172.21.15.67:0/1618055080 conn(0x7fbed00b2190 msgr2=0x7fbed00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:29.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.031+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 shutdown_connections 2024-04-17T07:26:29.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.031+0000 7fbed5600640 1 -- 172.21.15.67:0/1618055080 wait complete. 2024-04-17T07:26:29.058 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-04-17T07:26:29.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:29.059 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-17T07:26:29.122 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:29.124 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_01_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_01_UTC', 'scheduled-2024-04-17-07_19_01_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_01_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:29.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-04-17T07:26:29.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.360+0000 7fbb67200640 1 Processor -- start 2024-04-17T07:26:29.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.360+0000 7fbb67200640 1 -- start start 2024-04-17T07:26:29.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.361+0000 7fbb67200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb6806b8c0 0x7fbb680c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.361+0000 7fbb67200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb67200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb680c8110 con 0x7fbb6815fd30 2024-04-17T07:26:29.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb67200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6815f250 con 0x7fbb6806b8c0 2024-04-17T07:26:29.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb67200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6815f530 con 0x7fbb6815ed10 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb66800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb65e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb6806b8c0 0x7fbb680c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb65e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb6806b8c0 0x7fbb680c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:47824/0 (socket says 172.21.15.67:47824) 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb65400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbb6815fd30 0x7fbb6806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:38590/0 (socket says 172.21.15.67:38590) 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.362+0000 7fbb66800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:46126/0 (socket says 172.21.15.67:46126) 2024-04-17T07:26:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb65e00640 1 -- 172.21.15.67:0/1130572787 learned_addr learned my addr 172.21.15.67:0/1130572787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb65e00640 1 -- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 msgr2=0x7fbb680cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb65e00640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb65e00640 1 -- 172.21.15.67:0/1130572787 >> v1:172.21.15.187:6789/0 conn(0x7fbb6815fd30 legacy=0x7fbb6806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.364+0000 7fbb65e00640 1 -- 172.21.15.67:0/1130572787 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6815f810 con 0x7fbb6806b8c0 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.364+0000 7fbb66800640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:26:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.363+0000 7fbb64a00640 1 -- 172.21.15.67:0/1130572787 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523962443 0 0) 0x7fbb680c8110 con 0x7fbb6815fd30 2024-04-17T07:26:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.365+0000 7fbb65e00640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6806b8c0 0x7fbb680c79c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009e50 tx=0x7fbb5802be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feaf96f57907ab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.365+0000 7fbb64a00640 1 -- 172.21.15.67:0/1130572787 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5802c9f0 con 0x7fbb6806b8c0 2024-04-17T07:26:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb64a00640 1 -- 172.21.15.67:0/1130572787 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5802cb90 con 0x7fbb6806b8c0 2024-04-17T07:26:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb64a00640 1 -- 172.21.15.67:0/1130572787 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb580343f0 con 0x7fbb6806b8c0 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 -- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6806b8c0 msgr2=0x7fbb680c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6806b8c0 0x7fbb680c79c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009e50 tx=0x7fbb5802be50 comp rx=0 tx=0).stop 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 -- 172.21.15.67:0/1130572787 shutdown_connections 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6815ed10 0x7fbb680cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 --2- 172.21.15.67:0/1130572787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6806b8c0 0x7fbb680c79c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 -- 172.21.15.67:0/1130572787 >> 172.21.15.67:0/1130572787 conn(0x7fbb680b2fe0 msgr2=0x7fbb680b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 -- 172.21.15.67:0/1130572787 shutdown_connections 2024-04-17T07:26:29.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.366+0000 7fbb67200640 1 -- 172.21.15.67:0/1130572787 wait complete. 2024-04-17T07:26:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.368+0000 7fbb67200640 1 Processor -- start 2024-04-17T07:26:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.368+0000 7fbb67200640 1 -- start start 2024-04-17T07:26:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6806b8c0 0x7fbb681502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6815fd30 0x7fbb680ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6806c3b0 con 0x7fbb6815ed10 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6806c530 con 0x7fbb6815fd30 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.369+0000 7fbb67200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6806c6b0 con 0x7fbb6806b8c0 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42502/0 (socket says 172.21.15.67:42502) 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 -- 172.21.15.67:0/3692073531 learned_addr learned my addr 172.21.15.67:0/3692073531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65e00640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6806b8c0 0x7fbb681502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb66800640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6815fd30 0x7fbb680ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 -- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6806b8c0 msgr2=0x7fbb681502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6806b8c0 0x7fbb681502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 -- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6815fd30 msgr2=0x7fbb680ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6815fd30 0x7fbb680ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.370+0000 7fbb65400640 1 -- 172.21.15.67:0/3692073531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb680b4a80 con 0x7fbb6815ed10 2024-04-17T07:26:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb65400640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000c430 tx=0x7fbb5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500074e0 con 0x7fbb6815ed10 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb500047e0 con 0x7fbb6815ed10 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb680b4d60 con 0x7fbb6815ed10 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50004bd0 con 0x7fbb6815ed10 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.371+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb680b5600 con 0x7fbb6815ed10 2024-04-17T07:26:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.372+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb680b5800 con 0x7fbb6815ed10 2024-04-17T07:26:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.373+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fbb50011070 con 0x7fbb6815ed10 2024-04-17T07:26:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.374+0000 7fbb5e000640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 0x7fbb4406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.374+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fbb5008bd60 con 0x7fbb6815ed10 2024-04-17T07:26:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.375+0000 7fbb65e00640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 0x7fbb4406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.376+0000 7fbb65e00640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 0x7fbb4406efb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb58035000 tx=0x7fbb58030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.379+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb680b5800 con 0x7fbb6815ed10 2024-04-17T07:26:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.607+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb68150840 con 0x7fbb4406caf0 2024-04-17T07:26:29.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.627+0000 7fbb5e000640 1 -- 172.21.15.67:0/3692073531 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fbb68150840 con 0x7fbb4406caf0 2024-04-17T07:26:29.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:29.626 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:32", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.629+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 msgr2=0x7fbb4406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.629+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 0x7fbb4406efb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbb58035000 tx=0x7fbb58030000 comp rx=0 tx=0).stop 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.629+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 msgr2=0x7fbb680b6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.629+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000c430 tx=0x7fbb5000c8f0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 shutdown_connections 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbb4406caf0 0x7fbb4406efb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb6815fd30 0x7fbb680ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb6815ed10 0x7fbb680b6660 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 --2- 172.21.15.67:0/3692073531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb6806b8c0 0x7fbb681502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 >> 172.21.15.67:0/3692073531 conn(0x7fbb680b2fe0 msgr2=0x7fbb680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 shutdown_connections 2024-04-17T07:26:29.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.630+0000 7fbb67200640 1 -- 172.21.15.67:0/3692073531 wait complete. 2024-04-17T07:26:29.648 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:32", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:29.648 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:32', 'first': '2024-04-17T07:16:01', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:29.648 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:29.648 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:29.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:29.648 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:29.700 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:29.701 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:29.701 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_01_UTC', 'scheduled-2024-04-17-07_18_01_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_01_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_01_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_01_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:29.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-04-17T07:26:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.926+0000 7f0285800640 1 Processor -- start 2024-04-17T07:26:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.926+0000 7f0285800640 1 -- start start 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.927+0000 7f0285800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f028006daf0 0x7f02800d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.927+0000 7f0285800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f0280070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.927+0000 7f0285800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02800d1520 con 0x7f02800d1060 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.927+0000 7f0285800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02800d1800 con 0x7f028006daf0 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.927+0000 7f0285800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02800d1ae0 con 0x7f028009b170 2024-04-17T07:26:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f028006daf0 0x7f02800d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02800d1060 0x7f028009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51368/0 (socket says 172.21.15.67:51368) 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027e000640 1 -- 172.21.15.67:0/1074628092 learned_addr learned my addr 172.21.15.67:0/1074628092 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 --2- 172.21.15.67:0/1074628092 >> v2:172.21.15.67:3300/0 conn(0x7f028006daf0 0x7f02800d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42506/0 (socket says 172.21.15.67:42506) 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027f400640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f0280070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 -- 172.21.15.67:0/1074628092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 msgr2=0x7f0280070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f0280070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 -- 172.21.15.67:0/1074628092 >> v1:172.21.15.67:6789/0 conn(0x7f02800d1060 legacy=0x7f028009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027ea00640 1 -- 172.21.15.67:0/1074628092 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800d1dc0 con 0x7f028006daf0 2024-04-17T07:26:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.928+0000 7f027d600640 1 -- 172.21.15.67:0/1074628092 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2829089843 0 0) 0x7f02800d1520 con 0x7f02800d1060 2024-04-17T07:26:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f027ea00640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800d0b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f02700097a0 tx=0x7f027002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142f54fb0e2b57e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f027d600640 1 -- 172.21.15.67:0/1074628092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270035020 con 0x7f028006daf0 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f027d600640 1 -- 172.21.15.67:0/1074628092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0270031830 con 0x7f028006daf0 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f027d600640 1 -- 172.21.15.67:0/1074628092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270031c00 con 0x7f028006daf0 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f0285800640 1 -- 172.21.15.67:0/1074628092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 msgr2=0x7f02800d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.930+0000 7f0285800640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800d0b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f02700097a0 tx=0x7f027002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 -- 172.21.15.67:0/1074628092 shutdown_connections 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f0280070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 --2- 172.21.15.67:0/1074628092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800d0b20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 -- 172.21.15.67:0/1074628092 >> 172.21.15.67:0/1074628092 conn(0x7f02800b41a0 msgr2=0x7f02800b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 -- 172.21.15.67:0/1074628092 shutdown_connections 2024-04-17T07:26:29.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.931+0000 7f0285800640 1 -- 172.21.15.67:0/1074628092 wait complete. 2024-04-17T07:26:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.932+0000 7f0285800640 1 Processor -- start 2024-04-17T07:26:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 -- start start 2024-04-17T07:26:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f02800cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02800d1060 0x7f02801f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f028006e050 con 0x7f028006daf0 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f028006ded0 con 0x7f02800d1060 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.933+0000 7f0285800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f028006ce00 con 0x7f028009b170 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42522/0 (socket says 172.21.15.67:42522) 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 -- 172.21.15.67:0/4165878805 learned_addr learned my addr 172.21.15.67:0/4165878805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027e000640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f02800cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027f400640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02800d1060 0x7f02801f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 -- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 msgr2=0x7f02800cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f02800cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 -- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02800d1060 msgr2=0x7f02801f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02800d1060 0x7f02801f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027ea00640 1 -- 172.21.15.67:0/4165878805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800b6e40 con 0x7f028006daf0 2024-04-17T07:26:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.934+0000 7f027e000640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f02800cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f027ea00640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02700098c0 tx=0x7f02700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270041070 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027003aa30 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02800b5f10 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270035040 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.935+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02800b64d0 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.936+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0280068ca0 con 0x7f028006daf0 2024-04-17T07:26:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.937+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f027004c060 con 0x7f028006daf0 2024-04-17T07:26:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.938+0000 7f0276a00640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 0x7f025c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.938+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f02700ada90 con 0x7f028006daf0 2024-04-17T07:26:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.939+0000 7f027e000640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 0x7f025c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:29.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.940+0000 7f027e000640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 0x7f025c06efb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0268007fd0 tx=0x7f0268009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:29.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:29.942+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f027007baa0 con 0x7f028006daf0 2024-04-17T07:26:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.169+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f02800c73d0 con 0x7f025c06caf0 2024-04-17T07:26:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.189+0000 7f0276a00640 1 -- 172.21.15.67:0/4165878805 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f02800c73d0 con 0x7f025c06caf0 2024-04-17T07:26:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:30.189 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:32", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 msgr2=0x7f025c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 0x7f025c06efb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0268007fd0 tx=0x7f0268009040 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 msgr2=0x7f02800b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02700098c0 tx=0x7f02700046c0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 shutdown_connections 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f025c06caf0 0x7f025c06efb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02800d1060 0x7f02801f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f028009b170 0x7f02800cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 --2- 172.21.15.67:0/4165878805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f028006daf0 0x7f02800b6860 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.192+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 >> 172.21.15.67:0/4165878805 conn(0x7f02800b41a0 msgr2=0x7f02800b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.193+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 shutdown_connections 2024-04-17T07:26:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.193+0000 7f0285800640 1 -- 172.21.15.67:0/4165878805 wait complete. 2024-04-17T07:26:30.210 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:32", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:30.210 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:32', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:30.210 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:30.210 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:30.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:30.210 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:30.268 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:30.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:30.269 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_01_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_01_UTC', 'scheduled-2024-04-17-07_25_01_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:30.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-04-17T07:26:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.498+0000 7f1039600640 1 Processor -- start 2024-04-17T07:26:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.498+0000 7f1039600640 1 -- start start 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.499+0000 7f1039600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 0x7f10340d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1039600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340d3210 0x7f10340cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1039600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 0x7f10340d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1039600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10340d3670 con 0x7f10340d3210 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1039600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10340d0c30 con 0x7f10340cf8e0 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1039600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10340d00b0 con 0x7f103415ecb0 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340d3210 0x7f10340cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340d3210 0x7f10340cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42526/0 (socket says 172.21.15.67:42526) 2024-04-17T07:26:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 -- 172.21.15.67:0/2685623436 learned_addr learned my addr 172.21.15.67:0/2685623436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:30.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1033e00640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 0x7f10340d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f1038c00640 1 --2- 172.21.15.67:0/2685623436 >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 0x7f10340d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 -- 172.21.15.67:0/2685623436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 msgr2=0x7f10340d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:30.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 0x7f10340d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 -- 172.21.15.67:0/2685623436 >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 msgr2=0x7f10340d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 --2- 172.21.15.67:0/2685623436 >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 0x7f10340d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.500+0000 7f102be00640 1 -- 172.21.15.67:0/2685623436 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10340d0390 con 0x7f10340d3210 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.501+0000 7f1033e00640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 0x7f10340d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.501+0000 7f1038c00640 1 --2- 172.21.15.67:0/2685623436 >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 0x7f10340d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.502+0000 7f102be00640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340d3210 0x7f10340cf3a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f10340d2200 tx=0x7f101c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca38ae5a759e1594 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.502+0000 7f1033400640 1 -- 172.21.15.67:0/2685623436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c02e070 con 0x7f10340d3210 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.502+0000 7f1033400640 1 -- 172.21.15.67:0/2685623436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f101c037070 con 0x7f10340d3210 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.502+0000 7f1033400640 1 -- 172.21.15.67:0/2685623436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101c031a00 con 0x7f10340d3210 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 -- 172.21.15.67:0/2685623436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340d3210 msgr2=0x7f10340cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340d3210 0x7f10340cf3a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f10340d2200 tx=0x7f101c02b880 comp rx=0 tx=0).stop 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 -- 172.21.15.67:0/2685623436 shutdown_connections 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 --2- 172.21.15.67:0/2685623436 >> v2:172.21.15.187:3300/0 conn(0x7f10340cf8e0 0x7f10340d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340d3210 0x7f10340cf3a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 --2- 172.21.15.67:0/2685623436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f103415ecb0 0x7f10340d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 -- 172.21.15.67:0/2685623436 >> 172.21.15.67:0/2685623436 conn(0x7f103408bff0 msgr2=0x7f10340d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 -- 172.21.15.67:0/2685623436 shutdown_connections 2024-04-17T07:26:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.503+0000 7f1039600640 1 -- 172.21.15.67:0/2685623436 wait complete. 2024-04-17T07:26:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.505+0000 7f1039600640 1 Processor -- start 2024-04-17T07:26:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.505+0000 7f1039600640 1 -- start start 2024-04-17T07:26:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.506+0000 7f1039600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10340cf8e0 0x7f1034158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.506+0000 7f1039600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10340d3210 0x7f10340b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1039600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1039600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10340cfd50 con 0x7f103415ecb0 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1039600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10340d0950 con 0x7f10340cf8e0 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1039600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f103415f090 con 0x7f10340d3210 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42538/0 (socket says 172.21.15.67:42538) 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 -- 172.21.15.67:0/2833029936 learned_addr learned my addr 172.21.15.67:0/2833029936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f102be00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10340d3210 0x7f10340b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1033e00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10340cf8e0 0x7f1034158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 -- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10340d3210 msgr2=0x7f10340b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10340d3210 0x7f10340b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 -- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10340cf8e0 msgr2=0x7f1034158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10340cf8e0 0x7f1034158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.507+0000 7f1038c00640 1 -- 172.21.15.67:0/2833029936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10341f5a70 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1038c00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cd40 tx=0x7f102c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c00f680 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f102c00f840 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c00fc30 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10341f5d50 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.508+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10341f6310 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.509+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1034068a10 con 0x7f103415ecb0 2024-04-17T07:26:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.511+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f102c0213f0 con 0x7f103415ecb0 2024-04-17T07:26:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.511+0000 7f1030c00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 0x7f101006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.512+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f102c08cf10 con 0x7f103415ecb0 2024-04-17T07:26:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.512+0000 7f1033e00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 0x7f101006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.514+0000 7f1033e00640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 0x7f101006efb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f102400a990 tx=0x7f1024008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.516+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f102c014030 con 0x7f103415ecb0 2024-04-17T07:26:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.740+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f10341f6550 con 0x7f101006caf0 2024-04-17T07:26:30.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.760+0000 7f1030c00640 1 -- 172.21.15.67:0/2833029936 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f102400a660 con 0x7f101006caf0 2024-04-17T07:26:30.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:30.765 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:33", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 msgr2=0x7f101006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 0x7f101006efb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f102400a990 tx=0x7f1024008040 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 msgr2=0x7f10340b8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f102c00cd40 tx=0x7f102c00a650 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 shutdown_connections 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f101006caf0 0x7f101006efb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f103415ecb0 0x7f10340b8630 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10340d3210 0x7f10340b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 --2- 172.21.15.67:0/2833029936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10340cf8e0 0x7f1034158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 >> 172.21.15.67:0/2833029936 conn(0x7f103408bff0 msgr2=0x7f10340b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 shutdown_connections 2024-04-17T07:26:30.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:30.763+0000 7f1039600640 1 -- 172.21.15.67:0/2833029936 wait complete. 2024-04-17T07:26:30.780 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:33", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:30.781 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:33', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:30.781 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:30.781 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:30.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:30.781 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-17T07:26:30.835 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:30.837 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_01_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_01_UTC', 'scheduled-2024-04-17-07_20_01_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_01_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_01_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:30.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-04-17T07:26:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.055+0000 7f2a99600640 1 Processor -- start 2024-04-17T07:26:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.055+0000 7f2a99600640 1 -- start start 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a99600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2a94159020 0x7f2a9415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a99600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a9406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a99600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9415f780 con 0x7f2a940af400 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a99600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2a941633b0 con 0x7f2a94159020 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a99600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2a9415fa60 con 0x7f2a9406b8c0 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a8be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a940af400 0x7f2a94162ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51378/0 (socket says 172.21.15.67:51378) 2024-04-17T07:26:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.057+0000 7f2a8be00640 1 -- 172.21.15.67:0/1833249653 learned_addr learned my addr 172.21.15.67:0/1833249653 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:31.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a98c00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a9406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93e00640 1 --2- 172.21.15.67:0/1833249653 >> v2:172.21.15.187:3300/0 conn(0x7f2a94159020 0x7f2a9415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271282444 0 0) 0x7f2a9415f780 con 0x7f2a940af400 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a78003520 con 0x7f2a940af400 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2263076681 0 0) 0x7f2a78003520 con 0x7f2a940af400 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 msgr2=0x7f2a9406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a9406bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 >> v2:172.21.15.187:3300/0 conn(0x7f2a94159020 msgr2=0x7f2a9415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 --2- 172.21.15.67:0/1833249653 >> v2:172.21.15.187:3300/0 conn(0x7f2a94159020 0x7f2a9415ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a941624a0 con 0x7f2a940af400 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2a7c001670 con 0x7f2a940af400 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 >> v1:172.21.15.67:6789/0 conn(0x7f2a940af400 legacy=0x7f2a94162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a780039b0 0x7f2a78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.058+0000 7f2a93400640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93400640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941633b0 con 0x7f2a78004d40 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9415fa60 con 0x7f2a780039b0 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7800b9c0 con 0x7f2a9406b8c0 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93e00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a780039b0 0x7f2a78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a8be00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 -- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 msgr2=0x7f2a78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 -- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a780039b0 msgr2=0x7f2a78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a780039b0 0x7f2a78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a98c00640 1 -- 172.21.15.67:0/1833249653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a78003520 con 0x7f2a78004d40 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a93e00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a780039b0 0x7f2a78003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.059+0000 7f2a8be00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9406b8c0 0x7f2a78004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:31.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.060+0000 7f2a98c00640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a780051c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009fc0 tx=0x7f2a78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6c0996f4266ee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c039070 con 0x7f2a78004d40 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8c02cbf0 con 0x7f2a78004d40 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a93400640 1 -- 172.21.15.67:0/1833249653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c009c10 con 0x7f2a78004d40 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a99600640 1 -- 172.21.15.67:0/1833249653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 msgr2=0x7f2a780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a99600640 1 --2- 172.21.15.67:0/1833249653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a780051c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009fc0 tx=0x7f2a78005830 comp rx=0 tx=0).stop 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a98c00640 1 -- 172.21.15.67:0/1833249653 reap_dead start 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a99600640 1 -- 172.21.15.67:0/1833249653 shutdown_connections 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a99600640 1 -- 172.21.15.67:0/1833249653 >> 172.21.15.67:0/1833249653 conn(0x7f2a940b2190 msgr2=0x7f2a940b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.061+0000 7f2a99600640 1 -- 172.21.15.67:0/1833249653 shutdown_connections 2024-04-17T07:26:31.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.062+0000 7f2a99600640 1 -- 172.21.15.67:0/1833249653 wait complete. 2024-04-17T07:26:31.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.063+0000 7f2a99600640 1 Processor -- start 2024-04-17T07:26:31.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.063+0000 7f2a99600640 1 -- start start 2024-04-17T07:26:31.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a9413c500 0x7f2a9415b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9415b8a0 0x7f2a94159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9415f370 con 0x7f2a78004d40 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9415fd40 con 0x7f2a9413c500 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.064+0000 7f2a99600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2a9415f4f0 con 0x7f2a9415b8a0 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42560/0 (socket says 172.21.15.67:42560) 2024-04-17T07:26:31.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 -- 172.21.15.67:0/349920963 learned_addr learned my addr 172.21.15.67:0/349920963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 -- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9415b8a0 msgr2=0x7f2a94159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9415b8a0 0x7f2a94159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 -- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a9413c500 msgr2=0x7f2a9415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a9413c500 0x7f2a9415b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 -- 172.21.15.67:0/349920963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9413fdf0 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.065+0000 7f2a93e00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2a84010a00 tx=0x7f2a84010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.066+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400f040 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.066+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8400eba0 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.066+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8401c400 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.066+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9415bf20 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.066+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9413f0b0 con 0x7f2a78004d40 2024-04-17T07:26:31.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.067+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9413eea0 con 0x7f2a78004d40 2024-04-17T07:26:31.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.068+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f2a8401c5a0 con 0x7f2a78004d40 2024-04-17T07:26:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.069+0000 7f2a90c00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 0x7f2a7006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.069+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f2a8408c6f0 con 0x7f2a78004d40 2024-04-17T07:26:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.070+0000 7f2a8be00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 0x7f2a7006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.071+0000 7f2a8be00640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 0x7f2a7006efb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0017e0 tx=0x7f2a7c009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.074+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2a9413fdf0 con 0x7f2a78004d40 2024-04-17T07:26:31.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.306+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a9415bcc0 con 0x7f2a7006caf0 2024-04-17T07:26:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.326+0000 7f2a90c00640 1 -- 172.21.15.67:0/349920963 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f2a7c00d840 con 0x7f2a7006caf0 2024-04-17T07:26:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:31.325 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:34", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 msgr2=0x7f2a7006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 0x7f2a7006efb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0017e0 tx=0x7f2a7c009040 comp rx=0 tx=0).stop 2024-04-17T07:26:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 msgr2=0x7f2a9413f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2a84010a00 tx=0x7f2a84010ec0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 shutdown_connections 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2a7006caf0 0x7f2a7006efb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a9415b8a0 0x7f2a94159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a9413c500 0x7f2a9415b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 --2- 172.21.15.67:0/349920963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a78004d40 0x7f2a9413f8b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 >> 172.21.15.67:0/349920963 conn(0x7f2a940b2190 msgr2=0x7f2a9406ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 shutdown_connections 2024-04-17T07:26:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.329+0000 7f2a99600640 1 -- 172.21.15.67:0/349920963 wait complete. 2024-04-17T07:26:31.348 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:34", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:31.348 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:34', 'first': '2024-04-17T07:16:01', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:31.349 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:31.349 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:31.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:31.349 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:31.405 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:31.406 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:31.406 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:31.407 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_01_UTC', 'scheduled-2024-04-17-07_19_01_UTC', 'scheduled-2024-04-17-07_20_01_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_01_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_01_UTC'] 2024-04-17T07:26:31.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-04-17T07:26:31.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.634+0000 7fe40b200640 1 Processor -- start 2024-04-17T07:26:31.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.634+0000 7fe40b200640 1 -- start start 2024-04-17T07:26:31.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.635+0000 7fe40b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe40c145f80 0x7fe40c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.635+0000 7fe40b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.635+0000 7fe40b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40c06f1d0 con 0x7fe40c06c0a0 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.635+0000 7fe40b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe40c0727a0 con 0x7fe40c145f80 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.635+0000 7fe40b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe40c146360 con 0x7fe40c0cd750 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe40c145f80 0x7fe40c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe40c145f80 0x7fe40c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42566/0 (socket says 172.21.15.67:42566) 2024-04-17T07:26:31.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 -- 172.21.15.67:0/2248594214 learned_addr learned my addr 172.21.15.67:0/2248594214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409400640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 -- 172.21.15.67:0/2248594214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 msgr2=0x7fe40c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 -- 172.21.15.67:0/2248594214 >> v1:172.21.15.67:6789/0 conn(0x7fe40c06c0a0 legacy=0x7fe40c160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409e00640 1 -- 172.21.15.67:0/2248594214 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c071e70 con 0x7fe40c145f80 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe408a00640 1 -- 172.21.15.67:0/2248594214 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242555097 0 0) 0x7fe404001490 con 0x7fe40c06c0a0 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.636+0000 7fe409400640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:31.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.637+0000 7fe409e00640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c145f80 0x7fe40c0cd210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c79e10af966ced server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe408a00640 1 -- 172.21.15.67:0/2248594214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc035020 con 0x7fe40c145f80 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe408a00640 1 -- 172.21.15.67:0/2248594214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3fc031640 con 0x7fe40c145f80 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe408a00640 1 -- 172.21.15.67:0/2248594214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc02d040 con 0x7fe40c145f80 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 -- 172.21.15.67:0/2248594214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c145f80 msgr2=0x7fe40c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c145f80 0x7fe40c0cd210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 -- 172.21.15.67:0/2248594214 shutdown_connections 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 --2- 172.21.15.67:0/2248594214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c145f80 0x7fe40c0cd210 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 -- 172.21.15.67:0/2248594214 >> 172.21.15.67:0/2248594214 conn(0x7fe40c0b2190 msgr2=0x7fe40c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.638+0000 7fe40b200640 1 -- 172.21.15.67:0/2248594214 shutdown_connections 2024-04-17T07:26:31.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.639+0000 7fe40b200640 1 -- 172.21.15.67:0/2248594214 wait complete. 2024-04-17T07:26:31.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.640+0000 7fe40b200640 1 Processor -- start 2024-04-17T07:26:31.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.640+0000 7fe40b200640 1 -- start start 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c15b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 0x7fe40c140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c072c30 con 0x7fe40c06c0a0 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c072430 con 0x7fe40c145f80 2024-04-17T07:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.641+0000 7fe40b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe40c072db0 con 0x7fe40c0cd750 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42572/0 (socket says 172.21.15.67:42572) 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 -- 172.21.15.67:0/2670823171 learned_addr learned my addr 172.21.15.67:0/2670823171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409400640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c15b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe40a800640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 0x7fe40c140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 -- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 msgr2=0x7fe40c15b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c15b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 -- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 msgr2=0x7fe40c140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 0x7fe40c140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe409e00640 1 -- 172.21.15.67:0/2670823171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c1413f0 con 0x7fe40c06c0a0 2024-04-17T07:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.642+0000 7fe40a800640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 0x7fe40c140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe409e00640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0090b0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc040070 con 0x7fe40c06c0a0 2024-04-17T07:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3fc039bf0 con 0x7fe40c06c0a0 2024-04-17T07:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc035040 con 0x7fe40c06c0a0 2024-04-17T07:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40c1416d0 con 0x7fe40c06c0a0 2024-04-17T07:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.643+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40c141c90 con 0x7fe40c06c0a0 2024-04-17T07:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.644+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40c068a10 con 0x7fe40c06c0a0 2024-04-17T07:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.646+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fe3fc038020 con 0x7fe40c06c0a0 2024-04-17T07:26:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.647+0000 7fe402000640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 0x7fe3ec06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.647+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fe3fc0adc10 con 0x7fe40c06c0a0 2024-04-17T07:26:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.647+0000 7fe409400640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 0x7fe3ec06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.648+0000 7fe409400640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 0x7fe3ec06efb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe40c15c740 tx=0x7fe3f8008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.651+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe3fc07bc20 con 0x7fe40c06c0a0 2024-04-17T07:26:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.887+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe40c15db90 con 0x7fe3ec06caf0 2024-04-17T07:26:31.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.907+0000 7fe402000640 1 -- 172.21.15.67:0/2670823171 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fe40c15db90 con 0x7fe3ec06caf0 2024-04-17T07:26:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:31.908 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:35", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:31.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 msgr2=0x7fe3ec06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 0x7fe3ec06efb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe40c15c740 tx=0x7fe3f8008040 comp rx=0 tx=0).stop 2024-04-17T07:26:31.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 msgr2=0x7fe40c142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0090b0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 shutdown_connections 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe3ec06caf0 0x7fe3ec06efb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.910+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe40c145f80 0x7fe40c140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.911+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe40c0cd750 0x7fe40c15b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.911+0000 7fe40b200640 1 --2- 172.21.15.67:0/2670823171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe40c06c0a0 0x7fe40c142490 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:31.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.911+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 >> 172.21.15.67:0/2670823171 conn(0x7fe40c0b2190 msgr2=0x7fe40c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:31.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.911+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 shutdown_connections 2024-04-17T07:26:31.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:31.911+0000 7fe40b200640 1 -- 172.21.15.67:0/2670823171 wait complete. 2024-04-17T07:26:31.930 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:35", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:31.931 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:35', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:31.931 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:31.931 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:31.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:31.931 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-17T07:26:31.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:31.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:31.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:31.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:31.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:31.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:31.988 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_01_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_01_UTC', 'scheduled-2024-04-17-07_23_01_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:31.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-04-17T07:26:32.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.216+0000 7f919b200640 1 Processor -- start 2024-04-17T07:26:32.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.217+0000 7f919b200640 1 -- start start 2024-04-17T07:26:32.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.218+0000 7f919b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f919c0b46f0 0x7f919c0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.218+0000 7f919b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c0d4510 0x7f919c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.218+0000 7f919b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c0b51d0 con 0x7f919c15e9b0 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.218+0000 7f919b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f919c0b5790 con 0x7f919c0d4510 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.218+0000 7f919b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f919c0d2830 con 0x7f919c0b46f0 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c0d4510 0x7f919c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c0d4510 0x7f919c0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42582/0 (socket says 172.21.15.67:42582) 2024-04-17T07:26:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 -- 172.21.15.67:0/1605951818 learned_addr learned my addr 172.21.15.67:0/1605951818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:32.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 -- 172.21.15.67:0/1605951818 >> v2:172.21.15.187:3300/0 conn(0x7f919c0b46f0 msgr2=0x7f919c0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 --2- 172.21.15.67:0/1605951818 >> v2:172.21.15.187:3300/0 conn(0x7f919c0b46f0 0x7f919c0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 -- 172.21.15.67:0/1605951818 >> v1:172.21.15.187:6789/0 conn(0x7f919c15e9b0 legacy=0x7f919c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.219+0000 7f919a800640 1 -- 172.21.15.67:0/1605951818 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c0b54b0 con 0x7f919c0d4510 2024-04-17T07:26:32.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.220+0000 7f919a800640 1 --2- 172.21.15.67:0/1605951818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0d4510 0x7f919c0d4920 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f919c08dee0 tx=0x7f919402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba0980c0a3c723c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.221+0000 7f9198a00640 1 -- 172.21.15.67:0/1605951818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919402c9f0 con 0x7f919c0d4510 2024-04-17T07:26:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.221+0000 7f9198a00640 1 -- 172.21.15.67:0/1605951818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919402cb90 con 0x7f919c0d4510 2024-04-17T07:26:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.221+0000 7f9198a00640 1 -- 172.21.15.67:0/1605951818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91940343f0 con 0x7f919c0d4510 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.221+0000 7f919b200640 1 -- 172.21.15.67:0/1605951818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0d4510 msgr2=0x7f919c0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.221+0000 7f919b200640 1 --2- 172.21.15.67:0/1605951818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0d4510 0x7f919c0d4920 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f919c08dee0 tx=0x7f919402be50 comp rx=0 tx=0).stop 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 -- 172.21.15.67:0/1605951818 shutdown_connections 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 --2- 172.21.15.67:0/1605951818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0d4510 0x7f919c0d4920 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 --2- 172.21.15.67:0/1605951818 >> v2:172.21.15.187:3300/0 conn(0x7f919c0b46f0 0x7f919c0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 -- 172.21.15.67:0/1605951818 >> 172.21.15.67:0/1605951818 conn(0x7f919c09daa0 msgr2=0x7f919c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 -- 172.21.15.67:0/1605951818 shutdown_connections 2024-04-17T07:26:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.222+0000 7f919b200640 1 -- 172.21.15.67:0/1605951818 wait complete. 2024-04-17T07:26:32.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.223+0000 7f919b200640 1 Processor -- start 2024-04-17T07:26:32.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.224+0000 7f919b200640 1 -- start start 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f919c0b46f0 0x7f919c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 0x7f919c14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f919c0d4e60 con 0x7f919c15e9b0 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f919c0cfae0 con 0x7f919c0d4510 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f919c162830 con 0x7f919c0b46f0 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42584/0 (socket says 172.21.15.67:42584) 2024-04-17T07:26:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 -- 172.21.15.67:0/721463098 learned_addr learned my addr 172.21.15.67:0/721463098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f9199400640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 0x7f919c14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 -- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f919c0b46f0 msgr2=0x7f919c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f9199e00640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f919c0b46f0 0x7f919c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f919c0b46f0 0x7f919c14e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 -- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 msgr2=0x7f919c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.225+0000 7f919a800640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 0x7f919c14b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f919a800640 1 -- 172.21.15.67:0/721463098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c0b7df0 con 0x7f919c15e9b0 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f9199400640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 0x7f919c14b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f919a800640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9194009f40 tx=0x7f9194004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194043020 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919403e450 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c0b80d0 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194009510 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.226+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c0b8690 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.228+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f919c0755d0 con 0x7f919c15e9b0 2024-04-17T07:26:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.229+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f9194046020 con 0x7f919c15e9b0 2024-04-17T07:26:32.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.230+0000 7f9192000640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 0x7f917806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.230+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f91940aea40 con 0x7f919c15e9b0 2024-04-17T07:26:32.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.230+0000 7f9199e00640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 0x7f917806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.231+0000 7f9199e00640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 0x7f917806efb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f918c0015d0 tx=0x7f918c009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.234+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f919404d030 con 0x7f919c15e9b0 2024-04-17T07:26:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.469+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f919c0b6e00 con 0x7f917806caf0 2024-04-17T07:26:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.488+0000 7f9192000640 1 -- 172.21.15.67:0/721463098 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f919c0b6e00 con 0x7f917806caf0 2024-04-17T07:26:32.488 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:32.488 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:35", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 msgr2=0x7f917806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 0x7f917806efb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f918c0015d0 tx=0x7f918c009040 comp rx=0 tx=0).stop 2024-04-17T07:26:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 msgr2=0x7f919c14ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9194009f40 tx=0x7f9194004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 shutdown_connections 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f917806caf0 0x7f917806efb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c15e9b0 0x7f919c14ba30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f919c0d4510 0x7f919c14b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 --2- 172.21.15.67:0/721463098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f919c0b46f0 0x7f919c14e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.491+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 >> 172.21.15.67:0/721463098 conn(0x7f919c09daa0 msgr2=0x7f919c06d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.492+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 shutdown_connections 2024-04-17T07:26:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.492+0000 7f919b200640 1 -- 172.21.15.67:0/721463098 wait complete. 2024-04-17T07:26:32.511 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:35", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:32.511 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:35', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:32.511 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:32.511 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:32.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:32.511 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:32.567 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_01_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_01_UTC', 'scheduled-2024-04-17-07_22_01_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_00_UTC'] 2024-04-17T07:26:32.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-04-17T07:26:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.797+0000 7f2014200640 1 Processor -- start 2024-04-17T07:26:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.797+0000 7f2014200640 1 -- start start 2024-04-17T07:26:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2014200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f200c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2014200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f200c0d17a0 0x7f200c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2014200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200c163860 con 0x7f200c0cfa10 2024-04-17T07:26:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2014200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f200c0d03b0 con 0x7f200c0d17a0 2024-04-17T07:26:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2014200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f200c0d1d40 con 0x7f200c15eca0 2024-04-17T07:26:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2010e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f200c0cfa10 0x7f200c0d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51390/0 (socket says 172.21.15.67:51390) 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.799+0000 7f2010e00640 1 -- 172.21.15.67:0/1526632896 learned_addr learned my addr 172.21.15.67:0/1526632896 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f2011800640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f200c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f2012200640 1 --2- 172.21.15.67:0/1526632896 >> v2:172.21.15.187:3300/0 conn(0x7f200c0d17a0 0x7f200c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326828106 0 0) 0x7f200c163860 con 0x7f200c0cfa10 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ff4003520 con 0x7f200c0cfa10 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3398475015 0 0) 0x7f1ff4003520 con 0x7f200c0cfa10 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 msgr2=0x7f200c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f200c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 >> v2:172.21.15.187:3300/0 conn(0x7f200c0d17a0 msgr2=0x7f200c0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 --2- 172.21.15.67:0/1526632896 >> v2:172.21.15.187:3300/0 conn(0x7f200c0d17a0 0x7f200c0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c0d2c30 con 0x7f200c0cfa10 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1ff8001670 con 0x7f200c0cfa10 2024-04-17T07:26:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 >> v1:172.21.15.67:6789/0 conn(0x7f200c0cfa10 legacy=0x7f200c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff40039b0 0x7f1ff4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f1ff4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff4004d40 0x7f1ff40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f200c0d1d40 con 0x7f1ff40039b0 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff400b9c0 con 0x7f1ff4004d40 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.800+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff400bb40 con 0x7f200c15eca0 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff40039b0 0x7f1ff4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2010e00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f1ff4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 -- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 msgr2=0x7f1ff4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f1ff4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2012200640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff4004d40 0x7f1ff40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 -- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff4004d40 msgr2=0x7f1ff40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ff4004d40 0x7f1ff40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2011800640 1 -- 172.21.15.67:0/1526632896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff4003520 con 0x7f1ff40039b0 2024-04-17T07:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.801+0000 7f2010e00640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f200c15eca0 0x7f1ff4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.802+0000 7f2011800640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff40039b0 0x7f1ff4003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2000009ac0 tx=0x7f1ff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0db4f7b19145147 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000039070 con 0x7f1ff40039b0 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200002c6b0 con 0x7f1ff40039b0 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f200be00640 1 -- 172.21.15.67:0/1526632896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000009690 con 0x7f1ff40039b0 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 -- 172.21.15.67:0/1526632896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff40039b0 msgr2=0x7f1ff4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 --2- 172.21.15.67:0/1526632896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff40039b0 0x7f1ff4003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2000009ac0 tx=0x7f1ff4005830 comp rx=0 tx=0).stop 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2012200640 1 -- 172.21.15.67:0/1526632896 reap_dead start 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 -- 172.21.15.67:0/1526632896 shutdown_connections 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 -- 172.21.15.67:0/1526632896 >> 172.21.15.67:0/1526632896 conn(0x7f200c067600 msgr2=0x7f200c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 -- 172.21.15.67:0/1526632896 shutdown_connections 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.803+0000 7f2014200640 1 -- 172.21.15.67:0/1526632896 wait complete. 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.805+0000 7f2014200640 1 Processor -- start 2024-04-17T07:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.805+0000 7f2014200640 1 -- start start 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 0x7f200c149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f200c15b0c0 0x7f200c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f200c0d34d0 con 0x7f200c0cdf60 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f200c0d31f0 con 0x7f200c15b0c0 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.806+0000 7f2014200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f200c15f080 con 0x7f1ff40039b0 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42594/0 (socket says 172.21.15.67:42594) 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 -- 172.21.15.67:0/3649784677 learned_addr learned my addr 172.21.15.67:0/3649784677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2011800640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 0x7f200c149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2012200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f200c15b0c0 0x7f200c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 -- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 msgr2=0x7f200c149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 0x7f200c149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 -- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f200c15b0c0 msgr2=0x7f200c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f200c15b0c0 0x7f200c15b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 -- 172.21.15.67:0/3649784677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c15ba80 con 0x7f200c0cdf60 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2011800640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 0x7f200c149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.807+0000 7f2010e00640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800d810 tx=0x7f1ff800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.808+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8019070 con 0x7f200c0cdf60 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.808+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff80074e0 con 0x7f200c0cdf60 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.808+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8004a20 con 0x7f200c0cdf60 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.808+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c0c8540 con 0x7f200c0cdf60 2024-04-17T07:26:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.808+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c0c8b00 con 0x7f200c0cdf60 2024-04-17T07:26:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.809+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200c068ca0 con 0x7f200c0cdf60 2024-04-17T07:26:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.810+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f1ff801c070 con 0x7f200c0cdf60 2024-04-17T07:26:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.811+0000 7f2009600640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 0x7f1fec06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.811+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f1ff808bbc0 con 0x7f200c0cdf60 2024-04-17T07:26:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.812+0000 7f2011800640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 0x7f1fec06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.813+0000 7f2011800640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 0x7f1fec06efb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f2000030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:32.815+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1ff8059b50 con 0x7f200c0cdf60 2024-04-17T07:26:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.043+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f200c0d0000 con 0x7f1fec06caf0 2024-04-17T07:26:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.061+0000 7f2009600640 1 -- 172.21.15.67:0/3649784677 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f200c0d0000 con 0x7f1fec06caf0 2024-04-17T07:26:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:33.061 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:36", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 msgr2=0x7f1fec06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 0x7f1fec06efb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f2000030000 comp rx=0 tx=0).stop 2024-04-17T07:26:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 msgr2=0x7f200c0ce3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800d810 tx=0x7f1ff800dcd0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 shutdown_connections 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f1fec06caf0 0x7f1fec06efb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f200c15b0c0 0x7f200c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200c0cdf60 0x7f200c0ce3e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 --2- 172.21.15.67:0/3649784677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ff40039b0 0x7f200c149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 >> 172.21.15.67:0/3649784677 conn(0x7f200c067600 msgr2=0x7f200c1629a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 shutdown_connections 2024-04-17T07:26:33.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.064+0000 7f2014200640 1 -- 172.21.15.67:0/3649784677 wait complete. 2024-04-17T07:26:33.084 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:36", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:33.084 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:36', 'first': '2024-04-17T07:16:01', 'last': '2024-04-17T07:26:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:33.084 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:33.084 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:33.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:33.085 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-17T07:26:33.140 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:33.141 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:33.142 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:33.143 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_01_UTC', 'scheduled-2024-04-17-07_21_01_UTC', 'scheduled-2024-04-17-07_22_00_UTC', 'scheduled-2024-04-17-07_23_01_UTC', 'scheduled-2024-04-17-07_24_01_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_01_UTC'] 2024-04-17T07:26:33.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-04-17T07:26:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.375+0000 7f452be00640 1 Processor -- start 2024-04-17T07:26:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.376+0000 7f452be00640 1 -- start start 2024-04-17T07:26:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c15ecb0 0x7f452c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 0x7f452c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452c0d00b0 con 0x7f452c0d3210 2024-04-17T07:26:33.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f452c0d0c30 con 0x7f452c15ecb0 2024-04-17T07:26:33.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f452c0d0390 con 0x7f452c0cfab0 2024-04-17T07:26:33.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.377+0000 7f452aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c15ecb0 0x7f452c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c15ecb0 0x7f452c0d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50844/0 (socket says 172.21.15.67:50844) 2024-04-17T07:26:33.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 -- 172.21.15.67:0/4202532873 learned_addr learned my addr 172.21.15.67:0/4202532873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452b400640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 0x7f452c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 -- 172.21.15.67:0/4202532873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 msgr2=0x7f452c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 0x7f452c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 -- 172.21.15.67:0/4202532873 >> v1:172.21.15.67:6789/0 conn(0x7f452c0d3210 legacy=0x7f452c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452aa00640 1 -- 172.21.15.67:0/4202532873 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c0d0670 con 0x7f452c15ecb0 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f4529600640 1 -- 172.21.15.67:0/4202532873 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511076310 0 0) 0x7f452c0d00b0 con 0x7f452c0d3210 2024-04-17T07:26:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.378+0000 7f452b400640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 0x7f452c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.379+0000 7f452aa00640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c15ecb0 0x7f452c0d2cd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f451c009870 tx=0x7f451c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d0bbf64c88efd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f4529600640 1 -- 172.21.15.67:0/4202532873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c035020 con 0x7f452c15ecb0 2024-04-17T07:26:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f4529600640 1 -- 172.21.15.67:0/4202532873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f451c031640 con 0x7f452c15ecb0 2024-04-17T07:26:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f4529600640 1 -- 172.21.15.67:0/4202532873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c02d040 con 0x7f452c15ecb0 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f452be00640 1 -- 172.21.15.67:0/4202532873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c15ecb0 msgr2=0x7f452c0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f452be00640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c15ecb0 0x7f452c0d2cd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f451c009870 tx=0x7f451c02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f452be00640 1 -- 172.21.15.67:0/4202532873 shutdown_connections 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f452be00640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c0cfab0 0x7f452c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.380+0000 7f452be00640 1 --2- 172.21.15.67:0/4202532873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c15ecb0 0x7f452c0d2cd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.381+0000 7f452be00640 1 -- 172.21.15.67:0/4202532873 >> 172.21.15.67:0/4202532873 conn(0x7f452c08bff0 msgr2=0x7f452c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.381+0000 7f452be00640 1 -- 172.21.15.67:0/4202532873 shutdown_connections 2024-04-17T07:26:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.381+0000 7f452be00640 1 -- 172.21.15.67:0/4202532873 wait complete. 2024-04-17T07:26:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.382+0000 7f452be00640 1 Processor -- start 2024-04-17T07:26:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.382+0000 7f452be00640 1 -- start start 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 0x7f452c0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c15ecb0 0x7f452c06ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f452c15f210 con 0x7f452c0d3210 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f452c15f090 con 0x7f452c0cfab0 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.383+0000 7f452be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f452c0b4500 con 0x7f452c15ecb0 2024-04-17T07:26:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50860/0 (socket says 172.21.15.67:50860) 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 -- 172.21.15.67:0/2634372117 learned_addr learned my addr 172.21.15.67:0/2634372117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452aa00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 0x7f452c0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 -- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c15ecb0 msgr2=0x7f452c06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452b400640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c15ecb0 0x7f452c06ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c15ecb0 0x7f452c06ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 -- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 msgr2=0x7f452c0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 0x7f452c0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452a000640 1 -- 172.21.15.67:0/2634372117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c0c8f00 con 0x7f452c0d3210 2024-04-17T07:26:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.384+0000 7f452aa00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 0x7f452c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f452a000640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f451400c4e0 tx=0x7f451400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514017070 con 0x7f452c0d3210 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45140035f0 con 0x7f452c0d3210 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45140039e0 con 0x7f452c0d3210 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c0ca8c0 con 0x7f452c0d3210 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.385+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c0cae70 con 0x7f452c0d3210 2024-04-17T07:26:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.386+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c068a10 con 0x7f452c0d3210 2024-04-17T07:26:33.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.387+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f451401a070 con 0x7f452c0d3210 2024-04-17T07:26:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.388+0000 7f4523400640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 0x7f450806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.388+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f451408b310 con 0x7f452c0d3210 2024-04-17T07:26:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.390+0000 7f452aa00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 0x7f450806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.392+0000 7f452aa00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 0x7f450806efe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f451c036000 tx=0x7f451c008030 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.394+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4514058cd0 con 0x7f452c0d3210 2024-04-17T07:26:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.623+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f452c0b5a70 con 0x7f450806cb20 2024-04-17T07:26:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.641+0000 7f4523400640 1 -- 172.21.15.67:0/2634372117 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f452c0b5a70 con 0x7f450806cb20 2024-04-17T07:26:33.640 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:33.640 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:37", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:33.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 msgr2=0x7f450806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 0x7f450806efe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f451c036000 tx=0x7f451c008030 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 msgr2=0x7f452c157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f451400c4e0 tx=0x7f451400c9a0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 shutdown_connections 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f450806cb20 0x7f450806efe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f452c15ecb0 0x7f452c06ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.643+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c0d3210 0x7f452c157cb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.644+0000 7f452be00640 1 --2- 172.21.15.67:0/2634372117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f452c0cfab0 0x7f452c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.644+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 >> 172.21.15.67:0/2634372117 conn(0x7f452c08bff0 msgr2=0x7f452c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.644+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 shutdown_connections 2024-04-17T07:26:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.644+0000 7f452be00640 1 -- 172.21.15.67:0/2634372117 wait complete. 2024-04-17T07:26:33.663 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:37", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:33.664 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:37', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:33.664 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:33.664 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:33.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:33.664 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-17T07:26:33.721 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:33.721 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:33.721 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:33.722 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:33.723 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_00_UTC', 'scheduled-2024-04-17-07_17_00_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_00_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_00_UTC', 'scheduled-2024-04-17-07_22_01_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_01_UTC', 'scheduled-2024-04-17-07_25_00_UTC', 'scheduled-2024-04-17-07_26_01_UTC'] 2024-04-17T07:26:33.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-04-17T07:26:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.950+0000 7f2287e00640 1 Processor -- start 2024-04-17T07:26:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.950+0000 7f2287e00640 1 -- start start 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22880b41a0 0x7f22880b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f22880b4580 0x7f2288162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f228806f800 0x7f2288162bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2288163110 con 0x7f228806f800 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2288166810 con 0x7f22880b4580 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2288166a60 con 0x7f22880b41a0 2024-04-17T07:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f228806f800 0x7f2288162bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f228806f800 0x7f2288162bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50876/0 (socket says 172.21.15.67:50876) 2024-04-17T07:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.952+0000 7f2287400640 1 -- 172.21.15.67:0/1755774250 learned_addr learned my addr 172.21.15.67:0/1755774250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2286a00640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22880b41a0 0x7f22880b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2286000640 1 --2- 172.21.15.67:0/1755774250 >> v2:172.21.15.187:3300/0 conn(0x7f22880b4580 0x7f2288162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2287400640 1 -- 172.21.15.67:0/1755774250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22880b41a0 msgr2=0x7f22880b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2287400640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22880b41a0 0x7f22880b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2287400640 1 -- 172.21.15.67:0/1755774250 >> v2:172.21.15.187:3300/0 conn(0x7f22880b4580 msgr2=0x7f2288162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2287400640 1 --2- 172.21.15.67:0/1755774250 >> v2:172.21.15.187:3300/0 conn(0x7f22880b4580 0x7f2288162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.953+0000 7f2287400640 1 -- 172.21.15.67:0/1755774250 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2288166d40 con 0x7f228806f800 2024-04-17T07:26:33.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.954+0000 7f2287400640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f228806f800 0x7f2288162bd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f22880701e0 tx=0x7f228002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d8ec186f879361 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.955+0000 7f2285600640 1 -- 172.21.15.67:0/1755774250 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228002ca80 con 0x7f228806f800 2024-04-17T07:26:33.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.955+0000 7f2285600640 1 -- 172.21.15.67:0/1755774250 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228002cc20 con 0x7f228806f800 2024-04-17T07:26:33.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.955+0000 7f2285600640 1 -- 172.21.15.67:0/1755774250 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22800343f0 con 0x7f228806f800 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 -- 172.21.15.67:0/1755774250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f228806f800 msgr2=0x7f2288162bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f228806f800 0x7f2288162bd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f22880701e0 tx=0x7f228002bee0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 -- 172.21.15.67:0/1755774250 shutdown_connections 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f228806f800 0x7f2288162bd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 --2- 172.21.15.67:0/1755774250 >> v2:172.21.15.187:3300/0 conn(0x7f22880b4580 0x7f2288162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 --2- 172.21.15.67:0/1755774250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22880b41a0 0x7f22880b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 -- 172.21.15.67:0/1755774250 >> 172.21.15.67:0/1755774250 conn(0x7f22880b2190 msgr2=0x7f22880b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 -- 172.21.15.67:0/1755774250 shutdown_connections 2024-04-17T07:26:33.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.956+0000 7f2287e00640 1 -- 172.21.15.67:0/1755774250 wait complete. 2024-04-17T07:26:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.958+0000 7f2287e00640 1 Processor -- start 2024-04-17T07:26:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.958+0000 7f2287e00640 1 -- start start 2024-04-17T07:26:33.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f228806f800 0x7f22880af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22880b41a0 0x7f22880a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2288071cd0 con 0x7f22880b4580 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2288167300 con 0x7f22880b41a0 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2288071e50 con 0x7f228806f800 2024-04-17T07:26:33.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50880/0 (socket says 172.21.15.67:50880) 2024-04-17T07:26:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2287400640 1 -- 172.21.15.67:0/3304790481 learned_addr learned my addr 172.21.15.67:0/3304790481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2286000640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22880b41a0 0x7f22880a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.959+0000 7f2286a00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f228806f800 0x7f22880af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 -- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f228806f800 msgr2=0x7f22880af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f228806f800 0x7f22880af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 -- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22880b41a0 msgr2=0x7f22880a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22880b41a0 0x7f22880a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 -- 172.21.15.67:0/3304790481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22880aa410 con 0x7f22880b4580 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287400640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2280009f40 tx=0x7f2280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280043020 con 0x7f22880b4580 2024-04-17T07:26:33.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228003e450 con 0x7f22880b4580 2024-04-17T07:26:33.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280009510 con 0x7f22880b4580 2024-04-17T07:26:33.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.960+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22880ac860 con 0x7f22880b4580 2024-04-17T07:26:33.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.961+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22880ace20 con 0x7f22880b4580 2024-04-17T07:26:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.962+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2288068a10 con 0x7f22880b4580 2024-04-17T07:26:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.963+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f2280046020 con 0x7f22880b4580 2024-04-17T07:26:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.964+0000 7f227ea00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 0x7f226806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.964+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7f22800ae840 con 0x7f22880b4580 2024-04-17T07:26:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.964+0000 7f2286a00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 0x7f226806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:33.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.966+0000 7f2286a00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 0x7f226806efb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:33.969+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f228004d030 con 0x7f22880b4580 2024-04-17T07:26:34.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.196+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f22881613d0 con 0x7f226806caf0 2024-04-17T07:26:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.217+0000 7f227ea00640 1 -- 172.21.15.67:0/3304790481 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f22881613d0 con 0x7f226806caf0 2024-04-17T07:26:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:38", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 msgr2=0x7f226806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 0x7f226806efb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2278006fd0 tx=0x7f2278008040 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 msgr2=0x7f228815fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2280009f40 tx=0x7f2280004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 shutdown_connections 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f226806caf0 0x7f226806efb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22880b4580 0x7f228815fa50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22880b41a0 0x7f22880a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 --2- 172.21.15.67:0/3304790481 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f228806f800 0x7f22880af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.220+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 >> 172.21.15.67:0/3304790481 conn(0x7f22880b2190 msgr2=0x7f22880cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.221+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 shutdown_connections 2024-04-17T07:26:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.221+0000 7f2287e00640 1 -- 172.21.15.67:0/3304790481 wait complete. 2024-04-17T07:26:34.240 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:38", "first": "2024-04-17T07:16:00", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:34.241 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:38', 'first': '2024-04-17T07:16:00', 'last': '2024-04-17T07:26:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:34.241 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:34.241 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:34.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:34.241 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:34.296 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:34.296 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:34.296 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_16_01_UTC', 'scheduled-2024-04-17-07_17_01_UTC', 'scheduled-2024-04-17-07_18_00_UTC', 'scheduled-2024-04-17-07_19_01_UTC', 'scheduled-2024-04-17-07_20_00_UTC', 'scheduled-2024-04-17-07_21_01_UTC', 'scheduled-2024-04-17-07_22_01_UTC', 'scheduled-2024-04-17-07_23_00_UTC', 'scheduled-2024-04-17-07_24_00_UTC', 'scheduled-2024-04-17-07_25_01_UTC', 'scheduled-2024-04-17-07_26_01_UTC'] 2024-04-17T07:26:34.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-04-17T07:26:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.511+0000 7fe971000640 1 Processor -- start 2024-04-17T07:26:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.511+0000 7fe971000640 1 -- start start 2024-04-17T07:26:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.512+0000 7fe971000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe96c0b0440 0x7fe96c1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.512+0000 7fe971000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96c0aef70 con 0x7fe96c1417f0 2024-04-17T07:26:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.512+0000 7fe971000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96c0af250 con 0x7fe96c142000 2024-04-17T07:26:34.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.512+0000 7fe971000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe96c163270 con 0x7fe96c0b0440 2024-04-17T07:26:34.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe96c1417f0 0x7fe96c162b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53766/0 (socket says 172.21.15.67:53766) 2024-04-17T07:26:34.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96b400640 1 -- 172.21.15.67:0/3006676943 learned_addr learned my addr 172.21.15.67:0/3006676943 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009435856 0 0) 0x7fe96c0aef70 con 0x7fe96c1417f0 2024-04-17T07:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe950003520 con 0x7fe96c1417f0 2024-04-17T07:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4021100464 0 0) 0x7fe950003520 con 0x7fe96c1417f0 2024-04-17T07:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 >> v2:172.21.15.187:3300/0 conn(0x7fe96c0b0440 msgr2=0x7fe96c1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 --2- 172.21.15.67:0/3006676943 >> v2:172.21.15.187:3300/0 conn(0x7fe96c0b0440 0x7fe96c1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 >> v1:172.21.15.187:6789/0 conn(0x7fe96c142000 legacy=0x7fe96c0afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96c0af530 con 0x7fe96c1417f0 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe9640019f0 con 0x7fe96c1417f0 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 >> v1:172.21.15.67:6789/0 conn(0x7fe96c1417f0 legacy=0x7fe96c162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.513+0000 7fe96a000640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe950003a00 0x7fe950003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96a000640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 0x7fe950004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96a000640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9500053c0 0x7fe95000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe96c163270 con 0x7fe950004800 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe95000c230 con 0x7fe950003a00 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe95000c430 con 0x7fe9500053c0 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 0x7fe950004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96b400640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe950003a00 0x7fe950003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 -- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9500053c0 msgr2=0x7fe95000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9500053c0 0x7fe95000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 -- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe950003a00 msgr2=0x7fe950003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe950003a00 0x7fe950003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96aa00640 1 -- 172.21.15.67:0/3006676943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe950003520 con 0x7fe950004800 2024-04-17T07:26:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.514+0000 7fe96be00640 1 -- 172.21.15.67:0/3006676943 reap_dead start 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe96aa00640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 0x7fe950004c80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe954001750 tx=0x7fe950005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8c68ebfa486d56 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954001370 con 0x7fe950004800 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9540052d0 con 0x7fe950004800 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe96a000640 1 -- 172.21.15.67:0/3006676943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954003830 con 0x7fe950004800 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe971000640 1 -- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 msgr2=0x7fe950004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.516+0000 7fe971000640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 0x7fe950004c80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fe954001750 tx=0x7fe950005830 comp rx=0 tx=0).stop 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.517+0000 7fe971000640 1 -- 172.21.15.67:0/3006676943 shutdown_connections 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.517+0000 7fe971000640 1 --2- 172.21.15.67:0/3006676943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe950004800 0x7fe950004c80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.517+0000 7fe971000640 1 -- 172.21.15.67:0/3006676943 >> 172.21.15.67:0/3006676943 conn(0x7fe96c072460 msgr2=0x7fe96c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.517+0000 7fe971000640 1 -- 172.21.15.67:0/3006676943 shutdown_connections 2024-04-17T07:26:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.517+0000 7fe971000640 1 -- 172.21.15.67:0/3006676943 wait complete. 2024-04-17T07:26:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.518+0000 7fe971000640 1 Processor -- start 2024-04-17T07:26:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.519+0000 7fe971000640 1 -- start start 2024-04-17T07:26:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.519+0000 7fe971000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe950004800 0x7fe96c1e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.519+0000 7fe971000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe971000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe96c1f8d00 0x7fe96c1e9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe971000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe96c141d50 con 0x7fe96c150d40 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe971000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe96c141bd0 con 0x7fe96c1f8d00 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe971000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe96c0aeb90 con 0x7fe950004800 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50898/0 (socket says 172.21.15.67:50898) 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 -- 172.21.15.67:0/2640269445 learned_addr learned my addr 172.21.15.67:0/2640269445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96b400640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe950004800 0x7fe96c1e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 -- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe950004800 msgr2=0x7fe96c1e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe950004800 0x7fe96c1e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 -- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe96c1f8d00 msgr2=0x7fe96c1e9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe96c1f8d00 0x7fe96c1e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.520+0000 7fe96aa00640 1 -- 172.21.15.67:0/2640269445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96c06e3a0 con 0x7fe96c150d40 2024-04-17T07:26:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe96aa00640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe954001870 tx=0x7fe954001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954041020 con 0x7fe96c150d40 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe954005480 con 0x7fe96c150d40 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954001370 con 0x7fe96c150d40 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe96c0b4440 con 0x7fe96c150d40 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.521+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96c1e8720 con 0x7fe96c150d40 2024-04-17T07:26:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.522+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe96c0755d0 con 0x7fe96c150d40 2024-04-17T07:26:34.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.523+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fe954044020 con 0x7fe96c150d40 2024-04-17T07:26:34.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.525+0000 7fe963400640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 0x7fe94806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:34.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.525+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fe9540adab0 con 0x7fe96c150d40 2024-04-17T07:26:34.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.525+0000 7fe96b400640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 0x7fe94806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.527+0000 7fe96b400640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 0x7fe94806efb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe9640013d0 tx=0x7fe964009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:34.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.529+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe95404b030 con 0x7fe96c150d40 2024-04-17T07:26:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.756+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe96c144820 con 0x7fe94806caf0 2024-04-17T07:26:34.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.774+0000 7fe963400640 1 -- 172.21.15.67:0/2640269445 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fe96c144820 con 0x7fe94806caf0 2024-04-17T07:26:34.774 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:34.774 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:39", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 msgr2=0x7fe94806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 0x7fe94806efb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe9640013d0 tx=0x7fe964009040 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 msgr2=0x7fe96c1e9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe954001870 tx=0x7fe954001120 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 shutdown_connections 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fe94806caf0 0x7fe94806efb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe96c1f8d00 0x7fe96c1e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe96c150d40 0x7fe96c1e9130 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 --2- 172.21.15.67:0/2640269445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe950004800 0x7fe96c1e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.777+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 >> 172.21.15.67:0/2640269445 conn(0x7fe96c072460 msgr2=0x7fe96c160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.778+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 shutdown_connections 2024-04-17T07:26:34.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:34.778+0000 7fe971000640 1 -- 172.21.15.67:0/2640269445 wait complete. 2024-04-17T07:26:34.796 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:15:39", "first": "2024-04-17T07:16:01", "last": "2024-04-17T07:26:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-04-17T07:26:34.796 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:15:39', 'first': '2024-04-17T07:16:01', 'last': '2024-04-17T07:26:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-04-17T07:26:34.797 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T07:26:34.797 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T07:26:34.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-04-17T07:26:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.027+0000 7fbf7ca00640 1 Processor -- start 2024-04-17T07:26:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.028+0000 7fbf7ca00640 1 -- start start 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.029+0000 7fbf7ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 0x7fbf780b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.029+0000 7fbf7ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf7806f800 0x7fbf780b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf7ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7806df40 con 0x7fbf7815b6b0 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf7ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbf7806ed00 con 0x7fbf7806f800 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf7ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbf7806e870 con 0x7fbf7806bfd0 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf7815b6b0 0x7fbf7806b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53768/0 (socket says 172.21.15.67:53768) 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77400640 1 -- 172.21.15.67:0/2643873103 learned_addr learned my addr 172.21.15.67:0/2643873103 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77e00640 1 --2- 172.21.15.67:0/2643873103 >> v2:172.21.15.67:3300/0 conn(0x7fbf7806f800 0x7fbf780b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf76a00640 1 --2- 172.21.15.67:0/2643873103 >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 0x7fbf780b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77e00640 1 -- 172.21.15.67:0/2643873103 >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 msgr2=0x7fbf780b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77e00640 1 --2- 172.21.15.67:0/2643873103 >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 0x7fbf780b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77e00640 1 -- 172.21.15.67:0/2643873103 >> v1:172.21.15.67:6789/0 conn(0x7fbf7815b6b0 legacy=0x7fbf7806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf77e00640 1 -- 172.21.15.67:0/2643873103 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7806e220 con 0x7fbf7806f800 2024-04-17T07:26:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.030+0000 7fbf76a00640 1 --2- 172.21.15.67:0/2643873103 >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 0x7fbf780b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.031+0000 7fbf77e00640 1 --2- 172.21.15.67:0/2643873103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7806f800 0x7fbf780b39d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbf7806e9f0 tx=0x7fbf7002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff40b65de614262 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.032+0000 7fbf76000640 1 -- 172.21.15.67:0/2643873103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70039070 con 0x7fbf7806f800 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.032+0000 7fbf76000640 1 -- 172.21.15.67:0/2643873103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7002cbd0 con 0x7fbf7806f800 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.032+0000 7fbf76000640 1 -- 172.21.15.67:0/2643873103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70034440 con 0x7fbf7806f800 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.032+0000 7fbf7ca00640 1 -- 172.21.15.67:0/2643873103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7806f800 msgr2=0x7fbf780b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.032+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/2643873103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7806f800 0x7fbf780b39d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbf7806e9f0 tx=0x7fbf7002be20 comp rx=0 tx=0).stop 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 -- 172.21.15.67:0/2643873103 shutdown_connections 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/2643873103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7806f800 0x7fbf780b39d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/2643873103 >> v2:172.21.15.187:3300/0 conn(0x7fbf7806bfd0 0x7fbf780b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 -- 172.21.15.67:0/2643873103 >> 172.21.15.67:0/2643873103 conn(0x7fbf780b2190 msgr2=0x7fbf780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 -- 172.21.15.67:0/2643873103 shutdown_connections 2024-04-17T07:26:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.033+0000 7fbf7ca00640 1 -- 172.21.15.67:0/2643873103 wait complete. 2024-04-17T07:26:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.035+0000 7fbf7ca00640 1 Processor -- start 2024-04-17T07:26:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.035+0000 7fbf7ca00640 1 -- start start 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf7806bfd0 0x7fbf78148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf7806f800 0x7fbf7815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf780afee0 con 0x7fbf7815b6b0 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbf780afd60 con 0x7fbf7806bfd0 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf7ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf780cfc10 con 0x7fbf7806f800 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50916/0 (socket says 172.21.15.67:50916) 2024-04-17T07:26:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 -- 172.21.15.67:0/551093192 learned_addr learned my addr 172.21.15.67:0/551093192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 -- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf7806f800 msgr2=0x7fbf7815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf7806f800 0x7fbf7815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 -- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf7806bfd0 msgr2=0x7fbf78148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77400640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf7806bfd0 0x7fbf78148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf7806bfd0 0x7fbf78148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.036+0000 7fbf77e00640 1 -- 172.21.15.67:0/551093192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7815eec0 con 0x7fbf7815b6b0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf77e00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf7002c430 tx=0x7fbf70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70044020 con 0x7fbf7815b6b0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7003f670 con 0x7fbf7815b6b0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70039040 con 0x7fbf7815b6b0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7815f1a0 con 0x7fbf7815b6b0 2024-04-17T07:26:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.037+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7815f760 con 0x7fbf7815b6b0 2024-04-17T07:26:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.038+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf78068a10 con 0x7fbf7815b6b0 2024-04-17T07:26:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.039+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7fbf70047020 con 0x7fbf7815b6b0 2024-04-17T07:26:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.041+0000 7fbf6f400640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 0x7fbf5806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.041+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7fbf700aece0 con 0x7fbf7815b6b0 2024-04-17T07:26:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.041+0000 7fbf77400640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 0x7fbf5806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.043+0000 7fbf77400640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 0x7fbf5806efe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000f60 tx=0x7fbf68009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.045+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbf7004e030 con 0x7fbf7815b6b0 2024-04-17T07:26:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.275+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbf7806ee80 con 0x7fbf5806cb20 2024-04-17T07:26:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.530+0000 7fbf6f400640 1 -- 172.21.15.67:0/551093192 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7fbf7806ee80 con 0x7fbf5806cb20 2024-04-17T07:26:35.529 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 msgr2=0x7fbf5806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 0x7fbf5806efe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf68000f60 tx=0x7fbf68009040 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 msgr2=0x7fbf78071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbf7002c430 tx=0x7fbf70004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 shutdown_connections 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fbf5806cb20 0x7fbf5806efe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf7815b6b0 0x7fbf78071cd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf7806f800 0x7fbf7815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.533+0000 7fbf7ca00640 1 --2- 172.21.15.67:0/551093192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf7806bfd0 0x7fbf78148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.534+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 >> 172.21.15.67:0/551093192 conn(0x7fbf780b2190 msgr2=0x7fbf780b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.534+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 shutdown_connections 2024-04-17T07:26:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:35.534+0000 7fbf7ca00640 1 -- 172.21.15.67:0/551093192 wait complete. 2024-04-17T07:26:35.555 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-04-17T07:26:35.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.555 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:35.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:35.615 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:35.615 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:35.615 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:35.615 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:35.616 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:35.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.616 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_16_01_UTC) 2024-04-17T07:26:35.686 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:35.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.686 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:35.752 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:35.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.752 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_18_01_UTC) 2024-04-17T07:26:35.821 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:35.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.822 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_19_01_UTC) 2024-04-17T07:26:35.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:35.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.896 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:35.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:35.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:35.965 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_21_01_UTC) 2024-04-17T07:26:36.033 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:36.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.034 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:36.107 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:36.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.107 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:36.181 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:36.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.181 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:36.254 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:36.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.254 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:36.325 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:36.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.325 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:36.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:36.395 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-04-17T07:26:36.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-04-17T07:26:36.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.690+0000 7efcf7200640 1 Processor -- start 2024-04-17T07:26:36.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.690+0000 7efcf7200640 1 -- start start 2024-04-17T07:26:36.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efcf7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcf80cd920 0x7efcf806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efcf7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efcf806c0a0 0x7efcf806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efcf7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf8146360 con 0x7efcf8145f80 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efcf7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efcf80727a0 con 0x7efcf80cd920 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efcf7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efcf8071e70 con 0x7efcf806c0a0 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efced400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcf80cd920 0x7efcf806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efced400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcf80cd920 0x7efcf806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50922/0 (socket says 172.21.15.67:50922) 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.692+0000 7efced400640 1 -- 172.21.15.67:0/3005387684 learned_addr learned my addr 172.21.15.67:0/3005387684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efcf6800640 1 --2- 172.21.15.67:0/3005387684 >> v2:172.21.15.187:3300/0 conn(0x7efcf806c0a0 0x7efcf806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efced400640 1 -- 172.21.15.67:0/3005387684 >> v2:172.21.15.187:3300/0 conn(0x7efcf806c0a0 msgr2=0x7efcf806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efced400640 1 --2- 172.21.15.67:0/3005387684 >> v2:172.21.15.187:3300/0 conn(0x7efcf806c0a0 0x7efcf806e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efced400640 1 -- 172.21.15.67:0/3005387684 >> v1:172.21.15.67:6789/0 conn(0x7efcf8145f80 legacy=0x7efcf80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efced400640 1 -- 172.21.15.67:0/3005387684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf8072150 con 0x7efcf80cd920 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.693+0000 7efcf5400640 1 -- 172.21.15.67:0/3005387684 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211912430 0 0) 0x7efcf8146360 con 0x7efcf8145f80 2024-04-17T07:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.694+0000 7efced400640 1 --2- 172.21.15.67:0/3005387684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf806bb60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcf815cc90 tx=0x7efce002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359f1a45ea702489 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.694+0000 7efcf5400640 1 -- 172.21.15.67:0/3005387684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0035020 con 0x7efcf80cd920 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.694+0000 7efcf5400640 1 -- 172.21.15.67:0/3005387684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce0031870 con 0x7efcf80cd920 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.694+0000 7efcf5400640 1 -- 172.21.15.67:0/3005387684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0031c40 con 0x7efcf80cd920 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 -- 172.21.15.67:0/3005387684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 msgr2=0x7efcf806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 --2- 172.21.15.67:0/3005387684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf806bb60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcf815cc90 tx=0x7efce002b910 comp rx=0 tx=0).stop 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 -- 172.21.15.67:0/3005387684 shutdown_connections 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 --2- 172.21.15.67:0/3005387684 >> v2:172.21.15.187:3300/0 conn(0x7efcf806c0a0 0x7efcf806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 --2- 172.21.15.67:0/3005387684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf806bb60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 -- 172.21.15.67:0/3005387684 >> 172.21.15.67:0/3005387684 conn(0x7efcf80b2190 msgr2=0x7efcf80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 -- 172.21.15.67:0/3005387684 shutdown_connections 2024-04-17T07:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.695+0000 7efcf7200640 1 -- 172.21.15.67:0/3005387684 wait complete. 2024-04-17T07:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.697+0000 7efcf7200640 1 Processor -- start 2024-04-17T07:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.697+0000 7efcf7200640 1 -- start start 2024-04-17T07:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 0x7efcf815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcf8145f80 0x7efcf80ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efcf8072c30 con 0x7efcf80cd920 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efcf806c530 con 0x7efcf8145f80 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efcf7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcf8072db0 con 0x7efcf806c0a0 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efced400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efced400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50928/0 (socket says 172.21.15.67:50928) 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.698+0000 7efced400640 1 -- 172.21.15.67:0/49367945 learned_addr learned my addr 172.21.15.67:0/49367945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efcf5e00640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 0x7efcf815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efcf6800640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcf8145f80 0x7efcf80ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 -- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 msgr2=0x7efcf815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 0x7efcf815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 -- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcf8145f80 msgr2=0x7efcf80ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcf8145f80 0x7efcf80ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 -- 172.21.15.67:0/49367945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf80aabd0 con 0x7efcf80cd920 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efcf5e00640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 0x7efcf815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efced400640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efce0009960 tx=0x7efce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0035040 con 0x7efcf80cd920 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce0045480 con 0x7efcf80cd920 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce0045870 con 0x7efcf80cd920 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.699+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf80aaeb0 con 0x7efcf80cd920 2024-04-17T07:26:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.700+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf815c0b0 con 0x7efcf80cd920 2024-04-17T07:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.701+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf80cec90 con 0x7efcf80cd920 2024-04-17T07:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.702+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7efce0038020 con 0x7efcf80cd920 2024-04-17T07:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.702+0000 7efceea00640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 0x7efcd406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.703+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5967+0+0 (secure 0 0 0) 0x7efce00ae440 con 0x7efcf80cd920 2024-04-17T07:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.703+0000 7efcf5e00640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 0x7efcd406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:36.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.704+0000 7efcf5e00640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 0x7efcd406efb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efce8000f60 tx=0x7efce8009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.707+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efcf80cec90 con 0x7efcf80cd920 2024-04-17T07:26:36.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:36.939+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7efcf806eba0 con 0x7efcd406caf0 2024-04-17T07:26:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.192+0000 7efceea00640 1 -- 172.21.15.67:0/49367945 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7efcf806eba0 con 0x7efcd406caf0 2024-04-17T07:26:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.195+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 msgr2=0x7efcd406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.195+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 0x7efcd406efb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efce8000f60 tx=0x7efce8009040 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.195+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 msgr2=0x7efcf81337a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.195+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efce0009960 tx=0x7efce00046c0 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 shutdown_connections 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7efcd406caf0 0x7efcd406efb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcf8145f80 0x7efcf80ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf80cd920 0x7efcf81337a0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 --2- 172.21.15.67:0/49367945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcf806c0a0 0x7efcf815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 >> 172.21.15.67:0/49367945 conn(0x7efcf80b2190 msgr2=0x7efcf815cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 shutdown_connections 2024-04-17T07:26:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:37.196+0000 7efcf7200640 1 -- 172.21.15.67:0/49367945 wait complete. 2024-04-17T07:26:37.215 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-04-17T07:26:37.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.216 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:37.271 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:37.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.271 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:37.340 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:37.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.341 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_17_01_UTC) 2024-04-17T07:26:37.411 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:37.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.411 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_18_01_UTC) 2024-04-17T07:26:37.485 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:37.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.485 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:37.562 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:37.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.562 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_20_01_UTC) 2024-04-17T07:26:37.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:37.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.630 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:37.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:37.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.698 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:37.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:37.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.768 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_23_01_UTC) 2024-04-17T07:26:37.841 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:37.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.841 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:37.905 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:37.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.906 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_25_01_UTC) 2024-04-17T07:26:37.963 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:37.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:37.964 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:38.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:38.030 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-04-17T07:26:38.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-04-17T07:26:38.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.310+0000 7f42a7200640 1 Processor -- start 2024-04-17T07:26:38.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.310+0000 7f42a7200640 1 -- start start 2024-04-17T07:26:38.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.312+0000 7f42a7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f42a8145f80 0x7f42a80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.312+0000 7f42a7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42a806c270 0x7f42a806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.312+0000 7f42a7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42a8146360 con 0x7f42a80cd750 2024-04-17T07:26:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.312+0000 7f42a7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f42a80727a0 con 0x7f42a806c270 2024-04-17T07:26:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.312+0000 7f42a7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f42a8071e70 con 0x7f42a8145f80 2024-04-17T07:26:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.313+0000 7f42a6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42a806c270 0x7f42a806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:38.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.313+0000 7f42a6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42a806c270 0x7f42a806e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50934/0 (socket says 172.21.15.67:50934) 2024-04-17T07:26:38.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.313+0000 7f42a6800640 1 -- 172.21.15.67:0/2081100759 learned_addr learned my addr 172.21.15.67:0/2081100759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:38.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.313+0000 7f42a5e00640 1 --2- 172.21.15.67:0/2081100759 >> v2:172.21.15.187:3300/0 conn(0x7f42a8145f80 0x7f42a80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:38.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.314+0000 7f42a6800640 1 -- 172.21.15.67:0/2081100759 >> v2:172.21.15.187:3300/0 conn(0x7f42a8145f80 msgr2=0x7f42a80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:38.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.314+0000 7f42a6800640 1 --2- 172.21.15.67:0/2081100759 >> v2:172.21.15.187:3300/0 conn(0x7f42a8145f80 0x7f42a80cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.314+0000 7f42a6800640 1 -- 172.21.15.67:0/2081100759 >> v1:172.21.15.187:6789/0 conn(0x7f42a80cd750 legacy=0x7f42a806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:38.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.314+0000 7f42a6800640 1 -- 172.21.15.67:0/2081100759 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a8072150 con 0x7f42a806c270 2024-04-17T07:26:38.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.316+0000 7f42a6800640 1 --2- 172.21.15.67:0/2081100759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a806c270 0x7f42a806e1c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f42a806c9c0 tx=0x7f42a002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41c1a99f1bf4474 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:38.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.316+0000 7f42a4a00640 1 -- 172.21.15.67:0/2081100759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a002c9f0 con 0x7f42a806c270 2024-04-17T07:26:38.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.317+0000 7f42a4a00640 1 -- 172.21.15.67:0/2081100759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42a002cb90 con 0x7f42a806c270 2024-04-17T07:26:38.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.317+0000 7f42a4a00640 1 -- 172.21.15.67:0/2081100759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a00343f0 con 0x7f42a806c270 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 -- 172.21.15.67:0/2081100759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a806c270 msgr2=0x7f42a806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 --2- 172.21.15.67:0/2081100759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a806c270 0x7f42a806e1c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f42a806c9c0 tx=0x7f42a002be50 comp rx=0 tx=0).stop 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 -- 172.21.15.67:0/2081100759 shutdown_connections 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 --2- 172.21.15.67:0/2081100759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a806c270 0x7f42a806e1c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 --2- 172.21.15.67:0/2081100759 >> v2:172.21.15.187:3300/0 conn(0x7f42a8145f80 0x7f42a80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.318+0000 7f42a7200640 1 -- 172.21.15.67:0/2081100759 >> 172.21.15.67:0/2081100759 conn(0x7f42a80b2190 msgr2=0x7f42a80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:38.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.319+0000 7f42a7200640 1 -- 172.21.15.67:0/2081100759 shutdown_connections 2024-04-17T07:26:38.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.319+0000 7f42a7200640 1 -- 172.21.15.67:0/2081100759 wait complete. 2024-04-17T07:26:38.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.321+0000 7f42a7200640 1 Processor -- start 2024-04-17T07:26:38.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.322+0000 7f42a7200640 1 -- start start 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42a806c270 0x7f42a80aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 0x7f42a80b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42a8072db0 con 0x7f42a8145f80 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f42a8072c30 con 0x7f42a80cd750 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f42a806e700 con 0x7f42a806c270 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50940/0 (socket says 172.21.15.67:50940) 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.323+0000 7f42a6800640 1 -- 172.21.15.67:0/2127922795 learned_addr learned my addr 172.21.15.67:0/2127922795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a5400640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 0x7f42a80b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a6800640 1 -- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42a806c270 msgr2=0x7f42a80aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a6800640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42a806c270 0x7f42a80aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a6800640 1 -- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 msgr2=0x7f42a80b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a6800640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 0x7f42a80b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a6800640 1 -- 172.21.15.67:0/2127922795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a813f200 con 0x7f42a8145f80 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.324+0000 7f42a5400640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 0x7f42a80b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.326+0000 7f42a6800640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f42a0036040 tx=0x7f42a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.327+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a0043020 con 0x7f42a8145f80 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.327+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42a003e450 con 0x7f42a8145f80 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.327+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a003e860 con 0x7f42a8145f80 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.327+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a8145700 con 0x7f42a8145f80 2024-04-17T07:26:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.328+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a8140930 con 0x7f42a8145f80 2024-04-17T07:26:38.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.331+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a80b1820 con 0x7f42a8145f80 2024-04-17T07:26:38.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.331+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f42a0046020 con 0x7f42a8145f80 2024-04-17T07:26:38.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.333+0000 7f429e000640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 0x7f428806f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:38.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.333+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6631+0+0 (secure 0 0 0) 0x7f42a00aeb00 con 0x7f42a8145f80 2024-04-17T07:26:38.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.334+0000 7f42a5e00640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 0x7f428806f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:38.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.337+0000 7f42a5e00640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 0x7f428806f000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4298006a90 tx=0x7f4298006a20 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.341+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f42a80b1820 con 0x7f42a8145f80 2024-04-17T07:26:38.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.607+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f42a80ce690 con 0x7f428806cb40 2024-04-17T07:26:38.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.838+0000 7f429e000640 1 -- 172.21.15.67:0/2127922795 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f42a80ce690 con 0x7f428806cb40 2024-04-17T07:26:38.839 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-04-17T07:26:38.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 msgr2=0x7f428806f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 0x7f428806f000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4298006a90 tx=0x7f4298006a20 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 msgr2=0x7f42a80b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f42a0036040 tx=0x7f42a0004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 shutdown_connections 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f428806cb40 0x7f428806f000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42a8145f80 0x7f42a80b5180 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42a80cd750 0x7f42a80b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 --2- 172.21.15.67:0/2127922795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42a806c270 0x7f42a80aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 >> 172.21.15.67:0/2127922795 conn(0x7f42a80b2190 msgr2=0x7f42a80b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 shutdown_connections 2024-04-17T07:26:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:38.841+0000 7f42a7200640 1 -- 172.21.15.67:0/2127922795 wait complete. 2024-04-17T07:26:38.860 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-04-17T07:26:38.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:38.861 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-04-17T07:26:38.904 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:38.905 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:38.906 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:38.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:38.906 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:39.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:39.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.040 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_17_01_UTC) 2024-04-17T07:26:39.122 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:39.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.122 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:39.173 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:39.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.174 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:39.230 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:39.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.231 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:39.294 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:39.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.294 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:39.362 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:39.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.362 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:39.414 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:39.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.414 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:39.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:39.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.467 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_24_01_UTC) 2024-04-17T07:26:39.523 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:39.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.523 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_25_01_UTC) 2024-04-17T07:26:39.583 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:39.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.584 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:39.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:39.648 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-04-17T07:26:39.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-04-17T07:26:39.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.944+0000 7f2b4b200640 1 Processor -- start 2024-04-17T07:26:39.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.945+0000 7f2b4b200640 1 -- start start 2024-04-17T07:26:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.946+0000 7f2b4b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b440b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.946+0000 7f2b4b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2b44162da0 0x7f2b4406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.946+0000 7f2b4b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b44166a60 con 0x7f2b440b41a0 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.946+0000 7f2b4b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2b44166810 con 0x7f2b44162da0 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.946+0000 7f2b4b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b44166d40 con 0x7f2b440b4580 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.947+0000 7f2b49e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b440b41a0 0x7f2b44162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53786/0 (socket says 172.21.15.67:53786) 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.947+0000 7f2b49e00640 1 -- 172.21.15.67:0/3865160730 learned_addr learned my addr 172.21.15.67:0/3865160730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:39.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.947+0000 7f2b49400640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b440b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.947+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> v2:172.21.15.187:3300/0 conn(0x7f2b44162da0 0x7f2b4406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.947+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229490683 0 0) 0x7f2b44166a60 con 0x7f2b440b41a0 2024-04-17T07:26:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b2c003520 con 0x7f2b440b41a0 2024-04-17T07:26:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1391096675 0 0) 0x7f2b2c003520 con 0x7f2b440b41a0 2024-04-17T07:26:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 msgr2=0x7f2b440b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b440b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 >> v2:172.21.15.187:3300/0 conn(0x7f2b44162da0 msgr2=0x7f2b4406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 --2- 172.21.15.67:0/3865160730 >> v2:172.21.15.187:3300/0 conn(0x7f2b44162da0 0x7f2b4406dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b44167020 con 0x7f2b440b41a0 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2b38001670 con 0x7f2b440b41a0 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 >> v1:172.21.15.67:6789/0 conn(0x7f2b440b41a0 legacy=0x7f2b44162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b2c0039b0 0x7f2b2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c004d40 0x7f2b2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44166810 con 0x7f2b2c004d40 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44166d40 con 0x7f2b2c0039b0 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b2c00b9f0 con 0x7f2b440b4580 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.948+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> v2:172.21.15.187:3300/0 conn(0x7f2b44162da0 0x7f2b4406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c004d40 0x7f2b2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b49e00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b2c0039b0 0x7f2b2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b49400640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 -- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 msgr2=0x7f2b2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440b4580 0x7f2b2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 -- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b2c0039b0 msgr2=0x7f2b2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b2c0039b0 0x7f2b2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b4a800640 1 -- 172.21.15.67:0/3865160730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2c003520 con 0x7f2b2c004d40 2024-04-17T07:26:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.949+0000 7f2b49e00640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b2c0039b0 0x7f2b2c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.950+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c004d40 0x7f2b2c0051c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2b400073e0 tx=0x7f2b2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46972511bfd44a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.951+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40039070 con 0x7f2b2c004d40 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.951+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4002cdc0 con 0x7f2b2c004d40 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.951+0000 7f2b48a00640 1 -- 172.21.15.67:0/3865160730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40009c70 con 0x7f2b2c004d40 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 -- 172.21.15.67:0/3865160730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c004d40 msgr2=0x7f2b2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:39.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3865160730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c004d40 0x7f2b2c0051c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2b400073e0 tx=0x7f2b2c005830 comp rx=0 tx=0).stop 2024-04-17T07:26:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4a800640 1 -- 172.21.15.67:0/3865160730 reap_dead start 2024-04-17T07:26:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 -- 172.21.15.67:0/3865160730 shutdown_connections 2024-04-17T07:26:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 -- 172.21.15.67:0/3865160730 >> 172.21.15.67:0/3865160730 conn(0x7f2b440b2190 msgr2=0x7f2b440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 -- 172.21.15.67:0/3865160730 shutdown_connections 2024-04-17T07:26:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.952+0000 7f2b4b200640 1 -- 172.21.15.67:0/3865160730 wait complete. 2024-04-17T07:26:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.954+0000 7f2b4b200640 1 Processor -- start 2024-04-17T07:26:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.954+0000 7f2b4b200640 1 -- start start 2024-04-17T07:26:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.955+0000 7f2b4b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b4414c890 0x7f2b44165d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.955+0000 7f2b4b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b4b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440725c0 0x7f2b44072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b4b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4406f980 con 0x7f2b441646a0 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b4b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4406f800 con 0x7f2b4414c890 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b4b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4406fb00 con 0x7f2b440725c0 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50946/0 (socket says 172.21.15.67:50946) 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 -- 172.21.15.67:0/3612920597 learned_addr learned my addr 172.21.15.67:0/3612920597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49e00640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b4414c890 0x7f2b44165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b4a800640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440725c0 0x7f2b44072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 -- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440725c0 msgr2=0x7f2b44072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440725c0 0x7f2b44072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 -- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b4414c890 msgr2=0x7f2b44165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b4414c890 0x7f2b44165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:39.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.956+0000 7f2b49400640 1 -- 172.21.15.67:0/3612920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4415a820 con 0x7f2b441646a0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b49400640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2b34011660 tx=0x7f2b34011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400f960 con 0x7f2b441646a0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3400fb00 con 0x7f2b441646a0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b34015460 con 0x7f2b441646a0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b44164b80 con 0x7f2b441646a0 2024-04-17T07:26:39.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.957+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b44165560 con 0x7f2b441646a0 2024-04-17T07:26:39.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.959+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b44068a10 con 0x7f2b441646a0 2024-04-17T07:26:39.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.960+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f2b34015600 con 0x7f2b441646a0 2024-04-17T07:26:39.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.961+0000 7f2b3e000640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 0x7f2b2806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:39.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.961+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6359+0+0 (secure 0 0 0) 0x7f2b3408caf0 con 0x7f2b441646a0 2024-04-17T07:26:39.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.961+0000 7f2b49e00640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 0x7f2b2806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:39.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.962+0000 7f2b49e00640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 0x7f2b2806efe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b38001030 tx=0x7f2b38009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:39.965+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2b34019030 con 0x7f2b441646a0 2024-04-17T07:26:40.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.195+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2b44161310 con 0x7f2b2806cb20 2024-04-17T07:26:40.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.434+0000 7f2b3e000640 1 -- 172.21.15.67:0/3612920597 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f2b44161310 con 0x7f2b2806cb20 2024-04-17T07:26:40.435 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.437+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 msgr2=0x7f2b2806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.437+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 0x7f2b2806efe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b38001030 tx=0x7f2b38009040 comp rx=0 tx=0).stop 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.437+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 msgr2=0x7f2b44158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.437+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2b34011660 tx=0x7f2b34011b20 comp rx=0 tx=0).stop 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 shutdown_connections 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f2b2806cb20 0x7f2b2806efe0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b440725c0 0x7f2b44072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b441646a0 0x7f2b44158a80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:40.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 --2- 172.21.15.67:0/3612920597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b4414c890 0x7f2b44165d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:40.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 >> 172.21.15.67:0/3612920597 conn(0x7f2b440b2190 msgr2=0x7f2b440b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:40.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 shutdown_connections 2024-04-17T07:26:40.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:40.438+0000 7f2b4b200640 1 -- 172.21.15.67:0/3612920597 wait complete. 2024-04-17T07:26:40.459 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-04-17T07:26:40.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.459 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:40.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:40.512 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:40.512 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:40.512 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:40.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.513 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_16_01_UTC) 2024-04-17T07:26:40.607 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:40.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.607 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:40.665 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:40.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.665 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:40.724 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:40.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.725 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_19_01_UTC) 2024-04-17T07:26:40.790 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:40.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.790 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_20_01_UTC) 2024-04-17T07:26:40.864 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:40.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.865 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:40.952 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:40.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:40.952 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:41.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:41.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:41.005 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_23_01_UTC) 2024-04-17T07:26:41.064 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:41.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:41.064 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:41.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:41.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:41.127 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_25_01_UTC) 2024-04-17T07:26:41.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:41.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:41.188 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:41.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:41.252 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-04-17T07:26:41.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-04-17T07:26:41.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.540+0000 7f2782000640 1 Processor -- start 2024-04-17T07:26:41.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.540+0000 7f2782000640 1 -- start start 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f277c145f80 0x7f277c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f277c0cd750 0x7f277c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f277c0727a0 con 0x7f277c145f80 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f277c146360 con 0x7f277c0cd750 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2782000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f277c071e70 con 0x7f277c06c0a0 2024-04-17T07:26:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2780c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f277c145f80 0x7f277c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2780c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f277c145f80 0x7f277c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50954/0 (socket says 172.21.15.67:50954) 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.542+0000 7f2780c00640 1 -- 172.21.15.67:0/3497211019 learned_addr learned my addr 172.21.15.67:0/3497211019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.543+0000 7f2780c00640 1 -- 172.21.15.67:0/3497211019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 msgr2=0x7f277c160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.543+0000 7f2780c00640 1 --2- 172.21.15.67:0/3497211019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.543+0000 7f2780c00640 1 -- 172.21.15.67:0/3497211019 >> v2:172.21.15.187:3300/0 conn(0x7f277c0cd750 msgr2=0x7f277c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.543+0000 7f2780c00640 1 --2- 172.21.15.67:0/3497211019 >> v2:172.21.15.187:3300/0 conn(0x7f277c0cd750 0x7f277c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.543+0000 7f2780c00640 1 -- 172.21.15.67:0/3497211019 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277c072150 con 0x7f277c145f80 2024-04-17T07:26:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.544+0000 7f2780c00640 1 --2- 172.21.15.67:0/3497211019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0cd210 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f277c06e140 tx=0x7f277002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a936544fb6feea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.544+0000 7f277b400640 1 -- 172.21.15.67:0/3497211019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277002e070 con 0x7f277c145f80 2024-04-17T07:26:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.544+0000 7f277b400640 1 -- 172.21.15.67:0/3497211019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2770037070 con 0x7f277c145f80 2024-04-17T07:26:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.544+0000 7f277b400640 1 -- 172.21.15.67:0/3497211019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770031a00 con 0x7f277c145f80 2024-04-17T07:26:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 -- 172.21.15.67:0/3497211019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 msgr2=0x7f277c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 --2- 172.21.15.67:0/3497211019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0cd210 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f277c06e140 tx=0x7f277002b880 comp rx=0 tx=0).stop 2024-04-17T07:26:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 -- 172.21.15.67:0/3497211019 shutdown_connections 2024-04-17T07:26:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 --2- 172.21.15.67:0/3497211019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 --2- 172.21.15.67:0/3497211019 >> v2:172.21.15.187:3300/0 conn(0x7f277c0cd750 0x7f277c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 --2- 172.21.15.67:0/3497211019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0cd210 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 -- 172.21.15.67:0/3497211019 >> 172.21.15.67:0/3497211019 conn(0x7f277c0b2190 msgr2=0x7f277c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 -- 172.21.15.67:0/3497211019 shutdown_connections 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.545+0000 7f2782000640 1 -- 172.21.15.67:0/3497211019 wait complete. 2024-04-17T07:26:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.547+0000 7f2782000640 1 Processor -- start 2024-04-17T07:26:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.547+0000 7f2782000640 1 -- start start 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 0x7f277c0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f277c072db0 con 0x7f277c145f80 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f277c072c30 con 0x7f277c0cd750 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2782000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f277c06c4d0 con 0x7f277c06c0a0 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2781600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2781600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50960/0 (socket says 172.21.15.67:50960) 2024-04-17T07:26:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.548+0000 7f2781600640 1 -- 172.21.15.67:0/1515759068 learned_addr learned my addr 172.21.15.67:0/1515759068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2780c00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f277be00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 0x7f277c0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 -- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 msgr2=0x7f277c0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c0c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 -- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 msgr2=0x7f277c0b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 0x7f277c0b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 -- 172.21.15.67:0/1515759068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277c0c8d60 con 0x7f277c145f80 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f277be00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 0x7f277c0b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2781600640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f277400b970 tx=0x7f277400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774019070 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277400cb60 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774014440 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f277c0c9040 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.549+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f277c0c9600 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.551+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f277c068a10 con 0x7f277c145f80 2024-04-17T07:26:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.552+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f277401c070 con 0x7f277c145f80 2024-04-17T07:26:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.553+0000 7f2778c00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 0x7f275c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.553+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6519+0+0 (secure 0 0 0) 0x7f277408bf60 con 0x7f277c145f80 2024-04-17T07:26:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.554+0000 7f2780c00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 0x7f275c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.555+0000 7f2780c00640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 0x7f275c06efb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2770036000 tx=0x7f2770008030 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.558+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2774059dd0 con 0x7f277c145f80 2024-04-17T07:26:41.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:41.823+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f277c06f1a0 con 0x7f275c06caf0 2024-04-17T07:26:42.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.069+0000 7f2778c00640 1 -- 172.21.15.67:0/1515759068 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f277c06f1a0 con 0x7f275c06caf0 2024-04-17T07:26:42.069 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 msgr2=0x7f275c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 0x7f275c06efb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2770036000 tx=0x7f2770008030 comp rx=0 tx=0).stop 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 msgr2=0x7f277c0a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f277400b970 tx=0x7f277400be30 comp rx=0 tx=0).stop 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 shutdown_connections 2024-04-17T07:26:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f275c06caf0 0x7f275c06efb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f277c145f80 0x7f277c0a9680 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f277c0cd750 0x7f277c0b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 --2- 172.21.15.67:0/1515759068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f277c06c0a0 0x7f277c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 >> 172.21.15.67:0/1515759068 conn(0x7f277c0b2190 msgr2=0x7f277c15ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 shutdown_connections 2024-04-17T07:26:42.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:42.073+0000 7f2782000640 1 -- 172.21.15.67:0/1515759068 wait complete. 2024-04-17T07:26:42.098 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-04-17T07:26:42.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.098 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-04-17T07:26:42.153 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:42.153 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:42.153 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:42.154 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:42.155 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:42.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.155 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:42.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:42.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.224 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:42.293 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:42.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.294 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_18_01_UTC) 2024-04-17T07:26:42.358 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:42.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.358 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_19_01_UTC) 2024-04-17T07:26:42.478 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:42.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.479 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_20_01_UTC) 2024-04-17T07:26:42.531 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:42.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.532 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:42.578 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:42.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:42.989 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:43.060 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:43.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:43.061 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:43.130 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:43.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:43.131 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_24_01_UTC) 2024-04-17T07:26:43.199 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:43.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:43.199 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:43.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:43.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:43.270 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-04-17-07_26_01_UTC) 2024-04-17T07:26:43.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:43.342 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-04-17T07:26:43.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-04-17T07:26:43.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.604+0000 7f3e0fa00640 1 Processor -- start 2024-04-17T07:26:43.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.604+0000 7f3e0fa00640 1 -- start start 2024-04-17T07:26:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3e0fa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3e0815b860 0x7f3e080d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3e0fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3e080d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3e0fa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e08071f60 con 0x7f3e080ce4d0 2024-04-17T07:26:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3e0fa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3e08071d10 con 0x7f3e0815b860 2024-04-17T07:26:43.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3e0fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e08072240 con 0x7f3e080b4f80 2024-04-17T07:26:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3dffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3e080ce4d0 0x7f3e080ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51334/0 (socket says 172.21.15.67:51334) 2024-04-17T07:26:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.606+0000 7f3dffe00640 1 -- 172.21.15.67:0/371347714 learned_addr learned my addr 172.21.15.67:0/371347714 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e0d000640 1 --2- 172.21.15.67:0/371347714 >> v2:172.21.15.187:3300/0 conn(0x7f3e0815b860 0x7f3e080d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e0da00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3e080d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423762499 0 0) 0x7f3e08071f60 con 0x7f3e080ce4d0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dec003520 con 0x7f3e080ce4d0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 324525457 0 0) 0x7f3dec003520 con 0x7f3e080ce4d0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 msgr2=0x7f3e080d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3e080d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 >> v2:172.21.15.187:3300/0 conn(0x7f3e0815b860 msgr2=0x7f3e080d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 --2- 172.21.15.67:0/371347714 >> v2:172.21.15.187:3300/0 conn(0x7f3e0815b860 0x7f3e080d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.607+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e08072520 con 0x7f3e080ce4d0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3df0001670 con 0x7f3e080ce4d0 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 >> v1:172.21.15.67:6789/0 conn(0x7f3e080ce4d0 legacy=0x7f3e080ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dec0039b0 0x7f3dec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dec004800 0x7f3dec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3dec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08071d10 con 0x7f3dec0039b0 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08072240 con 0x7f3dec004800 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec00dc20 con 0x7f3e080b4f80 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dec0039b0 0x7f3dec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3dffe00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dec004800 0x7f3dec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0da00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3dec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 -- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 msgr2=0x7f3dec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e080b4f80 0x7f3dec0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 -- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dec004800 msgr2=0x7f3dec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dec004800 0x7f3dec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.608+0000 7f3e0d000640 1 -- 172.21.15.67:0/371347714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec003520 con 0x7f3dec0039b0 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e0d000640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dec0039b0 0x7f3dec003df0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009af0 tx=0x7f3dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f8d1c0d946312e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8039070 con 0x7f3dec0039b0 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df802c710 con 0x7f3dec0039b0 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e07e00640 1 -- 172.21.15.67:0/371347714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8009690 con 0x7f3dec0039b0 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e0fa00640 1 -- 172.21.15.67:0/371347714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dec0039b0 msgr2=0x7f3dec003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/371347714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dec0039b0 0x7f3dec003df0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009af0 tx=0x7f3dec005830 comp rx=0 tx=0).stop 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.610+0000 7f3e0da00640 1 -- 172.21.15.67:0/371347714 reap_dead start 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.611+0000 7f3e0fa00640 1 -- 172.21.15.67:0/371347714 shutdown_connections 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.611+0000 7f3e0fa00640 1 -- 172.21.15.67:0/371347714 >> 172.21.15.67:0/371347714 conn(0x7f3e08067ee0 msgr2=0x7f3e0806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.611+0000 7f3e0fa00640 1 -- 172.21.15.67:0/371347714 shutdown_connections 2024-04-17T07:26:43.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.611+0000 7f3e0fa00640 1 -- 172.21.15.67:0/371347714 wait complete. 2024-04-17T07:26:43.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.612+0000 7f3e0fa00640 1 Processor -- start 2024-04-17T07:26:43.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.613+0000 7f3e0fa00640 1 -- start start 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dec0039b0 0x7f3e08147240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 0x7f3e08149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0815bc40 con 0x7f3e0815c240 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08072ae0 con 0x7f3e0815bdc0 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0806f380 con 0x7f3dec0039b0 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60890/0 (socket says 172.21.15.67:60890) 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 -- 172.21.15.67:0/1084231308 learned_addr learned my addr 172.21.15.67:0/1084231308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0d000640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dec0039b0 0x7f3e08147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3dffe00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 0x7f3e08149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 -- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dec0039b0 msgr2=0x7f3e08147240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dec0039b0 0x7f3e08147240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 -- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 msgr2=0x7f3e08149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 0x7f3e08149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3e0da00640 1 -- 172.21.15.67:0/1084231308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e08147fb0 con 0x7f3e0815c240 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.614+0000 7f3dffe00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 0x7f3e08149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e0da00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3e0806c5d0 tx=0x7f3e0000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0001c070 con 0x7f3e0815c240 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00017400 con 0x7f3e0815c240 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000177f0 con 0x7f3e0815c240 2024-04-17T07:26:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e08148290 con 0x7f3e0815c240 2024-04-17T07:26:43.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.615+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e08148850 con 0x7f3e0815c240 2024-04-17T07:26:43.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.616+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e08068ca0 con 0x7f3e0815c240 2024-04-17T07:26:43.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.617+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f3e0001f070 con 0x7f3e0815c240 2024-04-17T07:26:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.618+0000 7f3e05600640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 0x7f3de406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.618+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6519+0+0 (secure 0 0 0) 0x7f3e0008c3e0 con 0x7f3e0815c240 2024-04-17T07:26:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.619+0000 7f3e0d000640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 0x7f3de406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.620+0000 7f3e0d000640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 0x7f3de406efe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3df8036000 tx=0x7f3df8030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.623+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3e00012070 con 0x7f3e0815c240 2024-04-17T07:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:43.850+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3e0806ccd0 con 0x7f3de406cb20 2024-04-17T07:26:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.082+0000 7f3e05600640 1 -- 172.21.15.67:0/1084231308 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f3e0806ccd0 con 0x7f3de406cb20 2024-04-17T07:26:44.083 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.085+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 msgr2=0x7f3de406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.085+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 0x7f3de406efe0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3df8036000 tx=0x7f3df8030000 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.085+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 msgr2=0x7f3e08149590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.085+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3e0806c5d0 tx=0x7f3e0000f4a0 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 shutdown_connections 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f3de406cb20 0x7f3de406efe0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e0815c240 0x7f3e08149590 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e0815bdc0 0x7f3e08149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 --2- 172.21.15.67:0/1084231308 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dec0039b0 0x7f3e08147240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 >> 172.21.15.67:0/1084231308 conn(0x7f3e08067ee0 msgr2=0x7f3e081585c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 shutdown_connections 2024-04-17T07:26:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:44.086+0000 7f3e0fa00640 1 -- 172.21.15.67:0/1084231308 wait complete. 2024-04-17T07:26:44.109 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-04-17T07:26:44.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.109 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-04-17T07:26:44.161 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:44.162 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:44.163 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:44.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.164 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:44.230 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:44.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.231 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:44.302 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_18_01_UTC 2024-04-17T07:26:44.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.303 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_18_01_UTC) 2024-04-17T07:26:44.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:44.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.377 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:44.465 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:44.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.466 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:44.511 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:44.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.512 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:44.604 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:44.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.604 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_22_01_UTC) 2024-04-17T07:26:44.660 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:44.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.660 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_23_01_UTC) 2024-04-17T07:26:44.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:44.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.719 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:44.778 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:44.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.778 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:44.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:44.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.838 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:44.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:44.898 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-04-17T07:26:44.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-04-17T07:26:45.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.174+0000 7f9db8a00640 1 Processor -- start 2024-04-17T07:26:45.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.174+0000 7f9db8a00640 1 -- start start 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.175+0000 7f9db8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9db406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db415ff40 con 0x7f9db40cfc10 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db8a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db406df40 con 0x7f9db406f800 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9db406ed00 con 0x7f9db415b6b0 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db3200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9db40cfc10 0x7f9db415f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51348/0 (socket says 172.21.15.67:51348) 2024-04-17T07:26:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db3200640 1 -- 172.21.15.67:0/690617296 learned_addr learned my addr 172.21.15.67:0/690617296 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db2800640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9db406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.176+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623487470 0 0) 0x7f9db415ff40 con 0x7f9db40cfc10 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d98005730 con 0x7f9db40cfc10 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3436129960 0 0) 0x7f9d98005730 con 0x7f9db40cfc10 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 msgr2=0x7f9db406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:45.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9db406b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 >> v1:172.21.15.187:6789/0 conn(0x7f9db406f800 legacy=0x7f9db415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db406e220 con 0x7f9db40cfc10 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9dac0019f0 con 0x7f9db40cfc10 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 >> v1:172.21.15.67:6789/0 conn(0x7f9db40cfc10 legacy=0x7f9db415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d98005bf0 0x7f9d98005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d980065a0 0x7f9d98006a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9d98007370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9db406ed00 con 0x7f9d980065a0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9800dc80 con 0x7f9d98005bf0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.177+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9800de80 con 0x7f9db415b6b0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d980065a0 0x7f9d98006a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9db2800640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d98005bf0 0x7f9d98005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9db3200640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9d98007370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 -- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 msgr2=0x7f9d98007370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9db415b6b0 0x7f9d98007370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 -- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d98005bf0 msgr2=0x7f9d98005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d98005bf0 0x7f9d98005fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.178+0000 7f9da9e00640 1 -- 172.21.15.67:0/690617296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d98005730 con 0x7f9d980065a0 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.179+0000 7f9da9e00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d980065a0 0x7f9d98006a70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c001860 tx=0x7f9d98007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db7d1dba787a453 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.179+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c038070 con 0x7f9d980065a0 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.179+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d9c033410 con 0x7f9d980065a0 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.179+0000 7f9db1e00640 1 -- 172.21.15.67:0/690617296 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c001430 con 0x7f9d980065a0 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 -- 172.21.15.67:0/690617296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d980065a0 msgr2=0x7f9d98006a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 --2- 172.21.15.67:0/690617296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d980065a0 0x7f9d98006a70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c001860 tx=0x7f9d98007a40 comp rx=0 tx=0).stop 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db3200640 1 -- 172.21.15.67:0/690617296 reap_dead start 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 -- 172.21.15.67:0/690617296 shutdown_connections 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 -- 172.21.15.67:0/690617296 >> 172.21.15.67:0/690617296 conn(0x7f9db40b2190 msgr2=0x7f9db40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 -- 172.21.15.67:0/690617296 shutdown_connections 2024-04-17T07:26:45.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.180+0000 7f9db8a00640 1 -- 172.21.15.67:0/690617296 wait complete. 2024-04-17T07:26:45.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.182+0000 7f9db8a00640 1 Processor -- start 2024-04-17T07:26:45.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.182+0000 7f9db8a00640 1 -- start start 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 0x7f9db40b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 0x7f9db414aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9db40afd60 con 0x7f9db415bf10 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9db406e870 con 0x7f9db415ba90 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9db40afee0 con 0x7f9d980065a0 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.183+0000 7f9db3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60912/0 (socket says 172.21.15.67:60912) 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 -- 172.21.15.67:0/4083417421 learned_addr learned my addr 172.21.15.67:0/4083417421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db2800640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 0x7f9db40b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9da9e00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 0x7f9db414aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 -- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 msgr2=0x7f9db40b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 0x7f9db40b1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 -- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 msgr2=0x7f9db414aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 0x7f9db414aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 -- 172.21.15.67:0/4083417421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db415c4e0 con 0x7f9db415bf10 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9da9e00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 0x7f9db414aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db2800640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 0x7f9db40b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:45.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.184+0000 7f9db3200640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9dac007c40 tx=0x7f9dac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.185+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac00dea0 con 0x7f9db415bf10 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.185+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dac00bb80 con 0x7f9db415bf10 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.185+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac007070 con 0x7f9db415bf10 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.185+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db415c7c0 con 0x7f9db415bf10 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.185+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db415cd80 con 0x7f9db415bf10 2024-04-17T07:26:45.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.186+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db41580a0 con 0x7f9db415bf10 2024-04-17T07:26:45.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.187+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f9dac007210 con 0x7f9db415bf10 2024-04-17T07:26:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.188+0000 7f9dab400640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 0x7f9d9006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.188+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6519+0+0 (secure 0 0 0) 0x7f9dac08c610 con 0x7f9db415bf10 2024-04-17T07:26:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.189+0000 7f9db2800640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 0x7f9d9006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:45.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.190+0000 7f9db2800640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 0x7f9d9006efe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9da40099e0 tx=0x7f9da4008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:45.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.193+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9dac05a250 con 0x7f9db415bf10 2024-04-17T07:26:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.437+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9db40b4590 con 0x7f9d9006cb20 2024-04-17T07:26:45.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.688+0000 7f9dab400640 1 -- 172.21.15.67:0/4083417421 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f9db40b4590 con 0x7f9d9006cb20 2024-04-17T07:26:45.688 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-04-17T07:26:45.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 msgr2=0x7f9d9006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 0x7f9d9006efe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9da40099e0 tx=0x7f9da4008040 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 msgr2=0x7f9db414b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9dac007c40 tx=0x7f9dac00b4a0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 shutdown_connections 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f9d9006cb20 0x7f9d9006efe0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9db415bf10 0x7f9db414b000 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9db415ba90 0x7f9db414aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 --2- 172.21.15.67:0/4083417421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9d980065a0 0x7f9db40b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.691+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 >> 172.21.15.67:0/4083417421 conn(0x7f9db40b2190 msgr2=0x7f9db40ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.692+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 shutdown_connections 2024-04-17T07:26:45.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:45.692+0000 7f9db8a00640 1 -- 172.21.15.67:0/4083417421 wait complete. 2024-04-17T07:26:45.713 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-04-17T07:26:45.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:45.713 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-04-17T07:26:45.765 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:45.765 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:45.766 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:45.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:45.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:45.767 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_16_01_UTC) 2024-04-17T07:26:45.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:45.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:45.831 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:45.895 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:45.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:45.895 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:45.959 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:45.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:45.959 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:46.020 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:46.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.021 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:46.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:46.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.087 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_21_01_UTC) 2024-04-17T07:26:46.156 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:46.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.157 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_22_01_UTC) 2024-04-17T07:26:46.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:46.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.224 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:46.285 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:46.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.285 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:46.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:46.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.353 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:46.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_26_00_UTC 2024-04-17T07:26:46.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.444 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-04-17-07_26_00_UTC) 2024-04-17T07:26:46.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:46.494 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-04-17T07:26:46.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-04-17T07:26:46.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.803+0000 7f2049a00640 1 Processor -- start 2024-04-17T07:26:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.804+0000 7f2049a00640 1 -- start start 2024-04-17T07:26:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2044145f80 0x7f20440cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2044146360 con 0x7f204406c270 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2044071e70 con 0x7f20440cd750 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20440727a0 con 0x7f2044145f80 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f204406c270 0x7f204406e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51350/0 (socket says 172.21.15.67:51350) 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.806+0000 7f2049000640 1 -- 172.21.15.67:0/1563141092 learned_addr learned my addr 172.21.15.67:0/1563141092 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2043e00640 1 --2- 172.21.15.67:0/1563141092 >> v2:172.21.15.187:3300/0 conn(0x7f2044145f80 0x7f20440cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213114407 0 0) 0x7f2044146360 con 0x7f204406c270 2024-04-17T07:26:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2028003520 con 0x7f204406c270 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3174883042 0 0) 0x7f2028003520 con 0x7f204406c270 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 >> v2:172.21.15.187:3300/0 conn(0x7f2044145f80 msgr2=0x7f20440cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 --2- 172.21.15.67:0/1563141092 >> v2:172.21.15.187:3300/0 conn(0x7f2044145f80 0x7f20440cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 >> v1:172.21.15.187:6789/0 conn(0x7f20440cd750 legacy=0x7f204406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2044072150 con 0x7f204406c270 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f203c0019f0 con 0x7f204406c270 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 >> v1:172.21.15.67:6789/0 conn(0x7f204406c270 legacy=0x7f204406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.807+0000 7f2042a00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 0x7f2028003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2042a00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2028004800 0x7f2028004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2042a00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028005210 0x7f202800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20440727a0 con 0x7f2028003a00 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f202800c160 con 0x7f2028004800 2024-04-17T07:26:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f202800c360 con 0x7f2028005210 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 0x7f2028003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043400640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2028004800 0x7f2028004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 -- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028005210 msgr2=0x7f202800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028005210 0x7f202800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 -- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2028004800 msgr2=0x7f2028004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2028004800 0x7f2028004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2043e00640 1 -- 172.21.15.67:0/1563141092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2028003520 con 0x7f2028003a00 2024-04-17T07:26:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.808+0000 7f2049000640 1 -- 172.21.15.67:0/1563141092 reap_dead start 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.810+0000 7f2043e00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 0x7f2028003e20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2034009a90 tx=0x7f2028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79fd262fcded6df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.810+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034039070 con 0x7f2028003a00 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.810+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203402c680 con 0x7f2028003a00 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.810+0000 7f2042a00640 1 -- 172.21.15.67:0/1563141092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034009690 con 0x7f2028003a00 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 -- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 msgr2=0x7f2028003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 0x7f2028003e20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2034009a90 tx=0x7f2028005830 comp rx=0 tx=0).stop 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 -- 172.21.15.67:0/1563141092 shutdown_connections 2024-04-17T07:26:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 --2- 172.21.15.67:0/1563141092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2028003a00 0x7f2028003e20 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 -- 172.21.15.67:0/1563141092 >> 172.21.15.67:0/1563141092 conn(0x7f20440b2190 msgr2=0x7f20440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 -- 172.21.15.67:0/1563141092 shutdown_connections 2024-04-17T07:26:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.811+0000 7f2049a00640 1 -- 172.21.15.67:0/1563141092 wait complete. 2024-04-17T07:26:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.813+0000 7f2049a00640 1 Processor -- start 2024-04-17T07:26:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.813+0000 7f2049a00640 1 -- start start 2024-04-17T07:26:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028003a00 0x7f20440ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044140240 0x7f20440cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2044072db0 con 0x7f204415ec30 2024-04-17T07:26:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2044072c30 con 0x7f2044140240 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.814+0000 7f2049a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20440cd210 con 0x7f2028003a00 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60936/0 (socket says 172.21.15.67:60936) 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 -- 172.21.15.67:0/1506139289 learned_addr learned my addr 172.21.15.67:0/1506139289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2043400640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044140240 0x7f20440cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2043e00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028003a00 0x7f20440ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 -- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028003a00 msgr2=0x7f20440ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028003a00 0x7f20440ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 -- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044140240 msgr2=0x7f20440cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044140240 0x7f20440cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 -- 172.21.15.67:0/1506139289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20440b37a0 con 0x7f204415ec30 2024-04-17T07:26:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.815+0000 7f2049000640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f20440d1470 tx=0x7f203c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.816+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c00a610 con 0x7f204415ec30 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.816+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203c00a7b0 con 0x7f204415ec30 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.816+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203c00aba0 con 0x7f204415ec30 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.816+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20440b3a80 con 0x7f204415ec30 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.816+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20440b4040 con 0x7f204415ec30 2024-04-17T07:26:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.817+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2044068a10 con 0x7f204415ec30 2024-04-17T07:26:46.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.818+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f203c011070 con 0x7f204415ec30 2024-04-17T07:26:46.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.819+0000 7f203be00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 0x7f202406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:46.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.819+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6551+0+0 (secure 0 0 0) 0x7f203c08bb40 con 0x7f204415ec30 2024-04-17T07:26:46.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.820+0000 7f2043e00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 0x7f202406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.821+0000 7f2043e00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 0x7f202406efb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2034036000 tx=0x7f2034030000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:46.824+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f203c059360 con 0x7f204415ec30 2024-04-17T07:26:47.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.087+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2044140620 con 0x7f202406caf0 2024-04-17T07:26:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.345+0000 7f203be00640 1 -- 172.21.15.67:0/1506139289 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f2044140620 con 0x7f202406caf0 2024-04-17T07:26:47.345 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.348+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 msgr2=0x7f202406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.348+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 0x7f202406efb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2034036000 tx=0x7f2034030000 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.348+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 msgr2=0x7f204415f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.348+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f20440d1470 tx=0x7f203c00ccd0 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 shutdown_connections 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f202406caf0 0x7f202406efb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f204415ec30 0x7f204415f0e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044140240 0x7f20440cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 --2- 172.21.15.67:0/1506139289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2028003a00 0x7f20440ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 >> 172.21.15.67:0/1506139289 conn(0x7f20440b2190 msgr2=0x7f204406baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 shutdown_connections 2024-04-17T07:26:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:47.349+0000 7f2049a00640 1 -- 172.21.15.67:0/1506139289 wait complete. 2024-04-17T07:26:47.370 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-04-17T07:26:47.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.370 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-04-17T07:26:47.436 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:47.436 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:47.436 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:47.436 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:47.436 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:47.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:47.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.438 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:47.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:47.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.521 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:47.592 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:47.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.592 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:47.638 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:47.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.639 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:47.693 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_20_01_UTC 2024-04-17T07:26:47.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.693 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_20_01_UTC) 2024-04-17T07:26:47.750 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:47.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.751 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_21_01_UTC) 2024-04-17T07:26:47.819 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_22_00_UTC 2024-04-17T07:26:47.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.819 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_22_00_UTC) 2024-04-17T07:26:47.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_23_01_UTC 2024-04-17T07:26:47.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.885 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_23_01_UTC) 2024-04-17T07:26:47.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:47.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:47.950 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_24_01_UTC) 2024-04-17T07:26:48.019 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:48.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:48.019 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:48.097 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:48.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:48.098 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-04-17-07_26_01_UTC) 2024-04-17T07:26:48.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:48.172 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-04-17T07:26:48.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-04-17T07:26:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.483+0000 7f4aa8600640 1 Processor -- start 2024-04-17T07:26:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.483+0000 7f4aa8600640 1 -- start start 2024-04-17T07:26:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.484+0000 7f4aa8600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4aa006daf0 0x7f4aa00d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa8600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa00d1520 con 0x7f4aa00d1060 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa8600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa00d1800 con 0x7f4aa009b170 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa8600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa00d1ae0 con 0x7f4aa006daf0 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa5200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4aa00d1060 0x7f4aa009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51366/0 (socket says 172.21.15.67:51366) 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa5200640 1 -- 172.21.15.67:0/1944955284 learned_addr learned my addr 172.21.15.67:0/1944955284 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/1944955284 >> v2:172.21.15.187:3300/0 conn(0x7f4aa006daf0 0x7f4aa00d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793427493 0 0) 0x7f4aa00d1520 con 0x7f4aa00d1060 2024-04-17T07:26:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.485+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a88003520 con 0x7f4aa00d1060 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 965955766 0 0) 0x7f4a88003520 con 0x7f4aa00d1060 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 >> v2:172.21.15.187:3300/0 conn(0x7f4aa006daf0 msgr2=0x7f4aa00d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 --2- 172.21.15.67:0/1944955284 >> v2:172.21.15.187:3300/0 conn(0x7f4aa006daf0 0x7f4aa00d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 >> v1:172.21.15.187:6789/0 conn(0x7f4aa009b170 legacy=0x7f4aa0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa00d1dc0 con 0x7f4aa00d1060 2024-04-17T07:26:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4a8c001670 con 0x7f4aa00d1060 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 >> v1:172.21.15.67:6789/0 conn(0x7f4aa00d1060 legacy=0x7f4aa009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a880039b0 0x7f4a88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a880043b0 0x7f4a88004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4a8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa00d1ae0 con 0x7f4a88005080 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8800bfe0 con 0x7f4a880039b0 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8800c1e0 con 0x7f4a880043b0 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.486+0000 7f4aa6600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4a8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a880039b0 0x7f4a88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa5200640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a880043b0 0x7f4a88004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa6600640 1 -- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a880043b0 msgr2=0x7f4a88004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa6600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a880043b0 0x7f4a88004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa6600640 1 -- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a880039b0 msgr2=0x7f4a88003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa6600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a880039b0 0x7f4a88003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa6600640 1 -- 172.21.15.67:0/1944955284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a88003520 con 0x7f4a88005080 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa5200640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a880043b0 0x7f4a88004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.487+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a880039b0 0x7f4a88003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.488+0000 7f4aa6600640 1 -- 172.21.15.67:0/1944955284 reap_dead start 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.488+0000 7f4aa6600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4a8800b9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0162800 tx=0x7f4a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425c7189fcd3ba8f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.488+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98001b30 con 0x7f4a88005080 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.488+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a98003b90 con 0x7f4a88005080 2024-04-17T07:26:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.488+0000 7f4a9fe00640 1 -- 172.21.15.67:0/1944955284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98005070 con 0x7f4a88005080 2024-04-17T07:26:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 -- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 msgr2=0x7f4a8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4a8800b9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0162800 tx=0x7f4a88005830 comp rx=0 tx=0).stop 2024-04-17T07:26:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 -- 172.21.15.67:0/1944955284 shutdown_connections 2024-04-17T07:26:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 --2- 172.21.15.67:0/1944955284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4a8800b9c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 -- 172.21.15.67:0/1944955284 >> 172.21.15.67:0/1944955284 conn(0x7f4aa00b41a0 msgr2=0x7f4aa00b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 -- 172.21.15.67:0/1944955284 shutdown_connections 2024-04-17T07:26:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.489+0000 7f4aa8600640 1 -- 172.21.15.67:0/1944955284 wait complete. 2024-04-17T07:26:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.491+0000 7f4aa8600640 1 Processor -- start 2024-04-17T07:26:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.491+0000 7f4aa8600640 1 -- start start 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4aa00b7650 0x7f4aa00c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4aa00c8d30 0x7f4aa00b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa006e050 con 0x7f4a88005080 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa006ded0 con 0x7f4aa00c8d30 2024-04-17T07:26:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa8600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa006ce00 con 0x7f4aa00b7650 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.492+0000 7f4aa5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60950/0 (socket says 172.21.15.67:60950) 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 -- 172.21.15.67:0/3243863634 learned_addr learned my addr 172.21.15.67:0/3243863634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5200640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4aa00b7650 0x7f4aa00c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa6600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4aa00c8d30 0x7f4aa00b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 -- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4aa00b7650 msgr2=0x7f4aa00c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4aa00b7650 0x7f4aa00c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 -- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4aa00c8d30 msgr2=0x7f4aa00b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4aa00c8d30 0x7f4aa00b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 -- 172.21.15.67:0/3243863634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa00b65b0 con 0x7f4a88005080 2024-04-17T07:26:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.493+0000 7f4aa5c00640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011540 tx=0x7f4a94011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.494+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94015870 con 0x7f4a88005080 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.494+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a94015a10 con 0x7f4a88005080 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.494+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94015dc0 con 0x7f4a88005080 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.494+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa00b6890 con 0x7f4a88005080 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.494+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa00b6e50 con 0x7f4a88005080 2024-04-17T07:26:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.495+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa00c9110 con 0x7f4a88005080 2024-04-17T07:26:48.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.496+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f4a9400f460 con 0x7f4a88005080 2024-04-17T07:26:48.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.498+0000 7f4a9d600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 0x7f4a8006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:48.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.498+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 7107+0+0 (secure 0 0 0) 0x7f4a9408cc90 con 0x7f4a88005080 2024-04-17T07:26:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.498+0000 7f4aa5200640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 0x7f4a8006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.500+0000 7f4aa5200640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 0x7f4a8006efe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cf0d0 tx=0x7f4a8c009040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:48.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.503+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a94019030 con 0x7f4a88005080 2024-04-17T07:26:48.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.732+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4aa0068ca0 con 0x7f4a8006cb20 2024-04-17T07:26:49.000 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-04-17T07:26:49.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:48.999+0000 7f4a9d600640 1 -- 172.21.15.67:0/3243863634 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f4aa0068ca0 con 0x7f4a8006cb20 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.005+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 msgr2=0x7f4a8006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.005+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 0x7f4a8006efe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cf0d0 tx=0x7f4a8c009040 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.005+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 msgr2=0x7f4aa015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.005+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011540 tx=0x7f4a94011a00 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 shutdown_connections 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f4a8006cb20 0x7f4a8006efe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4aa00c8d30 0x7f4aa00b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4aa00b7650 0x7f4aa00c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 --2- 172.21.15.67:0/3243863634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88005080 0x7f4aa015d580 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011540 tx=0x7f4a94011a00 comp rx=0 tx=0).stop 2024-04-17T07:26:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 >> 172.21.15.67:0/3243863634 conn(0x7f4aa00b41a0 msgr2=0x7f4aa00af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 shutdown_connections 2024-04-17T07:26:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:49.006+0000 7f4aa8600640 1 -- 172.21.15.67:0/3243863634 wait complete. 2024-04-17T07:26:49.036 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-04-17T07:26:49.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.037 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:49.089 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:49.089 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:49.089 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_16_00_UTC 2024-04-17T07:26:49.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.090 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_16_00_UTC) 2024-04-17T07:26:49.201 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_17_00_UTC 2024-04-17T07:26:49.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.202 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_17_00_UTC) 2024-04-17T07:26:49.273 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:49.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.274 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:49.360 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_19_00_UTC 2024-04-17T07:26:49.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.360 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_19_00_UTC) 2024-04-17T07:26:49.421 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:49.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.422 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:49.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_21_00_UTC 2024-04-17T07:26:49.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.489 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_21_00_UTC) 2024-04-17T07:26:49.559 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:49.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.559 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_22_01_UTC) 2024-04-17T07:26:49.628 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:49.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.628 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:49.703 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_24_01_UTC 2024-04-17T07:26:49.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.704 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_24_01_UTC) 2024-04-17T07:26:49.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_25_00_UTC 2024-04-17T07:26:49.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.780 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_25_00_UTC) 2024-04-17T07:26:49.856 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:49.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.857 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-04-17-07_26_01_UTC) 2024-04-17T07:26:49.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:49.929 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-04-17T07:26:49.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-04-17T07:26:50.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.192+0000 7f6107200640 1 Processor -- start 2024-04-17T07:26:50.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.192+0000 7f6107200640 1 -- start start 2024-04-17T07:26:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6107200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6108145f80 0x7f61080cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6107200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f610806c270 0x7f610806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6107200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6108146360 con 0x7f61080cd750 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6107200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f61080727a0 con 0x7f610806c270 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6107200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6108071e70 con 0x7f6108145f80 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6106800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f610806c270 0x7f610806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6105400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61080cd750 0x7f610806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51370/0 (socket says 172.21.15.67:51370) 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.194+0000 7f6105400640 1 -- 172.21.15.67:0/374858880 learned_addr learned my addr 172.21.15.67:0/374858880 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.195+0000 7f6105e00640 1 --2- 172.21.15.67:0/374858880 >> v2:172.21.15.187:3300/0 conn(0x7f6108145f80 0x7f61080cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.195+0000 7f6106800640 1 -- 172.21.15.67:0/374858880 >> v2:172.21.15.187:3300/0 conn(0x7f6108145f80 msgr2=0x7f61080cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.195+0000 7f6106800640 1 --2- 172.21.15.67:0/374858880 >> v2:172.21.15.187:3300/0 conn(0x7f6108145f80 0x7f61080cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.195+0000 7f6106800640 1 -- 172.21.15.67:0/374858880 >> v1:172.21.15.67:6789/0 conn(0x7f61080cd750 legacy=0x7f610806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.195+0000 7f6106800640 1 -- 172.21.15.67:0/374858880 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6108072150 con 0x7f610806c270 2024-04-17T07:26:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.196+0000 7f6106800640 1 --2- 172.21.15.67:0/374858880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f610806c270 0x7f610806e1c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f610806c9c0 tx=0x7f610002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93d2e513a659133 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.196+0000 7f6104a00640 1 -- 172.21.15.67:0/374858880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100039070 con 0x7f610806c270 2024-04-17T07:26:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.196+0000 7f6104a00640 1 -- 172.21.15.67:0/374858880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f610002cbe0 con 0x7f610806c270 2024-04-17T07:26:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.196+0000 7f6104a00640 1 -- 172.21.15.67:0/374858880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100034440 con 0x7f610806c270 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 -- 172.21.15.67:0/374858880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f610806c270 msgr2=0x7f610806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 --2- 172.21.15.67:0/374858880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f610806c270 0x7f610806e1c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f610806c9c0 tx=0x7f610002be30 comp rx=0 tx=0).stop 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 -- 172.21.15.67:0/374858880 shutdown_connections 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 --2- 172.21.15.67:0/374858880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f610806c270 0x7f610806e1c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 --2- 172.21.15.67:0/374858880 >> v2:172.21.15.187:3300/0 conn(0x7f6108145f80 0x7f61080cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 -- 172.21.15.67:0/374858880 >> 172.21.15.67:0/374858880 conn(0x7f61080b2190 msgr2=0x7f61080b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 -- 172.21.15.67:0/374858880 shutdown_connections 2024-04-17T07:26:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.197+0000 7f6107200640 1 -- 172.21.15.67:0/374858880 wait complete. 2024-04-17T07:26:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.199+0000 7f6107200640 1 Processor -- start 2024-04-17T07:26:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.200+0000 7f6107200640 1 -- start start 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 0x7f61080b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61080cd750 0x7f6108142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6108072db0 con 0x7f6108145f80 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6108072c30 con 0x7f610806c270 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6107200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f610806e700 con 0x7f61080cd750 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6106800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6106800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60976/0 (socket says 172.21.15.67:60976) 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.201+0000 7f6106800640 1 -- 172.21.15.67:0/3413097727 learned_addr learned my addr 172.21.15.67:0/3413097727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6105e00640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 0x7f61080b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6105400640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61080cd750 0x7f6108142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 -- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61080cd750 msgr2=0x7f6108142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61080cd750 0x7f6108142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 -- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 msgr2=0x7f61080b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 0x7f61080b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 -- 172.21.15.67:0/3413097727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61081435f0 con 0x7f6108145f80 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6105e00640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 0x7f61080b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f6106800640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6100009e80 tx=0x7f6100004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100039050 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6100045070 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.202+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610003e830 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.203+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61081438d0 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.203+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6108143e90 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.204+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6108068a10 con 0x7f6108145f80 2024-04-17T07:26:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.205+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f610003e9d0 con 0x7f6108145f80 2024-04-17T07:26:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.206+0000 7f60fe000640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 0x7f60e806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.206+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6583+0+0 (secure 0 0 0) 0x7f61000aefe0 con 0x7f6108145f80 2024-04-17T07:26:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.206+0000 7f6105e00640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 0x7f60e806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.208+0000 7f6105e00640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 0x7f60e806efb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006fd0 tx=0x7f60f8008040 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.210+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f610007cee0 con 0x7f6108145f80 2024-04-17T07:26:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.482+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 --> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f61080b0c70 con 0x7f60e806caf0 2024-04-17T07:26:50.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.728+0000 7f60fe000640 1 -- 172.21.15.67:0/3413097727 <== mgr.4712 v2:172.21.15.187:6832/2964624313 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f61080b0c70 con 0x7f60e806caf0 2024-04-17T07:26:50.729 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-04-17T07:26:50.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.731+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 msgr2=0x7f60e806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.731+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 0x7f60e806efb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006fd0 tx=0x7f60f8008040 comp rx=0 tx=0).stop 2024-04-17T07:26:50.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.731+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 msgr2=0x7f610815bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:50.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.731+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6100009e80 tx=0x7f6100004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:50.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 shutdown_connections 2024-04-17T07:26:50.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f60e806caf0 0x7f60e806efb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6108145f80 0x7f610815bc50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61080cd750 0x7f6108142bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 --2- 172.21.15.67:0/3413097727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f610806c270 0x7f61080b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 >> 172.21.15.67:0/3413097727 conn(0x7f61080b2190 msgr2=0x7f61080af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 shutdown_connections 2024-04-17T07:26:50.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:50.732+0000 7f6107200640 1 -- 172.21.15.67:0/3413097727 wait complete. 2024-04-17T07:26:50.753 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-04-17T07:26:50.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:50.753 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-04-17T07:26:50.806 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:50.807 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:50.808 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_16_01_UTC 2024-04-17T07:26:50.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:50.809 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_16_01_UTC) 2024-04-17T07:26:50.883 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_17_01_UTC 2024-04-17T07:26:50.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:50.884 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_17_01_UTC) 2024-04-17T07:26:50.957 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_18_00_UTC 2024-04-17T07:26:50.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:50.957 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_18_00_UTC) 2024-04-17T07:26:51.026 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_19_01_UTC 2024-04-17T07:26:51.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.027 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_19_01_UTC) 2024-04-17T07:26:51.096 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_20_00_UTC 2024-04-17T07:26:51.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.097 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_20_00_UTC) 2024-04-17T07:26:51.168 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_21_01_UTC 2024-04-17T07:26:51.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.169 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_21_01_UTC) 2024-04-17T07:26:51.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_22_01_UTC 2024-04-17T07:26:51.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.243 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_22_01_UTC) 2024-04-17T07:26:51.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_23_00_UTC 2024-04-17T07:26:51.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.344 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_23_00_UTC) 2024-04-17T07:26:51.409 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_24_00_UTC 2024-04-17T07:26:51.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.409 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_24_00_UTC) 2024-04-17T07:26:51.472 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_25_01_UTC 2024-04-17T07:26:51.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_25_01_UTC) 2024-04-17T07:26:51.542 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_26_01_UTC 2024-04-17T07:26:51.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.542 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-04-17-07_26_01_UTC) 2024-04-17T07:26:51.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:51.591 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-04-17T07:26:51.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:26:51.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.894+0000 7f81ea800640 1 Processor -- start 2024-04-17T07:26:51.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.894+0000 7f81ea800640 1 -- start start 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81ea800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81e40b41a0 0x7f81e40b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81ea800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e4166a60 con 0x7f81e4162da0 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81ea800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e4166d40 con 0x7f81e40b4580 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81ea800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f81e4166810 con 0x7f81e40b41a0 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81e9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81e40b41a0 0x7f81e40b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81e9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81e4162da0 0x7f81e406dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51376/0 (socket says 172.21.15.67:51376) 2024-04-17T07:26:51.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.896+0000 7f81e9e00640 1 -- 172.21.15.67:0/3985041893 learned_addr learned my addr 172.21.15.67:0/3985041893 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.897+0000 7f81e9400640 1 -- 172.21.15.67:0/3985041893 >> v1:172.21.15.187:6789/0 conn(0x7f81e40b4580 legacy=0x7f81e4162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.897+0000 7f81e9400640 1 -- 172.21.15.67:0/3985041893 >> v1:172.21.15.67:6789/0 conn(0x7f81e4162da0 legacy=0x7f81e406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.897+0000 7f81e9400640 1 -- 172.21.15.67:0/3985041893 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e4167020 con 0x7f81e40b41a0 2024-04-17T07:26:51.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.897+0000 7f81e3e00640 1 -- 172.21.15.67:0/3985041893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799552425 0 0) 0x7f81e4166a60 con 0x7f81e4162da0 2024-04-17T07:26:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.898+0000 7f81e9400640 1 --2- 172.21.15.67:0/3985041893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b41a0 0x7f81e40b3490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009d90 tx=0x7f81d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1996f4cda613ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.898+0000 7f81e3e00640 1 -- 172.21.15.67:0/3985041893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8039070 con 0x7f81e40b41a0 2024-04-17T07:26:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.898+0000 7f81e3e00640 1 -- 172.21.15.67:0/3985041893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d802cc60 con 0x7f81e40b41a0 2024-04-17T07:26:51.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.898+0000 7f81e3e00640 1 -- 172.21.15.67:0/3985041893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8034660 con 0x7f81e40b41a0 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 -- 172.21.15.67:0/3985041893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b41a0 msgr2=0x7f81e40b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 --2- 172.21.15.67:0/3985041893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b41a0 0x7f81e40b3490 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009d90 tx=0x7f81d802beb0 comp rx=0 tx=0).stop 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 -- 172.21.15.67:0/3985041893 shutdown_connections 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 --2- 172.21.15.67:0/3985041893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b41a0 0x7f81e40b3490 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 -- 172.21.15.67:0/3985041893 >> 172.21.15.67:0/3985041893 conn(0x7f81e40b2190 msgr2=0x7f81e40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:51.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 -- 172.21.15.67:0/3985041893 shutdown_connections 2024-04-17T07:26:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.899+0000 7f81ea800640 1 -- 172.21.15.67:0/3985041893 wait complete. 2024-04-17T07:26:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.901+0000 7f81ea800640 1 Processor -- start 2024-04-17T07:26:51.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.901+0000 7f81ea800640 1 -- start start 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81e40b41a0 0x7f81e4156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81e4162da0 0x7f81e4165460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81e406f980 con 0x7f81e40b4580 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81e406f800 con 0x7f81e4162da0 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81ea800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81e406fb00 con 0x7f81e40b41a0 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81e8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81e8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60990/0 (socket says 172.21.15.67:60990) 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.902+0000 7f81e8a00640 1 -- 172.21.15.67:0/2596976855 learned_addr learned my addr 172.21.15.67:0/2596976855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 -- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81e40b41a0 msgr2=0x7f81e4156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e9e00640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81e4162da0 0x7f81e4165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81e40b41a0 0x7f81e4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 -- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81e4162da0 msgr2=0x7f81e4165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81e4162da0 0x7f81e4165460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 -- 172.21.15.67:0/2596976855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e4157a60 con 0x7f81e40b4580 2024-04-17T07:26:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e8a00640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c410 tx=0x7f81d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0017070 con 0x7f81e40b4580 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d00035f0 con 0x7f81e40b4580 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0003a20 con 0x7f81e40b4580 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.903+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e4157d40 con 0x7f81e40b4580 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.904+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e4158300 con 0x7f81e40b4580 2024-04-17T07:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.905+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e40d1370 con 0x7f81e40b4580 2024-04-17T07:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.906+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 91228+0+0 (secure 0 0 0) 0x7f81d0003bc0 con 0x7f81e40b4580 2024-04-17T07:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.907+0000 7f81e1600640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 0x7f81c406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.907+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6631+0+0 (secure 0 0 0) 0x7f81d008b630 con 0x7f81e40b4580 2024-04-17T07:26:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.907+0000 7f81e9400640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 0x7f81c406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.908+0000 7f81e9400640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 0x7f81c406efe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81d8036000 tx=0x7f81d802e000 comp rx=0 tx=0).ready entity=mgr.4712 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:51.911+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f81e40d1370 con 0x7f81e40b4580 2024-04-17T07:26:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.170+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f81e406c7b0 con 0x7f81e40b4580 2024-04-17T07:26:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.420+0000 7f81e1600640 1 -- 172.21.15.67:0/2596976855 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) ==== 93+0+0 (secure 0 0 0) 0x7f81d00593a0 con 0x7f81e40b4580 2024-04-17T07:26:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.425+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 msgr2=0x7f81c406efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.425+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 0x7f81c406efe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81d8036000 tx=0x7f81d802e000 comp rx=0 tx=0).stop 2024-04-17T07:26:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.425+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 msgr2=0x7f81e41592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.425+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f81d000c410 tx=0x7f81d000c8d0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 shutdown_connections 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f81c406cb20 0x7f81c406efe0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81e4162da0 0x7f81e4165460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e40b4580 0x7f81e41592b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 --2- 172.21.15.67:0/2596976855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81e40b41a0 0x7f81e4156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.426+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 >> 172.21.15.67:0/2596976855 conn(0x7f81e40b2190 msgr2=0x7f81e4071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.427+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 shutdown_connections 2024-04-17T07:26:52.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.427+0000 7f81ea800640 1 -- 172.21.15.67:0/2596976855 wait complete. 2024-04-17T07:26:52.447 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:26:52.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:26:52.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:26:52.626 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:26:52.626 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:26:52.626 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:26:52.626 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:26:52.684 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:26:52.684 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:26:52.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:26:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.972+0000 7fddd5200640 1 Processor -- start 2024-04-17T07:26:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.972+0000 7fddd5200640 1 -- start start 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddd5200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddd006f5d0 0x7fddd00b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddd5200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd006c4d0 con 0x7fddd006bfd0 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddd5200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd006c7b0 con 0x7fddd00b4340 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddd5200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fddd006ca90 con 0x7fddd006f5d0 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddcea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddd006bfd0 0x7fddd00afdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51392/0 (socket says 172.21.15.67:51392) 2024-04-17T07:26:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddcea00640 1 -- 172.21.15.67:0/2061975126 learned_addr learned my addr 172.21.15.67:0/2061975126 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.974+0000 7fddcfe00640 1 --2- 172.21.15.67:0/2061975126 >> v2:172.21.15.187:3300/0 conn(0x7fddd006f5d0 0x7fddd00b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781981234 0 0) 0x7fddd006c4d0 con 0x7fddd006bfd0 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddb4003520 con 0x7fddd006bfd0 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328307364 0 0) 0x7fddd006c7b0 con 0x7fddd00b4340 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddd006c4d0 con 0x7fddd00b4340 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3687318621 0 0) 0x7fddb4003520 con 0x7fddd006bfd0 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 >> v2:172.21.15.187:3300/0 conn(0x7fddd006f5d0 msgr2=0x7fddd00b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 --2- 172.21.15.67:0/2061975126 >> v2:172.21.15.187:3300/0 conn(0x7fddd006f5d0 0x7fddd00b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 >> v1:172.21.15.187:6789/0 conn(0x7fddd00b4340 legacy=0x7fddd006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd006cd70 con 0x7fddd006bfd0 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fddbc001670 con 0x7fddd006bfd0 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 >> v1:172.21.15.67:6789/0 conn(0x7fddd006bfd0 legacy=0x7fddd00afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddb4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb4004800 0x7fddb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb40051c0 0x7fddb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddd006ca90 con 0x7fddb4003bb0 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddb400c0c0 con 0x7fddb4004800 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.975+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddb400c2c0 con 0x7fddb40051c0 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddb4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcfe00640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb40051c0 0x7fddb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 -- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb40051c0 msgr2=0x7fddb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb40051c0 0x7fddb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcea00640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb4004800 0x7fddb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 -- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb4004800 msgr2=0x7fddb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb4004800 0x7fddb4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcfe00640 1 -- 172.21.15.67:0/2061975126 reap_dead start 2024-04-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.976+0000 7fddcf400640 1 -- 172.21.15.67:0/2061975126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb4003520 con 0x7fddb4003bb0 2024-04-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.977+0000 7fddcf400640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddb4003fb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fddc00018d0 tx=0x7fddb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1aeda69d668b53 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.977+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0037070 con 0x7fddb4003bb0 2024-04-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.977+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc00052d0 con 0x7fddb4003bb0 2024-04-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.977+0000 7fddce000640 1 -- 172.21.15.67:0/2061975126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc00014f0 con 0x7fddb4003bb0 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 -- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 msgr2=0x7fddb4003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddb4003fb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fddc00018d0 tx=0x7fddb4005830 comp rx=0 tx=0).stop 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 -- 172.21.15.67:0/2061975126 shutdown_connections 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 --2- 172.21.15.67:0/2061975126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddb4003fb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 -- 172.21.15.67:0/2061975126 >> 172.21.15.67:0/2061975126 conn(0x7fddd00b2ac0 msgr2=0x7fddd00b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 -- 172.21.15.67:0/2061975126 shutdown_connections 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.978+0000 7fddd5200640 1 -- 172.21.15.67:0/2061975126 wait complete. 2024-04-17T07:26:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.980+0000 7fddd5200640 1 Processor -- start 2024-04-17T07:26:52.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.980+0000 7fddd5200640 1 -- start start 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 0x7fddd014b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 0x7fddd014be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddd00b48a0 con 0x7fddb4003bb0 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddd00b4720 con 0x7fddd013d610 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddd5200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddd00b4a20 con 0x7fddd013da90 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddcf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddcf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32798/0 (socket says 172.21.15.67:32798) 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.981+0000 7fddcf400640 1 -- 172.21.15.67:0/1848768093 learned_addr learned my addr 172.21.15.67:0/1848768093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcea00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 0x7fddd014b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcfe00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 0x7fddd014be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 msgr2=0x7fddd014be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 0x7fddd014be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 msgr2=0x7fddd014b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 0x7fddd014b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 -- 172.21.15.67:0/1848768093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd0160ac0 con 0x7fddb4003bb0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcea00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 0x7fddd014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcfe00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 0x7fddd014be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.982+0000 7fddcf400640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc0035040 tx=0x7fddc00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.983+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0042020 con 0x7fddb4003bb0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.983+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc0038040 con 0x7fddb4003bb0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.983+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd0160da0 con 0x7fddb4003bb0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.983+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0037070 con 0x7fddb4003bb0 2024-04-17T07:26:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.983+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd0161360 con 0x7fddb4003bb0 2024-04-17T07:26:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.984+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd00755d0 con 0x7fddb4003bb0 2024-04-17T07:26:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.985+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7fddc0045020 con 0x7fddb4003bb0 2024-04-17T07:26:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.986+0000 7fddc7400640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.987+0000 7fddcea00640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.987+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6615+0+0 (secure 0 0 0) 0x7fddc00ad960 con 0x7fddb4003bb0 2024-04-17T07:26:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.987+0000 7fddcea00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:52.991+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fddc004c030 con 0x7fddb4003bb0 2024-04-17T07:26:53.020 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.020+0000 7fd1d043e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:26:53.054 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.054+0000 7efe13046040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:26:53.077 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.077+0000 7fd1d043e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:26:53.114 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.114+0000 7efe13046040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:26:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.188+0000 7fddcea00640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.188+0000 7fddcea00640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:53.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.204+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddd0067ba0 con 0x7fddb4003bb0 2024-04-17T07:26:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.205+0000 7fddc7400640 1 -- 172.21.15.67:0/1848768093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+18007 (secure 0 0 0) 0x7fddc007b660 con 0x7fddb4003bb0 2024-04-17T07:26:53.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:53.206 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":51,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:26:52.288989+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":215,"snap_epoch":51,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2778463946":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/3485602680":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6833/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6832/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/618455126":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/2322153279":"2024-04-18T07:15:31.130810+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2},{"begin":5,"length":2},{"begin":8,"length":3},{"begin":12,"length":2},{"begin":15,"length":4},{"begin":21,"length":1},{"begin":23,"length":6},{"begin":30,"length":1},{"begin":32,"length":2},{"begin":36,"length":5},{"begin":43,"length":1},{"begin":45,"length":5},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":57,"length":1},{"begin":59,"length":2},{"begin":62,"length":3},{"begin":68,"length":1},{"begin":70,"length":2},{"begin":73,"length":1},{"begin":75,"length":1},{"begin":77,"length":2},{"begin":80,"length":2},{"begin":84,"length":2},{"begin":87,"length":1},{"begin":89,"length":7},{"begin":97,"length":1},{"begin":100,"length":1},{"begin":103,"length":4},{"begin":108,"length":2},{"begin":111,"length":1},{"begin":144,"length":72}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":40,"length":1},{"begin":43,"length":1},{"begin":111,"length":1},{"begin":210,"length":6}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 msgr2=0x7fddd0160290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc0035040 tx=0x7fddc00012d0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 shutdown_connections 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd013da90 0x7fddd014be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd013d610 0x7fddd014b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 --2- 172.21.15.67:0/1848768093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb4003bb0 0x7fddd0160290 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.208+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 >> 172.21.15.67:0/1848768093 conn(0x7fddd00b2ac0 msgr2=0x7fddd00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.209+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 shutdown_connections 2024-04-17T07:26:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.209+0000 7fddd5200640 1 -- 172.21.15.67:0/1848768093 wait complete. 2024-04-17T07:26:53.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:15:09.343098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:15:10.202353+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 215, 'snap_epoch': 51, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:26:53.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:26:53.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.439+0000 7f4228800640 1 Processor -- start 2024-04-17T07:26:53.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.440+0000 7f4228800640 1 -- start start 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.441+0000 7f4228800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f42200b18f0 0x7f42201679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.441+0000 7f4228800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42201607a0 con 0x7f4220143200 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.441+0000 7f4228800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4220160a60 con 0x7f4220142e20 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.441+0000 7f4228800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4220160d40 con 0x7f42200b18f0 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.441+0000 7f4225400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4220143200 0x7f42200cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35104/0 (socket says 172.21.15.67:35104) 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.442+0000 7f4225400640 1 -- 172.21.15.67:0/2069016901 learned_addr learned my addr 172.21.15.67:0/2069016901 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.442+0000 7f4226800640 1 --2- 172.21.15.67:0/2069016901 >> v2:172.21.15.187:3300/0 conn(0x7f42200b18f0 0x7f42201679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.443+0000 7f4226800640 1 -- 172.21.15.67:0/2069016901 >> v1:172.21.15.187:6789/0 conn(0x7f4220142e20 legacy=0x7f42200cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.443+0000 7f4224a00640 1 -- 172.21.15.67:0/2069016901 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582114770 0 0) 0x7f42201607a0 con 0x7f4220143200 2024-04-17T07:26:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.443+0000 7f4226800640 1 -- 172.21.15.67:0/2069016901 >> v1:172.21.15.67:6789/0 conn(0x7f4220143200 legacy=0x7f42200cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.443+0000 7f4226800640 1 -- 172.21.15.67:0/2069016901 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4220161020 con 0x7f42200b18f0 2024-04-17T07:26:53.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.444+0000 7f4226800640 1 --2- 172.21.15.67:0/2069016901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42200b18f0 0x7f42201679b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f421c009dc0 tx=0x7f421c02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f90a99476983b6c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4224a00640 1 -- 172.21.15.67:0/2069016901 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c039070 con 0x7f42200b18f0 2024-04-17T07:26:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4224a00640 1 -- 172.21.15.67:0/2069016901 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f421c02cc40 con 0x7f42200b18f0 2024-04-17T07:26:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4224a00640 1 -- 172.21.15.67:0/2069016901 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c034440 con 0x7f42200b18f0 2024-04-17T07:26:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 -- 172.21.15.67:0/2069016901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42200b18f0 msgr2=0x7f42201679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 --2- 172.21.15.67:0/2069016901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42200b18f0 0x7f42201679b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f421c009dc0 tx=0x7f421c02be90 comp rx=0 tx=0).stop 2024-04-17T07:26:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 -- 172.21.15.67:0/2069016901 shutdown_connections 2024-04-17T07:26:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 --2- 172.21.15.67:0/2069016901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42200b18f0 0x7f42201679b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 -- 172.21.15.67:0/2069016901 >> 172.21.15.67:0/2069016901 conn(0x7f422006b8c0 msgr2=0x7f422006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 -- 172.21.15.67:0/2069016901 shutdown_connections 2024-04-17T07:26:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.448+0000 7f4228800640 1 -- 172.21.15.67:0/2069016901 wait complete. 2024-04-17T07:26:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.450+0000 7f4228800640 1 Processor -- start 2024-04-17T07:26:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.450+0000 7f4228800640 1 -- start start 2024-04-17T07:26:53.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 0x7f42200b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 0x7f4220158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42200ce730 con 0x7f4220142e20 2024-04-17T07:26:53.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f42200cf330 con 0x7f4220143200 2024-04-17T07:26:53.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4228800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f42200cd750 con 0x7f42200b18f0 2024-04-17T07:26:53.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4225400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4225400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46936/0 (socket says 172.21.15.67:46936) 2024-04-17T07:26:53.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.451+0000 7f4225400640 1 -- 172.21.15.67:0/1659992702 learned_addr learned my addr 172.21.15.67:0/1659992702 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:53.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4226800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 0x7f4220158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 msgr2=0x7f42200b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225e00640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 0x7f42200b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 0x7f42200b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 msgr2=0x7f4220158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 0x7f4220158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 -- 172.21.15.67:0/1659992702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42200b3c30 con 0x7f4220142e20 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4226800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 0x7f4220158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225e00640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 0x7f42200b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.452+0000 7f4225400640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f421000c3f0 tx=0x7f421000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.453+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42100074e0 con 0x7f4220142e20 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.453+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42100047e0 con 0x7f4220142e20 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.453+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42200b3f10 con 0x7f4220142e20 2024-04-17T07:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.453+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210004bd0 con 0x7f4220142e20 2024-04-17T07:26:53.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.453+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42200b44d0 con 0x7f4220142e20 2024-04-17T07:26:53.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.454+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4220068b90 con 0x7f4220142e20 2024-04-17T07:26:53.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.455+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7f4210011070 con 0x7f4220142e20 2024-04-17T07:26:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.456+0000 7f421a000640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 0x7f420406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.456+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6627+0+0 (secure 0 0 0) 0x7f421008be20 con 0x7f4220142e20 2024-04-17T07:26:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.457+0000 7f4225e00640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 msgr2=0x7f420406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.457+0000 7f4225e00640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 0x7f420406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:53.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.461+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f421001a2b0 con 0x7f4220142e20 2024-04-17T07:26:53.558 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.558+0000 7fd1d043e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:26:53.610 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.610+0000 7fd1d043e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:26:53.617 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.618+0000 7efe13046040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:26:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.658+0000 7f4225e00640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 msgr2=0x7f420406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.658+0000 7f4225e00640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 0x7f420406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:53.659 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.659+0000 7fd1d043e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:26:53.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.661+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f422015b290 con 0x7f4220142e20 2024-04-17T07:26:53.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.662+0000 7f421a000640 1 -- 172.21.15.67:0/1659992702 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f4210059c40 con 0x7f4220142e20 2024-04-17T07:26:53.662 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:26:53.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 msgr2=0x7f420406ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 0x7f420406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 msgr2=0x7f42200ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f421000c3f0 tx=0x7f421000c8b0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 shutdown_connections 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f420406caa0 0x7f420406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4220143200 0x7f4220158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4220142e20 0x7f42200ccdc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 --2- 172.21.15.67:0/1659992702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42200b18f0 0x7f42200b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 >> 172.21.15.67:0/1659992702 conn(0x7f422006b8c0 msgr2=0x7f422015fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 shutdown_connections 2024-04-17T07:26:53.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.664+0000 7f4228800640 1 -- 172.21.15.67:0/1659992702 wait complete. 2024-04-17T07:26:53.675 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.675+0000 7efe13046040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:26:53.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:26:53.720 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.719+0000 7fd1d043e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:26:53.731 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.731+0000 7efe13046040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:26:53.768 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.768+0000 7fd1d043e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:26:53.798 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.798+0000 7efe13046040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:26:53.857 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:53.857+0000 7efe13046040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:26:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.881+0000 7f6a2ba00640 1 Processor -- start 2024-04-17T07:26:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.881+0000 7f6a2ba00640 1 -- start start 2024-04-17T07:26:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a2ba00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a240d0980 0x7f6a240d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a2ba00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2415d800 con 0x7f6a24160c30 2024-04-17T07:26:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a2ba00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2415dac0 con 0x7f6a24161010 2024-04-17T07:26:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a2ba00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6a240d1330 con 0x7f6a240d0980 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a29000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6a24160c30 0x7f6a2415cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35114/0 (socket says 172.21.15.67:35114) 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a29000640 1 -- 172.21.15.67:0/1170249384 learned_addr learned my addr 172.21.15.67:0/1170249384 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.882+0000 7f6a29a00640 1 --2- 172.21.15.67:0/1170249384 >> v2:172.21.15.67:3300/0 conn(0x7f6a240d0980 0x7f6a240d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.883+0000 7f6a23400640 1 -- 172.21.15.67:0/1170249384 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3119602451 0 0) 0x7f6a2415d800 con 0x7f6a24160c30 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.883+0000 7f6a23400640 1 -- 172.21.15.67:0/1170249384 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c003520 con 0x7f6a24160c30 2024-04-17T07:26:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.883+0000 7f6a29a00640 1 -- 172.21.15.67:0/1170249384 >> v1:172.21.15.187:6789/0 conn(0x7f6a24161010 legacy=0x7f6a2406c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.883+0000 7f6a29a00640 1 -- 172.21.15.67:0/1170249384 >> v1:172.21.15.67:6789/0 conn(0x7f6a24160c30 legacy=0x7f6a2415cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.883+0000 7f6a29a00640 1 -- 172.21.15.67:0/1170249384 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2415dda0 con 0x7f6a240d0980 2024-04-17T07:26:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.884+0000 7f6a29a00640 1 --2- 172.21.15.67:0/1170249384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a240d0980 0x7f6a240d0df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009d90 tx=0x7f6a1c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa47f6292be46cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.885+0000 7f6a23400640 1 -- 172.21.15.67:0/1170249384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c039070 con 0x7f6a240d0980 2024-04-17T07:26:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.885+0000 7f6a23400640 1 -- 172.21.15.67:0/1170249384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1c02ccf0 con 0x7f6a240d0980 2024-04-17T07:26:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.885+0000 7f6a23400640 1 -- 172.21.15.67:0/1170249384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c034660 con 0x7f6a240d0980 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.885+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1170249384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a240d0980 msgr2=0x7f6a240d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.885+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1170249384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a240d0980 0x7f6a240d0df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009d90 tx=0x7f6a1c02bf40 comp rx=0 tx=0).stop 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.886+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1170249384 shutdown_connections 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.886+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1170249384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a240d0980 0x7f6a240d0df0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.886+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1170249384 >> 172.21.15.67:0/1170249384 conn(0x7f6a24073360 msgr2=0x7f6a2409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.886+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1170249384 shutdown_connections 2024-04-17T07:26:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.886+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1170249384 wait complete. 2024-04-17T07:26:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.887+0000 7f6a2ba00640 1 Processor -- start 2024-04-17T07:26:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 -- start start 2024-04-17T07:26:53.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 0x7f6a240b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 0x7f6a240cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2406bba0 con 0x7f6a24161010 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2406b8c0 con 0x7f6a24160c30 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.888+0000 7f6a2ba00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6a241613f0 con 0x7f6a240d0980 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46960/0 (socket says 172.21.15.67:46960) 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 -- 172.21.15.67:0/1147187102 learned_addr learned my addr 172.21.15.67:0/1147187102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29000640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 0x7f6a240b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a23e00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 0x7f6a240cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 msgr2=0x7f6a240b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 0x7f6a240b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 msgr2=0x7f6a240cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 0x7f6a240cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29a00640 1 -- 172.21.15.67:0/1147187102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a240c7cf0 con 0x7f6a24161010 2024-04-17T07:26:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a29000640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 0x7f6a240b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.889+0000 7f6a23e00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 0x7f6a240cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a29a00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009eb0 tx=0x7f6a1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c039050 con 0x7f6a24161010 2024-04-17T07:26:53.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1c045070 con 0x7f6a24161010 2024-04-17T07:26:53.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c009450 con 0x7f6a24161010 2024-04-17T07:26:53.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a240c7fd0 con 0x7f6a24161010 2024-04-17T07:26:53.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.890+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a24156af0 con 0x7f6a24161010 2024-04-17T07:26:53.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.891+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a24068ca0 con 0x7f6a24161010 2024-04-17T07:26:53.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.893+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7f6a1c03e850 con 0x7f6a24161010 2024-04-17T07:26:53.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.894+0000 7f6a20c00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 0x7f6a0406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:53.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.894+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6627+0+0 (secure 0 0 0) 0x7f6a1c0aee30 con 0x7f6a24161010 2024-04-17T07:26:53.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.894+0000 7f6a29000640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 msgr2=0x7f6a0406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:53.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.894+0000 7f6a29000640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 0x7f6a0406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:53.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:53.903+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6a1c032e20 con 0x7f6a24161010 2024-04-17T07:26:53.905 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.905+0000 7fd1d043e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:26:53.974 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:53.972+0000 7fd1d043e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:26:54.020 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.020+0000 7efe13046040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:26:54.039 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.040+0000 7fd1d043e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:26:54.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.095+0000 7f6a29000640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 msgr2=0x7f6a0406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:54.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.095+0000 7f6a29000640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 0x7f6a0406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:54.100 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.100+0000 7fd1d043e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:26:54.102 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.102+0000 7efe13046040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:26:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.130+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6a240b0950 con 0x7f6a24161010 2024-04-17T07:26:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.131+0000 7f6a20c00640 1 -- 172.21.15.67:0/1147187102 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) ==== 104+0+11 (secure 0 0 0) 0x7f6a1c07cc20 con 0x7f6a24161010 2024-04-17T07:26:54.131 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:26:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 msgr2=0x7f6a0406ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 0x7f6a0406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 msgr2=0x7f6a240af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009eb0 tx=0x7f6a1c004a40 comp rx=0 tx=0).stop 2024-04-17T07:26:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 shutdown_connections 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f6a0406caa0 0x7f6a0406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a24161010 0x7f6a240af410 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a24160c30 0x7f6a240cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 --2- 172.21.15.67:0/1147187102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a240d0980 0x7f6a240b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 >> 172.21.15.67:0/1147187102 conn(0x7f6a24073360 msgr2=0x7f6a240d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 shutdown_connections 2024-04-17T07:26:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.133+0000 7f6a2ba00640 1 -- 172.21.15.67:0/1147187102 wait complete. 2024-04-17T07:26:54.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:26:54.151 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.150+0000 7fd1d043e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:26:54.157 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.158+0000 7efe13046040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:26:54.221 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.221+0000 7efe13046040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:26:54.277 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.276+0000 7efe13046040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:26:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.364+0000 7f24d8a00640 1 Processor -- start 2024-04-17T07:26:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.364+0000 7f24d8a00640 1 -- start start 2024-04-17T07:26:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.366+0000 7f24d8a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f24d406eda0 0x7f24d406f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.367+0000 7f24d8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d406dfe0 con 0x7f24d4071e40 2024-04-17T07:26:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.367+0000 7f24d8a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d406e2c0 con 0x7f24d41675e0 2024-04-17T07:26:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.367+0000 7f24d8a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f24d406e5a0 con 0x7f24d406eda0 2024-04-17T07:26:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.368+0000 7f24d2800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24d4071e40 0x7f24d4072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35120/0 (socket says 172.21.15.67:35120) 2024-04-17T07:26:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.368+0000 7f24d2800640 1 -- 172.21.15.67:0/464814222 learned_addr learned my addr 172.21.15.67:0/464814222 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.368+0000 7f24d1e00640 1 --2- 172.21.15.67:0/464814222 >> v2:172.21.15.187:3300/0 conn(0x7f24d406eda0 0x7f24d406f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.369+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849011382 0 0) 0x7f24d406e2c0 con 0x7f24d41675e0 2024-04-17T07:26:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.369+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b8003520 con 0x7f24d41675e0 2024-04-17T07:26:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.369+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167585391 0 0) 0x7f24d406dfe0 con 0x7f24d4071e40 2024-04-17T07:26:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.369+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24d406e2c0 con 0x7f24d4071e40 2024-04-17T07:26:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.369+0000 7f24d1e00640 1 -- 172.21.15.67:0/464814222 >> v1:172.21.15.187:6789/0 conn(0x7f24d41675e0 legacy=0x7f24d40b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.370+0000 7f24d1e00640 1 -- 172.21.15.67:0/464814222 >> v1:172.21.15.67:6789/0 conn(0x7f24d4071e40 legacy=0x7f24d4072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.370+0000 7f24d1e00640 1 -- 172.21.15.67:0/464814222 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d406e880 con 0x7f24d406eda0 2024-04-17T07:26:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.371+0000 7f24d1e00640 1 --2- 172.21.15.67:0/464814222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d406eda0 0x7f24d406f3b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097a0 tx=0x7f24c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1947d3a71012409 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.372+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0035020 con 0x7f24d406eda0 2024-04-17T07:26:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.372+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c0031830 con 0x7f24d406eda0 2024-04-17T07:26:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.372+0000 7f24d1400640 1 -- 172.21.15.67:0/464814222 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c0031be0 con 0x7f24d406eda0 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 -- 172.21.15.67:0/464814222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d406eda0 msgr2=0x7f24d406f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 --2- 172.21.15.67:0/464814222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d406eda0 0x7f24d406f3b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097a0 tx=0x7f24c002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 -- 172.21.15.67:0/464814222 shutdown_connections 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 --2- 172.21.15.67:0/464814222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d406eda0 0x7f24d406f3b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 -- 172.21.15.67:0/464814222 >> 172.21.15.67:0/464814222 conn(0x7f24d40b23c0 msgr2=0x7f24d40b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 -- 172.21.15.67:0/464814222 shutdown_connections 2024-04-17T07:26:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.373+0000 7f24d8a00640 1 -- 172.21.15.67:0/464814222 wait complete. 2024-04-17T07:26:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.375+0000 7f24d8a00640 1 Processor -- start 2024-04-17T07:26:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.375+0000 7f24d8a00640 1 -- start start 2024-04-17T07:26:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d4071e40 0x7f24d415e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24d41675e0 0x7f24d414d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24d406fa70 con 0x7f24d406eda0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f24d406f8f0 con 0x7f24d4071e40 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24d406fbf0 con 0x7f24d41675e0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46972/0 (socket says 172.21.15.67:46972) 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 -- 172.21.15.67:0/851840612 learned_addr learned my addr 172.21.15.67:0/851840612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d1e00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d4071e40 0x7f24d415e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d3200640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24d41675e0 0x7f24d414d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24d41675e0 msgr2=0x7f24d414d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24d41675e0 0x7f24d414d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d4071e40 msgr2=0x7f24d415e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d4071e40 0x7f24d415e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.376+0000 7f24d2800640 1 -- 172.21.15.67:0/851840612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d41655f0 con 0x7f24d406eda0 2024-04-17T07:26:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24d2800640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c500 tx=0x7f24c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4017070 con 0x7f24d406eda0 2024-04-17T07:26:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c4005ce0 con 0x7f24d406eda0 2024-04-17T07:26:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4007330 con 0x7f24d406eda0 2024-04-17T07:26:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d41658d0 con 0x7f24d406eda0 2024-04-17T07:26:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.377+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d4165e90 con 0x7f24d406eda0 2024-04-17T07:26:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.378+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d4068a10 con 0x7f24d406eda0 2024-04-17T07:26:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.379+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7f24c401a070 con 0x7f24d406eda0 2024-04-17T07:26:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.380+0000 7f24caa00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 0x7f24b406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.380+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6455+0+0 (secure 0 0 0) 0x7f24c408b590 con 0x7f24d406eda0 2024-04-17T07:26:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.380+0000 7f24d1e00640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 msgr2=0x7f24b406ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.380+0000 7f24d1e00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 0x7f24b406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.389+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f24c4058eb0 con 0x7f24d406eda0 2024-04-17T07:26:54.554 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.554+0000 7fd1d043e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:26:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.581+0000 7f24d1e00640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 msgr2=0x7f24b406ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.581+0000 7f24d1e00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 0x7f24b406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.592+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f24d4068610 con 0x7f24d406eda0 2024-04-17T07:26:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.593+0000 7f24caa00640 1 -- 172.21.15.67:0/851840612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) ==== 100+0+11 (secure 0 0 0) 0x7f24c405d1b0 con 0x7f24d406eda0 2024-04-17T07:26:54.593 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.595+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 msgr2=0x7f24b406ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.595+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 0x7f24b406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.595+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 msgr2=0x7f24d4163230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.595+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c500 tx=0x7f24c400c9c0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 shutdown_connections 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f24b406cad0 0x7f24b406ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24d41675e0 0x7f24d414d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24d4071e40 0x7f24d415e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 --2- 172.21.15.67:0/851840612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d406eda0 0x7f24d4163230 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 >> 172.21.15.67:0/851840612 conn(0x7f24d40b23c0 msgr2=0x7f24d4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 shutdown_connections 2024-04-17T07:26:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.596+0000 7f24d8a00640 1 -- 172.21.15.67:0/851840612 wait complete. 2024-04-17T07:26:54.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:26:54.712 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.711+0000 7fd1d043e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:26:54.718 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.718+0000 7efe13046040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:26:54.823 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.823+0000 7fd1d043e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:26:54.874 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:54.874+0000 7fd1d043e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:26:54.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.874+0000 7fddd5600640 1 Processor -- start 2024-04-17T07:26:54.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.874+0000 7fddd5600640 1 -- start start 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddd0145ac0 0x7fddd0071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd00ce3b0 0x7fddd0072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0072bb0 0x7fddd00ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fddd00ce840 con 0x7fddd00ce3b0 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fddd00cea70 con 0x7fddd0145ac0 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddd5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddd00ced50 con 0x7fddd0072bb0 2024-04-17T07:26:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddce000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd00ce3b0 0x7fddd0072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddce000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddd00ce3b0 0x7fddd0072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46980/0 (socket says 172.21.15.67:46980) 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.876+0000 7fddce000640 1 -- 172.21.15.67:0/740115981 learned_addr learned my addr 172.21.15.67:0/740115981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddcea00640 1 --2- 172.21.15.67:0/740115981 >> v2:172.21.15.187:3300/0 conn(0x7fddd0145ac0 0x7fddd0071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddce000640 1 -- 172.21.15.67:0/740115981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0072bb0 msgr2=0x7fddd00ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddce000640 1 --2- 172.21.15.67:0/740115981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0072bb0 0x7fddd00ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddce000640 1 -- 172.21.15.67:0/740115981 >> v2:172.21.15.187:3300/0 conn(0x7fddd0145ac0 msgr2=0x7fddd0071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddce000640 1 --2- 172.21.15.67:0/740115981 >> v2:172.21.15.187:3300/0 conn(0x7fddd0145ac0 0x7fddd0071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.877+0000 7fddce000640 1 -- 172.21.15.67:0/740115981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd00cf030 con 0x7fddd00ce3b0 2024-04-17T07:26:54.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.878+0000 7fddce000640 1 --2- 172.21.15.67:0/740115981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd00ce3b0 0x7fddd0072670 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009870 tx=0x7fddbc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d474a1311968ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:54.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.878+0000 7fddcd600640 1 -- 172.21.15.67:0/740115981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc035020 con 0x7fddd00ce3b0 2024-04-17T07:26:54.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.878+0000 7fddcd600640 1 -- 172.21.15.67:0/740115981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddbc031640 con 0x7fddd00ce3b0 2024-04-17T07:26:54.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.878+0000 7fddcd600640 1 -- 172.21.15.67:0/740115981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc02d040 con 0x7fddd00ce3b0 2024-04-17T07:26:54.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 -- 172.21.15.67:0/740115981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd00ce3b0 msgr2=0x7fddd0072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 --2- 172.21.15.67:0/740115981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd00ce3b0 0x7fddd0072670 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009870 tx=0x7fddbc02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 -- 172.21.15.67:0/740115981 shutdown_connections 2024-04-17T07:26:54.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 --2- 172.21.15.67:0/740115981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0072bb0 0x7fddd00ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 --2- 172.21.15.67:0/740115981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd00ce3b0 0x7fddd0072670 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 --2- 172.21.15.67:0/740115981 >> v2:172.21.15.187:3300/0 conn(0x7fddd0145ac0 0x7fddd0071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 -- 172.21.15.67:0/740115981 >> 172.21.15.67:0/740115981 conn(0x7fddd0067ee0 msgr2=0x7fddd006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 -- 172.21.15.67:0/740115981 shutdown_connections 2024-04-17T07:26:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.879+0000 7fddd5600640 1 -- 172.21.15.67:0/740115981 wait complete. 2024-04-17T07:26:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.881+0000 7fddd5600640 1 Processor -- start 2024-04-17T07:26:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.881+0000 7fddd5600640 1 -- start start 2024-04-17T07:26:54.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd00ce3b0 0x7fddd0131590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0145ac0 0x7fddd01e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddd0145ea0 con 0x7fddd0072bb0 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddd00cf5f0 con 0x7fddd00ce3b0 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddd5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddd00cd3e0 con 0x7fddd0145ac0 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46994/0 (socket says 172.21.15.67:46994) 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.882+0000 7fddcea00640 1 -- 172.21.15.67:0/3188476756 learned_addr learned my addr 172.21.15.67:0/3188476756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddce000640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd00ce3b0 0x7fddd0131590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0145ac0 msgr2=0x7fddd01e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0145ac0 0x7fddd01e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd00ce3b0 msgr2=0x7fddd0131590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd00ce3b0 0x7fddd0131590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 -- 172.21.15.67:0/3188476756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd01f56a0 con 0x7fddd0072bb0 2024-04-17T07:26:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddcea00640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fddc000d620 tx=0x7fddc000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc000b620 con 0x7fddd0072bb0 2024-04-17T07:26:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc0019070 con 0x7fddd0072bb0 2024-04-17T07:26:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd01f4e00 con 0x7fddd0072bb0 2024-04-17T07:26:54.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc000ba00 con 0x7fddd0072bb0 2024-04-17T07:26:54.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.883+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd01f53c0 con 0x7fddd0072bb0 2024-04-17T07:26:54.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.884+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd0068ca0 con 0x7fddd0072bb0 2024-04-17T07:26:54.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.886+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7fddc000bba0 con 0x7fddd0072bb0 2024-04-17T07:26:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.887+0000 7fddc6a00640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.887+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6455+0+0 (secure 0 0 0) 0x7fddc008c5f0 con 0x7fddd0072bb0 2024-04-17T07:26:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.887+0000 7fddce000640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.887+0000 7fddce000640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:54.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:54.891+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fddc005a420 con 0x7fddd0072bb0 2024-04-17T07:26:54.897 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:54.897+0000 7efe13046040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:26:55.002 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:55.002+0000 7fd1d043e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:26:55.016 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.015+0000 7efe13046040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:26:55.071 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.071+0000 7efe13046040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:26:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.088+0000 7fddce000640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.088+0000 7fddce000640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:55.108 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:55.108+0000 7fd1d043e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:26:55.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.110+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fddd0144d20 con 0x7fddd0072bb0 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.111+0000 7fddc6a00640 1 -- 172.21.15.67:0/3188476756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v53) ==== 80+0+359 (secure 0 0 0) 0x7fddc0059d20 con 0x7fddd0072bb0 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2778463946 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3485602680 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/3291261531 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/3291261531 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/618455126 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.112 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2322153279 2024-04-18T07:15:31.130810+0000 2024-04-17T07:26:55.112 INFO:teuthology.orchestra.run.smithi067.stderr:listed 6 entries 2024-04-17T07:26:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 msgr2=0x7fddb006ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 msgr2=0x7fddd01f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fddc000d620 tx=0x7fddc000dae0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 shutdown_connections 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fddb006caa0 0x7fddb006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddd0145ac0 0x7fddd01e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddd00ce3b0 0x7fddd0131590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 --2- 172.21.15.67:0/3188476756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddd0072bb0 0x7fddd01f46d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 >> 172.21.15.67:0/3188476756 conn(0x7fddd0067ee0 msgr2=0x7fddd00cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 shutdown_connections 2024-04-17T07:26:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.114+0000 7fddd5600640 1 -- 172.21.15.67:0/3188476756 wait complete. 2024-04-17T07:26:55.133 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:26:55.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:55.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:26:55.208 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.208+0000 7efe13046040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:26:55.234 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:26:55.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:26:55.235 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:26:55.279 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:26:55.280 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:26:55.280 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:26:55.291 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:55.291+0000 7fd1d043e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:26:55.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:26:55.326 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:26:55.326 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:26:55.326 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:26:55.333 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.329+0000 7efe13046040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:26:55.345 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:55.345+0000 7fd1d043e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:26:55.377 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:26:55.378 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:26:55.378 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:26:55.378 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:26:55.424 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:26:55.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:26:55.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:55.490+0000 7fd1d043e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:26:55.521 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.521+0000 7efe13046040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:26:55.575 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.575+0000 7efe13046040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:26:55.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.628+0000 7ff4d4c00640 1 Processor -- start 2024-04-17T07:26:55.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.628+0000 7ff4d4c00640 1 -- start start 2024-04-17T07:26:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.630+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.630+0000 7ff4d4c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff4d00d1bd0 0x7ff4d00ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.630+0000 7ff4d4c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d00d11c0 con 0x7ff4d00b2730 2024-04-17T07:26:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.630+0000 7ff4d4c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d00cf7a0 con 0x7ff4d00d1bd0 2024-04-17T07:26:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.630+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d00b2ca0 con 0x7ff4d00b0000 2024-04-17T07:26:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.631+0000 7ff4cf400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.631+0000 7ff4cf400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:55372/0 (socket says 172.21.15.67:55372) 2024-04-17T07:26:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.631+0000 7ff4cf400640 1 -- 172.21.15.67:0/3926363489 learned_addr learned my addr 172.21.15.67:0/3926363489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.631+0000 7ff4cfe00640 1 --2- 172.21.15.67:0/3926363489 >> v2:172.21.15.187:3300/0 conn(0x7ff4d00d1bd0 0x7ff4d00ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.632+0000 7ff4cf400640 1 -- 172.21.15.67:0/3926363489 >> v2:172.21.15.187:3300/0 conn(0x7ff4d00d1bd0 msgr2=0x7ff4d00ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.632+0000 7ff4cf400640 1 --2- 172.21.15.67:0/3926363489 >> v2:172.21.15.187:3300/0 conn(0x7ff4d00d1bd0 0x7ff4d00ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.632+0000 7ff4cf400640 1 -- 172.21.15.67:0/3926363489 >> v1:172.21.15.187:6789/0 conn(0x7ff4d00b2730 legacy=0x7ff4d00d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.632+0000 7ff4ce000640 1 -- 172.21.15.67:0/3926363489 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 162089824 0 0) 0x7ff4d00d11c0 con 0x7ff4d00b2730 2024-04-17T07:26:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.632+0000 7ff4cf400640 1 -- 172.21.15.67:0/3926363489 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d00b19b0 con 0x7ff4d00b0000 2024-04-17T07:26:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.633+0000 7ff4cf400640 1 --2- 172.21.15.67:0/3926363489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009870 tx=0x7ff4c002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dff3b6da7318cc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4ce000640 1 -- 172.21.15.67:0/3926363489 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0034070 con 0x7ff4d00b0000 2024-04-17T07:26:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4ce000640 1 -- 172.21.15.67:0/3926363489 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c0030840 con 0x7ff4d00b0000 2024-04-17T07:26:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4ce000640 1 -- 172.21.15.67:0/3926363489 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0030c10 con 0x7ff4d00b0000 2024-04-17T07:26:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 -- 172.21.15.67:0/3926363489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 msgr2=0x7ff4d00d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/3926363489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009870 tx=0x7ff4c002b980 comp rx=0 tx=0).stop 2024-04-17T07:26:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 -- 172.21.15.67:0/3926363489 shutdown_connections 2024-04-17T07:26:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/3926363489 >> v2:172.21.15.187:3300/0 conn(0x7ff4d00d1bd0 0x7ff4d00ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/3926363489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d00d0c80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 -- 172.21.15.67:0/3926363489 >> 172.21.15.67:0/3926363489 conn(0x7ff4d009aa60 msgr2=0x7ff4d00b6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:55.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 -- 172.21.15.67:0/3926363489 shutdown_connections 2024-04-17T07:26:55.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.634+0000 7ff4d4c00640 1 -- 172.21.15.67:0/3926363489 wait complete. 2024-04-17T07:26:55.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.636+0000 7ff4d4c00640 1 Processor -- start 2024-04-17T07:26:55.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.636+0000 7ff4d4c00640 1 -- start start 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d01f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d00d1bd0 0x7ff4d00b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d00b03e0 con 0x7ff4d00b2730 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d00b1f70 con 0x7ff4d00d1bd0 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.637+0000 7ff4d4c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d00ae900 con 0x7ff4d00b0000 2024-04-17T07:26:55.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47006/0 (socket says 172.21.15.67:47006) 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 -- 172.21.15.67:0/200566243 learned_addr learned my addr 172.21.15.67:0/200566243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cfe00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d00d1bd0 0x7ff4d00b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cf400640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d01f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 msgr2=0x7ff4d01f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d01f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d00d1bd0 msgr2=0x7ff4d00b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d00d1bd0 0x7ff4d00b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 -- 172.21.15.67:0/200566243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d01f4fe0 con 0x7ff4d00b2730 2024-04-17T07:26:55.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.638+0000 7ff4cea00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c4b0 tx=0x7ff4bc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.639+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc017070 con 0x7ff4d00b2730 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.639+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc005480 con 0x7ff4d00b2730 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.639+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc003830 con 0x7ff4d00b2730 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.639+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d01f52c0 con 0x7ff4d00b2730 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.639+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d01f5880 con 0x7ff4d00b2730 2024-04-17T07:26:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.640+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d00cee80 con 0x7ff4d00b2730 2024-04-17T07:26:55.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.641+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7ff4bc01a070 con 0x7ff4d00b2730 2024-04-17T07:26:55.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.642+0000 7ff4c7400640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 0x7ff4b006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:55.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.642+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6455+0+0 (secure 0 0 0) 0x7ff4bc08b630 con 0x7ff4d00b2730 2024-04-17T07:26:55.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.642+0000 7ff4cf400640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 msgr2=0x7ff4b006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:55.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.642+0000 7ff4cf400640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 0x7ff4b006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.646+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff4bc059450 con 0x7ff4d00b2730 2024-04-17T07:26:55.725 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:55.725+0000 7efe13046040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:26:55.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.843+0000 7ff4cf400640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 msgr2=0x7ff4b006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:55.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.843+0000 7ff4cf400640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 0x7ff4b006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:55.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.892+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4d01e6c30 con 0x7ff4d00b2730 2024-04-17T07:26:55.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.893+0000 7ff4c7400640 1 -- 172.21.15.67:0/200566243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7ff4bc058d50 con 0x7ff4d00b2730 2024-04-17T07:26:55.894 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:55.894 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:26:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:26:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 msgr2=0x7ff4b006ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 0x7ff4b006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 msgr2=0x7ff4d015bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c4b0 tx=0x7ff4bc00c970 comp rx=0 tx=0).stop 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 shutdown_connections 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7ff4b006caa0 0x7ff4b006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d00d1bd0 0x7ff4d00b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d00b2730 0x7ff4d015bda0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 --2- 172.21.15.67:0/200566243 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4d00b0000 0x7ff4d01f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 >> 172.21.15.67:0/200566243 conn(0x7ff4d009aa60 msgr2=0x7ff4d00aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 shutdown_connections 2024-04-17T07:26:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:55.896+0000 7ff4d4c00640 1 -- 172.21.15.67:0/200566243 wait complete. 2024-04-17T07:26:55.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:26:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.111+0000 7f76fca00640 1 Processor -- start 2024-04-17T07:26:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.111+0000 7f76fca00640 1 -- start start 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76fca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76f815e9f0 0x7f76f80ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76fca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76f80cd1b0 0x7f76f80c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76fca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f80cbbd0 con 0x7f76f815edd0 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76fca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f76f80cbeb0 con 0x7f76f815e9f0 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76fca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f76f80cc190 con 0x7f76f80cd1b0 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76f6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76f815e9f0 0x7f76f80ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76f6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76f815e9f0 0x7f76f80ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47020/0 (socket says 172.21.15.67:47020) 2024-04-17T07:26:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.113+0000 7f76f6800640 1 -- 172.21.15.67:0/145457584 learned_addr learned my addr 172.21.15.67:0/145457584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.114+0000 7f76f6800640 1 -- 172.21.15.67:0/145457584 >> v2:172.21.15.187:3300/0 conn(0x7f76f80cd1b0 msgr2=0x7f76f80c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.114+0000 7f76f6800640 1 --2- 172.21.15.67:0/145457584 >> v2:172.21.15.187:3300/0 conn(0x7f76f80cd1b0 0x7f76f80c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.114+0000 7f76f6800640 1 -- 172.21.15.67:0/145457584 >> v1:172.21.15.187:6789/0 conn(0x7f76f815edd0 legacy=0x7f76f80ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.114+0000 7f76f6800640 1 -- 172.21.15.67:0/145457584 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f80cc470 con 0x7f76f815e9f0 2024-04-17T07:26:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.115+0000 7f76f6800640 1 --2- 172.21.15.67:0/145457584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80ccc70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f76f80c7b50 tx=0x7f76e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b900f586f808012 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.115+0000 7f76f5400640 1 -- 172.21.15.67:0/145457584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8035020 con 0x7f76f815e9f0 2024-04-17T07:26:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.115+0000 7f76f5400640 1 -- 172.21.15.67:0/145457584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e8031870 con 0x7f76f815e9f0 2024-04-17T07:26:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.115+0000 7f76f5400640 1 -- 172.21.15.67:0/145457584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8031c40 con 0x7f76f815e9f0 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 -- 172.21.15.67:0/145457584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 msgr2=0x7f76f80ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 --2- 172.21.15.67:0/145457584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80ccc70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f76f80c7b50 tx=0x7f76e802b910 comp rx=0 tx=0).stop 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 -- 172.21.15.67:0/145457584 shutdown_connections 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 --2- 172.21.15.67:0/145457584 >> v2:172.21.15.187:3300/0 conn(0x7f76f80cd1b0 0x7f76f80c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 --2- 172.21.15.67:0/145457584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80ccc70 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 -- 172.21.15.67:0/145457584 >> 172.21.15.67:0/145457584 conn(0x7f76f80cd710 msgr2=0x7f76f80cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 -- 172.21.15.67:0/145457584 shutdown_connections 2024-04-17T07:26:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.116+0000 7f76fca00640 1 -- 172.21.15.67:0/145457584 wait complete. 2024-04-17T07:26:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.118+0000 7f76fca00640 1 Processor -- start 2024-04-17T07:26:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.118+0000 7f76fca00640 1 -- start start 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76f80cd1b0 0x7f76f8160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76f815edd0 0x7f76f80c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76f80cf4b0 con 0x7f76f815e9f0 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f76f80cb8a0 con 0x7f76f80cd1b0 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76fca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76f80cf630 con 0x7f76f815edd0 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.119+0000 7f76f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47036/0 (socket says 172.21.15.67:47036) 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 -- 172.21.15.67:0/2129713490 learned_addr learned my addr 172.21.15.67:0/2129713490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f6800640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76f80cd1b0 0x7f76f8160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76f815edd0 msgr2=0x7f76f80c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76f815edd0 0x7f76f80c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76f80cd1b0 msgr2=0x7f76f8160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76f80cd1b0 0x7f76f8160310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 -- 172.21.15.67:0/2129713490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f8146260 con 0x7f76f815e9f0 2024-04-17T07:26:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.120+0000 7f76f5e00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f76e000b3e0 tx=0x7f76e000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.121+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0015020 con 0x7f76f815e9f0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.121+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e0011850 con 0x7f76f815e9f0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.121+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0011c40 con 0x7f76f815e9f0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.121+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f81465a0 con 0x7f76f815e9f0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.121+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f8146de0 con 0x7f76f815e9f0 2024-04-17T07:26:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.122+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f80755d0 con 0x7f76f815e9f0 2024-04-17T07:26:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.123+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7f76e0018020 con 0x7f76f815e9f0 2024-04-17T07:26:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.125+0000 7f76eea00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 0x7f76d406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.125+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6455+0+0 (secure 0 0 0) 0x7f76e008c140 con 0x7f76f815e9f0 2024-04-17T07:26:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.125+0000 7f76f6800640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 msgr2=0x7f76d406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.125+0000 7f76f6800640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 0x7f76d406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:56.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.130+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f76e001f020 con 0x7f76f815e9f0 2024-04-17T07:26:56.296 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.296+0000 7fd1d043e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:26:56.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.326+0000 7f76f6800640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 msgr2=0x7f76d406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:56.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.326+0000 7f76f6800640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 0x7f76d406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.354+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76f80b1fb0 con 0x7f76f815e9f0 2024-04-17T07:26:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.356+0000 7f76eea00640 1 -- 172.21.15.67:0/2129713490 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+17767 (secure 0 0 0) 0x7f76f81465a0 con 0x7f76f815e9f0 2024-04-17T07:26:56.356 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:56.356 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:26:54.282511+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2778463946":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/3485602680":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6833/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6832/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/618455126":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/2322153279":"2024-04-18T07:15:31.130810+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1},{"begin":8,"length":2},{"begin":12,"length":1},{"begin":15,"length":2},{"begin":21,"length":1},{"begin":23,"length":6},{"begin":30,"length":1},{"begin":32,"length":2},{"begin":36,"length":5},{"begin":43,"length":1},{"begin":45,"length":5},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":57,"length":4},{"begin":62,"length":3},{"begin":68,"length":6},{"begin":75,"length":1},{"begin":77,"length":2},{"begin":80,"length":2},{"begin":84,"length":4},{"begin":89,"length":7},{"begin":97,"length":2},{"begin":100,"length":1},{"begin":103,"length":4},{"begin":108,"length":4},{"begin":144,"length":78}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":110,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:26:56.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 msgr2=0x7f76d406ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:56.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 0x7f76d406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 msgr2=0x7f76f80c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f76e000b3e0 tx=0x7f76e000b8a0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 shutdown_connections 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7f76d406caa0 0x7f76d406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76f815edd0 0x7f76f80c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76f815e9f0 0x7f76f80c8e10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 --2- 172.21.15.67:0/2129713490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76f80cd1b0 0x7f76f8160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 >> 172.21.15.67:0/2129713490 conn(0x7f76f80cd710 msgr2=0x7f76f80c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 shutdown_connections 2024-04-17T07:26:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.358+0000 7f76fca00640 1 -- 172.21.15.67:0/2129713490 wait complete. 2024-04-17T07:26:56.377 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:15:09.343098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:15:10.202353+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:26:56.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:26:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.588+0000 7fba36e00640 1 Processor -- start 2024-04-17T07:26:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.588+0000 7fba36e00640 1 -- start start 2024-04-17T07:26:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.589+0000 7fba36e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba300d16f0 0x7fba300d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba36e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba300d34a0 con 0x7fba3015ee80 2024-04-17T07:26:56.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba36e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba300d3780 con 0x7fba300b8370 2024-04-17T07:26:56.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba36e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fba300b7c20 con 0x7fba300d16f0 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba300d16f0 0x7fba300d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba300d16f0 0x7fba300d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47052/0 (socket says 172.21.15.67:47052) 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba27e00640 1 -- 172.21.15.67:0/3191267949 learned_addr learned my addr 172.21.15.67:0/3191267949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.590+0000 7fba2fe00640 1 --1- 172.21.15.67:0/3191267949 >> v1:172.21.15.67:6789/0 conn(0x7fba3015ee80 0x7fba300d2a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35128/0 (socket says 172.21.15.67:35128) 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.591+0000 7fba27e00640 1 -- 172.21.15.67:0/3191267949 >> v1:172.21.15.187:6789/0 conn(0x7fba300b8370 legacy=0x7fba300b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.591+0000 7fba27e00640 1 -- 172.21.15.67:0/3191267949 >> v1:172.21.15.67:6789/0 conn(0x7fba3015ee80 legacy=0x7fba300d2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.591+0000 7fba27e00640 1 -- 172.21.15.67:0/3191267949 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba300d3a60 con 0x7fba300d16f0 2024-04-17T07:26:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.592+0000 7fba27e00640 1 --2- 172.21.15.67:0/3191267949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300d16f0 0x7fba300d1b50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba18009870 tx=0x7fba1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772a9f22c1c2ab99 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:56.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.593+0000 7fba2f400640 1 -- 172.21.15.67:0/3191267949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18034070 con 0x7fba300d16f0 2024-04-17T07:26:56.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.593+0000 7fba2f400640 1 -- 172.21.15.67:0/3191267949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba18030860 con 0x7fba300d16f0 2024-04-17T07:26:56.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.593+0000 7fba2f400640 1 -- 172.21.15.67:0/3191267949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18030c30 con 0x7fba300d16f0 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.593+0000 7fba36e00640 1 -- 172.21.15.67:0/3191267949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300d16f0 msgr2=0x7fba300d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.593+0000 7fba36e00640 1 --2- 172.21.15.67:0/3191267949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300d16f0 0x7fba300d1b50 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba18009870 tx=0x7fba1802b9a0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.594+0000 7fba36e00640 1 -- 172.21.15.67:0/3191267949 shutdown_connections 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.594+0000 7fba36e00640 1 --2- 172.21.15.67:0/3191267949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300d16f0 0x7fba300d1b50 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.594+0000 7fba36e00640 1 -- 172.21.15.67:0/3191267949 >> 172.21.15.67:0/3191267949 conn(0x7fba300677e0 msgr2=0x7fba30067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.594+0000 7fba36e00640 1 -- 172.21.15.67:0/3191267949 shutdown_connections 2024-04-17T07:26:56.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.594+0000 7fba36e00640 1 -- 172.21.15.67:0/3191267949 wait complete. 2024-04-17T07:26:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.596+0000 7fba36e00640 1 Processor -- start 2024-04-17T07:26:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.596+0000 7fba36e00640 1 -- start start 2024-04-17T07:26:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba300b8370 0x7fba3014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba300d16f0 0x7fba301e8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba300d3d40 con 0x7fba3015ee80 2024-04-17T07:26:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba300b7ee0 con 0x7fba300d16f0 2024-04-17T07:26:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba36e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba300d42c0 con 0x7fba300b8370 2024-04-17T07:26:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba34e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba34e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47062/0 (socket says 172.21.15.67:47062) 2024-04-17T07:26:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.597+0000 7fba34e00640 1 -- 172.21.15.67:0/4184445926 learned_addr learned my addr 172.21.15.67:0/4184445926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba2fe00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba300b8370 0x7fba3014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba27e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba300d16f0 0x7fba301e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba300b8370 msgr2=0x7fba3014d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba300b8370 0x7fba3014d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba300d16f0 msgr2=0x7fba301e8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba300d16f0 0x7fba301e8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 -- 172.21.15.67:0/4184445926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba301e9120 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba34e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c6b0 tx=0x7fba2800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28003b70 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.598+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba28003d30 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.599+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28005310 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.599+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba301e9400 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.599+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba301e99c0 con 0x7fba3015ee80 2024-04-17T07:26:56.600 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.600+0000 7fd1d043e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:26:56.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.600+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba30068b70 con 0x7fba3015ee80 2024-04-17T07:26:56.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.601+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) ==== 91211+0+0 (secure 0 0 0) 0x7fba280054b0 con 0x7fba3015ee80 2024-04-17T07:26:56.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.602+0000 7fba2cc00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 0x7fba0c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:56.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.602+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6455+0+0 (secure 0 0 0) 0x7fba2808bfe0 con 0x7fba3015ee80 2024-04-17T07:26:56.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.602+0000 7fba2fe00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 msgr2=0x7fba0c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:56.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.602+0000 7fba2fe00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 0x7fba0c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:26:56.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.607+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba28059ed0 con 0x7fba3015ee80 2024-04-17T07:26:56.611 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:56.612+0000 7efe13046040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:26:56.657 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.657+0000 7fd1d043e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:26:56.787 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.787+0000 7fd1d043e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:26:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.804+0000 7fba2fe00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 msgr2=0x7fba0c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2964624313 2024-04-17T07:26:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.804+0000 7fba2fe00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 0x7fba0c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:26:56.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.808+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 15) ==== 90751+0+0 (secure 0 0 0) 0x7fba28055e40 con 0x7fba3015ee80 2024-04-17T07:26:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.808+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 msgr2=0x7fba0c06ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:26:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.808+0000 7fba2cc00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 0x7fba0c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.810+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fba301e9e50 con 0x7fba3015ee80 2024-04-17T07:26:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.814+0000 7fba2cc00640 1 -- 172.21.15.67:0/4184445926 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) ==== 93+0+10 (secure 0 0 0) 0x7fba28078690 con 0x7fba3015ee80 2024-04-17T07:26:56.814 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:26:56.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 msgr2=0x7fba301ea1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:56.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c6b0 tx=0x7fba2800cb70 comp rx=0 tx=0).stop 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 shutdown_connections 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:6832/2964624313,v1:172.21.15.187:6833/2964624313] conn(0x7fba0c06caa0 0x7fba0c06ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3015ee80 0x7fba301ea1c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba300d16f0 0x7fba301e8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 --2- 172.21.15.67:0/4184445926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba300b8370 0x7fba3014d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 >> 172.21.15.67:0/4184445926 conn(0x7fba300677e0 msgr2=0x7fba300cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.817+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 shutdown_connections 2024-04-17T07:26:56.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:56.818+0000 7fba36e00640 1 -- 172.21.15.67:0/4184445926 wait complete. 2024-04-17T07:26:56.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:26:56.881 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.881+0000 7fd14a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.881 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.881+0000 7fd14a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.881 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.881+0000 7fd14a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.881+0000 7fd14a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.881+0000 7fd14a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.882+0000 7fd14de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.882+0000 7fd14de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.882+0000 7fd14de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.882+0000 7fd14de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.882 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:26:56.882+0000 7fd14de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:26:56.953 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:56.953+0000 7efe13046040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:26:57.015 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:57.015+0000 7efe13046040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:26:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.037+0000 7f498a000640 1 Processor -- start 2024-04-17T07:26:57.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.038+0000 7f498a000640 1 -- start start 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f498a000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49840b46f0 0x7f49840b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f498a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49840b51d0 con 0x7f49840d4510 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f498a000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49840b54b0 con 0x7f498415e9b0 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f498a000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f49840b5790 con 0x7f49840b46f0 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f4989600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49840d4510 0x7f49840d4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35136/0 (socket says 172.21.15.67:35136) 2024-04-17T07:26:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.039+0000 7f4989600640 1 -- 172.21.15.67:0/1868719165 learned_addr learned my addr 172.21.15.67:0/1868719165 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983e00640 1 --2- 172.21.15.67:0/1868719165 >> v2:172.21.15.187:3300/0 conn(0x7f49840b46f0 0x7f49840b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710764220 0 0) 0x7f49840b51d0 con 0x7f49840d4510 2024-04-17T07:26:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f496c003520 con 0x7f49840d4510 2024-04-17T07:26:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 648127640 0 0) 0x7f496c003520 con 0x7f49840d4510 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 >> v2:172.21.15.187:3300/0 conn(0x7f49840b46f0 msgr2=0x7f49840b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 --2- 172.21.15.67:0/1868719165 >> v2:172.21.15.187:3300/0 conn(0x7f49840b46f0 0x7f49840b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 >> v1:172.21.15.187:6789/0 conn(0x7f498415e9b0 legacy=0x7f49840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49840d24c0 con 0x7f49840d4510 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f497c0019f0 con 0x7f49840d4510 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 >> v1:172.21.15.67:6789/0 conn(0x7f49840d4510 legacy=0x7f49840d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.040+0000 7f4983400640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f496c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4983400640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f496c004800 0x7f496c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4983400640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f496c005210 0x7f496c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49840b5790 con 0x7f496c0039b0 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f496c00c140 con 0x7f496c005210 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f496c00c340 con 0x7f496c004800 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f496c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 -- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f496c004800 msgr2=0x7f496c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f496c004800 0x7f496c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 -- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f496c005210 msgr2=0x7f496c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f496c005210 0x7f496c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4988c00640 1 -- 172.21.15.67:0/1868719165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c003520 con 0x7f496c0039b0 2024-04-17T07:26:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.041+0000 7f4989600640 1 -- 172.21.15.67:0/1868719165 reap_dead start 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.042+0000 7f4988c00640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f496c003df0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4978001750 tx=0x7f496c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41861f9d92f405d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.043+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978001370 con 0x7f496c0039b0 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.043+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49780052d0 con 0x7f496c0039b0 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.043+0000 7f4983400640 1 -- 172.21.15.67:0/1868719165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978003830 con 0x7f496c0039b0 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.043+0000 7f498a000640 1 -- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 msgr2=0x7f496c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.043+0000 7f498a000640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f496c003df0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4978001750 tx=0x7f496c005830 comp rx=0 tx=0).stop 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.044+0000 7f498a000640 1 -- 172.21.15.67:0/1868719165 shutdown_connections 2024-04-17T07:26:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.044+0000 7f498a000640 1 --2- 172.21.15.67:0/1868719165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f496c003df0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.044+0000 7f498a000640 1 -- 172.21.15.67:0/1868719165 >> 172.21.15.67:0/1868719165 conn(0x7f498409daa0 msgr2=0x7f498408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.044+0000 7f498a000640 1 -- 172.21.15.67:0/1868719165 shutdown_connections 2024-04-17T07:26:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.044+0000 7f498a000640 1 -- 172.21.15.67:0/1868719165 wait complete. 2024-04-17T07:26:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.045+0000 7f498a000640 1 Processor -- start 2024-04-17T07:26:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.046+0000 7f498a000640 1 -- start start 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.046+0000 7f498a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f498a000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 0x7f498414a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f498a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840c8e40 0x7f49840c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f498a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f498415ed90 con 0x7f496c0039b0 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f498a000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49840d2830 con 0x7f498414a540 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f498a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f498415ef10 con 0x7f49840c8e40 2024-04-17T07:26:57.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47074/0 (socket says 172.21.15.67:47074) 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 -- 172.21.15.67:0/490112739 learned_addr learned my addr 172.21.15.67:0/490112739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4983e00640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 0x7f498414a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4989600640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840c8e40 0x7f49840c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 -- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840c8e40 msgr2=0x7f49840c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840c8e40 0x7f49840c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 -- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 msgr2=0x7f498414a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 0x7f498414a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4988c00640 1 -- 172.21.15.67:0/490112739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49840c83a0 con 0x7f496c0039b0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.047+0000 7f4983e00640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 0x7f498414a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f4988c00640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4978005040 tx=0x7f4978001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978041020 con 0x7f496c0039b0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4978005480 con 0x7f496c0039b0 2024-04-17T07:26:57.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978001370 con 0x7f496c0039b0 2024-04-17T07:26:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4984160bb0 con 0x7f496c0039b0 2024-04-17T07:26:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.048+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49840c8920 con 0x7f496c0039b0 2024-04-17T07:26:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.049+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49840755d0 con 0x7f496c0039b0 2024-04-17T07:26:57.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.050+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) ==== 90751+0+0 (secure 0 0 0) 0x7f4978044020 con 0x7f496c0039b0 2024-04-17T07:26:57.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.051+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f49780ad330 con 0x7f496c0039b0 2024-04-17T07:26:57.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.061+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f497804b030 con 0x7f496c0039b0 2024-04-17T07:26:57.162 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:26:57.162+0000 7efe13046040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:26:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.266+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f49840b3900 con 0x7f496c0039b0 2024-04-17T07:26:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.267+0000 7f4980c00640 1 -- 172.21.15.67:0/490112739 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) ==== 104+0+11 (secure 0 0 0) 0x7f49780303c0 con 0x7f496c0039b0 2024-04-17T07:26:57.267 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 msgr2=0x7f49840b7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4978005040 tx=0x7f4978001120 comp rx=0 tx=0).stop 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 shutdown_connections 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840c8e40 0x7f49840c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f498414a540 0x7f498414a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 --2- 172.21.15.67:0/490112739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f496c0039b0 0x7f49840b7450 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 >> 172.21.15.67:0/490112739 conn(0x7f498409daa0 msgr2=0x7f498406ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 shutdown_connections 2024-04-17T07:26:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.270+0000 7f498a000640 1 -- 172.21.15.67:0/490112739 wait complete. 2024-04-17T07:26:57.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:26:57.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:26:57.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:26:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.500+0000 7f3626000640 1 Processor -- start 2024-04-17T07:26:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.500+0000 7f3626000640 1 -- start start 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3620145b00 0x7f3620160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36200b4ac0 0x7f36200b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f36200b5470 con 0x7f3620145b00 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3620160920 con 0x7f36200b4ac0 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3626000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3620160c00 con 0x7f3620146060 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3624c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3620145b00 0x7f3620160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3624c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3620145b00 0x7f3620160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47086/0 (socket says 172.21.15.67:47086) 2024-04-17T07:26:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3624c00640 1 -- 172.21.15.67:0/2773017404 learned_addr learned my addr 172.21.15.67:0/2773017404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3617e00640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.502+0000 7f3625600640 1 --2- 172.21.15.67:0/2773017404 >> v2:172.21.15.187:3300/0 conn(0x7f36200b4ac0 0x7f36200b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.503+0000 7f3624c00640 1 -- 172.21.15.67:0/2773017404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 msgr2=0x7f362015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.503+0000 7f3624c00640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.503+0000 7f3624c00640 1 -- 172.21.15.67:0/2773017404 >> v2:172.21.15.187:3300/0 conn(0x7f36200b4ac0 msgr2=0x7f36200b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.503+0000 7f3624c00640 1 --2- 172.21.15.67:0/2773017404 >> v2:172.21.15.187:3300/0 conn(0x7f36200b4ac0 0x7f36200b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.503+0000 7f3624c00640 1 -- 172.21.15.67:0/2773017404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3620160ee0 con 0x7f3620145b00 2024-04-17T07:26:57.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.504+0000 7f3624c00640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f3620160340 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f36200b6190 tx=0x7f361002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790386a448ecdd6d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.504+0000 7f361fe00640 1 -- 172.21.15.67:0/2773017404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361002c9e0 con 0x7f3620145b00 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.504+0000 7f361fe00640 1 -- 172.21.15.67:0/2773017404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f361002cb80 con 0x7f3620145b00 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.504+0000 7f361fe00640 1 -- 172.21.15.67:0/2773017404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36100343f0 con 0x7f3620145b00 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 -- 172.21.15.67:0/2773017404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 msgr2=0x7f3620160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f3620160340 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f36200b6190 tx=0x7f361002be40 comp rx=0 tx=0).stop 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 -- 172.21.15.67:0/2773017404 shutdown_connections 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 --2- 172.21.15.67:0/2773017404 >> v2:172.21.15.187:3300/0 conn(0x7f36200b4ac0 0x7f36200b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 --2- 172.21.15.67:0/2773017404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f3620160340 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 -- 172.21.15.67:0/2773017404 >> 172.21.15.67:0/2773017404 conn(0x7f36200b23c0 msgr2=0x7f36200b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 -- 172.21.15.67:0/2773017404 shutdown_connections 2024-04-17T07:26:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.505+0000 7f3626000640 1 -- 172.21.15.67:0/2773017404 wait complete. 2024-04-17T07:26:57.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.507+0000 7f3626000640 1 Processor -- start 2024-04-17T07:26:57.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.507+0000 7f3626000640 1 -- start start 2024-04-17T07:26:57.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36200b4ac0 0x7f3620144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f36200cfba0 con 0x7f3620145b00 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f36200b3760 con 0x7f36200b4ac0 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3626000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f362015d2c0 con 0x7f3620146060 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3617e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3617e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47092/0 (socket says 172.21.15.67:47092) 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3617e00640 1 -- 172.21.15.67:0/3677817799 learned_addr learned my addr 172.21.15.67:0/3677817799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3624c00640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36200b4ac0 0x7f3620144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.508+0000 7f3625600640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 -- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 msgr2=0x7f362015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 -- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36200b4ac0 msgr2=0x7f3620144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36200b4ac0 0x7f3620144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 -- 172.21.15.67:0/3677817799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36201f4940 con 0x7f3620145b00 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3625600640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3617e00640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f360800d5f0 tx=0x7f360800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36080125f0 con 0x7f3620145b00 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3608019070 con 0x7f3620145b00 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36080129d0 con 0x7f3620145b00 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362015c400 con 0x7f3620145b00 2024-04-17T07:26:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.509+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3620141a10 con 0x7f3620145b00 2024-04-17T07:26:57.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.511+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36200706f0 con 0x7f3620145b00 2024-04-17T07:26:57.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.512+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) ==== 90751+0+0 (secure 0 0 0) 0x7f3608012b70 con 0x7f3620145b00 2024-04-17T07:26:57.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.513+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f360808bd90 con 0x7f3620145b00 2024-04-17T07:26:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.517+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f36200706f0 con 0x7f3620145b00 2024-04-17T07:26:57.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.720+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3620160340 con 0x7f3620145b00 2024-04-17T07:26:57.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.720+0000 7f361d600640 1 -- 172.21.15.67:0/3677817799 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) ==== 100+0+11 (secure 0 0 0) 0x7f3620160340 con 0x7f3620145b00 2024-04-17T07:26:57.721 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:26:57.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 msgr2=0x7f362015bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f360800d5f0 tx=0x7f360800dab0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 shutdown_connections 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3620146060 0x7f362015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3620145b00 0x7f362015bc80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 --2- 172.21.15.67:0/3677817799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36200b4ac0 0x7f3620144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.723+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 >> 172.21.15.67:0/3677817799 conn(0x7f36200b23c0 msgr2=0x7f3620074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.724+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 shutdown_connections 2024-04-17T07:26:57.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.724+0000 7f3626000640 1 -- 172.21.15.67:0/3677817799 wait complete. 2024-04-17T07:26:57.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:26:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.955+0000 7f08de400640 1 Processor -- start 2024-04-17T07:26:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.955+0000 7f08de400640 1 -- start start 2024-04-17T07:26:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08de400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08de400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f08d80af980 0x7f08d80ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08de400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08d815f440 con 0x7f08d815c5a0 2024-04-17T07:26:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08de400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f08d80b4920 con 0x7f08d80af980 2024-04-17T07:26:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08de400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08d815f720 con 0x7f08d815ee60 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.957+0000 7f08d7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:55420/0 (socket says 172.21.15.67:55420) 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 -- 172.21.15.67:0/2095761397 learned_addr learned my addr 172.21.15.67:0/2095761397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08dda00640 1 --2- 172.21.15.67:0/2095761397 >> v2:172.21.15.187:3300/0 conn(0x7f08d80af980 0x7f08d80ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 -- 172.21.15.67:0/2095761397 >> v2:172.21.15.187:3300/0 conn(0x7f08d80af980 msgr2=0x7f08d80ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 --2- 172.21.15.67:0/2095761397 >> v2:172.21.15.187:3300/0 conn(0x7f08d80af980 0x7f08d80ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 -- 172.21.15.67:0/2095761397 >> v1:172.21.15.187:6789/0 conn(0x7f08d815c5a0 legacy=0x7f08d80b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.958+0000 7f08d7e00640 1 -- 172.21.15.67:0/2095761397 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d815fa00 con 0x7f08d815ee60 2024-04-17T07:26:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.960+0000 7f08d7e00640 1 --2- 172.21.15.67:0/2095761397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009870 tx=0x7f08c402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b4bf3e721c5099b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.960+0000 7f08d7400640 1 -- 172.21.15.67:0/2095761397 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40317c0 con 0x7f08d815ee60 2024-04-17T07:26:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.960+0000 7f08d7400640 1 -- 172.21.15.67:0/2095761397 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c4037070 con 0x7f08d815ee60 2024-04-17T07:26:57.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.960+0000 7f08d7400640 1 -- 172.21.15.67:0/2095761397 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4031b80 con 0x7f08d815ee60 2024-04-17T07:26:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 -- 172.21.15.67:0/2095761397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 msgr2=0x7f08d80af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 --2- 172.21.15.67:0/2095761397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009870 tx=0x7f08c402b860 comp rx=0 tx=0).stop 2024-04-17T07:26:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 -- 172.21.15.67:0/2095761397 shutdown_connections 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 --2- 172.21.15.67:0/2095761397 >> v2:172.21.15.187:3300/0 conn(0x7f08d80af980 0x7f08d80ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 --2- 172.21.15.67:0/2095761397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d815ee60 0x7f08d80af440 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 -- 172.21.15.67:0/2095761397 >> 172.21.15.67:0/2095761397 conn(0x7f08d8067d10 msgr2=0x7f08d80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 -- 172.21.15.67:0/2095761397 shutdown_connections 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.961+0000 7f08de400640 1 -- 172.21.15.67:0/2095761397 wait complete. 2024-04-17T07:26:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.963+0000 7f08de400640 1 Processor -- start 2024-04-17T07:26:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.963+0000 7f08de400640 1 -- start start 2024-04-17T07:26:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d80af980 0x7f08d8154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 0x7f08d81555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08d80adc30 con 0x7f08d815c5a0 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f08d80adab0 con 0x7f08d815ee60 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08de400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08d806e670 con 0x7f08d80af980 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47098/0 (socket says 172.21.15.67:47098) 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.964+0000 7f08d7e00640 1 -- 172.21.15.67:0/3064643980 learned_addr learned my addr 172.21.15.67:0/3064643980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08dd000640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d80af980 0x7f08d8154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08dda00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 0x7f08d81555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 -- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d80af980 msgr2=0x7f08d8154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d80af980 0x7f08d8154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 -- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 msgr2=0x7f08d81555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:57.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 0x7f08d81555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 -- 172.21.15.67:0/3064643980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d8145fd0 con 0x7f08d815c5a0 2024-04-17T07:26:57.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08dda00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 0x7f08d81555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d7e00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08c402b8a0 tx=0x7f08c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4032650 con 0x7f08d815c5a0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c4032830 con 0x7f08d815c5a0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4032c20 con 0x7f08d815c5a0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.965+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d81462b0 con 0x7f08d815c5a0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.966+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d81535d0 con 0x7f08d815c5a0 2024-04-17T07:26:57.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.967+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d8153890 con 0x7f08d815c5a0 2024-04-17T07:26:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.968+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 90878+0+0 (secure 0 0 0) 0x7f08c404c060 con 0x7f08d815c5a0 2024-04-17T07:26:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.968+0000 7f08d4c00640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 0x7f08b806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:57.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.969+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f08c40ad280 con 0x7f08d815c5a0 2024-04-17T07:26:57.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.969+0000 7f08dd000640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 0x7f08b806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:57.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.970+0000 7f08dd000640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 0x7f08b806ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f08cc007fd0 tx=0x7f08cc009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:57.973+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f08c4040070 con 0x7f08d815c5a0 2024-04-17T07:26:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.205+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08d8068a10 con 0x7f08d815c5a0 2024-04-17T07:26:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.207+0000 7f08d4c00640 1 -- 172.21.15.67:0/3064643980 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7f08c407b4e0 con 0x7f08d815c5a0 2024-04-17T07:26:58.207 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:58.207 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:26:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 msgr2=0x7f08b806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 0x7f08b806ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f08cc007fd0 tx=0x7f08cc009040 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 msgr2=0x7f08d8153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08c402b8a0 tx=0x7f08c40046c0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 shutdown_connections 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f08b806c740 0x7f08b806ec00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08d815ee60 0x7f08d81555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08d815c5a0 0x7f08d8153090 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 --2- 172.21.15.67:0/3064643980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08d80af980 0x7f08d8154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.209+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 >> 172.21.15.67:0/3064643980 conn(0x7f08d8067d10 msgr2=0x7f08d806edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.210+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 shutdown_connections 2024-04-17T07:26:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.210+0000 7f08de400640 1 -- 172.21.15.67:0/3064643980 wait complete. 2024-04-17T07:26:58.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:26:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.440+0000 7fe5fe200640 1 Processor -- start 2024-04-17T07:26:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.440+0000 7fe5fe200640 1 -- start start 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.442+0000 7fe5fe200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5f8160280 0x7fe5f806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.442+0000 7fe5fe200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5f8141ed0 0x7fe5f8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.442+0000 7fe5fe200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f80cd4a0 con 0x7fe5f806b8c0 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.442+0000 7fe5fe200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f815ce30 con 0x7fe5f8160280 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.442+0000 7fe5fe200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f81607f0 con 0x7fe5f8141ed0 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5fce00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe5f806b8c0 0x7fe5f80cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35148/0 (socket says 172.21.15.67:35148) 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5fce00640 1 -- 172.21.15.67:0/2325275855 learned_addr learned my addr 172.21.15.67:0/2325275855 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7e00640 1 --2- 172.21.15.67:0/2325275855 >> v2:172.21.15.67:3300/0 conn(0x7fe5f8160280 0x7fe5f806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5fd800640 1 --2- 172.21.15.67:0/2325275855 >> v2:172.21.15.187:3300/0 conn(0x7fe5f8141ed0 0x7fe5f8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7400640 1 -- 172.21.15.67:0/2325275855 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412943597 0 0) 0x7fe5f80cd4a0 con 0x7fe5f806b8c0 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7400640 1 -- 172.21.15.67:0/2325275855 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e0003520 con 0x7fe5f806b8c0 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7e00640 1 -- 172.21.15.67:0/2325275855 >> v2:172.21.15.187:3300/0 conn(0x7fe5f8141ed0 msgr2=0x7fe5f8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7e00640 1 --2- 172.21.15.67:0/2325275855 >> v2:172.21.15.187:3300/0 conn(0x7fe5f8141ed0 0x7fe5f8142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7e00640 1 -- 172.21.15.67:0/2325275855 >> v1:172.21.15.67:6789/0 conn(0x7fe5f806b8c0 legacy=0x7fe5f80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.443+0000 7fe5f7e00640 1 -- 172.21.15.67:0/2325275855 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f8160ad0 con 0x7fe5f8160280 2024-04-17T07:26:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.445+0000 7fe5f7e00640 1 --2- 172.21.15.67:0/2325275855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f806bca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80b2120 tx=0x7fe5e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b840e411f779ed64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.445+0000 7fe5f7400640 1 -- 172.21.15.67:0/2325275855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8035020 con 0x7fe5f8160280 2024-04-17T07:26:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.445+0000 7fe5f7400640 1 -- 172.21.15.67:0/2325275855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e8031830 con 0x7fe5f8160280 2024-04-17T07:26:58.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.445+0000 7fe5f7400640 1 -- 172.21.15.67:0/2325275855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8031be0 con 0x7fe5f8160280 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 -- 172.21.15.67:0/2325275855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 msgr2=0x7fe5f806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2325275855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f806bca0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80b2120 tx=0x7fe5e802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 -- 172.21.15.67:0/2325275855 shutdown_connections 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2325275855 >> v2:172.21.15.187:3300/0 conn(0x7fe5f8141ed0 0x7fe5f8142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2325275855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f806bca0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 -- 172.21.15.67:0/2325275855 >> 172.21.15.67:0/2325275855 conn(0x7fe5f8067d10 msgr2=0x7fe5f80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 -- 172.21.15.67:0/2325275855 shutdown_connections 2024-04-17T07:26:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.446+0000 7fe5fe200640 1 -- 172.21.15.67:0/2325275855 wait complete. 2024-04-17T07:26:58.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.448+0000 7fe5fe200640 1 Processor -- start 2024-04-17T07:26:58.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.448+0000 7fe5fe200640 1 -- start start 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 0x7fe5f813ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 0x7fe5f80b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f8161090 con 0x7fe5f8160280 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f806c1e0 con 0x7fe5f806b8c0 2024-04-17T07:26:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fe200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f813f9b0 con 0x7fe5f8141ed0 2024-04-17T07:26:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47128/0 (socket says 172.21.15.67:47128) 2024-04-17T07:26:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.449+0000 7fe5fd800640 1 -- 172.21.15.67:0/2214338795 learned_addr learned my addr 172.21.15.67:0/2214338795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fce00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 0x7fe5f813ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5f7e00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 0x7fe5f80b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 -- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 msgr2=0x7fe5f80b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 0x7fe5f80b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 -- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 msgr2=0x7fe5f813ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 0x7fe5f813ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 -- 172.21.15.67:0/2214338795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f8158be0 con 0x7fe5f8160280 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5f7e00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 0x7fe5f80b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fce00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 0x7fe5f813ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.450+0000 7fe5fd800640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0007e70 tx=0x7fe5f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.451+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0014620 con 0x7fe5f8160280 2024-04-17T07:26:58.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.451+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f0019070 con 0x7fe5f8160280 2024-04-17T07:26:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.451+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0014a20 con 0x7fe5f8160280 2024-04-17T07:26:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.451+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f8143160 con 0x7fe5f8160280 2024-04-17T07:26:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.451+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f8143720 con 0x7fe5f8160280 2024-04-17T07:26:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.452+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f815ca70 con 0x7fe5f8160280 2024-04-17T07:26:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.453+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 90878+0+0 (secure 0 0 0) 0x7fe5f0014bc0 con 0x7fe5f8160280 2024-04-17T07:26:58.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.454+0000 7fe5f4c00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 0x7fe5dc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.454+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7fe5f008c520 con 0x7fe5f8160280 2024-04-17T07:26:58.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.455+0000 7fe5fce00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 0x7fe5dc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.456+0000 7fe5fce00640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 0x7fe5dc06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec005880 tx=0x7fe5ec009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.459+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe5f815ca70 con 0x7fe5f8160280 2024-04-17T07:26:58.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.667+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5f80b2260 con 0x7fe5f8160280 2024-04-17T07:26:58.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.669+0000 7fe5f4c00640 1 -- 172.21.15.67:0/2214338795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+18268 (secure 0 0 0) 0x7fe5f005a850 con 0x7fe5f8160280 2024-04-17T07:26:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:58.670 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:26:56.792802+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2322153279":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1982372125":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1699979613":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6833/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/618455126":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/3485602680":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/2778463946":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1528295426":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6833/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2513891628":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/14109251":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1536403455":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2851019947":"2024-04-18T07:26:56.792716+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1},{"begin":8,"length":2},{"begin":12,"length":1},{"begin":15,"length":2},{"begin":21,"length":1},{"begin":23,"length":6},{"begin":30,"length":1},{"begin":32,"length":2},{"begin":36,"length":5},{"begin":43,"length":1},{"begin":45,"length":5},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":57,"length":4},{"begin":62,"length":3},{"begin":68,"length":6},{"begin":75,"length":1},{"begin":77,"length":2},{"begin":80,"length":2},{"begin":84,"length":4},{"begin":89,"length":7},{"begin":97,"length":2},{"begin":100,"length":1},{"begin":103,"length":4},{"begin":108,"length":4},{"begin":144,"length":78}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:26:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 msgr2=0x7fe5dc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 0x7fe5dc06ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec005880 tx=0x7fe5ec009040 comp rx=0 tx=0).stop 2024-04-17T07:26:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 msgr2=0x7fe5f80b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0007e70 tx=0x7fe5f000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 shutdown_connections 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe5dc06c740 0x7fe5dc06ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5f8160280 0x7fe5f80b47d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5f8141ed0 0x7fe5f80b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 --2- 172.21.15.67:0/2214338795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5f806b8c0 0x7fe5f813ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.672+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 >> 172.21.15.67:0/2214338795 conn(0x7fe5f8067d10 msgr2=0x7fe5f8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.673+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 shutdown_connections 2024-04-17T07:26:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.673+0000 7fe5fe200640 1 -- 172.21.15.67:0/2214338795 wait complete. 2024-04-17T07:26:58.692 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:26:58.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:26:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.915+0000 7fb555a00640 1 Processor -- start 2024-04-17T07:26:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.915+0000 7fb555a00640 1 -- start start 2024-04-17T07:26:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.917+0000 7fb555a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 0x7fb55006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.917+0000 7fb555a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 0x7fb55015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.917+0000 7fb555a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb55015bc80 con 0x7fb55015b570 2024-04-17T07:26:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.917+0000 7fb555a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb55015bf40 con 0x7fb55006be70 2024-04-17T07:26:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.917+0000 7fb555a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb55015c220 con 0x7fb55006feb0 2024-04-17T07:26:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.918+0000 7fb54fe00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb55015b570 0x7fb55006f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:59776/0 (socket says 172.21.15.67:59776) 2024-04-17T07:26:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.918+0000 7fb54fe00640 1 -- 172.21.15.67:0/2637617927 learned_addr learned my addr 172.21.15.67:0/2637617927 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.918+0000 7fb54f400640 1 --2- 172.21.15.67:0/2637617927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 0x7fb55006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.918+0000 7fb555000640 1 --2- 172.21.15.67:0/2637617927 >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 0x7fb55015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54ea00640 1 -- 172.21.15.67:0/2637617927 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703807815 0 0) 0x7fb55015bc80 con 0x7fb55015b570 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54ea00640 1 -- 172.21.15.67:0/2637617927 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb534005730 con 0x7fb55015b570 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54f400640 1 -- 172.21.15.67:0/2637617927 >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 msgr2=0x7fb55015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54f400640 1 --2- 172.21.15.67:0/2637617927 >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 0x7fb55015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54f400640 1 -- 172.21.15.67:0/2637617927 >> v1:172.21.15.187:6789/0 conn(0x7fb55015b570 legacy=0x7fb55006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb54f400640 1 -- 172.21.15.67:0/2637617927 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55015c500 con 0x7fb55006feb0 2024-04-17T07:26:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.919+0000 7fb555000640 1 --2- 172.21.15.67:0/2637617927 >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 0x7fb55015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:26:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.920+0000 7fb54f400640 1 --2- 172.21.15.67:0/2637617927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 0x7fb55006b930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0097a0 tx=0x7fb53c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c87ef151ea3215da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.921+0000 7fb54ea00640 1 -- 172.21.15.67:0/2637617927 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c031780 con 0x7fb55006feb0 2024-04-17T07:26:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.921+0000 7fb54ea00640 1 -- 172.21.15.67:0/2637617927 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53c037070 con 0x7fb55006feb0 2024-04-17T07:26:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.921+0000 7fb54ea00640 1 -- 172.21.15.67:0/2637617927 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c031b40 con 0x7fb55006feb0 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.921+0000 7fb555a00640 1 -- 172.21.15.67:0/2637617927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 msgr2=0x7fb55006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.921+0000 7fb555a00640 1 --2- 172.21.15.67:0/2637617927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 0x7fb55006b930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0097a0 tx=0x7fb53c02b820 comp rx=0 tx=0).stop 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 -- 172.21.15.67:0/2637617927 shutdown_connections 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 --2- 172.21.15.67:0/2637617927 >> v2:172.21.15.187:3300/0 conn(0x7fb55006be70 0x7fb55015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 --2- 172.21.15.67:0/2637617927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006feb0 0x7fb55006b930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 -- 172.21.15.67:0/2637617927 >> 172.21.15.67:0/2637617927 conn(0x7fb550067d10 msgr2=0x7fb5500b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 -- 172.21.15.67:0/2637617927 shutdown_connections 2024-04-17T07:26:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.922+0000 7fb555a00640 1 -- 172.21.15.67:0/2637617927 wait complete. 2024-04-17T07:26:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.923+0000 7fb555a00640 1 Processor -- start 2024-04-17T07:26:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.924+0000 7fb555a00640 1 -- start start 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006be70 0x7fb5501f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55006feb0 0x7fb5501f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb55006c460 con 0x7fb55015b570 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb55006c2e0 con 0x7fb55006feb0 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5500b47b0 con 0x7fb55006be70 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006be70 0x7fb5501f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47144/0 (socket says 172.21.15.67:47144) 2024-04-17T07:26:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 -- 172.21.15.67:0/577658881 learned_addr learned my addr 172.21.15.67:0/577658881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb54f400640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55006feb0 0x7fb5501f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 -- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006be70 msgr2=0x7fb5501f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006be70 0x7fb5501f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 -- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55006feb0 msgr2=0x7fb5501f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55006feb0 0x7fb5501f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.925+0000 7fb555000640 1 -- 172.21.15.67:0/577658881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5501f7c70 con 0x7fb55015b570 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb555000640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb54800efc0 tx=0x7fb548009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548018030 con 0x7fb55015b570 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54800cde0 con 0x7fb55015b570 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548003bb0 con 0x7fb55015b570 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5501e3e10 con 0x7fb55015b570 2024-04-17T07:26:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.926+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5501e43d0 con 0x7fb55015b570 2024-04-17T07:26:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.928+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5500692b0 con 0x7fb55015b570 2024-04-17T07:26:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.929+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 90878+0+0 (secure 0 0 0) 0x7fb548005d10 con 0x7fb55015b570 2024-04-17T07:26:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.930+0000 7fb547e00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 0x7fb53006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.930+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7fb54808c5b0 con 0x7fb55015b570 2024-04-17T07:26:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.930+0000 7fb54fe00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 0x7fb53006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:58.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.931+0000 7fb54fe00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 0x7fb53006ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb540005880 tx=0x7fb540009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:58.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:58.934+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb5500692b0 con 0x7fb55015b570 2024-04-17T07:26:59.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.185+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb55006f350 con 0x7fb55015b570 2024-04-17T07:26:59.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.186+0000 7fb547e00640 1 -- 172.21.15.67:0/577658881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7fb548022070 con 0x7fb55015b570 2024-04-17T07:26:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:59.187 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:26:59.187 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:26:59.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 msgr2=0x7fb53006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 0x7fb53006ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb540005880 tx=0x7fb540009040 comp rx=0 tx=0).stop 2024-04-17T07:26:59.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 msgr2=0x7fb5500aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb54800efc0 tx=0x7fb548009f90 comp rx=0 tx=0).stop 2024-04-17T07:26:59.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 shutdown_connections 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb53006c740 0x7fb53006ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb55015b570 0x7fb5500aedd0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb55006feb0 0x7fb5501f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 --2- 172.21.15.67:0/577658881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb55006be70 0x7fb5501f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 >> 172.21.15.67:0/577658881 conn(0x7fb550067d10 msgr2=0x7fb550156050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 shutdown_connections 2024-04-17T07:26:59.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.189+0000 7fb555a00640 1 -- 172.21.15.67:0/577658881 wait complete. 2024-04-17T07:26:59.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:26:59.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.432+0000 7f6589200640 1 Processor -- start 2024-04-17T07:26:59.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.432+0000 7f6589200640 1 -- start start 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f6589200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65840cfae0 0x7f6584169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f6589200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65840d2d30 con 0x7f65840b6d00 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f6589200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65840d3010 con 0x7f658415ec60 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f6589200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f65840d49c0 con 0x7f65840cfae0 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f657be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f65840b6d00 0x7f65840cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35150/0 (socket says 172.21.15.67:35150) 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f657be00640 1 -- 172.21.15.67:0/698067863 learned_addr learned my addr 172.21.15.67:0/698067863 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.434+0000 7f6583e00640 1 --2- 172.21.15.67:0/698067863 >> v2:172.21.15.67:3300/0 conn(0x7f65840cfae0 0x7f6584169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.435+0000 7f6582a00640 1 -- 172.21.15.67:0/698067863 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736573761 0 0) 0x7f65840d2d30 con 0x7f65840b6d00 2024-04-17T07:26:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.435+0000 7f6582a00640 1 -- 172.21.15.67:0/698067863 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6568003520 con 0x7f65840b6d00 2024-04-17T07:26:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.435+0000 7f6583e00640 1 -- 172.21.15.67:0/698067863 >> v1:172.21.15.187:6789/0 conn(0x7f658415ec60 legacy=0x7f65840b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.435+0000 7f6583e00640 1 -- 172.21.15.67:0/698067863 >> v1:172.21.15.67:6789/0 conn(0x7f65840b6d00 legacy=0x7f65840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.435+0000 7f6583e00640 1 -- 172.21.15.67:0/698067863 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65840d32f0 con 0x7f65840cfae0 2024-04-17T07:26:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.436+0000 7f6583e00640 1 --2- 172.21.15.67:0/698067863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f6584169810 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f657c009d90 tx=0x7f657c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece7d69c9dd9e3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.436+0000 7f6582a00640 1 -- 172.21.15.67:0/698067863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c039070 con 0x7f65840cfae0 2024-04-17T07:26:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.436+0000 7f6582a00640 1 -- 172.21.15.67:0/698067863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f657c02cd00 con 0x7f65840cfae0 2024-04-17T07:26:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.436+0000 7f6582a00640 1 -- 172.21.15.67:0/698067863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c034640 con 0x7f65840cfae0 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 -- 172.21.15.67:0/698067863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 msgr2=0x7f6584169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 --2- 172.21.15.67:0/698067863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f6584169810 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f657c009d90 tx=0x7f657c02bf50 comp rx=0 tx=0).stop 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 -- 172.21.15.67:0/698067863 shutdown_connections 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 --2- 172.21.15.67:0/698067863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f6584169810 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 -- 172.21.15.67:0/698067863 >> 172.21.15.67:0/698067863 conn(0x7f65840d15f0 msgr2=0x7f65840d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 -- 172.21.15.67:0/698067863 shutdown_connections 2024-04-17T07:26:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.437+0000 7f6589200640 1 -- 172.21.15.67:0/698067863 wait complete. 2024-04-17T07:26:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.439+0000 7f6589200640 1 Processor -- start 2024-04-17T07:26:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.439+0000 7f6589200640 1 -- start start 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65840b6d00 0x7f65841580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 0x7f6584156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65840b8b80 con 0x7f65840cfae0 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f65840d3660 con 0x7f658415ec60 2024-04-17T07:26:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f6589200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f65840d00c0 con 0x7f65840b6d00 2024-04-17T07:26:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f657be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f657be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47160/0 (socket says 172.21.15.67:47160) 2024-04-17T07:26:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.440+0000 7f657be00640 1 -- 172.21.15.67:0/5328727 learned_addr learned my addr 172.21.15.67:0/5328727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f6583e00640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 0x7f6584156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 -- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65840b6d00 msgr2=0x7f65841580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f6583400640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65840b6d00 0x7f65841580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65840b6d00 0x7f65841580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 -- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 msgr2=0x7f6584156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 0x7f6584156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 -- 172.21.15.67:0/5328727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65841578f0 con 0x7f65840cfae0 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f6583e00640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 0x7f6584156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657be00640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f656c00c500 tx=0x7f656c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.441+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c017070 con 0x7f65840cfae0 2024-04-17T07:26:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.442+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f656c005ce0 con 0x7f65840cfae0 2024-04-17T07:26:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.442+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c007330 con 0x7f65840cfae0 2024-04-17T07:26:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.442+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65840ce1c0 con 0x7f65840cfae0 2024-04-17T07:26:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.442+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f658415fc60 con 0x7f65840cfae0 2024-04-17T07:26:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.443+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6584068a10 con 0x7f65840cfae0 2024-04-17T07:26:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.444+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 90878+0+0 (secure 0 0 0) 0x7f656c01a070 con 0x7f65840cfae0 2024-04-17T07:26:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.445+0000 7f657b400640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 0x7f656006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.445+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f656c08b070 con 0x7f65840cfae0 2024-04-17T07:26:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.446+0000 7f6583400640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 0x7f656006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.447+0000 7f6583400640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 0x7f656006ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6574000f60 tx=0x7f6574009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.450+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f656c059460 con 0x7f65840cfae0 2024-04-17T07:26:59.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.470+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 17) ==== 91086+0+0 (secure 0 0 0) 0x7f656c0553d0 con 0x7f65840cfae0 2024-04-17T07:26:59.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.704+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65840b9870 con 0x7f65840cfae0 2024-04-17T07:26:59.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.706+0000 7f657b400640 1 -- 172.21.15.67:0/5328727 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5193 (secure 0 0 0) 0x7f656c07cc60 con 0x7f65840cfae0 2024-04-17T07:26:59.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:26:59.706 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:15:11.212642+0000","modified":"2024-04-17T07:15:14.302921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5079},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5079":{"gid":5079,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/504171556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":504171556},{"type":"v1","addr":"172.21.15.67:6835","nonce":504171556}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5079,"qdb_cluster":[5079]},"id":2}]} 2024-04-17T07:26:59.706 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 msgr2=0x7f656006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 0x7f656006ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6574000f60 tx=0x7f6574009040 comp rx=0 tx=0).stop 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 msgr2=0x7f658415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f656c00c500 tx=0x7f656c00c9c0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 shutdown_connections 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f656006c740 0x7f656006ec00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f658415ec60 0x7f6584156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65840cfae0 0x7f658415ac00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 --2- 172.21.15.67:0/5328727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65840b6d00 0x7f65841580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 >> 172.21.15.67:0/5328727 conn(0x7f65840d15f0 msgr2=0x7f65840b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 shutdown_connections 2024-04-17T07:26:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.709+0000 7f6589200640 1 -- 172.21.15.67:0/5328727 wait complete. 2024-04-17T07:26:59.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:26:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.948+0000 7f1dce600640 1 Processor -- start 2024-04-17T07:26:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.948+0000 7f1dce600640 1 -- start start 2024-04-17T07:26:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.950+0000 7f1dce600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 0x7f1dc80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.950+0000 7f1dce600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.950+0000 7f1dce600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc80cd4a0 con 0x7f1dc8141ed0 2024-04-17T07:26:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.950+0000 7f1dce600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc815ce30 con 0x7f1dc806b8c0 2024-04-17T07:26:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.950+0000 7f1dce600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc81607f0 con 0x7f1dc8160280 2024-04-17T07:26:59.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.951+0000 7f1dc7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.951+0000 7f1dcdc00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1dc8141ed0 0x7f1dc815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:59790/0 (socket says 172.21.15.67:59790) 2024-04-17T07:26:59.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.951+0000 7f1dcdc00640 1 -- 172.21.15.67:0/3238386586 learned_addr learned my addr 172.21.15.67:0/3238386586 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:26:59.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.951+0000 7f1dcd200640 1 --2- 172.21.15.67:0/3238386586 >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 0x7f1dc80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.952+0000 7f1dc7e00640 1 -- 172.21.15.67:0/3238386586 >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 msgr2=0x7f1dc80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.952+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/3238386586 >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 0x7f1dc80cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.952+0000 7f1dc7e00640 1 -- 172.21.15.67:0/3238386586 >> v1:172.21.15.187:6789/0 conn(0x7f1dc8141ed0 legacy=0x7f1dc815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.952+0000 7f1dc7e00640 1 -- 172.21.15.67:0/3238386586 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc8160ad0 con 0x7f1dc8160280 2024-04-17T07:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.952+0000 7f1dcd200640 1 --2- 172.21.15.67:0/3238386586 >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 0x7f1dc80cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.953+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/3238386586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc806bca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1db40097a0 tx=0x7f1db402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f08ef376afedd9e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.953+0000 7f1dc7400640 1 -- 172.21.15.67:0/3238386586 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4035020 con 0x7f1dc8160280 2024-04-17T07:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.954+0000 7f1dc7400640 1 -- 172.21.15.67:0/3238386586 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db4031810 con 0x7f1dc8160280 2024-04-17T07:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.954+0000 7f1dc7400640 1 -- 172.21.15.67:0/3238386586 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4031be0 con 0x7f1dc8160280 2024-04-17T07:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.954+0000 7f1dce600640 1 -- 172.21.15.67:0/3238386586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 msgr2=0x7f1dc806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.954+0000 7f1dce600640 1 --2- 172.21.15.67:0/3238386586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc806bca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1db40097a0 tx=0x7f1db402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 -- 172.21.15.67:0/3238386586 shutdown_connections 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 --2- 172.21.15.67:0/3238386586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc806bca0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 --2- 172.21.15.67:0/3238386586 >> v2:172.21.15.187:3300/0 conn(0x7f1dc806b8c0 0x7f1dc80cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 -- 172.21.15.67:0/3238386586 >> 172.21.15.67:0/3238386586 conn(0x7f1dc8067d10 msgr2=0x7f1dc80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 -- 172.21.15.67:0/3238386586 shutdown_connections 2024-04-17T07:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.955+0000 7f1dce600640 1 -- 172.21.15.67:0/3238386586 wait complete. 2024-04-17T07:26:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.957+0000 7f1dce600640 1 Processor -- start 2024-04-17T07:26:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.957+0000 7f1dce600640 1 -- start start 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 0x7f1dc8155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc8143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc806c1e0 con 0x7f1dc8141ed0 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80cddf0 con 0x7f1dc806b8c0 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dce600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc8161090 con 0x7f1dc8160280 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.958+0000 7f1dc7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47168/0 (socket says 172.21.15.67:47168) 2024-04-17T07:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 -- 172.21.15.67:0/2043064671 learned_addr learned my addr 172.21.15.67:0/2043064671 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dcdc00640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc8143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dcd200640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 0x7f1dc8155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 -- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 msgr2=0x7f1dc8143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc8143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 -- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 msgr2=0x7f1dc8155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 0x7f1dc8155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 -- 172.21.15.67:0/2043064671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc8155e70 con 0x7f1dc8141ed0 2024-04-17T07:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dcd200640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 0x7f1dc8155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.959+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1db40097a0 tx=0x7f1db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.960+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4006e10 con 0x7f1dc8141ed0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.960+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db403ad60 con 0x7f1dc8141ed0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.960+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4044630 con 0x7f1dc8141ed0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.960+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc81f7600 con 0x7f1dc8141ed0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.960+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc81f7bc0 con 0x7f1dc8141ed0 2024-04-17T07:26:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.961+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc8068a10 con 0x7f1dc8141ed0 2024-04-17T07:26:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.962+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 91086+0+0 (secure 0 0 0) 0x7f1db404b060 con 0x7f1dc8141ed0 2024-04-17T07:26:59.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.963+0000 7f1dc4c00640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 0x7f1da806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:26:59.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.963+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f1db40ae060 con 0x7f1dc8141ed0 2024-04-17T07:26:59.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.964+0000 7f1dcd200640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 0x7f1da806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:26:59.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.965+0000 7f1dcd200640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 0x7f1da806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009ac0 tx=0x7f1dbc008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:26:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:26:59.968+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1db403f070 con 0x7f1dc8141ed0 2024-04-17T07:27:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.176+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1dc81f8280 con 0x7f1dc8141ed0 2024-04-17T07:27:00.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.180+0000 7f1dc4c00640 1 -- 172.21.15.67:0/2043064671 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+18268 (secure 0 0 0) 0x7f1db40ad090 con 0x7f1dc8141ed0 2024-04-17T07:27:00.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:00.181 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:26:56.792802+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:15:09.343098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-17T07:15:10.202353+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":53,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2322153279":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1982372125":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1699979613":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6833/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/618455126":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/3485602680":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/2778463946":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1528295426":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6833/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2513891628":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/14109251":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1536403455":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2851019947":"2024-04-18T07:26:56.792716+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1},{"begin":8,"length":2},{"begin":12,"length":1},{"begin":15,"length":2},{"begin":21,"length":1},{"begin":23,"length":6},{"begin":30,"length":1},{"begin":32,"length":2},{"begin":36,"length":5},{"begin":43,"length":1},{"begin":45,"length":5},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":57,"length":4},{"begin":62,"length":3},{"begin":68,"length":6},{"begin":75,"length":1},{"begin":77,"length":2},{"begin":80,"length":2},{"begin":84,"length":4},{"begin":89,"length":7},{"begin":97,"length":2},{"begin":100,"length":1},{"begin":103,"length":4},{"begin":108,"length":4},{"begin":144,"length":78}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.182+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 msgr2=0x7f1da806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.182+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 0x7f1da806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009ac0 tx=0x7f1dbc008040 comp rx=0 tx=0).stop 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.182+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 msgr2=0x7f1dc8142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.182+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1db40097a0 tx=0x7f1db40046c0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 shutdown_connections 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1da806c980 0x7f1da806ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1dc8160280 0x7f1dc8143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc8141ed0 0x7f1dc8142fc0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 --2- 172.21.15.67:0/2043064671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1dc806b8c0 0x7f1dc8155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 >> 172.21.15.67:0/2043064671 conn(0x7f1dc8067d10 msgr2=0x7f1dc806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:00.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 shutdown_connections 2024-04-17T07:27:00.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.183+0000 7f1dce600640 1 -- 172.21.15.67:0/2043064671 wait complete. 2024-04-17T07:27:00.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:27:00.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.419+0000 7f67acc00640 1 Processor -- start 2024-04-17T07:27:00.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.419+0000 7f67acc00640 1 -- start start 2024-04-17T07:27:00.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67a80d18e0 0x7f67a80d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a80d3350 0x7f67a8067600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f67a80b83b0 con 0x7f67a80d3350 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67a80d1d20 con 0x7f67a80d18e0 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67acc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67a80d4140 con 0x7f67a815ee20 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a80d3350 0x7f67a8067600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a80d3350 0x7f67a8067600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47178/0 (socket says 172.21.15.67:47178) 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.421+0000 7f67a7e00640 1 -- 172.21.15.67:0/712905254 learned_addr learned my addr 172.21.15.67:0/712905254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7400640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a6a00640 1 --2- 172.21.15.67:0/712905254 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d18e0 0x7f67a80d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7e00640 1 -- 172.21.15.67:0/712905254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 msgr2=0x7f67a80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7e00640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a80d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7e00640 1 -- 172.21.15.67:0/712905254 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d18e0 msgr2=0x7f67a80d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7e00640 1 --2- 172.21.15.67:0/712905254 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d18e0 0x7f67a80d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.422+0000 7f67a7e00640 1 -- 172.21.15.67:0/712905254 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a80d3b80 con 0x7f67a80d3350 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.423+0000 7f67a7e00640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d3350 0x7f67a8067600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f67a80686d0 tx=0x7f67a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296c1013c5774dec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67a6000640 1 -- 172.21.15.67:0/712905254 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a002ca70 con 0x7f67a80d3350 2024-04-17T07:27:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67a6000640 1 -- 172.21.15.67:0/712905254 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a002cc10 con 0x7f67a80d3350 2024-04-17T07:27:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67a6000640 1 -- 172.21.15.67:0/712905254 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a00343f0 con 0x7f67a80d3350 2024-04-17T07:27:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 -- 172.21.15.67:0/712905254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d3350 msgr2=0x7f67a8067600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d3350 0x7f67a8067600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f67a80686d0 tx=0x7f67a002bed0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 -- 172.21.15.67:0/712905254 shutdown_connections 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d3350 0x7f67a8067600 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 --2- 172.21.15.67:0/712905254 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d18e0 0x7f67a80d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 --2- 172.21.15.67:0/712905254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a80d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 -- 172.21.15.67:0/712905254 >> 172.21.15.67:0/712905254 conn(0x7f67a806d3c0 msgr2=0x7f67a80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 -- 172.21.15.67:0/712905254 shutdown_connections 2024-04-17T07:27:00.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.424+0000 7f67acc00640 1 -- 172.21.15.67:0/712905254 wait complete. 2024-04-17T07:27:00.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.426+0000 7f67acc00640 1 Processor -- start 2024-04-17T07:27:00.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.426+0000 7f67acc00640 1 -- start start 2024-04-17T07:27:00.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a80d3350 0x7f67a81f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a8159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67a815f200 con 0x7f67a80d18e0 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f67a80b8040 con 0x7f67a80d3350 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.427+0000 7f67acc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67a80d37e0 con 0x7f67a815ee20 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47188/0 (socket says 172.21.15.67:47188) 2024-04-17T07:27:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 -- 172.21.15.67:0/2931691828 learned_addr learned my addr 172.21.15.67:0/2931691828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7e00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a8159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a6a00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a80d3350 0x7f67a81f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 -- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 msgr2=0x7f67a8159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a8159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 -- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a80d3350 msgr2=0x7f67a81f8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a80d3350 0x7f67a81f8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.428+0000 7f67a7400640 1 -- 172.21.15.67:0/2931691828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a81f94b0 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f67a7400640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f679800d600 tx=0x7f679800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67980125f0 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6798019070 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67980129f0 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a80cce50 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.429+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a80cd410 con 0x7f67a80d18e0 2024-04-17T07:27:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.430+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a8068a10 con 0x7f67a80d18e0 2024-04-17T07:27:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.431+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 91086+0+0 (secure 0 0 0) 0x7f6798012b90 con 0x7f67a80d18e0 2024-04-17T07:27:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.432+0000 7f679f400640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 0x7f678806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.432+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6798+0+0 (secure 0 0 0) 0x7f679808c2d0 con 0x7f67a80d18e0 2024-04-17T07:27:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.433+0000 7f67a6a00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 0x7f678806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.434+0000 7f67a6a00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 0x7f678806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67a80b59c0 tx=0x7f6794008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.437+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f67a81f94b0 con 0x7f67a80d18e0 2024-04-17T07:27:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.681+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f67a80d09b0 con 0x7f67a80d18e0 2024-04-17T07:27:00.682 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:27:00.682+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:27:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.686+0000 7f679f400640 1 -- 172.21.15.67:0/2931691828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7f679805a360 con 0x7f67a80d18e0 2024-04-17T07:27:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:27:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 msgr2=0x7f678806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 0x7f678806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f67a80b59c0 tx=0x7f6794008040 comp rx=0 tx=0).stop 2024-04-17T07:27:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 msgr2=0x7f67a81549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f679800d600 tx=0x7f679800dac0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 shutdown_connections 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f678806c9b0 0x7f678806ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a815ee20 0x7f67a8159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a80d3350 0x7f67a81f8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 --2- 172.21.15.67:0/2931691828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d18e0 0x7f67a81549e0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.689+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 >> 172.21.15.67:0/2931691828 conn(0x7f67a806d3c0 msgr2=0x7f67a80b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.690+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 shutdown_connections 2024-04-17T07:27:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.690+0000 7f67acc00640 1 -- 172.21.15.67:0/2931691828 wait complete. 2024-04-17T07:27:00.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:27:00.809 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:27:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.924+0000 7f06ef600640 1 Processor -- start 2024-04-17T07:27:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.924+0000 7f06ef600640 1 -- start start 2024-04-17T07:27:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.925+0000 7f06ef600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.925+0000 7f06ef600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06e8072d80 0x7f06e80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.925+0000 7f06ef600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e80cea70 con 0x7f06e80ce3b0 2024-04-17T07:27:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.925+0000 7f06ef600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06e80ce810 con 0x7f06e8072d80 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.925+0000 7f06ef600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e80ced50 con 0x7f06e8145ac0 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06e8072d80 0x7f06e80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06e8072d80 0x7f06e80cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47212/0 (socket says 172.21.15.67:47212) 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 -- 172.21.15.67:0/4069626140 learned_addr learned my addr 172.21.15.67:0/4069626140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ecc00640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 -- 172.21.15.67:0/4069626140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 msgr2=0x7f06e8071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e8071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 -- 172.21.15.67:0/4069626140 >> v1:172.21.15.187:6789/0 conn(0x7f06e80ce3b0 legacy=0x7f06e8072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:00.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.926+0000 7f06ed600640 1 -- 172.21.15.67:0/4069626140 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e80cf030 con 0x7f06e8072d80 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.927+0000 7f06ed600640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e80cfb10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f06e0009e50 tx=0x7f06e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325834d430887838 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.928+0000 7f06e7400640 1 -- 172.21.15.67:0/4069626140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e002ca80 con 0x7f06e8072d80 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.928+0000 7f06e7400640 1 -- 172.21.15.67:0/4069626140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e002cc20 con 0x7f06e8072d80 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.928+0000 7f06e7400640 1 -- 172.21.15.67:0/4069626140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00343f0 con 0x7f06e8072d80 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.928+0000 7f06ef600640 1 -- 172.21.15.67:0/4069626140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 msgr2=0x7f06e80cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.928+0000 7f06ef600640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e80cfb10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f06e0009e50 tx=0x7f06e002bee0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 -- 172.21.15.67:0/4069626140 shutdown_connections 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e80cfb10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 --2- 172.21.15.67:0/4069626140 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e8071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 -- 172.21.15.67:0/4069626140 >> 172.21.15.67:0/4069626140 conn(0x7f06e8067ee0 msgr2=0x7f06e806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 -- 172.21.15.67:0/4069626140 shutdown_connections 2024-04-17T07:27:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.929+0000 7f06ef600640 1 -- 172.21.15.67:0/4069626140 wait complete. 2024-04-17T07:27:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.931+0000 7f06ef600640 1 Processor -- start 2024-04-17T07:27:00.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.931+0000 7f06ef600640 1 -- start start 2024-04-17T07:27:00.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e80ce3b0 0x7f06e80cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e81ea650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06e8145ea0 con 0x7f06e8072d80 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06e80cf5f0 con 0x7f06e80ce3b0 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ef600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e80d0050 con 0x7f06e8145ac0 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47218/0 (socket says 172.21.15.67:47218) 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.932+0000 7f06ecc00640 1 -- 172.21.15.67:0/1668768451 learned_addr learned my addr 172.21.15.67:0/1668768451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 -- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 msgr2=0x7f06e81ea650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e81ea650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 -- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e80ce3b0 msgr2=0x7f06e80cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:00.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e80ce3b0 0x7f06e80cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 -- 172.21.15.67:0/1668768451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e81400c0 con 0x7f06e8072d80 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ecc00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c7f0 tx=0x7f06dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc015070 con 0x7f06e8072d80 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06dc004220 con 0x7f06e8072d80 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06dc007620 con 0x7f06e8072d80 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.933+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e80ca670 con 0x7f06e8072d80 2024-04-17T07:27:00.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.934+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e80cac30 con 0x7f06e8072d80 2024-04-17T07:27:00.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.935+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06e8068ca0 con 0x7f06e8072d80 2024-04-17T07:27:00.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.936+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 91086+0+0 (secure 0 0 0) 0x7f06dc0079d0 con 0x7f06e8072d80 2024-04-17T07:27:00.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.936+0000 7f06e4c00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 0x7f06c806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:00.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.937+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6884+0+0 (secure 0 0 0) 0x7f06dc08c140 con 0x7f06e8072d80 2024-04-17T07:27:00.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.937+0000 7f06e7e00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 0x7f06c806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:00.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.938+0000 7f06e7e00640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 0x7f06c806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06e81f7f20 tx=0x7f06d40079b0 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:00.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:00.941+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f06dc059f60 con 0x7f06e8072d80 2024-04-17T07:27:01.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.182+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f06e80cc6c0 con 0x7f06e8072d80 2024-04-17T07:27:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.701+0000 7f06e4c00640 1 -- 172.21.15.67:0/1668768451 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f06dc059860 con 0x7f06e8072d80 2024-04-17T07:27:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.704+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 msgr2=0x7f06c806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.704+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 0x7f06c806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06e81f7f20 tx=0x7f06d40079b0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 msgr2=0x7f06e813fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c7f0 tx=0x7f06dc00ccb0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 shutdown_connections 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06c806c9b0 0x7f06c806ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e8145ac0 0x7f06e81ea650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e80ce3b0 0x7f06e80cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 --2- 172.21.15.67:0/1668768451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e8072d80 0x7f06e813fb80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 >> 172.21.15.67:0/1668768451 conn(0x7f06e8067ee0 msgr2=0x7f06e8144ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.705+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 shutdown_connections 2024-04-17T07:27:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.706+0000 7f06ef600640 1 -- 172.21.15.67:0/1668768451 wait complete. 2024-04-17T07:27:01.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:27:01.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.951+0000 7f2ab7200640 1 Processor -- start 2024-04-17T07:27:01.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.951+0000 7f2ab7200640 1 -- start start 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 0x7f2ab0071d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 0x7f2ab0072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab01460a0 0x7f2ab006dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab01431f0 con 0x7f2ab01460a0 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab0160e40 con 0x7f2ab0142e10 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab01612d0 con 0x7f2ab006f220 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab01460a0 0x7f2ab006dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab01460a0 0x7f2ab006dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47220/0 (socket says 172.21.15.67:47220) 2024-04-17T07:27:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 -- 172.21.15.67:0/3347118556 learned_addr learned my addr 172.21.15.67:0/3347118556 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/3347118556 >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 0x7f2ab0071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 -- 172.21.15.67:0/3347118556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 msgr2=0x7f2ab0072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab5400640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 0x7f2ab0072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 0x7f2ab0072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 -- 172.21.15.67:0/3347118556 >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 msgr2=0x7f2ab0071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 --2- 172.21.15.67:0/3347118556 >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 0x7f2ab0071d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab6800640 1 -- 172.21.15.67:0/3347118556 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab0160510 con 0x7f2ab01460a0 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/3347118556 >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 0x7f2ab0071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.953+0000 7f2ab5400640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 0x7f2ab0072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.954+0000 7f2ab6800640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab006dd40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00b5730 tx=0x7f2aac02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e093c51288aed96 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.955+0000 7f2ab4a00640 1 -- 172.21.15.67:0/3347118556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac02c9f0 con 0x7f2ab01460a0 2024-04-17T07:27:01.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.955+0000 7f2ab4a00640 1 -- 172.21.15.67:0/3347118556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac02cb90 con 0x7f2ab01460a0 2024-04-17T07:27:01.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.955+0000 7f2ab4a00640 1 -- 172.21.15.67:0/3347118556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0343f0 con 0x7f2ab01460a0 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.955+0000 7f2ab7200640 1 -- 172.21.15.67:0/3347118556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 msgr2=0x7f2ab006dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.955+0000 7f2ab7200640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab006dd40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00b5730 tx=0x7f2aac02be50 comp rx=0 tx=0).stop 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 -- 172.21.15.67:0/3347118556 shutdown_connections 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab006dd40 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 --2- 172.21.15.67:0/3347118556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab006f220 0x7f2ab0072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 --2- 172.21.15.67:0/3347118556 >> v2:172.21.15.187:3300/0 conn(0x7f2ab0142e10 0x7f2ab0071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 -- 172.21.15.67:0/3347118556 >> 172.21.15.67:0/3347118556 conn(0x7f2ab006b8c0 msgr2=0x7f2ab006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 -- 172.21.15.67:0/3347118556 shutdown_connections 2024-04-17T07:27:01.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.956+0000 7f2ab7200640 1 -- 172.21.15.67:0/3347118556 wait complete. 2024-04-17T07:27:01.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.957+0000 7f2ab7200640 1 Processor -- start 2024-04-17T07:27:01.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.958+0000 7f2ab7200640 1 -- start start 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 0x7f2ab013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab013fa20 0x7f2ab0131880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab0145ac0 con 0x7f2ab01460a0 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab0160ad0 con 0x7f2ab0142e10 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab0145c40 con 0x7f2ab013fa20 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47226/0 (socket says 172.21.15.67:47226) 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 -- 172.21.15.67:0/805953071 learned_addr learned my addr 172.21.15.67:0/805953071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 0x7f2ab013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 -- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab013fa20 msgr2=0x7f2ab0131880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab6800640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab013fa20 0x7f2ab0131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab013fa20 0x7f2ab0131880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 -- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 msgr2=0x7f2ab013f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 0x7f2ab013f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5400640 1 -- 172.21.15.67:0/805953071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab013ff50 con 0x7f2ab01460a0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.959+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 0x7f2ab013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2ab5400640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c7f0 tx=0x7f2aa000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0015070 con 0x7f2ab01460a0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa00077c0 con 0x7f2ab01460a0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0007bb0 con 0x7f2ab01460a0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab0140230 con 0x7f2ab01460a0 2024-04-17T07:27:01.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.960+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab013d3d0 con 0x7f2ab01460a0 2024-04-17T07:27:01.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.961+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab0140750 con 0x7f2ab01460a0 2024-04-17T07:27:01.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.962+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f2aa0018070 con 0x7f2ab01460a0 2024-04-17T07:27:01.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.964+0000 7f2aaa000640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 0x7f2a9406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:01.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.964+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6884+0+0 (secure 0 0 0) 0x7f2aa008c3b0 con 0x7f2ab01460a0 2024-04-17T07:27:01.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.964+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 0x7f2a9406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:01.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.965+0000 7f2ab5e00640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 0x7f2a9406ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006fd0 tx=0x7f2aa4008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:01.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:01.968+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2ab0140750 con 0x7f2ab01460a0 2024-04-17T07:27:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.164+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2ab00b3170 con 0x7f2ab01460a0 2024-04-17T07:27:02.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.705+0000 7f2aaa000640 1 -- 172.21.15.67:0/805953071 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v56) ==== 181+0+0 (secure 0 0 0) 0x7f2aa005a000 con 0x7f2ab01460a0 2024-04-17T07:27:02.705 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.708+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 msgr2=0x7f2a9406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 0x7f2a9406ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006fd0 tx=0x7f2aa4008040 comp rx=0 tx=0).stop 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 msgr2=0x7f2ab0131340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c7f0 tx=0x7f2aa000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 shutdown_connections 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2a9406ca40 0x7f2a9406ef00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab013fa20 0x7f2ab0131880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab01460a0 0x7f2ab0131340 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 --2- 172.21.15.67:0/805953071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab0142e10 0x7f2ab013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 >> 172.21.15.67:0/805953071 conn(0x7f2ab006b8c0 msgr2=0x7f2ab0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:02.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 shutdown_connections 2024-04-17T07:27:02.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.709+0000 7f2ab7200640 1 -- 172.21.15.67:0/805953071 wait complete. 2024-04-17T07:27:02.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:27:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.947+0000 7f4692e00640 1 Processor -- start 2024-04-17T07:27:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.948+0000 7f4692e00640 1 -- start start 2024-04-17T07:27:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.949+0000 7f4692e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f468c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.949+0000 7f4692e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f468c0b4e20 0x7f468c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.949+0000 7f4692e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468c0ce0e0 con 0x7f468c0cfc60 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.949+0000 7f4692e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f468c0d00f0 con 0x7f468c0b4e20 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.949+0000 7f4692e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f468c0b5860 con 0x7f468c0cd7c0 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f4690e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f468c0cfc60 0x7f468c071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35152/0 (socket says 172.21.15.67:35152) 2024-04-17T07:27:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f4690e00640 1 -- 172.21.15.67:0/3229264754 learned_addr learned my addr 172.21.15.67:0/3229264754 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f468be00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f468c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f4683e00640 1 --2- 172.21.15.67:0/3229264754 >> v2:172.21.15.187:3300/0 conn(0x7f468c0b4e20 0x7f468c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545424988 0 0) 0x7f468c0ce0e0 con 0x7f468c0cfc60 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4670003520 con 0x7f468c0cfc60 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.950+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 275409412 0 0) 0x7f4670003520 con 0x7f468c0cfc60 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 msgr2=0x7f468c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f468c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 >> v2:172.21.15.187:3300/0 conn(0x7f468c0b4e20 msgr2=0x7f468c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 --2- 172.21.15.67:0/3229264754 >> v2:172.21.15.187:3300/0 conn(0x7f468c0b4e20 0x7f468c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c0b4060 con 0x7f468c0cfc60 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f46840019f0 con 0x7f468c0cfc60 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 >> v1:172.21.15.67:6789/0 conn(0x7f468c0cfc60 legacy=0x7f468c071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46700039b0 0x7f4670003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f4670004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f46700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0d00f0 con 0x7f4670004d40 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0b5860 con 0x7f46700039b0 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f467000dbd0 con 0x7f468c0cd7c0 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f4690e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f46700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f4683e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f4670004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f468be00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46700039b0 0x7f4670003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f4690e00640 1 -- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 msgr2=0x7f4670004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.951+0000 7f4690e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c0cd7c0 0x7f4670004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.952+0000 7f4690e00640 1 -- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46700039b0 msgr2=0x7f4670003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.952+0000 7f4690e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46700039b0 0x7f4670003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.952+0000 7f4690e00640 1 -- 172.21.15.67:0/3229264754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4670003520 con 0x7f4670004d40 2024-04-17T07:27:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f4690e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f46700051c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f468400a870 tx=0x7f4670005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fabfb55e95b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468400a4c0 con 0x7f4670004d40 2024-04-17T07:27:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4684034630 con 0x7f4670004d40 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f468b400640 1 -- 172.21.15.67:0/3229264754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684034a20 con 0x7f4670004d40 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f4692e00640 1 -- 172.21.15.67:0/3229264754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 msgr2=0x7f46700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.953+0000 7f4692e00640 1 --2- 172.21.15.67:0/3229264754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f46700051c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f468400a870 tx=0x7f4670005830 comp rx=0 tx=0).stop 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.954+0000 7f4690e00640 1 -- 172.21.15.67:0/3229264754 reap_dead start 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.954+0000 7f4692e00640 1 -- 172.21.15.67:0/3229264754 shutdown_connections 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.954+0000 7f4692e00640 1 -- 172.21.15.67:0/3229264754 >> 172.21.15.67:0/3229264754 conn(0x7f468c06b8c0 msgr2=0x7f468c06bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.954+0000 7f4692e00640 1 -- 172.21.15.67:0/3229264754 shutdown_connections 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.954+0000 7f4692e00640 1 -- 172.21.15.67:0/3229264754 wait complete. 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.955+0000 7f4692e00640 1 Processor -- start 2024-04-17T07:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.956+0000 7f4692e00640 1 -- start start 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f468c0c5a90 0x7f468c1e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 0x7f468c0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0d0350 con 0x7f4670004d40 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f468c0b4620 con 0x7f468c0c5a90 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4692e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f468c145390 con 0x7f468c1e65f0 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47242/0 (socket says 172.21.15.67:47242) 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 -- 172.21.15.67:0/1261810222 learned_addr learned my addr 172.21.15.67:0/1261810222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4690e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 0x7f468c0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4683e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f468c0c5a90 0x7f468c1e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 -- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 msgr2=0x7f468c0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 0x7f468c0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 -- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f468c0c5a90 msgr2=0x7f468c1e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f468c0c5a90 0x7f468c1e60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f468be00640 1 -- 172.21.15.67:0/1261810222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c0c8090 con 0x7f4670004d40 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.957+0000 7f4690e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 0x7f468c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f468be00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f467c010860 tx=0x7f467c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c00f040 con 0x7f4670004d40 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467c00eb00 con 0x7f4670004d40 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c01c400 con 0x7f4670004d40 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c135170 con 0x7f4670004d40 2024-04-17T07:27:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.958+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c135730 con 0x7f4670004d40 2024-04-17T07:27:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.959+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c073a50 con 0x7f4670004d40 2024-04-17T07:27:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.960+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f467c01c5a0 con 0x7f4670004d40 2024-04-17T07:27:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.961+0000 7f4688c00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 0x7f466806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.961+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6446+0+0 (secure 0 0 0) 0x7f467c08c5f0 con 0x7f4670004d40 2024-04-17T07:27:02.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.962+0000 7f4683e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 0x7f466806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:02.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.963+0000 7f4683e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 0x7f466806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f468c0b3670 tx=0x7f4674008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:02.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:02.966+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f467c018030 con 0x7f4670004d40 2024-04-17T07:27:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.169+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f468c1e69d0 con 0x7f4670004d40 2024-04-17T07:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.286+0000 7f4688c00640 1 -- 172.21.15.67:0/1261810222 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v57) ==== 169+0+0 (secure 0 0 0) 0x7f467c059fe0 con 0x7f4670004d40 2024-04-17T07:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:27:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 msgr2=0x7f466806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 0x7f466806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f468c0b3670 tx=0x7f4674008040 comp rx=0 tx=0).stop 2024-04-17T07:27:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 msgr2=0x7f468c0c7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f467c010860 tx=0x7f467c010d20 comp rx=0 tx=0).stop 2024-04-17T07:27:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 shutdown_connections 2024-04-17T07:27:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f466806ca10 0x7f466806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f468c1e65f0 0x7f468c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f468c0c5a90 0x7f468c1e60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 --2- 172.21.15.67:0/1261810222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4670004d40 0x7f468c0c7b50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.289+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 >> 172.21.15.67:0/1261810222 conn(0x7f468c06b8c0 msgr2=0x7f468c0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.290+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 shutdown_connections 2024-04-17T07:27:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.290+0000 7f4692e00640 1 -- 172.21.15.67:0/1261810222 wait complete. 2024-04-17T07:27:03.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:27:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.529+0000 7f4bec400640 1 Processor -- start 2024-04-17T07:27:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.529+0000 7f4bec400640 1 -- start start 2024-04-17T07:27:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4bec400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4be406daf0 0x7f4be40d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4bec400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4be409b170 0x7f4be4070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4bec400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be40d1520 con 0x7f4be40d1060 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4bec400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4be40d1800 con 0x7f4be406daf0 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4bec400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4be40d1ae0 con 0x7f4be409b170 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4be9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4be406daf0 0x7f4be40d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4be9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4be406daf0 0x7f4be40d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33298/0 (socket says 172.21.15.67:33298) 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.531+0000 7f4be9a00640 1 -- 172.21.15.67:0/62079350 learned_addr learned my addr 172.21.15.67:0/62079350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.532+0000 7f4be9a00640 1 -- 172.21.15.67:0/62079350 >> v2:172.21.15.187:3300/0 conn(0x7f4be409b170 msgr2=0x7f4be4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.532+0000 7f4be9a00640 1 --2- 172.21.15.67:0/62079350 >> v2:172.21.15.187:3300/0 conn(0x7f4be409b170 0x7f4be4070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.532+0000 7f4be9a00640 1 -- 172.21.15.67:0/62079350 >> v1:172.21.15.187:6789/0 conn(0x7f4be40d1060 legacy=0x7f4be409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.532+0000 7f4be9a00640 1 -- 172.21.15.67:0/62079350 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be40d1dc0 con 0x7f4be406daf0 2024-04-17T07:27:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.533+0000 7f4be9a00640 1 --2- 172.21.15.67:0/62079350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be406daf0 0x7f4be40d0b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4be40d3710 tx=0x7f4bd402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29d93a5e41db674 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.533+0000 7f4be3e00640 1 -- 172.21.15.67:0/62079350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4035020 con 0x7f4be406daf0 2024-04-17T07:27:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.533+0000 7f4be3e00640 1 -- 172.21.15.67:0/62079350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd4031890 con 0x7f4be406daf0 2024-04-17T07:27:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.533+0000 7f4be3e00640 1 -- 172.21.15.67:0/62079350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4031c60 con 0x7f4be406daf0 2024-04-17T07:27:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 -- 172.21.15.67:0/62079350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be406daf0 msgr2=0x7f4be40d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 --2- 172.21.15.67:0/62079350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be406daf0 0x7f4be40d0b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4be40d3710 tx=0x7f4bd402b930 comp rx=0 tx=0).stop 2024-04-17T07:27:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 -- 172.21.15.67:0/62079350 shutdown_connections 2024-04-17T07:27:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 --2- 172.21.15.67:0/62079350 >> v2:172.21.15.187:3300/0 conn(0x7f4be409b170 0x7f4be4070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 --2- 172.21.15.67:0/62079350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be406daf0 0x7f4be40d0b20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 -- 172.21.15.67:0/62079350 >> 172.21.15.67:0/62079350 conn(0x7f4be40b41a0 msgr2=0x7f4be40b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 -- 172.21.15.67:0/62079350 shutdown_connections 2024-04-17T07:27:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.534+0000 7f4bec400640 1 -- 172.21.15.67:0/62079350 wait complete. 2024-04-17T07:27:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.536+0000 7f4bec400640 1 Processor -- start 2024-04-17T07:27:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.536+0000 7f4bec400640 1 -- start start 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 0x7f4be40c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4be40d1060 0x7f4be4156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4be406e050 con 0x7f4be409b170 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4be406ded0 con 0x7f4be406daf0 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.537+0000 7f4bec400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4be406ce00 con 0x7f4be40d1060 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33300/0 (socket says 172.21.15.67:33300) 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 -- 172.21.15.67:0/3655779557 learned_addr learned my addr 172.21.15.67:0/3655779557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9a00640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 0x7f4be40c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4bea400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4be40d1060 0x7f4be4156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 -- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4be40d1060 msgr2=0x7f4be4156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4be40d1060 0x7f4be4156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 -- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 msgr2=0x7f4be40c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 0x7f4be40c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9000640 1 -- 172.21.15.67:0/3655779557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be40cabe0 con 0x7f4be409b170 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.538+0000 7f4be9a00640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 0x7f4be40c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4be9000640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b410 tx=0x7f4bd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8015020 con 0x7f4be409b170 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd8011830 con 0x7f4be409b170 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8011c20 con 0x7f4be409b170 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be40cae80 con 0x7f4be409b170 2024-04-17T07:27:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.539+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be40cb120 con 0x7f4be409b170 2024-04-17T07:27:03.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.540+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be4068ca0 con 0x7f4be409b170 2024-04-17T07:27:03.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.541+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f4bd8018020 con 0x7f4be409b170 2024-04-17T07:27:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.542+0000 7f4be1600640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 0x7f4bc806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:03.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.543+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f4bd808bcd0 con 0x7f4be409b170 2024-04-17T07:27:03.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.543+0000 7f4be9a00640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 0x7f4bc806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:03.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.544+0000 7f4be9a00640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 0x7f4bc806eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4036000 tx=0x7f4bd403c040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.548+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4bd801f020 con 0x7f4be409b170 2024-04-17T07:27:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.758+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f4be40b5040 con 0x7f4be409b170 2024-04-17T07:27:03.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.765+0000 7f4be1600640 1 -- 172.21.15.67:0/3655779557 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) ==== 115+0+0 (secure 0 0 0) 0x7f4bd80172c0 con 0x7f4be409b170 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 msgr2=0x7f4bc806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 0x7f4bc806eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4036000 tx=0x7f4bd403c040 comp rx=0 tx=0).stop 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 msgr2=0x7f4be41567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b410 tx=0x7f4bd800b8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 shutdown_connections 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f4bc806ca10 0x7f4bc806eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4be40d1060 0x7f4be4156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be409b170 0x7f4be41567f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 --2- 172.21.15.67:0/3655779557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be406daf0 0x7f4be40c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 >> 172.21.15.67:0/3655779557 conn(0x7f4be40b41a0 msgr2=0x7f4be40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 shutdown_connections 2024-04-17T07:27:03.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:03.768+0000 7f4bec400640 1 -- 172.21.15.67:0/3655779557 wait complete. 2024-04-17T07:27:03.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:27:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.004+0000 7f25e9000640 1 Processor -- start 2024-04-17T07:27:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.004+0000 7f25e9000640 1 -- start start 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.005+0000 7f25e9000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25e415eb10 0x7f25e40b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.005+0000 7f25e9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40b36d0 0x7f25e40d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.006+0000 7f25e9000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e40d28a0 con 0x7f25e40d2fe0 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.006+0000 7f25e9000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f25e40d3450 con 0x7f25e415eb10 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.006+0000 7f25e9000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f25e40d51e0 con 0x7f25e40b36d0 2024-04-17T07:27:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.006+0000 7f25e2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40b36d0 0x7f25e40d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.006+0000 7f25e3400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f25e415eb10 0x7f25e40b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.007+0000 7f25e3e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f25e40d2fe0 0x7f25e40d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:46562/0 (socket says 172.21.15.67:46562) 2024-04-17T07:27:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.007+0000 7f25e3e00640 1 -- 172.21.15.67:0/3931832962 learned_addr learned my addr 172.21.15.67:0/3931832962 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.007+0000 7f25e3400640 1 -- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40b36d0 msgr2=0x7f25e40d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.007+0000 7f25e3400640 1 --2- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40b36d0 0x7f25e40d2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.007+0000 7f25e3400640 1 -- 172.21.15.67:0/3931832962 >> v1:172.21.15.187:6789/0 conn(0x7f25e40d2fe0 legacy=0x7f25e40d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.008+0000 7f25e3400640 1 -- 172.21.15.67:0/3931832962 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e40d54c0 con 0x7f25e415eb10 2024-04-17T07:27:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.008+0000 7f25e2000640 1 -- 172.21.15.67:0/3931832962 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600900032 0 0) 0x7f25dc001110 con 0x7f25e40d2fe0 2024-04-17T07:27:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.009+0000 7f25e3400640 1 --2- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e415eb10 0x7f25e40b3c20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009d90 tx=0x7f25d402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bdf12bd3512c185 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.009+0000 7f25e2000640 1 -- 172.21.15.67:0/3931832962 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4039070 con 0x7f25e415eb10 2024-04-17T07:27:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.009+0000 7f25e2000640 1 -- 172.21.15.67:0/3931832962 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d402cc50 con 0x7f25e415eb10 2024-04-17T07:27:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e2000640 1 -- 172.21.15.67:0/3931832962 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4034440 con 0x7f25e415eb10 2024-04-17T07:27:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 -- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e415eb10 msgr2=0x7f25e40b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 --2- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e415eb10 0x7f25e40b3c20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009d90 tx=0x7f25d402bea0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 -- 172.21.15.67:0/3931832962 shutdown_connections 2024-04-17T07:27:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 --2- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40b36d0 0x7f25e40d2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 --2- 172.21.15.67:0/3931832962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e415eb10 0x7f25e40b3c20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 -- 172.21.15.67:0/3931832962 >> 172.21.15.67:0/3931832962 conn(0x7f25e408bff0 msgr2=0x7f25e40d4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 -- 172.21.15.67:0/3931832962 shutdown_connections 2024-04-17T07:27:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.010+0000 7f25e9000640 1 -- 172.21.15.67:0/3931832962 wait complete. 2024-04-17T07:27:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.012+0000 7f25e9000640 1 Processor -- start 2024-04-17T07:27:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.012+0000 7f25e9000640 1 -- start start 2024-04-17T07:27:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e40b36d0 0x7f25e4157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40d2fe0 0x7f25e40c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25e415eef0 con 0x7f25e415eb10 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f25e40d0210 con 0x7f25e40b36d0 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e9000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f25e406da70 con 0x7f25e40d2fe0 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33312/0 (socket says 172.21.15.67:33312) 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.013+0000 7f25e3e00640 1 -- 172.21.15.67:0/1143631276 learned_addr learned my addr 172.21.15.67:0/1143631276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3400640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e40b36d0 0x7f25e4157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e2a00640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40d2fe0 0x7f25e40c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 -- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40d2fe0 msgr2=0x7f25e40c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40d2fe0 0x7f25e40c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 -- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e40b36d0 msgr2=0x7f25e4157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e40b36d0 0x7f25e4157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 -- 172.21.15.67:0/1143631276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e41f42d0 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.014+0000 7f25e3e00640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00c410 tx=0x7f25dc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.015+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc0074e0 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.015+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e40c9a90 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.015+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25dc0047e0 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.015+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc004bd0 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.015+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e40ca050 con 0x7f25e415eb10 2024-04-17T07:27:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.016+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e4068a10 con 0x7f25e415eb10 2024-04-17T07:27:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.017+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f25dc011070 con 0x7f25e415eb10 2024-04-17T07:27:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.018+0000 7f25db400640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 0x7f25c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.018+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f25dc08b9e0 con 0x7f25e415eb10 2024-04-17T07:27:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.019+0000 7f25e3400640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 0x7f25c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.020+0000 7f25e3400640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 0x7f25c006eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f25d4036000 tx=0x7f25d4030000 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.023+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f25e40d28a0 con 0x7f25e415eb10 2024-04-17T07:27:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.235+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f25e40d1dc0 con 0x7f25e415eb10 2024-04-17T07:27:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.241+0000 7f25db400640 1 -- 172.21.15.67:0/1143631276 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7f25dc01a2b0 con 0x7f25e415eb10 2024-04-17T07:27:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 msgr2=0x7f25c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 0x7f25c006eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f25d4036000 tx=0x7f25d4030000 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 msgr2=0x7f25e40c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00c410 tx=0x7f25dc00c8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 shutdown_connections 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f25c006ca10 0x7f25c006eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25e415eb10 0x7f25e40c9460 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f25e40d2fe0 0x7f25e40c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 --2- 172.21.15.67:0/1143631276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f25e40b36d0 0x7f25e4157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.244+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 >> 172.21.15.67:0/1143631276 conn(0x7f25e408bff0 msgr2=0x7f25e40c82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.245+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 shutdown_connections 2024-04-17T07:27:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.245+0000 7f25e9000640 1 -- 172.21.15.67:0/1143631276 wait complete. 2024-04-17T07:27:04.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:27:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.471+0000 7f1e53200640 1 Processor -- start 2024-04-17T07:27:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.471+0000 7f1e53200640 1 -- start start 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 0x7f1e540cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e540cd750 0x7f1e5406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e5406c0a0 0x7f1e54160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e540cdbb0 con 0x7f1e540cd750 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1e540727a0 con 0x7f1e5406c0a0 2024-04-17T07:27:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e53200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e54146360 con 0x7f1e54145f80 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e51400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e540cd750 0x7f1e5406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e51400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e540cd750 0x7f1e5406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33314/0 (socket says 172.21.15.67:33314) 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e51400640 1 -- 172.21.15.67:0/2202044118 learned_addr learned my addr 172.21.15.67:0/2202044118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e52800640 1 --2- 172.21.15.67:0/2202044118 >> v2:172.21.15.187:3300/0 conn(0x7f1e5406c0a0 0x7f1e54160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.473+0000 7f1e51e00640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 0x7f1e540cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51400640 1 -- 172.21.15.67:0/2202044118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 msgr2=0x7f1e540cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51400640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 0x7f1e540cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51400640 1 -- 172.21.15.67:0/2202044118 >> v2:172.21.15.187:3300/0 conn(0x7f1e5406c0a0 msgr2=0x7f1e54160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51400640 1 --2- 172.21.15.67:0/2202044118 >> v2:172.21.15.187:3300/0 conn(0x7f1e5406c0a0 0x7f1e54160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51400640 1 -- 172.21.15.67:0/2202044118 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e54071e70 con 0x7f1e540cd750 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.474+0000 7f1e51e00640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 0x7f1e540cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.475+0000 7f1e51400640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e540cd750 0x7f1e5406bb60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1e540d03d0 tx=0x7f1e3c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe018f0db8e4cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.475+0000 7f1e50a00640 1 -- 172.21.15.67:0/2202044118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0317d0 con 0x7f1e540cd750 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.475+0000 7f1e50a00640 1 -- 172.21.15.67:0/2202044118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3c037070 con 0x7f1e540cd750 2024-04-17T07:27:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.475+0000 7f1e50a00640 1 -- 172.21.15.67:0/2202044118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c031b90 con 0x7f1e540cd750 2024-04-17T07:27:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 -- 172.21.15.67:0/2202044118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e540cd750 msgr2=0x7f1e5406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e540cd750 0x7f1e5406bb60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1e540d03d0 tx=0x7f1e3c02b870 comp rx=0 tx=0).stop 2024-04-17T07:27:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 -- 172.21.15.67:0/2202044118 shutdown_connections 2024-04-17T07:27:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 --2- 172.21.15.67:0/2202044118 >> v2:172.21.15.187:3300/0 conn(0x7f1e5406c0a0 0x7f1e54160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e540cd750 0x7f1e5406bb60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 --2- 172.21.15.67:0/2202044118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e54145f80 0x7f1e540cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 -- 172.21.15.67:0/2202044118 >> 172.21.15.67:0/2202044118 conn(0x7f1e540b2190 msgr2=0x7f1e540b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 -- 172.21.15.67:0/2202044118 shutdown_connections 2024-04-17T07:27:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.476+0000 7f1e53200640 1 -- 172.21.15.67:0/2202044118 wait complete. 2024-04-17T07:27:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.478+0000 7f1e53200640 1 Processor -- start 2024-04-17T07:27:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.478+0000 7f1e53200640 1 -- start start 2024-04-17T07:27:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e540cd750 0x7f1e540ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e54145f80 0x7f1e540ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e54072c30 con 0x7f1e5406c0a0 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e54072430 con 0x7f1e54145f80 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e53200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e54072db0 con 0x7f1e540cd750 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33316/0 (socket says 172.21.15.67:33316) 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.479+0000 7f1e51e00640 1 -- 172.21.15.67:0/4203504304 learned_addr learned my addr 172.21.15.67:0/4203504304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 -- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e540cd750 msgr2=0x7f1e540ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e540cd750 0x7f1e540ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 -- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e54145f80 msgr2=0x7f1e540ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e54145f80 0x7f1e540ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 -- 172.21.15.67:0/4203504304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5415cd30 con 0x7f1e5406c0a0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e51e00640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400ce00 tx=0x7f1e4400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4400f890 con 0x7f1e5406c0a0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e4400fa30 con 0x7f1e5406c0a0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4400fe40 con 0x7f1e5406c0a0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e54141300 con 0x7f1e5406c0a0 2024-04-17T07:27:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.480+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5415d280 con 0x7f1e5406c0a0 2024-04-17T07:27:04.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.482+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e540ce9b0 con 0x7f1e5406c0a0 2024-04-17T07:27:04.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.483+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f1e44018400 con 0x7f1e5406c0a0 2024-04-17T07:27:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.483+0000 7f1e4a000640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 0x7f1e3006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.484+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f1e4408cd70 con 0x7f1e5406c0a0 2024-04-17T07:27:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.484+0000 7f1e51400640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 0x7f1e3006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.485+0000 7f1e51400640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 0x7f1e3006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c036000 tx=0x7f1e3c030000 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.488+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e540ce9b0 con 0x7f1e5406c0a0 2024-04-17T07:27:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.685+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f1e54068a10 con 0x7f1e5406c0a0 2024-04-17T07:27:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.691+0000 7f1e4a000640 1 -- 172.21.15.67:0/4203504304 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7f1e44014030 con 0x7f1e5406c0a0 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 msgr2=0x7f1e3006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 0x7f1e3006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c036000 tx=0x7f1e3c030000 comp rx=0 tx=0).stop 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 msgr2=0x7f1e54140dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400ce00 tx=0x7f1e4400a650 comp rx=0 tx=0).stop 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 shutdown_connections 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f1e3006ca10 0x7f1e3006eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e54145f80 0x7f1e540ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e540cd750 0x7f1e540ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 --2- 172.21.15.67:0/4203504304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e5406c0a0 0x7f1e54140dc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 >> 172.21.15.67:0/4203504304 conn(0x7f1e540b2190 msgr2=0x7f1e540af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 shutdown_connections 2024-04-17T07:27:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.695+0000 7f1e53200640 1 -- 172.21.15.67:0/4203504304 wait complete. 2024-04-17T07:27:04.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-04-17T07:27:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.928+0000 7f0701400640 1 Processor -- start 2024-04-17T07:27:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.929+0000 7f0701400640 1 -- start start 2024-04-17T07:27:04.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.930+0000 7f0701400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06fc0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.930+0000 7f0701400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06fc0cd4a0 con 0x7f06fc141ed0 2024-04-17T07:27:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.930+0000 7f0701400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06fc1607f0 con 0x7f06fc160280 2024-04-17T07:27:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.930+0000 7f0701400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06fc15ce30 con 0x7f06fc06b8c0 2024-04-17T07:27:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.931+0000 7f0700a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06fc141ed0 0x7f06fc15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42310/0 (socket says 172.21.15.67:42310) 2024-04-17T07:27:04.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.931+0000 7f0700a00640 1 -- 172.21.15.67:0/3045421010 learned_addr learned my addr 172.21.15.67:0/3045421010 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.931+0000 7f06fbe00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06fc0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.931+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226633044 0 0) 0x7f06fc0cd4a0 con 0x7f06fc141ed0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.931+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e0003520 con 0x7f06fc141ed0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1073030015 0 0) 0x7f06e0003520 con 0x7f06fc141ed0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 msgr2=0x7f06fc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06fc0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 >> v1:172.21.15.187:6789/0 conn(0x7f06fc160280 legacy=0x7f06fc06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06fc160ad0 con 0x7f06fc141ed0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f06f40019f0 con 0x7f06fc141ed0 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 >> v1:172.21.15.67:6789/0 conn(0x7f06fc141ed0 legacy=0x7f06fc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06e0003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0003fd0 0x7f06e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06fc15ce30 con 0x7f06e0004d40 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06e000bb20 con 0x7f06e0003fd0 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e000bd20 con 0x7f06fc06b8c0 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.932+0000 7f0700a00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f06fbe00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06e0003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f0700a00640 1 -- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 msgr2=0x7f06e0003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f0700a00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06e0003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f0700a00640 1 -- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0003fd0 msgr2=0x7f06e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f0700a00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e0003fd0 0x7f06e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f0700a00640 1 -- 172.21.15.67:0/3045421010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e0003520 con 0x7f06e0004d40 2024-04-17T07:27:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.933+0000 7f06fbe00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc06b8c0 0x7f06e0003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.934+0000 7f0700a00640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f06f400a850 tx=0x7f06e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2947d285af7dd8bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.934+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f400a4a0 con 0x7f06e0004d40 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.934+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f4034650 con 0x7f06e0004d40 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.934+0000 7f06fb400640 1 -- 172.21.15.67:0/3045421010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4034a80 con 0x7f06e0004d40 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 -- 172.21.15.67:0/3045421010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 msgr2=0x7f06e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 --2- 172.21.15.67:0/3045421010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f06f400a850 tx=0x7f06e0005830 comp rx=0 tx=0).stop 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0700a00640 1 -- 172.21.15.67:0/3045421010 reap_dead start 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 -- 172.21.15.67:0/3045421010 shutdown_connections 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 -- 172.21.15.67:0/3045421010 >> 172.21.15.67:0/3045421010 conn(0x7f06fc067d10 msgr2=0x7f06fc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 -- 172.21.15.67:0/3045421010 shutdown_connections 2024-04-17T07:27:04.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.935+0000 7f0701400640 1 -- 172.21.15.67:0/3045421010 wait complete. 2024-04-17T07:27:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.937+0000 7f0701400640 1 Processor -- start 2024-04-17T07:27:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.937+0000 7f0701400640 1 -- start start 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06fc13ef20 0x7f06fc06e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc15b6a0 0x7f06fc155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06fc13f9b0 con 0x7f06e0004d40 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06fc161090 con 0x7f06fc13ef20 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0701400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06fc13fb30 con 0x7f06fc15b6a0 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33336/0 (socket says 172.21.15.67:33336) 2024-04-17T07:27:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 -- 172.21.15.67:0/362742123 learned_addr learned my addr 172.21.15.67:0/362742123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f0700a00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc15b6a0 0x7f06fc155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06f3e00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06fc13ef20 0x7f06fc06e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 -- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc15b6a0 msgr2=0x7f06fc155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc15b6a0 0x7f06fc155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.938+0000 7f06fbe00640 1 -- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06fc13ef20 msgr2=0x7f06fc06e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06fbe00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06fc13ef20 0x7f06fc06e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06fbe00640 1 -- 172.21.15.67:0/362742123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06fc06e6b0 con 0x7f06e0004d40 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06fbe00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0108b0 tx=0x7f06ec010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00f040 con 0x7f06e0004d40 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ec00eb60 con 0x7f06e0004d40 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec01c400 con 0x7f06e0004d40 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06fc06e930 con 0x7f06e0004d40 2024-04-17T07:27:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.939+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06fc06eef0 con 0x7f06e0004d40 2024-04-17T07:27:04.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.940+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06fc068a10 con 0x7f06e0004d40 2024-04-17T07:27:04.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.941+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f06ec01c5a0 con 0x7f06e0004d40 2024-04-17T07:27:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.942+0000 7f06f8c00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 0x7f06d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.942+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f06ec08c660 con 0x7f06e0004d40 2024-04-17T07:27:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.943+0000 7f06f3e00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 0x7f06d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.944+0000 7f06f3e00640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 0x7f06d806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0b5ba0 tx=0x7f06e4009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:04.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:04.947+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f06fc06e6b0 con 0x7f06e0004d40 2024-04-17T07:27:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.224+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7f06fc0b0150 con 0x7f06e0004d40 2024-04-17T07:27:05.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.731+0000 7f06f8c00640 1 -- 172.21.15.67:0/362742123 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v478) ==== 158+0+0 (secure 0 0 0) 0x7f06ec018030 con 0x7f06e0004d40 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 msgr2=0x7f06d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 0x7f06d806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0b5ba0 tx=0x7f06e4009040 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 msgr2=0x7f06fc06f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0108b0 tx=0x7f06ec010d70 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 shutdown_connections 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f06d806ca10 0x7f06d806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.734+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06fc15b6a0 0x7f06fc155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.735+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06fc13ef20 0x7f06fc06e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.735+0000 7f0701400640 1 --2- 172.21.15.67:0/362742123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06e0004d40 0x7f06fc06f9d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.735+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 >> 172.21.15.67:0/362742123 conn(0x7f06fc067d10 msgr2=0x7f06fc140b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.735+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 shutdown_connections 2024-04-17T07:27:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.735+0000 7f0701400640 1 -- 172.21.15.67:0/362742123 wait complete. 2024-04-17T07:27:05.758 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-04-17T07:27:05.758 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-04-17T07:27:05.758 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:27:05.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:27:05.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.977+0000 7f28f3000640 1 Processor -- start 2024-04-17T07:27:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.978+0000 7f28f3000640 1 -- start start 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec15b570 0x7f28ec06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f28ec06fce0 0x7f28ec06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28ec06be70 0x7f28ec06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f28ec15bc80 con 0x7f28ec06be70 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f28ec15bf40 con 0x7f28ec06fce0 2024-04-17T07:27:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.979+0000 7f28f3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec15c220 con 0x7f28ec15b570 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28ec06be70 0x7f28ec06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28ec06be70 0x7f28ec06ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33346/0 (socket says 172.21.15.67:33346) 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 -- 172.21.15.67:0/1472715860 learned_addr learned my addr 172.21.15.67:0/1472715860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f1c00640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec15b570 0x7f28ec06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f1200640 1 --2- 172.21.15.67:0/1472715860 >> v2:172.21.15.187:3300/0 conn(0x7f28ec06fce0 0x7f28ec06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 -- 172.21.15.67:0/1472715860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec15b570 msgr2=0x7f28ec06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec15b570 0x7f28ec06f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 -- 172.21.15.67:0/1472715860 >> v2:172.21.15.187:3300/0 conn(0x7f28ec06fce0 msgr2=0x7f28ec06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 --2- 172.21.15.67:0/1472715860 >> v2:172.21.15.187:3300/0 conn(0x7f28ec06fce0 0x7f28ec06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.980+0000 7f28f2600640 1 -- 172.21.15.67:0/1472715860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec15c500 con 0x7f28ec06be70 2024-04-17T07:27:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.982+0000 7f28f2600640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec06be70 0x7f28ec06ddf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0b3250 tx=0x7f28e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4df1519904b804d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.982+0000 7f28ebe00640 1 -- 172.21.15.67:0/1472715860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e402c9e0 con 0x7f28ec06be70 2024-04-17T07:27:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.982+0000 7f28ebe00640 1 -- 172.21.15.67:0/1472715860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e402cb80 con 0x7f28ec06be70 2024-04-17T07:27:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.982+0000 7f28ebe00640 1 -- 172.21.15.67:0/1472715860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40343f0 con 0x7f28ec06be70 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 -- 172.21.15.67:0/1472715860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec06be70 msgr2=0x7f28ec06ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec06be70 0x7f28ec06ddf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0b3250 tx=0x7f28e402be40 comp rx=0 tx=0).stop 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 -- 172.21.15.67:0/1472715860 shutdown_connections 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec06be70 0x7f28ec06ddf0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 --2- 172.21.15.67:0/1472715860 >> v2:172.21.15.187:3300/0 conn(0x7f28ec06fce0 0x7f28ec06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 --2- 172.21.15.67:0/1472715860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec15b570 0x7f28ec06f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 -- 172.21.15.67:0/1472715860 >> 172.21.15.67:0/1472715860 conn(0x7f28ec067d10 msgr2=0x7f28ec0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 -- 172.21.15.67:0/1472715860 shutdown_connections 2024-04-17T07:27:05.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.983+0000 7f28f3000640 1 -- 172.21.15.67:0/1472715860 wait complete. 2024-04-17T07:27:05.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.985+0000 7f28f3000640 1 Processor -- start 2024-04-17T07:27:05.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.985+0000 7f28f3000640 1 -- start start 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ec06be70 0x7f28ec1f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec06fce0 0x7f28ec15a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec06c460 con 0x7f28ec15b570 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec06c2e0 con 0x7f28ec06be70 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec0b47b0 con 0x7f28ec06fce0 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33360/0 (socket says 172.21.15.67:33360) 2024-04-17T07:27:05.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.986+0000 7f28f2600640 1 -- 172.21.15.67:0/2879737415 learned_addr learned my addr 172.21.15.67:0/2879737415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f1200640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec06fce0 0x7f28ec15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f1c00640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ec06be70 0x7f28ec1f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 -- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec06fce0 msgr2=0x7f28ec15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec06fce0 0x7f28ec15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 -- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ec06be70 msgr2=0x7f28ec1f7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ec06be70 0x7f28ec1f7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 -- 172.21.15.67:0/2879737415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec1f85d0 con 0x7f28ec15b570 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.987+0000 7f28f2600640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f28e4036040 tx=0x7f28e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.988+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4043020 con 0x7f28ec15b570 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.988+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e403e450 con 0x7f28ec15b570 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.988+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4009510 con 0x7f28ec15b570 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.988+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec1f88b0 con 0x7f28ec15b570 2024-04-17T07:27:05.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.988+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec159b10 con 0x7f28ec15b570 2024-04-17T07:27:05.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.989+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ec068a10 con 0x7f28ec15b570 2024-04-17T07:27:05.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.990+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f28e4046020 con 0x7f28ec15b570 2024-04-17T07:27:05.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.991+0000 7f28e9600640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 0x7f28cc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:05.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.991+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f28e40ae600 con 0x7f28ec15b570 2024-04-17T07:27:05.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.992+0000 7f28f1c00640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 0x7f28cc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:05.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.993+0000 7f28f1c00640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 0x7f28cc06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28e0006fd0 tx=0x7f28e0008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:05.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:05.995+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f28e404d030 con 0x7f28ec15b570 2024-04-17T07:27:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.201+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28ec0b5810 con 0x7f28ec15b570 2024-04-17T07:27:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.202+0000 7f28e9600640 1 -- 172.21.15.67:0/2879737415 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+14594 (secure 0 0 0) 0x7f28e407c870 con 0x7f28ec15b570 2024-04-17T07:27:06.203 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:06.203 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:27:03.280948+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/504171556":"2024-04-18T07:27:00.683486+0000","172.21.15.67:6833/504171556":"2024-04-18T07:27:00.683486+0000","172.21.15.187:0/2322153279":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1982372125":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1699979613":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:6833/3291261531":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/618455126":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/3485602680":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/2778463946":"2024-04-18T07:15:31.130810+0000","172.21.15.187:0/1528295426":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6833/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:6832/2964624313":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2513891628":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/14109251":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/1536403455":"2024-04-18T07:26:56.792716+0000","172.21.15.187:0/2851019947":"2024-04-18T07:26:56.792716+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1},{"begin":8,"length":2},{"begin":12,"length":1},{"begin":15,"length":2},{"begin":21,"length":1},{"begin":23,"length":6},{"begin":30,"length":1},{"begin":32,"length":2},{"begin":36,"length":5},{"begin":43,"length":1},{"begin":45,"length":5},{"begin":51,"length":2},{"begin":54,"length":2},{"begin":57,"length":4},{"begin":62,"length":3},{"begin":68,"length":6},{"begin":75,"length":1},{"begin":77,"length":2},{"begin":80,"length":2},{"begin":84,"length":4},{"begin":89,"length":7},{"begin":97,"length":2},{"begin":100,"length":1},{"begin":103,"length":4},{"begin":108,"length":4},{"begin":144,"length":78}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 msgr2=0x7f28cc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 0x7f28cc06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28e0006fd0 tx=0x7f28e0008040 comp rx=0 tx=0).stop 2024-04-17T07:27:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 msgr2=0x7f28ec1e8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f28e4036040 tx=0x7f28e4004a40 comp rx=0 tx=0).stop 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 shutdown_connections 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f28cc06ca10 0x7f28cc06eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ec15b570 0x7f28ec1e8180 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ec06fce0 0x7f28ec15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 --2- 172.21.15.67:0/2879737415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ec06be70 0x7f28ec1f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 >> 172.21.15.67:0/2879737415 conn(0x7f28ec067d10 msgr2=0x7f28ec1575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 shutdown_connections 2024-04-17T07:27:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.205+0000 7f28f3000640 1 -- 172.21.15.67:0/2879737415 wait complete. 2024-04-17T07:27:06.225 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:27:06.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:27:06.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.444+0000 7fa94c800640 1 Processor -- start 2024-04-17T07:27:06.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.444+0000 7fa94c800640 1 -- start start 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa94c800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa9440b18f0 0x7fa9441679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa94c800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9441607a0 con 0x7fa944142e20 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa94c800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa944160a60 con 0x7fa944143200 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa94c800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa944160d40 con 0x7fa9440b18f0 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa949e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa944142e20 0x7fa9440cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42326/0 (socket says 172.21.15.67:42326) 2024-04-17T07:27:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.446+0000 7fa949e00640 1 -- 172.21.15.67:0/494101775 learned_addr learned my addr 172.21.15.67:0/494101775 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa94a800640 1 --2- 172.21.15.67:0/494101775 >> v2:172.21.15.187:3300/0 conn(0x7fa9440b18f0 0x7fa9441679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747556055 0 0) 0x7fa9441607a0 con 0x7fa944142e20 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa92c003520 con 0x7fa944142e20 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2826841854 0 0) 0x7fa92c003520 con 0x7fa944142e20 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 >> v2:172.21.15.187:3300/0 conn(0x7fa9440b18f0 msgr2=0x7fa9441679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 --2- 172.21.15.67:0/494101775 >> v2:172.21.15.187:3300/0 conn(0x7fa9440b18f0 0x7fa9441679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 >> v1:172.21.15.187:6789/0 conn(0x7fa944143200 legacy=0x7fa9440cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa944161020 con 0x7fa944142e20 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa938001670 con 0x7fa944142e20 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 >> v1:172.21.15.67:6789/0 conn(0x7fa944142e20 legacy=0x7fa9440cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c003a30 0x7fa92c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa92c004800 0x7fa92c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 0x7fa92c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa944160d40 con 0x7fa92c005230 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c00c190 con 0x7fa92c003a30 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.447+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa92c00c390 con 0x7fa92c004800 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 0x7fa92c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 -- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa92c004800 msgr2=0x7fa92c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa92c004800 0x7fa92c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 -- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c003a30 msgr2=0x7fa92c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa949e00640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c003a30 0x7fa92c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c003a30 0x7fa92c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa94a800640 1 -- 172.21.15.67:0/494101775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92c003520 con 0x7fa92c005230 2024-04-17T07:27:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.448+0000 7fa949e00640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c003a30 0x7fa92c003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.449+0000 7fa94a800640 1 -- 172.21.15.67:0/494101775 reap_dead start 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.449+0000 7fa94a800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 0x7fa92c00b9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa940009f90 tx=0x7fa92c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59559ba0a166525 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940009c10 con 0x7fa92c005230 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94002cc20 con 0x7fa92c005230 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa948a00640 1 -- 172.21.15.67:0/494101775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940003990 con 0x7fa92c005230 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 -- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 msgr2=0x7fa92c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 0x7fa92c00b9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa940009f90 tx=0x7fa92c005830 comp rx=0 tx=0).stop 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 -- 172.21.15.67:0/494101775 shutdown_connections 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 --2- 172.21.15.67:0/494101775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa92c005230 0x7fa92c00b9c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 -- 172.21.15.67:0/494101775 >> 172.21.15.67:0/494101775 conn(0x7fa94406b8c0 msgr2=0x7fa94406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 -- 172.21.15.67:0/494101775 shutdown_connections 2024-04-17T07:27:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.450+0000 7fa94c800640 1 -- 172.21.15.67:0/494101775 wait complete. 2024-04-17T07:27:06.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.452+0000 7fa94c800640 1 Processor -- start 2024-04-17T07:27:06.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.452+0000 7fa94c800640 1 -- start start 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 0x7fa944157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa94413d320 0x7fa94413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440cec20 con 0x7fa94413f260 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440ce730 con 0x7fa92c005230 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa94c800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9440ceda0 con 0x7fa94413d320 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.453+0000 7fa949400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33376/0 (socket says 172.21.15.67:33376) 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 -- 172.21.15.67:0/2973840016 learned_addr learned my addr 172.21.15.67:0/2973840016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa94a800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa94413d320 0x7fa94413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 -- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa94413d320 msgr2=0x7fa94413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949e00640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 0x7fa944157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa94413d320 0x7fa94413d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 -- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 msgr2=0x7fa944157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 0x7fa944157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 -- 172.21.15.67:0/2973840016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94413cbb0 con 0x7fa94413f260 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949e00640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 0x7fa944157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.454+0000 7fa949400640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa93400c420 tx=0x7fa93400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.455+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934017070 con 0x7fa94413f260 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.455+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9340052d0 con 0x7fa94413f260 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.455+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934003810 con 0x7fa94413f260 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.455+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94415d060 con 0x7fa94413f260 2024-04-17T07:27:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.455+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94413fcb0 con 0x7fa94413f260 2024-04-17T07:27:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.456+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa944068b90 con 0x7fa94413f260 2024-04-17T07:27:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.457+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fa93401a070 con 0x7fa94413f260 2024-04-17T07:27:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.458+0000 7fa93e000640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 0x7fa92806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.458+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7fa93408b1d0 con 0x7fa94413f260 2024-04-17T07:27:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.459+0000 7fa949e00640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 0x7fa92806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.460+0000 7fa949e00640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 0x7fa92806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa938007910 tx=0x7fa938009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.462+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa9340593c0 con 0x7fa94413f260 2024-04-17T07:27:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.671+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa94415c160 con 0x7fa94413f260 2024-04-17T07:27:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.672+0000 7fa93e000640 1 -- 172.21.15.67:0/2973840016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7fa934058cc0 con 0x7fa94413f260 2024-04-17T07:27:06.672 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.674+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 msgr2=0x7fa92806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.674+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 0x7fa92806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa938007910 tx=0x7fa938009040 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.674+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 msgr2=0x7fa94413c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.674+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa93400c420 tx=0x7fa93400c8e0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 shutdown_connections 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa92806ca10 0x7fa92806eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa94413d320 0x7fa94413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa94413f260 0x7fa94413c670 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 --2- 172.21.15.67:0/2973840016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa92c005230 0x7fa944157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 >> 172.21.15.67:0/2973840016 conn(0x7fa94406b8c0 msgr2=0x7fa94415c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 shutdown_connections 2024-04-17T07:27:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.675+0000 7fa94c800640 1 -- 172.21.15.67:0/2973840016 wait complete. 2024-04-17T07:27:06.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-17T07:27:06.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.915+0000 7fb422600640 1 Processor -- start 2024-04-17T07:27:06.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.915+0000 7fb422600640 1 -- start start 2024-04-17T07:27:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb422600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb41c15c5a0 0x7fb41c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb422600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb41c0b4210 0x7fb41c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb422600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41c15f440 con 0x7fb41c0b46c0 2024-04-17T07:27:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb422600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb41c0af0a0 con 0x7fb41c15c5a0 2024-04-17T07:27:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb422600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb41c15f720 con 0x7fb41c0b4210 2024-04-17T07:27:06.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb421200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb41c15c5a0 0x7fb41c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb421200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb41c15c5a0 0x7fb41c15ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33386/0 (socket says 172.21.15.67:33386) 2024-04-17T07:27:06.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb421c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb41c0b46c0 0x7fb41c071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42342/0 (socket says 172.21.15.67:42342) 2024-04-17T07:27:06.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.917+0000 7fb421200640 1 -- 172.21.15.67:0/4012723783 learned_addr learned my addr 172.21.15.67:0/4012723783 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:06.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb41be00640 1 --2- 172.21.15.67:0/4012723783 >> v2:172.21.15.187:3300/0 conn(0x7fb41c0b4210 0x7fb41c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb421200640 1 -- 172.21.15.67:0/4012723783 >> v2:172.21.15.187:3300/0 conn(0x7fb41c0b4210 msgr2=0x7fb41c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb421200640 1 --2- 172.21.15.67:0/4012723783 >> v2:172.21.15.187:3300/0 conn(0x7fb41c0b4210 0x7fb41c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb421200640 1 -- 172.21.15.67:0/4012723783 >> v1:172.21.15.67:6789/0 conn(0x7fb41c0b46c0 legacy=0x7fb41c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb421200640 1 -- 172.21.15.67:0/4012723783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c15fa00 con 0x7fb41c15c5a0 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.918+0000 7fb41b400640 1 -- 172.21.15.67:0/4012723783 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460785461 0 0) 0x7fb41c15f440 con 0x7fb41c0b46c0 2024-04-17T07:27:06.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.919+0000 7fb421200640 1 --2- 172.21.15.67:0/4012723783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c15ee60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb41c073150 tx=0x7fb41002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcb5659fdf9d5f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.919+0000 7fb41b400640 1 -- 172.21.15.67:0/4012723783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41002e070 con 0x7fb41c15c5a0 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.919+0000 7fb41b400640 1 -- 172.21.15.67:0/4012723783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb410037070 con 0x7fb41c15c5a0 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.919+0000 7fb41b400640 1 -- 172.21.15.67:0/4012723783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410031a00 con 0x7fb41c15c5a0 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 -- 172.21.15.67:0/4012723783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 msgr2=0x7fb41c15ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 --2- 172.21.15.67:0/4012723783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c15ee60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb41c073150 tx=0x7fb41002b880 comp rx=0 tx=0).stop 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 -- 172.21.15.67:0/4012723783 shutdown_connections 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 --2- 172.21.15.67:0/4012723783 >> v2:172.21.15.187:3300/0 conn(0x7fb41c0b4210 0x7fb41c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 --2- 172.21.15.67:0/4012723783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c15ee60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 -- 172.21.15.67:0/4012723783 >> 172.21.15.67:0/4012723783 conn(0x7fb41c067d10 msgr2=0x7fb41c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 -- 172.21.15.67:0/4012723783 shutdown_connections 2024-04-17T07:27:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.920+0000 7fb422600640 1 -- 172.21.15.67:0/4012723783 wait complete. 2024-04-17T07:27:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.922+0000 7fb422600640 1 Processor -- start 2024-04-17T07:27:06.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.922+0000 7fb422600640 1 -- start start 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41c0b4210 0x7fb41c1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb41c15db00 0x7fb41c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c0adc30 con 0x7fb41c15c5a0 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c0adab0 con 0x7fb41c15db00 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb422600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb41c06e670 con 0x7fb41c0b4210 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb41be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb41be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33394/0 (socket says 172.21.15.67:33394) 2024-04-17T07:27:06.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.923+0000 7fb41be00640 1 -- 172.21.15.67:0/3235354197 learned_addr learned my addr 172.21.15.67:0/3235354197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb421200640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41c0b4210 0x7fb41c1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 -- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41c0b4210 msgr2=0x7fb41c1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41c0b4210 0x7fb41c1560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 -- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb41c15db00 msgr2=0x7fb41c15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb41c15db00 0x7fb41c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 -- 172.21.15.67:0/3235354197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c154540 con 0x7fb41c15c5a0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb41be00640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb40800d830 tx=0x7fb40800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40800b6c0 con 0x7fb41c15c5a0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb408019070 con 0x7fb41c15c5a0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40800ba80 con 0x7fb41c15c5a0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.924+0000 7fb422600640 1 -- 172.21.15.67:0/3235354197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c1555e0 con 0x7fb41c15c5a0 2024-04-17T07:27:06.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.925+0000 7fb422600640 1 -- 172.21.15.67:0/3235354197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c155ba0 con 0x7fb41c15c5a0 2024-04-17T07:27:06.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.926+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41c068a10 con 0x7fb41c15c5a0 2024-04-17T07:27:06.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.927+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fb40801e460 con 0x7fb41c15c5a0 2024-04-17T07:27:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.928+0000 7fb418c00640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 0x7fb3fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:06.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.928+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7fb40808c550 con 0x7fb41c15c5a0 2024-04-17T07:27:06.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.929+0000 7fb421200640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 0x7fb3fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:06.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.930+0000 7fb421200640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 0x7fb3fc06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb410036000 tx=0x7fb410008030 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:06.933+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb40805a740 con 0x7fb41c15c5a0 2024-04-17T07:27:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.211+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7fb41c0d0050 con 0x7fb41c15c5a0 2024-04-17T07:27:07.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.737+0000 7fb418c00640 1 -- 172.21.15.67:0/3235354197 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v480) ==== 157+0+0 (secure 0 0 0) 0x7fb40805a040 con 0x7fb41c15c5a0 2024-04-17T07:27:07.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 msgr2=0x7fb3fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:07.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 0x7fb3fc06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb410036000 tx=0x7fb410008030 comp rx=0 tx=0).stop 2024-04-17T07:27:07.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 msgr2=0x7fb41c149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:07.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb40800d830 tx=0x7fb40800dcf0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 shutdown_connections 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb3fc06ca10 0x7fb3fc06eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb41c15db00 0x7fb41c15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb41c15c5a0 0x7fb41c149d70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 --2- 172.21.15.67:0/3235354197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41c0b4210 0x7fb41c1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 >> 172.21.15.67:0/3235354197 conn(0x7fb41c067d10 msgr2=0x7fb41c0d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.740+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 shutdown_connections 2024-04-17T07:27:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.741+0000 7fb40e000640 1 -- 172.21.15.67:0/3235354197 wait complete. 2024-04-17T07:27:07.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:27:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.991+0000 7f2b10a00640 1 Processor -- start 2024-04-17T07:27:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.992+0000 7f2b10a00640 1 -- start start 2024-04-17T07:27:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.993+0000 7f2b10a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b0c0cd920 0x7f2b0c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.993+0000 7f2b10a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2b0c06c0a0 0x7f2b0c06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.993+0000 7f2b10a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b0c146360 con 0x7f2b0c145f80 2024-04-17T07:27:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b10a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0c0727a0 con 0x7f2b0c0cd920 2024-04-17T07:27:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b10a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0c071e70 con 0x7f2b0c06c0a0 2024-04-17T07:27:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b0c0cd920 0x7f2b0c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b0c0cd920 0x7f2b0c06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33410/0 (socket says 172.21.15.67:33410) 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 -- 172.21.15.67:0/3685940146 learned_addr learned my addr 172.21.15.67:0/3685940146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b0b200640 1 --2- 172.21.15.67:0/3685940146 >> v2:172.21.15.187:3300/0 conn(0x7f2b0c06c0a0 0x7f2b0c06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 -- 172.21.15.67:0/3685940146 >> v2:172.21.15.187:3300/0 conn(0x7f2b0c06c0a0 msgr2=0x7f2b0c06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 --2- 172.21.15.67:0/3685940146 >> v2:172.21.15.187:3300/0 conn(0x7f2b0c06c0a0 0x7f2b0c06e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 -- 172.21.15.67:0/3685940146 >> v1:172.21.15.187:6789/0 conn(0x7f2b0c145f80 legacy=0x7f2b0c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.994+0000 7f2b03e00640 1 -- 172.21.15.67:0/3685940146 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c072150 con 0x7f2b0c0cd920 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.996+0000 7f2b03e00640 1 --2- 172.21.15.67:0/3685940146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c06bb60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c15cc90 tx=0x7f2af402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae0b836033bae7d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.996+0000 7f2b09e00640 1 -- 172.21.15.67:0/3685940146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af402e070 con 0x7f2b0c0cd920 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.996+0000 7f2b09e00640 1 -- 172.21.15.67:0/3685940146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af4037070 con 0x7f2b0c0cd920 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.996+0000 7f2b09e00640 1 -- 172.21.15.67:0/3685940146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af4031a00 con 0x7f2b0c0cd920 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 -- 172.21.15.67:0/3685940146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c0cd920 msgr2=0x7f2b0c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 --2- 172.21.15.67:0/3685940146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c06bb60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c15cc90 tx=0x7f2af402b880 comp rx=0 tx=0).stop 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 -- 172.21.15.67:0/3685940146 shutdown_connections 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 --2- 172.21.15.67:0/3685940146 >> v2:172.21.15.187:3300/0 conn(0x7f2b0c06c0a0 0x7f2b0c06e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 --2- 172.21.15.67:0/3685940146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c06bb60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 -- 172.21.15.67:0/3685940146 >> 172.21.15.67:0/3685940146 conn(0x7f2b0c0b2190 msgr2=0x7f2b0c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 -- 172.21.15.67:0/3685940146 shutdown_connections 2024-04-17T07:27:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.997+0000 7f2b10a00640 1 -- 172.21.15.67:0/3685940146 wait complete. 2024-04-17T07:27:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.999+0000 7f2b10a00640 1 Processor -- start 2024-04-17T07:27:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:07.999+0000 7f2b10a00640 1 -- start start 2024-04-17T07:27:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c1e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b0c145f80 0x7f2b0c1f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c072c30 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c06c530 con 0x7f2b0c0cd920 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b10a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0c072db0 con 0x7f2b0c145f80 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33414/0 (socket says 172.21.15.67:33414) 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 -- 172.21.15.67:0/592400504 learned_addr learned my addr 172.21.15.67:0/592400504 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 -- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b0c145f80 msgr2=0x7f2b0c1f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b0c145f80 0x7f2b0c1f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 -- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b0c0cd920 msgr2=0x7f2b0c1e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c1e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.000+0000 7f2b0a800640 1 -- 172.21.15.67:0/592400504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c1e5770 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b0a800640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00b420 tx=0x7f2afc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc011830 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc016070 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc011c10 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c0b4120 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.001+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c0b43e0 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.002+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b0c068a10 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.003+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f2afc01b460 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.004+0000 7f2b02a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 0x7f2ae806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.004+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f2afc08b910 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.005+0000 7f2b03e00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 0x7f2ae806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.006+0000 7f2b03e00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 0x7f2ae806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2af4036000 tx=0x7f2af4030000 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.009+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2afc020020 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.204+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f2b0c0b2780 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.210+0000 7f2b02a00640 1 -- 172.21.15.67:0/592400504 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f2afc059a70 con 0x7f2b0c06c0a0 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.212+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 msgr2=0x7f2ae806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.212+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 0x7f2ae806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2af4036000 tx=0x7f2af4030000 comp rx=0 tx=0).stop 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 msgr2=0x7f2b0c140140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00b420 tx=0x7f2afc00b8e0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 shutdown_connections 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f2ae806ca10 0x7f2ae806eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b0c145f80 0x7f2b0c1f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b0c0cd920 0x7f2b0c1e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 --2- 172.21.15.67:0/592400504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0c06c0a0 0x7f2b0c140140 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 >> 172.21.15.67:0/592400504 conn(0x7f2b0c0b2190 msgr2=0x7f2b0c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 shutdown_connections 2024-04-17T07:27:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.213+0000 7f2b10a00640 1 -- 172.21.15.67:0/592400504 wait complete. 2024-04-17T07:27:08.231 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:27:08.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:27:08.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:27:08.470 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:27:08.470 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:27:08.470 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:27:08.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:27:08.532 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:27:08.532 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:27:08.588 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:27:08.588 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:27:08.589 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:27:08.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:27:08.652 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:27:08.652 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:27:08.708 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:27:08.709 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:27:08.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:27:08.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.936+0000 7f2745200640 1 Processor -- start 2024-04-17T07:27:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.936+0000 7f2745200640 1 -- start start 2024-04-17T07:27:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f2745200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f274006c040 0x7f27401664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f2745200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f274006c450 con 0x7f274015b570 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f2745200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f274015bc80 con 0x7f274006feb0 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f2745200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f274015bf40 con 0x7f274006c040 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f273f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f274015b570 0x7f274006f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42344/0 (socket says 172.21.15.67:42344) 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.938+0000 7f273f400640 1 -- 172.21.15.67:0/3017067532 learned_addr learned my addr 172.21.15.67:0/3017067532 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.939+0000 7f273fe00640 1 --2- 172.21.15.67:0/3017067532 >> v2:172.21.15.67:3300/0 conn(0x7f274006c040 0x7f27401664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.939+0000 7f273e000640 1 -- 172.21.15.67:0/3017067532 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747333540 0 0) 0x7f274006c450 con 0x7f274015b570 2024-04-17T07:27:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.939+0000 7f273fe00640 1 -- 172.21.15.67:0/3017067532 >> v1:172.21.15.187:6789/0 conn(0x7f274006feb0 legacy=0x7f274006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.939+0000 7f273fe00640 1 -- 172.21.15.67:0/3017067532 >> v1:172.21.15.67:6789/0 conn(0x7f274015b570 legacy=0x7f274006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.939+0000 7f273fe00640 1 -- 172.21.15.67:0/3017067532 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274015c220 con 0x7f274006c040 2024-04-17T07:27:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.940+0000 7f273fe00640 1 --2- 172.21.15.67:0/3017067532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006c040 0x7f27401664d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27380097a0 tx=0x7f273802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d463f9898d9ae898 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f273e000640 1 -- 172.21.15.67:0/3017067532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738035020 con 0x7f274006c040 2024-04-17T07:27:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f273e000640 1 -- 172.21.15.67:0/3017067532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2738031830 con 0x7f274006c040 2024-04-17T07:27:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f273e000640 1 -- 172.21.15.67:0/3017067532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738031c00 con 0x7f274006c040 2024-04-17T07:27:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 -- 172.21.15.67:0/3017067532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006c040 msgr2=0x7f27401664d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 --2- 172.21.15.67:0/3017067532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006c040 0x7f27401664d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27380097a0 tx=0x7f273802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 -- 172.21.15.67:0/3017067532 shutdown_connections 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 --2- 172.21.15.67:0/3017067532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006c040 0x7f27401664d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 -- 172.21.15.67:0/3017067532 >> 172.21.15.67:0/3017067532 conn(0x7f2740067d10 msgr2=0x7f27400b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 -- 172.21.15.67:0/3017067532 shutdown_connections 2024-04-17T07:27:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.941+0000 7f2745200640 1 -- 172.21.15.67:0/3017067532 wait complete. 2024-04-17T07:27:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.943+0000 7f2745200640 1 Processor -- start 2024-04-17T07:27:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.943+0000 7f2745200640 1 -- start start 2024-04-17T07:27:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f274006c040 0x7f2740146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f274015b570 0x7f27400ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27400b47b0 con 0x7f274006feb0 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f274015b950 con 0x7f274015b570 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.944+0000 7f2745200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27400b4930 con 0x7f274006c040 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33436/0 (socket says 172.21.15.67:33436) 2024-04-17T07:27:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 -- 172.21.15.67:0/2865210821 learned_addr learned my addr 172.21.15.67:0/2865210821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273f400640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f274006c040 0x7f2740146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273fe00640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f274015b570 0x7f27400ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 -- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f274006c040 msgr2=0x7f2740146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f274006c040 0x7f2740146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 -- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f274015b570 msgr2=0x7f27400ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f274015b570 0x7f27400ce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 -- 172.21.15.67:0/2865210821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27400af760 con 0x7f274006feb0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.945+0000 7f273ea00640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f272800c430 tx=0x7f272800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.946+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27280052d0 con 0x7f274006feb0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.946+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2728005490 con 0x7f274006feb0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.946+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728007260 con 0x7f274006feb0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.946+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274014e570 con 0x7f274006feb0 2024-04-17T07:27:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.946+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274014eb30 con 0x7f274006feb0 2024-04-17T07:27:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.947+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2740068a10 con 0x7f274006feb0 2024-04-17T07:27:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.948+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f2728007400 con 0x7f274006feb0 2024-04-17T07:27:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.949+0000 7f2737400640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 0x7f271c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.949+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f272808b400 con 0x7f274006feb0 2024-04-17T07:27:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.950+0000 7f273f400640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 0x7f271c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.951+0000 7f273f400640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 0x7f271c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:08.955+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2728059550 con 0x7f274006feb0 2024-04-17T07:27:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.199+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27400b21e0 con 0x7f274006feb0 2024-04-17T07:27:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.200+0000 7f2737400640 1 -- 172.21.15.67:0/2865210821 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3866 (secure 0 0 0) 0x7f2728058e50 con 0x7f274006feb0 2024-04-17T07:27:09.201 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:09.201 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/688666051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":688666051},{"type":"v1","addr":"172.21.15.187:6836","nonce":688666051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4786,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/271989184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":271989184},{"type":"v1","addr":"172.21.15.187:6837","nonce":271989184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/381338418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":381338418},{"type":"v1","addr":"172.21.15.67:6834","nonce":381338418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5835,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2204656627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2204656627},{"type":"v1","addr":"172.21.15.67:6835","nonce":2204656627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-04-17T07:27:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 msgr2=0x7f271c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 0x7f271c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2730007fd0 tx=0x7f2730009040 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 msgr2=0x7f27401457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f272800c430 tx=0x7f272800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 shutdown_connections 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f271c06ca10 0x7f271c06eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f274015b570 0x7f27400ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274006feb0 0x7f27401457c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 --2- 172.21.15.67:0/2865210821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f274006c040 0x7f2740146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 >> 172.21.15.67:0/2865210821 conn(0x7f2740067d10 msgr2=0x7f27400b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 shutdown_connections 2024-04-17T07:27:09.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.204+0000 7f2745200640 1 -- 172.21.15.67:0/2865210821 wait complete. 2024-04-17T07:27:09.225 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:27:09.225 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:27:09.225 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:27:09.225 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:27:09.226 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:27:09.226 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:27:09.226 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:27:09.226 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:27:09.226 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:27:09.226 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:27:09.227 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:27:09.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:27:09.227 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:27:09.227 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:27:09.227 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:27:09.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:27:09.247 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:27:09.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:27:09.250 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:27:09.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:27:09.252 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:27:09.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:27:09.254 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:27:09.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:27:09.298 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:27:09.298 DEBUG:teuthology.parallel:result is None 2024-04-17T07:27:09.301 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:27:09.302 DEBUG:teuthology.parallel:result is None 2024-04-17T07:27:09.304 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:27:09.305 DEBUG:teuthology.parallel:result is None 2024-04-17T07:27:09.307 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:27:09.308 DEBUG:teuthology.parallel:result is None 2024-04-17T07:27:09.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:27:09.455 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:27:09.465 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:27:09.465 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:27:09.479 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:27:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.498+0000 7f89b8600640 1 Processor -- start 2024-04-17T07:27:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.499+0000 7f89b8600640 1 -- start start 2024-04-17T07:27:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b8600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b00d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b8600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b0163860 con 0x7f89b00cfa10 2024-04-17T07:27:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b8600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b00d1d40 con 0x7f89b00d17a0 2024-04-17T07:27:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b8600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89b00d2c30 con 0x7f89b015eca0 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b5200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89b00cfa10 0x7f89b00d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42388/0 (socket says 172.21.15.67:42388) 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.500+0000 7f89b5200640 1 -- 172.21.15.67:0/3271455166 learned_addr learned my addr 172.21.15.67:0/3271455166 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89b5c00640 1 --2- 172.21.15.67:0/3271455166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b00d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3727972112 0 0) 0x7f89b0163860 con 0x7f89b00cfa10 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8998003520 con 0x7f89b00cfa10 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881275807 0 0) 0x7f89b00d1d40 con 0x7f89b00d17a0 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89b0163860 con 0x7f89b00d17a0 2024-04-17T07:27:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2248810821 0 0) 0x7f8998003520 con 0x7f89b00cfa10 2024-04-17T07:27:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89b5c00640 1 -- 172.21.15.67:0/3271455166 >> v1:172.21.15.187:6789/0 conn(0x7f89b00d17a0 legacy=0x7f89b00b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89b5c00640 1 -- 172.21.15.67:0/3271455166 >> v1:172.21.15.67:6789/0 conn(0x7f89b00cfa10 legacy=0x7f89b00d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.501+0000 7f89b5c00640 1 -- 172.21.15.67:0/3271455166 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b00d2f10 con 0x7f89b015eca0 2024-04-17T07:27:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.502+0000 7f89b5c00640 1 --2- 172.21.15.67:0/3271455166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b00d39f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097d0 tx=0x7f89a402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b364130610ac56ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4037070 con 0x7f89b015eca0 2024-04-17T07:27:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4030510 con 0x7f89b015eca0 2024-04-17T07:27:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89afe00640 1 -- 172.21.15.67:0/3271455166 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c001670 con 0x7f89b015eca0 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 -- 172.21.15.67:0/3271455166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 msgr2=0x7f89b00d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 --2- 172.21.15.67:0/3271455166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b00d39f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097d0 tx=0x7f89a402b800 comp rx=0 tx=0).stop 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 -- 172.21.15.67:0/3271455166 shutdown_connections 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 --2- 172.21.15.67:0/3271455166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b00d39f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 -- 172.21.15.67:0/3271455166 >> 172.21.15.67:0/3271455166 conn(0x7f89b0067600 msgr2=0x7f89b00679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.503+0000 7f89b8600640 1 -- 172.21.15.67:0/3271455166 shutdown_connections 2024-04-17T07:27:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.504+0000 7f89b8600640 1 -- 172.21.15.67:0/3271455166 wait complete. 2024-04-17T07:27:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.505+0000 7f89b8600640 1 Processor -- start 2024-04-17T07:27:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.505+0000 7f89b8600640 1 -- start start 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 0x7f89b014e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89b015f080 con 0x7f89b00cfa10 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89b00d34d0 con 0x7f89b00d17a0 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.506+0000 7f89b8600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89b015f200 con 0x7f89b015eca0 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33474/0 (socket says 172.21.15.67:33474) 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 -- 172.21.15.67:0/897553848 learned_addr learned my addr 172.21.15.67:0/897553848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5200640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 0x7f89b014e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b6600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 -- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 msgr2=0x7f89b014e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b014e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 -- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 msgr2=0x7f89b014e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 0x7f89b014e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 -- 172.21.15.67:0/897553848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b00cb340 con 0x7f89b00cfa10 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5200640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 0x7f89b014e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.507+0000 7f89b5c00640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.508+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4037040 con 0x7f89b00cfa10 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.508+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a4004250 con 0x7f89b00cfa10 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.508+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a403c670 con 0x7f89b00cfa10 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.508+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b00c8680 con 0x7f89b00cfa10 2024-04-17T07:27:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.508+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b00c8920 con 0x7f89b00cfa10 2024-04-17T07:27:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.509+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b0068ca0 con 0x7f89b00cfa10 2024-04-17T07:27:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.510+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f89a403a070 con 0x7f89b00cfa10 2024-04-17T07:27:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.511+0000 7f89ad600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 0x7f899006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.511+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6032+0+0 (secure 0 0 0) 0x7f89a40ad8e0 con 0x7f89b00cfa10 2024-04-17T07:27:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.512+0000 7f89b5200640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 0x7f899006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.512+0000 7f89b5200640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 0x7f899006eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f899c001eb0 tx=0x7f899c009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.515+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f89a4041020 con 0x7f89b00cfa10 2024-04-17T07:27:09.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.734+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f89b00d2230 con 0x7f89b00cfa10 2024-04-17T07:27:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.766+0000 7f89ad600640 1 -- 172.21.15.67:0/897553848 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v59) ==== 97+0+0 (secure 0 0 0) 0x7f89a407b930 con 0x7f89b00cfa10 2024-04-17T07:27:09.766 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:27:09.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 msgr2=0x7f899006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 0x7f899006eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f899c001eb0 tx=0x7f899c009040 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 msgr2=0x7f89b0149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a40046c0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 shutdown_connections 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f899006ca10 0x7f899006eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89b015eca0 0x7f89b014e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89b00d17a0 0x7f89b014e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 --2- 172.21.15.67:0/897553848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00cfa10 0x7f89b0149950 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.769+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 >> 172.21.15.67:0/897553848 conn(0x7f89b0067600 msgr2=0x7f89b00cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.770+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 shutdown_connections 2024-04-17T07:27:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:09.770+0000 7f89b8600640 1 -- 172.21.15.67:0/897553848 wait complete. 2024-04-17T07:27:09.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:27:10.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.005+0000 7f1327600640 1 Processor -- start 2024-04-17T07:27:10.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.005+0000 7f1327600640 1 -- start start 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f1327600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13200cd040 0x7f13200cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f1327600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13200cd960 con 0x7f13200ce610 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f1327600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1320071f70 con 0x7f132015ed00 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f1327600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1320072250 con 0x7f13200cd040 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f131fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f13200ce610 0x7f13200b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42394/0 (socket says 172.21.15.67:42394) 2024-04-17T07:27:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.007+0000 7f131fe00640 1 -- 172.21.15.67:0/1916426848 learned_addr learned my addr 172.21.15.67:0/1916426848 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f1324c00640 1 --2- 172.21.15.67:0/1916426848 >> v2:172.21.15.187:3300/0 conn(0x7f13200cd040 0x7f13200cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315205412 0 0) 0x7f13200cd960 con 0x7f13200ce610 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1308003520 con 0x7f13200ce610 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1439555408 0 0) 0x7f1308003520 con 0x7f13200ce610 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 >> v2:172.21.15.187:3300/0 conn(0x7f13200cd040 msgr2=0x7f13200cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 --2- 172.21.15.67:0/1916426848 >> v2:172.21.15.187:3300/0 conn(0x7f13200cd040 0x7f13200cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 >> v1:172.21.15.187:6789/0 conn(0x7f132015ed00 legacy=0x7f13200b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1320072530 con 0x7f13200ce610 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f130c001670 con 0x7f13200ce610 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 >> v1:172.21.15.67:6789/0 conn(0x7f13200ce610 legacy=0x7f13200b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1308003a00 0x7f1308003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1308004800 0x7f1308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 0x7f130800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1320072250 con 0x7f13080051c0 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f130800c0c0 con 0x7f1308004800 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.008+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f130800c2c0 con 0x7f1308003a00 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.009+0000 7f1325600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 0x7f130800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.009+0000 7f1324c00640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1308003a00 0x7f1308003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.009+0000 7f131fe00640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1308004800 0x7f1308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.010+0000 7f1325600640 1 -- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1308003a00 msgr2=0x7f1308003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.010+0000 7f1325600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1308003a00 0x7f1308003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.010+0000 7f1325600640 1 -- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1308004800 msgr2=0x7f1308004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.010+0000 7f1325600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1308004800 0x7f1308004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.010+0000 7f1325600640 1 -- 172.21.15.67:0/1916426848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1308003520 con 0x7f13080051c0 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.011+0000 7f1325600640 1 -- 172.21.15.67:0/1916426848 reap_dead start 2024-04-17T07:27:10.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.011+0000 7f1325600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 0x7f130800b9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1318001e50 tx=0x7f1308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67090c476a1c257 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318001a70 con 0x7f13080051c0 2024-04-17T07:27:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1318003b90 con 0x7f13080051c0 2024-04-17T07:27:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f131f400640 1 -- 172.21.15.67:0/1916426848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180050d0 con 0x7f13080051c0 2024-04-17T07:27:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 -- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 msgr2=0x7f130800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 0x7f130800b9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1318001e50 tx=0x7f1308005830 comp rx=0 tx=0).stop 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 -- 172.21.15.67:0/1916426848 shutdown_connections 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 --2- 172.21.15.67:0/1916426848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080051c0 0x7f130800b9c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 -- 172.21.15.67:0/1916426848 >> 172.21.15.67:0/1916426848 conn(0x7f132006b8c0 msgr2=0x7f132006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 -- 172.21.15.67:0/1916426848 shutdown_connections 2024-04-17T07:27:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.012+0000 7f1327600640 1 -- 172.21.15.67:0/1916426848 wait complete. 2024-04-17T07:27:10.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.014+0000 7f1327600640 1 Processor -- start 2024-04-17T07:27:10.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.014+0000 7f1327600640 1 -- start start 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 0x7f1320148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1320161880 0x7f1320154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13200cd420 con 0x7f1320155350 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1320072af0 con 0x7f13080051c0 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.015+0000 7f1327600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13200cd5a0 con 0x7f1320161880 2024-04-17T07:27:10.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33492/0 (socket says 172.21.15.67:33492) 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 -- 172.21.15.67:0/2985429081 learned_addr learned my addr 172.21.15.67:0/2985429081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1324c00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 0x7f1320148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f131fe00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1320161880 0x7f1320154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 -- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1320161880 msgr2=0x7f1320154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1320161880 0x7f1320154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 -- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 msgr2=0x7f1320148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 0x7f1320148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 -- 172.21.15.67:0/2985429081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1320153de0 con 0x7f1320155350 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1324c00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 0x7f1320148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:10.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.016+0000 7f1325600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f131802c060 tx=0x7f13180014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.017+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318006d50 con 0x7f1320155350 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.017+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1318007030 con 0x7f1320155350 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.017+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318007400 con 0x7f1320155350 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.017+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1320154050 con 0x7f1320155350 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.017+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1320154610 con 0x7f1320155350 2024-04-17T07:27:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.018+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1320068b90 con 0x7f1320155350 2024-04-17T07:27:10.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.019+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f13180075a0 con 0x7f1320155350 2024-04-17T07:27:10.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.020+0000 7f131cc00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 0x7f130006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.020+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4857+0+0 (secure 0 0 0) 0x7f13180ad800 con 0x7f1320155350 2024-04-17T07:27:10.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.021+0000 7f1324c00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 0x7f130006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.022+0000 7f1324c00640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 0x7f130006ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1314009a40 tx=0x7f1314008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.025+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1318033030 con 0x7f1320155350 2024-04-17T07:27:10.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.306+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f13200b4ee0 con 0x7f1320155350 2024-04-17T07:27:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.307+0000 7f131cc00640 1 -- 172.21.15.67:0/2985429081 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7f131807bcc0 con 0x7f1320155350 2024-04-17T07:27:10.306 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:10.306 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:27:10.306 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:27:10.307 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:27:10.308 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:27:10.309 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.311 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:27:10.313 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:27:10.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.316 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:27:10.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:27:10.318 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:27:10.319 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 msgr2=0x7f130006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 0x7f130006ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1314009a40 tx=0x7f1314008040 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 msgr2=0x7f132015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f131802c060 tx=0x7f13180014a0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 shutdown_connections 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f130006ca40 0x7f130006ef00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1320155350 0x7f132015cdd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1320161880 0x7f1320154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 --2- 172.21.15.67:0/2985429081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13080051c0 0x7f1320148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 >> 172.21.15.67:0/2985429081 conn(0x7f132006b8c0 msgr2=0x7f13200aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 shutdown_connections 2024-04-17T07:27:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.310+0000 7f1327600640 1 -- 172.21.15.67:0/2985429081 wait complete. 2024-04-17T07:27:10.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:27:10.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.550+0000 7f0806a00640 1 Processor -- start 2024-04-17T07:27:10.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.550+0000 7f0806a00640 1 -- start start 2024-04-17T07:27:10.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.551+0000 7f0806a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f080006b8c0 0x7f08000cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.551+0000 7f0806a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08000cd4a0 con 0x7f0800141ed0 2024-04-17T07:27:10.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.551+0000 7f0806a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08001607f0 con 0x7f0800160280 2024-04-17T07:27:10.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.551+0000 7f0806a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f080015ce30 con 0x7f080006b8c0 2024-04-17T07:27:10.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.552+0000 7f0806000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0800141ed0 0x7f080015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42408/0 (socket says 172.21.15.67:42408) 2024-04-17T07:27:10.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.552+0000 7f0806000640 1 -- 172.21.15.67:0/2580933455 learned_addr learned my addr 172.21.15.67:0/2580933455 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.552+0000 7f0805600640 1 --2- 172.21.15.67:0/2580933455 >> v2:172.21.15.187:3300/0 conn(0x7f080006b8c0 0x7f08000cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.552+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641666314 0 0) 0x7f08000cd4a0 con 0x7f0800141ed0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e8003520 con 0x7f0800141ed0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1951767143 0 0) 0x7f07e8003520 con 0x7f0800141ed0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 >> v2:172.21.15.187:3300/0 conn(0x7f080006b8c0 msgr2=0x7f08000cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 --2- 172.21.15.67:0/2580933455 >> v2:172.21.15.187:3300/0 conn(0x7f080006b8c0 0x7f08000cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 >> v1:172.21.15.187:6789/0 conn(0x7f0800160280 legacy=0x7f080006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0800160ad0 con 0x7f0800141ed0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f07f80019f0 con 0x7f0800141ed0 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 >> v1:172.21.15.67:6789/0 conn(0x7f0800141ed0 legacy=0x7f080015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8003a70 0x7f07e8003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 0x7f07e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07e8005210 0x7f07e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f080015ce30 con 0x7f07e8004800 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07e800c140 con 0x7f07e8003a70 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07e800c340 con 0x7f07e8005210 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.553+0000 7f0804c00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 0x7f07e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0805600640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8003a70 0x7f07e8003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0804c00640 1 -- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07e8005210 msgr2=0x7f07e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0804c00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07e8005210 0x7f07e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0804c00640 1 -- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8003a70 msgr2=0x7f07e8003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:10.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0804c00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8003a70 0x7f07e8003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0804c00640 1 -- 172.21.15.67:0/2580933455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e8003520 con 0x7f07e8004800 2024-04-17T07:27:10.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0806000640 1 -- 172.21.15.67:0/2580933455 reap_dead start 2024-04-17T07:27:10.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.554+0000 7f0805600640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8003a70 0x7f07e8003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.555+0000 7f0804c00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 0x7f07e8004c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f07ec001750 tx=0x7f07e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a078c87233ea88 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.555+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec001370 con 0x7f07e8004800 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.555+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec0052d0 con 0x7f07e8004800 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.555+0000 7f07ffe00640 1 -- 172.21.15.67:0/2580933455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec003830 con 0x7f07e8004800 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 -- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 msgr2=0x7f07e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 0x7f07e8004c80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f07ec001750 tx=0x7f07e8005830 comp rx=0 tx=0).stop 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 -- 172.21.15.67:0/2580933455 shutdown_connections 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 --2- 172.21.15.67:0/2580933455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e8004800 0x7f07e8004c80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 -- 172.21.15.67:0/2580933455 >> 172.21.15.67:0/2580933455 conn(0x7f0800067d10 msgr2=0x7f08000b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 -- 172.21.15.67:0/2580933455 shutdown_connections 2024-04-17T07:27:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.556+0000 7f0806a00640 1 -- 172.21.15.67:0/2580933455 wait complete. 2024-04-17T07:27:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.558+0000 7f0806a00640 1 Processor -- start 2024-04-17T07:27:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.558+0000 7f0806a00640 1 -- start start 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8004800 0x7f080015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0800143980 0x7f0800143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f080013f9b0 con 0x7f0800144680 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0800161090 con 0x7f07e8004800 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0806a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f080013fb30 con 0x7f0800143980 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0804c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0804c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33524/0 (socket says 172.21.15.67:33524) 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.559+0000 7f0804c00640 1 -- 172.21.15.67:0/3258928304 learned_addr learned my addr 172.21.15.67:0/3258928304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0805600640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8004800 0x7f080015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 -- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0800143980 msgr2=0x7f0800143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0800143980 0x7f0800143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 -- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8004800 msgr2=0x7f080015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8004800 0x7f080015db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 -- 172.21.15.67:0/3258928304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080015f5d0 con 0x7f0800144680 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0804c00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f07ec001870 tx=0x7f07ec001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec041020 con 0x7f0800144680 2024-04-17T07:27:10.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec005480 con 0x7f0800144680 2024-04-17T07:27:10.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec001370 con 0x7f0800144680 2024-04-17T07:27:10.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.560+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08001407c0 con 0x7f0800144680 2024-04-17T07:27:10.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.561+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0800140d80 con 0x7f0800144680 2024-04-17T07:27:10.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.562+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0800068a10 con 0x7f0800144680 2024-04-17T07:27:10.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.563+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f07ec044020 con 0x7f0800144680 2024-04-17T07:27:10.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.564+0000 7f07fd600640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 0x7f07e006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:10.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.564+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4857+0+0 (secure 0 0 0) 0x7f07ec0ad300 con 0x7f0800144680 2024-04-17T07:27:10.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.565+0000 7f0805600640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 0x7f07e006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:10.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.566+0000 7f0805600640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 0x7f07e006ef00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f08000516a0 tx=0x7f07f4008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:10.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.568+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f07ec04b030 con 0x7f0800144680 2024-04-17T07:27:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.778+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08000d0de0 con 0x7f0800144680 2024-04-17T07:27:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.779+0000 7f07fd600640 1 -- 172.21.15.67:0/3258928304 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+12843 (secure 0 0 0) 0x7f07ec0303c0 con 0x7f0800144680 2024-04-17T07:27:10.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:10.779 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:27:09.784871+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.782+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 msgr2=0x7f07e006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.782+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 0x7f07e006ef00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f08000516a0 tx=0x7f07f4008040 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.782+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 msgr2=0x7f0800155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.782+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f07ec001870 tx=0x7f07ec001120 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 shutdown_connections 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f07e006ca40 0x7f07e006ef00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0800143980 0x7f0800143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0800144680 0x7f0800155e10 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 --2- 172.21.15.67:0/3258928304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07e8004800 0x7f080015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 >> 172.21.15.67:0/3258928304 conn(0x7f0800067d10 msgr2=0x7f08000b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:10.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 shutdown_connections 2024-04-17T07:27:10.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:10.783+0000 7f0806a00640 1 -- 172.21.15.67:0/3258928304 wait complete. 2024-04-17T07:27:10.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:27:10.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:27:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.012+0000 7f75e4000640 1 Processor -- start 2024-04-17T07:27:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.012+0000 7f75e4000640 1 -- start start 2024-04-17T07:27:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e4000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0b17c0 0x7f75dc0b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e4000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75dc0b4fa0 0x7f75dc15be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e4000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75dc0b20e0 con 0x7f75dc0cfd00 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e4000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75dc0ce920 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e4000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75dc0b5560 con 0x7f75dc0b17c0 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75dc0b4fa0 0x7f75dc15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75dc0b4fa0 0x7f75dc15be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33534/0 (socket says 172.21.15.67:33534) 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.014+0000 7f75e0c00640 1 -- 172.21.15.67:0/2649537085 learned_addr learned my addr 172.21.15.67:0/2649537085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.015+0000 7f75e1600640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0b17c0 0x7f75dc0b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.015+0000 7f75e0c00640 1 -- 172.21.15.67:0/2649537085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0b17c0 msgr2=0x7f75dc0b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.015+0000 7f75e0c00640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0b17c0 0x7f75dc0b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.015+0000 7f75e0c00640 1 -- 172.21.15.67:0/2649537085 >> v1:172.21.15.187:6789/0 conn(0x7f75dc0cfd00 legacy=0x7f75dc0cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.015+0000 7f75e0c00640 1 -- 172.21.15.67:0/2649537085 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc0b5840 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.016+0000 7f75e0c00640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc15be50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009870 tx=0x7f75c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca94a5caa00eb30 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.016+0000 7f75dbe00640 1 -- 172.21.15.67:0/2649537085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8035020 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.016+0000 7f75dbe00640 1 -- 172.21.15.67:0/2649537085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c8031640 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.016+0000 7f75dbe00640 1 -- 172.21.15.67:0/2649537085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c802d040 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 -- 172.21.15.67:0/2649537085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b4fa0 msgr2=0x7f75dc15be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc15be50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009870 tx=0x7f75c802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 -- 172.21.15.67:0/2649537085 shutdown_connections 2024-04-17T07:27:11.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc15be50 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 --2- 172.21.15.67:0/2649537085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0b17c0 0x7f75dc0b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 -- 172.21.15.67:0/2649537085 >> 172.21.15.67:0/2649537085 conn(0x7f75dc06b8c0 msgr2=0x7f75dc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:11.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 -- 172.21.15.67:0/2649537085 shutdown_connections 2024-04-17T07:27:11.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.017+0000 7f75e4000640 1 -- 172.21.15.67:0/2649537085 wait complete. 2024-04-17T07:27:11.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.019+0000 7f75e4000640 1 Processor -- start 2024-04-17T07:27:11.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.019+0000 7f75e4000640 1 -- start start 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc1e5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 0x7f75dc1e5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc0b5b20 con 0x7f75dc0b17c0 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc15c390 con 0x7f75dc0b4fa0 2024-04-17T07:27:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e4000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75dc145490 con 0x7f75dc0cfd00 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.020+0000 7f75e1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33548/0 (socket says 172.21.15.67:33548) 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 -- 172.21.15.67:0/2587789143 learned_addr learned my addr 172.21.15.67:0/2587789143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e2000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 0x7f75dc1e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e0c00640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc1e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 -- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 msgr2=0x7f75dc1e5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 0x7f75dc1e5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 -- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75dc0b4fa0 msgr2=0x7f75dc1e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc1e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e1600640 1 -- 172.21.15.67:0/2587789143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc0ae800 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.021+0000 7f75e2000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 0x7f75dc1e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75e1600640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f75d000d800 tx=0x7f75d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d000b6c0 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75d0019070 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d000ba80 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc0aeae0 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.022+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc0af0a0 con 0x7f75dc0b17c0 2024-04-17T07:27:11.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.023+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc0d0190 con 0x7f75dc0b17c0 2024-04-17T07:27:11.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.024+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f75d000bc20 con 0x7f75dc0b17c0 2024-04-17T07:27:11.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.026+0000 7f75d9600640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 0x7f75bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.026+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4857+0+0 (secure 0 0 0) 0x7f75d008bea0 con 0x7f75dc0b17c0 2024-04-17T07:27:11.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.026+0000 7f75e0c00640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 0x7f75bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.027+0000 7f75e0c00640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 0x7f75bc06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75c8036000 tx=0x7f75c803c040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.030+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f75dc0d0190 con 0x7f75dc0b17c0 2024-04-17T07:27:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.232+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f75dc068b90 con 0x7f75dc0b17c0 2024-04-17T07:27:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.233+0000 7f75d9600640 1 -- 172.21.15.67:0/2587789143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7f75d005a4f0 con 0x7f75dc0b17c0 2024-04-17T07:27:11.232 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:27:11.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 msgr2=0x7f75bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 0x7f75bc06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f75c8036000 tx=0x7f75c803c040 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 msgr2=0x7f75dc1f4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f75d000d800 tx=0x7f75d000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 shutdown_connections 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f75bc06ca10 0x7f75bc06eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75dc0cfd00 0x7f75dc1e5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75dc0b4fa0 0x7f75dc1e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 --2- 172.21.15.67:0/2587789143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc0b17c0 0x7f75dc1f4700 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 >> 172.21.15.67:0/2587789143 conn(0x7f75dc06b8c0 msgr2=0x7f75dc142c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 shutdown_connections 2024-04-17T07:27:11.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.236+0000 7f75e4000640 1 -- 172.21.15.67:0/2587789143 wait complete. 2024-04-17T07:27:11.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:27:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.469+0000 7fd7ea200640 1 Processor -- start 2024-04-17T07:27:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.469+0000 7fd7ea200640 1 -- start start 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.470+0000 7fd7ea200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd7e40cf8f0 0x7fd7e40b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.470+0000 7fd7ea200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd7e40d12a0 0x7fd7e40d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.470+0000 7fd7ea200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e40cfe20 con 0x7fd7e40b9f00 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.470+0000 7fd7ea200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e40d1c50 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.470+0000 7fd7ea200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e40cf640 con 0x7fd7e40d12a0 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e8e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd7e40b9f00 0x7fd7e41634e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42410/0 (socket says 172.21.15.67:42410) 2024-04-17T07:27:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e8e00640 1 -- 172.21.15.67:0/1971308145 learned_addr learned my addr 172.21.15.67:0/1971308145 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/1971308145 >> v2:172.21.15.67:3300/0 conn(0x7fd7e40cf8f0 0x7fd7e40b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e9800640 1 --2- 172.21.15.67:0/1971308145 >> v2:172.21.15.187:3300/0 conn(0x7fd7e40d12a0 0x7fd7e40d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3400640 1 -- 172.21.15.67:0/1971308145 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912804506 0 0) 0x7fd7e40cfe20 con 0x7fd7e40b9f00 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3400640 1 -- 172.21.15.67:0/1971308145 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7c8003520 con 0x7fd7e40b9f00 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3e00640 1 -- 172.21.15.67:0/1971308145 >> v2:172.21.15.187:3300/0 conn(0x7fd7e40d12a0 msgr2=0x7fd7e40d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/1971308145 >> v2:172.21.15.187:3300/0 conn(0x7fd7e40d12a0 0x7fd7e40d1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3e00640 1 -- 172.21.15.67:0/1971308145 >> v1:172.21.15.67:6789/0 conn(0x7fd7e40b9f00 legacy=0x7fd7e41634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.471+0000 7fd7e3e00640 1 -- 172.21.15.67:0/1971308145 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e40d0100 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.472+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/1971308145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40cf8f0 0x7fd7e40b7f80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40b7810 tx=0x7fd7cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22960eea4902b46a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7e3400640 1 -- 172.21.15.67:0/1971308145 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0315a0 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7e3400640 1 -- 172.21.15.67:0/1971308145 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc037070 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7e3400640 1 -- 172.21.15.67:0/1971308145 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc031960 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 -- 172.21.15.67:0/1971308145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40cf8f0 msgr2=0x7fd7e40b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 --2- 172.21.15.67:0/1971308145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40cf8f0 0x7fd7e40b7f80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40b7810 tx=0x7fd7cc02b840 comp rx=0 tx=0).stop 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 -- 172.21.15.67:0/1971308145 shutdown_connections 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 --2- 172.21.15.67:0/1971308145 >> v2:172.21.15.187:3300/0 conn(0x7fd7e40d12a0 0x7fd7e40d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 --2- 172.21.15.67:0/1971308145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40cf8f0 0x7fd7e40b7f80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.473+0000 7fd7ea200640 1 -- 172.21.15.67:0/1971308145 >> 172.21.15.67:0/1971308145 conn(0x7fd7e406cb60 msgr2=0x7fd7e406cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.474+0000 7fd7ea200640 1 -- 172.21.15.67:0/1971308145 shutdown_connections 2024-04-17T07:27:11.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.474+0000 7fd7ea200640 1 -- 172.21.15.67:0/1971308145 wait complete. 2024-04-17T07:27:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.475+0000 7fd7ea200640 1 Processor -- start 2024-04-17T07:27:11.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.476+0000 7fd7ea200640 1 -- start start 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.476+0000 7fd7ea200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7ea200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 0x7fd7e40c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7ea200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e40d12a0 0x7fd7e415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7ea200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40ba2e0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7ea200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40d3420 con 0x7fd7e40d12a0 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7ea200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e40d03e0 con 0x7fd7e40cf8f0 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33564/0 (socket says 172.21.15.67:33564) 2024-04-17T07:27:11.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 -- 172.21.15.67:0/478198781 learned_addr learned my addr 172.21.15.67:0/478198781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 0x7fd7e40c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 -- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 msgr2=0x7fd7e40c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e9800640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e40d12a0 0x7fd7e415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 0x7fd7e40c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 -- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e40d12a0 msgr2=0x7fd7e415bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e40d12a0 0x7fd7e415bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e8e00640 1 -- 172.21.15.67:0/478198781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e40c7f40 con 0x7fd7e40b9f00 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.477+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 0x7fd7e40c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7e8e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400c430 tx=0x7fd7d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4017070 con 0x7fd7e40b9f00 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d4005ce0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40072f0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e40c8220 con 0x7fd7e40b9f00 2024-04-17T07:27:11.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.478+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e40cb6a0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.479+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e40b6b60 con 0x7fd7e40b9f00 2024-04-17T07:27:11.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.480+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fd7d401a070 con 0x7fd7e40b9f00 2024-04-17T07:27:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.481+0000 7fd7e0c00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 0x7fd7c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.481+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4857+0+0 (secure 0 0 0) 0x7fd7d408af30 con 0x7fd7e40b9f00 2024-04-17T07:27:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.482+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 0x7fd7c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.483+0000 7fd7e3e00640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 0x7fd7c006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc036000 tx=0x7fd7cc008030 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.485+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd7e40b6b60 con 0x7fd7e40b9f00 2024-04-17T07:27:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.689+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd7e40b82b0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.690+0000 7fd7e0c00640 1 -- 172.21.15.67:0/478198781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) ==== 84+0+65 (secure 0 0 0) 0x7fd7e40b82b0 con 0x7fd7e40b9f00 2024-04-17T07:27:11.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:11.689 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:27:11.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:11.689 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:27:11.690 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:27:11.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:11.690 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:27:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 msgr2=0x7fd7c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 0x7fd7c006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc036000 tx=0x7fd7cc008030 comp rx=0 tx=0).stop 2024-04-17T07:27:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 msgr2=0x7fd7e40cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400c430 tx=0x7fd7d400c8f0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 shutdown_connections 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd7c006ca10 0x7fd7c006eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7e40d12a0 0x7fd7e415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7e40cf8f0 0x7fd7e40c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 --2- 172.21.15.67:0/478198781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e40b9f00 0x7fd7e40cb160 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 >> 172.21.15.67:0/478198781 conn(0x7fd7e406cb60 msgr2=0x7fd7e415ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 shutdown_connections 2024-04-17T07:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.693+0000 7fd7ea200640 1 -- 172.21.15.67:0/478198781 wait complete. 2024-04-17T07:27:11.711 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:27:11.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:27:11.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.928+0000 7fb7e8c00640 1 Processor -- start 2024-04-17T07:27:11.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.928+0000 7fb7e8c00640 1 -- start start 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e8c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb7e40cfc80 0x7fb7e40b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7e4155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e8c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7e415f890 con 0x7fb7e40b3800 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e8c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb7e41629b0 con 0x7fb7e40cfc80 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e8c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e415fb70 con 0x7fb7e4162e40 2024-04-17T07:27:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7e40b3800 0x7fb7e415f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42412/0 (socket says 172.21.15.67:42412) 2024-04-17T07:27:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.930+0000 7fb7e3400640 1 -- 172.21.15.67:0/2683952975 learned_addr learned my addr 172.21.15.67:0/2683952975 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e3e00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7e4155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/2683952975 >> v2:172.21.15.187:3300/0 conn(0x7fb7e40cfc80 0x7fb7e40b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269889583 0 0) 0x7fb7e415f890 con 0x7fb7e40b3800 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c8003520 con 0x7fb7e40b3800 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2536631389 0 0) 0x7fb7c8003520 con 0x7fb7e40b3800 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 msgr2=0x7fb7e4155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7e4155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 >> v2:172.21.15.187:3300/0 conn(0x7fb7e40cfc80 msgr2=0x7fb7e40b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 --2- 172.21.15.67:0/2683952975 >> v2:172.21.15.187:3300/0 conn(0x7fb7e40cfc80 0x7fb7e40b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e415fe50 con 0x7fb7e40b3800 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.931+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb7d4001670 con 0x7fb7e40b3800 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 >> v1:172.21.15.67:6789/0 conn(0x7fb7e40b3800 legacy=0x7fb7e415f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7c80039b0 0x7fb7c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c8004800 0x7fb7c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41629b0 con 0x7fb7c8004800 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e415fb70 con 0x7fb7c80039b0 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c800ba40 con 0x7fb7e4162e40 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c8004800 0x7fb7c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e3400640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7c80039b0 0x7fb7c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.932+0000 7fb7e3e00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.933+0000 7fb7e2a00640 1 -- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 msgr2=0x7fb7c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.933+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7e4162e40 0x7fb7c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.933+0000 7fb7e2a00640 1 -- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7c80039b0 msgr2=0x7fb7c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.933+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7c80039b0 0x7fb7c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.933+0000 7fb7e2a00640 1 -- 172.21.15.67:0/2683952975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c8003520 con 0x7fb7c8004800 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.934+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c8004800 0x7fb7c8004c80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009ac0 tx=0x7fb7c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33aa49c392f9154e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.934+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0039070 con 0x7fb7c8004800 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.934+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d002c710 con 0x7fb7c8004800 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.934+0000 7fb7e2000640 1 -- 172.21.15.67:0/2683952975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d0009690 con 0x7fb7c8004800 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 -- 172.21.15.67:0/2683952975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c8004800 msgr2=0x7fb7c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/2683952975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7c8004800 0x7fb7c8004c80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009ac0 tx=0x7fb7c8005830 comp rx=0 tx=0).stop 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e3e00640 1 -- 172.21.15.67:0/2683952975 reap_dead start 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 -- 172.21.15.67:0/2683952975 shutdown_connections 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 -- 172.21.15.67:0/2683952975 >> 172.21.15.67:0/2683952975 conn(0x7fb7e4067600 msgr2=0x7fb7e40679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 -- 172.21.15.67:0/2683952975 shutdown_connections 2024-04-17T07:27:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.935+0000 7fb7e8c00640 1 -- 172.21.15.67:0/2683952975 wait complete. 2024-04-17T07:27:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.937+0000 7fb7e8c00640 1 Processor -- start 2024-04-17T07:27:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.937+0000 7fb7e8c00640 1 -- start start 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7c8004800 0x7fb7e4152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7e41f4780 0x7fb7e40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e4163450 con 0x7fb7e41f4c00 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41632d0 con 0x7fb7e41f4780 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.938+0000 7fb7e8c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e4155560 con 0x7fb7c8004800 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33586/0 (socket says 172.21.15.67:33586) 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 -- 172.21.15.67:0/352996913 learned_addr learned my addr 172.21.15.67:0/352996913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3400640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7c8004800 0x7fb7e4152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e2a00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7e41f4780 0x7fb7e40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 -- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7c8004800 msgr2=0x7fb7e4152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7c8004800 0x7fb7e4152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 -- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7e41f4780 msgr2=0x7fb7e40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:11.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7e41f4780 0x7fb7e40b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:11.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 -- 172.21.15.67:0/352996913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e41f51d0 con 0x7fb7e41f4c00 2024-04-17T07:27:11.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.939+0000 7fb7e3e00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4163220 tx=0x7fb7dc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.940+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc015430 con 0x7fb7e41f4c00 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.940+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7dc01d070 con 0x7fb7e41f4c00 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.940+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc015830 con 0x7fb7e41f4c00 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.940+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e41f54b0 con 0x7fb7e41f4c00 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.940+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e40cd180 con 0x7fb7e41f4c00 2024-04-17T07:27:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.941+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e4068a10 con 0x7fb7e41f4c00 2024-04-17T07:27:11.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.942+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fb7dc0159d0 con 0x7fb7e41f4c00 2024-04-17T07:27:11.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.943+0000 7fb7db400640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 0x7fb7c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:11.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.943+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4857+0+0 (secure 0 0 0) 0x7fb7dc08c490 con 0x7fb7e41f4c00 2024-04-17T07:27:11.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.944+0000 7fb7e3400640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 0x7fb7c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:11.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.945+0000 7fb7e3400640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 0x7fb7c406eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4000fb0 tx=0x7fb7d4009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:11.948+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb7dc010040 con 0x7fb7e41f4c00 2024-04-17T07:27:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.153+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7e40cd350 con 0x7fb7e41f4c00 2024-04-17T07:27:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.797+0000 7fb7db400640 1 -- 172.21.15.67:0/352996913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v62) ==== 138+0+0 (secure 0 0 0) 0x7fb7dc05a300 con 0x7fb7e41f4c00 2024-04-17T07:27:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:27:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 msgr2=0x7fb7c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 0x7fb7c406eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4000fb0 tx=0x7fb7d4009040 comp rx=0 tx=0).stop 2024-04-17T07:27:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 msgr2=0x7fb7e40b5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4163220 tx=0x7fb7dc00f580 comp rx=0 tx=0).stop 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 shutdown_connections 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fb7c406ca10 0x7fb7c406eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e41f4c00 0x7fb7e40b5160 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7e41f4780 0x7fb7e40b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 --2- 172.21.15.67:0/352996913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7c8004800 0x7fb7e4152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 >> 172.21.15.67:0/352996913 conn(0x7fb7e4067600 msgr2=0x7fb7e4160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 shutdown_connections 2024-04-17T07:27:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:12.801+0000 7fb7e8c00640 1 -- 172.21.15.67:0/352996913 wait complete. 2024-04-17T07:27:12.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:27:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.048+0000 7f7c98800640 1 Processor -- start 2024-04-17T07:27:13.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.048+0000 7f7c98800640 1 -- start start 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.049+0000 7f7c98800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c90146150 0x7f7c90160d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.049+0000 7f7c98800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c900cd060 0x7f7c90071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c98800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c90161240 con 0x7f7c900cd4d0 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c98800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7c90146530 con 0x7f7c90146150 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c98800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7c900cda40 con 0x7f7c900cd060 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c95e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c90146150 0x7f7c90160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c95e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c90146150 0x7f7c90160d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33594/0 (socket says 172.21.15.67:33594) 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c95e00640 1 -- 172.21.15.67:0/2643778116 learned_addr learned my addr 172.21.15.67:0/2643778116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:13.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c96800640 1 --1- 172.21.15.67:0/2643778116 >> v1:172.21.15.67:6789/0 conn(0x7f7c900cd4d0 0x7f7c9015cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42416/0 (socket says 172.21.15.67:42416) 2024-04-17T07:27:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.050+0000 7f7c95400640 1 --2- 172.21.15.67:0/2643778116 >> v2:172.21.15.187:3300/0 conn(0x7f7c900cd060 0x7f7c90071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.051+0000 7f7c95e00640 1 -- 172.21.15.67:0/2643778116 >> v2:172.21.15.187:3300/0 conn(0x7f7c900cd060 msgr2=0x7f7c90071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.051+0000 7f7c95e00640 1 --2- 172.21.15.67:0/2643778116 >> v2:172.21.15.187:3300/0 conn(0x7f7c900cd060 0x7f7c90071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.051+0000 7f7c95e00640 1 -- 172.21.15.67:0/2643778116 >> v1:172.21.15.67:6789/0 conn(0x7f7c900cd4d0 legacy=0x7f7c9015cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:13.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.051+0000 7f7c94a00640 1 -- 172.21.15.67:0/2643778116 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3022309097 0 0) 0x7f7c8c001490 con 0x7f7c900cd4d0 2024-04-17T07:27:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.051+0000 7f7c95e00640 1 -- 172.21.15.67:0/2643778116 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c900cdd20 con 0x7f7c90146150 2024-04-17T07:27:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.052+0000 7f7c95e00640 1 --2- 172.21.15.67:0/2643778116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90146150 0x7f7c90160d00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7c90072e30 tx=0x7f7c8402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ba0b62d19d6821 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.052+0000 7f7c94a00640 1 -- 172.21.15.67:0/2643778116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84034070 con 0x7f7c90146150 2024-04-17T07:27:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.052+0000 7f7c94a00640 1 -- 172.21.15.67:0/2643778116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c84030860 con 0x7f7c90146150 2024-04-17T07:27:13.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.052+0000 7f7c94a00640 1 -- 172.21.15.67:0/2643778116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84030c30 con 0x7f7c90146150 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 -- 172.21.15.67:0/2643778116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90146150 msgr2=0x7f7c90160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 --2- 172.21.15.67:0/2643778116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90146150 0x7f7c90160d00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7c90072e30 tx=0x7f7c8402b9a0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 -- 172.21.15.67:0/2643778116 shutdown_connections 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 --2- 172.21.15.67:0/2643778116 >> v2:172.21.15.187:3300/0 conn(0x7f7c900cd060 0x7f7c90071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 --2- 172.21.15.67:0/2643778116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c90146150 0x7f7c90160d00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 -- 172.21.15.67:0/2643778116 >> 172.21.15.67:0/2643778116 conn(0x7f7c90067c30 msgr2=0x7f7c9006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 -- 172.21.15.67:0/2643778116 shutdown_connections 2024-04-17T07:27:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.053+0000 7f7c98800640 1 -- 172.21.15.67:0/2643778116 wait complete. 2024-04-17T07:27:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.055+0000 7f7c98800640 1 Processor -- start 2024-04-17T07:27:13.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.055+0000 7f7c98800640 1 -- start start 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c900cd4d0 0x7f7c90141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c90146150 0x7f7c90140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9006f310 con 0x7f7c900cd060 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7c901607e0 con 0x7f7c900cd4d0 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c98800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c9006f490 con 0x7f7c90146150 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:13.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33600/0 (socket says 172.21.15.67:33600) 2024-04-17T07:27:13.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c95e00640 1 -- 172.21.15.67:0/1805388197 learned_addr learned my addr 172.21.15.67:0/1805388197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.056+0000 7f7c95e00640 1 -- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c90146150 msgr2=0x7f7c90140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c95e00640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c90146150 0x7f7c90140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c95e00640 1 -- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c900cd4d0 msgr2=0x7f7c90141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c95e00640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c900cd4d0 0x7f7c90141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c95e00640 1 -- 172.21.15.67:0/1805388197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9013e9f0 con 0x7f7c900cd060 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c95e00640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c8402b9e0 tx=0x7f7c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84034040 con 0x7f7c900cd060 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c84039d90 con 0x7f7c900cd060 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84043610 con 0x7f7c900cd060 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c90140e90 con 0x7f7c900cd060 2024-04-17T07:27:13.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.057+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c90141450 con 0x7f7c900cd060 2024-04-17T07:27:13.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.059+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c9013e7e0 con 0x7f7c900cd060 2024-04-17T07:27:13.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.060+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f7c8404a060 con 0x7f7c900cd060 2024-04-17T07:27:13.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.061+0000 7f7c8a000640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 0x7f7c7406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:13.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.061+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5227+0+0 (secure 0 0 0) 0x7f7c840ade10 con 0x7f7c900cd060 2024-04-17T07:27:13.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.062+0000 7f7c95400640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 0x7f7c7406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.063+0000 7f7c95400640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 0x7f7c7406eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c90141f40 tx=0x7f7c80008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.066+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7c840b1030 con 0x7f7c900cd060 2024-04-17T07:27:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.272+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7c9013ebc0 con 0x7f7c900cd060 2024-04-17T07:27:13.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.815+0000 7f7c8a000640 1 -- 172.21.15.67:0/1805388197 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v63) ==== 170+0+0 (secure 0 0 0) 0x7f7c8403e070 con 0x7f7c900cd060 2024-04-17T07:27:14.310 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.818+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 msgr2=0x7f7c7406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.818+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 0x7f7c7406eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7c90141f40 tx=0x7f7c80008040 comp rx=0 tx=0).stop 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.818+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 msgr2=0x7f7c901f89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7c8402b9e0 tx=0x7f7c840046c0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 shutdown_connections 2024-04-17T07:27:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f7c7406ca10 0x7f7c7406eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c90146150 0x7f7c90140780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c900cd4d0 0x7f7c90141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 --2- 172.21.15.67:0/1805388197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c900cd060 0x7f7c901f89e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 >> 172.21.15.67:0/1805388197 conn(0x7f7c90067c30 msgr2=0x7f7c900b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 shutdown_connections 2024-04-17T07:27:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:13.819+0000 7f7c98800640 1 -- 172.21.15.67:0/1805388197 wait complete. 2024-04-17T07:27:14.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:27:14.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.551+0000 7fe647200640 1 Processor -- start 2024-04-17T07:27:14.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.551+0000 7fe647200640 1 -- start start 2024-04-17T07:27:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.552+0000 7fe647200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe640159020 0x7fe64015ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.552+0000 7fe647200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.552+0000 7fe647200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64015f780 con 0x7fe6400af400 2024-04-17T07:27:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.552+0000 7fe647200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe6401633b0 con 0x7fe640159020 2024-04-17T07:27:14.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.552+0000 7fe647200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe64015fa60 con 0x7fe64006b8c0 2024-04-17T07:27:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.553+0000 7fe645e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe640159020 0x7fe64015ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.553+0000 7fe645e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe640159020 0x7fe64015ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:51996/0 (socket says 172.21.15.67:51996) 2024-04-17T07:27:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.553+0000 7fe645e00640 1 -- 172.21.15.67:0/2990917088 learned_addr learned my addr 172.21.15.67:0/2990917088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.553+0000 7fe646800640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.554+0000 7fe645e00640 1 -- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 msgr2=0x7fe64006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.554+0000 7fe645e00640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64006bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.554+0000 7fe645e00640 1 -- 172.21.15.67:0/2990917088 >> v1:172.21.15.187:6789/0 conn(0x7fe6400af400 legacy=0x7fe640162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.554+0000 7fe645e00640 1 -- 172.21.15.67:0/2990917088 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6401624a0 con 0x7fe640159020 2024-04-17T07:27:14.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.555+0000 7fe645e00640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015ee30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe634009870 tx=0x7fe63402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ccac33a1483bd99 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.556+0000 7fe644a00640 1 -- 172.21.15.67:0/2990917088 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634035020 con 0x7fe640159020 2024-04-17T07:27:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.556+0000 7fe644a00640 1 -- 172.21.15.67:0/2990917088 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe634031640 con 0x7fe640159020 2024-04-17T07:27:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.556+0000 7fe644a00640 1 -- 172.21.15.67:0/2990917088 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63402d040 con 0x7fe640159020 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.556+0000 7fe647200640 1 -- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 msgr2=0x7fe64015ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.556+0000 7fe647200640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015ee30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe634009870 tx=0x7fe63402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 -- 172.21.15.67:0/2990917088 shutdown_connections 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64006bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 --2- 172.21.15.67:0/2990917088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015ee30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 -- 172.21.15.67:0/2990917088 >> 172.21.15.67:0/2990917088 conn(0x7fe6400b2190 msgr2=0x7fe6400b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 -- 172.21.15.67:0/2990917088 shutdown_connections 2024-04-17T07:27:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.557+0000 7fe647200640 1 -- 172.21.15.67:0/2990917088 wait complete. 2024-04-17T07:27:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.558+0000 7fe647200640 1 Processor -- start 2024-04-17T07:27:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.559+0000 7fe647200640 1 -- start start 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.559+0000 7fe647200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64014af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.559+0000 7fe647200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.559+0000 7fe647200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe647200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe64015f370 con 0x7fe6400af400 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe647200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe64015fd40 con 0x7fe640159020 2024-04-17T07:27:14.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe647200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe64015f4f0 con 0x7fe64006b8c0 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35810/0 (socket says 172.21.15.67:35810) 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 -- 172.21.15.67:0/2965282637 learned_addr learned my addr 172.21.15.67:0/2965282637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe646800640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645e00640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64014af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 -- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 msgr2=0x7fe64014af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64014af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 -- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 msgr2=0x7fe64015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.560+0000 7fe645400640 1 -- 172.21.15.67:0/2965282637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6400c5400 con 0x7fe6400af400 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe645400640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c430 tx=0x7fe63000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630017070 con 0x7fe6400af400 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6300052d0 con 0x7fe6400af400 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630003830 con 0x7fe6400af400 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6400c56e0 con 0x7fe6400af400 2024-04-17T07:27:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.561+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6400c5ca0 con 0x7fe6400af400 2024-04-17T07:27:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.562+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe640068a10 con 0x7fe6400af400 2024-04-17T07:27:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.564+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fe63001a070 con 0x7fe6400af400 2024-04-17T07:27:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.565+0000 7fe63a000640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 0x7fe62406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.565+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5609+0+0 (secure 0 0 0) 0x7fe63008b0f0 con 0x7fe6400af400 2024-04-17T07:27:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.565+0000 7fe645e00640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 0x7fe62406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:14.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.566+0000 7fe645e00640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 0x7fe62406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe634036000 tx=0x7fe63403c040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:14.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.570+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe6300593d0 con 0x7fe6400af400 2024-04-17T07:27:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:14.818+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe64006c7b0 con 0x7fe6400af400 2024-04-17T07:27:15.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.822+0000 7fe63a000640 1 -- 172.21.15.67:0/2965282637 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) ==== 172+0+0 (secure 0 0 0) 0x7fe630058cd0 con 0x7fe6400af400 2024-04-17T07:27:15.821 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 6 and data pool 7 2024-04-17T07:27:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 msgr2=0x7fe62406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 0x7fe62406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe634036000 tx=0x7fe63403c040 comp rx=0 tx=0).stop 2024-04-17T07:27:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 msgr2=0x7fe6400c64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c430 tx=0x7fe63000c8f0 comp rx=0 tx=0).stop 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 shutdown_connections 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fe62406ca40 0x7fe62406ef00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe640159020 0x7fe64015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6400af400 0x7fe6400c64a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 --2- 172.21.15.67:0/2965282637 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe64006b8c0 0x7fe64014af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 >> 172.21.15.67:0/2965282637 conn(0x7fe6400b2190 msgr2=0x7fe6400d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 shutdown_connections 2024-04-17T07:27:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:15.827+0000 7fe647200640 1 -- 172.21.15.67:0/2965282637 wait complete. 2024-04-17T07:27:15.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:27:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.047+0000 7f9847200640 1 Processor -- start 2024-04-17T07:27:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.047+0000 7f9847200640 1 -- start start 2024-04-17T07:27:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9847200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f984806b8c0 0x7f98480cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9847200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9848141ed0 0x7f9848142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9847200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98480cd4a0 con 0x7f9848160280 2024-04-17T07:27:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9847200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f984815ce30 con 0x7f984806b8c0 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9847200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f98481607f0 con 0x7f9848141ed0 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9845e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f984806b8c0 0x7f98480cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f983d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9848160280 0x7f984806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38992/0 (socket says 172.21.15.67:38992) 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f983d400640 1 -- 172.21.15.67:0/330776076 learned_addr learned my addr 172.21.15.67:0/330776076 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.049+0000 7f9846800640 1 --2- 172.21.15.67:0/330776076 >> v2:172.21.15.187:3300/0 conn(0x7f9848141ed0 0x7f9848142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.050+0000 7f9845e00640 1 -- 172.21.15.67:0/330776076 >> v2:172.21.15.187:3300/0 conn(0x7f9848141ed0 msgr2=0x7f9848142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.050+0000 7f9845e00640 1 --2- 172.21.15.67:0/330776076 >> v2:172.21.15.187:3300/0 conn(0x7f9848141ed0 0x7f9848142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.050+0000 7f9845e00640 1 -- 172.21.15.67:0/330776076 >> v1:172.21.15.67:6789/0 conn(0x7f9848160280 legacy=0x7f984806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.050+0000 7f9845e00640 1 -- 172.21.15.67:0/330776076 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9848160ad0 con 0x7f984806b8c0 2024-04-17T07:27:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.050+0000 7f9845400640 1 -- 172.21.15.67:0/330776076 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2844817301 0 0) 0x7f98480cd4a0 con 0x7f9848160280 2024-04-17T07:27:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.051+0000 7f9845e00640 1 --2- 172.21.15.67:0/330776076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984806b8c0 0x7f98480cd8b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f98480b2120 tx=0x7f983802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a574991881cb48 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.051+0000 7f9845400640 1 -- 172.21.15.67:0/330776076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838035020 con 0x7f984806b8c0 2024-04-17T07:27:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.051+0000 7f9845400640 1 -- 172.21.15.67:0/330776076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9838031830 con 0x7f984806b8c0 2024-04-17T07:27:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.051+0000 7f9845400640 1 -- 172.21.15.67:0/330776076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838031c00 con 0x7f984806b8c0 2024-04-17T07:27:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 -- 172.21.15.67:0/330776076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984806b8c0 msgr2=0x7f98480cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 --2- 172.21.15.67:0/330776076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984806b8c0 0x7f98480cd8b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f98480b2120 tx=0x7f983802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 -- 172.21.15.67:0/330776076 shutdown_connections 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 --2- 172.21.15.67:0/330776076 >> v2:172.21.15.187:3300/0 conn(0x7f9848141ed0 0x7f9848142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 --2- 172.21.15.67:0/330776076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984806b8c0 0x7f98480cd8b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 -- 172.21.15.67:0/330776076 >> 172.21.15.67:0/330776076 conn(0x7f9848067d10 msgr2=0x7f98480b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 -- 172.21.15.67:0/330776076 shutdown_connections 2024-04-17T07:27:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.052+0000 7f9847200640 1 -- 172.21.15.67:0/330776076 wait complete. 2024-04-17T07:27:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.054+0000 7f9847200640 1 Processor -- start 2024-04-17T07:27:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.054+0000 7f9847200640 1 -- start start 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 0x7f9848156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 0x7f984815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9848161090 con 0x7f9848141ed0 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f98480cddf0 con 0x7f9848160280 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9847200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f984813f9b0 con 0x7f984806b8c0 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35830/0 (socket says 172.21.15.67:35830) 2024-04-17T07:27:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 -- 172.21.15.67:0/3559496259 learned_addr learned my addr 172.21.15.67:0/3559496259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9846800640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 0x7f984815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9845e00640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 0x7f9848156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 -- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 msgr2=0x7f9848156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 0x7f9848156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 -- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 msgr2=0x7f984815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 0x7f984815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f983d400640 1 -- 172.21.15.67:0/3559496259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98481569a0 con 0x7f9848141ed0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9845e00640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 0x7f9848156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.055+0000 7f9846800640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 0x7f984815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f983d400640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f983000c7c0 tx=0x7f983000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830018070 con 0x7f9848141ed0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f983000a640 con 0x7f9848141ed0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984806eaa0 con 0x7f9848141ed0 2024-04-17T07:27:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983000aa30 con 0x7f9848141ed0 2024-04-17T07:27:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.056+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984806f060 con 0x7f9848141ed0 2024-04-17T07:27:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.057+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9848068a10 con 0x7f9848141ed0 2024-04-17T07:27:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.058+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f983001b070 con 0x7f9848141ed0 2024-04-17T07:27:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.059+0000 7f983ea00640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 0x7f982406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.059+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f983008b350 con 0x7f9848141ed0 2024-04-17T07:27:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.060+0000 7f9845e00640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 0x7f982406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.061+0000 7f9845e00640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 0x7f982406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9838036000 tx=0x7f9838008030 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.063+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9830059660 con 0x7f9848141ed0 2024-04-17T07:27:16.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.271+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9848146220 con 0x7f9848141ed0 2024-04-17T07:27:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.273+0000 7f983ea00640 1 -- 172.21.15.67:0/3559496259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16565 (secure 0 0 0) 0x7f9830058f60 con 0x7f9848141ed0 2024-04-17T07:27:16.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:16.273 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:27:15.855819+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.275+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 msgr2=0x7f982406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.275+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 0x7f982406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9838036000 tx=0x7f9838008030 comp rx=0 tx=0).stop 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.275+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 msgr2=0x7f984806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.275+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f983000c7c0 tx=0x7f983000cc80 comp rx=0 tx=0).stop 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 shutdown_connections 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f982406ca40 0x7f982406ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9848160280 0x7f984815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9848141ed0 0x7f984806dfe0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 --2- 172.21.15.67:0/3559496259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f984806b8c0 0x7f9848156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 >> 172.21.15.67:0/3559496259 conn(0x7f9848067d10 msgr2=0x7f984815f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 shutdown_connections 2024-04-17T07:27:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.276+0000 7f9847200640 1 -- 172.21.15.67:0/3559496259 wait complete. 2024-04-17T07:27:16.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:27:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.507+0000 7fdee5800640 1 Processor -- start 2024-04-17T07:27:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.508+0000 7fdee5800640 1 -- start start 2024-04-17T07:27:16.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdee5800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdee00b9f00 0x7fdee00b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdee5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee00d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdee5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee00cfe20 con 0x7fdee00cf8f0 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdee5800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdee00d1c50 con 0x7fdee00b9f00 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdee5800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdee00cf640 con 0x7fdee00d12a0 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.509+0000 7fdedf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdee00cf8f0 0x7fdee01634e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38994/0 (socket says 172.21.15.67:38994) 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedf400640 1 -- 172.21.15.67:0/3281503351 learned_addr learned my addr 172.21.15.67:0/3281503351 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3281503351 >> v2:172.21.15.67:3300/0 conn(0x7fdee00b9f00 0x7fdee00b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdee4e00640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee00d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedfe00640 1 -- 172.21.15.67:0/3281503351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 msgr2=0x7fdee00d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:16.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee00d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedfe00640 1 -- 172.21.15.67:0/3281503351 >> v1:172.21.15.67:6789/0 conn(0x7fdee00cf8f0 legacy=0x7fdee01634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.510+0000 7fdedfe00640 1 -- 172.21.15.67:0/3281503351 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee00d0100 con 0x7fdee00b9f00 2024-04-17T07:27:16.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.511+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00b9f00 0x7fdee00b7f80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fded00097a0 tx=0x7fded002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8a6aae6a9bd87f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.511+0000 7fdedea00640 1 -- 172.21.15.67:0/3281503351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0031780 con 0x7fdee00b9f00 2024-04-17T07:27:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdedea00640 1 -- 172.21.15.67:0/3281503351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded0037070 con 0x7fdee00b9f00 2024-04-17T07:27:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdedea00640 1 -- 172.21.15.67:0/3281503351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0031b40 con 0x7fdee00b9f00 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 -- 172.21.15.67:0/3281503351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00b9f00 msgr2=0x7fdee00b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00b9f00 0x7fdee00b7f80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fded00097a0 tx=0x7fded002b820 comp rx=0 tx=0).stop 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 -- 172.21.15.67:0/3281503351 shutdown_connections 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee00d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 --2- 172.21.15.67:0/3281503351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00b9f00 0x7fdee00b7f80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 -- 172.21.15.67:0/3281503351 >> 172.21.15.67:0/3281503351 conn(0x7fdee006cb60 msgr2=0x7fdee006cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 -- 172.21.15.67:0/3281503351 shutdown_connections 2024-04-17T07:27:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.512+0000 7fdee5800640 1 -- 172.21.15.67:0/3281503351 wait complete. 2024-04-17T07:27:16.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.514+0000 7fdee5800640 1 Processor -- start 2024-04-17T07:27:16.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.514+0000 7fdee5800640 1 -- start start 2024-04-17T07:27:16.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdee00b9f00 0x7fdee00cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee01f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdee00d3420 con 0x7fdee00cf8f0 2024-04-17T07:27:16.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdee00b84c0 con 0x7fdee00b9f00 2024-04-17T07:27:16.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdee5800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdee00ba2e0 con 0x7fdee00d12a0 2024-04-17T07:27:16.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.515+0000 7fdedf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35848/0 (socket says 172.21.15.67:35848) 2024-04-17T07:27:16.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 -- 172.21.15.67:0/3183336084 learned_addr learned my addr 172.21.15.67:0/3183336084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdee00b9f00 0x7fdee00cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdee4e00640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee01f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 -- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 msgr2=0x7fdee01f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee01f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 -- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdee00b9f00 msgr2=0x7fdee00cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdee00b9f00 0x7fdee00cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 -- 172.21.15.67:0/3183336084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee00cba00 con 0x7fdee00cf8f0 2024-04-17T07:27:16.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.516+0000 7fdedf400640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdec800cb10 tx=0x7fdec800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.517+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8017070 con 0x7fdee00cf8f0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.517+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec800ac70 con 0x7fdee00cf8f0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.517+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee00cbce0 con 0x7fdee00cf8f0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.517+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8012620 con 0x7fdee00cf8f0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.517+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee00cc2a0 con 0x7fdee00cf8f0 2024-04-17T07:27:16.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.518+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee00755d0 con 0x7fdee00cf8f0 2024-04-17T07:27:16.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.519+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fdec801a070 con 0x7fdee00cf8f0 2024-04-17T07:27:16.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.520+0000 7fded7e00640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 0x7fdebc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.520+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fdec808bb60 con 0x7fdee00cf8f0 2024-04-17T07:27:16.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.521+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 0x7fdebc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.522+0000 7fdedfe00640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 0x7fdebc06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fded0036000 tx=0x7fded003d040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.524+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdec8059de0 con 0x7fdee00cf8f0 2024-04-17T07:27:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.728+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdee015e930 con 0x7fdee00cf8f0 2024-04-17T07:27:16.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.732+0000 7fded7e00640 1 -- 172.21.15.67:0/3183336084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16565 (secure 0 0 0) 0x7fdec80596e0 con 0x7fdee00cf8f0 2024-04-17T07:27:16.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:16.731 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:27:15.855819+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:16.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.734+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 msgr2=0x7fdebc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.734+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 0x7fdebc06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fded0036000 tx=0x7fded003d040 comp rx=0 tx=0).stop 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.734+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 msgr2=0x7fdee00ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.734+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdec800cb10 tx=0x7fdec800a060 comp rx=0 tx=0).stop 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 shutdown_connections 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdebc06ca10 0x7fdebc06eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdee00d12a0 0x7fdee01f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00cf8f0 0x7fdee00ccaa0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 --2- 172.21.15.67:0/3183336084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdee00b9f00 0x7fdee00cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 >> 172.21.15.67:0/3183336084 conn(0x7fdee006cb60 msgr2=0x7fdee00cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 shutdown_connections 2024-04-17T07:27:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.735+0000 7fdee5800640 1 -- 172.21.15.67:0/3183336084 wait complete. 2024-04-17T07:27:16.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:27:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.979+0000 7f014e800640 1 Processor -- start 2024-04-17T07:27:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.980+0000 7f014e800640 1 -- start start 2024-04-17T07:27:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014815eb40 0x7f014806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014e800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 0x7f01480d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014e800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014809bbe0 con 0x7f014806d3e0 2024-04-17T07:27:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f01480d1080 con 0x7f014815eb40 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014e800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f01480d0a30 con 0x7f014809b470 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.981+0000 7f014d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014815eb40 0x7f014806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014815eb40 0x7f014806cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35856/0 (socket says 172.21.15.67:35856) 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 -- 172.21.15.67:0/835472956 learned_addr learned my addr 172.21.15.67:0/835472956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014ca00640 1 --2- 172.21.15.67:0/835472956 >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 0x7f01480d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 -- 172.21.15.67:0/835472956 >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 msgr2=0x7f01480d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 --2- 172.21.15.67:0/835472956 >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 0x7f01480d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 -- 172.21.15.67:0/835472956 >> v1:172.21.15.187:6789/0 conn(0x7f014806d3e0 legacy=0x7f014806d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014d400640 1 -- 172.21.15.67:0/835472956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01480d0d10 con 0x7f014815eb40 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.982+0000 7f014ca00640 1 --2- 172.21.15.67:0/835472956 >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 0x7f01480d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.983+0000 7f014d400640 1 --2- 172.21.15.67:0/835472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014815eb40 0x7f014806cda0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f01480b8590 tx=0x7f013802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83142d493703c419 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f0147e00640 1 -- 172.21.15.67:0/835472956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138035020 con 0x7f014815eb40 2024-04-17T07:27:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f0147e00640 1 -- 172.21.15.67:0/835472956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0138031870 con 0x7f014815eb40 2024-04-17T07:27:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f0147e00640 1 -- 172.21.15.67:0/835472956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138031c40 con 0x7f014815eb40 2024-04-17T07:27:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 -- 172.21.15.67:0/835472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014815eb40 msgr2=0x7f014806cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 --2- 172.21.15.67:0/835472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014815eb40 0x7f014806cda0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f01480b8590 tx=0x7f013802b910 comp rx=0 tx=0).stop 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 -- 172.21.15.67:0/835472956 shutdown_connections 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 --2- 172.21.15.67:0/835472956 >> v2:172.21.15.187:3300/0 conn(0x7f014809b470 0x7f01480d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 --2- 172.21.15.67:0/835472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014815eb40 0x7f014806cda0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 -- 172.21.15.67:0/835472956 >> 172.21.15.67:0/835472956 conn(0x7f01480b3fd0 msgr2=0x7f01480b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 -- 172.21.15.67:0/835472956 shutdown_connections 2024-04-17T07:27:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.984+0000 7f014e800640 1 -- 172.21.15.67:0/835472956 wait complete. 2024-04-17T07:27:16.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.986+0000 7f014e800640 1 Processor -- start 2024-04-17T07:27:16.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.986+0000 7f014e800640 1 -- start start 2024-04-17T07:27:16.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f014806d3e0 0x7f01480c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 0x7f01481535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01480d06e0 con 0x7f014809b470 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f014815ef20 con 0x7f014806d3e0 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014e800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01480d1a50 con 0x7f014815eb40 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35872/0 (socket says 172.21.15.67:35872) 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.987+0000 7f014ca00640 1 -- 172.21.15.67:0/2798533954 learned_addr learned my addr 172.21.15.67:0/2798533954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014d400640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f014806d3e0 0x7f01480c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014de00640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 0x7f01481535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 -- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 msgr2=0x7f01481535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 0x7f01481535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 -- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f014806d3e0 msgr2=0x7f01480c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f014806d3e0 0x7f01480c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 -- 172.21.15.67:0/2798533954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01480c8860 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014de00640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 0x7f01481535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.988+0000 7f014ca00640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f013400c7f0 tx=0x7f013400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.989+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134015070 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.989+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01340079d0 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.989+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134007da0 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.989+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01480c8b40 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.989+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01480c9100 con 0x7f014809b470 2024-04-17T07:27:16.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.990+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01480755d0 con 0x7f014809b470 2024-04-17T07:27:16.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.991+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f0134018070 con 0x7f014809b470 2024-04-17T07:27:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.992+0000 7f0145600640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 0x7f012806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:16.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.992+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f013408bdf0 con 0x7f014809b470 2024-04-17T07:27:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.993+0000 7f014d400640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 0x7f012806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.994+0000 7f014d400640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 0x7f012806ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f013803c040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:16.996+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0134059f60 con 0x7f014809b470 2024-04-17T07:27:17.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.242+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f01480709e0 con 0x7f014809b470 2024-04-17T07:27:17.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.882+0000 7f0145600640 1 -- 172.21.15.67:0/2798533954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f0134059860 con 0x7f014809b470 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 msgr2=0x7f012806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 0x7f012806ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f013803c040 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 msgr2=0x7f01481530a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f013400c7f0 tx=0x7f013400cbf0 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 shutdown_connections 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.885+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f012806ca40 0x7f012806ef00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f014815eb40 0x7f01481535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014809b470 0x7f01481530a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 --2- 172.21.15.67:0/2798533954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f014806d3e0 0x7f01480c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 >> 172.21.15.67:0/2798533954 conn(0x7f01480b3fd0 msgr2=0x7f014806ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 shutdown_connections 2024-04-17T07:27:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:17.886+0000 7f014e800640 1 -- 172.21.15.67:0/2798533954 wait complete. 2024-04-17T07:27:17.905 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:27:17.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:27:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.125+0000 7f6433e00640 1 Processor -- start 2024-04-17T07:27:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.126+0000 7f6433e00640 1 -- start start 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.127+0000 7f6433e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340af400 0x7f6434162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.127+0000 7f6433e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f643406b8c0 0x7f643406bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.127+0000 7f6433e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643415f780 con 0x7f6434159020 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.127+0000 7f6433e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f643415f540 con 0x7f643406b8c0 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.127+0000 7f6433e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f643415fa60 con 0x7f64340af400 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6432a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6434159020 0x7f643415ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39004/0 (socket says 172.21.15.67:39004) 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6432a00640 1 -- 172.21.15.67:0/4283123894 learned_addr learned my addr 172.21.15.67:0/4283123894 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6433400640 1 --2- 172.21.15.67:0/4283123894 >> v2:172.21.15.67:3300/0 conn(0x7f643406b8c0 0x7f643406bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f642a000640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340af400 0x7f6434162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6433400640 1 -- 172.21.15.67:0/4283123894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340af400 msgr2=0x7f6434162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6433400640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340af400 0x7f6434162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6432000640 1 -- 172.21.15.67:0/4283123894 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018890193 0 0) 0x7f643415f780 con 0x7f6434159020 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6433400640 1 -- 172.21.15.67:0/4283123894 >> v1:172.21.15.67:6789/0 conn(0x7f6434159020 legacy=0x7f643415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.128+0000 7f6433400640 1 -- 172.21.15.67:0/4283123894 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64341624a0 con 0x7f643406b8c0 2024-04-17T07:27:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6433400640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643406b8c0 0x7f643406bd70 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d733533178f607 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6432000640 1 -- 172.21.15.67:0/4283123894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c039070 con 0x7f643406b8c0 2024-04-17T07:27:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6432000640 1 -- 172.21.15.67:0/4283123894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642c02cbd0 con 0x7f643406b8c0 2024-04-17T07:27:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6432000640 1 -- 172.21.15.67:0/4283123894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c034440 con 0x7f643406b8c0 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6433e00640 1 -- 172.21.15.67:0/4283123894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643406b8c0 msgr2=0x7f643406bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.130+0000 7f6433e00640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643406b8c0 0x7f643406bd70 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c02be20 comp rx=0 tx=0).stop 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 -- 172.21.15.67:0/4283123894 shutdown_connections 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643406b8c0 0x7f643406bd70 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 --2- 172.21.15.67:0/4283123894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340af400 0x7f6434162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 -- 172.21.15.67:0/4283123894 >> 172.21.15.67:0/4283123894 conn(0x7f64340b2190 msgr2=0x7f64340b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 -- 172.21.15.67:0/4283123894 shutdown_connections 2024-04-17T07:27:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.131+0000 7f6433e00640 1 -- 172.21.15.67:0/4283123894 wait complete. 2024-04-17T07:27:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.132+0000 7f6433e00640 1 Processor -- start 2024-04-17T07:27:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 -- start start 2024-04-17T07:27:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643406b8c0 0x7f6434159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6434159020 0x7f643415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64341631e0 con 0x7f64340af400 2024-04-17T07:27:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f643415fd40 con 0x7f643406b8c0 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.133+0000 7f6433e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6434163360 con 0x7f6434159020 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35902/0 (socket says 172.21.15.67:35902) 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 -- 172.21.15.67:0/3565465773 learned_addr learned my addr 172.21.15.67:0/3565465773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f6432a00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643406b8c0 0x7f6434159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f6433400640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6434159020 0x7f643415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 -- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6434159020 msgr2=0x7f643415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6434159020 0x7f643415dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 -- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643406b8c0 msgr2=0x7f6434159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643406b8c0 0x7f6434159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.134+0000 7f642a000640 1 -- 172.21.15.67:0/3565465773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434159970 con 0x7f64340af400 2024-04-17T07:27:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f642a000640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f641c00d800 tx=0x7f641c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:18.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c00b660 con 0x7f64340af400 2024-04-17T07:27:18.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f641c019070 con 0x7f64340af400 2024-04-17T07:27:18.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64341f45d0 con 0x7f64340af400 2024-04-17T07:27:18.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c00ba20 con 0x7f64340af400 2024-04-17T07:27:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.135+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64341f4b90 con 0x7f64340af400 2024-04-17T07:27:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.136+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64340b3790 con 0x7f64340af400 2024-04-17T07:27:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.137+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f641c00bbc0 con 0x7f64340af400 2024-04-17T07:27:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.138+0000 7f642b400640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 0x7f641006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.139+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f641c08c140 con 0x7f64340af400 2024-04-17T07:27:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.139+0000 7f6432a00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 0x7f641006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.140+0000 7f6432a00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 0x7f641006eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6424000f60 tx=0x7f6424009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:18.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.143+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f64340b3790 con 0x7f64340af400 2024-04-17T07:27:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.390+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6434068a10 con 0x7f64340af400 2024-04-17T07:27:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.885+0000 7f642b400640 1 -- 172.21.15.67:0/3565465773 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7f641c01c360 con 0x7f64340af400 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.888+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 msgr2=0x7f641006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 0x7f641006eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6424000f60 tx=0x7f6424009040 comp rx=0 tx=0).stop 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 msgr2=0x7f6434150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f641c00d800 tx=0x7f641c00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 shutdown_connections 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f641006ca10 0x7f641006eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6434159020 0x7f643415dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340af400 0x7f6434150370 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 --2- 172.21.15.67:0/3565465773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643406b8c0 0x7f6434159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 >> 172.21.15.67:0/3565465773 conn(0x7f64340b2190 msgr2=0x7f6434074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 shutdown_connections 2024-04-17T07:27:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:18.889+0000 7f6433e00640 1 -- 172.21.15.67:0/3565465773 wait complete. 2024-04-17T07:27:18.913 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:27:18.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:27:19.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.135+0000 7ff41fe00640 1 Processor -- start 2024-04-17T07:27:19.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.135+0000 7ff41fe00640 1 -- start start 2024-04-17T07:27:19.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.137+0000 7ff41fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff42006b8c0 0x7ff4200cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.137+0000 7ff41fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.137+0000 7ff41fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4200cd4a0 con 0x7ff420160280 2024-04-17T07:27:19.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.137+0000 7ff41fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff42015ce30 con 0x7ff42006b8c0 2024-04-17T07:27:19.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.137+0000 7ff41fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4201607f0 con 0x7ff420141ed0 2024-04-17T07:27:19.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.138+0000 7ff41f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.138+0000 7ff41ea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff42006b8c0 0x7ff4200cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.138+0000 7ff41f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:39896/0 (socket says 172.21.15.67:39896) 2024-04-17T07:27:19.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.138+0000 7ff41f400640 1 -- 172.21.15.67:0/2774182116 learned_addr learned my addr 172.21.15.67:0/2774182116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:19.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.139+0000 7ff41f400640 1 -- 172.21.15.67:0/2774182116 >> v2:172.21.15.187:3300/0 conn(0x7ff42006b8c0 msgr2=0x7ff4200cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:19.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.139+0000 7ff41f400640 1 --2- 172.21.15.67:0/2774182116 >> v2:172.21.15.187:3300/0 conn(0x7ff42006b8c0 0x7ff4200cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.139+0000 7ff41f400640 1 -- 172.21.15.67:0/2774182116 >> v1:172.21.15.187:6789/0 conn(0x7ff420160280 legacy=0x7ff42006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.139+0000 7ff41f400640 1 -- 172.21.15.67:0/2774182116 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff420160ad0 con 0x7ff420141ed0 2024-04-17T07:27:19.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.140+0000 7ff41f400640 1 --2- 172.21.15.67:0/2774182116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff418009e50 tx=0x7ff41802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d635fa5b8acac51 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.141+0000 7ff41e000640 1 -- 172.21.15.67:0/2774182116 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41802cb00 con 0x7ff420141ed0 2024-04-17T07:27:19.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.141+0000 7ff41e000640 1 -- 172.21.15.67:0/2774182116 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff41802cca0 con 0x7ff420141ed0 2024-04-17T07:27:19.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.141+0000 7ff41e000640 1 -- 172.21.15.67:0/2774182116 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff418034610 con 0x7ff420141ed0 2024-04-17T07:27:19.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.141+0000 7ff41fe00640 1 -- 172.21.15.67:0/2774182116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 msgr2=0x7ff420142380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.141+0000 7ff41fe00640 1 --2- 172.21.15.67:0/2774182116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff418009e50 tx=0x7ff41802bf60 comp rx=0 tx=0).stop 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 -- 172.21.15.67:0/2774182116 shutdown_connections 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 --2- 172.21.15.67:0/2774182116 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420141ed0 0x7ff420142380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 --2- 172.21.15.67:0/2774182116 >> v2:172.21.15.187:3300/0 conn(0x7ff42006b8c0 0x7ff4200cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 -- 172.21.15.67:0/2774182116 >> 172.21.15.67:0/2774182116 conn(0x7ff420067d10 msgr2=0x7ff4200b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 -- 172.21.15.67:0/2774182116 shutdown_connections 2024-04-17T07:27:19.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.142+0000 7ff41fe00640 1 -- 172.21.15.67:0/2774182116 wait complete. 2024-04-17T07:27:19.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.144+0000 7ff41fe00640 1 Processor -- start 2024-04-17T07:27:19.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.144+0000 7ff41fe00640 1 -- start start 2024-04-17T07:27:19.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff42006b8c0 0x7ff42015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420160280 0x7ff4201e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff420161090 con 0x7ff420141ed0 2024-04-17T07:27:19.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4200cddf0 con 0x7ff42006b8c0 2024-04-17T07:27:19.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.145+0000 7ff41fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff42013f9b0 con 0x7ff420160280 2024-04-17T07:27:19.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35912/0 (socket says 172.21.15.67:35912) 2024-04-17T07:27:19.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 -- 172.21.15.67:0/689477715 learned_addr learned my addr 172.21.15.67:0/689477715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:19.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff41f400640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420160280 0x7ff4201e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff41ea00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff42006b8c0 0x7ff42015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 -- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420160280 msgr2=0x7ff4201e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420160280 0x7ff4201e80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 -- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff42006b8c0 msgr2=0x7ff42015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff42006b8c0 0x7ff42015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.146+0000 7ff417e00640 1 -- 172.21.15.67:0/689477715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4201e8ad0 con 0x7ff420141ed0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff417e00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff40800c3c0 tx=0x7ff40800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4080052d0 con 0x7ff420141ed0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff408005490 con 0x7ff420141ed0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408007260 con 0x7ff420141ed0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4201e8db0 con 0x7ff420141ed0 2024-04-17T07:27:19.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.147+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4201e8610 con 0x7ff420141ed0 2024-04-17T07:27:19.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.148+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff420068a10 con 0x7ff420141ed0 2024-04-17T07:27:19.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.149+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7ff408007400 con 0x7ff420141ed0 2024-04-17T07:27:19.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.150+0000 7ff416a00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 0x7ff3fc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.151+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7ff40808b190 con 0x7ff420141ed0 2024-04-17T07:27:19.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.151+0000 7ff41ea00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 0x7ff3fc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.152+0000 7ff41ea00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 0x7ff3fc06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff410009ad0 tx=0x7ff410008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.156+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff4080594a0 con 0x7ff420141ed0 2024-04-17T07:27:19.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.399+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4200d0f60 con 0x7ff420141ed0 2024-04-17T07:27:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.401+0000 7ff416a00640 1 -- 172.21.15.67:0/689477715 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7ff408058da0 con 0x7ff420141ed0 2024-04-17T07:27:19.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:19.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:27:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:27:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 msgr2=0x7ff3fc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 0x7ff3fc06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff410009ad0 tx=0x7ff410008040 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 msgr2=0x7ff42012e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff40800c3c0 tx=0x7ff40800c880 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 shutdown_connections 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7ff3fc06ca40 0x7ff3fc06ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff420160280 0x7ff4201e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff420141ed0 0x7ff42012e0e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 --2- 172.21.15.67:0/689477715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff42006b8c0 0x7ff42015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.403+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 >> 172.21.15.67:0/689477715 conn(0x7ff420067d10 msgr2=0x7ff4200b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.404+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 shutdown_connections 2024-04-17T07:27:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.404+0000 7ff41fe00640 1 -- 172.21.15.67:0/689477715 wait complete. 2024-04-17T07:27:19.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:27:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.638+0000 7fdc31000640 1 Processor -- start 2024-04-17T07:27:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.639+0000 7fdc31000640 1 -- start start 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.640+0000 7fdc31000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2c161e70 0x7fdc2c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.640+0000 7fdc31000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc2c0ce4a0 0x7fdc2c162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.640+0000 7fdc31000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc2c0cc500 con 0x7fdc2c0cce90 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.640+0000 7fdc31000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2c0c71e0 con 0x7fdc2c161e70 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.640+0000 7fdc31000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2c0cc7a0 con 0x7fdc2c0ce4a0 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2c161e70 0x7fdc2c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2c161e70 0x7fdc2c0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35924/0 (socket says 172.21.15.67:35924) 2024-04-17T07:27:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 -- 172.21.15.67:0/1863713640 learned_addr learned my addr 172.21.15.67:0/1863713640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29400640 1 --2- 172.21.15.67:0/1863713640 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c0ce4a0 0x7fdc2c162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 -- 172.21.15.67:0/1863713640 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c0ce4a0 msgr2=0x7fdc2c162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 --2- 172.21.15.67:0/1863713640 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c0ce4a0 0x7fdc2c162fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 -- 172.21.15.67:0/1863713640 >> v1:172.21.15.187:6789/0 conn(0x7fdc2c0cce90 legacy=0x7fdc2c0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.641+0000 7fdc29e00640 1 -- 172.21.15.67:0/1863713640 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c0cca80 con 0x7fdc2c161e70 2024-04-17T07:27:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.643+0000 7fdc29e00640 1 --2- 172.21.15.67:0/1863713640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c161e70 0x7fdc2c0c6c00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0701f0 tx=0x7fdc1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915c41e13a638eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.643+0000 7fdc28a00640 1 -- 172.21.15.67:0/1863713640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c035020 con 0x7fdc2c161e70 2024-04-17T07:27:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.643+0000 7fdc28a00640 1 -- 172.21.15.67:0/1863713640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1c031870 con 0x7fdc2c161e70 2024-04-17T07:27:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.643+0000 7fdc28a00640 1 -- 172.21.15.67:0/1863713640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c031c40 con 0x7fdc2c161e70 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 -- 172.21.15.67:0/1863713640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c161e70 msgr2=0x7fdc2c0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 --2- 172.21.15.67:0/1863713640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c161e70 0x7fdc2c0c6c00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0701f0 tx=0x7fdc1c02b910 comp rx=0 tx=0).stop 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 -- 172.21.15.67:0/1863713640 shutdown_connections 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 --2- 172.21.15.67:0/1863713640 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c0ce4a0 0x7fdc2c162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 --2- 172.21.15.67:0/1863713640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c161e70 0x7fdc2c0c6c00 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 -- 172.21.15.67:0/1863713640 >> 172.21.15.67:0/1863713640 conn(0x7fdc2c06bf50 msgr2=0x7fdc2c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 -- 172.21.15.67:0/1863713640 shutdown_connections 2024-04-17T07:27:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.644+0000 7fdc31000640 1 -- 172.21.15.67:0/1863713640 wait complete. 2024-04-17T07:27:19.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.646+0000 7fdc31000640 1 Processor -- start 2024-04-17T07:27:19.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.646+0000 7fdc31000640 1 -- start start 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c0ce4a0 0x7fdc2c0ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c161e70 0x7fdc2c150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c0cd460 con 0x7fdc2c0cce90 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c0cd2e0 con 0x7fdc2c0ce4a0 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc31000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2c0cd5e0 con 0x7fdc2c161e70 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35930/0 (socket says 172.21.15.67:35930) 2024-04-17T07:27:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.647+0000 7fdc29e00640 1 -- 172.21.15.67:0/257310692 learned_addr learned my addr 172.21.15.67:0/257310692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 -- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c161e70 msgr2=0x7fdc2c150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c161e70 0x7fdc2c150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29400640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c0ce4a0 0x7fdc2c0ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 -- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c0ce4a0 msgr2=0x7fdc2c0ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c0ce4a0 0x7fdc2c0ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 -- 172.21.15.67:0/257310692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c0ae820 con 0x7fdc2c0cce90 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc29e00640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000c00 tx=0x7fdc1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.648+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c040070 con 0x7fdc2c0cce90 2024-04-17T07:27:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.649+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1c03a420 con 0x7fdc2c0cce90 2024-04-17T07:27:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.649+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c035040 con 0x7fdc2c0cce90 2024-04-17T07:27:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.649+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c0aeaa0 con 0x7fdc2c0cce90 2024-04-17T07:27:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.649+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c0af060 con 0x7fdc2c0cce90 2024-04-17T07:27:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.650+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c068b90 con 0x7fdc2c0cce90 2024-04-17T07:27:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.651+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fdc1c04b060 con 0x7fdc2c0cce90 2024-04-17T07:27:19.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.652+0000 7fdc22000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:19.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.652+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fdc1c0add10 con 0x7fdc2c0cce90 2024-04-17T07:27:19.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.653+0000 7fdc29400640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.654+0000 7fdc29400640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 0x7fdc0806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc14006fd0 tx=0x7fdc14008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.656+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdc1c0b3020 con 0x7fdc2c0cce90 2024-04-17T07:27:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.861+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc2c0cbb00 con 0x7fdc2c0cce90 2024-04-17T07:27:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.863+0000 7fdc22000640 1 -- 172.21.15.67:0/257310692 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16565 (secure 0 0 0) 0x7fdc2c0ae820 con 0x7fdc2c0cce90 2024-04-17T07:27:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:19.862 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:27:15.855819+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 msgr2=0x7fdc0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 0x7fdc0806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc14006fd0 tx=0x7fdc14008040 comp rx=0 tx=0).stop 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 msgr2=0x7fdc2c155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000c00 tx=0x7fdc1c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 shutdown_connections 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c161e70 0x7fdc2c150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc2c0ce4a0 0x7fdc2c0ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 --2- 172.21.15.67:0/257310692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c0cce90 0x7fdc2c155c20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 >> 172.21.15.67:0/257310692 conn(0x7fdc2c06bf50 msgr2=0x7fdc2c1581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 shutdown_connections 2024-04-17T07:27:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:19.866+0000 7fdc31000640 1 -- 172.21.15.67:0/257310692 wait complete. 2024-04-17T07:27:19.885 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:27:19.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:27:20.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.107+0000 7f50bb200640 1 Processor -- start 2024-04-17T07:27:20.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.108+0000 7f50bb200640 1 -- start start 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50bb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50bc15ec60 0x7f50bc0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50bb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50bc0d2d30 con 0x7f50bc0cfae0 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50bb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50bc0d3010 con 0x7f50bc0b6b30 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50bb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f50bc0d49c0 con 0x7f50bc15ec60 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50b9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50bc15ec60 0x7f50bc0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50ba800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50bc0cfae0 0x7f50bc0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39012/0 (socket says 172.21.15.67:39012) 2024-04-17T07:27:20.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.109+0000 7f50ba800640 1 -- 172.21.15.67:0/2855073043 learned_addr learned my addr 172.21.15.67:0/2855073043 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:20.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.110+0000 7f50b9400640 1 -- 172.21.15.67:0/2855073043 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043540292 0 0) 0x7f50bc0d2d30 con 0x7f50bc0cfae0 2024-04-17T07:27:20.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.110+0000 7f50b9400640 1 -- 172.21.15.67:0/2855073043 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50a0003520 con 0x7f50bc0cfae0 2024-04-17T07:27:20.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.110+0000 7f50b9e00640 1 -- 172.21.15.67:0/2855073043 >> v1:172.21.15.187:6789/0 conn(0x7f50bc0b6b30 legacy=0x7f50bc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:20.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.110+0000 7f50b9e00640 1 -- 172.21.15.67:0/2855073043 >> v1:172.21.15.67:6789/0 conn(0x7f50bc0cfae0 legacy=0x7f50bc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.110+0000 7f50b9e00640 1 -- 172.21.15.67:0/2855073043 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc0d32f0 con 0x7f50bc15ec60 2024-04-17T07:27:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.112+0000 7f50b9e00640 1 --2- 172.21.15.67:0/2855073043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0b65f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0097a0 tx=0x7f50ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30d737ac3fe6979 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.112+0000 7f50b9400640 1 -- 172.21.15.67:0/2855073043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac035020 con 0x7f50bc15ec60 2024-04-17T07:27:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.112+0000 7f50b9400640 1 -- 172.21.15.67:0/2855073043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ac031830 con 0x7f50bc15ec60 2024-04-17T07:27:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.112+0000 7f50b9400640 1 -- 172.21.15.67:0/2855073043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac031be0 con 0x7f50bc15ec60 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 -- 172.21.15.67:0/2855073043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 msgr2=0x7f50bc0b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 --2- 172.21.15.67:0/2855073043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0b65f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0097a0 tx=0x7f50ac02b8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 -- 172.21.15.67:0/2855073043 shutdown_connections 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 --2- 172.21.15.67:0/2855073043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0b65f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 -- 172.21.15.67:0/2855073043 >> 172.21.15.67:0/2855073043 conn(0x7f50bc0d15f0 msgr2=0x7f50bc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 -- 172.21.15.67:0/2855073043 shutdown_connections 2024-04-17T07:27:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.113+0000 7f50bb200640 1 -- 172.21.15.67:0/2855073043 wait complete. 2024-04-17T07:27:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.115+0000 7f50bb200640 1 Processor -- start 2024-04-17T07:27:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.115+0000 7f50bb200640 1 -- start start 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 0x7f50bc160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50bc0cfae0 0x7f50bc160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc0b6f70 con 0x7f50bc15ec60 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f50bc0d3660 con 0x7f50bc0b6b30 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50bb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50bc0d00c0 con 0x7f50bc0cfae0 2024-04-17T07:27:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50ba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50ba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35934/0 (socket says 172.21.15.67:35934) 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50ba800640 1 -- 172.21.15.67:0/1284187651 learned_addr learned my addr 172.21.15.67:0/1284187651 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50b1400640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50bc0cfae0 0x7f50bc160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.116+0000 7f50b9e00640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 0x7f50bc160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 -- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50bc0cfae0 msgr2=0x7f50bc160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50bc0cfae0 0x7f50bc160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 -- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 msgr2=0x7f50bc160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 0x7f50bc160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 -- 172.21.15.67:0/1284187651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50bc0cc8c0 con 0x7f50bc15ec60 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50b9e00640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 0x7f50bc160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50ba800640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f50b400cb70 tx=0x7f50b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4017070 con 0x7f50bc15ec60 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50b400ad40 con 0x7f50bc15ec60 2024-04-17T07:27:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4012620 con 0x7f50bc15ec60 2024-04-17T07:27:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50bc0ccba0 con 0x7f50bc15ec60 2024-04-17T07:27:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.117+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50bc0cd160 con 0x7f50bc15ec60 2024-04-17T07:27:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.118+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50bc0d3af0 con 0x7f50bc15ec60 2024-04-17T07:27:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.120+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f50b401a070 con 0x7f50bc15ec60 2024-04-17T07:27:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.120+0000 7f50b2a00640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 0x7f509806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.121+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f50b408bb80 con 0x7f50bc15ec60 2024-04-17T07:27:20.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.121+0000 7f50b9e00640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 0x7f509806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.123+0000 7f50b9e00640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 0x7f509806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50ac036000 tx=0x7f50ac008030 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.126+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f50bc0d3af0 con 0x7f50bc15ec60 2024-04-17T07:27:20.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.323+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 --> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f50bc068a10 con 0x7f509806ca10 2024-04-17T07:27:20.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.347+0000 7f50b2a00640 1 -- 172.21.15.67:0/1284187651 <== mgr.5134 v2:172.21.15.187:6832/1235852913 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251956 (secure 0 0 0) 0x7f50bc068a10 con 0x7f509806ca10 2024-04-17T07:27:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:20.353 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 msgr2=0x7f509806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 0x7f509806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50ac036000 tx=0x7f50ac008030 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 msgr2=0x7f50bc0cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f50b400cb70 tx=0x7f50b400a430 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 shutdown_connections 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f509806ca10 0x7f509806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50bc15ec60 0x7f50bc0cbc40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50bc0cfae0 0x7f50bc160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 --2- 172.21.15.67:0/1284187651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50bc0b6b30 0x7f50bc160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 >> 172.21.15.67:0/1284187651 conn(0x7f50bc0d15f0 msgr2=0x7f50bc0b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 shutdown_connections 2024-04-17T07:27:20.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.354+0000 7f50bb200640 1 -- 172.21.15.67:0/1284187651 wait complete. 2024-04-17T07:27:20.370 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T07:27:18.838326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309564,"kb_used_data":7336,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754665156,"statfs":{"total":773094113280,"available":772777119744,"internally_reserved":0,"allocated":7512064,"data_stored":4122652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":309317168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.018807"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891034+0000","last_change":"2024-04-17T07:27:14.850104+0000","last_active":"2024-04-17T07:27:15.891034+0000","last_peered":"2024-04-17T07:27:15.891034+0000","last_clean":"2024-04-17T07:27:15.891034+0000","last_became_active":"2024-04-17T07:27:14.849784+0000","last_became_peered":"2024-04-17T07:27:14.849784+0000","last_unstale":"2024-04-17T07:27:15.891034+0000","last_undegraded":"2024-04-17T07:27:15.891034+0000","last_fullsized":"2024-04-17T07:27:15.891034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:18.786520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874000+0000","last_change":"2024-04-17T07:27:14.850674+0000","last_active":"2024-04-17T07:27:15.874000+0000","last_peered":"2024-04-17T07:27:15.874000+0000","last_clean":"2024-04-17T07:27:15.874000+0000","last_became_active":"2024-04-17T07:27:14.850359+0000","last_became_peered":"2024-04-17T07:27:14.850359+0000","last_unstale":"2024-04-17T07:27:15.874000+0000","last_undegraded":"2024-04-17T07:27:15.874000+0000","last_fullsized":"2024-04-17T07:27:15.874000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:36.075466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871118+0000","last_change":"2024-04-17T07:27:14.857351+0000","last_active":"2024-04-17T07:27:15.871118+0000","last_peered":"2024-04-17T07:27:15.871118+0000","last_clean":"2024-04-17T07:27:15.871118+0000","last_became_active":"2024-04-17T07:27:14.857039+0000","last_became_peered":"2024-04-17T07:27:14.857039+0000","last_unstale":"2024-04-17T07:27:15.871118+0000","last_undegraded":"2024-04-17T07:27:15.871118+0000","last_fullsized":"2024-04-17T07:27:15.871118+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:17.548403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869596+0000","last_change":"2024-04-17T07:27:14.866403+0000","last_active":"2024-04-17T07:27:15.869596+0000","last_peered":"2024-04-17T07:27:15.869596+0000","last_clean":"2024-04-17T07:27:15.869596+0000","last_became_active":"2024-04-17T07:27:14.866102+0000","last_became_peered":"2024-04-17T07:27:14.866102+0000","last_unstale":"2024-04-17T07:27:15.869596+0000","last_undegraded":"2024-04-17T07:27:15.869596+0000","last_fullsized":"2024-04-17T07:27:15.869596+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:32:54.077129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873799+0000","last_change":"2024-04-17T07:27:14.856407+0000","last_active":"2024-04-17T07:27:15.873799+0000","last_peered":"2024-04-17T07:27:15.873799+0000","last_clean":"2024-04-17T07:27:15.873799+0000","last_became_active":"2024-04-17T07:27:14.854791+0000","last_became_peered":"2024-04-17T07:27:14.854791+0000","last_unstale":"2024-04-17T07:27:15.873799+0000","last_undegraded":"2024-04-17T07:27:15.873799+0000","last_fullsized":"2024-04-17T07:27:15.873799+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:38.812985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100153+0000","last_change":"2024-04-17T07:27:14.840525+0000","last_active":"2024-04-17T07:27:16.100153+0000","last_peered":"2024-04-17T07:27:16.100153+0000","last_clean":"2024-04-17T07:27:16.100153+0000","last_became_active":"2024-04-17T07:27:14.840109+0000","last_became_peered":"2024-04-17T07:27:14.840109+0000","last_unstale":"2024-04-17T07:27:16.100153+0000","last_undegraded":"2024-04-17T07:27:16.100153+0000","last_fullsized":"2024-04-17T07:27:16.100153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:50.036601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890574+0000","last_change":"2024-04-17T07:27:14.847501+0000","last_active":"2024-04-17T07:27:15.890574+0000","last_peered":"2024-04-17T07:27:15.890574+0000","last_clean":"2024-04-17T07:27:15.890574+0000","last_became_active":"2024-04-17T07:27:14.847182+0000","last_became_peered":"2024-04-17T07:27:14.847182+0000","last_unstale":"2024-04-17T07:27:15.890574+0000","last_undegraded":"2024-04-17T07:27:15.890574+0000","last_fullsized":"2024-04-17T07:27:15.890574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:43.196106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868844+0000","last_change":"2024-04-17T07:27:14.855179+0000","last_active":"2024-04-17T07:27:15.868844+0000","last_peered":"2024-04-17T07:27:15.868844+0000","last_clean":"2024-04-17T07:27:15.868844+0000","last_became_active":"2024-04-17T07:27:14.854862+0000","last_became_peered":"2024-04-17T07:27:14.854862+0000","last_unstale":"2024-04-17T07:27:15.868844+0000","last_undegraded":"2024-04-17T07:27:15.868844+0000","last_fullsized":"2024-04-17T07:27:15.868844+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:12.064243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869946+0000","last_change":"2024-04-17T07:27:14.851225+0000","last_active":"2024-04-17T07:27:15.869946+0000","last_peered":"2024-04-17T07:27:15.869946+0000","last_clean":"2024-04-17T07:27:15.869946+0000","last_became_active":"2024-04-17T07:27:14.849005+0000","last_became_peered":"2024-04-17T07:27:14.849005+0000","last_unstale":"2024-04-17T07:27:15.869946+0000","last_undegraded":"2024-04-17T07:27:15.869946+0000","last_fullsized":"2024-04-17T07:27:15.869946+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:53.364064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869993+0000","last_change":"2024-04-17T07:27:14.866356+0000","last_active":"2024-04-17T07:27:15.869993+0000","last_peered":"2024-04-17T07:27:15.869993+0000","last_clean":"2024-04-17T07:27:15.869993+0000","last_became_active":"2024-04-17T07:27:14.866024+0000","last_became_peered":"2024-04-17T07:27:14.866024+0000","last_unstale":"2024-04-17T07:27:15.869993+0000","last_undegraded":"2024-04-17T07:27:15.869993+0000","last_fullsized":"2024-04-17T07:27:15.869993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:52.549915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870137+0000","last_change":"2024-04-17T07:27:14.845860+0000","last_active":"2024-04-17T07:27:15.870137+0000","last_peered":"2024-04-17T07:27:15.870137+0000","last_clean":"2024-04-17T07:27:15.870137+0000","last_became_active":"2024-04-17T07:27:14.845489+0000","last_became_peered":"2024-04-17T07:27:14.845489+0000","last_unstale":"2024-04-17T07:27:15.870137+0000","last_undegraded":"2024-04-17T07:27:15.870137+0000","last_fullsized":"2024-04-17T07:27:15.870137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:13.892289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873888+0000","last_change":"2024-04-17T07:27:14.844211+0000","last_active":"2024-04-17T07:27:15.873888+0000","last_peered":"2024-04-17T07:27:15.873888+0000","last_clean":"2024-04-17T07:27:15.873888+0000","last_became_active":"2024-04-17T07:27:14.843136+0000","last_became_peered":"2024-04-17T07:27:14.843136+0000","last_unstale":"2024-04-17T07:27:15.873888+0000","last_undegraded":"2024-04-17T07:27:15.873888+0000","last_fullsized":"2024-04-17T07:27:15.873888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:33.102060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100061+0000","last_change":"2024-04-17T07:27:14.841860+0000","last_active":"2024-04-17T07:27:16.100061+0000","last_peered":"2024-04-17T07:27:16.100061+0000","last_clean":"2024-04-17T07:27:16.100061+0000","last_became_active":"2024-04-17T07:27:14.841509+0000","last_became_peered":"2024-04-17T07:27:14.841509+0000","last_unstale":"2024-04-17T07:27:16.100061+0000","last_undegraded":"2024-04-17T07:27:16.100061+0000","last_fullsized":"2024-04-17T07:27:16.100061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:32.186156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869442+0000","last_change":"2024-04-17T07:27:14.850251+0000","last_active":"2024-04-17T07:27:15.869442+0000","last_peered":"2024-04-17T07:27:15.869442+0000","last_clean":"2024-04-17T07:27:15.869442+0000","last_became_active":"2024-04-17T07:27:14.849987+0000","last_became_peered":"2024-04-17T07:27:14.849987+0000","last_unstale":"2024-04-17T07:27:15.869442+0000","last_undegraded":"2024-04-17T07:27:15.869442+0000","last_fullsized":"2024-04-17T07:27:15.869442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:32:43.797253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099664+0000","last_change":"2024-04-17T07:27:14.847912+0000","last_active":"2024-04-17T07:27:16.099664+0000","last_peered":"2024-04-17T07:27:16.099664+0000","last_clean":"2024-04-17T07:27:16.099664+0000","last_became_active":"2024-04-17T07:27:14.847661+0000","last_became_peered":"2024-04-17T07:27:14.847661+0000","last_unstale":"2024-04-17T07:27:16.099664+0000","last_undegraded":"2024-04-17T07:27:16.099664+0000","last_fullsized":"2024-04-17T07:27:16.099664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:23.259168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100319+0000","last_change":"2024-04-17T07:27:14.851680+0000","last_active":"2024-04-17T07:27:16.100319+0000","last_peered":"2024-04-17T07:27:16.100319+0000","last_clean":"2024-04-17T07:27:16.100319+0000","last_became_active":"2024-04-17T07:27:14.851412+0000","last_became_peered":"2024-04-17T07:27:14.851412+0000","last_unstale":"2024-04-17T07:27:16.100319+0000","last_undegraded":"2024-04-17T07:27:16.100319+0000","last_fullsized":"2024-04-17T07:27:16.100319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:34.337772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871417+0000","last_change":"2024-04-17T07:27:14.863921+0000","last_active":"2024-04-17T07:27:15.871417+0000","last_peered":"2024-04-17T07:27:15.871417+0000","last_clean":"2024-04-17T07:27:15.871417+0000","last_became_active":"2024-04-17T07:27:14.863203+0000","last_became_peered":"2024-04-17T07:27:14.863203+0000","last_unstale":"2024-04-17T07:27:15.871417+0000","last_undegraded":"2024-04-17T07:27:15.871417+0000","last_fullsized":"2024-04-17T07:27:15.871417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:45.464925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871344+0000","last_change":"2024-04-17T07:27:14.856184+0000","last_active":"2024-04-17T07:27:15.871344+0000","last_peered":"2024-04-17T07:27:15.871344+0000","last_clean":"2024-04-17T07:27:15.871344+0000","last_became_active":"2024-04-17T07:27:14.855889+0000","last_became_peered":"2024-04-17T07:27:14.855889+0000","last_unstale":"2024-04-17T07:27:15.871344+0000","last_undegraded":"2024-04-17T07:27:15.871344+0000","last_fullsized":"2024-04-17T07:27:15.871344+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:51.116693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101410+0000","last_change":"2024-04-17T07:27:14.858943+0000","last_active":"2024-04-17T07:27:16.101410+0000","last_peered":"2024-04-17T07:27:16.101410+0000","last_clean":"2024-04-17T07:27:16.101410+0000","last_became_active":"2024-04-17T07:27:14.857675+0000","last_became_peered":"2024-04-17T07:27:14.857675+0000","last_unstale":"2024-04-17T07:27:16.101410+0000","last_undegraded":"2024-04-17T07:27:16.101410+0000","last_fullsized":"2024-04-17T07:27:16.101410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:33.261066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101664+0000","last_change":"2024-04-17T07:27:14.845291+0000","last_active":"2024-04-17T07:27:16.101664+0000","last_peered":"2024-04-17T07:27:16.101664+0000","last_clean":"2024-04-17T07:27:16.101664+0000","last_became_active":"2024-04-17T07:27:14.843921+0000","last_became_peered":"2024-04-17T07:27:14.843921+0000","last_unstale":"2024-04-17T07:27:16.101664+0000","last_undegraded":"2024-04-17T07:27:16.101664+0000","last_fullsized":"2024-04-17T07:27:16.101664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:10.937006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100430+0000","last_change":"2024-04-17T07:27:14.854916+0000","last_active":"2024-04-17T07:27:16.100430+0000","last_peered":"2024-04-17T07:27:16.100430+0000","last_clean":"2024-04-17T07:27:16.100430+0000","last_became_active":"2024-04-17T07:27:14.853852+0000","last_became_peered":"2024-04-17T07:27:14.853852+0000","last_unstale":"2024-04-17T07:27:16.100430+0000","last_undegraded":"2024-04-17T07:27:16.100430+0000","last_fullsized":"2024-04-17T07:27:16.100430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:58.078350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869480+0000","last_change":"2024-04-17T07:27:14.854635+0000","last_active":"2024-04-17T07:27:15.869480+0000","last_peered":"2024-04-17T07:27:15.869480+0000","last_clean":"2024-04-17T07:27:15.869480+0000","last_became_active":"2024-04-17T07:27:14.854305+0000","last_became_peered":"2024-04-17T07:27:14.854305+0000","last_unstale":"2024-04-17T07:27:15.869480+0000","last_undegraded":"2024-04-17T07:27:15.869480+0000","last_fullsized":"2024-04-17T07:27:15.869480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:50.778807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100370+0000","last_change":"2024-04-17T07:27:14.850991+0000","last_active":"2024-04-17T07:27:16.100370+0000","last_peered":"2024-04-17T07:27:16.100370+0000","last_clean":"2024-04-17T07:27:16.100370+0000","last_became_active":"2024-04-17T07:27:14.850703+0000","last_became_peered":"2024-04-17T07:27:14.850703+0000","last_unstale":"2024-04-17T07:27:16.100370+0000","last_undegraded":"2024-04-17T07:27:16.100370+0000","last_fullsized":"2024-04-17T07:27:16.100370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:11.944098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100694+0000","last_change":"2024-04-17T07:27:14.854500+0000","last_active":"2024-04-17T07:27:16.100694+0000","last_peered":"2024-04-17T07:27:16.100694+0000","last_clean":"2024-04-17T07:27:16.100694+0000","last_became_active":"2024-04-17T07:27:14.851054+0000","last_became_peered":"2024-04-17T07:27:14.851054+0000","last_unstale":"2024-04-17T07:27:16.100694+0000","last_undegraded":"2024-04-17T07:27:16.100694+0000","last_fullsized":"2024-04-17T07:27:16.100694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:27.880355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101388+0000","last_change":"2024-04-17T07:27:14.859453+0000","last_active":"2024-04-17T07:27:16.101388+0000","last_peered":"2024-04-17T07:27:16.101388+0000","last_clean":"2024-04-17T07:27:16.101388+0000","last_became_active":"2024-04-17T07:27:14.858768+0000","last_became_peered":"2024-04-17T07:27:14.858768+0000","last_unstale":"2024-04-17T07:27:16.101388+0000","last_undegraded":"2024-04-17T07:27:16.101388+0000","last_fullsized":"2024-04-17T07:27:16.101388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:16.990145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100971+0000","last_change":"2024-04-17T07:27:14.841562+0000","last_active":"2024-04-17T07:27:16.100971+0000","last_peered":"2024-04-17T07:27:16.100971+0000","last_clean":"2024-04-17T07:27:16.100971+0000","last_became_active":"2024-04-17T07:27:14.841184+0000","last_became_peered":"2024-04-17T07:27:14.841184+0000","last_unstale":"2024-04-17T07:27:16.100971+0000","last_undegraded":"2024-04-17T07:27:16.100971+0000","last_fullsized":"2024-04-17T07:27:16.100971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.745936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871435+0000","last_change":"2024-04-17T07:27:14.864853+0000","last_active":"2024-04-17T07:27:15.871435+0000","last_peered":"2024-04-17T07:27:15.871435+0000","last_clean":"2024-04-17T07:27:15.871435+0000","last_became_active":"2024-04-17T07:27:14.863242+0000","last_became_peered":"2024-04-17T07:27:14.863242+0000","last_unstale":"2024-04-17T07:27:15.871435+0000","last_undegraded":"2024-04-17T07:27:15.871435+0000","last_fullsized":"2024-04-17T07:27:15.871435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:38.646382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874102+0000","last_change":"2024-04-17T07:27:14.843938+0000","last_active":"2024-04-17T07:27:15.874102+0000","last_peered":"2024-04-17T07:27:15.874102+0000","last_clean":"2024-04-17T07:27:15.874102+0000","last_became_active":"2024-04-17T07:27:14.842663+0000","last_became_peered":"2024-04-17T07:27:14.842663+0000","last_unstale":"2024-04-17T07:27:15.874102+0000","last_undegraded":"2024-04-17T07:27:15.874102+0000","last_fullsized":"2024-04-17T07:27:15.874102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:55:22.035015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100516+0000","last_change":"2024-04-17T07:27:14.856120+0000","last_active":"2024-04-17T07:27:16.100516+0000","last_peered":"2024-04-17T07:27:16.100516+0000","last_clean":"2024-04-17T07:27:16.100516+0000","last_became_active":"2024-04-17T07:27:14.855840+0000","last_became_peered":"2024-04-17T07:27:14.855840+0000","last_unstale":"2024-04-17T07:27:16.100516+0000","last_undegraded":"2024-04-17T07:27:16.100516+0000","last_fullsized":"2024-04-17T07:27:16.100516+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:33.265743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870724+0000","last_change":"2024-04-17T07:27:14.856154+0000","last_active":"2024-04-17T07:27:15.870724+0000","last_peered":"2024-04-17T07:27:15.870724+0000","last_clean":"2024-04-17T07:27:15.870724+0000","last_became_active":"2024-04-17T07:27:14.855810+0000","last_became_peered":"2024-04-17T07:27:14.855810+0000","last_unstale":"2024-04-17T07:27:15.870724+0000","last_undegraded":"2024-04-17T07:27:15.870724+0000","last_fullsized":"2024-04-17T07:27:15.870724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:12:59.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100100+0000","last_change":"2024-04-17T07:27:14.849723+0000","last_active":"2024-04-17T07:27:16.100100+0000","last_peered":"2024-04-17T07:27:16.100100+0000","last_clean":"2024-04-17T07:27:16.100100+0000","last_became_active":"2024-04-17T07:27:14.849412+0000","last_became_peered":"2024-04-17T07:27:14.849412+0000","last_unstale":"2024-04-17T07:27:16.100100+0000","last_undegraded":"2024-04-17T07:27:16.100100+0000","last_fullsized":"2024-04-17T07:27:16.100100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:14.345316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098183+0000","last_change":"2024-04-17T07:27:14.829777+0000","last_active":"2024-04-17T07:27:16.098183+0000","last_peered":"2024-04-17T07:27:16.098183+0000","last_clean":"2024-04-17T07:27:16.098183+0000","last_became_active":"2024-04-17T07:27:14.829261+0000","last_became_peered":"2024-04-17T07:27:14.829261+0000","last_unstale":"2024-04-17T07:27:16.098183+0000","last_undegraded":"2024-04-17T07:27:16.098183+0000","last_fullsized":"2024-04-17T07:27:16.098183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:22.628938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099357+0000","last_change":"2024-04-17T07:27:14.841566+0000","last_active":"2024-04-17T07:27:16.099357+0000","last_peered":"2024-04-17T07:27:16.099357+0000","last_clean":"2024-04-17T07:27:16.099357+0000","last_became_active":"2024-04-17T07:27:14.841286+0000","last_became_peered":"2024-04-17T07:27:14.841286+0000","last_unstale":"2024-04-17T07:27:16.099357+0000","last_undegraded":"2024-04-17T07:27:16.099357+0000","last_fullsized":"2024-04-17T07:27:16.099357+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:24.967416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"65'4","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101492+0000","last_change":"2024-04-17T07:27:13.857238+0000","last_active":"2024-04-17T07:27:16.101492+0000","last_peered":"2024-04-17T07:27:16.101492+0000","last_clean":"2024-04-17T07:27:16.101492+0000","last_became_active":"2024-04-17T07:27:13.856976+0000","last_became_peered":"2024-04-17T07:27:13.856976+0000","last_unstale":"2024-04-17T07:27:16.101492+0000","last_undegraded":"2024-04-17T07:27:16.101492+0000","last_fullsized":"2024-04-17T07:27:16.101492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:18.495414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101365+0000","last_change":"2024-04-17T07:27:14.860028+0000","last_active":"2024-04-17T07:27:16.101365+0000","last_peered":"2024-04-17T07:27:16.101365+0000","last_clean":"2024-04-17T07:27:16.101365+0000","last_became_active":"2024-04-17T07:27:14.859755+0000","last_became_peered":"2024-04-17T07:27:14.859755+0000","last_unstale":"2024-04-17T07:27:16.101365+0000","last_undegraded":"2024-04-17T07:27:16.101365+0000","last_fullsized":"2024-04-17T07:27:16.101365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:21.514275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871682+0000","last_change":"2024-04-17T07:27:13.858318+0000","last_active":"2024-04-17T07:27:15.871682+0000","last_peered":"2024-04-17T07:27:15.871682+0000","last_clean":"2024-04-17T07:27:15.871682+0000","last_became_active":"2024-04-17T07:27:13.857431+0000","last_became_peered":"2024-04-17T07:27:13.857431+0000","last_unstale":"2024-04-17T07:27:15.871682+0000","last_undegraded":"2024-04-17T07:27:15.871682+0000","last_fullsized":"2024-04-17T07:27:15.871682+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:19:58.869362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874442+0000","last_change":"2024-04-17T07:27:14.844434+0000","last_active":"2024-04-17T07:27:15.874442+0000","last_peered":"2024-04-17T07:27:15.874442+0000","last_clean":"2024-04-17T07:27:15.874442+0000","last_became_active":"2024-04-17T07:27:14.843578+0000","last_became_peered":"2024-04-17T07:27:14.843578+0000","last_unstale":"2024-04-17T07:27:15.874442+0000","last_undegraded":"2024-04-17T07:27:15.874442+0000","last_fullsized":"2024-04-17T07:27:15.874442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:23.381721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098991+0000","last_change":"2024-04-17T07:27:13.849492+0000","last_active":"2024-04-17T07:27:16.098991+0000","last_peered":"2024-04-17T07:27:16.098991+0000","last_clean":"2024-04-17T07:27:16.098991+0000","last_became_active":"2024-04-17T07:27:13.849193+0000","last_became_peered":"2024-04-17T07:27:13.849193+0000","last_unstale":"2024-04-17T07:27:16.098991+0000","last_undegraded":"2024-04-17T07:27:16.098991+0000","last_fullsized":"2024-04-17T07:27:16.098991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:47.339048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098862+0000","last_change":"2024-04-17T07:27:14.861388+0000","last_active":"2024-04-17T07:27:16.098862+0000","last_peered":"2024-04-17T07:27:16.098862+0000","last_clean":"2024-04-17T07:27:16.098862+0000","last_became_active":"2024-04-17T07:27:14.861092+0000","last_became_peered":"2024-04-17T07:27:14.861092+0000","last_unstale":"2024-04-17T07:27:16.098862+0000","last_undegraded":"2024-04-17T07:27:16.098862+0000","last_fullsized":"2024-04-17T07:27:16.098862+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:26.261040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890391+0000","last_change":"2024-04-17T07:27:13.852852+0000","last_active":"2024-04-17T07:27:15.890391+0000","last_peered":"2024-04-17T07:27:15.890391+0000","last_clean":"2024-04-17T07:27:15.890391+0000","last_became_active":"2024-04-17T07:27:13.851479+0000","last_became_peered":"2024-04-17T07:27:13.851479+0000","last_unstale":"2024-04-17T07:27:15.890391+0000","last_undegraded":"2024-04-17T07:27:15.890391+0000","last_fullsized":"2024-04-17T07:27:15.890391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:01.491535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870952+0000","last_change":"2024-04-17T07:27:14.861157+0000","last_active":"2024-04-17T07:27:15.870952+0000","last_peered":"2024-04-17T07:27:15.870952+0000","last_clean":"2024-04-17T07:27:15.870952+0000","last_became_active":"2024-04-17T07:27:14.860813+0000","last_became_peered":"2024-04-17T07:27:14.860813+0000","last_unstale":"2024-04-17T07:27:15.870952+0000","last_undegraded":"2024-04-17T07:27:15.870952+0000","last_fullsized":"2024-04-17T07:27:15.870952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:31:07.484053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100661+0000","last_change":"2024-04-17T07:27:13.856181+0000","last_active":"2024-04-17T07:27:16.100661+0000","last_peered":"2024-04-17T07:27:16.100661+0000","last_clean":"2024-04-17T07:27:16.100661+0000","last_became_active":"2024-04-17T07:27:13.855812+0000","last_became_peered":"2024-04-17T07:27:13.855812+0000","last_unstale":"2024-04-17T07:27:16.100661+0000","last_undegraded":"2024-04-17T07:27:16.100661+0000","last_fullsized":"2024-04-17T07:27:16.100661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:32.476602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869587+0000","last_change":"2024-04-17T07:27:14.845314+0000","last_active":"2024-04-17T07:27:15.869587+0000","last_peered":"2024-04-17T07:27:15.869587+0000","last_clean":"2024-04-17T07:27:15.869587+0000","last_became_active":"2024-04-17T07:27:14.844970+0000","last_became_peered":"2024-04-17T07:27:14.844970+0000","last_unstale":"2024-04-17T07:27:15.869587+0000","last_undegraded":"2024-04-17T07:27:15.869587+0000","last_fullsized":"2024-04-17T07:27:15.869587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:52.076888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870818+0000","last_change":"2024-04-17T07:27:13.854450+0000","last_active":"2024-04-17T07:27:15.870818+0000","last_peered":"2024-04-17T07:27:15.870818+0000","last_clean":"2024-04-17T07:27:15.870818+0000","last_became_active":"2024-04-17T07:27:13.854086+0000","last_became_peered":"2024-04-17T07:27:13.854086+0000","last_unstale":"2024-04-17T07:27:15.870818+0000","last_undegraded":"2024-04-17T07:27:15.870818+0000","last_fullsized":"2024-04-17T07:27:15.870818+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:42.646383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868926+0000","last_change":"2024-04-17T07:27:14.854625+0000","last_active":"2024-04-17T07:27:15.868926+0000","last_peered":"2024-04-17T07:27:15.868926+0000","last_clean":"2024-04-17T07:27:15.868926+0000","last_became_active":"2024-04-17T07:27:14.853892+0000","last_became_peered":"2024-04-17T07:27:14.853892+0000","last_unstale":"2024-04-17T07:27:15.868926+0000","last_undegraded":"2024-04-17T07:27:15.868926+0000","last_fullsized":"2024-04-17T07:27:15.868926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:35.235084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873414+0000","last_change":"2024-04-17T07:27:13.836090+0000","last_active":"2024-04-17T07:27:15.873414+0000","last_peered":"2024-04-17T07:27:15.873414+0000","last_clean":"2024-04-17T07:27:15.873414+0000","last_became_active":"2024-04-17T07:27:13.835575+0000","last_became_peered":"2024-04-17T07:27:13.835575+0000","last_unstale":"2024-04-17T07:27:15.873414+0000","last_undegraded":"2024-04-17T07:27:15.873414+0000","last_fullsized":"2024-04-17T07:27:15.873414+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:48.370837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100702+0000","last_change":"2024-04-17T07:27:14.854695+0000","last_active":"2024-04-17T07:27:16.100702+0000","last_peered":"2024-04-17T07:27:16.100702+0000","last_clean":"2024-04-17T07:27:16.100702+0000","last_became_active":"2024-04-17T07:27:14.854045+0000","last_became_peered":"2024-04-17T07:27:14.854045+0000","last_unstale":"2024-04-17T07:27:16.100702+0000","last_undegraded":"2024-04-17T07:27:16.100702+0000","last_fullsized":"2024-04-17T07:27:16.100702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:52.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868888+0000","last_change":"2024-04-17T07:27:13.852535+0000","last_active":"2024-04-17T07:27:15.868888+0000","last_peered":"2024-04-17T07:27:15.868888+0000","last_clean":"2024-04-17T07:27:15.868888+0000","last_became_active":"2024-04-17T07:27:13.852194+0000","last_became_peered":"2024-04-17T07:27:13.852194+0000","last_unstale":"2024-04-17T07:27:15.868888+0000","last_undegraded":"2024-04-17T07:27:15.868888+0000","last_fullsized":"2024-04-17T07:27:15.868888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:30.546379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871557+0000","last_change":"2024-04-17T07:27:14.855804+0000","last_active":"2024-04-17T07:27:15.871557+0000","last_peered":"2024-04-17T07:27:15.871557+0000","last_clean":"2024-04-17T07:27:15.871557+0000","last_became_active":"2024-04-17T07:27:14.855469+0000","last_became_peered":"2024-04-17T07:27:14.855469+0000","last_unstale":"2024-04-17T07:27:15.871557+0000","last_undegraded":"2024-04-17T07:27:15.871557+0000","last_fullsized":"2024-04-17T07:27:15.871557+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:20.172072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871775+0000","last_change":"2024-04-17T07:27:13.849897+0000","last_active":"2024-04-17T07:27:15.871775+0000","last_peered":"2024-04-17T07:27:15.871775+0000","last_clean":"2024-04-17T07:27:15.871775+0000","last_became_active":"2024-04-17T07:27:13.849594+0000","last_became_peered":"2024-04-17T07:27:13.849594+0000","last_unstale":"2024-04-17T07:27:15.871775+0000","last_undegraded":"2024-04-17T07:27:15.871775+0000","last_fullsized":"2024-04-17T07:27:15.871775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:45:56.681690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890875+0000","last_change":"2024-04-17T07:27:14.857423+0000","last_active":"2024-04-17T07:27:15.890875+0000","last_peered":"2024-04-17T07:27:15.890875+0000","last_clean":"2024-04-17T07:27:15.890875+0000","last_became_active":"2024-04-17T07:27:14.857094+0000","last_became_peered":"2024-04-17T07:27:14.857094+0000","last_unstale":"2024-04-17T07:27:15.890875+0000","last_undegraded":"2024-04-17T07:27:15.890875+0000","last_fullsized":"2024-04-17T07:27:15.890875+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:53.000609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870077+0000","last_change":"2024-04-17T07:27:13.853190+0000","last_active":"2024-04-17T07:27:15.870077+0000","last_peered":"2024-04-17T07:27:15.870077+0000","last_clean":"2024-04-17T07:27:15.870077+0000","last_became_active":"2024-04-17T07:27:13.852035+0000","last_became_peered":"2024-04-17T07:27:13.852035+0000","last_unstale":"2024-04-17T07:27:15.870077+0000","last_undegraded":"2024-04-17T07:27:15.870077+0000","last_fullsized":"2024-04-17T07:27:15.870077+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:26.307009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100490+0000","last_change":"2024-04-17T07:27:14.843845+0000","last_active":"2024-04-17T07:27:16.100490+0000","last_peered":"2024-04-17T07:27:16.100490+0000","last_clean":"2024-04-17T07:27:16.100490+0000","last_became_active":"2024-04-17T07:27:14.843344+0000","last_became_peered":"2024-04-17T07:27:14.843344+0000","last_unstale":"2024-04-17T07:27:16.100490+0000","last_undegraded":"2024-04-17T07:27:16.100490+0000","last_fullsized":"2024-04-17T07:27:16.100490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:11.355030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"65'3","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891041+0000","last_change":"2024-04-17T07:27:13.854788+0000","last_active":"2024-04-17T07:27:15.891041+0000","last_peered":"2024-04-17T07:27:15.891041+0000","last_clean":"2024-04-17T07:27:15.891041+0000","last_became_active":"2024-04-17T07:27:13.854298+0000","last_became_peered":"2024-04-17T07:27:13.854298+0000","last_unstale":"2024-04-17T07:27:15.891041+0000","last_undegraded":"2024-04-17T07:27:15.891041+0000","last_fullsized":"2024-04-17T07:27:15.891041+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:24.337854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890903+0000","last_change":"2024-04-17T07:27:14.846915+0000","last_active":"2024-04-17T07:27:15.890903+0000","last_peered":"2024-04-17T07:27:15.890903+0000","last_clean":"2024-04-17T07:27:15.890903+0000","last_became_active":"2024-04-17T07:27:14.846328+0000","last_became_peered":"2024-04-17T07:27:14.846328+0000","last_unstale":"2024-04-17T07:27:15.890903+0000","last_undegraded":"2024-04-17T07:27:15.890903+0000","last_fullsized":"2024-04-17T07:27:15.890903+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:29.636974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"65'6","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100501+0000","last_change":"2024-04-17T07:27:13.849239+0000","last_active":"2024-04-17T07:27:16.100501+0000","last_peered":"2024-04-17T07:27:16.100501+0000","last_clean":"2024-04-17T07:27:16.100501+0000","last_became_active":"2024-04-17T07:27:13.848540+0000","last_became_peered":"2024-04-17T07:27:13.848540+0000","last_unstale":"2024-04-17T07:27:16.100501+0000","last_undegraded":"2024-04-17T07:27:16.100501+0000","last_fullsized":"2024-04-17T07:27:16.100501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:44:50.889450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100684+0000","last_change":"2024-04-17T07:27:14.864103+0000","last_active":"2024-04-17T07:27:16.100684+0000","last_peered":"2024-04-17T07:27:16.100684+0000","last_clean":"2024-04-17T07:27:16.100684+0000","last_became_active":"2024-04-17T07:27:14.863547+0000","last_became_peered":"2024-04-17T07:27:14.863547+0000","last_unstale":"2024-04-17T07:27:16.100684+0000","last_undegraded":"2024-04-17T07:27:16.100684+0000","last_fullsized":"2024-04-17T07:27:16.100684+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:32.088976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100757+0000","last_change":"2024-04-17T07:27:13.854642+0000","last_active":"2024-04-17T07:27:16.100757+0000","last_peered":"2024-04-17T07:27:16.100757+0000","last_clean":"2024-04-17T07:27:16.100757+0000","last_became_active":"2024-04-17T07:27:13.854116+0000","last_became_peered":"2024-04-17T07:27:13.854116+0000","last_unstale":"2024-04-17T07:27:16.100757+0000","last_undegraded":"2024-04-17T07:27:16.100757+0000","last_fullsized":"2024-04-17T07:27:16.100757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:33.510590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869595+0000","last_change":"2024-04-17T07:27:14.842733+0000","last_active":"2024-04-17T07:27:15.869595+0000","last_peered":"2024-04-17T07:27:15.869595+0000","last_clean":"2024-04-17T07:27:15.869595+0000","last_became_active":"2024-04-17T07:27:14.842479+0000","last_became_peered":"2024-04-17T07:27:14.842479+0000","last_unstale":"2024-04-17T07:27:15.869595+0000","last_undegraded":"2024-04-17T07:27:15.869595+0000","last_fullsized":"2024-04-17T07:27:15.869595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:41.834950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890230+0000","last_change":"2024-04-17T07:27:13.851191+0000","last_active":"2024-04-17T07:27:15.890230+0000","last_peered":"2024-04-17T07:27:15.890230+0000","last_clean":"2024-04-17T07:27:15.890230+0000","last_became_active":"2024-04-17T07:27:13.850898+0000","last_became_peered":"2024-04-17T07:27:13.850898+0000","last_unstale":"2024-04-17T07:27:15.890230+0000","last_undegraded":"2024-04-17T07:27:15.890230+0000","last_fullsized":"2024-04-17T07:27:15.890230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:20.393223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099607+0000","last_change":"2024-04-17T07:27:14.842821+0000","last_active":"2024-04-17T07:27:16.099607+0000","last_peered":"2024-04-17T07:27:16.099607+0000","last_clean":"2024-04-17T07:27:16.099607+0000","last_became_active":"2024-04-17T07:27:14.842468+0000","last_became_peered":"2024-04-17T07:27:14.842468+0000","last_unstale":"2024-04-17T07:27:16.099607+0000","last_undegraded":"2024-04-17T07:27:16.099607+0000","last_fullsized":"2024-04-17T07:27:16.099607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:02.510156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099812+0000","last_change":"2024-04-17T07:27:13.855175+0000","last_active":"2024-04-17T07:27:16.099812+0000","last_peered":"2024-04-17T07:27:16.099812+0000","last_clean":"2024-04-17T07:27:16.099812+0000","last_became_active":"2024-04-17T07:27:13.854416+0000","last_became_peered":"2024-04-17T07:27:13.854416+0000","last_unstale":"2024-04-17T07:27:16.099812+0000","last_undegraded":"2024-04-17T07:27:16.099812+0000","last_fullsized":"2024-04-17T07:27:16.099812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:37.794364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890580+0000","last_change":"2024-04-17T07:27:14.853138+0000","last_active":"2024-04-17T07:27:15.890580+0000","last_peered":"2024-04-17T07:27:15.890580+0000","last_clean":"2024-04-17T07:27:15.890580+0000","last_became_active":"2024-04-17T07:27:14.852812+0000","last_became_peered":"2024-04-17T07:27:14.852812+0000","last_unstale":"2024-04-17T07:27:15.890580+0000","last_undegraded":"2024-04-17T07:27:15.890580+0000","last_fullsized":"2024-04-17T07:27:15.890580+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:35:52.840396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"65'8","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869021+0000","last_change":"2024-04-17T07:27:13.857020+0000","last_active":"2024-04-17T07:27:15.869021+0000","last_peered":"2024-04-17T07:27:15.869021+0000","last_clean":"2024-04-17T07:27:15.869021+0000","last_became_active":"2024-04-17T07:27:13.854881+0000","last_became_peered":"2024-04-17T07:27:13.854881+0000","last_unstale":"2024-04-17T07:27:15.869021+0000","last_undegraded":"2024-04-17T07:27:15.869021+0000","last_fullsized":"2024-04-17T07:27:15.869021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:38.918738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101363+0000","last_change":"2024-04-17T07:27:14.844212+0000","last_active":"2024-04-17T07:27:16.101363+0000","last_peered":"2024-04-17T07:27:16.101363+0000","last_clean":"2024-04-17T07:27:16.101363+0000","last_became_active":"2024-04-17T07:27:14.843895+0000","last_became_peered":"2024-04-17T07:27:14.843895+0000","last_unstale":"2024-04-17T07:27:16.101363+0000","last_undegraded":"2024-04-17T07:27:16.101363+0000","last_fullsized":"2024-04-17T07:27:16.101363+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:12:55.331688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099111+0000","last_change":"2024-04-17T07:27:13.845685+0000","last_active":"2024-04-17T07:27:16.099111+0000","last_peered":"2024-04-17T07:27:16.099111+0000","last_clean":"2024-04-17T07:27:16.099111+0000","last_became_active":"2024-04-17T07:27:13.845360+0000","last_became_peered":"2024-04-17T07:27:13.845360+0000","last_unstale":"2024-04-17T07:27:16.099111+0000","last_undegraded":"2024-04-17T07:27:16.099111+0000","last_fullsized":"2024-04-17T07:27:16.099111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:46.712162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870044+0000","last_change":"2024-04-17T07:27:14.846632+0000","last_active":"2024-04-17T07:27:15.870044+0000","last_peered":"2024-04-17T07:27:15.870044+0000","last_clean":"2024-04-17T07:27:15.870044+0000","last_became_active":"2024-04-17T07:27:14.846345+0000","last_became_peered":"2024-04-17T07:27:14.846345+0000","last_unstale":"2024-04-17T07:27:15.870044+0000","last_undegraded":"2024-04-17T07:27:15.870044+0000","last_fullsized":"2024-04-17T07:27:15.870044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:02.450730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871825+0000","last_change":"2024-04-17T07:27:13.847245+0000","last_active":"2024-04-17T07:27:15.871825+0000","last_peered":"2024-04-17T07:27:15.871825+0000","last_clean":"2024-04-17T07:27:15.871825+0000","last_became_active":"2024-04-17T07:27:13.846942+0000","last_became_peered":"2024-04-17T07:27:13.846942+0000","last_unstale":"2024-04-17T07:27:15.871825+0000","last_undegraded":"2024-04-17T07:27:15.871825+0000","last_fullsized":"2024-04-17T07:27:15.871825+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:44.946243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100749+0000","last_change":"2024-04-17T07:27:14.857652+0000","last_active":"2024-04-17T07:27:16.100749+0000","last_peered":"2024-04-17T07:27:16.100749+0000","last_clean":"2024-04-17T07:27:16.100749+0000","last_became_active":"2024-04-17T07:27:14.857146+0000","last_became_peered":"2024-04-17T07:27:14.857146+0000","last_unstale":"2024-04-17T07:27:16.100749+0000","last_undegraded":"2024-04-17T07:27:16.100749+0000","last_fullsized":"2024-04-17T07:27:16.100749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:29.764710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874341+0000","last_change":"2024-04-17T07:27:13.847077+0000","last_active":"2024-04-17T07:27:15.874341+0000","last_peered":"2024-04-17T07:27:15.874341+0000","last_clean":"2024-04-17T07:27:15.874341+0000","last_became_active":"2024-04-17T07:27:13.842412+0000","last_became_peered":"2024-04-17T07:27:13.842412+0000","last_unstale":"2024-04-17T07:27:15.874341+0000","last_undegraded":"2024-04-17T07:27:15.874341+0000","last_fullsized":"2024-04-17T07:27:15.874341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:21.242882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101070+0000","last_change":"2024-04-17T07:27:14.855342+0000","last_active":"2024-04-17T07:27:16.101070+0000","last_peered":"2024-04-17T07:27:16.101070+0000","last_clean":"2024-04-17T07:27:16.101070+0000","last_became_active":"2024-04-17T07:27:14.854997+0000","last_became_peered":"2024-04-17T07:27:14.854997+0000","last_unstale":"2024-04-17T07:27:16.101070+0000","last_undegraded":"2024-04-17T07:27:16.101070+0000","last_fullsized":"2024-04-17T07:27:16.101070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:25.534091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100547+0000","last_change":"2024-04-17T07:27:13.856675+0000","last_active":"2024-04-17T07:27:16.100547+0000","last_peered":"2024-04-17T07:27:16.100547+0000","last_clean":"2024-04-17T07:27:16.100547+0000","last_became_active":"2024-04-17T07:27:13.856394+0000","last_became_peered":"2024-04-17T07:27:13.856394+0000","last_unstale":"2024-04-17T07:27:16.100547+0000","last_undegraded":"2024-04-17T07:27:16.100547+0000","last_fullsized":"2024-04-17T07:27:16.100547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:14.344042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100208+0000","last_change":"2024-04-17T07:27:14.843225+0000","last_active":"2024-04-17T07:27:16.100208+0000","last_peered":"2024-04-17T07:27:16.100208+0000","last_clean":"2024-04-17T07:27:16.100208+0000","last_became_active":"2024-04-17T07:27:14.842912+0000","last_became_peered":"2024-04-17T07:27:14.842912+0000","last_unstale":"2024-04-17T07:27:16.100208+0000","last_undegraded":"2024-04-17T07:27:16.100208+0000","last_fullsized":"2024-04-17T07:27:16.100208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:25.356475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100913+0000","last_change":"2024-04-17T07:27:13.855887+0000","last_active":"2024-04-17T07:27:16.100913+0000","last_peered":"2024-04-17T07:27:16.100913+0000","last_clean":"2024-04-17T07:27:16.100913+0000","last_became_active":"2024-04-17T07:27:13.855595+0000","last_became_peered":"2024-04-17T07:27:13.855595+0000","last_unstale":"2024-04-17T07:27:16.100913+0000","last_undegraded":"2024-04-17T07:27:16.100913+0000","last_fullsized":"2024-04-17T07:27:16.100913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:44:55.683309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098826+0000","last_change":"2024-04-17T07:27:14.840379+0000","last_active":"2024-04-17T07:27:16.098826+0000","last_peered":"2024-04-17T07:27:16.098826+0000","last_clean":"2024-04-17T07:27:16.098826+0000","last_became_active":"2024-04-17T07:27:14.840089+0000","last_became_peered":"2024-04-17T07:27:14.840089+0000","last_unstale":"2024-04-17T07:27:16.098826+0000","last_undegraded":"2024-04-17T07:27:16.098826+0000","last_fullsized":"2024-04-17T07:27:16.098826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:43.485046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873400+0000","last_change":"2024-04-17T07:27:13.852469+0000","last_active":"2024-04-17T07:27:15.873400+0000","last_peered":"2024-04-17T07:27:15.873400+0000","last_clean":"2024-04-17T07:27:15.873400+0000","last_became_active":"2024-04-17T07:27:13.851418+0000","last_became_peered":"2024-04-17T07:27:13.851418+0000","last_unstale":"2024-04-17T07:27:15.873400+0000","last_undegraded":"2024-04-17T07:27:15.873400+0000","last_fullsized":"2024-04-17T07:27:15.873400+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:02:31.421337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099149+0000","last_change":"2024-04-17T07:27:14.835711+0000","last_active":"2024-04-17T07:27:16.099149+0000","last_peered":"2024-04-17T07:27:16.099149+0000","last_clean":"2024-04-17T07:27:16.099149+0000","last_became_active":"2024-04-17T07:27:14.835097+0000","last_became_peered":"2024-04-17T07:27:14.835097+0000","last_unstale":"2024-04-17T07:27:16.099149+0000","last_undegraded":"2024-04-17T07:27:16.099149+0000","last_fullsized":"2024-04-17T07:27:16.099149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:15.766689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869159+0000","last_change":"2024-04-17T07:27:13.857309+0000","last_active":"2024-04-17T07:27:15.869159+0000","last_peered":"2024-04-17T07:27:15.869159+0000","last_clean":"2024-04-17T07:27:15.869159+0000","last_became_active":"2024-04-17T07:27:13.857020+0000","last_became_peered":"2024-04-17T07:27:13.857020+0000","last_unstale":"2024-04-17T07:27:15.869159+0000","last_undegraded":"2024-04-17T07:27:15.869159+0000","last_fullsized":"2024-04-17T07:27:15.869159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.333114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869011+0000","last_change":"2024-04-17T07:27:14.846546+0000","last_active":"2024-04-17T07:27:15.869011+0000","last_peered":"2024-04-17T07:27:15.869011+0000","last_clean":"2024-04-17T07:27:15.869011+0000","last_became_active":"2024-04-17T07:27:14.846194+0000","last_became_peered":"2024-04-17T07:27:14.846194+0000","last_unstale":"2024-04-17T07:27:15.869011+0000","last_undegraded":"2024-04-17T07:27:15.869011+0000","last_fullsized":"2024-04-17T07:27:15.869011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:44.394197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.889774+0000","last_change":"2024-04-17T07:27:13.849552+0000","last_active":"2024-04-17T07:27:15.889774+0000","last_peered":"2024-04-17T07:27:15.889774+0000","last_clean":"2024-04-17T07:27:15.889774+0000","last_became_active":"2024-04-17T07:27:13.849194+0000","last_became_peered":"2024-04-17T07:27:13.849194+0000","last_unstale":"2024-04-17T07:27:15.889774+0000","last_undegraded":"2024-04-17T07:27:15.889774+0000","last_fullsized":"2024-04-17T07:27:15.889774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:15.682578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101615+0000","last_change":"2024-04-17T07:27:14.858105+0000","last_active":"2024-04-17T07:27:16.101615+0000","last_peered":"2024-04-17T07:27:16.101615+0000","last_clean":"2024-04-17T07:27:16.101615+0000","last_became_active":"2024-04-17T07:27:14.857040+0000","last_became_peered":"2024-04-17T07:27:14.857040+0000","last_unstale":"2024-04-17T07:27:16.101615+0000","last_undegraded":"2024-04-17T07:27:16.101615+0000","last_fullsized":"2024-04-17T07:27:16.101615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:14.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100816+0000","last_change":"2024-04-17T07:27:13.850499+0000","last_active":"2024-04-17T07:27:16.100816+0000","last_peered":"2024-04-17T07:27:16.100816+0000","last_clean":"2024-04-17T07:27:16.100816+0000","last_became_active":"2024-04-17T07:27:13.849205+0000","last_became_peered":"2024-04-17T07:27:13.849205+0000","last_unstale":"2024-04-17T07:27:16.100816+0000","last_undegraded":"2024-04-17T07:27:16.100816+0000","last_fullsized":"2024-04-17T07:27:16.100816+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:20.393223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869807+0000","last_change":"2024-04-17T07:27:14.846354+0000","last_active":"2024-04-17T07:27:15.869807+0000","last_peered":"2024-04-17T07:27:15.869807+0000","last_clean":"2024-04-17T07:27:15.869807+0000","last_became_active":"2024-04-17T07:27:14.846084+0000","last_became_peered":"2024-04-17T07:27:14.846084+0000","last_unstale":"2024-04-17T07:27:15.869807+0000","last_undegraded":"2024-04-17T07:27:15.869807+0000","last_fullsized":"2024-04-17T07:27:15.869807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:10.916043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891244+0000","last_change":"2024-04-17T07:27:13.855123+0000","last_active":"2024-04-17T07:27:15.891244+0000","last_peered":"2024-04-17T07:27:15.891244+0000","last_clean":"2024-04-17T07:27:15.891244+0000","last_became_active":"2024-04-17T07:27:13.854212+0000","last_became_peered":"2024-04-17T07:27:13.854212+0000","last_unstale":"2024-04-17T07:27:15.891244+0000","last_undegraded":"2024-04-17T07:27:15.891244+0000","last_fullsized":"2024-04-17T07:27:15.891244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:57.059729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891088+0000","last_change":"2024-04-17T07:27:14.842027+0000","last_active":"2024-04-17T07:27:15.891088+0000","last_peered":"2024-04-17T07:27:15.891088+0000","last_clean":"2024-04-17T07:27:15.891088+0000","last_became_active":"2024-04-17T07:27:14.841669+0000","last_became_peered":"2024-04-17T07:27:14.841669+0000","last_unstale":"2024-04-17T07:27:15.891088+0000","last_undegraded":"2024-04-17T07:27:15.891088+0000","last_fullsized":"2024-04-17T07:27:15.891088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:11.641523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101019+0000","last_change":"2024-04-17T07:27:13.850353+0000","last_active":"2024-04-17T07:27:16.101019+0000","last_peered":"2024-04-17T07:27:16.101019+0000","last_clean":"2024-04-17T07:27:16.101019+0000","last_became_active":"2024-04-17T07:27:13.850003+0000","last_became_peered":"2024-04-17T07:27:13.850003+0000","last_unstale":"2024-04-17T07:27:16.101019+0000","last_undegraded":"2024-04-17T07:27:16.101019+0000","last_fullsized":"2024-04-17T07:27:16.101019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:51.501836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100429+0000","last_change":"2024-04-17T07:27:14.839822+0000","last_active":"2024-04-17T07:27:16.100429+0000","last_peered":"2024-04-17T07:27:16.100429+0000","last_clean":"2024-04-17T07:27:16.100429+0000","last_became_active":"2024-04-17T07:27:14.839492+0000","last_became_peered":"2024-04-17T07:27:14.839492+0000","last_unstale":"2024-04-17T07:27:16.100429+0000","last_undegraded":"2024-04-17T07:27:16.100429+0000","last_fullsized":"2024-04-17T07:27:16.100429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:55:09.728826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890388+0000","last_change":"2024-04-17T07:27:13.857311+0000","last_active":"2024-04-17T07:27:15.890388+0000","last_peered":"2024-04-17T07:27:15.890388+0000","last_clean":"2024-04-17T07:27:15.890388+0000","last_became_active":"2024-04-17T07:27:13.856977+0000","last_became_peered":"2024-04-17T07:27:13.856977+0000","last_unstale":"2024-04-17T07:27:15.890388+0000","last_undegraded":"2024-04-17T07:27:15.890388+0000","last_fullsized":"2024-04-17T07:27:15.890388+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:49.017980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100609+0000","last_change":"2024-04-17T07:27:14.844776+0000","last_active":"2024-04-17T07:27:16.100609+0000","last_peered":"2024-04-17T07:27:16.100609+0000","last_clean":"2024-04-17T07:27:16.100609+0000","last_became_active":"2024-04-17T07:27:14.844528+0000","last_became_peered":"2024-04-17T07:27:14.844528+0000","last_unstale":"2024-04-17T07:27:16.100609+0000","last_undegraded":"2024-04-17T07:27:16.100609+0000","last_fullsized":"2024-04-17T07:27:16.100609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:11.677344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870976+0000","last_change":"2024-04-17T07:27:13.853202+0000","last_active":"2024-04-17T07:27:15.870976+0000","last_peered":"2024-04-17T07:27:15.870976+0000","last_clean":"2024-04-17T07:27:15.870976+0000","last_became_active":"2024-04-17T07:27:13.852669+0000","last_became_peered":"2024-04-17T07:27:13.852669+0000","last_unstale":"2024-04-17T07:27:15.870976+0000","last_undegraded":"2024-04-17T07:27:15.870976+0000","last_fullsized":"2024-04-17T07:27:15.870976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:36.199769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874051+0000","last_change":"2024-04-17T07:27:14.854577+0000","last_active":"2024-04-17T07:27:15.874051+0000","last_peered":"2024-04-17T07:27:15.874051+0000","last_clean":"2024-04-17T07:27:15.874051+0000","last_became_active":"2024-04-17T07:27:14.854276+0000","last_became_peered":"2024-04-17T07:27:14.854276+0000","last_unstale":"2024-04-17T07:27:15.874051+0000","last_undegraded":"2024-04-17T07:27:15.874051+0000","last_fullsized":"2024-04-17T07:27:15.874051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:10.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098408+0000","last_change":"2024-04-17T07:27:13.850748+0000","last_active":"2024-04-17T07:27:16.098408+0000","last_peered":"2024-04-17T07:27:16.098408+0000","last_clean":"2024-04-17T07:27:16.098408+0000","last_became_active":"2024-04-17T07:27:13.849092+0000","last_became_peered":"2024-04-17T07:27:13.849092+0000","last_unstale":"2024-04-17T07:27:16.098408+0000","last_undegraded":"2024-04-17T07:27:16.098408+0000","last_fullsized":"2024-04-17T07:27:16.098408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:36.621658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870838+0000","last_change":"2024-04-17T07:27:14.854138+0000","last_active":"2024-04-17T07:27:15.870838+0000","last_peered":"2024-04-17T07:27:15.870838+0000","last_clean":"2024-04-17T07:27:15.870838+0000","last_became_active":"2024-04-17T07:27:14.853769+0000","last_became_peered":"2024-04-17T07:27:14.853769+0000","last_unstale":"2024-04-17T07:27:15.870838+0000","last_undegraded":"2024-04-17T07:27:15.870838+0000","last_fullsized":"2024-04-17T07:27:15.870838+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:41.523336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869022+0000","last_change":"2024-04-17T07:27:13.846014+0000","last_active":"2024-04-17T07:27:15.869022+0000","last_peered":"2024-04-17T07:27:15.869022+0000","last_clean":"2024-04-17T07:27:15.869022+0000","last_became_active":"2024-04-17T07:27:13.845656+0000","last_became_peered":"2024-04-17T07:27:13.845656+0000","last_unstale":"2024-04-17T07:27:15.869022+0000","last_undegraded":"2024-04-17T07:27:15.869022+0000","last_fullsized":"2024-04-17T07:27:15.869022+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:23:22.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870074+0000","last_change":"2024-04-17T07:27:14.841602+0000","last_active":"2024-04-17T07:27:15.870074+0000","last_peered":"2024-04-17T07:27:15.870074+0000","last_clean":"2024-04-17T07:27:15.870074+0000","last_became_active":"2024-04-17T07:27:14.841228+0000","last_became_peered":"2024-04-17T07:27:14.841228+0000","last_unstale":"2024-04-17T07:27:15.870074+0000","last_undegraded":"2024-04-17T07:27:15.870074+0000","last_fullsized":"2024-04-17T07:27:15.870074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:01.370852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873993+0000","last_change":"2024-04-17T07:27:13.846264+0000","last_active":"2024-04-17T07:27:15.873993+0000","last_peered":"2024-04-17T07:27:15.873993+0000","last_clean":"2024-04-17T07:27:15.873993+0000","last_became_active":"2024-04-17T07:27:13.845951+0000","last_became_peered":"2024-04-17T07:27:13.845951+0000","last_unstale":"2024-04-17T07:27:15.873993+0000","last_undegraded":"2024-04-17T07:27:15.873993+0000","last_fullsized":"2024-04-17T07:27:15.873993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:39:13.218154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'139","reported_seq":339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.889967+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:27:15.889967+0000","last_peered":"2024-04-17T07:27:15.889967+0000","last_clean":"2024-04-17T07:27:15.889967+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:27:15.889967+0000","last_undegraded":"2024-04-17T07:27:15.889967+0000","last_fullsized":"2024-04-17T07:27:15.889967+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542308,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1380352,"data_stored":957595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":790528,"data_stored":367809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":786432,"data_stored":367207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542307,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58336,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313504,"statfs":{"total":96636764160,"available":96577028096,"internally_reserved":0,"allocated":794624,"data_stored":368749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542308,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":367775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40284,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331556,"statfs":{"total":96636764160,"available":96595513344,"internally_reserved":0,"allocated":798720,"data_stored":368771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542308,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50004,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321836,"statfs":{"total":96636764160,"available":96585560064,"internally_reserved":0,"allocated":786432,"data_stored":367207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542307,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1380352,"data_stored":957539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:27:20.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:27:20.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.632+0000 7f60e3e00640 1 Processor -- start 2024-04-17T07:27:20.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.632+0000 7f60e3e00640 1 -- start start 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 0x7f60e415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60e40b4210 0x7f60e40af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e40b46c0 0x7f60e40af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f60e415f440 con 0x7f60e40b4210 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f60e415f720 con 0x7f60e415c5a0 2024-04-17T07:27:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f60e415fa00 con 0x7f60e40b46c0 2024-04-17T07:27:20.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60e40b4210 0x7f60e40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60e40b4210 0x7f60e40af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35936/0 (socket says 172.21.15.67:35936) 2024-04-17T07:27:20.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.634+0000 7f60e2000640 1 -- 172.21.15.67:0/2469345153 learned_addr learned my addr 172.21.15.67:0/2469345153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e3400640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e40b46c0 0x7f60e40af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2a00640 1 --2- 172.21.15.67:0/2469345153 >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 0x7f60e415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2000640 1 -- 172.21.15.67:0/2469345153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e40b46c0 msgr2=0x7f60e40af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2000640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e40b46c0 0x7f60e40af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2000640 1 -- 172.21.15.67:0/2469345153 >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 msgr2=0x7f60e415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2000640 1 --2- 172.21.15.67:0/2469345153 >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 0x7f60e415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2000640 1 -- 172.21.15.67:0/2469345153 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e415fce0 con 0x7f60e40b4210 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.635+0000 7f60e2a00640 1 --2- 172.21.15.67:0/2469345153 >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 0x7f60e415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:20.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.636+0000 7f60e2000640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e40b4210 0x7f60e40af440 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60e40b3010 tx=0x7f60d002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb73051a0adf9b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.636+0000 7f60e1600640 1 -- 172.21.15.67:0/2469345153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0038070 con 0x7f60e40b4210 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.636+0000 7f60e1600640 1 -- 172.21.15.67:0/2469345153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d0031740 con 0x7f60e40b4210 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.636+0000 7f60e1600640 1 -- 172.21.15.67:0/2469345153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d002f050 con 0x7f60e40b4210 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 -- 172.21.15.67:0/2469345153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e40b4210 msgr2=0x7f60e40af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e40b4210 0x7f60e40af440 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60e40b3010 tx=0x7f60d002b7e0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 -- 172.21.15.67:0/2469345153 shutdown_connections 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e40b46c0 0x7f60e40af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 --2- 172.21.15.67:0/2469345153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e40b4210 0x7f60e40af440 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 --2- 172.21.15.67:0/2469345153 >> v2:172.21.15.187:3300/0 conn(0x7f60e415c5a0 0x7f60e415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 -- 172.21.15.67:0/2469345153 >> 172.21.15.67:0/2469345153 conn(0x7f60e4067d10 msgr2=0x7f60e40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 -- 172.21.15.67:0/2469345153 shutdown_connections 2024-04-17T07:27:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.637+0000 7f60e3e00640 1 -- 172.21.15.67:0/2469345153 wait complete. 2024-04-17T07:27:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.639+0000 7f60e3e00640 1 Processor -- start 2024-04-17T07:27:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.639+0000 7f60e3e00640 1 -- start start 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 0x7f60e406f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e415c5a0 0x7f60e4157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60e406e7f0 con 0x7f60e4157da0 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f60e406e670 con 0x7f60e40b4210 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f60e4071df0 con 0x7f60e415c5a0 2024-04-17T07:27:20.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35938/0 (socket says 172.21.15.67:35938) 2024-04-17T07:27:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.640+0000 7f60e3400640 1 -- 172.21.15.67:0/3552091218 learned_addr learned my addr 172.21.15.67:0/3552091218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e2a00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 0x7f60e406f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e2000640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e415c5a0 0x7f60e4157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 -- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e415c5a0 msgr2=0x7f60e4157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e415c5a0 0x7f60e4157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 -- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 msgr2=0x7f60e406f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 0x7f60e406f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 -- 172.21.15.67:0/3552091218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e4158470 con 0x7f60e4157da0 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e2a00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 0x7f60e406f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3400640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f60dc007e30 tx=0x7f60dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc014420 con 0x7f60e4157da0 2024-04-17T07:27:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc019070 con 0x7f60e4157da0 2024-04-17T07:27:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc014800 con 0x7f60e4157da0 2024-04-17T07:27:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e4158750 con 0x7f60e4157da0 2024-04-17T07:27:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.641+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e406f9e0 con 0x7f60e4157da0 2024-04-17T07:27:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.643+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e4068a10 con 0x7f60e4157da0 2024-04-17T07:27:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.643+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f60dc0149a0 con 0x7f60e4157da0 2024-04-17T07:27:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.644+0000 7f60daa00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 0x7f60c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.644+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f60dc08c550 con 0x7f60e4157da0 2024-04-17T07:27:20.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.645+0000 7f60e2a00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 0x7f60c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:20.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.646+0000 7f60e2a00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 0x7f60c406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:20.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.649+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f60dc05a860 con 0x7f60e4157da0 2024-04-17T07:27:20.782 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T07:27:20.783+0000 7f7642800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:27:20.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.859+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 --> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60e406fc10 con 0x7f60c406ca10 2024-04-17T07:27:20.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.878+0000 7f60daa00640 1 -- 172.21.15.67:0/3552091218 <== mgr.5134 v2:172.21.15.187:6832/1235852913 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251956 (secure 0 0 0) 0x7f60e406fc10 con 0x7f60c406ca10 2024-04-17T07:27:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:20.883 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:27:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 msgr2=0x7f60c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 0x7f60c406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2024-04-17T07:27:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 msgr2=0x7f60e40c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f60dc007e30 tx=0x7f60dc00e4b0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 shutdown_connections 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f60c406ca10 0x7f60c406eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60e4157da0 0x7f60e40c83e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60e415c5a0 0x7f60e4157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 --2- 172.21.15.67:0/3552091218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60e40b4210 0x7f60e406f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 >> 172.21.15.67:0/3552091218 conn(0x7f60e4067d10 msgr2=0x7f60e40adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 shutdown_connections 2024-04-17T07:27:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:20.885+0000 7f60e3e00640 1 -- 172.21.15.67:0/3552091218 wait complete. 2024-04-17T07:27:20.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-17T07:27:20.839289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309564,"kb_used_data":7336,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754665156,"statfs":{"total":773094113280,"available":772777119744,"internally_reserved":0,"allocated":7512064,"data_stored":4122652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":309317168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.017615"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891034+0000","last_change":"2024-04-17T07:27:14.850104+0000","last_active":"2024-04-17T07:27:15.891034+0000","last_peered":"2024-04-17T07:27:15.891034+0000","last_clean":"2024-04-17T07:27:15.891034+0000","last_became_active":"2024-04-17T07:27:14.849784+0000","last_became_peered":"2024-04-17T07:27:14.849784+0000","last_unstale":"2024-04-17T07:27:15.891034+0000","last_undegraded":"2024-04-17T07:27:15.891034+0000","last_fullsized":"2024-04-17T07:27:15.891034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:47:18.786520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874000+0000","last_change":"2024-04-17T07:27:14.850674+0000","last_active":"2024-04-17T07:27:15.874000+0000","last_peered":"2024-04-17T07:27:15.874000+0000","last_clean":"2024-04-17T07:27:15.874000+0000","last_became_active":"2024-04-17T07:27:14.850359+0000","last_became_peered":"2024-04-17T07:27:14.850359+0000","last_unstale":"2024-04-17T07:27:15.874000+0000","last_undegraded":"2024-04-17T07:27:15.874000+0000","last_fullsized":"2024-04-17T07:27:15.874000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:36.075466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871118+0000","last_change":"2024-04-17T07:27:14.857351+0000","last_active":"2024-04-17T07:27:15.871118+0000","last_peered":"2024-04-17T07:27:15.871118+0000","last_clean":"2024-04-17T07:27:15.871118+0000","last_became_active":"2024-04-17T07:27:14.857039+0000","last_became_peered":"2024-04-17T07:27:14.857039+0000","last_unstale":"2024-04-17T07:27:15.871118+0000","last_undegraded":"2024-04-17T07:27:15.871118+0000","last_fullsized":"2024-04-17T07:27:15.871118+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:17.548403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869596+0000","last_change":"2024-04-17T07:27:14.866403+0000","last_active":"2024-04-17T07:27:15.869596+0000","last_peered":"2024-04-17T07:27:15.869596+0000","last_clean":"2024-04-17T07:27:15.869596+0000","last_became_active":"2024-04-17T07:27:14.866102+0000","last_became_peered":"2024-04-17T07:27:14.866102+0000","last_unstale":"2024-04-17T07:27:15.869596+0000","last_undegraded":"2024-04-17T07:27:15.869596+0000","last_fullsized":"2024-04-17T07:27:15.869596+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:32:54.077129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873799+0000","last_change":"2024-04-17T07:27:14.856407+0000","last_active":"2024-04-17T07:27:15.873799+0000","last_peered":"2024-04-17T07:27:15.873799+0000","last_clean":"2024-04-17T07:27:15.873799+0000","last_became_active":"2024-04-17T07:27:14.854791+0000","last_became_peered":"2024-04-17T07:27:14.854791+0000","last_unstale":"2024-04-17T07:27:15.873799+0000","last_undegraded":"2024-04-17T07:27:15.873799+0000","last_fullsized":"2024-04-17T07:27:15.873799+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:38.812985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100153+0000","last_change":"2024-04-17T07:27:14.840525+0000","last_active":"2024-04-17T07:27:16.100153+0000","last_peered":"2024-04-17T07:27:16.100153+0000","last_clean":"2024-04-17T07:27:16.100153+0000","last_became_active":"2024-04-17T07:27:14.840109+0000","last_became_peered":"2024-04-17T07:27:14.840109+0000","last_unstale":"2024-04-17T07:27:16.100153+0000","last_undegraded":"2024-04-17T07:27:16.100153+0000","last_fullsized":"2024-04-17T07:27:16.100153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:50.036601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890574+0000","last_change":"2024-04-17T07:27:14.847501+0000","last_active":"2024-04-17T07:27:15.890574+0000","last_peered":"2024-04-17T07:27:15.890574+0000","last_clean":"2024-04-17T07:27:15.890574+0000","last_became_active":"2024-04-17T07:27:14.847182+0000","last_became_peered":"2024-04-17T07:27:14.847182+0000","last_unstale":"2024-04-17T07:27:15.890574+0000","last_undegraded":"2024-04-17T07:27:15.890574+0000","last_fullsized":"2024-04-17T07:27:15.890574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:43.196106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868844+0000","last_change":"2024-04-17T07:27:14.855179+0000","last_active":"2024-04-17T07:27:15.868844+0000","last_peered":"2024-04-17T07:27:15.868844+0000","last_clean":"2024-04-17T07:27:15.868844+0000","last_became_active":"2024-04-17T07:27:14.854862+0000","last_became_peered":"2024-04-17T07:27:14.854862+0000","last_unstale":"2024-04-17T07:27:15.868844+0000","last_undegraded":"2024-04-17T07:27:15.868844+0000","last_fullsized":"2024-04-17T07:27:15.868844+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:12.064243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869946+0000","last_change":"2024-04-17T07:27:14.851225+0000","last_active":"2024-04-17T07:27:15.869946+0000","last_peered":"2024-04-17T07:27:15.869946+0000","last_clean":"2024-04-17T07:27:15.869946+0000","last_became_active":"2024-04-17T07:27:14.849005+0000","last_became_peered":"2024-04-17T07:27:14.849005+0000","last_unstale":"2024-04-17T07:27:15.869946+0000","last_undegraded":"2024-04-17T07:27:15.869946+0000","last_fullsized":"2024-04-17T07:27:15.869946+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:53.364064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869993+0000","last_change":"2024-04-17T07:27:14.866356+0000","last_active":"2024-04-17T07:27:15.869993+0000","last_peered":"2024-04-17T07:27:15.869993+0000","last_clean":"2024-04-17T07:27:15.869993+0000","last_became_active":"2024-04-17T07:27:14.866024+0000","last_became_peered":"2024-04-17T07:27:14.866024+0000","last_unstale":"2024-04-17T07:27:15.869993+0000","last_undegraded":"2024-04-17T07:27:15.869993+0000","last_fullsized":"2024-04-17T07:27:15.869993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:52.549915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870137+0000","last_change":"2024-04-17T07:27:14.845860+0000","last_active":"2024-04-17T07:27:15.870137+0000","last_peered":"2024-04-17T07:27:15.870137+0000","last_clean":"2024-04-17T07:27:15.870137+0000","last_became_active":"2024-04-17T07:27:14.845489+0000","last_became_peered":"2024-04-17T07:27:14.845489+0000","last_unstale":"2024-04-17T07:27:15.870137+0000","last_undegraded":"2024-04-17T07:27:15.870137+0000","last_fullsized":"2024-04-17T07:27:15.870137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:13.892289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873888+0000","last_change":"2024-04-17T07:27:14.844211+0000","last_active":"2024-04-17T07:27:15.873888+0000","last_peered":"2024-04-17T07:27:15.873888+0000","last_clean":"2024-04-17T07:27:15.873888+0000","last_became_active":"2024-04-17T07:27:14.843136+0000","last_became_peered":"2024-04-17T07:27:14.843136+0000","last_unstale":"2024-04-17T07:27:15.873888+0000","last_undegraded":"2024-04-17T07:27:15.873888+0000","last_fullsized":"2024-04-17T07:27:15.873888+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:33.102060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100061+0000","last_change":"2024-04-17T07:27:14.841860+0000","last_active":"2024-04-17T07:27:16.100061+0000","last_peered":"2024-04-17T07:27:16.100061+0000","last_clean":"2024-04-17T07:27:16.100061+0000","last_became_active":"2024-04-17T07:27:14.841509+0000","last_became_peered":"2024-04-17T07:27:14.841509+0000","last_unstale":"2024-04-17T07:27:16.100061+0000","last_undegraded":"2024-04-17T07:27:16.100061+0000","last_fullsized":"2024-04-17T07:27:16.100061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:32.186156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869442+0000","last_change":"2024-04-17T07:27:14.850251+0000","last_active":"2024-04-17T07:27:15.869442+0000","last_peered":"2024-04-17T07:27:15.869442+0000","last_clean":"2024-04-17T07:27:15.869442+0000","last_became_active":"2024-04-17T07:27:14.849987+0000","last_became_peered":"2024-04-17T07:27:14.849987+0000","last_unstale":"2024-04-17T07:27:15.869442+0000","last_undegraded":"2024-04-17T07:27:15.869442+0000","last_fullsized":"2024-04-17T07:27:15.869442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:32:43.797253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099664+0000","last_change":"2024-04-17T07:27:14.847912+0000","last_active":"2024-04-17T07:27:16.099664+0000","last_peered":"2024-04-17T07:27:16.099664+0000","last_clean":"2024-04-17T07:27:16.099664+0000","last_became_active":"2024-04-17T07:27:14.847661+0000","last_became_peered":"2024-04-17T07:27:14.847661+0000","last_unstale":"2024-04-17T07:27:16.099664+0000","last_undegraded":"2024-04-17T07:27:16.099664+0000","last_fullsized":"2024-04-17T07:27:16.099664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:23.259168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100319+0000","last_change":"2024-04-17T07:27:14.851680+0000","last_active":"2024-04-17T07:27:16.100319+0000","last_peered":"2024-04-17T07:27:16.100319+0000","last_clean":"2024-04-17T07:27:16.100319+0000","last_became_active":"2024-04-17T07:27:14.851412+0000","last_became_peered":"2024-04-17T07:27:14.851412+0000","last_unstale":"2024-04-17T07:27:16.100319+0000","last_undegraded":"2024-04-17T07:27:16.100319+0000","last_fullsized":"2024-04-17T07:27:16.100319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:34.337772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871417+0000","last_change":"2024-04-17T07:27:14.863921+0000","last_active":"2024-04-17T07:27:15.871417+0000","last_peered":"2024-04-17T07:27:15.871417+0000","last_clean":"2024-04-17T07:27:15.871417+0000","last_became_active":"2024-04-17T07:27:14.863203+0000","last_became_peered":"2024-04-17T07:27:14.863203+0000","last_unstale":"2024-04-17T07:27:15.871417+0000","last_undegraded":"2024-04-17T07:27:15.871417+0000","last_fullsized":"2024-04-17T07:27:15.871417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:45.464925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871344+0000","last_change":"2024-04-17T07:27:14.856184+0000","last_active":"2024-04-17T07:27:15.871344+0000","last_peered":"2024-04-17T07:27:15.871344+0000","last_clean":"2024-04-17T07:27:15.871344+0000","last_became_active":"2024-04-17T07:27:14.855889+0000","last_became_peered":"2024-04-17T07:27:14.855889+0000","last_unstale":"2024-04-17T07:27:15.871344+0000","last_undegraded":"2024-04-17T07:27:15.871344+0000","last_fullsized":"2024-04-17T07:27:15.871344+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:51.116693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101410+0000","last_change":"2024-04-17T07:27:14.858943+0000","last_active":"2024-04-17T07:27:16.101410+0000","last_peered":"2024-04-17T07:27:16.101410+0000","last_clean":"2024-04-17T07:27:16.101410+0000","last_became_active":"2024-04-17T07:27:14.857675+0000","last_became_peered":"2024-04-17T07:27:14.857675+0000","last_unstale":"2024-04-17T07:27:16.101410+0000","last_undegraded":"2024-04-17T07:27:16.101410+0000","last_fullsized":"2024-04-17T07:27:16.101410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:33.261066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101664+0000","last_change":"2024-04-17T07:27:14.845291+0000","last_active":"2024-04-17T07:27:16.101664+0000","last_peered":"2024-04-17T07:27:16.101664+0000","last_clean":"2024-04-17T07:27:16.101664+0000","last_became_active":"2024-04-17T07:27:14.843921+0000","last_became_peered":"2024-04-17T07:27:14.843921+0000","last_unstale":"2024-04-17T07:27:16.101664+0000","last_undegraded":"2024-04-17T07:27:16.101664+0000","last_fullsized":"2024-04-17T07:27:16.101664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:10.937006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100430+0000","last_change":"2024-04-17T07:27:14.854916+0000","last_active":"2024-04-17T07:27:16.100430+0000","last_peered":"2024-04-17T07:27:16.100430+0000","last_clean":"2024-04-17T07:27:16.100430+0000","last_became_active":"2024-04-17T07:27:14.853852+0000","last_became_peered":"2024-04-17T07:27:14.853852+0000","last_unstale":"2024-04-17T07:27:16.100430+0000","last_undegraded":"2024-04-17T07:27:16.100430+0000","last_fullsized":"2024-04-17T07:27:16.100430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:58.078350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869480+0000","last_change":"2024-04-17T07:27:14.854635+0000","last_active":"2024-04-17T07:27:15.869480+0000","last_peered":"2024-04-17T07:27:15.869480+0000","last_clean":"2024-04-17T07:27:15.869480+0000","last_became_active":"2024-04-17T07:27:14.854305+0000","last_became_peered":"2024-04-17T07:27:14.854305+0000","last_unstale":"2024-04-17T07:27:15.869480+0000","last_undegraded":"2024-04-17T07:27:15.869480+0000","last_fullsized":"2024-04-17T07:27:15.869480+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:50.778807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100370+0000","last_change":"2024-04-17T07:27:14.850991+0000","last_active":"2024-04-17T07:27:16.100370+0000","last_peered":"2024-04-17T07:27:16.100370+0000","last_clean":"2024-04-17T07:27:16.100370+0000","last_became_active":"2024-04-17T07:27:14.850703+0000","last_became_peered":"2024-04-17T07:27:14.850703+0000","last_unstale":"2024-04-17T07:27:16.100370+0000","last_undegraded":"2024-04-17T07:27:16.100370+0000","last_fullsized":"2024-04-17T07:27:16.100370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:11.944098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100694+0000","last_change":"2024-04-17T07:27:14.854500+0000","last_active":"2024-04-17T07:27:16.100694+0000","last_peered":"2024-04-17T07:27:16.100694+0000","last_clean":"2024-04-17T07:27:16.100694+0000","last_became_active":"2024-04-17T07:27:14.851054+0000","last_became_peered":"2024-04-17T07:27:14.851054+0000","last_unstale":"2024-04-17T07:27:16.100694+0000","last_undegraded":"2024-04-17T07:27:16.100694+0000","last_fullsized":"2024-04-17T07:27:16.100694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:27.880355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101388+0000","last_change":"2024-04-17T07:27:14.859453+0000","last_active":"2024-04-17T07:27:16.101388+0000","last_peered":"2024-04-17T07:27:16.101388+0000","last_clean":"2024-04-17T07:27:16.101388+0000","last_became_active":"2024-04-17T07:27:14.858768+0000","last_became_peered":"2024-04-17T07:27:14.858768+0000","last_unstale":"2024-04-17T07:27:16.101388+0000","last_undegraded":"2024-04-17T07:27:16.101388+0000","last_fullsized":"2024-04-17T07:27:16.101388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:16.990145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100971+0000","last_change":"2024-04-17T07:27:14.841562+0000","last_active":"2024-04-17T07:27:16.100971+0000","last_peered":"2024-04-17T07:27:16.100971+0000","last_clean":"2024-04-17T07:27:16.100971+0000","last_became_active":"2024-04-17T07:27:14.841184+0000","last_became_peered":"2024-04-17T07:27:14.841184+0000","last_unstale":"2024-04-17T07:27:16.100971+0000","last_undegraded":"2024-04-17T07:27:16.100971+0000","last_fullsized":"2024-04-17T07:27:16.100971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.745936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871435+0000","last_change":"2024-04-17T07:27:14.864853+0000","last_active":"2024-04-17T07:27:15.871435+0000","last_peered":"2024-04-17T07:27:15.871435+0000","last_clean":"2024-04-17T07:27:15.871435+0000","last_became_active":"2024-04-17T07:27:14.863242+0000","last_became_peered":"2024-04-17T07:27:14.863242+0000","last_unstale":"2024-04-17T07:27:15.871435+0000","last_undegraded":"2024-04-17T07:27:15.871435+0000","last_fullsized":"2024-04-17T07:27:15.871435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:38.646382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874102+0000","last_change":"2024-04-17T07:27:14.843938+0000","last_active":"2024-04-17T07:27:15.874102+0000","last_peered":"2024-04-17T07:27:15.874102+0000","last_clean":"2024-04-17T07:27:15.874102+0000","last_became_active":"2024-04-17T07:27:14.842663+0000","last_became_peered":"2024-04-17T07:27:14.842663+0000","last_unstale":"2024-04-17T07:27:15.874102+0000","last_undegraded":"2024-04-17T07:27:15.874102+0000","last_fullsized":"2024-04-17T07:27:15.874102+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:55:22.035015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100516+0000","last_change":"2024-04-17T07:27:14.856120+0000","last_active":"2024-04-17T07:27:16.100516+0000","last_peered":"2024-04-17T07:27:16.100516+0000","last_clean":"2024-04-17T07:27:16.100516+0000","last_became_active":"2024-04-17T07:27:14.855840+0000","last_became_peered":"2024-04-17T07:27:14.855840+0000","last_unstale":"2024-04-17T07:27:16.100516+0000","last_undegraded":"2024-04-17T07:27:16.100516+0000","last_fullsized":"2024-04-17T07:27:16.100516+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:33.265743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870724+0000","last_change":"2024-04-17T07:27:14.856154+0000","last_active":"2024-04-17T07:27:15.870724+0000","last_peered":"2024-04-17T07:27:15.870724+0000","last_clean":"2024-04-17T07:27:15.870724+0000","last_became_active":"2024-04-17T07:27:14.855810+0000","last_became_peered":"2024-04-17T07:27:14.855810+0000","last_unstale":"2024-04-17T07:27:15.870724+0000","last_undegraded":"2024-04-17T07:27:15.870724+0000","last_fullsized":"2024-04-17T07:27:15.870724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:12:59.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100100+0000","last_change":"2024-04-17T07:27:14.849723+0000","last_active":"2024-04-17T07:27:16.100100+0000","last_peered":"2024-04-17T07:27:16.100100+0000","last_clean":"2024-04-17T07:27:16.100100+0000","last_became_active":"2024-04-17T07:27:14.849412+0000","last_became_peered":"2024-04-17T07:27:14.849412+0000","last_unstale":"2024-04-17T07:27:16.100100+0000","last_undegraded":"2024-04-17T07:27:16.100100+0000","last_fullsized":"2024-04-17T07:27:16.100100+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:14.345316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098183+0000","last_change":"2024-04-17T07:27:14.829777+0000","last_active":"2024-04-17T07:27:16.098183+0000","last_peered":"2024-04-17T07:27:16.098183+0000","last_clean":"2024-04-17T07:27:16.098183+0000","last_became_active":"2024-04-17T07:27:14.829261+0000","last_became_peered":"2024-04-17T07:27:14.829261+0000","last_unstale":"2024-04-17T07:27:16.098183+0000","last_undegraded":"2024-04-17T07:27:16.098183+0000","last_fullsized":"2024-04-17T07:27:16.098183+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:16:22.628938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099357+0000","last_change":"2024-04-17T07:27:14.841566+0000","last_active":"2024-04-17T07:27:16.099357+0000","last_peered":"2024-04-17T07:27:16.099357+0000","last_clean":"2024-04-17T07:27:16.099357+0000","last_became_active":"2024-04-17T07:27:14.841286+0000","last_became_peered":"2024-04-17T07:27:14.841286+0000","last_unstale":"2024-04-17T07:27:16.099357+0000","last_undegraded":"2024-04-17T07:27:16.099357+0000","last_fullsized":"2024-04-17T07:27:16.099357+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:24.967416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"65'4","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101492+0000","last_change":"2024-04-17T07:27:13.857238+0000","last_active":"2024-04-17T07:27:16.101492+0000","last_peered":"2024-04-17T07:27:16.101492+0000","last_clean":"2024-04-17T07:27:16.101492+0000","last_became_active":"2024-04-17T07:27:13.856976+0000","last_became_peered":"2024-04-17T07:27:13.856976+0000","last_unstale":"2024-04-17T07:27:16.101492+0000","last_undegraded":"2024-04-17T07:27:16.101492+0000","last_fullsized":"2024-04-17T07:27:16.101492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:18.495414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101365+0000","last_change":"2024-04-17T07:27:14.860028+0000","last_active":"2024-04-17T07:27:16.101365+0000","last_peered":"2024-04-17T07:27:16.101365+0000","last_clean":"2024-04-17T07:27:16.101365+0000","last_became_active":"2024-04-17T07:27:14.859755+0000","last_became_peered":"2024-04-17T07:27:14.859755+0000","last_unstale":"2024-04-17T07:27:16.101365+0000","last_undegraded":"2024-04-17T07:27:16.101365+0000","last_fullsized":"2024-04-17T07:27:16.101365+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:21.514275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871682+0000","last_change":"2024-04-17T07:27:13.858318+0000","last_active":"2024-04-17T07:27:15.871682+0000","last_peered":"2024-04-17T07:27:15.871682+0000","last_clean":"2024-04-17T07:27:15.871682+0000","last_became_active":"2024-04-17T07:27:13.857431+0000","last_became_peered":"2024-04-17T07:27:13.857431+0000","last_unstale":"2024-04-17T07:27:15.871682+0000","last_undegraded":"2024-04-17T07:27:15.871682+0000","last_fullsized":"2024-04-17T07:27:15.871682+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:19:58.869362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874442+0000","last_change":"2024-04-17T07:27:14.844434+0000","last_active":"2024-04-17T07:27:15.874442+0000","last_peered":"2024-04-17T07:27:15.874442+0000","last_clean":"2024-04-17T07:27:15.874442+0000","last_became_active":"2024-04-17T07:27:14.843578+0000","last_became_peered":"2024-04-17T07:27:14.843578+0000","last_unstale":"2024-04-17T07:27:15.874442+0000","last_undegraded":"2024-04-17T07:27:15.874442+0000","last_fullsized":"2024-04-17T07:27:15.874442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:23.381721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098991+0000","last_change":"2024-04-17T07:27:13.849492+0000","last_active":"2024-04-17T07:27:16.098991+0000","last_peered":"2024-04-17T07:27:16.098991+0000","last_clean":"2024-04-17T07:27:16.098991+0000","last_became_active":"2024-04-17T07:27:13.849193+0000","last_became_peered":"2024-04-17T07:27:13.849193+0000","last_unstale":"2024-04-17T07:27:16.098991+0000","last_undegraded":"2024-04-17T07:27:16.098991+0000","last_fullsized":"2024-04-17T07:27:16.098991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:47.339048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098862+0000","last_change":"2024-04-17T07:27:14.861388+0000","last_active":"2024-04-17T07:27:16.098862+0000","last_peered":"2024-04-17T07:27:16.098862+0000","last_clean":"2024-04-17T07:27:16.098862+0000","last_became_active":"2024-04-17T07:27:14.861092+0000","last_became_peered":"2024-04-17T07:27:14.861092+0000","last_unstale":"2024-04-17T07:27:16.098862+0000","last_undegraded":"2024-04-17T07:27:16.098862+0000","last_fullsized":"2024-04-17T07:27:16.098862+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:26.261040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890391+0000","last_change":"2024-04-17T07:27:13.852852+0000","last_active":"2024-04-17T07:27:15.890391+0000","last_peered":"2024-04-17T07:27:15.890391+0000","last_clean":"2024-04-17T07:27:15.890391+0000","last_became_active":"2024-04-17T07:27:13.851479+0000","last_became_peered":"2024-04-17T07:27:13.851479+0000","last_unstale":"2024-04-17T07:27:15.890391+0000","last_undegraded":"2024-04-17T07:27:15.890391+0000","last_fullsized":"2024-04-17T07:27:15.890391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:01.491535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870952+0000","last_change":"2024-04-17T07:27:14.861157+0000","last_active":"2024-04-17T07:27:15.870952+0000","last_peered":"2024-04-17T07:27:15.870952+0000","last_clean":"2024-04-17T07:27:15.870952+0000","last_became_active":"2024-04-17T07:27:14.860813+0000","last_became_peered":"2024-04-17T07:27:14.860813+0000","last_unstale":"2024-04-17T07:27:15.870952+0000","last_undegraded":"2024-04-17T07:27:15.870952+0000","last_fullsized":"2024-04-17T07:27:15.870952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:31:07.484053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100661+0000","last_change":"2024-04-17T07:27:13.856181+0000","last_active":"2024-04-17T07:27:16.100661+0000","last_peered":"2024-04-17T07:27:16.100661+0000","last_clean":"2024-04-17T07:27:16.100661+0000","last_became_active":"2024-04-17T07:27:13.855812+0000","last_became_peered":"2024-04-17T07:27:13.855812+0000","last_unstale":"2024-04-17T07:27:16.100661+0000","last_undegraded":"2024-04-17T07:27:16.100661+0000","last_fullsized":"2024-04-17T07:27:16.100661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:32.476602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869587+0000","last_change":"2024-04-17T07:27:14.845314+0000","last_active":"2024-04-17T07:27:15.869587+0000","last_peered":"2024-04-17T07:27:15.869587+0000","last_clean":"2024-04-17T07:27:15.869587+0000","last_became_active":"2024-04-17T07:27:14.844970+0000","last_became_peered":"2024-04-17T07:27:14.844970+0000","last_unstale":"2024-04-17T07:27:15.869587+0000","last_undegraded":"2024-04-17T07:27:15.869587+0000","last_fullsized":"2024-04-17T07:27:15.869587+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:52.076888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870818+0000","last_change":"2024-04-17T07:27:13.854450+0000","last_active":"2024-04-17T07:27:15.870818+0000","last_peered":"2024-04-17T07:27:15.870818+0000","last_clean":"2024-04-17T07:27:15.870818+0000","last_became_active":"2024-04-17T07:27:13.854086+0000","last_became_peered":"2024-04-17T07:27:13.854086+0000","last_unstale":"2024-04-17T07:27:15.870818+0000","last_undegraded":"2024-04-17T07:27:15.870818+0000","last_fullsized":"2024-04-17T07:27:15.870818+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:42.646383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868926+0000","last_change":"2024-04-17T07:27:14.854625+0000","last_active":"2024-04-17T07:27:15.868926+0000","last_peered":"2024-04-17T07:27:15.868926+0000","last_clean":"2024-04-17T07:27:15.868926+0000","last_became_active":"2024-04-17T07:27:14.853892+0000","last_became_peered":"2024-04-17T07:27:14.853892+0000","last_unstale":"2024-04-17T07:27:15.868926+0000","last_undegraded":"2024-04-17T07:27:15.868926+0000","last_fullsized":"2024-04-17T07:27:15.868926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:31:35.235084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873414+0000","last_change":"2024-04-17T07:27:13.836090+0000","last_active":"2024-04-17T07:27:15.873414+0000","last_peered":"2024-04-17T07:27:15.873414+0000","last_clean":"2024-04-17T07:27:15.873414+0000","last_became_active":"2024-04-17T07:27:13.835575+0000","last_became_peered":"2024-04-17T07:27:13.835575+0000","last_unstale":"2024-04-17T07:27:15.873414+0000","last_undegraded":"2024-04-17T07:27:15.873414+0000","last_fullsized":"2024-04-17T07:27:15.873414+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:48.370837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100702+0000","last_change":"2024-04-17T07:27:14.854695+0000","last_active":"2024-04-17T07:27:16.100702+0000","last_peered":"2024-04-17T07:27:16.100702+0000","last_clean":"2024-04-17T07:27:16.100702+0000","last_became_active":"2024-04-17T07:27:14.854045+0000","last_became_peered":"2024-04-17T07:27:14.854045+0000","last_unstale":"2024-04-17T07:27:16.100702+0000","last_undegraded":"2024-04-17T07:27:16.100702+0000","last_fullsized":"2024-04-17T07:27:16.100702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:52.811181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.868888+0000","last_change":"2024-04-17T07:27:13.852535+0000","last_active":"2024-04-17T07:27:15.868888+0000","last_peered":"2024-04-17T07:27:15.868888+0000","last_clean":"2024-04-17T07:27:15.868888+0000","last_became_active":"2024-04-17T07:27:13.852194+0000","last_became_peered":"2024-04-17T07:27:13.852194+0000","last_unstale":"2024-04-17T07:27:15.868888+0000","last_undegraded":"2024-04-17T07:27:15.868888+0000","last_fullsized":"2024-04-17T07:27:15.868888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:30.546379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871557+0000","last_change":"2024-04-17T07:27:14.855804+0000","last_active":"2024-04-17T07:27:15.871557+0000","last_peered":"2024-04-17T07:27:15.871557+0000","last_clean":"2024-04-17T07:27:15.871557+0000","last_became_active":"2024-04-17T07:27:14.855469+0000","last_became_peered":"2024-04-17T07:27:14.855469+0000","last_unstale":"2024-04-17T07:27:15.871557+0000","last_undegraded":"2024-04-17T07:27:15.871557+0000","last_fullsized":"2024-04-17T07:27:15.871557+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:20.172072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871775+0000","last_change":"2024-04-17T07:27:13.849897+0000","last_active":"2024-04-17T07:27:15.871775+0000","last_peered":"2024-04-17T07:27:15.871775+0000","last_clean":"2024-04-17T07:27:15.871775+0000","last_became_active":"2024-04-17T07:27:13.849594+0000","last_became_peered":"2024-04-17T07:27:13.849594+0000","last_unstale":"2024-04-17T07:27:15.871775+0000","last_undegraded":"2024-04-17T07:27:15.871775+0000","last_fullsized":"2024-04-17T07:27:15.871775+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:45:56.681690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890875+0000","last_change":"2024-04-17T07:27:14.857423+0000","last_active":"2024-04-17T07:27:15.890875+0000","last_peered":"2024-04-17T07:27:15.890875+0000","last_clean":"2024-04-17T07:27:15.890875+0000","last_became_active":"2024-04-17T07:27:14.857094+0000","last_became_peered":"2024-04-17T07:27:14.857094+0000","last_unstale":"2024-04-17T07:27:15.890875+0000","last_undegraded":"2024-04-17T07:27:15.890875+0000","last_fullsized":"2024-04-17T07:27:15.890875+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:53.000609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870077+0000","last_change":"2024-04-17T07:27:13.853190+0000","last_active":"2024-04-17T07:27:15.870077+0000","last_peered":"2024-04-17T07:27:15.870077+0000","last_clean":"2024-04-17T07:27:15.870077+0000","last_became_active":"2024-04-17T07:27:13.852035+0000","last_became_peered":"2024-04-17T07:27:13.852035+0000","last_unstale":"2024-04-17T07:27:15.870077+0000","last_undegraded":"2024-04-17T07:27:15.870077+0000","last_fullsized":"2024-04-17T07:27:15.870077+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:26.307009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100490+0000","last_change":"2024-04-17T07:27:14.843845+0000","last_active":"2024-04-17T07:27:16.100490+0000","last_peered":"2024-04-17T07:27:16.100490+0000","last_clean":"2024-04-17T07:27:16.100490+0000","last_became_active":"2024-04-17T07:27:14.843344+0000","last_became_peered":"2024-04-17T07:27:14.843344+0000","last_unstale":"2024-04-17T07:27:16.100490+0000","last_undegraded":"2024-04-17T07:27:16.100490+0000","last_fullsized":"2024-04-17T07:27:16.100490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:11.355030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"65'3","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891041+0000","last_change":"2024-04-17T07:27:13.854788+0000","last_active":"2024-04-17T07:27:15.891041+0000","last_peered":"2024-04-17T07:27:15.891041+0000","last_clean":"2024-04-17T07:27:15.891041+0000","last_became_active":"2024-04-17T07:27:13.854298+0000","last_became_peered":"2024-04-17T07:27:13.854298+0000","last_unstale":"2024-04-17T07:27:15.891041+0000","last_undegraded":"2024-04-17T07:27:15.891041+0000","last_fullsized":"2024-04-17T07:27:15.891041+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:24.337854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890903+0000","last_change":"2024-04-17T07:27:14.846915+0000","last_active":"2024-04-17T07:27:15.890903+0000","last_peered":"2024-04-17T07:27:15.890903+0000","last_clean":"2024-04-17T07:27:15.890903+0000","last_became_active":"2024-04-17T07:27:14.846328+0000","last_became_peered":"2024-04-17T07:27:14.846328+0000","last_unstale":"2024-04-17T07:27:15.890903+0000","last_undegraded":"2024-04-17T07:27:15.890903+0000","last_fullsized":"2024-04-17T07:27:15.890903+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:29.636974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"65'6","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100501+0000","last_change":"2024-04-17T07:27:13.849239+0000","last_active":"2024-04-17T07:27:16.100501+0000","last_peered":"2024-04-17T07:27:16.100501+0000","last_clean":"2024-04-17T07:27:16.100501+0000","last_became_active":"2024-04-17T07:27:13.848540+0000","last_became_peered":"2024-04-17T07:27:13.848540+0000","last_unstale":"2024-04-17T07:27:16.100501+0000","last_undegraded":"2024-04-17T07:27:16.100501+0000","last_fullsized":"2024-04-17T07:27:16.100501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:44:50.889450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100684+0000","last_change":"2024-04-17T07:27:14.864103+0000","last_active":"2024-04-17T07:27:16.100684+0000","last_peered":"2024-04-17T07:27:16.100684+0000","last_clean":"2024-04-17T07:27:16.100684+0000","last_became_active":"2024-04-17T07:27:14.863547+0000","last_became_peered":"2024-04-17T07:27:14.863547+0000","last_unstale":"2024-04-17T07:27:16.100684+0000","last_undegraded":"2024-04-17T07:27:16.100684+0000","last_fullsized":"2024-04-17T07:27:16.100684+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:32.088976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100757+0000","last_change":"2024-04-17T07:27:13.854642+0000","last_active":"2024-04-17T07:27:16.100757+0000","last_peered":"2024-04-17T07:27:16.100757+0000","last_clean":"2024-04-17T07:27:16.100757+0000","last_became_active":"2024-04-17T07:27:13.854116+0000","last_became_peered":"2024-04-17T07:27:13.854116+0000","last_unstale":"2024-04-17T07:27:16.100757+0000","last_undegraded":"2024-04-17T07:27:16.100757+0000","last_fullsized":"2024-04-17T07:27:16.100757+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:33.510590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869595+0000","last_change":"2024-04-17T07:27:14.842733+0000","last_active":"2024-04-17T07:27:15.869595+0000","last_peered":"2024-04-17T07:27:15.869595+0000","last_clean":"2024-04-17T07:27:15.869595+0000","last_became_active":"2024-04-17T07:27:14.842479+0000","last_became_peered":"2024-04-17T07:27:14.842479+0000","last_unstale":"2024-04-17T07:27:15.869595+0000","last_undegraded":"2024-04-17T07:27:15.869595+0000","last_fullsized":"2024-04-17T07:27:15.869595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:44:41.834950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890230+0000","last_change":"2024-04-17T07:27:13.851191+0000","last_active":"2024-04-17T07:27:15.890230+0000","last_peered":"2024-04-17T07:27:15.890230+0000","last_clean":"2024-04-17T07:27:15.890230+0000","last_became_active":"2024-04-17T07:27:13.850898+0000","last_became_peered":"2024-04-17T07:27:13.850898+0000","last_unstale":"2024-04-17T07:27:15.890230+0000","last_undegraded":"2024-04-17T07:27:15.890230+0000","last_fullsized":"2024-04-17T07:27:15.890230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:20.393223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099607+0000","last_change":"2024-04-17T07:27:14.842821+0000","last_active":"2024-04-17T07:27:16.099607+0000","last_peered":"2024-04-17T07:27:16.099607+0000","last_clean":"2024-04-17T07:27:16.099607+0000","last_became_active":"2024-04-17T07:27:14.842468+0000","last_became_peered":"2024-04-17T07:27:14.842468+0000","last_unstale":"2024-04-17T07:27:16.099607+0000","last_undegraded":"2024-04-17T07:27:16.099607+0000","last_fullsized":"2024-04-17T07:27:16.099607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:02.510156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099812+0000","last_change":"2024-04-17T07:27:13.855175+0000","last_active":"2024-04-17T07:27:16.099812+0000","last_peered":"2024-04-17T07:27:16.099812+0000","last_clean":"2024-04-17T07:27:16.099812+0000","last_became_active":"2024-04-17T07:27:13.854416+0000","last_became_peered":"2024-04-17T07:27:13.854416+0000","last_unstale":"2024-04-17T07:27:16.099812+0000","last_undegraded":"2024-04-17T07:27:16.099812+0000","last_fullsized":"2024-04-17T07:27:16.099812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:37.794364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890580+0000","last_change":"2024-04-17T07:27:14.853138+0000","last_active":"2024-04-17T07:27:15.890580+0000","last_peered":"2024-04-17T07:27:15.890580+0000","last_clean":"2024-04-17T07:27:15.890580+0000","last_became_active":"2024-04-17T07:27:14.852812+0000","last_became_peered":"2024-04-17T07:27:14.852812+0000","last_unstale":"2024-04-17T07:27:15.890580+0000","last_undegraded":"2024-04-17T07:27:15.890580+0000","last_fullsized":"2024-04-17T07:27:15.890580+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:35:52.840396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"65'8","reported_seq":25,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869021+0000","last_change":"2024-04-17T07:27:13.857020+0000","last_active":"2024-04-17T07:27:15.869021+0000","last_peered":"2024-04-17T07:27:15.869021+0000","last_clean":"2024-04-17T07:27:15.869021+0000","last_became_active":"2024-04-17T07:27:13.854881+0000","last_became_peered":"2024-04-17T07:27:13.854881+0000","last_unstale":"2024-04-17T07:27:15.869021+0000","last_undegraded":"2024-04-17T07:27:15.869021+0000","last_fullsized":"2024-04-17T07:27:15.869021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:38.918738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101363+0000","last_change":"2024-04-17T07:27:14.844212+0000","last_active":"2024-04-17T07:27:16.101363+0000","last_peered":"2024-04-17T07:27:16.101363+0000","last_clean":"2024-04-17T07:27:16.101363+0000","last_became_active":"2024-04-17T07:27:14.843895+0000","last_became_peered":"2024-04-17T07:27:14.843895+0000","last_unstale":"2024-04-17T07:27:16.101363+0000","last_undegraded":"2024-04-17T07:27:16.101363+0000","last_fullsized":"2024-04-17T07:27:16.101363+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:12:55.331688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099111+0000","last_change":"2024-04-17T07:27:13.845685+0000","last_active":"2024-04-17T07:27:16.099111+0000","last_peered":"2024-04-17T07:27:16.099111+0000","last_clean":"2024-04-17T07:27:16.099111+0000","last_became_active":"2024-04-17T07:27:13.845360+0000","last_became_peered":"2024-04-17T07:27:13.845360+0000","last_unstale":"2024-04-17T07:27:16.099111+0000","last_undegraded":"2024-04-17T07:27:16.099111+0000","last_fullsized":"2024-04-17T07:27:16.099111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:46.712162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870044+0000","last_change":"2024-04-17T07:27:14.846632+0000","last_active":"2024-04-17T07:27:15.870044+0000","last_peered":"2024-04-17T07:27:15.870044+0000","last_clean":"2024-04-17T07:27:15.870044+0000","last_became_active":"2024-04-17T07:27:14.846345+0000","last_became_peered":"2024-04-17T07:27:14.846345+0000","last_unstale":"2024-04-17T07:27:15.870044+0000","last_undegraded":"2024-04-17T07:27:15.870044+0000","last_fullsized":"2024-04-17T07:27:15.870044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:02.450730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.871825+0000","last_change":"2024-04-17T07:27:13.847245+0000","last_active":"2024-04-17T07:27:15.871825+0000","last_peered":"2024-04-17T07:27:15.871825+0000","last_clean":"2024-04-17T07:27:15.871825+0000","last_became_active":"2024-04-17T07:27:13.846942+0000","last_became_peered":"2024-04-17T07:27:13.846942+0000","last_unstale":"2024-04-17T07:27:15.871825+0000","last_undegraded":"2024-04-17T07:27:15.871825+0000","last_fullsized":"2024-04-17T07:27:15.871825+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:44.946243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100749+0000","last_change":"2024-04-17T07:27:14.857652+0000","last_active":"2024-04-17T07:27:16.100749+0000","last_peered":"2024-04-17T07:27:16.100749+0000","last_clean":"2024-04-17T07:27:16.100749+0000","last_became_active":"2024-04-17T07:27:14.857146+0000","last_became_peered":"2024-04-17T07:27:14.857146+0000","last_unstale":"2024-04-17T07:27:16.100749+0000","last_undegraded":"2024-04-17T07:27:16.100749+0000","last_fullsized":"2024-04-17T07:27:16.100749+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:29.764710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"65'1","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874341+0000","last_change":"2024-04-17T07:27:13.847077+0000","last_active":"2024-04-17T07:27:15.874341+0000","last_peered":"2024-04-17T07:27:15.874341+0000","last_clean":"2024-04-17T07:27:15.874341+0000","last_became_active":"2024-04-17T07:27:13.842412+0000","last_became_peered":"2024-04-17T07:27:13.842412+0000","last_unstale":"2024-04-17T07:27:15.874341+0000","last_undegraded":"2024-04-17T07:27:15.874341+0000","last_fullsized":"2024-04-17T07:27:15.874341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:21.242882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101070+0000","last_change":"2024-04-17T07:27:14.855342+0000","last_active":"2024-04-17T07:27:16.101070+0000","last_peered":"2024-04-17T07:27:16.101070+0000","last_clean":"2024-04-17T07:27:16.101070+0000","last_became_active":"2024-04-17T07:27:14.854997+0000","last_became_peered":"2024-04-17T07:27:14.854997+0000","last_unstale":"2024-04-17T07:27:16.101070+0000","last_undegraded":"2024-04-17T07:27:16.101070+0000","last_fullsized":"2024-04-17T07:27:16.101070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:25.534091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100547+0000","last_change":"2024-04-17T07:27:13.856675+0000","last_active":"2024-04-17T07:27:16.100547+0000","last_peered":"2024-04-17T07:27:16.100547+0000","last_clean":"2024-04-17T07:27:16.100547+0000","last_became_active":"2024-04-17T07:27:13.856394+0000","last_became_peered":"2024-04-17T07:27:13.856394+0000","last_unstale":"2024-04-17T07:27:16.100547+0000","last_undegraded":"2024-04-17T07:27:16.100547+0000","last_fullsized":"2024-04-17T07:27:16.100547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:14.344042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100208+0000","last_change":"2024-04-17T07:27:14.843225+0000","last_active":"2024-04-17T07:27:16.100208+0000","last_peered":"2024-04-17T07:27:16.100208+0000","last_clean":"2024-04-17T07:27:16.100208+0000","last_became_active":"2024-04-17T07:27:14.842912+0000","last_became_peered":"2024-04-17T07:27:14.842912+0000","last_unstale":"2024-04-17T07:27:16.100208+0000","last_undegraded":"2024-04-17T07:27:16.100208+0000","last_fullsized":"2024-04-17T07:27:16.100208+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:28:25.356475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100913+0000","last_change":"2024-04-17T07:27:13.855887+0000","last_active":"2024-04-17T07:27:16.100913+0000","last_peered":"2024-04-17T07:27:16.100913+0000","last_clean":"2024-04-17T07:27:16.100913+0000","last_became_active":"2024-04-17T07:27:13.855595+0000","last_became_peered":"2024-04-17T07:27:13.855595+0000","last_unstale":"2024-04-17T07:27:16.100913+0000","last_undegraded":"2024-04-17T07:27:16.100913+0000","last_fullsized":"2024-04-17T07:27:16.100913+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:44:55.683309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098826+0000","last_change":"2024-04-17T07:27:14.840379+0000","last_active":"2024-04-17T07:27:16.098826+0000","last_peered":"2024-04-17T07:27:16.098826+0000","last_clean":"2024-04-17T07:27:16.098826+0000","last_became_active":"2024-04-17T07:27:14.840089+0000","last_became_peered":"2024-04-17T07:27:14.840089+0000","last_unstale":"2024-04-17T07:27:16.098826+0000","last_undegraded":"2024-04-17T07:27:16.098826+0000","last_fullsized":"2024-04-17T07:27:16.098826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:43.485046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873400+0000","last_change":"2024-04-17T07:27:13.852469+0000","last_active":"2024-04-17T07:27:15.873400+0000","last_peered":"2024-04-17T07:27:15.873400+0000","last_clean":"2024-04-17T07:27:15.873400+0000","last_became_active":"2024-04-17T07:27:13.851418+0000","last_became_peered":"2024-04-17T07:27:13.851418+0000","last_unstale":"2024-04-17T07:27:15.873400+0000","last_undegraded":"2024-04-17T07:27:15.873400+0000","last_fullsized":"2024-04-17T07:27:15.873400+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:02:31.421337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.099149+0000","last_change":"2024-04-17T07:27:14.835711+0000","last_active":"2024-04-17T07:27:16.099149+0000","last_peered":"2024-04-17T07:27:16.099149+0000","last_clean":"2024-04-17T07:27:16.099149+0000","last_became_active":"2024-04-17T07:27:14.835097+0000","last_became_peered":"2024-04-17T07:27:14.835097+0000","last_unstale":"2024-04-17T07:27:16.099149+0000","last_undegraded":"2024-04-17T07:27:16.099149+0000","last_fullsized":"2024-04-17T07:27:16.099149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:15.766689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869159+0000","last_change":"2024-04-17T07:27:13.857309+0000","last_active":"2024-04-17T07:27:15.869159+0000","last_peered":"2024-04-17T07:27:15.869159+0000","last_clean":"2024-04-17T07:27:15.869159+0000","last_became_active":"2024-04-17T07:27:13.857020+0000","last_became_peered":"2024-04-17T07:27:13.857020+0000","last_unstale":"2024-04-17T07:27:15.869159+0000","last_undegraded":"2024-04-17T07:27:15.869159+0000","last_fullsized":"2024-04-17T07:27:15.869159+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.333114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869011+0000","last_change":"2024-04-17T07:27:14.846546+0000","last_active":"2024-04-17T07:27:15.869011+0000","last_peered":"2024-04-17T07:27:15.869011+0000","last_clean":"2024-04-17T07:27:15.869011+0000","last_became_active":"2024-04-17T07:27:14.846194+0000","last_became_peered":"2024-04-17T07:27:14.846194+0000","last_unstale":"2024-04-17T07:27:15.869011+0000","last_undegraded":"2024-04-17T07:27:15.869011+0000","last_fullsized":"2024-04-17T07:27:15.869011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:44.394197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.889774+0000","last_change":"2024-04-17T07:27:13.849552+0000","last_active":"2024-04-17T07:27:15.889774+0000","last_peered":"2024-04-17T07:27:15.889774+0000","last_clean":"2024-04-17T07:27:15.889774+0000","last_became_active":"2024-04-17T07:27:13.849194+0000","last_became_peered":"2024-04-17T07:27:13.849194+0000","last_unstale":"2024-04-17T07:27:15.889774+0000","last_undegraded":"2024-04-17T07:27:15.889774+0000","last_fullsized":"2024-04-17T07:27:15.889774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:15.682578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101615+0000","last_change":"2024-04-17T07:27:14.858105+0000","last_active":"2024-04-17T07:27:16.101615+0000","last_peered":"2024-04-17T07:27:16.101615+0000","last_clean":"2024-04-17T07:27:16.101615+0000","last_became_active":"2024-04-17T07:27:14.857040+0000","last_became_peered":"2024-04-17T07:27:14.857040+0000","last_unstale":"2024-04-17T07:27:16.101615+0000","last_undegraded":"2024-04-17T07:27:16.101615+0000","last_fullsized":"2024-04-17T07:27:16.101615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:14.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100816+0000","last_change":"2024-04-17T07:27:13.850499+0000","last_active":"2024-04-17T07:27:16.100816+0000","last_peered":"2024-04-17T07:27:16.100816+0000","last_clean":"2024-04-17T07:27:16.100816+0000","last_became_active":"2024-04-17T07:27:13.849205+0000","last_became_peered":"2024-04-17T07:27:13.849205+0000","last_unstale":"2024-04-17T07:27:16.100816+0000","last_undegraded":"2024-04-17T07:27:16.100816+0000","last_fullsized":"2024-04-17T07:27:16.100816+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:20.393223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869807+0000","last_change":"2024-04-17T07:27:14.846354+0000","last_active":"2024-04-17T07:27:15.869807+0000","last_peered":"2024-04-17T07:27:15.869807+0000","last_clean":"2024-04-17T07:27:15.869807+0000","last_became_active":"2024-04-17T07:27:14.846084+0000","last_became_peered":"2024-04-17T07:27:14.846084+0000","last_unstale":"2024-04-17T07:27:15.869807+0000","last_undegraded":"2024-04-17T07:27:15.869807+0000","last_fullsized":"2024-04-17T07:27:15.869807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:10.916043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891244+0000","last_change":"2024-04-17T07:27:13.855123+0000","last_active":"2024-04-17T07:27:15.891244+0000","last_peered":"2024-04-17T07:27:15.891244+0000","last_clean":"2024-04-17T07:27:15.891244+0000","last_became_active":"2024-04-17T07:27:13.854212+0000","last_became_peered":"2024-04-17T07:27:13.854212+0000","last_unstale":"2024-04-17T07:27:15.891244+0000","last_undegraded":"2024-04-17T07:27:15.891244+0000","last_fullsized":"2024-04-17T07:27:15.891244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:57.059729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.891088+0000","last_change":"2024-04-17T07:27:14.842027+0000","last_active":"2024-04-17T07:27:15.891088+0000","last_peered":"2024-04-17T07:27:15.891088+0000","last_clean":"2024-04-17T07:27:15.891088+0000","last_became_active":"2024-04-17T07:27:14.841669+0000","last_became_peered":"2024-04-17T07:27:14.841669+0000","last_unstale":"2024-04-17T07:27:15.891088+0000","last_undegraded":"2024-04-17T07:27:15.891088+0000","last_fullsized":"2024-04-17T07:27:15.891088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:11.641523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.101019+0000","last_change":"2024-04-17T07:27:13.850353+0000","last_active":"2024-04-17T07:27:16.101019+0000","last_peered":"2024-04-17T07:27:16.101019+0000","last_clean":"2024-04-17T07:27:16.101019+0000","last_became_active":"2024-04-17T07:27:13.850003+0000","last_became_peered":"2024-04-17T07:27:13.850003+0000","last_unstale":"2024-04-17T07:27:16.101019+0000","last_undegraded":"2024-04-17T07:27:16.101019+0000","last_fullsized":"2024-04-17T07:27:16.101019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:51.501836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100429+0000","last_change":"2024-04-17T07:27:14.839822+0000","last_active":"2024-04-17T07:27:16.100429+0000","last_peered":"2024-04-17T07:27:16.100429+0000","last_clean":"2024-04-17T07:27:16.100429+0000","last_became_active":"2024-04-17T07:27:14.839492+0000","last_became_peered":"2024-04-17T07:27:14.839492+0000","last_unstale":"2024-04-17T07:27:16.100429+0000","last_undegraded":"2024-04-17T07:27:16.100429+0000","last_fullsized":"2024-04-17T07:27:16.100429+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:55:09.728826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.890388+0000","last_change":"2024-04-17T07:27:13.857311+0000","last_active":"2024-04-17T07:27:15.890388+0000","last_peered":"2024-04-17T07:27:15.890388+0000","last_clean":"2024-04-17T07:27:15.890388+0000","last_became_active":"2024-04-17T07:27:13.856977+0000","last_became_peered":"2024-04-17T07:27:13.856977+0000","last_unstale":"2024-04-17T07:27:15.890388+0000","last_undegraded":"2024-04-17T07:27:15.890388+0000","last_fullsized":"2024-04-17T07:27:15.890388+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:49.017980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.100609+0000","last_change":"2024-04-17T07:27:14.844776+0000","last_active":"2024-04-17T07:27:16.100609+0000","last_peered":"2024-04-17T07:27:16.100609+0000","last_clean":"2024-04-17T07:27:16.100609+0000","last_became_active":"2024-04-17T07:27:14.844528+0000","last_became_peered":"2024-04-17T07:27:14.844528+0000","last_unstale":"2024-04-17T07:27:16.100609+0000","last_undegraded":"2024-04-17T07:27:16.100609+0000","last_fullsized":"2024-04-17T07:27:16.100609+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:11.677344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870976+0000","last_change":"2024-04-17T07:27:13.853202+0000","last_active":"2024-04-17T07:27:15.870976+0000","last_peered":"2024-04-17T07:27:15.870976+0000","last_clean":"2024-04-17T07:27:15.870976+0000","last_became_active":"2024-04-17T07:27:13.852669+0000","last_became_peered":"2024-04-17T07:27:13.852669+0000","last_unstale":"2024-04-17T07:27:15.870976+0000","last_undegraded":"2024-04-17T07:27:15.870976+0000","last_fullsized":"2024-04-17T07:27:15.870976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:36.199769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.874051+0000","last_change":"2024-04-17T07:27:14.854577+0000","last_active":"2024-04-17T07:27:15.874051+0000","last_peered":"2024-04-17T07:27:15.874051+0000","last_clean":"2024-04-17T07:27:15.874051+0000","last_became_active":"2024-04-17T07:27:14.854276+0000","last_became_peered":"2024-04-17T07:27:14.854276+0000","last_unstale":"2024-04-17T07:27:15.874051+0000","last_undegraded":"2024-04-17T07:27:15.874051+0000","last_fullsized":"2024-04-17T07:27:15.874051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:10.927524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:16.098408+0000","last_change":"2024-04-17T07:27:13.850748+0000","last_active":"2024-04-17T07:27:16.098408+0000","last_peered":"2024-04-17T07:27:16.098408+0000","last_clean":"2024-04-17T07:27:16.098408+0000","last_became_active":"2024-04-17T07:27:13.849092+0000","last_became_peered":"2024-04-17T07:27:13.849092+0000","last_unstale":"2024-04-17T07:27:16.098408+0000","last_undegraded":"2024-04-17T07:27:16.098408+0000","last_fullsized":"2024-04-17T07:27:16.098408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:36.621658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870838+0000","last_change":"2024-04-17T07:27:14.854138+0000","last_active":"2024-04-17T07:27:15.870838+0000","last_peered":"2024-04-17T07:27:15.870838+0000","last_clean":"2024-04-17T07:27:15.870838+0000","last_became_active":"2024-04-17T07:27:14.853769+0000","last_became_peered":"2024-04-17T07:27:14.853769+0000","last_unstale":"2024-04-17T07:27:15.870838+0000","last_undegraded":"2024-04-17T07:27:15.870838+0000","last_fullsized":"2024-04-17T07:27:15.870838+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:41.523336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.869022+0000","last_change":"2024-04-17T07:27:13.846014+0000","last_active":"2024-04-17T07:27:15.869022+0000","last_peered":"2024-04-17T07:27:15.869022+0000","last_clean":"2024-04-17T07:27:15.869022+0000","last_became_active":"2024-04-17T07:27:13.845656+0000","last_became_peered":"2024-04-17T07:27:13.845656+0000","last_unstale":"2024-04-17T07:27:15.869022+0000","last_undegraded":"2024-04-17T07:27:15.869022+0000","last_fullsized":"2024-04-17T07:27:15.869022+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:23:22.123264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.870074+0000","last_change":"2024-04-17T07:27:14.841602+0000","last_active":"2024-04-17T07:27:15.870074+0000","last_peered":"2024-04-17T07:27:15.870074+0000","last_clean":"2024-04-17T07:27:15.870074+0000","last_became_active":"2024-04-17T07:27:14.841228+0000","last_became_peered":"2024-04-17T07:27:14.841228+0000","last_unstale":"2024-04-17T07:27:15.870074+0000","last_undegraded":"2024-04-17T07:27:15.870074+0000","last_fullsized":"2024-04-17T07:27:15.870074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:13.799045+0000","last_clean_scrub_stamp":"2024-04-17T07:27:13.799045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:01.370852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"65'2","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.873993+0000","last_change":"2024-04-17T07:27:13.846264+0000","last_active":"2024-04-17T07:27:15.873993+0000","last_peered":"2024-04-17T07:27:15.873993+0000","last_clean":"2024-04-17T07:27:15.873993+0000","last_became_active":"2024-04-17T07:27:13.845951+0000","last_became_peered":"2024-04-17T07:27:13.845951+0000","last_unstale":"2024-04-17T07:27:15.873993+0000","last_undegraded":"2024-04-17T07:27:15.873993+0000","last_fullsized":"2024-04-17T07:27:15.873993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:27:12.780423+0000","last_clean_scrub_stamp":"2024-04-17T07:27:12.780423+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:39:13.218154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"54'139","reported_seq":339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-17T07:27:15.889967+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:27:15.889967+0000","last_peered":"2024-04-17T07:27:15.889967+0000","last_clean":"2024-04-17T07:27:15.889967+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:27:15.889967+0000","last_undegraded":"2024-04-17T07:27:15.889967+0000","last_fullsized":"2024-04-17T07:27:15.889967+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542308,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1380352,"data_stored":957595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542308,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":790528,"data_stored":367809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":786432,"data_stored":367207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542307,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58336,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313504,"statfs":{"total":96636764160,"available":96577028096,"internally_reserved":0,"allocated":794624,"data_stored":368749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542308,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":794624,"data_stored":367775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542308,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40284,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331556,"statfs":{"total":96636764160,"available":96595513344,"internally_reserved":0,"allocated":798720,"data_stored":368771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542308,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50004,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321836,"statfs":{"total":96636764160,"available":96585560064,"internally_reserved":0,"allocated":786432,"data_stored":367207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542307,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1380352,"data_stored":957539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:27:20.910 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:27:20.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.183+0000 7f8f73200640 1 Processor -- start 2024-04-17T07:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.183+0000 7f8f73200640 1 -- start start 2024-04-17T07:27:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f73200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8f74071d00 0x7f8f740b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f73200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f740b25f0 0x7f8f74142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f73200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f7415cbe0 con 0x7f8f7413f6d0 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f73200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8f740b2a60 con 0x7f8f740b25f0 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f73200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8f74072190 con 0x7f8f74071d00 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f71e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8f7413f6d0 0x7f8f740cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39018/0 (socket says 172.21.15.67:39018) 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.185+0000 7f8f71e00640 1 -- 172.21.15.67:0/2401628836 learned_addr learned my addr 172.21.15.67:0/2401628836 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f72800640 1 --2- 172.21.15.67:0/2401628836 >> v2:172.21.15.67:3300/0 conn(0x7f8f740b25f0 0x7f8f74142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f71400640 1 --2- 172.21.15.67:0/2401628836 >> v2:172.21.15.187:3300/0 conn(0x7f8f74071d00 0x7f8f740b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f70a00640 1 -- 172.21.15.67:0/2401628836 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3851522760 0 0) 0x7f8f7415cbe0 con 0x7f8f7413f6d0 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f70a00640 1 -- 172.21.15.67:0/2401628836 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f58003520 con 0x7f8f7413f6d0 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f72800640 1 -- 172.21.15.67:0/2401628836 >> v2:172.21.15.187:3300/0 conn(0x7f8f74071d00 msgr2=0x7f8f740b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f72800640 1 --2- 172.21.15.67:0/2401628836 >> v2:172.21.15.187:3300/0 conn(0x7f8f74071d00 0x7f8f740b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f72800640 1 -- 172.21.15.67:0/2401628836 >> v1:172.21.15.67:6789/0 conn(0x7f8f7413f6d0 legacy=0x7f8f740cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.186+0000 7f8f72800640 1 -- 172.21.15.67:0/2401628836 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7415cea0 con 0x7f8f740b25f0 2024-04-17T07:27:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.188+0000 7f8f72800640 1 --2- 172.21.15.67:0/2401628836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f74142020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8f740d02d0 tx=0x7f8f6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b508113f753dfa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.188+0000 7f8f70a00640 1 -- 172.21.15.67:0/2401628836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c039070 con 0x7f8f740b25f0 2024-04-17T07:27:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.188+0000 7f8f70a00640 1 -- 172.21.15.67:0/2401628836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6c02cc80 con 0x7f8f740b25f0 2024-04-17T07:27:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.188+0000 7f8f70a00640 1 -- 172.21.15.67:0/2401628836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c034660 con 0x7f8f740b25f0 2024-04-17T07:27:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 -- 172.21.15.67:0/2401628836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 msgr2=0x7f8f74142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 --2- 172.21.15.67:0/2401628836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f74142020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8f740d02d0 tx=0x7f8f6c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 -- 172.21.15.67:0/2401628836 shutdown_connections 2024-04-17T07:27:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 --2- 172.21.15.67:0/2401628836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f74142020 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 --2- 172.21.15.67:0/2401628836 >> v2:172.21.15.187:3300/0 conn(0x7f8f74071d00 0x7f8f740b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 -- 172.21.15.67:0/2401628836 >> 172.21.15.67:0/2401628836 conn(0x7f8f74067d10 msgr2=0x7f8f740d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 -- 172.21.15.67:0/2401628836 shutdown_connections 2024-04-17T07:27:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.189+0000 7f8f73200640 1 -- 172.21.15.67:0/2401628836 wait complete. 2024-04-17T07:27:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.191+0000 7f8f73200640 1 Processor -- start 2024-04-17T07:27:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.191+0000 7f8f73200640 1 -- start start 2024-04-17T07:27:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 0x7f8f74160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7413f6d0 0x7f8f74155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7415d740 con 0x7f8f740b25f0 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7415d460 con 0x7f8f74071d00 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f73200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f740d1120 con 0x7f8f7413f6d0 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.192+0000 7f8f71400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35942/0 (socket says 172.21.15.67:35942) 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 -- 172.21.15.67:0/104196331 learned_addr learned my addr 172.21.15.67:0/104196331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 -- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7413f6d0 msgr2=0x7f8f74155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7413f6d0 0x7f8f74155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71e00640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 0x7f8f74160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 -- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 msgr2=0x7f8f74160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 0x7f8f74160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 -- 172.21.15.67:0/104196331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f741efc00 con 0x7f8f740b25f0 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71e00640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 0x7f8f74160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.193+0000 7f8f71400640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000d830 tx=0x7f8f6000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.194+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6000b6c0 con 0x7f8f740b25f0 2024-04-17T07:27:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.194+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f60019070 con 0x7f8f740b25f0 2024-04-17T07:27:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.194+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6000ba80 con 0x7f8f740b25f0 2024-04-17T07:27:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.194+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f741efee0 con 0x7f8f740b25f0 2024-04-17T07:27:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.194+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741f04a0 con 0x7f8f740b25f0 2024-04-17T07:27:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.195+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f74068a10 con 0x7f8f740b25f0 2024-04-17T07:27:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.196+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f8f6001e460 con 0x7f8f740b25f0 2024-04-17T07:27:21.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.197+0000 7f8f6a000640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 0x7f8f5406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.198+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f8f6008c3c0 con 0x7f8f740b25f0 2024-04-17T07:27:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.198+0000 7f8f71e00640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 0x7f8f5406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.199+0000 7f8f71e00640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 0x7f8f5406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f64005880 tx=0x7f8f64009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.202+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8f6005a6d0 con 0x7f8f740b25f0 2024-04-17T07:27:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.487+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8f7406cd40 con 0x7f8f740b25f0 2024-04-17T07:27:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.492+0000 7f8f6a000640 1 -- 172.21.15.67:0/104196331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7f8f60059fd0 con 0x7f8f740b25f0 2024-04-17T07:27:21.492 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:27:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:27:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:27:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:27:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:27:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 msgr2=0x7f8f5406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 0x7f8f5406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f64005880 tx=0x7f8f64009040 comp rx=0 tx=0).stop 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 msgr2=0x7f8f741556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000d830 tx=0x7f8f6000dcf0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 shutdown_connections 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f8f5406ca10 0x7f8f5406eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7413f6d0 0x7f8f74155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f740b25f0 0x7f8f741556e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 --2- 172.21.15.67:0/104196331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f74071d00 0x7f8f74160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 >> 172.21.15.67:0/104196331 conn(0x7f8f74067d10 msgr2=0x7f8f7415f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 shutdown_connections 2024-04-17T07:27:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.496+0000 7f8f73200640 1 -- 172.21.15.67:0/104196331 wait complete. 2024-04-17T07:27:21.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:27:21.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.733+0000 7f49ebe00640 1 Processor -- start 2024-04-17T07:27:21.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.734+0000 7f49ebe00640 1 -- start start 2024-04-17T07:27:21.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49ebe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49ec0d06b0 0x7f49ec0ae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49ebe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49ec06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49ebe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49ec0d1310 con 0x7f49ec161ec0 2024-04-17T07:27:21.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49ebe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec06f060 con 0x7f49ec0d06b0 2024-04-17T07:27:21.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49ebe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec0d0d50 con 0x7f49ec06b8c0 2024-04-17T07:27:21.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.735+0000 7f49eaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49ec161ec0 0x7f49ec0b0800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39028/0 (socket says 172.21.15.67:39028) 2024-04-17T07:27:21.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49eaa00640 1 -- 172.21.15.67:0/2336517164 learned_addr learned my addr 172.21.15.67:0/2336517164 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2336517164 >> v2:172.21.15.187:3300/0 conn(0x7f49ec0d06b0 0x7f49ec0ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49eb400640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49ec06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871002499 0 0) 0x7f49ec0d1310 con 0x7f49ec161ec0 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d0003520 con 0x7f49ec161ec0 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.736+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 154937060 0 0) 0x7f49d0003520 con 0x7f49ec161ec0 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 msgr2=0x7f49ec06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49ec06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 >> v2:172.21.15.187:3300/0 conn(0x7f49ec0d06b0 msgr2=0x7f49ec0ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 --2- 172.21.15.67:0/2336517164 >> v2:172.21.15.187:3300/0 conn(0x7f49ec0d06b0 0x7f49ec0ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0d1030 con 0x7f49ec161ec0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f49dc001670 con 0x7f49ec161ec0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 >> v1:172.21.15.67:6789/0 conn(0x7f49ec161ec0 legacy=0x7f49ec0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49d0003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d0003ff0 0x7f49d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0004d40 0x7f49d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec06f060 con 0x7f49d0003ff0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0d0d50 con 0x7f49d0004d40 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49d000bac0 con 0x7f49ec06b8c0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d0003ff0 0x7f49d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.737+0000 7f49eaa00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49d0003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.738+0000 7f49e3e00640 1 -- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 msgr2=0x7f49d0003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.738+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec06b8c0 0x7f49d0003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.738+0000 7f49e3e00640 1 -- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0004d40 msgr2=0x7f49d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.738+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0004d40 0x7f49d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.738+0000 7f49e3e00640 1 -- 172.21.15.67:0/2336517164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d0003520 con 0x7f49d0003ff0 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.739+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d0003ff0 0x7f49d0004800 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009ac0 tx=0x7f49d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4924bf42f2816df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.739+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4039070 con 0x7f49d0003ff0 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.739+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d402c710 con 0x7f49d0003ff0 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.739+0000 7f49ea000640 1 -- 172.21.15.67:0/2336517164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d4009690 con 0x7f49d0003ff0 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 -- 172.21.15.67:0/2336517164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d0003ff0 msgr2=0x7f49d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2336517164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49d0003ff0 0x7f49d0004800 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009ac0 tx=0x7f49d0005830 comp rx=0 tx=0).stop 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49eb400640 1 -- 172.21.15.67:0/2336517164 reap_dead start 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 -- 172.21.15.67:0/2336517164 shutdown_connections 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 -- 172.21.15.67:0/2336517164 >> 172.21.15.67:0/2336517164 conn(0x7f49ec09aa60 msgr2=0x7f49ec0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 -- 172.21.15.67:0/2336517164 shutdown_connections 2024-04-17T07:27:21.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.740+0000 7f49ebe00640 1 -- 172.21.15.67:0/2336517164 wait complete. 2024-04-17T07:27:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.742+0000 7f49ebe00640 1 Processor -- start 2024-04-17T07:27:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.742+0000 7f49ebe00640 1 -- start start 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 0x7f49ec1f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec156200 0x7f49ec0b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0ae870 con 0x7f49ec158080 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1622a0 con 0x7f49d0003ff0 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49ebe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec06bd50 con 0x7f49ec156200 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49eb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49eb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35964/0 (socket says 172.21.15.67:35964) 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49eb400640 1 -- 172.21.15.67:0/2114870591 learned_addr learned my addr 172.21.15.67:0/2114870591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49eaa00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 0x7f49ec1f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49e3e00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec156200 0x7f49ec0b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.743+0000 7f49eb400640 1 -- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec156200 msgr2=0x7f49ec0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eb400640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec156200 0x7f49ec0b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eb400640 1 -- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 msgr2=0x7f49ec1f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eb400640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 0x7f49ec1f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eb400640 1 -- 172.21.15.67:0/2114870591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1f5540 con 0x7f49ec158080 2024-04-17T07:27:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eaa00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 0x7f49ec1f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49eb400640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0d2590 tx=0x7f49e400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e400bb70 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e401a070 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1567a0 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e4014400 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.744+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec156d60 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.745+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec0755d0 con 0x7f49ec158080 2024-04-17T07:27:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.746+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f49e4005d10 con 0x7f49ec158080 2024-04-17T07:27:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.747+0000 7f49e3400640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 0x7f49c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.748+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f49e408be80 con 0x7f49ec158080 2024-04-17T07:27:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.748+0000 7f49eaa00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 0x7f49c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.749+0000 7f49eaa00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 0x7f49c806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0016f0 tx=0x7f49dc000f30 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:21.752+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f49ec1f5540 con 0x7f49ec158080 2024-04-17T07:27:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.002+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49ec0b7480 con 0x7f49ec158080 2024-04-17T07:27:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.003+0000 7f49e3400640 1 -- 172.21.15.67:0/2114870591 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7f49e4059ff0 con 0x7f49ec158080 2024-04-17T07:27:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:22.003 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:27:22.004 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 msgr2=0x7f49c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 0x7f49c806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0016f0 tx=0x7f49dc000f30 comp rx=0 tx=0).stop 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 msgr2=0x7f49ec158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0d2590 tx=0x7f49e400ef30 comp rx=0 tx=0).stop 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 shutdown_connections 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f49c806ca10 0x7f49c806eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49ec158080 0x7f49ec158500 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49ec156200 0x7f49ec0b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 --2- 172.21.15.67:0/2114870591 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49d0003ff0 0x7f49ec1f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 >> 172.21.15.67:0/2114870591 conn(0x7f49ec09aa60 msgr2=0x7f49ec15e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.006+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 shutdown_connections 2024-04-17T07:27:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.007+0000 7f49ebe00640 1 -- 172.21.15.67:0/2114870591 wait complete. 2024-04-17T07:27:22.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:27:14.836845+0000', 'modified': '2024-04-17T07:27:18.391129+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5931}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5931': {'gid': 5931, 'name': 'c', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/2094216919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 2094216919}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2094216919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5931, 'qdb_cluster': [5931]} 2024-04-17T07:27:22.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:27:22.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T07:27:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.663+0000 7f874f400640 1 Processor -- start 2024-04-17T07:27:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.663+0000 7f874f400640 1 -- start start 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8748160c30 0x7f8748161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874815cfc0 0x7f874806c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f874806c570 0x7f87480d0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f874806b8c0 con 0x7f874806c570 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f874806bba0 con 0x7f8748160c30 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f874815d800 con 0x7f874815cfc0 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f874806c570 0x7f87480d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f874806c570 0x7f87480d0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35980/0 (socket says 172.21.15.67:35980) 2024-04-17T07:27:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874d400640 1 -- 172.21.15.67:0/494018951 learned_addr learned my addr 172.21.15.67:0/494018951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f8747e00640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874815cfc0 0x7f874806c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.665+0000 7f874ca00640 1 --2- 172.21.15.67:0/494018951 >> v2:172.21.15.187:3300/0 conn(0x7f8748160c30 0x7f8748161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.666+0000 7f874d400640 1 -- 172.21.15.67:0/494018951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874815cfc0 msgr2=0x7f874806c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.666+0000 7f874d400640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874815cfc0 0x7f874806c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.666+0000 7f874d400640 1 -- 172.21.15.67:0/494018951 >> v2:172.21.15.187:3300/0 conn(0x7f8748160c30 msgr2=0x7f8748161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.666+0000 7f874d400640 1 --2- 172.21.15.67:0/494018951 >> v2:172.21.15.187:3300/0 conn(0x7f8748160c30 0x7f8748161010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.666+0000 7f874d400640 1 -- 172.21.15.67:0/494018951 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874815dac0 con 0x7f874806c570 2024-04-17T07:27:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.667+0000 7f874d400640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874806c570 0x7f87480d0980 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2690 tx=0x7f874002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca61a0967571458 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.667+0000 7f8747400640 1 -- 172.21.15.67:0/494018951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740034070 con 0x7f874806c570 2024-04-17T07:27:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.667+0000 7f8747400640 1 -- 172.21.15.67:0/494018951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8740030850 con 0x7f874806c570 2024-04-17T07:27:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.667+0000 7f8747400640 1 -- 172.21.15.67:0/494018951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740030c20 con 0x7f874806c570 2024-04-17T07:27:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 -- 172.21.15.67:0/494018951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874806c570 msgr2=0x7f87480d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874806c570 0x7f87480d0980 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2690 tx=0x7f874002b990 comp rx=0 tx=0).stop 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 -- 172.21.15.67:0/494018951 shutdown_connections 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874806c570 0x7f87480d0980 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 --2- 172.21.15.67:0/494018951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874815cfc0 0x7f874806c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 --2- 172.21.15.67:0/494018951 >> v2:172.21.15.187:3300/0 conn(0x7f8748160c30 0x7f8748161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 -- 172.21.15.67:0/494018951 >> 172.21.15.67:0/494018951 conn(0x7f8748073360 msgr2=0x7f874809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 -- 172.21.15.67:0/494018951 shutdown_connections 2024-04-17T07:27:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.668+0000 7f874f400640 1 -- 172.21.15.67:0/494018951 wait complete. 2024-04-17T07:27:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.670+0000 7f874f400640 1 Processor -- start 2024-04-17T07:27:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.670+0000 7f874f400640 1 -- start start 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 0x7f87480c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8748160c30 0x7f8748156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f874815d520 con 0x7f874815cfc0 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f874815d3a0 con 0x7f8748160c30 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f874f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f87480d0ec0 con 0x7f874806c570 2024-04-17T07:27:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f8747e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f8747e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35992/0 (socket says 172.21.15.67:35992) 2024-04-17T07:27:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.671+0000 7f8747e00640 1 -- 172.21.15.67:0/2377641072 learned_addr learned my addr 172.21.15.67:0/2377641072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f874ca00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 0x7f87480c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f874d400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8748160c30 0x7f8748156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 -- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 msgr2=0x7f87480c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 0x7f87480c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 -- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8748160c30 msgr2=0x7f8748156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8748160c30 0x7f8748156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 -- 172.21.15.67:0/2377641072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87480c8c00 con 0x7f874815cfc0 2024-04-17T07:27:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f874ca00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 0x7f87480c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8747e00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8738007e90 tx=0x7f873800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8744c00640 1 -- 172.21.15.67:0/2377641072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738014620 con 0x7f874815cfc0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.672+0000 7f8744c00640 1 -- 172.21.15.67:0/2377641072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8738019070 con 0x7f874815cfc0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.673+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8748154250 con 0x7f874815cfc0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.673+0000 7f8744c00640 1 -- 172.21.15.67:0/2377641072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738014a00 con 0x7f874815cfc0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.673+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87480c52a0 con 0x7f874815cfc0 2024-04-17T07:27:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.675+0000 7f8744c00640 1 -- 172.21.15.67:0/2377641072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f8738014ba0 con 0x7f874815cfc0 2024-04-17T07:27:22.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.676+0000 7f8744c00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 0x7f872406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.676+0000 7f8744c00640 1 -- 172.21.15.67:0/2377641072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f873808c570 con 0x7f874815cfc0 2024-04-17T07:27:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.676+0000 7f874ca00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 0x7f872406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.677+0000 7f87521c3740 1 Processor -- start 2024-04-17T07:27:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.677+0000 7f87521c3740 1 -- start start 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f874ca00640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 0x7f872406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b910f70 0x55e43b92fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b8e9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e43b84f980 con 0x55e43b913720 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e43b7f6220 con 0x55e43b92f160 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e43b84b940 con 0x55e43b910f70 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f874d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f874d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36002/0 (socket says 172.21.15.67:36002) 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.678+0000 7f874d400640 1 -- 172.21.15.67:0/2128386364 learned_addr learned my addr 172.21.15.67:0/2128386364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874ca00640 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b910f70 0x55e43b92fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f8747e00640 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b8e9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 -- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b910f70 msgr2=0x55e43b92fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b910f70 0x55e43b92fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 -- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 msgr2=0x55e43b8e9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b8e9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 -- 172.21.15.67:0/2128386364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43b5622d0 con 0x55e43b913720 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f874d400640 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f874002bfa0 tx=0x7f8740008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff702ff23effecc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.679+0000 7f873d200640 1 -- 172.21.15.67:0/2128386364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740034040 con 0x55e43b913720 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f873d200640 1 -- 172.21.15.67:0/2128386364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8740049070 con 0x55e43b913720 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f873d200640 1 -- 172.21.15.67:0/2128386364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740030b50 con 0x55e43b913720 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 -- 172.21.15.67:0/2128386364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 msgr2=0x55e43b913ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f874002bfa0 tx=0x7f8740008030 comp rx=0 tx=0).stop 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 -- 172.21.15.67:0/2128386364 shutdown_connections 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b913ba0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b8e9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 --2- 172.21.15.67:0/2128386364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b910f70 0x55e43b92fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 -- 172.21.15.67:0/2128386364 >> 172.21.15.67:0/2128386364 conn(0x55e43b8e5ba0 msgr2=0x55e43b92e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 -- 172.21.15.67:0/2128386364 shutdown_connections 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.680+0000 7f87521c3740 1 -- 172.21.15.67:0/2128386364 wait complete. 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.681+0000 7f87521c3740 1 -- start start 2024-04-17T07:27:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.681+0000 7f87521c3740 1 Processor -- start 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b913720 0x55e43b946750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b931440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e43b911350 con 0x55e43b931980 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e43b809590 con 0x55e43b92f160 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e43b8e38e0 con 0x55e43b913720 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.682+0000 7f87521c3740 10 client.0 _ll_register_callbacks cb 0x55e43b561ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36016/0 (socket says 172.21.15.67:36016) 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 -- 172.21.15.67:0/285690045 learned_addr learned my addr 172.21.15.67:0/285690045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f8747e00640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b931440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 -- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b913720 msgr2=0x55e43b946750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874ca00640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b913720 0x55e43b946750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b913720 0x55e43b946750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 -- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 msgr2=0x55e43b931440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b931440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 -- 172.21.15.67:0/285690045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43b62c6b0 con 0x55e43b931980 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f8747e00640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b931440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f874d400640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8740045000 tx=0x7f8740004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8816d76f0aa380 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.683+0000 7f8735600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740051070 con 0x55e43b931980 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f8740003610 con 0x55e43b931980 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8740049070 con 0x55e43b931980 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8740059050 con 0x55e43b931980 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 10 client.6072 fetch_fsmap learned FSMap version 26 2024-04-17T07:27:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x55e43b900c60 con 0x55e43b931980 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8740049070 con 0x55e43b931980 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) ==== 3495+0+0 (secure 0 0 0) 0x7f874005c020 con 0x55e43b931980 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 10 client.6072 fetch_fsmap finished waiting for FSMap version 26 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 10 client.6072 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 20 client.6072 populate_metadata read hostname 'smithi067' 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b8ffe60 0x55e43b900240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 4 client.6072 mds_command: new command op to 5931 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.684+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 --> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e43b85d3a0 con 0x55e43b8ffe60 2024-04-17T07:27:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.685+0000 7f874ca00640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b8ffe60 0x55e43b900240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.685+0000 7f874ca00640 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b8ffe60 0x55e43b900240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdd7f4b562a9737b server_cookie=1d2837bc8225f18c in_seq=0 out_seq=0 2024-04-17T07:27:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.685+0000 7f8735600640 10 client.6072 ms_handle_connect on v2:172.21.15.67:6834/2094216919 2024-04-17T07:27:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.686+0000 7f8735600640 1 -- 172.21.15.67:0/285690045 <== mds.0 v2:172.21.15.67:6834/2094216919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e43b85d3a0 con 0x55e43b8ffe60 2024-04-17T07:27:22.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.686+0000 7f8735600640 10 client.6072 handle_command_reply: tid=0 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.686+0000 7f87521c3740 1 client.6072 shutdown 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.686+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 msgr2=0x55e43b92d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.686+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8740045000 tx=0x7f8740004730 comp rx=0 tx=0).stop 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 shutdown_connections 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b8ffe60 0x55e43b900240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b931980 0x55e43b92d700 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b931440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 --2- 172.21.15.67:0/285690045 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b913720 0x55e43b946750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f8735600640 0 client.6072 ms_handle_reset on v2:172.21.15.67:6834/2094216919 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 >> 172.21.15.67:0/285690045 conn(0x55e43b910f70 msgr2=0x55e43b912990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 shutdown_connections 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 1 -- 172.21.15.67:0/285690045 wait complete. 2024-04-17T07:27:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.687+0000 7f87521c3740 20 client.6072 trim_cache size 0 max 16384 2024-04-17T07:27:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.707+0000 7f87521c3740 1 Processor -- start 2024-04-17T07:27:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.707+0000 7f87521c3740 1 -- start start 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b92c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b931980 0x55e43b92ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f87480cdaf0 con 0x55e43b913720 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f87480cce50 con 0x55e43b92f160 2024-04-17T07:27:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f87480cd4a0 con 0x55e43b931980 2024-04-17T07:27:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f874ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f874ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36026/0 (socket says 172.21.15.67:36026) 2024-04-17T07:27:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.708+0000 7f874ca00640 1 -- 172.21.15.67:0/3938925004 learned_addr learned my addr 172.21.15.67:0/3938925004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f8747e00640 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b92c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874d400640 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b931980 0x55e43b92ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 -- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b931980 msgr2=0x55e43b92ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b931980 0x55e43b92ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 -- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 msgr2=0x55e43b92c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b92c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 -- 172.21.15.67:0/3938925004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43b8fde60 con 0x55e43b913720 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f874ca00640 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87300041e0 tx=0x7f8730014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f15459e5ec7a372 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.709+0000 7f8737e00640 1 -- 172.21.15.67:0/3938925004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873002b070 con 0x55e43b913720 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f8737e00640 1 -- 172.21.15.67:0/3938925004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873000ee10 con 0x55e43b913720 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f8737e00640 1 -- 172.21.15.67:0/3938925004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873000f040 con 0x55e43b913720 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 -- 172.21.15.67:0/3938925004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 msgr2=0x55e43b936840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87300041e0 tx=0x7f8730014000 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 -- 172.21.15.67:0/3938925004 shutdown_connections 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b931980 0x55e43b92ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b92f160 0x55e43b92c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 --2- 172.21.15.67:0/3938925004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b936840 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 -- 172.21.15.67:0/3938925004 >> 172.21.15.67:0/3938925004 conn(0x55e43b910f70 msgr2=0x55e43b913b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 -- 172.21.15.67:0/3938925004 shutdown_connections 2024-04-17T07:27:22.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.710+0000 7f87521c3740 1 -- 172.21.15.67:0/3938925004 wait complete. 2024-04-17T07:27:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.711+0000 7f87521c3740 1 -- start start 2024-04-17T07:27:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.711+0000 7f87521c3740 1 Processor -- start 2024-04-17T07:27:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b92f160 0x55e43b9ad040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 0x55e43b9ad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e43b8fdc20 con 0x55e43b913720 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e43b911350 con 0x55e43b931980 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e43b903760 con 0x55e43b92f160 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f87521c3740 10 client.0 _ll_register_callbacks cb 0x7f8740006a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f874ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.712+0000 7f874ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36034/0 (socket says 172.21.15.67:36034) 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 -- 172.21.15.67:0/940241362 learned_addr learned my addr 172.21.15.67:0/940241362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874d400640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 0x55e43b9ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f8747e00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b92f160 0x55e43b9ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 -- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b92f160 msgr2=0x55e43b9ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b92f160 0x55e43b9ad040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 -- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 msgr2=0x55e43b9ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 0x55e43b9ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 -- 172.21.15.67:0/940241362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43b9168a0 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874d400640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 0x55e43b9ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f874ca00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f87300091a0 tx=0x7f8730004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7dd11e2a6388be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f8735600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.713+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873002b040 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f8730031af0 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f873000ee10 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8730031cf0 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 10 client.6078 fetch_fsmap learned FSMap version 26 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x55e43b910a50 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873000ee10 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) ==== 3495+0+0 (secure 0 0 0) 0x7f873003b080 con 0x55e43b913720 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 10 client.6078 fetch_fsmap finished waiting for FSMap version 26 2024-04-17T07:27:22.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 10 client.6078 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:27:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 20 client.6078 populate_metadata read hostname 'smithi067' 2024-04-17T07:27:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b92eab0 0x55e43b9350f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 4 client.6078 mds_command: new command op to 5931 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:27:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.714+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 --> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] -- command(tid 0: {"prefix": "status"}) -- 0x55e43b857210 con 0x55e43b92eab0 2024-04-17T07:27:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f8747e00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b92eab0 0x55e43b9350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f8747e00640 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b92eab0 0x55e43b9350f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c1030c6c5a5017f server_cookie=2eea042f4fca37a7 in_seq=0 out_seq=0 2024-04-17T07:27:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f8735600640 10 client.6078 ms_handle_connect on v2:172.21.15.67:6834/2094216919 2024-04-17T07:27:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f8735600640 1 -- 172.21.15.67:0/940241362 <== mds.0 v2:172.21.15.67:6834/2094216919 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55e43b857210 con 0x55e43b92eab0 2024-04-17T07:27:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f8735600640 10 client.6078 handle_command_reply: tid=0 2024-04-17T07:27:22.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.715+0000 7f87521c3740 1 client.6078 shutdown 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 msgr2=0x55e43b957150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f87300091a0 tx=0x7f8730004730 comp rx=0 tx=0).stop 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 shutdown_connections 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:6834/2094216919,v1:172.21.15.67:6835/2094216919] conn(0x55e43b92eab0 0x55e43b9350f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e43b931980 0x55e43b9ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e43b92f160 0x55e43b9ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 --2- 172.21.15.67:0/940241362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e43b913720 0x55e43b957150 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f8735600640 0 client.6078 ms_handle_reset on v2:172.21.15.67:6834/2094216919 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 >> 172.21.15.67:0/940241362 conn(0x55e43b910f70 msgr2=0x55e43b913b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 shutdown_connections 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 1 -- 172.21.15.67:0/940241362 wait complete. 2024-04-17T07:27:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.716+0000 7f87521c3740 20 client.6078 trim_cache size 0 max 16384 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5931, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8931669490000003, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 26, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 66, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 66, 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.246144691 2024-04-17T07:27:22.720 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 msgr2=0x7f872406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 0x7f872406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8730006fd0 tx=0x7f8730008040 comp rx=0 tx=0).stop 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 msgr2=0x7f8748156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8738007e90 tx=0x7f873800e4b0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 shutdown_connections 2024-04-17T07:27:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f872406ca10 0x7f872406eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8748160c30 0x7f8748156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874815cfc0 0x7f8748156a20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 --2- 172.21.15.67:0/2377641072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f874806c570 0x7f87480c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 >> 172.21.15.67:0/2377641072 conn(0x7f8748073360 msgr2=0x7f87480d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 shutdown_connections 2024-04-17T07:27:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.717+0000 7f874f400640 1 -- 172.21.15.67:0/2377641072 wait complete. 2024-04-17T07:27:22.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:27:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.948+0000 7fa0ade00640 1 Processor -- start 2024-04-17T07:27:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.948+0000 7fa0ade00640 1 -- start start 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0ade00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0a815c5a0 0x7fa0a815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0ade00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a815f440 con 0x7fa0a80b46c0 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0ade00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a815f720 con 0x7fa0a80b4210 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0ade00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a815fa00 con 0x7fa0a815c5a0 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0aca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0a815c5a0 0x7fa0a815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0aca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0a815c5a0 0x7fa0a815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36042/0 (socket says 172.21.15.67:36042) 2024-04-17T07:27:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0ad400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0a80b46c0 0x7fa0a80ae090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39034/0 (socket says 172.21.15.67:39034) 2024-04-17T07:27:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.950+0000 7fa0aca00640 1 -- 172.21.15.67:0/1641661871 learned_addr learned my addr 172.21.15.67:0/1641661871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.951+0000 7fa0aca00640 1 -- 172.21.15.67:0/1641661871 >> v1:172.21.15.187:6789/0 conn(0x7fa0a80b4210 legacy=0x7fa0a80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.951+0000 7fa0aca00640 1 -- 172.21.15.67:0/1641661871 >> v1:172.21.15.67:6789/0 conn(0x7fa0a80b46c0 legacy=0x7fa0a80ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.951+0000 7fa0aca00640 1 -- 172.21.15.67:0/1641661871 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a815fce0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.952+0000 7fa0aca00640 1 --2- 172.21.15.67:0/1641661871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a815ee60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa09c009870 tx=0x7fa09c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d7f72ab5ea317a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.952+0000 7fa0a7400640 1 -- 172.21.15.67:0/1641661871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c02e070 con 0x7fa0a815c5a0 2024-04-17T07:27:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.952+0000 7fa0a7400640 1 -- 172.21.15.67:0/1641661871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa09c037070 con 0x7fa0a815c5a0 2024-04-17T07:27:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.952+0000 7fa0a7400640 1 -- 172.21.15.67:0/1641661871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c031a00 con 0x7fa0a815c5a0 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 -- 172.21.15.67:0/1641661871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 msgr2=0x7fa0a815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 --2- 172.21.15.67:0/1641661871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a815ee60 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa09c009870 tx=0x7fa09c02b880 comp rx=0 tx=0).stop 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 -- 172.21.15.67:0/1641661871 shutdown_connections 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 --2- 172.21.15.67:0/1641661871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a815ee60 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 -- 172.21.15.67:0/1641661871 >> 172.21.15.67:0/1641661871 conn(0x7fa0a8067d10 msgr2=0x7fa0a80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 -- 172.21.15.67:0/1641661871 shutdown_connections 2024-04-17T07:27:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.953+0000 7fa0ade00640 1 -- 172.21.15.67:0/1641661871 wait complete. 2024-04-17T07:27:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.955+0000 7fa0ade00640 1 Processor -- start 2024-04-17T07:27:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.955+0000 7fa0ade00640 1 -- start start 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a80b4210 0x7fa0a814c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 0x7fa0a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a806e7f0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a806e670 con 0x7fa0a81f49c0 2024-04-17T07:27:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0ade00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a80afb50 con 0x7fa0a80b4210 2024-04-17T07:27:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36054/0 (socket says 172.21.15.67:36054) 2024-04-17T07:27:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.956+0000 7fa0a7e00640 1 -- 172.21.15.67:0/2391709958 learned_addr learned my addr 172.21.15.67:0/2391709958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0aca00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a80b4210 0x7fa0a814c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0ad400640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 0x7fa0a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 -- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a80b4210 msgr2=0x7fa0a814c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a80b4210 0x7fa0a814c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 -- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 msgr2=0x7fa0a806e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 0x7fa0a806e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 -- 172.21.15.67:0/2391709958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a81f4f10 con 0x7fa0a815c5a0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0ad400640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 0x7fa0a806e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a7e00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa09400c410 tx=0x7fa09400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa094017070 con 0x7fa0a815c5a0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0940035f0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa094003a20 con 0x7fa0a815c5a0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a81f51f0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.957+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a81f57b0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.959+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a8068a10 con 0x7fa0a815c5a0 2024-04-17T07:27:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.960+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fa094003bc0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.961+0000 7fa0a4c00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 0x7fa08806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.961+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fa09408b0c0 con 0x7fa0a815c5a0 2024-04-17T07:27:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.961+0000 7fa0aca00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 0x7fa08806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.963+0000 7fa0aca00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 0x7fa08806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa09c036000 tx=0x7fa09c008030 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:22.965+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa094059340 con 0x7fa0a815c5a0 2024-04-17T07:27:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.210+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa0a815eb70 con 0x7fa0a815c5a0 2024-04-17T07:27:23.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.211+0000 7fa0a4c00640 1 -- 172.21.15.67:0/2391709958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7fa094058c40 con 0x7fa0a815c5a0 2024-04-17T07:27:23.211 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:27:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 msgr2=0x7fa08806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 0x7fa08806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa09c036000 tx=0x7fa09c008030 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 msgr2=0x7fa0a81f89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa09400c410 tx=0x7fa09400c8d0 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 shutdown_connections 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fa08806ca10 0x7fa08806eed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a81f49c0 0x7fa0a806e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0a815c5a0 0x7fa0a81f89c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 --2- 172.21.15.67:0/2391709958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a80b4210 0x7fa0a814c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 >> 172.21.15.67:0/2391709958 conn(0x7fa0a8067d10 msgr2=0x7fa0a80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 shutdown_connections 2024-04-17T07:27:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23.214+0000 7fa0ade00640 1 -- 172.21.15.67:0/2391709958 wait complete. 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:27:23.234 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:27:23.235 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:27:23.235 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:27:23.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.235 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:27:23.287 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41801sec preferred_lft 41801sec 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.288 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:27:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:27:23.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.290 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:27:23.382 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:27:23.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.387 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:27:23.448 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:27:23.448 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:27:23.448 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:27:23.448 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:27:23.448 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:27:23.449 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:27:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:27:23.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:27:23.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.713 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:27:23.764 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:27:23.764 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:27:23.765 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:27:23.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:27:23.854 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:27:23.855 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:27:23.905 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:27:23.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:27:23.905 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:27:23.955 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:27:23.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:23.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:27:24.099 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:27:24.099 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:27:24.136 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:27:24.136 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:27:24.136 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:27:24.136 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:27:24.136 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:27:24.140 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:27:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:27:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:27:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:27:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:27:24.145 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:27:24.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:24.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:27:24.244 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:27:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:27:24.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:27:24.370 INFO:teuthology.orchestra.run.smithi067.stdout:client5383 2024-04-17T07:27:24.371 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:27:24.371 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:27:24.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:27:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.645+0000 7fd3c2000640 1 Processor -- start 2024-04-17T07:27:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.645+0000 7fd3c2000640 1 -- start start 2024-04-17T07:27:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3c2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3bc15b860 0x7fd3bc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3c2000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3bc0cfcc0 con 0x7fd3bc0ce4d0 2024-04-17T07:27:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3c2000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3bc071f60 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3c2000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd3bc071d10 con 0x7fd3bc15b860 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3bb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3bc15b860 0x7fd3bc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3bb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3bc15b860 0x7fd3bc0d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50044/0 (socket says 172.21.15.67:50044) 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.647+0000 7fd3bb400640 1 -- 172.21.15.67:0/4046406961 learned_addr learned my addr 172.21.15.67:0/4046406961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.648+0000 7fd3bb400640 1 -- 172.21.15.67:0/4046406961 >> v1:172.21.15.187:6789/0 conn(0x7fd3bc0b4f80 legacy=0x7fd3bc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.648+0000 7fd3bb400640 1 -- 172.21.15.67:0/4046406961 >> v1:172.21.15.67:6789/0 conn(0x7fd3bc0ce4d0 legacy=0x7fd3bc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.648+0000 7fd3bb400640 1 -- 172.21.15.67:0/4046406961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc072240 con 0x7fd3bc15b860 2024-04-17T07:27:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.649+0000 7fd3bb400640 1 --2- 172.21.15.67:0/4046406961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc15b860 0x7fd3bc0d0420 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd23df77c7bc7403 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.649+0000 7fd3baa00640 1 -- 172.21.15.67:0/4046406961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac034070 con 0x7fd3bc15b860 2024-04-17T07:27:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.649+0000 7fd3baa00640 1 -- 172.21.15.67:0/4046406961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac030860 con 0x7fd3bc15b860 2024-04-17T07:27:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.649+0000 7fd3baa00640 1 -- 172.21.15.67:0/4046406961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac030c30 con 0x7fd3bc15b860 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 -- 172.21.15.67:0/4046406961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc15b860 msgr2=0x7fd3bc0d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 --2- 172.21.15.67:0/4046406961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc15b860 0x7fd3bc0d0420 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 -- 172.21.15.67:0/4046406961 shutdown_connections 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 --2- 172.21.15.67:0/4046406961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc15b860 0x7fd3bc0d0420 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 -- 172.21.15.67:0/4046406961 >> 172.21.15.67:0/4046406961 conn(0x7fd3bc067ee0 msgr2=0x7fd3bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 -- 172.21.15.67:0/4046406961 shutdown_connections 2024-04-17T07:27:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.650+0000 7fd3c2000640 1 -- 172.21.15.67:0/4046406961 wait complete. 2024-04-17T07:27:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.652+0000 7fd3c2000640 1 Processor -- start 2024-04-17T07:27:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.652+0000 7fd3c2000640 1 -- start start 2024-04-17T07:27:24.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:24.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 0x7fd3bc146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 0x7fd3bc147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc072ae0 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc072800 con 0x7fd3bc15b860 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.653+0000 7fd3c2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3bc15bc40 con 0x7fd3bc0ce4d0 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50050/0 (socket says 172.21.15.67:50050) 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 -- 172.21.15.67:0/401518687 learned_addr learned my addr 172.21.15.67:0/401518687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3b3e00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 0x7fd3bc146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bbe00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 0x7fd3bc147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 -- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 msgr2=0x7fd3bc146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 0x7fd3bc146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 -- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 msgr2=0x7fd3bc147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 0x7fd3bc147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bb400640 1 -- 172.21.15.67:0/401518687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc15d2b0 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3bbe00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 0x7fd3bc147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.654+0000 7fd3b3e00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 0x7fd3bc146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3bb400640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac048070 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac030660 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac034040 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3bc15d590 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.655+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3bc144ad0 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.656+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3bc068ca0 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.657+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7fd3ac037070 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.658+0000 7fd3b3400640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 0x7fd39806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.658+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fd3ac0ade10 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.659+0000 7fd3b3e00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 0x7fd39806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.660+0000 7fd3b3e00640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 0x7fd39806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc1f7f70 tx=0x7fd3a4009040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.663+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3ac0b1030 con 0x7fd3bc0b4f80 2024-04-17T07:27:24.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.895+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 --> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd3bc159290 con 0x7fd39806ca10 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.899+0000 7fd3b3400640 1 -- 172.21.15.67:0/401518687 <== mgr.5134 v2:172.21.15.187:6832/1235852913 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fd3bc159290 con 0x7fd39806ca10 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:27:24.899 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.901+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 msgr2=0x7fd39806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.901+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 0x7fd39806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc1f7f70 tx=0x7fd3a4009040 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.901+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 msgr2=0x7fd3bc15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.901+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009870 tx=0x7fd3ac004730 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 shutdown_connections 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd39806ca10 0x7fd39806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3bc15b860 0x7fd3bc147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3bc0ce4d0 0x7fd3bc146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 --2- 172.21.15.67:0/401518687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3bc0b4f80 0x7fd3bc15c830 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 >> 172.21.15.67:0/401518687 conn(0x7fd3bc067ee0 msgr2=0x7fd3bc0b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 shutdown_connections 2024-04-17T07:27:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:24.902+0000 7fd3c2000640 1 -- 172.21.15.67:0/401518687 wait complete. 2024-04-17T07:27:24.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:27:25.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.130+0000 7f690e000640 1 Processor -- start 2024-04-17T07:27:25.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.131+0000 7f690e000640 1 -- start start 2024-04-17T07:27:25.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.132+0000 7f690e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69080b6770 0x7f690806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.132+0000 7f690e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69080b3bf0 0x7f690806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.132+0000 7f690e000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69080ce970 con 0x7f690815c710 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.132+0000 7f690e000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6908073800 con 0x7f69080b6770 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.132+0000 7f690e000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f69080cec10 con 0x7f69080b3bf0 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69080b6770 0x7f690806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69080b6770 0x7f690806c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50066/0 (socket says 172.21.15.67:50066) 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 -- 172.21.15.67:0/3998903838 learned_addr learned my addr 172.21.15.67:0/3998903838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f6907e00640 1 --2- 172.21.15.67:0/3998903838 >> v2:172.21.15.187:3300/0 conn(0x7f69080b3bf0 0x7f690806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 -- 172.21.15.67:0/3998903838 >> v2:172.21.15.187:3300/0 conn(0x7f69080b3bf0 msgr2=0x7f690806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 --2- 172.21.15.67:0/3998903838 >> v2:172.21.15.187:3300/0 conn(0x7f69080b3bf0 0x7f690806ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 -- 172.21.15.67:0/3998903838 >> v1:172.21.15.187:6789/0 conn(0x7f690815c710 legacy=0x7f69080730f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.133+0000 7f68ffe00640 1 -- 172.21.15.67:0/3998903838 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69080ceef0 con 0x7f69080b6770 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.134+0000 7f68ffe00640 1 --2- 172.21.15.67:0/3998903838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b6770 0x7f690806c8d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f69080cd6e0 tx=0x7f68f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283796dd12455f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.135+0000 7f6906a00640 1 -- 172.21.15.67:0/3998903838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0035020 con 0x7f69080b6770 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.135+0000 7f6906a00640 1 -- 172.21.15.67:0/3998903838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f0031870 con 0x7f69080b6770 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.135+0000 7f6906a00640 1 -- 172.21.15.67:0/3998903838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0031c40 con 0x7f69080b6770 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 -- 172.21.15.67:0/3998903838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b6770 msgr2=0x7f690806c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 --2- 172.21.15.67:0/3998903838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b6770 0x7f690806c8d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f69080cd6e0 tx=0x7f68f002b910 comp rx=0 tx=0).stop 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 -- 172.21.15.67:0/3998903838 shutdown_connections 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 --2- 172.21.15.67:0/3998903838 >> v2:172.21.15.187:3300/0 conn(0x7f69080b3bf0 0x7f690806ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 --2- 172.21.15.67:0/3998903838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b6770 0x7f690806c8d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 -- 172.21.15.67:0/3998903838 >> 172.21.15.67:0/3998903838 conn(0x7f69080725d0 msgr2=0x7f69080729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 -- 172.21.15.67:0/3998903838 shutdown_connections 2024-04-17T07:27:25.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.136+0000 7f690e000640 1 -- 172.21.15.67:0/3998903838 wait complete. 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.138+0000 7f690e000640 1 Processor -- start 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.138+0000 7f690e000640 1 -- start start 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69080b6770 0x7f690815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 0x7f690815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f690815caf0 con 0x7f69080b3bf0 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69080cf4b0 con 0x7f69080b6770 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f690e000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f690815cc70 con 0x7f690815c710 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f6907400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f6907400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50078/0 (socket says 172.21.15.67:50078) 2024-04-17T07:27:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.139+0000 7f6907400640 1 -- 172.21.15.67:0/3446421616 learned_addr learned my addr 172.21.15.67:0/3446421616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f68ffe00640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69080b6770 0x7f690815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907e00640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 0x7f690815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 -- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 msgr2=0x7f690815b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 0x7f690815b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 -- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69080b6770 msgr2=0x7f690815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69080b6770 0x7f690815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 -- 172.21.15.67:0/3446421616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69080cce40 con 0x7f69080b3bf0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907e00640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 0x7f690815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f6907400640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f68f800b420 tx=0x7f68f800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8011830 con 0x7f69080b3bf0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f8016070 con 0x7f69080b3bf0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.140+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8011c10 con 0x7f69080b3bf0 2024-04-17T07:27:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.141+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69080cd120 con 0x7f69080b3bf0 2024-04-17T07:27:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.141+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69080c5dc0 con 0x7f69080b3bf0 2024-04-17T07:27:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.142+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6908068ca0 con 0x7f69080b3bf0 2024-04-17T07:27:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.143+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 91142+0+0 (secure 0 0 0) 0x7f68f801b460 con 0x7f69080b3bf0 2024-04-17T07:27:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.144+0000 7f68ff400640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 0x7f68e406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.144+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f68f808b880 con 0x7f69080b3bf0 2024-04-17T07:27:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.145+0000 7f68ffe00640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 0x7f68e406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:25.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.146+0000 7f68ffe00640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 0x7f68e406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68f0036000 tx=0x7f68f003c040 comp rx=0 tx=0).ready entity=mgr.5134 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.148+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f68f8020020 con 0x7f69080b3bf0 2024-04-17T07:27:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.425+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f69080c4000 con 0x7f69080b3bf0 2024-04-17T07:27:25.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.947+0000 7f68ff400640 1 -- 172.21.15.67:0/3446421616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f68f8059b00 con 0x7f69080b3bf0 2024-04-17T07:27:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 msgr2=0x7f68e406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 0x7f68e406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f68f0036000 tx=0x7f68f003c040 comp rx=0 tx=0).stop 2024-04-17T07:27:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 msgr2=0x7f69081f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f68f800b420 tx=0x7f68f800b8e0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 shutdown_connections 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f68e406ca10 0x7f68e406eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f690815c710 0x7f690815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69080b6770 0x7f690815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.950+0000 7f690e000640 1 --2- 172.21.15.67:0/3446421616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69080b3bf0 0x7f69081f7c70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.951+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 >> 172.21.15.67:0/3446421616 conn(0x7f69080725d0 msgr2=0x7f6908158a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.951+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 shutdown_connections 2024-04-17T07:27:25.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:25.953+0000 7f690e000640 1 -- 172.21.15.67:0/3446421616 wait complete. 2024-04-17T07:27:25.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:27:26.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.176+0000 7fd0df200640 1 Processor -- start 2024-04-17T07:27:26.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.176+0000 7fd0df200640 1 -- start start 2024-04-17T07:27:26.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.177+0000 7fd0df200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0d815b800 0x7fd0d815ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0df200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d80d02c0 0x7fd0d815c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0df200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0d815c740 con 0x7fd0d815bbe0 2024-04-17T07:27:26.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0df200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d81604b0 con 0x7fd0d815b800 2024-04-17T07:27:26.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0df200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d815ca20 con 0x7fd0d80d02c0 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0d7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0d815b800 0x7fd0d815ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0d7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0d815b800 0x7fd0d815ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:43410/0 (socket says 172.21.15.67:43410) 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0d7e00640 1 -- 172.21.15.67:0/3084943001 learned_addr learned my addr 172.21.15.67:0/3084943001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.178+0000 7fd0dd200640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d80d02c0 0x7fd0d815c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.179+0000 7fd0d7e00640 1 -- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d80d02c0 msgr2=0x7fd0d815c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.179+0000 7fd0d7e00640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d80d02c0 0x7fd0d815c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.179+0000 7fd0d7e00640 1 -- 172.21.15.67:0/3084943001 >> v1:172.21.15.187:6789/0 conn(0x7fd0d815bbe0 legacy=0x7fd0d80cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.179+0000 7fd0d7e00640 1 -- 172.21.15.67:0/3084943001 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d815cd00 con 0x7fd0d815b800 2024-04-17T07:27:26.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.180+0000 7fd0d7e00640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815b800 0x7fd0d815ff70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009870 tx=0x7fd0c802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b8506aac716db2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:26.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.180+0000 7fd0d7400640 1 -- 172.21.15.67:0/3084943001 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8034070 con 0x7fd0d815b800 2024-04-17T07:27:26.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0d7400640 1 -- 172.21.15.67:0/3084943001 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c8030840 con 0x7fd0d815b800 2024-04-17T07:27:26.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0d7400640 1 -- 172.21.15.67:0/3084943001 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8030c10 con 0x7fd0d815b800 2024-04-17T07:27:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 -- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815b800 msgr2=0x7fd0d815ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815b800 0x7fd0d815ff70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009870 tx=0x7fd0c802b980 comp rx=0 tx=0).stop 2024-04-17T07:27:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 -- 172.21.15.67:0/3084943001 shutdown_connections 2024-04-17T07:27:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d80d02c0 0x7fd0d815c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 --2- 172.21.15.67:0/3084943001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815b800 0x7fd0d815ff70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 -- 172.21.15.67:0/3084943001 >> 172.21.15.67:0/3084943001 conn(0x7fd0d8073360 msgr2=0x7fd0d809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 -- 172.21.15.67:0/3084943001 shutdown_connections 2024-04-17T07:27:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.181+0000 7fd0df200640 1 -- 172.21.15.67:0/3084943001 wait complete. 2024-04-17T07:27:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.183+0000 7fd0df200640 1 Processor -- start 2024-04-17T07:27:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.183+0000 7fd0df200640 1 -- start start 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 0x7fd0d81561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 0x7fd0d80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d80d1400 con 0x7fd0d80d02c0 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d80d1280 con 0x7fd0d815bbe0 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0df200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d80d1580 con 0x7fd0d815b800 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50104/0 (socket says 172.21.15.67:50104) 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.184+0000 7fd0d7e00640 1 -- 172.21.15.67:0/2631243489 learned_addr learned my addr 172.21.15.67:0/2631243489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0dd200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 0x7fd0d80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 msgr2=0x7fd0d81561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0cfe00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 0x7fd0d81561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 0x7fd0d81561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 msgr2=0x7fd0d80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 0x7fd0d80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 -- 172.21.15.67:0/2631243489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d80c6b20 con 0x7fd0d80d02c0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0cfe00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 0x7fd0d81561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0dd200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 0x7fd0d80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d7e00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000c00 tx=0x7fd0c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.185+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8048070 con 0x7fd0d80d02c0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.186+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c8030640 con 0x7fd0d80d02c0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.186+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8034040 con 0x7fd0d80d02c0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.186+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d80c6e00 con 0x7fd0d80d02c0 2024-04-17T07:27:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.186+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d80c73c0 con 0x7fd0d80d02c0 2024-04-17T07:27:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.187+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d80d2430 con 0x7fd0d80d02c0 2024-04-17T07:27:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.188+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) ==== 91159+0+0 (secure 0 0 0) 0x7fd0c8037070 con 0x7fd0d80d02c0 2024-04-17T07:27:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.189+0000 7fd0d4c00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 0x7fd0b406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.189+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fd0c80adef0 con 0x7fd0d80d02c0 2024-04-17T07:27:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.189+0000 7fd0cfe00640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 msgr2=0x7fd0b406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:26.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.189+0000 7fd0cfe00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 0x7fd0b406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:27:26.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.193+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd0c8051020 con 0x7fd0d80d02c0 2024-04-17T07:27:26.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.391+0000 7fd0cfe00640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 msgr2=0x7fd0b406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:26.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.391+0000 7fd0cfe00640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 0x7fd0b406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:27:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.396+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fd0d815f8e0 con 0x7fd0d80d02c0 2024-04-17T07:27:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.401+0000 7fd0d4c00640 1 -- 172.21.15.67:0/2631243489 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) ==== 138+0+0 (secure 0 0 0) 0x7fd0c807c100 con 0x7fd0d80d02c0 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 msgr2=0x7fd0b406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 0x7fd0b406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 msgr2=0x7fd0d80c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000c00 tx=0x7fd0c8004730 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 shutdown_connections 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fd0b406ca60 0x7fd0b406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d815bbe0 0x7fd0d80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d815b800 0x7fd0d81561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 --2- 172.21.15.67:0/2631243489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d02c0 0x7fd0d80c6410 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 >> 172.21.15.67:0/2631243489 conn(0x7fd0d8073360 msgr2=0x7fd0d8156d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.403+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 shutdown_connections 2024-04-17T07:27:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.404+0000 7fd0df200640 1 -- 172.21.15.67:0/2631243489 wait complete. 2024-04-17T07:27:26.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:27:26.521 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:26.522+0000 7efe4f042040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:27:26.538 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:26.538+0000 7fa6bb617040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:27:26.581 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:26.581+0000 7efe4f042040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:27:26.597 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:26.597+0000 7fa6bb617040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:27:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.646+0000 7f3b2e600640 1 Processor -- start 2024-04-17T07:27:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.646+0000 7f3b2e600640 1 -- start start 2024-04-17T07:27:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.647+0000 7f3b2e600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b280b46f0 0x7f3b280b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.647+0000 7f3b2e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.647+0000 7f3b2e600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b280b51d0 con 0x7f3b2815e9b0 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.647+0000 7f3b2e600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3b280b5790 con 0x7f3b280b46f0 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.647+0000 7f3b2e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b280d2830 con 0x7f3b280d4510 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47444/0 (socket says 172.21.15.67:47444) 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 -- 172.21.15.67:0/3551359313 learned_addr learned my addr 172.21.15.67:0/3551359313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b27e00640 1 --2- 172.21.15.67:0/3551359313 >> v2:172.21.15.187:3300/0 conn(0x7f3b280b46f0 0x7f3b280b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 -- 172.21.15.67:0/3551359313 >> v2:172.21.15.187:3300/0 conn(0x7f3b280b46f0 msgr2=0x7f3b280b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 --2- 172.21.15.67:0/3551359313 >> v2:172.21.15.187:3300/0 conn(0x7f3b280b46f0 0x7f3b280b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.648+0000 7f3b2dc00640 1 -- 172.21.15.67:0/3551359313 >> v1:172.21.15.187:6789/0 conn(0x7f3b2815e9b0 legacy=0x7f3b280cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.649+0000 7f3b2dc00640 1 -- 172.21.15.67:0/3551359313 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b280b54b0 con 0x7f3b280d4510 2024-04-17T07:27:26.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.650+0000 7f3b2dc00640 1 --2- 172.21.15.67:0/3551359313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009870 tx=0x7f3b2002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b976d197699be7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:26.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.650+0000 7f3b27400640 1 -- 172.21.15.67:0/3551359313 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b200317c0 con 0x7f3b280d4510 2024-04-17T07:27:26.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.650+0000 7f3b27400640 1 -- 172.21.15.67:0/3551359313 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b20037070 con 0x7f3b280d4510 2024-04-17T07:27:26.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.650+0000 7f3b27400640 1 -- 172.21.15.67:0/3551359313 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031b80 con 0x7f3b280d4510 2024-04-17T07:27:26.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 -- 172.21.15.67:0/3551359313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 msgr2=0x7f3b280d4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 --2- 172.21.15.67:0/3551359313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009870 tx=0x7f3b2002b860 comp rx=0 tx=0).stop 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 -- 172.21.15.67:0/3551359313 shutdown_connections 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 --2- 172.21.15.67:0/3551359313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b280d4960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 --2- 172.21.15.67:0/3551359313 >> v2:172.21.15.187:3300/0 conn(0x7f3b280b46f0 0x7f3b280b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 -- 172.21.15.67:0/3551359313 >> 172.21.15.67:0/3551359313 conn(0x7f3b2809daa0 msgr2=0x7f3b2808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 -- 172.21.15.67:0/3551359313 shutdown_connections 2024-04-17T07:27:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.651+0000 7f3b2e600640 1 -- 172.21.15.67:0/3551359313 wait complete. 2024-04-17T07:27:26.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.653+0000 7f3b2e600640 1 Processor -- start 2024-04-17T07:27:26.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.653+0000 7f3b2e600640 1 -- start start 2024-04-17T07:27:26.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.654+0000 7f3b2e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b28158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b2815e9b0 0x7f3b280ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b280cfae0 con 0x7f3b280b46f0 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3b280d4ea0 con 0x7f3b2815e9b0 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b28162830 con 0x7f3b280d4510 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50120/0 (socket says 172.21.15.67:50120) 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 -- 172.21.15.67:0/1327227173 learned_addr learned my addr 172.21.15.67:0/1327227173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b27e00640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b28158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2dc00640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b2815e9b0 0x7f3b280ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 msgr2=0x7f3b28158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b28158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b2815e9b0 msgr2=0x7f3b280ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.655+0000 7f3b2d200640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b2815e9b0 0x7f3b280ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b2d200640 1 -- 172.21.15.67:0/1327227173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b280cd3f0 con 0x7f3b280b46f0 2024-04-17T07:27:26.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b2d200640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800ba50 tx=0x7f3b1800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:26.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18019070 con 0x7f3b280b46f0 2024-04-17T07:27:26.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1800cc20 con 0x7f3b280b46f0 2024-04-17T07:27:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b280cd6d0 con 0x7f3b280b46f0 2024-04-17T07:27:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.656+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18014680 con 0x7f3b280b46f0 2024-04-17T07:27:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.657+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b280cbb30 con 0x7f3b280b46f0 2024-04-17T07:27:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.658+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b280755d0 con 0x7f3b280b46f0 2024-04-17T07:27:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.659+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) ==== 91159+0+0 (secure 0 0 0) 0x7f3b1801c070 con 0x7f3b280b46f0 2024-04-17T07:27:26.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.660+0000 7f3b24c00640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 0x7f3b0406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.661+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f3b1808b960 con 0x7f3b280b46f0 2024-04-17T07:27:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.661+0000 7f3b27e00640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 msgr2=0x7f3b0406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.661+0000 7f3b27e00640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 0x7f3b0406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:27:26.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.665+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3b180596c0 con 0x7f3b280b46f0 2024-04-17T07:27:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.862+0000 7f3b27e00640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 msgr2=0x7f3b0406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.862+0000 7f3b27e00640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 0x7f3b0406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:27:26.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.867+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f3b280b66f0 con 0x7f3b280b46f0 2024-04-17T07:27:26.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.872+0000 7f3b24c00640 1 -- 172.21.15.67:0/1327227173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) ==== 133+0+0 (secure 0 0 0) 0x7f3b1805d9c0 con 0x7f3b280b46f0 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 msgr2=0x7f3b0406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 0x7f3b0406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 msgr2=0x7f3b281f8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800ba50 tx=0x7f3b1800bf10 comp rx=0 tx=0).stop 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 shutdown_connections 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7f3b0406ca60 0x7f3b0406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b2815e9b0 0x7f3b280ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b280d4510 0x7f3b28158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 --2- 172.21.15.67:0/1327227173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b280b46f0 0x7f3b281f8a80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:26.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 >> 172.21.15.67:0/1327227173 conn(0x7f3b2809daa0 msgr2=0x7f3b280b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:26.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 shutdown_connections 2024-04-17T07:27:26.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:26.875+0000 7f3b2e600640 1 -- 172.21.15.67:0/1327227173 wait complete. 2024-04-17T07:27:26.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:26.895 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T07:27:26.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-17T07:27:27.042 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.042+0000 7fa6bb617040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:27:27.062 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.062+0000 7efe4f042040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:27:27.094 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.094+0000 7fa6bb617040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:27:27.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.118+0000 7efe4f042040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:27:27.143 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.143+0000 7fa6bb617040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:27:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.159+0000 7fec72000640 1 Processor -- start 2024-04-17T07:27:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.160+0000 7fec72000640 1 -- start start 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec72000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec72000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec6c0d4510 0x7fec6c0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec72000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c0b51d0 con 0x7fec6c15e9b0 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec72000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fec6c0b5790 con 0x7fec6c0b46f0 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec72000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fec6c0d2830 con 0x7fec6c0d4510 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.161+0000 7fec6be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50126/0 (socket says 172.21.15.67:50126) 2024-04-17T07:27:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 -- 172.21.15.67:0/3542328319 learned_addr learned my addr 172.21.15.67:0/3542328319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec71600640 1 --2- 172.21.15.67:0/3542328319 >> v2:172.21.15.187:3300/0 conn(0x7fec6c0d4510 0x7fec6c0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 -- 172.21.15.67:0/3542328319 >> v2:172.21.15.187:3300/0 conn(0x7fec6c0d4510 msgr2=0x7fec6c0d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 --2- 172.21.15.67:0/3542328319 >> v2:172.21.15.187:3300/0 conn(0x7fec6c0d4510 0x7fec6c0d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 -- 172.21.15.67:0/3542328319 >> v1:172.21.15.187:6789/0 conn(0x7fec6c15e9b0 legacy=0x7fec6c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.162+0000 7fec6be00640 1 -- 172.21.15.67:0/3542328319 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c0b54b0 con 0x7fec6c0b46f0 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.163+0000 7fec6be00640 1 --2- 172.21.15.67:0/3542328319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0b50e0 tx=0x7fec5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154ba9c8dfc14699 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.163+0000 7fec6b400640 1 -- 172.21.15.67:0/3542328319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5802e070 con 0x7fec6c0b46f0 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.163+0000 7fec6b400640 1 -- 172.21.15.67:0/3542328319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec58037070 con 0x7fec6c0b46f0 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec6b400640 1 -- 172.21.15.67:0/3542328319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58031a00 con 0x7fec6c0b46f0 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 -- 172.21.15.67:0/3542328319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0b46f0 msgr2=0x7fec6c0b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 --2- 172.21.15.67:0/3542328319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0b50e0 tx=0x7fec5802b880 comp rx=0 tx=0).stop 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 -- 172.21.15.67:0/3542328319 shutdown_connections 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 --2- 172.21.15.67:0/3542328319 >> v2:172.21.15.187:3300/0 conn(0x7fec6c0d4510 0x7fec6c0d4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 --2- 172.21.15.67:0/3542328319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0b46f0 0x7fec6c0b4ba0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 -- 172.21.15.67:0/3542328319 >> 172.21.15.67:0/3542328319 conn(0x7fec6c09daa0 msgr2=0x7fec6c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 -- 172.21.15.67:0/3542328319 shutdown_connections 2024-04-17T07:27:27.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.164+0000 7fec72000640 1 -- 172.21.15.67:0/3542328319 wait complete. 2024-04-17T07:27:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.166+0000 7fec72000640 1 Processor -- start 2024-04-17T07:27:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.166+0000 7fec72000640 1 -- start start 2024-04-17T07:27:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec6c0b46f0 0x7fec6c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 0x7fec6c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec6c15e9b0 0x7fec6c0cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c0d4e60 con 0x7fec6c0d4510 2024-04-17T07:27:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c0cfae0 con 0x7fec6c15e9b0 2024-04-17T07:27:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec72000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec6c162830 con 0x7fec6c0b46f0 2024-04-17T07:27:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.167+0000 7fec6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 0x7fec6c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 0x7fec6c0cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50132/0 (socket says 172.21.15.67:50132) 2024-04-17T07:27:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 -- 172.21.15.67:0/283310945 learned_addr learned my addr 172.21.15.67:0/283310945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec6c0b46f0 0x7fec6c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec71600640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec6c15e9b0 0x7fec6c0cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec6c0b46f0 msgr2=0x7fec6c15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec6c0b46f0 0x7fec6c15f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec6c15e9b0 msgr2=0x7fec6c0cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec6c15e9b0 0x7fec6c0cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c1601d0 con 0x7fec6c0d4510 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec6c0b46f0 0x7fec6c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec71600640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec6c15e9b0 0x7fec6c0cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.168+0000 7fec6be00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 0x7fec6c0cce20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fec5802b8c0 tx=0x7fec580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.169+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58032880 con 0x7fec6c0d4510 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.169+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec58032a20 con 0x7fec6c0d4510 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.169+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec6c1604b0 con 0x7fec6c0d4510 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.169+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58032e10 con 0x7fec6c0d4510 2024-04-17T07:27:27.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.169+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec6c1f6170 con 0x7fec6c0d4510 2024-04-17T07:27:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.170+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec6c0755d0 con 0x7fec6c0d4510 2024-04-17T07:27:27.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.171+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) ==== 91159+0+0 (secure 0 0 0) 0x7fec580453f0 con 0x7fec6c0d4510 2024-04-17T07:27:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.172+0000 7fec68c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.173+0000 7fec70c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.173+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7fec580adfb0 con 0x7fec6c0d4510 2024-04-17T07:27:27.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.173+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:27:27.173 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.174+0000 7efe4f042040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.177+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fec58040070 con 0x7fec6c0d4510 2024-04-17T07:27:27.206 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.206+0000 7fa6bb617040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:27:27.239 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.239+0000 7efe4f042040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:27:27.254 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.254+0000 7fa6bb617040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:27:27.292 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.292+0000 7efe4f042040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:27:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.374+0000 7fec70c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.374+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:27:27.383 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.383+0000 7fa6bb617040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:27:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.403+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fec6c0b6970 con 0x7fec4c06ca60 2024-04-17T07:27:27.428 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.428+0000 7efe4f042040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:27:27.445 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.445+0000 7fa6bb617040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:27:27.495 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.495+0000 7efe4f042040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:27:27.495 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.495+0000 7fa6bb617040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:27:27.547 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.547+0000 7efe4f042040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:27:27.553 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.553+0000 7fa6bb617040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:27:27.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:27:27.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:27:27.604 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.604+0000 7fa6bb617040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:27:27.608 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.608+0000 7efe4f042040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:27:27.662 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:27.662+0000 7efe4f042040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:27:27.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.775+0000 7fec70c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:27.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:27.775+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:27:27.995 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:27.995+0000 7fa6bb617040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:27:28.063 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.063+0000 7efe4f042040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:27:28.153 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.153+0000 7fa6bb617040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:27:28.232 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.231+0000 7efe4f042040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:27:28.263 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.263+0000 7fa6bb617040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:27:28.314 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.314+0000 7fa6bb617040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:27:28.347 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.346+0000 7efe4f042040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:27:28.403 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.399+0000 7efe4f042040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:27:28.439 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.439+0000 7fa6bb617040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:27:28.531 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.531+0000 7efe4f042040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:27:28.544 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.544+0000 7fa6bb617040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:27:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:28.576+0000 7fec70c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:28.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:28.576+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:27:28.643 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.643+0000 7efe4f042040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:27:28.718 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.717+0000 7fa6bb617040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:27:28.768 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.768+0000 7fa6bb617040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:27:28.827 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.827+0000 7efe4f042040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:27:28.879 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:28.879+0000 7efe4f042040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:27:28.903 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:28.903+0000 7fa6bb617040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:27:29.022 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:29.022+0000 7efe4f042040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:27:29.701 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:29.701+0000 7fa6bb617040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:27:29.876 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:29.876+0000 7efe4f042040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:27:30.006 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.006+0000 7fa6bb617040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:27:30.063 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.063+0000 7fa6bb617040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:27:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:30.178+0000 7fec70c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1235852913 2024-04-17T07:27:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:30.179+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T07:27:30.193 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.193+0000 7fa6bb617040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:27:30.196 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:30.196+0000 7efe4f042040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:27:30.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:30.220+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 20) ==== 90768+0+0 (secure 0 0 0) 0x7fec58078120 con 0x7fec6c0d4510 2024-04-17T07:27:30.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:30.221+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 msgr2=0x7fec4c06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:27:30.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:30.221+0000 7fec68c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/1235852913,v1:172.21.15.187:6833/1235852913] conn(0x7fec4c06ca60 0x7fec4c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:30.265 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:30.265+0000 7efe4f042040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:27:30.421 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:27:30.421+0000 7efe4f042040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:27:30.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.528+0000 7fa62e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.528+0000 7fa62e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.528+0000 7fa62e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.528+0000 7fa62e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.528+0000 7fa62e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.529+0000 7fa630a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.529+0000 7fa630a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.529+0000 7fa630a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.529+0000 7fa630a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:30.530 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:27:30.529+0000 7fa630a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:27:31.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.287+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 21) ==== 90895+0+0 (secure 0 0 0) 0x7fec58094290 con 0x7fec6c0d4510 2024-04-17T07:27:31.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.288+0000 7fec68c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fec4c074900 0x7fec4c076cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.289+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 --> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fec58043ec0 con 0x7fec4c074900 2024-04-17T07:27:31.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.297+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fec4c074900 0x7fec4c076cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.301+0000 7fec70c00640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fec4c074900 0x7fec4c076cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec60006ad0 tx=0x7fec600091f0 comp rx=0 tx=0).ready entity=mgr.5072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.494+0000 7fec68c00640 1 -- 172.21.15.67:0/283310945 <== mgr.5072 v2:172.21.15.187:6832/2009310366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fec58043ec0 con 0x7fec4c074900 2024-04-17T07:27:31.495 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T07:27:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fec4c074900 msgr2=0x7fec4c076cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fec4c074900 0x7fec4c076cf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec60006ad0 tx=0x7fec600091f0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 msgr2=0x7fec6c0cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 --2- 172.21.15.67:0/283310945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec6c0d4510 0x7fec6c0cce20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fec5802b8c0 tx=0x7fec580046c0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec71600640 1 -- 172.21.15.67:0/283310945 reap_dead start 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 shutdown_connections 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.497+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 >> 172.21.15.67:0/283310945 conn(0x7fec6c09daa0 msgr2=0x7fec6c0d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.498+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 shutdown_connections 2024-04-17T07:27:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.498+0000 7fec72000640 1 -- 172.21.15.67:0/283310945 wait complete. 2024-04-17T07:27:31.519 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T07:27:31.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-04-17T07:27:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.742+0000 7f33e1200640 1 Processor -- start 2024-04-17T07:27:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.742+0000 7f33e1200640 1 -- start start 2024-04-17T07:27:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33e1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc15b6b0 0x7f33dc06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33e1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 0x7f33dc06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33e1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33dc06df40 con 0x7f33dc06f800 2024-04-17T07:27:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33e1200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f33dc06ed00 con 0x7f33dc15b6b0 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33e1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f33dc06e870 con 0x7f33dc0cfc10 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.744+0000 7f33db400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc15b6b0 0x7f33dc06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc15b6b0 0x7f33dc06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50144/0 (socket says 172.21.15.67:50144) 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 -- 172.21.15.67:0/192870355 learned_addr learned my addr 172.21.15.67:0/192870355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33dbe00640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 0x7f33dc06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 -- 172.21.15.67:0/192870355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 msgr2=0x7f33dc06be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 0x7f33dc06be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 -- 172.21.15.67:0/192870355 >> v1:172.21.15.67:6789/0 conn(0x7f33dc06f800 legacy=0x7f33dc15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33db400640 1 -- 172.21.15.67:0/192870355 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc06e220 con 0x7f33dc15b6b0 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.745+0000 7f33dbe00640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 0x7f33dc06be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.746+0000 7f33db400640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc15b6b0 0x7f33dc06b8c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2a8bd57fdbffea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.747+0000 7f33da000640 1 -- 172.21.15.67:0/192870355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0317c0 con 0x7f33dc15b6b0 2024-04-17T07:27:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.747+0000 7f33da000640 1 -- 172.21.15.67:0/192870355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33cc037070 con 0x7f33dc15b6b0 2024-04-17T07:27:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.747+0000 7f33da000640 1 -- 172.21.15.67:0/192870355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031b80 con 0x7f33dc15b6b0 2024-04-17T07:27:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.747+0000 7f33e1200640 1 -- 172.21.15.67:0/192870355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc15b6b0 msgr2=0x7f33dc06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.747+0000 7f33e1200640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc15b6b0 0x7f33dc06b8c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc02b860 comp rx=0 tx=0).stop 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 -- 172.21.15.67:0/192870355 shutdown_connections 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc0cfc10 0x7f33dc06be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 --2- 172.21.15.67:0/192870355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc15b6b0 0x7f33dc06b8c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 -- 172.21.15.67:0/192870355 >> 172.21.15.67:0/192870355 conn(0x7f33dc0b2190 msgr2=0x7f33dc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 -- 172.21.15.67:0/192870355 shutdown_connections 2024-04-17T07:27:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.748+0000 7f33e1200640 1 -- 172.21.15.67:0/192870355 wait complete. 2024-04-17T07:27:31.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.749+0000 7f33e1200640 1 Processor -- start 2024-04-17T07:27:31.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.750+0000 7f33e1200640 1 -- start start 2024-04-17T07:27:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.750+0000 7f33e1200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 0x7f33dc14b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.750+0000 7f33e1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33e1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc15b6b0 0x7f33dc0cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33e1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0afee0 con 0x7f33dc0cfc10 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33e1200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0afd60 con 0x7f33dc06f800 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33e1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f33dc0b44a0 con 0x7f33dc15b6b0 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50146/0 (socket says 172.21.15.67:50146) 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 -- 172.21.15.67:0/2620863772 learned_addr learned my addr 172.21.15.67:0/2620863772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33db400640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 0x7f33dc14b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 -- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc15b6b0 msgr2=0x7f33dc0cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc15b6b0 0x7f33dc0cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 -- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 msgr2=0x7f33dc14b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 0x7f33dc14b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33daa00640 1 -- 172.21.15.67:0/2620863772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc14c690 con 0x7f33dc0cfc10 2024-04-17T07:27:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.751+0000 7f33db400640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 0x7f33dc14b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:27:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33daa00640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c340 tx=0x7f33c400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4017070 con 0x7f33dc0cfc10 2024-04-17T07:27:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c40052d0 con 0x7f33dc0cfc10 2024-04-17T07:27:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4003850 con 0x7f33dc0cfc10 2024-04-17T07:27:31.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc14c970 con 0x7f33dc0cfc10 2024-04-17T07:27:31.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.752+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc14cf30 con 0x7f33dc0cfc10 2024-04-17T07:27:31.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.753+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc068a10 con 0x7f33dc0cfc10 2024-04-17T07:27:31.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.754+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 21) ==== 90895+0+0 (secure 0 0 0) 0x7f33c401a070 con 0x7f33dc0cfc10 2024-04-17T07:27:31.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.755+0000 7f33d3400640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 0x7f33b806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:31.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.755+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6053+0+0 (secure 0 0 0) 0x7f33c408ad30 con 0x7f33dc0cfc10 2024-04-17T07:27:31.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.756+0000 7f33db400640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 0x7f33b806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:31.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.757+0000 7f33db400640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 0x7f33b806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc008030 comp rx=0 tx=0).ready entity=mgr.5072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:31.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.760+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f33c4059270 con 0x7f33dc0cfc10 2024-04-17T07:27:31.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:31.989+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 --> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) -- 0x7f33dc0b3210 con 0x7f33b806c790 2024-04-17T07:27:32.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.166+0000 7f33d3400640 1 -- 172.21.15.67:0/2620863772 <== mgr.5072 v2:172.21.15.187:6832/2009310366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f33dc0b3210 con 0x7f33b806c790 2024-04-17T07:27:32.167 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T07:27:32.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.170+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 msgr2=0x7f33b806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.170+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 0x7f33b806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc008030 comp rx=0 tx=0).stop 2024-04-17T07:27:32.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.170+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 msgr2=0x7f33dc0ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.170+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c340 tx=0x7f33c400c800 comp rx=0 tx=0).stop 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.170+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 shutdown_connections 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f33b806c790 0x7f33b806ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33dc15b6b0 0x7f33dc0cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0cfc10 0x7f33dc0ceb60 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 --2- 172.21.15.67:0/2620863772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc06f800 0x7f33dc14b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 >> 172.21.15.67:0/2620863772 conn(0x7f33dc0b2190 msgr2=0x7f33dc0d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 shutdown_connections 2024-04-17T07:27:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.171+0000 7f33e1200640 1 -- 172.21.15.67:0/2620863772 wait complete. 2024-04-17T07:27:32.196 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T07:27:32.196 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-17-07_28_00 in ~28s... 2024-04-17T07:27:32.196 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-17-07_29_00 in ~88s... 2024-04-17T07:27:32.196 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-04-17T07:27:32.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-17T07:27:32.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.422+0000 7f7c4be00640 1 Processor -- start 2024-04-17T07:27:32.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.422+0000 7f7c4be00640 1 -- start start 2024-04-17T07:27:32.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.424+0000 7f7c4be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.424+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.424+0000 7f7c4be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4c146360 con 0x7f7c4c0cd750 2024-04-17T07:27:32.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.424+0000 7f7c4be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4c0727a0 con 0x7f7c4c145f80 2024-04-17T07:27:32.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.424+0000 7f7c4be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c071e70 con 0x7f7c4c06c270 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.425+0000 7f7c4aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.425+0000 7f7c4b400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.425+0000 7f7c43e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7c4c0cd750 0x7f7c4c06bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:50344/0 (socket says 172.21.15.67:50344) 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.425+0000 7f7c4aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:43484/0 (socket says 172.21.15.67:43484) 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.425+0000 7f7c4b400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47482/0 (socket says 172.21.15.67:47482) 2024-04-17T07:27:32.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c43e00640 1 -- 172.21.15.67:0/898856709 learned_addr learned my addr 172.21.15.67:0/898856709 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c4b400640 1 -- 172.21.15.67:0/898856709 >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 msgr2=0x7f7c4c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c4b400640 1 --2- 172.21.15.67:0/898856709 >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c4b400640 1 -- 172.21.15.67:0/898856709 >> v1:172.21.15.187:6789/0 conn(0x7f7c4c0cd750 legacy=0x7f7c4c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c4b400640 1 -- 172.21.15.67:0/898856709 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c072150 con 0x7f7c4c06c270 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.426+0000 7f7c4aa00640 1 --2- 172.21.15.67:0/898856709 >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:27:32.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.428+0000 7f7c4b400640 1 --2- 172.21.15.67:0/898856709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009d90 tx=0x7f7c4402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21090025a7ade086 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.428+0000 7f7c4a000640 1 -- 172.21.15.67:0/898856709 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44039070 con 0x7f7c4c06c270 2024-04-17T07:27:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.428+0000 7f7c4a000640 1 -- 172.21.15.67:0/898856709 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4402cbc0 con 0x7f7c4c06c270 2024-04-17T07:27:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.428+0000 7f7c4a000640 1 -- 172.21.15.67:0/898856709 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44034440 con 0x7f7c4c06c270 2024-04-17T07:27:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 -- 172.21.15.67:0/898856709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 msgr2=0x7f7c4c06e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 --2- 172.21.15.67:0/898856709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009d90 tx=0x7f7c4402be10 comp rx=0 tx=0).stop 2024-04-17T07:27:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 -- 172.21.15.67:0/898856709 shutdown_connections 2024-04-17T07:27:32.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 --2- 172.21.15.67:0/898856709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c06c270 0x7f7c4c06e1c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 --2- 172.21.15.67:0/898856709 >> v2:172.21.15.187:3300/0 conn(0x7f7c4c145f80 0x7f7c4c0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 -- 172.21.15.67:0/898856709 >> 172.21.15.67:0/898856709 conn(0x7f7c4c0b2190 msgr2=0x7f7c4c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 -- 172.21.15.67:0/898856709 shutdown_connections 2024-04-17T07:27:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.429+0000 7f7c4be00640 1 -- 172.21.15.67:0/898856709 wait complete. 2024-04-17T07:27:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.431+0000 7f7c4be00640 1 Processor -- start 2024-04-17T07:27:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.432+0000 7f7c4be00640 1 -- start start 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c0cd750 0x7f7c4c142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c4c145f80 0x7f7c4c143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c072db0 con 0x7f7c4c06c270 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c072c30 con 0x7f7c4c145f80 2024-04-17T07:27:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c06e700 con 0x7f7c4c0cd750 2024-04-17T07:27:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50152/0 (socket says 172.21.15.67:50152) 2024-04-17T07:27:32.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.433+0000 7f7c4aa00640 1 -- 172.21.15.67:0/4127195072 learned_addr learned my addr 172.21.15.67:0/4127195072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 -- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c0cd750 msgr2=0x7f7c4c142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c0cd750 0x7f7c4c142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 -- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c4c145f80 msgr2=0x7f7c4c143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c4c145f80 0x7f7c4c143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 -- 172.21.15.67:0/4127195072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c15c260 con 0x7f7c4c06c270 2024-04-17T07:27:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4aa00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00e960 tx=0x7f7c3c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c017070 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c00ca00 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c00cdd0 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c13e870 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.434+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c13ee30 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.436+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c068a10 con 0x7f7c4c06c270 2024-04-17T07:27:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.436+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 22) ==== 91160+0+0 (secure 0 0 0) 0x7f7c3c01a070 con 0x7f7c4c06c270 2024-04-17T07:27:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.437+0000 7f7c43400640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 0x7f7c2806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:27:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.437+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6053+0+0 (secure 0 0 0) 0x7f7c3c08bd30 con 0x7f7c4c06c270 2024-04-17T07:27:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.438+0000 7f7c43e00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 0x7f7c2806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:27:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.439+0000 7f7c43e00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 0x7f7c2806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000f60 tx=0x7f7c34007a80 comp rx=0 tx=0).ready entity=mgr.5072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:27:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.442+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7c3c059f80 con 0x7f7c4c06c270 2024-04-17T07:27:32.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.668+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 --> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c4c0ce7c0 con 0x7f7c2806ca60 2024-04-17T07:27:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.687+0000 7f7c43400640 1 -- 172.21.15.67:0/4127195072 <== mgr.5072 v2:172.21.15.187:6832/2009310366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+76 (secure 0 0 0) 0x7f7c4c0ce7c0 con 0x7f7c2806ca60 2024-04-17T07:27:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:27:32.687 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-17T07:27:32.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 msgr2=0x7f7c2806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 0x7f7c2806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000f60 tx=0x7f7c34007a80 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 msgr2=0x7f7c4c15bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00e960 tx=0x7f7c3c00ee20 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 shutdown_connections 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f7c2806ca60 0x7f7c2806ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c4c145f80 0x7f7c4c143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c4c0cd750 0x7f7c4c142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 --2- 172.21.15.67:0/4127195072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c4c06c270 0x7f7c4c15bd20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 >> 172.21.15.67:0/4127195072 conn(0x7f7c4c0b2190 msgr2=0x7f7c4c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.689+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 shutdown_connections 2024-04-17T07:27:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:27:32.690+0000 7f7c4be00640 1 -- 172.21.15.67:0/4127195072 wait complete. 2024-04-17T07:27:32.711 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-04-17T07:27:32.711 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-04-17T07:27:32.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:32.711 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:32.766 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:32.766 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:32.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:33.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:33.767 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:33.829 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:33.829 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:33.830 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:34.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:34.831 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:34.894 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:34.894 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:34.894 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:35.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:35.896 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:35.956 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:35.956 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:35.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:36.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:36.958 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:37.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:37.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:37.021 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:38.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:38.022 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:38.084 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:38.084 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:38.084 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:39.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:39.085 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:39.147 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:39.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:39.148 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:40.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:40.349 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:40.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:40.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:40.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:41.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:41.415 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:41.479 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:41.479 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:41.479 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:42.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:42.480 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:42.547 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:42.547 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:42.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:43.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:43.548 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:43.611 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:43.611 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:43.611 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:44.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:44.612 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:44.676 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:44.676 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:44.676 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:45.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:45.677 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:45.740 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:45.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:45.740 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:46.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:46.742 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:46.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:46.807 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:46.807 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:47.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:47.808 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:47.873 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:47.873 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:47.873 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:48.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:48.874 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:48.937 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:48.937 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:48.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:49.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:49.939 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:50.001 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:50.002 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:50.002 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:51.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:51.003 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:51.067 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:51.068 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:51.068 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:52.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:52.069 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:52.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:52.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:52.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:53.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:53.136 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:53.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:53.202 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:53.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:54.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:54.203 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:54.263 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:54.263 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:54.263 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:55.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:55.264 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:55.328 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:55.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:55.329 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:56.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:56.330 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:56.392 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:56.392 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:56.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:57.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:57.393 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:57.456 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:57.456 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:57.456 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:57.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:27:57.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:27:58.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:58.458 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:58.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:58.522 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:58.522 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:27:59.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:27:59.523 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:27:59.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:27:59.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:27:59.589 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:00.191 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:28:00.191+0000 7fa61ca00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:28:00.191 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:28:00.191+0000 7fa61ca00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:28:00.191 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:28:00.191+0000 7fa61ca00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:28:00.192 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:28:00.191+0000 7fa61ca00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:28:00.192 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:28:00.191+0000 7fa61ca00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:28:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:00.590 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:00.654 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:00.656 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:00.656 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:00.656 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:00.656 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-17-07_28_00_UTC'] 2024-04-17T07:28:00.656 DEBUG:tasks.cephfs.test_snap_schedules:exec=1713338852.1966465, now = 1713338880.6564963, timo = 28 2024-04-17T07:28:01.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:01.657 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:01.717 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:01.718 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:01.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:01.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:02.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:02.720 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:02.781 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:02.782 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:02.783 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:02.783 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:03.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:03.784 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:03.844 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:03.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:03.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:03.845 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:04.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:04.846 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:04.909 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:04.911 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:04.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:04.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:05.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:05.912 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:05.976 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:05.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:05.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:05.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:06.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:06.979 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:07.042 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:07.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:07.043 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:07.043 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:08.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:08.044 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:08.104 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:08.105 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:08.105 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:08.105 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:09.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:09.107 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:09.170 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:09.172 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:09.172 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:09.172 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:10.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:10.172 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:10.234 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:10.235 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:10.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:10.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:11.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:11.236 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:11.298 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:11.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:11.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:11.300 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:12.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:12.300 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:12.363 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:12.365 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:12.365 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:12.365 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:13.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:13.366 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:13.428 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:13.429 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:13.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:13.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:14.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:14.431 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:14.494 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:14.495 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:14.495 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:14.495 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:15.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:15.496 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:15.558 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:15.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:15.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:15.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:16.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:16.560 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:16.623 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:16.624 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:16.624 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:16.625 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:17.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:17.626 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:17.688 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:17.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:17.690 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:17.690 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:18.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:18.691 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:18.755 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:18.756 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:18.757 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:18.757 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:19.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:19.758 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:19.821 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:19.823 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:19.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:19.823 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:20.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:20.824 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:20.887 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:20.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:20.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:20.888 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:21.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:21.890 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:21.952 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:21.954 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:21.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:21.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:22.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:22.955 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:23.017 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:23.018 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:23.018 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:23.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:24.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:24.019 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:24.080 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:24.082 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:24.082 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:24.082 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:25.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:25.083 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:25.145 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:25.146 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:25.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:25.147 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:26.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:26.148 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:26.209 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:26.211 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:26.211 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:26.211 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:27.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:27.212 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:27.274 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:27.275 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:27.275 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:27.275 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:27.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:28:27.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:28:28.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:28.276 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:28.338 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:28.339 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:28.339 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:28.339 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:29.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:29.340 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:29.403 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:29.406 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:29.406 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:29.406 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:30.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:30.407 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:30.470 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:30.471 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:30.471 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:30.471 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:31.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:31.473 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:31.536 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:31.537 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:31.537 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:31.537 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:32.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:32.538 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:32.599 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:32.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:32.601 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:32.601 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:33.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:33.602 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:33.664 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:33.665 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:33.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:33.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:34.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:34.666 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:34.730 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:34.731 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:34.731 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:34.731 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:35.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:35.732 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:35.795 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:35.796 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:35.797 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:35.797 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:36.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:36.797 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:36.858 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:36.859 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:36.860 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:36.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:37.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:37.861 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:37.924 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:37.925 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:37.925 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:37.925 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:38.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:38.926 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:38.987 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:38.988 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:38.988 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:38.988 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:39.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:39.989 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:40.050 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:40.052 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:40.052 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:40.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:41.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:41.053 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:41.115 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:41.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:41.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:41.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:42.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:42.117 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:42.179 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:42.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:42.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:42.180 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:43.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:43.182 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:43.245 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:43.247 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:43.247 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:43.247 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:44.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:44.247 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:44.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:44.308 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:44.308 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:44.308 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:45.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:45.309 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:45.370 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:45.372 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:45.372 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:45.372 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:46.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:46.372 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:46.435 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:46.436 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:46.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:46.437 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:47.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:47.437 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:47.500 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:47.501 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:47.501 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:47.501 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:48.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:48.502 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:48.566 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:48.567 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:48.567 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:48.567 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:49.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:49.569 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:49.632 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:49.633 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:49.634 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:49.634 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:50.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:50.635 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:50.697 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:50.698 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:50.698 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:50.698 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:51.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:51.699 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:51.761 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:51.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:51.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:51.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:52.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:52.764 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:52.825 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:52.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:52.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:52.827 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:53.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:53.829 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:53.894 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:53.895 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:53.895 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:53.895 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:54.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:54.896 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:54.958 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:54.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:54.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:54.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:55.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:55.961 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:56.023 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:56.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:56.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:56.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:57.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:57.025 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:57.089 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:57.090 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:57.091 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:57.091 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:57.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:28:57.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:28:58.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:58.092 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:58.151 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:58.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:58.153 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:58.153 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:28:59.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:28:59.154 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:28:59.216 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:28:59.218 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:28:59.218 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:28:59.218 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:29:00.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:00.219 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:29:00.282 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:29:00.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:29:00.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:29:00.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:29:01.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:01.284 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:29:01.347 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:29:01.347 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_29_00_UTC 2024-04-17T07:29:01.348 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_29_00_UTC', 'scheduled-2024-04-17-07_28_00_UTC'} 2024-04-17T07:29:01.348 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-17-07_29_00_UTC'} 2024-04-17T07:29:01.348 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:29:01.348 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-17-07_29_00_UTC'] 2024-04-17T07:29:01.348 DEBUG:tasks.cephfs.test_snap_schedules:exec=1713338852.1966465, now = 1713338941.3488111, timo = 88 2024-04-17T07:29:02.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-17T07:29:02.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.585+0000 7f4c35200640 1 Processor -- start 2024-04-17T07:29:02.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.585+0000 7f4c35200640 1 -- start start 2024-04-17T07:29:02.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.586+0000 7f4c35200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4c300b46f0 0x7f4c300b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.586+0000 7f4c35200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c300b51d0 con 0x7f4c3015e9b0 2024-04-17T07:29:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.586+0000 7f4c35200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c300b54b0 con 0x7f4c300d4510 2024-04-17T07:29:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.586+0000 7f4c35200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4c300b5790 con 0x7f4c300b46f0 2024-04-17T07:29:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.587+0000 7f4c2f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c3015e9b0 0x7f4c300cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55594/0 (socket says 172.21.15.67:55594) 2024-04-17T07:29:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.587+0000 7f4c2f400640 1 -- 172.21.15.67:0/90949159 learned_addr learned my addr 172.21.15.67:0/90949159 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.587+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214308537 0 0) 0x7f4c300b51d0 con 0x7f4c3015e9b0 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.587+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c14003520 con 0x7f4c3015e9b0 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.587+0000 7f4c2ea00640 1 --2- 172.21.15.67:0/90949159 >> v2:172.21.15.187:3300/0 conn(0x7f4c300b46f0 0x7f4c300b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3778437406 0 0) 0x7f4c14003520 con 0x7f4c3015e9b0 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 >> v2:172.21.15.187:3300/0 conn(0x7f4c300b46f0 msgr2=0x7f4c300b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 --2- 172.21.15.67:0/90949159 >> v2:172.21.15.187:3300/0 conn(0x7f4c300b46f0 0x7f4c300b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 >> v1:172.21.15.187:6789/0 conn(0x7f4c300d4510 legacy=0x7f4c300d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c300d24c0 con 0x7f4c3015e9b0 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4c20001670 con 0x7f4c3015e9b0 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 >> v1:172.21.15.67:6789/0 conn(0x7f4c3015e9b0 legacy=0x7f4c300cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c14003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c14004800 0x7f4c14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c140052e0 0x7f4c1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c300b5790 con 0x7f4c140039b0 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1400c1e0 con 0x7f4c140052e0 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c1400c3e0 con 0x7f4c14004800 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.588+0000 7f4c2f400640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c14003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2ea00640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c14004800 0x7f4c14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2f400640 1 -- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c14004800 msgr2=0x7f4c14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2fe00640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c140052e0 0x7f4c1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2f400640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c14004800 0x7f4c14004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2f400640 1 -- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c140052e0 msgr2=0x7f4c1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2f400640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c140052e0 0x7f4c1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2f400640 1 -- 172.21.15.67:0/90949159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c14003520 con 0x7f4c140039b0 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2fe00640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c140052e0 0x7f4c1400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.589+0000 7f4c2fe00640 1 -- 172.21.15.67:0/90949159 reap_dead start 2024-04-17T07:29:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.590+0000 7f4c2f400640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c14003df0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c210 tx=0x7f4c14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef8586da90c58d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.590+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20007cf0 con 0x7f4c140039b0 2024-04-17T07:29:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.590+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c20030070 con 0x7f4c140039b0 2024-04-17T07:29:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.590+0000 7f4c2e000640 1 -- 172.21.15.67:0/90949159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20005d70 con 0x7f4c140039b0 2024-04-17T07:29:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 -- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 msgr2=0x7f4c14003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c14003df0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c210 tx=0x7f4c14005830 comp rx=0 tx=0).stop 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 -- 172.21.15.67:0/90949159 shutdown_connections 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 --2- 172.21.15.67:0/90949159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c14003df0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 -- 172.21.15.67:0/90949159 >> 172.21.15.67:0/90949159 conn(0x7f4c3009daa0 msgr2=0x7f4c3008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 -- 172.21.15.67:0/90949159 shutdown_connections 2024-04-17T07:29:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.591+0000 7f4c35200640 1 -- 172.21.15.67:0/90949159 wait complete. 2024-04-17T07:29:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.593+0000 7f4c35200640 1 Processor -- start 2024-04-17T07:29:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.593+0000 7f4c35200640 1 -- start start 2024-04-17T07:29:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c300cb890 0x7f4c30153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c300b73e0 0x7f4c300cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3015ed90 con 0x7f4c140039b0 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4c300d2830 con 0x7f4c300b73e0 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c35200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3015ef10 con 0x7f4c300cb890 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49096/0 (socket says 172.21.15.67:49096) 2024-04-17T07:29:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.594+0000 7f4c2f400640 1 -- 172.21.15.67:0/979294349 learned_addr learned my addr 172.21.15.67:0/979294349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:02.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 -- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c300cb890 msgr2=0x7f4c30153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:02.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c300cb890 0x7f4c30153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 -- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c300b73e0 msgr2=0x7f4c300cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:02.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c300b73e0 0x7f4c300cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 -- 172.21.15.67:0/979294349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c300b7c30 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c2f400640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c210 tx=0x7f4c300d27a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20045070 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.595+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c20004bf0 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.596+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20001c30 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.596+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c300b7f10 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.596+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c300b84d0 con 0x7f4c140039b0 2024-04-17T07:29:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.597+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c300755d0 con 0x7f4c140039b0 2024-04-17T07:29:02.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.598+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 24) ==== 91273+0+0 (secure 0 0 0) 0x7f4c20007070 con 0x7f4c140039b0 2024-04-17T07:29:02.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.599+0000 7f4c27400640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 0x7f4c1006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:02.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.599+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6111+0+0 (secure 0 0 0) 0x7f4c200ad3d0 con 0x7f4c140039b0 2024-04-17T07:29:02.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.600+0000 7f4c2ea00640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 0x7f4c1006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.601+0000 7f4c2ea00640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 0x7f4c1006f070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0094c0 tx=0x7f4c1c00b040 comp rx=0 tx=0).ready entity=mgr.5072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.604+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4c2007ab90 con 0x7f4c140039b0 2024-04-17T07:29:02.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.696+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 25) ==== 91171+0+0 (secure 0 0 0) 0x7f4c20077200 con 0x7f4c140039b0 2024-04-17T07:29:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:02.837+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 --> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f4c300d0530 con 0x7f4c1006cbb0 2024-04-17T07:29:03.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.039+0000 7f4c27400640 1 -- 172.21.15.67:0/979294349 <== mgr.5072 v2:172.21.15.187:6832/2009310366 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f4c300d0530 con 0x7f4c1006cbb0 2024-04-17T07:29:03.039 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path snap_test_dir1 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 msgr2=0x7f4c1006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 0x7f4c1006f070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0094c0 tx=0x7f4c1c00b040 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 msgr2=0x7f4c300b9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c210 tx=0x7f4c300d27a0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 shutdown_connections 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f4c1006cbb0 0x7f4c1006f070 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c300b73e0 0x7f4c300cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c300cb890 0x7f4c30153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 --2- 172.21.15.67:0/979294349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c140039b0 0x7f4c300b9920 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.042+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 >> 172.21.15.67:0/979294349 conn(0x7f4c3009daa0 msgr2=0x7f4c300b66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.043+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 shutdown_connections 2024-04-17T07:29:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.043+0000 7f4c35200640 1 -- 172.21.15.67:0/979294349 wait complete. 2024-04-17T07:29:03.066 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-17T07:29:03.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:03.066 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:29:03.121 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:29:03.121 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_29_00_UTC 2024-04-17T07:29:03.122 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_28_00_UTC 2024-04-17T07:29:03.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:03.122 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_28_00_UTC) 2024-04-17T07:29:03.187 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_29_00_UTC 2024-04-17T07:29:03.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:03.187 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_29_00_UTC) 2024-04-17T07:29:03.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:03.254 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T07:29:03.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:29:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.536+0000 7fc82b200640 1 Processor -- start 2024-04-17T07:29:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.536+0000 7fc82b200640 1 -- start start 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc82b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc82c0d0260 0x7fc82c0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc82b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc82c0d0720 0x7fc82c0d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc82b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc82c0d1080 con 0x7fc82c15fda0 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc82b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc82c160180 con 0x7fc82c0d0260 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc82b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc82c06c0f0 con 0x7fc82c0d0720 2024-04-17T07:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc829400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc82c0d0260 0x7fc82c0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc829400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc82c0d0260 0x7fc82c0ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59374/0 (socket says 172.21.15.67:59374) 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.538+0000 7fc829400640 1 -- 172.21.15.67:0/4107205711 learned_addr learned my addr 172.21.15.67:0/4107205711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.539+0000 7fc829400640 1 -- 172.21.15.67:0/4107205711 >> v2:172.21.15.187:3300/0 conn(0x7fc82c0d0720 msgr2=0x7fc82c0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.539+0000 7fc829400640 1 --2- 172.21.15.67:0/4107205711 >> v2:172.21.15.187:3300/0 conn(0x7fc82c0d0720 0x7fc82c0d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.539+0000 7fc829400640 1 -- 172.21.15.67:0/4107205711 >> v1:172.21.15.187:6789/0 conn(0x7fc82c15fda0 legacy=0x7fc82c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.539+0000 7fc829400640 1 -- 172.21.15.67:0/4107205711 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c0cdb70 con 0x7fc82c0d0260 2024-04-17T07:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.540+0000 7fc829400640 1 --2- 172.21.15.67:0/4107205711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0260 0x7fc82c0ce650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc82c06c970 tx=0x7fc81802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5e2bf3afb7a1c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.540+0000 7fc828a00640 1 -- 172.21.15.67:0/4107205711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81802c9f0 con 0x7fc82c0d0260 2024-04-17T07:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.540+0000 7fc828a00640 1 -- 172.21.15.67:0/4107205711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81802cb90 con 0x7fc82c0d0260 2024-04-17T07:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.540+0000 7fc828a00640 1 -- 172.21.15.67:0/4107205711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8180343f0 con 0x7fc82c0d0260 2024-04-17T07:29:03.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 -- 172.21.15.67:0/4107205711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0260 msgr2=0x7fc82c0ce650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:03.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 --2- 172.21.15.67:0/4107205711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0260 0x7fc82c0ce650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc82c06c970 tx=0x7fc81802be50 comp rx=0 tx=0).stop 2024-04-17T07:29:03.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 -- 172.21.15.67:0/4107205711 shutdown_connections 2024-04-17T07:29:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 --2- 172.21.15.67:0/4107205711 >> v2:172.21.15.187:3300/0 conn(0x7fc82c0d0720 0x7fc82c0d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 --2- 172.21.15.67:0/4107205711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0260 0x7fc82c0ce650 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 -- 172.21.15.67:0/4107205711 >> 172.21.15.67:0/4107205711 conn(0x7fc82c09aa60 msgr2=0x7fc82c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 -- 172.21.15.67:0/4107205711 shutdown_connections 2024-04-17T07:29:03.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.541+0000 7fc82b200640 1 -- 172.21.15.67:0/4107205711 wait complete. 2024-04-17T07:29:03.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.543+0000 7fc82b200640 1 Processor -- start 2024-04-17T07:29:03.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.543+0000 7fc82b200640 1 -- start start 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc82c0d0260 0x7fc82c153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 0x7fc82c06f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c0ceb90 con 0x7fc82c0d0720 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c0ce130 con 0x7fc82c15fda0 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc82b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc82c0ced10 con 0x7fc82c0d0260 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc829400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc829400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59382/0 (socket says 172.21.15.67:59382) 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.544+0000 7fc829400640 1 -- 172.21.15.67:0/3251658894 learned_addr learned my addr 172.21.15.67:0/3251658894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:03.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc82a800640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 0x7fc82c06f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 -- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc82c0d0260 msgr2=0x7fc82c153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc82c0d0260 0x7fc82c153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 -- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 msgr2=0x7fc82c06f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 0x7fc82c06f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 -- 172.21.15.67:0/3251658894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c0b5ac0 con 0x7fc82c0d0720 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc82a800640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 0x7fc82c06f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc829400640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc8180093e0 tx=0x7fc818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818043020 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.545+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81803e450 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.546+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c0b5da0 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.546+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81803e840 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.546+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c0b6360 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.547+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c0755d0 con 0x7fc82c0d0720 2024-04-17T07:29:03.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.548+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 25) ==== 91171+0+0 (secure 0 0 0) 0x7fc818046020 con 0x7fc82c0d0720 2024-04-17T07:29:03.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.549+0000 7fc822000640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 0x7fc80c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:03.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.549+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6111+0+0 (secure 0 0 0) 0x7fc8180aea00 con 0x7fc82c0d0720 2024-04-17T07:29:03.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.550+0000 7fc829e00640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 0x7fc80c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.551+0000 7fc829e00640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 0x7fc80c06ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc81c006fd0 tx=0x7fc81c006bc0 comp rx=0 tx=0).ready entity=mgr.5072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.554+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc81804d030 con 0x7fc82c0d0720 2024-04-17T07:29:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:03.817+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fc82c0b6740 con 0x7fc82c0d0720 2024-04-17T07:29:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.704+0000 7fc822000640 1 -- 172.21.15.67:0/3251658894 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) ==== 93+0+0 (secure 0 0 0) 0x7fc81807caf0 con 0x7fc82c0d0720 2024-04-17T07:29:04.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.709+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 msgr2=0x7fc80c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:04.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.709+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 0x7fc80c06ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc81c006fd0 tx=0x7fc81c006bc0 comp rx=0 tx=0).stop 2024-04-17T07:29:04.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.709+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 msgr2=0x7fc82c0b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:04.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.709+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc8180093e0 tx=0x7fc818004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 shutdown_connections 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fc80c06ca60 0x7fc80c06ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc82c15fda0 0x7fc82c06f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc82c0d0720 0x7fc82c0b6b60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 --2- 172.21.15.67:0/3251658894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc82c0d0260 0x7fc82c153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:04.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.710+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 >> 172.21.15.67:0/3251658894 conn(0x7fc82c09aa60 msgr2=0x7fc82c0ae320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:04.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.713+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 shutdown_connections 2024-04-17T07:29:04.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:04.714+0000 7fc82b200640 1 -- 172.21.15.67:0/3251658894 wait complete. 2024-04-17T07:29:04.738 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:29:04.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:04.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:04.952 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:29:04.953 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:04.953 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:04.953 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:05.003 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:05.003 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:05.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:05.279 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.279+0000 7f3bb27be040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:05.291 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:05.291+0000 7fa75422b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.303+0000 7fae17200640 1 Processor -- start 2024-04-17T07:29:05.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.303+0000 7fae17200640 1 -- start start 2024-04-17T07:29:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.304+0000 7fae17200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae1815ec60 0x7fae180b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.304+0000 7fae17200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.304+0000 7fae17200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae180d2d30 con 0x7fae180b6b30 2024-04-17T07:29:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae17200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fae180d49c0 con 0x7fae1815ec60 2024-04-17T07:29:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae17200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae180d3660 con 0x7fae180cfae0 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae16800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae16800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:36812/0 (socket says 172.21.15.67:36812) 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae16800640 1 -- 172.21.15.67:0/2003428969 learned_addr learned my addr 172.21.15.67:0/2003428969 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.305+0000 7fae15e00640 1 --2- 172.21.15.67:0/2003428969 >> v2:172.21.15.187:3300/0 conn(0x7fae1815ec60 0x7fae180b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.306+0000 7fae16800640 1 -- 172.21.15.67:0/2003428969 >> v2:172.21.15.187:3300/0 conn(0x7fae1815ec60 msgr2=0x7fae180b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.306+0000 7fae16800640 1 --2- 172.21.15.67:0/2003428969 >> v2:172.21.15.187:3300/0 conn(0x7fae1815ec60 0x7fae180b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.306+0000 7fae16800640 1 -- 172.21.15.67:0/2003428969 >> v1:172.21.15.187:6789/0 conn(0x7fae180b6b30 legacy=0x7fae180cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.306+0000 7fae16800640 1 -- 172.21.15.67:0/2003428969 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae180d3010 con 0x7fae180cfae0 2024-04-17T07:29:05.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.307+0000 7fae16800640 1 --2- 172.21.15.67:0/2003428969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae10009e50 tx=0x7fae1002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c11c9e597750a74c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:05.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.308+0000 7fae15400640 1 -- 172.21.15.67:0/2003428969 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1002ca70 con 0x7fae180cfae0 2024-04-17T07:29:05.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.308+0000 7fae15400640 1 -- 172.21.15.67:0/2003428969 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1002cc10 con 0x7fae180cfae0 2024-04-17T07:29:05.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.308+0000 7fae15400640 1 -- 172.21.15.67:0/2003428969 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae100343f0 con 0x7fae180cfae0 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.308+0000 7fae17200640 1 -- 172.21.15.67:0/2003428969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 msgr2=0x7fae180b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.308+0000 7fae17200640 1 --2- 172.21.15.67:0/2003428969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae10009e50 tx=0x7fae1002bed0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 -- 172.21.15.67:0/2003428969 shutdown_connections 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 --2- 172.21.15.67:0/2003428969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae180cfae0 0x7fae180b7b70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 --2- 172.21.15.67:0/2003428969 >> v2:172.21.15.187:3300/0 conn(0x7fae1815ec60 0x7fae180b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 -- 172.21.15.67:0/2003428969 >> 172.21.15.67:0/2003428969 conn(0x7fae180d15f0 msgr2=0x7fae180d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 -- 172.21.15.67:0/2003428969 shutdown_connections 2024-04-17T07:29:05.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.309+0000 7fae17200640 1 -- 172.21.15.67:0/2003428969 wait complete. 2024-04-17T07:29:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.311+0000 7fae17200640 1 Processor -- start 2024-04-17T07:29:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.311+0000 7fae17200640 1 -- start start 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae180cfae0 0x7fae180ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae1815ec60 0x7fae180ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae1815f040 con 0x7fae180b6b30 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae180b6f70 con 0x7fae180cfae0 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.312+0000 7fae17200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae180d00c0 con 0x7fae1815ec60 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59412/0 (socket says 172.21.15.67:59412) 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 -- 172.21.15.67:0/936882159 learned_addr learned my addr 172.21.15.67:0/936882159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae16800640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae1815ec60 0x7fae180ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae0fe00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae180cfae0 0x7fae180ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae1815ec60 msgr2=0x7fae180ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae1815ec60 0x7fae180ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae180cfae0 msgr2=0x7fae180ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae180cfae0 0x7fae180ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 -- 172.21.15.67:0/936882159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae180ac060 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.313+0000 7fae15e00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9b0 tx=0x7fae0800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.314+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08017070 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.314+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0800c9a0 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.314+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae180ac340 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.314+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0800cd90 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.314+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae180ac900 con 0x7fae180b6b30 2024-04-17T07:29:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.315+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae18068a10 con 0x7fae180b6b30 2024-04-17T07:29:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.317+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7fae0801a070 con 0x7fae180b6b30 2024-04-17T07:29:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.318+0000 7fae0e000640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 0x7fadf406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.318+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6199+0+0 (secure 0 0 0) 0x7fae0808c600 con 0x7fae180b6b30 2024-04-17T07:29:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.318+0000 7fae0fe00640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 msgr2=0x7fadf406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.318+0000 7fae0fe00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 0x7fadf406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:05.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.321+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fae0805a150 con 0x7fae180b6b30 2024-04-17T07:29:05.336 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.336+0000 7f3bb27be040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:05.351 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:05.351+0000 7fa75422b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.519+0000 7fae0fe00640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 msgr2=0x7fadf406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.519+0000 7fae0fe00640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 0x7fadf406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.526+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae180b45e0 con 0x7fae180b6b30 2024-04-17T07:29:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.527+0000 7fae0e000640 1 -- 172.21.15.67:0/936882159 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+17318 (secure 0 0 0) 0x7fae0805e450 con 0x7fae180b6b30 2024-04-17T07:29:05.528 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:05.528 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":70,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:04.691131+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1671674456":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/4131277908":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3437520709":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3840279087":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3804745495":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1222851080":"2024-04-18T07:27:30.199060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 msgr2=0x7fadf406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 0x7fadf406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 msgr2=0x7fae180d3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9b0 tx=0x7fae0800ee70 comp rx=0 tx=0).stop 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 shutdown_connections 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fadf406ca10 0x7fadf406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae1815ec60 0x7fae180ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae180cfae0 0x7fae180ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 --2- 172.21.15.67:0/936882159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae180b6b30 0x7fae180d3d60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 >> 172.21.15.67:0/936882159 conn(0x7fae180d15f0 msgr2=0x7fae180b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 shutdown_connections 2024-04-17T07:29:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.530+0000 7fae17200640 1 -- 172.21.15.67:0/936882159 wait complete. 2024-04-17T07:29:05.550 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:27:12.167534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:27:13.288606+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:05.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.762+0000 7fda1da00640 1 Processor -- start 2024-04-17T07:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.762+0000 7fda1da00640 1 -- start start 2024-04-17T07:29:05.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.763+0000 7fda1da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 0x7fda180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.763+0000 7fda1da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda180cfcd0 0x7fda180d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.763+0000 7fda1da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda180d4f70 con 0x7fda1815ec20 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.763+0000 7fda1da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fda180d5250 con 0x7fda180cfcd0 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.763+0000 7fda1da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fda180cf4d0 con 0x7fda180b3b50 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda17e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fda1815ec20 0x7fda1806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51874/0 (socket says 172.21.15.67:51874) 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda17e00640 1 -- 172.21.15.67:0/203657890 learned_addr learned my addr 172.21.15.67:0/203657890 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda1d000640 1 --2- 172.21.15.67:0/203657890 >> v2:172.21.15.67:3300/0 conn(0x7fda180cfcd0 0x7fda180d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda0fe00640 1 --2- 172.21.15.67:0/203657890 >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 0x7fda180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda1d000640 1 -- 172.21.15.67:0/203657890 >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 msgr2=0x7fda180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda1d000640 1 --2- 172.21.15.67:0/203657890 >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 0x7fda180d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda1d000640 1 -- 172.21.15.67:0/203657890 >> v1:172.21.15.67:6789/0 conn(0x7fda1815ec20 legacy=0x7fda1806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda1d000640 1 -- 172.21.15.67:0/203657890 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda180cf7b0 con 0x7fda180cfcd0 2024-04-17T07:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.764+0000 7fda0fe00640 1 --2- 172.21.15.67:0/203657890 >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 0x7fda180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.765+0000 7fda1d000640 1 --2- 172.21.15.67:0/203657890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180cfcd0 0x7fda180d0180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda180b71e0 tx=0x7fda1002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab2143317d404ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda17400640 1 -- 172.21.15.67:0/203657890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10039070 con 0x7fda180cfcd0 2024-04-17T07:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda17400640 1 -- 172.21.15.67:0/203657890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1002cb60 con 0x7fda180cfcd0 2024-04-17T07:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda17400640 1 -- 172.21.15.67:0/203657890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10034440 con 0x7fda180cfcd0 2024-04-17T07:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 -- 172.21.15.67:0/203657890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180cfcd0 msgr2=0x7fda180d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 --2- 172.21.15.67:0/203657890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180cfcd0 0x7fda180d0180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda180b71e0 tx=0x7fda1002bdb0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 -- 172.21.15.67:0/203657890 shutdown_connections 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 --2- 172.21.15.67:0/203657890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180cfcd0 0x7fda180d0180 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 --2- 172.21.15.67:0/203657890 >> v2:172.21.15.187:3300/0 conn(0x7fda180b3b50 0x7fda180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.766+0000 7fda1da00640 1 -- 172.21.15.67:0/203657890 >> 172.21.15.67:0/203657890 conn(0x7fda1808bff0 msgr2=0x7fda180d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.767+0000 7fda1da00640 1 -- 172.21.15.67:0/203657890 shutdown_connections 2024-04-17T07:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.767+0000 7fda1da00640 1 -- 172.21.15.67:0/203657890 wait complete. 2024-04-17T07:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.768+0000 7fda1da00640 1 Processor -- start 2024-04-17T07:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.769+0000 7fda1da00640 1 -- start start 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda180cfcd0 0x7fda18074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda1815ec20 0x7fda1814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda1806d2d0 con 0x7fda180b3b50 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fda1806d150 con 0x7fda1815ec20 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fda180d10d0 con 0x7fda180cfcd0 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59440/0 (socket says 172.21.15.67:59440) 2024-04-17T07:29:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 -- 172.21.15.67:0/26921125 learned_addr learned my addr 172.21.15.67:0/26921125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda1d000640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda1815ec20 0x7fda1814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda180cfcd0 msgr2=0x7fda18074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda180cfcd0 0x7fda18074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda1815ec20 msgr2=0x7fda1814f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda1815ec20 0x7fda1814f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.770+0000 7fda17e00640 1 -- 172.21.15.67:0/26921125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda18161550 con 0x7fda180b3b50 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda17e00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c3f0 tx=0x7fda0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08017070 con 0x7fda180b3b50 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda08005ce0 con 0x7fda180b3b50 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda18161830 con 0x7fda180b3b50 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080072f0 con 0x7fda180b3b50 2024-04-17T07:29:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.771+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda18161df0 con 0x7fda180b3b50 2024-04-17T07:29:05.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.772+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda18068a10 con 0x7fda180b3b50 2024-04-17T07:29:05.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.773+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7fda0801a070 con 0x7fda180b3b50 2024-04-17T07:29:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.774+0000 7fda14c00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 0x7fd9f406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.774+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7fda0808b250 con 0x7fda180b3b50 2024-04-17T07:29:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.774+0000 7fda0fe00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 msgr2=0x7fd9f406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.774+0000 7fda0fe00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 0x7fd9f406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:05.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.778+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fda08059390 con 0x7fda180b3b50 2024-04-17T07:29:05.787 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.787+0000 7f3bb27be040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:05.834 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:05.834+0000 7fa75422b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:05.839 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.839+0000 7f3bb27be040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:05.892 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.892+0000 7f3bb27be040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:05.892 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:05.892+0000 7fa75422b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:05.956 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:05.956+0000 7fa75422b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:05.964 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:05.964+0000 7f3bb27be040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.975+0000 7fda0fe00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 msgr2=0x7fd9f406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.975+0000 7fda0fe00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 0x7fd9f406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:05.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.993+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda180d55f0 con 0x7fda180b3b50 2024-04-17T07:29:05.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.994+0000 7fda14c00640 1 -- 172.21.15.67:0/26921125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7fda08058c90 con 0x7fda180b3b50 2024-04-17T07:29:05.994 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.996+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 msgr2=0x7fd9f406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.996+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 0x7fd9f406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 msgr2=0x7fda18160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c3f0 tx=0x7fda0800c8b0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 shutdown_connections 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fd9f406ca10 0x7fd9f406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda1815ec20 0x7fda1814f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda180cfcd0 0x7fda18074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 --2- 172.21.15.67:0/26921125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda180b3b50 0x7fda18160d00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 >> 172.21.15.67:0/26921125 conn(0x7fda1808bff0 msgr2=0x7fda180d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 shutdown_connections 2024-04-17T07:29:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:05.997+0000 7fda1da00640 1 -- 172.21.15.67:0/26921125 wait complete. 2024-04-17T07:29:06.014 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.014+0000 7f3bb27be040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:06.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:29:06.023 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.023+0000 7fa75422b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:06.076 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.076+0000 7fa75422b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:06.142 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.142+0000 7f3bb27be040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.201+0000 7f29f2400640 1 Processor -- start 2024-04-17T07:29:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.202+0000 7f29f2400640 1 -- start start 2024-04-17T07:29:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.203+0000 7f29f2400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.203+0000 7f29f2400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec0d00b0 con 0x7f29ec15ecb0 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.203+0000 7f29f2400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29ec0d0390 con 0x7f29ec0d33e0 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.203+0000 7f29f2400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f29ec0d0c30 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59444/0 (socket says 172.21.15.67:59444) 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 -- 172.21.15.67:0/96740961 learned_addr learned my addr 172.21.15.67:0/96740961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 -- 172.21.15.67:0/96740961 >> v1:172.21.15.187:6789/0 conn(0x7f29ec0d33e0 legacy=0x7f29ec0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 -- 172.21.15.67:0/96740961 >> v1:172.21.15.67:6789/0 conn(0x7f29ec15ecb0 legacy=0x7f29ec0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.204+0000 7f29f1a00640 1 -- 172.21.15.67:0/96740961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec0d0670 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.205+0000 7f29f1a00640 1 --2- 172.21.15.67:0/96740961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0d0db0 tx=0x7f29e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197c58d07f64b990 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:06.206 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.205+0000 7f3bb27be040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29eb400640 1 -- 172.21.15.67:0/96740961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e402caf0 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29eb400640 1 -- 172.21.15.67:0/96740961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e402cc90 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29eb400640 1 -- 172.21.15.67:0/96740961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4034610 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29f2400640 1 -- 172.21.15.67:0/96740961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec0b4fd0 msgr2=0x7f29ec0cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29f2400640 1 --2- 172.21.15.67:0/96740961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0d0db0 tx=0x7f29e402bf50 comp rx=0 tx=0).stop 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29f2400640 1 -- 172.21.15.67:0/96740961 shutdown_connections 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29f2400640 1 --2- 172.21.15.67:0/96740961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec0cfab0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.206+0000 7f29f2400640 1 -- 172.21.15.67:0/96740961 >> 172.21.15.67:0/96740961 conn(0x7f29ec08bff0 msgr2=0x7f29ec0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.207+0000 7f29f2400640 1 -- 172.21.15.67:0/96740961 shutdown_connections 2024-04-17T07:29:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.207+0000 7f29f2400640 1 -- 172.21.15.67:0/96740961 wait complete. 2024-04-17T07:29:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.208+0000 7f29f2400640 1 Processor -- start 2024-04-17T07:29:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.208+0000 7f29f2400640 1 -- start start 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29ec0d33e0 0x7f29ec0b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec15f210 con 0x7f29ec15ecb0 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec15f090 con 0x7f29ec0b4fd0 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f2400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29ec1698f0 con 0x7f29ec0d33e0 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.209+0000 7f29f1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59452/0 (socket says 172.21.15.67:59452) 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 -- 172.21.15.67:0/3065291603 learned_addr learned my addr 172.21.15.67:0/3065291603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1000640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29ec0d33e0 msgr2=0x7f29ec0b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29ec0d33e0 0x7f29ec0b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 msgr2=0x7f29ec155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 -- 172.21.15.67:0/3065291603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec15b430 con 0x7f29ec15ecb0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1000640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29f1a00640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c560 tx=0x7f29e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.210+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4043020 con 0x7f29ec15ecb0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.211+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e403e450 con 0x7f29ec15ecb0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.211+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4009510 con 0x7f29ec15ecb0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.211+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec0b8f40 con 0x7f29ec15ecb0 2024-04-17T07:29:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.211+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec0b9500 con 0x7f29ec15ecb0 2024-04-17T07:29:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.212+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec068a10 con 0x7f29ec15ecb0 2024-04-17T07:29:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.213+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7f29e4046020 con 0x7f29ec15ecb0 2024-04-17T07:29:06.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.214+0000 7f29e8c00640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 0x7f29cc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.214+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7f29e40ae850 con 0x7f29ec15ecb0 2024-04-17T07:29:06.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.214+0000 7f29f1000640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 msgr2=0x7f29cc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:06.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.214+0000 7f29f1000640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 0x7f29cc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:06.217 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.217+0000 7fa75422b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.218+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f29e404d030 con 0x7f29ec15ecb0 2024-04-17T07:29:06.256 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.256+0000 7f3bb27be040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:06.285 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.285+0000 7fa75422b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:06.313 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.313+0000 7f3bb27be040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:06.339 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.338+0000 7fa75422b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:06.364 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.364+0000 7f3bb27be040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:06.399 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.399+0000 7fa75422b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:06.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.415+0000 7f29f1000640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 msgr2=0x7f29cc06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:06.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.415+0000 7f29f1000640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 0x7f29cc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:06.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.419+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f29ec15b600 con 0x7f29ec15ecb0 2024-04-17T07:29:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.420+0000 7f29e8c00640 1 -- 172.21.15.67:0/3065291603 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) ==== 104+0+11 (secure 0 0 0) 0x7f29e407c8f0 con 0x7f29ec15ecb0 2024-04-17T07:29:06.420 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.422+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 msgr2=0x7f29cc06ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.422+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 0x7f29cc06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.422+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 msgr2=0x7f29ec1567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.422+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c560 tx=0x7f29e4004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 shutdown_connections 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f29cc06ca40 0x7f29cc06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29ec15ecb0 0x7f29ec1567c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29ec0d33e0 0x7f29ec0b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 --2- 172.21.15.67:0/3065291603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29ec0b4fd0 0x7f29ec155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 >> 172.21.15.67:0/3065291603 conn(0x7f29ec08bff0 msgr2=0x7f29ec0d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 shutdown_connections 2024-04-17T07:29:06.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.423+0000 7f29f2400640 1 -- 172.21.15.67:0/3065291603 wait complete. 2024-04-17T07:29:06.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:29:06.455 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.455+0000 7fa75422b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.661+0000 7f44dce00640 1 Processor -- start 2024-04-17T07:29:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.662+0000 7f44dce00640 1 -- start start 2024-04-17T07:29:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.663+0000 7f44dce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.663+0000 7f44dce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d80cfcd0 0x7f44d80d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.663+0000 7f44dce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d80d4f70 con 0x7f44d80b3b50 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.663+0000 7f44dce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f44d80d5250 con 0x7f44d80cfcd0 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.663+0000 7f44dce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f44d80cf4d0 con 0x7f44d815ec20 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d80cfcd0 0x7f44d80d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d80cfcd0 0x7f44d80d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59456/0 (socket says 172.21.15.67:59456) 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 -- 172.21.15.67:0/1335873577 learned_addr learned my addr 172.21.15.67:0/1335873577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7400640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 -- 172.21.15.67:0/1335873577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 msgr2=0x7f44d80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 -- 172.21.15.67:0/1335873577 >> v1:172.21.15.187:6789/0 conn(0x7f44d80b3b50 legacy=0x7f44d806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.664+0000 7f44d7e00640 1 -- 172.21.15.67:0/1335873577 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d80cf7b0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.666+0000 7f44d7e00640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80d0180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009e50 tx=0x7f44d002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4720080ff363d42e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.666+0000 7f44d6000640 1 -- 172.21.15.67:0/1335873577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d002c950 con 0x7f44d80cfcd0 2024-04-17T07:29:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.666+0000 7f44d6000640 1 -- 172.21.15.67:0/1335873577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d002caf0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.666+0000 7f44d6000640 1 -- 172.21.15.67:0/1335873577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d00343f0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 -- 172.21.15.67:0/1335873577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 msgr2=0x7f44d80d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80d0180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009e50 tx=0x7f44d002bdb0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 -- 172.21.15.67:0/1335873577 shutdown_connections 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80d0180 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 --2- 172.21.15.67:0/1335873577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 -- 172.21.15.67:0/1335873577 >> 172.21.15.67:0/1335873577 conn(0x7f44d808bff0 msgr2=0x7f44d80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 -- 172.21.15.67:0/1335873577 shutdown_connections 2024-04-17T07:29:06.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.667+0000 7f44dce00640 1 -- 172.21.15.67:0/1335873577 wait complete. 2024-04-17T07:29:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.669+0000 7f44dce00640 1 Processor -- start 2024-04-17T07:29:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.669+0000 7f44dce00640 1 -- start start 2024-04-17T07:29:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44d80b3b50 0x7f44d815ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806d2d0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806d150 con 0x7f44d80b3b50 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44dce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f44d80d10d0 con 0x7f44d815ec20 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59464/0 (socket says 172.21.15.67:59464) 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44d6a00640 1 -- 172.21.15.67:0/2779059798 learned_addr learned my addr 172.21.15.67:0/2779059798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.670+0000 7f44d7400640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44d80b3b50 0x7f44d815ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 msgr2=0x7f44d80a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44d80b3b50 msgr2=0x7f44d815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44d80b3b50 0x7f44d815ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 -- 172.21.15.67:0/2779059798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d80cca80 con 0x7f44d80cfcd0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44d6a00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004640 tx=0x7f44c000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0011790 con 0x7f44d80cfcd0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c0017070 con 0x7f44d80cfcd0 2024-04-17T07:29:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0011bb0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d80ccd60 con 0x7f44d80cfcd0 2024-04-17T07:29:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.671+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d815d3f0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.672+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d8068a10 con 0x7f44d80cfcd0 2024-04-17T07:29:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.673+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7f44c0011d50 con 0x7f44d80cfcd0 2024-04-17T07:29:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.674+0000 7f44cf400640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 0x7f44b406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.674+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7f44c008ba80 con 0x7f44d80cfcd0 2024-04-17T07:29:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.675+0000 7f44d7400640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 msgr2=0x7f44b406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.675+0000 7f44d7400640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 0x7f44b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:06.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.679+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f44c008de20 con 0x7f44d80cfcd0 2024-04-17T07:29:06.753 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.753+0000 7f3bb27be040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:06.873 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:06.873+0000 7fa75422b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.876+0000 7f44d7400640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 msgr2=0x7f44b406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.876+0000 7f44d7400640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 0x7f44b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.885+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f44d80d55f0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.886+0000 7f44cf400640 1 -- 172.21.15.67:0/2779059798 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) ==== 100+0+11 (secure 0 0 0) 0x7f44c0059bb0 con 0x7f44d80cfcd0 2024-04-17T07:29:06.886 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 msgr2=0x7f44b406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 0x7f44b406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 msgr2=0x7f44d80a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f44c0004640 tx=0x7f44c000c8e0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 shutdown_connections 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f44b406ca10 0x7f44b406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f44d815ec20 0x7f44d80a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d80cfcd0 0x7f44d80a18a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 --2- 172.21.15.67:0/2779059798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44d80b3b50 0x7f44d815ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.888+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 >> 172.21.15.67:0/2779059798 conn(0x7f44d808bff0 msgr2=0x7f44d80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.889+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 shutdown_connections 2024-04-17T07:29:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:06.889+0000 7f44dce00640 1 -- 172.21.15.67:0/2779059798 wait complete. 2024-04-17T07:29:06.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:29:06.918 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:06.918+0000 7f3bb27be040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:07.028 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.027+0000 7f3bb27be040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:07.047 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.047+0000 7fa75422b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:07.078 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.078+0000 7f3bb27be040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.114+0000 7f179d000640 1 Processor -- start 2024-04-17T07:29:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.114+0000 7f179d000640 1 -- start start 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f179d000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f179815eba0 0x7f17980ba400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f179d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17980d4150 0x7f17980d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f179d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17980d2e10 con 0x7f17980cf560 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f179d000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17980d45d0 con 0x7f17980d4150 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f179d000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f17980d30b0 con 0x7f179815eba0 2024-04-17T07:29:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f1796a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17980d4150 0x7f17980d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f1796a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17980d4150 0x7f17980d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59480/0 (socket says 172.21.15.67:59480) 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.116+0000 7f1796a00640 1 -- 172.21.15.67:0/4175384617 learned_addr learned my addr 172.21.15.67:0/4175384617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.117+0000 7f1796a00640 1 -- 172.21.15.67:0/4175384617 >> v2:172.21.15.187:3300/0 conn(0x7f179815eba0 msgr2=0x7f17980ba400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.117+0000 7f1796a00640 1 --2- 172.21.15.67:0/4175384617 >> v2:172.21.15.187:3300/0 conn(0x7f179815eba0 0x7f17980ba400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.117+0000 7f1796a00640 1 -- 172.21.15.67:0/4175384617 >> v1:172.21.15.67:6789/0 conn(0x7f17980cf560 legacy=0x7f17980b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.117+0000 7f1796a00640 1 -- 172.21.15.67:0/4175384617 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17980d3390 con 0x7f17980d4150 2024-04-17T07:29:07.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.118+0000 7f1796a00640 1 --2- 172.21.15.67:0/4175384617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17980d4150 0x7f17980d28d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1798068d50 tx=0x7f178402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca835219bcfc2816 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:07.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.118+0000 7f1796000640 1 -- 172.21.15.67:0/4175384617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784035020 con 0x7f17980d4150 2024-04-17T07:29:07.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.118+0000 7f1796000640 1 -- 172.21.15.67:0/4175384617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1784031870 con 0x7f17980d4150 2024-04-17T07:29:07.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.118+0000 7f1796000640 1 -- 172.21.15.67:0/4175384617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784031c40 con 0x7f17980d4150 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 -- 172.21.15.67:0/4175384617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17980d4150 msgr2=0x7f17980d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 --2- 172.21.15.67:0/4175384617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17980d4150 0x7f17980d28d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1798068d50 tx=0x7f178402b910 comp rx=0 tx=0).stop 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 -- 172.21.15.67:0/4175384617 shutdown_connections 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 --2- 172.21.15.67:0/4175384617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17980d4150 0x7f17980d28d0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 --2- 172.21.15.67:0/4175384617 >> v2:172.21.15.187:3300/0 conn(0x7f179815eba0 0x7f17980ba400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 -- 172.21.15.67:0/4175384617 >> 172.21.15.67:0/4175384617 conn(0x7f179806cac0 msgr2=0x7f179806cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 -- 172.21.15.67:0/4175384617 shutdown_connections 2024-04-17T07:29:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.119+0000 7f179d000640 1 -- 172.21.15.67:0/4175384617 wait complete. 2024-04-17T07:29:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.120+0000 7f179d000640 1 Processor -- start 2024-04-17T07:29:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 -- start start 2024-04-17T07:29:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17980cf560 0x7f1798151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 0x7f179815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17980d3c30 con 0x7f179815eba0 2024-04-17T07:29:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f17980d3950 con 0x7f17980d4150 2024-04-17T07:29:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.121+0000 7f179d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17980cf9f0 con 0x7f17980cf560 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59494/0 (socket says 172.21.15.67:59494) 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 -- 172.21.15.67:0/3756392456 learned_addr learned my addr 172.21.15.67:0/3756392456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797400640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17980cf560 0x7f1798151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1796a00640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 0x7f179815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17980cf560 msgr2=0x7f1798151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17980cf560 0x7f1798151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 msgr2=0x7f179815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 0x7f179815ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 -- 172.21.15.67:0/3756392456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1798160740 con 0x7f179815eba0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1796a00640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 0x7f179815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.122+0000 7f1797e00640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f179000c7a0 tx=0x7f179000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.123+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790003b70 con 0x7f179815eba0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.123+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1790003d30 con 0x7f179815eba0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.123+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179000aa90 con 0x7f179815eba0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.123+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1798160a20 con 0x7f179815eba0 2024-04-17T07:29:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.123+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1798160fe0 con 0x7f179815eba0 2024-04-17T07:29:07.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.124+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17980755d0 con 0x7f179815eba0 2024-04-17T07:29:07.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.125+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7f179000ac30 con 0x7f179815eba0 2024-04-17T07:29:07.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.126+0000 7f178f400640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 0x7f177806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.126+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7f179008be80 con 0x7f179815eba0 2024-04-17T07:29:07.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.126+0000 7f1797400640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 msgr2=0x7f177806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:07.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.126+0000 7f1797400640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 0x7f177806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.130+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1790059fb0 con 0x7f179815eba0 2024-04-17T07:29:07.167 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.166+0000 7fa75422b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:07.204 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.204+0000 7f3bb27be040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:07.222 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.221+0000 7fa75422b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:07.311 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.311+0000 7f3bb27be040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:07.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.327+0000 7f1797400640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 msgr2=0x7f177806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:07.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.327+0000 7f1797400640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 0x7f177806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:07.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.344+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f17980c8240 con 0x7f179815eba0 2024-04-17T07:29:07.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.344+0000 7f178f400640 1 -- 172.21.15.67:0/3756392456 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v71) ==== 80+0+478 (secure 0 0 0) 0x7f17900598b0 con 0x7f179815eba0 2024-04-17T07:29:07.344 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/1235852913 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/1235852913 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1671674456 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/4131277908 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3437520709 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3840279087 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3804745495 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1222851080 2024-04-18T07:27:30.199060+0000 2024-04-17T07:29:07.345 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 msgr2=0x7f177806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 0x7f177806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 msgr2=0x7f17980ab480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f179000c7a0 tx=0x7f179000cc60 comp rx=0 tx=0).stop 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 shutdown_connections 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f177806ca10 0x7f177806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f179815eba0 0x7f17980ab480 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17980d4150 0x7f179815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 --2- 172.21.15.67:0/3756392456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17980cf560 0x7f1798151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 >> 172.21.15.67:0/3756392456 conn(0x7f179806cac0 msgr2=0x7f17980d23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:07.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 shutdown_connections 2024-04-17T07:29:07.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.347+0000 7f179d000640 1 -- 172.21.15.67:0/3756392456 wait complete. 2024-04-17T07:29:07.356 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.356+0000 7fa75422b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:07.364 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:29:07.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:07.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:07.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:29:07.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:07.467 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:29:07.477 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.478+0000 7fa75422b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:07.487 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.487+0000 7f3bb27be040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:07.512 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:07.512 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:07.512 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:07.538 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.538+0000 7f3bb27be040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:07.564 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:07.565 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:07.565 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:07.565 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:07.615 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:07.616 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:07.616 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:07.616 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:07.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:07.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:07.674 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.668+0000 7fa75422b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:07.683 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:07.683+0000 7f3bb27be040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:07.721 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.721+0000 7fa75422b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:07.870 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:07.870+0000 7fa75422b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.880+0000 7f101dc00640 1 Processor -- start 2024-04-17T07:29:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.880+0000 7f101dc00640 1 -- start start 2024-04-17T07:29:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 0x7f10180cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 0x7f10180d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10180d2ab0 0x7f10180b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10180d3c30 con 0x7f10180d2ab0 2024-04-17T07:29:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10180cff60 con 0x7f10180cf640 2024-04-17T07:29:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f101dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10180d3020 con 0x7f101815ed10 2024-04-17T07:29:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f1017400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10180d2ab0 0x7f10180b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f1017400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10180d2ab0 0x7f10180b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59508/0 (socket says 172.21.15.67:59508) 2024-04-17T07:29:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.882+0000 7f1017400640 1 -- 172.21.15.67:0/3297655493 learned_addr learned my addr 172.21.15.67:0/3297655493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1016a00640 1 --2- 172.21.15.67:0/3297655493 >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 0x7f10180cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f100fe00640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 0x7f10180d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1017400640 1 -- 172.21.15.67:0/3297655493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 msgr2=0x7f10180d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1017400640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 0x7f10180d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1017400640 1 -- 172.21.15.67:0/3297655493 >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 msgr2=0x7f10180cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1017400640 1 --2- 172.21.15.67:0/3297655493 >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 0x7f10180cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1017400640 1 -- 172.21.15.67:0/3297655493 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10180d3300 con 0x7f10180d2ab0 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f1016a00640 1 --2- 172.21.15.67:0/3297655493 >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 0x7f10180cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.883+0000 7f100fe00640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 0x7f10180d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.884+0000 7f1017400640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180b7b10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f101808cc00 tx=0x7f101002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef9254d321f4894 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.884+0000 7f1016000640 1 -- 172.21.15.67:0/3297655493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101002cb00 con 0x7f10180d2ab0 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.884+0000 7f1016000640 1 -- 172.21.15.67:0/3297655493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f101002cca0 con 0x7f10180d2ab0 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.884+0000 7f1016000640 1 -- 172.21.15.67:0/3297655493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010034610 con 0x7f10180d2ab0 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 -- 172.21.15.67:0/3297655493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 msgr2=0x7f10180b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180b7b10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f101808cc00 tx=0x7f101002bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 -- 172.21.15.67:0/3297655493 shutdown_connections 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180b7b10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 --2- 172.21.15.67:0/3297655493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f101815ed10 0x7f10180d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 --2- 172.21.15.67:0/3297655493 >> v2:172.21.15.187:3300/0 conn(0x7f10180cf640 0x7f10180cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 -- 172.21.15.67:0/3297655493 >> 172.21.15.67:0/3297655493 conn(0x7f101806cbd0 msgr2=0x7f101806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 -- 172.21.15.67:0/3297655493 shutdown_connections 2024-04-17T07:29:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.885+0000 7f101dc00640 1 -- 172.21.15.67:0/3297655493 wait complete. 2024-04-17T07:29:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.887+0000 7f101dc00640 1 Processor -- start 2024-04-17T07:29:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.887+0000 7f101dc00640 1 -- start start 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf640 0x7f10181f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f101815ed10 0x7f10180c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10180d25b0 con 0x7f10180d2ab0 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10180d38c0 con 0x7f101815ed10 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.888+0000 7f101dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10180d4600 con 0x7f10180cf640 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59516/0 (socket says 172.21.15.67:59516) 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 -- 172.21.15.67:0/2023802137 learned_addr learned my addr 172.21.15.67:0/2023802137 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f1017400640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f101815ed10 0x7f10180c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f1016a00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf640 0x7f10181f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf640 msgr2=0x7f10181f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf640 0x7f10181f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f101815ed10 msgr2=0x7f10180c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f101815ed10 0x7f10180c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.889+0000 7f100fe00640 1 -- 172.21.15.67:0/2023802137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10181f7ef0 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f100fe00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f100000c7c0 tx=0x7f100000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000015070 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1000007820 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000007bf0 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10181f81c0 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.890+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10181f8780 con 0x7f10180d2ab0 2024-04-17T07:29:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.891+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1018068b90 con 0x7f10180d2ab0 2024-04-17T07:29:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.892+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7f1000018070 con 0x7f10180d2ab0 2024-04-17T07:29:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.893+0000 7f100ea00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 0x7f0ff406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.893+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7f100008bed0 con 0x7f10180d2ab0 2024-04-17T07:29:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.893+0000 7f1016a00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 msgr2=0x7f0ff406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.893+0000 7f1016a00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 0x7f0ff406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:07.898+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1000059e60 con 0x7f10180d2ab0 2024-04-17T07:29:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.094+0000 7f1016a00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 msgr2=0x7f0ff406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.094+0000 7f1016a00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 0x7f0ff406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.138+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10180d0910 con 0x7f10180d2ab0 2024-04-17T07:29:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.139+0000 7f100ea00640 1 -- 172.21.15.67:0/2023802137 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7f1000059760 con 0x7f10180d2ab0 2024-04-17T07:29:08.139 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:08.140 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:29:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:29:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 msgr2=0x7f0ff406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 0x7f0ff406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 msgr2=0x7f10180ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f100000c7c0 tx=0x7f100000cc80 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 shutdown_connections 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f0ff406ca10 0x7f0ff406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f101815ed10 0x7f10180c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10180d2ab0 0x7f10180ac010 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 --2- 172.21.15.67:0/2023802137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf640 0x7f10181f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 >> 172.21.15.67:0/2023802137 conn(0x7f101806cbd0 msgr2=0x7f10180c84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 shutdown_connections 2024-04-17T07:29:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.142+0000 7f101dc00640 1 -- 172.21.15.67:0/2023802137 wait complete. 2024-04-17T07:29:08.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.367+0000 7fb4b2a00640 1 Processor -- start 2024-04-17T07:29:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.367+0000 7fb4b2a00640 1 -- start start 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4b2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb4ac0d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4b2a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4ac0d2ab0 0x7fb4ac0b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4b2a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4ac0d3020 con 0x7fb4ac0cf640 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4b2a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb4ac0d3c30 con 0x7fb4ac0d2ab0 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4b2a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4ac0d3300 con 0x7fb4ac15ed10 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4abe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4ac0cf640 0x7fb4ac0cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51898/0 (socket says 172.21.15.67:51898) 2024-04-17T07:29:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4abe00640 1 -- 172.21.15.67:0/1173560777 learned_addr learned my addr 172.21.15.67:0/1173560777 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.369+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb4ac0d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1173560777 >> v2:172.21.15.187:3300/0 conn(0x7fb4ac0d2ab0 0x7fb4ac0b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468639948 0 0) 0x7fb4ac0d3020 con 0x7fb4ac0cf640 2024-04-17T07:29:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb490003520 con 0x7fb4ac0cf640 2024-04-17T07:29:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2643935934 0 0) 0x7fb490003520 con 0x7fb4ac0cf640 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 msgr2=0x7fb4ac0d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb4ac0d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 >> v2:172.21.15.187:3300/0 conn(0x7fb4ac0d2ab0 msgr2=0x7fb4ac0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 --2- 172.21.15.67:0/1173560777 >> v2:172.21.15.187:3300/0 conn(0x7fb4ac0d2ab0 0x7fb4ac0b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ac0d35e0 con 0x7fb4ac0cf640 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb49c001670 con 0x7fb4ac0cf640 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 >> v1:172.21.15.67:6789/0 conn(0x7fb4ac0cf640 legacy=0x7fb4ac0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4900039b0 0x7fb490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb490004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac0d3c30 con 0x7fb490004d40 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac0d3300 con 0x7fb4900039b0 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.370+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb49000b9c0 con 0x7fb4ac15ed10 2024-04-17T07:29:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4abe00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4900039b0 0x7fb490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb490004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 -- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 msgr2=0x7fb490004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac15ed10 0x7fb490004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 -- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4900039b0 msgr2=0x7fb490003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4900039b0 0x7fb490003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.371+0000 7fb4b0a00640 1 -- 172.21.15.67:0/1173560777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb490003520 con 0x7fb490004d40 2024-04-17T07:29:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.372+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4900051c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009fc0 tx=0x7fb490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f124b77b7e94ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.372+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4039070 con 0x7fb490004d40 2024-04-17T07:29:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.372+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a402cc60 con 0x7fb490004d40 2024-04-17T07:29:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.372+0000 7fb4aaa00640 1 -- 172.21.15.67:0/1173560777 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4009c10 con 0x7fb490004d40 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1173560777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 msgr2=0x7fb4900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1173560777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4900051c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009fc0 tx=0x7fb490005830 comp rx=0 tx=0).stop 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b0a00640 1 -- 172.21.15.67:0/1173560777 reap_dead start 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1173560777 shutdown_connections 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1173560777 >> 172.21.15.67:0/1173560777 conn(0x7fb4ac06cbd0 msgr2=0x7fb4ac06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1173560777 shutdown_connections 2024-04-17T07:29:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.373+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1173560777 wait complete. 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.375+0000 7fb4b2a00640 1 Processor -- start 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.375+0000 7fb4b2a00640 1 -- start start 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 0x7fb4ac1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac1f4b40 0x7fb4ac160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac0ba300 con 0x7fb490004d40 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac0d25b0 con 0x7fb4ac0cba40 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4b2a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4ac0ba480 con 0x7fb4ac1f4b40 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59546/0 (socket says 172.21.15.67:59546) 2024-04-17T07:29:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.376+0000 7fb4abe00640 1 -- 172.21.15.67:0/1474158858 learned_addr learned my addr 172.21.15.67:0/1474158858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 0x7fb4ac1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4b0a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac1f4b40 0x7fb4ac160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac1f4b40 msgr2=0x7fb4ac160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac1f4b40 0x7fb4ac160190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 msgr2=0x7fb4ac1f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 0x7fb4ac1f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 -- 172.21.15.67:0/1474158858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ac0ce8d0 con 0x7fb490004d40 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 0x7fb4ac1f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4abe00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e820 tx=0x7fb49c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00c870 con 0x7fb490004d40 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb49c018070 con 0x7fb490004d40 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00cc50 con 0x7fb490004d40 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.377+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ac0cebb0 con 0x7fb490004d40 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.378+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ac0b5810 con 0x7fb490004d40 2024-04-17T07:29:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.379+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ac068b90 con 0x7fb490004d40 2024-04-17T07:29:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.380+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7fb49c01d460 con 0x7fb490004d40 2024-04-17T07:29:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.381+0000 7fb4a3e00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 0x7fb48c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.381+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7fb49c08be80 con 0x7fb490004d40 2024-04-17T07:29:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.381+0000 7fb4ab400640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 msgr2=0x7fb48c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.381+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 0x7fb48c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:08.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.386+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb49c059fc0 con 0x7fb490004d40 2024-04-17T07:29:08.498 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:08.497+0000 7f3bb27be040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:08.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.582+0000 7fb4ab400640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 msgr2=0x7fb48c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:08.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.582+0000 7fb4ab400640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 0x7fb48c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.589+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4ac15e3c0 con 0x7fb490004d40 2024-04-17T07:29:08.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.591+0000 7fb4a3e00640 1 -- 172.21.15.67:0/1474158858 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+17295 (secure 0 0 0) 0x7fb49c0598c0 con 0x7fb490004d40 2024-04-17T07:29:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:08.591 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":71,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:05.704397+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1671674456":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/4131277908":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3437520709":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3840279087":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3804745495":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1222851080":"2024-04-18T07:27:30.199060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 msgr2=0x7fb48c06ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 0x7fb48c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 msgr2=0x7fb4ac15a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e820 tx=0x7fb49c00ece0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 shutdown_connections 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7fb48c06ca40 0x7fb48c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ac1f4b40 0x7fb4ac160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ac0cba40 0x7fb4ac1f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 --2- 172.21.15.67:0/1474158858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb490004d40 0x7fb4ac15a5d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 >> 172.21.15.67:0/1474158858 conn(0x7fb4ac06cbd0 msgr2=0x7fb4ac0b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 shutdown_connections 2024-04-17T07:29:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.594+0000 7fb4b2a00640 1 -- 172.21.15.67:0/1474158858 wait complete. 2024-04-17T07:29:08.613 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:27:12.167534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:27:13.288606+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:08.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:08.751 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:08.751+0000 7fa75422b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:08.802 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:08.802+0000 7f3bb27be040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:08.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.830+0000 7f0034a00640 1 Processor -- start 2024-04-17T07:29:08.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.830+0000 7f0034a00640 1 -- start start 2024-04-17T07:29:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f0034a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00300b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f0034a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00300ce3a0 0x7f00301627c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f0034a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003015f520 con 0x7f00300d0580 2024-04-17T07:29:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f0034a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00300cfc90 con 0x7f00300ce3a0 2024-04-17T07:29:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f0034a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f003015fe50 con 0x7f00300d01a0 2024-04-17T07:29:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f002e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00300d0580 0x7f0030162d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51912/0 (socket says 172.21.15.67:51912) 2024-04-17T07:29:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f002e800640 1 -- 172.21.15.67:0/1532151643 learned_addr learned my addr 172.21.15.67:0/1532151643 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f002de00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00300b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.832+0000 7f002d400640 1 --2- 172.21.15.67:0/1532151643 >> v2:172.21.15.187:3300/0 conn(0x7f00300ce3a0 0x7f00301627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528985686 0 0) 0x7f003015f520 con 0x7f00300d0580 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0014003520 con 0x7f00300d0580 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 984716691 0 0) 0x7f0014003520 con 0x7f00300d0580 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 msgr2=0x7f00300b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00300b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 >> v2:172.21.15.187:3300/0 conn(0x7f00300ce3a0 msgr2=0x7f00301627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 --2- 172.21.15.67:0/1532151643 >> v2:172.21.15.187:3300/0 conn(0x7f00300ce3a0 0x7f00301627c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003015f800 con 0x7f00300d0580 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0028001670 con 0x7f00300d0580 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 >> v1:172.21.15.67:6789/0 conn(0x7f00300d0580 legacy=0x7f0030162d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014003f30 0x7f0014004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014004d40 0x7f00140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f003015fe50 con 0x7f0014004d40 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f001400b9c0 con 0x7f0014003f30 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.833+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f001400bb90 con 0x7f00300d01a0 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014004d40 0x7f00140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002de00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002d400640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014003f30 0x7f0014004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 -- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 msgr2=0x7f00140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00300d01a0 0x7f00140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 -- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014003f30 msgr2=0x7f0014004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014003f30 0x7f0014004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.834+0000 7f002e800640 1 -- 172.21.15.67:0/1532151643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0014003520 con 0x7f0014004d40 2024-04-17T07:29:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.835+0000 7f002e800640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014004d40 0x7f00140051c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f002800c2c0 tx=0x7f0014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac314070fa6cfe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028007d50 con 0x7f0014004d40 2024-04-17T07:29:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0028030070 con 0x7f0014004d40 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f002ca00640 1 -- 172.21.15.67:0/1532151643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002802cda0 con 0x7f0014004d40 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f0034a00640 1 -- 172.21.15.67:0/1532151643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014004d40 msgr2=0x7f00140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f0034a00640 1 --2- 172.21.15.67:0/1532151643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0014004d40 0x7f00140051c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f002800c2c0 tx=0x7f0014005830 comp rx=0 tx=0).stop 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f002e800640 1 -- 172.21.15.67:0/1532151643 reap_dead start 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f0034a00640 1 -- 172.21.15.67:0/1532151643 shutdown_connections 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f0034a00640 1 -- 172.21.15.67:0/1532151643 >> 172.21.15.67:0/1532151643 conn(0x7f00300cd880 msgr2=0x7f00300cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.836+0000 7f0034a00640 1 -- 172.21.15.67:0/1532151643 shutdown_connections 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.837+0000 7f0034a00640 1 -- 172.21.15.67:0/1532151643 wait complete. 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.838+0000 7f0034a00640 1 Processor -- start 2024-04-17T07:29:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.838+0000 7f0034a00640 1 -- start start 2024-04-17T07:29:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014004d40 0x7f003015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00301f8740 0x7f0030152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00300cd5f0 con 0x7f00301f8300 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f003015cc30 con 0x7f0014004d40 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.839+0000 7f0034a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f003015f1f0 con 0x7f00301f8740 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59566/0 (socket says 172.21.15.67:59566) 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 -- 172.21.15.67:0/2939271043 learned_addr learned my addr 172.21.15.67:0/2939271043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002e800640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00301f8740 0x7f0030152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002de00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014004d40 0x7f003015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 -- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00301f8740 msgr2=0x7f0030152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00301f8740 0x7f0030152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 -- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014004d40 msgr2=0x7f003015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014004d40 0x7f003015d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.840+0000 7f002d400640 1 -- 172.21.15.67:0/2939271043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00300aea70 con 0x7f00301f8300 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f002d400640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0020007f00 tx=0x7f00200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002001a020 con 0x7f00301f8300 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020016470 con 0x7f00301f8300 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00300aed50 con 0x7f00301f8300 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020016840 con 0x7f00301f8300 2024-04-17T07:29:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.841+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00300af310 con 0x7f00301f8300 2024-04-17T07:29:08.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.842+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0030068b90 con 0x7f00301f8300 2024-04-17T07:29:08.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.843+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) ==== 91154+0+0 (secure 0 0 0) 0x7f0020005d10 con 0x7f00301f8300 2024-04-17T07:29:08.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.844+0000 7f0026000640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 0x7f000c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:08.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.844+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6183+0+0 (secure 0 0 0) 0x7f002008cfe0 con 0x7f00301f8300 2024-04-17T07:29:08.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.844+0000 7f002de00640 1 -- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 msgr2=0x7f000c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2009310366 2024-04-17T07:29:08.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.845+0000 7f002de00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 0x7f000c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:08.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:08.854+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0020024030 con 0x7f00301f8300 2024-04-17T07:29:08.862 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:08.862+0000 7f3bb27be040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:08.993 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:08.992+0000 7f3bb27be040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:09.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.017+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 27) ==== 90751+0+0 (secure 0 0 0) 0x7f002001d020 con 0x7f00301f8300 2024-04-17T07:29:09.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.018+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 msgr2=0x7f000c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:09.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.018+0000 7f0026000640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 0x7f000c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.059+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00300cb460 con 0x7f00301f8300 2024-04-17T07:29:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.059+0000 7f0026000640 1 -- 172.21.15.67:0/2939271043 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f0020057ee0 con 0x7f00301f8300 2024-04-17T07:29:09.059 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 msgr2=0x7f0030151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0020007f00 tx=0x7f00200115a0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 shutdown_connections 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:6832/2009310366,v1:172.21.15.187:6833/2009310366] conn(0x7f000c06ca10 0x7f000c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00301f8740 0x7f0030152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00301f8300 0x7f0030151fc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 --2- 172.21.15.67:0/2939271043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0014004d40 0x7f003015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 >> 172.21.15.67:0/2939271043 conn(0x7f00300cd880 msgr2=0x7f00301615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 shutdown_connections 2024-04-17T07:29:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.062+0000 7f0034a00640 1 -- 172.21.15.67:0/2939271043 wait complete. 2024-04-17T07:29:09.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:29:09.097 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.097 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.097 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2b200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.099 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:09.098+0000 7f3b2ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:09.109 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:09.109+0000 7fa75422b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:09.172 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:09.172+0000 7fa75422b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:09.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.268+0000 7f78b3e00640 1 Processor -- start 2024-04-17T07:29:09.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.268+0000 7f78b3e00640 1 -- start start 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.269+0000 7f78b3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78b415ecb0 0x7f78b40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.269+0000 7f78b3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b40d00b0 con 0x7f78b40d3210 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.269+0000 7f78b3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b40d0390 con 0x7f78b40cfab0 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.269+0000 7f78b3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f78b40d0c30 con 0x7f78b415ecb0 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78b415ecb0 0x7f78b40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78b40d3210 0x7f78b40cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51916/0 (socket says 172.21.15.67:51916) 2024-04-17T07:29:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2000640 1 -- 172.21.15.67:0/553258785 learned_addr learned my addr 172.21.15.67:0/553258785 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2a00640 1 -- 172.21.15.67:0/553258785 >> v1:172.21.15.187:6789/0 conn(0x7f78b40cfab0 legacy=0x7f78b40b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2a00640 1 -- 172.21.15.67:0/553258785 >> v1:172.21.15.67:6789/0 conn(0x7f78b40d3210 legacy=0x7f78b40cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.270+0000 7f78b2a00640 1 -- 172.21.15.67:0/553258785 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b40d0670 con 0x7f78b415ecb0 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.271+0000 7f78b2a00640 1 --2- 172.21.15.67:0/553258785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40d2cd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f78a40097a0 tx=0x7f78a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16b06670d3a588b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.271+0000 7f78b1600640 1 -- 172.21.15.67:0/553258785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40315a0 con 0x7f78b415ecb0 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b1600640 1 -- 172.21.15.67:0/553258785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a4037070 con 0x7f78b415ecb0 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b1600640 1 -- 172.21.15.67:0/553258785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4031960 con 0x7f78b415ecb0 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 -- 172.21.15.67:0/553258785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 msgr2=0x7f78b40d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 --2- 172.21.15.67:0/553258785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40d2cd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f78a40097a0 tx=0x7f78a402b840 comp rx=0 tx=0).stop 2024-04-17T07:29:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 -- 172.21.15.67:0/553258785 shutdown_connections 2024-04-17T07:29:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 --2- 172.21.15.67:0/553258785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40d2cd0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 -- 172.21.15.67:0/553258785 >> 172.21.15.67:0/553258785 conn(0x7f78b408bff0 msgr2=0x7f78b40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 -- 172.21.15.67:0/553258785 shutdown_connections 2024-04-17T07:29:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.272+0000 7f78b3e00640 1 -- 172.21.15.67:0/553258785 wait complete. 2024-04-17T07:29:09.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.274+0000 7f78b3e00640 1 Processor -- start 2024-04-17T07:29:09.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.274+0000 7f78b3e00640 1 -- start start 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b40cfab0 0x7f78b414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 0x7f78b4161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78b415f210 con 0x7f78b415ecb0 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f78b415f090 con 0x7f78b40cfab0 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f78b40b46d0 con 0x7f78b40d3210 2024-04-17T07:29:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59580/0 (socket says 172.21.15.67:59580) 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.275+0000 7f78b3400640 1 -- 172.21.15.67:0/3427465267 learned_addr learned my addr 172.21.15.67:0/3427465267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b2a00640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b40cfab0 0x7f78b414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b2000640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 0x7f78b4161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 -- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 msgr2=0x7f78b4161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 0x7f78b4161cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 -- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b40cfab0 msgr2=0x7f78b414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b40cfab0 0x7f78b414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 -- 172.21.15.67:0/3427465267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b414e030 con 0x7f78b415ecb0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b2000640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 0x7f78b4161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78b3400640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00c780 tx=0x7f78ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.276+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac004b60 con 0x7f78b415ecb0 2024-04-17T07:29:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.277+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ac004d00 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.277+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b40b7840 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.277+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00aa90 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.277+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b40b7e00 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.278+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b4068a10 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.279+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 27) ==== 90751+0+0 (secure 0 0 0) 0x7f78ac00ac30 con 0x7f78b415ecb0 2024-04-17T07:29:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.280+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7f78ac08bb80 con 0x7f78b415ecb0 2024-04-17T07:29:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.284+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f78ac020070 con 0x7f78b415ecb0 2024-04-17T07:29:09.314 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:09.314+0000 7fa75422b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.488+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78b40cad80 con 0x7f78b415ecb0 2024-04-17T07:29:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.488+0000 7f78aaa00640 1 -- 172.21.15.67:0/3427465267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) ==== 104+0+11 (secure 0 0 0) 0x7f78ac05a0f0 con 0x7f78b415ecb0 2024-04-17T07:29:09.489 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 msgr2=0x7f78b40b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00c780 tx=0x7f78ac00cc40 comp rx=0 tx=0).stop 2024-04-17T07:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 shutdown_connections 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b415ecb0 0x7f78b40b88e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b40d3210 0x7f78b4161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 --2- 172.21.15.67:0/3427465267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b40cfab0 0x7f78b414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 >> 172.21.15.67:0/3427465267 conn(0x7f78b408bff0 msgr2=0x7f78b40cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 shutdown_connections 2024-04-17T07:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.491+0000 7f78b3e00640 1 -- 172.21.15.67:0/3427465267 wait complete. 2024-04-17T07:29:09.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:29:09.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.777+0000 7ff042c00640 1 Processor -- start 2024-04-17T07:29:09.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.777+0000 7ff042c00640 1 -- start start 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff042c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c143200 0x7ff03c0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff042c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03c1607a0 con 0x7ff03c142e20 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff042c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03c160a60 con 0x7ff03c0cd580 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff042c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff03c160d40 con 0x7ff03c143200 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff03be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff03c142e20 0x7ff03c0cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51920/0 (socket says 172.21.15.67:51920) 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff03be00640 1 -- 172.21.15.67:0/2191287067 learned_addr learned my addr 172.21.15.67:0/2191287067 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.779+0000 7ff033400640 1 --2- 172.21.15.67:0/2191287067 >> v2:172.21.15.67:3300/0 conn(0x7ff03c143200 0x7ff03c0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.780+0000 7ff03b400640 1 -- 172.21.15.67:0/2191287067 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584067784 0 0) 0x7ff03c1607a0 con 0x7ff03c142e20 2024-04-17T07:29:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.780+0000 7ff03b400640 1 -- 172.21.15.67:0/2191287067 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff020003520 con 0x7ff03c142e20 2024-04-17T07:29:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.780+0000 7ff033400640 1 -- 172.21.15.67:0/2191287067 >> v1:172.21.15.187:6789/0 conn(0x7ff03c0cd580 legacy=0x7ff03c0b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.780+0000 7ff033400640 1 -- 172.21.15.67:0/2191287067 >> v1:172.21.15.67:6789/0 conn(0x7ff03c142e20 legacy=0x7ff03c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.780+0000 7ff033400640 1 -- 172.21.15.67:0/2191287067 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c161020 con 0x7ff03c143200 2024-04-17T07:29:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.781+0000 7ff033400640 1 --2- 172.21.15.67:0/2191287067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0cd040 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fb3c55487f8c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.781+0000 7ff03b400640 1 -- 172.21.15.67:0/2191287067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0240315a0 con 0x7ff03c143200 2024-04-17T07:29:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff03b400640 1 -- 172.21.15.67:0/2191287067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff024037070 con 0x7ff03c143200 2024-04-17T07:29:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff03b400640 1 -- 172.21.15.67:0/2191287067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024031940 con 0x7ff03c143200 2024-04-17T07:29:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 -- 172.21.15.67:0/2191287067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 msgr2=0x7ff03c0cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 --2- 172.21.15.67:0/2191287067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0cd040 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b840 comp rx=0 tx=0).stop 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 -- 172.21.15.67:0/2191287067 shutdown_connections 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 --2- 172.21.15.67:0/2191287067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0cd040 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 -- 172.21.15.67:0/2191287067 >> 172.21.15.67:0/2191287067 conn(0x7ff03c06b8c0 msgr2=0x7ff03c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 -- 172.21.15.67:0/2191287067 shutdown_connections 2024-04-17T07:29:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.782+0000 7ff042c00640 1 -- 172.21.15.67:0/2191287067 wait complete. 2024-04-17T07:29:09.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.784+0000 7ff042c00640 1 Processor -- start 2024-04-17T07:29:09.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.784+0000 7ff042c00640 1 -- start start 2024-04-17T07:29:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff03c0cd580 0x7ff03c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff03c142e20 0x7ff03c141e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c0ce730 con 0x7ff03c143200 2024-04-17T07:29:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c0cf330 con 0x7ff03c0cd580 2024-04-17T07:29:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff042c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff03c0b0ff0 con 0x7ff03c142e20 2024-04-17T07:29:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff040c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff040c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59586/0 (socket says 172.21.15.67:59586) 2024-04-17T07:29:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.785+0000 7ff040c00640 1 -- 172.21.15.67:0/1559698350 learned_addr learned my addr 172.21.15.67:0/1559698350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 -- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff03c142e20 msgr2=0x7ff03c141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff03c142e20 0x7ff03c141e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 -- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff03c0cd580 msgr2=0x7ff03c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff03c0cd580 0x7ff03c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 -- 172.21.15.67:0/1559698350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c15d8b0 con 0x7ff03c143200 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff040c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c440 tx=0x7ff03400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034011790 con 0x7ff03c143200 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff034017070 con 0x7ff03c143200 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034011b70 con 0x7ff03c143200 2024-04-17T07:29:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.786+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c15db90 con 0x7ff03c143200 2024-04-17T07:29:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.787+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c0c89a0 con 0x7ff03c143200 2024-04-17T07:29:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.788+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03c15e0b0 con 0x7ff03c143200 2024-04-17T07:29:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.789+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 27) ==== 90751+0+0 (secure 0 0 0) 0x7ff034011d10 con 0x7ff03c143200 2024-04-17T07:29:09.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.789+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7ff03408b700 con 0x7ff03c143200 2024-04-17T07:29:09.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.794+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff034059c70 con 0x7ff03c143200 2024-04-17T07:29:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.998+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff03c068b90 con 0x7ff03c143200 2024-04-17T07:29:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:09.999+0000 7ff038c00640 1 -- 172.21.15.67:0/1559698350 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) ==== 100+0+11 (secure 0 0 0) 0x7ff034059570 con 0x7ff03c143200 2024-04-17T07:29:09.999 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.001+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 msgr2=0x7ff03c0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.001+0000 7ff042c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff03400c440 tx=0x7ff03400c900 comp rx=0 tx=0).stop 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 shutdown_connections 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c143200 0x7ff03c0b44d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff03c142e20 0x7ff03c141e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 --2- 172.21.15.67:0/1559698350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff03c0cd580 0x7ff03c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 >> 172.21.15.67:0/1559698350 conn(0x7ff03c06b8c0 msgr2=0x7ff03c06e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 shutdown_connections 2024-04-17T07:29:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.002+0000 7ff042c00640 1 -- 172.21.15.67:0/1559698350 wait complete. 2024-04-17T07:29:10.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:10.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.222+0000 7f3481000640 1 Processor -- start 2024-04-17T07:29:10.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.222+0000 7f3481000640 1 -- start start 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f3481000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f347c06b8c0 0x7f347c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f3481000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347c0cd4a0 con 0x7f347c160280 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f3481000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347c1607f0 con 0x7f347c141ed0 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f3481000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f347c15ce30 con 0x7f347c06b8c0 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f347aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f347c160280 0x7f347c06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51926/0 (socket says 172.21.15.67:51926) 2024-04-17T07:29:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.224+0000 7f347aa00640 1 -- 172.21.15.67:0/1579365485 learned_addr learned my addr 172.21.15.67:0/1579365485 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347b400640 1 --2- 172.21.15.67:0/1579365485 >> v2:172.21.15.187:3300/0 conn(0x7f347c06b8c0 0x7f347c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894757889 0 0) 0x7f347c0cd4a0 con 0x7f347c160280 2024-04-17T07:29:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3460003520 con 0x7f347c160280 2024-04-17T07:29:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3771120882 0 0) 0x7f3460003520 con 0x7f347c160280 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 >> v2:172.21.15.187:3300/0 conn(0x7f347c06b8c0 msgr2=0x7f347c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 --2- 172.21.15.67:0/1579365485 >> v2:172.21.15.187:3300/0 conn(0x7f347c06b8c0 0x7f347c0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 >> v1:172.21.15.187:6789/0 conn(0x7f347c141ed0 legacy=0x7f347c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c160ad0 con 0x7f347c160280 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3468001670 con 0x7f347c160280 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 >> v1:172.21.15.67:6789/0 conn(0x7f347c160280 legacy=0x7f347c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460003a70 0x7f3460003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 0x7f3460004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3460005210 0x7f346000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f347c15ce30 con 0x7f3460004800 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f346000c140 con 0x7f3460003a70 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.225+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f346000c340 con 0x7f3460005210 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 0x7f3460004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347b400640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460003a70 0x7f3460003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347be00640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3460005210 0x7f346000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 -- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3460005210 msgr2=0x7f346000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3460005210 0x7f346000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 -- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460003a70 msgr2=0x7f3460003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460003a70 0x7f3460003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347aa00640 1 -- 172.21.15.67:0/1579365485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3460003520 con 0x7f3460004800 2024-04-17T07:29:10.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.226+0000 7f347be00640 1 -- 172.21.15.67:0/1579365485 reap_dead start 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.227+0000 7f347aa00640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 0x7f3460004c80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f346800c110 tx=0x7f3460005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64646685e57da24f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468007c40 con 0x7f3460004800 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3468030070 con 0x7f3460004800 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f347a000640 1 -- 172.21.15.67:0/1579365485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468005d70 con 0x7f3460004800 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 -- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 msgr2=0x7f3460004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 0x7f3460004c80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f346800c110 tx=0x7f3460005830 comp rx=0 tx=0).stop 2024-04-17T07:29:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 -- 172.21.15.67:0/1579365485 shutdown_connections 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 --2- 172.21.15.67:0/1579365485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3460004800 0x7f3460004c80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 -- 172.21.15.67:0/1579365485 >> 172.21.15.67:0/1579365485 conn(0x7f347c067d10 msgr2=0x7f347c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 -- 172.21.15.67:0/1579365485 shutdown_connections 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.228+0000 7f3481000640 1 -- 172.21.15.67:0/1579365485 wait complete. 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.230+0000 7f3481000640 1 Processor -- start 2024-04-17T07:29:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.230+0000 7f3481000640 1 -- start start 2024-04-17T07:29:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460004800 0x7f347c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c13e780 0x7f347c1458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f347c13f9b0 con 0x7f347c0b40a0 2024-04-17T07:29:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f347c161090 con 0x7f3460004800 2024-04-17T07:29:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f3481000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f347c13fb30 con 0x7f347c13e780 2024-04-17T07:29:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f347be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f347be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59610/0 (socket says 172.21.15.67:59610) 2024-04-17T07:29:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.231+0000 7f347be00640 1 -- 172.21.15.67:0/1815657437 learned_addr learned my addr 172.21.15.67:0/1815657437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347aa00640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c13e780 0x7f347c1458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347b400640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460004800 0x7f347c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 -- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c13e780 msgr2=0x7f347c1458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c13e780 0x7f347c1458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 -- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460004800 msgr2=0x7f347c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460004800 0x7f347c156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 -- 172.21.15.67:0/1815657437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c145e20 con 0x7f347c0b40a0 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f347be00640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f347c071d80 tx=0x7f347400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474018070 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3474003b70 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474005070 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.232+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c06ddf0 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.233+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c06e3b0 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.234+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f347c0b4a60 con 0x7f347c0b40a0 2024-04-17T07:29:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.235+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) ==== 90878+0+0 (secure 0 0 0) 0x7f347401b070 con 0x7f347c0b40a0 2024-04-17T07:29:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.236+0000 7f3473400640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 0x7f345c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.236+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7f347408b190 con 0x7f347c0b40a0 2024-04-17T07:29:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.237+0000 7f347b400640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 0x7f345c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.238+0000 7f347b400640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 0x7f345c06ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f347c0516a0 tx=0x7f346c008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.241+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34740595b0 con 0x7f347c0b40a0 2024-04-17T07:29:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.485+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f347c068a10 con 0x7f347c0b40a0 2024-04-17T07:29:10.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.486+0000 7f3473400640 1 -- 172.21.15.67:0/1815657437 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7f3474058eb0 con 0x7f347c0b40a0 2024-04-17T07:29:10.487 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:10.487 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:29:10.487 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:29:10.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 msgr2=0x7f345c06ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 0x7f345c06ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f347c0516a0 tx=0x7f346c008040 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 msgr2=0x7f347c0b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f347c071d80 tx=0x7f347400dc00 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 shutdown_connections 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f345c06c770 0x7f345c06ec30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0b40a0 0x7f347c0b4520 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c13e780 0x7f347c1458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 --2- 172.21.15.67:0/1815657437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3460004800 0x7f347c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.489+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 >> 172.21.15.67:0/1815657437 conn(0x7f347c067d10 msgr2=0x7f347c075290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.490+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 shutdown_connections 2024-04-17T07:29:10.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.490+0000 7f3481000640 1 -- 172.21.15.67:0/1815657437 wait complete. 2024-04-17T07:29:10.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.720+0000 7ff19b200640 1 Processor -- start 2024-04-17T07:29:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.720+0000 7ff19b200640 1 -- start start 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff19c06be70 0x7ff19c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19c15bc80 con 0x7ff19c15b570 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c15bf40 con 0x7ff19c06be70 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff19c15c220 con 0x7ff19c06feb0 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff199e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff19c15b570 0x7ff19c06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51930/0 (socket says 172.21.15.67:51930) 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff199e00640 1 -- 172.21.15.67:0/3857717992 learned_addr learned my addr 172.21.15.67:0/3857717992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.722+0000 7ff19a800640 1 --2- 172.21.15.67:0/3857717992 >> v2:172.21.15.67:3300/0 conn(0x7ff19c06be70 0x7ff19c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff198a00640 1 -- 172.21.15.67:0/3857717992 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994849204 0 0) 0x7ff19c15bc80 con 0x7ff19c15b570 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff199400640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff198a00640 1 -- 172.21.15.67:0/3857717992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff180003520 con 0x7ff19c15b570 2024-04-17T07:29:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff19a800640 1 -- 172.21.15.67:0/3857717992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 msgr2=0x7ff19c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff19a800640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff19a800640 1 -- 172.21.15.67:0/3857717992 >> v1:172.21.15.67:6789/0 conn(0x7ff19c15b570 legacy=0x7ff19c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.723+0000 7ff19a800640 1 -- 172.21.15.67:0/3857717992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c15c500 con 0x7ff19c06be70 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.724+0000 7ff19a800640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15f140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff194009d90 tx=0x7ff19402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6417d996233e5646 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.724+0000 7ff198a00640 1 -- 172.21.15.67:0/3857717992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194039070 con 0x7ff19c06be70 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.724+0000 7ff198a00640 1 -- 172.21.15.67:0/3857717992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19402cbc0 con 0x7ff19c06be70 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.724+0000 7ff198a00640 1 -- 172.21.15.67:0/3857717992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff194034440 con 0x7ff19c06be70 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 -- 172.21.15.67:0/3857717992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 msgr2=0x7ff19c15f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15f140 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff194009d90 tx=0x7ff19402be10 comp rx=0 tx=0).stop 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 -- 172.21.15.67:0/3857717992 shutdown_connections 2024-04-17T07:29:10.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15f140 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 --2- 172.21.15.67:0/3857717992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 -- 172.21.15.67:0/3857717992 >> 172.21.15.67:0/3857717992 conn(0x7ff19c067d10 msgr2=0x7ff19c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 -- 172.21.15.67:0/3857717992 shutdown_connections 2024-04-17T07:29:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.725+0000 7ff19b200640 1 -- 172.21.15.67:0/3857717992 wait complete. 2024-04-17T07:29:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.727+0000 7ff19b200640 1 Processor -- start 2024-04-17T07:29:10.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.727+0000 7ff19b200640 1 -- start start 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff19c15b570 0x7ff19c15e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c06c460 con 0x7ff19c06be70 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c06c2e0 con 0x7ff19c15b570 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff19b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff19c0b47b0 con 0x7ff19c06feb0 2024-04-17T07:29:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff199e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff199e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59630/0 (socket says 172.21.15.67:59630) 2024-04-17T07:29:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.728+0000 7ff199e00640 1 -- 172.21.15.67:0/34189331 learned_addr learned my addr 172.21.15.67:0/34189331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 -- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 msgr2=0x7ff19c155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 -- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff19c15b570 msgr2=0x7ff19c15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff19c15b570 0x7ff19c15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 -- 172.21.15.67:0/34189331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c15e100 con 0x7ff19c06be70 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff199e00640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00c430 tx=0x7ff18c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c017070 con 0x7ff19c06be70 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18c005ce0 con 0x7ff19c06be70 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c0072f0 con 0x7ff19c06be70 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c14e7a0 con 0x7ff19c06be70 2024-04-17T07:29:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.729+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c14ed60 con 0x7ff19c06be70 2024-04-17T07:29:10.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.731+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19c14ef60 con 0x7ff19c06be70 2024-04-17T07:29:10.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.732+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) ==== 90878+0+0 (secure 0 0 0) 0x7ff18c01a070 con 0x7ff19c06be70 2024-04-17T07:29:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.733+0000 7ff192000640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 0x7ff17c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.733+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7ff18c08aff0 con 0x7ff19c06be70 2024-04-17T07:29:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.734+0000 7ff199400640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 0x7ff17c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:10.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.735+0000 7ff199400640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 0x7ff17c06ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff19c072700 tx=0x7ff188008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:10.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.739+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff19c14ef60 con 0x7ff19c06be70 2024-04-17T07:29:10.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.943+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff19c068a10 con 0x7ff19c06be70 2024-04-17T07:29:10.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.946+0000 7ff192000640 1 -- 172.21.15.67:0/34189331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+17739 (secure 0 0 0) 0x7ff18c0593a0 con 0x7ff19c06be70 2024-04-17T07:29:10.946 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:10.946 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:08.999313+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4215758024":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2703892812":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1695242533":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2375944225":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3898372684":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/2716101762":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1671674456":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/4131277908":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3437520709":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3840279087":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3804745495":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1222851080":"2024-04-18T07:27:30.199060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 msgr2=0x7ff17c06ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 0x7ff17c06ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff19c072700 tx=0x7ff188008040 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 msgr2=0x7ff19c15da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00c430 tx=0x7ff18c00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 shutdown_connections 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff17c06c770 0x7ff17c06ec30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff19c15b570 0x7ff19c15e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff19c06feb0 0x7ff19c155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 --2- 172.21.15.67:0/34189331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19c06be70 0x7ff19c15da90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.948+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 >> 172.21.15.67:0/34189331 conn(0x7ff19c067d10 msgr2=0x7ff19c154500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.949+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 shutdown_connections 2024-04-17T07:29:10.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:10.949+0000 7ff19b200640 1 -- 172.21.15.67:0/34189331 wait complete. 2024-04-17T07:29:10.968 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:29:10.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.180+0000 7f6339400640 1 Processor -- start 2024-04-17T07:29:11.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.181+0000 7f6339400640 1 -- start start 2024-04-17T07:29:11.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6339400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f633409b170 0x7f633409b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6339400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 0x7f63340d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6339400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f633409bd80 con 0x7f633406e320 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6339400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f633406c810 con 0x7f633409b170 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6339400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63340cdeb0 con 0x7f63340b3180 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.182+0000 7f6333400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f633409b170 0x7f633409b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f633409b170 0x7f633409b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59640/0 (socket says 172.21.15.67:59640) 2024-04-17T07:29:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 -- 172.21.15.67:0/1093158127 learned_addr learned my addr 172.21.15.67:0/1093158127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 -- 172.21.15.67:0/1093158127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 msgr2=0x7f63340d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6338a00640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 0x7f63340d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 0x7f63340d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 -- 172.21.15.67:0/1093158127 >> v1:172.21.15.187:6789/0 conn(0x7f633406e320 legacy=0x7f633409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6333400640 1 -- 172.21.15.67:0/1093158127 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63340ce190 con 0x7f633409b170 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.183+0000 7f6338a00640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 0x7f63340d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.184+0000 7f6333400640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f633409b7a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6324009e50 tx=0x7f632402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d41383854a196ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6332a00640 1 -- 172.21.15.67:0/1093158127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632402ca80 con 0x7f633409b170 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6332a00640 1 -- 172.21.15.67:0/1093158127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632402cc20 con 0x7f633409b170 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6332a00640 1 -- 172.21.15.67:0/1093158127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63240343f0 con 0x7f633409b170 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 -- 172.21.15.67:0/1093158127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 msgr2=0x7f633409b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f633409b7a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6324009e50 tx=0x7f632402bee0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 -- 172.21.15.67:0/1093158127 shutdown_connections 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63340b3180 0x7f63340d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 --2- 172.21.15.67:0/1093158127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f633409b7a0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.185+0000 7f6339400640 1 -- 172.21.15.67:0/1093158127 >> 172.21.15.67:0/1093158127 conn(0x7f63340b4170 msgr2=0x7f63340b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.186+0000 7f6339400640 1 -- 172.21.15.67:0/1093158127 shutdown_connections 2024-04-17T07:29:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.186+0000 7f6339400640 1 -- 172.21.15.67:0/1093158127 wait complete. 2024-04-17T07:29:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.187+0000 7f6339400640 1 Processor -- start 2024-04-17T07:29:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.187+0000 7f6339400640 1 -- start start 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 0x7f63340c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63340b3180 0x7f63340b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f63340b3a00 con 0x7f633409b170 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f63340b3880 con 0x7f63340b3180 2024-04-17T07:29:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.188+0000 7f6339400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63340d0b70 con 0x7f633406e320 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59642/0 (socket says 172.21.15.67:59642) 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 -- 172.21.15.67:0/2064818744 learned_addr learned my addr 172.21.15.67:0/2064818744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6338a00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63340b3180 0x7f63340b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333e00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 0x7f63340c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 -- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 msgr2=0x7f63340c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 0x7f63340c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 -- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63340b3180 msgr2=0x7f63340b13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63340b3180 0x7f63340b13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 -- 172.21.15.67:0/2064818744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6334153130 con 0x7f633409b170 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333e00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 0x7f63340c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:11.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.189+0000 7f6333400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6324000f80 tx=0x7f6324004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.190+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324043020 con 0x7f633409b170 2024-04-17T07:29:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.190+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632403e450 con 0x7f633409b170 2024-04-17T07:29:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.190+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632403e840 con 0x7f633409b170 2024-04-17T07:29:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.190+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6334153410 con 0x7f633409b170 2024-04-17T07:29:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.190+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63341539d0 con 0x7f633409b170 2024-04-17T07:29:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.191+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6334068a10 con 0x7f633409b170 2024-04-17T07:29:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.192+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) ==== 90878+0+0 (secure 0 0 0) 0x7f6324046020 con 0x7f633409b170 2024-04-17T07:29:11.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.193+0000 7f632be00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 0x7f631006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.193+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7f63240ae420 con 0x7f633409b170 2024-04-17T07:29:11.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.194+0000 7f6333e00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 0x7f631006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.195+0000 7f6333e00640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 0x7f631006ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f631c001680 tx=0x7f631c009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.198+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f632404d030 con 0x7f633409b170 2024-04-17T07:29:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.445+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63340b4f80 con 0x7f633409b170 2024-04-17T07:29:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.446+0000 7f632be00640 1 -- 172.21.15.67:0/2064818744 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7f632407c7b0 con 0x7f633409b170 2024-04-17T07:29:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:11.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:29:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.448+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 msgr2=0x7f631006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.448+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 0x7f631006ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f631c001680 tx=0x7f631c009040 comp rx=0 tx=0).stop 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.448+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 msgr2=0x7f6334150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.448+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6324000f80 tx=0x7f6324004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 shutdown_connections 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f631006c740 0x7f631006ec00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63340b3180 0x7f63340b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f633409b170 0x7f6334150720 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 --2- 172.21.15.67:0/2064818744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f633406e320 0x7f63340c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 >> 172.21.15.67:0/2064818744 conn(0x7f63340b4170 msgr2=0x7f6334070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 shutdown_connections 2024-04-17T07:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.449+0000 7f6339400640 1 -- 172.21.15.67:0/2064818744 wait complete. 2024-04-17T07:29:11.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:11.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.686+0000 7f409be00640 1 Processor -- start 2024-04-17T07:29:11.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.687+0000 7f409be00640 1 -- start start 2024-04-17T07:29:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.688+0000 7f409be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 0x7f409c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.688+0000 7f409be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 0x7f409c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.688+0000 7f409be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c0d2d30 con 0x7f409c15ec60 2024-04-17T07:29:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.688+0000 7f409be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f409c0d49c0 con 0x7f409c0b6d00 2024-04-17T07:29:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.688+0000 7f409be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f409c0d3660 con 0x7f409c0cf910 2024-04-17T07:29:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.689+0000 7f409a000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 0x7f409c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.689+0000 7f409b400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 0x7f409c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.689+0000 7f409a000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 0x7f409c0cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:50628/0 (socket says 172.21.15.67:50628) 2024-04-17T07:29:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.689+0000 7f409a000640 1 -- 172.21.15.67:0/522802380 learned_addr learned my addr 172.21.15.67:0/522802380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.690+0000 7f409b400640 1 -- 172.21.15.67:0/522802380 >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 msgr2=0x7f409c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.690+0000 7f409b400640 1 --2- 172.21.15.67:0/522802380 >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 0x7f409c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.690+0000 7f409b400640 1 -- 172.21.15.67:0/522802380 >> v1:172.21.15.187:6789/0 conn(0x7f409c15ec60 legacy=0x7f409c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.690+0000 7f409b400640 1 -- 172.21.15.67:0/522802380 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c0d3010 con 0x7f409c0cf910 2024-04-17T07:29:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.691+0000 7f409b400640 1 --2- 172.21.15.67:0/522802380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 0x7f409c0b7b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b5e047345752415 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.692+0000 7f4099600640 1 -- 172.21.15.67:0/522802380 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094035020 con 0x7f409c0cf910 2024-04-17T07:29:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.692+0000 7f4099600640 1 -- 172.21.15.67:0/522802380 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094031830 con 0x7f409c0cf910 2024-04-17T07:29:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.692+0000 7f4099600640 1 -- 172.21.15.67:0/522802380 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094031c00 con 0x7f409c0cf910 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.692+0000 7f409be00640 1 -- 172.21.15.67:0/522802380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 msgr2=0x7f409c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.692+0000 7f409be00640 1 --2- 172.21.15.67:0/522802380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 0x7f409c0b7b70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 -- 172.21.15.67:0/522802380 shutdown_connections 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 --2- 172.21.15.67:0/522802380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c0cf910 0x7f409c0b7b70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 --2- 172.21.15.67:0/522802380 >> v2:172.21.15.187:3300/0 conn(0x7f409c0b6d00 0x7f409c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 -- 172.21.15.67:0/522802380 >> 172.21.15.67:0/522802380 conn(0x7f409c0d15f0 msgr2=0x7f409c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 -- 172.21.15.67:0/522802380 shutdown_connections 2024-04-17T07:29:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.693+0000 7f409be00640 1 -- 172.21.15.67:0/522802380 wait complete. 2024-04-17T07:29:11.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.695+0000 7f409be00640 1 Processor -- start 2024-04-17T07:29:11.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.695+0000 7f409be00640 1 -- start start 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 0x7f409c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c15ec60 0x7f409c1f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f409c15f040 con 0x7f409c0b6d00 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f409c0cfdc0 con 0x7f409c0cf910 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f409c0d00c0 con 0x7f409c15ec60 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59644/0 (socket says 172.21.15.67:59644) 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409aa00640 1 -- 172.21.15.67:0/586305026 learned_addr learned my addr 172.21.15.67:0/586305026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409a000640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 0x7f409c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409b400640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c15ec60 0x7f409c1f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409aa00640 1 -- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c15ec60 msgr2=0x7f409c1f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.696+0000 7f409aa00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c15ec60 0x7f409c1f7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409aa00640 1 -- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 msgr2=0x7f409c154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409aa00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 0x7f409c154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409aa00640 1 -- 172.21.15.67:0/586305026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c157720 con 0x7f409c0b6d00 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409a000640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 0x7f409c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409aa00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c730 tx=0x7f408c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c016020 con 0x7f409c0b6d00 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c006d50 con 0x7f409c0b6d00 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c003bb0 con 0x7f409c0b6d00 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c157a00 con 0x7f409c0b6d00 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.697+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c1603b0 con 0x7f409c0b6d00 2024-04-17T07:29:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.698+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c068a10 con 0x7f409c0b6d00 2024-04-17T07:29:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.699+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) ==== 90878+0+0 (secure 0 0 0) 0x7f408c00a850 con 0x7f409c0b6d00 2024-04-17T07:29:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.700+0000 7f4092a00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 0x7f407806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.700+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7f408c08b820 con 0x7f409c0b6d00 2024-04-17T07:29:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.701+0000 7f409a000640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 0x7f407806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.702+0000 7f409a000640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 0x7f407806ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b7ae0 tx=0x7f4084008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.705+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f408c020020 con 0x7f409c0b6d00 2024-04-17T07:29:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.746+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f408c08dce0 con 0x7f409c0b6d00 2024-04-17T07:29:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.954+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f409c160580 con 0x7f409c0b6d00 2024-04-17T07:29:11.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.955+0000 7f4092a00640 1 -- 172.21.15.67:0/586305026 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5199 (secure 0 0 0) 0x7f408c059bb0 con 0x7f409c0b6d00 2024-04-17T07:29:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:11.955 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:27:14.836845+0000","modified":"2024-04-17T07:27:18.391129+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5931},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5931":{"gid":5931,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2094216919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2094216919},{"type":"v1","addr":"172.21.15.67:6835","nonce":2094216919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5931,"qdb_cluster":[5931]},"id":3}]} 2024-04-17T07:29:11.956 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-17T07:29:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 msgr2=0x7f407806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 0x7f407806ec00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f409c0b7ae0 tx=0x7f4084008040 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 msgr2=0x7f409c0ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c730 tx=0x7f408c00cbf0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 shutdown_connections 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f407806c740 0x7f407806ec00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f409c15ec60 0x7f409c1f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f409c0cf910 0x7f409c154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 --2- 172.21.15.67:0/586305026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409c0b6d00 0x7f409c0ce760 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 >> 172.21.15.67:0/586305026 conn(0x7f409c0d15f0 msgr2=0x7f409c0b8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 shutdown_connections 2024-04-17T07:29:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:11.958+0000 7f409be00640 1 -- 172.21.15.67:0/586305026 wait complete. 2024-04-17T07:29:11.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.192+0000 7f0814a00640 1 Processor -- start 2024-04-17T07:29:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.193+0000 7f0814a00640 1 -- start start 2024-04-17T07:29:12.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.194+0000 7f0814a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08100b6d00 0x7f08100cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.194+0000 7f0814a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08100d2d30 con 0x7f08100cf910 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.194+0000 7f0814a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08100d3010 con 0x7f081015ec60 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.194+0000 7f0814a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f08100d49c0 con 0x7f08100b6d00 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08100b6d00 0x7f08100cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080f200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08100cf910 0x7f08100b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51936/0 (socket says 172.21.15.67:51936) 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08100b6d00 0x7f08100cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59660/0 (socket says 172.21.15.67:59660) 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080f200640 1 -- 172.21.15.67:0/3998781726 learned_addr learned my addr 172.21.15.67:0/3998781726 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:12.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080de00640 1 -- 172.21.15.67:0/3998781726 >> v1:172.21.15.187:6789/0 conn(0x7f081015ec60 legacy=0x7f08100b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:12.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080de00640 1 -- 172.21.15.67:0/3998781726 >> v1:172.21.15.67:6789/0 conn(0x7f08100cf910 legacy=0x7f08100b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.195+0000 7f080de00640 1 -- 172.21.15.67:0/3998781726 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08100d32f0 con 0x7f08100b6d00 2024-04-17T07:29:12.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.196+0000 7f080de00640 1 --2- 172.21.15.67:0/3998781726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100b6d00 0x7f08100cf3d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009d90 tx=0x7f07f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e730fa26642dc1e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f080d400640 1 -- 172.21.15.67:0/3998781726 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8039070 con 0x7f08100b6d00 2024-04-17T07:29:12.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f080d400640 1 -- 172.21.15.67:0/3998781726 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f802cc60 con 0x7f08100b6d00 2024-04-17T07:29:12.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f080d400640 1 -- 172.21.15.67:0/3998781726 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8034660 con 0x7f08100b6d00 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f0814a00640 1 -- 172.21.15.67:0/3998781726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100b6d00 msgr2=0x7f08100cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f0814a00640 1 --2- 172.21.15.67:0/3998781726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100b6d00 0x7f08100cf3d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009d90 tx=0x7f07f802beb0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f0814a00640 1 -- 172.21.15.67:0/3998781726 shutdown_connections 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f0814a00640 1 --2- 172.21.15.67:0/3998781726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100b6d00 0x7f08100cf3d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.197+0000 7f0814a00640 1 -- 172.21.15.67:0/3998781726 >> 172.21.15.67:0/3998781726 conn(0x7f08100d15f0 msgr2=0x7f08100d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.198+0000 7f0814a00640 1 -- 172.21.15.67:0/3998781726 shutdown_connections 2024-04-17T07:29:12.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.198+0000 7f0814a00640 1 -- 172.21.15.67:0/3998781726 wait complete. 2024-04-17T07:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.199+0000 7f0814a00640 1 Processor -- start 2024-04-17T07:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.199+0000 7f0814a00640 1 -- start start 2024-04-17T07:29:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08100b6d00 0x7f08101f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f081015ec60 0x7f08100ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f081015f040 con 0x7f08100cf910 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f08100d3660 con 0x7f08100b6d00 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.200+0000 7f0814a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08100d00c0 con 0x7f081015ec60 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59672/0 (socket says 172.21.15.67:59672) 2024-04-17T07:29:12.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 -- 172.21.15.67:0/2469998190 learned_addr learned my addr 172.21.15.67:0/2469998190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:12.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080e800640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08100b6d00 0x7f08101f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080f200640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f081015ec60 0x7f08100ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 -- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f081015ec60 msgr2=0x7f08100ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f081015ec60 0x7f08100ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 -- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08100b6d00 msgr2=0x7f08101f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08100b6d00 0x7f08101f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.201+0000 7f080de00640 1 -- 172.21.15.67:0/2469998190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08101f5b40 con 0x7f08100cf910 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f080de00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f07f8037040 tx=0x7f07f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8039050 con 0x7f08100cf910 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f8045070 con 0x7f08100cf910 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f803ea50 con 0x7f08100cf910 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08100cc5b0 con 0x7f08100cf910 2024-04-17T07:29:12.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.202+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08100ccb70 con 0x7f08100cf910 2024-04-17T07:29:12.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.203+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08100d3ad0 con 0x7f08100cf910 2024-04-17T07:29:12.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.204+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7f07f802cb20 con 0x7f08100cf910 2024-04-17T07:29:12.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.205+0000 7f0806a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 0x7f07ec06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.205+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7f07f80ae780 con 0x7f08100cf910 2024-04-17T07:29:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.206+0000 7f080e800640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 0x7f07ec06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.207+0000 7f080e800640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 0x7f07ec06ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0800006fd0 tx=0x7f0800008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.211+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f07f804e020 con 0x7f08100cf910 2024-04-17T07:29:12.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.417+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0810068a10 con 0x7f08100cf910 2024-04-17T07:29:12.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.420+0000 7f0806a00640 1 -- 172.21.15.67:0/2469998190 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+17739 (secure 0 0 0) 0x7f08101f5b40 con 0x7f08100cf910 2024-04-17T07:29:12.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:12.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:08.999313+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:27:12.167534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-17T07:27:13.288606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":4}]},{"pgid":"7.f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4215758024":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2703892812":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1695242533":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2375944225":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3898372684":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/2716101762":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1671674456":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/4131277908":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3437520709":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3840279087":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3804745495":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1222851080":"2024-04-18T07:27:30.199060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:12.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 msgr2=0x7f07ec06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 0x7f07ec06ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0800006fd0 tx=0x7f0800008040 comp rx=0 tx=0).stop 2024-04-17T07:29:12.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 msgr2=0x7f08100cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f07f8037040 tx=0x7f07f8004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 shutdown_connections 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f07ec06c9b0 0x7f07ec06ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f081015ec60 0x7f08100ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08100cf910 0x7f08100cbc40 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 --2- 172.21.15.67:0/2469998190 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08100b6d00 0x7f08101f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 >> 172.21.15.67:0/2469998190 conn(0x7f08100d15f0 msgr2=0x7f08100b95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 shutdown_connections 2024-04-17T07:29:12.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.423+0000 7f0814a00640 1 -- 172.21.15.67:0/2469998190 wait complete. 2024-04-17T07:29:12.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:29:12.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.663+0000 7fb8c3000640 1 Processor -- start 2024-04-17T07:29:12.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.663+0000 7fb8c3000640 1 -- start start 2024-04-17T07:29:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8c3000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8bc145ac0 0x7fb8bc071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8c3000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 0x7fb8bc0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8c3000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8bc0cea70 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8c3000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb8bc0ce840 con 0x7fb8bc145ac0 2024-04-17T07:29:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8c3000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb8bc0ced50 con 0x7fb8bc072d80 2024-04-17T07:29:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8bc145ac0 0x7fb8bc071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8bc145ac0 0x7fb8bc071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59684/0 (socket says 172.21.15.67:59684) 2024-04-17T07:29:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.665+0000 7fb8bbe00640 1 -- 172.21.15.67:0/3124372373 learned_addr learned my addr 172.21.15.67:0/3124372373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8c1000640 1 --2- 172.21.15.67:0/3124372373 >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 0x7fb8bc0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8bbe00640 1 -- 172.21.15.67:0/3124372373 >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 msgr2=0x7fb8bc0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8bbe00640 1 --2- 172.21.15.67:0/3124372373 >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 0x7fb8bc0cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8bbe00640 1 -- 172.21.15.67:0/3124372373 >> v1:172.21.15.187:6789/0 conn(0x7fb8bc0ce3b0 legacy=0x7fb8bc072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8bbe00640 1 -- 172.21.15.67:0/3124372373 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc0cf030 con 0x7fb8bc145ac0 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.666+0000 7fb8c1000640 1 --2- 172.21.15.67:0/3124372373 >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 0x7fb8bc0cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.667+0000 7fb8bbe00640 1 --2- 172.21.15.67:0/3124372373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc071ea0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009870 tx=0x7fb8ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bea33b24a62922 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.667+0000 7fb8bb400640 1 -- 172.21.15.67:0/3124372373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac035020 con 0x7fb8bc145ac0 2024-04-17T07:29:12.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.667+0000 7fb8bb400640 1 -- 172.21.15.67:0/3124372373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ac031870 con 0x7fb8bc145ac0 2024-04-17T07:29:12.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.667+0000 7fb8bb400640 1 -- 172.21.15.67:0/3124372373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac031c40 con 0x7fb8bc145ac0 2024-04-17T07:29:12.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 -- 172.21.15.67:0/3124372373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc145ac0 msgr2=0x7fb8bc071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 --2- 172.21.15.67:0/3124372373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc071ea0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009870 tx=0x7fb8ac02b910 comp rx=0 tx=0).stop 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 -- 172.21.15.67:0/3124372373 shutdown_connections 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 --2- 172.21.15.67:0/3124372373 >> v2:172.21.15.187:3300/0 conn(0x7fb8bc072d80 0x7fb8bc0cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 --2- 172.21.15.67:0/3124372373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc071ea0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 -- 172.21.15.67:0/3124372373 >> 172.21.15.67:0/3124372373 conn(0x7fb8bc067ee0 msgr2=0x7fb8bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 -- 172.21.15.67:0/3124372373 shutdown_connections 2024-04-17T07:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.668+0000 7fb8c3000640 1 -- 172.21.15.67:0/3124372373 wait complete. 2024-04-17T07:29:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.670+0000 7fb8c3000640 1 Processor -- start 2024-04-17T07:29:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.670+0000 7fb8c3000640 1 -- start start 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8bc072d80 0x7fb8bc0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc1f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc145ea0 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc0cf5f0 con 0x7fb8bc145ac0 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8c3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8bc0d0050 con 0x7fb8bc072d80 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.671+0000 7fb8b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59694/0 (socket says 172.21.15.67:59694) 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 -- 172.21.15.67:0/2628188634 learned_addr learned my addr 172.21.15.67:0/2628188634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8bbe00640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8bc072d80 0x7fb8bc0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 -- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8bc072d80 msgr2=0x7fb8bc0cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8bc072d80 0x7fb8bc0cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8c1000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc1f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 -- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8bc145ac0 msgr2=0x7fb8bc1f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc1f4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 -- 172.21.15.67:0/2628188634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc0cbb70 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b3400640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c410 tx=0x7fb8a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.672+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4017070 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.673+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a40035f0 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.673+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc0cbe50 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.673+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4003a20 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.673+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1f5a80 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.674+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8bc0cc6c0 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.675+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7fb8a401a070 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.676+0000 7fb8b8c00640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 0x7fb89806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.676+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6483+0+0 (secure 0 0 0) 0x7fb8a408b3c0 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.677+0000 7fb8bbe00640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 0x7fb89806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.678+0000 7fb8bbe00640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 0x7fb89806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac036000 tx=0x7fb8ac008030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.681+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb8a4059480 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.928+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb8bc072a80 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.929 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:29:12.929+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:29:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.933+0000 7fb8b8c00640 1 -- 172.21.15.67:0/2628188634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7fb8a40010c0 con 0x7fb8bc0ce3b0 2024-04-17T07:29:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:29:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 msgr2=0x7fb89806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 0x7fb89806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac036000 tx=0x7fb8ac008030 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 msgr2=0x7fb8bc1f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c410 tx=0x7fb8a400c8d0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 shutdown_connections 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb89806c980 0x7fb89806ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8bc145ac0 0x7fb8bc1f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8bc0ce3b0 0x7fb8bc1f4870 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 --2- 172.21.15.67:0/2628188634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8bc072d80 0x7fb8bc0cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.936+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 >> 172.21.15.67:0/2628188634 conn(0x7fb8bc067ee0 msgr2=0x7fb8bc06ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.937+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 shutdown_connections 2024-04-17T07:29:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:12.937+0000 7fb8c3000640 1 -- 172.21.15.67:0/2628188634 wait complete. 2024-04-17T07:29:12.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:29:13.035 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:29:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.160+0000 7fbba1400640 1 Processor -- start 2024-04-17T07:29:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.161+0000 7fbba1400640 1 -- start start 2024-04-17T07:29:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.162+0000 7fbba1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.162+0000 7fbba1400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb9c0cf8e0 0x7fbb9c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.162+0000 7fbba1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb9c0d00b0 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.162+0000 7fbba1400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9c0d0c30 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.162+0000 7fbba1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9c0d0390 con 0x7fbb9c0d33e0 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.163+0000 7fbb93e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.163+0000 7fbba0a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb9c0cf8e0 0x7fbb9c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.163+0000 7fbb9be00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbb9c15ecb0 0x7fbb9c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:57942/0 (socket says 172.21.15.67:57942) 2024-04-17T07:29:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.163+0000 7fbb93e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:37096/0 (socket says 172.21.15.67:37096) 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.163+0000 7fbb9be00640 1 -- 172.21.15.67:0/1768821015 learned_addr learned my addr 172.21.15.67:0/1768821015 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.164+0000 7fbba0a00640 1 -- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 msgr2=0x7fbb9c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.164+0000 7fbba0a00640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.164+0000 7fbba0a00640 1 -- 172.21.15.67:0/1768821015 >> v1:172.21.15.187:6789/0 conn(0x7fbb9c15ecb0 legacy=0x7fbb9c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.164+0000 7fbba0a00640 1 -- 172.21.15.67:0/1768821015 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c0d0670 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.164+0000 7fbb93e00640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:29:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.165+0000 7fbba0a00640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0b3fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009d90 tx=0x7fbb9402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d759e62dc0c980af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:13.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.166+0000 7fbb9b400640 1 -- 172.21.15.67:0/1768821015 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94039070 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.166+0000 7fbb9b400640 1 -- 172.21.15.67:0/1768821015 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb9402cbc0 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.166+0000 7fbb9b400640 1 -- 172.21.15.67:0/1768821015 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94034440 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 -- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0cf8e0 msgr2=0x7fbb9c0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0b3fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009d90 tx=0x7fbb9402be10 comp rx=0 tx=0).stop 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 -- 172.21.15.67:0/1768821015 shutdown_connections 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0b3fc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 --2- 172.21.15.67:0/1768821015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0d33e0 0x7fbb9c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 -- 172.21.15.67:0/1768821015 >> 172.21.15.67:0/1768821015 conn(0x7fbb9c08bff0 msgr2=0x7fbb9c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 -- 172.21.15.67:0/1768821015 shutdown_connections 2024-04-17T07:29:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.167+0000 7fbba1400640 1 -- 172.21.15.67:0/1768821015 wait complete. 2024-04-17T07:29:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.169+0000 7fbba1400640 1 Processor -- start 2024-04-17T07:29:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.169+0000 7fbba1400640 1 -- start start 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 0x7fbb9c0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c15f090 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c0cfd30 con 0x7fbb9c0d33e0 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9c15f210 con 0x7fbb9c0cf8e0 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59704/0 (socket says 172.21.15.67:59704) 2024-04-17T07:29:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.171+0000 7fbba0a00640 1 -- 172.21.15.67:0/2981496473 learned_addr learned my addr 172.21.15.67:0/2981496473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb93e00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 0x7fbb9c0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb9be00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 -- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0cf8e0 msgr2=0x7fbb9c0c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 -- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 msgr2=0x7fbb9c0c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 0x7fbb9c0c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 -- 172.21.15.67:0/2981496473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c15beb0 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb93e00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 0x7fbb9c0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba0a00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009eb0 tx=0x7fbb94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94039050 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb94045070 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c0b8ad0 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.172+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94009450 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.173+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9c0b9310 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.174+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb9c068a10 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.175+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 29) ==== 91086+0+0 (secure 0 0 0) 0x7fbb9403e850 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.176+0000 7fbb98c00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 0x7fbb7806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.176+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6569+0+0 (secure 0 0 0) 0x7fbb940aed50 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.176+0000 7fbb9be00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 0x7fbb7806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:13.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.177+0000 7fbb9be00640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 0x7fbb7806ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000f60 tx=0x7fbb8c009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:13.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.180+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb9c0b8ad0 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.429+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbb9c06b930 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.946+0000 7fbb98c00640 1 -- 172.21.15.67:0/2981496473 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7fbb94032e20 con 0x7fbb9c15ecb0 2024-04-17T07:29:13.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 msgr2=0x7fbb7806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 0x7fbb7806ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000f60 tx=0x7fbb8c009040 comp rx=0 tx=0).stop 2024-04-17T07:29:13.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 msgr2=0x7fbb9c0b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:13.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009eb0 tx=0x7fbb94004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 shutdown_connections 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7806c9d0 0x7fbb7806ee90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c15ecb0 0x7fbb9c0b1f10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb9c0d33e0 0x7fbb9c0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 --2- 172.21.15.67:0/2981496473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb9c0cf8e0 0x7fbb9c0c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.950+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 >> 172.21.15.67:0/2981496473 conn(0x7fbb9c08bff0 msgr2=0x7fbb9c0b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.951+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 shutdown_connections 2024-04-17T07:29:13.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:13.951+0000 7fbba1400640 1 -- 172.21.15.67:0/2981496473 wait complete. 2024-04-17T07:29:13.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:29:14.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.200+0000 7fba32400640 1 Processor -- start 2024-04-17T07:29:14.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.201+0000 7fba32400640 1 -- start start 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba32400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba2c15ec80 0x7fba2c0d2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba32400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba2c0d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba32400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba2c15f060 con 0x7fba2c0d4c40 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba32400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fba2c0cf4f0 con 0x7fba2c15ec80 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba32400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba2c0d3800 con 0x7fba2c0d3230 2024-04-17T07:29:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba31a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba2c0d4c40 0x7fba2c0d0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44426/0 (socket says 172.21.15.67:44426) 2024-04-17T07:29:14.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.202+0000 7fba31a00640 1 -- 172.21.15.67:0/3552629292 learned_addr learned my addr 172.21.15.67:0/3552629292 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba31000640 1 --2- 172.21.15.67:0/3552629292 >> v2:172.21.15.187:3300/0 conn(0x7fba2c15ec80 0x7fba2c0d2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2be00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba2c0d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348872471 0 0) 0x7fba2c15f060 con 0x7fba2c0d4c40 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba10003520 con 0x7fba2c0d4c40 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2004264346 0 0) 0x7fba10003520 con 0x7fba2c0d4c40 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 msgr2=0x7fba2c0d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba2c0d4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 >> v2:172.21.15.187:3300/0 conn(0x7fba2c15ec80 msgr2=0x7fba2c0d2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 --2- 172.21.15.67:0/3552629292 >> v2:172.21.15.187:3300/0 conn(0x7fba2c15ec80 0x7fba2c0d2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2c0d3ae0 con 0x7fba2c0d4c40 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fba24001670 con 0x7fba2c0d4c40 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.203+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 >> v1:172.21.15.67:6789/0 conn(0x7fba2c0d4c40 legacy=0x7fba2c0d0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba10003a20 0x7fba10003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10004800 0x7fba10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c0cf4f0 con 0x7fba10004800 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c0d3800 con 0x7fba10003a20 2024-04-17T07:29:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba1000b9c0 con 0x7fba2c0d3230 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10004800 0x7fba10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba31000640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba10003a20 0x7fba10003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba31a00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 -- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 msgr2=0x7fba100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 -- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba10003a20 msgr2=0x7fba10003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba10003a20 0x7fba10003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba2be00640 1 -- 172.21.15.67:0/3552629292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba10003520 con 0x7fba10004800 2024-04-17T07:29:14.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.204+0000 7fba31a00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba2c0d3230 0x7fba100051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.206+0000 7fba2be00640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10004800 0x7fba10004c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fba14009af0 tx=0x7fba10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9b25325ee1f611 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.206+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14039070 con 0x7fba10004800 2024-04-17T07:29:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.206+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba1402c6b0 con 0x7fba10004800 2024-04-17T07:29:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.206+0000 7fba2b400640 1 -- 172.21.15.67:0/3552629292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14009690 con 0x7fba10004800 2024-04-17T07:29:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 -- 172.21.15.67:0/3552629292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10004800 msgr2=0x7fba10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 --2- 172.21.15.67:0/3552629292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10004800 0x7fba10004c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fba14009af0 tx=0x7fba10005830 comp rx=0 tx=0).stop 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba31a00640 1 -- 172.21.15.67:0/3552629292 reap_dead start 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 -- 172.21.15.67:0/3552629292 shutdown_connections 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 -- 172.21.15.67:0/3552629292 >> 172.21.15.67:0/3552629292 conn(0x7fba2c067600 msgr2=0x7fba2c067a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 -- 172.21.15.67:0/3552629292 shutdown_connections 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.207+0000 7fba32400640 1 -- 172.21.15.67:0/3552629292 wait complete. 2024-04-17T07:29:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.209+0000 7fba32400640 1 Processor -- start 2024-04-17T07:29:14.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.209+0000 7fba32400640 1 -- start start 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba10004800 0x7fba2c0c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 0x7fba2c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c0b7ce0 con 0x7fba2c0c9d50 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c0b8030 con 0x7fba2c0ca900 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba32400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba2c0b9560 con 0x7fba10004800 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55980/0 (socket says 172.21.15.67:55980) 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba2be00640 1 -- 172.21.15.67:0/3193507796 learned_addr learned my addr 172.21.15.67:0/3193507796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.210+0000 7fba31000640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba10004800 0x7fba2c0c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 -- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba10004800 msgr2=0x7fba2c0c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba31a00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 0x7fba2c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba10004800 0x7fba2c0c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 -- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 msgr2=0x7fba2c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 0x7fba2c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 -- 172.21.15.67:0/3193507796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2c0c8cb0 con 0x7fba2c0c9d50 2024-04-17T07:29:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba31a00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 0x7fba2c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba2be00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fba14037040 tx=0x7fba14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14042020 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba1403a040 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2c0c8f90 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.211+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14039070 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.212+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba2c0c9550 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.213+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2c068a10 con 0x7fba2c0c9d50 2024-04-17T07:29:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.214+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fba14045020 con 0x7fba2c0c9d50 2024-04-17T07:29:14.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.214+0000 7fba28c00640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 0x7fba0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:14.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.215+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6569+0+0 (secure 0 0 0) 0x7fba140ad7c0 con 0x7fba2c0c9d50 2024-04-17T07:29:14.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.215+0000 7fba31000640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 0x7fba0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:14.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.216+0000 7fba31000640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 0x7fba0806eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba1c007380 tx=0x7fba1c00b040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:14.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.219+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba1404c030 con 0x7fba2c0c9d50 2024-04-17T07:29:14.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.425+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fba2c0755d0 con 0x7fba2c0c9d50 2024-04-17T07:29:14.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.950+0000 7fba28c00640 1 -- 172.21.15.67:0/3193507796 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v75) ==== 181+0+0 (secure 0 0 0) 0x7fba1407b770 con 0x7fba2c0c9d50 2024-04-17T07:29:14.951 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:29:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 msgr2=0x7fba0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 0x7fba0806eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba1c007380 tx=0x7fba1c00b040 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 msgr2=0x7fba2c14fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fba14037040 tx=0x7fba14004730 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 shutdown_connections 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fba0806ca10 0x7fba0806eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba2c0ca900 0x7fba2c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c0c9d50 0x7fba2c14fb90 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 --2- 172.21.15.67:0/3193507796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba10004800 0x7fba2c0c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 >> 172.21.15.67:0/3193507796 conn(0x7fba2c067600 msgr2=0x7fba2c0b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.954+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 shutdown_connections 2024-04-17T07:29:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:14.955+0000 7fba32400640 1 -- 172.21.15.67:0/3193507796 wait complete. 2024-04-17T07:29:14.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:29:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.199+0000 7f0ac4e00640 1 Processor -- start 2024-04-17T07:29:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.199+0000 7f0ac4e00640 1 -- start start 2024-04-17T07:29:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.200+0000 7f0ac4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ac00cd6d0 0x7f0ac0145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0ac4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac0146080 0x7f0ac0160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0ac4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac0161000 con 0x7f0ac00cd2f0 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0ac4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0ac00cfa60 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0ac4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac0146530 con 0x7f0ac0146080 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0abf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0ac00cd2f0 0x7f0ac00cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44440/0 (socket says 172.21.15.67:44440) 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0abf400640 1 -- 172.21.15.67:0/735168632 learned_addr learned my addr 172.21.15.67:0/735168632 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0abea00640 1 --2- 172.21.15.67:0/735168632 >> v2:172.21.15.67:3300/0 conn(0x7f0ac00cd6d0 0x7f0ac0145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.201+0000 7f0abfe00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac0146080 0x7f0ac0160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.202+0000 7f0abea00640 1 -- 172.21.15.67:0/735168632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac0146080 msgr2=0x7f0ac0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.202+0000 7f0abea00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac0146080 0x7f0ac0160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.202+0000 7f0abea00640 1 -- 172.21.15.67:0/735168632 >> v1:172.21.15.67:6789/0 conn(0x7f0ac00cd2f0 legacy=0x7f0ac00cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.202+0000 7f0abea00640 1 -- 172.21.15.67:0/735168632 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac00ce870 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.203+0000 7f0abea00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac0145b40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097d0 tx=0x7f0aac02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92df8b2ed913e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.203+0000 7f0abe000640 1 -- 172.21.15.67:0/735168632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac035020 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.203+0000 7f0abe000640 1 -- 172.21.15.67:0/735168632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aac031810 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.203+0000 7f0abe000640 1 -- 172.21.15.67:0/735168632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac031be0 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 -- 172.21.15.67:0/735168632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 msgr2=0x7f0ac0145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac0145b40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097d0 tx=0x7f0aac02b8b0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 -- 172.21.15.67:0/735168632 shutdown_connections 2024-04-17T07:29:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac0146080 0x7f0ac0160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/735168632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac0145b40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 -- 172.21.15.67:0/735168632 >> 172.21.15.67:0/735168632 conn(0x7f0ac006b8c0 msgr2=0x7f0ac006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:15.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 -- 172.21.15.67:0/735168632 shutdown_connections 2024-04-17T07:29:15.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.204+0000 7f0ac4e00640 1 -- 172.21.15.67:0/735168632 wait complete. 2024-04-17T07:29:15.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.206+0000 7f0ac4e00640 1 Processor -- start 2024-04-17T07:29:15.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.206+0000 7f0ac4e00640 1 -- start start 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac00cd2f0 0x7f0ac00af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac0146080 0x7f0ac00ab8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01608a0 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac00cee30 con 0x7f0ac0146080 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.207+0000 7f0ac4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac0160a20 con 0x7f0ac00cd2f0 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55996/0 (socket says 172.21.15.67:55996) 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 -- 172.21.15.67:0/2779781211 learned_addr learned my addr 172.21.15.67:0/2779781211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abf400640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac00cd2f0 0x7f0ac00af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 -- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac00cd2f0 msgr2=0x7f0ac00af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac00cd2f0 0x7f0ac00af2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 -- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac0146080 msgr2=0x7f0ac00ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac0146080 0x7f0ac00ab8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 -- 172.21.15.67:0/2779781211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac01e6f70 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.208+0000 7f0abea00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0098f0 tx=0x7f0aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.209+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac041070 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.209+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aac03ad00 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.209+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac01e7250 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.209+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac035040 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.209+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac01e7810 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.210+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac0068a10 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.211+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f0aac038020 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.212+0000 7f0ab7400640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 0x7f0aa006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:15.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.212+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6103+0+0 (secure 0 0 0) 0x7f0aac0adbe0 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.213+0000 7f0abf400640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 0x7f0aa006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:15.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.214+0000 7f0abf400640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 0x7f0aa006ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f60 tx=0x7f0ab0009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:15.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.216+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0aac07bcd0 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.433+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0ac0068610 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.964+0000 7f0ab7400640 1 -- 172.21.15.67:0/2779781211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v76) ==== 169+0+0 (secure 0 0 0) 0x7f0aac07b5d0 con 0x7f0ac00cd6d0 2024-04-17T07:29:15.964 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.967+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 msgr2=0x7f0aa006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.967+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 0x7f0aa006ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f60 tx=0x7f0ab0009040 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.967+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 msgr2=0x7f0ac00ab3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.967+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0098f0 tx=0x7f0aac0046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 shutdown_connections 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0aa006ca40 0x7f0aa006ef00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac0146080 0x7f0ac00ab8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac00cd6d0 0x7f0ac00ab3b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 --2- 172.21.15.67:0/2779781211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ac00cd2f0 0x7f0ac00af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:15.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 >> 172.21.15.67:0/2779781211 conn(0x7f0ac006b8c0 msgr2=0x7f0ac0142860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:15.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 shutdown_connections 2024-04-17T07:29:15.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:15.968+0000 7f0ac4e00640 1 -- 172.21.15.67:0/2779781211 wait complete. 2024-04-17T07:29:15.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:29:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.216+0000 7fb283e00640 1 Processor -- start 2024-04-17T07:29:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.216+0000 7fb283e00640 1 -- start start 2024-04-17T07:29:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb283e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb284162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb283e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28406b8c0 con 0x7fb2840cdce0 2024-04-17T07:29:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb283e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb28406bb80 con 0x7fb284070c30 2024-04-17T07:29:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb283e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2841626d0 con 0x7fb2840d0e80 2024-04-17T07:29:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb27be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb2840cdce0 0x7fb2840d0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44456/0 (socket says 172.21.15.67:44456) 2024-04-17T07:29:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.218+0000 7fb27be00640 1 -- 172.21.15.67:0/1804546132 learned_addr learned my addr 172.21.15.67:0/1804546132 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb283400640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb284162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112575259 0 0) 0x7fb28406b8c0 con 0x7fb2840cdce0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb268003520 con 0x7fb2840cdce0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3208139568 0 0) 0x7fb268003520 con 0x7fb2840cdce0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 msgr2=0x7fb284162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb284162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 >> v1:172.21.15.187:6789/0 conn(0x7fb284070c30 legacy=0x7fb2840b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28406be60 con 0x7fb2840cdce0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb26c001690 con 0x7fb2840cdce0 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 >> v1:172.21.15.67:6789/0 conn(0x7fb2840cdce0 legacy=0x7fb2840d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb268003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.219+0000 7fb282000640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb268004030 0x7fb268004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb282000640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2841626d0 con 0x7fb268004d40 2024-04-17T07:29:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb26800bc10 con 0x7fb268004030 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb26800be10 con 0x7fb2840d0e80 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb27be00640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb268004030 0x7fb268004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb282a00640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb268003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 -- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 msgr2=0x7fb268003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2840d0e80 0x7fb268003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 -- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb268004030 msgr2=0x7fb268004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb268004030 0x7fb268004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb283400640 1 -- 172.21.15.67:0/1804546132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb268003520 con 0x7fb268004d40 2024-04-17T07:29:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.220+0000 7fb27be00640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb268004030 0x7fb268004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.221+0000 7fb283400640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2680051c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb2840d1290 tx=0x7fb268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc5117ea8eb6ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c039070 con 0x7fb268004d40 2024-04-17T07:29:16.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c02cc70 con 0x7fb268004d40 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb282000640 1 -- 172.21.15.67:0/1804546132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c009be0 con 0x7fb268004d40 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb283e00640 1 -- 172.21.15.67:0/1804546132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 msgr2=0x7fb2680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb283e00640 1 --2- 172.21.15.67:0/1804546132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2680051c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb2840d1290 tx=0x7fb268005830 comp rx=0 tx=0).stop 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.222+0000 7fb283400640 1 -- 172.21.15.67:0/1804546132 reap_dead start 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.223+0000 7fb283e00640 1 -- 172.21.15.67:0/1804546132 shutdown_connections 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.223+0000 7fb283e00640 1 -- 172.21.15.67:0/1804546132 >> 172.21.15.67:0/1804546132 conn(0x7fb28409aa60 msgr2=0x7fb2840b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.223+0000 7fb283e00640 1 -- 172.21.15.67:0/1804546132 shutdown_connections 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.223+0000 7fb283e00640 1 -- 172.21.15.67:0/1804546132 wait complete. 2024-04-17T07:29:16.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.224+0000 7fb283e00640 1 Processor -- start 2024-04-17T07:29:16.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 -- start start 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2840c8f00 0x7fb28414e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb28414e8d0 0x7fb28414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2840cd820 con 0x7fb268004d40 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb28406c420 con 0x7fb2840c8f00 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.225+0000 7fb283e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2840ce150 con 0x7fb28414e8d0 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56014/0 (socket says 172.21.15.67:56014) 2024-04-17T07:29:16.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 -- 172.21.15.67:0/4095651718 learned_addr learned my addr 172.21.15.67:0/4095651718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 -- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb28414e8d0 msgr2=0x7fb28414ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb28414e8d0 0x7fb28414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 -- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2840c8f00 msgr2=0x7fb28414e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb27be00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2840c8f00 0x7fb28414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2840c8f00 0x7fb28414e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.226+0000 7fb282a00640 1 -- 172.21.15.67:0/4095651718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2840b5380 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb282a00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb27400e500 tx=0x7fb27400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2740052e0 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb274018070 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2840c9730 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274003810 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.227+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2840c9cf0 con 0x7fb268004d40 2024-04-17T07:29:16.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.228+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2840755d0 con 0x7fb268004d40 2024-04-17T07:29:16.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.229+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fb2740039b0 con 0x7fb268004d40 2024-04-17T07:29:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.230+0000 7fb27aa00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 0x7fb26006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.230+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7fb27408b8e0 con 0x7fb268004d40 2024-04-17T07:29:16.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.231+0000 7fb27be00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 0x7fb26006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.232+0000 7fb27be00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 0x7fb26006ef00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb2840b7260 tx=0x7fb26c001490 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.235+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb274059c60 con 0x7fb268004d40 2024-04-17T07:29:16.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.439+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7fb2840b3eb0 con 0x7fb268004d40 2024-04-17T07:29:16.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.444+0000 7fb27aa00640 1 -- 172.21.15.67:0/4095651718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7fb274059560 con 0x7fb268004d40 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.447+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 msgr2=0x7fb26006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.447+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 0x7fb26006ef00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb2840b7260 tx=0x7fb26c001490 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.447+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 msgr2=0x7fb2840ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.447+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb27400e500 tx=0x7fb27400e9c0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 shutdown_connections 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb26006ca40 0x7fb26006ef00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb28414e8d0 0x7fb28414ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2840c8f00 0x7fb28414e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 --2- 172.21.15.67:0/4095651718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb268004d40 0x7fb2840ca400 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 >> 172.21.15.67:0/4095651718 conn(0x7fb28409aa60 msgr2=0x7fb28415e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:16.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 shutdown_connections 2024-04-17T07:29:16.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.448+0000 7fb283e00640 1 -- 172.21.15.67:0/4095651718 wait complete. 2024-04-17T07:29:16.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:29:16.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.681+0000 7fb1d1600640 1 Processor -- start 2024-04-17T07:29:16.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.681+0000 7fb1d1600640 1 -- start start 2024-04-17T07:29:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d1600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1cc15fd30 0x7fb1cc0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d1600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 0x7fb1cc0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1cc0c8110 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d1600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb1cc15f250 con 0x7fb1cc15fd30 2024-04-17T07:29:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d1600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1cc15f530 con 0x7fb1cc15ed10 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1cbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb1cc06b8c0 0x7fb1cc06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44462/0 (socket says 172.21.15.67:44462) 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1cbe00640 1 -- 172.21.15.67:0/964498804 learned_addr learned my addr 172.21.15.67:0/964498804 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1d0c00640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 0x7fb1cc0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.683+0000 7fb1cb400640 1 --2- 172.21.15.67:0/964498804 >> v2:172.21.15.187:3300/0 conn(0x7fb1cc15fd30 0x7fb1cc0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323031683 0 0) 0x7fb1cc0c8110 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b0003520 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 534027873 0 0) 0x7fb1b0003520 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 msgr2=0x7fb1cc0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 0x7fb1cc0cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 >> v2:172.21.15.187:3300/0 conn(0x7fb1cc15fd30 msgr2=0x7fb1cc0c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 --2- 172.21.15.67:0/964498804 >> v2:172.21.15.187:3300/0 conn(0x7fb1cc15fd30 0x7fb1cc0c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc15f810 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb1bc001670 con 0x7fb1cc06b8c0 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 >> v1:172.21.15.67:6789/0 conn(0x7fb1cc06b8c0 legacy=0x7fb1cc06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b00039b0 0x7fb1b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 0x7fb1b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc15f250 con 0x7fb1b0004800 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc15f530 con 0x7fb1b00039b0 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.684+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b000ba40 con 0x7fb1cc15ed10 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 -- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 msgr2=0x7fb1b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc15ed10 0x7fb1b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 -- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b00039b0 msgr2=0x7fb1b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b00039b0 0x7fb1b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.685+0000 7fb1cb400640 1 -- 172.21.15.67:0/964498804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b0003520 con 0x7fb1b0004800 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.686+0000 7fb1cb400640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009fc0 tx=0x7fb1b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e963ae59790c774 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.686+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4039070 con 0x7fb1b0004800 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b402cc60 con 0x7fb1b0004800 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1caa00640 1 -- 172.21.15.67:0/964498804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4009c10 con 0x7fb1b0004800 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 -- 172.21.15.67:0/964498804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1b0004800 msgr2=0x7fb1b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 --2- 172.21.15.67:0/964498804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009fc0 tx=0x7fb1b0005830 comp rx=0 tx=0).stop 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d0c00640 1 -- 172.21.15.67:0/964498804 reap_dead start 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 -- 172.21.15.67:0/964498804 shutdown_connections 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 -- 172.21.15.67:0/964498804 >> 172.21.15.67:0/964498804 conn(0x7fb1cc0b2fe0 msgr2=0x7fb1cc0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 -- 172.21.15.67:0/964498804 shutdown_connections 2024-04-17T07:29:16.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.687+0000 7fb1d1600640 1 -- 172.21.15.67:0/964498804 wait complete. 2024-04-17T07:29:16.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.689+0000 7fb1d1600640 1 Processor -- start 2024-04-17T07:29:16.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.689+0000 7fb1d1600640 1 -- start start 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 0x7fb1cc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 0x7fb1cc0b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc06c530 con 0x7fb1cc0b6020 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc06c3b0 con 0x7fb1b0004800 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1d1600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1cc06c6b0 con 0x7fb1cc0b40f0 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56034/0 (socket says 172.21.15.67:56034) 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.690+0000 7fb1cb400640 1 -- 172.21.15.67:0/1800209336 learned_addr learned my addr 172.21.15.67:0/1800209336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1d0c00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 0x7fb1cc0b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cbe00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 0x7fb1cc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 -- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 msgr2=0x7fb1cc0b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 0x7fb1cc0b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 -- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 msgr2=0x7fb1cc1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 0x7fb1cc1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 -- 172.21.15.67:0/1800209336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc0b4e90 con 0x7fb1cc0b6020 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1d0c00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 0x7fb1cc0b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cbe00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 0x7fb1cc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.691+0000 7fb1cb400640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4037040 tx=0x7fb1b4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.692+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4039050 con 0x7fb1cc0b6020 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.692+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b4045070 con 0x7fb1cc0b6020 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.692+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b403e850 con 0x7fb1cc0b6020 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.692+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1cc0b5170 con 0x7fb1cc0b6020 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.692+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1cc0b5730 con 0x7fb1cc0b6020 2024-04-17T07:29:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.693+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc0755d0 con 0x7fb1cc0b6020 2024-04-17T07:29:16.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.694+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fb1b403e9f0 con 0x7fb1cc0b6020 2024-04-17T07:29:16.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.695+0000 7fb1c3e00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 0x7fb1a806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:16.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.695+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7fb1b40ae9c0 con 0x7fb1cc0b6020 2024-04-17T07:29:16.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.696+0000 7fb1cbe00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 0x7fb1a806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:16.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.697+0000 7fb1cbe00640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 0x7fb1a806ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc001eb0 tx=0x7fb1bc009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.700+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb1b407cd20 con 0x7fb1cc0b6020 2024-04-17T07:29:16.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.917+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fb1cc0cb700 con 0x7fb1cc0b6020 2024-04-17T07:29:16.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.925+0000 7fb1c3e00640 1 -- 172.21.15.67:0/1800209336 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) ==== 104+0+0 (secure 0 0 0) 0x7fb1b407c620 con 0x7fb1cc0b6020 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 msgr2=0x7fb1a806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 0x7fb1a806ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc001eb0 tx=0x7fb1bc009040 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 msgr2=0x7fb1cc0b64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4037040 tx=0x7fb1b4009750 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 shutdown_connections 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb1a806ca40 0x7fb1a806ef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1cc0b40f0 0x7fb1cc0b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1cc0b6020 0x7fb1cc0b64a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 --2- 172.21.15.67:0/1800209336 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1b0004800 0x7fb1cc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 >> 172.21.15.67:0/1800209336 conn(0x7fb1cc0b2fe0 msgr2=0x7fb1cc06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:16.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 shutdown_connections 2024-04-17T07:29:16.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:16.928+0000 7fb1d1600640 1 -- 172.21.15.67:0/1800209336 wait complete. 2024-04-17T07:29:16.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:29:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.136+0000 7f9231c00640 1 Processor -- start 2024-04-17T07:29:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.137+0000 7f9231c00640 1 -- start start 2024-04-17T07:29:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f9231c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f922c0d33e0 0x7f922c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f9231c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f922c0d00b0 con 0x7f922c15ecb0 2024-04-17T07:29:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f9231c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f922c0d0390 con 0x7f922c0cf8e0 2024-04-17T07:29:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f9231c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f922c0d0c30 con 0x7f922c0d33e0 2024-04-17T07:29:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f922be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f922c15ecb0 0x7f922c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44476/0 (socket says 172.21.15.67:44476) 2024-04-17T07:29:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.138+0000 7f922be00640 1 -- 172.21.15.67:0/2031438920 learned_addr learned my addr 172.21.15.67:0/2031438920 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f9223400640 1 --2- 172.21.15.67:0/2031438920 >> v2:172.21.15.187:3300/0 conn(0x7f922c0d33e0 0x7f922c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703010340 0 0) 0x7f922c0d00b0 con 0x7f922c15ecb0 2024-04-17T07:29:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9210003520 con 0x7f922c15ecb0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2320504206 0 0) 0x7f9210003520 con 0x7f922c15ecb0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 >> v2:172.21.15.187:3300/0 conn(0x7f922c0d33e0 msgr2=0x7f922c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 --2- 172.21.15.67:0/2031438920 >> v2:172.21.15.187:3300/0 conn(0x7f922c0d33e0 0x7f922c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 >> v1:172.21.15.187:6789/0 conn(0x7f922c0cf8e0 legacy=0x7f922c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c0d0670 con 0x7f922c15ecb0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f921c001670 con 0x7f922c15ecb0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.139+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 >> v1:172.21.15.67:6789/0 conn(0x7f922c15ecb0 legacy=0x7f922c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 0x7f9210003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9210004800 0x7f9210004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100051c0 0x7f921000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f922c0d0c30 con 0x7f92100039b0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f921000c160 con 0x7f92100051c0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f921000c360 con 0x7f9210004800 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 0x7f9210003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f9231200640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100051c0 0x7f921000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 -- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9210004800 msgr2=0x7f9210004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9210004800 0x7f9210004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 -- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100051c0 msgr2=0x7f921000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100051c0 0x7f921000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f922be00640 1 -- 172.21.15.67:0/2031438920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9210003520 con 0x7f92100039b0 2024-04-17T07:29:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.140+0000 7f9231200640 1 -- 172.21.15.67:0/2031438920 reap_dead start 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f922be00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 0x7f9210003dd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f921c00c240 tx=0x7f9210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c56c28316a0322 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c007cf0 con 0x7f92100039b0 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f921c030070 con 0x7f92100039b0 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f922b400640 1 -- 172.21.15.67:0/2031438920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c005d10 con 0x7f92100039b0 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f9231c00640 1 -- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 msgr2=0x7f9210003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f9231c00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 0x7f9210003dd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f921c00c240 tx=0x7f9210005830 comp rx=0 tx=0).stop 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f9231c00640 1 -- 172.21.15.67:0/2031438920 shutdown_connections 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f9231c00640 1 --2- 172.21.15.67:0/2031438920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92100039b0 0x7f9210003dd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.142+0000 7f9231c00640 1 -- 172.21.15.67:0/2031438920 >> 172.21.15.67:0/2031438920 conn(0x7f922c08bff0 msgr2=0x7f922c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.143+0000 7f9231c00640 1 -- 172.21.15.67:0/2031438920 shutdown_connections 2024-04-17T07:29:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.143+0000 7f9231c00640 1 -- 172.21.15.67:0/2031438920 wait complete. 2024-04-17T07:29:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.144+0000 7f9231c00640 1 Processor -- start 2024-04-17T07:29:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.144+0000 7f9231c00640 1 -- start start 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100039b0 0x7f922c15f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f922c0cbcf0 0x7f922c0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f922c15f090 con 0x7f922c157cb0 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f922c0cfd30 con 0x7f92100039b0 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.145+0000 7f9231c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f922c15f210 con 0x7f922c0cbcf0 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56048/0 (socket says 172.21.15.67:56048) 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 -- 172.21.15.67:0/2956855369 learned_addr learned my addr 172.21.15.67:0/2956855369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f922be00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100039b0 0x7f922c15f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9231200640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f922c0cbcf0 0x7f922c0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 -- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f922c0cbcf0 msgr2=0x7f922c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f922c0cbcf0 0x7f922c0ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 -- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100039b0 msgr2=0x7f922c15f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100039b0 0x7f922c15f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.146+0000 7f9223400640 1 -- 172.21.15.67:0/2956855369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c0c9440 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9223400640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f921400e770 tx=0x7f921400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214017070 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f921400c730 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921400cb20 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922c0c9720 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.147+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922c0c9ce0 con 0x7f922c157cb0 2024-04-17T07:29:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.148+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922c068a10 con 0x7f922c157cb0 2024-04-17T07:29:17.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.149+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f921401a070 con 0x7f922c157cb0 2024-04-17T07:29:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.151+0000 7f9228c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 0x7f920806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.151+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7f921408bbd0 con 0x7f922c157cb0 2024-04-17T07:29:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.151+0000 7f922be00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 0x7f920806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.152+0000 7f922be00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 0x7f920806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f921c034000 tx=0x7f921c039040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.155+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9214059f40 con 0x7f922c157cb0 2024-04-17T07:29:17.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.368+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f922c0cc0d0 con 0x7f922c157cb0 2024-04-17T07:29:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.377+0000 7f9228c00640 1 -- 172.21.15.67:0/2956855369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v21) ==== 120+0+0 (secure 0 0 0) 0x7f9214059840 con 0x7f922c157cb0 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 msgr2=0x7f920806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 0x7f920806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f921c034000 tx=0x7f921c039040 comp rx=0 tx=0).stop 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 msgr2=0x7f922c0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f921400e770 tx=0x7f921400ec30 comp rx=0 tx=0).stop 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 shutdown_connections 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f920806ca10 0x7f920806eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f922c0cbcf0 0x7f922c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f922c157cb0 0x7f922c0cb7b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 --2- 172.21.15.67:0/2956855369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92100039b0 0x7f922c15f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 >> 172.21.15.67:0/2956855369 conn(0x7f922c08bff0 msgr2=0x7f922c06ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:17.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 shutdown_connections 2024-04-17T07:29:17.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.380+0000 7f9231c00640 1 -- 172.21.15.67:0/2956855369 wait complete. 2024-04-17T07:29:17.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-04-17T07:29:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.624+0000 7ff553e00640 1 Processor -- start 2024-04-17T07:29:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.624+0000 7ff553e00640 1 -- start start 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff553e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff5540cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff553e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff554160280 0x7ff55406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff553e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5540cd4a0 con 0x7ff554141ed0 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff553e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff55415ce30 con 0x7ff554160280 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff553e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff5541607f0 con 0x7ff55406b8c0 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff552000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff554160280 0x7ff55406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff552000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff554160280 0x7ff55406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56060/0 (socket says 172.21.15.67:56060) 2024-04-17T07:29:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff552000640 1 -- 172.21.15.67:0/2634141305 learned_addr learned my addr 172.21.15.67:0/2634141305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff552000640 1 -- 172.21.15.67:0/2634141305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 msgr2=0x7ff5540cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.626+0000 7ff552000640 1 --2- 172.21.15.67:0/2634141305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff5540cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.627+0000 7ff552000640 1 -- 172.21.15.67:0/2634141305 >> v1:172.21.15.187:6789/0 conn(0x7ff554141ed0 legacy=0x7ff55415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.627+0000 7ff552000640 1 -- 172.21.15.67:0/2634141305 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff554160ad0 con 0x7ff554160280 2024-04-17T07:29:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff552000640 1 --2- 172.21.15.67:0/2634141305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff55406bca0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009870 tx=0x7ff53c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc21f6ee43648af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff551600640 1 -- 172.21.15.67:0/2634141305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c035020 con 0x7ff554160280 2024-04-17T07:29:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff551600640 1 -- 172.21.15.67:0/2634141305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c031640 con 0x7ff554160280 2024-04-17T07:29:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff551600640 1 -- 172.21.15.67:0/2634141305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c02d040 con 0x7ff554160280 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff553e00640 1 -- 172.21.15.67:0/2634141305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 msgr2=0x7ff55406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.628+0000 7ff553e00640 1 --2- 172.21.15.67:0/2634141305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff55406bca0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009870 tx=0x7ff53c02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 -- 172.21.15.67:0/2634141305 shutdown_connections 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 --2- 172.21.15.67:0/2634141305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff55406bca0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 --2- 172.21.15.67:0/2634141305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff5540cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 -- 172.21.15.67:0/2634141305 >> 172.21.15.67:0/2634141305 conn(0x7ff554067d10 msgr2=0x7ff5540b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 -- 172.21.15.67:0/2634141305 shutdown_connections 2024-04-17T07:29:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.629+0000 7ff553e00640 1 -- 172.21.15.67:0/2634141305 wait complete. 2024-04-17T07:29:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.630+0000 7ff553e00640 1 Processor -- start 2024-04-17T07:29:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.631+0000 7ff553e00640 1 -- start start 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff554155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff554141ed0 0x7ff55415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff55406c1e0 con 0x7ff554160280 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff5540cddf0 con 0x7ff554141ed0 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff554161090 con 0x7ff55406b8c0 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56076/0 (socket says 172.21.15.67:56076) 2024-04-17T07:29:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 -- 172.21.15.67:0/2491321005 learned_addr learned my addr 172.21.15.67:0/2491321005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff552000640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff554141ed0 0x7ff55415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff552a00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff554155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 -- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 msgr2=0x7ff554155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff554155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 -- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff554141ed0 msgr2=0x7ff55415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff554141ed0 0x7ff55415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff553400640 1 -- 172.21.15.67:0/2491321005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55415dd10 con 0x7ff554160280 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.632+0000 7ff552a00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff554155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff553400640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00c780 tx=0x7ff54c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c017070 con 0x7ff554160280 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff54c003da0 con 0x7ff554160280 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c0052b0 con 0x7ff554160280 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55415dff0 con 0x7ff554160280 2024-04-17T07:29:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.633+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55415e5b0 con 0x7ff554160280 2024-04-17T07:29:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.634+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff554068a10 con 0x7ff554160280 2024-04-17T07:29:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.635+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7ff54c01a070 con 0x7ff554160280 2024-04-17T07:29:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.636+0000 7ff54aa00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 0x7ff53006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.636+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7ff54c08b340 con 0x7ff554160280 2024-04-17T07:29:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.637+0000 7ff552a00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 0x7ff53006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.638+0000 7ff552a00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 0x7ff53006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff544006fd0 tx=0x7ff544008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.641+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff54c0596a0 con 0x7ff554160280 2024-04-17T07:29:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.920+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7ff5540d10c0 con 0x7ff554160280 2024-04-17T07:29:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.976+0000 7ff54aa00640 1 -- 172.21.15.67:0/2491321005 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v564) ==== 154+0+0 (secure 0 0 0) 0x7ff54c058fa0 con 0x7ff554160280 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 msgr2=0x7ff53006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 0x7ff53006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff544006fd0 tx=0x7ff544008040 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 msgr2=0x7ff554140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ff54c00c780 tx=0x7ff54c00cc40 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 shutdown_connections 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff53006ca10 0x7ff53006eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff554160280 0x7ff554140a00 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff554141ed0 0x7ff55415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 --2- 172.21.15.67:0/2491321005 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff55406b8c0 0x7ff554155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.979+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 >> 172.21.15.67:0/2491321005 conn(0x7ff554067d10 msgr2=0x7ff5540b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.980+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 shutdown_connections 2024-04-17T07:29:17.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:17.980+0000 7ff553e00640 1 -- 172.21.15.67:0/2491321005 wait complete. 2024-04-17T07:29:17.998 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-04-17T07:29:17.998 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-04-17T07:29:17.999 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:29:17.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:18.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.219+0000 7fb5a2000640 1 Processor -- start 2024-04-17T07:29:18.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.219+0000 7fb5a2000640 1 -- start start 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.220+0000 7fb5a2000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb59c0b6d00 0x7fb59c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb59c0cf910 0x7fb59c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a2000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb59c0d2d30 con 0x7fb59c15ec60 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a2000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb59c0d49c0 con 0x7fb59c0cf910 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a2000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb59c0d3660 con 0x7fb59c0b6d00 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb59c0cf910 0x7fb59c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb59c0cf910 0x7fb59c0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56078/0 (socket says 172.21.15.67:56078) 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a0c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb59c15ec60 0x7fb59c0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44478/0 (socket says 172.21.15.67:44478) 2024-04-17T07:29:18.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb5a1600640 1 -- 172.21.15.67:0/277828756 learned_addr learned my addr 172.21.15.67:0/277828756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:18.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.221+0000 7fb59be00640 1 --2- 172.21.15.67:0/277828756 >> v2:172.21.15.187:3300/0 conn(0x7fb59c0b6d00 0x7fb59c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.222+0000 7fb5a1600640 1 -- 172.21.15.67:0/277828756 >> v2:172.21.15.187:3300/0 conn(0x7fb59c0b6d00 msgr2=0x7fb59c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:18.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.222+0000 7fb5a1600640 1 --2- 172.21.15.67:0/277828756 >> v2:172.21.15.187:3300/0 conn(0x7fb59c0b6d00 0x7fb59c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.222+0000 7fb5a1600640 1 -- 172.21.15.67:0/277828756 >> v1:172.21.15.67:6789/0 conn(0x7fb59c15ec60 legacy=0x7fb59c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.222+0000 7fb5a1600640 1 -- 172.21.15.67:0/277828756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c0d3010 con 0x7fb59c0cf910 2024-04-17T07:29:18.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.223+0000 7fb5a1600640 1 --2- 172.21.15.67:0/277828756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0cf910 0x7fb59c0b7b70 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ba590 tx=0x7fb59402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2e39d062f3bb7c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.223+0000 7fb59b400640 1 -- 172.21.15.67:0/277828756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59402cb10 con 0x7fb59c0cf910 2024-04-17T07:29:18.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.223+0000 7fb59b400640 1 -- 172.21.15.67:0/277828756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59402ccb0 con 0x7fb59c0cf910 2024-04-17T07:29:18.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.223+0000 7fb59b400640 1 -- 172.21.15.67:0/277828756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59402ce60 con 0x7fb59c0cf910 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 -- 172.21.15.67:0/277828756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0cf910 msgr2=0x7fb59c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 --2- 172.21.15.67:0/277828756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0cf910 0x7fb59c0b7b70 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ba590 tx=0x7fb59402bf70 comp rx=0 tx=0).stop 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 -- 172.21.15.67:0/277828756 shutdown_connections 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 --2- 172.21.15.67:0/277828756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0cf910 0x7fb59c0b7b70 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 --2- 172.21.15.67:0/277828756 >> v2:172.21.15.187:3300/0 conn(0x7fb59c0b6d00 0x7fb59c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 -- 172.21.15.67:0/277828756 >> 172.21.15.67:0/277828756 conn(0x7fb59c0d15f0 msgr2=0x7fb59c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 -- 172.21.15.67:0/277828756 shutdown_connections 2024-04-17T07:29:18.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.224+0000 7fb5a2000640 1 -- 172.21.15.67:0/277828756 wait complete. 2024-04-17T07:29:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.226+0000 7fb5a2000640 1 Processor -- start 2024-04-17T07:29:18.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.226+0000 7fb5a2000640 1 -- start start 2024-04-17T07:29:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb59c0cf910 0x7fb59c1e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb59c15ec60 0x7fb59c1f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c15f040 con 0x7fb59c0b6d00 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c0cfdc0 con 0x7fb59c0cf910 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb59c0d00c0 con 0x7fb59c15ec60 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56092/0 (socket says 172.21.15.67:56092) 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.227+0000 7fb5a0c00640 1 -- 172.21.15.67:0/4111139824 learned_addr learned my addr 172.21.15.67:0/4111139824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 -- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb59c15ec60 msgr2=0x7fb59c1f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb59c15ec60 0x7fb59c1f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 -- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb59c0cf910 msgr2=0x7fb59c1e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb59c0cf910 0x7fb59c1e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 -- 172.21.15.67:0/4111139824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c1f5400 con 0x7fb59c0b6d00 2024-04-17T07:29:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a0c00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c3f0 tx=0x7fb58c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0074e0 con 0x7fb59c0b6d00 2024-04-17T07:29:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb58c007680 con 0x7fb59c0b6d00 2024-04-17T07:29:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c004a00 con 0x7fb59c0b6d00 2024-04-17T07:29:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59c1f56e0 con 0x7fb59c0b6d00 2024-04-17T07:29:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.228+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59c0cc490 con 0x7fb59c0b6d00 2024-04-17T07:29:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.230+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb59c068a10 con 0x7fb59c0b6d00 2024-04-17T07:29:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.231+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fb58c01a020 con 0x7fb59c0b6d00 2024-04-17T07:29:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.232+0000 7fb598c00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 0x7fb57806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.232+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7fb58c08b4a0 con 0x7fb59c0b6d00 2024-04-17T07:29:18.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.232+0000 7fb59be00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 0x7fb57806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.234+0000 7fb59be00640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 0x7fb57806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb59c1ea480 tx=0x7fb584008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.236+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb58c010300 con 0x7fb59c0b6d00 2024-04-17T07:29:18.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.444+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb59c0b5450 con 0x7fb59c0b6d00 2024-04-17T07:29:18.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.447+0000 7fb598c00640 1 -- 172.21.15.67:0/4111139824 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+13992 (secure 0 0 0) 0x7fb58c059810 con 0x7fb59c0b6d00 2024-04-17T07:29:18.447 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:15.952254+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/2094216919":"2024-04-18T07:29:12.930630+0000","172.21.15.67:6834/2094216919":"2024-04-18T07:29:12.930630+0000","172.21.15.187:0/4215758024":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2703892812":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1695242533":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/2375944225":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3898372684":"2024-04-18T07:29:08.999244+0000","172.21.15.187:6833/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/1235852913":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/2716101762":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/1671674456":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/4131277908":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3437520709":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/3840279087":"2024-04-18T07:27:30.199060+0000","172.21.15.187:6832/2009310366":"2024-04-18T07:29:08.999244+0000","172.21.15.187:0/3804745495":"2024-04-18T07:27:30.199060+0000","172.21.15.187:0/1222851080":"2024-04-18T07:27:30.199060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 msgr2=0x7fb57806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 0x7fb57806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb59c1ea480 tx=0x7fb584008040 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 msgr2=0x7fb59c1e97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c3f0 tx=0x7fb58c00c8b0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 shutdown_connections 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb57806ca10 0x7fb57806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb59c15ec60 0x7fb59c1f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb59c0cf910 0x7fb59c1e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 --2- 172.21.15.67:0/4111139824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb59c0b6d00 0x7fb59c1e97c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.449+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 >> 172.21.15.67:0/4111139824 conn(0x7fb59c0d15f0 msgr2=0x7fb59c15d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:18.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.450+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 shutdown_connections 2024-04-17T07:29:18.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.450+0000 7fb5a2000640 1 -- 172.21.15.67:0/4111139824 wait complete. 2024-04-17T07:29:18.469 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:18.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.693+0000 7fe3df200640 1 Processor -- start 2024-04-17T07:29:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.693+0000 7fe3df200640 1 -- start start 2024-04-17T07:29:18.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.695+0000 7fe3df200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe3e0146060 0x7fe3e0160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.695+0000 7fe3df200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e0160920 con 0x7fe3e0145b00 2024-04-17T07:29:18.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.695+0000 7fe3df200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e0160c00 con 0x7fe3e00b4ac0 2024-04-17T07:29:18.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.695+0000 7fe3df200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e0160ee0 con 0x7fe3e0146060 2024-04-17T07:29:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.696+0000 7fe3dde00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3e0145b00 0x7fe3e015cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44480/0 (socket says 172.21.15.67:44480) 2024-04-17T07:29:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.696+0000 7fe3dde00640 1 -- 172.21.15.67:0/2491567723 learned_addr learned my addr 172.21.15.67:0/2491567723 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.696+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2491567723 >> v2:172.21.15.187:3300/0 conn(0x7fe3e0146060 0x7fe3e0160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.696+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1576410570 0 0) 0x7fe3e0160920 con 0x7fe3e0145b00 2024-04-17T07:29:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.696+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3c4003520 con 0x7fe3e0145b00 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 654609041 0 0) 0x7fe3c4003520 con 0x7fe3e0145b00 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 >> v2:172.21.15.187:3300/0 conn(0x7fe3e0146060 msgr2=0x7fe3e0160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 --2- 172.21.15.67:0/2491567723 >> v2:172.21.15.187:3300/0 conn(0x7fe3e0146060 0x7fe3e0160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 >> v1:172.21.15.187:6789/0 conn(0x7fe3e00b4ac0 legacy=0x7fe3e00b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e01611c0 con 0x7fe3e0145b00 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe3d0001670 con 0x7fe3e0145b00 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 >> v1:172.21.15.67:6789/0 conn(0x7fe3e0145b00 legacy=0x7fe3e015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40039b0 0x7fe3c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3c4004800 0x7fe3c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 0x7fe3c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0160ee0 con 0x7fe3c40051c0 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c400c160 con 0x7fe3c4004800 2024-04-17T07:29:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c400c360 con 0x7fe3c40039b0 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3de800640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 0x7fe3c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.697+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3c4004800 0x7fe3c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3dde00640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40039b0 0x7fe3c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3de800640 1 -- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40039b0 msgr2=0x7fe3c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3de800640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40039b0 0x7fe3c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3de800640 1 -- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3c4004800 msgr2=0x7fe3c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3de800640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3c4004800 0x7fe3c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3de800640 1 -- 172.21.15.67:0/2491567723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c4003520 con 0x7fe3c40051c0 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3c4004800 0x7fe3c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.698+0000 7fe3dde00640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40039b0 0x7fe3c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.699+0000 7fe3de800640 1 -- 172.21.15.67:0/2491567723 reap_dead start 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.699+0000 7fe3de800640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 0x7fe3c400b9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8001e50 tx=0x7fe3c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33588b2ba6d0350a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.699+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8001a70 con 0x7fe3c40051c0 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.699+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d8003bb0 con 0x7fe3c40051c0 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.699+0000 7fe3dd400640 1 -- 172.21.15.67:0/2491567723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80050d0 con 0x7fe3c40051c0 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 -- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 msgr2=0x7fe3c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 0x7fe3c400b9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8001e50 tx=0x7fe3c4005830 comp rx=0 tx=0).stop 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 -- 172.21.15.67:0/2491567723 shutdown_connections 2024-04-17T07:29:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 --2- 172.21.15.67:0/2491567723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3c40051c0 0x7fe3c400b9c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 -- 172.21.15.67:0/2491567723 >> 172.21.15.67:0/2491567723 conn(0x7fe3e00b23c0 msgr2=0x7fe3e00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 -- 172.21.15.67:0/2491567723 shutdown_connections 2024-04-17T07:29:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.700+0000 7fe3df200640 1 -- 172.21.15.67:0/2491567723 wait complete. 2024-04-17T07:29:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.702+0000 7fe3df200640 1 Processor -- start 2024-04-17T07:29:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.702+0000 7fe3df200640 1 -- start start 2024-04-17T07:29:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40051c0 0x7fe3e00c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 0x7fe3e014df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0160340 con 0x7fe3e015bff0 2024-04-17T07:29:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e00cfba0 con 0x7fe3e014bf00 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.703+0000 7fe3df200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e01604c0 con 0x7fe3c40051c0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56114/0 (socket says 172.21.15.67:56114) 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 -- 172.21.15.67:0/2331606774 learned_addr learned my addr 172.21.15.67:0/2331606774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3dde00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40051c0 0x7fe3e00c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3de800640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 0x7fe3e014df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 -- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40051c0 msgr2=0x7fe3e00c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40051c0 0x7fe3e00c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 -- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 msgr2=0x7fe3e014df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 0x7fe3e014df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 -- 172.21.15.67:0/2331606774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e014dd30 con 0x7fe3e015bff0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3de800640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 0x7fe3e014df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.704+0000 7fe3d7e00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8011540 tx=0x7fe3c8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.705+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8015870 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.705+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c8015a10 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.705+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8015dc0 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.705+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e014c750 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.705+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e014cd10 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.706+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e0068a10 con 0x7fe3e015bff0 2024-04-17T07:29:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.707+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fe3c801d400 con 0x7fe3e015bff0 2024-04-17T07:29:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.708+0000 7fe3d6000640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 0x7fe3bc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.708+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7fe3c808c3e0 con 0x7fe3e015bff0 2024-04-17T07:29:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.709+0000 7fe3dde00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 0x7fe3bc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:18.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.710+0000 7fe3dde00640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 0x7fe3bc06ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0001920 tx=0x7fe3d0009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.713+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe3c8019030 con 0x7fe3e015bff0 2024-04-17T07:29:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.907+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe3e0068610 con 0x7fe3e015bff0 2024-04-17T07:29:18.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.908+0000 7fe3d6000640 1 -- 172.21.15.67:0/2331606774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) ==== 93+0+10 (secure 0 0 0) 0x7fe3c805a7e0 con 0x7fe3e015bff0 2024-04-17T07:29:18.908 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 msgr2=0x7fe3bc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 0x7fe3bc06ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0001920 tx=0x7fe3d0009040 comp rx=0 tx=0).stop 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 msgr2=0x7fe3e014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8011540 tx=0x7fe3c8011a00 comp rx=0 tx=0).stop 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 shutdown_connections 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fe3bc06ca40 0x7fe3bc06ef00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3e014bf00 0x7fe3e014df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3e015bff0 0x7fe3e014d7f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.910+0000 7fe3df200640 1 --2- 172.21.15.67:0/2331606774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3c40051c0 0x7fe3e00c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.911+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 >> 172.21.15.67:0/2331606774 conn(0x7fe3e00b23c0 msgr2=0x7fe3e015f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.911+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 shutdown_connections 2024-04-17T07:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:18.911+0000 7fe3df200640 1 -- 172.21.15.67:0/2331606774 wait complete. 2024-04-17T07:29:18.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-17T07:29:19.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.151+0000 7f98a5e00640 1 Processor -- start 2024-04-17T07:29:19.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.151+0000 7f98a5e00640 1 -- start start 2024-04-17T07:29:19.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f98a5e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f98a00c68c0 0x7f98a006e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f98a5e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98a015f3c0 con 0x7f98a0072420 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f98a5e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98a015f660 con 0x7f98a015b530 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f98a5e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f98a015f940 con 0x7f98a00c68c0 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f9896000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f98a0072420 0x7f98a006dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44482/0 (socket says 172.21.15.67:44482) 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.153+0000 7f9896000640 1 -- 172.21.15.67:0/572832500 learned_addr learned my addr 172.21.15.67:0/572832500 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231027624 0 0) 0x7f98a015f3c0 con 0x7f98a0072420 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989f400640 1 --2- 172.21.15.67:0/572832500 >> v2:172.21.15.187:3300/0 conn(0x7f98a00c68c0 0x7f98a006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9884003520 con 0x7f98a0072420 2024-04-17T07:29:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2302553262 0 0) 0x7f9884003520 con 0x7f98a0072420 2024-04-17T07:29:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 >> v2:172.21.15.187:3300/0 conn(0x7f98a00c68c0 msgr2=0x7f98a006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 --2- 172.21.15.67:0/572832500 >> v2:172.21.15.187:3300/0 conn(0x7f98a00c68c0 0x7f98a006e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 >> v1:172.21.15.187:6789/0 conn(0x7f98a015b530 legacy=0x7f98a0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a015fc20 con 0x7f98a0072420 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9888001670 con 0x7f98a0072420 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 >> v1:172.21.15.67:6789/0 conn(0x7f98a0072420 legacy=0x7f98a006dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9884003a00 0x7f9884003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f9884004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98840051c0 0x7f988400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98a015f940 con 0x7f9884004800 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f988400c0c0 con 0x7f98840051c0 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.154+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f988400c2c0 con 0x7f9884003a00 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f9884004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f989ea00640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9884003a00 0x7f9884003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f989f400640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98840051c0 0x7f988400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 -- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9884003a00 msgr2=0x7f9884003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9884003a00 0x7f9884003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 -- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98840051c0 msgr2=0x7f988400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98840051c0 0x7f988400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f9896000640 1 -- 172.21.15.67:0/572832500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9884003520 con 0x7f9884004800 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f989f400640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98840051c0 0x7f988400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.155+0000 7f989f400640 1 -- 172.21.15.67:0/572832500 reap_dead start 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.156+0000 7f9896000640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f9884004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9888007c00 tx=0x7f9884005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3b3693740cfb7a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.156+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888007c40 con 0x7f9884004800 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9888032070 con 0x7f9884004800 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f989e000640 1 -- 172.21.15.67:0/572832500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98880058d0 con 0x7f9884004800 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 -- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 msgr2=0x7f9884004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f9884004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9888007c00 tx=0x7f9884005830 comp rx=0 tx=0).stop 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 -- 172.21.15.67:0/572832500 shutdown_connections 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 --2- 172.21.15.67:0/572832500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f9884004c80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 -- 172.21.15.67:0/572832500 >> 172.21.15.67:0/572832500 conn(0x7f98a006bf50 msgr2=0x7f98a00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 -- 172.21.15.67:0/572832500 shutdown_connections 2024-04-17T07:29:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.157+0000 7f98a5e00640 1 -- 172.21.15.67:0/572832500 wait complete. 2024-04-17T07:29:19.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.159+0000 7f98a5e00640 1 Processor -- start 2024-04-17T07:29:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.159+0000 7f98a5e00640 1 -- start start 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.160+0000 7f98a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f98a5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98a015db80 0x7f98a00cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f98a5e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a00ce410 0x7f98a0154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f98a5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98a0160140 con 0x7f9884004800 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f98a5e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f98a015b910 con 0x7f98a00ce410 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f98a5e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f98a0154590 con 0x7f98a015db80 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56130/0 (socket says 172.21.15.67:56130) 2024-04-17T07:29:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 -- 172.21.15.67:0/461871619 learned_addr learned my addr 172.21.15.67:0/461871619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f9896000640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98a015db80 0x7f98a00cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989f400640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a00ce410 0x7f98a0154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 -- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98a015db80 msgr2=0x7f98a00cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98a015db80 0x7f98a00cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 -- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a00ce410 msgr2=0x7f98a0154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a00ce410 0x7f98a0154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.161+0000 7f989ea00640 1 -- 172.21.15.67:0/461871619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a015e140 con 0x7f9884004800 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f989ea00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f989000e8c0 tx=0x7f989000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890003b80 con 0x7f9884004800 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9890018070 con 0x7f9884004800 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98900050d0 con 0x7f9884004800 2024-04-17T07:29:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a015e420 con 0x7f9884004800 2024-04-17T07:29:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.162+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a015e9e0 con 0x7f9884004800 2024-04-17T07:29:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.163+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a0068b90 con 0x7f9884004800 2024-04-17T07:29:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.164+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f9890005270 con 0x7f9884004800 2024-04-17T07:29:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.165+0000 7f9897e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 0x7f987c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.165+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7f989008bd00 con 0x7f9884004800 2024-04-17T07:29:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.166+0000 7f9896000640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 0x7f987c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:19.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.167+0000 7f9896000640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 0x7f987c06ee70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9888036000 tx=0x7f9888009030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.170+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f989001a310 con 0x7f9884004800 2024-04-17T07:29:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.447+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7f98a00c6390 con 0x7f9884004800 2024-04-17T07:29:19.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.992+0000 7f9897e00640 1 -- 172.21.15.67:0/461871619 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v566) ==== 159+0+0 (secure 0 0 0) 0x7f989005a100 con 0x7f9884004800 2024-04-17T07:29:19.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 msgr2=0x7f987c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 0x7f987c06ee70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9888036000 tx=0x7f9888009030 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 msgr2=0x7f98a0154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f989000e8c0 tx=0x7f989000ed80 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 shutdown_connections 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f987c06c9b0 0x7f987c06ee70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98a00ce410 0x7f98a0154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98a015db80 0x7f98a00cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 --2- 172.21.15.67:0/461871619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9884004800 0x7f98a0154050 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 >> 172.21.15.67:0/461871619 conn(0x7f98a006bf50 msgr2=0x7f98a00b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.996+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 shutdown_connections 2024-04-17T07:29:19.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:19.997+0000 7f98a5e00640 1 -- 172.21.15.67:0/461871619 wait complete. 2024-04-17T07:29:20.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:29:20.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.246+0000 7ff879c00640 1 Processor -- start 2024-04-17T07:29:20.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.246+0000 7ff879c00640 1 -- start start 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.247+0000 7ff879c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff87415b6b0 0x7ff87406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff879c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8740cfc10 0x7ff87406be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff879c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87406df40 con 0x7ff87406f800 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff879c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff87406ed00 con 0x7ff87415b6b0 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff879c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff87406e870 con 0x7ff8740cfc10 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff873e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff87415b6b0 0x7ff87406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff879200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8740cfc10 0x7ff87406be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff873e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff87415b6b0 0x7ff87406b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:41054/0 (socket says 172.21.15.67:41054) 2024-04-17T07:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.248+0000 7ff873e00640 1 -- 172.21.15.67:0/4270909820 learned_addr learned my addr 172.21.15.67:0/4270909820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.249+0000 7ff873e00640 1 -- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8740cfc10 msgr2=0x7ff87406be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.249+0000 7ff873e00640 1 --2- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8740cfc10 0x7ff87406be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.249+0000 7ff873e00640 1 -- 172.21.15.67:0/4270909820 >> v1:172.21.15.187:6789/0 conn(0x7ff87406f800 legacy=0x7ff87415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.249+0000 7ff873e00640 1 -- 172.21.15.67:0/4270909820 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87406e220 con 0x7ff87415b6b0 2024-04-17T07:29:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.250+0000 7ff873e00640 1 --2- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff87415b6b0 0x7ff87406b8c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff864009e50 tx=0x7ff86402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172c8e2f7fa3c1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.251+0000 7ff872a00640 1 -- 172.21.15.67:0/4270909820 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86402c9e0 con 0x7ff87415b6b0 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.251+0000 7ff872a00640 1 -- 172.21.15.67:0/4270909820 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86402cb80 con 0x7ff87415b6b0 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.251+0000 7ff872a00640 1 -- 172.21.15.67:0/4270909820 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8640343f0 con 0x7ff87415b6b0 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.251+0000 7ff879c00640 1 -- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff87415b6b0 msgr2=0x7ff87406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.251+0000 7ff879c00640 1 --2- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff87415b6b0 0x7ff87406b8c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff864009e50 tx=0x7ff86402be40 comp rx=0 tx=0).stop 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 -- 172.21.15.67:0/4270909820 shutdown_connections 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 --2- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8740cfc10 0x7ff87406be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 --2- 172.21.15.67:0/4270909820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff87415b6b0 0x7ff87406b8c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 -- 172.21.15.67:0/4270909820 >> 172.21.15.67:0/4270909820 conn(0x7ff8740b2190 msgr2=0x7ff8740b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 -- 172.21.15.67:0/4270909820 shutdown_connections 2024-04-17T07:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.252+0000 7ff879c00640 1 -- 172.21.15.67:0/4270909820 wait complete. 2024-04-17T07:29:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.253+0000 7ff879c00640 1 Processor -- start 2024-04-17T07:29:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.254+0000 7ff879c00640 1 -- start start 2024-04-17T07:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.254+0000 7ff879c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.254+0000 7ff879c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 0x7ff874147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff879c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87415b6b0 0x7ff874150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff879c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8740afee0 con 0x7ff87406f800 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff879c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8740afd60 con 0x7ff8740cfc10 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff879c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8740b44a0 con 0x7ff87415b6b0 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56138/0 (socket says 172.21.15.67:56138) 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 -- 172.21.15.67:0/3402924237 learned_addr learned my addr 172.21.15.67:0/3402924237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873400640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 0x7ff874147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 -- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87415b6b0 msgr2=0x7ff874150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87415b6b0 0x7ff874150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 -- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 msgr2=0x7ff874147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 0x7ff874147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873e00640 1 -- 172.21.15.67:0/3402924237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87414f7c0 con 0x7ff87406f800 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.255+0000 7ff873400640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 0x7ff874147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff873e00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff864000f80 tx=0x7ff864004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff864043020 con 0x7ff87406f800 2024-04-17T07:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86403e450 con 0x7ff87406f800 2024-04-17T07:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86403e840 con 0x7ff87406f800 2024-04-17T07:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87414faa0 con 0x7ff87406f800 2024-04-17T07:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.256+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff874150060 con 0x7ff87406f800 2024-04-17T07:29:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.257+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff874068a10 con 0x7ff87406f800 2024-04-17T07:29:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.258+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7ff864046020 con 0x7ff87406f800 2024-04-17T07:29:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.260+0000 7ff86be00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 0x7ff85406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.260+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7ff8640ae570 con 0x7ff87406f800 2024-04-17T07:29:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.260+0000 7ff873400640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 0x7ff85406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.262+0000 7ff873400640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 0x7ff85406ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff87414dc10 tx=0x7ff860009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.264+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff86404d030 con 0x7ff87406f800 2024-04-17T07:29:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.469+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff87406ee80 con 0x7ff87406f800 2024-04-17T07:29:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.477+0000 7ff86be00640 1 -- 172.21.15.67:0/3402924237 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7ff86407c850 con 0x7ff87406f800 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 msgr2=0x7ff85406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 0x7ff85406ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff87414dc10 tx=0x7ff860009040 comp rx=0 tx=0).stop 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 msgr2=0x7ff8741461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff864000f80 tx=0x7ff864004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 shutdown_connections 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7ff85406ca40 0x7ff85406ef00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87415b6b0 0x7ff874150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8740cfc10 0x7ff874147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 --2- 172.21.15.67:0/3402924237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff87406f800 0x7ff8741461a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 >> 172.21.15.67:0/3402924237 conn(0x7ff8740b2190 msgr2=0x7ff87414df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 shutdown_connections 2024-04-17T07:29:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:20.480+0000 7ff879c00640 1 -- 172.21.15.67:0/3402924237 wait complete. 2024-04-17T07:29:20.498 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:29:20.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:20.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:20.727 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:29:20.727 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:20.727 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:20.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:20.787 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:20.787 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:20.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:20.843 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:20.843 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:20.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:20.903 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:20.903 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:20.958 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:20.959 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:29:20.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.185+0000 7f710f400640 1 Processor -- start 2024-04-17T07:29:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.186+0000 7f710f400640 1 -- start start 2024-04-17T07:29:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710f400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f710815ed10 0x7f71080d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710f400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71080d3020 con 0x7f71080cf640 2024-04-17T07:29:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710f400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71080d3300 con 0x7f71080d2ab0 2024-04-17T07:29:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710f400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f71080d3c30 con 0x7f710815ed10 2024-04-17T07:29:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f71080cf640 0x7f71080cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44486/0 (socket says 172.21.15.67:44486) 2024-04-17T07:29:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.187+0000 7f710ca00640 1 -- 172.21.15.67:0/1893141538 learned_addr learned my addr 172.21.15.67:0/1893141538 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107e00640 1 --2- 172.21.15.67:0/1893141538 >> v2:172.21.15.187:3300/0 conn(0x7f710815ed10 0x7f71080d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 440881766 0 0) 0x7f71080d3020 con 0x7f71080cf640 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70f0003520 con 0x7f71080cf640 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 710394530 0 0) 0x7f70f0003520 con 0x7f71080cf640 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 >> v2:172.21.15.187:3300/0 conn(0x7f710815ed10 msgr2=0x7f71080d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 --2- 172.21.15.67:0/1893141538 >> v2:172.21.15.187:3300/0 conn(0x7f710815ed10 0x7f71080d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 >> v1:172.21.15.187:6789/0 conn(0x7f71080d2ab0 legacy=0x7f71080b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71080d35e0 con 0x7f71080cf640 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.188+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f70fc001670 con 0x7f71080cf640 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 >> v1:172.21.15.67:6789/0 conn(0x7f71080cf640 legacy=0x7f71080cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70f0003a00 0x7f70f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 0x7f70f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71080d3c30 con 0x7f70f00051c0 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70f000c070 con 0x7f70f0003a00 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70f000c270 con 0x7f70f0004800 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 0x7f70f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710ca00640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70f0003a00 0x7f70f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f7107e00640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 -- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f0004800 msgr2=0x7f70f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 -- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70f0003a00 msgr2=0x7f70f0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70f0003a00 0x7f70f0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.189+0000 7f710d400640 1 -- 172.21.15.67:0/1893141538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f0003520 con 0x7f70f00051c0 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.190+0000 7f710d400640 1 -- 172.21.15.67:0/1893141538 reap_dead start 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f710d400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 0x7f70f000b9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f71000044b0 tx=0x7f70f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8f9094ac7e2633 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100034070 con 0x7f70f00051c0 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7100006d70 con 0x7f70f00051c0 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f7107400640 1 -- 172.21.15.67:0/1893141538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100004080 con 0x7f70f00051c0 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f710f400640 1 -- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 msgr2=0x7f70f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.191+0000 7f710f400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 0x7f70f000b9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f71000044b0 tx=0x7f70f0005830 comp rx=0 tx=0).stop 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.192+0000 7f710f400640 1 -- 172.21.15.67:0/1893141538 shutdown_connections 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.192+0000 7f710f400640 1 --2- 172.21.15.67:0/1893141538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70f00051c0 0x7f70f000b9c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.192+0000 7f710f400640 1 -- 172.21.15.67:0/1893141538 >> 172.21.15.67:0/1893141538 conn(0x7f710806cbd0 msgr2=0x7f710806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.192+0000 7f710f400640 1 -- 172.21.15.67:0/1893141538 shutdown_connections 2024-04-17T07:29:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.192+0000 7f710f400640 1 -- 172.21.15.67:0/1893141538 wait complete. 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.194+0000 7f710f400640 1 Processor -- start 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.194+0000 7f710f400640 1 -- start start 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f00051c0 0x7f71080b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 0x7f71080b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71080ba300 con 0x7f7108161310 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71080d25b0 con 0x7f71081539b0 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71080ba480 con 0x7f70f00051c0 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56150/0 (socket says 172.21.15.67:56150) 2024-04-17T07:29:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 -- 172.21.15.67:0/1104826169 learned_addr learned my addr 172.21.15.67:0/1104826169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710ca00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f00051c0 0x7f71080b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f7107e00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 0x7f71080b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 -- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f00051c0 msgr2=0x7f71080b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f00051c0 0x7f71080b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 -- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 msgr2=0x7f71080b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.195+0000 7f710d400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 0x7f71080b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f710d400640 1 -- 172.21.15.67:0/1104826169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71080b5a00 con 0x7f7108161310 2024-04-17T07:29:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f7107e00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 0x7f71080b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f710d400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7100006fd0 tx=0x7f7100003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100047070 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7100007560 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7100034040 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71080b6100 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.196+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71080b66c0 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.198+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7108068b90 con 0x7f7108161310 2024-04-17T07:29:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.198+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f710004a070 con 0x7f7108161310 2024-04-17T07:29:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.199+0000 7f7104c00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 0x7f70e806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.200+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7f71000ad160 con 0x7f7108161310 2024-04-17T07:29:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.200+0000 7f710ca00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 0x7f70e806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.201+0000 7f710ca00640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 0x7f70e806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70fc001050 tx=0x7f70fc009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.204+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f710007b4d0 con 0x7f7108161310 2024-04-17T07:29:21.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.447+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71080b6890 con 0x7f7108161310 2024-04-17T07:29:21.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.448+0000 7f7104c00640 1 -- 172.21.15.67:0/1104826169 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+3869 (secure 0 0 0) 0x7f710007add0 con 0x7f7108161310 2024-04-17T07:29:21.449 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:21.449 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5003,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/686713075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":686713075},{"type":"v1","addr":"172.21.15.187:6835","nonce":686713075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/942188417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":942188417},{"type":"v1","addr":"172.21.15.187:6837","nonce":942188417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2580088470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2580088470},{"type":"v1","addr":"172.21.15.67:6833","nonce":2580088470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6345,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2538702821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2538702821},{"type":"v1","addr":"172.21.15.67:6835","nonce":2538702821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[]} 2024-04-17T07:29:21.449 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 msgr2=0x7f70e806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 0x7f70e806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70fc001050 tx=0x7f70fc009040 comp rx=0 tx=0).stop 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 msgr2=0x7f71081617c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7100006fd0 tx=0x7f7100003a70 comp rx=0 tx=0).stop 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 shutdown_connections 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70e806ca40 0x7f70e806ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7108161310 0x7f71081617c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71081539b0 0x7f71080b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 --2- 172.21.15.67:0/1104826169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70f00051c0 0x7f71080b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 >> 172.21.15.67:0/1104826169 conn(0x7f710806cbd0 msgr2=0x7f71080b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 shutdown_connections 2024-04-17T07:29:21.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.451+0000 7f710f400640 1 -- 172.21.15.67:0/1104826169 wait complete. 2024-04-17T07:29:21.471 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:29:21.471 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:29:21.471 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:29:21.471 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:29:21.472 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:29:21.472 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:29:21.472 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:29:21.472 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:29:21.472 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:29:21.472 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:29:21.472 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:29:21.472 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:29:21.473 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:29:21.473 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:29:21.473 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:29:21.473 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:29:21.489 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:29:21.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:29:21.490 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:29:21.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:29:21.497 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:29:21.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:29:21.502 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:29:21.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:29:21.534 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:29:21.535 DEBUG:teuthology.parallel:result is None 2024-04-17T07:29:21.539 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:29:21.539 DEBUG:teuthology.parallel:result is None 2024-04-17T07:29:21.549 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:29:21.550 DEBUG:teuthology.parallel:result is None 2024-04-17T07:29:21.556 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:29:21.556 DEBUG:teuthology.parallel:result is None 2024-04-17T07:29:21.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:29:21.687 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:29:21.687 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:29:21.712 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:29:21.720 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:29:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.742+0000 7f1bfda00640 1 Processor -- start 2024-04-17T07:29:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.742+0000 7f1bfda00640 1 -- start start 2024-04-17T07:29:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.743+0000 7f1bfda00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bf815ee20 0x7f1bf80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.743+0000 7f1bfda00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf80d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.743+0000 7f1bfda00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf80d4140 con 0x7f1bf80d3350 2024-04-17T07:29:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.743+0000 7f1bfda00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf80b83b0 con 0x7f1bf815ee20 2024-04-17T07:29:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.743+0000 7f1bfda00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf80d1d20 con 0x7f1bf80d18e0 2024-04-17T07:29:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf80d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bfd000640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1bf80d3350 0x7f1bf8067600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:52800/0 (socket says 172.21.15.67:52800) 2024-04-17T07:29:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bfd000640 1 -- 172.21.15.67:0/2567645362 learned_addr learned my addr 172.21.15.67:0/2567645362 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1bf815ee20 0x7f1bf80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7400640 1 -- 172.21.15.67:0/2567645362 >> v2:172.21.15.187:3300/0 conn(0x7f1bf815ee20 msgr2=0x7f1bf80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7400640 1 --2- 172.21.15.67:0/2567645362 >> v2:172.21.15.187:3300/0 conn(0x7f1bf815ee20 0x7f1bf80d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7400640 1 -- 172.21.15.67:0/2567645362 >> v1:172.21.15.187:6789/0 conn(0x7f1bf80d3350 legacy=0x7f1bf8067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf6a00640 1 -- 172.21.15.67:0/2567645362 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582409021 0 0) 0x7f1bf80d4140 con 0x7f1bf80d3350 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.744+0000 7f1bf7400640 1 -- 172.21.15.67:0/2567645362 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf80d3b80 con 0x7f1bf80d18e0 2024-04-17T07:29:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.745+0000 7f1bf7400640 1 --2- 172.21.15.67:0/2567645362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf80d2e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1be40097a0 tx=0x7f1be402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84e85dcc532cd4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bf6a00640 1 -- 172.21.15.67:0/2567645362 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4035020 con 0x7f1bf80d18e0 2024-04-17T07:29:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bf6a00640 1 -- 172.21.15.67:0/2567645362 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be4031830 con 0x7f1bf80d18e0 2024-04-17T07:29:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bf6a00640 1 -- 172.21.15.67:0/2567645362 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4031c00 con 0x7f1bf80d18e0 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 -- 172.21.15.67:0/2567645362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 msgr2=0x7f1bf80d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 --2- 172.21.15.67:0/2567645362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf80d2e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1be40097a0 tx=0x7f1be402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 -- 172.21.15.67:0/2567645362 shutdown_connections 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 --2- 172.21.15.67:0/2567645362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf80d2e10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 --2- 172.21.15.67:0/2567645362 >> v2:172.21.15.187:3300/0 conn(0x7f1bf815ee20 0x7f1bf80d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.746+0000 7f1bfda00640 1 -- 172.21.15.67:0/2567645362 >> 172.21.15.67:0/2567645362 conn(0x7f1bf806d3c0 msgr2=0x7f1bf80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.747+0000 7f1bfda00640 1 -- 172.21.15.67:0/2567645362 shutdown_connections 2024-04-17T07:29:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.747+0000 7f1bfda00640 1 -- 172.21.15.67:0/2567645362 wait complete. 2024-04-17T07:29:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.748+0000 7f1bfda00640 1 Processor -- start 2024-04-17T07:29:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.748+0000 7f1bfda00640 1 -- start start 2024-04-17T07:29:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf8155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bf80d3350 0x7f1bf80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf815f200 con 0x7f1bf815ee20 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf80b8040 con 0x7f1bf80d3350 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.749+0000 7f1bfda00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf80d37e0 con 0x7f1bf80d18e0 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56184/0 (socket says 172.21.15.67:56184) 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 -- 172.21.15.67:0/1604933652 learned_addr learned my addr 172.21.15.67:0/1604933652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bf7e00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf8155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bf7400640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bf80d3350 0x7f1bf80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 -- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 msgr2=0x7f1bf8155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf8155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 -- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bf80d3350 msgr2=0x7f1bf80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bf80d3350 0x7f1bf80cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 -- 172.21.15.67:0/1604933652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf8156700 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.750+0000 7f1bfd000640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c7a0 tx=0x7f1bf000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.751+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0018070 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.751+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf000a640 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.751+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf81569e0 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.751+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf000aab0 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.751+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf81600e0 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.752+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf8068a10 con 0x7f1bf815ee20 2024-04-17T07:29:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.753+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f1bf001b070 con 0x7f1bf815ee20 2024-04-17T07:29:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.754+0000 7f1befe00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 0x7f1bd806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.754+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5631+0+0 (secure 0 0 0) 0x7f1bf008b210 con 0x7f1bf815ee20 2024-04-17T07:29:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.754+0000 7f1bf7e00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 0x7f1bd806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.755+0000 7f1bf7e00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 0x7f1bd806ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009aa0 tx=0x7f1be8009190 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.758+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1bf0058f20 con 0x7f1bf815ee20 2024-04-17T07:29:21.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:21.967+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f1bf8162640 con 0x7f1bf815ee20 2024-04-17T07:29:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.020+0000 7f1befe00640 1 -- 172.21.15.67:0/1604933652 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) ==== 97+0+0 (secure 0 0 0) 0x7f1bf005d220 con 0x7f1bf815ee20 2024-04-17T07:29:22.020 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 msgr2=0x7f1bd806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 0x7f1bd806ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009aa0 tx=0x7f1be8009190 comp rx=0 tx=0).stop 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 msgr2=0x7f1bf80ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000c7a0 tx=0x7f1bf000cc60 comp rx=0 tx=0).stop 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 shutdown_connections 2024-04-17T07:29:22.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f1bd806ca40 0x7f1bd806ef00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bf815ee20 0x7f1bf80ce3b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bf80d3350 0x7f1bf80cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 --2- 172.21.15.67:0/1604933652 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bf80d18e0 0x7f1bf8155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 >> 172.21.15.67:0/1604933652 conn(0x7f1bf806d3c0 msgr2=0x7f1bf8067d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 shutdown_connections 2024-04-17T07:29:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.023+0000 7f1bfda00640 1 -- 172.21.15.67:0/1604933652 wait complete. 2024-04-17T07:29:22.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:29:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.257+0000 7f89aea00640 1 Processor -- start 2024-04-17T07:29:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.257+0000 7f89aea00640 1 -- start start 2024-04-17T07:29:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89aea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89a8147d60 0x7f89a8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89aea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89a80b3eb0 0x7f89a80722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89aea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a80aedd0 con 0x7f89a80afb90 2024-04-17T07:29:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89aea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89a80af0b0 con 0x7f89a8147d60 2024-04-17T07:29:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89aea00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89a80af390 con 0x7f89a80b3eb0 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89a8147d60 0x7f89a8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89a8147d60 0x7f89a8071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56192/0 (socket says 172.21.15.67:56192) 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89acc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89a80afb90 0x7f89a80b37a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44514/0 (socket says 172.21.15.67:44514) 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 -- 172.21.15.67:0/891225852 learned_addr learned my addr 172.21.15.67:0/891225852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ae000640 1 --2- 172.21.15.67:0/891225852 >> v2:172.21.15.187:3300/0 conn(0x7f89a80b3eb0 0x7f89a80722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 -- 172.21.15.67:0/891225852 >> v2:172.21.15.187:3300/0 conn(0x7f89a80b3eb0 msgr2=0x7f89a80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 --2- 172.21.15.67:0/891225852 >> v2:172.21.15.187:3300/0 conn(0x7f89a80b3eb0 0x7f89a80722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 -- 172.21.15.67:0/891225852 >> v1:172.21.15.67:6789/0 conn(0x7f89a80afb90 legacy=0x7f89a80b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.259+0000 7f89ad600640 1 -- 172.21.15.67:0/891225852 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a80af670 con 0x7f89a8147d60 2024-04-17T07:29:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.260+0000 7f89ad600640 1 --2- 172.21.15.67:0/891225852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a8071d60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b0020 tx=0x7f899c02b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d14e7195a786f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.260+0000 7f89a7e00640 1 -- 172.21.15.67:0/891225852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c037070 con 0x7f89a8147d60 2024-04-17T07:29:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.260+0000 7f89a7e00640 1 -- 172.21.15.67:0/891225852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f899c030710 con 0x7f89a8147d60 2024-04-17T07:29:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.260+0000 7f89a7e00640 1 -- 172.21.15.67:0/891225852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c030ae0 con 0x7f89a8147d60 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 -- 172.21.15.67:0/891225852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 msgr2=0x7f89a8071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 --2- 172.21.15.67:0/891225852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a8071d60 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b0020 tx=0x7f899c02b760 comp rx=0 tx=0).stop 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 -- 172.21.15.67:0/891225852 shutdown_connections 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 --2- 172.21.15.67:0/891225852 >> v2:172.21.15.187:3300/0 conn(0x7f89a80b3eb0 0x7f89a80722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 --2- 172.21.15.67:0/891225852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a8071d60 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 -- 172.21.15.67:0/891225852 >> 172.21.15.67:0/891225852 conn(0x7f89a80b5f70 msgr2=0x7f89a806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 -- 172.21.15.67:0/891225852 shutdown_connections 2024-04-17T07:29:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.261+0000 7f89aea00640 1 -- 172.21.15.67:0/891225852 wait complete. 2024-04-17T07:29:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.263+0000 7f89aea00640 1 Processor -- start 2024-04-17T07:29:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.263+0000 7f89aea00640 1 -- start start 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89a80afb90 0x7f89a815de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89a80b3eb0 0x7f89a80c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81482c0 con 0x7f89a8147d60 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8148140 con 0x7f89a80b3eb0 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.264+0000 7f89aea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89a806dd80 con 0x7f89a80afb90 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56198/0 (socket says 172.21.15.67:56198) 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 -- 172.21.15.67:0/3302383942 learned_addr learned my addr 172.21.15.67:0/3302383942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ad600640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89a80afb90 0x7f89a815de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89acc00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89a80b3eb0 0x7f89a80c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 -- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89a80afb90 msgr2=0x7f89a815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89a80afb90 0x7f89a815de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 -- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89a80b3eb0 msgr2=0x7f89a80c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89a80b3eb0 0x7f89a80c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.265+0000 7f89ae000640 1 -- 172.21.15.67:0/3302383942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a815e3a0 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89ae000640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007e70 tx=0x7f89a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0014620 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a0019070 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0014a20 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a8144490 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.266+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a80c61a0 con 0x7f89a8147d60 2024-04-17T07:29:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.267+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a8068a10 con 0x7f89a8147d60 2024-04-17T07:29:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.268+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f89a0014bc0 con 0x7f89a8147d60 2024-04-17T07:29:22.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.269+0000 7f89a5600640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 0x7f898806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.270+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 4857+0+0 (secure 0 0 0) 0x7f89a008c260 con 0x7f89a8147d60 2024-04-17T07:29:22.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.270+0000 7f89ad600640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 0x7f898806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.271+0000 7f89ad600640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 0x7f898806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f899c034000 tx=0x7f899c008030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.274+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f89a005a960 con 0x7f89a8147d60 2024-04-17T07:29:22.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.558+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f89a8146d70 con 0x7f89a8147d60 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.559+0000 7f89a5600640 1 -- 172.21.15.67:0/3302383942 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7f89a005a260 con 0x7f89a8147d60 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:29:22.559 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:29:22.561 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.564 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:29:22.565 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:29:22.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.568 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:29:22.570 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:29:22.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:22.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:29:22.571 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:29:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.561+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 msgr2=0x7f898806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.561+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 0x7f898806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f899c034000 tx=0x7f899c008030 comp rx=0 tx=0).stop 2024-04-17T07:29:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.561+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 msgr2=0x7f89a80c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.561+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007e70 tx=0x7f89a000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 shutdown_connections 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f898806ca10 0x7f898806eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89a8147d60 0x7f89a80c58d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89a80b3eb0 0x7f89a80c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 --2- 172.21.15.67:0/3302383942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89a80afb90 0x7f89a815de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 >> 172.21.15.67:0/3302383942 conn(0x7f89a80b5f70 msgr2=0x7f89a80b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 shutdown_connections 2024-04-17T07:29:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.562+0000 7f89aea00640 1 -- 172.21.15.67:0/3302383942 wait complete. 2024-04-17T07:29:22.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:22.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.803+0000 7fbed5600640 1 Processor -- start 2024-04-17T07:29:22.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.805+0000 7fbed5600640 1 -- start start 2024-04-17T07:29:22.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbed5600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbed00cd020 0x7fbed00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbed5600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed00ceb00 con 0x7fbed006f670 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbed5600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbed00cede0 con 0x7fbed00cd020 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00cf0c0 con 0x7fbed006e020 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbecf400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbec7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbed00cd020 0x7fbed00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbecea00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbed006f670 0x7fbed006fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:52808/0 (socket says 172.21.15.67:52808) 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbecea00640 1 -- 172.21.15.67:0/2280472170 learned_addr learned my addr 172.21.15.67:0/2280472170 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.806+0000 7fbecf400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:60776/0 (socket says 172.21.15.67:60776) 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.807+0000 7fbec7e00640 1 -- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 msgr2=0x7fbed006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.807+0000 7fbec7e00640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.807+0000 7fbec7e00640 1 -- 172.21.15.67:0/2280472170 >> v1:172.21.15.187:6789/0 conn(0x7fbed006f670 legacy=0x7fbed006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.807+0000 7fbec7e00640 1 -- 172.21.15.67:0/2280472170 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed00cf3a0 con 0x7fbed00cd020 2024-04-17T07:29:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.807+0000 7fbecf400640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:29:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.808+0000 7fbec7e00640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed00cd020 0x7fbed00cd480 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80097a0 tx=0x7fbeb802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db8fdeee96ae2fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.809+0000 7fbece000640 1 -- 172.21.15.67:0/2280472170 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8035020 con 0x7fbed00cd020 2024-04-17T07:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.809+0000 7fbece000640 1 -- 172.21.15.67:0/2280472170 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb8031810 con 0x7fbed00cd020 2024-04-17T07:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.809+0000 7fbece000640 1 -- 172.21.15.67:0/2280472170 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8031be0 con 0x7fbed00cd020 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 -- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed00cd020 msgr2=0x7fbed00cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed00cd020 0x7fbed00cd480 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80097a0 tx=0x7fbeb802b8b0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 -- 172.21.15.67:0/2280472170 shutdown_connections 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed006e020 0x7fbed006e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 --2- 172.21.15.67:0/2280472170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed00cd020 0x7fbed00cd480 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 -- 172.21.15.67:0/2280472170 >> 172.21.15.67:0/2280472170 conn(0x7fbed006bf50 msgr2=0x7fbed00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 -- 172.21.15.67:0/2280472170 shutdown_connections 2024-04-17T07:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.810+0000 7fbed5600640 1 -- 172.21.15.67:0/2280472170 wait complete. 2024-04-17T07:29:22.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.813+0000 7fbed5600640 1 Processor -- start 2024-04-17T07:29:22.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.813+0000 7fbed5600640 1 -- start start 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f670 0x7fbed01f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed00cd020 0x7fbed01f86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00ce8a0 con 0x7fbed006e020 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00ce720 con 0x7fbed006f670 2024-04-17T07:29:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbed5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbed015cb00 con 0x7fbed00cd020 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56200/0 (socket says 172.21.15.67:56200) 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 -- 172.21.15.67:0/594561061 learned_addr learned my addr 172.21.15.67:0/594561061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecf400640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed00cd020 0x7fbed01f86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbec7e00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f670 0x7fbed01f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 -- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed00cd020 msgr2=0x7fbed01f86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed00cd020 0x7fbed01f86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 -- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f670 msgr2=0x7fbed01f81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.814+0000 7fbecea00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f670 0x7fbed01f81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbecea00640 1 -- 172.21.15.67:0/594561061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed00aea40 con 0x7fbed006e020 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbecea00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c3c0 tx=0x7fbec000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0017070 con 0x7fbed006e020 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec0005ce0 con 0x7fbed006e020 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00072f0 con 0x7fbed006e020 2024-04-17T07:29:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed00aed20 con 0x7fbed006e020 2024-04-17T07:29:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.815+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed00af2e0 con 0x7fbed006e020 2024-04-17T07:29:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.816+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed0068b90 con 0x7fbed006e020 2024-04-17T07:29:22.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.818+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fbec001a070 con 0x7fbed006e020 2024-04-17T07:29:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.819+0000 7fbec6a00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 0x7fbeac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.819+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 4857+0+0 (secure 0 0 0) 0x7fbec008ad00 con 0x7fbed006e020 2024-04-17T07:29:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.819+0000 7fbec7e00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 0x7fbeac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.820+0000 7fbec7e00640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 0x7fbeac06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8036000 tx=0x7fbeb8008030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:22.823+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbec0059400 con 0x7fbed006e020 2024-04-17T07:29:23.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.033+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbed006f000 con 0x7fbed006e020 2024-04-17T07:29:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.035+0000 7fbec6a00640 1 -- 172.21.15.67:0/594561061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+12843 (secure 0 0 0) 0x7fbec0058d00 con 0x7fbed006e020 2024-04-17T07:29:23.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:23.035 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:22.034672+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:23.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 msgr2=0x7fbeac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 0x7fbeac06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8036000 tx=0x7fbeb8008030 comp rx=0 tx=0).stop 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 msgr2=0x7fbed00c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c3c0 tx=0x7fbec000c880 comp rx=0 tx=0).stop 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 shutdown_connections 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeac06ca10 0x7fbeac06eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbed00cd020 0x7fbed01f86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed006f670 0x7fbed01f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 --2- 172.21.15.67:0/594561061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed006e020 0x7fbed00c8c00 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.038+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 >> 172.21.15.67:0/594561061 conn(0x7fbed006bf50 msgr2=0x7fbed015b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:23.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.039+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 shutdown_connections 2024-04-17T07:29:23.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.039+0000 7fbed5600640 1 -- 172.21.15.67:0/594561061 wait complete. 2024-04-17T07:29:23.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:23.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.278+0000 7f70d6800640 1 Processor -- start 2024-04-17T07:29:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.279+0000 7f70d6800640 1 -- start start 2024-04-17T07:29:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d6800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006fd30 0x7f70d015b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d015b990 0x7f70d00726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d6800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d0070110 con 0x7f70d00ce510 2024-04-17T07:29:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d6800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f70d015eb70 con 0x7f70d015b990 2024-04-17T07:29:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d6800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70d00cf300 con 0x7f70d006fd30 2024-04-17T07:29:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d015b990 0x7f70d00726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d015b990 0x7f70d00726d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56204/0 (socket says 172.21.15.67:56204) 2024-04-17T07:29:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.281+0000 7f70d4a00640 1 -- 172.21.15.67:0/2434145272 learned_addr learned my addr 172.21.15.67:0/2434145272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.282+0000 7f70d5400640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006fd30 0x7f70d015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.282+0000 7f70d4a00640 1 -- 172.21.15.67:0/2434145272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006fd30 msgr2=0x7f70d015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.282+0000 7f70d4a00640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006fd30 0x7f70d015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.282+0000 7f70d4a00640 1 -- 172.21.15.67:0/2434145272 >> v1:172.21.15.187:6789/0 conn(0x7f70d00ce510 legacy=0x7f70d015f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.282+0000 7f70d4a00640 1 -- 172.21.15.67:0/2434145272 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d00cea60 con 0x7f70d015b990 2024-04-17T07:29:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.283+0000 7f70d4a00640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d015b990 0x7f70d00726d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009870 tx=0x7f70bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a65b2e86bf36fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70cfe00640 1 -- 172.21.15.67:0/2434145272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc034070 con 0x7f70d015b990 2024-04-17T07:29:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70cfe00640 1 -- 172.21.15.67:0/2434145272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70bc030840 con 0x7f70d015b990 2024-04-17T07:29:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70cfe00640 1 -- 172.21.15.67:0/2434145272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc030c10 con 0x7f70d015b990 2024-04-17T07:29:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 -- 172.21.15.67:0/2434145272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d015b990 msgr2=0x7f70d00726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d015b990 0x7f70d00726d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009870 tx=0x7f70bc02b980 comp rx=0 tx=0).stop 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 -- 172.21.15.67:0/2434145272 shutdown_connections 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d015b990 0x7f70d00726d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 --2- 172.21.15.67:0/2434145272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006fd30 0x7f70d015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 -- 172.21.15.67:0/2434145272 >> 172.21.15.67:0/2434145272 conn(0x7f70d00b23c0 msgr2=0x7f70d00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.284+0000 7f70d6800640 1 -- 172.21.15.67:0/2434145272 shutdown_connections 2024-04-17T07:29:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.285+0000 7f70d6800640 1 -- 172.21.15.67:0/2434145272 wait complete. 2024-04-17T07:29:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.286+0000 7f70d6800640 1 Processor -- start 2024-04-17T07:29:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.286+0000 7f70d6800640 1 -- start start 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00ce510 0x7f70d0136880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d015b990 0x7f70d014a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00cfa80 con 0x7f70d006fd30 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00cf020 con 0x7f70d00ce510 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.287+0000 7f70d6800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70d00cfc00 con 0x7f70d015b990 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59150/0 (socket says 172.21.15.67:59150) 2024-04-17T07:29:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d5400640 1 -- 172.21.15.67:0/16892424 learned_addr learned my addr 172.21.15.67:0/16892424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:23.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d4a00640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00ce510 0x7f70d0136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d5e00640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d015b990 0x7f70d014a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.288+0000 7f70d5400640 1 -- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d015b990 msgr2=0x7f70d014a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70d5400640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d015b990 0x7f70d014a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70d5400640 1 -- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00ce510 msgr2=0x7f70d0136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70d5400640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00ce510 0x7f70d0136880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70d5400640 1 -- 172.21.15.67:0/16892424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d0136dc0 con 0x7f70d006fd30 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70d5400640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f70c400d600 tx=0x7f70c400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c40125f0 con 0x7f70d006fd30 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.289+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c4019070 con 0x7f70d006fd30 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.290+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d0137040 con 0x7f70d006fd30 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.290+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c40129f0 con 0x7f70d006fd30 2024-04-17T07:29:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.290+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d0137600 con 0x7f70d006fd30 2024-04-17T07:29:23.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.291+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0068a10 con 0x7f70d006fd30 2024-04-17T07:29:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.292+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f70c4012b90 con 0x7f70d006fd30 2024-04-17T07:29:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.293+0000 7f70cd600640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 0x7f70b006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.293+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 4857+0+0 (secure 0 0 0) 0x7f70c408bd70 con 0x7f70d006fd30 2024-04-17T07:29:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.294+0000 7f70d4a00640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 0x7f70b006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.295+0000 7f70d4a00640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 0x7f70b006ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036000 tx=0x7f70bc03b040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.297+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f70d0136dc0 con 0x7f70d006fd30 2024-04-17T07:29:23.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.504+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70d0068610 con 0x7f70d006fd30 2024-04-17T07:29:23.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.505+0000 7f70cd600640 1 -- 172.21.15.67:0/16892424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f70c405a4c0 con 0x7f70d006fd30 2024-04-17T07:29:23.504 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 msgr2=0x7f70b006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 0x7f70b006ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036000 tx=0x7f70bc03b040 comp rx=0 tx=0).stop 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 msgr2=0x7f70d014b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f70c400d600 tx=0x7f70c400dac0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 shutdown_connections 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f70b006ca60 0x7f70b006ef20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d015b990 0x7f70d014a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00ce510 0x7f70d0136880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 --2- 172.21.15.67:0/16892424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d006fd30 0x7f70d014b880 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 >> 172.21.15.67:0/16892424 conn(0x7f70d00b23c0 msgr2=0x7f70d00b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 shutdown_connections 2024-04-17T07:29:23.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.508+0000 7f70d6800640 1 -- 172.21.15.67:0/16892424 wait complete. 2024-04-17T07:29:23.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:29:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.748+0000 7fb766800640 1 Processor -- start 2024-04-17T07:29:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.749+0000 7fb766800640 1 -- start start 2024-04-17T07:29:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.750+0000 7fb766800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb760160210 0x7fb7600cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.750+0000 7fb766800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7600ceea0 con 0x7fb76009b170 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.750+0000 7fb766800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7600b21c0 con 0x7fb7600ce2e0 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.750+0000 7fb766800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7600b2af0 con 0x7fb760160210 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb760160210 0x7fb7600cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb760160210 0x7fb7600cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59162/0 (socket says 172.21.15.67:59162) 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb76009b170 0x7fb7600b2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58438/0 (socket says 172.21.15.67:58438) 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 -- 172.21.15.67:0/3945362990 learned_addr learned my addr 172.21.15.67:0/3945362990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 -- 172.21.15.67:0/3945362990 >> v1:172.21.15.187:6789/0 conn(0x7fb7600ce2e0 legacy=0x7fb76009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 -- 172.21.15.67:0/3945362990 >> v1:172.21.15.67:6789/0 conn(0x7fb76009b170 legacy=0x7fb7600b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75f400640 1 -- 172.21.15.67:0/3945362990 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7600b24a0 con 0x7fb760160210 2024-04-17T07:29:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.751+0000 7fb75e000640 1 -- 172.21.15.67:0/3945362990 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2826871920 0 0) 0x7fb7600ceea0 con 0x7fb76009b170 2024-04-17T07:29:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.752+0000 7fb75f400640 1 --2- 172.21.15.67:0/3945362990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600cdda0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb750009870 tx=0x7fb75002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c528068c78fec97 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.753+0000 7fb75e000640 1 -- 172.21.15.67:0/3945362990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75002f840 con 0x7fb760160210 2024-04-17T07:29:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.753+0000 7fb75e000640 1 -- 172.21.15.67:0/3945362990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb75002f9e0 con 0x7fb760160210 2024-04-17T07:29:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.753+0000 7fb75e000640 1 -- 172.21.15.67:0/3945362990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75002fdb0 con 0x7fb760160210 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.753+0000 7fb766800640 1 -- 172.21.15.67:0/3945362990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 msgr2=0x7fb7600cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.753+0000 7fb766800640 1 --2- 172.21.15.67:0/3945362990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600cdda0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb750009870 tx=0x7fb75002ba20 comp rx=0 tx=0).stop 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.754+0000 7fb766800640 1 -- 172.21.15.67:0/3945362990 shutdown_connections 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.754+0000 7fb766800640 1 --2- 172.21.15.67:0/3945362990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600cdda0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.754+0000 7fb766800640 1 -- 172.21.15.67:0/3945362990 >> 172.21.15.67:0/3945362990 conn(0x7fb7600b41b0 msgr2=0x7fb7600b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.754+0000 7fb766800640 1 -- 172.21.15.67:0/3945362990 shutdown_connections 2024-04-17T07:29:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.754+0000 7fb766800640 1 -- 172.21.15.67:0/3945362990 wait complete. 2024-04-17T07:29:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.755+0000 7fb766800640 1 Processor -- start 2024-04-17T07:29:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 -- start start 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb76009b170 0x7fb7600c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 0x7fb7600b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7600af4d0 con 0x7fb760160210 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb7600af350 con 0x7fb76009b170 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.756+0000 7fb766800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7600b3690 con 0x7fb7600ce2e0 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59164/0 (socket says 172.21.15.67:59164) 2024-04-17T07:29:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 -- 172.21.15.67:0/4131009313 learned_addr learned my addr 172.21.15.67:0/4131009313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75ea00640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 0x7fb7600b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 -- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 msgr2=0x7fb7600b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 0x7fb7600b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75f400640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb76009b170 0x7fb7600c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 -- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb76009b170 msgr2=0x7fb7600c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb76009b170 0x7fb7600c6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75fe00640 1 -- 172.21.15.67:0/4131009313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7600c76e0 con 0x7fb760160210 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.757+0000 7fb75ea00640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 0x7fb7600b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb75fe00640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c450 tx=0x7fb75800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758018070 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb758007500 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7600b68a0 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758004a20 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.758+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7600c73f0 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.759+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb760068b90 con 0x7fb760160210 2024-04-17T07:29:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.760+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fb75801b070 con 0x7fb760160210 2024-04-17T07:29:23.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.762+0000 7fb757400640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 0x7fb74006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.762+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 4857+0+0 (secure 0 0 0) 0x7fb75808ac80 con 0x7fb760160210 2024-04-17T07:29:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.762+0000 7fb75f400640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 0x7fb74006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.763+0000 7fb75f400640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 0x7fb74006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb750036000 tx=0x7fb750039040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:23.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.767+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb7580592e0 con 0x7fb760160210 2024-04-17T07:29:23.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.974+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb76006d780 con 0x7fb760160210 2024-04-17T07:29:23.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.974+0000 7fb757400640 1 -- 172.21.15.67:0/4131009313 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) ==== 84+0+65 (secure 0 0 0) 0x7fb758058be0 con 0x7fb760160210 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:23.974 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 msgr2=0x7fb74006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 0x7fb74006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb750036000 tx=0x7fb750039040 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 msgr2=0x7fb7600b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c450 tx=0x7fb75800c910 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 shutdown_connections 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fb74006ca10 0x7fb74006eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb760160210 0x7fb7600b0be0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7600ce2e0 0x7fb7600b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 --2- 172.21.15.67:0/4131009313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb76009b170 0x7fb7600c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.977+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 >> 172.21.15.67:0/4131009313 conn(0x7fb7600b41b0 msgr2=0x7fb7600cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.978+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 shutdown_connections 2024-04-17T07:29:23.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:23.978+0000 7fb766800640 1 -- 172.21.15.67:0/4131009313 wait complete. 2024-04-17T07:29:23.995 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:29:23.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:29:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.214+0000 7f13c4a00640 1 Processor -- start 2024-04-17T07:29:24.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.215+0000 7f13c4a00640 1 -- start start 2024-04-17T07:29:24.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.216+0000 7f13c4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13c009aa60 0x7f13c009b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.216+0000 7f13c4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c00ceeb0 con 0x7f13c00ccd40 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.216+0000 7f13c4a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c00d13c0 con 0x7f13c006d3e0 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.216+0000 7f13c4a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f13c0162530 con 0x7f13c009aa60 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13c009aa60 0x7f13c009b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13c009aa60 0x7f13c009b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59178/0 (socket says 172.21.15.67:59178) 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 -- 172.21.15.67:0/1145429580 learned_addr learned my addr 172.21.15.67:0/1145429580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 -- 172.21.15.67:0/1145429580 >> v1:172.21.15.187:6789/0 conn(0x7f13c006d3e0 legacy=0x7f13c009ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 -- 172.21.15.67:0/1145429580 >> v1:172.21.15.67:6789/0 conn(0x7f13c00ccd40 legacy=0x7f13c00cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.217+0000 7f13be800640 1 -- 172.21.15.67:0/1145429580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c00d16a0 con 0x7f13c009aa60 2024-04-17T07:29:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.218+0000 7f13be800640 1 --2- 172.21.15.67:0/1145429580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c009aa60 0x7f13c009b740 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f13b00098a0 tx=0x7f13b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf12ed9beb4ed95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:24.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13bd400640 1 -- 172.21.15.67:0/1145429580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0035020 con 0x7f13c009aa60 2024-04-17T07:29:24.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13bd400640 1 -- 172.21.15.67:0/1145429580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b0031640 con 0x7f13c009aa60 2024-04-17T07:29:24.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13bd400640 1 -- 172.21.15.67:0/1145429580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b002d040 con 0x7f13c009aa60 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 -- 172.21.15.67:0/1145429580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c009aa60 msgr2=0x7f13c009b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1145429580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c009aa60 0x7f13c009b740 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f13b00098a0 tx=0x7f13b002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 -- 172.21.15.67:0/1145429580 shutdown_connections 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1145429580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c009aa60 0x7f13c009b740 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 -- 172.21.15.67:0/1145429580 >> 172.21.15.67:0/1145429580 conn(0x7f13c00b41e0 msgr2=0x7f13c00b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.219+0000 7f13c4a00640 1 -- 172.21.15.67:0/1145429580 shutdown_connections 2024-04-17T07:29:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.220+0000 7f13c4a00640 1 -- 172.21.15.67:0/1145429580 wait complete. 2024-04-17T07:29:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.222+0000 7f13c4a00640 1 Processor -- start 2024-04-17T07:29:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.222+0000 7f13c4a00640 1 -- start start 2024-04-17T07:29:24.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 0x7f13c00b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13c009aa60 0x7f13c00b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00d1cf0 con 0x7f13c00ccd40 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f13c006dbc0 con 0x7f13c006d3e0 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.223+0000 7f13c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13c006d890 con 0x7f13c009aa60 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59182/0 (socket says 172.21.15.67:59182) 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 -- 172.21.15.67:0/1247270575 learned_addr learned my addr 172.21.15.67:0/1247270575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13be800640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 0x7f13c00b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bde00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13c009aa60 0x7f13c00b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 -- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13c009aa60 msgr2=0x7f13c00b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13c009aa60 0x7f13c00b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 -- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 msgr2=0x7f13c00b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 0x7f13c00b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13bf200640 1 -- 172.21.15.67:0/1247270575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c00b19a0 con 0x7f13c00ccd40 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.224+0000 7f13be800640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 0x7f13c00b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13bf200640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f13b800c6b0 tx=0x7f13b800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8003b70 con 0x7f13c00ccd40 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b8003d30 con 0x7f13c00ccd40 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8005310 con 0x7f13c00ccd40 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c00b1c80 con 0x7f13c00ccd40 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.225+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c00b2240 con 0x7f13c00ccd40 2024-04-17T07:29:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.226+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c00b2720 con 0x7f13c00ccd40 2024-04-17T07:29:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.227+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f13b80054b0 con 0x7f13c00ccd40 2024-04-17T07:29:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.228+0000 7f13b6a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 0x7f139c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.228+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 4857+0+0 (secure 0 0 0) 0x7f13b808b8a0 con 0x7f13c00ccd40 2024-04-17T07:29:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.229+0000 7f13be800640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 0x7f139c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:24.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.230+0000 7f13be800640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 0x7f139c06eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036000 tx=0x7f13b0008030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:24.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.235+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f13c00b2720 con 0x7f13c00ccd40 2024-04-17T07:29:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:24.441+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f13c01621f0 con 0x7f13c00ccd40 2024-04-17T07:29:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.048+0000 7f13b6a00640 1 -- 172.21.15.67:0/1247270575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) ==== 138+0+0 (secure 0 0 0) 0x7f13b8059e80 con 0x7f13c00ccd40 2024-04-17T07:29:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.052+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 msgr2=0x7f139c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.052+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 0x7f139c06eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036000 tx=0x7f13b0008030 comp rx=0 tx=0).stop 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.052+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 msgr2=0x7f13c00c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.052+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f13b800c6b0 tx=0x7f13b800cb70 comp rx=0 tx=0).stop 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 shutdown_connections 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f139c06ca10 0x7f139c06eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13c00ccd40 0x7f13c00c9d70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13c009aa60 0x7f13c00b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 --2- 172.21.15.67:0/1247270575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13c006d3e0 0x7f13c00b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.053+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 >> 172.21.15.67:0/1247270575 conn(0x7f13c00b41e0 msgr2=0x7f13c00d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:25.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.054+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 shutdown_connections 2024-04-17T07:29:25.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.055+0000 7f13c4a00640 1 -- 172.21.15.67:0/1247270575 wait complete. 2024-04-17T07:29:25.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:29:25.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.293+0000 7fbede000640 1 Processor -- start 2024-04-17T07:29:25.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.293+0000 7fbede000640 1 -- start start 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbede000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbed80b7f00 0x7fbed80cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbede000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed80b64e0 con 0x7fbed80cff10 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbede000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed80b67c0 con 0x7fbed815ecc0 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbede000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbed80b6290 con 0x7fbed80b7f00 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbed7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbed80cff10 0x7fbed8162940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58444/0 (socket says 172.21.15.67:58444) 2024-04-17T07:29:25.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.295+0000 7fbed7e00640 1 -- 172.21.15.67:0/3095123622 learned_addr learned my addr 172.21.15.67:0/3095123622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbedd600640 1 --2- 172.21.15.67:0/3095123622 >> v2:172.21.15.187:3300/0 conn(0x7fbed80b7f00 0x7fbed80cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754700716 0 0) 0x7fbed80b64e0 con 0x7fbed80cff10 2024-04-17T07:29:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbebc003520 con 0x7fbed80cff10 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3810083593 0 0) 0x7fbebc003520 con 0x7fbed80cff10 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 >> v2:172.21.15.187:3300/0 conn(0x7fbed80b7f00 msgr2=0x7fbed80cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 --2- 172.21.15.67:0/3095123622 >> v2:172.21.15.187:3300/0 conn(0x7fbed80b7f00 0x7fbed80cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 >> v1:172.21.15.187:6789/0 conn(0x7fbed815ecc0 legacy=0x7fbed80cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed80b6aa0 con 0x7fbed80cff10 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbec00017f0 con 0x7fbed80cff10 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 >> v1:172.21.15.67:6789/0 conn(0x7fbed80cff10 legacy=0x7fbed8162940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0039b0 0x7fbebc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbebc004800 0x7fbebc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 0x7fbebc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbed80b6290 con 0x7fbebc0051c0 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc00c160 con 0x7fbebc004800 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.296+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc00c360 con 0x7fbebc0039b0 2024-04-17T07:29:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 0x7fbebc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedcc00640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0039b0 0x7fbebc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbed7e00640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbebc004800 0x7fbebc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 -- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0039b0 msgr2=0x7fbebc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0039b0 0x7fbebc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 -- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbebc004800 msgr2=0x7fbebc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbebc004800 0x7fbebc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.297+0000 7fbedd600640 1 -- 172.21.15.67:0/3095123622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc003520 con 0x7fbebc0051c0 2024-04-17T07:29:25.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.298+0000 7fbedd600640 1 -- 172.21.15.67:0/3095123622 reap_dead start 2024-04-17T07:29:25.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.299+0000 7fbedd600640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 0x7fbebc00b9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009a90 tx=0x7fbebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fb7730d6b0434d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:25.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.299+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0039070 con 0x7fbebc0051c0 2024-04-17T07:29:25.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.299+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed002c6b0 con 0x7fbebc0051c0 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.299+0000 7fbed7400640 1 -- 172.21.15.67:0/3095123622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0009690 con 0x7fbebc0051c0 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 -- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 msgr2=0x7fbebc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 0x7fbebc00b9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009a90 tx=0x7fbebc005830 comp rx=0 tx=0).stop 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 -- 172.21.15.67:0/3095123622 shutdown_connections 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 --2- 172.21.15.67:0/3095123622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbebc0051c0 0x7fbebc00b9c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 -- 172.21.15.67:0/3095123622 >> 172.21.15.67:0/3095123622 conn(0x7fbed806cb60 msgr2=0x7fbed806cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 -- 172.21.15.67:0/3095123622 shutdown_connections 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.300+0000 7fbede000640 1 -- 172.21.15.67:0/3095123622 wait complete. 2024-04-17T07:29:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.302+0000 7fbede000640 1 Processor -- start 2024-04-17T07:29:25.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.302+0000 7fbede000640 1 -- start start 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 0x7fbed8153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed815ae30 0x7fbed81569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbed80b6e10 con 0x7fbed815b2b0 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbed80cee60 con 0x7fbed815ae30 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbede000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbed80cefe0 con 0x7fbebc0051c0 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbedd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbedd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59204/0 (socket says 172.21.15.67:59204) 2024-04-17T07:29:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.303+0000 7fbedd600640 1 -- 172.21.15.67:0/2841356423 learned_addr learned my addr 172.21.15.67:0/2841356423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbed7e00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed815ae30 0x7fbed81569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedcc00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 0x7fbed8153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 -- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 msgr2=0x7fbed8153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 0x7fbed8153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 -- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed815ae30 msgr2=0x7fbed81569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed815ae30 0x7fbed81569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 -- 172.21.15.67:0/2841356423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed80c94e0 con 0x7fbed815b2b0 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedcc00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 0x7fbed8153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.304+0000 7fbedd600640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbed0037040 tx=0x7fbed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.305+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0046070 con 0x7fbed815b2b0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.305+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed003e890 con 0x7fbed815b2b0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.305+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed80c97c0 con 0x7fbed815b2b0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.305+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0039040 con 0x7fbed815b2b0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.305+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed80c9d80 con 0x7fbed815b2b0 2024-04-17T07:29:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.306+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed80755d0 con 0x7fbed815b2b0 2024-04-17T07:29:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.307+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fbed0049070 con 0x7fbed815b2b0 2024-04-17T07:29:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.309+0000 7fbed4c00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 0x7fbeb406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.309+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5227+0+0 (secure 0 0 0) 0x7fbed00ac8e0 con 0x7fbed815b2b0 2024-04-17T07:29:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.309+0000 7fbedcc00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 0x7fbeb406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:25.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.311+0000 7fbedcc00640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 0x7fbeb406ef00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbed8157510 tx=0x7fbec8009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.314+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbed007a790 con 0x7fbed815b2b0 2024-04-17T07:29:25.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:25.519+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbed80d2980 con 0x7fbed815b2b0 2024-04-17T07:29:26.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.055+0000 7fbed4c00640 1 -- 172.21.15.67:0/2841356423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v82) ==== 170+0+0 (secure 0 0 0) 0x7fbed007ea90 con 0x7fbed815b2b0 2024-04-17T07:29:26.055 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:29:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 msgr2=0x7fbeb406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 0x7fbeb406ef00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbed8157510 tx=0x7fbec8009040 comp rx=0 tx=0).stop 2024-04-17T07:29:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 msgr2=0x7fbed8156f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbed0037040 tx=0x7fbed0004730 comp rx=0 tx=0).stop 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 shutdown_connections 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.059+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbeb406ca40 0x7fbeb406ef00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed815b2b0 0x7fbed8156f20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbed815ae30 0x7fbed81569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 --2- 172.21.15.67:0/2841356423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbebc0051c0 0x7fbed8153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 >> 172.21.15.67:0/2841356423 conn(0x7fbed806cb60 msgr2=0x7fbed80b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:26.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 shutdown_connections 2024-04-17T07:29:26.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.060+0000 7fbede000640 1 -- 172.21.15.67:0/2841356423 wait complete. 2024-04-17T07:29:26.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:29:26.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.283+0000 7f0ed1800640 1 Processor -- start 2024-04-17T07:29:26.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.284+0000 7f0ed1800640 1 -- start start 2024-04-17T07:29:26.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc15b860 0x7f0ecc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0ecc06dd10 0x7f0ecc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0ecc071d10 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0ecc071f60 con 0x7f0ecc06dd10 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.285+0000 7f0ed1800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ecc072240 con 0x7f0ecc15b860 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59216/0 (socket says 172.21.15.67:59216) 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 -- 172.21.15.67:0/1864407167 learned_addr learned my addr 172.21.15.67:0/1864407167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ecaa00640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc15b860 0x7f0ecc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ecb400640 1 --2- 172.21.15.67:0/1864407167 >> v2:172.21.15.187:3300/0 conn(0x7f0ecc06dd10 0x7f0ecc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 -- 172.21.15.67:0/1864407167 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc15b860 msgr2=0x7f0ecc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc15b860 0x7f0ecc0d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 -- 172.21.15.67:0/1864407167 >> v2:172.21.15.187:3300/0 conn(0x7f0ecc06dd10 msgr2=0x7f0ecc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 --2- 172.21.15.67:0/1864407167 >> v2:172.21.15.187:3300/0 conn(0x7f0ecc06dd10 0x7f0ecc0ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.286+0000 7f0ec2000640 1 -- 172.21.15.67:0/1864407167 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc072520 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.287+0000 7f0ec2000640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0d1550 tx=0x7f0eb402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1281ef9357280fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:26.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0eca000640 1 -- 172.21.15.67:0/1864407167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4035020 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0eca000640 1 -- 172.21.15.67:0/1864407167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb402d040 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0eca000640 1 -- 172.21.15.67:0/1864407167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4031a80 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 -- 172.21.15.67:0/1864407167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc0ce4d0 msgr2=0x7f0ecc0d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:26.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0d1550 tx=0x7f0eb402b900 comp rx=0 tx=0).stop 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 -- 172.21.15.67:0/1864407167 shutdown_connections 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 --2- 172.21.15.67:0/1864407167 >> v2:172.21.15.187:3300/0 conn(0x7f0ecc06dd10 0x7f0ecc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc0d0960 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.288+0000 7f0ed1800640 1 --2- 172.21.15.67:0/1864407167 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc15b860 0x7f0ecc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.289+0000 7f0ed1800640 1 -- 172.21.15.67:0/1864407167 >> 172.21.15.67:0/1864407167 conn(0x7f0ecc067ee0 msgr2=0x7f0ecc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.289+0000 7f0ed1800640 1 -- 172.21.15.67:0/1864407167 shutdown_connections 2024-04-17T07:29:26.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.289+0000 7f0ed1800640 1 -- 172.21.15.67:0/1864407167 wait complete. 2024-04-17T07:29:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.290+0000 7f0ed1800640 1 Processor -- start 2024-04-17T07:29:26.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.290+0000 7f0ed1800640 1 -- start start 2024-04-17T07:29:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc06dd10 0x7f0ecc14aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc15bc40 con 0x7f0ecc15b860 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc072ae0 con 0x7f0ecc0ce4d0 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.291+0000 7f0ed1800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ecc06f380 con 0x7f0ecc06dd10 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59228/0 (socket says 172.21.15.67:59228) 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 -- 172.21.15.67:0/3366633522 learned_addr learned my addr 172.21.15.67:0/3366633522 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecaa00640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc06dd10 0x7f0ecc14aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ec2000640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 -- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc06dd10 msgr2=0x7f0ecc14aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc06dd10 0x7f0ecc14aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 -- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 msgr2=0x7f0ecc144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 -- 172.21.15.67:0/3366633522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc147360 con 0x7f0ecc15b860 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ec2000640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:26.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.292+0000 7f0ecb400640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4007e40 tx=0x7f0ec400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:26.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.293+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4014620 con 0x7f0ecc15b860 2024-04-17T07:29:26.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.293+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec4019070 con 0x7f0ecc15b860 2024-04-17T07:29:26.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.293+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4014a20 con 0x7f0ecc15b860 2024-04-17T07:29:26.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.293+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ecc144db0 con 0x7f0ecc15b860 2024-04-17T07:29:26.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.293+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ecc1478e0 con 0x7f0ecc15b860 2024-04-17T07:29:26.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.294+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ecc068ca0 con 0x7f0ecc15b860 2024-04-17T07:29:26.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.297+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f0ec4014bc0 con 0x7f0ecc15b860 2024-04-17T07:29:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.297+0000 7f0ec3400640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 0x7f0ea806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:26.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.298+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5609+0+0 (secure 0 0 0) 0x7f0ec408d3b0 con 0x7f0ecc15b860 2024-04-17T07:29:26.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.298+0000 7f0ecaa00640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 0x7f0ea806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:26.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.299+0000 7f0ecaa00640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 0x7f0ea806ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:26.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.302+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0ec405b810 con 0x7f0ecc15b860 2024-04-17T07:29:26.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:26.550+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0ecc15abe0 con 0x7f0ecc15b860 2024-04-17T07:29:27.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.083+0000 7f0ec3400640 1 -- 172.21.15.67:0/3366633522 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) ==== 172+0+0 (secure 0 0 0) 0x7f0ec405b110 con 0x7f0ecc15b860 2024-04-17T07:29:27.082 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 8 and data pool 9 2024-04-17T07:29:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 msgr2=0x7f0ea806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 0x7f0ea806ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc008040 comp rx=0 tx=0).stop 2024-04-17T07:29:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 msgr2=0x7f0ecc146e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4007e40 tx=0x7f0ec400e4b0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 shutdown_connections 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0ea806ca40 0x7f0ea806ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ecc15b860 0x7f0ecc146e20 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ecc0ce4d0 0x7f0ecc144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 --2- 172.21.15.67:0/3366633522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ecc06dd10 0x7f0ecc14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.086+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 >> 172.21.15.67:0/3366633522 conn(0x7f0ecc067ee0 msgr2=0x7f0ecc0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:27.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.087+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 shutdown_connections 2024-04-17T07:29:27.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.087+0000 7f0ed1800640 1 -- 172.21.15.67:0/3366633522 wait complete. 2024-04-17T07:29:27.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:27.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.319+0000 7f222e600640 1 Processor -- start 2024-04-17T07:29:27.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.319+0000 7f222e600640 1 -- start start 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.320+0000 7f222e600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f222806b8c0 0x7f22280cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.320+0000 7f222e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f222806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22280cd4a0 con 0x7f2228141ed0 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222e600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f222815ce30 con 0x7f222806b8c0 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f22281607f0 con 0x7f2228160280 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2228141ed0 0x7f222815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58454/0 (socket says 172.21.15.67:58454) 2024-04-17T07:29:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222dc00640 1 -- 172.21.15.67:0/3296030835 learned_addr learned my addr 172.21.15.67:0/3296030835 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f221fe00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f222806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.321+0000 7f222d200640 1 --2- 172.21.15.67:0/3296030835 >> v2:172.21.15.187:3300/0 conn(0x7f222806b8c0 0x7f22280cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630985139 0 0) 0x7f22280cd4a0 con 0x7f2228141ed0 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2208003520 con 0x7f2228141ed0 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2965922803 0 0) 0x7f2208003520 con 0x7f2228141ed0 2024-04-17T07:29:27.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 msgr2=0x7f222806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f222806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 >> v2:172.21.15.187:3300/0 conn(0x7f222806b8c0 msgr2=0x7f22280cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 --2- 172.21.15.67:0/3296030835 >> v2:172.21.15.187:3300/0 conn(0x7f222806b8c0 0x7f22280cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2228160ad0 con 0x7f2228141ed0 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f22200019f0 con 0x7f2228141ed0 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 >> v1:172.21.15.67:6789/0 conn(0x7f2228141ed0 legacy=0x7f222815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2208003a20 0x7f2208003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208004800 0x7f2208004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f22080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f222815ce30 con 0x7f2208003a20 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f22281607f0 con 0x7f2208004800 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f220800b9c0 con 0x7f2228160280 2024-04-17T07:29:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.322+0000 7f222d200640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2208003a20 0x7f2208003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222dc00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f22080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222d200640 1 -- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 msgr2=0x7f22080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222d200640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f22080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222d200640 1 -- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208004800 msgr2=0x7f2208004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222d200640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208004800 0x7f2208004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222d200640 1 -- 172.21.15.67:0/3296030835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2208003520 con 0x7f2208003a20 2024-04-17T07:29:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.323+0000 7f222dc00640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2228160280 0x7f22080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.324+0000 7f222d200640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2208003a20 0x7f2208003e40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2210009af0 tx=0x7f2208005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72556c3c4315fc83 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.325+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210039070 con 0x7f2208003a20 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.325+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221002c710 con 0x7f2208003a20 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.325+0000 7f2227e00640 1 -- 172.21.15.67:0/3296030835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210009690 con 0x7f2208003a20 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.325+0000 7f222e600640 1 -- 172.21.15.67:0/3296030835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2208003a20 msgr2=0x7f2208003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.325+0000 7f222e600640 1 --2- 172.21.15.67:0/3296030835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2208003a20 0x7f2208003e40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2210009af0 tx=0x7f2208005830 comp rx=0 tx=0).stop 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.326+0000 7f222dc00640 1 -- 172.21.15.67:0/3296030835 reap_dead start 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.326+0000 7f222e600640 1 -- 172.21.15.67:0/3296030835 shutdown_connections 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.326+0000 7f222e600640 1 -- 172.21.15.67:0/3296030835 >> 172.21.15.67:0/3296030835 conn(0x7f2228067d10 msgr2=0x7f22280b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.326+0000 7f222e600640 1 -- 172.21.15.67:0/3296030835 shutdown_connections 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.326+0000 7f222e600640 1 -- 172.21.15.67:0/3296030835 wait complete. 2024-04-17T07:29:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.328+0000 7f222e600640 1 Processor -- start 2024-04-17T07:29:27.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.328+0000 7f222e600640 1 -- start start 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 0x7f22280b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22281561d0 0x7f2228143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2228161090 con 0x7f2228155d50 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f22280cddf0 con 0x7f2208003a20 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f222813f9b0 con 0x7f22281561d0 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f221fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f221fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59246/0 (socket says 172.21.15.67:59246) 2024-04-17T07:29:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f221fe00640 1 -- 172.21.15.67:0/3269658530 learned_addr learned my addr 172.21.15.67:0/3269658530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.329+0000 7f222d200640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 0x7f22280b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f222dc00640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22281561d0 0x7f2228143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 -- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22281561d0 msgr2=0x7f2228143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22281561d0 0x7f2228143840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 -- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 msgr2=0x7f22280b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 0x7f22280b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 -- 172.21.15.67:0/3269658530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22280b4290 con 0x7f2228155d50 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f222d200640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 0x7f22280b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f221fe00640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f221800e7a0 tx=0x7f221800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218017070 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221800c940 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221800cd10 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22280b4570 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.330+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22280b4b30 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.332+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2228068a10 con 0x7f2228155d50 2024-04-17T07:29:27.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.333+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f221801a070 con 0x7f2228155d50 2024-04-17T07:29:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.334+0000 7f2225600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 0x7f220406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.334+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7f221808bfa0 con 0x7f2228155d50 2024-04-17T07:29:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.335+0000 7f222d200640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 0x7f220406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.336+0000 7f222d200640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 0x7f220406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2210036000 tx=0x7f2210030000 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.339+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f221805a220 con 0x7f2228155d50 2024-04-17T07:29:27.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.548+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22280b1900 con 0x7f2228155d50 2024-04-17T07:29:27.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.550+0000 7f2225600640 1 -- 172.21.15.67:0/3269658530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+16630 (secure 0 0 0) 0x7f2218059b20 con 0x7f2228155d50 2024-04-17T07:29:27.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:27.549 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":83,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:26.551558+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 msgr2=0x7f220406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 0x7f220406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2210036000 tx=0x7f2210030000 comp rx=0 tx=0).stop 2024-04-17T07:29:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 msgr2=0x7f22280b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f221800e7a0 tx=0x7f221800ec60 comp rx=0 tx=0).stop 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 shutdown_connections 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f220406ca10 0x7f220406eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f22281561d0 0x7f2228143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2228155d50 0x7f22280b38b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 --2- 172.21.15.67:0/3269658530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2208003a20 0x7f22280b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 >> 172.21.15.67:0/3269658530 conn(0x7f2228067d10 msgr2=0x7f22281567d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 shutdown_connections 2024-04-17T07:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.553+0000 7f222e600640 1 -- 172.21.15.67:0/3269658530 wait complete. 2024-04-17T07:29:27.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.790+0000 7fc0e7200640 1 Processor -- start 2024-04-17T07:29:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.790+0000 7fc0e7200640 1 -- start start 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0e8160280 0x7fc0e806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e80cd4a0 con 0x7fc0e806b8c0 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e81607f0 con 0x7fc0e8141ed0 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e815ce30 con 0x7fc0e8160280 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc0e806b8c0 0x7fc0e80cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58468/0 (socket says 172.21.15.67:58468) 2024-04-17T07:29:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.792+0000 7fc0e5e00640 1 -- 172.21.15.67:0/3453459137 learned_addr learned my addr 172.21.15.67:0/3453459137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346443641 0 0) 0x7fc0e80cd4a0 con 0x7fc0e806b8c0 2024-04-17T07:29:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0cc003520 con 0x7fc0e806b8c0 2024-04-17T07:29:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/3453459137 >> v2:172.21.15.187:3300/0 conn(0x7fc0e8160280 0x7fc0e806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1857679780 0 0) 0x7fc0cc003520 con 0x7fc0e806b8c0 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 >> v2:172.21.15.187:3300/0 conn(0x7fc0e8160280 msgr2=0x7fc0e806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 --2- 172.21.15.67:0/3453459137 >> v2:172.21.15.187:3300/0 conn(0x7fc0e8160280 0x7fc0e806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 >> v1:172.21.15.187:6789/0 conn(0x7fc0e8141ed0 legacy=0x7fc0e815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e8160ad0 con 0x7fc0e806b8c0 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc0d8001670 con 0x7fc0e806b8c0 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 >> v1:172.21.15.67:6789/0 conn(0x7fc0e806b8c0 legacy=0x7fc0e80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0cc003a20 0x7fc0cc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 0x7fc0cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc005230 0x7fc0cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e815ce30 con 0x7fc0cc004800 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc00c190 con 0x7fc0cc005230 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.793+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0cc00c390 con 0x7fc0cc003a20 2024-04-17T07:29:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 0x7fc0cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0e6800640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc005230 0x7fc0cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0cc003a20 0x7fc0cc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 -- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0cc003a20 msgr2=0x7fc0cc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0cc003a20 0x7fc0cc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 -- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc005230 msgr2=0x7fc0cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc005230 0x7fc0cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0dfe00640 1 -- 172.21.15.67:0/3453459137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0cc003520 con 0x7fc0cc004800 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0e6800640 1 -- 172.21.15.67:0/3453459137 reap_dead start 2024-04-17T07:29:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.794+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0cc003a20 0x7fc0cc003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.795+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 0x7fc0cc004c80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000ba30 tx=0x7fc0cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2197a7865090a47d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.795+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0036020 con 0x7fc0cc004800 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.795+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d0032640 con 0x7fc0cc004800 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.795+0000 7fc0e5400640 1 -- 172.21.15.67:0/3453459137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000b600 con 0x7fc0cc004800 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 -- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 msgr2=0x7fc0cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 0x7fc0cc004c80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000ba30 tx=0x7fc0cc005830 comp rx=0 tx=0).stop 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 -- 172.21.15.67:0/3453459137 shutdown_connections 2024-04-17T07:29:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 --2- 172.21.15.67:0/3453459137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0cc004800 0x7fc0cc004c80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 -- 172.21.15.67:0/3453459137 >> 172.21.15.67:0/3453459137 conn(0x7fc0e8067d10 msgr2=0x7fc0e80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 -- 172.21.15.67:0/3453459137 shutdown_connections 2024-04-17T07:29:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.796+0000 7fc0e7200640 1 -- 172.21.15.67:0/3453459137 wait complete. 2024-04-17T07:29:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.798+0000 7fc0e7200640 1 Processor -- start 2024-04-17T07:29:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.798+0000 7fc0e7200640 1 -- start start 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 0x7fc0e815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e80b4be0 0x7fc0e81439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e813f9b0 con 0x7fc0e81f57a0 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e8161090 con 0x7fc0cc004800 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e813fb30 con 0x7fc0e80b4be0 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59260/0 (socket says 172.21.15.67:59260) 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.799+0000 7fc0dfe00640 1 -- 172.21.15.67:0/4032641659 learned_addr learned my addr 172.21.15.67:0/4032641659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0e6800640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e80b4be0 0x7fc0e81439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 0x7fc0e815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 -- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e80b4be0 msgr2=0x7fc0e81439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e80b4be0 0x7fc0e81439d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 -- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 msgr2=0x7fc0e815e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 0x7fc0e815e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 -- 172.21.15.67:0/4032641659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e815ebe0 con 0x7fc0e81f57a0 2024-04-17T07:29:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 0x7fc0e815e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.800+0000 7fc0dfe00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002bf90 tx=0x7fc0d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.801+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0041070 con 0x7fc0e81f57a0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.801+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d0030a50 con 0x7fc0e81f57a0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.801+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0036040 con 0x7fc0e81f57a0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.801+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e80b3b40 con 0x7fc0e81f57a0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.801+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e80b4100 con 0x7fc0e81f57a0 2024-04-17T07:29:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.802+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e8068a10 con 0x7fc0e81f57a0 2024-04-17T07:29:27.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.803+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fc0d004c060 con 0x7fc0e81f57a0 2024-04-17T07:29:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.804+0000 7fc0de000640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 0x7fc0c406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.804+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5709+0+0 (secure 0 0 0) 0x7fc0d00ada80 con 0x7fc0e81f57a0 2024-04-17T07:29:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.805+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 0x7fc0c406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.806+0000 7fc0e5e00640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 0x7fc0c406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80516a0 tx=0x7fc0d8001470 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:27.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:27.808+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc0d007bda0 con 0x7fc0e81f57a0 2024-04-17T07:29:28.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.013+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0e8074ee0 con 0x7fc0e81f57a0 2024-04-17T07:29:28.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.017+0000 7fc0de000640 1 -- 172.21.15.67:0/4032641659 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+16630 (secure 0 0 0) 0x7fc0d007b6a0 con 0x7fc0e81f57a0 2024-04-17T07:29:28.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:28.016 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":83,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:26.551558+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 msgr2=0x7fc0c406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 0x7fc0c406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80516a0 tx=0x7fc0d8001470 comp rx=0 tx=0).stop 2024-04-17T07:29:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 msgr2=0x7fc0e8142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002bf90 tx=0x7fc0d0004730 comp rx=0 tx=0).stop 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 shutdown_connections 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc0c406ca40 0x7fc0c406ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e80b4be0 0x7fc0e81439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0e81f57a0 0x7fc0e8142fc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 --2- 172.21.15.67:0/4032641659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0cc004800 0x7fc0e815e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.019+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 >> 172.21.15.67:0/4032641659 conn(0x7fc0e8067d10 msgr2=0x7fc0e80cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.020+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 shutdown_connections 2024-04-17T07:29:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.020+0000 7fc0e7200640 1 -- 172.21.15.67:0/4032641659 wait complete. 2024-04-17T07:29:28.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:29:28.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:29:28.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:29:28.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.262+0000 7fa672e00640 1 Processor -- start 2024-04-17T07:29:28.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.262+0000 7fa672e00640 1 -- start start 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa66c0cdce0 0x7fa66c0d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa66c0ce120 0x7fa66c0d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c1626d0 con 0x7fa66c0ce120 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c06b8c0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c06bb80 con 0x7fa66c070c30 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa66c0ce120 0x7fa66c0d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa66c0ce120 0x7fa66c0d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59262/0 (socket says 172.21.15.67:59262) 2024-04-17T07:29:28.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa672400640 1 -- 172.21.15.67:0/1851260740 learned_addr learned my addr 172.21.15.67:0/1851260740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:28.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa671000640 1 --2- 172.21.15.67:0/1851260740 >> v2:172.21.15.187:3300/0 conn(0x7fa66c0cdce0 0x7fa66c0d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.264+0000 7fa671a00640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa672400640 1 -- 172.21.15.67:0/1851260740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 msgr2=0x7fa66c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa672400640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c0b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa672400640 1 -- 172.21.15.67:0/1851260740 >> v2:172.21.15.187:3300/0 conn(0x7fa66c0cdce0 msgr2=0x7fa66c0d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa672400640 1 --2- 172.21.15.67:0/1851260740 >> v2:172.21.15.187:3300/0 conn(0x7fa66c0cdce0 0x7fa66c0d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa672400640 1 -- 172.21.15.67:0/1851260740 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c06be60 con 0x7fa66c0ce120 2024-04-17T07:29:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.265+0000 7fa671a00640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.266+0000 7fa672400640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0ce120 0x7fa66c0d0cb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa66c071660 tx=0x7fa66402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98152deecca5cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.266+0000 7fa66be00640 1 -- 172.21.15.67:0/1851260740 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66402cb00 con 0x7fa66c0ce120 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.266+0000 7fa66be00640 1 -- 172.21.15.67:0/1851260740 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66402cca0 con 0x7fa66c0ce120 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.266+0000 7fa66be00640 1 -- 172.21.15.67:0/1851260740 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664034610 con 0x7fa66c0ce120 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 -- 172.21.15.67:0/1851260740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0ce120 msgr2=0x7fa66c0d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0ce120 0x7fa66c0d0cb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa66c071660 tx=0x7fa66402bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 -- 172.21.15.67:0/1851260740 shutdown_connections 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0ce120 0x7fa66c0d0cb0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 --2- 172.21.15.67:0/1851260740 >> v2:172.21.15.187:3300/0 conn(0x7fa66c0cdce0 0x7fa66c0d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 --2- 172.21.15.67:0/1851260740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 -- 172.21.15.67:0/1851260740 >> 172.21.15.67:0/1851260740 conn(0x7fa66c09aa60 msgr2=0x7fa66c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 -- 172.21.15.67:0/1851260740 shutdown_connections 2024-04-17T07:29:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.267+0000 7fa672e00640 1 -- 172.21.15.67:0/1851260740 wait complete. 2024-04-17T07:29:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.269+0000 7fa672e00640 1 Processor -- start 2024-04-17T07:29:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.269+0000 7fa672e00640 1 -- start start 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa66c0ce120 0x7fa66c0af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c0b3c30 con 0x7fa66c0cdce0 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c06c420 con 0x7fa66c0ce120 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa672e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c0cd820 con 0x7fa66c070c30 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa671000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.270+0000 7fa671000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59268/0 (socket says 172.21.15.67:59268) 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 -- 172.21.15.67:0/1247916421 learned_addr learned my addr 172.21.15.67:0/1247916421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671a00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa672400640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa66c0ce120 0x7fa66c0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 -- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 msgr2=0x7fa66c15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c15d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 -- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa66c0ce120 msgr2=0x7fa66c0af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa66c0ce120 0x7fa66c0af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 -- 172.21.15.67:0/1247916421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c0c94c0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.271+0000 7fa671000640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa65800d800 tx=0x7fa65800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.272+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65800b6c0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.272+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa658019070 con 0x7fa66c0cdce0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.272+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65800ba80 con 0x7fa66c0cdce0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.272+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c15dde0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.272+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c0b4450 con 0x7fa66c0cdce0 2024-04-17T07:29:28.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.273+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66c0755d0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.274+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fa65800bc20 con 0x7fa66c0cdce0 2024-04-17T07:29:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.275+0000 7fa669600640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 0x7fa64c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.275+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fa65808c190 con 0x7fa66c0cdce0 2024-04-17T07:29:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.276+0000 7fa671a00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 0x7fa64c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.277+0000 7fa671a00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 0x7fa64c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa660006fd0 tx=0x7fa660008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.280+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa65805a4b0 con 0x7fa66c0cdce0 2024-04-17T07:29:28.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:28.525+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa66c0ce7d0 con 0x7fa66c0cdce0 2024-04-17T07:29:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.135+0000 7fa669600640 1 -- 172.21.15.67:0/1247916421 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7fa658059db0 con 0x7fa66c0cdce0 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 msgr2=0x7fa64c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 0x7fa64c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa660006fd0 tx=0x7fa660008040 comp rx=0 tx=0).stop 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 msgr2=0x7fa66c0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa65800d800 tx=0x7fa65800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 shutdown_connections 2024-04-17T07:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fa64c06ca10 0x7fa64c06eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa66c0ce120 0x7fa66c0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa66c0cdce0 0x7fa66c0af440 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 --2- 172.21.15.67:0/1247916421 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa66c070c30 0x7fa66c15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 >> 172.21.15.67:0/1247916421 conn(0x7fa66c09aa60 msgr2=0x7fa66c161e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 shutdown_connections 2024-04-17T07:29:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.140+0000 7fa672e00640 1 -- 172.21.15.67:0/1247916421 wait complete. 2024-04-17T07:29:29.160 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:29:29.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:29:29.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.437+0000 7f2c48a00640 1 Processor -- start 2024-04-17T07:29:29.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.437+0000 7f2c48a00640 1 -- start start 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.439+0000 7f2c48a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c440b41a0 0x7f2c440b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.439+0000 7f2c48a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c440b4580 0x7f2c44162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.439+0000 7f2c48a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c44166a60 con 0x7f2c4406f800 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.439+0000 7f2c48a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c44166810 con 0x7f2c440b41a0 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.439+0000 7f2c48a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c44166d40 con 0x7f2c440b4580 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c440b41a0 0x7f2c440b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c440b41a0 0x7f2c440b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59278/0 (socket says 172.21.15.67:59278) 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 -- 172.21.15.67:0/2638115128 learned_addr learned my addr 172.21.15.67:0/2638115128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c43200640 1 --1- 172.21.15.67:0/2638115128 >> v1:172.21.15.67:6789/0 conn(0x7f2c4406f800 0x7f2c44071cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58476/0 (socket says 172.21.15.67:58476) 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c41e00640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c440b4580 0x7f2c44162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 -- 172.21.15.67:0/2638115128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c440b4580 msgr2=0x7f2c44162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c440b4580 0x7f2c44162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 -- 172.21.15.67:0/2638115128 >> v1:172.21.15.67:6789/0 conn(0x7f2c4406f800 legacy=0x7f2c44071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.440+0000 7f2c42800640 1 -- 172.21.15.67:0/2638115128 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c44167020 con 0x7f2c440b41a0 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.441+0000 7f2c42800640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c440b3490 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009870 tx=0x7f2c3402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8180a306c29dc3c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c41400640 1 -- 172.21.15.67:0/2638115128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34034070 con 0x7f2c440b41a0 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c41400640 1 -- 172.21.15.67:0/2638115128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c34030840 con 0x7f2c440b41a0 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c41400640 1 -- 172.21.15.67:0/2638115128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34030c10 con 0x7f2c440b41a0 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c48a00640 1 -- 172.21.15.67:0/2638115128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 msgr2=0x7f2c440b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c48a00640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c440b3490 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009870 tx=0x7f2c3402b980 comp rx=0 tx=0).stop 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.442+0000 7f2c48a00640 1 -- 172.21.15.67:0/2638115128 shutdown_connections 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.443+0000 7f2c48a00640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c440b4580 0x7f2c44162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.443+0000 7f2c48a00640 1 --2- 172.21.15.67:0/2638115128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c440b3490 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.443+0000 7f2c48a00640 1 -- 172.21.15.67:0/2638115128 >> 172.21.15.67:0/2638115128 conn(0x7f2c440b2190 msgr2=0x7f2c440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.443+0000 7f2c48a00640 1 -- 172.21.15.67:0/2638115128 shutdown_connections 2024-04-17T07:29:29.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.443+0000 7f2c48a00640 1 -- 172.21.15.67:0/2638115128 wait complete. 2024-04-17T07:29:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.445+0000 7f2c48a00640 1 Processor -- start 2024-04-17T07:29:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.445+0000 7f2c48a00640 1 -- start start 2024-04-17T07:29:29.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4406f800 0x7f2c440ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c440b4580 0x7f2c441fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4406eec0 con 0x7f2c440b41a0 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4406ed40 con 0x7f2c440b4580 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.446+0000 7f2c48a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c44162bd0 con 0x7f2c4406f800 2024-04-17T07:29:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59292/0 (socket says 172.21.15.67:59292) 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 -- 172.21.15.67:0/28379776 learned_addr learned my addr 172.21.15.67:0/28379776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c42800640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4406f800 0x7f2c440ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c43200640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c440b4580 0x7f2c441fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 -- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4406f800 msgr2=0x7f2c440ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4406f800 0x7f2c440ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 -- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c440b4580 msgr2=0x7f2c441fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c440b4580 0x7f2c441fb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 -- 172.21.15.67:0/28379776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c44164f20 con 0x7f2c440b41a0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.447+0000 7f2c41e00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00d5f0 tx=0x7f2c2c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.448+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0125f0 con 0x7f2c440b41a0 2024-04-17T07:29:29.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.448+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2c019070 con 0x7f2c440b41a0 2024-04-17T07:29:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.448+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0129d0 con 0x7f2c440b41a0 2024-04-17T07:29:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.448+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c44165200 con 0x7f2c440b41a0 2024-04-17T07:29:29.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.448+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c441657c0 con 0x7f2c440b41a0 2024-04-17T07:29:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.449+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c44068a10 con 0x7f2c440b41a0 2024-04-17T07:29:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.450+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f2c2c012b70 con 0x7f2c440b41a0 2024-04-17T07:29:29.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.451+0000 7f2c3aa00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 0x7f2c2006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.451+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f2c2c08c090 con 0x7f2c440b41a0 2024-04-17T07:29:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.452+0000 7f2c42800640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 0x7f2c2006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:29.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.453+0000 7f2c42800640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 0x7f2c2006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c34036000 tx=0x7f2c3403b040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.456+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c2c05a440 con 0x7f2c440b41a0 2024-04-17T07:29:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:29.711+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2c4406c7b0 con 0x7f2c440b41a0 2024-04-17T07:29:30.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.138+0000 7f2c3aa00640 1 -- 172.21.15.67:0/28379776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f2c2c059d40 con 0x7f2c440b41a0 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 msgr2=0x7f2c2006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 0x7f2c2006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c34036000 tx=0x7f2c3403b040 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 msgr2=0x7f2c44164540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00d5f0 tx=0x7f2c2c00dab0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 shutdown_connections 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2c2006ca10 0x7f2c2006eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c440b4580 0x7f2c441fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c440b41a0 0x7f2c44164540 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 --2- 172.21.15.67:0/28379776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4406f800 0x7f2c440ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 >> 172.21.15.67:0/28379776 conn(0x7f2c440b2190 msgr2=0x7f2c440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 shutdown_connections 2024-04-17T07:29:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.143+0000 7f2c48a00640 1 -- 172.21.15.67:0/28379776 wait complete. 2024-04-17T07:29:30.165 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:29:30.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.392+0000 7f520da00640 1 Processor -- start 2024-04-17T07:29:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.392+0000 7f520da00640 1 -- start start 2024-04-17T07:29:30.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f520da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f520815c5a0 0x7f520815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f520da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 0x7f52080af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f520da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f520815f440 con 0x7f52080b46c0 2024-04-17T07:29:30.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f520da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f52080af0a0 con 0x7f520815c5a0 2024-04-17T07:29:30.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f520da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f520815f720 con 0x7f52080b4210 2024-04-17T07:29:30.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f5207e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f520815c5a0 0x7f520815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f5207e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f520815c5a0 0x7f520815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59300/0 (socket says 172.21.15.67:59300) 2024-04-17T07:29:30.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.394+0000 7f5207e00640 1 -- 172.21.15.67:0/1089672220 learned_addr learned my addr 172.21.15.67:0/1089672220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:30.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f51ff400640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 0x7f52080af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f5207e00640 1 -- 172.21.15.67:0/1089672220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 msgr2=0x7f52080af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f5207e00640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 0x7f52080af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f5207e00640 1 -- 172.21.15.67:0/1089672220 >> v1:172.21.15.187:6789/0 conn(0x7f52080b46c0 legacy=0x7f5208071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f5207e00640 1 -- 172.21.15.67:0/1089672220 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520815fa00 con 0x7f520815c5a0 2024-04-17T07:29:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.395+0000 7f51ff400640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 0x7f52080af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:30.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.396+0000 7f5207e00640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520815c5a0 0x7f520815ee60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098a0 tx=0x7f51f802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df6ea2203307acf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.396+0000 7f5207400640 1 -- 172.21.15.67:0/1089672220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8038070 con 0x7f520815c5a0 2024-04-17T07:29:30.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.396+0000 7f5207400640 1 -- 172.21.15.67:0/1089672220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f8031520 con 0x7f520815c5a0 2024-04-17T07:29:30.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.396+0000 7f5207400640 1 -- 172.21.15.67:0/1089672220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f80318f0 con 0x7f520815c5a0 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 -- 172.21.15.67:0/1089672220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520815c5a0 msgr2=0x7f520815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520815c5a0 0x7f520815ee60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098a0 tx=0x7f51f802b7c0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 -- 172.21.15.67:0/1089672220 shutdown_connections 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52080b4210 0x7f52080af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 --2- 172.21.15.67:0/1089672220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520815c5a0 0x7f520815ee60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 -- 172.21.15.67:0/1089672220 >> 172.21.15.67:0/1089672220 conn(0x7f5208067d10 msgr2=0x7f52080b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 -- 172.21.15.67:0/1089672220 shutdown_connections 2024-04-17T07:29:30.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.397+0000 7f520da00640 1 -- 172.21.15.67:0/1089672220 wait complete. 2024-04-17T07:29:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.399+0000 7f520da00640 1 Processor -- start 2024-04-17T07:29:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.399+0000 7f520da00640 1 -- start start 2024-04-17T07:29:30.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f520815c5a0 0x7f52080d0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 0x7f52081f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52080adc30 con 0x7f52080b4210 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f52080adab0 con 0x7f52081f44c0 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.400+0000 7f520da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f520806e670 con 0x7f520815c5a0 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59310/0 (socket says 172.21.15.67:59310) 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 -- 172.21.15.67:0/3368153271 learned_addr learned my addr 172.21.15.67:0/3368153271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f520d000640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 0x7f52081f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f51ff400640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f520815c5a0 0x7f52080d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 -- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f520815c5a0 msgr2=0x7f52080d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f520815c5a0 0x7f52080d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 -- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 msgr2=0x7f52081f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 0x7f52081f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f5207e00640 1 -- 172.21.15.67:0/3368153271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52081f7a40 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.401+0000 7f520d000640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 0x7f52081f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f5207e00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098a0 tx=0x7f51f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8042070 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f803d870 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8038040 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52081f4f80 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.402+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52081f7d00 con 0x7f52080b4210 2024-04-17T07:29:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.403+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5208068a10 con 0x7f52080b4210 2024-04-17T07:29:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.404+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f51f8045070 con 0x7f52080b4210 2024-04-17T07:29:30.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.405+0000 7f5204c00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 0x7f51e406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.405+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f51f80ad980 con 0x7f52080b4210 2024-04-17T07:29:30.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.406+0000 7f51ff400640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 0x7f51e406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.407+0000 7f51ff400640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 0x7f51e406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520815bdf0 tx=0x7f51f0008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.410+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f51f807bcb0 con 0x7f52080b4210 2024-04-17T07:29:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.656+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52080b3dd0 con 0x7f52080b4210 2024-04-17T07:29:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.658+0000 7f5204c00640 1 -- 172.21.15.67:0/3368153271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7f51f807b5b0 con 0x7f52080b4210 2024-04-17T07:29:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:30.657 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 msgr2=0x7f51e406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 0x7f51e406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520815bdf0 tx=0x7f51f0008040 comp rx=0 tx=0).stop 2024-04-17T07:29:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 msgr2=0x7f52081f8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098a0 tx=0x7f51f80046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 shutdown_connections 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f51e406ca10 0x7f51e406eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52081f44c0 0x7f52081f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f520815c5a0 0x7f52080d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 --2- 172.21.15.67:0/3368153271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52080b4210 0x7f52081f8500 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 >> 172.21.15.67:0/3368153271 conn(0x7f5208067d10 msgr2=0x7f52080b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 shutdown_connections 2024-04-17T07:29:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.661+0000 7f520da00640 1 -- 172.21.15.67:0/3368153271 wait complete. 2024-04-17T07:29:30.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:30.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.893+0000 7fbb93200640 1 Processor -- start 2024-04-17T07:29:30.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.893+0000 7fbb93200640 1 -- start start 2024-04-17T07:29:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.894+0000 7fbb93200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb8c06b8c0 0x7fbb8c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.894+0000 7fbb93200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8c160280 0x7fbb8c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.894+0000 7fbb93200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb8c0cd4a0 con 0x7fbb8c141ed0 2024-04-17T07:29:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.894+0000 7fbb93200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8c15ce30 con 0x7fbb8c160280 2024-04-17T07:29:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.894+0000 7fbb93200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8c1607f0 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8c160280 0x7fbb8c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb8c160280 0x7fbb8c06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59320/0 (socket says 172.21.15.67:59320) 2024-04-17T07:29:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 -- 172.21.15.67:0/1804949359 learned_addr learned my addr 172.21.15.67:0/1804949359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91e00640 1 --2- 172.21.15.67:0/1804949359 >> v2:172.21.15.187:3300/0 conn(0x7fbb8c06b8c0 0x7fbb8c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 -- 172.21.15.67:0/1804949359 >> v2:172.21.15.187:3300/0 conn(0x7fbb8c06b8c0 msgr2=0x7fbb8c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 --2- 172.21.15.67:0/1804949359 >> v2:172.21.15.187:3300/0 conn(0x7fbb8c06b8c0 0x7fbb8c0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 -- 172.21.15.67:0/1804949359 >> v1:172.21.15.187:6789/0 conn(0x7fbb8c141ed0 legacy=0x7fbb8c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.895+0000 7fbb91400640 1 -- 172.21.15.67:0/1804949359 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c160ad0 con 0x7fbb8c160280 2024-04-17T07:29:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb91400640 1 --2- 172.21.15.67:0/1804949359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c160280 0x7fbb8c06bca0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0b3210 tx=0x7fbb7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2f43deab652969 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb90a00640 1 -- 172.21.15.67:0/1804949359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c034070 con 0x7fbb8c160280 2024-04-17T07:29:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb90a00640 1 -- 172.21.15.67:0/1804949359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7c030860 con 0x7fbb8c160280 2024-04-17T07:29:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb90a00640 1 -- 172.21.15.67:0/1804949359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c030c30 con 0x7fbb8c160280 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb93200640 1 -- 172.21.15.67:0/1804949359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c160280 msgr2=0x7fbb8c06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.897+0000 7fbb93200640 1 --2- 172.21.15.67:0/1804949359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c160280 0x7fbb8c06bca0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0b3210 tx=0x7fbb7c02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 -- 172.21.15.67:0/1804949359 shutdown_connections 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 --2- 172.21.15.67:0/1804949359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c160280 0x7fbb8c06bca0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 --2- 172.21.15.67:0/1804949359 >> v2:172.21.15.187:3300/0 conn(0x7fbb8c06b8c0 0x7fbb8c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 -- 172.21.15.67:0/1804949359 >> 172.21.15.67:0/1804949359 conn(0x7fbb8c067d10 msgr2=0x7fbb8c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 -- 172.21.15.67:0/1804949359 shutdown_connections 2024-04-17T07:29:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.898+0000 7fbb93200640 1 -- 172.21.15.67:0/1804949359 wait complete. 2024-04-17T07:29:30.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.899+0000 7fbb93200640 1 Processor -- start 2024-04-17T07:29:30.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.900+0000 7fbb93200640 1 -- start start 2024-04-17T07:29:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 0x7fbb8c142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb8c160280 0x7fbb8c1f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c06c1e0 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c0cddf0 con 0x7fbb8c141ed0 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb93200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8c161090 con 0x7fbb8c160280 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59336/0 (socket says 172.21.15.67:59336) 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 -- 172.21.15.67:0/1864854444 learned_addr learned my addr 172.21.15.67:0/1864854444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91400640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 0x7fbb8c142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb92800640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb8c160280 0x7fbb8c1f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 -- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb8c160280 msgr2=0x7fbb8c1f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb8c160280 0x7fbb8c1f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 -- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 msgr2=0x7fbb8c142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 0x7fbb8c142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91e00640 1 -- 172.21.15.67:0/1864854444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c143500 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.901+0000 7fbb91400640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 0x7fbb8c142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb91e00640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000e9b0 tx=0x7fbb8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80017070 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8000c9a0 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000cd90 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8c1f7920 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.902+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb8c0b45f0 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.903+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb8c068a10 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.905+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fbb8001a070 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.906+0000 7fbb86000640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 0x7fbb7006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.906+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fbb8008bc40 con 0x7fbb8c06b8c0 2024-04-17T07:29:30.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.907+0000 7fbb91400640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 0x7fbb7006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.908+0000 7fbb91400640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 0x7fbb7006ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c036000 tx=0x7fbb7c008030 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:30.911+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb80059ff0 con 0x7fbb8c06b8c0 2024-04-17T07:29:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.119+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb8c0d04d0 con 0x7fbb8c06b8c0 2024-04-17T07:29:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.120+0000 7fbb86000640 1 -- 172.21.15.67:0/1864854444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16617 (secure 0 0 0) 0x7fbb800598f0 con 0x7fbb8c06b8c0 2024-04-17T07:29:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:31.120 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:28.100215+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 msgr2=0x7fbb7006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 0x7fbb7006ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c036000 tx=0x7fbb7c008030 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 msgr2=0x7fbb8c0b1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000e9b0 tx=0x7fbb8000ee70 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 shutdown_connections 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fbb7006ca40 0x7fbb7006ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb8c160280 0x7fbb8c1f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb8c141ed0 0x7fbb8c142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 --2- 172.21.15.67:0/1864854444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb8c06b8c0 0x7fbb8c0b1b60 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.123+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 >> 172.21.15.67:0/1864854444 conn(0x7fbb8c067d10 msgr2=0x7fbb8c13f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.124+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 shutdown_connections 2024-04-17T07:29:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.124+0000 7fbb93200640 1 -- 172.21.15.67:0/1864854444 wait complete. 2024-04-17T07:29:31.143 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:29:31.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:29:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.358+0000 7f0cab200640 1 Processor -- start 2024-04-17T07:29:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.358+0000 7f0cab200640 1 -- start start 2024-04-17T07:29:31.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0cab200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cac0b9f00 0x7f0cac0b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0cab200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 0x7f0cac1634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0cab200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac163a20 con 0x7f0cac0d12a0 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0cab200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0cac0cf640 con 0x7f0cac0b9f00 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0cab200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cac0b84c0 con 0x7f0cac0cf8f0 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0ca9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cac0b9f00 0x7f0cac0b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0caa800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cac0d12a0 0x7f0cac0d3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58482/0 (socket says 172.21.15.67:58482) 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0ca9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cac0b9f00 0x7f0cac0b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59352/0 (socket says 172.21.15.67:59352) 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.359+0000 7f0caa800640 1 -- 172.21.15.67:0/1794447256 learned_addr learned my addr 172.21.15.67:0/1794447256 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca1400640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 0x7f0cac1634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca9e00640 1 -- 172.21.15.67:0/1794447256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 msgr2=0x7f0cac1634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca9e00640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 0x7f0cac1634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca9400640 1 -- 172.21.15.67:0/1794447256 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203297630 0 0) 0x7f0ca4001490 con 0x7f0cac0d12a0 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca9e00640 1 -- 172.21.15.67:0/1794447256 >> v1:172.21.15.67:6789/0 conn(0x7f0cac0d12a0 legacy=0x7f0cac0d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca9e00640 1 -- 172.21.15.67:0/1794447256 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac0cfe20 con 0x7f0cac0b9f00 2024-04-17T07:29:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.360+0000 7f0ca1400640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 0x7f0cac1634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.361+0000 7f0ca9e00640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac0b7f80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0097a0 tx=0x7f0c9c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02f827fdd864cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.361+0000 7f0ca9400640 1 -- 172.21.15.67:0/1794447256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c031780 con 0x7f0cac0b9f00 2024-04-17T07:29:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.361+0000 7f0ca9400640 1 -- 172.21.15.67:0/1794447256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c037070 con 0x7f0cac0b9f00 2024-04-17T07:29:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.361+0000 7f0ca9400640 1 -- 172.21.15.67:0/1794447256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c031b40 con 0x7f0cac0b9f00 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 -- 172.21.15.67:0/1794447256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 msgr2=0x7f0cac0b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac0b7f80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0097a0 tx=0x7f0c9c02b820 comp rx=0 tx=0).stop 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 -- 172.21.15.67:0/1794447256 shutdown_connections 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0cf8f0 0x7f0cac1634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 --2- 172.21.15.67:0/1794447256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac0b7f80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 -- 172.21.15.67:0/1794447256 >> 172.21.15.67:0/1794447256 conn(0x7f0cac06cb60 msgr2=0x7f0cac06cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 -- 172.21.15.67:0/1794447256 shutdown_connections 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.362+0000 7f0cab200640 1 -- 172.21.15.67:0/1794447256 wait complete. 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.364+0000 7f0cab200640 1 Processor -- start 2024-04-17T07:29:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.364+0000 7f0cab200640 1 -- start start 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 0x7f0cac0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 0x7f0cac154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac0d3420 con 0x7f0cac0b9f00 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac0d3140 con 0x7f0cac0cf8f0 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0cab200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cac0ba2e0 con 0x7f0cac0d12a0 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0ca9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0ca9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59356/0 (socket says 172.21.15.67:59356) 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.365+0000 7f0ca9e00640 1 -- 172.21.15.67:0/973666438 learned_addr learned my addr 172.21.15.67:0/973666438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:31.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0caa800640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 0x7f0cac154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca1400640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 0x7f0cac0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 -- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 msgr2=0x7f0cac154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 0x7f0cac154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 -- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 msgr2=0x7f0cac0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 0x7f0cac0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 -- 172.21.15.67:0/973666438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac0ccb30 con 0x7f0cac0b9f00 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0caa800640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 0x7f0cac154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca1400640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 0x7f0cac0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca9e00640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0098c0 tx=0x7f0c9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c032820 con 0x7f0cac0b9f00 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.366+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c0329e0 con 0x7f0cac0b9f00 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.367+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c032dd0 con 0x7f0cac0b9f00 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.367+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac0cce10 con 0x7f0cac0b9f00 2024-04-17T07:29:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.367+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac0cd3d0 con 0x7f0cac0b9f00 2024-04-17T07:29:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.368+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cac0755d0 con 0x7f0cac0b9f00 2024-04-17T07:29:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.369+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f0c9c04e440 con 0x7f0cac0b9f00 2024-04-17T07:29:31.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.370+0000 7f0ca2a00640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 0x7f0c8806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.370+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f0c9c0adcf0 con 0x7f0cac0b9f00 2024-04-17T07:29:31.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.371+0000 7f0ca1400640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 0x7f0c8806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.372+0000 7f0ca1400640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 0x7f0c8806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006fd0 tx=0x7f0c94008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.375+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0c9c040070 con 0x7f0cac0b9f00 2024-04-17T07:29:31.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.576+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 --> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0cac0b9390 con 0x7f0c8806ca10 2024-04-17T07:29:31.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.599+0000 7f0ca2a00640 1 -- 172.21.15.67:0/973666438 <== mgr.5488 v2:172.21.15.187:6832/1436704971 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251961 (secure 0 0 0) 0x7f0cac0b9390 con 0x7f0c8806ca10 2024-04-17T07:29:31.602 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:29:31.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 msgr2=0x7f0c8806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 0x7f0c8806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006fd0 tx=0x7f0c94008040 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 msgr2=0x7f0cac1f44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0098c0 tx=0x7f0c9c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 shutdown_connections 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0c8806ca10 0x7f0c8806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cac0d12a0 0x7f0cac154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cac0cf8f0 0x7f0cac0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 --2- 172.21.15.67:0/973666438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cac0b9f00 0x7f0cac1f44c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.604+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 >> 172.21.15.67:0/973666438 conn(0x7f0cac06cb60 msgr2=0x7f0cac0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.605+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 shutdown_connections 2024-04-17T07:29:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.605+0000 7f0cab200640 1 -- 172.21.15.67:0/973666438 wait complete. 2024-04-17T07:29:31.622 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T07:29:31.047680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":284,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319628,"kb_used_data":9128,"kb_used_omap":12,"kb_used_meta":310387,"kb_avail":754655092,"statfs":{"total":773094113280,"available":772766814208,"internally_reserved":0,"allocated":9347072,"data_stored":5030260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":317836848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.996746"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849592+0000","last_change":"2024-04-17T07:29:27.123160+0000","last_active":"2024-04-17T07:29:28.849592+0000","last_peered":"2024-04-17T07:29:28.849592+0000","last_clean":"2024-04-17T07:29:28.849592+0000","last_became_active":"2024-04-17T07:29:27.122861+0000","last_became_peered":"2024-04-17T07:29:27.122861+0000","last_unstale":"2024-04-17T07:29:28.849592+0000","last_undegraded":"2024-04-17T07:29:28.849592+0000","last_fullsized":"2024-04-17T07:29:28.849592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:47.210435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119417+0000","last_change":"2024-04-17T07:29:27.124629+0000","last_active":"2024-04-17T07:29:28.119417+0000","last_peered":"2024-04-17T07:29:28.119417+0000","last_clean":"2024-04-17T07:29:28.119417+0000","last_became_active":"2024-04-17T07:29:27.124335+0000","last_became_peered":"2024-04-17T07:29:27.124335+0000","last_unstale":"2024-04-17T07:29:28.119417+0000","last_undegraded":"2024-04-17T07:29:28.119417+0000","last_fullsized":"2024-04-17T07:29:28.119417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:41.463570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.122157+0000","last_change":"2024-04-17T07:29:27.120262+0000","last_active":"2024-04-17T07:29:28.122157+0000","last_peered":"2024-04-17T07:29:28.122157+0000","last_clean":"2024-04-17T07:29:28.122157+0000","last_became_active":"2024-04-17T07:29:27.119638+0000","last_became_peered":"2024-04-17T07:29:27.119638+0000","last_unstale":"2024-04-17T07:29:28.122157+0000","last_undegraded":"2024-04-17T07:29:28.122157+0000","last_fullsized":"2024-04-17T07:29:28.122157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:11.186057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121216+0000","last_change":"2024-04-17T07:29:27.133151+0000","last_active":"2024-04-17T07:29:28.121216+0000","last_peered":"2024-04-17T07:29:28.121216+0000","last_clean":"2024-04-17T07:29:28.121216+0000","last_became_active":"2024-04-17T07:29:27.132815+0000","last_became_peered":"2024-04-17T07:29:27.132815+0000","last_unstale":"2024-04-17T07:29:28.121216+0000","last_undegraded":"2024-04-17T07:29:28.121216+0000","last_fullsized":"2024-04-17T07:29:28.121216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:55.742836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848922+0000","last_change":"2024-04-17T07:29:27.123839+0000","last_active":"2024-04-17T07:29:28.848922+0000","last_peered":"2024-04-17T07:29:28.848922+0000","last_clean":"2024-04-17T07:29:28.848922+0000","last_became_active":"2024-04-17T07:29:27.123571+0000","last_became_peered":"2024-04-17T07:29:27.123571+0000","last_unstale":"2024-04-17T07:29:28.848922+0000","last_undegraded":"2024-04-17T07:29:28.848922+0000","last_fullsized":"2024-04-17T07:29:28.848922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:28:39.676530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848825+0000","last_change":"2024-04-17T07:29:27.125583+0000","last_active":"2024-04-17T07:29:28.848825+0000","last_peered":"2024-04-17T07:29:28.848825+0000","last_clean":"2024-04-17T07:29:28.848825+0000","last_became_active":"2024-04-17T07:29:27.125316+0000","last_became_peered":"2024-04-17T07:29:27.125316+0000","last_unstale":"2024-04-17T07:29:28.848825+0000","last_undegraded":"2024-04-17T07:29:28.848825+0000","last_fullsized":"2024-04-17T07:29:28.848825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:36.533900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845586+0000","last_change":"2024-04-17T07:29:27.098100+0000","last_active":"2024-04-17T07:29:28.845586+0000","last_peered":"2024-04-17T07:29:28.845586+0000","last_clean":"2024-04-17T07:29:28.845586+0000","last_became_active":"2024-04-17T07:29:27.097713+0000","last_became_peered":"2024-04-17T07:29:27.097713+0000","last_unstale":"2024-04-17T07:29:28.845586+0000","last_undegraded":"2024-04-17T07:29:28.845586+0000","last_fullsized":"2024-04-17T07:29:28.845586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:43.298621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117812+0000","last_change":"2024-04-17T07:29:27.119712+0000","last_active":"2024-04-17T07:29:28.117812+0000","last_peered":"2024-04-17T07:29:28.117812+0000","last_clean":"2024-04-17T07:29:28.117812+0000","last_became_active":"2024-04-17T07:29:27.119237+0000","last_became_peered":"2024-04-17T07:29:27.119237+0000","last_unstale":"2024-04-17T07:29:28.117812+0000","last_undegraded":"2024-04-17T07:29:28.117812+0000","last_fullsized":"2024-04-17T07:29:28.117812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:28.723379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848317+0000","last_change":"2024-04-17T07:29:27.101889+0000","last_active":"2024-04-17T07:29:28.848317+0000","last_peered":"2024-04-17T07:29:28.848317+0000","last_clean":"2024-04-17T07:29:28.848317+0000","last_became_active":"2024-04-17T07:29:27.101344+0000","last_became_peered":"2024-04-17T07:29:27.101344+0000","last_unstale":"2024-04-17T07:29:28.848317+0000","last_undegraded":"2024-04-17T07:29:28.848317+0000","last_fullsized":"2024-04-17T07:29:28.848317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:08:16.663790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848213+0000","last_change":"2024-04-17T07:29:27.130829+0000","last_active":"2024-04-17T07:29:28.848213+0000","last_peered":"2024-04-17T07:29:28.848213+0000","last_clean":"2024-04-17T07:29:28.848213+0000","last_became_active":"2024-04-17T07:29:27.130172+0000","last_became_peered":"2024-04-17T07:29:27.130172+0000","last_unstale":"2024-04-17T07:29:28.848213+0000","last_undegraded":"2024-04-17T07:29:28.848213+0000","last_fullsized":"2024-04-17T07:29:28.848213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:37.605549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848578+0000","last_change":"2024-04-17T07:29:27.114471+0000","last_active":"2024-04-17T07:29:28.848578+0000","last_peered":"2024-04-17T07:29:28.848578+0000","last_clean":"2024-04-17T07:29:28.848578+0000","last_became_active":"2024-04-17T07:29:27.114157+0000","last_became_peered":"2024-04-17T07:29:27.114157+0000","last_unstale":"2024-04-17T07:29:28.848578+0000","last_undegraded":"2024-04-17T07:29:28.848578+0000","last_fullsized":"2024-04-17T07:29:28.848578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:10.327423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121387+0000","last_change":"2024-04-17T07:29:27.110816+0000","last_active":"2024-04-17T07:29:28.121387+0000","last_peered":"2024-04-17T07:29:28.121387+0000","last_clean":"2024-04-17T07:29:28.121387+0000","last_became_active":"2024-04-17T07:29:27.109912+0000","last_became_peered":"2024-04-17T07:29:27.109912+0000","last_unstale":"2024-04-17T07:29:28.121387+0000","last_undegraded":"2024-04-17T07:29:28.121387+0000","last_fullsized":"2024-04-17T07:29:28.121387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:59:41.437930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846903+0000","last_change":"2024-04-17T07:29:27.122361+0000","last_active":"2024-04-17T07:29:28.846903+0000","last_peered":"2024-04-17T07:29:28.846903+0000","last_clean":"2024-04-17T07:29:28.846903+0000","last_became_active":"2024-04-17T07:29:27.122056+0000","last_became_peered":"2024-04-17T07:29:27.122056+0000","last_unstale":"2024-04-17T07:29:28.846903+0000","last_undegraded":"2024-04-17T07:29:28.846903+0000","last_fullsized":"2024-04-17T07:29:28.846903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:21:50.758032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847641+0000","last_change":"2024-04-17T07:29:27.124263+0000","last_active":"2024-04-17T07:29:28.847641+0000","last_peered":"2024-04-17T07:29:28.847641+0000","last_clean":"2024-04-17T07:29:28.847641+0000","last_became_active":"2024-04-17T07:29:27.123970+0000","last_became_peered":"2024-04-17T07:29:27.123970+0000","last_unstale":"2024-04-17T07:29:28.847641+0000","last_undegraded":"2024-04-17T07:29:28.847641+0000","last_fullsized":"2024-04-17T07:29:28.847641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:46.200756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846879+0000","last_change":"2024-04-17T07:29:27.120074+0000","last_active":"2024-04-17T07:29:28.846879+0000","last_peered":"2024-04-17T07:29:28.846879+0000","last_clean":"2024-04-17T07:29:28.846879+0000","last_became_active":"2024-04-17T07:29:27.119794+0000","last_became_peered":"2024-04-17T07:29:27.119794+0000","last_unstale":"2024-04-17T07:29:28.846879+0000","last_undegraded":"2024-04-17T07:29:28.846879+0000","last_fullsized":"2024-04-17T07:29:28.846879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:07.203688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846815+0000","last_change":"2024-04-17T07:29:27.115712+0000","last_active":"2024-04-17T07:29:28.846815+0000","last_peered":"2024-04-17T07:29:28.846815+0000","last_clean":"2024-04-17T07:29:28.846815+0000","last_became_active":"2024-04-17T07:29:27.115066+0000","last_became_peered":"2024-04-17T07:29:27.115066+0000","last_unstale":"2024-04-17T07:29:28.846815+0000","last_undegraded":"2024-04-17T07:29:28.846815+0000","last_fullsized":"2024-04-17T07:29:28.846815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:27.463176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119528+0000","last_change":"2024-04-17T07:29:27.121481+0000","last_active":"2024-04-17T07:29:28.119528+0000","last_peered":"2024-04-17T07:29:28.119528+0000","last_clean":"2024-04-17T07:29:28.119528+0000","last_became_active":"2024-04-17T07:29:27.117310+0000","last_became_peered":"2024-04-17T07:29:27.117310+0000","last_unstale":"2024-04-17T07:29:28.119528+0000","last_undegraded":"2024-04-17T07:29:28.119528+0000","last_fullsized":"2024-04-17T07:29:28.119528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:04.328403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120316+0000","last_change":"2024-04-17T07:29:27.128333+0000","last_active":"2024-04-17T07:29:28.120316+0000","last_peered":"2024-04-17T07:29:28.120316+0000","last_clean":"2024-04-17T07:29:28.120316+0000","last_became_active":"2024-04-17T07:29:27.127651+0000","last_became_peered":"2024-04-17T07:29:27.127651+0000","last_unstale":"2024-04-17T07:29:28.120316+0000","last_undegraded":"2024-04-17T07:29:28.120316+0000","last_fullsized":"2024-04-17T07:29:28.120316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.442353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849548+0000","last_change":"2024-04-17T07:29:27.133121+0000","last_active":"2024-04-17T07:29:28.849548+0000","last_peered":"2024-04-17T07:29:28.849548+0000","last_clean":"2024-04-17T07:29:28.849548+0000","last_became_active":"2024-04-17T07:29:27.132857+0000","last_became_peered":"2024-04-17T07:29:27.132857+0000","last_unstale":"2024-04-17T07:29:28.849548+0000","last_undegraded":"2024-04-17T07:29:28.849548+0000","last_fullsized":"2024-04-17T07:29:28.849548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:19.932955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118878+0000","last_change":"2024-04-17T07:29:27.123664+0000","last_active":"2024-04-17T07:29:28.118878+0000","last_peered":"2024-04-17T07:29:28.118878+0000","last_clean":"2024-04-17T07:29:28.118878+0000","last_became_active":"2024-04-17T07:29:27.123183+0000","last_became_peered":"2024-04-17T07:29:27.123183+0000","last_unstale":"2024-04-17T07:29:28.118878+0000","last_undegraded":"2024-04-17T07:29:28.118878+0000","last_fullsized":"2024-04-17T07:29:28.118878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:26.575870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118976+0000","last_change":"2024-04-17T07:29:27.111149+0000","last_active":"2024-04-17T07:29:28.118976+0000","last_peered":"2024-04-17T07:29:28.118976+0000","last_clean":"2024-04-17T07:29:28.118976+0000","last_became_active":"2024-04-17T07:29:27.110812+0000","last_became_peered":"2024-04-17T07:29:27.110812+0000","last_unstale":"2024-04-17T07:29:28.118976+0000","last_undegraded":"2024-04-17T07:29:28.118976+0000","last_fullsized":"2024-04-17T07:29:28.118976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:40.840787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118029+0000","last_change":"2024-04-17T07:29:27.131525+0000","last_active":"2024-04-17T07:29:28.118029+0000","last_peered":"2024-04-17T07:29:28.118029+0000","last_clean":"2024-04-17T07:29:28.118029+0000","last_became_active":"2024-04-17T07:29:27.131167+0000","last_became_peered":"2024-04-17T07:29:27.131167+0000","last_unstale":"2024-04-17T07:29:28.118029+0000","last_undegraded":"2024-04-17T07:29:28.118029+0000","last_fullsized":"2024-04-17T07:29:28.118029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:53.439903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119460+0000","last_change":"2024-04-17T07:29:27.125657+0000","last_active":"2024-04-17T07:29:28.119460+0000","last_peered":"2024-04-17T07:29:28.119460+0000","last_clean":"2024-04-17T07:29:28.119460+0000","last_became_active":"2024-04-17T07:29:27.124055+0000","last_became_peered":"2024-04-17T07:29:27.124055+0000","last_unstale":"2024-04-17T07:29:28.119460+0000","last_undegraded":"2024-04-17T07:29:28.119460+0000","last_fullsized":"2024-04-17T07:29:28.119460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:14:24.705764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846768+0000","last_change":"2024-04-17T07:29:27.114081+0000","last_active":"2024-04-17T07:29:28.846768+0000","last_peered":"2024-04-17T07:29:28.846768+0000","last_clean":"2024-04-17T07:29:28.846768+0000","last_became_active":"2024-04-17T07:29:27.113789+0000","last_became_peered":"2024-04-17T07:29:27.113789+0000","last_unstale":"2024-04-17T07:29:28.846768+0000","last_undegraded":"2024-04-17T07:29:28.846768+0000","last_fullsized":"2024-04-17T07:29:28.846768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:06.881589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850196+0000","last_change":"2024-04-17T07:29:27.120628+0000","last_active":"2024-04-17T07:29:28.850196+0000","last_peered":"2024-04-17T07:29:28.850196+0000","last_clean":"2024-04-17T07:29:28.850196+0000","last_became_active":"2024-04-17T07:29:27.119868+0000","last_became_peered":"2024-04-17T07:29:27.119868+0000","last_unstale":"2024-04-17T07:29:28.850196+0000","last_undegraded":"2024-04-17T07:29:28.850196+0000","last_fullsized":"2024-04-17T07:29:28.850196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:52.801878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121767+0000","last_change":"2024-04-17T07:29:27.101251+0000","last_active":"2024-04-17T07:29:28.121767+0000","last_peered":"2024-04-17T07:29:28.121767+0000","last_clean":"2024-04-17T07:29:28.121767+0000","last_became_active":"2024-04-17T07:29:27.100928+0000","last_became_peered":"2024-04-17T07:29:27.100928+0000","last_unstale":"2024-04-17T07:29:28.121767+0000","last_undegraded":"2024-04-17T07:29:28.121767+0000","last_fullsized":"2024-04-17T07:29:28.121767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:26:46.645918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846604+0000","last_change":"2024-04-17T07:29:27.118930+0000","last_active":"2024-04-17T07:29:28.846604+0000","last_peered":"2024-04-17T07:29:28.846604+0000","last_clean":"2024-04-17T07:29:28.846604+0000","last_became_active":"2024-04-17T07:29:27.113604+0000","last_became_peered":"2024-04-17T07:29:27.113604+0000","last_unstale":"2024-04-17T07:29:28.846604+0000","last_undegraded":"2024-04-17T07:29:28.846604+0000","last_fullsized":"2024-04-17T07:29:28.846604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:23.782427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848129+0000","last_change":"2024-04-17T07:29:27.129615+0000","last_active":"2024-04-17T07:29:28.848129+0000","last_peered":"2024-04-17T07:29:28.848129+0000","last_clean":"2024-04-17T07:29:28.848129+0000","last_became_active":"2024-04-17T07:29:27.127368+0000","last_became_peered":"2024-04-17T07:29:27.127368+0000","last_unstale":"2024-04-17T07:29:28.848129+0000","last_undegraded":"2024-04-17T07:29:28.848129+0000","last_fullsized":"2024-04-17T07:29:28.848129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:27.463176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119029+0000","last_change":"2024-04-17T07:29:27.114300+0000","last_active":"2024-04-17T07:29:28.119029+0000","last_peered":"2024-04-17T07:29:28.119029+0000","last_clean":"2024-04-17T07:29:28.119029+0000","last_became_active":"2024-04-17T07:29:27.114015+0000","last_became_peered":"2024-04-17T07:29:27.114015+0000","last_unstale":"2024-04-17T07:29:28.119029+0000","last_undegraded":"2024-04-17T07:29:28.119029+0000","last_fullsized":"2024-04-17T07:29:28.119029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:14.604494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118950+0000","last_change":"2024-04-17T07:29:27.120623+0000","last_active":"2024-04-17T07:29:28.118950+0000","last_peered":"2024-04-17T07:29:28.118950+0000","last_clean":"2024-04-17T07:29:28.118950+0000","last_became_active":"2024-04-17T07:29:27.120352+0000","last_became_peered":"2024-04-17T07:29:27.120352+0000","last_unstale":"2024-04-17T07:29:28.118950+0000","last_undegraded":"2024-04-17T07:29:28.118950+0000","last_fullsized":"2024-04-17T07:29:28.118950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:59.438049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847741+0000","last_change":"2024-04-17T07:29:27.117219+0000","last_active":"2024-04-17T07:29:28.847741+0000","last_peered":"2024-04-17T07:29:28.847741+0000","last_clean":"2024-04-17T07:29:28.847741+0000","last_became_active":"2024-04-17T07:29:27.114752+0000","last_became_peered":"2024-04-17T07:29:27.114752+0000","last_unstale":"2024-04-17T07:29:28.847741+0000","last_undegraded":"2024-04-17T07:29:28.847741+0000","last_fullsized":"2024-04-17T07:29:28.847741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:22.024199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847224+0000","last_change":"2024-04-17T07:29:27.115972+0000","last_active":"2024-04-17T07:29:28.847224+0000","last_peered":"2024-04-17T07:29:28.847224+0000","last_clean":"2024-04-17T07:29:28.847224+0000","last_became_active":"2024-04-17T07:29:27.115442+0000","last_became_peered":"2024-04-17T07:29:27.115442+0000","last_unstale":"2024-04-17T07:29:28.847224+0000","last_undegraded":"2024-04-17T07:29:28.847224+0000","last_fullsized":"2024-04-17T07:29:28.847224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:19.932955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850064+0000","last_change":"2024-04-17T07:29:27.112651+0000","last_active":"2024-04-17T07:29:28.850064+0000","last_peered":"2024-04-17T07:29:28.850064+0000","last_clean":"2024-04-17T07:29:28.850064+0000","last_became_active":"2024-04-17T07:29:27.112371+0000","last_became_peered":"2024-04-17T07:29:27.112371+0000","last_unstale":"2024-04-17T07:29:28.850064+0000","last_undegraded":"2024-04-17T07:29:28.850064+0000","last_fullsized":"2024-04-17T07:29:28.850064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:30.631706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850471+0000","last_change":"2024-04-17T07:29:26.098941+0000","last_active":"2024-04-17T07:29:28.850471+0000","last_peered":"2024-04-17T07:29:28.850471+0000","last_clean":"2024-04-17T07:29:28.850471+0000","last_became_active":"2024-04-17T07:29:26.097390+0000","last_became_peered":"2024-04-17T07:29:26.097390+0000","last_unstale":"2024-04-17T07:29:28.850471+0000","last_undegraded":"2024-04-17T07:29:28.850471+0000","last_fullsized":"2024-04-17T07:29:28.850471+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:50:27.726872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849131+0000","last_change":"2024-04-17T07:29:27.114601+0000","last_active":"2024-04-17T07:29:28.849131+0000","last_peered":"2024-04-17T07:29:28.849131+0000","last_clean":"2024-04-17T07:29:28.849131+0000","last_became_active":"2024-04-17T07:29:27.114224+0000","last_became_peered":"2024-04-17T07:29:27.114224+0000","last_unstale":"2024-04-17T07:29:28.849131+0000","last_undegraded":"2024-04-17T07:29:28.849131+0000","last_fullsized":"2024-04-17T07:29:28.849131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:21.977899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119327+0000","last_change":"2024-04-17T07:29:26.091637+0000","last_active":"2024-04-17T07:29:28.119327+0000","last_peered":"2024-04-17T07:29:28.119327+0000","last_clean":"2024-04-17T07:29:28.119327+0000","last_became_active":"2024-04-17T07:29:26.091341+0000","last_became_peered":"2024-04-17T07:29:26.091341+0000","last_unstale":"2024-04-17T07:29:28.119327+0000","last_undegraded":"2024-04-17T07:29:28.119327+0000","last_fullsized":"2024-04-17T07:29:28.119327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:48.870665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846766+0000","last_change":"2024-04-17T07:29:27.120465+0000","last_active":"2024-04-17T07:29:28.846766+0000","last_peered":"2024-04-17T07:29:28.846766+0000","last_clean":"2024-04-17T07:29:28.846766+0000","last_became_active":"2024-04-17T07:29:27.119444+0000","last_became_peered":"2024-04-17T07:29:27.119444+0000","last_unstale":"2024-04-17T07:29:28.846766+0000","last_undegraded":"2024-04-17T07:29:28.846766+0000","last_fullsized":"2024-04-17T07:29:28.846766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:22.024199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849767+0000","last_change":"2024-04-17T07:29:26.089766+0000","last_active":"2024-04-17T07:29:28.849767+0000","last_peered":"2024-04-17T07:29:28.849767+0000","last_clean":"2024-04-17T07:29:28.849767+0000","last_became_active":"2024-04-17T07:29:26.089492+0000","last_became_peered":"2024-04-17T07:29:26.089492+0000","last_unstale":"2024-04-17T07:29:28.849767+0000","last_undegraded":"2024-04-17T07:29:28.849767+0000","last_fullsized":"2024-04-17T07:29:28.849767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:38.813978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847045+0000","last_change":"2024-04-17T07:29:27.100857+0000","last_active":"2024-04-17T07:29:28.847045+0000","last_peered":"2024-04-17T07:29:28.847045+0000","last_clean":"2024-04-17T07:29:28.847045+0000","last_became_active":"2024-04-17T07:29:27.100211+0000","last_became_peered":"2024-04-17T07:29:27.100211+0000","last_unstale":"2024-04-17T07:29:28.847045+0000","last_undegraded":"2024-04-17T07:29:28.847045+0000","last_fullsized":"2024-04-17T07:29:28.847045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:28:39.676530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121567+0000","last_change":"2024-04-17T07:29:26.091899+0000","last_active":"2024-04-17T07:29:28.121567+0000","last_peered":"2024-04-17T07:29:28.121567+0000","last_clean":"2024-04-17T07:29:28.121567+0000","last_became_active":"2024-04-17T07:29:26.091563+0000","last_became_peered":"2024-04-17T07:29:26.091563+0000","last_unstale":"2024-04-17T07:29:28.121567+0000","last_undegraded":"2024-04-17T07:29:28.121567+0000","last_fullsized":"2024-04-17T07:29:28.121567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:09.557891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118507+0000","last_change":"2024-04-17T07:29:27.126462+0000","last_active":"2024-04-17T07:29:28.118507+0000","last_peered":"2024-04-17T07:29:28.118507+0000","last_clean":"2024-04-17T07:29:28.118507+0000","last_became_active":"2024-04-17T07:29:27.124129+0000","last_became_peered":"2024-04-17T07:29:27.124129+0000","last_unstale":"2024-04-17T07:29:28.118507+0000","last_undegraded":"2024-04-17T07:29:28.118507+0000","last_fullsized":"2024-04-17T07:29:28.118507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:11.186057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847812+0000","last_change":"2024-04-17T07:29:26.089084+0000","last_active":"2024-04-17T07:29:28.847812+0000","last_peered":"2024-04-17T07:29:28.847812+0000","last_clean":"2024-04-17T07:29:28.847812+0000","last_became_active":"2024-04-17T07:29:26.088682+0000","last_became_peered":"2024-04-17T07:29:26.088682+0000","last_unstale":"2024-04-17T07:29:28.847812+0000","last_undegraded":"2024-04-17T07:29:28.847812+0000","last_fullsized":"2024-04-17T07:29:28.847812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:21:52.734778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118632+0000","last_change":"2024-04-17T07:29:27.123083+0000","last_active":"2024-04-17T07:29:28.118632+0000","last_peered":"2024-04-17T07:29:28.118632+0000","last_clean":"2024-04-17T07:29:28.118632+0000","last_became_active":"2024-04-17T07:29:27.114606+0000","last_became_peered":"2024-04-17T07:29:27.114606+0000","last_unstale":"2024-04-17T07:29:28.118632+0000","last_undegraded":"2024-04-17T07:29:28.118632+0000","last_fullsized":"2024-04-17T07:29:28.118632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:41:23.134601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849248+0000","last_change":"2024-04-17T07:29:26.095815+0000","last_active":"2024-04-17T07:29:28.849248+0000","last_peered":"2024-04-17T07:29:28.849248+0000","last_clean":"2024-04-17T07:29:28.849248+0000","last_became_active":"2024-04-17T07:29:26.095254+0000","last_became_peered":"2024-04-17T07:29:26.095254+0000","last_unstale":"2024-04-17T07:29:28.849248+0000","last_undegraded":"2024-04-17T07:29:28.849248+0000","last_fullsized":"2024-04-17T07:29:28.849248+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:53.455644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846402+0000","last_change":"2024-04-17T07:29:27.112818+0000","last_active":"2024-04-17T07:29:28.846402+0000","last_peered":"2024-04-17T07:29:28.846402+0000","last_clean":"2024-04-17T07:29:28.846402+0000","last_became_active":"2024-04-17T07:29:27.112514+0000","last_became_peered":"2024-04-17T07:29:27.112514+0000","last_unstale":"2024-04-17T07:29:28.846402+0000","last_undegraded":"2024-04-17T07:29:28.846402+0000","last_fullsized":"2024-04-17T07:29:28.846402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:12:12.066931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120309+0000","last_change":"2024-04-17T07:29:26.093741+0000","last_active":"2024-04-17T07:29:28.120309+0000","last_peered":"2024-04-17T07:29:28.120309+0000","last_clean":"2024-04-17T07:29:28.120309+0000","last_became_active":"2024-04-17T07:29:26.092840+0000","last_became_peered":"2024-04-17T07:29:26.092840+0000","last_unstale":"2024-04-17T07:29:28.120309+0000","last_undegraded":"2024-04-17T07:29:28.120309+0000","last_fullsized":"2024-04-17T07:29:28.120309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:51.647606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117845+0000","last_change":"2024-04-17T07:29:27.117212+0000","last_active":"2024-04-17T07:29:28.117845+0000","last_peered":"2024-04-17T07:29:28.117845+0000","last_clean":"2024-04-17T07:29:28.117845+0000","last_became_active":"2024-04-17T07:29:27.116833+0000","last_became_peered":"2024-04-17T07:29:27.116833+0000","last_unstale":"2024-04-17T07:29:28.117845+0000","last_undegraded":"2024-04-17T07:29:28.117845+0000","last_fullsized":"2024-04-17T07:29:28.117845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:16.252763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845606+0000","last_change":"2024-04-17T07:29:26.090724+0000","last_active":"2024-04-17T07:29:28.845606+0000","last_peered":"2024-04-17T07:29:28.845606+0000","last_clean":"2024-04-17T07:29:28.845606+0000","last_became_active":"2024-04-17T07:29:26.089486+0000","last_became_peered":"2024-04-17T07:29:26.089486+0000","last_unstale":"2024-04-17T07:29:28.845606+0000","last_undegraded":"2024-04-17T07:29:28.845606+0000","last_fullsized":"2024-04-17T07:29:28.845606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:31:47.889530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848292+0000","last_change":"2024-04-17T07:29:27.114364+0000","last_active":"2024-04-17T07:29:28.848292+0000","last_peered":"2024-04-17T07:29:28.848292+0000","last_clean":"2024-04-17T07:29:28.848292+0000","last_became_active":"2024-04-17T07:29:27.114055+0000","last_became_peered":"2024-04-17T07:29:27.114055+0000","last_unstale":"2024-04-17T07:29:28.848292+0000","last_undegraded":"2024-04-17T07:29:28.848292+0000","last_fullsized":"2024-04-17T07:29:28.848292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:14.337338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121029+0000","last_change":"2024-04-17T07:29:26.097035+0000","last_active":"2024-04-17T07:29:28.121029+0000","last_peered":"2024-04-17T07:29:28.121029+0000","last_clean":"2024-04-17T07:29:28.121029+0000","last_became_active":"2024-04-17T07:29:26.096751+0000","last_became_peered":"2024-04-17T07:29:26.096751+0000","last_unstale":"2024-04-17T07:29:28.121029+0000","last_undegraded":"2024-04-17T07:29:28.121029+0000","last_fullsized":"2024-04-17T07:29:28.121029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:39.920812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120918+0000","last_change":"2024-04-17T07:29:27.130355+0000","last_active":"2024-04-17T07:29:28.120918+0000","last_peered":"2024-04-17T07:29:28.120918+0000","last_clean":"2024-04-17T07:29:28.120918+0000","last_became_active":"2024-04-17T07:29:27.129970+0000","last_became_peered":"2024-04-17T07:29:27.129970+0000","last_unstale":"2024-04-17T07:29:28.120918+0000","last_undegraded":"2024-04-17T07:29:28.120918+0000","last_fullsized":"2024-04-17T07:29:28.120918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:33.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119497+0000","last_change":"2024-04-17T07:29:26.095323+0000","last_active":"2024-04-17T07:29:28.119497+0000","last_peered":"2024-04-17T07:29:28.119497+0000","last_clean":"2024-04-17T07:29:28.119497+0000","last_became_active":"2024-04-17T07:29:26.094924+0000","last_became_peered":"2024-04-17T07:29:26.094924+0000","last_unstale":"2024-04-17T07:29:28.119497+0000","last_undegraded":"2024-04-17T07:29:28.119497+0000","last_fullsized":"2024-04-17T07:29:28.119497+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:17:34.892795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847550+0000","last_change":"2024-04-17T07:29:27.119851+0000","last_active":"2024-04-17T07:29:28.847550+0000","last_peered":"2024-04-17T07:29:28.847550+0000","last_clean":"2024-04-17T07:29:28.847550+0000","last_became_active":"2024-04-17T07:29:27.119332+0000","last_became_peered":"2024-04-17T07:29:27.119332+0000","last_unstale":"2024-04-17T07:29:28.847550+0000","last_undegraded":"2024-04-17T07:29:28.847550+0000","last_fullsized":"2024-04-17T07:29:28.847550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:35:26.594389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121395+0000","last_change":"2024-04-17T07:29:26.091377+0000","last_active":"2024-04-17T07:29:28.121395+0000","last_peered":"2024-04-17T07:29:28.121395+0000","last_clean":"2024-04-17T07:29:28.121395+0000","last_became_active":"2024-04-17T07:29:26.091045+0000","last_became_peered":"2024-04-17T07:29:26.091045+0000","last_unstale":"2024-04-17T07:29:28.121395+0000","last_undegraded":"2024-04-17T07:29:28.121395+0000","last_fullsized":"2024-04-17T07:29:28.121395+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:14.757061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120635+0000","last_change":"2024-04-17T07:29:27.130344+0000","last_active":"2024-04-17T07:29:28.120635+0000","last_peered":"2024-04-17T07:29:28.120635+0000","last_clean":"2024-04-17T07:29:28.120635+0000","last_became_active":"2024-04-17T07:29:27.130018+0000","last_became_peered":"2024-04-17T07:29:27.130018+0000","last_unstale":"2024-04-17T07:29:28.120635+0000","last_undegraded":"2024-04-17T07:29:28.120635+0000","last_fullsized":"2024-04-17T07:29:28.120635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:16.252763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"83'6","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847793+0000","last_change":"2024-04-17T07:29:26.085726+0000","last_active":"2024-04-17T07:29:28.847793+0000","last_peered":"2024-04-17T07:29:28.847793+0000","last_clean":"2024-04-17T07:29:28.847793+0000","last_became_active":"2024-04-17T07:29:26.085392+0000","last_became_peered":"2024-04-17T07:29:26.085392+0000","last_unstale":"2024-04-17T07:29:28.847793+0000","last_undegraded":"2024-04-17T07:29:28.847793+0000","last_fullsized":"2024-04-17T07:29:28.847793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:45.018010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120529+0000","last_change":"2024-04-17T07:29:27.117916+0000","last_active":"2024-04-17T07:29:28.120529+0000","last_peered":"2024-04-17T07:29:28.120529+0000","last_clean":"2024-04-17T07:29:28.120529+0000","last_became_active":"2024-04-17T07:29:27.117586+0000","last_became_peered":"2024-04-17T07:29:27.117586+0000","last_unstale":"2024-04-17T07:29:28.120529+0000","last_undegraded":"2024-04-17T07:29:28.120529+0000","last_fullsized":"2024-04-17T07:29:28.120529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:15:20.981413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118610+0000","last_change":"2024-04-17T07:29:26.096371+0000","last_active":"2024-04-17T07:29:28.118610+0000","last_peered":"2024-04-17T07:29:28.118610+0000","last_clean":"2024-04-17T07:29:28.118610+0000","last_became_active":"2024-04-17T07:29:26.095070+0000","last_became_peered":"2024-04-17T07:29:26.095070+0000","last_unstale":"2024-04-17T07:29:28.118610+0000","last_undegraded":"2024-04-17T07:29:28.118610+0000","last_fullsized":"2024-04-17T07:29:28.118610+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:55.256610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119192+0000","last_change":"2024-04-17T07:29:27.116477+0000","last_active":"2024-04-17T07:29:28.119192+0000","last_peered":"2024-04-17T07:29:28.119192+0000","last_clean":"2024-04-17T07:29:28.119192+0000","last_became_active":"2024-04-17T07:29:27.115928+0000","last_became_peered":"2024-04-17T07:29:27.115928+0000","last_unstale":"2024-04-17T07:29:28.119192+0000","last_undegraded":"2024-04-17T07:29:28.119192+0000","last_fullsized":"2024-04-17T07:29:28.119192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:03.728886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846620+0000","last_change":"2024-04-17T07:29:26.093862+0000","last_active":"2024-04-17T07:29:28.846620+0000","last_peered":"2024-04-17T07:29:28.846620+0000","last_clean":"2024-04-17T07:29:28.846620+0000","last_became_active":"2024-04-17T07:29:26.092195+0000","last_became_peered":"2024-04-17T07:29:26.092195+0000","last_unstale":"2024-04-17T07:29:28.846620+0000","last_undegraded":"2024-04-17T07:29:28.846620+0000","last_fullsized":"2024-04-17T07:29:28.846620+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:12.339387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845606+0000","last_change":"2024-04-17T07:29:27.112844+0000","last_active":"2024-04-17T07:29:28.845606+0000","last_peered":"2024-04-17T07:29:28.845606+0000","last_clean":"2024-04-17T07:29:28.845606+0000","last_became_active":"2024-04-17T07:29:27.112549+0000","last_became_peered":"2024-04-17T07:29:27.112549+0000","last_unstale":"2024-04-17T07:29:28.845606+0000","last_undegraded":"2024-04-17T07:29:28.845606+0000","last_fullsized":"2024-04-17T07:29:28.845606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:25.626227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120456+0000","last_change":"2024-04-17T07:29:26.092792+0000","last_active":"2024-04-17T07:29:28.120456+0000","last_peered":"2024-04-17T07:29:28.120456+0000","last_clean":"2024-04-17T07:29:28.120456+0000","last_became_active":"2024-04-17T07:29:26.091708+0000","last_became_peered":"2024-04-17T07:29:26.091708+0000","last_unstale":"2024-04-17T07:29:28.120456+0000","last_undegraded":"2024-04-17T07:29:28.120456+0000","last_fullsized":"2024-04-17T07:29:28.120456+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:07.572381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119722+0000","last_change":"2024-04-17T07:29:27.125408+0000","last_active":"2024-04-17T07:29:28.119722+0000","last_peered":"2024-04-17T07:29:28.119722+0000","last_clean":"2024-04-17T07:29:28.119722+0000","last_became_active":"2024-04-17T07:29:27.125113+0000","last_became_peered":"2024-04-17T07:29:27.125113+0000","last_unstale":"2024-04-17T07:29:28.119722+0000","last_undegraded":"2024-04-17T07:29:28.119722+0000","last_fullsized":"2024-04-17T07:29:28.119722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:17:45.846224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"83'8","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118789+0000","last_change":"2024-04-17T07:29:26.097547+0000","last_active":"2024-04-17T07:29:28.118789+0000","last_peered":"2024-04-17T07:29:28.118789+0000","last_clean":"2024-04-17T07:29:28.118789+0000","last_became_active":"2024-04-17T07:29:26.095575+0000","last_became_peered":"2024-04-17T07:29:26.095575+0000","last_unstale":"2024-04-17T07:29:28.118789+0000","last_undegraded":"2024-04-17T07:29:28.118789+0000","last_fullsized":"2024-04-17T07:29:28.118789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:57.584344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120500+0000","last_change":"2024-04-17T07:29:27.120867+0000","last_active":"2024-04-17T07:29:28.120500+0000","last_peered":"2024-04-17T07:29:28.120500+0000","last_clean":"2024-04-17T07:29:28.120500+0000","last_became_active":"2024-04-17T07:29:27.120574+0000","last_became_peered":"2024-04-17T07:29:27.120574+0000","last_unstale":"2024-04-17T07:29:28.120500+0000","last_undegraded":"2024-04-17T07:29:28.120500+0000","last_fullsized":"2024-04-17T07:29:28.120500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:28.723379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119761+0000","last_change":"2024-04-17T07:29:26.097309+0000","last_active":"2024-04-17T07:29:28.119761+0000","last_peered":"2024-04-17T07:29:28.119761+0000","last_clean":"2024-04-17T07:29:28.119761+0000","last_became_active":"2024-04-17T07:29:26.095696+0000","last_became_peered":"2024-04-17T07:29:26.095696+0000","last_unstale":"2024-04-17T07:29:28.119761+0000","last_undegraded":"2024-04-17T07:29:28.119761+0000","last_fullsized":"2024-04-17T07:29:28.119761+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:39.642342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121866+0000","last_change":"2024-04-17T07:29:27.117536+0000","last_active":"2024-04-17T07:29:28.121866+0000","last_peered":"2024-04-17T07:29:28.121866+0000","last_clean":"2024-04-17T07:29:28.121866+0000","last_became_active":"2024-04-17T07:29:27.117213+0000","last_became_peered":"2024-04-17T07:29:27.117213+0000","last_unstale":"2024-04-17T07:29:28.121866+0000","last_undegraded":"2024-04-17T07:29:28.121866+0000","last_fullsized":"2024-04-17T07:29:28.121866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:58.599330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850058+0000","last_change":"2024-04-17T07:29:26.093682+0000","last_active":"2024-04-17T07:29:28.850058+0000","last_peered":"2024-04-17T07:29:28.850058+0000","last_clean":"2024-04-17T07:29:28.850058+0000","last_became_active":"2024-04-17T07:29:26.090291+0000","last_became_peered":"2024-04-17T07:29:26.090291+0000","last_unstale":"2024-04-17T07:29:28.850058+0000","last_undegraded":"2024-04-17T07:29:28.850058+0000","last_fullsized":"2024-04-17T07:29:28.850058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:49:17.064563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119180+0000","last_change":"2024-04-17T07:29:27.119864+0000","last_active":"2024-04-17T07:29:28.119180+0000","last_peered":"2024-04-17T07:29:28.119180+0000","last_clean":"2024-04-17T07:29:28.119180+0000","last_became_active":"2024-04-17T07:29:27.118284+0000","last_became_peered":"2024-04-17T07:29:27.118284+0000","last_unstale":"2024-04-17T07:29:28.119180+0000","last_undegraded":"2024-04-17T07:29:28.119180+0000","last_fullsized":"2024-04-17T07:29:28.119180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:04:55.685793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121457+0000","last_change":"2024-04-17T07:29:26.092015+0000","last_active":"2024-04-17T07:29:28.121457+0000","last_peered":"2024-04-17T07:29:28.121457+0000","last_clean":"2024-04-17T07:29:28.121457+0000","last_became_active":"2024-04-17T07:29:26.091433+0000","last_became_peered":"2024-04-17T07:29:26.091433+0000","last_unstale":"2024-04-17T07:29:28.121457+0000","last_undegraded":"2024-04-17T07:29:28.121457+0000","last_fullsized":"2024-04-17T07:29:28.121457+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:40.112094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121047+0000","last_change":"2024-04-17T07:29:27.132427+0000","last_active":"2024-04-17T07:29:28.121047+0000","last_peered":"2024-04-17T07:29:28.121047+0000","last_clean":"2024-04-17T07:29:28.121047+0000","last_became_active":"2024-04-17T07:29:27.132208+0000","last_became_peered":"2024-04-17T07:29:27.132208+0000","last_unstale":"2024-04-17T07:29:28.121047+0000","last_undegraded":"2024-04-17T07:29:28.121047+0000","last_fullsized":"2024-04-17T07:29:28.121047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:53.439903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847226+0000","last_change":"2024-04-17T07:29:26.096974+0000","last_active":"2024-04-17T07:29:28.847226+0000","last_peered":"2024-04-17T07:29:28.847226+0000","last_clean":"2024-04-17T07:29:28.847226+0000","last_became_active":"2024-04-17T07:29:26.096715+0000","last_became_peered":"2024-04-17T07:29:26.096715+0000","last_unstale":"2024-04-17T07:29:28.847226+0000","last_undegraded":"2024-04-17T07:29:28.847226+0000","last_fullsized":"2024-04-17T07:29:28.847226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.796473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120859+0000","last_change":"2024-04-17T07:29:27.116830+0000","last_active":"2024-04-17T07:29:28.120859+0000","last_peered":"2024-04-17T07:29:28.120859+0000","last_clean":"2024-04-17T07:29:28.120859+0000","last_became_active":"2024-04-17T07:29:27.116566+0000","last_became_peered":"2024-04-17T07:29:27.116566+0000","last_unstale":"2024-04-17T07:29:28.120859+0000","last_undegraded":"2024-04-17T07:29:28.120859+0000","last_fullsized":"2024-04-17T07:29:28.120859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:46.916850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118892+0000","last_change":"2024-04-17T07:29:26.090683+0000","last_active":"2024-04-17T07:29:28.118892+0000","last_peered":"2024-04-17T07:29:28.118892+0000","last_clean":"2024-04-17T07:29:28.118892+0000","last_became_active":"2024-04-17T07:29:26.090420+0000","last_became_peered":"2024-04-17T07:29:26.090420+0000","last_unstale":"2024-04-17T07:29:28.118892+0000","last_undegraded":"2024-04-17T07:29:28.118892+0000","last_fullsized":"2024-04-17T07:29:28.118892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:19.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848291+0000","last_change":"2024-04-17T07:29:27.114402+0000","last_active":"2024-04-17T07:29:28.848291+0000","last_peered":"2024-04-17T07:29:28.848291+0000","last_clean":"2024-04-17T07:29:28.848291+0000","last_became_active":"2024-04-17T07:29:27.114078+0000","last_became_peered":"2024-04-17T07:29:27.114078+0000","last_unstale":"2024-04-17T07:29:28.848291+0000","last_undegraded":"2024-04-17T07:29:28.848291+0000","last_fullsized":"2024-04-17T07:29:28.848291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:52.801878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847417+0000","last_change":"2024-04-17T07:29:26.088859+0000","last_active":"2024-04-17T07:29:28.847417+0000","last_peered":"2024-04-17T07:29:28.847417+0000","last_clean":"2024-04-17T07:29:28.847417+0000","last_became_active":"2024-04-17T07:29:26.088533+0000","last_became_peered":"2024-04-17T07:29:26.088533+0000","last_unstale":"2024-04-17T07:29:28.847417+0000","last_undegraded":"2024-04-17T07:29:28.847417+0000","last_fullsized":"2024-04-17T07:29:28.847417+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:34.615808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846314+0000","last_change":"2024-04-17T07:29:27.111112+0000","last_active":"2024-04-17T07:29:28.846314+0000","last_peered":"2024-04-17T07:29:28.846314+0000","last_clean":"2024-04-17T07:29:28.846314+0000","last_became_active":"2024-04-17T07:29:27.110821+0000","last_became_peered":"2024-04-17T07:29:27.110821+0000","last_unstale":"2024-04-17T07:29:28.846314+0000","last_undegraded":"2024-04-17T07:29:28.846314+0000","last_fullsized":"2024-04-17T07:29:28.846314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:36.820348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120131+0000","last_change":"2024-04-17T07:29:26.096511+0000","last_active":"2024-04-17T07:29:28.120131+0000","last_peered":"2024-04-17T07:29:28.120131+0000","last_clean":"2024-04-17T07:29:28.120131+0000","last_became_active":"2024-04-17T07:29:26.095227+0000","last_became_peered":"2024-04-17T07:29:26.095227+0000","last_unstale":"2024-04-17T07:29:28.120131+0000","last_undegraded":"2024-04-17T07:29:28.120131+0000","last_fullsized":"2024-04-17T07:29:28.120131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:47.774884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121929+0000","last_change":"2024-04-17T07:29:27.135341+0000","last_active":"2024-04-17T07:29:28.121929+0000","last_peered":"2024-04-17T07:29:28.121929+0000","last_clean":"2024-04-17T07:29:28.121929+0000","last_became_active":"2024-04-17T07:29:27.134993+0000","last_became_peered":"2024-04-17T07:29:27.134993+0000","last_unstale":"2024-04-17T07:29:28.121929+0000","last_undegraded":"2024-04-17T07:29:28.121929+0000","last_fullsized":"2024-04-17T07:29:28.121929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:40.935799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849085+0000","last_change":"2024-04-17T07:29:26.094172+0000","last_active":"2024-04-17T07:29:28.849085+0000","last_peered":"2024-04-17T07:29:28.849085+0000","last_clean":"2024-04-17T07:29:28.849085+0000","last_became_active":"2024-04-17T07:29:26.093851+0000","last_became_peered":"2024-04-17T07:29:26.093851+0000","last_unstale":"2024-04-17T07:29:28.849085+0000","last_undegraded":"2024-04-17T07:29:28.849085+0000","last_fullsized":"2024-04-17T07:29:28.849085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:44.729310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.122030+0000","last_change":"2024-04-17T07:29:27.112876+0000","last_active":"2024-04-17T07:29:28.122030+0000","last_peered":"2024-04-17T07:29:28.122030+0000","last_clean":"2024-04-17T07:29:28.122030+0000","last_became_active":"2024-04-17T07:29:27.112604+0000","last_became_peered":"2024-04-17T07:29:27.112604+0000","last_unstale":"2024-04-17T07:29:28.122030+0000","last_undegraded":"2024-04-17T07:29:28.122030+0000","last_fullsized":"2024-04-17T07:29:28.122030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:15.772047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"84'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:29.096699+0000","last_change":"2024-04-17T07:29:26.095078+0000","last_active":"2024-04-17T07:29:29.096699+0000","last_peered":"2024-04-17T07:29:29.096699+0000","last_clean":"2024-04-17T07:29:29.096699+0000","last_became_active":"2024-04-17T07:29:26.094818+0000","last_became_peered":"2024-04-17T07:29:26.094818+0000","last_unstale":"2024-04-17T07:29:29.096699+0000","last_undegraded":"2024-04-17T07:29:29.096699+0000","last_fullsized":"2024-04-17T07:29:29.096699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:24.202316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847756+0000","last_change":"2024-04-17T07:29:27.119627+0000","last_active":"2024-04-17T07:29:28.847756+0000","last_peered":"2024-04-17T07:29:28.847756+0000","last_clean":"2024-04-17T07:29:28.847756+0000","last_became_active":"2024-04-17T07:29:27.118954+0000","last_became_peered":"2024-04-17T07:29:27.118954+0000","last_unstale":"2024-04-17T07:29:28.847756+0000","last_undegraded":"2024-04-17T07:29:28.847756+0000","last_fullsized":"2024-04-17T07:29:28.847756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:02.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846646+0000","last_change":"2024-04-17T07:29:26.090200+0000","last_active":"2024-04-17T07:29:28.846646+0000","last_peered":"2024-04-17T07:29:28.846646+0000","last_clean":"2024-04-17T07:29:28.846646+0000","last_became_active":"2024-04-17T07:29:26.089907+0000","last_became_peered":"2024-04-17T07:29:26.089907+0000","last_unstale":"2024-04-17T07:29:28.846646+0000","last_undegraded":"2024-04-17T07:29:28.846646+0000","last_fullsized":"2024-04-17T07:29:28.846646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:58.559638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847494+0000","last_change":"2024-04-17T07:29:27.100514+0000","last_active":"2024-04-17T07:29:28.847494+0000","last_peered":"2024-04-17T07:29:28.847494+0000","last_clean":"2024-04-17T07:29:28.847494+0000","last_became_active":"2024-04-17T07:29:27.099313+0000","last_became_peered":"2024-04-17T07:29:27.099313+0000","last_unstale":"2024-04-17T07:29:28.847494+0000","last_undegraded":"2024-04-17T07:29:28.847494+0000","last_fullsized":"2024-04-17T07:29:28.847494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:47.210435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121764+0000","last_change":"2024-04-17T07:29:26.080436+0000","last_active":"2024-04-17T07:29:28.121764+0000","last_peered":"2024-04-17T07:29:28.121764+0000","last_clean":"2024-04-17T07:29:28.121764+0000","last_became_active":"2024-04-17T07:29:26.080137+0000","last_became_peered":"2024-04-17T07:29:26.080137+0000","last_unstale":"2024-04-17T07:29:28.121764+0000","last_undegraded":"2024-04-17T07:29:28.121764+0000","last_fullsized":"2024-04-17T07:29:28.121764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:32:38.414749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118549+0000","last_change":"2024-04-17T07:29:27.123716+0000","last_active":"2024-04-17T07:29:28.118549+0000","last_peered":"2024-04-17T07:29:28.118549+0000","last_clean":"2024-04-17T07:29:28.118549+0000","last_became_active":"2024-04-17T07:29:27.123440+0000","last_became_peered":"2024-04-17T07:29:27.123440+0000","last_unstale":"2024-04-17T07:29:28.118549+0000","last_undegraded":"2024-04-17T07:29:28.118549+0000","last_fullsized":"2024-04-17T07:29:28.118549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:54.006694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846852+0000","last_change":"2024-04-17T07:29:26.079687+0000","last_active":"2024-04-17T07:29:28.846852+0000","last_peered":"2024-04-17T07:29:28.846852+0000","last_clean":"2024-04-17T07:29:28.846852+0000","last_became_active":"2024-04-17T07:29:26.079372+0000","last_became_peered":"2024-04-17T07:29:26.079372+0000","last_unstale":"2024-04-17T07:29:28.846852+0000","last_undegraded":"2024-04-17T07:29:28.846852+0000","last_fullsized":"2024-04-17T07:29:28.846852+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:55.688138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119369+0000","last_change":"2024-04-17T07:29:27.126613+0000","last_active":"2024-04-17T07:29:28.119369+0000","last_peered":"2024-04-17T07:29:28.119369+0000","last_clean":"2024-04-17T07:29:28.119369+0000","last_became_active":"2024-04-17T07:29:27.126235+0000","last_became_peered":"2024-04-17T07:29:27.126235+0000","last_unstale":"2024-04-17T07:29:28.119369+0000","last_undegraded":"2024-04-17T07:29:28.119369+0000","last_fullsized":"2024-04-17T07:29:28.119369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847974+0000","last_change":"2024-04-17T07:29:26.097087+0000","last_active":"2024-04-17T07:29:28.847974+0000","last_peered":"2024-04-17T07:29:28.847974+0000","last_clean":"2024-04-17T07:29:28.847974+0000","last_became_active":"2024-04-17T07:29:26.096536+0000","last_became_peered":"2024-04-17T07:29:26.096536+0000","last_unstale":"2024-04-17T07:29:28.847974+0000","last_undegraded":"2024-04-17T07:29:28.847974+0000","last_fullsized":"2024-04-17T07:29:28.847974+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:08.636535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845967+0000","last_change":"2024-04-17T07:29:27.116889+0000","last_active":"2024-04-17T07:29:28.845967+0000","last_peered":"2024-04-17T07:29:28.845967+0000","last_clean":"2024-04-17T07:29:28.845967+0000","last_became_active":"2024-04-17T07:29:27.115083+0000","last_became_peered":"2024-04-17T07:29:27.115083+0000","last_unstale":"2024-04-17T07:29:28.845967+0000","last_undegraded":"2024-04-17T07:29:28.845967+0000","last_fullsized":"2024-04-17T07:29:28.845967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:35.508241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846483+0000","last_change":"2024-04-17T07:29:26.080915+0000","last_active":"2024-04-17T07:29:28.846483+0000","last_peered":"2024-04-17T07:29:28.846483+0000","last_clean":"2024-04-17T07:29:28.846483+0000","last_became_active":"2024-04-17T07:29:26.080470+0000","last_became_peered":"2024-04-17T07:29:26.080470+0000","last_unstale":"2024-04-17T07:29:28.846483+0000","last_undegraded":"2024-04-17T07:29:28.846483+0000","last_fullsized":"2024-04-17T07:29:28.846483+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:33.269102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117908+0000","last_change":"2024-04-17T07:29:27.117132+0000","last_active":"2024-04-17T07:29:28.117908+0000","last_peered":"2024-04-17T07:29:28.117908+0000","last_clean":"2024-04-17T07:29:28.117908+0000","last_became_active":"2024-04-17T07:29:27.116469+0000","last_became_peered":"2024-04-17T07:29:27.116469+0000","last_unstale":"2024-04-17T07:29:28.117908+0000","last_undegraded":"2024-04-17T07:29:28.117908+0000","last_fullsized":"2024-04-17T07:29:28.117908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:33.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118050+0000","last_change":"2024-04-17T07:29:26.088644+0000","last_active":"2024-04-17T07:29:28.118050+0000","last_peered":"2024-04-17T07:29:28.118050+0000","last_clean":"2024-04-17T07:29:28.118050+0000","last_became_active":"2024-04-17T07:29:26.088306+0000","last_became_peered":"2024-04-17T07:29:26.088306+0000","last_unstale":"2024-04-17T07:29:28.118050+0000","last_undegraded":"2024-04-17T07:29:28.118050+0000","last_fullsized":"2024-04-17T07:29:28.118050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:14.757061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117818+0000","last_change":"2024-04-17T07:29:27.116094+0000","last_active":"2024-04-17T07:29:28.117818+0000","last_peered":"2024-04-17T07:29:28.117818+0000","last_clean":"2024-04-17T07:29:28.117818+0000","last_became_active":"2024-04-17T07:29:27.115395+0000","last_became_peered":"2024-04-17T07:29:27.115395+0000","last_unstale":"2024-04-17T07:29:28.117818+0000","last_undegraded":"2024-04-17T07:29:28.117818+0000","last_fullsized":"2024-04-17T07:29:28.117818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.442353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846320+0000","last_change":"2024-04-17T07:29:26.096004+0000","last_active":"2024-04-17T07:29:28.846320+0000","last_peered":"2024-04-17T07:29:28.846320+0000","last_clean":"2024-04-17T07:29:28.846320+0000","last_became_active":"2024-04-17T07:29:26.095465+0000","last_became_peered":"2024-04-17T07:29:26.095465+0000","last_unstale":"2024-04-17T07:29:28.846320+0000","last_undegraded":"2024-04-17T07:29:28.846320+0000","last_fullsized":"2024-04-17T07:29:28.846320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:08:13.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"72'161","reported_seq":452,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118921+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:29:28.118921+0000","last_peered":"2024-04-17T07:29:28.118921+0000","last_clean":"2024-04-17T07:29:28.118921+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:29:28.118921+0000","last_undegraded":"2024-04-17T07:29:28.118921+0000","last_fullsized":"2024-04-17T07:29:28.118921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542333,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1613824,"data_stored":1071116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":480588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":481240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58548,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313292,"statfs":{"total":96636764160,"available":96576811008,"internally_reserved":0,"allocated":1024000,"data_stored":482200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":480644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542333,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40496,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331344,"statfs":{"total":96636764160,"available":96595296256,"internally_reserved":0,"allocated":1019904,"data_stored":480644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542333,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50220,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321620,"statfs":{"total":96636764160,"available":96585338880,"internally_reserved":0,"allocated":1015808,"data_stored":480678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542332,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1622016,"data_stored":1073150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:29:31.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:29:31.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.888+0000 7fca1f000640 1 Processor -- start 2024-04-17T07:29:31.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.888+0000 7fca1f000640 1 -- start start 2024-04-17T07:29:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca180d0720 0x7fca1809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1f000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1815ce00 con 0x7fca1809afa0 2024-04-17T07:29:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fca1815d090 con 0x7fca180d0720 2024-04-17T07:29:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fca1815d330 con 0x7fca1815ca20 2024-04-17T07:29:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca180d0720 0x7fca1809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca180d0720 0x7fca1809aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59364/0 (socket says 172.21.15.67:59364) 2024-04-17T07:29:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.890+0000 7fca1d200640 1 -- 172.21.15.67:0/1128789397 learned_addr learned my addr 172.21.15.67:0/1128789397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.891+0000 7fca1dc00640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.891+0000 7fca1d200640 1 -- 172.21.15.67:0/1128789397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 msgr2=0x7fca180d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.891+0000 7fca1d200640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.891+0000 7fca1d200640 1 -- 172.21.15.67:0/1128789397 >> v1:172.21.15.187:6789/0 conn(0x7fca1809afa0 legacy=0x7fca181604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.891+0000 7fca1d200640 1 -- 172.21.15.67:0/1128789397 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1815d610 con 0x7fca180d0720 2024-04-17T07:29:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.892+0000 7fca1d200640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca1809aa60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fca04009870 tx=0x7fca0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d51ce395640e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.892+0000 7fca17e00640 1 -- 172.21.15.67:0/1128789397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04035020 con 0x7fca180d0720 2024-04-17T07:29:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.892+0000 7fca17e00640 1 -- 172.21.15.67:0/1128789397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca04031640 con 0x7fca180d0720 2024-04-17T07:29:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.892+0000 7fca17e00640 1 -- 172.21.15.67:0/1128789397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0402d040 con 0x7fca180d0720 2024-04-17T07:29:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 -- 172.21.15.67:0/1128789397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 msgr2=0x7fca1809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca1809aa60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fca04009870 tx=0x7fca0402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 -- 172.21.15.67:0/1128789397 shutdown_connections 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca1809aa60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 --2- 172.21.15.67:0/1128789397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 -- 172.21.15.67:0/1128789397 >> 172.21.15.67:0/1128789397 conn(0x7fca180b4100 msgr2=0x7fca180b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 -- 172.21.15.67:0/1128789397 shutdown_connections 2024-04-17T07:29:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.893+0000 7fca1f000640 1 -- 172.21.15.67:0/1128789397 wait complete. 2024-04-17T07:29:31.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.895+0000 7fca1f000640 1 Processor -- start 2024-04-17T07:29:31.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.895+0000 7fca1f000640 1 -- start start 2024-04-17T07:29:31.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca1809afa0 0x7fca180c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca1809bea0 con 0x7fca180d0720 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fca1809bd20 con 0x7fca1809afa0 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fca180ce1a0 con 0x7fca1815ca20 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.896+0000 7fca1d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59368/0 (socket says 172.21.15.67:59368) 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 -- 172.21.15.67:0/3171597867 learned_addr learned my addr 172.21.15.67:0/3171597867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1e600640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1dc00640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca1809afa0 0x7fca180c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 -- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 msgr2=0x7fca180ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 -- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca1809afa0 msgr2=0x7fca180c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca1809afa0 0x7fca180c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:31.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 -- 172.21.15.67:0/3171597867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca180c7170 con 0x7fca180d0720 2024-04-17T07:29:31.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.897+0000 7fca1d200640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fca04000c00 tx=0x7fca040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.898+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04040070 con 0x7fca180d0720 2024-04-17T07:29:31.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.898+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca04039bf0 con 0x7fca180d0720 2024-04-17T07:29:31.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.898+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04035040 con 0x7fca180d0720 2024-04-17T07:29:31.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.898+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca180c7450 con 0x7fca180d0720 2024-04-17T07:29:31.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.898+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca180c7a10 con 0x7fca180d0720 2024-04-17T07:29:31.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.899+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1806bc90 con 0x7fca180d0720 2024-04-17T07:29:31.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.900+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7fca0404b060 con 0x7fca180d0720 2024-04-17T07:29:31.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.901+0000 7fca15600640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 0x7fc9f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:31.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.902+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fca040ad780 con 0x7fca180d0720 2024-04-17T07:29:31.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.902+0000 7fca1dc00640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 0x7fc9f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:31.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.903+0000 7fca1dc00640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 0x7fc9f806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006fd0 tx=0x7fca0c008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:31.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:31.906+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fca1806bc90 con 0x7fca180d0720 2024-04-17T07:29:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.108+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 --> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca180755d0 con 0x7fc9f806ca10 2024-04-17T07:29:32.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.129+0000 7fca15600640 1 -- 172.21.15.67:0/3171597867 <== mgr.5488 v2:172.21.15.187:6832/1436704971 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251961 (secure 0 0 0) 0x7fca180755d0 con 0x7fc9f806ca10 2024-04-17T07:29:32.131 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:29:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 msgr2=0x7fc9f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 0x7fc9f806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006fd0 tx=0x7fca0c008040 comp rx=0 tx=0).stop 2024-04-17T07:29:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 msgr2=0x7fca18154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fca04000c00 tx=0x7fca040046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 shutdown_connections 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fc9f806ca10 0x7fc9f806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca1815ca20 0x7fca180ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca180d0720 0x7fca18154b60 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 --2- 172.21.15.67:0/3171597867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca1809afa0 0x7fca180c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 >> 172.21.15.67:0/3171597867 conn(0x7fca180b4100 msgr2=0x7fca18070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 shutdown_connections 2024-04-17T07:29:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.136+0000 7fca1f000640 1 -- 172.21.15.67:0/3171597867 wait complete. 2024-04-17T07:29:32.158 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T07:29:31.047680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":284,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319628,"kb_used_data":9128,"kb_used_omap":12,"kb_used_meta":310387,"kb_avail":754655092,"statfs":{"total":773094113280,"available":772766814208,"internally_reserved":0,"allocated":9347072,"data_stored":5030260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":317836848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.996746"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849592+0000","last_change":"2024-04-17T07:29:27.123160+0000","last_active":"2024-04-17T07:29:28.849592+0000","last_peered":"2024-04-17T07:29:28.849592+0000","last_clean":"2024-04-17T07:29:28.849592+0000","last_became_active":"2024-04-17T07:29:27.122861+0000","last_became_peered":"2024-04-17T07:29:27.122861+0000","last_unstale":"2024-04-17T07:29:28.849592+0000","last_undegraded":"2024-04-17T07:29:28.849592+0000","last_fullsized":"2024-04-17T07:29:28.849592+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:47.210435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119417+0000","last_change":"2024-04-17T07:29:27.124629+0000","last_active":"2024-04-17T07:29:28.119417+0000","last_peered":"2024-04-17T07:29:28.119417+0000","last_clean":"2024-04-17T07:29:28.119417+0000","last_became_active":"2024-04-17T07:29:27.124335+0000","last_became_peered":"2024-04-17T07:29:27.124335+0000","last_unstale":"2024-04-17T07:29:28.119417+0000","last_undegraded":"2024-04-17T07:29:28.119417+0000","last_fullsized":"2024-04-17T07:29:28.119417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:41.463570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.122157+0000","last_change":"2024-04-17T07:29:27.120262+0000","last_active":"2024-04-17T07:29:28.122157+0000","last_peered":"2024-04-17T07:29:28.122157+0000","last_clean":"2024-04-17T07:29:28.122157+0000","last_became_active":"2024-04-17T07:29:27.119638+0000","last_became_peered":"2024-04-17T07:29:27.119638+0000","last_unstale":"2024-04-17T07:29:28.122157+0000","last_undegraded":"2024-04-17T07:29:28.122157+0000","last_fullsized":"2024-04-17T07:29:28.122157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:11.186057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121216+0000","last_change":"2024-04-17T07:29:27.133151+0000","last_active":"2024-04-17T07:29:28.121216+0000","last_peered":"2024-04-17T07:29:28.121216+0000","last_clean":"2024-04-17T07:29:28.121216+0000","last_became_active":"2024-04-17T07:29:27.132815+0000","last_became_peered":"2024-04-17T07:29:27.132815+0000","last_unstale":"2024-04-17T07:29:28.121216+0000","last_undegraded":"2024-04-17T07:29:28.121216+0000","last_fullsized":"2024-04-17T07:29:28.121216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:55.742836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848922+0000","last_change":"2024-04-17T07:29:27.123839+0000","last_active":"2024-04-17T07:29:28.848922+0000","last_peered":"2024-04-17T07:29:28.848922+0000","last_clean":"2024-04-17T07:29:28.848922+0000","last_became_active":"2024-04-17T07:29:27.123571+0000","last_became_peered":"2024-04-17T07:29:27.123571+0000","last_unstale":"2024-04-17T07:29:28.848922+0000","last_undegraded":"2024-04-17T07:29:28.848922+0000","last_fullsized":"2024-04-17T07:29:28.848922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:28:39.676530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848825+0000","last_change":"2024-04-17T07:29:27.125583+0000","last_active":"2024-04-17T07:29:28.848825+0000","last_peered":"2024-04-17T07:29:28.848825+0000","last_clean":"2024-04-17T07:29:28.848825+0000","last_became_active":"2024-04-17T07:29:27.125316+0000","last_became_peered":"2024-04-17T07:29:27.125316+0000","last_unstale":"2024-04-17T07:29:28.848825+0000","last_undegraded":"2024-04-17T07:29:28.848825+0000","last_fullsized":"2024-04-17T07:29:28.848825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:36.533900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845586+0000","last_change":"2024-04-17T07:29:27.098100+0000","last_active":"2024-04-17T07:29:28.845586+0000","last_peered":"2024-04-17T07:29:28.845586+0000","last_clean":"2024-04-17T07:29:28.845586+0000","last_became_active":"2024-04-17T07:29:27.097713+0000","last_became_peered":"2024-04-17T07:29:27.097713+0000","last_unstale":"2024-04-17T07:29:28.845586+0000","last_undegraded":"2024-04-17T07:29:28.845586+0000","last_fullsized":"2024-04-17T07:29:28.845586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:43.298621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117812+0000","last_change":"2024-04-17T07:29:27.119712+0000","last_active":"2024-04-17T07:29:28.117812+0000","last_peered":"2024-04-17T07:29:28.117812+0000","last_clean":"2024-04-17T07:29:28.117812+0000","last_became_active":"2024-04-17T07:29:27.119237+0000","last_became_peered":"2024-04-17T07:29:27.119237+0000","last_unstale":"2024-04-17T07:29:28.117812+0000","last_undegraded":"2024-04-17T07:29:28.117812+0000","last_fullsized":"2024-04-17T07:29:28.117812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:28.723379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848317+0000","last_change":"2024-04-17T07:29:27.101889+0000","last_active":"2024-04-17T07:29:28.848317+0000","last_peered":"2024-04-17T07:29:28.848317+0000","last_clean":"2024-04-17T07:29:28.848317+0000","last_became_active":"2024-04-17T07:29:27.101344+0000","last_became_peered":"2024-04-17T07:29:27.101344+0000","last_unstale":"2024-04-17T07:29:28.848317+0000","last_undegraded":"2024-04-17T07:29:28.848317+0000","last_fullsized":"2024-04-17T07:29:28.848317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:08:16.663790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848213+0000","last_change":"2024-04-17T07:29:27.130829+0000","last_active":"2024-04-17T07:29:28.848213+0000","last_peered":"2024-04-17T07:29:28.848213+0000","last_clean":"2024-04-17T07:29:28.848213+0000","last_became_active":"2024-04-17T07:29:27.130172+0000","last_became_peered":"2024-04-17T07:29:27.130172+0000","last_unstale":"2024-04-17T07:29:28.848213+0000","last_undegraded":"2024-04-17T07:29:28.848213+0000","last_fullsized":"2024-04-17T07:29:28.848213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:37.605549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848578+0000","last_change":"2024-04-17T07:29:27.114471+0000","last_active":"2024-04-17T07:29:28.848578+0000","last_peered":"2024-04-17T07:29:28.848578+0000","last_clean":"2024-04-17T07:29:28.848578+0000","last_became_active":"2024-04-17T07:29:27.114157+0000","last_became_peered":"2024-04-17T07:29:27.114157+0000","last_unstale":"2024-04-17T07:29:28.848578+0000","last_undegraded":"2024-04-17T07:29:28.848578+0000","last_fullsized":"2024-04-17T07:29:28.848578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:10.327423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121387+0000","last_change":"2024-04-17T07:29:27.110816+0000","last_active":"2024-04-17T07:29:28.121387+0000","last_peered":"2024-04-17T07:29:28.121387+0000","last_clean":"2024-04-17T07:29:28.121387+0000","last_became_active":"2024-04-17T07:29:27.109912+0000","last_became_peered":"2024-04-17T07:29:27.109912+0000","last_unstale":"2024-04-17T07:29:28.121387+0000","last_undegraded":"2024-04-17T07:29:28.121387+0000","last_fullsized":"2024-04-17T07:29:28.121387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:59:41.437930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846903+0000","last_change":"2024-04-17T07:29:27.122361+0000","last_active":"2024-04-17T07:29:28.846903+0000","last_peered":"2024-04-17T07:29:28.846903+0000","last_clean":"2024-04-17T07:29:28.846903+0000","last_became_active":"2024-04-17T07:29:27.122056+0000","last_became_peered":"2024-04-17T07:29:27.122056+0000","last_unstale":"2024-04-17T07:29:28.846903+0000","last_undegraded":"2024-04-17T07:29:28.846903+0000","last_fullsized":"2024-04-17T07:29:28.846903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:21:50.758032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847641+0000","last_change":"2024-04-17T07:29:27.124263+0000","last_active":"2024-04-17T07:29:28.847641+0000","last_peered":"2024-04-17T07:29:28.847641+0000","last_clean":"2024-04-17T07:29:28.847641+0000","last_became_active":"2024-04-17T07:29:27.123970+0000","last_became_peered":"2024-04-17T07:29:27.123970+0000","last_unstale":"2024-04-17T07:29:28.847641+0000","last_undegraded":"2024-04-17T07:29:28.847641+0000","last_fullsized":"2024-04-17T07:29:28.847641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:46.200756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846879+0000","last_change":"2024-04-17T07:29:27.120074+0000","last_active":"2024-04-17T07:29:28.846879+0000","last_peered":"2024-04-17T07:29:28.846879+0000","last_clean":"2024-04-17T07:29:28.846879+0000","last_became_active":"2024-04-17T07:29:27.119794+0000","last_became_peered":"2024-04-17T07:29:27.119794+0000","last_unstale":"2024-04-17T07:29:28.846879+0000","last_undegraded":"2024-04-17T07:29:28.846879+0000","last_fullsized":"2024-04-17T07:29:28.846879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:07.203688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846815+0000","last_change":"2024-04-17T07:29:27.115712+0000","last_active":"2024-04-17T07:29:28.846815+0000","last_peered":"2024-04-17T07:29:28.846815+0000","last_clean":"2024-04-17T07:29:28.846815+0000","last_became_active":"2024-04-17T07:29:27.115066+0000","last_became_peered":"2024-04-17T07:29:27.115066+0000","last_unstale":"2024-04-17T07:29:28.846815+0000","last_undegraded":"2024-04-17T07:29:28.846815+0000","last_fullsized":"2024-04-17T07:29:28.846815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:27.463176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119528+0000","last_change":"2024-04-17T07:29:27.121481+0000","last_active":"2024-04-17T07:29:28.119528+0000","last_peered":"2024-04-17T07:29:28.119528+0000","last_clean":"2024-04-17T07:29:28.119528+0000","last_became_active":"2024-04-17T07:29:27.117310+0000","last_became_peered":"2024-04-17T07:29:27.117310+0000","last_unstale":"2024-04-17T07:29:28.119528+0000","last_undegraded":"2024-04-17T07:29:28.119528+0000","last_fullsized":"2024-04-17T07:29:28.119528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:04.328403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120316+0000","last_change":"2024-04-17T07:29:27.128333+0000","last_active":"2024-04-17T07:29:28.120316+0000","last_peered":"2024-04-17T07:29:28.120316+0000","last_clean":"2024-04-17T07:29:28.120316+0000","last_became_active":"2024-04-17T07:29:27.127651+0000","last_became_peered":"2024-04-17T07:29:27.127651+0000","last_unstale":"2024-04-17T07:29:28.120316+0000","last_undegraded":"2024-04-17T07:29:28.120316+0000","last_fullsized":"2024-04-17T07:29:28.120316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.442353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849548+0000","last_change":"2024-04-17T07:29:27.133121+0000","last_active":"2024-04-17T07:29:28.849548+0000","last_peered":"2024-04-17T07:29:28.849548+0000","last_clean":"2024-04-17T07:29:28.849548+0000","last_became_active":"2024-04-17T07:29:27.132857+0000","last_became_peered":"2024-04-17T07:29:27.132857+0000","last_unstale":"2024-04-17T07:29:28.849548+0000","last_undegraded":"2024-04-17T07:29:28.849548+0000","last_fullsized":"2024-04-17T07:29:28.849548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:19.932955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118878+0000","last_change":"2024-04-17T07:29:27.123664+0000","last_active":"2024-04-17T07:29:28.118878+0000","last_peered":"2024-04-17T07:29:28.118878+0000","last_clean":"2024-04-17T07:29:28.118878+0000","last_became_active":"2024-04-17T07:29:27.123183+0000","last_became_peered":"2024-04-17T07:29:27.123183+0000","last_unstale":"2024-04-17T07:29:28.118878+0000","last_undegraded":"2024-04-17T07:29:28.118878+0000","last_fullsized":"2024-04-17T07:29:28.118878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:26.575870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118976+0000","last_change":"2024-04-17T07:29:27.111149+0000","last_active":"2024-04-17T07:29:28.118976+0000","last_peered":"2024-04-17T07:29:28.118976+0000","last_clean":"2024-04-17T07:29:28.118976+0000","last_became_active":"2024-04-17T07:29:27.110812+0000","last_became_peered":"2024-04-17T07:29:27.110812+0000","last_unstale":"2024-04-17T07:29:28.118976+0000","last_undegraded":"2024-04-17T07:29:28.118976+0000","last_fullsized":"2024-04-17T07:29:28.118976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:40.840787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118029+0000","last_change":"2024-04-17T07:29:27.131525+0000","last_active":"2024-04-17T07:29:28.118029+0000","last_peered":"2024-04-17T07:29:28.118029+0000","last_clean":"2024-04-17T07:29:28.118029+0000","last_became_active":"2024-04-17T07:29:27.131167+0000","last_became_peered":"2024-04-17T07:29:27.131167+0000","last_unstale":"2024-04-17T07:29:28.118029+0000","last_undegraded":"2024-04-17T07:29:28.118029+0000","last_fullsized":"2024-04-17T07:29:28.118029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:53.439903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119460+0000","last_change":"2024-04-17T07:29:27.125657+0000","last_active":"2024-04-17T07:29:28.119460+0000","last_peered":"2024-04-17T07:29:28.119460+0000","last_clean":"2024-04-17T07:29:28.119460+0000","last_became_active":"2024-04-17T07:29:27.124055+0000","last_became_peered":"2024-04-17T07:29:27.124055+0000","last_unstale":"2024-04-17T07:29:28.119460+0000","last_undegraded":"2024-04-17T07:29:28.119460+0000","last_fullsized":"2024-04-17T07:29:28.119460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:14:24.705764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846768+0000","last_change":"2024-04-17T07:29:27.114081+0000","last_active":"2024-04-17T07:29:28.846768+0000","last_peered":"2024-04-17T07:29:28.846768+0000","last_clean":"2024-04-17T07:29:28.846768+0000","last_became_active":"2024-04-17T07:29:27.113789+0000","last_became_peered":"2024-04-17T07:29:27.113789+0000","last_unstale":"2024-04-17T07:29:28.846768+0000","last_undegraded":"2024-04-17T07:29:28.846768+0000","last_fullsized":"2024-04-17T07:29:28.846768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:06.881589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850196+0000","last_change":"2024-04-17T07:29:27.120628+0000","last_active":"2024-04-17T07:29:28.850196+0000","last_peered":"2024-04-17T07:29:28.850196+0000","last_clean":"2024-04-17T07:29:28.850196+0000","last_became_active":"2024-04-17T07:29:27.119868+0000","last_became_peered":"2024-04-17T07:29:27.119868+0000","last_unstale":"2024-04-17T07:29:28.850196+0000","last_undegraded":"2024-04-17T07:29:28.850196+0000","last_fullsized":"2024-04-17T07:29:28.850196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:52.801878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121767+0000","last_change":"2024-04-17T07:29:27.101251+0000","last_active":"2024-04-17T07:29:28.121767+0000","last_peered":"2024-04-17T07:29:28.121767+0000","last_clean":"2024-04-17T07:29:28.121767+0000","last_became_active":"2024-04-17T07:29:27.100928+0000","last_became_peered":"2024-04-17T07:29:27.100928+0000","last_unstale":"2024-04-17T07:29:28.121767+0000","last_undegraded":"2024-04-17T07:29:28.121767+0000","last_fullsized":"2024-04-17T07:29:28.121767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:26:46.645918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846604+0000","last_change":"2024-04-17T07:29:27.118930+0000","last_active":"2024-04-17T07:29:28.846604+0000","last_peered":"2024-04-17T07:29:28.846604+0000","last_clean":"2024-04-17T07:29:28.846604+0000","last_became_active":"2024-04-17T07:29:27.113604+0000","last_became_peered":"2024-04-17T07:29:27.113604+0000","last_unstale":"2024-04-17T07:29:28.846604+0000","last_undegraded":"2024-04-17T07:29:28.846604+0000","last_fullsized":"2024-04-17T07:29:28.846604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:23.782427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848129+0000","last_change":"2024-04-17T07:29:27.129615+0000","last_active":"2024-04-17T07:29:28.848129+0000","last_peered":"2024-04-17T07:29:28.848129+0000","last_clean":"2024-04-17T07:29:28.848129+0000","last_became_active":"2024-04-17T07:29:27.127368+0000","last_became_peered":"2024-04-17T07:29:27.127368+0000","last_unstale":"2024-04-17T07:29:28.848129+0000","last_undegraded":"2024-04-17T07:29:28.848129+0000","last_fullsized":"2024-04-17T07:29:28.848129+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:27.463176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119029+0000","last_change":"2024-04-17T07:29:27.114300+0000","last_active":"2024-04-17T07:29:28.119029+0000","last_peered":"2024-04-17T07:29:28.119029+0000","last_clean":"2024-04-17T07:29:28.119029+0000","last_became_active":"2024-04-17T07:29:27.114015+0000","last_became_peered":"2024-04-17T07:29:27.114015+0000","last_unstale":"2024-04-17T07:29:28.119029+0000","last_undegraded":"2024-04-17T07:29:28.119029+0000","last_fullsized":"2024-04-17T07:29:28.119029+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:14.604494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118950+0000","last_change":"2024-04-17T07:29:27.120623+0000","last_active":"2024-04-17T07:29:28.118950+0000","last_peered":"2024-04-17T07:29:28.118950+0000","last_clean":"2024-04-17T07:29:28.118950+0000","last_became_active":"2024-04-17T07:29:27.120352+0000","last_became_peered":"2024-04-17T07:29:27.120352+0000","last_unstale":"2024-04-17T07:29:28.118950+0000","last_undegraded":"2024-04-17T07:29:28.118950+0000","last_fullsized":"2024-04-17T07:29:28.118950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:59.438049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847741+0000","last_change":"2024-04-17T07:29:27.117219+0000","last_active":"2024-04-17T07:29:28.847741+0000","last_peered":"2024-04-17T07:29:28.847741+0000","last_clean":"2024-04-17T07:29:28.847741+0000","last_became_active":"2024-04-17T07:29:27.114752+0000","last_became_peered":"2024-04-17T07:29:27.114752+0000","last_unstale":"2024-04-17T07:29:28.847741+0000","last_undegraded":"2024-04-17T07:29:28.847741+0000","last_fullsized":"2024-04-17T07:29:28.847741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:22.024199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847224+0000","last_change":"2024-04-17T07:29:27.115972+0000","last_active":"2024-04-17T07:29:28.847224+0000","last_peered":"2024-04-17T07:29:28.847224+0000","last_clean":"2024-04-17T07:29:28.847224+0000","last_became_active":"2024-04-17T07:29:27.115442+0000","last_became_peered":"2024-04-17T07:29:27.115442+0000","last_unstale":"2024-04-17T07:29:28.847224+0000","last_undegraded":"2024-04-17T07:29:28.847224+0000","last_fullsized":"2024-04-17T07:29:28.847224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:19.932955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850064+0000","last_change":"2024-04-17T07:29:27.112651+0000","last_active":"2024-04-17T07:29:28.850064+0000","last_peered":"2024-04-17T07:29:28.850064+0000","last_clean":"2024-04-17T07:29:28.850064+0000","last_became_active":"2024-04-17T07:29:27.112371+0000","last_became_peered":"2024-04-17T07:29:27.112371+0000","last_unstale":"2024-04-17T07:29:28.850064+0000","last_undegraded":"2024-04-17T07:29:28.850064+0000","last_fullsized":"2024-04-17T07:29:28.850064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:30.631706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850471+0000","last_change":"2024-04-17T07:29:26.098941+0000","last_active":"2024-04-17T07:29:28.850471+0000","last_peered":"2024-04-17T07:29:28.850471+0000","last_clean":"2024-04-17T07:29:28.850471+0000","last_became_active":"2024-04-17T07:29:26.097390+0000","last_became_peered":"2024-04-17T07:29:26.097390+0000","last_unstale":"2024-04-17T07:29:28.850471+0000","last_undegraded":"2024-04-17T07:29:28.850471+0000","last_fullsized":"2024-04-17T07:29:28.850471+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:50:27.726872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849131+0000","last_change":"2024-04-17T07:29:27.114601+0000","last_active":"2024-04-17T07:29:28.849131+0000","last_peered":"2024-04-17T07:29:28.849131+0000","last_clean":"2024-04-17T07:29:28.849131+0000","last_became_active":"2024-04-17T07:29:27.114224+0000","last_became_peered":"2024-04-17T07:29:27.114224+0000","last_unstale":"2024-04-17T07:29:28.849131+0000","last_undegraded":"2024-04-17T07:29:28.849131+0000","last_fullsized":"2024-04-17T07:29:28.849131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:21.977899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119327+0000","last_change":"2024-04-17T07:29:26.091637+0000","last_active":"2024-04-17T07:29:28.119327+0000","last_peered":"2024-04-17T07:29:28.119327+0000","last_clean":"2024-04-17T07:29:28.119327+0000","last_became_active":"2024-04-17T07:29:26.091341+0000","last_became_peered":"2024-04-17T07:29:26.091341+0000","last_unstale":"2024-04-17T07:29:28.119327+0000","last_undegraded":"2024-04-17T07:29:28.119327+0000","last_fullsized":"2024-04-17T07:29:28.119327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:48.870665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846766+0000","last_change":"2024-04-17T07:29:27.120465+0000","last_active":"2024-04-17T07:29:28.846766+0000","last_peered":"2024-04-17T07:29:28.846766+0000","last_clean":"2024-04-17T07:29:28.846766+0000","last_became_active":"2024-04-17T07:29:27.119444+0000","last_became_peered":"2024-04-17T07:29:27.119444+0000","last_unstale":"2024-04-17T07:29:28.846766+0000","last_undegraded":"2024-04-17T07:29:28.846766+0000","last_fullsized":"2024-04-17T07:29:28.846766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:22.024199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849767+0000","last_change":"2024-04-17T07:29:26.089766+0000","last_active":"2024-04-17T07:29:28.849767+0000","last_peered":"2024-04-17T07:29:28.849767+0000","last_clean":"2024-04-17T07:29:28.849767+0000","last_became_active":"2024-04-17T07:29:26.089492+0000","last_became_peered":"2024-04-17T07:29:26.089492+0000","last_unstale":"2024-04-17T07:29:28.849767+0000","last_undegraded":"2024-04-17T07:29:28.849767+0000","last_fullsized":"2024-04-17T07:29:28.849767+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:38.813978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847045+0000","last_change":"2024-04-17T07:29:27.100857+0000","last_active":"2024-04-17T07:29:28.847045+0000","last_peered":"2024-04-17T07:29:28.847045+0000","last_clean":"2024-04-17T07:29:28.847045+0000","last_became_active":"2024-04-17T07:29:27.100211+0000","last_became_peered":"2024-04-17T07:29:27.100211+0000","last_unstale":"2024-04-17T07:29:28.847045+0000","last_undegraded":"2024-04-17T07:29:28.847045+0000","last_fullsized":"2024-04-17T07:29:28.847045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:28:39.676530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121567+0000","last_change":"2024-04-17T07:29:26.091899+0000","last_active":"2024-04-17T07:29:28.121567+0000","last_peered":"2024-04-17T07:29:28.121567+0000","last_clean":"2024-04-17T07:29:28.121567+0000","last_became_active":"2024-04-17T07:29:26.091563+0000","last_became_peered":"2024-04-17T07:29:26.091563+0000","last_unstale":"2024-04-17T07:29:28.121567+0000","last_undegraded":"2024-04-17T07:29:28.121567+0000","last_fullsized":"2024-04-17T07:29:28.121567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:09.557891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118507+0000","last_change":"2024-04-17T07:29:27.126462+0000","last_active":"2024-04-17T07:29:28.118507+0000","last_peered":"2024-04-17T07:29:28.118507+0000","last_clean":"2024-04-17T07:29:28.118507+0000","last_became_active":"2024-04-17T07:29:27.124129+0000","last_became_peered":"2024-04-17T07:29:27.124129+0000","last_unstale":"2024-04-17T07:29:28.118507+0000","last_undegraded":"2024-04-17T07:29:28.118507+0000","last_fullsized":"2024-04-17T07:29:28.118507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:11.186057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847812+0000","last_change":"2024-04-17T07:29:26.089084+0000","last_active":"2024-04-17T07:29:28.847812+0000","last_peered":"2024-04-17T07:29:28.847812+0000","last_clean":"2024-04-17T07:29:28.847812+0000","last_became_active":"2024-04-17T07:29:26.088682+0000","last_became_peered":"2024-04-17T07:29:26.088682+0000","last_unstale":"2024-04-17T07:29:28.847812+0000","last_undegraded":"2024-04-17T07:29:28.847812+0000","last_fullsized":"2024-04-17T07:29:28.847812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:21:52.734778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118632+0000","last_change":"2024-04-17T07:29:27.123083+0000","last_active":"2024-04-17T07:29:28.118632+0000","last_peered":"2024-04-17T07:29:28.118632+0000","last_clean":"2024-04-17T07:29:28.118632+0000","last_became_active":"2024-04-17T07:29:27.114606+0000","last_became_peered":"2024-04-17T07:29:27.114606+0000","last_unstale":"2024-04-17T07:29:28.118632+0000","last_undegraded":"2024-04-17T07:29:28.118632+0000","last_fullsized":"2024-04-17T07:29:28.118632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:41:23.134601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849248+0000","last_change":"2024-04-17T07:29:26.095815+0000","last_active":"2024-04-17T07:29:28.849248+0000","last_peered":"2024-04-17T07:29:28.849248+0000","last_clean":"2024-04-17T07:29:28.849248+0000","last_became_active":"2024-04-17T07:29:26.095254+0000","last_became_peered":"2024-04-17T07:29:26.095254+0000","last_unstale":"2024-04-17T07:29:28.849248+0000","last_undegraded":"2024-04-17T07:29:28.849248+0000","last_fullsized":"2024-04-17T07:29:28.849248+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:53.455644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846402+0000","last_change":"2024-04-17T07:29:27.112818+0000","last_active":"2024-04-17T07:29:28.846402+0000","last_peered":"2024-04-17T07:29:28.846402+0000","last_clean":"2024-04-17T07:29:28.846402+0000","last_became_active":"2024-04-17T07:29:27.112514+0000","last_became_peered":"2024-04-17T07:29:27.112514+0000","last_unstale":"2024-04-17T07:29:28.846402+0000","last_undegraded":"2024-04-17T07:29:28.846402+0000","last_fullsized":"2024-04-17T07:29:28.846402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:12:12.066931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120309+0000","last_change":"2024-04-17T07:29:26.093741+0000","last_active":"2024-04-17T07:29:28.120309+0000","last_peered":"2024-04-17T07:29:28.120309+0000","last_clean":"2024-04-17T07:29:28.120309+0000","last_became_active":"2024-04-17T07:29:26.092840+0000","last_became_peered":"2024-04-17T07:29:26.092840+0000","last_unstale":"2024-04-17T07:29:28.120309+0000","last_undegraded":"2024-04-17T07:29:28.120309+0000","last_fullsized":"2024-04-17T07:29:28.120309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:51.647606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117845+0000","last_change":"2024-04-17T07:29:27.117212+0000","last_active":"2024-04-17T07:29:28.117845+0000","last_peered":"2024-04-17T07:29:28.117845+0000","last_clean":"2024-04-17T07:29:28.117845+0000","last_became_active":"2024-04-17T07:29:27.116833+0000","last_became_peered":"2024-04-17T07:29:27.116833+0000","last_unstale":"2024-04-17T07:29:28.117845+0000","last_undegraded":"2024-04-17T07:29:28.117845+0000","last_fullsized":"2024-04-17T07:29:28.117845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:16.252763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845606+0000","last_change":"2024-04-17T07:29:26.090724+0000","last_active":"2024-04-17T07:29:28.845606+0000","last_peered":"2024-04-17T07:29:28.845606+0000","last_clean":"2024-04-17T07:29:28.845606+0000","last_became_active":"2024-04-17T07:29:26.089486+0000","last_became_peered":"2024-04-17T07:29:26.089486+0000","last_unstale":"2024-04-17T07:29:28.845606+0000","last_undegraded":"2024-04-17T07:29:28.845606+0000","last_fullsized":"2024-04-17T07:29:28.845606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:31:47.889530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848292+0000","last_change":"2024-04-17T07:29:27.114364+0000","last_active":"2024-04-17T07:29:28.848292+0000","last_peered":"2024-04-17T07:29:28.848292+0000","last_clean":"2024-04-17T07:29:28.848292+0000","last_became_active":"2024-04-17T07:29:27.114055+0000","last_became_peered":"2024-04-17T07:29:27.114055+0000","last_unstale":"2024-04-17T07:29:28.848292+0000","last_undegraded":"2024-04-17T07:29:28.848292+0000","last_fullsized":"2024-04-17T07:29:28.848292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:14.337338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121029+0000","last_change":"2024-04-17T07:29:26.097035+0000","last_active":"2024-04-17T07:29:28.121029+0000","last_peered":"2024-04-17T07:29:28.121029+0000","last_clean":"2024-04-17T07:29:28.121029+0000","last_became_active":"2024-04-17T07:29:26.096751+0000","last_became_peered":"2024-04-17T07:29:26.096751+0000","last_unstale":"2024-04-17T07:29:28.121029+0000","last_undegraded":"2024-04-17T07:29:28.121029+0000","last_fullsized":"2024-04-17T07:29:28.121029+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:39.920812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120918+0000","last_change":"2024-04-17T07:29:27.130355+0000","last_active":"2024-04-17T07:29:28.120918+0000","last_peered":"2024-04-17T07:29:28.120918+0000","last_clean":"2024-04-17T07:29:28.120918+0000","last_became_active":"2024-04-17T07:29:27.129970+0000","last_became_peered":"2024-04-17T07:29:27.129970+0000","last_unstale":"2024-04-17T07:29:28.120918+0000","last_undegraded":"2024-04-17T07:29:28.120918+0000","last_fullsized":"2024-04-17T07:29:28.120918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:33.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119497+0000","last_change":"2024-04-17T07:29:26.095323+0000","last_active":"2024-04-17T07:29:28.119497+0000","last_peered":"2024-04-17T07:29:28.119497+0000","last_clean":"2024-04-17T07:29:28.119497+0000","last_became_active":"2024-04-17T07:29:26.094924+0000","last_became_peered":"2024-04-17T07:29:26.094924+0000","last_unstale":"2024-04-17T07:29:28.119497+0000","last_undegraded":"2024-04-17T07:29:28.119497+0000","last_fullsized":"2024-04-17T07:29:28.119497+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:17:34.892795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847550+0000","last_change":"2024-04-17T07:29:27.119851+0000","last_active":"2024-04-17T07:29:28.847550+0000","last_peered":"2024-04-17T07:29:28.847550+0000","last_clean":"2024-04-17T07:29:28.847550+0000","last_became_active":"2024-04-17T07:29:27.119332+0000","last_became_peered":"2024-04-17T07:29:27.119332+0000","last_unstale":"2024-04-17T07:29:28.847550+0000","last_undegraded":"2024-04-17T07:29:28.847550+0000","last_fullsized":"2024-04-17T07:29:28.847550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:35:26.594389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121395+0000","last_change":"2024-04-17T07:29:26.091377+0000","last_active":"2024-04-17T07:29:28.121395+0000","last_peered":"2024-04-17T07:29:28.121395+0000","last_clean":"2024-04-17T07:29:28.121395+0000","last_became_active":"2024-04-17T07:29:26.091045+0000","last_became_peered":"2024-04-17T07:29:26.091045+0000","last_unstale":"2024-04-17T07:29:28.121395+0000","last_undegraded":"2024-04-17T07:29:28.121395+0000","last_fullsized":"2024-04-17T07:29:28.121395+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:14.757061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120635+0000","last_change":"2024-04-17T07:29:27.130344+0000","last_active":"2024-04-17T07:29:28.120635+0000","last_peered":"2024-04-17T07:29:28.120635+0000","last_clean":"2024-04-17T07:29:28.120635+0000","last_became_active":"2024-04-17T07:29:27.130018+0000","last_became_peered":"2024-04-17T07:29:27.130018+0000","last_unstale":"2024-04-17T07:29:28.120635+0000","last_undegraded":"2024-04-17T07:29:28.120635+0000","last_fullsized":"2024-04-17T07:29:28.120635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:47:16.252763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"83'6","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847793+0000","last_change":"2024-04-17T07:29:26.085726+0000","last_active":"2024-04-17T07:29:28.847793+0000","last_peered":"2024-04-17T07:29:28.847793+0000","last_clean":"2024-04-17T07:29:28.847793+0000","last_became_active":"2024-04-17T07:29:26.085392+0000","last_became_peered":"2024-04-17T07:29:26.085392+0000","last_unstale":"2024-04-17T07:29:28.847793+0000","last_undegraded":"2024-04-17T07:29:28.847793+0000","last_fullsized":"2024-04-17T07:29:28.847793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:45.018010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120529+0000","last_change":"2024-04-17T07:29:27.117916+0000","last_active":"2024-04-17T07:29:28.120529+0000","last_peered":"2024-04-17T07:29:28.120529+0000","last_clean":"2024-04-17T07:29:28.120529+0000","last_became_active":"2024-04-17T07:29:27.117586+0000","last_became_peered":"2024-04-17T07:29:27.117586+0000","last_unstale":"2024-04-17T07:29:28.120529+0000","last_undegraded":"2024-04-17T07:29:28.120529+0000","last_fullsized":"2024-04-17T07:29:28.120529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:15:20.981413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118610+0000","last_change":"2024-04-17T07:29:26.096371+0000","last_active":"2024-04-17T07:29:28.118610+0000","last_peered":"2024-04-17T07:29:28.118610+0000","last_clean":"2024-04-17T07:29:28.118610+0000","last_became_active":"2024-04-17T07:29:26.095070+0000","last_became_peered":"2024-04-17T07:29:26.095070+0000","last_unstale":"2024-04-17T07:29:28.118610+0000","last_undegraded":"2024-04-17T07:29:28.118610+0000","last_fullsized":"2024-04-17T07:29:28.118610+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:55.256610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119192+0000","last_change":"2024-04-17T07:29:27.116477+0000","last_active":"2024-04-17T07:29:28.119192+0000","last_peered":"2024-04-17T07:29:28.119192+0000","last_clean":"2024-04-17T07:29:28.119192+0000","last_became_active":"2024-04-17T07:29:27.115928+0000","last_became_peered":"2024-04-17T07:29:27.115928+0000","last_unstale":"2024-04-17T07:29:28.119192+0000","last_undegraded":"2024-04-17T07:29:28.119192+0000","last_fullsized":"2024-04-17T07:29:28.119192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:03.728886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846620+0000","last_change":"2024-04-17T07:29:26.093862+0000","last_active":"2024-04-17T07:29:28.846620+0000","last_peered":"2024-04-17T07:29:28.846620+0000","last_clean":"2024-04-17T07:29:28.846620+0000","last_became_active":"2024-04-17T07:29:26.092195+0000","last_became_peered":"2024-04-17T07:29:26.092195+0000","last_unstale":"2024-04-17T07:29:28.846620+0000","last_undegraded":"2024-04-17T07:29:28.846620+0000","last_fullsized":"2024-04-17T07:29:28.846620+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:12.339387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845606+0000","last_change":"2024-04-17T07:29:27.112844+0000","last_active":"2024-04-17T07:29:28.845606+0000","last_peered":"2024-04-17T07:29:28.845606+0000","last_clean":"2024-04-17T07:29:28.845606+0000","last_became_active":"2024-04-17T07:29:27.112549+0000","last_became_peered":"2024-04-17T07:29:27.112549+0000","last_unstale":"2024-04-17T07:29:28.845606+0000","last_undegraded":"2024-04-17T07:29:28.845606+0000","last_fullsized":"2024-04-17T07:29:28.845606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:25.626227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120456+0000","last_change":"2024-04-17T07:29:26.092792+0000","last_active":"2024-04-17T07:29:28.120456+0000","last_peered":"2024-04-17T07:29:28.120456+0000","last_clean":"2024-04-17T07:29:28.120456+0000","last_became_active":"2024-04-17T07:29:26.091708+0000","last_became_peered":"2024-04-17T07:29:26.091708+0000","last_unstale":"2024-04-17T07:29:28.120456+0000","last_undegraded":"2024-04-17T07:29:28.120456+0000","last_fullsized":"2024-04-17T07:29:28.120456+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:07.572381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119722+0000","last_change":"2024-04-17T07:29:27.125408+0000","last_active":"2024-04-17T07:29:28.119722+0000","last_peered":"2024-04-17T07:29:28.119722+0000","last_clean":"2024-04-17T07:29:28.119722+0000","last_became_active":"2024-04-17T07:29:27.125113+0000","last_became_peered":"2024-04-17T07:29:27.125113+0000","last_unstale":"2024-04-17T07:29:28.119722+0000","last_undegraded":"2024-04-17T07:29:28.119722+0000","last_fullsized":"2024-04-17T07:29:28.119722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:17:45.846224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"83'8","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118789+0000","last_change":"2024-04-17T07:29:26.097547+0000","last_active":"2024-04-17T07:29:28.118789+0000","last_peered":"2024-04-17T07:29:28.118789+0000","last_clean":"2024-04-17T07:29:28.118789+0000","last_became_active":"2024-04-17T07:29:26.095575+0000","last_became_peered":"2024-04-17T07:29:26.095575+0000","last_unstale":"2024-04-17T07:29:28.118789+0000","last_undegraded":"2024-04-17T07:29:28.118789+0000","last_fullsized":"2024-04-17T07:29:28.118789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:57.584344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120500+0000","last_change":"2024-04-17T07:29:27.120867+0000","last_active":"2024-04-17T07:29:28.120500+0000","last_peered":"2024-04-17T07:29:28.120500+0000","last_clean":"2024-04-17T07:29:28.120500+0000","last_became_active":"2024-04-17T07:29:27.120574+0000","last_became_peered":"2024-04-17T07:29:27.120574+0000","last_unstale":"2024-04-17T07:29:28.120500+0000","last_undegraded":"2024-04-17T07:29:28.120500+0000","last_fullsized":"2024-04-17T07:29:28.120500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:28.723379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119761+0000","last_change":"2024-04-17T07:29:26.097309+0000","last_active":"2024-04-17T07:29:28.119761+0000","last_peered":"2024-04-17T07:29:28.119761+0000","last_clean":"2024-04-17T07:29:28.119761+0000","last_became_active":"2024-04-17T07:29:26.095696+0000","last_became_peered":"2024-04-17T07:29:26.095696+0000","last_unstale":"2024-04-17T07:29:28.119761+0000","last_undegraded":"2024-04-17T07:29:28.119761+0000","last_fullsized":"2024-04-17T07:29:28.119761+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:39.642342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121866+0000","last_change":"2024-04-17T07:29:27.117536+0000","last_active":"2024-04-17T07:29:28.121866+0000","last_peered":"2024-04-17T07:29:28.121866+0000","last_clean":"2024-04-17T07:29:28.121866+0000","last_became_active":"2024-04-17T07:29:27.117213+0000","last_became_peered":"2024-04-17T07:29:27.117213+0000","last_unstale":"2024-04-17T07:29:28.121866+0000","last_undegraded":"2024-04-17T07:29:28.121866+0000","last_fullsized":"2024-04-17T07:29:28.121866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:58.599330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.850058+0000","last_change":"2024-04-17T07:29:26.093682+0000","last_active":"2024-04-17T07:29:28.850058+0000","last_peered":"2024-04-17T07:29:28.850058+0000","last_clean":"2024-04-17T07:29:28.850058+0000","last_became_active":"2024-04-17T07:29:26.090291+0000","last_became_peered":"2024-04-17T07:29:26.090291+0000","last_unstale":"2024-04-17T07:29:28.850058+0000","last_undegraded":"2024-04-17T07:29:28.850058+0000","last_fullsized":"2024-04-17T07:29:28.850058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:49:17.064563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119180+0000","last_change":"2024-04-17T07:29:27.119864+0000","last_active":"2024-04-17T07:29:28.119180+0000","last_peered":"2024-04-17T07:29:28.119180+0000","last_clean":"2024-04-17T07:29:28.119180+0000","last_became_active":"2024-04-17T07:29:27.118284+0000","last_became_peered":"2024-04-17T07:29:27.118284+0000","last_unstale":"2024-04-17T07:29:28.119180+0000","last_undegraded":"2024-04-17T07:29:28.119180+0000","last_fullsized":"2024-04-17T07:29:28.119180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:04:55.685793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121457+0000","last_change":"2024-04-17T07:29:26.092015+0000","last_active":"2024-04-17T07:29:28.121457+0000","last_peered":"2024-04-17T07:29:28.121457+0000","last_clean":"2024-04-17T07:29:28.121457+0000","last_became_active":"2024-04-17T07:29:26.091433+0000","last_became_peered":"2024-04-17T07:29:26.091433+0000","last_unstale":"2024-04-17T07:29:28.121457+0000","last_undegraded":"2024-04-17T07:29:28.121457+0000","last_fullsized":"2024-04-17T07:29:28.121457+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:40.112094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121047+0000","last_change":"2024-04-17T07:29:27.132427+0000","last_active":"2024-04-17T07:29:28.121047+0000","last_peered":"2024-04-17T07:29:28.121047+0000","last_clean":"2024-04-17T07:29:28.121047+0000","last_became_active":"2024-04-17T07:29:27.132208+0000","last_became_peered":"2024-04-17T07:29:27.132208+0000","last_unstale":"2024-04-17T07:29:28.121047+0000","last_undegraded":"2024-04-17T07:29:28.121047+0000","last_fullsized":"2024-04-17T07:29:28.121047+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:53.439903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847226+0000","last_change":"2024-04-17T07:29:26.096974+0000","last_active":"2024-04-17T07:29:28.847226+0000","last_peered":"2024-04-17T07:29:28.847226+0000","last_clean":"2024-04-17T07:29:28.847226+0000","last_became_active":"2024-04-17T07:29:26.096715+0000","last_became_peered":"2024-04-17T07:29:26.096715+0000","last_unstale":"2024-04-17T07:29:28.847226+0000","last_undegraded":"2024-04-17T07:29:28.847226+0000","last_fullsized":"2024-04-17T07:29:28.847226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:55.796473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120859+0000","last_change":"2024-04-17T07:29:27.116830+0000","last_active":"2024-04-17T07:29:28.120859+0000","last_peered":"2024-04-17T07:29:28.120859+0000","last_clean":"2024-04-17T07:29:28.120859+0000","last_became_active":"2024-04-17T07:29:27.116566+0000","last_became_peered":"2024-04-17T07:29:27.116566+0000","last_unstale":"2024-04-17T07:29:28.120859+0000","last_undegraded":"2024-04-17T07:29:28.120859+0000","last_fullsized":"2024-04-17T07:29:28.120859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:46.916850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118892+0000","last_change":"2024-04-17T07:29:26.090683+0000","last_active":"2024-04-17T07:29:28.118892+0000","last_peered":"2024-04-17T07:29:28.118892+0000","last_clean":"2024-04-17T07:29:28.118892+0000","last_became_active":"2024-04-17T07:29:26.090420+0000","last_became_peered":"2024-04-17T07:29:26.090420+0000","last_unstale":"2024-04-17T07:29:28.118892+0000","last_undegraded":"2024-04-17T07:29:28.118892+0000","last_fullsized":"2024-04-17T07:29:28.118892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:19.542838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.848291+0000","last_change":"2024-04-17T07:29:27.114402+0000","last_active":"2024-04-17T07:29:28.848291+0000","last_peered":"2024-04-17T07:29:28.848291+0000","last_clean":"2024-04-17T07:29:28.848291+0000","last_became_active":"2024-04-17T07:29:27.114078+0000","last_became_peered":"2024-04-17T07:29:27.114078+0000","last_unstale":"2024-04-17T07:29:28.848291+0000","last_undegraded":"2024-04-17T07:29:28.848291+0000","last_fullsized":"2024-04-17T07:29:28.848291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:52.801878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847417+0000","last_change":"2024-04-17T07:29:26.088859+0000","last_active":"2024-04-17T07:29:28.847417+0000","last_peered":"2024-04-17T07:29:28.847417+0000","last_clean":"2024-04-17T07:29:28.847417+0000","last_became_active":"2024-04-17T07:29:26.088533+0000","last_became_peered":"2024-04-17T07:29:26.088533+0000","last_unstale":"2024-04-17T07:29:28.847417+0000","last_undegraded":"2024-04-17T07:29:28.847417+0000","last_fullsized":"2024-04-17T07:29:28.847417+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:34.615808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846314+0000","last_change":"2024-04-17T07:29:27.111112+0000","last_active":"2024-04-17T07:29:28.846314+0000","last_peered":"2024-04-17T07:29:28.846314+0000","last_clean":"2024-04-17T07:29:28.846314+0000","last_became_active":"2024-04-17T07:29:27.110821+0000","last_became_peered":"2024-04-17T07:29:27.110821+0000","last_unstale":"2024-04-17T07:29:28.846314+0000","last_undegraded":"2024-04-17T07:29:28.846314+0000","last_fullsized":"2024-04-17T07:29:28.846314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:36.820348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.120131+0000","last_change":"2024-04-17T07:29:26.096511+0000","last_active":"2024-04-17T07:29:28.120131+0000","last_peered":"2024-04-17T07:29:28.120131+0000","last_clean":"2024-04-17T07:29:28.120131+0000","last_became_active":"2024-04-17T07:29:26.095227+0000","last_became_peered":"2024-04-17T07:29:26.095227+0000","last_unstale":"2024-04-17T07:29:28.120131+0000","last_undegraded":"2024-04-17T07:29:28.120131+0000","last_fullsized":"2024-04-17T07:29:28.120131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:47.774884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121929+0000","last_change":"2024-04-17T07:29:27.135341+0000","last_active":"2024-04-17T07:29:28.121929+0000","last_peered":"2024-04-17T07:29:28.121929+0000","last_clean":"2024-04-17T07:29:28.121929+0000","last_became_active":"2024-04-17T07:29:27.134993+0000","last_became_peered":"2024-04-17T07:29:27.134993+0000","last_unstale":"2024-04-17T07:29:28.121929+0000","last_undegraded":"2024-04-17T07:29:28.121929+0000","last_fullsized":"2024-04-17T07:29:28.121929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:40.935799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"83'2","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.849085+0000","last_change":"2024-04-17T07:29:26.094172+0000","last_active":"2024-04-17T07:29:28.849085+0000","last_peered":"2024-04-17T07:29:28.849085+0000","last_clean":"2024-04-17T07:29:28.849085+0000","last_became_active":"2024-04-17T07:29:26.093851+0000","last_became_peered":"2024-04-17T07:29:26.093851+0000","last_unstale":"2024-04-17T07:29:28.849085+0000","last_undegraded":"2024-04-17T07:29:28.849085+0000","last_fullsized":"2024-04-17T07:29:28.849085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:44.729310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.122030+0000","last_change":"2024-04-17T07:29:27.112876+0000","last_active":"2024-04-17T07:29:28.122030+0000","last_peered":"2024-04-17T07:29:28.122030+0000","last_clean":"2024-04-17T07:29:28.122030+0000","last_became_active":"2024-04-17T07:29:27.112604+0000","last_became_peered":"2024-04-17T07:29:27.112604+0000","last_unstale":"2024-04-17T07:29:28.122030+0000","last_undegraded":"2024-04-17T07:29:28.122030+0000","last_fullsized":"2024-04-17T07:29:28.122030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:15.772047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"84'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:29.096699+0000","last_change":"2024-04-17T07:29:26.095078+0000","last_active":"2024-04-17T07:29:29.096699+0000","last_peered":"2024-04-17T07:29:29.096699+0000","last_clean":"2024-04-17T07:29:29.096699+0000","last_became_active":"2024-04-17T07:29:26.094818+0000","last_became_peered":"2024-04-17T07:29:26.094818+0000","last_unstale":"2024-04-17T07:29:29.096699+0000","last_undegraded":"2024-04-17T07:29:29.096699+0000","last_fullsized":"2024-04-17T07:29:29.096699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:24.202316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847756+0000","last_change":"2024-04-17T07:29:27.119627+0000","last_active":"2024-04-17T07:29:28.847756+0000","last_peered":"2024-04-17T07:29:28.847756+0000","last_clean":"2024-04-17T07:29:28.847756+0000","last_became_active":"2024-04-17T07:29:27.118954+0000","last_became_peered":"2024-04-17T07:29:27.118954+0000","last_unstale":"2024-04-17T07:29:28.847756+0000","last_undegraded":"2024-04-17T07:29:28.847756+0000","last_fullsized":"2024-04-17T07:29:28.847756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:02.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846646+0000","last_change":"2024-04-17T07:29:26.090200+0000","last_active":"2024-04-17T07:29:28.846646+0000","last_peered":"2024-04-17T07:29:28.846646+0000","last_clean":"2024-04-17T07:29:28.846646+0000","last_became_active":"2024-04-17T07:29:26.089907+0000","last_became_peered":"2024-04-17T07:29:26.089907+0000","last_unstale":"2024-04-17T07:29:28.846646+0000","last_undegraded":"2024-04-17T07:29:28.846646+0000","last_fullsized":"2024-04-17T07:29:28.846646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:58.559638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847494+0000","last_change":"2024-04-17T07:29:27.100514+0000","last_active":"2024-04-17T07:29:28.847494+0000","last_peered":"2024-04-17T07:29:28.847494+0000","last_clean":"2024-04-17T07:29:28.847494+0000","last_became_active":"2024-04-17T07:29:27.099313+0000","last_became_peered":"2024-04-17T07:29:27.099313+0000","last_unstale":"2024-04-17T07:29:28.847494+0000","last_undegraded":"2024-04-17T07:29:28.847494+0000","last_fullsized":"2024-04-17T07:29:28.847494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:47.210435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.121764+0000","last_change":"2024-04-17T07:29:26.080436+0000","last_active":"2024-04-17T07:29:28.121764+0000","last_peered":"2024-04-17T07:29:28.121764+0000","last_clean":"2024-04-17T07:29:28.121764+0000","last_became_active":"2024-04-17T07:29:26.080137+0000","last_became_peered":"2024-04-17T07:29:26.080137+0000","last_unstale":"2024-04-17T07:29:28.121764+0000","last_undegraded":"2024-04-17T07:29:28.121764+0000","last_fullsized":"2024-04-17T07:29:28.121764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:32:38.414749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118549+0000","last_change":"2024-04-17T07:29:27.123716+0000","last_active":"2024-04-17T07:29:28.118549+0000","last_peered":"2024-04-17T07:29:28.118549+0000","last_clean":"2024-04-17T07:29:28.118549+0000","last_became_active":"2024-04-17T07:29:27.123440+0000","last_became_peered":"2024-04-17T07:29:27.123440+0000","last_unstale":"2024-04-17T07:29:28.118549+0000","last_undegraded":"2024-04-17T07:29:28.118549+0000","last_fullsized":"2024-04-17T07:29:28.118549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:54.006694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846852+0000","last_change":"2024-04-17T07:29:26.079687+0000","last_active":"2024-04-17T07:29:28.846852+0000","last_peered":"2024-04-17T07:29:28.846852+0000","last_clean":"2024-04-17T07:29:28.846852+0000","last_became_active":"2024-04-17T07:29:26.079372+0000","last_became_peered":"2024-04-17T07:29:26.079372+0000","last_unstale":"2024-04-17T07:29:28.846852+0000","last_undegraded":"2024-04-17T07:29:28.846852+0000","last_fullsized":"2024-04-17T07:29:28.846852+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:55.688138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.119369+0000","last_change":"2024-04-17T07:29:27.126613+0000","last_active":"2024-04-17T07:29:28.119369+0000","last_peered":"2024-04-17T07:29:28.119369+0000","last_clean":"2024-04-17T07:29:28.119369+0000","last_became_active":"2024-04-17T07:29:27.126235+0000","last_became_peered":"2024-04-17T07:29:27.126235+0000","last_unstale":"2024-04-17T07:29:28.119369+0000","last_undegraded":"2024-04-17T07:29:28.119369+0000","last_fullsized":"2024-04-17T07:29:28.119369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.847974+0000","last_change":"2024-04-17T07:29:26.097087+0000","last_active":"2024-04-17T07:29:28.847974+0000","last_peered":"2024-04-17T07:29:28.847974+0000","last_clean":"2024-04-17T07:29:28.847974+0000","last_became_active":"2024-04-17T07:29:26.096536+0000","last_became_peered":"2024-04-17T07:29:26.096536+0000","last_unstale":"2024-04-17T07:29:28.847974+0000","last_undegraded":"2024-04-17T07:29:28.847974+0000","last_fullsized":"2024-04-17T07:29:28.847974+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:08.636535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.845967+0000","last_change":"2024-04-17T07:29:27.116889+0000","last_active":"2024-04-17T07:29:28.845967+0000","last_peered":"2024-04-17T07:29:28.845967+0000","last_clean":"2024-04-17T07:29:28.845967+0000","last_became_active":"2024-04-17T07:29:27.115083+0000","last_became_peered":"2024-04-17T07:29:27.115083+0000","last_unstale":"2024-04-17T07:29:28.845967+0000","last_undegraded":"2024-04-17T07:29:28.845967+0000","last_fullsized":"2024-04-17T07:29:28.845967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:35.508241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846483+0000","last_change":"2024-04-17T07:29:26.080915+0000","last_active":"2024-04-17T07:29:28.846483+0000","last_peered":"2024-04-17T07:29:28.846483+0000","last_clean":"2024-04-17T07:29:28.846483+0000","last_became_active":"2024-04-17T07:29:26.080470+0000","last_became_peered":"2024-04-17T07:29:26.080470+0000","last_unstale":"2024-04-17T07:29:28.846483+0000","last_undegraded":"2024-04-17T07:29:28.846483+0000","last_fullsized":"2024-04-17T07:29:28.846483+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:33.269102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117908+0000","last_change":"2024-04-17T07:29:27.117132+0000","last_active":"2024-04-17T07:29:28.117908+0000","last_peered":"2024-04-17T07:29:28.117908+0000","last_clean":"2024-04-17T07:29:28.117908+0000","last_became_active":"2024-04-17T07:29:27.116469+0000","last_became_peered":"2024-04-17T07:29:27.116469+0000","last_unstale":"2024-04-17T07:29:28.117908+0000","last_undegraded":"2024-04-17T07:29:28.117908+0000","last_fullsized":"2024-04-17T07:29:28.117908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:33.137982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118050+0000","last_change":"2024-04-17T07:29:26.088644+0000","last_active":"2024-04-17T07:29:28.118050+0000","last_peered":"2024-04-17T07:29:28.118050+0000","last_clean":"2024-04-17T07:29:28.118050+0000","last_became_active":"2024-04-17T07:29:26.088306+0000","last_became_peered":"2024-04-17T07:29:26.088306+0000","last_unstale":"2024-04-17T07:29:28.118050+0000","last_undegraded":"2024-04-17T07:29:28.118050+0000","last_fullsized":"2024-04-17T07:29:28.118050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:21:14.757061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.117818+0000","last_change":"2024-04-17T07:29:27.116094+0000","last_active":"2024-04-17T07:29:28.117818+0000","last_peered":"2024-04-17T07:29:28.117818+0000","last_clean":"2024-04-17T07:29:28.117818+0000","last_became_active":"2024-04-17T07:29:27.115395+0000","last_became_peered":"2024-04-17T07:29:27.115395+0000","last_unstale":"2024-04-17T07:29:28.117818+0000","last_undegraded":"2024-04-17T07:29:28.117818+0000","last_fullsized":"2024-04-17T07:29:28.117818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:26.048118+0000","last_clean_scrub_stamp":"2024-04-17T07:29:26.048118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.442353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.846320+0000","last_change":"2024-04-17T07:29:26.096004+0000","last_active":"2024-04-17T07:29:28.846320+0000","last_peered":"2024-04-17T07:29:28.846320+0000","last_clean":"2024-04-17T07:29:28.846320+0000","last_became_active":"2024-04-17T07:29:26.095465+0000","last_became_peered":"2024-04-17T07:29:26.095465+0000","last_unstale":"2024-04-17T07:29:28.846320+0000","last_undegraded":"2024-04-17T07:29:28.846320+0000","last_fullsized":"2024-04-17T07:29:28.846320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:29:25.033132+0000","last_clean_scrub_stamp":"2024-04-17T07:29:25.033132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:08:13.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"72'161","reported_seq":452,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-17T07:29:28.118921+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:29:28.118921+0000","last_peered":"2024-04-17T07:29:28.118921+0000","last_clean":"2024-04-17T07:29:28.118921+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:29:28.118921+0000","last_undegraded":"2024-04-17T07:29:28.118921+0000","last_fullsized":"2024-04-17T07:29:28.118921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542333,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1613824,"data_stored":1071116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":480588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542332,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":481240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542332,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58548,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313292,"statfs":{"total":96636764160,"available":96576811008,"internally_reserved":0,"allocated":1024000,"data_stored":482200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":480644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542333,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40496,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331344,"statfs":{"total":96636764160,"available":96595296256,"internally_reserved":0,"allocated":1019904,"data_stored":480644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542333,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50220,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321620,"statfs":{"total":96636764160,"available":96585338880,"internally_reserved":0,"allocated":1015808,"data_stored":480678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542332,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1622016,"data_stored":1073150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:29:32.162 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:29:32.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:29:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.434+0000 7f6596c00640 1 Processor -- start 2024-04-17T07:29:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.434+0000 7f6596c00640 1 -- start start 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f659013f6d0 0x7f6590071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65900b27c0 0x7f65900cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f659015cbe0 con 0x7f65900cd5d0 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f659015cea0 con 0x7f65900b27c0 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f659015d180 con 0x7f659013f6d0 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65900b27c0 0x7f65900cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65900b27c0 0x7f65900cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59384/0 (socket says 172.21.15.67:59384) 2024-04-17T07:29:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.436+0000 7f6596200640 1 -- 172.21.15.67:0/1245129989 learned_addr learned my addr 172.21.15.67:0/1245129989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.437+0000 7f6595800640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f659013f6d0 0x7f6590071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.437+0000 7f6596200640 1 -- 172.21.15.67:0/1245129989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f659013f6d0 msgr2=0x7f6590071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.437+0000 7f6596200640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f659013f6d0 0x7f6590071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.437+0000 7f6596200640 1 -- 172.21.15.67:0/1245129989 >> v1:172.21.15.67:6789/0 conn(0x7f65900cd5d0 legacy=0x7f65900b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.437+0000 7f6596200640 1 -- 172.21.15.67:0/1245129989 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659015d460 con 0x7f65900b27c0 2024-04-17T07:29:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.438+0000 7f6596200640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65900b27c0 0x7f65900cda30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6588009e50 tx=0x7f658802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37a40b7d0785703 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.438+0000 7f658fe00640 1 -- 172.21.15.67:0/1245129989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658802ca80 con 0x7f65900b27c0 2024-04-17T07:29:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.438+0000 7f658fe00640 1 -- 172.21.15.67:0/1245129989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658802cc20 con 0x7f65900b27c0 2024-04-17T07:29:32.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.438+0000 7f658fe00640 1 -- 172.21.15.67:0/1245129989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65880343f0 con 0x7f65900b27c0 2024-04-17T07:29:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 -- 172.21.15.67:0/1245129989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65900b27c0 msgr2=0x7f65900cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65900b27c0 0x7f65900cda30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6588009e50 tx=0x7f658802bee0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 -- 172.21.15.67:0/1245129989 shutdown_connections 2024-04-17T07:29:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65900b27c0 0x7f65900cda30 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 --2- 172.21.15.67:0/1245129989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f659013f6d0 0x7f6590071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 -- 172.21.15.67:0/1245129989 >> 172.21.15.67:0/1245129989 conn(0x7f6590067d10 msgr2=0x7f65900d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 -- 172.21.15.67:0/1245129989 shutdown_connections 2024-04-17T07:29:32.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.439+0000 7f6596c00640 1 -- 172.21.15.67:0/1245129989 wait complete. 2024-04-17T07:29:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.441+0000 7f6596c00640 1 Processor -- start 2024-04-17T07:29:32.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.441+0000 7f6596c00640 1 -- start start 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 0x7f65901568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65900cd5d0 0x7f659013e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f659015da20 con 0x7f659013f6d0 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f65900d1120 con 0x7f65900b27c0 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6590142020 con 0x7f65900cd5d0 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59400/0 (socket says 172.21.15.67:59400) 2024-04-17T07:29:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.442+0000 7f6596200640 1 -- 172.21.15.67:0/1565953655 learned_addr learned my addr 172.21.15.67:0/1565953655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6594e00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65900cd5d0 0x7f659013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6595800640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 0x7f65901568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 -- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65900cd5d0 msgr2=0x7f659013e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65900cd5d0 0x7f659013e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 -- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 msgr2=0x7f65901568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 0x7f65901568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 -- 172.21.15.67:0/1565953655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6590141550 con 0x7f659013f6d0 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6595800640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 0x7f65901568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:32.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f6596200640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6588036040 tx=0x7f6588004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.443+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588043020 con 0x7f659013f6d0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.444+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658803e450 con 0x7f659013f6d0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.444+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588009510 con 0x7f659013f6d0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.444+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6590141830 con 0x7f659013f6d0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.444+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6590155e80 con 0x7f659013f6d0 2024-04-17T07:29:32.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.445+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659015e6f0 con 0x7f659013f6d0 2024-04-17T07:29:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.446+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f6588046020 con 0x7f659013f6d0 2024-04-17T07:29:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.447+0000 7f658d600640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 0x7f657406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.447+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f65880ae510 con 0x7f659013f6d0 2024-04-17T07:29:32.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.448+0000 7f6595800640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 0x7f657406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.449+0000 7f6595800640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 0x7f657406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f658400b820 tx=0x7f6584008040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.451+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f659015e6f0 con 0x7f659013f6d0 2024-04-17T07:29:32.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.740+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f6590160360 con 0x7f659013f6d0 2024-04-17T07:29:32.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.745+0000 7f658d600640 1 -- 172.21.15.67:0/1565953655 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f658804d030 con 0x7f659013f6d0 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:29:32.745 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:29:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 msgr2=0x7f657406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 0x7f657406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f658400b820 tx=0x7f6584008040 comp rx=0 tx=0).stop 2024-04-17T07:29:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 msgr2=0x7f65900b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6588036040 tx=0x7f6588004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 shutdown_connections 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f657406ca10 0x7f657406eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f659013f6d0 0x7f65900b10e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65900cd5d0 0x7f659013e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 --2- 172.21.15.67:0/1565953655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65900b27c0 0x7f65901568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.748+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 >> 172.21.15.67:0/1565953655 conn(0x7f6590067d10 msgr2=0x7f659015ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.749+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 shutdown_connections 2024-04-17T07:29:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.749+0000 7f6596c00640 1 -- 172.21.15.67:0/1565953655 wait complete. 2024-04-17T07:29:32.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:32.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.996+0000 7faa28c00640 1 Processor -- start 2024-04-17T07:29:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.997+0000 7faa28c00640 1 -- start start 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa2415ec60 0x7faa240b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faa240b6b30 0x7faa240cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa240cf910 0x7faa2406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faa240d49c0 con 0x7faa2415ec60 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faa240d3660 con 0x7faa240b6b30 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa28c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faa240b6f70 con 0x7faa240cf910 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.998+0000 7faa23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa2415ec60 0x7faa240b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa2415ec60 0x7faa240b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59404/0 (socket says 172.21.15.67:59404) 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 -- 172.21.15.67:0/1384313307 learned_addr learned my addr 172.21.15.67:0/1384313307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 -- 172.21.15.67:0/1384313307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa240cf910 msgr2=0x7faa2406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 --2- 172.21.15.67:0/1384313307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa240cf910 0x7faa2406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 -- 172.21.15.67:0/1384313307 >> v2:172.21.15.187:3300/0 conn(0x7faa240b6b30 msgr2=0x7faa240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 --2- 172.21.15.67:0/1384313307 >> v2:172.21.15.187:3300/0 conn(0x7faa240b6b30 0x7faa240cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:32.999+0000 7faa23400640 1 -- 172.21.15.67:0/1384313307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa240d2d30 con 0x7faa2415ec60 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.000+0000 7faa23400640 1 --2- 172.21.15.67:0/1384313307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa2415ec60 0x7faa240b65f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faa240ba590 tx=0x7faa0c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fffb0817780da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.000+0000 7faa22000640 1 -- 172.21.15.67:0/1384313307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c035020 con 0x7faa2415ec60 2024-04-17T07:29:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.000+0000 7faa22000640 1 -- 172.21.15.67:0/1384313307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa0c031880 con 0x7faa2415ec60 2024-04-17T07:29:33.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.000+0000 7faa22000640 1 -- 172.21.15.67:0/1384313307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c031c50 con 0x7faa2415ec60 2024-04-17T07:29:33.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 -- 172.21.15.67:0/1384313307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa2415ec60 msgr2=0x7faa240b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 --2- 172.21.15.67:0/1384313307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa2415ec60 0x7faa240b65f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faa240ba590 tx=0x7faa0c02b920 comp rx=0 tx=0).stop 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 -- 172.21.15.67:0/1384313307 shutdown_connections 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 --2- 172.21.15.67:0/1384313307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa240cf910 0x7faa2406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 --2- 172.21.15.67:0/1384313307 >> v2:172.21.15.187:3300/0 conn(0x7faa240b6b30 0x7faa240cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 --2- 172.21.15.67:0/1384313307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa2415ec60 0x7faa240b65f0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 -- 172.21.15.67:0/1384313307 >> 172.21.15.67:0/1384313307 conn(0x7faa240d15f0 msgr2=0x7faa240d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 -- 172.21.15.67:0/1384313307 shutdown_connections 2024-04-17T07:29:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.001+0000 7faa28c00640 1 -- 172.21.15.67:0/1384313307 wait complete. 2024-04-17T07:29:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.003+0000 7faa28c00640 1 Processor -- start 2024-04-17T07:29:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.003+0000 7faa28c00640 1 -- start start 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa240cf910 0x7faa240ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa2415ec60 0x7faa240b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faa240cfda0 con 0x7faa240b6b30 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faa240d32f0 con 0x7faa240cf910 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.004+0000 7faa28c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faa2415f040 con 0x7faa2415ec60 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59418/0 (socket says 172.21.15.67:59418) 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 -- 172.21.15.67:0/3289381796 learned_addr learned my addr 172.21.15.67:0/3289381796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa22a00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa240cf910 0x7faa240ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 -- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa2415ec60 msgr2=0x7faa240b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa2415ec60 0x7faa240b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 -- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa240cf910 msgr2=0x7faa240ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa240cf910 0x7faa240ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 -- 172.21.15.67:0/3289381796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241f9410 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.005+0000 7faa23400640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0090b0 tx=0x7faa0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.006+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0443f0 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.006+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa0c0445b0 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.006+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0449a0 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.006+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa240cadc0 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.006+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa240ce030 con 0x7faa240b6b30 2024-04-17T07:29:33.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.007+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa24068a10 con 0x7faa240b6b30 2024-04-17T07:29:33.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.008+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7faa0c04b060 con 0x7faa240b6b30 2024-04-17T07:29:33.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.009+0000 7faa1b400640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 0x7faa0006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.009+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7faa0c0adf50 con 0x7faa240b6b30 2024-04-17T07:29:33.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.010+0000 7faa22a00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 0x7faa0006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.011+0000 7faa22a00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 0x7faa0006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa240d0200 tx=0x7faa140072f0 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.014+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faa240cadc0 con 0x7faa240b6b30 2024-04-17T07:29:33.038 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-17T07:29:33.039+0000 7efd57400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:29:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.263+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa2415e530 con 0x7faa240b6b30 2024-04-17T07:29:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.265+0000 7faa1b400640 1 -- 172.21.15.67:0/3289381796 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7faa0c03f070 con 0x7faa240b6b30 2024-04-17T07:29:33.264 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:33.264 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.267+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 msgr2=0x7faa0006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.267+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 0x7faa0006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa240d0200 tx=0x7faa140072f0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.267+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 msgr2=0x7faa240ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.267+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0090b0 tx=0x7faa0c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 shutdown_connections 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7faa0006ca10 0x7faa0006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa2415ec60 0x7faa240b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa240cf910 0x7faa240ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 --2- 172.21.15.67:0/3289381796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa240b6b30 0x7faa240ceb10 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 >> 172.21.15.67:0/3289381796 conn(0x7faa240d15f0 msgr2=0x7faa240ba040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 shutdown_connections 2024-04-17T07:29:33.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.268+0000 7faa28c00640 1 -- 172.21.15.67:0/3289381796 wait complete. 2024-04-17T07:29:33.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:29:26.551509+0000', 'modified': '2024-04-17T07:29:29.712215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6444': {'gid': 6444, 'name': 'd', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/2437593714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 2437593714}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 2437593714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6444, 'qdb_cluster': [6444]} 2024-04-17T07:29:33.689 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:29:33.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T07:29:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.923+0000 7f0b37200640 1 Processor -- start 2024-04-17T07:29:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.923+0000 7f0b37200640 1 -- start start 2024-04-17T07:29:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.924+0000 7f0b37200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b37200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0b3815fad0 0x7f0b380ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b37200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b380cb420 con 0x7f0b3806eec0 2024-04-17T07:29:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b37200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0b380cc030 con 0x7f0b3815fad0 2024-04-17T07:29:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b37200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b380cb700 con 0x7f0b380afbf0 2024-04-17T07:29:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.925+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:40250/0 (socket says 172.21.15.67:40250) 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b35e00640 1 -- 172.21.15.67:0/2824072908 learned_addr learned my addr 172.21.15.67:0/2824072908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b36800640 1 --2- 172.21.15.67:0/2824072908 >> v2:172.21.15.187:3300/0 conn(0x7f0b3815fad0 0x7f0b380ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b35e00640 1 -- 172.21.15.67:0/2824072908 >> v2:172.21.15.187:3300/0 conn(0x7f0b3815fad0 msgr2=0x7f0b380ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2824072908 >> v2:172.21.15.187:3300/0 conn(0x7f0b3815fad0 0x7f0b380ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b35e00640 1 -- 172.21.15.67:0/2824072908 >> v1:172.21.15.187:6789/0 conn(0x7f0b3806eec0 legacy=0x7f0b3815f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.926+0000 7f0b35e00640 1 -- 172.21.15.67:0/2824072908 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b380cb9e0 con 0x7f0b380afbf0 2024-04-17T07:29:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.928+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2824072908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009e50 tx=0x7f0b2802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df63071fbf48e4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.928+0000 7f0b35400640 1 -- 172.21.15.67:0/2824072908 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2802cb00 con 0x7f0b380afbf0 2024-04-17T07:29:33.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.928+0000 7f0b35400640 1 -- 172.21.15.67:0/2824072908 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2802cca0 con 0x7f0b380afbf0 2024-04-17T07:29:33.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.928+0000 7f0b35400640 1 -- 172.21.15.67:0/2824072908 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28034610 con 0x7f0b380afbf0 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 -- 172.21.15.67:0/2824072908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 msgr2=0x7f0b3806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 --2- 172.21.15.67:0/2824072908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009e50 tx=0x7f0b2802bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 -- 172.21.15.67:0/2824072908 shutdown_connections 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 --2- 172.21.15.67:0/2824072908 >> v2:172.21.15.187:3300/0 conn(0x7f0b3815fad0 0x7f0b380ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 --2- 172.21.15.67:0/2824072908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b380afbf0 0x7f0b3806e980 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 -- 172.21.15.67:0/2824072908 >> 172.21.15.67:0/2824072908 conn(0x7f0b3806bef0 msgr2=0x7f0b380b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 -- 172.21.15.67:0/2824072908 shutdown_connections 2024-04-17T07:29:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.929+0000 7f0b37200640 1 -- 172.21.15.67:0/2824072908 wait complete. 2024-04-17T07:29:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.931+0000 7f0b37200640 1 Processor -- start 2024-04-17T07:29:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.931+0000 7f0b37200640 1 -- start start 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b3806eec0 0x7f0b38168750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 0x7f0b380b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b380affd0 con 0x7f0b380afbf0 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38160110 con 0x7f0b3815fad0 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b37200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b380b0150 con 0x7f0b3806eec0 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b2d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b2d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49004/0 (socket says 172.21.15.67:49004) 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.932+0000 7f0b2d400640 1 -- 172.21.15.67:0/611578581 learned_addr learned my addr 172.21.15.67:0/611578581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b35e00640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b3806eec0 0x7f0b38168750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b36800640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 0x7f0b380b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 -- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b3806eec0 msgr2=0x7f0b38168750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b3806eec0 0x7f0b38168750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 -- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 msgr2=0x7f0b380b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 0x7f0b380b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 -- 172.21.15.67:0/611578581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b381472b0 con 0x7f0b380afbf0 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b36800640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 0x7f0b380b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:33.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2d400640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c400 tx=0x7f0b2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2ea00640 1 -- 172.21.15.67:0/611578581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200052d0 con 0x7f0b380afbf0 2024-04-17T07:29:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b38168d30 con 0x7f0b380afbf0 2024-04-17T07:29:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2ea00640 1 -- 172.21.15.67:0/611578581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b20005490 con 0x7f0b380afbf0 2024-04-17T07:29:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.933+0000 7f0b2ea00640 1 -- 172.21.15.67:0/611578581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20007260 con 0x7f0b380afbf0 2024-04-17T07:29:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.934+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b381692f0 con 0x7f0b380afbf0 2024-04-17T07:29:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.936+0000 7f0b2ea00640 1 -- 172.21.15.67:0/611578581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f0b20007400 con 0x7f0b380afbf0 2024-04-17T07:29:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.937+0000 7f0b2ea00640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 0x7f0b1406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.937+0000 7f0b2ea00640 1 -- 172.21.15.67:0/611578581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f0b2008b390 con 0x7f0b380afbf0 2024-04-17T07:29:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.938+0000 7f0b3f9e2740 1 Processor -- start 2024-04-17T07:29:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.938+0000 7f0b3f9e2740 1 -- start start 2024-04-17T07:29:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.938+0000 7f0b35e00640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 0x7f0b1406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.938+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fc8d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 0x560c7fcb7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560c7fbf3980 con 0x560c7fcd5620 2024-04-17T07:29:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560c7fb9a220 con 0x560c7fc8b6a0 2024-04-17T07:29:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560c7fbef940 con 0x560c7fc87b40 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49018/0 (socket says 172.21.15.67:49018) 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 -- 172.21.15.67:0/2408175057 learned_addr learned my addr 172.21.15.67:0/2408175057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b2d400640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fc8d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b36800640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 0x560c7fcb7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 -- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 msgr2=0x560c7fcb7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 0x560c7fcb7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 -- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 msgr2=0x560c7fc8d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fc8d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b35e00640 1 -- 172.21.15.67:0/2408175057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c7f9062d0 con 0x560c7fcd5620 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.939+0000 7f0b36800640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 0x560c7fcb7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.940+0000 7f0b35e00640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 0x7f0b1406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0b28035000 tx=0x7f0b2802e000 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.940+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0b28036040 tx=0x7f0b28040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813ab6ea6694edbd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.940+0000 7f0b26a00640 1 -- 172.21.15.67:0/2408175057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28051070 con 0x560c7fcd5620 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.940+0000 7f0b26a00640 1 -- 172.21.15.67:0/2408175057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2802ce50 con 0x560c7fcd5620 2024-04-17T07:29:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.940+0000 7f0b26a00640 1 -- 172.21.15.67:0/2408175057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28040d40 con 0x560c7fcd5620 2024-04-17T07:29:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2408175057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 msgr2=0x560c7fc8cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0b28036040 tx=0x7f0b28040000 comp rx=0 tx=0).stop 2024-04-17T07:29:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2408175057 shutdown_connections 2024-04-17T07:29:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc87b40 0x560c7fcb7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fc8d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2408175057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fc8cc50 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2408175057 >> 172.21.15.67:0/2408175057 conn(0x560c7fc89db0 msgr2=0x560c7fcd2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2408175057 shutdown_connections 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.941+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2408175057 wait complete. 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.942+0000 7f0b3f9e2740 1 -- start start 2024-04-17T07:29:33.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.942+0000 7f0b3f9e2740 1 Processor -- start 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcd9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 0x560c7fc9f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560c7fc8bbb0 con 0x560c7fcd5620 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560c7fbad590 con 0x560c7fc8b6a0 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560c7fc8be90 con 0x560c7fcd7580 2024-04-17T07:29:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b3f9e2740 10 client.0 _ll_register_callbacks cb 0x560c7f905ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b2d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b2d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49028/0 (socket says 172.21.15.67:49028) 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b2d400640 1 -- 172.21.15.67:0/623788783 learned_addr learned my addr 172.21.15.67:0/623788783 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.943+0000 7f0b36800640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 0x560c7fc9f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 -- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 msgr2=0x560c7fc9f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 0x560c7fc9f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 -- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 msgr2=0x560c7fcd9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b35e00640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcd9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcd9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 -- 172.21.15.67:0/623788783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c7f9d06b0 con 0x560c7fcd5620 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b36800640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 0x560c7fc9f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b2d400640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0b20001eb0 tx=0x7f0b20058e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d526ee228935b4c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b1b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:29:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20061a70 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f0b20055500 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2802ce50 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0b20076050 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b3f9e2740 10 client.6576 fetch_fsmap learned FSMap version 35 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.944+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x560c7fcd8d10 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2802ce50 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 3495+0+0 (secure 0 0 0) 0x7f0b200711b0 con 0x560c7fcd5620 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 10 client.6576 fetch_fsmap finished waiting for FSMap version 35 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 10 client.6576 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 20 client.6576 populate_metadata read hostname 'smithi067' 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fce3c70 0x560c7fcb3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 4 client.6576 mds_command: new command op to 6444 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.945+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 --> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c7fc013a0 con 0x560c7fce3c70 2024-04-17T07:29:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.946+0000 7f0b35e00640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fce3c70 0x560c7fcb3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.947+0000 7f0b35e00640 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fce3c70 0x560c7fcb3370 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33189af65c69a677 server_cookie=2cceaa8835019492 in_seq=0 out_seq=0 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.947+0000 7f0b1b400640 10 client.6576 ms_handle_connect on v2:172.21.15.187:6834/2437593714 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.948+0000 7f0b1b400640 1 -- 172.21.15.67:0/623788783 <== mds.0 v2:172.21.15.187:6834/2437593714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c7fc013a0 con 0x560c7fce3c70 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.948+0000 7f0b1b400640 10 client.6576 handle_command_reply: tid=0 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.948+0000 7f0b3f9e2740 1 client.6576 shutdown 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 msgr2=0x560c7fcb47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0b20001eb0 tx=0x7f0b20058e90 comp rx=0 tx=0).stop 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 shutdown_connections 2024-04-17T07:29:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fce3c70 0x560c7fcb3370 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd7580 0x560c7fc9f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd5620 0x560c7fcb47c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/623788783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcd9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b1b400640 0 client.6576 ms_handle_reset on v2:172.21.15.187:6834/2437593714 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 >> 172.21.15.67:0/623788783 conn(0x560c7fc87b40 msgr2=0x560c7fc8d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 shutdown_connections 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/623788783 wait complete. 2024-04-17T07:29:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.949+0000 7f0b3f9e2740 20 client.6576 trim_cache size 0 max 16384 2024-04-17T07:29:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.969+0000 7f0b3f9e2740 1 Processor -- start 2024-04-17T07:29:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.969+0000 7f0b3f9e2740 1 -- start start 2024-04-17T07:29:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc8b6a0 0x560c7fc8cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fcd5620 0x560c7fcd2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38168340 con 0x560c7fcd7580 2024-04-17T07:29:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381676a0 con 0x560c7fcd5620 2024-04-17T07:29:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.970+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b38167cf0 con 0x560c7fc8b6a0 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49036/0 (socket says 172.21.15.67:49036) 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 -- 172.21.15.67:0/1775390742 learned_addr learned my addr 172.21.15.67:0/1775390742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b35e00640 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc8b6a0 0x560c7fc8cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b2d400640 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fcd5620 0x560c7fcd2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 -- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc8b6a0 msgr2=0x560c7fc8cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc8b6a0 0x560c7fc8cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 -- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fcd5620 msgr2=0x560c7fcd2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fcd5620 0x560c7fcd2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.971+0000 7f0b36800640 1 -- 172.21.15.67:0/1775390742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c7fc89a90 con 0x560c7fcd7580 2024-04-17T07:29:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.972+0000 7f0b36800640 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0b30003770 tx=0x7f0b30009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3dcae91cb4f454 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.972+0000 7f0b26000640 1 -- 172.21.15.67:0/1775390742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30019020 con 0x560c7fcd7580 2024-04-17T07:29:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.972+0000 7f0b26000640 1 -- 172.21.15.67:0/1775390742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b300097e0 con 0x560c7fcd7580 2024-04-17T07:29:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.972+0000 7f0b26000640 1 -- 172.21.15.67:0/1775390742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30009bd0 con 0x560c7fcd7580 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/1775390742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 msgr2=0x560c7fcd97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0b30003770 tx=0x7f0b30009190 comp rx=0 tx=0).stop 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/1775390742 shutdown_connections 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcd97c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fcd5620 0x560c7fcd2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/1775390742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fc8b6a0 0x560c7fc8cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/1775390742 >> 172.21.15.67:0/1775390742 conn(0x560c7fc87b40 msgr2=0x560c7fc83230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/1775390742 shutdown_connections 2024-04-17T07:29:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.973+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/1775390742 wait complete. 2024-04-17T07:29:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.974+0000 7f0b3f9e2740 1 -- start start 2024-04-17T07:29:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.974+0000 7f0b3f9e2740 1 Processor -- start 2024-04-17T07:29:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcaea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd5620 0x560c7fcaef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560c7fce3af0 con 0x560c7fcd7580 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560c7fcb4d60 con 0x560c7fc8b6a0 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560c7fce4140 con 0x560c7fcd5620 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b3f9e2740 10 client.0 _ll_register_callbacks cb 0x7f0b20007770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b36800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b36800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49042/0 (socket says 172.21.15.67:49042) 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b36800640 1 -- 172.21.15.67:0/2260257900 learned_addr learned my addr 172.21.15.67:0/2260257900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.975+0000 7f0b36800640 1 -- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd5620 msgr2=0x560c7fcaef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b36800640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd5620 0x560c7fcaef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b36800640 1 -- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 msgr2=0x560c7fcaea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcaea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b36800640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcaea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b36800640 1 -- 172.21.15.67:0/2260257900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c7fcb0310 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b36800640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0b30008010 tx=0x7f0b30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44d7cfb255d4747 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b1b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3002e060 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f0b30004320 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b300097e0 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.976+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0b30034050 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 10 client.6582 fetch_fsmap learned FSMap version 35 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x560c7fcb45d0 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b300097e0 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 3495+0+0 (secure 0 0 0) 0x7f0b30037020 con 0x560c7fcd7580 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 10 client.6582 fetch_fsmap finished waiting for FSMap version 35 2024-04-17T07:29:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 10 client.6582 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T07:29:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 20 client.6582 populate_metadata read hostname 'smithi067' 2024-04-17T07:29:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fd4ddf0 0x560c7fd4e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 4 client.6582 mds_command: new command op to 6444 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:29:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.977+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 --> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] -- command(tid 0: {"prefix": "status"}) -- 0x560c7fcb5550 con 0x560c7fd4ddf0 2024-04-17T07:29:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.978+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fd4ddf0 0x560c7fd4e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b35e00640 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fd4ddf0 0x560c7fd4e1d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde092e8ce1c0c4 server_cookie=bf8ce5fbf86d4084 in_seq=0 out_seq=0 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b1b400640 10 client.6582 ms_handle_connect on v2:172.21.15.187:6834/2437593714 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b1b400640 1 -- 172.21.15.67:0/2260257900 <== mds.0 v2:172.21.15.187:6834/2437593714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x560c7fcb5550 con 0x560c7fd4ddf0 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b1b400640 10 client.6582 handle_command_reply: tid=0 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b3f9e2740 1 client.6582 shutdown 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 msgr2=0x560c7fcc82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.979+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0b30008010 tx=0x7f0b30004730 comp rx=0 tx=0).stop 2024-04-17T07:29:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 shutdown_connections 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:6834/2437593714,v1:172.21.15.187:6835/2437593714] conn(0x560c7fd4ddf0 0x560c7fd4e1d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560c7fcd7580 0x560c7fcc82c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560c7fcd5620 0x560c7fcaef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 --2- 172.21.15.67:0/2260257900 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560c7fc8b6a0 0x560c7fcaea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b1b400640 0 client.6582 ms_handle_reset on v2:172.21.15.187:6834/2437593714 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 >> 172.21.15.67:0/2260257900 conn(0x560c7fc87b40 msgr2=0x560c7fcd2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 shutdown_connections 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 1 -- 172.21.15.67:0/2260257900 wait complete. 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b3f9e2740 20 client.6582 trim_cache size 0 max 16384 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:29:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6444, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.894322517, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 35, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 84, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 84, 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 12.260800997 2024-04-17T07:29:33.982 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 msgr2=0x7f0b1406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 0x7f0b1406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0b28035000 tx=0x7f0b2802e000 comp rx=0 tx=0).stop 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 msgr2=0x7f0b3814fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.980+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c400 tx=0x7f0b2000c8c0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 shutdown_connections 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f0b1406ca10 0x7f0b1406eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b3815fad0 0x7f0b380b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380afbf0 0x7f0b3814fb30 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 --2- 172.21.15.67:0/611578581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b3806eec0 0x7f0b38168750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 >> 172.21.15.67:0/611578581 conn(0x7f0b3806bef0 msgr2=0x7f0b38153950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 shutdown_connections 2024-04-17T07:29:33.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:33.981+0000 7f0b37200640 1 -- 172.21.15.67:0/611578581 wait complete. 2024-04-17T07:29:33.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:29:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.219+0000 7f78b6200640 1 Processor -- start 2024-04-17T07:29:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.220+0000 7f78b6200640 1 -- start start 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.221+0000 7f78b6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f78b0169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78b6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b00d2d30 con 0x7f78b00b6d00 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78b6200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b00d3010 con 0x7f78b015ec60 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78b6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f78b00d49c0 con 0x7f78b00cfae0 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78afe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78b00b6d00 0x7f78b00cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42152/0 (socket says 172.21.15.67:42152) 2024-04-17T07:29:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78afe00640 1 -- 172.21.15.67:0/2651882251 learned_addr learned my addr 172.21.15.67:0/2651882251 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78b5800640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f78b0169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.222+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720901745 0 0) 0x7f78b00d2d30 con 0x7f78b00b6d00 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7890003520 con 0x7f78b00b6d00 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3991876392 0 0) 0x7f7890003520 con 0x7f78b00b6d00 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 msgr2=0x7f78b0169810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f78b0169810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 >> v1:172.21.15.187:6789/0 conn(0x7f78b015ec60 legacy=0x7f78b00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b00d32f0 con 0x7f78b00b6d00 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f78980019f0 con 0x7f78b00b6d00 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 >> v1:172.21.15.67:6789/0 conn(0x7f78b00b6d00 legacy=0x7f78b00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7890003a50 0x7f7890003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f7890004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78b00d49c0 con 0x7f7890004d40 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f789000bb80 con 0x7f7890003a50 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f789000bd80 con 0x7f78b00cfae0 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.223+0000 7f78b5800640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78afe00640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f7890004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78b5800640 1 -- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 msgr2=0x7f7890004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78b5800640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00cfae0 0x7f7890004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78b5800640 1 -- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7890003a50 msgr2=0x7f7890003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78b5800640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7890003a50 0x7f7890003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.224+0000 7f78b5800640 1 -- 172.21.15.67:0/2651882251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7890003520 con 0x7f7890004d40 2024-04-17T07:29:34.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.225+0000 7f78b5800640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78900051c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009ac0 tx=0x7f7890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecde82ead553de09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:34.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.225+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8039070 con 0x7f7890004d40 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.225+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a802c6b0 con 0x7f7890004d40 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.225+0000 7f78af400640 1 -- 172.21.15.67:0/2651882251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8009690 con 0x7f7890004d40 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 -- 172.21.15.67:0/2651882251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 msgr2=0x7f78900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 --2- 172.21.15.67:0/2651882251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78900051c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009ac0 tx=0x7f7890005830 comp rx=0 tx=0).stop 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b5800640 1 -- 172.21.15.67:0/2651882251 reap_dead start 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 -- 172.21.15.67:0/2651882251 shutdown_connections 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 -- 172.21.15.67:0/2651882251 >> 172.21.15.67:0/2651882251 conn(0x7f78b00d15f0 msgr2=0x7f78b00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 -- 172.21.15.67:0/2651882251 shutdown_connections 2024-04-17T07:29:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.226+0000 7f78b6200640 1 -- 172.21.15.67:0/2651882251 wait complete. 2024-04-17T07:29:34.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.228+0000 7f78b6200640 1 Processor -- start 2024-04-17T07:29:34.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.228+0000 7f78b6200640 1 -- start start 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00abf40 0x7f78b00adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b01f75c0 0x7f78b01f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78b00b8b80 con 0x7f7890004d40 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f78b00d3660 con 0x7f78b01f75c0 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f78b00d00c0 con 0x7f78b00abf40 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49058/0 (socket says 172.21.15.67:49058) 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.229+0000 7f78b4e00640 1 -- 172.21.15.67:0/1641651992 learned_addr learned my addr 172.21.15.67:0/1641651992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78afe00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00abf40 0x7f78b00adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 -- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00abf40 msgr2=0x7f78b00adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00abf40 0x7f78b00adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 -- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b01f75c0 msgr2=0x7f78b01f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b01f75c0 0x7f78b01f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 -- 172.21.15.67:0/1641651992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b00ac790 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78b4e00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c3c0 tx=0x7f78a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.230+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0017070 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.231+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a00052d0 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.231+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0003830 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.231+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b00aca70 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.231+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b00ad030 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.232+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b0068a10 con 0x7f7890004d40 2024-04-17T07:29:34.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.233+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f78a001a070 con 0x7f7890004d40 2024-04-17T07:29:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.234+0000 7f78acc00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 0x7f788c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.234+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f78a008b090 con 0x7f7890004d40 2024-04-17T07:29:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.235+0000 7f78afe00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 0x7f788c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:34.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.236+0000 7f78afe00640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 0x7f788c06ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7898001f70 tx=0x7f78980012f0 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:34.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.239+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f78a0059320 con 0x7f7890004d40 2024-04-17T07:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.486+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f78b00ba350 con 0x7f7890004d40 2024-04-17T07:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.487+0000 7f78acc00640 1 -- 172.21.15.67:0/1641651992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f78a0058c20 con 0x7f7890004d40 2024-04-17T07:29:34.486 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:29:34.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 msgr2=0x7f788c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 0x7f788c06ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7898001f70 tx=0x7f78980012f0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 msgr2=0x7f78b00ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f78a000c3c0 tx=0x7f78a000c880 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 shutdown_connections 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f788c06ca40 0x7f788c06ef00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f78b01f75c0 0x7f78b01f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f78b00abf40 0x7f78b00adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 --2- 172.21.15.67:0/1641651992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7890004d40 0x7f78b00ce570 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.489+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 >> 172.21.15.67:0/1641651992 conn(0x7f78b00d15f0 msgr2=0x7f78b015e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.490+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 shutdown_connections 2024-04-17T07:29:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34.490+0000 7f78b6200640 1 -- 172.21.15.67:0/1641651992 wait complete. 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:29:34.508 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:29:34.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:34.508 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:29:34.560 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:29:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:29:34.560 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:29:34.561 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41670sec preferred_lft 41670sec 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:29:34.562 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:29:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:29:34.563 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:29:34.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:34.564 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:29:34.657 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:29:34.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:34.662 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:29:34.717 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:29:34.718 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:29:34.718 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:29:34.718 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:29:34.718 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:29:34.719 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:29:34.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:29:34.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:29:34.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:34.979 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:29:35.029 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:29:35.029 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:29:35.030 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:29:35.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:35.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:29:35.112 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:29:35.112 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:35.161 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:29:35.161 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:29:35.161 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:35.210 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:29:35.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:35.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:29:35.351 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:29:35.351 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:29:35.388 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:29:35.388 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:29:35.388 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:29:35.388 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:29:35.388 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:29:35.392 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:29:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:29:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:29:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:29:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:29:35.399 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:29:35.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:35.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:35.500 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:29:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:29:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:29:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:client6591 2024-04-17T07:29:35.643 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:35.643 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:35.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:29:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.922+0000 7f2066000640 1 Processor -- start 2024-04-17T07:29:35.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.923+0000 7f2066000640 1 -- start start 2024-04-17T07:29:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.924+0000 7f2066000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f206015ee20 0x7f20600d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.924+0000 7f2066000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20600d4140 con 0x7f20600d18e0 2024-04-17T07:29:35.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.924+0000 7f2066000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20600d3b80 con 0x7f20600d3520 2024-04-17T07:29:35.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.924+0000 7f2066000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20600b83b0 con 0x7f206015ee20 2024-04-17T07:29:35.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20600d18e0 0x7f20600d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42166/0 (socket says 172.21.15.67:42166) 2024-04-17T07:29:35.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205fe00640 1 -- 172.21.15.67:0/1175868918 learned_addr learned my addr 172.21.15.67:0/1175868918 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133047406 0 0) 0x7f20600d4140 con 0x7f20600d18e0 2024-04-17T07:29:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2044003520 con 0x7f20600d18e0 2024-04-17T07:29:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f2064c00640 1 --2- 172.21.15.67:0/1175868918 >> v2:172.21.15.187:3300/0 conn(0x7f206015ee20 0x7f20600d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3026997942 0 0) 0x7f2044003520 con 0x7f20600d18e0 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 >> v2:172.21.15.187:3300/0 conn(0x7f206015ee20 msgr2=0x7f20600d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.925+0000 7f205f400640 1 --2- 172.21.15.67:0/1175868918 >> v2:172.21.15.187:3300/0 conn(0x7f206015ee20 0x7f20600d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 >> v1:172.21.15.187:6789/0 conn(0x7f20600d3520 legacy=0x7f20600b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20600d3e60 con 0x7f20600d18e0 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f20480019f0 con 0x7f20600d18e0 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 >> v1:172.21.15.67:6789/0 conn(0x7f20600d18e0 legacy=0x7f20600d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044003a20 0x7f2044003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2044004800 0x7f2044004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f204400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20600b83b0 con 0x7f2044005210 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f204400c160 con 0x7f2044003a20 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f204400c360 con 0x7f2044004800 2024-04-17T07:29:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2065600640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f204400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2064c00640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044003a20 0x7f2044003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f205fe00640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2044004800 0x7f2044004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2065600640 1 -- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2044004800 msgr2=0x7f2044004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2065600640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2044004800 0x7f2044004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2065600640 1 -- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044003a20 msgr2=0x7f2044003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.926+0000 7f2065600640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2044003a20 0x7f2044003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.927+0000 7f2065600640 1 -- 172.21.15.67:0/1175868918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2044003520 con 0x7f2044005210 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.927+0000 7f2065600640 1 -- 172.21.15.67:0/1175868918 reap_dead start 2024-04-17T07:29:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f2065600640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f204400b9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2058003db0 tx=0x7f2044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1e1d17f5980d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058003980 con 0x7f2044005210 2024-04-17T07:29:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2058007240 con 0x7f2044005210 2024-04-17T07:29:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f205f400640 1 -- 172.21.15.67:0/1175868918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058007630 con 0x7f2044005210 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f2066000640 1 -- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 msgr2=0x7f204400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.928+0000 7f2066000640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f204400b9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f2058003db0 tx=0x7f2044005830 comp rx=0 tx=0).stop 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.929+0000 7f2066000640 1 -- 172.21.15.67:0/1175868918 shutdown_connections 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.929+0000 7f2066000640 1 --2- 172.21.15.67:0/1175868918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f204400b9c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.929+0000 7f2066000640 1 -- 172.21.15.67:0/1175868918 >> 172.21.15.67:0/1175868918 conn(0x7f206006d3c0 msgr2=0x7f20600b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.929+0000 7f2066000640 1 -- 172.21.15.67:0/1175868918 shutdown_connections 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.929+0000 7f2066000640 1 -- 172.21.15.67:0/1175868918 wait complete. 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.930+0000 7f2066000640 1 Processor -- start 2024-04-17T07:29:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.931+0000 7f2066000640 1 -- start start 2024-04-17T07:29:35.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.931+0000 7f2066000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.931+0000 7f2066000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 0x7f20601f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2066000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 0x7f20601e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2066000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f206015f200 con 0x7f2044005210 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2066000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20600d1d20 con 0x7f20601e7dd0 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2066000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20600d13a0 con 0x7f20600cc470 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49074/0 (socket says 172.21.15.67:49074) 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 -- 172.21.15.67:0/2289304759 learned_addr learned my addr 172.21.15.67:0/2289304759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f205fe00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 0x7f20601f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 -- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 msgr2=0x7f20601f7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2065600640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 0x7f20601e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 0x7f20601f7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 -- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 msgr2=0x7f20601e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:35.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 0x7f20601e8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2064c00640 1 -- 172.21.15.67:0/2289304759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20600ccc50 con 0x7f2044005210 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f205fe00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 0x7f20601f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.932+0000 7f2065600640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 0x7f20601e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f2064c00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f205000f5b0 tx=0x7f205000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2050013810 con 0x7f2044005210 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f205001a070 con 0x7f2044005210 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2050013bb0 con 0x7f2044005210 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20600ccf30 con 0x7f2044005210 2024-04-17T07:29:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.933+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20601e78b0 con 0x7f2044005210 2024-04-17T07:29:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.934+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2060068a10 con 0x7f2044005210 2024-04-17T07:29:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.936+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f2050012070 con 0x7f2044005210 2024-04-17T07:29:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.937+0000 7f205cc00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 0x7f203c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.937+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f205008bb40 con 0x7f2044005210 2024-04-17T07:29:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.938+0000 7f205fe00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 0x7f203c06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:35.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.939+0000 7f205fe00640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 0x7f203c06ef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2048001b60 tx=0x7f2048009040 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:35.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:35.942+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f205001c310 con 0x7f2044005210 2024-04-17T07:29:36.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.174+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 --> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f20600cd450 con 0x7f203c06ca90 2024-04-17T07:29:36.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.179+0000 7f205cc00640 1 -- 172.21.15.67:0/2289304759 <== mgr.5488 v2:172.21.15.187:6832/1436704971 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2048002050 con 0x7f203c06ca90 2024-04-17T07:29:36.179 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:29:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:29:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:29:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:29:36.180 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:29:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 msgr2=0x7f203c06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 0x7f203c06ef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2048001b60 tx=0x7f2048009040 comp rx=0 tx=0).stop 2024-04-17T07:29:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 msgr2=0x7f20600cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f205000f5b0 tx=0x7f205000fa70 comp rx=0 tx=0).stop 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 shutdown_connections 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f203c06ca90 0x7f203c06ef50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20601e7dd0 0x7f20601e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cc470 0x7f20601f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 --2- 172.21.15.67:0/2289304759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2044005210 0x7f20600cb020 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 >> 172.21.15.67:0/2289304759 conn(0x7f206006d3c0 msgr2=0x7f2060154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 shutdown_connections 2024-04-17T07:29:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.182+0000 7f2066000640 1 -- 172.21.15.67:0/2289304759 wait complete. 2024-04-17T07:29:36.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:29:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.420+0000 7f9ee6000640 1 Processor -- start 2024-04-17T07:29:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.421+0000 7f9ee6000640 1 -- start start 2024-04-17T07:29:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.422+0000 7f9ee6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.422+0000 7f9ee6000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9ee00d3210 0x7f9ee00cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.422+0000 7f9ee6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee00d00b0 con 0x7f9ee00cf8e0 2024-04-17T07:29:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.422+0000 7f9ee6000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee00d3670 con 0x7f9ee00d3210 2024-04-17T07:29:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.422+0000 7f9ee6000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee00d0c30 con 0x7f9ee015ecb0 2024-04-17T07:29:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9ee5600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9ee00cf8e0 0x7f9ee00d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42180/0 (socket says 172.21.15.67:42180) 2024-04-17T07:29:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9ee5600640 1 -- 172.21.15.67:0/1571639977 learned_addr learned my addr 172.21.15.67:0/1571639977 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9ee4c00640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9edfe00640 1 --2- 172.21.15.67:0/1571639977 >> v2:172.21.15.187:3300/0 conn(0x7f9ee00d3210 0x7f9ee00cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436985099 0 0) 0x7f9ee00d00b0 con 0x7f9ee00cf8e0 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.423+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ec8003520 con 0x7f9ee00cf8e0 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.424+0000 7f9edfe00640 1 -- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 msgr2=0x7f9ee00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.424+0000 7f9edfe00640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee00d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.424+0000 7f9edfe00640 1 -- 172.21.15.67:0/1571639977 >> v1:172.21.15.67:6789/0 conn(0x7f9ee00cf8e0 legacy=0x7f9ee00d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.424+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2728670458 0 0) 0x7f9ec8003520 con 0x7f9ee00cf8e0 2024-04-17T07:29:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.424+0000 7f9edfe00640 1 -- 172.21.15.67:0/1571639977 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee00d0390 con 0x7f9ee00d3210 2024-04-17T07:29:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.425+0000 7f9edfe00640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00d3210 0x7f9ee00cf3a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00097a0 tx=0x7f9ed002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=469a0200378be304 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0037070 con 0x7f9ee00d3210 2024-04-17T07:29:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed0030510 con 0x7f9ee00d3210 2024-04-17T07:29:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9edf400640 1 -- 172.21.15.67:0/1571639977 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8001a10 con 0x7f9ee00d3210 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9ee6000640 1 -- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00d3210 msgr2=0x7f9ee00cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9ee6000640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00d3210 0x7f9ee00cf3a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00097a0 tx=0x7f9ed002b710 comp rx=0 tx=0).stop 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.426+0000 7f9ee6000640 1 -- 172.21.15.67:0/1571639977 shutdown_connections 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.427+0000 7f9ee6000640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00d3210 0x7f9ee00cf3a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.427+0000 7f9ee6000640 1 --2- 172.21.15.67:0/1571639977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee00d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.427+0000 7f9ee6000640 1 -- 172.21.15.67:0/1571639977 >> 172.21.15.67:0/1571639977 conn(0x7f9ee008bff0 msgr2=0x7f9ee00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.427+0000 7f9ee6000640 1 -- 172.21.15.67:0/1571639977 shutdown_connections 2024-04-17T07:29:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.427+0000 7f9ee6000640 1 -- 172.21.15.67:0/1571639977 wait complete. 2024-04-17T07:29:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.429+0000 7f9ee6000640 1 Processor -- start 2024-04-17T07:29:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.429+0000 7f9ee6000640 1 -- start start 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.430+0000 7f9ee6000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00cf8e0 0x7f9ee014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.430+0000 7f9ee6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.430+0000 7f9ee6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee014d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9ee6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee00cfd50 con 0x7f9ee00d3210 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9ee6000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee00d0950 con 0x7f9ee00cf8e0 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9ee6000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee015f090 con 0x7f9ee015ecb0 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49078/0 (socket says 172.21.15.67:49078) 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 -- 172.21.15.67:0/3472921509 learned_addr learned my addr 172.21.15.67:0/3472921509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9ee4c00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00cf8e0 0x7f9ee014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9ee5600640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee014d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 -- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 msgr2=0x7f9ee014d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee014d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 -- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00cf8e0 msgr2=0x7f9ee014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00cf8e0 0x7f9ee014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.431+0000 7f9edfe00640 1 -- 172.21.15.67:0/3472921509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee014e1a0 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9edfe00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9ed002b750 tx=0x7f9ed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0041020 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed003c870 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0037040 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee014e480 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.432+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee014ea40 con 0x7f9ee00d3210 2024-04-17T07:29:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.434+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee0068a10 con 0x7f9ee00d3210 2024-04-17T07:29:36.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.435+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 30) ==== 91142+0+0 (secure 0 0 0) 0x7f9ed0044020 con 0x7f9ee00d3210 2024-04-17T07:29:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.436+0000 7f9edcc00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 0x7f9ec406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.436+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f9ed00ad7b0 con 0x7f9ee00d3210 2024-04-17T07:29:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.437+0000 7f9ee4c00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 0x7f9ec406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.438+0000 7f9ee4c00640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 0x7f9ec406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009aa0 tx=0x7f9ed4009190 comp rx=0 tx=0).ready entity=mgr.5488 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.443+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9ed004b030 con 0x7f9ee00d3210 2024-04-17T07:29:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:36.708+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f9ee00b5a70 con 0x7f9ee00d3210 2024-04-17T07:29:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.186+0000 7f9edcc00640 1 -- 172.21.15.67:0/3472921509 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) ==== 92+0+0 (secure 0 0 0) 0x7f9ed007b9c0 con 0x7f9ee00d3210 2024-04-17T07:29:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.190+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 msgr2=0x7f9ec406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.190+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 0x7f9ec406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009aa0 tx=0x7f9ed4009190 comp rx=0 tx=0).stop 2024-04-17T07:29:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.190+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 msgr2=0x7f9ee00b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.190+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9ed002b750 tx=0x7f9ed00046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 shutdown_connections 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f9ec406ca10 0x7f9ec406eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ee015ecb0 0x7f9ee014d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ee00d3210 0x7f9ee00b4ca0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 --2- 172.21.15.67:0/3472921509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ee00cf8e0 0x7f9ee014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.191+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 >> 172.21.15.67:0/3472921509 conn(0x7f9ee008bff0 msgr2=0x7f9ee00d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:37.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.192+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 shutdown_connections 2024-04-17T07:29:37.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.193+0000 7f9ee6000640 1 -- 172.21.15.67:0/3472921509 wait complete. 2024-04-17T07:29:37.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:29:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.434+0000 7fcd1da00640 1 Processor -- start 2024-04-17T07:29:37.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.434+0000 7fcd1da00640 1 -- start start 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.435+0000 7fcd1da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fcd180d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd1da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd180d51d0 con 0x7fcd180b3660 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd1da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd180d54b0 con 0x7fcd1815ec40 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd1da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1815f3a0 con 0x7fcd180d33f0 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd1d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd180b3660 0x7fcd180cf280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42204/0 (socket says 172.21.15.67:42204) 2024-04-17T07:29:37.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd1d000640 1 -- 172.21.15.67:0/4132315813 learned_addr learned my addr 172.21.15.67:0/4132315813 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd0fe00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fcd180d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480055805 0 0) 0x7fcd180d51d0 con 0x7fcd180b3660 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.436+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccfc003520 con 0x7fcd180b3660 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 101316152 0 0) 0x7fccfc003520 con 0x7fcd180b3660 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 msgr2=0x7fcd180d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fcd180d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 >> v1:172.21.15.187:6789/0 conn(0x7fcd1815ec40 legacy=0x7fcd180d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1815f680 con 0x7fcd180b3660 2024-04-17T07:29:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcd100019f0 con 0x7fcd180b3660 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 >> v1:172.21.15.67:6789/0 conn(0x7fcd180b3660 legacy=0x7fcd180cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fccfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fccfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccfc004d40 0x7fccfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1815f3a0 con 0x7fccfc0039e0 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc00baa0 con 0x7fccfc004d40 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc00bca0 con 0x7fcd180d33f0 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.437+0000 7fcd17e00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fccfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd17e00640 1 -- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 msgr2=0x7fccfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd0fe00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fccfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd1d000640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccfc004d40 0x7fccfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd17e00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd180d33f0 0x7fccfc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd17e00640 1 -- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccfc004d40 msgr2=0x7fccfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd17e00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccfc004d40 0x7fccfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd17e00640 1 -- 172.21.15.67:0/4132315813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc003520 con 0x7fccfc0039e0 2024-04-17T07:29:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.438+0000 7fcd1d000640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccfc004d40 0x7fccfc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.439+0000 7fcd17e00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fccfc003e00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001720 tx=0x7fccfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2512571b19e2b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.439+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08001370 con 0x7fccfc0039e0 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.439+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd080052d0 con 0x7fccfc0039e0 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.439+0000 7fcd17400640 1 -- 172.21.15.67:0/4132315813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08003830 con 0x7fccfc0039e0 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 -- 172.21.15.67:0/4132315813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 msgr2=0x7fccfc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 --2- 172.21.15.67:0/4132315813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fccfc003e00 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001720 tx=0x7fccfc005830 comp rx=0 tx=0).stop 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1d000640 1 -- 172.21.15.67:0/4132315813 reap_dead start 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 -- 172.21.15.67:0/4132315813 shutdown_connections 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 -- 172.21.15.67:0/4132315813 >> 172.21.15.67:0/4132315813 conn(0x7fcd1808bff0 msgr2=0x7fcd180d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 -- 172.21.15.67:0/4132315813 shutdown_connections 2024-04-17T07:29:37.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.440+0000 7fcd1da00640 1 -- 172.21.15.67:0/4132315813 wait complete. 2024-04-17T07:29:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.443+0000 7fcd1da00640 1 Processor -- start 2024-04-17T07:29:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.443+0000 7fcd1da00640 1 -- start start 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd18152ce0 0x7fcd180b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd18152730 0x7fcd180b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcd180d18a0 con 0x7fccfc0039e0 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcd180d1720 con 0x7fcd18152730 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd1da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcd180d1a20 con 0x7fcd18152ce0 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49116/0 (socket says 172.21.15.67:49116) 2024-04-17T07:29:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 -- 172.21.15.67:0/2204777890 learned_addr learned my addr 172.21.15.67:0/2204777890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd18152ce0 msgr2=0x7fcd180b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd18152ce0 0x7fcd180b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd18152730 msgr2=0x7fcd180b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd18152730 0x7fcd180b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.445+0000 7fcd17e00640 1 -- 172.21.15.67:0/2204777890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd180b8080 con 0x7fccfc0039e0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd17e00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001720 tx=0x7fcd08001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08042070 con 0x7fccfc0039e0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd08005480 con 0x7fccfc0039e0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd180b6250 con 0x7fccfc0039e0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08007a30 con 0x7fccfc0039e0 2024-04-17T07:29:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.446+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd180b67d0 con 0x7fccfc0039e0 2024-04-17T07:29:37.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.448+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd18068a10 con 0x7fccfc0039e0 2024-04-17T07:29:37.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.449+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 31) ==== 91159+0+0 (secure 0 0 0) 0x7fcd08045070 con 0x7fccfc0039e0 2024-04-17T07:29:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.450+0000 7fcd14c00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 0x7fccf406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.450+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fcd080ad990 con 0x7fccfc0039e0 2024-04-17T07:29:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.450+0000 7fcd0fe00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 msgr2=0x7fccf406ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.450+0000 7fcd0fe00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 0x7fccf406ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.454+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcd0807bc30 con 0x7fccfc0039e0 2024-04-17T07:29:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.651+0000 7fcd0fe00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 msgr2=0x7fccf406ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.651+0000 7fcd0fe00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 0x7fccf406ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:37.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.661+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fcd180d4e10 con 0x7fccfc0039e0 2024-04-17T07:29:37.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.667+0000 7fcd14c00640 1 -- 172.21.15.67:0/2204777890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) ==== 138+0+0 (secure 0 0 0) 0x7fcd0807b530 con 0x7fccfc0039e0 2024-04-17T07:29:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 msgr2=0x7fccf406ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 0x7fccf406ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 msgr2=0x7fcd180b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fcd08001720 tx=0x7fcd08001120 comp rx=0 tx=0).stop 2024-04-17T07:29:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 shutdown_connections 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7fccf406ca90 0x7fccf406ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd18152730 0x7fcd180b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd18152ce0 0x7fcd180b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 --2- 172.21.15.67:0/2204777890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccfc0039e0 0x7fcd180b5290 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 >> 172.21.15.67:0/2204777890 conn(0x7fcd1808bff0 msgr2=0x7fcd1806e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 shutdown_connections 2024-04-17T07:29:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.670+0000 7fcd1da00640 1 -- 172.21.15.67:0/2204777890 wait complete. 2024-04-17T07:29:37.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:29:37.738 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:37.739+0000 7f39b784c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:37.792 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:37.793+0000 7f39b784c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:37.795 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:37.795+0000 7f552e8da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:37.858 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:37.858+0000 7f552e8da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.914+0000 7f300ea00640 1 Processor -- start 2024-04-17T07:29:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.914+0000 7f300ea00640 1 -- start start 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30080d31b0 0x7f30080b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30080b36d0 0x7f300806ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300ea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30080d28a0 con 0x7f300815eb10 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300ea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f30080d51e0 con 0x7f30080b36d0 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f30080d54c0 con 0x7f30080d31b0 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30080b36d0 0x7f300806ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300d600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f300815eb10 0x7f30080d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42212/0 (socket says 172.21.15.67:42212) 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30080b36d0 0x7f300806ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49126/0 (socket says 172.21.15.67:49126) 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.915+0000 7f300d600640 1 -- 172.21.15.67:0/3024839921 learned_addr learned my addr 172.21.15.67:0/3024839921 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.916+0000 7f300cc00640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30080d31b0 0x7f30080b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.916+0000 7f300e000640 1 -- 172.21.15.67:0/3024839921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30080d31b0 msgr2=0x7f30080b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.916+0000 7f300e000640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30080d31b0 0x7f30080b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.916+0000 7f300e000640 1 -- 172.21.15.67:0/3024839921 >> v1:172.21.15.67:6789/0 conn(0x7f300815eb10 legacy=0x7f30080d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.916+0000 7f300e000640 1 -- 172.21.15.67:0/3024839921 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30080cff30 con 0x7f30080b36d0 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.917+0000 7f300e000640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f300806ca60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7aef5d3a08227f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.917+0000 7f3007e00640 1 -- 172.21.15.67:0/3024839921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000039070 con 0x7f30080b36d0 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.917+0000 7f3007e00640 1 -- 172.21.15.67:0/3024839921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f300002cc60 con 0x7f30080b36d0 2024-04-17T07:29:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.917+0000 7f3007e00640 1 -- 172.21.15.67:0/3024839921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000034660 con 0x7f30080b36d0 2024-04-17T07:29:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 -- 172.21.15.67:0/3024839921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 msgr2=0x7f300806ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f300806ca60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300002beb0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 -- 172.21.15.67:0/3024839921 shutdown_connections 2024-04-17T07:29:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f300806ca60 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 --2- 172.21.15.67:0/3024839921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30080d31b0 0x7f30080b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 -- 172.21.15.67:0/3024839921 >> 172.21.15.67:0/3024839921 conn(0x7f300808bff0 msgr2=0x7f30080d4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 -- 172.21.15.67:0/3024839921 shutdown_connections 2024-04-17T07:29:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.918+0000 7f300ea00640 1 -- 172.21.15.67:0/3024839921 wait complete. 2024-04-17T07:29:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.920+0000 7f300ea00640 1 Processor -- start 2024-04-17T07:29:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.920+0000 7f300ea00640 1 -- start start 2024-04-17T07:29:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080d31b0 0x7f30081f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f300815eb10 0x7f30081f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f300806cfa0 con 0x7f30080b36d0 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f300815eef0 con 0x7f30080d31b0 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f300806d120 con 0x7f300815eb10 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49130/0 (socket says 172.21.15.67:49130) 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.921+0000 7f300d600640 1 -- 172.21.15.67:0/3830993382 learned_addr learned my addr 172.21.15.67:0/3830993382 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300cc00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080d31b0 0x7f30081f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300e000640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f300815eb10 0x7f30081f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f300815eb10 msgr2=0x7f30081f8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f300815eb10 0x7f30081f8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080d31b0 msgr2=0x7f30081f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080d31b0 0x7f30081f7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 -- 172.21.15.67:0/3830993382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30080b9410 con 0x7f30080b36d0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f300d600640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c3f0 tx=0x7f2ffc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.922+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc017070 con 0x7f30080b36d0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.923+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc005ce0 con 0x7f30080b36d0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.923+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0072f0 con 0x7f30080b36d0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.923+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3008155150 con 0x7f30080b36d0 2024-04-17T07:29:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.923+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30080cbdc0 con 0x7f30080b36d0 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.924+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30080d2fd0 con 0x7f30080b36d0 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.925+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 31) ==== 91159+0+0 (secure 0 0 0) 0x7f2ffc01a070 con 0x7f30080b36d0 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.926+0000 7f3005600640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 0x7f2fe806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.926+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f2ffc08b020 con 0x7f30080b36d0 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.926+0000 7f300cc00640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 msgr2=0x7f2fe806ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:37.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.926+0000 7f300cc00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 0x7f2fe806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:37.930+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f30080d2fd0 con 0x7f30080b36d0 2024-04-17T07:29:38.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.127+0000 7f300cc00640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 msgr2=0x7f2fe806ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:38.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.127+0000 7f300cc00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 0x7f2fe806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.133+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f3008068a10 con 0x7f30080b36d0 2024-04-17T07:29:38.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.138+0000 7f3005600640 1 -- 172.21.15.67:0/3830993382 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) ==== 133+0+0 (secure 0 0 0) 0x7f3008155150 con 0x7f30080b36d0 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 msgr2=0x7f2fe806ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 0x7f2fe806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 msgr2=0x7f3008154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c3f0 tx=0x7f2ffc00c8b0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 shutdown_connections 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f2fe806cab0 0x7f2fe806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f300815eb10 0x7f30081f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080d31b0 0x7f30081f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 --2- 172.21.15.67:0/3830993382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30080b36d0 0x7f3008154c10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 >> 172.21.15.67:0/3830993382 conn(0x7f300808bff0 msgr2=0x7f30080c84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 shutdown_connections 2024-04-17T07:29:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.141+0000 7f300ea00640 1 -- 172.21.15.67:0/3830993382 wait complete. 2024-04-17T07:29:38.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:38.158 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T07:29:38.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-17T07:29:38.239 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.240+0000 7f39b784c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:38.292 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.293+0000 7f39b784c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:38.344 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.345+0000 7f39b784c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:38.347 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.348+0000 7f552e8da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:38.405 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.405+0000 7f552e8da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:38.405 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.406+0000 7f39b784c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.422+0000 7f0189e00640 1 Processor -- start 2024-04-17T07:29:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.422+0000 7f0189e00640 1 -- start start 2024-04-17T07:29:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.423+0000 7f0189e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01840d2ff0 0x7f01840b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.423+0000 7f0189e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01841622b0 0x7f01840cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.423+0000 7f0189e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01840b5110 con 0x7f01840b4c70 2024-04-17T07:29:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.423+0000 7f0189e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f01840b53f0 con 0x7f01841622b0 2024-04-17T07:29:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.423+0000 7f0189e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01840b56d0 con 0x7f01840d2ff0 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.424+0000 7f0189400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01841622b0 0x7f01840cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.424+0000 7f0189400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01841622b0 0x7f01840cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:60684/0 (socket says 172.21.15.67:60684) 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.424+0000 7f0189400640 1 -- 172.21.15.67:0/734240719 learned_addr learned my addr 172.21.15.67:0/734240719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.424+0000 7f0188a00640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01840d2ff0 0x7f01840b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.425+0000 7f0189400640 1 -- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01840d2ff0 msgr2=0x7f01840b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.425+0000 7f0189400640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01840d2ff0 0x7f01840b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.425+0000 7f0189400640 1 -- 172.21.15.67:0/734240719 >> v1:172.21.15.187:6789/0 conn(0x7f01840b4c70 legacy=0x7f0184161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.425+0000 7f0189400640 1 -- 172.21.15.67:0/734240719 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01840cf340 con 0x7f01841622b0 2024-04-17T07:29:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.426+0000 7f0189400640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01841622b0 0x7f01840cfc40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f017c009e50 tx=0x7f017c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc48465a68e87a93 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0183400640 1 -- 172.21.15.67:0/734240719 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c02cb00 con 0x7f01841622b0 2024-04-17T07:29:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0183400640 1 -- 172.21.15.67:0/734240719 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f017c02cca0 con 0x7f01841622b0 2024-04-17T07:29:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0183400640 1 -- 172.21.15.67:0/734240719 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c034610 con 0x7f01841622b0 2024-04-17T07:29:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 -- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01841622b0 msgr2=0x7f01840cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01841622b0 0x7f01840cfc40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f017c009e50 tx=0x7f017c02bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 -- 172.21.15.67:0/734240719 shutdown_connections 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01841622b0 0x7f01840cfc40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 --2- 172.21.15.67:0/734240719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01840d2ff0 0x7f01840b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 -- 172.21.15.67:0/734240719 >> 172.21.15.67:0/734240719 conn(0x7f018409da80 msgr2=0x7f018408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.427+0000 7f0189e00640 1 -- 172.21.15.67:0/734240719 shutdown_connections 2024-04-17T07:29:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.428+0000 7f0189e00640 1 -- 172.21.15.67:0/734240719 wait complete. 2024-04-17T07:29:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.429+0000 7f0189e00640 1 Processor -- start 2024-04-17T07:29:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.429+0000 7f0189e00640 1 -- start start 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01840b4c70 0x7f018406de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01840d2ff0 0x7f01840cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01841622b0 0x7f01841f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01840d3550 con 0x7f01840b4c70 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f01840d33d0 con 0x7f01840d2ff0 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0189e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01840d36d0 con 0x7f01841622b0 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.430+0000 7f0188a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01840b4c70 0x7f018406de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01840b4c70 0x7f018406de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49134/0 (socket says 172.21.15.67:49134) 2024-04-17T07:29:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 -- 172.21.15.67:0/1630950707 learned_addr learned my addr 172.21.15.67:0/1630950707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01840d2ff0 0x7f01840cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0189400640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01841622b0 0x7f01841f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01841622b0 msgr2=0x7f01841f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01841622b0 0x7f01841f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01840d2ff0 msgr2=0x7f01840cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01840d2ff0 0x7f01840cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0184157f90 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.431+0000 7f0188a00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01840b4c70 0x7f018406de00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f017800e870 tx=0x7f017800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.432+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178017070 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.432+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f017800c940 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.432+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017800cd30 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.432+0000 7f0189e00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0184158270 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.432+0000 7f0189e00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0184158830 con 0x7f01840b4c70 2024-04-17T07:29:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.433+0000 7f0189e00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01840755d0 con 0x7f01840b4c70 2024-04-17T07:29:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.434+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 31) ==== 91159+0+0 (secure 0 0 0) 0x7f017801a070 con 0x7f01840b4c70 2024-04-17T07:29:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.435+0000 7f0180c00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.435+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f017808bfb0 con 0x7f01840b4c70 2024-04-17T07:29:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.435+0000 7f0183e00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.435+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:38.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.440+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f017805a2b0 con 0x7f01840b4c70 2024-04-17T07:29:38.457 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.457+0000 7f39b784c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:38.462 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.462+0000 7f552e8da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:38.529 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.529+0000 7f552e8da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:38.582 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.582+0000 7f552e8da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:38.586 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.587+0000 7f39b784c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.636+0000 7f0183e00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.636+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:38.648 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.649+0000 7f39b784c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:38.667+0000 7f0189e00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f018408c2a0 con 0x7f016406cab0 2024-04-17T07:29:38.700 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.701+0000 7f39b784c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:38.720 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.720+0000 7f552e8da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:38.756 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.757+0000 7f39b784c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:38.787 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.787+0000 7f552e8da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:38.806 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:38.806+0000 7f39b784c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:38.841 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.841+0000 7f552e8da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:38.902 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.902+0000 7f552e8da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:38.957 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:38.957+0000 7f552e8da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:39.037+0000 7f0183e00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:39.037+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:29:39.189 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.189+0000 7f39b784c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:39.348 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.348+0000 7f39b784c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:39.368 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.368+0000 7f552e8da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:39.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.464+0000 7f39b784c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:39.516 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.516+0000 7f39b784c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:39.538 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.538+0000 7f552e8da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:39.643 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.643+0000 7f39b784c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:39.654 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.654+0000 7f552e8da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:39.708 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.708+0000 7f552e8da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:39.749 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.749+0000 7f39b784c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:39.838+0000 7f0183e00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:39.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:39.838+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:29:39.842 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.843+0000 7f552e8da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:39.923 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.922+0000 7f39b784c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:39.957 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:39.957+0000 7f552e8da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:39.972 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:39.972+0000 7f39b784c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:40.108 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:40.107+0000 7f39b784c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:40.144 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:40.144+0000 7f552e8da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:40.197 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:40.197+0000 7f552e8da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:40.344 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:40.343+0000 7f552e8da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:40.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:40.947+0000 7f39b784c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:41.199 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:41.199+0000 7f552e8da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:41.271 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.271+0000 7f39b784c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:41.329 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.329+0000 7f39b784c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:41.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:41.440+0000 7f0183e00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1436704971 2024-04-17T07:29:41.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:41.440+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T07:29:41.461 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.461+0000 7f39b784c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:41.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:41.482+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 32) ==== 90768+0+0 (secure 0 0 0) 0x7f0178056220 con 0x7f01840b4c70 2024-04-17T07:29:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:41.483+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 msgr2=0x7f016406ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:41.483+0000 7f0180c00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/1436704971,v1:172.21.15.187:6833/1436704971] conn(0x7f016406cab0 0x7f016406ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:41.531 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:41.531+0000 7f552e8da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:41.599 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:41.599+0000 7f552e8da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:41.745 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:41.745+0000 7f552e8da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:41.794 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.794 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.794 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392da00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:41.795 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:41.794+0000 7f392a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.549+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 33) ==== 90895+0+0 (secure 0 0 0) 0x7f017800cd30 con 0x7f01840b4c70 2024-04-17T07:29:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.551+0000 7f0180c00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f0164074960 0x7f0164076d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.551+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 --> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f018408c2a0 con 0x7f0164074960 2024-04-17T07:29:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.557+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f0164074960 0x7f0164076d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.569+0000 7f0183e00640 1 --2- 172.21.15.67:0/1630950707 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f0164074960 0x7f0164076d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f01841617e0 tx=0x7f01700010e0 comp rx=0 tx=0).ready entity=mgr.5731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:42.590 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:42.590+0000 7f3937000640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-17T07:29:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.590+0000 7f0180c00640 1 -- 172.21.15.67:0/1630950707 <== mgr.5731 v2:172.21.15.187:6832/2378630274 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f018408c2a0 con 0x7f0164074960 2024-04-17T07:29:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-17T07:29:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:29:42.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:42.604 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T07:29:42.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:29:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.891+0000 7f716be00640 1 Processor -- start 2024-04-17T07:29:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.891+0000 7f716be00640 1 -- start start 2024-04-17T07:29:42.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.892+0000 7f716be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 0x7f716c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.892+0000 7f716be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716c0d00b0 con 0x7f716c0d33e0 2024-04-17T07:29:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.892+0000 7f716be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716c0d0390 con 0x7f716c15ecb0 2024-04-17T07:29:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.892+0000 7f716be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f716c0d0c30 con 0x7f716c0b4fd0 2024-04-17T07:29:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.893+0000 7f716a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f716c0d33e0 0x7f716c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42226/0 (socket says 172.21.15.67:42226) 2024-04-17T07:29:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.893+0000 7f716a000640 1 -- 172.21.15.67:0/2420513489 learned_addr learned my addr 172.21.15.67:0/2420513489 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.893+0000 7f716b400640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 0x7f716c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.893+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452330257 0 0) 0x7f716c0d00b0 con 0x7f716c0d33e0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.893+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7150003520 con 0x7f716c0d33e0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2278056494 0 0) 0x7f7150003520 con 0x7f716c0d33e0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 msgr2=0x7f716c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 0x7f716c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 >> v1:172.21.15.187:6789/0 conn(0x7f716c15ecb0 legacy=0x7f716c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c0d0670 con 0x7f716c0d33e0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f715c0019f0 con 0x7f716c0d33e0 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 >> v1:172.21.15.67:6789/0 conn(0x7f716c0d33e0 legacy=0x7f716c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71500039e0 0x7f7150003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f7150004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 0x7f7150005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f716c0d0c30 con 0x7f7150004800 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f715000bb80 con 0x7f71500039e0 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f715000bd80 con 0x7f716c0b4fd0 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.894+0000 7f716a000640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f7150004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716aa00640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71500039e0 0x7f7150003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716a000640 1 -- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 msgr2=0x7f7150005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716a000640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4fd0 0x7f7150005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716a000640 1 -- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71500039e0 msgr2=0x7f7150003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716a000640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71500039e0 0x7f7150003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.895+0000 7f716a000640 1 -- 172.21.15.67:0/2420513489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7150003520 con 0x7f7150004800 2024-04-17T07:29:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.896+0000 7f716a000640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f7150004c80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f715c00a840 tx=0x7f7150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76bff6568ec20e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.896+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c00a460 con 0x7f7150004800 2024-04-17T07:29:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.896+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715c034650 con 0x7f7150004800 2024-04-17T07:29:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.896+0000 7f7169600640 1 -- 172.21.15.67:0/2420513489 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c034a40 con 0x7f7150004800 2024-04-17T07:29:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 -- 172.21.15.67:0/2420513489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 msgr2=0x7f7150004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 --2- 172.21.15.67:0/2420513489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f7150004c80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f715c00a840 tx=0x7f7150005830 comp rx=0 tx=0).stop 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716b400640 1 -- 172.21.15.67:0/2420513489 reap_dead start 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 -- 172.21.15.67:0/2420513489 shutdown_connections 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 -- 172.21.15.67:0/2420513489 >> 172.21.15.67:0/2420513489 conn(0x7f716c08bff0 msgr2=0x7f716c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 -- 172.21.15.67:0/2420513489 shutdown_connections 2024-04-17T07:29:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.897+0000 7f716be00640 1 -- 172.21.15.67:0/2420513489 wait complete. 2024-04-17T07:29:42.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.899+0000 7f716be00640 1 Processor -- start 2024-04-17T07:29:42.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.899+0000 7f716be00640 1 -- start start 2024-04-17T07:29:42.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f716c162230 0x7f716c0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4400 0x7f716c14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f716c15f210 con 0x7f7150004800 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f716c15f090 con 0x7f716c162230 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f716c0cfab0 con 0x7f716c0b4400 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49150/0 (socket says 172.21.15.67:49150) 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.900+0000 7f716aa00640 1 -- 172.21.15.67:0/3529569465 learned_addr learned my addr 172.21.15.67:0/3529569465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 -- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4400 msgr2=0x7f716c14db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4400 0x7f716c14db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 -- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f716c162230 msgr2=0x7f716c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f716c162230 0x7f716c0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 -- 172.21.15.67:0/3529569465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c14b470 con 0x7f7150004800 2024-04-17T07:29:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716aa00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f715400e550 tx=0x7f715400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71540052e0 con 0x7f7150004800 2024-04-17T07:29:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7154018070 con 0x7f7150004800 2024-04-17T07:29:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154003830 con 0x7f7150004800 2024-04-17T07:29:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.901+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c161470 con 0x7f7150004800 2024-04-17T07:29:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.902+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c161a30 con 0x7f7150004800 2024-04-17T07:29:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.903+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716c068a10 con 0x7f7150004800 2024-04-17T07:29:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.904+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) ==== 90895+0+0 (secure 0 0 0) 0x7f71540039d0 con 0x7f7150004800 2024-04-17T07:29:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.905+0000 7f7162a00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 0x7f714806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.905+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f715408b620 con 0x7f7150004800 2024-04-17T07:29:42.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.905+0000 7f716a000640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 0x7f714806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.907+0000 7f716a000640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 0x7f714806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f715c034000 tx=0x7f715c007000 comp rx=0 tx=0).ready entity=mgr.5731 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:42.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:42.910+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7154059b60 con 0x7f7150004800 2024-04-17T07:29:43.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.173+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f716c0b5a70 con 0x7f7150004800 2024-04-17T07:29:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.548+0000 7f7162a00640 1 -- 172.21.15.67:0/3529569465 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) ==== 93+0+0 (secure 0 0 0) 0x7f7154059460 con 0x7f7150004800 2024-04-17T07:29:43.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.551+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 msgr2=0x7f714806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:43.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.551+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 0x7f714806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f715c034000 tx=0x7f715c007000 comp rx=0 tx=0).stop 2024-04-17T07:29:43.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.551+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 msgr2=0x7f716c14af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:43.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.551+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f715400e550 tx=0x7f715400ea10 comp rx=0 tx=0).stop 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 shutdown_connections 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f716c0b4400 0x7f716c14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f716c162230 0x7f716c0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f714806c790 0x7f714806ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 --2- 172.21.15.67:0/3529569465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7150004800 0x7f716c14af30 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.552+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 >> 172.21.15.67:0/3529569465 conn(0x7f716c08bff0 msgr2=0x7f716c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.553+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 shutdown_connections 2024-04-17T07:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:43.553+0000 7f716be00640 1 -- 172.21.15.67:0/3529569465 wait complete. 2024-04-17T07:29:43.577 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:29:43.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:43.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:43.771 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:29:43.771 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:43.771 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:43.771 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:43.823 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:43.823 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:43.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:44.097 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.097+0000 7fa306242040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:44.127 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.127+0000 7f7132a42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:29:44.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.130+0000 7fddb7200640 1 Processor -- start 2024-04-17T07:29:44.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.130+0000 7fddb7200640 1 -- start start 2024-04-17T07:29:44.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddb7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddb815ca20 0x7fddb80d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddb7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddb815ce00 con 0x7fddb80d0720 2024-04-17T07:29:44.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddb7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddb815d090 con 0x7fddb809b170 2024-04-17T07:29:44.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddb7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fddb815d330 con 0x7fddb815ca20 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddb5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddb815ca20 0x7fddb80d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddafe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddb80d0720 0x7fddb809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57168/0 (socket says 172.21.15.67:57168) 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.132+0000 7fddafe00640 1 -- 172.21.15.67:0/1299679077 learned_addr learned my addr 172.21.15.67:0/1299679077 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.133+0000 7fddb5e00640 1 -- 172.21.15.67:0/1299679077 >> v1:172.21.15.187:6789/0 conn(0x7fddb809b170 legacy=0x7fddb80cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.133+0000 7fddb5e00640 1 -- 172.21.15.67:0/1299679077 >> v1:172.21.15.67:6789/0 conn(0x7fddb80d0720 legacy=0x7fddb809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.133+0000 7fddb5e00640 1 -- 172.21.15.67:0/1299679077 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb815d610 con 0x7fddb815ca20 2024-04-17T07:29:44.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.134+0000 7fddb5e00640 1 --2- 172.21.15.67:0/1299679077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb815ca20 0x7fddb80d01e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097a0 tx=0x7fdda802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0542fa9ecbd3bbc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:44.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.134+0000 7fddb5400640 1 -- 172.21.15.67:0/1299679077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda8035020 con 0x7fddb815ca20 2024-04-17T07:29:44.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.134+0000 7fddb5400640 1 -- 172.21.15.67:0/1299679077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda8031830 con 0x7fddb815ca20 2024-04-17T07:29:44.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.134+0000 7fddb5400640 1 -- 172.21.15.67:0/1299679077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda8031c00 con 0x7fddb815ca20 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 -- 172.21.15.67:0/1299679077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb815ca20 msgr2=0x7fddb80d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 --2- 172.21.15.67:0/1299679077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb815ca20 0x7fddb80d01e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097a0 tx=0x7fdda802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 -- 172.21.15.67:0/1299679077 shutdown_connections 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 --2- 172.21.15.67:0/1299679077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb815ca20 0x7fddb80d01e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 -- 172.21.15.67:0/1299679077 >> 172.21.15.67:0/1299679077 conn(0x7fddb80b4100 msgr2=0x7fddb80b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 -- 172.21.15.67:0/1299679077 shutdown_connections 2024-04-17T07:29:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.135+0000 7fddb7200640 1 -- 172.21.15.67:0/1299679077 wait complete. 2024-04-17T07:29:44.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.136+0000 7fddb7200640 1 Processor -- start 2024-04-17T07:29:44.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.137+0000 7fddb7200640 1 -- start start 2024-04-17T07:29:44.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb809b170 0x7fddb80ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb815ca20 0x7fddb80cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddb809bea0 con 0x7fddb80d0720 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddb809bd20 con 0x7fddb815ca20 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddb80cd8a0 con 0x7fddb809b170 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57900/0 (socket says 172.21.15.67:57900) 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 -- 172.21.15.67:0/318372805 learned_addr learned my addr 172.21.15.67:0/318372805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddb5e00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb809b170 0x7fddb80ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb809b170 msgr2=0x7fddb80ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb809b170 0x7fddb80ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb815ca20 msgr2=0x7fddb80cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb815ca20 0x7fddb80cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.138+0000 7fddafe00640 1 -- 172.21.15.67:0/318372805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb80cb260 con 0x7fddb80d0720 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddafe00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c470 tx=0x7fdda0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0017070 con 0x7fddb80d0720 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda0005ce0 con 0x7fddb80d0720 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda00072f0 con 0x7fddb80d0720 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb80cb540 con 0x7fddb80d0720 2024-04-17T07:29:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.139+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb80cbb00 con 0x7fddb80d0720 2024-04-17T07:29:44.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.141+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb80755d0 con 0x7fddb80d0720 2024-04-17T07:29:44.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.142+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7fdda001a070 con 0x7fddb80d0720 2024-04-17T07:29:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.142+0000 7fddae000640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 0x7fdd9406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.143+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7fdda008b2e0 con 0x7fddb80d0720 2024-04-17T07:29:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.143+0000 7fddb5e00640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 msgr2=0x7fdd9406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.143+0000 7fddb5e00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 0x7fdd9406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.148+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdda0059400 con 0x7fddb80d0720 2024-04-17T07:29:44.152 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.152+0000 7fa306242040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:44.189 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.189+0000 7f7132a42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:29:44.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.344+0000 7fddb5e00640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 msgr2=0x7fdd9406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:44.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.344+0000 7fddb5e00640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 0x7fdd9406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:44.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.354+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddb809b590 con 0x7fddb80d0720 2024-04-17T07:29:44.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.356+0000 7fddae000640 1 -- 172.21.15.67:0/318372805 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+17123 (secure 0 0 0) 0x7fdda0058d00 con 0x7fddb80d0720 2024-04-17T07:29:44.356 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:44.356 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:41.466143+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6832/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1567626846":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/831200072":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/3826805311":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/12378685":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1955748639":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1795500056":"2024-04-18T07:29:41.466074+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 msgr2=0x7fdd9406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 0x7fdd9406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 msgr2=0x7fddb8150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdda000c470 tx=0x7fdda0001e70 comp rx=0 tx=0).stop 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 shutdown_connections 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddb815ca20 0x7fddb80cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7fdd9406ca10 0x7fdd9406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddb80d0720 0x7fddb8150a00 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 --2- 172.21.15.67:0/318372805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddb809b170 0x7fddb80ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 >> 172.21.15.67:0/318372805 conn(0x7fddb80b4100 msgr2=0x7fddb80cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 shutdown_connections 2024-04-17T07:29:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.359+0000 7fddb7200640 1 -- 172.21.15.67:0/318372805 wait complete. 2024-04-17T07:29:44.377 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:29:24.457001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:29:25.538749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:44.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:44.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.583+0000 7f9ba3e00640 1 Processor -- start 2024-04-17T07:29:44.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.583+0000 7f9ba3e00640 1 -- start start 2024-04-17T07:29:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ba415ec20 0x7f9ba40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9ba40b3b50 0x7f9ba406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba406cf80 0x7f9ba40cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9ba40d4f70 con 0x7f9ba415ec20 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9ba40d5250 con 0x7f9ba40b3b50 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba40cf4d0 con 0x7f9ba406cf80 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ba415ec20 0x7f9ba40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ba415ec20 0x7f9ba40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57916/0 (socket says 172.21.15.67:57916) 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.585+0000 7f9ba2a00640 1 -- 172.21.15.67:0/2205734665 learned_addr learned my addr 172.21.15.67:0/2205734665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9b9a000640 1 --2- 172.21.15.67:0/2205734665 >> v2:172.21.15.187:3300/0 conn(0x7f9ba40b3b50 0x7f9ba406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba3400640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba406cf80 0x7f9ba40cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba2a00640 1 -- 172.21.15.67:0/2205734665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba406cf80 msgr2=0x7f9ba40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba406cf80 0x7f9ba40cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba2a00640 1 -- 172.21.15.67:0/2205734665 >> v2:172.21.15.187:3300/0 conn(0x7f9ba40b3b50 msgr2=0x7f9ba406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/2205734665 >> v2:172.21.15.187:3300/0 conn(0x7f9ba40b3b50 0x7f9ba406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.586+0000 7f9ba2a00640 1 -- 172.21.15.67:0/2205734665 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba40cf7b0 con 0x7f9ba415ec20 2024-04-17T07:29:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.587+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba415ec20 0x7f9ba40d22b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b71e0 tx=0x7f9b9402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a224e1ddbe405609 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.587+0000 7f9ba2000640 1 -- 172.21.15.67:0/2205734665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9402c960 con 0x7f9ba415ec20 2024-04-17T07:29:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.587+0000 7f9ba2000640 1 -- 172.21.15.67:0/2205734665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b9402cb00 con 0x7f9ba415ec20 2024-04-17T07:29:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.587+0000 7f9ba2000640 1 -- 172.21.15.67:0/2205734665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b940343f0 con 0x7f9ba415ec20 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 -- 172.21.15.67:0/2205734665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba415ec20 msgr2=0x7f9ba40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba415ec20 0x7f9ba40d22b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40b71e0 tx=0x7f9b9402bdc0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 -- 172.21.15.67:0/2205734665 shutdown_connections 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba406cf80 0x7f9ba40cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/2205734665 >> v2:172.21.15.187:3300/0 conn(0x7f9ba40b3b50 0x7f9ba406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/2205734665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba415ec20 0x7f9ba40d22b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 -- 172.21.15.67:0/2205734665 >> 172.21.15.67:0/2205734665 conn(0x7f9ba408bff0 msgr2=0x7f9ba40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 -- 172.21.15.67:0/2205734665 shutdown_connections 2024-04-17T07:29:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.588+0000 7f9ba3e00640 1 -- 172.21.15.67:0/2205734665 wait complete. 2024-04-17T07:29:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.590+0000 7f9ba3e00640 1 Processor -- start 2024-04-17T07:29:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.590+0000 7f9ba3e00640 1 -- start start 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ba40b3b50 0x7f9ba40ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba415ec20 0x7f9ba40b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba40d0390 con 0x7f9ba406cf80 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba40d0210 con 0x7f9ba40b3b50 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.591+0000 7f9ba3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba40d0510 con 0x7f9ba415ec20 2024-04-17T07:29:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57930/0 (socket says 172.21.15.67:57930) 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 -- 172.21.15.67:0/1911257977 learned_addr learned my addr 172.21.15.67:0/1911257977 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9b9a000640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ba40b3b50 0x7f9ba40ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba3400640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba415ec20 0x7f9ba40b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba415ec20 msgr2=0x7f9ba40b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba415ec20 0x7f9ba40b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ba40b3b50 msgr2=0x7f9ba40ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ba40b3b50 0x7f9ba40ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 -- 172.21.15.67:0/1911257977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba40cbb60 con 0x7f9ba406cf80 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.592+0000 7f9ba2a00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9b94000f80 tx=0x7f9b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.593+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94043020 con 0x7f9ba406cf80 2024-04-17T07:29:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.593+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b9403e450 con 0x7f9ba406cf80 2024-04-17T07:29:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.593+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9403e840 con 0x7f9ba406cf80 2024-04-17T07:29:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.593+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba40cbe40 con 0x7f9ba406cf80 2024-04-17T07:29:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.593+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba40cc400 con 0x7f9ba406cf80 2024-04-17T07:29:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.594+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba4068a10 con 0x7f9ba406cf80 2024-04-17T07:29:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.595+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f9b94046020 con 0x7f9ba406cf80 2024-04-17T07:29:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.596+0000 7f9b9be00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 0x7f9b8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.597+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f9b940ae780 con 0x7f9ba406cf80 2024-04-17T07:29:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.597+0000 7f9b9a000640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 msgr2=0x7f9b8006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.597+0000 7f9b9a000640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 0x7f9b8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:44.599 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.599+0000 7fa306242040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:44.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.601+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9b9404d030 con 0x7f9ba406cf80 2024-04-17T07:29:44.652 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.652+0000 7fa306242040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:44.667 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.667+0000 7f7132a42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:29:44.703 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.703+0000 7fa306242040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:44.723 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.723+0000 7f7132a42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:29:44.764 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.764+0000 7fa306242040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:44.776 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.776+0000 7f7132a42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:29:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.798+0000 7f9b9a000640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 msgr2=0x7f9b8006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.798+0000 7f9b9a000640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 0x7f9b8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.803+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ba40cacb0 con 0x7f9ba406cf80 2024-04-17T07:29:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.804+0000 7f9b9be00640 1 -- 172.21.15.67:0/1911257977 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f9b9407c940 con 0x7f9ba406cf80 2024-04-17T07:29:44.804 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.806+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 msgr2=0x7f9b8006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.806+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 0x7f9b8006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.806+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 msgr2=0x7f9ba41527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.806+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9b94000f80 tx=0x7f9b94004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 shutdown_connections 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ba415ec20 0x7f9ba40b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ba40b3b50 0x7f9ba40ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f9b8006ca10 0x7f9b8006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 --2- 172.21.15.67:0/1911257977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba406cf80 0x7f9ba41527a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 >> 172.21.15.67:0/1911257977 conn(0x7f9ba408bff0 msgr2=0x7f9ba4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 shutdown_connections 2024-04-17T07:29:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:44.807+0000 7f9ba3e00640 1 -- 172.21.15.67:0/1911257977 wait complete. 2024-04-17T07:29:44.813 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.813+0000 7fa306242040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:44.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:29:44.842 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.842+0000 7f7132a42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:29:44.895 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:44.895+0000 7f7132a42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:29:44.941 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:44.941+0000 7fa306242040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:45.004 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.004+0000 7fa306242040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:45.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.034+0000 7f7132a42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:29:45.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.039+0000 7f1d90c00640 1 Processor -- start 2024-04-17T07:29:45.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.039+0000 7f1d90c00640 1 -- start start 2024-04-17T07:29:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d90c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d8c0ae790 0x7f1d8c0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d90c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 0x7f1d8c06c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d90c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d8c0b3cc0 con 0x7f1d8c06ce10 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d90c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1d8c15c3a0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d90c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1d8c15c680 con 0x7f1d8c15bae0 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d8c0ae790 0x7f1d8c0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d8c0ae790 0x7f1d8c0aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57940/0 (socket says 172.21.15.67:57940) 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.041+0000 7f1d8b400640 1 -- 172.21.15.67:0/3871764620 learned_addr learned my addr 172.21.15.67:0/3871764620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:45.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8aa00640 1 --2- 172.21.15.67:0/3871764620 >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 0x7f1d8c06c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8b400640 1 -- 172.21.15.67:0/3871764620 >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 msgr2=0x7f1d8c06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8b400640 1 --2- 172.21.15.67:0/3871764620 >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 0x7f1d8c06c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8b400640 1 -- 172.21.15.67:0/3871764620 >> v1:172.21.15.187:6789/0 conn(0x7f1d8c06ce10 legacy=0x7f1d8c06f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8b400640 1 -- 172.21.15.67:0/3871764620 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c15c960 con 0x7f1d8c0ae790 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.042+0000 7f1d8aa00640 1 --2- 172.21.15.67:0/3871764620 >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 0x7f1d8c06c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:45.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.043+0000 7f1d8b400640 1 --2- 172.21.15.67:0/3871764620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c0aff10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c06dd60 tx=0x7f1d7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2366f862ecdedb95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.043+0000 7f1d8a000640 1 -- 172.21.15.67:0/3871764620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c034070 con 0x7f1d8c0ae790 2024-04-17T07:29:45.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.043+0000 7f1d8a000640 1 -- 172.21.15.67:0/3871764620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7c030860 con 0x7f1d8c0ae790 2024-04-17T07:29:45.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.043+0000 7f1d8a000640 1 -- 172.21.15.67:0/3871764620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c030c30 con 0x7f1d8c0ae790 2024-04-17T07:29:45.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 -- 172.21.15.67:0/3871764620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 msgr2=0x7f1d8c0aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 --2- 172.21.15.67:0/3871764620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c0aff10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c06dd60 tx=0x7f1d7c02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 -- 172.21.15.67:0/3871764620 shutdown_connections 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 --2- 172.21.15.67:0/3871764620 >> v2:172.21.15.187:3300/0 conn(0x7f1d8c15bae0 0x7f1d8c06c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 --2- 172.21.15.67:0/3871764620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c0aff10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 -- 172.21.15.67:0/3871764620 >> 172.21.15.67:0/3871764620 conn(0x7f1d8c072460 msgr2=0x7f1d8c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 -- 172.21.15.67:0/3871764620 shutdown_connections 2024-04-17T07:29:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.044+0000 7f1d90c00640 1 -- 172.21.15.67:0/3871764620 wait complete. 2024-04-17T07:29:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.046+0000 7f1d90c00640 1 Processor -- start 2024-04-17T07:29:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.046+0000 7f1d90c00640 1 -- start start 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c06ce10 0x7f1d8c0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d8c15bae0 0x7f1d8c06e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c15c0e0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c15bf60 con 0x7f1d8c06ce10 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d90c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8c0b5490 con 0x7f1d8c15bae0 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d8aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d8aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57952/0 (socket says 172.21.15.67:57952) 2024-04-17T07:29:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d8aa00640 1 -- 172.21.15.67:0/1293595129 learned_addr learned my addr 172.21.15.67:0/1293595129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.047+0000 7f1d8b400640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c06ce10 0x7f1d8c0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8be00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d8c15bae0 0x7f1d8c06e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d8c15bae0 msgr2=0x7f1d8c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d8c15bae0 0x7f1d8c06e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c06ce10 msgr2=0x7f1d8c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c06ce10 0x7f1d8c0cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 -- 172.21.15.67:0/1293595129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c132db0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.048+0000 7f1d8aa00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800d7d0 tx=0x7f1d7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.049+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78004030 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.049+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d780041d0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.049+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800b6c0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.049+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d8c133090 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.049+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d8c15d360 con 0x7f1d8c0ae790 2024-04-17T07:29:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.050+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d8c0755d0 con 0x7f1d8c0ae790 2024-04-17T07:29:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.052+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f1d7800b860 con 0x7f1d8c0ae790 2024-04-17T07:29:45.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.053+0000 7f1d83400640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 0x7f1d6c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.054+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f1d7808c060 con 0x7f1d8c0ae790 2024-04-17T07:29:45.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.054+0000 7f1d8b400640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 msgr2=0x7f1d6c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:45.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.054+0000 7f1d8b400640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 0x7f1d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:45.055 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.055+0000 7fa306242040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.058+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1d7805a210 con 0x7f1d8c0ae790 2024-04-17T07:29:45.103 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.103+0000 7f7132a42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:45.112 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.112+0000 7fa306242040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:45.156 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.156+0000 7f7132a42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:29:45.163 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.163+0000 7fa306242040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:45.216 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.216+0000 7f7132a42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:29:45.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.255+0000 7f1d8b400640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 msgr2=0x7f1d6c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:45.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.255+0000 7f1d8b400640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 0x7f1d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:45.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.263+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d8c13c170 con 0x7f1d8c0ae790 2024-04-17T07:29:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.264+0000 7f1d83400640 1 -- 172.21.15.67:0/1293595129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f1d78059b10 con 0x7f1d8c0ae790 2024-04-17T07:29:45.264 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:29:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 msgr2=0x7f1d6c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 0x7f1d6c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 msgr2=0x7f1d8c154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800d7d0 tx=0x7f1d7800dc90 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 shutdown_connections 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d8c15bae0 0x7f1d8c06e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f1d6c06ca10 0x7f1d6c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c0ae790 0x7f1d8c154c40 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 --2- 172.21.15.67:0/1293595129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d8c06ce10 0x7f1d8c0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 >> 172.21.15.67:0/1293595129 conn(0x7f1d8c072460 msgr2=0x7f1d8c13e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 shutdown_connections 2024-04-17T07:29:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.267+0000 7f1d90c00640 1 -- 172.21.15.67:0/1293595129 wait complete. 2024-04-17T07:29:45.271 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.271+0000 7f7132a42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:29:45.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:29:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.537+0000 7f8b21a00640 1 Processor -- start 2024-04-17T07:29:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.538+0000 7f8b21a00640 1 -- start start 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b1c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8b1c0cfcd0 0x7f8b1c0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1c0d4f70 con 0x7f8b1c15ec20 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8b1c0d5250 con 0x7f8b1c0cfcd0 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b1c0cf4d0 con 0x7f8b1c0b3b50 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b1be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b1c15ec20 0x7f8b1c06ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57174/0 (socket says 172.21.15.67:57174) 2024-04-17T07:29:45.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b1be00640 1 -- 172.21.15.67:0/1005247939 learned_addr learned my addr 172.21.15.67:0/1005247939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.539+0000 7f8b21000640 1 --2- 172.21.15.67:0/1005247939 >> v2:172.21.15.187:3300/0 conn(0x7f8b1c0cfcd0 0x7f8b1c0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b13e00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b1c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605413522 0 0) 0x7f8b1c0d4f70 con 0x7f8b1c15ec20 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b00003520 con 0x7f8b1c15ec20 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2727774683 0 0) 0x7f8b00003520 con 0x7f8b1c15ec20 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 msgr2=0x7f8b1c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b1c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 >> v2:172.21.15.187:3300/0 conn(0x7f8b1c0cfcd0 msgr2=0x7f8b1c0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 --2- 172.21.15.67:0/1005247939 >> v2:172.21.15.187:3300/0 conn(0x7f8b1c0cfcd0 0x7f8b1c0d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c0cf7b0 con 0x7f8b1c15ec20 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8b0c001670 con 0x7f8b1c15ec20 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 >> v1:172.21.15.67:6789/0 conn(0x7f8b1c15ec20 legacy=0x7f8b1c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b00003f30 0x7f8b00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00004d40 0x7f8b000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c0d5250 con 0x7f8b00003f30 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c0cf4d0 con 0x7f8b00004d40 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.540+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0000baa0 con 0x7f8b1c0b3b50 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b00003f30 0x7f8b00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b21000640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00004d40 0x7f8b000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b1be00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 -- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 msgr2=0x7f8b000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b3b50 0x7f8b000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 -- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00004d40 msgr2=0x7f8b000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00004d40 0x7f8b000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.541+0000 7f8b13e00640 1 -- 172.21.15.67:0/1005247939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b00003520 con 0x7f8b00003f30 2024-04-17T07:29:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.542+0000 7f8b13e00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b00003f30 0x7f8b00004800 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009ab0 tx=0x7f8b00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa8e0d011de1d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.542+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04009680 con 0x7f8b00003f30 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.542+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b04005990 con 0x7f8b00003f30 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.542+0000 7f8b1b400640 1 -- 172.21.15.67:0/1005247939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04005d80 con 0x7f8b00003f30 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.543+0000 7f8b21a00640 1 -- 172.21.15.67:0/1005247939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b00003f30 msgr2=0x7f8b00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.543+0000 7f8b21a00640 1 --2- 172.21.15.67:0/1005247939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b00003f30 0x7f8b00004800 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009ab0 tx=0x7f8b00005830 comp rx=0 tx=0).stop 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.543+0000 7f8b21000640 1 -- 172.21.15.67:0/1005247939 reap_dead start 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.544+0000 7f8b21a00640 1 -- 172.21.15.67:0/1005247939 shutdown_connections 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.544+0000 7f8b21a00640 1 -- 172.21.15.67:0/1005247939 >> 172.21.15.67:0/1005247939 conn(0x7f8b1c08bff0 msgr2=0x7f8b1c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.544+0000 7f8b21a00640 1 -- 172.21.15.67:0/1005247939 shutdown_connections 2024-04-17T07:29:45.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.544+0000 7f8b21a00640 1 -- 172.21.15.67:0/1005247939 wait complete. 2024-04-17T07:29:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.545+0000 7f8b21a00640 1 Processor -- start 2024-04-17T07:29:45.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 -- start start 2024-04-17T07:29:45.546 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.546+0000 7fa306242040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 0x7f8b1c1f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b7c50 0x7f8b1c0b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c06d2d0 con 0x7f8b1c0b5190 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1c06d150 con 0x7f8b00003f30 2024-04-17T07:29:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.546+0000 7f8b21a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b1c0d22b0 con 0x7f8b1c0b7c50 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57972/0 (socket says 172.21.15.67:57972) 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 -- 172.21.15.67:0/2840642981 learned_addr learned my addr 172.21.15.67:0/2840642981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b1be00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 0x7f8b1c1f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b13e00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b7c50 0x7f8b1c0b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b7c50 msgr2=0x7f8b1c0b80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b7c50 0x7f8b1c0b80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 msgr2=0x7f8b1c1f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 0x7f8b1c1f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b21000640 1 -- 172.21.15.67:0/2840642981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c0b86f0 con 0x7f8b1c0b5190 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.547+0000 7f8b1be00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 0x7f8b1c1f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b21000640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8b14007cf0 tx=0x7f8b140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14011de0 con 0x7f8b1c0b5190 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1401b070 con 0x7f8b1c0b5190 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1c0b89d0 con 0x7f8b1c0b5190 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14003990 con 0x7f8b1c0b5190 2024-04-17T07:29:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.548+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1c15f490 con 0x7f8b1c0b5190 2024-04-17T07:29:45.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.549+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b1c068a10 con 0x7f8b1c0b5190 2024-04-17T07:29:45.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.550+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f8b14005d10 con 0x7f8b1c0b5190 2024-04-17T07:29:45.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.551+0000 7f8b18c00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 0x7f8af806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.551+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f8b1408c2a0 con 0x7f8b1c0b5190 2024-04-17T07:29:45.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.551+0000 7f8b1be00640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 msgr2=0x7f8af806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:45.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.551+0000 7f8b1be00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 0x7f8af806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.555+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8b14014350 con 0x7f8b1c0b5190 2024-04-17T07:29:45.686 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.686+0000 7f7132a42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:29:45.708 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.708+0000 7fa306242040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:45.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.752+0000 7f8b1be00640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 msgr2=0x7f8af806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:45.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.752+0000 7f8b1be00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 0x7f8af806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.757+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8b1c15f690 con 0x7f8b1c0b5190 2024-04-17T07:29:45.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.758+0000 7f8b18c00640 1 -- 172.21.15.67:0/2840642981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f8b1c15f690 con 0x7f8b1c0b5190 2024-04-17T07:29:45.758 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:29:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 msgr2=0x7f8af806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 0x7f8af806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 msgr2=0x7f8b1c0b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8b14007cf0 tx=0x7f8b140115a0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 shutdown_connections 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8af806ca40 0x7f8af806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b1c0b5190 0x7f8b1c0b5640 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b1c0b7c50 0x7f8b1c0b80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 --2- 172.21.15.67:0/2840642981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b00003f30 0x7f8b1c1f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.760+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 >> 172.21.15.67:0/2840642981 conn(0x7f8b1c08bff0 msgr2=0x7f8b1c162550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.761+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 shutdown_connections 2024-04-17T07:29:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.761+0000 7f8b21a00640 1 -- 172.21.15.67:0/2840642981 wait complete. 2024-04-17T07:29:45.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:29:45.817 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.817+0000 7fa306242040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:45.859 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.859+0000 7f7132a42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:29:45.867 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.867+0000 7fa306242040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:45.975 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:45.975+0000 7f7132a42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:29:45.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.985+0000 7f8a73c00640 1 Processor -- start 2024-04-17T07:29:45.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.985+0000 7f8a73c00640 1 -- start start 2024-04-17T07:29:45.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a73c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a73c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 0x7f8a6c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a73c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6c163860 con 0x7f8a6c0d1090 2024-04-17T07:29:45.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a73c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8a6c0d03b0 con 0x7f8a6c15eca0 2024-04-17T07:29:45.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a73c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c0d1d40 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a71200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a71200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57978/0 (socket says 172.21.15.67:57978) 2024-04-17T07:29:45.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.987+0000 7f8a71200640 1 -- 172.21.15.67:0/501986839 learned_addr learned my addr 172.21.15.67:0/501986839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:45.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a71200640 1 -- 172.21.15.67:0/501986839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 msgr2=0x7f8a6c0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a6be00640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 0x7f8a6c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a71200640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 0x7f8a6c0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a71200640 1 -- 172.21.15.67:0/501986839 >> v1:172.21.15.187:6789/0 conn(0x7f8a6c0d1090 legacy=0x7f8a6c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:45.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a71200640 1 -- 172.21.15.67:0/501986839 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c0d2c30 con 0x7f8a6c15eca0 2024-04-17T07:29:45.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.988+0000 7f8a6be00640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 0x7f8a6c0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:45.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.989+0000 7f8a71200640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009870 tx=0x7f8a6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50029c6276c8f4a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.989+0000 7f8a6b400640 1 -- 172.21.15.67:0/501986839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60035020 con 0x7f8a6c15eca0 2024-04-17T07:29:45.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.989+0000 7f8a6b400640 1 -- 172.21.15.67:0/501986839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a60031640 con 0x7f8a6c15eca0 2024-04-17T07:29:45.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.989+0000 7f8a6b400640 1 -- 172.21.15.67:0/501986839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6002d040 con 0x7f8a6c15eca0 2024-04-17T07:29:45.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 -- 172.21.15.67:0/501986839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c15eca0 msgr2=0x7f8a6c0d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009870 tx=0x7f8a6002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 -- 172.21.15.67:0/501986839 shutdown_connections 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c0cfa10 0x7f8a6c0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 --2- 172.21.15.67:0/501986839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c15eca0 0x7f8a6c0d39f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 -- 172.21.15.67:0/501986839 >> 172.21.15.67:0/501986839 conn(0x7f8a6c067600 msgr2=0x7f8a6c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 -- 172.21.15.67:0/501986839 shutdown_connections 2024-04-17T07:29:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.990+0000 7f8a73c00640 1 -- 172.21.15.67:0/501986839 wait complete. 2024-04-17T07:29:45.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.992+0000 7f8a73c00640 1 Processor -- start 2024-04-17T07:29:45.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.992+0000 7f8a73c00640 1 -- start start 2024-04-17T07:29:45.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:45.992+0000 7fa306242040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a6c0d1090 0x7f8a6c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c15eca0 0x7f8a6c1494b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0d34d0 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0d31f0 con 0x7f8a6c0d1090 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a73c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c15f080 con 0x7f8a6c15eca0 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57982/0 (socket says 172.21.15.67:57982) 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 -- 172.21.15.67:0/853393004 learned_addr learned my addr 172.21.15.67:0/853393004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a6be00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a6c0d1090 0x7f8a6c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c15eca0 0x7f8a6c1494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c15eca0 msgr2=0x7f8a6c1494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c15eca0 0x7f8a6c1494b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a6c0d1090 msgr2=0x7f8a6c0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.993+0000 7f8a71200640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a6c0d1090 0x7f8a6c0b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a71200640 1 -- 172.21.15.67:0/853393004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c0b5eb0 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a71200640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8a6002b930 tx=0x7f8a600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60040070 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a60039c10 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c0b6190 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60035040 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.994+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c1f8900 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.995+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6c068ca0 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.996+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f8a6004b060 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.997+0000 7f8a68c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 0x7f8a4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.997+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f8a600ad9f0 con 0x7f8a6c0cfa10 2024-04-17T07:29:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.997+0000 7f8a6be00640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 msgr2=0x7f8a4c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:45.997+0000 7f8a6be00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 0x7f8a4c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:46.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.002+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8a6002f300 con 0x7f8a6c0cfa10 2024-04-17T07:29:46.031 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.032+0000 7f7132a42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:29:46.100 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:46.100+0000 7fa306242040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:46.167 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.167+0000 7f7132a42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.198+0000 7f8a6be00640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 msgr2=0x7f8a4c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.198+0000 7f8a6be00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 0x7f8a4c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:46.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.220+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8a6c0d2970 con 0x7f8a6c0cfa10 2024-04-17T07:29:46.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.221+0000 7f8a68c00640 1 -- 172.21.15.67:0/853393004 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v85) ==== 80+0+475 (secure 0 0 0) 0x7f8a6007bb10 con 0x7f8a6c0cfa10 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/1436704971 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/1436704971 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1567626846 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/831200072 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3826805311 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/12378685 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1955748639 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1795500056 2024-04-18T07:29:41.466074+0000 2024-04-17T07:29:46.222 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.225+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 msgr2=0x7f8a4c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.225+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 0x7f8a4c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.225+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 msgr2=0x7f8a6c1f8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.225+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8a6002b930 tx=0x7f8a600046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 shutdown_connections 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a6c15eca0 0x7f8a6c1494b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a6c0d1090 0x7f8a6c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f8a4c06ca10 0x7f8a4c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 --2- 172.21.15.67:0/853393004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0cfa10 0x7f8a6c1f8e00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 >> 172.21.15.67:0/853393004 conn(0x7f8a6c067600 msgr2=0x7f8a6c0cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 shutdown_connections 2024-04-17T07:29:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.226+0000 7f8a73c00640 1 -- 172.21.15.67:0/853393004 wait complete. 2024-04-17T07:29:46.245 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:29:46.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:46.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:29:46.276 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:46.276+0000 7fa306242040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:46.284 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.284+0000 7f7132a42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:29:46.326 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:46.325+0000 7fa306242040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:46.348 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:29:46.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:29:46.348 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:29:46.437 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:46.437 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:46.438 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:46.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:46.464+0000 7fa306242040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:46.474 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.473+0000 7f7132a42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:29:46.489 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:46.489 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:46.489 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:46.489 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:46.526 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.526+0000 7f7132a42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:29:46.543 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:46.543 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:29:46.543 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:46.543 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:46.595 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:46.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:46.673 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:46.673+0000 7f7132a42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:29:46.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.798+0000 7f6361a00640 1 Processor -- start 2024-04-17T07:29:46.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.798+0000 7f6361a00640 1 -- start start 2024-04-17T07:29:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 0x7f635c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f635c141ed0 0x7f635c142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635c0cd4a0 con 0x7f635c06b8c0 2024-04-17T07:29:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f635c15ce30 con 0x7f635c141ed0 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f635c1607f0 con 0x7f635c160280 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f635c141ed0 0x7f635c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f635c141ed0 0x7f635c142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57996/0 (socket says 172.21.15.67:57996) 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.800+0000 7f6361000640 1 -- 172.21.15.67:0/831408185 learned_addr learned my addr 172.21.15.67:0/831408185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f635b400640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 0x7f635c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f6361000640 1 -- 172.21.15.67:0/831408185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 msgr2=0x7f635c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f6361000640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 0x7f635c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f6361000640 1 -- 172.21.15.67:0/831408185 >> v1:172.21.15.187:6789/0 conn(0x7f635c06b8c0 legacy=0x7f635c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f6361000640 1 -- 172.21.15.67:0/831408185 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c160ad0 con 0x7f635c141ed0 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.801+0000 7f635b400640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 0x7f635c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.802+0000 7f6361000640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c141ed0 0x7f635c142360 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e50 tx=0x7f635402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb9622320d04feb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:46.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.802+0000 7f635aa00640 1 -- 172.21.15.67:0/831408185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635402cb00 con 0x7f635c141ed0 2024-04-17T07:29:46.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.802+0000 7f635aa00640 1 -- 172.21.15.67:0/831408185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635402cca0 con 0x7f635c141ed0 2024-04-17T07:29:46.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.802+0000 7f635aa00640 1 -- 172.21.15.67:0/831408185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354034610 con 0x7f635c141ed0 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 -- 172.21.15.67:0/831408185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c141ed0 msgr2=0x7f635c142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c141ed0 0x7f635c142360 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e50 tx=0x7f635402bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 -- 172.21.15.67:0/831408185 shutdown_connections 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c141ed0 0x7f635c142360 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 --2- 172.21.15.67:0/831408185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c160280 0x7f635c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 -- 172.21.15.67:0/831408185 >> 172.21.15.67:0/831408185 conn(0x7f635c067d10 msgr2=0x7f635c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 -- 172.21.15.67:0/831408185 shutdown_connections 2024-04-17T07:29:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.803+0000 7f6361a00640 1 -- 172.21.15.67:0/831408185 wait complete. 2024-04-17T07:29:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.805+0000 7f6361a00640 1 Processor -- start 2024-04-17T07:29:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.805+0000 7f6361a00640 1 -- start start 2024-04-17T07:29:46.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 0x7f635c133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f635c141ed0 0x7f635c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f635c161090 con 0x7f635c160280 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f635c06c1e0 con 0x7f635c141ed0 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f635c13f9b0 con 0x7f635c06b8c0 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58012/0 (socket says 172.21.15.67:58012) 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.806+0000 7f6361000640 1 -- 172.21.15.67:0/473476174 learned_addr learned my addr 172.21.15.67:0/473476174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f635b400640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f635c141ed0 0x7f635c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f635be00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 0x7f635c133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 msgr2=0x7f635c133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 0x7f635c133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f635c141ed0 msgr2=0x7f635c155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f635c141ed0 0x7f635c155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 -- 172.21.15.67:0/473476174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c06e250 con 0x7f635c160280 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f635be00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 0x7f635c133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6361000640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6354036040 tx=0x7f6354004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.807+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354043020 con 0x7f635c160280 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.808+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635402ce50 con 0x7f635c160280 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.808+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635403e670 con 0x7f635c160280 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.808+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635c06e530 con 0x7f635c160280 2024-04-17T07:29:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.808+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f635c06eaf0 con 0x7f635c160280 2024-04-17T07:29:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.809+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f635c068a10 con 0x7f635c160280 2024-04-17T07:29:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.810+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f6354046020 con 0x7f635c160280 2024-04-17T07:29:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.811+0000 7f6353e00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 0x7f633c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.811+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f63540ae690 con 0x7f635c160280 2024-04-17T07:29:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.811+0000 7f635be00640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 msgr2=0x7f633c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.811+0000 7f635be00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 0x7f633c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:46.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:46.815+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f635404d030 con 0x7f635c160280 2024-04-17T07:29:47.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.012+0000 7f635be00640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 msgr2=0x7f633c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:47.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.012+0000 7f635be00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 0x7f633c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.059+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f635c0cdb80 con 0x7f635c160280 2024-04-17T07:29:47.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.060+0000 7f6353e00640 1 -- 172.21.15.67:0/473476174 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7f635407c7b0 con 0x7f635c160280 2024-04-17T07:29:47.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:47.061 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:47.061 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:47.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 msgr2=0x7f633c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:47.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 0x7f633c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 msgr2=0x7f635c1456d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6354036040 tx=0x7f6354004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 shutdown_connections 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f633c06ca10 0x7f633c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f635c160280 0x7f635c1456d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f635c141ed0 0x7f635c155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 --2- 172.21.15.67:0/473476174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f635c06b8c0 0x7f635c133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 >> 172.21.15.67:0/473476174 conn(0x7f635c067d10 msgr2=0x7f635c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 shutdown_connections 2024-04-17T07:29:47.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.063+0000 7f6361a00640 1 -- 172.21.15.67:0/473476174 wait complete. 2024-04-17T07:29:47.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:47.284 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.283+0000 7fa306242040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.284+0000 7f2dac000640 1 Processor -- start 2024-04-17T07:29:47.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.284+0000 7f2dac000640 1 -- start start 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2dac000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2dac000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2da406f030 0x7f2da40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2dac000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da406ff90 con 0x7f2da40cd020 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2dac000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2da40ceb00 con 0x7f2da406f030 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2dac000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2da40cede0 con 0x7f2da406f670 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2daa000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2da406f030 0x7f2da40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2daa000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2da406f030 0x7f2da40cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58028/0 (socket says 172.21.15.67:58028) 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2daa000640 1 -- 172.21.15.67:0/1781720371 learned_addr learned my addr 172.21.15.67:0/1781720371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.286+0000 7f2da9600640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.287+0000 7f2daa000640 1 -- 172.21.15.67:0/1781720371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 msgr2=0x7f2da406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.287+0000 7f2daa000640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.287+0000 7f2daa000640 1 -- 172.21.15.67:0/1781720371 >> v1:172.21.15.187:6789/0 conn(0x7f2da40cd020 legacy=0x7f2da4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.287+0000 7f2daa000640 1 -- 172.21.15.67:0/1781720371 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da40cf0c0 con 0x7f2da406f030 2024-04-17T07:29:47.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.288+0000 7f2daa000640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da40cd480 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009e50 tx=0x7f2d9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccfca18487e77fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:47.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.288+0000 7f2da3e00640 1 -- 172.21.15.67:0/1781720371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c02ca70 con 0x7f2da406f030 2024-04-17T07:29:47.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.289+0000 7f2da3e00640 1 -- 172.21.15.67:0/1781720371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9c02cc10 con 0x7f2da406f030 2024-04-17T07:29:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.289+0000 7f2da3e00640 1 -- 172.21.15.67:0/1781720371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c0343f0 con 0x7f2da406f030 2024-04-17T07:29:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.289+0000 7f2dac000640 1 -- 172.21.15.67:0/1781720371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 msgr2=0x7f2da40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.289+0000 7f2dac000640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da40cd480 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009e50 tx=0x7f2d9c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 -- 172.21.15.67:0/1781720371 shutdown_connections 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da40cd480 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 --2- 172.21.15.67:0/1781720371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 -- 172.21.15.67:0/1781720371 >> 172.21.15.67:0/1781720371 conn(0x7f2da406bf50 msgr2=0x7f2da40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 -- 172.21.15.67:0/1781720371 shutdown_connections 2024-04-17T07:29:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.290+0000 7f2dac000640 1 -- 172.21.15.67:0/1781720371 wait complete. 2024-04-17T07:29:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.291+0000 7f2dac000640 1 Processor -- start 2024-04-17T07:29:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.292+0000 7f2dac000640 1 -- start start 2024-04-17T07:29:47.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.292+0000 7f2dac000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2dac000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da414cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2dac000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2da40cd020 0x7f2da41568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2dac000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40ce720 con 0x7f2da406f030 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2dac000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2da40cf680 con 0x7f2da40cd020 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2dac000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2da40ce8a0 con 0x7f2da406f670 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58042/0 (socket says 172.21.15.67:58042) 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 -- 172.21.15.67:0/1458679552 learned_addr learned my addr 172.21.15.67:0/1458679552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da8c00640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da414cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 msgr2=0x7f2da414cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2daa000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2da40cd020 0x7f2da41568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da414cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2da40cd020 msgr2=0x7f2da41568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2da40cd020 0x7f2da41568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da9600640 1 -- 172.21.15.67:0/1458679552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da4156630 con 0x7f2da406f030 2024-04-17T07:29:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.293+0000 7f2da8c00640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da414cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2da9600640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800d800 tx=0x7f2d9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800b6c0 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98019070 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da4155050 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800ba80 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.294+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da4155610 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.295+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da4068b90 con 0x7f2da406f030 2024-04-17T07:29:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.297+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f2d9800bc20 con 0x7f2da406f030 2024-04-17T07:29:47.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.298+0000 7f2da1600640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 0x7f2d8406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.298+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f2d9808c380 con 0x7f2da406f030 2024-04-17T07:29:47.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.298+0000 7f2da8c00640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 msgr2=0x7f2d8406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:47.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.298+0000 7f2da8c00640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 0x7f2d8406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.302+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2d9805a4b0 con 0x7f2da406f030 2024-04-17T07:29:47.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.499+0000 7f2da8c00640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 msgr2=0x7f2d8406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:47.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.499+0000 7f2da8c00640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 0x7f2d8406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:29:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.528+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2da406ccd0 con 0x7f2da406f030 2024-04-17T07:29:47.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.531+0000 7f2da1600640 1 -- 172.21.15.67:0/1458679552 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+17123 (secure 0 0 0) 0x7f2d98059db0 con 0x7f2da406f030 2024-04-17T07:29:47.531 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:47.532 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:41.466143+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6832/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1567626846":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/831200072":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/3826805311":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/12378685":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1955748639":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1795500056":"2024-04-18T07:29:41.466074+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 msgr2=0x7f2d8406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 0x7f2d8406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 msgr2=0x7f2da41560f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800d800 tx=0x7f2d9800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 shutdown_connections 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2da40cd020 0x7f2da41568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2da406f670 0x7f2da414cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f2d8406ca10 0x7f2d8406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 --2- 172.21.15.67:0/1458679552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2da406f030 0x7f2da41560f0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 >> 172.21.15.67:0/1458679552 conn(0x7f2da406bf50 msgr2=0x7f2da40c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 shutdown_connections 2024-04-17T07:29:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.533+0000 7f2dac000640 1 -- 172.21.15.67:0/1458679552 wait complete. 2024-04-17T07:29:47.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:29:24.457001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:29:25.538749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:47.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:47.560 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:47.560+0000 7f7132a42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:29:47.589 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.589+0000 7fa306242040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:47.646 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.646+0000 7fa306242040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:47.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.753+0000 7f656e200640 1 Processor -- start 2024-04-17T07:29:47.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.753+0000 7f656e200640 1 -- start start 2024-04-17T07:29:47.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65680d2ff0 0x7f65680b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656e200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 0x7f65680cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656e200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65680b5110 con 0x7f65680b4c70 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f65680b53f0 con 0x7f65680d2ff0 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656e200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f65680b56d0 con 0x7f65681622b0 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65680d2ff0 0x7f65680b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65680d2ff0 0x7f65680b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58056/0 (socket says 172.21.15.67:58056) 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 -- 172.21.15.67:0/4221497042 learned_addr learned my addr 172.21.15.67:0/4221497042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656d800640 1 --2- 172.21.15.67:0/4221497042 >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 0x7f65680cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 -- 172.21.15.67:0/4221497042 >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 msgr2=0x7f65680cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 --2- 172.21.15.67:0/4221497042 >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 0x7f65680cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 -- 172.21.15.67:0/4221497042 >> v1:172.21.15.187:6789/0 conn(0x7f65680b4c70 legacy=0x7f6568161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656ce00640 1 -- 172.21.15.67:0/4221497042 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65680cf340 con 0x7f65680d2ff0 2024-04-17T07:29:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.755+0000 7f656d800640 1 --2- 172.21.15.67:0/4221497042 >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 0x7f65680cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f656ce00640 1 --2- 172.21.15.67:0/4221497042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680d2ff0 0x7f65680b4730 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f65680d5850 tx=0x7f655c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61eb32daa9042f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f6567400640 1 -- 172.21.15.67:0/4221497042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c035020 con 0x7f65680d2ff0 2024-04-17T07:29:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f6567400640 1 -- 172.21.15.67:0/4221497042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c031870 con 0x7f65680d2ff0 2024-04-17T07:29:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f6567400640 1 -- 172.21.15.67:0/4221497042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c031c40 con 0x7f65680d2ff0 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f656e200640 1 -- 172.21.15.67:0/4221497042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680d2ff0 msgr2=0x7f65680b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.757+0000 7f656e200640 1 --2- 172.21.15.67:0/4221497042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680d2ff0 0x7f65680b4730 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f65680d5850 tx=0x7f655c02b910 comp rx=0 tx=0).stop 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 -- 172.21.15.67:0/4221497042 shutdown_connections 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 --2- 172.21.15.67:0/4221497042 >> v2:172.21.15.187:3300/0 conn(0x7f65681622b0 0x7f65680cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 --2- 172.21.15.67:0/4221497042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680d2ff0 0x7f65680b4730 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 -- 172.21.15.67:0/4221497042 >> 172.21.15.67:0/4221497042 conn(0x7f656809da80 msgr2=0x7f656808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 -- 172.21.15.67:0/4221497042 shutdown_connections 2024-04-17T07:29:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.758+0000 7f656e200640 1 -- 172.21.15.67:0/4221497042 wait complete. 2024-04-17T07:29:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.759+0000 7f656e200640 1 Processor -- start 2024-04-17T07:29:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.759+0000 7f656e200640 1 -- start start 2024-04-17T07:29:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 0x7f6568156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65681622b0 0x7f6568156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65680d3550 con 0x7f65680b4c70 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f65680d33d0 con 0x7f65680d2ff0 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.760+0000 7f656e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f65680d36d0 con 0x7f65681622b0 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58062/0 (socket says 172.21.15.67:58062) 2024-04-17T07:29:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 -- 172.21.15.67:0/2039131275 learned_addr learned my addr 172.21.15.67:0/2039131275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f6567e00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 0x7f6568156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 -- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65681622b0 msgr2=0x7f6568156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656d800640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65681622b0 0x7f6568156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65681622b0 0x7f6568156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 -- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 msgr2=0x7f6568156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 0x7f6568156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 -- 172.21.15.67:0/2039131275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65680cc6c0 con 0x7f65680b4c70 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f6567e00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 0x7f6568156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.761+0000 7f656ce00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.762+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c0443f0 con 0x7f65680b4c70 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.762+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c044590 con 0x7f65680b4c70 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.762+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c044980 con 0x7f65680b4c70 2024-04-17T07:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.762+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65680cc9a0 con 0x7f65680b4c70 2024-04-17T07:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.762+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65680ccf60 con 0x7f65680b4c70 2024-04-17T07:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.763+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65680cd1a0 con 0x7f65680b4c70 2024-04-17T07:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.764+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 34) ==== 91143+0+0 (secure 0 0 0) 0x7f655c04b060 con 0x7f65680b4c70 2024-04-17T07:29:47.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.765+0000 7f6564c00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 0x7f654806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.766+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 6053+0+0 (secure 0 0 0) 0x7f655c0ae2d0 con 0x7f65680b4c70 2024-04-17T07:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.766+0000 7f6567e00640 1 -- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 msgr2=0x7f654806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2378630274 2024-04-17T07:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.766+0000 7f6567e00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 0x7f654806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:29:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.770+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f65680cd1a0 con 0x7f65680b4c70 2024-04-17T07:29:47.779 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.779+0000 7fa306242040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.794+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 35) ==== 90751+0+0 (secure 0 0 0) 0x7f655c038020 con 0x7f65680b4c70 2024-04-17T07:29:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.794+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 msgr2=0x7f654806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:29:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.794+0000 7f6564c00640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 0x7f654806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa27fe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa27fe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa27fe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa27fe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa27fe00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa283000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa283000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa283000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa283000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:29:47.870+0000 7fa283000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:29:47.896 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:47.896+0000 7f7132a42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:29:47.957 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:47.957+0000 7f7132a42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:29:47.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.979+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65680755d0 con 0x7f65680b4c70 2024-04-17T07:29:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.980+0000 7f6564c00640 1 -- 172.21.15.67:0/2039131275 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7f655c03f070 con 0x7f65680b4c70 2024-04-17T07:29:47.980 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.982+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 msgr2=0x7f65680cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.982+0000 7f656e200640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 shutdown_connections 2024-04-17T07:29:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65681622b0 0x7f6568156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65680d2ff0 0x7f6568156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.187:6832/2378630274,v1:172.21.15.187:6833/2378630274] conn(0x7f654806ca10 0x7f654806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 --2- 172.21.15.67:0/2039131275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65680b4c70 0x7f65680cdfa0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 >> 172.21.15.67:0/2039131275 conn(0x7f656809da80 msgr2=0x7f65680b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 shutdown_connections 2024-04-17T07:29:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:47.983+0000 7f656e200640 1 -- 172.21.15.67:0/2039131275 wait complete. 2024-04-17T07:29:48.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:29:48.098 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:29:48.098+0000 7f7132a42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:29:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.255+0000 7ffb74e00640 1 Processor -- start 2024-04-17T07:29:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.255+0000 7ffb74e00640 1 -- start start 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb74e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb7006bfd0 0x7ffb700b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb74e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb700b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb74e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7006df40 con 0x7ffb7015b6b0 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb74e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7006ed00 con 0x7ffb7006bfd0 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb74e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7006e870 con 0x7ffb7006f800 2024-04-17T07:29:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb6ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb7006bfd0 0x7ffb700b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb6ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb7006bfd0 0x7ffb700b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58078/0 (socket says 172.21.15.67:58078) 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.257+0000 7ffb6ea00640 1 -- 172.21.15.67:0/1273534134 learned_addr learned my addr 172.21.15.67:0/1273534134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.258+0000 7ffb6ea00640 1 -- 172.21.15.67:0/1273534134 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 msgr2=0x7ffb700b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.258+0000 7ffb6fe00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb700b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.258+0000 7ffb6ea00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb700b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.258+0000 7ffb6ea00640 1 -- 172.21.15.67:0/1273534134 >> v1:172.21.15.187:6789/0 conn(0x7ffb7015b6b0 legacy=0x7ffb7006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:48.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.258+0000 7ffb6ea00640 1 -- 172.21.15.67:0/1273534134 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7006e220 con 0x7ffb7006bfd0 2024-04-17T07:29:48.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.259+0000 7ffb6ea00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7006bfd0 0x7ffb700b3490 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009870 tx=0x7ffb5c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a1ec623ec510bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:48.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.259+0000 7ffb6e000640 1 -- 172.21.15.67:0/1273534134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0317c0 con 0x7ffb7006bfd0 2024-04-17T07:29:48.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.259+0000 7ffb6e000640 1 -- 172.21.15.67:0/1273534134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c037070 con 0x7ffb7006bfd0 2024-04-17T07:29:48.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.259+0000 7ffb6e000640 1 -- 172.21.15.67:0/1273534134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c031b80 con 0x7ffb7006bfd0 2024-04-17T07:29:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 -- 172.21.15.67:0/1273534134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7006bfd0 msgr2=0x7ffb700b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7006bfd0 0x7ffb700b3490 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c009870 tx=0x7ffb5c02b860 comp rx=0 tx=0).stop 2024-04-17T07:29:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 -- 172.21.15.67:0/1273534134 shutdown_connections 2024-04-17T07:29:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb700b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 --2- 172.21.15.67:0/1273534134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7006bfd0 0x7ffb700b3490 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 -- 172.21.15.67:0/1273534134 >> 172.21.15.67:0/1273534134 conn(0x7ffb700b2190 msgr2=0x7ffb700b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 -- 172.21.15.67:0/1273534134 shutdown_connections 2024-04-17T07:29:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.260+0000 7ffb74e00640 1 -- 172.21.15.67:0/1273534134 wait complete. 2024-04-17T07:29:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.262+0000 7ffb74e00640 1 Processor -- start 2024-04-17T07:29:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.262+0000 7ffb74e00640 1 -- start start 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb7006bfd0 0x7ffb7014c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb70144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb700afee0 con 0x7ffb7015b6b0 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffb700afd60 con 0x7ffb7006bfd0 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb74e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb700cfc10 con 0x7ffb7006f800 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58084/0 (socket says 172.21.15.67:58084) 2024-04-17T07:29:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb6fe00640 1 -- 172.21.15.67:0/148107901 learned_addr learned my addr 172.21.15.67:0/148107901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb6ea00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb70144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.264+0000 7ffb6f400640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb7006bfd0 0x7ffb7014c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 -- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 msgr2=0x7ffb70144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb70144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 -- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb7006bfd0 msgr2=0x7ffb7014c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb7006bfd0 0x7ffb7014c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 -- 172.21.15.67:0/148107901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb70143d60 con 0x7ffb7015b6b0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb6fe00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800cd50 tx=0x7ffb6800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68003b90 con 0x7ffb7015b6b0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb68003d30 con 0x7ffb7015b6b0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68004240 con 0x7ffb7015b6b0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7014d870 con 0x7ffb7015b6b0 2024-04-17T07:29:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.265+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb70144010 con 0x7ffb7015b6b0 2024-04-17T07:29:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.267+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb70068a10 con 0x7ffb7015b6b0 2024-04-17T07:29:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.268+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 35) ==== 90751+0+0 (secure 0 0 0) 0x7ffb680043e0 con 0x7ffb7015b6b0 2024-04-17T07:29:48.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.269+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7ffb6808c310 con 0x7ffb7015b6b0 2024-04-17T07:29:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.273+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ffb6805a8e0 con 0x7ffb7015b6b0 2024-04-17T07:29:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.474+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb7006bd80 con 0x7ffb7015b6b0 2024-04-17T07:29:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.474+0000 7ffb67400640 1 -- 172.21.15.67:0/148107901 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) ==== 104+0+11 (secure 0 0 0) 0x7ffb6805a1e0 con 0x7ffb7015b6b0 2024-04-17T07:29:48.475 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 msgr2=0x7ffb700b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800cd50 tx=0x7ffb6800a660 comp rx=0 tx=0).stop 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 shutdown_connections 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb7015b6b0 0x7ffb700b13c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb7006f800 0x7ffb70144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 --2- 172.21.15.67:0/148107901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb7006bfd0 0x7ffb7014c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 >> 172.21.15.67:0/148107901 conn(0x7ffb700b2190 msgr2=0x7ffb700cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 shutdown_connections 2024-04-17T07:29:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.477+0000 7ffb74e00640 1 -- 172.21.15.67:0/148107901 wait complete. 2024-04-17T07:29:48.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:29:48.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.717+0000 7f3865800640 1 Processor -- start 2024-04-17T07:29:48.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.718+0000 7f3865800640 1 -- start start 2024-04-17T07:29:48.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.719+0000 7f3865800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38600cd1d0 0x7f3860160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.719+0000 7f3865800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38600700c0 0x7f38600cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f3865800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38600677a0 con 0x7f386009ec30 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f3865800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3860160b10 con 0x7f38600cd1d0 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f3865800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3860067a80 con 0x7f38600700c0 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38600cd1d0 0x7f3860160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38600cd1d0 0x7f3860160fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58098/0 (socket says 172.21.15.67:58098) 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 -- 172.21.15.67:0/3093483559 learned_addr learned my addr 172.21.15.67:0/3093483559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385e000640 1 --2- 172.21.15.67:0/3093483559 >> v2:172.21.15.187:3300/0 conn(0x7f38600700c0 0x7f38600cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 -- 172.21.15.67:0/3093483559 >> v2:172.21.15.187:3300/0 conn(0x7f38600700c0 msgr2=0x7f38600cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:48.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 --2- 172.21.15.67:0/3093483559 >> v2:172.21.15.187:3300/0 conn(0x7f38600700c0 0x7f38600cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 -- 172.21.15.67:0/3093483559 >> v1:172.21.15.67:6789/0 conn(0x7f386009ec30 legacy=0x7f386006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.720+0000 7f385ea00640 1 -- 172.21.15.67:0/3093483559 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38601607a0 con 0x7f38600cd1d0 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f385ea00640 1 --2- 172.21.15.67:0/3093483559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f3860160fa0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f386006d9c0 tx=0x7f385002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51412d9fd76be21 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f385d600640 1 -- 172.21.15.67:0/3093483559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850035020 con 0x7f38600cd1d0 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f385d600640 1 -- 172.21.15.67:0/3093483559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3850031870 con 0x7f38600cd1d0 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f385d600640 1 -- 172.21.15.67:0/3093483559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850031c40 con 0x7f38600cd1d0 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f3865800640 1 -- 172.21.15.67:0/3093483559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 msgr2=0x7f3860160fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.722+0000 7f3865800640 1 --2- 172.21.15.67:0/3093483559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f3860160fa0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f386006d9c0 tx=0x7f385002b910 comp rx=0 tx=0).stop 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 -- 172.21.15.67:0/3093483559 shutdown_connections 2024-04-17T07:29:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 --2- 172.21.15.67:0/3093483559 >> v2:172.21.15.187:3300/0 conn(0x7f38600700c0 0x7f38600cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 --2- 172.21.15.67:0/3093483559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f3860160fa0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 -- 172.21.15.67:0/3093483559 >> 172.21.15.67:0/3093483559 conn(0x7f38600cd850 msgr2=0x7f38600cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 -- 172.21.15.67:0/3093483559 shutdown_connections 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.723+0000 7f3865800640 1 -- 172.21.15.67:0/3093483559 wait complete. 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.724+0000 7f3865800640 1 Processor -- start 2024-04-17T07:29:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 -- start start 2024-04-17T07:29:48.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 0x7f3860146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f386009ec30 0x7f38600b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38600ce4f0 con 0x7f38600cd1d0 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38600ce370 con 0x7f38600700c0 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.725+0000 7f3865800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38601614e0 con 0x7f386009ec30 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58102/0 (socket says 172.21.15.67:58102) 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 -- 172.21.15.67:0/4249922815 learned_addr learned my addr 172.21.15.67:0/4249922815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385ea00640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 0x7f3860146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385e000640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f386009ec30 0x7f38600b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 -- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f386009ec30 msgr2=0x7f38600b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f386009ec30 0x7f38600b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 -- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 msgr2=0x7f3860146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 0x7f3860146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385f400640 1 -- 172.21.15.67:0/4249922815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3860146d80 con 0x7f38600cd1d0 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.726+0000 7f385ea00640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 0x7f3860146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f385f400640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f385800c730 tx=0x7f385800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858003b90 con 0x7f38600cd1d0 2024-04-17T07:29:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3858003d30 con 0x7f38600cd1d0 2024-04-17T07:29:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385800aa90 con 0x7f38600cd1d0 2024-04-17T07:29:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3860147100 con 0x7f38600cd1d0 2024-04-17T07:29:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.727+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38601476c0 con 0x7f38600cd1d0 2024-04-17T07:29:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.728+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3860068b90 con 0x7f38600cd1d0 2024-04-17T07:29:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.729+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 35) ==== 90751+0+0 (secure 0 0 0) 0x7f385800ac30 con 0x7f38600cd1d0 2024-04-17T07:29:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.730+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f385808b7a0 con 0x7f38600cd1d0 2024-04-17T07:29:48.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.735+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3860146d80 con 0x7f38600cd1d0 2024-04-17T07:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.809+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 36) ==== 90878+0+0 (secure 0 0 0) 0x7f38580553c0 con 0x7f38600cd1d0 2024-04-17T07:29:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.810+0000 7f3856a00640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 0x7f383c074400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:48.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.811+0000 7f385ea00640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 0x7f383c074400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:48.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.825+0000 7f385ea00640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 0x7f383c074400 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3850036000 tx=0x7f385003c040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:48.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.948+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f38600737d0 con 0x7f38600cd1d0 2024-04-17T07:29:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.949+0000 7f3856a00640 1 -- 172.21.15.67:0/4249922815 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) ==== 100+0+11 (secure 0 0 0) 0x7f3858059d70 con 0x7f38600cd1d0 2024-04-17T07:29:48.949 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:29:48.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 msgr2=0x7f383c074400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 0x7f383c074400 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3850036000 tx=0x7f385003c040 comp rx=0 tx=0).stop 2024-04-17T07:29:48.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 msgr2=0x7f38600b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:48.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f385800c730 tx=0x7f385800cbf0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 shutdown_connections 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38600cd1d0 0x7f38600b24c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f386009ec30 0x7f38600b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f383c072010 0x7f383c074400 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 --2- 172.21.15.67:0/4249922815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38600700c0 0x7f3860146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 >> 172.21.15.67:0/4249922815 conn(0x7f38600cd850 msgr2=0x7f386015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 shutdown_connections 2024-04-17T07:29:48.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:48.951+0000 7f3865800640 1 -- 172.21.15.67:0/4249922815 wait complete. 2024-04-17T07:29:48.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.193+0000 7f1536800640 1 Processor -- start 2024-04-17T07:29:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.194+0000 7f1536800640 1 -- start start 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.195+0000 7f1536800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1530161e40 0x7f15300ba3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.195+0000 7f1536800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1530162220 0x7f15301626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.195+0000 7f1536800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1530162e60 con 0x7f15300cf650 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.195+0000 7f1536800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1530162c30 con 0x7f1530161e40 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.195+0000 7f1536800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1530163140 con 0x7f1530162220 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1530161e40 0x7f15300ba3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1530161e40 0x7f15300ba3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58116/0 (socket says 172.21.15.67:58116) 2024-04-17T07:29:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 -- 172.21.15.67:0/57018986 learned_addr learned my addr 172.21.15.67:0/57018986 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 -- 172.21.15.67:0/57018986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1530162220 msgr2=0x7f15301626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 --2- 172.21.15.67:0/57018986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1530162220 0x7f15301626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 -- 172.21.15.67:0/57018986 >> v1:172.21.15.67:6789/0 conn(0x7f15300cf650 legacy=0x7f15300b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.196+0000 7f1535400640 1 -- 172.21.15.67:0/57018986 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1530163420 con 0x7f1530161e40 2024-04-17T07:29:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.198+0000 7f1535400640 1 --2- 172.21.15.67:0/57018986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530161e40 0x7f15300ba3d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1524009870 tx=0x7f152402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b703e4dbe863b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.198+0000 7f152fe00640 1 -- 172.21.15.67:0/57018986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524034070 con 0x7f1530161e40 2024-04-17T07:29:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.198+0000 7f152fe00640 1 -- 172.21.15.67:0/57018986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524030840 con 0x7f1530161e40 2024-04-17T07:29:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.198+0000 7f152fe00640 1 -- 172.21.15.67:0/57018986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524030c10 con 0x7f1530161e40 2024-04-17T07:29:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 -- 172.21.15.67:0/57018986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530161e40 msgr2=0x7f15300ba3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 --2- 172.21.15.67:0/57018986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530161e40 0x7f15300ba3d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1524009870 tx=0x7f152402b980 comp rx=0 tx=0).stop 2024-04-17T07:29:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 -- 172.21.15.67:0/57018986 shutdown_connections 2024-04-17T07:29:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 --2- 172.21.15.67:0/57018986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1530162220 0x7f15301626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 --2- 172.21.15.67:0/57018986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530161e40 0x7f15300ba3d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 -- 172.21.15.67:0/57018986 >> 172.21.15.67:0/57018986 conn(0x7f153006cbf0 msgr2=0x7f153006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 -- 172.21.15.67:0/57018986 shutdown_connections 2024-04-17T07:29:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.199+0000 7f1536800640 1 -- 172.21.15.67:0/57018986 wait complete. 2024-04-17T07:29:49.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.201+0000 7f1536800640 1 Processor -- start 2024-04-17T07:29:49.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.201+0000 7f1536800640 1 -- start start 2024-04-17T07:29:49.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15300cf650 0x7f15300cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1530161e40 0x7f15301f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15300d44f0 con 0x7f1530162220 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15300d4370 con 0x7f1530161e40 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1536800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15300d4670 con 0x7f15300cf650 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1535e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1535e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58118/0 (socket says 172.21.15.67:58118) 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.202+0000 7f1535e00640 1 -- 172.21.15.67:0/1523187780 learned_addr learned my addr 172.21.15.67:0/1523187780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1534a00640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1530161e40 0x7f15301f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535400640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15300cf650 0x7f15300cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 -- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15300cf650 msgr2=0x7f15300cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15300cf650 0x7f15300cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 -- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1530161e40 msgr2=0x7f15301f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1530161e40 0x7f15301f7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 -- 172.21.15.67:0/1523187780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153015a040 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1535e00640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f152800c730 tx=0x7f152800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528003b70 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1528003d30 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528005310 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.203+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153015a320 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.204+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153015a8e0 con 0x7f1530162220 2024-04-17T07:29:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.205+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15301626f0 con 0x7f1530162220 2024-04-17T07:29:49.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.206+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 36) ==== 90878+0+0 (secure 0 0 0) 0x7f15280056c0 con 0x7f1530162220 2024-04-17T07:29:49.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.207+0000 7f152d600640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 0x7f151006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.207+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f152808bb60 con 0x7f1530162220 2024-04-17T07:29:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.207+0000 7f1535400640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 0x7f151006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.209+0000 7f1535400640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 0x7f151006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1524036000 tx=0x7f152403b040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.211+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f15301626f0 con 0x7f1530162220 2024-04-17T07:29:49.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.456+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1530068a10 con 0x7f1530162220 2024-04-17T07:29:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.458+0000 7f152d600640 1 -- 172.21.15.67:0/1523187780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7f1528059f50 con 0x7f1530162220 2024-04-17T07:29:49.458 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:49.458 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.460+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 msgr2=0x7f151006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.460+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 0x7f151006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1524036000 tx=0x7f152403b040 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.460+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 msgr2=0x7f15301f7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.460+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f152800c730 tx=0x7f152800cbf0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 shutdown_connections 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530162220 0x7f15301f7980 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f151006c740 0x7f151006ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1530161e40 0x7f15301f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 --2- 172.21.15.67:0/1523187780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15300cf650 0x7f15300cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 >> 172.21.15.67:0/1523187780 conn(0x7f153006cbf0 msgr2=0x7f153006cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 shutdown_connections 2024-04-17T07:29:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.461+0000 7f1536800640 1 -- 172.21.15.67:0/1523187780 wait complete. 2024-04-17T07:29:49.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:49.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.704+0000 7f6cd5a00640 1 Processor -- start 2024-04-17T07:29:49.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.705+0000 7f6cd5a00640 1 -- start start 2024-04-17T07:29:49.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.706+0000 7f6cd5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 0x7f6cd006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.706+0000 7f6cd5a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6cd00c68c0 0x7f6cd006e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.706+0000 7f6cd5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd015f3c0 con 0x7f6cd015b530 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.706+0000 7f6cd5a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd015f660 con 0x7f6cd00c68c0 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.706+0000 7f6cd5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd015f940 con 0x7f6cd0072420 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6ccea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6cd015b530 0x7f6cd0071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57202/0 (socket says 172.21.15.67:57202) 2024-04-17T07:29:49.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6ccea00640 1 -- 172.21.15.67:0/2614631756 learned_addr learned my addr 172.21.15.67:0/2614631756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6cc7e00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 0x7f6cd006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6ccf400640 1 --2- 172.21.15.67:0/2614631756 >> v2:172.21.15.187:3300/0 conn(0x7f6cd00c68c0 0x7f6cd006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1534669332 0 0) 0x7f6cd015f3c0 con 0x7f6cd015b530 2024-04-17T07:29:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.707+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb4003520 con 0x7f6cd015b530 2024-04-17T07:29:49.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 457417103 0 0) 0x7f6cb4003520 con 0x7f6cd015b530 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 msgr2=0x7f6cd006dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 0x7f6cd006dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 >> v2:172.21.15.187:3300/0 conn(0x7f6cd00c68c0 msgr2=0x7f6cd006e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 --2- 172.21.15.67:0/2614631756 >> v2:172.21.15.187:3300/0 conn(0x7f6cd00c68c0 0x7f6cd006e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd015fc20 con 0x7f6cd015b530 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6cc0001690 con 0x7f6cd015b530 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 >> v1:172.21.15.67:6789/0 conn(0x7f6cd015b530 legacy=0x7f6cd0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cb4004800 0x7f6cb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 0x7f6cb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd015f660 con 0x7f6cb40039b0 2024-04-17T07:29:49.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd015f940 con 0x7f6cb4004800 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb400ba70 con 0x7f6cd0072420 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccf400640 1 --2- 172.21.15.67:0/2614631756 >> v2:172.21.15.187:3300/0 conn(0x7f6cd00c68c0 0x7f6cd006e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccea00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccea00640 1 -- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 msgr2=0x7f6cb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccea00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd0072420 0x7f6cb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccea00640 1 -- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cb4004800 msgr2=0x7f6cb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.708+0000 7f6ccea00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cb4004800 0x7f6cb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.709+0000 7f6ccea00640 1 -- 172.21.15.67:0/2614631756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb4003520 con 0x7f6cb40039b0 2024-04-17T07:29:49.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6ccea00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cb4003dd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6cd015f590 tx=0x7f6cb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ffe4d36c111d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0007d60 con 0x7f6cb40039b0 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc0030070 con 0x7f6cb40039b0 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6cce000640 1 -- 172.21.15.67:0/2614631756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0005d10 con 0x7f6cb40039b0 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6cd5a00640 1 -- 172.21.15.67:0/2614631756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 msgr2=0x7f6cb4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/2614631756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cb4003dd0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6cd015f590 tx=0x7f6cb4005830 comp rx=0 tx=0).stop 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.710+0000 7f6ccf400640 1 -- 172.21.15.67:0/2614631756 reap_dead start 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.711+0000 7f6cd5a00640 1 -- 172.21.15.67:0/2614631756 shutdown_connections 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.711+0000 7f6cd5a00640 1 -- 172.21.15.67:0/2614631756 >> 172.21.15.67:0/2614631756 conn(0x7f6cd006bf50 msgr2=0x7f6cd00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.711+0000 7f6cd5a00640 1 -- 172.21.15.67:0/2614631756 shutdown_connections 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.711+0000 7f6cd5a00640 1 -- 172.21.15.67:0/2614631756 wait complete. 2024-04-17T07:29:49.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.712+0000 7f6cd5a00640 1 Processor -- start 2024-04-17T07:29:49.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.713+0000 7f6cd5a00640 1 -- start start 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.713+0000 7f6cd5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cd5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd00b2170 0x7f6cd0149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cd5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cd00c9a20 0x7f6cd00c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cd5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0160140 con 0x7f6cb40039b0 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cd5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd015b910 con 0x7f6cd00c9a20 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cd5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd00ca3e0 con 0x7f6cd00b2170 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58138/0 (socket says 172.21.15.67:58138) 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 -- 172.21.15.67:0/1938178849 learned_addr learned my addr 172.21.15.67:0/1938178849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 -- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd00b2170 msgr2=0x7f6cd0149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6cc7e00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd00b2170 0x7f6cd0149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccf400640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cd00c9a20 0x7f6cd00c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd00b2170 0x7f6cd0149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 -- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cd00c9a20 msgr2=0x7f6cd00c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cd00c9a20 0x7f6cd00c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.714+0000 7f6ccea00640 1 -- 172.21.15.67:0/1938178849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd00ce360 con 0x7f6cb40039b0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6ccea00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0001990 tx=0x7f6cd015fdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0001ca0 con 0x7f6cb40039b0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc0004bf0 con 0x7f6cb40039b0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0007070 con 0x7f6cb40039b0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd00b25f0 con 0x7f6cb40039b0 2024-04-17T07:29:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.715+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01f8cb0 con 0x7f6cb40039b0 2024-04-17T07:29:49.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.716+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd0068b90 con 0x7f6cb40039b0 2024-04-17T07:29:49.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.717+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 36) ==== 90878+0+0 (secure 0 0 0) 0x7f6cc003f070 con 0x7f6cb40039b0 2024-04-17T07:29:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.719+0000 7f6cc6a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 0x7f6cac06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.719+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f6cc00ace80 con 0x7f6cb40039b0 2024-04-17T07:29:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.719+0000 7f6cc7e00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 0x7f6cac06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.721+0000 7f6cc7e00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 0x7f6cac06eb70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8006a90 tx=0x7f6cb8008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:49.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.723+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6cd00ce360 con 0x7f6cb40039b0 2024-04-17T07:29:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.823+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f6cc00b1070 con 0x7f6cb40039b0 2024-04-17T07:29:49.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.939+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cd00c6390 con 0x7f6cb40039b0 2024-04-17T07:29:49.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.942+0000 7f6cc6a00640 1 -- 172.21.15.67:0/1938178849 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+17631 (secure 0 0 0) 0x7f6cc003f360 con 0x7f6cb40039b0 2024-04-17T07:29:49.942 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:49.943 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:47.783926+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/939506877":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6832/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/139946814":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/2814172476":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/3268338662":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1164934579":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6832/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1567626846":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/831200072":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/3826805311":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/12378685":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1955748639":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1146224552":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1795500056":"2024-04-18T07:29:41.466074+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 msgr2=0x7f6cac06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 0x7f6cac06eb70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8006a90 tx=0x7f6cb8008040 comp rx=0 tx=0).stop 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 msgr2=0x7f6cd01f82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0001990 tx=0x7f6cd015fdf0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 shutdown_connections 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cac06c6b0 0x7f6cac06eb70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cd00c9a20 0x7f6cd00c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cd00b2170 0x7f6cd0149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 --2- 172.21.15.67:0/1938178849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40039b0 0x7f6cd01f82d0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 >> 172.21.15.67:0/1938178849 conn(0x7f6cd006bf50 msgr2=0x7f6cd0159610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 shutdown_connections 2024-04-17T07:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:49.945+0000 7f6cd5a00640 1 -- 172.21.15.67:0/1938178849 wait complete. 2024-04-17T07:29:49.965 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:29:49.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.185+0000 7f3482400640 1 Processor -- start 2024-04-17T07:29:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.186+0000 7f3482400640 1 -- start start 2024-04-17T07:29:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.187+0000 7f3482400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f347c15d430 0x7f347c0b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.187+0000 7f3482400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f347c15d810 0x7f347c0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.187+0000 7f3482400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347c0b1430 con 0x7f347c0b29b0 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.187+0000 7f3482400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f347c0b2df0 con 0x7f347c15d810 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.187+0000 7f3482400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f347c0b1710 con 0x7f347c15d430 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f347c15d810 0x7f347c0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f347c15d810 0x7f347c0b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58150/0 (socket says 172.21.15.67:58150) 2024-04-17T07:29:50.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 -- 172.21.15.67:0/3076408303 learned_addr learned my addr 172.21.15.67:0/3076408303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347b400640 1 --2- 172.21.15.67:0/3076408303 >> v2:172.21.15.187:3300/0 conn(0x7f347c15d430 0x7f347c0b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 -- 172.21.15.67:0/3076408303 >> v2:172.21.15.187:3300/0 conn(0x7f347c15d430 msgr2=0x7f347c0b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 --2- 172.21.15.67:0/3076408303 >> v2:172.21.15.187:3300/0 conn(0x7f347c15d430 0x7f347c0b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 -- 172.21.15.67:0/3076408303 >> v1:172.21.15.187:6789/0 conn(0x7f347c0b29b0 legacy=0x7f347c0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.188+0000 7f347aa00640 1 -- 172.21.15.67:0/3076408303 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c0b19f0 con 0x7f347c15d810 2024-04-17T07:29:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.189+0000 7f347aa00640 1 --2- 172.21.15.67:0/3076408303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c0b2470 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f347c06c510 tx=0x7f346402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce865ac87356b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.189+0000 7f347a000640 1 -- 172.21.15.67:0/3076408303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464035020 con 0x7f347c15d810 2024-04-17T07:29:50.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.189+0000 7f347a000640 1 -- 172.21.15.67:0/3076408303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3464031870 con 0x7f347c15d810 2024-04-17T07:29:50.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.189+0000 7f347a000640 1 -- 172.21.15.67:0/3076408303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464031c40 con 0x7f347c15d810 2024-04-17T07:29:50.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 -- 172.21.15.67:0/3076408303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 msgr2=0x7f347c0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 --2- 172.21.15.67:0/3076408303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c0b2470 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f347c06c510 tx=0x7f346402b910 comp rx=0 tx=0).stop 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 -- 172.21.15.67:0/3076408303 shutdown_connections 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 --2- 172.21.15.67:0/3076408303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c0b2470 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 --2- 172.21.15.67:0/3076408303 >> v2:172.21.15.187:3300/0 conn(0x7f347c15d430 0x7f347c0b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 -- 172.21.15.67:0/3076408303 >> 172.21.15.67:0/3076408303 conn(0x7f347c073360 msgr2=0x7f347c09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 -- 172.21.15.67:0/3076408303 shutdown_connections 2024-04-17T07:29:50.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.190+0000 7f3482400640 1 -- 172.21.15.67:0/3076408303 wait complete. 2024-04-17T07:29:50.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.192+0000 7f3482400640 1 Processor -- start 2024-04-17T07:29:50.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.192+0000 7f3482400640 1 -- start start 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f347c0b29b0 0x7f347c0c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c15d430 0x7f347c1f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f347c15dca0 con 0x7f347c15d810 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f347c0b1fb0 con 0x7f347c0b29b0 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f3482400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f347c0ce3a0 con 0x7f347c15d430 2024-04-17T07:29:50.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.193+0000 7f347be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58160/0 (socket says 172.21.15.67:58160) 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 -- 172.21.15.67:0/3083222126 learned_addr learned my addr 172.21.15.67:0/3083222126 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347aa00640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c15d430 0x7f347c1f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347b400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f347c0b29b0 0x7f347c0c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 -- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c15d430 msgr2=0x7f347c1f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c15d430 0x7f347c1f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 -- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f347c0b29b0 msgr2=0x7f347c0c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f347c0b29b0 0x7f347c0c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 -- 172.21.15.67:0/3083222126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c0cf0f0 con 0x7f347c15d810 2024-04-17T07:29:50.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.194+0000 7f347be00640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f347400c390 tx=0x7f347400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.195+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474017070 con 0x7f347c15d810 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.195+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34740035f0 con 0x7f347c15d810 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.195+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34740039e0 con 0x7f347c15d810 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.195+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c0cf3d0 con 0x7f347c15d810 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.195+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c0c73a0 con 0x7f347c15d810 2024-04-17T07:29:50.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.196+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f347c068b90 con 0x7f347c15d810 2024-04-17T07:29:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.197+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f347401a070 con 0x7f347c15d810 2024-04-17T07:29:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.198+0000 7f3473400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 0x7f345806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.198+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f347408b330 con 0x7f347c15d810 2024-04-17T07:29:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.198+0000 7f347b400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 0x7f345806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.200+0000 7f347b400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 0x7f345806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f347c0c5a50 tx=0x7f346c008460 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.203+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34740593c0 con 0x7f347c15d810 2024-04-17T07:29:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.450+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f347c0cf870 con 0x7f347c15d810 2024-04-17T07:29:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.451+0000 7f3473400640 1 -- 172.21.15.67:0/3083222126 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7f3474058cc0 con 0x7f347c15d810 2024-04-17T07:29:50.451 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:50.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:50.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 msgr2=0x7f345806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 0x7f345806ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f347c0c5a50 tx=0x7f346c008460 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 msgr2=0x7f347c1f7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f347400c390 tx=0x7f347400c850 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 shutdown_connections 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c15d810 0x7f347c1f7200 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f347c15d430 0x7f347c1f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f345806c980 0x7f345806ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 --2- 172.21.15.67:0/3083222126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f347c0b29b0 0x7f347c0c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 >> 172.21.15.67:0/3083222126 conn(0x7f347c073360 msgr2=0x7f347c15aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 shutdown_connections 2024-04-17T07:29:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.454+0000 7f3482400640 1 -- 172.21.15.67:0/3083222126 wait complete. 2024-04-17T07:29:50.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:29:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.688+0000 7f972a800640 1 Processor -- start 2024-04-17T07:29:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.689+0000 7f972a800640 1 -- start start 2024-04-17T07:29:50.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.690+0000 7f972a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240b6d00 0x7f97240cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.690+0000 7f972a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97240cf910 0x7f97240b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.690+0000 7f972a800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97240d2d30 con 0x7f972415ec60 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.690+0000 7f972a800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f97240d49c0 con 0x7f97240cf910 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.690+0000 7f972a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97240d3660 con 0x7f97240b6d00 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.691+0000 7f9729e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97240cf910 0x7f97240b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.691+0000 7f9728a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240b6d00 0x7f97240cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.691+0000 7f9729e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f97240cf910 0x7f97240b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:33298/0 (socket says 172.21.15.67:33298) 2024-04-17T07:29:50.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.691+0000 7f9729e00640 1 -- 172.21.15.67:0/1639402136 learned_addr learned my addr 172.21.15.67:0/1639402136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.692+0000 7f9729e00640 1 -- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240b6d00 msgr2=0x7f97240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.692+0000 7f9729e00640 1 --2- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240b6d00 0x7f97240cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.692+0000 7f9729e00640 1 -- 172.21.15.67:0/1639402136 >> v1:172.21.15.187:6789/0 conn(0x7f972415ec60 legacy=0x7f97240b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.692+0000 7f9729e00640 1 -- 172.21.15.67:0/1639402136 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97240d3010 con 0x7f97240cf910 2024-04-17T07:29:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.693+0000 7f9729e00640 1 --2- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97240cf910 0x7f97240b7b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f971c009e50 tx=0x7f971c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d5d141afe33fd5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.694+0000 7f9723e00640 1 -- 172.21.15.67:0/1639402136 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c02cb00 con 0x7f97240cf910 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.694+0000 7f9723e00640 1 -- 172.21.15.67:0/1639402136 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f971c02cca0 con 0x7f97240cf910 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.694+0000 7f9723e00640 1 -- 172.21.15.67:0/1639402136 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c034610 con 0x7f97240cf910 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 -- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97240cf910 msgr2=0x7f97240b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 --2- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97240cf910 0x7f97240b7b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f971c009e50 tx=0x7f971c02bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 -- 172.21.15.67:0/1639402136 shutdown_connections 2024-04-17T07:29:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 --2- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97240cf910 0x7f97240b7b70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 --2- 172.21.15.67:0/1639402136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240b6d00 0x7f97240cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 -- 172.21.15.67:0/1639402136 >> 172.21.15.67:0/1639402136 conn(0x7f97240d15f0 msgr2=0x7f97240d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 -- 172.21.15.67:0/1639402136 shutdown_connections 2024-04-17T07:29:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.695+0000 7f972a800640 1 -- 172.21.15.67:0/1639402136 wait complete. 2024-04-17T07:29:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.697+0000 7f972a800640 1 Processor -- start 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.697+0000 7f972a800640 1 -- start start 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 0x7f97241f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f972415ec60 0x7f9724157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f972415f040 con 0x7f97240b6d00 2024-04-17T07:29:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f97240cfdc0 con 0x7f972415ec60 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f972a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97240d00c0 con 0x7f97240cf910 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58174/0 (socket says 172.21.15.67:58174) 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 -- 172.21.15.67:0/1387003314 learned_addr learned my addr 172.21.15.67:0/1387003314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9728a00640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 0x7f97241f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729e00640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f972415ec60 0x7f9724157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 -- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 msgr2=0x7f97241f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 0x7f97241f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 -- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f972415ec60 msgr2=0x7f9724157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f972415ec60 0x7f9724157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9729400640 1 -- 172.21.15.67:0/1387003314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97241f5a10 con 0x7f97240b6d00 2024-04-17T07:29:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.698+0000 7f9728a00640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 0x7f97241f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f9729400640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f971800c420 tx=0x7f971800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97180052d0 con 0x7f97240b6d00 2024-04-17T07:29:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9718005490 con 0x7f97240b6d00 2024-04-17T07:29:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972415f2f0 con 0x7f97240b6d00 2024-04-17T07:29:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718007260 con 0x7f97240b6d00 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.699+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9724161040 con 0x7f97240b6d00 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.701+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9724068a10 con 0x7f97240b6d00 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.701+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f9718007400 con 0x7f97240b6d00 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.702+0000 7f9721600640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 0x7f970406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.702+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f971808b210 con 0x7f97240b6d00 2024-04-17T07:29:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.703+0000 7f9728a00640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 0x7f970406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.704+0000 7f9728a00640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 0x7f970406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9724147f30 tx=0x7f9710008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.707+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f97241f5a10 con 0x7f97240b6d00 2024-04-17T07:29:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.964+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9724161210 con 0x7f97240b6d00 2024-04-17T07:29:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.965+0000 7f9721600640 1 -- 172.21.15.67:0/1387003314 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5205 (secure 0 0 0) 0x7f9718059430 con 0x7f97240b6d00 2024-04-17T07:29:50.965 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:50.965 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:29:26.551509+0000","modified":"2024-04-17T07:29:29.712215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6444":{"gid":6444,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/2437593714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2437593714},{"type":"v1","addr":"172.21.15.187:6835","nonce":2437593714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6444,"qdb_cluster":[6444]},"id":4}]} 2024-04-17T07:29:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 msgr2=0x7f970406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 0x7f970406ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9724147f30 tx=0x7f9710008040 comp rx=0 tx=0).stop 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 msgr2=0x7f97240b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f971800c420 tx=0x7f971800c8e0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 shutdown_connections 2024-04-17T07:29:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f970406c9b0 0x7f970406ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f972415ec60 0x7f9724157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97240cf910 0x7f97241f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 --2- 172.21.15.67:0/1387003314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97240b6d00 0x7f97240b4540 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 >> 172.21.15.67:0/1387003314 conn(0x7f97240d15f0 msgr2=0x7f972415e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 shutdown_connections 2024-04-17T07:29:50.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:50.968+0000 7f972a800640 1 -- 172.21.15.67:0/1387003314 wait complete. 2024-04-17T07:29:50.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.216+0000 7f76bda00640 1 Processor -- start 2024-04-17T07:29:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.216+0000 7f76bda00640 1 -- start start 2024-04-17T07:29:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.217+0000 7f76bda00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f76b806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.217+0000 7f76bda00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76b8141ed0 0x7f76b8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.217+0000 7f76bda00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b80cd4a0 con 0x7f76b806b8c0 2024-04-17T07:29:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.217+0000 7f76bda00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f76b815ce30 con 0x7f76b8141ed0 2024-04-17T07:29:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.217+0000 7f76bda00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76b81607f0 con 0x7f76b8160280 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76b7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76b806b8c0 0x7f76b80cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57208/0 (socket says 172.21.15.67:57208) 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76b7e00640 1 -- 172.21.15.67:0/3822774138 learned_addr learned my addr 172.21.15.67:0/3822774138 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76b7400640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f76b806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76bd000640 1 --2- 172.21.15.67:0/3822774138 >> v2:172.21.15.187:3300/0 conn(0x7f76b8141ed0 0x7f76b8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 36342459 0 0) 0x7f76b80cd4a0 con 0x7f76b806b8c0 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.218+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f769c003520 con 0x7f76b806b8c0 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1364904165 0 0) 0x7f769c003520 con 0x7f76b806b8c0 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 msgr2=0x7f76b806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f76b806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 >> v2:172.21.15.187:3300/0 conn(0x7f76b8141ed0 msgr2=0x7f76b8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 --2- 172.21.15.67:0/3822774138 >> v2:172.21.15.187:3300/0 conn(0x7f76b8141ed0 0x7f76b8142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b8160ad0 con 0x7f76b806b8c0 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f76a8001670 con 0x7f76b806b8c0 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 >> v1:172.21.15.67:6789/0 conn(0x7f76b806b8c0 legacy=0x7f76b80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f769c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f769c003ff0 0x7f769c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c004d40 0x7f769c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76b815ce30 con 0x7f769c004d40 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f76b81607f0 con 0x7f769c003ff0 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f769c00ba40 con 0x7f76b8160280 2024-04-17T07:29:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.219+0000 7f76bd000640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c004d40 0x7f769c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76b7e00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f769c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76bd000640 1 -- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 msgr2=0x7f769c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76b7400640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f769c003ff0 0x7f769c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76bd000640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f769c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76bd000640 1 -- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f769c003ff0 msgr2=0x7f769c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76bd000640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f769c003ff0 0x7f769c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76bd000640 1 -- 172.21.15.67:0/3822774138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c003520 con 0x7f769c004d40 2024-04-17T07:29:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.220+0000 7f76b7e00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76b8160280 0x7f769c003a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.221+0000 7f76bd000640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c004d40 0x7f769c0051c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009ac0 tx=0x7f769c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8df3fdf8e3b961c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.221+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0039070 con 0x7f769c004d40 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.221+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b002c710 con 0x7f769c004d40 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.221+0000 7f76b6a00640 1 -- 172.21.15.67:0/3822774138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0009690 con 0x7f769c004d40 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 -- 172.21.15.67:0/3822774138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c004d40 msgr2=0x7f769c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 --2- 172.21.15.67:0/3822774138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769c004d40 0x7f769c0051c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009ac0 tx=0x7f769c005830 comp rx=0 tx=0).stop 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bd000640 1 -- 172.21.15.67:0/3822774138 reap_dead start 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 -- 172.21.15.67:0/3822774138 shutdown_connections 2024-04-17T07:29:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 -- 172.21.15.67:0/3822774138 >> 172.21.15.67:0/3822774138 conn(0x7f76b8067d10 msgr2=0x7f76b80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 -- 172.21.15.67:0/3822774138 shutdown_connections 2024-04-17T07:29:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.222+0000 7f76bda00640 1 -- 172.21.15.67:0/3822774138 wait complete. 2024-04-17T07:29:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.224+0000 7f76bda00640 1 Processor -- start 2024-04-17T07:29:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.224+0000 7f76bda00640 1 -- start start 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f769c004d40 0x7f76b815d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b80aec40 0x7f76b8145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76b813f9b0 con 0x7f76b80af0a0 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f76b8161090 con 0x7f76b80aec40 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bda00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76b813fb30 con 0x7f769c004d40 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58196/0 (socket says 172.21.15.67:58196) 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.225+0000 7f76bd000640 1 -- 172.21.15.67:0/2576785224 learned_addr learned my addr 172.21.15.67:0/2576785224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 -- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f769c004d40 msgr2=0x7f76b815d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f769c004d40 0x7f76b815d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 -- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b80aec40 msgr2=0x7f76b8145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b80aec40 0x7f76b8145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 -- 172.21.15.67:0/2576785224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b80b3340 con 0x7f76b80af0a0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bd000640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f76b002c380 tx=0x7f76b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0039070 con 0x7f76b80af0a0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b0043070 con 0x7f76b80af0a0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0009690 con 0x7f76b80af0a0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.226+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b8143cd0 con 0x7f76b80af0a0 2024-04-17T07:29:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.227+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b80b35e0 con 0x7f76b80af0a0 2024-04-17T07:29:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.228+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b8068a10 con 0x7f76b80af0a0 2024-04-17T07:29:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.229+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f76b0003850 con 0x7f76b80af0a0 2024-04-17T07:29:51.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.230+0000 7f76afe00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 0x7f769806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.230+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f76b00ad700 con 0x7f76b80af0a0 2024-04-17T07:29:51.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.231+0000 7f76b7e00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 0x7f769806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.232+0000 7f76b7e00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 0x7f769806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f76a8001200 tx=0x7f76a8009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.235+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f76b007b790 con 0x7f76b80af0a0 2024-04-17T07:29:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.443+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76b80d0dd0 con 0x7f76b80af0a0 2024-04-17T07:29:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.444+0000 7f76afe00640 1 -- 172.21.15.67:0/2576785224 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+17631 (secure 0 0 0) 0x7f76b00044a0 con 0x7f76b80af0a0 2024-04-17T07:29:51.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":86,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:47.783926+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:29:24.457001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-17T07:29:25.538749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/939506877":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6832/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/139946814":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/2814172476":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/3268338662":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1164934579":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6832/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1567626846":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/831200072":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/3826805311":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/12378685":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1955748639":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1146224552":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1795500056":"2024-04-18T07:29:41.466074+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 msgr2=0x7f769806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 0x7f769806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f76a8001200 tx=0x7f76a8009040 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 msgr2=0x7f76b81456b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f76b002c380 tx=0x7f76b0004730 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 shutdown_connections 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76b80af0a0 0x7f76b81456b0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f769806c980 0x7f769806ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76b80aec40 0x7f76b8145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 --2- 172.21.15.67:0/2576785224 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f769c004d40 0x7f76b815d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.447+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 >> 172.21.15.67:0/2576785224 conn(0x7f76b8067d10 msgr2=0x7f76b8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.448+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 shutdown_connections 2024-04-17T07:29:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.448+0000 7f76bda00640 1 -- 172.21.15.67:0/2576785224 wait complete. 2024-04-17T07:29:51.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:29:51.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.694+0000 7f3713000640 1 Processor -- start 2024-04-17T07:29:51.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.694+0000 7f3713000640 1 -- start start 2024-04-17T07:29:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.696+0000 7f3713000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.696+0000 7f3713000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f370c0ce3b0 0x7f370c0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.696+0000 7f3713000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c0cea70 con 0x7f370c145ac0 2024-04-17T07:29:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.696+0000 7f3713000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f370c0ce840 con 0x7f370c0ce3b0 2024-04-17T07:29:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.696+0000 7f3713000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f370c0ced50 con 0x7f370c072d80 2024-04-17T07:29:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f370c0ce3b0 0x7f370c0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f370c0ce3b0 0x7f370c0cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58210/0 (socket says 172.21.15.67:58210) 2024-04-17T07:29:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 -- 172.21.15.67:0/1614404813 learned_addr learned my addr 172.21.15.67:0/1614404813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3703400640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 -- 172.21.15.67:0/1614404813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 msgr2=0x7f370c071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 -- 172.21.15.67:0/1614404813 >> v1:172.21.15.67:6789/0 conn(0x7f370c145ac0 legacy=0x7f370c072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.697+0000 7f3711000640 1 -- 172.21.15.67:0/1614404813 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c0cf030 con 0x7f370c0ce3b0 2024-04-17T07:29:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.698+0000 7f3711000640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c0cfb10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3704009e50 tx=0x7f370402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e722497b5bcd74 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.699+0000 7f370b400640 1 -- 172.21.15.67:0/1614404813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370402cb00 con 0x7f370c0ce3b0 2024-04-17T07:29:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.699+0000 7f370b400640 1 -- 172.21.15.67:0/1614404813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370402cca0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.699+0000 7f370b400640 1 -- 172.21.15.67:0/1614404813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704034610 con 0x7f370c0ce3b0 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.699+0000 7f3713000640 1 -- 172.21.15.67:0/1614404813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 msgr2=0x7f370c0cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.699+0000 7f3713000640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c0cfb10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3704009e50 tx=0x7f370402bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 -- 172.21.15.67:0/1614404813 shutdown_connections 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c0cfb10 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 --2- 172.21.15.67:0/1614404813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 -- 172.21.15.67:0/1614404813 >> 172.21.15.67:0/1614404813 conn(0x7f370c067ee0 msgr2=0x7f370c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 -- 172.21.15.67:0/1614404813 shutdown_connections 2024-04-17T07:29:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.700+0000 7f3713000640 1 -- 172.21.15.67:0/1614404813 wait complete. 2024-04-17T07:29:51.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.701+0000 7f3713000640 1 Processor -- start 2024-04-17T07:29:51.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.702+0000 7f3713000640 1 -- start start 2024-04-17T07:29:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.702+0000 7f3713000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3713000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3713000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c145ac0 0x7f370c0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3713000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f370c145ea0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3713000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f370c0cf5f0 con 0x7f370c145ac0 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3713000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f370c0d0050 con 0x7f370c072d80 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58224/0 (socket says 172.21.15.67:58224) 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 -- 172.21.15.67:0/3464452477 learned_addr learned my addr 172.21.15.67:0/3464452477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f370be00640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 -- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 msgr2=0x7f370c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 -- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c145ac0 msgr2=0x7f370c0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c145ac0 0x7f370c0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f3703400640 1 -- 172.21.15.67:0/3464452477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c06e3a0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.703+0000 7f370be00640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:51.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3703400640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d800 tx=0x7f36f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f400b6c0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f4019070 con 0x7f370c0ce3b0 2024-04-17T07:29:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f400ba80 con 0x7f370c0ce3b0 2024-04-17T07:29:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c06e680 con 0x7f370c0ce3b0 2024-04-17T07:29:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.704+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c06ec40 con 0x7f370c0ce3b0 2024-04-17T07:29:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.705+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370c068ca0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.706+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 37) ==== 91086+0+0 (secure 0 0 0) 0x7f36f400bc20 con 0x7f370c0ce3b0 2024-04-17T07:29:51.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.707+0000 7f3708c00640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 0x7f36e806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:51.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.707+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 6397+0+0 (secure 0 0 0) 0x7f36f408c390 con 0x7f370c0ce3b0 2024-04-17T07:29:51.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.708+0000 7f370be00640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 0x7f36e806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.709+0000 7f370be00640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 0x7f36e806ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0015e0 tx=0x7f36fc009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:51.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.712+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f36f405a4b0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.852+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f36f4056420 con 0x7f370c0ce3b0 2024-04-17T07:29:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.965+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f370c1439f0 con 0x7f370c0ce3b0 2024-04-17T07:29:51.966 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:29:51.966+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:29:51.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.970+0000 7f3708c00640 1 -- 172.21.15.67:0/3464452477 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7f36f4076290 con 0x7f370c0ce3b0 2024-04-17T07:29:51.970 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:29:51.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 msgr2=0x7f36e806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 0x7f36e806ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0015e0 tx=0x7f36fc009040 comp rx=0 tx=0).stop 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 msgr2=0x7f370c13de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d800 tx=0x7f36f400dcc0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 shutdown_connections 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f36e806c980 0x7f36e806ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c145ac0 0x7f370c0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0ce3b0 0x7f370c13de00 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 --2- 172.21.15.67:0/3464452477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c072d80 0x7f370c06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:51.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 >> 172.21.15.67:0/3464452477 conn(0x7f370c067ee0 msgr2=0x7f370c142e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:51.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 shutdown_connections 2024-04-17T07:29:51.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:51.973+0000 7f3713000640 1 -- 172.21.15.67:0/3464452477 wait complete. 2024-04-17T07:29:51.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:29:52.003 INFO:tasks.ceph.mds.d.smithi187.stderr: -4096> 2024-04-17T07:29:33.039+0000 7efd57400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:29:52.097 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:29:52.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.218+0000 7fb2c7e00640 1 Processor -- start 2024-04-17T07:29:52.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.219+0000 7fb2c7e00640 1 -- start start 2024-04-17T07:29:52.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.220+0000 7fb2c7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.220+0000 7fb2c7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 0x7fb2c806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.220+0000 7fb2c7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c8146360 con 0x7fb2c80cd750 2024-04-17T07:29:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.220+0000 7fb2c7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c80cdbb0 con 0x7fb2c806c270 2024-04-17T07:29:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.220+0000 7fb2c7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c80727a0 con 0x7fb2c8145f80 2024-04-17T07:29:52.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.221+0000 7fb2c6a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.221+0000 7fb2c6a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:46140/0 (socket says 172.21.15.67:46140) 2024-04-17T07:29:52.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.221+0000 7fb2c6a00640 1 -- 172.21.15.67:0/419296102 learned_addr learned my addr 172.21.15.67:0/419296102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.221+0000 7fb2c7400640 1 --2- 172.21.15.67:0/419296102 >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 0x7fb2c806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.222+0000 7fb2c6a00640 1 -- 172.21.15.67:0/419296102 >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 msgr2=0x7fb2c806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.222+0000 7fb2c6a00640 1 --2- 172.21.15.67:0/419296102 >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 0x7fb2c806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.222+0000 7fb2c6a00640 1 -- 172.21.15.67:0/419296102 >> v1:172.21.15.187:6789/0 conn(0x7fb2c80cd750 legacy=0x7fb2c806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.222+0000 7fb2c6a00640 1 -- 172.21.15.67:0/419296102 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c8071e70 con 0x7fb2c8145f80 2024-04-17T07:29:52.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.222+0000 7fb2c7400640 1 --2- 172.21.15.67:0/419296102 >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 0x7fb2c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:29:52.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.224+0000 7fb2c6a00640 1 --2- 172.21.15.67:0/419296102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009e50 tx=0x7fb2b802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcee3589026b232 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:52.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c6000640 1 -- 172.21.15.67:0/419296102 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b802ca70 con 0x7fb2c8145f80 2024-04-17T07:29:52.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c6000640 1 -- 172.21.15.67:0/419296102 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b802cc10 con 0x7fb2c8145f80 2024-04-17T07:29:52.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c6000640 1 -- 172.21.15.67:0/419296102 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80343f0 con 0x7fb2c8145f80 2024-04-17T07:29:52.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 -- 172.21.15.67:0/419296102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 msgr2=0x7fb2c80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/419296102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009e50 tx=0x7fb2b802bed0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 -- 172.21.15.67:0/419296102 shutdown_connections 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/419296102 >> v2:172.21.15.187:3300/0 conn(0x7fb2c806c270 0x7fb2c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/419296102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c80cd210 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.225+0000 7fb2c7e00640 1 -- 172.21.15.67:0/419296102 >> 172.21.15.67:0/419296102 conn(0x7fb2c80b2190 msgr2=0x7fb2c80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:52.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.226+0000 7fb2c7e00640 1 -- 172.21.15.67:0/419296102 shutdown_connections 2024-04-17T07:29:52.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.226+0000 7fb2c7e00640 1 -- 172.21.15.67:0/419296102 wait complete. 2024-04-17T07:29:52.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.227+0000 7fb2c7e00640 1 Processor -- start 2024-04-17T07:29:52.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.227+0000 7fb2c7e00640 1 -- start start 2024-04-17T07:29:52.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c80cd750 0x7fb2c815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c81f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8072c30 con 0x7fb2c806c270 2024-04-17T07:29:52.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c8072430 con 0x7fb2c80cd750 2024-04-17T07:29:52.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.228+0000 7fb2c7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c8072db0 con 0x7fb2c8145f80 2024-04-17T07:29:52.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58234/0 (socket says 172.21.15.67:58234) 2024-04-17T07:29:52.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 -- 172.21.15.67:0/3106085166 learned_addr learned my addr 172.21.15.67:0/3106085166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2be000640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c80cd750 0x7fb2c815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c7400640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c81f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 -- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 msgr2=0x7fb2c81f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c81f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 -- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c80cd750 msgr2=0x7fb2c815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c80cd750 0x7fb2c815e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.229+0000 7fb2c6a00640 1 -- 172.21.15.67:0/3106085166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c80cac00 con 0x7fb2c806c270 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2c6a00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f70 tx=0x7fb2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8044020 con 0x7fb2c806c270 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b8040690 con 0x7fb2c806c270 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8009510 con 0x7fb2c806c270 2024-04-17T07:29:52.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c80caee0 con 0x7fb2c806c270 2024-04-17T07:29:52.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.230+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c80cb4a0 con 0x7fb2c806c270 2024-04-17T07:29:52.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.231+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c8068a10 con 0x7fb2c806c270 2024-04-17T07:29:52.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.233+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fb2b8047020 con 0x7fb2c806c270 2024-04-17T07:29:52.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.233+0000 7fb2bf400640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 0x7fb2a406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:52.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.234+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6483+0+0 (secure 0 0 0) 0x7fb2b80aeb70 con 0x7fb2c806c270 2024-04-17T07:29:52.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.234+0000 7fb2be000640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 0x7fb2a406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:52.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.235+0000 7fb2be000640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 0x7fb2a406eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80b1cb0 tx=0x7fb2b0009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:52.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.238+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb2b804e030 con 0x7fb2c806c270 2024-04-17T07:29:52.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.484+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb2c80ca8e0 con 0x7fb2c806c270 2024-04-17T07:29:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.973+0000 7fb2bf400640 1 -- 172.21.15.67:0/3106085166 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7fb2b807cae0 con 0x7fb2c806c270 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.977+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 msgr2=0x7fb2a406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.977+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 0x7fb2a406eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80b1cb0 tx=0x7fb2b0009040 comp rx=0 tx=0).stop 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.977+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 msgr2=0x7fb2c80ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.977+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f70 tx=0x7fb2b8004a40 comp rx=0 tx=0).stop 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 shutdown_connections 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c8145f80 0x7fb2c81f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fb2a406ca10 0x7fb2a406eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c80cd750 0x7fb2c815e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3106085166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806c270 0x7fb2c80ca3a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 >> 172.21.15.67:0/3106085166 conn(0x7fb2c80b2190 msgr2=0x7fb2c806e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 shutdown_connections 2024-04-17T07:29:52.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:52.978+0000 7fb2c7e00640 1 -- 172.21.15.67:0/3106085166 wait complete. 2024-04-17T07:29:53.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:29:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.219+0000 7fcbc7a00640 1 Processor -- start 2024-04-17T07:29:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.219+0000 7fcbc7a00640 1 -- start start 2024-04-17T07:29:53.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc7a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcbc00ba600 0x7fcbc00d4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc7a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcbc00b6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc7a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc00d31b0 con 0x7fcbc015ec60 2024-04-17T07:29:53.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc7a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc015f040 con 0x7fcbc00ba600 2024-04-17T07:29:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc7a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc00d3dc0 con 0x7fcbc00b63d0 2024-04-17T07:29:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc5000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcbc015ec60 0x7fcbc00d2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57228/0 (socket says 172.21.15.67:57228) 2024-04-17T07:29:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.220+0000 7fcbc5000640 1 -- 172.21.15.67:0/924609274 learned_addr learned my addr 172.21.15.67:0/924609274 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcbc00b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbfe00640 1 --2- 172.21.15.67:0/924609274 >> v2:172.21.15.187:3300/0 conn(0x7fcbc00ba600 0x7fcbc00d4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633288031 0 0) 0x7fcbc00d31b0 con 0x7fcbc015ec60 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba4003520 con 0x7fcbc015ec60 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3518369342 0 0) 0x7fcba4003520 con 0x7fcbc015ec60 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 msgr2=0x7fcbc00b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcbc00b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 >> v2:172.21.15.187:3300/0 conn(0x7fcbc00ba600 msgr2=0x7fcbc00d4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 --2- 172.21.15.67:0/924609274 >> v2:172.21.15.187:3300/0 conn(0x7fcbc00ba600 0x7fcbc00d4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc00d3490 con 0x7fcbc015ec60 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcbb0001670 con 0x7fcbc015ec60 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.221+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 >> v1:172.21.15.67:6789/0 conn(0x7fcbc015ec60 legacy=0x7fcbc00d2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba40039b0 0x7fcba4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcba4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcba4004d40 0x7fcba40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc015f040 con 0x7fcba4004d40 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00d3dc0 con 0x7fcba40039b0 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcba400b9c0 con 0x7fcbc00b63d0 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcba4004d40 0x7fcba40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbfe00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcba4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 -- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 msgr2=0x7fcba4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcba4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 -- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba40039b0 msgr2=0x7fcba4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba40039b0 0x7fcba4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbc5a00640 1 -- 172.21.15.67:0/924609274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba4003520 con 0x7fcba4004d40 2024-04-17T07:29:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.222+0000 7fcbbfe00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b63d0 0x7fcba4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcba4004d40 0x7fcba40051c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e170 tx=0x7fcba4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64a2558028e91cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8039070 con 0x7fcba4004d40 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb802cbf0 con 0x7fcba4004d40 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbbf400640 1 -- 172.21.15.67:0/924609274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8009c10 con 0x7fcba4004d40 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbc7a00640 1 -- 172.21.15.67:0/924609274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcba4004d40 msgr2=0x7fcba40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/924609274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcba4004d40 0x7fcba40051c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e170 tx=0x7fcba4005830 comp rx=0 tx=0).stop 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.224+0000 7fcbc5a00640 1 -- 172.21.15.67:0/924609274 reap_dead start 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.225+0000 7fcbc7a00640 1 -- 172.21.15.67:0/924609274 shutdown_connections 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.225+0000 7fcbc7a00640 1 -- 172.21.15.67:0/924609274 >> 172.21.15.67:0/924609274 conn(0x7fcbc0067c30 msgr2=0x7fcbc00d13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.225+0000 7fcbc7a00640 1 -- 172.21.15.67:0/924609274 shutdown_connections 2024-04-17T07:29:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.225+0000 7fcbc7a00640 1 -- 172.21.15.67:0/924609274 wait complete. 2024-04-17T07:29:53.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.226+0000 7fcbc7a00640 1 Processor -- start 2024-04-17T07:29:53.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.227+0000 7fcbc7a00640 1 -- start start 2024-04-17T07:29:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba4004d40 0x7fcbc014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b47f0 0x7fcbc00b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00b6860 con 0x7fcbc01593a0 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00d3a50 con 0x7fcba4004d40 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc7a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc00b69e0 con 0x7fcbc00b47f0 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58262/0 (socket says 172.21.15.67:58262) 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 -- 172.21.15.67:0/4024545672 learned_addr learned my addr 172.21.15.67:0/4024545672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbbfe00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b47f0 0x7fcbc00b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5000640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba4004d40 0x7fcbc014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 -- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b47f0 msgr2=0x7fcbc00b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b47f0 0x7fcbc00b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 -- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba4004d40 msgr2=0x7fcbc014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba4004d40 0x7fcbc014ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.228+0000 7fcbc5a00640 1 -- 172.21.15.67:0/4024545672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc01f7de0 con 0x7fcbc01593a0 2024-04-17T07:29:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbc5a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e170 tx=0x7fcbb8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8039030 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb803a040 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8009c10 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc01f80c0 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.229+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc01f8680 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.230+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc006e000 con 0x7fcbc01593a0 2024-04-17T07:29:53.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.231+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fcbb8045030 con 0x7fcbc01593a0 2024-04-17T07:29:53.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.232+0000 7fcbbcc00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 0x7fcba006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:53.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.232+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6483+0+0 (secure 0 0 0) 0x7fcbb80add50 con 0x7fcbc01593a0 2024-04-17T07:29:53.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.233+0000 7fcbc5000640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 0x7fcba006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:53.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.234+0000 7fcbc5000640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 0x7fcba006eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0000c00 tx=0x7fcbb00016c0 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:53.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.237+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcbb804c060 con 0x7fcbc01593a0 2024-04-17T07:29:53.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.440+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcbc0068d60 con 0x7fcbc01593a0 2024-04-17T07:29:53.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.983+0000 7fcbbcc00640 1 -- 172.21.15.67:0/4024545672 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) ==== 181+0+0 (secure 0 0 0) 0x7fcbb807bd50 con 0x7fcbc01593a0 2024-04-17T07:29:53.984 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:29:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 msgr2=0x7fcba006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 0x7fcba006eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0000c00 tx=0x7fcbb00016c0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 msgr2=0x7fcbc0159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e170 tx=0x7fcbb8009750 comp rx=0 tx=0).stop 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 shutdown_connections 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc01593a0 0x7fcbc0159850 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc00b47f0 0x7fcbc00b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fcba006ca10 0x7fcba006eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 --2- 172.21.15.67:0/4024545672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcba4004d40 0x7fcbc014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 >> 172.21.15.67:0/4024545672 conn(0x7fcbc0067c30 msgr2=0x7fcbc00b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:53.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.987+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 shutdown_connections 2024-04-17T07:29:53.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:53.988+0000 7fcbc7a00640 1 -- 172.21.15.67:0/4024545672 wait complete. 2024-04-17T07:29:54.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:29:54.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.239+0000 7f099be00640 1 Processor -- start 2024-04-17T07:29:54.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.239+0000 7f099be00640 1 -- start start 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.240+0000 7f099be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.241+0000 7f099be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f099c0ce510 0x7f099c0b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.241+0000 7f099be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099c0721c0 con 0x7f099c162f80 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.241+0000 7f099be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f099c0724a0 con 0x7f099c0ce510 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.241+0000 7f099be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f099c072780 con 0x7f099c0cd750 2024-04-17T07:29:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.241+0000 7f0993e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099aa00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f099c162f80 0x7f099c0cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:41460/0 (socket says 172.21.15.67:41460) 2024-04-17T07:29:54.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099aa00640 1 -- 172.21.15.67:0/3436537260 learned_addr learned my addr 172.21.15.67:0/3436537260 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:54.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f0993e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:45496/0 (socket says 172.21.15.67:45496) 2024-04-17T07:29:54.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099b400640 1 --2- 172.21.15.67:0/3436537260 >> v2:172.21.15.187:3300/0 conn(0x7f099c0ce510 0x7f099c0b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099b400640 1 -- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 msgr2=0x7f099c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099b400640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099b400640 1 -- 172.21.15.67:0/3436537260 >> v1:172.21.15.187:6789/0 conn(0x7f099c162f80 legacy=0x7f099c0cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f099b400640 1 -- 172.21.15.67:0/3436537260 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c072a60 con 0x7f099c0ce510 2024-04-17T07:29:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.242+0000 7f0993e00640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:29:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.244+0000 7f099b400640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c0ce510 0x7f099c0b48c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0994009d90 tx=0x7f099402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19332b0677cf1ee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:54.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.244+0000 7f099a000640 1 -- 172.21.15.67:0/3436537260 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994039070 con 0x7f099c0ce510 2024-04-17T07:29:54.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.244+0000 7f099a000640 1 -- 172.21.15.67:0/3436537260 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099402cc60 con 0x7f099c0ce510 2024-04-17T07:29:54.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.244+0000 7f099a000640 1 -- 172.21.15.67:0/3436537260 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994034660 con 0x7f099c0ce510 2024-04-17T07:29:54.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 -- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c0ce510 msgr2=0x7f099c0b48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c0ce510 0x7f099c0b48c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0994009d90 tx=0x7f099402beb0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 -- 172.21.15.67:0/3436537260 shutdown_connections 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c0ce510 0x7f099c0b48c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 --2- 172.21.15.67:0/3436537260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 -- 172.21.15.67:0/3436537260 >> 172.21.15.67:0/3436537260 conn(0x7f099c06b8c0 msgr2=0x7f099c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 -- 172.21.15.67:0/3436537260 shutdown_connections 2024-04-17T07:29:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.245+0000 7f099be00640 1 -- 172.21.15.67:0/3436537260 wait complete. 2024-04-17T07:29:54.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.247+0000 7f099be00640 1 Processor -- start 2024-04-17T07:29:54.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.247+0000 7f099be00640 1 -- start start 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c1568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c162f80 0x7f099c15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f099c0d0260 con 0x7f099c0ce510 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f099c0d00e0 con 0x7f099c162f80 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f099be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f099c0d03e0 con 0x7f099c0cd750 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f0993e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f0993e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38692/0 (socket says 172.21.15.67:38692) 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.248+0000 7f0993e00640 1 -- 172.21.15.67:0/94284710 learned_addr learned my addr 172.21.15.67:0/94284710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f099b400640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c162f80 0x7f099c15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f099aa00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c1568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 -- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 msgr2=0x7f099c1568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c1568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 -- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c162f80 msgr2=0x7f099c15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c162f80 0x7f099c15d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 -- 172.21.15.67:0/94284710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c15a950 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.249+0000 7f0993e00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f098400ebe0 tx=0x7f098400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.250+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984017070 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.250+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098400cc30 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.250+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984013470 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.250+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099c159bd0 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.250+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099c159e70 con 0x7f099c0ce510 2024-04-17T07:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.251+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099c068a10 con 0x7f099c0ce510 2024-04-17T07:29:54.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.252+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f098401a070 con 0x7f099c0ce510 2024-04-17T07:29:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.253+0000 7f0992a00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 0x7f097806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.253+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6045+0+0 (secure 0 0 0) 0x7f098408bdf0 con 0x7f099c0ce510 2024-04-17T07:29:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.254+0000 7f099aa00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 0x7f097806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.255+0000 7f099aa00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 0x7f097806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f098c000f60 tx=0x7f098c009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.258+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0984059fc0 con 0x7f099c0ce510 2024-04-17T07:29:54.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.477+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f099c068610 con 0x7f099c0ce510 2024-04-17T07:29:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:54.998+0000 7f0992a00640 1 -- 172.21.15.67:0/94284710 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) ==== 169+0+0 (secure 0 0 0) 0x7f09840598c0 con 0x7f099c0ce510 2024-04-17T07:29:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:29:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 msgr2=0x7f097806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 0x7f097806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f098c000f60 tx=0x7f098c009040 comp rx=0 tx=0).stop 2024-04-17T07:29:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 msgr2=0x7f099c152a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f098400ebe0 tx=0x7f098400c520 comp rx=0 tx=0).stop 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 shutdown_connections 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f097806ca10 0x7f097806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f099c162f80 0x7f099c15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c0ce510 0x7f099c152a50 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 --2- 172.21.15.67:0/94284710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f099c0cd750 0x7f099c1568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.003+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 >> 172.21.15.67:0/94284710 conn(0x7f099c06b8c0 msgr2=0x7f099c165930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.004+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 shutdown_connections 2024-04-17T07:29:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.004+0000 7f099be00640 1 -- 172.21.15.67:0/94284710 wait complete. 2024-04-17T07:29:55.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:29:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.254+0000 7f620fe00640 1 Processor -- start 2024-04-17T07:29:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.254+0000 7f620fe00640 1 -- start start 2024-04-17T07:29:55.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f6210167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62100b4d30 con 0x7f62100b2af0 2024-04-17T07:29:55.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6210163350 con 0x7f62100b3760 2024-04-17T07:29:55.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f621006c9d0 con 0x7f62100c5be0 2024-04-17T07:29:55.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62100b2af0 0x7f62100c54d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53776/0 (socket says 172.21.15.67:53776) 2024-04-17T07:29:55.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.256+0000 7f620ea00640 1 -- 172.21.15.67:0/128423903 learned_addr learned my addr 172.21.15.67:0/128423903 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:55.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620e000640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f6210167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215463402 0 0) 0x7f6200001110 con 0x7f62100b2af0 2024-04-17T07:29:55.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f4003520 con 0x7f62100b2af0 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1819984069 0 0) 0x7f61f4003520 con 0x7f62100b2af0 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 msgr2=0x7f6210167470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f6210167470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 >> v1:172.21.15.187:6789/0 conn(0x7f62100b3760 legacy=0x7f62100d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621006d250 con 0x7f62100b2af0 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6200001840 con 0x7f62100b2af0 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 >> v1:172.21.15.67:6789/0 conn(0x7f62100b2af0 legacy=0x7f62100c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f40039e0 0x7f61f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f61f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f4004d40 0x7f61f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61f400bb50 con 0x7f61f4004d40 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61f400bcd0 con 0x7f61f40039e0 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.257+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61f400bed0 con 0x7f62100c5be0 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f4004d40 0x7f61f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620ea00640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f40039e0 0x7f61f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620e000640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f61f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 -- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 msgr2=0x7f61f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62100c5be0 0x7f61f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 -- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f40039e0 msgr2=0x7f61f4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f40039e0 0x7f61f4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.258+0000 7f620f400640 1 -- 172.21.15.67:0/128423903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f4003520 con 0x7f61f4004d40 2024-04-17T07:29:55.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.259+0000 7f620f400640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f4004d40 0x7f61f40051c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6208001e20 tx=0x7f61f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c008117ba35abd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.259+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208001a70 con 0x7f61f4004d40 2024-04-17T07:29:55.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6208003b70 con 0x7f61f4004d40 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620d600640 1 -- 172.21.15.67:0/128423903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080050d0 con 0x7f61f4004d40 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 -- 172.21.15.67:0/128423903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f4004d40 msgr2=0x7f61f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 --2- 172.21.15.67:0/128423903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f4004d40 0x7f61f40051c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6208001e20 tx=0x7f61f4005830 comp rx=0 tx=0).stop 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620f400640 1 -- 172.21.15.67:0/128423903 reap_dead start 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 -- 172.21.15.67:0/128423903 shutdown_connections 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 -- 172.21.15.67:0/128423903 >> 172.21.15.67:0/128423903 conn(0x7f62100b3ff0 msgr2=0x7f62100b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 -- 172.21.15.67:0/128423903 shutdown_connections 2024-04-17T07:29:55.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.260+0000 7f620fe00640 1 -- 172.21.15.67:0/128423903 wait complete. 2024-04-17T07:29:55.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.262+0000 7f620fe00640 1 Processor -- start 2024-04-17T07:29:55.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.262+0000 7f620fe00640 1 -- start start 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f4004d40 0x7f62101645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 0x7f62100b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f621006cee0 con 0x7f6210164af0 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f621006cc00 con 0x7f61f4004d40 2024-04-17T07:29:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.263+0000 7f620fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6210167470 con 0x7f6210153270 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38706/0 (socket says 172.21.15.67:38706) 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 -- 172.21.15.67:0/627495049 learned_addr learned my addr 172.21.15.67:0/627495049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620ea00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f4004d40 0x7f62101645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620f400640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 0x7f62100b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 -- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 msgr2=0x7f62100b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 0x7f62100b1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 -- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f4004d40 msgr2=0x7f62101645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f4004d40 0x7f62101645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 -- 172.21.15.67:0/627495049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62101536d0 con 0x7f6210164af0 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620f400640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 0x7f62100b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:55.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.264+0000 7f620e000640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f61f8011560 tx=0x7f61f8011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.265+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8004220 con 0x7f6210164af0 2024-04-17T07:29:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.265+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f80043c0 con 0x7f6210164af0 2024-04-17T07:29:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.265+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8015870 con 0x7f6210164af0 2024-04-17T07:29:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.265+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6210165120 con 0x7f6210164af0 2024-04-17T07:29:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.265+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62101656e0 con 0x7f6210164af0 2024-04-17T07:29:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.266+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62100755d0 con 0x7f6210164af0 2024-04-17T07:29:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.267+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f61f8015a10 con 0x7f6210164af0 2024-04-17T07:29:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.268+0000 7f6206a00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 0x7f61ec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.268+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f61f808bf80 con 0x7f6210164af0 2024-04-17T07:29:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.269+0000 7f620ea00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 0x7f61ec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.270+0000 7f620ea00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 0x7f61ec06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6200001050 tx=0x7f6200009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.273+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f61f8019030 con 0x7f6210164af0 2024-04-17T07:29:55.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.479+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f6210162180 con 0x7f6210164af0 2024-04-17T07:29:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.486+0000 7f6206a00640 1 -- 172.21.15.67:0/627495049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f61f805a2d0 con 0x7f6210164af0 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.488+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 msgr2=0x7f61ec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.488+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 0x7f61ec06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6200001050 tx=0x7f6200009040 comp rx=0 tx=0).stop 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.488+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 msgr2=0x7f6210152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f61f8011560 tx=0x7f61f8011a20 comp rx=0 tx=0).stop 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 shutdown_connections 2024-04-17T07:29:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6210153270 0x7f62100b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6210164af0 0x7f6210152d30 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f61ec06ca10 0x7f61ec06eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 --2- 172.21.15.67:0/627495049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61f4004d40 0x7f62101645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 >> 172.21.15.67:0/627495049 conn(0x7f62100b3ff0 msgr2=0x7f62100d1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 shutdown_connections 2024-04-17T07:29:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.489+0000 7f620fe00640 1 -- 172.21.15.67:0/627495049 wait complete. 2024-04-17T07:29:55.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:29:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.724+0000 7f9911200640 1 Processor -- start 2024-04-17T07:29:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.724+0000 7f9911200640 1 -- start start 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f9911200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 0x7f990c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f9911200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0cf8e0 0x7f990c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f9911200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990c0d00b0 con 0x7f990c15ecb0 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f9911200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f990c0d0c30 con 0x7f990c0cf8e0 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f9911200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f990c0d0390 con 0x7f990c0d33e0 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0cf8e0 0x7f990c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0cf8e0 0x7f990c0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38716/0 (socket says 172.21.15.67:38716) 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 -- 172.21.15.67:0/633505941 learned_addr learned my addr 172.21.15.67:0/633505941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990aa00640 1 --2- 172.21.15.67:0/633505941 >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 0x7f990c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 -- 172.21.15.67:0/633505941 >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 msgr2=0x7f990c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 --2- 172.21.15.67:0/633505941 >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 0x7f990c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.726+0000 7f990be00640 1 -- 172.21.15.67:0/633505941 >> v1:172.21.15.67:6789/0 conn(0x7f990c15ecb0 legacy=0x7f990c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.727+0000 7f990be00640 1 -- 172.21.15.67:0/633505941 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c0d0670 con 0x7f990c0cf8e0 2024-04-17T07:29:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.727+0000 7f990aa00640 1 --2- 172.21.15.67:0/633505941 >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 0x7f990c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.728+0000 7f990be00640 1 --2- 172.21.15.67:0/633505941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0b3fc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b61a0 tx=0x7f990402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a3a9c0cd2ddf1f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.728+0000 7f990a000640 1 -- 172.21.15.67:0/633505941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990402ca80 con 0x7f990c0cf8e0 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.728+0000 7f990a000640 1 -- 172.21.15.67:0/633505941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990402cc20 con 0x7f990c0cf8e0 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.728+0000 7f990a000640 1 -- 172.21.15.67:0/633505941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040343f0 con 0x7f990c0cf8e0 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 -- 172.21.15.67:0/633505941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 msgr2=0x7f990c0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 --2- 172.21.15.67:0/633505941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0b3fc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b61a0 tx=0x7f990402bee0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 -- 172.21.15.67:0/633505941 shutdown_connections 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 --2- 172.21.15.67:0/633505941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0b3fc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 --2- 172.21.15.67:0/633505941 >> v2:172.21.15.187:3300/0 conn(0x7f990c0d33e0 0x7f990c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 -- 172.21.15.67:0/633505941 >> 172.21.15.67:0/633505941 conn(0x7f990c08bff0 msgr2=0x7f990c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 -- 172.21.15.67:0/633505941 shutdown_connections 2024-04-17T07:29:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.729+0000 7f9911200640 1 -- 172.21.15.67:0/633505941 wait complete. 2024-04-17T07:29:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.731+0000 7f9911200640 1 Processor -- start 2024-04-17T07:29:55.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.731+0000 7f9911200640 1 -- start start 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f990c0d33e0 0x7f990c09f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f990c15ecb0 0x7f990c0b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f990c15f090 con 0x7f990c0cf8e0 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f990c0cfd30 con 0x7f990c0d33e0 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f9911200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f990c15f210 con 0x7f990c15ecb0 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38722/0 (socket says 172.21.15.67:38722) 2024-04-17T07:29:55.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 -- 172.21.15.67:0/1897830299 learned_addr learned my addr 172.21.15.67:0/1897830299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990aa00640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f990c0d33e0 0x7f990c09f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 -- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f990c15ecb0 msgr2=0x7f990c0b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f990c15ecb0 0x7f990c0b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 -- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f990c0d33e0 msgr2=0x7f990c09f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f990c0d33e0 0x7f990c09f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.732+0000 7f990b400640 1 -- 172.21.15.67:0/1897830299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c0b7520 con 0x7f990c0cf8e0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f990b400640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c3c0 tx=0x7f98fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0052d0 con 0x7f990c0cf8e0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc005490 con 0x7f990c0cf8e0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc007260 con 0x7f990c0cf8e0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c0b7800 con 0x7f990c0cf8e0 2024-04-17T07:29:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.733+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c09f9e0 con 0x7f990c0cf8e0 2024-04-17T07:29:55.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.734+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990c068a10 con 0x7f990c0cf8e0 2024-04-17T07:29:55.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.736+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f98fc007400 con 0x7f990c0cf8e0 2024-04-17T07:29:55.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.736+0000 7f9903400640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 0x7f98ec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:55.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.737+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f98fc08b180 con 0x7f990c0cf8e0 2024-04-17T07:29:55.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.737+0000 7f990aa00640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 0x7f98ec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:55.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.738+0000 7f990aa00640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 0x7f98ec06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b4e80 tx=0x7f98f8008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:55.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.741+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f98fc0594e0 con 0x7f990c0cf8e0 2024-04-17T07:29:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.949+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f990c0b5a70 con 0x7f990c0cf8e0 2024-04-17T07:29:55.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.958+0000 7f9903400640 1 -- 172.21.15.67:0/1897830299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7f98fc058de0 con 0x7f990c0cf8e0 2024-04-17T07:29:55.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 msgr2=0x7f98ec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 0x7f98ec06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f990c0b4e80 tx=0x7f98f8008040 comp rx=0 tx=0).stop 2024-04-17T07:29:55.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 msgr2=0x7f990c0caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:55.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c3c0 tx=0x7f98fc00c880 comp rx=0 tx=0).stop 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 shutdown_connections 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f990c15ecb0 0x7f990c0b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f98ec06ca10 0x7f98ec06eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f990c0d33e0 0x7f990c09f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 --2- 172.21.15.67:0/1897830299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0cf8e0 0x7f990c0caac0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.960+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 >> 172.21.15.67:0/1897830299 conn(0x7f990c08bff0 msgr2=0x7f990c0d4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.961+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 shutdown_connections 2024-04-17T07:29:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:55.961+0000 7f9911200640 1 -- 172.21.15.67:0/1897830299 wait complete. 2024-04-17T07:29:55.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:29:56.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.203+0000 7fa302000640 1 Processor -- start 2024-04-17T07:29:56.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.203+0000 7fa302000640 1 -- start start 2024-04-17T07:29:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa302000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2fc0ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa302000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2fc0cea70 con 0x7fa2fc072d80 2024-04-17T07:29:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa302000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2fc0ced50 con 0x7fa2fc145ac0 2024-04-17T07:29:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa302000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc0722e0 con 0x7fa2fc071ea0 2024-04-17T07:29:56.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa2faa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2fc072d80 0x7fa2fc0cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53788/0 (socket says 172.21.15.67:53788) 2024-04-17T07:29:56.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.205+0000 7fa2faa00640 1 -- 172.21.15.67:0/1401153756 learned_addr learned my addr 172.21.15.67:0/1401153756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:56.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2fc0ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258301477 0 0) 0x7fa2fc0cea70 con 0x7fa2fc072d80 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e0003520 con 0x7fa2fc072d80 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2904457838 0 0) 0x7fa2e0003520 con 0x7fa2fc072d80 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 msgr2=0x7fa2fc0ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2fc0ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 >> v1:172.21.15.187:6789/0 conn(0x7fa2fc145ac0 legacy=0x7fa2fc072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.206+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc0cf030 con 0x7fa2fc072d80 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa2e8001670 con 0x7fa2fc072d80 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 >> v1:172.21.15.67:6789/0 conn(0x7fa2fc072d80 legacy=0x7fa2fc0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e00039e0 0x7fa2e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0722e0 con 0x7fa2e00039e0 2024-04-17T07:29:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e000ba70 con 0x7fa2e0004800 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e000bc70 con 0x7fa2fc071ea0 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e00039e0 0x7fa2e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2faa00640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 -- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 msgr2=0x7fa2e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc071ea0 0x7fa2e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 -- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2e0004800 msgr2=0x7fa2e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2fb400640 1 -- 172.21.15.67:0/1401153756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e0003520 con 0x7fa2e00039e0 2024-04-17T07:29:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.207+0000 7fa2faa00640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa2fb400640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e00039e0 0x7fa2e0003e00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001e20 tx=0x7fa2e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a284e5067b9f3b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4001a70 con 0x7fa2e00039e0 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f4003b70 con 0x7fa2e00039e0 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa2fa000640 1 -- 172.21.15.67:0/1401153756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40050d0 con 0x7fa2e00039e0 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa302000640 1 -- 172.21.15.67:0/1401153756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e00039e0 msgr2=0x7fa2e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.209+0000 7fa302000640 1 --2- 172.21.15.67:0/1401153756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e00039e0 0x7fa2e0003e00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001e20 tx=0x7fa2e0005830 comp rx=0 tx=0).stop 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.210+0000 7fa2fbe00640 1 -- 172.21.15.67:0/1401153756 reap_dead start 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.210+0000 7fa302000640 1 -- 172.21.15.67:0/1401153756 shutdown_connections 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.210+0000 7fa302000640 1 -- 172.21.15.67:0/1401153756 >> 172.21.15.67:0/1401153756 conn(0x7fa2fc067ee0 msgr2=0x7fa2fc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.210+0000 7fa302000640 1 -- 172.21.15.67:0/1401153756 shutdown_connections 2024-04-17T07:29:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.210+0000 7fa302000640 1 -- 172.21.15.67:0/1401153756 wait complete. 2024-04-17T07:29:56.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.211+0000 7fa302000640 1 Processor -- start 2024-04-17T07:29:56.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.212+0000 7fa302000640 1 -- start start 2024-04-17T07:29:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.212+0000 7fa302000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 0x7fa2fc1f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa302000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa302000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc139580 0x7fa2fc139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa302000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc145ea0 con 0x7fa2fc13f090 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa302000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0cf5f0 con 0x7fa2fc139580 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa302000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc0ce3b0 con 0x7fa2e00039e0 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38744/0 (socket says 172.21.15.67:38744) 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 -- 172.21.15.67:0/3154942028 learned_addr learned my addr 172.21.15.67:0/3154942028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2fb400640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 0x7fa2fc1f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 -- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 msgr2=0x7fa2fc1f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 0x7fa2fc1f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 -- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc139580 msgr2=0x7fa2fc139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc139580 0x7fa2fc139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2faa00640 1 -- 172.21.15.67:0/3154942028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc13a2e0 con 0x7fa2fc13f090 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.213+0000 7fa2fb400640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 0x7fa2fc1f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa2faa00640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800d840 tx=0x7fa2e800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8019070 con 0x7fa2fc13f090 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e80074e0 con 0x7fa2fc13f090 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80049e0 con 0x7fa2fc13f090 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc13a5c0 con 0x7fa2fc13f090 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.214+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc13ab80 con 0x7fa2fc13f090 2024-04-17T07:29:56.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.215+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2fc068ca0 con 0x7fa2fc13f090 2024-04-17T07:29:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.216+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fa2e801c070 con 0x7fa2fc13f090 2024-04-17T07:29:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.217+0000 7fa2f3400640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 0x7fa2dc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.217+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7fa2e808b8c0 con 0x7fa2fc13f090 2024-04-17T07:29:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.218+0000 7fa2fb400640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 0x7fa2dc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.219+0000 7fa2fb400640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 0x7fa2dc06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4034000 tx=0x7fa2f4039040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.221+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa2e8059a80 con 0x7fa2fc13f090 2024-04-17T07:29:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.424+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7fa2fc0d0500 con 0x7fa2fc13f090 2024-04-17T07:29:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.429+0000 7fa2f3400640 1 -- 172.21.15.67:0/3154942028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v27) ==== 120+0+0 (secure 0 0 0) 0x7fa2e8059380 con 0x7fa2fc13f090 2024-04-17T07:29:56.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 msgr2=0x7fa2dc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 0x7fa2dc06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4034000 tx=0x7fa2f4039040 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 msgr2=0x7fa2fc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800d840 tx=0x7fa2e800dd00 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 shutdown_connections 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa2dc06ca10 0x7fa2dc06eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc139580 0x7fa2fc139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc13f090 0x7fa2fc13f4d0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 --2- 172.21.15.67:0/3154942028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2e00039e0 0x7fa2fc1f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 >> 172.21.15.67:0/3154942028 conn(0x7fa2fc067ee0 msgr2=0x7fa2fc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 shutdown_connections 2024-04-17T07:29:56.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.432+0000 7fa302000640 1 -- 172.21.15.67:0/3154942028 wait complete. 2024-04-17T07:29:56.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-04-17T07:29:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.665+0000 7fab69a00640 1 Processor -- start 2024-04-17T07:29:56.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.666+0000 7fab69a00640 1 -- start start 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.667+0000 7fab69a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab64142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.667+0000 7fab69a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab640cd4a0 con 0x7fab6406b8c0 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.667+0000 7fab69a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab641607f0 con 0x7fab64160280 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.667+0000 7fab69a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab6415ce30 con 0x7fab64141ed0 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.668+0000 7fab63e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab6406b8c0 0x7fab640cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53798/0 (socket says 172.21.15.67:53798) 2024-04-17T07:29:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.668+0000 7fab63e00640 1 -- 172.21.15.67:0/2325351125 learned_addr learned my addr 172.21.15.67:0/2325351125 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.668+0000 7fab69000640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab64142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.668+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253476494 0 0) 0x7fab640cd4a0 con 0x7fab6406b8c0 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.668+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab48003520 con 0x7fab6406b8c0 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1173870080 0 0) 0x7fab48003520 con 0x7fab6406b8c0 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 msgr2=0x7fab64142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab64142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 >> v1:172.21.15.187:6789/0 conn(0x7fab64160280 legacy=0x7fab6406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab64160ad0 con 0x7fab6406b8c0 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fab54001670 con 0x7fab6406b8c0 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 >> v1:172.21.15.67:6789/0 conn(0x7fab6406b8c0 legacy=0x7fab640cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab48003a50 0x7fab48003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48004d40 0x7fab480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab6415ce30 con 0x7fab48004d40 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab4800ba70 con 0x7fab48003a50 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab4800bc70 con 0x7fab64141ed0 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.669+0000 7fab69000640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48004d40 0x7fab480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab63e00640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab48003a50 0x7fab48003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab5b400640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab69000640 1 -- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 msgr2=0x7fab48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab69000640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab64141ed0 0x7fab48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab69000640 1 -- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab48003a50 msgr2=0x7fab48003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab69000640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab48003a50 0x7fab48003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.670+0000 7fab69000640 1 -- 172.21.15.67:0/2325351125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab48003520 con 0x7fab48004d40 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.671+0000 7fab69000640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48004d40 0x7fab480051c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009f90 tx=0x7fab48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5bd74f85bdc286 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.671+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c039070 con 0x7fab48004d40 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.671+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab5c02cbd0 con 0x7fab48004d40 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.671+0000 7fab63400640 1 -- 172.21.15.67:0/2325351125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c009c10 con 0x7fab48004d40 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 -- 172.21.15.67:0/2325351125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48004d40 msgr2=0x7fab480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 --2- 172.21.15.67:0/2325351125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48004d40 0x7fab480051c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009f90 tx=0x7fab48005830 comp rx=0 tx=0).stop 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69000640 1 -- 172.21.15.67:0/2325351125 reap_dead start 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 -- 172.21.15.67:0/2325351125 shutdown_connections 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 -- 172.21.15.67:0/2325351125 >> 172.21.15.67:0/2325351125 conn(0x7fab64067d10 msgr2=0x7fab640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 -- 172.21.15.67:0/2325351125 shutdown_connections 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.672+0000 7fab69a00640 1 -- 172.21.15.67:0/2325351125 wait complete. 2024-04-17T07:29:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.674+0000 7fab69a00640 1 Processor -- start 2024-04-17T07:29:56.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.674+0000 7fab69a00640 1 -- start start 2024-04-17T07:29:56.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab48004d40 0x7fab64155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64143500 0x7fab6415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab6413f9b0 con 0x7fab6406f8e0 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab64161090 con 0x7fab64143500 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.675+0000 7fab69a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab6413fb30 con 0x7fab48004d40 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38750/0 (socket says 172.21.15.67:38750) 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 -- 172.21.15.67:0/1001059743 learned_addr learned my addr 172.21.15.67:0/1001059743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab63e00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab48004d40 0x7fab64155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab69000640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64143500 0x7fab6415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 -- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab48004d40 msgr2=0x7fab64155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab48004d40 0x7fab64155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 -- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64143500 msgr2=0x7fab6415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64143500 0x7fab6415b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.676+0000 7fab5b400640 1 -- 172.21.15.67:0/1001059743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6406e840 con 0x7fab6406f8e0 2024-04-17T07:29:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab5b400640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00e5a0 tx=0x7fab4c00ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c0053a0 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c018070 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6406eb20 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c003830 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.677+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6406f0e0 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.678+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab64160820 con 0x7fab6406f8e0 2024-04-17T07:29:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.679+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fab4c0039d0 con 0x7fab6406f8e0 2024-04-17T07:29:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.680+0000 7fab60c00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 0x7fab4006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.680+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7fab4c08bb60 con 0x7fab6406f8e0 2024-04-17T07:29:56.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.681+0000 7fab63e00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 0x7fab4006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.682+0000 7fab63e00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 0x7fab4006eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fab54005ad0 tx=0x7fab54009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:56.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.685+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fab64160820 con 0x7fab6406f8e0 2024-04-17T07:29:56.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:56.973+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7fab64068a10 con 0x7fab6406f8e0 2024-04-17T07:29:57.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.004+0000 7fab60c00640 1 -- 172.21.15.67:0/1001059743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v600) ==== 156+0+0 (secure 0 0 0) 0x7fab4c01a310 con 0x7fab6406f8e0 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 msgr2=0x7fab4006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 0x7fab4006eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fab54005ad0 tx=0x7fab54009040 comp rx=0 tx=0).stop 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 msgr2=0x7fab64142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00e5a0 tx=0x7fab4c00ea60 comp rx=0 tx=0).stop 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 shutdown_connections 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fab4006ca10 0x7fab4006eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab64143500 0x7fab6415b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6406f8e0 0x7fab64142fc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 --2- 172.21.15.67:0/1001059743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab48004d40 0x7fab64155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.008+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 >> 172.21.15.67:0/1001059743 conn(0x7fab64067d10 msgr2=0x7fab640cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.009+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 shutdown_connections 2024-04-17T07:29:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.009+0000 7fab69a00640 1 -- 172.21.15.67:0/1001059743 wait complete. 2024-04-17T07:29:57.030 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-04-17T07:29:57.030 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-04-17T07:29:57.030 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:29:57.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:29:57.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.256+0000 7f6f77c00640 1 Processor -- start 2024-04-17T07:29:57.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.256+0000 7f6f77c00640 1 -- start start 2024-04-17T07:29:57.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.257+0000 7f6f77c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7015ed10 0x7f6f700d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.257+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f700d2ab0 0x7f6f700b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f77c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f700d3020 con 0x7f6f700cf640 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f77c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6f700d3c30 con 0x7f6f7015ed10 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f77c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f700d3300 con 0x7f6f700d2ab0 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7015ed10 0x7f6f700d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f7015ed10 0x7f6f700d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38766/0 (socket says 172.21.15.67:38766) 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 -- 172.21.15.67:0/2408793064 learned_addr learned my addr 172.21.15.67:0/2408793064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 -- 172.21.15.67:0/2408793064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f700d2ab0 msgr2=0x7f6f700b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 --2- 172.21.15.67:0/2408793064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f700d2ab0 0x7f6f700b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 -- 172.21.15.67:0/2408793064 >> v1:172.21.15.187:6789/0 conn(0x7f6f700cf640 legacy=0x7f6f700cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.258+0000 7f6f6fe00640 1 -- 172.21.15.67:0/2408793064 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f700d35e0 con 0x7f6f7015ed10 2024-04-17T07:29:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.260+0000 7f6f6fe00640 1 --2- 172.21.15.67:0/2408793064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7015ed10 0x7f6f700d40c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009870 tx=0x7f6f6002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6787bde4673d9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.260+0000 7f6f6f400640 1 -- 172.21.15.67:0/2408793064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6002f830 con 0x7f6f7015ed10 2024-04-17T07:29:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.260+0000 7f6f6f400640 1 -- 172.21.15.67:0/2408793064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6002f9d0 con 0x7f6f7015ed10 2024-04-17T07:29:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.260+0000 7f6f6f400640 1 -- 172.21.15.67:0/2408793064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6002fda0 con 0x7f6f7015ed10 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 -- 172.21.15.67:0/2408793064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7015ed10 msgr2=0x7f6f700d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 --2- 172.21.15.67:0/2408793064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7015ed10 0x7f6f700d40c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009870 tx=0x7f6f6002ba10 comp rx=0 tx=0).stop 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 -- 172.21.15.67:0/2408793064 shutdown_connections 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 --2- 172.21.15.67:0/2408793064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f700d2ab0 0x7f6f700b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 --2- 172.21.15.67:0/2408793064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f7015ed10 0x7f6f700d40c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 -- 172.21.15.67:0/2408793064 >> 172.21.15.67:0/2408793064 conn(0x7f6f7006cbd0 msgr2=0x7f6f7006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 -- 172.21.15.67:0/2408793064 shutdown_connections 2024-04-17T07:29:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.261+0000 7f6f77c00640 1 -- 172.21.15.67:0/2408793064 wait complete. 2024-04-17T07:29:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.263+0000 7f6f77c00640 1 Processor -- start 2024-04-17T07:29:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.263+0000 7f6f77c00640 1 -- start start 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f700cf640 0x7f6f700cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 0x7f6f700b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f700d4600 con 0x7f6f700d2ab0 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6f700d25b0 con 0x7f6f700cf640 2024-04-17T07:29:57.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f77c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f700d4780 con 0x7f6f7015ed10 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38772/0 (socket says 172.21.15.67:38772) 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 -- 172.21.15.67:0/304742715 learned_addr learned my addr 172.21.15.67:0/304742715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f75c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 0x7f6f700b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 -- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 msgr2=0x7f6f700b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 0x7f6f700b8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 -- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f700cf640 msgr2=0x7f6f700cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f700cf640 0x7f6f700cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f6fe00640 1 -- 172.21.15.67:0/304742715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f700cc170 con 0x7f6f700d2ab0 2024-04-17T07:29:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.264+0000 7f6f75c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 0x7f6f700b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f6fe00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009870 tx=0x7f6f60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6002f7e0 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f60046070 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f700cc450 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60041640 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.265+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f700cca10 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.266+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f70068b90 con 0x7f6f700d2ab0 2024-04-17T07:29:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.268+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f6f60040070 con 0x7f6f700d2ab0 2024-04-17T07:29:57.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.269+0000 7f6f6cc00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 0x7f6f5406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.269+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f6f600ae5b0 con 0x7f6f700d2ab0 2024-04-17T07:29:57.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.270+0000 7f6f75200640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 0x7f6f5406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.271+0000 7f6f75200640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 0x7f6f5406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000f60 tx=0x7f6f640079d0 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.274+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6f60033030 con 0x7f6f700d2ab0 2024-04-17T07:29:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.481+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f70161ac0 con 0x7f6f700d2ab0 2024-04-17T07:29:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.482+0000 7f6f6cc00640 1 -- 172.21.15.67:0/304742715 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+13989 (secure 0 0 0) 0x7f6f6007c800 con 0x7f6f700d2ab0 2024-04-17T07:29:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:29:57.483 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":89,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:29:54.985138+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/2437593714":"2024-04-18T07:29:51.967080+0000","172.21.15.187:0/939506877":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6832/2378630274":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/139946814":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/2814172476":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/3268338662":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1164934579":"2024-04-18T07:29:47.783876+0000","172.21.15.187:6833/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6832/1436704971":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1567626846":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/831200072":"2024-04-18T07:29:41.466074+0000","172.21.15.187:6834/2437593714":"2024-04-18T07:29:51.967080+0000","172.21.15.187:0/3826805311":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/12378685":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1955748639":"2024-04-18T07:29:41.466074+0000","172.21.15.187:0/1146224552":"2024-04-18T07:29:47.783876+0000","172.21.15.187:0/1795500056":"2024-04-18T07:29:41.466074+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:29:57.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 msgr2=0x7f6f5406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 0x7f6f5406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f64000f60 tx=0x7f6f640079d0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 msgr2=0x7f6f70159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009870 tx=0x7f6f60004730 comp rx=0 tx=0).stop 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 shutdown_connections 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f7015ed10 0x7f6f700b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f700d2ab0 0x7f6f70159070 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6f5406ca10 0x7f6f5406eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 --2- 172.21.15.67:0/304742715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f700cf640 0x7f6f700cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 >> 172.21.15.67:0/304742715 conn(0x7f6f7006cbd0 msgr2=0x7f6f700ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 shutdown_connections 2024-04-17T07:29:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.485+0000 7f6f77c00640 1 -- 172.21.15.67:0/304742715 wait complete. 2024-04-17T07:29:57.503 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:29:57.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:29:57.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.729+0000 7f628fe00640 1 Processor -- start 2024-04-17T07:29:57.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.730+0000 7f628fe00640 1 -- start start 2024-04-17T07:29:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.731+0000 7f628fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f629015ec20 0x7f62900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.731+0000 7f628fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62900d4f70 con 0x7f62900b3b50 2024-04-17T07:29:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.731+0000 7f628fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62900d5250 con 0x7f62900cfcd0 2024-04-17T07:29:57.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.731+0000 7f628fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f62900cf4d0 con 0x7f629015ec20 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.732+0000 7f628e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62900b3b50 0x7f629006ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53802/0 (socket says 172.21.15.67:53802) 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.732+0000 7f628e000640 1 -- 172.21.15.67:0/694240206 learned_addr learned my addr 172.21.15.67:0/694240206 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.732+0000 7f628ea00640 1 --2- 172.21.15.67:0/694240206 >> v2:172.21.15.187:3300/0 conn(0x7f629015ec20 0x7f62900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.732+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047877074 0 0) 0x7f62900d4f70 con 0x7f62900b3b50 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.732+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6274003520 con 0x7f62900b3b50 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1085489746 0 0) 0x7f6274003520 con 0x7f62900b3b50 2024-04-17T07:29:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 >> v2:172.21.15.187:3300/0 conn(0x7f629015ec20 msgr2=0x7f62900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 --2- 172.21.15.67:0/694240206 >> v2:172.21.15.187:3300/0 conn(0x7f629015ec20 0x7f62900d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 >> v1:172.21.15.187:6789/0 conn(0x7f62900cfcd0 legacy=0x7f62900d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62900cf7b0 con 0x7f62900b3b50 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f627c0017f0 con 0x7f62900b3b50 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 >> v1:172.21.15.67:6789/0 conn(0x7f62900b3b50 legacy=0x7f629006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 0x7f6274003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6274004800 0x7f6274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62740052e0 0x7f627400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f62900cf4d0 con 0x7f6274003a00 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f627400c1e0 con 0x7f62740052e0 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f627400c3e0 con 0x7f6274004800 2024-04-17T07:29:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628ea00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 0x7f6274003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.733+0000 7f628e000640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6274004800 0x7f6274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628ea00640 1 -- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6274004800 msgr2=0x7f6274004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628ea00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6274004800 0x7f6274004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628f400640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62740052e0 0x7f627400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628ea00640 1 -- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62740052e0 msgr2=0x7f627400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628ea00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62740052e0 0x7f627400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628f400640 1 -- 172.21.15.67:0/694240206 reap_dead start 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.734+0000 7f628ea00640 1 -- 172.21.15.67:0/694240206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6274003520 con 0x7f6274003a00 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.735+0000 7f628ea00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 0x7f6274003e20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6280009fc0 tx=0x7f6274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cad1cea3fda411 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.735+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6280039070 con 0x7f6274003a00 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.735+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f628002cc20 con 0x7f6274003a00 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.735+0000 7f628d600640 1 -- 172.21.15.67:0/694240206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6280009c10 con 0x7f6274003a00 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 -- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 msgr2=0x7f6274003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 0x7f6274003e20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6280009fc0 tx=0x7f6274005830 comp rx=0 tx=0).stop 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 -- 172.21.15.67:0/694240206 shutdown_connections 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 --2- 172.21.15.67:0/694240206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274003a00 0x7f6274003e20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 -- 172.21.15.67:0/694240206 >> 172.21.15.67:0/694240206 conn(0x7f629008bff0 msgr2=0x7f62900d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 -- 172.21.15.67:0/694240206 shutdown_connections 2024-04-17T07:29:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.736+0000 7f628fe00640 1 -- 172.21.15.67:0/694240206 wait complete. 2024-04-17T07:29:57.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.738+0000 7f628fe00640 1 Processor -- start 2024-04-17T07:29:57.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.738+0000 7f628fe00640 1 -- start start 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6274003a00 0x7f62901f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6290160620 0x7f6290157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f62900d0300 con 0x7f6290156ba0 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f62900d0180 con 0x7f6274003a00 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f62900d0480 con 0x7f6290160620 2024-04-17T07:29:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.739+0000 7f628e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38792/0 (socket says 172.21.15.67:38792) 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 -- 172.21.15.67:0/694502433 learned_addr learned my addr 172.21.15.67:0/694502433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628f400640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6290160620 0x7f6290157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628ea00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6274003a00 0x7f62901f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 -- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6290160620 msgr2=0x7f6290157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6290160620 0x7f6290157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 -- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6274003a00 msgr2=0x7f62901f48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6274003a00 0x7f62901f48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 -- 172.21.15.67:0/694502433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6290158620 con 0x7f6290156ba0 2024-04-17T07:29:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.740+0000 7f628e000640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f627c00e780 tx=0x7f627c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.741+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0070d0 con 0x7f6290156ba0 2024-04-17T07:29:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.741+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f627c018070 con 0x7f6290156ba0 2024-04-17T07:29:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.741+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0074d0 con 0x7f6290156ba0 2024-04-17T07:29:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.741+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6290158900 con 0x7f6290156ba0 2024-04-17T07:29:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.741+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6290158ec0 con 0x7f6290156ba0 2024-04-17T07:29:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.742+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6290068a10 con 0x7f6290156ba0 2024-04-17T07:29:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.743+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f627c0057a0 con 0x7f6290156ba0 2024-04-17T07:29:57.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.744+0000 7f6286a00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 0x7f627006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:57.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.744+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f627c08bb30 con 0x7f6290156ba0 2024-04-17T07:29:57.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.745+0000 7f628ea00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 0x7f627006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:57.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.746+0000 7f628ea00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 0x7f627006eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6280036000 tx=0x7f628002e000 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.749+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f627c059e90 con 0x7f6290156ba0 2024-04-17T07:29:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.954+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6290160a00 con 0x7f6290156ba0 2024-04-17T07:29:57.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.954+0000 7f6286a00640 1 -- 172.21.15.67:0/694502433 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f627c059790 con 0x7f6290156ba0 2024-04-17T07:29:57.955 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:29:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 msgr2=0x7f627006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 0x7f627006eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6280036000 tx=0x7f628002e000 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 msgr2=0x7f62901600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f627c00e780 tx=0x7f627c00ec40 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 shutdown_connections 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6290160620 0x7f6290157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6290156ba0 0x7f62901600e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f627006ca10 0x7f627006eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 --2- 172.21.15.67:0/694502433 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6274003a00 0x7f62901f48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 >> 172.21.15.67:0/694502433 conn(0x7f629008bff0 msgr2=0x7f62900b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.957+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 shutdown_connections 2024-04-17T07:29:57.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:57.958+0000 7f628fe00640 1 -- 172.21.15.67:0/694502433 wait complete. 2024-04-17T07:29:57.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-17T07:29:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.204+0000 7f4096a00640 1 Processor -- start 2024-04-17T07:29:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.205+0000 7f4096a00640 1 -- start start 2024-04-17T07:29:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f409006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4090141ed0 0x7f4090142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40900cd4a0 con 0x7f409006b8c0 2024-04-17T07:29:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f409015ce30 con 0x7f4090141ed0 2024-04-17T07:29:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40901607f0 con 0x7f4090160280 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4090141ed0 0x7f4090142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4090141ed0 0x7f4090142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38796/0 (socket says 172.21.15.67:38796) 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.206+0000 7f4096000640 1 -- 172.21.15.67:0/885234084 learned_addr learned my addr 172.21.15.67:0/885234084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.207+0000 7f4094c00640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f409006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.207+0000 7f4096000640 1 -- 172.21.15.67:0/885234084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 msgr2=0x7f409006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.207+0000 7f4096000640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f409006bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.207+0000 7f4096000640 1 -- 172.21.15.67:0/885234084 >> v1:172.21.15.187:6789/0 conn(0x7f409006b8c0 legacy=0x7f40900cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.207+0000 7f4096000640 1 -- 172.21.15.67:0/885234084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4090160ad0 con 0x7f4090141ed0 2024-04-17T07:29:58.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.208+0000 7f4096000640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4090141ed0 0x7f4090142360 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997497ef8575e2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:58.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f408fe00640 1 -- 172.21.15.67:0/885234084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408802cb00 con 0x7f4090141ed0 2024-04-17T07:29:58.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f408fe00640 1 -- 172.21.15.67:0/885234084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408802cca0 con 0x7f4090141ed0 2024-04-17T07:29:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f408fe00640 1 -- 172.21.15.67:0/885234084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088034610 con 0x7f4090141ed0 2024-04-17T07:29:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 -- 172.21.15.67:0/885234084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4090141ed0 msgr2=0x7f4090142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4090141ed0 0x7f4090142360 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4088009e50 tx=0x7f408802bf60 comp rx=0 tx=0).stop 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 -- 172.21.15.67:0/885234084 shutdown_connections 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4090141ed0 0x7f4090142360 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 --2- 172.21.15.67:0/885234084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f409006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 -- 172.21.15.67:0/885234084 >> 172.21.15.67:0/885234084 conn(0x7f4090067d10 msgr2=0x7f40900b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 -- 172.21.15.67:0/885234084 shutdown_connections 2024-04-17T07:29:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.209+0000 7f4096a00640 1 -- 172.21.15.67:0/885234084 wait complete. 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.211+0000 7f4096a00640 1 Processor -- start 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.211+0000 7f4096a00640 1 -- start start 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 0x7f4090145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f40901430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4090161090 con 0x7f409006b8c0 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f409006c1e0 con 0x7f4090141ed0 2024-04-17T07:29:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4096a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f409013f9b0 con 0x7f4090160280 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.212+0000 7f4095600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38806/0 (socket says 172.21.15.67:38806) 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 -- 172.21.15.67:0/3860038380 learned_addr learned my addr 172.21.15.67:0/3860038380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4096000640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f40901430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4094c00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 0x7f4090145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 -- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 msgr2=0x7f40901430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f40901430e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 -- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 msgr2=0x7f4090145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 0x7f4090145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4095600640 1 -- 172.21.15.67:0/3860038380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409015dfe0 con 0x7f409006b8c0 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.213+0000 7f4094c00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 0x7f4090145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f4095600640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f408400c400 tx=0x7f408400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:58.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40840052d0 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4084005490 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084007260 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409015e2c0 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.214+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409015e880 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.215+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4090068a10 con 0x7f409006b8c0 2024-04-17T07:29:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.216+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f4084007400 con 0x7f409006b8c0 2024-04-17T07:29:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.217+0000 7f408d600640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 0x7f407006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.217+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f408408b2d0 con 0x7f409006b8c0 2024-04-17T07:29:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.218+0000 7f4094c00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 0x7f407006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.219+0000 7f4094c00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 0x7f407006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4090141b90 tx=0x7f407c008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.222+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f40840595a0 con 0x7f409006b8c0 2024-04-17T07:29:58.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:29:58.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:29:58.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:58.501+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7f40900ccf20 con 0x7f409006b8c0 2024-04-17T07:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.017+0000 7f408d600640 1 -- 172.21.15.67:0/3860038380 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v602) ==== 169+0+0 (secure 0 0 0) 0x7f4084058ea0 con 0x7f409006b8c0 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.020+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 msgr2=0x7f407006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.020+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 0x7f407006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4090141b90 tx=0x7f407c008040 comp rx=0 tx=0).stop 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.020+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 msgr2=0x7f409013e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.020+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f408400c400 tx=0x7f408400c8c0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 shutdown_connections 2024-04-17T07:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4090160280 0x7f40901430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f407006ca10 0x7f407006eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4090141ed0 0x7f4090145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 --2- 172.21.15.67:0/3860038380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f409006b8c0 0x7f409013e960 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 >> 172.21.15.67:0/3860038380 conn(0x7f4090067d10 msgr2=0x7f4090074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 shutdown_connections 2024-04-17T07:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.021+0000 7f4096a00640 1 -- 172.21.15.67:0/3860038380 wait complete. 2024-04-17T07:29:59.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:29:59.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.270+0000 7f4c57200640 1 Processor -- start 2024-04-17T07:29:59.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.271+0000 7f4c57200640 1 -- start start 2024-04-17T07:29:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c581626b0 0x7f4c580cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c57200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c580d2d40 0x7f4c580d31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c57200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c58162c20 con 0x7f4c5815eb00 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c57200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4c5806d430 con 0x7f4c580d2d40 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c58162f00 con 0x7f4c581626b0 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c55e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c5815eb00 0x7f4c580d2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53816/0 (socket says 172.21.15.67:53816) 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.272+0000 7f4c55e00640 1 -- 172.21.15.67:0/1340661877 learned_addr learned my addr 172.21.15.67:0/1340661877 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c56800640 1 --2- 172.21.15.67:0/1340661877 >> v2:172.21.15.67:3300/0 conn(0x7f4c580d2d40 0x7f4c580d31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c55400640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c581626b0 0x7f4c580cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c54a00640 1 -- 172.21.15.67:0/1340661877 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2049709468 0 0) 0x7f4c58162c20 con 0x7f4c5815eb00 2024-04-17T07:29:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c56800640 1 -- 172.21.15.67:0/1340661877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c581626b0 msgr2=0x7f4c580cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:29:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c56800640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c581626b0 0x7f4c580cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c56800640 1 -- 172.21.15.67:0/1340661877 >> v1:172.21.15.67:6789/0 conn(0x7f4c5815eb00 legacy=0x7f4c580d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.273+0000 7f4c56800640 1 -- 172.21.15.67:0/1340661877 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c581631e0 con 0x7f4c580d2d40 2024-04-17T07:29:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.274+0000 7f4c56800640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c580d2d40 0x7f4c580d31d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4c500097a0 tx=0x7f4c5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38c3b7f68fedad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c54a00640 1 -- 172.21.15.67:0/1340661877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50035020 con 0x7f4c580d2d40 2024-04-17T07:29:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c54a00640 1 -- 172.21.15.67:0/1340661877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c50031830 con 0x7f4c580d2d40 2024-04-17T07:29:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c54a00640 1 -- 172.21.15.67:0/1340661877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50031c00 con 0x7f4c580d2d40 2024-04-17T07:29:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 -- 172.21.15.67:0/1340661877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c580d2d40 msgr2=0x7f4c580d31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c580d2d40 0x7f4c580d31d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4c500097a0 tx=0x7f4c5002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 -- 172.21.15.67:0/1340661877 shutdown_connections 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c580d2d40 0x7f4c580d31d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 --2- 172.21.15.67:0/1340661877 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c581626b0 0x7f4c580cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 -- 172.21.15.67:0/1340661877 >> 172.21.15.67:0/1340661877 conn(0x7f4c5809da70 msgr2=0x7f4c5808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.275+0000 7f4c57200640 1 -- 172.21.15.67:0/1340661877 shutdown_connections 2024-04-17T07:29:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.276+0000 7f4c57200640 1 -- 172.21.15.67:0/1340661877 wait complete. 2024-04-17T07:29:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.277+0000 7f4c57200640 1 Processor -- start 2024-04-17T07:29:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.277+0000 7f4c57200640 1 -- start start 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 0x7f4c581577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c581626b0 0x7f4c58160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5806d8c0 con 0x7f4c5815eb00 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5815eee0 con 0x7f4c581626b0 2024-04-17T07:29:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.278+0000 7f4c57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c5806da40 con 0x7f4c580d2d40 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38822/0 (socket says 172.21.15.67:38822) 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 -- 172.21.15.67:0/4276823373 learned_addr learned my addr 172.21.15.67:0/4276823373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55e00640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 0x7f4c581577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c56800640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c581626b0 0x7f4c58160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 -- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 msgr2=0x7f4c581577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 0x7f4c581577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 -- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c581626b0 msgr2=0x7f4c58160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c581626b0 0x7f4c58160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 -- 172.21.15.67:0/4276823373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c580c83c0 con 0x7f4c5815eb00 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55e00640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 0x7f4c581577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:29:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.279+0000 7f4c55400640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4c40007e40 tx=0x7f4c4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.280+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40014620 con 0x7f4c5815eb00 2024-04-17T07:29:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.280+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c40019070 con 0x7f4c5815eb00 2024-04-17T07:29:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.280+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40014a00 con 0x7f4c5815eb00 2024-04-17T07:29:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.280+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c580c86a0 con 0x7f4c5815eb00 2024-04-17T07:29:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.280+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c58160340 con 0x7f4c5815eb00 2024-04-17T07:29:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.281+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c58068a10 con 0x7f4c5815eb00 2024-04-17T07:29:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.282+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f4c40014ba0 con 0x7f4c5815eb00 2024-04-17T07:29:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.283+0000 7f4c4e000640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 0x7f4c3406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:29:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.283+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7f4c4008c510 con 0x7f4c5815eb00 2024-04-17T07:29:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.284+0000 7f4c55e00640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 0x7f4c3406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:29:59.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.285+0000 7f4c55e00640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 0x7f4c3406eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c48000f60 tx=0x7f4c48009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:29:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.288+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4c4005a870 con 0x7f4c5815eb00 2024-04-17T07:29:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.493+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4c5815b340 con 0x7f4c5815eb00 2024-04-17T07:29:59.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.500+0000 7f4c4e000640 1 -- 172.21.15.67:0/4276823373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f4c4005a170 con 0x7f4c5815eb00 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.503+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 msgr2=0x7f4c3406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.503+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 0x7f4c3406eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c48000f60 tx=0x7f4c48009040 comp rx=0 tx=0).stop 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.503+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 msgr2=0x7f4c580ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.503+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4c40007e40 tx=0x7f4c4000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 shutdown_connections 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f4c3406ca10 0x7f4c3406eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c581626b0 0x7f4c58160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c5815eb00 0x7f4c580ccdf0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 --2- 172.21.15.67:0/4276823373 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c580d2d40 0x7f4c581577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:29:59.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 >> 172.21.15.67:0/4276823373 conn(0x7f4c5809da70 msgr2=0x7f4c580b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:29:59.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 shutdown_connections 2024-04-17T07:29:59.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:29:59.504+0000 7f4c57200640 1 -- 172.21.15.67:0/4276823373 wait complete. 2024-04-17T07:29:59.525 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:29:59.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:59.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:29:59.782 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:29:59.782 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:59.782 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:59.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:59.843 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:59.844 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:59.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:29:59.902 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:29:59.902 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:29:59.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:29:59.961 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:29:59.961 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:00.017 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:30:00.018 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:30:00.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.247+0000 7fa81be00640 1 Processor -- start 2024-04-17T07:30:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.247+0000 7fa81be00640 1 -- start start 2024-04-17T07:30:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.249+0000 7fa81be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa81c15c5a0 0x7fa81c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.249+0000 7fa81be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.249+0000 7fa81be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c15f440 con 0x7fa81c0b46c0 2024-04-17T07:30:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.249+0000 7fa81be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa81c0af0a0 con 0x7fa81c15c5a0 2024-04-17T07:30:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.249+0000 7fa81be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa81c15f720 con 0x7fa81c0b4210 2024-04-17T07:30:00.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.250+0000 7fa813e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.250+0000 7fa813e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:45682/0 (socket says 172.21.15.67:45682) 2024-04-17T07:30:00.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.250+0000 7fa813e00640 1 -- 172.21.15.67:0/1691203064 learned_addr learned my addr 172.21.15.67:0/1691203064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:00.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.250+0000 7fa81aa00640 1 --2- 172.21.15.67:0/1691203064 >> v2:172.21.15.187:3300/0 conn(0x7fa81c15c5a0 0x7fa81c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.251+0000 7fa813e00640 1 -- 172.21.15.67:0/1691203064 >> v2:172.21.15.187:3300/0 conn(0x7fa81c15c5a0 msgr2=0x7fa81c15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:00.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.251+0000 7fa813e00640 1 --2- 172.21.15.67:0/1691203064 >> v2:172.21.15.187:3300/0 conn(0x7fa81c15c5a0 0x7fa81c15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.251+0000 7fa813e00640 1 -- 172.21.15.67:0/1691203064 >> v1:172.21.15.187:6789/0 conn(0x7fa81c0b46c0 legacy=0x7fa81c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.251+0000 7fa813e00640 1 -- 172.21.15.67:0/1691203064 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c15fa00 con 0x7fa81c0b4210 2024-04-17T07:30:00.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.252+0000 7fa813e00640 1 --2- 172.21.15.67:0/1691203064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa8040098a0 tx=0x7fa80402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e013ffa1019aff86 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:00.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81a000640 1 -- 172.21.15.67:0/1691203064 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8040317c0 con 0x7fa81c0b4210 2024-04-17T07:30:00.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81a000640 1 -- 172.21.15.67:0/1691203064 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa804037070 con 0x7fa81c0b4210 2024-04-17T07:30:00.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81a000640 1 -- 172.21.15.67:0/1691203064 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804031b80 con 0x7fa81c0b4210 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 -- 172.21.15.67:0/1691203064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 msgr2=0x7fa81c0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 --2- 172.21.15.67:0/1691203064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa8040098a0 tx=0x7fa80402b860 comp rx=0 tx=0).stop 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 -- 172.21.15.67:0/1691203064 shutdown_connections 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 --2- 172.21.15.67:0/1691203064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c0af440 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 --2- 172.21.15.67:0/1691203064 >> v2:172.21.15.187:3300/0 conn(0x7fa81c15c5a0 0x7fa81c15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.253+0000 7fa81be00640 1 -- 172.21.15.67:0/1691203064 >> 172.21.15.67:0/1691203064 conn(0x7fa81c067d10 msgr2=0x7fa81c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.254+0000 7fa81be00640 1 -- 172.21.15.67:0/1691203064 shutdown_connections 2024-04-17T07:30:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.254+0000 7fa81be00640 1 -- 172.21.15.67:0/1691203064 wait complete. 2024-04-17T07:30:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.255+0000 7fa81be00640 1 Processor -- start 2024-04-17T07:30:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.255+0000 7fa81be00640 1 -- start start 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c1f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 0x7fa81c159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c0adc30 con 0x7fa81c15c5a0 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c0adab0 con 0x7fa81c1469c0 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.256+0000 7fa81be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa81c06e670 con 0x7fa81c0b4210 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38836/0 (socket says 172.21.15.67:38836) 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 -- 172.21.15.67:0/1153876187 learned_addr learned my addr 172.21.15.67:0/1153876187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa81b400640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 0x7fa81c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa81aa00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c1f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 -- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 msgr2=0x7fa81c1f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c1f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 -- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 msgr2=0x7fa81c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 0x7fa81c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa813e00640 1 -- 172.21.15.67:0/1153876187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c149000 con 0x7fa81c15c5a0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.257+0000 7fa81b400640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 0x7fa81c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa813e00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa80402b8a0 tx=0x7fa8040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804032650 con 0x7fa81c15c5a0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8040327f0 con 0x7fa81c15c5a0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa804032be0 con 0x7fa81c15c5a0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c1f4280 con 0x7fa81c15c5a0 2024-04-17T07:30:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.258+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c147220 con 0x7fa81c15c5a0 2024-04-17T07:30:00.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.260+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c068a10 con 0x7fa81c15c5a0 2024-04-17T07:30:00.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.261+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fa8040453f0 con 0x7fa81c15c5a0 2024-04-17T07:30:00.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.262+0000 7fa812a00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 0x7fa7f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:00.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.262+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5631+0+0 (secure 0 0 0) 0x7fa8040adf60 con 0x7fa81c15c5a0 2024-04-17T07:30:00.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.263+0000 7fa81aa00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 0x7fa7f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:00.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.264+0000 7fa81aa00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 0x7fa7f806eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009a30 tx=0x7fa80c008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:00.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.267+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa804040070 con 0x7fa81c15c5a0 2024-04-17T07:30:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.515+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa81c1491d0 con 0x7fa81c15c5a0 2024-04-17T07:30:00.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.516+0000 7fa812a00640 1 -- 172.21.15.67:0/1153876187 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+3875 (secure 0 0 0) 0x7fa80407bb50 con 0x7fa81c15c5a0 2024-04-17T07:30:00.516 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:00.517 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5626,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2819209820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2819209820},{"type":"v1","addr":"172.21.15.67:6835","nonce":2819209820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5638,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2139922413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2139922413},{"type":"v1","addr":"172.21.15.187:6837","nonce":2139922413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5887,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1649524828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1649524828},{"type":"v1","addr":"172.21.15.187:6835","nonce":1649524828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":6429,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1261188176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1261188176},{"type":"v1","addr":"172.21.15.67:6834","nonce":1261188176}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[]} 2024-04-17T07:30:00.517 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 37 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 msgr2=0x7fa7f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 0x7fa7f806eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009a30 tx=0x7fa80c008040 comp rx=0 tx=0).stop 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 msgr2=0x7fa81c146480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa80402b8a0 tx=0x7fa8040046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 shutdown_connections 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa7f806ca10 0x7fa7f806eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c1469c0 0x7fa81c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c15c5a0 0x7fa81c146480 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 --2- 172.21.15.67:0/1153876187 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c0b4210 0x7fa81c1f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:00.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 >> 172.21.15.67:0/1153876187 conn(0x7fa81c067d10 msgr2=0x7fa81c0b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:00.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 shutdown_connections 2024-04-17T07:30:00.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:00.519+0000 7fa81be00640 1 -- 172.21.15.67:0/1153876187 wait complete. 2024-04-17T07:30:00.540 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:30:00.541 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:30:00.541 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:30:00.541 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:00.541 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:30:00.541 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:30:00.541 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:30:00.541 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:00.542 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:30:00.542 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:30:00.542 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:30:00.542 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:00.542 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:30:00.542 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:30:00.542 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:30:00.542 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:00.558 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:30:00.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:30:00.559 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:30:00.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:30:00.574 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:30:00.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:30:00.606 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:30:00.606 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:00.611 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:30:00.611 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:00.620 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:30:00.621 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:00.768 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:30:00.768 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:30:00.807 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:30:05.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-17T07:30:06.574 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:30:06.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:30:06.582 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:30:06.582 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:06.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:30:06.770 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:30:06.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.818+0000 7fe34ec00640 1 Processor -- start 2024-04-17T07:30:06.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.818+0000 7fe34ec00640 1 -- start start 2024-04-17T07:30:06.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.819+0000 7fe34ec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe34809aa60 0x7fe34809ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:06.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.819+0000 7fe34ec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3481613c0 con 0x7fe3480d0280 2024-04-17T07:30:06.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.819+0000 7fe34ec00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34809b380 con 0x7fe3480cead0 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.819+0000 7fe34ec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe34815d350 con 0x7fe34809aa60 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe34809aa60 0x7fe34809ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3480d0280 0x7fe3481605a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45136/0 (socket says 172.21.15.67:45136) 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347400640 1 -- 172.21.15.67:0/4031327673 learned_addr learned my addr 172.21.15.67:0/4031327673 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347e00640 1 -- 172.21.15.67:0/4031327673 >> v1:172.21.15.187:6789/0 conn(0x7fe3480cead0 legacy=0x7fe348160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347e00640 1 -- 172.21.15.67:0/4031327673 >> v1:172.21.15.67:6789/0 conn(0x7fe3480d0280 legacy=0x7fe3481605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:06.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.820+0000 7fe347e00640 1 -- 172.21.15.67:0/4031327673 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3480d0e60 con 0x7fe34809aa60 2024-04-17T07:30:06.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.821+0000 7fe347e00640 1 --2- 172.21.15.67:0/4031327673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe34809aa60 0x7fe34809ae40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe338009dc0 tx=0x7fe33802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5f9236b17481c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:06.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.822+0000 7fe346a00640 1 -- 172.21.15.67:0/4031327673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338039070 con 0x7fe34809aa60 2024-04-17T07:30:06.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.822+0000 7fe346a00640 1 -- 172.21.15.67:0/4031327673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33802cc40 con 0x7fe34809aa60 2024-04-17T07:30:06.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.822+0000 7fe346a00640 1 -- 172.21.15.67:0/4031327673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338034440 con 0x7fe34809aa60 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.822+0000 7fe34ec00640 1 -- 172.21.15.67:0/4031327673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe34809aa60 msgr2=0x7fe34809ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.822+0000 7fe34ec00640 1 --2- 172.21.15.67:0/4031327673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe34809aa60 0x7fe34809ae40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe338009dc0 tx=0x7fe33802be90 comp rx=0 tx=0).stop 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.823+0000 7fe34ec00640 1 -- 172.21.15.67:0/4031327673 shutdown_connections 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.823+0000 7fe34ec00640 1 --2- 172.21.15.67:0/4031327673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe34809aa60 0x7fe34809ae40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.823+0000 7fe34ec00640 1 -- 172.21.15.67:0/4031327673 >> 172.21.15.67:0/4031327673 conn(0x7fe3480b41a0 msgr2=0x7fe3480b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.823+0000 7fe34ec00640 1 -- 172.21.15.67:0/4031327673 shutdown_connections 2024-04-17T07:30:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.823+0000 7fe34ec00640 1 -- 172.21.15.67:0/4031327673 wait complete. 2024-04-17T07:30:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.824+0000 7fe34ec00640 1 Processor -- start 2024-04-17T07:30:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.825+0000 7fe34ec00640 1 -- start start 2024-04-17T07:30:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.825+0000 7fe34ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34809aa60 0x7fe3480b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.825+0000 7fe34ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.825+0000 7fe34ec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3480d0280 0x7fe34815da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe34ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3480d1420 con 0x7fe3480cead0 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe34ec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe3480cd630 con 0x7fe3480d0280 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe34ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe3480d1700 con 0x7fe34809aa60 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58898/0 (socket says 172.21.15.67:58898) 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 -- 172.21.15.67:0/2450780065 learned_addr learned my addr 172.21.15.67:0/2450780065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347e00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34809aa60 0x7fe3480b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe34cc00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3480d0280 0x7fe34815da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 -- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34809aa60 msgr2=0x7fe3480b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34809aa60 0x7fe3480b7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 -- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3480d0280 msgr2=0x7fe34815da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3480d0280 0x7fe34815da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.826+0000 7fe347400640 1 -- 172.21.15.67:0/2450780065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3480b7890 con 0x7fe3480cead0 2024-04-17T07:30:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe347400640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c3c0 tx=0x7fe33000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330017070 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe330005ce0 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3300072f0 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3480b7b70 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.827+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3480b8130 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.828+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe348068ca0 con 0x7fe3480cead0 2024-04-17T07:30:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.829+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fe33001a070 con 0x7fe3480cead0 2024-04-17T07:30:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.830+0000 7fe33fe00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 0x7fe32406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.830+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5631+0+0 (secure 0 0 0) 0x7fe33008b090 con 0x7fe3480cead0 2024-04-17T07:30:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.831+0000 7fe347e00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 0x7fe32406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:06.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.832+0000 7fe347e00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 0x7fe32406eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe338036000 tx=0x7fe338030000 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:06.837+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe330058c80 con 0x7fe3480cead0 2024-04-17T07:30:07.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:07.056+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe3480b5040 con 0x7fe3480cead0 2024-04-17T07:30:08.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.551+0000 7fe33fe00640 1 -- 172.21.15.67:0/2450780065 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) ==== 97+0+0 (secure 0 0 0) 0x7fe33005cf80 con 0x7fe3480cead0 2024-04-17T07:30:08.551 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 msgr2=0x7fe32406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 0x7fe32406eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe338036000 tx=0x7fe338030000 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 msgr2=0x7fe34815d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe33000c3c0 tx=0x7fe33000c880 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 shutdown_connections 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe32406ca10 0x7fe32406eed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3480d0280 0x7fe34815da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3480cead0 0x7fe34815d4e0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.554+0000 7fe34ec00640 1 --2- 172.21.15.67:0/2450780065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34809aa60 0x7fe3480b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.555+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 >> 172.21.15.67:0/2450780065 conn(0x7fe3480b41a0 msgr2=0x7fe3480cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.555+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 shutdown_connections 2024-04-17T07:30:08.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.555+0000 7fe34ec00640 1 -- 172.21.15.67:0/2450780065 wait complete. 2024-04-17T07:30:08.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:30:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.782+0000 7f7d71a00640 1 Processor -- start 2024-04-17T07:30:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.782+0000 7f7d71a00640 1 -- start start 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.783+0000 7f7d71a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6c160e50 0x7f7d6c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.783+0000 7f7d71a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 0x7f7d6c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d71a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6c15cd40 con 0x7f7d6c1452f0 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d71a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6c072710 con 0x7f7d6c160e50 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d71a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6c15d670 con 0x7f7d6c0722a0 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6c160e50 0x7f7d6c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6c160e50 0x7f7d6c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58908/0 (socket says 172.21.15.67:58908) 2024-04-17T07:30:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 -- 172.21.15.67:0/3184387467 learned_addr learned my addr 172.21.15.67:0/3184387467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 -- 172.21.15.67:0/3184387467 >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 msgr2=0x7f7d6c15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d71000640 1 --2- 172.21.15.67:0/3184387467 >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 0x7f7d6c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 --2- 172.21.15.67:0/3184387467 >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 0x7f7d6c15db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 -- 172.21.15.67:0/3184387467 >> v1:172.21.15.187:6789/0 conn(0x7f7d6c1452f0 legacy=0x7f7d6c145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d6b400640 1 -- 172.21.15.67:0/3184387467 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c15d020 con 0x7f7d6c160e50 2024-04-17T07:30:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.784+0000 7f7d71000640 1 --2- 172.21.15.67:0/3184387467 >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 0x7f7d6c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.786+0000 7f7d6b400640 1 --2- 172.21.15.67:0/3184387467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c160e50 0x7f7d6c071d60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c072920 tx=0x7f7d5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad79309e28eba6cd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.786+0000 7f7d6aa00640 1 -- 172.21.15.67:0/3184387467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5802e070 con 0x7f7d6c160e50 2024-04-17T07:30:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.786+0000 7f7d6aa00640 1 -- 172.21.15.67:0/3184387467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58037070 con 0x7f7d6c160e50 2024-04-17T07:30:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.786+0000 7f7d6aa00640 1 -- 172.21.15.67:0/3184387467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58031a00 con 0x7f7d6c160e50 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 -- 172.21.15.67:0/3184387467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c160e50 msgr2=0x7f7d6c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 --2- 172.21.15.67:0/3184387467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c160e50 0x7f7d6c071d60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c072920 tx=0x7f7d5802b880 comp rx=0 tx=0).stop 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 -- 172.21.15.67:0/3184387467 shutdown_connections 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 --2- 172.21.15.67:0/3184387467 >> v2:172.21.15.187:3300/0 conn(0x7f7d6c0722a0 0x7f7d6c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 --2- 172.21.15.67:0/3184387467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c160e50 0x7f7d6c071d60 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 -- 172.21.15.67:0/3184387467 >> 172.21.15.67:0/3184387467 conn(0x7f7d6c0b5f70 msgr2=0x7f7d6c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 -- 172.21.15.67:0/3184387467 shutdown_connections 2024-04-17T07:30:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.787+0000 7f7d71a00640 1 -- 172.21.15.67:0/3184387467 wait complete. 2024-04-17T07:30:08.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.789+0000 7f7d71a00640 1 Processor -- start 2024-04-17T07:30:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.789+0000 7f7d71a00640 1 -- start start 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d6c0722a0 0x7f7d6c0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d6c160e50 0x7f7d6c0aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1456d0 con 0x7f7d6c1452f0 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c146180 con 0x7f7d6c160e50 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6c15e040 con 0x7f7d6c0722a0 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58920/0 (socket says 172.21.15.67:58920) 2024-04-17T07:30:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 -- 172.21.15.67:0/2043369696 learned_addr learned my addr 172.21.15.67:0/2043369696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6be00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d6c0722a0 0x7f7d6c0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d71000640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d6c160e50 0x7f7d6c0aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 -- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d6c0722a0 msgr2=0x7f7d6c0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d6c0722a0 0x7f7d6c0cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 -- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d6c160e50 msgr2=0x7f7d6c0aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d6c160e50 0x7f7d6c0aa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.790+0000 7f7d6b400640 1 -- 172.21.15.67:0/2043369696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c0ce340 con 0x7f7d6c1452f0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d6b400640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7d5802b8c0 tx=0x7f7d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58032880 con 0x7f7d6c1452f0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58032a40 con 0x7f7d6c1452f0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58032e30 con 0x7f7d6c1452f0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c0ac770 con 0x7f7d6c1452f0 2024-04-17T07:30:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.791+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c0acd30 con 0x7f7d6c1452f0 2024-04-17T07:30:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.792+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d6c068a10 con 0x7f7d6c1452f0 2024-04-17T07:30:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.793+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f7d5804c060 con 0x7f7d6c1452f0 2024-04-17T07:30:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.794+0000 7f7d63e00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 0x7f7d4c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.794+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 4857+0+0 (secure 0 0 0) 0x7f7d580ad920 con 0x7f7d6c1452f0 2024-04-17T07:30:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.795+0000 7f7d6be00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 0x7f7d4c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.796+0000 7f7d6be00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 0x7f7d4c06ef00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0015d0 tx=0x7f7d5c009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:08.799+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7d6c0ce340 con 0x7f7d6c1452f0 2024-04-17T07:30:09.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.084+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7d6c071d60 con 0x7f7d6c1452f0 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.085+0000 7f7d63e00640 1 -- 172.21.15.67:0/2043369696 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7f7d58040070 con 0x7f7d6c1452f0 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:30:09.258 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:30:09.260 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:30:09.262 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:30:09.264 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.265 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.266 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:30:09.269 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:30:09.269 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:09.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:30:09.273 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:30:09.274 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:30:09.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.087+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 msgr2=0x7f7d4c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.087+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 0x7f7d4c06ef00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0015d0 tx=0x7f7d5c009040 comp rx=0 tx=0).stop 2024-04-17T07:30:09.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.087+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 msgr2=0x7f7d6c15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.087+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7d5802b8c0 tx=0x7f7d580046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 shutdown_connections 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7d4c06ca40 0x7f7d4c06ef00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d6c160e50 0x7f7d6c0aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6c1452f0 0x7f7d6c15bc10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 --2- 172.21.15.67:0/2043369696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d6c0722a0 0x7f7d6c0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 >> 172.21.15.67:0/2043369696 conn(0x7f7d6c0b5f70 msgr2=0x7f7d6c15fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 shutdown_connections 2024-04-17T07:30:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.088+0000 7f7d71a00640 1 -- 172.21.15.67:0/2043369696 wait complete. 2024-04-17T07:30:09.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.505+0000 7fccfb000640 1 Processor -- start 2024-04-17T07:30:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.506+0000 7fccfb000640 1 -- start start 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccfb000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fccf406c040 0x7fccf41664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccfb000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf406c450 con 0x7fccf415b570 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccfb000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf415bc80 con 0x7fccf406feb0 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccfb000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fccf415bf40 con 0x7fccf406c040 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccf9c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fccf415b570 0x7fccf406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45140/0 (socket says 172.21.15.67:45140) 2024-04-17T07:30:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.507+0000 7fccf9c00640 1 -- 172.21.15.67:0/3784167729 learned_addr learned my addr 172.21.15.67:0/3784167729 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccfa600640 1 --2- 172.21.15.67:0/3784167729 >> v2:172.21.15.187:3300/0 conn(0x7fccf406c040 0x7fccf41664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576925866 0 0) 0x7fccf406c450 con 0x7fccf415b570 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccdc003520 con 0x7fccf415b570 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118900719 0 0) 0x7fccdc003520 con 0x7fccf415b570 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 >> v2:172.21.15.187:3300/0 conn(0x7fccf406c040 msgr2=0x7fccf41664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 --2- 172.21.15.67:0/3784167729 >> v2:172.21.15.187:3300/0 conn(0x7fccf406c040 0x7fccf41664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 >> v1:172.21.15.187:6789/0 conn(0x7fccf406feb0 legacy=0x7fccf406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf415c220 con 0x7fccf415b570 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcce8001670 con 0x7fccf415b570 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 >> v1:172.21.15.67:6789/0 conn(0x7fccf415b570 legacy=0x7fccf406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc003a20 0x7fccdc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 0x7fccdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccdc005230 0x7fccdc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fccf415bf40 con 0x7fccdc004800 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc00e3a0 con 0x7fccdc003a20 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.508+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fccdc00e5a0 con 0x7fccdc005230 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 0x7fccdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9c00640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc003a20 0x7fccdc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 -- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccdc005230 msgr2=0x7fccdc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccdc005230 0x7fccdc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 -- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc003a20 msgr2=0x7fccdc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc003a20 0x7fccdc003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccf9200640 1 -- 172.21.15.67:0/3784167729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc003520 con 0x7fccdc004800 2024-04-17T07:30:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.509+0000 7fccfa600640 1 -- 172.21.15.67:0/3784167729 reap_dead start 2024-04-17T07:30:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.510+0000 7fccf9200640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 0x7fccdc004c80 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcce4001750 tx=0x7fccdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c5e9b11fcd8035 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.510+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4001370 con 0x7fccdc004800 2024-04-17T07:30:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce40052d0 con 0x7fccdc004800 2024-04-17T07:30:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccf3e00640 1 -- 172.21.15.67:0/3784167729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4003830 con 0x7fccdc004800 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 -- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 msgr2=0x7fccdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 0x7fccdc004c80 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcce4001750 tx=0x7fccdc005830 comp rx=0 tx=0).stop 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 -- 172.21.15.67:0/3784167729 shutdown_connections 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 --2- 172.21.15.67:0/3784167729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccdc004800 0x7fccdc004c80 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 -- 172.21.15.67:0/3784167729 >> 172.21.15.67:0/3784167729 conn(0x7fccf4067d10 msgr2=0x7fccf40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 -- 172.21.15.67:0/3784167729 shutdown_connections 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.511+0000 7fccfb000640 1 -- 172.21.15.67:0/3784167729 wait complete. 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.513+0000 7fccfb000640 1 Processor -- start 2024-04-17T07:30:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.513+0000 7fccfb000640 1 -- start start 2024-04-17T07:30:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc004800 0x7fccf40aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf415a060 0x7fccf415dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fccf40b47b0 con 0x7fccf4154e30 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fccf415b950 con 0x7fccdc004800 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.514+0000 7fccfb000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fccf40b4930 con 0x7fccf415a060 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58936/0 (socket says 172.21.15.67:58936) 2024-04-17T07:30:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 -- 172.21.15.67:0/2197246487 learned_addr learned my addr 172.21.15.67:0/2197246487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9c00640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc004800 0x7fccf40aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 -- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf415a060 msgr2=0x7fccf415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf415a060 0x7fccf415dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 -- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc004800 msgr2=0x7fccf40aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc004800 0x7fccf40aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 -- 172.21.15.67:0/2197246487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf40a9660 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.515+0000 7fccf9200640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcce4005040 tx=0x7fcce4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.516+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4041020 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.516+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce4005480 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.516+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf40a9940 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.516+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4007a30 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.516+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf415e280 con 0x7fccf4154e30 2024-04-17T07:30:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.517+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf4068a10 con 0x7fccf4154e30 2024-04-17T07:30:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.518+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fcce4044020 con 0x7fccf4154e30 2024-04-17T07:30:09.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.519+0000 7fccf1600640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 0x7fccd806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.519+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 4857+0+0 (secure 0 0 0) 0x7fcce40ad4e0 con 0x7fccf4154e30 2024-04-17T07:30:09.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.520+0000 7fccf9c00640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 0x7fccd806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.521+0000 7fccf9c00640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 0x7fccd806ee40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcce80017e0 tx=0x7fcce8001190 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.524+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcce404b030 con 0x7fccf4154e30 2024-04-17T07:30:09.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.728+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccf406f350 con 0x7fccf4154e30 2024-04-17T07:30:09.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.729+0000 7fccf1600640 1 -- 172.21.15.67:0/2197246487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+12843 (secure 0 0 0) 0x7fcce407b9a0 con 0x7fccf4154e30 2024-04-17T07:30:09.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:09.729 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:08.547386+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:09.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 msgr2=0x7fccd806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 0x7fccd806ee40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcce80017e0 tx=0x7fcce8001190 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 msgr2=0x7fccf414b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcce4005040 tx=0x7fcce4001120 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 shutdown_connections 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf415a060 0x7fccf415dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf4154e30 0x7fccf414b040 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fccd806c980 0x7fccd806ee40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 --2- 172.21.15.67:0/2197246487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccdc004800 0x7fccf40aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 >> 172.21.15.67:0/2197246487 conn(0x7fccf4067d10 msgr2=0x7fccf40ce360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 shutdown_connections 2024-04-17T07:30:09.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.732+0000 7fccfb000640 1 -- 172.21.15.67:0/2197246487 wait complete. 2024-04-17T07:30:09.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:30:09.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:30:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.973+0000 7f1e36400640 1 Processor -- start 2024-04-17T07:30:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.974+0000 7f1e36400640 1 -- start start 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.975+0000 7f1e36400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e30146060 0x7f1e30160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.975+0000 7f1e36400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e300b4ac0 0x7f1e300b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.975+0000 7f1e36400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e30160920 con 0x7f1e30145b00 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.975+0000 7f1e36400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e300b5450 con 0x7f1e30146060 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.975+0000 7f1e36400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1e30160c00 con 0x7f1e300b4ac0 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e35000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e30145b00 0x7f1e3015cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45150/0 (socket says 172.21.15.67:45150) 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e35000640 1 -- 172.21.15.67:0/1197571033 learned_addr learned my addr 172.21.15.67:0/1197571033 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2fe00640 1 --2- 172.21.15.67:0/1197571033 >> v2:172.21.15.67:3300/0 conn(0x7f1e30146060 0x7f1e30160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e35a00640 1 --2- 172.21.15.67:0/1197571033 >> v2:172.21.15.187:3300/0 conn(0x7f1e300b4ac0 0x7f1e300b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2f400640 1 -- 172.21.15.67:0/1197571033 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346641721 0 0) 0x7f1e30160920 con 0x7f1e30145b00 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2f400640 1 -- 172.21.15.67:0/1197571033 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e18003520 con 0x7f1e30145b00 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2fe00640 1 -- 172.21.15.67:0/1197571033 >> v2:172.21.15.187:3300/0 conn(0x7f1e300b4ac0 msgr2=0x7f1e300b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2fe00640 1 --2- 172.21.15.67:0/1197571033 >> v2:172.21.15.187:3300/0 conn(0x7f1e300b4ac0 0x7f1e300b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2fe00640 1 -- 172.21.15.67:0/1197571033 >> v1:172.21.15.67:6789/0 conn(0x7f1e30145b00 legacy=0x7f1e3015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.976+0000 7f1e2fe00640 1 -- 172.21.15.67:0/1197571033 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e30160ee0 con 0x7f1e30146060 2024-04-17T07:30:09.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e2fe00640 1 --2- 172.21.15.67:0/1197571033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e30160340 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1e300b61c0 tx=0x7f1e1c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3419def34c91c5a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e2f400640 1 -- 172.21.15.67:0/1197571033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0315a0 con 0x7f1e30146060 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e2f400640 1 -- 172.21.15.67:0/1197571033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c037070 con 0x7f1e30146060 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e2f400640 1 -- 172.21.15.67:0/1197571033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c031940 con 0x7f1e30146060 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e36400640 1 -- 172.21.15.67:0/1197571033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 msgr2=0x7f1e30160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.978+0000 7f1e36400640 1 --2- 172.21.15.67:0/1197571033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e30160340 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1e300b61c0 tx=0x7f1e1c02b840 comp rx=0 tx=0).stop 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 -- 172.21.15.67:0/1197571033 shutdown_connections 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 --2- 172.21.15.67:0/1197571033 >> v2:172.21.15.187:3300/0 conn(0x7f1e300b4ac0 0x7f1e300b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 --2- 172.21.15.67:0/1197571033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e30160340 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 -- 172.21.15.67:0/1197571033 >> 172.21.15.67:0/1197571033 conn(0x7f1e300b23c0 msgr2=0x7f1e300b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 -- 172.21.15.67:0/1197571033 shutdown_connections 2024-04-17T07:30:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.979+0000 7f1e36400640 1 -- 172.21.15.67:0/1197571033 wait complete. 2024-04-17T07:30:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.980+0000 7f1e36400640 1 Processor -- start 2024-04-17T07:30:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.980+0000 7f1e36400640 1 -- start start 2024-04-17T07:30:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 0x7f1e3013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e30145b00 0x7f1e30140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300cfba0 con 0x7f1e30146060 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300b3760 con 0x7f1e30145b00 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.981+0000 7f1e36400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e300b3c80 con 0x7f1e300b4ac0 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58942/0 (socket says 172.21.15.67:58942) 2024-04-17T07:30:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 -- 172.21.15.67:0/4240395020 learned_addr learned my addr 172.21.15.67:0/4240395020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35000640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 0x7f1e3013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e2fe00640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e30145b00 0x7f1e30140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 -- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 msgr2=0x7f1e3013d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 0x7f1e3013d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 -- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e30145b00 msgr2=0x7f1e30140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e30145b00 0x7f1e30140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 -- 172.21.15.67:0/4240395020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300ca2e0 con 0x7f1e30146060 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35000640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 0x7f1e3013d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.982+0000 7f1e35a00640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1e28007e40 tx=0x7f1e2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.983+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28014620 con 0x7f1e30146060 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.983+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e28019070 con 0x7f1e30146060 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.983+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28014a20 con 0x7f1e30146060 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.983+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e300ca5c0 con 0x7f1e30146060 2024-04-17T07:30:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.983+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e300cab80 con 0x7f1e30146060 2024-04-17T07:30:09.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.984+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e300cb070 con 0x7f1e30146060 2024-04-17T07:30:09.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.985+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f1e28014bc0 con 0x7f1e30146060 2024-04-17T07:30:09.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.986+0000 7f1e2cc00640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 0x7f1e1006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:09.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.987+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 4857+0+0 (secure 0 0 0) 0x7f1e2808c360 con 0x7f1e30146060 2024-04-17T07:30:09.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.987+0000 7f1e35000640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 0x7f1e1006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:09.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.988+0000 7f1e35000640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 0x7f1e1006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e24005880 tx=0x7f1e24009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:09.991+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e2805a930 con 0x7f1e30146060 2024-04-17T07:30:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.190+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e30068a10 con 0x7f1e30146060 2024-04-17T07:30:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.191+0000 7f1e2cc00640 1 -- 172.21.15.67:0/4240395020 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f1e2805a230 con 0x7f1e30146060 2024-04-17T07:30:10.190 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 msgr2=0x7f1e1006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 0x7f1e1006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e24005880 tx=0x7f1e24009040 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 msgr2=0x7f1e3015e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1e28007e40 tx=0x7f1e2800e4b0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 shutdown_connections 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e30146060 0x7f1e3015e2a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f1e1006ca10 0x7f1e1006eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e30145b00 0x7f1e30140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 --2- 172.21.15.67:0/4240395020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e300b4ac0 0x7f1e3013d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.193+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 >> 172.21.15.67:0/4240395020 conn(0x7f1e300b23c0 msgr2=0x7f1e30160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.194+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 shutdown_connections 2024-04-17T07:30:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.194+0000 7f1e36400640 1 -- 172.21.15.67:0/4240395020 wait complete. 2024-04-17T07:30:10.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:30:10.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.434+0000 7f913be00640 1 Processor -- start 2024-04-17T07:30:10.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.434+0000 7f913be00640 1 -- start start 2024-04-17T07:30:10.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.435+0000 7f913be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f913c15c5a0 0x7f913c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0b4210 0x7f913c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913c15f440 con 0x7f913c0b46c0 2024-04-17T07:30:10.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0af0a0 con 0x7f913c0b4210 2024-04-17T07:30:10.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f913c15f720 con 0x7f913c15c5a0 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f9133e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0b4210 0x7f913c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f9133e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0b4210 0x7f913c0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58954/0 (socket says 172.21.15.67:58954) 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f9133e00640 1 -- 172.21.15.67:0/4187501820 learned_addr learned my addr 172.21.15.67:0/4187501820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913b400640 1 --1- 172.21.15.67:0/4187501820 >> v1:172.21.15.67:6789/0 conn(0x7f913c0b46c0 0x7f913c071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45164/0 (socket says 172.21.15.67:45164) 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.436+0000 7f913aa00640 1 --2- 172.21.15.67:0/4187501820 >> v2:172.21.15.187:3300/0 conn(0x7f913c15c5a0 0x7f913c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.437+0000 7f9133e00640 1 -- 172.21.15.67:0/4187501820 >> v2:172.21.15.187:3300/0 conn(0x7f913c15c5a0 msgr2=0x7f913c15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.437+0000 7f9133e00640 1 --2- 172.21.15.67:0/4187501820 >> v2:172.21.15.187:3300/0 conn(0x7f913c15c5a0 0x7f913c15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.437+0000 7f9133e00640 1 -- 172.21.15.67:0/4187501820 >> v1:172.21.15.67:6789/0 conn(0x7f913c0b46c0 legacy=0x7f913c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.437+0000 7f9133e00640 1 -- 172.21.15.67:0/4187501820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c15fa00 con 0x7f913c0b4210 2024-04-17T07:30:10.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.438+0000 7f9133e00640 1 --2- 172.21.15.67:0/4187501820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0b4210 0x7f913c0af440 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f913c0b3010 tx=0x7f912402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9be722a05b2df3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.438+0000 7f913a000640 1 -- 172.21.15.67:0/4187501820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912402e070 con 0x7f913c0b4210 2024-04-17T07:30:10.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.438+0000 7f913a000640 1 -- 172.21.15.67:0/4187501820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124037070 con 0x7f913c0b4210 2024-04-17T07:30:10.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.438+0000 7f913a000640 1 -- 172.21.15.67:0/4187501820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124031a00 con 0x7f913c0b4210 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 -- 172.21.15.67:0/4187501820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0b4210 msgr2=0x7f913c0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 --2- 172.21.15.67:0/4187501820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0b4210 0x7f913c0af440 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f913c0b3010 tx=0x7f912402b880 comp rx=0 tx=0).stop 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 -- 172.21.15.67:0/4187501820 shutdown_connections 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 --2- 172.21.15.67:0/4187501820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0b4210 0x7f913c0af440 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 --2- 172.21.15.67:0/4187501820 >> v2:172.21.15.187:3300/0 conn(0x7f913c15c5a0 0x7f913c15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 -- 172.21.15.67:0/4187501820 >> 172.21.15.67:0/4187501820 conn(0x7f913c067d10 msgr2=0x7f913c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 -- 172.21.15.67:0/4187501820 shutdown_connections 2024-04-17T07:30:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.439+0000 7f913be00640 1 -- 172.21.15.67:0/4187501820 wait complete. 2024-04-17T07:30:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.441+0000 7f913be00640 1 Processor -- start 2024-04-17T07:30:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.441+0000 7f913be00640 1 -- start start 2024-04-17T07:30:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 0x7f913c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c15c5a0 0x7f913c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0adc30 con 0x7f913c06f320 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0adab0 con 0x7f913c0b4210 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f913c06e670 con 0x7f913c15c5a0 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58956/0 (socket says 172.21.15.67:58956) 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 -- 172.21.15.67:0/3812975125 learned_addr learned my addr 172.21.15.67:0/3812975125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913aa00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 0x7f913c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f9133e00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c15c5a0 0x7f913c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 -- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c15c5a0 msgr2=0x7f913c06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c15c5a0 0x7f913c06ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 -- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 msgr2=0x7f913c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 0x7f913c0ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913b400640 1 -- 172.21.15.67:0/3812975125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c1e5c70 con 0x7f913c06f320 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.442+0000 7f913aa00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 0x7f913c0ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f913b400640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f913400c730 tx=0x7f913400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134017070 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913400ce40 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913400a800 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913c0cf250 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.443+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913c145080 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.445+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f913c068a10 con 0x7f913c06f320 2024-04-17T07:30:10.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.445+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f913401a070 con 0x7f913c06f320 2024-04-17T07:30:10.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.446+0000 7f9132a00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 0x7f911806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.446+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 4857+0+0 (secure 0 0 0) 0x7f913408aff0 con 0x7f913c06f320 2024-04-17T07:30:10.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.447+0000 7f913aa00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 0x7f911806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.448+0000 7f913aa00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 0x7f911806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f912c006fd0 tx=0x7f912c008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.451+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f91340595d0 con 0x7f913c06f320 2024-04-17T07:30:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.649+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f913c0ae1f0 con 0x7f913c06f320 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.650+0000 7f9132a00640 1 -- 172.21.15.67:0/3812975125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) ==== 84+0+65 (secure 0 0 0) 0x7f9134058ed0 con 0x7f913c06f320 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:30:10.649 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:30:10.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:10.650 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 msgr2=0x7f911806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 0x7f911806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f912c006fd0 tx=0x7f912c008040 comp rx=0 tx=0).stop 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 msgr2=0x7f913c1e4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f913400c730 tx=0x7f913400cbf0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 shutdown_connections 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c06f320 0x7f913c1e4ed0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c15c5a0 0x7f913c06ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f911806ca40 0x7f911806ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 --2- 172.21.15.67:0/3812975125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c0b4210 0x7f913c0ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 >> 172.21.15.67:0/3812975125 conn(0x7f913c067d10 msgr2=0x7f913c071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 shutdown_connections 2024-04-17T07:30:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.653+0000 7f913be00640 1 -- 172.21.15.67:0/3812975125 wait complete. 2024-04-17T07:30:10.672 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:30:10.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:30:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.894+0000 7f2085600640 1 Processor -- start 2024-04-17T07:30:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.894+0000 7f2085600640 1 -- start start 2024-04-17T07:30:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20800b3800 0x7f20800cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20800b3be0 0x7f208015f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2080162e40 0x7f2080155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f20801629b0 con 0x7f20800b3800 2024-04-17T07:30:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f208015f890 con 0x7f2080162e40 2024-04-17T07:30:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.895+0000 7f2085600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f208015fb70 con 0x7f20800b3be0 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20800b3800 0x7f20800cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20800b3800 0x7f20800cfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58972/0 (socket says 172.21.15.67:58972) 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 -- 172.21.15.67:0/2194421044 learned_addr learned my addr 172.21.15.67:0/2194421044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207f400640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20800b3be0 0x7f208015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f2084c00640 1 --2- 172.21.15.67:0/2194421044 >> v2:172.21.15.187:3300/0 conn(0x7f2080162e40 0x7f2080155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 -- 172.21.15.67:0/2194421044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20800b3be0 msgr2=0x7f208015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20800b3be0 0x7f208015f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 -- 172.21.15.67:0/2194421044 >> v2:172.21.15.187:3300/0 conn(0x7f2080162e40 msgr2=0x7f2080155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 --2- 172.21.15.67:0/2194421044 >> v2:172.21.15.187:3300/0 conn(0x7f2080162e40 0x7f2080155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.896+0000 7f207fe00640 1 -- 172.21.15.67:0/2194421044 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208015fe50 con 0x7f20800b3800 2024-04-17T07:30:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.898+0000 7f207fe00640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3800 0x7f20800cfc80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f20800b7880 tx=0x7f207002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02b1a9ddb18021a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.898+0000 7f207ea00640 1 -- 172.21.15.67:0/2194421044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207002ca80 con 0x7f20800b3800 2024-04-17T07:30:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.898+0000 7f207ea00640 1 -- 172.21.15.67:0/2194421044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207002cc20 con 0x7f20800b3800 2024-04-17T07:30:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.898+0000 7f207ea00640 1 -- 172.21.15.67:0/2194421044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20700343f0 con 0x7f20800b3800 2024-04-17T07:30:10.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 -- 172.21.15.67:0/2194421044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3800 msgr2=0x7f20800cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3800 0x7f20800cfc80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f20800b7880 tx=0x7f207002bee0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 -- 172.21.15.67:0/2194421044 shutdown_connections 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 --2- 172.21.15.67:0/2194421044 >> v2:172.21.15.187:3300/0 conn(0x7f2080162e40 0x7f2080155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20800b3be0 0x7f208015f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 --2- 172.21.15.67:0/2194421044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3800 0x7f20800cfc80 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 -- 172.21.15.67:0/2194421044 >> 172.21.15.67:0/2194421044 conn(0x7f2080067600 msgr2=0x7f20800679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 -- 172.21.15.67:0/2194421044 shutdown_connections 2024-04-17T07:30:10.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.899+0000 7f2085600640 1 -- 172.21.15.67:0/2194421044 wait complete. 2024-04-17T07:30:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.901+0000 7f2085600640 1 Processor -- start 2024-04-17T07:30:10.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.901+0000 7f2085600640 1 -- start start 2024-04-17T07:30:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 0x7f20801f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2080162e40 0x7f208015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f208015f560 con 0x7f20800b3be0 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20800b4040 con 0x7f20800b3800 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2085600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20801632d0 con 0x7f2080162e40 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f207f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f207f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58984/0 (socket says 172.21.15.67:58984) 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f207f400640 1 -- 172.21.15.67:0/1490233248 learned_addr learned my addr 172.21.15.67:0/1490233248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f2084c00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2080162e40 0x7f208015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.902+0000 7f207fe00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 0x7f20801f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 -- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2080162e40 msgr2=0x7f208015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2080162e40 0x7f208015ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 -- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 msgr2=0x7f20801f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 0x7f20801f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 -- 172.21.15.67:0/1490233248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800b5660 con 0x7f20800b3be0 2024-04-17T07:30:10.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207fe00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 0x7f20801f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:10.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f207f400640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f206800d800 tx=0x7f206800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206800b6c0 con 0x7f20800b3be0 2024-04-17T07:30:10.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2068019070 con 0x7f20800b3be0 2024-04-17T07:30:10.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206800ba80 con 0x7f20800b3be0 2024-04-17T07:30:10.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20800b5940 con 0x7f20800b3be0 2024-04-17T07:30:10.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.903+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20800b5f00 con 0x7f20800b3be0 2024-04-17T07:30:10.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.905+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2080068a10 con 0x7f20800b3be0 2024-04-17T07:30:10.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.906+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f206800bc20 con 0x7f20800b3be0 2024-04-17T07:30:10.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.907+0000 7f2077e00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 0x7f205c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:10.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.907+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 4857+0+0 (secure 0 0 0) 0x7f206808bf20 con 0x7f20800b3be0 2024-04-17T07:30:10.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.908+0000 7f207fe00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 0x7f205c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:10.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.909+0000 7f207fe00640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 0x7f205c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2070035000 tx=0x7f2070030000 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:10.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:10.911+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f206805a4f0 con 0x7f20800b3be0 2024-04-17T07:30:11.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.109+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f208006da00 con 0x7f20800b3be0 2024-04-17T07:30:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.603+0000 7f2077e00640 1 -- 172.21.15.67:0/1490233248 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) ==== 138+0+0 (secure 0 0 0) 0x7f2068059df0 con 0x7f20800b3be0 2024-04-17T07:30:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.606+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 msgr2=0x7f205c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.606+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 0x7f205c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2070035000 tx=0x7f2070030000 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.606+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 msgr2=0x7f20800b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.606+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f206800d800 tx=0x7f206800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 shutdown_connections 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2080162e40 0x7f208015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20800b3be0 0x7f20800b4ce0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f205c06ca10 0x7f205c06eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 --2- 172.21.15.67:0/1490233248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20800b3800 0x7f20801f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 >> 172.21.15.67:0/1490233248 conn(0x7f2080067600 msgr2=0x7f2080161740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:11.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 shutdown_connections 2024-04-17T07:30:11.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.607+0000 7f2085600640 1 -- 172.21.15.67:0/1490233248 wait complete. 2024-04-17T07:30:11.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:30:11.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.839+0000 7fe333e00640 1 Processor -- start 2024-04-17T07:30:11.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.839+0000 7fe333e00640 1 -- start start 2024-04-17T07:30:11.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.840+0000 7fe333e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe32c160d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe333e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe32c161240 con 0x7fe32c072630 2024-04-17T07:30:11.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe333e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe32c0cda40 con 0x7fe32c0cd060 2024-04-17T07:30:11.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe333e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe32c146530 con 0x7fe32c146150 2024-04-17T07:30:11.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe331e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe32c072630 0x7fe32c15cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45176/0 (socket says 172.21.15.67:45176) 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe331e00640 1 -- 172.21.15.67:0/3933963270 learned_addr learned my addr 172.21.15.67:0/3933963270 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.841+0000 7fe331400640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe32c160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764549902 0 0) 0x7fe324001110 con 0x7fe32c072630 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe314003520 con 0x7fe32c072630 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 296778116 0 0) 0x7fe314003520 con 0x7fe32c072630 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 msgr2=0x7fe32c160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe32c160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 >> v1:172.21.15.187:6789/0 conn(0x7fe32c0cd060 legacy=0x7fe32c071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c0cdd20 con 0x7fe32c072630 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe324001840 con 0x7fe32c072630 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 >> v1:172.21.15.67:6789/0 conn(0x7fe32c072630 legacy=0x7fe32c15cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe3140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe314003fb0 0x7fe314004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe3140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe31400bad0 con 0x7fe314004d40 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe31400bc80 con 0x7fe314003fb0 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.842+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe31400be80 con 0x7fe32c146150 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe3140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331400640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe3140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe330a00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe314003fb0 0x7fe314004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 -- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 msgr2=0x7fe3140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c146150 0x7fe3140039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 -- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe314003fb0 msgr2=0x7fe314004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe314003fb0 0x7fe314004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.843+0000 7fe331e00640 1 -- 172.21.15.67:0/3933963270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe314003520 con 0x7fe314004d40 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.844+0000 7fe331e00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe3140051c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe32c06fa50 tx=0x7fe314005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befe6d5e6785c9eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.844+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324007e00 con 0x7fe314004d40 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe324030070 con 0x7fe314004d40 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe32be00640 1 -- 172.21.15.67:0/3933963270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32402ce10 con 0x7fe314004d40 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 -- 172.21.15.67:0/3933963270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 msgr2=0x7fe3140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 --2- 172.21.15.67:0/3933963270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe3140051c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe32c06fa50 tx=0x7fe314005830 comp rx=0 tx=0).stop 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe331e00640 1 -- 172.21.15.67:0/3933963270 reap_dead start 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 -- 172.21.15.67:0/3933963270 shutdown_connections 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 -- 172.21.15.67:0/3933963270 >> 172.21.15.67:0/3933963270 conn(0x7fe32c067c30 msgr2=0x7fe32c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 -- 172.21.15.67:0/3933963270 shutdown_connections 2024-04-17T07:30:11.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.845+0000 7fe333e00640 1 -- 172.21.15.67:0/3933963270 wait complete. 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.847+0000 7fe333e00640 1 Processor -- start 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.847+0000 7fe333e00640 1 -- start start 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c0caf00 0x7fe32c144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe32c0aa490 0x7fe32c0aa910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c06f310 con 0x7fe314004d40 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c1607e0 con 0x7fe32c0aa490 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe333e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe32c06f490 con 0x7fe32c0caf00 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.848+0000 7fe331400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58998/0 (socket says 172.21.15.67:58998) 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 -- 172.21.15.67:0/1857488587 learned_addr learned my addr 172.21.15.67:0/1857488587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe330a00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c0caf00 0x7fe32c144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 -- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c0caf00 msgr2=0x7fe32c144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c0caf00 0x7fe32c144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 -- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe32c0aa490 msgr2=0x7fe32c0aa910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe32c0aa490 0x7fe32c0aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:11.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 -- 172.21.15.67:0/1857488587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c0cb660 con 0x7fe314004d40 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.849+0000 7fe331400640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fe320007f30 tx=0x7fe320011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.850+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32001a020 con 0x7fe314004d40 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.850+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe320016470 con 0x7fe314004d40 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.850+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320016860 con 0x7fe314004d40 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.850+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c0cb940 con 0x7fe314004d40 2024-04-17T07:30:11.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.850+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c0cbf00 con 0x7fe314004d40 2024-04-17T07:30:11.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.851+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c068ca0 con 0x7fe314004d40 2024-04-17T07:30:11.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.853+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fe320005d10 con 0x7fe314004d40 2024-04-17T07:30:11.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.854+0000 7fe329600640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 0x7fe30c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:11.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.854+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5227+0+0 (secure 0 0 0) 0x7fe32008ca90 con 0x7fe314004d40 2024-04-17T07:30:11.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.855+0000 7fe330a00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 0x7fe30c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.856+0000 7fe330a00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 0x7fe30c06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0a9a50 tx=0x7fe318009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:11.859+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe320024030 con 0x7fe314004d40 2024-04-17T07:30:11.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-17T07:30:12.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.062+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe32c145520 con 0x7fe314004d40 2024-04-17T07:30:12.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.619+0000 7fe329600640 1 -- 172.21.15.67:0/1857488587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) ==== 170+0+0 (secure 0 0 0) 0x7fe32005ae80 con 0x7fe314004d40 2024-04-17T07:30:12.618 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 msgr2=0x7fe30c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 0x7fe30c06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0a9a50 tx=0x7fe318009040 comp rx=0 tx=0).stop 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 msgr2=0x7fe32c0c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fe320007f30 tx=0x7fe320011660 comp rx=0 tx=0).stop 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 shutdown_connections 2024-04-17T07:30:12.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fe30c06ca10 0x7fe30c06eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe32c0aa490 0x7fe32c0aa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe32c0caf00 0x7fe32c144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 --2- 172.21.15.67:0/1857488587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314004d40 0x7fe32c0c9180 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 >> 172.21.15.67:0/1857488587 conn(0x7fe32c067c30 msgr2=0x7fe32c15f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 shutdown_connections 2024-04-17T07:30:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.623+0000 7fe333e00640 1 -- 172.21.15.67:0/1857488587 wait complete. 2024-04-17T07:30:12.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:30:12.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.848+0000 7f6d17200640 1 Processor -- start 2024-04-17T07:30:12.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.848+0000 7f6d17200640 1 -- start start 2024-04-17T07:30:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d17200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6d100b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d17200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1006df40 con 0x7f6d1015b6b0 2024-04-17T07:30:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d17200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d1006e220 con 0x7f6d1006bfd0 2024-04-17T07:30:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d17200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d1006ed00 con 0x7f6d1006f800 2024-04-17T07:30:12.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d15e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d1015b6b0 0x7f6d1006b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45178/0 (socket says 172.21.15.67:45178) 2024-04-17T07:30:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.850+0000 7f6d15e00640 1 -- 172.21.15.67:0/3275575259 learned_addr learned my addr 172.21.15.67:0/3275575259 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d16800640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6d100b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014755614 0 0) 0x7f6d1006df40 con 0x7f6d1015b6b0 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cf8005730 con 0x7f6d1015b6b0 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2752149144 0 0) 0x7f6cf8005730 con 0x7f6d1015b6b0 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 msgr2=0x7f6d100b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6d100b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 >> v1:172.21.15.187:6789/0 conn(0x7f6d1006bfd0 legacy=0x7f6d100b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1006e500 con 0x7f6d1015b6b0 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6d04001670 con 0x7f6d1015b6b0 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 >> v1:172.21.15.67:6789/0 conn(0x7f6d1015b6b0 legacy=0x7f6d1006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf8005bf0 0x7f6cf8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6cf8006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6cf800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1006ed00 con 0x7f6cf8006a10 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8007760 con 0x7f6cf8005bf0 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.851+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf800e110 con 0x7f6d1006f800 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6cf8006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15e00640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf8005bf0 0x7f6cf8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d16800640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6cf800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 -- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 msgr2=0x7f6cf800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6cf800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 -- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf8005bf0 msgr2=0x7f6cf8006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf8005bf0 0x7f6cf8006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d15400640 1 -- 172.21.15.67:0/3275575259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf8005730 con 0x7f6cf8006a10 2024-04-17T07:30:12.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.852+0000 7f6d16800640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1006f800 0x7f6cf800dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d15400640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6cf8006e90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6d00001750 tx=0x7f6cf8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce01f85e8eafd98 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00001370 con 0x7f6cf8006a10 2024-04-17T07:30:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d000052d0 con 0x7f6cf8006a10 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d14a00640 1 -- 172.21.15.67:0/3275575259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d00003830 con 0x7f6cf8006a10 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d17200640 1 -- 172.21.15.67:0/3275575259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 msgr2=0x7f6cf8006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d17200640 1 --2- 172.21.15.67:0/3275575259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6cf8006e90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6d00001750 tx=0x7f6cf8007a40 comp rx=0 tx=0).stop 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d16800640 1 -- 172.21.15.67:0/3275575259 reap_dead start 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d17200640 1 -- 172.21.15.67:0/3275575259 shutdown_connections 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.854+0000 7f6d17200640 1 -- 172.21.15.67:0/3275575259 >> 172.21.15.67:0/3275575259 conn(0x7f6d100b2190 msgr2=0x7f6d100b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.855+0000 7f6d17200640 1 -- 172.21.15.67:0/3275575259 shutdown_connections 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.855+0000 7f6d17200640 1 -- 172.21.15.67:0/3275575259 wait complete. 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.856+0000 7f6d17200640 1 Processor -- start 2024-04-17T07:30:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.856+0000 7f6d17200640 1 -- start start 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1015ba90 0x7f6d1015bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1013faa0 0x7f6d1013ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d100cfc10 con 0x7f6cf8006a10 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1006e870 con 0x7f6d1013faa0 2024-04-17T07:30:12.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.857+0000 7f6d17200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d100cfd90 con 0x7f6d1015ba90 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59010/0 (socket says 172.21.15.67:59010) 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 -- 172.21.15.67:0/3892617078 learned_addr learned my addr 172.21.15.67:0/3892617078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15400640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1015ba90 0x7f6d1015bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d16800640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1013faa0 0x7f6d1013ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 -- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1015ba90 msgr2=0x7f6d1015bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1015ba90 0x7f6d1015bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 -- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1013faa0 msgr2=0x7f6d1013ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1013faa0 0x7f6d1013ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.858+0000 7f6d15e00640 1 -- 172.21.15.67:0/3892617078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d101406b0 con 0x7f6cf8006a10 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d15e00640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400e800 tx=0x7f6d0400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400c870 con 0x7f6cf8006a10 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d04018070 con 0x7f6cf8006a10 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d10140990 con 0x7f6cf8006a10 2024-04-17T07:30:12.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400cc10 con 0x7f6cf8006a10 2024-04-17T07:30:12.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.859+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d10140f50 con 0x7f6cf8006a10 2024-04-17T07:30:12.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.860+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d10068a10 con 0x7f6cf8006a10 2024-04-17T07:30:12.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.861+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f6d04013420 con 0x7f6cf8006a10 2024-04-17T07:30:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.862+0000 7f6d0a000640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 0x7f6cf406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.862+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5609+0+0 (secure 0 0 0) 0x7f6d0408b830 con 0x7f6cf8006a10 2024-04-17T07:30:12.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.863+0000 7f6d15400640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 0x7f6cf406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:12.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.864+0000 7f6d15400640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 0x7f6cf406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6d00033000 tx=0x7f6d00009030 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:12.867+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d04059bb0 con 0x7f6cf8006a10 2024-04-17T07:30:13.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.116+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f6d1006ee80 con 0x7f6cf8006a10 2024-04-17T07:30:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.623+0000 7f6d0a000640 1 -- 172.21.15.67:0/3892617078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) ==== 174+0+0 (secure 0 0 0) 0x7f6d040594b0 con 0x7f6cf8006a10 2024-04-17T07:30:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 10 and data pool 11 2024-04-17T07:30:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.628+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 msgr2=0x7f6cf406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.628+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 0x7f6cf406ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6d00033000 tx=0x7f6d00009030 comp rx=0 tx=0).stop 2024-04-17T07:30:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.628+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 msgr2=0x7f6d1015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.628+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400e800 tx=0x7f6d0400ecc0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 shutdown_connections 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f6cf406ca40 0x7f6cf406ef00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d1013faa0 0x7f6d1013ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d1015ba90 0x7f6d1015bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 --2- 172.21.15.67:0/3892617078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf8006a10 0x7f6d1015f550 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.629+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 >> 172.21.15.67:0/3892617078 conn(0x7f6d100b2190 msgr2=0x7f6d100b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:13.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.630+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 shutdown_connections 2024-04-17T07:30:13.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.630+0000 7f6d17200640 1 -- 172.21.15.67:0/3892617078 wait complete. 2024-04-17T07:30:13.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.851+0000 7f3b5c000640 1 Processor -- start 2024-04-17T07:30:13.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.851+0000 7f3b5c000640 1 -- start start 2024-04-17T07:30:13.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.852+0000 7f3b5c000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b5406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.852+0000 7f3b5c000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b540cd020 0x7f3b540cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.852+0000 7f3b5c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b5406ff90 con 0x7f3b54071d10 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.852+0000 7f3b5c000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3b540ceb00 con 0x7f3b540cd020 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.852+0000 7f3b5c000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540cede0 con 0x7f3b5406f670 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b5a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b54071d10 0x7f3b540721c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39900/0 (socket says 172.21.15.67:39900) 2024-04-17T07:30:13.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b5a000640 1 -- 172.21.15.67:0/3840252686 learned_addr learned my addr 172.21.15.67:0/3840252686 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:13.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b58c00640 1 --2- 172.21.15.67:0/3840252686 >> v2:172.21.15.187:3300/0 conn(0x7f3b540cd020 0x7f3b540cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b59600640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b5406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429654203 0 0) 0x7f3b5406ff90 con 0x7f3b54071d10 2024-04-17T07:30:13.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b3c003520 con 0x7f3b54071d10 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.853+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3261806771 0 0) 0x7f3b3c003520 con 0x7f3b54071d10 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 msgr2=0x7f3b5406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b5406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 >> v2:172.21.15.187:3300/0 conn(0x7f3b540cd020 msgr2=0x7f3b540cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 --2- 172.21.15.67:0/3840252686 >> v2:172.21.15.187:3300/0 conn(0x7f3b540cd020 0x7f3b540cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b540cf0c0 con 0x7f3b54071d10 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3b4c0019f0 con 0x7f3b54071d10 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 >> v1:172.21.15.67:6789/0 conn(0x7f3b54071d10 legacy=0x7f3b540721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b3c0039b0 0x7f3b3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540ceb00 con 0x7f3b3c0039b0 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540cede0 con 0x7f3b3c004d40 2024-04-17T07:30:13.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c00ba40 con 0x7f3b5406f670 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b59600640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b3c0039b0 0x7f3b3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b5a000640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b58c00640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b59600640 1 -- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 msgr2=0x7f3b3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b59600640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b5406f670 0x7f3b3c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b59600640 1 -- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b3c004d40 msgr2=0x7f3b3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.854+0000 7f3b59600640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.855+0000 7f3b59600640 1 -- 172.21.15.67:0/3840252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c003520 con 0x7f3b3c0039b0 2024-04-17T07:30:13.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.856+0000 7f3b59600640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b3c0039b0 0x7f3b3c003dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800ba30 tx=0x7f3b3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eefccc6a71b67e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:13.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.856+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48036020 con 0x7f3b3c0039b0 2024-04-17T07:30:13.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.856+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b48032640 con 0x7f3b3c0039b0 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.856+0000 7f3b53e00640 1 -- 172.21.15.67:0/3840252686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4800b600 con 0x7f3b3c0039b0 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 -- 172.21.15.67:0/3840252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b3c0039b0 msgr2=0x7f3b3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 --2- 172.21.15.67:0/3840252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b3c0039b0 0x7f3b3c003dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800ba30 tx=0x7f3b3c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5a000640 1 -- 172.21.15.67:0/3840252686 reap_dead start 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 -- 172.21.15.67:0/3840252686 shutdown_connections 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 -- 172.21.15.67:0/3840252686 >> 172.21.15.67:0/3840252686 conn(0x7f3b5406bf50 msgr2=0x7f3b540b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 -- 172.21.15.67:0/3840252686 shutdown_connections 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.857+0000 7f3b5c000640 1 -- 172.21.15.67:0/3840252686 wait complete. 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.858+0000 7f3b5c000640 1 Processor -- start 2024-04-17T07:30:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 -- start start 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b3c0039b0 0x7f3b54149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 0x7f3b540b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540ce720 con 0x7f3b54148c70 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540cf680 con 0x7f3b5415f670 2024-04-17T07:30:13.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.859+0000 7f3b5c000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540ce8a0 con 0x7f3b3c0039b0 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35448/0 (socket says 172.21.15.67:35448) 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 -- 172.21.15.67:0/2298943937 learned_addr learned my addr 172.21.15.67:0/2298943937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b58c00640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 0x7f3b540b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b59600640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b3c0039b0 0x7f3b54149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 -- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b3c0039b0 msgr2=0x7f3b54149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b3c0039b0 0x7f3b54149830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 -- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 msgr2=0x7f3b540b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 0x7f3b540b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b5a000640 1 -- 172.21.15.67:0/2298943937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b540b5ae0 con 0x7f3b54148c70 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.860+0000 7f3b58c00640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 0x7f3b540b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b5a000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3b540ce670 tx=0x7f3b4c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00ebb0 con 0x7f3b54148c70 2024-04-17T07:30:13.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00ed50 con 0x7f3b54148c70 2024-04-17T07:30:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c014670 con 0x7f3b54148c70 2024-04-17T07:30:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5415faf0 con 0x7f3b54148c70 2024-04-17T07:30:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.861+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b540b5d80 con 0x7f3b54148c70 2024-04-17T07:30:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.862+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b54068b90 con 0x7f3b54148c70 2024-04-17T07:30:13.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.863+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f3b4c014810 con 0x7f3b54148c70 2024-04-17T07:30:13.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.864+0000 7f3b51600640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 0x7f3b3406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:13.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.864+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f3b4c08ba70 con 0x7f3b54148c70 2024-04-17T07:30:13.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.865+0000 7f3b59600640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 0x7f3b3406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:13.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.866+0000 7f3b59600640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 0x7f3b3406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800ba00 tx=0x7f3b48030000 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:13.869+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3b4c059d80 con 0x7f3b54148c70 2024-04-17T07:30:14.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.073+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b54075390 con 0x7f3b54148c70 2024-04-17T07:30:14.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.075+0000 7f3b51600640 1 -- 172.21.15.67:0/2298943937 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16633 (secure 0 0 0) 0x7f3b4c059680 con 0x7f3b54148c70 2024-04-17T07:30:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:14.075 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":97,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:13.117012+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:14.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 msgr2=0x7f3b3406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 0x7f3b3406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800ba00 tx=0x7f3b48030000 comp rx=0 tx=0).stop 2024-04-17T07:30:14.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 msgr2=0x7f3b541490f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3b540ce670 tx=0x7f3b4c00e530 comp rx=0 tx=0).stop 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 shutdown_connections 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b54148c70 0x7f3b541490f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f3b3406ca40 0x7f3b3406ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b5415f670 0x7f3b540b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 --2- 172.21.15.67:0/2298943937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b3c0039b0 0x7f3b54149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 >> 172.21.15.67:0/2298943937 conn(0x7f3b5406bf50 msgr2=0x7f3b540b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 shutdown_connections 2024-04-17T07:30:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.077+0000 7f3b5c000640 1 -- 172.21.15.67:0/2298943937 wait complete. 2024-04-17T07:30:14.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:14.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.311+0000 7fbe6f200640 1 Processor -- start 2024-04-17T07:30:14.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.311+0000 7fbe6f200640 1 -- start start 2024-04-17T07:30:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.312+0000 7fbe6f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006feb0 0x7fbe7006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe7006be70 0x7fbe7015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe7015bc80 con 0x7fbe7015b570 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe7015bf40 con 0x7fbe7006be70 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7015c220 con 0x7fbe7006feb0 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe7006be70 0x7fbe7015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe7006be70 0x7fbe7015f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35450/0 (socket says 172.21.15.67:35450) 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 -- 172.21.15.67:0/3007558352 learned_addr learned my addr 172.21.15.67:0/3007558352 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe67e00640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006feb0 0x7fbe7006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 -- 172.21.15.67:0/3007558352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006feb0 msgr2=0x7fbe7006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006feb0 0x7fbe7006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 -- 172.21.15.67:0/3007558352 >> v1:172.21.15.187:6789/0 conn(0x7fbe7015b570 legacy=0x7fbe7006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.313+0000 7fbe6e800640 1 -- 172.21.15.67:0/3007558352 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7015c500 con 0x7fbe7006be70 2024-04-17T07:30:14.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.314+0000 7fbe6e800640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006be70 0x7fbe7015f140 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009e50 tx=0x7fbe6802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fa512913e5d328 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6d400640 1 -- 172.21.15.67:0/3007558352 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6802c9e0 con 0x7fbe7006be70 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6d400640 1 -- 172.21.15.67:0/3007558352 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe6802cb80 con 0x7fbe7006be70 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6d400640 1 -- 172.21.15.67:0/3007558352 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe680343f0 con 0x7fbe7006be70 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 -- 172.21.15.67:0/3007558352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006be70 msgr2=0x7fbe7015f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006be70 0x7fbe7015f140 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009e50 tx=0x7fbe6802be40 comp rx=0 tx=0).stop 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 -- 172.21.15.67:0/3007558352 shutdown_connections 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006be70 0x7fbe7015f140 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 --2- 172.21.15.67:0/3007558352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006feb0 0x7fbe7006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.315+0000 7fbe6f200640 1 -- 172.21.15.67:0/3007558352 >> 172.21.15.67:0/3007558352 conn(0x7fbe70067d10 msgr2=0x7fbe700b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.316+0000 7fbe6f200640 1 -- 172.21.15.67:0/3007558352 shutdown_connections 2024-04-17T07:30:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.316+0000 7fbe6f200640 1 -- 172.21.15.67:0/3007558352 wait complete. 2024-04-17T07:30:14.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.317+0000 7fbe6f200640 1 Processor -- start 2024-04-17T07:30:14.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.317+0000 7fbe6f200640 1 -- start start 2024-04-17T07:30:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006be70 0x7fbe70149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe7015b570 0x7fbe7015e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7006c460 con 0x7fbe7006feb0 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7006c2e0 con 0x7fbe7015b570 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe6f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe700b47b0 con 0x7fbe7006be70 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.318+0000 7fbe67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35454/0 (socket says 172.21.15.67:35454) 2024-04-17T07:30:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 -- 172.21.15.67:0/2682729617 learned_addr learned my addr 172.21.15.67:0/2682729617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:14.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 -- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006be70 msgr2=0x7fbe70149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006be70 0x7fbe70149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 -- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe7015b570 msgr2=0x7fbe7015e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe6e800640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe7015b570 0x7fbe7015e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe7015b570 0x7fbe7015e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 -- 172.21.15.67:0/2682729617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe700af400 con 0x7fbe7006feb0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.319+0000 7fbe67e00640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d830 tx=0x7fbe5800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.320+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800b6c0 con 0x7fbe7006feb0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.320+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe58019070 con 0x7fbe7006feb0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.320+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ba80 con 0x7fbe7006feb0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.320+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe700af6e0 con 0x7fbe7006feb0 2024-04-17T07:30:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.320+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe701590a0 con 0x7fbe7006feb0 2024-04-17T07:30:14.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.321+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe70068a10 con 0x7fbe7006feb0 2024-04-17T07:30:14.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.322+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fbe5801e460 con 0x7fbe7006feb0 2024-04-17T07:30:14.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.323+0000 7fbe66000640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 0x7fbe4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.323+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fbe5808c320 con 0x7fbe7006feb0 2024-04-17T07:30:14.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.323+0000 7fbe6de00640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 0x7fbe4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.325+0000 7fbe6de00640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 0x7fbe4c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe60006fd0 tx=0x7fbe60006bc0 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.327+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbe5805a6d0 con 0x7fbe7006feb0 2024-04-17T07:30:14.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.523+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe7006f350 con 0x7fbe7006feb0 2024-04-17T07:30:14.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.526+0000 7fbe66000640 1 -- 172.21.15.67:0/2682729617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16633 (secure 0 0 0) 0x7fbe58059fd0 con 0x7fbe7006feb0 2024-04-17T07:30:14.526 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:14.526 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":97,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:13.117012+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 msgr2=0x7fbe4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 0x7fbe4c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe60006fd0 tx=0x7fbe60006bc0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 msgr2=0x7fbe70158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d830 tx=0x7fbe5800dcf0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 shutdown_connections 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fbe4c06ca10 0x7fbe4c06eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe7015b570 0x7fbe7015e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7006feb0 0x7fbe70158b60 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 --2- 172.21.15.67:0/2682729617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe7006be70 0x7fbe70149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 >> 172.21.15.67:0/2682729617 conn(0x7fbe70067d10 msgr2=0x7fbe700b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 shutdown_connections 2024-04-17T07:30:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.529+0000 7fbe6f200640 1 -- 172.21.15.67:0/2682729617 wait complete. 2024-04-17T07:30:14.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:30:14.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.764+0000 7ff807200640 1 Processor -- start 2024-04-17T07:30:14.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.765+0000 7ff807200640 1 -- start start 2024-04-17T07:30:14.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.766+0000 7ff807200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff80015eca0 0x7ff8000d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.766+0000 7ff807200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8000cfa10 0x7ff8000cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.766+0000 7ff807200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff800163860 con 0x7ff8000d1090 2024-04-17T07:30:14.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.766+0000 7ff807200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8000d03b0 con 0x7ff8000cfa10 2024-04-17T07:30:14.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.766+0000 7ff807200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff8000d1d40 con 0x7ff80015eca0 2024-04-17T07:30:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8000cfa10 0x7ff8000cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8000cfa10 0x7ff8000cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35460/0 (socket says 172.21.15.67:35460) 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 -- 172.21.15.67:0/731365095 learned_addr learned my addr 172.21.15.67:0/731365095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ffe00640 1 --2- 172.21.15.67:0/731365095 >> v2:172.21.15.187:3300/0 conn(0x7ff80015eca0 0x7ff8000d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 -- 172.21.15.67:0/731365095 >> v2:172.21.15.187:3300/0 conn(0x7ff80015eca0 msgr2=0x7ff8000d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 --2- 172.21.15.67:0/731365095 >> v2:172.21.15.187:3300/0 conn(0x7ff80015eca0 0x7ff8000d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 -- 172.21.15.67:0/731365095 >> v1:172.21.15.187:6789/0 conn(0x7ff8000d1090 legacy=0x7ff8000d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.767+0000 7ff7ff400640 1 -- 172.21.15.67:0/731365095 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8000d2c30 con 0x7ff8000cfa10 2024-04-17T07:30:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.768+0000 7ff7ff400640 1 --2- 172.21.15.67:0/731365095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cfa10 0x7ff8000cfe70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff8000d26a0 tx=0x7ff7e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec98554657d25ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.769+0000 7ff7fea00640 1 -- 172.21.15.67:0/731365095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8035020 con 0x7ff8000cfa10 2024-04-17T07:30:14.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.769+0000 7ff7fea00640 1 -- 172.21.15.67:0/731365095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e8031870 con 0x7ff8000cfa10 2024-04-17T07:30:14.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.769+0000 7ff7fea00640 1 -- 172.21.15.67:0/731365095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8031c40 con 0x7ff8000cfa10 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 -- 172.21.15.67:0/731365095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cfa10 msgr2=0x7ff8000cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 --2- 172.21.15.67:0/731365095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cfa10 0x7ff8000cfe70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff8000d26a0 tx=0x7ff7e802b910 comp rx=0 tx=0).stop 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 -- 172.21.15.67:0/731365095 shutdown_connections 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 --2- 172.21.15.67:0/731365095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8000cfa10 0x7ff8000cfe70 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 --2- 172.21.15.67:0/731365095 >> v2:172.21.15.187:3300/0 conn(0x7ff80015eca0 0x7ff8000d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 -- 172.21.15.67:0/731365095 >> 172.21.15.67:0/731365095 conn(0x7ff800067600 msgr2=0x7ff8000679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 -- 172.21.15.67:0/731365095 shutdown_connections 2024-04-17T07:30:14.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.770+0000 7ff807200640 1 -- 172.21.15.67:0/731365095 wait complete. 2024-04-17T07:30:14.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.772+0000 7ff807200640 1 Processor -- start 2024-04-17T07:30:14.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.772+0000 7ff807200640 1 -- start start 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8000cfa10 0x7ff8001f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d1090 0x7ff80015f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8000d34d0 con 0x7ff80015eca0 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8000d31f0 con 0x7ff8000cfa10 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff807200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff80015f080 con 0x7ff8000d1090 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35462/0 (socket says 172.21.15.67:35462) 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 -- 172.21.15.67:0/3588878059 learned_addr learned my addr 172.21.15.67:0/3588878059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff7ffe00640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8000cfa10 0x7ff8001f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 -- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d1090 msgr2=0x7ff80015f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d1090 0x7ff80015f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 -- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8000cfa10 msgr2=0x7ff8001f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8000cfa10 0x7ff8001f8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.773+0000 7ff805200640 1 -- 172.21.15.67:0/3588878059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8001f8c90 con 0x7ff80015eca0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff805200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800c440 tx=0x7ff7f800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8011790 con 0x7ff80015eca0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f8017070 con 0x7ff80015eca0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8011b70 con 0x7ff80015eca0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8001f8f70 con 0x7ff80015eca0 2024-04-17T07:30:14.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.774+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8001615b0 con 0x7ff80015eca0 2024-04-17T07:30:14.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.775+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff800068ca0 con 0x7ff80015eca0 2024-04-17T07:30:14.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.776+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7ff7f8011d10 con 0x7ff80015eca0 2024-04-17T07:30:14.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.777+0000 7ff7f7e00640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 0x7ff7dc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:14.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.777+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7ff7f808bb20 con 0x7ff80015eca0 2024-04-17T07:30:14.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.778+0000 7ff7ffe00640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 0x7ff7dc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:14.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.779+0000 7ff7ffe00640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 0x7ff7dc06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009a60 tx=0x7ff7f0003990 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:14.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:14.781+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff7f8059e40 con 0x7ff80015eca0 2024-04-17T07:30:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.016+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff8000b9ea0 con 0x7ff80015eca0 2024-04-17T07:30:15.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.675+0000 7ff7f7e00640 1 -- 172.21.15.67:0/3588878059 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) ==== 118+0+0 (secure 0 0 0) 0x7ff7f8059740 con 0x7ff80015eca0 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 msgr2=0x7ff7dc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 0x7ff7dc06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009a60 tx=0x7ff7f0003990 comp rx=0 tx=0).stop 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 msgr2=0x7ff80015f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800c440 tx=0x7ff7f800c900 comp rx=0 tx=0).stop 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 shutdown_connections 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff80015eca0 0x7ff80015f8f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d1090 0x7ff80015f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff7dc06ca10 0x7ff7dc06eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 --2- 172.21.15.67:0/3588878059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8000cfa10 0x7ff8001f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 >> 172.21.15.67:0/3588878059 conn(0x7ff800067600 msgr2=0x7ff80015db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:15.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 shutdown_connections 2024-04-17T07:30:15.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.679+0000 7ff807200640 1 -- 172.21.15.67:0/3588878059 wait complete. 2024-04-17T07:30:15.702 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:30:15.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:30:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.933+0000 7f43dd600640 1 Processor -- start 2024-04-17T07:30:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.933+0000 7f43dd600640 1 -- start start 2024-04-17T07:30:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.934+0000 7f43dd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43d80d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.934+0000 7f43dd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d809bd00 con 0x7f43d815ee90 2024-04-17T07:30:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.934+0000 7f43dd600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d80d1820 con 0x7f43d806d940 2024-04-17T07:30:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.934+0000 7f43dd600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43d80d1b00 con 0x7f43d80b15a0 2024-04-17T07:30:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.935+0000 7f43d7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43d815ee90 0x7f43d806d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39914/0 (socket says 172.21.15.67:39914) 2024-04-17T07:30:15.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.935+0000 7f43d7e00640 1 -- 172.21.15.67:0/354694558 learned_addr learned my addr 172.21.15.67:0/354694558 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.935+0000 7f43dcc00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43d80d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.935+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339665306 0 0) 0x7f43d809bd00 con 0x7f43d815ee90 2024-04-17T07:30:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.935+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43bc003520 con 0x7f43d815ee90 2024-04-17T07:30:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3335606309 0 0) 0x7f43bc003520 con 0x7f43d815ee90 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 msgr2=0x7f43d80d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43d80d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 >> v1:172.21.15.187:6789/0 conn(0x7f43d806d940 legacy=0x7f43d80cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d80d1de0 con 0x7f43d815ee90 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f43c8001670 con 0x7f43d815ee90 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 >> v1:172.21.15.67:6789/0 conn(0x7f43d815ee90 legacy=0x7f43d806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43bc0039e0 0x7f43bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43bc004d40 0x7f43bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43d80d1b00 con 0x7f43bc004d40 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc00bb50 con 0x7f43bc0039e0 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43bc00bd50 con 0x7f43d80b15a0 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.936+0000 7f43dcc00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43bc004d40 0x7f43bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43dcc00640 1 -- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 msgr2=0x7f43bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43d7400640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43dcc00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43dcc00640 1 -- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43bc0039e0 msgr2=0x7f43bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43d7e00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43bc0039e0 0x7f43bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43dcc00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43bc0039e0 0x7f43bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43dcc00640 1 -- 172.21.15.67:0/354694558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc003520 con 0x7f43bc004d40 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43d7400640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43d80b15a0 0x7f43bc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.937+0000 7f43d7e00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43bc0039e0 0x7f43bc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.938+0000 7f43dcc00640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43bc004d40 0x7f43bc0051c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009fc0 tx=0x7f43bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41dd8eb76a4903b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.938+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0039070 con 0x7f43bc004d40 2024-04-17T07:30:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.938+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d002cbf0 con 0x7f43bc004d40 2024-04-17T07:30:15.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.938+0000 7f43d6a00640 1 -- 172.21.15.67:0/354694558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0009c10 con 0x7f43bc004d40 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 -- 172.21.15.67:0/354694558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43bc004d40 msgr2=0x7f43bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 --2- 172.21.15.67:0/354694558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43bc004d40 0x7f43bc0051c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009fc0 tx=0x7f43bc005830 comp rx=0 tx=0).stop 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dcc00640 1 -- 172.21.15.67:0/354694558 reap_dead start 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 -- 172.21.15.67:0/354694558 shutdown_connections 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 -- 172.21.15.67:0/354694558 >> 172.21.15.67:0/354694558 conn(0x7f43d80b4090 msgr2=0x7f43d80b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 -- 172.21.15.67:0/354694558 shutdown_connections 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.939+0000 7f43dd600640 1 -- 172.21.15.67:0/354694558 wait complete. 2024-04-17T07:30:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.941+0000 7f43dd600640 1 Processor -- start 2024-04-17T07:30:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.941+0000 7f43dd600640 1 -- start start 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43bc004d40 0x7f43d80b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43d80c6d70 0x7f43d80c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43d80d10f0 con 0x7f43d8159880 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43d80ccbf0 con 0x7f43d80c6d70 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43dd600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43d80d1270 con 0x7f43bc004d40 2024-04-17T07:30:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35478/0 (socket says 172.21.15.67:35478) 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.942+0000 7f43d7400640 1 -- 172.21.15.67:0/573816777 learned_addr learned my addr 172.21.15.67:0/573816777 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7e00640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43bc004d40 0x7f43d80b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43dcc00640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43d80c6d70 0x7f43d80c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 -- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43bc004d40 msgr2=0x7f43d80b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43bc004d40 0x7f43d80b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 -- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43d80c6d70 msgr2=0x7f43d80c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43d80c6d70 0x7f43d80c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 -- 172.21.15.67:0/573816777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d80b1220 con 0x7f43d8159880 2024-04-17T07:30:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43d7400640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d7a0 tx=0x7f43c000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.943+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0005070 con 0x7f43d8159880 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.944+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c0005210 con 0x7f43d8159880 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.944+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0007030 con 0x7f43d8159880 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.944+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d80b0c60 con 0x7f43d8159880 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.944+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d81e9410 con 0x7f43d8159880 2024-04-17T07:30:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.945+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d8068a10 con 0x7f43d8159880 2024-04-17T07:30:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.946+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f43c001a020 con 0x7f43d8159880 2024-04-17T07:30:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.947+0000 7f43cfe00640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 0x7f43b406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.947+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f43c008b650 con 0x7f43d8159880 2024-04-17T07:30:15.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.948+0000 7f43d7e00640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 0x7f43b406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.949+0000 7f43d7e00640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 0x7f43b406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f43d81f8130 tx=0x7f43c8009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:15.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:15.951+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f43c0059850 con 0x7f43d8159880 2024-04-17T07:30:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.195+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f43d806e9c0 con 0x7f43d8159880 2024-04-17T07:30:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.688+0000 7f43cfe00640 1 -- 172.21.15.67:0/573816777 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7f43c0059150 con 0x7f43d8159880 2024-04-17T07:30:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 msgr2=0x7f43b406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 0x7f43b406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f43d81f8130 tx=0x7f43c8009040 comp rx=0 tx=0).stop 2024-04-17T07:30:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 msgr2=0x7f43d80c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f43c000d7a0 tx=0x7f43c000dc60 comp rx=0 tx=0).stop 2024-04-17T07:30:16.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 shutdown_connections 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f43b406ca10 0x7f43b406eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43d80c6d70 0x7f43d80c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d8159880 0x7f43d80c6830 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 --2- 172.21.15.67:0/573816777 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43bc004d40 0x7f43d80b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 >> 172.21.15.67:0/573816777 conn(0x7f43d80b4090 msgr2=0x7f43d815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 shutdown_connections 2024-04-17T07:30:16.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.692+0000 7f43dd600640 1 -- 172.21.15.67:0/573816777 wait complete. 2024-04-17T07:30:16.715 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:30:16.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.938+0000 7ff130c00640 1 Processor -- start 2024-04-17T07:30:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.939+0000 7ff130c00640 1 -- start start 2024-04-17T07:30:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff130c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff130c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c0d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff130c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12c0cfe20 con 0x7ff12c0b9f00 2024-04-17T07:30:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff130c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c0d1c50 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff130c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0cf640 con 0x7ff12c0d12a0 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff123e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff123e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35490/0 (socket says 172.21.15.67:35490) 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.940+0000 7ff123e00640 1 -- 172.21.15.67:0/1983132636 learned_addr learned my addr 172.21.15.67:0/1983132636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff12be00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c0d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff123e00640 1 -- 172.21.15.67:0/1983132636 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 msgr2=0x7ff12c0d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff123e00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c0d1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff123e00640 1 -- 172.21.15.67:0/1983132636 >> v1:172.21.15.67:6789/0 conn(0x7ff12c0b9f00 legacy=0x7ff12c1634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff123e00640 1 -- 172.21.15.67:0/1983132636 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c0d0100 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.941+0000 7ff12be00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c0d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.942+0000 7ff123e00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff114009870 tx=0x7ff11402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5543c05c51064e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.942+0000 7ff12aa00640 1 -- 172.21.15.67:0/1983132636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114035020 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.942+0000 7ff12aa00640 1 -- 172.21.15.67:0/1983132636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff114031640 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.942+0000 7ff12aa00640 1 -- 172.21.15.67:0/1983132636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11402d040 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 -- 172.21.15.67:0/1983132636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0cf8f0 msgr2=0x7ff12c0b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff114009870 tx=0x7ff11402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 -- 172.21.15.67:0/1983132636 shutdown_connections 2024-04-17T07:30:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c0d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 --2- 172.21.15.67:0/1983132636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c0b7f80 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 -- 172.21.15.67:0/1983132636 >> 172.21.15.67:0/1983132636 conn(0x7ff12c06cb60 msgr2=0x7ff12c06cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 -- 172.21.15.67:0/1983132636 shutdown_connections 2024-04-17T07:30:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.943+0000 7ff130c00640 1 -- 172.21.15.67:0/1983132636 wait complete. 2024-04-17T07:30:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.945+0000 7ff130c00640 1 Processor -- start 2024-04-17T07:30:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.945+0000 7ff130c00640 1 -- start start 2024-04-17T07:30:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c161b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0d3420 con 0x7ff12c0b9f00 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0b84c0 con 0x7ff12c0cf8f0 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff130c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0ba2e0 con 0x7ff12c0d12a0 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff12b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff12b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35498/0 (socket says 172.21.15.67:35498) 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.946+0000 7ff12b400640 1 -- 172.21.15.67:0/2342965432 learned_addr learned my addr 172.21.15.67:0/2342965432 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 -- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 msgr2=0x7ff12c161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 -- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff12c0cf8f0 msgr2=0x7ff12c1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 -- 172.21.15.67:0/2342965432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c1504b0 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff12b400640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00c3c0 tx=0x7ff11c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0052d0 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c005490 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c007260 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.947+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c0c90a0 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.948+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c0c9660 con 0x7ff12c0b9f00 2024-04-17T07:30:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.949+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff12c0755d0 con 0x7ff12c0b9f00 2024-04-17T07:30:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.950+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7ff11c007400 con 0x7ff12c0b9f00 2024-04-17T07:30:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.951+0000 7ff123400640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 0x7ff10806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.951+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7ff11c08b130 con 0x7ff12c0b9f00 2024-04-17T07:30:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.952+0000 7ff123e00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 0x7ff10806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.953+0000 7ff123e00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 0x7ff10806eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff114036000 tx=0x7ff114030000 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:16.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:16.956+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff12c1504b0 con 0x7ff12c0b9f00 2024-04-17T07:30:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.199+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff12c0d2a20 con 0x7ff12c0b9f00 2024-04-17T07:30:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.200+0000 7ff123400640 1 -- 172.21.15.67:0/2342965432 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7ff11c0594e0 con 0x7ff12c0b9f00 2024-04-17T07:30:17.199 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:17.199 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.202+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 msgr2=0x7ff10806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.202+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 0x7ff10806eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff114036000 tx=0x7ff114030000 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.202+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 msgr2=0x7ff12c14ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.202+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00c3c0 tx=0x7ff11c00c880 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 shutdown_connections 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff12c0d12a0 0x7ff12c161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7ff10806ca10 0x7ff10806eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff12c0cf8f0 0x7ff12c1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 --2- 172.21.15.67:0/2342965432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff12c0b9f00 0x7ff12c14ff70 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 >> 172.21.15.67:0/2342965432 conn(0x7ff12c06cb60 msgr2=0x7ff12c0cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 shutdown_connections 2024-04-17T07:30:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.203+0000 7ff130c00640 1 -- 172.21.15.67:0/2342965432 wait complete. 2024-04-17T07:30:17.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.446+0000 7f522be00640 1 Processor -- start 2024-04-17T07:30:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.447+0000 7f522be00640 1 -- start start 2024-04-17T07:30:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.448+0000 7f522be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c15ec60 0x7f522c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.448+0000 7f522be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522c0b6b30 0x7f522c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.448+0000 7f522be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522c0d2d30 con 0x7f522c0cf910 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.448+0000 7f522be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f522c0d49c0 con 0x7f522c0b6b30 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.448+0000 7f522be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f522c0d3660 con 0x7f522c15ec60 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522c0b6b30 0x7f522c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f522c0b6b30 0x7f522c0cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35502/0 (socket says 172.21.15.67:35502) 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 -- 172.21.15.67:0/2565109227 learned_addr learned my addr 172.21.15.67:0/2565109227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f522aa00640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c15ec60 0x7f522c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 -- 172.21.15.67:0/2565109227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c15ec60 msgr2=0x7f522c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c15ec60 0x7f522c0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 -- 172.21.15.67:0/2565109227 >> v1:172.21.15.67:6789/0 conn(0x7f522c0cf910 legacy=0x7f522c06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.449+0000 7f5222000640 1 -- 172.21.15.67:0/2565109227 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c0d3010 con 0x7f522c0b6b30 2024-04-17T07:30:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f5222000640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c0cf3d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5214009870 tx=0x7f521402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b746486f7164c7ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f522a000640 1 -- 172.21.15.67:0/2565109227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214034070 con 0x7f522c0b6b30 2024-04-17T07:30:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f522a000640 1 -- 172.21.15.67:0/2565109227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5214030840 con 0x7f522c0b6b30 2024-04-17T07:30:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f522a000640 1 -- 172.21.15.67:0/2565109227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214030c10 con 0x7f522c0b6b30 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f522be00640 1 -- 172.21.15.67:0/2565109227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 msgr2=0x7f522c0cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.451+0000 7f522be00640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c0cf3d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5214009870 tx=0x7f521402b980 comp rx=0 tx=0).stop 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 -- 172.21.15.67:0/2565109227 shutdown_connections 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c0cf3d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 --2- 172.21.15.67:0/2565109227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c15ec60 0x7f522c0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 -- 172.21.15.67:0/2565109227 >> 172.21.15.67:0/2565109227 conn(0x7f522c0d15f0 msgr2=0x7f522c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 -- 172.21.15.67:0/2565109227 shutdown_connections 2024-04-17T07:30:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.452+0000 7f522be00640 1 -- 172.21.15.67:0/2565109227 wait complete. 2024-04-17T07:30:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.454+0000 7f522be00640 1 Processor -- start 2024-04-17T07:30:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.454+0000 7f522be00640 1 -- start start 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c0cf910 0x7f522c14d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 0x7f522c0b9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0cfda0 con 0x7f522c0b6b30 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0b6f70 con 0x7f522c15ec60 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f522c15f040 con 0x7f522c0cf910 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35514/0 (socket says 172.21.15.67:35514) 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.455+0000 7f522aa00640 1 -- 172.21.15.67:0/3577646904 learned_addr learned my addr 172.21.15.67:0/3577646904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 -- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c0cf910 msgr2=0x7f522c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f5222000640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c0cf910 0x7f522c14d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c0cf910 0x7f522c14d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 -- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 msgr2=0x7f522c0b9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522b400640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 0x7f522c0b9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 0x7f522c0b9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 -- 172.21.15.67:0/3577646904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c14e010 con 0x7f522c0b6b30 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522b400640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 0x7f522c0b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f522aa00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f521c00d5f0 tx=0x7f521c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0125f0 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c019070 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.456+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0129d0 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.457+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c14e2f0 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.457+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c14e8b0 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.458+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c068a10 con 0x7f522c0b6b30 2024-04-17T07:30:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.459+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f521c012b70 con 0x7f522c0b6b30 2024-04-17T07:30:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.460+0000 7f5223400640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 0x7f520806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.460+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f521c08c130 con 0x7f522c0b6b30 2024-04-17T07:30:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.460+0000 7f5222000640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 0x7f520806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.462+0000 7f5222000640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 0x7f520806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403b040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.465+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f521c05a440 con 0x7f522c0b6b30 2024-04-17T07:30:17.545 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T07:30:17.545+0000 7fdac7000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:30:17.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.670+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f522c0d2600 con 0x7f522c0b6b30 2024-04-17T07:30:17.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.673+0000 7f5223400640 1 -- 172.21.15.67:0/3577646904 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+16620 (secure 0 0 0) 0x7f521c059d40 con 0x7f522c0b6b30 2024-04-17T07:30:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:17.673 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":98,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:14.623116+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:17.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 msgr2=0x7f520806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 0x7f520806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403b040 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 msgr2=0x7f522c150940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f521c00d5f0 tx=0x7f521c00dab0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 shutdown_connections 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f520806ca10 0x7f520806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f522c15ec60 0x7f522c0b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f522c0cf910 0x7f522c14d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 --2- 172.21.15.67:0/3577646904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f522c0b6b30 0x7f522c150940 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 >> 172.21.15.67:0/3577646904 conn(0x7f522c0d15f0 msgr2=0x7f522c15f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 shutdown_connections 2024-04-17T07:30:17.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.677+0000 7f522be00640 1 -- 172.21.15.67:0/3577646904 wait complete. 2024-04-17T07:30:17.695 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:30:17.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:30:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.920+0000 7f13ed600640 1 Processor -- start 2024-04-17T07:30:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.920+0000 7f13ed600640 1 -- start start 2024-04-17T07:30:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13ed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13e8160280 0x7f13e806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13ed600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13e8141ed0 0x7f13e8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13ed600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e80cd4a0 con 0x7f13e806b8c0 2024-04-17T07:30:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13ed600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f13e815ce30 con 0x7f13e8160280 2024-04-17T07:30:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13ed600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f13e81607f0 con 0x7f13e8141ed0 2024-04-17T07:30:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13e7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f13e806b8c0 0x7f13e80cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39948/0 (socket says 172.21.15.67:39948) 2024-04-17T07:30:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.922+0000 7f13e7e00640 1 -- 172.21.15.67:0/1017778119 learned_addr learned my addr 172.21.15.67:0/1017778119 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:17.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13dfe00640 1 --2- 172.21.15.67:0/1017778119 >> v2:172.21.15.67:3300/0 conn(0x7f13e8160280 0x7f13e806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13ecc00640 1 --2- 172.21.15.67:0/1017778119 >> v2:172.21.15.187:3300/0 conn(0x7f13e8141ed0 0x7f13e8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13e7400640 1 -- 172.21.15.67:0/1017778119 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874152276 0 0) 0x7f13e80cd4a0 con 0x7f13e806b8c0 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13e7400640 1 -- 172.21.15.67:0/1017778119 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13cc003520 con 0x7f13e806b8c0 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13dfe00640 1 -- 172.21.15.67:0/1017778119 >> v2:172.21.15.187:3300/0 conn(0x7f13e8141ed0 msgr2=0x7f13e8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13dfe00640 1 --2- 172.21.15.67:0/1017778119 >> v2:172.21.15.187:3300/0 conn(0x7f13e8141ed0 0x7f13e8142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13dfe00640 1 -- 172.21.15.67:0/1017778119 >> v1:172.21.15.67:6789/0 conn(0x7f13e806b8c0 legacy=0x7f13e80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.923+0000 7f13dfe00640 1 -- 172.21.15.67:0/1017778119 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e8160ad0 con 0x7f13e8160280 2024-04-17T07:30:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.924+0000 7f13dfe00640 1 --2- 172.21.15.67:0/1017778119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e806bca0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f13e80b2120 tx=0x7f13d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b82adf8d2f1bea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.924+0000 7f13e7400640 1 -- 172.21.15.67:0/1017778119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0035020 con 0x7f13e8160280 2024-04-17T07:30:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13e7400640 1 -- 172.21.15.67:0/1017778119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0031830 con 0x7f13e8160280 2024-04-17T07:30:17.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13e7400640 1 -- 172.21.15.67:0/1017778119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0031c00 con 0x7f13e8160280 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 -- 172.21.15.67:0/1017778119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 msgr2=0x7f13e806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 --2- 172.21.15.67:0/1017778119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e806bca0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f13e80b2120 tx=0x7f13d002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 -- 172.21.15.67:0/1017778119 shutdown_connections 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 --2- 172.21.15.67:0/1017778119 >> v2:172.21.15.187:3300/0 conn(0x7f13e8141ed0 0x7f13e8142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 --2- 172.21.15.67:0/1017778119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e806bca0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 -- 172.21.15.67:0/1017778119 >> 172.21.15.67:0/1017778119 conn(0x7f13e8067d10 msgr2=0x7f13e80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 -- 172.21.15.67:0/1017778119 shutdown_connections 2024-04-17T07:30:17.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.925+0000 7f13ed600640 1 -- 172.21.15.67:0/1017778119 wait complete. 2024-04-17T07:30:17.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.927+0000 7f13ed600640 1 Processor -- start 2024-04-17T07:30:17.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.927+0000 7f13ed600640 1 -- start start 2024-04-17T07:30:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 0x7f13e8159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 0x7f13e8155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13e8161090 con 0x7f13e8160280 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f13e806c1e0 con 0x7f13e806b8c0 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.928+0000 7f13ed600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13e813f9b0 con 0x7f13e8141ed0 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35526/0 (socket says 172.21.15.67:35526) 2024-04-17T07:30:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 -- 172.21.15.67:0/2187940577 learned_addr learned my addr 172.21.15.67:0/2187940577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13dfe00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 0x7f13e8155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13e7e00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 0x7f13e8159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 -- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 msgr2=0x7f13e8155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 0x7f13e8155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 -- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 msgr2=0x7f13e8159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 0x7f13e8159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 -- 172.21.15.67:0/2187940577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e81587b0 con 0x7f13e8160280 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13dfe00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 0x7f13e8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13e7e00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 0x7f13e8159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.929+0000 7f13ecc00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f13e0007e40 tx=0x7f13e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.930+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0014620 con 0x7f13e8160280 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.930+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e0019070 con 0x7f13e8160280 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.930+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0014a20 con 0x7f13e8160280 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.930+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e81581f0 con 0x7f13e8160280 2024-04-17T07:30:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.930+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e81444f0 con 0x7f13e8160280 2024-04-17T07:30:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.931+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e8068a10 con 0x7f13e8160280 2024-04-17T07:30:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.932+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f13e0014bc0 con 0x7f13e8160280 2024-04-17T07:30:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.933+0000 7f13e4c00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 0x7f13c406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.933+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f13e008c540 con 0x7f13e8160280 2024-04-17T07:30:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.934+0000 7f13e7e00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 0x7f13c406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.935+0000 7f13e7e00640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 0x7f13c406ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13d8005880 tx=0x7f13d8009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:17.937+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f13e005a8f0 con 0x7f13e8160280 2024-04-17T07:30:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.139+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 --> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13e8141450 con 0x7f13c406ca40 2024-04-17T07:30:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.163+0000 7f13e4c00640 1 -- 172.21.15.67:0/2187940577 <== mgr.5794 v2:172.21.15.187:6832/3743052516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252016 (secure 0 0 0) 0x7f13e8141450 con 0x7f13c406ca40 2024-04-17T07:30:18.166 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:30:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 msgr2=0x7f13c406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 0x7f13c406ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13d8005880 tx=0x7f13d8009040 comp rx=0 tx=0).stop 2024-04-17T07:30:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 msgr2=0x7f13e80b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f13e0007e40 tx=0x7f13e000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 shutdown_connections 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e8160280 0x7f13e80b4290 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13e8141ed0 0x7f13e8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f13c406ca40 0x7f13c406ef00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 --2- 172.21.15.67:0/2187940577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13e806b8c0 0x7f13e8159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 >> 172.21.15.67:0/2187940577 conn(0x7f13e8067d10 msgr2=0x7f13e80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 shutdown_connections 2024-04-17T07:30:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.170+0000 7f13ed600640 1 -- 172.21.15.67:0/2187940577 wait complete. 2024-04-17T07:30:18.189 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-17T07:30:17.825155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324960,"kb_used_data":10280,"kb_used_omap":12,"kb_used_meta":314547,"kb_avail":754649760,"statfs":{"total":773094113280,"available":772761354240,"internally_reserved":0,"allocated":10526720,"data_stored":5620272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":322096688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002112"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678500+0000","last_change":"2024-04-17T07:30:13.669171+0000","last_active":"2024-04-17T07:30:13.678500+0000","last_peered":"2024-04-17T07:30:13.678500+0000","last_clean":"2024-04-17T07:30:13.678500+0000","last_became_active":"2024-04-17T07:30:13.668533+0000","last_became_peered":"2024-04-17T07:30:13.668533+0000","last_unstale":"2024-04-17T07:30:13.678500+0000","last_undegraded":"2024-04-17T07:30:13.678500+0000","last_fullsized":"2024-04-17T07:30:13.678500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:50.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.674094+0000","last_change":"2024-04-17T07:30:13.669267+0000","last_active":"2024-04-17T07:30:13.674094+0000","last_peered":"2024-04-17T07:30:13.674094+0000","last_clean":"2024-04-17T07:30:13.674094+0000","last_became_active":"2024-04-17T07:30:13.666694+0000","last_became_peered":"2024-04-17T07:30:13.666694+0000","last_unstale":"2024-04-17T07:30:13.674094+0000","last_undegraded":"2024-04-17T07:30:13.674094+0000","last_fullsized":"2024-04-17T07:30:13.674094+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:51.645485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658831+0000","last_change":"2024-04-17T07:30:13.655889+0000","last_active":"2024-04-17T07:30:13.658831+0000","last_peered":"2024-04-17T07:30:13.658831+0000","last_clean":"2024-04-17T07:30:13.658831+0000","last_became_active":"2024-04-17T07:30:13.655348+0000","last_became_peered":"2024-04-17T07:30:13.655348+0000","last_unstale":"2024-04-17T07:30:13.658831+0000","last_undegraded":"2024-04-17T07:30:13.658831+0000","last_fullsized":"2024-04-17T07:30:13.658831+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.205842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664719+0000","last_change":"2024-04-17T07:30:13.664065+0000","last_active":"2024-04-17T07:30:13.664719+0000","last_peered":"2024-04-17T07:30:13.664719+0000","last_clean":"2024-04-17T07:30:13.664719+0000","last_became_active":"2024-04-17T07:30:13.663775+0000","last_became_peered":"2024-04-17T07:30:13.663775+0000","last_unstale":"2024-04-17T07:30:13.664719+0000","last_undegraded":"2024-04-17T07:30:13.664719+0000","last_fullsized":"2024-04-17T07:30:13.664719+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:32.756772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.670895+0000","last_change":"2024-04-17T07:30:13.667872+0000","last_active":"2024-04-17T07:30:13.670895+0000","last_peered":"2024-04-17T07:30:13.670895+0000","last_clean":"2024-04-17T07:30:13.670895+0000","last_became_active":"2024-04-17T07:30:13.667522+0000","last_became_peered":"2024-04-17T07:30:13.667522+0000","last_unstale":"2024-04-17T07:30:13.670895+0000","last_undegraded":"2024-04-17T07:30:13.670895+0000","last_fullsized":"2024-04-17T07:30:13.670895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:16.746091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671700+0000","last_change":"2024-04-17T07:30:13.665710+0000","last_active":"2024-04-17T07:30:13.671700+0000","last_peered":"2024-04-17T07:30:13.671700+0000","last_clean":"2024-04-17T07:30:13.671700+0000","last_became_active":"2024-04-17T07:30:13.665387+0000","last_became_peered":"2024-04-17T07:30:13.665387+0000","last_unstale":"2024-04-17T07:30:13.671700+0000","last_undegraded":"2024-04-17T07:30:13.671700+0000","last_fullsized":"2024-04-17T07:30:13.671700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:08.533915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.654553+0000","last_change":"2024-04-17T07:30:13.648062+0000","last_active":"2024-04-17T07:30:13.654553+0000","last_peered":"2024-04-17T07:30:13.654553+0000","last_clean":"2024-04-17T07:30:13.654553+0000","last_became_active":"2024-04-17T07:30:13.647564+0000","last_became_peered":"2024-04-17T07:30:13.647564+0000","last_unstale":"2024-04-17T07:30:13.654553+0000","last_undegraded":"2024-04-17T07:30:13.654553+0000","last_fullsized":"2024-04-17T07:30:13.654553+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:19.693998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672083+0000","last_change":"2024-04-17T07:30:13.660078+0000","last_active":"2024-04-17T07:30:13.672083+0000","last_peered":"2024-04-17T07:30:13.672083+0000","last_clean":"2024-04-17T07:30:13.672083+0000","last_became_active":"2024-04-17T07:30:13.659614+0000","last_became_peered":"2024-04-17T07:30:13.659614+0000","last_unstale":"2024-04-17T07:30:13.672083+0000","last_undegraded":"2024-04-17T07:30:13.672083+0000","last_fullsized":"2024-04-17T07:30:13.672083+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:15.506289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659234+0000","last_change":"2024-04-17T07:30:13.647760+0000","last_active":"2024-04-17T07:30:13.659234+0000","last_peered":"2024-04-17T07:30:13.659234+0000","last_clean":"2024-04-17T07:30:13.659234+0000","last_became_active":"2024-04-17T07:30:13.647464+0000","last_became_peered":"2024-04-17T07:30:13.647464+0000","last_unstale":"2024-04-17T07:30:13.659234+0000","last_undegraded":"2024-04-17T07:30:13.659234+0000","last_fullsized":"2024-04-17T07:30:13.659234+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:55.369293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662552+0000","last_change":"2024-04-17T07:30:13.656408+0000","last_active":"2024-04-17T07:30:13.662552+0000","last_peered":"2024-04-17T07:30:13.662552+0000","last_clean":"2024-04-17T07:30:13.662552+0000","last_became_active":"2024-04-17T07:30:13.656087+0000","last_became_peered":"2024-04-17T07:30:13.656087+0000","last_unstale":"2024-04-17T07:30:13.662552+0000","last_undegraded":"2024-04-17T07:30:13.662552+0000","last_fullsized":"2024-04-17T07:30:13.662552+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:55.143383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678842+0000","last_change":"2024-04-17T07:30:13.669775+0000","last_active":"2024-04-17T07:30:13.678842+0000","last_peered":"2024-04-17T07:30:13.678842+0000","last_clean":"2024-04-17T07:30:13.678842+0000","last_became_active":"2024-04-17T07:30:13.667975+0000","last_became_peered":"2024-04-17T07:30:13.667975+0000","last_unstale":"2024-04-17T07:30:13.678842+0000","last_undegraded":"2024-04-17T07:30:13.678842+0000","last_fullsized":"2024-04-17T07:30:13.678842+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:11.540131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659531+0000","last_change":"2024-04-17T07:30:13.657903+0000","last_active":"2024-04-17T07:30:13.659531+0000","last_peered":"2024-04-17T07:30:13.659531+0000","last_clean":"2024-04-17T07:30:13.659531+0000","last_became_active":"2024-04-17T07:30:13.657607+0000","last_became_peered":"2024-04-17T07:30:13.657607+0000","last_unstale":"2024-04-17T07:30:13.659531+0000","last_undegraded":"2024-04-17T07:30:13.659531+0000","last_fullsized":"2024-04-17T07:30:13.659531+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:53.759703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671375+0000","last_change":"2024-04-17T07:30:13.669070+0000","last_active":"2024-04-17T07:30:13.671375+0000","last_peered":"2024-04-17T07:30:13.671375+0000","last_clean":"2024-04-17T07:30:13.671375+0000","last_became_active":"2024-04-17T07:30:13.668518+0000","last_became_peered":"2024-04-17T07:30:13.668518+0000","last_unstale":"2024-04-17T07:30:13.671375+0000","last_undegraded":"2024-04-17T07:30:13.671375+0000","last_fullsized":"2024-04-17T07:30:13.671375+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:22.861234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.660631+0000","last_change":"2024-04-17T07:30:13.650894+0000","last_active":"2024-04-17T07:30:13.660631+0000","last_peered":"2024-04-17T07:30:13.660631+0000","last_clean":"2024-04-17T07:30:13.660631+0000","last_became_active":"2024-04-17T07:30:13.649940+0000","last_became_peered":"2024-04-17T07:30:13.649940+0000","last_unstale":"2024-04-17T07:30:13.660631+0000","last_undegraded":"2024-04-17T07:30:13.660631+0000","last_fullsized":"2024-04-17T07:30:13.660631+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:10.515398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644991+0000","last_change":"2024-04-17T07:30:13.640177+0000","last_active":"2024-04-17T07:30:13.644991+0000","last_peered":"2024-04-17T07:30:13.644991+0000","last_clean":"2024-04-17T07:30:13.644991+0000","last_became_active":"2024-04-17T07:30:13.639859+0000","last_became_peered":"2024-04-17T07:30:13.639859+0000","last_unstale":"2024-04-17T07:30:13.644991+0000","last_undegraded":"2024-04-17T07:30:13.644991+0000","last_fullsized":"2024-04-17T07:30:13.644991+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:58.328130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.676238+0000","last_change":"2024-04-17T07:30:13.669587+0000","last_active":"2024-04-17T07:30:13.676238+0000","last_peered":"2024-04-17T07:30:13.676238+0000","last_clean":"2024-04-17T07:30:13.676238+0000","last_became_active":"2024-04-17T07:30:13.669204+0000","last_became_peered":"2024-04-17T07:30:13.669204+0000","last_unstale":"2024-04-17T07:30:13.676238+0000","last_undegraded":"2024-04-17T07:30:13.676238+0000","last_fullsized":"2024-04-17T07:30:13.676238+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:54.721472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659216+0000","last_change":"2024-04-17T07:30:13.647730+0000","last_active":"2024-04-17T07:30:13.659216+0000","last_peered":"2024-04-17T07:30:13.659216+0000","last_clean":"2024-04-17T07:30:13.659216+0000","last_became_active":"2024-04-17T07:30:13.647246+0000","last_became_peered":"2024-04-17T07:30:13.647246+0000","last_unstale":"2024-04-17T07:30:13.659216+0000","last_undegraded":"2024-04-17T07:30:13.659216+0000","last_fullsized":"2024-04-17T07:30:13.659216+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:09.100360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.675896+0000","last_change":"2024-04-17T07:30:13.675200+0000","last_active":"2024-04-17T07:30:13.675896+0000","last_peered":"2024-04-17T07:30:13.675896+0000","last_clean":"2024-04-17T07:30:13.675896+0000","last_became_active":"2024-04-17T07:30:13.674731+0000","last_became_peered":"2024-04-17T07:30:13.674731+0000","last_unstale":"2024-04-17T07:30:13.675896+0000","last_undegraded":"2024-04-17T07:30:13.675896+0000","last_fullsized":"2024-04-17T07:30:13.675896+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:00.893354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672327+0000","last_change":"2024-04-17T07:30:13.665805+0000","last_active":"2024-04-17T07:30:13.672327+0000","last_peered":"2024-04-17T07:30:13.672327+0000","last_clean":"2024-04-17T07:30:13.672327+0000","last_became_active":"2024-04-17T07:30:13.662003+0000","last_became_peered":"2024-04-17T07:30:13.662003+0000","last_unstale":"2024-04-17T07:30:13.672327+0000","last_undegraded":"2024-04-17T07:30:13.672327+0000","last_fullsized":"2024-04-17T07:30:13.672327+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:00:10.160119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652660+0000","last_change":"2024-04-17T07:30:13.649298+0000","last_active":"2024-04-17T07:30:13.652660+0000","last_peered":"2024-04-17T07:30:13.652660+0000","last_clean":"2024-04-17T07:30:13.652660+0000","last_became_active":"2024-04-17T07:30:13.648993+0000","last_became_peered":"2024-04-17T07:30:13.648993+0000","last_unstale":"2024-04-17T07:30:13.652660+0000","last_undegraded":"2024-04-17T07:30:13.652660+0000","last_fullsized":"2024-04-17T07:30:13.652660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:26.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.663127+0000","last_change":"2024-04-17T07:30:13.658063+0000","last_active":"2024-04-17T07:30:13.663127+0000","last_peered":"2024-04-17T07:30:13.663127+0000","last_clean":"2024-04-17T07:30:13.663127+0000","last_became_active":"2024-04-17T07:30:13.653621+0000","last_became_peered":"2024-04-17T07:30:13.653621+0000","last_unstale":"2024-04-17T07:30:13.663127+0000","last_undegraded":"2024-04-17T07:30:13.663127+0000","last_fullsized":"2024-04-17T07:30:13.663127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:59.528501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.660853+0000","last_change":"2024-04-17T07:30:13.654547+0000","last_active":"2024-04-17T07:30:13.660853+0000","last_peered":"2024-04-17T07:30:13.660853+0000","last_clean":"2024-04-17T07:30:13.660853+0000","last_became_active":"2024-04-17T07:30:13.654254+0000","last_became_peered":"2024-04-17T07:30:13.654254+0000","last_unstale":"2024-04-17T07:30:13.660853+0000","last_undegraded":"2024-04-17T07:30:13.660853+0000","last_fullsized":"2024-04-17T07:30:13.660853+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:39.218608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678406+0000","last_change":"2024-04-17T07:30:13.669107+0000","last_active":"2024-04-17T07:30:13.678406+0000","last_peered":"2024-04-17T07:30:13.678406+0000","last_clean":"2024-04-17T07:30:13.678406+0000","last_became_active":"2024-04-17T07:30:13.668387+0000","last_became_peered":"2024-04-17T07:30:13.668387+0000","last_unstale":"2024-04-17T07:30:13.678406+0000","last_undegraded":"2024-04-17T07:30:13.678406+0000","last_fullsized":"2024-04-17T07:30:13.678406+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:08:21.335506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679024+0000","last_change":"2024-04-17T07:30:13.674745+0000","last_active":"2024-04-17T07:30:13.679024+0000","last_peered":"2024-04-17T07:30:13.679024+0000","last_clean":"2024-04-17T07:30:13.679024+0000","last_became_active":"2024-04-17T07:30:13.674200+0000","last_became_peered":"2024-04-17T07:30:13.674200+0000","last_unstale":"2024-04-17T07:30:13.679024+0000","last_undegraded":"2024-04-17T07:30:13.679024+0000","last_fullsized":"2024-04-17T07:30:13.679024+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.205842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678636+0000","last_change":"2024-04-17T07:30:13.673963+0000","last_active":"2024-04-17T07:30:13.678636+0000","last_peered":"2024-04-17T07:30:13.678636+0000","last_clean":"2024-04-17T07:30:13.678636+0000","last_became_active":"2024-04-17T07:30:13.673685+0000","last_became_peered":"2024-04-17T07:30:13.673685+0000","last_unstale":"2024-04-17T07:30:13.678636+0000","last_undegraded":"2024-04-17T07:30:13.678636+0000","last_fullsized":"2024-04-17T07:30:13.678636+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:40.684090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.681020+0000","last_change":"2024-04-17T07:30:13.675871+0000","last_active":"2024-04-17T07:30:13.681020+0000","last_peered":"2024-04-17T07:30:13.681020+0000","last_clean":"2024-04-17T07:30:13.681020+0000","last_became_active":"2024-04-17T07:30:13.674702+0000","last_became_peered":"2024-04-17T07:30:13.674702+0000","last_unstale":"2024-04-17T07:30:13.681020+0000","last_undegraded":"2024-04-17T07:30:13.681020+0000","last_fullsized":"2024-04-17T07:30:13.681020+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:04:13.131885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669859+0000","last_change":"2024-04-17T07:30:13.667510+0000","last_active":"2024-04-17T07:30:13.669859+0000","last_peered":"2024-04-17T07:30:13.669859+0000","last_clean":"2024-04-17T07:30:13.669859+0000","last_became_active":"2024-04-17T07:30:13.667213+0000","last_became_peered":"2024-04-17T07:30:13.667213+0000","last_unstale":"2024-04-17T07:30:13.669859+0000","last_undegraded":"2024-04-17T07:30:13.669859+0000","last_fullsized":"2024-04-17T07:30:13.669859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:39.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.653321+0000","last_change":"2024-04-17T07:30:13.648762+0000","last_active":"2024-04-17T07:30:13.653321+0000","last_peered":"2024-04-17T07:30:13.653321+0000","last_clean":"2024-04-17T07:30:13.653321+0000","last_became_active":"2024-04-17T07:30:13.648195+0000","last_became_peered":"2024-04-17T07:30:13.648195+0000","last_unstale":"2024-04-17T07:30:13.653321+0000","last_undegraded":"2024-04-17T07:30:13.653321+0000","last_fullsized":"2024-04-17T07:30:13.653321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:57.128893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658824+0000","last_change":"2024-04-17T07:30:13.656087+0000","last_active":"2024-04-17T07:30:13.658824+0000","last_peered":"2024-04-17T07:30:13.658824+0000","last_clean":"2024-04-17T07:30:13.658824+0000","last_became_active":"2024-04-17T07:30:13.655801+0000","last_became_peered":"2024-04-17T07:30:13.655801+0000","last_unstale":"2024-04-17T07:30:13.658824+0000","last_undegraded":"2024-04-17T07:30:13.658824+0000","last_fullsized":"2024-04-17T07:30:13.658824+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:14.338883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678700+0000","last_change":"2024-04-17T07:30:13.674886+0000","last_active":"2024-04-17T07:30:13.678700+0000","last_peered":"2024-04-17T07:30:13.678700+0000","last_clean":"2024-04-17T07:30:13.678700+0000","last_became_active":"2024-04-17T07:30:13.674489+0000","last_became_peered":"2024-04-17T07:30:13.674489+0000","last_unstale":"2024-04-17T07:30:13.678700+0000","last_undegraded":"2024-04-17T07:30:13.678700+0000","last_fullsized":"2024-04-17T07:30:13.678700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:09.690617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.654413+0000","last_change":"2024-04-17T07:30:13.646770+0000","last_active":"2024-04-17T07:30:13.654413+0000","last_peered":"2024-04-17T07:30:13.654413+0000","last_clean":"2024-04-17T07:30:13.654413+0000","last_became_active":"2024-04-17T07:30:13.645793+0000","last_became_peered":"2024-04-17T07:30:13.645793+0000","last_unstale":"2024-04-17T07:30:13.654413+0000","last_undegraded":"2024-04-17T07:30:13.654413+0000","last_fullsized":"2024-04-17T07:30:13.654413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:02.808778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672111+0000","last_change":"2024-04-17T07:30:13.660716+0000","last_active":"2024-04-17T07:30:13.672111+0000","last_peered":"2024-04-17T07:30:13.672111+0000","last_clean":"2024-04-17T07:30:13.672111+0000","last_became_active":"2024-04-17T07:30:13.660410+0000","last_became_peered":"2024-04-17T07:30:13.660410+0000","last_unstale":"2024-04-17T07:30:13.672111+0000","last_undegraded":"2024-04-17T07:30:13.672111+0000","last_fullsized":"2024-04-17T07:30:13.672111+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:01.315245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678929+0000","last_change":"2024-04-17T07:30:13.673514+0000","last_active":"2024-04-17T07:30:13.678929+0000","last_peered":"2024-04-17T07:30:13.678929+0000","last_clean":"2024-04-17T07:30:13.678929+0000","last_became_active":"2024-04-17T07:30:13.672881+0000","last_became_peered":"2024-04-17T07:30:13.672881+0000","last_unstale":"2024-04-17T07:30:13.678929+0000","last_undegraded":"2024-04-17T07:30:13.678929+0000","last_fullsized":"2024-04-17T07:30:13.678929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:09.983999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"97'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.668351+0000","last_change":"2024-04-17T07:30:12.657902+0000","last_active":"2024-04-17T07:30:13.668351+0000","last_peered":"2024-04-17T07:30:13.668351+0000","last_clean":"2024-04-17T07:30:13.668351+0000","last_became_active":"2024-04-17T07:30:12.657500+0000","last_became_peered":"2024-04-17T07:30:12.657500+0000","last_unstale":"2024-04-17T07:30:13.668351+0000","last_undegraded":"2024-04-17T07:30:13.668351+0000","last_fullsized":"2024-04-17T07:30:13.668351+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:01.322054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659195+0000","last_change":"2024-04-17T07:30:13.647547+0000","last_active":"2024-04-17T07:30:13.659195+0000","last_peered":"2024-04-17T07:30:13.659195+0000","last_clean":"2024-04-17T07:30:13.659195+0000","last_became_active":"2024-04-17T07:30:13.647080+0000","last_became_peered":"2024-04-17T07:30:13.647080+0000","last_unstale":"2024-04-17T07:30:13.659195+0000","last_undegraded":"2024-04-17T07:30:13.659195+0000","last_fullsized":"2024-04-17T07:30:13.659195+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:26.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662824+0000","last_change":"2024-04-17T07:30:12.655187+0000","last_active":"2024-04-17T07:30:13.662824+0000","last_peered":"2024-04-17T07:30:13.662824+0000","last_clean":"2024-04-17T07:30:13.662824+0000","last_became_active":"2024-04-17T07:30:12.654903+0000","last_became_peered":"2024-04-17T07:30:12.654903+0000","last_unstale":"2024-04-17T07:30:13.662824+0000","last_undegraded":"2024-04-17T07:30:13.662824+0000","last_fullsized":"2024-04-17T07:30:13.662824+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:33.201476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679195+0000","last_change":"2024-04-17T07:30:13.675693+0000","last_active":"2024-04-17T07:30:13.679195+0000","last_peered":"2024-04-17T07:30:13.679195+0000","last_clean":"2024-04-17T07:30:13.679195+0000","last_became_active":"2024-04-17T07:30:13.674987+0000","last_became_peered":"2024-04-17T07:30:13.674987+0000","last_unstale":"2024-04-17T07:30:13.679195+0000","last_undegraded":"2024-04-17T07:30:13.679195+0000","last_fullsized":"2024-04-17T07:30:13.679195+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:51.805698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678227+0000","last_change":"2024-04-17T07:30:12.658562+0000","last_active":"2024-04-17T07:30:13.678227+0000","last_peered":"2024-04-17T07:30:13.678227+0000","last_clean":"2024-04-17T07:30:13.678227+0000","last_became_active":"2024-04-17T07:30:12.658241+0000","last_became_peered":"2024-04-17T07:30:12.658241+0000","last_unstale":"2024-04-17T07:30:13.678227+0000","last_undegraded":"2024-04-17T07:30:13.678227+0000","last_fullsized":"2024-04-17T07:30:13.678227+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:18.826813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678869+0000","last_change":"2024-04-17T07:30:13.670975+0000","last_active":"2024-04-17T07:30:13.678869+0000","last_peered":"2024-04-17T07:30:13.678869+0000","last_clean":"2024-04-17T07:30:13.678869+0000","last_became_active":"2024-04-17T07:30:13.669421+0000","last_became_peered":"2024-04-17T07:30:13.669421+0000","last_unstale":"2024-04-17T07:30:13.678869+0000","last_undegraded":"2024-04-17T07:30:13.678869+0000","last_fullsized":"2024-04-17T07:30:13.678869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:57.823931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677608+0000","last_change":"2024-04-17T07:30:12.655238+0000","last_active":"2024-04-17T07:30:13.677608+0000","last_peered":"2024-04-17T07:30:13.677608+0000","last_clean":"2024-04-17T07:30:13.677608+0000","last_became_active":"2024-04-17T07:30:12.654932+0000","last_became_peered":"2024-04-17T07:30:12.654932+0000","last_unstale":"2024-04-17T07:30:13.677608+0000","last_undegraded":"2024-04-17T07:30:13.677608+0000","last_fullsized":"2024-04-17T07:30:13.677608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:31.282535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658886+0000","last_change":"2024-04-17T07:30:13.649608+0000","last_active":"2024-04-17T07:30:13.658886+0000","last_peered":"2024-04-17T07:30:13.658886+0000","last_clean":"2024-04-17T07:30:13.658886+0000","last_became_active":"2024-04-17T07:30:13.648550+0000","last_became_peered":"2024-04-17T07:30:13.648550+0000","last_unstale":"2024-04-17T07:30:13.658886+0000","last_undegraded":"2024-04-17T07:30:13.658886+0000","last_fullsized":"2024-04-17T07:30:13.658886+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:39.995919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664958+0000","last_change":"2024-04-17T07:30:12.649158+0000","last_active":"2024-04-17T07:30:13.664958+0000","last_peered":"2024-04-17T07:30:13.664958+0000","last_clean":"2024-04-17T07:30:13.664958+0000","last_became_active":"2024-04-17T07:30:12.648678+0000","last_became_peered":"2024-04-17T07:30:12.648678+0000","last_unstale":"2024-04-17T07:30:13.664958+0000","last_undegraded":"2024-04-17T07:30:13.664958+0000","last_fullsized":"2024-04-17T07:30:13.664958+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:24.764730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.665999+0000","last_change":"2024-04-17T07:30:13.663624+0000","last_active":"2024-04-17T07:30:13.665999+0000","last_peered":"2024-04-17T07:30:13.665999+0000","last_clean":"2024-04-17T07:30:13.665999+0000","last_became_active":"2024-04-17T07:30:13.663286+0000","last_became_peered":"2024-04-17T07:30:13.663286+0000","last_unstale":"2024-04-17T07:30:13.665999+0000","last_undegraded":"2024-04-17T07:30:13.665999+0000","last_fullsized":"2024-04-17T07:30:13.665999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:56.469111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679160+0000","last_change":"2024-04-17T07:30:12.659376+0000","last_active":"2024-04-17T07:30:13.679160+0000","last_peered":"2024-04-17T07:30:13.679160+0000","last_clean":"2024-04-17T07:30:13.679160+0000","last_became_active":"2024-04-17T07:30:12.659077+0000","last_became_peered":"2024-04-17T07:30:12.659077+0000","last_unstale":"2024-04-17T07:30:13.679160+0000","last_undegraded":"2024-04-17T07:30:13.679160+0000","last_fullsized":"2024-04-17T07:30:13.679160+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:12.132627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.661155+0000","last_change":"2024-04-17T07:30:13.657914+0000","last_active":"2024-04-17T07:30:13.661155+0000","last_peered":"2024-04-17T07:30:13.661155+0000","last_clean":"2024-04-17T07:30:13.661155+0000","last_became_active":"2024-04-17T07:30:13.656819+0000","last_became_peered":"2024-04-17T07:30:13.656819+0000","last_unstale":"2024-04-17T07:30:13.661155+0000","last_undegraded":"2024-04-17T07:30:13.661155+0000","last_fullsized":"2024-04-17T07:30:13.661155+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:08.997705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.632983+0000","last_change":"2024-04-17T07:30:12.656941+0000","last_active":"2024-04-17T07:30:13.632983+0000","last_peered":"2024-04-17T07:30:13.632983+0000","last_clean":"2024-04-17T07:30:13.632983+0000","last_became_active":"2024-04-17T07:30:12.656633+0000","last_became_peered":"2024-04-17T07:30:12.656633+0000","last_unstale":"2024-04-17T07:30:13.632983+0000","last_undegraded":"2024-04-17T07:30:13.632983+0000","last_fullsized":"2024-04-17T07:30:13.632983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:02.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664448+0000","last_change":"2024-04-17T07:30:13.663839+0000","last_active":"2024-04-17T07:30:13.664448+0000","last_peered":"2024-04-17T07:30:13.664448+0000","last_clean":"2024-04-17T07:30:13.664448+0000","last_became_active":"2024-04-17T07:30:13.663202+0000","last_became_peered":"2024-04-17T07:30:13.663202+0000","last_unstale":"2024-04-17T07:30:13.664448+0000","last_undegraded":"2024-04-17T07:30:13.664448+0000","last_fullsized":"2024-04-17T07:30:13.664448+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:09.100360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.638318+0000","last_change":"2024-04-17T07:30:12.651522+0000","last_active":"2024-04-17T07:30:13.638318+0000","last_peered":"2024-04-17T07:30:13.638318+0000","last_clean":"2024-04-17T07:30:13.638318+0000","last_became_active":"2024-04-17T07:30:12.651212+0000","last_became_peered":"2024-04-17T07:30:12.651212+0000","last_unstale":"2024-04-17T07:30:13.638318+0000","last_undegraded":"2024-04-17T07:30:13.638318+0000","last_fullsized":"2024-04-17T07:30:13.638318+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:16:37.898883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.676480+0000","last_change":"2024-04-17T07:30:13.668864+0000","last_active":"2024-04-17T07:30:13.676480+0000","last_peered":"2024-04-17T07:30:13.676480+0000","last_clean":"2024-04-17T07:30:13.676480+0000","last_became_active":"2024-04-17T07:30:13.667245+0000","last_became_peered":"2024-04-17T07:30:13.667245+0000","last_unstale":"2024-04-17T07:30:13.676480+0000","last_undegraded":"2024-04-17T07:30:13.676480+0000","last_fullsized":"2024-04-17T07:30:13.676480+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:40.562710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644618+0000","last_change":"2024-04-17T07:30:12.646128+0000","last_active":"2024-04-17T07:30:13.644618+0000","last_peered":"2024-04-17T07:30:13.644618+0000","last_clean":"2024-04-17T07:30:13.644618+0000","last_became_active":"2024-04-17T07:30:12.643925+0000","last_became_peered":"2024-04-17T07:30:12.643925+0000","last_unstale":"2024-04-17T07:30:13.644618+0000","last_undegraded":"2024-04-17T07:30:13.644618+0000","last_fullsized":"2024-04-17T07:30:13.644618+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:26.665319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678890+0000","last_change":"2024-04-17T07:30:13.673312+0000","last_active":"2024-04-17T07:30:13.678890+0000","last_peered":"2024-04-17T07:30:13.678890+0000","last_clean":"2024-04-17T07:30:13.678890+0000","last_became_active":"2024-04-17T07:30:13.673041+0000","last_became_peered":"2024-04-17T07:30:13.673041+0000","last_unstale":"2024-04-17T07:30:13.678890+0000","last_undegraded":"2024-04-17T07:30:13.678890+0000","last_fullsized":"2024-04-17T07:30:13.678890+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:25.782507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.657616+0000","last_change":"2024-04-17T07:30:12.654905+0000","last_active":"2024-04-17T07:30:13.657616+0000","last_peered":"2024-04-17T07:30:13.657616+0000","last_clean":"2024-04-17T07:30:13.657616+0000","last_became_active":"2024-04-17T07:30:12.651462+0000","last_became_peered":"2024-04-17T07:30:12.651462+0000","last_unstale":"2024-04-17T07:30:13.657616+0000","last_undegraded":"2024-04-17T07:30:13.657616+0000","last_fullsized":"2024-04-17T07:30:13.657616+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:17.301348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.673999+0000","last_change":"2024-04-17T07:30:13.667226+0000","last_active":"2024-04-17T07:30:13.673999+0000","last_peered":"2024-04-17T07:30:13.673999+0000","last_clean":"2024-04-17T07:30:13.673999+0000","last_became_active":"2024-04-17T07:30:13.666881+0000","last_became_peered":"2024-04-17T07:30:13.666881+0000","last_unstale":"2024-04-17T07:30:13.673999+0000","last_undegraded":"2024-04-17T07:30:13.673999+0000","last_fullsized":"2024-04-17T07:30:13.673999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:26:10.482433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662777+0000","last_change":"2024-04-17T07:30:12.653752+0000","last_active":"2024-04-17T07:30:13.662777+0000","last_peered":"2024-04-17T07:30:13.662777+0000","last_clean":"2024-04-17T07:30:13.662777+0000","last_became_active":"2024-04-17T07:30:12.653082+0000","last_became_peered":"2024-04-17T07:30:12.653082+0000","last_unstale":"2024-04-17T07:30:13.662777+0000","last_undegraded":"2024-04-17T07:30:13.662777+0000","last_fullsized":"2024-04-17T07:30:13.662777+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:55.660313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.650615+0000","last_change":"2024-04-17T07:30:13.645185+0000","last_active":"2024-04-17T07:30:13.650615+0000","last_peered":"2024-04-17T07:30:13.650615+0000","last_clean":"2024-04-17T07:30:13.650615+0000","last_became_active":"2024-04-17T07:30:13.644874+0000","last_became_peered":"2024-04-17T07:30:13.644874+0000","last_unstale":"2024-04-17T07:30:13.650615+0000","last_undegraded":"2024-04-17T07:30:13.650615+0000","last_fullsized":"2024-04-17T07:30:13.650615+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:34.219253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"97'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.667070+0000","last_change":"2024-04-17T07:30:12.655810+0000","last_active":"2024-04-17T07:30:13.667070+0000","last_peered":"2024-04-17T07:30:13.667070+0000","last_clean":"2024-04-17T07:30:13.667070+0000","last_became_active":"2024-04-17T07:30:12.655483+0000","last_became_peered":"2024-04-17T07:30:12.655483+0000","last_unstale":"2024-04-17T07:30:13.667070+0000","last_undegraded":"2024-04-17T07:30:13.667070+0000","last_fullsized":"2024-04-17T07:30:13.667070+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:23:23.121888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678040+0000","last_change":"2024-04-17T07:30:13.668303+0000","last_active":"2024-04-17T07:30:13.678040+0000","last_peered":"2024-04-17T07:30:13.678040+0000","last_clean":"2024-04-17T07:30:13.678040+0000","last_became_active":"2024-04-17T07:30:13.667668+0000","last_became_peered":"2024-04-17T07:30:13.667668+0000","last_unstale":"2024-04-17T07:30:13.678040+0000","last_undegraded":"2024-04-17T07:30:13.678040+0000","last_fullsized":"2024-04-17T07:30:13.678040+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:59:42.830490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669573+0000","last_change":"2024-04-17T07:30:12.656865+0000","last_active":"2024-04-17T07:30:13.669573+0000","last_peered":"2024-04-17T07:30:13.669573+0000","last_clean":"2024-04-17T07:30:13.669573+0000","last_became_active":"2024-04-17T07:30:12.656431+0000","last_became_peered":"2024-04-17T07:30:12.656431+0000","last_unstale":"2024-04-17T07:30:13.669573+0000","last_undegraded":"2024-04-17T07:30:13.669573+0000","last_fullsized":"2024-04-17T07:30:13.669573+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:10:09.600827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659131+0000","last_change":"2024-04-17T07:30:13.652616+0000","last_active":"2024-04-17T07:30:13.659131+0000","last_peered":"2024-04-17T07:30:13.659131+0000","last_clean":"2024-04-17T07:30:13.659131+0000","last_became_active":"2024-04-17T07:30:13.652240+0000","last_became_peered":"2024-04-17T07:30:13.652240+0000","last_unstale":"2024-04-17T07:30:13.659131+0000","last_undegraded":"2024-04-17T07:30:13.659131+0000","last_fullsized":"2024-04-17T07:30:13.659131+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:57.742073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662763+0000","last_change":"2024-04-17T07:30:12.654008+0000","last_active":"2024-04-17T07:30:13.662763+0000","last_peered":"2024-04-17T07:30:13.662763+0000","last_clean":"2024-04-17T07:30:13.662763+0000","last_became_active":"2024-04-17T07:30:12.650771+0000","last_became_peered":"2024-04-17T07:30:12.650771+0000","last_unstale":"2024-04-17T07:30:13.662763+0000","last_undegraded":"2024-04-17T07:30:13.662763+0000","last_fullsized":"2024-04-17T07:30:13.662763+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:47.823913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671291+0000","last_change":"2024-04-17T07:30:13.668776+0000","last_active":"2024-04-17T07:30:13.671291+0000","last_peered":"2024-04-17T07:30:13.671291+0000","last_clean":"2024-04-17T07:30:13.671291+0000","last_became_active":"2024-04-17T07:30:13.668443+0000","last_became_peered":"2024-04-17T07:30:13.668443+0000","last_unstale":"2024-04-17T07:30:13.671291+0000","last_undegraded":"2024-04-17T07:30:13.671291+0000","last_fullsized":"2024-04-17T07:30:13.671291+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:55.369293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.635196+0000","last_change":"2024-04-17T07:30:12.648100+0000","last_active":"2024-04-17T07:30:13.635196+0000","last_peered":"2024-04-17T07:30:13.635196+0000","last_clean":"2024-04-17T07:30:13.635196+0000","last_became_active":"2024-04-17T07:30:12.647810+0000","last_became_peered":"2024-04-17T07:30:12.647810+0000","last_unstale":"2024-04-17T07:30:13.635196+0000","last_undegraded":"2024-04-17T07:30:13.635196+0000","last_fullsized":"2024-04-17T07:30:13.635196+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:10.244003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.649943+0000","last_change":"2024-04-17T07:30:13.645406+0000","last_active":"2024-04-17T07:30:13.649943+0000","last_peered":"2024-04-17T07:30:13.649943+0000","last_clean":"2024-04-17T07:30:13.649943+0000","last_became_active":"2024-04-17T07:30:13.645067+0000","last_became_peered":"2024-04-17T07:30:13.645067+0000","last_unstale":"2024-04-17T07:30:13.649943+0000","last_undegraded":"2024-04-17T07:30:13.649943+0000","last_fullsized":"2024-04-17T07:30:13.649943+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:58.328130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"97'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679452+0000","last_change":"2024-04-17T07:30:12.653114+0000","last_active":"2024-04-17T07:30:13.679452+0000","last_peered":"2024-04-17T07:30:13.679452+0000","last_clean":"2024-04-17T07:30:13.679452+0000","last_became_active":"2024-04-17T07:30:12.652238+0000","last_became_peered":"2024-04-17T07:30:12.652238+0000","last_unstale":"2024-04-17T07:30:13.679452+0000","last_undegraded":"2024-04-17T07:30:13.679452+0000","last_fullsized":"2024-04-17T07:30:13.679452+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:34.825945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659466+0000","last_change":"2024-04-17T07:30:13.649456+0000","last_active":"2024-04-17T07:30:13.659466+0000","last_peered":"2024-04-17T07:30:13.659466+0000","last_clean":"2024-04-17T07:30:13.659466+0000","last_became_active":"2024-04-17T07:30:13.648987+0000","last_became_peered":"2024-04-17T07:30:13.648987+0000","last_unstale":"2024-04-17T07:30:13.659466+0000","last_undegraded":"2024-04-17T07:30:13.659466+0000","last_fullsized":"2024-04-17T07:30:13.659466+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:25.985751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679557+0000","last_change":"2024-04-17T07:30:12.658068+0000","last_active":"2024-04-17T07:30:13.679557+0000","last_peered":"2024-04-17T07:30:13.679557+0000","last_clean":"2024-04-17T07:30:13.679557+0000","last_became_active":"2024-04-17T07:30:12.657511+0000","last_became_peered":"2024-04-17T07:30:12.657511+0000","last_unstale":"2024-04-17T07:30:13.679557+0000","last_undegraded":"2024-04-17T07:30:13.679557+0000","last_fullsized":"2024-04-17T07:30:13.679557+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:54.489242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678865+0000","last_change":"2024-04-17T07:30:13.669432+0000","last_active":"2024-04-17T07:30:13.678865+0000","last_peered":"2024-04-17T07:30:13.678865+0000","last_clean":"2024-04-17T07:30:13.678865+0000","last_became_active":"2024-04-17T07:30:13.668882+0000","last_became_peered":"2024-04-17T07:30:13.668882+0000","last_unstale":"2024-04-17T07:30:13.678865+0000","last_undegraded":"2024-04-17T07:30:13.678865+0000","last_fullsized":"2024-04-17T07:30:13.678865+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:46:45.505751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677933+0000","last_change":"2024-04-17T07:30:12.654105+0000","last_active":"2024-04-17T07:30:13.677933+0000","last_peered":"2024-04-17T07:30:13.677933+0000","last_clean":"2024-04-17T07:30:13.677933+0000","last_became_active":"2024-04-17T07:30:12.653826+0000","last_became_peered":"2024-04-17T07:30:12.653826+0000","last_unstale":"2024-04-17T07:30:13.677933+0000","last_undegraded":"2024-04-17T07:30:13.677933+0000","last_fullsized":"2024-04-17T07:30:13.677933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:34.328109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.656029+0000","last_change":"2024-04-17T07:30:13.650355+0000","last_active":"2024-04-17T07:30:13.656029+0000","last_peered":"2024-04-17T07:30:13.656029+0000","last_clean":"2024-04-17T07:30:13.656029+0000","last_became_active":"2024-04-17T07:30:13.648213+0000","last_became_peered":"2024-04-17T07:30:13.648213+0000","last_unstale":"2024-04-17T07:30:13.656029+0000","last_undegraded":"2024-04-17T07:30:13.656029+0000","last_fullsized":"2024-04-17T07:30:13.656029+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:12.182242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679155+0000","last_change":"2024-04-17T07:30:12.656280+0000","last_active":"2024-04-17T07:30:13.679155+0000","last_peered":"2024-04-17T07:30:13.679155+0000","last_clean":"2024-04-17T07:30:13.679155+0000","last_became_active":"2024-04-17T07:30:12.653013+0000","last_became_peered":"2024-04-17T07:30:12.653013+0000","last_unstale":"2024-04-17T07:30:13.679155+0000","last_undegraded":"2024-04-17T07:30:13.679155+0000","last_fullsized":"2024-04-17T07:30:13.679155+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:32.316524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658562+0000","last_change":"2024-04-17T07:30:13.650130+0000","last_active":"2024-04-17T07:30:13.658562+0000","last_peered":"2024-04-17T07:30:13.658562+0000","last_clean":"2024-04-17T07:30:13.658562+0000","last_became_active":"2024-04-17T07:30:13.648619+0000","last_became_peered":"2024-04-17T07:30:13.648619+0000","last_unstale":"2024-04-17T07:30:13.658562+0000","last_undegraded":"2024-04-17T07:30:13.658562+0000","last_fullsized":"2024-04-17T07:30:13.658562+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:22.861234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652996+0000","last_change":"2024-04-17T07:30:12.655148+0000","last_active":"2024-04-17T07:30:13.652996+0000","last_peered":"2024-04-17T07:30:13.652996+0000","last_clean":"2024-04-17T07:30:13.652996+0000","last_became_active":"2024-04-17T07:30:12.654883+0000","last_became_peered":"2024-04-17T07:30:12.654883+0000","last_unstale":"2024-04-17T07:30:13.652996+0000","last_undegraded":"2024-04-17T07:30:13.652996+0000","last_fullsized":"2024-04-17T07:30:13.652996+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:00.519603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.649052+0000","last_change":"2024-04-17T07:30:13.646685+0000","last_active":"2024-04-17T07:30:13.649052+0000","last_peered":"2024-04-17T07:30:13.649052+0000","last_clean":"2024-04-17T07:30:13.649052+0000","last_became_active":"2024-04-17T07:30:13.645474+0000","last_became_peered":"2024-04-17T07:30:13.645474+0000","last_unstale":"2024-04-17T07:30:13.649052+0000","last_undegraded":"2024-04-17T07:30:13.649052+0000","last_fullsized":"2024-04-17T07:30:13.649052+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:09.983999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662932+0000","last_change":"2024-04-17T07:30:12.655925+0000","last_active":"2024-04-17T07:30:13.662932+0000","last_peered":"2024-04-17T07:30:13.662932+0000","last_clean":"2024-04-17T07:30:13.662932+0000","last_became_active":"2024-04-17T07:30:12.655632+0000","last_became_peered":"2024-04-17T07:30:12.655632+0000","last_unstale":"2024-04-17T07:30:13.662932+0000","last_undegraded":"2024-04-17T07:30:13.662932+0000","last_fullsized":"2024-04-17T07:30:13.662932+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:23.143787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671799+0000","last_change":"2024-04-17T07:30:13.669204+0000","last_active":"2024-04-17T07:30:13.671799+0000","last_peered":"2024-04-17T07:30:13.671799+0000","last_clean":"2024-04-17T07:30:13.671799+0000","last_became_active":"2024-04-17T07:30:13.668319+0000","last_became_peered":"2024-04-17T07:30:13.668319+0000","last_unstale":"2024-04-17T07:30:13.671799+0000","last_undegraded":"2024-04-17T07:30:13.671799+0000","last_fullsized":"2024-04-17T07:30:13.671799+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:52.619149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.635749+0000","last_change":"2024-04-17T07:30:12.645417+0000","last_active":"2024-04-17T07:30:13.635749+0000","last_peered":"2024-04-17T07:30:13.635749+0000","last_clean":"2024-04-17T07:30:13.635749+0000","last_became_active":"2024-04-17T07:30:12.642333+0000","last_became_peered":"2024-04-17T07:30:12.642333+0000","last_unstale":"2024-04-17T07:30:13.635749+0000","last_undegraded":"2024-04-17T07:30:13.635749+0000","last_fullsized":"2024-04-17T07:30:13.635749+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:31.384463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672197+0000","last_change":"2024-04-17T07:30:13.670675+0000","last_active":"2024-04-17T07:30:13.672197+0000","last_peered":"2024-04-17T07:30:13.672197+0000","last_clean":"2024-04-17T07:30:13.672197+0000","last_became_active":"2024-04-17T07:30:13.670430+0000","last_became_peered":"2024-04-17T07:30:13.670430+0000","last_unstale":"2024-04-17T07:30:13.672197+0000","last_undegraded":"2024-04-17T07:30:13.672197+0000","last_fullsized":"2024-04-17T07:30:13.672197+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:38:23.868600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.627757+0000","last_change":"2024-04-17T07:30:12.650955+0000","last_active":"2024-04-17T07:30:13.627757+0000","last_peered":"2024-04-17T07:30:13.627757+0000","last_clean":"2024-04-17T07:30:13.627757+0000","last_became_active":"2024-04-17T07:30:12.650676+0000","last_became_peered":"2024-04-17T07:30:12.650676+0000","last_unstale":"2024-04-17T07:30:13.627757+0000","last_undegraded":"2024-04-17T07:30:13.627757+0000","last_fullsized":"2024-04-17T07:30:13.627757+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:55.648288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.651589+0000","last_change":"2024-04-17T07:30:13.646440+0000","last_active":"2024-04-17T07:30:13.651589+0000","last_peered":"2024-04-17T07:30:13.651589+0000","last_clean":"2024-04-17T07:30:13.651589+0000","last_became_active":"2024-04-17T07:30:13.646167+0000","last_became_peered":"2024-04-17T07:30:13.646167+0000","last_unstale":"2024-04-17T07:30:13.651589+0000","last_undegraded":"2024-04-17T07:30:13.651589+0000","last_fullsized":"2024-04-17T07:30:13.651589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:57.635005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679423+0000","last_change":"2024-04-17T07:30:12.656371+0000","last_active":"2024-04-17T07:30:13.679423+0000","last_peered":"2024-04-17T07:30:13.679423+0000","last_clean":"2024-04-17T07:30:13.679423+0000","last_became_active":"2024-04-17T07:30:12.655692+0000","last_became_peered":"2024-04-17T07:30:12.655692+0000","last_unstale":"2024-04-17T07:30:13.679423+0000","last_undegraded":"2024-04-17T07:30:13.679423+0000","last_fullsized":"2024-04-17T07:30:13.679423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:13.149975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652058+0000","last_change":"2024-04-17T07:30:13.644622+0000","last_active":"2024-04-17T07:30:13.652058+0000","last_peered":"2024-04-17T07:30:13.652058+0000","last_clean":"2024-04-17T07:30:13.652058+0000","last_became_active":"2024-04-17T07:30:13.643777+0000","last_became_peered":"2024-04-17T07:30:13.643777+0000","last_unstale":"2024-04-17T07:30:13.652058+0000","last_undegraded":"2024-04-17T07:30:13.652058+0000","last_fullsized":"2024-04-17T07:30:13.652058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:15.279395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.637106+0000","last_change":"2024-04-17T07:30:12.653927+0000","last_active":"2024-04-17T07:30:13.637106+0000","last_peered":"2024-04-17T07:30:13.637106+0000","last_clean":"2024-04-17T07:30:13.637106+0000","last_became_active":"2024-04-17T07:30:12.652224+0000","last_became_peered":"2024-04-17T07:30:12.652224+0000","last_unstale":"2024-04-17T07:30:13.637106+0000","last_undegraded":"2024-04-17T07:30:13.637106+0000","last_fullsized":"2024-04-17T07:30:13.637106+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:59.980592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679208+0000","last_change":"2024-04-17T07:30:13.671196+0000","last_active":"2024-04-17T07:30:13.679208+0000","last_peered":"2024-04-17T07:30:13.679208+0000","last_clean":"2024-04-17T07:30:13.679208+0000","last_became_active":"2024-04-17T07:30:13.670825+0000","last_became_peered":"2024-04-17T07:30:13.670825+0000","last_unstale":"2024-04-17T07:30:13.679208+0000","last_undegraded":"2024-04-17T07:30:13.679208+0000","last_fullsized":"2024-04-17T07:30:13.679208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:56.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.639395+0000","last_change":"2024-04-17T07:30:12.648039+0000","last_active":"2024-04-17T07:30:13.639395+0000","last_peered":"2024-04-17T07:30:13.639395+0000","last_clean":"2024-04-17T07:30:13.639395+0000","last_became_active":"2024-04-17T07:30:12.647761+0000","last_became_peered":"2024-04-17T07:30:12.647761+0000","last_unstale":"2024-04-17T07:30:13.639395+0000","last_undegraded":"2024-04-17T07:30:13.639395+0000","last_fullsized":"2024-04-17T07:30:13.639395+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:55.648288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678891+0000","last_change":"2024-04-17T07:30:13.671045+0000","last_active":"2024-04-17T07:30:13.678891+0000","last_peered":"2024-04-17T07:30:13.678891+0000","last_clean":"2024-04-17T07:30:13.678891+0000","last_became_active":"2024-04-17T07:30:13.669552+0000","last_became_peered":"2024-04-17T07:30:13.669552+0000","last_unstale":"2024-04-17T07:30:13.678891+0000","last_undegraded":"2024-04-17T07:30:13.678891+0000","last_fullsized":"2024-04-17T07:30:13.678891+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:12.897536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.629227+0000","last_change":"2024-04-17T07:30:12.648394+0000","last_active":"2024-04-17T07:30:13.629227+0000","last_peered":"2024-04-17T07:30:13.629227+0000","last_clean":"2024-04-17T07:30:13.629227+0000","last_became_active":"2024-04-17T07:30:12.648115+0000","last_became_peered":"2024-04-17T07:30:12.648115+0000","last_unstale":"2024-04-17T07:30:13.629227+0000","last_undegraded":"2024-04-17T07:30:13.629227+0000","last_fullsized":"2024-04-17T07:30:13.629227+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:17.639100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.656825+0000","last_change":"2024-04-17T07:30:13.646481+0000","last_active":"2024-04-17T07:30:13.656825+0000","last_peered":"2024-04-17T07:30:13.656825+0000","last_clean":"2024-04-17T07:30:13.656825+0000","last_became_active":"2024-04-17T07:30:13.646191+0000","last_became_peered":"2024-04-17T07:30:13.646191+0000","last_unstale":"2024-04-17T07:30:13.656825+0000","last_undegraded":"2024-04-17T07:30:13.656825+0000","last_fullsized":"2024-04-17T07:30:13.656825+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:57.635005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.640292+0000","last_change":"2024-04-17T07:30:12.647508+0000","last_active":"2024-04-17T07:30:13.640292+0000","last_peered":"2024-04-17T07:30:13.640292+0000","last_clean":"2024-04-17T07:30:13.640292+0000","last_became_active":"2024-04-17T07:30:12.647203+0000","last_became_peered":"2024-04-17T07:30:12.647203+0000","last_unstale":"2024-04-17T07:30:13.640292+0000","last_undegraded":"2024-04-17T07:30:13.640292+0000","last_fullsized":"2024-04-17T07:30:13.640292+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:17.639100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678767+0000","last_change":"2024-04-17T07:30:13.675054+0000","last_active":"2024-04-17T07:30:13.678767+0000","last_peered":"2024-04-17T07:30:13.678767+0000","last_clean":"2024-04-17T07:30:13.678767+0000","last_became_active":"2024-04-17T07:30:13.672833+0000","last_became_peered":"2024-04-17T07:30:13.672833+0000","last_unstale":"2024-04-17T07:30:13.678767+0000","last_undegraded":"2024-04-17T07:30:13.678767+0000","last_fullsized":"2024-04-17T07:30:13.678767+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:14:42.001194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.634491+0000","last_change":"2024-04-17T07:30:12.650282+0000","last_active":"2024-04-17T07:30:13.634491+0000","last_peered":"2024-04-17T07:30:13.634491+0000","last_clean":"2024-04-17T07:30:13.634491+0000","last_became_active":"2024-04-17T07:30:12.649660+0000","last_became_peered":"2024-04-17T07:30:12.649660+0000","last_unstale":"2024-04-17T07:30:13.634491+0000","last_undegraded":"2024-04-17T07:30:13.634491+0000","last_fullsized":"2024-04-17T07:30:13.634491+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:07:45.129127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644783+0000","last_change":"2024-04-17T07:30:13.642151+0000","last_active":"2024-04-17T07:30:13.644783+0000","last_peered":"2024-04-17T07:30:13.644783+0000","last_clean":"2024-04-17T07:30:13.644783+0000","last_became_active":"2024-04-17T07:30:13.641296+0000","last_became_peered":"2024-04-17T07:30:13.641296+0000","last_unstale":"2024-04-17T07:30:13.644783+0000","last_undegraded":"2024-04-17T07:30:13.644783+0000","last_fullsized":"2024-04-17T07:30:13.644783+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:29.854636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.637938+0000","last_change":"2024-04-17T07:30:12.652338+0000","last_active":"2024-04-17T07:30:13.637938+0000","last_peered":"2024-04-17T07:30:13.637938+0000","last_clean":"2024-04-17T07:30:13.637938+0000","last_became_active":"2024-04-17T07:30:12.650501+0000","last_became_peered":"2024-04-17T07:30:12.650501+0000","last_unstale":"2024-04-17T07:30:13.637938+0000","last_undegraded":"2024-04-17T07:30:13.637938+0000","last_fullsized":"2024-04-17T07:30:13.637938+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:14.343231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669784+0000","last_change":"2024-04-17T07:30:13.664542+0000","last_active":"2024-04-17T07:30:13.669784+0000","last_peered":"2024-04-17T07:30:13.669784+0000","last_clean":"2024-04-17T07:30:13.669784+0000","last_became_active":"2024-04-17T07:30:13.664266+0000","last_became_peered":"2024-04-17T07:30:13.664266+0000","last_unstale":"2024-04-17T07:30:13.669784+0000","last_undegraded":"2024-04-17T07:30:13.669784+0000","last_fullsized":"2024-04-17T07:30:13.669784+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:56.883439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658305+0000","last_change":"2024-04-17T07:30:12.656053+0000","last_active":"2024-04-17T07:30:13.658305+0000","last_peered":"2024-04-17T07:30:13.658305+0000","last_clean":"2024-04-17T07:30:13.658305+0000","last_became_active":"2024-04-17T07:30:12.655766+0000","last_became_peered":"2024-04-17T07:30:12.655766+0000","last_unstale":"2024-04-17T07:30:13.658305+0000","last_undegraded":"2024-04-17T07:30:13.658305+0000","last_fullsized":"2024-04-17T07:30:13.658305+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:19.199156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671723+0000","last_change":"2024-04-17T07:30:13.669172+0000","last_active":"2024-04-17T07:30:13.671723+0000","last_peered":"2024-04-17T07:30:13.671723+0000","last_clean":"2024-04-17T07:30:13.671723+0000","last_became_active":"2024-04-17T07:30:13.668895+0000","last_became_peered":"2024-04-17T07:30:13.668895+0000","last_unstale":"2024-04-17T07:30:13.671723+0000","last_undegraded":"2024-04-17T07:30:13.671723+0000","last_fullsized":"2024-04-17T07:30:13.671723+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:12.621274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677796+0000","last_change":"2024-04-17T07:30:12.656824+0000","last_active":"2024-04-17T07:30:13.677796+0000","last_peered":"2024-04-17T07:30:13.677796+0000","last_clean":"2024-04-17T07:30:13.677796+0000","last_became_active":"2024-04-17T07:30:12.656476+0000","last_became_peered":"2024-04-17T07:30:12.656476+0000","last_unstale":"2024-04-17T07:30:13.677796+0000","last_undegraded":"2024-04-17T07:30:13.677796+0000","last_fullsized":"2024-04-17T07:30:13.677796+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:21.046479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"86'175","reported_seq":526,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.653005+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:30:13.653005+0000","last_peered":"2024-04-17T07:30:13.653005+0000","last_clean":"2024-04-17T07:30:13.653005+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:30:13.653005+0000","last_undegraded":"2024-04-17T07:30:13.653005+0000","last_fullsized":"2024-04-17T07:30:13.653005+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":1145310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542341,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542340,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542340,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58692,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313148,"statfs":{"total":96636764160,"available":96576663552,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542341,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542341,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40648,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331192,"statfs":{"total":96636764160,"available":96595140608,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50368,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321472,"statfs":{"total":96636764160,"available":96585187328,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1761280,"data_stored":1145310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:30:18.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:30:18.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.458+0000 7f34efa00640 1 Processor -- start 2024-04-17T07:30:18.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.459+0000 7f34efa00640 1 -- start start 2024-04-17T07:30:18.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.460+0000 7f34efa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34e806f670 0x7f34e806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.460+0000 7f34efa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34e80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.460+0000 7f34efa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e806ff90 con 0x7f34e80cd020 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.460+0000 7f34efa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f34e80ceb00 con 0x7f34e806f670 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.460+0000 7f34efa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34e80cede0 con 0x7f34e806f030 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34e7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34e80cd020 0x7f34e8071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39956/0 (socket says 172.21.15.67:39956) 2024-04-17T07:30:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34e7e00640 1 -- 172.21.15.67:0/634667100 learned_addr learned my addr 172.21.15.67:0/634667100 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34ed000640 1 --2- 172.21.15.67:0/634667100 >> v2:172.21.15.187:3300/0 conn(0x7f34e806f670 0x7f34e806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34eda00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34e80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364333624 0 0) 0x7f34e806ff90 con 0x7f34e80cd020 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.461+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34cc003520 con 0x7f34e80cd020 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3886088164 0 0) 0x7f34cc003520 con 0x7f34e80cd020 2024-04-17T07:30:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 msgr2=0x7f34e80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34e80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 >> v2:172.21.15.187:3300/0 conn(0x7f34e806f670 msgr2=0x7f34e806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 --2- 172.21.15.67:0/634667100 >> v2:172.21.15.187:3300/0 conn(0x7f34e806f670 0x7f34e806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e80cf0c0 con 0x7f34e80cd020 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f34d80019f0 con 0x7f34e80cd020 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 >> v1:172.21.15.67:6789/0 conn(0x7f34e80cd020 legacy=0x7f34e8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc0039b0 0x7f34cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34cc004800 0x7f34cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34e80ceb00 con 0x7f34cc004800 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34e80cede0 con 0x7f34cc0039b0 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc00b9c0 con 0x7f34e806f030 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.462+0000 7f34e7e00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34cc004800 0x7f34cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34ed000640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc0039b0 0x7f34cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34eda00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34e7e00640 1 -- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 msgr2=0x7f34cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34e7e00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e806f030 0x7f34cc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34e7e00640 1 -- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc0039b0 msgr2=0x7f34cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34e7e00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc0039b0 0x7f34cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34e7e00640 1 -- 172.21.15.67:0/634667100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc003520 con 0x7f34cc004800 2024-04-17T07:30:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.463+0000 7f34ed000640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc0039b0 0x7f34cc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.464+0000 7f34e7e00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34cc004800 0x7f34cc004c80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f34d80013d0 tx=0x7f34cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b22d9f4adb2901 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.464+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d800a410 con 0x7f34cc004800 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.464+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d8034630 con 0x7f34cc004800 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.464+0000 7f34e7400640 1 -- 172.21.15.67:0/634667100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8034a20 con 0x7f34cc004800 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 -- 172.21.15.67:0/634667100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34cc004800 msgr2=0x7f34cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 --2- 172.21.15.67:0/634667100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34cc004800 0x7f34cc004c80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f34d80013d0 tx=0x7f34cc005830 comp rx=0 tx=0).stop 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34eda00640 1 -- 172.21.15.67:0/634667100 reap_dead start 2024-04-17T07:30:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 -- 172.21.15.67:0/634667100 shutdown_connections 2024-04-17T07:30:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 -- 172.21.15.67:0/634667100 >> 172.21.15.67:0/634667100 conn(0x7f34e806bf50 msgr2=0x7f34e80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 -- 172.21.15.67:0/634667100 shutdown_connections 2024-04-17T07:30:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.465+0000 7f34efa00640 1 -- 172.21.15.67:0/634667100 wait complete. 2024-04-17T07:30:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.467+0000 7f34efa00640 1 Processor -- start 2024-04-17T07:30:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.468+0000 7f34efa00640 1 -- start start 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 0x7f34e80c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e814af30 0x7f34e81580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34e80ce720 con 0x7f34e81585e0 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34e80cf680 con 0x7f34cc004800 2024-04-17T07:30:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34efa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34e80ce8a0 con 0x7f34e814af30 2024-04-17T07:30:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35548/0 (socket says 172.21.15.67:35548) 2024-04-17T07:30:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.469+0000 7f34eda00640 1 -- 172.21.15.67:0/1546315667 learned_addr learned my addr 172.21.15.67:0/1546315667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34ed000640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 0x7f34e80c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34e7e00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e814af30 0x7f34e81580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 -- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e814af30 msgr2=0x7f34e81580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e814af30 0x7f34e81580a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 -- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 msgr2=0x7f34e80c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 0x7f34e80c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 -- 172.21.15.67:0/1546315667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e8157000 con 0x7f34e81585e0 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34ed000640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 0x7f34e80c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34eda00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f34e80ce1e0 tx=0x7f34e0010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e000f040 con 0x7f34e81585e0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e000e8c0 con 0x7f34e81585e0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e000ecb0 con 0x7f34e81585e0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.470+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e81572e0 con 0x7f34e81585e0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.471+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e81578a0 con 0x7f34e81585e0 2024-04-17T07:30:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.472+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e8068b90 con 0x7f34e81585e0 2024-04-17T07:30:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.473+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f34e00143f0 con 0x7f34e81585e0 2024-04-17T07:30:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.474+0000 7f34e4c00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 0x7f34c406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.474+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f34e008bbe0 con 0x7f34e81585e0 2024-04-17T07:30:18.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.475+0000 7f34ed000640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 0x7f34c406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:18.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.476+0000 7f34ed000640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 0x7f34c406ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009b20 tx=0x7f34d0008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.479+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34e0023030 con 0x7f34e81585e0 2024-04-17T07:30:18.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.679+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 --> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f34e8072550 con 0x7f34c406ca40 2024-04-17T07:30:18.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.700+0000 7f34e4c00640 1 -- 172.21.15.67:0/1546315667 <== mgr.5794 v2:172.21.15.187:6832/3743052516 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252016 (secure 0 0 0) 0x7f34e8072550 con 0x7f34c406ca40 2024-04-17T07:30:18.702 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:30:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 msgr2=0x7f34c406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 0x7f34c406ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009b20 tx=0x7f34d0008040 comp rx=0 tx=0).stop 2024-04-17T07:30:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 msgr2=0x7f34e814b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f34e80ce1e0 tx=0x7f34e0010c20 comp rx=0 tx=0).stop 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 shutdown_connections 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e81585e0 0x7f34e814b9a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e814af30 0x7f34e81580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f34c406ca40 0x7f34c406ef00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 --2- 172.21.15.67:0/1546315667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34cc004800 0x7f34e80c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 >> 172.21.15.67:0/1546315667 conn(0x7f34e806bf50 msgr2=0x7f34e80b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 shutdown_connections 2024-04-17T07:30:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:18.705+0000 7f34efa00640 1 -- 172.21.15.67:0/1546315667 wait complete. 2024-04-17T07:30:18.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-17T07:30:17.825155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":307,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324960,"kb_used_data":10280,"kb_used_omap":12,"kb_used_meta":314547,"kb_avail":754649760,"statfs":{"total":773094113280,"available":772761354240,"internally_reserved":0,"allocated":10526720,"data_stored":5620272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":322096688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002112"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678500+0000","last_change":"2024-04-17T07:30:13.669171+0000","last_active":"2024-04-17T07:30:13.678500+0000","last_peered":"2024-04-17T07:30:13.678500+0000","last_clean":"2024-04-17T07:30:13.678500+0000","last_became_active":"2024-04-17T07:30:13.668533+0000","last_became_peered":"2024-04-17T07:30:13.668533+0000","last_unstale":"2024-04-17T07:30:13.678500+0000","last_undegraded":"2024-04-17T07:30:13.678500+0000","last_fullsized":"2024-04-17T07:30:13.678500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:50.734086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.674094+0000","last_change":"2024-04-17T07:30:13.669267+0000","last_active":"2024-04-17T07:30:13.674094+0000","last_peered":"2024-04-17T07:30:13.674094+0000","last_clean":"2024-04-17T07:30:13.674094+0000","last_became_active":"2024-04-17T07:30:13.666694+0000","last_became_peered":"2024-04-17T07:30:13.666694+0000","last_unstale":"2024-04-17T07:30:13.674094+0000","last_undegraded":"2024-04-17T07:30:13.674094+0000","last_fullsized":"2024-04-17T07:30:13.674094+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:51.645485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658831+0000","last_change":"2024-04-17T07:30:13.655889+0000","last_active":"2024-04-17T07:30:13.658831+0000","last_peered":"2024-04-17T07:30:13.658831+0000","last_clean":"2024-04-17T07:30:13.658831+0000","last_became_active":"2024-04-17T07:30:13.655348+0000","last_became_peered":"2024-04-17T07:30:13.655348+0000","last_unstale":"2024-04-17T07:30:13.658831+0000","last_undegraded":"2024-04-17T07:30:13.658831+0000","last_fullsized":"2024-04-17T07:30:13.658831+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.205842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664719+0000","last_change":"2024-04-17T07:30:13.664065+0000","last_active":"2024-04-17T07:30:13.664719+0000","last_peered":"2024-04-17T07:30:13.664719+0000","last_clean":"2024-04-17T07:30:13.664719+0000","last_became_active":"2024-04-17T07:30:13.663775+0000","last_became_peered":"2024-04-17T07:30:13.663775+0000","last_unstale":"2024-04-17T07:30:13.664719+0000","last_undegraded":"2024-04-17T07:30:13.664719+0000","last_fullsized":"2024-04-17T07:30:13.664719+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:32.756772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.670895+0000","last_change":"2024-04-17T07:30:13.667872+0000","last_active":"2024-04-17T07:30:13.670895+0000","last_peered":"2024-04-17T07:30:13.670895+0000","last_clean":"2024-04-17T07:30:13.670895+0000","last_became_active":"2024-04-17T07:30:13.667522+0000","last_became_peered":"2024-04-17T07:30:13.667522+0000","last_unstale":"2024-04-17T07:30:13.670895+0000","last_undegraded":"2024-04-17T07:30:13.670895+0000","last_fullsized":"2024-04-17T07:30:13.670895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:16.746091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671700+0000","last_change":"2024-04-17T07:30:13.665710+0000","last_active":"2024-04-17T07:30:13.671700+0000","last_peered":"2024-04-17T07:30:13.671700+0000","last_clean":"2024-04-17T07:30:13.671700+0000","last_became_active":"2024-04-17T07:30:13.665387+0000","last_became_peered":"2024-04-17T07:30:13.665387+0000","last_unstale":"2024-04-17T07:30:13.671700+0000","last_undegraded":"2024-04-17T07:30:13.671700+0000","last_fullsized":"2024-04-17T07:30:13.671700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:08.533915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.654553+0000","last_change":"2024-04-17T07:30:13.648062+0000","last_active":"2024-04-17T07:30:13.654553+0000","last_peered":"2024-04-17T07:30:13.654553+0000","last_clean":"2024-04-17T07:30:13.654553+0000","last_became_active":"2024-04-17T07:30:13.647564+0000","last_became_peered":"2024-04-17T07:30:13.647564+0000","last_unstale":"2024-04-17T07:30:13.654553+0000","last_undegraded":"2024-04-17T07:30:13.654553+0000","last_fullsized":"2024-04-17T07:30:13.654553+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:19.693998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672083+0000","last_change":"2024-04-17T07:30:13.660078+0000","last_active":"2024-04-17T07:30:13.672083+0000","last_peered":"2024-04-17T07:30:13.672083+0000","last_clean":"2024-04-17T07:30:13.672083+0000","last_became_active":"2024-04-17T07:30:13.659614+0000","last_became_peered":"2024-04-17T07:30:13.659614+0000","last_unstale":"2024-04-17T07:30:13.672083+0000","last_undegraded":"2024-04-17T07:30:13.672083+0000","last_fullsized":"2024-04-17T07:30:13.672083+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:15.506289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659234+0000","last_change":"2024-04-17T07:30:13.647760+0000","last_active":"2024-04-17T07:30:13.659234+0000","last_peered":"2024-04-17T07:30:13.659234+0000","last_clean":"2024-04-17T07:30:13.659234+0000","last_became_active":"2024-04-17T07:30:13.647464+0000","last_became_peered":"2024-04-17T07:30:13.647464+0000","last_unstale":"2024-04-17T07:30:13.659234+0000","last_undegraded":"2024-04-17T07:30:13.659234+0000","last_fullsized":"2024-04-17T07:30:13.659234+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:55.369293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662552+0000","last_change":"2024-04-17T07:30:13.656408+0000","last_active":"2024-04-17T07:30:13.662552+0000","last_peered":"2024-04-17T07:30:13.662552+0000","last_clean":"2024-04-17T07:30:13.662552+0000","last_became_active":"2024-04-17T07:30:13.656087+0000","last_became_peered":"2024-04-17T07:30:13.656087+0000","last_unstale":"2024-04-17T07:30:13.662552+0000","last_undegraded":"2024-04-17T07:30:13.662552+0000","last_fullsized":"2024-04-17T07:30:13.662552+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:55.143383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678842+0000","last_change":"2024-04-17T07:30:13.669775+0000","last_active":"2024-04-17T07:30:13.678842+0000","last_peered":"2024-04-17T07:30:13.678842+0000","last_clean":"2024-04-17T07:30:13.678842+0000","last_became_active":"2024-04-17T07:30:13.667975+0000","last_became_peered":"2024-04-17T07:30:13.667975+0000","last_unstale":"2024-04-17T07:30:13.678842+0000","last_undegraded":"2024-04-17T07:30:13.678842+0000","last_fullsized":"2024-04-17T07:30:13.678842+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:11.540131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659531+0000","last_change":"2024-04-17T07:30:13.657903+0000","last_active":"2024-04-17T07:30:13.659531+0000","last_peered":"2024-04-17T07:30:13.659531+0000","last_clean":"2024-04-17T07:30:13.659531+0000","last_became_active":"2024-04-17T07:30:13.657607+0000","last_became_peered":"2024-04-17T07:30:13.657607+0000","last_unstale":"2024-04-17T07:30:13.659531+0000","last_undegraded":"2024-04-17T07:30:13.659531+0000","last_fullsized":"2024-04-17T07:30:13.659531+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:53.759703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671375+0000","last_change":"2024-04-17T07:30:13.669070+0000","last_active":"2024-04-17T07:30:13.671375+0000","last_peered":"2024-04-17T07:30:13.671375+0000","last_clean":"2024-04-17T07:30:13.671375+0000","last_became_active":"2024-04-17T07:30:13.668518+0000","last_became_peered":"2024-04-17T07:30:13.668518+0000","last_unstale":"2024-04-17T07:30:13.671375+0000","last_undegraded":"2024-04-17T07:30:13.671375+0000","last_fullsized":"2024-04-17T07:30:13.671375+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:22.861234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.660631+0000","last_change":"2024-04-17T07:30:13.650894+0000","last_active":"2024-04-17T07:30:13.660631+0000","last_peered":"2024-04-17T07:30:13.660631+0000","last_clean":"2024-04-17T07:30:13.660631+0000","last_became_active":"2024-04-17T07:30:13.649940+0000","last_became_peered":"2024-04-17T07:30:13.649940+0000","last_unstale":"2024-04-17T07:30:13.660631+0000","last_undegraded":"2024-04-17T07:30:13.660631+0000","last_fullsized":"2024-04-17T07:30:13.660631+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:10.515398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644991+0000","last_change":"2024-04-17T07:30:13.640177+0000","last_active":"2024-04-17T07:30:13.644991+0000","last_peered":"2024-04-17T07:30:13.644991+0000","last_clean":"2024-04-17T07:30:13.644991+0000","last_became_active":"2024-04-17T07:30:13.639859+0000","last_became_peered":"2024-04-17T07:30:13.639859+0000","last_unstale":"2024-04-17T07:30:13.644991+0000","last_undegraded":"2024-04-17T07:30:13.644991+0000","last_fullsized":"2024-04-17T07:30:13.644991+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:58.328130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.676238+0000","last_change":"2024-04-17T07:30:13.669587+0000","last_active":"2024-04-17T07:30:13.676238+0000","last_peered":"2024-04-17T07:30:13.676238+0000","last_clean":"2024-04-17T07:30:13.676238+0000","last_became_active":"2024-04-17T07:30:13.669204+0000","last_became_peered":"2024-04-17T07:30:13.669204+0000","last_unstale":"2024-04-17T07:30:13.676238+0000","last_undegraded":"2024-04-17T07:30:13.676238+0000","last_fullsized":"2024-04-17T07:30:13.676238+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:54.721472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659216+0000","last_change":"2024-04-17T07:30:13.647730+0000","last_active":"2024-04-17T07:30:13.659216+0000","last_peered":"2024-04-17T07:30:13.659216+0000","last_clean":"2024-04-17T07:30:13.659216+0000","last_became_active":"2024-04-17T07:30:13.647246+0000","last_became_peered":"2024-04-17T07:30:13.647246+0000","last_unstale":"2024-04-17T07:30:13.659216+0000","last_undegraded":"2024-04-17T07:30:13.659216+0000","last_fullsized":"2024-04-17T07:30:13.659216+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:09.100360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.675896+0000","last_change":"2024-04-17T07:30:13.675200+0000","last_active":"2024-04-17T07:30:13.675896+0000","last_peered":"2024-04-17T07:30:13.675896+0000","last_clean":"2024-04-17T07:30:13.675896+0000","last_became_active":"2024-04-17T07:30:13.674731+0000","last_became_peered":"2024-04-17T07:30:13.674731+0000","last_unstale":"2024-04-17T07:30:13.675896+0000","last_undegraded":"2024-04-17T07:30:13.675896+0000","last_fullsized":"2024-04-17T07:30:13.675896+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:00.893354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672327+0000","last_change":"2024-04-17T07:30:13.665805+0000","last_active":"2024-04-17T07:30:13.672327+0000","last_peered":"2024-04-17T07:30:13.672327+0000","last_clean":"2024-04-17T07:30:13.672327+0000","last_became_active":"2024-04-17T07:30:13.662003+0000","last_became_peered":"2024-04-17T07:30:13.662003+0000","last_unstale":"2024-04-17T07:30:13.672327+0000","last_undegraded":"2024-04-17T07:30:13.672327+0000","last_fullsized":"2024-04-17T07:30:13.672327+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:00:10.160119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652660+0000","last_change":"2024-04-17T07:30:13.649298+0000","last_active":"2024-04-17T07:30:13.652660+0000","last_peered":"2024-04-17T07:30:13.652660+0000","last_clean":"2024-04-17T07:30:13.652660+0000","last_became_active":"2024-04-17T07:30:13.648993+0000","last_became_peered":"2024-04-17T07:30:13.648993+0000","last_unstale":"2024-04-17T07:30:13.652660+0000","last_undegraded":"2024-04-17T07:30:13.652660+0000","last_fullsized":"2024-04-17T07:30:13.652660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:26.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.663127+0000","last_change":"2024-04-17T07:30:13.658063+0000","last_active":"2024-04-17T07:30:13.663127+0000","last_peered":"2024-04-17T07:30:13.663127+0000","last_clean":"2024-04-17T07:30:13.663127+0000","last_became_active":"2024-04-17T07:30:13.653621+0000","last_became_peered":"2024-04-17T07:30:13.653621+0000","last_unstale":"2024-04-17T07:30:13.663127+0000","last_undegraded":"2024-04-17T07:30:13.663127+0000","last_fullsized":"2024-04-17T07:30:13.663127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:59.528501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.660853+0000","last_change":"2024-04-17T07:30:13.654547+0000","last_active":"2024-04-17T07:30:13.660853+0000","last_peered":"2024-04-17T07:30:13.660853+0000","last_clean":"2024-04-17T07:30:13.660853+0000","last_became_active":"2024-04-17T07:30:13.654254+0000","last_became_peered":"2024-04-17T07:30:13.654254+0000","last_unstale":"2024-04-17T07:30:13.660853+0000","last_undegraded":"2024-04-17T07:30:13.660853+0000","last_fullsized":"2024-04-17T07:30:13.660853+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:39.218608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678406+0000","last_change":"2024-04-17T07:30:13.669107+0000","last_active":"2024-04-17T07:30:13.678406+0000","last_peered":"2024-04-17T07:30:13.678406+0000","last_clean":"2024-04-17T07:30:13.678406+0000","last_became_active":"2024-04-17T07:30:13.668387+0000","last_became_peered":"2024-04-17T07:30:13.668387+0000","last_unstale":"2024-04-17T07:30:13.678406+0000","last_undegraded":"2024-04-17T07:30:13.678406+0000","last_fullsized":"2024-04-17T07:30:13.678406+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:08:21.335506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679024+0000","last_change":"2024-04-17T07:30:13.674745+0000","last_active":"2024-04-17T07:30:13.679024+0000","last_peered":"2024-04-17T07:30:13.679024+0000","last_clean":"2024-04-17T07:30:13.679024+0000","last_became_active":"2024-04-17T07:30:13.674200+0000","last_became_peered":"2024-04-17T07:30:13.674200+0000","last_unstale":"2024-04-17T07:30:13.679024+0000","last_undegraded":"2024-04-17T07:30:13.679024+0000","last_fullsized":"2024-04-17T07:30:13.679024+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.205842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678636+0000","last_change":"2024-04-17T07:30:13.673963+0000","last_active":"2024-04-17T07:30:13.678636+0000","last_peered":"2024-04-17T07:30:13.678636+0000","last_clean":"2024-04-17T07:30:13.678636+0000","last_became_active":"2024-04-17T07:30:13.673685+0000","last_became_peered":"2024-04-17T07:30:13.673685+0000","last_unstale":"2024-04-17T07:30:13.678636+0000","last_undegraded":"2024-04-17T07:30:13.678636+0000","last_fullsized":"2024-04-17T07:30:13.678636+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:40.684090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.681020+0000","last_change":"2024-04-17T07:30:13.675871+0000","last_active":"2024-04-17T07:30:13.681020+0000","last_peered":"2024-04-17T07:30:13.681020+0000","last_clean":"2024-04-17T07:30:13.681020+0000","last_became_active":"2024-04-17T07:30:13.674702+0000","last_became_peered":"2024-04-17T07:30:13.674702+0000","last_unstale":"2024-04-17T07:30:13.681020+0000","last_undegraded":"2024-04-17T07:30:13.681020+0000","last_fullsized":"2024-04-17T07:30:13.681020+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:04:13.131885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669859+0000","last_change":"2024-04-17T07:30:13.667510+0000","last_active":"2024-04-17T07:30:13.669859+0000","last_peered":"2024-04-17T07:30:13.669859+0000","last_clean":"2024-04-17T07:30:13.669859+0000","last_became_active":"2024-04-17T07:30:13.667213+0000","last_became_peered":"2024-04-17T07:30:13.667213+0000","last_unstale":"2024-04-17T07:30:13.669859+0000","last_undegraded":"2024-04-17T07:30:13.669859+0000","last_fullsized":"2024-04-17T07:30:13.669859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:39.621695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.653321+0000","last_change":"2024-04-17T07:30:13.648762+0000","last_active":"2024-04-17T07:30:13.653321+0000","last_peered":"2024-04-17T07:30:13.653321+0000","last_clean":"2024-04-17T07:30:13.653321+0000","last_became_active":"2024-04-17T07:30:13.648195+0000","last_became_peered":"2024-04-17T07:30:13.648195+0000","last_unstale":"2024-04-17T07:30:13.653321+0000","last_undegraded":"2024-04-17T07:30:13.653321+0000","last_fullsized":"2024-04-17T07:30:13.653321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:57.128893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658824+0000","last_change":"2024-04-17T07:30:13.656087+0000","last_active":"2024-04-17T07:30:13.658824+0000","last_peered":"2024-04-17T07:30:13.658824+0000","last_clean":"2024-04-17T07:30:13.658824+0000","last_became_active":"2024-04-17T07:30:13.655801+0000","last_became_peered":"2024-04-17T07:30:13.655801+0000","last_unstale":"2024-04-17T07:30:13.658824+0000","last_undegraded":"2024-04-17T07:30:13.658824+0000","last_fullsized":"2024-04-17T07:30:13.658824+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:14.338883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678700+0000","last_change":"2024-04-17T07:30:13.674886+0000","last_active":"2024-04-17T07:30:13.678700+0000","last_peered":"2024-04-17T07:30:13.678700+0000","last_clean":"2024-04-17T07:30:13.678700+0000","last_became_active":"2024-04-17T07:30:13.674489+0000","last_became_peered":"2024-04-17T07:30:13.674489+0000","last_unstale":"2024-04-17T07:30:13.678700+0000","last_undegraded":"2024-04-17T07:30:13.678700+0000","last_fullsized":"2024-04-17T07:30:13.678700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:09.690617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.654413+0000","last_change":"2024-04-17T07:30:13.646770+0000","last_active":"2024-04-17T07:30:13.654413+0000","last_peered":"2024-04-17T07:30:13.654413+0000","last_clean":"2024-04-17T07:30:13.654413+0000","last_became_active":"2024-04-17T07:30:13.645793+0000","last_became_peered":"2024-04-17T07:30:13.645793+0000","last_unstale":"2024-04-17T07:30:13.654413+0000","last_undegraded":"2024-04-17T07:30:13.654413+0000","last_fullsized":"2024-04-17T07:30:13.654413+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:48:02.808778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672111+0000","last_change":"2024-04-17T07:30:13.660716+0000","last_active":"2024-04-17T07:30:13.672111+0000","last_peered":"2024-04-17T07:30:13.672111+0000","last_clean":"2024-04-17T07:30:13.672111+0000","last_became_active":"2024-04-17T07:30:13.660410+0000","last_became_peered":"2024-04-17T07:30:13.660410+0000","last_unstale":"2024-04-17T07:30:13.672111+0000","last_undegraded":"2024-04-17T07:30:13.672111+0000","last_fullsized":"2024-04-17T07:30:13.672111+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:01.315245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678929+0000","last_change":"2024-04-17T07:30:13.673514+0000","last_active":"2024-04-17T07:30:13.678929+0000","last_peered":"2024-04-17T07:30:13.678929+0000","last_clean":"2024-04-17T07:30:13.678929+0000","last_became_active":"2024-04-17T07:30:13.672881+0000","last_became_peered":"2024-04-17T07:30:13.672881+0000","last_unstale":"2024-04-17T07:30:13.678929+0000","last_undegraded":"2024-04-17T07:30:13.678929+0000","last_fullsized":"2024-04-17T07:30:13.678929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:09.983999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"97'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.668351+0000","last_change":"2024-04-17T07:30:12.657902+0000","last_active":"2024-04-17T07:30:13.668351+0000","last_peered":"2024-04-17T07:30:13.668351+0000","last_clean":"2024-04-17T07:30:13.668351+0000","last_became_active":"2024-04-17T07:30:12.657500+0000","last_became_peered":"2024-04-17T07:30:12.657500+0000","last_unstale":"2024-04-17T07:30:13.668351+0000","last_undegraded":"2024-04-17T07:30:13.668351+0000","last_fullsized":"2024-04-17T07:30:13.668351+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:01.322054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659195+0000","last_change":"2024-04-17T07:30:13.647547+0000","last_active":"2024-04-17T07:30:13.659195+0000","last_peered":"2024-04-17T07:30:13.659195+0000","last_clean":"2024-04-17T07:30:13.659195+0000","last_became_active":"2024-04-17T07:30:13.647080+0000","last_became_peered":"2024-04-17T07:30:13.647080+0000","last_unstale":"2024-04-17T07:30:13.659195+0000","last_undegraded":"2024-04-17T07:30:13.659195+0000","last_fullsized":"2024-04-17T07:30:13.659195+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:26.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662824+0000","last_change":"2024-04-17T07:30:12.655187+0000","last_active":"2024-04-17T07:30:13.662824+0000","last_peered":"2024-04-17T07:30:13.662824+0000","last_clean":"2024-04-17T07:30:13.662824+0000","last_became_active":"2024-04-17T07:30:12.654903+0000","last_became_peered":"2024-04-17T07:30:12.654903+0000","last_unstale":"2024-04-17T07:30:13.662824+0000","last_undegraded":"2024-04-17T07:30:13.662824+0000","last_fullsized":"2024-04-17T07:30:13.662824+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:33.201476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679195+0000","last_change":"2024-04-17T07:30:13.675693+0000","last_active":"2024-04-17T07:30:13.679195+0000","last_peered":"2024-04-17T07:30:13.679195+0000","last_clean":"2024-04-17T07:30:13.679195+0000","last_became_active":"2024-04-17T07:30:13.674987+0000","last_became_peered":"2024-04-17T07:30:13.674987+0000","last_unstale":"2024-04-17T07:30:13.679195+0000","last_undegraded":"2024-04-17T07:30:13.679195+0000","last_fullsized":"2024-04-17T07:30:13.679195+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:51.805698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678227+0000","last_change":"2024-04-17T07:30:12.658562+0000","last_active":"2024-04-17T07:30:13.678227+0000","last_peered":"2024-04-17T07:30:13.678227+0000","last_clean":"2024-04-17T07:30:13.678227+0000","last_became_active":"2024-04-17T07:30:12.658241+0000","last_became_peered":"2024-04-17T07:30:12.658241+0000","last_unstale":"2024-04-17T07:30:13.678227+0000","last_undegraded":"2024-04-17T07:30:13.678227+0000","last_fullsized":"2024-04-17T07:30:13.678227+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:18.826813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678869+0000","last_change":"2024-04-17T07:30:13.670975+0000","last_active":"2024-04-17T07:30:13.678869+0000","last_peered":"2024-04-17T07:30:13.678869+0000","last_clean":"2024-04-17T07:30:13.678869+0000","last_became_active":"2024-04-17T07:30:13.669421+0000","last_became_peered":"2024-04-17T07:30:13.669421+0000","last_unstale":"2024-04-17T07:30:13.678869+0000","last_undegraded":"2024-04-17T07:30:13.678869+0000","last_fullsized":"2024-04-17T07:30:13.678869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:08:57.823931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677608+0000","last_change":"2024-04-17T07:30:12.655238+0000","last_active":"2024-04-17T07:30:13.677608+0000","last_peered":"2024-04-17T07:30:13.677608+0000","last_clean":"2024-04-17T07:30:13.677608+0000","last_became_active":"2024-04-17T07:30:12.654932+0000","last_became_peered":"2024-04-17T07:30:12.654932+0000","last_unstale":"2024-04-17T07:30:13.677608+0000","last_undegraded":"2024-04-17T07:30:13.677608+0000","last_fullsized":"2024-04-17T07:30:13.677608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:31.282535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658886+0000","last_change":"2024-04-17T07:30:13.649608+0000","last_active":"2024-04-17T07:30:13.658886+0000","last_peered":"2024-04-17T07:30:13.658886+0000","last_clean":"2024-04-17T07:30:13.658886+0000","last_became_active":"2024-04-17T07:30:13.648550+0000","last_became_peered":"2024-04-17T07:30:13.648550+0000","last_unstale":"2024-04-17T07:30:13.658886+0000","last_undegraded":"2024-04-17T07:30:13.658886+0000","last_fullsized":"2024-04-17T07:30:13.658886+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:39.995919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664958+0000","last_change":"2024-04-17T07:30:12.649158+0000","last_active":"2024-04-17T07:30:13.664958+0000","last_peered":"2024-04-17T07:30:13.664958+0000","last_clean":"2024-04-17T07:30:13.664958+0000","last_became_active":"2024-04-17T07:30:12.648678+0000","last_became_peered":"2024-04-17T07:30:12.648678+0000","last_unstale":"2024-04-17T07:30:13.664958+0000","last_undegraded":"2024-04-17T07:30:13.664958+0000","last_fullsized":"2024-04-17T07:30:13.664958+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:24.764730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.665999+0000","last_change":"2024-04-17T07:30:13.663624+0000","last_active":"2024-04-17T07:30:13.665999+0000","last_peered":"2024-04-17T07:30:13.665999+0000","last_clean":"2024-04-17T07:30:13.665999+0000","last_became_active":"2024-04-17T07:30:13.663286+0000","last_became_peered":"2024-04-17T07:30:13.663286+0000","last_unstale":"2024-04-17T07:30:13.665999+0000","last_undegraded":"2024-04-17T07:30:13.665999+0000","last_fullsized":"2024-04-17T07:30:13.665999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:56.469111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679160+0000","last_change":"2024-04-17T07:30:12.659376+0000","last_active":"2024-04-17T07:30:13.679160+0000","last_peered":"2024-04-17T07:30:13.679160+0000","last_clean":"2024-04-17T07:30:13.679160+0000","last_became_active":"2024-04-17T07:30:12.659077+0000","last_became_peered":"2024-04-17T07:30:12.659077+0000","last_unstale":"2024-04-17T07:30:13.679160+0000","last_undegraded":"2024-04-17T07:30:13.679160+0000","last_fullsized":"2024-04-17T07:30:13.679160+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:12.132627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.661155+0000","last_change":"2024-04-17T07:30:13.657914+0000","last_active":"2024-04-17T07:30:13.661155+0000","last_peered":"2024-04-17T07:30:13.661155+0000","last_clean":"2024-04-17T07:30:13.661155+0000","last_became_active":"2024-04-17T07:30:13.656819+0000","last_became_peered":"2024-04-17T07:30:13.656819+0000","last_unstale":"2024-04-17T07:30:13.661155+0000","last_undegraded":"2024-04-17T07:30:13.661155+0000","last_fullsized":"2024-04-17T07:30:13.661155+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:08.997705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.632983+0000","last_change":"2024-04-17T07:30:12.656941+0000","last_active":"2024-04-17T07:30:13.632983+0000","last_peered":"2024-04-17T07:30:13.632983+0000","last_clean":"2024-04-17T07:30:13.632983+0000","last_became_active":"2024-04-17T07:30:12.656633+0000","last_became_peered":"2024-04-17T07:30:12.656633+0000","last_unstale":"2024-04-17T07:30:13.632983+0000","last_undegraded":"2024-04-17T07:30:13.632983+0000","last_fullsized":"2024-04-17T07:30:13.632983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:02.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.664448+0000","last_change":"2024-04-17T07:30:13.663839+0000","last_active":"2024-04-17T07:30:13.664448+0000","last_peered":"2024-04-17T07:30:13.664448+0000","last_clean":"2024-04-17T07:30:13.664448+0000","last_became_active":"2024-04-17T07:30:13.663202+0000","last_became_peered":"2024-04-17T07:30:13.663202+0000","last_unstale":"2024-04-17T07:30:13.664448+0000","last_undegraded":"2024-04-17T07:30:13.664448+0000","last_fullsized":"2024-04-17T07:30:13.664448+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:09.100360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.638318+0000","last_change":"2024-04-17T07:30:12.651522+0000","last_active":"2024-04-17T07:30:13.638318+0000","last_peered":"2024-04-17T07:30:13.638318+0000","last_clean":"2024-04-17T07:30:13.638318+0000","last_became_active":"2024-04-17T07:30:12.651212+0000","last_became_peered":"2024-04-17T07:30:12.651212+0000","last_unstale":"2024-04-17T07:30:13.638318+0000","last_undegraded":"2024-04-17T07:30:13.638318+0000","last_fullsized":"2024-04-17T07:30:13.638318+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:16:37.898883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.676480+0000","last_change":"2024-04-17T07:30:13.668864+0000","last_active":"2024-04-17T07:30:13.676480+0000","last_peered":"2024-04-17T07:30:13.676480+0000","last_clean":"2024-04-17T07:30:13.676480+0000","last_became_active":"2024-04-17T07:30:13.667245+0000","last_became_peered":"2024-04-17T07:30:13.667245+0000","last_unstale":"2024-04-17T07:30:13.676480+0000","last_undegraded":"2024-04-17T07:30:13.676480+0000","last_fullsized":"2024-04-17T07:30:13.676480+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:40.562710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644618+0000","last_change":"2024-04-17T07:30:12.646128+0000","last_active":"2024-04-17T07:30:13.644618+0000","last_peered":"2024-04-17T07:30:13.644618+0000","last_clean":"2024-04-17T07:30:13.644618+0000","last_became_active":"2024-04-17T07:30:12.643925+0000","last_became_peered":"2024-04-17T07:30:12.643925+0000","last_unstale":"2024-04-17T07:30:13.644618+0000","last_undegraded":"2024-04-17T07:30:13.644618+0000","last_fullsized":"2024-04-17T07:30:13.644618+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:26.665319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678890+0000","last_change":"2024-04-17T07:30:13.673312+0000","last_active":"2024-04-17T07:30:13.678890+0000","last_peered":"2024-04-17T07:30:13.678890+0000","last_clean":"2024-04-17T07:30:13.678890+0000","last_became_active":"2024-04-17T07:30:13.673041+0000","last_became_peered":"2024-04-17T07:30:13.673041+0000","last_unstale":"2024-04-17T07:30:13.678890+0000","last_undegraded":"2024-04-17T07:30:13.678890+0000","last_fullsized":"2024-04-17T07:30:13.678890+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:25.782507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.657616+0000","last_change":"2024-04-17T07:30:12.654905+0000","last_active":"2024-04-17T07:30:13.657616+0000","last_peered":"2024-04-17T07:30:13.657616+0000","last_clean":"2024-04-17T07:30:13.657616+0000","last_became_active":"2024-04-17T07:30:12.651462+0000","last_became_peered":"2024-04-17T07:30:12.651462+0000","last_unstale":"2024-04-17T07:30:13.657616+0000","last_undegraded":"2024-04-17T07:30:13.657616+0000","last_fullsized":"2024-04-17T07:30:13.657616+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:17.301348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.673999+0000","last_change":"2024-04-17T07:30:13.667226+0000","last_active":"2024-04-17T07:30:13.673999+0000","last_peered":"2024-04-17T07:30:13.673999+0000","last_clean":"2024-04-17T07:30:13.673999+0000","last_became_active":"2024-04-17T07:30:13.666881+0000","last_became_peered":"2024-04-17T07:30:13.666881+0000","last_unstale":"2024-04-17T07:30:13.673999+0000","last_undegraded":"2024-04-17T07:30:13.673999+0000","last_fullsized":"2024-04-17T07:30:13.673999+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:26:10.482433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662777+0000","last_change":"2024-04-17T07:30:12.653752+0000","last_active":"2024-04-17T07:30:13.662777+0000","last_peered":"2024-04-17T07:30:13.662777+0000","last_clean":"2024-04-17T07:30:13.662777+0000","last_became_active":"2024-04-17T07:30:12.653082+0000","last_became_peered":"2024-04-17T07:30:12.653082+0000","last_unstale":"2024-04-17T07:30:13.662777+0000","last_undegraded":"2024-04-17T07:30:13.662777+0000","last_fullsized":"2024-04-17T07:30:13.662777+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:55.660313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.650615+0000","last_change":"2024-04-17T07:30:13.645185+0000","last_active":"2024-04-17T07:30:13.650615+0000","last_peered":"2024-04-17T07:30:13.650615+0000","last_clean":"2024-04-17T07:30:13.650615+0000","last_became_active":"2024-04-17T07:30:13.644874+0000","last_became_peered":"2024-04-17T07:30:13.644874+0000","last_unstale":"2024-04-17T07:30:13.650615+0000","last_undegraded":"2024-04-17T07:30:13.650615+0000","last_fullsized":"2024-04-17T07:30:13.650615+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:34.219253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"97'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.667070+0000","last_change":"2024-04-17T07:30:12.655810+0000","last_active":"2024-04-17T07:30:13.667070+0000","last_peered":"2024-04-17T07:30:13.667070+0000","last_clean":"2024-04-17T07:30:13.667070+0000","last_became_active":"2024-04-17T07:30:12.655483+0000","last_became_peered":"2024-04-17T07:30:12.655483+0000","last_unstale":"2024-04-17T07:30:13.667070+0000","last_undegraded":"2024-04-17T07:30:13.667070+0000","last_fullsized":"2024-04-17T07:30:13.667070+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:23:23.121888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678040+0000","last_change":"2024-04-17T07:30:13.668303+0000","last_active":"2024-04-17T07:30:13.678040+0000","last_peered":"2024-04-17T07:30:13.678040+0000","last_clean":"2024-04-17T07:30:13.678040+0000","last_became_active":"2024-04-17T07:30:13.667668+0000","last_became_peered":"2024-04-17T07:30:13.667668+0000","last_unstale":"2024-04-17T07:30:13.678040+0000","last_undegraded":"2024-04-17T07:30:13.678040+0000","last_fullsized":"2024-04-17T07:30:13.678040+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:59:42.830490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669573+0000","last_change":"2024-04-17T07:30:12.656865+0000","last_active":"2024-04-17T07:30:13.669573+0000","last_peered":"2024-04-17T07:30:13.669573+0000","last_clean":"2024-04-17T07:30:13.669573+0000","last_became_active":"2024-04-17T07:30:12.656431+0000","last_became_peered":"2024-04-17T07:30:12.656431+0000","last_unstale":"2024-04-17T07:30:13.669573+0000","last_undegraded":"2024-04-17T07:30:13.669573+0000","last_fullsized":"2024-04-17T07:30:13.669573+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:10:09.600827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659131+0000","last_change":"2024-04-17T07:30:13.652616+0000","last_active":"2024-04-17T07:30:13.659131+0000","last_peered":"2024-04-17T07:30:13.659131+0000","last_clean":"2024-04-17T07:30:13.659131+0000","last_became_active":"2024-04-17T07:30:13.652240+0000","last_became_peered":"2024-04-17T07:30:13.652240+0000","last_unstale":"2024-04-17T07:30:13.659131+0000","last_undegraded":"2024-04-17T07:30:13.659131+0000","last_fullsized":"2024-04-17T07:30:13.659131+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:57.742073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662763+0000","last_change":"2024-04-17T07:30:12.654008+0000","last_active":"2024-04-17T07:30:13.662763+0000","last_peered":"2024-04-17T07:30:13.662763+0000","last_clean":"2024-04-17T07:30:13.662763+0000","last_became_active":"2024-04-17T07:30:12.650771+0000","last_became_peered":"2024-04-17T07:30:12.650771+0000","last_unstale":"2024-04-17T07:30:13.662763+0000","last_undegraded":"2024-04-17T07:30:13.662763+0000","last_fullsized":"2024-04-17T07:30:13.662763+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:47.823913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671291+0000","last_change":"2024-04-17T07:30:13.668776+0000","last_active":"2024-04-17T07:30:13.671291+0000","last_peered":"2024-04-17T07:30:13.671291+0000","last_clean":"2024-04-17T07:30:13.671291+0000","last_became_active":"2024-04-17T07:30:13.668443+0000","last_became_peered":"2024-04-17T07:30:13.668443+0000","last_unstale":"2024-04-17T07:30:13.671291+0000","last_undegraded":"2024-04-17T07:30:13.671291+0000","last_fullsized":"2024-04-17T07:30:13.671291+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:55.369293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.635196+0000","last_change":"2024-04-17T07:30:12.648100+0000","last_active":"2024-04-17T07:30:13.635196+0000","last_peered":"2024-04-17T07:30:13.635196+0000","last_clean":"2024-04-17T07:30:13.635196+0000","last_became_active":"2024-04-17T07:30:12.647810+0000","last_became_peered":"2024-04-17T07:30:12.647810+0000","last_unstale":"2024-04-17T07:30:13.635196+0000","last_undegraded":"2024-04-17T07:30:13.635196+0000","last_fullsized":"2024-04-17T07:30:13.635196+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:10.244003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.649943+0000","last_change":"2024-04-17T07:30:13.645406+0000","last_active":"2024-04-17T07:30:13.649943+0000","last_peered":"2024-04-17T07:30:13.649943+0000","last_clean":"2024-04-17T07:30:13.649943+0000","last_became_active":"2024-04-17T07:30:13.645067+0000","last_became_peered":"2024-04-17T07:30:13.645067+0000","last_unstale":"2024-04-17T07:30:13.649943+0000","last_undegraded":"2024-04-17T07:30:13.649943+0000","last_fullsized":"2024-04-17T07:30:13.649943+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:58.328130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"97'8","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679452+0000","last_change":"2024-04-17T07:30:12.653114+0000","last_active":"2024-04-17T07:30:13.679452+0000","last_peered":"2024-04-17T07:30:13.679452+0000","last_clean":"2024-04-17T07:30:13.679452+0000","last_became_active":"2024-04-17T07:30:12.652238+0000","last_became_peered":"2024-04-17T07:30:12.652238+0000","last_unstale":"2024-04-17T07:30:13.679452+0000","last_undegraded":"2024-04-17T07:30:13.679452+0000","last_fullsized":"2024-04-17T07:30:13.679452+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:34.825945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.659466+0000","last_change":"2024-04-17T07:30:13.649456+0000","last_active":"2024-04-17T07:30:13.659466+0000","last_peered":"2024-04-17T07:30:13.659466+0000","last_clean":"2024-04-17T07:30:13.659466+0000","last_became_active":"2024-04-17T07:30:13.648987+0000","last_became_peered":"2024-04-17T07:30:13.648987+0000","last_unstale":"2024-04-17T07:30:13.659466+0000","last_undegraded":"2024-04-17T07:30:13.659466+0000","last_fullsized":"2024-04-17T07:30:13.659466+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:25.985751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679557+0000","last_change":"2024-04-17T07:30:12.658068+0000","last_active":"2024-04-17T07:30:13.679557+0000","last_peered":"2024-04-17T07:30:13.679557+0000","last_clean":"2024-04-17T07:30:13.679557+0000","last_became_active":"2024-04-17T07:30:12.657511+0000","last_became_peered":"2024-04-17T07:30:12.657511+0000","last_unstale":"2024-04-17T07:30:13.679557+0000","last_undegraded":"2024-04-17T07:30:13.679557+0000","last_fullsized":"2024-04-17T07:30:13.679557+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:54.489242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678865+0000","last_change":"2024-04-17T07:30:13.669432+0000","last_active":"2024-04-17T07:30:13.678865+0000","last_peered":"2024-04-17T07:30:13.678865+0000","last_clean":"2024-04-17T07:30:13.678865+0000","last_became_active":"2024-04-17T07:30:13.668882+0000","last_became_peered":"2024-04-17T07:30:13.668882+0000","last_unstale":"2024-04-17T07:30:13.678865+0000","last_undegraded":"2024-04-17T07:30:13.678865+0000","last_fullsized":"2024-04-17T07:30:13.678865+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:46:45.505751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677933+0000","last_change":"2024-04-17T07:30:12.654105+0000","last_active":"2024-04-17T07:30:13.677933+0000","last_peered":"2024-04-17T07:30:13.677933+0000","last_clean":"2024-04-17T07:30:13.677933+0000","last_became_active":"2024-04-17T07:30:12.653826+0000","last_became_peered":"2024-04-17T07:30:12.653826+0000","last_unstale":"2024-04-17T07:30:13.677933+0000","last_undegraded":"2024-04-17T07:30:13.677933+0000","last_fullsized":"2024-04-17T07:30:13.677933+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:34.328109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.656029+0000","last_change":"2024-04-17T07:30:13.650355+0000","last_active":"2024-04-17T07:30:13.656029+0000","last_peered":"2024-04-17T07:30:13.656029+0000","last_clean":"2024-04-17T07:30:13.656029+0000","last_became_active":"2024-04-17T07:30:13.648213+0000","last_became_peered":"2024-04-17T07:30:13.648213+0000","last_unstale":"2024-04-17T07:30:13.656029+0000","last_undegraded":"2024-04-17T07:30:13.656029+0000","last_fullsized":"2024-04-17T07:30:13.656029+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:12.182242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679155+0000","last_change":"2024-04-17T07:30:12.656280+0000","last_active":"2024-04-17T07:30:13.679155+0000","last_peered":"2024-04-17T07:30:13.679155+0000","last_clean":"2024-04-17T07:30:13.679155+0000","last_became_active":"2024-04-17T07:30:12.653013+0000","last_became_peered":"2024-04-17T07:30:12.653013+0000","last_unstale":"2024-04-17T07:30:13.679155+0000","last_undegraded":"2024-04-17T07:30:13.679155+0000","last_fullsized":"2024-04-17T07:30:13.679155+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:32.316524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658562+0000","last_change":"2024-04-17T07:30:13.650130+0000","last_active":"2024-04-17T07:30:13.658562+0000","last_peered":"2024-04-17T07:30:13.658562+0000","last_clean":"2024-04-17T07:30:13.658562+0000","last_became_active":"2024-04-17T07:30:13.648619+0000","last_became_peered":"2024-04-17T07:30:13.648619+0000","last_unstale":"2024-04-17T07:30:13.658562+0000","last_undegraded":"2024-04-17T07:30:13.658562+0000","last_fullsized":"2024-04-17T07:30:13.658562+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:22.861234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652996+0000","last_change":"2024-04-17T07:30:12.655148+0000","last_active":"2024-04-17T07:30:13.652996+0000","last_peered":"2024-04-17T07:30:13.652996+0000","last_clean":"2024-04-17T07:30:13.652996+0000","last_became_active":"2024-04-17T07:30:12.654883+0000","last_became_peered":"2024-04-17T07:30:12.654883+0000","last_unstale":"2024-04-17T07:30:13.652996+0000","last_undegraded":"2024-04-17T07:30:13.652996+0000","last_fullsized":"2024-04-17T07:30:13.652996+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:00.519603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.649052+0000","last_change":"2024-04-17T07:30:13.646685+0000","last_active":"2024-04-17T07:30:13.649052+0000","last_peered":"2024-04-17T07:30:13.649052+0000","last_clean":"2024-04-17T07:30:13.649052+0000","last_became_active":"2024-04-17T07:30:13.645474+0000","last_became_peered":"2024-04-17T07:30:13.645474+0000","last_unstale":"2024-04-17T07:30:13.649052+0000","last_undegraded":"2024-04-17T07:30:13.649052+0000","last_fullsized":"2024-04-17T07:30:13.649052+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:09.983999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.662932+0000","last_change":"2024-04-17T07:30:12.655925+0000","last_active":"2024-04-17T07:30:13.662932+0000","last_peered":"2024-04-17T07:30:13.662932+0000","last_clean":"2024-04-17T07:30:13.662932+0000","last_became_active":"2024-04-17T07:30:12.655632+0000","last_became_peered":"2024-04-17T07:30:12.655632+0000","last_unstale":"2024-04-17T07:30:13.662932+0000","last_undegraded":"2024-04-17T07:30:13.662932+0000","last_fullsized":"2024-04-17T07:30:13.662932+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:23.143787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671799+0000","last_change":"2024-04-17T07:30:13.669204+0000","last_active":"2024-04-17T07:30:13.671799+0000","last_peered":"2024-04-17T07:30:13.671799+0000","last_clean":"2024-04-17T07:30:13.671799+0000","last_became_active":"2024-04-17T07:30:13.668319+0000","last_became_peered":"2024-04-17T07:30:13.668319+0000","last_unstale":"2024-04-17T07:30:13.671799+0000","last_undegraded":"2024-04-17T07:30:13.671799+0000","last_fullsized":"2024-04-17T07:30:13.671799+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:52.619149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.635749+0000","last_change":"2024-04-17T07:30:12.645417+0000","last_active":"2024-04-17T07:30:13.635749+0000","last_peered":"2024-04-17T07:30:13.635749+0000","last_clean":"2024-04-17T07:30:13.635749+0000","last_became_active":"2024-04-17T07:30:12.642333+0000","last_became_peered":"2024-04-17T07:30:12.642333+0000","last_unstale":"2024-04-17T07:30:13.635749+0000","last_undegraded":"2024-04-17T07:30:13.635749+0000","last_fullsized":"2024-04-17T07:30:13.635749+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:31.384463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.672197+0000","last_change":"2024-04-17T07:30:13.670675+0000","last_active":"2024-04-17T07:30:13.672197+0000","last_peered":"2024-04-17T07:30:13.672197+0000","last_clean":"2024-04-17T07:30:13.672197+0000","last_became_active":"2024-04-17T07:30:13.670430+0000","last_became_peered":"2024-04-17T07:30:13.670430+0000","last_unstale":"2024-04-17T07:30:13.672197+0000","last_undegraded":"2024-04-17T07:30:13.672197+0000","last_fullsized":"2024-04-17T07:30:13.672197+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:38:23.868600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.627757+0000","last_change":"2024-04-17T07:30:12.650955+0000","last_active":"2024-04-17T07:30:13.627757+0000","last_peered":"2024-04-17T07:30:13.627757+0000","last_clean":"2024-04-17T07:30:13.627757+0000","last_became_active":"2024-04-17T07:30:12.650676+0000","last_became_peered":"2024-04-17T07:30:12.650676+0000","last_unstale":"2024-04-17T07:30:13.627757+0000","last_undegraded":"2024-04-17T07:30:13.627757+0000","last_fullsized":"2024-04-17T07:30:13.627757+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:55.648288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.651589+0000","last_change":"2024-04-17T07:30:13.646440+0000","last_active":"2024-04-17T07:30:13.651589+0000","last_peered":"2024-04-17T07:30:13.651589+0000","last_clean":"2024-04-17T07:30:13.651589+0000","last_became_active":"2024-04-17T07:30:13.646167+0000","last_became_peered":"2024-04-17T07:30:13.646167+0000","last_unstale":"2024-04-17T07:30:13.651589+0000","last_undegraded":"2024-04-17T07:30:13.651589+0000","last_fullsized":"2024-04-17T07:30:13.651589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:57.635005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679423+0000","last_change":"2024-04-17T07:30:12.656371+0000","last_active":"2024-04-17T07:30:13.679423+0000","last_peered":"2024-04-17T07:30:13.679423+0000","last_clean":"2024-04-17T07:30:13.679423+0000","last_became_active":"2024-04-17T07:30:12.655692+0000","last_became_peered":"2024-04-17T07:30:12.655692+0000","last_unstale":"2024-04-17T07:30:13.679423+0000","last_undegraded":"2024-04-17T07:30:13.679423+0000","last_fullsized":"2024-04-17T07:30:13.679423+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:50:13.149975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.652058+0000","last_change":"2024-04-17T07:30:13.644622+0000","last_active":"2024-04-17T07:30:13.652058+0000","last_peered":"2024-04-17T07:30:13.652058+0000","last_clean":"2024-04-17T07:30:13.652058+0000","last_became_active":"2024-04-17T07:30:13.643777+0000","last_became_peered":"2024-04-17T07:30:13.643777+0000","last_unstale":"2024-04-17T07:30:13.652058+0000","last_undegraded":"2024-04-17T07:30:13.652058+0000","last_fullsized":"2024-04-17T07:30:13.652058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:15.279395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.637106+0000","last_change":"2024-04-17T07:30:12.653927+0000","last_active":"2024-04-17T07:30:13.637106+0000","last_peered":"2024-04-17T07:30:13.637106+0000","last_clean":"2024-04-17T07:30:13.637106+0000","last_became_active":"2024-04-17T07:30:12.652224+0000","last_became_peered":"2024-04-17T07:30:12.652224+0000","last_unstale":"2024-04-17T07:30:13.637106+0000","last_undegraded":"2024-04-17T07:30:13.637106+0000","last_fullsized":"2024-04-17T07:30:13.637106+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:59.980592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.679208+0000","last_change":"2024-04-17T07:30:13.671196+0000","last_active":"2024-04-17T07:30:13.679208+0000","last_peered":"2024-04-17T07:30:13.679208+0000","last_clean":"2024-04-17T07:30:13.679208+0000","last_became_active":"2024-04-17T07:30:13.670825+0000","last_became_peered":"2024-04-17T07:30:13.670825+0000","last_unstale":"2024-04-17T07:30:13.679208+0000","last_undegraded":"2024-04-17T07:30:13.679208+0000","last_fullsized":"2024-04-17T07:30:13.679208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:56.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.639395+0000","last_change":"2024-04-17T07:30:12.648039+0000","last_active":"2024-04-17T07:30:13.639395+0000","last_peered":"2024-04-17T07:30:13.639395+0000","last_clean":"2024-04-17T07:30:13.639395+0000","last_became_active":"2024-04-17T07:30:12.647761+0000","last_became_peered":"2024-04-17T07:30:12.647761+0000","last_unstale":"2024-04-17T07:30:13.639395+0000","last_undegraded":"2024-04-17T07:30:13.639395+0000","last_fullsized":"2024-04-17T07:30:13.639395+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:55.648288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678891+0000","last_change":"2024-04-17T07:30:13.671045+0000","last_active":"2024-04-17T07:30:13.678891+0000","last_peered":"2024-04-17T07:30:13.678891+0000","last_clean":"2024-04-17T07:30:13.678891+0000","last_became_active":"2024-04-17T07:30:13.669552+0000","last_became_peered":"2024-04-17T07:30:13.669552+0000","last_unstale":"2024-04-17T07:30:13.678891+0000","last_undegraded":"2024-04-17T07:30:13.678891+0000","last_fullsized":"2024-04-17T07:30:13.678891+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:12.897536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.629227+0000","last_change":"2024-04-17T07:30:12.648394+0000","last_active":"2024-04-17T07:30:13.629227+0000","last_peered":"2024-04-17T07:30:13.629227+0000","last_clean":"2024-04-17T07:30:13.629227+0000","last_became_active":"2024-04-17T07:30:12.648115+0000","last_became_peered":"2024-04-17T07:30:12.648115+0000","last_unstale":"2024-04-17T07:30:13.629227+0000","last_undegraded":"2024-04-17T07:30:13.629227+0000","last_fullsized":"2024-04-17T07:30:13.629227+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:17.639100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.656825+0000","last_change":"2024-04-17T07:30:13.646481+0000","last_active":"2024-04-17T07:30:13.656825+0000","last_peered":"2024-04-17T07:30:13.656825+0000","last_clean":"2024-04-17T07:30:13.656825+0000","last_became_active":"2024-04-17T07:30:13.646191+0000","last_became_peered":"2024-04-17T07:30:13.646191+0000","last_unstale":"2024-04-17T07:30:13.656825+0000","last_undegraded":"2024-04-17T07:30:13.656825+0000","last_fullsized":"2024-04-17T07:30:13.656825+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:57.635005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.640292+0000","last_change":"2024-04-17T07:30:12.647508+0000","last_active":"2024-04-17T07:30:13.640292+0000","last_peered":"2024-04-17T07:30:13.640292+0000","last_clean":"2024-04-17T07:30:13.640292+0000","last_became_active":"2024-04-17T07:30:12.647203+0000","last_became_peered":"2024-04-17T07:30:12.647203+0000","last_unstale":"2024-04-17T07:30:13.640292+0000","last_undegraded":"2024-04-17T07:30:13.640292+0000","last_fullsized":"2024-04-17T07:30:13.640292+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:17.639100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.678767+0000","last_change":"2024-04-17T07:30:13.675054+0000","last_active":"2024-04-17T07:30:13.678767+0000","last_peered":"2024-04-17T07:30:13.678767+0000","last_clean":"2024-04-17T07:30:13.678767+0000","last_became_active":"2024-04-17T07:30:13.672833+0000","last_became_peered":"2024-04-17T07:30:13.672833+0000","last_unstale":"2024-04-17T07:30:13.678767+0000","last_undegraded":"2024-04-17T07:30:13.678767+0000","last_fullsized":"2024-04-17T07:30:13.678767+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:14:42.001194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.634491+0000","last_change":"2024-04-17T07:30:12.650282+0000","last_active":"2024-04-17T07:30:13.634491+0000","last_peered":"2024-04-17T07:30:13.634491+0000","last_clean":"2024-04-17T07:30:13.634491+0000","last_became_active":"2024-04-17T07:30:12.649660+0000","last_became_peered":"2024-04-17T07:30:12.649660+0000","last_unstale":"2024-04-17T07:30:13.634491+0000","last_undegraded":"2024-04-17T07:30:13.634491+0000","last_fullsized":"2024-04-17T07:30:13.634491+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:07:45.129127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.644783+0000","last_change":"2024-04-17T07:30:13.642151+0000","last_active":"2024-04-17T07:30:13.644783+0000","last_peered":"2024-04-17T07:30:13.644783+0000","last_clean":"2024-04-17T07:30:13.644783+0000","last_became_active":"2024-04-17T07:30:13.641296+0000","last_became_peered":"2024-04-17T07:30:13.641296+0000","last_unstale":"2024-04-17T07:30:13.644783+0000","last_undegraded":"2024-04-17T07:30:13.644783+0000","last_fullsized":"2024-04-17T07:30:13.644783+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:29.854636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.637938+0000","last_change":"2024-04-17T07:30:12.652338+0000","last_active":"2024-04-17T07:30:13.637938+0000","last_peered":"2024-04-17T07:30:13.637938+0000","last_clean":"2024-04-17T07:30:13.637938+0000","last_became_active":"2024-04-17T07:30:12.650501+0000","last_became_peered":"2024-04-17T07:30:12.650501+0000","last_unstale":"2024-04-17T07:30:13.637938+0000","last_undegraded":"2024-04-17T07:30:13.637938+0000","last_fullsized":"2024-04-17T07:30:13.637938+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:14.343231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.669784+0000","last_change":"2024-04-17T07:30:13.664542+0000","last_active":"2024-04-17T07:30:13.669784+0000","last_peered":"2024-04-17T07:30:13.669784+0000","last_clean":"2024-04-17T07:30:13.669784+0000","last_became_active":"2024-04-17T07:30:13.664266+0000","last_became_peered":"2024-04-17T07:30:13.664266+0000","last_unstale":"2024-04-17T07:30:13.669784+0000","last_undegraded":"2024-04-17T07:30:13.669784+0000","last_fullsized":"2024-04-17T07:30:13.669784+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:56.883439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.658305+0000","last_change":"2024-04-17T07:30:12.656053+0000","last_active":"2024-04-17T07:30:13.658305+0000","last_peered":"2024-04-17T07:30:13.658305+0000","last_clean":"2024-04-17T07:30:13.658305+0000","last_became_active":"2024-04-17T07:30:12.655766+0000","last_became_peered":"2024-04-17T07:30:12.655766+0000","last_unstale":"2024-04-17T07:30:13.658305+0000","last_undegraded":"2024-04-17T07:30:13.658305+0000","last_fullsized":"2024-04-17T07:30:13.658305+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:19.199156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.671723+0000","last_change":"2024-04-17T07:30:13.669172+0000","last_active":"2024-04-17T07:30:13.671723+0000","last_peered":"2024-04-17T07:30:13.671723+0000","last_clean":"2024-04-17T07:30:13.671723+0000","last_became_active":"2024-04-17T07:30:13.668895+0000","last_became_peered":"2024-04-17T07:30:13.668895+0000","last_unstale":"2024-04-17T07:30:13.671723+0000","last_undegraded":"2024-04-17T07:30:13.671723+0000","last_fullsized":"2024-04-17T07:30:13.671723+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:12.604134+0000","last_clean_scrub_stamp":"2024-04-17T07:30:12.604134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:12.621274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.677796+0000","last_change":"2024-04-17T07:30:12.656824+0000","last_active":"2024-04-17T07:30:13.677796+0000","last_peered":"2024-04-17T07:30:13.677796+0000","last_clean":"2024-04-17T07:30:13.677796+0000","last_became_active":"2024-04-17T07:30:12.656476+0000","last_became_peered":"2024-04-17T07:30:12.656476+0000","last_unstale":"2024-04-17T07:30:13.677796+0000","last_undegraded":"2024-04-17T07:30:13.677796+0000","last_fullsized":"2024-04-17T07:30:13.677796+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:11.586357+0000","last_clean_scrub_stamp":"2024-04-17T07:30:11.586357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:21.046479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"86'175","reported_seq":526,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-17T07:30:13.653005+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:30:13.653005+0000","last_peered":"2024-04-17T07:30:13.653005+0000","last_clean":"2024-04-17T07:30:13.653005+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:30:13.653005+0000","last_undegraded":"2024-04-17T07:30:13.653005+0000","last_fullsized":"2024-04-17T07:30:13.653005+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":619,"num_read_kb":1212,"num_write":263,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1761280,"data_stored":1145310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542341,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542340,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542340,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58692,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313148,"statfs":{"total":96636764160,"available":96576663552,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542341,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542341,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40648,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331192,"statfs":{"total":96636764160,"available":96595140608,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542341,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50368,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321472,"statfs":{"total":96636764160,"available":96585187328,"internally_reserved":0,"allocated":1167360,"data_stored":554942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1761280,"data_stored":1145310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:30:18.727 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:30:18.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:30:19.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.004+0000 7fa95e600640 1 Processor -- start 2024-04-17T07:30:19.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.004+0000 7fa95e600640 1 -- start start 2024-04-17T07:30:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.005+0000 7fa95e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9580d4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.005+0000 7fa95e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9580b79b0 con 0x7fa9580b6bf0 2024-04-17T07:30:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.005+0000 7fa95e600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9580d3330 con 0x7fa9580d2bc0 2024-04-17T07:30:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.005+0000 7fa95e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9580d3bd0 con 0x7fa95815ebe0 2024-04-17T07:30:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.006+0000 7fa95dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa9580b6bf0 0x7fa9580b8150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39966/0 (socket says 172.21.15.67:39966) 2024-04-17T07:30:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.006+0000 7fa95dc00640 1 -- 172.21.15.67:0/1186239918 learned_addr learned my addr 172.21.15.67:0/1186239918 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.006+0000 7fa95d200640 1 --2- 172.21.15.67:0/1186239918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9580d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.006+0000 7fa957400640 1 -- 172.21.15.67:0/1186239918 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2738084728 0 0) 0x7fa950001490 con 0x7fa9580b6bf0 2024-04-17T07:30:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.006+0000 7fa957400640 1 -- 172.21.15.67:0/1186239918 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa940003520 con 0x7fa9580b6bf0 2024-04-17T07:30:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.007+0000 7fa95d200640 1 -- 172.21.15.67:0/1186239918 >> v1:172.21.15.187:6789/0 conn(0x7fa9580d2bc0 legacy=0x7fa9580b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.007+0000 7fa95d200640 1 -- 172.21.15.67:0/1186239918 >> v1:172.21.15.67:6789/0 conn(0x7fa9580b6bf0 legacy=0x7fa9580b8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.007+0000 7fa95d200640 1 -- 172.21.15.67:0/1186239918 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9580d3610 con 0x7fa95815ebe0 2024-04-17T07:30:19.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.009+0000 7fa95d200640 1 --2- 172.21.15.67:0/1186239918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9580d4370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097d0 tx=0x7fa94c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b96970188cdc8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.009+0000 7fa957400640 1 -- 172.21.15.67:0/1186239918 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c034070 con 0x7fa95815ebe0 2024-04-17T07:30:19.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.009+0000 7fa957400640 1 -- 172.21.15.67:0/1186239918 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c030800 con 0x7fa95815ebe0 2024-04-17T07:30:19.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.009+0000 7fa957400640 1 -- 172.21.15.67:0/1186239918 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c030bd0 con 0x7fa95815ebe0 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 -- 172.21.15.67:0/1186239918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 msgr2=0x7fa9580d4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 --2- 172.21.15.67:0/1186239918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9580d4370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0097d0 tx=0x7fa94c02b940 comp rx=0 tx=0).stop 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 -- 172.21.15.67:0/1186239918 shutdown_connections 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 --2- 172.21.15.67:0/1186239918 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9580d4370 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 -- 172.21.15.67:0/1186239918 >> 172.21.15.67:0/1186239918 conn(0x7fa9580d16c0 msgr2=0x7fa9580d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 -- 172.21.15.67:0/1186239918 shutdown_connections 2024-04-17T07:30:19.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.010+0000 7fa95e600640 1 -- 172.21.15.67:0/1186239918 wait complete. 2024-04-17T07:30:19.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.012+0000 7fa95e600640 1 Processor -- start 2024-04-17T07:30:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.012+0000 7fa95e600640 1 -- start start 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9580d2bc0 0x7fa95815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9581f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa95815efc0 con 0x7fa9580b6bf0 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9580b7c30 con 0x7fa9580d2bc0 2024-04-17T07:30:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9580d3020 con 0x7fa95815ebe0 2024-04-17T07:30:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35562/0 (socket says 172.21.15.67:35562) 2024-04-17T07:30:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.013+0000 7fa95d200640 1 -- 172.21.15.67:0/1423724086 learned_addr learned my addr 172.21.15.67:0/1423724086 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95dc00640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9581f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa957e00640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9580d2bc0 0x7fa95815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 -- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 msgr2=0x7fa9581f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9581f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 -- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9580d2bc0 msgr2=0x7fa95815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9580d2bc0 0x7fa95815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 -- 172.21.15.67:0/1423724086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa958160430 con 0x7fa9580b6bf0 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95d200640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa94c000c00 tx=0x7fa94c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c034040 con 0x7fa9580b6bf0 2024-04-17T07:30:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c030600 con 0x7fa9580b6bf0 2024-04-17T07:30:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c006e10 con 0x7fa9580b6bf0 2024-04-17T07:30:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.014+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa958160710 con 0x7fa9580b6bf0 2024-04-17T07:30:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.015+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa958160cd0 con 0x7fa9580b6bf0 2024-04-17T07:30:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.016+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa958068a10 con 0x7fa9580b6bf0 2024-04-17T07:30:19.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.017+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fa94c04c060 con 0x7fa9580b6bf0 2024-04-17T07:30:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.018+0000 7fa954c00640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 0x7fa93806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.018+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fa94c0ada30 con 0x7fa9580b6bf0 2024-04-17T07:30:19.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.018+0000 7fa957e00640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 0x7fa93806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.020+0000 7fa957e00640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 0x7fa93806ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa9580b4440 tx=0x7fa944009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.022+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa94c07bd50 con 0x7fa9580b6bf0 2024-04-17T07:30:19.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.304+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fa958160120 con 0x7fa9580b6bf0 2024-04-17T07:30:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.309+0000 7fa954c00640 1 -- 172.21.15.67:0/1423724086 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7fa94c07b650 con 0x7fa9580b6bf0 2024-04-17T07:30:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:30:19.309 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:30:19.309 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:30:19.310 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:30:19.310 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:30:19.310 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:30:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 msgr2=0x7fa93806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 0x7fa93806ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa9580b4440 tx=0x7fa944009040 comp rx=0 tx=0).stop 2024-04-17T07:30:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 msgr2=0x7fa9580ac340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa94c000c00 tx=0x7fa94c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 shutdown_connections 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa95815ebe0 0x7fa9581f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fa93806ca40 0x7fa93806ef00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9580d2bc0 0x7fa95815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 --2- 172.21.15.67:0/1423724086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9580b6bf0 0x7fa9580ac340 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 >> 172.21.15.67:0/1423724086 conn(0x7fa9580d16c0 msgr2=0x7fa9580b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 shutdown_connections 2024-04-17T07:30:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.312+0000 7fa95e600640 1 -- 172.21.15.67:0/1423724086 wait complete. 2024-04-17T07:30:19.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.551+0000 7f5dd4800640 1 Processor -- start 2024-04-17T07:30:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.552+0000 7f5dd4800640 1 -- start start 2024-04-17T07:30:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.553+0000 7f5dd4800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5dcc0cd020 0x7f5dcc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.553+0000 7f5dd4800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5dcc06e020 0x7f5dcc06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd4800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dcc0ceb00 con 0x7f5dcc06f670 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd4800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc0cede0 con 0x7f5dcc06e020 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd4800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc0cf0c0 con 0x7f5dcc0cd020 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5dcc06f670 0x7f5dcc06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39982/0 (socket says 172.21.15.67:39982) 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd1e00640 1 -- 172.21.15.67:0/721126214 learned_addr learned my addr 172.21.15.67:0/721126214 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd2800640 1 --2- 172.21.15.67:0/721126214 >> v2:172.21.15.67:3300/0 conn(0x7f5dcc06e020 0x7f5dcc06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd1400640 1 --2- 172.21.15.67:0/721126214 >> v2:172.21.15.187:3300/0 conn(0x7f5dcc0cd020 0x7f5dcc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd0a00640 1 -- 172.21.15.67:0/721126214 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009058902 0 0) 0x7f5dcc0ceb00 con 0x7f5dcc06f670 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd0a00640 1 -- 172.21.15.67:0/721126214 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5db4003520 con 0x7f5dcc06f670 2024-04-17T07:30:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd2800640 1 -- 172.21.15.67:0/721126214 >> v2:172.21.15.187:3300/0 conn(0x7f5dcc0cd020 msgr2=0x7f5dcc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:19.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd2800640 1 --2- 172.21.15.67:0/721126214 >> v2:172.21.15.187:3300/0 conn(0x7f5dcc0cd020 0x7f5dcc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd2800640 1 -- 172.21.15.67:0/721126214 >> v1:172.21.15.67:6789/0 conn(0x7f5dcc06f670 legacy=0x7f5dcc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.554+0000 7f5dd2800640 1 -- 172.21.15.67:0/721126214 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc0cf3a0 con 0x7f5dcc06e020 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd2800640 1 --2- 172.21.15.67:0/721126214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06e020 0x7f5dcc06e490 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0cce60 tx=0x7f5dc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7664f525e2df541 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd0a00640 1 -- 172.21.15.67:0/721126214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8039070 con 0x7f5dcc06e020 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd0a00640 1 -- 172.21.15.67:0/721126214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc802cc80 con 0x7f5dcc06e020 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd0a00640 1 -- 172.21.15.67:0/721126214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8034660 con 0x7f5dcc06e020 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd4800640 1 -- 172.21.15.67:0/721126214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06e020 msgr2=0x7f5dcc06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.556+0000 7f5dd4800640 1 --2- 172.21.15.67:0/721126214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06e020 0x7f5dcc06e490 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc0cce60 tx=0x7f5dc802bed0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 -- 172.21.15.67:0/721126214 shutdown_connections 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 --2- 172.21.15.67:0/721126214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06e020 0x7f5dcc06e490 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 --2- 172.21.15.67:0/721126214 >> v2:172.21.15.187:3300/0 conn(0x7f5dcc0cd020 0x7f5dcc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 -- 172.21.15.67:0/721126214 >> 172.21.15.67:0/721126214 conn(0x7f5dcc06bf50 msgr2=0x7f5dcc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 -- 172.21.15.67:0/721126214 shutdown_connections 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.557+0000 7f5dd4800640 1 -- 172.21.15.67:0/721126214 wait complete. 2024-04-17T07:30:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.558+0000 7f5dd4800640 1 Processor -- start 2024-04-17T07:30:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.559+0000 7f5dd4800640 1 -- start start 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.559+0000 7f5dd4800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 0x7f5dcc1f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd4800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd4800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5dcc0cd020 0x7f5dcc158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd4800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc0ce8a0 con 0x7f5dcc06f670 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd4800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc0ce720 con 0x7f5dcc06e020 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd4800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5dcc15cb00 con 0x7f5dcc0cd020 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35570/0 (socket says 172.21.15.67:35570) 2024-04-17T07:30:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 -- 172.21.15.67:0/1490954896 learned_addr learned my addr 172.21.15.67:0/1490954896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1e00640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 0x7f5dcc1f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 -- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5dcc0cd020 msgr2=0x7f5dcc158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5dcc0cd020 0x7f5dcc158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 -- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 msgr2=0x7f5dcc1f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 0x7f5dcc1f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1400640 1 -- 172.21.15.67:0/1490954896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc1584f0 con 0x7f5dcc06f670 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.560+0000 7f5dd1e00640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 0x7f5dcc1f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dd1400640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00d800 tx=0x7f5dbc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00b6c0 con 0x7f5dcc06f670 2024-04-17T07:30:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc019070 con 0x7f5dcc06f670 2024-04-17T07:30:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00ba80 con 0x7f5dcc06f670 2024-04-17T07:30:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc1587d0 con 0x7f5dcc06f670 2024-04-17T07:30:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.561+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc0c76d0 con 0x7f5dcc06f670 2024-04-17T07:30:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.562+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc068b90 con 0x7f5dcc06f670 2024-04-17T07:30:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.563+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f5dbc00bc20 con 0x7f5dcc06f670 2024-04-17T07:30:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.564+0000 7f5dc6000640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 0x7f5db006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.564+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f5dbc08c1a0 con 0x7f5dcc06f670 2024-04-17T07:30:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.565+0000 7f5dd1e00640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 0x7f5db006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.566+0000 7f5dd1e00640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 0x7f5db006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0005880 tx=0x7f5dc0009040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.569+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5dbc05a4b0 con 0x7f5dcc06f670 2024-04-17T07:30:19.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.817+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dcc1f3d50 con 0x7f5dcc06f670 2024-04-17T07:30:19.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.818+0000 7f5dc6000640 1 -- 172.21.15.67:0/1490954896 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7f5dbc059db0 con 0x7f5dcc06f670 2024-04-17T07:30:19.818 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:19.819 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:19.819 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:19.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 msgr2=0x7f5db006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 0x7f5db006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0005880 tx=0x7f5dc0009040 comp rx=0 tx=0).stop 2024-04-17T07:30:19.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 msgr2=0x7f5dcc157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:19.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00d800 tx=0x7f5dbc00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 shutdown_connections 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5dcc0cd020 0x7f5dcc158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dcc06f670 0x7f5dcc157b80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f5db006ca10 0x7f5db006eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 --2- 172.21.15.67:0/1490954896 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5dcc06e020 0x7f5dcc1f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 >> 172.21.15.67:0/1490954896 conn(0x7f5dcc06bf50 msgr2=0x7f5dcc15b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 shutdown_connections 2024-04-17T07:30:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:19.821+0000 7f5dd4800640 1 -- 172.21.15.67:0/1490954896 wait complete. 2024-04-17T07:30:20.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:30:13.116971+0000', 'modified': '2024-04-17T07:30:16.196310+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6906}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6906': {'gid': 6906, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.67:6835/381512147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 381512147}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 381512147}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6906, 'qdb_cluster': [6906]} 2024-04-17T07:30:20.246 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:30:20.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T07:30:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.476+0000 7fdd0f200640 1 Processor -- start 2024-04-17T07:30:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.476+0000 7fdd0f200640 1 -- start start 2024-04-17T07:30:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd100afbf0 0x7fdd1006e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd1006eec0 0x7fdd1015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd100cb420 con 0x7fdd1015f900 2024-04-17T07:30:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd100cc030 con 0x7fdd1006eec0 2024-04-17T07:30:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd100cb700 con 0x7fdd100afbf0 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd05400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd1006eec0 0x7fdd1015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd05400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd1006eec0 0x7fdd1015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35574/0 (socket says 172.21.15.67:35574) 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd05400640 1 -- 172.21.15.67:0/2406742705 learned_addr learned my addr 172.21.15.67:0/2406742705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.478+0000 7fdd0de00640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd100afbf0 0x7fdd1006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.479+0000 7fdd05400640 1 -- 172.21.15.67:0/2406742705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd100afbf0 msgr2=0x7fdd1006e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.479+0000 7fdd05400640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd100afbf0 0x7fdd1006e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.479+0000 7fdd05400640 1 -- 172.21.15.67:0/2406742705 >> v1:172.21.15.187:6789/0 conn(0x7fdd1015f900 legacy=0x7fdd1015bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.479+0000 7fdd05400640 1 -- 172.21.15.67:0/2406742705 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd100cb9e0 con 0x7fdd1006eec0 2024-04-17T07:30:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.480+0000 7fdd05400640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd1015f3c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80098a0 tx=0x7fdcf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0cfc186ac3c4cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.480+0000 7fdd0d400640 1 -- 172.21.15.67:0/2406742705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8035020 con 0x7fdd1006eec0 2024-04-17T07:30:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.480+0000 7fdd0d400640 1 -- 172.21.15.67:0/2406742705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf8031640 con 0x7fdd1006eec0 2024-04-17T07:30:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.480+0000 7fdd0d400640 1 -- 172.21.15.67:0/2406742705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf802d040 con 0x7fdd1006eec0 2024-04-17T07:30:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 -- 172.21.15.67:0/2406742705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 msgr2=0x7fdd1015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd1015f3c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80098a0 tx=0x7fdcf802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 -- 172.21.15.67:0/2406742705 shutdown_connections 2024-04-17T07:30:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd1015f3c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 --2- 172.21.15.67:0/2406742705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd100afbf0 0x7fdd1006e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 -- 172.21.15.67:0/2406742705 >> 172.21.15.67:0/2406742705 conn(0x7fdd1006bef0 msgr2=0x7fdd100b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 -- 172.21.15.67:0/2406742705 shutdown_connections 2024-04-17T07:30:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.481+0000 7fdd0f200640 1 -- 172.21.15.67:0/2406742705 wait complete. 2024-04-17T07:30:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.483+0000 7fdd0f200640 1 Processor -- start 2024-04-17T07:30:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.483+0000 7fdd0f200640 1 -- start start 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100afbf0 0x7fdd1014a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 0x7fdd10154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd100affd0 con 0x7fdd1006eec0 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdd10160110 con 0x7fdd100afbf0 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd100b0150 con 0x7fdd1015f900 2024-04-17T07:30:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35586/0 (socket says 172.21.15.67:35586) 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.484+0000 7fdd0de00640 1 -- 172.21.15.67:0/1590023231 learned_addr learned my addr 172.21.15.67:0/1590023231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 -- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 msgr2=0x7fdd10154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0e800640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 0x7fdd10154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 0x7fdd10154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd05400640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100afbf0 0x7fdd1014a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 -- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100afbf0 msgr2=0x7fdd1014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100afbf0 0x7fdd1014a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 -- 172.21.15.67:0/1590023231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1014a540 con 0x7fdd1006eec0 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0e800640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 0x7fdd10154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000d800 tx=0x7fdd0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd06a00640 1 -- 172.21.15.67:0/1590023231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000b6c0 con 0x7fdd1006eec0 2024-04-17T07:30:20.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd06a00640 1 -- 172.21.15.67:0/1590023231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd00019070 con 0x7fdd1006eec0 2024-04-17T07:30:20.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.485+0000 7fdd06a00640 1 -- 172.21.15.67:0/1590023231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000ba80 con 0x7fdd1006eec0 2024-04-17T07:30:20.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.486+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd101f4a20 con 0x7fdd1006eec0 2024-04-17T07:30:20.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.486+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd101f4fe0 con 0x7fdd1006eec0 2024-04-17T07:30:20.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.489+0000 7fdd06a00640 1 -- 172.21.15.67:0/1590023231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fdd0000bc20 con 0x7fdd1006eec0 2024-04-17T07:30:20.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.490+0000 7fdd06a00640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 0x7fdcec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.490+0000 7fdd06a00640 1 -- 172.21.15.67:0/1590023231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fdd0008c270 con 0x7fdd1006eec0 2024-04-17T07:30:20.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.490+0000 7fdd17967740 1 Processor -- start 2024-04-17T07:30:20.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.490+0000 7fdd05400640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 0x7fdcec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.490+0000 7fdd17967740 1 -- start start 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 0x555ce57cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555ce5739980 con 0x555ce580ee70 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x555ce56e0220 con 0x555ce57cdb40 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.491+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x555ce5735940 con 0x555ce57fdb20 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35602/0 (socket says 172.21.15.67:35602) 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 -- 172.21.15.67:0/1481030771 learned_addr learned my addr 172.21.15.67:0/1481030771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd05400640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 0x7fdcec06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8036000 tx=0x7fdcf8008030 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd05400640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 0x555ce57cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 -- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 msgr2=0x555ce57fdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 -- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 msgr2=0x555ce57cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 0x555ce57cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd0e800640 1 -- 172.21.15.67:0/1481030771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce544c2d0 con 0x555ce580ee70 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.492+0000 7fdd05400640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 0x555ce57cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdd0e800640 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fdd08007f10 tx=0x7fdd0800c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b454b03b76cd3e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdcfe600640 1 -- 172.21.15.67:0/1481030771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08014070 con 0x555ce580ee70 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdcfe600640 1 -- 172.21.15.67:0/1481030771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0800f830 con 0x555ce580ee70 2024-04-17T07:30:20.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdcfe600640 1 -- 172.21.15.67:0/1481030771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800fc00 con 0x555ce580ee70 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdd17967740 1 -- 172.21.15.67:0/1481030771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 msgr2=0x555ce580f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.493+0000 7fdd17967740 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fdd08007f10 tx=0x7fdd0800c670 comp rx=0 tx=0).stop 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 -- 172.21.15.67:0/1481030771 shutdown_connections 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce580f2f0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57cdb40 0x555ce57cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 --2- 172.21.15.67:0/1481030771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 -- 172.21.15.67:0/1481030771 >> 172.21.15.67:0/1481030771 conn(0x555ce57cfdb0 msgr2=0x555ce5818930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 -- 172.21.15.67:0/1481030771 shutdown_connections 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.494+0000 7fdd17967740 1 -- 172.21.15.67:0/1481030771 wait complete. 2024-04-17T07:30:20.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.495+0000 7fdd17967740 1 -- start start 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.495+0000 7fdd17967740 1 Processor -- start 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555ce57d3660 con 0x555ce580ee70 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x555ce56f3590 con 0x555ce57d16a0 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x555ce57fe770 con 0x555ce57fdb20 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd17967740 10 client.0 _ll_register_callbacks cb 0x555ce544bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd05400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd05400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35612/0 (socket says 172.21.15.67:35612) 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.496+0000 7fdd05400640 1 -- 172.21.15.67:0/3511989407 learned_addr learned my addr 172.21.15.67:0/3511989407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd0e800640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd0de00640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 -- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 msgr2=0x555ce57fb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 -- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 msgr2=0x555ce57d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 -- 172.21.15.67:0/3511989407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce55166b0 con 0x555ce580ee70 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd0e800640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd0de00640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd05400640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80099c0 tx=0x7fdcf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985f32b3c2792bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdcf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8035040 con 0x555ce580ee70 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fdcf803ad80 con 0x555ce580ee70 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.497+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd0800f830 con 0x555ce580ee70 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdcf8058050 con 0x555ce580ee70 2024-04-17T07:30:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 10 client.7053 fetch_fsmap learned FSMap version 45 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x555ce580d000 con 0x555ce580ee70 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0800f830 con 0x555ce580ee70 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 45) ==== 3495+0+0 (secure 0 0 0) 0x7fdcf8050480 con 0x555ce580ee70 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 10 client.7053 fetch_fsmap finished waiting for FSMap version 45 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 10 client.7053 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 20 client.7053 populate_metadata read hostname 'smithi067' 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce5831800 0x555ce580b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 4 client.7053 mds_command: new command op to 6906 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 --> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ce57473a0 con 0x555ce5831800 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.498+0000 7fdd0de00640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce5831800 0x555ce580b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.499+0000 7fdd0de00640 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce5831800 0x555ce580b960 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb4a7d4d41de403 server_cookie=260ca47f77d575af in_seq=0 out_seq=0 2024-04-17T07:30:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.499+0000 7fdcf2a00640 10 client.7053 ms_handle_connect on v2:172.21.15.67:6833/381512147 2024-04-17T07:30:20.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdcf2a00640 1 -- 172.21.15.67:0/3511989407 <== mds.0 v2:172.21.15.67:6833/381512147 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ce57473a0 con 0x555ce5831800 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdcf2a00640 10 client.7053 handle_command_reply: tid=0 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 client.7053 shutdown 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 msgr2=0x555ce58012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80099c0 tx=0x7fdcf8004730 comp rx=0 tx=0).stop 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 shutdown_connections 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce5831800 0x555ce580b960 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58012c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 --2- 172.21.15.67:0/3511989407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce57fb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdcf2a00640 0 client.7053 ms_handle_reset on v2:172.21.15.67:6833/381512147 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 >> 172.21.15.67:0/3511989407 conn(0x555ce57cdb40 msgr2=0x555ce57cd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 shutdown_connections 2024-04-17T07:30:20.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.500+0000 7fdd17967740 1 -- 172.21.15.67:0/3511989407 wait complete. 2024-04-17T07:30:20.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.501+0000 7fdd17967740 20 client.7053 trim_cache size 0 max 16384 2024-04-17T07:30:20.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.521+0000 7fdd17967740 1 Processor -- start 2024-04-17T07:30:20.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.521+0000 7fdd17967740 1 -- start start 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce5817b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd100b56d0 con 0x555ce580ee70 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdd100b4a80 con 0x555ce57d16a0 2024-04-17T07:30:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd100b5080 con 0x555ce57fdb20 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35614/0 (socket says 172.21.15.67:35614) 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 -- 172.21.15.67:0/4054764270 learned_addr learned my addr 172.21.15.67:0/4054764270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd05400640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce5817b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 -- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 msgr2=0x555ce5817b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0de00640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce5817b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 -- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 msgr2=0x555ce57ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd0e800640 1 -- 172.21.15.67:0/4054764270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce57d1e70 con 0x555ce580ee70 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.522+0000 7fdd05400640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce5817b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.523+0000 7fdd0e800640 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800cb70 tx=0x7fdd08011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34af312b60ce326a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.523+0000 7fdcfdc00640 1 -- 172.21.15.67:0/4054764270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08014040 con 0x555ce580ee70 2024-04-17T07:30:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.523+0000 7fdcfdc00640 1 -- 172.21.15.67:0/4054764270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd08007a20 con 0x555ce580ee70 2024-04-17T07:30:20.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.523+0000 7fdcfdc00640 1 -- 172.21.15.67:0/4054764270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08003b70 con 0x555ce580ee70 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 -- 172.21.15.67:0/4054764270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 msgr2=0x555ce58180c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800cb70 tx=0x7fdd08011000 comp rx=0 tx=0).stop 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 -- 172.21.15.67:0/4054764270 shutdown_connections 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce580ee70 0x555ce58180c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57fdb20 0x555ce5817b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 --2- 172.21.15.67:0/4054764270 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce57d16a0 0x555ce57ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 -- 172.21.15.67:0/4054764270 >> 172.21.15.67:0/4054764270 conn(0x555ce57cdb40 msgr2=0x555ce580e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 -- 172.21.15.67:0/4054764270 shutdown_connections 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.524+0000 7fdd17967740 1 -- 172.21.15.67:0/4054764270 wait complete. 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.525+0000 7fdd17967740 1 -- start start 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.525+0000 7fdd17967740 1 Processor -- start 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57d16a0 0x555ce589ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce580ee70 0x555ce57febb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555ce5831680 con 0x555ce57fdb20 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x555ce5831ef0 con 0x555ce580ee70 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x555ce5831cd0 con 0x555ce57d16a0 2024-04-17T07:30:20.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd17967740 10 client.0 _ll_register_callbacks cb 0x7fdcf80096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35616/0 (socket says 172.21.15.67:35616) 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 -- 172.21.15.67:0/1365450456 learned_addr learned my addr 172.21.15.67:0/1365450456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 -- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57d16a0 msgr2=0x555ce589ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57d16a0 0x555ce589ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 -- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce580ee70 msgr2=0x555ce57febb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce580ee70 0x555ce57febb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.526+0000 7fdd05400640 1 -- 172.21.15.67:0/1365450456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce57f94a0 con 0x555ce57fdb20 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdd05400640 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80090b0 tx=0x7fdcf8009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d2e1d350884bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdcf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8055070 con 0x555ce57fdb20 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fdcf805eaf0 con 0x555ce57fdb20 2024-04-17T07:30:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd08007a20 con 0x555ce57fdb20 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fdcf8035040 con 0x555ce57fdb20 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdd17967740 10 client.7059 fetch_fsmap learned FSMap version 45 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.527+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x555ce589c470 con 0x555ce57fdb20 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd08007a20 con 0x555ce57fdb20 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 45) ==== 3495+0+0 (secure 0 0 0) 0x7fdcf8065050 con 0x555ce57fdb20 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 10 client.7059 fetch_fsmap finished waiting for FSMap version 45 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 10 client.7059 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 20 client.7059 populate_metadata read hostname 'smithi067' 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce589b3f0 0x555ce589b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 4 client.7059 mds_command: new command op to 6906 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 --> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] -- command(tid 0: {"prefix": "status"}) -- 0x555ce5741210 con 0x555ce589b3f0 2024-04-17T07:30:20.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.528+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce589b3f0 0x555ce589b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd0de00640 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce589b3f0 0x555ce589b7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4afb2b11b3a5bb11 server_cookie=4ee16386a315efe3 in_seq=0 out_seq=0 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdcf2a00640 10 client.7059 ms_handle_connect on v2:172.21.15.67:6833/381512147 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdcf2a00640 1 -- 172.21.15.67:0/1365450456 <== mds.0 v2:172.21.15.67:6833/381512147 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555ce5741210 con 0x555ce589b3f0 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdcf2a00640 10 client.7059 handle_command_reply: tid=0 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 client.7059 shutdown 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 msgr2=0x555ce57fe670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80090b0 tx=0x7fdcf8009240 comp rx=0 tx=0).stop 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 shutdown_connections 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:6833/381512147,v1:172.21.15.67:6835/381512147] conn(0x555ce589b3f0 0x555ce589b7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x555ce580ee70 0x555ce57febb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ce57fdb20 0x555ce57fe670 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 --2- 172.21.15.67:0/1365450456 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x555ce57d16a0 0x555ce589ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdcf2a00640 0 client.7059 ms_handle_reset on v2:172.21.15.67:6833/381512147 2024-04-17T07:30:20.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 >> 172.21.15.67:0/1365450456 conn(0x555ce57cdb40 msgr2=0x555ce5828f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.529+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 shutdown_connections 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd17967740 1 -- 172.21.15.67:0/1365450456 wait complete. 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd17967740 20 client.7059 trim_cache size 0 max 16384 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6906, 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:30:20.534 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.9048432069999999, 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 45, 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 98, 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 98, 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 19.756494714999999 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:30:20.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 msgr2=0x7fdcec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 0x7fdcec06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8036000 tx=0x7fdcf8008030 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 msgr2=0x7fdd101520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000d800 tx=0x7fdd0000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 shutdown_connections 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd1015f900 0x7fdd10154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fdcec06ca10 0x7fdcec06eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100afbf0 0x7fdd1014a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 --2- 172.21.15.67:0/1590023231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd1006eec0 0x7fdd101520c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.530+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 >> 172.21.15.67:0/1590023231 conn(0x7fdd1006bef0 msgr2=0x7fdd1015c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.531+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 shutdown_connections 2024-04-17T07:30:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.531+0000 7fdd0f200640 1 -- 172.21.15.67:0/1590023231 wait complete. 2024-04-17T07:30:20.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:30:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.774+0000 7f05f5400640 1 Processor -- start 2024-04-17T07:30:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.774+0000 7f05f5400640 1 -- start start 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f5400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f05f00cf8f0 0x7f05f00b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05f00d12a0 0x7f05f00d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f5400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05f00cfe20 con 0x7f05f00b9f00 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f5400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f05f00d1c50 con 0x7f05f00d12a0 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f5400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f05f00cf640 con 0x7f05f00cf8f0 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05f00d12a0 0x7f05f00d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05f00d12a0 0x7f05f00d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35618/0 (socket says 172.21.15.67:35618) 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05f4a00640 1 -- 172.21.15.67:0/2971822069 learned_addr learned my addr 172.21.15.67:0/2971822069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.776+0000 7f05ef400640 1 --2- 172.21.15.67:0/2971822069 >> v2:172.21.15.187:3300/0 conn(0x7f05f00cf8f0 0x7f05f00b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.777+0000 7f05f4a00640 1 -- 172.21.15.67:0/2971822069 >> v2:172.21.15.187:3300/0 conn(0x7f05f00cf8f0 msgr2=0x7f05f00b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.777+0000 7f05f4a00640 1 --2- 172.21.15.67:0/2971822069 >> v2:172.21.15.187:3300/0 conn(0x7f05f00cf8f0 0x7f05f00b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.777+0000 7f05f4a00640 1 -- 172.21.15.67:0/2971822069 >> v1:172.21.15.187:6789/0 conn(0x7f05f00b9f00 legacy=0x7f05f01634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.777+0000 7f05f4a00640 1 -- 172.21.15.67:0/2971822069 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f00d0100 con 0x7f05f00d12a0 2024-04-17T07:30:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.778+0000 7f05f4a00640 1 --2- 172.21.15.67:0/2971822069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00d12a0 0x7f05f00d1710 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f05f00d2370 tx=0x7f05e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfff9678c9e43c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.778+0000 7f05eea00640 1 -- 172.21.15.67:0/2971822069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e802e070 con 0x7f05f00d12a0 2024-04-17T07:30:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.778+0000 7f05eea00640 1 -- 172.21.15.67:0/2971822069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e8037070 con 0x7f05f00d12a0 2024-04-17T07:30:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.778+0000 7f05eea00640 1 -- 172.21.15.67:0/2971822069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8031a00 con 0x7f05f00d12a0 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 -- 172.21.15.67:0/2971822069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00d12a0 msgr2=0x7f05f00d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 --2- 172.21.15.67:0/2971822069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00d12a0 0x7f05f00d1710 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f05f00d2370 tx=0x7f05e802b880 comp rx=0 tx=0).stop 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 -- 172.21.15.67:0/2971822069 shutdown_connections 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 --2- 172.21.15.67:0/2971822069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00d12a0 0x7f05f00d1710 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 --2- 172.21.15.67:0/2971822069 >> v2:172.21.15.187:3300/0 conn(0x7f05f00cf8f0 0x7f05f00b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 -- 172.21.15.67:0/2971822069 >> 172.21.15.67:0/2971822069 conn(0x7f05f006cb60 msgr2=0x7f05f006cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 -- 172.21.15.67:0/2971822069 shutdown_connections 2024-04-17T07:30:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.779+0000 7f05f5400640 1 -- 172.21.15.67:0/2971822069 wait complete. 2024-04-17T07:30:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.781+0000 7f05f5400640 1 Processor -- start 2024-04-17T07:30:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.781+0000 7f05f5400640 1 -- start start 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05f00cf8f0 0x7f05f00cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05f00d12a0 0x7f05f0152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f05f00ba2e0 con 0x7f05f00b9f00 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f05f00d3420 con 0x7f05f00d12a0 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05f5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f05f00d03e0 con 0x7f05f00cf8f0 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35628/0 (socket says 172.21.15.67:35628) 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.782+0000 7f05efe00640 1 -- 172.21.15.67:0/4125761321 learned_addr learned my addr 172.21.15.67:0/4125761321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 -- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05f00cf8f0 msgr2=0x7f05f00cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05f00cf8f0 0x7f05f00cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 -- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05f00d12a0 msgr2=0x7f05f0152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05f00d12a0 0x7f05f0152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 -- 172.21.15.67:0/4125761321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f00c8f30 con 0x7f05f00b9f00 2024-04-17T07:30:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05efe00640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c7f0 tx=0x7f05e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0016020 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e000aa50 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e000ae40 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f00c9210 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.783+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f00c97d0 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.785+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f00cbba0 con 0x7f05f00b9f00 2024-04-17T07:30:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.786+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f05e0019020 con 0x7f05f00b9f00 2024-04-17T07:30:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.787+0000 7f05e7e00640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 0x7f05d006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:20.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.787+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f05e008ba40 con 0x7f05f00b9f00 2024-04-17T07:30:20.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.787+0000 7f05ef400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 0x7f05d006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:20.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.789+0000 7f05ef400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 0x7f05d006eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05f01f8220 tx=0x7f05dc008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:20.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:20.792+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f05e0020070 con 0x7f05f00b9f00 2024-04-17T07:30:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.037+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f05f00755d0 con 0x7f05f00b9f00 2024-04-17T07:30:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.038+0000 7f05e7e00640 1 -- 172.21.15.67:0/4125761321 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) ==== 53+0+73 (secure 0 0 0) 0x7f05e008be20 con 0x7f05f00b9f00 2024-04-17T07:30:21.038 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:30:21.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 msgr2=0x7f05d006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:21.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 0x7f05d006eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05f01f8220 tx=0x7f05dc008040 comp rx=0 tx=0).stop 2024-04-17T07:30:21.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 msgr2=0x7f05f00c8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:21.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f05e000c7f0 tx=0x7f05e000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:30:21.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 shutdown_connections 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f05d006ca10 0x7f05d006eed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05f00d12a0 0x7f05f0152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05f00cf8f0 0x7f05f00cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 --2- 172.21.15.67:0/4125761321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f00b9f00 0x7f05f00c8910 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 >> 172.21.15.67:0/4125761321 conn(0x7f05f006cb60 msgr2=0x7f05f00b5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 shutdown_connections 2024-04-17T07:30:21.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21.041+0000 7f05f5400640 1 -- 172.21.15.67:0/4125761321 wait complete. 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:30:21.061 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:30:21.062 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:30:21.062 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:30:21.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.062 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:30:21.117 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:30:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:30:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:30:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:30:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41623sec preferred_lft 41623sec 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:30:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:30:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:30:21.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.121 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:30:21.217 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:30:21.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.222 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:30:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:30:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:30:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:30:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:30:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:30:21.281 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:30:21.282 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:30:21.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:30:21.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:30:21.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:30:21.282 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:30:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:30:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:30:21.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.539 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:30:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:30:21.586 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:30:21.587 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:30:21.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:30:21.680 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:30:21.680 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:30:21.733 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:30:21.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:30:21.734 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:30:21.784 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:30:21.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:30:21.925 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:30:21.925 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:30:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:30:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:30:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:30:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:30:21.958 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:30:21.960 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:30:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:30:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:30:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:30:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:30:21.965 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:30:21.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:21.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:30:22.058 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:30:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:30:22.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:30:22.183 INFO:teuthology.orchestra.run.smithi067.stdout:client6019 2024-04-17T07:30:22.183 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:22.183 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:22.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:30:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.511+0000 7f9cdbe00640 1 Processor -- start 2024-04-17T07:30:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.511+0000 7f9cdbe00640 1 -- start start 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9cdc0cbc00 0x7f9cdc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0b63c0 0x7f9cdc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9cdc0723a0 0x7f9cdc160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9cdc0cfe90 con 0x7f9cdc0723a0 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9cdc15dc70 con 0x7f9cdc0cbc00 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9cdc15d060 con 0x7f9cdc0b63c0 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9cdc0723a0 0x7f9cdc160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9cdc0723a0 0x7f9cdc160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35630/0 (socket says 172.21.15.67:35630) 2024-04-17T07:30:22.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cdb400640 1 -- 172.21.15.67:0/3000749505 learned_addr learned my addr 172.21.15.67:0/3000749505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.513+0000 7f9cd3e00640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0b63c0 0x7f9cdc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdaa00640 1 --2- 172.21.15.67:0/3000749505 >> v2:172.21.15.187:3300/0 conn(0x7f9cdc0cbc00 0x7f9cdc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdb400640 1 -- 172.21.15.67:0/3000749505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0b63c0 msgr2=0x7f9cdc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdb400640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0b63c0 0x7f9cdc071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdb400640 1 -- 172.21.15.67:0/3000749505 >> v2:172.21.15.187:3300/0 conn(0x7f9cdc0cbc00 msgr2=0x7f9cdc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdb400640 1 --2- 172.21.15.67:0/3000749505 >> v2:172.21.15.187:3300/0 conn(0x7f9cdc0cbc00 0x7f9cdc0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.514+0000 7f9cdb400640 1 -- 172.21.15.67:0/3000749505 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc15d340 con 0x7f9cdc0723a0 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.515+0000 7f9cdb400640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc160540 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0702d0 tx=0x7f9cd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770fd97786e1d94 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.515+0000 7f9cda000640 1 -- 172.21.15.67:0/3000749505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd402ca70 con 0x7f9cdc0723a0 2024-04-17T07:30:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.515+0000 7f9cda000640 1 -- 172.21.15.67:0/3000749505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd402cc10 con 0x7f9cdc0723a0 2024-04-17T07:30:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.515+0000 7f9cda000640 1 -- 172.21.15.67:0/3000749505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd40343f0 con 0x7f9cdc0723a0 2024-04-17T07:30:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 -- 172.21.15.67:0/3000749505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 msgr2=0x7f9cdc160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc160540 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0702d0 tx=0x7f9cd402bed0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 -- 172.21.15.67:0/3000749505 shutdown_connections 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc160540 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/3000749505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0b63c0 0x7f9cdc071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/3000749505 >> v2:172.21.15.187:3300/0 conn(0x7f9cdc0cbc00 0x7f9cdc0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 -- 172.21.15.67:0/3000749505 >> 172.21.15.67:0/3000749505 conn(0x7f9cdc06bef0 msgr2=0x7f9cdc0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 -- 172.21.15.67:0/3000749505 shutdown_connections 2024-04-17T07:30:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.516+0000 7f9cdbe00640 1 -- 172.21.15.67:0/3000749505 wait complete. 2024-04-17T07:30:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.518+0000 7f9cdbe00640 1 Processor -- start 2024-04-17T07:30:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.518+0000 7f9cdbe00640 1 -- start start 2024-04-17T07:30:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cdc0b63c0 0x7f9cdc15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0cbc00 0x7f9cdc15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc0d02d0 con 0x7f9cdc0723a0 2024-04-17T07:30:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc15d900 con 0x7f9cdc0b63c0 2024-04-17T07:30:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9cdc0cbfe0 con 0x7f9cdc0cbc00 2024-04-17T07:30:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35644/0 (socket says 172.21.15.67:35644) 2024-04-17T07:30:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.519+0000 7f9cdaa00640 1 -- 172.21.15.67:0/2615703026 learned_addr learned my addr 172.21.15.67:0/2615703026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdb400640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0cbc00 0x7f9cdc15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cd3e00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cdc0b63c0 0x7f9cdc15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 -- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0cbc00 msgr2=0x7f9cdc15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0cbc00 0x7f9cdc15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 -- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cdc0b63c0 msgr2=0x7f9cdc15eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cdc0b63c0 0x7f9cdc15eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 -- 172.21.15.67:0/2615703026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc06e340 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.520+0000 7f9cdaa00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00d5f0 tx=0x7f9ccc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.521+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0125f0 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.521+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ccc019070 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.521+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0129d0 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.521+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc06e620 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.521+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc15f520 con 0x7f9cdc0723a0 2024-04-17T07:30:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.522+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cdc15f8d0 con 0x7f9cdc0723a0 2024-04-17T07:30:22.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.523+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7f9ccc012b70 con 0x7f9cdc0723a0 2024-04-17T07:30:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.524+0000 7f9cd2a00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 0x7f9cb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.524+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f9ccc08c140 con 0x7f9cdc0723a0 2024-04-17T07:30:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.525+0000 7f9cd3e00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 0x7f9cb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:22.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.526+0000 7f9cd3e00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 0x7f9cb806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc160fa0 tx=0x7f9cc4008040 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.529+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9ccc05a450 con 0x7f9cdc0723a0 2024-04-17T07:30:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.761+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 --> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f9cdc068a10 con 0x7f9cb806ca10 2024-04-17T07:30:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.765+0000 7f9cd2a00640 1 -- 172.21.15.67:0/2615703026 <== mgr.5794 v2:172.21.15.187:6832/3743052516 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f9cdc068a10 con 0x7f9cb806ca10 2024-04-17T07:30:22.765 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:30:22.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:22.766 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:30:22.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:22.766 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:30:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 msgr2=0x7f9cb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 0x7f9cb806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc160fa0 tx=0x7f9cc4008040 comp rx=0 tx=0).stop 2024-04-17T07:30:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 msgr2=0x7f9cdc1605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00d5f0 tx=0x7f9ccc00dab0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 shutdown_connections 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cdc0cbc00 0x7f9cdc15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f9cb806ca10 0x7f9cb806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cdc0b63c0 0x7f9cdc15eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 --2- 172.21.15.67:0/2615703026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0723a0 0x7f9cdc1605a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 >> 172.21.15.67:0/2615703026 conn(0x7f9cdc06bef0 msgr2=0x7f9cdc0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 shutdown_connections 2024-04-17T07:30:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:22.768+0000 7f9cdbe00640 1 -- 172.21.15.67:0/2615703026 wait complete. 2024-04-17T07:30:22.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:30:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.006+0000 7fc156600640 1 Processor -- start 2024-04-17T07:30:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.006+0000 7fc156600640 1 -- start start 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.007+0000 7fc156600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc15015ecb0 0x7fc1500d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.007+0000 7fc156600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1500cfab0 0x7fc1500b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.007+0000 7fc156600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1500d00b0 con 0x7fc1500d3210 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.007+0000 7fc156600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc1500d0c30 con 0x7fc15015ecb0 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.007+0000 7fc156600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc1500d0390 con 0x7fc1500cfab0 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc15015ecb0 0x7fc1500d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc15015ecb0 0x7fc1500d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35660/0 (socket says 172.21.15.67:35660) 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 -- 172.21.15.67:0/4292333206 learned_addr learned my addr 172.21.15.67:0/4292333206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 -- 172.21.15.67:0/4292333206 >> v2:172.21.15.187:3300/0 conn(0x7fc1500cfab0 msgr2=0x7fc1500b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 --2- 172.21.15.67:0/4292333206 >> v2:172.21.15.187:3300/0 conn(0x7fc1500cfab0 0x7fc1500b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 -- 172.21.15.67:0/4292333206 >> v1:172.21.15.187:6789/0 conn(0x7fc1500d3210 legacy=0x7fc1500cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.008+0000 7fc155200640 1 -- 172.21.15.67:0/4292333206 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1500d0670 con 0x7fc15015ecb0 2024-04-17T07:30:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.009+0000 7fc155200640 1 --2- 172.21.15.67:0/4292333206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc15015ecb0 0x7fc1500d2cd0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc1500c81b0 tx=0x7fc14402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b517edd197baa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.009+0000 7fc14f400640 1 -- 172.21.15.67:0/4292333206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14402c9f0 con 0x7fc15015ecb0 2024-04-17T07:30:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc14f400640 1 -- 172.21.15.67:0/4292333206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14402cb90 con 0x7fc15015ecb0 2024-04-17T07:30:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc14f400640 1 -- 172.21.15.67:0/4292333206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1440343f0 con 0x7fc15015ecb0 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 -- 172.21.15.67:0/4292333206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc15015ecb0 msgr2=0x7fc1500d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 --2- 172.21.15.67:0/4292333206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc15015ecb0 0x7fc1500d2cd0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc1500c81b0 tx=0x7fc14402be50 comp rx=0 tx=0).stop 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 -- 172.21.15.67:0/4292333206 shutdown_connections 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 --2- 172.21.15.67:0/4292333206 >> v2:172.21.15.187:3300/0 conn(0x7fc1500cfab0 0x7fc1500b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 --2- 172.21.15.67:0/4292333206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc15015ecb0 0x7fc1500d2cd0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 -- 172.21.15.67:0/4292333206 >> 172.21.15.67:0/4292333206 conn(0x7fc15008bff0 msgr2=0x7fc1500d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 -- 172.21.15.67:0/4292333206 shutdown_connections 2024-04-17T07:30:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.010+0000 7fc156600640 1 -- 172.21.15.67:0/4292333206 wait complete. 2024-04-17T07:30:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.012+0000 7fc156600640 1 Processor -- start 2024-04-17T07:30:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.012+0000 7fc156600640 1 -- start start 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1500d3210 0x7fc150158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc15015ecb0 0x7fc1500b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc15015f210 con 0x7fc1500cfab0 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc15015f090 con 0x7fc15015ecb0 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc156600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1500b4500 con 0x7fc1500d3210 2024-04-17T07:30:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.013+0000 7fc155200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35664/0 (socket says 172.21.15.67:35664) 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 -- 172.21.15.67:0/3766665059 learned_addr learned my addr 172.21.15.67:0/3766665059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 -- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1500d3210 msgr2=0x7fc150158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1500d3210 0x7fc150158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 -- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc15015ecb0 msgr2=0x7fc1500b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc15015ecb0 0x7fc1500b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 -- 172.21.15.67:0/3766665059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1500cbcb0 con 0x7fc1500cfab0 2024-04-17T07:30:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.014+0000 7fc155200640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc144036040 tx=0x7fc144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.015+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144043020 con 0x7fc1500cfab0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.015+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14403e450 con 0x7fc1500cfab0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.015+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1500cbf90 con 0x7fc1500cfab0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.015+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144009510 con 0x7fc1500cfab0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.015+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1500cc550 con 0x7fc1500cfab0 2024-04-17T07:30:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.016+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc150068a10 con 0x7fc1500cfab0 2024-04-17T07:30:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.017+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 38) ==== 91142+0+0 (secure 0 0 0) 0x7fc144046020 con 0x7fc1500cfab0 2024-04-17T07:30:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.018+0000 7fc14cc00640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 0x7fc13006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.018+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fc1440ae630 con 0x7fc1500cfab0 2024-04-17T07:30:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.018+0000 7fc14fe00640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 0x7fc13006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.019+0000 7fc14fe00640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 0x7fc13006eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc15015d2d0 tx=0x7fc13c0079b0 comp rx=0 tx=0).ready entity=mgr.5794 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.022+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc14404d030 con 0x7fc1500cfab0 2024-04-17T07:30:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.277+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc1500b37a0 con 0x7fc1500cfab0 2024-04-17T07:30:23.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.745+0000 7fc14cc00640 1 -- 172.21.15.67:0/3766665059 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) ==== 92+0+0 (secure 0 0 0) 0x7fc14407c9e0 con 0x7fc1500cfab0 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 msgr2=0x7fc13006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 0x7fc13006eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc15015d2d0 tx=0x7fc13c0079b0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 msgr2=0x7fc1500b9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc144036040 tx=0x7fc144004a40 comp rx=0 tx=0).stop 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 shutdown_connections 2024-04-17T07:30:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc13006ca10 0x7fc13006eed0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc15015d2d0 tx=0x7fc13c0079b0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc15015ecb0 0x7fc1500b74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1500d3210 0x7fc150158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 --2- 172.21.15.67:0/3766665059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500cfab0 0x7fc1500b9ef0 secure :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc144036040 tx=0x7fc144004a40 comp rx=0 tx=0).stop 2024-04-17T07:30:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.750+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 >> 172.21.15.67:0/3766665059 conn(0x7fc15008bff0 msgr2=0x7fc1500d3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.751+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 shutdown_connections 2024-04-17T07:30:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.752+0000 7fc156600640 1 -- 172.21.15.67:0/3766665059 wait complete. 2024-04-17T07:30:23.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:30:23.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.985+0000 7fc4a9800640 1 Processor -- start 2024-04-17T07:30:23.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.985+0000 7fc4a9800640 1 -- start start 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.986+0000 7fc4a9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc4a4072d80 0x7fc4a4071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a9800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a40cea70 con 0x7fc4a4145ac0 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a9800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a40ced50 con 0x7fc4a40ce3b0 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a9800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a40ce820 con 0x7fc4a4072d80 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc4a4145ac0 0x7fc4a4072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39138/0 (socket says 172.21.15.67:39138) 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a2a00640 1 -- 172.21.15.67:0/1030287526 learned_addr learned my addr 172.21.15.67:0/1030287526 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc49a000640 1 --2- 172.21.15.67:0/1030287526 >> v2:172.21.15.67:3300/0 conn(0x7fc4a4072d80 0x7fc4a4071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a2000640 1 -- 172.21.15.67:0/1030287526 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688548036 0 0) 0x7fc4a40cea70 con 0x7fc4a4145ac0 2024-04-17T07:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.987+0000 7fc4a2000640 1 -- 172.21.15.67:0/1030287526 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc488003520 con 0x7fc4a4145ac0 2024-04-17T07:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.988+0000 7fc49a000640 1 -- 172.21.15.67:0/1030287526 >> v1:172.21.15.187:6789/0 conn(0x7fc4a40ce3b0 legacy=0x7fc4a40cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.988+0000 7fc49a000640 1 -- 172.21.15.67:0/1030287526 >> v1:172.21.15.67:6789/0 conn(0x7fc4a4145ac0 legacy=0x7fc4a4072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.988+0000 7fc49a000640 1 -- 172.21.15.67:0/1030287526 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a40cf030 con 0x7fc4a4072d80 2024-04-17T07:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.989+0000 7fc49a000640 1 --2- 172.21.15.67:0/1030287526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4072d80 0x7fc4a4071ea0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc48c0097a0 tx=0x7fc48c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea701b59227642d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.989+0000 7fc4a2000640 1 -- 172.21.15.67:0/1030287526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c034070 con 0x7fc4a4072d80 2024-04-17T07:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.989+0000 7fc4a2000640 1 -- 172.21.15.67:0/1030287526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc48c030610 con 0x7fc4a4072d80 2024-04-17T07:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.989+0000 7fc4a2000640 1 -- 172.21.15.67:0/1030287526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c0309c0 con 0x7fc4a4072d80 2024-04-17T07:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 -- 172.21.15.67:0/1030287526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4072d80 msgr2=0x7fc4a4071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 --2- 172.21.15.67:0/1030287526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4072d80 0x7fc4a4071ea0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc48c0097a0 tx=0x7fc48c02b960 comp rx=0 tx=0).stop 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 -- 172.21.15.67:0/1030287526 shutdown_connections 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 --2- 172.21.15.67:0/1030287526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4072d80 0x7fc4a4071ea0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 -- 172.21.15.67:0/1030287526 >> 172.21.15.67:0/1030287526 conn(0x7fc4a4067ee0 msgr2=0x7fc4a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 -- 172.21.15.67:0/1030287526 shutdown_connections 2024-04-17T07:30:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.990+0000 7fc4a9800640 1 -- 172.21.15.67:0/1030287526 wait complete. 2024-04-17T07:30:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.993+0000 7fc4a9800640 1 Processor -- start 2024-04-17T07:30:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.993+0000 7fc4a9800640 1 -- start start 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 0x7fc4a4139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 0x7fc4a415e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a4145ea0 con 0x7fc4a4145ac0 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a40cf5f0 con 0x7fc4a4072d80 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a9800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a40d0220 con 0x7fc4a40ce3b0 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50392/0 (socket says 172.21.15.67:50392) 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a2a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 0x7fc4a4139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc4a3400640 1 -- 172.21.15.67:0/3239472956 learned_addr learned my addr 172.21.15.67:0/3239472956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.995+0000 7fc49a000640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 0x7fc4a415e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 msgr2=0x7fc4a415e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 0x7fc4a415e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 msgr2=0x7fc4a4139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 0x7fc4a4139e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 -- 172.21.15.67:0/3239472956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a415f420 con 0x7fc4a4145ac0 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a2a00640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 0x7fc4a4139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc49a000640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 0x7fc4a415e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a3400640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c7a0 tx=0x7fc49c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c003b70 con 0x7fc4a4145ac0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc49c003d30 con 0x7fc4a4145ac0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c00aa90 con 0x7fc4a4145ac0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a415f700 con 0x7fc4a4145ac0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.996+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a414ac10 con 0x7fc4a4145ac0 2024-04-17T07:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.998+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a4068ca0 con 0x7fc4a4145ac0 2024-04-17T07:30:23.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.999+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) ==== 91159+0+0 (secure 0 0 0) 0x7fc49c00ac30 con 0x7fc4a4145ac0 2024-04-17T07:30:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:23.999+0000 7fc49b400640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 0x7fc48006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.000+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7fc49c08bd60 con 0x7fc4a4145ac0 2024-04-17T07:30:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.000+0000 7fc4a2a00640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 msgr2=0x7fc48006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.000+0000 7fc4a2a00640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 0x7fc48006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.004+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc49c059fe0 con 0x7fc4a4145ac0 2024-04-17T07:30:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.201+0000 7fc4a2a00640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 msgr2=0x7fc48006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.201+0000 7fc4a2a00640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 0x7fc48006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:24.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.209+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc4a406b960 con 0x7fc4a4145ac0 2024-04-17T07:30:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.214+0000 7fc49b400640 1 -- 172.21.15.67:0/3239472956 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7fc49c05ee20 con 0x7fc4a4145ac0 2024-04-17T07:30:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 msgr2=0x7fc48006ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 0x7fc48006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 msgr2=0x7fc4a415ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c7a0 tx=0x7fc49c00cc60 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 shutdown_connections 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4a4145ac0 0x7fc4a415ed50 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4a40ce3b0 0x7fc4a415e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7fc48006ca60 0x7fc48006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 --2- 172.21.15.67:0/3239472956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4a4072d80 0x7fc4a4139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 >> 172.21.15.67:0/3239472956 conn(0x7fc4a4067ee0 msgr2=0x7fc4a40b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 shutdown_connections 2024-04-17T07:30:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.216+0000 7fc4a9800640 1 -- 172.21.15.67:0/3239472956 wait complete. 2024-04-17T07:30:24.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:30:24.301 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.301+0000 7f1b48e13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:30:24.336 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:24.336+0000 7fe51fa93040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:30:24.355 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.355+0000 7f1b48e13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:30:24.394 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:24.394+0000 7fe51fa93040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:30:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.441+0000 7f77a3600640 1 Processor -- start 2024-04-17T07:30:24.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.442+0000 7f77a3600640 1 -- start start 2024-04-17T07:30:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f77a3600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f779c15eca0 0x7f779c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f77a3600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 0x7f779c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f77a3600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f779c163860 con 0x7f779c0cfa10 2024-04-17T07:30:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f77a3600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f779c0d03b0 con 0x7f779c15eca0 2024-04-17T07:30:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f77a3600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f779c0d1d40 con 0x7f779c0d17a0 2024-04-17T07:30:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f779be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f779c0cfa10 0x7f779c0d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39154/0 (socket says 172.21.15.67:39154) 2024-04-17T07:30:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.443+0000 7f779be00640 1 -- 172.21.15.67:0/689944327 learned_addr learned my addr 172.21.15.67:0/689944327 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f77a0c00640 1 --2- 172.21.15.67:0/689944327 >> v2:172.21.15.187:3300/0 conn(0x7f779c15eca0 0x7f779c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f77a1600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 0x7f779c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311215012 0 0) 0x7f779c163860 con 0x7f779c0cfa10 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7784003520 con 0x7f779c0cfa10 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 526915257 0 0) 0x7f7784003520 con 0x7f779c0cfa10 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 msgr2=0x7f779c0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 0x7f779c0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 >> v2:172.21.15.187:3300/0 conn(0x7f779c15eca0 msgr2=0x7f779c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 --2- 172.21.15.67:0/689944327 >> v2:172.21.15.187:3300/0 conn(0x7f779c15eca0 0x7f779c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c0d2c30 con 0x7f779c0cfa10 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.444+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7788001670 con 0x7f779c0cfa10 2024-04-17T07:30:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 >> v1:172.21.15.67:6789/0 conn(0x7f779c0cfa10 legacy=0x7f779c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 0x7f77840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7784003f80 0x7f7784004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7784004d40 0x7f77840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f779c0d1d40 con 0x7f7784004d40 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f778400b9f0 con 0x7f7784003f80 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f778400bbf0 con 0x7f779c0d17a0 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7784004d40 0x7f77840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 -- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 msgr2=0x7f77840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f779c0d17a0 0x7f77840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 -- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7784003f80 msgr2=0x7f7784004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7784003f80 0x7f7784004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.445+0000 7f77a1600640 1 -- 172.21.15.67:0/689944327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7784003520 con 0x7f7784004d40 2024-04-17T07:30:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.446+0000 7f77a1600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7784004d40 0x7f77840051c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7794009fc0 tx=0x7f7784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378d922f00bb0ade server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794039070 con 0x7f7784004d40 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f779402cc60 con 0x7f7784004d40 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f779b400640 1 -- 172.21.15.67:0/689944327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794009c10 con 0x7f7784004d40 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f77a3600640 1 -- 172.21.15.67:0/689944327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7784004d40 msgr2=0x7f77840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f77a3600640 1 --2- 172.21.15.67:0/689944327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7784004d40 0x7f77840051c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f7794009fc0 tx=0x7f7784005830 comp rx=0 tx=0).stop 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.447+0000 7f77a1600640 1 -- 172.21.15.67:0/689944327 reap_dead start 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.448+0000 7f77a3600640 1 -- 172.21.15.67:0/689944327 shutdown_connections 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.448+0000 7f77a3600640 1 -- 172.21.15.67:0/689944327 >> 172.21.15.67:0/689944327 conn(0x7f779c067600 msgr2=0x7f779c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.448+0000 7f77a3600640 1 -- 172.21.15.67:0/689944327 shutdown_connections 2024-04-17T07:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.448+0000 7f77a3600640 1 -- 172.21.15.67:0/689944327 wait complete. 2024-04-17T07:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.449+0000 7f77a3600640 1 Processor -- start 2024-04-17T07:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 -- start start 2024-04-17T07:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7784004d40 0x7f779c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 0x7f779c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f779c0d34d0 con 0x7f779c160f30 2024-04-17T07:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f779c0d31f0 con 0x7f779c15e010 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.450+0000 7f77a3600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f779c15f080 con 0x7f7784004d40 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50406/0 (socket says 172.21.15.67:50406) 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 -- 172.21.15.67:0/1452989971 learned_addr learned my addr 172.21.15.67:0/1452989971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f779be00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 0x7f779c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a0c00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7784004d40 0x7f779c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7784004d40 msgr2=0x7f779c156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7784004d40 0x7f779c156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 msgr2=0x7f779c1609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 0x7f779c1609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f77a1600640 1 -- 172.21.15.67:0/1452989971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c0c8840 con 0x7f779c160f30 2024-04-17T07:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.451+0000 7f779be00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 0x7f779c1609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f77a1600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f779402c6d0 tx=0x7f7794009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794039030 con 0x7f779c160f30 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f779403a040 con 0x7f779c160f30 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794003990 con 0x7f779c160f30 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779c0c8b20 con 0x7f779c160f30 2024-04-17T07:30:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.452+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779c0c90e0 con 0x7f779c160f30 2024-04-17T07:30:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.453+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f779c068ca0 con 0x7f779c160f30 2024-04-17T07:30:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.454+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) ==== 91159+0+0 (secure 0 0 0) 0x7f7794045020 con 0x7f779c160f30 2024-04-17T07:30:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.455+0000 7f7798c00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 0x7f777c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.455+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f77940adac0 con 0x7f779c160f30 2024-04-17T07:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.455+0000 7f77a0c00640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 msgr2=0x7f777c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.455+0000 7f77a0c00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 0x7f777c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.460+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f779407be70 con 0x7f779c160f30 2024-04-17T07:30:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.656+0000 7f77a0c00640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 msgr2=0x7f777c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.656+0000 7f77a0c00640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 0x7f777c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.662+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f779c0d2970 con 0x7f779c160f30 2024-04-17T07:30:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.667+0000 7f7798c00640 1 -- 172.21.15.67:0/1452989971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f779407b770 con 0x7f779c160f30 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 msgr2=0x7f777c06ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 0x7f777c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 msgr2=0x7f779c154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f779402c6d0 tx=0x7f7794009750 comp rx=0 tx=0).stop 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 shutdown_connections 2024-04-17T07:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f779c160f30 0x7f779c154800 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f777c06ca90 0x7f777c06ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f779c15e010 0x7f779c1609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.669+0000 7f77a3600640 1 --2- 172.21.15.67:0/1452989971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7784004d40 0x7f779c156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.670+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 >> 172.21.15.67:0/1452989971 conn(0x7f779c067600 msgr2=0x7f779c0cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.670+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 shutdown_connections 2024-04-17T07:30:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.670+0000 7f77a3600640 1 -- 172.21.15.67:0/1452989971 wait complete. 2024-04-17T07:30:24.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-17T07:30:24.805 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.805+0000 7f1b48e13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:30:24.858 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.858+0000 7f1b48e13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:30:24.871 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:24.871+0000 7fe51fa93040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.900+0000 7f7a2be00640 1 Processor -- start 2024-04-17T07:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.900+0000 7f7a2be00640 1 -- start start 2024-04-17T07:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7a2c09aa60 0x7f7a2c09aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 0x7f7a2c15f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a2c0d0350 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2c09b400 con 0x7f7a2c09aa60 2024-04-17T07:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c0b2ee0 con 0x7f7a2c15f1f0 2024-04-17T07:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7a2c0b3c80 0x7f7a2c0cdc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39160/0 (socket says 172.21.15.67:39160) 2024-04-17T07:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.902+0000 7f7a2aa00640 1 -- 172.21.15.67:0/3740424412 learned_addr learned my addr 172.21.15.67:0/3740424412 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a2b400640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 0x7f7a2c15f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a2a000640 1 --2- 172.21.15.67:0/3740424412 >> v2:172.21.15.187:3300/0 conn(0x7f7a2c09aa60 0x7f7a2c09aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624889119 0 0) 0x7f7a2c0d0350 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a10003520 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3969431821 0 0) 0x7f7a10003520 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 msgr2=0x7f7a2c15f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 0x7f7a2c15f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 >> v2:172.21.15.187:3300/0 conn(0x7f7a2c09aa60 msgr2=0x7f7a2c09aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 --2- 172.21.15.67:0/3740424412 >> v2:172.21.15.187:3300/0 conn(0x7f7a2c09aa60 0x7f7a2c09aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c0d0630 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7a1c001670 con 0x7f7a2c0b3c80 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 >> v1:172.21.15.67:6789/0 conn(0x7f7a2c0b3c80 legacy=0x7f7a2c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a100039b0 0x7f7a10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 0x7f7a10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1000b9c0 con 0x7f7a10004d40 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1000bb40 con 0x7f7a100039b0 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.903+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1000bcc0 con 0x7f7a2c15f1f0 2024-04-17T07:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2aa00640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a100039b0 0x7f7a10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 -- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 msgr2=0x7f7a10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15f1f0 0x7f7a10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 -- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a100039b0 msgr2=0x7f7a10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a100039b0 0x7f7a10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2b400640 1 -- 172.21.15.67:0/3740424412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a10003520 con 0x7f7a10004d40 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.904+0000 7f7a2aa00640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a100039b0 0x7f7a10003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:24.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.905+0000 7f7a2b400640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a100051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009fc0 tx=0x7f7a10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d19e1e1e87196 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.905+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a24039070 con 0x7f7a10004d40 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.905+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2402cc60 con 0x7f7a10004d40 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.905+0000 7f7a29600640 1 -- 172.21.15.67:0/3740424412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a24009c10 con 0x7f7a10004d40 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 -- 172.21.15.67:0/3740424412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 msgr2=0x7f7a100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 --2- 172.21.15.67:0/3740424412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a100051c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009fc0 tx=0x7f7a10005830 comp rx=0 tx=0).stop 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2b400640 1 -- 172.21.15.67:0/3740424412 reap_dead start 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 -- 172.21.15.67:0/3740424412 shutdown_connections 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 -- 172.21.15.67:0/3740424412 >> 172.21.15.67:0/3740424412 conn(0x7f7a2c0b4060 msgr2=0x7f7a2c0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 -- 172.21.15.67:0/3740424412 shutdown_connections 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.906+0000 7f7a2be00640 1 -- 172.21.15.67:0/3740424412 wait complete. 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.908+0000 7f7a2be00640 1 Processor -- start 2024-04-17T07:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.908+0000 7f7a2be00640 1 -- start start 2024-04-17T07:30:24.910 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.909+0000 7f1b48e13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a2c1f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15ae10 0x7f7a2c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a2c0b5190 0x7f7a2c15e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0d0ed0 con 0x7f7a10004d40 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0d0bf0 con 0x7f7a2c0b5190 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c0ce330 con 0x7f7a2c15ae10 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.909+0000 7f7a2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a2c1f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a2c1f86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50420/0 (socket says 172.21.15.67:50420) 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 -- 172.21.15.67:0/1473582782 learned_addr learned my addr 172.21.15.67:0/1473582782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2b400640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a2c0b5190 0x7f7a2c15e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15ae10 msgr2=0x7f7a2c15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a2c15ae10 0x7f7a2c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a2c0b5190 msgr2=0x7f7a2c15e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a2c0b5190 0x7f7a2c15e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:24.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c0c5f60 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.910+0000 7f7a2aa00640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a10004d40 0x7f7a2c1f86a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00d8b0 tx=0x7f7a1c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.911+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c019070 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.911+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c0074e0 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.911+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0049e0 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.911+0000 7f7a2be00640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c0c6200 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.911+0000 7f7a2be00640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c0c67c0 con 0x7f7a10004d40 2024-04-17T07:30:24.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.912+0000 7f7a2be00640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c0b7440 con 0x7f7a10004d40 2024-04-17T07:30:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.913+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) ==== 91159+0+0 (secure 0 0 0) 0x7f7a1c004b80 con 0x7f7a10004d40 2024-04-17T07:30:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.914+0000 7f7a23400640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.914+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5709+0+0 (secure 0 0 0) 0x7f7a1c08ba50 con 0x7f7a10004d40 2024-04-17T07:30:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.914+0000 7f7a2a000640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.914+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:24.922+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7a1c059bd0 con 0x7f7a10004d40 2024-04-17T07:30:24.932 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:24.932+0000 7fe51fa93040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:30:24.971 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:24.971+0000 7f1b48e13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:30:24.986 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:24.986+0000 7fe51fa93040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:30:25.021 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.021+0000 7f1b48e13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:30:25.051 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.050+0000 7fe51fa93040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:30:25.102 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.102+0000 7fe51fa93040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:30:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:25.115+0000 7f7a2a000640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:25.115+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:25.148+0000 7f7a2be00640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f7a2c0c9190 con 0x7f7a0c06ca90 2024-04-17T07:30:25.149 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.149+0000 7f1b48e13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:30:25.211 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.211+0000 7f1b48e13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:25.238 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.238+0000 7fe51fa93040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:30:25.262 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.262+0000 7f1b48e13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:30:25.304 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.304+0000 7fe51fa93040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:25.319 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.319+0000 7f1b48e13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:30:25.356 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.356+0000 7fe51fa93040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:30:25.370 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.370+0000 7f1b48e13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:30:25.416 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.416+0000 7fe51fa93040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:30:25.470 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.469+0000 7fe51fa93040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:30:25.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:25.516+0000 7f7a2a000640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:25.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:25.516+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:30:25.753 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.753+0000 7f1b48e13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:30:25.870 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:25.870+0000 7fe51fa93040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:30:25.911 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:25.911+0000 7f1b48e13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:30:26.019 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.019+0000 7f1b48e13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:30:26.038 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.038+0000 7fe51fa93040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:30:26.070 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.070+0000 7f1b48e13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:30:26.152 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.151+0000 7fe51fa93040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:30:26.196 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.195+0000 7f1b48e13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:26.205 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.205+0000 7fe51fa93040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:30:26.302 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.302+0000 7f1b48e13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:30:26.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:26.317+0000 7f7a2a000640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:26.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:26.317+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:30:26.337 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.337+0000 7fe51fa93040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:26.450 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.450+0000 7fe51fa93040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:30:26.478 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.477+0000 7f1b48e13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:26.527 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.527+0000 7f1b48e13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:26.633 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.632+0000 7fe51fa93040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:26.663 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:26.663+0000 7f1b48e13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:30:26.684 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.684+0000 7fe51fa93040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:26.828 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:26.827+0000 7fe51fa93040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:30:27.484 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:27.484+0000 7f1b48e13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:30:27.679 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:27.678+0000 7fe51fa93040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:30:27.788 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:27.788+0000 7f1b48e13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:30:27.846 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:27.845+0000 7f1b48e13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:30:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:27.920+0000 7f7a2a000640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3743052516 2024-04-17T07:30:27.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:27.920+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T07:30:27.975 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:27.975+0000 7f1b48e13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:30:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:27.999+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 40) ==== 90768+0+0 (secure 0 0 0) 0x7f7a1c089f70 con 0x7f7a10004d40 2024-04-17T07:30:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:27.999+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 msgr2=0x7f7a0c06ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:28.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:27.999+0000 7f7a23400640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/3743052516,v1:172.21.15.187:6833/3743052516] conn(0x7f7a0c06ca90 0x7f7a0c06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:28.004 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:28.004+0000 7fe51fa93040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:30:28.067 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:28.067+0000 7fe51fa93040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:30:28.211 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:28.211+0000 7fe51fa93040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abba00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.265 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:28.265+0000 7f1abec00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:28.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:30:28.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:30:29.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.011+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 41) ==== 90895+0+0 (secure 0 0 0) 0x7f7a1c055b40 con 0x7f7a10004d40 2024-04-17T07:30:29.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.012+0000 7f7a23400640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f7a0c074930 0x7f7a0c076d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.012+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 --> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f7a2c0c9190 con 0x7f7a0c074930 2024-04-17T07:30:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.014+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f7a0c074930 0x7f7a0c076d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.021+0000 7f7a2a000640 1 --2- 172.21.15.67:0/1473582782 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f7a0c074930 0x7f7a0c076d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a180093d0 tx=0x7f7a18004620 comp rx=0 tx=0).ready entity=mgr.6034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:29.039 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:29.041+0000 7f1ac8200640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-17T07:30:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.040+0000 7f7a23400640 1 -- 172.21.15.67:0/1473582782 <== mgr.6034 v2:172.21.15.187:6832/1013507418 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f7a2c0c9190 con 0x7f7a0c074930 2024-04-17T07:30:29.040 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-17T07:30:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:30:29.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:30:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.279+0000 7f6b7f000640 1 Processor -- start 2024-04-17T07:30:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.279+0000 7f6b7f000640 1 -- start start 2024-04-17T07:30:29.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 0x7f6b7815ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b780af400 0x7f6b78162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7f000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7815f780 con 0x7f6b7806b8c0 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6b7815fa60 con 0x7f6b780af400 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b781624a0 con 0x7f6b78159020 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b780af400 0x7f6b78162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b780af400 0x7f6b78162ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50426/0 (socket says 172.21.15.67:50426) 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.281+0000 7f6b7d200640 1 -- 172.21.15.67:0/455377062 learned_addr learned my addr 172.21.15.67:0/455377062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7dc00640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 0x7f6b7815ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7d200640 1 -- 172.21.15.67:0/455377062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 msgr2=0x7f6b7815ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7d200640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 0x7f6b7815ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7d200640 1 -- 172.21.15.67:0/455377062 >> v1:172.21.15.187:6789/0 conn(0x7f6b7806b8c0 legacy=0x7f6b780adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7d200640 1 -- 172.21.15.67:0/455377062 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b78162780 con 0x7f6b780af400 2024-04-17T07:30:29.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.282+0000 7f6b7dc00640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 0x7f6b7815ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:29.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.283+0000 7f6b7d200640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b780af400 0x7f6b78162ca0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009870 tx=0x7f6b6402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec15a157500d1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:29.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.283+0000 7f6b77e00640 1 -- 172.21.15.67:0/455377062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640317c0 con 0x7f6b780af400 2024-04-17T07:30:29.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.283+0000 7f6b77e00640 1 -- 172.21.15.67:0/455377062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b64037070 con 0x7f6b780af400 2024-04-17T07:30:29.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.283+0000 7f6b77e00640 1 -- 172.21.15.67:0/455377062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64031b80 con 0x7f6b780af400 2024-04-17T07:30:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 -- 172.21.15.67:0/455377062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b780af400 msgr2=0x7f6b78162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b780af400 0x7f6b78162ca0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009870 tx=0x7f6b6402b860 comp rx=0 tx=0).stop 2024-04-17T07:30:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 -- 172.21.15.67:0/455377062 shutdown_connections 2024-04-17T07:30:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b780af400 0x7f6b78162ca0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 --2- 172.21.15.67:0/455377062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b78159020 0x7f6b7815ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 -- 172.21.15.67:0/455377062 >> 172.21.15.67:0/455377062 conn(0x7f6b780b2190 msgr2=0x7f6b780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 -- 172.21.15.67:0/455377062 shutdown_connections 2024-04-17T07:30:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.284+0000 7f6b7f000640 1 -- 172.21.15.67:0/455377062 wait complete. 2024-04-17T07:30:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.286+0000 7f6b7f000640 1 Processor -- start 2024-04-17T07:30:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.287+0000 7f6b7f000640 1 -- start start 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780af400 0x7f6b78154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b78159020 0x7f6b78154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7815f4f0 con 0x7f6b7806b8c0 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7815f370 con 0x7f6b78159020 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b781631e0 con 0x7f6b780af400 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50434/0 (socket says 172.21.15.67:50434) 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.288+0000 7f6b7dc00640 1 -- 172.21.15.67:0/828760450 learned_addr learned my addr 172.21.15.67:0/828760450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7d200640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780af400 0x7f6b78154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7e600640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b78159020 0x7f6b78154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 -- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780af400 msgr2=0x7f6b78154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780af400 0x7f6b78154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 -- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b78159020 msgr2=0x7f6b78154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b78159020 0x7f6b78154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 -- 172.21.15.67:0/828760450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b781540d0 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.289+0000 7f6b7dc00640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00c7f0 tx=0x7f6b6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.290+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c015070 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.290+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6c007a30 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.290+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b78142e60 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.290+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c007e00 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.290+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b78143420 con 0x7f6b7806b8c0 2024-04-17T07:30:29.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.291+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b78068a10 con 0x7f6b7806b8c0 2024-04-17T07:30:29.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.292+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 41) ==== 90895+0+0 (secure 0 0 0) 0x7f6b6c018070 con 0x7f6b7806b8c0 2024-04-17T07:30:29.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.294+0000 7f6b75600640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 0x7f6b5806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:29.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.294+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f6b6c08b9a0 con 0x7f6b7806b8c0 2024-04-17T07:30:29.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.294+0000 7f6b7d200640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 0x7f6b5806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:29.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.296+0000 7f6b7d200640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 0x7f6b5806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b64036000 tx=0x7f6b6403d040 comp rx=0 tx=0).ready entity=mgr.6034 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:29.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.299+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6b6c059f20 con 0x7f6b7806b8c0 2024-04-17T07:30:29.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:29.562+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f6b7806c7b0 con 0x7f6b7806b8c0 2024-04-17T07:30:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.054+0000 7f6b75600640 1 -- 172.21.15.67:0/828760450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) ==== 93+0+0 (secure 0 0 0) 0x7f6b6c059820 con 0x7f6b7806b8c0 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.059+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 msgr2=0x7f6b5806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.059+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 0x7f6b5806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6b64036000 tx=0x7f6b6403d040 comp rx=0 tx=0).stop 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.059+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 msgr2=0x7f6b78142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.059+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00c7f0 tx=0x7f6b6c00cbf0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 shutdown_connections 2024-04-17T07:30:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f6b5806c790 0x7f6b5806ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b78159020 0x7f6b78154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780af400 0x7f6b78154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 --2- 172.21.15.67:0/828760450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7806b8c0 0x7f6b78142530 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.060+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 >> 172.21.15.67:0/828760450 conn(0x7f6b780b2190 msgr2=0x7f6b7806e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.061+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 shutdown_connections 2024-04-17T07:30:30.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.061+0000 7f6b7f000640 1 -- 172.21.15.67:0/828760450 wait complete. 2024-04-17T07:30:30.087 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:30:30.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:30:30.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:30:30.295 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:30:30.295 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:30:30.295 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:30.295 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:30.348 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:30.348 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:30.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:30.616 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:30.617+0000 7f5963c48040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:30:30.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.648+0000 7f1cf9e00640 1 Processor -- start 2024-04-17T07:30:30.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.649+0000 7f1cf9e00640 1 -- start start 2024-04-17T07:30:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 0x7f1cf40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40b6b30 0x7f1cf40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cf40cf910 0x7f1cf406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf40d49c0 con 0x7f1cf40cf910 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf40d3660 con 0x7f1cf415ec60 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.650+0000 7f1cf9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf40b6f70 con 0x7f1cf40b6b30 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cf40cf910 0x7f1cf406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cf40cf910 0x7f1cf406cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50446/0 (socket says 172.21.15.67:50446) 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 -- 172.21.15.67:0/892546776 learned_addr learned my addr 172.21.15.67:0/892546776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf8a00640 1 --2- 172.21.15.67:0/892546776 >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 0x7f1cf40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf3e00640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40b6b30 0x7f1cf40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 -- 172.21.15.67:0/892546776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40b6b30 msgr2=0x7f1cf40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40b6b30 0x7f1cf40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 -- 172.21.15.67:0/892546776 >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 msgr2=0x7f1cf40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 --2- 172.21.15.67:0/892546776 >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 0x7f1cf40b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf9400640 1 -- 172.21.15.67:0/892546776 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf40d2d30 con 0x7f1cf40cf910 2024-04-17T07:30:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.651+0000 7f1cf8a00640 1 --2- 172.21.15.67:0/892546776 >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 0x7f1cf40b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.653+0000 7f1cf9400640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40cf910 0x7f1cf406cdb0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40d1550 tx=0x7f1cec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62eee37cd778a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.653+0000 7f1cf3400640 1 -- 172.21.15.67:0/892546776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec035020 con 0x7f1cf40cf910 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.653+0000 7f1cf3400640 1 -- 172.21.15.67:0/892546776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cec031870 con 0x7f1cf40cf910 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.653+0000 7f1cf3400640 1 -- 172.21.15.67:0/892546776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec031c40 con 0x7f1cf40cf910 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 -- 172.21.15.67:0/892546776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40cf910 msgr2=0x7f1cf406cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40cf910 0x7f1cf406cdb0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40d1550 tx=0x7f1cec02b910 comp rx=0 tx=0).stop 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 -- 172.21.15.67:0/892546776 shutdown_connections 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40cf910 0x7f1cf406cdb0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/892546776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40b6b30 0x7f1cf40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/892546776 >> v2:172.21.15.187:3300/0 conn(0x7f1cf415ec60 0x7f1cf40b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 -- 172.21.15.67:0/892546776 >> 172.21.15.67:0/892546776 conn(0x7f1cf40d15f0 msgr2=0x7f1cf40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 -- 172.21.15.67:0/892546776 shutdown_connections 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.654+0000 7f1cf9e00640 1 -- 172.21.15.67:0/892546776 wait complete. 2024-04-17T07:30:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.656+0000 7f1cf9e00640 1 Processor -- start 2024-04-17T07:30:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.656+0000 7f1cf9e00640 1 -- start start 2024-04-17T07:30:30.656 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f741a23f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40cf910 0x7f1cf414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 0x7f1cf41f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf40cfda0 con 0x7f1cf40b6b30 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf40d32f0 con 0x7f1cf415ec60 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf415f040 con 0x7f1cf40cf910 2024-04-17T07:30:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50456/0 (socket says 172.21.15.67:50456) 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 -- 172.21.15.67:0/3597343575 learned_addr learned my addr 172.21.15.67:0/3597343575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9400640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 0x7f1cf41f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40cf910 msgr2=0x7f1cf414cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40cf910 0x7f1cf414cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 msgr2=0x7f1cf41f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 0x7f1cf41f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf8a00640 1 -- 172.21.15.67:0/3597343575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf41f5160 con 0x7f1cf40b6b30 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.657+0000 7f1cf9400640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 0x7f1cf41f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf8a00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4007ef0 tx=0x7f1ce400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce401a070 con 0x7f1cf40b6b30 2024-04-17T07:30:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce4003b90 con 0x7f1cf40b6b30 2024-04-17T07:30:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4004030 con 0x7f1cf40b6b30 2024-04-17T07:30:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf41f5440 con 0x7f1cf40b6b30 2024-04-17T07:30:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.658+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf41f8130 con 0x7f1cf40b6b30 2024-04-17T07:30:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.659+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf4068a10 con 0x7f1cf40b6b30 2024-04-17T07:30:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.660+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f1ce401d070 con 0x7f1cf40b6b30 2024-04-17T07:30:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.661+0000 7f1cf0c00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 0x7f1cd006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.661+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f1ce408c0b0 con 0x7f1cf40b6b30 2024-04-17T07:30:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.662+0000 7f1cf3e00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 msgr2=0x7f1cd006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.662+0000 7f1cf3e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 0x7f1cd006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.666+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1ce4010070 con 0x7f1cf40b6b30 2024-04-17T07:30:30.672 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:30.673+0000 7f5963c48040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:30:30.716 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:30.716+0000 7f741a23f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:30:30.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.863+0000 7f1cf3e00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 msgr2=0x7f1cd006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:30.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.863+0000 7f1cf3e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 0x7f1cd006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:30.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.868+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cf415d830 con 0x7f1cf40b6b30 2024-04-17T07:30:30.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.870+0000 7f1cf0c00640 1 -- 172.21.15.67:0/3597343575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+17126 (secure 0 0 0) 0x7f1ce4059b80 con 0x7f1cf40b6b30 2024-04-17T07:30:30.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:30.870 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:27.980713+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:6832/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1419379931":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/201608361":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1684018318":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/35793766":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/3592827215":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/2290658878":"2024-04-18T07:30:27.980642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:30.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 msgr2=0x7f1cd006ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:30.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 0x7f1cd006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 msgr2=0x7f1cf4161350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4007ef0 tx=0x7f1ce400d4a0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 shutdown_connections 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f1cd006ca60 0x7f1cd006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cf415ec60 0x7f1cf41f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cf40cf910 0x7f1cf414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 --2- 172.21.15.67:0/3597343575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cf40b6b30 0x7f1cf4161350 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 >> 172.21.15.67:0/3597343575 conn(0x7f1cf40d15f0 msgr2=0x7f1cf40b89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 shutdown_connections 2024-04-17T07:30:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:30.872+0000 7f1cf9e00640 1 -- 172.21.15.67:0/3597343575 wait complete. 2024-04-17T07:30:30.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:30:11.123171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:30:12.078044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:30:30.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:30:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.103+0000 7f89d9400640 1 Processor -- start 2024-04-17T07:30:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.103+0000 7f89d9400640 1 -- start start 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89d4153950 0x7f89d40b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 0x7f89d4160390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89d4153d30 con 0x7f89d4153950 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89d40b8880 con 0x7f89d40cd5d0 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d9400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89d40cfdf0 con 0x7f89d40b8420 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89d4153950 0x7f89d40b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89d4153950 0x7f89d40b7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50472/0 (socket says 172.21.15.67:50472) 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d2800640 1 -- 172.21.15.67:0/2940200546 learned_addr learned my addr 172.21.15.67:0/2940200546 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.105+0000 7f89d3200640 1 --2- 172.21.15.67:0/2940200546 >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 0x7f89d4160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d2800640 1 -- 172.21.15.67:0/2940200546 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 msgr2=0x7f89d40cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d2800640 1 --2- 172.21.15.67:0/2940200546 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d2800640 1 -- 172.21.15.67:0/2940200546 >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 msgr2=0x7f89d4160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d2800640 1 --2- 172.21.15.67:0/2940200546 >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 0x7f89d4160390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d2800640 1 -- 172.21.15.67:0/2940200546 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d40cf4c0 con 0x7f89d4153950 2024-04-17T07:30:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.106+0000 7f89d3200640 1 --2- 172.21.15.67:0/2940200546 >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 0x7f89d4160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d2800640 1 --2- 172.21.15.67:0/2940200546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d40b7ee0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f89d4161430 tx=0x7f89c402b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8675d9232c5ce31 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d1e00640 1 -- 172.21.15.67:0/2940200546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c40317f0 con 0x7f89d4153950 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d1e00640 1 -- 172.21.15.67:0/2940200546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c4037070 con 0x7f89d4153950 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d1e00640 1 -- 172.21.15.67:0/2940200546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4031bb0 con 0x7f89d4153950 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d9400640 1 -- 172.21.15.67:0/2940200546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 msgr2=0x7f89d40b7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.107+0000 7f89d9400640 1 --2- 172.21.15.67:0/2940200546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d40b7ee0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f89d4161430 tx=0x7f89c402b890 comp rx=0 tx=0).stop 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 -- 172.21.15.67:0/2940200546 shutdown_connections 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 --2- 172.21.15.67:0/2940200546 >> v2:172.21.15.187:3300/0 conn(0x7f89d40cd5d0 0x7f89d4160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 --2- 172.21.15.67:0/2940200546 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 --2- 172.21.15.67:0/2940200546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d40b7ee0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 -- 172.21.15.67:0/2940200546 >> 172.21.15.67:0/2940200546 conn(0x7f89d40b4400 msgr2=0x7f89d40b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 -- 172.21.15.67:0/2940200546 shutdown_connections 2024-04-17T07:30:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.108+0000 7f89d9400640 1 -- 172.21.15.67:0/2940200546 wait complete. 2024-04-17T07:30:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.109+0000 7f89d9400640 1 Processor -- start 2024-04-17T07:30:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.109+0000 7f89d9400640 1 -- start start 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40cd5d0 0x7f89d40b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89d40cda40 con 0x7f89d4153950 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89d40cfa80 con 0x7f89d40cd5d0 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.110+0000 7f89d9400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89d40cf120 con 0x7f89d40b8420 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50476/0 (socket says 172.21.15.67:50476) 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 -- 172.21.15.67:0/1216944780 learned_addr learned my addr 172.21.15.67:0/1216944780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89cbe00640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40cd5d0 0x7f89d40b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d2800640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 msgr2=0x7f89d40cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40cd5d0 msgr2=0x7f89d40b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40cd5d0 0x7f89d40b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 -- 172.21.15.67:0/1216944780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d414f640 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.111+0000 7f89d3200640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00ce70 tx=0x7f89cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.112+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc01a070 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.112+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc00d840 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.112+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00dc30 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.112+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d40b4f50 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.112+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d40b5510 con 0x7f89d4153950 2024-04-17T07:30:31.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.113+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4068b90 con 0x7f89d4153950 2024-04-17T07:30:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.114+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f89cc01d070 con 0x7f89d4153950 2024-04-17T07:30:31.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.115+0000 7f89caa00640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 0x7f89b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.115+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f89cc08c640 con 0x7f89d4153950 2024-04-17T07:30:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.116+0000 7f89d2800640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 msgr2=0x7f89b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:31.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.116+0000 7f89d2800640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 0x7f89b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:31.117 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.118+0000 7f5963c48040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:30:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.120+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f89cc05a800 con 0x7f89d4153950 2024-04-17T07:30:31.171 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.172+0000 7f5963c48040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:30:31.194 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.192+0000 7f741a23f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:30:31.222 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.223+0000 7f5963c48040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:30:31.248 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.248+0000 7f741a23f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:30:31.283 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.284+0000 7f5963c48040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:30:31.301 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.301+0000 7f741a23f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:30:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.317+0000 7f89d2800640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 msgr2=0x7f89b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.317+0000 7f89d2800640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 0x7f89b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.321+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f89d4153320 con 0x7f89d4153950 2024-04-17T07:30:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.322+0000 7f89caa00640 1 -- 172.21.15.67:0/1216944780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f89cc05a100 con 0x7f89d4153950 2024-04-17T07:30:31.322 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 msgr2=0x7f89b006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 0x7f89b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 msgr2=0x7f89d414f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00ce70 tx=0x7f89cc00a710 comp rx=0 tx=0).stop 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 shutdown_connections 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d4153950 0x7f89d414f100 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f89b006ca10 0x7f89b006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40cd5d0 0x7f89d40b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 --2- 172.21.15.67:0/1216944780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b8420 0x7f89d40cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 >> 172.21.15.67:0/1216944780 conn(0x7f89d40b4400 msgr2=0x7f89d406c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 shutdown_connections 2024-04-17T07:30:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.325+0000 7f89d9400640 1 -- 172.21.15.67:0/1216944780 wait complete. 2024-04-17T07:30:31.332 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.333+0000 7f5963c48040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:30:31.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:30:31.369 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.369+0000 7f741a23f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:30:31.422 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.422+0000 7f741a23f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:30:31.461 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.462+0000 7f5963c48040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:30:31.523 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.524+0000 7f5963c48040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:31.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.551+0000 7f5144600640 1 Processor -- start 2024-04-17T07:30:31.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.552+0000 7f5144600640 1 -- start start 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c0cfa10 0x7f513c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f513c0d1090 0x7f513c0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f513c0d03b0 con 0x7f513c0cfa10 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f513c163860 con 0x7f513c0d1090 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.553+0000 7f5144600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0d1d40 con 0x7f513c15eca0 2024-04-17T07:30:31.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c0cfa10 0x7f513c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c0cfa10 0x7f513c0cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50484/0 (socket says 172.21.15.67:50484) 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 -- 172.21.15.67:0/3198094208 learned_addr learned my addr 172.21.15.67:0/3198094208 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5142600640 1 --2- 172.21.15.67:0/3198094208 >> v2:172.21.15.187:3300/0 conn(0x7f513c0d1090 0x7f513c0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141c00640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 -- 172.21.15.67:0/3198094208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 msgr2=0x7f513c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 -- 172.21.15.67:0/3198094208 >> v2:172.21.15.187:3300/0 conn(0x7f513c0d1090 msgr2=0x7f513c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 --2- 172.21.15.67:0/3198094208 >> v2:172.21.15.187:3300/0 conn(0x7f513c0d1090 0x7f513c0d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.554+0000 7f5141200640 1 -- 172.21.15.67:0/3198094208 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0d2c30 con 0x7f513c0cfa10 2024-04-17T07:30:31.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.555+0000 7f5141200640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c0cfe70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b43e0 tx=0x7f512802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78868c81b100f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:31.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.555+0000 7f513be00640 1 -- 172.21.15.67:0/3198094208 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128035020 con 0x7f513c0cfa10 2024-04-17T07:30:31.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f513be00640 1 -- 172.21.15.67:0/3198094208 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5128031870 con 0x7f513c0cfa10 2024-04-17T07:30:31.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f513be00640 1 -- 172.21.15.67:0/3198094208 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128031c40 con 0x7f513c0cfa10 2024-04-17T07:30:31.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 -- 172.21.15.67:0/3198094208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 msgr2=0x7f513c0cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c0cfe70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f513c0b43e0 tx=0x7f512802b910 comp rx=0 tx=0).stop 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 -- 172.21.15.67:0/3198094208 shutdown_connections 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 --2- 172.21.15.67:0/3198094208 >> v2:172.21.15.187:3300/0 conn(0x7f513c0d1090 0x7f513c0d1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c0cfe70 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 --2- 172.21.15.67:0/3198094208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 -- 172.21.15.67:0/3198094208 >> 172.21.15.67:0/3198094208 conn(0x7f513c067600 msgr2=0x7f513c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 -- 172.21.15.67:0/3198094208 shutdown_connections 2024-04-17T07:30:31.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.556+0000 7f5144600640 1 -- 172.21.15.67:0/3198094208 wait complete. 2024-04-17T07:30:31.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.558+0000 7f5144600640 1 Processor -- start 2024-04-17T07:30:31.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.558+0000 7f5144600640 1 -- start start 2024-04-17T07:30:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513c0d1090 0x7f513c15fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0d34d0 con 0x7f513c0cfa10 2024-04-17T07:30:31.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0d31f0 con 0x7f513c0d1090 2024-04-17T07:30:31.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5144600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0d1a60 con 0x7f513c15eca0 2024-04-17T07:30:31.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5141c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5141c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50500/0 (socket says 172.21.15.67:50500) 2024-04-17T07:30:31.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.559+0000 7f5141c00640 1 -- 172.21.15.67:0/2960619201 learned_addr learned my addr 172.21.15.67:0/2960619201 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141200640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513c0d1090 0x7f513c15fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 msgr2=0x7f513c0b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513c0d1090 msgr2=0x7f513c15fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513c0d1090 0x7f513c15fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 -- 172.21.15.67:0/2960619201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0caed0 con 0x7f513c0cfa10 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5141c00640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f513000d800 tx=0x7f513000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000b6c0 con 0x7f513c0cfa10 2024-04-17T07:30:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5130019070 con 0x7f513c0cfa10 2024-04-17T07:30:31.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c15f360 con 0x7f513c0cfa10 2024-04-17T07:30:31.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.560+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000ba80 con 0x7f513c0cfa10 2024-04-17T07:30:31.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.561+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c160380 con 0x7f513c0cfa10 2024-04-17T07:30:31.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.562+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f513c068ca0 con 0x7f513c0cfa10 2024-04-17T07:30:31.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.563+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f513000bc20 con 0x7f513c0cfa10 2024-04-17T07:30:31.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.564+0000 7f5139600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 0x7f511c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.564+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f513008c270 con 0x7f513c0cfa10 2024-04-17T07:30:31.564 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.564+0000 7f741a23f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:30:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.564+0000 7f5141200640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 msgr2=0x7f511c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.564+0000 7f5141200640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 0x7f511c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:31.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.568+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f513005a420 con 0x7f513c0cfa10 2024-04-17T07:30:31.576 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.577+0000 7f5963c48040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:30:31.631 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.631+0000 7f741a23f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:31.632 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.633+0000 7f5963c48040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:30:31.683 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:31.684+0000 7f5963c48040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:30:31.684 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.685+0000 7f741a23f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:30:31.745 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.745+0000 7f741a23f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:30:31.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.762+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f513c0d2970 con 0x7f513c0cfa10 2024-04-17T07:30:31.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.763+0000 7f5139600640 1 -- 172.21.15.67:0/2960619201 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f5130059d20 con 0x7f513c0cfa10 2024-04-17T07:30:31.763 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5141200640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 msgr2=0x7f511c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 msgr2=0x7f511c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5141200640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 0x7f511c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 0x7f511c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 msgr2=0x7f513c14ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f513000d800 tx=0x7f513000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 shutdown_connections 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f513c15eca0 0x7f513c0b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f511c06ca40 0x7f511c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f513c0d1090 0x7f513c15fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 --2- 172.21.15.67:0/2960619201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0cfa10 0x7f513c14ae20 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.765+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 >> 172.21.15.67:0/2960619201 conn(0x7f513c067600 msgr2=0x7f513c0d4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:31.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.766+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 shutdown_connections 2024-04-17T07:30:31.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:31.766+0000 7f5144600640 1 -- 172.21.15.67:0/2960619201 wait complete. 2024-04-17T07:30:31.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:30:31.801 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:31.801+0000 7f741a23f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:30:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.009+0000 7f47f2c00640 1 Processor -- start 2024-04-17T07:30:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.010+0000 7f47f2c00640 1 -- start start 2024-04-17T07:30:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 0x7f47ec09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 0x7f47ec0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47ec0cdf90 0x7f47ec165a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f47ec09beb0 con 0x7f47ec0cdf90 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f47ec0d0c90 con 0x7f47ec160950 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47ec0d0ee0 con 0x7f47ec160d30 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.011+0000 7f47f2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47ec0cdf90 0x7f47ec165a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47ec0cdf90 0x7f47ec165a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50508/0 (socket says 172.21.15.67:50508) 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 -- 172.21.15.67:0/3292238952 learned_addr learned my addr 172.21.15.67:0/3292238952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f1800640 1 --2- 172.21.15.67:0/3292238952 >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 0x7f47ec09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f0e00640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 0x7f47ec0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 -- 172.21.15.67:0/3292238952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 msgr2=0x7f47ec0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 0x7f47ec0cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 -- 172.21.15.67:0/3292238952 >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 msgr2=0x7f47ec09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 --2- 172.21.15.67:0/3292238952 >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 0x7f47ec09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f2200640 1 -- 172.21.15.67:0/3292238952 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec0d11c0 con 0x7f47ec0cdf90 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f0e00640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 0x7f47ec0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.012+0000 7f47f1800640 1 --2- 172.21.15.67:0/3292238952 >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 0x7f47ec09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.013+0000 7f47f2200640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec0cdf90 0x7f47ec165a00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06c510 tx=0x7f47e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913efde33822ed25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.013+0000 7f47ebe00640 1 -- 172.21.15.67:0/3292238952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e402ca80 con 0x7f47ec0cdf90 2024-04-17T07:30:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.013+0000 7f47ebe00640 1 -- 172.21.15.67:0/3292238952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e402cc20 con 0x7f47ec0cdf90 2024-04-17T07:30:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.013+0000 7f47ebe00640 1 -- 172.21.15.67:0/3292238952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40343f0 con 0x7f47ec0cdf90 2024-04-17T07:30:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 -- 172.21.15.67:0/3292238952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec0cdf90 msgr2=0x7f47ec165a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec0cdf90 0x7f47ec165a00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06c510 tx=0x7f47e402bee0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 -- 172.21.15.67:0/3292238952 shutdown_connections 2024-04-17T07:30:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec0cdf90 0x7f47ec165a00 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 --2- 172.21.15.67:0/3292238952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec160d30 0x7f47ec0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 --2- 172.21.15.67:0/3292238952 >> v2:172.21.15.187:3300/0 conn(0x7f47ec160950 0x7f47ec09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 -- 172.21.15.67:0/3292238952 >> 172.21.15.67:0/3292238952 conn(0x7f47ec0b3f90 msgr2=0x7f47ec0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 -- 172.21.15.67:0/3292238952 shutdown_connections 2024-04-17T07:30:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.014+0000 7f47f2c00640 1 -- 172.21.15.67:0/3292238952 wait complete. 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.016+0000 7f47f2c00640 1 Processor -- start 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.016+0000 7f47f2c00640 1 -- start start 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 0x7f47ec153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 0x7f47ec15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec0ce400 con 0x7f47ec160950 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47ec0cd530 con 0x7f47ec160d30 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f2c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47ec0d14a0 con 0x7f47ec0cdf90 2024-04-17T07:30:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.017+0000 7f47f0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50518/0 (socket says 172.21.15.67:50518) 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 -- 172.21.15.67:0/1149691016 learned_addr learned my addr 172.21.15.67:0/1149691016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f2200640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 0x7f47ec15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f1800640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 0x7f47ec153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 msgr2=0x7f47ec153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 0x7f47ec153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 msgr2=0x7f47ec15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 0x7f47ec15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 -- 172.21.15.67:0/1149691016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ec15e010 con 0x7f47ec160950 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f2200640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 0x7f47ec15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f1800640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 0x7f47ec153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.018+0000 7f47f0e00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f47d800c820 tx=0x7f47d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.019+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8015070 con 0x7f47ec160950 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.019+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d8007a30 con 0x7f47ec160950 2024-04-17T07:30:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.019+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8007e00 con 0x7f47ec160950 2024-04-17T07:30:32.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.019+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ec15e2f0 con 0x7f47ec160950 2024-04-17T07:30:32.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.019+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ec15e8b0 con 0x7f47ec160950 2024-04-17T07:30:32.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.020+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47ec0755d0 con 0x7f47ec160950 2024-04-17T07:30:32.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.021+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f47d8018070 con 0x7f47ec160950 2024-04-17T07:30:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.022+0000 7f47e9600640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 0x7f47cc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.022+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f47d808bf50 con 0x7f47ec160950 2024-04-17T07:30:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.022+0000 7f47f1800640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 msgr2=0x7f47cc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.022+0000 7f47f1800640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 0x7f47cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:32.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.026+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f47d8059f60 con 0x7f47ec160950 2024-04-17T07:30:32.071 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.072+0000 7f5963c48040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:30:32.217 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.217+0000 7f741a23f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:30:32.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.223+0000 7f47f1800640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 msgr2=0x7f47cc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:32.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.223+0000 7f47f1800640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 0x7f47cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.230+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47ec0b4b80 con 0x7f47ec160950 2024-04-17T07:30:32.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.231+0000 7f47e9600640 1 -- 172.21.15.67:0/1149691016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7f47d8059860 con 0x7f47ec160950 2024-04-17T07:30:32.231 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:30:32.232 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.233+0000 7f5963c48040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:30:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.233+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 msgr2=0x7f47cc06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.233+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 0x7f47cc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.233+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 msgr2=0x7f47ec15f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.233+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f47d800c820 tx=0x7f47d800cc20 comp rx=0 tx=0).stop 2024-04-17T07:30:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 shutdown_connections 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f47cc06ca10 0x7f47cc06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47ec160d30 0x7f47ec15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47ec160950 0x7f47ec15f5d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 --2- 172.21.15.67:0/1149691016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47ec0cdf90 0x7f47ec153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 >> 172.21.15.67:0/1149691016 conn(0x7f47ec0b3f90 msgr2=0x7f47ec06da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 shutdown_connections 2024-04-17T07:30:32.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.234+0000 7f47f2c00640 1 -- 172.21.15.67:0/1149691016 wait complete. 2024-04-17T07:30:32.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:30:32.341 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.342+0000 7f5963c48040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:30:32.392 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.393+0000 7f5963c48040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:30:32.392 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.392+0000 7f741a23f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:30:32.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.456+0000 7f21cae00640 1 Processor -- start 2024-04-17T07:30:32.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.457+0000 7f21cae00640 1 -- start start 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c409aa60 0x7f21c409ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c40d0280 0x7f21c40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21c41605a0 0x7f21c4160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21c415d350 con 0x7f21c40d0280 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f21c40d0c20 con 0x7f21c41605a0 2024-04-17T07:30:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.458+0000 7f21cae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21c40cd630 con 0x7f21c409aa60 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c40d0280 0x7f21c40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c40d0280 0x7f21c40d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50534/0 (socket says 172.21.15.67:50534) 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 -- 172.21.15.67:0/1835109929 learned_addr learned my addr 172.21.15.67:0/1835109929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21c8e00640 1 --2- 172.21.15.67:0/1835109929 >> v2:172.21.15.187:3300/0 conn(0x7f21c41605a0 0x7f21c4160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21c3e00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c409aa60 0x7f21c409ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 -- 172.21.15.67:0/1835109929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c409aa60 msgr2=0x7f21c409ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c409aa60 0x7f21c409ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 -- 172.21.15.67:0/1835109929 >> v2:172.21.15.187:3300/0 conn(0x7f21c41605a0 msgr2=0x7f21c4160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 --2- 172.21.15.67:0/1835109929 >> v2:172.21.15.187:3300/0 conn(0x7f21c41605a0 0x7f21c4160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.459+0000 7f21bbe00640 1 -- 172.21.15.67:0/1835109929 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c409b380 con 0x7f21c40d0280 2024-04-17T07:30:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.461+0000 7f21bbe00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c40d0280 0x7f21c40d06e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f21c4073ed0 tx=0x7f21ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c71a33f9f77b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.461+0000 7f21c3400640 1 -- 172.21.15.67:0/1835109929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac035020 con 0x7f21c40d0280 2024-04-17T07:30:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.461+0000 7f21c3400640 1 -- 172.21.15.67:0/1835109929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ac031640 con 0x7f21c40d0280 2024-04-17T07:30:32.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.461+0000 7f21c3400640 1 -- 172.21.15.67:0/1835109929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac02d040 con 0x7f21c40d0280 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 -- 172.21.15.67:0/1835109929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c40d0280 msgr2=0x7f21c40d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c40d0280 0x7f21c40d06e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f21c4073ed0 tx=0x7f21ac02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 -- 172.21.15.67:0/1835109929 shutdown_connections 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 --2- 172.21.15.67:0/1835109929 >> v2:172.21.15.187:3300/0 conn(0x7f21c41605a0 0x7f21c4160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c40d0280 0x7f21c40d06e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 --2- 172.21.15.67:0/1835109929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c409aa60 0x7f21c409ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 -- 172.21.15.67:0/1835109929 >> 172.21.15.67:0/1835109929 conn(0x7f21c40b41a0 msgr2=0x7f21c40b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:32.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 -- 172.21.15.67:0/1835109929 shutdown_connections 2024-04-17T07:30:32.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.462+0000 7f21cae00640 1 -- 172.21.15.67:0/1835109929 wait complete. 2024-04-17T07:30:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.464+0000 7f21cae00640 1 Processor -- start 2024-04-17T07:30:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.464+0000 7f21cae00640 1 -- start start 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21c40d0280 0x7f21c415f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c41605a0 0x7f21c40c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40d1420 con 0x7f21c409aa60 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f21c40d1140 con 0x7f21c40d0280 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21cae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21c40d1700 con 0x7f21c41605a0 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50536/0 (socket says 172.21.15.67:50536) 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.465+0000 7f21c3e00640 1 -- 172.21.15.67:0/3407788211 learned_addr learned my addr 172.21.15.67:0/3407788211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21bbe00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21c40d0280 0x7f21c415f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c8e00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c41605a0 0x7f21c40c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c41605a0 msgr2=0x7f21c40c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c41605a0 0x7f21c40c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21c40d0280 msgr2=0x7f21c415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21c40d0280 0x7f21c415f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 -- 172.21.15.67:0/3407788211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c4156240 con 0x7f21c409aa60 2024-04-17T07:30:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c3e00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f21b400d800 tx=0x7f21b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4018020 con 0x7f21c409aa60 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21b40096a0 con 0x7f21c409aa60 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.466+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4009a70 con 0x7f21c409aa60 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.467+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c41572a0 con 0x7f21c409aa60 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.467+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c41567c0 con 0x7f21c409aa60 2024-04-17T07:30:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.468+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c4068ca0 con 0x7f21c409aa60 2024-04-17T07:30:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.469+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f21b401b020 con 0x7f21c409aa60 2024-04-17T07:30:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.470+0000 7f21c0c00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 0x7f21a006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.470+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f21b408c140 con 0x7f21c409aa60 2024-04-17T07:30:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.470+0000 7f21bbe00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 msgr2=0x7f21a006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.470+0000 7f21bbe00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 0x7f21a006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.474+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f21b4022030 con 0x7f21c409aa60 2024-04-17T07:30:32.511 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.511+0000 7f741a23f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:30:32.519 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.520+0000 7f5963c48040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:32.565 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.565+0000 7f741a23f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:30:32.626 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.627+0000 7f5963c48040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:30:32.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.671+0000 7f21bbe00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 msgr2=0x7f21a006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:32.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.671+0000 7f21bbe00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 0x7f21a006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:32.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.688+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f21c40b5040 con 0x7f21c409aa60 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.689+0000 7f21c0c00640 1 -- 172.21.15.67:0/3407788211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v99) ==== 80+0+475 (secure 0 0 0) 0x7f21b405a260 con 0x7f21c409aa60 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/3743052516 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/3743052516 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1419379931 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/201608361 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1684018318 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/35793766 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.689 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3592827215 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.690 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2290658878 2024-04-18T07:30:27.980642+0000 2024-04-17T07:30:32.690 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:30:32.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 msgr2=0x7f21a006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 0x7f21a006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 msgr2=0x7f21c415f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f21b400d800 tx=0x7f21b400dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 shutdown_connections 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21c41605a0 0x7f21c40c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f21a006ca10 0x7f21a006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21c40d0280 0x7f21c415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 --2- 172.21.15.67:0/3407788211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c409aa60 0x7f21c415f1a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.691+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 >> 172.21.15.67:0/3407788211 conn(0x7f21c40b41a0 msgr2=0x7f21c40717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.692+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 shutdown_connections 2024-04-17T07:30:32.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:32.692+0000 7f21cae00640 1 -- 172.21.15.67:0/3407788211 wait complete. 2024-04-17T07:30:32.701 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.702+0000 7f741a23f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:32.709 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:30:32.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:32.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:30:32.799 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:30:32.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:30:32.800 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:30:32.803 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.803+0000 7f5963c48040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:32.820 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:32.820+0000 7f741a23f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:30:32.844 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:30:32.844 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:30:32.845 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:32.854 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.855+0000 7f5963c48040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:32.897 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:30:32.897 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:30:32.897 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:32.897 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:32.953 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:30:32.953 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:30:32.953 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:30:32.953 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:32.998 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:32.999+0000 7f5963c48040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:30:33.004 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:30:33.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:33.012 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:33.009+0000 7f741a23f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:30:33.063 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:33.063+0000 7f741a23f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:30:33.211 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:33.210+0000 7f741a23f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:30:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.219+0000 7fcc9d000640 1 Processor -- start 2024-04-17T07:30:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.219+0000 7fcc9d000640 1 -- start start 2024-04-17T07:30:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc9d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 0x7fcc980b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc9d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc980cfe20 con 0x7fcc980d12a0 2024-04-17T07:30:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc9d000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc980d0100 con 0x7fcc980cf8f0 2024-04-17T07:30:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc9d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc980cf640 con 0x7fcc980b9f00 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc97e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc980d12a0 0x7fcc980d3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39168/0 (socket says 172.21.15.67:39168) 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc97e00640 1 -- 172.21.15.67:0/288919058 learned_addr learned my addr 172.21.15.67:0/288919058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.221+0000 7fcc97400640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 0x7fcc980b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817316960 0 0) 0x7fcc980cfe20 con 0x7fcc980d12a0 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc7c003520 con 0x7fcc980d12a0 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1888208892 0 0) 0x7fcc7c003520 con 0x7fcc980d12a0 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 msgr2=0x7fcc980b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 0x7fcc980b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 >> v1:172.21.15.187:6789/0 conn(0x7fcc980cf8f0 legacy=0x7fcc981634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc980d3140 con 0x7fcc980d12a0 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcc900019f0 con 0x7fcc980d12a0 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 >> v1:172.21.15.67:6789/0 conn(0x7fcc980d12a0 legacy=0x7fcc980d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc7c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c004800 0x7fcc7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 0x7fcc7c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc980cf640 con 0x7fcc7c003a50 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c00bb80 con 0x7fcc7c004800 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7c00bd80 con 0x7fcc980b9f00 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.222+0000 7fcc97400640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc7c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc96a00640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c004800 0x7fcc7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc97400640 1 -- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 msgr2=0x7fcc7c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc97400640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980b9f00 0x7fcc7c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc97400640 1 -- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c004800 msgr2=0x7fcc7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc97400640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c004800 0x7fcc7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc97400640 1 -- 172.21.15.67:0/288919058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c003520 con 0x7fcc7c003a50 2024-04-17T07:30:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.223+0000 7fcc96a00640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc7c004800 0x7fcc7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.224+0000 7fcc97400640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc7c003e70 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009ac0 tx=0x7fcc7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f547cf9aa622b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.224+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88039070 con 0x7fcc7c003a50 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.224+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc8802c710 con 0x7fcc7c003a50 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.224+0000 7fcc96000640 1 -- 172.21.15.67:0/288919058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88009690 con 0x7fcc7c003a50 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 -- 172.21.15.67:0/288919058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 msgr2=0x7fcc7c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 --2- 172.21.15.67:0/288919058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc7c003e70 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009ac0 tx=0x7fcc7c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc97e00640 1 -- 172.21.15.67:0/288919058 reap_dead start 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 -- 172.21.15.67:0/288919058 shutdown_connections 2024-04-17T07:30:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 -- 172.21.15.67:0/288919058 >> 172.21.15.67:0/288919058 conn(0x7fcc9806cb60 msgr2=0x7fcc9806cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 -- 172.21.15.67:0/288919058 shutdown_connections 2024-04-17T07:30:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.225+0000 7fcc9d000640 1 -- 172.21.15.67:0/288919058 wait complete. 2024-04-17T07:30:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.227+0000 7fcc9d000640 1 Processor -- start 2024-04-17T07:30:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.227+0000 7fcc9d000640 1 -- start start 2024-04-17T07:30:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980cd900 0x7fcc980ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc980cdd80 0x7fcc98155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc980d03e0 con 0x7fcc7c003a50 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcc980ba2e0 con 0x7fcc980cdd80 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc9d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc980d0560 con 0x7fcc980cd900 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50556/0 (socket says 172.21.15.67:50556) 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc97400640 1 -- 172.21.15.67:0/2904089972 learned_addr learned my addr 172.21.15.67:0/2904089972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc96a00640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980cd900 0x7fcc980ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.228+0000 7fcc97400640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980cd900 msgr2=0x7fcc980ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc97400640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980cd900 0x7fcc980ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc97400640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc980cdd80 msgr2=0x7fcc98155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc97400640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc980cdd80 0x7fcc98155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc97400640 1 -- 172.21.15.67:0/2904089972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc980abcf0 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc97400640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fcc88037040 tx=0x7fcc88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88042020 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc8803a040 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88039070 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc980abfd0 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.229+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc980ac590 con 0x7fcc7c003a50 2024-04-17T07:30:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.231+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc980755d0 con 0x7fcc7c003a50 2024-04-17T07:30:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.232+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7fcc88045020 con 0x7fcc7c003a50 2024-04-17T07:30:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.232+0000 7fcc8f400640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 0x7fcc7806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.233+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7fcc880ad270 con 0x7fcc7c003a50 2024-04-17T07:30:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.233+0000 7fcc96a00640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 msgr2=0x7fcc7806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.233+0000 7fcc96a00640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 0x7fcc7806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:33.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.237+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcc8804c070 con 0x7fcc7c003a50 2024-04-17T07:30:33.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.434+0000 7fcc96a00640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 msgr2=0x7fcc7806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:33.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.434+0000 7fcc96a00640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 0x7fcc7806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:33.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.479+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc980d2a20 con 0x7fcc7c003a50 2024-04-17T07:30:33.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.480+0000 7fcc8f400640 1 -- 172.21.15.67:0/2904089972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7fcc8807b430 con 0x7fcc7c003a50 2024-04-17T07:30:33.480 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:33.481 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 msgr2=0x7fcc7806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 0x7fcc7806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 msgr2=0x7fcc980acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fcc88037040 tx=0x7fcc88004730 comp rx=0 tx=0).stop 2024-04-17T07:30:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 shutdown_connections 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7fcc7806ca10 0x7fcc7806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc980cdd80 0x7fcc98155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc980cd900 0x7fcc980ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 --2- 172.21.15.67:0/2904089972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c003a50 0x7fcc980acd90 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 >> 172.21.15.67:0/2904089972 conn(0x7fcc9806cb60 msgr2=0x7fcc980b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 shutdown_connections 2024-04-17T07:30:33.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.483+0000 7fcc9d000640 1 -- 172.21.15.67:0/2904089972 wait complete. 2024-04-17T07:30:33.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:33.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.713+0000 7f916ca00640 1 Processor -- start 2024-04-17T07:30:33.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.713+0000 7f916ca00640 1 -- start start 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f916ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f916815ec60 0x7f91680b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f916ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91680cfae0 0x7f91680b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f916ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91680d2d30 con 0x7f91680b6b30 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f916ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91680d49c0 con 0x7f91680cfae0 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f916ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f91680d3660 con 0x7f916815ec60 2024-04-17T07:30:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f9167200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91680cfae0 0x7f91680b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f9167200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91680cfae0 0x7f91680b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50538/0 (socket says 172.21.15.67:50538) 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.714+0000 7f9167200640 1 -- 172.21.15.67:0/325771882 learned_addr learned my addr 172.21.15.67:0/325771882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.715+0000 7f9167200640 1 -- 172.21.15.67:0/325771882 >> v2:172.21.15.187:3300/0 conn(0x7f916815ec60 msgr2=0x7f91680b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.715+0000 7f9167200640 1 --2- 172.21.15.67:0/325771882 >> v2:172.21.15.187:3300/0 conn(0x7f916815ec60 0x7f91680b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.715+0000 7f9167200640 1 -- 172.21.15.67:0/325771882 >> v1:172.21.15.187:6789/0 conn(0x7f91680b6b30 legacy=0x7f91680cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:33.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.715+0000 7f9167200640 1 -- 172.21.15.67:0/325771882 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91680d3010 con 0x7f91680cfae0 2024-04-17T07:30:33.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.716+0000 7f9167200640 1 --2- 172.21.15.67:0/325771882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b7b70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f91680d1550 tx=0x7f916002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48ddcd7c9082b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:33.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.716+0000 7f9165400640 1 -- 172.21.15.67:0/325771882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160035020 con 0x7f91680cfae0 2024-04-17T07:30:33.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.716+0000 7f9165400640 1 -- 172.21.15.67:0/325771882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9160031870 con 0x7f91680cfae0 2024-04-17T07:30:33.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.716+0000 7f9165400640 1 -- 172.21.15.67:0/325771882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160031c40 con 0x7f91680cfae0 2024-04-17T07:30:33.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 -- 172.21.15.67:0/325771882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 msgr2=0x7f91680b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 --2- 172.21.15.67:0/325771882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b7b70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f91680d1550 tx=0x7f916002b910 comp rx=0 tx=0).stop 2024-04-17T07:30:33.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 -- 172.21.15.67:0/325771882 shutdown_connections 2024-04-17T07:30:33.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 --2- 172.21.15.67:0/325771882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b7b70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 --2- 172.21.15.67:0/325771882 >> v2:172.21.15.187:3300/0 conn(0x7f916815ec60 0x7f91680b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 -- 172.21.15.67:0/325771882 >> 172.21.15.67:0/325771882 conn(0x7f91680d15f0 msgr2=0x7f91680d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:33.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 -- 172.21.15.67:0/325771882 shutdown_connections 2024-04-17T07:30:33.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.717+0000 7f916ca00640 1 -- 172.21.15.67:0/325771882 wait complete. 2024-04-17T07:30:33.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.719+0000 7f916ca00640 1 Processor -- start 2024-04-17T07:30:33.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.719+0000 7f916ca00640 1 -- start start 2024-04-17T07:30:33.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91680b6b30 0x7f916815d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f916815ec60 0x7f91680b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f916815f040 con 0x7f91680cfae0 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f91680b6f70 con 0x7f916815ec60 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f916ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91680d00c0 con 0x7f91680b6b30 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f9165e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f9165e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50550/0 (socket says 172.21.15.67:50550) 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f9165e00640 1 -- 172.21.15.67:0/3422823475 learned_addr learned my addr 172.21.15.67:0/3422823475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f9167200640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f916815ec60 0x7f91680b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.720+0000 7f9166800640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91680b6b30 0x7f916815d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91680b6b30 msgr2=0x7f916815d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91680b6b30 0x7f916815d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f916815ec60 msgr2=0x7f91680b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f916815ec60 0x7f91680b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 -- 172.21.15.67:0/3422823475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681f5910 con 0x7f91680cfae0 2024-04-17T07:30:33.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f9165e00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f915000c410 tx=0x7f915000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150017070 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91500035f0 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91681f5bf0 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91500039e0 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.721+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916815d5b0 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.722+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168068a10 con 0x7f91680cfae0 2024-04-17T07:30:33.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.723+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f915001a070 con 0x7f91680cfae0 2024-04-17T07:30:33.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.724+0000 7f915ea00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 0x7f914406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:33.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.724+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f915008b3e0 con 0x7f91680cfae0 2024-04-17T07:30:33.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.725+0000 7f9166800640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 msgr2=0x7f914406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:33.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.725+0000 7f9166800640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 0x7f914406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.729+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9150059500 con 0x7f91680cfae0 2024-04-17T07:30:33.814 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:33.815+0000 7f5963c48040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:30:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.926+0000 7f9166800640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 msgr2=0x7f914406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:33.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.926+0000 7f9166800640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 0x7f914406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:30:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.937+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9168161730 con 0x7f91680cfae0 2024-04-17T07:30:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.940+0000 7f915ea00640 1 -- 172.21.15.67:0/3422823475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+17126 (secure 0 0 0) 0x7f9150058e00 con 0x7f91680cfae0 2024-04-17T07:30:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:33.940 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:27.980713+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:6832/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1419379931":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/201608361":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1684018318":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/35793766":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/3592827215":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/2290658878":"2024-04-18T07:30:27.980642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 msgr2=0x7f914406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 0x7f914406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 msgr2=0x7f91680b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f915000c410 tx=0x7f915000c8d0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 shutdown_connections 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f914406ca10 0x7f914406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f916815ec60 0x7f91680b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91680cfae0 0x7f91680b5cb0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 --2- 172.21.15.67:0/3422823475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91680b6b30 0x7f916815d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.942+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 >> 172.21.15.67:0/3422823475 conn(0x7f91680d15f0 msgr2=0x7f9168074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.943+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 shutdown_connections 2024-04-17T07:30:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:33.943+0000 7f916ca00640 1 -- 172.21.15.67:0/3422823475 wait complete. 2024-04-17T07:30:33.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:30:11.123171+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:30:12.078044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:30:33.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:30:34.079 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:34.078+0000 7f741a23f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:30:34.119 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.120+0000 7f5963c48040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:30:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.169+0000 7f0e57200640 1 Processor -- start 2024-04-17T07:30:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.169+0000 7f0e57200640 1 -- start start 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e5815ec20 0x7f0e580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e580b3b50 0x7f0e5806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e5806cf80 0x7f0e580cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0e580d4f70 con 0x7f0e580b3b50 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0e580d5250 con 0x7f0e5806cf80 2024-04-17T07:30:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e580cf4d0 con 0x7f0e5815ec20 2024-04-17T07:30:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.171+0000 7f0e55400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e580b3b50 0x7f0e5806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e580b3b50 0x7f0e5806ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50560/0 (socket says 172.21.15.67:50560) 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 -- 172.21.15.67:0/1748178129 learned_addr learned my addr 172.21.15.67:0/1748178129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e56800640 1 --2- 172.21.15.67:0/1748178129 >> v2:172.21.15.187:3300/0 conn(0x7f0e5806cf80 0x7f0e580cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55e00640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e5815ec20 0x7f0e580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 -- 172.21.15.67:0/1748178129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e5815ec20 msgr2=0x7f0e580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e5815ec20 0x7f0e580d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 -- 172.21.15.67:0/1748178129 >> v2:172.21.15.187:3300/0 conn(0x7f0e5806cf80 msgr2=0x7f0e580cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 --2- 172.21.15.67:0/1748178129 >> v2:172.21.15.187:3300/0 conn(0x7f0e5806cf80 0x7f0e580cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.172+0000 7f0e55400640 1 -- 172.21.15.67:0/1748178129 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e580cf7b0 con 0x7f0e580b3b50 2024-04-17T07:30:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.173+0000 7f0e55400640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e580b3b50 0x7f0e5806ca40 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0e5806c460 tx=0x7f0e4402b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9020016c09fd0c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e54a00640 1 -- 172.21.15.67:0/1748178129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44038070 con 0x7f0e580b3b50 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e54a00640 1 -- 172.21.15.67:0/1748178129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e44004230 con 0x7f0e580b3b50 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e54a00640 1 -- 172.21.15.67:0/1748178129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44003610 con 0x7f0e580b3b50 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 -- 172.21.15.67:0/1748178129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e580b3b50 msgr2=0x7f0e5806ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e580b3b50 0x7f0e5806ca40 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0e5806c460 tx=0x7f0e4402b7d0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 -- 172.21.15.67:0/1748178129 shutdown_connections 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 --2- 172.21.15.67:0/1748178129 >> v2:172.21.15.187:3300/0 conn(0x7f0e5806cf80 0x7f0e580cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e580b3b50 0x7f0e5806ca40 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 --2- 172.21.15.67:0/1748178129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e5815ec20 0x7f0e580d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 -- 172.21.15.67:0/1748178129 >> 172.21.15.67:0/1748178129 conn(0x7f0e5808bff0 msgr2=0x7f0e580d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 -- 172.21.15.67:0/1748178129 shutdown_connections 2024-04-17T07:30:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.174+0000 7f0e57200640 1 -- 172.21.15.67:0/1748178129 wait complete. 2024-04-17T07:30:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.176+0000 7f0e57200640 1 Processor -- start 2024-04-17T07:30:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.176+0000 7f0e57200640 1 -- start start 2024-04-17T07:30:34.177 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.178+0000 7f5963c48040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e5806cf80 0x7f0e580ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e580b3b50 0x7f0e581e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e580d0390 con 0x7f0e5815ec20 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0e580d0210 con 0x7f0e5806cf80 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.177+0000 7f0e57200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e580d0510 con 0x7f0e580b3b50 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50568/0 (socket says 172.21.15.67:50568) 2024-04-17T07:30:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 -- 172.21.15.67:0/3520431696 learned_addr learned my addr 172.21.15.67:0/3520431696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e55400640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e580b3b50 0x7f0e581e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e55e00640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e5806cf80 0x7f0e580ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 -- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e580b3b50 msgr2=0x7f0e581e8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e580b3b50 0x7f0e581e8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 -- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e5806cf80 msgr2=0x7f0e580ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e5806cf80 0x7f0e580ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 -- 172.21.15.67:0/3520431696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e580cba00 con 0x7f0e5815ec20 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.178+0000 7f0e56800640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007d40 tx=0x7f0e5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.179+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50014420 con 0x7f0e5815ec20 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.179+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e50019070 con 0x7f0e5815ec20 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.179+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50014820 con 0x7f0e5815ec20 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.179+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e580cbce0 con 0x7f0e5815ec20 2024-04-17T07:30:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.179+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e580cc2a0 con 0x7f0e5815ec20 2024-04-17T07:30:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.180+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e58068a10 con 0x7f0e5815ec20 2024-04-17T07:30:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.181+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 42) ==== 91143+0+0 (secure 0 0 0) 0x7f0e500149c0 con 0x7f0e5815ec20 2024-04-17T07:30:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.182+0000 7f0e4e000640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 0x7f0e3806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.182+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6053+0+0 (secure 0 0 0) 0x7f0e5008c630 con 0x7f0e5815ec20 2024-04-17T07:30:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.182+0000 7f0e55e00640 1 -- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 msgr2=0x7f0e3806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1013507418 2024-04-17T07:30:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.182+0000 7f0e55e00640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 0x7f0e3806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:30:34.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.186+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0e5005a750 con 0x7f0e5815ec20 2024-04-17T07:30:34.309 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.310+0000 7f5963c48040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:30:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.333+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 43) ==== 90751+0+0 (secure 0 0 0) 0x7f0e500566c0 con 0x7f0e5815ec20 2024-04-17T07:30:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.333+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 msgr2=0x7f0e3806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:30:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.333+0000 7f0e4e000640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 0x7f0e3806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.397+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e580d55f0 con 0x7f0e5815ec20 2024-04-17T07:30:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.398+0000 7f0e4e000640 1 -- 172.21.15.67:0/3520431696 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f0e500767f0 con 0x7f0e5815ec20 2024-04-17T07:30:34.399 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 msgr2=0x7f0e5815b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007d40 tx=0x7f0e5000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 shutdown_connections 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e5815ec20 0x7f0e5815b920 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e580b3b50 0x7f0e581e8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:6832/1013507418,v1:172.21.15.187:6833/1013507418] conn(0x7f0e3806ca10 0x7f0e3806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 --2- 172.21.15.67:0/3520431696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e5806cf80 0x7f0e580ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 >> 172.21.15.67:0/3520431696 conn(0x7f0e5808bff0 msgr2=0x7f0e580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 shutdown_connections 2024-04-17T07:30:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.401+0000 7f0e57200640 1 -- 172.21.15.67:0/3520431696 wait complete. 2024-04-17T07:30:34.414 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.415+0000 7f58e1600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.414 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.415+0000 7f58e1600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.414 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.415+0000 7f58e1600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.414 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.415+0000 7f58e1600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.414 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.415+0000 7f58e1600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.415 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.416+0000 7f58dda00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.415 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.416+0000 7f58dda00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.415 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.416+0000 7f58dda00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.415 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.416+0000 7f58dda00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.415 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:30:34.416+0000 7f58dda00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:30:34.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:30:34.420 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:34.420+0000 7f741a23f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:30:34.484 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:34.484+0000 7f741a23f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:30:34.625 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:30:34.625+0000 7f741a23f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:30:34.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.635+0000 7f143b200640 1 Processor -- start 2024-04-17T07:30:34.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.635+0000 7f143b200640 1 -- start start 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f143b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f143c15ecb0 0x7f143c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f143b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 0x7f143c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f143b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f143c0d00b0 con 0x7f143c0d3210 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f143b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f143c0d0c30 con 0x7f143c15ecb0 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f143b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f143c0d0390 con 0x7f143c0cfab0 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f1439e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f143c15ecb0 0x7f143c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f1433e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f143c0d3210 0x7f143c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46614/0 (socket says 172.21.15.67:46614) 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.637+0000 7f1433e00640 1 -- 172.21.15.67:0/1372016752 learned_addr learned my addr 172.21.15.67:0/1372016752 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f143a800640 1 --2- 172.21.15.67:0/1372016752 >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 0x7f143c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f1439e00640 1 -- 172.21.15.67:0/1372016752 >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 msgr2=0x7f143c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:34.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f1439e00640 1 --2- 172.21.15.67:0/1372016752 >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 0x7f143c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f1439e00640 1 -- 172.21.15.67:0/1372016752 >> v1:172.21.15.67:6789/0 conn(0x7f143c0d3210 legacy=0x7f143c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f1439e00640 1 -- 172.21.15.67:0/1372016752 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c0d0670 con 0x7f143c15ecb0 2024-04-17T07:30:34.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f143a800640 1 --2- 172.21.15.67:0/1372016752 >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 0x7f143c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:34.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.638+0000 7f1439400640 1 -- 172.21.15.67:0/1372016752 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850802614 0 0) 0x7f143c0d00b0 con 0x7f143c0d3210 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.639+0000 7f1439e00640 1 --2- 172.21.15.67:0/1372016752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c15ecb0 0x7f143c0d2cd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f143c0c81b0 tx=0x7f142c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbee00e45d7a3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.639+0000 7f1439400640 1 -- 172.21.15.67:0/1372016752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c0315a0 con 0x7f143c15ecb0 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.639+0000 7f1439400640 1 -- 172.21.15.67:0/1372016752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142c037070 con 0x7f143c15ecb0 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.639+0000 7f1439400640 1 -- 172.21.15.67:0/1372016752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c031960 con 0x7f143c15ecb0 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 -- 172.21.15.67:0/1372016752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c15ecb0 msgr2=0x7f143c0d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 --2- 172.21.15.67:0/1372016752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c15ecb0 0x7f143c0d2cd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f143c0c81b0 tx=0x7f142c02b840 comp rx=0 tx=0).stop 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 -- 172.21.15.67:0/1372016752 shutdown_connections 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 --2- 172.21.15.67:0/1372016752 >> v2:172.21.15.187:3300/0 conn(0x7f143c0cfab0 0x7f143c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 --2- 172.21.15.67:0/1372016752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c15ecb0 0x7f143c0d2cd0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 -- 172.21.15.67:0/1372016752 >> 172.21.15.67:0/1372016752 conn(0x7f143c08bff0 msgr2=0x7f143c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 -- 172.21.15.67:0/1372016752 shutdown_connections 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.640+0000 7f143b200640 1 -- 172.21.15.67:0/1372016752 wait complete. 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.641+0000 7f143b200640 1 Processor -- start 2024-04-17T07:30:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.642+0000 7f143b200640 1 -- start start 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.642+0000 7f143b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f143c0d3210 0x7f143c1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 0x7f143c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f143c15f210 con 0x7f143c0cfab0 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f143c15f090 con 0x7f143c15ecb0 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f143c0b4500 con 0x7f143c0d3210 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50586/0 (socket says 172.21.15.67:50586) 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 -- 172.21.15.67:0/898316179 learned_addr learned my addr 172.21.15.67:0/898316179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143a800640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 0x7f143c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 -- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f143c0d3210 msgr2=0x7f143c1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f143c0d3210 0x7f143c1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 -- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 msgr2=0x7f143c15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 0x7f143c15cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f1439e00640 1 -- 172.21.15.67:0/898316179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c1f47b0 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.643+0000 7f143a800640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 0x7f143c15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f1439e00640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f142c000c00 tx=0x7f142c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c032840 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f142c0329e0 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143c15d240 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c032dd0 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.644+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f143c1f4a30 con 0x7f143c0cfab0 2024-04-17T07:30:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.645+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f143c068a10 con 0x7f143c0cfab0 2024-04-17T07:30:34.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.646+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 43) ==== 90751+0+0 (secure 0 0 0) 0x7f142c04d3f0 con 0x7f143c0cfab0 2024-04-17T07:30:34.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.647+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7f142c02f030 con 0x7f143c0cfab0 2024-04-17T07:30:34.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.651+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f143c15d240 con 0x7f143c0cfab0 2024-04-17T07:30:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.854+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f143c15de40 con 0x7f143c0cfab0 2024-04-17T07:30:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.855+0000 7f1432000640 1 -- 172.21.15.67:0/898316179 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) ==== 104+0+11 (secure 0 0 0) 0x7f143c1f47b0 con 0x7f143c0cfab0 2024-04-17T07:30:34.856 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:30:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 msgr2=0x7f143c0cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f142c000c00 tx=0x7f142c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 shutdown_connections 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f143c15ecb0 0x7f143c15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f143c0d3210 0x7f143c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 --2- 172.21.15.67:0/898316179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c0cfab0 0x7f143c0cc800 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 >> 172.21.15.67:0/898316179 conn(0x7f143c08bff0 msgr2=0x7f143c0b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.858+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 shutdown_connections 2024-04-17T07:30:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:34.859+0000 7f143b200640 1 -- 172.21.15.67:0/898316179 wait complete. 2024-04-17T07:30:34.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:30:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.101+0000 7fd496a00640 1 Processor -- start 2024-04-17T07:30:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.102+0000 7fd496a00640 1 -- start start 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd49015ee80 0x7fd4900d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d2f90 0x7fd4900d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd4900d1c30 0x7fd4900b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd4900b7c20 con 0x7fd49015ee80 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd4900d3d40 con 0x7fd4900d1c30 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd496a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd4900b7ee0 con 0x7fd4900d2f90 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.103+0000 7fd48fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd49015ee80 0x7fd4900d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd49015ee80 0x7fd4900d2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50598/0 (socket says 172.21.15.67:50598) 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 -- 172.21.15.67:0/2200886536 learned_addr learned my addr 172.21.15.67:0/2200886536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 -- 172.21.15.67:0/2200886536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d2f90 msgr2=0x7fd4900d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 --2- 172.21.15.67:0/2200886536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d2f90 0x7fd4900d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 -- 172.21.15.67:0/2200886536 >> v2:172.21.15.187:3300/0 conn(0x7fd4900d1c30 msgr2=0x7fd4900b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 --2- 172.21.15.67:0/2200886536 >> v2:172.21.15.187:3300/0 conn(0x7fd4900d1c30 0x7fd4900b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.104+0000 7fd48fe00640 1 -- 172.21.15.67:0/2200886536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4900d34a0 con 0x7fd49015ee80 2024-04-17T07:30:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.105+0000 7fd48fe00640 1 --2- 172.21.15.67:0/2200886536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd4900d2a50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd4900ceef0 tx=0x7fd48002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607674147fa5b69e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.105+0000 7fd48ea00640 1 -- 172.21.15.67:0/2200886536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48002c9e0 con 0x7fd49015ee80 2024-04-17T07:30:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd48ea00640 1 -- 172.21.15.67:0/2200886536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd48002cb80 con 0x7fd49015ee80 2024-04-17T07:30:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd48ea00640 1 -- 172.21.15.67:0/2200886536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4800343f0 con 0x7fd49015ee80 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 -- 172.21.15.67:0/2200886536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 msgr2=0x7fd4900d2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 --2- 172.21.15.67:0/2200886536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd4900d2a50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd4900ceef0 tx=0x7fd48002be40 comp rx=0 tx=0).stop 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 -- 172.21.15.67:0/2200886536 shutdown_connections 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 --2- 172.21.15.67:0/2200886536 >> v2:172.21.15.187:3300/0 conn(0x7fd4900d1c30 0x7fd4900b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 --2- 172.21.15.67:0/2200886536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d2f90 0x7fd4900d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 --2- 172.21.15.67:0/2200886536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd4900d2a50 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.106+0000 7fd496a00640 1 -- 172.21.15.67:0/2200886536 >> 172.21.15.67:0/2200886536 conn(0x7fd4900677e0 msgr2=0x7fd490067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.107+0000 7fd496a00640 1 -- 172.21.15.67:0/2200886536 shutdown_connections 2024-04-17T07:30:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.107+0000 7fd496a00640 1 -- 172.21.15.67:0/2200886536 wait complete. 2024-04-17T07:30:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.108+0000 7fd496a00640 1 Processor -- start 2024-04-17T07:30:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.108+0000 7fd496a00640 1 -- start start 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d1c30 0x7fd49015a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900d2f90 0x7fd4901f8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4900d42c0 con 0x7fd49015ee80 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd4900d3a60 con 0x7fd4900d2f90 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.109+0000 7fd496a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd4900d4440 con 0x7fd4900d1c30 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50608/0 (socket says 172.21.15.67:50608) 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 -- 172.21.15.67:0/1397690087 learned_addr learned my addr 172.21.15.67:0/1397690087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd48f400640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900d2f90 0x7fd4901f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd48fe00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d1c30 0x7fd49015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 -- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d1c30 msgr2=0x7fd49015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d1c30 0x7fd49015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 -- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900d2f90 msgr2=0x7fd4901f8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900d2f90 0x7fd4901f8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 -- 172.21.15.67:0/1397690087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd490161450 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.110+0000 7fd494a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd488009fd0 tx=0x7fd48800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.111+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd488011830 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.111+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd488016070 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.111+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd488011bf0 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.111+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd490161730 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.111+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4901eb080 con 0x7fd49015ee80 2024-04-17T07:30:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.112+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd490068b90 con 0x7fd49015ee80 2024-04-17T07:30:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.113+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 43) ==== 90751+0+0 (secure 0 0 0) 0x7fd48801b460 con 0x7fd49015ee80 2024-04-17T07:30:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.114+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7fd48808b6b0 con 0x7fd49015ee80 2024-04-17T07:30:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.118+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd488059b90 con 0x7fd49015ee80 2024-04-17T07:30:35.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.323+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd490161900 con 0x7fd49015ee80 2024-04-17T07:30:35.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.325+0000 7fd487e00640 1 -- 172.21.15.67:0/1397690087 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) ==== 100+0+11 (secure 0 0 0) 0x7fd488059490 con 0x7fd49015ee80 2024-04-17T07:30:35.325 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:30:35.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.327+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 msgr2=0x7fd49015c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.327+0000 7fd496a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd488009fd0 tx=0x7fd48800b8e0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 shutdown_connections 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd49015ee80 0x7fd49015c990 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4900d2f90 0x7fd4901f8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 --2- 172.21.15.67:0/1397690087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd4900d1c30 0x7fd49015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 >> 172.21.15.67:0/1397690087 conn(0x7fd4900677e0 msgr2=0x7fd490067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:35.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 shutdown_connections 2024-04-17T07:30:35.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.328+0000 7fd496a00640 1 -- 172.21.15.67:0/1397690087 wait complete. 2024-04-17T07:30:35.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.569+0000 7faa77200640 1 Processor -- start 2024-04-17T07:30:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.569+0000 7faa77200640 1 -- start start 2024-04-17T07:30:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa77200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faa700b6d00 0x7faa700cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa77200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa700d2d30 con 0x7faa7015ec60 2024-04-17T07:30:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa77200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa700d3010 con 0x7faa700cf910 2024-04-17T07:30:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa77200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faa700d49c0 con 0x7faa700b6d00 2024-04-17T07:30:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa75e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faa7015ec60 0x7faa700b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46626/0 (socket says 172.21.15.67:46626) 2024-04-17T07:30:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.571+0000 7faa75e00640 1 -- 172.21.15.67:0/918952677 learned_addr learned my addr 172.21.15.67:0/918952677 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351452261 0 0) 0x7faa700d2d30 con 0x7faa7015ec60 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa58003520 con 0x7faa7015ec60 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa75400640 1 --2- 172.21.15.67:0/918952677 >> v2:172.21.15.187:3300/0 conn(0x7faa700b6d00 0x7faa700cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3120945310 0 0) 0x7faa58003520 con 0x7faa7015ec60 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 >> v2:172.21.15.187:3300/0 conn(0x7faa700b6d00 msgr2=0x7faa700cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 --2- 172.21.15.67:0/918952677 >> v2:172.21.15.187:3300/0 conn(0x7faa700b6d00 0x7faa700cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 >> v1:172.21.15.187:6789/0 conn(0x7faa700cf910 legacy=0x7faa700b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa700d32f0 con 0x7faa7015ec60 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7faa64001670 con 0x7faa7015ec60 2024-04-17T07:30:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.572+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 >> v1:172.21.15.67:6789/0 conn(0x7faa7015ec60 legacy=0x7faa700b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 0x7faa58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58004800 0x7faa58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa58005210 0x7faa5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faa700d49c0 con 0x7faa58003a20 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faa5800c160 con 0x7faa58005210 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faa5800c360 con 0x7faa58004800 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa75e00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 0x7faa58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa75400640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58004800 0x7faa58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.573+0000 7faa76800640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa58005210 0x7faa5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa75e00640 1 -- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58004800 msgr2=0x7faa58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa75e00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58004800 0x7faa58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa75e00640 1 -- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa58005210 msgr2=0x7faa5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa75e00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa58005210 0x7faa5800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa76800640 1 -- 172.21.15.67:0/918952677 reap_dead start 2024-04-17T07:30:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.574+0000 7faa75e00640 1 -- 172.21.15.67:0/918952677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa58003520 con 0x7faa58003a20 2024-04-17T07:30:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.576+0000 7faa75e00640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 0x7faa58003e20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c240 tx=0x7faa58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b5bf49956c9d02 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64007d10 con 0x7faa58003a20 2024-04-17T07:30:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa64030070 con 0x7faa58003a20 2024-04-17T07:30:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa74a00640 1 -- 172.21.15.67:0/918952677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6402cda0 con 0x7faa58003a20 2024-04-17T07:30:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 -- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 msgr2=0x7faa58003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 0x7faa58003e20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c240 tx=0x7faa58005830 comp rx=0 tx=0).stop 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 -- 172.21.15.67:0/918952677 shutdown_connections 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 --2- 172.21.15.67:0/918952677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa58003a20 0x7faa58003e20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 -- 172.21.15.67:0/918952677 >> 172.21.15.67:0/918952677 conn(0x7faa700d15f0 msgr2=0x7faa700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.577+0000 7faa77200640 1 -- 172.21.15.67:0/918952677 shutdown_connections 2024-04-17T07:30:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.578+0000 7faa77200640 1 -- 172.21.15.67:0/918952677 wait complete. 2024-04-17T07:30:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.579+0000 7faa77200640 1 Processor -- start 2024-04-17T07:30:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.579+0000 7faa77200640 1 -- start start 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58003a20 0x7faa700ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 0x7faa701f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faa7015f040 con 0x7faa70156fb0 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faa700d3660 con 0x7faa701f3c90 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.580+0000 7faa77200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faa700d00c0 con 0x7faa58003a20 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50618/0 (socket says 172.21.15.67:50618) 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 -- 172.21.15.67:0/2798322353 learned_addr learned my addr 172.21.15.67:0/2798322353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75e00640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58003a20 0x7faa700ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa76800640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 0x7faa701f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 -- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58003a20 msgr2=0x7faa700ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58003a20 0x7faa700ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 -- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 msgr2=0x7faa701f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 0x7faa701f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa75400640 1 -- 172.21.15.67:0/2798322353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa701f4ca0 con 0x7faa70156fb0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.581+0000 7faa76800640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 0x7faa701f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa75400640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faa6000f600 tx=0x7faa6000fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa60019020 con 0x7faa70156fb0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa60014870 con 0x7faa70156fb0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa60014c20 con 0x7faa70156fb0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa701f4f80 con 0x7faa70156fb0 2024-04-17T07:30:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.582+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa700ccef0 con 0x7faa70156fb0 2024-04-17T07:30:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.583+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa700ccce0 con 0x7faa70156fb0 2024-04-17T07:30:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.584+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 44) ==== 90878+0+0 (secure 0 0 0) 0x7faa6001c020 con 0x7faa70156fb0 2024-04-17T07:30:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.586+0000 7faa6a000640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 0x7faa5406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.586+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7faa6008c200 con 0x7faa70156fb0 2024-04-17T07:30:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.586+0000 7faa75e00640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 0x7faa5406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.587+0000 7faa75e00640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 0x7faa5406ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa64034000 tx=0x7faa64007000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.590+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faa60023020 con 0x7faa70156fb0 2024-04-17T07:30:35.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.837+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa70068a10 con 0x7faa70156fb0 2024-04-17T07:30:35.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.838+0000 7faa6a000640 1 -- 172.21.15.67:0/2798322353 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7faa6005a630 con 0x7faa70156fb0 2024-04-17T07:30:35.839 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:35.839 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.840+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 msgr2=0x7faa5406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.840+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 0x7faa5406ec30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa64034000 tx=0x7faa64007000 comp rx=0 tx=0).stop 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 msgr2=0x7faa70157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7faa6000f600 tx=0x7faa6000fac0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 shutdown_connections 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faa5406c770 0x7faa5406ec30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa701f3c90 0x7faa701f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa70156fb0 0x7faa70157430 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 --2- 172.21.15.67:0/2798322353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faa58003a20 0x7faa700ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 >> 172.21.15.67:0/2798322353 conn(0x7faa700d15f0 msgr2=0x7faa700b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 shutdown_connections 2024-04-17T07:30:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:35.841+0000 7faa77200640 1 -- 172.21.15.67:0/2798322353 wait complete. 2024-04-17T07:30:35.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:36.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.078+0000 7f4c9f400640 1 Processor -- start 2024-04-17T07:30:36.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.078+0000 7f4c9f400640 1 -- start start 2024-04-17T07:30:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b530 0x7f4c98071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c9815b910 0x7f4c98072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9f400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c9815f3c0 con 0x7f4c980c68c0 2024-04-17T07:30:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4c9806ed60 con 0x7f4c9815b910 2024-04-17T07:30:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9815f660 con 0x7f4c9815b530 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c97e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c980c68c0 0x7f4c980c5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46640/0 (socket says 172.21.15.67:46640) 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c97e00640 1 -- 172.21.15.67:0/3402466001 learned_addr learned my addr 172.21.15.67:0/3402466001 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.080+0000 7f4c9d400640 1 --2- 172.21.15.67:0/3402466001 >> v2:172.21.15.67:3300/0 conn(0x7f4c9815b910 0x7f4c98072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.081+0000 7f4c9ca00640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b530 0x7f4c98071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.081+0000 7f4c9d400640 1 -- 172.21.15.67:0/3402466001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b530 msgr2=0x7f4c98071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.081+0000 7f4c9d400640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b530 0x7f4c98071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.081+0000 7f4c9d400640 1 -- 172.21.15.67:0/3402466001 >> v1:172.21.15.67:6789/0 conn(0x7f4c980c68c0 legacy=0x7f4c980c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.081+0000 7f4c9d400640 1 -- 172.21.15.67:0/3402466001 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9815f940 con 0x7f4c9815b910 2024-04-17T07:30:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.082+0000 7f4c9d400640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b910 0x7f4c98072250 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d90 tx=0x7f4c9002bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39af2f4ed1d9d84e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.082+0000 7f4c97400640 1 -- 172.21.15.67:0/3402466001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90039070 con 0x7f4c9815b910 2024-04-17T07:30:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.082+0000 7f4c97400640 1 -- 172.21.15.67:0/3402466001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9002ca10 con 0x7f4c9815b910 2024-04-17T07:30:36.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.082+0000 7f4c97400640 1 -- 172.21.15.67:0/3402466001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9002cde0 con 0x7f4c9815b910 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 -- 172.21.15.67:0/3402466001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b910 msgr2=0x7f4c98072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b910 0x7f4c98072250 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d90 tx=0x7f4c9002bce0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 -- 172.21.15.67:0/3402466001 shutdown_connections 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b910 0x7f4c98072250 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 --2- 172.21.15.67:0/3402466001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b530 0x7f4c98071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 -- 172.21.15.67:0/3402466001 >> 172.21.15.67:0/3402466001 conn(0x7f4c9806bf50 msgr2=0x7f4c980b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 -- 172.21.15.67:0/3402466001 shutdown_connections 2024-04-17T07:30:36.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.083+0000 7f4c9f400640 1 -- 172.21.15.67:0/3402466001 wait complete. 2024-04-17T07:30:36.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.085+0000 7f4c9f400640 1 Processor -- start 2024-04-17T07:30:36.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.085+0000 7f4c9f400640 1 -- start start 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c980c68c0 0x7f4c980b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b910 0x7f4c981ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c980cc950 con 0x7f4c9815b530 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4c980ccca0 con 0x7f4c980c68c0 2024-04-17T07:30:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c9f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c98160140 con 0x7f4c9815b910 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50642/0 (socket says 172.21.15.67:50642) 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.086+0000 7f4c97e00640 1 -- 172.21.15.67:0/2978985089 learned_addr learned my addr 172.21.15.67:0/2978985089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c9ca00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c980c68c0 0x7f4c980b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 -- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b910 msgr2=0x7f4c981ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c9d400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b910 0x7f4c981ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b910 0x7f4c981ef8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 -- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c980c68c0 msgr2=0x7f4c980b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c980c68c0 0x7f4c980b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 -- 172.21.15.67:0/2978985089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c980b4350 con 0x7f4c9815b530 2024-04-17T07:30:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c97e00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400c780 tx=0x7f4c8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.087+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84018070 con 0x7f4c9815b530 2024-04-17T07:30:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.088+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c840072d0 con 0x7f4c9815b530 2024-04-17T07:30:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.088+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c840047e0 con 0x7f4c9815b530 2024-04-17T07:30:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.088+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c980b4630 con 0x7f4c9815b530 2024-04-17T07:30:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.088+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c981f2300 con 0x7f4c9815b530 2024-04-17T07:30:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.089+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c98068b90 con 0x7f4c9815b530 2024-04-17T07:30:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.090+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 44) ==== 90878+0+0 (secure 0 0 0) 0x7f4c84004980 con 0x7f4c9815b530 2024-04-17T07:30:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.091+0000 7f4c94c00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 0x7f4c7806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.091+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7f4c8408aff0 con 0x7f4c9815b530 2024-04-17T07:30:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.091+0000 7f4c9ca00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 0x7f4c7806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.093+0000 7f4c9ca00640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 0x7f4c7806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c006fd0 tx=0x7f4c8c008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.095+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4c84004c70 con 0x7f4c9815b530 2024-04-17T07:30:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.304+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c980c6390 con 0x7f4c9815b530 2024-04-17T07:30:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.307+0000 7f4c94c00640 1 -- 172.21.15.67:0/2978985089 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+17637 (secure 0 0 0) 0x7f4c840012b0 con 0x7f4c9815b530 2024-04-17T07:30:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:36.309 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":100,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:34.315210+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1236658145":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4268794575":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3413674157":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/1120045762":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6832/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4218129231":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3033007882":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:6832/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1419379931":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/201608361":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1684018318":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/35793766":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/3592827215":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/2290658878":"2024-04-18T07:30:27.980642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 msgr2=0x7f4c7806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 0x7f4c7806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c006fd0 tx=0x7f4c8c008040 comp rx=0 tx=0).stop 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 msgr2=0x7f4c980b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400c780 tx=0x7f4c8400cc40 comp rx=0 tx=0).stop 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 shutdown_connections 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f4c7806c740 0x7f4c7806ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c9815b910 0x7f4c981ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c9815b530 0x7f4c980b34d0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 --2- 172.21.15.67:0/2978985089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c980c68c0 0x7f4c980b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 >> 172.21.15.67:0/2978985089 conn(0x7f4c9806bf50 msgr2=0x7f4c980c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.310+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 shutdown_connections 2024-04-17T07:30:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.311+0000 7f4c9f400640 1 -- 172.21.15.67:0/2978985089 wait complete. 2024-04-17T07:30:36.331 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:30:36.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.551+0000 7f64c2e00640 1 Processor -- start 2024-04-17T07:30:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.551+0000 7f64c2e00640 1 -- start start 2024-04-17T07:30:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f64bc06feb0 0x7f64bc06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64bc15bc80 con 0x7f64bc06be70 2024-04-17T07:30:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64bc15bf40 con 0x7f64bc15b570 2024-04-17T07:30:36.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f64bc15c220 con 0x7f64bc06feb0 2024-04-17T07:30:36.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64bc06be70 0x7f64bc15f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46648/0 (socket says 172.21.15.67:46648) 2024-04-17T07:30:36.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.553+0000 7f64c2400640 1 -- 172.21.15.67:0/442517333 learned_addr learned my addr 172.21.15.67:0/442517333 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64c1000640 1 --2- 172.21.15.67:0/442517333 >> v2:172.21.15.187:3300/0 conn(0x7f64bc06feb0 0x7f64bc06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1643335089 0 0) 0x7f64bc15bc80 con 0x7f64bc06be70 2024-04-17T07:30:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a4003420 con 0x7f64bc06be70 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3496269388 0 0) 0x7f64a4003420 con 0x7f64bc06be70 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 >> v2:172.21.15.187:3300/0 conn(0x7f64bc06feb0 msgr2=0x7f64bc06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 --2- 172.21.15.67:0/442517333 >> v2:172.21.15.187:3300/0 conn(0x7f64bc06feb0 0x7f64bc06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 >> v1:172.21.15.187:6789/0 conn(0x7f64bc15b570 legacy=0x7f64bc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc15c500 con 0x7f64bc06be70 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f64b40019f0 con 0x7f64bc06be70 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 >> v1:172.21.15.67:6789/0 conn(0x7f64bc06be70 legacy=0x7f64bc15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.554+0000 7f64bbe00640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64a4004b50 0x7f64a4004f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64bbe00640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a40036a0 0x7f64a4003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64bbe00640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 0x7f64a400ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc15c220 con 0x7f64a400db60 2024-04-17T07:30:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f64a40105c0 con 0x7f64a40036a0 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64a40107c0 con 0x7f64a4004b50 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 0x7f64a400ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c1000640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a40036a0 0x7f64a4003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 -- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64a4004b50 msgr2=0x7f64a4004f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64a4004b50 0x7f64a4004f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 -- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a40036a0 msgr2=0x7f64a4003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a40036a0 0x7f64a4003b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.555+0000 7f64c2400640 1 -- 172.21.15.67:0/442517333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a4003420 con 0x7f64a400db60 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.556+0000 7f64c2400640 1 -- 172.21.15.67:0/442517333 reap_dead start 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.556+0000 7f64c2400640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 0x7f64a400ff50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f64b400a790 tx=0x7f64a4004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928041607a89a936 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b400a3b0 con 0x7f64a400db60 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b4034670 con 0x7f64a400db60 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64bbe00640 1 -- 172.21.15.67:0/442517333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4034a60 con 0x7f64a400db60 2024-04-17T07:30:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 -- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 msgr2=0x7f64a400ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 0x7f64a400ff50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f64b400a790 tx=0x7f64a4004720 comp rx=0 tx=0).stop 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 -- 172.21.15.67:0/442517333 shutdown_connections 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 --2- 172.21.15.67:0/442517333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a400db60 0x7f64a400ff50 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 -- 172.21.15.67:0/442517333 >> 172.21.15.67:0/442517333 conn(0x7f64bc067d10 msgr2=0x7f64bc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.557+0000 7f64c2e00640 1 -- 172.21.15.67:0/442517333 shutdown_connections 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.558+0000 7f64c2e00640 1 -- 172.21.15.67:0/442517333 wait complete. 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.559+0000 7f64c2e00640 1 Processor -- start 2024-04-17T07:30:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.560+0000 7f64c2e00640 1 -- start start 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a400db60 0x7f64bc14f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 0x7f64bc151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc0b4930 con 0x7f64bc0b13f0 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc0b47b0 con 0x7f64a400db60 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c2e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64bc06c280 con 0x7f64bc151a90 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50666/0 (socket says 172.21.15.67:50666) 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.561+0000 7f64c1000640 1 -- 172.21.15.67:0/2481380036 learned_addr learned my addr 172.21.15.67:0/2481380036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1a00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a400db60 0x7f64bc14f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c2400640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 0x7f64bc151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 -- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 msgr2=0x7f64bc151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 0x7f64bc151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 -- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a400db60 msgr2=0x7f64bc14f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a400db60 0x7f64bc14f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 -- 172.21.15.67:0/2481380036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc1527e0 con 0x7f64bc0b13f0 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c2400640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 0x7f64bc151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64c1000640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f64a8010890 tx=0x7f64a8010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a800f040 con 0x7f64bc0b13f0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a800eb60 con 0x7f64bc0b13f0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.562+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a801c400 con 0x7f64bc0b13f0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.563+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64bc152ac0 con 0x7f64bc0b13f0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.563+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64bc153080 con 0x7f64bc0b13f0 2024-04-17T07:30:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.564+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc068a10 con 0x7f64bc0b13f0 2024-04-17T07:30:36.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.565+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 44) ==== 90878+0+0 (secure 0 0 0) 0x7f64a801c5a0 con 0x7f64bc0b13f0 2024-04-17T07:30:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.566+0000 7f64b9600640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 0x7f649c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.566+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7f64a808c160 con 0x7f64bc0b13f0 2024-04-17T07:30:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.566+0000 7f64c1a00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 0x7f649c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:36.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.568+0000 7f64c1a00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 0x7f649c06ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64b0001630 tx=0x7f64b0009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:36.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.570+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f64a8018030 con 0x7f64bc0b13f0 2024-04-17T07:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.822+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64bc0b2680 con 0x7f64bc0b13f0 2024-04-17T07:30:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.823+0000 7f64b9600640 1 -- 172.21.15.67:0/2481380036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7f64a805a550 con 0x7f64bc0b13f0 2024-04-17T07:30:36.824 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:36.824 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 msgr2=0x7f649c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 0x7f649c06ec00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64b0001630 tx=0x7f64b0009040 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 msgr2=0x7f64bc0b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f64a8010890 tx=0x7f64a8010d50 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 shutdown_connections 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f649c06c740 0x7f649c06ec00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64bc151a90 0x7f64bc151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64bc0b13f0 0x7f64bc0b1850 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 --2- 172.21.15.67:0/2481380036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64a400db60 0x7f64bc14f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 >> 172.21.15.67:0/2481380036 conn(0x7f64bc067d10 msgr2=0x7f64bc0b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.826+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 shutdown_connections 2024-04-17T07:30:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:36.827+0000 7f64c2e00640 1 -- 172.21.15.67:0/2481380036 wait complete. 2024-04-17T07:30:36.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:37.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.074+0000 7f3519a00640 1 Processor -- start 2024-04-17T07:30:37.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.075+0000 7f3519a00640 1 -- start start 2024-04-17T07:30:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.077+0000 7f3519a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35140b6d00 0x7f35140cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3519a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f35140cf910 0x7f35140b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3519a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35140d2d30 con 0x7f351415ec60 2024-04-17T07:30:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3519a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f35140d49c0 con 0x7f35140b6d00 2024-04-17T07:30:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3519a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f35140d3660 con 0x7f35140cf910 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3513400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35140b6d00 0x7f35140cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3513400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35140b6d00 0x7f35140cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50674/0 (socket says 172.21.15.67:50674) 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.078+0000 7f3513400640 1 -- 172.21.15.67:0/3689502742 learned_addr learned my addr 172.21.15.67:0/3689502742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.079+0000 7f3519000640 1 --2- 172.21.15.67:0/3689502742 >> v2:172.21.15.187:3300/0 conn(0x7f35140cf910 0x7f35140b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.079+0000 7f3513400640 1 -- 172.21.15.67:0/3689502742 >> v2:172.21.15.187:3300/0 conn(0x7f35140cf910 msgr2=0x7f35140b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.079+0000 7f3513400640 1 --2- 172.21.15.67:0/3689502742 >> v2:172.21.15.187:3300/0 conn(0x7f35140cf910 0x7f35140b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.079+0000 7f3513400640 1 -- 172.21.15.67:0/3689502742 >> v1:172.21.15.187:6789/0 conn(0x7f351415ec60 legacy=0x7f35140b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.079+0000 7f3513400640 1 -- 172.21.15.67:0/3689502742 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35140d3010 con 0x7f35140b6d00 2024-04-17T07:30:37.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.080+0000 7f3513400640 1 --2- 172.21.15.67:0/3689502742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35140b6d00 0x7f35140cf3d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f35140cf170 tx=0x7f350002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c826fb9dce1ced8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.080+0000 7f3512a00640 1 -- 172.21.15.67:0/3689502742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500034070 con 0x7f35140b6d00 2024-04-17T07:30:37.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.080+0000 7f3512a00640 1 -- 172.21.15.67:0/3689502742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3500030880 con 0x7f35140b6d00 2024-04-17T07:30:37.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.080+0000 7f3512a00640 1 -- 172.21.15.67:0/3689502742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500030c50 con 0x7f35140b6d00 2024-04-17T07:30:37.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 -- 172.21.15.67:0/3689502742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35140b6d00 msgr2=0x7f35140cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 --2- 172.21.15.67:0/3689502742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35140b6d00 0x7f35140cf3d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f35140cf170 tx=0x7f350002b9c0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 -- 172.21.15.67:0/3689502742 shutdown_connections 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 --2- 172.21.15.67:0/3689502742 >> v2:172.21.15.187:3300/0 conn(0x7f35140cf910 0x7f35140b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 --2- 172.21.15.67:0/3689502742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35140b6d00 0x7f35140cf3d0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 -- 172.21.15.67:0/3689502742 >> 172.21.15.67:0/3689502742 conn(0x7f35140d15f0 msgr2=0x7f35140d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.081+0000 7f3519a00640 1 -- 172.21.15.67:0/3689502742 shutdown_connections 2024-04-17T07:30:37.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.082+0000 7f3519a00640 1 -- 172.21.15.67:0/3689502742 wait complete. 2024-04-17T07:30:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.083+0000 7f3519a00640 1 Processor -- start 2024-04-17T07:30:37.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.084+0000 7f3519a00640 1 -- start start 2024-04-17T07:30:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.084+0000 7f3519a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35140b6d00 0x7f351415cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.084+0000 7f3519a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35140cf910 0x7f35140ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f351415f040 con 0x7f351415ec60 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f35140cfdc0 con 0x7f35140cf910 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f35140d00c0 con 0x7f35140b6d00 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50684/0 (socket says 172.21.15.67:50684) 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 -- 172.21.15.67:0/1662772366 learned_addr learned my addr 172.21.15.67:0/1662772366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 -- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35140b6d00 msgr2=0x7f351415cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35140b6d00 0x7f351415cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 -- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35140cf910 msgr2=0x7f35140ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35140cf910 0x7f35140ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.085+0000 7f3519000640 1 -- 172.21.15.67:0/1662772366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351415d180 con 0x7f351415ec60 2024-04-17T07:30:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f3519000640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f350c00cda0 tx=0x7f350c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c003b90 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f350c003d30 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35140cd7d0 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c004240 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.086+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35140cdd50 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.087+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3514068a10 con 0x7f351415ec60 2024-04-17T07:30:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.089+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f350c0043e0 con 0x7f351415ec60 2024-04-17T07:30:37.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.090+0000 7f350be00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 0x7f34f406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.090+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7f350c08cb00 con 0x7f351415ec60 2024-04-17T07:30:37.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.090+0000 7f3513e00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 0x7f34f406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.092+0000 7f3513e00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 0x7f34f406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3504000f60 tx=0x7f35040079d0 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.094+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f350c01d340 con 0x7f351415ec60 2024-04-17T07:30:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.345+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35140d3f00 con 0x7f351415ec60 2024-04-17T07:30:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.346+0000 7f350be00640 1 -- 172.21.15.67:0/1662772366 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5204 (secure 0 0 0) 0x7f350c05ab10 con 0x7f351415ec60 2024-04-17T07:30:37.346 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:37.347 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:13.116971+0000","modified":"2024-04-17T07:30:16.196310+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/381512147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":381512147},{"type":"v1","addr":"172.21.15.67:6835","nonce":381512147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6906,"qdb_cluster":[6906]},"id":5}]} 2024-04-17T07:30:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 msgr2=0x7f34f406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 0x7f34f406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3504000f60 tx=0x7f35040079d0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 msgr2=0x7f35140add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f350c00cda0 tx=0x7f350c00a660 comp rx=0 tx=0).stop 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 shutdown_connections 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f34f406c980 0x7f34f406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f351415ec60 0x7f35140add20 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35140cf910 0x7f35140ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 --2- 172.21.15.67:0/1662772366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f35140b6d00 0x7f351415cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 >> 172.21.15.67:0/1662772366 conn(0x7f35140d15f0 msgr2=0x7f35140b9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 shutdown_connections 2024-04-17T07:30:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.349+0000 7f3519a00640 1 -- 172.21.15.67:0/1662772366 wait complete. 2024-04-17T07:30:37.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:37.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.590+0000 7fdfafe00640 1 Processor -- start 2024-04-17T07:30:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.591+0000 7fdfafe00640 1 -- start start 2024-04-17T07:30:37.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.592+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 0x7fdfb00b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.592+0000 7fdfafe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfb00b6b30 0x7fdfb00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.592+0000 7fdfafe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb00d2d30 con 0x7fdfb00cf910 2024-04-17T07:30:37.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.592+0000 7fdfafe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb00d49c0 con 0x7fdfb00b6b30 2024-04-17T07:30:37.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.592+0000 7fdfafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb00d3660 con 0x7fdfb015ec60 2024-04-17T07:30:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.593+0000 7fdfaea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 0x7fdfb00b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.593+0000 7fdfae000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfb00b6b30 0x7fdfb00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.593+0000 7fdfaf400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdfb00cf910 0x7fdfb006cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:51818/0 (socket says 172.21.15.67:51818) 2024-04-17T07:30:37.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.593+0000 7fdfaf400640 1 -- 172.21.15.67:0/995284609 learned_addr learned my addr 172.21.15.67:0/995284609 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:37.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.594+0000 7fdfaea00640 1 -- 172.21.15.67:0/995284609 >> v2:172.21.15.187:3300/0 conn(0x7fdfb00b6b30 msgr2=0x7fdfb00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:37.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.594+0000 7fdfaea00640 1 --2- 172.21.15.67:0/995284609 >> v2:172.21.15.187:3300/0 conn(0x7fdfb00b6b30 0x7fdfb00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.594+0000 7fdfaea00640 1 -- 172.21.15.67:0/995284609 >> v1:172.21.15.187:6789/0 conn(0x7fdfb00cf910 legacy=0x7fdfb006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.594+0000 7fdfaea00640 1 -- 172.21.15.67:0/995284609 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb00d3010 con 0x7fdfb015ec60 2024-04-17T07:30:37.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.595+0000 7fdfaea00640 1 --2- 172.21.15.67:0/995284609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 0x7fdfb00b65f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00097a0 tx=0x7fdfa002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86ab99e9bd5f4fad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfad600640 1 -- 172.21.15.67:0/995284609 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0035020 con 0x7fdfb015ec60 2024-04-17T07:30:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfad600640 1 -- 172.21.15.67:0/995284609 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa0031810 con 0x7fdfb015ec60 2024-04-17T07:30:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfad600640 1 -- 172.21.15.67:0/995284609 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0031be0 con 0x7fdfb015ec60 2024-04-17T07:30:37.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 -- 172.21.15.67:0/995284609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 msgr2=0x7fdfb00b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 --2- 172.21.15.67:0/995284609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 0x7fdfb00b65f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00097a0 tx=0x7fdfa002b8b0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 -- 172.21.15.67:0/995284609 shutdown_connections 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 --2- 172.21.15.67:0/995284609 >> v2:172.21.15.187:3300/0 conn(0x7fdfb00b6b30 0x7fdfb00cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 --2- 172.21.15.67:0/995284609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb015ec60 0x7fdfb00b65f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.596+0000 7fdfafe00640 1 -- 172.21.15.67:0/995284609 >> 172.21.15.67:0/995284609 conn(0x7fdfb00d15f0 msgr2=0x7fdfb00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.597+0000 7fdfafe00640 1 -- 172.21.15.67:0/995284609 shutdown_connections 2024-04-17T07:30:37.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.597+0000 7fdfafe00640 1 -- 172.21.15.67:0/995284609 wait complete. 2024-04-17T07:30:37.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.598+0000 7fdfafe00640 1 Processor -- start 2024-04-17T07:30:37.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 -- start start 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb00b6b30 0x7fdfb00cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 0x7fdfb00b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb00cfda0 con 0x7fdfb00cf910 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb00b6f70 con 0x7fdfb015ec60 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.599+0000 7fdfafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb015f040 con 0x7fdfb00b6b30 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50696/0 (socket says 172.21.15.67:50696) 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 -- 172.21.15.67:0/3991645471 learned_addr learned my addr 172.21.15.67:0/3991645471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:37.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfaea00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb00b6b30 0x7fdfb00cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfaf400640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 0x7fdfb00b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 -- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb00b6b30 msgr2=0x7fdfb00cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb00b6b30 0x7fdfb00cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 -- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 msgr2=0x7fdfb00b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 0x7fdfb00b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfae000640 1 -- 172.21.15.67:0/3991645471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb0157ce0 con 0x7fdfb00cf910 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.600+0000 7fdfaf400640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 0x7fdfb00b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfae000640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00e9b0 tx=0x7fdf9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c017070 con 0x7fdfb00cf910 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c00c9a0 con 0x7fdfb00cf910 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c00cd90 con 0x7fdfb00cf910 2024-04-17T07:30:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb00b9b20 con 0x7fdfb00cf910 2024-04-17T07:30:37.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.601+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb00ba360 con 0x7fdfb00cf910 2024-04-17T07:30:37.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.603+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb0068a10 con 0x7fdfb00cf910 2024-04-17T07:30:37.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.604+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fdf9c01a070 con 0x7fdfb00cf910 2024-04-17T07:30:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.605+0000 7fdfa6a00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 0x7fdf9006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.605+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7fdf9c08bf50 con 0x7fdfb00cf910 2024-04-17T07:30:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.605+0000 7fdfaea00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 0x7fdf9006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.607+0000 7fdfaea00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 0x7fdf9006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0036000 tx=0x7fdfa003c040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.609+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdfb00b9b20 con 0x7fdfb00cf910 2024-04-17T07:30:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.820+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdfb015ba30 con 0x7fdfb00cf910 2024-04-17T07:30:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.822+0000 7fdfa6a00640 1 -- 172.21.15.67:0/3991645471 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+17637 (secure 0 0 0) 0x7fdf9c05ee20 con 0x7fdfb00cf910 2024-04-17T07:30:37.823 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:37.823 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":100,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:34.315210+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:11.123171+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:12.078044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1236658145":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4268794575":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3413674157":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/1120045762":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6832/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4218129231":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3033007882":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:6832/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1419379931":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/201608361":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1684018318":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/35793766":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/3592827215":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/2290658878":"2024-04-18T07:30:27.980642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 msgr2=0x7fdf9006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 0x7fdf9006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0036000 tx=0x7fdfa003c040 comp rx=0 tx=0).stop 2024-04-17T07:30:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 msgr2=0x7fdfb00cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00e9b0 tx=0x7fdf9c00ee70 comp rx=0 tx=0).stop 2024-04-17T07:30:37.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 shutdown_connections 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fdf9006c980 0x7fdf9006ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb015ec60 0x7fdfb00b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfb00cf910 0x7fdfb00cb360 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 --2- 172.21.15.67:0/3991645471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfb00b6b30 0x7fdfb00cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 >> 172.21.15.67:0/3991645471 conn(0x7fdfb00d15f0 msgr2=0x7fdfb0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 shutdown_connections 2024-04-17T07:30:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:37.825+0000 7fdfafe00640 1 -- 172.21.15.67:0/3991645471 wait complete. 2024-04-17T07:30:37.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:30:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.060+0000 7fde43200640 1 Processor -- start 2024-04-17T07:30:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.061+0000 7fde43200640 1 -- start start 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.062+0000 7fde43200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde3c0c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde43200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3c162ce0 con 0x7fde3c0afc70 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde43200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3c162fc0 con 0x7fde3c0c9f80 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde43200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c06e1b0 con 0x7fde3c162740 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde41e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde3c0afc70 0x7fde3c15fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46650/0 (socket says 172.21.15.67:46650) 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde41e00640 1 -- 172.21.15.67:0/119392802 learned_addr learned my addr 172.21.15.67:0/119392802 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385359737 0 0) 0x7fde3c162ce0 con 0x7fde3c0afc70 2024-04-17T07:30:38.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde41400640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde3c0c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.063+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde28003520 con 0x7fde3c0afc70 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2109466203 0 0) 0x7fde28003520 con 0x7fde3c0afc70 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 msgr2=0x7fde3c0c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde3c0c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 >> v1:172.21.15.187:6789/0 conn(0x7fde3c0c9f80 legacy=0x7fde3c06e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1632a0 con 0x7fde3c0afc70 2024-04-17T07:30:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fde34001670 con 0x7fde3c0afc70 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 >> v1:172.21.15.67:6789/0 conn(0x7fde3c0afc70 legacy=0x7fde3c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28003a50 0x7fde28003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde28004800 0x7fde28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c06e1b0 con 0x7fde28004800 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fde28005550 con 0x7fde28003a50 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fde2800bf00 con 0x7fde3c162740 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde41400640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde28004800 0x7fde28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde41e00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28003a50 0x7fde28003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.064+0000 7fde42800640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41400640 1 -- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 msgr2=0x7fde2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41400640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c162740 0x7fde2800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41400640 1 -- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28003a50 msgr2=0x7fde28003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41400640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28003a50 0x7fde28003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41400640 1 -- 172.21.15.67:0/119392802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde28003520 con 0x7fde28004800 2024-04-17T07:30:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.065+0000 7fde41e00640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28003a50 0x7fde28003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.066+0000 7fde41400640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde28004800 0x7fde28004c80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00ba30 tx=0x7fde28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7a474621d377dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.066+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c036020 con 0x7fde28004800 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.066+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c032640 con 0x7fde28004800 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.066+0000 7fde40a00640 1 -- 172.21.15.67:0/119392802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00b600 con 0x7fde28004800 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 -- 172.21.15.67:0/119392802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde28004800 msgr2=0x7fde28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 --2- 172.21.15.67:0/119392802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde28004800 0x7fde28004c80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00ba30 tx=0x7fde28005830 comp rx=0 tx=0).stop 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde42800640 1 -- 172.21.15.67:0/119392802 reap_dead start 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 -- 172.21.15.67:0/119392802 shutdown_connections 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 -- 172.21.15.67:0/119392802 >> 172.21.15.67:0/119392802 conn(0x7fde3c06bef0 msgr2=0x7fde3c0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 -- 172.21.15.67:0/119392802 shutdown_connections 2024-04-17T07:30:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.067+0000 7fde43200640 1 -- 172.21.15.67:0/119392802 wait complete. 2024-04-17T07:30:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.069+0000 7fde43200640 1 Processor -- start 2024-04-17T07:30:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.069+0000 7fde43200640 1 -- start start 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28004800 0x7fde3c1462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c0cb010 0x7fde3c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c0c9a40 con 0x7fde3c15d8a0 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c0b0050 con 0x7fde28004800 2024-04-17T07:30:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde43200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c0c9bc0 con 0x7fde3c0cb010 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50714/0 (socket says 172.21.15.67:50714) 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 -- 172.21.15.67:0/1622790848 learned_addr learned my addr 172.21.15.67:0/1622790848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 -- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c0cb010 msgr2=0x7fde3c0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c0cb010 0x7fde3c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 -- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28004800 msgr2=0x7fde3c1462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28004800 0x7fde3c1462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.070+0000 7fde42800640 1 -- 172.21.15.67:0/1622790848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c15b7a0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde42800640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fde28004780 tx=0x7fde3800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38019070 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde38003b70 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde380050d0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c15ba80 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.071+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c146820 con 0x7fde3c15d8a0 2024-04-17T07:30:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.072+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3c15c090 con 0x7fde3c15d8a0 2024-04-17T07:30:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.073+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7fde3801c070 con 0x7fde3c15d8a0 2024-04-17T07:30:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.074+0000 7fde32000640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 0x7fde2006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.074+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6397+0+0 (secure 0 0 0) 0x7fde3808b2b0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.075+0000 7fde41e00640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 0x7fde2006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.076+0000 7fde41e00640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 0x7fde2006ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde34000ff0 tx=0x7fde34009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.078+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fde38058be0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.326+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fde3c0cbee0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.328 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:30:38.328+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:30:38.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.331+0000 7fde32000640 1 -- 172.21.15.67:0/1622790848 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7fde3c0cbee0 con 0x7fde3c15d8a0 2024-04-17T07:30:38.331 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.335+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 msgr2=0x7fde2006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.335+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 0x7fde2006ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde34000ff0 tx=0x7fde34009040 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.335+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 msgr2=0x7fde3c15dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.335+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fde28004780 tx=0x7fde3800dc00 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 shutdown_connections 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fde2006c9b0 0x7fde2006ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d8a0 0x7fde3c15dd20 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde3c0cb010 0x7fde3c0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 --2- 172.21.15.67:0/1622790848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde28004800 0x7fde3c1462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 >> 172.21.15.67:0/1622790848 conn(0x7fde3c06bef0 msgr2=0x7fde3c161950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:38.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 shutdown_connections 2024-04-17T07:30:38.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.336+0000 7fde43200640 1 -- 172.21.15.67:0/1622790848 wait complete. 2024-04-17T07:30:38.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:30:38.355 INFO:tasks.ceph.mds.c.smithi067.stderr: -3007> 2024-04-17T07:30:17.545+0000 7fdac7000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:30:38.417 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:30:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.546+0000 7f1b8ae00640 1 Processor -- start 2024-04-17T07:30:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.546+0000 7f1b8ae00640 1 -- start start 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8ae00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1b8415ecb0 0x7f1b840d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b840cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b840d00b0 con 0x7f1b840cf8e0 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8ae00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1b840d0c30 con 0x7f1b8415ecb0 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b840d0390 con 0x7f1b840d3210 2024-04-17T07:30:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8a400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b840cf8e0 0x7f1b840d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46672/0 (socket says 172.21.15.67:46672) 2024-04-17T07:30:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.548+0000 7f1b8a400640 1 -- 172.21.15.67:0/1777677556 learned_addr learned my addr 172.21.15.67:0/1777677556 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b89000640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b840cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b89a00640 1 --2- 172.21.15.67:0/1777677556 >> v2:172.21.15.187:3300/0 conn(0x7f1b8415ecb0 0x7f1b840d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790036775 0 0) 0x7f1b840d00b0 con 0x7f1b840cf8e0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b6c003520 con 0x7f1b840cf8e0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4067853536 0 0) 0x7f1b6c003520 con 0x7f1b840cf8e0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 msgr2=0x7f1b840cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b840cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 >> v2:172.21.15.187:3300/0 conn(0x7f1b8415ecb0 msgr2=0x7f1b840d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 --2- 172.21.15.67:0/1777677556 >> v2:172.21.15.187:3300/0 conn(0x7f1b8415ecb0 0x7f1b840d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b840d0670 con 0x7f1b840cf8e0 2024-04-17T07:30:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1b7c0019f0 con 0x7f1b840cf8e0 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 >> v1:172.21.15.67:6789/0 conn(0x7f1b840cf8e0 legacy=0x7f1b840d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b6c0039b0 0x7f1b6c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b6c004d40 0x7f1b6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b840d0c30 con 0x7f1b6c004d40 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b840d0390 con 0x7f1b6c0039b0 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.549+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6c00ba40 con 0x7f1b840d3210 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b6c004d40 0x7f1b6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b89000640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b89a00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b6c0039b0 0x7f1b6c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 -- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 msgr2=0x7f1b6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b840d3210 0x7f1b6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 -- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b6c0039b0 msgr2=0x7f1b6c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b6c0039b0 0x7f1b6c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.550+0000 7f1b8a400640 1 -- 172.21.15.67:0/1777677556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c003520 con 0x7f1b6c004d40 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.552+0000 7f1b8a400640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b6c004d40 0x7f1b6c0051c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00a7b0 tx=0x7f1b6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5091a0124f031617 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.552+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00a3d0 con 0x7f1b6c004d40 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.552+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7c034650 con 0x7f1b6c004d40 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.552+0000 7f1b83e00640 1 -- 172.21.15.67:0/1777677556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c034a40 con 0x7f1b6c004d40 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.553+0000 7f1b8ae00640 1 -- 172.21.15.67:0/1777677556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b6c004d40 msgr2=0x7f1b6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.553+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/1777677556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b6c004d40 0x7f1b6c0051c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00a7b0 tx=0x7f1b6c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.553+0000 7f1b8a400640 1 -- 172.21.15.67:0/1777677556 reap_dead start 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.553+0000 7f1b8ae00640 1 -- 172.21.15.67:0/1777677556 shutdown_connections 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.553+0000 7f1b8ae00640 1 -- 172.21.15.67:0/1777677556 >> 172.21.15.67:0/1777677556 conn(0x7f1b8408bff0 msgr2=0x7f1b840d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.554+0000 7f1b8ae00640 1 -- 172.21.15.67:0/1777677556 shutdown_connections 2024-04-17T07:30:38.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.554+0000 7f1b8ae00640 1 -- 172.21.15.67:0/1777677556 wait complete. 2024-04-17T07:30:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.556+0000 7f1b8ae00640 1 Processor -- start 2024-04-17T07:30:38.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.557+0000 7f1b8ae00640 1 -- start start 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.557+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 0x7f1b8415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.557+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b840cbc80 0x7f1b840cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8415f090 con 0x7f1b840cc640 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b840cfd70 con 0x7f1b840cbc80 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b8415f210 con 0x7f1b6c004d40 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50754/0 (socket says 172.21.15.67:50754) 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 -- 172.21.15.67:0/2418451830 learned_addr learned my addr 172.21.15.67:0/2418451830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b89a00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 0x7f1b8415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b89000640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b840cbc80 0x7f1b840cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 -- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 msgr2=0x7f1b8415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 0x7f1b8415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 -- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b840cbc80 msgr2=0x7f1b840cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b840cbc80 0x7f1b840cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b8a400640 1 -- 172.21.15.67:0/2418451830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b840ccac0 con 0x7f1b840cc640 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.558+0000 7f1b89a00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 0x7f1b8415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:38.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b8a400640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c005c20 tx=0x7f1b840d0280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c034a00 con 0x7f1b840cc640 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7c044070 con 0x7f1b840cc640 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c034da0 con 0x7f1b840cc640 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b840c8e40 con 0x7f1b840cc640 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.559+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b840c9400 con 0x7f1b840cc640 2024-04-17T07:30:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.561+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b84068a10 con 0x7f1b840cc640 2024-04-17T07:30:38.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.562+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 45) ==== 91086+0+0 (secure 0 0 0) 0x7f1b7c049460 con 0x7f1b840cc640 2024-04-17T07:30:38.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.564+0000 7f1b81600640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 0x7f1b6406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:38.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.564+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6483+0+0 (secure 0 0 0) 0x7f1b7c0ae610 con 0x7f1b840cc640 2024-04-17T07:30:38.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.564+0000 7f1b89a00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 0x7f1b6406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:38.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.565+0000 7f1b89a00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 0x7f1b6406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009a00 tx=0x7f1b78008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:38.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.569+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1b7c07c530 con 0x7f1b840cc640 2024-04-17T07:30:38.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:38.810+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1b840b5a70 con 0x7f1b840cc640 2024-04-17T07:30:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.115+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f1b7c0784a0 con 0x7f1b840cc640 2024-04-17T07:30:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.342+0000 7f1b81600640 1 -- 172.21.15.67:0/2418451830 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f1b7c0a7530 con 0x7f1b840cc640 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.346+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 msgr2=0x7f1b6406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.346+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 0x7f1b6406ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009a00 tx=0x7f1b78008040 comp rx=0 tx=0).stop 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.346+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 msgr2=0x7f1b840b8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.346+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c005c20 tx=0x7f1b840d0280 comp rx=0 tx=0).stop 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 shutdown_connections 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1b6406c9b0 0x7f1b6406ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b840cc640 0x7f1b840b8910 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b840cbc80 0x7f1b840cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 --2- 172.21.15.67:0/2418451830 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b6c004d40 0x7f1b8415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 >> 172.21.15.67:0/2418451830 conn(0x7f1b8408bff0 msgr2=0x7f1b840b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 shutdown_connections 2024-04-17T07:30:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.347+0000 7f1b8ae00640 1 -- 172.21.15.67:0/2418451830 wait complete. 2024-04-17T07:30:39.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:30:39.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.583+0000 7fa7b8000640 1 Processor -- start 2024-04-17T07:30:39.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.584+0000 7fa7b8000640 1 -- start start 2024-04-17T07:30:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.585+0000 7fa7b8000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7b00d55e0 0x7fa7b00cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.585+0000 7fa7b8000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 0x7fa7b00cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.585+0000 7fa7b8000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b00b3920 con 0x7fa7b0162110 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.585+0000 7fa7b8000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b00d2940 con 0x7fa7b00d55e0 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.585+0000 7fa7b8000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b00d2610 con 0x7fa7b01624f0 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b5600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7b0162110 0x7fa7b00ba2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46686/0 (socket says 172.21.15.67:46686) 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7b00d55e0 0x7fa7b00cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b5600640 1 -- 172.21.15.67:0/228704567 learned_addr learned my addr 172.21.15.67:0/228704567 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b6000640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 0x7fa7b00cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b4c00640 1 -- 172.21.15.67:0/228704567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 msgr2=0x7fa7b00cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b4c00640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 0x7fa7b00cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b4c00640 1 -- 172.21.15.67:0/228704567 >> v1:172.21.15.67:6789/0 conn(0x7fa7b0162110 legacy=0x7fa7b00ba2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b4c00640 1 -- 172.21.15.67:0/228704567 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b00b3bc0 con 0x7fa7b00d55e0 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7afe00640 1 -- 172.21.15.67:0/228704567 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666474609 0 0) 0x7fa7b00b3920 con 0x7fa7b0162110 2024-04-17T07:30:39.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.586+0000 7fa7b6000640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 0x7fa7b00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.587+0000 7fa7b4c00640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b00d55e0 0x7fa7b00cf620 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0097a0 tx=0x7fa79c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256cd7ed83f59cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:39.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7afe00640 1 -- 172.21.15.67:0/228704567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c031780 con 0x7fa7b00d55e0 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7afe00640 1 -- 172.21.15.67:0/228704567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79c037070 con 0x7fa7b00d55e0 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7afe00640 1 -- 172.21.15.67:0/228704567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c031b40 con 0x7fa7b00d55e0 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 -- 172.21.15.67:0/228704567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b00d55e0 msgr2=0x7fa7b00cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b00d55e0 0x7fa7b00cf620 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0097a0 tx=0x7fa79c02b820 comp rx=0 tx=0).stop 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 -- 172.21.15.67:0/228704567 shutdown_connections 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b01624f0 0x7fa7b00cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 --2- 172.21.15.67:0/228704567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b00d55e0 0x7fa7b00cf620 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 -- 172.21.15.67:0/228704567 >> 172.21.15.67:0/228704567 conn(0x7fa7b00d1a50 msgr2=0x7fa7b00d49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 -- 172.21.15.67:0/228704567 shutdown_connections 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.588+0000 7fa7b8000640 1 -- 172.21.15.67:0/228704567 wait complete. 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.590+0000 7fa7b8000640 1 Processor -- start 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.590+0000 7fa7b8000640 1 -- start start 2024-04-17T07:30:39.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 0x7fa7b0156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b0162110 0x7fa7b00ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00d2f50 con 0x7fa7b01624f0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00d2dd0 con 0x7fa7b00d55e0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.591+0000 7fa7b8000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b00d30d0 con 0x7fa7b0162110 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50774/0 (socket says 172.21.15.67:50774) 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 -- 172.21.15.67:0/504793240 learned_addr learned my addr 172.21.15.67:0/504793240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b4c00640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b0162110 0x7fa7b00ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b5600640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 0x7fa7b0156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 -- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b0162110 msgr2=0x7fa7b00ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b0162110 0x7fa7b00ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 -- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 msgr2=0x7fa7b0156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 0x7fa7b0156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b6000640 1 -- 172.21.15.67:0/504793240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b00ca930 con 0x7fa7b01624f0 2024-04-17T07:30:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.592+0000 7fa7b5600640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 0x7fa7b0156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7b6000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800cd50 tx=0x7fa7a800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a800f680 con 0x7fa7b01624f0 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a800f840 con 0x7fa7b01624f0 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b00c7580 con 0x7fa7b01624f0 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a800fc30 con 0x7fa7b01624f0 2024-04-17T07:30:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.593+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b00c7b40 con 0x7fa7b01624f0 2024-04-17T07:30:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.594+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b006e0c0 con 0x7fa7b01624f0 2024-04-17T07:30:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.595+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fa7a80213f0 con 0x7fa7b01624f0 2024-04-17T07:30:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.596+0000 7fa7ad600640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 0x7fa79006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.596+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6483+0+0 (secure 0 0 0) 0x7fa7a808cf70 con 0x7fa7b01624f0 2024-04-17T07:30:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.597+0000 7fa7b5600640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 0x7fa79006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.598+0000 7fa7b5600640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 0x7fa79006eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0007fd0 tx=0x7fa7a0009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.601+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa7b00ca930 con 0x7fa7b01624f0 2024-04-17T07:30:39.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:39.803+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa7b006db80 con 0x7fa7b01624f0 2024-04-17T07:30:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.346+0000 7fa7ad600640 1 -- 172.21.15.67:0/504793240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) ==== 181+0+0 (secure 0 0 0) 0x7fa7a8014030 con 0x7fa7b01624f0 2024-04-17T07:30:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:30:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 msgr2=0x7fa79006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 0x7fa79006eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0007fd0 tx=0x7fa7a0009040 comp rx=0 tx=0).stop 2024-04-17T07:30:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 msgr2=0x7fa7b01f8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800cd50 tx=0x7fa7a800a650 comp rx=0 tx=0).stop 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 shutdown_connections 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa79006ca10 0x7fa79006eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7b01624f0 0x7fa7b01f8d10 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7b0162110 0x7fa7b00ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 --2- 172.21.15.67:0/504793240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b00d55e0 0x7fa7b0156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.350+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 >> 172.21.15.67:0/504793240 conn(0x7fa7b00d1a50 msgr2=0x7fa7b00b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:40.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.351+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 shutdown_connections 2024-04-17T07:30:40.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.351+0000 7fa7b8000640 1 -- 172.21.15.67:0/504793240 wait complete. 2024-04-17T07:30:40.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:30:40.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.598+0000 7f7957e00640 1 Processor -- start 2024-04-17T07:30:40.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.598+0000 7f7957e00640 1 -- start start 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.599+0000 7f7957e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 0x7f7958155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.599+0000 7f7957e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79580b4070 con 0x7f79580b3800 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7957e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795815f890 con 0x7f79580cfc80 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7957e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79581629b0 con 0x7f79580b3be0 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7956a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79580b3800 0x7f795815f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46690/0 (socket says 172.21.15.67:46690) 2024-04-17T07:30:40.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7956a00640 1 -- 172.21.15.67:0/3055414742 learned_addr learned my addr 172.21.15.67:0/3055414742 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7957400640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 0x7f7958155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2273438820 0 0) 0x7f79580b4070 con 0x7f79580b3800 2024-04-17T07:30:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.600+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f793c003520 con 0x7f79580b3800 2024-04-17T07:30:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1052085676 0 0) 0x7f793c003520 con 0x7f79580b3800 2024-04-17T07:30:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 msgr2=0x7f7958155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 0x7f7958155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 >> v1:172.21.15.187:6789/0 conn(0x7f79580cfc80 legacy=0x7f7958162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795815fb70 con 0x7f79580b3800 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f79480019f0 con 0x7f79580b3800 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 >> v1:172.21.15.67:6789/0 conn(0x7f79580b3800 legacy=0x7f795815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f793c0039e0 0x7f793c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f793c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 0x7f793c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79581629b0 con 0x7f793c004800 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f793c00bb60 con 0x7f793c0039e0 2024-04-17T07:30:40.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f793c00bd60 con 0x7f79580b3be0 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.601+0000 7f7956000640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f793c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956000640 1 -- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 msgr2=0x7f793c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956000640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79580b3be0 0x7f793c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956000640 1 -- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f793c0039e0 msgr2=0x7f793c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956a00640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f793c0039e0 0x7f793c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956000640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f793c0039e0 0x7f793c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956000640 1 -- 172.21.15.67:0/3055414742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793c003520 con 0x7f793c004800 2024-04-17T07:30:40.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.602+0000 7f7956a00640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f793c0039e0 0x7f793c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:40.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.603+0000 7f7956000640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f793c004c80 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7940001720 tx=0x7f793c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c81794dbe81c281 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.603+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940001370 con 0x7f793c004800 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.603+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79400052d0 con 0x7f793c004800 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.603+0000 7f7955600640 1 -- 172.21.15.67:0/3055414742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940003830 con 0x7f793c004800 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 -- 172.21.15.67:0/3055414742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 msgr2=0x7f793c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 --2- 172.21.15.67:0/3055414742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f793c004c80 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7940001720 tx=0x7f793c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957400640 1 -- 172.21.15.67:0/3055414742 reap_dead start 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 -- 172.21.15.67:0/3055414742 shutdown_connections 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 -- 172.21.15.67:0/3055414742 >> 172.21.15.67:0/3055414742 conn(0x7f7958067600 msgr2=0x7f79580679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 -- 172.21.15.67:0/3055414742 shutdown_connections 2024-04-17T07:30:40.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.604+0000 7f7957e00640 1 -- 172.21.15.67:0/3055414742 wait complete. 2024-04-17T07:30:40.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.606+0000 7f7957e00640 1 Processor -- start 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.606+0000 7f7957e00640 1 -- start start 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580cba70 0x7f79580b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79581e93e0 0x7f79581f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7958162640 con 0x7f793c004800 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7958163550 con 0x7f79580cba70 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7957e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7958155560 con 0x7f79581e93e0 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7956a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.607+0000 7f7956a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50782/0 (socket says 172.21.15.67:50782) 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 -- 172.21.15.67:0/1076105043 learned_addr learned my addr 172.21.15.67:0/1076105043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 -- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79581e93e0 msgr2=0x7f79581f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79581e93e0 0x7f79581f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 -- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580cba70 msgr2=0x7f79580b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580cba70 0x7f79580b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 -- 172.21.15.67:0/1076105043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79581e8620 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f7956a00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7948002190 tx=0x7f794800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.608+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948019070 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.609+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7948004d60 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.609+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79480072d0 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.609+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79581e8900 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.609+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7958153100 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.610+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79580d0250 con 0x7f793c004800 2024-04-17T07:30:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.611+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f7948007470 con 0x7f793c004800 2024-04-17T07:30:40.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.612+0000 7f794f400640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 0x7f793406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:40.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.612+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6045+0+0 (secure 0 0 0) 0x7f794808c150 con 0x7f793c004800 2024-04-17T07:30:40.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.613+0000 7f7956000640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 0x7f793406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:40.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.615+0000 7f7956000640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 0x7f793406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7940033000 tx=0x7f794002f000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:40.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.617+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f79580d0250 con 0x7f793c004800 2024-04-17T07:30:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:40.822+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7958068a10 con 0x7f793c004800 2024-04-17T07:30:41.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.349+0000 7f794f400640 1 -- 172.21.15.67:0/1076105043 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) ==== 169+0+0 (secure 0 0 0) 0x7f794805a170 con 0x7f793c004800 2024-04-17T07:30:41.350 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:30:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 msgr2=0x7f793406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 0x7f793406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7940033000 tx=0x7f794002f000 comp rx=0 tx=0).stop 2024-04-17T07:30:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 msgr2=0x7f79580b83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7948002190 tx=0x7f794800b560 comp rx=0 tx=0).stop 2024-04-17T07:30:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 shutdown_connections 2024-04-17T07:30:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f793406ca10 0x7f793406eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79581e93e0 0x7f79581f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580cba70 0x7f79580b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.353+0000 7f7957e00640 1 --2- 172.21.15.67:0/1076105043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793c004800 0x7f79580b83d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.354+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 >> 172.21.15.67:0/1076105043 conn(0x7f7958067600 msgr2=0x7f7958160180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.354+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 shutdown_connections 2024-04-17T07:30:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.354+0000 7f7957e00640 1 -- 172.21.15.67:0/1076105043 wait complete. 2024-04-17T07:30:41.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:30:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.611+0000 7f0de2600640 1 Processor -- start 2024-04-17T07:30:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.611+0000 7f0de2600640 1 -- start start 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0de2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0ddc0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0de2600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ddc071f60 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0de2600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ddc072240 con 0x7f0ddc15b860 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0de2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc071d10 con 0x7f0ddc0d0b30 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0ddbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0ddc0ce4d0 0x7f0ddc0b44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46696/0 (socket says 172.21.15.67:46696) 2024-04-17T07:30:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.613+0000 7f0ddbe00640 1 -- 172.21.15.67:0/2870039759 learned_addr learned my addr 172.21.15.67:0/2870039759 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084898330 0 0) 0x7f0ddc071f60 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0ddc0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc0003520 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 95687167 0 0) 0x7f0dc0003520 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 msgr2=0x7f0ddc0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0ddc0b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 >> v1:172.21.15.187:6789/0 conn(0x7f0ddc15b860 legacy=0x7f0ddc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc072520 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0dd40019f0 con 0x7f0ddc0ce4d0 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 >> v1:172.21.15.67:6789/0 conn(0x7f0ddc0ce4d0 legacy=0x7f0ddc0b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc00039e0 0x7f0dc0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0dc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc071d10 con 0x7f0dc00039e0 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc000baa0 con 0x7f0dc0004800 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.614+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc000bca0 con 0x7f0ddc0d0b30 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc00039e0 0x7f0dc0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddbe00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0dc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 -- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 msgr2=0x7f0dc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0dc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 -- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0004800 msgr2=0x7f0dc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddb400640 1 -- 172.21.15.67:0/2870039759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0003520 con 0x7f0dc00039e0 2024-04-17T07:30:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.615+0000 7f0ddbe00640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ddc0d0b30 0x7f0dc00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.616+0000 7f0ddb400640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc00039e0 0x7f0dc0003e20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc001720 tx=0x7f0dc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbfea3b84599a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.616+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc001370 con 0x7f0dc00039e0 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dcc0052d0 con 0x7f0dc00039e0 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0ddaa00640 1 -- 172.21.15.67:0/2870039759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc003830 con 0x7f0dc00039e0 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 -- 172.21.15.67:0/2870039759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc00039e0 msgr2=0x7f0dc0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 --2- 172.21.15.67:0/2870039759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc00039e0 0x7f0dc0003e20 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc001720 tx=0x7f0dc0005830 comp rx=0 tx=0).stop 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0ddbe00640 1 -- 172.21.15.67:0/2870039759 reap_dead start 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 -- 172.21.15.67:0/2870039759 shutdown_connections 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 -- 172.21.15.67:0/2870039759 >> 172.21.15.67:0/2870039759 conn(0x7f0ddc067ee0 msgr2=0x7f0ddc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 -- 172.21.15.67:0/2870039759 shutdown_connections 2024-04-17T07:30:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.617+0000 7f0de2600640 1 -- 172.21.15.67:0/2870039759 wait complete. 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.619+0000 7f0de2600640 1 Processor -- start 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.619+0000 7f0de2600640 1 -- start start 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 0x7f0ddc14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ddc138dc0 0x7f0ddc139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc15bc40 con 0x7f0ddc136f40 2024-04-17T07:30:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc072ae0 con 0x7f0ddc138dc0 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0de2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc06f380 con 0x7f0dc00039e0 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0ddbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0ddbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50806/0 (socket says 172.21.15.67:50806) 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.620+0000 7f0ddbe00640 1 -- 172.21.15.67:0/1447424112 learned_addr learned my addr 172.21.15.67:0/1447424112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddb400640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 0x7f0ddc14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ddc138dc0 0x7f0ddc139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 -- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 msgr2=0x7f0ddc14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 0x7f0ddc14dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 -- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ddc138dc0 msgr2=0x7f0ddc139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ddc138dc0 0x7f0ddc139240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 -- 172.21.15.67:0/1447424112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc14e220 con 0x7f0ddc136f40 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddb400640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 0x7f0ddc14dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0ddbe00640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0ce210 tx=0x7f0dd400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:41.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.621+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400bb20 con 0x7f0ddc136f40 2024-04-17T07:30:41.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.622+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd400bce0 con 0x7f0ddc136f40 2024-04-17T07:30:41.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.622+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd40072b0 con 0x7f0ddc136f40 2024-04-17T07:30:41.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.622+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ddc137c70 con 0x7f0ddc136f40 2024-04-17T07:30:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.622+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ddc138230 con 0x7f0ddc136f40 2024-04-17T07:30:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.623+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ddc068ca0 con 0x7f0ddc136f40 2024-04-17T07:30:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.624+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f0dd401b020 con 0x7f0ddc136f40 2024-04-17T07:30:41.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.625+0000 7f0dd3400640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 0x7f0db806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:41.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.625+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f0dd408bff0 con 0x7f0ddc136f40 2024-04-17T07:30:41.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.626+0000 7f0ddb400640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 0x7f0db806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:41.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.627+0000 7f0ddb400640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 0x7f0db806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc033000 tx=0x7f0dcc009030 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:41.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.630+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0dd405a210 con 0x7f0ddc136f40 2024-04-17T07:30:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.836+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f0ddc0b4a40 con 0x7f0ddc136f40 2024-04-17T07:30:41.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.842+0000 7f0dd3400640 1 -- 172.21.15.67:0/1447424112 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) ==== 115+0+0 (secure 0 0 0) 0x7f0dd4059b10 con 0x7f0ddc136f40 2024-04-17T07:30:41.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 msgr2=0x7f0db806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 0x7f0db806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc033000 tx=0x7f0dcc009030 comp rx=0 tx=0).stop 2024-04-17T07:30:41.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 msgr2=0x7f0ddc1373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0ce210 tx=0x7f0dd400b4a0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 shutdown_connections 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0db806ca10 0x7f0db806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ddc136f40 0x7f0ddc1373d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ddc138dc0 0x7f0ddc139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 --2- 172.21.15.67:0/1447424112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dc00039e0 0x7f0ddc14dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 >> 172.21.15.67:0/1447424112 conn(0x7f0ddc067ee0 msgr2=0x7f0ddc0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 shutdown_connections 2024-04-17T07:30:41.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:41.845+0000 7f0de2600640 1 -- 172.21.15.67:0/1447424112 wait complete. 2024-04-17T07:30:41.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:30:42.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.077+0000 7f817d400640 1 Processor -- start 2024-04-17T07:30:42.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.077+0000 7f817d400640 1 -- start start 2024-04-17T07:30:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.078+0000 7f817d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 0x7f81780d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.078+0000 7f817d400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f81780d3210 0x7f81780cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.078+0000 7f817d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81780d00b0 con 0x7f81780cf8e0 2024-04-17T07:30:42.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.078+0000 7f817d400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f81780d3670 con 0x7f81780d3210 2024-04-17T07:30:42.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.078+0000 7f817d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81780d0c30 con 0x7f817815ecb0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f817ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81780cf8e0 0x7f81780d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46710/0 (socket says 172.21.15.67:46710) 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f817ca00640 1 -- 172.21.15.67:0/3559795191 learned_addr learned my addr 172.21.15.67:0/3559795191 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f8177400640 1 --2- 172.21.15.67:0/3559795191 >> v2:172.21.15.187:3300/0 conn(0x7f81780d3210 0x7f81780cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f8177e00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 0x7f81780d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189589067 0 0) 0x7f81780d00b0 con 0x7f81780cf8e0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.079+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f815c003520 con 0x7f81780cf8e0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3040535560 0 0) 0x7f815c003520 con 0x7f81780cf8e0 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 msgr2=0x7f81780d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 0x7f81780d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 >> v2:172.21.15.187:3300/0 conn(0x7f81780d3210 msgr2=0x7f81780cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 --2- 172.21.15.67:0/3559795191 >> v2:172.21.15.187:3300/0 conn(0x7f81780d3210 0x7f81780cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81780d0390 con 0x7f81780cf8e0 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8170001690 con 0x7f81780cf8e0 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 >> v1:172.21.15.67:6789/0 conn(0x7f81780cf8e0 legacy=0x7f81780d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f815c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f815c004800 0x7f815c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 0x7f815c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81780d0c30 con 0x7f815c0039b0 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f815c00b9c0 con 0x7f815c004800 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f815c00bb90 con 0x7f817815ecb0 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.080+0000 7f8177e00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f815c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.081+0000 7f8177e00640 1 -- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 msgr2=0x7f815c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.081+0000 7f8177e00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815ecb0 0x7f815c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.081+0000 7f8177e00640 1 -- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f815c004800 msgr2=0x7f815c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.081+0000 7f8177e00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f815c004800 0x7f815c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.081+0000 7f8177e00640 1 -- 172.21.15.67:0/3559795191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815c003520 con 0x7f815c0039b0 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.082+0000 7f8177e00640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f815c003dd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f816800e170 tx=0x7f815c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9b82bcccdcdf75 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.082+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168039070 con 0x7f815c0039b0 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.082+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816802cc20 con 0x7f815c0039b0 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.082+0000 7f8176a00640 1 -- 172.21.15.67:0/3559795191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168009c10 con 0x7f815c0039b0 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 -- 172.21.15.67:0/3559795191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 msgr2=0x7f815c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 --2- 172.21.15.67:0/3559795191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f815c003dd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f816800e170 tx=0x7f815c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817ca00640 1 -- 172.21.15.67:0/3559795191 reap_dead start 2024-04-17T07:30:42.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 -- 172.21.15.67:0/3559795191 shutdown_connections 2024-04-17T07:30:42.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 -- 172.21.15.67:0/3559795191 >> 172.21.15.67:0/3559795191 conn(0x7f817808bff0 msgr2=0x7f81780d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:42.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 -- 172.21.15.67:0/3559795191 shutdown_connections 2024-04-17T07:30:42.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.083+0000 7f817d400640 1 -- 172.21.15.67:0/3559795191 wait complete. 2024-04-17T07:30:42.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.085+0000 7f817d400640 1 Processor -- start 2024-04-17T07:30:42.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.085+0000 7f817d400640 1 -- start start 2024-04-17T07:30:42.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81780cda10 0x7f817806cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815c540 0x7f817815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81780cfd50 con 0x7f815c0039b0 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81780d0950 con 0x7f81780cda10 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f817d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f817815f090 con 0x7f817815c540 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50828/0 (socket says 172.21.15.67:50828) 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 -- 172.21.15.67:0/3720889522 learned_addr learned my addr 172.21.15.67:0/3720889522 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 -- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815c540 msgr2=0x7f817815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815c540 0x7f817815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.086+0000 7f8177e00640 1 -- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81780cda10 msgr2=0x7f817806cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f8177e00640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81780cda10 0x7f817806cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f8177e00640 1 -- 172.21.15.67:0/3720889522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817815cef0 con 0x7f815c0039b0 2024-04-17T07:30:42.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f8177e00640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8168009f90 tx=0x7f8168009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168039050 con 0x7f815c0039b0 2024-04-17T07:30:42.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8168045070 con 0x7f815c0039b0 2024-04-17T07:30:42.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81781f4a30 con 0x7f815c0039b0 2024-04-17T07:30:42.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168009c10 con 0x7f815c0039b0 2024-04-17T07:30:42.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.087+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81781f4ff0 con 0x7f815c0039b0 2024-04-17T07:30:42.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.088+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8178068a10 con 0x7f815c0039b0 2024-04-17T07:30:42.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.089+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f8168005d10 con 0x7f815c0039b0 2024-04-17T07:30:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.090+0000 7f816fe00640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 0x7f815406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.090+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f81680ae530 con 0x7f815c0039b0 2024-04-17T07:30:42.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.091+0000 7f8177400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 0x7f815406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.092+0000 7f8177400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 0x7f815406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81600093d0 tx=0x7f8160009300 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.095+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f816807c800 con 0x7f815c0039b0 2024-04-17T07:30:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.299+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f81780b5a70 con 0x7f815c0039b0 2024-04-17T07:30:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.306+0000 7f816fe00640 1 -- 172.21.15.67:0/3720889522 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v32) ==== 104+0+0 (secure 0 0 0) 0x7f816807c100 con 0x7f815c0039b0 2024-04-17T07:30:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 msgr2=0x7f815406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 0x7f815406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81600093d0 tx=0x7f8160009300 comp rx=0 tx=0).stop 2024-04-17T07:30:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 msgr2=0x7f81781f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8168009f90 tx=0x7f8168009750 comp rx=0 tx=0).stop 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 shutdown_connections 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f815406ca10 0x7f815406eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f817815c540 0x7f817815c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81780cda10 0x7f817806cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 --2- 172.21.15.67:0/3720889522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815c0039b0 0x7f81781f4150 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.309+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 >> 172.21.15.67:0/3720889522 conn(0x7f817808bff0 msgr2=0x7f817806d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:42.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.310+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 shutdown_connections 2024-04-17T07:30:42.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.310+0000 7f817d400640 1 -- 172.21.15.67:0/3720889522 wait complete. 2024-04-17T07:30:42.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:30:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.563+0000 7f1aa8a00640 1 Processor -- start 2024-04-17T07:30:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.564+0000 7f1aa8a00640 1 -- start start 2024-04-17T07:30:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.565+0000 7f1aa8a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1aa415ecb0 0x7f1aa40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.565+0000 7f1aa8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 0x7f1aa40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.565+0000 7f1aa8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa40d00b0 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.565+0000 7f1aa8a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa40d0c30 con 0x7f1aa415ecb0 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.565+0000 7f1aa8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa40d0390 con 0x7f1aa40d3210 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1aa40cf8e0 0x7f1aa40d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46712/0 (socket says 172.21.15.67:46712) 2024-04-17T07:30:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa3e00640 1 -- 172.21.15.67:0/258792761 learned_addr learned my addr 172.21.15.67:0/258792761 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> v2:172.21.15.187:3300/0 conn(0x7f1aa415ecb0 0x7f1aa40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa2a00640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 0x7f1aa40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803739399 0 0) 0x7f1aa40d00b0 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a88003520 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.566+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3377323813 0 0) 0x7f1a88003520 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 msgr2=0x7f1aa40cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 0x7f1aa40cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 >> v2:172.21.15.187:3300/0 conn(0x7f1aa415ecb0 msgr2=0x7f1aa40d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 --2- 172.21.15.67:0/258792761 >> v2:172.21.15.187:3300/0 conn(0x7f1aa415ecb0 0x7f1aa40d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa40d0670 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1a9c001670 con 0x7f1aa40cf8e0 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> v2:172.21.15.187:3300/0 conn(0x7f1aa415ecb0 0x7f1aa40d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 >> v1:172.21.15.67:6789/0 conn(0x7f1aa40cf8e0 legacy=0x7f1aa40d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a880039b0 0x7f1a88003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a88004800 0x7f1a88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 0x7f1a880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa40d0c30 con 0x7f1a880039b0 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa40d0390 con 0x7f1a88004800 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1a8800bb70 con 0x7f1aa40d3210 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a880039b0 0x7f1a88003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 -- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 msgr2=0x7f1a880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa40d3210 0x7f1a880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 -- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a88004800 msgr2=0x7f1a88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a88004800 0x7f1a88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.567+0000 7f1aa3400640 1 -- 172.21.15.67:0/258792761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a88003520 con 0x7f1a880039b0 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa3400640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a880039b0 0x7f1a88003df0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1a940073e0 tx=0x7f1a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be70c61641f9968a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a94039070 con 0x7f1a880039b0 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9402cdc0 con 0x7f1a880039b0 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa2000640 1 -- 172.21.15.67:0/258792761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a94009c80 con 0x7f1a880039b0 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa8a00640 1 -- 172.21.15.67:0/258792761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a880039b0 msgr2=0x7f1a88003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/258792761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a880039b0 0x7f1a88003df0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1a940073e0 tx=0x7f1a88005830 comp rx=0 tx=0).stop 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.569+0000 7f1aa3e00640 1 -- 172.21.15.67:0/258792761 reap_dead start 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.570+0000 7f1aa8a00640 1 -- 172.21.15.67:0/258792761 shutdown_connections 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.570+0000 7f1aa8a00640 1 -- 172.21.15.67:0/258792761 >> 172.21.15.67:0/258792761 conn(0x7f1aa408bff0 msgr2=0x7f1aa40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.570+0000 7f1aa8a00640 1 -- 172.21.15.67:0/258792761 shutdown_connections 2024-04-17T07:30:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.570+0000 7f1aa8a00640 1 -- 172.21.15.67:0/258792761 wait complete. 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.571+0000 7f1aa8a00640 1 Processor -- start 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.571+0000 7f1aa8a00640 1 -- start start 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a880039b0 0x7f1aa41607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa41f3f00 0x7f1aa4161b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa415f090 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa40cfd70 con 0x7f1a880039b0 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.572+0000 7f1aa8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa415f210 con 0x7f1aa41f3f00 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50848/0 (socket says 172.21.15.67:50848) 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 -- 172.21.15.67:0/50498724 learned_addr learned my addr 172.21.15.67:0/50498724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa2a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa41f3f00 0x7f1aa4161b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3400640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a880039b0 0x7f1aa41607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 -- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa41f3f00 msgr2=0x7f1aa4161b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa41f3f00 0x7f1aa4161b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 -- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a880039b0 msgr2=0x7f1aa41607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a880039b0 0x7f1aa41607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.573+0000 7f1aa3e00640 1 -- 172.21.15.67:0/50498724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa40ccec0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1aa3e00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4069630 tx=0x7f1a9c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007440 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9c0075e0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c004a20 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa40cd1a0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.574+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa41542b0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.575+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa4068a10 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.576+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f1a9c004bc0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.577+0000 7f1a9b400640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 0x7f1a8006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.577+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f1a9c08b300 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.578+0000 7f1aa3400640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 0x7f1a8006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.579+0000 7f1aa3400640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 0x7f1a8006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1a94036000 tx=0x7f1a94030000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.582+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1a9c0595d0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.786+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f1aa40b5a70 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.793+0000 7f1a9b400640 1 -- 172.21.15.67:0/50498724 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v33) ==== 120+0+0 (secure 0 0 0) 0x7f1a9c058ed0 con 0x7f1aa40cd8a0 2024-04-17T07:30:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.796+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 msgr2=0x7f1a8006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.796+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 0x7f1a8006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1a94036000 tx=0x7f1a94030000 comp rx=0 tx=0).stop 2024-04-17T07:30:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.796+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 msgr2=0x7f1aa40cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.796+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4069630 tx=0x7f1a9c00c960 comp rx=0 tx=0).stop 2024-04-17T07:30:42.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 shutdown_connections 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1a8006ca40 0x7f1a8006ef00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa40cd8a0 0x7f1aa40cdd20 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa41f3f00 0x7f1aa4161b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 --2- 172.21.15.67:0/50498724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a880039b0 0x7f1aa41607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 >> 172.21.15.67:0/50498724 conn(0x7f1aa408bff0 msgr2=0x7f1aa406d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 shutdown_connections 2024-04-17T07:30:42.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:42.797+0000 7f1aa8a00640 1 -- 172.21.15.67:0/50498724 wait complete. 2024-04-17T07:30:42.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-04-17T07:30:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.038+0000 7f7f29a00640 1 Processor -- start 2024-04-17T07:30:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.038+0000 7f7f29a00640 1 -- start start 2024-04-17T07:30:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.040+0000 7f7f29a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 0x7f7f240b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.040+0000 7f7f29a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 0x7f7f240cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.040+0000 7f7f29a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f240cfaf0 con 0x7f7f2415b690 2024-04-17T07:30:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.040+0000 7f7f29a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7f2415c460 con 0x7f7f240cce90 2024-04-17T07:30:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.040+0000 7f7f29a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f2415c230 con 0x7f7f240cf710 2024-04-17T07:30:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.041+0000 7f7f22000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 0x7f7f240cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.041+0000 7f7f23400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f7f2415b690 0x7f7f2415bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:51846/0 (socket says 172.21.15.67:51846) 2024-04-17T07:30:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.041+0000 7f7f23400640 1 -- 172.21.15.67:0/140491088 learned_addr learned my addr 172.21.15.67:0/140491088 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.041+0000 7f7f22a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 0x7f7f240b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.042+0000 7f7f22a00640 1 -- 172.21.15.67:0/140491088 >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 msgr2=0x7f7f240cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.042+0000 7f7f22a00640 1 --2- 172.21.15.67:0/140491088 >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 0x7f7f240cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.042+0000 7f7f22a00640 1 -- 172.21.15.67:0/140491088 >> v1:172.21.15.187:6789/0 conn(0x7f7f2415b690 legacy=0x7f7f2415bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.042+0000 7f7f22a00640 1 -- 172.21.15.67:0/140491088 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f240ce950 con 0x7f7f240cf710 2024-04-17T07:30:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.042+0000 7f7f22000640 1 --2- 172.21.15.67:0/140491088 >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 0x7f7f240cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:30:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.043+0000 7f7f22a00640 1 --2- 172.21.15.67:0/140491088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 0x7f7f240b27c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097a0 tx=0x7f7f1402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=250edcccd14921ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.044+0000 7f7f21600640 1 -- 172.21.15.67:0/140491088 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f14035020 con 0x7f7f240cf710 2024-04-17T07:30:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.044+0000 7f7f21600640 1 -- 172.21.15.67:0/140491088 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f14031810 con 0x7f7f240cf710 2024-04-17T07:30:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.044+0000 7f7f21600640 1 -- 172.21.15.67:0/140491088 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f14031be0 con 0x7f7f240cf710 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.044+0000 7f7f29a00640 1 -- 172.21.15.67:0/140491088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 msgr2=0x7f7f240b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.044+0000 7f7f29a00640 1 --2- 172.21.15.67:0/140491088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 0x7f7f240b27c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097a0 tx=0x7f7f1402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 -- 172.21.15.67:0/140491088 shutdown_connections 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 --2- 172.21.15.67:0/140491088 >> v2:172.21.15.187:3300/0 conn(0x7f7f240cce90 0x7f7f240cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 --2- 172.21.15.67:0/140491088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cf710 0x7f7f240b27c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 -- 172.21.15.67:0/140491088 >> 172.21.15.67:0/140491088 conn(0x7f7f2406bf50 msgr2=0x7f7f240b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 -- 172.21.15.67:0/140491088 shutdown_connections 2024-04-17T07:30:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.045+0000 7f7f29a00640 1 -- 172.21.15.67:0/140491088 wait complete. 2024-04-17T07:30:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.047+0000 7f7f29a00640 1 Processor -- start 2024-04-17T07:30:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.047+0000 7f7f29a00640 1 -- start start 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cce90 0x7f7f2415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 0x7f7f24149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240cef10 con 0x7f7f240cf710 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7f24155960 con 0x7f7f2415b690 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f29a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f240cf1f0 con 0x7f7f240cce90 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f22000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f22000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50862/0 (socket says 172.21.15.67:50862) 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f22000640 1 -- 172.21.15.67:0/2700927865 learned_addr learned my addr 172.21.15.67:0/2700927865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.048+0000 7f7f22a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cce90 0x7f7f2415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f23400640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 0x7f7f24149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 -- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cce90 msgr2=0x7f7f2415eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cce90 0x7f7f2415eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 -- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 msgr2=0x7f7f24149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 0x7f7f24149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 -- 172.21.15.67:0/2700927865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f24148da0 con 0x7f7f240cf710 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f23400640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 0x7f7f24149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f22000640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000ebe0 tx=0x7f7f1000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10017070 con 0x7f7f240cf710 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1000cc30 con 0x7f7f240cf710 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10013470 con 0x7f7f240cf710 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f24149080 con 0x7f7f240cf710 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.049+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f24149640 con 0x7f7f240cf710 2024-04-17T07:30:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.051+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f24068b90 con 0x7f7f240cf710 2024-04-17T07:30:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.052+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f7f1001a070 con 0x7f7f240cf710 2024-04-17T07:30:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.053+0000 7f7f1aa00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 0x7f7f0406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.053+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f7f1008bc60 con 0x7f7f240cf710 2024-04-17T07:30:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.053+0000 7f7f22a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 0x7f7f0406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.055+0000 7f7f22a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 0x7f7f0406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7f14036000 tx=0x7f7f1403c040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.057+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7f10059fc0 con 0x7f7f240cf710 2024-04-17T07:30:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.343+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7f7f24075950 con 0x7f7f240cf710 2024-04-17T07:30:43.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.371+0000 7f7f1aa00640 1 -- 172.21.15.67:0/2700927865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v641) ==== 166+0+0 (secure 0 0 0) 0x7f7f100598c0 con 0x7f7f240cf710 2024-04-17T07:30:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.374+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 msgr2=0x7f7f0406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.374+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 0x7f7f0406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7f14036000 tx=0x7f7f1403c040 comp rx=0 tx=0).stop 2024-04-17T07:30:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 msgr2=0x7f7f240c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000ebe0 tx=0x7f7f1000c520 comp rx=0 tx=0).stop 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 shutdown_connections 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7f0406ca10 0x7f7f0406eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f2415b690 0x7f7f24149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f240cf710 0x7f7f240c97f0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 --2- 172.21.15.67:0/2700927865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f240cce90 0x7f7f2415eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 >> 172.21.15.67:0/2700927865 conn(0x7f7f2406bf50 msgr2=0x7f7f2406fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 shutdown_connections 2024-04-17T07:30:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.375+0000 7f7f29a00640 1 -- 172.21.15.67:0/2700927865 wait complete. 2024-04-17T07:30:43.396 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-04-17T07:30:43.396 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-04-17T07:30:43.396 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:30:43.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.616+0000 7f9dcb200640 1 Processor -- start 2024-04-17T07:30:43.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.616+0000 7f9dcb200640 1 -- start start 2024-04-17T07:30:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dcb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9dc415ec60 0x7f9dc40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dcb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc40b6b30 0x7f9dc40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dcb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc40d2d30 con 0x7f9dc40cf910 2024-04-17T07:30:43.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dcb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc40d49c0 con 0x7f9dc415ec60 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dcb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc40d3660 con 0x7f9dc40b6b30 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dc9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9dc415ec60 0x7f9dc40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dca800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9dc40cf910 0x7f9dc406cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46898/0 (socket says 172.21.15.67:46898) 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.618+0000 7f9dca800640 1 -- 172.21.15.67:0/3784674240 learned_addr learned my addr 172.21.15.67:0/3784674240 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc9400640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc40b6b30 0x7f9dc40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc9e00640 1 -- 172.21.15.67:0/3784674240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc40b6b30 msgr2=0x7f9dc40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:43.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc9e00640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc40b6b30 0x7f9dc40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc9e00640 1 -- 172.21.15.67:0/3784674240 >> v1:172.21.15.67:6789/0 conn(0x7f9dc40cf910 legacy=0x7f9dc406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc9e00640 1 -- 172.21.15.67:0/3784674240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc40d3010 con 0x7f9dc415ec60 2024-04-17T07:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.619+0000 7f9dc8a00640 1 -- 172.21.15.67:0/3784674240 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587763478 0 0) 0x7f9dc40d2d30 con 0x7f9dc40cf910 2024-04-17T07:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.620+0000 7f9dc9e00640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc415ec60 0x7f9dc40b65f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9db80097a0 tx=0x7f9db802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee1599d5e013a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.620+0000 7f9dc8a00640 1 -- 172.21.15.67:0/3784674240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8035020 con 0x7f9dc415ec60 2024-04-17T07:30:43.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dc8a00640 1 -- 172.21.15.67:0/3784674240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db8031830 con 0x7f9dc415ec60 2024-04-17T07:30:43.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dc8a00640 1 -- 172.21.15.67:0/3784674240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8031c00 con 0x7f9dc415ec60 2024-04-17T07:30:43.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 -- 172.21.15.67:0/3784674240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc415ec60 msgr2=0x7f9dc40b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc415ec60 0x7f9dc40b65f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9db80097a0 tx=0x7f9db802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 -- 172.21.15.67:0/3784674240 shutdown_connections 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc40b6b30 0x7f9dc40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 --2- 172.21.15.67:0/3784674240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc415ec60 0x7f9dc40b65f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 -- 172.21.15.67:0/3784674240 >> 172.21.15.67:0/3784674240 conn(0x7f9dc40d15f0 msgr2=0x7f9dc40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 -- 172.21.15.67:0/3784674240 shutdown_connections 2024-04-17T07:30:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.621+0000 7f9dcb200640 1 -- 172.21.15.67:0/3784674240 wait complete. 2024-04-17T07:30:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.623+0000 7f9dcb200640 1 Processor -- start 2024-04-17T07:30:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.623+0000 7f9dcb200640 1 -- start start 2024-04-17T07:30:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dc40b6b30 0x7f9dc40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc415ec60 0x7f9dc41f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc40cfda0 con 0x7f9dc40cf910 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc40b6f70 con 0x7f9dc40b6b30 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dcb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc415f040 con 0x7f9dc415ec60 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.624+0000 7f9dc9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43486/0 (socket says 172.21.15.67:43486) 2024-04-17T07:30:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 -- 172.21.15.67:0/395915846 learned_addr learned my addr 172.21.15.67:0/395915846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dca800640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc415ec60 0x7f9dc41f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9e00640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dc40b6b30 0x7f9dc40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 -- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc415ec60 msgr2=0x7f9dc41f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc415ec60 0x7f9dc41f45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 -- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dc40b6b30 msgr2=0x7f9dc40b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dc40b6b30 0x7f9dc40b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 -- 172.21.15.67:0/395915846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc41f5060 con 0x7f9dc40cf910 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.625+0000 7f9dc9400640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9db400d800 tx=0x7f9db400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.626+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db400b6c0 con 0x7f9dc40cf910 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.626+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db4019070 con 0x7f9dc40cf910 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.626+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db400ba80 con 0x7f9dc40cf910 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.626+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc41f5340 con 0x7f9dc40cf910 2024-04-17T07:30:43.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.626+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc415b780 con 0x7f9dc40cf910 2024-04-17T07:30:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.627+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc4068a10 con 0x7f9dc40cf910 2024-04-17T07:30:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.628+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f9db400bc20 con 0x7f9dc40cf910 2024-04-17T07:30:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.629+0000 7f9dbe000640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 0x7f9da806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.629+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f9db408c210 con 0x7f9dc40cf910 2024-04-17T07:30:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.630+0000 7f9dc9e00640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 0x7f9da806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.631+0000 7f9dc9e00640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 0x7f9da806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9db8036000 tx=0x7f9db8008030 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:43.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.634+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9db405a4f0 con 0x7f9dc40cf910 2024-04-17T07:30:43.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.843+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dc40c80a0 con 0x7f9dc40cf910 2024-04-17T07:30:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.844+0000 7f9dbe000640 1 -- 172.21.15.67:0/395915846 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+13989 (secure 0 0 0) 0x7f9db4059df0 con 0x7f9dc40cf910 2024-04-17T07:30:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:43.844 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":103,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:41.346961+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/381512147":"2024-04-18T07:30:38.328653+0000","172.21.15.67:6833/381512147":"2024-04-18T07:30:38.328653+0000","172.21.15.187:0/1236658145":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4268794575":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3413674157":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/1120045762":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6832/1013507418":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/4218129231":"2024-04-18T07:30:34.315141+0000","172.21.15.187:0/3033007882":"2024-04-18T07:30:34.315141+0000","172.21.15.187:6833/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:6832/3743052516":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1419379931":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/201608361":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/1684018318":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/35793766":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/3592827215":"2024-04-18T07:30:27.980642+0000","172.21.15.187:0/2290658878":"2024-04-18T07:30:27.980642+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 msgr2=0x7f9da806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 0x7f9da806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9db8036000 tx=0x7f9db8008030 comp rx=0 tx=0).stop 2024-04-17T07:30:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 msgr2=0x7f9dc415fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9db400d800 tx=0x7f9db400dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 shutdown_connections 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f9da806ca10 0x7f9da806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dc415ec60 0x7f9dc41f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc40cf910 0x7f9dc415fc30 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 --2- 172.21.15.67:0/395915846 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dc40b6b30 0x7f9dc40b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 >> 172.21.15.67:0/395915846 conn(0x7f9dc40d15f0 msgr2=0x7f9dc40b9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 shutdown_connections 2024-04-17T07:30:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:43.847+0000 7f9dcb200640 1 -- 172.21.15.67:0/395915846 wait complete. 2024-04-17T07:30:43.867 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:30:43.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:30:44.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.097+0000 7f0dcbe00640 1 Processor -- start 2024-04-17T07:30:44.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.098+0000 7f0dcbe00640 1 -- start start 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.099+0000 7f0dcbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dcc0aea80 0x7f0dcc071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.099+0000 7f0dcbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dcc0b0150 con 0x7f0dcc072380 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.099+0000 7f0dcbe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dcc145420 con 0x7f0dcc145040 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.099+0000 7f0dcbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0dcc0af0b0 con 0x7f0dcc0aea80 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.100+0000 7f0dca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dcc0aea80 0x7f0dcc071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.100+0000 7f0dca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dcc0aea80 0x7f0dcc071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43492/0 (socket says 172.21.15.67:43492) 2024-04-17T07:30:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.100+0000 7f0dca000640 1 -- 172.21.15.67:0/3686319492 learned_addr learned my addr 172.21.15.67:0/3686319492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.101+0000 7f0dca000640 1 -- 172.21.15.67:0/3686319492 >> v1:172.21.15.187:6789/0 conn(0x7f0dcc145040 legacy=0x7f0dcc162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.101+0000 7f0dca000640 1 -- 172.21.15.67:0/3686319492 >> v1:172.21.15.67:6789/0 conn(0x7f0dcc072380 legacy=0x7f0dcc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.101+0000 7f0dca000640 1 -- 172.21.15.67:0/3686319492 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc0af390 con 0x7f0dcc0aea80 2024-04-17T07:30:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.101+0000 7f0dc9600640 1 -- 172.21.15.67:0/3686319492 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564024536 0 0) 0x7f0dc4001110 con 0x7f0dcc072380 2024-04-17T07:30:44.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.102+0000 7f0dca000640 1 --2- 172.21.15.67:0/3686319492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc071e40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009870 tx=0x7f0db802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e5c4405bd9d170 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.102+0000 7f0dc9600640 1 -- 172.21.15.67:0/3686319492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8034070 con 0x7f0dcc0aea80 2024-04-17T07:30:44.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.102+0000 7f0dc9600640 1 -- 172.21.15.67:0/3686319492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db8030860 con 0x7f0dcc0aea80 2024-04-17T07:30:44.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.102+0000 7f0dc9600640 1 -- 172.21.15.67:0/3686319492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8030c30 con 0x7f0dcc0aea80 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 -- 172.21.15.67:0/3686319492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 msgr2=0x7f0dcc071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/3686319492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc071e40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009870 tx=0x7f0db802b9a0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 -- 172.21.15.67:0/3686319492 shutdown_connections 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/3686319492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc071e40 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 -- 172.21.15.67:0/3686319492 >> 172.21.15.67:0/3686319492 conn(0x7f0dcc0b23c0 msgr2=0x7f0dcc0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 -- 172.21.15.67:0/3686319492 shutdown_connections 2024-04-17T07:30:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.103+0000 7f0dcbe00640 1 -- 172.21.15.67:0/3686319492 wait complete. 2024-04-17T07:30:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.105+0000 7f0dcbe00640 1 Processor -- start 2024-04-17T07:30:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.105+0000 7f0dcbe00640 1 -- start start 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 0x7f0dcc0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 0x7f0dcc13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0afc30 con 0x7f0dcc0aea80 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0af950 con 0x7f0dcc145040 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dcbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc0cec20 con 0x7f0dcc072380 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43500/0 (socket says 172.21.15.67:43500) 2024-04-17T07:30:44.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.106+0000 7f0dca000640 1 -- 172.21.15.67:0/2528133633 learned_addr learned my addr 172.21.15.67:0/2528133633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dcaa00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 0x7f0dcc0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dcb400640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 0x7f0dcc13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 -- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 msgr2=0x7f0dcc0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 0x7f0dcc0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 -- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 msgr2=0x7f0dcc13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 0x7f0dcc13cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 -- 172.21.15.67:0/2528133633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc140b20 con 0x7f0dcc0aea80 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dcaa00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 0x7f0dcc0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dcb400640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 0x7f0dcc13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dca000640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009870 tx=0x7f0db8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8048070 con 0x7f0dcc0aea80 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db8039d90 con 0x7f0dcc0aea80 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.107+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc140e00 con 0x7f0dcc0aea80 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.108+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8034040 con 0x7f0dcc0aea80 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.108+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc13e630 con 0x7f0dcc0aea80 2024-04-17T07:30:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.109+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc068a10 con 0x7f0dcc0aea80 2024-04-17T07:30:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.110+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f0db804b070 con 0x7f0dcc0aea80 2024-04-17T07:30:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.111+0000 7f0dc2a00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 0x7f0dac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.111+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f0db80adee0 con 0x7f0dcc0aea80 2024-04-17T07:30:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.112+0000 7f0dcaa00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 0x7f0dac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.113+0000 7f0dcaa00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 0x7f0dac06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc161ff0 tx=0x7f0dbc009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.115+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0db807c1d0 con 0x7f0dcc0aea80 2024-04-17T07:30:44.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.319+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0dcc068610 con 0x7f0dcc0aea80 2024-04-17T07:30:44.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.320+0000 7f0dc2a00640 1 -- 172.21.15.67:0/2528133633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f0db807bad0 con 0x7f0dcc0aea80 2024-04-17T07:30:44.320 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 msgr2=0x7f0dac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 0x7f0dac06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc161ff0 tx=0x7f0dbc009040 comp rx=0 tx=0).stop 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 msgr2=0x7f0dcc1f8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009870 tx=0x7f0db8004730 comp rx=0 tx=0).stop 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 shutdown_connections 2024-04-17T07:30:44.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0dac06ca10 0x7f0dac06eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0dcc145040 0x7f0dcc13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc0aea80 0x7f0dcc1f8320 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 --2- 172.21.15.67:0/2528133633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0dcc072380 0x7f0dcc0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 >> 172.21.15.67:0/2528133633 conn(0x7f0dcc0b23c0 msgr2=0x7f0dcc0ce510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 shutdown_connections 2024-04-17T07:30:44.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.323+0000 7f0dcbe00640 1 -- 172.21.15.67:0/2528133633 wait complete. 2024-04-17T07:30:44.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-17T07:30:44.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.566+0000 7f8c32600640 1 Processor -- start 2024-04-17T07:30:44.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.566+0000 7f8c32600640 1 -- start start 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.567+0000 7f8c32600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c2c144da0 0x7f8c2c0b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.567+0000 7f8c32600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c2c0cd080 0x7f8c2c0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.567+0000 7f8c32600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c2c072cc0 con 0x7f8c2c06ee90 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c32600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8c2c0cda20 con 0x7f8c2c144da0 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c32600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8c2c0cdc40 con 0x7f8c2c0cd080 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c31200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c2c144da0 0x7f8c2c0b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c31c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c2c06ee90 0x7f8c2c15baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46916/0 (socket says 172.21.15.67:46916) 2024-04-17T07:30:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c31c00640 1 -- 172.21.15.67:0/2517418278 learned_addr learned my addr 172.21.15.67:0/2517418278 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.568+0000 7f8c2be00640 1 --2- 172.21.15.67:0/2517418278 >> v2:172.21.15.187:3300/0 conn(0x7f8c2c0cd080 0x7f8c2c0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.569+0000 7f8c31200640 1 -- 172.21.15.67:0/2517418278 >> v2:172.21.15.187:3300/0 conn(0x7f8c2c0cd080 msgr2=0x7f8c2c0cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.569+0000 7f8c31200640 1 --2- 172.21.15.67:0/2517418278 >> v2:172.21.15.187:3300/0 conn(0x7f8c2c0cd080 0x7f8c2c0cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.569+0000 7f8c31200640 1 -- 172.21.15.67:0/2517418278 >> v1:172.21.15.67:6789/0 conn(0x7f8c2c06ee90 legacy=0x7f8c2c15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.569+0000 7f8c31200640 1 -- 172.21.15.67:0/2517418278 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c0b3b50 con 0x7f8c2c144da0 2024-04-17T07:30:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.570+0000 7f8c31200640 1 --2- 172.21.15.67:0/2517418278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0b4910 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c06fff0 tx=0x7f8c2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9317b475a634ace3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.570+0000 7f8c2b400640 1 -- 172.21.15.67:0/2517418278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20035020 con 0x7f8c2c144da0 2024-04-17T07:30:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.570+0000 7f8c2b400640 1 -- 172.21.15.67:0/2517418278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c20031830 con 0x7f8c2c144da0 2024-04-17T07:30:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.570+0000 7f8c2b400640 1 -- 172.21.15.67:0/2517418278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20031c00 con 0x7f8c2c144da0 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 -- 172.21.15.67:0/2517418278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 msgr2=0x7f8c2c0b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 --2- 172.21.15.67:0/2517418278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0b4910 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c06fff0 tx=0x7f8c2002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 -- 172.21.15.67:0/2517418278 shutdown_connections 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 --2- 172.21.15.67:0/2517418278 >> v2:172.21.15.187:3300/0 conn(0x7f8c2c0cd080 0x7f8c2c0cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 --2- 172.21.15.67:0/2517418278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0b4910 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 -- 172.21.15.67:0/2517418278 >> 172.21.15.67:0/2517418278 conn(0x7f8c2c0b5f70 msgr2=0x7f8c2c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 -- 172.21.15.67:0/2517418278 shutdown_connections 2024-04-17T07:30:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.571+0000 7f8c32600640 1 -- 172.21.15.67:0/2517418278 wait complete. 2024-04-17T07:30:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.573+0000 7f8c32600640 1 Processor -- start 2024-04-17T07:30:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.573+0000 7f8c32600640 1 -- start start 2024-04-17T07:30:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c2c06ee90 0x7f8c2c15cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c2c0cd080 0x7f8c2c141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b43f0 con 0x7f8c2c144da0 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b4110 con 0x7f8c2c06ee90 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c32600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b4e50 con 0x7f8c2c0cd080 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c31c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c31c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43518/0 (socket says 172.21.15.67:43518) 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.574+0000 7f8c31c00640 1 -- 172.21.15.67:0/1056137851 learned_addr learned my addr 172.21.15.67:0/1056137851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 -- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c2c0cd080 msgr2=0x7f8c2c141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c2c0cd080 0x7f8c2c141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 -- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c2c06ee90 msgr2=0x7f8c2c15cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c2c06ee90 0x7f8c2c15cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 -- 172.21.15.67:0/1056137851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c15bf80 con 0x7f8c2c144da0 2024-04-17T07:30:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c31c00640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400ca80 tx=0x7f8c2400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24017070 con 0x7f8c2c144da0 2024-04-17T07:30:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c2400ac30 con 0x7f8c2c144da0 2024-04-17T07:30:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24012400 con 0x7f8c2c144da0 2024-04-17T07:30:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.575+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c15c260 con 0x7f8c2c144da0 2024-04-17T07:30:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.576+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c15c820 con 0x7f8c2c144da0 2024-04-17T07:30:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.577+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c069270 con 0x7f8c2c144da0 2024-04-17T07:30:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.578+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f8c2401a070 con 0x7f8c2c144da0 2024-04-17T07:30:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.579+0000 7f8c28c00640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 0x7f8c0c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.579+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7f8c2408b860 con 0x7f8c2c144da0 2024-04-17T07:30:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.579+0000 7f8c31200640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 0x7f8c0c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:44.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.581+0000 7f8c31200640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 0x7f8c0c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c20036000 tx=0x7f8c20030000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:44.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.583+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8c2c069270 con 0x7f8c2c144da0 2024-04-17T07:30:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:44.866+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7f8c2c068a10 con 0x7f8c2c144da0 2024-04-17T07:30:45.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.383+0000 7f8c28c00640 1 -- 172.21.15.67:0/1056137851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v643) ==== 186+0+0 (secure 0 0 0) 0x7f8c24059570 con 0x7f8c2c144da0 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 msgr2=0x7f8c0c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 0x7f8c0c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c20036000 tx=0x7f8c20030000 comp rx=0 tx=0).stop 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 msgr2=0x7f8c2c0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400ca80 tx=0x7f8c2400cf40 comp rx=0 tx=0).stop 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 shutdown_connections 2024-04-17T07:30:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f8c0c06ca10 0x7f8c0c06eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c144da0 0x7f8c2c0ad8d0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c2c0cd080 0x7f8c2c141920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 --2- 172.21.15.67:0/1056137851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c2c06ee90 0x7f8c2c15cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 >> 172.21.15.67:0/1056137851 conn(0x7f8c2c0b5f70 msgr2=0x7f8c2c15dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 shutdown_connections 2024-04-17T07:30:45.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.387+0000 7f8c32600640 1 -- 172.21.15.67:0/1056137851 wait complete. 2024-04-17T07:30:45.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:30:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.641+0000 7fcb0a800640 1 Processor -- start 2024-04-17T07:30:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.642+0000 7fcb0a800640 1 -- start start 2024-04-17T07:30:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.643+0000 7fcb0a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb0406daf0 0x7fcb040d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.643+0000 7fcb0a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb040d1060 0x7fcb0409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.643+0000 7fcb0a800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb040d1520 con 0x7fcb0406ce00 2024-04-17T07:30:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.643+0000 7fcb0a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb040d1800 con 0x7fcb040d1060 2024-04-17T07:30:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.643+0000 7fcb0a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb040d1ae0 con 0x7fcb0406daf0 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.644+0000 7fcb02a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb040d1060 0x7fcb0409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.644+0000 7fcb03e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb0406ce00 0x7fcb041628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46926/0 (socket says 172.21.15.67:46926) 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.644+0000 7fcb03e00640 1 -- 172.21.15.67:0/385374521 learned_addr learned my addr 172.21.15.67:0/385374521 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.644+0000 7fcb03400640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb0406daf0 0x7fcb040d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.644+0000 7fcb02a00640 1 -- 172.21.15.67:0/385374521 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb0406daf0 msgr2=0x7fcb040d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.645+0000 7fcb02a00640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb0406daf0 0x7fcb040d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.645+0000 7fcb02a00640 1 -- 172.21.15.67:0/385374521 >> v1:172.21.15.67:6789/0 conn(0x7fcb0406ce00 legacy=0x7fcb041628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.645+0000 7fcb02000640 1 -- 172.21.15.67:0/385374521 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241970530 0 0) 0x7fcb040d1520 con 0x7fcb0406ce00 2024-04-17T07:30:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.645+0000 7fcb02a00640 1 -- 172.21.15.67:0/385374521 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb040d1dc0 con 0x7fcb040d1060 2024-04-17T07:30:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.646+0000 7fcb02a00640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb040d1060 0x7fcb0409aa60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0097a0 tx=0x7fcaec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a5e69b54f8abc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.646+0000 7fcb02000640 1 -- 172.21.15.67:0/385374521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec035020 con 0x7fcb040d1060 2024-04-17T07:30:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.646+0000 7fcb02000640 1 -- 172.21.15.67:0/385374521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaec031810 con 0x7fcb040d1060 2024-04-17T07:30:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.646+0000 7fcb02000640 1 -- 172.21.15.67:0/385374521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec031be0 con 0x7fcb040d1060 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 -- 172.21.15.67:0/385374521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb040d1060 msgr2=0x7fcb0409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb040d1060 0x7fcb0409aa60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0097a0 tx=0x7fcaec02b8b0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 -- 172.21.15.67:0/385374521 shutdown_connections 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb040d1060 0x7fcb0409aa60 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 --2- 172.21.15.67:0/385374521 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb0406daf0 0x7fcb040d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 -- 172.21.15.67:0/385374521 >> 172.21.15.67:0/385374521 conn(0x7fcb040b41a0 msgr2=0x7fcb040b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 -- 172.21.15.67:0/385374521 shutdown_connections 2024-04-17T07:30:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.647+0000 7fcb0a800640 1 -- 172.21.15.67:0/385374521 wait complete. 2024-04-17T07:30:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.649+0000 7fcb0a800640 1 Processor -- start 2024-04-17T07:30:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.649+0000 7fcb0a800640 1 -- start start 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 0x7fcb0414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb040d1060 0x7fcb040cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0406e050 con 0x7fcb0406ce00 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0406ded0 con 0x7fcb0406daf0 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb0a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb04071050 con 0x7fcb040d1060 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb03400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb03400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43532/0 (socket says 172.21.15.67:43532) 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.650+0000 7fcb03400640 1 -- 172.21.15.67:0/1657603471 learned_addr learned my addr 172.21.15.67:0/1657603471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb02a00640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 0x7fcb0414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 -- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb040d1060 msgr2=0x7fcb040cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03e00640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb040d1060 0x7fcb040cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb040d1060 0x7fcb040cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 -- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 msgr2=0x7fcb0414f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 0x7fcb0414f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 -- 172.21.15.67:0/1657603471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb040c98d0 con 0x7fcb0406ce00 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb02a00640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 0x7fcb0414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb03400640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400efc0 tx=0x7fcaf400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4018020 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf4014480 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb040b8140 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4014850 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.651+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb040cc0d0 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.653+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb040cbc20 con 0x7fcb0406ce00 2024-04-17T07:30:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.654+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fcaf4005d10 con 0x7fcb0406ce00 2024-04-17T07:30:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.655+0000 7fcafb400640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 0x7fcae006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.655+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7fcaf408c8c0 con 0x7fcb0406ce00 2024-04-17T07:30:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.656+0000 7fcb02a00640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 0x7fcae006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:45.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.657+0000 7fcb02a00640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 0x7fcae006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcaec036000 tx=0x7fcaec008030 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:45.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.660+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcaf4022070 con 0x7fcb0406ce00 2024-04-17T07:30:45.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.857+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fcb04068ca0 con 0x7fcb0406ce00 2024-04-17T07:30:45.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.864+0000 7fcafb400640 1 -- 172.21.15.67:0/1657603471 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7fcaf4010040 con 0x7fcb0406ce00 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 msgr2=0x7fcae006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 0x7fcae006eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcaec036000 tx=0x7fcaec008030 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 msgr2=0x7fcb040b7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400efc0 tx=0x7fcaf400c490 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 shutdown_connections 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcae006ca10 0x7fcae006eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb040d1060 0x7fcb040cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb0406daf0 0x7fcb0414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 --2- 172.21.15.67:0/1657603471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0406ce00 0x7fcb040b7a50 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 >> 172.21.15.67:0/1657603471 conn(0x7fcb040b41a0 msgr2=0x7fcb040b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 shutdown_connections 2024-04-17T07:30:45.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:45.867+0000 7fcb0a800640 1 -- 172.21.15.67:0/1657603471 wait complete. 2024-04-17T07:30:45.886 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:30:45.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:30:46.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:30:46.145 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:30:46.145 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:46.145 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:46.202 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:30:46.202 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:30:46.202 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:46.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:30:46.259 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:30:46.259 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:46.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:30:46.319 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:30:46.319 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:30:46.375 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:30:46.376 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:30:46.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:46.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.605+0000 7ff17be00640 1 Processor -- start 2024-04-17T07:30:46.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.605+0000 7ff17be00640 1 -- start start 2024-04-17T07:30:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.606+0000 7ff17be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.606+0000 7ff17be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 0x7ff17c0d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.606+0000 7ff17be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff17c0cfe20 con 0x7ff17c0b9f00 2024-04-17T07:30:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.606+0000 7ff17be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff17c0d1c50 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.606+0000 7ff17be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff17c0cf640 con 0x7ff17c0d12a0 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43538/0 (socket says 172.21.15.67:43538) 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 -- 172.21.15.67:0/3158027919 learned_addr learned my addr 172.21.15.67:0/3158027919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 -- 172.21.15.67:0/3158027919 >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 msgr2=0x7ff17c0d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17b400640 1 --2- 172.21.15.67:0/3158027919 >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 0x7ff17c0d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 --2- 172.21.15.67:0/3158027919 >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 0x7ff17c0d1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 -- 172.21.15.67:0/3158027919 >> v1:172.21.15.67:6789/0 conn(0x7ff17c0b9f00 legacy=0x7ff17c1634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17a000640 1 -- 172.21.15.67:0/3158027919 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c0d0100 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.607+0000 7ff17b400640 1 --2- 172.21.15.67:0/3158027919 >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 0x7ff17c0d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.609+0000 7ff17a000640 1 --2- 172.21.15.67:0/3158027919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0b7810 tx=0x7ff16402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2043a1d4ef5d2951 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.609+0000 7ff179600640 1 -- 172.21.15.67:0/3158027919 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164035020 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.609+0000 7ff179600640 1 -- 172.21.15.67:0/3158027919 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff164031870 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.609+0000 7ff179600640 1 -- 172.21.15.67:0/3158027919 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164031c40 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 -- 172.21.15.67:0/3158027919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0cf8f0 msgr2=0x7ff17c0b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 --2- 172.21.15.67:0/3158027919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0b7810 tx=0x7ff16402b910 comp rx=0 tx=0).stop 2024-04-17T07:30:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 -- 172.21.15.67:0/3158027919 shutdown_connections 2024-04-17T07:30:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 --2- 172.21.15.67:0/3158027919 >> v2:172.21.15.187:3300/0 conn(0x7ff17c0d12a0 0x7ff17c0d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 --2- 172.21.15.67:0/3158027919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0b7f80 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 -- 172.21.15.67:0/3158027919 >> 172.21.15.67:0/3158027919 conn(0x7ff17c06cb60 msgr2=0x7ff17c06cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:46.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 -- 172.21.15.67:0/3158027919 shutdown_connections 2024-04-17T07:30:46.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.610+0000 7ff17be00640 1 -- 172.21.15.67:0/3158027919 wait complete. 2024-04-17T07:30:46.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.612+0000 7ff17be00640 1 Processor -- start 2024-04-17T07:30:46.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.612+0000 7ff17be00640 1 -- start start 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 0x7ff17c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0ba2e0 con 0x7ff17c0b9f00 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c0d3420 con 0x7ff17c0cf8f0 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.613+0000 7ff17be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff17c0d03e0 con 0x7ff17c0d12a0 2024-04-17T07:30:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43552/0 (socket says 172.21.15.67:43552) 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 -- 172.21.15.67:0/1938802921 learned_addr learned my addr 172.21.15.67:0/1938802921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17a000640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17b400640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 0x7ff17c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 -- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 msgr2=0x7ff17c154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 0x7ff17c154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 -- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff17c0cf8f0 msgr2=0x7ff17c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17aa00640 1 -- 172.21.15.67:0/1938802921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c1f52e0 con 0x7ff17c0b9f00 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.614+0000 7ff17b400640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 0x7ff17c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff17aa00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c430 tx=0x7ff16c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0052d0 con 0x7ff17c0b9f00 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff16c005490 con 0x7ff17c0b9f00 2024-04-17T07:30:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c007260 con 0x7ff17c0b9f00 2024-04-17T07:30:46.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c0ce800 con 0x7ff17c0b9f00 2024-04-17T07:30:46.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.615+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c146000 con 0x7ff17c0b9f00 2024-04-17T07:30:46.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.616+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c0755d0 con 0x7ff17c0b9f00 2024-04-17T07:30:46.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.617+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7ff16c007400 con 0x7ff17c0b9f00 2024-04-17T07:30:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.619+0000 7ff172a00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 0x7ff15806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.619+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7ff16c08b170 con 0x7ff17c0b9f00 2024-04-17T07:30:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.620+0000 7ff17a000640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 0x7ff15806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:46.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.621+0000 7ff17a000640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 0x7ff15806ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff164036000 tx=0x7ff16403c040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:46.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.624+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff16c059550 con 0x7ff17c0b9f00 2024-04-17T07:30:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.883+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff17c0b3ae0 con 0x7ff17c0b9f00 2024-04-17T07:30:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.884+0000 7ff172a00640 1 -- 172.21.15.67:0/1938802921 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+3875 (secure 0 0 0) 0x7ff16c058e50 con 0x7ff17c0b9f00 2024-04-17T07:30:46.884 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:46.884 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5956,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1889559837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1889559837},{"type":"v1","addr":"172.21.15.187:6835","nonce":1889559837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5962,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3175699620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3175699620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3175699620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2260014101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2260014101},{"type":"v1","addr":"172.21.15.67:6834","nonce":2260014101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1193757064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1193757064},{"type":"v1","addr":"172.21.15.67:6835","nonce":1193757064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[]} 2024-04-17T07:30:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T07:30:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 msgr2=0x7ff15806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 0x7ff15806ef20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff164036000 tx=0x7ff16403c040 comp rx=0 tx=0).stop 2024-04-17T07:30:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 msgr2=0x7ff17c148780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00c430 tx=0x7ff16c00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 shutdown_connections 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff15806ca60 0x7ff15806ef20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.886+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff17c0d12a0 0x7ff17c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.887+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff17c0cf8f0 0x7ff17c0ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.887+0000 7ff17be00640 1 --2- 172.21.15.67:0/1938802921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff17c0b9f00 0x7ff17c148780 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.887+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 >> 172.21.15.67:0/1938802921 conn(0x7ff17c06cb60 msgr2=0x7ff17c0b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.887+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 shutdown_connections 2024-04-17T07:30:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:46.887+0000 7ff17be00640 1 -- 172.21.15.67:0/1938802921 wait complete. 2024-04-17T07:30:46.906 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:30:46.906 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:30:46.906 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:30:46.906 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:46.907 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:30:46.907 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:30:46.907 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:30:46.907 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:46.907 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:30:46.907 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:30:46.907 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:30:46.908 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:46.908 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:30:46.908 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:30:46.908 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:30:46.908 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:30:46.926 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:30:46.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:30:46.932 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:30:46.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:30:46.935 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:30:46.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:30:46.940 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:30:46.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:30:46.983 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:30:46.983 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:46.986 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:30:46.987 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:46.989 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:30:46.989 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:46.993 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:30:46.993 DEBUG:teuthology.parallel:result is None 2024-04-17T07:30:46.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:30:47.146 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:30:47.146 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:30:47.149 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T07:30:47.149+0000 7fb4e0c00640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6834/0: (98) Address already in use 2024-04-17T07:30:47.153 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:30:47.163 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:30:47.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.184+0000 7ff7cd400640 1 Processor -- start 2024-04-17T07:30:47.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.184+0000 7ff7cd400640 1 -- start start 2024-04-17T07:30:47.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.185+0000 7ff7cd400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7c8145ac0 0x7ff7c8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.185+0000 7ff7cd400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.185+0000 7ff7cd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7c80cea70 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.185+0000 7ff7cd400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff7c80ce840 con 0x7ff7c8145ac0 2024-04-17T07:30:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.185+0000 7ff7cd400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c80ced50 con 0x7ff7c8072d80 2024-04-17T07:30:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7bde00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7c80ce3b0 0x7ff7c8072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46954/0 (socket says 172.21.15.67:46954) 2024-04-17T07:30:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7bde00640 1 -- 172.21.15.67:0/1681887337 learned_addr learned my addr 172.21.15.67:0/1681887337 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c6800640 1 --2- 172.21.15.67:0/1681887337 >> v2:172.21.15.187:3300/0 conn(0x7ff7c8145ac0 0x7ff7c8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c7200640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c5e00640 1 -- 172.21.15.67:0/1681887337 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499943018 0 0) 0x7ff7c80cea70 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c5e00640 1 -- 172.21.15.67:0/1681887337 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7ac003520 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c6800640 1 -- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 msgr2=0x7ff7c80cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c6800640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c80cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c6800640 1 -- 172.21.15.67:0/1681887337 >> v1:172.21.15.67:6789/0 conn(0x7ff7c80ce3b0 legacy=0x7ff7c8072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.186+0000 7ff7c6800640 1 -- 172.21.15.67:0/1681887337 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c80cf030 con 0x7ff7c8145ac0 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.187+0000 7ff7c6800640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145ac0 0x7ff7c8071ea0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009dc0 tx=0x7ff7b802be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2606ee588bd408 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7c5e00640 1 -- 172.21.15.67:0/1681887337 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8039070 con 0x7ff7c8145ac0 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7c5e00640 1 -- 172.21.15.67:0/1681887337 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7b802cc30 con 0x7ff7c8145ac0 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7c5e00640 1 -- 172.21.15.67:0/1681887337 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8034440 con 0x7ff7c8145ac0 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 -- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145ac0 msgr2=0x7ff7c8071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145ac0 0x7ff7c8071ea0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009dc0 tx=0x7ff7b802be80 comp rx=0 tx=0).stop 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 -- 172.21.15.67:0/1681887337 shutdown_connections 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c80cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 --2- 172.21.15.67:0/1681887337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c8145ac0 0x7ff7c8071ea0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.188+0000 7ff7cd400640 1 -- 172.21.15.67:0/1681887337 >> 172.21.15.67:0/1681887337 conn(0x7ff7c8067ee0 msgr2=0x7ff7c806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.189+0000 7ff7cd400640 1 -- 172.21.15.67:0/1681887337 shutdown_connections 2024-04-17T07:30:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.189+0000 7ff7cd400640 1 -- 172.21.15.67:0/1681887337 wait complete. 2024-04-17T07:30:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.190+0000 7ff7cd400640 1 Processor -- start 2024-04-17T07:30:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.190+0000 7ff7cd400640 1 -- start start 2024-04-17T07:30:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c815c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 0x7ff7c80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 0x7ff7c80b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8145ea0 con 0x7ff7c8145ac0 2024-04-17T07:30:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c80cf5f0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.191+0000 7ff7cd400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c80d0050 con 0x7ff7c8072d80 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 0x7ff7c80b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7c6800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c815c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 0x7ff7c80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 0x7ff7c80b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43604/0 (socket says 172.21.15.67:43604) 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7c7200640 1 -- 172.21.15.67:0/3941817047 learned_addr learned my addr 172.21.15.67:0/3941817047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 -- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 msgr2=0x7ff7c815c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c815c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 -- 172.21.15.67:0/3941817047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 msgr2=0x7ff7c80b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 0x7ff7c80b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.192+0000 7ff7bde00640 1 -- 172.21.15.67:0/3941817047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c81f8980 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7bde00640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 0x7ff7c80b3150 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c400 tx=0x7ff7b000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0017070 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7b0005ce0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c81e4fa0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b00072f0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.193+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c81e5560 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.194+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c8068ca0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.195+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7ff7b001a070 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.196+0000 7ff7bf400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 0x7ff7a406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:47.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.196+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5631+0+0 (secure 0 0 0) 0x7ff7b008afc0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.197+0000 7ff7c6800640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 0x7ff7a406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.197+0000 7ff7c6800640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 0x7ff7a406ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8036000 tx=0x7ff7b8030000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.202+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff7b0058cc0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.421+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff7c80cc6c0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.896+0000 7ff7bf400640 1 -- 172.21.15.67:0/3941817047 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) ==== 97+0+0 (secure 0 0 0) 0x7ff7b005cfc0 con 0x7ff7c80ce3b0 2024-04-17T07:30:47.897 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 msgr2=0x7ff7a406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 0x7ff7a406ef20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8036000 tx=0x7ff7b8030000 comp rx=0 tx=0).stop 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 msgr2=0x7ff7c80b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 0x7ff7c80b3150 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c400 tx=0x7ff7b000c8c0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 shutdown_connections 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff7a406ca60 0x7ff7a406ef20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7c8145ac0 0x7ff7c80b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7c80ce3b0 0x7ff7c80b3150 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3941817047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7c8072d80 0x7ff7c815c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 >> 172.21.15.67:0/3941817047 conn(0x7ff7c8067ee0 msgr2=0x7ff7c80b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.901+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 shutdown_connections 2024-04-17T07:30:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:47.902+0000 7ff7cd400640 1 -- 172.21.15.67:0/3941817047 wait complete. 2024-04-17T07:30:47.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:30:48.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.143+0000 7f3aada00640 1 Processor -- start 2024-04-17T07:30:48.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.143+0000 7f3aada00640 1 -- start start 2024-04-17T07:30:48.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aada00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3aa81452f0 0x7f3aa8145950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aada00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3aa8072470 0x7f3aa815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aada00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa815cd40 con 0x7f3aa8160e50 2024-04-17T07:30:48.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aada00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa815d670 con 0x7f3aa8072470 2024-04-17T07:30:48.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aada00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa815d020 con 0x7f3aa81452f0 2024-04-17T07:30:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aa7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3aa8160e50 0x7f3aa8071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46968/0 (socket says 172.21.15.67:46968) 2024-04-17T07:30:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aa7400640 1 -- 172.21.15.67:0/368491680 learned_addr learned my addr 172.21.15.67:0/368491680 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.145+0000 7f3aad000640 1 --2- 172.21.15.67:0/368491680 >> v2:172.21.15.67:3300/0 conn(0x7f3aa8072470 0x7f3aa815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aa7e00640 1 --2- 172.21.15.67:0/368491680 >> v2:172.21.15.187:3300/0 conn(0x7f3aa81452f0 0x7f3aa8145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aa6a00640 1 -- 172.21.15.67:0/368491680 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524328417 0 0) 0x7f3aa815cd40 con 0x7f3aa8160e50 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aa6a00640 1 -- 172.21.15.67:0/368491680 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a8c003520 con 0x7f3aa8160e50 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aad000640 1 -- 172.21.15.67:0/368491680 >> v2:172.21.15.187:3300/0 conn(0x7f3aa81452f0 msgr2=0x7f3aa8145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aad000640 1 --2- 172.21.15.67:0/368491680 >> v2:172.21.15.187:3300/0 conn(0x7f3aa81452f0 0x7f3aa8145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aad000640 1 -- 172.21.15.67:0/368491680 >> v1:172.21.15.67:6789/0 conn(0x7f3aa8160e50 legacy=0x7f3aa8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.146+0000 7f3aad000640 1 -- 172.21.15.67:0/368491680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa815d300 con 0x7f3aa8072470 2024-04-17T07:30:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.147+0000 7f3aad000640 1 --2- 172.21.15.67:0/368491680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa815db00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8073200 tx=0x7f3aa002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a481a0cb62c3119c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aa6a00640 1 -- 172.21.15.67:0/368491680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0039070 con 0x7f3aa8072470 2024-04-17T07:30:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aa6a00640 1 -- 172.21.15.67:0/368491680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa002cbf0 con 0x7f3aa8072470 2024-04-17T07:30:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aa6a00640 1 -- 172.21.15.67:0/368491680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0034440 con 0x7f3aa8072470 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 -- 172.21.15.67:0/368491680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 msgr2=0x7f3aa815db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 --2- 172.21.15.67:0/368491680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa815db00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8073200 tx=0x7f3aa002be40 comp rx=0 tx=0).stop 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 -- 172.21.15.67:0/368491680 shutdown_connections 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 --2- 172.21.15.67:0/368491680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa815db00 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 --2- 172.21.15.67:0/368491680 >> v2:172.21.15.187:3300/0 conn(0x7f3aa81452f0 0x7f3aa8145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.148+0000 7f3aada00640 1 -- 172.21.15.67:0/368491680 >> 172.21.15.67:0/368491680 conn(0x7f3aa80b5f70 msgr2=0x7f3aa806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.149+0000 7f3aada00640 1 -- 172.21.15.67:0/368491680 shutdown_connections 2024-04-17T07:30:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.149+0000 7f3aada00640 1 -- 172.21.15.67:0/368491680 wait complete. 2024-04-17T07:30:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.150+0000 7f3aada00640 1 Processor -- start 2024-04-17T07:30:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.151+0000 7f3aada00640 1 -- start start 2024-04-17T07:30:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.151+0000 7f3aada00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aada00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3aa81452f0 0x7f3aa80ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aada00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3aa8160e50 0x7f3aa8142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aada00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa81456d0 con 0x7f3aa8072470 2024-04-17T07:30:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aada00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa8145e90 con 0x7f3aa81452f0 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aada00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa815e040 con 0x7f3aa8160e50 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43632/0 (socket says 172.21.15.67:43632) 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 -- 172.21.15.67:0/2225943815 learned_addr learned my addr 172.21.15.67:0/2225943815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 -- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3aa8160e50 msgr2=0x7f3aa8142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3aa8160e50 0x7f3aa8142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 -- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3aa81452f0 msgr2=0x7f3aa80ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3aa81452f0 0x7f3aa80ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.152+0000 7f3aa7e00640 1 -- 172.21.15.67:0/2225943815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa81f2330 con 0x7f3aa8072470 2024-04-17T07:30:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3aa7e00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800b6c0 con 0x7f3aa8072470 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a98019070 con 0x7f3aa8072470 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800ba60 con 0x7f3aa8072470 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa81f25f0 con 0x7f3aa8072470 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.153+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa81f2bb0 con 0x7f3aa8072470 2024-04-17T07:30:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.154+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa81f3270 con 0x7f3aa8072470 2024-04-17T07:30:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.155+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f3a9800bc00 con 0x7f3aa8072470 2024-04-17T07:30:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.156+0000 7f3a9fe00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 0x7f3a8806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.156+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4857+0+0 (secure 0 0 0) 0x7f3a9808beb0 con 0x7f3aa8072470 2024-04-17T07:30:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.157+0000 7f3aa7400640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 0x7f3a8806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.158+0000 7f3aa7400640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 0x7f3a8806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a94000f60 tx=0x7f3a940078f0 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.161+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3aa81f3270 con 0x7f3aa8072470 2024-04-17T07:30:48.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.444+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3aa8072e50 con 0x7f3aa8072470 2024-04-17T07:30:48.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.446+0000 7f3a9fe00640 1 -- 172.21.15.67:0/2225943815 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f3a9805a520 con 0x7f3aa8072470 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.446 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.448 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.449 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:30:48.451 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:30:48.453 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:30:48.454 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:30:48.456 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:30:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:30:48.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:30:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 msgr2=0x7f3a8806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 0x7f3a8806eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3a94000f60 tx=0x7f3a940078f0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 msgr2=0x7f3aa813ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d800 tx=0x7f3a9800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 shutdown_connections 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f3a8806ca10 0x7f3a8806eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3aa8160e50 0x7f3aa8142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3aa81452f0 0x7f3aa80ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 --2- 172.21.15.67:0/2225943815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3aa8072470 0x7f3aa813ec20 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 >> 172.21.15.67:0/2225943815 conn(0x7f3aa80b5f70 msgr2=0x7f3aa806e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 shutdown_connections 2024-04-17T07:30:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.449+0000 7f3aada00640 1 -- 172.21.15.67:0/2225943815 wait complete. 2024-04-17T07:30:48.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:48.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.696+0000 7f579fe00640 1 Processor -- start 2024-04-17T07:30:48.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.697+0000 7f579fe00640 1 -- start start 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.698+0000 7f579fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57a00b6d00 0x7f57a00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.698+0000 7f579fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f57a00cf910 0x7f57a00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.698+0000 7f579fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57a00d2d30 con 0x7f57a015ec60 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.698+0000 7f579fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57a00d49c0 con 0x7f57a00b6d00 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.698+0000 7f579fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f57a00d3660 con 0x7f57a00cf910 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57a00b6d00 0x7f57a00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57a00b6d00 0x7f57a00cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43642/0 (socket says 172.21.15.67:43642) 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 -- 172.21.15.67:0/3137659192 learned_addr learned my addr 172.21.15.67:0/3137659192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579f400640 1 --2- 172.21.15.67:0/3137659192 >> v2:172.21.15.187:3300/0 conn(0x7f57a00cf910 0x7f57a00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 -- 172.21.15.67:0/3137659192 >> v2:172.21.15.187:3300/0 conn(0x7f57a00cf910 msgr2=0x7f57a00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 --2- 172.21.15.67:0/3137659192 >> v2:172.21.15.187:3300/0 conn(0x7f57a00cf910 0x7f57a00b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 -- 172.21.15.67:0/3137659192 >> v1:172.21.15.187:6789/0 conn(0x7f57a015ec60 legacy=0x7f57a00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:48.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.699+0000 7f579e000640 1 -- 172.21.15.67:0/3137659192 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a00d3010 con 0x7f57a00b6d00 2024-04-17T07:30:48.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.700+0000 7f579e000640 1 --2- 172.21.15.67:0/3137659192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00b6d00 0x7f57a00cf3d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f57a00cf170 tx=0x7f579002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e0a6227bf459ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579d600640 1 -- 172.21.15.67:0/3137659192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579002ca80 con 0x7f57a00b6d00 2024-04-17T07:30:48.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579d600640 1 -- 172.21.15.67:0/3137659192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f579002cc20 con 0x7f57a00b6d00 2024-04-17T07:30:48.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579d600640 1 -- 172.21.15.67:0/3137659192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57900343f0 con 0x7f57a00b6d00 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 -- 172.21.15.67:0/3137659192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00b6d00 msgr2=0x7f57a00cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 --2- 172.21.15.67:0/3137659192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00b6d00 0x7f57a00cf3d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f57a00cf170 tx=0x7f579002bee0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 -- 172.21.15.67:0/3137659192 shutdown_connections 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 --2- 172.21.15.67:0/3137659192 >> v2:172.21.15.187:3300/0 conn(0x7f57a00cf910 0x7f57a00b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 --2- 172.21.15.67:0/3137659192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00b6d00 0x7f57a00cf3d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 -- 172.21.15.67:0/3137659192 >> 172.21.15.67:0/3137659192 conn(0x7f57a00d15f0 msgr2=0x7f57a00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 -- 172.21.15.67:0/3137659192 shutdown_connections 2024-04-17T07:30:48.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.701+0000 7f579fe00640 1 -- 172.21.15.67:0/3137659192 wait complete. 2024-04-17T07:30:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.703+0000 7f579fe00640 1 Processor -- start 2024-04-17T07:30:48.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.703+0000 7f579fe00640 1 -- start start 2024-04-17T07:30:48.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57a00b6d00 0x7f57a00abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57a00cf910 0x7f57a01f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57a015f040 con 0x7f57a015ec60 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00cfdc0 con 0x7f57a00b6d00 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57a00d00c0 con 0x7f57a00cf910 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.704+0000 7f579f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43658/0 (socket says 172.21.15.67:43658) 2024-04-17T07:30:48.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 -- 172.21.15.67:0/950372458 learned_addr learned my addr 172.21.15.67:0/950372458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579e000640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57a00cf910 0x7f57a01f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 -- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57a00cf910 msgr2=0x7f57a01f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57a00cf910 0x7f57a01f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 -- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57a00b6d00 msgr2=0x7f57a00abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579ea00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57a00b6d00 0x7f57a00abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57a00b6d00 0x7f57a00abf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 -- 172.21.15.67:0/950372458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a00ac780 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.705+0000 7f579f400640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f579800d830 tx=0x7f579800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.706+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579800b6c0 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.706+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5798019070 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.706+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579800bac0 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.706+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a00aca60 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.706+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a00ad020 con 0x7f57a015ec60 2024-04-17T07:30:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.707+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a00ad4c0 con 0x7f57a015ec60 2024-04-17T07:30:48.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.708+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f579801e460 con 0x7f57a015ec60 2024-04-17T07:30:48.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.709+0000 7f5796a00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 0x7f577c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:48.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.709+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4857+0+0 (secure 0 0 0) 0x7f579808bff0 con 0x7f57a015ec60 2024-04-17T07:30:48.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.710+0000 7f579ea00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 0x7f577c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:48.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.711+0000 7f579ea00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 0x7f577c06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f578800a750 tx=0x7f5788008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:48.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.714+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f579805a6b0 con 0x7f57a015ec60 2024-04-17T07:30:48.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.925+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57a0068a10 con 0x7f57a015ec60 2024-04-17T07:30:48.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.927+0000 7f5796a00640 1 -- 172.21.15.67:0/950372458 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+12845 (secure 0 0 0) 0x7f5798059fb0 con 0x7f57a015ec60 2024-04-17T07:30:48.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:47.909087+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 msgr2=0x7f577c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 0x7f577c06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f578800a750 tx=0x7f5788008040 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 msgr2=0x7f57a0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f579800d830 tx=0x7f579800dcf0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 shutdown_connections 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f577c06ca10 0x7f577c06eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a015ec60 0x7f57a0074d40 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.929+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57a00cf910 0x7f57a01f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.930+0000 7f579fe00640 1 --2- 172.21.15.67:0/950372458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57a00b6d00 0x7f57a00abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.930+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 >> 172.21.15.67:0/950372458 conn(0x7f57a00d15f0 msgr2=0x7f57a00b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.930+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 shutdown_connections 2024-04-17T07:30:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:48.930+0000 7f579fe00640 1 -- 172.21.15.67:0/950372458 wait complete. 2024-04-17T07:30:48.948 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:30:48.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:30:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.158+0000 7fea73200640 1 Processor -- start 2024-04-17T07:30:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.159+0000 7fea73200640 1 -- start start 2024-04-17T07:30:49.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.160+0000 7fea73200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea6c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.160+0000 7fea73200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea6c160920 con 0x7fea6c145b00 2024-04-17T07:30:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea73200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea6c160c00 con 0x7fea6c0b4ac0 2024-04-17T07:30:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea73200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fea6c160ee0 con 0x7fea6c146060 2024-04-17T07:30:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea71e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea6c145b00 0x7fea6c15cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46974/0 (socket says 172.21.15.67:46974) 2024-04-17T07:30:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea71e00640 1 -- 172.21.15.67:0/2012221743 learned_addr learned my addr 172.21.15.67:0/2012221743 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea71400640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea6c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491208392 0 0) 0x7fea6c160920 con 0x7fea6c145b00 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.161+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea54003520 con 0x7fea6c145b00 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2642843326 0 0) 0x7fea54003520 con 0x7fea6c145b00 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 msgr2=0x7fea6c160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea6c160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 >> v1:172.21.15.187:6789/0 conn(0x7fea6c0b4ac0 legacy=0x7fea6c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6c1611c0 con 0x7fea6c145b00 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fea60001670 con 0x7fea6c145b00 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 >> v1:172.21.15.67:6789/0 conn(0x7fea6c145b00 legacy=0x7fea6c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea540039e0 0x7fea54003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea54004800 0x7fea54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c160ee0 con 0x7fea540039e0 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fea54005550 con 0x7fea54004800 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fea5400bf00 con 0x7fea6c146060 2024-04-17T07:30:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea71e00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea540039e0 0x7fea54003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea71400640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea54004800 0x7fea54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.162+0000 7fea72800640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71e00640 1 -- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 msgr2=0x7fea5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71e00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71e00640 1 -- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea54004800 msgr2=0x7fea54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71e00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea54004800 0x7fea54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71e00640 1 -- 172.21.15.67:0/2012221743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea54003520 con 0x7fea540039e0 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea72800640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea6c146060 0x7fea5400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.163+0000 7fea71400640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea54004800 0x7fea54004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.164+0000 7fea71e00640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea540039e0 0x7fea54003e00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fea6000c240 tx=0x7fea54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92754524587b486 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.164+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60007d10 con 0x7fea540039e0 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.164+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea60030070 con 0x7fea540039e0 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.164+0000 7fea70a00640 1 -- 172.21.15.67:0/2012221743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6002cd00 con 0x7fea540039e0 2024-04-17T07:30:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 -- 172.21.15.67:0/2012221743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea540039e0 msgr2=0x7fea54003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 --2- 172.21.15.67:0/2012221743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea540039e0 0x7fea54003e00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fea6000c240 tx=0x7fea54005830 comp rx=0 tx=0).stop 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea72800640 1 -- 172.21.15.67:0/2012221743 reap_dead start 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 -- 172.21.15.67:0/2012221743 shutdown_connections 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 -- 172.21.15.67:0/2012221743 >> 172.21.15.67:0/2012221743 conn(0x7fea6c0b23c0 msgr2=0x7fea6c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 -- 172.21.15.67:0/2012221743 shutdown_connections 2024-04-17T07:30:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.165+0000 7fea73200640 1 -- 172.21.15.67:0/2012221743 wait complete. 2024-04-17T07:30:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.167+0000 7fea73200640 1 Processor -- start 2024-04-17T07:30:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.167+0000 7fea73200640 1 -- start start 2024-04-17T07:30:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea540039e0 0x7fea6c0af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea6c0aa100 0x7fea6c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c160340 con 0x7fea6c0aa560 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c0cfba0 con 0x7fea6c0aa100 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea73200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fea6c1604c0 con 0x7fea540039e0 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea72800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea72800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43676/0 (socket says 172.21.15.67:43676) 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.168+0000 7fea72800640 1 -- 172.21.15.67:0/4109713369 learned_addr learned my addr 172.21.15.67:0/4109713369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 -- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea540039e0 msgr2=0x7fea6c0af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea540039e0 0x7fea6c0af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea71400640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea6c0aa100 0x7fea6c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 -- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea6c0aa100 msgr2=0x7fea6c15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea6c0aa100 0x7fea6c15bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 -- 172.21.15.67:0/4109713369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6c0aaa70 con 0x7fea6c0aa560 2024-04-17T07:30:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea72800640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fea6c073550 tx=0x7fea6800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68006d50 con 0x7fea6c0aa560 2024-04-17T07:30:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea68003970 con 0x7fea6c0aa560 2024-04-17T07:30:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68003d60 con 0x7fea6c0aa560 2024-04-17T07:30:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6c0aad50 con 0x7fea6c0aa560 2024-04-17T07:30:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.169+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6c0ac530 con 0x7fea6c0aa560 2024-04-17T07:30:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.171+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6c068a10 con 0x7fea6c0aa560 2024-04-17T07:30:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.171+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fea6801b020 con 0x7fea6c0aa560 2024-04-17T07:30:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.172+0000 7fea66000640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 0x7fea5006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.172+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4857+0+0 (secure 0 0 0) 0x7fea6808b5b0 con 0x7fea6c0aa560 2024-04-17T07:30:49.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.173+0000 7fea71e00640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 0x7fea5006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.174+0000 7fea71e00640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 0x7fea5006eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea60034000 tx=0x7fea60007000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.177+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fea6801ce40 con 0x7fea6c0aa560 2024-04-17T07:30:49.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.381+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fea6c068610 con 0x7fea6c0aa560 2024-04-17T07:30:49.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.381+0000 7fea66000640 1 -- 172.21.15.67:0/4109713369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7fea6808b020 con 0x7fea6c0aa560 2024-04-17T07:30:49.382 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 msgr2=0x7fea5006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 0x7fea5006eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea60034000 tx=0x7fea60007000 comp rx=0 tx=0).stop 2024-04-17T07:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 msgr2=0x7fea6c15c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fea6c073550 tx=0x7fea6800dc00 comp rx=0 tx=0).stop 2024-04-17T07:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 shutdown_connections 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fea5006ca10 0x7fea5006eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea6c0aa560 0x7fea6c15c150 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea6c0aa100 0x7fea6c15bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 --2- 172.21.15.67:0/4109713369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea540039e0 0x7fea6c0af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 >> 172.21.15.67:0/4109713369 conn(0x7fea6c0b23c0 msgr2=0x7fea6c15cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 shutdown_connections 2024-04-17T07:30:49.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.384+0000 7fea73200640 1 -- 172.21.15.67:0/4109713369 wait complete. 2024-04-17T07:30:49.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:30:49.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.633+0000 7f55f4e00640 1 Processor -- start 2024-04-17T07:30:49.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.634+0000 7f55f4e00640 1 -- start start 2024-04-17T07:30:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.635+0000 7f55f4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.635+0000 7f55f4e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f55f006e020 0x7f55f006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.635+0000 7f55f4e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55f00ceb00 con 0x7f55f006f670 2024-04-17T07:30:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.635+0000 7f55f4e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f55f00cede0 con 0x7f55f006e020 2024-04-17T07:30:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.635+0000 7f55f4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55f00cf0c0 con 0x7f55f00cd020 2024-04-17T07:30:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.636+0000 7f55e7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.636+0000 7f55ede00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f55f006f670 0x7f55f006fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:45570/0 (socket says 172.21.15.67:45570) 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.636+0000 7f55ee800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f55f006e020 0x7f55f006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.636+0000 7f55ede00640 1 -- 172.21.15.67:0/1015926818 learned_addr learned my addr 172.21.15.67:0/1015926818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.637+0000 7f55ee800640 1 -- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 msgr2=0x7f55f00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.637+0000 7f55ee800640 1 --2- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.637+0000 7f55ee800640 1 -- 172.21.15.67:0/1015926818 >> v1:172.21.15.187:6789/0 conn(0x7f55f006f670 legacy=0x7f55f006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.637+0000 7f55ee800640 1 -- 172.21.15.67:0/1015926818 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f00cf3a0 con 0x7f55f006e020 2024-04-17T07:30:49.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.639+0000 7f55ee800640 1 --2- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006e020 0x7f55f006e4d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009d90 tx=0x7f55e802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42718c2acf8d8e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.639+0000 7f55ed400640 1 -- 172.21.15.67:0/1015926818 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8039070 con 0x7f55f006e020 2024-04-17T07:30:49.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.639+0000 7f55ed400640 1 -- 172.21.15.67:0/1015926818 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e802cc50 con 0x7f55f006e020 2024-04-17T07:30:49.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.639+0000 7f55ed400640 1 -- 172.21.15.67:0/1015926818 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8034440 con 0x7f55f006e020 2024-04-17T07:30:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 -- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006e020 msgr2=0x7f55f006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 --2- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006e020 0x7f55f006e4d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009d90 tx=0x7f55e802bea0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 -- 172.21.15.67:0/1015926818 shutdown_connections 2024-04-17T07:30:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 --2- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006e020 0x7f55f006e4d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 --2- 172.21.15.67:0/1015926818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 -- 172.21.15.67:0/1015926818 >> 172.21.15.67:0/1015926818 conn(0x7f55f006bf50 msgr2=0x7f55f00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:49.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 -- 172.21.15.67:0/1015926818 shutdown_connections 2024-04-17T07:30:49.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.640+0000 7f55f4e00640 1 -- 172.21.15.67:0/1015926818 wait complete. 2024-04-17T07:30:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.642+0000 7f55f4e00640 1 Processor -- start 2024-04-17T07:30:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.642+0000 7f55f4e00640 1 -- start start 2024-04-17T07:30:49.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006f670 0x7f55f015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f014dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55f00ce8a0 con 0x7f55f006e020 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f55f00ce720 con 0x7f55f006f670 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55f4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55f015cb00 con 0x7f55f00cd020 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43686/0 (socket says 172.21.15.67:43686) 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.643+0000 7f55ede00640 1 -- 172.21.15.67:0/3427790796 learned_addr learned my addr 172.21.15.67:0/3427790796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55e7e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006f670 0x7f55f015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ee800640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f014dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 -- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 msgr2=0x7f55f014dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f014dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 -- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006f670 msgr2=0x7f55f015ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006f670 0x7f55f015ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 -- 172.21.15.67:0/3427790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f0151930 con 0x7f55f006e020 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.644+0000 7f55ede00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c3c0 tx=0x7f55e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.645+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e0017070 con 0x7f55f006e020 2024-04-17T07:30:49.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.645+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e0005ce0 con 0x7f55f006e020 2024-04-17T07:30:49.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.645+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e00072f0 con 0x7f55f006e020 2024-04-17T07:30:49.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.645+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f015d400 con 0x7f55f006e020 2024-04-17T07:30:49.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.645+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f014ede0 con 0x7f55f006e020 2024-04-17T07:30:49.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.646+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f00cbc80 con 0x7f55f006e020 2024-04-17T07:30:49.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.647+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f55e001a070 con 0x7f55f006e020 2024-04-17T07:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.648+0000 7f55e6000640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 0x7f55cc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.648+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4857+0+0 (secure 0 0 0) 0x7f55e008ad20 con 0x7f55f006e020 2024-04-17T07:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.649+0000 7f55e7e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 0x7f55cc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.650+0000 7f55e7e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 0x7f55cc06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f55f0158b40 tx=0x7f55d8008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.653+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f55f00cbc80 con 0x7f55f006e020 2024-04-17T07:30:49.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.863+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f55f00c5450 con 0x7f55f006e020 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.864+0000 7f55e6000640 1 -- 172.21.15.67:0/3427790796 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) ==== 84+0+65 (secure 0 0 0) 0x7f55e0059380 con 0x7f55f006e020 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:30:49.864 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 msgr2=0x7f55cc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 0x7f55cc06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f55f0158b40 tx=0x7f55d8008040 comp rx=0 tx=0).stop 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 msgr2=0x7f55f015f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c3c0 tx=0x7f55e000c880 comp rx=0 tx=0).stop 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 shutdown_connections 2024-04-17T07:30:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55cc06ca10 0x7f55cc06eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55f00cd020 0x7f55f014dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55f006f670 0x7f55f015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 --2- 172.21.15.67:0/3427790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f006e020 0x7f55f015f840 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 >> 172.21.15.67:0/3427790796 conn(0x7f55f006bf50 msgr2=0x7f55f006e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 shutdown_connections 2024-04-17T07:30:49.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:49.867+0000 7f55f4e00640 1 -- 172.21.15.67:0/3427790796 wait complete. 2024-04-17T07:30:49.888 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:30:49.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:30:50.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.104+0000 7f9141000640 1 Processor -- start 2024-04-17T07:30:50.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.104+0000 7f9141000640 1 -- start start 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.105+0000 7f9141000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f913c15eab0 0x7f913c0d1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.105+0000 7f9141000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0d33e0 0x7f913c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.105+0000 7f9141000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913c0cf430 con 0x7f913c0d01b0 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.105+0000 7f9141000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0d2820 con 0x7f913c0d33e0 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.105+0000 7f9141000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0d0610 con 0x7f913c15eab0 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0d33e0 0x7f913c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f913c0d33e0 0x7f913c0d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43698/0 (socket says 172.21.15.67:43698) 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 -- 172.21.15.67:0/2649213933 learned_addr learned my addr 172.21.15.67:0/2649213933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:50.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913aa00640 1 --1- 172.21.15.67:0/2649213933 >> v1:172.21.15.67:6789/0 conn(0x7f913c0d01b0 0x7f913c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46980/0 (socket says 172.21.15.67:46980) 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913b400640 1 --2- 172.21.15.67:0/2649213933 >> v2:172.21.15.187:3300/0 conn(0x7f913c15eab0 0x7f913c0d1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 -- 172.21.15.67:0/2649213933 >> v2:172.21.15.187:3300/0 conn(0x7f913c15eab0 msgr2=0x7f913c0d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 --2- 172.21.15.67:0/2649213933 >> v2:172.21.15.187:3300/0 conn(0x7f913c15eab0 0x7f913c0d1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 -- 172.21.15.67:0/2649213933 >> v1:172.21.15.67:6789/0 conn(0x7f913c0d01b0 legacy=0x7f913c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913be00640 1 -- 172.21.15.67:0/2649213933 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c0cf6d0 con 0x7f913c0d33e0 2024-04-17T07:30:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.106+0000 7f913a000640 1 -- 172.21.15.67:0/2649213933 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895780714 0 0) 0x7f913c0cf430 con 0x7f913c0d01b0 2024-04-17T07:30:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f913be00640 1 --2- 172.21.15.67:0/2649213933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c0d22e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f913c08ea00 tx=0x7f913402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a62b2e5623ba59 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f913a000640 1 -- 172.21.15.67:0/2649213933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134034070 con 0x7f913c0d33e0 2024-04-17T07:30:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f913a000640 1 -- 172.21.15.67:0/2649213933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9134030860 con 0x7f913c0d33e0 2024-04-17T07:30:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f913a000640 1 -- 172.21.15.67:0/2649213933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134030c30 con 0x7f913c0d33e0 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f9141000640 1 -- 172.21.15.67:0/2649213933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 msgr2=0x7f913c0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.108+0000 7f9141000640 1 --2- 172.21.15.67:0/2649213933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c0d22e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f913c08ea00 tx=0x7f913402b9a0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 -- 172.21.15.67:0/2649213933 shutdown_connections 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 --2- 172.21.15.67:0/2649213933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c0d22e0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 --2- 172.21.15.67:0/2649213933 >> v2:172.21.15.187:3300/0 conn(0x7f913c15eab0 0x7f913c0d1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 -- 172.21.15.67:0/2649213933 >> 172.21.15.67:0/2649213933 conn(0x7f913c08bff0 msgr2=0x7f913c0d4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 -- 172.21.15.67:0/2649213933 shutdown_connections 2024-04-17T07:30:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.109+0000 7f9141000640 1 -- 172.21.15.67:0/2649213933 wait complete. 2024-04-17T07:30:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.110+0000 7f9141000640 1 Processor -- start 2024-04-17T07:30:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.111+0000 7f9141000640 1 -- start start 2024-04-17T07:30:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 0x7f913c14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c15eab0 0x7f913c15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0d2a50 con 0x7f913c0d33e0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0cfc90 con 0x7f913c15eab0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f9141000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f913c0b7560 con 0x7f913c0d01b0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43702/0 (socket says 172.21.15.67:43702) 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 -- 172.21.15.67:0/2402602386 learned_addr learned my addr 172.21.15.67:0/2402602386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913b400640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 0x7f913c14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913be00640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c15eab0 0x7f913c15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 -- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 msgr2=0x7f913c14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 0x7f913c14ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 -- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c15eab0 msgr2=0x7f913c15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c15eab0 0x7f913c15f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913aa00640 1 -- 172.21.15.67:0/2402602386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c14b3a0 con 0x7f913c0d33e0 2024-04-17T07:30:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.112+0000 7f913b400640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 0x7f913c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f913aa00640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f912400c500 tx=0x7f912400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124018070 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124007580 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913c14b680 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124004a20 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.113+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913c15f010 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.114+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f913c0755d0 con 0x7f913c0d33e0 2024-04-17T07:30:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.116+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f912401b070 con 0x7f913c0d33e0 2024-04-17T07:30:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.117+0000 7f9133400640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 0x7f911806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.117+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4857+0+0 (secure 0 0 0) 0x7f912408ad20 con 0x7f913c0d33e0 2024-04-17T07:30:50.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.117+0000 7f913b400640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 0x7f911806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:50.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.119+0000 7f913b400640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 0x7f911806eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f912c009f50 tx=0x7f912c00b040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:50.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.121+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9124059380 con 0x7f913c0d33e0 2024-04-17T07:30:50.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.330+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f913c0d4e20 con 0x7f913c0d33e0 2024-04-17T07:30:50.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.905+0000 7f9133400640 1 -- 172.21.15.67:0/2402602386 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) ==== 138+0+0 (secure 0 0 0) 0x7f9124058c80 con 0x7f913c0d33e0 2024-04-17T07:30:50.905 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 msgr2=0x7f911806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 0x7f911806eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f912c009f50 tx=0x7f912c00b040 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 msgr2=0x7f913c1617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f912400c500 tx=0x7f912400c9c0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 shutdown_connections 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f911806ca10 0x7f911806eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913c15eab0 0x7f913c15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f913c0d33e0 0x7f913c1617f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 --2- 172.21.15.67:0/2402602386 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913c0d01b0 0x7f913c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:50.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.908+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 >> 172.21.15.67:0/2402602386 conn(0x7f913c08bff0 msgr2=0x7f913c06dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:50.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.909+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 shutdown_connections 2024-04-17T07:30:50.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:50.909+0000 7f9141000640 1 -- 172.21.15.67:0/2402602386 wait complete. 2024-04-17T07:30:50.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:30:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.145+0000 7fd176200640 1 Processor -- start 2024-04-17T07:30:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.145+0000 7fd176200640 1 -- start start 2024-04-17T07:30:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd176200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 0x7fd1700ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd176200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1700b20b0 0x7fd1700b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd176200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17015f050 con 0x7fd17015c1a0 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd176200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd1700b2a60 con 0x7fd1700b20b0 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd176200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1700b3770 con 0x7fd17015c580 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.147+0000 7fd175800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1700b20b0 0x7fd1700b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1700b20b0 0x7fd1700b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43706/0 (socket says 172.21.15.67:43706) 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 -- 172.21.15.67:0/737869872 learned_addr learned my addr 172.21.15.67:0/737869872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd174e00640 1 --1- 172.21.15.67:0/737869872 >> v1:172.21.15.67:6789/0 conn(0x7fd17015c1a0 0x7fd1700af470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46996/0 (socket says 172.21.15.67:46996) 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd167e00640 1 --2- 172.21.15.67:0/737869872 >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 0x7fd1700ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 -- 172.21.15.67:0/737869872 >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 msgr2=0x7fd1700ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 --2- 172.21.15.67:0/737869872 >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 0x7fd1700ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 -- 172.21.15.67:0/737869872 >> v1:172.21.15.67:6789/0 conn(0x7fd17015c1a0 legacy=0x7fd1700af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd175800640 1 -- 172.21.15.67:0/737869872 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17015f330 con 0x7fd1700b20b0 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.148+0000 7fd167e00640 1 --2- 172.21.15.67:0/737869872 >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 0x7fd1700ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.149+0000 7fd175800640 1 --2- 172.21.15.67:0/737869872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700b20b0 0x7fd1700b2520 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd1700af200 tx=0x7fd16802b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f202aa0dd1ce1c6d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:51.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.149+0000 7fd16fe00640 1 -- 172.21.15.67:0/737869872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168034070 con 0x7fd1700b20b0 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd16fe00640 1 -- 172.21.15.67:0/737869872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd168030880 con 0x7fd1700b20b0 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd16fe00640 1 -- 172.21.15.67:0/737869872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168030c50 con 0x7fd1700b20b0 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 -- 172.21.15.67:0/737869872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700b20b0 msgr2=0x7fd1700b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 --2- 172.21.15.67:0/737869872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700b20b0 0x7fd1700b2520 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd1700af200 tx=0x7fd16802b9c0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 -- 172.21.15.67:0/737869872 shutdown_connections 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 --2- 172.21.15.67:0/737869872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1700b20b0 0x7fd1700b2520 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 --2- 172.21.15.67:0/737869872 >> v2:172.21.15.187:3300/0 conn(0x7fd17015c580 0x7fd1700ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 -- 172.21.15.67:0/737869872 >> 172.21.15.67:0/737869872 conn(0x7fd170067d10 msgr2=0x7fd1700d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 -- 172.21.15.67:0/737869872 shutdown_connections 2024-04-17T07:30:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.150+0000 7fd176200640 1 -- 172.21.15.67:0/737869872 wait complete. 2024-04-17T07:30:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.152+0000 7fd176200640 1 Processor -- start 2024-04-17T07:30:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.152+0000 7fd176200640 1 -- start start 2024-04-17T07:30:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 0x7fd170153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd17015c1a0 0x7fd1701f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1700afb80 con 0x7fd17015c580 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd17015f8f0 con 0x7fd17015c1a0 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.153+0000 7fd176200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1700afd00 con 0x7fd1700b20b0 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43722/0 (socket says 172.21.15.67:43722) 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 -- 172.21.15.67:0/1272472792 learned_addr learned my addr 172.21.15.67:0/1272472792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd167e00640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd17015c1a0 0x7fd1701f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd174e00640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 0x7fd170153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 -- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 msgr2=0x7fd170153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 0x7fd170153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 -- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd17015c1a0 msgr2=0x7fd1701f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd17015c1a0 0x7fd1701f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd175800640 1 -- 172.21.15.67:0/1272472792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1700adaa0 con 0x7fd17015c580 2024-04-17T07:30:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.154+0000 7fd174e00640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 0x7fd170153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd175800640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd16802ba00 tx=0x7fd168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168048070 con 0x7fd17015c580 2024-04-17T07:30:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd168039d10 con 0x7fd17015c580 2024-04-17T07:30:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168034040 con 0x7fd17015c580 2024-04-17T07:30:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1700add80 con 0x7fd17015c580 2024-04-17T07:30:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.155+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1701540f0 con 0x7fd17015c580 2024-04-17T07:30:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.157+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd170068a10 con 0x7fd17015c580 2024-04-17T07:30:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.158+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fd16804b070 con 0x7fd17015c580 2024-04-17T07:30:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.159+0000 7fd16d600640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 0x7fd14c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.159+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5227+0+0 (secure 0 0 0) 0x7fd1680adf50 con 0x7fd17015c580 2024-04-17T07:30:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.160+0000 7fd174e00640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 0x7fd14c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:51.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.161+0000 7fd174e00640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 0x7fd14c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd17015cea0 tx=0x7fd160009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:51.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.164+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd16807c330 con 0x7fd17015c580 2024-04-17T07:30:51.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.368+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd170157410 con 0x7fd17015c580 2024-04-17T07:30:51.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.925+0000 7fd16d600640 1 -- 172.21.15.67:0/1272472792 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) ==== 170+0+0 (secure 0 0 0) 0x7fd16807bc30 con 0x7fd17015c580 2024-04-17T07:30:51.925 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:30:51.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.929+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 msgr2=0x7fd14c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.929+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 0x7fd14c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd17015cea0 tx=0x7fd160009040 comp rx=0 tx=0).stop 2024-04-17T07:30:51.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.929+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 msgr2=0x7fd1701f7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:51.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.929+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd16802ba00 tx=0x7fd168004730 comp rx=0 tx=0).stop 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 shutdown_connections 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fd14c06ca10 0x7fd14c06eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd17015c580 0x7fd1701f7670 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd17015c1a0 0x7fd1701f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 --2- 172.21.15.67:0/1272472792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1700b20b0 0x7fd170153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 >> 172.21.15.67:0/1272472792 conn(0x7fd170067d10 msgr2=0x7fd17006c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 shutdown_connections 2024-04-17T07:30:51.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:51.930+0000 7fd176200640 1 -- 172.21.15.67:0/1272472792 wait complete. 2024-04-17T07:30:51.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:30:52.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.153+0000 7f6981000640 1 Processor -- start 2024-04-17T07:30:52.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.153+0000 7f6981000640 1 -- start start 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f6981000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f697c06c270 0x7f697c167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f6981000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697c06f1d0 con 0x7f697c0cd920 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f6981000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697c146360 con 0x7f697c145f80 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f6981000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f697c0727a0 con 0x7f697c06c270 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f697be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f697c06c270 0x7f697c167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f697be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f697c06c270 0x7f697c167380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43728/0 (socket says 172.21.15.67:43728) 2024-04-17T07:30:52.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f697aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f697c0cd920 0x7f697c06bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47006/0 (socket says 172.21.15.67:47006) 2024-04-17T07:30:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.155+0000 7f697be00640 1 -- 172.21.15.67:0/2669242898 learned_addr learned my addr 172.21.15.67:0/2669242898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.156+0000 7f697be00640 1 -- 172.21.15.67:0/2669242898 >> v1:172.21.15.187:6789/0 conn(0x7f697c145f80 legacy=0x7f697c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.156+0000 7f697be00640 1 -- 172.21.15.67:0/2669242898 >> v1:172.21.15.67:6789/0 conn(0x7f697c0cd920 legacy=0x7f697c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.156+0000 7f697a000640 1 -- 172.21.15.67:0/2669242898 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788605698 0 0) 0x7f6964001110 con 0x7f697c0cd920 2024-04-17T07:30:52.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.156+0000 7f697be00640 1 -- 172.21.15.67:0/2669242898 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c071e70 con 0x7f697c06c270 2024-04-17T07:30:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.157+0000 7f697be00640 1 --2- 172.21.15.67:0/2669242898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c06c270 0x7f697c167380 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6974009e50 tx=0x7f697402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388d6c6cb75ddbd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.157+0000 7f697a000640 1 -- 172.21.15.67:0/2669242898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697402cb20 con 0x7f697c06c270 2024-04-17T07:30:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f697a000640 1 -- 172.21.15.67:0/2669242898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697402ccc0 con 0x7f697c06c270 2024-04-17T07:30:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f697a000640 1 -- 172.21.15.67:0/2669242898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974034610 con 0x7f697c06c270 2024-04-17T07:30:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 -- 172.21.15.67:0/2669242898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c06c270 msgr2=0x7f697c167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 --2- 172.21.15.67:0/2669242898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c06c270 0x7f697c167380 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6974009e50 tx=0x7f697402bf80 comp rx=0 tx=0).stop 2024-04-17T07:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 -- 172.21.15.67:0/2669242898 shutdown_connections 2024-04-17T07:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 --2- 172.21.15.67:0/2669242898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c06c270 0x7f697c167380 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 -- 172.21.15.67:0/2669242898 >> 172.21.15.67:0/2669242898 conn(0x7f697c0b2190 msgr2=0x7f697c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 -- 172.21.15.67:0/2669242898 shutdown_connections 2024-04-17T07:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.158+0000 7f6981000640 1 -- 172.21.15.67:0/2669242898 wait complete. 2024-04-17T07:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.160+0000 7f6981000640 1 Processor -- start 2024-04-17T07:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.160+0000 7f6981000640 1 -- start start 2024-04-17T07:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f697c06c270 0x7f697c0aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 0x7f697c13b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f697c072c30 con 0x7f697c0cd920 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f697c072430 con 0x7f697c06c270 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f6981000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f697c072db0 con 0x7f697c145f80 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.161+0000 7f697aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43742/0 (socket says 172.21.15.67:43742) 2024-04-17T07:30:52.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 -- 172.21.15.67:0/3491839305 learned_addr learned my addr 172.21.15.67:0/3491839305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697be00640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 0x7f697c13b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 -- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 msgr2=0x7f697c13b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 0x7f697c13b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697b400640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f697c06c270 0x7f697c0aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 -- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f697c06c270 msgr2=0x7f697c0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f697c06c270 0x7f697c0aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 -- 172.21.15.67:0/3491839305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c15f2b0 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697be00640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 0x7f697c13b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.162+0000 7f697aa00640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f696400c780 tx=0x7f696400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.163+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964018070 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.163+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69640070d0 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.163+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69640074c0 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.163+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c15f590 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.163+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c15cd00 con 0x7f697c0cd920 2024-04-17T07:30:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.164+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c068a10 con 0x7f697c0cd920 2024-04-17T07:30:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.166+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f696400a640 con 0x7f697c0cd920 2024-04-17T07:30:52.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.167+0000 7f6973400640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 0x7f695806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:52.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.167+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5609+0+0 (secure 0 0 0) 0x7f696408b0f0 con 0x7f697c0cd920 2024-04-17T07:30:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.168+0000 7f697b400640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 0x7f695806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.169+0000 7f697b400640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 0x7f695806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f60 tx=0x7f696c009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.171+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f697c06f1d0 con 0x7f697c0cd920 2024-04-17T07:30:52.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.419+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f697c0b5280 con 0x7f697c0cd920 2024-04-17T07:30:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.950+0000 7f6973400640 1 -- 172.21.15.67:0/3491839305 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v50) ==== 174+0+0 (secure 0 0 0) 0x7f69640595e0 con 0x7f697c0cd920 2024-04-17T07:30:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 12 and data pool 13 2024-04-17T07:30:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.958+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 msgr2=0x7f695806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.958+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 0x7f695806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f60 tx=0x7f696c009040 comp rx=0 tx=0).stop 2024-04-17T07:30:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.958+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 msgr2=0x7f697c13ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.958+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f696400c780 tx=0x7f696400cc40 comp rx=0 tx=0).stop 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 shutdown_connections 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f695806ca10 0x7f695806eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f697c145f80 0x7f697c13b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f697c0cd920 0x7f697c13ad40 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 --2- 172.21.15.67:0/3491839305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f697c06c270 0x7f697c0aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.960+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 >> 172.21.15.67:0/3491839305 conn(0x7f697c0b2190 msgr2=0x7f697c06ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.961+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 shutdown_connections 2024-04-17T07:30:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:52.961+0000 7f6981000640 1 -- 172.21.15.67:0/3491839305 wait complete. 2024-04-17T07:30:52.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.178+0000 7f0fb3e00640 1 Processor -- start 2024-04-17T07:30:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.178+0000 7f0fb3e00640 1 -- start start 2024-04-17T07:30:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.179+0000 7f0fb3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0fb415c5a0 0x7f0fb415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.179+0000 7f0fb3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0fb40af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.179+0000 7f0fb3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fb415f440 con 0x7f0fb40b46c0 2024-04-17T07:30:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.179+0000 7f0fb3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb40af0a0 con 0x7f0fb415c5a0 2024-04-17T07:30:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.179+0000 7f0fb3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb415f720 con 0x7f0fb40b4210 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fb3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0fb40b46c0 0x7f0fb4071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47018/0 (socket says 172.21.15.67:47018) 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fb3400640 1 -- 172.21.15.67:0/483829720 learned_addr learned my addr 172.21.15.67:0/483829720 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fabe00640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0fb40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fb2a00640 1 --2- 172.21.15.67:0/483829720 >> v2:172.21.15.187:3300/0 conn(0x7f0fb415c5a0 0x7f0fb415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278573527 0 0) 0x7f0fb415f440 con 0x7f0fb40b46c0 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.180+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f98003520 con 0x7f0fb40b46c0 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3621307015 0 0) 0x7f0f98003520 con 0x7f0fb40b46c0 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 msgr2=0x7f0fb40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0fb40af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 >> v2:172.21.15.187:3300/0 conn(0x7f0fb415c5a0 msgr2=0x7f0fb415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 --2- 172.21.15.67:0/483829720 >> v2:172.21.15.187:3300/0 conn(0x7f0fb415c5a0 0x7f0fb415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb415fa00 con 0x7f0fb40b46c0 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0fac0019f0 con 0x7f0fb40b46c0 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 >> v1:172.21.15.67:6789/0 conn(0x7f0fb40b46c0 legacy=0x7f0fb4071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f98003a20 0x7f0f98003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0f98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f98004d40 0x7f0f980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb415f720 con 0x7f0f98004d40 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9800baa0 con 0x7f0f98003a20 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9800bca0 con 0x7f0fb40b4210 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.181+0000 7f0fb3400640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f98004d40 0x7f0f980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fabe00640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0f98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fb3400640 1 -- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 msgr2=0x7f0f98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fb3400640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb40b4210 0x7f0f98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fb3400640 1 -- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f98003a20 msgr2=0x7f0f98003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fb3400640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f98003a20 0x7f0f98003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.182+0000 7f0fb3400640 1 -- 172.21.15.67:0/483829720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f98003520 con 0x7f0f98004d40 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.183+0000 7f0fb3400640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f98004d40 0x7f0f980051c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4073170 tx=0x7f0f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85eeab80d8c2a991 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.183+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00a480 con 0x7f0f98004d40 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.183+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac034630 con 0x7f0f98004d40 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.183+0000 7f0fb2000640 1 -- 172.21.15.67:0/483829720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac034a40 con 0x7f0f98004d40 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 -- 172.21.15.67:0/483829720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f98004d40 msgr2=0x7f0f980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/483829720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f98004d40 0x7f0f980051c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4073170 tx=0x7f0f98005830 comp rx=0 tx=0).stop 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3400640 1 -- 172.21.15.67:0/483829720 reap_dead start 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 -- 172.21.15.67:0/483829720 shutdown_connections 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 -- 172.21.15.67:0/483829720 >> 172.21.15.67:0/483829720 conn(0x7f0fb4067d10 msgr2=0x7f0fb40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 -- 172.21.15.67:0/483829720 shutdown_connections 2024-04-17T07:30:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.184+0000 7f0fb3e00640 1 -- 172.21.15.67:0/483829720 wait complete. 2024-04-17T07:30:53.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.186+0000 7f0fb3e00640 1 Processor -- start 2024-04-17T07:30:53.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.186+0000 7f0fb3e00640 1 -- start start 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 0x7f0fb41537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 0x7f0fb415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb40adc30 con 0x7f0fb41555e0 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb40adab0 con 0x7f0fb4146f70 2024-04-17T07:30:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fb3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb406e670 con 0x7f0fb415a880 2024-04-17T07:30:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43766/0 (socket says 172.21.15.67:43766) 2024-04-17T07:30:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.187+0000 7f0fabe00640 1 -- 172.21.15.67:0/1910526448 learned_addr learned my addr 172.21.15.67:0/1910526448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 -- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 msgr2=0x7f0fb415ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fb3400640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 0x7f0fb415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fb2a00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 0x7f0fb41537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 0x7f0fb415ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 -- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 msgr2=0x7f0fb41537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 0x7f0fb41537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 -- 172.21.15.67:0/1910526448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb4155ae0 con 0x7f0fb41555e0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fb3400640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 0x7f0fb415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fb2a00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 0x7f0fb41537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0fabe00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c011600 tx=0x7f0f9c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.188+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00f920 con 0x7f0fb41555e0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.189+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f9c00fac0 con 0x7f0fb41555e0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.189+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb4155dc0 con 0x7f0fb41555e0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.189+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00fe70 con 0x7f0fb41555e0 2024-04-17T07:30:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.189+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb4156380 con 0x7f0fb41555e0 2024-04-17T07:30:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.190+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb4068a10 con 0x7f0fb41555e0 2024-04-17T07:30:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.191+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f0f9c015460 con 0x7f0fb41555e0 2024-04-17T07:30:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.192+0000 7f0faaa00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 0x7f0f9006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.192+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5709+0+0 (secure 0 0 0) 0x7f0f9c08c740 con 0x7f0fb41555e0 2024-04-17T07:30:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.193+0000 7f0fb2a00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 0x7f0f9006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.194+0000 7f0fb2a00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 0x7f0f9006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a40 tx=0x7f0fa4008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.197+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0f9c019030 con 0x7f0fb41555e0 2024-04-17T07:30:53.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.404+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fb40cf860 con 0x7f0fb41555e0 2024-04-17T07:30:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.407+0000 7f0faaa00640 1 -- 172.21.15.67:0/1910526448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+16636 (secure 0 0 0) 0x7f0f9c05a9c0 con 0x7f0fb41555e0 2024-04-17T07:30:53.406 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:53.406 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":110,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:52.420052+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:53.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 msgr2=0x7f0f9006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 0x7f0f9006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a40 tx=0x7f0fa4008040 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 msgr2=0x7f0fb4156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c011600 tx=0x7f0f9c011ac0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 shutdown_connections 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f0f9006ca10 0x7f0f9006eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fb415a880 0x7f0fb415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb41555e0 0x7f0fb4156b80 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 --2- 172.21.15.67:0/1910526448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fb4146f70 0x7f0fb41537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 >> 172.21.15.67:0/1910526448 conn(0x7f0fb4067d10 msgr2=0x7f0fb40b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 shutdown_connections 2024-04-17T07:30:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.409+0000 7f0fb3e00640 1 -- 172.21.15.67:0/1910526448 wait complete. 2024-04-17T07:30:53.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.647+0000 7f2c4e400640 1 Processor -- start 2024-04-17T07:30:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.647+0000 7f2c4e400640 1 -- start start 2024-04-17T07:30:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.648+0000 7f2c4e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c4815cac0 0x7f2c480d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.648+0000 7f2c4e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c48160b70 0x7f2c4815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.648+0000 7f2c4e400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4809aa60 con 0x7f2c4809bae0 2024-04-17T07:30:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.648+0000 7f2c4e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4809b650 con 0x7f2c4815cac0 2024-04-17T07:30:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.648+0000 7f2c4e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4809ad20 con 0x7f2c48160b70 2024-04-17T07:30:53.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c47e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c4809bae0 0x7f2c48160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54658/0 (socket says 172.21.15.67:54658) 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c47e00640 1 -- 172.21.15.67:0/804995431 learned_addr learned my addr 172.21.15.67:0/804995431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c4815cac0 0x7f2c480d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4da00640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c48160b70 0x7f2c4815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c47400640 1 -- 172.21.15.67:0/804995431 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433083096 0 0) 0x7f2c4809aa60 con 0x7f2c4809bae0 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c47400640 1 -- 172.21.15.67:0/804995431 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c2c003520 con 0x7f2c4809bae0 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4d000640 1 -- 172.21.15.67:0/804995431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c48160b70 msgr2=0x7f2c4815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4d000640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c48160b70 0x7f2c4815cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4d000640 1 -- 172.21.15.67:0/804995431 >> v1:172.21.15.67:6789/0 conn(0x7f2c4809bae0 legacy=0x7f2c48160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.649+0000 7f2c4d000640 1 -- 172.21.15.67:0/804995431 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4809b000 con 0x7f2c4815cac0 2024-04-17T07:30:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c4d000640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4815cac0 0x7f2c480d0ba0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009d90 tx=0x7f2c3802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5dab626dc844da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c47400640 1 -- 172.21.15.67:0/804995431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38039070 con 0x7f2c4815cac0 2024-04-17T07:30:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c47400640 1 -- 172.21.15.67:0/804995431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c3802cc60 con 0x7f2c4815cac0 2024-04-17T07:30:53.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c47400640 1 -- 172.21.15.67:0/804995431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38034660 con 0x7f2c4815cac0 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c4e400640 1 -- 172.21.15.67:0/804995431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4815cac0 msgr2=0x7f2c480d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.651+0000 7f2c4e400640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4815cac0 0x7f2c480d0ba0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009d90 tx=0x7f2c3802beb0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 -- 172.21.15.67:0/804995431 shutdown_connections 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c48160b70 0x7f2c4815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 --2- 172.21.15.67:0/804995431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4815cac0 0x7f2c480d0ba0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 -- 172.21.15.67:0/804995431 >> 172.21.15.67:0/804995431 conn(0x7f2c480b40e0 msgr2=0x7f2c480b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 -- 172.21.15.67:0/804995431 shutdown_connections 2024-04-17T07:30:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.652+0000 7f2c4e400640 1 -- 172.21.15.67:0/804995431 wait complete. 2024-04-17T07:30:53.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.653+0000 7f2c4e400640 1 Processor -- start 2024-04-17T07:30:53.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.653+0000 7f2c4e400640 1 -- start start 2024-04-17T07:30:53.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4815cac0 0x7f2c481f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 0x7f2c48157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c480d10e0 con 0x7f2c4809bae0 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c480b1bc0 con 0x7f2c48160b70 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.654+0000 7f2c4e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c480d1260 con 0x7f2c4815cac0 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51650/0 (socket says 172.21.15.67:51650) 2024-04-17T07:30:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 -- 172.21.15.67:0/3162497614 learned_addr learned my addr 172.21.15.67:0/3162497614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:53.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c47e00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4815cac0 0x7f2c481f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4da00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 0x7f2c48157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 -- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4815cac0 msgr2=0x7f2c481f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4815cac0 0x7f2c481f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 -- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 msgr2=0x7f2c48157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 0x7f2c48157460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4d000640 1 -- 172.21.15.67:0/3162497614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c481f6960 con 0x7f2c4809bae0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.655+0000 7f2c4da00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 0x7f2c48157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c4d000640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009d90 tx=0x7f2c38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c38039050 con 0x7f2c4809bae0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c38045070 con 0x7f2c4809bae0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3803e850 con 0x7f2c4809bae0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c481f6c40 con 0x7f2c4809bae0 2024-04-17T07:30:53.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.656+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c480cad60 con 0x7f2c4809bae0 2024-04-17T07:30:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.657+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c480755d0 con 0x7f2c4809bae0 2024-04-17T07:30:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.658+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f2c3803e9f0 con 0x7f2c4809bae0 2024-04-17T07:30:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.659+0000 7f2c44c00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 0x7f2c2806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.659+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5709+0+0 (secure 0 0 0) 0x7f2c380aeb50 con 0x7f2c4809bae0 2024-04-17T07:30:53.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.660+0000 7f2c47e00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 0x7f2c2806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:53.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.661+0000 7f2c47e00640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 0x7f2c2806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c480cab10 tx=0x7f2c34009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:53.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.664+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c3807cf80 con 0x7f2c4809bae0 2024-04-17T07:30:53.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.872+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c480b4e20 con 0x7f2c4809bae0 2024-04-17T07:30:53.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.875+0000 7f2c44c00640 1 -- 172.21.15.67:0/3162497614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+16636 (secure 0 0 0) 0x7f2c3807c880 con 0x7f2c4809bae0 2024-04-17T07:30:53.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:53.875 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":110,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:52.420052+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 msgr2=0x7f2c2806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 0x7f2c2806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c480cab10 tx=0x7f2c34009040 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 msgr2=0x7f2c480b8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009d90 tx=0x7f2c38004a40 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 shutdown_connections 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f2c2806ca10 0x7f2c2806eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.877+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c48160b70 0x7f2c48157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.878+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c4815cac0 0x7f2c481f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.878+0000 7f2c4e400640 1 --2- 172.21.15.67:0/3162497614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4809bae0 0x7f2c480b8040 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.878+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 >> 172.21.15.67:0/3162497614 conn(0x7f2c480b40e0 msgr2=0x7f2c480716b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.878+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 shutdown_connections 2024-04-17T07:30:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:53.878+0000 7f2c4e400640 1 -- 172.21.15.67:0/3162497614 wait complete. 2024-04-17T07:30:53.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:30:53.899 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-17T07:30:53.899+0000 7f8874400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:30:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.123+0000 7f54a9600640 1 Processor -- start 2024-04-17T07:30:54.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.123+0000 7f54a9600640 1 -- start start 2024-04-17T07:30:54.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.124+0000 7f54a9600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f54a4145f80 0x7f54a40cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a40cd750 0x7f54a406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a9600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a406f1d0 con 0x7f54a406c0a0 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a9600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f54a40727a0 con 0x7f54a40cd750 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a9600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f54a4146360 con 0x7f54a4145f80 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a40cd750 0x7f54a406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54a40cd750 0x7f54a406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51658/0 (socket says 172.21.15.67:51658) 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a3400640 1 -- 172.21.15.67:0/3390832695 learned_addr learned my addr 172.21.15.67:0/3390832695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a8c00640 1 --1- 172.21.15.67:0/3390832695 >> v1:172.21.15.67:6789/0 conn(0x7f54a406c0a0 0x7f54a4160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54668/0 (socket says 172.21.15.67:54668) 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.125+0000 7f54a3e00640 1 --2- 172.21.15.67:0/3390832695 >> v2:172.21.15.187:3300/0 conn(0x7f54a4145f80 0x7f54a40cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.126+0000 7f54a3400640 1 -- 172.21.15.67:0/3390832695 >> v2:172.21.15.187:3300/0 conn(0x7f54a4145f80 msgr2=0x7f54a40cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.126+0000 7f54a3400640 1 --2- 172.21.15.67:0/3390832695 >> v2:172.21.15.187:3300/0 conn(0x7f54a4145f80 0x7f54a40cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.126+0000 7f54a3400640 1 -- 172.21.15.67:0/3390832695 >> v1:172.21.15.67:6789/0 conn(0x7f54a406c0a0 legacy=0x7f54a4160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.126+0000 7f54a3400640 1 -- 172.21.15.67:0/3390832695 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a4071e70 con 0x7f54a40cd750 2024-04-17T07:30:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.127+0000 7f54a3400640 1 --2- 172.21.15.67:0/3390832695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a40cd750 0x7f54a406bb60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c9c0 tx=0x7f549002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6927e89e5193cbc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.127+0000 7f54a2a00640 1 -- 172.21.15.67:0/3390832695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490035020 con 0x7f54a40cd750 2024-04-17T07:30:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.127+0000 7f54a2a00640 1 -- 172.21.15.67:0/3390832695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5490031870 con 0x7f54a40cd750 2024-04-17T07:30:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.127+0000 7f54a2a00640 1 -- 172.21.15.67:0/3390832695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490031c40 con 0x7f54a40cd750 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 -- 172.21.15.67:0/3390832695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a40cd750 msgr2=0x7f54a406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 --2- 172.21.15.67:0/3390832695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a40cd750 0x7f54a406bb60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f54a406c9c0 tx=0x7f549002b910 comp rx=0 tx=0).stop 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 -- 172.21.15.67:0/3390832695 shutdown_connections 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 --2- 172.21.15.67:0/3390832695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a40cd750 0x7f54a406bb60 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 --2- 172.21.15.67:0/3390832695 >> v2:172.21.15.187:3300/0 conn(0x7f54a4145f80 0x7f54a40cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 -- 172.21.15.67:0/3390832695 >> 172.21.15.67:0/3390832695 conn(0x7f54a40b2190 msgr2=0x7f54a40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 -- 172.21.15.67:0/3390832695 shutdown_connections 2024-04-17T07:30:54.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.128+0000 7f54a9600640 1 -- 172.21.15.67:0/3390832695 wait complete. 2024-04-17T07:30:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.130+0000 7f54a9600640 1 Processor -- start 2024-04-17T07:30:54.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.130+0000 7f54a9600640 1 -- start start 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40cd750 0x7f54a41e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a4145f80 0x7f54a41e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4072c30 con 0x7f54a406c0a0 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4072430 con 0x7f54a40cd750 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54a4072db0 con 0x7f54a4145f80 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51668/0 (socket says 172.21.15.67:51668) 2024-04-17T07:30:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 -- 172.21.15.67:0/3084760022 learned_addr learned my addr 172.21.15.67:0/3084760022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 -- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a4145f80 msgr2=0x7f54a41e5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a8c00640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a4145f80 0x7f54a41e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3400640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40cd750 0x7f54a41e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a4145f80 0x7f54a41e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 -- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40cd750 msgr2=0x7f54a41e55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40cd750 0x7f54a41e55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.131+0000 7f54a3e00640 1 -- 172.21.15.67:0/3084760022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a413fdd0 con 0x7f54a406c0a0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f54a3e00640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f549400c7f0 tx=0x7f549400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494015070 con 0x7f54a406c0a0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494004220 con 0x7f54a406c0a0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494007620 con 0x7f54a406c0a0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a41400b0 con 0x7f54a406c0a0 2024-04-17T07:30:54.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.132+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a4140670 con 0x7f54a406c0a0 2024-04-17T07:30:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.134+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a4068a10 con 0x7f54a406c0a0 2024-04-17T07:30:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.134+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f54940079d0 con 0x7f54a406c0a0 2024-04-17T07:30:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.135+0000 7f549be00640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 0x7f548406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.135+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f549408c1d0 con 0x7f54a406c0a0 2024-04-17T07:30:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.136+0000 7f54a3400640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 0x7f548406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:54.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.137+0000 7f54a3400640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 0x7f548406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5490036000 tx=0x7f5490008030 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.140+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f549405a3e0 con 0x7f54a406c0a0 2024-04-17T07:30:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.390+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f54a40afc40 con 0x7f54a406c0a0 2024-04-17T07:30:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.977+0000 7f549be00640 1 -- 172.21.15.67:0/3084760022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) ==== 118+0+0 (secure 0 0 0) 0x7f5494059ce0 con 0x7f54a406c0a0 2024-04-17T07:30:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 msgr2=0x7f548406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 0x7f548406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5490036000 tx=0x7f5490008030 comp rx=0 tx=0).stop 2024-04-17T07:30:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 msgr2=0x7f54a413ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f549400c7f0 tx=0x7f549400ccb0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 shutdown_connections 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f548406ca10 0x7f548406eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54a4145f80 0x7f54a41e5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54a40cd750 0x7f54a41e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 --2- 172.21.15.67:0/3084760022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a406c0a0 0x7f54a413ce60 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 >> 172.21.15.67:0/3084760022 conn(0x7f54a40b2190 msgr2=0x7f54a415cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 shutdown_connections 2024-04-17T07:30:54.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:54.981+0000 7f54a9600640 1 -- 172.21.15.67:0/3084760022 wait complete. 2024-04-17T07:30:55.002 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:30:55.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:30:55.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.232+0000 7f0043e00640 1 Processor -- start 2024-04-17T07:30:55.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.232+0000 7f0043e00640 1 -- start start 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0043e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f0044162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0043e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004415f780 con 0x7f0044159020 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0043e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004415fa60 con 0x7f004406b8c0 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0043e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f004415f540 con 0x7f00440af400 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0042a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0044159020 0x7f004415ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54676/0 (socket says 172.21.15.67:54676) 2024-04-17T07:30:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.234+0000 7f0042a00640 1 -- 172.21.15.67:0/1303922020 learned_addr learned my addr 172.21.15.67:0/1303922020 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f003a000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f0044162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499982835 0 0) 0x7f004415f780 con 0x7f0044159020 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0028003520 con 0x7f0044159020 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2865441000 0 0) 0x7f0028003520 con 0x7f0044159020 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 msgr2=0x7f0044162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f0044162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 >> v1:172.21.15.187:6789/0 conn(0x7f004406b8c0 legacy=0x7f00440adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00441624a0 con 0x7f0044159020 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f00340019f0 con 0x7f0044159020 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 >> v1:172.21.15.67:6789/0 conn(0x7f0044159020 legacy=0x7f004415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f00280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0028003fb0 0x7f0028004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0028004d40 0x7f00280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.235+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f004415f540 con 0x7f0028003fb0 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f002800bbb0 con 0x7f0028004d40 2024-04-17T07:30:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f002800bdb0 con 0x7f00440af400 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0028003fb0 0x7f0028004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f0042a00640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f00280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f0043400640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0028004d40 0x7f00280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 -- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 msgr2=0x7f00280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00440af400 0x7f00280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 -- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0028004d40 msgr2=0x7f00280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0028004d40 0x7f00280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.236+0000 7f003a000640 1 -- 172.21.15.67:0/1303922020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028003520 con 0x7f0028003fb0 2024-04-17T07:30:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f003a000640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0028003fb0 0x7f0028004800 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f002c009a90 tx=0x7f0028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c4e5647a4fd120 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c039070 con 0x7f0028003fb0 2024-04-17T07:30:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c02c680 con 0x7f0028003fb0 2024-04-17T07:30:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0042000640 1 -- 172.21.15.67:0/1303922020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c009690 con 0x7f0028003fb0 2024-04-17T07:30:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0043e00640 1 -- 172.21.15.67:0/1303922020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0028003fb0 msgr2=0x7f0028004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0043e00640 1 --2- 172.21.15.67:0/1303922020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0028003fb0 0x7f0028004800 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f002c009a90 tx=0x7f0028005830 comp rx=0 tx=0).stop 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.238+0000 7f0043400640 1 -- 172.21.15.67:0/1303922020 reap_dead start 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.239+0000 7f0043e00640 1 -- 172.21.15.67:0/1303922020 shutdown_connections 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.239+0000 7f0043e00640 1 -- 172.21.15.67:0/1303922020 >> 172.21.15.67:0/1303922020 conn(0x7f00440b2190 msgr2=0x7f00440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.239+0000 7f0043e00640 1 -- 172.21.15.67:0/1303922020 shutdown_connections 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.239+0000 7f0043e00640 1 -- 172.21.15.67:0/1303922020 wait complete. 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.240+0000 7f0043e00640 1 Processor -- start 2024-04-17T07:30:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.241+0000 7f0043e00640 1 -- start start 2024-04-17T07:30:55.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.241+0000 7f0043e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0028003fb0 0x7f0044155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.241+0000 7f0043e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 0x7f004406e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.241+0000 7f0043e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00440ae250 con 0x7f004406e640 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f004415fd40 con 0x7f00441f46b0 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00440ae3d0 con 0x7f0028003fb0 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51684/0 (socket says 172.21.15.67:51684) 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 -- 172.21.15.67:0/989612124 learned_addr learned my addr 172.21.15.67:0/989612124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0042a00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0028003fb0 0x7f0044155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f003a000640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 0x7f004406e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 -- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0028003fb0 msgr2=0x7f0044155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0028003fb0 0x7f0044155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 -- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 msgr2=0x7f004406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 0x7f004406e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f0043400640 1 -- 172.21.15.67:0/989612124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00441f4c70 con 0x7f004406e640 2024-04-17T07:30:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.242+0000 7f003a000640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 0x7f004406e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f0043400640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0044070020 tx=0x7f003c00e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c019070 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c005280 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c007030 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00441f4f50 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.243+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441f5510 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.244+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0044068a10 con 0x7f004406e640 2024-04-17T07:30:55.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.245+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f003c01c070 con 0x7f004406e640 2024-04-17T07:30:55.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.246+0000 7f003b400640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 0x7f002006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:55.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.246+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f003c08ab20 con 0x7f004406e640 2024-04-17T07:30:55.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.247+0000 7f0042a00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 0x7f002006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:55.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.248+0000 7f0042a00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 0x7f002006ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f00340013d0 tx=0x7f0034009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.250+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f003c058f20 con 0x7f004406e640 2024-04-17T07:30:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.497+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f00440cfd90 con 0x7f004406e640 2024-04-17T07:30:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.980+0000 7f003b400640 1 -- 172.21.15.67:0/989612124 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) ==== 122+0+0 (secure 0 0 0) 0x7f003c058820 con 0x7f004406e640 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.984+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 msgr2=0x7f002006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.984+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 0x7f002006ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f00340013d0 tx=0x7f0034009040 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.984+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 msgr2=0x7f0044159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.984+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0044070020 tx=0x7f003c00e8a0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 shutdown_connections 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f002006ca60 0x7f002006ef20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004406e640 0x7f0044159400 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00441f46b0 0x7f004406e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 --2- 172.21.15.67:0/989612124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0028003fb0 0x7f0044155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 >> 172.21.15.67:0/989612124 conn(0x7f00440b2190 msgr2=0x7f004415c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:55.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 shutdown_connections 2024-04-17T07:30:55.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:55.985+0000 7f0043e00640 1 -- 172.21.15.67:0/989612124 wait complete. 2024-04-17T07:30:56.008 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:30:56.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:56.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.236+0000 7fc216400640 1 Processor -- start 2024-04-17T07:30:56.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.236+0000 7fc216400640 1 -- start start 2024-04-17T07:30:56.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc216400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc21006bfd0 0x7fc2100cd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc216400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2100b3760 0x7fc2100d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc216400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc21015a750 con 0x7fc21015a370 2024-04-17T07:30:56.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc216400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc2100b3f60 con 0x7fc2100b3760 2024-04-17T07:30:56.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc216400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc21015cd40 con 0x7fc21006bfd0 2024-04-17T07:30:56.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc215000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc21015a370 0x7fc21006b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54692/0 (socket says 172.21.15.67:54692) 2024-04-17T07:30:56.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc215000640 1 -- 172.21.15.67:0/372491184 learned_addr learned my addr 172.21.15.67:0/372491184 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:56.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.237+0000 7fc215a00640 1 --2- 172.21.15.67:0/372491184 >> v2:172.21.15.67:3300/0 conn(0x7fc2100b3760 0x7fc2100d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc20fe00640 1 --2- 172.21.15.67:0/372491184 >> v2:172.21.15.187:3300/0 conn(0x7fc21006bfd0 0x7fc2100cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc20f400640 1 -- 172.21.15.67:0/372491184 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387762737 0 0) 0x7fc21015a750 con 0x7fc21015a370 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc20f400640 1 -- 172.21.15.67:0/372491184 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f8003520 con 0x7fc21015a370 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc215a00640 1 -- 172.21.15.67:0/372491184 >> v2:172.21.15.187:3300/0 conn(0x7fc21006bfd0 msgr2=0x7fc2100cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc215a00640 1 --2- 172.21.15.67:0/372491184 >> v2:172.21.15.187:3300/0 conn(0x7fc21006bfd0 0x7fc2100cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc215a00640 1 -- 172.21.15.67:0/372491184 >> v1:172.21.15.67:6789/0 conn(0x7fc21015a370 legacy=0x7fc21006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.238+0000 7fc215a00640 1 -- 172.21.15.67:0/372491184 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21015d000 con 0x7fc2100b3760 2024-04-17T07:30:56.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.239+0000 7fc215a00640 1 --2- 172.21.15.67:0/372491184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2100b3760 0x7fc2100d04a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc210072e20 tx=0x7fc20802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8efadac9b3fcec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc20f400640 1 -- 172.21.15.67:0/372491184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208039070 con 0x7fc2100b3760 2024-04-17T07:30:56.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc20f400640 1 -- 172.21.15.67:0/372491184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20802cbf0 con 0x7fc2100b3760 2024-04-17T07:30:56.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc20f400640 1 -- 172.21.15.67:0/372491184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208034440 con 0x7fc2100b3760 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 -- 172.21.15.67:0/372491184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2100b3760 msgr2=0x7fc2100d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 --2- 172.21.15.67:0/372491184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2100b3760 0x7fc2100d04a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc210072e20 tx=0x7fc20802be40 comp rx=0 tx=0).stop 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 -- 172.21.15.67:0/372491184 shutdown_connections 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 --2- 172.21.15.67:0/372491184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2100b3760 0x7fc2100d04a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 --2- 172.21.15.67:0/372491184 >> v2:172.21.15.187:3300/0 conn(0x7fc21006bfd0 0x7fc2100cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 -- 172.21.15.67:0/372491184 >> 172.21.15.67:0/372491184 conn(0x7fc21009aa60 msgr2=0x7fc2100b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 -- 172.21.15.67:0/372491184 shutdown_connections 2024-04-17T07:30:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.240+0000 7fc216400640 1 -- 172.21.15.67:0/372491184 wait complete. 2024-04-17T07:30:56.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.242+0000 7fc216400640 1 Processor -- start 2024-04-17T07:30:56.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.242+0000 7fc216400640 1 -- start start 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 0x7fc210070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2100b3760 0x7fc2100b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc21015d8a0 con 0x7fc21015a370 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc21015d5c0 con 0x7fc21006bfd0 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.243+0000 7fc216400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc210160330 con 0x7fc2100b3760 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51700/0 (socket says 172.21.15.67:51700) 2024-04-17T07:30:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 -- 172.21.15.67:0/3842859420 learned_addr learned my addr 172.21.15.67:0/3842859420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215000640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 0x7fc210070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc20fe00640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2100b3760 0x7fc2100b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 -- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2100b3760 msgr2=0x7fc2100b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2100b3760 0x7fc2100b6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 -- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 msgr2=0x7fc210070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 0x7fc210070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 -- 172.21.15.67:0/3842859420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2101581d0 con 0x7fc21015a370 2024-04-17T07:30:56.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215000640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 0x7fc210070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.244+0000 7fc215a00640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc208009e80 tx=0x7fc208004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.245+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208039050 con 0x7fc21015a370 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.245+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc208045070 con 0x7fc21015a370 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.245+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208009450 con 0x7fc21015a370 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.245+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21006f060 con 0x7fc21015a370 2024-04-17T07:30:56.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.245+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2100b4560 con 0x7fc21015a370 2024-04-17T07:30:56.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.246+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2100755d0 con 0x7fc21015a370 2024-04-17T07:30:56.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.248+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fc20803e850 con 0x7fc21015a370 2024-04-17T07:30:56.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.248+0000 7fc20cc00640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 0x7fc1f006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.249+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7fc2080ae1f0 con 0x7fc21015a370 2024-04-17T07:30:56.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.249+0000 7fc215000640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 0x7fc1f006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.251+0000 7fc215000640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 0x7fc1f006ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc204005880 tx=0x7fc204009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.253+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc20804e020 con 0x7fc21015a370 2024-04-17T07:30:56.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.501+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2100b7210 con 0x7fc21015a370 2024-04-17T07:30:56.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.502+0000 7fc20cc00640 1 -- 172.21.15.67:0/3842859420 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7fc2101581d0 con 0x7fc21015a370 2024-04-17T07:30:56.502 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:56.502 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:30:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 msgr2=0x7fc1f006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 0x7fc1f006ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc204005880 tx=0x7fc204009040 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 msgr2=0x7fc2100b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc208009e80 tx=0x7fc208004a40 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 shutdown_connections 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fc1f006ca40 0x7fc1f006ef00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.505+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc21015a370 0x7fc2100b2f20 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.506+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2100b3760 0x7fc2100b6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.506+0000 7fc216400640 1 --2- 172.21.15.67:0/3842859420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc21006bfd0 0x7fc210070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.506+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 >> 172.21.15.67:0/3842859420 conn(0x7fc21009aa60 msgr2=0x7fc2100d1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:56.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.506+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 shutdown_connections 2024-04-17T07:30:56.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.506+0000 7fc216400640 1 -- 172.21.15.67:0/3842859420 wait complete. 2024-04-17T07:30:56.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:30:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.750+0000 7f1cb7e00640 1 Processor -- start 2024-04-17T07:30:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.750+0000 7f1cb7e00640 1 -- start start 2024-04-17T07:30:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb806b8c0 0x7f1cb80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cb80cd4a0 con 0x7f1cb8160280 2024-04-17T07:30:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cb81607f0 con 0x7f1cb8141ed0 2024-04-17T07:30:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb815ce30 con 0x7f1cb806b8c0 2024-04-17T07:30:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb6000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1cb8160280 0x7f1cb806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54706/0 (socket says 172.21.15.67:54706) 2024-04-17T07:30:56.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.752+0000 7f1cb6000640 1 -- 172.21.15.67:0/4193855795 learned_addr learned my addr 172.21.15.67:0/4193855795 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb6a00640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb806b8c0 0x7f1cb80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633462460 0 0) 0x7f1cb80cd4a0 con 0x7f1cb8160280 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c9c003520 con 0x7f1cb8160280 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077192666 0 0) 0x7f1cb81607f0 con 0x7f1cb8141ed0 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb80cd4a0 con 0x7f1cb8141ed0 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1080730363 0 0) 0x7f1c9c003520 con 0x7f1cb8160280 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb806b8c0 msgr2=0x7f1cb80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.753+0000 7f1cb5600640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb806b8c0 0x7f1cb80cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 >> v1:172.21.15.187:6789/0 conn(0x7f1cb8141ed0 legacy=0x7f1cb815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb8160ad0 con 0x7f1cb8160280 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1ca4001670 con 0x7f1cb8160280 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 >> v1:172.21.15.67:6789/0 conn(0x7f1cb8160280 legacy=0x7f1cb806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c9c003be0 0x7f1c9c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1c9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c9c0051c0 0x7f1c9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb815ce30 con 0x7f1c9c004800 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c00c010 con 0x7f1c9c0051c0 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9c00c210 con 0x7f1c9c003be0 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb6000640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1c9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb6a00640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c9c003be0 0x7f1c9c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb7400640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c9c0051c0 0x7f1c9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.754+0000 7f1cb6000640 1 -- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c9c003be0 msgr2=0x7f1c9c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.755+0000 7f1cb6000640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1c9c003be0 0x7f1c9c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.755+0000 7f1cb6000640 1 -- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c9c0051c0 msgr2=0x7f1c9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.755+0000 7f1cb6000640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c9c0051c0 0x7f1c9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.755+0000 7f1cb6000640 1 -- 172.21.15.67:0/4193855795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c003520 con 0x7f1c9c004800 2024-04-17T07:30:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.756+0000 7f1cb6000640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1c9c004c80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400c190 tx=0x7f1c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f227324e11a667 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.756+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4007ce0 con 0x7f1c9c004800 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.756+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca4030070 con 0x7f1c9c004800 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.756+0000 7f1cb5600640 1 -- 172.21.15.67:0/4193855795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4005d90 con 0x7f1c9c004800 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 -- 172.21.15.67:0/4193855795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 msgr2=0x7f1c9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/4193855795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1c9c004c80 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400c190 tx=0x7f1c9c005830 comp rx=0 tx=0).stop 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7400640 1 -- 172.21.15.67:0/4193855795 reap_dead start 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 -- 172.21.15.67:0/4193855795 shutdown_connections 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 -- 172.21.15.67:0/4193855795 >> 172.21.15.67:0/4193855795 conn(0x7f1cb8067d10 msgr2=0x7f1cb80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 -- 172.21.15.67:0/4193855795 shutdown_connections 2024-04-17T07:30:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.757+0000 7f1cb7e00640 1 -- 172.21.15.67:0/4193855795 wait complete. 2024-04-17T07:30:56.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.759+0000 7f1cb7e00640 1 Processor -- start 2024-04-17T07:30:56.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.759+0000 7f1cb7e00640 1 -- start start 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb8133010 0x7f1cb8142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb815e210 0x7f1cb815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb813f9b0 con 0x7f1c9c004800 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8161090 con 0x7f1cb8133010 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1cb813fb30 con 0x7f1cb815e210 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51716/0 (socket says 172.21.15.67:51716) 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 -- 172.21.15.67:0/2047982699 learned_addr learned my addr 172.21.15.67:0/2047982699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 -- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb815e210 msgr2=0x7f1cb815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb815e210 0x7f1cb815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 -- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb8133010 msgr2=0x7f1cb8142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6000640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb8133010 0x7f1cb8142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb8133010 0x7f1cb8142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.760+0000 7f1cb6a00640 1 -- 172.21.15.67:0/2047982699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb8144920 con 0x7f1c9c004800 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1cb6a00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80115d0 tx=0x7f1ca8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800f8f0 con 0x7f1c9c004800 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca800fa90 con 0x7f1c9c004800 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800fe80 con 0x7f1c9c004800 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb8143500 con 0x7f1c9c004800 2024-04-17T07:30:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.761+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81e74c0 con 0x7f1c9c004800 2024-04-17T07:30:56.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.762+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb81f25e0 con 0x7f1c9c004800 2024-04-17T07:30:56.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.764+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f1ca8015460 con 0x7f1c9c004800 2024-04-17T07:30:56.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.765+0000 7f1caea00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 0x7f1c9806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:56.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.765+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f1ca808c5d0 con 0x7f1c9c004800 2024-04-17T07:30:56.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.766+0000 7f1cb6000640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 0x7f1c9806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:56.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.767+0000 7f1cb6000640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 0x7f1c9806ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4034000 tx=0x7f1ca4007000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:56.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.770+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1cb8144920 con 0x7f1c9c004800 2024-04-17T07:30:56.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.973+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cb8068a10 con 0x7f1c9c004800 2024-04-17T07:30:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.976+0000 7f1caea00640 1 -- 172.21.15.67:0/2047982699 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+16623 (secure 0 0 0) 0x7f1ca8019030 con 0x7f1c9c004800 2024-04-17T07:30:56.976 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:56.976 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":111,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:30:53.950333+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:30:56.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 msgr2=0x7f1c9806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 0x7f1c9806ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4034000 tx=0x7f1ca4007000 comp rx=0 tx=0).stop 2024-04-17T07:30:56.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 msgr2=0x7f1cb81443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80115d0 tx=0x7f1ca8011a90 comp rx=0 tx=0).stop 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 shutdown_connections 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f1c9806ca40 0x7f1c9806ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1cb815e210 0x7f1cb815e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1cb8133010 0x7f1cb8142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 --2- 172.21.15.67:0/2047982699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c004800 0x7f1cb81443e0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 >> 172.21.15.67:0/2047982699 conn(0x7f1cb8067d10 msgr2=0x7f1cb806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 shutdown_connections 2024-04-17T07:30:56.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:56.980+0000 7f1cb7e00640 1 -- 172.21.15.67:0/2047982699 wait complete. 2024-04-17T07:30:57.004 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:30:57.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:30:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.226+0000 7fb4f1000640 1 Processor -- start 2024-04-17T07:30:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.226+0000 7fb4f1000640 1 -- start start 2024-04-17T07:30:57.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec160280 0x7fb4ec06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb4ec141ed0 0x7fb4ec142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb4ec15ce30 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb4ec0cd4a0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4f1000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4ec1607f0 con 0x7fb4ec160280 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4eb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4eb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51728/0 (socket says 172.21.15.67:51728) 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.228+0000 7fb4eb400640 1 -- 172.21.15.67:0/3321840837 learned_addr learned my addr 172.21.15.67:0/3321840837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4eb400640 1 -- 172.21.15.67:0/3321840837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec160280 msgr2=0x7fb4ec06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4e3e00640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec160280 0x7fb4ec06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4eb400640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec160280 0x7fb4ec06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4eb400640 1 -- 172.21.15.67:0/3321840837 >> v2:172.21.15.187:3300/0 conn(0x7fb4ec141ed0 msgr2=0x7fb4ec142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4eb400640 1 --2- 172.21.15.67:0/3321840837 >> v2:172.21.15.187:3300/0 conn(0x7fb4ec141ed0 0x7fb4ec142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.229+0000 7fb4eb400640 1 -- 172.21.15.67:0/3321840837 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ec160ad0 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.230+0000 7fb4eb400640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec071d80 tx=0x7fb4dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3269e4dac2ec391 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.230+0000 7fb4eaa00640 1 -- 172.21.15.67:0/3321840837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc035020 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.230+0000 7fb4eaa00640 1 -- 172.21.15.67:0/3321840837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4dc031870 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.230+0000 7fb4eaa00640 1 -- 172.21.15.67:0/3321840837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc031c40 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 -- 172.21.15.67:0/3321840837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec06b8c0 msgr2=0x7fb4ec0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec071d80 tx=0x7fb4dc02b910 comp rx=0 tx=0).stop 2024-04-17T07:30:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 -- 172.21.15.67:0/3321840837 shutdown_connections 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 --2- 172.21.15.67:0/3321840837 >> v2:172.21.15.187:3300/0 conn(0x7fb4ec141ed0 0x7fb4ec142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec160280 0x7fb4ec06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 --2- 172.21.15.67:0/3321840837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec06b8c0 0x7fb4ec0cd8b0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 -- 172.21.15.67:0/3321840837 >> 172.21.15.67:0/3321840837 conn(0x7fb4ec067d10 msgr2=0x7fb4ec0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 -- 172.21.15.67:0/3321840837 shutdown_connections 2024-04-17T07:30:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.231+0000 7fb4f1000640 1 -- 172.21.15.67:0/3321840837 wait complete. 2024-04-17T07:30:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.233+0000 7fb4f1000640 1 Processor -- start 2024-04-17T07:30:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.233+0000 7fb4f1000640 1 -- start start 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec06b8c0 0x7fb4ec13e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ec160280 0x7fb4ec141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec06c1e0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ec0cddf0 con 0x7fb4ec160280 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4f1000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4ec161090 con 0x7fb4ec06b8c0 2024-04-17T07:30:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51740/0 (socket says 172.21.15.67:51740) 2024-04-17T07:30:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.234+0000 7fb4e3e00640 1 -- 172.21.15.67:0/1407005284 learned_addr learned my addr 172.21.15.67:0/1407005284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 -- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec06b8c0 msgr2=0x7fb4ec13e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec06b8c0 0x7fb4ec13e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 -- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ec160280 msgr2=0x7fb4ec141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ec160280 0x7fb4ec141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 -- 172.21.15.67:0/1407005284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ec15e460 con 0x7fb4ec141ed0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3e00640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c820 tx=0x7fb4d400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4015070 con 0x7fb4ec141ed0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4d4004020 con 0x7fb4ec141ed0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40043f0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ec15e740 con 0x7fb4ec141ed0 2024-04-17T07:30:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.235+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ec13ddb0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.236+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ec068a10 con 0x7fb4ec141ed0 2024-04-17T07:30:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.238+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fb4d40077d0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.239+0000 7fb4e3400640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 0x7fb4c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.239+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7fb4d408bc50 con 0x7fb4ec141ed0 2024-04-17T07:30:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.239+0000 7fb4eb400640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 0x7fb4c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.241+0000 7fb4eb400640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 0x7fb4c806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc036000 tx=0x7fb4dc030000 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.244+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb4d4059dc0 con 0x7fb4ec141ed0 2024-04-17T07:30:57.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.449+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 --> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4ec1565c0 con 0x7fb4c806ca10 2024-04-17T07:30:57.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.472+0000 7fb4e3400640 1 -- 172.21.15.67:0/1407005284 <== mgr.6091 v2:172.21.15.187:6832/158797128 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252491 (secure 0 0 0) 0x7fb4ec1565c0 con 0x7fb4c806ca10 2024-04-17T07:30:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:30:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.478+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 msgr2=0x7fb4c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.478+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 0x7fb4c806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc036000 tx=0x7fb4dc030000 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.478+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 msgr2=0x7fb4ec13d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c820 tx=0x7fb4d400cce0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 shutdown_connections 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fb4c806ca10 0x7fb4c806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4ec160280 0x7fb4ec141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4ec141ed0 0x7fb4ec13d870 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 --2- 172.21.15.67:0/1407005284 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4ec06b8c0 0x7fb4ec13e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 >> 172.21.15.67:0/1407005284 conn(0x7fb4ec067d10 msgr2=0x7fb4ec0d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 shutdown_connections 2024-04-17T07:30:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.479+0000 7fb4f1000640 1 -- 172.21.15.67:0/1407005284 wait complete. 2024-04-17T07:30:57.500 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-17T07:30:56.365114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":332,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330720,"kb_used_data":11848,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754644000,"statfs":{"total":773094113280,"available":772755456000,"internally_reserved":0,"allocated":12132352,"data_stored":6370092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456635"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351352+0000","last_change":"2024-04-17T07:30:52.953522+0000","last_active":"2024-04-17T07:30:54.351352+0000","last_peered":"2024-04-17T07:30:54.351352+0000","last_clean":"2024-04-17T07:30:54.351352+0000","last_became_active":"2024-04-17T07:30:52.953220+0000","last_became_peered":"2024-04-17T07:30:52.953220+0000","last_unstale":"2024-04-17T07:30:54.351352+0000","last_undegraded":"2024-04-17T07:30:54.351352+0000","last_fullsized":"2024-04-17T07:30:54.351352+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:12.504889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971548+0000","last_change":"2024-04-17T07:30:52.973940+0000","last_active":"2024-04-17T07:30:53.971548+0000","last_peered":"2024-04-17T07:30:53.971548+0000","last_clean":"2024-04-17T07:30:53.971548+0000","last_became_active":"2024-04-17T07:30:52.972663+0000","last_became_peered":"2024-04-17T07:30:52.972663+0000","last_unstale":"2024-04-17T07:30:53.971548+0000","last_undegraded":"2024-04-17T07:30:53.971548+0000","last_fullsized":"2024-04-17T07:30:53.971548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:40:22.130567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962476+0000","last_change":"2024-04-17T07:30:52.961899+0000","last_active":"2024-04-17T07:30:53.962476+0000","last_peered":"2024-04-17T07:30:53.962476+0000","last_clean":"2024-04-17T07:30:53.962476+0000","last_became_active":"2024-04-17T07:30:52.961623+0000","last_became_peered":"2024-04-17T07:30:52.961623+0000","last_unstale":"2024-04-17T07:30:53.962476+0000","last_undegraded":"2024-04-17T07:30:53.962476+0000","last_fullsized":"2024-04-17T07:30:53.962476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:36:21.177559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956895+0000","last_change":"2024-04-17T07:30:52.973925+0000","last_active":"2024-04-17T07:30:53.956895+0000","last_peered":"2024-04-17T07:30:53.956895+0000","last_clean":"2024-04-17T07:30:53.956895+0000","last_became_active":"2024-04-17T07:30:52.973641+0000","last_became_peered":"2024-04-17T07:30:52.973641+0000","last_unstale":"2024-04-17T07:30:53.956895+0000","last_undegraded":"2024-04-17T07:30:53.956895+0000","last_fullsized":"2024-04-17T07:30:53.956895+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:07.348218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348313+0000","last_change":"2024-04-17T07:30:52.968791+0000","last_active":"2024-04-17T07:30:54.348313+0000","last_peered":"2024-04-17T07:30:54.348313+0000","last_clean":"2024-04-17T07:30:54.348313+0000","last_became_active":"2024-04-17T07:30:52.968540+0000","last_became_peered":"2024-04-17T07:30:52.968540+0000","last_unstale":"2024-04-17T07:30:54.348313+0000","last_undegraded":"2024-04-17T07:30:54.348313+0000","last_fullsized":"2024-04-17T07:30:54.348313+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:07.322541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960641+0000","last_change":"2024-04-17T07:30:52.963724+0000","last_active":"2024-04-17T07:30:53.960641+0000","last_peered":"2024-04-17T07:30:53.960641+0000","last_clean":"2024-04-17T07:30:53.960641+0000","last_became_active":"2024-04-17T07:30:52.963401+0000","last_became_peered":"2024-04-17T07:30:52.963401+0000","last_unstale":"2024-04-17T07:30:53.960641+0000","last_undegraded":"2024-04-17T07:30:53.960641+0000","last_fullsized":"2024-04-17T07:30:53.960641+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:33.061753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956171+0000","last_change":"2024-04-17T07:30:52.962235+0000","last_active":"2024-04-17T07:30:53.956171+0000","last_peered":"2024-04-17T07:30:53.956171+0000","last_clean":"2024-04-17T07:30:53.956171+0000","last_became_active":"2024-04-17T07:30:52.958407+0000","last_became_peered":"2024-04-17T07:30:52.958407+0000","last_unstale":"2024-04-17T07:30:53.956171+0000","last_undegraded":"2024-04-17T07:30:53.956171+0000","last_fullsized":"2024-04-17T07:30:53.956171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:11.653065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960088+0000","last_change":"2024-04-17T07:30:52.976325+0000","last_active":"2024-04-17T07:30:53.960088+0000","last_peered":"2024-04-17T07:30:53.960088+0000","last_clean":"2024-04-17T07:30:53.960088+0000","last_became_active":"2024-04-17T07:30:52.975636+0000","last_became_peered":"2024-04-17T07:30:52.975636+0000","last_unstale":"2024-04-17T07:30:53.960088+0000","last_undegraded":"2024-04-17T07:30:53.960088+0000","last_fullsized":"2024-04-17T07:30:53.960088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:41.928147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961419+0000","last_change":"2024-04-17T07:30:52.973754+0000","last_active":"2024-04-17T07:30:53.961419+0000","last_peered":"2024-04-17T07:30:53.961419+0000","last_clean":"2024-04-17T07:30:53.961419+0000","last_became_active":"2024-04-17T07:30:52.973416+0000","last_became_peered":"2024-04-17T07:30:52.973416+0000","last_unstale":"2024-04-17T07:30:53.961419+0000","last_undegraded":"2024-04-17T07:30:53.961419+0000","last_fullsized":"2024-04-17T07:30:53.961419+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:01.659599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351123+0000","last_change":"2024-04-17T07:30:52.966095+0000","last_active":"2024-04-17T07:30:54.351123+0000","last_peered":"2024-04-17T07:30:54.351123+0000","last_clean":"2024-04-17T07:30:54.351123+0000","last_became_active":"2024-04-17T07:30:52.965735+0000","last_became_peered":"2024-04-17T07:30:52.965735+0000","last_unstale":"2024-04-17T07:30:54.351123+0000","last_undegraded":"2024-04-17T07:30:54.351123+0000","last_fullsized":"2024-04-17T07:30:54.351123+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:01:07.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971430+0000","last_change":"2024-04-17T07:30:52.964690+0000","last_active":"2024-04-17T07:30:53.971430+0000","last_peered":"2024-04-17T07:30:53.971430+0000","last_clean":"2024-04-17T07:30:53.971430+0000","last_became_active":"2024-04-17T07:30:52.964340+0000","last_became_peered":"2024-04-17T07:30:52.964340+0000","last_unstale":"2024-04-17T07:30:53.971430+0000","last_undegraded":"2024-04-17T07:30:53.971430+0000","last_fullsized":"2024-04-17T07:30:53.971430+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:06.794769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350413+0000","last_change":"2024-04-17T07:30:52.965459+0000","last_active":"2024-04-17T07:30:54.350413+0000","last_peered":"2024-04-17T07:30:54.350413+0000","last_clean":"2024-04-17T07:30:54.350413+0000","last_became_active":"2024-04-17T07:30:52.965156+0000","last_became_peered":"2024-04-17T07:30:52.965156+0000","last_unstale":"2024-04-17T07:30:54.350413+0000","last_undegraded":"2024-04-17T07:30:54.350413+0000","last_fullsized":"2024-04-17T07:30:54.350413+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:16.623613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962295+0000","last_change":"2024-04-17T07:30:52.969490+0000","last_active":"2024-04-17T07:30:53.962295+0000","last_peered":"2024-04-17T07:30:53.962295+0000","last_clean":"2024-04-17T07:30:53.962295+0000","last_became_active":"2024-04-17T07:30:52.969178+0000","last_became_peered":"2024-04-17T07:30:52.969178+0000","last_unstale":"2024-04-17T07:30:53.962295+0000","last_undegraded":"2024-04-17T07:30:53.962295+0000","last_fullsized":"2024-04-17T07:30:53.962295+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:59.001615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956407+0000","last_change":"2024-04-17T07:30:52.978051+0000","last_active":"2024-04-17T07:30:53.956407+0000","last_peered":"2024-04-17T07:30:53.956407+0000","last_clean":"2024-04-17T07:30:53.956407+0000","last_became_active":"2024-04-17T07:30:52.976371+0000","last_became_peered":"2024-04-17T07:30:52.976371+0000","last_unstale":"2024-04-17T07:30:53.956407+0000","last_undegraded":"2024-04-17T07:30:53.956407+0000","last_fullsized":"2024-04-17T07:30:53.956407+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:36.782623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347549+0000","last_change":"2024-04-17T07:30:52.961329+0000","last_active":"2024-04-17T07:30:54.347549+0000","last_peered":"2024-04-17T07:30:54.347549+0000","last_clean":"2024-04-17T07:30:54.347549+0000","last_became_active":"2024-04-17T07:30:52.960990+0000","last_became_peered":"2024-04-17T07:30:52.960990+0000","last_unstale":"2024-04-17T07:30:54.347549+0000","last_undegraded":"2024-04-17T07:30:54.347549+0000","last_fullsized":"2024-04-17T07:30:54.347549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:33.550564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347795+0000","last_change":"2024-04-17T07:30:52.951515+0000","last_active":"2024-04-17T07:30:54.347795+0000","last_peered":"2024-04-17T07:30:54.347795+0000","last_clean":"2024-04-17T07:30:54.347795+0000","last_became_active":"2024-04-17T07:30:52.951036+0000","last_became_peered":"2024-04-17T07:30:52.951036+0000","last_unstale":"2024-04-17T07:30:54.347795+0000","last_undegraded":"2024-04-17T07:30:54.347795+0000","last_fullsized":"2024-04-17T07:30:54.347795+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:40.463465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349694+0000","last_change":"2024-04-17T07:30:52.965372+0000","last_active":"2024-04-17T07:30:54.349694+0000","last_peered":"2024-04-17T07:30:54.349694+0000","last_clean":"2024-04-17T07:30:54.349694+0000","last_became_active":"2024-04-17T07:30:52.965061+0000","last_became_peered":"2024-04-17T07:30:52.965061+0000","last_unstale":"2024-04-17T07:30:54.349694+0000","last_undegraded":"2024-04-17T07:30:54.349694+0000","last_fullsized":"2024-04-17T07:30:54.349694+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:27.930242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.957113+0000","last_change":"2024-04-17T07:30:52.955873+0000","last_active":"2024-04-17T07:30:53.957113+0000","last_peered":"2024-04-17T07:30:53.957113+0000","last_clean":"2024-04-17T07:30:53.957113+0000","last_became_active":"2024-04-17T07:30:52.955558+0000","last_became_peered":"2024-04-17T07:30:52.955558+0000","last_unstale":"2024-04-17T07:30:53.957113+0000","last_undegraded":"2024-04-17T07:30:53.957113+0000","last_fullsized":"2024-04-17T07:30:53.957113+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:31.353125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975140+0000","last_change":"2024-04-17T07:30:52.983064+0000","last_active":"2024-04-17T07:30:53.975140+0000","last_peered":"2024-04-17T07:30:53.975140+0000","last_clean":"2024-04-17T07:30:53.975140+0000","last_became_active":"2024-04-17T07:30:52.981954+0000","last_became_peered":"2024-04-17T07:30:52.981954+0000","last_unstale":"2024-04-17T07:30:53.975140+0000","last_undegraded":"2024-04-17T07:30:53.975140+0000","last_fullsized":"2024-04-17T07:30:53.975140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:47.172943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.957106+0000","last_change":"2024-04-17T07:30:52.963736+0000","last_active":"2024-04-17T07:30:53.957106+0000","last_peered":"2024-04-17T07:30:53.957106+0000","last_clean":"2024-04-17T07:30:53.957106+0000","last_became_active":"2024-04-17T07:30:52.958146+0000","last_became_peered":"2024-04-17T07:30:52.958146+0000","last_unstale":"2024-04-17T07:30:53.957106+0000","last_undegraded":"2024-04-17T07:30:53.957106+0000","last_fullsized":"2024-04-17T07:30:53.957106+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:02:40.068613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956637+0000","last_change":"2024-04-17T07:30:52.960480+0000","last_active":"2024-04-17T07:30:53.956637+0000","last_peered":"2024-04-17T07:30:53.956637+0000","last_clean":"2024-04-17T07:30:53.956637+0000","last_became_active":"2024-04-17T07:30:52.958156+0000","last_became_peered":"2024-04-17T07:30:52.958156+0000","last_unstale":"2024-04-17T07:30:53.956637+0000","last_undegraded":"2024-04-17T07:30:53.956637+0000","last_fullsized":"2024-04-17T07:30:53.956637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:36.125345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971346+0000","last_change":"2024-04-17T07:30:52.978117+0000","last_active":"2024-04-17T07:30:53.971346+0000","last_peered":"2024-04-17T07:30:53.971346+0000","last_clean":"2024-04-17T07:30:53.971346+0000","last_became_active":"2024-04-17T07:30:52.974801+0000","last_became_peered":"2024-04-17T07:30:52.974801+0000","last_unstale":"2024-04-17T07:30:53.971346+0000","last_undegraded":"2024-04-17T07:30:53.971346+0000","last_fullsized":"2024-04-17T07:30:53.971346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.458301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349069+0000","last_change":"2024-04-17T07:30:52.978968+0000","last_active":"2024-04-17T07:30:54.349069+0000","last_peered":"2024-04-17T07:30:54.349069+0000","last_clean":"2024-04-17T07:30:54.349069+0000","last_became_active":"2024-04-17T07:30:52.978240+0000","last_became_peered":"2024-04-17T07:30:52.978240+0000","last_unstale":"2024-04-17T07:30:54.349069+0000","last_undegraded":"2024-04-17T07:30:54.349069+0000","last_fullsized":"2024-04-17T07:30:54.349069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:00:22.313647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956048+0000","last_change":"2024-04-17T07:30:52.970047+0000","last_active":"2024-04-17T07:30:53.956048+0000","last_peered":"2024-04-17T07:30:53.956048+0000","last_clean":"2024-04-17T07:30:53.956048+0000","last_became_active":"2024-04-17T07:30:52.969690+0000","last_became_peered":"2024-04-17T07:30:52.969690+0000","last_unstale":"2024-04-17T07:30:53.956048+0000","last_undegraded":"2024-04-17T07:30:53.956048+0000","last_fullsized":"2024-04-17T07:30:53.956048+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:42.419341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971471+0000","last_change":"2024-04-17T07:30:52.966745+0000","last_active":"2024-04-17T07:30:53.971471+0000","last_peered":"2024-04-17T07:30:53.971471+0000","last_clean":"2024-04-17T07:30:53.971471+0000","last_became_active":"2024-04-17T07:30:52.966427+0000","last_became_peered":"2024-04-17T07:30:52.966427+0000","last_unstale":"2024-04-17T07:30:53.971471+0000","last_undegraded":"2024-04-17T07:30:53.971471+0000","last_fullsized":"2024-04-17T07:30:53.971471+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:06.516299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962518+0000","last_change":"2024-04-17T07:30:52.975543+0000","last_active":"2024-04-17T07:30:53.962518+0000","last_peered":"2024-04-17T07:30:53.962518+0000","last_clean":"2024-04-17T07:30:53.962518+0000","last_became_active":"2024-04-17T07:30:52.975219+0000","last_became_peered":"2024-04-17T07:30:52.975219+0000","last_unstale":"2024-04-17T07:30:53.962518+0000","last_undegraded":"2024-04-17T07:30:53.962518+0000","last_fullsized":"2024-04-17T07:30:53.962518+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:45.152848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348633+0000","last_change":"2024-04-17T07:30:52.957742+0000","last_active":"2024-04-17T07:30:54.348633+0000","last_peered":"2024-04-17T07:30:54.348633+0000","last_clean":"2024-04-17T07:30:54.348633+0000","last_became_active":"2024-04-17T07:30:52.957473+0000","last_became_peered":"2024-04-17T07:30:52.957473+0000","last_unstale":"2024-04-17T07:30:54.348633+0000","last_undegraded":"2024-04-17T07:30:54.348633+0000","last_fullsized":"2024-04-17T07:30:54.348633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:36.153994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350626+0000","last_change":"2024-04-17T07:30:52.954271+0000","last_active":"2024-04-17T07:30:54.350626+0000","last_peered":"2024-04-17T07:30:54.350626+0000","last_clean":"2024-04-17T07:30:54.350626+0000","last_became_active":"2024-04-17T07:30:52.953990+0000","last_became_peered":"2024-04-17T07:30:52.953990+0000","last_unstale":"2024-04-17T07:30:54.350626+0000","last_undegraded":"2024-04-17T07:30:54.350626+0000","last_fullsized":"2024-04-17T07:30:54.350626+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:46.840383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971060+0000","last_change":"2024-04-17T07:30:52.967804+0000","last_active":"2024-04-17T07:30:53.971060+0000","last_peered":"2024-04-17T07:30:53.971060+0000","last_clean":"2024-04-17T07:30:53.971060+0000","last_became_active":"2024-04-17T07:30:52.967470+0000","last_became_peered":"2024-04-17T07:30:52.967470+0000","last_unstale":"2024-04-17T07:30:53.971060+0000","last_undegraded":"2024-04-17T07:30:53.971060+0000","last_fullsized":"2024-04-17T07:30:53.971060+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:01:07.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970933+0000","last_change":"2024-04-17T07:30:52.971295+0000","last_active":"2024-04-17T07:30:53.970933+0000","last_peered":"2024-04-17T07:30:53.970933+0000","last_clean":"2024-04-17T07:30:53.970933+0000","last_became_active":"2024-04-17T07:30:52.970780+0000","last_became_peered":"2024-04-17T07:30:52.970780+0000","last_unstale":"2024-04-17T07:30:53.970933+0000","last_undegraded":"2024-04-17T07:30:53.970933+0000","last_fullsized":"2024-04-17T07:30:53.970933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:21.601807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347825+0000","last_change":"2024-04-17T07:30:52.972634+0000","last_active":"2024-04-17T07:30:54.347825+0000","last_peered":"2024-04-17T07:30:54.347825+0000","last_clean":"2024-04-17T07:30:54.347825+0000","last_became_active":"2024-04-17T07:30:52.972294+0000","last_became_peered":"2024-04-17T07:30:52.972294+0000","last_unstale":"2024-04-17T07:30:54.347825+0000","last_undegraded":"2024-04-17T07:30:54.347825+0000","last_fullsized":"2024-04-17T07:30:54.347825+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:21.544764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348919+0000","last_change":"2024-04-17T07:30:52.969843+0000","last_active":"2024-04-17T07:30:54.348919+0000","last_peered":"2024-04-17T07:30:54.348919+0000","last_clean":"2024-04-17T07:30:54.348919+0000","last_became_active":"2024-04-17T07:30:52.966533+0000","last_became_peered":"2024-04-17T07:30:52.966533+0000","last_unstale":"2024-04-17T07:30:54.348919+0000","last_undegraded":"2024-04-17T07:30:54.348919+0000","last_fullsized":"2024-04-17T07:30:54.348919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:55.924704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349422+0000","last_change":"2024-04-17T07:30:52.961011+0000","last_active":"2024-04-17T07:30:54.349422+0000","last_peered":"2024-04-17T07:30:54.349422+0000","last_clean":"2024-04-17T07:30:54.349422+0000","last_became_active":"2024-04-17T07:30:52.960733+0000","last_became_peered":"2024-04-17T07:30:52.960733+0000","last_unstale":"2024-04-17T07:30:54.349422+0000","last_undegraded":"2024-04-17T07:30:54.349422+0000","last_fullsized":"2024-04-17T07:30:54.349422+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:54.024907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351818+0000","last_change":"2024-04-17T07:30:51.964315+0000","last_active":"2024-04-17T07:30:54.351818+0000","last_peered":"2024-04-17T07:30:54.351818+0000","last_clean":"2024-04-17T07:30:54.351818+0000","last_became_active":"2024-04-17T07:30:51.963522+0000","last_became_peered":"2024-04-17T07:30:51.963522+0000","last_unstale":"2024-04-17T07:30:54.351818+0000","last_undegraded":"2024-04-17T07:30:54.351818+0000","last_fullsized":"2024-04-17T07:30:54.351818+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:17:30.429277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351549+0000","last_change":"2024-04-17T07:30:52.978864+0000","last_active":"2024-04-17T07:30:54.351549+0000","last_peered":"2024-04-17T07:30:54.351549+0000","last_clean":"2024-04-17T07:30:54.351549+0000","last_became_active":"2024-04-17T07:30:52.978253+0000","last_became_peered":"2024-04-17T07:30:52.978253+0000","last_unstale":"2024-04-17T07:30:54.351549+0000","last_undegraded":"2024-04-17T07:30:54.351549+0000","last_fullsized":"2024-04-17T07:30:54.351549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.458301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349703+0000","last_change":"2024-04-17T07:30:51.954624+0000","last_active":"2024-04-17T07:30:54.349703+0000","last_peered":"2024-04-17T07:30:54.349703+0000","last_clean":"2024-04-17T07:30:54.349703+0000","last_became_active":"2024-04-17T07:30:51.954151+0000","last_became_peered":"2024-04-17T07:30:51.954151+0000","last_unstale":"2024-04-17T07:30:54.349703+0000","last_undegraded":"2024-04-17T07:30:54.349703+0000","last_fullsized":"2024-04-17T07:30:54.349703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:11.204766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350798+0000","last_change":"2024-04-17T07:30:52.970973+0000","last_active":"2024-04-17T07:30:54.350798+0000","last_peered":"2024-04-17T07:30:54.350798+0000","last_clean":"2024-04-17T07:30:54.350798+0000","last_became_active":"2024-04-17T07:30:52.970625+0000","last_became_peered":"2024-04-17T07:30:52.970625+0000","last_unstale":"2024-04-17T07:30:54.350798+0000","last_undegraded":"2024-04-17T07:30:54.350798+0000","last_fullsized":"2024-04-17T07:30:54.350798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:12.775821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975346+0000","last_change":"2024-04-17T07:30:51.957357+0000","last_active":"2024-04-17T07:30:53.975346+0000","last_peered":"2024-04-17T07:30:53.975346+0000","last_clean":"2024-04-17T07:30:53.975346+0000","last_became_active":"2024-04-17T07:30:51.956633+0000","last_became_peered":"2024-04-17T07:30:51.956633+0000","last_unstale":"2024-04-17T07:30:53.975346+0000","last_undegraded":"2024-04-17T07:30:53.975346+0000","last_fullsized":"2024-04-17T07:30:53.975346+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:07.870223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962880+0000","last_change":"2024-04-17T07:30:52.969777+0000","last_active":"2024-04-17T07:30:53.962880+0000","last_peered":"2024-04-17T07:30:53.962880+0000","last_clean":"2024-04-17T07:30:53.962880+0000","last_became_active":"2024-04-17T07:30:52.969101+0000","last_became_peered":"2024-04-17T07:30:52.969101+0000","last_unstale":"2024-04-17T07:30:53.962880+0000","last_undegraded":"2024-04-17T07:30:53.962880+0000","last_fullsized":"2024-04-17T07:30:53.962880+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:35.282675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975038+0000","last_change":"2024-04-17T07:30:51.957716+0000","last_active":"2024-04-17T07:30:53.975038+0000","last_peered":"2024-04-17T07:30:53.975038+0000","last_clean":"2024-04-17T07:30:53.975038+0000","last_became_active":"2024-04-17T07:30:51.957423+0000","last_became_peered":"2024-04-17T07:30:51.957423+0000","last_unstale":"2024-04-17T07:30:53.975038+0000","last_undegraded":"2024-04-17T07:30:53.975038+0000","last_fullsized":"2024-04-17T07:30:53.975038+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:18.127344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971116+0000","last_change":"2024-04-17T07:30:52.971044+0000","last_active":"2024-04-17T07:30:53.971116+0000","last_peered":"2024-04-17T07:30:53.971116+0000","last_clean":"2024-04-17T07:30:53.971116+0000","last_became_active":"2024-04-17T07:30:52.970457+0000","last_became_peered":"2024-04-17T07:30:52.970457+0000","last_unstale":"2024-04-17T07:30:53.971116+0000","last_undegraded":"2024-04-17T07:30:53.971116+0000","last_fullsized":"2024-04-17T07:30:53.971116+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:12.504889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348720+0000","last_change":"2024-04-17T07:30:51.957272+0000","last_active":"2024-04-17T07:30:54.348720+0000","last_peered":"2024-04-17T07:30:54.348720+0000","last_clean":"2024-04-17T07:30:54.348720+0000","last_became_active":"2024-04-17T07:30:51.954523+0000","last_became_peered":"2024-04-17T07:30:51.954523+0000","last_unstale":"2024-04-17T07:30:54.348720+0000","last_undegraded":"2024-04-17T07:30:54.348720+0000","last_fullsized":"2024-04-17T07:30:54.348720+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:28.571235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348430+0000","last_change":"2024-04-17T07:30:52.961446+0000","last_active":"2024-04-17T07:30:54.348430+0000","last_peered":"2024-04-17T07:30:54.348430+0000","last_clean":"2024-04-17T07:30:54.348430+0000","last_became_active":"2024-04-17T07:30:52.958924+0000","last_became_peered":"2024-04-17T07:30:52.958924+0000","last_unstale":"2024-04-17T07:30:54.348430+0000","last_undegraded":"2024-04-17T07:30:54.348430+0000","last_fullsized":"2024-04-17T07:30:54.348430+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:30.187374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348040+0000","last_change":"2024-04-17T07:30:51.953890+0000","last_active":"2024-04-17T07:30:54.348040+0000","last_peered":"2024-04-17T07:30:54.348040+0000","last_clean":"2024-04-17T07:30:54.348040+0000","last_became_active":"2024-04-17T07:30:51.952980+0000","last_became_peered":"2024-04-17T07:30:51.952980+0000","last_unstale":"2024-04-17T07:30:54.348040+0000","last_undegraded":"2024-04-17T07:30:54.348040+0000","last_fullsized":"2024-04-17T07:30:54.348040+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:15:24.280399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960151+0000","last_change":"2024-04-17T07:30:52.979175+0000","last_active":"2024-04-17T07:30:53.960151+0000","last_peered":"2024-04-17T07:30:53.960151+0000","last_clean":"2024-04-17T07:30:53.960151+0000","last_became_active":"2024-04-17T07:30:52.978799+0000","last_became_peered":"2024-04-17T07:30:52.978799+0000","last_unstale":"2024-04-17T07:30:53.960151+0000","last_undegraded":"2024-04-17T07:30:53.960151+0000","last_fullsized":"2024-04-17T07:30:53.960151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:33.550564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975541+0000","last_change":"2024-04-17T07:30:51.952630+0000","last_active":"2024-04-17T07:30:53.975541+0000","last_peered":"2024-04-17T07:30:53.975541+0000","last_clean":"2024-04-17T07:30:53.975541+0000","last_became_active":"2024-04-17T07:30:51.951413+0000","last_became_peered":"2024-04-17T07:30:51.951413+0000","last_unstale":"2024-04-17T07:30:53.975541+0000","last_undegraded":"2024-04-17T07:30:53.975541+0000","last_fullsized":"2024-04-17T07:30:53.975541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:52.513196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347371+0000","last_change":"2024-04-17T07:30:52.961176+0000","last_active":"2024-04-17T07:30:54.347371+0000","last_peered":"2024-04-17T07:30:54.347371+0000","last_clean":"2024-04-17T07:30:54.347371+0000","last_became_active":"2024-04-17T07:30:52.960901+0000","last_became_peered":"2024-04-17T07:30:52.960901+0000","last_unstale":"2024-04-17T07:30:54.347371+0000","last_undegraded":"2024-04-17T07:30:54.347371+0000","last_fullsized":"2024-04-17T07:30:54.347371+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.846951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970566+0000","last_change":"2024-04-17T07:30:51.958218+0000","last_active":"2024-04-17T07:30:53.970566+0000","last_peered":"2024-04-17T07:30:53.970566+0000","last_clean":"2024-04-17T07:30:53.970566+0000","last_became_active":"2024-04-17T07:30:51.957924+0000","last_became_peered":"2024-04-17T07:30:51.957924+0000","last_unstale":"2024-04-17T07:30:53.970566+0000","last_undegraded":"2024-04-17T07:30:53.970566+0000","last_fullsized":"2024-04-17T07:30:53.970566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:14:13.632220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348609+0000","last_change":"2024-04-17T07:30:52.973407+0000","last_active":"2024-04-17T07:30:54.348609+0000","last_peered":"2024-04-17T07:30:54.348609+0000","last_clean":"2024-04-17T07:30:54.348609+0000","last_became_active":"2024-04-17T07:30:52.973087+0000","last_became_peered":"2024-04-17T07:30:52.973087+0000","last_unstale":"2024-04-17T07:30:54.348609+0000","last_undegraded":"2024-04-17T07:30:54.348609+0000","last_fullsized":"2024-04-17T07:30:54.348609+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.108657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.352136+0000","last_change":"2024-04-17T07:30:51.964718+0000","last_active":"2024-04-17T07:30:54.352136+0000","last_peered":"2024-04-17T07:30:54.352136+0000","last_clean":"2024-04-17T07:30:54.352136+0000","last_became_active":"2024-04-17T07:30:51.964349+0000","last_became_peered":"2024-04-17T07:30:51.964349+0000","last_unstale":"2024-04-17T07:30:54.352136+0000","last_undegraded":"2024-04-17T07:30:54.352136+0000","last_fullsized":"2024-04-17T07:30:54.352136+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:53.647342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961614+0000","last_change":"2024-04-17T07:30:52.965649+0000","last_active":"2024-04-17T07:30:53.961614+0000","last_peered":"2024-04-17T07:30:53.961614+0000","last_clean":"2024-04-17T07:30:53.961614+0000","last_became_active":"2024-04-17T07:30:52.965374+0000","last_became_peered":"2024-04-17T07:30:52.965374+0000","last_unstale":"2024-04-17T07:30:53.961614+0000","last_undegraded":"2024-04-17T07:30:53.961614+0000","last_fullsized":"2024-04-17T07:30:53.961614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:55.645023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351007+0000","last_change":"2024-04-17T07:30:51.961616+0000","last_active":"2024-04-17T07:30:54.351007+0000","last_peered":"2024-04-17T07:30:54.351007+0000","last_clean":"2024-04-17T07:30:54.351007+0000","last_became_active":"2024-04-17T07:30:51.961322+0000","last_became_peered":"2024-04-17T07:30:51.961322+0000","last_unstale":"2024-04-17T07:30:54.351007+0000","last_undegraded":"2024-04-17T07:30:54.351007+0000","last_fullsized":"2024-04-17T07:30:54.351007+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:30.526673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975054+0000","last_change":"2024-04-17T07:30:52.960080+0000","last_active":"2024-04-17T07:30:53.975054+0000","last_peered":"2024-04-17T07:30:53.975054+0000","last_clean":"2024-04-17T07:30:53.975054+0000","last_became_active":"2024-04-17T07:30:52.958896+0000","last_became_peered":"2024-04-17T07:30:52.958896+0000","last_unstale":"2024-04-17T07:30:53.975054+0000","last_undegraded":"2024-04-17T07:30:53.975054+0000","last_fullsized":"2024-04-17T07:30:53.975054+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:55.835840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350493+0000","last_change":"2024-04-17T07:30:51.962920+0000","last_active":"2024-04-17T07:30:54.350493+0000","last_peered":"2024-04-17T07:30:54.350493+0000","last_clean":"2024-04-17T07:30:54.350493+0000","last_became_active":"2024-04-17T07:30:51.962642+0000","last_became_peered":"2024-04-17T07:30:51.962642+0000","last_unstale":"2024-04-17T07:30:54.350493+0000","last_undegraded":"2024-04-17T07:30:54.350493+0000","last_fullsized":"2024-04-17T07:30:54.350493+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:06.226285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349052+0000","last_change":"2024-04-17T07:30:52.972717+0000","last_active":"2024-04-17T07:30:54.349052+0000","last_peered":"2024-04-17T07:30:54.349052+0000","last_clean":"2024-04-17T07:30:54.349052+0000","last_became_active":"2024-04-17T07:30:52.972202+0000","last_became_peered":"2024-04-17T07:30:52.972202+0000","last_unstale":"2024-04-17T07:30:54.349052+0000","last_undegraded":"2024-04-17T07:30:54.349052+0000","last_fullsized":"2024-04-17T07:30:54.349052+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:37.546019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962084+0000","last_change":"2024-04-17T07:30:51.948347+0000","last_active":"2024-04-17T07:30:53.962084+0000","last_peered":"2024-04-17T07:30:53.962084+0000","last_clean":"2024-04-17T07:30:53.962084+0000","last_became_active":"2024-04-17T07:30:51.947001+0000","last_became_peered":"2024-04-17T07:30:51.947001+0000","last_unstale":"2024-04-17T07:30:53.962084+0000","last_undegraded":"2024-04-17T07:30:53.962084+0000","last_fullsized":"2024-04-17T07:30:53.962084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:48.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975103+0000","last_change":"2024-04-17T07:30:52.969081+0000","last_active":"2024-04-17T07:30:53.975103+0000","last_peered":"2024-04-17T07:30:53.975103+0000","last_clean":"2024-04-17T07:30:53.975103+0000","last_became_active":"2024-04-17T07:30:52.968804+0000","last_became_peered":"2024-04-17T07:30:52.968804+0000","last_unstale":"2024-04-17T07:30:53.975103+0000","last_undegraded":"2024-04-17T07:30:53.975103+0000","last_fullsized":"2024-04-17T07:30:53.975103+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:37.913622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970967+0000","last_change":"2024-04-17T07:30:51.961385+0000","last_active":"2024-04-17T07:30:53.970967+0000","last_peered":"2024-04-17T07:30:53.970967+0000","last_clean":"2024-04-17T07:30:53.970967+0000","last_became_active":"2024-04-17T07:30:51.961094+0000","last_became_peered":"2024-04-17T07:30:51.961094+0000","last_unstale":"2024-04-17T07:30:53.970967+0000","last_undegraded":"2024-04-17T07:30:53.970967+0000","last_fullsized":"2024-04-17T07:30:53.970967+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:05.969430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962534+0000","last_change":"2024-04-17T07:30:52.978270+0000","last_active":"2024-04-17T07:30:53.962534+0000","last_peered":"2024-04-17T07:30:53.962534+0000","last_clean":"2024-04-17T07:30:53.962534+0000","last_became_active":"2024-04-17T07:30:52.977962+0000","last_became_peered":"2024-04-17T07:30:52.977962+0000","last_unstale":"2024-04-17T07:30:53.962534+0000","last_undegraded":"2024-04-17T07:30:53.962534+0000","last_fullsized":"2024-04-17T07:30:53.962534+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:08.754125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347882+0000","last_change":"2024-04-17T07:30:51.949376+0000","last_active":"2024-04-17T07:30:54.347882+0000","last_peered":"2024-04-17T07:30:54.347882+0000","last_clean":"2024-04-17T07:30:54.347882+0000","last_became_active":"2024-04-17T07:30:51.949024+0000","last_became_peered":"2024-04-17T07:30:51.949024+0000","last_unstale":"2024-04-17T07:30:54.347882+0000","last_undegraded":"2024-04-17T07:30:54.347882+0000","last_fullsized":"2024-04-17T07:30:54.347882+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:59:10.386075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970498+0000","last_change":"2024-04-17T07:30:52.967073+0000","last_active":"2024-04-17T07:30:53.970498+0000","last_peered":"2024-04-17T07:30:53.970498+0000","last_clean":"2024-04-17T07:30:53.970498+0000","last_became_active":"2024-04-17T07:30:52.966787+0000","last_became_peered":"2024-04-17T07:30:52.966787+0000","last_unstale":"2024-04-17T07:30:53.970498+0000","last_undegraded":"2024-04-17T07:30:53.970498+0000","last_fullsized":"2024-04-17T07:30:53.970498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:12.775821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960658+0000","last_change":"2024-04-17T07:30:51.959594+0000","last_active":"2024-04-17T07:30:53.960658+0000","last_peered":"2024-04-17T07:30:53.960658+0000","last_clean":"2024-04-17T07:30:53.960658+0000","last_became_active":"2024-04-17T07:30:51.959280+0000","last_became_peered":"2024-04-17T07:30:51.959280+0000","last_unstale":"2024-04-17T07:30:53.960658+0000","last_undegraded":"2024-04-17T07:30:53.960658+0000","last_fullsized":"2024-04-17T07:30:53.960658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:04.068840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970437+0000","last_change":"2024-04-17T07:30:52.966883+0000","last_active":"2024-04-17T07:30:53.970437+0000","last_peered":"2024-04-17T07:30:53.970437+0000","last_clean":"2024-04-17T07:30:53.970437+0000","last_became_active":"2024-04-17T07:30:52.966572+0000","last_became_peered":"2024-04-17T07:30:52.966572+0000","last_unstale":"2024-04-17T07:30:53.970437+0000","last_undegraded":"2024-04-17T07:30:53.970437+0000","last_fullsized":"2024-04-17T07:30:53.970437+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:46.840383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":22,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.974754+0000","last_change":"2024-04-17T07:30:51.960788+0000","last_active":"2024-04-17T07:30:53.974754+0000","last_peered":"2024-04-17T07:30:53.974754+0000","last_clean":"2024-04-17T07:30:53.974754+0000","last_became_active":"2024-04-17T07:30:51.960435+0000","last_became_peered":"2024-04-17T07:30:51.960435+0000","last_unstale":"2024-04-17T07:30:53.974754+0000","last_undegraded":"2024-04-17T07:30:53.974754+0000","last_fullsized":"2024-04-17T07:30:53.974754+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:31.671927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962695+0000","last_change":"2024-04-17T07:30:52.966356+0000","last_active":"2024-04-17T07:30:53.962695+0000","last_peered":"2024-04-17T07:30:53.962695+0000","last_clean":"2024-04-17T07:30:53.962695+0000","last_became_active":"2024-04-17T07:30:52.964607+0000","last_became_peered":"2024-04-17T07:30:52.964607+0000","last_unstale":"2024-04-17T07:30:53.962695+0000","last_undegraded":"2024-04-17T07:30:53.962695+0000","last_fullsized":"2024-04-17T07:30:53.962695+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:24.915493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.352301+0000","last_change":"2024-04-17T07:30:51.960793+0000","last_active":"2024-04-17T07:30:54.352301+0000","last_peered":"2024-04-17T07:30:54.352301+0000","last_clean":"2024-04-17T07:30:54.352301+0000","last_became_active":"2024-04-17T07:30:51.960106+0000","last_became_peered":"2024-04-17T07:30:51.960106+0000","last_unstale":"2024-04-17T07:30:54.352301+0000","last_undegraded":"2024-04-17T07:30:54.352301+0000","last_fullsized":"2024-04-17T07:30:54.352301+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:14.130056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351752+0000","last_change":"2024-04-17T07:30:52.954834+0000","last_active":"2024-04-17T07:30:54.351752+0000","last_peered":"2024-04-17T07:30:54.351752+0000","last_clean":"2024-04-17T07:30:54.351752+0000","last_became_active":"2024-04-17T07:30:52.954281+0000","last_became_peered":"2024-04-17T07:30:52.954281+0000","last_unstale":"2024-04-17T07:30:54.351752+0000","last_undegraded":"2024-04-17T07:30:54.351752+0000","last_fullsized":"2024-04-17T07:30:54.351752+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:06.794769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975523+0000","last_change":"2024-04-17T07:30:51.964058+0000","last_active":"2024-04-17T07:30:53.975523+0000","last_peered":"2024-04-17T07:30:53.975523+0000","last_clean":"2024-04-17T07:30:53.975523+0000","last_became_active":"2024-04-17T07:30:51.963735+0000","last_became_peered":"2024-04-17T07:30:51.963735+0000","last_unstale":"2024-04-17T07:30:53.975523+0000","last_undegraded":"2024-04-17T07:30:53.975523+0000","last_fullsized":"2024-04-17T07:30:53.975523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:35.278335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349511+0000","last_change":"2024-04-17T07:30:52.967661+0000","last_active":"2024-04-17T07:30:54.349511+0000","last_peered":"2024-04-17T07:30:54.349511+0000","last_clean":"2024-04-17T07:30:54.349511+0000","last_became_active":"2024-04-17T07:30:52.967377+0000","last_became_peered":"2024-04-17T07:30:52.967377+0000","last_unstale":"2024-04-17T07:30:54.349511+0000","last_undegraded":"2024-04-17T07:30:54.349511+0000","last_fullsized":"2024-04-17T07:30:54.349511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:30.663701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975173+0000","last_change":"2024-04-17T07:30:51.958304+0000","last_active":"2024-04-17T07:30:53.975173+0000","last_peered":"2024-04-17T07:30:53.975173+0000","last_clean":"2024-04-17T07:30:53.975173+0000","last_became_active":"2024-04-17T07:30:51.958030+0000","last_became_peered":"2024-04-17T07:30:51.958030+0000","last_unstale":"2024-04-17T07:30:53.975173+0000","last_undegraded":"2024-04-17T07:30:53.975173+0000","last_fullsized":"2024-04-17T07:30:53.975173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:03.425591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956633+0000","last_change":"2024-04-17T07:30:52.976683+0000","last_active":"2024-04-17T07:30:53.956633+0000","last_peered":"2024-04-17T07:30:53.956633+0000","last_clean":"2024-04-17T07:30:53.956633+0000","last_became_active":"2024-04-17T07:30:52.975990+0000","last_became_peered":"2024-04-17T07:30:52.975990+0000","last_unstale":"2024-04-17T07:30:53.956633+0000","last_undegraded":"2024-04-17T07:30:53.956633+0000","last_fullsized":"2024-04-17T07:30:53.956633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:08.067694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348204+0000","last_change":"2024-04-17T07:30:51.950098+0000","last_active":"2024-04-17T07:30:54.348204+0000","last_peered":"2024-04-17T07:30:54.348204+0000","last_clean":"2024-04-17T07:30:54.348204+0000","last_became_active":"2024-04-17T07:30:51.947543+0000","last_became_peered":"2024-04-17T07:30:51.947543+0000","last_unstale":"2024-04-17T07:30:54.348204+0000","last_undegraded":"2024-04-17T07:30:54.348204+0000","last_fullsized":"2024-04-17T07:30:54.348204+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:16.806530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960975+0000","last_change":"2024-04-17T07:30:52.960137+0000","last_active":"2024-04-17T07:30:53.960975+0000","last_peered":"2024-04-17T07:30:53.960975+0000","last_clean":"2024-04-17T07:30:53.960975+0000","last_became_active":"2024-04-17T07:30:52.959821+0000","last_became_peered":"2024-04-17T07:30:52.959821+0000","last_unstale":"2024-04-17T07:30:53.960975+0000","last_undegraded":"2024-04-17T07:30:53.960975+0000","last_fullsized":"2024-04-17T07:30:53.960975+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:39.534083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956662+0000","last_change":"2024-04-17T07:30:51.942316+0000","last_active":"2024-04-17T07:30:53.956662+0000","last_peered":"2024-04-17T07:30:53.956662+0000","last_clean":"2024-04-17T07:30:53.956662+0000","last_became_active":"2024-04-17T07:30:51.940838+0000","last_became_peered":"2024-04-17T07:30:51.940838+0000","last_unstale":"2024-04-17T07:30:53.956662+0000","last_undegraded":"2024-04-17T07:30:53.956662+0000","last_fullsized":"2024-04-17T07:30:53.956662+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:07.870223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956088+0000","last_change":"2024-04-17T07:30:52.964136+0000","last_active":"2024-04-17T07:30:53.956088+0000","last_peered":"2024-04-17T07:30:53.956088+0000","last_clean":"2024-04-17T07:30:53.956088+0000","last_became_active":"2024-04-17T07:30:52.957533+0000","last_became_peered":"2024-04-17T07:30:52.957533+0000","last_unstale":"2024-04-17T07:30:53.956088+0000","last_undegraded":"2024-04-17T07:30:53.956088+0000","last_fullsized":"2024-04-17T07:30:53.956088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:51.942966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956271+0000","last_change":"2024-04-17T07:30:51.961102+0000","last_active":"2024-04-17T07:30:53.956271+0000","last_peered":"2024-04-17T07:30:53.956271+0000","last_clean":"2024-04-17T07:30:53.956271+0000","last_became_active":"2024-04-17T07:30:51.960823+0000","last_became_peered":"2024-04-17T07:30:51.960823+0000","last_unstale":"2024-04-17T07:30:53.956271+0000","last_undegraded":"2024-04-17T07:30:53.956271+0000","last_fullsized":"2024-04-17T07:30:53.956271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:52.513196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961297+0000","last_change":"2024-04-17T07:30:52.973772+0000","last_active":"2024-04-17T07:30:53.961297+0000","last_peered":"2024-04-17T07:30:53.961297+0000","last_clean":"2024-04-17T07:30:53.961297+0000","last_became_active":"2024-04-17T07:30:52.973498+0000","last_became_peered":"2024-04-17T07:30:52.973498+0000","last_unstale":"2024-04-17T07:30:53.961297+0000","last_undegraded":"2024-04-17T07:30:53.961297+0000","last_fullsized":"2024-04-17T07:30:53.961297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:17.951301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347388+0000","last_change":"2024-04-17T07:30:51.949336+0000","last_active":"2024-04-17T07:30:54.347388+0000","last_peered":"2024-04-17T07:30:54.347388+0000","last_clean":"2024-04-17T07:30:54.347388+0000","last_became_active":"2024-04-17T07:30:51.948756+0000","last_became_peered":"2024-04-17T07:30:51.948756+0000","last_unstale":"2024-04-17T07:30:54.347388+0000","last_undegraded":"2024-04-17T07:30:54.347388+0000","last_fullsized":"2024-04-17T07:30:54.347388+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:09:31.157119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349536+0000","last_change":"2024-04-17T07:30:52.953516+0000","last_active":"2024-04-17T07:30:54.349536+0000","last_peered":"2024-04-17T07:30:54.349536+0000","last_clean":"2024-04-17T07:30:54.349536+0000","last_became_active":"2024-04-17T07:30:52.953127+0000","last_became_peered":"2024-04-17T07:30:52.953127+0000","last_unstale":"2024-04-17T07:30:54.349536+0000","last_undegraded":"2024-04-17T07:30:54.349536+0000","last_fullsized":"2024-04-17T07:30:54.349536+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:47.405703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971558+0000","last_change":"2024-04-17T07:30:51.955313+0000","last_active":"2024-04-17T07:30:53.971558+0000","last_peered":"2024-04-17T07:30:53.971558+0000","last_clean":"2024-04-17T07:30:53.971558+0000","last_became_active":"2024-04-17T07:30:51.954895+0000","last_became_peered":"2024-04-17T07:30:51.954895+0000","last_unstale":"2024-04-17T07:30:53.971558+0000","last_undegraded":"2024-04-17T07:30:53.971558+0000","last_fullsized":"2024-04-17T07:30:53.971558+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:04.272085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962769+0000","last_change":"2024-04-17T07:30:52.972154+0000","last_active":"2024-04-17T07:30:53.962769+0000","last_peered":"2024-04-17T07:30:53.962769+0000","last_clean":"2024-04-17T07:30:53.962769+0000","last_became_active":"2024-04-17T07:30:52.971886+0000","last_became_peered":"2024-04-17T07:30:52.971886+0000","last_unstale":"2024-04-17T07:30:53.962769+0000","last_undegraded":"2024-04-17T07:30:53.962769+0000","last_fullsized":"2024-04-17T07:30:53.962769+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:07.228548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"111'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.973623+0000","last_change":"2024-04-17T07:30:51.953140+0000","last_active":"2024-04-17T07:30:53.973623+0000","last_peered":"2024-04-17T07:30:53.973623+0000","last_clean":"2024-04-17T07:30:53.973623+0000","last_became_active":"2024-04-17T07:30:51.952808+0000","last_became_peered":"2024-04-17T07:30:51.952808+0000","last_unstale":"2024-04-17T07:30:53.973623+0000","last_undegraded":"2024-04-17T07:30:53.973623+0000","last_fullsized":"2024-04-17T07:30:53.973623+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:09:35.415227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351069+0000","last_change":"2024-04-17T07:30:52.959706+0000","last_active":"2024-04-17T07:30:54.351069+0000","last_peered":"2024-04-17T07:30:54.351069+0000","last_clean":"2024-04-17T07:30:54.351069+0000","last_became_active":"2024-04-17T07:30:52.959401+0000","last_became_peered":"2024-04-17T07:30:52.959401+0000","last_unstale":"2024-04-17T07:30:54.351069+0000","last_undegraded":"2024-04-17T07:30:54.351069+0000","last_fullsized":"2024-04-17T07:30:54.351069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:21.601807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350735+0000","last_change":"2024-04-17T07:30:51.960925+0000","last_active":"2024-04-17T07:30:54.350735+0000","last_peered":"2024-04-17T07:30:54.350735+0000","last_clean":"2024-04-17T07:30:54.350735+0000","last_became_active":"2024-04-17T07:30:51.960619+0000","last_became_peered":"2024-04-17T07:30:51.960619+0000","last_unstale":"2024-04-17T07:30:54.350735+0000","last_undegraded":"2024-04-17T07:30:54.350735+0000","last_fullsized":"2024-04-17T07:30:54.350735+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:32.948486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349621+0000","last_change":"2024-04-17T07:30:52.971908+0000","last_active":"2024-04-17T07:30:54.349621+0000","last_peered":"2024-04-17T07:30:54.349621+0000","last_clean":"2024-04-17T07:30:54.349621+0000","last_became_active":"2024-04-17T07:30:52.970476+0000","last_became_peered":"2024-04-17T07:30:52.970476+0000","last_unstale":"2024-04-17T07:30:54.349621+0000","last_undegraded":"2024-04-17T07:30:54.349621+0000","last_fullsized":"2024-04-17T07:30:54.349621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:57.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351061+0000","last_change":"2024-04-17T07:30:51.963126+0000","last_active":"2024-04-17T07:30:54.351061+0000","last_peered":"2024-04-17T07:30:54.351061+0000","last_clean":"2024-04-17T07:30:54.351061+0000","last_became_active":"2024-04-17T07:30:51.959481+0000","last_became_peered":"2024-04-17T07:30:51.959481+0000","last_unstale":"2024-04-17T07:30:54.351061+0000","last_undegraded":"2024-04-17T07:30:54.351061+0000","last_fullsized":"2024-04-17T07:30:54.351061+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:25.004071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961883+0000","last_change":"2024-04-17T07:30:52.958100+0000","last_active":"2024-04-17T07:30:53.961883+0000","last_peered":"2024-04-17T07:30:53.961883+0000","last_clean":"2024-04-17T07:30:53.961883+0000","last_became_active":"2024-04-17T07:30:52.957780+0000","last_became_peered":"2024-04-17T07:30:52.957780+0000","last_unstale":"2024-04-17T07:30:53.961883+0000","last_undegraded":"2024-04-17T07:30:53.961883+0000","last_fullsized":"2024-04-17T07:30:53.961883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:00:13.982768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975453+0000","last_change":"2024-04-17T07:30:51.962209+0000","last_active":"2024-04-17T07:30:53.975453+0000","last_peered":"2024-04-17T07:30:53.975453+0000","last_clean":"2024-04-17T07:30:53.975453+0000","last_became_active":"2024-04-17T07:30:51.961902+0000","last_became_peered":"2024-04-17T07:30:51.961902+0000","last_unstale":"2024-04-17T07:30:53.975453+0000","last_undegraded":"2024-04-17T07:30:53.975453+0000","last_fullsized":"2024-04-17T07:30:53.975453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:05.514327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975286+0000","last_change":"2024-04-17T07:30:52.959950+0000","last_active":"2024-04-17T07:30:53.975286+0000","last_peered":"2024-04-17T07:30:53.975286+0000","last_clean":"2024-04-17T07:30:53.975286+0000","last_became_active":"2024-04-17T07:30:52.958965+0000","last_became_peered":"2024-04-17T07:30:52.958965+0000","last_unstale":"2024-04-17T07:30:53.975286+0000","last_undegraded":"2024-04-17T07:30:53.975286+0000","last_fullsized":"2024-04-17T07:30:53.975286+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:42.419341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348012+0000","last_change":"2024-04-17T07:30:51.955420+0000","last_active":"2024-04-17T07:30:54.348012+0000","last_peered":"2024-04-17T07:30:54.348012+0000","last_clean":"2024-04-17T07:30:54.348012+0000","last_became_active":"2024-04-17T07:30:51.955099+0000","last_became_peered":"2024-04-17T07:30:51.955099+0000","last_unstale":"2024-04-17T07:30:54.348012+0000","last_undegraded":"2024-04-17T07:30:54.348012+0000","last_fullsized":"2024-04-17T07:30:54.348012+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:05.683136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960268+0000","last_change":"2024-04-17T07:30:52.976237+0000","last_active":"2024-04-17T07:30:53.960268+0000","last_peered":"2024-04-17T07:30:53.960268+0000","last_clean":"2024-04-17T07:30:53.960268+0000","last_became_active":"2024-04-17T07:30:52.975910+0000","last_became_peered":"2024-04-17T07:30:52.975910+0000","last_unstale":"2024-04-17T07:30:53.960268+0000","last_undegraded":"2024-04-17T07:30:53.960268+0000","last_fullsized":"2024-04-17T07:30:53.960268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.108657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350145+0000","last_change":"2024-04-17T07:30:51.965391+0000","last_active":"2024-04-17T07:30:54.350145+0000","last_peered":"2024-04-17T07:30:54.350145+0000","last_clean":"2024-04-17T07:30:54.350145+0000","last_became_active":"2024-04-17T07:30:51.964945+0000","last_became_peered":"2024-04-17T07:30:51.964945+0000","last_unstale":"2024-04-17T07:30:54.350145+0000","last_undegraded":"2024-04-17T07:30:54.350145+0000","last_fullsized":"2024-04-17T07:30:54.350145+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:30.790943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349867+0000","last_change":"2024-04-17T07:30:52.958641+0000","last_active":"2024-04-17T07:30:54.349867+0000","last_peered":"2024-04-17T07:30:54.349867+0000","last_clean":"2024-04-17T07:30:54.349867+0000","last_became_active":"2024-04-17T07:30:52.958307+0000","last_became_peered":"2024-04-17T07:30:52.958307+0000","last_unstale":"2024-04-17T07:30:54.349867+0000","last_undegraded":"2024-04-17T07:30:54.349867+0000","last_fullsized":"2024-04-17T07:30:54.349867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:10.602698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347983+0000","last_change":"2024-04-17T07:30:51.962718+0000","last_active":"2024-04-17T07:30:54.347983+0000","last_peered":"2024-04-17T07:30:54.347983+0000","last_clean":"2024-04-17T07:30:54.347983+0000","last_became_active":"2024-04-17T07:30:51.961694+0000","last_became_peered":"2024-04-17T07:30:51.961694+0000","last_unstale":"2024-04-17T07:30:54.347983+0000","last_undegraded":"2024-04-17T07:30:54.347983+0000","last_fullsized":"2024-04-17T07:30:54.347983+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:46.032006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971492+0000","last_change":"2024-04-17T07:30:52.969957+0000","last_active":"2024-04-17T07:30:53.971492+0000","last_peered":"2024-04-17T07:30:53.971492+0000","last_clean":"2024-04-17T07:30:53.971492+0000","last_became_active":"2024-04-17T07:30:52.969578+0000","last_became_peered":"2024-04-17T07:30:52.969578+0000","last_unstale":"2024-04-17T07:30:53.971492+0000","last_undegraded":"2024-04-17T07:30:53.971492+0000","last_fullsized":"2024-04-17T07:30:53.971492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:22:41.631018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.963050+0000","last_change":"2024-04-17T07:30:51.966520+0000","last_active":"2024-04-17T07:30:53.963050+0000","last_peered":"2024-04-17T07:30:53.963050+0000","last_clean":"2024-04-17T07:30:53.963050+0000","last_became_active":"2024-04-17T07:30:51.966147+0000","last_became_peered":"2024-04-17T07:30:51.966147+0000","last_unstale":"2024-04-17T07:30:53.963050+0000","last_undegraded":"2024-04-17T07:30:53.963050+0000","last_fullsized":"2024-04-17T07:30:53.963050+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:46.555431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"100'189","reported_seq":600,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960413+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:30:53.960413+0000","last_peered":"2024-04-17T07:30:53.960413+0000","last_clean":"2024-04-17T07:30:53.960413+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:30:53.960413+0000","last_undegraded":"2024-04-17T07:30:53.960413+0000","last_fullsized":"2024-04-17T07:30:53.960413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":287,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":287,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542348,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1957888,"data_stored":1238505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542348,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":648761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542347,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1376256,"data_stored":649701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542347,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1359872,"data_stored":648067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542349,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36688,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335152,"statfs":{"total":96636764160,"available":96599195648,"internally_reserved":0,"allocated":1372160,"data_stored":649679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542348,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1363968,"data_stored":648649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542349,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54736,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317104,"statfs":{"total":96636764160,"available":96580714496,"internally_reserved":0,"allocated":1372160,"data_stored":648261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542347,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":1238469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:30:57.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:30:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.771+0000 7ff15fe00640 1 Processor -- start 2024-04-17T07:30:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.771+0000 7ff15fe00640 1 -- start start 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff15fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff16015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff15fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16015f440 con 0x7ff1600b4210 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff15fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16015f720 con 0x7ff1600afb50 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff15fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff16015fa00 con 0x7ff16015c5a0 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff156000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff1600b4210 0x7ff1600af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54714/0 (socket says 172.21.15.67:54714) 2024-04-17T07:30:57.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff156000640 1 -- 172.21.15.67:0/1925225844 learned_addr learned my addr 172.21.15.67:0/1925225844 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.773+0000 7ff15ea00640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff16015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724103281 0 0) 0x7ff16015f440 con 0x7ff1600b4210 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff144003520 con 0x7ff1600b4210 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2143597804 0 0) 0x7ff144003520 con 0x7ff1600b4210 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 msgr2=0x7ff16015ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff16015ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 >> v1:172.21.15.187:6789/0 conn(0x7ff1600afb50 legacy=0x7ff1600ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16015fce0 con 0x7ff1600b4210 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff148001670 con 0x7ff1600b4210 2024-04-17T07:30:57.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 >> v1:172.21.15.67:6789/0 conn(0x7ff1600b4210 legacy=0x7ff1600af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff144003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff144004020 0x7ff144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff144005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff16015fa00 con 0x7ff144004d90 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff14400bca0 con 0x7ff144004020 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.774+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff14400bea0 con 0x7ff16015c5a0 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff144005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15ea00640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff144003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff156000640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff144004020 0x7ff144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 -- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 msgr2=0x7ff144003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff16015c5a0 0x7ff144003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 -- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff144004020 msgr2=0x7ff144004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff144004020 0x7ff144004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.775+0000 7ff15f400640 1 -- 172.21.15.67:0/1925225844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff144003520 con 0x7ff144004d90 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.776+0000 7ff15f400640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff144005210 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff160073170 tx=0x7ff144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f41ae38c52541d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158001a70 con 0x7ff144004d90 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff158003b70 con 0x7ff144004d90 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15e000640 1 -- 172.21.15.67:0/1925225844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1580050d0 con 0x7ff144004d90 2024-04-17T07:30:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15fe00640 1 -- 172.21.15.67:0/1925225844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 msgr2=0x7ff144005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15fe00640 1 --2- 172.21.15.67:0/1925225844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff144005210 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff160073170 tx=0x7ff144005830 comp rx=0 tx=0).stop 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15f400640 1 -- 172.21.15.67:0/1925225844 reap_dead start 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15fe00640 1 -- 172.21.15.67:0/1925225844 shutdown_connections 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15fe00640 1 -- 172.21.15.67:0/1925225844 >> 172.21.15.67:0/1925225844 conn(0x7ff160067d10 msgr2=0x7ff1600b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.777+0000 7ff15fe00640 1 -- 172.21.15.67:0/1925225844 shutdown_connections 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.778+0000 7ff15fe00640 1 -- 172.21.15.67:0/1925225844 wait complete. 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.779+0000 7ff15fe00640 1 Processor -- start 2024-04-17T07:30:57.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.779+0000 7ff15fe00640 1 -- start start 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff160156be0 0x7ff160157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1601e5a00 0x7ff1601e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff16006e7f0 con 0x7ff144004d90 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff16006e670 con 0x7ff1601e5a00 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.780+0000 7ff15fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1600b4690 con 0x7ff160156be0 2024-04-17T07:30:57.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51748/0 (socket says 172.21.15.67:51748) 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 -- 172.21.15.67:0/944247191 learned_addr learned my addr 172.21.15.67:0/944247191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff156000640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff160156be0 0x7ff160157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 -- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff160156be0 msgr2=0x7ff160157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff160156be0 0x7ff160157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 -- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1601e5a00 msgr2=0x7ff1601e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1601e5a00 0x7ff1601e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:57.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.781+0000 7ff15ea00640 1 -- 172.21.15.67:0/944247191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff160157940 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff15ea00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff150011630 tx=0x7ff150011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15000f920 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff15000fac0 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1601e4f20 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15000fe70 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.782+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1601575a0 con 0x7ff144004d90 2024-04-17T07:30:57.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.783+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff160068a10 con 0x7ff144004d90 2024-04-17T07:30:57.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.784+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7ff150015460 con 0x7ff144004d90 2024-04-17T07:30:57.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.785+0000 7ff157e00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 0x7ff13c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:57.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.785+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7ff15008c630 con 0x7ff144004d90 2024-04-17T07:30:57.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.786+0000 7ff156000640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 0x7ff13c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:57.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.787+0000 7ff156000640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 0x7ff13c06ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff148001eb0 tx=0x7ff148001d40 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:57.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.790+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff150019030 con 0x7ff144004d90 2024-04-17T07:30:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:57.993+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 --> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff16006c010 con 0x7ff13c06ca40 2024-04-17T07:30:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.013+0000 7ff157e00640 1 -- 172.21.15.67:0/944247191 <== mgr.6091 v2:172.21.15.187:6832/158797128 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252491 (secure 0 0 0) 0x7ff16006c010 con 0x7ff13c06ca40 2024-04-17T07:30:58.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 msgr2=0x7ff13c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 0x7ff13c06ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff148001eb0 tx=0x7ff148001d40 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 msgr2=0x7ff1601e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff150011630 tx=0x7ff150011af0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 shutdown_connections 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7ff13c06ca40 0x7ff13c06ef00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1601e5a00 0x7ff1601e5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff160156be0 0x7ff160157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 --2- 172.21.15.67:0/944247191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144004d90 0x7ff1601e7200 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.019+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 >> 172.21.15.67:0/944247191 conn(0x7ff160067d10 msgr2=0x7ff1600b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.020+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 shutdown_connections 2024-04-17T07:30:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.020+0000 7ff15fe00640 1 -- 172.21.15.67:0/944247191 wait complete. 2024-04-17T07:30:58.038 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-17T07:30:56.365114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":332,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330720,"kb_used_data":11848,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754644000,"statfs":{"total":773094113280,"available":772755456000,"internally_reserved":0,"allocated":12132352,"data_stored":6370092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":326356528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456635"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351352+0000","last_change":"2024-04-17T07:30:52.953522+0000","last_active":"2024-04-17T07:30:54.351352+0000","last_peered":"2024-04-17T07:30:54.351352+0000","last_clean":"2024-04-17T07:30:54.351352+0000","last_became_active":"2024-04-17T07:30:52.953220+0000","last_became_peered":"2024-04-17T07:30:52.953220+0000","last_unstale":"2024-04-17T07:30:54.351352+0000","last_undegraded":"2024-04-17T07:30:54.351352+0000","last_fullsized":"2024-04-17T07:30:54.351352+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:12.504889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971548+0000","last_change":"2024-04-17T07:30:52.973940+0000","last_active":"2024-04-17T07:30:53.971548+0000","last_peered":"2024-04-17T07:30:53.971548+0000","last_clean":"2024-04-17T07:30:53.971548+0000","last_became_active":"2024-04-17T07:30:52.972663+0000","last_became_peered":"2024-04-17T07:30:52.972663+0000","last_unstale":"2024-04-17T07:30:53.971548+0000","last_undegraded":"2024-04-17T07:30:53.971548+0000","last_fullsized":"2024-04-17T07:30:53.971548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:40:22.130567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962476+0000","last_change":"2024-04-17T07:30:52.961899+0000","last_active":"2024-04-17T07:30:53.962476+0000","last_peered":"2024-04-17T07:30:53.962476+0000","last_clean":"2024-04-17T07:30:53.962476+0000","last_became_active":"2024-04-17T07:30:52.961623+0000","last_became_peered":"2024-04-17T07:30:52.961623+0000","last_unstale":"2024-04-17T07:30:53.962476+0000","last_undegraded":"2024-04-17T07:30:53.962476+0000","last_fullsized":"2024-04-17T07:30:53.962476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:36:21.177559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956895+0000","last_change":"2024-04-17T07:30:52.973925+0000","last_active":"2024-04-17T07:30:53.956895+0000","last_peered":"2024-04-17T07:30:53.956895+0000","last_clean":"2024-04-17T07:30:53.956895+0000","last_became_active":"2024-04-17T07:30:52.973641+0000","last_became_peered":"2024-04-17T07:30:52.973641+0000","last_unstale":"2024-04-17T07:30:53.956895+0000","last_undegraded":"2024-04-17T07:30:53.956895+0000","last_fullsized":"2024-04-17T07:30:53.956895+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:07.348218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348313+0000","last_change":"2024-04-17T07:30:52.968791+0000","last_active":"2024-04-17T07:30:54.348313+0000","last_peered":"2024-04-17T07:30:54.348313+0000","last_clean":"2024-04-17T07:30:54.348313+0000","last_became_active":"2024-04-17T07:30:52.968540+0000","last_became_peered":"2024-04-17T07:30:52.968540+0000","last_unstale":"2024-04-17T07:30:54.348313+0000","last_undegraded":"2024-04-17T07:30:54.348313+0000","last_fullsized":"2024-04-17T07:30:54.348313+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:07.322541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960641+0000","last_change":"2024-04-17T07:30:52.963724+0000","last_active":"2024-04-17T07:30:53.960641+0000","last_peered":"2024-04-17T07:30:53.960641+0000","last_clean":"2024-04-17T07:30:53.960641+0000","last_became_active":"2024-04-17T07:30:52.963401+0000","last_became_peered":"2024-04-17T07:30:52.963401+0000","last_unstale":"2024-04-17T07:30:53.960641+0000","last_undegraded":"2024-04-17T07:30:53.960641+0000","last_fullsized":"2024-04-17T07:30:53.960641+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:33.061753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956171+0000","last_change":"2024-04-17T07:30:52.962235+0000","last_active":"2024-04-17T07:30:53.956171+0000","last_peered":"2024-04-17T07:30:53.956171+0000","last_clean":"2024-04-17T07:30:53.956171+0000","last_became_active":"2024-04-17T07:30:52.958407+0000","last_became_peered":"2024-04-17T07:30:52.958407+0000","last_unstale":"2024-04-17T07:30:53.956171+0000","last_undegraded":"2024-04-17T07:30:53.956171+0000","last_fullsized":"2024-04-17T07:30:53.956171+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:11.653065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960088+0000","last_change":"2024-04-17T07:30:52.976325+0000","last_active":"2024-04-17T07:30:53.960088+0000","last_peered":"2024-04-17T07:30:53.960088+0000","last_clean":"2024-04-17T07:30:53.960088+0000","last_became_active":"2024-04-17T07:30:52.975636+0000","last_became_peered":"2024-04-17T07:30:52.975636+0000","last_unstale":"2024-04-17T07:30:53.960088+0000","last_undegraded":"2024-04-17T07:30:53.960088+0000","last_fullsized":"2024-04-17T07:30:53.960088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:41.928147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961419+0000","last_change":"2024-04-17T07:30:52.973754+0000","last_active":"2024-04-17T07:30:53.961419+0000","last_peered":"2024-04-17T07:30:53.961419+0000","last_clean":"2024-04-17T07:30:53.961419+0000","last_became_active":"2024-04-17T07:30:52.973416+0000","last_became_peered":"2024-04-17T07:30:52.973416+0000","last_unstale":"2024-04-17T07:30:53.961419+0000","last_undegraded":"2024-04-17T07:30:53.961419+0000","last_fullsized":"2024-04-17T07:30:53.961419+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:01.659599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351123+0000","last_change":"2024-04-17T07:30:52.966095+0000","last_active":"2024-04-17T07:30:54.351123+0000","last_peered":"2024-04-17T07:30:54.351123+0000","last_clean":"2024-04-17T07:30:54.351123+0000","last_became_active":"2024-04-17T07:30:52.965735+0000","last_became_peered":"2024-04-17T07:30:52.965735+0000","last_unstale":"2024-04-17T07:30:54.351123+0000","last_undegraded":"2024-04-17T07:30:54.351123+0000","last_fullsized":"2024-04-17T07:30:54.351123+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:01:07.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971430+0000","last_change":"2024-04-17T07:30:52.964690+0000","last_active":"2024-04-17T07:30:53.971430+0000","last_peered":"2024-04-17T07:30:53.971430+0000","last_clean":"2024-04-17T07:30:53.971430+0000","last_became_active":"2024-04-17T07:30:52.964340+0000","last_became_peered":"2024-04-17T07:30:52.964340+0000","last_unstale":"2024-04-17T07:30:53.971430+0000","last_undegraded":"2024-04-17T07:30:53.971430+0000","last_fullsized":"2024-04-17T07:30:53.971430+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:06.794769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350413+0000","last_change":"2024-04-17T07:30:52.965459+0000","last_active":"2024-04-17T07:30:54.350413+0000","last_peered":"2024-04-17T07:30:54.350413+0000","last_clean":"2024-04-17T07:30:54.350413+0000","last_became_active":"2024-04-17T07:30:52.965156+0000","last_became_peered":"2024-04-17T07:30:52.965156+0000","last_unstale":"2024-04-17T07:30:54.350413+0000","last_undegraded":"2024-04-17T07:30:54.350413+0000","last_fullsized":"2024-04-17T07:30:54.350413+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:16.623613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962295+0000","last_change":"2024-04-17T07:30:52.969490+0000","last_active":"2024-04-17T07:30:53.962295+0000","last_peered":"2024-04-17T07:30:53.962295+0000","last_clean":"2024-04-17T07:30:53.962295+0000","last_became_active":"2024-04-17T07:30:52.969178+0000","last_became_peered":"2024-04-17T07:30:52.969178+0000","last_unstale":"2024-04-17T07:30:53.962295+0000","last_undegraded":"2024-04-17T07:30:53.962295+0000","last_fullsized":"2024-04-17T07:30:53.962295+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:59.001615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956407+0000","last_change":"2024-04-17T07:30:52.978051+0000","last_active":"2024-04-17T07:30:53.956407+0000","last_peered":"2024-04-17T07:30:53.956407+0000","last_clean":"2024-04-17T07:30:53.956407+0000","last_became_active":"2024-04-17T07:30:52.976371+0000","last_became_peered":"2024-04-17T07:30:52.976371+0000","last_unstale":"2024-04-17T07:30:53.956407+0000","last_undegraded":"2024-04-17T07:30:53.956407+0000","last_fullsized":"2024-04-17T07:30:53.956407+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:36.782623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347549+0000","last_change":"2024-04-17T07:30:52.961329+0000","last_active":"2024-04-17T07:30:54.347549+0000","last_peered":"2024-04-17T07:30:54.347549+0000","last_clean":"2024-04-17T07:30:54.347549+0000","last_became_active":"2024-04-17T07:30:52.960990+0000","last_became_peered":"2024-04-17T07:30:52.960990+0000","last_unstale":"2024-04-17T07:30:54.347549+0000","last_undegraded":"2024-04-17T07:30:54.347549+0000","last_fullsized":"2024-04-17T07:30:54.347549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:33.550564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347795+0000","last_change":"2024-04-17T07:30:52.951515+0000","last_active":"2024-04-17T07:30:54.347795+0000","last_peered":"2024-04-17T07:30:54.347795+0000","last_clean":"2024-04-17T07:30:54.347795+0000","last_became_active":"2024-04-17T07:30:52.951036+0000","last_became_peered":"2024-04-17T07:30:52.951036+0000","last_unstale":"2024-04-17T07:30:54.347795+0000","last_undegraded":"2024-04-17T07:30:54.347795+0000","last_fullsized":"2024-04-17T07:30:54.347795+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:40.463465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349694+0000","last_change":"2024-04-17T07:30:52.965372+0000","last_active":"2024-04-17T07:30:54.349694+0000","last_peered":"2024-04-17T07:30:54.349694+0000","last_clean":"2024-04-17T07:30:54.349694+0000","last_became_active":"2024-04-17T07:30:52.965061+0000","last_became_peered":"2024-04-17T07:30:52.965061+0000","last_unstale":"2024-04-17T07:30:54.349694+0000","last_undegraded":"2024-04-17T07:30:54.349694+0000","last_fullsized":"2024-04-17T07:30:54.349694+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:27.930242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.957113+0000","last_change":"2024-04-17T07:30:52.955873+0000","last_active":"2024-04-17T07:30:53.957113+0000","last_peered":"2024-04-17T07:30:53.957113+0000","last_clean":"2024-04-17T07:30:53.957113+0000","last_became_active":"2024-04-17T07:30:52.955558+0000","last_became_peered":"2024-04-17T07:30:52.955558+0000","last_unstale":"2024-04-17T07:30:53.957113+0000","last_undegraded":"2024-04-17T07:30:53.957113+0000","last_fullsized":"2024-04-17T07:30:53.957113+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:31.353125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975140+0000","last_change":"2024-04-17T07:30:52.983064+0000","last_active":"2024-04-17T07:30:53.975140+0000","last_peered":"2024-04-17T07:30:53.975140+0000","last_clean":"2024-04-17T07:30:53.975140+0000","last_became_active":"2024-04-17T07:30:52.981954+0000","last_became_peered":"2024-04-17T07:30:52.981954+0000","last_unstale":"2024-04-17T07:30:53.975140+0000","last_undegraded":"2024-04-17T07:30:53.975140+0000","last_fullsized":"2024-04-17T07:30:53.975140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:47.172943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.957106+0000","last_change":"2024-04-17T07:30:52.963736+0000","last_active":"2024-04-17T07:30:53.957106+0000","last_peered":"2024-04-17T07:30:53.957106+0000","last_clean":"2024-04-17T07:30:53.957106+0000","last_became_active":"2024-04-17T07:30:52.958146+0000","last_became_peered":"2024-04-17T07:30:52.958146+0000","last_unstale":"2024-04-17T07:30:53.957106+0000","last_undegraded":"2024-04-17T07:30:53.957106+0000","last_fullsized":"2024-04-17T07:30:53.957106+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:02:40.068613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956637+0000","last_change":"2024-04-17T07:30:52.960480+0000","last_active":"2024-04-17T07:30:53.956637+0000","last_peered":"2024-04-17T07:30:53.956637+0000","last_clean":"2024-04-17T07:30:53.956637+0000","last_became_active":"2024-04-17T07:30:52.958156+0000","last_became_peered":"2024-04-17T07:30:52.958156+0000","last_unstale":"2024-04-17T07:30:53.956637+0000","last_undegraded":"2024-04-17T07:30:53.956637+0000","last_fullsized":"2024-04-17T07:30:53.956637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:36.125345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971346+0000","last_change":"2024-04-17T07:30:52.978117+0000","last_active":"2024-04-17T07:30:53.971346+0000","last_peered":"2024-04-17T07:30:53.971346+0000","last_clean":"2024-04-17T07:30:53.971346+0000","last_became_active":"2024-04-17T07:30:52.974801+0000","last_became_peered":"2024-04-17T07:30:52.974801+0000","last_unstale":"2024-04-17T07:30:53.971346+0000","last_undegraded":"2024-04-17T07:30:53.971346+0000","last_fullsized":"2024-04-17T07:30:53.971346+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.458301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349069+0000","last_change":"2024-04-17T07:30:52.978968+0000","last_active":"2024-04-17T07:30:54.349069+0000","last_peered":"2024-04-17T07:30:54.349069+0000","last_clean":"2024-04-17T07:30:54.349069+0000","last_became_active":"2024-04-17T07:30:52.978240+0000","last_became_peered":"2024-04-17T07:30:52.978240+0000","last_unstale":"2024-04-17T07:30:54.349069+0000","last_undegraded":"2024-04-17T07:30:54.349069+0000","last_fullsized":"2024-04-17T07:30:54.349069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:00:22.313647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956048+0000","last_change":"2024-04-17T07:30:52.970047+0000","last_active":"2024-04-17T07:30:53.956048+0000","last_peered":"2024-04-17T07:30:53.956048+0000","last_clean":"2024-04-17T07:30:53.956048+0000","last_became_active":"2024-04-17T07:30:52.969690+0000","last_became_peered":"2024-04-17T07:30:52.969690+0000","last_unstale":"2024-04-17T07:30:53.956048+0000","last_undegraded":"2024-04-17T07:30:53.956048+0000","last_fullsized":"2024-04-17T07:30:53.956048+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:42.419341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971471+0000","last_change":"2024-04-17T07:30:52.966745+0000","last_active":"2024-04-17T07:30:53.971471+0000","last_peered":"2024-04-17T07:30:53.971471+0000","last_clean":"2024-04-17T07:30:53.971471+0000","last_became_active":"2024-04-17T07:30:52.966427+0000","last_became_peered":"2024-04-17T07:30:52.966427+0000","last_unstale":"2024-04-17T07:30:53.971471+0000","last_undegraded":"2024-04-17T07:30:53.971471+0000","last_fullsized":"2024-04-17T07:30:53.971471+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:06.516299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962518+0000","last_change":"2024-04-17T07:30:52.975543+0000","last_active":"2024-04-17T07:30:53.962518+0000","last_peered":"2024-04-17T07:30:53.962518+0000","last_clean":"2024-04-17T07:30:53.962518+0000","last_became_active":"2024-04-17T07:30:52.975219+0000","last_became_peered":"2024-04-17T07:30:52.975219+0000","last_unstale":"2024-04-17T07:30:53.962518+0000","last_undegraded":"2024-04-17T07:30:53.962518+0000","last_fullsized":"2024-04-17T07:30:53.962518+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:45.152848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348633+0000","last_change":"2024-04-17T07:30:52.957742+0000","last_active":"2024-04-17T07:30:54.348633+0000","last_peered":"2024-04-17T07:30:54.348633+0000","last_clean":"2024-04-17T07:30:54.348633+0000","last_became_active":"2024-04-17T07:30:52.957473+0000","last_became_peered":"2024-04-17T07:30:52.957473+0000","last_unstale":"2024-04-17T07:30:54.348633+0000","last_undegraded":"2024-04-17T07:30:54.348633+0000","last_fullsized":"2024-04-17T07:30:54.348633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:36.153994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350626+0000","last_change":"2024-04-17T07:30:52.954271+0000","last_active":"2024-04-17T07:30:54.350626+0000","last_peered":"2024-04-17T07:30:54.350626+0000","last_clean":"2024-04-17T07:30:54.350626+0000","last_became_active":"2024-04-17T07:30:52.953990+0000","last_became_peered":"2024-04-17T07:30:52.953990+0000","last_unstale":"2024-04-17T07:30:54.350626+0000","last_undegraded":"2024-04-17T07:30:54.350626+0000","last_fullsized":"2024-04-17T07:30:54.350626+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:46.840383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971060+0000","last_change":"2024-04-17T07:30:52.967804+0000","last_active":"2024-04-17T07:30:53.971060+0000","last_peered":"2024-04-17T07:30:53.971060+0000","last_clean":"2024-04-17T07:30:53.971060+0000","last_became_active":"2024-04-17T07:30:52.967470+0000","last_became_peered":"2024-04-17T07:30:52.967470+0000","last_unstale":"2024-04-17T07:30:53.971060+0000","last_undegraded":"2024-04-17T07:30:53.971060+0000","last_fullsized":"2024-04-17T07:30:53.971060+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:01:07.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970933+0000","last_change":"2024-04-17T07:30:52.971295+0000","last_active":"2024-04-17T07:30:53.970933+0000","last_peered":"2024-04-17T07:30:53.970933+0000","last_clean":"2024-04-17T07:30:53.970933+0000","last_became_active":"2024-04-17T07:30:52.970780+0000","last_became_peered":"2024-04-17T07:30:52.970780+0000","last_unstale":"2024-04-17T07:30:53.970933+0000","last_undegraded":"2024-04-17T07:30:53.970933+0000","last_fullsized":"2024-04-17T07:30:53.970933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:21.601807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347825+0000","last_change":"2024-04-17T07:30:52.972634+0000","last_active":"2024-04-17T07:30:54.347825+0000","last_peered":"2024-04-17T07:30:54.347825+0000","last_clean":"2024-04-17T07:30:54.347825+0000","last_became_active":"2024-04-17T07:30:52.972294+0000","last_became_peered":"2024-04-17T07:30:52.972294+0000","last_unstale":"2024-04-17T07:30:54.347825+0000","last_undegraded":"2024-04-17T07:30:54.347825+0000","last_fullsized":"2024-04-17T07:30:54.347825+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:06:21.544764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348919+0000","last_change":"2024-04-17T07:30:52.969843+0000","last_active":"2024-04-17T07:30:54.348919+0000","last_peered":"2024-04-17T07:30:54.348919+0000","last_clean":"2024-04-17T07:30:54.348919+0000","last_became_active":"2024-04-17T07:30:52.966533+0000","last_became_peered":"2024-04-17T07:30:52.966533+0000","last_unstale":"2024-04-17T07:30:54.348919+0000","last_undegraded":"2024-04-17T07:30:54.348919+0000","last_fullsized":"2024-04-17T07:30:54.348919+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:55.924704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349422+0000","last_change":"2024-04-17T07:30:52.961011+0000","last_active":"2024-04-17T07:30:54.349422+0000","last_peered":"2024-04-17T07:30:54.349422+0000","last_clean":"2024-04-17T07:30:54.349422+0000","last_became_active":"2024-04-17T07:30:52.960733+0000","last_became_peered":"2024-04-17T07:30:52.960733+0000","last_unstale":"2024-04-17T07:30:54.349422+0000","last_undegraded":"2024-04-17T07:30:54.349422+0000","last_fullsized":"2024-04-17T07:30:54.349422+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:54.024907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"110'4","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351818+0000","last_change":"2024-04-17T07:30:51.964315+0000","last_active":"2024-04-17T07:30:54.351818+0000","last_peered":"2024-04-17T07:30:54.351818+0000","last_clean":"2024-04-17T07:30:54.351818+0000","last_became_active":"2024-04-17T07:30:51.963522+0000","last_became_peered":"2024-04-17T07:30:51.963522+0000","last_unstale":"2024-04-17T07:30:54.351818+0000","last_undegraded":"2024-04-17T07:30:54.351818+0000","last_fullsized":"2024-04-17T07:30:54.351818+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:17:30.429277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351549+0000","last_change":"2024-04-17T07:30:52.978864+0000","last_active":"2024-04-17T07:30:54.351549+0000","last_peered":"2024-04-17T07:30:54.351549+0000","last_clean":"2024-04-17T07:30:54.351549+0000","last_became_active":"2024-04-17T07:30:52.978253+0000","last_became_peered":"2024-04-17T07:30:52.978253+0000","last_unstale":"2024-04-17T07:30:54.351549+0000","last_undegraded":"2024-04-17T07:30:54.351549+0000","last_fullsized":"2024-04-17T07:30:54.351549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.458301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349703+0000","last_change":"2024-04-17T07:30:51.954624+0000","last_active":"2024-04-17T07:30:54.349703+0000","last_peered":"2024-04-17T07:30:54.349703+0000","last_clean":"2024-04-17T07:30:54.349703+0000","last_became_active":"2024-04-17T07:30:51.954151+0000","last_became_peered":"2024-04-17T07:30:51.954151+0000","last_unstale":"2024-04-17T07:30:54.349703+0000","last_undegraded":"2024-04-17T07:30:54.349703+0000","last_fullsized":"2024-04-17T07:30:54.349703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:11.204766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350798+0000","last_change":"2024-04-17T07:30:52.970973+0000","last_active":"2024-04-17T07:30:54.350798+0000","last_peered":"2024-04-17T07:30:54.350798+0000","last_clean":"2024-04-17T07:30:54.350798+0000","last_became_active":"2024-04-17T07:30:52.970625+0000","last_became_peered":"2024-04-17T07:30:52.970625+0000","last_unstale":"2024-04-17T07:30:54.350798+0000","last_undegraded":"2024-04-17T07:30:54.350798+0000","last_fullsized":"2024-04-17T07:30:54.350798+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:12.775821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975346+0000","last_change":"2024-04-17T07:30:51.957357+0000","last_active":"2024-04-17T07:30:53.975346+0000","last_peered":"2024-04-17T07:30:53.975346+0000","last_clean":"2024-04-17T07:30:53.975346+0000","last_became_active":"2024-04-17T07:30:51.956633+0000","last_became_peered":"2024-04-17T07:30:51.956633+0000","last_unstale":"2024-04-17T07:30:53.975346+0000","last_undegraded":"2024-04-17T07:30:53.975346+0000","last_fullsized":"2024-04-17T07:30:53.975346+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:07.870223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962880+0000","last_change":"2024-04-17T07:30:52.969777+0000","last_active":"2024-04-17T07:30:53.962880+0000","last_peered":"2024-04-17T07:30:53.962880+0000","last_clean":"2024-04-17T07:30:53.962880+0000","last_became_active":"2024-04-17T07:30:52.969101+0000","last_became_peered":"2024-04-17T07:30:52.969101+0000","last_unstale":"2024-04-17T07:30:53.962880+0000","last_undegraded":"2024-04-17T07:30:53.962880+0000","last_fullsized":"2024-04-17T07:30:53.962880+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:35.282675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975038+0000","last_change":"2024-04-17T07:30:51.957716+0000","last_active":"2024-04-17T07:30:53.975038+0000","last_peered":"2024-04-17T07:30:53.975038+0000","last_clean":"2024-04-17T07:30:53.975038+0000","last_became_active":"2024-04-17T07:30:51.957423+0000","last_became_peered":"2024-04-17T07:30:51.957423+0000","last_unstale":"2024-04-17T07:30:53.975038+0000","last_undegraded":"2024-04-17T07:30:53.975038+0000","last_fullsized":"2024-04-17T07:30:53.975038+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:18.127344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971116+0000","last_change":"2024-04-17T07:30:52.971044+0000","last_active":"2024-04-17T07:30:53.971116+0000","last_peered":"2024-04-17T07:30:53.971116+0000","last_clean":"2024-04-17T07:30:53.971116+0000","last_became_active":"2024-04-17T07:30:52.970457+0000","last_became_peered":"2024-04-17T07:30:52.970457+0000","last_unstale":"2024-04-17T07:30:53.971116+0000","last_undegraded":"2024-04-17T07:30:53.971116+0000","last_fullsized":"2024-04-17T07:30:53.971116+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:12.504889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348720+0000","last_change":"2024-04-17T07:30:51.957272+0000","last_active":"2024-04-17T07:30:54.348720+0000","last_peered":"2024-04-17T07:30:54.348720+0000","last_clean":"2024-04-17T07:30:54.348720+0000","last_became_active":"2024-04-17T07:30:51.954523+0000","last_became_peered":"2024-04-17T07:30:51.954523+0000","last_unstale":"2024-04-17T07:30:54.348720+0000","last_undegraded":"2024-04-17T07:30:54.348720+0000","last_fullsized":"2024-04-17T07:30:54.348720+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:28.571235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348430+0000","last_change":"2024-04-17T07:30:52.961446+0000","last_active":"2024-04-17T07:30:54.348430+0000","last_peered":"2024-04-17T07:30:54.348430+0000","last_clean":"2024-04-17T07:30:54.348430+0000","last_became_active":"2024-04-17T07:30:52.958924+0000","last_became_peered":"2024-04-17T07:30:52.958924+0000","last_unstale":"2024-04-17T07:30:54.348430+0000","last_undegraded":"2024-04-17T07:30:54.348430+0000","last_fullsized":"2024-04-17T07:30:54.348430+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:30.187374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348040+0000","last_change":"2024-04-17T07:30:51.953890+0000","last_active":"2024-04-17T07:30:54.348040+0000","last_peered":"2024-04-17T07:30:54.348040+0000","last_clean":"2024-04-17T07:30:54.348040+0000","last_became_active":"2024-04-17T07:30:51.952980+0000","last_became_peered":"2024-04-17T07:30:51.952980+0000","last_unstale":"2024-04-17T07:30:54.348040+0000","last_undegraded":"2024-04-17T07:30:54.348040+0000","last_fullsized":"2024-04-17T07:30:54.348040+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:15:24.280399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960151+0000","last_change":"2024-04-17T07:30:52.979175+0000","last_active":"2024-04-17T07:30:53.960151+0000","last_peered":"2024-04-17T07:30:53.960151+0000","last_clean":"2024-04-17T07:30:53.960151+0000","last_became_active":"2024-04-17T07:30:52.978799+0000","last_became_peered":"2024-04-17T07:30:52.978799+0000","last_unstale":"2024-04-17T07:30:53.960151+0000","last_undegraded":"2024-04-17T07:30:53.960151+0000","last_fullsized":"2024-04-17T07:30:53.960151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:33.550564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975541+0000","last_change":"2024-04-17T07:30:51.952630+0000","last_active":"2024-04-17T07:30:53.975541+0000","last_peered":"2024-04-17T07:30:53.975541+0000","last_clean":"2024-04-17T07:30:53.975541+0000","last_became_active":"2024-04-17T07:30:51.951413+0000","last_became_peered":"2024-04-17T07:30:51.951413+0000","last_unstale":"2024-04-17T07:30:53.975541+0000","last_undegraded":"2024-04-17T07:30:53.975541+0000","last_fullsized":"2024-04-17T07:30:53.975541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:52.513196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347371+0000","last_change":"2024-04-17T07:30:52.961176+0000","last_active":"2024-04-17T07:30:54.347371+0000","last_peered":"2024-04-17T07:30:54.347371+0000","last_clean":"2024-04-17T07:30:54.347371+0000","last_became_active":"2024-04-17T07:30:52.960901+0000","last_became_peered":"2024-04-17T07:30:52.960901+0000","last_unstale":"2024-04-17T07:30:54.347371+0000","last_undegraded":"2024-04-17T07:30:54.347371+0000","last_fullsized":"2024-04-17T07:30:54.347371+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.846951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970566+0000","last_change":"2024-04-17T07:30:51.958218+0000","last_active":"2024-04-17T07:30:53.970566+0000","last_peered":"2024-04-17T07:30:53.970566+0000","last_clean":"2024-04-17T07:30:53.970566+0000","last_became_active":"2024-04-17T07:30:51.957924+0000","last_became_peered":"2024-04-17T07:30:51.957924+0000","last_unstale":"2024-04-17T07:30:53.970566+0000","last_undegraded":"2024-04-17T07:30:53.970566+0000","last_fullsized":"2024-04-17T07:30:53.970566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:14:13.632220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348609+0000","last_change":"2024-04-17T07:30:52.973407+0000","last_active":"2024-04-17T07:30:54.348609+0000","last_peered":"2024-04-17T07:30:54.348609+0000","last_clean":"2024-04-17T07:30:54.348609+0000","last_became_active":"2024-04-17T07:30:52.973087+0000","last_became_peered":"2024-04-17T07:30:52.973087+0000","last_unstale":"2024-04-17T07:30:54.348609+0000","last_undegraded":"2024-04-17T07:30:54.348609+0000","last_fullsized":"2024-04-17T07:30:54.348609+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.108657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.352136+0000","last_change":"2024-04-17T07:30:51.964718+0000","last_active":"2024-04-17T07:30:54.352136+0000","last_peered":"2024-04-17T07:30:54.352136+0000","last_clean":"2024-04-17T07:30:54.352136+0000","last_became_active":"2024-04-17T07:30:51.964349+0000","last_became_peered":"2024-04-17T07:30:51.964349+0000","last_unstale":"2024-04-17T07:30:54.352136+0000","last_undegraded":"2024-04-17T07:30:54.352136+0000","last_fullsized":"2024-04-17T07:30:54.352136+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:53.647342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961614+0000","last_change":"2024-04-17T07:30:52.965649+0000","last_active":"2024-04-17T07:30:53.961614+0000","last_peered":"2024-04-17T07:30:53.961614+0000","last_clean":"2024-04-17T07:30:53.961614+0000","last_became_active":"2024-04-17T07:30:52.965374+0000","last_became_peered":"2024-04-17T07:30:52.965374+0000","last_unstale":"2024-04-17T07:30:53.961614+0000","last_undegraded":"2024-04-17T07:30:53.961614+0000","last_fullsized":"2024-04-17T07:30:53.961614+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:51:55.645023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351007+0000","last_change":"2024-04-17T07:30:51.961616+0000","last_active":"2024-04-17T07:30:54.351007+0000","last_peered":"2024-04-17T07:30:54.351007+0000","last_clean":"2024-04-17T07:30:54.351007+0000","last_became_active":"2024-04-17T07:30:51.961322+0000","last_became_peered":"2024-04-17T07:30:51.961322+0000","last_unstale":"2024-04-17T07:30:54.351007+0000","last_undegraded":"2024-04-17T07:30:54.351007+0000","last_fullsized":"2024-04-17T07:30:54.351007+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:30.526673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975054+0000","last_change":"2024-04-17T07:30:52.960080+0000","last_active":"2024-04-17T07:30:53.975054+0000","last_peered":"2024-04-17T07:30:53.975054+0000","last_clean":"2024-04-17T07:30:53.975054+0000","last_became_active":"2024-04-17T07:30:52.958896+0000","last_became_peered":"2024-04-17T07:30:52.958896+0000","last_unstale":"2024-04-17T07:30:53.975054+0000","last_undegraded":"2024-04-17T07:30:53.975054+0000","last_fullsized":"2024-04-17T07:30:53.975054+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:55.835840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"110'3","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350493+0000","last_change":"2024-04-17T07:30:51.962920+0000","last_active":"2024-04-17T07:30:54.350493+0000","last_peered":"2024-04-17T07:30:54.350493+0000","last_clean":"2024-04-17T07:30:54.350493+0000","last_became_active":"2024-04-17T07:30:51.962642+0000","last_became_peered":"2024-04-17T07:30:51.962642+0000","last_unstale":"2024-04-17T07:30:54.350493+0000","last_undegraded":"2024-04-17T07:30:54.350493+0000","last_fullsized":"2024-04-17T07:30:54.350493+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:06.226285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349052+0000","last_change":"2024-04-17T07:30:52.972717+0000","last_active":"2024-04-17T07:30:54.349052+0000","last_peered":"2024-04-17T07:30:54.349052+0000","last_clean":"2024-04-17T07:30:54.349052+0000","last_became_active":"2024-04-17T07:30:52.972202+0000","last_became_peered":"2024-04-17T07:30:52.972202+0000","last_unstale":"2024-04-17T07:30:54.349052+0000","last_undegraded":"2024-04-17T07:30:54.349052+0000","last_fullsized":"2024-04-17T07:30:54.349052+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:37.546019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"110'6","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962084+0000","last_change":"2024-04-17T07:30:51.948347+0000","last_active":"2024-04-17T07:30:53.962084+0000","last_peered":"2024-04-17T07:30:53.962084+0000","last_clean":"2024-04-17T07:30:53.962084+0000","last_became_active":"2024-04-17T07:30:51.947001+0000","last_became_peered":"2024-04-17T07:30:51.947001+0000","last_unstale":"2024-04-17T07:30:53.962084+0000","last_undegraded":"2024-04-17T07:30:53.962084+0000","last_fullsized":"2024-04-17T07:30:53.962084+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:13:48.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975103+0000","last_change":"2024-04-17T07:30:52.969081+0000","last_active":"2024-04-17T07:30:53.975103+0000","last_peered":"2024-04-17T07:30:53.975103+0000","last_clean":"2024-04-17T07:30:53.975103+0000","last_became_active":"2024-04-17T07:30:52.968804+0000","last_became_peered":"2024-04-17T07:30:52.968804+0000","last_unstale":"2024-04-17T07:30:53.975103+0000","last_undegraded":"2024-04-17T07:30:53.975103+0000","last_fullsized":"2024-04-17T07:30:53.975103+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:37.913622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970967+0000","last_change":"2024-04-17T07:30:51.961385+0000","last_active":"2024-04-17T07:30:53.970967+0000","last_peered":"2024-04-17T07:30:53.970967+0000","last_clean":"2024-04-17T07:30:53.970967+0000","last_became_active":"2024-04-17T07:30:51.961094+0000","last_became_peered":"2024-04-17T07:30:51.961094+0000","last_unstale":"2024-04-17T07:30:53.970967+0000","last_undegraded":"2024-04-17T07:30:53.970967+0000","last_fullsized":"2024-04-17T07:30:53.970967+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:05.969430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962534+0000","last_change":"2024-04-17T07:30:52.978270+0000","last_active":"2024-04-17T07:30:53.962534+0000","last_peered":"2024-04-17T07:30:53.962534+0000","last_clean":"2024-04-17T07:30:53.962534+0000","last_became_active":"2024-04-17T07:30:52.977962+0000","last_became_peered":"2024-04-17T07:30:52.977962+0000","last_unstale":"2024-04-17T07:30:53.962534+0000","last_undegraded":"2024-04-17T07:30:53.962534+0000","last_fullsized":"2024-04-17T07:30:53.962534+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:08.754125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347882+0000","last_change":"2024-04-17T07:30:51.949376+0000","last_active":"2024-04-17T07:30:54.347882+0000","last_peered":"2024-04-17T07:30:54.347882+0000","last_clean":"2024-04-17T07:30:54.347882+0000","last_became_active":"2024-04-17T07:30:51.949024+0000","last_became_peered":"2024-04-17T07:30:51.949024+0000","last_unstale":"2024-04-17T07:30:54.347882+0000","last_undegraded":"2024-04-17T07:30:54.347882+0000","last_fullsized":"2024-04-17T07:30:54.347882+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:59:10.386075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970498+0000","last_change":"2024-04-17T07:30:52.967073+0000","last_active":"2024-04-17T07:30:53.970498+0000","last_peered":"2024-04-17T07:30:53.970498+0000","last_clean":"2024-04-17T07:30:53.970498+0000","last_became_active":"2024-04-17T07:30:52.966787+0000","last_became_peered":"2024-04-17T07:30:52.966787+0000","last_unstale":"2024-04-17T07:30:53.970498+0000","last_undegraded":"2024-04-17T07:30:53.970498+0000","last_fullsized":"2024-04-17T07:30:53.970498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:12.775821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960658+0000","last_change":"2024-04-17T07:30:51.959594+0000","last_active":"2024-04-17T07:30:53.960658+0000","last_peered":"2024-04-17T07:30:53.960658+0000","last_clean":"2024-04-17T07:30:53.960658+0000","last_became_active":"2024-04-17T07:30:51.959280+0000","last_became_peered":"2024-04-17T07:30:51.959280+0000","last_unstale":"2024-04-17T07:30:53.960658+0000","last_undegraded":"2024-04-17T07:30:53.960658+0000","last_fullsized":"2024-04-17T07:30:53.960658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:04.068840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.970437+0000","last_change":"2024-04-17T07:30:52.966883+0000","last_active":"2024-04-17T07:30:53.970437+0000","last_peered":"2024-04-17T07:30:53.970437+0000","last_clean":"2024-04-17T07:30:53.970437+0000","last_became_active":"2024-04-17T07:30:52.966572+0000","last_became_peered":"2024-04-17T07:30:52.966572+0000","last_unstale":"2024-04-17T07:30:53.970437+0000","last_undegraded":"2024-04-17T07:30:53.970437+0000","last_fullsized":"2024-04-17T07:30:53.970437+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:16:46.840383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"110'8","reported_seq":22,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.974754+0000","last_change":"2024-04-17T07:30:51.960788+0000","last_active":"2024-04-17T07:30:53.974754+0000","last_peered":"2024-04-17T07:30:53.974754+0000","last_clean":"2024-04-17T07:30:53.974754+0000","last_became_active":"2024-04-17T07:30:51.960435+0000","last_became_peered":"2024-04-17T07:30:51.960435+0000","last_unstale":"2024-04-17T07:30:53.974754+0000","last_undegraded":"2024-04-17T07:30:53.974754+0000","last_fullsized":"2024-04-17T07:30:53.974754+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:31.671927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962695+0000","last_change":"2024-04-17T07:30:52.966356+0000","last_active":"2024-04-17T07:30:53.962695+0000","last_peered":"2024-04-17T07:30:53.962695+0000","last_clean":"2024-04-17T07:30:53.962695+0000","last_became_active":"2024-04-17T07:30:52.964607+0000","last_became_peered":"2024-04-17T07:30:52.964607+0000","last_unstale":"2024-04-17T07:30:53.962695+0000","last_undegraded":"2024-04-17T07:30:53.962695+0000","last_fullsized":"2024-04-17T07:30:53.962695+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:57:24.915493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.352301+0000","last_change":"2024-04-17T07:30:51.960793+0000","last_active":"2024-04-17T07:30:54.352301+0000","last_peered":"2024-04-17T07:30:54.352301+0000","last_clean":"2024-04-17T07:30:54.352301+0000","last_became_active":"2024-04-17T07:30:51.960106+0000","last_became_peered":"2024-04-17T07:30:51.960106+0000","last_unstale":"2024-04-17T07:30:54.352301+0000","last_undegraded":"2024-04-17T07:30:54.352301+0000","last_fullsized":"2024-04-17T07:30:54.352301+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:14.130056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351752+0000","last_change":"2024-04-17T07:30:52.954834+0000","last_active":"2024-04-17T07:30:54.351752+0000","last_peered":"2024-04-17T07:30:54.351752+0000","last_clean":"2024-04-17T07:30:54.351752+0000","last_became_active":"2024-04-17T07:30:52.954281+0000","last_became_peered":"2024-04-17T07:30:52.954281+0000","last_unstale":"2024-04-17T07:30:54.351752+0000","last_undegraded":"2024-04-17T07:30:54.351752+0000","last_fullsized":"2024-04-17T07:30:54.351752+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:06.794769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975523+0000","last_change":"2024-04-17T07:30:51.964058+0000","last_active":"2024-04-17T07:30:53.975523+0000","last_peered":"2024-04-17T07:30:53.975523+0000","last_clean":"2024-04-17T07:30:53.975523+0000","last_became_active":"2024-04-17T07:30:51.963735+0000","last_became_peered":"2024-04-17T07:30:51.963735+0000","last_unstale":"2024-04-17T07:30:53.975523+0000","last_undegraded":"2024-04-17T07:30:53.975523+0000","last_fullsized":"2024-04-17T07:30:53.975523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:35.278335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349511+0000","last_change":"2024-04-17T07:30:52.967661+0000","last_active":"2024-04-17T07:30:54.349511+0000","last_peered":"2024-04-17T07:30:54.349511+0000","last_clean":"2024-04-17T07:30:54.349511+0000","last_became_active":"2024-04-17T07:30:52.967377+0000","last_became_peered":"2024-04-17T07:30:52.967377+0000","last_unstale":"2024-04-17T07:30:54.349511+0000","last_undegraded":"2024-04-17T07:30:54.349511+0000","last_fullsized":"2024-04-17T07:30:54.349511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:30.663701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975173+0000","last_change":"2024-04-17T07:30:51.958304+0000","last_active":"2024-04-17T07:30:53.975173+0000","last_peered":"2024-04-17T07:30:53.975173+0000","last_clean":"2024-04-17T07:30:53.975173+0000","last_became_active":"2024-04-17T07:30:51.958030+0000","last_became_peered":"2024-04-17T07:30:51.958030+0000","last_unstale":"2024-04-17T07:30:53.975173+0000","last_undegraded":"2024-04-17T07:30:53.975173+0000","last_fullsized":"2024-04-17T07:30:53.975173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:03.425591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956633+0000","last_change":"2024-04-17T07:30:52.976683+0000","last_active":"2024-04-17T07:30:53.956633+0000","last_peered":"2024-04-17T07:30:53.956633+0000","last_clean":"2024-04-17T07:30:53.956633+0000","last_became_active":"2024-04-17T07:30:52.975990+0000","last_became_peered":"2024-04-17T07:30:52.975990+0000","last_unstale":"2024-04-17T07:30:53.956633+0000","last_undegraded":"2024-04-17T07:30:53.956633+0000","last_fullsized":"2024-04-17T07:30:53.956633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:08.067694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348204+0000","last_change":"2024-04-17T07:30:51.950098+0000","last_active":"2024-04-17T07:30:54.348204+0000","last_peered":"2024-04-17T07:30:54.348204+0000","last_clean":"2024-04-17T07:30:54.348204+0000","last_became_active":"2024-04-17T07:30:51.947543+0000","last_became_peered":"2024-04-17T07:30:51.947543+0000","last_unstale":"2024-04-17T07:30:54.348204+0000","last_undegraded":"2024-04-17T07:30:54.348204+0000","last_fullsized":"2024-04-17T07:30:54.348204+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:16.806530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960975+0000","last_change":"2024-04-17T07:30:52.960137+0000","last_active":"2024-04-17T07:30:53.960975+0000","last_peered":"2024-04-17T07:30:53.960975+0000","last_clean":"2024-04-17T07:30:53.960975+0000","last_became_active":"2024-04-17T07:30:52.959821+0000","last_became_peered":"2024-04-17T07:30:52.959821+0000","last_unstale":"2024-04-17T07:30:53.960975+0000","last_undegraded":"2024-04-17T07:30:53.960975+0000","last_fullsized":"2024-04-17T07:30:53.960975+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:39.534083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956662+0000","last_change":"2024-04-17T07:30:51.942316+0000","last_active":"2024-04-17T07:30:53.956662+0000","last_peered":"2024-04-17T07:30:53.956662+0000","last_clean":"2024-04-17T07:30:53.956662+0000","last_became_active":"2024-04-17T07:30:51.940838+0000","last_became_peered":"2024-04-17T07:30:51.940838+0000","last_unstale":"2024-04-17T07:30:53.956662+0000","last_undegraded":"2024-04-17T07:30:53.956662+0000","last_fullsized":"2024-04-17T07:30:53.956662+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:49:07.870223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956088+0000","last_change":"2024-04-17T07:30:52.964136+0000","last_active":"2024-04-17T07:30:53.956088+0000","last_peered":"2024-04-17T07:30:53.956088+0000","last_clean":"2024-04-17T07:30:53.956088+0000","last_became_active":"2024-04-17T07:30:52.957533+0000","last_became_peered":"2024-04-17T07:30:52.957533+0000","last_unstale":"2024-04-17T07:30:53.956088+0000","last_undegraded":"2024-04-17T07:30:53.956088+0000","last_fullsized":"2024-04-17T07:30:53.956088+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:51.942966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.956271+0000","last_change":"2024-04-17T07:30:51.961102+0000","last_active":"2024-04-17T07:30:53.956271+0000","last_peered":"2024-04-17T07:30:53.956271+0000","last_clean":"2024-04-17T07:30:53.956271+0000","last_became_active":"2024-04-17T07:30:51.960823+0000","last_became_peered":"2024-04-17T07:30:51.960823+0000","last_unstale":"2024-04-17T07:30:53.956271+0000","last_undegraded":"2024-04-17T07:30:53.956271+0000","last_fullsized":"2024-04-17T07:30:53.956271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:52.513196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961297+0000","last_change":"2024-04-17T07:30:52.973772+0000","last_active":"2024-04-17T07:30:53.961297+0000","last_peered":"2024-04-17T07:30:53.961297+0000","last_clean":"2024-04-17T07:30:53.961297+0000","last_became_active":"2024-04-17T07:30:52.973498+0000","last_became_peered":"2024-04-17T07:30:52.973498+0000","last_unstale":"2024-04-17T07:30:53.961297+0000","last_undegraded":"2024-04-17T07:30:53.961297+0000","last_fullsized":"2024-04-17T07:30:53.961297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:17.951301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347388+0000","last_change":"2024-04-17T07:30:51.949336+0000","last_active":"2024-04-17T07:30:54.347388+0000","last_peered":"2024-04-17T07:30:54.347388+0000","last_clean":"2024-04-17T07:30:54.347388+0000","last_became_active":"2024-04-17T07:30:51.948756+0000","last_became_peered":"2024-04-17T07:30:51.948756+0000","last_unstale":"2024-04-17T07:30:54.347388+0000","last_undegraded":"2024-04-17T07:30:54.347388+0000","last_fullsized":"2024-04-17T07:30:54.347388+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:09:31.157119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349536+0000","last_change":"2024-04-17T07:30:52.953516+0000","last_active":"2024-04-17T07:30:54.349536+0000","last_peered":"2024-04-17T07:30:54.349536+0000","last_clean":"2024-04-17T07:30:54.349536+0000","last_became_active":"2024-04-17T07:30:52.953127+0000","last_became_peered":"2024-04-17T07:30:52.953127+0000","last_unstale":"2024-04-17T07:30:54.349536+0000","last_undegraded":"2024-04-17T07:30:54.349536+0000","last_fullsized":"2024-04-17T07:30:54.349536+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:47.405703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971558+0000","last_change":"2024-04-17T07:30:51.955313+0000","last_active":"2024-04-17T07:30:53.971558+0000","last_peered":"2024-04-17T07:30:53.971558+0000","last_clean":"2024-04-17T07:30:53.971558+0000","last_became_active":"2024-04-17T07:30:51.954895+0000","last_became_peered":"2024-04-17T07:30:51.954895+0000","last_unstale":"2024-04-17T07:30:53.971558+0000","last_undegraded":"2024-04-17T07:30:53.971558+0000","last_fullsized":"2024-04-17T07:30:53.971558+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:04.272085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.962769+0000","last_change":"2024-04-17T07:30:52.972154+0000","last_active":"2024-04-17T07:30:53.962769+0000","last_peered":"2024-04-17T07:30:53.962769+0000","last_clean":"2024-04-17T07:30:53.962769+0000","last_became_active":"2024-04-17T07:30:52.971886+0000","last_became_peered":"2024-04-17T07:30:52.971886+0000","last_unstale":"2024-04-17T07:30:53.962769+0000","last_undegraded":"2024-04-17T07:30:53.962769+0000","last_fullsized":"2024-04-17T07:30:53.962769+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:07.228548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"111'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.973623+0000","last_change":"2024-04-17T07:30:51.953140+0000","last_active":"2024-04-17T07:30:53.973623+0000","last_peered":"2024-04-17T07:30:53.973623+0000","last_clean":"2024-04-17T07:30:53.973623+0000","last_became_active":"2024-04-17T07:30:51.952808+0000","last_became_peered":"2024-04-17T07:30:51.952808+0000","last_unstale":"2024-04-17T07:30:53.973623+0000","last_undegraded":"2024-04-17T07:30:53.973623+0000","last_fullsized":"2024-04-17T07:30:53.973623+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:09:35.415227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351069+0000","last_change":"2024-04-17T07:30:52.959706+0000","last_active":"2024-04-17T07:30:54.351069+0000","last_peered":"2024-04-17T07:30:54.351069+0000","last_clean":"2024-04-17T07:30:54.351069+0000","last_became_active":"2024-04-17T07:30:52.959401+0000","last_became_peered":"2024-04-17T07:30:52.959401+0000","last_unstale":"2024-04-17T07:30:54.351069+0000","last_undegraded":"2024-04-17T07:30:54.351069+0000","last_fullsized":"2024-04-17T07:30:54.351069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:21.601807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350735+0000","last_change":"2024-04-17T07:30:51.960925+0000","last_active":"2024-04-17T07:30:54.350735+0000","last_peered":"2024-04-17T07:30:54.350735+0000","last_clean":"2024-04-17T07:30:54.350735+0000","last_became_active":"2024-04-17T07:30:51.960619+0000","last_became_peered":"2024-04-17T07:30:51.960619+0000","last_unstale":"2024-04-17T07:30:54.350735+0000","last_undegraded":"2024-04-17T07:30:54.350735+0000","last_fullsized":"2024-04-17T07:30:54.350735+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:32.948486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349621+0000","last_change":"2024-04-17T07:30:52.971908+0000","last_active":"2024-04-17T07:30:54.349621+0000","last_peered":"2024-04-17T07:30:54.349621+0000","last_clean":"2024-04-17T07:30:54.349621+0000","last_became_active":"2024-04-17T07:30:52.970476+0000","last_became_peered":"2024-04-17T07:30:52.970476+0000","last_unstale":"2024-04-17T07:30:54.349621+0000","last_undegraded":"2024-04-17T07:30:54.349621+0000","last_fullsized":"2024-04-17T07:30:54.349621+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:57.401250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.351061+0000","last_change":"2024-04-17T07:30:51.963126+0000","last_active":"2024-04-17T07:30:54.351061+0000","last_peered":"2024-04-17T07:30:54.351061+0000","last_clean":"2024-04-17T07:30:54.351061+0000","last_became_active":"2024-04-17T07:30:51.959481+0000","last_became_peered":"2024-04-17T07:30:51.959481+0000","last_unstale":"2024-04-17T07:30:54.351061+0000","last_undegraded":"2024-04-17T07:30:54.351061+0000","last_fullsized":"2024-04-17T07:30:54.351061+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:25.004071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.961883+0000","last_change":"2024-04-17T07:30:52.958100+0000","last_active":"2024-04-17T07:30:53.961883+0000","last_peered":"2024-04-17T07:30:53.961883+0000","last_clean":"2024-04-17T07:30:53.961883+0000","last_became_active":"2024-04-17T07:30:52.957780+0000","last_became_peered":"2024-04-17T07:30:52.957780+0000","last_unstale":"2024-04-17T07:30:53.961883+0000","last_undegraded":"2024-04-17T07:30:53.961883+0000","last_fullsized":"2024-04-17T07:30:53.961883+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:00:13.982768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975453+0000","last_change":"2024-04-17T07:30:51.962209+0000","last_active":"2024-04-17T07:30:53.975453+0000","last_peered":"2024-04-17T07:30:53.975453+0000","last_clean":"2024-04-17T07:30:53.975453+0000","last_became_active":"2024-04-17T07:30:51.961902+0000","last_became_peered":"2024-04-17T07:30:51.961902+0000","last_unstale":"2024-04-17T07:30:53.975453+0000","last_undegraded":"2024-04-17T07:30:53.975453+0000","last_fullsized":"2024-04-17T07:30:53.975453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:05.514327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.975286+0000","last_change":"2024-04-17T07:30:52.959950+0000","last_active":"2024-04-17T07:30:53.975286+0000","last_peered":"2024-04-17T07:30:53.975286+0000","last_clean":"2024-04-17T07:30:53.975286+0000","last_became_active":"2024-04-17T07:30:52.958965+0000","last_became_peered":"2024-04-17T07:30:52.958965+0000","last_unstale":"2024-04-17T07:30:53.975286+0000","last_undegraded":"2024-04-17T07:30:53.975286+0000","last_fullsized":"2024-04-17T07:30:53.975286+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:42.419341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.348012+0000","last_change":"2024-04-17T07:30:51.955420+0000","last_active":"2024-04-17T07:30:54.348012+0000","last_peered":"2024-04-17T07:30:54.348012+0000","last_clean":"2024-04-17T07:30:54.348012+0000","last_became_active":"2024-04-17T07:30:51.955099+0000","last_became_peered":"2024-04-17T07:30:51.955099+0000","last_unstale":"2024-04-17T07:30:54.348012+0000","last_undegraded":"2024-04-17T07:30:54.348012+0000","last_fullsized":"2024-04-17T07:30:54.348012+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:05.683136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960268+0000","last_change":"2024-04-17T07:30:52.976237+0000","last_active":"2024-04-17T07:30:53.960268+0000","last_peered":"2024-04-17T07:30:53.960268+0000","last_clean":"2024-04-17T07:30:53.960268+0000","last_became_active":"2024-04-17T07:30:52.975910+0000","last_became_peered":"2024-04-17T07:30:52.975910+0000","last_unstale":"2024-04-17T07:30:53.960268+0000","last_undegraded":"2024-04-17T07:30:53.960268+0000","last_fullsized":"2024-04-17T07:30:53.960268+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:19.108657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.350145+0000","last_change":"2024-04-17T07:30:51.965391+0000","last_active":"2024-04-17T07:30:54.350145+0000","last_peered":"2024-04-17T07:30:54.350145+0000","last_clean":"2024-04-17T07:30:54.350145+0000","last_became_active":"2024-04-17T07:30:51.964945+0000","last_became_peered":"2024-04-17T07:30:51.964945+0000","last_unstale":"2024-04-17T07:30:54.350145+0000","last_undegraded":"2024-04-17T07:30:54.350145+0000","last_fullsized":"2024-04-17T07:30:54.350145+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:30.790943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.349867+0000","last_change":"2024-04-17T07:30:52.958641+0000","last_active":"2024-04-17T07:30:54.349867+0000","last_peered":"2024-04-17T07:30:54.349867+0000","last_clean":"2024-04-17T07:30:54.349867+0000","last_became_active":"2024-04-17T07:30:52.958307+0000","last_became_peered":"2024-04-17T07:30:52.958307+0000","last_unstale":"2024-04-17T07:30:54.349867+0000","last_undegraded":"2024-04-17T07:30:54.349867+0000","last_fullsized":"2024-04-17T07:30:54.349867+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:10.602698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:54.347983+0000","last_change":"2024-04-17T07:30:51.962718+0000","last_active":"2024-04-17T07:30:54.347983+0000","last_peered":"2024-04-17T07:30:54.347983+0000","last_clean":"2024-04-17T07:30:54.347983+0000","last_became_active":"2024-04-17T07:30:51.961694+0000","last_became_peered":"2024-04-17T07:30:51.961694+0000","last_unstale":"2024-04-17T07:30:54.347983+0000","last_undegraded":"2024-04-17T07:30:54.347983+0000","last_fullsized":"2024-04-17T07:30:54.347983+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:46.032006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.971492+0000","last_change":"2024-04-17T07:30:52.969957+0000","last_active":"2024-04-17T07:30:53.971492+0000","last_peered":"2024-04-17T07:30:53.971492+0000","last_clean":"2024-04-17T07:30:53.971492+0000","last_became_active":"2024-04-17T07:30:52.969578+0000","last_became_peered":"2024-04-17T07:30:52.969578+0000","last_unstale":"2024-04-17T07:30:53.971492+0000","last_undegraded":"2024-04-17T07:30:53.971492+0000","last_fullsized":"2024-04-17T07:30:53.971492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:51.907089+0000","last_clean_scrub_stamp":"2024-04-17T07:30:51.907089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:22:41.631018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.963050+0000","last_change":"2024-04-17T07:30:51.966520+0000","last_active":"2024-04-17T07:30:53.963050+0000","last_peered":"2024-04-17T07:30:53.963050+0000","last_clean":"2024-04-17T07:30:53.963050+0000","last_became_active":"2024-04-17T07:30:51.966147+0000","last_became_peered":"2024-04-17T07:30:51.966147+0000","last_unstale":"2024-04-17T07:30:53.963050+0000","last_undegraded":"2024-04-17T07:30:53.963050+0000","last_fullsized":"2024-04-17T07:30:53.963050+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:30:50.890467+0000","last_clean_scrub_stamp":"2024-04-17T07:30:50.890467+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:46.555431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"100'189","reported_seq":600,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-17T07:30:53.960413+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:30:53.960413+0000","last_peered":"2024-04-17T07:30:53.960413+0000","last_clean":"2024-04-17T07:30:53.960413+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:30:53.960413+0000","last_undegraded":"2024-04-17T07:30:53.960413+0000","last_fullsized":"2024-04-17T07:30:53.960413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":287,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":738,"num_read_kb":1500,"num_write":287,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542348,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37260,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334580,"statfs":{"total":96636764160,"available":96598609920,"internally_reserved":0,"allocated":1957888,"data_stored":1238505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542348,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":648761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542347,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1376256,"data_stored":649701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542347,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58880,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312960,"statfs":{"total":96636764160,"available":96576471040,"internally_reserved":0,"allocated":1359872,"data_stored":648067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542349,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36688,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335152,"statfs":{"total":96636764160,"available":96599195648,"internally_reserved":0,"allocated":1372160,"data_stored":649679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542348,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1363968,"data_stored":648649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542349,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54736,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317104,"statfs":{"total":96636764160,"available":96580714496,"internally_reserved":0,"allocated":1372160,"data_stored":648261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542347,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":1238469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:30:58.041 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:30:58.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:30:58.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.318+0000 7fbbe7200640 1 Processor -- start 2024-04-17T07:30:58.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.318+0000 7fbbe7200640 1 -- start start 2024-04-17T07:30:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.320+0000 7fbbe7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.320+0000 7fbbe7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbbe80cd5d0 0x7fbbe80cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.320+0000 7fbbe7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe815cbe0 con 0x7fbbe80b20b0 2024-04-17T07:30:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.320+0000 7fbbe7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbbe815cea0 con 0x7fbbe80cd5d0 2024-04-17T07:30:58.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.320+0000 7fbbe7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe815d180 con 0x7fbbe813f6d0 2024-04-17T07:30:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.321+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.321+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:45980/0 (socket says 172.21.15.67:45980) 2024-04-17T07:30:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.321+0000 7fbbe5e00640 1 -- 172.21.15.67:0/2803434740 learned_addr learned my addr 172.21.15.67:0/2803434740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.321+0000 7fbbe5400640 1 --2- 172.21.15.67:0/2803434740 >> v2:172.21.15.187:3300/0 conn(0x7fbbe80cd5d0 0x7fbbe80cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.322+0000 7fbbe5e00640 1 -- 172.21.15.67:0/2803434740 >> v2:172.21.15.187:3300/0 conn(0x7fbbe80cd5d0 msgr2=0x7fbbe80cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:58.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.322+0000 7fbbe5e00640 1 --2- 172.21.15.67:0/2803434740 >> v2:172.21.15.187:3300/0 conn(0x7fbbe80cd5d0 0x7fbbe80cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.322+0000 7fbbe5e00640 1 -- 172.21.15.67:0/2803434740 >> v1:172.21.15.187:6789/0 conn(0x7fbbe80b20b0 legacy=0x7fbbe8142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.322+0000 7fbbe5e00640 1 -- 172.21.15.67:0/2803434740 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe815d460 con 0x7fbbe813f6d0 2024-04-17T07:30:58.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.324+0000 7fbbe5e00640 1 --2- 172.21.15.67:0/2803434740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8009870 tx=0x7fbbd802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600450f676aef3d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.324+0000 7fbbe4a00640 1 -- 172.21.15.67:0/2803434740 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8035020 con 0x7fbbe813f6d0 2024-04-17T07:30:58.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe4a00640 1 -- 172.21.15.67:0/2803434740 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd8031640 con 0x7fbbe813f6d0 2024-04-17T07:30:58.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe4a00640 1 -- 172.21.15.67:0/2803434740 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd802d040 con 0x7fbbe813f6d0 2024-04-17T07:30:58.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 -- 172.21.15.67:0/2803434740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 msgr2=0x7fbbe8071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 --2- 172.21.15.67:0/2803434740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8009870 tx=0x7fbbd802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 -- 172.21.15.67:0/2803434740 shutdown_connections 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 --2- 172.21.15.67:0/2803434740 >> v2:172.21.15.187:3300/0 conn(0x7fbbe80cd5d0 0x7fbbe80cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 --2- 172.21.15.67:0/2803434740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe813f6d0 0x7fbbe8071d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.325+0000 7fbbe7200640 1 -- 172.21.15.67:0/2803434740 >> 172.21.15.67:0/2803434740 conn(0x7fbbe8067d10 msgr2=0x7fbbe80d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.326+0000 7fbbe7200640 1 -- 172.21.15.67:0/2803434740 shutdown_connections 2024-04-17T07:30:58.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.326+0000 7fbbe7200640 1 -- 172.21.15.67:0/2803434740 wait complete. 2024-04-17T07:30:58.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.328+0000 7fbbe7200640 1 Processor -- start 2024-04-17T07:30:58.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.328+0000 7fbbe7200640 1 -- start start 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe80cd5d0 0x7fbbe81f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbe813f6d0 0x7fbbe8153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe815da20 con 0x7fbbe80b20b0 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe80d1120 con 0x7fbbe813f6d0 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe80b2540 con 0x7fbbe80cd5d0 2024-04-17T07:30:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51764/0 (socket says 172.21.15.67:51764) 2024-04-17T07:30:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 -- 172.21.15.67:0/1366592825 learned_addr learned my addr 172.21.15.67:0/1366592825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 -- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe80cd5d0 msgr2=0x7fbbe81f5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe80cd5d0 0x7fbbe81f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 -- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbe813f6d0 msgr2=0x7fbbe8153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbe813f6d0 0x7fbbe8153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.330+0000 7fbbe5e00640 1 -- 172.21.15.67:0/1366592825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe81f5550 con 0x7fbbe80b20b0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbe5e00640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000c00 tx=0x7fbbd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8040070 con 0x7fbbe80b20b0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd80399f0 con 0x7fbbe80b20b0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8035040 con 0x7fbbe80b20b0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe8141280 con 0x7fbbe80b20b0 2024-04-17T07:30:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.331+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe8156590 con 0x7fbbe80b20b0 2024-04-17T07:30:58.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.332+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe8068a10 con 0x7fbbe80b20b0 2024-04-17T07:30:58.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.334+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fbbd804b060 con 0x7fbbe80b20b0 2024-04-17T07:30:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.334+0000 7fbbde000640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 0x7fbbc806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.335+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7fbbd80ad580 con 0x7fbbe80b20b0 2024-04-17T07:30:58.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.335+0000 7fbbe5400640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 0x7fbbc806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.336+0000 7fbbe5400640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 0x7fbbc806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4006a90 tx=0x7fbbd4006a20 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.339+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbbe81f5550 con 0x7fbbe80b20b0 2024-04-17T07:30:58.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:30:58.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:30:58.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.621+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbbe813da60 con 0x7fbbe80b20b0 2024-04-17T07:30:58.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.626+0000 7fbbde000640 1 -- 172.21.15.67:0/1366592825 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7fbbd807b890 con 0x7fbbe80b20b0 2024-04-17T07:30:58.625 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:30:58.626 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:30:58.626 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:30:58.626 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:30:58.626 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:30:58.626 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 msgr2=0x7fbbc806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 0x7fbbc806eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4006a90 tx=0x7fbbd4006a20 comp rx=0 tx=0).stop 2024-04-17T07:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 msgr2=0x7fbbe8155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000c00 tx=0x7fbbd80046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 shutdown_connections 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fbbc806ca10 0x7fbbc806eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbe813f6d0 0x7fbbe8153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbbe80cd5d0 0x7fbbe81f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 --2- 172.21.15.67:0/1366592825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80b20b0 0x7fbbe8155bf0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 >> 172.21.15.67:0/1366592825 conn(0x7fbbe8067d10 msgr2=0x7fbbe813df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 shutdown_connections 2024-04-17T07:30:58.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.629+0000 7fbbe7200640 1 -- 172.21.15.67:0/1366592825 wait complete. 2024-04-17T07:30:58.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:30:58.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.867+0000 7f662d000640 1 Processor -- start 2024-04-17T07:30:58.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.867+0000 7f662d000640 1 -- start start 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f662d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f662806b8c0 0x7f66280cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f662d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628160280 0x7f662806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f662d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66280cd4a0 con 0x7f6628141ed0 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f662d000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f662815ce30 con 0x7f662806b8c0 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f662d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66281607f0 con 0x7f6628160280 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f6627400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f662806b8c0 0x7f66280cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f6627400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f662806b8c0 0x7f66280cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51768/0 (socket says 172.21.15.67:51768) 2024-04-17T07:30:58.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f6627e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6628141ed0 0x7f662815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54718/0 (socket says 172.21.15.67:54718) 2024-04-17T07:30:58.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f6627400640 1 -- 172.21.15.67:0/869855479 learned_addr learned my addr 172.21.15.67:0/869855479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.869+0000 7f6627400640 1 -- 172.21.15.67:0/869855479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628160280 msgr2=0x7f662806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.870+0000 7f661fe00640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628160280 0x7f662806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.870+0000 7f6627400640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628160280 0x7f662806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.870+0000 7f6627400640 1 -- 172.21.15.67:0/869855479 >> v1:172.21.15.67:6789/0 conn(0x7f6628141ed0 legacy=0x7f662815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.870+0000 7f6626a00640 1 -- 172.21.15.67:0/869855479 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799206920 0 0) 0x7f66280cd4a0 con 0x7f6628141ed0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.870+0000 7f6627400640 1 -- 172.21.15.67:0/869855479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6628160ad0 con 0x7f662806b8c0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.871+0000 7f6627400640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f662806b8c0 0x7f66280cd8b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6618009870 tx=0x7f661802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62ca3d360bfd0a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.871+0000 7f6626a00640 1 -- 172.21.15.67:0/869855479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618034070 con 0x7f662806b8c0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.871+0000 7f6626a00640 1 -- 172.21.15.67:0/869855479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6618030840 con 0x7f662806b8c0 2024-04-17T07:30:58.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.871+0000 7f6626a00640 1 -- 172.21.15.67:0/869855479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618030c10 con 0x7f662806b8c0 2024-04-17T07:30:58.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 -- 172.21.15.67:0/869855479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f662806b8c0 msgr2=0x7f66280cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f662806b8c0 0x7f66280cd8b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6618009870 tx=0x7f661802b980 comp rx=0 tx=0).stop 2024-04-17T07:30:58.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 -- 172.21.15.67:0/869855479 shutdown_connections 2024-04-17T07:30:58.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628160280 0x7f662806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 --2- 172.21.15.67:0/869855479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f662806b8c0 0x7f66280cd8b0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 -- 172.21.15.67:0/869855479 >> 172.21.15.67:0/869855479 conn(0x7f6628067d10 msgr2=0x7f66280b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 -- 172.21.15.67:0/869855479 shutdown_connections 2024-04-17T07:30:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.872+0000 7f662d000640 1 -- 172.21.15.67:0/869855479 wait complete. 2024-04-17T07:30:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.874+0000 7f662d000640 1 Processor -- start 2024-04-17T07:30:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.874+0000 7f662d000640 1 -- start start 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f662806b8c0 0x7f6628144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628141ed0 0x7f6628155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f662806c1e0 con 0x7f6628160280 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66280cddf0 con 0x7f662806b8c0 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f662d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6628161090 con 0x7f6628141ed0 2024-04-17T07:30:58.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f6627e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f6627e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51784/0 (socket says 172.21.15.67:51784) 2024-04-17T07:30:58.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f6627e00640 1 -- 172.21.15.67:0/3931683919 learned_addr learned my addr 172.21.15.67:0/3931683919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.875+0000 7f6627e00640 1 -- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628141ed0 msgr2=0x7f6628155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627e00640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628141ed0 0x7f6628155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627e00640 1 -- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f662806b8c0 msgr2=0x7f6628144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627400640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f662806b8c0 0x7f6628144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627e00640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f662806b8c0 0x7f6628144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627e00640 1 -- 172.21.15.67:0/3931683919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6628145430 con 0x7f6628160280 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f6627e00640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f662000c810 tx=0x7f662000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620018070 con 0x7f6628160280 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662000a640 con 0x7f6628160280 2024-04-17T07:30:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6628145710 con 0x7f6628160280 2024-04-17T07:30:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662000aa30 con 0x7f6628160280 2024-04-17T07:30:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.876+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6628145cd0 con 0x7f6628160280 2024-04-17T07:30:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.878+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6628068a10 con 0x7f6628160280 2024-04-17T07:30:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.879+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f662001b070 con 0x7f6628160280 2024-04-17T07:30:58.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.880+0000 7f661f400640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 0x7f660406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:58.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.880+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f662008b450 con 0x7f6628160280 2024-04-17T07:30:58.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.880+0000 7f6627400640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 0x7f660406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.881+0000 7f6627400640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 0x7f660406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6618036000 tx=0x7f661803b040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:58.884+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f66200596d0 con 0x7f6628160280 2024-04-17T07:30:59.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.138+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f662815c640 con 0x7f6628160280 2024-04-17T07:30:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.139+0000 7f661f400640 1 -- 172.21.15.67:0/3931683919 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7f6620058fd0 con 0x7f6628160280 2024-04-17T07:30:59.139 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:30:59.139 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:30:59.139 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:30:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 msgr2=0x7f660406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 0x7f660406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6618036000 tx=0x7f661803b040 comp rx=0 tx=0).stop 2024-04-17T07:30:59.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 msgr2=0x7f662815e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f662000c810 tx=0x7f662000ccd0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 shutdown_connections 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f660406ca10 0x7f660406eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6628160280 0x7f662815e310 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6628141ed0 0x7f6628155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.142+0000 7f662d000640 1 --2- 172.21.15.67:0/3931683919 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f662806b8c0 0x7f6628144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.143+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 >> 172.21.15.67:0/3931683919 conn(0x7f6628067d10 msgr2=0x7f662813f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.143+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 shutdown_connections 2024-04-17T07:30:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.143+0000 7f662d000640 1 -- 172.21.15.67:0/3931683919 wait complete. 2024-04-17T07:30:59.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:30:52.419982+0000', 'modified': '2024-04-17T07:30:55.498917+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6232': {'gid': 6232, 'name': 'd', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3534305613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 3534305613}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3534305613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6232, 'qdb_cluster': [6232]} 2024-04-17T07:30:59.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:30:59.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T07:30:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.797+0000 7f49cd400640 1 Processor -- start 2024-04-17T07:30:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.797+0000 7f49cd400640 1 -- start start 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.798+0000 7f49cd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49c815b860 0x7f49c80d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49cd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c80cfcc0 con 0x7f49c80ce4d0 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49cd400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c8071f60 con 0x7f49c80b4f80 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49cd400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f49c8071d10 con 0x7f49c815b860 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49c815b860 0x7f49c80d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49c815b860 0x7f49c80d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51792/0 (socket says 172.21.15.67:51792) 2024-04-17T07:30:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 -- 172.21.15.67:0/671156321 learned_addr learned my addr 172.21.15.67:0/671156321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 -- 172.21.15.67:0/671156321 >> v1:172.21.15.187:6789/0 conn(0x7f49c80b4f80 legacy=0x7f49c80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 -- 172.21.15.67:0/671156321 >> v1:172.21.15.67:6789/0 conn(0x7f49c80ce4d0 legacy=0x7f49c80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.799+0000 7f49c6800640 1 -- 172.21.15.67:0/671156321 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c8072240 con 0x7f49c815b860 2024-04-17T07:30:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.800+0000 7f49c6800640 1 --2- 172.21.15.67:0/671156321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c815b860 0x7f49c80d0420 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f49b800bb10 tx=0x7f49b802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43022f0e7cb75106 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.801+0000 7f49c5400640 1 -- 172.21.15.67:0/671156321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b803b070 con 0x7f49c815b860 2024-04-17T07:30:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.801+0000 7f49c5400640 1 -- 172.21.15.67:0/671156321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b8036440 con 0x7f49c815b860 2024-04-17T07:30:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.801+0000 7f49c5400640 1 -- 172.21.15.67:0/671156321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8036810 con 0x7f49c815b860 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.801+0000 7f49cd400640 1 -- 172.21.15.67:0/671156321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c815b860 msgr2=0x7f49c80d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.801+0000 7f49cd400640 1 --2- 172.21.15.67:0/671156321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c815b860 0x7f49c80d0420 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f49b800bb10 tx=0x7f49b802e170 comp rx=0 tx=0).stop 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.802+0000 7f49cd400640 1 -- 172.21.15.67:0/671156321 shutdown_connections 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.802+0000 7f49cd400640 1 --2- 172.21.15.67:0/671156321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c815b860 0x7f49c80d0420 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.802+0000 7f49cd400640 1 -- 172.21.15.67:0/671156321 >> 172.21.15.67:0/671156321 conn(0x7f49c8067ee0 msgr2=0x7f49c806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.802+0000 7f49cd400640 1 -- 172.21.15.67:0/671156321 shutdown_connections 2024-04-17T07:30:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.802+0000 7f49cd400640 1 -- 172.21.15.67:0/671156321 wait complete. 2024-04-17T07:30:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.803+0000 7f49cd400640 1 Processor -- start 2024-04-17T07:30:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 -- start start 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49c80ce4d0 0x7f49c80aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49c815b860 0x7f49c80aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49c8072ae0 con 0x7f49c80b4f80 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49c8072800 con 0x7f49c815b860 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.804+0000 7f49cd400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49c815bc40 con 0x7f49c80ce4d0 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51804/0 (socket says 172.21.15.67:51804) 2024-04-17T07:30:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 -- 172.21.15.67:0/895141684 learned_addr learned my addr 172.21.15.67:0/895141684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c7200640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49c815b860 0x7f49c80aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 -- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49c80ce4d0 msgr2=0x7f49c80aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49c80ce4d0 0x7f49c80aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 -- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49c815b860 msgr2=0x7f49c80aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49c815b860 0x7f49c80aa610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.805+0000 7f49c6800640 1 -- 172.21.15.67:0/895141684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c8148310 con 0x7f49c80b4f80 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49c6800640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b2c0 tx=0x7f49b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49bea00640 1 -- 172.21.15.67:0/895141684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8045070 con 0x7f49c80b4f80 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49bea00640 1 -- 172.21.15.67:0/895141684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b8040890 con 0x7f49c80b4f80 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49bea00640 1 -- 172.21.15.67:0/895141684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b803b040 con 0x7f49c80b4f80 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c81485f0 con 0x7f49c80b4f80 2024-04-17T07:30:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.806+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c8148bb0 con 0x7f49c80b4f80 2024-04-17T07:30:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.809+0000 7f49bea00640 1 -- 172.21.15.67:0/895141684 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f49b8048070 con 0x7f49c80b4f80 2024-04-17T07:30:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.809+0000 7f49bea00640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 0x7f49a806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.810+0000 7f49bea00640 1 -- 172.21.15.67:0/895141684 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f49b80b0c50 con 0x7f49c80b4f80 2024-04-17T07:30:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.810+0000 7f49d0190740 1 Processor -- start 2024-04-17T07:30:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.810+0000 7f49c5e00640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 0x7f49a806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.810+0000 7f49d0190740 1 -- start start 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe9088ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908898a0 0x55fe9088b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe907ed980 con 0x55fe90885cf0 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55fe90794220 con 0x55fe908d3820 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49c5e00640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 0x7f49a806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49b4000f60 tx=0x7f49b40079d0 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.811+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fe907e9940 con 0x55fe908898a0 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51810/0 (socket says 172.21.15.67:51810) 2024-04-17T07:30:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 -- 172.21.15.67:0/4173338364 learned_addr learned my addr 172.21.15.67:0/4173338364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c6800640 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe9088ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c5e00640 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908898a0 0x55fe9088b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 -- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908898a0 msgr2=0x55fe9088b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908898a0 0x55fe9088b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 -- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 msgr2=0x55fe9088ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe9088ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.812+0000 7f49c7200640 1 -- 172.21.15.67:0/4173338364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe905002d0 con 0x55fe90885cf0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49c7200640 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f49c0007d60 tx=0x7f49c000c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701ab8e9945aab3b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49b3000640 1 -- 172.21.15.67:0/4173338364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c0015030 con 0x55fe90885cf0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49b3000640 1 -- 172.21.15.67:0/4173338364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c0003970 con 0x55fe90885cf0 2024-04-17T07:30:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49b3000640 1 -- 172.21.15.67:0/4173338364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c0003d40 con 0x55fe90885cf0 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49d0190740 1 -- 172.21.15.67:0/4173338364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 msgr2=0x55fe908b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.813+0000 7f49d0190740 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f49c0007d60 tx=0x7f49c000c560 comp rx=0 tx=0).stop 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 -- 172.21.15.67:0/4173338364 shutdown_connections 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe90885cf0 0x55fe908b5b30 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908898a0 0x55fe9088b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 --2- 172.21.15.67:0/4173338364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe9088ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 -- 172.21.15.67:0/4173338364 >> 172.21.15.67:0/4173338364 conn(0x55fe90887fb0 msgr2=0x55fe908d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 -- 172.21.15.67:0/4173338364 shutdown_connections 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 -- 172.21.15.67:0/4173338364 wait complete. 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.814+0000 7f49d0190740 1 -- start start 2024-04-17T07:30:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.815+0000 7f49d0190740 1 Processor -- start 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908d3820 0x55fe908b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe908d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe90889db0 con 0x55fe908898a0 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55fe907a7590 con 0x55fe908abc70 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fe9088a090 con 0x55fe908d3820 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49d0190740 10 client.0 _ll_register_callbacks cb 0x55fe904ffac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51814/0 (socket says 172.21.15.67:51814) 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 -- 172.21.15.67:0/2441026562 learned_addr learned my addr 172.21.15.67:0/2441026562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 -- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908d3820 msgr2=0x55fe908b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908d3820 0x55fe908b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 -- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 msgr2=0x55fe908d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe908d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.816+0000 7f49c6800640 1 -- 172.21.15.67:0/2441026562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe905ca6b0 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49c6800640 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f49b807e880 tx=0x7f49b807f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22dee448237b7da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49a7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b804f040 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f49b8040c80 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49c0003970 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f49b8087040 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49d0190740 10 client.7548 fetch_fsmap learned FSMap version 53 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55fe908d6ea0 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.817+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c0003970 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 53) ==== 3495+0+0 (secure 0 0 0) 0x7f49b800e040 con 0x55fe908898a0 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 10 client.7548 fetch_fsmap finished waiting for FSMap version 53 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 10 client.7548 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 20 client.7548 populate_metadata read hostname 'smithi067' 2024-04-17T07:30:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe908d8530 0x55fe908efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 4 client.7548 mds_command: new command op to 6232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:30:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.818+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 --> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe907fb3a0 con 0x55fe908d8530 2024-04-17T07:30:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.819+0000 7f49c5e00640 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe908d8530 0x55fe908efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.820+0000 7f49c5e00640 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe908d8530 0x55fe908efcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd9570e2460a88 server_cookie=2359d67cb0ba5d0c in_seq=0 out_seq=0 2024-04-17T07:30:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.820+0000 7f49a7400640 10 client.7548 ms_handle_connect on v2:172.21.15.187:6834/3534305613 2024-04-17T07:30:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49a7400640 1 -- 172.21.15.67:0/2441026562 <== mds.0 v2:172.21.15.187:6834/3534305613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe907fb3a0 con 0x55fe908d8530 2024-04-17T07:30:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49a7400640 10 client.7548 handle_command_reply: tid=0 2024-04-17T07:30:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 client.7548 shutdown 2024-04-17T07:30:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 msgr2=0x55fe908b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f49b807e880 tx=0x7f49b807f130 comp rx=0 tx=0).stop 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 shutdown_connections 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe908d8530 0x55fe908efcb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe908d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908d3820 0x55fe908b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 --2- 172.21.15.67:0/2441026562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908898a0 0x55fe908b25c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 >> 172.21.15.67:0/2441026562 conn(0x55fe90885cf0 msgr2=0x55fe9088b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49a7400640 0 client.7548 ms_handle_reset on v2:172.21.15.187:6834/3534305613 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 shutdown_connections 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 1 -- 172.21.15.67:0/2441026562 wait complete. 2024-04-17T07:30:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.822+0000 7f49d0190740 20 client.7548 trim_cache size 0 max 16384 2024-04-17T07:30:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.843+0000 7f49d0190740 1 Processor -- start 2024-04-17T07:30:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.843+0000 7f49d0190740 1 -- start start 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.843+0000 7f49d0190740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe908adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe908ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49d0190740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49c81fcde0 con 0x55fe908abc70 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49c81f4970 con 0x55fe908d3820 2024-04-17T07:30:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49c81f3cd0 con 0x55fe908cf870 2024-04-17T07:30:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51828/0 (socket says 172.21.15.67:51828) 2024-04-17T07:30:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49c6800640 1 -- 172.21.15.67:0/3387455744 learned_addr learned my addr 172.21.15.67:0/3387455744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49c7200640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe908ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.844+0000 7f49c5e00640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe908adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 -- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 msgr2=0x55fe908ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe908ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 -- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 msgr2=0x55fe908adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe908adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 -- 172.21.15.67:0/3387455744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe908eb7d0 con 0x55fe908abc70 2024-04-17T07:30:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c7200640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe908ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49c6800640 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f49b8036fd0 tx=0x7f49b804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739032053148c33a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49b2600640 1 -- 172.21.15.67:0/3387455744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b804f070 con 0x55fe908abc70 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49b2600640 1 -- 172.21.15.67:0/3387455744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b80871e0 con 0x55fe908abc70 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.845+0000 7f49b2600640 1 -- 172.21.15.67:0/3387455744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8040c80 con 0x55fe908abc70 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 -- 172.21.15.67:0/3387455744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 msgr2=0x55fe908d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f49b8036fd0 tx=0x7f49b804c000 comp rx=0 tx=0).stop 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 -- 172.21.15.67:0/3387455744 shutdown_connections 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe908ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908d3820 0x55fe908adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 --2- 172.21.15.67:0/3387455744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908abc70 0x55fe908d0b20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 -- 172.21.15.67:0/3387455744 >> 172.21.15.67:0/3387455744 conn(0x55fe90885cf0 msgr2=0x55fe908d6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 -- 172.21.15.67:0/3387455744 shutdown_connections 2024-04-17T07:30:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.846+0000 7f49d0190740 1 -- 172.21.15.67:0/3387455744 wait complete. 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.847+0000 7f49d0190740 1 -- start start 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.847+0000 7f49d0190740 1 Processor -- start 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe9088a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe9088af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe908d7180 con 0x55fe908d3820 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49c81f4320 con 0x55fe908abc70 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fe908ab9b0 con 0x55fe908cf870 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49d0190740 10 client.0 _ll_register_callbacks cb 0x7f49b800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:30:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51830/0 (socket says 172.21.15.67:51830) 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.848+0000 7f49c7200640 1 -- 172.21.15.67:0/2766090973 learned_addr learned my addr 172.21.15.67:0/2766090973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c5e00640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe9088af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c6800640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe9088a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 -- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 msgr2=0x55fe9088af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe9088af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 -- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 msgr2=0x55fe9088a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe9088a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 -- 172.21.15.67:0/2766090973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe908d2110 con 0x55fe908d3820 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49c7200640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f49c000c5a0 tx=0x7f49c0004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3a52f30d500873 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49a7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c0028060 con 0x55fe908d3820 2024-04-17T07:30:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f49c0004250 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49b80871e0 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f49c002e050 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.849+0000 7f49d0190740 10 client.7554 fetch_fsmap learned FSMap version 53 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55fe90951130 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b80871e0 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 53) ==== 3495+0+0 (secure 0 0 0) 0x7f49c0031020 con 0x55fe908d3820 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 10 client.7554 fetch_fsmap finished waiting for FSMap version 53 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 10 client.7554 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 20 client.7554 populate_metadata read hostname 'smithi067' 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe9094fbe0 0x55fe9094ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 4 client.7554 mds_command: new command op to 6232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:30:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.850+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 --> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] -- command(tid 0: {"prefix": "status"}) -- 0x55fe907f5210 con 0x55fe9094fbe0 2024-04-17T07:30:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.851+0000 7f49c6800640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe9094fbe0 0x55fe9094ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:30:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49c6800640 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe9094fbe0 0x55fe9094ffc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4185ec14f886053b server_cookie=11afbd6c8abb9aa8 in_seq=0 out_seq=0 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49a7400640 10 client.7554 ms_handle_connect on v2:172.21.15.187:6834/3534305613 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49a7400640 1 -- 172.21.15.67:0/2766090973 <== mds.0 v2:172.21.15.187:6834/3534305613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fe907f5210 con 0x55fe9094fbe0 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49a7400640 10 client.7554 handle_command_reply: tid=0 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 client.7554 shutdown 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 msgr2=0x55fe908829f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f49c000c5a0 tx=0x7f49c0004a70 comp rx=0 tx=0).stop 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 shutdown_connections 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:6834/3534305613,v1:172.21.15.187:6835/3534305613] conn(0x55fe9094fbe0 0x55fe9094ffc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe908d3820 0x55fe908829f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fe908cf870 0x55fe9088af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 --2- 172.21.15.67:0/2766090973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fe908abc70 0x55fe9088a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 >> 172.21.15.67:0/2766090973 conn(0x55fe90885cf0 msgr2=0x55fe9088b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49a7400640 0 client.7554 ms_handle_reset on v2:172.21.15.187:6834/3534305613 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 shutdown_connections 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.852+0000 7f49d0190740 1 -- 172.21.15.67:0/2766090973 wait complete. 2024-04-17T07:30:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49d0190740 20 client.7554 trim_cache size 0 max 16384 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6232, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8967976780000004, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 53, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 111, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 111, 2024-04-17T07:30:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 12.693310667 2024-04-17T07:30:59.855 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:30:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 msgr2=0x7f49a806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 0x7f49a806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49b4000f60 tx=0x7f49b40079d0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 msgr2=0x7f49c80abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f49b800b2c0 tx=0x7f49b80046c0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 shutdown_connections 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f49a806ca10 0x7f49a806eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49c815b860 0x7f49c80aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49c80ce4d0 0x7f49c80aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 --2- 172.21.15.67:0/895141684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49c80b4f80 0x7f49c80abd90 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 >> 172.21.15.67:0/895141684 conn(0x7f49c8067ee0 msgr2=0x7f49c815de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.853+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 shutdown_connections 2024-04-17T07:30:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:30:59.854+0000 7f49cd400640 1 -- 172.21.15.67:0/895141684 wait complete. 2024-04-17T07:30:59.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:31:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.086+0000 7fcfafe00640 1 Processor -- start 2024-04-17T07:31:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.086+0000 7fcfafe00640 1 -- start start 2024-04-17T07:31:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.088+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb015c5a0 0x7fcfb015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.088+0000 7fcfafe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfb00afb50 0x7fcfb00ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.088+0000 7fcfafe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfb015f440 con 0x7fcfb00b4210 2024-04-17T07:31:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.088+0000 7fcfafe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcfb015f720 con 0x7fcfb00afb50 2024-04-17T07:31:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.088+0000 7fcfafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb015fa00 con 0x7fcfb015c5a0 2024-04-17T07:31:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfb00afb50 0x7fcfb00ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfb00afb50 0x7fcfb00ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51832/0 (socket says 172.21.15.67:51832) 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 -- 172.21.15.67:0/1706414344 learned_addr learned my addr 172.21.15.67:0/1706414344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaea00640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb015c5a0 0x7fcfb015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 -- 172.21.15.67:0/1706414344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb015c5a0 msgr2=0x7fcfb015ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb015c5a0 0x7fcfb015ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 -- 172.21.15.67:0/1706414344 >> v1:172.21.15.187:6789/0 conn(0x7fcfb00b4210 legacy=0x7fcfb00af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.089+0000 7fcfaf400640 1 -- 172.21.15.67:0/1706414344 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb015fce0 con 0x7fcfb00afb50 2024-04-17T07:31:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.090+0000 7fcfaf400640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb00afb50 0x7fcfb00ae090 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009e80 tx=0x7fcfa802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f93741a35331cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.091+0000 7fcfae000640 1 -- 172.21.15.67:0/1706414344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa802c9c0 con 0x7fcfb00afb50 2024-04-17T07:31:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.091+0000 7fcfae000640 1 -- 172.21.15.67:0/1706414344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa802cb60 con 0x7fcfb00afb50 2024-04-17T07:31:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.091+0000 7fcfae000640 1 -- 172.21.15.67:0/1706414344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80343f0 con 0x7fcfb00afb50 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.091+0000 7fcfafe00640 1 -- 172.21.15.67:0/1706414344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb00afb50 msgr2=0x7fcfb00ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.091+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb00afb50 0x7fcfb00ae090 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009e80 tx=0x7fcfa802be20 comp rx=0 tx=0).stop 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 -- 172.21.15.67:0/1706414344 shutdown_connections 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb00afb50 0x7fcfb00ae090 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1706414344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb015c5a0 0x7fcfb015ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 -- 172.21.15.67:0/1706414344 >> 172.21.15.67:0/1706414344 conn(0x7fcfb0067d10 msgr2=0x7fcfb00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 -- 172.21.15.67:0/1706414344 shutdown_connections 2024-04-17T07:31:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.092+0000 7fcfafe00640 1 -- 172.21.15.67:0/1706414344 wait complete. 2024-04-17T07:31:00.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.093+0000 7fcfafe00640 1 Processor -- start 2024-04-17T07:31:00.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.094+0000 7fcfafe00640 1 -- start start 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfb00afb50 0x7fcfb01430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 0x7fcfb0146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb006e7f0 con 0x7fcfb015c5a0 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb006e670 con 0x7fcfb00afb50 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb00b4690 con 0x7fcfb00b4210 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51836/0 (socket says 172.21.15.67:51836) 2024-04-17T07:31:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 -- 172.21.15.67:0/1851703064 learned_addr learned my addr 172.21.15.67:0/1851703064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfa6000640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 0x7fcfb0146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaea00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfb00afb50 0x7fcfb01430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 -- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 msgr2=0x7fcfb0146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 0x7fcfb0146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 -- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfb00afb50 msgr2=0x7fcfb01430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfb00afb50 0x7fcfb01430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfaf400640 1 -- 172.21.15.67:0/1851703064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb01472d0 con 0x7fcfb015c5a0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.095+0000 7fcfa6000640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 0x7fcfb0146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfaf400640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8036040 tx=0x7fcfa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8043020 con 0x7fcfb015c5a0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa803e450 con 0x7fcfb015c5a0 2024-04-17T07:31:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa803e840 con 0x7fcfb015c5a0 2024-04-17T07:31:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb0143a90 con 0x7fcfb015c5a0 2024-04-17T07:31:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.096+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb0144050 con 0x7fcfb015c5a0 2024-04-17T07:31:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.097+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb00692b0 con 0x7fcfb015c5a0 2024-04-17T07:31:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.098+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fcfa8046020 con 0x7fcfb015c5a0 2024-04-17T07:31:00.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.099+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 0x7fcf8c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:00.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.099+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7fcfa80ae600 con 0x7fcfb015c5a0 2024-04-17T07:31:00.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.100+0000 7fcfaea00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 0x7fcf8c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:00.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.101+0000 7fcfaea00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 0x7fcf8c06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0008040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:00.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.104+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcfb00692b0 con 0x7fcfb015c5a0 2024-04-17T07:31:00.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.351+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcfb0068a10 con 0x7fcfb015c5a0 2024-04-17T07:31:00.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.352+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1851703064 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) ==== 53+0+73 (secure 0 0 0) 0x7fcfa804d030 con 0x7fcfb015c5a0 2024-04-17T07:31:00.351 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:31:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 msgr2=0x7fcf8c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 0x7fcf8c06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0008040 comp rx=0 tx=0).stop 2024-04-17T07:31:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 msgr2=0x7fcfb01462d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8036040 tx=0x7fcfa8004a40 comp rx=0 tx=0).stop 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 shutdown_connections 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fcf8c06ca10 0x7fcf8c06eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb015c5a0 0x7fcfb01462d0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfb00b4210 0x7fcfb0146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 --2- 172.21.15.67:0/1851703064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfb00afb50 0x7fcfb01430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 >> 172.21.15.67:0/1851703064 conn(0x7fcfb0067d10 msgr2=0x7fcfb015a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 shutdown_connections 2024-04-17T07:31:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00.355+0000 7fcfafe00640 1 -- 172.21.15.67:0/1851703064 wait complete. 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:31:00.375 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:31:00.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:00.376 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:31:00.430 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:31:00.430 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:31:00.430 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:31:00.431 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:31:00.432 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:31:00.432 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:31:00.432 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:31:00.432 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:31:00.432 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41584sec preferred_lft 41584sec 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:31:00.433 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:31:00.434 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:31:00.434 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.434 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:31:00.434 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.434 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:31:00.435 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:31:00.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:00.436 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:31:00.534 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:31:00.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:00.538 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:31:00.592 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:31:00.592 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:31:00.592 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:31:00.592 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:31:00.592 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:31:00.593 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:31:00.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:31:00.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:31:00.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:00.856 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:31:00.905 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:31:00.905 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:31:00.905 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:31:00.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:00.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:31:01.006 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:31:01.006 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:31:01.062 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:31:01.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:31:01.063 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:31:01.114 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:31:01.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:01.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:31:01.259 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:31:01.259 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:31:01.294 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:31:01.295 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:31:01.295 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:31:01.295 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:31:01.295 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:31:01.297 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:31:01.297 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:31:01.297 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:31:01.297 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:31:01.297 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:31:01.301 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:31:01.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:31:01.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:31:01.391 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:31:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:31:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:31:01.537 INFO:teuthology.orchestra.run.smithi067.stdout:client7563 2024-04-17T07:31:01.538 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:31:01.538 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:31:01.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:31:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.809+0000 7fa434c00640 1 Processor -- start 2024-04-17T07:31:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.810+0000 7fa434c00640 1 -- start start 2024-04-17T07:31:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.811+0000 7fa434c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 0x7fa43015f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.811+0000 7fa434c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 0x7fa4300ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.811+0000 7fa434c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4300cb420 con 0x7fa4300afbf0 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.811+0000 7fa434c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa43015fd60 con 0x7fa4300ce510 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.811+0000 7fa434c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4300cc030 con 0x7fa43015f3c0 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.812+0000 7fa427e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 0x7fa43015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.812+0000 7fa42fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 0x7fa4300ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.812+0000 7fa42f400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa4300afbf0 0x7fa43006e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:39038/0 (socket says 172.21.15.67:39038) 2024-04-17T07:31:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.812+0000 7fa42f400640 1 -- 172.21.15.67:0/2274632658 learned_addr learned my addr 172.21.15.67:0/2274632658 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:31:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.813+0000 7fa427e00640 1 -- 172.21.15.67:0/2274632658 >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 msgr2=0x7fa4300ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.813+0000 7fa427e00640 1 --2- 172.21.15.67:0/2274632658 >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 0x7fa4300ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.813+0000 7fa427e00640 1 -- 172.21.15.67:0/2274632658 >> v1:172.21.15.187:6789/0 conn(0x7fa4300afbf0 legacy=0x7fa43006e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:01.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.813+0000 7fa427e00640 1 -- 172.21.15.67:0/2274632658 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4300cb700 con 0x7fa43015f3c0 2024-04-17T07:31:01.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.813+0000 7fa42fe00640 1 --2- 172.21.15.67:0/2274632658 >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 0x7fa4300ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:31:01.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.815+0000 7fa427e00640 1 --2- 172.21.15.67:0/2274632658 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 0x7fa43015f820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa4180097d0 tx=0x7fa41802dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d32b71c645ce357 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:01.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.815+0000 7fa42ea00640 1 -- 172.21.15.67:0/2274632658 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41802f880 con 0x7fa43015f3c0 2024-04-17T07:31:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa42ea00640 1 -- 172.21.15.67:0/2274632658 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41802fa20 con 0x7fa43015f3c0 2024-04-17T07:31:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa42ea00640 1 -- 172.21.15.67:0/2274632658 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41802fdf0 con 0x7fa43015f3c0 2024-04-17T07:31:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa434c00640 1 -- 172.21.15.67:0/2274632658 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 msgr2=0x7fa43015f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa434c00640 1 --2- 172.21.15.67:0/2274632658 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 0x7fa43015f820 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa4180097d0 tx=0x7fa41802dac0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa434c00640 1 -- 172.21.15.67:0/2274632658 shutdown_connections 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa434c00640 1 --2- 172.21.15.67:0/2274632658 >> v2:172.21.15.187:3300/0 conn(0x7fa4300ce510 0x7fa4300ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.816+0000 7fa434c00640 1 --2- 172.21.15.67:0/2274632658 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa43015f3c0 0x7fa43015f820 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.817+0000 7fa434c00640 1 -- 172.21.15.67:0/2274632658 >> 172.21.15.67:0/2274632658 conn(0x7fa43006bef0 msgr2=0x7fa4300b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.817+0000 7fa434c00640 1 -- 172.21.15.67:0/2274632658 shutdown_connections 2024-04-17T07:31:01.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.817+0000 7fa434c00640 1 -- 172.21.15.67:0/2274632658 wait complete. 2024-04-17T07:31:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.819+0000 7fa434c00640 1 Processor -- start 2024-04-17T07:31:01.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.819+0000 7fa434c00640 1 -- start start 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4300afbf0 0x7fa4301563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43015f3c0 0x7fa4301548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa43006f090 con 0x7fa4300ce510 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4300cbcc0 con 0x7fa43015f3c0 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.820+0000 7fa434c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa430160110 con 0x7fa4300afbf0 2024-04-17T07:31:01.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51838/0 (socket says 172.21.15.67:51838) 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 -- 172.21.15.67:0/4058320812 learned_addr learned my addr 172.21.15.67:0/4058320812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa42f400640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4300afbf0 0x7fa4301563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa42fe00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43015f3c0 0x7fa4301548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 -- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4300afbf0 msgr2=0x7fa4301563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4300afbf0 0x7fa4301563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 -- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43015f3c0 msgr2=0x7fa4301548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43015f3c0 0x7fa4301548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:01.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.821+0000 7fa427e00640 1 -- 172.21.15.67:0/4058320812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa430155330 con 0x7fa4300ce510 2024-04-17T07:31:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa427e00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa41802e0d0 tx=0x7fa4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418030b20 con 0x7fa4300ce510 2024-04-17T07:31:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418030ce0 con 0x7fa4300ce510 2024-04-17T07:31:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180446c0 con 0x7fa4300ce510 2024-04-17T07:31:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa430155610 con 0x7fa4300ce510 2024-04-17T07:31:01.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.822+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa430155bd0 con 0x7fa4300ce510 2024-04-17T07:31:01.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.824+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa430068a10 con 0x7fa4300ce510 2024-04-17T07:31:01.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.825+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7fa41804b030 con 0x7fa4300ce510 2024-04-17T07:31:01.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.826+0000 7fa427400640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 0x7fa40c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:01.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.826+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7fa4180b0480 con 0x7fa4300ce510 2024-04-17T07:31:01.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.826+0000 7fa42f400640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 0x7fa40c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:01.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.828+0000 7fa42f400640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 0x7fa40c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa420005880 tx=0x7fa420009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:01.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:01.830+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa418052020 con 0x7fa4300ce510 2024-04-17T07:31:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.061+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 --> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa4300b5d60 con 0x7fa40c06ca40 2024-04-17T07:31:02.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.065+0000 7fa427400640 1 -- 172.21.15.67:0/4058320812 <== mgr.6091 v2:172.21.15.187:6832/158797128 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa4300b5d60 con 0x7fa40c06ca40 2024-04-17T07:31:02.065 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:31:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:31:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:31:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:31:02.065 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:31:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 msgr2=0x7fa40c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 0x7fa40c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa420005880 tx=0x7fa420009040 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 msgr2=0x7fa43013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa41802e0d0 tx=0x7fa4180046c0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 shutdown_connections 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7fa40c06ca40 0x7fa40c06ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43015f3c0 0x7fa4301548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4300ce510 0x7fa43013edb0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 --2- 172.21.15.67:0/4058320812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4300afbf0 0x7fa4301563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 >> 172.21.15.67:0/4058320812 conn(0x7fa43006bef0 msgr2=0x7fa430071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 shutdown_connections 2024-04-17T07:31:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.068+0000 7fa434c00640 1 -- 172.21.15.67:0/4058320812 wait complete. 2024-04-17T07:31:02.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:31:02.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.308+0000 7f233c400640 1 Processor -- start 2024-04-17T07:31:02.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.308+0000 7f233c400640 1 -- start start 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.309+0000 7f233c400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f2334072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.309+0000 7f233c400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f233415fc00 0x7f23341686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233c400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f233406f990 con 0x7f233406f4f0 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233c400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f23340d0af0 con 0x7f233415fc00 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233c400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f233406fc70 con 0x7f233415f110 2024-04-17T07:31:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f233415fc00 0x7f23341686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f233415fc00 0x7f23341686c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51852/0 (socket says 172.21.15.67:51852) 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f233a400640 1 -- 172.21.15.67:0/4088823679 learned_addr learned my addr 172.21.15.67:0/4088823679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.310+0000 7f2339a00640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f2334072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.311+0000 7f233a400640 1 -- 172.21.15.67:0/4088823679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 msgr2=0x7f2334072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.311+0000 7f233a400640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f2334072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.311+0000 7f233a400640 1 -- 172.21.15.67:0/4088823679 >> v1:172.21.15.187:6789/0 conn(0x7f233406f4f0 legacy=0x7f233415f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.311+0000 7f233a400640 1 -- 172.21.15.67:0/4088823679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23340d04a0 con 0x7f233415fc00 2024-04-17T07:31:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.312+0000 7f233a400640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233415fc00 0x7f23341686c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f232c009e50 tx=0x7f232c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b5bd03fc44133d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.312+0000 7f2333e00640 1 -- 172.21.15.67:0/4088823679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c02cb00 con 0x7f233415fc00 2024-04-17T07:31:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.312+0000 7f2333e00640 1 -- 172.21.15.67:0/4088823679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232c02cca0 con 0x7f233415fc00 2024-04-17T07:31:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.312+0000 7f2333e00640 1 -- 172.21.15.67:0/4088823679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c034610 con 0x7f233415fc00 2024-04-17T07:31:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 -- 172.21.15.67:0/4088823679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233415fc00 msgr2=0x7f23341686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233415fc00 0x7f23341686c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f232c009e50 tx=0x7f232c02bf60 comp rx=0 tx=0).stop 2024-04-17T07:31:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 -- 172.21.15.67:0/4088823679 shutdown_connections 2024-04-17T07:31:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233415fc00 0x7f23341686c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 --2- 172.21.15.67:0/4088823679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f2334072870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 -- 172.21.15.67:0/4088823679 >> 172.21.15.67:0/4088823679 conn(0x7f2334067ee0 msgr2=0x7f233406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 -- 172.21.15.67:0/4088823679 shutdown_connections 2024-04-17T07:31:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.313+0000 7f233c400640 1 -- 172.21.15.67:0/4088823679 wait complete. 2024-04-17T07:31:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.315+0000 7f233c400640 1 Processor -- start 2024-04-17T07:31:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.315+0000 7f233c400640 1 -- start start 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f23341e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f233415fc00 0x7f2334147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2334072db0 con 0x7f233406f4f0 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f23340afdc0 con 0x7f233415fc00 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.316+0000 7f233c400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23340cf3c0 con 0x7f233415f110 2024-04-17T07:31:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51854/0 (socket says 172.21.15.67:51854) 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 -- 172.21.15.67:0/172311646 learned_addr learned my addr 172.21.15.67:0/172311646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339000640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f23341e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f233a400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f233415fc00 0x7f2334147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 -- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 msgr2=0x7f23341e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f23341e72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 -- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f233415fc00 msgr2=0x7f2334147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f233415fc00 0x7f2334147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.317+0000 7f2339a00640 1 -- 172.21.15.67:0/172311646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23341e79d0 con 0x7f233406f4f0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f2339a00640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f232800e9b0 tx=0x7f232800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328017070 con 0x7f233406f4f0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232800c9a0 con 0x7f233406f4f0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23341e7cb0 con 0x7f233406f4f0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800cd90 con 0x7f233406f4f0 2024-04-17T07:31:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.318+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23341e8270 con 0x7f233406f4f0 2024-04-17T07:31:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.319+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2334068ca0 con 0x7f233406f4f0 2024-04-17T07:31:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.320+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 46) ==== 91142+0+0 (secure 0 0 0) 0x7f232801a070 con 0x7f233406f4f0 2024-04-17T07:31:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.321+0000 7f2331600640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 0x7f231406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:02.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.321+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f232808c090 con 0x7f233406f4f0 2024-04-17T07:31:02.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.322+0000 7f2339000640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 0x7f231406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:02.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.323+0000 7f2339000640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 0x7f231406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f23340cf9f0 tx=0x7f2320009040 comp rx=0 tx=0).ready entity=mgr.6091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.326+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f232805a3b0 con 0x7f233406f4f0 2024-04-17T07:31:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:02.584+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f233415f590 con 0x7f233406f4f0 2024-04-17T07:31:03.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.039+0000 7f2331600640 1 -- 172.21.15.67:0/172311646 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) ==== 92+0+0 (secure 0 0 0) 0x7f2328059cb0 con 0x7f233406f4f0 2024-04-17T07:31:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.044+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 msgr2=0x7f231406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.044+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 0x7f231406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f23340cf9f0 tx=0x7f2320009040 comp rx=0 tx=0).stop 2024-04-17T07:31:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.044+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 msgr2=0x7f233414a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.044+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f232800e9b0 tx=0x7f232800ee70 comp rx=0 tx=0).stop 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.045+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 shutdown_connections 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.045+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f231406ca10 0x7f231406eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.045+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f233415fc00 0x7f2334147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.045+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f233415f110 0x7f23341e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.045+0000 7f233c400640 1 --2- 172.21.15.67:0/172311646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f233406f4f0 0x7f233414a980 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.046+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 >> 172.21.15.67:0/172311646 conn(0x7f2334067ee0 msgr2=0x7f23340b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.046+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 shutdown_connections 2024-04-17T07:31:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.046+0000 7f233c400640 1 -- 172.21.15.67:0/172311646 wait complete. 2024-04-17T07:31:03.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:31:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.286+0000 7f7da9a00640 1 Processor -- start 2024-04-17T07:31:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.286+0000 7f7da9a00640 1 -- start start 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.287+0000 7f7da9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7da40d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da9a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da40cfcc0 con 0x7f7da40ce4d0 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da9a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da4071f60 con 0x7f7da40b4f80 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7da4071d10 con 0x7f7da415b860 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7da40ce4d0 0x7f7da40ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47778/0 (socket says 172.21.15.67:47778) 2024-04-17T07:31:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da2000640 1 -- 172.21.15.67:0/3443811770 learned_addr learned my addr 172.21.15.67:0/3443811770 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da2a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7da40d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.288+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111584258 0 0) 0x7f7da40cfcc0 con 0x7f7da40ce4d0 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d88003520 con 0x7f7da40ce4d0 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290203544 0 0) 0x7f7da4071f60 con 0x7f7da40b4f80 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da40cfcc0 con 0x7f7da40b4f80 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3423175786 0 0) 0x7f7d88003520 con 0x7f7da40ce4d0 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 msgr2=0x7f7da40d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7da40d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 >> v1:172.21.15.187:6789/0 conn(0x7f7da40b4f80 legacy=0x7f7da40cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da4072240 con 0x7f7da40ce4d0 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7d8c001670 con 0x7f7da40ce4d0 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 >> v1:172.21.15.67:6789/0 conn(0x7f7da40ce4d0 legacy=0x7f7da40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d88003b70 0x7f7d88003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d88004800 0x7f7d88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7d880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7da4071d10 con 0x7f7d88003b70 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8800ba70 con 0x7f7d88004800 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8800bc40 con 0x7f7da415b860 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.289+0000 7f7da2a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d88003b70 0x7f7d88003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da3400640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7d880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2a00640 1 -- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 msgr2=0x7f7d880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7da415b860 0x7f7d880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2000640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d88004800 0x7f7d88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2a00640 1 -- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d88004800 msgr2=0x7f7d88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d88004800 0x7f7d88004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.290+0000 7f7da2a00640 1 -- 172.21.15.67:0/3443811770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d88003520 con 0x7f7d88003b70 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.291+0000 7f7da2a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d88003b70 0x7f7d88003f70 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009ac0 tx=0x7f7d88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d129fa4af46db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.291+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94039070 con 0x7f7d88003b70 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.291+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9402c660 con 0x7f7d88003b70 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.291+0000 7f7da1600640 1 -- 172.21.15.67:0/3443811770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94009690 con 0x7f7d88003b70 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 -- 172.21.15.67:0/3443811770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d88003b70 msgr2=0x7f7d88003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 --2- 172.21.15.67:0/3443811770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d88003b70 0x7f7d88003f70 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009ac0 tx=0x7f7d88005830 comp rx=0 tx=0).stop 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da3400640 1 -- 172.21.15.67:0/3443811770 reap_dead start 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 -- 172.21.15.67:0/3443811770 shutdown_connections 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 -- 172.21.15.67:0/3443811770 >> 172.21.15.67:0/3443811770 conn(0x7f7da4067ee0 msgr2=0x7f7da406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 -- 172.21.15.67:0/3443811770 shutdown_connections 2024-04-17T07:31:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.292+0000 7f7da9a00640 1 -- 172.21.15.67:0/3443811770 wait complete. 2024-04-17T07:31:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.294+0000 7f7da9a00640 1 Processor -- start 2024-04-17T07:31:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.294+0000 7f7da9a00640 1 -- start start 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d88003b70 0x7f7da41448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 0x7f7da40ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7da4072ae0 con 0x7f7da415d900 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7da4072800 con 0x7f7da40ac100 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7da415bc40 con 0x7f7d88003b70 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.295+0000 7f7da3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49518/0 (socket says 172.21.15.67:49518) 2024-04-17T07:31:03.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 -- 172.21.15.67:0/2110190020 learned_addr learned my addr 172.21.15.67:0/2110190020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da2000640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 0x7f7da40ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da2a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d88003b70 0x7f7da41448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d88003b70 msgr2=0x7f7da41448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d88003b70 0x7f7da41448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 msgr2=0x7f7da40ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 0x7f7da40ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 -- 172.21.15.67:0/2110190020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da40acac0 con 0x7f7da415d900 2024-04-17T07:31:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da2000640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 0x7f7da40ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.296+0000 7f7da3400640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f7da40b53f0 tx=0x7f7d9c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.297+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c003970 con 0x7f7da415d900 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.297+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c018070 con 0x7f7da415d900 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.297+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c003d50 con 0x7f7da415d900 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.297+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da415e8a0 con 0x7f7da415d900 2024-04-17T07:31:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.297+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da40ae320 con 0x7f7da415d900 2024-04-17T07:31:03.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.298+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da4068ca0 con 0x7f7da415d900 2024-04-17T07:31:03.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.300+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 47) ==== 91159+0+0 (secure 0 0 0) 0x7f7d9c0050d0 con 0x7f7da415d900 2024-04-17T07:31:03.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.300+0000 7f7d9aa00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 0x7f7d8006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.301+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f7d9c08bb90 con 0x7f7da415d900 2024-04-17T07:31:03.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.301+0000 7f7da2a00640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 msgr2=0x7f7d8006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:03.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.301+0000 7f7da2a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 0x7f7d8006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:31:03.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.305+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7d9c059e10 con 0x7f7da415d900 2024-04-17T07:31:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.501+0000 7f7da2a00640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 msgr2=0x7f7d8006ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.501+0000 7f7da2a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 0x7f7d8006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:31:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.510+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7da4158800 con 0x7f7da415d900 2024-04-17T07:31:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.515+0000 7f7d9aa00640 1 -- 172.21.15.67:0/2110190020 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) ==== 138+0+0 (secure 0 0 0) 0x7f7d9c059710 con 0x7f7da415d900 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.517+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 msgr2=0x7f7d8006ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.517+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 0x7f7d8006ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.517+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 msgr2=0x7f7da415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.517+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f7da40b53f0 tx=0x7f7d9c00eb20 comp rx=0 tx=0).stop 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.517+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 shutdown_connections 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f7d8006ca90 0x7f7d8006ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da415d900 0x7f7da415dd80 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7da40ac100 0x7f7da40ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 --2- 172.21.15.67:0/2110190020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7d88003b70 0x7f7da41448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 >> 172.21.15.67:0/2110190020 conn(0x7f7da4067ee0 msgr2=0x7f7da406dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:03.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 shutdown_connections 2024-04-17T07:31:03.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.518+0000 7f7da9a00640 1 -- 172.21.15.67:0/2110190020 wait complete. 2024-04-17T07:31:03.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:31:03.603 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:03.603+0000 7fe883652040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:31:03.636 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:03.636+0000 7f56daa2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:31:03.657 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:03.657+0000 7fe883652040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:31:03.694 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:03.694+0000 7f56daa2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:31:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.745+0000 7f560de00640 1 Processor -- start 2024-04-17T07:31:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.745+0000 7f560de00640 1 -- start start 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.746+0000 7f560de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f560815ecb0 0x7f56080d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.746+0000 7f560de00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f56080d3210 0x7f56080cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.746+0000 7f560de00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56080d00b0 con 0x7f56080cf8e0 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.746+0000 7f560de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f56080d0c30 con 0x7f560815ecb0 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.746+0000 7f560de00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f56080d0390 con 0x7f56080d3210 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f560815ecb0 0x7f56080d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f560815ecb0 0x7f56080d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49534/0 (socket says 172.21.15.67:49534) 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 -- 172.21.15.67:0/3768486027 learned_addr learned my addr 172.21.15.67:0/3768486027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 -- 172.21.15.67:0/3768486027 >> v2:172.21.15.187:3300/0 conn(0x7f56080d3210 msgr2=0x7f56080cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 --2- 172.21.15.67:0/3768486027 >> v2:172.21.15.187:3300/0 conn(0x7f56080d3210 0x7f56080cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 -- 172.21.15.67:0/3768486027 >> v1:172.21.15.187:6789/0 conn(0x7f56080cf8e0 legacy=0x7f56080d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.747+0000 7f560ca00640 1 -- 172.21.15.67:0/3768486027 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56080d0670 con 0x7f560815ecb0 2024-04-17T07:31:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.748+0000 7f560ca00640 1 --2- 172.21.15.67:0/3768486027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f56080d2cd0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f56081627f0 tx=0x7f55fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f45682aae5f962a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.749+0000 7f5607400640 1 -- 172.21.15.67:0/3768486027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc02e070 con 0x7f560815ecb0 2024-04-17T07:31:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.749+0000 7f5607400640 1 -- 172.21.15.67:0/3768486027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc037070 con 0x7f560815ecb0 2024-04-17T07:31:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.749+0000 7f5607400640 1 -- 172.21.15.67:0/3768486027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc031a00 con 0x7f560815ecb0 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.749+0000 7f560de00640 1 -- 172.21.15.67:0/3768486027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 msgr2=0x7f56080d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.749+0000 7f560de00640 1 --2- 172.21.15.67:0/3768486027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f56080d2cd0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f56081627f0 tx=0x7f55fc02b880 comp rx=0 tx=0).stop 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 -- 172.21.15.67:0/3768486027 shutdown_connections 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 --2- 172.21.15.67:0/3768486027 >> v2:172.21.15.187:3300/0 conn(0x7f56080d3210 0x7f56080cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 --2- 172.21.15.67:0/3768486027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f56080d2cd0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 -- 172.21.15.67:0/3768486027 >> 172.21.15.67:0/3768486027 conn(0x7f560808bff0 msgr2=0x7f56080d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 -- 172.21.15.67:0/3768486027 shutdown_connections 2024-04-17T07:31:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.750+0000 7f560de00640 1 -- 172.21.15.67:0/3768486027 wait complete. 2024-04-17T07:31:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.751+0000 7f560de00640 1 Processor -- start 2024-04-17T07:31:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.751+0000 7f560de00640 1 -- start start 2024-04-17T07:31:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56080cf8e0 0x7f56081510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56080d3210 0x7f560815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56080cfd00 con 0x7f560815ecb0 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f560815f090 con 0x7f56080cf8e0 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.752+0000 7f560de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f56080cc800 con 0x7f56080d3210 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49542/0 (socket says 172.21.15.67:49542) 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 -- 172.21.15.67:0/2378563720 learned_addr learned my addr 172.21.15.67:0/2378563720 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560ca00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56080cf8e0 0x7f56081510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f5607e00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56080d3210 0x7f560815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56080d3210 msgr2=0x7f560815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56080d3210 0x7f560815f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56080cf8e0 msgr2=0x7f56081510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56080cf8e0 0x7f56081510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.753+0000 7f560d400640 1 -- 172.21.15.67:0/2378563720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5608161c80 con 0x7f560815ecb0 2024-04-17T07:31:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f560d400640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f560000c780 tx=0x7f560000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600004b60 con 0x7f560815ecb0 2024-04-17T07:31:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5600004d00 con 0x7f560815ecb0 2024-04-17T07:31:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560000aa30 con 0x7f560815ecb0 2024-04-17T07:31:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5608161f60 con 0x7f560815ecb0 2024-04-17T07:31:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.754+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5608162520 con 0x7f560815ecb0 2024-04-17T07:31:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.755+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5608068a10 con 0x7f560815ecb0 2024-04-17T07:31:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.757+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 47) ==== 91159+0+0 (secure 0 0 0) 0x7f560000abd0 con 0x7f560815ecb0 2024-04-17T07:31:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.758+0000 7f5604c00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 0x7f55e806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.759+0000 7f560ca00640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 msgr2=0x7f55e806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.759+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7f560008c210 con 0x7f560815ecb0 2024-04-17T07:31:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.759+0000 7f560ca00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 0x7f55e806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:31:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.764+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f560005a5c0 con 0x7f560815ecb0 2024-04-17T07:31:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.960+0000 7f560ca00640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 msgr2=0x7f55e806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:03.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.960+0000 7f560ca00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 0x7f55e806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:31:03.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.967+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f56080b3760 con 0x7f560815ecb0 2024-04-17T07:31:03.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.971+0000 7f5604c00640 1 -- 172.21.15.67:0/2378563720 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) ==== 133+0+0 (secure 0 0 0) 0x7f5600059ec0 con 0x7f560815ecb0 2024-04-17T07:31:03.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 msgr2=0x7f55e806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:31:03.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 0x7f55e806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 msgr2=0x7f560814d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f560000c780 tx=0x7f560000cc40 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 shutdown_connections 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7f55e806ca60 0x7f55e806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560815ecb0 0x7f560814d720 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56080d3210 0x7f560815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 --2- 172.21.15.67:0/2378563720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56080cf8e0 0x7f56081510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 >> 172.21.15.67:0/2378563720 conn(0x7f560808bff0 msgr2=0x7f56080b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 shutdown_connections 2024-04-17T07:31:03.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:03.974+0000 7f560de00640 1 -- 172.21.15.67:0/2378563720 wait complete. 2024-04-17T07:31:03.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-04-17T07:31:04.103 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.103+0000 7fe883652040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:31:04.155 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.155+0000 7fe883652040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:31:04.177 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.177+0000 7f56daa2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:31:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.194+0000 7faf9ca00640 1 Processor -- start 2024-04-17T07:31:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.195+0000 7faf9ca00640 1 -- start start 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.196+0000 7faf9ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf9809aec0 0x7faf980cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.196+0000 7faf9ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf980d07a0 con 0x7faf9809aa60 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.196+0000 7faf9ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf980d1ae0 con 0x7faf980ce480 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.196+0000 7faf9ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faf980d1650 con 0x7faf9809aec0 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf9809aec0 0x7faf980cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf9809aec0 0x7faf980cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49546/0 (socket says 172.21.15.67:49546) 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 -- 172.21.15.67:0/1198632124 learned_addr learned my addr 172.21.15.67:0/1198632124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 -- 172.21.15.67:0/1198632124 >> v1:172.21.15.187:6789/0 conn(0x7faf980ce480 legacy=0x7faf9806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:04.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 -- 172.21.15.67:0/1198632124 >> v1:172.21.15.67:6789/0 conn(0x7faf9809aa60 legacy=0x7faf980ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:04.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.197+0000 7faf97e00640 1 -- 172.21.15.67:0/1198632124 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9809ba50 con 0x7faf9809aec0 2024-04-17T07:31:04.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.198+0000 7faf97e00640 1 --2- 172.21.15.67:0/1198632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf9809aec0 0x7faf980cd2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7faf90009e50 tx=0x7faf9002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587a512b2ae61a6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:04.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf96a00640 1 -- 172.21.15.67:0/1198632124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9002cbb0 con 0x7faf9809aec0 2024-04-17T07:31:04.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf96a00640 1 -- 172.21.15.67:0/1198632124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9002cd50 con 0x7faf9809aec0 2024-04-17T07:31:04.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf96a00640 1 -- 172.21.15.67:0/1198632124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90034610 con 0x7faf9809aec0 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf9ca00640 1 -- 172.21.15.67:0/1198632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf9809aec0 msgr2=0x7faf980cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf9ca00640 1 --2- 172.21.15.67:0/1198632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf9809aec0 0x7faf980cd2c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7faf90009e50 tx=0x7faf9002c010 comp rx=0 tx=0).stop 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf9ca00640 1 -- 172.21.15.67:0/1198632124 shutdown_connections 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf9ca00640 1 --2- 172.21.15.67:0/1198632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf9809aec0 0x7faf980cd2c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.199+0000 7faf9ca00640 1 -- 172.21.15.67:0/1198632124 >> 172.21.15.67:0/1198632124 conn(0x7faf980b4050 msgr2=0x7faf980b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.200+0000 7faf9ca00640 1 -- 172.21.15.67:0/1198632124 shutdown_connections 2024-04-17T07:31:04.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.200+0000 7faf9ca00640 1 -- 172.21.15.67:0/1198632124 wait complete. 2024-04-17T07:31:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.201+0000 7faf9ca00640 1 Processor -- start 2024-04-17T07:31:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.201+0000 7faf9ca00640 1 -- start start 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf9809aa60 0x7faf981f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf9809aec0 0x7faf980c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 0x7faf980cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faf980d1000 con 0x7faf980ce480 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faf980d0d20 con 0x7faf9809aec0 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf9ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faf980d12e0 con 0x7faf9809aa60 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 0x7faf980cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.202+0000 7faf97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 0x7faf980cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49558/0 (socket says 172.21.15.67:49558) 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 -- 172.21.15.67:0/1582966838 learned_addr learned my addr 172.21.15.67:0/1582966838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf8fe00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf9809aec0 0x7faf980c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf9809aa60 0x7faf981f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf9809aa60 msgr2=0x7faf981f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf9809aa60 0x7faf981f3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf9809aec0 msgr2=0x7faf980c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf9809aec0 0x7faf980c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf980ca0b0 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf97e00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 0x7faf980cb4d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7faf90036040 tx=0x7faf90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.203+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90043020 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.204+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9003e450 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.204+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9003e820 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.204+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf980ca390 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.204+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9815fdf0 con 0x7faf980ce480 2024-04-17T07:31:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.205+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf980755d0 con 0x7faf980ce480 2024-04-17T07:31:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.206+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 47) ==== 91159+0+0 (secure 0 0 0) 0x7faf90046020 con 0x7faf980ce480 2024-04-17T07:31:04.206 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.206+0000 7fe883652040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:31:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.207+0000 7faf8f400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.207+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5709+0+0 (secure 0 0 0) 0x7faf900ae630 con 0x7faf980ce480 2024-04-17T07:31:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.207+0000 7faf97400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.207+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:31:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.211+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faf9004d030 con 0x7faf980ce480 2024-04-17T07:31:04.236 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.235+0000 7f56daa2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:31:04.268 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.268+0000 7fe883652040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:31:04.290 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.290+0000 7f56daa2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:31:04.318 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.318+0000 7fe883652040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:31:04.355 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.355+0000 7f56daa2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:31:04.408 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.408+0000 7f56daa2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:31:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.408+0000 7faf97400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.408+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:31:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.431+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7faf980b4db0 con 0x7faf7406ca60 2024-04-17T07:31:04.446 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.446+0000 7fe883652040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:31:04.508 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.508+0000 7fe883652040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:31:04.544 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.544+0000 7f56daa2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:31:04.559 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.559+0000 7fe883652040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:31:04.610 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.610+0000 7f56daa2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:31:04.616 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.616+0000 7fe883652040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:31:04.663 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.663+0000 7f56daa2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:31:04.666 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:04.666+0000 7fe883652040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:31:04.723 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.723+0000 7f56daa2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:31:04.777 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:04.776+0000 7f56daa2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:31:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.809+0000 7faf97400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:04.809+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:31:05.049 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.049+0000 7fe883652040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:31:05.185 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.185+0000 7f56daa2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:31:05.207 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.207+0000 7fe883652040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:31:05.316 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.316+0000 7fe883652040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:31:05.353 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.353+0000 7f56daa2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:31:05.366 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.366+0000 7fe883652040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:31:05.469 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.469+0000 7f56daa2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:31:05.581 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.492+0000 7fe883652040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:31:05.581 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.522+0000 7f56daa2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:31:05.598 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.598+0000 7fe883652040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:31:05.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:05.610+0000 7faf97400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:05.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:05.610+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:31:05.655 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.655+0000 7f56daa2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:31:05.768 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.768+0000 7f56daa2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:31:05.772 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.772+0000 7fe883652040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:31:05.821 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.821+0000 7fe883652040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:31:05.953 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:05.953+0000 7f56daa2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:31:05.956 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:05.956+0000 7fe883652040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:31:06.005 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:06.005+0000 7f56daa2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:31:06.149 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:06.149+0000 7f56daa2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:31:06.759 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:06.759+0000 7fe883652040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:31:07.004 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:07.004+0000 7f56daa2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:31:07.061 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.061+0000 7fe883652040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:31:07.118 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.118+0000 7fe883652040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:31:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:07.212+0000 7faf97400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/158797128 2024-04-17T07:31:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:07.213+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T07:31:07.249 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.249+0000 7fe883652040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:31:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:07.271+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 48) ==== 90768+0+0 (secure 0 0 0) 0x7faf90078950 con 0x7faf980ce480 2024-04-17T07:31:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:07.272+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 msgr2=0x7faf7406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:31:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:07.272+0000 7faf8f400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/158797128,v1:172.21.15.187:6833/158797128] conn(0x7faf7406ca60 0x7faf7406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:07.337 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:07.337+0000 7f56daa2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:31:07.407 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:07.407+0000 7f56daa2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:31:07.563 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:31:07.563+0000 7f56daa2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:31:07.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.579+0000 7fe7f6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.579+0000 7fe7f6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.579+0000 7fe7f6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.579+0000 7fe7f6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.579+0000 7fe7f6600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.580 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.580+0000 7fe7f8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.580 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.580+0000 7fe7f8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.580 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.580+0000 7fe7f8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.580 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.580+0000 7fe7f8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:07.580 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:31:07.580+0000 7fe7f8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:31:08.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.282+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 49) ==== 90895+0+0 (secure 0 0 0) 0x7faf90080310 con 0x7faf980ce480 2024-04-17T07:31:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.282+0000 7faf8f400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7faf74074910 0x7faf74076d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.282+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 --> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7faf74007ea0 con 0x7faf74074910 2024-04-17T07:31:08.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.291+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7faf74074910 0x7faf74076d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.306+0000 7faf97400640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7faf74074910 0x7faf74076d00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf88007fd0 tx=0x7faf88007d40 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.507+0000 7faf8f400640 1 -- 172.21.15.67:0/1582966838 <== mgr.5855 v2:172.21.15.187:6832/1997545601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7faf74007ea0 con 0x7faf74074910 2024-04-17T07:31:08.507 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /bad-path 2024-04-17T07:31:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7faf74074910 msgr2=0x7faf74076d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7faf74074910 0x7faf74076d00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf88007fd0 tx=0x7faf88007d40 comp rx=0 tx=0).stop 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 msgr2=0x7faf980cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 --2- 172.21.15.67:0/1582966838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf980ce480 0x7faf980cb4d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7faf90036040 tx=0x7faf90004a40 comp rx=0 tx=0).stop 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf97e00640 1 -- 172.21.15.67:0/1582966838 reap_dead start 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 shutdown_connections 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 >> 172.21.15.67:0/1582966838 conn(0x7faf980b4050 msgr2=0x7faf980cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.510+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 shutdown_connections 2024-04-17T07:31:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.511+0000 7faf9ca00640 1 -- 172.21.15.67:0/1582966838 wait complete. 2024-04-17T07:31:08.533 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-04-17T07:31:08.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-17T07:31:08.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.757+0000 7fa53f200640 1 Processor -- start 2024-04-17T07:31:08.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.757+0000 7fa53f200640 1 -- start start 2024-04-17T07:31:08.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa540145b00 0x7fa540160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa540160920 con 0x7fa5400b4ac0 2024-04-17T07:31:08.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa540160c00 con 0x7fa540146060 2024-04-17T07:31:08.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa540160ee0 con 0x7fa540145b00 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa5400b4ac0 0x7fa5400b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47804/0 (socket says 172.21.15.67:47804) 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53e800640 1 -- 172.21.15.67:0/2682373926 learned_addr learned my addr 172.21.15.67:0/2682373926 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.759+0000 7fa53de00640 1 --2- 172.21.15.67:0/2682373926 >> v2:172.21.15.67:3300/0 conn(0x7fa540145b00 0x7fa540160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.760+0000 7fa53d400640 1 -- 172.21.15.67:0/2682373926 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836067436 0 0) 0x7fa540160920 con 0x7fa5400b4ac0 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.760+0000 7fa53d400640 1 -- 172.21.15.67:0/2682373926 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa524003520 con 0x7fa5400b4ac0 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.760+0000 7fa53de00640 1 -- 172.21.15.67:0/2682373926 >> v1:172.21.15.187:6789/0 conn(0x7fa540146060 legacy=0x7fa54015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.760+0000 7fa53de00640 1 -- 172.21.15.67:0/2682373926 >> v1:172.21.15.67:6789/0 conn(0x7fa5400b4ac0 legacy=0x7fa5400b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.760+0000 7fa53de00640 1 -- 172.21.15.67:0/2682373926 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5401611c0 con 0x7fa540145b00 2024-04-17T07:31:08.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.761+0000 7fa53de00640 1 --2- 172.21.15.67:0/2682373926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540145b00 0x7fa540160340 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fa5300097a0 tx=0x7fa53002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c032e43b19837fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:08.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.762+0000 7fa53d400640 1 -- 172.21.15.67:0/2682373926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530035020 con 0x7fa540145b00 2024-04-17T07:31:08.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.762+0000 7fa53d400640 1 -- 172.21.15.67:0/2682373926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa530031830 con 0x7fa540145b00 2024-04-17T07:31:08.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.762+0000 7fa53d400640 1 -- 172.21.15.67:0/2682373926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530031be0 con 0x7fa540145b00 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.762+0000 7fa53f200640 1 -- 172.21.15.67:0/2682373926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540145b00 msgr2=0x7fa540160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.762+0000 7fa53f200640 1 --2- 172.21.15.67:0/2682373926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540145b00 0x7fa540160340 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fa5300097a0 tx=0x7fa53002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.763+0000 7fa53f200640 1 -- 172.21.15.67:0/2682373926 shutdown_connections 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.763+0000 7fa53f200640 1 --2- 172.21.15.67:0/2682373926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540145b00 0x7fa540160340 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.763+0000 7fa53f200640 1 -- 172.21.15.67:0/2682373926 >> 172.21.15.67:0/2682373926 conn(0x7fa5400b23c0 msgr2=0x7fa5400b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.763+0000 7fa53f200640 1 -- 172.21.15.67:0/2682373926 shutdown_connections 2024-04-17T07:31:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.763+0000 7fa53f200640 1 -- 172.21.15.67:0/2682373926 wait complete. 2024-04-17T07:31:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.765+0000 7fa53f200640 1 Processor -- start 2024-04-17T07:31:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.765+0000 7fa53f200640 1 -- start start 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5400b4ac0 0x7fa54015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa540145b00 0x7fa54006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5400b4f10 con 0x7fa540146060 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa5400cfba0 con 0x7fa5400b4ac0 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.766+0000 7fa53f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5400b5090 con 0x7fa540145b00 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49582/0 (socket says 172.21.15.67:49582) 2024-04-17T07:31:08.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 -- 172.21.15.67:0/577042571 learned_addr learned my addr 172.21.15.67:0/577042571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:31:08.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 -- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa540145b00 msgr2=0x7fa54006ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa540145b00 0x7fa54006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53de00640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5400b4ac0 0x7fa54015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 -- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5400b4ac0 msgr2=0x7fa54015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5400b4ac0 0x7fa54015bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 -- 172.21.15.67:0/577042571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54015c530 con 0x7fa540146060 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.767+0000 7fa53e800640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa53800cb70 tx=0x7fa53800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.768+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538017070 con 0x7fa540146060 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.768+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53800ad40 con 0x7fa540146060 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.768+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538012620 con 0x7fa540146060 2024-04-17T07:31:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.768+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa54013c020 con 0x7fa540146060 2024-04-17T07:31:08.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.768+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa54013c5e0 con 0x7fa540146060 2024-04-17T07:31:08.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.769+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa540068a10 con 0x7fa540146060 2024-04-17T07:31:08.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.770+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 49) ==== 90895+0+0 (secure 0 0 0) 0x7fa53801a070 con 0x7fa540146060 2024-04-17T07:31:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.771+0000 7fa536000640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 0x7fa51c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:31:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.771+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 6053+0+0 (secure 0 0 0) 0x7fa53808b770 con 0x7fa540146060 2024-04-17T07:31:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.772+0000 7fa53de00640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 0x7fa51c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:31:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.773+0000 7fa53de00640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 0x7fa51c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa530036000 tx=0x7fa530008030 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:31:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:08.776+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa538059d80 con 0x7fa540146060 2024-04-17T07:31:09.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.008+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 --> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa540068610 con 0x7fa51c06c790 2024-04-17T07:31:09.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.024+0000 7fa536000640 1 -- 172.21.15.67:0/577042571 <== mgr.5855 v2:172.21.15.187:6832/1997545601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7fa540068610 con 0x7fa51c06c790 2024-04-17T07:31:09.025 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:31:09.026 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:31:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 msgr2=0x7fa51c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 0x7fa51c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa530036000 tx=0x7fa530008030 comp rx=0 tx=0).stop 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 msgr2=0x7fa54006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fa53800cb70 tx=0x7fa53800a430 comp rx=0 tx=0).stop 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 shutdown_connections 2024-04-17T07:31:09.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa51c06c790 0x7fa51c06ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa540146060 0x7fa54006f140 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa540145b00 0x7fa54006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 --2- 172.21.15.67:0/577042571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5400b4ac0 0x7fa54015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 >> 172.21.15.67:0/577042571 conn(0x7fa5400b23c0 msgr2=0x7fa540071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 shutdown_connections 2024-04-17T07:31:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:31:09.027+0000 7fa53f200640 1 -- 172.21.15.67:0/577042571 wait complete. 2024-04-17T07:31:09.049 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:31:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-17T07:31:28.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:31:28.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:31:58.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:31:58.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:32:00.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:00.535+0000 7fe7e5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:00.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:00.535+0000 7fe7e5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:00.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:00.535+0000 7fe7e5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:00.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:00.535+0000 7fe7e5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:00.535 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:00.535+0000 7fe7e5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:09.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-04-17T07:32:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.284+0000 7f2068200640 1 Processor -- start 2024-04-17T07:32:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.284+0000 7f2068200640 1 -- start start 2024-04-17T07:32:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2068200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20600b4fa0 0x7f206015be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2068200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2068200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20600b5560 con 0x7f20600b17c0 2024-04-17T07:32:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2068200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20600ce920 con 0x7f20600b4fa0 2024-04-17T07:32:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2068200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20600b5840 con 0x7f20600cfd00 2024-04-17T07:32:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.286+0000 7f2066200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.287+0000 7f2066200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:42794/0 (socket says 172.21.15.67:42794) 2024-04-17T07:32:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.287+0000 7f2066200640 1 -- 172.21.15.67:0/2881288198 learned_addr learned my addr 172.21.15.67:0/2881288198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.287+0000 7f2065800640 1 --1- 172.21.15.67:0/2881288198 >> v1:172.21.15.187:6789/0 conn(0x7f20600b17c0 0x7f20600b1ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:54846/0 (socket says 172.21.15.67:54846) 2024-04-17T07:32:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.287+0000 7f2064e00640 1 --2- 172.21.15.67:0/2881288198 >> v2:172.21.15.187:3300/0 conn(0x7f20600b4fa0 0x7f206015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.288+0000 7f2066200640 1 -- 172.21.15.67:0/2881288198 >> v2:172.21.15.187:3300/0 conn(0x7f20600b4fa0 msgr2=0x7f206015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.288+0000 7f2066200640 1 --2- 172.21.15.67:0/2881288198 >> v2:172.21.15.187:3300/0 conn(0x7f20600b4fa0 0x7f206015be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.288+0000 7f205fe00640 1 -- 172.21.15.67:0/2881288198 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566387157 0 0) 0x7f20600b5560 con 0x7f20600b17c0 2024-04-17T07:32:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.288+0000 7f2066200640 1 -- 172.21.15.67:0/2881288198 >> v1:172.21.15.187:6789/0 conn(0x7f20600b17c0 legacy=0x7f20600b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.288+0000 7f2066200640 1 -- 172.21.15.67:0/2881288198 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20600ce5b0 con 0x7f20600cfd00 2024-04-17T07:32:09.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.289+0000 7f2066200640 1 --2- 172.21.15.67:0/2881288198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f205800bb10 tx=0x7f205802e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58cca59a63e91b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f205fe00640 1 -- 172.21.15.67:0/2881288198 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205803b070 con 0x7f20600cfd00 2024-04-17T07:32:09.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f205fe00640 1 -- 172.21.15.67:0/2881288198 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2058036440 con 0x7f20600cfd00 2024-04-17T07:32:09.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f205fe00640 1 -- 172.21.15.67:0/2881288198 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058036810 con 0x7f20600cfd00 2024-04-17T07:32:09.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 -- 172.21.15.67:0/2881288198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 msgr2=0x7f20600cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 --2- 172.21.15.67:0/2881288198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f205800bb10 tx=0x7f205802e150 comp rx=0 tx=0).stop 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 -- 172.21.15.67:0/2881288198 shutdown_connections 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 --2- 172.21.15.67:0/2881288198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600cfd00 0x7f20600cedb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 --2- 172.21.15.67:0/2881288198 >> v2:172.21.15.187:3300/0 conn(0x7f20600b4fa0 0x7f206015be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.290+0000 7f2068200640 1 -- 172.21.15.67:0/2881288198 >> 172.21.15.67:0/2881288198 conn(0x7f206006b8c0 msgr2=0x7f206006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.291+0000 7f2068200640 1 -- 172.21.15.67:0/2881288198 shutdown_connections 2024-04-17T07:32:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.291+0000 7f2068200640 1 -- 172.21.15.67:0/2881288198 wait complete. 2024-04-17T07:32:09.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.292+0000 7f2068200640 1 Processor -- start 2024-04-17T07:32:09.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.292+0000 7f2068200640 1 -- start start 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20600b17c0 0x7f2060138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600b4fa0 0x7f206015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2060145490 con 0x7f20600cfd00 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20600b5b20 con 0x7f20600b17c0 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.293+0000 7f2068200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2060145610 con 0x7f20600b4fa0 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59604/0 (socket says 172.21.15.67:59604) 2024-04-17T07:32:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 -- 172.21.15.67:0/3597735298 learned_addr learned my addr 172.21.15.67:0/3597735298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:09.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2065800640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20600b17c0 0x7f2060138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2064e00640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600b4fa0 0x7f206015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 -- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600b4fa0 msgr2=0x7f206015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600b4fa0 0x7f206015ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 -- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20600b17c0 msgr2=0x7f2060138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20600b17c0 0x7f2060138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.294+0000 7f2066200640 1 -- 172.21.15.67:0/3597735298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2060133ee0 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f2066200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f205800b2c0 tx=0x7f20580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205803b070 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f205803d050 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058034a90 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20601341c0 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.295+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2060134780 con 0x7f20600cfd00 2024-04-17T07:32:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.296+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2060068b90 con 0x7f20600cfd00 2024-04-17T07:32:09.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.297+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 52) ==== 91273+0+0 (secure 0 0 0) 0x7f2058048060 con 0x7f20600cfd00 2024-04-17T07:32:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.298+0000 7f205d600640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 0x7f204006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.298+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 6082+0+0 (secure 0 0 0) 0x7f20580add90 con 0x7f20600cfd00 2024-04-17T07:32:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.299+0000 7f2065800640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 0x7f204006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.300+0000 7f2065800640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 0x7f204006f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20540012e0 tx=0x7f2054007ef0 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.303+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f205807b680 con 0x7f20600cfd00 2024-04-17T07:32:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.530+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 --> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f206013d640 con 0x7f204006cb80 2024-04-17T07:32:09.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.547+0000 7f205d600640 1 -- 172.21.15.67:0/3597735298 <== mgr.5855 v2:172.21.15.187:6832/1997545601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f206013d640 con 0x7f204006cb80 2024-04-17T07:32:09.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:09.547 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:31:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-17T07:32:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 msgr2=0x7f204006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 0x7f204006f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f20540012e0 tx=0x7f2054007ef0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 msgr2=0x7f206015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f205800b2c0 tx=0x7f20580046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 shutdown_connections 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f204006cb80 0x7f204006f040 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20600cfd00 0x7f206015b370 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20600b4fa0 0x7f206015ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 --2- 172.21.15.67:0/3597735298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20600b17c0 0x7f2060138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 >> 172.21.15.67:0/3597735298 conn(0x7f206006b8c0 msgr2=0x7f206006fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 shutdown_connections 2024-04-17T07:32:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.550+0000 7f2068200640 1 -- 172.21.15.67:0/3597735298 wait complete. 2024-04-17T07:32:09.570 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:31:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-04-17T07:32:09.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-04-17T07:32:09.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.793+0000 7f0bc6e00640 1 Processor -- start 2024-04-17T07:32:09.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.793+0000 7f0bc6e00640 1 -- start start 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.794+0000 7f0bc6e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc006d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.794+0000 7f0bc6e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0bc00cf400 0x7f0bc00cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.794+0000 7f0bc6e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bc00d0070 con 0x7f0bc015e8a0 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.794+0000 7f0bc6e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc00d2310 con 0x7f0bc00cf400 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.794+0000 7f0bc6e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc00d0350 con 0x7f0bc00d4de0 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bc5a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0bc015e8a0 0x7f0bc00d46d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43696/0 (socket says 172.21.15.67:43696) 2024-04-17T07:32:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bc5a00640 1 -- 172.21.15.67:0/18694052 learned_addr learned my addr 172.21.15.67:0/18694052 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bc6400640 1 --2- 172.21.15.67:0/18694052 >> v2:172.21.15.67:3300/0 conn(0x7f0bc00cf400 0x7f0bc00cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bc5000640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bbfe00640 1 -- 172.21.15.67:0/18694052 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257562443 0 0) 0x7f0bc00d0070 con 0x7f0bc015e8a0 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.795+0000 7f0bbfe00640 1 -- 172.21.15.67:0/18694052 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ba4003520 con 0x7f0bc015e8a0 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.796+0000 7f0bc6400640 1 -- 172.21.15.67:0/18694052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 msgr2=0x7f0bc006d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.796+0000 7f0bc6400640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc006d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.796+0000 7f0bc6400640 1 -- 172.21.15.67:0/18694052 >> v1:172.21.15.67:6789/0 conn(0x7f0bc015e8a0 legacy=0x7f0bc00d46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.796+0000 7f0bc6400640 1 -- 172.21.15.67:0/18694052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc00d0630 con 0x7f0bc00cf400 2024-04-17T07:32:09.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.797+0000 7f0bc6400640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc00cf400 0x7f0bc00cf850 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80097a0 tx=0x7f0bb802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15eef0939c66ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.797+0000 7f0bbfe00640 1 -- 172.21.15.67:0/18694052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80315a0 con 0x7f0bc00cf400 2024-04-17T07:32:09.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.797+0000 7f0bbfe00640 1 -- 172.21.15.67:0/18694052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bb8037070 con 0x7f0bc00cf400 2024-04-17T07:32:09.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.797+0000 7f0bbfe00640 1 -- 172.21.15.67:0/18694052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8031960 con 0x7f0bc00cf400 2024-04-17T07:32:09.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 -- 172.21.15.67:0/18694052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc00cf400 msgr2=0x7f0bc00cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc00cf400 0x7f0bc00cf850 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80097a0 tx=0x7f0bb802b840 comp rx=0 tx=0).stop 2024-04-17T07:32:09.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 -- 172.21.15.67:0/18694052 shutdown_connections 2024-04-17T07:32:09.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc00cf400 0x7f0bc00cf850 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/18694052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc006d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 -- 172.21.15.67:0/18694052 >> 172.21.15.67:0/18694052 conn(0x7f0bc009da50 msgr2=0x7f0bc008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:09.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 -- 172.21.15.67:0/18694052 shutdown_connections 2024-04-17T07:32:09.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.798+0000 7f0bc6e00640 1 -- 172.21.15.67:0/18694052 wait complete. 2024-04-17T07:32:09.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.800+0000 7f0bc6e00640 1 Processor -- start 2024-04-17T07:32:09.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.800+0000 7f0bc6e00640 1 -- start start 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bc00cf400 0x7f0bc01f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc0159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc00b3cc0 con 0x7f0bc015e8a0 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc00cfd90 con 0x7f0bc00cf400 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc015ec80 con 0x7f0bc00d4de0 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59612/0 (socket says 172.21.15.67:59612) 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.801+0000 7f0bc6400640 1 -- 172.21.15.67:0/2972424055 learned_addr learned my addr 172.21.15.67:0/2972424055 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc5a00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bc00cf400 0x7f0bc01f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc5000640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc0159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 -- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 msgr2=0x7f0bc0159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc0159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 -- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bc00cf400 msgr2=0x7f0bc01f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bc00cf400 0x7f0bc01f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 -- 172.21.15.67:0/2972424055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc0160e80 con 0x7f0bc015e8a0 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.802+0000 7f0bc6400640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80098c0 tx=0x7f0bb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.803+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8032630 con 0x7f0bc015e8a0 2024-04-17T07:32:09.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.803+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bb80327f0 con 0x7f0bc015e8a0 2024-04-17T07:32:09.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.803+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8032be0 con 0x7f0bc015e8a0 2024-04-17T07:32:09.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.803+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc015a510 con 0x7f0bc015e8a0 2024-04-17T07:32:09.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.803+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc015aad0 con 0x7f0bc015e8a0 2024-04-17T07:32:09.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.805+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc0162260 con 0x7f0bc015e8a0 2024-04-17T07:32:09.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.806+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 52) ==== 91273+0+0 (secure 0 0 0) 0x7f0bb804c020 con 0x7f0bc015e8a0 2024-04-17T07:32:09.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.807+0000 7f0bbd600640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 0x7f0ba006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:09.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.807+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f0bb802f020 con 0x7f0bc015e8a0 2024-04-17T07:32:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.810+0000 7f0bc5a00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 0x7f0ba006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.811+0000 7f0bc5a00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 0x7f0ba006f070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0005c00 tx=0x7f0bb0009040 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:09.811+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0bc0162260 con 0x7f0bc015e8a0 2024-04-17T07:32:10.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.039+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 --> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) -- 0x7f0bc006ccb0 con 0x7f0ba006cbb0 2024-04-17T07:32:10.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.249+0000 7f0bbd600640 1 -- 172.21.15.67:0/2972424055 <== mgr.5855 v2:172.21.15.187:6832/1997545601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7f0bc006ccb0 con 0x7f0ba006cbb0 2024-04-17T07:32:10.250 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /bad-path 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.252+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 msgr2=0x7f0ba006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.252+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 0x7f0ba006f070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0005c00 tx=0x7f0bb0009040 comp rx=0 tx=0).stop 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 msgr2=0x7f0bc0161640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80098c0 tx=0x7f0bb80046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 shutdown_connections 2024-04-17T07:32:10.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f0ba006cbb0 0x7f0ba006f070 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc015e8a0 0x7f0bc0161640 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0bc00d4de0 0x7f0bc0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 --2- 172.21.15.67:0/2972424055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bc00cf400 0x7f0bc01f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 >> 172.21.15.67:0/2972424055 conn(0x7f0bc009da50 msgr2=0x7f0bc00b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 shutdown_connections 2024-04-17T07:32:10.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.253+0000 7f0bc6e00640 1 -- 172.21.15.67:0/2972424055 wait complete. 2024-04-17T07:32:10.277 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-04-17T07:32:10.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:32:10.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.502+0000 7f1e32a00640 1 Processor -- start 2024-04-17T07:32:10.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.502+0000 7f1e32a00640 1 -- start start 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e32a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e2c06c270 0x7f1e2c167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e32a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c06f1d0 con 0x7f1e2c145f80 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e32a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c146360 con 0x7f1e2c0cd920 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e32a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e2c0727a0 con 0x7f1e2c06c270 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e31600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e2c145f80 0x7f1e2c0cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43702/0 (socket says 172.21.15.67:43702) 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e31600640 1 -- 172.21.15.67:0/3623635569 learned_addr learned my addr 172.21.15.67:0/3623635569 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.504+0000 7f1e32000640 1 --2- 172.21.15.67:0/3623635569 >> v2:172.21.15.67:3300/0 conn(0x7f1e2c06c270 0x7f1e2c167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.505+0000 7f1e32000640 1 -- 172.21.15.67:0/3623635569 >> v1:172.21.15.187:6789/0 conn(0x7f1e2c0cd920 legacy=0x7f1e2c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.505+0000 7f1e32000640 1 -- 172.21.15.67:0/3623635569 >> v1:172.21.15.67:6789/0 conn(0x7f1e2c145f80 legacy=0x7f1e2c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.505+0000 7f1e32000640 1 -- 172.21.15.67:0/3623635569 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c071e70 con 0x7f1e2c06c270 2024-04-17T07:32:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.506+0000 7f1e32000640 1 --2- 172.21.15.67:0/3623635569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c167380 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009d90 tx=0x7f1e2402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5a3f7c36958acc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.506+0000 7f1e2be00640 1 -- 172.21.15.67:0/3623635569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e24039070 con 0x7f1e2c06c270 2024-04-17T07:32:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.506+0000 7f1e2be00640 1 -- 172.21.15.67:0/3623635569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e2402cbd0 con 0x7f1e2c06c270 2024-04-17T07:32:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.506+0000 7f1e2be00640 1 -- 172.21.15.67:0/3623635569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e24034440 con 0x7f1e2c06c270 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 -- 172.21.15.67:0/3623635569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 msgr2=0x7f1e2c167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 --2- 172.21.15.67:0/3623635569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c167380 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009d90 tx=0x7f1e2402be20 comp rx=0 tx=0).stop 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 -- 172.21.15.67:0/3623635569 shutdown_connections 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 --2- 172.21.15.67:0/3623635569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c167380 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 -- 172.21.15.67:0/3623635569 >> 172.21.15.67:0/3623635569 conn(0x7f1e2c0b2190 msgr2=0x7f1e2c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 -- 172.21.15.67:0/3623635569 shutdown_connections 2024-04-17T07:32:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.507+0000 7f1e32a00640 1 -- 172.21.15.67:0/3623635569 wait complete. 2024-04-17T07:32:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.509+0000 7f1e32a00640 1 Processor -- start 2024-04-17T07:32:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.509+0000 7f1e32a00640 1 -- start start 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 0x7f1e2c0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c145f80 0x7f1e2c15eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c072c30 con 0x7f1e2c06c270 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c072430 con 0x7f1e2c0cd920 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e32a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2c072db0 con 0x7f1e2c145f80 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e31600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e31600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59628/0 (socket says 172.21.15.67:59628) 2024-04-17T07:32:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.510+0000 7f1e31600640 1 -- 172.21.15.67:0/783426939 learned_addr learned my addr 172.21.15.67:0/783426939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e32000640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c145f80 0x7f1e2c15eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 -- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c145f80 msgr2=0x7f1e2c15eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c145f80 0x7f1e2c15eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 -- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 msgr2=0x7f1e2c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e30c00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 0x7f1e2c0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 0x7f1e2c0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 -- 172.21.15.67:0/783426939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c1430d0 con 0x7f1e2c06c270 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e30c00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 0x7f1e2c0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e31600640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c3c0 tx=0x7f1e2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20017070 con 0x7f1e2c06c270 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e20005ce0 con 0x7f1e2c06c270 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2c1433b0 con 0x7f1e2c06c270 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.511+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e200072f0 con 0x7f1e2c06c270 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.512+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e2c143970 con 0x7f1e2c06c270 2024-04-17T07:32:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.513+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e2c143b70 con 0x7f1e2c06c270 2024-04-17T07:32:10.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.514+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 52) ==== 91273+0+0 (secure 0 0 0) 0x7f1e2001a070 con 0x7f1e2c06c270 2024-04-17T07:32:10.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.515+0000 7f1e29600640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 0x7f1e0c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:10.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.515+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f1e2008b510 con 0x7f1e2c06c270 2024-04-17T07:32:10.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.515+0000 7f1e30c00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 0x7f1e0c06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:10.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.516+0000 7f1e30c00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 0x7f1e0c06f040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c1e53e0 tx=0x7f1e18009040 comp rx=0 tx=0).ready entity=mgr.5855 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.519+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e2c143b70 con 0x7f1e2c06c270 2024-04-17T07:32:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:10.779+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f1e2c068a10 con 0x7f1e2c06c270 2024-04-17T07:32:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.597+0000 7f1e29600640 1 -- 172.21.15.67:0/783426939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) ==== 93+0+0 (secure 0 0 0) 0x7f1e2c06f1d0 con 0x7f1e2c06c270 2024-04-17T07:32:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.601+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 msgr2=0x7f1e0c06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.601+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 0x7f1e0c06f040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c1e53e0 tx=0x7f1e18009040 comp rx=0 tx=0).stop 2024-04-17T07:32:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.601+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 msgr2=0x7f1e2c15bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:11.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.601+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c3c0 tx=0x7f1e2000c880 comp rx=0 tx=0).stop 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 shutdown_connections 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1e0c06cb80 0x7f1e0c06f040 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c145f80 0x7f1e2c15eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cd920 0x7f1e2c0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 --2- 172.21.15.67:0/783426939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c06c270 0x7f1e2c15bf50 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.602+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 >> 172.21.15.67:0/783426939 conn(0x7f1e2c0b2190 msgr2=0x7f1e2c0af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:11.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.603+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 shutdown_connections 2024-04-17T07:32:11.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:11.603+0000 7f1e32a00640 1 -- 172.21.15.67:0/783426939 wait complete. 2024-04-17T07:32:11.629 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:32:11.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:32:11.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:32:11.824 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:32:11.824 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:32:11.824 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:11.824 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:11.869 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:11.869 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:11.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:12.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.166+0000 7f3125e00640 1 Processor -- start 2024-04-17T07:32:12.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.166+0000 7f3125e00640 1 -- start start 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.167+0000 7f3125e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f312015b570 0x7f312006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.167+0000 7f3125e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006fce0 0x7f312006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.167+0000 7f3125e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f312015bc80 con 0x7f312006be70 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.167+0000 7f3125e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f312015bf40 con 0x7f312015b570 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.167+0000 7f3125e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f312015c220 con 0x7f312006fce0 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f312015b570 0x7f312006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f312015b570 0x7f312006f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59648/0 (socket says 172.21.15.67:59648) 2024-04-17T07:32:12.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 -- 172.21.15.67:0/1345638269 learned_addr learned my addr 172.21.15.67:0/1345638269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 -- 172.21.15.67:0/1345638269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006fce0 msgr2=0x7f312006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 --2- 172.21.15.67:0/1345638269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006fce0 0x7f312006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 -- 172.21.15.67:0/1345638269 >> v1:172.21.15.187:6789/0 conn(0x7f312006be70 legacy=0x7f312006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.168+0000 7f3124a00640 1 -- 172.21.15.67:0/1345638269 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312015c500 con 0x7f312015b570 2024-04-17T07:32:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.169+0000 7f3124a00640 1 --2- 172.21.15.67:0/1345638269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f312006f7a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3114009870 tx=0x7f311402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448a59711bc3fa91 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.170+0000 7f311f400640 1 -- 172.21.15.67:0/1345638269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31140317c0 con 0x7f312015b570 2024-04-17T07:32:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.170+0000 7f311f400640 1 -- 172.21.15.67:0/1345638269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3114037070 con 0x7f312015b570 2024-04-17T07:32:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.170+0000 7f311f400640 1 -- 172.21.15.67:0/1345638269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3114031b80 con 0x7f312015b570 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.170+0000 7f3125e00640 1 -- 172.21.15.67:0/1345638269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 msgr2=0x7f312006f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.170+0000 7f3125e00640 1 --2- 172.21.15.67:0/1345638269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f312006f7a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3114009870 tx=0x7f311402b860 comp rx=0 tx=0).stop 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 -- 172.21.15.67:0/1345638269 shutdown_connections 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 --2- 172.21.15.67:0/1345638269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006fce0 0x7f312006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 --2- 172.21.15.67:0/1345638269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f312006f7a0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 -- 172.21.15.67:0/1345638269 >> 172.21.15.67:0/1345638269 conn(0x7f3120067d10 msgr2=0x7f31200b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 -- 172.21.15.67:0/1345638269 shutdown_connections 2024-04-17T07:32:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.171+0000 7f3125e00640 1 -- 172.21.15.67:0/1345638269 wait complete. 2024-04-17T07:32:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.172+0000 7f3125e00640 1 Processor -- start 2024-04-17T07:32:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.173+0000 7f3125e00640 1 -- start start 2024-04-17T07:32:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.173+0000 7f3125e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006be70 0x7f312014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 0x7f31201f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f312006c460 con 0x7f312015b570 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f312006c2e0 con 0x7f312006fce0 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31200b47b0 con 0x7f312006be70 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59652/0 (socket says 172.21.15.67:59652) 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 -- 172.21.15.67:0/2588253258 learned_addr learned my addr 172.21.15.67:0/2588253258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f311fe00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 0x7f31201f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3124a00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006be70 0x7f312014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006be70 msgr2=0x7f312014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006be70 0x7f312014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 msgr2=0x7f31201f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 0x7f31201f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f3125400640 1 -- 172.21.15.67:0/2588253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31201f4d80 con 0x7f312015b570 2024-04-17T07:32:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.174+0000 7f311fe00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 0x7f31201f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f3125400640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f311800b970 tx=0x7f311800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:12.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118019070 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311800cb40 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31201587f0 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118014440 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.175+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31201595e0 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.177+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3120068a10 con 0x7f312015b570 2024-04-17T07:32:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.177+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f311801c070 con 0x7f312015b570 2024-04-17T07:32:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.178+0000 7f311cc00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 0x7f310006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.178+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f311808be20 con 0x7f312015b570 2024-04-17T07:32:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.178+0000 7f3124a00640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 msgr2=0x7f310006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.179+0000 7f3124a00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 0x7f310006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:12.180 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.180+0000 7f7100c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:32:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.183+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3118059cd0 con 0x7f312015b570 2024-04-17T07:32:12.209 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.209+0000 7f5f8b9b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:32:12.240 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.240+0000 7f7100c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:32:12.267 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.267+0000 7f5f8b9b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:32:12.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.380+0000 7f3124a00640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 msgr2=0x7f310006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:12.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.380+0000 7f3124a00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 0x7f310006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:12.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.390+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f312006f350 con 0x7f312015b570 2024-04-17T07:32:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.392+0000 7f311cc00640 1 -- 172.21.15.67:0/2588253258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+17179 (secure 0 0 0) 0x7f31180595d0 con 0x7f312015b570 2024-04-17T07:32:12.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:12.393 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":114,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:09.570881+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.1a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/403098035":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3476034436":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6833/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6832/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3513519163":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1745494580":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1260936234":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/2471807753":"2024-04-18T07:31:07.255351+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:12.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 msgr2=0x7f310006eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:12.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 0x7f310006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 msgr2=0x7f31201f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f311800b970 tx=0x7f311800be30 comp rx=0 tx=0).stop 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 shutdown_connections 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f310006cb30 0x7f310006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f312015b570 0x7f31201f4470 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f312006fce0 0x7f31201f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 --2- 172.21.15.67:0/2588253258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f312006be70 0x7f312014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 >> 172.21.15.67:0/2588253258 conn(0x7f3120067d10 msgr2=0x7f31200b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 shutdown_connections 2024-04-17T07:32:12.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.395+0000 7f3125e00640 1 -- 172.21.15.67:0/2588253258 wait complete. 2024-04-17T07:32:12.415 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:30:50.351588+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:30:51.385068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:32:12.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:32:12.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.626+0000 7f2d4ec00640 1 Processor -- start 2024-04-17T07:32:12.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.626+0000 7f2d4ec00640 1 -- start start 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.627+0000 7f2d4ec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d4809aa60 0x7f2d4809aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.627+0000 7f2d4ec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d480b3cc0 con 0x7f2d480ccb20 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ec00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d480d0b80 con 0x7f2d480b38e0 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4809b400 con 0x7f2d4809aa60 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d4809aa60 0x7f2d4809aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4e200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2d480ccb20 0x7f2d480ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43714/0 (socket says 172.21.15.67:43714) 2024-04-17T07:32:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4e200640 1 -- 172.21.15.67:0/2096759464 learned_addr learned my addr 172.21.15.67:0/2096759464 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:12.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ce00640 1 -- 172.21.15.67:0/2096759464 >> v1:172.21.15.187:6789/0 conn(0x7f2d480b38e0 legacy=0x7f2d4806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ce00640 1 -- 172.21.15.67:0/2096759464 >> v1:172.21.15.67:6789/0 conn(0x7f2d480ccb20 legacy=0x7f2d480ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.628+0000 7f2d4ce00640 1 -- 172.21.15.67:0/2096759464 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d480d0e60 con 0x7f2d4809aa60 2024-04-17T07:32:12.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.630+0000 7f2d4ce00640 1 --2- 172.21.15.67:0/2096759464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d4809aa60 0x7f2d4809aec0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f2d340097a0 tx=0x7f2d3402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f445931a3666e1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.630+0000 7f2d47e00640 1 -- 172.21.15.67:0/2096759464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34034070 con 0x7f2d4809aa60 2024-04-17T07:32:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.630+0000 7f2d47e00640 1 -- 172.21.15.67:0/2096759464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d34030610 con 0x7f2d4809aa60 2024-04-17T07:32:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.630+0000 7f2d47e00640 1 -- 172.21.15.67:0/2096759464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340309e0 con 0x7f2d4809aa60 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 -- 172.21.15.67:0/2096759464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d4809aa60 msgr2=0x7f2d4809aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/2096759464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d4809aa60 0x7f2d4809aec0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f2d340097a0 tx=0x7f2d3402b960 comp rx=0 tx=0).stop 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 -- 172.21.15.67:0/2096759464 shutdown_connections 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/2096759464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d4809aa60 0x7f2d4809aec0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 -- 172.21.15.67:0/2096759464 >> 172.21.15.67:0/2096759464 conn(0x7f2d480b3fc0 msgr2=0x7f2d480b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 -- 172.21.15.67:0/2096759464 shutdown_connections 2024-04-17T07:32:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.631+0000 7f2d4ec00640 1 -- 172.21.15.67:0/2096759464 wait complete. 2024-04-17T07:32:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.633+0000 7f2d4ec00640 1 Processor -- start 2024-04-17T07:32:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.633+0000 7f2d4ec00640 1 -- start start 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 0x7f2d481558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d480b38e0 0x7f2d4806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d480d1420 con 0x7f2d480ccb20 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d480d1790 con 0x7f2d4809aa60 2024-04-17T07:32:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.634+0000 7f2d4ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4809b9a0 con 0x7f2d480b38e0 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59662/0 (socket says 172.21.15.67:59662) 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 -- 172.21.15.67:0/154549928 learned_addr learned my addr 172.21.15.67:0/154549928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4d800640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 0x7f2d481558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4ce00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d480b38e0 0x7f2d4806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d480b38e0 msgr2=0x7f2d4806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d480b38e0 0x7f2d4806d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 msgr2=0x7f2d481558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 0x7f2d481558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4e200640 1 -- 172.21.15.67:0/154549928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d480ca440 con 0x7f2d480ccb20 2024-04-17T07:32:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.635+0000 7f2d4d800640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 0x7f2d481558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d4e200640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c3a0 tx=0x7f2d4000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40017070 con 0x7f2d480ccb20 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d40005ce0 con 0x7f2d480ccb20 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d400072d0 con 0x7f2d480ccb20 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d480ca720 con 0x7f2d480ccb20 2024-04-17T07:32:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.636+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d480b07b0 con 0x7f2d480ccb20 2024-04-17T07:32:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.637+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d480755d0 con 0x7f2d480ccb20 2024-04-17T07:32:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.638+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f2d4001a070 con 0x7f2d480ccb20 2024-04-17T07:32:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.639+0000 7f2d45600640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 0x7f2d2806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.640+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f2d4008b340 con 0x7f2d480ccb20 2024-04-17T07:32:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.640+0000 7f2d4d800640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 msgr2=0x7f2d2806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.640+0000 7f2d4d800640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 0x7f2d2806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:12.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.648+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2d40058c40 con 0x7f2d480ccb20 2024-04-17T07:32:12.691 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.691+0000 7f7100c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:32:12.742 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.742+0000 7f5f8b9b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:32:12.743 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.742+0000 7f7100c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:32:12.793 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.793+0000 7f7100c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:32:12.797 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.797+0000 7f5f8b9b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:32:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.841+0000 7f2d4d800640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 msgr2=0x7f2d2806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.841+0000 7f2d4d800640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 0x7f2d2806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:12.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.850+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2d480b54e0 con 0x7f2d480ccb20 2024-04-17T07:32:12.851 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.850+0000 7f5f8b9b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:32:12.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.851+0000 7f2d45600640 1 -- 172.21.15.67:0/154549928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) ==== 93+0+10 (secure 0 0 0) 0x7f2d4005cf40 con 0x7f2d480ccb20 2024-04-17T07:32:12.852 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:32:12.853 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.853+0000 7f7100c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 msgr2=0x7f2d2806eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 0x7f2d2806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 msgr2=0x7f2d4806d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c3a0 tx=0x7f2d4000c860 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 shutdown_connections 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2d2806cb30 0x7f2d2806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d480ccb20 0x7f2d4806d920 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d480b38e0 0x7f2d4806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 --2- 172.21.15.67:0/154549928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d4809aa60 0x7f2d481558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 >> 172.21.15.67:0/154549928 conn(0x7f2d480b3fc0 msgr2=0x7f2d480b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 shutdown_connections 2024-04-17T07:32:12.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:12.854+0000 7f2d4ec00640 1 -- 172.21.15.67:0/154549928 wait complete. 2024-04-17T07:32:12.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:32:12.903 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:12.903+0000 7f7100c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:32:12.917 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.917+0000 7f5f8b9b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:32:12.973 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:12.973+0000 7f5f8b9b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:32:13.036 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.036+0000 7f7100c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:32:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.073+0000 7f5aa5a00640 1 Processor -- start 2024-04-17T07:32:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.074+0000 7f5aa5a00640 1 -- start start 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5aa015ff30 0x7f5aa009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aa00d0170 0x7f5aa00af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa00b1970 con 0x7f5aa00b1400 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa009b7e0 con 0x7f5aa00d0170 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa009bac0 con 0x7f5aa015ff30 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.075+0000 7f5aa5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aa00d0170 0x7f5aa00af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aa00d0170 0x7f5aa00af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59670/0 (socket says 172.21.15.67:59670) 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 -- 172.21.15.67:0/3142097995 learned_addr learned my addr 172.21.15.67:0/3142097995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5a9fe00640 1 --2- 172.21.15.67:0/3142097995 >> v2:172.21.15.187:3300/0 conn(0x7f5aa015ff30 0x7f5aa009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 -- 172.21.15.67:0/3142097995 >> v2:172.21.15.187:3300/0 conn(0x7f5aa015ff30 msgr2=0x7f5aa009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 --2- 172.21.15.67:0/3142097995 >> v2:172.21.15.187:3300/0 conn(0x7f5aa015ff30 0x7f5aa009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 -- 172.21.15.67:0/3142097995 >> v1:172.21.15.187:6789/0 conn(0x7f5aa00b1400 legacy=0x7f5aa009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.076+0000 7f5aa5000640 1 -- 172.21.15.67:0/3142097995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa0160310 con 0x7f5aa00d0170 2024-04-17T07:32:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.077+0000 7f5aa5000640 1 --2- 172.21.15.67:0/3142097995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00d0170 0x7f5aa00af450 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0072e50 tx=0x7f5a9802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2a35e49bc92b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.077+0000 7f5a9ea00640 1 -- 172.21.15.67:0/3142097995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9802c960 con 0x7f5aa00d0170 2024-04-17T07:32:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5a9ea00640 1 -- 172.21.15.67:0/3142097995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9802cb00 con 0x7f5aa00d0170 2024-04-17T07:32:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5a9ea00640 1 -- 172.21.15.67:0/3142097995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a980343f0 con 0x7f5aa00d0170 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5aa5a00640 1 -- 172.21.15.67:0/3142097995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00d0170 msgr2=0x7f5aa00af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/3142097995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00d0170 0x7f5aa00af450 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0072e50 tx=0x7f5a9802bdc0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5aa5a00640 1 -- 172.21.15.67:0/3142097995 shutdown_connections 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.078+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/3142097995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00d0170 0x7f5aa00af450 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.079+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/3142097995 >> v2:172.21.15.187:3300/0 conn(0x7f5aa015ff30 0x7f5aa009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.079+0000 7f5aa5a00640 1 -- 172.21.15.67:0/3142097995 >> 172.21.15.67:0/3142097995 conn(0x7f5aa00b4090 msgr2=0x7f5aa00b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.079+0000 7f5aa5a00640 1 -- 172.21.15.67:0/3142097995 shutdown_connections 2024-04-17T07:32:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.079+0000 7f5aa5a00640 1 -- 172.21.15.67:0/3142097995 wait complete. 2024-04-17T07:32:13.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.081+0000 7f5aa5a00640 1 Processor -- start 2024-04-17T07:32:13.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.081+0000 7f5aa5a00640 1 -- start start 2024-04-17T07:32:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa00d0170 0x7f5aa00c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aa015ff30 0x7f5aa00c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa00afb10 con 0x7f5aa00b1400 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa00af990 con 0x7f5aa00d0170 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5aa5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa00afc90 con 0x7f5aa015ff30 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59672/0 (socket says 172.21.15.67:59672) 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.083+0000 7f5a9fe00640 1 -- 172.21.15.67:0/648257444 learned_addr learned my addr 172.21.15.67:0/648257444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9f400640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa00d0170 0x7f5aa00c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aa015ff30 msgr2=0x7f5aa00c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aa015ff30 0x7f5aa00c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa00d0170 msgr2=0x7f5aa00c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa00d0170 0x7f5aa00c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 -- 172.21.15.67:0/648257444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa00b7320 con 0x7f5aa00b1400 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a9fe00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000d5f0 tx=0x7f5a9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a900125f0 con 0x7f5aa00b1400 2024-04-17T07:32:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a90019070 con 0x7f5aa00b1400 2024-04-17T07:32:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a900129d0 con 0x7f5aa00b1400 2024-04-17T07:32:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa00b7600 con 0x7f5aa00b1400 2024-04-17T07:32:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.084+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa00c9790 con 0x7f5aa00b1400 2024-04-17T07:32:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.086+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa0068a10 con 0x7f5aa00b1400 2024-04-17T07:32:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.087+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f5a90012b70 con 0x7f5aa00b1400 2024-04-17T07:32:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.088+0000 7f5a97e00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 0x7f5a8006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.089+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f5a9008c510 con 0x7f5aa00b1400 2024-04-17T07:32:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.089+0000 7f5a9f400640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 msgr2=0x7f5a8006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.089+0000 7f5a9f400640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 0x7f5a8006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.093+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5a9005a460 con 0x7f5aa00b1400 2024-04-17T07:32:13.100 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.100+0000 7f7100c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:13.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.117+0000 7f5f8b9b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:32:13.150 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.150+0000 7f7100c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:32:13.184 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.184+0000 7f5f8b9b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:13.207 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.207+0000 7f7100c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:32:13.236 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.236+0000 7f5f8b9b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:32:13.258 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.258+0000 7f7100c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:32:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.290+0000 7f5a9f400640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 msgr2=0x7f5a8006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.290+0000 7f5a9f400640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 0x7f5a8006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:13.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.293+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5aa00b4610 con 0x7f5aa00b1400 2024-04-17T07:32:13.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.293+0000 7f5a97e00640 1 -- 172.21.15.67:0/648257444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) ==== 104+0+11 (secure 0 0 0) 0x7f5a90059d60 con 0x7f5aa00b1400 2024-04-17T07:32:13.294 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 msgr2=0x7f5a8006eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 0x7f5a8006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 msgr2=0x7f5aa014bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000d5f0 tx=0x7f5a9000dab0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 shutdown_connections 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f5a8006cb30 0x7f5a8006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aa015ff30 0x7f5aa00c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa00d0170 0x7f5aa00c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 --2- 172.21.15.67:0/648257444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aa00b1400 0x7f5aa014bad0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 >> 172.21.15.67:0/648257444 conn(0x7f5aa00b4090 msgr2=0x7f5aa0070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.296+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 shutdown_connections 2024-04-17T07:32:13.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.297+0000 7f5aa5a00640 1 -- 172.21.15.67:0/648257444 wait complete. 2024-04-17T07:32:13.298 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.297+0000 7f5f8b9b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:32:13.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:32:13.351 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.351+0000 7f5f8b9b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:32:13.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.571+0000 7f2e5ce00640 1 Processor -- start 2024-04-17T07:32:13.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.571+0000 7f2e5ce00640 1 -- start start 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.573+0000 7f2e5ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e580b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.573+0000 7f2e5ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e580b5110 con 0x7f2e581622b0 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.573+0000 7f2e5ce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e580b53f0 con 0x7f2e580b4c70 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.573+0000 7f2e5ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2e580b56d0 con 0x7f2e580d2ff0 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e57e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2e581622b0 0x7f2e580cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54248/0 (socket says 172.21.15.67:54248) 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e57e00640 1 -- 172.21.15.67:0/409014023 learned_addr learned my addr 172.21.15.67:0/409014023 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e57400640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e580b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2876370472 0 0) 0x7f2e580b5110 con 0x7f2e581622b0 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e3c003520 con 0x7f2e581622b0 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456760030 0 0) 0x7f2e580b53f0 con 0x7f2e580b4c70 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e580b5110 con 0x7f2e580b4c70 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.574+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 307971671 0 0) 0x7f2e3c003520 con 0x7f2e581622b0 2024-04-17T07:32:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 msgr2=0x7f2e580b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e580b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 >> v1:172.21.15.187:6789/0 conn(0x7f2e580b4c70 legacy=0x7f2e58161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e580cf340 con 0x7f2e581622b0 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2e500019f0 con 0x7f2e581622b0 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e57400640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e580b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 >> v1:172.21.15.67:6789/0 conn(0x7f2e581622b0 legacy=0x7f2e580cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e3c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e3c0040b0 0x7f2e3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e580b56d0 con 0x7f2e3c004d40 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c00ba70 con 0x7f2e3c0040b0 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3c00bc70 con 0x7f2e580d2ff0 2024-04-17T07:32:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e57e00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e4ea00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e3c0040b0 0x7f2e3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.575+0000 7f2e57400640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e3c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.576+0000 7f2e57e00640 1 -- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 msgr2=0x7f2e3c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.576+0000 7f2e57e00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e580d2ff0 0x7f2e3c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.576+0000 7f2e57e00640 1 -- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e3c0040b0 msgr2=0x7f2e3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.576+0000 7f2e57e00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e3c0040b0 0x7f2e3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.576+0000 7f2e57e00640 1 -- 172.21.15.67:0/409014023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3c003520 con 0x7f2e3c004d40 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.577+0000 7f2e57e00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e3c0051c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000a7d0 tx=0x7f2e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225374cda0e59269 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000a430 con 0x7f2e3c004d40 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e50034630 con 0x7f2e3c004d40 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e56a00640 1 -- 172.21.15.67:0/409014023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50034a20 con 0x7f2e3c004d40 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e5ce00640 1 -- 172.21.15.67:0/409014023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 msgr2=0x7f2e3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/409014023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e3c0051c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000a7d0 tx=0x7f2e3c005830 comp rx=0 tx=0).stop 2024-04-17T07:32:13.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e57e00640 1 -- 172.21.15.67:0/409014023 reap_dead start 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e5ce00640 1 -- 172.21.15.67:0/409014023 shutdown_connections 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.578+0000 7f2e5ce00640 1 -- 172.21.15.67:0/409014023 >> 172.21.15.67:0/409014023 conn(0x7f2e5809da80 msgr2=0x7f2e5808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.579+0000 7f2e5ce00640 1 -- 172.21.15.67:0/409014023 shutdown_connections 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.579+0000 7f2e5ce00640 1 -- 172.21.15.67:0/409014023 wait complete. 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.580+0000 7f2e5ce00640 1 Processor -- start 2024-04-17T07:32:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.580+0000 7f2e5ce00640 1 -- start start 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e58150e90 0x7f2e581f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e581f57c0 0x7f2e5815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e580d3550 con 0x7f2e3c004d40 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2e580d33d0 con 0x7f2e581f57c0 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.581+0000 7f2e5ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2e580d36d0 con 0x7f2e58150e90 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33486/0 (socket says 172.21.15.67:33486) 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 -- 172.21.15.67:0/782095073 learned_addr learned my addr 172.21.15.67:0/782095073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57e00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e581f57c0 0x7f2e5815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e4ea00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e58150e90 0x7f2e581f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e58150e90 msgr2=0x7f2e581f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e58150e90 0x7f2e581f45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e581f57c0 msgr2=0x7f2e5815f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e581f57c0 0x7f2e5815f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 -- 172.21.15.67:0/782095073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e581f4e70 con 0x7f2e3c004d40 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.582+0000 7f2e57400640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2e48011a30 tx=0x7f2e48011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.583+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4800fbb0 con 0x7f2e3c004d40 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.583+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4800fd50 con 0x7f2e3c004d40 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.583+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48014680 con 0x7f2e3c004d40 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.583+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e581f5150 con 0x7f2e3c004d40 2024-04-17T07:32:13.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.583+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e580c84e0 con 0x7f2e3c004d40 2024-04-17T07:32:13.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.584+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e580755d0 con 0x7f2e3c004d40 2024-04-17T07:32:13.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.585+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f2e48012040 con 0x7f2e3c004d40 2024-04-17T07:32:13.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.586+0000 7f2e4fe00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 0x7f2e3406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:13.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.586+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f2e4808c810 con 0x7f2e3c004d40 2024-04-17T07:32:13.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.586+0000 7f2e4ea00640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 msgr2=0x7f2e3406f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:13.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.586+0000 7f2e4ea00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 0x7f2e3406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:13.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.591+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2e48019030 con 0x7f2e3c004d40 2024-04-17T07:32:13.642 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.642+0000 7f7100c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:32:13.764 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.764+0000 7f5f8b9b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:32:13.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.787+0000 7f2e4ea00640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 msgr2=0x7f2e3406f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:13.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.787+0000 7f2e4ea00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 0x7f2e3406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.795+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e580b34e0 con 0x7f2e3c004d40 2024-04-17T07:32:13.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.796+0000 7f2e4fe00640 1 -- 172.21.15.67:0/782095073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) ==== 100+0+11 (secure 0 0 0) 0x7f2e4805a750 con 0x7f2e3c004d40 2024-04-17T07:32:13.796 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 msgr2=0x7f2e3406f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 0x7f2e3406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 msgr2=0x7f2e580cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2e48011a30 tx=0x7f2e48011ef0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 shutdown_connections 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f2e3406cb60 0x7f2e3406f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e581f57c0 0x7f2e5815f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e58150e90 0x7f2e581f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 --2- 172.21.15.67:0/782095073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e3c004d40 0x7f2e580cdac0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:13.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 >> 172.21.15.67:0/782095073 conn(0x7f2e5809da80 msgr2=0x7f2e580d1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:13.800 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.800+0000 7f7100c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:32:13.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 shutdown_connections 2024-04-17T07:32:13.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:13.799+0000 7f2e5ce00640 1 -- 172.21.15.67:0/782095073 wait complete. 2024-04-17T07:32:13.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:32:13.909 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.909+0000 7f7100c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:32:13.936 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:13.936+0000 7f5f8b9b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:32:13.959 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:13.959+0000 7f7100c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:32:14.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.024+0000 7fd3a7200640 1 Processor -- start 2024-04-17T07:32:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.024+0000 7fd3a7200640 1 -- start start 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3a815ee20 0x7fd3a80d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a80d4140 con 0x7fd3a80d18e0 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a80d3b80 con 0x7fd3a80d3520 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a80b83b0 con 0x7fd3a815ee20 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd39fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd3a80d18e0 0x7fd3a80d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54260/0 (socket says 172.21.15.67:54260) 2024-04-17T07:32:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd39fe00640 1 -- 172.21.15.67:0/636060483 learned_addr learned my addr 172.21.15.67:0/636060483 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a5e00640 1 --2- 172.21.15.67:0/636060483 >> v2:172.21.15.187:3300/0 conn(0x7fd3a815ee20 0x7fd3a80d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.026+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781048296 0 0) 0x7fd3a80d4140 con 0x7fd3a80d18e0 2024-04-17T07:32:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd38c003520 con 0x7fd3a80d18e0 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1271639093 0 0) 0x7fd38c003520 con 0x7fd3a80d18e0 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 >> v2:172.21.15.187:3300/0 conn(0x7fd3a815ee20 msgr2=0x7fd3a80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 --2- 172.21.15.67:0/636060483 >> v2:172.21.15.187:3300/0 conn(0x7fd3a815ee20 0x7fd3a80d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 >> v1:172.21.15.187:6789/0 conn(0x7fd3a80d3520 legacy=0x7fd3a80b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a80d3e60 con 0x7fd3a80d18e0 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd390001670 con 0x7fd3a80d18e0 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 >> v1:172.21.15.67:6789/0 conn(0x7fd3a80d18e0 legacy=0x7fd3a80d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c003a70 0x7fd38c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd38c005210 0x7fd38c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a80b83b0 con 0x7fd38c004800 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c00c140 con 0x7fd38c005210 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd38c00c340 con 0x7fd38c003a70 2024-04-17T07:32:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.027+0000 7fd39fe00640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd3a6800640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd38c005210 0x7fd38c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd3a5e00640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c003a70 0x7fd38c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd39fe00640 1 -- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c003a70 msgr2=0x7fd38c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd39fe00640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c003a70 0x7fd38c003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd39fe00640 1 -- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd38c005210 msgr2=0x7fd38c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd39fe00640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd38c005210 0x7fd38c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd39fe00640 1 -- 172.21.15.67:0/636060483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c003520 con 0x7fd38c004800 2024-04-17T07:32:14.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.028+0000 7fd3a6800640 1 -- 172.21.15.67:0/636060483 reap_dead start 2024-04-17T07:32:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.029+0000 7fd39fe00640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c190 tx=0x7fd38c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771d7e438624cc70 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.029+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390007c60 con 0x7fd38c004800 2024-04-17T07:32:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.029+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd390030070 con 0x7fd38c004800 2024-04-17T07:32:14.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.029+0000 7fd3a5400640 1 -- 172.21.15.67:0/636060483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390005db0 con 0x7fd38c004800 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 -- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 msgr2=0x7fd38c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c190 tx=0x7fd38c005830 comp rx=0 tx=0).stop 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 -- 172.21.15.67:0/636060483 shutdown_connections 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 --2- 172.21.15.67:0/636060483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 -- 172.21.15.67:0/636060483 >> 172.21.15.67:0/636060483 conn(0x7fd3a806d3c0 msgr2=0x7fd3a80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 -- 172.21.15.67:0/636060483 shutdown_connections 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.030+0000 7fd3a7200640 1 -- 172.21.15.67:0/636060483 wait complete. 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.032+0000 7fd3a7200640 1 Processor -- start 2024-04-17T07:32:14.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.032+0000 7fd3a7200640 1 -- start start 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c004800 0x7fd3a814bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a80cdb20 0x7fd3a80cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a815f200 con 0x7fd3a80cbad0 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a80d1d20 con 0x7fd3a80cdb20 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a80d13a0 con 0x7fd38c004800 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33506/0 (socket says 172.21.15.67:33506) 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.033+0000 7fd3a6800640 1 -- 172.21.15.67:0/1415431221 learned_addr learned my addr 172.21.15.67:0/1415431221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:14.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd39fe00640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a80cdb20 0x7fd3a80cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a5e00640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c004800 0x7fd3a814bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c004800 msgr2=0x7fd3a814bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c004800 0x7fd3a814bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a80cdb20 msgr2=0x7fd3a80cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a80cdb20 0x7fd3a80cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 -- 172.21.15.67:0/1415431221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a814c2e0 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.034+0000 7fd3a6800640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80d45b0 tx=0x7fd3a000e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.035+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0019070 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.035+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a0006d60 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.035+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0003b90 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.035+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a80cc840 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.035+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a80cce00 con 0x7fd3a80cbad0 2024-04-17T07:32:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.036+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3a8068a10 con 0x7fd3a80cbad0 2024-04-17T07:32:14.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.037+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7fd3a0005070 con 0x7fd3a80cbad0 2024-04-17T07:32:14.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.038+0000 7fd39e000640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 0x7fd38406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.038+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7fd3a008b580 con 0x7fd3a80cbad0 2024-04-17T07:32:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.039+0000 7fd3a5e00640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 msgr2=0x7fd38406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.039+0000 7fd3a5e00640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 0x7fd38406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:14.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.043+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3a814c2e0 con 0x7fd3a80cbad0 2024-04-17T07:32:14.054 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.054+0000 7f5f8b9b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:32:14.087 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:14.087+0000 7f7100c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:14.108 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.108+0000 7f5f8b9b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:32:14.194 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:14.194+0000 7f7100c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:32:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.240+0000 7fd3a5e00640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 msgr2=0x7fd38406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.240+0000 7fd3a5e00640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 0x7fd38406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:14.241 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.241+0000 7f5f8b9b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:14.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.257+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd3a80b87d0 con 0x7fd3a80cbad0 2024-04-17T07:32:14.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.258+0000 7fd39e000640 1 -- 172.21.15.67:0/1415431221 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v114) ==== 80+0+475 (secure 0 0 0) 0x7fd3a0058eb0 con 0x7fd3a80cbad0 2024-04-17T07:32:14.258 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/403098035 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.258 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3476034436 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.258 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/158797128 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/158797128 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3513519163 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1745494580 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1260936234 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2471807753 2024-04-18T07:31:07.255351+0000 2024-04-17T07:32:14.259 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.260+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 msgr2=0x7fd38406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.260+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 0x7fd38406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.260+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 msgr2=0x7fd3a80cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.260+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80d45b0 tx=0x7fd3a000e8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 shutdown_connections 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fd38406cb30 0x7fd38406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3a80cbad0 0x7fd3a80cbf50 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3a80cdb20 0x7fd3a80cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 --2- 172.21.15.67:0/1415431221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd38c004800 0x7fd3a814bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 >> 172.21.15.67:0/1415431221 conn(0x7fd3a806d3c0 msgr2=0x7fd3a80b8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 shutdown_connections 2024-04-17T07:32:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.261+0000 7fd3a7200640 1 -- 172.21.15.67:0/1415431221 wait complete. 2024-04-17T07:32:14.279 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:32:14.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:14.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:32:14.357 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.357+0000 7f5f8b9b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:32:14.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:32:14.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:14.370 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:32:14.370 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:14.369+0000 7f7100c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:14.411 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:32:14.411 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:32:14.411 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:14.420 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:14.420+0000 7f7100c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:14.462 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:32:14.462 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:32:14.462 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:14.462 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:14.516 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:32:14.516 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:32:14.516 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:32:14.516 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:14.546 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.546+0000 7f5f8b9b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:14.563 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:14.563+0000 7f7100c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:32:14.568 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:32:14.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:14.599 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.599+0000 7f5f8b9b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:14.747 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:14.746+0000 7f5f8b9b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:32:14.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.783+0000 7f8412c00640 1 Processor -- start 2024-04-17T07:32:14.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.783+0000 7f8412c00640 1 -- start start 2024-04-17T07:32:14.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.784+0000 7f8412c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f840c160210 0x7f840c0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.784+0000 7f8412c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c09b170 0x7f840c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.784+0000 7f8412c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840c0ceea0 con 0x7f840c0ce2e0 2024-04-17T07:32:14.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.784+0000 7f8412c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f840c0b2af0 con 0x7f840c160210 2024-04-17T07:32:14.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.784+0000 7f8412c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f840c0b21c0 con 0x7f840c09b170 2024-04-17T07:32:14.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.785+0000 7f840be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f840c160210 0x7f840c0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.785+0000 7f840be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f840c160210 0x7f840c0cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:52696/0 (socket says 172.21.15.67:52696) 2024-04-17T07:32:14.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.785+0000 7f840be00640 1 -- 172.21.15.67:0/1500089043 learned_addr learned my addr 172.21.15.67:0/1500089043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:14.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.785+0000 7f8410c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c09b170 0x7f840c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.786+0000 7f840be00640 1 -- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c09b170 msgr2=0x7f840c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:14.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.786+0000 7f840be00640 1 --2- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c09b170 0x7f840c0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.786+0000 7f840be00640 1 -- 172.21.15.67:0/1500089043 >> v1:172.21.15.187:6789/0 conn(0x7f840c0ce2e0 legacy=0x7f840c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.786+0000 7f840be00640 1 -- 172.21.15.67:0/1500089043 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840c0b24a0 con 0x7f840c160210 2024-04-17T07:32:14.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.787+0000 7f840be00640 1 --2- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0cdda0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009870 tx=0x7f83fc02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edbea26b463b79ee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:14.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f840aa00640 1 -- 172.21.15.67:0/1500089043 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc02f830 con 0x7f840c160210 2024-04-17T07:32:14.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f840aa00640 1 -- 172.21.15.67:0/1500089043 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc02f9d0 con 0x7f840c160210 2024-04-17T07:32:14.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f840aa00640 1 -- 172.21.15.67:0/1500089043 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc02fda0 con 0x7f840c160210 2024-04-17T07:32:14.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 -- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 msgr2=0x7f840c0cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 --2- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0cdda0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009870 tx=0x7f83fc02ba10 comp rx=0 tx=0).stop 2024-04-17T07:32:14.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 -- 172.21.15.67:0/1500089043 shutdown_connections 2024-04-17T07:32:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 --2- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c09b170 0x7f840c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 --2- 172.21.15.67:0/1500089043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0cdda0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 -- 172.21.15.67:0/1500089043 >> 172.21.15.67:0/1500089043 conn(0x7f840c0b41b0 msgr2=0x7f840c0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.788+0000 7f8412c00640 1 -- 172.21.15.67:0/1500089043 shutdown_connections 2024-04-17T07:32:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.789+0000 7f8412c00640 1 -- 172.21.15.67:0/1500089043 wait complete. 2024-04-17T07:32:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.790+0000 7f8412c00640 1 Processor -- start 2024-04-17T07:32:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.790+0000 7f8412c00640 1 -- start start 2024-04-17T07:32:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 0x7f840c1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f840c0af4d0 con 0x7f840c09b170 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f840c0af350 con 0x7f840c160210 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f8412c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f840c0b34c0 con 0x7f840c0ce2e0 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.791+0000 7f840be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33522/0 (socket says 172.21.15.67:33522) 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 -- 172.21.15.67:0/169177792 learned_addr learned my addr 172.21.15.67:0/169177792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f8410c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 msgr2=0x7f840c1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840b400640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 0x7f840c1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 0x7f840c1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 msgr2=0x7f840c0c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 -- 172.21.15.67:0/169177792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840c159de0 con 0x7f840c09b170 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f8410c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840b400640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 0x7f840c1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.792+0000 7f840be00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f83fc02ba50 tx=0x7f83fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.793+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc047070 con 0x7f840c09b170 2024-04-17T07:32:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.793+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc037c40 con 0x7f840c09b170 2024-04-17T07:32:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.793+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc042480 con 0x7f840c09b170 2024-04-17T07:32:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.793+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840c0b52e0 con 0x7f840c09b170 2024-04-17T07:32:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.793+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f840c0b58a0 con 0x7f840c09b170 2024-04-17T07:32:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.794+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840c068b90 con 0x7f840c09b170 2024-04-17T07:32:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.796+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f83fc02d3c0 con 0x7f840c09b170 2024-04-17T07:32:14.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.797+0000 7f8403e00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 0x7f83e806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:14.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.797+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f83fc0ae490 con 0x7f840c09b170 2024-04-17T07:32:14.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.797+0000 7f840b400640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 msgr2=0x7f83e806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:14.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.797+0000 7f840b400640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 0x7f83e806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:14.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.801+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f83fc033030 con 0x7f840c09b170 2024-04-17T07:32:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.998+0000 7f840b400640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 msgr2=0x7f83e806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:14.998+0000 7f840b400640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 0x7f83e806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.045+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840c0cfa80 con 0x7f840c09b170 2024-04-17T07:32:15.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.046+0000 7f8403e00640 1 -- 172.21.15.67:0/169177792 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7f83fc07c230 con 0x7f840c09b170 2024-04-17T07:32:15.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:15.047 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:32:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 msgr2=0x7f83e806eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 0x7f83e806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 msgr2=0x7f840c0c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f83fc02ba50 tx=0x7f83fc004730 comp rx=0 tx=0).stop 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 shutdown_connections 2024-04-17T07:32:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f83e806cb30 0x7f83e806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f840c160210 0x7f840c0c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f840c0ce2e0 0x7f840c1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 --2- 172.21.15.67:0/169177792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840c09b170 0x7f840c0c9790 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 >> 172.21.15.67:0/169177792 conn(0x7f840c0b41b0 msgr2=0x7f840c0cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 shutdown_connections 2024-04-17T07:32:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.049+0000 7f8412c00640 1 -- 172.21.15.67:0/169177792 wait complete. 2024-04-17T07:32:15.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:15.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.327+0000 7fa8f6600640 1 Processor -- start 2024-04-17T07:32:15.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.327+0000 7fa8f6600640 1 -- start start 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8f6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f006feb0 0x7fa8f006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8f6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8f6600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f015bc80 con 0x7fa8f015b570 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8f6600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f015bf40 con 0x7fa8f006feb0 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8f6600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f015c220 con 0x7fa8f006be70 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.328+0000 7fa8efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f006feb0 0x7fa8f006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f006feb0 0x7fa8f006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33530/0 (socket says 172.21.15.67:33530) 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 -- 172.21.15.67:0/800213331 learned_addr learned my addr 172.21.15.67:0/800213331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:15.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8f5c00640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 -- 172.21.15.67:0/800213331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 msgr2=0x7fa8f015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 -- 172.21.15.67:0/800213331 >> v1:172.21.15.187:6789/0 conn(0x7fa8f015b570 legacy=0x7fa8f006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.329+0000 7fa8efe00640 1 -- 172.21.15.67:0/800213331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f015c500 con 0x7fa8f006feb0 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.330+0000 7fa8efe00640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f006b930 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00bb10 tx=0x7fa8dc02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122a91e308f5a8e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:15.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.330+0000 7fa8ef400640 1 -- 172.21.15.67:0/800213331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc03b070 con 0x7fa8f006feb0 2024-04-17T07:32:15.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.330+0000 7fa8ef400640 1 -- 172.21.15.67:0/800213331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8dc02ede0 con 0x7fa8f006feb0 2024-04-17T07:32:15.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8ef400640 1 -- 172.21.15.67:0/800213331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc036660 con 0x7fa8f006feb0 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 -- 172.21.15.67:0/800213331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 msgr2=0x7fa8f006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f006b930 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00bb10 tx=0x7fa8dc02e030 comp rx=0 tx=0).stop 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 -- 172.21.15.67:0/800213331 shutdown_connections 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 --2- 172.21.15.67:0/800213331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f006b930 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 -- 172.21.15.67:0/800213331 >> 172.21.15.67:0/800213331 conn(0x7fa8f0067d10 msgr2=0x7fa8f00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 -- 172.21.15.67:0/800213331 shutdown_connections 2024-04-17T07:32:15.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.331+0000 7fa8f6600640 1 -- 172.21.15.67:0/800213331 wait complete. 2024-04-17T07:32:15.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.333+0000 7fa8f6600640 1 Processor -- start 2024-04-17T07:32:15.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.333+0000 7fa8f6600640 1 -- start start 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f0072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8f015b570 0x7fa8f014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f006c440 con 0x7fa8f006feb0 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f006c2c0 con 0x7fa8f015b570 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.334+0000 7fa8f6600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00b47b0 con 0x7fa8f006be70 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33544/0 (socket says 172.21.15.67:33544) 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 -- 172.21.15.67:0/1038001741 learned_addr learned my addr 172.21.15.67:0/1038001741 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8f5200640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f0072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8f5c00640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8f015b570 0x7fa8f014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 msgr2=0x7fa8f0072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f0072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8f015b570 msgr2=0x7fa8f014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8f015b570 0x7fa8f014b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.335+0000 7fa8efe00640 1 -- 172.21.15.67:0/1038001741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f015a3a0 con 0x7fa8f006feb0 2024-04-17T07:32:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8efe00640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00b2c0 tx=0x7fa8dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc046070 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8dc03d050 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc03b030 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f015a680 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.336+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f0159610 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.338+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f0068a10 con 0x7fa8f006feb0 2024-04-17T07:32:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.339+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7fa8dc034550 con 0x7fa8f006feb0 2024-04-17T07:32:15.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.340+0000 7fa8ecc00640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 0x7fa8d006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.340+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7fa8dc0ad810 con 0x7fa8f006feb0 2024-04-17T07:32:15.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.340+0000 7fa8f5200640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 msgr2=0x7fa8d006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:15.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.340+0000 7fa8f5200640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 0x7fa8d006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:15.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.345+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa8dc07b100 con 0x7fa8f006feb0 2024-04-17T07:32:15.391 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.390+0000 7f7100c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:32:15.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.541+0000 7fa8f5200640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 msgr2=0x7fa8d006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:15.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.541+0000 7fa8f5200640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 0x7fa8d006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.553+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8f0072be0 con 0x7fa8f006feb0 2024-04-17T07:32:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.556+0000 7fa8ecc00640 1 -- 172.21.15.67:0/1038001741 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+17179 (secure 0 0 0) 0x7fa8dc0aa330 con 0x7fa8f006feb0 2024-04-17T07:32:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:15.557 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":114,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:09.570881+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.1a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/403098035":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3476034436":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6833/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6832/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3513519163":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1745494580":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1260936234":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/2471807753":"2024-04-18T07:31:07.255351+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.558+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 msgr2=0x7fa8d006eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.558+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 0x7fa8d006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 msgr2=0x7fa8f00af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00b2c0 tx=0x7fa8dc004730 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 shutdown_connections 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7fa8d006cb30 0x7fa8d006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8f015b570 0x7fa8f014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f006feb0 0x7fa8f00af000 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 --2- 172.21.15.67:0/1038001741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8f006be70 0x7fa8f0072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 >> 172.21.15.67:0/1038001741 conn(0x7fa8f0067d10 msgr2=0x7fa8f00b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:15.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 shutdown_connections 2024-04-17T07:32:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.559+0000 7fa8f6600640 1 -- 172.21.15.67:0/1038001741 wait complete. 2024-04-17T07:32:15.576 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:30:50.351588+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:30:51.385068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:32:15.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:32:15.619 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:15.619+0000 7f5f8b9b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:32:15.695 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.695+0000 7f7100c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:32:15.752 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.752+0000 7f7100c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:32:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.775+0000 7f1d9b200640 1 Processor -- start 2024-04-17T07:32:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.775+0000 7f1d9b200640 1 -- start start 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d9c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1d9c0cfcd0 0x7f1d9c0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d9c0d4f70 con 0x7f1d9c15ec20 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1d9c0d5250 con 0x7f1d9c0cfcd0 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9c0cf4d0 con 0x7f1d9c0b3b50 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d99e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1d9c15ec20 0x7f1d9c06ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54270/0 (socket says 172.21.15.67:54270) 2024-04-17T07:32:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d99e00640 1 -- 172.21.15.67:0/452180123 learned_addr learned my addr 172.21.15.67:0/452180123 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d99400640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d9c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815581825 0 0) 0x7f1d9c0d4f70 con 0x7f1d9c15ec20 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.777+0000 7f1d9a800640 1 --2- 172.21.15.67:0/452180123 >> v2:172.21.15.187:3300/0 conn(0x7f1d9c0cfcd0 0x7f1d9c0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d80003520 con 0x7f1d9c15ec20 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2509600440 0 0) 0x7f1d80003520 con 0x7f1d9c15ec20 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 msgr2=0x7f1d9c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d9c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 >> v2:172.21.15.187:3300/0 conn(0x7f1d9c0cfcd0 msgr2=0x7f1d9c0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 --2- 172.21.15.67:0/452180123 >> v2:172.21.15.187:3300/0 conn(0x7f1d9c0cfcd0 0x7f1d9c0d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9c0cf7b0 con 0x7f1d9c15ec20 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1d8c001670 con 0x7f1d9c15ec20 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 >> v1:172.21.15.67:6789/0 conn(0x7f1d9c15ec20 legacy=0x7f1d9c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d80004d40 0x7f1d800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c0d5250 con 0x7f1d80003f30 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c0cf4d0 con 0x7f1d80004d40 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8000bb50 con 0x7f1d9c0b3b50 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d9a800640 1 --2- 172.21.15.67:0/452180123 >> v2:172.21.15.187:3300/0 conn(0x7f1d9c0cfcd0 0x7f1d9c0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.778+0000 7f1d99400640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99e00640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d9a800640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d80004d40 0x7f1d800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99400640 1 -- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 msgr2=0x7f1d800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99400640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c0b3b50 0x7f1d800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99400640 1 -- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d80004d40 msgr2=0x7f1d800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99400640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d80004d40 0x7f1d800051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.779+0000 7f1d99400640 1 -- 172.21.15.67:0/452180123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d80003520 con 0x7f1d80003f30 2024-04-17T07:32:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.780+0000 7f1d99400640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d80004800 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800e170 tx=0x7f1d80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb52ecbcd54dba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.780+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88039070 con 0x7f1d80003f30 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.780+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8802cc60 con 0x7f1d80003f30 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.780+0000 7f1d98a00640 1 -- 172.21.15.67:0/452180123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88009c60 con 0x7f1d80003f30 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 -- 172.21.15.67:0/452180123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 msgr2=0x7f1d80004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 --2- 172.21.15.67:0/452180123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d80004800 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800e170 tx=0x7f1d80005830 comp rx=0 tx=0).stop 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9a800640 1 -- 172.21.15.67:0/452180123 reap_dead start 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 -- 172.21.15.67:0/452180123 shutdown_connections 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 -- 172.21.15.67:0/452180123 >> 172.21.15.67:0/452180123 conn(0x7f1d9c08bff0 msgr2=0x7f1d9c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 -- 172.21.15.67:0/452180123 shutdown_connections 2024-04-17T07:32:15.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.781+0000 7f1d9b200640 1 -- 172.21.15.67:0/452180123 wait complete. 2024-04-17T07:32:15.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.784+0000 7f1d9b200640 1 Processor -- start 2024-04-17T07:32:15.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.784+0000 7f1d9b200640 1 -- start start 2024-04-17T07:32:15.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d9c159a60 0x7f1d9c15f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c14c560 0x7f1d9c14ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c06d2d0 con 0x7f1d80003f30 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c06d150 con 0x7f1d9c159a60 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9c0d22b0 con 0x7f1d9c14c560 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33562/0 (socket says 172.21.15.67:33562) 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 -- 172.21.15.67:0/3761485424 learned_addr learned my addr 172.21.15.67:0/3761485424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d9a800640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c14c560 0x7f1d9c14ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99400640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d9c159a60 0x7f1d9c15f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 -- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c14c560 msgr2=0x7f1d9c14ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c14c560 0x7f1d9c14ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 -- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d9c159a60 msgr2=0x7f1d9c15f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d9c159a60 0x7f1d9c15f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.786+0000 7f1d99e00640 1 -- 172.21.15.67:0/3761485424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9c14cf50 con 0x7f1d80003f30 2024-04-17T07:32:15.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d99e00640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00e820 tx=0x7f1d8c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:15.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c00c870 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8c018070 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9c15f970 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c00cc50 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.787+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9c15ff30 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.788+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d9c160640 con 0x7f1d80003f30 2024-04-17T07:32:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.789+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 53) ==== 91256+0+0 (secure 0 0 0) 0x7f1d8c01d460 con 0x7f1d80003f30 2024-04-17T07:32:15.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.790+0000 7f1d92000640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 0x7f1d7c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:15.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.790+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6082+0+0 (secure 0 0 0) 0x7f1d8c08bfd0 con 0x7f1d80003f30 2024-04-17T07:32:15.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.791+0000 7f1d99400640 1 -- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 msgr2=0x7f1d7c06eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1997545601 2024-04-17T07:32:15.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.791+0000 7f1d99400640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 0x7f1d7c06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:15.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.795+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1d8c05a140 con 0x7f1d80003f30 2024-04-17T07:32:15.884 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.884+0000 7f7100c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:32:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.910+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 54) ==== 90751+0+0 (secure 0 0 0) 0x7f1d8c0560b0 con 0x7f1d80003f30 2024-04-17T07:32:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.910+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 msgr2=0x7f1d7c06eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:15.910+0000 7f1d92000640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 0x7f1d7c06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:15.956 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:15.955+0000 7f5f8b9b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:32:15.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707dc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707dc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707dc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707dc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:15.988 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:15.987+0000 7f707dc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.011+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d9c0d55f0 con 0x7f1d80003f30 2024-04-17T07:32:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.012+0000 7f1d92000640 1 -- 172.21.15.67:0/3761485424 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7f1d8c044680 con 0x7f1d80003f30 2024-04-17T07:32:16.012 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:32:16.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 msgr2=0x7f1d9c0b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00e820 tx=0x7f1d8c00ece0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 shutdown_connections 2024-04-17T07:32:16.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:6832/1997545601,v1:172.21.15.187:6833/1997545601] conn(0x7f1d7c06cb30 0x7f1d7c06eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d9c14c560 0x7f1d9c14ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d9c159a60 0x7f1d9c15f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 --2- 172.21.15.67:0/3761485424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d80003f30 0x7f1d9c0b5400 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 >> 172.21.15.67:0/3761485424 conn(0x7f1d9c08bff0 msgr2=0x7f1d9c074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 shutdown_connections 2024-04-17T07:32:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.015+0000 7f1d9b200640 1 -- 172.21.15.67:0/3761485424 wait complete. 2024-04-17T07:32:16.020 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:16.020+0000 7f5f8b9b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:32:16.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:32:16.161 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:16.161+0000 7f5f8b9b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.282+0000 7f0549600640 1 Processor -- start 2024-04-17T07:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.282+0000 7f0549600640 1 -- start start 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0549600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f054415ad10 0x7f054406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0549600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f054406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0549600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054406e1f0 con 0x7f05440b3ea0 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0549600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f054406e4d0 con 0x7f054415ad10 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0549600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f054406e7b0 con 0x7f05440ce070 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0543400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f05440b3ea0 0x7f05440cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54274/0 (socket says 172.21.15.67:54274) 2024-04-17T07:32:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.284+0000 7f0543400640 1 -- 172.21.15.67:0/685386415 learned_addr learned my addr 172.21.15.67:0/685386415 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0543e00640 1 --2- 172.21.15.67:0/685386415 >> v2:172.21.15.187:3300/0 conn(0x7f054415ad10 0x7f054406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0548c00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f054406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877965746 0 0) 0x7f054406e1f0 con 0x7f05440b3ea0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0528003520 con 0x7f05440b3ea0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2618707777 0 0) 0x7f0528003520 con 0x7f05440b3ea0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 msgr2=0x7f054406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f054406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 >> v2:172.21.15.187:3300/0 conn(0x7f054415ad10 msgr2=0x7f054406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 --2- 172.21.15.67:0/685386415 >> v2:172.21.15.187:3300/0 conn(0x7f054415ad10 0x7f054406f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054406ea90 con 0x7f05440b3ea0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f05300019f0 con 0x7f05440b3ea0 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 >> v1:172.21.15.67:6789/0 conn(0x7f05440b3ea0 legacy=0x7f05440cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f0528004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528004d40 0x7f05280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f054406e4d0 con 0x7f0528004d40 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f054406e7b0 con 0x7f05280039b0 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.285+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f052800b9c0 con 0x7f05440ce070 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528004d40 0x7f05280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0543400640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f0528004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0543e00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 -- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 msgr2=0x7f0528004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05440ce070 0x7f0528004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 -- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05280039b0 msgr2=0x7f0528003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0548c00640 1 -- 172.21.15.67:0/685386415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0528003520 con 0x7f0528004d40 2024-04-17T07:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.286+0000 7f0543e00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05280039b0 0x7f0528003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.287+0000 7f0548c00640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528004d40 0x7f05280051c0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f05440afe20 tx=0x7f0528005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc791b0d33dd2d0b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.287+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c039070 con 0x7f0528004d40 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.287+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c02c660 con 0x7f0528004d40 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.287+0000 7f0542a00640 1 -- 172.21.15.67:0/685386415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c009690 con 0x7f0528004d40 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 -- 172.21.15.67:0/685386415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528004d40 msgr2=0x7f05280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 --2- 172.21.15.67:0/685386415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0528004d40 0x7f05280051c0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f05440afe20 tx=0x7f0528005830 comp rx=0 tx=0).stop 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0548c00640 1 -- 172.21.15.67:0/685386415 reap_dead start 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 -- 172.21.15.67:0/685386415 shutdown_connections 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 -- 172.21.15.67:0/685386415 >> 172.21.15.67:0/685386415 conn(0x7f05440b2190 msgr2=0x7f05440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 -- 172.21.15.67:0/685386415 shutdown_connections 2024-04-17T07:32:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.288+0000 7f0549600640 1 -- 172.21.15.67:0/685386415 wait complete. 2024-04-17T07:32:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.290+0000 7f0549600640 1 Processor -- start 2024-04-17T07:32:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.290+0000 7f0549600640 1 -- start start 2024-04-17T07:32:16.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05441581e0 0x7f05441e73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05441e4cc0 0x7f05441e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f054406dd30 con 0x7f05441574a0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f054415b0f0 con 0x7f05441e4cc0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0549600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f054406deb0 con 0x7f05441581e0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0543e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0543e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33584/0 (socket says 172.21.15.67:33584) 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.291+0000 7f0543e00640 1 -- 172.21.15.67:0/2328893203 learned_addr learned my addr 172.21.15.67:0/2328893203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 -- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05441581e0 msgr2=0x7f05441e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05441581e0 0x7f05441e73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 -- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05441e4cc0 msgr2=0x7f05441e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05441e4cc0 0x7f05441e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 -- 172.21.15.67:0/2328893203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05441e51c0 con 0x7f05441574a0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0543e00640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f053400f910 tx=0x7f053400fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053400b040 con 0x7f05441574a0 2024-04-17T07:32:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053400dc90 con 0x7f05441574a0 2024-04-17T07:32:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05441e54a0 con 0x7f05441574a0 2024-04-17T07:32:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534012460 con 0x7f05441574a0 2024-04-17T07:32:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.292+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05441e5a60 con 0x7f05441574a0 2024-04-17T07:32:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.293+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0544068a10 con 0x7f05441574a0 2024-04-17T07:32:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.294+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 54) ==== 90751+0+0 (secure 0 0 0) 0x7f0534012600 con 0x7f05441574a0 2024-04-17T07:32:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.295+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f053408c270 con 0x7f05441574a0 2024-04-17T07:32:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.300+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0534017030 con 0x7f05441574a0 2024-04-17T07:32:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.507+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f054415d8d0 con 0x7f05441574a0 2024-04-17T07:32:16.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.508+0000 7f053be00640 1 -- 172.21.15.67:0/2328893203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) ==== 104+0+11 (secure 0 0 0) 0x7f053405a700 con 0x7f05441574a0 2024-04-17T07:32:16.509 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:32:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 msgr2=0x7f05441e6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f053400f910 tx=0x7f053400fdd0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 shutdown_connections 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05441e4cc0 0x7f05441e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f05441581e0 0x7f05441e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 --2- 172.21.15.67:0/2328893203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05441574a0 0x7f05441e6260 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 >> 172.21.15.67:0/2328893203 conn(0x7f05440b2190 msgr2=0x7f0544075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 shutdown_connections 2024-04-17T07:32:16.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.511+0000 7f0549600640 1 -- 172.21.15.67:0/2328893203 wait complete. 2024-04-17T07:32:16.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:32:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.757+0000 7f9666e00640 1 Processor -- start 2024-04-17T07:32:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.757+0000 7f9666e00640 1 -- start start 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.758+0000 7f9666e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96600cd040 0x7f96600cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.758+0000 7f9666e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f96600ce610 0x7f96600b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.758+0000 7f9666e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96600cd960 con 0x7f966015ed00 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f9666e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9660071f70 con 0x7f96600cd040 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f9666e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9660072250 con 0x7f96600ce610 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f9664e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f966015ed00 0x7f96600b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54290/0 (socket says 172.21.15.67:54290) 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f9664e00640 1 -- 172.21.15.67:0/1860173405 learned_addr learned my addr 172.21.15.67:0/1860173405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f965fe00640 1 --2- 172.21.15.67:0/1860173405 >> v2:172.21.15.67:3300/0 conn(0x7f96600cd040 0x7f96600cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.759+0000 7f9657400640 1 --2- 172.21.15.67:0/1860173405 >> v2:172.21.15.187:3300/0 conn(0x7f96600ce610 0x7f96600b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965f400640 1 -- 172.21.15.67:0/1860173405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170972981 0 0) 0x7f96600cd960 con 0x7f966015ed00 2024-04-17T07:32:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965f400640 1 -- 172.21.15.67:0/1860173405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9644003520 con 0x7f966015ed00 2024-04-17T07:32:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965fe00640 1 -- 172.21.15.67:0/1860173405 >> v2:172.21.15.187:3300/0 conn(0x7f96600ce610 msgr2=0x7f96600b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965fe00640 1 --2- 172.21.15.67:0/1860173405 >> v2:172.21.15.187:3300/0 conn(0x7f96600ce610 0x7f96600b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965fe00640 1 -- 172.21.15.67:0/1860173405 >> v1:172.21.15.67:6789/0 conn(0x7f966015ed00 legacy=0x7f96600b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.760+0000 7f965fe00640 1 -- 172.21.15.67:0/1860173405 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9660072530 con 0x7f96600cd040 2024-04-17T07:32:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.761+0000 7f965fe00640 1 --2- 172.21.15.67:0/1860173405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600cd040 0x7f96600cd420 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f9660070300 tx=0x7f965002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6079e29043dd76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:16.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f965f400640 1 -- 172.21.15.67:0/1860173405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650035020 con 0x7f96600cd040 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f965f400640 1 -- 172.21.15.67:0/1860173405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650031830 con 0x7f96600cd040 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f965f400640 1 -- 172.21.15.67:0/1860173405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650031c00 con 0x7f96600cd040 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 -- 172.21.15.67:0/1860173405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600cd040 msgr2=0x7f96600cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 --2- 172.21.15.67:0/1860173405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600cd040 0x7f96600cd420 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f9660070300 tx=0x7f965002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 -- 172.21.15.67:0/1860173405 shutdown_connections 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 --2- 172.21.15.67:0/1860173405 >> v2:172.21.15.187:3300/0 conn(0x7f96600ce610 0x7f96600b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 --2- 172.21.15.67:0/1860173405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600cd040 0x7f96600cd420 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 -- 172.21.15.67:0/1860173405 >> 172.21.15.67:0/1860173405 conn(0x7f966006b8c0 msgr2=0x7f966006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.762+0000 7f9666e00640 1 -- 172.21.15.67:0/1860173405 shutdown_connections 2024-04-17T07:32:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.763+0000 7f9666e00640 1 -- 172.21.15.67:0/1860173405 wait complete. 2024-04-17T07:32:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.764+0000 7f9666e00640 1 Processor -- start 2024-04-17T07:32:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.764+0000 7f9666e00640 1 -- start start 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.765+0000 7f9666e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96600cd040 0x7f9660161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.765+0000 7f9666e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.765+0000 7f9666e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966015ed00 0x7f96600aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.765+0000 7f9666e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96600b2100 con 0x7f96600ce610 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9666e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9660072af0 con 0x7f966015ed00 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9666e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96600b50d0 con 0x7f96600cd040 2024-04-17T07:32:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33592/0 (socket says 172.21.15.67:33592) 2024-04-17T07:32:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 -- 172.21.15.67:0/1004171550 learned_addr learned my addr 172.21.15.67:0/1004171550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9664e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966015ed00 0x7f96600aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f965fe00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96600cd040 0x7f9660161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 -- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96600cd040 msgr2=0x7f9660161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96600cd040 0x7f9660161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 -- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966015ed00 msgr2=0x7f96600aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966015ed00 0x7f96600aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.766+0000 7f9657400640 1 -- 172.21.15.67:0/1004171550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96601f3f80 con 0x7f96600ce610 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f9657400640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f964800d800 tx=0x7f964800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964800b660 con 0x7f96600ce610 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9648019070 con 0x7f96600ce610 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964800ba40 con 0x7f96600ce610 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96601515b0 con 0x7f96600ce610 2024-04-17T07:32:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.767+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96601f44e0 con 0x7f96600ce610 2024-04-17T07:32:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.769+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9660068b90 con 0x7f96600ce610 2024-04-17T07:32:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.769+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 54) ==== 90751+0+0 (secure 0 0 0) 0x7f964800bbe0 con 0x7f96600ce610 2024-04-17T07:32:16.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.770+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f964808bd30 con 0x7f96600ce610 2024-04-17T07:32:16.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.775+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f96601f3f80 con 0x7f96600ce610 2024-04-17T07:32:16.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.916+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 55) ==== 90878+0+0 (secure 0 0 0) 0x7f96480558e0 con 0x7f96600ce610 2024-04-17T07:32:16.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.917+0000 7f965cc00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 0x7f963c074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:16.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.919+0000 7f965fe00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 0x7f963c074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.934+0000 7f965fe00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 0x7f963c074500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9650036000 tx=0x7f9650008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:16.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.976+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f966006cf00 con 0x7f96600ce610 2024-04-17T07:32:16.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.977+0000 7f965cc00640 1 -- 172.21.15.67:0/1004171550 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) ==== 100+0+11 (secure 0 0 0) 0x7f96480606a0 con 0x7f96600ce610 2024-04-17T07:32:16.977 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.979+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 msgr2=0x7f963c074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.979+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 0x7f963c074500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9650036000 tx=0x7f9650008030 comp rx=0 tx=0).stop 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.979+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 msgr2=0x7f9660153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.979+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f964800d800 tx=0x7f964800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.979+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 shutdown_connections 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f963c072110 0x7f963c074500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f966015ed00 0x7f96600aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96600ce610 0x7f9660153800 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 --2- 172.21.15.67:0/1004171550 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96600cd040 0x7f9660161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 >> 172.21.15.67:0/1004171550 conn(0x7f966006b8c0 msgr2=0x7f96600b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 shutdown_connections 2024-04-17T07:32:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:16.980+0000 7f9666e00640 1 -- 172.21.15.67:0/1004171550 wait complete. 2024-04-17T07:32:16.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.209+0000 7f249b200640 1 Processor -- start 2024-04-17T07:32:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.209+0000 7f249b200640 1 -- start start 2024-04-17T07:32:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f249406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2494072500 0x7f24940ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24940cd920 con 0x7f2494072eb0 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f24940cdc30 con 0x7f2494072500 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f249406f980 con 0x7f249415ac80 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2494072500 0x7f24940ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2494072500 0x7f24940ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33598/0 (socket says 172.21.15.67:33598) 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249a800640 1 -- 172.21.15.67:0/3231940665 learned_addr learned my addr 172.21.15.67:0/3231940665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f2499e00640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f249406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249a800640 1 -- 172.21.15.67:0/3231940665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 msgr2=0x7f249406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.211+0000 7f249a800640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f249406f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.212+0000 7f249a800640 1 -- 172.21.15.67:0/3231940665 >> v1:172.21.15.187:6789/0 conn(0x7f2494072eb0 legacy=0x7f2494071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.212+0000 7f249a800640 1 -- 172.21.15.67:0/3231940665 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249406fc60 con 0x7f2494072500 2024-04-17T07:32:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f249a800640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072500 0x7f24940ce0c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f249000d150 tx=0x7f249002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3c83490cafbe72 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f2498a00640 1 -- 172.21.15.67:0/3231940665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249002e9e0 con 0x7f2494072500 2024-04-17T07:32:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f2498a00640 1 -- 172.21.15.67:0/3231940665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249002eb80 con 0x7f2494072500 2024-04-17T07:32:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f2498a00640 1 -- 172.21.15.67:0/3231940665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490009600 con 0x7f2494072500 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f249b200640 1 -- 172.21.15.67:0/3231940665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072500 msgr2=0x7f24940ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.213+0000 7f249b200640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072500 0x7f24940ce0c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f249000d150 tx=0x7f249002e040 comp rx=0 tx=0).stop 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 -- 172.21.15.67:0/3231940665 shutdown_connections 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072500 0x7f24940ce0c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 --2- 172.21.15.67:0/3231940665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f249406f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 -- 172.21.15.67:0/3231940665 >> 172.21.15.67:0/3231940665 conn(0x7f2494067d10 msgr2=0x7f24940b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 -- 172.21.15.67:0/3231940665 shutdown_connections 2024-04-17T07:32:17.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.214+0000 7f249b200640 1 -- 172.21.15.67:0/3231940665 wait complete. 2024-04-17T07:32:17.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.215+0000 7f249b200640 1 Processor -- start 2024-04-17T07:32:17.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.215+0000 7f249b200640 1 -- start start 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 0x7f24941f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f2494153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f249415b060 con 0x7f2494072eb0 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f249406f550 con 0x7f2494072500 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.216+0000 7f249b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24941f4490 con 0x7f249415ac80 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33608/0 (socket says 172.21.15.67:33608) 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 -- 172.21.15.67:0/2813120206 learned_addr learned my addr 172.21.15.67:0/2813120206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499e00640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 0x7f24941f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 -- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 msgr2=0x7f2494153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f249a800640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f2494153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f2494153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 -- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 msgr2=0x7f24941f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 0x7f24941f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 -- 172.21.15.67:0/2813120206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249414ee00 con 0x7f2494072eb0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499e00640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 0x7f24941f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.217+0000 7f2499400640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f248400c400 tx=0x7f248400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.218+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484011790 con 0x7f2494072eb0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.218+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2484017070 con 0x7f2494072eb0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.218+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249414daa0 con 0x7f2494072eb0 2024-04-17T07:32:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.218+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484011bb0 con 0x7f2494072eb0 2024-04-17T07:32:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.218+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249414e040 con 0x7f2494072eb0 2024-04-17T07:32:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.219+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2494068a10 con 0x7f2494072eb0 2024-04-17T07:32:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.220+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 55) ==== 90878+0+0 (secure 0 0 0) 0x7f248401c460 con 0x7f2494072eb0 2024-04-17T07:32:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.221+0000 7f248e000640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 0x7f247806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.221+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f248408b930 con 0x7f2494072eb0 2024-04-17T07:32:17.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.222+0000 7f2499e00640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 0x7f247806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.223+0000 7f2499e00640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 0x7f247806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f248800b820 tx=0x7f2488008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.226+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f249414ee00 con 0x7f2494072eb0 2024-04-17T07:32:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.473+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24940b3d80 con 0x7f2494072eb0 2024-04-17T07:32:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.475+0000 7f248e000640 1 -- 172.21.15.67:0/2813120206 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7f2484059d80 con 0x7f2494072eb0 2024-04-17T07:32:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:17.475 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:32:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.477+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 msgr2=0x7f247806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.477+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 0x7f247806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f248800b820 tx=0x7f2488008040 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.477+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 msgr2=0x7f2494152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.477+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f248400c400 tx=0x7f248400c8c0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 shutdown_connections 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f247806c740 0x7f247806ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249415ac80 0x7f2494153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2494072eb0 0x7f2494152e90 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 --2- 172.21.15.67:0/2813120206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2494072500 0x7f24941f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 >> 172.21.15.67:0/2813120206 conn(0x7f2494067d10 msgr2=0x7f249406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 shutdown_connections 2024-04-17T07:32:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.478+0000 7f249b200640 1 -- 172.21.15.67:0/2813120206 wait complete. 2024-04-17T07:32:17.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.719+0000 7f6564e00640 1 Processor -- start 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.720+0000 7f6564e00640 1 -- start start 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.721+0000 7f6564e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f65600ce4a0 0x7f65600c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.721+0000 7f6564e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65600cc500 con 0x7f65600cd5a0 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.721+0000 7f6564e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65600c71e0 con 0x7f6560161e70 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.721+0000 7f6564e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f65600cc7a0 con 0x7f65600ce4a0 2024-04-17T07:32:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.722+0000 7f6555400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f65600cd5a0 0x7f65600c54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54302/0 (socket says 172.21.15.67:54302) 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.722+0000 7f6555400640 1 -- 172.21.15.67:0/309760257 learned_addr learned my addr 172.21.15.67:0/309760257 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.722+0000 7f655e800640 1 --2- 172.21.15.67:0/309760257 >> v2:172.21.15.187:3300/0 conn(0x7f65600ce4a0 0x7f65600c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.722+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985476932 0 0) 0x7f65600cc500 con 0x7f65600cd5a0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.722+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6544003520 con 0x7f65600cd5a0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144472355 0 0) 0x7f6544003520 con 0x7f65600cd5a0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 >> v2:172.21.15.187:3300/0 conn(0x7f65600ce4a0 msgr2=0x7f65600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 --2- 172.21.15.67:0/309760257 >> v2:172.21.15.187:3300/0 conn(0x7f65600ce4a0 0x7f65600c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 >> v1:172.21.15.187:6789/0 conn(0x7f6560161e70 legacy=0x7f65600cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65600cca80 con 0x7f65600cd5a0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6548001690 con 0x7f65600cd5a0 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 >> v1:172.21.15.67:6789/0 conn(0x7f65600cd5a0 legacy=0x7f65600c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 0x7f6544003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544004800 0x7f6544004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65440051c0 0x7f654400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65600cc7a0 con 0x7f6544003a00 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f654400c0c0 con 0x7f65440051c0 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f654400c2c0 con 0x7f6544004800 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655de00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 0x7f6544003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f6555400640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544004800 0x7f6544004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.723+0000 7f655e800640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65440051c0 0x7f654400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655de00640 1 -- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544004800 msgr2=0x7f6544004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655de00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544004800 0x7f6544004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655de00640 1 -- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65440051c0 msgr2=0x7f654400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655de00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65440051c0 0x7f654400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655de00640 1 -- 172.21.15.67:0/309760257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6544003520 con 0x7f6544003a00 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f655e800640 1 -- 172.21.15.67:0/309760257 reap_dead start 2024-04-17T07:32:17.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.724+0000 7f6555400640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544004800 0x7f6544004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f655de00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 0x7f6544003e20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f6550001750 tx=0x7f6544005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c67691220d3440 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550001370 con 0x7f6544003a00 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65500052d0 con 0x7f6544003a00 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f655d400640 1 -- 172.21.15.67:0/309760257 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550003830 con 0x7f6544003a00 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f6564e00640 1 -- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 msgr2=0x7f6544003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.725+0000 7f6564e00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 0x7f6544003e20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f6550001750 tx=0x7f6544005830 comp rx=0 tx=0).stop 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.726+0000 7f6564e00640 1 -- 172.21.15.67:0/309760257 shutdown_connections 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.726+0000 7f6564e00640 1 --2- 172.21.15.67:0/309760257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544003a00 0x7f6544003e20 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.726+0000 7f6564e00640 1 -- 172.21.15.67:0/309760257 >> 172.21.15.67:0/309760257 conn(0x7f656006bf50 msgr2=0x7f65600b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.726+0000 7f6564e00640 1 -- 172.21.15.67:0/309760257 shutdown_connections 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.726+0000 7f6564e00640 1 -- 172.21.15.67:0/309760257 wait complete. 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.727+0000 7f6564e00640 1 Processor -- start 2024-04-17T07:32:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.728+0000 7f6564e00640 1 -- start start 2024-04-17T07:32:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544003a00 0x7f656006df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 0x7f65601557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65600c6d80 con 0x7f656006f6a0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f65600c6c00 con 0x7f6560154c00 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6564e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f65600c6f00 con 0x7f6544003a00 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33638/0 (socket says 172.21.15.67:33638) 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 -- 172.21.15.67:0/4130109768 learned_addr learned my addr 172.21.15.67:0/4130109768 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f655de00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544003a00 0x7f656006df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f655e800640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 0x7f65601557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 -- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544003a00 msgr2=0x7f656006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544003a00 0x7f656006df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 -- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 msgr2=0x7f65601557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 0x7f65601557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f6555400640 1 -- 172.21.15.67:0/4130109768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656006e600 con 0x7f656006f6a0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.729+0000 7f655e800640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 0x7f65601557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6555400640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6548000c00 tx=0x7f654800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548019070 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65480072d0 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480047e0 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656006e8e0 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.730+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656006eea0 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.731+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6560068b90 con 0x7f656006f6a0 2024-04-17T07:32:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.732+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 55) ==== 90878+0+0 (secure 0 0 0) 0x7f6548004980 con 0x7f656006f6a0 2024-04-17T07:32:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.733+0000 7f6556a00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 0x7f653c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.734+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f654808b340 con 0x7f656006f6a0 2024-04-17T07:32:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.734+0000 7f655de00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 0x7f653c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.735+0000 7f655de00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 0x7f653c06ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6550033000 tx=0x7f6550009030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:17.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.738+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6548059090 con 0x7f656006f6a0 2024-04-17T07:32:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.927+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 56) ==== 91086+0+0 (secure 0 0 0) 0x7f6548055700 con 0x7f656006f6a0 2024-04-17T07:32:17.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.948+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65600c6120 con 0x7f656006f6a0 2024-04-17T07:32:17.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.951+0000 7f6556a00640 1 -- 172.21.15.67:0/4130109768 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+17815 (secure 0 0 0) 0x7f654805faf0 con 0x7f656006f6a0 2024-04-17T07:32:17.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:17.952 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:15.890299+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.1a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3905455841":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2921215638":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2142021007":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/4125580629":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2371263017":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/403098035":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3476034436":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6832/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1195480689":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6832/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/2232663954":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/3513519163":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1745494580":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1260936234":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1890541551":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2471807753":"2024-04-18T07:31:07.255351+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:17.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 msgr2=0x7f653c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 0x7f653c06ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6550033000 tx=0x7f6550009030 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 msgr2=0x7f6560150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6548000c00 tx=0x7f654800dc00 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 shutdown_connections 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f653c06c740 0x7f653c06ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6560154c00 0x7f65601557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656006f6a0 0x7f6560150130 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 --2- 172.21.15.67:0/4130109768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6544003a00 0x7f656006df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 >> 172.21.15.67:0/4130109768 conn(0x7f656006bf50 msgr2=0x7f65600cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 shutdown_connections 2024-04-17T07:32:17.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:17.954+0000 7f6564e00640 1 -- 172.21.15.67:0/4130109768 wait complete. 2024-04-17T07:32:17.976 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:32:17.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.200+0000 7f2426a00640 1 Processor -- start 2024-04-17T07:32:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.201+0000 7f2426a00640 1 -- start start 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.202+0000 7f2426a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 0x7f24200d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.202+0000 7f2426a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24200ba470 0x7f242015e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24200d3170 con 0x7f24200d14f0 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f24200d3450 con 0x7f24200ba470 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24200d3730 con 0x7f24200d18d0 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2425600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24200d14f0 0x7f24200d2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54314/0 (socket says 172.21.15.67:54314) 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2425600640 1 -- 172.21.15.67:0/1559929304 learned_addr learned my addr 172.21.15.67:0/1559929304 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426000640 1 --2- 172.21.15.67:0/1559929304 >> v2:172.21.15.67:3300/0 conn(0x7f24200ba470 0x7f242015e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2424c00640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 0x7f24200d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426000640 1 -- 172.21.15.67:0/1559929304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 msgr2=0x7f24200d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426000640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 0x7f24200d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f241fe00640 1 -- 172.21.15.67:0/1559929304 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1452996481 0 0) 0x7f24200d3170 con 0x7f24200d14f0 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426000640 1 -- 172.21.15.67:0/1559929304 >> v1:172.21.15.67:6789/0 conn(0x7f24200d14f0 legacy=0x7f24200d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2426000640 1 -- 172.21.15.67:0/1559929304 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24200d3a10 con 0x7f24200ba470 2024-04-17T07:32:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.203+0000 7f2424c00640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 0x7f24200d4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f2426000640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f242015e4a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2418009d90 tx=0x7f241802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90c4ae7c7e52a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f241fe00640 1 -- 172.21.15.67:0/1559929304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418039070 con 0x7f24200ba470 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f241fe00640 1 -- 172.21.15.67:0/1559929304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f241802cc60 con 0x7f24200ba470 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f241fe00640 1 -- 172.21.15.67:0/1559929304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418034660 con 0x7f24200ba470 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f2426a00640 1 -- 172.21.15.67:0/1559929304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 msgr2=0x7f242015e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.205+0000 7f2426a00640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f242015e4a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2418009d90 tx=0x7f241802beb0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 -- 172.21.15.67:0/1559929304 shutdown_connections 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f242015e4a0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 --2- 172.21.15.67:0/1559929304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d18d0 0x7f24200d4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 -- 172.21.15.67:0/1559929304 >> 172.21.15.67:0/1559929304 conn(0x7f242006cbd0 msgr2=0x7f242006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 -- 172.21.15.67:0/1559929304 shutdown_connections 2024-04-17T07:32:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.206+0000 7f2426a00640 1 -- 172.21.15.67:0/1559929304 wait complete. 2024-04-17T07:32:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.207+0000 7f2426a00640 1 Processor -- start 2024-04-17T07:32:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.207+0000 7f2426a00640 1 -- start start 2024-04-17T07:32:18.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 0x7f24200cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24200d18d0 0x7f2420153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f242015eb60 con 0x7f24200ba470 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f242015e9e0 con 0x7f24200d18d0 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.208+0000 7f2426a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f242015ece0 con 0x7f24200d14f0 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33646/0 (socket says 172.21.15.67:33646) 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 -- 172.21.15.67:0/2182517920 learned_addr learned my addr 172.21.15.67:0/2182517920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2424c00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 0x7f24200cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 -- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 msgr2=0x7f24200cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 0x7f24200cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2426000640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24200d18d0 0x7f2420153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 -- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24200d18d0 msgr2=0x7f2420153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24200d18d0 0x7f2420153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 -- 172.21.15.67:0/2182517920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2420155800 con 0x7f24200ba470 2024-04-17T07:32:18.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2424c00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 0x7f24200cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.209+0000 7f2425600640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f241400c4d0 tx=0x7f241400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.210+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414018070 con 0x7f24200ba470 2024-04-17T07:32:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.210+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2414007500 con 0x7f24200ba470 2024-04-17T07:32:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.210+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414004a20 con 0x7f24200ba470 2024-04-17T07:32:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.210+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24200cf450 con 0x7f24200ba470 2024-04-17T07:32:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.210+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2420155a80 con 0x7f24200ba470 2024-04-17T07:32:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.211+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2420068a10 con 0x7f24200ba470 2024-04-17T07:32:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.212+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 56) ==== 91086+0+0 (secure 0 0 0) 0x7f2414004bc0 con 0x7f24200ba470 2024-04-17T07:32:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.213+0000 7f241d600640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 0x7f240006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.213+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f241408b1e0 con 0x7f24200ba470 2024-04-17T07:32:18.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.214+0000 7f2424c00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 0x7f240006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.215+0000 7f2424c00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 0x7f240006ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f240c006fd0 tx=0x7f240c008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.217+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2420155800 con 0x7f24200ba470 2024-04-17T07:32:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.461+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24200b9460 con 0x7f24200ba470 2024-04-17T07:32:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.462+0000 7f241d600640 1 -- 172.21.15.67:0/2182517920 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7f2414059220 con 0x7f24200ba470 2024-04-17T07:32:18.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:18.463 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:32:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 msgr2=0x7f240006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 0x7f240006ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f240c006fd0 tx=0x7f240c008040 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 msgr2=0x7f2420156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f241400c4d0 tx=0x7f241400c990 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 shutdown_connections 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f240006c9b0 0x7f240006ee70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24200d18d0 0x7f2420153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f24200d14f0 0x7f24200cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 --2- 172.21.15.67:0/2182517920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24200ba470 0x7f2420156560 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 >> 172.21.15.67:0/2182517920 conn(0x7f242006cbd0 msgr2=0x7f2420158f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 shutdown_connections 2024-04-17T07:32:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.466+0000 7f2426a00640 1 -- 172.21.15.67:0/2182517920 wait complete. 2024-04-17T07:32:18.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.698+0000 7f49b9e00640 1 Processor -- start 2024-04-17T07:32:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.698+0000 7f49b9e00640 1 -- start start 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.699+0000 7f49b9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49b415ec60 0x7f49b40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.699+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.699+0000 7f49b9e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49b40d2d30 con 0x7f49b40b6b30 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.699+0000 7f49b9e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f49b40d49c0 con 0x7f49b415ec60 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.699+0000 7f49b9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49b40d3660 con 0x7f49b40cfae0 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49b415ec60 0x7f49b40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49b415ec60 0x7f49b40b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33652/0 (socket says 172.21.15.67:33652) 2024-04-17T07:32:18.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 -- 172.21.15.67:0/3646418664 learned_addr learned my addr 172.21.15.67:0/3646418664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:18.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b9400640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 -- 172.21.15.67:0/3646418664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 msgr2=0x7f49b40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:18.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 -- 172.21.15.67:0/3646418664 >> v1:172.21.15.187:6789/0 conn(0x7f49b40b6b30 legacy=0x7f49b40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b8a00640 1 -- 172.21.15.67:0/3646418664 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b40d3010 con 0x7f49b415ec60 2024-04-17T07:32:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.700+0000 7f49b9400640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.702+0000 7f49b8a00640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b40b65f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009870 tx=0x7f49a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70eb48df0a8aa9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.702+0000 7f49b3400640 1 -- 172.21.15.67:0/3646418664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4034070 con 0x7f49b415ec60 2024-04-17T07:32:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.702+0000 7f49b3400640 1 -- 172.21.15.67:0/3646418664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a4030840 con 0x7f49b415ec60 2024-04-17T07:32:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.702+0000 7f49b3400640 1 -- 172.21.15.67:0/3646418664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4030c10 con 0x7f49b415ec60 2024-04-17T07:32:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 -- 172.21.15.67:0/3646418664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 msgr2=0x7f49b40b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b40b65f0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009870 tx=0x7f49a402b980 comp rx=0 tx=0).stop 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 -- 172.21.15.67:0/3646418664 shutdown_connections 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 --2- 172.21.15.67:0/3646418664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b40b65f0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 -- 172.21.15.67:0/3646418664 >> 172.21.15.67:0/3646418664 conn(0x7f49b40d15f0 msgr2=0x7f49b40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 -- 172.21.15.67:0/3646418664 shutdown_connections 2024-04-17T07:32:18.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.703+0000 7f49b9e00640 1 -- 172.21.15.67:0/3646418664 wait complete. 2024-04-17T07:32:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.705+0000 7f49b9e00640 1 Processor -- start 2024-04-17T07:32:18.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.705+0000 7f49b9e00640 1 -- start start 2024-04-17T07:32:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 0x7f49b4160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49b415f040 con 0x7f49b415ec60 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49b40b6f70 con 0x7f49b40b6b30 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.706+0000 7f49b9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49b40d00c0 con 0x7f49b40cfae0 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33662/0 (socket says 172.21.15.67:33662) 2024-04-17T07:32:18.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 -- 172.21.15.67:0/2535030046 learned_addr learned my addr 172.21.15.67:0/2535030046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b8a00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 0x7f49b4160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b3e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 -- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 msgr2=0x7f49b40ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 -- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 msgr2=0x7f49b4160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 0x7f49b4160b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 -- 172.21.15.67:0/2535030046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b41f8aa0 con 0x7f49b415ec60 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b3e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b8a00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 0x7f49b4160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.707+0000 7f49b9400640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00e7e0 tx=0x7f49ac00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.708+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac00f040 con 0x7f49b415ec60 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.708+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ac004030 con 0x7f49b415ec60 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.708+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0043e0 con 0x7f49b415ec60 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.708+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b41f7f20 con 0x7f49b415ec60 2024-04-17T07:32:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.708+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b41f84e0 con 0x7f49b415ec60 2024-04-17T07:32:18.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.709+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b4068a10 con 0x7f49b415ec60 2024-04-17T07:32:18.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.710+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 56) ==== 91086+0+0 (secure 0 0 0) 0x7f49ac00c6d0 con 0x7f49b415ec60 2024-04-17T07:32:18.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.711+0000 7f49b0c00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 0x7f499406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:18.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.711+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f49ac08bca0 con 0x7f49b415ec60 2024-04-17T07:32:18.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.712+0000 7f49b8a00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 0x7f499406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.713+0000 7f49b8a00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 0x7f499406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49a4036000 tx=0x7f49a403b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:18.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.716+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f49ac017020 con 0x7f49b415ec60 2024-04-17T07:32:18.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.963+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49b40d5710 con 0x7f49b415ec60 2024-04-17T07:32:18.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.963+0000 7f49b0c00640 1 -- 172.21.15.67:0/2535030046 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5201 (secure 0 0 0) 0x7f49ac059cd0 con 0x7f49b415ec60 2024-04-17T07:32:18.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:18.964 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:30:52.419982+0000","modified":"2024-04-17T07:30:55.498917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6232":{"gid":6232,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3534305613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3534305613},{"type":"v1","addr":"172.21.15.187:6835","nonce":3534305613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6232,"qdb_cluster":[6232]},"id":6}]} 2024-04-17T07:32:18.964 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 53 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.966+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 msgr2=0x7f499406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.966+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 0x7f499406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49a4036000 tx=0x7f49a403b040 comp rx=0 tx=0).stop 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.966+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 msgr2=0x7f49b414fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.966+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00e7e0 tx=0x7f49ac00eca0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 shutdown_connections 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f499406c980 0x7f499406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49b415ec60 0x7f49b414fb20 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49b40cfae0 0x7f49b40ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 --2- 172.21.15.67:0/2535030046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49b40b6b30 0x7f49b4160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:18.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 >> 172.21.15.67:0/2535030046 conn(0x7f49b40d15f0 msgr2=0x7f49b40b7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:18.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 shutdown_connections 2024-04-17T07:32:18.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:18.967+0000 7f49b9e00640 1 -- 172.21.15.67:0/2535030046 wait complete. 2024-04-17T07:32:18.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.213+0000 7f12e0400640 1 Processor -- start 2024-04-17T07:32:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.213+0000 7f12e0400640 1 -- start start 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12e0400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12d8072420 0x7f12d80ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12e0400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d8072b20 con 0x7f12d8166440 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12e0400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d80c5eb0 con 0x7f12d80c6f90 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12e0400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f12d80ce960 con 0x7f12d8072420 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12d8072420 0x7f12d80ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12d8072420 0x7f12d80ce050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33672/0 (socket says 172.21.15.67:33672) 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 -- 172.21.15.67:0/2799341388 learned_addr learned my addr 172.21.15.67:0/2799341388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 -- 172.21.15.67:0/2799341388 >> v1:172.21.15.187:6789/0 conn(0x7f12d80c6f90 legacy=0x7f12d806dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 -- 172.21.15.67:0/2799341388 >> v1:172.21.15.67:6789/0 conn(0x7f12d8166440 legacy=0x7f12d8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.215+0000 7f12dd000640 1 -- 172.21.15.67:0/2799341388 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d80c6190 con 0x7f12d8072420 2024-04-17T07:32:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.216+0000 7f12dd000640 1 --2- 172.21.15.67:0/2799341388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8072420 0x7f12d80ce050 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009870 tx=0x7f12c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb66447380cb2f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.216+0000 7f12d7e00640 1 -- 172.21.15.67:0/2799341388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4035020 con 0x7f12d8072420 2024-04-17T07:32:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.216+0000 7f12d7e00640 1 -- 172.21.15.67:0/2799341388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c4031870 con 0x7f12d8072420 2024-04-17T07:32:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.216+0000 7f12d7e00640 1 -- 172.21.15.67:0/2799341388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4031c40 con 0x7f12d8072420 2024-04-17T07:32:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 -- 172.21.15.67:0/2799341388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8072420 msgr2=0x7f12d80ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 --2- 172.21.15.67:0/2799341388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8072420 0x7f12d80ce050 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009870 tx=0x7f12c402b910 comp rx=0 tx=0).stop 2024-04-17T07:32:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 -- 172.21.15.67:0/2799341388 shutdown_connections 2024-04-17T07:32:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 --2- 172.21.15.67:0/2799341388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8072420 0x7f12d80ce050 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 -- 172.21.15.67:0/2799341388 >> 172.21.15.67:0/2799341388 conn(0x7f12d806bf50 msgr2=0x7f12d80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 -- 172.21.15.67:0/2799341388 shutdown_connections 2024-04-17T07:32:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.217+0000 7f12e0400640 1 -- 172.21.15.67:0/2799341388 wait complete. 2024-04-17T07:32:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.219+0000 7f12e0400640 1 Processor -- start 2024-04-17T07:32:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.219+0000 7f12e0400640 1 -- start start 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 0x7f12d80c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12d8166440 0x7f12d815f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12d80cf0d0 con 0x7f12d80c6f90 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f12d806e460 con 0x7f12d8072420 2024-04-17T07:32:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12e0400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12d815cb50 con 0x7f12d8166440 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.220+0000 7f12dd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33682/0 (socket says 172.21.15.67:33682) 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 -- 172.21.15.67:0/315284016 learned_addr learned my addr 172.21.15.67:0/315284016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 -- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12d8166440 msgr2=0x7f12d815f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12d8166440 0x7f12d815f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 -- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 msgr2=0x7f12d80c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dda00640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 0x7f12d80c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 0x7f12d80c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 -- 172.21.15.67:0/315284016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d81fd300 con 0x7f12d80c6f90 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dda00640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 0x7f12d80c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.221+0000 7f12dd000640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009870 tx=0x7f12c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.222+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c40443f0 con 0x7f12d80c6f90 2024-04-17T07:32:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.222+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c40445b0 con 0x7f12d80c6f90 2024-04-17T07:32:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.222+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c40449a0 con 0x7f12d80c6f90 2024-04-17T07:32:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.222+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d80b42a0 con 0x7f12d80c6f90 2024-04-17T07:32:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.222+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d815d080 con 0x7f12d80c6f90 2024-04-17T07:32:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.223+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d8068b90 con 0x7f12d80c6f90 2024-04-17T07:32:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.224+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 56) ==== 91086+0+0 (secure 0 0 0) 0x7f12c404b060 con 0x7f12d80c6f90 2024-04-17T07:32:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.225+0000 7f12d5600640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 0x7f12b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.225+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f12c40ae380 con 0x7f12d80c6f90 2024-04-17T07:32:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.226+0000 7f12dda00640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 0x7f12b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.227+0000 7f12dda00640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 0x7f12b806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12cc000f60 tx=0x7f12cc009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.229+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f12c403f070 con 0x7f12d80c6f90 2024-04-17T07:32:19.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.441+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12d806e7a0 con 0x7f12d80c6f90 2024-04-17T07:32:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.443+0000 7f12d5600640 1 -- 172.21.15.67:0/315284016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+17815 (secure 0 0 0) 0x7f12c407c2b0 con 0x7f12d80c6f90 2024-04-17T07:32:19.444 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:19.444 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:15.890299+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:30:50.351588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-17T07:30:51.385068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.1a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3905455841":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2921215638":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2142021007":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/4125580629":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2371263017":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/403098035":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3476034436":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6832/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1195480689":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6832/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/2232663954":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/3513519163":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1745494580":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1260936234":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1890541551":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2471807753":"2024-04-18T07:31:07.255351+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:19.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 msgr2=0x7f12b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 0x7f12b806ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12cc000f60 tx=0x7f12cc009040 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 msgr2=0x7f12d81fcdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009870 tx=0x7f12c40046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 shutdown_connections 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f12b806c980 0x7f12b806ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12d8166440 0x7f12d815f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d80c6f90 0x7f12d81fcdc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 --2- 172.21.15.67:0/315284016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12d8072420 0x7f12d80c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 >> 172.21.15.67:0/315284016 conn(0x7f12d806bf50 msgr2=0x7f12d8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.446+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 shutdown_connections 2024-04-17T07:32:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.447+0000 7f12e0400640 1 -- 172.21.15.67:0/315284016 wait complete. 2024-04-17T07:32:19.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:32:19.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.677+0000 7f2c6be00640 1 Processor -- start 2024-04-17T07:32:19.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.678+0000 7f2c6be00640 1 -- start start 2024-04-17T07:32:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.679+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c15ee20 0x7f2c6c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.679+0000 7f2c6be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c6c0d18e0 0x7f2c6c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.679+0000 7f2c6be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0d4140 con 0x7f2c6c0d3350 2024-04-17T07:32:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.679+0000 7f2c6be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2c6c0b83b0 con 0x7f2c6c0d18e0 2024-04-17T07:32:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.679+0000 7f2c6be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6c0d1d20 con 0x7f2c6c15ee20 2024-04-17T07:32:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c6b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c6c0d3350 0x7f2c6c067600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54344/0 (socket says 172.21.15.67:54344) 2024-04-17T07:32:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c6b400640 1 -- 172.21.15.67:0/2295880093 learned_addr learned my addr 172.21.15.67:0/2295880093 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c6a000640 1 --2- 172.21.15.67:0/2295880093 >> v2:172.21.15.187:3300/0 conn(0x7f2c6c0d18e0 0x7f2c6c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c6aa00640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c15ee20 0x7f2c6c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685394575 0 0) 0x7f2c6c0d4140 con 0x7f2c6c0d3350 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.680+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c50003520 con 0x7f2c6c0d3350 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2900702892 0 0) 0x7f2c50003520 con 0x7f2c6c0d3350 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c15ee20 msgr2=0x7f2c6c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c15ee20 0x7f2c6c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 >> v2:172.21.15.187:3300/0 conn(0x7f2c6c0d18e0 msgr2=0x7f2c6c0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 --2- 172.21.15.67:0/2295880093 >> v2:172.21.15.187:3300/0 conn(0x7f2c6c0d18e0 0x7f2c6c0d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c0d3b80 con 0x7f2c6c0d3350 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2c64001670 con 0x7f2c6c0d3350 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 >> v1:172.21.15.67:6789/0 conn(0x7f2c6c0d3350 legacy=0x7f2c6c067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c50003a20 0x7f2c50003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c50004800 0x7f2c50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c500051c0 0x7f2c5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0b83b0 con 0x7f2c500051c0 2024-04-17T07:32:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0d1d20 con 0x7f2c50004800 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5000bf00 con 0x7f2c50003a20 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.681+0000 7f2c6b400640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c500051c0 0x7f2c5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6aa00640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c50003a20 0x7f2c50003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6a000640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c50004800 0x7f2c50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6b400640 1 -- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c50003a20 msgr2=0x7f2c50003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6b400640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c50003a20 0x7f2c50003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6b400640 1 -- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c50004800 msgr2=0x7f2c50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6b400640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c50004800 0x7f2c50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.682+0000 7f2c6b400640 1 -- 172.21.15.67:0/2295880093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c50003520 con 0x7f2c500051c0 2024-04-17T07:32:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c6b400640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c500051c0 0x7f2c5000b9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c240 tx=0x7f2c50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eacbd5362d22e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64007d50 con 0x7f2c500051c0 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c64030070 con 0x7f2c500051c0 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c69600640 1 -- 172.21.15.67:0/2295880093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6402cdb0 con 0x7f2c500051c0 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c6be00640 1 -- 172.21.15.67:0/2295880093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c500051c0 msgr2=0x7f2c5000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c6be00640 1 --2- 172.21.15.67:0/2295880093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c500051c0 0x7f2c5000b9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f2c6400c240 tx=0x7f2c50005830 comp rx=0 tx=0).stop 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.684+0000 7f2c6b400640 1 -- 172.21.15.67:0/2295880093 reap_dead start 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.685+0000 7f2c6be00640 1 -- 172.21.15.67:0/2295880093 shutdown_connections 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.685+0000 7f2c6be00640 1 -- 172.21.15.67:0/2295880093 >> 172.21.15.67:0/2295880093 conn(0x7f2c6c06d3c0 msgr2=0x7f2c6c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.685+0000 7f2c6be00640 1 -- 172.21.15.67:0/2295880093 shutdown_connections 2024-04-17T07:32:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.685+0000 7f2c6be00640 1 -- 172.21.15.67:0/2295880093 wait complete. 2024-04-17T07:32:19.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.686+0000 7f2c6be00640 1 Processor -- start 2024-04-17T07:32:19.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.687+0000 7f2c6be00640 1 -- start start 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.687+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c0ca400 0x7f2c6c0ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c6c0ccff0 0x7f2c6c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c15f200 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c0b8040 con 0x7f2c6c0ccff0 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6c0d37e0 con 0x7f2c6c0ca400 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33702/0 (socket says 172.21.15.67:33702) 2024-04-17T07:32:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 -- 172.21.15.67:0/3968192205 learned_addr learned my addr 172.21.15.67:0/3968192205 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6a000640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c0ca400 0x7f2c6c0ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 -- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c0ca400 msgr2=0x7f2c6c0ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c0ca400 0x7f2c6c0ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 -- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c6c0ccff0 msgr2=0x7f2c6c0c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c6c0ccff0 0x7f2c6c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.688+0000 7f2c6aa00640 1 -- 172.21.15.67:0/3968192205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c0cadf0 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c6aa00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c011c80 tx=0x7f2c5c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c019070 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5c003990 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c003d80 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c0cb0d0 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.689+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c158330 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.690+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c068a10 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.691+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 56) ==== 91086+0+0 (secure 0 0 0) 0x7f2c5c01c070 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.692+0000 7f2c62a00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 0x7f2c4806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.692+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6512+0+0 (secure 0 0 0) 0x7f2c5c08cb90 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.693+0000 7f2c6a000640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 0x7f2c4806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:19.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.694+0000 7f2c6a000640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 0x7f2c4806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ce620 tx=0x7f2c54008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:19.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.697+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c5c010040 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.941+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2c6c15db00 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.941 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:32:19.942+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:32:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.953+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f2c5c005d10 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.968+0000 7f2c62a00640 1 -- 172.21.15.67:0/3968192205 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) ==== 161+0+0 (secure 0 0 0) 0x7f2c5c08ae20 con 0x7f2c6c0c84c0 2024-04-17T07:32:19.967 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:32:19.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 msgr2=0x7f2c4806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 0x7f2c4806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ce620 tx=0x7f2c54008040 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 msgr2=0x7f2c6c158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c011c80 tx=0x7f2c5c00f580 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 shutdown_connections 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f2c4806c9b0 0x7f2c4806ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c6c0ccff0 0x7f2c6c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c6c0ca400 0x7f2c6c0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 --2- 172.21.15.67:0/3968192205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c6c0c84c0 0x7f2c6c158b30 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 >> 172.21.15.67:0/3968192205 conn(0x7f2c6c06d3c0 msgr2=0x7f2c6c067d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.971+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 shutdown_connections 2024-04-17T07:32:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:19.972+0000 7f2c6be00640 1 -- 172.21.15.67:0/3968192205 wait complete. 2024-04-17T07:32:19.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:32:19.992 INFO:tasks.ceph.mds.d.smithi187.stderr: -7931> 2024-04-17T07:30:53.899+0000 7f8874400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:32:20.053 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:32:20.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.212+0000 7f6739200640 1 Processor -- start 2024-04-17T07:32:20.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.212+0000 7f6739200640 1 -- start start 2024-04-17T07:32:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.214+0000 7f6739200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f673406f4f0 0x7f673415f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.214+0000 7f6739200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.214+0000 7f6739200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673406f990 con 0x7f673415f110 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.214+0000 7f6739200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67340d0af0 con 0x7f673406f4f0 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.214+0000 7f6739200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f673406fc70 con 0x7f673415fa30 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.215+0000 7f6729400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f673406f4f0 0x7f673415f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.215+0000 7f6732800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.215+0000 7f6732800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:53114/0 (socket says 172.21.15.67:53114) 2024-04-17T07:32:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.215+0000 7f6732800640 1 -- 172.21.15.67:0/306044514 learned_addr learned my addr 172.21.15.67:0/306044514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.216+0000 7f6732800640 1 -- 172.21.15.67:0/306044514 >> v2:172.21.15.187:3300/0 conn(0x7f673406f4f0 msgr2=0x7f673415f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.216+0000 7f6732800640 1 --2- 172.21.15.67:0/306044514 >> v2:172.21.15.187:3300/0 conn(0x7f673406f4f0 0x7f673415f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.216+0000 7f6732800640 1 -- 172.21.15.67:0/306044514 >> v1:172.21.15.187:6789/0 conn(0x7f673415f110 legacy=0x7f6734072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.216+0000 7f6732800640 1 -- 172.21.15.67:0/306044514 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67340d04a0 con 0x7f673415fa30 2024-04-17T07:32:20.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.217+0000 7f6732800640 1 --2- 172.21.15.67:0/306044514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f672c009e50 tx=0x7f672c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc2209d64ce37044 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:20.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6731400640 1 -- 172.21.15.67:0/306044514 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c02cb00 con 0x7f673415fa30 2024-04-17T07:32:20.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6731400640 1 -- 172.21.15.67:0/306044514 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c02cca0 con 0x7f673415fa30 2024-04-17T07:32:20.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6731400640 1 -- 172.21.15.67:0/306044514 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c034610 con 0x7f673415fa30 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 -- 172.21.15.67:0/306044514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 msgr2=0x7f67341686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 --2- 172.21.15.67:0/306044514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f672c009e50 tx=0x7f672c02bf60 comp rx=0 tx=0).stop 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 -- 172.21.15.67:0/306044514 shutdown_connections 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 --2- 172.21.15.67:0/306044514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673415fa30 0x7f67341686c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 --2- 172.21.15.67:0/306044514 >> v2:172.21.15.187:3300/0 conn(0x7f673406f4f0 0x7f673415f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 -- 172.21.15.67:0/306044514 >> 172.21.15.67:0/306044514 conn(0x7f6734067ee0 msgr2=0x7f673406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 -- 172.21.15.67:0/306044514 shutdown_connections 2024-04-17T07:32:20.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.218+0000 7f6739200640 1 -- 172.21.15.67:0/306044514 wait complete. 2024-04-17T07:32:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.220+0000 7f6739200640 1 Processor -- start 2024-04-17T07:32:20.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.220+0000 7f6739200640 1 -- start start 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673406f4f0 0x7f67341569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 0x7f673415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f673415fec0 con 0x7f673415fa30 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f67340afdc0 con 0x7f673415f110 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.221+0000 7f6739200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67340cf3c0 con 0x7f673406f4f0 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33708/0 (socket says 172.21.15.67:33708) 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 -- 172.21.15.67:0/2733558836 learned_addr learned my addr 172.21.15.67:0/2733558836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6729400640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 0x7f673415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6731e00640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673406f4f0 0x7f67341569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 -- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673406f4f0 msgr2=0x7f67341569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673406f4f0 0x7f67341569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 -- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 msgr2=0x7f673415e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 0x7f673415e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6732800640 1 -- 172.21.15.67:0/2733558836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67341e7730 con 0x7f673415fa30 2024-04-17T07:32:20.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.222+0000 7f6729400640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 0x7f673415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f6732800640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f672c009e50 tx=0x7f672c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c043020 con 0x7f673415fa30 2024-04-17T07:32:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c02ce50 con 0x7f673415fa30 2024-04-17T07:32:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67341e7a10 con 0x7f673415fa30 2024-04-17T07:32:20.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c009510 con 0x7f673415fa30 2024-04-17T07:32:20.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.223+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67341e7fd0 con 0x7f673415fa30 2024-04-17T07:32:20.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.224+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6734068ca0 con 0x7f673415fa30 2024-04-17T07:32:20.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.225+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f672c046020 con 0x7f673415fa30 2024-04-17T07:32:20.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.226+0000 7f672aa00640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 0x7f671006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:20.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.226+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6598+0+0 (secure 0 0 0) 0x7f672c0ae7e0 con 0x7f673415fa30 2024-04-17T07:32:20.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.227+0000 7f6731e00640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 0x7f671006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:20.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.228+0000 7f6731e00640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 0x7f671006eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6724001640 tx=0x7f6724009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:20.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.230+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f672c04d030 con 0x7f673415fa30 2024-04-17T07:32:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.479+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f673406ba60 con 0x7f673415fa30 2024-04-17T07:32:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.985+0000 7f672aa00640 1 -- 172.21.15.67:0/2733558836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) ==== 104+0+0 (secure 0 0 0) 0x7f672c03c350 con 0x7f673415fa30 2024-04-17T07:32:20.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 msgr2=0x7f671006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 0x7f671006eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6724001640 tx=0x7f6724009040 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 msgr2=0x7f673414be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f672c009e50 tx=0x7f672c004a40 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 shutdown_connections 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f671006ca10 0x7f671006eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673415fa30 0x7f673414be90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f673415f110 0x7f673415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 --2- 172.21.15.67:0/2733558836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f673406f4f0 0x7f67341569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 >> 172.21.15.67:0/2733558836 conn(0x7f6734067ee0 msgr2=0x7f67340b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 shutdown_connections 2024-04-17T07:32:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:20.989+0000 7f6739200640 1 -- 172.21.15.67:0/2733558836 wait complete. 2024-04-17T07:32:21.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:32:21.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.241+0000 7f004ca00640 1 Processor -- start 2024-04-17T07:32:21.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.241+0000 7f004ca00640 1 -- start start 2024-04-17T07:32:21.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f004ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f004806f220 0x7f00481434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f004ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0048072430 0x7f00481439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f004ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00481612d0 con 0x7f0048142e10 2024-04-17T07:32:21.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f004ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f00481431f0 con 0x7f0048072430 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f004ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0048160e40 con 0x7f004806f220 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f0047400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0048142e10 0x7f0048071d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54350/0 (socket says 172.21.15.67:54350) 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f0047400640 1 -- 172.21.15.67:0/671407843 learned_addr learned my addr 172.21.15.67:0/671407843 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.243+0000 7f0047e00640 1 --2- 172.21.15.67:0/671407843 >> v2:172.21.15.67:3300/0 conn(0x7f0048072430 0x7f00481439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0046a00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f004806f220 0x7f00481434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:21.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0046000640 1 -- 172.21.15.67:0/671407843 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1706682440 0 0) 0x7f00481612d0 con 0x7f0048142e10 2024-04-17T07:32:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0046000640 1 -- 172.21.15.67:0/671407843 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f002c003520 con 0x7f0048142e10 2024-04-17T07:32:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0047e00640 1 -- 172.21.15.67:0/671407843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f004806f220 msgr2=0x7f00481434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0047e00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f004806f220 0x7f00481434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0047e00640 1 -- 172.21.15.67:0/671407843 >> v1:172.21.15.67:6789/0 conn(0x7f0048142e10 legacy=0x7f0048071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.244+0000 7f0047e00640 1 -- 172.21.15.67:0/671407843 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0048160510 con 0x7f0048072430 2024-04-17T07:32:21.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.245+0000 7f0047e00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048072430 0x7f00481439e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0040009d90 tx=0x7f004002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708212ce953ad6d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:21.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.245+0000 7f0046000640 1 -- 172.21.15.67:0/671407843 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040039070 con 0x7f0048072430 2024-04-17T07:32:21.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.245+0000 7f0046000640 1 -- 172.21.15.67:0/671407843 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f004002cbc0 con 0x7f0048072430 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.245+0000 7f0046000640 1 -- 172.21.15.67:0/671407843 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040034440 con 0x7f0048072430 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 -- 172.21.15.67:0/671407843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048072430 msgr2=0x7f00481439e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048072430 0x7f00481439e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f0040009d90 tx=0x7f004002be10 comp rx=0 tx=0).stop 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 -- 172.21.15.67:0/671407843 shutdown_connections 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048072430 0x7f00481439e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 --2- 172.21.15.67:0/671407843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f004806f220 0x7f00481434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 -- 172.21.15.67:0/671407843 >> 172.21.15.67:0/671407843 conn(0x7f004806b8c0 msgr2=0x7f004806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 -- 172.21.15.67:0/671407843 shutdown_connections 2024-04-17T07:32:21.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.246+0000 7f004ca00640 1 -- 172.21.15.67:0/671407843 wait complete. 2024-04-17T07:32:21.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.248+0000 7f004ca00640 1 Processor -- start 2024-04-17T07:32:21.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.248+0000 7f004ca00640 1 -- start start 2024-04-17T07:32:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0048072430 0x7f00480b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00481f1f00 0x7f00481f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00481460a0 con 0x7f0048142e10 2024-04-17T07:32:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0048160ad0 con 0x7f00481f1f00 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.249+0000 7f004ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0048146220 con 0x7f0048072430 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33730/0 (socket says 172.21.15.67:33730) 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 -- 172.21.15.67:0/4200075963 learned_addr learned my addr 172.21.15.67:0/4200075963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0047400640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0048072430 0x7f00480b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 -- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0048072430 msgr2=0x7f00480b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0048072430 0x7f00480b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 -- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00481f1f00 msgr2=0x7f00481f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00481f1f00 0x7f00481f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 -- 172.21.15.67:0/4200075963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00480b3bb0 con 0x7f0048142e10 2024-04-17T07:32:21.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.250+0000 7f0046a00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f003400d620 tx=0x7f003400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:21.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.251+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400b620 con 0x7f0048142e10 2024-04-17T07:32:21.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.251+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0034019070 con 0x7f0048142e10 2024-04-17T07:32:21.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.251+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00480b3e90 con 0x7f0048142e10 2024-04-17T07:32:21.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.251+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400ba00 con 0x7f0048142e10 2024-04-17T07:32:21.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.251+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00480b4450 con 0x7f0048142e10 2024-04-17T07:32:21.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.252+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0048068a10 con 0x7f0048142e10 2024-04-17T07:32:21.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.253+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f003400bba0 con 0x7f0048142e10 2024-04-17T07:32:21.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.254+0000 7f003f400640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 0x7f002806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:21.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.254+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6598+0+0 (secure 0 0 0) 0x7f003408c450 con 0x7f0048142e10 2024-04-17T07:32:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.255+0000 7f0047400640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 0x7f002806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.256+0000 7f0047400640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 0x7f002806eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00481f7d10 tx=0x7f0038009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:21.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.258+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f003405a3e0 con 0x7f0048142e10 2024-04-17T07:32:21.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.472+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0048072e30 con 0x7f0048142e10 2024-04-17T07:32:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:21.995+0000 7f003f400640 1 -- 172.21.15.67:0/4200075963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v117) ==== 181+0+0 (secure 0 0 0) 0x7f0034059ce0 con 0x7f0048142e10 2024-04-17T07:32:21.995 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:32:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.001+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 msgr2=0x7f002806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.001+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 0x7f002806eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00481f7d10 tx=0x7f0038009040 comp rx=0 tx=0).stop 2024-04-17T07:32:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.001+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 msgr2=0x7f00481f5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.001+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f003400d620 tx=0x7f003400dae0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 shutdown_connections 2024-04-17T07:32:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f002806ca10 0x7f002806eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00481f1f00 0x7f00481f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048142e10 0x7f00481f5260 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 --2- 172.21.15.67:0/4200075963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0048072430 0x7f00480b3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 >> 172.21.15.67:0/4200075963 conn(0x7f004806b8c0 msgr2=0x7f0048143550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 shutdown_connections 2024-04-17T07:32:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.002+0000 7f004ca00640 1 -- 172.21.15.67:0/4200075963 wait complete. 2024-04-17T07:32:22.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:32:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.250+0000 7f34ed600640 1 Processor -- start 2024-04-17T07:32:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.250+0000 7f34ed600640 1 -- start start 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.251+0000 7f34ed600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e80cd2f0 0x7f34e80cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.251+0000 7f34ed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e8146250 0x7f34e8160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.251+0000 7f34ed600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e8161000 con 0x7f34e80cd6d0 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.251+0000 7f34ed600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f34e80ce870 con 0x7f34e8146250 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.251+0000 7f34ed600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34e80ceb50 con 0x7f34e80cd2f0 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e8146250 0x7f34e8160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e8146250 0x7f34e8160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33740/0 (socket says 172.21.15.67:33740) 2024-04-17T07:32:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 -- 172.21.15.67:0/184357342 learned_addr learned my addr 172.21.15.67:0/184357342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34e7e00640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e80cd2f0 0x7f34e80cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 -- 172.21.15.67:0/184357342 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e80cd2f0 msgr2=0x7f34e80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e80cd2f0 0x7f34e80cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.252+0000 7f34ecc00640 1 -- 172.21.15.67:0/184357342 >> v1:172.21.15.187:6789/0 conn(0x7f34e80cd6d0 legacy=0x7f34e8145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.253+0000 7f34ecc00640 1 -- 172.21.15.67:0/184357342 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e80cee30 con 0x7f34e8146250 2024-04-17T07:32:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.254+0000 7f34ecc00640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e8146250 0x7f34e8160360 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e43e0115b528d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.254+0000 7f34e7400640 1 -- 172.21.15.67:0/184357342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0035020 con 0x7f34e8146250 2024-04-17T07:32:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.254+0000 7f34e7400640 1 -- 172.21.15.67:0/184357342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e0031640 con 0x7f34e8146250 2024-04-17T07:32:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.254+0000 7f34e7400640 1 -- 172.21.15.67:0/184357342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e002d040 con 0x7f34e8146250 2024-04-17T07:32:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 -- 172.21.15.67:0/184357342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e8146250 msgr2=0x7f34e8160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e8146250 0x7f34e8160360 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009870 tx=0x7f34e002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 -- 172.21.15.67:0/184357342 shutdown_connections 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e8146250 0x7f34e8160360 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 --2- 172.21.15.67:0/184357342 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e80cd2f0 0x7f34e80cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 -- 172.21.15.67:0/184357342 >> 172.21.15.67:0/184357342 conn(0x7f34e806b8c0 msgr2=0x7f34e806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 -- 172.21.15.67:0/184357342 shutdown_connections 2024-04-17T07:32:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.255+0000 7f34ed600640 1 -- 172.21.15.67:0/184357342 wait complete. 2024-04-17T07:32:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.257+0000 7f34ed600640 1 Processor -- start 2024-04-17T07:32:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.257+0000 7f34ed600640 1 -- start start 2024-04-17T07:32:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34e80cd6d0 0x7f34e81f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e8146250 0x7f34e806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8160a20 con 0x7f34e80cd2f0 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81608a0 con 0x7f34e80cd6d0 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34ed600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34e8160ba0 con 0x7f34e8146250 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33748/0 (socket says 172.21.15.67:33748) 2024-04-17T07:32:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.258+0000 7f34e7e00640 1 -- 172.21.15.67:0/2708456287 learned_addr learned my addr 172.21.15.67:0/2708456287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 -- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e8146250 msgr2=0x7f34e806ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e8146250 0x7f34e806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 -- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34e80cd6d0 msgr2=0x7f34e81f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34df400640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34e80cd6d0 0x7f34e81f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34e80cd6d0 0x7f34e81f85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 -- 172.21.15.67:0/2708456287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e80adf50 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.259+0000 7f34e7e00640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007e40 tx=0x7f34d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.260+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8014620 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.260+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d8019070 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.260+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8014a00 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.260+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e80ae230 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.260+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e80ae7f0 con 0x7f34e80cd2f0 2024-04-17T07:32:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.261+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e8072ea0 con 0x7f34e80cd2f0 2024-04-17T07:32:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.262+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f34d8014ba0 con 0x7f34e80cd2f0 2024-04-17T07:32:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.263+0000 7f34e4c00640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 0x7f34c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.263+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6160+0+0 (secure 0 0 0) 0x7f34d808c700 con 0x7f34e80cd2f0 2024-04-17T07:32:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.264+0000 7f34df400640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 0x7f34c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.265+0000 7f34df400640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 0x7f34c406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f34e80cf690 tx=0x7f34d0008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.267+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34e8072ea0 con 0x7f34e80cd2f0 2024-04-17T07:32:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:22.471+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f34e815c9d0 con 0x7f34e80cd2f0 2024-04-17T07:32:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.008+0000 7f34e4c00640 1 -- 172.21.15.67:0/2708456287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v118) ==== 169+0+0 (secure 0 0 0) 0x7f34d805a870 con 0x7f34e80cd2f0 2024-04-17T07:32:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:32:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.013+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 msgr2=0x7f34c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.013+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 0x7f34c406eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f34e80cf690 tx=0x7f34d0008040 comp rx=0 tx=0).stop 2024-04-17T07:32:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.013+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 msgr2=0x7f34e80aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.013+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007e40 tx=0x7f34d800e4b0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 shutdown_connections 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f34c406ca10 0x7f34c406eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34e8146250 0x7f34e806ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34e80cd6d0 0x7f34e81f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 --2- 172.21.15.67:0/2708456287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80cd2f0 0x7f34e80aeff0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 >> 172.21.15.67:0/2708456287 conn(0x7f34e806b8c0 msgr2=0x7f34e80cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 shutdown_connections 2024-04-17T07:32:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.014+0000 7f34ed600640 1 -- 172.21.15.67:0/2708456287 wait complete. 2024-04-17T07:32:23.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:32:23.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.244+0000 7f8bbb200640 1 Processor -- start 2024-04-17T07:32:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.244+0000 7f8bbb200640 1 -- start start 2024-04-17T07:32:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bbb200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8bbc0af400 0x7f8bbc162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8bbc06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bbb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bbc15f780 con 0x7f8bbc159020 2024-04-17T07:32:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bbb200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8bbc15f540 con 0x7f8bbc0af400 2024-04-17T07:32:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bbb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8bbc15fa60 con 0x7f8bbc06b8c0 2024-04-17T07:32:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bb9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8bbc159020 0x7f8bbc15ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54354/0 (socket says 172.21.15.67:54354) 2024-04-17T07:32:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.246+0000 7f8bb9e00640 1 -- 172.21.15.67:0/3175001946 learned_addr learned my addr 172.21.15.67:0/3175001946 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb9400640 1 --2- 172.21.15.67:0/3175001946 >> v2:172.21.15.187:3300/0 conn(0x7f8bbc0af400 0x7f8bbc162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bba800640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8bbc06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953583197 0 0) 0x7f8bbc15f780 con 0x7f8bbc159020 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba0003520 con 0x7f8bbc159020 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 472572481 0 0) 0x7f8ba0003520 con 0x7f8bbc159020 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 msgr2=0x7f8bbc06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8bbc06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 >> v2:172.21.15.187:3300/0 conn(0x7f8bbc0af400 msgr2=0x7f8bbc162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 --2- 172.21.15.67:0/3175001946 >> v2:172.21.15.187:3300/0 conn(0x7f8bbc0af400 0x7f8bbc162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc1624a0 con 0x7f8bbc159020 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8bac001670 con 0x7f8bbc159020 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.247+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 >> v1:172.21.15.67:6789/0 conn(0x7f8bbc159020 legacy=0x7f8bbc15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba00039b0 0x7f8ba0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8ba0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc15f540 con 0x7f8ba0004d40 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc15fa60 con 0x7f8ba00039b0 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba000b9c0 con 0x7f8bbc06b8c0 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb9400640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8ba0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bb9e00640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba00039b0 0x7f8ba0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 -- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 msgr2=0x7f8ba0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc06b8c0 0x7f8ba0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 -- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba00039b0 msgr2=0x7f8ba0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba00039b0 0x7f8ba0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.248+0000 7f8bba800640 1 -- 172.21.15.67:0/3175001946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba0003520 con 0x7f8ba0004d40 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bba800640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009fc0 tx=0x7f8ba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5edb4d4ba4312f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4039070 con 0x7f8ba0004d40 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bb402cc20 con 0x7f8ba0004d40 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bb8a00640 1 -- 172.21.15.67:0/3175001946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4009c10 con 0x7f8ba0004d40 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bbb200640 1 -- 172.21.15.67:0/3175001946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba0004d40 msgr2=0x7f8ba00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bbb200640 1 --2- 172.21.15.67:0/3175001946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba0004d40 0x7f8ba00051c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009fc0 tx=0x7f8ba0005830 comp rx=0 tx=0).stop 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.250+0000 7f8bba800640 1 -- 172.21.15.67:0/3175001946 reap_dead start 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.251+0000 7f8bbb200640 1 -- 172.21.15.67:0/3175001946 shutdown_connections 2024-04-17T07:32:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.251+0000 7f8bbb200640 1 -- 172.21.15.67:0/3175001946 >> 172.21.15.67:0/3175001946 conn(0x7f8bbc0b2190 msgr2=0x7f8bbc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.251+0000 7f8bbb200640 1 -- 172.21.15.67:0/3175001946 shutdown_connections 2024-04-17T07:32:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.251+0000 7f8bbb200640 1 -- 172.21.15.67:0/3175001946 wait complete. 2024-04-17T07:32:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.252+0000 7f8bbb200640 1 Processor -- start 2024-04-17T07:32:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.253+0000 7f8bbb200640 1 -- start start 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.253+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bbc153630 0x7f8bbc154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.253+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc1550c0 0x7f8bbc15db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bbb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc1631e0 con 0x7f8bbc15e070 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bbb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc15fd40 con 0x7f8bbc153630 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bbb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8bbc163360 con 0x7f8bbc1550c0 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33778/0 (socket says 172.21.15.67:33778) 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 -- 172.21.15.67:0/2764890212 learned_addr learned my addr 172.21.15.67:0/2764890212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bb9e00640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bbc153630 0x7f8bbc154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 -- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc1550c0 msgr2=0x7f8bbc15db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bb9400640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc1550c0 0x7f8bbc15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc1550c0 0x7f8bbc15db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 -- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bbc153630 msgr2=0x7f8bbc154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bbc153630 0x7f8bbc154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.254+0000 7f8bba800640 1 -- 172.21.15.67:0/2764890212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc15a6f0 con 0x7f8bbc15e070 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bba800640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8bb402c690 tx=0x7f8bb4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4039070 con 0x7f8bbc15e070 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bb4044070 con 0x7f8bbc15e070 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4009c10 con 0x7f8bbc15e070 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc155660 con 0x7f8bbc15e070 2024-04-17T07:32:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.255+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc15a990 con 0x7f8bbc15e070 2024-04-17T07:32:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.257+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bbc15e4f0 con 0x7f8bbc15e070 2024-04-17T07:32:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.257+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f8bb403e810 con 0x7f8bbc15e070 2024-04-17T07:32:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.258+0000 7f8bb2000640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 0x7f8b9c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.259+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f8bb40adef0 con 0x7f8bbc15e070 2024-04-17T07:32:23.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.259+0000 7f8bb9e00640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 0x7f8b9c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.260+0000 7f8bb9e00640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 0x7f8b9c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bac004840 tx=0x7f8bac009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.263+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8bb407c290 con 0x7f8bbc15e070 2024-04-17T07:32:23.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.484+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f8bbc068a10 con 0x7f8bbc15e070 2024-04-17T07:32:23.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.492+0000 7f8bb2000640 1 -- 172.21.15.67:0/2764890212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) ==== 115+0+0 (secure 0 0 0) 0x7f8bb407bb90 con 0x7f8bbc15e070 2024-04-17T07:32:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 msgr2=0x7f8b9c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 0x7f8b9c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bac004840 tx=0x7f8bac009040 comp rx=0 tx=0).stop 2024-04-17T07:32:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 msgr2=0x7f8bbc159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8bb402c690 tx=0x7f8bb4009750 comp rx=0 tx=0).stop 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 shutdown_connections 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f8b9c06ca40 0x7f8b9c06ef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bbc15e070 0x7f8bbc159490 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bbc1550c0 0x7f8bbc15db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 --2- 172.21.15.67:0/2764890212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bbc153630 0x7f8bbc154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 >> 172.21.15.67:0/2764890212 conn(0x7f8bbc0b2190 msgr2=0x7f8bbc0d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 shutdown_connections 2024-04-17T07:32:23.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.496+0000 7f8bbb200640 1 -- 172.21.15.67:0/2764890212 wait complete. 2024-04-17T07:32:23.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:32:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.724+0000 7f14ed400640 1 Processor -- start 2024-04-17T07:32:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.724+0000 7f14ed400640 1 -- start start 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14ed400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14e8145f80 0x7f14e80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14ed400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 0x7f14e806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14ed400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e806f1d0 con 0x7f14e806c0a0 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14ed400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f14e80727a0 con 0x7f14e8145f80 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14ed400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8146360 con 0x7f14e80cd750 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14e8145f80 0x7f14e80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14e8145f80 0x7f14e80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34738/0 (socket says 172.21.15.67:34738) 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14e7e00640 1 -- 172.21.15.67:0/2778301640 learned_addr learned my addr 172.21.15.67:0/2778301640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.726+0000 7f14eca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f14e806c0a0 0x7f14e8160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42638/0 (socket says 172.21.15.67:42638) 2024-04-17T07:32:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7400640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 0x7f14e806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7e00640 1 -- 172.21.15.67:0/2778301640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 msgr2=0x7f14e806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7e00640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 0x7f14e806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7e00640 1 -- 172.21.15.67:0/2778301640 >> v1:172.21.15.67:6789/0 conn(0x7f14e806c0a0 legacy=0x7f14e8160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7e00640 1 -- 172.21.15.67:0/2778301640 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e8071e70 con 0x7f14e8145f80 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e7400640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 0x7f14e806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.727+0000 7f14e6a00640 1 -- 172.21.15.67:0/2778301640 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355288727 0 0) 0x7f14e0001490 con 0x7f14e806c0a0 2024-04-17T07:32:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.728+0000 7f14e7e00640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80cd210 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009870 tx=0x7f14d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb262e2522de282 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.728+0000 7f14e6a00640 1 -- 172.21.15.67:0/2778301640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8035020 con 0x7f14e8145f80 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.728+0000 7f14e6a00640 1 -- 172.21.15.67:0/2778301640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d8031640 con 0x7f14e8145f80 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.728+0000 7f14e6a00640 1 -- 172.21.15.67:0/2778301640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d802d040 con 0x7f14e8145f80 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 -- 172.21.15.67:0/2778301640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 msgr2=0x7f14e80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80cd210 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009870 tx=0x7f14d802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 -- 172.21.15.67:0/2778301640 shutdown_connections 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e80cd750 0x7f14e806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 --2- 172.21.15.67:0/2778301640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80cd210 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 -- 172.21.15.67:0/2778301640 >> 172.21.15.67:0/2778301640 conn(0x7f14e80b2190 msgr2=0x7f14e80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 -- 172.21.15.67:0/2778301640 shutdown_connections 2024-04-17T07:32:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.729+0000 7f14ed400640 1 -- 172.21.15.67:0/2778301640 wait complete. 2024-04-17T07:32:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.731+0000 7f14ed400640 1 Processor -- start 2024-04-17T07:32:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.731+0000 7f14ed400640 1 -- start start 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e806c0a0 0x7f14e81406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 0x7f14e8140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14e8072c30 con 0x7f14e8145f80 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14e8072430 con 0x7f14e80cd750 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14ed400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8072db0 con 0x7f14e806c0a0 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14eca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14eca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34740/0 (socket says 172.21.15.67:34740) 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.732+0000 7f14eca00640 1 -- 172.21.15.67:0/3774211232 learned_addr learned my addr 172.21.15.67:0/3774211232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14e7e00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e806c0a0 0x7f14e81406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 -- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e806c0a0 msgr2=0x7f14e81406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e806c0a0 0x7f14e81406e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14e7400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 0x7f14e8140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 -- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 msgr2=0x7f14e8140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 0x7f14e8140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 -- 172.21.15.67:0/3774211232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e815c500 con 0x7f14e8145f80 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14e7400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 0x7f14e8140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14eca00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f14e000c7f0 tx=0x7f14e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0018070 con 0x7f14e8145f80 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e000a5e0 con 0x7f14e8145f80 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.733+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e000aa10 con 0x7f14e8145f80 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.734+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e80c9b40 con 0x7f14e8145f80 2024-04-17T07:32:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.734+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e80ca100 con 0x7f14e8145f80 2024-04-17T07:32:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.735+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e8068a10 con 0x7f14e8145f80 2024-04-17T07:32:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.736+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f14e000abb0 con 0x7f14e8145f80 2024-04-17T07:32:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.737+0000 7f14dfe00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 0x7f14c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.737+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f14e008b350 con 0x7f14e8145f80 2024-04-17T07:32:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.738+0000 7f14e7e00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 0x7f14c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.739+0000 7f14e7e00640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 0x7f14c806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14d8036000 tx=0x7f14d8008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:23.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.742+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f14e806f1d0 con 0x7f14e8145f80 2024-04-17T07:32:23.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.948+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f14e815de00 con 0x7f14e8145f80 2024-04-17T07:32:23.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.955+0000 7f14dfe00640 1 -- 172.21.15.67:0/3774211232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7f14e0059650 con 0x7f14e8145f80 2024-04-17T07:32:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 msgr2=0x7f14c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 0x7f14c806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f14d8036000 tx=0x7f14d8008030 comp rx=0 tx=0).stop 2024-04-17T07:32:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 msgr2=0x7f14e80ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f14e000c7f0 tx=0x7f14e000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 shutdown_connections 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f14c806ca10 0x7f14c806eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e8145f80 0x7f14e80ca900 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80cd750 0x7f14e8140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 --2- 172.21.15.67:0/3774211232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14e806c0a0 0x7f14e81406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 >> 172.21.15.67:0/3774211232 conn(0x7f14e80b2190 msgr2=0x7f14e815ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 shutdown_connections 2024-04-17T07:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:23.958+0000 7f14ed400640 1 -- 172.21.15.67:0/3774211232 wait complete. 2024-04-17T07:32:23.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:32:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.202+0000 7f3fd8200640 1 Processor -- start 2024-04-17T07:32:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.202+0000 7f3fd8200640 1 -- start start 2024-04-17T07:32:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd8200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3fd0071ea0 0x7f3fd00ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd8200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fd00cea70 con 0x7f3fd0145ac0 2024-04-17T07:32:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd8200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fd00ced50 con 0x7f3fd0072d80 2024-04-17T07:32:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd8200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3fd00cf030 con 0x7f3fd0071ea0 2024-04-17T07:32:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd5800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3fd0145ac0 0x7f3fd0072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42654/0 (socket says 172.21.15.67:42654) 2024-04-17T07:32:24.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.204+0000 7f3fd5800640 1 -- 172.21.15.67:0/2909427922 learned_addr learned my addr 172.21.15.67:0/2909427922 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:24.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018401693 0 0) 0x7f3fd00cea70 con 0x7f3fd0145ac0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fd6200640 1 --2- 172.21.15.67:0/2909427922 >> v2:172.21.15.187:3300/0 conn(0x7f3fd0071ea0 0x7f3fd00ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fb8003520 con 0x7f3fd0145ac0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2155963457 0 0) 0x7f3fb8003520 con 0x7f3fd0145ac0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 >> v2:172.21.15.187:3300/0 conn(0x7f3fd0071ea0 msgr2=0x7f3fd00ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 --2- 172.21.15.67:0/2909427922 >> v2:172.21.15.187:3300/0 conn(0x7f3fd0071ea0 0x7f3fd00ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 >> v1:172.21.15.187:6789/0 conn(0x7f3fd0072d80 legacy=0x7f3fd00cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd00cf310 con 0x7f3fd0145ac0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3fc4001670 con 0x7f3fd0145ac0 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 >> v1:172.21.15.67:6789/0 conn(0x7f3fd0145ac0 legacy=0x7f3fd0072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb80039b0 0x7f3fb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 0x7f3fb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.205+0000 7f3fcfe00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fb80051c0 0x7f3fb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd00cf030 con 0x7f3fb8004800 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb800c110 con 0x7f3fb80039b0 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb800c310 con 0x7f3fb80051c0 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd6200640 1 --2- 172.21.15.67:0/2909427922 >> v2:172.21.15.187:3300/0 conn(0x7f3fd0071ea0 0x7f3fd00ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 0x7f3fb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd5800640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb80039b0 0x7f3fb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 -- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fb80051c0 msgr2=0x7f3fb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fb80051c0 0x7f3fb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 -- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb80039b0 msgr2=0x7f3fb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb80039b0 0x7f3fb8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd4e00640 1 -- 172.21.15.67:0/2909427922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb8003520 con 0x7f3fb8004800 2024-04-17T07:32:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.206+0000 7f3fd6200640 1 -- 172.21.15.67:0/2909427922 reap_dead start 2024-04-17T07:32:24.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.207+0000 7f3fd4e00640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 0x7f3fb8004c80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc001750 tx=0x7f3fb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1977967f8e951b21 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc001370 con 0x7f3fb8004800 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc0052d0 con 0x7f3fb8004800 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fcfe00640 1 -- 172.21.15.67:0/2909427922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc003830 con 0x7f3fb8004800 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 -- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 msgr2=0x7f3fb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 0x7f3fb8004c80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc001750 tx=0x7f3fb8005830 comp rx=0 tx=0).stop 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 -- 172.21.15.67:0/2909427922 shutdown_connections 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 --2- 172.21.15.67:0/2909427922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fb8004800 0x7f3fb8004c80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 -- 172.21.15.67:0/2909427922 >> 172.21.15.67:0/2909427922 conn(0x7f3fd0067ee0 msgr2=0x7f3fd006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.208+0000 7f3fd8200640 1 -- 172.21.15.67:0/2909427922 shutdown_connections 2024-04-17T07:32:24.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.209+0000 7f3fd8200640 1 -- 172.21.15.67:0/2909427922 wait complete. 2024-04-17T07:32:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.210+0000 7f3fd8200640 1 Processor -- start 2024-04-17T07:32:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.210+0000 7f3fd8200640 1 -- start start 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 0x7f3fd0152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fd006e720 0x7f3fd01f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd00ce3b0 con 0x7f3fd015cf40 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd0145ea0 con 0x7f3fb8004800 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.211+0000 7f3fd8200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd00ce530 con 0x7f3fd006e720 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34750/0 (socket says 172.21.15.67:34750) 2024-04-17T07:32:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 -- 172.21.15.67:0/250823409 learned_addr learned my addr 172.21.15.67:0/250823409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd4e00640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fd006e720 0x7f3fd01f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd5800640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 0x7f3fd0152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 -- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fd006e720 msgr2=0x7f3fd01f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fd006e720 0x7f3fd01f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 -- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 msgr2=0x7f3fd0152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 0x7f3fd0152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd6200640 1 -- 172.21.15.67:0/250823409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd006ef50 con 0x7f3fd015cf40 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.212+0000 7f3fd5800640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 0x7f3fd0152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fd6200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fd01467c0 tx=0x7f3fc800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc801a070 con 0x7f3fd015cf40 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc8003990 con 0x7f3fd015cf40 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8003d80 con 0x7f3fd015cf40 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd006f230 con 0x7f3fd015cf40 2024-04-17T07:32:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.213+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd014f980 con 0x7f3fd015cf40 2024-04-17T07:32:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.214+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd00cc6c0 con 0x7f3fd015cf40 2024-04-17T07:32:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.215+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f3fc801d070 con 0x7f3fd015cf40 2024-04-17T07:32:24.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.216+0000 7f3fcd600640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 0x7f3fb006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.216+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f3fc808bd10 con 0x7f3fd015cf40 2024-04-17T07:32:24.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.217+0000 7f3fd5800640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 0x7f3fb006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.218+0000 7f3fd5800640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 0x7f3fb006ef00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4001050 tx=0x7f3fc4001d50 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.223+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3fc8091020 con 0x7f3fd015cf40 2024-04-17T07:32:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.433+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f3fd0144b70 con 0x7f3fd015cf40 2024-04-17T07:32:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.440+0000 7f3fcd600640 1 -- 172.21.15.67:0/250823409 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v39) ==== 120+0+0 (secure 0 0 0) 0x7f3fc8010070 con 0x7f3fd015cf40 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 msgr2=0x7f3fb006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 0x7f3fb006ef00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4001050 tx=0x7f3fc4001d50 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 msgr2=0x7f3fd015d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fd01467c0 tx=0x7f3fc800d4a0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 shutdown_connections 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f3fb006ca40 0x7f3fb006ef00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fd015cf40 0x7f3fd015d3c0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fd006e720 0x7f3fd01f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 --2- 172.21.15.67:0/250823409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fb8004800 0x7f3fd0152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 >> 172.21.15.67:0/250823409 conn(0x7f3fd0067ee0 msgr2=0x7f3fd01440c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:24.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 shutdown_connections 2024-04-17T07:32:24.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.443+0000 7f3fd8200640 1 -- 172.21.15.67:0/250823409 wait complete. 2024-04-17T07:32:24.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-04-17T07:32:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.685+0000 7f75c7200640 1 Processor -- start 2024-04-17T07:32:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.685+0000 7f75c7200640 1 -- start start 2024-04-17T07:32:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.687+0000 7f75c7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.687+0000 7f75c7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.687+0000 7f75c7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75c80cd4a0 con 0x7f75c8141ed0 2024-04-17T07:32:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.687+0000 7f75c7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f75c815ce30 con 0x7f75c8160280 2024-04-17T07:32:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.687+0000 7f75c7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75c81607f0 con 0x7f75c806b8c0 2024-04-17T07:32:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.688+0000 7f75c5400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.688+0000 7f75c5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.688+0000 7f75c5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:43824/0 (socket says 172.21.15.67:43824) 2024-04-17T07:32:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.688+0000 7f75c5400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:41100/0 (socket says 172.21.15.67:41100) 2024-04-17T07:32:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.688+0000 7f75c5e00640 1 -- 172.21.15.67:0/819774540 learned_addr learned my addr 172.21.15.67:0/819774540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.689+0000 7f75c5e00640 1 -- 172.21.15.67:0/819774540 >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 msgr2=0x7f75c806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.689+0000 7f75c5e00640 1 --2- 172.21.15.67:0/819774540 >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.689+0000 7f75c5e00640 1 -- 172.21.15.67:0/819774540 >> v1:172.21.15.187:6789/0 conn(0x7f75c8141ed0 legacy=0x7f75c815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.689+0000 7f75c5e00640 1 -- 172.21.15.67:0/819774540 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c8160ad0 con 0x7f75c806b8c0 2024-04-17T07:32:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.689+0000 7f75c5400640 1 --2- 172.21.15.67:0/819774540 >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.690+0000 7f75c5e00640 1 --2- 172.21.15.67:0/819774540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009870 tx=0x7f75b802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dfd8ba3e9bfa9d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c4a00640 1 -- 172.21.15.67:0/819774540 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8034070 con 0x7f75c806b8c0 2024-04-17T07:32:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c4a00640 1 -- 172.21.15.67:0/819774540 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b8030840 con 0x7f75c806b8c0 2024-04-17T07:32:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c4a00640 1 -- 172.21.15.67:0/819774540 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8030c10 con 0x7f75c806b8c0 2024-04-17T07:32:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 -- 172.21.15.67:0/819774540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 msgr2=0x7f75c80cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 --2- 172.21.15.67:0/819774540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009870 tx=0x7f75b802b980 comp rx=0 tx=0).stop 2024-04-17T07:32:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 -- 172.21.15.67:0/819774540 shutdown_connections 2024-04-17T07:32:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 --2- 172.21.15.67:0/819774540 >> v2:172.21.15.187:3300/0 conn(0x7f75c8160280 0x7f75c806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 --2- 172.21.15.67:0/819774540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c80cd8b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.691+0000 7f75c7200640 1 -- 172.21.15.67:0/819774540 >> 172.21.15.67:0/819774540 conn(0x7f75c8067d10 msgr2=0x7f75c80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.692+0000 7f75c7200640 1 -- 172.21.15.67:0/819774540 shutdown_connections 2024-04-17T07:32:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.692+0000 7f75c7200640 1 -- 172.21.15.67:0/819774540 wait complete. 2024-04-17T07:32:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.693+0000 7f75c7200640 1 Processor -- start 2024-04-17T07:32:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.694+0000 7f75c7200640 1 -- start start 2024-04-17T07:32:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c81f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75c8141ed0 0x7f75c8133360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75c806c1e0 con 0x7f75c8160280 2024-04-17T07:32:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75c80cddf0 con 0x7f75c8141ed0 2024-04-17T07:32:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75c8161090 con 0x7f75c806b8c0 2024-04-17T07:32:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34766/0 (socket says 172.21.15.67:34766) 2024-04-17T07:32:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c6800640 1 -- 172.21.15.67:0/3534669070 learned_addr learned my addr 172.21.15.67:0/3534669070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c5e00640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c81f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.695+0000 7f75c5400640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75c8141ed0 0x7f75c8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 -- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 msgr2=0x7f75c81f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c81f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 -- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75c8141ed0 msgr2=0x7f75c8133360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75c8141ed0 0x7f75c8133360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 -- 172.21.15.67:0/3534669070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c81338a0 con 0x7f75c8160280 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c5e00640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c81f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c6800640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f75c000c710 tx=0x7f75c000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0016020 con 0x7f75c8160280 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c0006d50 con 0x7f75c8160280 2024-04-17T07:32:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0003bb0 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.696+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c81455b0 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.697+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c8145b70 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.698+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c8068a10 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.699+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f75c000a7f0 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.700+0000 7f75be000640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 0x7f75a406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.700+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f75c008b8a0 con 0x7f75c8160280 2024-04-17T07:32:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.700+0000 7f75c5e00640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 0x7f75a406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:24.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.702+0000 7f75c5e00640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 0x7f75a406ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f75b8036000 tx=0x7f75b803b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.704+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f75c0020020 con 0x7f75c8160280 2024-04-17T07:32:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:24.986+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7f75c80d0a90 con 0x7f75c8160280 2024-04-17T07:32:25.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.015+0000 7f75be000640 1 -- 172.21.15.67:0/3534669070 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v710) ==== 183+0+0 (secure 0 0 0) 0x7f75c0059c40 con 0x7f75c8160280 2024-04-17T07:32:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 msgr2=0x7f75a406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 0x7f75a406ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f75b8036000 tx=0x7f75b803b040 comp rx=0 tx=0).stop 2024-04-17T07:32:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 msgr2=0x7f75c80b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f75c000c710 tx=0x7f75c000cbd0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 shutdown_connections 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f75a406ca40 0x7f75a406ef00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c8160280 0x7f75c80b4610 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75c8141ed0 0x7f75c8133360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 --2- 172.21.15.67:0/3534669070 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75c806b8c0 0x7f75c81f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 >> 172.21.15.67:0/3534669070 conn(0x7f75c8067d10 msgr2=0x7f75c80d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 shutdown_connections 2024-04-17T07:32:25.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.018+0000 7f75c7200640 1 -- 172.21.15.67:0/3534669070 wait complete. 2024-04-17T07:32:25.039 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-04-17T07:32:25.039 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-04-17T07:32:25.039 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:32:25.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:25.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.254+0000 7fc5e1e00640 1 Processor -- start 2024-04-17T07:32:25.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.254+0000 7fc5e1e00640 1 -- start start 2024-04-17T07:32:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5e1e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc5dc072420 0x7fc5dc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5e1e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 0x7fc5dc06e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5e1e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5dc15f3c0 con 0x7fc5dc15b530 2024-04-17T07:32:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5e1e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc5dc15f660 con 0x7fc5dc072420 2024-04-17T07:32:25.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5e1e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5dc15f940 con 0x7fc5dc0c68c0 2024-04-17T07:32:25.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5dc15b530 0x7fc5dc071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42670/0 (socket says 172.21.15.67:42670) 2024-04-17T07:32:25.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.256+0000 7fc5daa00640 1 -- 172.21.15.67:0/3421656498 learned_addr learned my addr 172.21.15.67:0/3421656498 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:25.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895215598 0 0) 0x7fc5dc15f3c0 con 0x7fc5dc15b530 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5bc005730 con 0x7fc5dc15b530 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5db400640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 0x7fc5dc06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5da000640 1 --2- 172.21.15.67:0/3421656498 >> v2:172.21.15.187:3300/0 conn(0x7fc5dc072420 0x7fc5dc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3345826343 0 0) 0x7fc5bc005730 con 0x7fc5dc15b530 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 msgr2=0x7fc5dc06e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 0x7fc5dc06e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 >> v2:172.21.15.187:3300/0 conn(0x7fc5dc072420 msgr2=0x7fc5dc06dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 --2- 172.21.15.67:0/3421656498 >> v2:172.21.15.187:3300/0 conn(0x7fc5dc072420 0x7fc5dc06dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5dc15fc20 con 0x7fc5dc15b530 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc5cc001670 con 0x7fc5dc15b530 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 >> v1:172.21.15.67:6789/0 conn(0x7fc5dc15b530 legacy=0x7fc5dc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5bc005bc0 0x7fc5bc005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 0x7fc5bc006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5bc0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc15f660 con 0x7fc5bc006f50 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc15f940 con 0x7fc5bc005bc0 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.257+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc00dd00 con 0x7fc5dc0c68c0 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5bc0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5daa00640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5bc005bc0 0x7fc5bc005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 -- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 msgr2=0x7fc5bc006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc0c68c0 0x7fc5bc006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 -- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5bc005bc0 msgr2=0x7fc5bc005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5bc005bc0 0x7fc5bc005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5db400640 1 -- 172.21.15.67:0/3421656498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc005730 con 0x7fc5bc006f50 2024-04-17T07:32:25.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.258+0000 7fc5daa00640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5bc005bc0 0x7fc5bc005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:25.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.259+0000 7fc5db400640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5bc0073d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400ba30 tx=0x7fc5bc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfb115dc4779695 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4036020 con 0x7fc5bc006f50 2024-04-17T07:32:25.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d4032680 con 0x7fc5bc006f50 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5d9600640 1 -- 172.21.15.67:0/3421656498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d400b600 con 0x7fc5bc006f50 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 -- 172.21.15.67:0/3421656498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 msgr2=0x7fc5bc0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/3421656498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5bc0073d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400ba30 tx=0x7fc5bc007a40 comp rx=0 tx=0).stop 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5db400640 1 -- 172.21.15.67:0/3421656498 reap_dead start 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 -- 172.21.15.67:0/3421656498 shutdown_connections 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 -- 172.21.15.67:0/3421656498 >> 172.21.15.67:0/3421656498 conn(0x7fc5dc06bf50 msgr2=0x7fc5dc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 -- 172.21.15.67:0/3421656498 shutdown_connections 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.260+0000 7fc5e1e00640 1 -- 172.21.15.67:0/3421656498 wait complete. 2024-04-17T07:32:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.262+0000 7fc5e1e00640 1 Processor -- start 2024-04-17T07:32:25.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.262+0000 7fc5e1e00640 1 -- start start 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5dc1f0220 0x7fc5dc1f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc06e340 0x7fc5dc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc160140 con 0x7fc5bc006f50 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc15b910 con 0x7fc5dc1f0220 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5e1e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5dc0b47d0 con 0x7fc5dc06e340 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.263+0000 7fc5daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34794/0 (socket says 172.21.15.67:34794) 2024-04-17T07:32:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 -- 172.21.15.67:0/596869961 learned_addr learned my addr 172.21.15.67:0/596869961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5da000640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5dc1f0220 0x7fc5dc1f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 -- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc06e340 msgr2=0x7fc5dc06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc06e340 0x7fc5dc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 -- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5dc1f0220 msgr2=0x7fc5dc1f0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5dc1f0220 0x7fc5dc1f0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 -- 172.21.15.67:0/596869961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5dc0b4ab0 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.264+0000 7fc5daa00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00d860 tx=0x7fc5cc00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.265+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc019070 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.265+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5cc0074e0 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.265+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5dc0b4d90 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.265+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc0049e0 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.265+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5dc156570 con 0x7fc5bc006f50 2024-04-17T07:32:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.266+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5dc0b52b0 con 0x7fc5bc006f50 2024-04-17T07:32:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.267+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fc5cc01c070 con 0x7fc5bc006f50 2024-04-17T07:32:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.268+0000 7fc5d3400640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 0x7fc5b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.268+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7fc5cc08ba60 con 0x7fc5bc006f50 2024-04-17T07:32:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.269+0000 7fc5da000640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 0x7fc5b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.270+0000 7fc5da000640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 0x7fc5b806eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4004b30 tx=0x7fc5c400b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.273+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc5dc0b52b0 con 0x7fc5bc006f50 2024-04-17T07:32:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.479+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5dc068b90 con 0x7fc5bc006f50 2024-04-17T07:32:25.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.481+0000 7fc5d3400640 1 -- 172.21.15.67:0/596869961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+14127 (secure 0 0 0) 0x7fc5cc059bc0 con 0x7fc5bc006f50 2024-04-17T07:32:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:25.481 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:22.996614+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6834/3534305613":"2024-04-18T07:32:19.942093+0000","172.21.15.187:0/3905455841":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2921215638":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2142021007":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/4125580629":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2371263017":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/403098035":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/3476034436":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6832/1997545601":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6833/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1195480689":"2024-04-18T07:32:15.890182+0000","172.21.15.187:6832/158797128":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/2232663954":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/3513519163":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1745494580":"2024-04-18T07:31:07.255351+0000","172.21.15.187:6835/3534305613":"2024-04-18T07:32:19.942093+0000","172.21.15.187:0/1260936234":"2024-04-18T07:31:07.255351+0000","172.21.15.187:0/1890541551":"2024-04-18T07:32:15.890182+0000","172.21.15.187:0/2471807753":"2024-04-18T07:31:07.255351+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 msgr2=0x7fc5b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 0x7fc5b806eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4004b30 tx=0x7fc5c400b040 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 msgr2=0x7fc5dc0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00d860 tx=0x7fc5cc00dd20 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 shutdown_connections 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc5b806ca10 0x7fc5b806eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5dc06e340 0x7fc5dc06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5dc1f0220 0x7fc5dc1f0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 --2- 172.21.15.67:0/596869961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5bc006f50 0x7fc5dc0b41f0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.484+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 >> 172.21.15.67:0/596869961 conn(0x7fc5dc06bf50 msgr2=0x7fc5dc159650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.485+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 shutdown_connections 2024-04-17T07:32:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.485+0000 7fc5e1e00640 1 -- 172.21.15.67:0/596869961 wait complete. 2024-04-17T07:32:25.502 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:32:25.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:32:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.731+0000 7fd2f7e00640 1 Processor -- start 2024-04-17T07:32:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.732+0000 7fd2f7e00640 1 -- start start 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.733+0000 7fd2f7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2f815ec20 0x7fd2f80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.733+0000 7fd2f7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f80d4f70 con 0x7fd2f80b3b50 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.733+0000 7fd2f7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f80d5250 con 0x7fd2f80cfcd0 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.733+0000 7fd2f7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f80cf4d0 con 0x7fd2f815ec20 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.734+0000 7fd2f6000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2f80b3b50 0x7fd2f806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42684/0 (socket says 172.21.15.67:42684) 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.734+0000 7fd2f6000640 1 -- 172.21.15.67:0/4129587285 learned_addr learned my addr 172.21.15.67:0/4129587285 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.734+0000 7fd2f6a00640 1 --2- 172.21.15.67:0/4129587285 >> v2:172.21.15.187:3300/0 conn(0x7fd2f815ec20 0x7fd2f80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.734+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596295009 0 0) 0x7fd2f80d4f70 con 0x7fd2f80b3b50 2024-04-17T07:32:25.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.735+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2dc003520 con 0x7fd2f80b3b50 2024-04-17T07:32:25.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.735+0000 7fd2f6a00640 1 -- 172.21.15.67:0/4129587285 >> v1:172.21.15.187:6789/0 conn(0x7fd2f80cfcd0 legacy=0x7fd2f80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.735+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2301936381 0 0) 0x7fd2dc003520 con 0x7fd2f80b3b50 2024-04-17T07:32:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.735+0000 7fd2f6a00640 1 -- 172.21.15.67:0/4129587285 >> v1:172.21.15.67:6789/0 conn(0x7fd2f80b3b50 legacy=0x7fd2f806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.735+0000 7fd2f6a00640 1 -- 172.21.15.67:0/4129587285 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f80cf7b0 con 0x7fd2f815ec20 2024-04-17T07:32:25.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.736+0000 7fd2f6a00640 1 --2- 172.21.15.67:0/4129587285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80d22b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009d90 tx=0x7fd2e802bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d2182ce71c1625 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.737+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8039070 con 0x7fd2f815ec20 2024-04-17T07:32:25.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.737+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e802c820 con 0x7fd2f815ec20 2024-04-17T07:32:25.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.737+0000 7fd2f5600640 1 -- 172.21.15.67:0/4129587285 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0005090 con 0x7fd2f815ec20 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 -- 172.21.15.67:0/4129587285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 msgr2=0x7fd2f80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/4129587285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80d22b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009d90 tx=0x7fd2e802bcf0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 -- 172.21.15.67:0/4129587285 shutdown_connections 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/4129587285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80d22b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 -- 172.21.15.67:0/4129587285 >> 172.21.15.67:0/4129587285 conn(0x7fd2f808bff0 msgr2=0x7fd2f80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:25.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 -- 172.21.15.67:0/4129587285 shutdown_connections 2024-04-17T07:32:25.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.738+0000 7fd2f7e00640 1 -- 172.21.15.67:0/4129587285 wait complete. 2024-04-17T07:32:25.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.740+0000 7fd2f7e00640 1 Processor -- start 2024-04-17T07:32:25.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.740+0000 7fd2f7e00640 1 -- start start 2024-04-17T07:32:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2f80cfcd0 0x7fd2f80cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f806d2d0 con 0x7fd2f80b3b50 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f806d150 con 0x7fd2f815ec20 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f80d17e0 con 0x7fd2f80cfcd0 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34806/0 (socket says 172.21.15.67:34806) 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f6a00640 1 -- 172.21.15.67:0/3754062420 learned_addr learned my addr 172.21.15.67:0/3754062420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f6a00640 1 -- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2f80cfcd0 msgr2=0x7fd2f80cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.741+0000 7fd2f6a00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2f80cfcd0 0x7fd2f80cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f6a00640 1 -- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 msgr2=0x7fd2f80b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f6a00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f6a00640 1 -- 172.21.15.67:0/3754062420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f80cda00 con 0x7fd2f80b3b50 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f6a00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8037040 tx=0x7fd2e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8039050 con 0x7fd2f80b3b50 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e803a040 con 0x7fd2f80b3b50 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8009450 con 0x7fd2f80b3b50 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f815f600 con 0x7fd2f80b3b50 2024-04-17T07:32:25.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.742+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f815fea0 con 0x7fd2f80b3b50 2024-04-17T07:32:25.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.743+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f8068a10 con 0x7fd2f80b3b50 2024-04-17T07:32:25.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.744+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fd2e8003bd0 con 0x7fd2f80b3b50 2024-04-17T07:32:25.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.745+0000 7fd2eea00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 0x7fd2d406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:25.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.745+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7fd2e80ae300 con 0x7fd2f80b3b50 2024-04-17T07:32:25.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.746+0000 7fd2f6000640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 0x7fd2d406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:25.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.747+0000 7fd2f6000640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 0x7fd2d406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80ceac0 tx=0x7fd2e0004a60 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:25.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.750+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd2e807c570 con 0x7fd2f80b3b50 2024-04-17T07:32:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.951+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2f8160630 con 0x7fd2f80b3b50 2024-04-17T07:32:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.952+0000 7fd2eea00640 1 -- 172.21.15.67:0/3754062420 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) ==== 93+0+10 (secure 0 0 0) 0x7fd2e807be70 con 0x7fd2f80b3b50 2024-04-17T07:32:25.952 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:32:25.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 msgr2=0x7fd2d406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 0x7fd2d406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80ceac0 tx=0x7fd2e0004a60 comp rx=0 tx=0).stop 2024-04-17T07:32:25.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 msgr2=0x7fd2f815f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8037040 tx=0x7fd2e8004a40 comp rx=0 tx=0).stop 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 shutdown_connections 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd2d406ca10 0x7fd2d406eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2f815ec20 0x7fd2f80b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2f80cfcd0 0x7fd2f80cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 --2- 172.21.15.67:0/3754062420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f80b3b50 0x7fd2f815f000 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 >> 172.21.15.67:0/3754062420 conn(0x7fd2f808bff0 msgr2=0x7fd2f80b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 shutdown_connections 2024-04-17T07:32:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:25.955+0000 7fd2f7e00640 1 -- 172.21.15.67:0/3754062420 wait complete. 2024-04-17T07:32:25.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-17T07:32:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.199+0000 7fa2fbe00640 1 Processor -- start 2024-04-17T07:32:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.200+0000 7fa2fbe00640 1 -- start start 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc15c5a0 0x7fa2fc15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2fc0b4210 0x7fa2fc0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2fc0b4690 0x7fa2fc0af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc15f440 con 0x7fa2fc0b4210 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc15f720 con 0x7fa2fc0b4690 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.201+0000 7fa2fbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc15fa00 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2fc0b4210 0x7fa2fc0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2fc0b4210 0x7fa2fc0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34816/0 (socket says 172.21.15.67:34816) 2024-04-17T07:32:26.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 -- 172.21.15.67:0/2432330336 learned_addr learned my addr 172.21.15.67:0/2432330336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2faa00640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc15c5a0 0x7fa2fc15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2fb400640 1 --2- 172.21.15.67:0/2432330336 >> v2:172.21.15.187:3300/0 conn(0x7fa2fc0b4690 0x7fa2fc0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 -- 172.21.15.67:0/2432330336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc15c5a0 msgr2=0x7fa2fc15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc15c5a0 0x7fa2fc15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 -- 172.21.15.67:0/2432330336 >> v2:172.21.15.187:3300/0 conn(0x7fa2fc0b4690 msgr2=0x7fa2fc0af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 --2- 172.21.15.67:0/2432330336 >> v2:172.21.15.187:3300/0 conn(0x7fa2fc0b4690 0x7fa2fc0af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.202+0000 7fa2f2000640 1 -- 172.21.15.67:0/2432330336 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc15fce0 con 0x7fa2fc0b4210 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2f2000640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc0af440 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009870 tx=0x7fa2e402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a46b32786541c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:26.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2fa000640 1 -- 172.21.15.67:0/2432330336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4038070 con 0x7fa2fc0b4210 2024-04-17T07:32:26.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2fa000640 1 -- 172.21.15.67:0/2432330336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e4031740 con 0x7fa2fc0b4210 2024-04-17T07:32:26.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2fa000640 1 -- 172.21.15.67:0/2432330336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e402f050 con 0x7fa2fc0b4210 2024-04-17T07:32:26.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2432330336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc0b4210 msgr2=0x7fa2fc0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:26.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.204+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc0af440 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009870 tx=0x7fa2e402b7e0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2432330336 shutdown_connections 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2432330336 >> v2:172.21.15.187:3300/0 conn(0x7fa2fc0b4690 0x7fa2fc0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc0af440 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2432330336 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc15c5a0 0x7fa2fc15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2432330336 >> 172.21.15.67:0/2432330336 conn(0x7fa2fc067d10 msgr2=0x7fa2fc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2432330336 shutdown_connections 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.205+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2432330336 wait complete. 2024-04-17T07:32:26.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.207+0000 7fa2fbe00640 1 Processor -- start 2024-04-17T07:32:26.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.207+0000 7fa2fbe00640 1 -- start start 2024-04-17T07:32:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc1e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 0x7fa2fc1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc06e7f0 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc06e670 con 0x7fa2fc0b4210 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc071df0 con 0x7fa2fc0b4690 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34824/0 (socket says 172.21.15.67:34824) 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2fb400640 1 -- 172.21.15.67:0/2980556791 learned_addr learned my addr 172.21.15.67:0/2980556791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2faa00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc1e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.208+0000 7fa2f2000640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 0x7fa2fc1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 -- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 msgr2=0x7fa2fc1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 0x7fa2fc1569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 -- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc0b4210 msgr2=0x7fa2fc1e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc1e72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 -- 172.21.15.67:0/2980556791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc158500 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2f2000640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 0x7fa2fc1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fb400640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4007d60 tx=0x7fa2f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4014420 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f4019070 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4014820 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc157460 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.209+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc157a20 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.211+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2fc068a10 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.212+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fa2f40149c0 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.213+0000 7fa2f3400640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 0x7fa2d806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.213+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7fa2f408c4c0 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.213+0000 7fa2faa00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 0x7fa2d806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.214+0000 7fa2faa00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 0x7fa2d806ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec006fd0 tx=0x7fa2ec008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.217+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa2fc158500 con 0x7fa2fc15c5a0 2024-04-17T07:32:26.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:26.497+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7fa2fc0cfbe0 con 0x7fa2fc15c5a0 2024-04-17T07:32:27.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.029+0000 7fa2f3400640 1 -- 172.21.15.67:0/2980556791 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v712) ==== 151+0+0 (secure 0 0 0) 0x7fa2f405a840 con 0x7fa2fc15c5a0 2024-04-17T07:32:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 msgr2=0x7fa2d806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 0x7fa2d806ef20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec006fd0 tx=0x7fa2ec008040 comp rx=0 tx=0).stop 2024-04-17T07:32:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 msgr2=0x7fa2fc142bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4007d60 tx=0x7fa2f400e4b0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 shutdown_connections 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fa2d806ca60 0x7fa2d806ef20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2fc15c5a0 0x7fa2fc142bc0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2fc0b4690 0x7fa2fc1569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 --2- 172.21.15.67:0/2980556791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2fc0b4210 0x7fa2fc1e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.032+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 >> 172.21.15.67:0/2980556791 conn(0x7fa2fc067d10 msgr2=0x7fa2fc0751b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.033+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 shutdown_connections 2024-04-17T07:32:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.033+0000 7fa2fbe00640 1 -- 172.21.15.67:0/2980556791 wait complete. 2024-04-17T07:32:27.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:32:27.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.283+0000 7f27fdc00640 1 Processor -- start 2024-04-17T07:32:27.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.284+0000 7f27fdc00640 1 -- start start 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.285+0000 7f27fdc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f27f815ecb0 0x7f27f80d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.285+0000 7f27fdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27f80cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.285+0000 7f27fdc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f80d00b0 con 0x7f27f80cf8e0 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.285+0000 7f27fdc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f27f80d0c30 con 0x7f27f815ecb0 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.285+0000 7f27fdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80d0390 con 0x7f27f80d3210 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27fd200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27f80cf8e0 0x7f27f80d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42700/0 (socket says 172.21.15.67:42700) 2024-04-17T07:32:27.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27fd200640 1 -- 172.21.15.67:0/2257426660 learned_addr learned my addr 172.21.15.67:0/2257426660 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:27.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f7e00640 1 --2- 172.21.15.67:0/2257426660 >> v2:172.21.15.187:3300/0 conn(0x7f27f815ecb0 0x7f27f80d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f7400640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27f80cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228340130 0 0) 0x7f27f80d00b0 con 0x7f27f80cf8e0 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d8003520 con 0x7f27f80cf8e0 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 894755844 0 0) 0x7f27d8003520 con 0x7f27f80cf8e0 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 msgr2=0x7f27f80cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27f80cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 >> v2:172.21.15.187:3300/0 conn(0x7f27f815ecb0 msgr2=0x7f27f80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 --2- 172.21.15.67:0/2257426660 >> v2:172.21.15.187:3300/0 conn(0x7f27f815ecb0 0x7f27f80d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.286+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80d0670 con 0x7f27f80cf8e0 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f27f00019f0 con 0x7f27f80cf8e0 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 >> v1:172.21.15.67:6789/0 conn(0x7f27f80cf8e0 legacy=0x7f27f80d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d80039b0 0x7f27d8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80d0c30 con 0x7f27d8004d40 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80d0390 con 0x7f27d80039b0 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27d800ba40 con 0x7f27f80d3210 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27fd200640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f7400640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27f7e00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d80039b0 0x7f27d8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27fd200640 1 -- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 msgr2=0x7f27d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27fd200640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27f80d3210 0x7f27d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27fd200640 1 -- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d80039b0 msgr2=0x7f27d8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.287+0000 7f27fd200640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27d80039b0 0x7f27d8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.288+0000 7f27fd200640 1 -- 172.21.15.67:0/2257426660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d8003520 con 0x7f27d8004d40 2024-04-17T07:32:27.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27fd200640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f27f000a7b0 tx=0x7f27d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843315c45ae9e9cd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f000a3d0 con 0x7f27d8004d40 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f0034650 con 0x7f27d8004d40 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27f6a00640 1 -- 172.21.15.67:0/2257426660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0034a40 con 0x7f27d8004d40 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27fdc00640 1 -- 172.21.15.67:0/2257426660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d8004d40 msgr2=0x7f27d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2257426660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f27f000a7b0 tx=0x7f27d8005830 comp rx=0 tx=0).stop 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.289+0000 7f27fd200640 1 -- 172.21.15.67:0/2257426660 reap_dead start 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.290+0000 7f27fdc00640 1 -- 172.21.15.67:0/2257426660 shutdown_connections 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.290+0000 7f27fdc00640 1 -- 172.21.15.67:0/2257426660 >> 172.21.15.67:0/2257426660 conn(0x7f27f808bff0 msgr2=0x7f27f80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.290+0000 7f27fdc00640 1 -- 172.21.15.67:0/2257426660 shutdown_connections 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.290+0000 7f27fdc00640 1 -- 172.21.15.67:0/2257426660 wait complete. 2024-04-17T07:32:27.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.291+0000 7f27fdc00640 1 Processor -- start 2024-04-17T07:32:27.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.291+0000 7f27fdc00640 1 -- start start 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27d8004d40 0x7f27f815ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27f80cc640 0x7f27f80b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27f815f090 con 0x7f27f80cbc80 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27f80cfd70 con 0x7f27f80cc640 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.292+0000 7f27fdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27f815f210 con 0x7f27d8004d40 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34846/0 (socket says 172.21.15.67:34846) 2024-04-17T07:32:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 -- 172.21.15.67:0/2292649790 learned_addr learned my addr 172.21.15.67:0/2292649790 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7e00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27d8004d40 0x7f27f815ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27fd200640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27f80cc640 0x7f27f80b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 -- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27d8004d40 msgr2=0x7f27f815ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27d8004d40 0x7f27f815ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 -- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27f80cc640 msgr2=0x7f27f80b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27f80cc640 0x7f27f80b8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.293+0000 7f27f7400640 1 -- 172.21.15.67:0/2292649790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80ccac0 con 0x7f27f80cbc80 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27f7400640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f27e0011600 tx=0x7f27e0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e000f920 con 0x7f27f80cbc80 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e000fac0 con 0x7f27f80cbc80 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e000fe70 con 0x7f27f80cbc80 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f80c8e40 con 0x7f27f80cbc80 2024-04-17T07:32:27.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.294+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f80c9400 con 0x7f27f80cbc80 2024-04-17T07:32:27.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.295+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f80c9ac0 con 0x7f27f80cbc80 2024-04-17T07:32:27.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.296+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f27e0015460 con 0x7f27f80cbc80 2024-04-17T07:32:27.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.297+0000 7f27efe00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 0x7f27d406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:27.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.297+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f27e008c2d0 con 0x7f27f80cbc80 2024-04-17T07:32:27.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.298+0000 7f27f7e00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 0x7f27d406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:27.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.299+0000 7f27f7e00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 0x7f27d406ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009a90 tx=0x7f27e8008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:27.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.301+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f27f80c9ac0 con 0x7f27f80cbc80 2024-04-17T07:32:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.503+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f27f80b5a70 con 0x7f27f80cbc80 2024-04-17T07:32:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.511+0000 7f27efe00640 1 -- 172.21.15.67:0/2292649790 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f27e0019030 con 0x7f27f80cbc80 2024-04-17T07:32:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 msgr2=0x7f27d406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 0x7f27d406ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009a90 tx=0x7f27e8008040 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 msgr2=0x7f27f80cc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f27e0011600 tx=0x7f27e0011ac0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 shutdown_connections 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f27d406ca40 0x7f27d406ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27f80cc640 0x7f27f80b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f80cbc80 0x7f27f80cc100 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 --2- 172.21.15.67:0/2292649790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27d8004d40 0x7f27f815ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 >> 172.21.15.67:0/2292649790 conn(0x7f27f808bff0 msgr2=0x7f27f80b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 shutdown_connections 2024-04-17T07:32:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:27.514+0000 7f27fdc00640 1 -- 172.21.15.67:0/2292649790 wait complete. 2024-04-17T07:32:27.531 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:32:27.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:32:27.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:32:27.776 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:32:27.777 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:27.777 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:27.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:32:27.835 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:32:27.835 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:27.894 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:32:27.894 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:27.894 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:27.955 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:32:27.955 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:32:27.955 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:28.010 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:32:28.011 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:32:28.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.240+0000 7f4a74e00640 1 Processor -- start 2024-04-17T07:32:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.240+0000 7f4a74e00640 1 -- start start 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4a700ce4a0 0x7f4a70162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a700cce90 0x7f4a700cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a700cd7f0 con 0x7f4a700cce90 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4a700cc500 con 0x7f4a700ce4a0 2024-04-17T07:32:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.241+0000 7f4a74e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a700c71e0 con 0x7f4a70161e70 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.242+0000 7f4a6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a700cce90 0x7f4a700cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.242+0000 7f4a6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a700cce90 0x7f4a700cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34860/0 (socket says 172.21.15.67:34860) 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.242+0000 7f4a6e800640 1 -- 172.21.15.67:0/185979610 learned_addr learned my addr 172.21.15.67:0/185979610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.242+0000 7f4a6de00640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.242+0000 7f4a67e00640 1 --2- 172.21.15.67:0/185979610 >> v2:172.21.15.187:3300/0 conn(0x7f4a700ce4a0 0x7f4a70162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.243+0000 7f4a6e800640 1 -- 172.21.15.67:0/185979610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 msgr2=0x7f4a700c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.243+0000 7f4a6e800640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.243+0000 7f4a6e800640 1 -- 172.21.15.67:0/185979610 >> v2:172.21.15.187:3300/0 conn(0x7f4a700ce4a0 msgr2=0x7f4a70162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.243+0000 7f4a6e800640 1 --2- 172.21.15.67:0/185979610 >> v2:172.21.15.187:3300/0 conn(0x7f4a700ce4a0 0x7f4a70162fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.243+0000 7f4a6e800640 1 -- 172.21.15.67:0/185979610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a700cc7a0 con 0x7f4a700cce90 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.244+0000 7f4a6e800640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700cce90 0x7f4a700cd2b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f4a70072de0 tx=0x7f4a6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7dc67596eb392 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.244+0000 7f4a6d400640 1 -- 172.21.15.67:0/185979610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6802ca80 con 0x7f4a700cce90 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.244+0000 7f4a6d400640 1 -- 172.21.15.67:0/185979610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6802cc20 con 0x7f4a700cce90 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.244+0000 7f4a6d400640 1 -- 172.21.15.67:0/185979610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a680343f0 con 0x7f4a700cce90 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 -- 172.21.15.67:0/185979610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700cce90 msgr2=0x7f4a700cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700cce90 0x7f4a700cd2b0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f4a70072de0 tx=0x7f4a6802bee0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 -- 172.21.15.67:0/185979610 shutdown_connections 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700cce90 0x7f4a700cd2b0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 --2- 172.21.15.67:0/185979610 >> v2:172.21.15.187:3300/0 conn(0x7f4a700ce4a0 0x7f4a70162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 --2- 172.21.15.67:0/185979610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 -- 172.21.15.67:0/185979610 >> 172.21.15.67:0/185979610 conn(0x7f4a7006bf50 msgr2=0x7f4a700b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 -- 172.21.15.67:0/185979610 shutdown_connections 2024-04-17T07:32:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.245+0000 7f4a74e00640 1 -- 172.21.15.67:0/185979610 wait complete. 2024-04-17T07:32:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.247+0000 7f4a74e00640 1 Processor -- start 2024-04-17T07:32:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.248+0000 7f4a74e00640 1 -- start start 2024-04-17T07:32:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700cce90 0x7f4a70145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a70162250 con 0x7f4a700ce4a0 2024-04-17T07:32:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a700c54b0 con 0x7f4a700cce90 2024-04-17T07:32:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a74e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a70146440 con 0x7f4a70161e70 2024-04-17T07:32:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34874/0 (socket says 172.21.15.67:34874) 2024-04-17T07:32:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.249+0000 7f4a67e00640 1 -- 172.21.15.67:0/1341202279 learned_addr learned my addr 172.21.15.67:0/1341202279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a6de00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700cce90 0x7f4a70145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 -- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 msgr2=0x7f4a700aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 -- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700cce90 msgr2=0x7f4a70145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700cce90 0x7f4a70145f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 -- 172.21.15.67:0/1341202279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a701554a0 con 0x7f4a700ce4a0 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a67e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800d800 tx=0x7f4a5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58018020 con 0x7f4a700ce4a0 2024-04-17T07:32:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a58009640 con 0x7f4a700ce4a0 2024-04-17T07:32:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58009a10 con 0x7f4a700ce4a0 2024-04-17T07:32:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.250+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a70155780 con 0x7f4a700ce4a0 2024-04-17T07:32:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.251+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a70155d40 con 0x7f4a700ce4a0 2024-04-17T07:32:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.252+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a70068b90 con 0x7f4a700ce4a0 2024-04-17T07:32:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.253+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f4a5801b020 con 0x7f4a700ce4a0 2024-04-17T07:32:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.254+0000 7f4a66000640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 0x7f4a4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.254+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f4a5808c190 con 0x7f4a700ce4a0 2024-04-17T07:32:28.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.255+0000 7f4a6de00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 0x7f4a4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.256+0000 7f4a6de00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 0x7f4a4c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4a60006fd0 tx=0x7f4a60008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.259+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a58022020 con 0x7f4a700ce4a0 2024-04-17T07:32:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.508+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a70154550 con 0x7f4a700ce4a0 2024-04-17T07:32:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.510+0000 7f4a66000640 1 -- 172.21.15.67:0/1341202279 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+3866 (secure 0 0 0) 0x7f4a5805a530 con 0x7f4a700ce4a0 2024-04-17T07:32:28.509 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:28.509 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/908634691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":908634691},{"type":"v1","addr":"172.21.15.67:6835","nonce":908634691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":5789,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2463189426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":2463189426},{"type":"v1","addr":"172.21.15.187:6837","nonce":2463189426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6226,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/968366654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":968366654},{"type":"v1","addr":"172.21.15.67:6834","nonce":968366654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/605187075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":605187075},{"type":"v1","addr":"172.21.15.187:6835","nonce":605187075}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[]} 2024-04-17T07:32:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 55 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.512+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 msgr2=0x7f4a4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.512+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 0x7f4a4c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4a60006fd0 tx=0x7f4a60008040 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.512+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 msgr2=0x7f4a700aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.512+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800d800 tx=0x7f4a5800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 shutdown_connections 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f4a4c06ca10 0x7f4a4c06eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a70161e70 0x7f4a700aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a700ce4a0 0x7f4a700aea50 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 --2- 172.21.15.67:0/1341202279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700cce90 0x7f4a70145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 >> 172.21.15.67:0/1341202279 conn(0x7f4a7006bf50 msgr2=0x7f4a700c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 shutdown_connections 2024-04-17T07:32:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.513+0000 7f4a74e00640 1 -- 172.21.15.67:0/1341202279 wait complete. 2024-04-17T07:32:28.531 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:32:28.532 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:32:28.532 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:32:28.532 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:32:28.532 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:32:28.532 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:32:28.532 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:32:28.532 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:32:28.533 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:32:28.533 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:32:28.533 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:32:28.533 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:32:28.533 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:32:28.533 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:32:28.533 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:32:28.533 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:32:28.549 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:32:28.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:32:28.551 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:32:28.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:32:28.555 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:32:28.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:32:28.563 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:32:28.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:32:28.596 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:32:28.596 DEBUG:teuthology.parallel:result is None 2024-04-17T07:32:28.600 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:32:28.600 DEBUG:teuthology.parallel:result is None 2024-04-17T07:32:28.610 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:32:28.611 DEBUG:teuthology.parallel:result is None 2024-04-17T07:32:28.617 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:32:28.617 DEBUG:teuthology.parallel:result is None 2024-04-17T07:32:28.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:32:28.760 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:32:28.763 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:32:28.773 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:32:28.773 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:32:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.815+0000 7f9bdcc00640 1 Processor -- start 2024-04-17T07:32:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.815+0000 7f9bdcc00640 1 -- start start 2024-04-17T07:32:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bdcc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bdcc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bdcc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd80cea70 con 0x7f9bd8145ac0 2024-04-17T07:32:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bdcc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd80ce840 con 0x7f9bd80ce3b0 2024-04-17T07:32:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bdcc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd80ced50 con 0x7f9bd8072d80 2024-04-17T07:32:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bd5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.816+0000 7f9bd6800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:43988/0 (socket says 172.21.15.67:43988) 2024-04-17T07:32:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 -- 172.21.15.67:0/1813504200 learned_addr learned my addr 172.21.15.67:0/1813504200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd6800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:41230/0 (socket says 172.21.15.67:41230) 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5e00640 1 --1- 172.21.15.67:0/1813504200 >> v1:172.21.15.187:6789/0 conn(0x7f9bd8145ac0 0x7f9bd8072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:38204/0 (socket says 172.21.15.67:38204) 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 -- 172.21.15.67:0/1813504200 >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 msgr2=0x7f9bd80cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 --2- 172.21.15.67:0/1813504200 >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 -- 172.21.15.67:0/1813504200 >> v1:172.21.15.187:6789/0 conn(0x7f9bd8145ac0 legacy=0x7f9bd8072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd5400640 1 -- 172.21.15.67:0/1813504200 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd80cf030 con 0x7f9bd8072d80 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd4a00640 1 -- 172.21.15.67:0/1813504200 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258573248 0 0) 0x7f9bd80cea70 con 0x7f9bd8145ac0 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.817+0000 7f9bd6800640 1 --2- 172.21.15.67:0/1813504200 >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.818+0000 7f9bd5400640 1 --2- 172.21.15.67:0/1813504200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89781ee6348af909 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bd4a00640 1 -- 172.21.15.67:0/1813504200 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4034070 con 0x7f9bd8072d80 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bd4a00640 1 -- 172.21.15.67:0/1813504200 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc4030840 con 0x7f9bd8072d80 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bd4a00640 1 -- 172.21.15.67:0/1813504200 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4030c10 con 0x7f9bd8072d80 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 -- 172.21.15.67:0/1813504200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 msgr2=0x7f9bd8071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/1813504200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc402b980 comp rx=0 tx=0).stop 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 -- 172.21.15.67:0/1813504200 shutdown_connections 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/1813504200 >> v2:172.21.15.187:3300/0 conn(0x7f9bd80ce3b0 0x7f9bd80cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/1813504200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd8072d80 0x7f9bd8071ea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.819+0000 7f9bdcc00640 1 -- 172.21.15.67:0/1813504200 >> 172.21.15.67:0/1813504200 conn(0x7f9bd8067ee0 msgr2=0x7f9bd806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.820+0000 7f9bdcc00640 1 -- 172.21.15.67:0/1813504200 shutdown_connections 2024-04-17T07:32:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.820+0000 7f9bdcc00640 1 -- 172.21.15.67:0/1813504200 wait complete. 2024-04-17T07:32:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.821+0000 7f9bdcc00640 1 Processor -- start 2024-04-17T07:32:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.821+0000 7f9bdcc00640 1 -- start start 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.822+0000 7f9bdcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.822+0000 7f9bdcc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd80ce3b0 0x7f9bd81e6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.822+0000 7f9bdcc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bd8145ac0 0x7f9bd815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.822+0000 7f9bdcc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd8145ea0 con 0x7f9bd8072d80 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.822+0000 7f9bdcc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd80cf5f0 con 0x7f9bd8145ac0 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bdcc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd80d0050 con 0x7f9bd80ce3b0 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34904/0 (socket says 172.21.15.67:34904) 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 -- 172.21.15.67:0/733781510 learned_addr learned my addr 172.21.15.67:0/733781510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5400640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd80ce3b0 0x7f9bd81e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd6800640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bd8145ac0 0x7f9bd815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 -- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd80ce3b0 msgr2=0x7f9bd81e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd80ce3b0 0x7f9bd81e6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 -- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bd8145ac0 msgr2=0x7f9bd815f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bd8145ac0 0x7f9bd815f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.823+0000 7f9bd5e00640 1 -- 172.21.15.67:0/733781510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd81e6d40 con 0x7f9bd8072d80 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bd5e00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c3f0 tx=0x7f9bc800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80074e0 con 0x7f9bd8072d80 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc80047e0 con 0x7f9bd8072d80 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8004c50 con 0x7f9bd8072d80 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd81e7020 con 0x7f9bd8072d80 2024-04-17T07:32:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.824+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd81e75e0 con 0x7f9bd8072d80 2024-04-17T07:32:28.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.826+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd80cc6c0 con 0x7f9bd8072d80 2024-04-17T07:32:28.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.827+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f9bc8011070 con 0x7f9bd8072d80 2024-04-17T07:32:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.827+0000 7f9bce000640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 0x7f9bb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.828+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5717+0+0 (secure 0 0 0) 0x7f9bc808b760 con 0x7f9bd8072d80 2024-04-17T07:32:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.828+0000 7f9bd5400640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 0x7f9bb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.829+0000 7f9bd5400640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 0x7f9bb806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4036000 tx=0x7f9bc4008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:28.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:28.832+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9bc8059b00 con 0x7f9bd8072d80 2024-04-17T07:32:28.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:32:29.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:32:29.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:29.059+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9bd80b3ad0 con 0x7f9bd8072d80 2024-04-17T07:32:30.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.056+0000 7f9bce000640 1 -- 172.21.15.67:0/733781510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v123) ==== 97+0+0 (secure 0 0 0) 0x7f9bc8059400 con 0x7f9bd8072d80 2024-04-17T07:32:30.056 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:32:30.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 msgr2=0x7f9bb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 0x7f9bb806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4036000 tx=0x7f9bc4008030 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 msgr2=0x7f9bd81e7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c3f0 tx=0x7f9bc800c7f0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 shutdown_connections 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f9bb806ca10 0x7f9bb806eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bd8145ac0 0x7f9bd815f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9bd80ce3b0 0x7f9bd81e6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 --2- 172.21.15.67:0/733781510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd8072d80 0x7f9bd81e7de0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 >> 172.21.15.67:0/733781510 conn(0x7f9bd8067ee0 msgr2=0x7f9bd8144cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 shutdown_connections 2024-04-17T07:32:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.060+0000 7f9bdcc00640 1 -- 172.21.15.67:0/733781510 wait complete. 2024-04-17T07:32:30.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:32:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.295+0000 7fdc2d600640 1 Processor -- start 2024-04-17T07:32:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.295+0000 7fdc2d600640 1 -- start start 2024-04-17T07:32:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.297+0000 7fdc2d600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc28160e50 0x7fdc28071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.297+0000 7fdc2d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.297+0000 7fdc2d600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc2815cd40 con 0x7fdc281452f0 2024-04-17T07:32:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.297+0000 7fdc2d600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2815d670 con 0x7fdc28160e50 2024-04-17T07:32:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.297+0000 7fdc2d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2815d020 con 0x7fdc280722a0 2024-04-17T07:32:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc2cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc2cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:44008/0 (socket says 172.21.15.67:44008) 2024-04-17T07:32:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc27400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc28160e50 0x7fdc28071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc2cc00640 1 -- 172.21.15.67:0/1856731859 learned_addr learned my addr 172.21.15.67:0/1856731859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc2cc00640 1 -- 172.21.15.67:0/1856731859 >> v2:172.21.15.187:3300/0 conn(0x7fdc28160e50 msgr2=0x7fdc28071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.298+0000 7fdc2cc00640 1 --2- 172.21.15.67:0/1856731859 >> v2:172.21.15.187:3300/0 conn(0x7fdc28160e50 0x7fdc28071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.299+0000 7fdc2cc00640 1 -- 172.21.15.67:0/1856731859 >> v1:172.21.15.187:6789/0 conn(0x7fdc281452f0 legacy=0x7fdc28145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.299+0000 7fdc2cc00640 1 -- 172.21.15.67:0/1856731859 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2815d300 con 0x7fdc280722a0 2024-04-17T07:32:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.300+0000 7fdc2cc00640 1 --2- 172.21.15.67:0/1856731859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009870 tx=0x7fdc2002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2410c575055ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.301+0000 7fdc26a00640 1 -- 172.21.15.67:0/1856731859 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20035020 con 0x7fdc280722a0 2024-04-17T07:32:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.301+0000 7fdc26a00640 1 -- 172.21.15.67:0/1856731859 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc20031640 con 0x7fdc280722a0 2024-04-17T07:32:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.301+0000 7fdc26a00640 1 -- 172.21.15.67:0/1856731859 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2002d040 con 0x7fdc280722a0 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.301+0000 7fdc2d600640 1 -- 172.21.15.67:0/1856731859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 msgr2=0x7fdc2815db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.301+0000 7fdc2d600640 1 --2- 172.21.15.67:0/1856731859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009870 tx=0x7fdc2002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 -- 172.21.15.67:0/1856731859 shutdown_connections 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 --2- 172.21.15.67:0/1856731859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc280722a0 0x7fdc2815db00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 --2- 172.21.15.67:0/1856731859 >> v2:172.21.15.187:3300/0 conn(0x7fdc28160e50 0x7fdc28071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 -- 172.21.15.67:0/1856731859 >> 172.21.15.67:0/1856731859 conn(0x7fdc280b5f70 msgr2=0x7fdc2806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 -- 172.21.15.67:0/1856731859 shutdown_connections 2024-04-17T07:32:30.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.302+0000 7fdc2d600640 1 -- 172.21.15.67:0/1856731859 wait complete. 2024-04-17T07:32:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.303+0000 7fdc2d600640 1 Processor -- start 2024-04-17T07:32:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.304+0000 7fdc2d600640 1 -- start start 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 0x7fdc281e6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc28160e50 0x7fdc2815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2815e040 con 0x7fdc281452f0 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281456d0 con 0x7fdc280722a0 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2815e1c0 con 0x7fdc28160e50 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34908/0 (socket says 172.21.15.67:34908) 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc27400640 1 -- 172.21.15.67:0/3707864205 learned_addr learned my addr 172.21.15.67:0/3707864205 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc27e00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 0x7fdc281e6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.305+0000 7fdc2cc00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc28160e50 0x7fdc2815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 -- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc28160e50 msgr2=0x7fdc2815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc28160e50 0x7fdc2815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 -- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 msgr2=0x7fdc281e6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 0x7fdc281e6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 -- 172.21.15.67:0/3707864205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc280b44d0 con 0x7fdc281452f0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27e00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 0x7fdc281e6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc27400640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400e870 tx=0x7fdc1400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14017070 con 0x7fdc281452f0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc1400c940 con 0x7fdc281452f0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400cd30 con 0x7fdc281452f0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.306+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc280b47b0 con 0x7fdc281452f0 2024-04-17T07:32:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.307+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc280b4d70 con 0x7fdc281452f0 2024-04-17T07:32:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.308+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc28068a10 con 0x7fdc281452f0 2024-04-17T07:32:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.309+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fdc1401a070 con 0x7fdc281452f0 2024-04-17T07:32:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.310+0000 7fdc1fe00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.310+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4857+0+0 (secure 0 0 0) 0x7fdc1408bcb0 con 0x7fdc281452f0 2024-04-17T07:32:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.311+0000 7fdc27e00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.312+0000 7fdc27e00640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 0x7fdc0806eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc180012e0 tx=0x7fdc18009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.315+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdc1405a280 con 0x7fdc281452f0 2024-04-17T07:32:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.598+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fdc28072e50 con 0x7fdc281452f0 2024-04-17T07:32:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.600+0000 7fdc1fe00640 1 -- 172.21.15.67:0/3707864205 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7fdc14059b80 con 0x7fdc281452f0 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:32:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:32:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:32:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:32:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:32:30.612 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:32:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 msgr2=0x7fdc0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 0x7fdc0806eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdc180012e0 tx=0x7fdc18009040 comp rx=0 tx=0).stop 2024-04-17T07:32:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 msgr2=0x7fdc2815b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400e870 tx=0x7fdc1400ed30 comp rx=0 tx=0).stop 2024-04-17T07:32:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 shutdown_connections 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fdc0806ca10 0x7fdc0806eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc28160e50 0x7fdc2815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc281452f0 0x7fdc2815b9c0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 --2- 172.21.15.67:0/3707864205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc280722a0 0x7fdc281e6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 >> 172.21.15.67:0/3707864205 conn(0x7fdc280b5f70 msgr2=0x7fdc28142030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 shutdown_connections 2024-04-17T07:32:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.602+0000 7fdc2d600640 1 -- 172.21.15.67:0/3707864205 wait complete. 2024-04-17T07:32:30.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.844+0000 7fac5ee00640 1 Processor -- start 2024-04-17T07:32:30.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.844+0000 7fac5ee00640 1 -- start start 2024-04-17T07:32:30.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac580cd270 0x7fac5815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac580ce950 con 0x7fac5815bbd0 2024-04-17T07:32:30.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fac5815c460 con 0x7fac580cd270 2024-04-17T07:32:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac58155960 con 0x7fac580cce90 2024-04-17T07:32:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ce00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac5815bbd0 0x7fac58071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42726/0 (socket says 172.21.15.67:42726) 2024-04-17T07:32:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac5ce00640 1 -- 172.21.15.67:0/3492815435 learned_addr learned my addr 172.21.15.67:0/3492815435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.846+0000 7fac57400640 1 --2- 172.21.15.67:0/3492815435 >> v2:172.21.15.67:3300/0 conn(0x7fac580cd270 0x7fac5815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac57e00640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac56a00640 1 -- 172.21.15.67:0/3492815435 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083163754 0 0) 0x7fac580ce950 con 0x7fac5815bbd0 2024-04-17T07:32:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac57400640 1 -- 172.21.15.67:0/3492815435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 msgr2=0x7fac580cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac57400640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac57400640 1 -- 172.21.15.67:0/3492815435 >> v1:172.21.15.67:6789/0 conn(0x7fac5815bbd0 legacy=0x7fac58071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.847+0000 7fac57400640 1 -- 172.21.15.67:0/3492815435 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac580cec30 con 0x7fac580cd270 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.848+0000 7fac57400640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580cd270 0x7fac5815b690 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fac440097a0 tx=0x7fac4402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48668e7ae88f716b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.848+0000 7fac56a00640 1 -- 172.21.15.67:0/3492815435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44035020 con 0x7fac580cd270 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.848+0000 7fac56a00640 1 -- 172.21.15.67:0/3492815435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac44031810 con 0x7fac580cd270 2024-04-17T07:32:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.848+0000 7fac56a00640 1 -- 172.21.15.67:0/3492815435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44031be0 con 0x7fac580cd270 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 -- 172.21.15.67:0/3492815435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580cd270 msgr2=0x7fac5815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580cd270 0x7fac5815b690 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fac440097a0 tx=0x7fac4402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 -- 172.21.15.67:0/3492815435 shutdown_connections 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac580cd270 0x7fac5815b690 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3492815435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 -- 172.21.15.67:0/3492815435 >> 172.21.15.67:0/3492815435 conn(0x7fac5806bf50 msgr2=0x7fac580b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 -- 172.21.15.67:0/3492815435 shutdown_connections 2024-04-17T07:32:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.849+0000 7fac5ee00640 1 -- 172.21.15.67:0/3492815435 wait complete. 2024-04-17T07:32:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.851+0000 7fac5ee00640 1 Processor -- start 2024-04-17T07:32:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.851+0000 7fac5ee00640 1 -- start start 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac580cd270 0x7fac58153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac580cd6f0 con 0x7fac5815bbd0 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac580cf1f0 con 0x7fac580cd270 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac5815c060 con 0x7fac580cce90 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34924/0 (socket says 172.21.15.67:34924) 2024-04-17T07:32:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.852+0000 7fac5ce00640 1 -- 172.21.15.67:0/3955069784 learned_addr learned my addr 172.21.15.67:0/3955069784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac57e00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac57400640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac580cd270 0x7fac58153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 -- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 msgr2=0x7fac580c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 -- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac580cd270 msgr2=0x7fac58153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac580cd270 0x7fac58153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 -- 172.21.15.67:0/3955069784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac580b05a0 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac5ce00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c7a0 tx=0x7fac5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.853+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50018070 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.854+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac5000a640 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.854+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5000aab0 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.854+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac580b0880 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.854+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac580b1120 con 0x7fac5815bbd0 2024-04-17T07:32:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.855+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac580b0a50 con 0x7fac5815bbd0 2024-04-17T07:32:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.856+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fac5001b070 con 0x7fac5815bbd0 2024-04-17T07:32:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.857+0000 7fac4fe00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 0x7fac3806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.858+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4857+0+0 (secure 0 0 0) 0x7fac5008af60 con 0x7fac5815bbd0 2024-04-17T07:32:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.858+0000 7fac57e00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 0x7fac3806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:30.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.859+0000 7fac57e00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 0x7fac3806eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac48006fd0 tx=0x7fac48008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:30.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:30.862+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fac50059660 con 0x7fac5815bbd0 2024-04-17T07:32:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.074+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac58068b90 con 0x7fac5815bbd0 2024-04-17T07:32:31.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.075+0000 7fac4fe00640 1 -- 172.21.15.67:0/3955069784 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+12853 (secure 0 0 0) 0x7fac50058f60 con 0x7fac5815bbd0 2024-04-17T07:32:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:31.075 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":123,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:30.052727+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.078+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 msgr2=0x7fac3806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.078+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 0x7fac3806eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fac48006fd0 tx=0x7fac48008040 comp rx=0 tx=0).stop 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.078+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 msgr2=0x7fac580c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.078+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c7a0 tx=0x7fac5000cc60 comp rx=0 tx=0).stop 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 shutdown_connections 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fac3806ca10 0x7fac3806eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5815bbd0 0x7fac580c7550 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac580cd270 0x7fac58153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 --2- 172.21.15.67:0/3955069784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac580cce90 0x7fac580c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 >> 172.21.15.67:0/3955069784 conn(0x7fac5806bf50 msgr2=0x7fac580c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 shutdown_connections 2024-04-17T07:32:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.079+0000 7fac5ee00640 1 -- 172.21.15.67:0/3955069784 wait complete. 2024-04-17T07:32:31.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:32:31.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:32:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.313+0000 7f6d11400640 1 Processor -- start 2024-04-17T07:32:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.313+0000 7f6d11400640 1 -- start start 2024-04-17T07:32:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d11400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0c146060 0x7f6d0c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d11400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c160920 con 0x7f6d0c145b00 2024-04-17T07:32:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d11400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c160c00 con 0x7f6d0c0b4ac0 2024-04-17T07:32:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d11400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0c160ee0 con 0x7f6d0c146060 2024-04-17T07:32:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d0be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d0c145b00 0x7f6d0c15cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42742/0 (socket says 172.21.15.67:42742) 2024-04-17T07:32:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.314+0000 7f6d0be00640 1 -- 172.21.15.67:0/3881291588 learned_addr learned my addr 172.21.15.67:0/3881291588 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0b400640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0c146060 0x7f6d0c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819913875 0 0) 0x7f6d0c160920 con 0x7f6d0c145b00 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cf0003520 con 0x7f6d0c145b00 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 705913669 0 0) 0x7f6cf0003520 con 0x7f6d0c145b00 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0c146060 msgr2=0x7f6d0c160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0c146060 0x7f6d0c160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 >> v1:172.21.15.187:6789/0 conn(0x7f6d0c0b4ac0 legacy=0x7f6d0c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1611c0 con 0x7f6d0c145b00 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6cfc0019f0 con 0x7f6d0c145b00 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 >> v1:172.21.15.67:6789/0 conn(0x7f6d0c145b00 legacy=0x7f6d0c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00039e0 0x7f6cf0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf0004800 0x7f6cf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.315+0000 7f6d0aa00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf0005210 0x7f6cf000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c160ee0 con 0x7f6cf0005210 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf000c110 con 0x7f6cf0004800 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf000c310 con 0x7f6cf00039e0 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0b400640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0c146060 0x7f6d0c160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf0005210 0x7f6cf000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0be00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00039e0 0x7f6cf0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 -- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00039e0 msgr2=0x7f6cf0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00039e0 0x7f6cf0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 -- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf0004800 msgr2=0x7f6cf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf0004800 0x7f6cf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d10a00640 1 -- 172.21.15.67:0/3881291588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf0003520 con 0x7f6cf0005210 2024-04-17T07:32:31.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.316+0000 7f6d0be00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00039e0 0x7f6cf0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:31.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.317+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf0005210 0x7f6cf000b9c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f6d04001750 tx=0x7f6cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff76be5d9bc0f95a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04001370 con 0x7f6cf0005210 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d040052d0 con 0x7f6cf0005210 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d0aa00640 1 -- 172.21.15.67:0/3881291588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003830 con 0x7f6cf0005210 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d11400640 1 -- 172.21.15.67:0/3881291588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf0005210 msgr2=0x7f6cf000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d11400640 1 --2- 172.21.15.67:0/3881291588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf0005210 0x7f6cf000b9c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f6d04001750 tx=0x7f6cf0005830 comp rx=0 tx=0).stop 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d10a00640 1 -- 172.21.15.67:0/3881291588 reap_dead start 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d11400640 1 -- 172.21.15.67:0/3881291588 shutdown_connections 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.318+0000 7f6d11400640 1 -- 172.21.15.67:0/3881291588 >> 172.21.15.67:0/3881291588 conn(0x7f6d0c0b23c0 msgr2=0x7f6d0c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.319+0000 7f6d11400640 1 -- 172.21.15.67:0/3881291588 shutdown_connections 2024-04-17T07:32:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.319+0000 7f6d11400640 1 -- 172.21.15.67:0/3881291588 wait complete. 2024-04-17T07:32:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.320+0000 7f6d11400640 1 Processor -- start 2024-04-17T07:32:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.320+0000 7f6d11400640 1 -- start start 2024-04-17T07:32:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf0005210 0x7f6d0c13a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 0x7f6d0c13dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c160340 con 0x7f6d0c13e240 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c0cfba0 con 0x7f6d0c15b4c0 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.321+0000 7f6d11400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0c1604c0 con 0x7f6cf0005210 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34956/0 (socket says 172.21.15.67:34956) 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 -- 172.21.15.67:0/3375744692 learned_addr learned my addr 172.21.15.67:0/3375744692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d0be00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf0005210 0x7f6d0c13a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d0b400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 0x7f6d0c13dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 -- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf0005210 msgr2=0x7f6d0c13a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf0005210 0x7f6d0c13a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 -- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 msgr2=0x7f6d0c13dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 0x7f6d0c13dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d10a00640 1 -- 172.21.15.67:0/3375744692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1396b0 con 0x7f6d0c13e240 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.322+0000 7f6d0b400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 0x7f6d0c13dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d10a00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005040 tx=0x7f6d04001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04001370 con 0x7f6d0c13e240 2024-04-17T07:32:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d04044070 con 0x7f6d0c13e240 2024-04-17T07:32:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04007a30 con 0x7f6d0c13e240 2024-04-17T07:32:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c139990 con 0x7f6d0c13e240 2024-04-17T07:32:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.323+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c139f50 con 0x7f6d0c13e240 2024-04-17T07:32:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.324+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0c068a10 con 0x7f6d0c13e240 2024-04-17T07:32:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.325+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f6d0403b040 con 0x7f6d0c13e240 2024-04-17T07:32:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.327+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 0x7f6cec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.327+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4857+0+0 (secure 0 0 0) 0x7f6d040acf10 con 0x7f6d0c13e240 2024-04-17T07:32:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.327+0000 7f6d0be00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 0x7f6cec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.329+0000 7f6d0be00640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 0x7f6cec06ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc001580 tx=0x7f6cfc009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.331+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d0407b4f0 con 0x7f6d0c13e240 2024-04-17T07:32:31.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.545+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d0c068610 con 0x7f6d0c13e240 2024-04-17T07:32:31.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.546+0000 7f6d03e00640 1 -- 172.21.15.67:0/3375744692 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) ==== 93+0+10 (secure 0 0 0) 0x7f6d0407adf0 con 0x7f6d0c13e240 2024-04-17T07:32:31.546 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:32:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 msgr2=0x7f6cec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 0x7f6cec06ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc001580 tx=0x7f6cfc009040 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 msgr2=0x7f6d0c13fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005040 tx=0x7f6d04001120 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 shutdown_connections 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6cec06ca40 0x7f6cec06ef00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c13e240 0x7f6d0c13fc10 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c15b4c0 0x7f6d0c13dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 --2- 172.21.15.67:0/3375744692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf0005210 0x7f6d0c13a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 >> 172.21.15.67:0/3375744692 conn(0x7f6d0c0b23c0 msgr2=0x7f6d0c06deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 shutdown_connections 2024-04-17T07:32:31.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.549+0000 7f6d11400640 1 -- 172.21.15.67:0/3375744692 wait complete. 2024-04-17T07:32:31.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:32:31.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.783+0000 7fc9d2000640 1 Processor -- start 2024-04-17T07:32:31.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.783+0000 7fc9d2000640 1 -- start start 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.784+0000 7fc9d2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9cc0cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.784+0000 7fc9d2000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc9cc15ed10 0x7fc9cc0d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9d2000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9cc0cff60 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9d2000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc9cc0d3c30 con 0x7fc9cc15ed10 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9d2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d3020 con 0x7fc9cc0cf640 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9cbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc9cc0d2ab0 0x7fc9cc0b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42748/0 (socket says 172.21.15.67:42748) 2024-04-17T07:32:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9cbe00640 1 -- 172.21.15.67:0/4275120101 learned_addr learned my addr 172.21.15.67:0/4275120101 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9cb400640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9cc0cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9c3e00640 1 --2- 172.21.15.67:0/4275120101 >> v2:172.21.15.187:3300/0 conn(0x7fc9cc15ed10 0x7fc9cc0d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028142728 0 0) 0x7fc9cc0cff60 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.785+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9b0003520 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1256348507 0 0) 0x7fc9b0003520 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 msgr2=0x7fc9cc0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9cc0cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 >> v2:172.21.15.187:3300/0 conn(0x7fc9cc15ed10 msgr2=0x7fc9cc0d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 --2- 172.21.15.67:0/4275120101 >> v2:172.21.15.187:3300/0 conn(0x7fc9cc15ed10 0x7fc9cc0d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc0d3300 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc9c40019f0 con 0x7fc9cc0d2ab0 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 >> v1:172.21.15.67:6789/0 conn(0x7fc9cc0d2ab0 legacy=0x7fc9cc0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00039b0 0x7fc9b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b0004d40 0x7fc9b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d3c30 con 0x7fc9b00039b0 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d3020 con 0x7fc9b0004d40 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b000b9f0 con 0x7fc9cc0cf640 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.786+0000 7fc9cb400640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00039b0 0x7fc9b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9c3e00640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9cb400640 1 -- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 msgr2=0x7fc9b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9cb400640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc0cf640 0x7fc9b0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9cb400640 1 -- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b0004d40 msgr2=0x7fc9b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9cb400640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b0004d40 0x7fc9b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.787+0000 7fc9cb400640 1 -- 172.21.15.67:0/4275120101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b0003520 con 0x7fc9b00039b0 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.788+0000 7fc9cb400640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00039b0 0x7fc9b0003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009ac0 tx=0x7fc9b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5b474be7e0ebe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.788+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc039070 con 0x7fc9b00039b0 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.788+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9bc02c6b0 con 0x7fc9b00039b0 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.788+0000 7fc9caa00640 1 -- 172.21.15.67:0/4275120101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc009690 con 0x7fc9b00039b0 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 -- 172.21.15.67:0/4275120101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00039b0 msgr2=0x7fc9b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 --2- 172.21.15.67:0/4275120101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b00039b0 0x7fc9b0003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009ac0 tx=0x7fc9b0005830 comp rx=0 tx=0).stop 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9cbe00640 1 -- 172.21.15.67:0/4275120101 reap_dead start 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 -- 172.21.15.67:0/4275120101 shutdown_connections 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 -- 172.21.15.67:0/4275120101 >> 172.21.15.67:0/4275120101 conn(0x7fc9cc06cbd0 msgr2=0x7fc9cc06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 -- 172.21.15.67:0/4275120101 shutdown_connections 2024-04-17T07:32:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.789+0000 7fc9d2000640 1 -- 172.21.15.67:0/4275120101 wait complete. 2024-04-17T07:32:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.791+0000 7fc9d2000640 1 Processor -- start 2024-04-17T07:32:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.791+0000 7fc9d2000640 1 -- start start 2024-04-17T07:32:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 0x7fc9cc15bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc15b7c0 0x7fc9cc0b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d25b0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d38c0 con 0x7fc9b00039b0 2024-04-17T07:32:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9d2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9cc0d4600 con 0x7fc9cc15b7c0 2024-04-17T07:32:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34982/0 (socket says 172.21.15.67:34982) 2024-04-17T07:32:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.792+0000 7fc9cbe00640 1 -- 172.21.15.67:0/2730104836 learned_addr learned my addr 172.21.15.67:0/2730104836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9c3e00640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc15b7c0 0x7fc9cc0b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cb400640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 0x7fc9cc15bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 -- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc15b7c0 msgr2=0x7fc9cc0b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc15b7c0 0x7fc9cc0b4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 -- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 msgr2=0x7fc9cc15bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 0x7fc9cc15bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 -- 172.21.15.67:0/2730104836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc15c4e0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cb400640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 0x7fc9cc15bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.793+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0d1060 tx=0x7fc9c400ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.794+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400a610 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.794+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c400a7b0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.794+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400aba0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.794+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9cc0b5710 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.794+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9cc0b5cd0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.795+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9cc0ce5d0 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.796+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fc9c4011070 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.797+0000 7fc9c3400640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 0x7fc9a806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.797+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4857+0+0 (secure 0 0 0) 0x7fc9c408b410 con 0x7fc9cc0cdc10 2024-04-17T07:32:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.798+0000 7fc9cb400640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 0x7fc9a806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.799+0000 7fc9cb400640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 0x7fc9a806eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009ac0 tx=0x7fc9bc030000 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:31.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:31.801+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc9c4059a70 con 0x7fc9cc0cdc10 2024-04-17T07:32:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.009+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc9cc0b9a90 con 0x7fc9cc0cdc10 2024-04-17T07:32:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.010+0000 7fc9c3400640 1 -- 172.21.15.67:0/2730104836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) ==== 84+0+65 (secure 0 0 0) 0x7fc9c4059370 con 0x7fc9cc0cdc10 2024-04-17T07:32:32.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:32.010 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:32:32.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:32.011 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:32:32.011 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:32:32.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:32.011 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:32:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 msgr2=0x7fc9a806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:32.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 0x7fc9a806eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009ac0 tx=0x7fc9bc030000 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 msgr2=0x7fc9cc0ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0d1060 tx=0x7fc9c400ce40 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 shutdown_connections 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fc9a806ca10 0x7fc9a806eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9cc0cdc10 0x7fc9cc0ce090 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9cc15b7c0 0x7fc9cc0b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 --2- 172.21.15.67:0/2730104836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b00039b0 0x7fc9cc15bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 >> 172.21.15.67:0/2730104836 conn(0x7fc9cc06cbd0 msgr2=0x7fc9cc161150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.013+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 shutdown_connections 2024-04-17T07:32:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.014+0000 7fc9d2000640 1 -- 172.21.15.67:0/2730104836 wait complete. 2024-04-17T07:32:32.034 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:32:32.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:32:32.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.260+0000 7f06d4a00640 1 Processor -- start 2024-04-17T07:32:32.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.260+0000 7f06d4a00640 1 -- start start 2024-04-17T07:32:32.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06d4a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f06d00cf650 0x7f06d00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06d4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06d0145030 0x7f06d006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06d4a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d006df10 con 0x7f06d0144a70 2024-04-17T07:32:32.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06d4a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06d00cd460 con 0x7f06d0145030 2024-04-17T07:32:32.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06d4a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f06d00cd740 con 0x7f06d00cf650 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06cde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06d0145030 0x7f06d006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06cde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06d0145030 0x7f06d006f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34994/0 (socket says 172.21.15.67:34994) 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.262+0000 7f06cde00640 1 -- 172.21.15.67:0/3161415263 learned_addr learned my addr 172.21.15.67:0/3161415263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.263+0000 7f06ce800640 1 --2- 172.21.15.67:0/3161415263 >> v2:172.21.15.187:3300/0 conn(0x7f06d00cf650 0x7f06d00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.263+0000 7f06cde00640 1 -- 172.21.15.67:0/3161415263 >> v2:172.21.15.187:3300/0 conn(0x7f06d00cf650 msgr2=0x7f06d00ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:32.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.263+0000 7f06cde00640 1 --2- 172.21.15.67:0/3161415263 >> v2:172.21.15.187:3300/0 conn(0x7f06d00cf650 0x7f06d00ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.263+0000 7f06cde00640 1 -- 172.21.15.67:0/3161415263 >> v1:172.21.15.187:6789/0 conn(0x7f06d0144a70 legacy=0x7f06d006e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:32.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.263+0000 7f06cde00640 1 -- 172.21.15.67:0/3161415263 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d00cda20 con 0x7f06d0145030 2024-04-17T07:32:32.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.264+0000 7f06cde00640 1 --2- 172.21.15.67:0/3161415263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d006f600 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f06d00735e0 tx=0x7f06bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fd87b043ba49be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:32.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.264+0000 7f06cd400640 1 -- 172.21.15.67:0/3161415263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc035020 con 0x7f06d0145030 2024-04-17T07:32:32.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06cd400640 1 -- 172.21.15.67:0/3161415263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06bc031870 con 0x7f06d0145030 2024-04-17T07:32:32.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06cd400640 1 -- 172.21.15.67:0/3161415263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc031c40 con 0x7f06d0145030 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 -- 172.21.15.67:0/3161415263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 msgr2=0x7f06d006f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 --2- 172.21.15.67:0/3161415263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d006f600 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f06d00735e0 tx=0x7f06bc02b910 comp rx=0 tx=0).stop 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 -- 172.21.15.67:0/3161415263 shutdown_connections 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 --2- 172.21.15.67:0/3161415263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d006f600 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 --2- 172.21.15.67:0/3161415263 >> v2:172.21.15.187:3300/0 conn(0x7f06d00cf650 0x7f06d00ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.265+0000 7f06d4a00640 1 -- 172.21.15.67:0/3161415263 >> 172.21.15.67:0/3161415263 conn(0x7f06d006b8c0 msgr2=0x7f06d006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.266+0000 7f06d4a00640 1 -- 172.21.15.67:0/3161415263 shutdown_connections 2024-04-17T07:32:32.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.266+0000 7f06d4a00640 1 -- 172.21.15.67:0/3161415263 wait complete. 2024-04-17T07:32:32.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.267+0000 7f06d4a00640 1 Processor -- start 2024-04-17T07:32:32.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.267+0000 7f06d4a00640 1 -- start start 2024-04-17T07:32:32.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06d00cf650 0x7f06d01f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 0x7f06d01f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d00cd070 con 0x7f06d0145030 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06d00cea50 con 0x7f06d0144a70 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.268+0000 7f06d4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06d00cd1f0 con 0x7f06d00cf650 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34996/0 (socket says 172.21.15.67:34996) 2024-04-17T07:32:32.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 -- 172.21.15.67:0/2795434351 learned_addr learned my addr 172.21.15.67:0/2795434351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:32.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cde00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 0x7f06d01f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 -- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06d00cf650 msgr2=0x7f06d01f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:32.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06ce800640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06d00cf650 0x7f06d01f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06d00cf650 0x7f06d01f2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 -- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 msgr2=0x7f06d01f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 0x7f06d01f57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 -- 172.21.15.67:0/2795434351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d00ac7e0 con 0x7f06d0145030 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cde00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 0x7f06d01f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.269+0000 7f06cf200640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b970 tx=0x7f06c800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.270+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8019070 con 0x7f06d0145030 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.270+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c800cb40 con 0x7f06d0145030 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.270+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8014440 con 0x7f06d0145030 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.270+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d00acac0 con 0x7f06d0145030 2024-04-17T07:32:32.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.270+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d01e42f0 con 0x7f06d0145030 2024-04-17T07:32:32.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.271+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d00b3c20 con 0x7f06d0145030 2024-04-17T07:32:32.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.272+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f06c801c070 con 0x7f06d0145030 2024-04-17T07:32:32.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.273+0000 7f06c6a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 0x7f06b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:32.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.273+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4857+0+0 (secure 0 0 0) 0x7f06c808b680 con 0x7f06d0145030 2024-04-17T07:32:32.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.274+0000 7f06ce800640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 0x7f06b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:32.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.275+0000 7f06ce800640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 0x7f06b006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006fd0 tx=0x7f06c0008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:32.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.278+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f06d00b3c20 con 0x7f06d0145030 2024-04-17T07:32:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:32.486+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f06d0068a10 con 0x7f06d0145030 2024-04-17T07:32:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.076+0000 7f06c6a00640 1 -- 172.21.15.67:0/2795434351 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) ==== 138+0+0 (secure 0 0 0) 0x7f06c8059c50 con 0x7f06d0145030 2024-04-17T07:32:33.076 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:32:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 msgr2=0x7f06b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:33.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 0x7f06b006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006fd0 tx=0x7f06c0008040 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 msgr2=0x7f06d00ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f06c800b970 tx=0x7f06c800be30 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 shutdown_connections 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f06b006ca10 0x7f06b006eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0145030 0x7f06d00ac0e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d0144a70 0x7f06d01f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 --2- 172.21.15.67:0/2795434351 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06d00cf650 0x7f06d01f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 >> 172.21.15.67:0/2795434351 conn(0x7f06d006b8c0 msgr2=0x7f06d00aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 shutdown_connections 2024-04-17T07:32:33.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.080+0000 7f06d4a00640 1 -- 172.21.15.67:0/2795434351 wait complete. 2024-04-17T07:32:33.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:32:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.326+0000 7fd027200640 1 Processor -- start 2024-04-17T07:32:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.326+0000 7fd027200640 1 -- start start 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd027200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 0x7fd0280d4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd027200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0280d2bc0 0x7fd0280b8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd027200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0280b79b0 con 0x7fd02815ebe0 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd027200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd0280d3bd0 con 0x7fd0280d2bc0 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd027200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0280d3330 con 0x7fd0280b6bf0 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.327+0000 7fd026800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0280d2bc0 0x7fd0280b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0280d2bc0 0x7fd0280b8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60862/0 (socket says 172.21.15.67:60862) 2024-04-17T07:32:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 -- 172.21.15.67:0/2667460669 learned_addr learned my addr 172.21.15.67:0/2667460669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd025400640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 0x7fd0280d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 -- 172.21.15.67:0/2667460669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 msgr2=0x7fd0280d4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 0x7fd0280d4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 -- 172.21.15.67:0/2667460669 >> v1:172.21.15.187:6789/0 conn(0x7fd02815ebe0 legacy=0x7fd0280b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd026800640 1 -- 172.21.15.67:0/2667460669 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0280d3610 con 0x7fd0280d2bc0 2024-04-17T07:32:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.328+0000 7fd025400640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 0x7fd0280d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.329+0000 7fd026800640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280d2bc0 0x7fd0280b8150 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd020009e50 tx=0x7fd02002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498e970a3a8318f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.329+0000 7fd024a00640 1 -- 172.21.15.67:0/2667460669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02002ca70 con 0x7fd0280d2bc0 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd024a00640 1 -- 172.21.15.67:0/2667460669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02002cc10 con 0x7fd0280d2bc0 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd024a00640 1 -- 172.21.15.67:0/2667460669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200343f0 con 0x7fd0280d2bc0 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 -- 172.21.15.67:0/2667460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280d2bc0 msgr2=0x7fd0280b8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280d2bc0 0x7fd0280b8150 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fd020009e50 tx=0x7fd02002bed0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 -- 172.21.15.67:0/2667460669 shutdown_connections 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280d2bc0 0x7fd0280b8150 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 --2- 172.21.15.67:0/2667460669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280b6bf0 0x7fd0280d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 -- 172.21.15.67:0/2667460669 >> 172.21.15.67:0/2667460669 conn(0x7fd0280d16c0 msgr2=0x7fd0280d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 -- 172.21.15.67:0/2667460669 shutdown_connections 2024-04-17T07:32:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.330+0000 7fd027200640 1 -- 172.21.15.67:0/2667460669 wait complete. 2024-04-17T07:32:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.332+0000 7fd027200640 1 Processor -- start 2024-04-17T07:32:33.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.332+0000 7fd027200640 1 -- start start 2024-04-17T07:32:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 0x7fd0281f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 0x7fd02815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0280d3030 con 0x7fd0280b6bf0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0280b7c30 con 0x7fd02815ebe0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd027200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd02815efc0 con 0x7fd0280d2bc0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.333+0000 7fd025e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60874/0 (socket says 172.21.15.67:60874) 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 -- 172.21.15.67:0/4242370952 learned_addr learned my addr 172.21.15.67:0/4242370952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd026800640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 0x7fd02815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025400640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 0x7fd0281f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 -- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 msgr2=0x7fd0281f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 0x7fd0281f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 -- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 msgr2=0x7fd02815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 0x7fd02815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 -- 172.21.15.67:0/4242370952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0280cd060 con 0x7fd0280b6bf0 2024-04-17T07:32:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd026800640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 0x7fd02815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025400640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 0x7fd0281f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.334+0000 7fd025e00640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c430 tx=0x7fd01800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.335+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018011790 con 0x7fd0280b6bf0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.335+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd018017070 con 0x7fd0280b6bf0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.335+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018011bb0 con 0x7fd0280b6bf0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.335+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0280cd340 con 0x7fd0280b6bf0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.335+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0280cd900 con 0x7fd0280b6bf0 2024-04-17T07:32:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.336+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd028068a10 con 0x7fd0280b6bf0 2024-04-17T07:32:33.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.338+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fd01801c460 con 0x7fd0280b6bf0 2024-04-17T07:32:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.339+0000 7fd01e000640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 0x7fd00806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.339+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5227+0+0 (secure 0 0 0) 0x7fd01808c4e0 con 0x7fd0280b6bf0 2024-04-17T07:32:33.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.339+0000 7fd025400640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 0x7fd00806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.341+0000 7fd025400640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 0x7fd00806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd014006fd0 tx=0x7fd014008040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.343+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd01805a9c0 con 0x7fd0280b6bf0 2024-04-17T07:32:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:33.553+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd0280c8b20 con 0x7fd0280b6bf0 2024-04-17T07:32:34.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.083+0000 7fd01e000640 1 -- 172.21.15.67:0/4242370952 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) ==== 170+0+0 (secure 0 0 0) 0x7fd0280c8b20 con 0x7fd0280b6bf0 2024-04-17T07:32:34.083 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.087+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 msgr2=0x7fd00806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.087+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 0x7fd00806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd014006fd0 tx=0x7fd014008040 comp rx=0 tx=0).stop 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.087+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 msgr2=0x7fd0281f5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.087+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd01800c430 tx=0x7fd01800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 shutdown_connections 2024-04-17T07:32:34.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fd00806ca10 0x7fd00806eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd02815ebe0 0x7fd02815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0280d2bc0 0x7fd0281f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 --2- 172.21.15.67:0/4242370952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0280b6bf0 0x7fd0281f5950 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 >> 172.21.15.67:0/4242370952 conn(0x7fd0280d16c0 msgr2=0x7fd028161240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 shutdown_connections 2024-04-17T07:32:34.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.088+0000 7fd027200640 1 -- 172.21.15.67:0/4242370952 wait complete. 2024-04-17T07:32:34.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:32:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.315+0000 7fb533e00640 1 Processor -- start 2024-04-17T07:32:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.315+0000 7fb533e00640 1 -- start start 2024-04-17T07:32:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb533e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb52c06daf0 0x7fb52c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb533e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 0x7fb52c070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb533e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb52c0d1520 con 0x7fb52c0d1060 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb533e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb52c0d1800 con 0x7fb52c06daf0 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb533e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb52c0d1ae0 con 0x7fb52c09b170 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb531400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb52c06daf0 0x7fb52c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb530a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb52c0d1060 0x7fb52c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46280/0 (socket says 172.21.15.67:46280) 2024-04-17T07:32:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.317+0000 7fb530a00640 1 -- 172.21.15.67:0/1762801299 learned_addr learned my addr 172.21.15.67:0/1762801299 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531e00640 1 --2- 172.21.15.67:0/1762801299 >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 0x7fb52c070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531400640 1 -- 172.21.15.67:0/1762801299 >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 msgr2=0x7fb52c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531400640 1 --2- 172.21.15.67:0/1762801299 >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 0x7fb52c070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531400640 1 -- 172.21.15.67:0/1762801299 >> v1:172.21.15.67:6789/0 conn(0x7fb52c0d1060 legacy=0x7fb52c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531400640 1 -- 172.21.15.67:0/1762801299 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c0d1dc0 con 0x7fb52c06daf0 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb52be00640 1 -- 172.21.15.67:0/1762801299 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707953258 0 0) 0x7fb52c0d1520 con 0x7fb52c0d1060 2024-04-17T07:32:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.318+0000 7fb531e00640 1 --2- 172.21.15.67:0/1762801299 >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 0x7fb52c070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.319+0000 7fb531400640 1 --2- 172.21.15.67:0/1762801299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c06daf0 0x7fb52c0d0b20 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0d3710 tx=0x7fb52002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3a99805980e725 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb52be00640 1 -- 172.21.15.67:0/1762801299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520035020 con 0x7fb52c06daf0 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb52be00640 1 -- 172.21.15.67:0/1762801299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb520031830 con 0x7fb52c06daf0 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb52be00640 1 -- 172.21.15.67:0/1762801299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520031c00 con 0x7fb52c06daf0 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 -- 172.21.15.67:0/1762801299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c06daf0 msgr2=0x7fb52c0d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 --2- 172.21.15.67:0/1762801299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c06daf0 0x7fb52c0d0b20 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0d3710 tx=0x7fb52002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 -- 172.21.15.67:0/1762801299 shutdown_connections 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 --2- 172.21.15.67:0/1762801299 >> v2:172.21.15.187:3300/0 conn(0x7fb52c09b170 0x7fb52c070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 --2- 172.21.15.67:0/1762801299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c06daf0 0x7fb52c0d0b20 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 -- 172.21.15.67:0/1762801299 >> 172.21.15.67:0/1762801299 conn(0x7fb52c0b41a0 msgr2=0x7fb52c0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 -- 172.21.15.67:0/1762801299 shutdown_connections 2024-04-17T07:32:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.320+0000 7fb533e00640 1 -- 172.21.15.67:0/1762801299 wait complete. 2024-04-17T07:32:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.322+0000 7fb533e00640 1 Processor -- start 2024-04-17T07:32:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.323+0000 7fb533e00640 1 -- start start 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 0x7fb52c0b7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb52c09b170 0x7fb52c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c06e050 con 0x7fb52c0d1060 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c06ded0 con 0x7fb52c06daf0 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb533e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c06ce00 con 0x7fb52c09b170 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60900/0 (socket says 172.21.15.67:60900) 2024-04-17T07:32:34.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 -- 172.21.15.67:0/4167546115 learned_addr learned my addr 172.21.15.67:0/4167546115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531400640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 0x7fb52c0b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb530a00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb52c09b170 0x7fb52c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 -- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb52c09b170 msgr2=0x7fb52c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb52c09b170 0x7fb52c0cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 -- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 msgr2=0x7fb52c0b7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 0x7fb52c0b7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531e00640 1 -- 172.21.15.67:0/4167546115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c0b8060 con 0x7fb52c0d1060 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.324+0000 7fb531400640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 0x7fb52c0b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb531e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb52400ce70 tx=0x7fb52400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb524003b90 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb524003d50 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb524004260 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c160060 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.325+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c160620 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.326+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c068ca0 con 0x7fb52c0d1060 2024-04-17T07:32:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.328+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fb524004400 con 0x7fb52c0d1060 2024-04-17T07:32:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.328+0000 7fb529600640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 0x7fb50c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.329+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5609+0+0 (secure 0 0 0) 0x7fb52408c6b0 con 0x7fb52c0d1060 2024-04-17T07:32:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.329+0000 7fb531400640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 0x7fb50c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.331+0000 7fb531400640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 0x7fb50c06eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb520036000 tx=0x7fb520008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.335+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb52405a990 con 0x7fb52c0d1060 2024-04-17T07:32:34.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:34.585+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb52c06ef00 con 0x7fb52c0d1060 2024-04-17T07:32:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.130+0000 7fb529600640 1 -- 172.21.15.67:0/4167546115 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v57) ==== 174+0+0 (secure 0 0 0) 0x7fb52405a290 con 0x7fb52c0d1060 2024-04-17T07:32:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 14 and data pool 15 2024-04-17T07:32:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.143+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 msgr2=0x7fb50c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.143+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 0x7fb50c06eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb520036000 tx=0x7fb520008030 comp rx=0 tx=0).stop 2024-04-17T07:32:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.143+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 msgr2=0x7fb52c0caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.143+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb52400ce70 tx=0x7fb52400a710 comp rx=0 tx=0).stop 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 shutdown_connections 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb50c06ca10 0x7fb50c06eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52c0d1060 0x7fb52c0caef0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb52c09b170 0x7fb52c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 --2- 172.21.15.67:0/4167546115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb52c06daf0 0x7fb52c0b7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 >> 172.21.15.67:0/4167546115 conn(0x7fb52c0b41a0 msgr2=0x7fb52c0b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:35.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.144+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 shutdown_connections 2024-04-17T07:32:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.146+0000 7fb533e00640 1 -- 172.21.15.67:0/4167546115 wait complete. 2024-04-17T07:32:35.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:35.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.364+0000 7f96b8e00640 1 Processor -- start 2024-04-17T07:32:35.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.365+0000 7f96b8e00640 1 -- start start 2024-04-17T07:32:35.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b8e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f96b40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b8e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b40d2d30 con 0x7f96b415ec60 2024-04-17T07:32:35.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b8e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b40d3010 con 0x7f96b40cf910 2024-04-17T07:32:35.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b8e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96b40d49c0 con 0x7f96b40b6d00 2024-04-17T07:32:35.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96b415ec60 0x7f96b40b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46284/0 (socket says 172.21.15.67:46284) 2024-04-17T07:32:35.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.366+0000 7f96b3400640 1 -- 172.21.15.67:0/556360587 learned_addr learned my addr 172.21.15.67:0/556360587 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:35.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96aaa00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f96b40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295848980 0 0) 0x7f96b40d2d30 con 0x7f96b415ec60 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9698003520 con 0x7f96b415ec60 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 836494181 0 0) 0x7f9698003520 con 0x7f96b415ec60 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 msgr2=0x7f96b40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f96b40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 >> v1:172.21.15.187:6789/0 conn(0x7f96b40cf910 legacy=0x7f96b40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b40d32f0 con 0x7f96b415ec60 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f96a4001670 con 0x7f96b415ec60 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 >> v1:172.21.15.67:6789/0 conn(0x7f96b415ec60 legacy=0x7f96b40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.367+0000 7f96b2a00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f9698003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b2a00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698003fd0 0x7f9698004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b2a00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9698004d40 0x7f96980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96b40d49c0 con 0x7f9698004d40 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f969800bb20 con 0x7f9698003fd0 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f969800bd20 con 0x7f96b40b6d00 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9698004d40 0x7f96980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3400640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f9698003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96aaa00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698003fd0 0x7f9698004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 -- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 msgr2=0x7f9698003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b40b6d00 0x7f9698003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 -- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698003fd0 msgr2=0x7f9698004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698003fd0 0x7f9698004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.368+0000 7f96b3e00640 1 -- 172.21.15.67:0/556360587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9698003520 con 0x7f9698004d40 2024-04-17T07:32:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.370+0000 7f96b3e00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9698004d40 0x7f96980051c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f96ac001e50 tx=0x7f9698005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc05f793e783ed9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.370+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac001a70 con 0x7f9698004d40 2024-04-17T07:32:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.370+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ac003b70 con 0x7f9698004d40 2024-04-17T07:32:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.370+0000 7f96b2a00640 1 -- 172.21.15.67:0/556360587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0050d0 con 0x7f9698004d40 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 -- 172.21.15.67:0/556360587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9698004d40 msgr2=0x7f96980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 --2- 172.21.15.67:0/556360587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9698004d40 0x7f96980051c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f96ac001e50 tx=0x7f9698005830 comp rx=0 tx=0).stop 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b3e00640 1 -- 172.21.15.67:0/556360587 reap_dead start 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 -- 172.21.15.67:0/556360587 shutdown_connections 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 -- 172.21.15.67:0/556360587 >> 172.21.15.67:0/556360587 conn(0x7f96b40d15f0 msgr2=0x7f96b40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 -- 172.21.15.67:0/556360587 shutdown_connections 2024-04-17T07:32:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.371+0000 7f96b8e00640 1 -- 172.21.15.67:0/556360587 wait complete. 2024-04-17T07:32:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.373+0000 7f96b8e00640 1 Processor -- start 2024-04-17T07:32:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.373+0000 7f96b8e00640 1 -- start start 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698004d40 0x7f96b4158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b406cc30 0x7f96b406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96b415f040 con 0x7f96b4158680 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f96b40d3660 con 0x7f9698004d40 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.374+0000 7f96b8e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96b40d00c0 con 0x7f96b406cc30 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60912/0 (socket says 172.21.15.67:60912) 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 -- 172.21.15.67:0/3953834139 learned_addr learned my addr 172.21.15.67:0/3953834139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96b3400640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698004d40 0x7f96b4158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96b3e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b406cc30 0x7f96b406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 -- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b406cc30 msgr2=0x7f96b406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b406cc30 0x7f96b406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 -- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698004d40 msgr2=0x7f96b4158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698004d40 0x7f96b4158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.375+0000 7f96aaa00640 1 -- 172.21.15.67:0/3953834139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b40c9820 con 0x7f96b4158680 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96aaa00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f969c011630 tx=0x7f969c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c00f920 con 0x7f96b4158680 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969c00fac0 con 0x7f96b4158680 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c00fe70 con 0x7f96b4158680 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b40c8f80 con 0x7f96b4158680 2024-04-17T07:32:35.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.376+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b40c9540 con 0x7f96b4158680 2024-04-17T07:32:35.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.377+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b4068a10 con 0x7f96b4158680 2024-04-17T07:32:35.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.378+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f969c015460 con 0x7f96b4158680 2024-04-17T07:32:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.379+0000 7f96abe00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 0x7f969006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.379+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5709+0+0 (secure 0 0 0) 0x7f969c08c2c0 con 0x7f96b4158680 2024-04-17T07:32:35.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.380+0000 7f96b3400640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 0x7f969006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.381+0000 7f96b3400640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 0x7f969006ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001b80 tx=0x7f96a4009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.384+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f969c019030 con 0x7f96b4158680 2024-04-17T07:32:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.590+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96b406bd30 con 0x7f96b4158680 2024-04-17T07:32:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.593+0000 7f96abe00640 1 -- 172.21.15.67:0/3953834139 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+16584 (secure 0 0 0) 0x7f969c05a540 con 0x7f96b4158680 2024-04-17T07:32:35.594 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:35.594 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":126,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:34.587242+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 msgr2=0x7f969006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 0x7f969006ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001b80 tx=0x7f96a4009040 comp rx=0 tx=0).stop 2024-04-17T07:32:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 msgr2=0x7f96b40c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f969c011630 tx=0x7f969c011af0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 shutdown_connections 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f969006ca40 0x7f969006ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96b406cc30 0x7f96b406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96b4158680 0x7f96b40c86a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 --2- 172.21.15.67:0/3953834139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9698004d40 0x7f96b4158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 >> 172.21.15.67:0/3953834139 conn(0x7f96b40d15f0 msgr2=0x7f96b40c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 shutdown_connections 2024-04-17T07:32:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.596+0000 7f96b8e00640 1 -- 172.21.15.67:0/3953834139 wait complete. 2024-04-17T07:32:35.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.839+0000 7f5fb3200640 1 Processor -- start 2024-04-17T07:32:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.840+0000 7f5fb3200640 1 -- start start 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.841+0000 7f5fb3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb415ec60 0x7f5fb40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.841+0000 7f5fb3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.841+0000 7f5fb3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5fb40cf910 0x7f5fb406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.841+0000 7f5fb3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb40d49c0 con 0x7f5fb40b6b30 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb40d3660 con 0x7f5fb40cf910 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb40b6f70 con 0x7f5fb415ec60 2024-04-17T07:32:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60924/0 (socket says 172.21.15.67:60924) 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 -- 172.21.15.67:0/485770679 learned_addr learned my addr 172.21.15.67:0/485770679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1e00640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb415ec60 0x7f5fb40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb2800640 1 --2- 172.21.15.67:0/485770679 >> v2:172.21.15.187:3300/0 conn(0x7f5fb40cf910 0x7f5fb406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 -- 172.21.15.67:0/485770679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb415ec60 msgr2=0x7f5fb40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb415ec60 0x7f5fb40b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 -- 172.21.15.67:0/485770679 >> v2:172.21.15.187:3300/0 conn(0x7f5fb40cf910 msgr2=0x7f5fb406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 --2- 172.21.15.67:0/485770679 >> v2:172.21.15.187:3300/0 conn(0x7f5fb40cf910 0x7f5fb406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.842+0000 7f5fb1400640 1 -- 172.21.15.67:0/485770679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb40d2d30 con 0x7f5fb40b6b30 2024-04-17T07:32:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.844+0000 7f5fb1400640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40c8670 tx=0x7f5fa002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb8fa98b3b6d88c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.844+0000 7f5fb0a00640 1 -- 172.21.15.67:0/485770679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0035020 con 0x7f5fb40b6b30 2024-04-17T07:32:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.844+0000 7f5fb0a00640 1 -- 172.21.15.67:0/485770679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa002d040 con 0x7f5fb40b6b30 2024-04-17T07:32:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.844+0000 7f5fb0a00640 1 -- 172.21.15.67:0/485770679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0031a80 con 0x7f5fb40b6b30 2024-04-17T07:32:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 -- 172.21.15.67:0/485770679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 msgr2=0x7f5fb40cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40c8670 tx=0x7f5fa002b900 comp rx=0 tx=0).stop 2024-04-17T07:32:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 -- 172.21.15.67:0/485770679 shutdown_connections 2024-04-17T07:32:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 --2- 172.21.15.67:0/485770679 >> v2:172.21.15.187:3300/0 conn(0x7f5fb40cf910 0x7f5fb406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb40cf3d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 --2- 172.21.15.67:0/485770679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb415ec60 0x7f5fb40b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 -- 172.21.15.67:0/485770679 >> 172.21.15.67:0/485770679 conn(0x7f5fb40d15f0 msgr2=0x7f5fb40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 -- 172.21.15.67:0/485770679 shutdown_connections 2024-04-17T07:32:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.845+0000 7f5fb3200640 1 -- 172.21.15.67:0/485770679 wait complete. 2024-04-17T07:32:35.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.847+0000 7f5fb3200640 1 Processor -- start 2024-04-17T07:32:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.848+0000 7f5fb3200640 1 -- start start 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 0x7f5fb40b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 0x7f5fb40abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb40cfda0 con 0x7f5fb40b6b30 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb40d32f0 con 0x7f5fb415ec60 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb415f040 con 0x7f5fb40cf910 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60938/0 (socket says 172.21.15.67:60938) 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.849+0000 7f5fb1e00640 1 -- 172.21.15.67:0/1804359898 learned_addr learned my addr 172.21.15.67:0/1804359898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb2800640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 0x7f5fb40abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1400640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 0x7f5fb40b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 -- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 msgr2=0x7f5fb40b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 0x7f5fb40b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 -- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 msgr2=0x7f5fb40abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 0x7f5fb40abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 -- 172.21.15.67:0/1804359898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb415b820 con 0x7f5fb40b6b30 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb2800640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 0x7f5fb40abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1400640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 0x7f5fb40b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb1e00640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d5f0 tx=0x7f5fa400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40125f0 con 0x7f5fb40b6b30 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa4019070 con 0x7f5fb40b6b30 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40129d0 con 0x7f5fb40b6b30 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.850+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb40b96a0 con 0x7f5fb40b6b30 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.851+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb40cc130 con 0x7f5fb40b6b30 2024-04-17T07:32:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.852+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb4068a10 con 0x7f5fb40b6b30 2024-04-17T07:32:35.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.853+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f5fa4012b70 con 0x7f5fb40b6b30 2024-04-17T07:32:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.854+0000 7f5faa000640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 0x7f5f9406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.854+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5709+0+0 (secure 0 0 0) 0x7f5fa408c170 con 0x7f5fb40b6b30 2024-04-17T07:32:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.855+0000 7f5fb1400640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 0x7f5f9406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:35.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.856+0000 7f5fb1400640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 0x7f5f9406ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0036000 tx=0x7f5fa0008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:35.859+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5fa405a3f0 con 0x7f5fb40b6b30 2024-04-17T07:32:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.065+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fb40b45e0 con 0x7f5fb40b6b30 2024-04-17T07:32:36.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.068+0000 7f5faa000640 1 -- 172.21.15.67:0/1804359898 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+16584 (secure 0 0 0) 0x7f5fa4059cf0 con 0x7f5fb40b6b30 2024-04-17T07:32:36.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:36.069 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":126,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:34.587242+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 msgr2=0x7f5f9406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 0x7f5f9406ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0036000 tx=0x7f5fa0008030 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 msgr2=0x7f5fb415b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d5f0 tx=0x7f5fa400dab0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 shutdown_connections 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f5f9406ca40 0x7f5f9406ef00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5fb415ec60 0x7f5fb40abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5fb40cf910 0x7f5fb40b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 --2- 172.21.15.67:0/1804359898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40b6b30 0x7f5fb415b2e0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.070+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 >> 172.21.15.67:0/1804359898 conn(0x7f5fb40d15f0 msgr2=0x7f5fb41612b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.071+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 shutdown_connections 2024-04-17T07:32:36.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.071+0000 7f5fb3200640 1 -- 172.21.15.67:0/1804359898 wait complete. 2024-04-17T07:32:36.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:32:36.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.294+0000 7f749d200640 1 Processor -- start 2024-04-17T07:32:36.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.295+0000 7f749d200640 1 -- start start 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.296+0000 7f749d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f74980b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.296+0000 7f749d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749806df40 con 0x7f749806bfd0 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.296+0000 7f749d200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749806e220 con 0x7f749815b6b0 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.297+0000 7f749d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f749806ed00 con 0x7f749806f800 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.297+0000 7f748fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f749806bfd0 0x7f74980b3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46290/0 (socket says 172.21.15.67:46290) 2024-04-17T07:32:36.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.297+0000 7f748fe00640 1 -- 172.21.15.67:0/2866026524 learned_addr learned my addr 172.21.15.67:0/2866026524 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:36.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.297+0000 7f7497e00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f74980b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:36.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.297+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2470415071 0 0) 0x7f749806df40 con 0x7f749806bfd0 2024-04-17T07:32:36.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f747c003520 con 0x7f749806bfd0 2024-04-17T07:32:36.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3953690866 0 0) 0x7f747c003520 con 0x7f749806bfd0 2024-04-17T07:32:36.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 msgr2=0x7f74980b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f74980b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 >> v1:172.21.15.187:6789/0 conn(0x7f749815b6b0 legacy=0x7f749806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749806e500 con 0x7f749806bfd0 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7480001670 con 0x7f749806bfd0 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 >> v1:172.21.15.67:6789/0 conn(0x7f749806bfd0 legacy=0x7f74980b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f747c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c003f20 0x7f747c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f747c004d40 0x7f747c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f749806ed00 con 0x7f747c003f20 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f747c00dce0 con 0x7f747c004d40 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f747c00dee0 con 0x7f749806f800 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.298+0000 7f748fe00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c003f20 0x7f747c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f7497400640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f747c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f748fe00640 1 -- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 msgr2=0x7f747c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f748fe00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f747c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f748fe00640 1 -- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f747c004d40 msgr2=0x7f747c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f748fe00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f747c004d40 0x7f747c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f748fe00640 1 -- 172.21.15.67:0/2866026524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c003520 con 0x7f747c003f20 2024-04-17T07:32:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.299+0000 7f7497400640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f749806f800 0x7f747c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.300+0000 7f748fe00640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c003f20 0x7f747c004800 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f747c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff76c889a103f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.300+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480007d30 con 0x7f747c003f20 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.300+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7480030070 con 0x7f747c003f20 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.300+0000 7f7496a00640 1 -- 172.21.15.67:0/2866026524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748002cd80 con 0x7f747c003f20 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 -- 172.21.15.67:0/2866026524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c003f20 msgr2=0x7f747c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 --2- 172.21.15.67:0/2866026524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c003f20 0x7f747c004800 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f747c005830 comp rx=0 tx=0).stop 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f7497e00640 1 -- 172.21.15.67:0/2866026524 reap_dead start 2024-04-17T07:32:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 -- 172.21.15.67:0/2866026524 shutdown_connections 2024-04-17T07:32:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 -- 172.21.15.67:0/2866026524 >> 172.21.15.67:0/2866026524 conn(0x7f74980b2190 msgr2=0x7f74980b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 -- 172.21.15.67:0/2866026524 shutdown_connections 2024-04-17T07:32:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.301+0000 7f749d200640 1 -- 172.21.15.67:0/2866026524 wait complete. 2024-04-17T07:32:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.302+0000 7f749d200640 1 Processor -- start 2024-04-17T07:32:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.303+0000 7f749d200640 1 -- start start 2024-04-17T07:32:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.303+0000 7f749d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f747c003f20 0x7f7498146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f749d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f749d200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74981f0440 0x7f74981f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f749d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74980cfc10 con 0x7f7498151d90 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f749d200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f749806e870 con 0x7f74981f0440 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f749d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74980cfd90 con 0x7f747c003f20 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60960/0 (socket says 172.21.15.67:60960) 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 -- 172.21.15.67:0/1743996049 learned_addr learned my addr 172.21.15.67:0/1743996049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 -- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f747c003f20 msgr2=0x7f7498146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f747c003f20 0x7f7498146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 -- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74981f0440 msgr2=0x7f74981f08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74981f0440 0x7f74981f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.304+0000 7f748fe00640 1 -- 172.21.15.67:0/1743996049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7498147230 con 0x7f7498151d90 2024-04-17T07:32:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f748fe00640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f749806e3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480007d30 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7480004bf0 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480007070 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749815bce0 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.305+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7498158740 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.306+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7498068a10 con 0x7f7498151d90 2024-04-17T07:32:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.307+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f7480047020 con 0x7f7498151d90 2024-04-17T07:32:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.308+0000 7f748f400640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 0x7f747406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.308+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f74800ace00 con 0x7f7498151d90 2024-04-17T07:32:36.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.309+0000 7f7497400640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 0x7f747406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.310+0000 7f7497400640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 0x7f747406ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7488000f60 tx=0x7f7488009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.313+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7480033e20 con 0x7f7498151d90 2024-04-17T07:32:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:36.557+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f749806ee80 con 0x7f7498151d90 2024-04-17T07:32:37.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.144+0000 7f748f400640 1 -- 172.21.15.67:0/1743996049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) ==== 118+0+0 (secure 0 0 0) 0x7f748007b110 con 0x7f7498151d90 2024-04-17T07:32:37.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 msgr2=0x7f747406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 0x7f747406ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7488000f60 tx=0x7f7488009040 comp rx=0 tx=0).stop 2024-04-17T07:32:37.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 msgr2=0x7f749815e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f748000c240 tx=0x7f749806e3f0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 shutdown_connections 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f747406ca40 0x7f747406ef00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74981f0440 0x7f74981f08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7498151d90 0x7f749815e4d0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 --2- 172.21.15.67:0/1743996049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f747c003f20 0x7f7498146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 >> 172.21.15.67:0/1743996049 conn(0x7f74980b2190 msgr2=0x7f74980ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 shutdown_connections 2024-04-17T07:32:37.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.148+0000 7f749d200640 1 -- 172.21.15.67:0/1743996049 wait complete. 2024-04-17T07:32:37.172 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:32:37.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:32:37.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.394+0000 7ff136600640 1 Processor -- start 2024-04-17T07:32:37.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.394+0000 7ff136600640 1 -- start start 2024-04-17T07:32:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.396+0000 7ff136600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.396+0000 7ff136600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 0x7ff130167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.396+0000 7ff136600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff130166c30 con 0x7ff130167650 2024-04-17T07:32:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.396+0000 7ff136600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff13006fde0 con 0x7ff130071f30 2024-04-17T07:32:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.396+0000 7ff136600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff13006f820 con 0x7ff1300cffe0 2024-04-17T07:32:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.397+0000 7ff12f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.397+0000 7ff127e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 0x7ff130167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.397+0000 7ff12f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:34224/0 (socket says 172.21.15.67:34224) 2024-04-17T07:32:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.397+0000 7ff12f400640 1 -- 172.21.15.67:0/2359743596 learned_addr learned my addr 172.21.15.67:0/2359743596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.398+0000 7ff12f400640 1 -- 172.21.15.67:0/2359743596 >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 msgr2=0x7ff130167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.398+0000 7ff12f400640 1 --2- 172.21.15.67:0/2359743596 >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 0x7ff130167110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.398+0000 7ff12f400640 1 -- 172.21.15.67:0/2359743596 >> v1:172.21.15.187:6789/0 conn(0x7ff130167650 legacy=0x7ff1301634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.398+0000 7ff12f400640 1 -- 172.21.15.67:0/2359743596 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13006fb00 con 0x7ff1300cffe0 2024-04-17T07:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.398+0000 7ff127e00640 1 --2- 172.21.15.67:0/2359743596 >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 0x7ff130167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.400+0000 7ff12f400640 1 --2- 172.21.15.67:0/2359743596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff120009870 tx=0x7ff12002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f0e284b18ea24d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.400+0000 7ff12ea00640 1 -- 172.21.15.67:0/2359743596 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120034070 con 0x7ff1300cffe0 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.400+0000 7ff12ea00640 1 -- 172.21.15.67:0/2359743596 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff120030840 con 0x7ff1300cffe0 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.400+0000 7ff12ea00640 1 -- 172.21.15.67:0/2359743596 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120030c10 con 0x7ff1300cffe0 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 -- 172.21.15.67:0/2359743596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 msgr2=0x7ff13006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 --2- 172.21.15.67:0/2359743596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff120009870 tx=0x7ff12002b980 comp rx=0 tx=0).stop 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 -- 172.21.15.67:0/2359743596 shutdown_connections 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 --2- 172.21.15.67:0/2359743596 >> v2:172.21.15.187:3300/0 conn(0x7ff130071f30 0x7ff130167110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 --2- 172.21.15.67:0/2359743596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1300cffe0 0x7ff13006dd10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 -- 172.21.15.67:0/2359743596 >> 172.21.15.67:0/2359743596 conn(0x7ff130067ee0 msgr2=0x7ff13006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 -- 172.21.15.67:0/2359743596 shutdown_connections 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.401+0000 7ff136600640 1 -- 172.21.15.67:0/2359743596 wait complete. 2024-04-17T07:32:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.403+0000 7ff136600640 1 Processor -- start 2024-04-17T07:32:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.403+0000 7ff136600640 1 -- start start 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff130071f30 0x7ff1300add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff130167650 0x7ff1300ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1300729f0 con 0x7ff1300cffe0 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff130072870 con 0x7ff130167650 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff136600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff130163bb0 con 0x7ff130071f30 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60964/0 (socket says 172.21.15.67:60964) 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 -- 172.21.15.67:0/3429946063 learned_addr learned my addr 172.21.15.67:0/3429946063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff12f400640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff130071f30 0x7ff1300add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff12fe00640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff130167650 0x7ff1300ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 -- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff130071f30 msgr2=0x7ff1300add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff130071f30 0x7ff1300add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.404+0000 7ff127e00640 1 -- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff130167650 msgr2=0x7ff1300ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127e00640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff130167650 0x7ff1300ac220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127e00640 1 -- 172.21.15.67:0/3429946063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13014d420 con 0x7ff1300cffe0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127e00640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff11800e9b0 tx=0x7ff11800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118017070 con 0x7ff1300cffe0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11800ca00 con 0x7ff1300cffe0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13014d700 con 0x7ff1300cffe0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11800cdf0 con 0x7ff1300cffe0 2024-04-17T07:32:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.405+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13014dcc0 con 0x7ff1300cffe0 2024-04-17T07:32:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.406+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff130068ca0 con 0x7ff1300cffe0 2024-04-17T07:32:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.407+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7ff11801a070 con 0x7ff1300cffe0 2024-04-17T07:32:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.408+0000 7ff127400640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 0x7ff10c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.408+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7ff11808bd50 con 0x7ff1300cffe0 2024-04-17T07:32:37.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.409+0000 7ff12f400640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 0x7ff10c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:37.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.410+0000 7ff12f400640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 0x7ff10c06ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff120036000 tx=0x7ff12003b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:37.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.413+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff118059fd0 con 0x7ff1300cffe0 2024-04-17T07:32:37.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:37.645+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff130164250 con 0x7ff1300cffe0 2024-04-17T07:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.147+0000 7ff127400640 1 -- 172.21.15.67:0/3429946063 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) ==== 122+0+0 (secure 0 0 0) 0x7ff1180598d0 con 0x7ff1300cffe0 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 msgr2=0x7ff10c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 0x7ff10c06ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff120036000 tx=0x7ff12003b040 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 msgr2=0x7ff1300abce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff11800e9b0 tx=0x7ff11800ee70 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 shutdown_connections 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff10c06ca40 0x7ff10c06ef00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff130167650 0x7ff1300ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1300cffe0 0x7ff1300abce0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 --2- 172.21.15.67:0/3429946063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff130071f30 0x7ff1300add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 >> 172.21.15.67:0/3429946063 conn(0x7ff130067ee0 msgr2=0x7ff1300b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.151+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 shutdown_connections 2024-04-17T07:32:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.152+0000 7ff136600640 1 -- 172.21.15.67:0/3429946063 wait complete. 2024-04-17T07:32:38.177 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:32:38.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.401+0000 7f52dba00640 1 Processor -- start 2024-04-17T07:32:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.401+0000 7f52dba00640 1 -- start start 2024-04-17T07:32:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d4161e70 0x7f52d40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d40ce4a0 0x7f52d4162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 0x7f52d40cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f52d40cd7f0 con 0x7f52d4161e70 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f52d40cc500 con 0x7f52d40cce90 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52dba00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f52d40c71e0 con 0x7f52d40ce4a0 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52d9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d4161e70 0x7f52d40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52d9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d4161e70 0x7f52d40c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60966/0 (socket says 172.21.15.67:60966) 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.403+0000 7f52d9000640 1 -- 172.21.15.67:0/3618337098 learned_addr learned my addr 172.21.15.67:0/3618337098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9000640 1 -- 172.21.15.67:0/3618337098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d40ce4a0 msgr2=0x7f52d4162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9000640 1 --2- 172.21.15.67:0/3618337098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d40ce4a0 0x7f52d4162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9000640 1 -- 172.21.15.67:0/3618337098 >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 msgr2=0x7f52d40cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9a00640 1 --2- 172.21.15.67:0/3618337098 >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 0x7f52d40cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9000640 1 --2- 172.21.15.67:0/3618337098 >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 0x7f52d40cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9000640 1 -- 172.21.15.67:0/3618337098 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d40cc7a0 con 0x7f52d4161e70 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.404+0000 7f52d9a00640 1 --2- 172.21.15.67:0/3618337098 >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 0x7f52d40cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.405+0000 7f52d9000640 1 --2- 172.21.15.67:0/3618337098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d4161e70 0x7f52d40c6c00 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f52d40701f0 tx=0x7f52c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34483536901afe2e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.405+0000 7f52d3400640 1 -- 172.21.15.67:0/3618337098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8035020 con 0x7f52d4161e70 2024-04-17T07:32:38.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.405+0000 7f52d3400640 1 -- 172.21.15.67:0/3618337098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c8031870 con 0x7f52d4161e70 2024-04-17T07:32:38.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.405+0000 7f52d3400640 1 -- 172.21.15.67:0/3618337098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8031c40 con 0x7f52d4161e70 2024-04-17T07:32:38.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 -- 172.21.15.67:0/3618337098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d4161e70 msgr2=0x7f52d40c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 --2- 172.21.15.67:0/3618337098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d4161e70 0x7f52d40c6c00 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f52d40701f0 tx=0x7f52c802b910 comp rx=0 tx=0).stop 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 -- 172.21.15.67:0/3618337098 shutdown_connections 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 --2- 172.21.15.67:0/3618337098 >> v2:172.21.15.187:3300/0 conn(0x7f52d40cce90 0x7f52d40cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 --2- 172.21.15.67:0/3618337098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d40ce4a0 0x7f52d4162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 --2- 172.21.15.67:0/3618337098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d4161e70 0x7f52d40c6c00 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 -- 172.21.15.67:0/3618337098 >> 172.21.15.67:0/3618337098 conn(0x7f52d406bf50 msgr2=0x7f52d40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 -- 172.21.15.67:0/3618337098 shutdown_connections 2024-04-17T07:32:38.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.406+0000 7f52dba00640 1 -- 172.21.15.67:0/3618337098 wait complete. 2024-04-17T07:32:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.408+0000 7f52dba00640 1 Processor -- start 2024-04-17T07:32:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.408+0000 7f52dba00640 1 -- start start 2024-04-17T07:32:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 0x7f52d41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 0x7f52d4143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4162250 con 0x7f52d40ce4a0 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40c54b0 con 0x7f52d40cce90 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.409+0000 7f52dba00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f52d41f7b10 con 0x7f52d4161e70 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60978/0 (socket says 172.21.15.67:60978) 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 -- 172.21.15.67:0/977736987 learned_addr learned my addr 172.21.15.67:0/977736987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d9000640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 0x7f52d41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d9a00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 0x7f52d4143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 -- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 msgr2=0x7f52d4143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 0x7f52d4143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 -- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 msgr2=0x7f52d41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 0x7f52d41571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 -- 172.21.15.67:0/977736987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d41f7df0 con 0x7f52d40ce4a0 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d9a00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 0x7f52d4143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d9000640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 0x7f52d41571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.410+0000 7f52d3e00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f52c000b4a0 tx=0x7f52c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.411+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0014070 con 0x7f52d40ce4a0 2024-04-17T07:32:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.411+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c0010610 con 0x7f52d40ce4a0 2024-04-17T07:32:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.411+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0010a00 con 0x7f52d40ce4a0 2024-04-17T07:32:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.411+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d41f80d0 con 0x7f52d40ce4a0 2024-04-17T07:32:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.411+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d41f8690 con 0x7f52d40ce4a0 2024-04-17T07:32:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.412+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d41f8890 con 0x7f52d40ce4a0 2024-04-17T07:32:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.413+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f52c0017070 con 0x7f52d40ce4a0 2024-04-17T07:32:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.414+0000 7f52d0c00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 0x7f52b406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.414+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f52c008b7f0 con 0x7f52d40ce4a0 2024-04-17T07:32:38.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.415+0000 7f52d9000640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 0x7f52b406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.416+0000 7f52d9000640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 0x7f52b406eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52c8036000 tx=0x7f52c8008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.419+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f52d41f8890 con 0x7f52d40ce4a0 2024-04-17T07:32:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.665+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52d4068b90 con 0x7f52d40ce4a0 2024-04-17T07:32:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.667+0000 7f52d0c00640 1 -- 172.21.15.67:0/977736987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7f52c0059a70 con 0x7f52d40ce4a0 2024-04-17T07:32:38.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:38.668 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:32:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:32:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.669+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 msgr2=0x7f52b406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.669+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 0x7f52b406eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52c8036000 tx=0x7f52c8008030 comp rx=0 tx=0).stop 2024-04-17T07:32:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.669+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 msgr2=0x7f52d4142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.669+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f52c000b4a0 tx=0x7f52c000b960 comp rx=0 tx=0).stop 2024-04-17T07:32:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 shutdown_connections 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f52b406ca10 0x7f52b406eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52d4161e70 0x7f52d4143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d40ce4a0 0x7f52d4142cd0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 --2- 172.21.15.67:0/977736987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52d40cce90 0x7f52d41571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 >> 172.21.15.67:0/977736987 conn(0x7f52d406bf50 msgr2=0x7f52d40b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 shutdown_connections 2024-04-17T07:32:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.670+0000 7f52dba00640 1 -- 172.21.15.67:0/977736987 wait complete. 2024-04-17T07:32:38.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:32:38.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.902+0000 7f00fea00640 1 Processor -- start 2024-04-17T07:32:38.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.902+0000 7f00fea00640 1 -- start start 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.903+0000 7f00fea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00f806b8c0 0x7f00f80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00f8141ed0 0x7f00f8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fea00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00f80cd4a0 con 0x7f00f8160280 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f00f815ce30 con 0x7f00f806b8c0 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fea00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00f81607f0 con 0x7f00f8141ed0 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00f806b8c0 0x7f00f80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00f806b8c0 0x7f00f80cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60980/0 (socket says 172.21.15.67:60980) 2024-04-17T07:32:38.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fd600640 1 -- 172.21.15.67:0/3935034281 learned_addr learned my addr 172.21.15.67:0/3935034281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:38.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fe000640 1 --2- 172.21.15.67:0/3935034281 >> v2:172.21.15.187:3300/0 conn(0x7f00f8141ed0 0x7f00f8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.904+0000 7f00fd600640 1 -- 172.21.15.67:0/3935034281 >> v2:172.21.15.187:3300/0 conn(0x7f00f8141ed0 msgr2=0x7f00f8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:38.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.905+0000 7f00fd600640 1 --2- 172.21.15.67:0/3935034281 >> v2:172.21.15.187:3300/0 conn(0x7f00f8141ed0 0x7f00f8142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.905+0000 7f00fd600640 1 -- 172.21.15.67:0/3935034281 >> v1:172.21.15.187:6789/0 conn(0x7f00f8160280 legacy=0x7f00f806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:38.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.905+0000 7f00fd600640 1 -- 172.21.15.67:0/3935034281 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f8160ad0 con 0x7f00f806b8c0 2024-04-17T07:32:38.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.906+0000 7f00fd600640 1 --2- 172.21.15.67:0/3935034281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f806b8c0 0x7f00f80cd8b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f00f80b2120 tx=0x7f00ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b995580011363b8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.906+0000 7f00f7e00640 1 -- 172.21.15.67:0/3935034281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec035020 con 0x7f00f806b8c0 2024-04-17T07:32:38.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.906+0000 7f00f7e00640 1 -- 172.21.15.67:0/3935034281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ec031870 con 0x7f00f806b8c0 2024-04-17T07:32:38.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.906+0000 7f00f7e00640 1 -- 172.21.15.67:0/3935034281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec031c40 con 0x7f00f806b8c0 2024-04-17T07:32:38.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 -- 172.21.15.67:0/3935034281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f806b8c0 msgr2=0x7f00f80cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 --2- 172.21.15.67:0/3935034281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f806b8c0 0x7f00f80cd8b0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f00f80b2120 tx=0x7f00ec02b910 comp rx=0 tx=0).stop 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 -- 172.21.15.67:0/3935034281 shutdown_connections 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 --2- 172.21.15.67:0/3935034281 >> v2:172.21.15.187:3300/0 conn(0x7f00f8141ed0 0x7f00f8142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 --2- 172.21.15.67:0/3935034281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f806b8c0 0x7f00f80cd8b0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 -- 172.21.15.67:0/3935034281 >> 172.21.15.67:0/3935034281 conn(0x7f00f8067d10 msgr2=0x7f00f80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 -- 172.21.15.67:0/3935034281 shutdown_connections 2024-04-17T07:32:38.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.907+0000 7f00fea00640 1 -- 172.21.15.67:0/3935034281 wait complete. 2024-04-17T07:32:38.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.910+0000 7f00fea00640 1 Processor -- start 2024-04-17T07:32:38.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.910+0000 7f00fea00640 1 -- start start 2024-04-17T07:32:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 0x7f00f813eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 0x7f00f81f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00f8161090 con 0x7f00f8160280 2024-04-17T07:32:38.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80cddf0 con 0x7f00f806b8c0 2024-04-17T07:32:38.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00f813f9b0 con 0x7f00f8141ed0 2024-04-17T07:32:38.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fe000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fe000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60986/0 (socket says 172.21.15.67:60986) 2024-04-17T07:32:38.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.911+0000 7f00fe000640 1 -- 172.21.15.67:0/1404069409 learned_addr learned my addr 172.21.15.67:0/1404069409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fd600640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 0x7f00f813eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 -- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 msgr2=0x7f00f81f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fcc00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 0x7f00f81f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 0x7f00f81f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 -- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 msgr2=0x7f00f813eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 0x7f00f813eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 -- 172.21.15.67:0/1404069409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f81f4230 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fd600640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 0x7f00f813eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fcc00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 0x7f00f81f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00fe000640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f00f000cd70 tx=0x7f00f000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.912+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0003b90 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.913+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f0003d30 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.913+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0004240 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.913+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f8144140 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.913+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f81f44d0 con 0x7f00f8160280 2024-04-17T07:32:38.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.914+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f8144820 con 0x7f00f8160280 2024-04-17T07:32:38.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.915+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f00f00043e0 con 0x7f00f8160280 2024-04-17T07:32:38.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.916+0000 7f00f5600640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 0x7f00d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:38.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.916+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f00f008c980 con 0x7f00f8160280 2024-04-17T07:32:38.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.917+0000 7f00fd600640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 0x7f00d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:38.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.918+0000 7f00fd600640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 0x7f00d806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00ec036000 tx=0x7f00ec008030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:38.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:38.920+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f00f005aaf0 con 0x7f00f8160280 2024-04-17T07:32:39.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.125+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00f81f46d0 con 0x7f00f8160280 2024-04-17T07:32:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.128+0000 7f00f5600640 1 -- 172.21.15.67:0/1404069409 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+16571 (secure 0 0 0) 0x7f00f005a3f0 con 0x7f00f8160280 2024-04-17T07:32:39.128 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:39.129 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":127,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:36.105839+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:32:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 msgr2=0x7f00d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 0x7f00d806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00ec036000 tx=0x7f00ec008030 comp rx=0 tx=0).stop 2024-04-17T07:32:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 msgr2=0x7f00f815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f00f000cd70 tx=0x7f00f000a660 comp rx=0 tx=0).stop 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 shutdown_connections 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f00d806ca10 0x7f00d806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00f8160280 0x7f00f815b5f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00f8141ed0 0x7f00f81f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 --2- 172.21.15.67:0/1404069409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00f806b8c0 0x7f00f813eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 >> 172.21.15.67:0/1404069409 conn(0x7f00f8067d10 msgr2=0x7f00f80b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 shutdown_connections 2024-04-17T07:32:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.131+0000 7f00fea00640 1 -- 172.21.15.67:0/1404069409 wait complete. 2024-04-17T07:32:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:32:39.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:32:39.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.367+0000 7f6191000640 1 Processor -- start 2024-04-17T07:32:39.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.367+0000 7f6191000640 1 -- start start 2024-04-17T07:32:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.369+0000 7f6191000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 0x7f618c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.369+0000 7f6191000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.369+0000 7f6191000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618c0cd4a0 con 0x7f618c06b8c0 2024-04-17T07:32:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.369+0000 7f6191000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f618c15ce30 con 0x7f618c141ed0 2024-04-17T07:32:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.369+0000 7f6191000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f618c1607f0 con 0x7f618c160280 2024-04-17T07:32:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.370+0000 7f618be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.370+0000 7f6183e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 0x7f618c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.370+0000 7f618be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:47692/0 (socket says 172.21.15.67:47692) 2024-04-17T07:32:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.370+0000 7f618be00640 1 -- 172.21.15.67:0/3049723192 learned_addr learned my addr 172.21.15.67:0/3049723192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.371+0000 7f6183e00640 1 -- 172.21.15.67:0/3049723192 >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 msgr2=0x7f618c142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.371+0000 7f6183e00640 1 --2- 172.21.15.67:0/3049723192 >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.371+0000 7f6183e00640 1 -- 172.21.15.67:0/3049723192 >> v1:172.21.15.187:6789/0 conn(0x7f618c06b8c0 legacy=0x7f618c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.371+0000 7f6183e00640 1 -- 172.21.15.67:0/3049723192 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c160ad0 con 0x7f618c160280 2024-04-17T07:32:39.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.371+0000 7f618be00640 1 --2- 172.21.15.67:0/3049723192 >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:39.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.372+0000 7f6183e00640 1 --2- 172.21.15.67:0/3049723192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 0x7f618c06bca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=430eb6bf69c5db97 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.373+0000 7f618aa00640 1 -- 172.21.15.67:0/3049723192 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174035020 con 0x7f618c160280 2024-04-17T07:32:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.373+0000 7f618aa00640 1 -- 172.21.15.67:0/3049723192 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174031810 con 0x7f618c160280 2024-04-17T07:32:39.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.373+0000 7f618aa00640 1 -- 172.21.15.67:0/3049723192 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174031be0 con 0x7f618c160280 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.373+0000 7f6191000640 1 -- 172.21.15.67:0/3049723192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 msgr2=0x7f618c06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.373+0000 7f6191000640 1 --2- 172.21.15.67:0/3049723192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 0x7f618c06bca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 -- 172.21.15.67:0/3049723192 shutdown_connections 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 --2- 172.21.15.67:0/3049723192 >> v2:172.21.15.187:3300/0 conn(0x7f618c141ed0 0x7f618c142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 --2- 172.21.15.67:0/3049723192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c160280 0x7f618c06bca0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 -- 172.21.15.67:0/3049723192 >> 172.21.15.67:0/3049723192 conn(0x7f618c067d10 msgr2=0x7f618c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 -- 172.21.15.67:0/3049723192 shutdown_connections 2024-04-17T07:32:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.374+0000 7f6191000640 1 -- 172.21.15.67:0/3049723192 wait complete. 2024-04-17T07:32:39.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.375+0000 7f6191000640 1 Processor -- start 2024-04-17T07:32:39.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 -- start start 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f618c06b8c0 0x7f618c145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c141ed0 0x7f618c136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f618c161090 con 0x7f618c160280 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f618c06c1e0 con 0x7f618c06b8c0 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.376+0000 7f6191000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f618c13f9b0 con 0x7f618c141ed0 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32770/0 (socket says 172.21.15.67:32770) 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 -- 172.21.15.67:0/3929914872 learned_addr learned my addr 172.21.15.67:0/3929914872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f6183e00640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c141ed0 0x7f618c136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618b400640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f618c06b8c0 0x7f618c145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 -- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c141ed0 msgr2=0x7f618c136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c141ed0 0x7f618c136bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 -- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f618c06b8c0 msgr2=0x7f618c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f618c06b8c0 0x7f618c145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.377+0000 7f618be00640 1 -- 172.21.15.67:0/3929914872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c1406b0 con 0x7f618c160280 2024-04-17T07:32:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f618be00640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6184009f10 tx=0x7f618400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618400cc40 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f618400cde0 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184003bb0 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c140990 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.378+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c140f50 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.379+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c068a10 con 0x7f618c160280 2024-04-17T07:32:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.380+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f618401b030 con 0x7f618c160280 2024-04-17T07:32:39.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.381+0000 7f6183400640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 0x7f616806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.381+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f618408c2a0 con 0x7f618c160280 2024-04-17T07:32:39.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.382+0000 7f618b400640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 0x7f616806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.384+0000 7f618b400640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 0x7f616806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f617c007fd0 tx=0x7f617c009040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.386+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6184022020 con 0x7f618c160280 2024-04-17T07:32:39.530 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T07:32:39.530+0000 7f0f59600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:32:39.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.588+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 --> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f618c0b1000 con 0x7f616806ca10 2024-04-17T07:32:39.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.612+0000 7f6183400640 1 -- 172.21.15.67:0/3929914872 <== mgr.6394 v2:172.21.15.187:6832/2517863062 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252489 (secure 0 0 0) 0x7f618c0b1000 con 0x7f616806ca10 2024-04-17T07:32:39.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:39.618 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:32:39.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 msgr2=0x7f616806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 0x7f616806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f617c007fd0 tx=0x7f617c009040 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 msgr2=0x7f618c143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6184009f10 tx=0x7f618400c490 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 shutdown_connections 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f616806ca10 0x7f616806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f618c160280 0x7f618c143d20 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618c141ed0 0x7f618c136bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 --2- 172.21.15.67:0/3929914872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f618c06b8c0 0x7f618c145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 >> 172.21.15.67:0/3929914872 conn(0x7f618c067d10 msgr2=0x7f618c0b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 shutdown_connections 2024-04-17T07:32:39.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.619+0000 7f6191000640 1 -- 172.21.15.67:0/3929914872 wait complete. 2024-04-17T07:32:39.641 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T07:32:37.933357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":372,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336444,"kb_used_data":13436,"kb_used_omap":12,"kb_used_meta":322867,"kb_avail":754638276,"statfs":{"total":773094113280,"available":772749594624,"internally_reserved":0,"allocated":13758464,"data_stored":7171616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":330616368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.854483"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175765+0000","last_change":"2024-04-17T07:32:35.141914+0000","last_active":"2024-04-17T07:32:36.175765+0000","last_peered":"2024-04-17T07:32:36.175765+0000","last_clean":"2024-04-17T07:32:36.175765+0000","last_became_active":"2024-04-17T07:32:35.140482+0000","last_became_peered":"2024-04-17T07:32:35.140482+0000","last_unstale":"2024-04-17T07:32:36.175765+0000","last_undegraded":"2024-04-17T07:32:36.175765+0000","last_fullsized":"2024-04-17T07:32:36.175765+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:18.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161746+0000","last_change":"2024-04-17T07:32:35.130073+0000","last_active":"2024-04-17T07:32:36.161746+0000","last_peered":"2024-04-17T07:32:36.161746+0000","last_clean":"2024-04-17T07:32:36.161746+0000","last_became_active":"2024-04-17T07:32:35.129360+0000","last_became_peered":"2024-04-17T07:32:35.129360+0000","last_unstale":"2024-04-17T07:32:36.161746+0000","last_undegraded":"2024-04-17T07:32:36.161746+0000","last_fullsized":"2024-04-17T07:32:36.161746+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:43.978087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175673+0000","last_change":"2024-04-17T07:32:35.143728+0000","last_active":"2024-04-17T07:32:36.175673+0000","last_peered":"2024-04-17T07:32:36.175673+0000","last_clean":"2024-04-17T07:32:36.175673+0000","last_became_active":"2024-04-17T07:32:35.142923+0000","last_became_peered":"2024-04-17T07:32:35.142923+0000","last_unstale":"2024-04-17T07:32:36.175673+0000","last_undegraded":"2024-04-17T07:32:36.175673+0000","last_fullsized":"2024-04-17T07:32:36.175673+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:11:19.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176517+0000","last_change":"2024-04-17T07:32:35.142131+0000","last_active":"2024-04-17T07:32:36.176517+0000","last_peered":"2024-04-17T07:32:36.176517+0000","last_clean":"2024-04-17T07:32:36.176517+0000","last_became_active":"2024-04-17T07:32:35.141836+0000","last_became_peered":"2024-04-17T07:32:35.141836+0000","last_unstale":"2024-04-17T07:32:36.176517+0000","last_undegraded":"2024-04-17T07:32:36.176517+0000","last_fullsized":"2024-04-17T07:32:36.176517+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:35:08.418431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116602+0000","last_change":"2024-04-17T07:32:35.155566+0000","last_active":"2024-04-17T07:32:36.116602+0000","last_peered":"2024-04-17T07:32:36.116602+0000","last_clean":"2024-04-17T07:32:36.116602+0000","last_became_active":"2024-04-17T07:32:35.155238+0000","last_became_peered":"2024-04-17T07:32:35.155238+0000","last_unstale":"2024-04-17T07:32:36.116602+0000","last_undegraded":"2024-04-17T07:32:36.116602+0000","last_fullsized":"2024-04-17T07:32:36.116602+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:12.206597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117781+0000","last_change":"2024-04-17T07:32:35.144948+0000","last_active":"2024-04-17T07:32:36.117781+0000","last_peered":"2024-04-17T07:32:36.117781+0000","last_clean":"2024-04-17T07:32:36.117781+0000","last_became_active":"2024-04-17T07:32:35.143203+0000","last_became_peered":"2024-04-17T07:32:35.143203+0000","last_unstale":"2024-04-17T07:32:36.117781+0000","last_undegraded":"2024-04-17T07:32:36.117781+0000","last_fullsized":"2024-04-17T07:32:36.117781+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:08:39.614569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116601+0000","last_change":"2024-04-17T07:32:35.144278+0000","last_active":"2024-04-17T07:32:36.116601+0000","last_peered":"2024-04-17T07:32:36.116601+0000","last_clean":"2024-04-17T07:32:36.116601+0000","last_became_active":"2024-04-17T07:32:35.143964+0000","last_became_peered":"2024-04-17T07:32:35.143964+0000","last_unstale":"2024-04-17T07:32:36.116601+0000","last_undegraded":"2024-04-17T07:32:36.116601+0000","last_fullsized":"2024-04-17T07:32:36.116601+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:46.435011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161312+0000","last_change":"2024-04-17T07:32:35.147820+0000","last_active":"2024-04-17T07:32:36.161312+0000","last_peered":"2024-04-17T07:32:36.161312+0000","last_clean":"2024-04-17T07:32:36.161312+0000","last_became_active":"2024-04-17T07:32:35.147552+0000","last_became_peered":"2024-04-17T07:32:35.147552+0000","last_unstale":"2024-04-17T07:32:36.161312+0000","last_undegraded":"2024-04-17T07:32:36.161312+0000","last_fullsized":"2024-04-17T07:32:36.161312+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:27.796262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178075+0000","last_change":"2024-04-17T07:32:35.141551+0000","last_active":"2024-04-17T07:32:36.178075+0000","last_peered":"2024-04-17T07:32:36.178075+0000","last_clean":"2024-04-17T07:32:36.178075+0000","last_became_active":"2024-04-17T07:32:35.140192+0000","last_became_peered":"2024-04-17T07:32:35.140192+0000","last_unstale":"2024-04-17T07:32:36.178075+0000","last_undegraded":"2024-04-17T07:32:36.178075+0000","last_fullsized":"2024-04-17T07:32:36.178075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:15.720120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116015+0000","last_change":"2024-04-17T07:32:35.160179+0000","last_active":"2024-04-17T07:32:36.116015+0000","last_peered":"2024-04-17T07:32:36.116015+0000","last_clean":"2024-04-17T07:32:36.116015+0000","last_became_active":"2024-04-17T07:32:35.159858+0000","last_became_peered":"2024-04-17T07:32:35.159858+0000","last_unstale":"2024-04-17T07:32:36.116015+0000","last_undegraded":"2024-04-17T07:32:36.116015+0000","last_fullsized":"2024-04-17T07:32:36.116015+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:04.303000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117667+0000","last_change":"2024-04-17T07:32:35.148431+0000","last_active":"2024-04-17T07:32:36.117667+0000","last_peered":"2024-04-17T07:32:36.117667+0000","last_clean":"2024-04-17T07:32:36.117667+0000","last_became_active":"2024-04-17T07:32:35.148110+0000","last_became_peered":"2024-04-17T07:32:35.148110+0000","last_unstale":"2024-04-17T07:32:36.117667+0000","last_undegraded":"2024-04-17T07:32:36.117667+0000","last_fullsized":"2024-04-17T07:32:36.117667+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:00:16.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177779+0000","last_change":"2024-04-17T07:32:35.139822+0000","last_active":"2024-04-17T07:32:36.177779+0000","last_peered":"2024-04-17T07:32:36.177779+0000","last_clean":"2024-04-17T07:32:36.177779+0000","last_became_active":"2024-04-17T07:32:35.139437+0000","last_became_peered":"2024-04-17T07:32:35.139437+0000","last_unstale":"2024-04-17T07:32:36.177779+0000","last_undegraded":"2024-04-17T07:32:36.177779+0000","last_fullsized":"2024-04-17T07:32:36.177779+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:11.257154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176304+0000","last_change":"2024-04-17T07:32:35.140029+0000","last_active":"2024-04-17T07:32:36.176304+0000","last_peered":"2024-04-17T07:32:36.176304+0000","last_clean":"2024-04-17T07:32:36.176304+0000","last_became_active":"2024-04-17T07:32:35.139056+0000","last_became_peered":"2024-04-17T07:32:35.139056+0000","last_unstale":"2024-04-17T07:32:36.176304+0000","last_undegraded":"2024-04-17T07:32:36.176304+0000","last_fullsized":"2024-04-17T07:32:36.176304+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:19.263171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111736+0000","last_change":"2024-04-17T07:32:35.147837+0000","last_active":"2024-04-17T07:32:36.111736+0000","last_peered":"2024-04-17T07:32:36.111736+0000","last_clean":"2024-04-17T07:32:36.111736+0000","last_became_active":"2024-04-17T07:32:35.147071+0000","last_became_peered":"2024-04-17T07:32:35.147071+0000","last_unstale":"2024-04-17T07:32:36.111736+0000","last_undegraded":"2024-04-17T07:32:36.111736+0000","last_fullsized":"2024-04-17T07:32:36.111736+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:02.543243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176209+0000","last_change":"2024-04-17T07:32:35.141660+0000","last_active":"2024-04-17T07:32:36.176209+0000","last_peered":"2024-04-17T07:32:36.176209+0000","last_clean":"2024-04-17T07:32:36.176209+0000","last_became_active":"2024-04-17T07:32:35.141127+0000","last_became_peered":"2024-04-17T07:32:35.141127+0000","last_unstale":"2024-04-17T07:32:36.176209+0000","last_undegraded":"2024-04-17T07:32:36.176209+0000","last_fullsized":"2024-04-17T07:32:36.176209+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:38.661050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116749+0000","last_change":"2024-04-17T07:32:35.158821+0000","last_active":"2024-04-17T07:32:36.116749+0000","last_peered":"2024-04-17T07:32:36.116749+0000","last_clean":"2024-04-17T07:32:36.116749+0000","last_became_active":"2024-04-17T07:32:35.158527+0000","last_became_peered":"2024-04-17T07:32:35.158527+0000","last_unstale":"2024-04-17T07:32:36.116749+0000","last_undegraded":"2024-04-17T07:32:36.116749+0000","last_fullsized":"2024-04-17T07:32:36.116749+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:57.877675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161641+0000","last_change":"2024-04-17T07:32:35.137005+0000","last_active":"2024-04-17T07:32:36.161641+0000","last_peered":"2024-04-17T07:32:36.161641+0000","last_clean":"2024-04-17T07:32:36.161641+0000","last_became_active":"2024-04-17T07:32:35.136712+0000","last_became_peered":"2024-04-17T07:32:35.136712+0000","last_unstale":"2024-04-17T07:32:36.161641+0000","last_undegraded":"2024-04-17T07:32:36.161641+0000","last_fullsized":"2024-04-17T07:32:36.161641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:32.292820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176710+0000","last_change":"2024-04-17T07:32:35.157700+0000","last_active":"2024-04-17T07:32:36.176710+0000","last_peered":"2024-04-17T07:32:36.176710+0000","last_clean":"2024-04-17T07:32:36.176710+0000","last_became_active":"2024-04-17T07:32:35.154715+0000","last_became_peered":"2024-04-17T07:32:35.154715+0000","last_unstale":"2024-04-17T07:32:36.176710+0000","last_undegraded":"2024-04-17T07:32:36.176710+0000","last_fullsized":"2024-04-17T07:32:36.176710+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:11.363240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112558+0000","last_change":"2024-04-17T07:32:35.157023+0000","last_active":"2024-04-17T07:32:36.112558+0000","last_peered":"2024-04-17T07:32:36.112558+0000","last_clean":"2024-04-17T07:32:36.112558+0000","last_became_active":"2024-04-17T07:32:35.154591+0000","last_became_peered":"2024-04-17T07:32:35.154591+0000","last_unstale":"2024-04-17T07:32:36.112558+0000","last_undegraded":"2024-04-17T07:32:36.112558+0000","last_fullsized":"2024-04-17T07:32:36.112558+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:37.978579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178230+0000","last_change":"2024-04-17T07:32:35.131815+0000","last_active":"2024-04-17T07:32:36.178230+0000","last_peered":"2024-04-17T07:32:36.178230+0000","last_clean":"2024-04-17T07:32:36.178230+0000","last_became_active":"2024-04-17T07:32:35.130681+0000","last_became_peered":"2024-04-17T07:32:35.130681+0000","last_unstale":"2024-04-17T07:32:36.178230+0000","last_undegraded":"2024-04-17T07:32:36.178230+0000","last_fullsized":"2024-04-17T07:32:36.178230+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:41.171171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112053+0000","last_change":"2024-04-17T07:32:35.145767+0000","last_active":"2024-04-17T07:32:36.112053+0000","last_peered":"2024-04-17T07:32:36.112053+0000","last_clean":"2024-04-17T07:32:36.112053+0000","last_became_active":"2024-04-17T07:32:35.145300+0000","last_became_peered":"2024-04-17T07:32:35.145300+0000","last_unstale":"2024-04-17T07:32:36.112053+0000","last_undegraded":"2024-04-17T07:32:36.112053+0000","last_fullsized":"2024-04-17T07:32:36.112053+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:32.292820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161414+0000","last_change":"2024-04-17T07:32:35.132096+0000","last_active":"2024-04-17T07:32:36.161414+0000","last_peered":"2024-04-17T07:32:36.161414+0000","last_clean":"2024-04-17T07:32:36.161414+0000","last_became_active":"2024-04-17T07:32:35.131218+0000","last_became_peered":"2024-04-17T07:32:35.131218+0000","last_unstale":"2024-04-17T07:32:36.161414+0000","last_undegraded":"2024-04-17T07:32:36.161414+0000","last_fullsized":"2024-04-17T07:32:36.161414+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:42.180598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161575+0000","last_change":"2024-04-17T07:32:35.129053+0000","last_active":"2024-04-17T07:32:36.161575+0000","last_peered":"2024-04-17T07:32:36.161575+0000","last_clean":"2024-04-17T07:32:36.161575+0000","last_became_active":"2024-04-17T07:32:35.128491+0000","last_became_peered":"2024-04-17T07:32:35.128491+0000","last_unstale":"2024-04-17T07:32:36.161575+0000","last_undegraded":"2024-04-17T07:32:36.161575+0000","last_fullsized":"2024-04-17T07:32:36.161575+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:52.866924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117144+0000","last_change":"2024-04-17T07:32:35.169272+0000","last_active":"2024-04-17T07:32:36.117144+0000","last_peered":"2024-04-17T07:32:36.117144+0000","last_clean":"2024-04-17T07:32:36.117144+0000","last_became_active":"2024-04-17T07:32:35.169027+0000","last_became_peered":"2024-04-17T07:32:35.169027+0000","last_unstale":"2024-04-17T07:32:36.117144+0000","last_undegraded":"2024-04-17T07:32:36.117144+0000","last_fullsized":"2024-04-17T07:32:36.117144+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:42.070845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117324+0000","last_change":"2024-04-17T07:32:35.159348+0000","last_active":"2024-04-17T07:32:36.117324+0000","last_peered":"2024-04-17T07:32:36.117324+0000","last_clean":"2024-04-17T07:32:36.117324+0000","last_became_active":"2024-04-17T07:32:35.158963+0000","last_became_peered":"2024-04-17T07:32:35.158963+0000","last_unstale":"2024-04-17T07:32:36.117324+0000","last_undegraded":"2024-04-17T07:32:36.117324+0000","last_fullsized":"2024-04-17T07:32:36.117324+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:28:33.012641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162020+0000","last_change":"2024-04-17T07:32:35.143295+0000","last_active":"2024-04-17T07:32:36.162020+0000","last_peered":"2024-04-17T07:32:36.162020+0000","last_clean":"2024-04-17T07:32:36.162020+0000","last_became_active":"2024-04-17T07:32:35.142984+0000","last_became_peered":"2024-04-17T07:32:35.142984+0000","last_unstale":"2024-04-17T07:32:36.162020+0000","last_undegraded":"2024-04-17T07:32:36.162020+0000","last_fullsized":"2024-04-17T07:32:36.162020+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:21.938044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118107+0000","last_change":"2024-04-17T07:32:35.154535+0000","last_active":"2024-04-17T07:32:36.118107+0000","last_peered":"2024-04-17T07:32:36.118107+0000","last_clean":"2024-04-17T07:32:36.118107+0000","last_became_active":"2024-04-17T07:32:35.154009+0000","last_became_peered":"2024-04-17T07:32:35.154009+0000","last_unstale":"2024-04-17T07:32:36.118107+0000","last_undegraded":"2024-04-17T07:32:36.118107+0000","last_fullsized":"2024-04-17T07:32:36.118107+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:47.489094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116314+0000","last_change":"2024-04-17T07:32:35.155640+0000","last_active":"2024-04-17T07:32:36.116314+0000","last_peered":"2024-04-17T07:32:36.116314+0000","last_clean":"2024-04-17T07:32:36.116314+0000","last_became_active":"2024-04-17T07:32:35.155334+0000","last_became_peered":"2024-04-17T07:32:35.155334+0000","last_unstale":"2024-04-17T07:32:36.116314+0000","last_undegraded":"2024-04-17T07:32:36.116314+0000","last_fullsized":"2024-04-17T07:32:36.116314+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:42.808017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161724+0000","last_change":"2024-04-17T07:32:35.145835+0000","last_active":"2024-04-17T07:32:36.161724+0000","last_peered":"2024-04-17T07:32:36.161724+0000","last_clean":"2024-04-17T07:32:36.161724+0000","last_became_active":"2024-04-17T07:32:35.145555+0000","last_became_peered":"2024-04-17T07:32:35.145555+0000","last_unstale":"2024-04-17T07:32:36.161724+0000","last_undegraded":"2024-04-17T07:32:36.161724+0000","last_fullsized":"2024-04-17T07:32:36.161724+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:37.978579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111765+0000","last_change":"2024-04-17T07:32:35.156628+0000","last_active":"2024-04-17T07:32:36.111765+0000","last_peered":"2024-04-17T07:32:36.111765+0000","last_clean":"2024-04-17T07:32:36.111765+0000","last_became_active":"2024-04-17T07:32:35.155845+0000","last_became_peered":"2024-04-17T07:32:35.155845+0000","last_unstale":"2024-04-17T07:32:36.111765+0000","last_undegraded":"2024-04-17T07:32:36.111765+0000","last_fullsized":"2024-04-17T07:32:36.111765+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:45.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176534+0000","last_change":"2024-04-17T07:32:35.146638+0000","last_active":"2024-04-17T07:32:36.176534+0000","last_peered":"2024-04-17T07:32:36.176534+0000","last_clean":"2024-04-17T07:32:36.176534+0000","last_became_active":"2024-04-17T07:32:35.146254+0000","last_became_peered":"2024-04-17T07:32:35.146254+0000","last_unstale":"2024-04-17T07:32:36.176534+0000","last_undegraded":"2024-04-17T07:32:36.176534+0000","last_fullsized":"2024-04-17T07:32:36.176534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:05.987532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175220+0000","last_change":"2024-04-17T07:32:35.144190+0000","last_active":"2024-04-17T07:32:36.175220+0000","last_peered":"2024-04-17T07:32:36.175220+0000","last_clean":"2024-04-17T07:32:36.175220+0000","last_became_active":"2024-04-17T07:32:35.143877+0000","last_became_peered":"2024-04-17T07:32:35.143877+0000","last_unstale":"2024-04-17T07:32:36.175220+0000","last_undegraded":"2024-04-17T07:32:36.175220+0000","last_fullsized":"2024-04-17T07:32:36.175220+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:12.206597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112563+0000","last_change":"2024-04-17T07:32:35.147344+0000","last_active":"2024-04-17T07:32:36.112563+0000","last_peered":"2024-04-17T07:32:36.112563+0000","last_clean":"2024-04-17T07:32:36.112563+0000","last_became_active":"2024-04-17T07:32:35.146989+0000","last_became_peered":"2024-04-17T07:32:35.146989+0000","last_unstale":"2024-04-17T07:32:36.112563+0000","last_undegraded":"2024-04-17T07:32:36.112563+0000","last_fullsized":"2024-04-17T07:32:36.112563+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:43.978087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161887+0000","last_change":"2024-04-17T07:32:34.120366+0000","last_active":"2024-04-17T07:32:36.161887+0000","last_peered":"2024-04-17T07:32:36.161887+0000","last_clean":"2024-04-17T07:32:36.161887+0000","last_became_active":"2024-04-17T07:32:34.120065+0000","last_became_peered":"2024-04-17T07:32:34.120065+0000","last_unstale":"2024-04-17T07:32:36.161887+0000","last_undegraded":"2024-04-17T07:32:36.161887+0000","last_fullsized":"2024-04-17T07:32:36.161887+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:06.125055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176830+0000","last_change":"2024-04-17T07:32:35.148547+0000","last_active":"2024-04-17T07:32:36.176830+0000","last_peered":"2024-04-17T07:32:36.176830+0000","last_clean":"2024-04-17T07:32:36.176830+0000","last_became_active":"2024-04-17T07:32:35.148258+0000","last_became_peered":"2024-04-17T07:32:35.148258+0000","last_unstale":"2024-04-17T07:32:36.176830+0000","last_undegraded":"2024-04-17T07:32:36.176830+0000","last_fullsized":"2024-04-17T07:32:36.176830+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:32:57.671400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117971+0000","last_change":"2024-04-17T07:32:34.119437+0000","last_active":"2024-04-17T07:32:36.117971+0000","last_peered":"2024-04-17T07:32:36.117971+0000","last_clean":"2024-04-17T07:32:36.117971+0000","last_became_active":"2024-04-17T07:32:34.118222+0000","last_became_peered":"2024-04-17T07:32:34.118222+0000","last_unstale":"2024-04-17T07:32:36.117971+0000","last_undegraded":"2024-04-17T07:32:36.117971+0000","last_fullsized":"2024-04-17T07:32:36.117971+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:41.002892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175767+0000","last_change":"2024-04-17T07:32:35.146873+0000","last_active":"2024-04-17T07:32:36.175767+0000","last_peered":"2024-04-17T07:32:36.175767+0000","last_clean":"2024-04-17T07:32:36.175767+0000","last_became_active":"2024-04-17T07:32:35.142068+0000","last_became_peered":"2024-04-17T07:32:35.142068+0000","last_unstale":"2024-04-17T07:32:36.175767+0000","last_undegraded":"2024-04-17T07:32:36.175767+0000","last_fullsized":"2024-04-17T07:32:36.175767+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:34.370047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161991+0000","last_change":"2024-04-17T07:32:34.119033+0000","last_active":"2024-04-17T07:32:36.161991+0000","last_peered":"2024-04-17T07:32:36.161991+0000","last_clean":"2024-04-17T07:32:36.161991+0000","last_became_active":"2024-04-17T07:32:34.118728+0000","last_became_peered":"2024-04-17T07:32:34.118728+0000","last_unstale":"2024-04-17T07:32:36.161991+0000","last_undegraded":"2024-04-17T07:32:36.161991+0000","last_fullsized":"2024-04-17T07:32:36.161991+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:28.471682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177991+0000","last_change":"2024-04-17T07:32:35.136460+0000","last_active":"2024-04-17T07:32:36.177991+0000","last_peered":"2024-04-17T07:32:36.177991+0000","last_clean":"2024-04-17T07:32:36.177991+0000","last_became_active":"2024-04-17T07:32:35.135961+0000","last_became_peered":"2024-04-17T07:32:35.135961+0000","last_unstale":"2024-04-17T07:32:36.177991+0000","last_undegraded":"2024-04-17T07:32:36.177991+0000","last_fullsized":"2024-04-17T07:32:36.177991+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:00.016507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161583+0000","last_change":"2024-04-17T07:32:34.117205+0000","last_active":"2024-04-17T07:32:36.161583+0000","last_peered":"2024-04-17T07:32:36.161583+0000","last_clean":"2024-04-17T07:32:36.161583+0000","last_became_active":"2024-04-17T07:32:34.116778+0000","last_became_peered":"2024-04-17T07:32:34.116778+0000","last_unstale":"2024-04-17T07:32:36.161583+0000","last_undegraded":"2024-04-17T07:32:36.161583+0000","last_fullsized":"2024-04-17T07:32:36.161583+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:30.083718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179087+0000","last_change":"2024-04-17T07:32:35.129661+0000","last_active":"2024-04-17T07:32:36.179087+0000","last_peered":"2024-04-17T07:32:36.179087+0000","last_clean":"2024-04-17T07:32:36.179087+0000","last_became_active":"2024-04-17T07:32:35.129155+0000","last_became_peered":"2024-04-17T07:32:35.129155+0000","last_unstale":"2024-04-17T07:32:36.179087+0000","last_undegraded":"2024-04-17T07:32:36.179087+0000","last_fullsized":"2024-04-17T07:32:36.179087+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:38.654110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175344+0000","last_change":"2024-04-17T07:32:34.118731+0000","last_active":"2024-04-17T07:32:36.175344+0000","last_peered":"2024-04-17T07:32:36.175344+0000","last_clean":"2024-04-17T07:32:36.175344+0000","last_became_active":"2024-04-17T07:32:34.118465+0000","last_became_peered":"2024-04-17T07:32:34.118465+0000","last_unstale":"2024-04-17T07:32:36.175344+0000","last_undegraded":"2024-04-17T07:32:36.175344+0000","last_fullsized":"2024-04-17T07:32:36.175344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:14:49.847153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118396+0000","last_change":"2024-04-17T07:32:35.143873+0000","last_active":"2024-04-17T07:32:36.118396+0000","last_peered":"2024-04-17T07:32:36.118396+0000","last_clean":"2024-04-17T07:32:36.118396+0000","last_became_active":"2024-04-17T07:32:35.143144+0000","last_became_peered":"2024-04-17T07:32:35.143144+0000","last_unstale":"2024-04-17T07:32:36.118396+0000","last_undegraded":"2024-04-17T07:32:36.118396+0000","last_fullsized":"2024-04-17T07:32:36.118396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:02:14.597223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161520+0000","last_change":"2024-04-17T07:32:34.109956+0000","last_active":"2024-04-17T07:32:36.161520+0000","last_peered":"2024-04-17T07:32:36.161520+0000","last_clean":"2024-04-17T07:32:36.161520+0000","last_became_active":"2024-04-17T07:32:34.109727+0000","last_became_peered":"2024-04-17T07:32:34.109727+0000","last_unstale":"2024-04-17T07:32:36.161520+0000","last_undegraded":"2024-04-17T07:32:36.161520+0000","last_fullsized":"2024-04-17T07:32:36.161520+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:33.146222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176067+0000","last_change":"2024-04-17T07:32:35.152746+0000","last_active":"2024-04-17T07:32:36.176067+0000","last_peered":"2024-04-17T07:32:36.176067+0000","last_clean":"2024-04-17T07:32:36.176067+0000","last_became_active":"2024-04-17T07:32:35.151895+0000","last_became_peered":"2024-04-17T07:32:35.151895+0000","last_unstale":"2024-04-17T07:32:36.176067+0000","last_undegraded":"2024-04-17T07:32:36.176067+0000","last_fullsized":"2024-04-17T07:32:36.176067+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:07.429754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116050+0000","last_change":"2024-04-17T07:32:34.116820+0000","last_active":"2024-04-17T07:32:36.116050+0000","last_peered":"2024-04-17T07:32:36.116050+0000","last_clean":"2024-04-17T07:32:36.116050+0000","last_became_active":"2024-04-17T07:32:34.116317+0000","last_became_peered":"2024-04-17T07:32:34.116317+0000","last_unstale":"2024-04-17T07:32:36.116050+0000","last_undegraded":"2024-04-17T07:32:36.116050+0000","last_fullsized":"2024-04-17T07:32:36.116050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:38.565254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177034+0000","last_change":"2024-04-17T07:32:35.129508+0000","last_active":"2024-04-17T07:32:36.177034+0000","last_peered":"2024-04-17T07:32:36.177034+0000","last_clean":"2024-04-17T07:32:36.177034+0000","last_became_active":"2024-04-17T07:32:35.128961+0000","last_became_peered":"2024-04-17T07:32:35.128961+0000","last_unstale":"2024-04-17T07:32:36.177034+0000","last_undegraded":"2024-04-17T07:32:36.177034+0000","last_fullsized":"2024-04-17T07:32:36.177034+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:59.992707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175075+0000","last_change":"2024-04-17T07:32:34.103139+0000","last_active":"2024-04-17T07:32:36.175075+0000","last_peered":"2024-04-17T07:32:36.175075+0000","last_clean":"2024-04-17T07:32:36.175075+0000","last_became_active":"2024-04-17T07:32:34.102635+0000","last_became_peered":"2024-04-17T07:32:34.102635+0000","last_unstale":"2024-04-17T07:32:36.175075+0000","last_undegraded":"2024-04-17T07:32:36.175075+0000","last_fullsized":"2024-04-17T07:32:36.175075+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:01.668959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161932+0000","last_change":"2024-04-17T07:32:35.124617+0000","last_active":"2024-04-17T07:32:36.161932+0000","last_peered":"2024-04-17T07:32:36.161932+0000","last_clean":"2024-04-17T07:32:36.161932+0000","last_became_active":"2024-04-17T07:32:35.123058+0000","last_became_peered":"2024-04-17T07:32:35.123058+0000","last_unstale":"2024-04-17T07:32:36.161932+0000","last_undegraded":"2024-04-17T07:32:36.161932+0000","last_fullsized":"2024-04-17T07:32:36.161932+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:52.230466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176118+0000","last_change":"2024-04-17T07:32:34.107372+0000","last_active":"2024-04-17T07:32:36.176118+0000","last_peered":"2024-04-17T07:32:36.176118+0000","last_clean":"2024-04-17T07:32:36.176118+0000","last_became_active":"2024-04-17T07:32:34.107087+0000","last_became_peered":"2024-04-17T07:32:34.107087+0000","last_unstale":"2024-04-17T07:32:36.176118+0000","last_undegraded":"2024-04-17T07:32:36.176118+0000","last_fullsized":"2024-04-17T07:32:36.176118+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:29.102580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178148+0000","last_change":"2024-04-17T07:32:35.137895+0000","last_active":"2024-04-17T07:32:36.178148+0000","last_peered":"2024-04-17T07:32:36.178148+0000","last_clean":"2024-04-17T07:32:36.178148+0000","last_became_active":"2024-04-17T07:32:35.137601+0000","last_became_peered":"2024-04-17T07:32:35.137601+0000","last_unstale":"2024-04-17T07:32:36.178148+0000","last_undegraded":"2024-04-17T07:32:36.178148+0000","last_fullsized":"2024-04-17T07:32:36.178148+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.278212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176721+0000","last_change":"2024-04-17T07:32:34.118513+0000","last_active":"2024-04-17T07:32:36.176721+0000","last_peered":"2024-04-17T07:32:36.176721+0000","last_clean":"2024-04-17T07:32:36.176721+0000","last_became_active":"2024-04-17T07:32:34.118231+0000","last_became_peered":"2024-04-17T07:32:34.118231+0000","last_unstale":"2024-04-17T07:32:36.176721+0000","last_undegraded":"2024-04-17T07:32:36.176721+0000","last_fullsized":"2024-04-17T07:32:36.176721+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:38.711925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117406+0000","last_change":"2024-04-17T07:32:35.154386+0000","last_active":"2024-04-17T07:32:36.117406+0000","last_peered":"2024-04-17T07:32:36.117406+0000","last_clean":"2024-04-17T07:32:36.117406+0000","last_became_active":"2024-04-17T07:32:35.153650+0000","last_became_peered":"2024-04-17T07:32:35.153650+0000","last_unstale":"2024-04-17T07:32:36.117406+0000","last_undegraded":"2024-04-17T07:32:36.117406+0000","last_fullsized":"2024-04-17T07:32:36.117406+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:11:19.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118584+0000","last_change":"2024-04-17T07:32:34.115451+0000","last_active":"2024-04-17T07:32:36.118584+0000","last_peered":"2024-04-17T07:32:36.118584+0000","last_clean":"2024-04-17T07:32:36.118584+0000","last_became_active":"2024-04-17T07:32:34.115118+0000","last_became_peered":"2024-04-17T07:32:34.115118+0000","last_unstale":"2024-04-17T07:32:36.118584+0000","last_undegraded":"2024-04-17T07:32:36.118584+0000","last_fullsized":"2024-04-17T07:32:36.118584+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:25.659014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118200+0000","last_change":"2024-04-17T07:32:35.147902+0000","last_active":"2024-04-17T07:32:36.118200+0000","last_peered":"2024-04-17T07:32:36.118200+0000","last_clean":"2024-04-17T07:32:36.118200+0000","last_became_active":"2024-04-17T07:32:35.146971+0000","last_became_peered":"2024-04-17T07:32:35.146971+0000","last_unstale":"2024-04-17T07:32:36.118200+0000","last_undegraded":"2024-04-17T07:32:36.118200+0000","last_fullsized":"2024-04-17T07:32:36.118200+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:16:16.519615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178508+0000","last_change":"2024-04-17T07:32:34.122434+0000","last_active":"2024-04-17T07:32:36.178508+0000","last_peered":"2024-04-17T07:32:36.178508+0000","last_clean":"2024-04-17T07:32:36.178508+0000","last_became_active":"2024-04-17T07:32:34.121792+0000","last_became_peered":"2024-04-17T07:32:34.121792+0000","last_unstale":"2024-04-17T07:32:36.178508+0000","last_undegraded":"2024-04-17T07:32:36.178508+0000","last_fullsized":"2024-04-17T07:32:36.178508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:52.433800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176297+0000","last_change":"2024-04-17T07:32:35.150620+0000","last_active":"2024-04-17T07:32:36.176297+0000","last_peered":"2024-04-17T07:32:36.176297+0000","last_clean":"2024-04-17T07:32:36.176297+0000","last_became_active":"2024-04-17T07:32:35.150343+0000","last_became_peered":"2024-04-17T07:32:35.150343+0000","last_unstale":"2024-04-17T07:32:36.176297+0000","last_undegraded":"2024-04-17T07:32:36.176297+0000","last_fullsized":"2024-04-17T07:32:36.176297+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:54.548737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112099+0000","last_change":"2024-04-17T07:32:34.119211+0000","last_active":"2024-04-17T07:32:36.112099+0000","last_peered":"2024-04-17T07:32:36.112099+0000","last_clean":"2024-04-17T07:32:36.112099+0000","last_became_active":"2024-04-17T07:32:34.118856+0000","last_became_peered":"2024-04-17T07:32:34.118856+0000","last_unstale":"2024-04-17T07:32:36.112099+0000","last_undegraded":"2024-04-17T07:32:36.112099+0000","last_fullsized":"2024-04-17T07:32:36.112099+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:33:09.831283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175236+0000","last_change":"2024-04-17T07:32:35.153895+0000","last_active":"2024-04-17T07:32:36.175236+0000","last_peered":"2024-04-17T07:32:36.175236+0000","last_clean":"2024-04-17T07:32:36.175236+0000","last_became_active":"2024-04-17T07:32:35.151794+0000","last_became_peered":"2024-04-17T07:32:35.151794+0000","last_unstale":"2024-04-17T07:32:36.175236+0000","last_undegraded":"2024-04-17T07:32:36.175236+0000","last_fullsized":"2024-04-17T07:32:36.175236+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.978262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112039+0000","last_change":"2024-04-17T07:32:34.117055+0000","last_active":"2024-04-17T07:32:36.112039+0000","last_peered":"2024-04-17T07:32:36.112039+0000","last_clean":"2024-04-17T07:32:36.112039+0000","last_became_active":"2024-04-17T07:32:34.116609+0000","last_became_peered":"2024-04-17T07:32:34.116609+0000","last_unstale":"2024-04-17T07:32:36.112039+0000","last_undegraded":"2024-04-17T07:32:36.112039+0000","last_fullsized":"2024-04-17T07:32:36.112039+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:15:57.845439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177348+0000","last_change":"2024-04-17T07:32:35.127746+0000","last_active":"2024-04-17T07:32:36.177348+0000","last_peered":"2024-04-17T07:32:36.177348+0000","last_clean":"2024-04-17T07:32:36.177348+0000","last_became_active":"2024-04-17T07:32:35.127026+0000","last_became_peered":"2024-04-17T07:32:35.127026+0000","last_unstale":"2024-04-17T07:32:36.177348+0000","last_undegraded":"2024-04-17T07:32:36.177348+0000","last_fullsized":"2024-04-17T07:32:36.177348+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:42:14.360457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177012+0000","last_change":"2024-04-17T07:32:34.118865+0000","last_active":"2024-04-17T07:32:36.177012+0000","last_peered":"2024-04-17T07:32:36.177012+0000","last_clean":"2024-04-17T07:32:36.177012+0000","last_became_active":"2024-04-17T07:32:34.118465+0000","last_became_peered":"2024-04-17T07:32:34.118465+0000","last_unstale":"2024-04-17T07:32:36.177012+0000","last_undegraded":"2024-04-17T07:32:36.177012+0000","last_fullsized":"2024-04-17T07:32:36.177012+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:01.394709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111659+0000","last_change":"2024-04-17T07:32:35.144776+0000","last_active":"2024-04-17T07:32:36.111659+0000","last_peered":"2024-04-17T07:32:36.111659+0000","last_clean":"2024-04-17T07:32:36.111659+0000","last_became_active":"2024-04-17T07:32:35.144482+0000","last_became_peered":"2024-04-17T07:32:35.144482+0000","last_unstale":"2024-04-17T07:32:36.111659+0000","last_undegraded":"2024-04-17T07:32:36.111659+0000","last_fullsized":"2024-04-17T07:32:36.111659+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:58.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175197+0000","last_change":"2024-04-17T07:32:34.110438+0000","last_active":"2024-04-17T07:32:36.175197+0000","last_peered":"2024-04-17T07:32:36.175197+0000","last_clean":"2024-04-17T07:32:36.175197+0000","last_became_active":"2024-04-17T07:32:34.109449+0000","last_became_peered":"2024-04-17T07:32:34.109449+0000","last_unstale":"2024-04-17T07:32:36.175197+0000","last_undegraded":"2024-04-17T07:32:36.175197+0000","last_fullsized":"2024-04-17T07:32:36.175197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:49.182032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117710+0000","last_change":"2024-04-17T07:32:35.148505+0000","last_active":"2024-04-17T07:32:36.117710+0000","last_peered":"2024-04-17T07:32:36.117710+0000","last_clean":"2024-04-17T07:32:36.117710+0000","last_became_active":"2024-04-17T07:32:35.148196+0000","last_became_peered":"2024-04-17T07:32:35.148196+0000","last_unstale":"2024-04-17T07:32:36.117710+0000","last_undegraded":"2024-04-17T07:32:36.117710+0000","last_fullsized":"2024-04-17T07:32:36.117710+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:34.370047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177861+0000","last_change":"2024-04-17T07:32:34.119465+0000","last_active":"2024-04-17T07:32:36.177861+0000","last_peered":"2024-04-17T07:32:36.177861+0000","last_clean":"2024-04-17T07:32:36.177861+0000","last_became_active":"2024-04-17T07:32:34.119062+0000","last_became_peered":"2024-04-17T07:32:34.119062+0000","last_unstale":"2024-04-17T07:32:36.177861+0000","last_undegraded":"2024-04-17T07:32:36.177861+0000","last_fullsized":"2024-04-17T07:32:36.177861+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:45.726103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179192+0000","last_change":"2024-04-17T07:32:35.142092+0000","last_active":"2024-04-17T07:32:36.179192+0000","last_peered":"2024-04-17T07:32:36.179192+0000","last_clean":"2024-04-17T07:32:36.179192+0000","last_became_active":"2024-04-17T07:32:35.141566+0000","last_became_peered":"2024-04-17T07:32:35.141566+0000","last_unstale":"2024-04-17T07:32:36.179192+0000","last_undegraded":"2024-04-17T07:32:36.179192+0000","last_fullsized":"2024-04-17T07:32:36.179192+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:41.490644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117581+0000","last_change":"2024-04-17T07:32:34.119203+0000","last_active":"2024-04-17T07:32:36.117581+0000","last_peered":"2024-04-17T07:32:36.117581+0000","last_clean":"2024-04-17T07:32:36.117581+0000","last_became_active":"2024-04-17T07:32:34.118889+0000","last_became_peered":"2024-04-17T07:32:34.118889+0000","last_unstale":"2024-04-17T07:32:36.117581+0000","last_undegraded":"2024-04-17T07:32:36.117581+0000","last_fullsized":"2024-04-17T07:32:36.117581+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:28.569706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118534+0000","last_change":"2024-04-17T07:32:35.144152+0000","last_active":"2024-04-17T07:32:36.118534+0000","last_peered":"2024-04-17T07:32:36.118534+0000","last_clean":"2024-04-17T07:32:36.118534+0000","last_became_active":"2024-04-17T07:32:35.143826+0000","last_became_peered":"2024-04-17T07:32:35.143826+0000","last_unstale":"2024-04-17T07:32:36.118534+0000","last_undegraded":"2024-04-17T07:32:36.118534+0000","last_fullsized":"2024-04-17T07:32:36.118534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:09.754340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161876+0000","last_change":"2024-04-17T07:32:34.115795+0000","last_active":"2024-04-17T07:32:36.161876+0000","last_peered":"2024-04-17T07:32:36.161876+0000","last_clean":"2024-04-17T07:32:36.161876+0000","last_became_active":"2024-04-17T07:32:34.115495+0000","last_became_peered":"2024-04-17T07:32:34.115495+0000","last_unstale":"2024-04-17T07:32:36.161876+0000","last_undegraded":"2024-04-17T07:32:36.161876+0000","last_fullsized":"2024-04-17T07:32:36.161876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:44.817097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112004+0000","last_change":"2024-04-17T07:32:35.152975+0000","last_active":"2024-04-17T07:32:36.112004+0000","last_peered":"2024-04-17T07:32:36.112004+0000","last_clean":"2024-04-17T07:32:36.112004+0000","last_became_active":"2024-04-17T07:32:35.152639+0000","last_became_peered":"2024-04-17T07:32:35.152639+0000","last_unstale":"2024-04-17T07:32:36.112004+0000","last_undegraded":"2024-04-17T07:32:36.112004+0000","last_fullsized":"2024-04-17T07:32:36.112004+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:42.180598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161718+0000","last_change":"2024-04-17T07:32:34.114014+0000","last_active":"2024-04-17T07:32:36.161718+0000","last_peered":"2024-04-17T07:32:36.161718+0000","last_clean":"2024-04-17T07:32:36.161718+0000","last_became_active":"2024-04-17T07:32:34.113722+0000","last_became_peered":"2024-04-17T07:32:34.113722+0000","last_unstale":"2024-04-17T07:32:36.161718+0000","last_undegraded":"2024-04-17T07:32:36.161718+0000","last_fullsized":"2024-04-17T07:32:36.161718+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:11.145740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111996+0000","last_change":"2024-04-17T07:32:35.161559+0000","last_active":"2024-04-17T07:32:36.111996+0000","last_peered":"2024-04-17T07:32:36.111996+0000","last_clean":"2024-04-17T07:32:36.111996+0000","last_became_active":"2024-04-17T07:32:35.161222+0000","last_became_peered":"2024-04-17T07:32:35.161222+0000","last_unstale":"2024-04-17T07:32:36.111996+0000","last_undegraded":"2024-04-17T07:32:36.111996+0000","last_fullsized":"2024-04-17T07:32:36.111996+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:52.866924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178990+0000","last_change":"2024-04-17T07:32:34.106442+0000","last_active":"2024-04-17T07:32:36.178990+0000","last_peered":"2024-04-17T07:32:36.178990+0000","last_clean":"2024-04-17T07:32:36.178990+0000","last_became_active":"2024-04-17T07:32:34.105981+0000","last_became_peered":"2024-04-17T07:32:34.105981+0000","last_unstale":"2024-04-17T07:32:36.178990+0000","last_undegraded":"2024-04-17T07:32:36.178990+0000","last_fullsized":"2024-04-17T07:32:36.178990+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:21.122385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111687+0000","last_change":"2024-04-17T07:32:35.157407+0000","last_active":"2024-04-17T07:32:36.111687+0000","last_peered":"2024-04-17T07:32:36.111687+0000","last_clean":"2024-04-17T07:32:36.111687+0000","last_became_active":"2024-04-17T07:32:35.156513+0000","last_became_peered":"2024-04-17T07:32:35.156513+0000","last_unstale":"2024-04-17T07:32:36.111687+0000","last_undegraded":"2024-04-17T07:32:36.111687+0000","last_fullsized":"2024-04-17T07:32:36.111687+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:08.175259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178935+0000","last_change":"2024-04-17T07:32:34.109456+0000","last_active":"2024-04-17T07:32:36.178935+0000","last_peered":"2024-04-17T07:32:36.178935+0000","last_clean":"2024-04-17T07:32:36.178935+0000","last_became_active":"2024-04-17T07:32:34.106867+0000","last_became_peered":"2024-04-17T07:32:34.106867+0000","last_unstale":"2024-04-17T07:32:36.178935+0000","last_undegraded":"2024-04-17T07:32:36.178935+0000","last_fullsized":"2024-04-17T07:32:36.178935+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:39.889609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.115788+0000","last_change":"2024-04-17T07:32:35.160126+0000","last_active":"2024-04-17T07:32:36.115788+0000","last_peered":"2024-04-17T07:32:36.115788+0000","last_clean":"2024-04-17T07:32:36.115788+0000","last_became_active":"2024-04-17T07:32:35.159786+0000","last_became_peered":"2024-04-17T07:32:35.159786+0000","last_unstale":"2024-04-17T07:32:36.115788+0000","last_undegraded":"2024-04-17T07:32:36.115788+0000","last_fullsized":"2024-04-17T07:32:36.115788+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:23.812342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161438+0000","last_change":"2024-04-17T07:32:34.104872+0000","last_active":"2024-04-17T07:32:36.161438+0000","last_peered":"2024-04-17T07:32:36.161438+0000","last_clean":"2024-04-17T07:32:36.161438+0000","last_became_active":"2024-04-17T07:32:34.104576+0000","last_became_peered":"2024-04-17T07:32:34.104576+0000","last_unstale":"2024-04-17T07:32:36.161438+0000","last_undegraded":"2024-04-17T07:32:36.161438+0000","last_fullsized":"2024-04-17T07:32:36.161438+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:33:09.831283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162067+0000","last_change":"2024-04-17T07:32:35.129628+0000","last_active":"2024-04-17T07:32:36.162067+0000","last_peered":"2024-04-17T07:32:36.162067+0000","last_clean":"2024-04-17T07:32:36.162067+0000","last_became_active":"2024-04-17T07:32:35.128422+0000","last_became_peered":"2024-04-17T07:32:35.128422+0000","last_unstale":"2024-04-17T07:32:36.162067+0000","last_undegraded":"2024-04-17T07:32:36.162067+0000","last_fullsized":"2024-04-17T07:32:36.162067+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.535287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179721+0000","last_change":"2024-04-17T07:32:34.115849+0000","last_active":"2024-04-17T07:32:36.179721+0000","last_peered":"2024-04-17T07:32:36.179721+0000","last_clean":"2024-04-17T07:32:36.179721+0000","last_became_active":"2024-04-17T07:32:34.115554+0000","last_became_peered":"2024-04-17T07:32:34.115554+0000","last_unstale":"2024-04-17T07:32:36.179721+0000","last_undegraded":"2024-04-17T07:32:36.179721+0000","last_fullsized":"2024-04-17T07:32:36.179721+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:24.993547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176019+0000","last_change":"2024-04-17T07:32:35.151950+0000","last_active":"2024-04-17T07:32:36.176019+0000","last_peered":"2024-04-17T07:32:36.176019+0000","last_clean":"2024-04-17T07:32:36.176019+0000","last_became_active":"2024-04-17T07:32:35.145334+0000","last_became_peered":"2024-04-17T07:32:35.145334+0000","last_unstale":"2024-04-17T07:32:36.176019+0000","last_undegraded":"2024-04-17T07:32:36.176019+0000","last_fullsized":"2024-04-17T07:32:36.176019+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:47.255018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.163881+0000","last_change":"2024-04-17T07:32:34.114148+0000","last_active":"2024-04-17T07:32:36.163881+0000","last_peered":"2024-04-17T07:32:36.163881+0000","last_clean":"2024-04-17T07:32:36.163881+0000","last_became_active":"2024-04-17T07:32:34.113529+0000","last_became_peered":"2024-04-17T07:32:34.113529+0000","last_unstale":"2024-04-17T07:32:36.163881+0000","last_undegraded":"2024-04-17T07:32:36.163881+0000","last_fullsized":"2024-04-17T07:32:36.163881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:45.667890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175610+0000","last_change":"2024-04-17T07:32:35.152780+0000","last_active":"2024-04-17T07:32:36.175610+0000","last_peered":"2024-04-17T07:32:36.175610+0000","last_clean":"2024-04-17T07:32:36.175610+0000","last_became_active":"2024-04-17T07:32:35.151186+0000","last_became_peered":"2024-04-17T07:32:35.151186+0000","last_unstale":"2024-04-17T07:32:36.175610+0000","last_undegraded":"2024-04-17T07:32:36.175610+0000","last_fullsized":"2024-04-17T07:32:36.175610+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:28:33.012641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162088+0000","last_change":"2024-04-17T07:32:34.114679+0000","last_active":"2024-04-17T07:32:36.162088+0000","last_peered":"2024-04-17T07:32:36.162088+0000","last_clean":"2024-04-17T07:32:36.162088+0000","last_became_active":"2024-04-17T07:32:34.114406+0000","last_became_peered":"2024-04-17T07:32:34.114406+0000","last_unstale":"2024-04-17T07:32:36.162088+0000","last_undegraded":"2024-04-17T07:32:36.162088+0000","last_fullsized":"2024-04-17T07:32:36.162088+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:37.930564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117322+0000","last_change":"2024-04-17T07:32:35.156586+0000","last_active":"2024-04-17T07:32:36.117322+0000","last_peered":"2024-04-17T07:32:36.117322+0000","last_clean":"2024-04-17T07:32:36.117322+0000","last_became_active":"2024-04-17T07:32:35.155514+0000","last_became_peered":"2024-04-17T07:32:35.155514+0000","last_unstale":"2024-04-17T07:32:36.117322+0000","last_undegraded":"2024-04-17T07:32:36.117322+0000","last_fullsized":"2024-04-17T07:32:36.117322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.978262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117692+0000","last_change":"2024-04-17T07:32:34.123668+0000","last_active":"2024-04-17T07:32:36.117692+0000","last_peered":"2024-04-17T07:32:36.117692+0000","last_clean":"2024-04-17T07:32:36.117692+0000","last_became_active":"2024-04-17T07:32:34.123317+0000","last_became_peered":"2024-04-17T07:32:34.123317+0000","last_unstale":"2024-04-17T07:32:36.117692+0000","last_undegraded":"2024-04-17T07:32:36.117692+0000","last_fullsized":"2024-04-17T07:32:36.117692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:16:35.188911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111950+0000","last_change":"2024-04-17T07:32:35.148068+0000","last_active":"2024-04-17T07:32:36.111950+0000","last_peered":"2024-04-17T07:32:36.111950+0000","last_clean":"2024-04-17T07:32:36.111950+0000","last_became_active":"2024-04-17T07:32:35.147557+0000","last_became_peered":"2024-04-17T07:32:35.147557+0000","last_unstale":"2024-04-17T07:32:36.111950+0000","last_undegraded":"2024-04-17T07:32:36.111950+0000","last_fullsized":"2024-04-17T07:32:36.111950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:27.796262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161789+0000","last_change":"2024-04-17T07:32:34.106683+0000","last_active":"2024-04-17T07:32:36.161789+0000","last_peered":"2024-04-17T07:32:36.161789+0000","last_clean":"2024-04-17T07:32:36.161789+0000","last_became_active":"2024-04-17T07:32:34.106255+0000","last_became_peered":"2024-04-17T07:32:34.106255+0000","last_unstale":"2024-04-17T07:32:36.161789+0000","last_undegraded":"2024-04-17T07:32:36.161789+0000","last_fullsized":"2024-04-17T07:32:36.161789+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:28.482375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177522+0000","last_change":"2024-04-17T07:32:35.146392+0000","last_active":"2024-04-17T07:32:36.177522+0000","last_peered":"2024-04-17T07:32:36.177522+0000","last_clean":"2024-04-17T07:32:36.177522+0000","last_became_active":"2024-04-17T07:32:35.146014+0000","last_became_peered":"2024-04-17T07:32:35.146014+0000","last_unstale":"2024-04-17T07:32:36.177522+0000","last_undegraded":"2024-04-17T07:32:36.177522+0000","last_fullsized":"2024-04-17T07:32:36.177522+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:33.587267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.115940+0000","last_change":"2024-04-17T07:32:34.117019+0000","last_active":"2024-04-17T07:32:36.115940+0000","last_peered":"2024-04-17T07:32:36.115940+0000","last_clean":"2024-04-17T07:32:36.115940+0000","last_became_active":"2024-04-17T07:32:34.116493+0000","last_became_peered":"2024-04-17T07:32:34.116493+0000","last_unstale":"2024-04-17T07:32:36.115940+0000","last_undegraded":"2024-04-17T07:32:36.115940+0000","last_fullsized":"2024-04-17T07:32:36.115940+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:11.827705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111646+0000","last_change":"2024-04-17T07:32:35.149351+0000","last_active":"2024-04-17T07:32:36.111646+0000","last_peered":"2024-04-17T07:32:36.111646+0000","last_clean":"2024-04-17T07:32:36.111646+0000","last_became_active":"2024-04-17T07:32:35.148610+0000","last_became_peered":"2024-04-17T07:32:35.148610+0000","last_unstale":"2024-04-17T07:32:36.111646+0000","last_undegraded":"2024-04-17T07:32:36.111646+0000","last_fullsized":"2024-04-17T07:32:36.111646+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:22.941744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176328+0000","last_change":"2024-04-17T07:32:34.114975+0000","last_active":"2024-04-17T07:32:36.176328+0000","last_peered":"2024-04-17T07:32:36.176328+0000","last_clean":"2024-04-17T07:32:36.176328+0000","last_became_active":"2024-04-17T07:32:34.114695+0000","last_became_peered":"2024-04-17T07:32:34.114695+0000","last_unstale":"2024-04-17T07:32:36.176328+0000","last_undegraded":"2024-04-17T07:32:36.176328+0000","last_fullsized":"2024-04-17T07:32:36.176328+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:23.375185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117971+0000","last_change":"2024-04-17T07:32:35.155946+0000","last_active":"2024-04-17T07:32:36.117971+0000","last_peered":"2024-04-17T07:32:36.117971+0000","last_clean":"2024-04-17T07:32:36.117971+0000","last_became_active":"2024-04-17T07:32:35.155625+0000","last_became_peered":"2024-04-17T07:32:35.155625+0000","last_unstale":"2024-04-17T07:32:36.117971+0000","last_undegraded":"2024-04-17T07:32:36.117971+0000","last_fullsized":"2024-04-17T07:32:36.117971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:18.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111815+0000","last_change":"2024-04-17T07:32:34.122836+0000","last_active":"2024-04-17T07:32:36.111815+0000","last_peered":"2024-04-17T07:32:36.111815+0000","last_clean":"2024-04-17T07:32:36.111815+0000","last_became_active":"2024-04-17T07:32:34.120873+0000","last_became_peered":"2024-04-17T07:32:34.120873+0000","last_unstale":"2024-04-17T07:32:36.111815+0000","last_undegraded":"2024-04-17T07:32:36.111815+0000","last_fullsized":"2024-04-17T07:32:36.111815+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:48.287591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112693+0000","last_change":"2024-04-17T07:32:35.160004+0000","last_active":"2024-04-17T07:32:36.112693+0000","last_peered":"2024-04-17T07:32:36.112693+0000","last_clean":"2024-04-17T07:32:36.112693+0000","last_became_active":"2024-04-17T07:32:35.158591+0000","last_became_peered":"2024-04-17T07:32:35.158591+0000","last_unstale":"2024-04-17T07:32:36.112693+0000","last_undegraded":"2024-04-17T07:32:36.112693+0000","last_fullsized":"2024-04-17T07:32:36.112693+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:52.230466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118125+0000","last_change":"2024-04-17T07:32:34.122654+0000","last_active":"2024-04-17T07:32:36.118125+0000","last_peered":"2024-04-17T07:32:36.118125+0000","last_clean":"2024-04-17T07:32:36.118125+0000","last_became_active":"2024-04-17T07:32:34.122324+0000","last_became_peered":"2024-04-17T07:32:34.122324+0000","last_unstale":"2024-04-17T07:32:36.118125+0000","last_undegraded":"2024-04-17T07:32:36.118125+0000","last_fullsized":"2024-04-17T07:32:36.118125+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:06.424202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"115'211","reported_seq":709,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117322+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:32:36.117322+0000","last_peered":"2024-04-17T07:32:36.117322+0000","last_clean":"2024-04-17T07:32:36.117322+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:32:36.117322+0000","last_undegraded":"2024-04-17T07:32:36.117322+0000","last_fullsized":"2024-04-17T07:32:36.117322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":327,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":327,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542367,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37444,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334396,"statfs":{"total":96636764160,"available":96598421504,"internally_reserved":0,"allocated":2154496,"data_stored":1333225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542367,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":749697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542366,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1568768,"data_stored":749115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542366,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59096,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312744,"statfs":{"total":96636764160,"available":96576249856,"internally_reserved":0,"allocated":1585152,"data_stored":750781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542368,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1581056,"data_stored":750679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542367,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41036,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330804,"statfs":{"total":96636764160,"available":96594743296,"internally_reserved":0,"allocated":1568768,"data_stored":749079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542368,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":1564672,"data_stored":749045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542366,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37456,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334384,"statfs":{"total":96636764160,"available":96598409216,"internally_reserved":0,"allocated":2162688,"data_stored":1339995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:32:39.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:32:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.900+0000 7ff338800640 1 Processor -- start 2024-04-17T07:32:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.901+0000 7ff338800640 1 -- start start 2024-04-17T07:32:39.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.902+0000 7ff338800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33009aa60 0x7ff33009aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.902+0000 7ff338800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3300ceea0 con 0x7ff330160210 2024-04-17T07:32:39.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.902+0000 7ff338800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3300b21c0 con 0x7ff3300b2f80 2024-04-17T07:32:39.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.902+0000 7ff338800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff3300b2af0 con 0x7ff33009aa60 2024-04-17T07:32:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff335e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff330160210 0x7ff3300cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46306/0 (socket says 172.21.15.67:46306) 2024-04-17T07:32:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff335e00640 1 -- 172.21.15.67:0/3358144179 learned_addr learned my addr 172.21.15.67:0/3358144179 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff335400640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33009aa60 0x7ff33009aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815407248 0 0) 0x7ff3300ceea0 con 0x7ff330160210 2024-04-17T07:32:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff31c003520 con 0x7ff330160210 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3079115512 0 0) 0x7ff31c003520 con 0x7ff330160210 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33009aa60 msgr2=0x7ff33009aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.903+0000 7ff334a00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33009aa60 0x7ff33009aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 >> v1:172.21.15.187:6789/0 conn(0x7ff3300b2f80 legacy=0x7ff3300b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3300b24a0 con 0x7ff330160210 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff3280019f0 con 0x7ff330160210 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 >> v1:172.21.15.67:6789/0 conn(0x7ff330160210 legacy=0x7ff3300cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff31c0039e0 0x7ff31c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c004800 0x7ff31c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff31c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3300b2af0 con 0x7ff31c0051c0 2024-04-17T07:32:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c00c070 con 0x7ff31c004800 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff31c00c270 con 0x7ff31c0039e0 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff336800640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff31c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff335400640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c004800 0x7ff31c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff335e00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff31c0039e0 0x7ff31c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.904+0000 7ff336800640 1 -- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff31c0039e0 msgr2=0x7ff31c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.905+0000 7ff336800640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff31c0039e0 0x7ff31c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.905+0000 7ff336800640 1 -- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c004800 msgr2=0x7ff31c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.905+0000 7ff336800640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c004800 0x7ff31c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.905+0000 7ff336800640 1 -- 172.21.15.67:0/3358144179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3300b24a0 con 0x7ff31c0051c0 2024-04-17T07:32:39.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.905+0000 7ff335e00640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff31c0039e0 0x7ff31c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:39.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.906+0000 7ff336800640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff31c00b9c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ff32c001750 tx=0x7ff31c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2862fe37d4b48f44 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.906+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c001370 con 0x7ff31c0051c0 2024-04-17T07:32:39.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.906+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32c0052d0 con 0x7ff31c0051c0 2024-04-17T07:32:39.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.906+0000 7ff334a00640 1 -- 172.21.15.67:0/3358144179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c003830 con 0x7ff31c0051c0 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 -- 172.21.15.67:0/3358144179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 msgr2=0x7ff31c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 --2- 172.21.15.67:0/3358144179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff31c00b9c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7ff32c001750 tx=0x7ff31c005830 comp rx=0 tx=0).stop 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff336800640 1 -- 172.21.15.67:0/3358144179 reap_dead start 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 -- 172.21.15.67:0/3358144179 shutdown_connections 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 -- 172.21.15.67:0/3358144179 >> 172.21.15.67:0/3358144179 conn(0x7ff3300b41b0 msgr2=0x7ff3300b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 -- 172.21.15.67:0/3358144179 shutdown_connections 2024-04-17T07:32:39.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.907+0000 7ff338800640 1 -- 172.21.15.67:0/3358144179 wait complete. 2024-04-17T07:32:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.909+0000 7ff338800640 1 Processor -- start 2024-04-17T07:32:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.909+0000 7ff338800640 1 -- start start 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33015b950 0x7ff33015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3300cb940 0x7ff3300cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3300af4d0 con 0x7ff31c0051c0 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff3300af350 con 0x7ff3300cb940 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff338800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff3300ce4b0 con 0x7ff33015b950 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff335e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff335e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32798/0 (socket says 172.21.15.67:32798) 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.910+0000 7ff335e00640 1 -- 172.21.15.67:0/1802725972 learned_addr learned my addr 172.21.15.67:0/1802725972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 -- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33015b950 msgr2=0x7ff33015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33015b950 0x7ff33015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 -- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3300cb940 msgr2=0x7ff3300cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:39.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff336800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3300cb940 0x7ff3300cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3300cb940 0x7ff3300cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:39.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 -- 172.21.15.67:0/1802725972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3300ca4c0 con 0x7ff31c0051c0 2024-04-17T07:32:39.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff335e00640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b100 tx=0x7ff32800b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328019070 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff328004d60 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3280072d0 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301f46e0 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.911+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301f4ca0 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.912+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff330068b90 con 0x7ff31c0051c0 2024-04-17T07:32:39.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.914+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7ff32801c070 con 0x7ff31c0051c0 2024-04-17T07:32:39.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.914+0000 7ff326000640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 0x7ff31406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:39.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.915+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7ff32808c190 con 0x7ff31c0051c0 2024-04-17T07:32:39.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.915+0000 7ff335400640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 0x7ff31406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:39.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.916+0000 7ff335400640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 0x7ff31406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff320006a90 tx=0x7ff320006a20 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:39.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:39.919+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff32805a390 con 0x7ff31c0051c0 2024-04-17T07:32:40.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.119+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 --> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff33015d1f0 con 0x7ff31406ca10 2024-04-17T07:32:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.139+0000 7ff326000640 1 -- 172.21.15.67:0/1802725972 <== mgr.6394 v2:172.21.15.187:6832/2517863062 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252489 (secure 0 0 0) 0x7ff33015d1f0 con 0x7ff31406ca10 2024-04-17T07:32:40.141 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 msgr2=0x7ff31406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 0x7ff31406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff320006a90 tx=0x7ff320006a20 comp rx=0 tx=0).stop 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 msgr2=0x7ff3300c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b100 tx=0x7ff32800b5c0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 shutdown_connections 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff31406ca10 0x7ff31406eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3300cb940 0x7ff3300cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff33015b950 0x7ff33015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 --2- 172.21.15.67:0/1802725972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31c0051c0 0x7ff3300c8c40 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 >> 172.21.15.67:0/1802725972 conn(0x7ff3300b41b0 msgr2=0x7ff3300cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 shutdown_connections 2024-04-17T07:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.146+0000 7ff338800640 1 -- 172.21.15.67:0/1802725972 wait complete. 2024-04-17T07:32:40.166 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-17T07:32:39.934264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":372,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336444,"kb_used_data":13436,"kb_used_omap":12,"kb_used_meta":322867,"kb_avail":754638276,"statfs":{"total":773094113280,"available":772749594624,"internally_reserved":0,"allocated":13758464,"data_stored":7171616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":330616368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002212"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175765+0000","last_change":"2024-04-17T07:32:35.141914+0000","last_active":"2024-04-17T07:32:36.175765+0000","last_peered":"2024-04-17T07:32:36.175765+0000","last_clean":"2024-04-17T07:32:36.175765+0000","last_became_active":"2024-04-17T07:32:35.140482+0000","last_became_peered":"2024-04-17T07:32:35.140482+0000","last_unstale":"2024-04-17T07:32:36.175765+0000","last_undegraded":"2024-04-17T07:32:36.175765+0000","last_fullsized":"2024-04-17T07:32:36.175765+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:18.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161746+0000","last_change":"2024-04-17T07:32:35.130073+0000","last_active":"2024-04-17T07:32:36.161746+0000","last_peered":"2024-04-17T07:32:36.161746+0000","last_clean":"2024-04-17T07:32:36.161746+0000","last_became_active":"2024-04-17T07:32:35.129360+0000","last_became_peered":"2024-04-17T07:32:35.129360+0000","last_unstale":"2024-04-17T07:32:36.161746+0000","last_undegraded":"2024-04-17T07:32:36.161746+0000","last_fullsized":"2024-04-17T07:32:36.161746+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:43.978087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175673+0000","last_change":"2024-04-17T07:32:35.143728+0000","last_active":"2024-04-17T07:32:36.175673+0000","last_peered":"2024-04-17T07:32:36.175673+0000","last_clean":"2024-04-17T07:32:36.175673+0000","last_became_active":"2024-04-17T07:32:35.142923+0000","last_became_peered":"2024-04-17T07:32:35.142923+0000","last_unstale":"2024-04-17T07:32:36.175673+0000","last_undegraded":"2024-04-17T07:32:36.175673+0000","last_fullsized":"2024-04-17T07:32:36.175673+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:11:19.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176517+0000","last_change":"2024-04-17T07:32:35.142131+0000","last_active":"2024-04-17T07:32:36.176517+0000","last_peered":"2024-04-17T07:32:36.176517+0000","last_clean":"2024-04-17T07:32:36.176517+0000","last_became_active":"2024-04-17T07:32:35.141836+0000","last_became_peered":"2024-04-17T07:32:35.141836+0000","last_unstale":"2024-04-17T07:32:36.176517+0000","last_undegraded":"2024-04-17T07:32:36.176517+0000","last_fullsized":"2024-04-17T07:32:36.176517+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:35:08.418431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116602+0000","last_change":"2024-04-17T07:32:35.155566+0000","last_active":"2024-04-17T07:32:36.116602+0000","last_peered":"2024-04-17T07:32:36.116602+0000","last_clean":"2024-04-17T07:32:36.116602+0000","last_became_active":"2024-04-17T07:32:35.155238+0000","last_became_peered":"2024-04-17T07:32:35.155238+0000","last_unstale":"2024-04-17T07:32:36.116602+0000","last_undegraded":"2024-04-17T07:32:36.116602+0000","last_fullsized":"2024-04-17T07:32:36.116602+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:12.206597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117781+0000","last_change":"2024-04-17T07:32:35.144948+0000","last_active":"2024-04-17T07:32:36.117781+0000","last_peered":"2024-04-17T07:32:36.117781+0000","last_clean":"2024-04-17T07:32:36.117781+0000","last_became_active":"2024-04-17T07:32:35.143203+0000","last_became_peered":"2024-04-17T07:32:35.143203+0000","last_unstale":"2024-04-17T07:32:36.117781+0000","last_undegraded":"2024-04-17T07:32:36.117781+0000","last_fullsized":"2024-04-17T07:32:36.117781+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:08:39.614569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116601+0000","last_change":"2024-04-17T07:32:35.144278+0000","last_active":"2024-04-17T07:32:36.116601+0000","last_peered":"2024-04-17T07:32:36.116601+0000","last_clean":"2024-04-17T07:32:36.116601+0000","last_became_active":"2024-04-17T07:32:35.143964+0000","last_became_peered":"2024-04-17T07:32:35.143964+0000","last_unstale":"2024-04-17T07:32:36.116601+0000","last_undegraded":"2024-04-17T07:32:36.116601+0000","last_fullsized":"2024-04-17T07:32:36.116601+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:46.435011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161312+0000","last_change":"2024-04-17T07:32:35.147820+0000","last_active":"2024-04-17T07:32:36.161312+0000","last_peered":"2024-04-17T07:32:36.161312+0000","last_clean":"2024-04-17T07:32:36.161312+0000","last_became_active":"2024-04-17T07:32:35.147552+0000","last_became_peered":"2024-04-17T07:32:35.147552+0000","last_unstale":"2024-04-17T07:32:36.161312+0000","last_undegraded":"2024-04-17T07:32:36.161312+0000","last_fullsized":"2024-04-17T07:32:36.161312+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:27.796262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178075+0000","last_change":"2024-04-17T07:32:35.141551+0000","last_active":"2024-04-17T07:32:36.178075+0000","last_peered":"2024-04-17T07:32:36.178075+0000","last_clean":"2024-04-17T07:32:36.178075+0000","last_became_active":"2024-04-17T07:32:35.140192+0000","last_became_peered":"2024-04-17T07:32:35.140192+0000","last_unstale":"2024-04-17T07:32:36.178075+0000","last_undegraded":"2024-04-17T07:32:36.178075+0000","last_fullsized":"2024-04-17T07:32:36.178075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:15.720120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116015+0000","last_change":"2024-04-17T07:32:35.160179+0000","last_active":"2024-04-17T07:32:36.116015+0000","last_peered":"2024-04-17T07:32:36.116015+0000","last_clean":"2024-04-17T07:32:36.116015+0000","last_became_active":"2024-04-17T07:32:35.159858+0000","last_became_peered":"2024-04-17T07:32:35.159858+0000","last_unstale":"2024-04-17T07:32:36.116015+0000","last_undegraded":"2024-04-17T07:32:36.116015+0000","last_fullsized":"2024-04-17T07:32:36.116015+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:04.303000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117667+0000","last_change":"2024-04-17T07:32:35.148431+0000","last_active":"2024-04-17T07:32:36.117667+0000","last_peered":"2024-04-17T07:32:36.117667+0000","last_clean":"2024-04-17T07:32:36.117667+0000","last_became_active":"2024-04-17T07:32:35.148110+0000","last_became_peered":"2024-04-17T07:32:35.148110+0000","last_unstale":"2024-04-17T07:32:36.117667+0000","last_undegraded":"2024-04-17T07:32:36.117667+0000","last_fullsized":"2024-04-17T07:32:36.117667+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:00:16.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177779+0000","last_change":"2024-04-17T07:32:35.139822+0000","last_active":"2024-04-17T07:32:36.177779+0000","last_peered":"2024-04-17T07:32:36.177779+0000","last_clean":"2024-04-17T07:32:36.177779+0000","last_became_active":"2024-04-17T07:32:35.139437+0000","last_became_peered":"2024-04-17T07:32:35.139437+0000","last_unstale":"2024-04-17T07:32:36.177779+0000","last_undegraded":"2024-04-17T07:32:36.177779+0000","last_fullsized":"2024-04-17T07:32:36.177779+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:11.257154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176304+0000","last_change":"2024-04-17T07:32:35.140029+0000","last_active":"2024-04-17T07:32:36.176304+0000","last_peered":"2024-04-17T07:32:36.176304+0000","last_clean":"2024-04-17T07:32:36.176304+0000","last_became_active":"2024-04-17T07:32:35.139056+0000","last_became_peered":"2024-04-17T07:32:35.139056+0000","last_unstale":"2024-04-17T07:32:36.176304+0000","last_undegraded":"2024-04-17T07:32:36.176304+0000","last_fullsized":"2024-04-17T07:32:36.176304+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:19.263171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111736+0000","last_change":"2024-04-17T07:32:35.147837+0000","last_active":"2024-04-17T07:32:36.111736+0000","last_peered":"2024-04-17T07:32:36.111736+0000","last_clean":"2024-04-17T07:32:36.111736+0000","last_became_active":"2024-04-17T07:32:35.147071+0000","last_became_peered":"2024-04-17T07:32:35.147071+0000","last_unstale":"2024-04-17T07:32:36.111736+0000","last_undegraded":"2024-04-17T07:32:36.111736+0000","last_fullsized":"2024-04-17T07:32:36.111736+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:02.543243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176209+0000","last_change":"2024-04-17T07:32:35.141660+0000","last_active":"2024-04-17T07:32:36.176209+0000","last_peered":"2024-04-17T07:32:36.176209+0000","last_clean":"2024-04-17T07:32:36.176209+0000","last_became_active":"2024-04-17T07:32:35.141127+0000","last_became_peered":"2024-04-17T07:32:35.141127+0000","last_unstale":"2024-04-17T07:32:36.176209+0000","last_undegraded":"2024-04-17T07:32:36.176209+0000","last_fullsized":"2024-04-17T07:32:36.176209+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:38.661050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116749+0000","last_change":"2024-04-17T07:32:35.158821+0000","last_active":"2024-04-17T07:32:36.116749+0000","last_peered":"2024-04-17T07:32:36.116749+0000","last_clean":"2024-04-17T07:32:36.116749+0000","last_became_active":"2024-04-17T07:32:35.158527+0000","last_became_peered":"2024-04-17T07:32:35.158527+0000","last_unstale":"2024-04-17T07:32:36.116749+0000","last_undegraded":"2024-04-17T07:32:36.116749+0000","last_fullsized":"2024-04-17T07:32:36.116749+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:57.877675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161641+0000","last_change":"2024-04-17T07:32:35.137005+0000","last_active":"2024-04-17T07:32:36.161641+0000","last_peered":"2024-04-17T07:32:36.161641+0000","last_clean":"2024-04-17T07:32:36.161641+0000","last_became_active":"2024-04-17T07:32:35.136712+0000","last_became_peered":"2024-04-17T07:32:35.136712+0000","last_unstale":"2024-04-17T07:32:36.161641+0000","last_undegraded":"2024-04-17T07:32:36.161641+0000","last_fullsized":"2024-04-17T07:32:36.161641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:32.292820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176710+0000","last_change":"2024-04-17T07:32:35.157700+0000","last_active":"2024-04-17T07:32:36.176710+0000","last_peered":"2024-04-17T07:32:36.176710+0000","last_clean":"2024-04-17T07:32:36.176710+0000","last_became_active":"2024-04-17T07:32:35.154715+0000","last_became_peered":"2024-04-17T07:32:35.154715+0000","last_unstale":"2024-04-17T07:32:36.176710+0000","last_undegraded":"2024-04-17T07:32:36.176710+0000","last_fullsized":"2024-04-17T07:32:36.176710+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:11.363240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112558+0000","last_change":"2024-04-17T07:32:35.157023+0000","last_active":"2024-04-17T07:32:36.112558+0000","last_peered":"2024-04-17T07:32:36.112558+0000","last_clean":"2024-04-17T07:32:36.112558+0000","last_became_active":"2024-04-17T07:32:35.154591+0000","last_became_peered":"2024-04-17T07:32:35.154591+0000","last_unstale":"2024-04-17T07:32:36.112558+0000","last_undegraded":"2024-04-17T07:32:36.112558+0000","last_fullsized":"2024-04-17T07:32:36.112558+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:37.978579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178230+0000","last_change":"2024-04-17T07:32:35.131815+0000","last_active":"2024-04-17T07:32:36.178230+0000","last_peered":"2024-04-17T07:32:36.178230+0000","last_clean":"2024-04-17T07:32:36.178230+0000","last_became_active":"2024-04-17T07:32:35.130681+0000","last_became_peered":"2024-04-17T07:32:35.130681+0000","last_unstale":"2024-04-17T07:32:36.178230+0000","last_undegraded":"2024-04-17T07:32:36.178230+0000","last_fullsized":"2024-04-17T07:32:36.178230+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:41.171171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112053+0000","last_change":"2024-04-17T07:32:35.145767+0000","last_active":"2024-04-17T07:32:36.112053+0000","last_peered":"2024-04-17T07:32:36.112053+0000","last_clean":"2024-04-17T07:32:36.112053+0000","last_became_active":"2024-04-17T07:32:35.145300+0000","last_became_peered":"2024-04-17T07:32:35.145300+0000","last_unstale":"2024-04-17T07:32:36.112053+0000","last_undegraded":"2024-04-17T07:32:36.112053+0000","last_fullsized":"2024-04-17T07:32:36.112053+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:06:32.292820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161414+0000","last_change":"2024-04-17T07:32:35.132096+0000","last_active":"2024-04-17T07:32:36.161414+0000","last_peered":"2024-04-17T07:32:36.161414+0000","last_clean":"2024-04-17T07:32:36.161414+0000","last_became_active":"2024-04-17T07:32:35.131218+0000","last_became_peered":"2024-04-17T07:32:35.131218+0000","last_unstale":"2024-04-17T07:32:36.161414+0000","last_undegraded":"2024-04-17T07:32:36.161414+0000","last_fullsized":"2024-04-17T07:32:36.161414+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:42.180598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161575+0000","last_change":"2024-04-17T07:32:35.129053+0000","last_active":"2024-04-17T07:32:36.161575+0000","last_peered":"2024-04-17T07:32:36.161575+0000","last_clean":"2024-04-17T07:32:36.161575+0000","last_became_active":"2024-04-17T07:32:35.128491+0000","last_became_peered":"2024-04-17T07:32:35.128491+0000","last_unstale":"2024-04-17T07:32:36.161575+0000","last_undegraded":"2024-04-17T07:32:36.161575+0000","last_fullsized":"2024-04-17T07:32:36.161575+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:52.866924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117144+0000","last_change":"2024-04-17T07:32:35.169272+0000","last_active":"2024-04-17T07:32:36.117144+0000","last_peered":"2024-04-17T07:32:36.117144+0000","last_clean":"2024-04-17T07:32:36.117144+0000","last_became_active":"2024-04-17T07:32:35.169027+0000","last_became_peered":"2024-04-17T07:32:35.169027+0000","last_unstale":"2024-04-17T07:32:36.117144+0000","last_undegraded":"2024-04-17T07:32:36.117144+0000","last_fullsized":"2024-04-17T07:32:36.117144+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:42.070845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117324+0000","last_change":"2024-04-17T07:32:35.159348+0000","last_active":"2024-04-17T07:32:36.117324+0000","last_peered":"2024-04-17T07:32:36.117324+0000","last_clean":"2024-04-17T07:32:36.117324+0000","last_became_active":"2024-04-17T07:32:35.158963+0000","last_became_peered":"2024-04-17T07:32:35.158963+0000","last_unstale":"2024-04-17T07:32:36.117324+0000","last_undegraded":"2024-04-17T07:32:36.117324+0000","last_fullsized":"2024-04-17T07:32:36.117324+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:28:33.012641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162020+0000","last_change":"2024-04-17T07:32:35.143295+0000","last_active":"2024-04-17T07:32:36.162020+0000","last_peered":"2024-04-17T07:32:36.162020+0000","last_clean":"2024-04-17T07:32:36.162020+0000","last_became_active":"2024-04-17T07:32:35.142984+0000","last_became_peered":"2024-04-17T07:32:35.142984+0000","last_unstale":"2024-04-17T07:32:36.162020+0000","last_undegraded":"2024-04-17T07:32:36.162020+0000","last_fullsized":"2024-04-17T07:32:36.162020+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:21.938044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118107+0000","last_change":"2024-04-17T07:32:35.154535+0000","last_active":"2024-04-17T07:32:36.118107+0000","last_peered":"2024-04-17T07:32:36.118107+0000","last_clean":"2024-04-17T07:32:36.118107+0000","last_became_active":"2024-04-17T07:32:35.154009+0000","last_became_peered":"2024-04-17T07:32:35.154009+0000","last_unstale":"2024-04-17T07:32:36.118107+0000","last_undegraded":"2024-04-17T07:32:36.118107+0000","last_fullsized":"2024-04-17T07:32:36.118107+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:47.489094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116314+0000","last_change":"2024-04-17T07:32:35.155640+0000","last_active":"2024-04-17T07:32:36.116314+0000","last_peered":"2024-04-17T07:32:36.116314+0000","last_clean":"2024-04-17T07:32:36.116314+0000","last_became_active":"2024-04-17T07:32:35.155334+0000","last_became_peered":"2024-04-17T07:32:35.155334+0000","last_unstale":"2024-04-17T07:32:36.116314+0000","last_undegraded":"2024-04-17T07:32:36.116314+0000","last_fullsized":"2024-04-17T07:32:36.116314+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:42.808017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161724+0000","last_change":"2024-04-17T07:32:35.145835+0000","last_active":"2024-04-17T07:32:36.161724+0000","last_peered":"2024-04-17T07:32:36.161724+0000","last_clean":"2024-04-17T07:32:36.161724+0000","last_became_active":"2024-04-17T07:32:35.145555+0000","last_became_peered":"2024-04-17T07:32:35.145555+0000","last_unstale":"2024-04-17T07:32:36.161724+0000","last_undegraded":"2024-04-17T07:32:36.161724+0000","last_fullsized":"2024-04-17T07:32:36.161724+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:37.978579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111765+0000","last_change":"2024-04-17T07:32:35.156628+0000","last_active":"2024-04-17T07:32:36.111765+0000","last_peered":"2024-04-17T07:32:36.111765+0000","last_clean":"2024-04-17T07:32:36.111765+0000","last_became_active":"2024-04-17T07:32:35.155845+0000","last_became_peered":"2024-04-17T07:32:35.155845+0000","last_unstale":"2024-04-17T07:32:36.111765+0000","last_undegraded":"2024-04-17T07:32:36.111765+0000","last_fullsized":"2024-04-17T07:32:36.111765+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:45.973978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176534+0000","last_change":"2024-04-17T07:32:35.146638+0000","last_active":"2024-04-17T07:32:36.176534+0000","last_peered":"2024-04-17T07:32:36.176534+0000","last_clean":"2024-04-17T07:32:36.176534+0000","last_became_active":"2024-04-17T07:32:35.146254+0000","last_became_peered":"2024-04-17T07:32:35.146254+0000","last_unstale":"2024-04-17T07:32:36.176534+0000","last_undegraded":"2024-04-17T07:32:36.176534+0000","last_fullsized":"2024-04-17T07:32:36.176534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:05.987532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175220+0000","last_change":"2024-04-17T07:32:35.144190+0000","last_active":"2024-04-17T07:32:36.175220+0000","last_peered":"2024-04-17T07:32:36.175220+0000","last_clean":"2024-04-17T07:32:36.175220+0000","last_became_active":"2024-04-17T07:32:35.143877+0000","last_became_peered":"2024-04-17T07:32:35.143877+0000","last_unstale":"2024-04-17T07:32:36.175220+0000","last_undegraded":"2024-04-17T07:32:36.175220+0000","last_fullsized":"2024-04-17T07:32:36.175220+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:12.206597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112563+0000","last_change":"2024-04-17T07:32:35.147344+0000","last_active":"2024-04-17T07:32:36.112563+0000","last_peered":"2024-04-17T07:32:36.112563+0000","last_clean":"2024-04-17T07:32:36.112563+0000","last_became_active":"2024-04-17T07:32:35.146989+0000","last_became_peered":"2024-04-17T07:32:35.146989+0000","last_unstale":"2024-04-17T07:32:36.112563+0000","last_undegraded":"2024-04-17T07:32:36.112563+0000","last_fullsized":"2024-04-17T07:32:36.112563+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:43.978087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161887+0000","last_change":"2024-04-17T07:32:34.120366+0000","last_active":"2024-04-17T07:32:36.161887+0000","last_peered":"2024-04-17T07:32:36.161887+0000","last_clean":"2024-04-17T07:32:36.161887+0000","last_became_active":"2024-04-17T07:32:34.120065+0000","last_became_peered":"2024-04-17T07:32:34.120065+0000","last_unstale":"2024-04-17T07:32:36.161887+0000","last_undegraded":"2024-04-17T07:32:36.161887+0000","last_fullsized":"2024-04-17T07:32:36.161887+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:06.125055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176830+0000","last_change":"2024-04-17T07:32:35.148547+0000","last_active":"2024-04-17T07:32:36.176830+0000","last_peered":"2024-04-17T07:32:36.176830+0000","last_clean":"2024-04-17T07:32:36.176830+0000","last_became_active":"2024-04-17T07:32:35.148258+0000","last_became_peered":"2024-04-17T07:32:35.148258+0000","last_unstale":"2024-04-17T07:32:36.176830+0000","last_undegraded":"2024-04-17T07:32:36.176830+0000","last_fullsized":"2024-04-17T07:32:36.176830+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:32:57.671400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117971+0000","last_change":"2024-04-17T07:32:34.119437+0000","last_active":"2024-04-17T07:32:36.117971+0000","last_peered":"2024-04-17T07:32:36.117971+0000","last_clean":"2024-04-17T07:32:36.117971+0000","last_became_active":"2024-04-17T07:32:34.118222+0000","last_became_peered":"2024-04-17T07:32:34.118222+0000","last_unstale":"2024-04-17T07:32:36.117971+0000","last_undegraded":"2024-04-17T07:32:36.117971+0000","last_fullsized":"2024-04-17T07:32:36.117971+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:41.002892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175767+0000","last_change":"2024-04-17T07:32:35.146873+0000","last_active":"2024-04-17T07:32:36.175767+0000","last_peered":"2024-04-17T07:32:36.175767+0000","last_clean":"2024-04-17T07:32:36.175767+0000","last_became_active":"2024-04-17T07:32:35.142068+0000","last_became_peered":"2024-04-17T07:32:35.142068+0000","last_unstale":"2024-04-17T07:32:36.175767+0000","last_undegraded":"2024-04-17T07:32:36.175767+0000","last_fullsized":"2024-04-17T07:32:36.175767+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:34.370047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161991+0000","last_change":"2024-04-17T07:32:34.119033+0000","last_active":"2024-04-17T07:32:36.161991+0000","last_peered":"2024-04-17T07:32:36.161991+0000","last_clean":"2024-04-17T07:32:36.161991+0000","last_became_active":"2024-04-17T07:32:34.118728+0000","last_became_peered":"2024-04-17T07:32:34.118728+0000","last_unstale":"2024-04-17T07:32:36.161991+0000","last_undegraded":"2024-04-17T07:32:36.161991+0000","last_fullsized":"2024-04-17T07:32:36.161991+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:28.471682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177991+0000","last_change":"2024-04-17T07:32:35.136460+0000","last_active":"2024-04-17T07:32:36.177991+0000","last_peered":"2024-04-17T07:32:36.177991+0000","last_clean":"2024-04-17T07:32:36.177991+0000","last_became_active":"2024-04-17T07:32:35.135961+0000","last_became_peered":"2024-04-17T07:32:35.135961+0000","last_unstale":"2024-04-17T07:32:36.177991+0000","last_undegraded":"2024-04-17T07:32:36.177991+0000","last_fullsized":"2024-04-17T07:32:36.177991+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:00.016507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161583+0000","last_change":"2024-04-17T07:32:34.117205+0000","last_active":"2024-04-17T07:32:36.161583+0000","last_peered":"2024-04-17T07:32:36.161583+0000","last_clean":"2024-04-17T07:32:36.161583+0000","last_became_active":"2024-04-17T07:32:34.116778+0000","last_became_peered":"2024-04-17T07:32:34.116778+0000","last_unstale":"2024-04-17T07:32:36.161583+0000","last_undegraded":"2024-04-17T07:32:36.161583+0000","last_fullsized":"2024-04-17T07:32:36.161583+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:30.083718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179087+0000","last_change":"2024-04-17T07:32:35.129661+0000","last_active":"2024-04-17T07:32:36.179087+0000","last_peered":"2024-04-17T07:32:36.179087+0000","last_clean":"2024-04-17T07:32:36.179087+0000","last_became_active":"2024-04-17T07:32:35.129155+0000","last_became_peered":"2024-04-17T07:32:35.129155+0000","last_unstale":"2024-04-17T07:32:36.179087+0000","last_undegraded":"2024-04-17T07:32:36.179087+0000","last_fullsized":"2024-04-17T07:32:36.179087+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:38.654110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175344+0000","last_change":"2024-04-17T07:32:34.118731+0000","last_active":"2024-04-17T07:32:36.175344+0000","last_peered":"2024-04-17T07:32:36.175344+0000","last_clean":"2024-04-17T07:32:36.175344+0000","last_became_active":"2024-04-17T07:32:34.118465+0000","last_became_peered":"2024-04-17T07:32:34.118465+0000","last_unstale":"2024-04-17T07:32:36.175344+0000","last_undegraded":"2024-04-17T07:32:36.175344+0000","last_fullsized":"2024-04-17T07:32:36.175344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:14:49.847153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118396+0000","last_change":"2024-04-17T07:32:35.143873+0000","last_active":"2024-04-17T07:32:36.118396+0000","last_peered":"2024-04-17T07:32:36.118396+0000","last_clean":"2024-04-17T07:32:36.118396+0000","last_became_active":"2024-04-17T07:32:35.143144+0000","last_became_peered":"2024-04-17T07:32:35.143144+0000","last_unstale":"2024-04-17T07:32:36.118396+0000","last_undegraded":"2024-04-17T07:32:36.118396+0000","last_fullsized":"2024-04-17T07:32:36.118396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:02:14.597223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161520+0000","last_change":"2024-04-17T07:32:34.109956+0000","last_active":"2024-04-17T07:32:36.161520+0000","last_peered":"2024-04-17T07:32:36.161520+0000","last_clean":"2024-04-17T07:32:36.161520+0000","last_became_active":"2024-04-17T07:32:34.109727+0000","last_became_peered":"2024-04-17T07:32:34.109727+0000","last_unstale":"2024-04-17T07:32:36.161520+0000","last_undegraded":"2024-04-17T07:32:36.161520+0000","last_fullsized":"2024-04-17T07:32:36.161520+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:33.146222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176067+0000","last_change":"2024-04-17T07:32:35.152746+0000","last_active":"2024-04-17T07:32:36.176067+0000","last_peered":"2024-04-17T07:32:36.176067+0000","last_clean":"2024-04-17T07:32:36.176067+0000","last_became_active":"2024-04-17T07:32:35.151895+0000","last_became_peered":"2024-04-17T07:32:35.151895+0000","last_unstale":"2024-04-17T07:32:36.176067+0000","last_undegraded":"2024-04-17T07:32:36.176067+0000","last_fullsized":"2024-04-17T07:32:36.176067+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:07.429754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.116050+0000","last_change":"2024-04-17T07:32:34.116820+0000","last_active":"2024-04-17T07:32:36.116050+0000","last_peered":"2024-04-17T07:32:36.116050+0000","last_clean":"2024-04-17T07:32:36.116050+0000","last_became_active":"2024-04-17T07:32:34.116317+0000","last_became_peered":"2024-04-17T07:32:34.116317+0000","last_unstale":"2024-04-17T07:32:36.116050+0000","last_undegraded":"2024-04-17T07:32:36.116050+0000","last_fullsized":"2024-04-17T07:32:36.116050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:38.565254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177034+0000","last_change":"2024-04-17T07:32:35.129508+0000","last_active":"2024-04-17T07:32:36.177034+0000","last_peered":"2024-04-17T07:32:36.177034+0000","last_clean":"2024-04-17T07:32:36.177034+0000","last_became_active":"2024-04-17T07:32:35.128961+0000","last_became_peered":"2024-04-17T07:32:35.128961+0000","last_unstale":"2024-04-17T07:32:36.177034+0000","last_undegraded":"2024-04-17T07:32:36.177034+0000","last_fullsized":"2024-04-17T07:32:36.177034+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:59.992707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175075+0000","last_change":"2024-04-17T07:32:34.103139+0000","last_active":"2024-04-17T07:32:36.175075+0000","last_peered":"2024-04-17T07:32:36.175075+0000","last_clean":"2024-04-17T07:32:36.175075+0000","last_became_active":"2024-04-17T07:32:34.102635+0000","last_became_peered":"2024-04-17T07:32:34.102635+0000","last_unstale":"2024-04-17T07:32:36.175075+0000","last_undegraded":"2024-04-17T07:32:36.175075+0000","last_fullsized":"2024-04-17T07:32:36.175075+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:36:01.668959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161932+0000","last_change":"2024-04-17T07:32:35.124617+0000","last_active":"2024-04-17T07:32:36.161932+0000","last_peered":"2024-04-17T07:32:36.161932+0000","last_clean":"2024-04-17T07:32:36.161932+0000","last_became_active":"2024-04-17T07:32:35.123058+0000","last_became_peered":"2024-04-17T07:32:35.123058+0000","last_unstale":"2024-04-17T07:32:36.161932+0000","last_undegraded":"2024-04-17T07:32:36.161932+0000","last_fullsized":"2024-04-17T07:32:36.161932+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:52.230466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176118+0000","last_change":"2024-04-17T07:32:34.107372+0000","last_active":"2024-04-17T07:32:36.176118+0000","last_peered":"2024-04-17T07:32:36.176118+0000","last_clean":"2024-04-17T07:32:36.176118+0000","last_became_active":"2024-04-17T07:32:34.107087+0000","last_became_peered":"2024-04-17T07:32:34.107087+0000","last_unstale":"2024-04-17T07:32:36.176118+0000","last_undegraded":"2024-04-17T07:32:36.176118+0000","last_fullsized":"2024-04-17T07:32:36.176118+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:29.102580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178148+0000","last_change":"2024-04-17T07:32:35.137895+0000","last_active":"2024-04-17T07:32:36.178148+0000","last_peered":"2024-04-17T07:32:36.178148+0000","last_clean":"2024-04-17T07:32:36.178148+0000","last_became_active":"2024-04-17T07:32:35.137601+0000","last_became_peered":"2024-04-17T07:32:35.137601+0000","last_unstale":"2024-04-17T07:32:36.178148+0000","last_undegraded":"2024-04-17T07:32:36.178148+0000","last_fullsized":"2024-04-17T07:32:36.178148+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.278212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176721+0000","last_change":"2024-04-17T07:32:34.118513+0000","last_active":"2024-04-17T07:32:36.176721+0000","last_peered":"2024-04-17T07:32:36.176721+0000","last_clean":"2024-04-17T07:32:36.176721+0000","last_became_active":"2024-04-17T07:32:34.118231+0000","last_became_peered":"2024-04-17T07:32:34.118231+0000","last_unstale":"2024-04-17T07:32:36.176721+0000","last_undegraded":"2024-04-17T07:32:36.176721+0000","last_fullsized":"2024-04-17T07:32:36.176721+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:38.711925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117406+0000","last_change":"2024-04-17T07:32:35.154386+0000","last_active":"2024-04-17T07:32:36.117406+0000","last_peered":"2024-04-17T07:32:36.117406+0000","last_clean":"2024-04-17T07:32:36.117406+0000","last_became_active":"2024-04-17T07:32:35.153650+0000","last_became_peered":"2024-04-17T07:32:35.153650+0000","last_unstale":"2024-04-17T07:32:36.117406+0000","last_undegraded":"2024-04-17T07:32:36.117406+0000","last_fullsized":"2024-04-17T07:32:36.117406+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:11:19.296441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118584+0000","last_change":"2024-04-17T07:32:34.115451+0000","last_active":"2024-04-17T07:32:36.118584+0000","last_peered":"2024-04-17T07:32:36.118584+0000","last_clean":"2024-04-17T07:32:36.118584+0000","last_became_active":"2024-04-17T07:32:34.115118+0000","last_became_peered":"2024-04-17T07:32:34.115118+0000","last_unstale":"2024-04-17T07:32:36.118584+0000","last_undegraded":"2024-04-17T07:32:36.118584+0000","last_fullsized":"2024-04-17T07:32:36.118584+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:25.659014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118200+0000","last_change":"2024-04-17T07:32:35.147902+0000","last_active":"2024-04-17T07:32:36.118200+0000","last_peered":"2024-04-17T07:32:36.118200+0000","last_clean":"2024-04-17T07:32:36.118200+0000","last_became_active":"2024-04-17T07:32:35.146971+0000","last_became_peered":"2024-04-17T07:32:35.146971+0000","last_unstale":"2024-04-17T07:32:36.118200+0000","last_undegraded":"2024-04-17T07:32:36.118200+0000","last_fullsized":"2024-04-17T07:32:36.118200+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:16:16.519615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178508+0000","last_change":"2024-04-17T07:32:34.122434+0000","last_active":"2024-04-17T07:32:36.178508+0000","last_peered":"2024-04-17T07:32:36.178508+0000","last_clean":"2024-04-17T07:32:36.178508+0000","last_became_active":"2024-04-17T07:32:34.121792+0000","last_became_peered":"2024-04-17T07:32:34.121792+0000","last_unstale":"2024-04-17T07:32:36.178508+0000","last_undegraded":"2024-04-17T07:32:36.178508+0000","last_fullsized":"2024-04-17T07:32:36.178508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:41:52.433800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176297+0000","last_change":"2024-04-17T07:32:35.150620+0000","last_active":"2024-04-17T07:32:36.176297+0000","last_peered":"2024-04-17T07:32:36.176297+0000","last_clean":"2024-04-17T07:32:36.176297+0000","last_became_active":"2024-04-17T07:32:35.150343+0000","last_became_peered":"2024-04-17T07:32:35.150343+0000","last_unstale":"2024-04-17T07:32:36.176297+0000","last_undegraded":"2024-04-17T07:32:36.176297+0000","last_fullsized":"2024-04-17T07:32:36.176297+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:54.548737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112099+0000","last_change":"2024-04-17T07:32:34.119211+0000","last_active":"2024-04-17T07:32:36.112099+0000","last_peered":"2024-04-17T07:32:36.112099+0000","last_clean":"2024-04-17T07:32:36.112099+0000","last_became_active":"2024-04-17T07:32:34.118856+0000","last_became_peered":"2024-04-17T07:32:34.118856+0000","last_unstale":"2024-04-17T07:32:36.112099+0000","last_undegraded":"2024-04-17T07:32:36.112099+0000","last_fullsized":"2024-04-17T07:32:36.112099+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:33:09.831283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175236+0000","last_change":"2024-04-17T07:32:35.153895+0000","last_active":"2024-04-17T07:32:36.175236+0000","last_peered":"2024-04-17T07:32:36.175236+0000","last_clean":"2024-04-17T07:32:36.175236+0000","last_became_active":"2024-04-17T07:32:35.151794+0000","last_became_peered":"2024-04-17T07:32:35.151794+0000","last_unstale":"2024-04-17T07:32:36.175236+0000","last_undegraded":"2024-04-17T07:32:36.175236+0000","last_fullsized":"2024-04-17T07:32:36.175236+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.978262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112039+0000","last_change":"2024-04-17T07:32:34.117055+0000","last_active":"2024-04-17T07:32:36.112039+0000","last_peered":"2024-04-17T07:32:36.112039+0000","last_clean":"2024-04-17T07:32:36.112039+0000","last_became_active":"2024-04-17T07:32:34.116609+0000","last_became_peered":"2024-04-17T07:32:34.116609+0000","last_unstale":"2024-04-17T07:32:36.112039+0000","last_undegraded":"2024-04-17T07:32:36.112039+0000","last_fullsized":"2024-04-17T07:32:36.112039+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:15:57.845439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177348+0000","last_change":"2024-04-17T07:32:35.127746+0000","last_active":"2024-04-17T07:32:36.177348+0000","last_peered":"2024-04-17T07:32:36.177348+0000","last_clean":"2024-04-17T07:32:36.177348+0000","last_became_active":"2024-04-17T07:32:35.127026+0000","last_became_peered":"2024-04-17T07:32:35.127026+0000","last_unstale":"2024-04-17T07:32:36.177348+0000","last_undegraded":"2024-04-17T07:32:36.177348+0000","last_fullsized":"2024-04-17T07:32:36.177348+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:42:14.360457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177012+0000","last_change":"2024-04-17T07:32:34.118865+0000","last_active":"2024-04-17T07:32:36.177012+0000","last_peered":"2024-04-17T07:32:36.177012+0000","last_clean":"2024-04-17T07:32:36.177012+0000","last_became_active":"2024-04-17T07:32:34.118465+0000","last_became_peered":"2024-04-17T07:32:34.118465+0000","last_unstale":"2024-04-17T07:32:36.177012+0000","last_undegraded":"2024-04-17T07:32:36.177012+0000","last_fullsized":"2024-04-17T07:32:36.177012+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:01.394709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111659+0000","last_change":"2024-04-17T07:32:35.144776+0000","last_active":"2024-04-17T07:32:36.111659+0000","last_peered":"2024-04-17T07:32:36.111659+0000","last_clean":"2024-04-17T07:32:36.111659+0000","last_became_active":"2024-04-17T07:32:35.144482+0000","last_became_peered":"2024-04-17T07:32:35.144482+0000","last_unstale":"2024-04-17T07:32:36.111659+0000","last_undegraded":"2024-04-17T07:32:36.111659+0000","last_fullsized":"2024-04-17T07:32:36.111659+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:58.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175197+0000","last_change":"2024-04-17T07:32:34.110438+0000","last_active":"2024-04-17T07:32:36.175197+0000","last_peered":"2024-04-17T07:32:36.175197+0000","last_clean":"2024-04-17T07:32:36.175197+0000","last_became_active":"2024-04-17T07:32:34.109449+0000","last_became_peered":"2024-04-17T07:32:34.109449+0000","last_unstale":"2024-04-17T07:32:36.175197+0000","last_undegraded":"2024-04-17T07:32:36.175197+0000","last_fullsized":"2024-04-17T07:32:36.175197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:49.182032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117710+0000","last_change":"2024-04-17T07:32:35.148505+0000","last_active":"2024-04-17T07:32:36.117710+0000","last_peered":"2024-04-17T07:32:36.117710+0000","last_clean":"2024-04-17T07:32:36.117710+0000","last_became_active":"2024-04-17T07:32:35.148196+0000","last_became_peered":"2024-04-17T07:32:35.148196+0000","last_unstale":"2024-04-17T07:32:36.117710+0000","last_undegraded":"2024-04-17T07:32:36.117710+0000","last_fullsized":"2024-04-17T07:32:36.117710+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:34.370047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177861+0000","last_change":"2024-04-17T07:32:34.119465+0000","last_active":"2024-04-17T07:32:36.177861+0000","last_peered":"2024-04-17T07:32:36.177861+0000","last_clean":"2024-04-17T07:32:36.177861+0000","last_became_active":"2024-04-17T07:32:34.119062+0000","last_became_peered":"2024-04-17T07:32:34.119062+0000","last_unstale":"2024-04-17T07:32:36.177861+0000","last_undegraded":"2024-04-17T07:32:36.177861+0000","last_fullsized":"2024-04-17T07:32:36.177861+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:45.726103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179192+0000","last_change":"2024-04-17T07:32:35.142092+0000","last_active":"2024-04-17T07:32:36.179192+0000","last_peered":"2024-04-17T07:32:36.179192+0000","last_clean":"2024-04-17T07:32:36.179192+0000","last_became_active":"2024-04-17T07:32:35.141566+0000","last_became_peered":"2024-04-17T07:32:35.141566+0000","last_unstale":"2024-04-17T07:32:36.179192+0000","last_undegraded":"2024-04-17T07:32:36.179192+0000","last_fullsized":"2024-04-17T07:32:36.179192+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:41.490644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117581+0000","last_change":"2024-04-17T07:32:34.119203+0000","last_active":"2024-04-17T07:32:36.117581+0000","last_peered":"2024-04-17T07:32:36.117581+0000","last_clean":"2024-04-17T07:32:36.117581+0000","last_became_active":"2024-04-17T07:32:34.118889+0000","last_became_peered":"2024-04-17T07:32:34.118889+0000","last_unstale":"2024-04-17T07:32:36.117581+0000","last_undegraded":"2024-04-17T07:32:36.117581+0000","last_fullsized":"2024-04-17T07:32:36.117581+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:28.569706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118534+0000","last_change":"2024-04-17T07:32:35.144152+0000","last_active":"2024-04-17T07:32:36.118534+0000","last_peered":"2024-04-17T07:32:36.118534+0000","last_clean":"2024-04-17T07:32:36.118534+0000","last_became_active":"2024-04-17T07:32:35.143826+0000","last_became_peered":"2024-04-17T07:32:35.143826+0000","last_unstale":"2024-04-17T07:32:36.118534+0000","last_undegraded":"2024-04-17T07:32:36.118534+0000","last_fullsized":"2024-04-17T07:32:36.118534+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:09.754340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161876+0000","last_change":"2024-04-17T07:32:34.115795+0000","last_active":"2024-04-17T07:32:36.161876+0000","last_peered":"2024-04-17T07:32:36.161876+0000","last_clean":"2024-04-17T07:32:36.161876+0000","last_became_active":"2024-04-17T07:32:34.115495+0000","last_became_peered":"2024-04-17T07:32:34.115495+0000","last_unstale":"2024-04-17T07:32:36.161876+0000","last_undegraded":"2024-04-17T07:32:36.161876+0000","last_fullsized":"2024-04-17T07:32:36.161876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:44.817097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112004+0000","last_change":"2024-04-17T07:32:35.152975+0000","last_active":"2024-04-17T07:32:36.112004+0000","last_peered":"2024-04-17T07:32:36.112004+0000","last_clean":"2024-04-17T07:32:36.112004+0000","last_became_active":"2024-04-17T07:32:35.152639+0000","last_became_peered":"2024-04-17T07:32:35.152639+0000","last_unstale":"2024-04-17T07:32:36.112004+0000","last_undegraded":"2024-04-17T07:32:36.112004+0000","last_fullsized":"2024-04-17T07:32:36.112004+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:42.180598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161718+0000","last_change":"2024-04-17T07:32:34.114014+0000","last_active":"2024-04-17T07:32:36.161718+0000","last_peered":"2024-04-17T07:32:36.161718+0000","last_clean":"2024-04-17T07:32:36.161718+0000","last_became_active":"2024-04-17T07:32:34.113722+0000","last_became_peered":"2024-04-17T07:32:34.113722+0000","last_unstale":"2024-04-17T07:32:36.161718+0000","last_undegraded":"2024-04-17T07:32:36.161718+0000","last_fullsized":"2024-04-17T07:32:36.161718+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:11.145740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111996+0000","last_change":"2024-04-17T07:32:35.161559+0000","last_active":"2024-04-17T07:32:36.111996+0000","last_peered":"2024-04-17T07:32:36.111996+0000","last_clean":"2024-04-17T07:32:36.111996+0000","last_became_active":"2024-04-17T07:32:35.161222+0000","last_became_peered":"2024-04-17T07:32:35.161222+0000","last_unstale":"2024-04-17T07:32:36.111996+0000","last_undegraded":"2024-04-17T07:32:36.111996+0000","last_fullsized":"2024-04-17T07:32:36.111996+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:52.866924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178990+0000","last_change":"2024-04-17T07:32:34.106442+0000","last_active":"2024-04-17T07:32:36.178990+0000","last_peered":"2024-04-17T07:32:36.178990+0000","last_clean":"2024-04-17T07:32:36.178990+0000","last_became_active":"2024-04-17T07:32:34.105981+0000","last_became_peered":"2024-04-17T07:32:34.105981+0000","last_unstale":"2024-04-17T07:32:36.178990+0000","last_undegraded":"2024-04-17T07:32:36.178990+0000","last_fullsized":"2024-04-17T07:32:36.178990+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:21.122385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111687+0000","last_change":"2024-04-17T07:32:35.157407+0000","last_active":"2024-04-17T07:32:36.111687+0000","last_peered":"2024-04-17T07:32:36.111687+0000","last_clean":"2024-04-17T07:32:36.111687+0000","last_became_active":"2024-04-17T07:32:35.156513+0000","last_became_peered":"2024-04-17T07:32:35.156513+0000","last_unstale":"2024-04-17T07:32:36.111687+0000","last_undegraded":"2024-04-17T07:32:36.111687+0000","last_fullsized":"2024-04-17T07:32:36.111687+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:08.175259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.178935+0000","last_change":"2024-04-17T07:32:34.109456+0000","last_active":"2024-04-17T07:32:36.178935+0000","last_peered":"2024-04-17T07:32:36.178935+0000","last_clean":"2024-04-17T07:32:36.178935+0000","last_became_active":"2024-04-17T07:32:34.106867+0000","last_became_peered":"2024-04-17T07:32:34.106867+0000","last_unstale":"2024-04-17T07:32:36.178935+0000","last_undegraded":"2024-04-17T07:32:36.178935+0000","last_fullsized":"2024-04-17T07:32:36.178935+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:39.889609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.115788+0000","last_change":"2024-04-17T07:32:35.160126+0000","last_active":"2024-04-17T07:32:36.115788+0000","last_peered":"2024-04-17T07:32:36.115788+0000","last_clean":"2024-04-17T07:32:36.115788+0000","last_became_active":"2024-04-17T07:32:35.159786+0000","last_became_peered":"2024-04-17T07:32:35.159786+0000","last_unstale":"2024-04-17T07:32:36.115788+0000","last_undegraded":"2024-04-17T07:32:36.115788+0000","last_fullsized":"2024-04-17T07:32:36.115788+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:23.812342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161438+0000","last_change":"2024-04-17T07:32:34.104872+0000","last_active":"2024-04-17T07:32:36.161438+0000","last_peered":"2024-04-17T07:32:36.161438+0000","last_clean":"2024-04-17T07:32:36.161438+0000","last_became_active":"2024-04-17T07:32:34.104576+0000","last_became_peered":"2024-04-17T07:32:34.104576+0000","last_unstale":"2024-04-17T07:32:36.161438+0000","last_undegraded":"2024-04-17T07:32:36.161438+0000","last_fullsized":"2024-04-17T07:32:36.161438+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:33:09.831283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162067+0000","last_change":"2024-04-17T07:32:35.129628+0000","last_active":"2024-04-17T07:32:36.162067+0000","last_peered":"2024-04-17T07:32:36.162067+0000","last_clean":"2024-04-17T07:32:36.162067+0000","last_became_active":"2024-04-17T07:32:35.128422+0000","last_became_peered":"2024-04-17T07:32:35.128422+0000","last_unstale":"2024-04-17T07:32:36.162067+0000","last_undegraded":"2024-04-17T07:32:36.162067+0000","last_fullsized":"2024-04-17T07:32:36.162067+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:14.535287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.179721+0000","last_change":"2024-04-17T07:32:34.115849+0000","last_active":"2024-04-17T07:32:36.179721+0000","last_peered":"2024-04-17T07:32:36.179721+0000","last_clean":"2024-04-17T07:32:36.179721+0000","last_became_active":"2024-04-17T07:32:34.115554+0000","last_became_peered":"2024-04-17T07:32:34.115554+0000","last_unstale":"2024-04-17T07:32:36.179721+0000","last_undegraded":"2024-04-17T07:32:36.179721+0000","last_fullsized":"2024-04-17T07:32:36.179721+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:24.993547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176019+0000","last_change":"2024-04-17T07:32:35.151950+0000","last_active":"2024-04-17T07:32:36.176019+0000","last_peered":"2024-04-17T07:32:36.176019+0000","last_clean":"2024-04-17T07:32:36.176019+0000","last_became_active":"2024-04-17T07:32:35.145334+0000","last_became_peered":"2024-04-17T07:32:35.145334+0000","last_unstale":"2024-04-17T07:32:36.176019+0000","last_undegraded":"2024-04-17T07:32:36.176019+0000","last_fullsized":"2024-04-17T07:32:36.176019+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:47.255018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.163881+0000","last_change":"2024-04-17T07:32:34.114148+0000","last_active":"2024-04-17T07:32:36.163881+0000","last_peered":"2024-04-17T07:32:36.163881+0000","last_clean":"2024-04-17T07:32:36.163881+0000","last_became_active":"2024-04-17T07:32:34.113529+0000","last_became_peered":"2024-04-17T07:32:34.113529+0000","last_unstale":"2024-04-17T07:32:36.163881+0000","last_undegraded":"2024-04-17T07:32:36.163881+0000","last_fullsized":"2024-04-17T07:32:36.163881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:45.667890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.175610+0000","last_change":"2024-04-17T07:32:35.152780+0000","last_active":"2024-04-17T07:32:36.175610+0000","last_peered":"2024-04-17T07:32:36.175610+0000","last_clean":"2024-04-17T07:32:36.175610+0000","last_became_active":"2024-04-17T07:32:35.151186+0000","last_became_peered":"2024-04-17T07:32:35.151186+0000","last_unstale":"2024-04-17T07:32:36.175610+0000","last_undegraded":"2024-04-17T07:32:36.175610+0000","last_fullsized":"2024-04-17T07:32:36.175610+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:28:33.012641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.162088+0000","last_change":"2024-04-17T07:32:34.114679+0000","last_active":"2024-04-17T07:32:36.162088+0000","last_peered":"2024-04-17T07:32:36.162088+0000","last_clean":"2024-04-17T07:32:36.162088+0000","last_became_active":"2024-04-17T07:32:34.114406+0000","last_became_peered":"2024-04-17T07:32:34.114406+0000","last_unstale":"2024-04-17T07:32:36.162088+0000","last_undegraded":"2024-04-17T07:32:36.162088+0000","last_fullsized":"2024-04-17T07:32:36.162088+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:37.930564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117322+0000","last_change":"2024-04-17T07:32:35.156586+0000","last_active":"2024-04-17T07:32:36.117322+0000","last_peered":"2024-04-17T07:32:36.117322+0000","last_clean":"2024-04-17T07:32:36.117322+0000","last_became_active":"2024-04-17T07:32:35.155514+0000","last_became_peered":"2024-04-17T07:32:35.155514+0000","last_unstale":"2024-04-17T07:32:36.117322+0000","last_undegraded":"2024-04-17T07:32:36.117322+0000","last_fullsized":"2024-04-17T07:32:36.117322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.978262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117692+0000","last_change":"2024-04-17T07:32:34.123668+0000","last_active":"2024-04-17T07:32:36.117692+0000","last_peered":"2024-04-17T07:32:36.117692+0000","last_clean":"2024-04-17T07:32:36.117692+0000","last_became_active":"2024-04-17T07:32:34.123317+0000","last_became_peered":"2024-04-17T07:32:34.123317+0000","last_unstale":"2024-04-17T07:32:36.117692+0000","last_undegraded":"2024-04-17T07:32:36.117692+0000","last_fullsized":"2024-04-17T07:32:36.117692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:16:35.188911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111950+0000","last_change":"2024-04-17T07:32:35.148068+0000","last_active":"2024-04-17T07:32:36.111950+0000","last_peered":"2024-04-17T07:32:36.111950+0000","last_clean":"2024-04-17T07:32:36.111950+0000","last_became_active":"2024-04-17T07:32:35.147557+0000","last_became_peered":"2024-04-17T07:32:35.147557+0000","last_unstale":"2024-04-17T07:32:36.111950+0000","last_undegraded":"2024-04-17T07:32:36.111950+0000","last_fullsized":"2024-04-17T07:32:36.111950+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:27.796262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.161789+0000","last_change":"2024-04-17T07:32:34.106683+0000","last_active":"2024-04-17T07:32:36.161789+0000","last_peered":"2024-04-17T07:32:36.161789+0000","last_clean":"2024-04-17T07:32:36.161789+0000","last_became_active":"2024-04-17T07:32:34.106255+0000","last_became_peered":"2024-04-17T07:32:34.106255+0000","last_unstale":"2024-04-17T07:32:36.161789+0000","last_undegraded":"2024-04-17T07:32:36.161789+0000","last_fullsized":"2024-04-17T07:32:36.161789+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:28.482375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.177522+0000","last_change":"2024-04-17T07:32:35.146392+0000","last_active":"2024-04-17T07:32:36.177522+0000","last_peered":"2024-04-17T07:32:36.177522+0000","last_clean":"2024-04-17T07:32:36.177522+0000","last_became_active":"2024-04-17T07:32:35.146014+0000","last_became_peered":"2024-04-17T07:32:35.146014+0000","last_unstale":"2024-04-17T07:32:36.177522+0000","last_undegraded":"2024-04-17T07:32:36.177522+0000","last_fullsized":"2024-04-17T07:32:36.177522+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:33.587267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.115940+0000","last_change":"2024-04-17T07:32:34.117019+0000","last_active":"2024-04-17T07:32:36.115940+0000","last_peered":"2024-04-17T07:32:36.115940+0000","last_clean":"2024-04-17T07:32:36.115940+0000","last_became_active":"2024-04-17T07:32:34.116493+0000","last_became_peered":"2024-04-17T07:32:34.116493+0000","last_unstale":"2024-04-17T07:32:36.115940+0000","last_undegraded":"2024-04-17T07:32:36.115940+0000","last_fullsized":"2024-04-17T07:32:36.115940+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:11.827705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111646+0000","last_change":"2024-04-17T07:32:35.149351+0000","last_active":"2024-04-17T07:32:36.111646+0000","last_peered":"2024-04-17T07:32:36.111646+0000","last_clean":"2024-04-17T07:32:36.111646+0000","last_became_active":"2024-04-17T07:32:35.148610+0000","last_became_peered":"2024-04-17T07:32:35.148610+0000","last_unstale":"2024-04-17T07:32:36.111646+0000","last_undegraded":"2024-04-17T07:32:36.111646+0000","last_fullsized":"2024-04-17T07:32:36.111646+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:22.941744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.176328+0000","last_change":"2024-04-17T07:32:34.114975+0000","last_active":"2024-04-17T07:32:36.176328+0000","last_peered":"2024-04-17T07:32:36.176328+0000","last_clean":"2024-04-17T07:32:36.176328+0000","last_became_active":"2024-04-17T07:32:34.114695+0000","last_became_peered":"2024-04-17T07:32:34.114695+0000","last_unstale":"2024-04-17T07:32:36.176328+0000","last_undegraded":"2024-04-17T07:32:36.176328+0000","last_fullsized":"2024-04-17T07:32:36.176328+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:23.375185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117971+0000","last_change":"2024-04-17T07:32:35.155946+0000","last_active":"2024-04-17T07:32:36.117971+0000","last_peered":"2024-04-17T07:32:36.117971+0000","last_clean":"2024-04-17T07:32:36.117971+0000","last_became_active":"2024-04-17T07:32:35.155625+0000","last_became_peered":"2024-04-17T07:32:35.155625+0000","last_unstale":"2024-04-17T07:32:36.117971+0000","last_undegraded":"2024-04-17T07:32:36.117971+0000","last_fullsized":"2024-04-17T07:32:36.117971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:18.150601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.111815+0000","last_change":"2024-04-17T07:32:34.122836+0000","last_active":"2024-04-17T07:32:36.111815+0000","last_peered":"2024-04-17T07:32:36.111815+0000","last_clean":"2024-04-17T07:32:36.111815+0000","last_became_active":"2024-04-17T07:32:34.120873+0000","last_became_peered":"2024-04-17T07:32:34.120873+0000","last_unstale":"2024-04-17T07:32:36.111815+0000","last_undegraded":"2024-04-17T07:32:36.111815+0000","last_fullsized":"2024-04-17T07:32:36.111815+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:48.287591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.112693+0000","last_change":"2024-04-17T07:32:35.160004+0000","last_active":"2024-04-17T07:32:36.112693+0000","last_peered":"2024-04-17T07:32:36.112693+0000","last_clean":"2024-04-17T07:32:36.112693+0000","last_became_active":"2024-04-17T07:32:35.158591+0000","last_became_peered":"2024-04-17T07:32:35.158591+0000","last_unstale":"2024-04-17T07:32:36.112693+0000","last_undegraded":"2024-04-17T07:32:36.112693+0000","last_fullsized":"2024-04-17T07:32:36.112693+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:34.076645+0000","last_clean_scrub_stamp":"2024-04-17T07:32:34.076645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:52.230466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.118125+0000","last_change":"2024-04-17T07:32:34.122654+0000","last_active":"2024-04-17T07:32:36.118125+0000","last_peered":"2024-04-17T07:32:36.118125+0000","last_clean":"2024-04-17T07:32:36.118125+0000","last_became_active":"2024-04-17T07:32:34.122324+0000","last_became_peered":"2024-04-17T07:32:34.122324+0000","last_unstale":"2024-04-17T07:32:36.118125+0000","last_undegraded":"2024-04-17T07:32:36.118125+0000","last_fullsized":"2024-04-17T07:32:36.118125+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:32:33.071093+0000","last_clean_scrub_stamp":"2024-04-17T07:32:33.071093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:06.424202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"115'211","reported_seq":709,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-04-17T07:32:36.117322+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:32:36.117322+0000","last_peered":"2024-04-17T07:32:36.117322+0000","last_clean":"2024-04-17T07:32:36.117322+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:32:36.117322+0000","last_undegraded":"2024-04-17T07:32:36.117322+0000","last_fullsized":"2024-04-17T07:32:36.117322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":327,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":930,"num_read_kb":1912,"num_write":327,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542367,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37444,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334396,"statfs":{"total":96636764160,"available":96598421504,"internally_reserved":0,"allocated":2154496,"data_stored":1333225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542367,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1572864,"data_stored":749697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542366,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1568768,"data_stored":749115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542366,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59096,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312744,"statfs":{"total":96636764160,"available":96576249856,"internally_reserved":0,"allocated":1585152,"data_stored":750781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542368,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1581056,"data_stored":750679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542367,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41036,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330804,"statfs":{"total":96636764160,"available":96594743296,"internally_reserved":0,"allocated":1568768,"data_stored":749079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542368,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":1564672,"data_stored":749045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542366,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37456,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334384,"statfs":{"total":96636764160,"available":96598409216,"internally_reserved":0,"allocated":2162688,"data_stored":1339995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:32:40.170 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:32:40.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:32:40.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.444+0000 7fde35c00640 1 Processor -- start 2024-04-17T07:32:40.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.444+0000 7fde35c00640 1 -- start start 2024-04-17T07:32:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.446+0000 7fde35c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde300b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.446+0000 7fde35c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fde300b4640 0x7fde3006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.446+0000 7fde35c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde30145c60 con 0x7fde3006be00 2024-04-17T07:32:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.446+0000 7fde35c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fde300b4e20 con 0x7fde300b4640 2024-04-17T07:32:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.446+0000 7fde35c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fde30145a10 con 0x7fde30145370 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fde300b4640 0x7fde3006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde300b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2f400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fde3006be00 0x7fde301627f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:40366/0 (socket says 172.21.15.67:40366) 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde300b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:34310/0 (socket says 172.21.15.67:34310) 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2ea00640 1 -- 172.21.15.67:0/158010893 learned_addr learned my addr 172.21.15.67:0/158010893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fde300b4640 0x7fde3006b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:47722/0 (socket says 172.21.15.67:47722) 2024-04-17T07:32:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 -- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 msgr2=0x7fde300b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 --2- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde300b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 -- 172.21.15.67:0/158010893 >> v1:172.21.15.187:6789/0 conn(0x7fde3006be00 legacy=0x7fde301627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.447+0000 7fde2e000640 1 -- 172.21.15.67:0/158010893 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde30145f40 con 0x7fde300b4640 2024-04-17T07:32:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.449+0000 7fde2e000640 1 --2- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde300b4640 0x7fde3006b8c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fde18009d90 tx=0x7fde1802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85cb84ecd1dbc6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.449+0000 7fde2d600640 1 -- 172.21.15.67:0/158010893 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde18039070 con 0x7fde300b4640 2024-04-17T07:32:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.449+0000 7fde2d600640 1 -- 172.21.15.67:0/158010893 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1802cc50 con 0x7fde300b4640 2024-04-17T07:32:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.449+0000 7fde2d600640 1 -- 172.21.15.67:0/158010893 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde18034440 con 0x7fde300b4640 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 -- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde300b4640 msgr2=0x7fde3006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 --2- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde300b4640 0x7fde3006b8c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fde18009d90 tx=0x7fde1802bea0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 -- 172.21.15.67:0/158010893 shutdown_connections 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 --2- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde300b4640 0x7fde3006b8c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 --2- 172.21.15.67:0/158010893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde300b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 -- 172.21.15.67:0/158010893 >> 172.21.15.67:0/158010893 conn(0x7fde300b6d80 msgr2=0x7fde3009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 -- 172.21.15.67:0/158010893 shutdown_connections 2024-04-17T07:32:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.450+0000 7fde35c00640 1 -- 172.21.15.67:0/158010893 wait complete. 2024-04-17T07:32:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.452+0000 7fde35c00640 1 Processor -- start 2024-04-17T07:32:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.452+0000 7fde35c00640 1 -- start start 2024-04-17T07:32:40.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3006be00 0x7fde3013d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde3015f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde30147a30 con 0x7fde300b4640 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fde300b4aa0 con 0x7fde3006be00 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde35c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fde30147bb0 con 0x7fde30145370 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde2e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde2e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32810/0 (socket says 172.21.15.67:32810) 2024-04-17T07:32:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.453+0000 7fde2e000640 1 -- 172.21.15.67:0/2966770337 learned_addr learned my addr 172.21.15.67:0/2966770337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 -- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 msgr2=0x7fde3015f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde3015f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 -- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3006be00 msgr2=0x7fde3013d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3006be00 0x7fde3013d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 -- 172.21.15.67:0/2966770337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3015fb40 con 0x7fde300b4640 2024-04-17T07:32:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.454+0000 7fde2e000640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fde18037040 tx=0x7fde18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.455+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde18039050 con 0x7fde300b4640 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.455+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde18045070 con 0x7fde300b4640 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.455+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3015fe20 con 0x7fde300b4640 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.455+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1803e610 con 0x7fde300b4640 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.455+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde301608e0 con 0x7fde300b4640 2024-04-17T07:32:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.456+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3006d960 con 0x7fde300b4640 2024-04-17T07:32:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.457+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7fde1803e7b0 con 0x7fde300b4640 2024-04-17T07:32:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.458+0000 7fde26a00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 0x7fde0c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.459+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7fde180ae940 con 0x7fde300b4640 2024-04-17T07:32:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.459+0000 7fde2ea00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 0x7fde0c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:40.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.460+0000 7fde2ea00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 0x7fde0c06ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde20006f40 tx=0x7fde20006e90 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.463+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fde1807cd70 con 0x7fde300b4640 2024-04-17T07:32:40.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.745+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fde30160340 con 0x7fde300b4640 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.750+0000 7fde26a00640 1 -- 172.21.15.67:0/2966770337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7fde1807c670 con 0x7fde300b4640 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:32:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.753+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 msgr2=0x7fde0c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.753+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 0x7fde0c06ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde20006f40 tx=0x7fde20006e90 comp rx=0 tx=0).stop 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.753+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 msgr2=0x7fde3015e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.753+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fde18037040 tx=0x7fde18004a40 comp rx=0 tx=0).stop 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 shutdown_connections 2024-04-17T07:32:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fde0c06ca40 0x7fde0c06ef00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fde30145370 0x7fde3015f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde300b4640 0x7fde3015e6b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 --2- 172.21.15.67:0/2966770337 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3006be00 0x7fde3013d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 >> 172.21.15.67:0/2966770337 conn(0x7fde300b6d80 msgr2=0x7fde30143960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 shutdown_connections 2024-04-17T07:32:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.754+0000 7fde35c00640 1 -- 172.21.15.67:0/2966770337 wait complete. 2024-04-17T07:32:40.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:32:40.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.996+0000 7ff7ad200640 1 Processor -- start 2024-04-17T07:32:40.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.997+0000 7ff7ad200640 1 -- start start 2024-04-17T07:32:40.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.998+0000 7ff7ad200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff7a806f670 0x7ff7a806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.998+0000 7ff7ad200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7a80cd020 0x7ff7a80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.998+0000 7ff7ad200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a806ff90 con 0x7ff7a8071d10 2024-04-17T07:32:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.998+0000 7ff7ad200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a80ceb00 con 0x7ff7a80cd020 2024-04-17T07:32:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.998+0000 7ff7ad200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a80cede0 con 0x7ff7a806f670 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.999+0000 7ff79d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7a80cd020 0x7ff7a80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.999+0000 7ff79d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7a80cd020 0x7ff7a80cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32820/0 (socket says 172.21.15.67:32820) 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.999+0000 7ff79d400640 1 -- 172.21.15.67:0/1879683545 learned_addr learned my addr 172.21.15.67:0/1879683545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:40.999+0000 7ff7a5e00640 1 --2- 172.21.15.67:0/1879683545 >> v2:172.21.15.187:3300/0 conn(0x7ff7a806f670 0x7ff7a806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.000+0000 7ff79d400640 1 -- 172.21.15.67:0/1879683545 >> v2:172.21.15.187:3300/0 conn(0x7ff7a806f670 msgr2=0x7ff7a806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.000+0000 7ff79d400640 1 --2- 172.21.15.67:0/1879683545 >> v2:172.21.15.187:3300/0 conn(0x7ff7a806f670 0x7ff7a806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.000+0000 7ff79d400640 1 -- 172.21.15.67:0/1879683545 >> v1:172.21.15.187:6789/0 conn(0x7ff7a8071d10 legacy=0x7ff7a8072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.000+0000 7ff79d400640 1 -- 172.21.15.67:0/1879683545 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a80cf0c0 con 0x7ff7a80cd020 2024-04-17T07:32:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.001+0000 7ff79d400640 1 --2- 172.21.15.67:0/1879683545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a80cd020 0x7ff7a80cd480 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff7a806c910 tx=0x7ff79002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b54bf58a474285 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.001+0000 7ff7a5400640 1 -- 172.21.15.67:0/1879683545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790034070 con 0x7ff7a80cd020 2024-04-17T07:32:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.001+0000 7ff7a5400640 1 -- 172.21.15.67:0/1879683545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff790030860 con 0x7ff7a80cd020 2024-04-17T07:32:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.001+0000 7ff7a5400640 1 -- 172.21.15.67:0/1879683545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790030c30 con 0x7ff7a80cd020 2024-04-17T07:32:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 -- 172.21.15.67:0/1879683545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a80cd020 msgr2=0x7ff7a80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 --2- 172.21.15.67:0/1879683545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a80cd020 0x7ff7a80cd480 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff7a806c910 tx=0x7ff79002b9a0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 -- 172.21.15.67:0/1879683545 shutdown_connections 2024-04-17T07:32:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 --2- 172.21.15.67:0/1879683545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a80cd020 0x7ff7a80cd480 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 --2- 172.21.15.67:0/1879683545 >> v2:172.21.15.187:3300/0 conn(0x7ff7a806f670 0x7ff7a806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 -- 172.21.15.67:0/1879683545 >> 172.21.15.67:0/1879683545 conn(0x7ff7a806bf50 msgr2=0x7ff7a80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 -- 172.21.15.67:0/1879683545 shutdown_connections 2024-04-17T07:32:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.002+0000 7ff7ad200640 1 -- 172.21.15.67:0/1879683545 wait complete. 2024-04-17T07:32:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.004+0000 7ff7ad200640 1 Processor -- start 2024-04-17T07:32:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.004+0000 7ff7ad200640 1 -- start start 2024-04-17T07:32:41.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 0x7ff7a814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 0x7ff7a814a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a80ce720 con 0x7ff7a806f670 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a80cf680 con 0x7ff7a80cd020 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7ad200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a80ce8a0 con 0x7ff7a8071d10 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32826/0 (socket says 172.21.15.67:32826) 2024-04-17T07:32:41.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.005+0000 7ff7a5e00640 1 -- 172.21.15.67:0/2337265404 learned_addr learned my addr 172.21.15.67:0/2337265404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a6800640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 0x7ff7a814a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff79d400640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 0x7ff7a814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 -- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 msgr2=0x7ff7a814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 0x7ff7a814a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 -- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 msgr2=0x7ff7a814a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 0x7ff7a814a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 -- 172.21.15.67:0/2337265404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a80b3320 con 0x7ff7a806f670 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff79d400640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 0x7ff7a814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a6800640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 0x7ff7a814a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff7a5e00640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e8a0 tx=0x7ff79800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.006+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798017070 con 0x7ff7a806f670 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.007+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79800c940 con 0x7ff7a806f670 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.007+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79800cd30 con 0x7ff7a806f670 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.007+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a815a530 con 0x7ff7a806f670 2024-04-17T07:32:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.007+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a815a7f0 con 0x7ff7a806f670 2024-04-17T07:32:41.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.008+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a8068b90 con 0x7ff7a806f670 2024-04-17T07:32:41.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.009+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7ff79801a070 con 0x7ff7a806f670 2024-04-17T07:32:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.010+0000 7ff79ea00640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 0x7ff78406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.010+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7ff79808bc90 con 0x7ff7a806f670 2024-04-17T07:32:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.010+0000 7ff79d400640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 0x7ff78406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.012+0000 7ff79d400640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 0x7ff78406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.014+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff798059f10 con 0x7ff7a806f670 2024-04-17T07:32:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.264+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7a80b6690 con 0x7ff7a806f670 2024-04-17T07:32:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.265+0000 7ff79ea00640 1 -- 172.21.15.67:0/2337265404 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7ff798059810 con 0x7ff7a806f670 2024-04-17T07:32:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:41.266 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:32:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:32:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.268+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 msgr2=0x7ff78406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.268+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 0x7ff78406eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003b040 comp rx=0 tx=0).stop 2024-04-17T07:32:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.268+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 msgr2=0x7ff7a815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.268+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff79800e8a0 tx=0x7ff79800eca0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 shutdown_connections 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7ff78406ca10 0x7ff78406eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7a80cd020 0x7ff7a814a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7a8071d10 0x7ff7a814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 --2- 172.21.15.67:0/2337265404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7a806f670 0x7ff7a815ad10 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 >> 172.21.15.67:0/2337265404 conn(0x7ff7a806bf50 msgr2=0x7ff7a8072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 shutdown_connections 2024-04-17T07:32:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.269+0000 7ff7ad200640 1 -- 172.21.15.67:0/2337265404 wait complete. 2024-04-17T07:32:41.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:32:34.587190+0000', 'modified': '2024-04-17T07:32:37.647893+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7899}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7899': {'gid': 7899, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1617335847', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 1617335847}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1617335847}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7899, 'qdb_cluster': [7899]} 2024-04-17T07:32:41.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:32:41.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T07:32:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.917+0000 7f240fe00640 1 Processor -- start 2024-04-17T07:32:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.917+0000 7f240fe00640 1 -- start start 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f2408071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24080cea70 con 0x7f2408145ac0 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24080ced50 con 0x7f24080ce3b0 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24080ce840 con 0x7f2408072d80 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2408145ac0 0x7f2408072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46318/0 (socket says 172.21.15.67:46318) 2024-04-17T07:32:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.919+0000 7f240d400640 1 -- 172.21.15.67:0/2114797310 learned_addr learned my addr 172.21.15.67:0/2114797310 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723573063 0 0) 0x7f24080cea70 con 0x7f2408145ac0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f0003520 con 0x7f2408145ac0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f240ca00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f2408071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 897459256 0 0) 0x7f23f0003520 con 0x7f2408145ac0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 msgr2=0x7f2408071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f2408071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 >> v1:172.21.15.187:6789/0 conn(0x7f24080ce3b0 legacy=0x7f24080cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24080cf030 con 0x7f2408145ac0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f23fc001670 con 0x7f2408145ac0 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 >> v1:172.21.15.67:6789/0 conn(0x7f2408145ac0 legacy=0x7f2408072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f23f00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f23f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f0004d40 0x7f23f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23f000bb50 con 0x7f23f0004010 2024-04-17T07:32:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f23f000bd00 con 0x7f23f0004d40 2024-04-17T07:32:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.920+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23f000bf00 con 0x7f2408072d80 2024-04-17T07:32:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f23f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240d400640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f23f00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240de00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f0004d40 0x7f23f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 -- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 msgr2=0x7f23f00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2408072d80 0x7f23f00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 -- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f0004d40 msgr2=0x7f23f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23f0004d40 0x7f23f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.921+0000 7f240ca00640 1 -- 172.21.15.67:0/2114797310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f0003520 con 0x7f23f0004010 2024-04-17T07:32:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.923+0000 7f240ca00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f23f0004800 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f23f400ba30 tx=0x7f23f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a02ac8e03453af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.923+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4036020 con 0x7f23f0004010 2024-04-17T07:32:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4032640 con 0x7f23f0004010 2024-04-17T07:32:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f2407e00640 1 -- 172.21.15.67:0/2114797310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f400b600 con 0x7f23f0004010 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 -- 172.21.15.67:0/2114797310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 msgr2=0x7f23f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 --2- 172.21.15.67:0/2114797310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f23f0004800 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f23f400ba30 tx=0x7f23f0005830 comp rx=0 tx=0).stop 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240de00640 1 -- 172.21.15.67:0/2114797310 reap_dead start 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 -- 172.21.15.67:0/2114797310 shutdown_connections 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 -- 172.21.15.67:0/2114797310 >> 172.21.15.67:0/2114797310 conn(0x7f2408067ee0 msgr2=0x7f240806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 -- 172.21.15.67:0/2114797310 shutdown_connections 2024-04-17T07:32:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.924+0000 7f240fe00640 1 -- 172.21.15.67:0/2114797310 wait complete. 2024-04-17T07:32:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.926+0000 7f240fe00640 1 Processor -- start 2024-04-17T07:32:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.926+0000 7f240fe00640 1 -- start start 2024-04-17T07:32:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f240813f3d0 0x7f24081387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2408140e30 0x7f24081412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2408145ea0 con 0x7f23f0004010 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f24080cf5f0 con 0x7f2408140e30 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2408071ea0 con 0x7f240813f3d0 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32848/0 (socket says 172.21.15.67:32848) 2024-04-17T07:32:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.927+0000 7f240d400640 1 -- 172.21.15.67:0/571461796 learned_addr learned my addr 172.21.15.67:0/571461796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 -- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f240813f3d0 msgr2=0x7f24081387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f240813f3d0 0x7f24081387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 -- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2408140e30 msgr2=0x7f24081412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2408140e30 0x7f24081412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 -- 172.21.15.67:0/571461796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240813bdd0 con 0x7f23f0004010 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f240d400640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e7e0 tx=0x7f23fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f2405600640 1 -- 172.21.15.67:0/571461796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00c870 con 0x7f23f0004010 2024-04-17T07:32:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f2405600640 1 -- 172.21.15.67:0/571461796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc018070 con 0x7f23f0004010 2024-04-17T07:32:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.928+0000 7f2405600640 1 -- 172.21.15.67:0/571461796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00cc50 con 0x7f23f0004010 2024-04-17T07:32:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.929+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240813ddf0 con 0x7f23f0004010 2024-04-17T07:32:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.929+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240813e3b0 con 0x7f23f0004010 2024-04-17T07:32:41.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.931+0000 7f2405600640 1 -- 172.21.15.67:0/571461796 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f23fc01d460 con 0x7f23f0004010 2024-04-17T07:32:41.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.933+0000 7f2405600640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 0x7f23e806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.933+0000 7f2405600640 1 -- 172.21.15.67:0/571461796 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f23fc08baf0 con 0x7f23f0004010 2024-04-17T07:32:41.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.933+0000 7f2412c0b740 1 Processor -- start 2024-04-17T07:32:41.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.934+0000 7f2412c0b740 1 -- start start 2024-04-17T07:32:41.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.934+0000 7f240ca00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 0x7f23e806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33e0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ecd32fe980 con 0x55ecd33bfda0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ecd32a5220 con 0x55ecd33dfda0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ecd32fa940 con 0x55ecd33d41b0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32860/0 (socket says 172.21.15.67:32860) 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240ca00640 1 -- 172.21.15.67:0/3190106366 learned_addr learned my addr 172.21.15.67:0/3190106366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240ca00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 0x7f23e806ef50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f23f400ba00 tx=0x7f23f4032000 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240de00640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.935+0000 7f240d400640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33e0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 -- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 msgr2=0x55ecd33c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 -- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 msgr2=0x55ecd33e0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33e0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 -- 172.21.15.67:0/3190106366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd30112d0 con 0x55ecd33bfda0 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240d400640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33e0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:41.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f240ca00640 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f23f402bf90 tx=0x7f23f4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e31be6deef08fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f23f9c00640 1 -- 172.21.15.67:0/3190106366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4041070 con 0x55ecd33bfda0 2024-04-17T07:32:41.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f23f9c00640 1 -- 172.21.15.67:0/3190106366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4032cf0 con 0x55ecd33bfda0 2024-04-17T07:32:41.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.936+0000 7f23f9c00640 1 -- 172.21.15.67:0/3190106366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4036040 con 0x55ecd33bfda0 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 -- 172.21.15.67:0/3190106366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 msgr2=0x55ecd33d3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f23f402bf90 tx=0x7f23f4030000 comp rx=0 tx=0).stop 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 -- 172.21.15.67:0/3190106366 shutdown_connections 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33bfda0 0x55ecd33d3c70 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 --2- 172.21.15.67:0/3190106366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33e0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 -- 172.21.15.67:0/3190106366 >> 172.21.15.67:0/3190106366 conn(0x55ecd3394ba0 msgr2=0x55ecd33dd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 -- 172.21.15.67:0/3190106366 shutdown_connections 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.937+0000 7f2412c0b740 1 -- 172.21.15.67:0/3190106366 wait complete. 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.938+0000 7f2412c0b740 1 -- start start 2024-04-17T07:32:41.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.938+0000 7f2412c0b740 1 Processor -- start 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33c0700 0x55ecd33c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ecd32b8590 con 0x55ecd33dfda0 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ecd33d4610 con 0x55ecd33c0700 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ecd33c4770 con 0x55ecd33d41b0 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f2412c0b740 10 client.0 _ll_register_callbacks cb 0x55ecd3010ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.939+0000 7f240ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32864/0 (socket says 172.21.15.67:32864) 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 -- 172.21.15.67:0/4212940663 learned_addr learned my addr 172.21.15.67:0/4212940663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240d400640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 -- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 msgr2=0x55ecd33c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 -- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33c0700 msgr2=0x55ecd33c3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240de00640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33c0700 0x55ecd33c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33c0700 0x55ecd33c3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 -- 172.21.15.67:0/4212940663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecd30db6b0 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f240ca00640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f23f400bb50 tx=0x7f23f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aebbf13eaeddba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.940+0000 7f23ee000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f405c060 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23f4032cf0 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f23f404f640 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f23f4062050 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 10 client.8016 fetch_fsmap learned FSMap version 61 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x55ecd33c44b0 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4032cf0 con 0x55ecd33dfda0 2024-04-17T07:32:41.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) ==== 3495+0+0 (secure 0 0 0) 0x7f23f4065020 con 0x55ecd33dfda0 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 10 client.8016 fetch_fsmap finished waiting for FSMap version 61 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 10 client.8016 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 20 client.8016 populate_metadata read hostname 'smithi067' 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd33f9b80 0x55ecd33f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 4 client.8016 mds_command: new command op to 7899 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.941+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 --> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecd330c3a0 con 0x55ecd33f9b80 2024-04-17T07:32:41.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.942+0000 7f240d400640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd33f9b80 0x55ecd33f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.942+0000 7f240d400640 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd33f9b80 0x55ecd33f9f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e17e46a5ccaad5c6 server_cookie=3f1a5a3639c8bb48 in_seq=0 out_seq=0 2024-04-17T07:32:41.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.942+0000 7f23ee000640 10 client.8016 ms_handle_connect on v2:172.21.15.67:6834/1617335847 2024-04-17T07:32:41.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.943+0000 7f23ee000640 1 -- 172.21.15.67:0/4212940663 <== mds.0 v2:172.21.15.67:6834/1617335847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecd330c3a0 con 0x55ecd33f9b80 2024-04-17T07:32:41.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.943+0000 7f23ee000640 10 client.8016 handle_command_reply: tid=0 2024-04-17T07:32:41.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.943+0000 7f2412c0b740 1 client.8016 shutdown 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.943+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 msgr2=0x55ecd33bf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.943+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f23f400bb50 tx=0x7f23f40046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 shutdown_connections 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd33f9b80 0x55ecd33f9f60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33c0700 0x55ecd33c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33dfda0 0x55ecd33bf1c0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4212940663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 >> 172.21.15.67:0/4212940663 conn(0x55ecd33bfda0 msgr2=0x55ecd33ded30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f23ee000640 0 client.8016 ms_handle_reset on v2:172.21.15.67:6834/1617335847 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 shutdown_connections 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 1 -- 172.21.15.67:0/4212940663 wait complete. 2024-04-17T07:32:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.944+0000 7f2412c0b740 20 client.8016 trim_cache size 0 max 16384 2024-04-17T07:32:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.964+0000 7f2412c0b740 1 Processor -- start 2024-04-17T07:32:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.964+0000 7f2412c0b740 1 -- start start 2024-04-17T07:32:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 0x55ecd3391d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33dfda0 0x55ecd33e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f240806f0c0 con 0x55ecd33c0700 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2408139f40 con 0x55ecd33d41b0 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f240813a590 con 0x55ecd33dfda0 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.965+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32870/0 (socket says 172.21.15.67:32870) 2024-04-17T07:32:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 -- 172.21.15.67:0/21108314 learned_addr learned my addr 172.21.15.67:0/21108314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240de00640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33dfda0 0x55ecd33e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 -- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33dfda0 msgr2=0x55ecd33e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240ca00640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 0x55ecd3391d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33dfda0 0x55ecd33e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 -- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 msgr2=0x55ecd3391d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 0x55ecd3391d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 -- 172.21.15.67:0/21108314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd33b9c30 con 0x55ecd33c0700 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240ca00640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 0x55ecd3391d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f240d400640 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f23fc059480 tx=0x7f23fc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c036865e768846d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.966+0000 7f23f9200640 1 -- 172.21.15.67:0/21108314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc075020 con 0x55ecd33c0700 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f23f9200640 1 -- 172.21.15.67:0/21108314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc05db30 con 0x55ecd33c0700 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f23f9200640 1 -- 172.21.15.67:0/21108314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc062360 con 0x55ecd33c0700 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 -- 172.21.15.67:0/21108314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 msgr2=0x55ecd3391800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f23fc059480 tx=0x7f23fc00cf20 comp rx=0 tx=0).stop 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 -- 172.21.15.67:0/21108314 shutdown_connections 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33dfda0 0x55ecd33e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33d41b0 0x55ecd3391d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 --2- 172.21.15.67:0/21108314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd3391800 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 -- 172.21.15.67:0/21108314 >> 172.21.15.67:0/21108314 conn(0x55ecd33bfda0 msgr2=0x55ecd3409fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 -- 172.21.15.67:0/21108314 shutdown_connections 2024-04-17T07:32:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.967+0000 7f2412c0b740 1 -- 172.21.15.67:0/21108314 wait complete. 2024-04-17T07:32:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.968+0000 7f2412c0b740 1 -- start start 2024-04-17T07:32:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.968+0000 7f2412c0b740 1 Processor -- start 2024-04-17T07:32:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33deba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ecd33b9ab0 con 0x55ecd33c0700 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ecd32b8590 con 0x55ecd33dfda0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ecd33bac10 con 0x55ecd33d41b0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f2412c0b740 10 client.0 _ll_register_callbacks cb 0x7f23f000baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.969+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32874/0 (socket says 172.21.15.67:32874) 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 -- 172.21.15.67:0/4010064943 learned_addr learned my addr 172.21.15.67:0/4010064943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 -- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 msgr2=0x55ecd33deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33deba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240de00640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 -- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 msgr2=0x55ecd33df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33df0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 -- 172.21.15.67:0/4010064943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecd3393600 con 0x55ecd33c0700 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f240d400640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f23fc059480 tx=0x7f23fc001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7183d3d946599b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f23ee000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc087070 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f23fc058bd0 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.970+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23fc05db30 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f23fc075050 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 10 client.8022 fetch_fsmap learned FSMap version 61 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x55ecd33c0d30 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc05db30 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 61) ==== 3495+0+0 (secure 0 0 0) 0x7f23fc0554d0 con 0x55ecd33c0700 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 10 client.8022 fetch_fsmap finished waiting for FSMap version 61 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 10 client.8022 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 20 client.8022 populate_metadata read hostname 'smithi067' 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd3396890 0x55ecd3408d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 4 client.8022 mds_command: new command op to 7899 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 --> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] -- command(tid 0: {"prefix": "status"}) -- 0x55ecd33e0860 con 0x55ecd3396890 2024-04-17T07:32:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.971+0000 7f240ca00640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd3396890 0x55ecd3408d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f240ca00640 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd3396890 0x55ecd3408d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3a508c2a133794e server_cookie=2f4e9397623de41e in_seq=0 out_seq=0 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f23ee000640 10 client.8022 ms_handle_connect on v2:172.21.15.67:6834/1617335847 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f23ee000640 1 -- 172.21.15.67:0/4010064943 <== mds.0 v2:172.21.15.67:6834/1617335847 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ecd33e0860 con 0x55ecd3396890 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f23ee000640 10 client.8022 handle_command_reply: tid=0 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 client.8022 shutdown 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 msgr2=0x55ecd345e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f23fc059480 tx=0x7f23fc001b50 comp rx=0 tx=0).stop 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 shutdown_connections 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:6834/1617335847,v1:172.21.15.67:6835/1617335847] conn(0x55ecd3396890 0x55ecd3408d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ecd33dfda0 0x55ecd33df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ecd33d41b0 0x55ecd33deba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 --2- 172.21.15.67:0/4010064943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecd33c0700 0x55ecd345e420 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f23ee000640 0 client.8022 ms_handle_reset on v2:172.21.15.67:6834/1617335847 2024-04-17T07:32:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.972+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 >> 172.21.15.67:0/4010064943 conn(0x55ecd33bfda0 msgr2=0x55ecd33dd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 shutdown_connections 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f2412c0b740 1 -- 172.21.15.67:0/4010064943 wait complete. 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f2412c0b740 20 client.8022 trim_cache size 0 max 16384 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7899, 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8376510059999998, 2024-04-17T07:32:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 61, 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 127, 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 127, 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.203307081 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 msgr2=0x7f23e806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 0x7f23e806ef50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f23f400ba00 tx=0x7f23f4032000 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 msgr2=0x7f240813d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.973+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e7e0 tx=0x7f23fc00eca0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 shutdown_connections 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f23e806ca90 0x7f23e806ef50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2408140e30 0x7f24081412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f240813f3d0 0x7f24081387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 --2- 172.21.15.67:0/571461796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23f0004010 0x7f240813d780 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 >> 172.21.15.67:0/571461796 conn(0x7f2408067ee0 msgr2=0x7f240815fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 shutdown_connections 2024-04-17T07:32:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:41.974+0000 7f240fe00640 1 -- 172.21.15.67:0/571461796 wait complete. 2024-04-17T07:32:41.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:32:42.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.200+0000 7f47d5400640 1 Processor -- start 2024-04-17T07:32:42.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.200+0000 7f47d5400640 1 -- start start 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.201+0000 7f47d5400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f47d0160280 0x7f47d006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.201+0000 7f47d5400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d00cd4a0 con 0x7f47d0141ed0 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.201+0000 7f47d5400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d01607f0 con 0x7f47d006b8c0 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.201+0000 7f47d5400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f47d015ce30 con 0x7f47d0160280 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.202+0000 7f47d4a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47d0141ed0 0x7f47d015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46324/0 (socket says 172.21.15.67:46324) 2024-04-17T07:32:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.202+0000 7f47d4a00640 1 -- 172.21.15.67:0/570742 learned_addr learned my addr 172.21.15.67:0/570742 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.202+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314570418 0 0) 0x7f47d00cd4a0 con 0x7f47d0141ed0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.202+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b4003520 con 0x7f47d0141ed0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.202+0000 7f47cf400640 1 --2- 172.21.15.67:0/570742 >> v2:172.21.15.187:3300/0 conn(0x7f47d0160280 0x7f47d006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4160603966 0 0) 0x7f47b4003520 con 0x7f47d0141ed0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 >> v2:172.21.15.187:3300/0 conn(0x7f47d0160280 msgr2=0x7f47d006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 --2- 172.21.15.67:0/570742 >> v2:172.21.15.187:3300/0 conn(0x7f47d0160280 0x7f47d006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 >> v1:172.21.15.187:6789/0 conn(0x7f47d006b8c0 legacy=0x7f47d00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d0160ad0 con 0x7f47d0141ed0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f47c80019f0 con 0x7f47d0141ed0 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 >> v1:172.21.15.67:6789/0 conn(0x7f47d0141ed0 legacy=0x7f47d015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 0x7f47b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4004800 0x7f47b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47b40052e0 0x7f47b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47d015ce30 con 0x7f47b4003a70 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47b400c1b0 con 0x7f47b4004800 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47b400c3b0 con 0x7f47b40052e0 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cfe00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 0x7f47b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cfe00640 1 -- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47b40052e0 msgr2=0x7f47b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.203+0000 7f47cfe00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47b40052e0 0x7f47b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.204+0000 7f47cfe00640 1 -- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4004800 msgr2=0x7f47b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.204+0000 7f47cfe00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4004800 0x7f47b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.204+0000 7f47cfe00640 1 -- 172.21.15.67:0/570742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b4003520 con 0x7f47b4003a70 2024-04-17T07:32:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.204+0000 7f47d4a00640 1 -- 172.21.15.67:0/570742 reap_dead start 2024-04-17T07:32:42.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47cfe00640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 0x7f47b4003e70 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f47c0003d80 tx=0x7f47b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b84821342def89c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:42.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0037070 con 0x7f47b4003a70 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c0032600 con 0x7f47b4003a70 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47cea00640 1 -- 172.21.15.67:0/570742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0003980 con 0x7f47b4003a70 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47d5400640 1 -- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 msgr2=0x7f47b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.205+0000 7f47d5400640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 0x7f47b4003e70 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f47c0003d80 tx=0x7f47b4005830 comp rx=0 tx=0).stop 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.206+0000 7f47d5400640 1 -- 172.21.15.67:0/570742 shutdown_connections 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.206+0000 7f47d5400640 1 --2- 172.21.15.67:0/570742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47b4003a70 0x7f47b4003e70 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.206+0000 7f47d5400640 1 -- 172.21.15.67:0/570742 >> 172.21.15.67:0/570742 conn(0x7f47d0067d10 msgr2=0x7f47d00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.206+0000 7f47d5400640 1 -- 172.21.15.67:0/570742 shutdown_connections 2024-04-17T07:32:42.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.206+0000 7f47d5400640 1 -- 172.21.15.67:0/570742 wait complete. 2024-04-17T07:32:42.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.208+0000 7f47d5400640 1 Processor -- start 2024-04-17T07:32:42.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.208+0000 7f47d5400640 1 -- start start 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4003a70 0x7f47d013e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47d006ee40 0x7f47d006f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47d013f9b0 con 0x7f47d015e780 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0161090 con 0x7f47b4003a70 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47d013fb30 con 0x7f47d006ee40 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32884/0 (socket says 172.21.15.67:32884) 2024-04-17T07:32:42.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 -- 172.21.15.67:0/1230681269 learned_addr learned my addr 172.21.15.67:0/1230681269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47d4a00640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47d006ee40 0x7f47d006f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cfe00640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4003a70 0x7f47d013e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 -- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47d006ee40 msgr2=0x7f47d006f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47d006ee40 0x7f47d006f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 -- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4003a70 msgr2=0x7f47d013e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4003a70 0x7f47d013e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.209+0000 7f47cf400640 1 -- 172.21.15.67:0/1230681269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d01f49d0 con 0x7f47d015e780 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47cf400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d560 tx=0x7f47bc00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0125f0 con 0x7f47d015e780 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47bc019070 con 0x7f47d015e780 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d01f4c90 con 0x7f47d015e780 2024-04-17T07:32:42.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0129d0 con 0x7f47d015e780 2024-04-17T07:32:42.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.210+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d01f5250 con 0x7f47d015e780 2024-04-17T07:32:42.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.211+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d0068a10 con 0x7f47d015e780 2024-04-17T07:32:42.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.212+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f47bc012b70 con 0x7f47d015e780 2024-04-17T07:32:42.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.213+0000 7f47c7e00640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 0x7f47b006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:42.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.214+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f47bc08bf10 con 0x7f47d015e780 2024-04-17T07:32:42.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.214+0000 7f47cfe00640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 0x7f47b006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:42.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.215+0000 7f47cfe00640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 0x7f47b006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034000 tx=0x7f47c0009030 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:42.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.218+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f47bc05a220 con 0x7f47d015e780 2024-04-17T07:32:42.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.464+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f47d0140750 con 0x7f47d015e780 2024-04-17T07:32:42.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.465+0000 7f47c7e00640 1 -- 172.21.15.67:0/1230681269 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) ==== 53+0+73 (secure 0 0 0) 0x7f47bc059b20 con 0x7f47d015e780 2024-04-17T07:32:42.465 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:32:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 msgr2=0x7f47b006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 0x7f47b006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034000 tx=0x7f47c0009030 comp rx=0 tx=0).stop 2024-04-17T07:32:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 msgr2=0x7f47d01f8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d560 tx=0x7f47bc00da20 comp rx=0 tx=0).stop 2024-04-17T07:32:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 shutdown_connections 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f47b006ca40 0x7f47b006ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47d006ee40 0x7f47d006f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47d015e780 0x7f47d01f8950 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 --2- 172.21.15.67:0/1230681269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47b4003a70 0x7f47d013e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 >> 172.21.15.67:0/1230681269 conn(0x7f47d0067d10 msgr2=0x7f47d00cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 shutdown_connections 2024-04-17T07:32:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42.468+0000 7f47d5400640 1 -- 172.21.15.67:0/1230681269 wait complete. 2024-04-17T07:32:42.487 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:32:42.487 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:32:42.488 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:32:42.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:42.488 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.546 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41482sec preferred_lft 41482sec 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:32:42.547 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:32:42.548 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:32:42.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:42.549 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:32:42.644 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:32:42.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:42.649 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:32:42.707 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:32:42.707 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:32:42.707 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:32:42.707 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:32:42.707 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:32:42.707 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:32:42.707 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:32:42.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:32:42.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:32:42.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:32:42.708 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:32:42.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:32:42.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:32:42.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:42.967 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:32:43.021 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:32:43.021 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:32:43.022 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:32:43.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:43.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:32:43.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:32:43.123 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:32:43.184 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:32:43.184 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:32:43.184 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:32:43.234 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:32:43.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:43.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:32:43.377 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:32:43.377 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:32:43.416 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:32:43.416 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:32:43.416 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:32:43.416 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:32:43.416 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:32:43.418 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:32:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:32:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:32:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:32:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:32:43.424 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:32:43.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:43.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:32:43.509 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:32:43.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:32:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:32:43.650 INFO:teuthology.orchestra.run.smithi067.stdout:client6640 2024-04-17T07:32:43.651 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:32:43.651 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:32:43.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:32:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.917+0000 7f1744c00640 1 Processor -- start 2024-04-17T07:32:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.917+0000 7f1744c00640 1 -- start start 2024-04-17T07:32:43.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f1744c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f174015f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f1744c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17400ce510 0x7f17400ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f1744c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17400cb420 con 0x7f17400afbf0 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f1744c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f174015fd60 con 0x7f17400ce510 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f1744c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17400cc030 con 0x7f174015f3c0 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f173f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17400afbf0 0x7f174006e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36158/0 (socket says 172.21.15.67:36158) 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.919+0000 7f173f400640 1 -- 172.21.15.67:0/2239160439 learned_addr learned my addr 172.21.15.67:0/2239160439 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173fe00640 1 --2- 172.21.15.67:0/2239160439 >> v2:172.21.15.67:3300/0 conn(0x7f17400ce510 0x7f17400ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f1737e00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f174015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173fe00640 1 -- 172.21.15.67:0/2239160439 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 msgr2=0x7f174015f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173fe00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f174015f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173fe00640 1 -- 172.21.15.67:0/2239160439 >> v1:172.21.15.67:6789/0 conn(0x7f17400afbf0 legacy=0x7f174006e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173fe00640 1 -- 172.21.15.67:0/2239160439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17400cb700 con 0x7f17400ce510 2024-04-17T07:32:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.920+0000 7f173ea00640 1 -- 172.21.15.67:0/2239160439 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161408764 0 0) 0x7f17400cb420 con 0x7f17400afbf0 2024-04-17T07:32:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.921+0000 7f173fe00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f17400ce9a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f1738009dc0 tx=0x7f173802be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457843531f383cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f173ea00640 1 -- 172.21.15.67:0/2239160439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738039070 con 0x7f17400ce510 2024-04-17T07:32:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f173ea00640 1 -- 172.21.15.67:0/2239160439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f173802cc30 con 0x7f17400ce510 2024-04-17T07:32:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f173ea00640 1 -- 172.21.15.67:0/2239160439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738034440 con 0x7f17400ce510 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 -- 172.21.15.67:0/2239160439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 msgr2=0x7f17400ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f17400ce9a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f1738009dc0 tx=0x7f173802be80 comp rx=0 tx=0).stop 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 -- 172.21.15.67:0/2239160439 shutdown_connections 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f17400ce9a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 --2- 172.21.15.67:0/2239160439 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f174015f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.922+0000 7f1744c00640 1 -- 172.21.15.67:0/2239160439 >> 172.21.15.67:0/2239160439 conn(0x7f174006bef0 msgr2=0x7f17400b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.923+0000 7f1744c00640 1 -- 172.21.15.67:0/2239160439 shutdown_connections 2024-04-17T07:32:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.923+0000 7f1744c00640 1 -- 172.21.15.67:0/2239160439 wait complete. 2024-04-17T07:32:43.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.924+0000 7f1744c00640 1 Processor -- start 2024-04-17T07:32:43.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.925+0000 7f1744c00640 1 -- start start 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.925+0000 7f1744c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17400afbf0 0x7f1740147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1744c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1744c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f1740152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1744c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f174006f090 con 0x7f17400ce510 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1744c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f17400cbcc0 con 0x7f17400afbf0 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1744c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1740160110 con 0x7f174015f3c0 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41922/0 (socket says 172.21.15.67:41922) 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 -- 172.21.15.67:0/3581422458 learned_addr learned my addr 172.21.15.67:0/3581422458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f173f400640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17400afbf0 0x7f1740147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f173fe00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f1740152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 -- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 msgr2=0x7f1740152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:43.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f1740152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 -- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17400afbf0 msgr2=0x7f1740147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17400afbf0 0x7f1740147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.926+0000 7f1737e00640 1 -- 172.21.15.67:0/3581422458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17401466b0 con 0x7f17400ce510 2024-04-17T07:32:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1737e00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f172800d5f0 tx=0x7f172800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17280125f0 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1728019070 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17280129d0 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1740146960 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.927+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1740146f20 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.928+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1740068a10 con 0x7f17400ce510 2024-04-17T07:32:43.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.929+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f1728012b70 con 0x7f17400ce510 2024-04-17T07:32:43.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.930+0000 7f1737400640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 0x7f171c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:43.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.930+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f172808c0e0 con 0x7f17400ce510 2024-04-17T07:32:43.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.931+0000 7f173f400640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 0x7f171c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:43.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.932+0000 7f173f400640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 0x7f171c06eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1730000f60 tx=0x7f1730007b70 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:43.935+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f172805a3f0 con 0x7f17400ce510 2024-04-17T07:32:44.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.169+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 --> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f17400754e0 con 0x7f171c06ca10 2024-04-17T07:32:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.173+0000 7f1737400640 1 -- 172.21.15.67:0/3581422458 <== mgr.6394 v2:172.21.15.187:6832/2517863062 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f17400754e0 con 0x7f171c06ca10 2024-04-17T07:32:44.173 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:32:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:32:44.173 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:32:44.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:32:44.174 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:32:44.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 msgr2=0x7f171c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 0x7f171c06eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1730000f60 tx=0x7f1730007b70 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 msgr2=0x7f1740146170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f172800d5f0 tx=0x7f172800dab0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 shutdown_connections 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f171c06ca10 0x7f171c06eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174015f3c0 0x7f1740152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17400ce510 0x7f1740146170 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 --2- 172.21.15.67:0/3581422458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17400afbf0 0x7f1740147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 >> 172.21.15.67:0/3581422458 conn(0x7f174006bef0 msgr2=0x7f174015e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 shutdown_connections 2024-04-17T07:32:44.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.176+0000 7f1744c00640 1 -- 172.21.15.67:0/3581422458 wait complete. 2024-04-17T07:32:44.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:32:44.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.421+0000 7f0591800640 1 Processor -- start 2024-04-17T07:32:44.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.422+0000 7f0591800640 1 -- start start 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.423+0000 7f0591800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f058c06daf0 0x7f058c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.423+0000 7f0591800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c09b170 0x7f058c070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.423+0000 7f0591800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c0d1520 con 0x7f058c0d1060 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.423+0000 7f0591800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f058c0d1800 con 0x7f058c06daf0 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.423+0000 7f0591800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f058c0d1ae0 con 0x7f058c09b170 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f058c06daf0 0x7f058c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f058c06daf0 0x7f058c0d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41930/0 (socket says 172.21.15.67:41930) 2024-04-17T07:32:44.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 -- 172.21.15.67:0/3799784609 learned_addr learned my addr 172.21.15.67:0/3799784609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058b400640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c09b170 0x7f058c070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 -- 172.21.15.67:0/3799784609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c09b170 msgr2=0x7f058c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c09b170 0x7f058c070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 -- 172.21.15.67:0/3799784609 >> v1:172.21.15.187:6789/0 conn(0x7f058c0d1060 legacy=0x7f058c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.424+0000 7f058aa00640 1 -- 172.21.15.67:0/3799784609 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c0d1dc0 con 0x7f058c06daf0 2024-04-17T07:32:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.425+0000 7f058aa00640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c06daf0 0x7f058c0d0b20 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0574009870 tx=0x7f057402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d410d7b5ed2478 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.426+0000 7f0589600640 1 -- 172.21.15.67:0/3799784609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0574034070 con 0x7f058c06daf0 2024-04-17T07:32:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.426+0000 7f0589600640 1 -- 172.21.15.67:0/3799784609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0574030840 con 0x7f058c06daf0 2024-04-17T07:32:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.426+0000 7f0589600640 1 -- 172.21.15.67:0/3799784609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0574030c10 con 0x7f058c06daf0 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 -- 172.21.15.67:0/3799784609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c06daf0 msgr2=0x7f058c0d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c06daf0 0x7f058c0d0b20 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0574009870 tx=0x7f057402b980 comp rx=0 tx=0).stop 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 -- 172.21.15.67:0/3799784609 shutdown_connections 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c09b170 0x7f058c070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 --2- 172.21.15.67:0/3799784609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c06daf0 0x7f058c0d0b20 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 -- 172.21.15.67:0/3799784609 >> 172.21.15.67:0/3799784609 conn(0x7f058c0b41a0 msgr2=0x7f058c0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 -- 172.21.15.67:0/3799784609 shutdown_connections 2024-04-17T07:32:44.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.427+0000 7f0591800640 1 -- 172.21.15.67:0/3799784609 wait complete. 2024-04-17T07:32:44.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.429+0000 7f0591800640 1 Processor -- start 2024-04-17T07:32:44.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.429+0000 7f0591800640 1 -- start start 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c06daf0 0x7f058c154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f058c09b170 0x7f058c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f058c06e050 con 0x7f058c0d1060 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f058c06ded0 con 0x7f058c09b170 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f0591800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f058c06ce00 con 0x7f058c06daf0 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f058b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f058b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41932/0 (socket says 172.21.15.67:41932) 2024-04-17T07:32:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.430+0000 7f058b400640 1 -- 172.21.15.67:0/1573573089 learned_addr learned my addr 172.21.15.67:0/1573573089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058aa00640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c06daf0 0x7f058c154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058a000640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f058c09b170 0x7f058c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 -- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c06daf0 msgr2=0x7f058c154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c06daf0 0x7f058c154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 -- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f058c09b170 msgr2=0x7f058c070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f058c09b170 0x7f058c070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:44.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 -- 172.21.15.67:0/1573573089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c0c8580 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.431+0000 7f058b400640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f058400d5d0 tx=0x7f058400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.432+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05840125f0 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.432+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0584019070 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.432+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05840129d0 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.432+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c0c8860 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.432+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c0b57e0 con 0x7f058c0d1060 2024-04-17T07:32:44.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.433+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c06fb90 con 0x7f058c0d1060 2024-04-17T07:32:44.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.434+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 57) ==== 91142+0+0 (secure 0 0 0) 0x7f0584012b70 con 0x7f058c0d1060 2024-04-17T07:32:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.435+0000 7f0582a00640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 0x7f056806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.435+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f058408c1a0 con 0x7f058c0d1060 2024-04-17T07:32:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.436+0000 7f058aa00640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 0x7f056806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:44.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.437+0000 7f058aa00640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 0x7f056806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0574036000 tx=0x7f057403b040 comp rx=0 tx=0).ready entity=mgr.6394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:44.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.439+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f058c06fb90 con 0x7f058c0d1060 2024-04-17T07:32:44.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:44.696+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f058c068ca0 con 0x7f058c0d1060 2024-04-17T07:32:45.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.212+0000 7f0582a00640 1 -- 172.21.15.67:0/1573573089 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) ==== 92+0+0 (secure 0 0 0) 0x7f058405a420 con 0x7f058c0d1060 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 msgr2=0x7f056806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 0x7f056806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0574036000 tx=0x7f057403b040 comp rx=0 tx=0).stop 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 msgr2=0x7f058c0714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f058400d5d0 tx=0x7f058400da90 comp rx=0 tx=0).stop 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 shutdown_connections 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f056806ca10 0x7f056806eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0d1060 0x7f058c0714d0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f058c09b170 0x7f058c070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 --2- 172.21.15.67:0/1573573089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f058c06daf0 0x7f058c154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.216+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 >> 172.21.15.67:0/1573573089 conn(0x7f058c0b41a0 msgr2=0x7f058c0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:45.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.217+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 shutdown_connections 2024-04-17T07:32:45.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.217+0000 7f0591800640 1 -- 172.21.15.67:0/1573573089 wait complete. 2024-04-17T07:32:45.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:32:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.498+0000 7f7cbe800640 1 Processor -- start 2024-04-17T07:32:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.498+0000 7f7cbe800640 1 -- start start 2024-04-17T07:32:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbe800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7cb815ecb0 0x7f7cb80d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbe800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbe800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb80d00b0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbe800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb80d0c30 con 0x7f7cb815ecb0 2024-04-17T07:32:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbe800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb80d0390 con 0x7f7cb80d3210 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7cb815ecb0 0x7f7cb80d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbde00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7cb80cf8e0 0x7f7cb80d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36182/0 (socket says 172.21.15.67:36182) 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.500+0000 7f7cbde00640 1 -- 172.21.15.67:0/3953172361 learned_addr learned my addr 172.21.15.67:0/3953172361 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.501+0000 7f7cbca00640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.501+0000 7f7cbd400640 1 -- 172.21.15.67:0/3953172361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 msgr2=0x7f7cb80cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.501+0000 7f7cbd400640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.501+0000 7f7cbd400640 1 -- 172.21.15.67:0/3953172361 >> v1:172.21.15.67:6789/0 conn(0x7f7cb80cf8e0 legacy=0x7f7cb80d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.501+0000 7f7cbd400640 1 -- 172.21.15.67:0/3953172361 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb80d0670 con 0x7f7cb815ecb0 2024-04-17T07:32:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.502+0000 7f7cbd400640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80d2cd0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80d0db0 tx=0x7f7cac02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33a1209e606c7dd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.502+0000 7f7cb7e00640 1 -- 172.21.15.67:0/3953172361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031780 con 0x7f7cb815ecb0 2024-04-17T07:32:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.502+0000 7f7cb7e00640 1 -- 172.21.15.67:0/3953172361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cac037070 con 0x7f7cb815ecb0 2024-04-17T07:32:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.502+0000 7f7cb7e00640 1 -- 172.21.15.67:0/3953172361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac031b40 con 0x7f7cb815ecb0 2024-04-17T07:32:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 -- 172.21.15.67:0/3953172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb815ecb0 msgr2=0x7f7cb80d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80d2cd0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80d0db0 tx=0x7f7cac02b820 comp rx=0 tx=0).stop 2024-04-17T07:32:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 -- 172.21.15.67:0/3953172361 shutdown_connections 2024-04-17T07:32:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 --2- 172.21.15.67:0/3953172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80d2cd0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 -- 172.21.15.67:0/3953172361 >> 172.21.15.67:0/3953172361 conn(0x7f7cb808bff0 msgr2=0x7f7cb80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 -- 172.21.15.67:0/3953172361 shutdown_connections 2024-04-17T07:32:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.503+0000 7f7cbe800640 1 -- 172.21.15.67:0/3953172361 wait complete. 2024-04-17T07:32:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.505+0000 7f7cbe800640 1 Processor -- start 2024-04-17T07:32:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.505+0000 7f7cbe800640 1 -- start start 2024-04-17T07:32:45.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb815f090 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb80cfd70 con 0x7f7cb815ecb0 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbe800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb815f210 con 0x7f7cb80d3210 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.506+0000 7f7cbd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41964/0 (socket says 172.21.15.67:41964) 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 -- 172.21.15.67:0/460091212 learned_addr learned my addr 172.21.15.67:0/460091212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 msgr2=0x7f7cb80b8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 msgr2=0x7f7cb80b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbde00640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 -- 172.21.15.67:0/460091212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb80b92c0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbde00640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.507+0000 7f7cbd400640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.508+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032820 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.508+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cac0329e0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.508+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032dd0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.508+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb80b95a0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.508+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb80b9b60 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.509+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb815ff80 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.510+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 58) ==== 91159+0+0 (secure 0 0 0) 0x7f7cac04c060 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.511+0000 7f7cb5600640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 0x7f7c9806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.511+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f7cac0ad5c0 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.512+0000 7f7cbca00640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 msgr2=0x7f7c9806ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:45.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.512+0000 7f7cbca00640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 0x7f7c9806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:45.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.520+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7cac040070 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.713+0000 7f7cbca00640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 msgr2=0x7f7c9806ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.713+0000 7f7cbca00640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 0x7f7c9806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.724+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7cb8068a10 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.731+0000 7f7cb5600640 1 -- 172.21.15.67:0/460091212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) ==== 138+0+0 (secure 0 0 0) 0x7f7cac02f050 con 0x7f7cb80cf8e0 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.733+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 msgr2=0x7f7c9806ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.733+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 0x7f7c9806ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.733+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 msgr2=0x7f7cb80b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.733+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 shutdown_connections 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f7c9806cab0 0x7f7c9806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7cb815ecb0 0x7f7cb80b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7cb80d3210 0x7f7cb80b8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 --2- 172.21.15.67:0/460091212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb80cf8e0 0x7f7cb80b42e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 >> 172.21.15.67:0/460091212 conn(0x7f7cb808bff0 msgr2=0x7f7cb806d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 shutdown_connections 2024-04-17T07:32:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.734+0000 7f7cbe800640 1 -- 172.21.15.67:0/460091212 wait complete. 2024-04-17T07:32:45.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:32:45.780 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:45.780+0000 7f4c8003e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:32:45.834 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:45.834+0000 7f7a08242040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:32:45.834 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:45.834+0000 7f4c8003e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:32:45.893 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:45.892+0000 7f7a08242040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:32:45.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.963+0000 7fb919c00640 1 Processor -- start 2024-04-17T07:32:45.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.964+0000 7fb919c00640 1 -- start start 2024-04-17T07:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb91409aa60 0x7fb91409ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 0x7fb914160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb91415d350 con 0x7fb91409aa60 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb9140d0c20 con 0x7fb9141605a0 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb919c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb9140cd630 con 0x7fb9140d0280 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb912a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb91409aa60 0x7fb91409ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb912a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb91409aa60 0x7fb91409ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41978/0 (socket says 172.21.15.67:41978) 2024-04-17T07:32:45.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.965+0000 7fb912a00640 1 -- 172.21.15.67:0/2319048221 learned_addr learned my addr 172.21.15.67:0/2319048221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912a00640 1 -- 172.21.15.67:0/2319048221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 msgr2=0x7fb9140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912000640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb913400640 1 --2- 172.21.15.67:0/2319048221 >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 0x7fb914160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912a00640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912a00640 1 -- 172.21.15.67:0/2319048221 >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 msgr2=0x7fb914160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912a00640 1 --2- 172.21.15.67:0/2319048221 >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 0x7fb914160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb912a00640 1 -- 172.21.15.67:0/2319048221 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91409b380 con 0x7fb91409aa60 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.966+0000 7fb913400640 1 --2- 172.21.15.67:0/2319048221 >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 0x7fb914160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.967+0000 7fb912a00640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb91409ae40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb914073ed0 tx=0x7fb90402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc35fc0d47c40a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.967+0000 7fb911600640 1 -- 172.21.15.67:0/2319048221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904035020 con 0x7fb91409aa60 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.967+0000 7fb911600640 1 -- 172.21.15.67:0/2319048221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb904031640 con 0x7fb91409aa60 2024-04-17T07:32:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.967+0000 7fb911600640 1 -- 172.21.15.67:0/2319048221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90402d040 con 0x7fb91409aa60 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 -- 172.21.15.67:0/2319048221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 msgr2=0x7fb91409ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb91409ae40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb914073ed0 tx=0x7fb90402b8f0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 -- 172.21.15.67:0/2319048221 shutdown_connections 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 --2- 172.21.15.67:0/2319048221 >> v2:172.21.15.187:3300/0 conn(0x7fb9141605a0 0x7fb914160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 --2- 172.21.15.67:0/2319048221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb91409ae40 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 -- 172.21.15.67:0/2319048221 >> 172.21.15.67:0/2319048221 conn(0x7fb9140b41a0 msgr2=0x7fb9140b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 -- 172.21.15.67:0/2319048221 shutdown_connections 2024-04-17T07:32:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.968+0000 7fb919c00640 1 -- 172.21.15.67:0/2319048221 wait complete. 2024-04-17T07:32:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.970+0000 7fb919c00640 1 Processor -- start 2024-04-17T07:32:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.970+0000 7fb919c00640 1 -- start start 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9141605a0 0x7fb9140c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140d1420 con 0x7fb91409aa60 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140d1140 con 0x7fb9141605a0 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb919c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb9140d1700 con 0x7fb9140d0280 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41988/0 (socket says 172.21.15.67:41988) 2024-04-17T07:32:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 -- 172.21.15.67:0/1858942981 learned_addr learned my addr 172.21.15.67:0/1858942981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 msgr2=0x7fb9140c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9141605a0 msgr2=0x7fb9140c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9141605a0 0x7fb9140c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.971+0000 7fb912a00640 1 -- 172.21.15.67:0/1858942981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9141e63e0 con 0x7fb91409aa60 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb912a00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb90402b930 tx=0x7fb9040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904040070 con 0x7fb91409aa60 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb904039c10 con 0x7fb91409aa60 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904035040 con 0x7fb91409aa60 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9141e66c0 con 0x7fb91409aa60 2024-04-17T07:32:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.972+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9141e6c80 con 0x7fb91409aa60 2024-04-17T07:32:45.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.973+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9140ccda0 con 0x7fb91409aa60 2024-04-17T07:32:45.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.975+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 58) ==== 91159+0+0 (secure 0 0 0) 0x7fb90404b060 con 0x7fb91409aa60 2024-04-17T07:32:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.975+0000 7fb90aa00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 0x7fb8f406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.976+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7fb9040ad8c0 con 0x7fb91409aa60 2024-04-17T07:32:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.976+0000 7fb912000640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 msgr2=0x7fb8f406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.976+0000 7fb912000640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 0x7fb8f406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:45.980+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb9140ccda0 con 0x7fb91409aa60 2024-04-17T07:32:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.177+0000 7fb912000640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 msgr2=0x7fb8f406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.177+0000 7fb912000640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 0x7fb8f406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:46.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.185+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fb9140b5040 con 0x7fb91409aa60 2024-04-17T07:32:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.190+0000 7fb90aa00640 1 -- 172.21.15.67:0/1858942981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) ==== 133+0+0 (secure 0 0 0) 0x7fb90402f300 con 0x7fb91409aa60 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 msgr2=0x7fb8f406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 0x7fb8f406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 msgr2=0x7fb9141e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fb90402b930 tx=0x7fb9040046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 shutdown_connections 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7fb8f406ca60 0x7fb8f406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9141605a0 0x7fb9140c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9140d0280 0x7fb9140c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 --2- 172.21.15.67:0/1858942981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91409aa60 0x7fb9141e7760 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.193+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 >> 172.21.15.67:0/1858942981 conn(0x7fb9140b41a0 msgr2=0x7fb9140cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:46.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.194+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 shutdown_connections 2024-04-17T07:32:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.194+0000 7fb919c00640 1 -- 172.21.15.67:0/1858942981 wait complete. 2024-04-17T07:32:46.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:46.212 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T07:32:46.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-17T07:32:46.280 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.280+0000 7f4c8003e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:32:46.332 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.331+0000 7f4c8003e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:32:46.371 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.371+0000 7f7a08242040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:32:46.381 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.381+0000 7f4c8003e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:32:46.428 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.428+0000 7f7a08242040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:32:46.442 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.442+0000 7f4c8003e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:32:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.459+0000 7f6a2f600640 1 Processor -- start 2024-04-17T07:32:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.459+0000 7f6a2f600640 1 -- start start 2024-04-17T07:32:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.461+0000 7f6a2f600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.461+0000 7f6a2f600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6a280d0b50 0x7f6a2815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.461+0000 7f6a2f600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a28163710 con 0x7f6a281627b0 2024-04-17T07:32:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.461+0000 7f6a2f600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6a2815f360 con 0x7f6a280d0b50 2024-04-17T07:32:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.461+0000 7f6a2f600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2806c1e0 con 0x7f6a2806b8c0 2024-04-17T07:32:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a2d600640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6a281627b0 0x7f6a280cdf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:59252/0 (socket says 172.21.15.67:59252) 2024-04-17T07:32:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a2d600640 1 -- 172.21.15.67:0/3068059290 learned_addr learned my addr 172.21.15.67:0/3068059290 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:32:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a27e00640 1 --2- 172.21.15.67:0/3068059290 >> v2:172.21.15.187:3300/0 conn(0x7f6a280d0b50 0x7f6a2815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a27e00640 1 -- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 msgr2=0x7f6a2806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a27e00640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a27e00640 1 -- 172.21.15.67:0/3068059290 >> v1:172.21.15.187:6789/0 conn(0x7f6a281627b0 legacy=0x7f6a280cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a27e00640 1 -- 172.21.15.67:0/3068059290 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a28163150 con 0x7f6a280d0b50 2024-04-17T07:32:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.462+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:32:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.464+0000 7f6a27e00640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a280d0b50 0x7f6a2815fb20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6a140097a0 tx=0x7f6a1402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b20252e8251b72d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.464+0000 7f6a27400640 1 -- 172.21.15.67:0/3068059290 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14035020 con 0x7f6a280d0b50 2024-04-17T07:32:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.464+0000 7f6a27400640 1 -- 172.21.15.67:0/3068059290 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a14031810 con 0x7f6a280d0b50 2024-04-17T07:32:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.464+0000 7f6a27400640 1 -- 172.21.15.67:0/3068059290 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14031be0 con 0x7f6a280d0b50 2024-04-17T07:32:46.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 -- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a280d0b50 msgr2=0x7f6a2815fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:46.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a280d0b50 0x7f6a2815fb20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6a140097a0 tx=0x7f6a1402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 -- 172.21.15.67:0/3068059290 shutdown_connections 2024-04-17T07:32:46.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a280d0b50 0x7f6a2815fb20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 --2- 172.21.15.67:0/3068059290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 -- 172.21.15.67:0/3068059290 >> 172.21.15.67:0/3068059290 conn(0x7f6a28073360 msgr2=0x7f6a2809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:46.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 -- 172.21.15.67:0/3068059290 shutdown_connections 2024-04-17T07:32:46.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.465+0000 7f6a2f600640 1 -- 172.21.15.67:0/3068059290 wait complete. 2024-04-17T07:32:46.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.467+0000 7f6a2f600640 1 Processor -- start 2024-04-17T07:32:46.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.467+0000 7f6a2f600640 1 -- start start 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2814a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 0x7f6a280c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a281627b0 0x7f6a2814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6a28160060 con 0x7f6a280d0b50 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2815f600 con 0x7f6a281627b0 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a2f600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6a281601e0 con 0x7f6a2806b8c0 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 0x7f6a280c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 0x7f6a280c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41992/0 (socket says 172.21.15.67:41992) 2024-04-17T07:32:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.468+0000 7f6a27e00640 1 -- 172.21.15.67:0/4294613081 learned_addr learned my addr 172.21.15.67:0/4294613081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2814a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a2d600640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a281627b0 0x7f6a2814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 msgr2=0x7f6a2814a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6a2806b8c0 0x7f6a2814a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a281627b0 msgr2=0x7f6a2814c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a281627b0 0x7f6a2814c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2815e530 con 0x7f6a280d0b50 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a27e00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 0x7f6a280c5a20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6a140098c0 tx=0x7f6a140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.469+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14041070 con 0x7f6a280d0b50 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.470+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1403ad00 con 0x7f6a280d0b50 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.470+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14035040 con 0x7f6a280d0b50 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.470+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a280b5350 con 0x7f6a280d0b50 2024-04-17T07:32:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.470+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a280b5910 con 0x7f6a280d0b50 2024-04-17T07:32:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.471+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a280b5140 con 0x7f6a280d0b50 2024-04-17T07:32:46.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.472+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 58) ==== 91159+0+0 (secure 0 0 0) 0x7f6a1404c060 con 0x7f6a280d0b50 2024-04-17T07:32:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.473+0000 7f6a24c00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.473+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f6a140ad8c0 con 0x7f6a280d0b50 2024-04-17T07:32:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.473+0000 7f6a2cc00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.473+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:32:46.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.477+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6a1404b2c0 con 0x7f6a280d0b50 2024-04-17T07:32:46.491 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.491+0000 7f7a08242040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:32:46.494 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.494+0000 7f4c8003e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:32:46.592 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.592+0000 7f7a08242040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:32:46.622 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.622+0000 7f4c8003e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:32:46.650 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.650+0000 7f7a08242040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:32:46.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.674+0000 7f6a2cc00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:46.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.674+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:32:46.683 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.683+0000 7f4c8003e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:46.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:46.700+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6a28068ca0 con 0x7f6a0806ca60 2024-04-17T07:32:46.735 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.734+0000 7f4c8003e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:32:46.787 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.787+0000 7f7a08242040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:32:46.791 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.791+0000 7f4c8003e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:32:46.842 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:46.842+0000 7f4c8003e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:32:46.854 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.854+0000 7f7a08242040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:46.906 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.906+0000 7f7a08242040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:32:46.966 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:46.966+0000 7f7a08242040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:32:47.020 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.019+0000 7f7a08242040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:32:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:47.075+0000 7f6a2cc00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:47.075+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:32:47.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.221+0000 7f4c8003e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:32:47.380 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.379+0000 7f4c8003e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:32:47.424 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.423+0000 7f7a08242040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:32:47.488 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.488+0000 7f4c8003e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:32:47.538 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.538+0000 7f4c8003e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:32:47.592 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.592+0000 7f7a08242040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:32:47.663 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.663+0000 7f4c8003e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:47.706 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.706+0000 7f7a08242040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:32:47.759 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.759+0000 7f7a08242040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:32:47.769 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.769+0000 7f4c8003e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:32:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:47.876+0000 7f6a2cc00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:47.876+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:32:47.892 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:47.892+0000 7f7a08242040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:47.942 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.941+0000 7f4c8003e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:47.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:47.991+0000 7f4c8003e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:48.005 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:48.005+0000 7f7a08242040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:32:48.125 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:48.125+0000 7f4c8003e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:32:48.189 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:48.189+0000 7f7a08242040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:32:48.241 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:48.241+0000 7f7a08242040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:32:48.385 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:48.384+0000 7f7a08242040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:32:48.976 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:48.975+0000 7f4c8003e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:32:49.275 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:49.275+0000 7f7a08242040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:32:49.278 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.278+0000 7f4c8003e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:32:49.336 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.336+0000 7f4c8003e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:32:49.468 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.467+0000 7f4c8003e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:32:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:49.478+0000 7f6a2cc00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2517863062 2024-04-17T07:32:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:49.478+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T07:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:49.491+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 59) ==== 90768+0+0 (secure 0 0 0) 0x7f6a14038020 con 0x7f6a280d0b50 2024-04-17T07:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:49.492+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 msgr2=0x7f6a0806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:49.492+0000 7f6a24c00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2517863062,v1:172.21.15.187:6833/2517863062] conn(0x7f6a0806ca60 0x7f6a0806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:49.616 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:49.615+0000 7f7a08242040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:32:49.682 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:49.682+0000 7f7a08242040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:32:49.814 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.814 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.814 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf5800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf3000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf3000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf3000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf3000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:32:49.814+0000 7f4bf3000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:32:49.833 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:32:49.833+0000 7f7a08242040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:32:50.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.502+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 60) ==== 90895+0+0 (secure 0 0 0) 0x7f6a14030070 con 0x7f6a280d0b50 2024-04-17T07:32:50.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.503+0000 7f6a24c00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f6a08074910 0x7f6a08076d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.503+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 --> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6a14008eb0 con 0x7f6a08074910 2024-04-17T07:32:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.506+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f6a08074910 0x7f6a08076d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.511+0000 7f6a2cc00640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f6a08074910 0x7f6a08076d00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009380 tx=0x7f6a1c009190 comp rx=0 tx=0).ready entity=mgr.6658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.707+0000 7f6a24c00640 1 -- 172.21.15.67:0/4294613081 <== mgr.6658 v2:172.21.15.187:6832/2982226538 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a14008eb0 con 0x7f6a08074910 2024-04-17T07:32:50.708 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.710+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f6a08074910 msgr2=0x7f6a08076d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.710+0000 7f6a2f600640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f6a08074910 0x7f6a08076d00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009380 tx=0x7f6a1c009190 comp rx=0 tx=0).stop 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.710+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 msgr2=0x7f6a280c5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.710+0000 7f6a2f600640 1 --2- 172.21.15.67:0/4294613081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a280d0b50 0x7f6a280c5a20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6a140098c0 tx=0x7f6a140046c0 comp rx=0 tx=0).stop 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.710+0000 7f6a2d600640 1 -- 172.21.15.67:0/4294613081 reap_dead start 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.711+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 shutdown_connections 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.711+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 >> 172.21.15.67:0/4294613081 conn(0x7f6a28073360 msgr2=0x7f6a280705b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.711+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 shutdown_connections 2024-04-17T07:32:50.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.711+0000 7f6a2f600640 1 -- 172.21.15.67:0/4294613081 wait complete. 2024-04-17T07:32:50.731 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T07:32:50.731 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-17-07_33_00 in ~10s... 2024-04-17T07:32:50.731 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-17T07:32:50.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-17T07:32:50.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.955+0000 7efcc1800640 1 Processor -- start 2024-04-17T07:32:50.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.956+0000 7efcc1800640 1 -- start start 2024-04-17T07:32:50.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcc1800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcc1800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc0d00b0 con 0x7efcbc15ecb0 2024-04-17T07:32:50.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcc1800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc0d0390 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcc1800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0d0c30 con 0x7efcbc0d33e0 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcb3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcb3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42000/0 (socket says 172.21.15.67:42000) 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcb3400640 1 -- 172.21.15.67:0/3510123005 learned_addr learned my addr 172.21.15.67:0/3510123005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.957+0000 7efcbbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcbc15ecb0 0x7efcbc0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36188/0 (socket says 172.21.15.67:36188) 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.958+0000 7efcb3400640 1 -- 172.21.15.67:0/3510123005 >> v1:172.21.15.187:6789/0 conn(0x7efcbc0cf8e0 legacy=0x7efcbc0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.958+0000 7efcb3400640 1 -- 172.21.15.67:0/3510123005 >> v1:172.21.15.67:6789/0 conn(0x7efcbc15ecb0 legacy=0x7efcbc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.958+0000 7efcb3400640 1 -- 172.21.15.67:0/3510123005 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0d0670 con 0x7efcbc0d33e0 2024-04-17T07:32:50.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.959+0000 7efcb3400640 1 --2- 172.21.15.67:0/3510123005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7efca4009870 tx=0x7efca402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f2f230022af320 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:50.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcbb400640 1 -- 172.21.15.67:0/3510123005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4035020 con 0x7efcbc0d33e0 2024-04-17T07:32:50.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcbb400640 1 -- 172.21.15.67:0/3510123005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca4031870 con 0x7efcbc0d33e0 2024-04-17T07:32:50.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcbb400640 1 -- 172.21.15.67:0/3510123005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4031c40 con 0x7efcbc0d33e0 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 -- 172.21.15.67:0/3510123005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d33e0 msgr2=0x7efcbc0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 --2- 172.21.15.67:0/3510123005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7efca4009870 tx=0x7efca402b910 comp rx=0 tx=0).stop 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 -- 172.21.15.67:0/3510123005 shutdown_connections 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 --2- 172.21.15.67:0/3510123005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0d33e0 0x7efcbc0cf3a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 -- 172.21.15.67:0/3510123005 >> 172.21.15.67:0/3510123005 conn(0x7efcbc08bff0 msgr2=0x7efcbc0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 -- 172.21.15.67:0/3510123005 shutdown_connections 2024-04-17T07:32:50.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.960+0000 7efcc1800640 1 -- 172.21.15.67:0/3510123005 wait complete. 2024-04-17T07:32:50.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.962+0000 7efcc1800640 1 Processor -- start 2024-04-17T07:32:50.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.962+0000 7efcc1800640 1 -- start start 2024-04-17T07:32:50.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc0d33e0 0x7efcbc0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 0x7efcbc1f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc15f090 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc0cfd30 con 0x7efcbc15ecb0 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.963+0000 7efcc1800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc15f210 con 0x7efcbc0d33e0 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42004/0 (socket says 172.21.15.67:42004) 2024-04-17T07:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 -- 172.21.15.67:0/198385948 learned_addr learned my addr 172.21.15.67:0/198385948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:32:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcb3400640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc0d33e0 0x7efcbc0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcc0e00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 0x7efcbc1f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 -- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc0d33e0 msgr2=0x7efcbc0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc0d33e0 0x7efcbc0cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 -- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 msgr2=0x7efcbc1f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 0x7efcbc1f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 -- 172.21.15.67:0/198385948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0ce2f0 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcc0e00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 0x7efcbc1f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.964+0000 7efcbbe00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c3c0 tx=0x7efcac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.965+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0074e0 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.965+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcac007680 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.965+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc0ce5d0 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.965+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac004a00 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.965+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc15adb0 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.966+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc068a10 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.967+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 60) ==== 90895+0+0 (secure 0 0 0) 0x7efcac01a020 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.968+0000 7efcb8c00640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 0x7efc9806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:32:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.968+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7efcac08b0f0 con 0x7efcbc0cf8e0 2024-04-17T07:32:50.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.969+0000 7efcb3400640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 0x7efc9806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:32:50.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.970+0000 7efcb3400640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 0x7efc9806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efca4036000 tx=0x7efca4008030 comp rx=0 tx=0).ready entity=mgr.6658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:32:50.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:50.973+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efcac059700 con 0x7efcbc0cf8e0 2024-04-17T07:32:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.214+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 --> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efcbc0b5a70 con 0x7efc9806c790 2024-04-17T07:32:51.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.231+0000 7efcb8c00640 1 -- 172.21.15.67:0/198385948 <== mgr.6658 v2:172.21.15.187:6832/2982226538 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7efcbc0b5a70 con 0x7efc9806c790 2024-04-17T07:32:51.231 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:32:51.231 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 msgr2=0x7efc9806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 0x7efc9806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efca4036000 tx=0x7efca4008030 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 msgr2=0x7efcbc0cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c3c0 tx=0x7efcac00c880 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 shutdown_connections 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7efc9806c790 0x7efc9806ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcbc15ecb0 0x7efcbc1f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcbc0d33e0 0x7efcbc0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 --2- 172.21.15.67:0/198385948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc0cf8e0 0x7efcbc0cba20 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:32:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.233+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 >> 172.21.15.67:0/198385948 conn(0x7efcbc08bff0 msgr2=0x7efcbc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:32:51.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.234+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 shutdown_connections 2024-04-17T07:32:51.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:32:51.234+0000 7efcc1800640 1 -- 172.21.15.67:0/198385948 wait complete. 2024-04-17T07:32:51.253 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-04-17T07:32:51.253 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-04-17T07:32:51.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:51.253 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:51.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:51.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:51.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:52.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:52.306 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:52.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:52.370 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:52.370 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:53.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:53.372 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:53.435 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:53.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:53.436 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:54.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:54.436 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:54.499 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:54.500 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:54.500 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:55.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:55.500 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:55.564 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:55.565 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:55.565 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:56.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:56.565 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:56.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:56.632 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:56.632 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:57.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:57.633 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:57.698 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:57.698 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:57.698 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:58.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:58.700 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:58.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:58.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:58.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:32:59.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:32:59.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:32:59.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:32:59.764 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:32:59.826 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T07:32:59.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T07:32:59.827 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:33:00.734 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:00.732+0000 7f4be2200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:00.734 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:00.732+0000 7f4be2200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:00.735 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:00.732+0000 7f4be2200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:00.735 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:00.732+0000 7f4be2200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:00.735 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:00.732+0000 7f4be2200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:00.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:00.828 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:33:00.888 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_33_00_UTC 2024-04-17T07:33:00.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-07_33_00_UTC'} 2024-04-17T07:33:00.889 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-17-07_33_00_UTC'} 2024-04-17T07:33:00.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T07:33:00.889 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-17-07_33_00_UTC'] 2024-04-17T07:33:00.889 DEBUG:tasks.cephfs.test_snap_schedules:exec=1713339170.7311687, now = 1713339180.8898757, timo = 10 2024-04-17T07:33:01.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-17T07:33:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.123+0000 7f8822600640 1 Processor -- start 2024-04-17T07:33:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.124+0000 7f8822600640 1 -- start start 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.125+0000 7f8822600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f881c0d33e0 0x7f881c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.125+0000 7f8822600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881c0d00b0 con 0x7f881c15ecb0 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.125+0000 7f8822600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881c0d0390 con 0x7f881c0cf8e0 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.125+0000 7f8822600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f881c0d0c30 con 0x7f881c0d33e0 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f8821200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f881c15ecb0 0x7f881c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56254/0 (socket says 172.21.15.67:56254) 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f8821200640 1 -- 172.21.15.67:0/555077129 learned_addr learned my addr 172.21.15.67:0/555077129 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f881be00640 1 --2- 172.21.15.67:0/555077129 >> v2:172.21.15.67:3300/0 conn(0x7f881c0d33e0 0x7f881c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:02.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f881be00640 1 -- 172.21.15.67:0/555077129 >> v1:172.21.15.187:6789/0 conn(0x7f881c0cf8e0 legacy=0x7f881c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:02.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f881b400640 1 -- 172.21.15.67:0/555077129 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439353876 0 0) 0x7f881c0d00b0 con 0x7f881c15ecb0 2024-04-17T07:33:02.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f881be00640 1 -- 172.21.15.67:0/555077129 >> v1:172.21.15.67:6789/0 conn(0x7f881c15ecb0 legacy=0x7f881c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.126+0000 7f881be00640 1 -- 172.21.15.67:0/555077129 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c0d0670 con 0x7f881c0d33e0 2024-04-17T07:33:02.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.127+0000 7f881be00640 1 --2- 172.21.15.67:0/555077129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c0d33e0 0x7f881c0cf3a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f880c009d90 tx=0x7f880c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdae8401c22f30e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:02.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f881b400640 1 -- 172.21.15.67:0/555077129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c039070 con 0x7f881c0d33e0 2024-04-17T07:33:02.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f881b400640 1 -- 172.21.15.67:0/555077129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c02cc60 con 0x7f881c0d33e0 2024-04-17T07:33:02.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f881b400640 1 -- 172.21.15.67:0/555077129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c034660 con 0x7f881c0d33e0 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f8822600640 1 -- 172.21.15.67:0/555077129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c0d33e0 msgr2=0x7f881c0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f8822600640 1 --2- 172.21.15.67:0/555077129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c0d33e0 0x7f881c0cf3a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f880c009d90 tx=0x7f880c02beb0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f8822600640 1 -- 172.21.15.67:0/555077129 shutdown_connections 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f8822600640 1 --2- 172.21.15.67:0/555077129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c0d33e0 0x7f881c0cf3a0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.128+0000 7f8822600640 1 -- 172.21.15.67:0/555077129 >> 172.21.15.67:0/555077129 conn(0x7f881c08bff0 msgr2=0x7f881c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.129+0000 7f8822600640 1 -- 172.21.15.67:0/555077129 shutdown_connections 2024-04-17T07:33:02.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.129+0000 7f8822600640 1 -- 172.21.15.67:0/555077129 wait complete. 2024-04-17T07:33:02.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.130+0000 7f8822600640 1 Processor -- start 2024-04-17T07:33:02.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.130+0000 7f8822600640 1 -- start start 2024-04-17T07:33:02.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f881c0cf8e0 0x7f881c157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:02.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f881c0d33e0 0x7f881c0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:02.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:02.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f881c15f090 con 0x7f881c15ecb0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0cfd30 con 0x7f881c0cf8e0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.131+0000 7f8822600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f881c15f210 con 0x7f881c0d33e0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55680/0 (socket says 172.21.15.67:55680) 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 -- 172.21.15.67:0/2936699112 learned_addr learned my addr 172.21.15.67:0/2936699112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821200640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f881c0cf8e0 0x7f881c157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f881be00640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f881c0d33e0 0x7f881c0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 -- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f881c0d33e0 msgr2=0x7f881c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f881c0d33e0 0x7f881c0b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 -- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f881c0cf8e0 msgr2=0x7f881c157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f881c0cf8e0 0x7f881c157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.132+0000 7f8821c00640 1 -- 172.21.15.67:0/2936699112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c0b5070 con 0x7f881c15ecb0 2024-04-17T07:33:02.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8821c00640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f881400c3c0 tx=0x7f881400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:02.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88140052d0 con 0x7f881c15ecb0 2024-04-17T07:33:02.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8814005490 con 0x7f881c15ecb0 2024-04-17T07:33:02.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814007260 con 0x7f881c15ecb0 2024-04-17T07:33:02.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881c0b5350 con 0x7f881c15ecb0 2024-04-17T07:33:02.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.133+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881c0b4940 con 0x7f881c15ecb0 2024-04-17T07:33:02.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.134+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f881c0b5870 con 0x7f881c15ecb0 2024-04-17T07:33:02.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.135+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 63) ==== 91273+0+0 (secure 0 0 0) 0x7f8814007400 con 0x7f881c15ecb0 2024-04-17T07:33:02.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.136+0000 7f8818c00640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 0x7f87f806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:02.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.136+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f881408b660 con 0x7f881c15ecb0 2024-04-17T07:33:02.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.137+0000 7f8821200640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 0x7f87f806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:02.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.138+0000 7f8821200640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 0x7f87f806f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8804000f60 tx=0x7f8804007d00 comp rx=0 tx=0).ready entity=mgr.6658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:02.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.141+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f881c0b5870 con 0x7f881c15ecb0 2024-04-17T07:33:02.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.372+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 --> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f881c068a10 con 0x7f87f806cb80 2024-04-17T07:33:02.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.584+0000 7f8818c00640 1 -- 172.21.15.67:0/2936699112 <== mgr.6658 v2:172.21.15.187:6832/2982226538 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f881c068a10 con 0x7f87f806cb80 2024-04-17T07:33:02.585 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path snap_test_dir1 2024-04-17T07:33:02.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 msgr2=0x7f87f806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 0x7f87f806f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8804000f60 tx=0x7f8804007d00 comp rx=0 tx=0).stop 2024-04-17T07:33:02.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 msgr2=0x7f881c0b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:02.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f881400c3c0 tx=0x7f881400c880 comp rx=0 tx=0).stop 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 shutdown_connections 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f87f806cb80 0x7f87f806f040 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881c15ecb0 0x7f881c0b4400 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f881c0d33e0 0x7f881c0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 --2- 172.21.15.67:0/2936699112 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f881c0cf8e0 0x7f881c157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 >> 172.21.15.67:0/2936699112 conn(0x7f881c08bff0 msgr2=0x7f881c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 shutdown_connections 2024-04-17T07:33:02.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:02.588+0000 7f8822600640 1 -- 172.21.15.67:0/2936699112 wait complete. 2024-04-17T07:33:02.612 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-17T07:33:02.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:02.612 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T07:33:02.666 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_33_00_UTC 2024-04-17T07:33:02.667 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_33_00_UTC 2024-04-17T07:33:02.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:02.667 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_33_00_UTC) 2024-04-17T07:33:02.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:02.733 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T07:33:02.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:33:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.018+0000 7f7e5b400640 1 Processor -- start 2024-04-17T07:33:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.019+0000 7f7e5b400640 1 -- start start 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.020+0000 7f7e5b400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7e540d17a0 0x7f7e540cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.020+0000 7f7e5b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e540d4100 con 0x7f7e5415eca0 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.020+0000 7f7e5b400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e54163860 con 0x7f7e540cfa10 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.020+0000 7f7e5b400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7e540d03b0 con 0x7f7e540d17a0 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.021+0000 7f7e58a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e5415eca0 0x7f7e540d39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56266/0 (socket says 172.21.15.67:56266) 2024-04-17T07:33:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.021+0000 7f7e58a00640 1 -- 172.21.15.67:0/3576835961 learned_addr learned my addr 172.21.15.67:0/3576835961 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.021+0000 7f7e59400640 1 --2- 172.21.15.67:0/3576835961 >> v2:172.21.15.187:3300/0 conn(0x7f7e540d17a0 0x7f7e540cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.021+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855417529 0 0) 0x7f7e540d4100 con 0x7f7e5415eca0 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.021+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e3c003520 con 0x7f7e5415eca0 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971153946 0 0) 0x7f7e3c003520 con 0x7f7e5415eca0 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 >> v2:172.21.15.187:3300/0 conn(0x7f7e540d17a0 msgr2=0x7f7e540cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 --2- 172.21.15.67:0/3576835961 >> v2:172.21.15.187:3300/0 conn(0x7f7e540d17a0 0x7f7e540cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 >> v1:172.21.15.187:6789/0 conn(0x7f7e540cfa10 legacy=0x7f7e540d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e540d1d40 con 0x7f7e5415eca0 2024-04-17T07:33:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7e48001670 con 0x7f7e5415eca0 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 >> v1:172.21.15.67:6789/0 conn(0x7f7e5415eca0 legacy=0x7f7e540d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c003a00 0x7f7e3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e3c004800 0x7f7e3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 0x7f7e3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c00c090 con 0x7f7e3c0051c0 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c00c240 con 0x7f7e3c003a00 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c00c440 con 0x7f7e3c004800 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e59400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 0x7f7e3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.022+0000 7f7e58a00640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c003a00 0x7f7e3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.023+0000 7f7e59400640 1 -- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e3c004800 msgr2=0x7f7e3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.023+0000 7f7e59400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e3c004800 0x7f7e3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.023+0000 7f7e59400640 1 -- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c003a00 msgr2=0x7f7e3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.023+0000 7f7e59400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c003a00 0x7f7e3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.023+0000 7f7e59400640 1 -- 172.21.15.67:0/3576835961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c003520 con 0x7f7e3c0051c0 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.024+0000 7f7e59400640 1 -- 172.21.15.67:0/3576835961 reap_dead start 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.024+0000 7f7e59400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 0x7f7e3c00b9c0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009fc0 tx=0x7f7e3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f567334cbf19091e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.024+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c009c10 con 0x7f7e3c0051c0 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.024+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c02cca0 con 0x7f7e3c0051c0 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.024+0000 7f7e53400640 1 -- 172.21.15.67:0/3576835961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c003bd0 con 0x7f7e3c0051c0 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 -- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 msgr2=0x7f7e3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 0x7f7e3c00b9c0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009fc0 tx=0x7f7e3c005830 comp rx=0 tx=0).stop 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 -- 172.21.15.67:0/3576835961 shutdown_connections 2024-04-17T07:33:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3576835961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e3c0051c0 0x7f7e3c00b9c0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 -- 172.21.15.67:0/3576835961 >> 172.21.15.67:0/3576835961 conn(0x7f7e54067600 msgr2=0x7f7e540679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 -- 172.21.15.67:0/3576835961 shutdown_connections 2024-04-17T07:33:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.025+0000 7f7e5b400640 1 -- 172.21.15.67:0/3576835961 wait complete. 2024-04-17T07:33:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.026+0000 7f7e5b400640 1 Processor -- start 2024-04-17T07:33:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.027+0000 7f7e5b400640 1 -- start start 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0051c0 0x7f7e54142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 0x7f7e540c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e540d31f0 con 0x7f7e540c9390 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e540d2f10 con 0x7f7e3c0051c0 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e5b400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e540d34d0 con 0x7f7e5415d770 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55694/0 (socket says 172.21.15.67:55694) 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 -- 172.21.15.67:0/3565899062 learned_addr learned my addr 172.21.15.67:0/3565899062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e58a00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0051c0 0x7f7e54142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e59400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 0x7f7e540c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 -- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 msgr2=0x7f7e540c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 0x7f7e540c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 -- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0051c0 msgr2=0x7f7e54142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0051c0 0x7f7e54142b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e53e00640 1 -- 172.21.15.67:0/3565899062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e541430d0 con 0x7f7e540c9390 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.028+0000 7f7e59400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 0x7f7e540c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e53e00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c480 tx=0x7f7e4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e440052d0 con 0x7f7e540c9390 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e44005470 con 0x7f7e540c9390 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44012870 con 0x7f7e540c9390 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e540c85d0 con 0x7f7e540c9390 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.029+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e540c8b90 con 0x7f7e540c9390 2024-04-17T07:33:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.031+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5415db80 con 0x7f7e540c9390 2024-04-17T07:33:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.032+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 63) ==== 91273+0+0 (secure 0 0 0) 0x7f7e44012a10 con 0x7f7e540c9390 2024-04-17T07:33:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.033+0000 7f7e50c00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 0x7f7e3806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.033+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f7e4408bdb0 con 0x7f7e540c9390 2024-04-17T07:33:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.034+0000 7f7e58a00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 0x7f7e3806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.035+0000 7f7e58a00640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 0x7f7e3806f040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e48001050 tx=0x7f7e48009040 comp rx=0 tx=0).ready entity=mgr.6658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.037+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7e5415db80 con 0x7f7e540c9390 2024-04-17T07:33:03.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.297+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f7e54068ca0 con 0x7f7e540c9390 2024-04-17T07:33:03.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.612+0000 7f7e50c00640 1 -- 172.21.15.67:0/3565899062 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) ==== 93+0+0 (secure 0 0 0) 0x7f7e44059db0 con 0x7f7e540c9390 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.617+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 msgr2=0x7f7e3806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.617+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 0x7f7e3806f040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e48001050 tx=0x7f7e48009040 comp rx=0 tx=0).stop 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.617+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 msgr2=0x7f7e5415d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.617+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c480 tx=0x7f7e4400c940 comp rx=0 tx=0).stop 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 shutdown_connections 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f7e3806cb80 0x7f7e3806f040 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e48001050 tx=0x7f7e48009040 comp rx=0 tx=0).stop 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e5415d770 0x7f7e540c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e540c9390 0x7f7e5415d230 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 --2- 172.21.15.67:0/3565899062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e3c0051c0 0x7f7e54142b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:03.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.618+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 >> 172.21.15.67:0/3565899062 conn(0x7f7e54067600 msgr2=0x7f7e54154f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:03.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.619+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 shutdown_connections 2024-04-17T07:33:03.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:03.620+0000 7f7e5b400640 1 -- 172.21.15.67:0/3565899062 wait complete. 2024-04-17T07:33:03.651 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:33:03.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:03.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:03.882 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:33:03.882 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:03.882 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:03.882 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:03.936 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:03.936 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:03.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:04.181 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.181+0000 7fb4749bc040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:04.200 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.201+0000 7f6c24a40040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:04.235 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.235+0000 7fb4749bc040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.237+0000 7f2fbdc00640 1 Processor -- start 2024-04-17T07:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.237+0000 7f2fbdc00640 1 -- start start 2024-04-17T07:33:04.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.239+0000 7f2fbdc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2fb8141ed0 0x7f2fb8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.239+0000 7f2fbdc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb80cd4a0 con 0x7f2fb806b8c0 2024-04-17T07:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.239+0000 7f2fbdc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb81607f0 con 0x7f2fb8160280 2024-04-17T07:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.239+0000 7f2fbdc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb815ce30 con 0x7f2fb8141ed0 2024-04-17T07:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fb806b8c0 0x7f2fb80cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49922/0 (socket says 172.21.15.67:49922) 2024-04-17T07:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7e00640 1 -- 172.21.15.67:0/1624515427 learned_addr learned my addr 172.21.15.67:0/1624515427 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fbd200640 1 --2- 172.21.15.67:0/1624515427 >> v2:172.21.15.187:3300/0 conn(0x7f2fb8141ed0 0x7f2fb8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603387508 0 0) 0x7f2fb80cd4a0 con 0x7f2fb806b8c0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f9c003520 con 0x7f2fb806b8c0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1449835029 0 0) 0x7f2f9c003520 con 0x7f2fb806b8c0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 >> v2:172.21.15.187:3300/0 conn(0x7f2fb8141ed0 msgr2=0x7f2fb8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.240+0000 7f2fb7400640 1 --2- 172.21.15.67:0/1624515427 >> v2:172.21.15.187:3300/0 conn(0x7f2fb8141ed0 0x7f2fb8142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 >> v1:172.21.15.187:6789/0 conn(0x7f2fb8160280 legacy=0x7f2fb806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8160ad0 con 0x7f2fb806b8c0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2fa8001670 con 0x7f2fb806b8c0 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 >> v1:172.21.15.67:6789/0 conn(0x7f2fb806b8c0 legacy=0x7f2fb80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 0x7f2f9c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f9c004800 0x7f2f9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c0051c0 0x7f2f9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb815ce30 con 0x7f2f9c003a20 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c00c0f0 con 0x7f2f9c0051c0 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c00c2f0 con 0x7f2f9c004800 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 0x7f2f9c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fbd200640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c0051c0 0x7f2f9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.241+0000 7f2fb7e00640 1 -- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f9c004800 msgr2=0x7f2f9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.242+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f9c004800 0x7f2f9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.242+0000 7f2fb7e00640 1 -- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c0051c0 msgr2=0x7f2f9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.242+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c0051c0 0x7f2f9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.242+0000 7f2fb7e00640 1 -- 172.21.15.67:0/1624515427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c003520 con 0x7f2f9c003a20 2024-04-17T07:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.242+0000 7f2fbd200640 1 -- 172.21.15.67:0/1624515427 reap_dead start 2024-04-17T07:33:04.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 0x7f2f9c003e40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c210 tx=0x7f2f9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da1ca2dfb3bb3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:04.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8007d30 con 0x7f2f9c003a20 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8030070 con 0x7f2f9c003a20 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fb7400640 1 -- 172.21.15.67:0/1624515427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8005d70 con 0x7f2f9c003a20 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fbdc00640 1 -- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 msgr2=0x7f2f9c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.243+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 0x7f2f9c003e40 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c210 tx=0x7f2f9c005830 comp rx=0 tx=0).stop 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.244+0000 7f2fbdc00640 1 -- 172.21.15.67:0/1624515427 shutdown_connections 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.244+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/1624515427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c003a20 0x7f2f9c003e40 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.244+0000 7f2fbdc00640 1 -- 172.21.15.67:0/1624515427 >> 172.21.15.67:0/1624515427 conn(0x7f2fb8067d10 msgr2=0x7f2fb80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.244+0000 7f2fbdc00640 1 -- 172.21.15.67:0/1624515427 shutdown_connections 2024-04-17T07:33:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.244+0000 7f2fbdc00640 1 -- 172.21.15.67:0/1624515427 wait complete. 2024-04-17T07:33:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.245+0000 7f2fbdc00640 1 Processor -- start 2024-04-17T07:33:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.246+0000 7f2fbdc00640 1 -- start start 2024-04-17T07:33:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c003a20 0x7f2fb815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 0x7f2fb8145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb813f9b0 con 0x7f2fb80aecd0 2024-04-17T07:33:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb8161090 con 0x7f2f9c003a20 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb813fb30 con 0x7f2fb81439d0 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fb7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c003a20 0x7f2fb815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34190/0 (socket says 172.21.15.67:34190) 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 -- 172.21.15.67:0/156946182 learned_addr learned my addr 172.21.15.67:0/156946182 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fafe00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 0x7f2fb8145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 msgr2=0x7f2fb8145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 0x7f2fb8145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.247+0000 7f2fbd200640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c003a20 msgr2=0x7f2fb815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fbd200640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c003a20 0x7f2fb815d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fbd200640 1 -- 172.21.15.67:0/156946182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8143e60 con 0x7f2fb80aecd0 2024-04-17T07:33:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fafe00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 0x7f2fb8145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fbd200640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8071d80 tx=0x7f2fb00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0011db0 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb001b070 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0003990 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb8144140 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.248+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb813efc0 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.250+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80ce370 con 0x7f2fb80aecd0 2024-04-17T07:33:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.251+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7f2fb0003b30 con 0x7f2fb80aecd0 2024-04-17T07:33:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.251+0000 7f2fb4c00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 0x7f2f9406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.252+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f2fb008ca50 con 0x7f2fb80aecd0 2024-04-17T07:33:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.252+0000 7f2fb7e00640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 msgr2=0x7f2f9406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.252+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 0x7f2f9406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:04.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.257+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2fb80ce370 con 0x7f2fb80aecd0 2024-04-17T07:33:04.261 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.262+0000 7f6c24a40040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.453+0000 7f2fb7e00640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 msgr2=0x7f2f9406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.453+0000 7f2fb7e00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 0x7f2f9406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.462+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fb8068a10 con 0x7f2fb80aecd0 2024-04-17T07:33:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.464+0000 7f2fb4c00640 1 -- 172.21.15.67:0/156946182 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+17078 (secure 0 0 0) 0x7f2fb0003e20 con 0x7f2fb80aecd0 2024-04-17T07:33:04.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:04.463 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":128,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:49.473630+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/757480182":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2989376162":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6833/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6832/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/3883423100":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/410647397":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2591672672":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1399211335":"2024-04-18T07:32:49.473454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 msgr2=0x7f2f9406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 0x7f2f9406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 msgr2=0x7f2fb81f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8071d80 tx=0x7f2fb00115a0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 shutdown_connections 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f2f9406cb30 0x7f2f9406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb80aecd0 0x7f2fb81f47f0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fb81439d0 0x7f2fb8145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 --2- 172.21.15.67:0/156946182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f9c003a20 0x7f2fb815d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 >> 172.21.15.67:0/156946182 conn(0x7f2fb8067d10 msgr2=0x7f2fb80cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.466+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 shutdown_connections 2024-04-17T07:33:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.467+0000 7f2fbdc00640 1 -- 172.21.15.67:0/156946182 wait complete. 2024-04-17T07:33:04.485 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:32:32.505992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:32:33.573434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:04.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:04.681 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.680+0000 7fb4749bc040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:04.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.695+0000 7fd275200640 1 Processor -- start 2024-04-17T07:33:04.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.695+0000 7fd275200640 1 -- start start 2024-04-17T07:33:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd275200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd270142e20 0x7fd2700cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd275200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2700cd750 0x7fd2700b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd275200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2701607a0 con 0x7fd2700cf160 2024-04-17T07:33:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd275200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd270160a60 con 0x7fd2700cd750 2024-04-17T07:33:04.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd275200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd270160d40 con 0x7fd270142e20 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd26e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2700cd750 0x7fd2700b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd26d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2700cf160 0x7fd2700cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49928/0 (socket says 172.21.15.67:49928) 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.697+0000 7fd26d400640 1 -- 172.21.15.67:0/2888783806 learned_addr learned my addr 172.21.15.67:0/2888783806 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.698+0000 7fd26de00640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd270142e20 0x7fd2700cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.698+0000 7fd26e800640 1 -- 172.21.15.67:0/2888783806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd270142e20 msgr2=0x7fd2700cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.698+0000 7fd26e800640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd270142e20 0x7fd2700cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.698+0000 7fd26e800640 1 -- 172.21.15.67:0/2888783806 >> v1:172.21.15.67:6789/0 conn(0x7fd2700cf160 legacy=0x7fd2700cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.698+0000 7fd26e800640 1 -- 172.21.15.67:0/2888783806 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd270161020 con 0x7fd2700cd750 2024-04-17T07:33:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.699+0000 7fd26e800640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2700cd750 0x7fd2700b08e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd268009d90 tx=0x7fd26802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccc84b99b09fb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.699+0000 7fd26ca00640 1 -- 172.21.15.67:0/2888783806 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268039070 con 0x7fd2700cd750 2024-04-17T07:33:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.699+0000 7fd26ca00640 1 -- 172.21.15.67:0/2888783806 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26802cbc0 con 0x7fd2700cd750 2024-04-17T07:33:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.699+0000 7fd26ca00640 1 -- 172.21.15.67:0/2888783806 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268034440 con 0x7fd2700cd750 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 -- 172.21.15.67:0/2888783806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2700cd750 msgr2=0x7fd2700b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2700cd750 0x7fd2700b08e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd268009d90 tx=0x7fd26802be10 comp rx=0 tx=0).stop 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 -- 172.21.15.67:0/2888783806 shutdown_connections 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2700cd750 0x7fd2700b08e0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 --2- 172.21.15.67:0/2888783806 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd270142e20 0x7fd2700cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 -- 172.21.15.67:0/2888783806 >> 172.21.15.67:0/2888783806 conn(0x7fd27006b8c0 msgr2=0x7fd27006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:04.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 -- 172.21.15.67:0/2888783806 shutdown_connections 2024-04-17T07:33:04.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.700+0000 7fd275200640 1 -- 172.21.15.67:0/2888783806 wait complete. 2024-04-17T07:33:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.702+0000 7fd275200640 1 Processor -- start 2024-04-17T07:33:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.702+0000 7fd275200640 1 -- start start 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2700cd750 0x7fd27013b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2700cf160 0x7fd2700b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd270143200 con 0x7fd270142e20 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2700ce730 con 0x7fd2700cf160 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd275200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd270143380 con 0x7fd2700cd750 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd26e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd26e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34206/0 (socket says 172.21.15.67:34206) 2024-04-17T07:33:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd26e800640 1 -- 172.21.15.67:0/1303583506 learned_addr learned my addr 172.21.15.67:0/1303583506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.703+0000 7fd26de00640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2700cd750 0x7fd27013b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26d400640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2700cf160 0x7fd2700b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2700cd750 msgr2=0x7fd27013b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2700cd750 0x7fd27013b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2700cf160 msgr2=0x7fd2700b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2700cf160 0x7fd2700b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 -- 172.21.15.67:0/1303583506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2700b47d0 con 0x7fd270142e20 2024-04-17T07:33:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd26e800640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd268009d90 tx=0x7fd268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268039050 con 0x7fd270142e20 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd268045070 con 0x7fd270142e20 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2700b4210 con 0x7fd270142e20 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.704+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26803e830 con 0x7fd270142e20 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.705+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27013d580 con 0x7fd270142e20 2024-04-17T07:33:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.706+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2700d0100 con 0x7fd270142e20 2024-04-17T07:33:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.707+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7fd26803e9d0 con 0x7fd270142e20 2024-04-17T07:33:04.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.707+0000 7fd266000640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 0x7fd24c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:04.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.708+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7fd2680aef80 con 0x7fd270142e20 2024-04-17T07:33:04.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.708+0000 7fd26de00640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 msgr2=0x7fd24c06eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:04.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.708+0000 7fd26de00640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 0x7fd24c06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:04.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.716+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd2700d0100 con 0x7fd270142e20 2024-04-17T07:33:04.734 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.734+0000 7fb4749bc040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:04.743 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.744+0000 7f6c24a40040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:04.785 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.785+0000 7fb4749bc040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:04.799 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.800+0000 7f6c24a40040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:04.848 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.848+0000 7fb4749bc040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:04.853 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.854+0000 7f6c24a40040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:04.897 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:04.897+0000 7fb4749bc040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:04.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.909+0000 7fd26de00640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 msgr2=0x7fd24c06eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:04.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.909+0000 7fd26de00640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 0x7fd24c06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:04.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.918+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd270068b90 con 0x7fd270142e20 2024-04-17T07:33:04.917 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.918+0000 7f6c24a40040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:04.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.919+0000 7fd266000640 1 -- 172.21.15.67:0/1303583506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) ==== 93+0+10 (secure 0 0 0) 0x7fd26807cf70 con 0x7fd270142e20 2024-04-17T07:33:04.918 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 msgr2=0x7fd24c06eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 0x7fd24c06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 msgr2=0x7fd27015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd268009d90 tx=0x7fd268004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 shutdown_connections 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fd24c06cb30 0x7fd24c06eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd270142e20 0x7fd27015e360 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2700cf160 0x7fd2700b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 --2- 172.21.15.67:0/1303583506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2700cd750 0x7fd27013b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.921+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 >> 172.21.15.67:0/1303583506 conn(0x7fd27006b8c0 msgr2=0x7fd27015ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.922+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 shutdown_connections 2024-04-17T07:33:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:04.922+0000 7fd275200640 1 -- 172.21.15.67:0/1303583506 wait complete. 2024-04-17T07:33:04.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:33:04.973 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:04.974+0000 7f6c24a40040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:05.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.026+0000 7fb4749bc040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:05.089 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.089+0000 7fb4749bc040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:05.112 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.113+0000 7f6c24a40040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:05.140 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.140+0000 7fb4749bc040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.150+0000 7fbc28800640 1 Processor -- start 2024-04-17T07:33:05.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.150+0000 7fbc28800640 1 -- start start 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc20071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbc200ce3b0 0x7fbc20072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc20072bb0 0x7fbc200ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbc200ce810 con 0x7fbc20072bb0 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbc200cea70 con 0x7fbc200ce3b0 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.152+0000 7fbc28800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200ced50 con 0x7fbc20145ac0 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc20072bb0 0x7fbc200ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc20072bb0 0x7fbc200ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34208/0 (socket says 172.21.15.67:34208) 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 -- 172.21.15.67:0/1151161367 learned_addr learned my addr 172.21.15.67:0/1151161367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc25400640 1 --2- 172.21.15.67:0/1151161367 >> v2:172.21.15.187:3300/0 conn(0x7fbc200ce3b0 0x7fbc20072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc25e00640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc20071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 -- 172.21.15.67:0/1151161367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 msgr2=0x7fbc20071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc20071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 -- 172.21.15.67:0/1151161367 >> v2:172.21.15.187:3300/0 conn(0x7fbc200ce3b0 msgr2=0x7fbc20072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 --2- 172.21.15.67:0/1151161367 >> v2:172.21.15.187:3300/0 conn(0x7fbc200ce3b0 0x7fbc20072670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.153+0000 7fbc26800640 1 -- 172.21.15.67:0/1151161367 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc200cf030 con 0x7fbc20072bb0 2024-04-17T07:33:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.155+0000 7fbc26800640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc20072bb0 0x7fbc200ccea0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbc200701a0 tx=0x7fbc1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c765590b7fdffcd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.155+0000 7fbc24a00640 1 -- 172.21.15.67:0/1151161367 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c02ca70 con 0x7fbc20072bb0 2024-04-17T07:33:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.155+0000 7fbc24a00640 1 -- 172.21.15.67:0/1151161367 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c02cc10 con 0x7fbc20072bb0 2024-04-17T07:33:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.155+0000 7fbc24a00640 1 -- 172.21.15.67:0/1151161367 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0343f0 con 0x7fbc20072bb0 2024-04-17T07:33:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 -- 172.21.15.67:0/1151161367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc20072bb0 msgr2=0x7fbc200ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc20072bb0 0x7fbc200ccea0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbc200701a0 tx=0x7fbc1c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 -- 172.21.15.67:0/1151161367 shutdown_connections 2024-04-17T07:33:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc20072bb0 0x7fbc200ccea0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 --2- 172.21.15.67:0/1151161367 >> v2:172.21.15.187:3300/0 conn(0x7fbc200ce3b0 0x7fbc20072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 --2- 172.21.15.67:0/1151161367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc20071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 -- 172.21.15.67:0/1151161367 >> 172.21.15.67:0/1151161367 conn(0x7fbc20067ee0 msgr2=0x7fbc2006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 -- 172.21.15.67:0/1151161367 shutdown_connections 2024-04-17T07:33:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.156+0000 7fbc28800640 1 -- 172.21.15.67:0/1151161367 wait complete. 2024-04-17T07:33:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.158+0000 7fbc28800640 1 Processor -- start 2024-04-17T07:33:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.158+0000 7fbc28800640 1 -- start start 2024-04-17T07:33:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc20072bb0 0x7fbc20141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc201e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc20145ea0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200cf5f0 con 0x7fbc20072bb0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc28800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200cd3e0 con 0x7fbc20145ac0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc25400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.159+0000 7fbc25400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34222/0 (socket says 172.21.15.67:34222) 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 -- 172.21.15.67:0/921679617 learned_addr learned my addr 172.21.15.67:0/921679617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25e00640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc20072bb0 0x7fbc20141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc26800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc201e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 msgr2=0x7fbc201e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc201e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc20072bb0 msgr2=0x7fbc20141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc20072bb0 0x7fbc20141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 -- 172.21.15.67:0/921679617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2015ec40 con 0x7fbc200ce3b0 2024-04-17T07:33:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.160+0000 7fbc25400640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9e0 tx=0x7fbc1000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.161+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc10017070 con 0x7fbc200ce3b0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.161+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1000c9d0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.161+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000cdc0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.161+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2015ef20 con 0x7fbc200ce3b0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.161+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2015f4e0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.162+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc20068ca0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.163+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7fbc1001a070 con 0x7fbc200ce3b0 2024-04-17T07:33:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.164+0000 7fbc1a000640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 0x7fbc0406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.164+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7fbc1008c4b0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.164+0000 7fbc25e00640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 msgr2=0x7fbc0406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.164+0000 7fbc25e00640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 0x7fbc0406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.173+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbc10059d30 con 0x7fbc200ce3b0 2024-04-17T07:33:05.185 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.186+0000 7f6c24a40040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:05.200 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.200+0000 7fb4749bc040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:05.238 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.239+0000 7f6c24a40040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:05.250 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.250+0000 7fb4749bc040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:05.299 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.299+0000 7f6c24a40040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:05.353 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.353+0000 7f6c24a40040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.365+0000 7fbc25e00640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 msgr2=0x7fbc0406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.365+0000 7fbc25e00640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 0x7fbc0406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.374+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbc200cc6c0 con 0x7fbc200ce3b0 2024-04-17T07:33:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.374+0000 7fbc1a000640 1 -- 172.21.15.67:0/921679617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) ==== 104+0+11 (secure 0 0 0) 0x7fbc1005e030 con 0x7fbc200ce3b0 2024-04-17T07:33:05.374 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:33:05.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 msgr2=0x7fbc0406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:05.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 0x7fbc0406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 msgr2=0x7fbc2015e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9e0 tx=0x7fbc1000eea0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 shutdown_connections 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fbc0406cb30 0x7fbc0406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc20145ac0 0x7fbc201e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc200ce3b0 0x7fbc2015e200 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 --2- 172.21.15.67:0/921679617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc20072bb0 0x7fbc20141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 >> 172.21.15.67:0/921679617 conn(0x7fbc20067ee0 msgr2=0x7fbc200cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 shutdown_connections 2024-04-17T07:33:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.377+0000 7fbc28800640 1 -- 172.21.15.67:0/921679617 wait complete. 2024-04-17T07:33:05.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:33:05.636 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.636+0000 7fb4749bc040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:05.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.650+0000 7f9f69800640 1 Processor -- start 2024-04-17T07:33:05.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.651+0000 7f9f69800640 1 -- start start 2024-04-17T07:33:05.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f69800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f6415ec00 0x7f9f640b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f69800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f640d1a80 0x7f9f640cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f69800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f640d4740 con 0x7f9f640cf870 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f69800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f640d3610 con 0x7f9f6415ec00 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f69800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f640d2a00 con 0x7f9f640d1a80 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f5be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f640cf870 0x7f9f640d1370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49936/0 (socket says 172.21.15.67:49936) 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f5be00640 1 -- 172.21.15.67:0/1494792389 learned_addr learned my addr 172.21.15.67:0/1494792389 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.652+0000 7f9f62a00640 1 --2- 172.21.15.67:0/1494792389 >> v2:172.21.15.67:3300/0 conn(0x7f9f6415ec00 0x7f9f640b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.653+0000 7f9f63400640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f640d1a80 0x7f9f640cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.653+0000 7f9f62a00640 1 -- 172.21.15.67:0/1494792389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f640d1a80 msgr2=0x7f9f640cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.653+0000 7f9f62a00640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f640d1a80 0x7f9f640cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.653+0000 7f9f62a00640 1 -- 172.21.15.67:0/1494792389 >> v1:172.21.15.67:6789/0 conn(0x7f9f640cf870 legacy=0x7f9f640d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.653+0000 7f9f62a00640 1 -- 172.21.15.67:0/1494792389 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f640d2ce0 con 0x7f9f6415ec00 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.654+0000 7f9f62a00640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6415ec00 0x7f9f640b7b60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9f540097a0 tx=0x7f9f5402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e96069f7cc3e668 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.654+0000 7f9f62000640 1 -- 172.21.15.67:0/1494792389 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54031780 con 0x7f9f6415ec00 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.654+0000 7f9f62000640 1 -- 172.21.15.67:0/1494792389 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f54037070 con 0x7f9f6415ec00 2024-04-17T07:33:05.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.654+0000 7f9f62000640 1 -- 172.21.15.67:0/1494792389 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54031b40 con 0x7f9f6415ec00 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 -- 172.21.15.67:0/1494792389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6415ec00 msgr2=0x7f9f640b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6415ec00 0x7f9f640b7b60 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9f540097a0 tx=0x7f9f5402b820 comp rx=0 tx=0).stop 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 -- 172.21.15.67:0/1494792389 shutdown_connections 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f640d1a80 0x7f9f640cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 --2- 172.21.15.67:0/1494792389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6415ec00 0x7f9f640b7b60 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 -- 172.21.15.67:0/1494792389 >> 172.21.15.67:0/1494792389 conn(0x7f9f6406ccb0 msgr2=0x7f9f6406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 -- 172.21.15.67:0/1494792389 shutdown_connections 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.655+0000 7f9f69800640 1 -- 172.21.15.67:0/1494792389 wait complete. 2024-04-17T07:33:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.657+0000 7f9f69800640 1 Processor -- start 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.657+0000 7f9f69800640 1 -- start start 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 0x7f9f640b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f6415ec00 0x7f9f6415cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6415efe0 con 0x7f9f640d1a80 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640d32a0 con 0x7f9f640cf870 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f69800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6415f160 con 0x7f9f6415ec00 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34250/0 (socket says 172.21.15.67:34250) 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.658+0000 7f9f5be00640 1 -- 172.21.15.67:0/4138204934 learned_addr learned my addr 172.21.15.67:0/4138204934 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f63400640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f6415ec00 0x7f9f6415cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f62a00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 0x7f9f640b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f6415ec00 msgr2=0x7f9f6415cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f6415ec00 0x7f9f6415cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 msgr2=0x7f9f640b94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 0x7f9f640b94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 -- 172.21.15.67:0/4138204934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f640c8030 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f62a00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 0x7f9f640b94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5be00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c430 tx=0x7f9f4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c017070 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4c005ce0 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c0072f0 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.659+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f640c8310 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.660+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f640c88d0 con 0x7f9f640d1a80 2024-04-17T07:33:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.661+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f64068b90 con 0x7f9f640d1a80 2024-04-17T07:33:05.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.662+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7f9f4c01a070 con 0x7f9f640d1a80 2024-04-17T07:33:05.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.662+0000 7f9f5aa00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 0x7f9f4006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:05.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.663+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f9f4c08b5b0 con 0x7f9f640d1a80 2024-04-17T07:33:05.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.663+0000 7f9f62a00640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 msgr2=0x7f9f4006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:05.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.663+0000 7f9f62a00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 0x7f9f4006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:05.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.667+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9f4c059480 con 0x7f9f640d1a80 2024-04-17T07:33:05.767 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.768+0000 7f6c24a40040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:05.796 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.796+0000 7fb4749bc040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.864+0000 7f9f62a00640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 msgr2=0x7f9f4006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.864+0000 7f9f62a00640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 0x7f9f4006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:05.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.870+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9f6415e0f0 con 0x7f9f640d1a80 2024-04-17T07:33:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.871+0000 7f9f5aa00640 1 -- 172.21.15.67:0/4138204934 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) ==== 100+0+11 (secure 0 0 0) 0x7f9f4c058d80 con 0x7f9f640d1a80 2024-04-17T07:33:05.870 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:33:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 msgr2=0x7f9f4006eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 0x7f9f4006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 msgr2=0x7f9f640b99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00c430 tx=0x7f9f4c00c8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 shutdown_connections 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f9f4006cb30 0x7f9f4006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f6415ec00 0x7f9f6415cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640d1a80 0x7f9f640b99f0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 --2- 172.21.15.67:0/4138204934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f640cf870 0x7f9f640b94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 >> 172.21.15.67:0/4138204934 conn(0x7f9f6406ccb0 msgr2=0x7f9f640b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 shutdown_connections 2024-04-17T07:33:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:05.874+0000 7f9f69800640 1 -- 172.21.15.67:0/4138204934 wait complete. 2024-04-17T07:33:05.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:33:05.906 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.906+0000 7fb4749bc040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:05.936 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:05.937+0000 7f6c24a40040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:05.957 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:05.957+0000 7fb4749bc040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:06.058 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.059+0000 7f6c24a40040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:06.082 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:06.082+0000 7fb4749bc040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:06.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.100+0000 7f9204a00640 1 Processor -- start 2024-04-17T07:33:06.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.100+0000 7f9204a00640 1 -- start start 2024-04-17T07:33:06.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f9204a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f92000cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f9204a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9200072570 0x7f92000ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f9204a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920015d060 con 0x7f92000b63c0 2024-04-17T07:33:06.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f9204a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f92000cfe90 con 0x7f9200072570 2024-04-17T07:33:06.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f9204a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f920015dc70 con 0x7f92000cbc00 2024-04-17T07:33:06.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f91f7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92000b63c0 0x7f9200071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49948/0 (socket says 172.21.15.67:49948) 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.102+0000 7f91f7e00640 1 -- 172.21.15.67:0/567171820 learned_addr learned my addr 172.21.15.67:0/567171820 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91ff200640 1 --2- 172.21.15.67:0/567171820 >> v2:172.21.15.67:3300/0 conn(0x7f9200072570 0x7f92000ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91fe800640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f92000cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91fde00640 1 -- 172.21.15.67:0/567171820 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594063851 0 0) 0x7f920015d060 con 0x7f92000b63c0 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91fde00640 1 -- 172.21.15.67:0/567171820 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91e4003520 con 0x7f92000b63c0 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91ff200640 1 -- 172.21.15.67:0/567171820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 msgr2=0x7f92000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91ff200640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f92000cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91ff200640 1 -- 172.21.15.67:0/567171820 >> v1:172.21.15.67:6789/0 conn(0x7f92000b63c0 legacy=0x7f9200071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91ff200640 1 -- 172.21.15.67:0/567171820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920015d340 con 0x7f9200072570 2024-04-17T07:33:06.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.103+0000 7f91fe800640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f92000cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:06.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.104+0000 7f91ff200640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f92000ce610 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009d90 tx=0x7f91f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325ce53a9827e95e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:06.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.105+0000 7f91fde00640 1 -- 172.21.15.67:0/567171820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f8039070 con 0x7f9200072570 2024-04-17T07:33:06.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.105+0000 7f91fde00640 1 -- 172.21.15.67:0/567171820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91f802cc60 con 0x7f9200072570 2024-04-17T07:33:06.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.105+0000 7f91fde00640 1 -- 172.21.15.67:0/567171820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f8034640 con 0x7f9200072570 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.105+0000 7f9204a00640 1 -- 172.21.15.67:0/567171820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 msgr2=0x7f92000ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.105+0000 7f9204a00640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f92000ce610 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009d90 tx=0x7f91f802beb0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 -- 172.21.15.67:0/567171820 shutdown_connections 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f92000ce610 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 --2- 172.21.15.67:0/567171820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f92000cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 -- 172.21.15.67:0/567171820 >> 172.21.15.67:0/567171820 conn(0x7f920006bef0 msgr2=0x7f92000b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 -- 172.21.15.67:0/567171820 shutdown_connections 2024-04-17T07:33:06.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.106+0000 7f9204a00640 1 -- 172.21.15.67:0/567171820 wait complete. 2024-04-17T07:33:06.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.108+0000 7f9204a00640 1 Processor -- start 2024-04-17T07:33:06.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.108+0000 7f9204a00640 1 -- start start 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 0x7f9200148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f920006fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92000d02d0 con 0x7f9200072570 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f920015d900 con 0x7f92000b63c0 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f9204a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92000cbfe0 con 0x7f92000cbc00 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34262/0 (socket says 172.21.15.67:34262) 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 -- 172.21.15.67:0/552786904 learned_addr learned my addr 172.21.15.67:0/552786904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91f7e00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 0x7f9200148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 msgr2=0x7f920006fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f920006fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 msgr2=0x7f9200148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 0x7f9200148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91fe800640 1 -- 172.21.15.67:0/552786904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920006f210 con 0x7f9200072570 2024-04-17T07:33:06.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.109+0000 7f91f7e00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 0x7f9200148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:06.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f91fe800640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f91f000d7d0 tx=0x7f91f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f0004030 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91f00041d0 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00096a0 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f920006e600 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.110+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f920006ebc0 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.112+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9200068a10 con 0x7f9200072570 2024-04-17T07:33:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.112+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7f91f0009840 con 0x7f9200072570 2024-04-17T07:33:06.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.113+0000 7f91f6a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 0x7f91dc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.113+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f91f008c3b0 con 0x7f9200072570 2024-04-17T07:33:06.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.113+0000 7f91f7e00640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 msgr2=0x7f91dc06eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:06.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.114+0000 7f91f7e00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 0x7f91dc06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:06.115 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.116+0000 7f6c24a40040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.118+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f91f005a320 con 0x7f9200072570 2024-04-17T07:33:06.193 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:06.193+0000 7fb4749bc040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:06.250 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.251+0000 7f6c24a40040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.315+0000 7f91f7e00640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 msgr2=0x7f91dc06eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.315+0000 7f91f7e00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 0x7f91dc06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.332+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9200153820 con 0x7f9200072570 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.333+0000 7f91f6a00640 1 -- 172.21.15.67:0/552786904 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v128) ==== 80+0+476 (secure 0 0 0) 0x7f91f0059c20 con 0x7f9200072570 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/757480182 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2989376162 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/2517863062 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/2517863062 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3883423100 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/410647397 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.332 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2591672672 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.333 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1399211335 2024-04-18T07:32:49.473454+0000 2024-04-17T07:33:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 msgr2=0x7f91dc06eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 0x7f91dc06eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 msgr2=0x7f920006df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f91f000d7d0 tx=0x7f91f000dc90 comp rx=0 tx=0).stop 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 shutdown_connections 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f91dc06cb30 0x7f91dc06eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92000cbc00 0x7f920006fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92000b63c0 0x7f9200148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 --2- 172.21.15.67:0/552786904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9200072570 0x7f920006df70 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 >> 172.21.15.67:0/552786904 conn(0x7f920006bef0 msgr2=0x7f9200159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:06.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.335+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 shutdown_connections 2024-04-17T07:33:06.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.336+0000 7f9204a00640 1 -- 172.21.15.67:0/552786904 wait complete. 2024-04-17T07:33:06.352 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:33:06.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:06.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:06.365 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.366+0000 7f6c24a40040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:06.368 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:06.368+0000 7fb4749bc040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:06.417 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:06.418+0000 7fb4749bc040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:06.472 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:33:06.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:06.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:33:06.519 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:06.520 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:06.520 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:06.556 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.556+0000 7f6c24a40040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:06.558 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:06.559+0000 7fb4749bc040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:06.570 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:06.571 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:06.571 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:06.571 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:06.608 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.609+0000 7f6c24a40040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:06.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:33:06.622 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:06.622 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:06.622 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:06.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:06.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:06.755 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:06.756+0000 7f6c24a40040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.880+0000 7fea96000640 1 Processor -- start 2024-04-17T07:33:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.880+0000 7fea96000640 1 -- start start 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea96000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fea900cd020 0x7fea900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea96000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea900ceb00 con 0x7fea9006e020 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea96000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea900cede0 con 0x7fea9006f670 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea96000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fea900cf0c0 con 0x7fea900cd020 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea8fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea9006e020 0x7fea9006e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49950/0 (socket says 172.21.15.67:49950) 2024-04-17T07:33:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea8fe00640 1 -- 172.21.15.67:0/1134158761 learned_addr learned my addr 172.21.15.67:0/1134158761 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea8ea00640 1 --2- 172.21.15.67:0/1134158761 >> v2:172.21.15.187:3300/0 conn(0x7fea900cd020 0x7fea900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.882+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458234435 0 0) 0x7fea900ceb00 con 0x7fea9006e020 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea74003520 con 0x7fea9006e020 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 970324703 0 0) 0x7fea900cede0 con 0x7fea9006f670 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea900ceb00 con 0x7fea9006f670 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 201937224 0 0) 0x7fea74003520 con 0x7fea9006e020 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 >> v2:172.21.15.187:3300/0 conn(0x7fea900cd020 msgr2=0x7fea900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 --2- 172.21.15.67:0/1134158761 >> v2:172.21.15.187:3300/0 conn(0x7fea900cd020 0x7fea900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 >> v1:172.21.15.187:6789/0 conn(0x7fea9006f670 legacy=0x7fea9006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea900cf3a0 con 0x7fea9006e020 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fea880019f0 con 0x7fea9006e020 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 >> v1:172.21.15.67:6789/0 conn(0x7fea9006e020 legacy=0x7fea9006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea74003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea74004800 0x7fea74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea74005210 0x7fea7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea900cf0c0 con 0x7fea74003b90 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fea7400c140 con 0x7fea74004800 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fea7400c340 con 0x7fea74005210 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.883+0000 7fea8f400640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea74003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8fe00640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea74005210 0x7fea7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8f400640 1 -- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea74005210 msgr2=0x7fea7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8f400640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea74005210 0x7fea7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8f400640 1 -- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea74004800 msgr2=0x7fea74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8f400640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea74004800 0x7fea74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8f400640 1 -- 172.21.15.67:0/1134158761 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea74003520 con 0x7fea74003b90 2024-04-17T07:33:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8fe00640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea74005210 0x7fea7400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.884+0000 7fea8fe00640 1 -- 172.21.15.67:0/1134158761 reap_dead start 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.885+0000 7fea8f400640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea74003f90 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fea800018a0 tx=0x7fea74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1081b0ff23103b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.885+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8002f050 con 0x7fea74003b90 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.885+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea800052d0 con 0x7fea74003b90 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.885+0000 7fea8e000640 1 -- 172.21.15.67:0/1134158761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea800014f0 con 0x7fea74003b90 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 -- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 msgr2=0x7fea74003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea74003f90 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fea800018a0 tx=0x7fea74005830 comp rx=0 tx=0).stop 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 -- 172.21.15.67:0/1134158761 shutdown_connections 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 --2- 172.21.15.67:0/1134158761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea74003f90 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 -- 172.21.15.67:0/1134158761 >> 172.21.15.67:0/1134158761 conn(0x7fea9006bf50 msgr2=0x7fea900b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 -- 172.21.15.67:0/1134158761 shutdown_connections 2024-04-17T07:33:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.886+0000 7fea96000640 1 -- 172.21.15.67:0/1134158761 wait complete. 2024-04-17T07:33:06.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.888+0000 7fea96000640 1 Processor -- start 2024-04-17T07:33:06.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.888+0000 7fea96000640 1 -- start start 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea901f5860 0x7fea901550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea9015aaa0 0x7fea9015af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea900ce8a0 con 0x7fea74003b90 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fea900ce720 con 0x7fea901f5860 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea96000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fea9015cb00 con 0x7fea9015aaa0 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea8f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea8f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34274/0 (socket says 172.21.15.67:34274) 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.889+0000 7fea8f400640 1 -- 172.21.15.67:0/3156095441 learned_addr learned my addr 172.21.15.67:0/3156095441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8ea00640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea901f5860 0x7fea901550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8fe00640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea9015aaa0 0x7fea9015af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea9015aaa0 msgr2=0x7fea9015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea9015aaa0 0x7fea9015af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea901f5860 msgr2=0x7fea901550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea901f5860 0x7fea901550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 -- 172.21.15.67:0/3156095441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea90155630 con 0x7fea74003b90 2024-04-17T07:33:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea8f400640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fea80035040 tx=0x7fea800012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.890+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80042070 con 0x7fea74003b90 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.891+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea80005480 con 0x7fea74003b90 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.891+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea800014f0 con 0x7fea74003b90 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.891+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9015d530 con 0x7fea74003b90 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.891+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9015daf0 con 0x7fea74003b90 2024-04-17T07:33:06.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.892+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea900cc600 con 0x7fea74003b90 2024-04-17T07:33:06.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.893+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7fea8003c040 con 0x7fea74003b90 2024-04-17T07:33:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.894+0000 7fea87400640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 0x7fea7006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.894+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7fea80007680 con 0x7fea74003b90 2024-04-17T07:33:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.894+0000 7fea8ea00640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 msgr2=0x7fea7006f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:06.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.894+0000 7fea8ea00640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 0x7fea7006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:06.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:06.898+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fea900cc600 con 0x7fea74003b90 2024-04-17T07:33:07.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.095+0000 7fea8ea00640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 msgr2=0x7fea7006f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:07.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.095+0000 7fea8ea00640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 0x7fea7006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.141+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea90068b90 con 0x7fea74003b90 2024-04-17T07:33:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.143+0000 7fea87400640 1 -- 172.21.15.67:0/3156095441 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7fea8004b030 con 0x7fea74003b90 2024-04-17T07:33:07.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:07.142 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:33:07.143 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:33:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.145+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 msgr2=0x7fea7006f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.145+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 0x7fea7006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.145+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 msgr2=0x7fea9015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.145+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fea80035040 tx=0x7fea800012d0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 shutdown_connections 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7fea7006cb60 0x7fea7006f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fea9015aaa0 0x7fea9015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea901f5860 0x7fea901550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 --2- 172.21.15.67:0/3156095441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea74003b90 0x7fea9015cea0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 >> 172.21.15.67:0/3156095441 conn(0x7fea9006bf50 msgr2=0x7fea900cfe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 shutdown_connections 2024-04-17T07:33:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.146+0000 7fea96000640 1 -- 172.21.15.67:0/3156095441 wait complete. 2024-04-17T07:33:07.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:07.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.358+0000 7f0715200640 1 Processor -- start 2024-04-17T07:33:07.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.359+0000 7f0715200640 1 -- start start 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f0715200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07100cf640 0x7f07100cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f0715200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f071015ed10 0x7f07100d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f0715200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07100cff60 con 0x7f07100d2ab0 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f0715200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f07100d3c30 con 0x7f07100cf640 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f0715200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f07100d3020 con 0x7f071015ed10 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f070de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07100cf640 0x7f07100cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f070e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07100d2ab0 0x7f07100b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49952/0 (socket says 172.21.15.67:49952) 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.360+0000 7f070e800640 1 -- 172.21.15.67:0/2756636377 learned_addr learned my addr 172.21.15.67:0/2756636377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.361+0000 7f070d400640 1 --2- 172.21.15.67:0/2756636377 >> v2:172.21.15.187:3300/0 conn(0x7f071015ed10 0x7f07100d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.361+0000 7f070de00640 1 -- 172.21.15.67:0/2756636377 >> v2:172.21.15.187:3300/0 conn(0x7f071015ed10 msgr2=0x7f07100d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.361+0000 7f070de00640 1 --2- 172.21.15.67:0/2756636377 >> v2:172.21.15.187:3300/0 conn(0x7f071015ed10 0x7f07100d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.361+0000 7f070de00640 1 -- 172.21.15.67:0/2756636377 >> v1:172.21.15.67:6789/0 conn(0x7f07100d2ab0 legacy=0x7f07100b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.361+0000 7f070de00640 1 -- 172.21.15.67:0/2756636377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07100d3300 con 0x7f07100cf640 2024-04-17T07:33:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.362+0000 7f070de00640 1 --2- 172.21.15.67:0/2756636377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07100cf640 0x7f07100cfa20 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f071008ddd0 tx=0x7f070002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8c8e4d4d7440d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.362+0000 7f070ca00640 1 -- 172.21.15.67:0/2756636377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700034070 con 0x7f07100cf640 2024-04-17T07:33:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.362+0000 7f070ca00640 1 -- 172.21.15.67:0/2756636377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0700030610 con 0x7f07100cf640 2024-04-17T07:33:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.362+0000 7f070ca00640 1 -- 172.21.15.67:0/2756636377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07000309e0 con 0x7f07100cf640 2024-04-17T07:33:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 -- 172.21.15.67:0/2756636377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07100cf640 msgr2=0x7f07100cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 --2- 172.21.15.67:0/2756636377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07100cf640 0x7f07100cfa20 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f071008ddd0 tx=0x7f070002b960 comp rx=0 tx=0).stop 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 -- 172.21.15.67:0/2756636377 shutdown_connections 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 --2- 172.21.15.67:0/2756636377 >> v2:172.21.15.187:3300/0 conn(0x7f071015ed10 0x7f07100d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 --2- 172.21.15.67:0/2756636377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07100cf640 0x7f07100cfa20 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 -- 172.21.15.67:0/2756636377 >> 172.21.15.67:0/2756636377 conn(0x7f071006cbd0 msgr2=0x7f071006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 -- 172.21.15.67:0/2756636377 shutdown_connections 2024-04-17T07:33:07.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.363+0000 7f0715200640 1 -- 172.21.15.67:0/2756636377 wait complete. 2024-04-17T07:33:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.365+0000 7f0715200640 1 Processor -- start 2024-04-17T07:33:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.365+0000 7f0715200640 1 -- start start 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07100cf640 0x7f0710159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07100d2ab0 0x7f07100b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07100d25b0 con 0x7f071015ed10 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07100d38c0 con 0x7f07100d2ab0 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f0715200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07100d4600 con 0x7f07100cf640 2024-04-17T07:33:07.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f070e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f070e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34298/0 (socket says 172.21.15.67:34298) 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f070e800640 1 -- 172.21.15.67:0/1182872907 learned_addr learned my addr 172.21.15.67:0/1182872907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.366+0000 7f070d400640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07100d2ab0 0x7f07100b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070de00640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07100cf640 0x7f0710159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07100cf640 msgr2=0x7f0710159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07100cf640 0x7f0710159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07100d2ab0 msgr2=0x7f07100b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07100d2ab0 0x7f07100b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 -- 172.21.15.67:0/1182872907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071015bdf0 con 0x7f071015ed10 2024-04-17T07:33:07.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f070e800640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f070800cbc0 tx=0x7f070800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708017070 con 0x7f071015ed10 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070800ace0 con 0x7f071015ed10 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.367+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708012600 con 0x7f071015ed10 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.368+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071015c0d0 con 0x7f071015ed10 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.368+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071015c690 con 0x7f071015ed10 2024-04-17T07:33:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.369+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0710068b90 con 0x7f071015ed10 2024-04-17T07:33:07.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.371+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7f070801a070 con 0x7f071015ed10 2024-04-17T07:33:07.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.372+0000 7f0706000640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 0x7f06f006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.372+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f070808bcd0 con 0x7f071015ed10 2024-04-17T07:33:07.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.372+0000 7f070de00640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 msgr2=0x7f06f006f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:07.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.372+0000 7f070de00640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 0x7f06f006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:07.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.376+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0708059680 con 0x7f071015ed10 2024-04-17T07:33:07.402 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.402+0000 7fb4749bc040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.573+0000 7f070de00640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 msgr2=0x7f06f006f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:07.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.573+0000 7f070de00640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 0x7f06f006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:07.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.580+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f071006d020 con 0x7f071015ed10 2024-04-17T07:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.582+0000 7f0706000640 1 -- 172.21.15.67:0/1182872907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+17078 (secure 0 0 0) 0x7f070805d980 con 0x7f071015ed10 2024-04-17T07:33:07.581 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:07.582 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":128,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:32:49.473630+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/757480182":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2989376162":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6833/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6832/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/3883423100":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/410647397":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2591672672":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1399211335":"2024-04-18T07:32:49.473454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 msgr2=0x7f06f006f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 0x7f06f006f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 msgr2=0x7f07100b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f070800cbc0 tx=0x7f070800a430 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 shutdown_connections 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f06f006cb60 0x7f06f006f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f071015ed10 0x7f07100b4810 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07100d2ab0 0x7f07100b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 --2- 172.21.15.67:0/1182872907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07100cf640 0x7f0710159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.584+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 >> 172.21.15.67:0/1182872907 conn(0x7f071006cbd0 msgr2=0x7f0710160a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.585+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 shutdown_connections 2024-04-17T07:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.585+0000 7f0715200640 1 -- 172.21.15.67:0/1182872907 wait complete. 2024-04-17T07:33:07.602 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:32:32.505992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:32:33.573434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:07.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:07.623 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:07.624+0000 7f6c24a40040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:07.708 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.708+0000 7fb4749bc040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:07.766 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.766+0000 7fb4749bc040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.805+0000 7f263be00640 1 Processor -- start 2024-04-17T07:33:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.806+0000 7f263be00640 1 -- start start 2024-04-17T07:33:07.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f263c1622b0 0x7f263c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263c0b5110 con 0x7f263c0b4c70 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f263c0b53f0 con 0x7f263c1622b0 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f263c0b56d0 con 0x7f263c0d2ff0 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.807+0000 7f263b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f263c1622b0 0x7f263c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f263c1622b0 0x7f263c0cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34304/0 (socket says 172.21.15.67:34304) 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 -- 172.21.15.67:0/1479891443 learned_addr learned my addr 172.21.15.67:0/1479891443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263aa00640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 -- 172.21.15.67:0/1479891443 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 msgr2=0x7f263c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c0b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 -- 172.21.15.67:0/1479891443 >> v1:172.21.15.187:6789/0 conn(0x7f263c0b4c70 legacy=0x7f263c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.808+0000 7f263b400640 1 -- 172.21.15.67:0/1479891443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c0cf340 con 0x7f263c1622b0 2024-04-17T07:33:07.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.809+0000 7f263b400640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0cfc40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2634009e50 tx=0x7f263402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2a33175258dd58 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.809+0000 7f263a000640 1 -- 172.21.15.67:0/1479891443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263402ca70 con 0x7f263c1622b0 2024-04-17T07:33:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.809+0000 7f263a000640 1 -- 172.21.15.67:0/1479891443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f263402cc10 con 0x7f263c1622b0 2024-04-17T07:33:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.809+0000 7f263a000640 1 -- 172.21.15.67:0/1479891443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340343f0 con 0x7f263c1622b0 2024-04-17T07:33:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 -- 172.21.15.67:0/1479891443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 msgr2=0x7f263c0cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0cfc40 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2634009e50 tx=0x7f263402bed0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 -- 172.21.15.67:0/1479891443 shutdown_connections 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0cfc40 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 --2- 172.21.15.67:0/1479891443 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c0b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 -- 172.21.15.67:0/1479891443 >> 172.21.15.67:0/1479891443 conn(0x7f263c09da80 msgr2=0x7f263c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 -- 172.21.15.67:0/1479891443 shutdown_connections 2024-04-17T07:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.810+0000 7f263be00640 1 -- 172.21.15.67:0/1479891443 wait complete. 2024-04-17T07:33:07.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.812+0000 7f263be00640 1 Processor -- start 2024-04-17T07:33:07.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.812+0000 7f263be00640 1 -- start start 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f263c0b4c70 0x7f263c0cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f263c0d3550 con 0x7f263c1622b0 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f263c0d33d0 con 0x7f263c0b4c70 2024-04-17T07:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f263c0d36d0 con 0x7f263c0d2ff0 2024-04-17T07:33:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34320/0 (socket says 172.21.15.67:34320) 2024-04-17T07:33:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.813+0000 7f263b400640 1 -- 172.21.15.67:0/146148553 learned_addr learned my addr 172.21.15.67:0/146148553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f2633e00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263aa00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f263c0b4c70 0x7f263c0cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 -- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 msgr2=0x7f263c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c15fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 -- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f263c0b4c70 msgr2=0x7f263c0cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f263c0b4c70 0x7f263c0cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 -- 172.21.15.67:0/146148553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c1572e0 con 0x7f263c1622b0 2024-04-17T07:33:07.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f2633e00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c15fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263b400640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2634036040 tx=0x7f2634004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634043020 con 0x7f263c1622b0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f263403e450 con 0x7f263c1622b0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634009510 con 0x7f263c1622b0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263c0b8a00 con 0x7f263c1622b0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.814+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263c0b92a0 con 0x7f263c1622b0 2024-04-17T07:33:07.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.816+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263c0755d0 con 0x7f263c1622b0 2024-04-17T07:33:07.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.817+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 64) ==== 91256+0+0 (secure 0 0 0) 0x7f2634005d10 con 0x7f263c1622b0 2024-04-17T07:33:07.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.818+0000 7f2633400640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 0x7f261806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:07.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.818+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6053+0+0 (secure 0 0 0) 0x7f26340aecc0 con 0x7f263c1622b0 2024-04-17T07:33:07.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.818+0000 7f263aa00640 1 -- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 msgr2=0x7f261806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2982226538 2024-04-17T07:33:07.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.818+0000 7f263aa00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 0x7f261806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:07.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.822+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f26340b5080 con 0x7f263c1622b0 2024-04-17T07:33:07.898 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.897+0000 7fb4749bc040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:07.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.918+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 65) ==== 90751+0+0 (secure 0 0 0) 0x7f2634046020 con 0x7f263c1622b0 2024-04-17T07:33:07.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.919+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 msgr2=0x7f261806eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:07.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:07.919+0000 7f2633400640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 0x7f261806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:07.959 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:07.960+0000 7f6c24a40040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:07.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.991+0000 7fb3efa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.991+0000 7fb3efa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.991+0000 7fb3efa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.991+0000 7fb3efa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.991 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.991+0000 7fb3efa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.992+0000 7fb3ed200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.992+0000 7fb3ed200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.992+0000 7fb3ed200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.992+0000 7fb3ed200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:07.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:07.992+0000 7fb3ed200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:08.022 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:08.023+0000 7f6c24a40040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.039+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f263c0b4460 con 0x7f263c1622b0 2024-04-17T07:33:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.040+0000 7f2633400640 1 -- 172.21.15.67:0/146148553 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7f263407cb90 con 0x7f263c1622b0 2024-04-17T07:33:08.039 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:08.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 msgr2=0x7f263c0ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2634036040 tx=0x7f2634004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 shutdown_connections 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:6832/2982226538,v1:172.21.15.187:6833/2982226538] conn(0x7f261806cb30 0x7f261806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f263c1622b0 0x7f263c0ca670 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f263c0d2ff0 0x7f263c15fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 --2- 172.21.15.67:0/146148553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f263c0b4c70 0x7f263c0cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 >> 172.21.15.67:0/146148553 conn(0x7f263c09da80 msgr2=0x7f263c0b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.042+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 shutdown_connections 2024-04-17T07:33:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.043+0000 7f263be00640 1 -- 172.21.15.67:0/146148553 wait complete. 2024-04-17T07:33:08.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:33:08.163 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:08.164+0000 7f6c24a40040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.314+0000 7f698ce00640 1 Processor -- start 2024-04-17T07:33:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.315+0000 7f698ce00640 1 -- start start 2024-04-17T07:33:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.316+0000 7f698ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69880d33e0 0x7f69880cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.316+0000 7f698ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880cf8e0 0x7f69880b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.316+0000 7f698ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69880d00b0 con 0x7f698815ecb0 2024-04-17T07:33:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.316+0000 7f698ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f69880d0c30 con 0x7f69880d33e0 2024-04-17T07:33:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.316+0000 7f698ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69880d0390 con 0x7f69880cf8e0 2024-04-17T07:33:08.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6987400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f698815ecb0 0x7f69880d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49962/0 (socket says 172.21.15.67:49962) 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6987400640 1 -- 172.21.15.67:0/3569579709 learned_addr learned my addr 172.21.15.67:0/3569579709 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986a00640 1 --2- 172.21.15.67:0/3569579709 >> v2:172.21.15.67:3300/0 conn(0x7f69880d33e0 0x7f69880cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6987e00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880cf8e0 0x7f69880b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986a00640 1 -- 172.21.15.67:0/3569579709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880cf8e0 msgr2=0x7f69880b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986a00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880cf8e0 0x7f69880b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986a00640 1 -- 172.21.15.67:0/3569579709 >> v1:172.21.15.67:6789/0 conn(0x7f698815ecb0 legacy=0x7f69880d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986000640 1 -- 172.21.15.67:0/3569579709 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436331593 0 0) 0x7f69880d00b0 con 0x7f698815ecb0 2024-04-17T07:33:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.317+0000 7f6986a00640 1 -- 172.21.15.67:0/3569579709 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880d0670 con 0x7f69880d33e0 2024-04-17T07:33:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.318+0000 7f6986a00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880cf3a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f69700097a0 tx=0x7f697002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc7ca97786fdc55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.319+0000 7f6986000640 1 -- 172.21.15.67:0/3569579709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69700315a0 con 0x7f69880d33e0 2024-04-17T07:33:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.319+0000 7f6986000640 1 -- 172.21.15.67:0/3569579709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970037070 con 0x7f69880d33e0 2024-04-17T07:33:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.319+0000 7f6986000640 1 -- 172.21.15.67:0/3569579709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970031960 con 0x7f69880d33e0 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.319+0000 7f698ce00640 1 -- 172.21.15.67:0/3569579709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 msgr2=0x7f69880cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.319+0000 7f698ce00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880cf3a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f69700097a0 tx=0x7f697002b840 comp rx=0 tx=0).stop 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 -- 172.21.15.67:0/3569579709 shutdown_connections 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880cf8e0 0x7f69880b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 --2- 172.21.15.67:0/3569579709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880cf3a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 -- 172.21.15.67:0/3569579709 >> 172.21.15.67:0/3569579709 conn(0x7f698808bff0 msgr2=0x7f69880d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 -- 172.21.15.67:0/3569579709 shutdown_connections 2024-04-17T07:33:08.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.320+0000 7f698ce00640 1 -- 172.21.15.67:0/3569579709 wait complete. 2024-04-17T07:33:08.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.321+0000 7f698ce00640 1 Processor -- start 2024-04-17T07:33:08.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.322+0000 7f698ce00640 1 -- start start 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 0x7f69880b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f698815ecb0 0x7f69880b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69880cfd00 con 0x7f69880d33e0 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f698815f090 con 0x7f69880cf8e0 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f698ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69880b7b00 con 0x7f698815ecb0 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34326/0 (socket says 172.21.15.67:34326) 2024-04-17T07:33:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 -- 172.21.15.67:0/1210632588 learned_addr learned my addr 172.21.15.67:0/1210632588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6987400640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 0x7f69880b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 -- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f698815ecb0 msgr2=0x7f69880b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f698815ecb0 0x7f69880b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 -- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 msgr2=0x7f69880b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 0x7f69880b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6986a00640 1 -- 172.21.15.67:0/1210632588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880b7de0 con 0x7f69880d33e0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.323+0000 7f6987400640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 0x7f69880b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f6986a00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6970034040 tx=0x7f69700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970032840 con 0x7f69880d33e0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970032a00 con 0x7f69880d33e0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970032df0 con 0x7f69880d33e0 2024-04-17T07:33:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69880b80c0 con 0x7f69880d33e0 2024-04-17T07:33:08.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.324+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69880b8680 con 0x7f69880d33e0 2024-04-17T07:33:08.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.325+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6988068a10 con 0x7f69880d33e0 2024-04-17T07:33:08.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.326+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 65) ==== 90751+0+0 (secure 0 0 0) 0x7f697004e430 con 0x7f69880d33e0 2024-04-17T07:33:08.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.328+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6483+0+0 (secure 0 0 0) 0x7f697002f030 con 0x7f69880d33e0 2024-04-17T07:33:08.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.333+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6970040070 con 0x7f69880d33e0 2024-04-17T07:33:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.528+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69880b8850 con 0x7f69880d33e0 2024-04-17T07:33:08.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.529+0000 7f697f400640 1 -- 172.21.15.67:0/1210632588 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7f697007bde0 con 0x7f69880d33e0 2024-04-17T07:33:08.528 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 msgr2=0x7f69880b8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f6970034040 tx=0x7f69700046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 shutdown_connections 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f698815ecb0 0x7f69880b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880d33e0 0x7f69880b8ba0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 --2- 172.21.15.67:0/1210632588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69880cf8e0 0x7f69880b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.531+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 >> 172.21.15.67:0/1210632588 conn(0x7f698808bff0 msgr2=0x7f698806dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.532+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 shutdown_connections 2024-04-17T07:33:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.532+0000 7f698ce00640 1 -- 172.21.15.67:0/1210632588 wait complete. 2024-04-17T07:33:08.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:33:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.771+0000 7f5241600640 1 Processor -- start 2024-04-17T07:33:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.772+0000 7f5241600640 1 -- start start 2024-04-17T07:33:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.773+0000 7f5241600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f523c15ec20 0x7f523c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.773+0000 7f5241600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f523c0b3b50 0x7f523c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.773+0000 7f5241600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523c0d4f70 con 0x7f523c06cf80 2024-04-17T07:33:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.773+0000 7f5241600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f523c0d5250 con 0x7f523c15ec20 2024-04-17T07:33:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.773+0000 7f5241600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f523c0cf4d0 con 0x7f523c0b3b50 2024-04-17T07:33:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f523c15ec20 0x7f523c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f523c15ec20 0x7f523c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34332/0 (socket says 172.21.15.67:34332) 2024-04-17T07:33:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 -- 172.21.15.67:0/3692994613 learned_addr learned my addr 172.21.15.67:0/3692994613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 -- 172.21.15.67:0/3692994613 >> v2:172.21.15.187:3300/0 conn(0x7f523c0b3b50 msgr2=0x7f523c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:08.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 --2- 172.21.15.67:0/3692994613 >> v2:172.21.15.187:3300/0 conn(0x7f523c0b3b50 0x7f523c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 -- 172.21.15.67:0/3692994613 >> v1:172.21.15.187:6789/0 conn(0x7f523c06cf80 legacy=0x7f523c0cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:08.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.774+0000 7f523be00640 1 -- 172.21.15.67:0/3692994613 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523c0cf7b0 con 0x7f523c15ec20 2024-04-17T07:33:08.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.776+0000 7f523be00640 1 --2- 172.21.15.67:0/3692994613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0d22b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f523c0b71e0 tx=0x7f522c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e4966cc3110fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:08.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.776+0000 7f523aa00640 1 -- 172.21.15.67:0/3692994613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c038070 con 0x7f523c15ec20 2024-04-17T07:33:08.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.776+0000 7f523aa00640 1 -- 172.21.15.67:0/3692994613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522c02f050 con 0x7f523c15ec20 2024-04-17T07:33:08.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.776+0000 7f523aa00640 1 -- 172.21.15.67:0/3692994613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c031970 con 0x7f523c15ec20 2024-04-17T07:33:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 -- 172.21.15.67:0/3692994613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 msgr2=0x7f523c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 --2- 172.21.15.67:0/3692994613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0d22b0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f523c0b71e0 tx=0x7f522c02b7f0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 -- 172.21.15.67:0/3692994613 shutdown_connections 2024-04-17T07:33:08.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 --2- 172.21.15.67:0/3692994613 >> v2:172.21.15.187:3300/0 conn(0x7f523c0b3b50 0x7f523c06ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 --2- 172.21.15.67:0/3692994613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0d22b0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 -- 172.21.15.67:0/3692994613 >> 172.21.15.67:0/3692994613 conn(0x7f523c08bff0 msgr2=0x7f523c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:08.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 -- 172.21.15.67:0/3692994613 shutdown_connections 2024-04-17T07:33:08.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.777+0000 7f5241600640 1 -- 172.21.15.67:0/3692994613 wait complete. 2024-04-17T07:33:08.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.779+0000 7f5241600640 1 Processor -- start 2024-04-17T07:33:08.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.779+0000 7f5241600640 1 -- start start 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 0x7f523c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f523c0b3b50 0x7f523c0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f523c0d0560 con 0x7f523c15ec20 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f523c0d03e0 con 0x7f523c0b3b50 2024-04-17T07:33:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.780+0000 7f5241600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f523c0d10d0 con 0x7f523c06cf80 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34336/0 (socket says 172.21.15.67:34336) 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 -- 172.21.15.67:0/1856142758 learned_addr learned my addr 172.21.15.67:0/1856142758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f523be00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 0x7f523c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f523b400640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f523c0b3b50 0x7f523c0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 -- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 msgr2=0x7f523c0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 0x7f523c0cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 -- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f523c0b3b50 msgr2=0x7f523c0cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f523c0b3b50 0x7f523c0cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f5240c00640 1 -- 172.21.15.67:0/1856142758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523c0cc6e0 con 0x7f523c15ec20 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.781+0000 7f523be00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 0x7f523c0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5240c00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f523400c780 tx=0x7f523400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5234004b10 con 0x7f523c15ec20 2024-04-17T07:33:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5234004cb0 con 0x7f523c15ec20 2024-04-17T07:33:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523400aa90 con 0x7f523c15ec20 2024-04-17T07:33:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523c0cc9c0 con 0x7f523c15ec20 2024-04-17T07:33:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.782+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f523c0cc3f0 con 0x7f523c15ec20 2024-04-17T07:33:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.783+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f523c0ccdb0 con 0x7f523c15ec20 2024-04-17T07:33:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.785+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 65) ==== 90751+0+0 (secure 0 0 0) 0x7f523400ac30 con 0x7f523c15ec20 2024-04-17T07:33:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.785+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6483+0+0 (secure 0 0 0) 0x7f523408bbb0 con 0x7f523c15ec20 2024-04-17T07:33:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.790+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5234020070 con 0x7f523c15ec20 2024-04-17T07:33:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.931+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 66) ==== 90878+0+0 (secure 0 0 0) 0x7f5234055710 con 0x7f523c15ec20 2024-04-17T07:33:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.932+0000 7f5233e00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 0x7f521c0744a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:08.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.937+0000 7f523be00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 0x7f521c0744a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:08.944+0000 7f523be00640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 0x7f521c0744a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f522c035000 tx=0x7f522c008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.009+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f523c068a10 con 0x7f523c15ec20 2024-04-17T07:33:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.010+0000 7f5233e00640 1 -- 172.21.15.67:0/1856142758 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) ==== 100+0+11 (secure 0 0 0) 0x7f523405a0c0 con 0x7f523c15ec20 2024-04-17T07:33:09.009 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.012+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 msgr2=0x7f521c0744a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.012+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 0x7f521c0744a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f522c035000 tx=0x7f522c008030 comp rx=0 tx=0).stop 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.012+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 msgr2=0x7f523c0cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.012+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f523400c780 tx=0x7f523400cc40 comp rx=0 tx=0).stop 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 shutdown_connections 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f523c15ec20 0x7f523c0cbeb0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f523c0b3b50 0x7f523c0cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f521c0720b0 0x7f521c0744a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 --2- 172.21.15.67:0/1856142758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f523c06cf80 0x7f523c0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 >> 172.21.15.67:0/1856142758 conn(0x7f523c08bff0 msgr2=0x7f523c0d2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 shutdown_connections 2024-04-17T07:33:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.013+0000 7f5241600640 1 -- 172.21.15.67:0/1856142758 wait complete. 2024-04-17T07:33:09.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:09.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.240+0000 7f8961200640 1 Processor -- start 2024-04-17T07:33:09.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.240+0000 7f8961200640 1 -- start start 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.241+0000 7f8961200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f895c15ec10 0x7f895c0b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.241+0000 7f8961200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f895c0b2c90 0x7f895c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.241+0000 7f8961200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f895c0ae330 con 0x7f895c0cdfb0 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.241+0000 7f8961200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f895c0ce440 con 0x7f895c15ec10 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.241+0000 7f8961200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f895c0ce720 con 0x7f895c0b2c90 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f8959e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f895c15ec10 0x7f895c0b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f8959e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f895c15ec10 0x7f895c0b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34338/0 (socket says 172.21.15.67:34338) 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f8959e00640 1 -- 172.21.15.67:0/2433014327 learned_addr learned my addr 172.21.15.67:0/2433014327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f895a800640 1 --2- 172.21.15.67:0/2433014327 >> v2:172.21.15.187:3300/0 conn(0x7f895c0b2c90 0x7f895c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f8959e00640 1 -- 172.21.15.67:0/2433014327 >> v2:172.21.15.187:3300/0 conn(0x7f895c0b2c90 msgr2=0x7f895c0b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:09.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.242+0000 7f8959e00640 1 --2- 172.21.15.67:0/2433014327 >> v2:172.21.15.187:3300/0 conn(0x7f895c0b2c90 0x7f895c0b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.243+0000 7f8959e00640 1 -- 172.21.15.67:0/2433014327 >> v1:172.21.15.187:6789/0 conn(0x7f895c0cdfb0 legacy=0x7f895c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.243+0000 7f8959e00640 1 -- 172.21.15.67:0/2433014327 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c0cea00 con 0x7f895c15ec10 2024-04-17T07:33:09.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.244+0000 7f8959e00640 1 --2- 172.21.15.67:0/2433014327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c15ec10 0x7f895c0b2750 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f895c071770 tx=0x7f894c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade3a8de433b29ba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.244+0000 7f8958a00640 1 -- 172.21.15.67:0/2433014327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c035020 con 0x7f895c15ec10 2024-04-17T07:33:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8958a00640 1 -- 172.21.15.67:0/2433014327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894c031870 con 0x7f895c15ec10 2024-04-17T07:33:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8958a00640 1 -- 172.21.15.67:0/2433014327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c031c40 con 0x7f895c15ec10 2024-04-17T07:33:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 -- 172.21.15.67:0/2433014327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c15ec10 msgr2=0x7f895c0b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 --2- 172.21.15.67:0/2433014327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c15ec10 0x7f895c0b2750 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f895c071770 tx=0x7f894c02b910 comp rx=0 tx=0).stop 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 -- 172.21.15.67:0/2433014327 shutdown_connections 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 --2- 172.21.15.67:0/2433014327 >> v2:172.21.15.187:3300/0 conn(0x7f895c0b2c90 0x7f895c0b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 --2- 172.21.15.67:0/2433014327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c15ec10 0x7f895c0b2750 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 -- 172.21.15.67:0/2433014327 >> 172.21.15.67:0/2433014327 conn(0x7f895c073360 msgr2=0x7f895c09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:09.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 -- 172.21.15.67:0/2433014327 shutdown_connections 2024-04-17T07:33:09.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.245+0000 7f8961200640 1 -- 172.21.15.67:0/2433014327 wait complete. 2024-04-17T07:33:09.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.247+0000 7f8961200640 1 Processor -- start 2024-04-17T07:33:09.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.247+0000 7f8961200640 1 -- start start 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 0x7f895c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f895c15ec10 0x7f895c151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f895c06c150 con 0x7f895c0b5e80 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f895c06bfd0 con 0x7f895c15ec10 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.248+0000 7f8961200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f895c0b1820 con 0x7f895c0b2c90 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f8959400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f895c15ec10 0x7f895c151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34340/0 (socket says 172.21.15.67:34340) 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 -- 172.21.15.67:0/314194872 learned_addr learned my addr 172.21.15.67:0/314194872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f8959e00640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 0x7f895c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 -- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 msgr2=0x7f895c0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 0x7f895c0c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 -- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f895c15ec10 msgr2=0x7f895c151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f895c15ec10 0x7f895c151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f895a800640 1 -- 172.21.15.67:0/314194872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c0c5d30 con 0x7f895c0b5e80 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.249+0000 7f8959e00640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 0x7f895c0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f895a800640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f895400efc0 tx=0x7f895400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954018020 con 0x7f895c0b5e80 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8954014480 con 0x7f895c0b5e80 2024-04-17T07:33:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954014870 con 0x7f895c0b5e80 2024-04-17T07:33:09.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c0c6010 con 0x7f895c0b5e80 2024-04-17T07:33:09.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.250+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c0c6af0 con 0x7f895c0b5e80 2024-04-17T07:33:09.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.251+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c068b90 con 0x7f895c0b5e80 2024-04-17T07:33:09.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.252+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 66) ==== 90878+0+0 (secure 0 0 0) 0x7f8954005d10 con 0x7f895c0b5e80 2024-04-17T07:33:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.253+0000 7f8952000640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 0x7f893c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.253+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7f895408c940 con 0x7f895c0b5e80 2024-04-17T07:33:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.254+0000 7f8959e00640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 0x7f893c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.255+0000 7f8959e00640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 0x7f893c06ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f894c036000 tx=0x7f894c008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.258+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8954022030 con 0x7f895c0b5e80 2024-04-17T07:33:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.509+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f895c0c7e10 con 0x7f895c0b5e80 2024-04-17T07:33:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.510+0000 7f8952000640 1 -- 172.21.15.67:0/314194872 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7f8954010040 con 0x7f895c0b5e80 2024-04-17T07:33:09.509 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:09.509 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:33:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.512+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 msgr2=0x7f893c06ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.512+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 0x7f893c06ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f894c036000 tx=0x7f894c008030 comp rx=0 tx=0).stop 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.512+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 msgr2=0x7f895c0b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.512+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f895400efc0 tx=0x7f895400c490 comp rx=0 tx=0).stop 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 shutdown_connections 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c0b5e80 0x7f895c0b6310 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f895c15ec10 0x7f895c151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f893c06c770 0x7f893c06ec30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 --2- 172.21.15.67:0/314194872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f895c0b2c90 0x7f895c0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 >> 172.21.15.67:0/314194872 conn(0x7f895c073360 msgr2=0x7f895c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 shutdown_connections 2024-04-17T07:33:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.513+0000 7f8961200640 1 -- 172.21.15.67:0/314194872 wait complete. 2024-04-17T07:33:09.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.753+0000 7fbe4f800640 1 Processor -- start 2024-04-17T07:33:09.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.753+0000 7fbe4f800640 1 -- start start 2024-04-17T07:33:09.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe4815b530 0x7fbe48071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4f800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe48072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4f800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe4815f3c0 con 0x7fbe4815b910 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe4806ed60 con 0x7fbe4815b530 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4f800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4815f660 con 0x7fbe480c68c0 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe4815b530 0x7fbe48071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe4815b530 0x7fbe48071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34354/0 (socket says 172.21.15.67:34354) 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.755+0000 7fbe4ce00640 1 -- 172.21.15.67:0/1379650212 learned_addr learned my addr 172.21.15.67:0/1379650212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe3fe00640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe48072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe4ce00640 1 -- 172.21.15.67:0/1379650212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 msgr2=0x7fbe48072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe4ce00640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe48072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe4ce00640 1 -- 172.21.15.67:0/1379650212 >> v1:172.21.15.187:6789/0 conn(0x7fbe4815b910 legacy=0x7fbe480c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe4ce00640 1 -- 172.21.15.67:0/1379650212 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4815f940 con 0x7fbe4815b530 2024-04-17T07:33:09.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.756+0000 7fbe3fe00640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe48072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.757+0000 7fbe4ce00640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe48071d10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61312b534163aaee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.757+0000 7fbe47e00640 1 -- 172.21.15.67:0/1379650212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe38035020 con 0x7fbe4815b530 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.757+0000 7fbe47e00640 1 -- 172.21.15.67:0/1379650212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe38031640 con 0x7fbe4815b530 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.757+0000 7fbe47e00640 1 -- 172.21.15.67:0/1379650212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3802d040 con 0x7fbe4815b530 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 -- 172.21.15.67:0/1379650212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 msgr2=0x7fbe48071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe48071d10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3802b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 -- 172.21.15.67:0/1379650212 shutdown_connections 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe48072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 --2- 172.21.15.67:0/1379650212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe48071d10 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 -- 172.21.15.67:0/1379650212 >> 172.21.15.67:0/1379650212 conn(0x7fbe4806bf50 msgr2=0x7fbe480b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 -- 172.21.15.67:0/1379650212 shutdown_connections 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.758+0000 7fbe4f800640 1 -- 172.21.15.67:0/1379650212 wait complete. 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.760+0000 7fbe4f800640 1 Processor -- start 2024-04-17T07:33:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.760+0000 7fbe4f800640 1 -- start start 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe4813cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe4815b910 0x7fbe480cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe48160140 con 0x7fbe4815b530 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbe480cc950 con 0x7fbe4815b910 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe4f800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4813d280 con 0x7fbe480c68c0 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.761+0000 7fbe3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34358/0 (socket says 172.21.15.67:34358) 2024-04-17T07:33:09.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 -- 172.21.15.67:0/3576566689 learned_addr learned my addr 172.21.15.67:0/3576566689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:09.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe4ce00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe4813cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe4d800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe4815b910 0x7fbe480cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 -- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 msgr2=0x7fbe4813cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe4813cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 -- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe4815b910 msgr2=0x7fbe480cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe4815b910 0x7fbe480cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 -- 172.21.15.67:0/3576566689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe480cf890 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.762+0000 7fbe3fe00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000c7f0 tx=0x7fbe3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.763+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30015070 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.763+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe30007a30 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.763+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe30007e00 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.763+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe480cfb70 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.763+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe480b33b0 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.764+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe48068b90 con 0x7fbe4815b530 2024-04-17T07:33:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.765+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 66) ==== 90878+0+0 (secure 0 0 0) 0x7fbe30018070 con 0x7fbe4815b530 2024-04-17T07:33:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.766+0000 7fbe45600640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 0x7fbe2406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.766+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7fbe3008bb70 con 0x7fbe4815b530 2024-04-17T07:33:09.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.767+0000 7fbe4ce00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 0x7fbe2406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.768+0000 7fbe4ce00640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 0x7fbe2406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe38036000 tx=0x7fbe38008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:09.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.770+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbe30059ed0 con 0x7fbe4815b530 2024-04-17T07:33:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.973+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 67) ==== 91086+0+0 (secure 0 0 0) 0x7fbe30055e40 con 0x7fbe4815b530 2024-04-17T07:33:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.990+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe480c6390 con 0x7fbe4815b530 2024-04-17T07:33:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.991+0000 7fbe45600640 1 -- 172.21.15.67:0/3576566689 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17825 (secure 0 0 0) 0x7fbe30018360 con 0x7fbe4815b530 2024-04-17T07:33:09.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:09.991 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":130,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:08.912658+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1738540495":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3255864985":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6833/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3710283205":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/4043320648":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/1830624579":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/2041663839":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/757480182":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2989376162":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6833/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1667247431":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/3546270980":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3883423100":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/410647397":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2591672672":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1399211335":"2024-04-18T07:32:49.473454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 msgr2=0x7fbe2406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 0x7fbe2406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe38036000 tx=0x7fbe38008030 comp rx=0 tx=0).stop 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 msgr2=0x7fbe480ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000c7f0 tx=0x7fbe3000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 shutdown_connections 2024-04-17T07:33:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe4815b910 0x7fbe480cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe4815b530 0x7fbe480ce8a0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fbe2406c740 0x7fbe2406ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 --2- 172.21.15.67:0/3576566689 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe480c68c0 0x7fbe4813cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 >> 172.21.15.67:0/3576566689 conn(0x7fbe4806bf50 msgr2=0x7fbe480c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 shutdown_connections 2024-04-17T07:33:09.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:09.995+0000 7fbe4f800640 1 -- 172.21.15.67:0/3576566689 wait complete. 2024-04-17T07:33:10.013 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:33:10.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.230+0000 7ff6c3e00640 1 Processor -- start 2024-04-17T07:33:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.231+0000 7ff6c3e00640 1 -- start start 2024-04-17T07:33:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.232+0000 7ff6c3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6c415ec60 0x7ff6c40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.232+0000 7ff6c3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6c40b6b30 0x7ff6c40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.232+0000 7ff6c3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c40d2d30 con 0x7ff6c40cf910 2024-04-17T07:33:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.232+0000 7ff6c3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c40d49c0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.232+0000 7ff6c3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c40d3660 con 0x7ff6c415ec60 2024-04-17T07:33:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6c40cf910 0x7ff6c406cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49972/0 (socket says 172.21.15.67:49972) 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6c40b6b30 0x7ff6c40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c3400640 1 -- 172.21.15.67:0/3605316780 learned_addr learned my addr 172.21.15.67:0/3605316780 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2a00640 1 --2- 172.21.15.67:0/3605316780 >> v2:172.21.15.187:3300/0 conn(0x7ff6c415ec60 0x7ff6c40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c1600640 1 -- 172.21.15.67:0/3605316780 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435063491 0 0) 0x7ff6c40d2d30 con 0x7ff6c40cf910 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c1600640 1 -- 172.21.15.67:0/3605316780 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a8003520 con 0x7ff6c40cf910 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2000640 1 -- 172.21.15.67:0/3605316780 >> v2:172.21.15.187:3300/0 conn(0x7ff6c415ec60 msgr2=0x7ff6c40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2000640 1 --2- 172.21.15.67:0/3605316780 >> v2:172.21.15.187:3300/0 conn(0x7ff6c415ec60 0x7ff6c40b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2000640 1 -- 172.21.15.67:0/3605316780 >> v1:172.21.15.67:6789/0 conn(0x7ff6c40cf910 legacy=0x7ff6c406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.233+0000 7ff6c2000640 1 -- 172.21.15.67:0/3605316780 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c40d3010 con 0x7ff6c40b6b30 2024-04-17T07:33:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.235+0000 7ff6c2000640 1 --2- 172.21.15.67:0/3605316780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c40cf3d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40d1550 tx=0x7ff6b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112dcbefe69916a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.235+0000 7ff6c1600640 1 -- 172.21.15.67:0/3605316780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0035020 con 0x7ff6c40b6b30 2024-04-17T07:33:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.235+0000 7ff6c1600640 1 -- 172.21.15.67:0/3605316780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b0031830 con 0x7ff6c40b6b30 2024-04-17T07:33:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.235+0000 7ff6c1600640 1 -- 172.21.15.67:0/3605316780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0031c00 con 0x7ff6c40b6b30 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 -- 172.21.15.67:0/3605316780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 msgr2=0x7ff6c40cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/3605316780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c40cf3d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40d1550 tx=0x7ff6b002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 -- 172.21.15.67:0/3605316780 shutdown_connections 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/3605316780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c40cf3d0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/3605316780 >> v2:172.21.15.187:3300/0 conn(0x7ff6c415ec60 0x7ff6c40b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 -- 172.21.15.67:0/3605316780 >> 172.21.15.67:0/3605316780 conn(0x7ff6c40d15f0 msgr2=0x7ff6c40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 -- 172.21.15.67:0/3605316780 shutdown_connections 2024-04-17T07:33:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.236+0000 7ff6c3e00640 1 -- 172.21.15.67:0/3605316780 wait complete. 2024-04-17T07:33:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.237+0000 7ff6c3e00640 1 Processor -- start 2024-04-17T07:33:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.238+0000 7ff6c3e00640 1 -- start start 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.238+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6c40cf910 0x7ff6c414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c415ec60 0x7ff6c40b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c40cfda0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c40b6f70 con 0x7ff6c415ec60 2024-04-17T07:33:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c415f040 con 0x7ff6c40cf910 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34366/0 (socket says 172.21.15.67:34366) 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 -- 172.21.15.67:0/2313644285 learned_addr learned my addr 172.21.15.67:0/2313644285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 -- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6c40cf910 msgr2=0x7ff6c414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6c40cf910 0x7ff6c414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 -- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c415ec60 msgr2=0x7ff6c40b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c3400640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c415ec60 0x7ff6c40b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c415ec60 0x7ff6c40b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.239+0000 7ff6c2a00640 1 -- 172.21.15.67:0/2313644285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c414e490 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6c2a00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d620 tx=0x7ff6b400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400b620 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b4019070 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400ba00 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c414fd40 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.240+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c414ea40 con 0x7ff6c40b6b30 2024-04-17T07:33:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.241+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c4068a10 con 0x7ff6c40b6b30 2024-04-17T07:33:10.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.242+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 67) ==== 91086+0+0 (secure 0 0 0) 0x7ff6b400bba0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.243+0000 7ff6baa00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 0x7ff6a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.243+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7ff6b408c2a0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.244+0000 7ff6c2000640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 0x7ff6a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.245+0000 7ff6c2000640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 0x7ff6a406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0036000 tx=0x7ff6b0030000 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.248+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff6b405a3a0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.491+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6c40cef20 con 0x7ff6c40b6b30 2024-04-17T07:33:10.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.492+0000 7ff6baa00640 1 -- 172.21.15.67:0/2313644285 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7ff6b4059ca0 con 0x7ff6c40b6b30 2024-04-17T07:33:10.491 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:10.492 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:33:10.492 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:33:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 msgr2=0x7ff6a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 0x7ff6a406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0036000 tx=0x7ff6b0030000 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 msgr2=0x7ff6c414df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d620 tx=0x7ff6b400dae0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 shutdown_connections 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6c415ec60 0x7ff6c40b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff6a406c980 0x7ff6a406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6c40cf910 0x7ff6c414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 --2- 172.21.15.67:0/2313644285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6c40b6b30 0x7ff6c414df50 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 >> 172.21.15.67:0/2313644285 conn(0x7ff6c40d15f0 msgr2=0x7ff6c4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 shutdown_connections 2024-04-17T07:33:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.495+0000 7ff6c3e00640 1 -- 172.21.15.67:0/2313644285 wait complete. 2024-04-17T07:33:10.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:10.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.740+0000 7f84da400640 1 Processor -- start 2024-04-17T07:33:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.740+0000 7f84da400640 1 -- start start 2024-04-17T07:33:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.742+0000 7f84da400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d40d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.742+0000 7f84da400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f84d40d2ab0 0x7f84d40b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.742+0000 7f84da400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d40d3020 con 0x7f84d40cf640 2024-04-17T07:33:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.742+0000 7f84da400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f84d40d3c30 con 0x7f84d40d2ab0 2024-04-17T07:33:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.742+0000 7f84da400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f84d40d3300 con 0x7f84d415ed10 2024-04-17T07:33:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.743+0000 7f84d3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f84d40d2ab0 0x7f84d40b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.743+0000 7f84d2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d40d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.743+0000 7f84d3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f84d40d2ab0 0x7f84d40b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:53940/0 (socket says 172.21.15.67:53940) 2024-04-17T07:33:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.743+0000 7f84d3e00640 1 -- 172.21.15.67:0/4040354604 learned_addr learned my addr 172.21.15.67:0/4040354604 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.744+0000 7f84d3e00640 1 -- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 msgr2=0x7f84d40d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.744+0000 7f84d3e00640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d40d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.744+0000 7f84d3e00640 1 -- 172.21.15.67:0/4040354604 >> v1:172.21.15.187:6789/0 conn(0x7f84d40cf640 legacy=0x7f84d40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.744+0000 7f84d3e00640 1 -- 172.21.15.67:0/4040354604 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40d35e0 con 0x7f84d40d2ab0 2024-04-17T07:33:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.744+0000 7f84d2a00640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d40d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:33:10.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.745+0000 7f84d3e00640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40d2ab0 0x7f84d40b7b10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009e50 tx=0x7f84cc02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94b7ca5c09aff6af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84d2000640 1 -- 172.21.15.67:0/4040354604 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc02cb90 con 0x7f84d40d2ab0 2024-04-17T07:33:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84d2000640 1 -- 172.21.15.67:0/4040354604 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc02cd30 con 0x7f84d40d2ab0 2024-04-17T07:33:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84d2000640 1 -- 172.21.15.67:0/4040354604 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc034610 con 0x7f84d40d2ab0 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 -- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40d2ab0 msgr2=0x7f84d40b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40d2ab0 0x7f84d40b7b10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009e50 tx=0x7f84cc02bff0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 -- 172.21.15.67:0/4040354604 shutdown_connections 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40d2ab0 0x7f84d40b7b10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 --2- 172.21.15.67:0/4040354604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d40d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.746+0000 7f84da400640 1 -- 172.21.15.67:0/4040354604 >> 172.21.15.67:0/4040354604 conn(0x7f84d406cbd0 msgr2=0x7f84d406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.747+0000 7f84da400640 1 -- 172.21.15.67:0/4040354604 shutdown_connections 2024-04-17T07:33:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.747+0000 7f84da400640 1 -- 172.21.15.67:0/4040354604 wait complete. 2024-04-17T07:33:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.748+0000 7f84da400640 1 Processor -- start 2024-04-17T07:33:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 -- start start 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40cf640 0x7f84d41f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d4160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84d40d4600 con 0x7f84d40d2ab0 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f84d40d25b0 con 0x7f84d40cf640 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.749+0000 7f84da400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f84d40d4780 con 0x7f84d415ed10 2024-04-17T07:33:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34370/0 (socket says 172.21.15.67:34370) 2024-04-17T07:33:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 -- 172.21.15.67:0/628704417 learned_addr learned my addr 172.21.15.67:0/628704417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d3e00640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d4160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d3400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40cf640 0x7f84d41f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 -- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 msgr2=0x7f84d4160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d4160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 -- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40cf640 msgr2=0x7f84d41f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40cf640 0x7f84d41f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.750+0000 7f84d2a00640 1 -- 172.21.15.67:0/628704417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40ccf90 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84d2a00640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f84c000e930 tx=0x7f84c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0017070 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c000ca00 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c000cdf0 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d40b5580 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.751+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d40ccbd0 con 0x7f84d40d2ab0 2024-04-17T07:33:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.752+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d4068b90 con 0x7f84d40d2ab0 2024-04-17T07:33:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.754+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 67) ==== 91086+0+0 (secure 0 0 0) 0x7f84c001a070 con 0x7f84d40d2ab0 2024-04-17T07:33:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.755+0000 7f84cb400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 0x7f84b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.755+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7f84c008bdb0 con 0x7f84d40d2ab0 2024-04-17T07:33:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.756+0000 7f84d3400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 0x7f84b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.757+0000 7f84d3400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 0x7f84b406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001640 tx=0x7f84c4009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:10.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:10.760+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f84c0059eb0 con 0x7f84d40d2ab0 2024-04-17T07:33:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.005+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84d406bb70 con 0x7f84d40d2ab0 2024-04-17T07:33:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.006+0000 7f84cb400640 1 -- 172.21.15.67:0/628704417 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5201 (secure 0 0 0) 0x7f84c00597b0 con 0x7f84d40d2ab0 2024-04-17T07:33:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:11.006 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:32:34.587190+0000","modified":"2024-04-17T07:32:37.647893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7899},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7899":{"gid":7899,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1617335847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1617335847},{"type":"v1","addr":"172.21.15.67:6835","nonce":1617335847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7899,"qdb_cluster":[7899]},"id":7}]} 2024-04-17T07:33:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 61 2024-04-17T07:33:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 msgr2=0x7f84b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 0x7f84b406ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001640 tx=0x7f84c4009040 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 msgr2=0x7f84d40ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f84c000e930 tx=0x7f84c000edf0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 shutdown_connections 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f84b406c980 0x7f84b406ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f84d415ed10 0x7f84d4160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d40d2ab0 0x7f84d40ce590 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 --2- 172.21.15.67:0/628704417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40cf640 0x7f84d41f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 >> 172.21.15.67:0/628704417 conn(0x7f84d406cbd0 msgr2=0x7f84d40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 shutdown_connections 2024-04-17T07:33:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.009+0000 7f84da400640 1 -- 172.21.15.67:0/628704417 wait complete. 2024-04-17T07:33:11.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:11.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.256+0000 7fd0cb200640 1 Processor -- start 2024-04-17T07:33:11.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.256+0000 7fd0cb200640 1 -- start start 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.257+0000 7fd0cb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.257+0000 7fd0cb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc160280 0x7fd0cc06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.257+0000 7fd0cb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0cc0cd4a0 con 0x7fd0cc141ed0 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.257+0000 7fd0cb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd0cc15ce30 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.257+0000 7fd0cb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0cc1607f0 con 0x7fd0cc160280 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34372/0 (socket says 172.21.15.67:34372) 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 -- 172.21.15.67:0/2519007991 learned_addr learned my addr 172.21.15.67:0/2519007991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9400640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc160280 0x7fd0cc06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 -- 172.21.15.67:0/2519007991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc160280 msgr2=0x7fd0cc06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc160280 0x7fd0cc06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 -- 172.21.15.67:0/2519007991 >> v1:172.21.15.187:6789/0 conn(0x7fd0cc141ed0 legacy=0x7fd0cc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.258+0000 7fd0c9e00640 1 -- 172.21.15.67:0/2519007991 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc160ad0 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.259+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8485629e1007af26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.260+0000 7fd0c8a00640 1 -- 172.21.15.67:0/2519007991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc035020 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.260+0000 7fd0c8a00640 1 -- 172.21.15.67:0/2519007991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0bc031640 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.260+0000 7fd0c8a00640 1 -- 172.21.15.67:0/2519007991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc02d040 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.260+0000 7fd0cb200640 1 -- 172.21.15.67:0/2519007991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc06b8c0 msgr2=0x7fd0cc0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.260+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009870 tx=0x7fd0bc02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 -- 172.21.15.67:0/2519007991 shutdown_connections 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc160280 0x7fd0cc06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2519007991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc06b8c0 0x7fd0cc0cd8b0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 -- 172.21.15.67:0/2519007991 >> 172.21.15.67:0/2519007991 conn(0x7fd0cc067d10 msgr2=0x7fd0cc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 -- 172.21.15.67:0/2519007991 shutdown_connections 2024-04-17T07:33:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.261+0000 7fd0cb200640 1 -- 172.21.15.67:0/2519007991 wait complete. 2024-04-17T07:33:11.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.262+0000 7fd0cb200640 1 Processor -- start 2024-04-17T07:33:11.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.262+0000 7fd0cb200640 1 -- start start 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 0x7fd0cc1f9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc141ed0 0x7fd0cc1456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc06c1e0 con 0x7fd0cc160280 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc0cddf0 con 0x7fd0cc141ed0 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.263+0000 7fd0cb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0cc161090 con 0x7fd0cc06b8c0 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34374/0 (socket says 172.21.15.67:34374) 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 -- 172.21.15.67:0/2400428078 learned_addr learned my addr 172.21.15.67:0/2400428078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 0x7fd0cc1f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0c9400640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc141ed0 0x7fd0cc1456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 -- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 msgr2=0x7fd0cc1f9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 0x7fd0cc1f9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 -- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc141ed0 msgr2=0x7fd0cc1456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc141ed0 0x7fd0cc1456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0ca800640 1 -- 172.21.15.67:0/2400428078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc143a90 con 0x7fd0cc160280 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.264+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 0x7fd0cc1f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0ca800640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c410 tx=0x7fd0c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4011790 con 0x7fd0cc160280 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c4017070 con 0x7fd0cc160280 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc15b040 con 0x7fd0cc160280 2024-04-17T07:33:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4011bb0 con 0x7fd0cc160280 2024-04-17T07:33:11.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.265+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc15d360 con 0x7fd0cc160280 2024-04-17T07:33:11.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.266+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc068a10 con 0x7fd0cc160280 2024-04-17T07:33:11.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.267+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 67) ==== 91086+0+0 (secure 0 0 0) 0x7fd0c401c460 con 0x7fd0cc160280 2024-04-17T07:33:11.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.268+0000 7fd0c2000640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 0x7fd0ac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.269+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7fd0c408bb10 con 0x7fd0cc160280 2024-04-17T07:33:11.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.269+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 0x7fd0ac06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.271+0000 7fd0c9e00640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 0x7fd0ac06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc03c040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.273+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd0c4021020 con 0x7fd0cc160280 2024-04-17T07:33:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.480+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0cc0cf590 con 0x7fd0cc160280 2024-04-17T07:33:11.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.484+0000 7fd0c2000640 1 -- 172.21.15.67:0/2400428078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) ==== 74+0+17825 (secure 0 0 0) 0x7fd0c4059ba0 con 0x7fd0cc160280 2024-04-17T07:33:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:11.483 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":130,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:08.912658+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:32:32.505992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-17T07:32:33.573434+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":130,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1738540495":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3255864985":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6833/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3710283205":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/4043320648":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/1830624579":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/2041663839":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/757480182":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2989376162":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6833/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1667247431":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/3546270980":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3883423100":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/410647397":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2591672672":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1399211335":"2024-04-18T07:32:49.473454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.486+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 msgr2=0x7fd0ac06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.486+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 0x7fd0ac06ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc036000 tx=0x7fd0bc03c040 comp rx=0 tx=0).stop 2024-04-17T07:33:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.486+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 msgr2=0x7fd0cc1ef130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.486+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400c410 tx=0x7fd0c400c8d0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 shutdown_connections 2024-04-17T07:33:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0cc160280 0x7fd0cc1ef130 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0cc141ed0 0x7fd0cc1456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd0ac06c980 0x7fd0ac06ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 --2- 172.21.15.67:0/2400428078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0cc06b8c0 0x7fd0cc1f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 >> 172.21.15.67:0/2400428078 conn(0x7fd0cc067d10 msgr2=0x7fd0cc0b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 shutdown_connections 2024-04-17T07:33:11.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.487+0000 7fd0cb200640 1 -- 172.21.15.67:0/2400428078 wait complete. 2024-04-17T07:33:11.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:33:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.733+0000 7f0a87e00640 1 Processor -- start 2024-04-17T07:33:11.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.733+0000 7f0a87e00640 1 -- start start 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.734+0000 7f0a87e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a8806f550 0x7f0a880ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.734+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 0x7f0a880722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.734+0000 7f0a87e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a8806fb10 con 0x7f0a8814c1d0 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.734+0000 7f0a87e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0a88167b10 con 0x7f0a8806f550 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.734+0000 7f0a87e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0a88166f70 con 0x7f0a88071e60 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a8806f550 0x7f0a880ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a8806f550 0x7f0a880ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34384/0 (socket says 172.21.15.67:34384) 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 -- 172.21.15.67:0/497678882 learned_addr learned my addr 172.21.15.67:0/497678882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a87400640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 0x7f0a880722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 -- 172.21.15.67:0/497678882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 msgr2=0x7f0a880722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 0x7f0a880722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 -- 172.21.15.67:0/497678882 >> v1:172.21.15.187:6789/0 conn(0x7f0a8814c1d0 legacy=0x7f0a88167400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a7e000640 1 -- 172.21.15.67:0/497678882 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8806fdf0 con 0x7f0a8806f550 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.735+0000 7f0a87400640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 0x7f0a880722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.736+0000 7f0a7e000640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8806f550 0x7f0a880ca890 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009870 tx=0x7f0a7002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5d0008bde9c01f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a86000640 1 -- 172.21.15.67:0/497678882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a700317c0 con 0x7f0a8806f550 2024-04-17T07:33:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a86000640 1 -- 172.21.15.67:0/497678882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a70037070 con 0x7f0a8806f550 2024-04-17T07:33:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a86000640 1 -- 172.21.15.67:0/497678882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70031b80 con 0x7f0a8806f550 2024-04-17T07:33:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 -- 172.21.15.67:0/497678882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8806f550 msgr2=0x7f0a880ca890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8806f550 0x7f0a880ca890 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009870 tx=0x7f0a7002b860 comp rx=0 tx=0).stop 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 -- 172.21.15.67:0/497678882 shutdown_connections 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a88071e60 0x7f0a880722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 --2- 172.21.15.67:0/497678882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8806f550 0x7f0a880ca890 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 -- 172.21.15.67:0/497678882 >> 172.21.15.67:0/497678882 conn(0x7f0a8806bef0 msgr2=0x7f0a880b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 -- 172.21.15.67:0/497678882 shutdown_connections 2024-04-17T07:33:11.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.737+0000 7f0a87e00640 1 -- 172.21.15.67:0/497678882 wait complete. 2024-04-17T07:33:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.739+0000 7f0a87e00640 1 Processor -- start 2024-04-17T07:33:11.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.739+0000 7f0a87e00640 1 -- start start 2024-04-17T07:33:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a8806f550 0x7f0a880af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a8814c1d0 0x7f0a88165400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8814ca80 con 0x7f0a88071e60 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8814c5b0 con 0x7f0a8814c1d0 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a87e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0a880700d0 con 0x7f0a8806f550 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.740+0000 7f0a7e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34388/0 (socket says 172.21.15.67:34388) 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 -- 172.21.15.67:0/1013748171 learned_addr learned my addr 172.21.15.67:0/1013748171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a86a00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a8806f550 0x7f0a880af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a87400640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a8814c1d0 0x7f0a88165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 -- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a8806f550 msgr2=0x7f0a880af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a8806f550 0x7f0a880af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 -- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a8814c1d0 msgr2=0x7f0a88165400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a8814c1d0 0x7f0a88165400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:11.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 -- 172.21.15.67:0/1013748171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a88164080 con 0x7f0a88071e60 2024-04-17T07:33:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.741+0000 7f0a7e000640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000c00 tx=0x7f0a700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.742+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70032860 con 0x7f0a88071e60 2024-04-17T07:33:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.742+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a70032a00 con 0x7f0a88071e60 2024-04-17T07:33:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.742+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70032df0 con 0x7f0a88071e60 2024-04-17T07:33:11.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.742+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a88164360 con 0x7f0a88071e60 2024-04-17T07:33:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.742+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a88164920 con 0x7f0a88071e60 2024-04-17T07:33:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.743+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a88068a10 con 0x7f0a88071e60 2024-04-17T07:33:11.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.744+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 67) ==== 91086+0+0 (secure 0 0 0) 0x7f0a7004c060 con 0x7f0a88071e60 2024-04-17T07:33:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.745+0000 7f0a7f400640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 0x7f0a6406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.745+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6539+0+0 (secure 0 0 0) 0x7f0a700ada20 con 0x7f0a88071e60 2024-04-17T07:33:11.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.746+0000 7f0a86a00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 0x7f0a6406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:11.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.747+0000 7f0a86a00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 0x7f0a6406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800a750 tx=0x7f0a78008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:11.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.750+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0a70040070 con 0x7f0a88071e60 2024-04-17T07:33:11.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:11.991+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f0a70077990 con 0x7f0a88071e60 2024-04-17T07:33:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.004+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0a880cbe90 con 0x7f0a88071e60 2024-04-17T07:33:12.006 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:33:12.005+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:33:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.009+0000 7f0a7f400640 1 -- 172.21.15.67:0/1013748171 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) ==== 161+0+0 (secure 0 0 0) 0x7f0a7003b3f0 con 0x7f0a88071e60 2024-04-17T07:33:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:33:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.012+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 msgr2=0x7f0a6406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.012+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 0x7f0a6406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800a750 tx=0x7f0a78008040 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.012+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 msgr2=0x7f0a881f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.012+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000c00 tx=0x7f0a700046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 shutdown_connections 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a8814c1d0 0x7f0a88165400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a88071e60 0x7f0a881f9040 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f0a6406c980 0x7f0a6406ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 --2- 172.21.15.67:0/1013748171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0a8806f550 0x7f0a880af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 >> 172.21.15.67:0/1013748171 conn(0x7f0a8806bef0 msgr2=0x7f0a880722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 shutdown_connections 2024-04-17T07:33:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.013+0000 7f0a87e00640 1 -- 172.21.15.67:0/1013748171 wait complete. 2024-04-17T07:33:12.035 INFO:tasks.ceph.mds.a.smithi067.stderr: -6657> 2024-04-17T07:32:39.530+0000 7f0f59600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:33:12.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:33:12.096 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:33:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.234+0000 7fb194e00640 1 Processor -- start 2024-04-17T07:33:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.234+0000 7fb194e00640 1 -- start start 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb194e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb190145f80 0x7fb1900cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb194e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 0x7fb19006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb194e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb190146360 con 0x7fb1900cd750 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb194e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb1900727a0 con 0x7fb190145f80 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb194e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb190071e70 con 0x7fb19006c270 2024-04-17T07:33:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb18f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb190145f80 0x7fb1900cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb18f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb190145f80 0x7fb1900cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34414/0 (socket says 172.21.15.67:34414) 2024-04-17T07:33:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb18f400640 1 -- 172.21.15.67:0/3135041100 learned_addr learned my addr 172.21.15.67:0/3135041100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.236+0000 7fb18ea00640 1 --1- 172.21.15.67:0/3135041100 >> v1:172.21.15.67:6789/0 conn(0x7fb1900cd750 0x7fb19006bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49984/0 (socket says 172.21.15.67:49984) 2024-04-17T07:33:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18fe00640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 0x7fb19006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18f400640 1 -- 172.21.15.67:0/3135041100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 msgr2=0x7fb19006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18f400640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 0x7fb19006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18f400640 1 -- 172.21.15.67:0/3135041100 >> v1:172.21.15.67:6789/0 conn(0x7fb1900cd750 legacy=0x7fb19006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18e000640 1 -- 172.21.15.67:0/3135041100 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301131977 0 0) 0x7fb190146360 con 0x7fb1900cd750 2024-04-17T07:33:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18f400640 1 -- 172.21.15.67:0/3135041100 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb190072150 con 0x7fb190145f80 2024-04-17T07:33:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.237+0000 7fb18fe00640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 0x7fb19006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.238+0000 7fb18f400640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb190145f80 0x7fb1900cd210 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb180009870 tx=0x7fb18002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f696a1d25dd0a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.238+0000 7fb18e000640 1 -- 172.21.15.67:0/3135041100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180035020 con 0x7fb190145f80 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.238+0000 7fb18e000640 1 -- 172.21.15.67:0/3135041100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb180031640 con 0x7fb190145f80 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.238+0000 7fb18e000640 1 -- 172.21.15.67:0/3135041100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18002d040 con 0x7fb190145f80 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 -- 172.21.15.67:0/3135041100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb190145f80 msgr2=0x7fb1900cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb190145f80 0x7fb1900cd210 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb180009870 tx=0x7fb18002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 -- 172.21.15.67:0/3135041100 shutdown_connections 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19006c270 0x7fb19006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 --2- 172.21.15.67:0/3135041100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb190145f80 0x7fb1900cd210 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 -- 172.21.15.67:0/3135041100 >> 172.21.15.67:0/3135041100 conn(0x7fb1900b2190 msgr2=0x7fb1900b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 -- 172.21.15.67:0/3135041100 shutdown_connections 2024-04-17T07:33:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.239+0000 7fb194e00640 1 -- 172.21.15.67:0/3135041100 wait complete. 2024-04-17T07:33:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.241+0000 7fb194e00640 1 Processor -- start 2024-04-17T07:33:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.241+0000 7fb194e00640 1 -- start start 2024-04-17T07:33:12.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1900cd750 0x7fb1900ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb190145f80 0x7fb1900acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb190072db0 con 0x7fb19006c270 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb190072c30 con 0x7fb190145f80 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb194e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb19006e700 con 0x7fb1900cd750 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.242+0000 7fb18f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34424/0 (socket says 172.21.15.67:34424) 2024-04-17T07:33:12.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 -- 172.21.15.67:0/795613125 learned_addr learned my addr 172.21.15.67:0/795613125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:12.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18fe00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb190145f80 0x7fb1900acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18ea00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1900cd750 0x7fb1900ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 -- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1900cd750 msgr2=0x7fb1900ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1900cd750 0x7fb1900ac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 -- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb190145f80 msgr2=0x7fb1900acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb190145f80 0x7fb1900acb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 -- 172.21.15.67:0/795613125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1900cb240 con 0x7fb19006c270 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.243+0000 7fb18f400640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fb1800090b0 tx=0x7fb1800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.244+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180040070 con 0x7fb19006c270 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.244+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb180039bf0 con 0x7fb19006c270 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.244+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1900cb520 con 0x7fb19006c270 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.244+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180035040 con 0x7fb19006c270 2024-04-17T07:33:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.244+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1900cbae0 con 0x7fb19006c270 2024-04-17T07:33:12.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.245+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb190068a10 con 0x7fb19006c270 2024-04-17T07:33:12.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.246+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fb180038020 con 0x7fb19006c270 2024-04-17T07:33:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.247+0000 7fb187400640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 0x7fb17006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.247+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6597+0+0 (secure 0 0 0) 0x7fb1800adc80 con 0x7fb19006c270 2024-04-17T07:33:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.248+0000 7fb18ea00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 0x7fb17006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:12.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.249+0000 7fb18ea00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 0x7fb17006eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1900b1f90 tx=0x7fb17c007b70 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:12.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.252+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb18007bb80 con 0x7fb19006c270 2024-04-17T07:33:12.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:12.497+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb19006bd70 con 0x7fb19006c270 2024-04-17T07:33:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.012+0000 7fb187400640 1 -- 172.21.15.67:0/795613125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) ==== 104+0+0 (secure 0 0 0) 0x7fb1800ad0a0 con 0x7fb19006c270 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 msgr2=0x7fb17006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 0x7fb17006eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb1900b1f90 tx=0x7fb17c007b70 comp rx=0 tx=0).stop 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 msgr2=0x7fb1900ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fb1800090b0 tx=0x7fb1800046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 shutdown_connections 2024-04-17T07:33:13.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb190145f80 0x7fb1900acb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fb17006ca10 0x7fb17006eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb1900cd750 0x7fb1900ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 --2- 172.21.15.67:0/795613125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19006c270 0x7fb1900ca860 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.016+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 >> 172.21.15.67:0/795613125 conn(0x7fb1900b2190 msgr2=0x7fb1900d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.017+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 shutdown_connections 2024-04-17T07:33:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.017+0000 7fb194e00640 1 -- 172.21.15.67:0/795613125 wait complete. 2024-04-17T07:33:13.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:33:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.265+0000 7f1aad600640 1 Processor -- start 2024-04-17T07:33:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.265+0000 7f1aad600640 1 -- start start 2024-04-17T07:33:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aad600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1aa806f220 0x7f1aa81434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aad600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8072430 0x7f1aa81439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aad600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa81612d0 con 0x7f1aa8142e10 2024-04-17T07:33:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aad600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa81431f0 con 0x7f1aa806f220 2024-04-17T07:33:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aad600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa8160e40 con 0x7f1aa8072430 2024-04-17T07:33:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aa7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1aa8142e10 0x7f1aa8071d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49988/0 (socket says 172.21.15.67:49988) 2024-04-17T07:33:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aa7e00640 1 -- 172.21.15.67:0/3411070746 learned_addr learned my addr 172.21.15.67:0/3411070746 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.267+0000 7f1aa7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1aa806f220 0x7f1aa81434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aacc00640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8072430 0x7f1aa81439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa6a00640 1 -- 172.21.15.67:0/3411070746 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696859435 0 0) 0x7f1aa81612d0 con 0x7f1aa8142e10 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa6a00640 1 -- 172.21.15.67:0/3411070746 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a8c003520 con 0x7f1aa8142e10 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa7400640 1 -- 172.21.15.67:0/3411070746 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8072430 msgr2=0x7f1aa81439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa7400640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8072430 0x7f1aa81439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa7400640 1 -- 172.21.15.67:0/3411070746 >> v1:172.21.15.67:6789/0 conn(0x7f1aa8142e10 legacy=0x7f1aa8071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.268+0000 7f1aa7400640 1 -- 172.21.15.67:0/3411070746 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa8160510 con 0x7f1aa806f220 2024-04-17T07:33:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.269+0000 7f1aa7400640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa806f220 0x7f1aa81434a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f1a940097a0 tx=0x7f1a9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917ef4e70607608f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aa6a00640 1 -- 172.21.15.67:0/3411070746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a940315a0 con 0x7f1aa806f220 2024-04-17T07:33:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aa6a00640 1 -- 172.21.15.67:0/3411070746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a94037070 con 0x7f1aa806f220 2024-04-17T07:33:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aa6a00640 1 -- 172.21.15.67:0/3411070746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a94031960 con 0x7f1aa806f220 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 -- 172.21.15.67:0/3411070746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa806f220 msgr2=0x7f1aa81434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa806f220 0x7f1aa81434a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f1a940097a0 tx=0x7f1a9402b840 comp rx=0 tx=0).stop 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 -- 172.21.15.67:0/3411070746 shutdown_connections 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8072430 0x7f1aa81439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 --2- 172.21.15.67:0/3411070746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa806f220 0x7f1aa81434a0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.270+0000 7f1aad600640 1 -- 172.21.15.67:0/3411070746 >> 172.21.15.67:0/3411070746 conn(0x7f1aa806b8c0 msgr2=0x7f1aa806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.271+0000 7f1aad600640 1 -- 172.21.15.67:0/3411070746 shutdown_connections 2024-04-17T07:33:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.271+0000 7f1aad600640 1 -- 172.21.15.67:0/3411070746 wait complete. 2024-04-17T07:33:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.272+0000 7f1aad600640 1 Processor -- start 2024-04-17T07:33:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.272+0000 7f1aad600640 1 -- start start 2024-04-17T07:33:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1aa8072430 0x7f1aa80ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8142e10 0x7f1aa8130770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa81460a0 con 0x7f1aa815bed0 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8160ad0 con 0x7f1aa8072430 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.273+0000 7f1aad600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa8146220 con 0x7f1aa8142e10 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34440/0 (socket says 172.21.15.67:34440) 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 -- 172.21.15.67:0/3709353916 learned_addr learned my addr 172.21.15.67:0/3709353916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aa7e00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1aa8072430 0x7f1aa80ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aa7400640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8142e10 0x7f1aa8130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 -- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8142e10 msgr2=0x7f1aa8130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8142e10 0x7f1aa8130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 -- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1aa8072430 msgr2=0x7f1aa80ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:13.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1aa8072430 0x7f1aa80ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.274+0000 7f1aacc00640 1 -- 172.21.15.67:0/3709353916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa8130cb0 con 0x7f1aa815bed0 2024-04-17T07:33:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1aacc00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007e40 tx=0x7f1aa000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:13.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0014620 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa0019070 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa80ae120 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0014a20 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.275+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa80ae6e0 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.276+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa8072c20 con 0x7f1aa815bed0 2024-04-17T07:33:13.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.277+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f1aa0014bc0 con 0x7f1aa815bed0 2024-04-17T07:33:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.279+0000 7f1a9fe00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 0x7f1a8806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:13.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.279+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 6597+0+0 (secure 0 0 0) 0x7f1aa008c980 con 0x7f1aa815bed0 2024-04-17T07:33:13.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.280+0000 7f1aa7e00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 0x7f1a8806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:13.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.281+0000 7f1aa7e00640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 0x7f1a8806eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a98005880 tx=0x7f1a98009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.286+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1aa8072c20 con 0x7f1aa815bed0 2024-04-17T07:33:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:13.493+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1aa8072e30 con 0x7f1aa815bed0 2024-04-17T07:33:14.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.016+0000 7f1a9fe00640 1 -- 172.21.15.67:0/3709353916 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) ==== 181+0+0 (secure 0 0 0) 0x7f1aa005a910 con 0x7f1aa815bed0 2024-04-17T07:33:14.016 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 msgr2=0x7f1a8806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 0x7f1a8806eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a98005880 tx=0x7f1a98009040 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 msgr2=0x7f1aa80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007e40 tx=0x7f1aa000e4b0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 shutdown_connections 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa815bed0 0x7f1aa80b2170 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1a8806ca10 0x7f1a8806eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1aa8142e10 0x7f1aa8130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 --2- 172.21.15.67:0/3709353916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1aa8072430 0x7f1aa80ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.020+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 >> 172.21.15.67:0/3709353916 conn(0x7f1aa806b8c0 msgr2=0x7f1aa81450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.021+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 shutdown_connections 2024-04-17T07:33:14.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.021+0000 7f1aad600640 1 -- 172.21.15.67:0/3709353916 wait complete. 2024-04-17T07:33:14.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:33:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.276+0000 7faa02a00640 1 Processor -- start 2024-04-17T07:33:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.276+0000 7faa02a00640 1 -- start start 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc0cd2f0 0x7fa9fc0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 0x7fa9fc145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9fc146080 0x7fa9fc160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa9fc146510 con 0x7fa9fc146080 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa9fc161000 con 0x7fa9fc0cd6d0 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9fc0ce870 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9fc146080 0x7fa9fc160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9fc146080 0x7fa9fc160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54336/0 (socket says 172.21.15.67:54336) 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.278+0000 7faa02000640 1 -- 172.21.15.67:0/1757503462 learned_addr learned my addr 172.21.15.67:0/1757503462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa00c00640 1 --2- 172.21.15.67:0/1757503462 >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 0x7fa9fc145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa02000640 1 -- 172.21.15.67:0/1757503462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc0cd2f0 msgr2=0x7fa9fc0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:14.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa02000640 1 --2- 172.21.15.67:0/1757503462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc0cd2f0 0x7fa9fc0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa02000640 1 -- 172.21.15.67:0/1757503462 >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 msgr2=0x7fa9fc145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa02000640 1 --2- 172.21.15.67:0/1757503462 >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 0x7fa9fc145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa02000640 1 -- 172.21.15.67:0/1757503462 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9fc0ceb50 con 0x7fa9fc146080 2024-04-17T07:33:14.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.279+0000 7faa00c00640 1 --2- 172.21.15.67:0/1757503462 >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 0x7fa9fc145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.280+0000 7faa02000640 1 --2- 172.21.15.67:0/1757503462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc146080 0x7fa9fc160360 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc073660 tx=0x7fa9f402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699c10836391c516 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.280+0000 7fa9fbe00640 1 -- 172.21.15.67:0/1757503462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4035020 con 0x7fa9fc146080 2024-04-17T07:33:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.280+0000 7fa9fbe00640 1 -- 172.21.15.67:0/1757503462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f402d040 con 0x7fa9fc146080 2024-04-17T07:33:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.280+0000 7fa9fbe00640 1 -- 172.21.15.67:0/1757503462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4031a80 con 0x7fa9fc146080 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 -- 172.21.15.67:0/1757503462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc146080 msgr2=0x7fa9fc160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 --2- 172.21.15.67:0/1757503462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc146080 0x7fa9fc160360 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc073660 tx=0x7fa9f402b900 comp rx=0 tx=0).stop 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 -- 172.21.15.67:0/1757503462 shutdown_connections 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 --2- 172.21.15.67:0/1757503462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc146080 0x7fa9fc160360 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 --2- 172.21.15.67:0/1757503462 >> v2:172.21.15.187:3300/0 conn(0x7fa9fc0cd6d0 0x7fa9fc145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 --2- 172.21.15.67:0/1757503462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc0cd2f0 0x7fa9fc0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 -- 172.21.15.67:0/1757503462 >> 172.21.15.67:0/1757503462 conn(0x7fa9fc06b8c0 msgr2=0x7fa9fc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 -- 172.21.15.67:0/1757503462 shutdown_connections 2024-04-17T07:33:14.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.281+0000 7faa02a00640 1 -- 172.21.15.67:0/1757503462 wait complete. 2024-04-17T07:33:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.283+0000 7faa02a00640 1 Processor -- start 2024-04-17T07:33:14.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.283+0000 7faa02a00640 1 -- start start 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9fc0cd6d0 0x7fa9fc14dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc146080 0x7fa9fc13db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9fc0cfa10 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9fc0cf890 con 0x7fa9fc0cd6d0 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa02a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9fc1608a0 con 0x7fa9fc146080 2024-04-17T07:33:14.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.284+0000 7faa01600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54350/0 (socket says 172.21.15.67:54350) 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 -- 172.21.15.67:0/2902834819 learned_addr learned my addr 172.21.15.67:0/2902834819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa02000640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc146080 0x7fa9fc13db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa00c00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9fc0cd6d0 0x7fa9fc14dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 -- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc146080 msgr2=0x7fa9fc13db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc146080 0x7fa9fc13db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 -- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9fc0cd6d0 msgr2=0x7fa9fc14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9fc0cd6d0 0x7fa9fc14dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:14.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 -- 172.21.15.67:0/2902834819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9fc0b15d0 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.285+0000 7faa01600640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00ba50 tx=0x7fa9ec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.286+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec019070 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.286+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ec00cc00 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.286+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec014440 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.286+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9fc14e200 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.286+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9fc0b1b20 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.287+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9fc068a10 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.288+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fa9ec01c070 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.289+0000 7fa9f9600640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 0x7fa9dc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:14.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.289+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 6159+0+0 (secure 0 0 0) 0x7fa9ec08bc70 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.290+0000 7faa00c00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 0x7fa9dc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.291+0000 7faa00c00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 0x7fa9dc06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc1430f0 tx=0x7fa9e8008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:14.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.294+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa9ec059dc0 con 0x7fa9fc0cd2f0 2024-04-17T07:33:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:14.499+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa9fc068610 con 0x7fa9fc0cd2f0 2024-04-17T07:33:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.035+0000 7fa9f9600640 1 -- 172.21.15.67:0/2902834819 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) ==== 169+0+0 (secure 0 0 0) 0x7fa9ec0596c0 con 0x7fa9fc0cd2f0 2024-04-17T07:33:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:33:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 msgr2=0x7fa9dc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 0x7fa9dc06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc1430f0 tx=0x7fa9e8008040 comp rx=0 tx=0).stop 2024-04-17T07:33:15.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 msgr2=0x7fa9fc14fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00ba50 tx=0x7fa9ec00bf10 comp rx=0 tx=0).stop 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 shutdown_connections 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa9dc06ca10 0x7fa9dc06eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9fc146080 0x7fa9fc13db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9fc0cd6d0 0x7fa9fc14dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.040+0000 7faa02a00640 1 --2- 172.21.15.67:0/2902834819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9fc0cd2f0 0x7fa9fc14fbb0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.041+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 >> 172.21.15.67:0/2902834819 conn(0x7fa9fc06b8c0 msgr2=0x7fa9fc0af530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.041+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 shutdown_connections 2024-04-17T07:33:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.041+0000 7faa02a00640 1 -- 172.21.15.67:0/2902834819 wait complete. 2024-04-17T07:33:15.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:33:15.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.292+0000 7fc0c5a00640 1 Processor -- start 2024-04-17T07:33:15.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.293+0000 7fc0c5a00640 1 -- start start 2024-04-17T07:33:15.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c015ecb0 0x7fc0c00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c00d00b0 con 0x7fc0c00d3210 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c00d3670 con 0x7fc0c00cfab0 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c00d0c30 con 0x7fc0c015ecb0 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.294+0000 7fc0c5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54362/0 (socket says 172.21.15.67:54362) 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 -- 172.21.15.67:0/2636876667 learned_addr learned my addr 172.21.15.67:0/2636876667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0b7e00640 1 --1- 172.21.15.67:0/2636876667 >> v1:172.21.15.67:6789/0 conn(0x7fc0c00d3210 0x7fc0c00cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42150/0 (socket says 172.21.15.67:42150) 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0bfe00640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c015ecb0 0x7fc0c00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 -- 172.21.15.67:0/2636876667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c015ecb0 msgr2=0x7fc0c00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c015ecb0 0x7fc0c00d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 -- 172.21.15.67:0/2636876667 >> v1:172.21.15.67:6789/0 conn(0x7fc0c00d3210 legacy=0x7fc0c00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.295+0000 7fc0c5000640 1 -- 172.21.15.67:0/2636876667 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c00d0390 con 0x7fc0c00cfab0 2024-04-17T07:33:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.296+0000 7fc0c5000640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009e50 tx=0x7fc0b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab714948877b352 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.296+0000 7fc0bf400640 1 -- 172.21.15.67:0/2636876667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b802ca80 con 0x7fc0c00cfab0 2024-04-17T07:33:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.296+0000 7fc0bf400640 1 -- 172.21.15.67:0/2636876667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b802cc20 con 0x7fc0c00cfab0 2024-04-17T07:33:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.296+0000 7fc0bf400640 1 -- 172.21.15.67:0/2636876667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b80343f0 con 0x7fc0c00cfab0 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 -- 172.21.15.67:0/2636876667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 msgr2=0x7fc0c00b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009e50 tx=0x7fc0b802bee0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 -- 172.21.15.67:0/2636876667 shutdown_connections 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c00b3fc0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/2636876667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c015ecb0 0x7fc0c00d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 -- 172.21.15.67:0/2636876667 >> 172.21.15.67:0/2636876667 conn(0x7fc0c008bff0 msgr2=0x7fc0c00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 -- 172.21.15.67:0/2636876667 shutdown_connections 2024-04-17T07:33:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.297+0000 7fc0c5a00640 1 -- 172.21.15.67:0/2636876667 wait complete. 2024-04-17T07:33:15.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.299+0000 7fc0c5a00640 1 Processor -- start 2024-04-17T07:33:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.299+0000 7fc0c5a00640 1 -- start start 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c00d3210 0x7fc0c00cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c015ecb0 0x7fc0c00cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c015f090 con 0x7fc0c00cfab0 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00d0950 con 0x7fc0c015ecb0 2024-04-17T07:33:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0c5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c015f210 con 0x7fc0c00d3210 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.300+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54368/0 (socket says 172.21.15.67:54368) 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 -- 172.21.15.67:0/3317492491 learned_addr learned my addr 172.21.15.67:0/3317492491 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0b7e00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c00d3210 0x7fc0c00cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0c5000640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c015ecb0 0x7fc0c00cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 -- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c00d3210 msgr2=0x7fc0c00cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c00d3210 0x7fc0c00cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 -- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c015ecb0 msgr2=0x7fc0c00cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c015ecb0 0x7fc0c00cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 -- 172.21.15.67:0/3317492491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c00cdb70 con 0x7fc0c00cfab0 2024-04-17T07:33:15.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.301+0000 7fc0bfe00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000d5f0 tx=0x7fc0b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.302+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00125f0 con 0x7fc0c00cfab0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.302+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b0019070 con 0x7fc0c00cfab0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.302+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00129d0 con 0x7fc0c00cfab0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.302+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c00cde50 con 0x7fc0c00cfab0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.302+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c00cc470 con 0x7fc0c00cfab0 2024-04-17T07:33:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.303+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c0068a10 con 0x7fc0c00cfab0 2024-04-17T07:33:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.304+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fc0b0012b70 con 0x7fc0c00cfab0 2024-04-17T07:33:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.305+0000 7fc0bcc00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 0x7fc09c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.306+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7fc0b008c050 con 0x7fc0c00cfab0 2024-04-17T07:33:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.306+0000 7fc0b7e00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 0x7fc09c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.307+0000 7fc0b7e00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 0x7fc09c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8007fd0 tx=0x7fc0a8007c40 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.310+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc0b005a3d0 con 0x7fc0c00cfab0 2024-04-17T07:33:15.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.530+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7fc0c00b4b20 con 0x7fc0c00cfab0 2024-04-17T07:33:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.535+0000 7fc0bcc00640 1 -- 172.21.15.67:0/3317492491 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) ==== 115+0+0 (secure 0 0 0) 0x7fc0b0059cd0 con 0x7fc0c00cfab0 2024-04-17T07:33:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 msgr2=0x7fc09c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 0x7fc09c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8007fd0 tx=0x7fc0a8007c40 comp rx=0 tx=0).stop 2024-04-17T07:33:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 msgr2=0x7fc0c006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000d5f0 tx=0x7fc0b000dab0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 shutdown_connections 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c015ecb0 0x7fc0c00cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc09c06ca40 0x7fc09c06ef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c00d3210 0x7fc0c00cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 --2- 172.21.15.67:0/3317492491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00cfab0 0x7fc0c006d510 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 >> 172.21.15.67:0/3317492491 conn(0x7fc0c008bff0 msgr2=0x7fc0c015dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 shutdown_connections 2024-04-17T07:33:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.538+0000 7fc0c5a00640 1 -- 172.21.15.67:0/3317492491 wait complete. 2024-04-17T07:33:15.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:33:15.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.770+0000 7f28cbe00640 1 Processor -- start 2024-04-17T07:33:15.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.771+0000 7f28cbe00640 1 -- start start 2024-04-17T07:33:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f28cc15ecb0 0x7f28cc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28cc0d00b0 con 0x7f28cc0cfab0 2024-04-17T07:33:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cbe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28cc0d0390 con 0x7f28cc0d3210 2024-04-17T07:33:15.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cbe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f28cc0d0c30 con 0x7f28cc15ecb0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f28cc0cfab0 0x7f28cc0b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42166/0 (socket says 172.21.15.67:42166) 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.772+0000 7f28cb400640 1 -- 172.21.15.67:0/1490998751 learned_addr learned my addr 172.21.15.67:0/1490998751 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28caa00640 1 --2- 172.21.15.67:0/1490998751 >> v2:172.21.15.187:3300/0 conn(0x7f28cc15ecb0 0x7f28cc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812595219 0 0) 0x7f28cc0d00b0 con 0x7f28cc0cfab0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ac003520 con 0x7f28cc0cfab0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 482687168 0 0) 0x7f28ac003520 con 0x7f28cc0cfab0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 >> v2:172.21.15.187:3300/0 conn(0x7f28cc15ecb0 msgr2=0x7f28cc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 --2- 172.21.15.67:0/1490998751 >> v2:172.21.15.187:3300/0 conn(0x7f28cc15ecb0 0x7f28cc0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 >> v1:172.21.15.187:6789/0 conn(0x7f28cc0d3210 legacy=0x7f28cc0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc0d0670 con 0x7f28cc0cfab0 2024-04-17T07:33:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f28c4001670 con 0x7f28cc0cfab0 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 >> v1:172.21.15.67:6789/0 conn(0x7f28cc0cfab0 legacy=0x7f28cc0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.773+0000 7f28c9600640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac003a00 0x7f28ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28c9600640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ac004800 0x7f28ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28c9600640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 0x7f28ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc0d0c30 con 0x7f28ac005210 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac00c140 con 0x7f28ac003a00 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f28ac00c340 con 0x7f28ac004800 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 0x7f28ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28caa00640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac003a00 0x7f28ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28ca000640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ac004800 0x7f28ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 -- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ac004800 msgr2=0x7f28ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28ac004800 0x7f28ac004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 -- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac003a00 msgr2=0x7f28ac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac003a00 0x7f28ac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28cb400640 1 -- 172.21.15.67:0/1490998751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac003520 con 0x7f28ac005210 2024-04-17T07:33:15.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.774+0000 7f28caa00640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac003a00 0x7f28ac003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.775+0000 7f28cb400640 1 -- 172.21.15.67:0/1490998751 reap_dead start 2024-04-17T07:33:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.775+0000 7f28cb400640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 0x7f28ac00b9c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c110 tx=0x7f28ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50e249532d22cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4007c40 con 0x7f28ac005210 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c4030070 con 0x7f28ac005210 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28c9600640 1 -- 172.21.15.67:0/1490998751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4005d30 con 0x7f28ac005210 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28cbe00640 1 -- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 msgr2=0x7f28ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28cbe00640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 0x7f28ac00b9c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f28c400c110 tx=0x7f28ac005830 comp rx=0 tx=0).stop 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28cbe00640 1 -- 172.21.15.67:0/1490998751 shutdown_connections 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28cbe00640 1 --2- 172.21.15.67:0/1490998751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28ac005210 0x7f28ac00b9c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.776+0000 7f28cbe00640 1 -- 172.21.15.67:0/1490998751 >> 172.21.15.67:0/1490998751 conn(0x7f28cc08bff0 msgr2=0x7f28cc0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.777+0000 7f28cbe00640 1 -- 172.21.15.67:0/1490998751 shutdown_connections 2024-04-17T07:33:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.777+0000 7f28cbe00640 1 -- 172.21.15.67:0/1490998751 wait complete. 2024-04-17T07:33:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.778+0000 7f28cbe00640 1 Processor -- start 2024-04-17T07:33:15.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.778+0000 7f28cbe00640 1 -- start start 2024-04-17T07:33:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.779+0000 7f28cbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac005210 0x7f28cc1f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28cc14f690 0x7f28cc1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc15f210 con 0x7f28cc14faf0 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc15f090 con 0x7f28ac005210 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f28cc0d2cd0 con 0x7f28cc14f690 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54384/0 (socket says 172.21.15.67:54384) 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 -- 172.21.15.67:0/757009171 learned_addr learned my addr 172.21.15.67:0/757009171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28caa00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac005210 0x7f28cc1f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 -- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28cc14f690 msgr2=0x7f28cc1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28cc14f690 0x7f28cc1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 -- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac005210 msgr2=0x7f28cc1f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac005210 0x7f28cc1f3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.780+0000 7f28cb400640 1 -- 172.21.15.67:0/757009171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc0a0910 con 0x7f28cc14faf0 2024-04-17T07:33:15.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28cb400640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f28c402c480 tx=0x7f28cc0d0560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4007c40 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c4004bf0 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28cc0a0bf0 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4001b80 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.781+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28cc0a11b0 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.782+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28cc068a10 con 0x7f28cc14faf0 2024-04-17T07:33:15.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.783+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f28c4007070 con 0x7f28cc14faf0 2024-04-17T07:33:15.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.785+0000 7f28c2a00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 0x7f28a806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:15.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.785+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7f28c40acc80 con 0x7f28cc14faf0 2024-04-17T07:33:15.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.786+0000 7f28caa00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 0x7f28a806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:15.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.787+0000 7f28caa00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 0x7f28a806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0099b0 tx=0x7f28bc008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.790+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f28c407af60 con 0x7f28cc14faf0 2024-04-17T07:33:15.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:15.998+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f28cc0b5a70 con 0x7f28cc14faf0 2024-04-17T07:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.004+0000 7f28c2a00640 1 -- 172.21.15.67:0/757009171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) ==== 104+0+0 (secure 0 0 0) 0x7f28c402de20 con 0x7f28cc14faf0 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 msgr2=0x7f28a806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 0x7f28a806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0099b0 tx=0x7f28bc008040 comp rx=0 tx=0).stop 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 msgr2=0x7f28cc155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f28c402c480 tx=0x7f28cc0d0560 comp rx=0 tx=0).stop 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 shutdown_connections 2024-04-17T07:33:16.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc14faf0 0x7f28cc155720 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f28a806ca10 0x7f28a806eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f28cc14f690 0x7f28cc1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 --2- 172.21.15.67:0/757009171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f28ac005210 0x7f28cc1f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 >> 172.21.15.67:0/757009171 conn(0x7f28cc08bff0 msgr2=0x7f28cc0cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 shutdown_connections 2024-04-17T07:33:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.007+0000 7f28cbe00640 1 -- 172.21.15.67:0/757009171 wait complete. 2024-04-17T07:33:16.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:33:16.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.254+0000 7f4a15600640 1 Processor -- start 2024-04-17T07:33:16.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.254+0000 7f4a15600640 1 -- start start 2024-04-17T07:33:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.255+0000 7f4a15600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4a10145f80 0x7f4a100cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.255+0000 7f4a15600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1006c270 0x7f4a1006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a15600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a10146360 con 0x7f4a100cd750 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a15600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a100727a0 con 0x7f4a1006c270 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a15600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4a10071e70 con 0x7f4a10145f80 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1006c270 0x7f4a1006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1006c270 0x7f4a1006e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54386/0 (socket says 172.21.15.67:54386) 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 -- 172.21.15.67:0/4057609258 learned_addr learned my addr 172.21.15.67:0/4057609258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a0fe00640 1 --2- 172.21.15.67:0/4057609258 >> v2:172.21.15.187:3300/0 conn(0x7f4a10145f80 0x7f4a100cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 -- 172.21.15.67:0/4057609258 >> v2:172.21.15.187:3300/0 conn(0x7f4a10145f80 msgr2=0x7f4a100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 --2- 172.21.15.67:0/4057609258 >> v2:172.21.15.187:3300/0 conn(0x7f4a10145f80 0x7f4a100cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 -- 172.21.15.67:0/4057609258 >> v1:172.21.15.187:6789/0 conn(0x7f4a100cd750 legacy=0x7f4a1006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.256+0000 7f4a14c00640 1 -- 172.21.15.67:0/4057609258 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a10072150 con 0x7f4a1006c270 2024-04-17T07:33:16.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a14c00640 1 --2- 172.21.15.67:0/4057609258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1006c270 0x7f4a1006e1c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f4a1006c9c0 tx=0x7f4a0802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ea88a4d1b85323 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a0f400640 1 -- 172.21.15.67:0/4057609258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0802e070 con 0x7f4a1006c270 2024-04-17T07:33:16.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a0f400640 1 -- 172.21.15.67:0/4057609258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a08037070 con 0x7f4a1006c270 2024-04-17T07:33:16.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a0f400640 1 -- 172.21.15.67:0/4057609258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08031a00 con 0x7f4a1006c270 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a15600640 1 -- 172.21.15.67:0/4057609258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1006c270 msgr2=0x7f4a1006e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.258+0000 7f4a15600640 1 --2- 172.21.15.67:0/4057609258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1006c270 0x7f4a1006e1c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f4a1006c9c0 tx=0x7f4a0802b880 comp rx=0 tx=0).stop 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 -- 172.21.15.67:0/4057609258 shutdown_connections 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 --2- 172.21.15.67:0/4057609258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1006c270 0x7f4a1006e1c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 --2- 172.21.15.67:0/4057609258 >> v2:172.21.15.187:3300/0 conn(0x7f4a10145f80 0x7f4a100cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 -- 172.21.15.67:0/4057609258 >> 172.21.15.67:0/4057609258 conn(0x7f4a100b2190 msgr2=0x7f4a100b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 -- 172.21.15.67:0/4057609258 shutdown_connections 2024-04-17T07:33:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.259+0000 7f4a15600640 1 -- 172.21.15.67:0/4057609258 wait complete. 2024-04-17T07:33:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.261+0000 7f4a15600640 1 Processor -- start 2024-04-17T07:33:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.261+0000 7f4a15600640 1 -- start start 2024-04-17T07:33:16.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1006c270 0x7f4a100c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a10145f80 0x7f4a10143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a10072d90 con 0x7f4a100cd750 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a10072c10 con 0x7f4a1006c270 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a15600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1006e700 con 0x7f4a10145f80 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54402/0 (socket says 172.21.15.67:54402) 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 -- 172.21.15.67:0/1525182435 learned_addr learned my addr 172.21.15.67:0/1525182435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a0fe00640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1006c270 0x7f4a100c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a14c00640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a10145f80 0x7f4a10143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 -- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a10145f80 msgr2=0x7f4a10143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a10145f80 0x7f4a10143b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 -- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1006c270 msgr2=0x7f4a100c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.263+0000 7f4a07400640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1006c270 0x7f4a100c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a07400640 1 -- 172.21.15.67:0/1525182435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a100c7730 con 0x7f4a100cd750 2024-04-17T07:33:16.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a07400640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c440 tx=0x7f49f800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8017070 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f80035f0 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a100c7a10 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f80039e0 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.264+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a10163b40 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.266+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a10068a10 con 0x7f4a100cd750 2024-04-17T07:33:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.267+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f49f801a070 con 0x7f4a100cd750 2024-04-17T07:33:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.268+0000 7f4a0cc00640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 0x7f49ec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.268+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7f49f808b170 con 0x7f4a100cd750 2024-04-17T07:33:16.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.268+0000 7f4a0fe00640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 0x7f49ec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.270+0000 7f4a0fe00640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 0x7f49ec06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009e90 tx=0x7f4a0000b040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.272+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f49f80593c0 con 0x7f4a100cd750 2024-04-17T07:33:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.483+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f4a10163d10 con 0x7f4a100cd750 2024-04-17T07:33:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.489+0000 7f4a0cc00640 1 -- 172.21.15.67:0/1525182435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v45) ==== 120+0+0 (secure 0 0 0) 0x7f49f80010c0 con 0x7f4a100cd750 2024-04-17T07:33:16.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 msgr2=0x7f49ec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 0x7f49ec06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009e90 tx=0x7f4a0000b040 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 msgr2=0x7f4a10145190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c440 tx=0x7f49f800c900 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 shutdown_connections 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49ec06ca10 0x7f49ec06eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a10145f80 0x7f4a10143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a100cd750 0x7f4a10145190 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 --2- 172.21.15.67:0/1525182435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1006c270 0x7f4a100c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 >> 172.21.15.67:0/1525182435 conn(0x7f4a100b2190 msgr2=0x7f4a100b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.492+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 shutdown_connections 2024-04-17T07:33:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.493+0000 7f4a15600640 1 -- 172.21.15.67:0/1525182435 wait complete. 2024-04-17T07:33:16.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-04-17T07:33:16.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.724+0000 7fced2600640 1 Processor -- start 2024-04-17T07:33:16.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.725+0000 7fced2600640 1 -- start start 2024-04-17T07:33:16.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.726+0000 7fced2600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcecc06b8c0 0x7fcecc0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.726+0000 7fced2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.726+0000 7fced2600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcecc0cd4a0 con 0x7fcecc141ed0 2024-04-17T07:33:16.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.726+0000 7fced2600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcecc15ce30 con 0x7fcecc06b8c0 2024-04-17T07:33:16.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.726+0000 7fced2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc1607f0 con 0x7fcecc160280 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcecc06b8c0 0x7fcecc0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcecc141ed0 0x7fcecc15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42170/0 (socket says 172.21.15.67:42170) 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1c00640 1 -- 172.21.15.67:0/271245618 learned_addr learned my addr 172.21.15.67:0/271245618 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fcecbe00640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1200640 1 -- 172.21.15.67:0/271245618 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 msgr2=0x7fcecc06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1200640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1200640 1 -- 172.21.15.67:0/271245618 >> v1:172.21.15.67:6789/0 conn(0x7fcecc141ed0 legacy=0x7fcecc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fced1200640 1 -- 172.21.15.67:0/271245618 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc160ad0 con 0x7fcecc06b8c0 2024-04-17T07:33:16.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.727+0000 7fcecb400640 1 -- 172.21.15.67:0/271245618 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389389894 0 0) 0x7fcecc0cd4a0 con 0x7fcecc141ed0 2024-04-17T07:33:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.729+0000 7fced1200640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0cd8b0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097a0 tx=0x7fcec002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc1b089cb571be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.729+0000 7fcecb400640 1 -- 172.21.15.67:0/271245618 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0035020 con 0x7fcecc06b8c0 2024-04-17T07:33:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.729+0000 7fcecb400640 1 -- 172.21.15.67:0/271245618 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec0031830 con 0x7fcecc06b8c0 2024-04-17T07:33:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.729+0000 7fcecb400640 1 -- 172.21.15.67:0/271245618 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0031c00 con 0x7fcecc06b8c0 2024-04-17T07:33:16.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 -- 172.21.15.67:0/271245618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 msgr2=0x7fcecc0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0cd8b0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097a0 tx=0x7fcec002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 -- 172.21.15.67:0/271245618 shutdown_connections 2024-04-17T07:33:16.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 --2- 172.21.15.67:0/271245618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0cd8b0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 -- 172.21.15.67:0/271245618 >> 172.21.15.67:0/271245618 conn(0x7fcecc067d10 msgr2=0x7fcecc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:16.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 -- 172.21.15.67:0/271245618 shutdown_connections 2024-04-17T07:33:16.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.730+0000 7fced2600640 1 -- 172.21.15.67:0/271245618 wait complete. 2024-04-17T07:33:16.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.732+0000 7fced2600640 1 Processor -- start 2024-04-17T07:33:16.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.732+0000 7fced2600640 1 -- start start 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcecc141ed0 0x7fcecc134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc06c1e0 con 0x7fcecc06b8c0 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc0cddf0 con 0x7fcecc141ed0 2024-04-17T07:33:16.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced2600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc161090 con 0x7fcecc160280 2024-04-17T07:33:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54416/0 (socket says 172.21.15.67:54416) 2024-04-17T07:33:16.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 -- 172.21.15.67:0/3779596380 learned_addr learned my addr 172.21.15.67:0/3779596380 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 -- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 msgr2=0x7fcecc15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 -- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcecc141ed0 msgr2=0x7fcecc134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcecc141ed0 0x7fcecc134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.734+0000 7fced1200640 1 -- 172.21.15.67:0/3779596380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc143180 con 0x7fcecc06b8c0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fced1200640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097a0 tx=0x7fcec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0004230 con 0x7fcecc06b8c0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec0004410 con 0x7fcecc06b8c0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcecc143460 con 0x7fcecc06b8c0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec003c910 con 0x7fcecc06b8c0 2024-04-17T07:33:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.735+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcecc143a20 con 0x7fcecc06b8c0 2024-04-17T07:33:16.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.737+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcecc075470 con 0x7fcecc06b8c0 2024-04-17T07:33:16.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.738+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fcec0038020 con 0x7fcecc06b8c0 2024-04-17T07:33:16.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.739+0000 7fcec8c00640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 0x7fceac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:16.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.739+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7fcec00ae0e0 con 0x7fcecc06b8c0 2024-04-17T07:33:16.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.739+0000 7fcecbe00640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 0x7fceac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.741+0000 7fcecbe00640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 0x7fceac06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fceb8006fd0 tx=0x7fceb8008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:16.743+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcecc075470 con 0x7fcecc06b8c0 2024-04-17T07:33:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.025+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7fcecc0cebf0 con 0x7fcecc06b8c0 2024-04-17T07:33:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.040+0000 7fcec8c00640 1 -- 172.21.15.67:0/3779596380 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v753) ==== 148+0+0 (secure 0 0 0) 0x7fcecc0cebf0 con 0x7fcecc06b8c0 2024-04-17T07:33:17.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 msgr2=0x7fceac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 0x7fceac06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fceb8006fd0 tx=0x7fceb8008040 comp rx=0 tx=0).stop 2024-04-17T07:33:17.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 msgr2=0x7fcecc0aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fcec00097a0 tx=0x7fcec00046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 shutdown_connections 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fceac06ca10 0x7fceac06eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcecc160280 0x7fcecc15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcecc141ed0 0x7fcecc134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 --2- 172.21.15.67:0/3779596380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc06b8c0 0x7fcecc0aee40 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 >> 172.21.15.67:0/3779596380 conn(0x7fcecc067d10 msgr2=0x7fcecc0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 shutdown_connections 2024-04-17T07:33:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.043+0000 7fced2600640 1 -- 172.21.15.67:0/3779596380 wait complete. 2024-04-17T07:33:17.063 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-04-17T07:33:17.063 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:33:17.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.290+0000 7fc9e3200640 1 Processor -- start 2024-04-17T07:33:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.290+0000 7fc9e3200640 1 -- start start 2024-04-17T07:33:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.292+0000 7fc9e3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.292+0000 7fc9e3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc9dc06be70 0x7fc9dc15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.292+0000 7fc9e3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9dc15bc80 con 0x7fc9dc15b570 2024-04-17T07:33:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.292+0000 7fc9e3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc9dc15bf40 con 0x7fc9dc06be70 2024-04-17T07:33:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.292+0000 7fc9e3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9dc15c220 con 0x7fc9dc06feb0 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e2800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc9dc06be70 0x7fc9dc15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:48830/0 (socket says 172.21.15.67:48830) 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1400640 1 -- 172.21.15.67:0/3991936616 learned_addr learned my addr 172.21.15.67:0/3991936616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1e00640 1 --1- 172.21.15.67:0/3991936616 >> v1:172.21.15.187:6789/0 conn(0x7fc9dc15b570 0x7fc9dc06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:52488/0 (socket says 172.21.15.67:52488) 2024-04-17T07:33:17.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1400640 1 -- 172.21.15.67:0/3991936616 >> v2:172.21.15.187:3300/0 conn(0x7fc9dc06be70 msgr2=0x7fc9dc15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:17.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.293+0000 7fc9e1400640 1 --2- 172.21.15.67:0/3991936616 >> v2:172.21.15.187:3300/0 conn(0x7fc9dc06be70 0x7fc9dc15f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.294+0000 7fc9e1400640 1 -- 172.21.15.67:0/3991936616 >> v1:172.21.15.187:6789/0 conn(0x7fc9dc15b570 legacy=0x7fc9dc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.294+0000 7fc9e1400640 1 -- 172.21.15.67:0/3991936616 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc15c500 con 0x7fc9dc06feb0 2024-04-17T07:33:17.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.295+0000 7fc9e1400640 1 --2- 172.21.15.67:0/3991936616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009870 tx=0x7fc9cc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2addac45533bca36 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.296+0000 7fc9e0a00640 1 -- 172.21.15.67:0/3991936616 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc035020 con 0x7fc9dc06feb0 2024-04-17T07:33:17.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.296+0000 7fc9e0a00640 1 -- 172.21.15.67:0/3991936616 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9cc031640 con 0x7fc9dc06feb0 2024-04-17T07:33:17.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.296+0000 7fc9e0a00640 1 -- 172.21.15.67:0/3991936616 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc02d040 con 0x7fc9dc06feb0 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 -- 172.21.15.67:0/3991936616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 msgr2=0x7fc9dc06b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 --2- 172.21.15.67:0/3991936616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009870 tx=0x7fc9cc02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 -- 172.21.15.67:0/3991936616 shutdown_connections 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 --2- 172.21.15.67:0/3991936616 >> v2:172.21.15.187:3300/0 conn(0x7fc9dc06be70 0x7fc9dc15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 --2- 172.21.15.67:0/3991936616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc06feb0 0x7fc9dc06b930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 -- 172.21.15.67:0/3991936616 >> 172.21.15.67:0/3991936616 conn(0x7fc9dc067d10 msgr2=0x7fc9dc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 -- 172.21.15.67:0/3991936616 shutdown_connections 2024-04-17T07:33:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.297+0000 7fc9e3200640 1 -- 172.21.15.67:0/3991936616 wait complete. 2024-04-17T07:33:17.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.299+0000 7fc9e3200640 1 Processor -- start 2024-04-17T07:33:17.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.299+0000 7fc9e3200640 1 -- start start 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 0x7fc9dc1f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc15b570 0x7fc9dc142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc06c460 con 0x7fc9dc06be70 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc06c2e0 con 0x7fc9dc06feb0 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9dc0b47b0 con 0x7fc9dc15b570 2024-04-17T07:33:17.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54428/0 (socket says 172.21.15.67:54428) 2024-04-17T07:33:17.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e1e00640 1 -- 172.21.15.67:0/4063918131 learned_addr learned my addr 172.21.15.67:0/4063918131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.300+0000 7fc9e1400640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 0x7fc9dc1f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e2800640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc15b570 0x7fc9dc142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 -- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc15b570 msgr2=0x7fc9dc142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc15b570 0x7fc9dc142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 -- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 msgr2=0x7fc9dc1f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 0x7fc9dc1f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 -- 172.21.15.67:0/4063918131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc1f55d0 con 0x7fc9dc06be70 2024-04-17T07:33:17.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1400640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 0x7fc9dc1f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e1e00640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000c3c0 tx=0x7fc9d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00074e0 con 0x7fc9dc06be70 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d0007680 con 0x7fc9dc06be70 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9dc142d70 con 0x7fc9dc06be70 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0004a00 con 0x7fc9dc06be70 2024-04-17T07:33:17.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.301+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9dc1563f0 con 0x7fc9dc06be70 2024-04-17T07:33:17.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.303+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9dc068a10 con 0x7fc9dc06be70 2024-04-17T07:33:17.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.304+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fc9d001a020 con 0x7fc9dc06be70 2024-04-17T07:33:17.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.304+0000 7fc9d6000640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 0x7fc9c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.305+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7fc9d008b560 con 0x7fc9dc06be70 2024-04-17T07:33:17.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.305+0000 7fc9e1400640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 0x7fc9c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.306+0000 7fc9e1400640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 0x7fc9c006eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc036000 tx=0x7fc9cc008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.309+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc9d00597b0 con 0x7fc9dc06be70 2024-04-17T07:33:17.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.514+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9dc158920 con 0x7fc9dc06be70 2024-04-17T07:33:17.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.516+0000 7fc9d6000640 1 -- 172.21.15.67:0/4063918131 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+14170 (secure 0 0 0) 0x7fc9d00590b0 con 0x7fc9dc06be70 2024-04-17T07:33:17.516 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:17.516 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":133,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:15.030832+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/1617335847":"2024-04-18T07:33:12.005666+0000","172.21.15.187:0/1738540495":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3255864985":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6833/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3710283205":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2982226538":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/4043320648":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/1830624579":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/2041663839":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/757480182":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/2989376162":"2024-04-18T07:32:49.473454+0000","172.21.15.187:6833/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1667247431":"2024-04-18T07:33:07.903110+0000","172.21.15.187:6832/2517863062":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/3546270980":"2024-04-18T07:33:07.903110+0000","172.21.15.187:0/3883423100":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/410647397":"2024-04-18T07:32:49.473454+0000","172.21.15.67:6835/1617335847":"2024-04-18T07:33:12.005666+0000","172.21.15.187:0/2591672672":"2024-04-18T07:32:49.473454+0000","172.21.15.187:0/1399211335":"2024-04-18T07:32:49.473454+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 msgr2=0x7fc9c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 0x7fc9c006eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc036000 tx=0x7fc9cc008030 comp rx=0 tx=0).stop 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 msgr2=0x7fc9dc1559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000c3c0 tx=0x7fc9d000c880 comp rx=0 tx=0).stop 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 shutdown_connections 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fc9c006ca10 0x7fc9c006eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9dc15b570 0x7fc9dc142830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9dc06feb0 0x7fc9dc1f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 --2- 172.21.15.67:0/4063918131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9dc06be70 0x7fc9dc1559c0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.518+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 >> 172.21.15.67:0/4063918131 conn(0x7fc9dc067d10 msgr2=0x7fc9dc0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.519+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 shutdown_connections 2024-04-17T07:33:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.519+0000 7fc9e3200640 1 -- 172.21.15.67:0/4063918131 wait complete. 2024-04-17T07:33:17.540 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:17.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:17.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.762+0000 7f470f200640 1 Processor -- start 2024-04-17T07:33:17.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.762+0000 7f470f200640 1 -- start start 2024-04-17T07:33:17.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47100d33d0 0x7f4710161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47101624f0 con 0x7f47101620e0 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47100b4e40 con 0x7f47100d2ff0 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f47100b5110 con 0x7f47100d33d0 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47100d33d0 0x7f4710161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.763+0000 7f470e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47101620e0 0x7f47100cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42186/0 (socket says 172.21.15.67:42186) 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.764+0000 7f470e800640 1 -- 172.21.15.67:0/3715931614 learned_addr learned my addr 172.21.15.67:0/3715931614 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.764+0000 7f470d400640 1 -- 172.21.15.67:0/3715931614 >> v1:172.21.15.187:6789/0 conn(0x7f47100d2ff0 legacy=0x7f47100b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.764+0000 7f470d400640 1 -- 172.21.15.67:0/3715931614 >> v1:172.21.15.67:6789/0 conn(0x7f47101620e0 legacy=0x7f47100cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.764+0000 7f470d400640 1 -- 172.21.15.67:0/3715931614 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47100b53f0 con 0x7f47100d33d0 2024-04-17T07:33:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.764+0000 7f470ca00640 1 -- 172.21.15.67:0/3715931614 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872658835 0 0) 0x7f47101624f0 con 0x7f47101620e0 2024-04-17T07:33:17.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.766+0000 7f470d400640 1 --2- 172.21.15.67:0/3715931614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d33d0 0x7f4710161ba0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f46f80097a0 tx=0x7f46f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c48cc31ed8b084e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.766+0000 7f470ca00640 1 -- 172.21.15.67:0/3715931614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8034070 con 0x7f47100d33d0 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.766+0000 7f470ca00640 1 -- 172.21.15.67:0/3715931614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f8030610 con 0x7f47100d33d0 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.766+0000 7f470ca00640 1 -- 172.21.15.67:0/3715931614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f80309e0 con 0x7f47100d33d0 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 -- 172.21.15.67:0/3715931614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d33d0 msgr2=0x7f4710161ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 --2- 172.21.15.67:0/3715931614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d33d0 0x7f4710161ba0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f46f80097a0 tx=0x7f46f802b960 comp rx=0 tx=0).stop 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 -- 172.21.15.67:0/3715931614 shutdown_connections 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 --2- 172.21.15.67:0/3715931614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47100d33d0 0x7f4710161ba0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 -- 172.21.15.67:0/3715931614 >> 172.21.15.67:0/3715931614 conn(0x7f471009da80 msgr2=0x7f471008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 -- 172.21.15.67:0/3715931614 shutdown_connections 2024-04-17T07:33:17.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.767+0000 7f470f200640 1 -- 172.21.15.67:0/3715931614 wait complete. 2024-04-17T07:33:17.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.769+0000 7f470f200640 1 Processor -- start 2024-04-17T07:33:17.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.769+0000 7f470f200640 1 -- start start 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47100d2ff0 0x7f4710159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47100d33d0 0x7f47101f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47100cf4c0 con 0x7f47101620e0 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47100cf340 con 0x7f47100d33d0 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47100d26d0 con 0x7f47100d2ff0 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54436/0 (socket says 172.21.15.67:54436) 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.770+0000 7f470e800640 1 -- 172.21.15.67:0/2447103358 learned_addr learned my addr 172.21.15.67:0/2447103358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:17.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470d400640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47100d33d0 0x7f47101f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 -- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47100d2ff0 msgr2=0x7f4710159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47100d2ff0 0x7f4710159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 -- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47100d33d0 msgr2=0x7f47101f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47100d33d0 0x7f47101f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 -- 172.21.15.67:0/2447103358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47101f4d10 con 0x7f47101620e0 2024-04-17T07:33:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470e800640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f470800caf0 tx=0x7f47080021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470800ac00 con 0x7f47101620e0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470800ada0 con 0x7f47101620e0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708012640 con 0x7f47101620e0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.771+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47101f4ff0 con 0x7f47101620e0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.772+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47101f59c0 con 0x7f47101620e0 2024-04-17T07:33:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.773+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47100755d0 con 0x7f47101620e0 2024-04-17T07:33:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.774+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f47080127e0 con 0x7f47101620e0 2024-04-17T07:33:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.775+0000 7f4706000640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 0x7f46ec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.775+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7f470808c320 con 0x7f47101620e0 2024-04-17T07:33:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.775+0000 7f470de00640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 0x7f46ec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.776+0000 7f470de00640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 0x7f46ec06ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4700000f60 tx=0x7f47000079b0 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:17.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.779+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f470805a600 con 0x7f47101620e0 2024-04-17T07:33:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.986+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47100b28b0 con 0x7f47101620e0 2024-04-17T07:33:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.987+0000 7f4706000640 1 -- 172.21.15.67:0/2447103358 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) ==== 93+0+10 (secure 0 0 0) 0x7f4708059f00 con 0x7f47101620e0 2024-04-17T07:33:17.987 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 msgr2=0x7f46ec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 0x7f46ec06ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4700000f60 tx=0x7f47000079b0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 msgr2=0x7f47101f4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f470800caf0 tx=0x7f47080021c0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 shutdown_connections 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47101620e0 0x7f47101f4730 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47100d33d0 0x7f47101f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f46ec06ca40 0x7f46ec06ef00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 --2- 172.21.15.67:0/2447103358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47100d2ff0 0x7f4710159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:17.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.989+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 >> 172.21.15.67:0/2447103358 conn(0x7f471009da80 msgr2=0x7f47100b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.990+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 shutdown_connections 2024-04-17T07:33:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:17.990+0000 7f470f200640 1 -- 172.21.15.67:0/2447103358 wait complete. 2024-04-17T07:33:18.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-17T07:33:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.232+0000 7f5b42800640 1 Processor -- start 2024-04-17T07:33:18.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.233+0000 7f5b42800640 1 -- start start 2024-04-17T07:33:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.234+0000 7f5b42800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5b3c0cd480 0x7f5b3c06e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.234+0000 7f5b42800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3c0ceb00 con 0x7f5b3c06f670 2024-04-17T07:33:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.234+0000 7f5b42800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3c0cede0 con 0x7f5b3c0cd020 2024-04-17T07:33:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.234+0000 7f5b42800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5b3c0cf0c0 con 0x7f5b3c0cd480 2024-04-17T07:33:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b3c06f670 0x7f5b3c06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42192/0 (socket says 172.21.15.67:42192) 2024-04-17T07:33:18.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3b400640 1 -- 172.21.15.67:0/1992336575 learned_addr learned my addr 172.21.15.67:0/1992336575 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3604557510 0 0) 0x7f5b3c0ceb00 con 0x7f5b3c06f670 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b1c003520 con 0x7f5b3c06f670 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3be00640 1 --2- 172.21.15.67:0/1992336575 >> v2:172.21.15.187:3300/0 conn(0x7f5b3c0cd480 0x7f5b3c06e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2147159019 0 0) 0x7f5b1c003520 con 0x7f5b3c06f670 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 >> v2:172.21.15.187:3300/0 conn(0x7f5b3c0cd480 msgr2=0x7f5b3c06e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 --2- 172.21.15.67:0/1992336575 >> v2:172.21.15.187:3300/0 conn(0x7f5b3c0cd480 0x7f5b3c06e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.235+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 >> v1:172.21.15.187:6789/0 conn(0x7f5b3c0cd020 legacy=0x7f5b3c06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c0cf3a0 con 0x7f5b3c06f670 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5b2c001670 con 0x7f5b3c06f670 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 >> v1:172.21.15.67:6789/0 conn(0x7f5b3c06f670 legacy=0x7f5b3c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c003a00 0x7f5b1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c004800 0x7f5b1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c0cf0c0 con 0x7f5b1c005210 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c00c160 con 0x7f5b1c004800 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1c00c360 con 0x7f5b1c003a00 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3be00640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3aa00640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c004800 0x7f5b1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3b400640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c003a00 0x7f5b1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3be00640 1 -- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c003a00 msgr2=0x7f5b1c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3be00640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c003a00 0x7f5b1c003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3be00640 1 -- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c004800 msgr2=0x7f5b1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.236+0000 7f5b3be00640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c004800 0x7f5b1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.237+0000 7f5b3be00640 1 -- 172.21.15.67:0/1992336575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c003520 con 0x7f5b1c005210 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.237+0000 7f5b3be00640 1 -- 172.21.15.67:0/1992336575 reap_dead start 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b3be00640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400ba30 tx=0x7f5b1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38840ed9ea01917 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34036030 con 0x7f5b1c005210 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b34004020 con 0x7f5b1c005210 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b3a000640 1 -- 172.21.15.67:0/1992336575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3400b600 con 0x7f5b1c005210 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b42800640 1 -- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 msgr2=0x7f5b1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.238+0000 7f5b42800640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400ba30 tx=0x7f5b1c005830 comp rx=0 tx=0).stop 2024-04-17T07:33:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.239+0000 7f5b42800640 1 -- 172.21.15.67:0/1992336575 shutdown_connections 2024-04-17T07:33:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.239+0000 7f5b42800640 1 --2- 172.21.15.67:0/1992336575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.239+0000 7f5b42800640 1 -- 172.21.15.67:0/1992336575 >> 172.21.15.67:0/1992336575 conn(0x7f5b3c06bf50 msgr2=0x7f5b3c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.239+0000 7f5b42800640 1 -- 172.21.15.67:0/1992336575 shutdown_connections 2024-04-17T07:33:18.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.239+0000 7f5b42800640 1 -- 172.21.15.67:0/1992336575 wait complete. 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.241+0000 7f5b42800640 1 Processor -- start 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.241+0000 7f5b42800640 1 -- start start 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b3c157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 0x7f5b3c0c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c0ce8a0 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c0ce720 con 0x7f5b3c0c9210 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b42800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3c15cb00 con 0x7f5b1c005210 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54458/0 (socket says 172.21.15.67:54458) 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.242+0000 7f5b3be00640 1 -- 172.21.15.67:0/204761440 learned_addr learned my addr 172.21.15.67:0/204761440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3aa00640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 0x7f5b3c0c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3b400640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b3c157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 -- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 msgr2=0x7f5b3c157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b3c157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 -- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 msgr2=0x7f5b3c0c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 0x7f5b3c0c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 -- 172.21.15.67:0/204761440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c0ca370 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3aa00640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 0x7f5b3c0c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3b400640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b3c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b3be00640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400bb50 tx=0x7f5b34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34041030 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b34037040 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.243+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3c0c9db0 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.244+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34036070 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.244+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b3c0c5450 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.245+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3c06df20 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.246+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f5b34044030 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.247+0000 7f5b33400640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 0x7f5b1806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:18.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.247+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7f5b340adbd0 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.248+0000 7f5b3b400640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 0x7f5b1806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:18.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.249+0000 7f5b3b400640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 0x7f5b1806ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0017e0 tx=0x7f5b2c009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:18.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.253+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5b3c06df20 con 0x7f5b3c14d8a0 2024-04-17T07:33:18.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:18.540+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7f5b3c068b90 con 0x7f5b3c14d8a0 2024-04-17T07:33:19.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.420+0000 7f5b33400640 1 -- 172.21.15.67:0/204761440 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v755) ==== 163+0+0 (secure 0 0 0) 0x7f5b3407be40 con 0x7f5b3c14d8a0 2024-04-17T07:33:19.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.425+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 msgr2=0x7f5b1806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.425+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 0x7f5b1806ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0017e0 tx=0x7f5b2c009040 comp rx=0 tx=0).stop 2024-04-17T07:33:19.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.425+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 msgr2=0x7f5b3c14dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.425+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400bb50 tx=0x7f5b34004730 comp rx=0 tx=0).stop 2024-04-17T07:33:19.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 shutdown_connections 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b3c14d8a0 0x7f5b3c14dd20 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b3c0c9210 0x7f5b3c0c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5b1806ca40 0x7f5b1806ef00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 --2- 172.21.15.67:0/204761440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b3c157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 >> 172.21.15.67:0/204761440 conn(0x7f5b3c06bf50 msgr2=0x7f5b3c15a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 shutdown_connections 2024-04-17T07:33:19.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.426+0000 7f5b42800640 1 -- 172.21.15.67:0/204761440 wait complete. 2024-04-17T07:33:19.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:33:19.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.675+0000 7efeca200640 1 Processor -- start 2024-04-17T07:33:19.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.676+0000 7efeca200640 1 -- start start 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.677+0000 7efeca200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.677+0000 7efeca200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efec40d3210 0x7efec40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.677+0000 7efeca200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec40d00b0 con 0x7efec40cf8e0 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.677+0000 7efeca200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efec40d3670 con 0x7efec40d3210 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.677+0000 7efeca200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efec40d0c30 con 0x7efec415ecb0 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efec40d3210 0x7efec40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efec40d3210 0x7efec40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54460/0 (socket says 172.21.15.67:54460) 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 -- 172.21.15.67:0/2635248898 learned_addr learned my addr 172.21.15.67:0/2635248898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec9800640 1 --1- 172.21.15.67:0/2635248898 >> v1:172.21.15.67:6789/0 conn(0x7efec40cf8e0 0x7efec40d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42206/0 (socket says 172.21.15.67:42206) 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec8e00640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 -- 172.21.15.67:0/2635248898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 msgr2=0x7efec40d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 -- 172.21.15.67:0/2635248898 >> v1:172.21.15.67:6789/0 conn(0x7efec40cf8e0 legacy=0x7efec40d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3e00640 1 -- 172.21.15.67:0/2635248898 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40d0390 con 0x7efec40d3210 2024-04-17T07:33:19.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.678+0000 7efec3400640 1 -- 172.21.15.67:0/2635248898 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661579405 0 0) 0x7efec40d00b0 con 0x7efec40cf8e0 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.679+0000 7efec3e00640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40d3210 0x7efec40cf3a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efeac009870 tx=0x7efeac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9164c47a17ad99f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efec3400640 1 -- 172.21.15.67:0/2635248898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac035020 con 0x7efec40d3210 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efec3400640 1 -- 172.21.15.67:0/2635248898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeac031640 con 0x7efec40d3210 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efec3400640 1 -- 172.21.15.67:0/2635248898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac02d040 con 0x7efec40d3210 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 -- 172.21.15.67:0/2635248898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40d3210 msgr2=0x7efec40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40d3210 0x7efec40cf3a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efeac009870 tx=0x7efeac02b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 -- 172.21.15.67:0/2635248898 shutdown_connections 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40d3210 0x7efec40cf3a0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 --2- 172.21.15.67:0/2635248898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.680+0000 7efeca200640 1 -- 172.21.15.67:0/2635248898 >> 172.21.15.67:0/2635248898 conn(0x7efec408bff0 msgr2=0x7efec40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.681+0000 7efeca200640 1 -- 172.21.15.67:0/2635248898 shutdown_connections 2024-04-17T07:33:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.681+0000 7efeca200640 1 -- 172.21.15.67:0/2635248898 wait complete. 2024-04-17T07:33:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.682+0000 7efeca200640 1 Processor -- start 2024-04-17T07:33:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.682+0000 7efeca200640 1 -- start start 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec40d3210 0x7efec414b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efec40cfd50 con 0x7efec40cf8e0 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efec40d0950 con 0x7efec40d3210 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.683+0000 7efeca200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efec415f090 con 0x7efec415ecb0 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54466/0 (socket says 172.21.15.67:54466) 2024-04-17T07:33:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 -- 172.21.15.67:0/2322166478 learned_addr learned my addr 172.21.15.67:0/2322166478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec9800640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec3e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec40d3210 0x7efec414b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 -- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 msgr2=0x7efec40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 -- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec40d3210 msgr2=0x7efec414b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec40d3210 0x7efec414b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.684+0000 7efec8e00640 1 -- 172.21.15.67:0/2322166478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40cdba0 con 0x7efec40cf8e0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efec8e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7efeb4007ef0 tx=0x7efeb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4014620 con 0x7efec40cf8e0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb4019070 con 0x7efec40cf8e0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4014a00 con 0x7efec40cf8e0 2024-04-17T07:33:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec40cde80 con 0x7efec40cf8e0 2024-04-17T07:33:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.685+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec40ce440 con 0x7efec40cf8e0 2024-04-17T07:33:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.686+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec4068a10 con 0x7efec40cf8e0 2024-04-17T07:33:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.687+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7efeb4014ba0 con 0x7efec40cf8e0 2024-04-17T07:33:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.688+0000 7efec0c00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 0x7efea006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.688+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7efeb408c710 con 0x7efec40cf8e0 2024-04-17T07:33:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.689+0000 7efec3e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 0x7efea006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.690+0000 7efec3e00640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 0x7efea006eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efeac036000 tx=0x7efeac008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.692+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efeb405a960 con 0x7efec40cf8e0 2024-04-17T07:33:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.896+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7efec40b5a70 con 0x7efec40cf8e0 2024-04-17T07:33:19.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.902+0000 7efec0c00640 1 -- 172.21.15.67:0/2322166478 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7efeb405a260 con 0x7efec40cf8e0 2024-04-17T07:33:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 msgr2=0x7efea006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 0x7efea006eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efeac036000 tx=0x7efeac008030 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 msgr2=0x7efec40ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7efeb4007ef0 tx=0x7efeb400e4b0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 shutdown_connections 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efea006ca10 0x7efea006eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efec415ecb0 0x7efec40cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efec40d3210 0x7efec414b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 --2- 172.21.15.67:0/2322166478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec40cf8e0 0x7efec40ca800 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 >> 172.21.15.67:0/2322166478 conn(0x7efec408bff0 msgr2=0x7efec406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 shutdown_connections 2024-04-17T07:33:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:19.905+0000 7efeca200640 1 -- 172.21.15.67:0/2322166478 wait complete. 2024-04-17T07:33:19.927 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:33:19.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:20.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:20.165 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:33:20.165 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:20.165 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:20.228 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:33:20.228 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:20.228 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:20.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:20.283 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:20.283 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:20.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:33:20.344 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:20.344 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:20.401 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:20.401 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:33:20.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:20.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.618+0000 7ff117a00640 1 Processor -- start 2024-04-17T07:33:20.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.618+0000 7ff117a00640 1 -- start start 2024-04-17T07:33:20.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff110166440 0x7ff1100ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1100c6f90 0x7ff110071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff110072250 0x7ff110166980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff1100726c0 con 0x7ff1100c6f90 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff1100ce960 con 0x7ff110166440 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.620+0000 7ff117a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff110072b20 con 0x7ff110072250 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1100c6f90 0x7ff110071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1100c6f90 0x7ff110071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54470/0 (socket says 172.21.15.67:54470) 2024-04-17T07:33:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 -- 172.21.15.67:0/3981256130 learned_addr learned my addr 172.21.15.67:0/3981256130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff115a00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff110072250 0x7ff110166980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff115000640 1 --2- 172.21.15.67:0/3981256130 >> v2:172.21.15.187:3300/0 conn(0x7ff110166440 0x7ff1100ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 -- 172.21.15.67:0/3981256130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff110072250 msgr2=0x7ff110166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff110072250 0x7ff110166980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 -- 172.21.15.67:0/3981256130 >> v2:172.21.15.187:3300/0 conn(0x7ff110166440 msgr2=0x7ff1100ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 --2- 172.21.15.67:0/3981256130 >> v2:172.21.15.187:3300/0 conn(0x7ff110166440 0x7ff1100ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.621+0000 7ff10fe00640 1 -- 172.21.15.67:0/3981256130 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1100c5eb0 con 0x7ff1100c6f90 2024-04-17T07:33:20.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.623+0000 7ff10fe00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1100c6f90 0x7ff110071d10 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7ff110072d80 tx=0x7ff10402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf761800666e96f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:20.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.623+0000 7ff10f400640 1 -- 172.21.15.67:0/3981256130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10402e070 con 0x7ff1100c6f90 2024-04-17T07:33:20.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.623+0000 7ff10f400640 1 -- 172.21.15.67:0/3981256130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff104037070 con 0x7ff1100c6f90 2024-04-17T07:33:20.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.623+0000 7ff10f400640 1 -- 172.21.15.67:0/3981256130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104031a00 con 0x7ff1100c6f90 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 -- 172.21.15.67:0/3981256130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1100c6f90 msgr2=0x7ff110071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1100c6f90 0x7ff110071d10 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7ff110072d80 tx=0x7ff10402b880 comp rx=0 tx=0).stop 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 -- 172.21.15.67:0/3981256130 shutdown_connections 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff110072250 0x7ff110166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 --2- 172.21.15.67:0/3981256130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1100c6f90 0x7ff110071d10 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 --2- 172.21.15.67:0/3981256130 >> v2:172.21.15.187:3300/0 conn(0x7ff110166440 0x7ff1100ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 -- 172.21.15.67:0/3981256130 >> 172.21.15.67:0/3981256130 conn(0x7ff11006bf50 msgr2=0x7ff1100b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:20.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 -- 172.21.15.67:0/3981256130 shutdown_connections 2024-04-17T07:33:20.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.624+0000 7ff117a00640 1 -- 172.21.15.67:0/3981256130 wait complete. 2024-04-17T07:33:20.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.626+0000 7ff117a00640 1 Processor -- start 2024-04-17T07:33:20.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.626+0000 7ff117a00640 1 -- start start 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1100c6f90 0x7ff11014da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff110166440 0x7ff11014df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100cf0d0 con 0x7ff110072250 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100cebb0 con 0x7ff110166440 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff117a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff11014e6c0 con 0x7ff1100c6f90 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff115000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff115000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54474/0 (socket says 172.21.15.67:54474) 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.627+0000 7ff115000640 1 -- 172.21.15.67:0/1776355106 learned_addr learned my addr 172.21.15.67:0/1776355106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 -- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1100c6f90 msgr2=0x7ff11014da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1100c6f90 0x7ff11014da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 -- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff110166440 msgr2=0x7ff11014df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff110166440 0x7ff11014df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 -- 172.21.15.67:0/1776355106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11014e910 con 0x7ff110072250 2024-04-17T07:33:20.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff115000640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff10000d5f0 tx=0x7ff10000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:20.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1000125f0 con 0x7ff110072250 2024-04-17T07:33:20.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff100019070 con 0x7ff110072250 2024-04-17T07:33:20.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1000129d0 con 0x7ff110072250 2024-04-17T07:33:20.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.628+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11014ebf0 con 0x7ff110072250 2024-04-17T07:33:20.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.629+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11014f1b0 con 0x7ff110072250 2024-04-17T07:33:20.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.630+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff110068b90 con 0x7ff110072250 2024-04-17T07:33:20.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.631+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7ff100012b70 con 0x7ff110072250 2024-04-17T07:33:20.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.632+0000 7ff10cc00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 0x7ff0f406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:20.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.632+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7ff10008c050 con 0x7ff110072250 2024-04-17T07:33:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.633+0000 7ff10fe00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 0x7ff0f406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:20.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.634+0000 7ff10fe00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 0x7ff0f406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff104036000 tx=0x7ff104030000 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.637+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff10005a3d0 con 0x7ff110072250 2024-04-17T07:33:20.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.885+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff110165e00 con 0x7ff110072250 2024-04-17T07:33:20.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.886+0000 7ff10cc00640 1 -- 172.21.15.67:0/1776355106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+3869 (secure 0 0 0) 0x7ff100059cd0 con 0x7ff110072250 2024-04-17T07:33:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:20.886 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":63,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/352119453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":352119453},{"type":"v1","addr":"172.21.15.187:6836","nonce":352119453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":6074,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1691986930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1691986930},{"type":"v1","addr":"172.21.15.187:6837","nonce":1691986930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7884,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/528048956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":528048956},{"type":"v1","addr":"172.21.15.67:6833","nonce":528048956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":8292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2676118027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2676118027},{"type":"v1","addr":"172.21.15.67:6835","nonce":2676118027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[]} 2024-04-17T07:33:20.887 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 63 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.888+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 msgr2=0x7ff0f406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.888+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 0x7ff0f406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff104036000 tx=0x7ff104030000 comp rx=0 tx=0).stop 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.888+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 msgr2=0x7ff1100b1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.888+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7ff10000d5f0 tx=0x7ff10000dab0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 shutdown_connections 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff110166440 0x7ff11014df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7ff0f406ca10 0x7ff0f406eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1100c6f90 0x7ff11014da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 --2- 172.21.15.67:0/1776355106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff110072250 0x7ff1100b1020 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:20.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 >> 172.21.15.67:0/1776355106 conn(0x7ff11006bf50 msgr2=0x7ff1100c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:20.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 shutdown_connections 2024-04-17T07:33:20.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:20.889+0000 7ff117a00640 1 -- 172.21.15.67:0/1776355106 wait complete. 2024-04-17T07:33:20.910 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:33:20.910 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:33:20.911 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:33:20.911 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:33:20.911 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:33:20.911 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:33:20.911 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:33:20.911 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:33:20.911 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:33:20.912 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:33:20.912 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:33:20.912 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:33:20.912 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:33:20.912 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:33:20.912 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:33:20.912 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:33:20.929 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:33:20.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:33:20.933 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:33:20.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:33:20.935 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:33:20.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:33:20.935 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:33:20.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:33:20.981 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:33:20.981 DEBUG:teuthology.parallel:result is None 2024-04-17T07:33:20.982 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:33:20.983 DEBUG:teuthology.parallel:result is None 2024-04-17T07:33:20.987 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:33:20.987 DEBUG:teuthology.parallel:result is None 2024-04-17T07:33:20.989 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:33:20.989 DEBUG:teuthology.parallel:result is None 2024-04-17T07:33:20.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:33:21.137 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:33:21.145 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:33:21.146 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:33:21.146 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:33:21.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.169+0000 7f34c3200640 1 Processor -- start 2024-04-17T07:33:21.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.169+0000 7f34c3200640 1 -- start start 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c415ee20 0x7f34c40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34c40d18e0 0x7f34c40d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34c40d4140 con 0x7f34c40d3350 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f34c40b83b0 con 0x7f34c415ee20 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f34c40d1d20 con 0x7f34c40d18e0 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c415ee20 0x7f34c40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34c415ee20 0x7f34c40d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54502/0 (socket says 172.21.15.67:54502) 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c1e00640 1 -- 172.21.15.67:0/2279916955 learned_addr learned my addr 172.21.15.67:0/2279916955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.170+0000 7f34c1400640 1 --2- 172.21.15.67:0/2279916955 >> v2:172.21.15.187:3300/0 conn(0x7f34c40d18e0 0x7f34c40d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.171+0000 7f34c1e00640 1 -- 172.21.15.67:0/2279916955 >> v2:172.21.15.187:3300/0 conn(0x7f34c40d18e0 msgr2=0x7f34c40d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.171+0000 7f34c1e00640 1 --2- 172.21.15.67:0/2279916955 >> v2:172.21.15.187:3300/0 conn(0x7f34c40d18e0 0x7f34c40d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.171+0000 7f34c1e00640 1 -- 172.21.15.67:0/2279916955 >> v1:172.21.15.187:6789/0 conn(0x7f34c40d3350 legacy=0x7f34c4067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.171+0000 7f34c1e00640 1 -- 172.21.15.67:0/2279916955 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c40d3b80 con 0x7f34c415ee20 2024-04-17T07:33:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.172+0000 7f34c1e00640 1 --2- 172.21.15.67:0/2279916955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40d13a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f34c40d5a90 tx=0x7f34b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51453103e616911e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.172+0000 7f34c0a00640 1 -- 172.21.15.67:0/2279916955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4034070 con 0x7f34c415ee20 2024-04-17T07:33:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.172+0000 7f34c0a00640 1 -- 172.21.15.67:0/2279916955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b4030860 con 0x7f34c415ee20 2024-04-17T07:33:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.172+0000 7f34c0a00640 1 -- 172.21.15.67:0/2279916955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4030c30 con 0x7f34c415ee20 2024-04-17T07:33:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 -- 172.21.15.67:0/2279916955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 msgr2=0x7f34c40d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 --2- 172.21.15.67:0/2279916955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40d13a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f34c40d5a90 tx=0x7f34b402b9a0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 -- 172.21.15.67:0/2279916955 shutdown_connections 2024-04-17T07:33:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 --2- 172.21.15.67:0/2279916955 >> v2:172.21.15.187:3300/0 conn(0x7f34c40d18e0 0x7f34c40d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 --2- 172.21.15.67:0/2279916955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40d13a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 -- 172.21.15.67:0/2279916955 >> 172.21.15.67:0/2279916955 conn(0x7f34c406d3c0 msgr2=0x7f34c40b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 -- 172.21.15.67:0/2279916955 shutdown_connections 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.173+0000 7f34c3200640 1 -- 172.21.15.67:0/2279916955 wait complete. 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.174+0000 7f34c3200640 1 Processor -- start 2024-04-17T07:33:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.175+0000 7f34c3200640 1 -- start start 2024-04-17T07:33:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34c40d18e0 0x7f34c40ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34c40d3350 0x7f34c41f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34c415f200 con 0x7f34c415ee20 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34c40b8040 con 0x7f34c40d18e0 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34c40d37c0 con 0x7f34c40d3350 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c1e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34c40d18e0 0x7f34c40ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54508/0 (socket says 172.21.15.67:54508) 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c2800640 1 -- 172.21.15.67:0/1727953073 learned_addr learned my addr 172.21.15.67:0/1727953073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.176+0000 7f34c1400640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34c40d3350 0x7f34c41f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 -- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34c40d3350 msgr2=0x7f34c41f8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34c40d3350 0x7f34c41f8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 -- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34c40d18e0 msgr2=0x7f34c40ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34c40d18e0 0x7f34c40ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 -- 172.21.15.67:0/1727953073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c40cd320 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c2800640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c710 tx=0x7f34bc00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc016020 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34bc006d50 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003bb0 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c40cd600 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.177+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c40c8710 con 0x7f34c415ee20 2024-04-17T07:33:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.179+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c40c8910 con 0x7f34c415ee20 2024-04-17T07:33:21.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.180+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f34bc00a850 con 0x7f34c415ee20 2024-04-17T07:33:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.181+0000 7f34ba000640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 0x7f34a406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.181+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5745+0+0 (secure 0 0 0) 0x7f34bc08b890 con 0x7f34c415ee20 2024-04-17T07:33:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.181+0000 7f34c1e00640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 0x7f34a406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.182+0000 7f34c1e00640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 0x7f34a406eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34b4036000 tx=0x7f34b403b040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.185+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34c40c8910 con 0x7f34c415ee20 2024-04-17T07:33:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:21.393+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f34c4068a10 con 0x7f34c415ee20 2024-04-17T07:33:22.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.462+0000 7f34ba000640 1 -- 172.21.15.67:0/1727953073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v138) ==== 97+0+0 (secure 0 0 0) 0x7f34bc020020 con 0x7f34c415ee20 2024-04-17T07:33:22.462 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.465+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 msgr2=0x7f34a406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 0x7f34a406eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34b4036000 tx=0x7f34b403b040 comp rx=0 tx=0).stop 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 msgr2=0x7f34c40ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c710 tx=0x7f34bc00cbd0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 shutdown_connections 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34c415ee20 0x7f34c40ccab0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f34a406ca10 0x7f34a406eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34c40d3350 0x7f34c41f8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 --2- 172.21.15.67:0/1727953073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34c40d18e0 0x7f34c40ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 >> 172.21.15.67:0/1727953073 conn(0x7f34c406d3c0 msgr2=0x7f34c40b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 shutdown_connections 2024-04-17T07:33:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.466+0000 7f34c3200640 1 -- 172.21.15.67:0/1727953073 wait complete. 2024-04-17T07:33:22.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:33:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.700+0000 7fcfa7e00640 1 Processor -- start 2024-04-17T07:33:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.701+0000 7fcfa7e00640 1 -- start start 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.702+0000 7fcfa7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 0x7fcfa80d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.702+0000 7fcfa7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa815fb20 0x7fcfa80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.702+0000 7fcfa7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa80cf730 con 0x7fcfa815e9a0 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.702+0000 7fcfa7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa815ed80 con 0x7fcfa815fb20 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.702+0000 7fcfa7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa80cfa10 con 0x7fcfa80d04f0 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa815fb20 0x7fcfa80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa815fb20 0x7fcfa80d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54512/0 (socket says 172.21.15.67:54512) 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 -- 172.21.15.67:0/1465658684 learned_addr learned my addr 172.21.15.67:0/1465658684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa6000640 1 --2- 172.21.15.67:0/1465658684 >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 0x7fcfa80d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 -- 172.21.15.67:0/1465658684 >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 msgr2=0x7fcfa80d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 --2- 172.21.15.67:0/1465658684 >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 0x7fcfa80d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 -- 172.21.15.67:0/1465658684 >> v1:172.21.15.187:6789/0 conn(0x7fcfa815e9a0 legacy=0x7fcfa80d4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa7400640 1 -- 172.21.15.67:0/1465658684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa80cfcf0 con 0x7fcfa815fb20 2024-04-17T07:33:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.703+0000 7fcfa6000640 1 --2- 172.21.15.67:0/1465658684 >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 0x7fcfa80d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:22.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.705+0000 7fcfa7400640 1 --2- 172.21.15.67:0/1465658684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa80d1550 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcfa808cfc0 tx=0x7fcfa002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653da1f0cdb0911c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:22.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.705+0000 7fcfa5600640 1 -- 172.21.15.67:0/1465658684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa002cb10 con 0x7fcfa815fb20 2024-04-17T07:33:22.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa5600640 1 -- 172.21.15.67:0/1465658684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa002ccb0 con 0x7fcfa815fb20 2024-04-17T07:33:22.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa5600640 1 -- 172.21.15.67:0/1465658684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa002ce60 con 0x7fcfa815fb20 2024-04-17T07:33:22.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1465658684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 msgr2=0x7fcfa80d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1465658684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa80d1550 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcfa808cfc0 tx=0x7fcfa002bf70 comp rx=0 tx=0).stop 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1465658684 shutdown_connections 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1465658684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa80d1550 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1465658684 >> v2:172.21.15.187:3300/0 conn(0x7fcfa80d04f0 0x7fcfa80d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1465658684 >> 172.21.15.67:0/1465658684 conn(0x7fcfa808bff0 msgr2=0x7fcfa80d4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1465658684 shutdown_connections 2024-04-17T07:33:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.706+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1465658684 wait complete. 2024-04-17T07:33:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.708+0000 7fcfa7e00640 1 Processor -- start 2024-04-17T07:33:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.708+0000 7fcfa7e00640 1 -- start start 2024-04-17T07:33:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfa80d04f0 0x7fcfa8159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfa815e9a0 0x7fcfa80cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80d22f0 con 0x7fcfa815fb20 2024-04-17T07:33:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa80cf3e0 con 0x7fcfa80d04f0 2024-04-17T07:33:22.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.709+0000 7fcfa7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa80d2470 con 0x7fcfa815e9a0 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54522/0 (socket says 172.21.15.67:54522) 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 -- 172.21.15.67:0/1279288599 learned_addr learned my addr 172.21.15.67:0/1279288599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa6000640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfa815e9a0 0x7fcfa80cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa6a00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfa80d04f0 0x7fcfa8159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 -- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfa815e9a0 msgr2=0x7fcfa80cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfa815e9a0 0x7fcfa80cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 -- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfa80d04f0 msgr2=0x7fcfa8159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfa80d04f0 0x7fcfa8159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.710+0000 7fcfa7400640 1 -- 172.21.15.67:0/1279288599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa80c8990 con 0x7fcfa815fb20 2024-04-17T07:33:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcfa7400640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f40 tx=0x7fcfa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:22.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0043020 con 0x7fcfa815fb20 2024-04-17T07:33:22.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa003e450 con 0x7fcfa815fb20 2024-04-17T07:33:22.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa80cbb60 con 0x7fcfa815fb20 2024-04-17T07:33:22.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa002ce60 con 0x7fcfa815fb20 2024-04-17T07:33:22.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.711+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa80cc120 con 0x7fcfa815fb20 2024-04-17T07:33:22.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.712+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa8068a10 con 0x7fcfa815fb20 2024-04-17T07:33:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.714+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fcfa0046020 con 0x7fcfa815fb20 2024-04-17T07:33:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.715+0000 7fcf9ea00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 0x7fcf8406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.715+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4885+0+0 (secure 0 0 0) 0x7fcfa00adfe0 con 0x7fcfa815fb20 2024-04-17T07:33:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.716+0000 7fcfa6a00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 0x7fcf8406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:22.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.717+0000 7fcfa6a00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 0x7fcf8406eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf98001640 tx=0x7fcf98009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:22.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.719+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcfa004d030 con 0x7fcfa815fb20 2024-04-17T07:33:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.988+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcfa80ce600 con 0x7fcfa815fb20 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.989+0000 7fcf9ea00640 1 -- 172.21.15.67:0/1279288599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7fcfa80c8990 con 0x7fcfa815fb20 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:33:22.989 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:33:22.990 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.993 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:33:22.994 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.996 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:33:22.997 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:33:22.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:33:22.999 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:23.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:33:23.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:33:23.002 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 msgr2=0x7fcf8406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 0x7fcf8406eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf98001640 tx=0x7fcf98009040 comp rx=0 tx=0).stop 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 msgr2=0x7fcfa815bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009f40 tx=0x7fcfa0004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 shutdown_connections 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa815fb20 0x7fcfa815bb20 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fcf8406ca10 0x7fcf8406eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcfa815e9a0 0x7fcfa80cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 --2- 172.21.15.67:0/1279288599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcfa80d04f0 0x7fcfa8159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 >> 172.21.15.67:0/1279288599 conn(0x7fcfa808bff0 msgr2=0x7fcfa815c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 shutdown_connections 2024-04-17T07:33:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:22.991+0000 7fcfa7e00640 1 -- 172.21.15.67:0/1279288599 wait complete. 2024-04-17T07:33:23.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.224+0000 7f4389400640 1 Processor -- start 2024-04-17T07:33:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.225+0000 7f4389400640 1 -- start start 2024-04-17T07:33:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.226+0000 7f4389400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f438415ec60 0x7f43840b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.226+0000 7f4389400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43840cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.226+0000 7f4389400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43840d2d30 con 0x7f43840cf910 2024-04-17T07:33:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.226+0000 7f4389400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f43840d49c0 con 0x7f438415ec60 2024-04-17T07:33:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.226+0000 7f4389400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43840d3660 con 0x7f43840b6b30 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4388a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43840cf910 0x7f438406cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42224/0 (socket says 172.21.15.67:42224) 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4388a00640 1 -- 172.21.15.67:0/166538820 learned_addr learned my addr 172.21.15.67:0/166538820 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4383e00640 1 --2- 172.21.15.67:0/166538820 >> v2:172.21.15.187:3300/0 conn(0x7f438415ec60 0x7f43840b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4383400640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43840cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678010304 0 0) 0x7f43840d2d30 con 0x7f43840cf910 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.227+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4364003520 con 0x7f43840cf910 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3476696224 0 0) 0x7f4364003520 con 0x7f43840cf910 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 msgr2=0x7f43840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43840cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 >> v2:172.21.15.187:3300/0 conn(0x7f438415ec60 msgr2=0x7f43840b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 --2- 172.21.15.67:0/166538820 >> v2:172.21.15.187:3300/0 conn(0x7f438415ec60 0x7f43840b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43840d3010 con 0x7f43840cf910 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f437c001670 con 0x7f43840cf910 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 >> v1:172.21.15.67:6789/0 conn(0x7f43840cf910 legacy=0x7f438406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f4364003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4364004800 0x7f4364004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43840d49c0 con 0x7f4364003a20 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43840d3660 con 0x7f4364004800 2024-04-17T07:33:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f436400b9f0 con 0x7f43840b6b30 2024-04-17T07:33:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.228+0000 7f4383e00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f4364003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383e00640 1 -- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 msgr2=0x7f43640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383400640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4364004800 0x7f4364004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4388a00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383e00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b6b30 0x7f43640051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383e00640 1 -- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4364004800 msgr2=0x7f4364004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383e00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4364004800 0x7f4364004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383e00640 1 -- 172.21.15.67:0/166538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4364003520 con 0x7f4364003a20 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.229+0000 7f4383400640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4364004800 0x7f4364004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.230+0000 7f4383e00640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f4364003e40 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4374009ac0 tx=0x7f4364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24ee321690a9046 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.230+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374039070 con 0x7f4364003a20 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.230+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437402c710 con 0x7f4364003a20 2024-04-17T07:33:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.230+0000 7f4382a00640 1 -- 172.21.15.67:0/166538820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374009690 con 0x7f4364003a20 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 -- 172.21.15.67:0/166538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 msgr2=0x7f4364003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 --2- 172.21.15.67:0/166538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f4364003e40 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f4374009ac0 tx=0x7f4364005830 comp rx=0 tx=0).stop 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4388a00640 1 -- 172.21.15.67:0/166538820 reap_dead start 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 -- 172.21.15.67:0/166538820 shutdown_connections 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 -- 172.21.15.67:0/166538820 >> 172.21.15.67:0/166538820 conn(0x7f43840d15f0 msgr2=0x7f43840d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:23.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 -- 172.21.15.67:0/166538820 shutdown_connections 2024-04-17T07:33:23.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.231+0000 7f4389400640 1 -- 172.21.15.67:0/166538820 wait complete. 2024-04-17T07:33:23.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.233+0000 7f4389400640 1 Processor -- start 2024-04-17T07:33:23.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.233+0000 7f4389400640 1 -- start start 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840ccc90 0x7f43840ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b5c20 0x7f43840b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43840cfda0 con 0x7f4364003a20 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43840b6f70 con 0x7f43840ccc90 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4389400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f438415f040 con 0x7f43840b5c20 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.234+0000 7f4383e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54546/0 (socket says 172.21.15.67:54546) 2024-04-17T07:33:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 -- 172.21.15.67:0/457604640 learned_addr learned my addr 172.21.15.67:0/457604640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4388a00640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b5c20 0x7f43840b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840ccc90 0x7f43840ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 -- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b5c20 msgr2=0x7f43840b60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b5c20 0x7f43840b60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 -- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840ccc90 msgr2=0x7f43840ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840ccc90 0x7f43840ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 -- 172.21.15.67:0/457604640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43840cdbc0 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.235+0000 7f4383e00640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4374009be0 tx=0x7f4374004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.236+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374039050 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.236+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437403a040 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.236+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374005bd0 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.236+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43840cd320 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.236+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43840cd8e0 con 0x7f4364003a20 2024-04-17T07:33:23.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.237+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4384068a10 con 0x7f4364003a20 2024-04-17T07:33:23.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.238+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f4374047020 con 0x7f4364003a20 2024-04-17T07:33:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.239+0000 7f437be00640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 0x7f436006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.239+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4885+0+0 (secure 0 0 0) 0x7f43740ad320 con 0x7f4364003a20 2024-04-17T07:33:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.240+0000 7f4383400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 0x7f436006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.241+0000 7f4383400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 0x7f436006ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43840ce6a0 tx=0x7f436c00d040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.243+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f437407b9f0 con 0x7f4364003a20 2024-04-17T07:33:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.450+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43840cd070 con 0x7f4364003a20 2024-04-17T07:33:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.452+0000 7f437be00640 1 -- 172.21.15.67:0/457604640 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+12897 (secure 0 0 0) 0x7f437407b2f0 con 0x7f4364003a20 2024-04-17T07:33:23.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:23.453 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":138,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:22.459438+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 msgr2=0x7f436006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 0x7f436006ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f43840ce6a0 tx=0x7f436c00d040 comp rx=0 tx=0).stop 2024-04-17T07:33:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 msgr2=0x7f438414db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f4374009be0 tx=0x7f4374004730 comp rx=0 tx=0).stop 2024-04-17T07:33:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 shutdown_connections 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f436006ca40 0x7f436006ef00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43840b5c20 0x7f43840b60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43840ccc90 0x7f43840ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 --2- 172.21.15.67:0/457604640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4364003a20 0x7f438414db50 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 >> 172.21.15.67:0/457604640 conn(0x7f43840d15f0 msgr2=0x7f43840b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 shutdown_connections 2024-04-17T07:33:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.455+0000 7f4389400640 1 -- 172.21.15.67:0/457604640 wait complete. 2024-04-17T07:33:23.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:23.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:23.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.702+0000 7f389f200640 1 Processor -- start 2024-04-17T07:33:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.703+0000 7f389f200640 1 -- start start 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.704+0000 7f389f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38a0145b00 0x7f38a0160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.704+0000 7f389f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a00b4ac0 0x7f38a00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.704+0000 7f389f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38a0160920 con 0x7f38a0146060 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.704+0000 7f389f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f38a00b5450 con 0x7f38a0145b00 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.704+0000 7f389f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38a0160c00 con 0x7f38a00b4ac0 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38a0145b00 0x7f38a0160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38a0145b00 0x7f38a0160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56596/0 (socket says 172.21.15.67:56596) 2024-04-17T07:33:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 -- 172.21.15.67:0/4007768227 learned_addr learned my addr 172.21.15.67:0/4007768227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389e800640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a00b4ac0 0x7f38a00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 -- 172.21.15.67:0/4007768227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a00b4ac0 msgr2=0x7f38a00b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a00b4ac0 0x7f38a00b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 -- 172.21.15.67:0/4007768227 >> v1:172.21.15.187:6789/0 conn(0x7f38a0146060 legacy=0x7f38a015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.705+0000 7f389de00640 1 -- 172.21.15.67:0/4007768227 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a0160ee0 con 0x7f38a0145b00 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.706+0000 7f389de00640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a0145b00 0x7f38a0160340 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f389002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b428e210ff82d1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.707+0000 7f389d400640 1 -- 172.21.15.67:0/4007768227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38900317c0 con 0x7f38a0145b00 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.707+0000 7f389d400640 1 -- 172.21.15.67:0/4007768227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3890037070 con 0x7f38a0145b00 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.707+0000 7f389d400640 1 -- 172.21.15.67:0/4007768227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890031b80 con 0x7f38a0145b00 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 -- 172.21.15.67:0/4007768227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a0145b00 msgr2=0x7f38a0160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a0145b00 0x7f38a0160340 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f389002b860 comp rx=0 tx=0).stop 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 -- 172.21.15.67:0/4007768227 shutdown_connections 2024-04-17T07:33:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a00b4ac0 0x7f38a00b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 --2- 172.21.15.67:0/4007768227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a0145b00 0x7f38a0160340 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 -- 172.21.15.67:0/4007768227 >> 172.21.15.67:0/4007768227 conn(0x7f38a00b23c0 msgr2=0x7f38a00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 -- 172.21.15.67:0/4007768227 shutdown_connections 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.708+0000 7f389f200640 1 -- 172.21.15.67:0/4007768227 wait complete. 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.709+0000 7f389f200640 1 Processor -- start 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.710+0000 7f389f200640 1 -- start start 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 0x7f38a013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38a0146060 0x7f38a013c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38a00cfba0 con 0x7f38a00b4ac0 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38a00b3760 con 0x7f38a0146060 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38a00b3c80 con 0x7f38a0145b00 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56608/0 (socket says 172.21.15.67:56608) 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 -- 172.21.15.67:0/4056072183 learned_addr learned my addr 172.21.15.67:0/4056072183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f3897e00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 0x7f38a013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 -- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 msgr2=0x7f38a013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 0x7f38a013d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 -- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38a0146060 msgr2=0x7f38a013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38a0146060 0x7f38a013c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f389de00640 1 -- 172.21.15.67:0/4056072183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a0139d20 con 0x7f38a00b4ac0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.711+0000 7f3897e00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 0x7f38a013d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f389de00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f38900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890032650 con 0x7f38a00b4ac0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38900327f0 con 0x7f38a00b4ac0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a013a000 con 0x7f38a00b4ac0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890032be0 con 0x7f38a00b4ac0 2024-04-17T07:33:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.712+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a013a5c0 con 0x7f38a00b4ac0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.713+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a0068a10 con 0x7f38a00b4ac0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.714+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f389004c060 con 0x7f38a00b4ac0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.715+0000 7f3896000640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 0x7f387c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.715+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4885+0+0 (secure 0 0 0) 0x7f38900ad5d0 con 0x7f38a00b4ac0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.716+0000 7f3897e00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 0x7f387c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.717+0000 7f3897e00640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 0x7f387c06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38a01397e0 tx=0x7f3888008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.721+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3890040070 con 0x7f38a00b4ac0 2024-04-17T07:33:23.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.929+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38a0068610 con 0x7f38a00b4ac0 2024-04-17T07:33:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.930+0000 7f3896000640 1 -- 172.21.15.67:0/4056072183 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) ==== 93+0+10 (secure 0 0 0) 0x7f38900ad030 con 0x7f38a00b4ac0 2024-04-17T07:33:23.930 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 msgr2=0x7f387c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 0x7f387c06eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38a01397e0 tx=0x7f3888008040 comp rx=0 tx=0).stop 2024-04-17T07:33:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 msgr2=0x7f38a015bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f38900046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 shutdown_connections 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38a0146060 0x7f38a013c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f387c06ca10 0x7f387c06eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38a0145b00 0x7f38a013d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.932+0000 7f389f200640 1 --2- 172.21.15.67:0/4056072183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38a00b4ac0 0x7f38a015bec0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.933+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 >> 172.21.15.67:0/4056072183 conn(0x7f38a00b23c0 msgr2=0x7f38a0145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.933+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 shutdown_connections 2024-04-17T07:33:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:23.933+0000 7f389f200640 1 -- 172.21.15.67:0/4056072183 wait complete. 2024-04-17T07:33:23.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:33:24.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.172+0000 7efd3a000640 1 Processor -- start 2024-04-17T07:33:24.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.172+0000 7efd3a000640 1 -- start start 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd3a000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efd3406e320 0x7efd3409b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd3a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd340d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd3a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3409bd80 con 0x7efd340b3180 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd3a000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efd3406c810 con 0x7efd3406e320 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd3a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd340cdeb0 con 0x7efd3409b170 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd33e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efd340b3180 0x7efd3409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56684/0 (socket says 172.21.15.67:56684) 2024-04-17T07:33:24.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd33e00640 1 -- 172.21.15.67:0/684588164 learned_addr learned my addr 172.21.15.67:0/684588164 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:24.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd38c00640 1 --2- 172.21.15.67:0/684588164 >> v2:172.21.15.187:3300/0 conn(0x7efd3406e320 0x7efd3409b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd39600640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd340d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342454113 0 0) 0x7efd3409bd80 con 0x7efd340b3180 2024-04-17T07:33:24.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.174+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd18003520 con 0x7efd340b3180 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2485732272 0 0) 0x7efd18003520 con 0x7efd340b3180 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 msgr2=0x7efd340d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd340d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 >> v2:172.21.15.187:3300/0 conn(0x7efd3406e320 msgr2=0x7efd3409b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 --2- 172.21.15.67:0/684588164 >> v2:172.21.15.187:3300/0 conn(0x7efd3406e320 0x7efd3409b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd340ce190 con 0x7efd340b3180 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7efd1c001670 con 0x7efd340b3180 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 >> v1:172.21.15.67:6789/0 conn(0x7efd340b3180 legacy=0x7efd3409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18003a20 0x7efd18003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd18004800 0x7efd18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efd3406c810 con 0x7efd18004800 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efd340cdeb0 con 0x7efd18003a20 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd1800baa0 con 0x7efd3409b170 2024-04-17T07:33:24.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.175+0000 7efd33e00640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd18004800 0x7efd18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd38c00640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18003a20 0x7efd18003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd39600640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd33e00640 1 -- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 msgr2=0x7efd180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd33e00640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd33e00640 1 -- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18003a20 msgr2=0x7efd18003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd33e00640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18003a20 0x7efd18003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd33e00640 1 -- 172.21.15.67:0/684588164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd18003520 con 0x7efd18004800 2024-04-17T07:33:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.176+0000 7efd39600640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd3409b170 0x7efd180051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.177+0000 7efd33e00640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd18004800 0x7efd18004c80 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001eb0 tx=0x7efd18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e33ffa876e5919 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.177+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c007ca0 con 0x7efd18004800 2024-04-17T07:33:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.177+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c030070 con 0x7efd18004800 2024-04-17T07:33:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.177+0000 7efd33400640 1 -- 172.21.15.67:0/684588164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c005d10 con 0x7efd18004800 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 -- 172.21.15.67:0/684588164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd18004800 msgr2=0x7efd18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 --2- 172.21.15.67:0/684588164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd18004800 0x7efd18004c80 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001eb0 tx=0x7efd18005830 comp rx=0 tx=0).stop 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd39600640 1 -- 172.21.15.67:0/684588164 reap_dead start 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 -- 172.21.15.67:0/684588164 shutdown_connections 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 -- 172.21.15.67:0/684588164 >> 172.21.15.67:0/684588164 conn(0x7efd340b4170 msgr2=0x7efd340b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 -- 172.21.15.67:0/684588164 shutdown_connections 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.178+0000 7efd3a000640 1 -- 172.21.15.67:0/684588164 wait complete. 2024-04-17T07:33:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.180+0000 7efd3a000640 1 Processor -- start 2024-04-17T07:33:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.180+0000 7efd3a000640 1 -- start start 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18004800 0x7efd340b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 0x7efd34071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efd340b3a00 con 0x7efd340b0650 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efd340b3880 con 0x7efd18004800 2024-04-17T07:33:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.181+0000 7efd3a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd340d0630 con 0x7efd340b7cb0 2024-04-17T07:33:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56624/0 (socket says 172.21.15.67:56624) 2024-04-17T07:33:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 -- 172.21.15.67:0/1447560801 learned_addr learned my addr 172.21.15.67:0/1447560801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd38c00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18004800 0x7efd340b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd39600640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 0x7efd34071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 -- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 msgr2=0x7efd34071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 0x7efd34071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 -- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18004800 msgr2=0x7efd340b6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18004800 0x7efd340b6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd33e00640 1 -- 172.21.15.67:0/1447560801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd340b0a90 con 0x7efd340b0650 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.182+0000 7efd39600640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 0x7efd34071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd33e00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000fb0 tx=0x7efd1c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c02cbf0 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c02cd90 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c0072b0 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd340b22e0 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.183+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd340b6ef0 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.184+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd34068a10 con 0x7efd340b0650 2024-04-17T07:33:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.185+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7efd1c007450 con 0x7efd340b0650 2024-04-17T07:33:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.186+0000 7efd30c00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 0x7efd1006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.186+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4885+0+0 (secure 0 0 0) 0x7efd1c0acfa0 con 0x7efd340b0650 2024-04-17T07:33:24.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.187+0000 7efd38c00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 0x7efd1006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.188+0000 7efd38c00640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 0x7efd1006ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd24009a40 tx=0x7efd24008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.191+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efd1c07b4c0 con 0x7efd340b0650 2024-04-17T07:33:24.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.395+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7efd340cf660 con 0x7efd340b0650 2024-04-17T07:33:24.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.396+0000 7efd30c00640 1 -- 172.21.15.67:0/1447560801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) ==== 84+0+65 (secure 0 0 0) 0x7efd1c07adc0 con 0x7efd340b0650 2024-04-17T07:33:24.396 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:24.397 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:33:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 msgr2=0x7efd1006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 0x7efd1006ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd24009a40 tx=0x7efd24008040 comp rx=0 tx=0).stop 2024-04-17T07:33:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 msgr2=0x7efd34154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7efd1c000fb0 tx=0x7efd1c000fe0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 shutdown_connections 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7efd1006ca40 0x7efd1006ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd340b7cb0 0x7efd34071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd340b0650 0x7efd34154ab0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 --2- 172.21.15.67:0/1447560801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd18004800 0x7efd340b6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 >> 172.21.15.67:0/1447560801 conn(0x7efd340b4170 msgr2=0x7efd340cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 shutdown_connections 2024-04-17T07:33:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.399+0000 7efd3a000640 1 -- 172.21.15.67:0/1447560801 wait complete. 2024-04-17T07:33:24.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:33:24.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:33:24.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.642+0000 7f3561800640 1 Processor -- start 2024-04-17T07:33:24.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.642+0000 7f3561800640 1 -- start start 2024-04-17T07:33:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f3561800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0d0330 0x7f355c0d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f3561800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f355c0cf170 0x7f355c0cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f3561800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355c0d0df0 con 0x7f355c15c950 2024-04-17T07:33:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f3561800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f355c0d13b0 con 0x7f355c0cf170 2024-04-17T07:33:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f3561800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f355c0d10d0 con 0x7f355c0d0330 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f355b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f355c0cf170 0x7f355c0cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f355b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f355c0cf170 0x7f355c0cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56626/0 (socket says 172.21.15.67:56626) 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.644+0000 7f355b400640 1 -- 172.21.15.67:0/2301894899 learned_addr learned my addr 172.21.15.67:0/2301894899 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.645+0000 7f355a000640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0d0330 0x7f355c0d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.645+0000 7f355b400640 1 -- 172.21.15.67:0/2301894899 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0d0330 msgr2=0x7f355c0d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.645+0000 7f355b400640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0d0330 0x7f355c0d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.645+0000 7f355b400640 1 -- 172.21.15.67:0/2301894899 >> v1:172.21.15.187:6789/0 conn(0x7f355c15c950 legacy=0x7f355c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:24.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.645+0000 7f355b400640 1 -- 172.21.15.67:0/2301894899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c0c79b0 con 0x7f355c0cf170 2024-04-17T07:33:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.646+0000 7f355b400640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c0cf170 0x7f355c0cf600 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f3554009e50 tx=0x7f355402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68724edeafa45cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3559600640 1 -- 172.21.15.67:0/2301894899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355402ca70 con 0x7f355c0cf170 2024-04-17T07:33:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3559600640 1 -- 172.21.15.67:0/2301894899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355402cc10 con 0x7f355c0cf170 2024-04-17T07:33:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3559600640 1 -- 172.21.15.67:0/2301894899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540343f0 con 0x7f355c0cf170 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 -- 172.21.15.67:0/2301894899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c0cf170 msgr2=0x7f355c0cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c0cf170 0x7f355c0cf600 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f3554009e50 tx=0x7f355402bed0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 -- 172.21.15.67:0/2301894899 shutdown_connections 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c0cf170 0x7f355c0cf600 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 --2- 172.21.15.67:0/2301894899 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0d0330 0x7f355c0d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 -- 172.21.15.67:0/2301894899 >> 172.21.15.67:0/2301894899 conn(0x7f355c0b4390 msgr2=0x7f355c0b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 -- 172.21.15.67:0/2301894899 shutdown_connections 2024-04-17T07:33:24.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.647+0000 7f3561800640 1 -- 172.21.15.67:0/2301894899 wait complete. 2024-04-17T07:33:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.649+0000 7f3561800640 1 Processor -- start 2024-04-17T07:33:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.649+0000 7f3561800640 1 -- start start 2024-04-17T07:33:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0cf170 0x7f355c1f8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f355c0d0330 0x7f355c151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f355c0c9250 con 0x7f355c15c950 2024-04-17T07:33:24.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f355c15cd30 con 0x7f355c0d0330 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.650+0000 7f3561800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f355c0c81b0 con 0x7f355c0cf170 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56634/0 (socket says 172.21.15.67:56634) 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 -- 172.21.15.67:0/610672498 learned_addr learned my addr 172.21.15.67:0/610672498 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355a000640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f355c0d0330 0x7f355c151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355aa00640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0cf170 0x7f355c1f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 -- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0cf170 msgr2=0x7f355c1f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0cf170 0x7f355c1f8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 -- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f355c0d0330 msgr2=0x7f355c151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f355c0d0330 0x7f355c151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.651+0000 7f355b400640 1 -- 172.21.15.67:0/610672498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c15ee30 con 0x7f355c15c950 2024-04-17T07:33:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f355b400640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f3554009e50 tx=0x7f3554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554043020 con 0x7f355c15c950 2024-04-17T07:33:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355403e450 con 0x7f355c15c950 2024-04-17T07:33:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554009510 con 0x7f355c15c950 2024-04-17T07:33:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355c152520 con 0x7f355c15c950 2024-04-17T07:33:24.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.652+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355c152ae0 con 0x7f355c15c950 2024-04-17T07:33:24.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.653+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355c071730 con 0x7f355c15c950 2024-04-17T07:33:24.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.654+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f3554046020 con 0x7f355c15c950 2024-04-17T07:33:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.655+0000 7f3552a00640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.655+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4885+0+0 (secure 0 0 0) 0x7f35540ae250 con 0x7f355c15c950 2024-04-17T07:33:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.656+0000 7f355aa00640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.657+0000 7f355aa00640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 0x7f353c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f354c006fd0 tx=0x7f354c008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.660+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f355404d030 con 0x7f355c15c950 2024-04-17T07:33:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:24.867+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f355c068d60 con 0x7f355c15c950 2024-04-17T07:33:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.479+0000 7f3552a00640 1 -- 172.21.15.67:0/610672498 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v139) ==== 138+0+0 (secure 0 0 0) 0x7f355407c890 con 0x7f355c15c950 2024-04-17T07:33:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:33:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.482+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 msgr2=0x7f353c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.482+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 0x7f353c06eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f354c006fd0 tx=0x7f354c008040 comp rx=0 tx=0).stop 2024-04-17T07:33:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.482+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 msgr2=0x7f355c15e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.482+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f3554009e50 tx=0x7f3554004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 shutdown_connections 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355c15c950 0x7f355c15e8f0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f355c0d0330 0x7f355c151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 --2- 172.21.15.67:0/610672498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f355c0cf170 0x7f355c1f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 >> 172.21.15.67:0/610672498 conn(0x7f355c0b4390 msgr2=0x7f355c14efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 shutdown_connections 2024-04-17T07:33:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.483+0000 7f3561800640 1 -- 172.21.15.67:0/610672498 wait complete. 2024-04-17T07:33:25.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:33:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.717+0000 7f836e200640 1 Processor -- start 2024-04-17T07:33:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.717+0000 7f836e200640 1 -- start start 2024-04-17T07:33:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f836e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83680b27c0 0x7f8368142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f836e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f836815cbe0 con 0x7f8368071d00 2024-04-17T07:33:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f836e200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f836815cea0 con 0x7f836813f6d0 2024-04-17T07:33:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f836e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8368072190 con 0x7f83680b27c0 2024-04-17T07:33:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f836d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83680b27c0 0x7f8368142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f8367e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8368071d00 0x7f83680b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56686/0 (socket says 172.21.15.67:56686) 2024-04-17T07:33:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.719+0000 7f8367e00640 1 -- 172.21.15.67:0/3487596487 learned_addr learned my addr 172.21.15.67:0/3487596487 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.720+0000 7f836d800640 1 -- 172.21.15.67:0/3487596487 >> v1:172.21.15.187:6789/0 conn(0x7f836813f6d0 legacy=0x7f83680cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.720+0000 7f836d800640 1 -- 172.21.15.67:0/3487596487 >> v1:172.21.15.67:6789/0 conn(0x7f8368071d00 legacy=0x7f83680b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.720+0000 7f836d800640 1 -- 172.21.15.67:0/3487596487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836815d180 con 0x7f83680b27c0 2024-04-17T07:33:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.721+0000 7f836d800640 1 --2- 172.21.15.67:0/3487596487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83680b27c0 0x7f8368142020 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f836002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3025776d16f20131 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.721+0000 7f8367400640 1 -- 172.21.15.67:0/3487596487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360039070 con 0x7f83680b27c0 2024-04-17T07:33:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.721+0000 7f8367400640 1 -- 172.21.15.67:0/3487596487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836002cc60 con 0x7f83680b27c0 2024-04-17T07:33:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.721+0000 7f8367400640 1 -- 172.21.15.67:0/3487596487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360034660 con 0x7f83680b27c0 2024-04-17T07:33:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 -- 172.21.15.67:0/3487596487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83680b27c0 msgr2=0x7f8368142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 --2- 172.21.15.67:0/3487596487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83680b27c0 0x7f8368142020 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f836002beb0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 -- 172.21.15.67:0/3487596487 shutdown_connections 2024-04-17T07:33:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 --2- 172.21.15.67:0/3487596487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83680b27c0 0x7f8368142020 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 -- 172.21.15.67:0/3487596487 >> 172.21.15.67:0/3487596487 conn(0x7f8368067d10 msgr2=0x7f83680d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 -- 172.21.15.67:0/3487596487 shutdown_connections 2024-04-17T07:33:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.722+0000 7f836e200640 1 -- 172.21.15.67:0/3487596487 wait complete. 2024-04-17T07:33:25.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.724+0000 7f836e200640 1 Processor -- start 2024-04-17T07:33:25.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.724+0000 7f836e200640 1 -- start start 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8368071d00 0x7f83681f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83680b27c0 0x7f83681564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83680d1120 con 0x7f836813f6d0 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f836815d740 con 0x7f83680b27c0 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f836815da20 con 0x7f8368071d00 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56654/0 (socket says 172.21.15.67:56654) 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836d800640 1 -- 172.21.15.67:0/4201903423 learned_addr learned my addr 172.21.15.67:0/4201903423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.725+0000 7f836ce00640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8368071d00 0x7f83681f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f8367e00640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83680b27c0 0x7f83681564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 -- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8368071d00 msgr2=0x7f83681f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8368071d00 0x7f83681f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 -- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83680b27c0 msgr2=0x7f83681564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83680b27c0 0x7f83681564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 -- 172.21.15.67:0/4201903423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836815c080 con 0x7f836813f6d0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f836d800640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f8360004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.726+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360039050 con 0x7f836813f6d0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.727+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836003a040 con 0x7f836813f6d0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.727+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836003e610 con 0x7f836813f6d0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.727+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836815c360 con 0x7f836813f6d0 2024-04-17T07:33:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.727+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8368157910 con 0x7f836813f6d0 2024-04-17T07:33:25.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.728+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8368068a10 con 0x7f836813f6d0 2024-04-17T07:33:25.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.730+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f8360047020 con 0x7f836813f6d0 2024-04-17T07:33:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.731+0000 7f8364c00640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 0x7f834806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:25.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.731+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5255+0+0 (secure 0 0 0) 0x7f83600ae930 con 0x7f836813f6d0 2024-04-17T07:33:25.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.732+0000 7f836ce00640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 0x7f834806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:25.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.733+0000 7f836ce00640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 0x7f834806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f835c006fd0 tx=0x7f835c008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:25.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.736+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f836004e020 con 0x7f836813f6d0 2024-04-17T07:33:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:25.943+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8368141ac0 con 0x7f836813f6d0 2024-04-17T07:33:26.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.485+0000 7f8364c00640 1 -- 172.21.15.67:0/4201903423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v140) ==== 170+0+0 (secure 0 0 0) 0x7f836007ceb0 con 0x7f836813f6d0 2024-04-17T07:33:26.486 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:33:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.490+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 msgr2=0x7f834806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.490+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 0x7f834806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f835c006fd0 tx=0x7f835c008040 comp rx=0 tx=0).stop 2024-04-17T07:33:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.490+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 msgr2=0x7f83681592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.490+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f8360004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 shutdown_connections 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836813f6d0 0x7f83681592e0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83680b27c0 0x7f83681564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f834806ca10 0x7f834806eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 --2- 172.21.15.67:0/4201903423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8368071d00 0x7f83681f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.492+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 >> 172.21.15.67:0/4201903423 conn(0x7f8368067d10 msgr2=0x7f83681608e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.493+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 shutdown_connections 2024-04-17T07:33:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.493+0000 7f836e200640 1 -- 172.21.15.67:0/4201903423 wait complete. 2024-04-17T07:33:26.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:33:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.716+0000 7f7116c00640 1 Processor -- start 2024-04-17T07:33:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.717+0000 7f7116c00640 1 -- start start 2024-04-17T07:33:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7116c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71100b4fd0 0x7f71100cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7116c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71100d00b0 con 0x7f71100d33e0 2024-04-17T07:33:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7116c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71100d0390 con 0x7f711015ecb0 2024-04-17T07:33:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7116c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f71100d0c30 con 0x7f71100b4fd0 2024-04-17T07:33:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7114e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f71100d33e0 0x7f71100cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56698/0 (socket says 172.21.15.67:56698) 2024-04-17T07:33:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.718+0000 7f7114e00640 1 -- 172.21.15.67:0/1384377221 learned_addr learned my addr 172.21.15.67:0/1384377221 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f7116200640 1 --2- 172.21.15.67:0/1384377221 >> v2:172.21.15.67:3300/0 conn(0x7f71100b4fd0 0x7f71100cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f710fe00640 1 -- 172.21.15.67:0/1384377221 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180753504 0 0) 0x7f71100d00b0 con 0x7f71100d33e0 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f710fe00640 1 -- 172.21.15.67:0/1384377221 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70f8003520 con 0x7f71100d33e0 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f7116200640 1 -- 172.21.15.67:0/1384377221 >> v1:172.21.15.187:6789/0 conn(0x7f711015ecb0 legacy=0x7f71100d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f7116200640 1 -- 172.21.15.67:0/1384377221 >> v1:172.21.15.67:6789/0 conn(0x7f71100d33e0 legacy=0x7f71100cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.719+0000 7f7116200640 1 -- 172.21.15.67:0/1384377221 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71100d0670 con 0x7f71100b4fd0 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.720+0000 7f7116200640 1 --2- 172.21.15.67:0/1384377221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71100cfab0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f71100d0db0 tx=0x7f710802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8bf4d31cc0235f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.720+0000 7f710fe00640 1 -- 172.21.15.67:0/1384377221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108035020 con 0x7f71100b4fd0 2024-04-17T07:33:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f710fe00640 1 -- 172.21.15.67:0/1384377221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7108031810 con 0x7f71100b4fd0 2024-04-17T07:33:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f710fe00640 1 -- 172.21.15.67:0/1384377221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108031bc0 con 0x7f71100b4fd0 2024-04-17T07:33:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 -- 172.21.15.67:0/1384377221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 msgr2=0x7f71100cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 --2- 172.21.15.67:0/1384377221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71100cfab0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f71100d0db0 tx=0x7f710802b8b0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 -- 172.21.15.67:0/1384377221 shutdown_connections 2024-04-17T07:33:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 --2- 172.21.15.67:0/1384377221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71100cfab0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 -- 172.21.15.67:0/1384377221 >> 172.21.15.67:0/1384377221 conn(0x7f711008bff0 msgr2=0x7f71100d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 -- 172.21.15.67:0/1384377221 shutdown_connections 2024-04-17T07:33:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.721+0000 7f7116c00640 1 -- 172.21.15.67:0/1384377221 wait complete. 2024-04-17T07:33:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.723+0000 7f7116c00640 1 Processor -- start 2024-04-17T07:33:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.723+0000 7f7116c00640 1 -- start start 2024-04-17T07:33:26.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71100d33e0 0x7f71100cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 0x7f7110156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f711015f210 con 0x7f71100b4fd0 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f711015f090 con 0x7f711015ecb0 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.724+0000 7f7116c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71101698f0 con 0x7f71100d33e0 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56666/0 (socket says 172.21.15.67:56666) 2024-04-17T07:33:26.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 -- 172.21.15.67:0/3392031577 learned_addr learned my addr 172.21.15.67:0/3392031577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:26.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7116200640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 0x7f7110156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:26.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7114e00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71100d33e0 0x7f71100cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 -- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71100d33e0 msgr2=0x7f71100cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71100d33e0 0x7f71100cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 -- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 msgr2=0x7f7110156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 0x7f7110156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 -- 172.21.15.67:0/3392031577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71100ce6a0 con 0x7f71100b4fd0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7116200640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 0x7f7110156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.725+0000 7f7115800640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f710400c730 tx=0x7f710400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.726+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104003b70 con 0x7f71100b4fd0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.726+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7104003d30 con 0x7f71100b4fd0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.726+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710400aa90 con 0x7f71100b4fd0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.726+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71100ce3e0 con 0x7f71100b4fd0 2024-04-17T07:33:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.726+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71101547f0 con 0x7f71100b4fd0 2024-04-17T07:33:26.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.727+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7110068a10 con 0x7f71100b4fd0 2024-04-17T07:33:26.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.728+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f710400ac30 con 0x7f71100b4fd0 2024-04-17T07:33:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.730+0000 7f710d600640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 0x7f70f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.730+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5637+0+0 (secure 0 0 0) 0x7f710408bbf0 con 0x7f71100b4fd0 2024-04-17T07:33:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.731+0000 7f7114e00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 0x7f70f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.732+0000 7f7114e00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 0x7f70f006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71101f7cb0 tx=0x7f70fc009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.735+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7104059f50 con 0x7f71100b4fd0 2024-04-17T07:33:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:26.986+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f711006d880 con 0x7f71100b4fd0 2024-04-17T07:33:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.557+0000 7f710d600640 1 -- 172.21.15.67:0/3392031577 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v65) ==== 174+0+0 (secure 0 0 0) 0x7f7104059850 con 0x7f71100b4fd0 2024-04-17T07:33:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 16 and data pool 17 2024-04-17T07:33:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 msgr2=0x7f70f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 0x7f70f006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f71101f7cb0 tx=0x7f70fc009040 comp rx=0 tx=0).stop 2024-04-17T07:33:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 msgr2=0x7f71101552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f710400c730 tx=0x7f710400cbf0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 shutdown_connections 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f711015ecb0 0x7f7110156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f70f006ca10 0x7f70f006eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71100d33e0 0x7f71100cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 --2- 172.21.15.67:0/3392031577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71100b4fd0 0x7f71101552d0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.564+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 >> 172.21.15.67:0/3392031577 conn(0x7f711008bff0 msgr2=0x7f7110159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.565+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 shutdown_connections 2024-04-17T07:33:27.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.566+0000 7f7116c00640 1 -- 172.21.15.67:0/3392031577 wait complete. 2024-04-17T07:33:27.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:27.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.780+0000 7f7f8c600640 1 Processor -- start 2024-04-17T07:33:27.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.780+0000 7f7f8c600640 1 -- start start 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f8c600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7f840cf640 0x7f7f840cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f8c600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 0x7f7f840b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f8c600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f840cff60 con 0x7f7f8415ed10 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f8c600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7f840d3c30 con 0x7f7f840cf640 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f8c600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f840d3020 con 0x7f7f840d2ab0 2024-04-17T07:33:27.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f89200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7f8415ed10 0x7f7f840d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56708/0 (socket says 172.21.15.67:56708) 2024-04-17T07:33:27.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.782+0000 7f7f89200640 1 -- 172.21.15.67:0/3002159221 learned_addr learned my addr 172.21.15.67:0/3002159221 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:27.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f8a600640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 0x7f7f840b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f89c00640 1 --2- 172.21.15.67:0/3002159221 >> v2:172.21.15.187:3300/0 conn(0x7f7f840cf640 0x7f7f840cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831714123 0 0) 0x7f7f840cff60 con 0x7f7f8415ed10 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f6c003520 con 0x7f7f8415ed10 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1499289517 0 0) 0x7f7f6c003520 con 0x7f7f8415ed10 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 msgr2=0x7f7f840b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 0x7f7f840b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 >> v2:172.21.15.187:3300/0 conn(0x7f7f840cf640 msgr2=0x7f7f840cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 --2- 172.21.15.67:0/3002159221 >> v2:172.21.15.187:3300/0 conn(0x7f7f840cf640 0x7f7f840cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f840d3300 con 0x7f7f8415ed10 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7f70001670 con 0x7f7f8415ed10 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 >> v1:172.21.15.67:6789/0 conn(0x7f7f8415ed10 legacy=0x7f7f840d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6c0039b0 0x7f7f6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f6c004800 0x7f7f6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 0x7f7f6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f840d3c30 con 0x7f7f6c0039b0 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7f840d3020 con 0x7f7f6c004800 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.783+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f6c00b9c0 con 0x7f7f840d2ab0 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6c0039b0 0x7f7f6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89200640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f6c004800 0x7f7f6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 -- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 msgr2=0x7f7f6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f840d2ab0 0x7f7f6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 -- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f6c004800 msgr2=0x7f7f6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f6c004800 0x7f7f6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.784+0000 7f7f89c00640 1 -- 172.21.15.67:0/3002159221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6c003520 con 0x7f7f6c0039b0 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.785+0000 7f7f89c00640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6c0039b0 0x7f7f6c003dd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009ac0 tx=0x7f7f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e5c34cbd5c495a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.785+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78039070 con 0x7f7f6c0039b0 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f7802c710 con 0x7f7f6c0039b0 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f83e00640 1 -- 172.21.15.67:0/3002159221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78009690 con 0x7f7f6c0039b0 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 -- 172.21.15.67:0/3002159221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6c0039b0 msgr2=0x7f7f6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 --2- 172.21.15.67:0/3002159221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6c0039b0 0x7f7f6c003dd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009ac0 tx=0x7f7f6c005830 comp rx=0 tx=0).stop 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8a600640 1 -- 172.21.15.67:0/3002159221 reap_dead start 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 -- 172.21.15.67:0/3002159221 shutdown_connections 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 -- 172.21.15.67:0/3002159221 >> 172.21.15.67:0/3002159221 conn(0x7f7f8406cbd0 msgr2=0x7f7f8406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 -- 172.21.15.67:0/3002159221 shutdown_connections 2024-04-17T07:33:27.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.786+0000 7f7f8c600640 1 -- 172.21.15.67:0/3002159221 wait complete. 2024-04-17T07:33:27.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.788+0000 7f7f8c600640 1 Processor -- start 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.788+0000 7f7f8c600640 1 -- start start 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f6c0039b0 0x7f7f841f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f841f7820 0x7f7f840ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f840d25b0 con 0x7f7f841f7ca0 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7f840d38c0 con 0x7f7f841f7820 2024-04-17T07:33:27.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.789+0000 7f7f8c600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f840ba300 con 0x7f7f6c0039b0 2024-04-17T07:33:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56678/0 (socket says 172.21.15.67:56678) 2024-04-17T07:33:27.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 -- 172.21.15.67:0/4098347434 learned_addr learned my addr 172.21.15.67:0/4098347434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:27.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f89c00640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f6c0039b0 0x7f7f841f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f89200640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f841f7820 0x7f7f840ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 -- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f6c0039b0 msgr2=0x7f7f841f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f6c0039b0 0x7f7f841f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 -- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f841f7820 msgr2=0x7f7f840ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f841f7820 0x7f7f840ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.790+0000 7f7f8a600640 1 -- 172.21.15.67:0/4098347434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f840cc6d0 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f8a600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f7f840d4920 tx=0x7f7f7c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c018020 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00bd10 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f840cc9b0 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c014640 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.791+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f840ccf70 con 0x7f7f841f7ca0 2024-04-17T07:33:27.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.792+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f84068b90 con 0x7f7f841f7ca0 2024-04-17T07:33:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.793+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f7f7c01b020 con 0x7f7f841f7ca0 2024-04-17T07:33:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.794+0000 7f7f81600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 0x7f7f6406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:27.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.794+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5737+0+0 (secure 0 0 0) 0x7f7f7c08c630 con 0x7f7f841f7ca0 2024-04-17T07:33:27.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.795+0000 7f7f89c00640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 0x7f7f6406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:27.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.796+0000 7f7f89c00640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 0x7f7f6406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f78036000 tx=0x7f7f78030000 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:27.799+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7f7c022020 con 0x7f7f841f7ca0 2024-04-17T07:33:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.004+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f840b40b0 con 0x7f7f841f7ca0 2024-04-17T07:33:28.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.006+0000 7f7f81600640 1 -- 172.21.15.67:0/4098347434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) ==== 74+0+16636 (secure 0 0 0) 0x7f7f7c05a970 con 0x7f7f841f7ca0 2024-04-17T07:33:28.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:28.007 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":141,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:26.988169+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 msgr2=0x7f7f6406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 0x7f7f6406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f78036000 tx=0x7f7f78030000 comp rx=0 tx=0).stop 2024-04-17T07:33:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 msgr2=0x7f7f840ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f7f840d4920 tx=0x7f7f7c00b480 comp rx=0 tx=0).stop 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 shutdown_connections 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f841f7ca0 0x7f7f840ab420 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f841f7820 0x7f7f840ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f7f6406ca10 0x7f7f6406eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 --2- 172.21.15.67:0/4098347434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f6c0039b0 0x7f7f841f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 >> 172.21.15.67:0/4098347434 conn(0x7f7f8406cbd0 msgr2=0x7f7f840d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 shutdown_connections 2024-04-17T07:33:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.009+0000 7f7f8c600640 1 -- 172.21.15.67:0/4098347434 wait complete. 2024-04-17T07:33:28.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:28.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.238+0000 7f5e32a00640 1 Processor -- start 2024-04-17T07:33:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.238+0000 7f5e32a00640 1 -- start start 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.240+0000 7f5e32a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c06f670 0x7f5e2c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.240+0000 7f5e32a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e2c0cd020 0x7f5e2c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.240+0000 7f5e32a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e2c06ff90 con 0x7f5e2c071d10 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.240+0000 7f5e32a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5e2c0ceb00 con 0x7f5e2c0cd020 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.240+0000 7f5e32a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c0cede0 con 0x7f5e2c06f670 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e2c0cd020 0x7f5e2c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e2c0cd020 0x7f5e2c0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56692/0 (socket says 172.21.15.67:56692) 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e30a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5e2c071d10 0x7f5e2c0721c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56724/0 (socket says 172.21.15.67:56724) 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 -- 172.21.15.67:0/644369195 learned_addr learned my addr 172.21.15.67:0/644369195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e2be00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c06f670 0x7f5e2c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 -- 172.21.15.67:0/644369195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c06f670 msgr2=0x7f5e2c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c06f670 0x7f5e2c06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 -- 172.21.15.67:0/644369195 >> v1:172.21.15.67:6789/0 conn(0x7f5e2c071d10 legacy=0x7f5e2c0721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e23e00640 1 -- 172.21.15.67:0/644369195 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c0cf0c0 con 0x7f5e2c0cd020 2024-04-17T07:33:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.241+0000 7f5e2b400640 1 -- 172.21.15.67:0/644369195 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034539923 0 0) 0x7f5e2c06ff90 con 0x7f5e2c071d10 2024-04-17T07:33:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.242+0000 7f5e23e00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c0cd480 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2091df9c8f985d72 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e2b400640 1 -- 172.21.15.67:0/644369195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14034070 con 0x7f5e2c0cd020 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e2b400640 1 -- 172.21.15.67:0/644369195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e14030840 con 0x7f5e2c0cd020 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e2b400640 1 -- 172.21.15.67:0/644369195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14030c10 con 0x7f5e2c0cd020 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 -- 172.21.15.67:0/644369195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c0cd020 msgr2=0x7f5e2c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c0cd480 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b980 comp rx=0 tx=0).stop 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 -- 172.21.15.67:0/644369195 shutdown_connections 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c0cd480 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 --2- 172.21.15.67:0/644369195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c06f670 0x7f5e2c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.243+0000 7f5e32a00640 1 -- 172.21.15.67:0/644369195 >> 172.21.15.67:0/644369195 conn(0x7f5e2c06bf50 msgr2=0x7f5e2c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.244+0000 7f5e32a00640 1 -- 172.21.15.67:0/644369195 shutdown_connections 2024-04-17T07:33:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.244+0000 7f5e32a00640 1 -- 172.21.15.67:0/644369195 wait complete. 2024-04-17T07:33:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.245+0000 7f5e32a00640 1 Processor -- start 2024-04-17T07:33:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.245+0000 7f5e32a00640 1 -- start start 2024-04-17T07:33:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c071d10 0x7f5e2c1f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0ce720 con 0x7f5e2c06f670 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0cf680 con 0x7f5e2c0cd020 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.246+0000 7f5e32a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c0ce8a0 con 0x7f5e2c071d10 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56696/0 (socket says 172.21.15.67:56696) 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 -- 172.21.15.67:0/3531485268 learned_addr learned my addr 172.21.15.67:0/3531485268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e23e00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c071d10 0x7f5e2c1f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e30a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 -- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c071d10 msgr2=0x7f5e2c1f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c071d10 0x7f5e2c1f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 -- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 msgr2=0x7f5e2c15dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c15dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 -- 172.21.15.67:0/3531485268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c15e400 con 0x7f5e2c06f670 2024-04-17T07:33:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e30a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c15dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.247+0000 7f5e2be00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00d5f0 tx=0x7f5e1c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.248+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0125f0 con 0x7f5e2c06f670 2024-04-17T07:33:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.248+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1c019070 con 0x7f5e2c06f670 2024-04-17T07:33:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.248+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0129d0 con 0x7f5e2c06f670 2024-04-17T07:33:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.248+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c158240 con 0x7f5e2c06f670 2024-04-17T07:33:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.248+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c158800 con 0x7f5e2c06f670 2024-04-17T07:33:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.249+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2c068b90 con 0x7f5e2c06f670 2024-04-17T07:33:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.250+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f5e1c012b70 con 0x7f5e2c06f670 2024-04-17T07:33:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.251+0000 7f5e28c00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 0x7f5e0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.251+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5737+0+0 (secure 0 0 0) 0x7f5e1c08c190 con 0x7f5e2c06f670 2024-04-17T07:33:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.252+0000 7f5e23e00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 0x7f5e0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.253+0000 7f5e23e00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 0x7f5e0806eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e14036000 tx=0x7f5e14008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.256+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5e1c05a3f0 con 0x7f5e2c06f670 2024-04-17T07:33:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.463+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e2c06ccd0 con 0x7f5e2c06f670 2024-04-17T07:33:28.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.466+0000 7f5e28c00640 1 -- 172.21.15.67:0/3531485268 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) ==== 74+0+16636 (secure 0 0 0) 0x7f5e1c059cf0 con 0x7f5e2c06f670 2024-04-17T07:33:28.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:28.468 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":141,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:26.988169+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 msgr2=0x7f5e0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 0x7f5e0806eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e14036000 tx=0x7f5e14008030 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 msgr2=0x7f5e2c157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00d5f0 tx=0x7f5e1c00dab0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 shutdown_connections 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5e2c0cd020 0x7f5e2c15dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f5e0806ca10 0x7f5e0806eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5e2c071d10 0x7f5e2c1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 --2- 172.21.15.67:0/3531485268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c06f670 0x7f5e2c157b80 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.469+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 >> 172.21.15.67:0/3531485268 conn(0x7f5e2c06bf50 msgr2=0x7f5e2c15bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.470+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 shutdown_connections 2024-04-17T07:33:28.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.470+0000 7f5e32a00640 1 -- 172.21.15.67:0/3531485268 wait complete. 2024-04-17T07:33:28.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:33:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.715+0000 7f9ee2000640 1 Processor -- start 2024-04-17T07:33:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.715+0000 7f9ee2000640 1 -- start start 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.717+0000 7f9ee2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9edc06c270 0x7f9edc167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.717+0000 7f9ee2000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9edc06f1d0 con 0x7f9edc145f80 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.717+0000 7f9ee2000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9edc146360 con 0x7f9edc0cd920 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.717+0000 7f9ee2000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9edc0727a0 con 0x7f9edc06c270 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.717+0000 7f9ee1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9edc06c270 0x7f9edc167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.718+0000 7f9ee1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9edc06c270 0x7f9edc167380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56708/0 (socket says 172.21.15.67:56708) 2024-04-17T07:33:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.718+0000 7f9ee1600640 1 -- 172.21.15.67:0/3864157119 learned_addr learned my addr 172.21.15.67:0/3864157119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:28.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.718+0000 7f9ee1600640 1 -- 172.21.15.67:0/3864157119 >> v1:172.21.15.187:6789/0 conn(0x7f9edc0cd920 legacy=0x7f9edc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:28.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.718+0000 7f9ee1600640 1 -- 172.21.15.67:0/3864157119 >> v1:172.21.15.67:6789/0 conn(0x7f9edc145f80 legacy=0x7f9edc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.718+0000 7f9ee1600640 1 -- 172.21.15.67:0/3864157119 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc071e70 con 0x7f9edc06c270 2024-04-17T07:33:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.719+0000 7f9ee1600640 1 --2- 172.21.15.67:0/3864157119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc167380 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009e50 tx=0x7f9ed402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb751de0fe2889b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.719+0000 7f9edb400640 1 -- 172.21.15.67:0/3864157119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed402ca80 con 0x7f9edc06c270 2024-04-17T07:33:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.719+0000 7f9edb400640 1 -- 172.21.15.67:0/3864157119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed402cc20 con 0x7f9edc06c270 2024-04-17T07:33:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.719+0000 7f9edb400640 1 -- 172.21.15.67:0/3864157119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40343f0 con 0x7f9edc06c270 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 -- 172.21.15.67:0/3864157119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 msgr2=0x7f9edc167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3864157119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc167380 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009e50 tx=0x7f9ed402bee0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 -- 172.21.15.67:0/3864157119 shutdown_connections 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3864157119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc167380 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 -- 172.21.15.67:0/3864157119 >> 172.21.15.67:0/3864157119 conn(0x7f9edc0b2190 msgr2=0x7f9edc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 -- 172.21.15.67:0/3864157119 shutdown_connections 2024-04-17T07:33:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.720+0000 7f9ee2000640 1 -- 172.21.15.67:0/3864157119 wait complete. 2024-04-17T07:33:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.722+0000 7f9ee2000640 1 Processor -- start 2024-04-17T07:33:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.722+0000 7f9ee2000640 1 -- start start 2024-04-17T07:33:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9edc0cd920 0x7f9edc1f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 0x7f9edc1e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc072c30 con 0x7f9edc06c270 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc072430 con 0x7f9edc145f80 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee2000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc072db0 con 0x7f9edc0cd920 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56712/0 (socket says 172.21.15.67:56712) 2024-04-17T07:33:28.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.723+0000 7f9ee0c00640 1 -- 172.21.15.67:0/3489637516 learned_addr learned my addr 172.21.15.67:0/3489637516 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:28.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee1600640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 0x7f9edc1e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9edbe00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9edc0cd920 0x7f9edc1f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 -- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9edc0cd920 msgr2=0x7f9edc1f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9edc0cd920 0x7f9edc1f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 -- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 msgr2=0x7f9edc1e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 0x7f9edc1e8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 -- 172.21.15.67:0/3489637516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc1f2c40 con 0x7f9edc06c270 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee1600640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 0x7f9edc1e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.724+0000 7f9ee0c00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c3c0 tx=0x7f9ed000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.725+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00052d0 con 0x7f9edc06c270 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.725+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed0005490 con 0x7f9edc06c270 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.725+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0007260 con 0x7f9edc06c270 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.725+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc1f2f20 con 0x7f9edc06c270 2024-04-17T07:33:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.725+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc1451c0 con 0x7f9edc06c270 2024-04-17T07:33:28.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.726+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc068a10 con 0x7f9edc06c270 2024-04-17T07:33:28.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.727+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f9ed0007400 con 0x7f9edc06c270 2024-04-17T07:33:28.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.728+0000 7f9ed8c00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 0x7f9ebc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:28.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.728+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f9ed008b150 con 0x7f9edc06c270 2024-04-17T07:33:28.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.729+0000 7f9edbe00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 0x7f9ebc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.730+0000 7f9edbe00640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 0x7f9ebc06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0757e0 tx=0x7f9ec8009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:28.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.733+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9edc06f1d0 con 0x7f9edc06c270 2024-04-17T07:33:28.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:28.983+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9edc15f040 con 0x7f9edc06c270 2024-04-17T07:33:29.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:33:29.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:33:29.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.570+0000 7f9ed8c00640 1 -- 172.21.15.67:0/3489637516 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v67) ==== 118+0+0 (secure 0 0 0) 0x7f9ed00594e0 con 0x7f9edc06c270 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 msgr2=0x7f9ebc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 0x7f9ebc06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0757e0 tx=0x7f9ec8009040 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 msgr2=0x7f9edc1444b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c3c0 tx=0x7f9ed000c880 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 shutdown_connections 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9edc145f80 0x7f9edc1e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f9ebc06ca10 0x7f9ebc06eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9edc0cd920 0x7f9edc1f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 --2- 172.21.15.67:0/3489637516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9edc06c270 0x7f9edc1444b0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.574+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 >> 172.21.15.67:0/3489637516 conn(0x7f9edc0b2190 msgr2=0x7f9edc0b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.575+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 shutdown_connections 2024-04-17T07:33:29.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.575+0000 7f9ee2000640 1 -- 172.21.15.67:0/3489637516 wait complete. 2024-04-17T07:33:29.599 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:33:29.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:33:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.830+0000 7fa87ec00640 1 Processor -- start 2024-04-17T07:33:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.830+0000 7fa87ec00640 1 -- start start 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa8780cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87ec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8780d00b0 con 0x7fa8780cf8e0 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87ec00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8780d0390 con 0x7fa87815ecb0 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8780d0c30 con 0x7fa8780d33e0 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87e200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa8780cf8e0 0x7fa8780b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56744/0 (socket says 172.21.15.67:56744) 2024-04-17T07:33:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.832+0000 7fa87e200640 1 -- 172.21.15.67:0/3832638518 learned_addr learned my addr 172.21.15.67:0/3832638518 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa87ce00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa8780cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198696521 0 0) 0x7fa8780d00b0 con 0x7fa8780cf8e0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa860003520 con 0x7fa8780cf8e0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 132872131 0 0) 0x7fa860003520 con 0x7fa8780cf8e0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 msgr2=0x7fa8780cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa8780cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 >> v1:172.21.15.187:6789/0 conn(0x7fa87815ecb0 legacy=0x7fa8780d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8780d0670 con 0x7fa8780cf8e0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa8700019f0 con 0x7fa8780cf8e0 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 >> v1:172.21.15.67:6789/0 conn(0x7fa8780cf8e0 legacy=0x7fa8780b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8600039e0 0x7fa860003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa860004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780d0c30 con 0x7fa860004d40 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa86000bb80 con 0x7fa8600039e0 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.833+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa86000bd80 con 0x7fa8780d33e0 2024-04-17T07:33:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87ce00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa860004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87d800640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8600039e0 0x7fa860003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 -- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 msgr2=0x7fa860004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8780d33e0 0x7fa860004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 -- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8600039e0 msgr2=0x7fa860003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8600039e0 0x7fa860003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.834+0000 7fa87e200640 1 -- 172.21.15.67:0/3832638518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa860003520 con 0x7fa860004d40 2024-04-17T07:33:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.836+0000 7fa87e200640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8600051c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa87000a840 tx=0x7fa860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1faf03c3b784c4cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.836+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87000a460 con 0x7fa860004d40 2024-04-17T07:33:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.836+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa870034650 con 0x7fa860004d40 2024-04-17T07:33:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.836+0000 7fa877e00640 1 -- 172.21.15.67:0/3832638518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa870034a40 con 0x7fa860004d40 2024-04-17T07:33:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 -- 172.21.15.67:0/3832638518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 msgr2=0x7fa8600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 --2- 172.21.15.67:0/3832638518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8600051c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa87000a840 tx=0x7fa860005830 comp rx=0 tx=0).stop 2024-04-17T07:33:29.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87e200640 1 -- 172.21.15.67:0/3832638518 reap_dead start 2024-04-17T07:33:29.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 -- 172.21.15.67:0/3832638518 shutdown_connections 2024-04-17T07:33:29.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 -- 172.21.15.67:0/3832638518 >> 172.21.15.67:0/3832638518 conn(0x7fa87808bff0 msgr2=0x7fa8780d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:29.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 -- 172.21.15.67:0/3832638518 shutdown_connections 2024-04-17T07:33:29.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.837+0000 7fa87ec00640 1 -- 172.21.15.67:0/3832638518 wait complete. 2024-04-17T07:33:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.839+0000 7fa87ec00640 1 Processor -- start 2024-04-17T07:33:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.839+0000 7fa87ec00640 1 -- start start 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa878155c90 0x7fa8780a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 0x7fa8780a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa87815f090 con 0x7fa860004d40 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780cfd30 con 0x7fa87809fdc0 2024-04-17T07:33:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa87815f210 con 0x7fa878155c90 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56732/0 (socket says 172.21.15.67:56732) 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.840+0000 7fa87d800640 1 -- 172.21.15.67:0/4211746734 learned_addr learned my addr 172.21.15.67:0/4211746734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87ce00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa878155c90 0x7fa8780a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87e200640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 0x7fa8780a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 -- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa878155c90 msgr2=0x7fa8780a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa878155c90 0x7fa8780a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 -- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 msgr2=0x7fa8780a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 0x7fa8780a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 -- 172.21.15.67:0/4211746734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87809edd0 con 0x7fa860004d40 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87e200640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 0x7fa8780a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:29.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.841+0000 7fa87d800640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e550 tx=0x7fa86c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:29.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.842+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0052e0 con 0x7fa860004d40 2024-04-17T07:33:29.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.842+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c018070 con 0x7fa860004d40 2024-04-17T07:33:29.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.842+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c003810 con 0x7fa860004d40 2024-04-17T07:33:29.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.842+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8780a23c0 con 0x7fa860004d40 2024-04-17T07:33:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.842+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8780cddb0 con 0x7fa860004d40 2024-04-17T07:33:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.843+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa878068a10 con 0x7fa860004d40 2024-04-17T07:33:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.844+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fa86c0039b0 con 0x7fa860004d40 2024-04-17T07:33:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.845+0000 7fa875600640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 0x7fa85806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.846+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7fa86c08b960 con 0x7fa860004d40 2024-04-17T07:33:29.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.846+0000 7fa87ce00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 0x7fa85806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:29.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.847+0000 7fa87ce00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 0x7fa85806ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8781f7940 tx=0x7fa864008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:29.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:29.850+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa86c059cf0 con 0x7fa860004d40 2024-04-17T07:33:30.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.095+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa8780b5a70 con 0x7fa860004d40 2024-04-17T07:33:30.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.583+0000 7fa875600640 1 -- 172.21.15.67:0/4211746734 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v68) ==== 122+0+0 (secure 0 0 0) 0x7fa86c0595f0 con 0x7fa860004d40 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 msgr2=0x7fa85806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 0x7fa85806ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8781f7940 tx=0x7fa864008040 comp rx=0 tx=0).stop 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 msgr2=0x7fa8780ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e550 tx=0x7fa86c00ea10 comp rx=0 tx=0).stop 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 shutdown_connections 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa87809fdc0 0x7fa8780a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa85806ca40 0x7fa85806ef00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa878155c90 0x7fa8780a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 --2- 172.21.15.67:0/4211746734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa860004d40 0x7fa8780ce330 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 >> 172.21.15.67:0/4211746734 conn(0x7fa87808bff0 msgr2=0x7fa8780b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 shutdown_connections 2024-04-17T07:33:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.587+0000 7fa87ec00640 1 -- 172.21.15.67:0/4211746734 wait complete. 2024-04-17T07:33:30.611 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:33:30.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.836+0000 7f8df7200640 1 Processor -- start 2024-04-17T07:33:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.837+0000 7f8df7200640 1 -- start start 2024-04-17T07:33:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df815eb60 0x7f8df80b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80708d0 0x7f8df806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df8071240 con 0x7f8df8070d10 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8df8071520 con 0x7f8df815eb60 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8df8071800 con 0x7f8df80708d0 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df815eb60 0x7f8df80b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df815eb60 0x7f8df80b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56744/0 (socket says 172.21.15.67:56744) 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.839+0000 7f8df5e00640 1 -- 172.21.15.67:0/1854297655 learned_addr learned my addr 172.21.15.67:0/1854297655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.840+0000 7f8df5400640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80708d0 0x7f8df806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.840+0000 7f8df5e00640 1 -- 172.21.15.67:0/1854297655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80708d0 msgr2=0x7f8df806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.840+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80708d0 0x7f8df806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.840+0000 7f8df5e00640 1 -- 172.21.15.67:0/1854297655 >> v1:172.21.15.187:6789/0 conn(0x7f8df8070d10 legacy=0x7f8df80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.840+0000 7f8df5e00640 1 -- 172.21.15.67:0/1854297655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df80cdb70 con 0x7f8df815eb60 2024-04-17T07:33:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.841+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df815eb60 0x7f8df80b45b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0cb4a8061ee212 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.841+0000 7f8df4a00640 1 -- 172.21.15.67:0/1854297655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034070 con 0x7f8df815eb60 2024-04-17T07:33:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df4a00640 1 -- 172.21.15.67:0/1854297655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8030840 con 0x7f8df815eb60 2024-04-17T07:33:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df4a00640 1 -- 172.21.15.67:0/1854297655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8030c10 con 0x7f8df815eb60 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 -- 172.21.15.67:0/1854297655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df815eb60 msgr2=0x7f8df80b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df815eb60 0x7f8df80b45b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de802b980 comp rx=0 tx=0).stop 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 -- 172.21.15.67:0/1854297655 shutdown_connections 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80708d0 0x7f8df806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 --2- 172.21.15.67:0/1854297655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df815eb60 0x7f8df80b45b0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 -- 172.21.15.67:0/1854297655 >> 172.21.15.67:0/1854297655 conn(0x7f8df806f060 msgr2=0x7f8df806f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 -- 172.21.15.67:0/1854297655 shutdown_connections 2024-04-17T07:33:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.842+0000 7f8df7200640 1 -- 172.21.15.67:0/1854297655 wait complete. 2024-04-17T07:33:30.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.844+0000 7f8df7200640 1 Processor -- start 2024-04-17T07:33:30.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.844+0000 7f8df7200640 1 -- start start 2024-04-17T07:33:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 0x7f8df80cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df815eb60 0x7f8df8151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8073d40 con 0x7f8df80708d0 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8df815ef40 con 0x7f8df8070d10 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.845+0000 7f8df7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8df8073ec0 con 0x7f8df815eb60 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56756/0 (socket says 172.21.15.67:56756) 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 -- 172.21.15.67:0/1712854931 learned_addr learned my addr 172.21.15.67:0/1712854931 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5400640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 0x7f8df80cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 -- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df815eb60 msgr2=0x7f8df8151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df6800640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df815eb60 0x7f8df8151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df815eb60 0x7f8df8151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 -- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 msgr2=0x7f8df80cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 0x7f8df80cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 -- 172.21.15.67:0/1712854931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df80cb880 con 0x7f8df80708d0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5400640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 0x7f8df80cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.846+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.847+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8048070 con 0x7f8df80708d0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.847+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8030640 con 0x7f8df80708d0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.847+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df80cbb50 con 0x7f8df80708d0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.847+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034040 con 0x7f8df80708d0 2024-04-17T07:33:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.847+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df80cc110 con 0x7f8df80708d0 2024-04-17T07:33:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.848+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df80755d0 con 0x7f8df80708d0 2024-04-17T07:33:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.849+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f8de804b070 con 0x7f8df80708d0 2024-04-17T07:33:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.850+0000 7f8dee000640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 0x7f8dd406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.850+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f8de80ade70 con 0x7f8df80708d0 2024-04-17T07:33:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.851+0000 7f8df5400640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 0x7f8dd406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.852+0000 7f8df5400640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 0x7f8dd406eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8df80b3160 tx=0x7f8de000b040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:30.855+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8de807c080 con 0x7f8df80708d0 2024-04-17T07:33:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.100+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8df815dda0 con 0x7f8df80708d0 2024-04-17T07:33:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.102+0000 7f8dee000640 1 -- 172.21.15.67:0/1712854931 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7f8de807b980 con 0x7f8df80708d0 2024-04-17T07:33:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:31.103 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 msgr2=0x7f8dd406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 0x7f8dd406eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8df80b3160 tx=0x7f8de000b040 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 msgr2=0x7f8df81613b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009870 tx=0x7f8de8004730 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 shutdown_connections 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f8dd406ca10 0x7f8dd406eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df815eb60 0x7f8df8151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df8070d10 0x7f8df80cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 --2- 172.21.15.67:0/1712854931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80708d0 0x7f8df81613b0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.104+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 >> 172.21.15.67:0/1712854931 conn(0x7f8df806f060 msgr2=0x7f8df80d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.105+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 shutdown_connections 2024-04-17T07:33:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.105+0000 7f8df7200640 1 -- 172.21.15.67:0/1712854931 wait complete. 2024-04-17T07:33:31.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.335+0000 7f4857e00640 1 Processor -- start 2024-04-17T07:33:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.335+0000 7f4857e00640 1 -- start start 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f48581625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48580d0e60 con 0x7f485806c9d0 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48580d1140 con 0x7f48580d1f00 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f48580ceb50 con 0x7f48580ce770 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f485806c9d0 0x7f485806ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56754/0 (socket says 172.21.15.67:56754) 2024-04-17T07:33:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.336+0000 7f4857400640 1 -- 172.21.15.67:0/2633798498 learned_addr learned my addr 172.21.15.67:0/2633798498 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4856a00640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f48581625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1671668107 0 0) 0x7f48580d0e60 con 0x7f485806c9d0 2024-04-17T07:33:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4838003520 con 0x7f485806c9d0 2024-04-17T07:33:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2272699447 0 0) 0x7f4838003520 con 0x7f485806c9d0 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 msgr2=0x7f48581625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f48581625a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 >> v1:172.21.15.187:6789/0 conn(0x7f48580d1f00 legacy=0x7f4858162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48580d1420 con 0x7f485806c9d0 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4850001670 con 0x7f485806c9d0 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 >> v1:172.21.15.67:6789/0 conn(0x7f485806c9d0 legacy=0x7f485806ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003a50 0x7f4838003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4838004800 0x7f4838004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f483800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f48580ceb50 con 0x7f4838004800 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4838005520 con 0x7f4838003a50 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.337+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f483800bf00 con 0x7f48580ce770 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4838004800 0x7f4838004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4857400640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f483800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856a00640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003a50 0x7f4838003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 -- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 msgr2=0x7f483800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f483800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 -- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003a50 msgr2=0x7f4838003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838003a50 0x7f4838003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4856000640 1 -- 172.21.15.67:0/2633798498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4838003520 con 0x7f4838004800 2024-04-17T07:33:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.338+0000 7f4857400640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48580ce770 0x7f483800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.339+0000 7f4856000640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4838004800 0x7f4838004c80 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f4840001e20 tx=0x7f4838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6979370586ebea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.339+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840001a70 con 0x7f4838004800 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.339+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4840003b70 con 0x7f4838004800 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.339+0000 7f4855600640 1 -- 172.21.15.67:0/2633798498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48400050d0 con 0x7f4838004800 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 -- 172.21.15.67:0/2633798498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4838004800 msgr2=0x7f4838004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 --2- 172.21.15.67:0/2633798498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4838004800 0x7f4838004c80 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f4840001e20 tx=0x7f4838005830 comp rx=0 tx=0).stop 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857400640 1 -- 172.21.15.67:0/2633798498 reap_dead start 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 -- 172.21.15.67:0/2633798498 shutdown_connections 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 -- 172.21.15.67:0/2633798498 >> 172.21.15.67:0/2633798498 conn(0x7f48580b4170 msgr2=0x7f48580b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 -- 172.21.15.67:0/2633798498 shutdown_connections 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.340+0000 7f4857e00640 1 -- 172.21.15.67:0/2633798498 wait complete. 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.342+0000 7f4857e00640 1 Processor -- start 2024-04-17T07:33:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.342+0000 7f4857e00640 1 -- start start 2024-04-17T07:33:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838004800 0x7f485815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f485809d5c0 0x7f48580b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f48581631f0 con 0x7f485809da40 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f48580d1a70 con 0x7f4838004800 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4858163370 con 0x7f485809d5c0 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56772/0 (socket says 172.21.15.67:56772) 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.343+0000 7f4857400640 1 -- 172.21.15.67:0/2457693339 learned_addr learned my addr 172.21.15.67:0/2457693339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4856000640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f485809d5c0 0x7f48580b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4856a00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838004800 0x7f485815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 -- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f485809d5c0 msgr2=0x7f48580b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f485809d5c0 0x7f48580b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 -- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838004800 msgr2=0x7f485815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838004800 0x7f485815fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 -- 172.21.15.67:0/2457693339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48580b6790 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857400640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f48580b8730 tx=0x7f485000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48500075a0 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48500047e0 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4858161ad0 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.344+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850004bd0 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.345+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48580c75f0 con 0x7f485809da40 2024-04-17T07:33:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.346+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48580755d0 con 0x7f485809da40 2024-04-17T07:33:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.347+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f485001b020 con 0x7f485809da40 2024-04-17T07:33:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.348+0000 7f484f400640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 0x7f483406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.348+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f485008b8d0 con 0x7f485809da40 2024-04-17T07:33:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.348+0000 7f4856a00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 0x7f483406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.349+0000 7f4856a00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 0x7f483406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4848006fd0 tx=0x7f4848008040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.352+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f485000f2f0 con 0x7f485809da40 2024-04-17T07:33:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.559+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f485806efe0 con 0x7f485809da40 2024-04-17T07:33:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.561+0000 7f484f400640 1 -- 172.21.15.67:0/2457693339 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+16623 (secure 0 0 0) 0x7f4850059ab0 con 0x7f485809da40 2024-04-17T07:33:31.562 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:31.562 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":142,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:28.555092+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 msgr2=0x7f483406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 0x7f483406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4848006fd0 tx=0x7f4848008040 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 msgr2=0x7f48580b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f48580b8730 tx=0x7f485000d9d0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 shutdown_connections 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f485809da40 0x7f48580b6130 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f483406ca40 0x7f483406ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f485809d5c0 0x7f48580b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 --2- 172.21.15.67:0/2457693339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4838004800 0x7f485815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 >> 172.21.15.67:0/2457693339 conn(0x7f48580b4170 msgr2=0x7f48580cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.564+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 shutdown_connections 2024-04-17T07:33:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.565+0000 7f4857e00640 1 -- 172.21.15.67:0/2457693339 wait complete. 2024-04-17T07:33:31.585 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:33:31.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:33:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.805+0000 7f55dec00640 1 Processor -- start 2024-04-17T07:33:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.806+0000 7f55dec00640 1 -- start start 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dec00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 0x7f55d80b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55d80b6b30 0x7f55d80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d80d2d30 con 0x7f55d80cf910 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f55d80d49c0 con 0x7f55d80b6b30 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dec00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f55d80d3660 con 0x7f55d815ec60 2024-04-17T07:33:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55dce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55d80b6b30 0x7f55d80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55de200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f55d80cf910 0x7f55d806cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56756/0 (socket says 172.21.15.67:56756) 2024-04-17T07:33:31.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.807+0000 7f55de200640 1 -- 172.21.15.67:0/2687293667 learned_addr learned my addr 172.21.15.67:0/2687293667 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:31.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dce00640 1 -- 172.21.15.67:0/2687293667 >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 msgr2=0x7f55d80b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dd800640 1 --2- 172.21.15.67:0/2687293667 >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 0x7f55d80b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55d7e00640 1 -- 172.21.15.67:0/2687293667 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654511438 0 0) 0x7f55d80d2d30 con 0x7f55d80cf910 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dce00640 1 --2- 172.21.15.67:0/2687293667 >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 0x7f55d80b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dce00640 1 -- 172.21.15.67:0/2687293667 >> v1:172.21.15.67:6789/0 conn(0x7f55d80cf910 legacy=0x7f55d806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dce00640 1 -- 172.21.15.67:0/2687293667 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d80d3010 con 0x7f55d80b6b30 2024-04-17T07:33:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.808+0000 7f55dd800640 1 --2- 172.21.15.67:0/2687293667 >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 0x7f55d80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.809+0000 7f55dce00640 1 --2- 172.21.15.67:0/2687293667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d80cf3d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f55d80d1550 tx=0x7f55c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f945f654dc364 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.809+0000 7f55d7e00640 1 -- 172.21.15.67:0/2687293667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4035020 con 0x7f55d80b6b30 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.809+0000 7f55d7e00640 1 -- 172.21.15.67:0/2687293667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c4031830 con 0x7f55d80b6b30 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.809+0000 7f55d7e00640 1 -- 172.21.15.67:0/2687293667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4031c00 con 0x7f55d80b6b30 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 -- 172.21.15.67:0/2687293667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 msgr2=0x7f55d80cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 --2- 172.21.15.67:0/2687293667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d80cf3d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f55d80d1550 tx=0x7f55c402b8d0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 -- 172.21.15.67:0/2687293667 shutdown_connections 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 --2- 172.21.15.67:0/2687293667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d80cf3d0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 --2- 172.21.15.67:0/2687293667 >> v2:172.21.15.187:3300/0 conn(0x7f55d815ec60 0x7f55d80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 -- 172.21.15.67:0/2687293667 >> 172.21.15.67:0/2687293667 conn(0x7f55d80d15f0 msgr2=0x7f55d80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 -- 172.21.15.67:0/2687293667 shutdown_connections 2024-04-17T07:33:31.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.810+0000 7f55dec00640 1 -- 172.21.15.67:0/2687293667 wait complete. 2024-04-17T07:33:31.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.812+0000 7f55dec00640 1 Processor -- start 2024-04-17T07:33:31.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.812+0000 7f55dec00640 1 -- start start 2024-04-17T07:33:31.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d80cf910 0x7f55d80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 0x7f55d80ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55d80cfda0 con 0x7f55d80b6b30 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f55d80b6f70 con 0x7f55d80cf910 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55d815f040 con 0x7f55d815ec60 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.813+0000 7f55dd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56790/0 (socket says 172.21.15.67:56790) 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 -- 172.21.15.67:0/2117888086 learned_addr learned my addr 172.21.15.67:0/2117888086 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dce00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d80cf910 0x7f55d80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55de200640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 0x7f55d80ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 -- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 msgr2=0x7f55d80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 0x7f55d80ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 -- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d80cf910 msgr2=0x7f55d80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d80cf910 0x7f55d80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 -- 172.21.15.67:0/2117888086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d80cce60 con 0x7f55d80b6b30 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55de200640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 0x7f55d80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.814+0000 7f55dd800640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00c7f0 tx=0x7f55cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.815+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc015070 con 0x7f55d80b6b30 2024-04-17T07:33:31.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.815+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55cc007820 con 0x7f55d80b6b30 2024-04-17T07:33:31.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.815+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d80cd140 con 0x7f55d80b6b30 2024-04-17T07:33:31.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.815+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc007bd0 con 0x7f55d80b6b30 2024-04-17T07:33:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.815+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d80cd700 con 0x7f55d80b6b30 2024-04-17T07:33:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.816+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d8068a10 con 0x7f55d80b6b30 2024-04-17T07:33:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.817+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f55cc018070 con 0x7f55d80b6b30 2024-04-17T07:33:31.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.819+0000 7f55d5600640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 0x7f55b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:31.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.819+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f55cc08bff0 con 0x7f55d80b6b30 2024-04-17T07:33:31.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.819+0000 7f55dce00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 0x7f55b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.820+0000 7f55dce00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 0x7f55b806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55c4036000 tx=0x7f55c4008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:31.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:31.823+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f55cc05a140 con 0x7f55d80b6b30 2024-04-17T07:33:31.921 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T07:33:31.920+0000 7f9775000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:33:32.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.022+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 --> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f55d80cdb20 con 0x7f55b806ca10 2024-04-17T07:33:32.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.046+0000 7f55d5600640 1 -- 172.21.15.67:0/2117888086 <== mgr.6221 v2:172.21.15.187:6832/1274719646 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252484 (secure 0 0 0) 0x7f55c4086a60 con 0x7f55b806ca10 2024-04-17T07:33:32.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:32.052 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.052+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 msgr2=0x7f55b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.052+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 0x7f55b806eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55c4036000 tx=0x7f55c4008030 comp rx=0 tx=0).stop 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.052+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 msgr2=0x7f55d815d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.052+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00c7f0 tx=0x7f55cc00ccb0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 shutdown_connections 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f55b806ca10 0x7f55b806eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55d815ec60 0x7f55d80ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d80cf910 0x7f55d80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 --2- 172.21.15.67:0/2117888086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55d80b6b30 0x7f55d815d100 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 >> 172.21.15.67:0/2117888086 conn(0x7f55d80d15f0 msgr2=0x7f55d815e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 shutdown_connections 2024-04-17T07:33:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.053+0000 7f55dec00640 1 -- 172.21.15.67:0/2117888086 wait complete. 2024-04-17T07:33:32.074 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-17T07:33:31.947704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":412,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355964,"kb_used_data":14892,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618756,"statfs":{"total":773094113280,"available":772729606144,"internally_reserved":0,"allocated":15249408,"data_stored":7912412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":349097520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002315"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759420+0000","last_change":"2024-04-17T07:33:27.582626+0000","last_active":"2024-04-17T07:33:28.759420+0000","last_peered":"2024-04-17T07:33:28.759420+0000","last_clean":"2024-04-17T07:33:28.759420+0000","last_became_active":"2024-04-17T07:33:27.580553+0000","last_became_peered":"2024-04-17T07:33:27.580553+0000","last_unstale":"2024-04-17T07:33:28.759420+0000","last_undegraded":"2024-04-17T07:33:28.759420+0000","last_fullsized":"2024-04-17T07:33:28.759420+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:19.845800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759029+0000","last_change":"2024-04-17T07:33:27.573363+0000","last_active":"2024-04-17T07:33:28.759029+0000","last_peered":"2024-04-17T07:33:28.759029+0000","last_clean":"2024-04-17T07:33:28.759029+0000","last_became_active":"2024-04-17T07:33:27.572971+0000","last_became_peered":"2024-04-17T07:33:27.572971+0000","last_unstale":"2024-04-17T07:33:28.759029+0000","last_undegraded":"2024-04-17T07:33:28.759029+0000","last_fullsized":"2024-04-17T07:33:28.759029+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:36:00.821543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567373+0000","last_change":"2024-04-17T07:33:27.574591+0000","last_active":"2024-04-17T07:33:28.567373+0000","last_peered":"2024-04-17T07:33:28.567373+0000","last_clean":"2024-04-17T07:33:28.567373+0000","last_became_active":"2024-04-17T07:33:27.573462+0000","last_became_peered":"2024-04-17T07:33:27.573462+0000","last_unstale":"2024-04-17T07:33:28.567373+0000","last_undegraded":"2024-04-17T07:33:28.567373+0000","last_fullsized":"2024-04-17T07:33:28.567373+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:39.157892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757942+0000","last_change":"2024-04-17T07:33:27.581825+0000","last_active":"2024-04-17T07:33:28.757942+0000","last_peered":"2024-04-17T07:33:28.757942+0000","last_clean":"2024-04-17T07:33:28.757942+0000","last_became_active":"2024-04-17T07:33:27.580867+0000","last_became_peered":"2024-04-17T07:33:27.580867+0000","last_unstale":"2024-04-17T07:33:28.757942+0000","last_undegraded":"2024-04-17T07:33:28.757942+0000","last_fullsized":"2024-04-17T07:33:28.757942+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:26.309376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760366+0000","last_change":"2024-04-17T07:33:27.577742+0000","last_active":"2024-04-17T07:33:28.760366+0000","last_peered":"2024-04-17T07:33:28.760366+0000","last_clean":"2024-04-17T07:33:28.760366+0000","last_became_active":"2024-04-17T07:33:27.577467+0000","last_became_peered":"2024-04-17T07:33:27.577467+0000","last_unstale":"2024-04-17T07:33:28.760366+0000","last_undegraded":"2024-04-17T07:33:28.760366+0000","last_fullsized":"2024-04-17T07:33:28.760366+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:53.801946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759012+0000","last_change":"2024-04-17T07:33:27.584765+0000","last_active":"2024-04-17T07:33:28.759012+0000","last_peered":"2024-04-17T07:33:28.759012+0000","last_clean":"2024-04-17T07:33:28.759012+0000","last_became_active":"2024-04-17T07:33:27.583713+0000","last_became_peered":"2024-04-17T07:33:27.583713+0000","last_unstale":"2024-04-17T07:33:28.759012+0000","last_undegraded":"2024-04-17T07:33:28.759012+0000","last_fullsized":"2024-04-17T07:33:28.759012+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:55.750226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566108+0000","last_change":"2024-04-17T07:33:27.564544+0000","last_active":"2024-04-17T07:33:28.566108+0000","last_peered":"2024-04-17T07:33:28.566108+0000","last_clean":"2024-04-17T07:33:28.566108+0000","last_became_active":"2024-04-17T07:33:27.564204+0000","last_became_peered":"2024-04-17T07:33:27.564204+0000","last_unstale":"2024-04-17T07:33:28.566108+0000","last_undegraded":"2024-04-17T07:33:28.566108+0000","last_fullsized":"2024-04-17T07:33:28.566108+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:03.766352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.564918+0000","last_change":"2024-04-17T07:33:27.551547+0000","last_active":"2024-04-17T07:33:28.564918+0000","last_peered":"2024-04-17T07:33:28.564918+0000","last_clean":"2024-04-17T07:33:28.564918+0000","last_became_active":"2024-04-17T07:33:27.549482+0000","last_became_peered":"2024-04-17T07:33:27.549482+0000","last_unstale":"2024-04-17T07:33:28.564918+0000","last_undegraded":"2024-04-17T07:33:28.564918+0000","last_fullsized":"2024-04-17T07:33:28.564918+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:23.155259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759210+0000","last_change":"2024-04-17T07:33:27.585802+0000","last_active":"2024-04-17T07:33:28.759210+0000","last_peered":"2024-04-17T07:33:28.759210+0000","last_clean":"2024-04-17T07:33:28.759210+0000","last_became_active":"2024-04-17T07:33:27.585431+0000","last_became_peered":"2024-04-17T07:33:27.585431+0000","last_unstale":"2024-04-17T07:33:28.759210+0000","last_undegraded":"2024-04-17T07:33:28.759210+0000","last_fullsized":"2024-04-17T07:33:28.759210+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:54.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757859+0000","last_change":"2024-04-17T07:33:27.571584+0000","last_active":"2024-04-17T07:33:28.757859+0000","last_peered":"2024-04-17T07:33:28.757859+0000","last_clean":"2024-04-17T07:33:28.757859+0000","last_became_active":"2024-04-17T07:33:27.570734+0000","last_became_peered":"2024-04-17T07:33:27.570734+0000","last_unstale":"2024-04-17T07:33:28.757859+0000","last_undegraded":"2024-04-17T07:33:28.757859+0000","last_fullsized":"2024-04-17T07:33:28.757859+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:59:10.617296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758761+0000","last_change":"2024-04-17T07:33:27.574249+0000","last_active":"2024-04-17T07:33:28.758761+0000","last_peered":"2024-04-17T07:33:28.758761+0000","last_clean":"2024-04-17T07:33:28.758761+0000","last_became_active":"2024-04-17T07:33:27.573915+0000","last_became_peered":"2024-04-17T07:33:27.573915+0000","last_unstale":"2024-04-17T07:33:28.758761+0000","last_undegraded":"2024-04-17T07:33:28.758761+0000","last_fullsized":"2024-04-17T07:33:28.758761+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:06:31.064161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757940+0000","last_change":"2024-04-17T07:33:27.572760+0000","last_active":"2024-04-17T07:33:28.757940+0000","last_peered":"2024-04-17T07:33:28.757940+0000","last_clean":"2024-04-17T07:33:28.757940+0000","last_became_active":"2024-04-17T07:33:27.570847+0000","last_became_peered":"2024-04-17T07:33:27.570847+0000","last_unstale":"2024-04-17T07:33:28.757940+0000","last_undegraded":"2024-04-17T07:33:28.757940+0000","last_fullsized":"2024-04-17T07:33:28.757940+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:22.211757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567374+0000","last_change":"2024-04-17T07:33:27.568093+0000","last_active":"2024-04-17T07:33:28.567374+0000","last_peered":"2024-04-17T07:33:28.567374+0000","last_clean":"2024-04-17T07:33:28.567374+0000","last_became_active":"2024-04-17T07:33:27.567794+0000","last_became_peered":"2024-04-17T07:33:27.567794+0000","last_unstale":"2024-04-17T07:33:28.567374+0000","last_undegraded":"2024-04-17T07:33:28.567374+0000","last_fullsized":"2024-04-17T07:33:28.567374+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:36.779561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566502+0000","last_change":"2024-04-17T07:33:27.568549+0000","last_active":"2024-04-17T07:33:28.566502+0000","last_peered":"2024-04-17T07:33:28.566502+0000","last_clean":"2024-04-17T07:33:28.566502+0000","last_became_active":"2024-04-17T07:33:27.568189+0000","last_became_peered":"2024-04-17T07:33:27.568189+0000","last_unstale":"2024-04-17T07:33:28.566502+0000","last_undegraded":"2024-04-17T07:33:28.566502+0000","last_fullsized":"2024-04-17T07:33:28.566502+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:43.033379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757378+0000","last_change":"2024-04-17T07:33:27.583143+0000","last_active":"2024-04-17T07:33:28.757378+0000","last_peered":"2024-04-17T07:33:28.757378+0000","last_clean":"2024-04-17T07:33:28.757378+0000","last_became_active":"2024-04-17T07:33:27.582815+0000","last_became_peered":"2024-04-17T07:33:27.582815+0000","last_unstale":"2024-04-17T07:33:28.757378+0000","last_undegraded":"2024-04-17T07:33:28.757378+0000","last_fullsized":"2024-04-17T07:33:28.757378+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:03.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756549+0000","last_change":"2024-04-17T07:33:27.575976+0000","last_active":"2024-04-17T07:33:28.756549+0000","last_peered":"2024-04-17T07:33:28.756549+0000","last_clean":"2024-04-17T07:33:28.756549+0000","last_became_active":"2024-04-17T07:33:27.575675+0000","last_became_peered":"2024-04-17T07:33:27.575675+0000","last_unstale":"2024-04-17T07:33:28.756549+0000","last_undegraded":"2024-04-17T07:33:28.756549+0000","last_fullsized":"2024-04-17T07:33:28.756549+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:07.512759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758979+0000","last_change":"2024-04-17T07:33:27.574954+0000","last_active":"2024-04-17T07:33:28.758979+0000","last_peered":"2024-04-17T07:33:28.758979+0000","last_clean":"2024-04-17T07:33:28.758979+0000","last_became_active":"2024-04-17T07:33:27.574702+0000","last_became_peered":"2024-04-17T07:33:27.574702+0000","last_unstale":"2024-04-17T07:33:28.758979+0000","last_undegraded":"2024-04-17T07:33:28.758979+0000","last_fullsized":"2024-04-17T07:33:28.758979+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:46.951849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594239+0000","last_change":"2024-04-17T07:33:27.577851+0000","last_active":"2024-04-17T07:33:28.594239+0000","last_peered":"2024-04-17T07:33:28.594239+0000","last_clean":"2024-04-17T07:33:28.594239+0000","last_became_active":"2024-04-17T07:33:27.577226+0000","last_became_peered":"2024-04-17T07:33:27.577226+0000","last_unstale":"2024-04-17T07:33:28.594239+0000","last_undegraded":"2024-04-17T07:33:28.594239+0000","last_fullsized":"2024-04-17T07:33:28.594239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:43:06.763568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567501+0000","last_change":"2024-04-17T07:33:27.575145+0000","last_active":"2024-04-17T07:33:28.567501+0000","last_peered":"2024-04-17T07:33:28.567501+0000","last_clean":"2024-04-17T07:33:28.567501+0000","last_became_active":"2024-04-17T07:33:27.573004+0000","last_became_peered":"2024-04-17T07:33:27.573004+0000","last_unstale":"2024-04-17T07:33:28.567501+0000","last_undegraded":"2024-04-17T07:33:28.567501+0000","last_fullsized":"2024-04-17T07:33:28.567501+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:00.349150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758585+0000","last_change":"2024-04-17T07:33:27.574545+0000","last_active":"2024-04-17T07:33:28.758585+0000","last_peered":"2024-04-17T07:33:28.758585+0000","last_clean":"2024-04-17T07:33:28.758585+0000","last_became_active":"2024-04-17T07:33:27.571629+0000","last_became_peered":"2024-04-17T07:33:27.571629+0000","last_unstale":"2024-04-17T07:33:28.758585+0000","last_undegraded":"2024-04-17T07:33:28.758585+0000","last_fullsized":"2024-04-17T07:33:28.758585+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:34.411556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759028+0000","last_change":"2024-04-17T07:33:27.584790+0000","last_active":"2024-04-17T07:33:28.759028+0000","last_peered":"2024-04-17T07:33:28.759028+0000","last_clean":"2024-04-17T07:33:28.759028+0000","last_became_active":"2024-04-17T07:33:27.584482+0000","last_became_peered":"2024-04-17T07:33:27.584482+0000","last_unstale":"2024-04-17T07:33:28.759028+0000","last_undegraded":"2024-04-17T07:33:28.759028+0000","last_fullsized":"2024-04-17T07:33:28.759028+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:11.666283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562163+0000","last_change":"2024-04-17T07:33:27.582454+0000","last_active":"2024-04-17T07:33:28.562163+0000","last_peered":"2024-04-17T07:33:28.562163+0000","last_clean":"2024-04-17T07:33:28.562163+0000","last_became_active":"2024-04-17T07:33:27.580089+0000","last_became_peered":"2024-04-17T07:33:27.580089+0000","last_unstale":"2024-04-17T07:33:28.562163+0000","last_undegraded":"2024-04-17T07:33:28.562163+0000","last_fullsized":"2024-04-17T07:33:28.562163+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:58.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757967+0000","last_change":"2024-04-17T07:33:27.573427+0000","last_active":"2024-04-17T07:33:28.757967+0000","last_peered":"2024-04-17T07:33:28.757967+0000","last_clean":"2024-04-17T07:33:28.757967+0000","last_became_active":"2024-04-17T07:33:27.573065+0000","last_became_peered":"2024-04-17T07:33:27.573065+0000","last_unstale":"2024-04-17T07:33:28.757967+0000","last_undegraded":"2024-04-17T07:33:28.757967+0000","last_fullsized":"2024-04-17T07:33:28.757967+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:11.653657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.561984+0000","last_change":"2024-04-17T07:33:27.580010+0000","last_active":"2024-04-17T07:33:28.561984+0000","last_peered":"2024-04-17T07:33:28.561984+0000","last_clean":"2024-04-17T07:33:28.561984+0000","last_became_active":"2024-04-17T07:33:27.579730+0000","last_became_peered":"2024-04-17T07:33:27.579730+0000","last_unstale":"2024-04-17T07:33:28.561984+0000","last_undegraded":"2024-04-17T07:33:28.561984+0000","last_fullsized":"2024-04-17T07:33:28.561984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:45:30.741505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759066+0000","last_change":"2024-04-17T07:33:27.576317+0000","last_active":"2024-04-17T07:33:28.759066+0000","last_peered":"2024-04-17T07:33:28.759066+0000","last_clean":"2024-04-17T07:33:28.759066+0000","last_became_active":"2024-04-17T07:33:27.575781+0000","last_became_peered":"2024-04-17T07:33:27.575781+0000","last_unstale":"2024-04-17T07:33:28.759066+0000","last_undegraded":"2024-04-17T07:33:28.759066+0000","last_fullsized":"2024-04-17T07:33:28.759066+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:58.390643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758068+0000","last_change":"2024-04-17T07:33:27.576045+0000","last_active":"2024-04-17T07:33:28.758068+0000","last_peered":"2024-04-17T07:33:28.758068+0000","last_clean":"2024-04-17T07:33:28.758068+0000","last_became_active":"2024-04-17T07:33:27.575081+0000","last_became_peered":"2024-04-17T07:33:27.575081+0000","last_unstale":"2024-04-17T07:33:28.758068+0000","last_undegraded":"2024-04-17T07:33:28.758068+0000","last_fullsized":"2024-04-17T07:33:28.758068+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:27.578908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758942+0000","last_change":"2024-04-17T07:33:27.569431+0000","last_active":"2024-04-17T07:33:28.758942+0000","last_peered":"2024-04-17T07:33:28.758942+0000","last_clean":"2024-04-17T07:33:28.758942+0000","last_became_active":"2024-04-17T07:33:27.569118+0000","last_became_peered":"2024-04-17T07:33:27.569118+0000","last_unstale":"2024-04-17T07:33:28.758942+0000","last_undegraded":"2024-04-17T07:33:28.758942+0000","last_fullsized":"2024-04-17T07:33:28.758942+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:33.574282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562542+0000","last_change":"2024-04-17T07:33:27.586836+0000","last_active":"2024-04-17T07:33:28.562542+0000","last_peered":"2024-04-17T07:33:28.562542+0000","last_clean":"2024-04-17T07:33:28.562542+0000","last_became_active":"2024-04-17T07:33:27.586576+0000","last_became_peered":"2024-04-17T07:33:27.586576+0000","last_unstale":"2024-04-17T07:33:28.562542+0000","last_undegraded":"2024-04-17T07:33:28.562542+0000","last_fullsized":"2024-04-17T07:33:28.562542+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:34.666286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758312+0000","last_change":"2024-04-17T07:33:27.574886+0000","last_active":"2024-04-17T07:33:28.758312+0000","last_peered":"2024-04-17T07:33:28.758312+0000","last_clean":"2024-04-17T07:33:28.758312+0000","last_became_active":"2024-04-17T07:33:27.574590+0000","last_became_peered":"2024-04-17T07:33:27.574590+0000","last_unstale":"2024-04-17T07:33:28.758312+0000","last_undegraded":"2024-04-17T07:33:28.758312+0000","last_fullsized":"2024-04-17T07:33:28.758312+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:43.664413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567300+0000","last_change":"2024-04-17T07:33:27.582859+0000","last_active":"2024-04-17T07:33:28.567300+0000","last_peered":"2024-04-17T07:33:28.567300+0000","last_clean":"2024-04-17T07:33:28.567300+0000","last_became_active":"2024-04-17T07:33:27.582550+0000","last_became_peered":"2024-04-17T07:33:27.582550+0000","last_unstale":"2024-04-17T07:33:28.567300+0000","last_undegraded":"2024-04-17T07:33:28.567300+0000","last_fullsized":"2024-04-17T07:33:28.567300+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:36.150233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757226+0000","last_change":"2024-04-17T07:33:27.571736+0000","last_active":"2024-04-17T07:33:28.757226+0000","last_peered":"2024-04-17T07:33:28.757226+0000","last_clean":"2024-04-17T07:33:28.757226+0000","last_became_active":"2024-04-17T07:33:27.568899+0000","last_became_peered":"2024-04-17T07:33:27.568899+0000","last_unstale":"2024-04-17T07:33:28.757226+0000","last_undegraded":"2024-04-17T07:33:28.757226+0000","last_fullsized":"2024-04-17T07:33:28.757226+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:53.305969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757104+0000","last_change":"2024-04-17T07:33:27.583936+0000","last_active":"2024-04-17T07:33:28.757104+0000","last_peered":"2024-04-17T07:33:28.757104+0000","last_clean":"2024-04-17T07:33:28.757104+0000","last_became_active":"2024-04-17T07:33:27.583653+0000","last_became_peered":"2024-04-17T07:33:27.583653+0000","last_unstale":"2024-04-17T07:33:28.757104+0000","last_undegraded":"2024-04-17T07:33:28.757104+0000","last_fullsized":"2024-04-17T07:33:28.757104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:25.990378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758338+0000","last_change":"2024-04-17T07:33:27.572126+0000","last_active":"2024-04-17T07:33:28.758338+0000","last_peered":"2024-04-17T07:33:28.758338+0000","last_clean":"2024-04-17T07:33:28.758338+0000","last_became_active":"2024-04-17T07:33:27.571857+0000","last_became_peered":"2024-04-17T07:33:27.571857+0000","last_unstale":"2024-04-17T07:33:28.758338+0000","last_undegraded":"2024-04-17T07:33:28.758338+0000","last_fullsized":"2024-04-17T07:33:28.758338+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:52.419618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"141'4","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568231+0000","last_change":"2024-04-17T07:33:26.520033+0000","last_active":"2024-04-17T07:33:28.568231+0000","last_peered":"2024-04-17T07:33:28.568231+0000","last_clean":"2024-04-17T07:33:28.568231+0000","last_became_active":"2024-04-17T07:33:26.519630+0000","last_became_peered":"2024-04-17T07:33:26.519630+0000","last_unstale":"2024-04-17T07:33:28.568231+0000","last_undegraded":"2024-04-17T07:33:28.568231+0000","last_fullsized":"2024-04-17T07:33:28.568231+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:58.827598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594672+0000","last_change":"2024-04-17T07:33:27.571039+0000","last_active":"2024-04-17T07:33:28.594672+0000","last_peered":"2024-04-17T07:33:28.594672+0000","last_clean":"2024-04-17T07:33:28.594672+0000","last_became_active":"2024-04-17T07:33:27.570721+0000","last_became_peered":"2024-04-17T07:33:27.570721+0000","last_unstale":"2024-04-17T07:33:28.594672+0000","last_undegraded":"2024-04-17T07:33:28.594672+0000","last_fullsized":"2024-04-17T07:33:28.594672+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:03.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567588+0000","last_change":"2024-04-17T07:33:26.527149+0000","last_active":"2024-04-17T07:33:28.567588+0000","last_peered":"2024-04-17T07:33:28.567588+0000","last_clean":"2024-04-17T07:33:28.567588+0000","last_became_active":"2024-04-17T07:33:26.526615+0000","last_became_peered":"2024-04-17T07:33:26.526615+0000","last_unstale":"2024-04-17T07:33:28.567588+0000","last_undegraded":"2024-04-17T07:33:28.567588+0000","last_fullsized":"2024-04-17T07:33:28.567588+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:45:29.736238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758710+0000","last_change":"2024-04-17T07:33:27.585562+0000","last_active":"2024-04-17T07:33:28.758710+0000","last_peered":"2024-04-17T07:33:28.758710+0000","last_clean":"2024-04-17T07:33:28.758710+0000","last_became_active":"2024-04-17T07:33:27.585302+0000","last_became_peered":"2024-04-17T07:33:27.585302+0000","last_unstale":"2024-04-17T07:33:28.758710+0000","last_undegraded":"2024-04-17T07:33:28.758710+0000","last_fullsized":"2024-04-17T07:33:28.758710+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:53.681324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760017+0000","last_change":"2024-04-17T07:33:26.515654+0000","last_active":"2024-04-17T07:33:28.760017+0000","last_peered":"2024-04-17T07:33:28.760017+0000","last_clean":"2024-04-17T07:33:28.760017+0000","last_became_active":"2024-04-17T07:33:26.515359+0000","last_became_peered":"2024-04-17T07:33:26.515359+0000","last_unstale":"2024-04-17T07:33:28.760017+0000","last_undegraded":"2024-04-17T07:33:28.760017+0000","last_fullsized":"2024-04-17T07:33:28.760017+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:52.977206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567763+0000","last_change":"2024-04-17T07:33:27.569020+0000","last_active":"2024-04-17T07:33:28.567763+0000","last_peered":"2024-04-17T07:33:28.567763+0000","last_clean":"2024-04-17T07:33:28.567763+0000","last_became_active":"2024-04-17T07:33:27.568740+0000","last_became_peered":"2024-04-17T07:33:27.568740+0000","last_unstale":"2024-04-17T07:33:28.567763+0000","last_undegraded":"2024-04-17T07:33:28.567763+0000","last_fullsized":"2024-04-17T07:33:28.567763+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:23.793208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760132+0000","last_change":"2024-04-17T07:33:26.516876+0000","last_active":"2024-04-17T07:33:28.760132+0000","last_peered":"2024-04-17T07:33:28.760132+0000","last_clean":"2024-04-17T07:33:28.760132+0000","last_became_active":"2024-04-17T07:33:26.515528+0000","last_became_peered":"2024-04-17T07:33:26.515528+0000","last_unstale":"2024-04-17T07:33:28.760132+0000","last_undegraded":"2024-04-17T07:33:28.760132+0000","last_fullsized":"2024-04-17T07:33:28.760132+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:53.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756998+0000","last_change":"2024-04-17T07:33:27.583685+0000","last_active":"2024-04-17T07:33:28.756998+0000","last_peered":"2024-04-17T07:33:28.756998+0000","last_clean":"2024-04-17T07:33:28.756998+0000","last_became_active":"2024-04-17T07:33:27.583415+0000","last_became_peered":"2024-04-17T07:33:27.583415+0000","last_unstale":"2024-04-17T07:33:28.756998+0000","last_undegraded":"2024-04-17T07:33:28.756998+0000","last_fullsized":"2024-04-17T07:33:28.756998+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:34.114306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567754+0000","last_change":"2024-04-17T07:33:26.515193+0000","last_active":"2024-04-17T07:33:28.567754+0000","last_peered":"2024-04-17T07:33:28.567754+0000","last_clean":"2024-04-17T07:33:28.567754+0000","last_became_active":"2024-04-17T07:33:26.514883+0000","last_became_peered":"2024-04-17T07:33:26.514883+0000","last_unstale":"2024-04-17T07:33:28.567754+0000","last_undegraded":"2024-04-17T07:33:28.567754+0000","last_fullsized":"2024-04-17T07:33:28.567754+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:18.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758346+0000","last_change":"2024-04-17T07:33:27.570290+0000","last_active":"2024-04-17T07:33:28.758346+0000","last_peered":"2024-04-17T07:33:28.758346+0000","last_clean":"2024-04-17T07:33:28.758346+0000","last_became_active":"2024-04-17T07:33:27.569990+0000","last_became_peered":"2024-04-17T07:33:27.569990+0000","last_unstale":"2024-04-17T07:33:28.758346+0000","last_undegraded":"2024-04-17T07:33:28.758346+0000","last_fullsized":"2024-04-17T07:33:28.758346+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:19.067677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562580+0000","last_change":"2024-04-17T07:33:26.523017+0000","last_active":"2024-04-17T07:33:28.562580+0000","last_peered":"2024-04-17T07:33:28.562580+0000","last_clean":"2024-04-17T07:33:28.562580+0000","last_became_active":"2024-04-17T07:33:26.522511+0000","last_became_peered":"2024-04-17T07:33:26.522511+0000","last_unstale":"2024-04-17T07:33:28.562580+0000","last_undegraded":"2024-04-17T07:33:28.562580+0000","last_fullsized":"2024-04-17T07:33:28.562580+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:19.220691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756821+0000","last_change":"2024-04-17T07:33:27.567138+0000","last_active":"2024-04-17T07:33:28.756821+0000","last_peered":"2024-04-17T07:33:28.756821+0000","last_clean":"2024-04-17T07:33:28.756821+0000","last_became_active":"2024-04-17T07:33:27.566324+0000","last_became_peered":"2024-04-17T07:33:27.566324+0000","last_unstale":"2024-04-17T07:33:28.756821+0000","last_undegraded":"2024-04-17T07:33:28.756821+0000","last_fullsized":"2024-04-17T07:33:28.756821+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:04.682658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758027+0000","last_change":"2024-04-17T07:33:26.517484+0000","last_active":"2024-04-17T07:33:28.758027+0000","last_peered":"2024-04-17T07:33:28.758027+0000","last_clean":"2024-04-17T07:33:28.758027+0000","last_became_active":"2024-04-17T07:33:26.516353+0000","last_became_peered":"2024-04-17T07:33:26.516353+0000","last_unstale":"2024-04-17T07:33:28.758027+0000","last_undegraded":"2024-04-17T07:33:28.758027+0000","last_fullsized":"2024-04-17T07:33:28.758027+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:55.881555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566128+0000","last_change":"2024-04-17T07:33:27.570261+0000","last_active":"2024-04-17T07:33:28.566128+0000","last_peered":"2024-04-17T07:33:28.566128+0000","last_clean":"2024-04-17T07:33:28.566128+0000","last_became_active":"2024-04-17T07:33:27.568399+0000","last_became_peered":"2024-04-17T07:33:27.568399+0000","last_unstale":"2024-04-17T07:33:28.566128+0000","last_undegraded":"2024-04-17T07:33:28.566128+0000","last_fullsized":"2024-04-17T07:33:28.566128+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:50.074512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756774+0000","last_change":"2024-04-17T07:33:26.512001+0000","last_active":"2024-04-17T07:33:28.756774+0000","last_peered":"2024-04-17T07:33:28.756774+0000","last_clean":"2024-04-17T07:33:28.756774+0000","last_became_active":"2024-04-17T07:33:26.511647+0000","last_became_peered":"2024-04-17T07:33:26.511647+0000","last_unstale":"2024-04-17T07:33:28.756774+0000","last_undegraded":"2024-04-17T07:33:28.756774+0000","last_fullsized":"2024-04-17T07:33:28.756774+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:00:37.696656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568038+0000","last_change":"2024-04-17T07:33:27.572498+0000","last_active":"2024-04-17T07:33:28.568038+0000","last_peered":"2024-04-17T07:33:28.568038+0000","last_clean":"2024-04-17T07:33:28.568038+0000","last_became_active":"2024-04-17T07:33:27.572185+0000","last_became_peered":"2024-04-17T07:33:27.572185+0000","last_unstale":"2024-04-17T07:33:28.568038+0000","last_undegraded":"2024-04-17T07:33:28.568038+0000","last_fullsized":"2024-04-17T07:33:28.568038+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:38.838123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759843+0000","last_change":"2024-04-17T07:33:26.517258+0000","last_active":"2024-04-17T07:33:28.759843+0000","last_peered":"2024-04-17T07:33:28.759843+0000","last_clean":"2024-04-17T07:33:28.759843+0000","last_became_active":"2024-04-17T07:33:26.516402+0000","last_became_peered":"2024-04-17T07:33:26.516402+0000","last_unstale":"2024-04-17T07:33:28.759843+0000","last_undegraded":"2024-04-17T07:33:28.759843+0000","last_fullsized":"2024-04-17T07:33:28.759843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:57.548656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594707+0000","last_change":"2024-04-17T07:33:27.579135+0000","last_active":"2024-04-17T07:33:28.594707+0000","last_peered":"2024-04-17T07:33:28.594707+0000","last_clean":"2024-04-17T07:33:28.594707+0000","last_became_active":"2024-04-17T07:33:27.578673+0000","last_became_peered":"2024-04-17T07:33:27.578673+0000","last_unstale":"2024-04-17T07:33:28.594707+0000","last_undegraded":"2024-04-17T07:33:28.594707+0000","last_fullsized":"2024-04-17T07:33:28.594707+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:52.419618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563960+0000","last_change":"2024-04-17T07:33:26.523217+0000","last_active":"2024-04-17T07:33:28.563960+0000","last_peered":"2024-04-17T07:33:28.563960+0000","last_clean":"2024-04-17T07:33:28.563960+0000","last_became_active":"2024-04-17T07:33:26.522701+0000","last_became_peered":"2024-04-17T07:33:26.522701+0000","last_unstale":"2024-04-17T07:33:28.563960+0000","last_undegraded":"2024-04-17T07:33:28.563960+0000","last_fullsized":"2024-04-17T07:33:28.563960+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:04.433074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562944+0000","last_change":"2024-04-17T07:33:27.584002+0000","last_active":"2024-04-17T07:33:28.562944+0000","last_peered":"2024-04-17T07:33:28.562944+0000","last_clean":"2024-04-17T07:33:28.562944+0000","last_became_active":"2024-04-17T07:33:27.583246+0000","last_became_peered":"2024-04-17T07:33:27.583246+0000","last_unstale":"2024-04-17T07:33:28.562944+0000","last_undegraded":"2024-04-17T07:33:28.562944+0000","last_fullsized":"2024-04-17T07:33:28.562944+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:06:15.984794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"141'3","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757821+0000","last_change":"2024-04-17T07:33:26.520309+0000","last_active":"2024-04-17T07:33:28.757821+0000","last_peered":"2024-04-17T07:33:28.757821+0000","last_clean":"2024-04-17T07:33:28.757821+0000","last_became_active":"2024-04-17T07:33:26.520006+0000","last_became_peered":"2024-04-17T07:33:26.520006+0000","last_unstale":"2024-04-17T07:33:28.757821+0000","last_undegraded":"2024-04-17T07:33:28.757821+0000","last_fullsized":"2024-04-17T07:33:28.757821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:21:06.788744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566856+0000","last_change":"2024-04-17T07:33:27.552575+0000","last_active":"2024-04-17T07:33:28.566856+0000","last_peered":"2024-04-17T07:33:28.566856+0000","last_clean":"2024-04-17T07:33:28.566856+0000","last_became_active":"2024-04-17T07:33:27.551759+0000","last_became_peered":"2024-04-17T07:33:27.551759+0000","last_unstale":"2024-04-17T07:33:28.566856+0000","last_undegraded":"2024-04-17T07:33:28.566856+0000","last_fullsized":"2024-04-17T07:33:28.566856+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:35.745460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"141'6","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758488+0000","last_change":"2024-04-17T07:33:26.498325+0000","last_active":"2024-04-17T07:33:28.758488+0000","last_peered":"2024-04-17T07:33:28.758488+0000","last_clean":"2024-04-17T07:33:28.758488+0000","last_became_active":"2024-04-17T07:33:26.498019+0000","last_became_peered":"2024-04-17T07:33:26.498019+0000","last_unstale":"2024-04-17T07:33:28.758488+0000","last_undegraded":"2024-04-17T07:33:28.758488+0000","last_fullsized":"2024-04-17T07:33:28.758488+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:02:47.669089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562684+0000","last_change":"2024-04-17T07:33:27.580429+0000","last_active":"2024-04-17T07:33:28.562684+0000","last_peered":"2024-04-17T07:33:28.562684+0000","last_clean":"2024-04-17T07:33:28.562684+0000","last_became_active":"2024-04-17T07:33:27.579746+0000","last_became_peered":"2024-04-17T07:33:27.579746+0000","last_unstale":"2024-04-17T07:33:28.562684+0000","last_undegraded":"2024-04-17T07:33:28.562684+0000","last_fullsized":"2024-04-17T07:33:28.562684+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:27.405509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758474+0000","last_change":"2024-04-17T07:33:26.527902+0000","last_active":"2024-04-17T07:33:28.758474+0000","last_peered":"2024-04-17T07:33:28.758474+0000","last_clean":"2024-04-17T07:33:28.758474+0000","last_became_active":"2024-04-17T07:33:26.527105+0000","last_became_peered":"2024-04-17T07:33:26.527105+0000","last_unstale":"2024-04-17T07:33:28.758474+0000","last_undegraded":"2024-04-17T07:33:28.758474+0000","last_fullsized":"2024-04-17T07:33:28.758474+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:37.065496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563596+0000","last_change":"2024-04-17T07:33:27.577044+0000","last_active":"2024-04-17T07:33:28.563596+0000","last_peered":"2024-04-17T07:33:28.563596+0000","last_clean":"2024-04-17T07:33:28.563596+0000","last_became_active":"2024-04-17T07:33:27.576715+0000","last_became_peered":"2024-04-17T07:33:27.576715+0000","last_unstale":"2024-04-17T07:33:28.563596+0000","last_undegraded":"2024-04-17T07:33:28.563596+0000","last_fullsized":"2024-04-17T07:33:28.563596+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:03.219137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594040+0000","last_change":"2024-04-17T07:33:26.526646+0000","last_active":"2024-04-17T07:33:28.594040+0000","last_peered":"2024-04-17T07:33:28.594040+0000","last_clean":"2024-04-17T07:33:28.594040+0000","last_became_active":"2024-04-17T07:33:26.526283+0000","last_became_peered":"2024-04-17T07:33:26.526283+0000","last_unstale":"2024-04-17T07:33:28.594040+0000","last_undegraded":"2024-04-17T07:33:28.594040+0000","last_fullsized":"2024-04-17T07:33:28.594040+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:37.065496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562468+0000","last_change":"2024-04-17T07:33:27.576979+0000","last_active":"2024-04-17T07:33:28.562468+0000","last_peered":"2024-04-17T07:33:28.562468+0000","last_clean":"2024-04-17T07:33:28.562468+0000","last_became_active":"2024-04-17T07:33:27.576579+0000","last_became_peered":"2024-04-17T07:33:27.576579+0000","last_unstale":"2024-04-17T07:33:28.562468+0000","last_undegraded":"2024-04-17T07:33:28.562468+0000","last_fullsized":"2024-04-17T07:33:28.562468+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:15:11.262084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759566+0000","last_change":"2024-04-17T07:33:26.520015+0000","last_active":"2024-04-17T07:33:28.759566+0000","last_peered":"2024-04-17T07:33:28.759566+0000","last_clean":"2024-04-17T07:33:28.759566+0000","last_became_active":"2024-04-17T07:33:26.519516+0000","last_became_peered":"2024-04-17T07:33:26.519516+0000","last_unstale":"2024-04-17T07:33:28.759566+0000","last_undegraded":"2024-04-17T07:33:28.759566+0000","last_fullsized":"2024-04-17T07:33:28.759566+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:02:47.669089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760375+0000","last_change":"2024-04-17T07:33:27.565708+0000","last_active":"2024-04-17T07:33:28.760375+0000","last_peered":"2024-04-17T07:33:28.760375+0000","last_clean":"2024-04-17T07:33:28.760375+0000","last_became_active":"2024-04-17T07:33:27.565421+0000","last_became_peered":"2024-04-17T07:33:27.565421+0000","last_unstale":"2024-04-17T07:33:28.760375+0000","last_undegraded":"2024-04-17T07:33:28.760375+0000","last_fullsized":"2024-04-17T07:33:28.760375+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:32.623989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759549+0000","last_change":"2024-04-17T07:33:26.523746+0000","last_active":"2024-04-17T07:33:28.759549+0000","last_peered":"2024-04-17T07:33:28.759549+0000","last_clean":"2024-04-17T07:33:28.759549+0000","last_became_active":"2024-04-17T07:33:26.523456+0000","last_became_peered":"2024-04-17T07:33:26.523456+0000","last_unstale":"2024-04-17T07:33:28.759549+0000","last_undegraded":"2024-04-17T07:33:28.759549+0000","last_fullsized":"2024-04-17T07:33:28.759549+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:47.626837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758424+0000","last_change":"2024-04-17T07:33:27.570319+0000","last_active":"2024-04-17T07:33:28.758424+0000","last_peered":"2024-04-17T07:33:28.758424+0000","last_clean":"2024-04-17T07:33:28.758424+0000","last_became_active":"2024-04-17T07:33:27.570024+0000","last_became_peered":"2024-04-17T07:33:27.570024+0000","last_unstale":"2024-04-17T07:33:28.758424+0000","last_undegraded":"2024-04-17T07:33:28.758424+0000","last_fullsized":"2024-04-17T07:33:28.758424+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:08.123232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760221+0000","last_change":"2024-04-17T07:33:26.514453+0000","last_active":"2024-04-17T07:33:28.760221+0000","last_peered":"2024-04-17T07:33:28.760221+0000","last_clean":"2024-04-17T07:33:28.760221+0000","last_became_active":"2024-04-17T07:33:26.512214+0000","last_became_peered":"2024-04-17T07:33:26.512214+0000","last_unstale":"2024-04-17T07:33:28.760221+0000","last_undegraded":"2024-04-17T07:33:28.760221+0000","last_fullsized":"2024-04-17T07:33:28.760221+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:36.359971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567107+0000","last_change":"2024-04-17T07:33:27.568901+0000","last_active":"2024-04-17T07:33:28.567107+0000","last_peered":"2024-04-17T07:33:28.567107+0000","last_clean":"2024-04-17T07:33:28.567107+0000","last_became_active":"2024-04-17T07:33:27.567363+0000","last_became_peered":"2024-04-17T07:33:27.567363+0000","last_unstale":"2024-04-17T07:33:28.567107+0000","last_undegraded":"2024-04-17T07:33:28.567107+0000","last_fullsized":"2024-04-17T07:33:28.567107+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:31:36.204058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594379+0000","last_change":"2024-04-17T07:33:26.517616+0000","last_active":"2024-04-17T07:33:28.594379+0000","last_peered":"2024-04-17T07:33:28.594379+0000","last_clean":"2024-04-17T07:33:28.594379+0000","last_became_active":"2024-04-17T07:33:26.517026+0000","last_became_peered":"2024-04-17T07:33:26.517026+0000","last_unstale":"2024-04-17T07:33:28.594379+0000","last_undegraded":"2024-04-17T07:33:28.594379+0000","last_fullsized":"2024-04-17T07:33:28.594379+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:21:30.056993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567888+0000","last_change":"2024-04-17T07:33:27.553105+0000","last_active":"2024-04-17T07:33:28.567888+0000","last_peered":"2024-04-17T07:33:28.567888+0000","last_clean":"2024-04-17T07:33:28.567888+0000","last_became_active":"2024-04-17T07:33:27.552366+0000","last_became_peered":"2024-04-17T07:33:27.552366+0000","last_unstale":"2024-04-17T07:33:28.567888+0000","last_undegraded":"2024-04-17T07:33:28.567888+0000","last_fullsized":"2024-04-17T07:33:28.567888+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:02.436650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.565732+0000","last_change":"2024-04-17T07:33:26.523742+0000","last_active":"2024-04-17T07:33:28.565732+0000","last_peered":"2024-04-17T07:33:28.565732+0000","last_clean":"2024-04-17T07:33:28.565732+0000","last_became_active":"2024-04-17T07:33:26.523369+0000","last_became_peered":"2024-04-17T07:33:26.523369+0000","last_unstale":"2024-04-17T07:33:28.565732+0000","last_undegraded":"2024-04-17T07:33:28.565732+0000","last_fullsized":"2024-04-17T07:33:28.565732+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:42.659145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566455+0000","last_change":"2024-04-17T07:33:27.573401+0000","last_active":"2024-04-17T07:33:28.566455+0000","last_peered":"2024-04-17T07:33:28.566455+0000","last_clean":"2024-04-17T07:33:28.566455+0000","last_became_active":"2024-04-17T07:33:27.573067+0000","last_became_peered":"2024-04-17T07:33:27.573067+0000","last_unstale":"2024-04-17T07:33:28.566455+0000","last_undegraded":"2024-04-17T07:33:28.566455+0000","last_fullsized":"2024-04-17T07:33:28.566455+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:28.864383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758594+0000","last_change":"2024-04-17T07:33:26.526987+0000","last_active":"2024-04-17T07:33:28.758594+0000","last_peered":"2024-04-17T07:33:28.758594+0000","last_clean":"2024-04-17T07:33:28.758594+0000","last_became_active":"2024-04-17T07:33:26.526684+0000","last_became_peered":"2024-04-17T07:33:26.526684+0000","last_unstale":"2024-04-17T07:33:28.758594+0000","last_undegraded":"2024-04-17T07:33:28.758594+0000","last_fullsized":"2024-04-17T07:33:28.758594+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:21:30.056993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757120+0000","last_change":"2024-04-17T07:33:27.576417+0000","last_active":"2024-04-17T07:33:28.757120+0000","last_peered":"2024-04-17T07:33:28.757120+0000","last_clean":"2024-04-17T07:33:28.757120+0000","last_became_active":"2024-04-17T07:33:27.575741+0000","last_became_peered":"2024-04-17T07:33:27.575741+0000","last_unstale":"2024-04-17T07:33:28.757120+0000","last_undegraded":"2024-04-17T07:33:28.757120+0000","last_fullsized":"2024-04-17T07:33:28.757120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:06.419086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759863+0000","last_change":"2024-04-17T07:33:26.518745+0000","last_active":"2024-04-17T07:33:28.759863+0000","last_peered":"2024-04-17T07:33:28.759863+0000","last_clean":"2024-04-17T07:33:28.759863+0000","last_became_active":"2024-04-17T07:33:26.518435+0000","last_became_peered":"2024-04-17T07:33:26.518435+0000","last_unstale":"2024-04-17T07:33:28.759863+0000","last_undegraded":"2024-04-17T07:33:28.759863+0000","last_fullsized":"2024-04-17T07:33:28.759863+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:21:06.788744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.565099+0000","last_change":"2024-04-17T07:33:27.570379+0000","last_active":"2024-04-17T07:33:28.565099+0000","last_peered":"2024-04-17T07:33:28.565099+0000","last_clean":"2024-04-17T07:33:28.565099+0000","last_became_active":"2024-04-17T07:33:27.570046+0000","last_became_peered":"2024-04-17T07:33:27.570046+0000","last_unstale":"2024-04-17T07:33:28.565099+0000","last_undegraded":"2024-04-17T07:33:28.565099+0000","last_fullsized":"2024-04-17T07:33:28.565099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:56:09.759290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757368+0000","last_change":"2024-04-17T07:33:26.526909+0000","last_active":"2024-04-17T07:33:28.757368+0000","last_peered":"2024-04-17T07:33:28.757368+0000","last_clean":"2024-04-17T07:33:28.757368+0000","last_became_active":"2024-04-17T07:33:26.526600+0000","last_became_peered":"2024-04-17T07:33:26.526600+0000","last_unstale":"2024-04-17T07:33:28.757368+0000","last_undegraded":"2024-04-17T07:33:28.757368+0000","last_fullsized":"2024-04-17T07:33:28.757368+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:22:26.699722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562889+0000","last_change":"2024-04-17T07:33:27.579248+0000","last_active":"2024-04-17T07:33:28.562889+0000","last_peered":"2024-04-17T07:33:28.562889+0000","last_clean":"2024-04-17T07:33:28.562889+0000","last_became_active":"2024-04-17T07:33:27.578961+0000","last_became_peered":"2024-04-17T07:33:27.578961+0000","last_unstale":"2024-04-17T07:33:28.562889+0000","last_undegraded":"2024-04-17T07:33:28.562889+0000","last_fullsized":"2024-04-17T07:33:28.562889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:35:42.086755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756911+0000","last_change":"2024-04-17T07:33:26.522674+0000","last_active":"2024-04-17T07:33:28.756911+0000","last_peered":"2024-04-17T07:33:28.756911+0000","last_clean":"2024-04-17T07:33:28.756911+0000","last_became_active":"2024-04-17T07:33:26.522210+0000","last_became_peered":"2024-04-17T07:33:26.522210+0000","last_unstale":"2024-04-17T07:33:28.756911+0000","last_undegraded":"2024-04-17T07:33:28.756911+0000","last_fullsized":"2024-04-17T07:33:28.756911+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:57.458153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594402+0000","last_change":"2024-04-17T07:33:27.574837+0000","last_active":"2024-04-17T07:33:28.594402+0000","last_peered":"2024-04-17T07:33:28.594402+0000","last_clean":"2024-04-17T07:33:28.594402+0000","last_became_active":"2024-04-17T07:33:27.574517+0000","last_became_peered":"2024-04-17T07:33:27.574517+0000","last_unstale":"2024-04-17T07:33:28.594402+0000","last_undegraded":"2024-04-17T07:33:28.594402+0000","last_fullsized":"2024-04-17T07:33:28.594402+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:25.990378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563564+0000","last_change":"2024-04-17T07:33:26.498707+0000","last_active":"2024-04-17T07:33:28.563564+0000","last_peered":"2024-04-17T07:33:28.563564+0000","last_clean":"2024-04-17T07:33:28.563564+0000","last_became_active":"2024-04-17T07:33:26.497493+0000","last_became_peered":"2024-04-17T07:33:26.497493+0000","last_unstale":"2024-04-17T07:33:28.563564+0000","last_undegraded":"2024-04-17T07:33:28.563564+0000","last_fullsized":"2024-04-17T07:33:28.563564+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:35.768018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759410+0000","last_change":"2024-04-17T07:33:27.581353+0000","last_active":"2024-04-17T07:33:28.759410+0000","last_peered":"2024-04-17T07:33:28.759410+0000","last_clean":"2024-04-17T07:33:28.759410+0000","last_became_active":"2024-04-17T07:33:27.581055+0000","last_became_peered":"2024-04-17T07:33:27.581055+0000","last_unstale":"2024-04-17T07:33:28.759410+0000","last_undegraded":"2024-04-17T07:33:28.759410+0000","last_fullsized":"2024-04-17T07:33:28.759410+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:23:49.773528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"142'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.598360+0000","last_change":"2024-04-17T07:33:26.521281+0000","last_active":"2024-04-17T07:33:28.598360+0000","last_peered":"2024-04-17T07:33:28.598360+0000","last_clean":"2024-04-17T07:33:28.598360+0000","last_became_active":"2024-04-17T07:33:26.518369+0000","last_became_peered":"2024-04-17T07:33:26.518369+0000","last_unstale":"2024-04-17T07:33:28.598360+0000","last_undegraded":"2024-04-17T07:33:28.598360+0000","last_fullsized":"2024-04-17T07:33:28.598360+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:26.993104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758808+0000","last_change":"2024-04-17T07:33:27.576584+0000","last_active":"2024-04-17T07:33:28.758808+0000","last_peered":"2024-04-17T07:33:28.758808+0000","last_clean":"2024-04-17T07:33:28.758808+0000","last_became_active":"2024-04-17T07:33:27.576287+0000","last_became_peered":"2024-04-17T07:33:27.576287+0000","last_unstale":"2024-04-17T07:33:28.758808+0000","last_undegraded":"2024-04-17T07:33:28.758808+0000","last_fullsized":"2024-04-17T07:33:28.758808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:38.531544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757977+0000","last_change":"2024-04-17T07:33:26.520712+0000","last_active":"2024-04-17T07:33:28.757977+0000","last_peered":"2024-04-17T07:33:28.757977+0000","last_clean":"2024-04-17T07:33:28.757977+0000","last_became_active":"2024-04-17T07:33:26.520387+0000","last_became_peered":"2024-04-17T07:33:26.520387+0000","last_unstale":"2024-04-17T07:33:28.757977+0000","last_undegraded":"2024-04-17T07:33:28.757977+0000","last_fullsized":"2024-04-17T07:33:28.757977+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:57.548656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594079+0000","last_change":"2024-04-17T07:33:27.570953+0000","last_active":"2024-04-17T07:33:28.594079+0000","last_peered":"2024-04-17T07:33:28.594079+0000","last_clean":"2024-04-17T07:33:28.594079+0000","last_became_active":"2024-04-17T07:33:27.570661+0000","last_became_peered":"2024-04-17T07:33:27.570661+0000","last_unstale":"2024-04-17T07:33:28.594079+0000","last_undegraded":"2024-04-17T07:33:28.594079+0000","last_fullsized":"2024-04-17T07:33:28.594079+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:52.395819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562767+0000","last_change":"2024-04-17T07:33:26.523984+0000","last_active":"2024-04-17T07:33:28.562767+0000","last_peered":"2024-04-17T07:33:28.562767+0000","last_clean":"2024-04-17T07:33:28.562767+0000","last_became_active":"2024-04-17T07:33:26.523666+0000","last_became_peered":"2024-04-17T07:33:26.523666+0000","last_unstale":"2024-04-17T07:33:28.562767+0000","last_undegraded":"2024-04-17T07:33:28.562767+0000","last_fullsized":"2024-04-17T07:33:28.562767+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:55:49.619601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758019+0000","last_change":"2024-04-17T07:33:27.567593+0000","last_active":"2024-04-17T07:33:28.758019+0000","last_peered":"2024-04-17T07:33:28.758019+0000","last_clean":"2024-04-17T07:33:28.758019+0000","last_became_active":"2024-04-17T07:33:27.560647+0000","last_became_peered":"2024-04-17T07:33:27.560647+0000","last_unstale":"2024-04-17T07:33:28.758019+0000","last_undegraded":"2024-04-17T07:33:28.758019+0000","last_fullsized":"2024-04-17T07:33:28.758019+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:23.389879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568010+0000","last_change":"2024-04-17T07:33:26.517942+0000","last_active":"2024-04-17T07:33:28.568010+0000","last_peered":"2024-04-17T07:33:28.568010+0000","last_clean":"2024-04-17T07:33:28.568010+0000","last_became_active":"2024-04-17T07:33:26.517627+0000","last_became_peered":"2024-04-17T07:33:26.517627+0000","last_unstale":"2024-04-17T07:33:28.568010+0000","last_undegraded":"2024-04-17T07:33:28.568010+0000","last_fullsized":"2024-04-17T07:33:28.568010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:33.406288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758736+0000","last_change":"2024-04-17T07:33:27.574379+0000","last_active":"2024-04-17T07:33:28.758736+0000","last_peered":"2024-04-17T07:33:28.758736+0000","last_clean":"2024-04-17T07:33:28.758736+0000","last_became_active":"2024-04-17T07:33:27.571823+0000","last_became_peered":"2024-04-17T07:33:27.571823+0000","last_unstale":"2024-04-17T07:33:28.758736+0000","last_undegraded":"2024-04-17T07:33:28.758736+0000","last_fullsized":"2024-04-17T07:33:28.758736+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:59.832865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758079+0000","last_change":"2024-04-17T07:33:26.514630+0000","last_active":"2024-04-17T07:33:28.758079+0000","last_peered":"2024-04-17T07:33:28.758079+0000","last_clean":"2024-04-17T07:33:28.758079+0000","last_became_active":"2024-04-17T07:33:26.514099+0000","last_became_peered":"2024-04-17T07:33:26.514099+0000","last_unstale":"2024-04-17T07:33:28.758079+0000","last_undegraded":"2024-04-17T07:33:28.758079+0000","last_fullsized":"2024-04-17T07:33:28.758079+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:07.211383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566110+0000","last_change":"2024-04-17T07:33:27.566792+0000","last_active":"2024-04-17T07:33:28.566110+0000","last_peered":"2024-04-17T07:33:28.566110+0000","last_clean":"2024-04-17T07:33:28.566110+0000","last_became_active":"2024-04-17T07:33:27.565484+0000","last_became_peered":"2024-04-17T07:33:27.565484+0000","last_unstale":"2024-04-17T07:33:28.566110+0000","last_undegraded":"2024-04-17T07:33:28.566110+0000","last_fullsized":"2024-04-17T07:33:28.566110+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:57.245452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759620+0000","last_change":"2024-04-17T07:33:26.517611+0000","last_active":"2024-04-17T07:33:28.759620+0000","last_peered":"2024-04-17T07:33:28.759620+0000","last_clean":"2024-04-17T07:33:28.759620+0000","last_became_active":"2024-04-17T07:33:26.517337+0000","last_became_peered":"2024-04-17T07:33:26.517337+0000","last_unstale":"2024-04-17T07:33:28.759620+0000","last_undegraded":"2024-04-17T07:33:28.759620+0000","last_fullsized":"2024-04-17T07:33:28.759620+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:47.626837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756927+0000","last_change":"2024-04-17T07:33:27.572048+0000","last_active":"2024-04-17T07:33:28.756927+0000","last_peered":"2024-04-17T07:33:28.756927+0000","last_clean":"2024-04-17T07:33:28.756927+0000","last_became_active":"2024-04-17T07:33:27.568829+0000","last_became_peered":"2024-04-17T07:33:27.568829+0000","last_unstale":"2024-04-17T07:33:28.756927+0000","last_undegraded":"2024-04-17T07:33:28.756927+0000","last_fullsized":"2024-04-17T07:33:28.756927+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:12.261602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757083+0000","last_change":"2024-04-17T07:33:26.521478+0000","last_active":"2024-04-17T07:33:28.757083+0000","last_peered":"2024-04-17T07:33:28.757083+0000","last_clean":"2024-04-17T07:33:28.757083+0000","last_became_active":"2024-04-17T07:33:26.519813+0000","last_became_peered":"2024-04-17T07:33:26.519813+0000","last_unstale":"2024-04-17T07:33:28.757083+0000","last_undegraded":"2024-04-17T07:33:28.757083+0000","last_fullsized":"2024-04-17T07:33:28.757083+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:52.977206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562433+0000","last_change":"2024-04-17T07:33:27.575375+0000","last_active":"2024-04-17T07:33:28.562433+0000","last_peered":"2024-04-17T07:33:28.562433+0000","last_clean":"2024-04-17T07:33:28.562433+0000","last_became_active":"2024-04-17T07:33:27.575054+0000","last_became_peered":"2024-04-17T07:33:27.575054+0000","last_unstale":"2024-04-17T07:33:28.562433+0000","last_undegraded":"2024-04-17T07:33:28.562433+0000","last_fullsized":"2024-04-17T07:33:28.562433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:12.276598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568338+0000","last_change":"2024-04-17T07:33:26.523730+0000","last_active":"2024-04-17T07:33:28.568338+0000","last_peered":"2024-04-17T07:33:28.568338+0000","last_clean":"2024-04-17T07:33:28.568338+0000","last_became_active":"2024-04-17T07:33:26.523341+0000","last_became_peered":"2024-04-17T07:33:26.523341+0000","last_unstale":"2024-04-17T07:33:28.568338+0000","last_undegraded":"2024-04-17T07:33:28.568338+0000","last_fullsized":"2024-04-17T07:33:28.568338+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:22.384611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"130'233","reported_seq":816,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566010+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:33:28.566010+0000","last_peered":"2024-04-17T07:33:28.566010+0000","last_clean":"2024-04-17T07:33:28.566010+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:33:28.566010+0000","last_undegraded":"2024-04-17T07:33:28.566010+0000","last_fullsized":"2024-04-17T07:33:28.566010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":367,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":367,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542376,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2359296,"data_stored":1435796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542376,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1748992,"data_stored":838554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542375,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1753088,"data_stored":839088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542375,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59272,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312568,"statfs":{"total":96636764160,"available":96576069632,"internally_reserved":0,"allocated":1757184,"data_stored":844812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542377,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":840166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":1744896,"data_stored":838484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542377,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55120,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316720,"statfs":{"total":96636764160,"available":96580321280,"internally_reserved":0,"allocated":1765376,"data_stored":844936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2359296,"data_stored":1430576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:33:32.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:33:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.346+0000 7f89e1200640 1 Processor -- start 2024-04-17T07:33:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.346+0000 7f89e1200640 1 -- start start 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.347+0000 7f89e1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89dc06d8a0 0x7f89dc09b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.347+0000 7f89e1200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89dc06ff60 0x7f89dc09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.347+0000 7f89e1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 0x7f89dc09afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.347+0000 7f89e1200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc0d1cd0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89e1200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc09bd40 con 0x7f89dc06ff60 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89e1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89dc0cd770 con 0x7f89dc06ca70 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89dc06d8a0 0x7f89dc09b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89dc06d8a0 0x7f89dc09b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56806/0 (socket says 172.21.15.67:56806) 2024-04-17T07:33:32.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 -- 172.21.15.67:0/601129136 learned_addr learned my addr 172.21.15.67:0/601129136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:32.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89daa00640 1 --2- 172.21.15.67:0/601129136 >> v2:172.21.15.187:3300/0 conn(0x7f89dc06ff60 0x7f89dc09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89dbe00640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 0x7f89dc09afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 -- 172.21.15.67:0/601129136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 msgr2=0x7f89dc09afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 0x7f89dc09afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 -- 172.21.15.67:0/601129136 >> v2:172.21.15.187:3300/0 conn(0x7f89dc06ff60 msgr2=0x7f89dc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 --2- 172.21.15.67:0/601129136 >> v2:172.21.15.187:3300/0 conn(0x7f89dc06ff60 0x7f89dc09aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89db400640 1 -- 172.21.15.67:0/601129136 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc0cce40 con 0x7f89dc06d8a0 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.348+0000 7f89dbe00640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 0x7f89dc09afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.349+0000 7f89db400640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc09b800 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06f850 tx=0x7f89cc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7efa4d0ccc7e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.350+0000 7f89da000640 1 -- 172.21.15.67:0/601129136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc034070 con 0x7f89dc06d8a0 2024-04-17T07:33:32.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.350+0000 7f89da000640 1 -- 172.21.15.67:0/601129136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc030850 con 0x7f89dc06d8a0 2024-04-17T07:33:32.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.350+0000 7f89da000640 1 -- 172.21.15.67:0/601129136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc030c20 con 0x7f89dc06d8a0 2024-04-17T07:33:32.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 -- 172.21.15.67:0/601129136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 msgr2=0x7f89dc09b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc09b800 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f89dc06f850 tx=0x7f89cc02b990 comp rx=0 tx=0).stop 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 -- 172.21.15.67:0/601129136 shutdown_connections 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ca70 0x7f89dc09afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 --2- 172.21.15.67:0/601129136 >> v2:172.21.15.187:3300/0 conn(0x7f89dc06ff60 0x7f89dc09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 --2- 172.21.15.67:0/601129136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc09b800 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 -- 172.21.15.67:0/601129136 >> 172.21.15.67:0/601129136 conn(0x7f89dc0b4140 msgr2=0x7f89dc0b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 -- 172.21.15.67:0/601129136 shutdown_connections 2024-04-17T07:33:32.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.351+0000 7f89e1200640 1 -- 172.21.15.67:0/601129136 wait complete. 2024-04-17T07:33:32.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.353+0000 7f89e1200640 1 Processor -- start 2024-04-17T07:33:32.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.353+0000 7f89e1200640 1 -- start start 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.354+0000 7f89e1200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 0x7f89dc14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.354+0000 7f89e1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.354+0000 7f89e1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ff60 0x7f89dc158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89e1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc06dc80 con 0x7f89dc06d8a0 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89e1200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc0cd400 con 0x7f89dc06ca70 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89e1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89dc06de00 con 0x7f89dc06ff60 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56820/0 (socket says 172.21.15.67:56820) 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 -- 172.21.15.67:0/2097858109 learned_addr learned my addr 172.21.15.67:0/2097858109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89db400640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 0x7f89dc14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 -- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ff60 msgr2=0x7f89dc158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89dbe00640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ff60 0x7f89dc158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ff60 0x7f89dc158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 -- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 msgr2=0x7f89dc14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 0x7f89dc14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89daa00640 1 -- 172.21.15.67:0/2097858109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc158620 con 0x7f89dc06d8a0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.355+0000 7f89db400640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 0x7f89dc14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89daa00640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c5f0 tx=0x7f89c800cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8015070 con 0x7f89dc06d8a0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c8004220 con 0x7f89dc06d8a0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8003610 con 0x7f89dc06d8a0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc0b75a0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.356+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1588c0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.357+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc0755d0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.358+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f89c80037b0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.359+0000 7f89d3400640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 0x7f89bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.359+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f89c808bbd0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.360+0000 7f89db400640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 0x7f89bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.361+0000 7f89db400640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 0x7f89bc06eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89cc036000 tx=0x7f89cc03b040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.364+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f89c8059dc0 con 0x7f89dc06d8a0 2024-04-17T07:33:32.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.568+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 --> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89dc0c9200 con 0x7f89bc06ca10 2024-04-17T07:33:32.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.587+0000 7f89d3400640 1 -- 172.21.15.67:0/2097858109 <== mgr.6221 v2:172.21.15.187:6832/1274719646 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252484 (secure 0 0 0) 0x7f89dc0c9200 con 0x7f89bc06ca10 2024-04-17T07:33:32.590 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:32.593 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:33:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 msgr2=0x7f89bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 0x7f89bc06eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89cc036000 tx=0x7f89cc03b040 comp rx=0 tx=0).stop 2024-04-17T07:33:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 msgr2=0x7f89dc0b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c5f0 tx=0x7f89c800cab0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 shutdown_connections 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f89bc06ca10 0x7f89bc06eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89dc06ff60 0x7f89dc158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc06d8a0 0x7f89dc0b6e30 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 --2- 172.21.15.67:0/2097858109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89dc06ca70 0x7f89dc14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.594+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 >> 172.21.15.67:0/2097858109 conn(0x7f89dc0b4140 msgr2=0x7f89dc0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:32.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.595+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 shutdown_connections 2024-04-17T07:33:32.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.595+0000 7f89e1200640 1 -- 172.21.15.67:0/2097858109 wait complete. 2024-04-17T07:33:32.614 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-17T07:33:31.947704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":412,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":355964,"kb_used_data":14892,"kb_used_omap":12,"kb_used_meta":340915,"kb_avail":754618756,"statfs":{"total":773094113280,"available":772729606144,"internally_reserved":0,"allocated":15249408,"data_stored":7912412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":349097520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002315"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759420+0000","last_change":"2024-04-17T07:33:27.582626+0000","last_active":"2024-04-17T07:33:28.759420+0000","last_peered":"2024-04-17T07:33:28.759420+0000","last_clean":"2024-04-17T07:33:28.759420+0000","last_became_active":"2024-04-17T07:33:27.580553+0000","last_became_peered":"2024-04-17T07:33:27.580553+0000","last_unstale":"2024-04-17T07:33:28.759420+0000","last_undegraded":"2024-04-17T07:33:28.759420+0000","last_fullsized":"2024-04-17T07:33:28.759420+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:19.845800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759029+0000","last_change":"2024-04-17T07:33:27.573363+0000","last_active":"2024-04-17T07:33:28.759029+0000","last_peered":"2024-04-17T07:33:28.759029+0000","last_clean":"2024-04-17T07:33:28.759029+0000","last_became_active":"2024-04-17T07:33:27.572971+0000","last_became_peered":"2024-04-17T07:33:27.572971+0000","last_unstale":"2024-04-17T07:33:28.759029+0000","last_undegraded":"2024-04-17T07:33:28.759029+0000","last_fullsized":"2024-04-17T07:33:28.759029+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:36:00.821543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567373+0000","last_change":"2024-04-17T07:33:27.574591+0000","last_active":"2024-04-17T07:33:28.567373+0000","last_peered":"2024-04-17T07:33:28.567373+0000","last_clean":"2024-04-17T07:33:28.567373+0000","last_became_active":"2024-04-17T07:33:27.573462+0000","last_became_peered":"2024-04-17T07:33:27.573462+0000","last_unstale":"2024-04-17T07:33:28.567373+0000","last_undegraded":"2024-04-17T07:33:28.567373+0000","last_fullsized":"2024-04-17T07:33:28.567373+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:39.157892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757942+0000","last_change":"2024-04-17T07:33:27.581825+0000","last_active":"2024-04-17T07:33:28.757942+0000","last_peered":"2024-04-17T07:33:28.757942+0000","last_clean":"2024-04-17T07:33:28.757942+0000","last_became_active":"2024-04-17T07:33:27.580867+0000","last_became_peered":"2024-04-17T07:33:27.580867+0000","last_unstale":"2024-04-17T07:33:28.757942+0000","last_undegraded":"2024-04-17T07:33:28.757942+0000","last_fullsized":"2024-04-17T07:33:28.757942+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:27:26.309376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760366+0000","last_change":"2024-04-17T07:33:27.577742+0000","last_active":"2024-04-17T07:33:28.760366+0000","last_peered":"2024-04-17T07:33:28.760366+0000","last_clean":"2024-04-17T07:33:28.760366+0000","last_became_active":"2024-04-17T07:33:27.577467+0000","last_became_peered":"2024-04-17T07:33:27.577467+0000","last_unstale":"2024-04-17T07:33:28.760366+0000","last_undegraded":"2024-04-17T07:33:28.760366+0000","last_fullsized":"2024-04-17T07:33:28.760366+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:53.801946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759012+0000","last_change":"2024-04-17T07:33:27.584765+0000","last_active":"2024-04-17T07:33:28.759012+0000","last_peered":"2024-04-17T07:33:28.759012+0000","last_clean":"2024-04-17T07:33:28.759012+0000","last_became_active":"2024-04-17T07:33:27.583713+0000","last_became_peered":"2024-04-17T07:33:27.583713+0000","last_unstale":"2024-04-17T07:33:28.759012+0000","last_undegraded":"2024-04-17T07:33:28.759012+0000","last_fullsized":"2024-04-17T07:33:28.759012+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:55.750226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566108+0000","last_change":"2024-04-17T07:33:27.564544+0000","last_active":"2024-04-17T07:33:28.566108+0000","last_peered":"2024-04-17T07:33:28.566108+0000","last_clean":"2024-04-17T07:33:28.566108+0000","last_became_active":"2024-04-17T07:33:27.564204+0000","last_became_peered":"2024-04-17T07:33:27.564204+0000","last_unstale":"2024-04-17T07:33:28.566108+0000","last_undegraded":"2024-04-17T07:33:28.566108+0000","last_fullsized":"2024-04-17T07:33:28.566108+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:03.766352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.564918+0000","last_change":"2024-04-17T07:33:27.551547+0000","last_active":"2024-04-17T07:33:28.564918+0000","last_peered":"2024-04-17T07:33:28.564918+0000","last_clean":"2024-04-17T07:33:28.564918+0000","last_became_active":"2024-04-17T07:33:27.549482+0000","last_became_peered":"2024-04-17T07:33:27.549482+0000","last_unstale":"2024-04-17T07:33:28.564918+0000","last_undegraded":"2024-04-17T07:33:28.564918+0000","last_fullsized":"2024-04-17T07:33:28.564918+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:23.155259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759210+0000","last_change":"2024-04-17T07:33:27.585802+0000","last_active":"2024-04-17T07:33:28.759210+0000","last_peered":"2024-04-17T07:33:28.759210+0000","last_clean":"2024-04-17T07:33:28.759210+0000","last_became_active":"2024-04-17T07:33:27.585431+0000","last_became_peered":"2024-04-17T07:33:27.585431+0000","last_unstale":"2024-04-17T07:33:28.759210+0000","last_undegraded":"2024-04-17T07:33:28.759210+0000","last_fullsized":"2024-04-17T07:33:28.759210+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:54.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757859+0000","last_change":"2024-04-17T07:33:27.571584+0000","last_active":"2024-04-17T07:33:28.757859+0000","last_peered":"2024-04-17T07:33:28.757859+0000","last_clean":"2024-04-17T07:33:28.757859+0000","last_became_active":"2024-04-17T07:33:27.570734+0000","last_became_peered":"2024-04-17T07:33:27.570734+0000","last_unstale":"2024-04-17T07:33:28.757859+0000","last_undegraded":"2024-04-17T07:33:28.757859+0000","last_fullsized":"2024-04-17T07:33:28.757859+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:59:10.617296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758761+0000","last_change":"2024-04-17T07:33:27.574249+0000","last_active":"2024-04-17T07:33:28.758761+0000","last_peered":"2024-04-17T07:33:28.758761+0000","last_clean":"2024-04-17T07:33:28.758761+0000","last_became_active":"2024-04-17T07:33:27.573915+0000","last_became_peered":"2024-04-17T07:33:27.573915+0000","last_unstale":"2024-04-17T07:33:28.758761+0000","last_undegraded":"2024-04-17T07:33:28.758761+0000","last_fullsized":"2024-04-17T07:33:28.758761+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:06:31.064161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757940+0000","last_change":"2024-04-17T07:33:27.572760+0000","last_active":"2024-04-17T07:33:28.757940+0000","last_peered":"2024-04-17T07:33:28.757940+0000","last_clean":"2024-04-17T07:33:28.757940+0000","last_became_active":"2024-04-17T07:33:27.570847+0000","last_became_peered":"2024-04-17T07:33:27.570847+0000","last_unstale":"2024-04-17T07:33:28.757940+0000","last_undegraded":"2024-04-17T07:33:28.757940+0000","last_fullsized":"2024-04-17T07:33:28.757940+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:22.211757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567374+0000","last_change":"2024-04-17T07:33:27.568093+0000","last_active":"2024-04-17T07:33:28.567374+0000","last_peered":"2024-04-17T07:33:28.567374+0000","last_clean":"2024-04-17T07:33:28.567374+0000","last_became_active":"2024-04-17T07:33:27.567794+0000","last_became_peered":"2024-04-17T07:33:27.567794+0000","last_unstale":"2024-04-17T07:33:28.567374+0000","last_undegraded":"2024-04-17T07:33:28.567374+0000","last_fullsized":"2024-04-17T07:33:28.567374+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:36.779561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566502+0000","last_change":"2024-04-17T07:33:27.568549+0000","last_active":"2024-04-17T07:33:28.566502+0000","last_peered":"2024-04-17T07:33:28.566502+0000","last_clean":"2024-04-17T07:33:28.566502+0000","last_became_active":"2024-04-17T07:33:27.568189+0000","last_became_peered":"2024-04-17T07:33:27.568189+0000","last_unstale":"2024-04-17T07:33:28.566502+0000","last_undegraded":"2024-04-17T07:33:28.566502+0000","last_fullsized":"2024-04-17T07:33:28.566502+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:43.033379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757378+0000","last_change":"2024-04-17T07:33:27.583143+0000","last_active":"2024-04-17T07:33:28.757378+0000","last_peered":"2024-04-17T07:33:28.757378+0000","last_clean":"2024-04-17T07:33:28.757378+0000","last_became_active":"2024-04-17T07:33:27.582815+0000","last_became_peered":"2024-04-17T07:33:27.582815+0000","last_unstale":"2024-04-17T07:33:28.757378+0000","last_undegraded":"2024-04-17T07:33:28.757378+0000","last_fullsized":"2024-04-17T07:33:28.757378+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:03.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756549+0000","last_change":"2024-04-17T07:33:27.575976+0000","last_active":"2024-04-17T07:33:28.756549+0000","last_peered":"2024-04-17T07:33:28.756549+0000","last_clean":"2024-04-17T07:33:28.756549+0000","last_became_active":"2024-04-17T07:33:27.575675+0000","last_became_peered":"2024-04-17T07:33:27.575675+0000","last_unstale":"2024-04-17T07:33:28.756549+0000","last_undegraded":"2024-04-17T07:33:28.756549+0000","last_fullsized":"2024-04-17T07:33:28.756549+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:10:07.512759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758979+0000","last_change":"2024-04-17T07:33:27.574954+0000","last_active":"2024-04-17T07:33:28.758979+0000","last_peered":"2024-04-17T07:33:28.758979+0000","last_clean":"2024-04-17T07:33:28.758979+0000","last_became_active":"2024-04-17T07:33:27.574702+0000","last_became_peered":"2024-04-17T07:33:27.574702+0000","last_unstale":"2024-04-17T07:33:28.758979+0000","last_undegraded":"2024-04-17T07:33:28.758979+0000","last_fullsized":"2024-04-17T07:33:28.758979+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:46.951849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594239+0000","last_change":"2024-04-17T07:33:27.577851+0000","last_active":"2024-04-17T07:33:28.594239+0000","last_peered":"2024-04-17T07:33:28.594239+0000","last_clean":"2024-04-17T07:33:28.594239+0000","last_became_active":"2024-04-17T07:33:27.577226+0000","last_became_peered":"2024-04-17T07:33:27.577226+0000","last_unstale":"2024-04-17T07:33:28.594239+0000","last_undegraded":"2024-04-17T07:33:28.594239+0000","last_fullsized":"2024-04-17T07:33:28.594239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:43:06.763568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567501+0000","last_change":"2024-04-17T07:33:27.575145+0000","last_active":"2024-04-17T07:33:28.567501+0000","last_peered":"2024-04-17T07:33:28.567501+0000","last_clean":"2024-04-17T07:33:28.567501+0000","last_became_active":"2024-04-17T07:33:27.573004+0000","last_became_peered":"2024-04-17T07:33:27.573004+0000","last_unstale":"2024-04-17T07:33:28.567501+0000","last_undegraded":"2024-04-17T07:33:28.567501+0000","last_fullsized":"2024-04-17T07:33:28.567501+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:00.349150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758585+0000","last_change":"2024-04-17T07:33:27.574545+0000","last_active":"2024-04-17T07:33:28.758585+0000","last_peered":"2024-04-17T07:33:28.758585+0000","last_clean":"2024-04-17T07:33:28.758585+0000","last_became_active":"2024-04-17T07:33:27.571629+0000","last_became_peered":"2024-04-17T07:33:27.571629+0000","last_unstale":"2024-04-17T07:33:28.758585+0000","last_undegraded":"2024-04-17T07:33:28.758585+0000","last_fullsized":"2024-04-17T07:33:28.758585+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:34.411556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759028+0000","last_change":"2024-04-17T07:33:27.584790+0000","last_active":"2024-04-17T07:33:28.759028+0000","last_peered":"2024-04-17T07:33:28.759028+0000","last_clean":"2024-04-17T07:33:28.759028+0000","last_became_active":"2024-04-17T07:33:27.584482+0000","last_became_peered":"2024-04-17T07:33:27.584482+0000","last_unstale":"2024-04-17T07:33:28.759028+0000","last_undegraded":"2024-04-17T07:33:28.759028+0000","last_fullsized":"2024-04-17T07:33:28.759028+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:11.666283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562163+0000","last_change":"2024-04-17T07:33:27.582454+0000","last_active":"2024-04-17T07:33:28.562163+0000","last_peered":"2024-04-17T07:33:28.562163+0000","last_clean":"2024-04-17T07:33:28.562163+0000","last_became_active":"2024-04-17T07:33:27.580089+0000","last_became_peered":"2024-04-17T07:33:27.580089+0000","last_unstale":"2024-04-17T07:33:28.562163+0000","last_undegraded":"2024-04-17T07:33:28.562163+0000","last_fullsized":"2024-04-17T07:33:28.562163+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:58.988571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757967+0000","last_change":"2024-04-17T07:33:27.573427+0000","last_active":"2024-04-17T07:33:28.757967+0000","last_peered":"2024-04-17T07:33:28.757967+0000","last_clean":"2024-04-17T07:33:28.757967+0000","last_became_active":"2024-04-17T07:33:27.573065+0000","last_became_peered":"2024-04-17T07:33:27.573065+0000","last_unstale":"2024-04-17T07:33:28.757967+0000","last_undegraded":"2024-04-17T07:33:28.757967+0000","last_fullsized":"2024-04-17T07:33:28.757967+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:11.653657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.561984+0000","last_change":"2024-04-17T07:33:27.580010+0000","last_active":"2024-04-17T07:33:28.561984+0000","last_peered":"2024-04-17T07:33:28.561984+0000","last_clean":"2024-04-17T07:33:28.561984+0000","last_became_active":"2024-04-17T07:33:27.579730+0000","last_became_peered":"2024-04-17T07:33:27.579730+0000","last_unstale":"2024-04-17T07:33:28.561984+0000","last_undegraded":"2024-04-17T07:33:28.561984+0000","last_fullsized":"2024-04-17T07:33:28.561984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:45:30.741505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759066+0000","last_change":"2024-04-17T07:33:27.576317+0000","last_active":"2024-04-17T07:33:28.759066+0000","last_peered":"2024-04-17T07:33:28.759066+0000","last_clean":"2024-04-17T07:33:28.759066+0000","last_became_active":"2024-04-17T07:33:27.575781+0000","last_became_peered":"2024-04-17T07:33:27.575781+0000","last_unstale":"2024-04-17T07:33:28.759066+0000","last_undegraded":"2024-04-17T07:33:28.759066+0000","last_fullsized":"2024-04-17T07:33:28.759066+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:26:58.390643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758068+0000","last_change":"2024-04-17T07:33:27.576045+0000","last_active":"2024-04-17T07:33:28.758068+0000","last_peered":"2024-04-17T07:33:28.758068+0000","last_clean":"2024-04-17T07:33:28.758068+0000","last_became_active":"2024-04-17T07:33:27.575081+0000","last_became_peered":"2024-04-17T07:33:27.575081+0000","last_unstale":"2024-04-17T07:33:28.758068+0000","last_undegraded":"2024-04-17T07:33:28.758068+0000","last_fullsized":"2024-04-17T07:33:28.758068+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:27.578908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758942+0000","last_change":"2024-04-17T07:33:27.569431+0000","last_active":"2024-04-17T07:33:28.758942+0000","last_peered":"2024-04-17T07:33:28.758942+0000","last_clean":"2024-04-17T07:33:28.758942+0000","last_became_active":"2024-04-17T07:33:27.569118+0000","last_became_peered":"2024-04-17T07:33:27.569118+0000","last_unstale":"2024-04-17T07:33:28.758942+0000","last_undegraded":"2024-04-17T07:33:28.758942+0000","last_fullsized":"2024-04-17T07:33:28.758942+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:33.574282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562542+0000","last_change":"2024-04-17T07:33:27.586836+0000","last_active":"2024-04-17T07:33:28.562542+0000","last_peered":"2024-04-17T07:33:28.562542+0000","last_clean":"2024-04-17T07:33:28.562542+0000","last_became_active":"2024-04-17T07:33:27.586576+0000","last_became_peered":"2024-04-17T07:33:27.586576+0000","last_unstale":"2024-04-17T07:33:28.562542+0000","last_undegraded":"2024-04-17T07:33:28.562542+0000","last_fullsized":"2024-04-17T07:33:28.562542+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:34.666286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758312+0000","last_change":"2024-04-17T07:33:27.574886+0000","last_active":"2024-04-17T07:33:28.758312+0000","last_peered":"2024-04-17T07:33:28.758312+0000","last_clean":"2024-04-17T07:33:28.758312+0000","last_became_active":"2024-04-17T07:33:27.574590+0000","last_became_peered":"2024-04-17T07:33:27.574590+0000","last_unstale":"2024-04-17T07:33:28.758312+0000","last_undegraded":"2024-04-17T07:33:28.758312+0000","last_fullsized":"2024-04-17T07:33:28.758312+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:43.664413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567300+0000","last_change":"2024-04-17T07:33:27.582859+0000","last_active":"2024-04-17T07:33:28.567300+0000","last_peered":"2024-04-17T07:33:28.567300+0000","last_clean":"2024-04-17T07:33:28.567300+0000","last_became_active":"2024-04-17T07:33:27.582550+0000","last_became_peered":"2024-04-17T07:33:27.582550+0000","last_unstale":"2024-04-17T07:33:28.567300+0000","last_undegraded":"2024-04-17T07:33:28.567300+0000","last_fullsized":"2024-04-17T07:33:28.567300+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:36.150233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757226+0000","last_change":"2024-04-17T07:33:27.571736+0000","last_active":"2024-04-17T07:33:28.757226+0000","last_peered":"2024-04-17T07:33:28.757226+0000","last_clean":"2024-04-17T07:33:28.757226+0000","last_became_active":"2024-04-17T07:33:27.568899+0000","last_became_peered":"2024-04-17T07:33:27.568899+0000","last_unstale":"2024-04-17T07:33:28.757226+0000","last_undegraded":"2024-04-17T07:33:28.757226+0000","last_fullsized":"2024-04-17T07:33:28.757226+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:53.305969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757104+0000","last_change":"2024-04-17T07:33:27.583936+0000","last_active":"2024-04-17T07:33:28.757104+0000","last_peered":"2024-04-17T07:33:28.757104+0000","last_clean":"2024-04-17T07:33:28.757104+0000","last_became_active":"2024-04-17T07:33:27.583653+0000","last_became_peered":"2024-04-17T07:33:27.583653+0000","last_unstale":"2024-04-17T07:33:28.757104+0000","last_undegraded":"2024-04-17T07:33:28.757104+0000","last_fullsized":"2024-04-17T07:33:28.757104+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:25.990378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758338+0000","last_change":"2024-04-17T07:33:27.572126+0000","last_active":"2024-04-17T07:33:28.758338+0000","last_peered":"2024-04-17T07:33:28.758338+0000","last_clean":"2024-04-17T07:33:28.758338+0000","last_became_active":"2024-04-17T07:33:27.571857+0000","last_became_peered":"2024-04-17T07:33:27.571857+0000","last_unstale":"2024-04-17T07:33:28.758338+0000","last_undegraded":"2024-04-17T07:33:28.758338+0000","last_fullsized":"2024-04-17T07:33:28.758338+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:52.419618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"141'4","reported_seq":18,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568231+0000","last_change":"2024-04-17T07:33:26.520033+0000","last_active":"2024-04-17T07:33:28.568231+0000","last_peered":"2024-04-17T07:33:28.568231+0000","last_clean":"2024-04-17T07:33:28.568231+0000","last_became_active":"2024-04-17T07:33:26.519630+0000","last_became_peered":"2024-04-17T07:33:26.519630+0000","last_unstale":"2024-04-17T07:33:28.568231+0000","last_undegraded":"2024-04-17T07:33:28.568231+0000","last_fullsized":"2024-04-17T07:33:28.568231+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:58.827598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594672+0000","last_change":"2024-04-17T07:33:27.571039+0000","last_active":"2024-04-17T07:33:28.594672+0000","last_peered":"2024-04-17T07:33:28.594672+0000","last_clean":"2024-04-17T07:33:28.594672+0000","last_became_active":"2024-04-17T07:33:27.570721+0000","last_became_peered":"2024-04-17T07:33:27.570721+0000","last_unstale":"2024-04-17T07:33:28.594672+0000","last_undegraded":"2024-04-17T07:33:28.594672+0000","last_fullsized":"2024-04-17T07:33:28.594672+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:03.660265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567588+0000","last_change":"2024-04-17T07:33:26.527149+0000","last_active":"2024-04-17T07:33:28.567588+0000","last_peered":"2024-04-17T07:33:28.567588+0000","last_clean":"2024-04-17T07:33:28.567588+0000","last_became_active":"2024-04-17T07:33:26.526615+0000","last_became_peered":"2024-04-17T07:33:26.526615+0000","last_unstale":"2024-04-17T07:33:28.567588+0000","last_undegraded":"2024-04-17T07:33:28.567588+0000","last_fullsized":"2024-04-17T07:33:28.567588+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:45:29.736238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758710+0000","last_change":"2024-04-17T07:33:27.585562+0000","last_active":"2024-04-17T07:33:28.758710+0000","last_peered":"2024-04-17T07:33:28.758710+0000","last_clean":"2024-04-17T07:33:28.758710+0000","last_became_active":"2024-04-17T07:33:27.585302+0000","last_became_peered":"2024-04-17T07:33:27.585302+0000","last_unstale":"2024-04-17T07:33:28.758710+0000","last_undegraded":"2024-04-17T07:33:28.758710+0000","last_fullsized":"2024-04-17T07:33:28.758710+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:53.681324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760017+0000","last_change":"2024-04-17T07:33:26.515654+0000","last_active":"2024-04-17T07:33:28.760017+0000","last_peered":"2024-04-17T07:33:28.760017+0000","last_clean":"2024-04-17T07:33:28.760017+0000","last_became_active":"2024-04-17T07:33:26.515359+0000","last_became_peered":"2024-04-17T07:33:26.515359+0000","last_unstale":"2024-04-17T07:33:28.760017+0000","last_undegraded":"2024-04-17T07:33:28.760017+0000","last_fullsized":"2024-04-17T07:33:28.760017+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:52.977206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567763+0000","last_change":"2024-04-17T07:33:27.569020+0000","last_active":"2024-04-17T07:33:28.567763+0000","last_peered":"2024-04-17T07:33:28.567763+0000","last_clean":"2024-04-17T07:33:28.567763+0000","last_became_active":"2024-04-17T07:33:27.568740+0000","last_became_peered":"2024-04-17T07:33:27.568740+0000","last_unstale":"2024-04-17T07:33:28.567763+0000","last_undegraded":"2024-04-17T07:33:28.567763+0000","last_fullsized":"2024-04-17T07:33:28.567763+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:23.793208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760132+0000","last_change":"2024-04-17T07:33:26.516876+0000","last_active":"2024-04-17T07:33:28.760132+0000","last_peered":"2024-04-17T07:33:28.760132+0000","last_clean":"2024-04-17T07:33:28.760132+0000","last_became_active":"2024-04-17T07:33:26.515528+0000","last_became_peered":"2024-04-17T07:33:26.515528+0000","last_unstale":"2024-04-17T07:33:28.760132+0000","last_undegraded":"2024-04-17T07:33:28.760132+0000","last_fullsized":"2024-04-17T07:33:28.760132+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:53.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756998+0000","last_change":"2024-04-17T07:33:27.583685+0000","last_active":"2024-04-17T07:33:28.756998+0000","last_peered":"2024-04-17T07:33:28.756998+0000","last_clean":"2024-04-17T07:33:28.756998+0000","last_became_active":"2024-04-17T07:33:27.583415+0000","last_became_peered":"2024-04-17T07:33:27.583415+0000","last_unstale":"2024-04-17T07:33:28.756998+0000","last_undegraded":"2024-04-17T07:33:28.756998+0000","last_fullsized":"2024-04-17T07:33:28.756998+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:54:34.114306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567754+0000","last_change":"2024-04-17T07:33:26.515193+0000","last_active":"2024-04-17T07:33:28.567754+0000","last_peered":"2024-04-17T07:33:28.567754+0000","last_clean":"2024-04-17T07:33:28.567754+0000","last_became_active":"2024-04-17T07:33:26.514883+0000","last_became_peered":"2024-04-17T07:33:26.514883+0000","last_unstale":"2024-04-17T07:33:28.567754+0000","last_undegraded":"2024-04-17T07:33:28.567754+0000","last_fullsized":"2024-04-17T07:33:28.567754+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:18.062410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758346+0000","last_change":"2024-04-17T07:33:27.570290+0000","last_active":"2024-04-17T07:33:28.758346+0000","last_peered":"2024-04-17T07:33:28.758346+0000","last_clean":"2024-04-17T07:33:28.758346+0000","last_became_active":"2024-04-17T07:33:27.569990+0000","last_became_peered":"2024-04-17T07:33:27.569990+0000","last_unstale":"2024-04-17T07:33:28.758346+0000","last_undegraded":"2024-04-17T07:33:28.758346+0000","last_fullsized":"2024-04-17T07:33:28.758346+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:19.067677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562580+0000","last_change":"2024-04-17T07:33:26.523017+0000","last_active":"2024-04-17T07:33:28.562580+0000","last_peered":"2024-04-17T07:33:28.562580+0000","last_clean":"2024-04-17T07:33:28.562580+0000","last_became_active":"2024-04-17T07:33:26.522511+0000","last_became_peered":"2024-04-17T07:33:26.522511+0000","last_unstale":"2024-04-17T07:33:28.562580+0000","last_undegraded":"2024-04-17T07:33:28.562580+0000","last_fullsized":"2024-04-17T07:33:28.562580+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:19.220691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756821+0000","last_change":"2024-04-17T07:33:27.567138+0000","last_active":"2024-04-17T07:33:28.756821+0000","last_peered":"2024-04-17T07:33:28.756821+0000","last_clean":"2024-04-17T07:33:28.756821+0000","last_became_active":"2024-04-17T07:33:27.566324+0000","last_became_peered":"2024-04-17T07:33:27.566324+0000","last_unstale":"2024-04-17T07:33:28.756821+0000","last_undegraded":"2024-04-17T07:33:28.756821+0000","last_fullsized":"2024-04-17T07:33:28.756821+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:04.682658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758027+0000","last_change":"2024-04-17T07:33:26.517484+0000","last_active":"2024-04-17T07:33:28.758027+0000","last_peered":"2024-04-17T07:33:28.758027+0000","last_clean":"2024-04-17T07:33:28.758027+0000","last_became_active":"2024-04-17T07:33:26.516353+0000","last_became_peered":"2024-04-17T07:33:26.516353+0000","last_unstale":"2024-04-17T07:33:28.758027+0000","last_undegraded":"2024-04-17T07:33:28.758027+0000","last_fullsized":"2024-04-17T07:33:28.758027+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:55.881555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566128+0000","last_change":"2024-04-17T07:33:27.570261+0000","last_active":"2024-04-17T07:33:28.566128+0000","last_peered":"2024-04-17T07:33:28.566128+0000","last_clean":"2024-04-17T07:33:28.566128+0000","last_became_active":"2024-04-17T07:33:27.568399+0000","last_became_peered":"2024-04-17T07:33:27.568399+0000","last_unstale":"2024-04-17T07:33:28.566128+0000","last_undegraded":"2024-04-17T07:33:28.566128+0000","last_fullsized":"2024-04-17T07:33:28.566128+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:50.074512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756774+0000","last_change":"2024-04-17T07:33:26.512001+0000","last_active":"2024-04-17T07:33:28.756774+0000","last_peered":"2024-04-17T07:33:28.756774+0000","last_clean":"2024-04-17T07:33:28.756774+0000","last_became_active":"2024-04-17T07:33:26.511647+0000","last_became_peered":"2024-04-17T07:33:26.511647+0000","last_unstale":"2024-04-17T07:33:28.756774+0000","last_undegraded":"2024-04-17T07:33:28.756774+0000","last_fullsized":"2024-04-17T07:33:28.756774+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:00:37.696656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568038+0000","last_change":"2024-04-17T07:33:27.572498+0000","last_active":"2024-04-17T07:33:28.568038+0000","last_peered":"2024-04-17T07:33:28.568038+0000","last_clean":"2024-04-17T07:33:28.568038+0000","last_became_active":"2024-04-17T07:33:27.572185+0000","last_became_peered":"2024-04-17T07:33:27.572185+0000","last_unstale":"2024-04-17T07:33:28.568038+0000","last_undegraded":"2024-04-17T07:33:28.568038+0000","last_fullsized":"2024-04-17T07:33:28.568038+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:38.838123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759843+0000","last_change":"2024-04-17T07:33:26.517258+0000","last_active":"2024-04-17T07:33:28.759843+0000","last_peered":"2024-04-17T07:33:28.759843+0000","last_clean":"2024-04-17T07:33:28.759843+0000","last_became_active":"2024-04-17T07:33:26.516402+0000","last_became_peered":"2024-04-17T07:33:26.516402+0000","last_unstale":"2024-04-17T07:33:28.759843+0000","last_undegraded":"2024-04-17T07:33:28.759843+0000","last_fullsized":"2024-04-17T07:33:28.759843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:57.548656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594707+0000","last_change":"2024-04-17T07:33:27.579135+0000","last_active":"2024-04-17T07:33:28.594707+0000","last_peered":"2024-04-17T07:33:28.594707+0000","last_clean":"2024-04-17T07:33:28.594707+0000","last_became_active":"2024-04-17T07:33:27.578673+0000","last_became_peered":"2024-04-17T07:33:27.578673+0000","last_unstale":"2024-04-17T07:33:28.594707+0000","last_undegraded":"2024-04-17T07:33:28.594707+0000","last_fullsized":"2024-04-17T07:33:28.594707+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:52.419618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563960+0000","last_change":"2024-04-17T07:33:26.523217+0000","last_active":"2024-04-17T07:33:28.563960+0000","last_peered":"2024-04-17T07:33:28.563960+0000","last_clean":"2024-04-17T07:33:28.563960+0000","last_became_active":"2024-04-17T07:33:26.522701+0000","last_became_peered":"2024-04-17T07:33:26.522701+0000","last_unstale":"2024-04-17T07:33:28.563960+0000","last_undegraded":"2024-04-17T07:33:28.563960+0000","last_fullsized":"2024-04-17T07:33:28.563960+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:04.433074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562944+0000","last_change":"2024-04-17T07:33:27.584002+0000","last_active":"2024-04-17T07:33:28.562944+0000","last_peered":"2024-04-17T07:33:28.562944+0000","last_clean":"2024-04-17T07:33:28.562944+0000","last_became_active":"2024-04-17T07:33:27.583246+0000","last_became_peered":"2024-04-17T07:33:27.583246+0000","last_unstale":"2024-04-17T07:33:28.562944+0000","last_undegraded":"2024-04-17T07:33:28.562944+0000","last_fullsized":"2024-04-17T07:33:28.562944+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:06:15.984794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"141'3","reported_seq":17,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757821+0000","last_change":"2024-04-17T07:33:26.520309+0000","last_active":"2024-04-17T07:33:28.757821+0000","last_peered":"2024-04-17T07:33:28.757821+0000","last_clean":"2024-04-17T07:33:28.757821+0000","last_became_active":"2024-04-17T07:33:26.520006+0000","last_became_peered":"2024-04-17T07:33:26.520006+0000","last_unstale":"2024-04-17T07:33:28.757821+0000","last_undegraded":"2024-04-17T07:33:28.757821+0000","last_fullsized":"2024-04-17T07:33:28.757821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:21:06.788744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566856+0000","last_change":"2024-04-17T07:33:27.552575+0000","last_active":"2024-04-17T07:33:28.566856+0000","last_peered":"2024-04-17T07:33:28.566856+0000","last_clean":"2024-04-17T07:33:28.566856+0000","last_became_active":"2024-04-17T07:33:27.551759+0000","last_became_peered":"2024-04-17T07:33:27.551759+0000","last_unstale":"2024-04-17T07:33:28.566856+0000","last_undegraded":"2024-04-17T07:33:28.566856+0000","last_fullsized":"2024-04-17T07:33:28.566856+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:35.745460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"141'6","reported_seq":20,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758488+0000","last_change":"2024-04-17T07:33:26.498325+0000","last_active":"2024-04-17T07:33:28.758488+0000","last_peered":"2024-04-17T07:33:28.758488+0000","last_clean":"2024-04-17T07:33:28.758488+0000","last_became_active":"2024-04-17T07:33:26.498019+0000","last_became_peered":"2024-04-17T07:33:26.498019+0000","last_unstale":"2024-04-17T07:33:28.758488+0000","last_undegraded":"2024-04-17T07:33:28.758488+0000","last_fullsized":"2024-04-17T07:33:28.758488+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:02:47.669089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562684+0000","last_change":"2024-04-17T07:33:27.580429+0000","last_active":"2024-04-17T07:33:28.562684+0000","last_peered":"2024-04-17T07:33:28.562684+0000","last_clean":"2024-04-17T07:33:28.562684+0000","last_became_active":"2024-04-17T07:33:27.579746+0000","last_became_peered":"2024-04-17T07:33:27.579746+0000","last_unstale":"2024-04-17T07:33:28.562684+0000","last_undegraded":"2024-04-17T07:33:28.562684+0000","last_fullsized":"2024-04-17T07:33:28.562684+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:27.405509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758474+0000","last_change":"2024-04-17T07:33:26.527902+0000","last_active":"2024-04-17T07:33:28.758474+0000","last_peered":"2024-04-17T07:33:28.758474+0000","last_clean":"2024-04-17T07:33:28.758474+0000","last_became_active":"2024-04-17T07:33:26.527105+0000","last_became_peered":"2024-04-17T07:33:26.527105+0000","last_unstale":"2024-04-17T07:33:28.758474+0000","last_undegraded":"2024-04-17T07:33:28.758474+0000","last_fullsized":"2024-04-17T07:33:28.758474+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:37.065496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563596+0000","last_change":"2024-04-17T07:33:27.577044+0000","last_active":"2024-04-17T07:33:28.563596+0000","last_peered":"2024-04-17T07:33:28.563596+0000","last_clean":"2024-04-17T07:33:28.563596+0000","last_became_active":"2024-04-17T07:33:27.576715+0000","last_became_peered":"2024-04-17T07:33:27.576715+0000","last_unstale":"2024-04-17T07:33:28.563596+0000","last_undegraded":"2024-04-17T07:33:28.563596+0000","last_fullsized":"2024-04-17T07:33:28.563596+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:03.219137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594040+0000","last_change":"2024-04-17T07:33:26.526646+0000","last_active":"2024-04-17T07:33:28.594040+0000","last_peered":"2024-04-17T07:33:28.594040+0000","last_clean":"2024-04-17T07:33:28.594040+0000","last_became_active":"2024-04-17T07:33:26.526283+0000","last_became_peered":"2024-04-17T07:33:26.526283+0000","last_unstale":"2024-04-17T07:33:28.594040+0000","last_undegraded":"2024-04-17T07:33:28.594040+0000","last_fullsized":"2024-04-17T07:33:28.594040+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:38:37.065496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562468+0000","last_change":"2024-04-17T07:33:27.576979+0000","last_active":"2024-04-17T07:33:28.562468+0000","last_peered":"2024-04-17T07:33:28.562468+0000","last_clean":"2024-04-17T07:33:28.562468+0000","last_became_active":"2024-04-17T07:33:27.576579+0000","last_became_peered":"2024-04-17T07:33:27.576579+0000","last_unstale":"2024-04-17T07:33:28.562468+0000","last_undegraded":"2024-04-17T07:33:28.562468+0000","last_fullsized":"2024-04-17T07:33:28.562468+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:15:11.262084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"141'8","reported_seq":22,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759566+0000","last_change":"2024-04-17T07:33:26.520015+0000","last_active":"2024-04-17T07:33:28.759566+0000","last_peered":"2024-04-17T07:33:28.759566+0000","last_clean":"2024-04-17T07:33:28.759566+0000","last_became_active":"2024-04-17T07:33:26.519516+0000","last_became_peered":"2024-04-17T07:33:26.519516+0000","last_unstale":"2024-04-17T07:33:28.759566+0000","last_undegraded":"2024-04-17T07:33:28.759566+0000","last_fullsized":"2024-04-17T07:33:28.759566+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:02:47.669089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760375+0000","last_change":"2024-04-17T07:33:27.565708+0000","last_active":"2024-04-17T07:33:28.760375+0000","last_peered":"2024-04-17T07:33:28.760375+0000","last_clean":"2024-04-17T07:33:28.760375+0000","last_became_active":"2024-04-17T07:33:27.565421+0000","last_became_peered":"2024-04-17T07:33:27.565421+0000","last_unstale":"2024-04-17T07:33:28.760375+0000","last_undegraded":"2024-04-17T07:33:28.760375+0000","last_fullsized":"2024-04-17T07:33:28.760375+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:32.623989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759549+0000","last_change":"2024-04-17T07:33:26.523746+0000","last_active":"2024-04-17T07:33:28.759549+0000","last_peered":"2024-04-17T07:33:28.759549+0000","last_clean":"2024-04-17T07:33:28.759549+0000","last_became_active":"2024-04-17T07:33:26.523456+0000","last_became_peered":"2024-04-17T07:33:26.523456+0000","last_unstale":"2024-04-17T07:33:28.759549+0000","last_undegraded":"2024-04-17T07:33:28.759549+0000","last_fullsized":"2024-04-17T07:33:28.759549+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:47.626837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758424+0000","last_change":"2024-04-17T07:33:27.570319+0000","last_active":"2024-04-17T07:33:28.758424+0000","last_peered":"2024-04-17T07:33:28.758424+0000","last_clean":"2024-04-17T07:33:28.758424+0000","last_became_active":"2024-04-17T07:33:27.570024+0000","last_became_peered":"2024-04-17T07:33:27.570024+0000","last_unstale":"2024-04-17T07:33:28.758424+0000","last_undegraded":"2024-04-17T07:33:28.758424+0000","last_fullsized":"2024-04-17T07:33:28.758424+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:08.123232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.760221+0000","last_change":"2024-04-17T07:33:26.514453+0000","last_active":"2024-04-17T07:33:28.760221+0000","last_peered":"2024-04-17T07:33:28.760221+0000","last_clean":"2024-04-17T07:33:28.760221+0000","last_became_active":"2024-04-17T07:33:26.512214+0000","last_became_peered":"2024-04-17T07:33:26.512214+0000","last_unstale":"2024-04-17T07:33:28.760221+0000","last_undegraded":"2024-04-17T07:33:28.760221+0000","last_fullsized":"2024-04-17T07:33:28.760221+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:36.359971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567107+0000","last_change":"2024-04-17T07:33:27.568901+0000","last_active":"2024-04-17T07:33:28.567107+0000","last_peered":"2024-04-17T07:33:28.567107+0000","last_clean":"2024-04-17T07:33:28.567107+0000","last_became_active":"2024-04-17T07:33:27.567363+0000","last_became_peered":"2024-04-17T07:33:27.567363+0000","last_unstale":"2024-04-17T07:33:28.567107+0000","last_undegraded":"2024-04-17T07:33:28.567107+0000","last_fullsized":"2024-04-17T07:33:28.567107+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:31:36.204058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"141'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594379+0000","last_change":"2024-04-17T07:33:26.517616+0000","last_active":"2024-04-17T07:33:28.594379+0000","last_peered":"2024-04-17T07:33:28.594379+0000","last_clean":"2024-04-17T07:33:28.594379+0000","last_became_active":"2024-04-17T07:33:26.517026+0000","last_became_peered":"2024-04-17T07:33:26.517026+0000","last_unstale":"2024-04-17T07:33:28.594379+0000","last_undegraded":"2024-04-17T07:33:28.594379+0000","last_fullsized":"2024-04-17T07:33:28.594379+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:21:30.056993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.567888+0000","last_change":"2024-04-17T07:33:27.553105+0000","last_active":"2024-04-17T07:33:28.567888+0000","last_peered":"2024-04-17T07:33:28.567888+0000","last_clean":"2024-04-17T07:33:28.567888+0000","last_became_active":"2024-04-17T07:33:27.552366+0000","last_became_peered":"2024-04-17T07:33:27.552366+0000","last_unstale":"2024-04-17T07:33:28.567888+0000","last_undegraded":"2024-04-17T07:33:28.567888+0000","last_fullsized":"2024-04-17T07:33:28.567888+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:02.436650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.565732+0000","last_change":"2024-04-17T07:33:26.523742+0000","last_active":"2024-04-17T07:33:28.565732+0000","last_peered":"2024-04-17T07:33:28.565732+0000","last_clean":"2024-04-17T07:33:28.565732+0000","last_became_active":"2024-04-17T07:33:26.523369+0000","last_became_peered":"2024-04-17T07:33:26.523369+0000","last_unstale":"2024-04-17T07:33:28.565732+0000","last_undegraded":"2024-04-17T07:33:28.565732+0000","last_fullsized":"2024-04-17T07:33:28.565732+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:42.659145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566455+0000","last_change":"2024-04-17T07:33:27.573401+0000","last_active":"2024-04-17T07:33:28.566455+0000","last_peered":"2024-04-17T07:33:28.566455+0000","last_clean":"2024-04-17T07:33:28.566455+0000","last_became_active":"2024-04-17T07:33:27.573067+0000","last_became_peered":"2024-04-17T07:33:27.573067+0000","last_unstale":"2024-04-17T07:33:28.566455+0000","last_undegraded":"2024-04-17T07:33:28.566455+0000","last_fullsized":"2024-04-17T07:33:28.566455+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:28.864383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758594+0000","last_change":"2024-04-17T07:33:26.526987+0000","last_active":"2024-04-17T07:33:28.758594+0000","last_peered":"2024-04-17T07:33:28.758594+0000","last_clean":"2024-04-17T07:33:28.758594+0000","last_became_active":"2024-04-17T07:33:26.526684+0000","last_became_peered":"2024-04-17T07:33:26.526684+0000","last_unstale":"2024-04-17T07:33:28.758594+0000","last_undegraded":"2024-04-17T07:33:28.758594+0000","last_fullsized":"2024-04-17T07:33:28.758594+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:21:30.056993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757120+0000","last_change":"2024-04-17T07:33:27.576417+0000","last_active":"2024-04-17T07:33:28.757120+0000","last_peered":"2024-04-17T07:33:28.757120+0000","last_clean":"2024-04-17T07:33:28.757120+0000","last_became_active":"2024-04-17T07:33:27.575741+0000","last_became_peered":"2024-04-17T07:33:27.575741+0000","last_unstale":"2024-04-17T07:33:28.757120+0000","last_undegraded":"2024-04-17T07:33:28.757120+0000","last_fullsized":"2024-04-17T07:33:28.757120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:20:06.419086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759863+0000","last_change":"2024-04-17T07:33:26.518745+0000","last_active":"2024-04-17T07:33:28.759863+0000","last_peered":"2024-04-17T07:33:28.759863+0000","last_clean":"2024-04-17T07:33:28.759863+0000","last_became_active":"2024-04-17T07:33:26.518435+0000","last_became_peered":"2024-04-17T07:33:26.518435+0000","last_unstale":"2024-04-17T07:33:28.759863+0000","last_undegraded":"2024-04-17T07:33:28.759863+0000","last_fullsized":"2024-04-17T07:33:28.759863+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:21:06.788744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.565099+0000","last_change":"2024-04-17T07:33:27.570379+0000","last_active":"2024-04-17T07:33:28.565099+0000","last_peered":"2024-04-17T07:33:28.565099+0000","last_clean":"2024-04-17T07:33:28.565099+0000","last_became_active":"2024-04-17T07:33:27.570046+0000","last_became_peered":"2024-04-17T07:33:27.570046+0000","last_unstale":"2024-04-17T07:33:28.565099+0000","last_undegraded":"2024-04-17T07:33:28.565099+0000","last_fullsized":"2024-04-17T07:33:28.565099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:56:09.759290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757368+0000","last_change":"2024-04-17T07:33:26.526909+0000","last_active":"2024-04-17T07:33:28.757368+0000","last_peered":"2024-04-17T07:33:28.757368+0000","last_clean":"2024-04-17T07:33:28.757368+0000","last_became_active":"2024-04-17T07:33:26.526600+0000","last_became_peered":"2024-04-17T07:33:26.526600+0000","last_unstale":"2024-04-17T07:33:28.757368+0000","last_undegraded":"2024-04-17T07:33:28.757368+0000","last_fullsized":"2024-04-17T07:33:28.757368+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:22:26.699722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562889+0000","last_change":"2024-04-17T07:33:27.579248+0000","last_active":"2024-04-17T07:33:28.562889+0000","last_peered":"2024-04-17T07:33:28.562889+0000","last_clean":"2024-04-17T07:33:28.562889+0000","last_became_active":"2024-04-17T07:33:27.578961+0000","last_became_peered":"2024-04-17T07:33:27.578961+0000","last_unstale":"2024-04-17T07:33:28.562889+0000","last_undegraded":"2024-04-17T07:33:28.562889+0000","last_fullsized":"2024-04-17T07:33:28.562889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:35:42.086755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756911+0000","last_change":"2024-04-17T07:33:26.522674+0000","last_active":"2024-04-17T07:33:28.756911+0000","last_peered":"2024-04-17T07:33:28.756911+0000","last_clean":"2024-04-17T07:33:28.756911+0000","last_became_active":"2024-04-17T07:33:26.522210+0000","last_became_peered":"2024-04-17T07:33:26.522210+0000","last_unstale":"2024-04-17T07:33:28.756911+0000","last_undegraded":"2024-04-17T07:33:28.756911+0000","last_fullsized":"2024-04-17T07:33:28.756911+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:57.458153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594402+0000","last_change":"2024-04-17T07:33:27.574837+0000","last_active":"2024-04-17T07:33:28.594402+0000","last_peered":"2024-04-17T07:33:28.594402+0000","last_clean":"2024-04-17T07:33:28.594402+0000","last_became_active":"2024-04-17T07:33:27.574517+0000","last_became_peered":"2024-04-17T07:33:27.574517+0000","last_unstale":"2024-04-17T07:33:28.594402+0000","last_undegraded":"2024-04-17T07:33:28.594402+0000","last_fullsized":"2024-04-17T07:33:28.594402+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:25.990378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"141'2","reported_seq":16,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.563564+0000","last_change":"2024-04-17T07:33:26.498707+0000","last_active":"2024-04-17T07:33:28.563564+0000","last_peered":"2024-04-17T07:33:28.563564+0000","last_clean":"2024-04-17T07:33:28.563564+0000","last_became_active":"2024-04-17T07:33:26.497493+0000","last_became_peered":"2024-04-17T07:33:26.497493+0000","last_unstale":"2024-04-17T07:33:28.563564+0000","last_undegraded":"2024-04-17T07:33:28.563564+0000","last_fullsized":"2024-04-17T07:33:28.563564+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:35.768018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759410+0000","last_change":"2024-04-17T07:33:27.581353+0000","last_active":"2024-04-17T07:33:28.759410+0000","last_peered":"2024-04-17T07:33:28.759410+0000","last_clean":"2024-04-17T07:33:28.759410+0000","last_became_active":"2024-04-17T07:33:27.581055+0000","last_became_peered":"2024-04-17T07:33:27.581055+0000","last_unstale":"2024-04-17T07:33:28.759410+0000","last_undegraded":"2024-04-17T07:33:28.759410+0000","last_fullsized":"2024-04-17T07:33:28.759410+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:23:49.773528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"142'1","reported_seq":15,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.598360+0000","last_change":"2024-04-17T07:33:26.521281+0000","last_active":"2024-04-17T07:33:28.598360+0000","last_peered":"2024-04-17T07:33:28.598360+0000","last_clean":"2024-04-17T07:33:28.598360+0000","last_became_active":"2024-04-17T07:33:26.518369+0000","last_became_peered":"2024-04-17T07:33:26.518369+0000","last_unstale":"2024-04-17T07:33:28.598360+0000","last_undegraded":"2024-04-17T07:33:28.598360+0000","last_fullsized":"2024-04-17T07:33:28.598360+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:53:26.993104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758808+0000","last_change":"2024-04-17T07:33:27.576584+0000","last_active":"2024-04-17T07:33:28.758808+0000","last_peered":"2024-04-17T07:33:28.758808+0000","last_clean":"2024-04-17T07:33:28.758808+0000","last_became_active":"2024-04-17T07:33:27.576287+0000","last_became_peered":"2024-04-17T07:33:27.576287+0000","last_unstale":"2024-04-17T07:33:28.758808+0000","last_undegraded":"2024-04-17T07:33:28.758808+0000","last_fullsized":"2024-04-17T07:33:28.758808+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:54:38.531544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757977+0000","last_change":"2024-04-17T07:33:26.520712+0000","last_active":"2024-04-17T07:33:28.757977+0000","last_peered":"2024-04-17T07:33:28.757977+0000","last_clean":"2024-04-17T07:33:28.757977+0000","last_became_active":"2024-04-17T07:33:26.520387+0000","last_became_peered":"2024-04-17T07:33:26.520387+0000","last_unstale":"2024-04-17T07:33:28.757977+0000","last_undegraded":"2024-04-17T07:33:28.757977+0000","last_fullsized":"2024-04-17T07:33:28.757977+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:47:57.548656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.594079+0000","last_change":"2024-04-17T07:33:27.570953+0000","last_active":"2024-04-17T07:33:28.594079+0000","last_peered":"2024-04-17T07:33:28.594079+0000","last_clean":"2024-04-17T07:33:28.594079+0000","last_became_active":"2024-04-17T07:33:27.570661+0000","last_became_peered":"2024-04-17T07:33:27.570661+0000","last_unstale":"2024-04-17T07:33:28.594079+0000","last_undegraded":"2024-04-17T07:33:28.594079+0000","last_fullsized":"2024-04-17T07:33:28.594079+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:52.395819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562767+0000","last_change":"2024-04-17T07:33:26.523984+0000","last_active":"2024-04-17T07:33:28.562767+0000","last_peered":"2024-04-17T07:33:28.562767+0000","last_clean":"2024-04-17T07:33:28.562767+0000","last_became_active":"2024-04-17T07:33:26.523666+0000","last_became_peered":"2024-04-17T07:33:26.523666+0000","last_unstale":"2024-04-17T07:33:28.562767+0000","last_undegraded":"2024-04-17T07:33:28.562767+0000","last_fullsized":"2024-04-17T07:33:28.562767+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:55:49.619601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758019+0000","last_change":"2024-04-17T07:33:27.567593+0000","last_active":"2024-04-17T07:33:28.758019+0000","last_peered":"2024-04-17T07:33:28.758019+0000","last_clean":"2024-04-17T07:33:28.758019+0000","last_became_active":"2024-04-17T07:33:27.560647+0000","last_became_peered":"2024-04-17T07:33:27.560647+0000","last_unstale":"2024-04-17T07:33:28.758019+0000","last_undegraded":"2024-04-17T07:33:28.758019+0000","last_fullsized":"2024-04-17T07:33:28.758019+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:23.389879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568010+0000","last_change":"2024-04-17T07:33:26.517942+0000","last_active":"2024-04-17T07:33:28.568010+0000","last_peered":"2024-04-17T07:33:28.568010+0000","last_clean":"2024-04-17T07:33:28.568010+0000","last_became_active":"2024-04-17T07:33:26.517627+0000","last_became_peered":"2024-04-17T07:33:26.517627+0000","last_unstale":"2024-04-17T07:33:28.568010+0000","last_undegraded":"2024-04-17T07:33:28.568010+0000","last_fullsized":"2024-04-17T07:33:28.568010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:33.406288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758736+0000","last_change":"2024-04-17T07:33:27.574379+0000","last_active":"2024-04-17T07:33:28.758736+0000","last_peered":"2024-04-17T07:33:28.758736+0000","last_clean":"2024-04-17T07:33:28.758736+0000","last_became_active":"2024-04-17T07:33:27.571823+0000","last_became_peered":"2024-04-17T07:33:27.571823+0000","last_unstale":"2024-04-17T07:33:28.758736+0000","last_undegraded":"2024-04-17T07:33:28.758736+0000","last_fullsized":"2024-04-17T07:33:28.758736+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:59.832865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.758079+0000","last_change":"2024-04-17T07:33:26.514630+0000","last_active":"2024-04-17T07:33:28.758079+0000","last_peered":"2024-04-17T07:33:28.758079+0000","last_clean":"2024-04-17T07:33:28.758079+0000","last_became_active":"2024-04-17T07:33:26.514099+0000","last_became_peered":"2024-04-17T07:33:26.514099+0000","last_unstale":"2024-04-17T07:33:28.758079+0000","last_undegraded":"2024-04-17T07:33:28.758079+0000","last_fullsized":"2024-04-17T07:33:28.758079+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:07.211383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566110+0000","last_change":"2024-04-17T07:33:27.566792+0000","last_active":"2024-04-17T07:33:28.566110+0000","last_peered":"2024-04-17T07:33:28.566110+0000","last_clean":"2024-04-17T07:33:28.566110+0000","last_became_active":"2024-04-17T07:33:27.565484+0000","last_became_peered":"2024-04-17T07:33:27.565484+0000","last_unstale":"2024-04-17T07:33:28.566110+0000","last_undegraded":"2024-04-17T07:33:28.566110+0000","last_fullsized":"2024-04-17T07:33:28.566110+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:57.245452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.759620+0000","last_change":"2024-04-17T07:33:26.517611+0000","last_active":"2024-04-17T07:33:28.759620+0000","last_peered":"2024-04-17T07:33:28.759620+0000","last_clean":"2024-04-17T07:33:28.759620+0000","last_became_active":"2024-04-17T07:33:26.517337+0000","last_became_peered":"2024-04-17T07:33:26.517337+0000","last_unstale":"2024-04-17T07:33:28.759620+0000","last_undegraded":"2024-04-17T07:33:28.759620+0000","last_fullsized":"2024-04-17T07:33:28.759620+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:47.626837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.756927+0000","last_change":"2024-04-17T07:33:27.572048+0000","last_active":"2024-04-17T07:33:28.756927+0000","last_peered":"2024-04-17T07:33:28.756927+0000","last_clean":"2024-04-17T07:33:28.756927+0000","last_became_active":"2024-04-17T07:33:27.568829+0000","last_became_peered":"2024-04-17T07:33:27.568829+0000","last_unstale":"2024-04-17T07:33:28.756927+0000","last_undegraded":"2024-04-17T07:33:28.756927+0000","last_fullsized":"2024-04-17T07:33:28.756927+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:12.261602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.757083+0000","last_change":"2024-04-17T07:33:26.521478+0000","last_active":"2024-04-17T07:33:28.757083+0000","last_peered":"2024-04-17T07:33:28.757083+0000","last_clean":"2024-04-17T07:33:28.757083+0000","last_became_active":"2024-04-17T07:33:26.519813+0000","last_became_peered":"2024-04-17T07:33:26.519813+0000","last_unstale":"2024-04-17T07:33:28.757083+0000","last_undegraded":"2024-04-17T07:33:28.757083+0000","last_fullsized":"2024-04-17T07:33:28.757083+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:52.977206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.562433+0000","last_change":"2024-04-17T07:33:27.575375+0000","last_active":"2024-04-17T07:33:28.562433+0000","last_peered":"2024-04-17T07:33:28.562433+0000","last_clean":"2024-04-17T07:33:28.562433+0000","last_became_active":"2024-04-17T07:33:27.575054+0000","last_became_peered":"2024-04-17T07:33:27.575054+0000","last_unstale":"2024-04-17T07:33:28.562433+0000","last_undegraded":"2024-04-17T07:33:28.562433+0000","last_fullsized":"2024-04-17T07:33:28.562433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:26.479756+0000","last_clean_scrub_stamp":"2024-04-17T07:33:26.479756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:12.276598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.568338+0000","last_change":"2024-04-17T07:33:26.523730+0000","last_active":"2024-04-17T07:33:28.568338+0000","last_peered":"2024-04-17T07:33:28.568338+0000","last_clean":"2024-04-17T07:33:28.568338+0000","last_became_active":"2024-04-17T07:33:26.523341+0000","last_became_peered":"2024-04-17T07:33:26.523341+0000","last_unstale":"2024-04-17T07:33:28.568338+0000","last_undegraded":"2024-04-17T07:33:28.568338+0000","last_fullsized":"2024-04-17T07:33:28.568338+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:33:25.474489+0000","last_clean_scrub_stamp":"2024-04-17T07:33:25.474489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:22.384611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"130'233","reported_seq":816,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-17T07:33:28.566010+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:33:28.566010+0000","last_peered":"2024-04-17T07:33:28.566010+0000","last_clean":"2024-04-17T07:33:28.566010+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:33:28.566010+0000","last_undegraded":"2024-04-17T07:33:28.566010+0000","last_fullsized":"2024-04-17T07:33:28.566010+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":367,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1122,"num_read_kb":2324,"num_write":367,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542376,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2359296,"data_stored":1435796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542376,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1748992,"data_stored":838554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542375,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1753088,"data_stored":839088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542375,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59272,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312568,"statfs":{"total":96636764160,"available":96576069632,"internally_reserved":0,"allocated":1757184,"data_stored":844812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542377,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":840166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":1744896,"data_stored":838484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542377,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55120,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316720,"statfs":{"total":96636764160,"available":96580321280,"internally_reserved":0,"allocated":1765376,"data_stored":844936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2359296,"data_stored":1430576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:33:32.618 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:33:32.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:33:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.890+0000 7f1ee8200640 1 Processor -- start 2024-04-17T07:33:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.890+0000 7f1ee8200640 1 -- start start 2024-04-17T07:33:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.892+0000 7f1ee8200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 0x7f1ee00cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.892+0000 7f1ee8200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1ee0072630 0x7f1ee00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.892+0000 7f1ee8200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee006fe10 con 0x7f1ee01669d0 2024-04-17T07:33:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.892+0000 7f1ee8200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee00ceb00 con 0x7f1ee0072630 2024-04-17T07:33:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.892+0000 7f1ee8200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee0166730 con 0x7f1ee0166350 2024-04-17T07:33:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.893+0000 7f1ee6200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1ee0072630 0x7f1ee00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.893+0000 7f1ee5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 0x7f1ee00cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.893+0000 7f1ee4e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f1ee01669d0 0x7f1ee0071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:41302/0 (socket says 172.21.15.67:41302) 2024-04-17T07:33:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.893+0000 7f1ee4e00640 1 -- 172.21.15.67:0/4287217858 learned_addr learned my addr 172.21.15.67:0/4287217858 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1edfe00640 1 -- 172.21.15.67:0/4287217858 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203111058 0 0) 0x7f1ee006fe10 con 0x7f1ee01669d0 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1edfe00640 1 -- 172.21.15.67:0/4287217858 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec8003520 con 0x7f1ee01669d0 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1ee5800640 1 -- 172.21.15.67:0/4287217858 >> v2:172.21.15.187:3300/0 conn(0x7f1ee0072630 msgr2=0x7f1ee00ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1ee5800640 1 --2- 172.21.15.67:0/4287217858 >> v2:172.21.15.187:3300/0 conn(0x7f1ee0072630 0x7f1ee00ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1ee5800640 1 -- 172.21.15.67:0/4287217858 >> v1:172.21.15.187:6789/0 conn(0x7f1ee01669d0 legacy=0x7f1ee0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.894+0000 7f1ee5800640 1 -- 172.21.15.67:0/4287217858 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee00ced30 con 0x7f1ee0166350 2024-04-17T07:33:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.896+0000 7f1ee5800640 1 --2- 172.21.15.67:0/4287217858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 0x7f1ee00cfdd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00097a0 tx=0x7f1ed002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=887b1876f11c4ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.896+0000 7f1edfe00640 1 -- 172.21.15.67:0/4287217858 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031780 con 0x7f1ee0166350 2024-04-17T07:33:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.896+0000 7f1edfe00640 1 -- 172.21.15.67:0/4287217858 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0037070 con 0x7f1ee0166350 2024-04-17T07:33:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.896+0000 7f1edfe00640 1 -- 172.21.15.67:0/4287217858 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031b20 con 0x7f1ee0166350 2024-04-17T07:33:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 -- 172.21.15.67:0/4287217858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 msgr2=0x7f1ee00cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 --2- 172.21.15.67:0/4287217858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 0x7f1ee00cfdd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00097a0 tx=0x7f1ed002b820 comp rx=0 tx=0).stop 2024-04-17T07:33:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 -- 172.21.15.67:0/4287217858 shutdown_connections 2024-04-17T07:33:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 --2- 172.21.15.67:0/4287217858 >> v2:172.21.15.187:3300/0 conn(0x7f1ee0072630 0x7f1ee00ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 --2- 172.21.15.67:0/4287217858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee0166350 0x7f1ee00cfdd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 -- 172.21.15.67:0/4287217858 >> 172.21.15.67:0/4287217858 conn(0x7f1ee0067c30 msgr2=0x7f1ee006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 -- 172.21.15.67:0/4287217858 shutdown_connections 2024-04-17T07:33:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.897+0000 7f1ee8200640 1 -- 172.21.15.67:0/4287217858 wait complete. 2024-04-17T07:33:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.899+0000 7f1ee8200640 1 Processor -- start 2024-04-17T07:33:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.899+0000 7f1ee8200640 1 -- start start 2024-04-17T07:33:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ee0166350 0x7f1ee014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee01669d0 0x7f1ee00b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00cf5d0 con 0x7f1ee0072630 2024-04-17T07:33:32.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00cf2f0 con 0x7f1ee0166350 2024-04-17T07:33:32.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee8200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee00cf8b0 con 0x7f1ee01669d0 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.900+0000 7f1ee5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56822/0 (socket says 172.21.15.67:56822) 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 -- 172.21.15.67:0/1294356403 learned_addr learned my addr 172.21.15.67:0/1294356403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee6200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee01669d0 0x7f1ee00b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee4e00640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ee0166350 0x7f1ee014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 -- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee01669d0 msgr2=0x7f1ee00b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee01669d0 0x7f1ee00b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 -- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ee0166350 msgr2=0x7f1ee014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:32.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ee0166350 0x7f1ee014e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 -- 172.21.15.67:0/1294356403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee014f3f0 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.901+0000 7f1ee5800640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00098c0 tx=0x7f1ed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.902+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031780 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.902+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0031920 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.902+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee014f6d0 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.902+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031cd0 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.902+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee014fc90 con 0x7f1ee0072630 2024-04-17T07:33:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.903+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee0068ca0 con 0x7f1ee0072630 2024-04-17T07:33:32.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.904+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f1ed0032410 con 0x7f1ee0072630 2024-04-17T07:33:32.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.905+0000 7f1edd600640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 0x7f1ec406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:32.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.906+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f1ed00adf50 con 0x7f1ee0072630 2024-04-17T07:33:32.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.906+0000 7f1ee4e00640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 0x7f1ec406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:32.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.907+0000 7f1ee4e00640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 0x7f1ec406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ee015f360 tx=0x7f1ed4009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:32.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:32.910+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1ed002f050 con 0x7f1ee0072630 2024-04-17T07:33:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.194+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1ee0072cd0 con 0x7f1ee0072630 2024-04-17T07:33:33.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.197+0000 7f1edd600640 1 -- 172.21.15.67:0/1294356403 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f1ed007c1b0 con 0x7f1ee0072630 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:33:33.199 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 msgr2=0x7f1ec406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 0x7f1ec406eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ee015f360 tx=0x7f1ed4009040 comp rx=0 tx=0).stop 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 msgr2=0x7f1ee014e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00098c0 tx=0x7f1ed00046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 shutdown_connections 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f1ec406ca10 0x7f1ec406eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.200+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1ee01669d0 0x7f1ee00b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.201+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1ee0166350 0x7f1ee014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.201+0000 7f1ee8200640 1 --2- 172.21.15.67:0/1294356403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ee0072630 0x7f1ee014e0c0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.201+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 >> 172.21.15.67:0/1294356403 conn(0x7f1ee0067c30 msgr2=0x7f1ee00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.201+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 shutdown_connections 2024-04-17T07:33:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.201+0000 7f1ee8200640 1 -- 172.21.15.67:0/1294356403 wait complete. 2024-04-17T07:33:33.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.443+0000 7f1885a00640 1 Processor -- start 2024-04-17T07:33:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.443+0000 7f1885a00640 1 -- start start 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880160280 0x7f188006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1880141ed0 0x7f1880142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18800cd4a0 con 0x7f188006b8c0 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f188015ce30 con 0x7f1880141ed0 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f18801607f0 con 0x7f1880160280 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f187fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f188006b8c0 0x7f18800cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56942/0 (socket says 172.21.15.67:56942) 2024-04-17T07:33:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f187fe00640 1 -- 172.21.15.67:0/2589820585 learned_addr learned my addr 172.21.15.67:0/2589820585 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.445+0000 7f1885000640 1 --2- 172.21.15.67:0/2589820585 >> v2:172.21.15.67:3300/0 conn(0x7f1880141ed0 0x7f1880142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f187ea00640 1 -- 172.21.15.67:0/2589820585 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395778834 0 0) 0x7f18800cd4a0 con 0x7f188006b8c0 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f187f400640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880160280 0x7f188006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f187ea00640 1 -- 172.21.15.67:0/2589820585 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1864003520 con 0x7f188006b8c0 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f1885000640 1 -- 172.21.15.67:0/2589820585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880160280 msgr2=0x7f188006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f1885000640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880160280 0x7f188006bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f1885000640 1 -- 172.21.15.67:0/2589820585 >> v1:172.21.15.67:6789/0 conn(0x7f188006b8c0 legacy=0x7f18800cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.446+0000 7f1885000640 1 -- 172.21.15.67:0/2589820585 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1880160ad0 con 0x7f1880141ed0 2024-04-17T07:33:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.447+0000 7f1885000640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880141ed0 0x7f1880142360 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1878009d90 tx=0x7f187802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a10a14f4b7fba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.447+0000 7f187ea00640 1 -- 172.21.15.67:0/2589820585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878039070 con 0x7f1880141ed0 2024-04-17T07:33:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.447+0000 7f187ea00640 1 -- 172.21.15.67:0/2589820585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187802cc60 con 0x7f1880141ed0 2024-04-17T07:33:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.447+0000 7f187ea00640 1 -- 172.21.15.67:0/2589820585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878034640 con 0x7f1880141ed0 2024-04-17T07:33:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 -- 172.21.15.67:0/2589820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880141ed0 msgr2=0x7f1880142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880141ed0 0x7f1880142360 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1878009d90 tx=0x7f187802beb0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 -- 172.21.15.67:0/2589820585 shutdown_connections 2024-04-17T07:33:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880141ed0 0x7f1880142360 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 --2- 172.21.15.67:0/2589820585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880160280 0x7f188006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 -- 172.21.15.67:0/2589820585 >> 172.21.15.67:0/2589820585 conn(0x7f1880067d10 msgr2=0x7f18800b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 -- 172.21.15.67:0/2589820585 shutdown_connections 2024-04-17T07:33:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.448+0000 7f1885a00640 1 -- 172.21.15.67:0/2589820585 wait complete. 2024-04-17T07:33:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.450+0000 7f1885a00640 1 Processor -- start 2024-04-17T07:33:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.450+0000 7f1885a00640 1 -- start start 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 0x7f1880145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 0x7f188013d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1880161090 con 0x7f1880160280 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f188006c1e0 con 0x7f188006b8c0 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f188013f9b0 con 0x7f1880141ed0 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51316/0 (socket says 172.21.15.67:51316) 2024-04-17T07:33:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.451+0000 7f1885000640 1 -- 172.21.15.67:0/4005736191 learned_addr learned my addr 172.21.15.67:0/4005736191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f187fe00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 0x7f1880145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f187f400640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 0x7f188013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 -- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 msgr2=0x7f188013d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 0x7f188013d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 -- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 msgr2=0x7f1880145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 0x7f1880145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 -- 172.21.15.67:0/4005736191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18800b1420 con 0x7f1880160280 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f187fe00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 0x7f1880145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f187f400640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 0x7f188013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.452+0000 7f1885000640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1878009d90 tx=0x7f1878004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.453+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878044020 con 0x7f1880160280 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.453+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187803f470 con 0x7f1880160280 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.453+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878039040 con 0x7f1880160280 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.453+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1880145870 con 0x7f1880160280 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.453+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18801f4300 con 0x7f1880160280 2024-04-17T07:33:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.454+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1880068a10 con 0x7f1880160280 2024-04-17T07:33:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.455+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f1878047020 con 0x7f1880160280 2024-04-17T07:33:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.456+0000 7f1877e00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 0x7f185c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.456+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f18780ae840 con 0x7f1880160280 2024-04-17T07:33:33.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.457+0000 7f187fe00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 0x7f185c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.458+0000 7f187fe00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 0x7f185c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1870005880 tx=0x7f1870009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.461+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f187804e030 con 0x7f1880160280 2024-04-17T07:33:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.703+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18800b2360 con 0x7f1880160280 2024-04-17T07:33:33.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.705+0000 7f1877e00640 1 -- 172.21.15.67:0/4005736191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7f187807cbd0 con 0x7f1880160280 2024-04-17T07:33:33.705 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:33.705 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:33.705 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 msgr2=0x7f185c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 0x7f185c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1870005880 tx=0x7f1870009040 comp rx=0 tx=0).stop 2024-04-17T07:33:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 msgr2=0x7f18801f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1878009d90 tx=0x7f1878004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 shutdown_connections 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1880160280 0x7f18801f4b00 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f185c06ca10 0x7f185c06eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1880141ed0 0x7f188013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 --2- 172.21.15.67:0/4005736191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f188006b8c0 0x7f1880145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.707+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 >> 172.21.15.67:0/4005736191 conn(0x7f1880067d10 msgr2=0x7f18800b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.708+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 shutdown_connections 2024-04-17T07:33:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:33.708+0000 7f1885a00640 1 -- 172.21.15.67:0/4005736191 wait complete. 2024-04-17T07:33:34.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:33:26.988125+0000', 'modified': '2024-04-17T07:33:30.097371+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8397}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8397': {'gid': 8397, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/996743914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 996743914}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 996743914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8397, 'qdb_cluster': [8397]} 2024-04-17T07:33:34.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:33:34.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T07:33:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.361+0000 7f1840c00640 1 Processor -- start 2024-04-17T07:33:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.361+0000 7f1840c00640 1 -- start start 2024-04-17T07:33:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.362+0000 7f1840c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.362+0000 7f1840c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f183c0cfa10 0x7f183c0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.362+0000 7f1840c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f183c163860 con 0x7f183c0d1090 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.362+0000 7f1840c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f183c0d03b0 con 0x7f183c0cfa10 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.362+0000 7f1840c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f183c0d1d40 con 0x7f183c15eca0 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.363+0000 7f1839e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.363+0000 7f1831400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f183c0cfa10 0x7f183c0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.363+0000 7f1839e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:59520/0 (socket says 172.21.15.67:59520) 2024-04-17T07:33:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.363+0000 7f1839e00640 1 -- 172.21.15.67:0/3748278677 learned_addr learned my addr 172.21.15.67:0/3748278677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.364+0000 7f1839e00640 1 -- 172.21.15.67:0/3748278677 >> v2:172.21.15.187:3300/0 conn(0x7f183c0cfa10 msgr2=0x7f183c0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.364+0000 7f1839e00640 1 --2- 172.21.15.67:0/3748278677 >> v2:172.21.15.187:3300/0 conn(0x7f183c0cfa10 0x7f183c0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.364+0000 7f1839e00640 1 -- 172.21.15.67:0/3748278677 >> v1:172.21.15.187:6789/0 conn(0x7f183c0d1090 legacy=0x7f183c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.364+0000 7f1839e00640 1 -- 172.21.15.67:0/3748278677 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c0d2c30 con 0x7f183c15eca0 2024-04-17T07:33:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.366+0000 7f1839e00640 1 --2- 172.21.15.67:0/3748278677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f182c009870 tx=0x7f182c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e39d6a55d8ee38d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.366+0000 7f1839400640 1 -- 172.21.15.67:0/3748278677 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c034070 con 0x7f183c15eca0 2024-04-17T07:33:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.366+0000 7f1839400640 1 -- 172.21.15.67:0/3748278677 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c030840 con 0x7f183c15eca0 2024-04-17T07:33:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.366+0000 7f1839400640 1 -- 172.21.15.67:0/3748278677 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c030c10 con 0x7f183c15eca0 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 -- 172.21.15.67:0/3748278677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 msgr2=0x7f183c0d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 --2- 172.21.15.67:0/3748278677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f182c009870 tx=0x7f182c02b980 comp rx=0 tx=0).stop 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 -- 172.21.15.67:0/3748278677 shutdown_connections 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 --2- 172.21.15.67:0/3748278677 >> v2:172.21.15.187:3300/0 conn(0x7f183c0cfa10 0x7f183c0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 --2- 172.21.15.67:0/3748278677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c15eca0 0x7f183c0d39f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 -- 172.21.15.67:0/3748278677 >> 172.21.15.67:0/3748278677 conn(0x7f183c067600 msgr2=0x7f183c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 -- 172.21.15.67:0/3748278677 shutdown_connections 2024-04-17T07:33:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.367+0000 7f1840c00640 1 -- 172.21.15.67:0/3748278677 wait complete. 2024-04-17T07:33:34.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.369+0000 7f1840c00640 1 Processor -- start 2024-04-17T07:33:34.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.370+0000 7f1840c00640 1 -- start start 2024-04-17T07:33:34.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0cfa10 0x7f183c0c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c0d1090 0x7f183c0b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0d34d0 con 0x7f183c15eca0 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0d31f0 con 0x7f183c0cfa10 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f1840c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f183c15f080 con 0x7f183c0d1090 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f183a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f183a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51320/0 (socket says 172.21.15.67:51320) 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.371+0000 7f183a800640 1 -- 172.21.15.67:0/2409735703 learned_addr learned my addr 172.21.15.67:0/2409735703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f1831400640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c0d1090 0x7f183c0b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f1839e00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0cfa10 0x7f183c0c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 -- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c0d1090 msgr2=0x7f183c0b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c0d1090 0x7f183c0b6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 -- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0cfa10 msgr2=0x7f183c0c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0cfa10 0x7f183c0c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 -- 172.21.15.67:0/2409735703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c0cbea0 con 0x7f183c15eca0 2024-04-17T07:33:34.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.372+0000 7f183a800640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f183400c7a0 tx=0x7f183400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.373+0000 7f1832a00640 1 -- 172.21.15.67:0/2409735703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834003b70 con 0x7f183c15eca0 2024-04-17T07:33:34.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.373+0000 7f1832a00640 1 -- 172.21.15.67:0/2409735703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1834003d30 con 0x7f183c15eca0 2024-04-17T07:33:34.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.373+0000 7f1832a00640 1 -- 172.21.15.67:0/2409735703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183400aa90 con 0x7f183c15eca0 2024-04-17T07:33:34.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.373+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183c0c8a80 con 0x7f183c15eca0 2024-04-17T07:33:34.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.373+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183c0c9040 con 0x7f183c15eca0 2024-04-17T07:33:34.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.375+0000 7f1832a00640 1 -- 172.21.15.67:0/2409735703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f183400ac30 con 0x7f183c15eca0 2024-04-17T07:33:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.376+0000 7f1832a00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 0x7f181806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.376+0000 7f1832a00640 1 -- 172.21.15.67:0/2409735703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f183408bdd0 con 0x7f183c15eca0 2024-04-17T07:33:34.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.377+0000 7f184399f740 1 Processor -- start 2024-04-17T07:33:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.377+0000 7f184399f740 1 -- start start 2024-04-17T07:33:34.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.377+0000 7f1839e00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 0x7f181806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad0339d8e0 0x55ad033cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad03309980 con 0x55ad033eb410 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ad032b0220 con 0x55ad033a1490 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ad03305940 con 0x55ad0339d8e0 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51332/0 (socket says 172.21.15.67:51332) 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 -- 172.21.15.67:0/1231146825 learned_addr learned my addr 172.21.15.67:0/1231146825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1831400640 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 -- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad0339d8e0 msgr2=0x55ad033cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad0339d8e0 0x55ad033cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 -- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 msgr2=0x55ad033a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033a2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.378+0000 7f1839e00640 1 -- 172.21.15.67:0/1231146825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad0301c2d0 con 0x55ad033eb410 2024-04-17T07:33:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.379+0000 7f1839e00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 0x7f181806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f182c036000 tx=0x7f182c008030 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.379+0000 7f1839e00640 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f182c009630 tx=0x7f182c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838534c4ae8420c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.379+0000 7f182a600640 1 -- 172.21.15.67:0/1231146825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c053070 con 0x55ad033eb410 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.379+0000 7f182a600640 1 -- 172.21.15.67:0/1231146825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c0398a0 con 0x55ad033eb410 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.379+0000 7f182a600640 1 -- 172.21.15.67:0/1231146825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c034040 con 0x55ad033eb410 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 -- 172.21.15.67:0/1231146825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 msgr2=0x55ad033a2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f182c009630 tx=0x7f182c009240 comp rx=0 tx=0).stop 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 -- 172.21.15.67:0/1231146825 shutdown_connections 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad0339d8e0 0x55ad033cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033a2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 --2- 172.21.15.67:0/1231146825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033eb410 0x55ad033a2a40 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 -- 172.21.15.67:0/1231146825 >> 172.21.15.67:0/1231146825 conn(0x55ad0339fba0 msgr2=0x55ad033e8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 -- 172.21.15.67:0/1231146825 shutdown_connections 2024-04-17T07:33:34.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.380+0000 7f184399f740 1 -- 172.21.15.67:0/1231146825 wait complete. 2024-04-17T07:33:34.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.381+0000 7f184399f740 1 -- start start 2024-04-17T07:33:34.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.381+0000 7f184399f740 1 Processor -- start 2024-04-17T07:33:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad033a19a0 con 0x55ad033c5d00 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ad032c3590 con 0x55ad033a1490 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ad033a1c80 con 0x55ad033eb410 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f184399f740 10 client.0 _ll_register_callbacks cb 0x55ad0301bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f183a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f183a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51344/0 (socket says 172.21.15.67:51344) 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.382+0000 7f183a800640 1 -- 172.21.15.67:0/3951123505 learned_addr learned my addr 172.21.15.67:0/3951123505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f1839e00640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 -- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 msgr2=0x55ad033f92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 -- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 msgr2=0x55ad033cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 -- 172.21.15.67:0/3951123505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad030e66b0 con 0x55ad033c5d00 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f183a800640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f183400a430 tx=0x7f1834059fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9156d6acf2f6f5a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f181ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183405dc50 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f1834062080 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.383+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f182c0398a0 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f1834062510 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 10 client.8535 fetch_fsmap learned FSMap version 68 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55ad033edc10 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c0398a0 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 68) ==== 3495+0+0 (secure 0 0 0) 0x7f1834058cf0 con 0x55ad033c5d00 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 10 client.8535 fetch_fsmap finished waiting for FSMap version 68 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 10 client.8535 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 20 client.8535 populate_metadata read hostname 'smithi067' 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cc470 0x55ad033efa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 4 client.8535 mds_command: new command op to 8397 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:33:34.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 --> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad033173a0 con 0x55ad033cc470 2024-04-17T07:33:34.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.384+0000 7f1839e00640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cc470 0x55ad033efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.385+0000 7f1839e00640 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cc470 0x55ad033efa70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9b8bf237939da94 server_cookie=4575326c0caf2510 in_seq=0 out_seq=0 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.385+0000 7f181ea00640 10 client.8535 ms_handle_connect on v2:172.21.15.67:6834/996743914 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f181ea00640 1 -- 172.21.15.67:0/3951123505 <== mds.0 v2:172.21.15.67:6834/996743914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad033173a0 con 0x55ad033cc470 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f181ea00640 10 client.8535 handle_command_reply: tid=0 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 client.8535 shutdown 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 msgr2=0x55ad033ee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f183400a430 tx=0x7f1834059fa0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 shutdown_connections 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cc470 0x55ad033efa70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033c5d00 0x55ad033ee130 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033f92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 --2- 172.21.15.67:0/3951123505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033a1490 0x55ad033cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f181ea00640 0 client.8535 ms_handle_reset on v2:172.21.15.67:6834/996743914 2024-04-17T07:33:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 >> 172.21.15.67:0/3951123505 conn(0x55ad0339d8e0 msgr2=0x55ad0339e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 shutdown_connections 2024-04-17T07:33:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.386+0000 7f184399f740 1 -- 172.21.15.67:0/3951123505 wait complete. 2024-04-17T07:33:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.387+0000 7f184399f740 20 client.8535 trim_cache size 0 max 16384 2024-04-17T07:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.408+0000 7f184399f740 1 Processor -- start 2024-04-17T07:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.408+0000 7f184399f740 1 -- start start 2024-04-17T07:33:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad033a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0b5c00 con 0x55ad033a1490 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f183c162e20 con 0x55ad033c5d00 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f183c0b55b0 con 0x55ad033eb410 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.409+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51350/0 (socket says 172.21.15.67:51350) 2024-04-17T07:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 -- 172.21.15.67:0/3733736892 learned_addr learned my addr 172.21.15.67:0/3733736892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1831400640 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad033a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f183a800640 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 -- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 msgr2=0x55ad033e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 -- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 msgr2=0x55ad033a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad033a3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 -- 172.21.15.67:0/3733736892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad033eabb0 con 0x55ad033a1490 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.410+0000 7f1839e00640 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f182c009960 tx=0x7f182c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bce73abef11fad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f1829c00640 1 -- 172.21.15.67:0/3733736892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c053070 con 0x55ad033a1490 2024-04-17T07:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f1829c00640 1 -- 172.21.15.67:0/3733736892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c054040 con 0x55ad033a1490 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f1829c00640 1 -- 172.21.15.67:0/3733736892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c034040 con 0x55ad033a1490 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 -- 172.21.15.67:0/3733736892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 msgr2=0x55ad033efde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f182c009960 tx=0x7f182c030000 comp rx=0 tx=0).stop 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 -- 172.21.15.67:0/3733736892 shutdown_connections 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad033a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 --2- 172.21.15.67:0/3733736892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad033efde0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 -- 172.21.15.67:0/3733736892 >> 172.21.15.67:0/3733736892 conn(0x55ad0339d8e0 msgr2=0x55ad0340cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 -- 172.21.15.67:0/3733736892 shutdown_connections 2024-04-17T07:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.411+0000 7f184399f740 1 -- 172.21.15.67:0/3733736892 wait complete. 2024-04-17T07:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.412+0000 7f184399f740 1 -- start start 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.412+0000 7f184399f740 1 Processor -- start 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad0340c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad033cc2f0 con 0x55ad033a1490 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55ad033f9870 con 0x55ad033c5d00 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55ad033cc940 con 0x55ad033eb410 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f184399f740 10 client.0 _ll_register_callbacks cb 0x7f183400af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:33:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f1839e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51358/0 (socket says 172.21.15.67:51358) 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.413+0000 7f1839e00640 1 -- 172.21.15.67:0/3825810800 learned_addr learned my addr 172.21.15.67:0/3825810800 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 -- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 msgr2=0x55ad033e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 -- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 msgr2=0x55ad0340c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad0340c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 -- 172.21.15.67:0/3825810800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad033ea3e0 con 0x55ad033a1490 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f1839e00640 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f182c0090b0 tx=0x7f182c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3390fbb298b5d14a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f181ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c063060 con 0x55ad033a1490 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f182c004030 con 0x55ad033a1490 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.414+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f182c054040 con 0x55ad033a1490 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f182c069050 con 0x55ad033a1490 2024-04-17T07:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 10 client.8541 fetch_fsmap learned FSMap version 68 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55ad033e9ed0 con 0x55ad033a1490 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c054040 con 0x55ad033a1490 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 68) ==== 3495+0+0 (secure 0 0 0) 0x7f182c06c020 con 0x55ad033a1490 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 10 client.8541 fetch_fsmap finished waiting for FSMap version 68 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 10 client.8541 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 20 client.8541 populate_metadata read hostname 'smithi067' 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cb990 0x55ad033eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 4 client.8541 mds_command: new command op to 8397 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 --> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] -- command(tid 0: {"prefix": "status"}) -- 0x55ad03311210 con 0x55ad033cb990 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.415+0000 7f1831400640 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cb990 0x55ad033eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f1831400640 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cb990 0x55ad033eb7f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751e9eb80613c991 server_cookie=1f5738693114accc in_seq=0 out_seq=0 2024-04-17T07:33:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f181ea00640 10 client.8541 ms_handle_connect on v2:172.21.15.67:6834/996743914 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f181ea00640 1 -- 172.21.15.67:0/3825810800 <== mds.0 v2:172.21.15.67:6834/996743914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55ad03311210 con 0x55ad033cb990 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f181ea00640 10 client.8541 handle_command_reply: tid=0 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f184399f740 1 client.8541 shutdown 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 msgr2=0x55ad03462bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.416+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f182c0090b0 tx=0x7f182c004730 comp rx=0 tx=0).stop 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 shutdown_connections 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:6834/996743914,v1:172.21.15.67:6835/996743914] conn(0x55ad033cb990 0x55ad033eb7f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55ad033eb410 0x55ad033e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55ad033c5d00 0x55ad0340c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 --2- 172.21.15.67:0/3825810800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad033a1490 0x55ad03462bc0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 >> 172.21.15.67:0/3825810800 conn(0x55ad0339d8e0 msgr2=0x55ad033a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f181ea00640 0 client.8541 ms_handle_reset on v2:172.21.15.67:6834/996743914 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 shutdown_connections 2024-04-17T07:33:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 1 -- 172.21.15.67:0/3825810800 wait complete. 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f184399f740 20 client.8541 trim_cache size 0 max 16384 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8397, 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8550684579999999, 2024-04-17T07:33:34.421 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 68, 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 142, 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 142, 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.266999156000001 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 msgr2=0x7f181806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 0x7f181806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f182c036000 tx=0x7f182c008030 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 msgr2=0x7f183c149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.417+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f183400c7a0 tx=0x7f183400cc60 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 shutdown_connections 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183c15eca0 0x7f183c149a70 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f181806ca10 0x7f181806eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f183c0d1090 0x7f183c0b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 --2- 172.21.15.67:0/2409735703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0cfa10 0x7f183c0c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 >> 172.21.15.67:0/2409735703 conn(0x7f183c067600 msgr2=0x7f183c0ba080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 shutdown_connections 2024-04-17T07:33:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.418+0000 7f1840c00640 1 -- 172.21.15.67:0/2409735703 wait complete. 2024-04-17T07:33:34.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:33:34.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.650+0000 7fa821000640 1 Processor -- start 2024-04-17T07:33:34.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.650+0000 7fa821000640 1 -- start start 2024-04-17T07:33:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.652+0000 7fa821000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa81c06f670 0x7fa81c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.652+0000 7fa821000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa81c0cd020 0x7fa81c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.652+0000 7fa821000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c06ff90 con 0x7fa81c071d10 2024-04-17T07:33:34.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.652+0000 7fa821000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa81c0ceb00 con 0x7fa81c06f670 2024-04-17T07:33:34.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.652+0000 7fa821000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa81c0cede0 con 0x7fa81c0cd020 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa81c06f670 0x7fa81c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa81a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa81c071d10 0x7fa81c072160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56948/0 (socket says 172.21.15.67:56948) 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa81a800640 1 -- 172.21.15.67:0/3239744453 learned_addr learned my addr 172.21.15.67:0/3239744453 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819400640 1 --2- 172.21.15.67:0/3239744453 >> v2:172.21.15.187:3300/0 conn(0x7fa81c0cd020 0x7fa81c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819e00640 1 -- 172.21.15.67:0/3239744453 >> v2:172.21.15.187:3300/0 conn(0x7fa81c0cd020 msgr2=0x7fa81c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819e00640 1 --2- 172.21.15.67:0/3239744453 >> v2:172.21.15.187:3300/0 conn(0x7fa81c0cd020 0x7fa81c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819e00640 1 -- 172.21.15.67:0/3239744453 >> v1:172.21.15.67:6789/0 conn(0x7fa81c071d10 legacy=0x7fa81c072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa819e00640 1 -- 172.21.15.67:0/3239744453 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c0cf0c0 con 0x7fa81c06f670 2024-04-17T07:33:34.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.653+0000 7fa818a00640 1 -- 172.21.15.67:0/3239744453 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317545586 0 0) 0x7fa81c06ff90 con 0x7fa81c071d10 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa819e00640 1 --2- 172.21.15.67:0/3239744453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c06f670 0x7fa81c06fa50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0cce60 tx=0x7fa80c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bbaca6d99aaf81 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa818a00640 1 -- 172.21.15.67:0/3239744453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c039070 con 0x7fa81c06f670 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa818a00640 1 -- 172.21.15.67:0/3239744453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80c02cc80 con 0x7fa81c06f670 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa818a00640 1 -- 172.21.15.67:0/3239744453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c034660 con 0x7fa81c06f670 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa821000640 1 -- 172.21.15.67:0/3239744453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c06f670 msgr2=0x7fa81c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.655+0000 7fa821000640 1 --2- 172.21.15.67:0/3239744453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c06f670 0x7fa81c06fa50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0cce60 tx=0x7fa80c02bed0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 -- 172.21.15.67:0/3239744453 shutdown_connections 2024-04-17T07:33:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 --2- 172.21.15.67:0/3239744453 >> v2:172.21.15.187:3300/0 conn(0x7fa81c0cd020 0x7fa81c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 --2- 172.21.15.67:0/3239744453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c06f670 0x7fa81c06fa50 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 -- 172.21.15.67:0/3239744453 >> 172.21.15.67:0/3239744453 conn(0x7fa81c06bf50 msgr2=0x7fa81c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 -- 172.21.15.67:0/3239744453 shutdown_connections 2024-04-17T07:33:34.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.656+0000 7fa821000640 1 -- 172.21.15.67:0/3239744453 wait complete. 2024-04-17T07:33:34.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.657+0000 7fa821000640 1 Processor -- start 2024-04-17T07:33:34.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.658+0000 7fa821000640 1 -- start start 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.658+0000 7fa821000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c06f670 0x7fa81c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.658+0000 7fa821000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa821000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 0x7fa81c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa821000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c0ce720 con 0x7fa81c071d10 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa821000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c0cf680 con 0x7fa81c0cd020 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa821000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa81c0ce8a0 con 0x7fa81c06f670 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51364/0 (socket says 172.21.15.67:51364) 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 -- 172.21.15.67:0/4138154054 learned_addr learned my addr 172.21.15.67:0/4138154054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa81a800640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 0x7fa81c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819e00640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c06f670 0x7fa81c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 -- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c06f670 msgr2=0x7fa81c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c06f670 0x7fa81c0c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 -- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 msgr2=0x7fa81c155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 0x7fa81c155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa819400640 1 -- 172.21.15.67:0/4138154054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c0c5e70 con 0x7fa81c071d10 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.659+0000 7fa81a800640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 0x7fa81c155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa819400640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa80800d830 tx=0x7fa80800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80800b660 con 0x7fa81c071d10 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa808019070 con 0x7fa81c071d10 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80800ba20 con 0x7fa81c071d10 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c0c6150 con 0x7fa81c071d10 2024-04-17T07:33:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.660+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c0c6710 con 0x7fa81c071d10 2024-04-17T07:33:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.661+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c068b90 con 0x7fa81c071d10 2024-04-17T07:33:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.662+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7fa80801e460 con 0x7fa81c071d10 2024-04-17T07:33:34.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.663+0000 7fa812000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 0x7fa7fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:34.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.663+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7fa80808c3b0 con 0x7fa81c071d10 2024-04-17T07:33:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.664+0000 7fa819e00640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 0x7fa7fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.665+0000 7fa819e00640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 0x7fa7fc06eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa80c036000 tx=0x7fa80c030000 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.668+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa80805a6a0 con 0x7fa81c071d10 2024-04-17T07:33:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.917+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa81c0c6b10 con 0x7fa81c071d10 2024-04-17T07:33:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.918+0000 7fa812000640 1 -- 172.21.15.67:0/4138154054 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) ==== 53+0+73 (secure 0 0 0) 0x7fa808059fa0 con 0x7fa81c071d10 2024-04-17T07:33:34.918 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 msgr2=0x7fa7fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 0x7fa7fc06eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa80c036000 tx=0x7fa80c030000 comp rx=0 tx=0).stop 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 msgr2=0x7fa81c0c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa80800d830 tx=0x7fa80800dcf0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 shutdown_connections 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa81c0cd020 0x7fa81c155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c071d10 0x7fa81c0c5420 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fa7fc06ca10 0x7fa7fc06eed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 --2- 172.21.15.67:0/4138154054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa81c06f670 0x7fa81c0c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 >> 172.21.15.67:0/4138154054 conn(0x7fa81c06bf50 msgr2=0x7fa81c0d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 shutdown_connections 2024-04-17T07:33:34.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:34.921+0000 7fa821000640 1 -- 172.21.15.67:0/4138154054 wait complete. 2024-04-17T07:33:34.941 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:33:34.941 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:33:34.941 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:33:34.941 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:33:34.941 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:34.942 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:33:34.942 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:33:34.942 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:33:34.942 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:33:34.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:34.942 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:33:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41429sec preferred_lft 41429sec 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:33:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:33:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:33:34.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:34.997 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:33:35.086 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:33:35.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:35.091 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:33:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:33:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:33:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:33:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:33:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:33:35.147 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:33:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:33:35.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:33:35.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:35.413 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:33:35.465 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:33:35.465 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:33:35.465 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:33:35.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:35.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:33:35.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:33:35.568 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:35.624 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:33:35.624 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:33:35.624 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:35.675 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:33:35.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:35.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:33:35.815 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:33:35.816 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:33:35.853 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:33:35.853 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:33:35.853 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:33:35.854 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:33:35.854 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:33:35.858 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:33:35.858 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:33:35.858 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:33:35.858 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:33:35.858 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:33:35.862 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:33:35.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:35.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:35.950 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:33:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:33:36.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:33:36.082 INFO:teuthology.orchestra.run.smithi067.stdout:client6413 2024-04-17T07:33:36.083 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:36.083 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:36.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:33:36.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.359+0000 7f3dfca00640 1 Processor -- start 2024-04-17T07:33:36.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.359+0000 7f3dfca00640 1 -- start start 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.360+0000 7f3dfca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3df80b2ad0 0x7f3df80720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3dfca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 0x7f3df815b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3dfca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df80cf470 con 0x7f3df80b2eb0 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3dfca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3df80ce6b0 con 0x7f3df80b2ad0 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3dfca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3df80ce990 con 0x7f3df80afef0 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3df80b2ad0 0x7f3df80720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3df80b2ad0 0x7f3df80720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51366/0 (socket says 172.21.15.67:51366) 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df6800640 1 -- 172.21.15.67:0/231649054 learned_addr learned my addr 172.21.15.67:0/231649054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3dede00640 1 --1- 172.21.15.67:0/231649054 >> v1:172.21.15.67:6789/0 conn(0x7f3df80b2eb0 0x7f3df80af7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56950/0 (socket says 172.21.15.67:56950) 2024-04-17T07:33:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df7200640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 0x7f3df815b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df6800640 1 -- 172.21.15.67:0/231649054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 msgr2=0x7f3df815b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df6800640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 0x7f3df815b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.362+0000 7f3df6800640 1 -- 172.21.15.67:0/231649054 >> v1:172.21.15.67:6789/0 conn(0x7f3df80b2eb0 legacy=0x7f3df80af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.362+0000 7f3df6800640 1 -- 172.21.15.67:0/231649054 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df80cec70 con 0x7f3df80b2ad0 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.362+0000 7f3df7200640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 0x7f3df815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.361+0000 7f3df5e00640 1 -- 172.21.15.67:0/231649054 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184055260 0 0) 0x7f3df80cf470 con 0x7f3df80b2eb0 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.363+0000 7f3df6800640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80b2ad0 0x7f3df80720d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6556ceeeb72924e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.363+0000 7f3df5e00640 1 -- 172.21.15.67:0/231649054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de802cb00 con 0x7f3df80b2ad0 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.363+0000 7f3df5e00640 1 -- 172.21.15.67:0/231649054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de802cca0 con 0x7f3df80b2ad0 2024-04-17T07:33:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.363+0000 7f3df5e00640 1 -- 172.21.15.67:0/231649054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8034610 con 0x7f3df80b2ad0 2024-04-17T07:33:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 -- 172.21.15.67:0/231649054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80b2ad0 msgr2=0x7f3df80720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80b2ad0 0x7f3df80720d0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de802bf60 comp rx=0 tx=0).stop 2024-04-17T07:33:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 -- 172.21.15.67:0/231649054 shutdown_connections 2024-04-17T07:33:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80afef0 0x7f3df815b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 --2- 172.21.15.67:0/231649054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80b2ad0 0x7f3df80720d0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 -- 172.21.15.67:0/231649054 >> 172.21.15.67:0/231649054 conn(0x7f3df806bef0 msgr2=0x7f3df80b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 -- 172.21.15.67:0/231649054 shutdown_connections 2024-04-17T07:33:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.364+0000 7f3dfca00640 1 -- 172.21.15.67:0/231649054 wait complete. 2024-04-17T07:33:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.366+0000 7f3dfca00640 1 Processor -- start 2024-04-17T07:33:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.366+0000 7f3dfca00640 1 -- start start 2024-04-17T07:33:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df80b2ad0 0x7f3df813e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80b2eb0 0x7f3df8156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8072790 con 0x7f3df80afef0 2024-04-17T07:33:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8072610 con 0x7f3df80b2ad0 2024-04-17T07:33:36.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3dfca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3df815bcd0 con 0x7f3df80b2eb0 2024-04-17T07:33:36.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51382/0 (socket says 172.21.15.67:51382) 2024-04-17T07:33:36.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 -- 172.21.15.67:0/2852372247 learned_addr learned my addr 172.21.15.67:0/2852372247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 -- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80b2eb0 msgr2=0x7f3df8156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80b2eb0 0x7f3df8156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.367+0000 7f3df6800640 1 -- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df80b2ad0 msgr2=0x7f3df813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3dede00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df80b2ad0 0x7f3df813e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3df6800640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df80b2ad0 0x7f3df813e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3df6800640 1 -- 172.21.15.67:0/2852372247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df81548e0 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3df6800640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8043020 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de802ce50 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8009510 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.368+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df813d7f0 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.369+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df813dd70 con 0x7f3df80afef0 2024-04-17T07:33:36.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.370+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df8068a10 con 0x7f3df80afef0 2024-04-17T07:33:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.371+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7f3de8046020 con 0x7f3df80afef0 2024-04-17T07:33:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.371+0000 7f3def400640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 0x7f3dd406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.372+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f3de80ae4e0 con 0x7f3df80afef0 2024-04-17T07:33:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.372+0000 7f3dede00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 0x7f3dd406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.373+0000 7f3dede00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 0x7f3dd406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3df813cb00 tx=0x7f3de0009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.376+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3de804d030 con 0x7f3df80afef0 2024-04-17T07:33:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.608+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 --> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f3df80cfb30 con 0x7f3dd406ca10 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.613+0000 7f3def400640 1 -- 172.21.15.67:0/2852372247 <== mgr.6221 v2:172.21.15.187:6832/1274719646 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f3df80cfb30 con 0x7f3dd406ca10 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:33:36.613 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:33:36.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.615+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 msgr2=0x7f3dd406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.615+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 0x7f3dd406eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3df813cb00 tx=0x7f3de0009040 comp rx=0 tx=0).stop 2024-04-17T07:33:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.615+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 msgr2=0x7f3df81543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.615+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de8004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 shutdown_connections 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f3dd406ca10 0x7f3dd406eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3df80b2eb0 0x7f3df8156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df80b2ad0 0x7f3df813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 --2- 172.21.15.67:0/2852372247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df80afef0 0x7f3df81543a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 >> 172.21.15.67:0/2852372247 conn(0x7f3df806bef0 msgr2=0x7f3df8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 shutdown_connections 2024-04-17T07:33:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.616+0000 7f3dfca00640 1 -- 172.21.15.67:0/2852372247 wait complete. 2024-04-17T07:33:36.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:33:36.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.857+0000 7febc5400640 1 Processor -- start 2024-04-17T07:33:36.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.857+0000 7febc5400640 1 -- start start 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.859+0000 7febc5400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 0x7febc006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.859+0000 7febc5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.859+0000 7febc5400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febc0146360 con 0x7febc0145f80 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.859+0000 7febc5400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7febc00727a0 con 0x7febc00cd920 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.859+0000 7febc5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7febc0071e70 con 0x7febc006c0a0 2024-04-17T07:33:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.860+0000 7febbf400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 0x7febc006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.860+0000 7febc4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.860+0000 7febbfe00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7febc0145f80 0x7febc00cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:46656/0 (socket says 172.21.15.67:46656) 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.860+0000 7febbfe00640 1 -- 172.21.15.67:0/2349895756 learned_addr learned my addr 172.21.15.67:0/2349895756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.861+0000 7febc4a00640 1 -- 172.21.15.67:0/2349895756 >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 msgr2=0x7febc006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.861+0000 7febc4a00640 1 --2- 172.21.15.67:0/2349895756 >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 0x7febc006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.861+0000 7febc4a00640 1 -- 172.21.15.67:0/2349895756 >> v1:172.21.15.187:6789/0 conn(0x7febc0145f80 legacy=0x7febc00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.861+0000 7febc4a00640 1 -- 172.21.15.67:0/2349895756 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc0072150 con 0x7febc006c0a0 2024-04-17T07:33:36.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.861+0000 7febbf400640 1 --2- 172.21.15.67:0/2349895756 >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 0x7febc006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:33:36.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.863+0000 7febc4a00640 1 --2- 172.21.15.67:0/2349895756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc006e1c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7febb8009d90 tx=0x7febb802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1903eca63d72d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.863+0000 7febbea00640 1 -- 172.21.15.67:0/2349895756 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8039070 con 0x7febc006c0a0 2024-04-17T07:33:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.863+0000 7febbea00640 1 -- 172.21.15.67:0/2349895756 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb802cbc0 con 0x7febc006c0a0 2024-04-17T07:33:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.863+0000 7febbea00640 1 -- 172.21.15.67:0/2349895756 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8034440 con 0x7febc006c0a0 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 -- 172.21.15.67:0/2349895756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 msgr2=0x7febc006e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 --2- 172.21.15.67:0/2349895756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc006e1c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7febb8009d90 tx=0x7febb802be10 comp rx=0 tx=0).stop 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 -- 172.21.15.67:0/2349895756 shutdown_connections 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 --2- 172.21.15.67:0/2349895756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc006e1c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 --2- 172.21.15.67:0/2349895756 >> v2:172.21.15.187:3300/0 conn(0x7febc00cd920 0x7febc006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 -- 172.21.15.67:0/2349895756 >> 172.21.15.67:0/2349895756 conn(0x7febc00b2190 msgr2=0x7febc00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:36.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 -- 172.21.15.67:0/2349895756 shutdown_connections 2024-04-17T07:33:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.864+0000 7febc5400640 1 -- 172.21.15.67:0/2349895756 wait complete. 2024-04-17T07:33:36.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.866+0000 7febc5400640 1 Processor -- start 2024-04-17T07:33:36.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.866+0000 7febc5400640 1 -- start start 2024-04-17T07:33:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc01e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 0x7febc01f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7febc0072c30 con 0x7febc0145f80 2024-04-17T07:33:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7febc006c530 con 0x7febc00cd920 2024-04-17T07:33:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7febc0072db0 con 0x7febc006c0a0 2024-04-17T07:33:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51384/0 (socket says 172.21.15.67:51384) 2024-04-17T07:33:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.867+0000 7febc4a00640 1 -- 172.21.15.67:0/2446396879 learned_addr learned my addr 172.21.15.67:0/2446396879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febbfe00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc01e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febbf400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 0x7febc01f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 -- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 msgr2=0x7febc01e76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc01e76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 -- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 msgr2=0x7febc01f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 0x7febc01f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 -- 172.21.15.67:0/2446396879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc01f8320 con 0x7febc0145f80 2024-04-17T07:33:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febbf400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 0x7febc01f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.868+0000 7febc4a00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7febb8009d90 tx=0x7febb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.869+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8039050 con 0x7febc0145f80 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.869+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb8045070 con 0x7febc0145f80 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.869+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc01f8600 con 0x7febc0145f80 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.869+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8009450 con 0x7febc0145f80 2024-04-17T07:33:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.869+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc0144df0 con 0x7febc0145f80 2024-04-17T07:33:36.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.870+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc0068a10 con 0x7febc0145f80 2024-04-17T07:33:36.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.871+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 68) ==== 91142+0+0 (secure 0 0 0) 0x7febb803e850 con 0x7febc0145f80 2024-04-17T07:33:36.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.873+0000 7febb7e00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 0x7feba006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.873+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7febb80aeb50 con 0x7febc0145f80 2024-04-17T07:33:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.873+0000 7febbfe00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 0x7feba006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:36.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.874+0000 7febbfe00640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 0x7feba006eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febc01f78a0 tx=0x7febb0009040 comp rx=0 tx=0).ready entity=mgr.6221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:36.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:36.877+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7febb8032e20 con 0x7febc0145f80 2024-04-17T07:33:37.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.143+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7febc00cf4c0 con 0x7febc0145f80 2024-04-17T07:33:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.624+0000 7febb7e00640 1 -- 172.21.15.67:0/2446396879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) ==== 92+0+0 (secure 0 0 0) 0x7febb807ced0 con 0x7febc0145f80 2024-04-17T07:33:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.628+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 msgr2=0x7feba006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.628+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 0x7feba006eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febc01f78a0 tx=0x7febb0009040 comp rx=0 tx=0).stop 2024-04-17T07:33:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.628+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 msgr2=0x7febc013d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.628+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7febb8009d90 tx=0x7febb8004a40 comp rx=0 tx=0).stop 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 shutdown_connections 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febc0145f80 0x7febc013d1d0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febc00cd920 0x7febc01f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7feba006ca10 0x7feba006eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 --2- 172.21.15.67:0/2446396879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febc006c0a0 0x7febc01e76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 >> 172.21.15.67:0/2446396879 conn(0x7febc00b2190 msgr2=0x7febc006eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:37.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 shutdown_connections 2024-04-17T07:33:37.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.629+0000 7febc5400640 1 -- 172.21.15.67:0/2446396879 wait complete. 2024-04-17T07:33:37.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:33:37.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.864+0000 7f4a23000640 1 Processor -- start 2024-04-17T07:33:37.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.865+0000 7f4a23000640 1 -- start start 2024-04-17T07:33:37.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.866+0000 7f4a23000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a1c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.866+0000 7f4a23000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1c163860 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.866+0000 7f4a23000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1c0d1d40 con 0x7f4a1c0d17a0 2024-04-17T07:33:37.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.866+0000 7f4a23000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1c0d2c30 con 0x7f4a1c15eca0 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.866+0000 7f4a13400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4a1c0cfa10 0x7f4a1c0d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56958/0 (socket says 172.21.15.67:56958) 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a13400640 1 -- 172.21.15.67:0/3108920150 learned_addr learned my addr 172.21.15.67:0/3108920150 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1be00640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a1c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074532017 0 0) 0x7f4a1c163860 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a00003520 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3334009492 0 0) 0x7f4a00003520 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 msgr2=0x7f4a1c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a1c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 >> v1:172.21.15.187:6789/0 conn(0x7f4a1c0d17a0 legacy=0x7f4a1c0b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c0d2f10 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.867+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4a0c001670 con 0x7f4a1c0cfa10 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 >> v1:172.21.15.67:6789/0 conn(0x7f4a1c0cfa10 legacy=0x7f4a1c0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a000039e0 0x7f4a00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a00004d40 0x7f4a000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c0d2c30 con 0x7f4a00004d40 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0000baa0 con 0x7f4a000039e0 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0000bca0 con 0x7f4a1c15eca0 2024-04-17T07:33:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a21000640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a00004d40 0x7f4a000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a1be00640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a000039e0 0x7f4a00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.868+0000 7f4a13400640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.869+0000 7f4a21000640 1 -- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 msgr2=0x7f4a00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.869+0000 7f4a21000640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c15eca0 0x7f4a00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.869+0000 7f4a21000640 1 -- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a000039e0 msgr2=0x7f4a00003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.869+0000 7f4a21000640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a000039e0 0x7f4a00003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.869+0000 7f4a21000640 1 -- 172.21.15.67:0/3108920150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a00003520 con 0x7f4a00004d40 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.870+0000 7f4a21000640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a00004d40 0x7f4a000051c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001e20 tx=0x7f4a00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2802eee9911b3856 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14001a70 con 0x7f4a00004d40 2024-04-17T07:33:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a14003b70 con 0x7f4a00004d40 2024-04-17T07:33:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a1b400640 1 -- 172.21.15.67:0/3108920150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a140050d0 con 0x7f4a00004d40 2024-04-17T07:33:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a23000640 1 -- 172.21.15.67:0/3108920150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a00004d40 msgr2=0x7f4a000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a23000640 1 --2- 172.21.15.67:0/3108920150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a00004d40 0x7f4a000051c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001e20 tx=0x7f4a00005830 comp rx=0 tx=0).stop 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.871+0000 7f4a21000640 1 -- 172.21.15.67:0/3108920150 reap_dead start 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.872+0000 7f4a23000640 1 -- 172.21.15.67:0/3108920150 shutdown_connections 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.872+0000 7f4a23000640 1 -- 172.21.15.67:0/3108920150 >> 172.21.15.67:0/3108920150 conn(0x7f4a1c067600 msgr2=0x7f4a1c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.872+0000 7f4a23000640 1 -- 172.21.15.67:0/3108920150 shutdown_connections 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.872+0000 7f4a23000640 1 -- 172.21.15.67:0/3108920150 wait complete. 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.873+0000 7f4a23000640 1 Processor -- start 2024-04-17T07:33:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.874+0000 7f4a23000640 1 -- start start 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.874+0000 7f4a23000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1c149ea0 0x7f4a1c1e7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a23000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c1f8f20 0x7f4a1c1f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a23000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a23000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c15f080 con 0x7f4a1c1f8830 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a23000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c0d34d0 con 0x7f4a1c149ea0 2024-04-17T07:33:37.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a23000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1c15f200 con 0x7f4a1c1f8f20 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51430/0 (socket says 172.21.15.67:51430) 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a21000640 1 -- 172.21.15.67:0/2856462006 learned_addr learned my addr 172.21.15.67:0/2856462006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a13400640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c1f8f20 0x7f4a1c1f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a1be00640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1c149ea0 0x7f4a1c1e7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a21000640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c1f8f20 msgr2=0x7f4a1c1f82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.875+0000 7f4a21000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c1f8f20 0x7f4a1c1f82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a21000640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1c149ea0 msgr2=0x7f4a1c1e7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a21000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1c149ea0 0x7f4a1c1e7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a21000640 1 -- 172.21.15.67:0/2856462006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c1e7610 con 0x7f4a1c1f8830 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a21000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f4a1402c040 tx=0x7f4a140014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14007db0 con 0x7f4a1c1f8830 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a14003970 con 0x7f4a1c1f8830 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14006d90 con 0x7f4a1c1f8830 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.876+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1c1e78f0 con 0x7f4a1c1f8830 2024-04-17T07:33:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.877+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1c160510 con 0x7f4a1c1f8830 2024-04-17T07:33:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.878+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a1c160710 con 0x7f4a1c1f8830 2024-04-17T07:33:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.880+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 69) ==== 91159+0+0 (secure 0 0 0) 0x7f4a14007030 con 0x7f4a1c1f8830 2024-04-17T07:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.880+0000 7f4a18c00640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 0x7f49f806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.881+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f4a140acf40 con 0x7f4a1c1f8830 2024-04-17T07:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.881+0000 7f4a1be00640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 msgr2=0x7f49f806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.881+0000 7f4a1be00640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 0x7f49f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:37.888+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a1c160710 con 0x7f4a1c1f8830 2024-04-17T07:33:38.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.082+0000 7f4a1be00640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 msgr2=0x7f49f806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:38.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.082+0000 7f4a1be00640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 0x7f49f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.095+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f4a1c068ca0 con 0x7f4a1c1f8830 2024-04-17T07:33:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.103+0000 7f4a18c00640 1 -- 172.21.15.67:0/2856462006 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) ==== 138+0+0 (secure 0 0 0) 0x7f4a14033020 con 0x7f4a1c1f8830 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 msgr2=0x7f49f806ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 0x7f49f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 msgr2=0x7f4a1c0c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f4a1402c040 tx=0x7f4a140014a0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 shutdown_connections 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c1f8830 0x7f4a1c0c9660 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f49f806ca90 0x7f49f806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a1c1f8f20 0x7f4a1c1f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 --2- 172.21.15.67:0/2856462006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a1c149ea0 0x7f4a1c1e7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.105+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 >> 172.21.15.67:0/2856462006 conn(0x7f4a1c067600 msgr2=0x7f4a1c0cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.106+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 shutdown_connections 2024-04-17T07:33:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.106+0000 7f4a23000640 1 -- 172.21.15.67:0/2856462006 wait complete. 2024-04-17T07:33:38.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:33:38.179 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.179+0000 7f993fbdb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:38.222 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.222+0000 7feaa8a32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:38.233 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.233+0000 7f993fbdb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:38.281 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.280+0000 7feaa8a32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.339+0000 7fd3d3000640 1 Processor -- start 2024-04-17T07:33:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.339+0000 7fd3d3000640 1 -- start start 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc15eca0 0x7fd3cc0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3cc0cfa10 0x7fd3cc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3cc0d1090 0x7fd3cc0d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc0d03b0 con 0x7fd3cc0d1090 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc163860 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc0d1d40 con 0x7fd3cc15eca0 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3cc0d1090 0x7fd3cc0d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3cc0d1090 0x7fd3cc0d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51446/0 (socket says 172.21.15.67:51446) 2024-04-17T07:33:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.341+0000 7fd3d1000640 1 -- 172.21.15.67:0/1846077670 learned_addr learned my addr 172.21.15.67:0/1846077670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3cbe00640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc15eca0 0x7fd3cc0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3cb400640 1 --2- 172.21.15.67:0/1846077670 >> v2:172.21.15.187:3300/0 conn(0x7fd3cc0cfa10 0x7fd3cc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3d1000640 1 -- 172.21.15.67:0/1846077670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc15eca0 msgr2=0x7fd3cc0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3d1000640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc15eca0 0x7fd3cc0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3d1000640 1 -- 172.21.15.67:0/1846077670 >> v2:172.21.15.187:3300/0 conn(0x7fd3cc0cfa10 msgr2=0x7fd3cc0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3d1000640 1 --2- 172.21.15.67:0/1846077670 >> v2:172.21.15.187:3300/0 conn(0x7fd3cc0cfa10 0x7fd3cc0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.342+0000 7fd3d1000640 1 -- 172.21.15.67:0/1846077670 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc0d2c30 con 0x7fd3cc0d1090 2024-04-17T07:33:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.343+0000 7fd3d1000640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0d1090 0x7fd3cc0d1520 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0b7530 tx=0x7fd3c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1e592d066b8502 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.343+0000 7fd3caa00640 1 -- 172.21.15.67:0/1846077670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402ca80 con 0x7fd3cc0d1090 2024-04-17T07:33:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.343+0000 7fd3caa00640 1 -- 172.21.15.67:0/1846077670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c402cc20 con 0x7fd3cc0d1090 2024-04-17T07:33:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.343+0000 7fd3caa00640 1 -- 172.21.15.67:0/1846077670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c40343f0 con 0x7fd3cc0d1090 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 -- 172.21.15.67:0/1846077670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0d1090 msgr2=0x7fd3cc0d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0d1090 0x7fd3cc0d1520 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0b7530 tx=0x7fd3c402bee0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 -- 172.21.15.67:0/1846077670 shutdown_connections 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0d1090 0x7fd3cc0d1520 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 --2- 172.21.15.67:0/1846077670 >> v2:172.21.15.187:3300/0 conn(0x7fd3cc0cfa10 0x7fd3cc0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 --2- 172.21.15.67:0/1846077670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc15eca0 0x7fd3cc0d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 -- 172.21.15.67:0/1846077670 >> 172.21.15.67:0/1846077670 conn(0x7fd3cc067600 msgr2=0x7fd3cc0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 -- 172.21.15.67:0/1846077670 shutdown_connections 2024-04-17T07:33:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.344+0000 7fd3d3000640 1 -- 172.21.15.67:0/1846077670 wait complete. 2024-04-17T07:33:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.346+0000 7fd3d3000640 1 Processor -- start 2024-04-17T07:33:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.346+0000 7fd3d3000640 1 -- start start 2024-04-17T07:33:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc0d1090 0x7fd3cc0c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc15eca0 0x7fd3cc0c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0d34d0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0d31f0 con 0x7fd3cc15eca0 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3d3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc0d1a60 con 0x7fd3cc0d1090 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51458/0 (socket says 172.21.15.67:51458) 2024-04-17T07:33:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.347+0000 7fd3cbe00640 1 -- 172.21.15.67:0/2847021480 learned_addr learned my addr 172.21.15.67:0/2847021480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cb400640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc0d1090 0x7fd3cc0c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3d1000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc15eca0 0x7fd3cc0c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc0d1090 msgr2=0x7fd3cc0c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc0d1090 0x7fd3cc0c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc15eca0 msgr2=0x7fd3cc0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc15eca0 0x7fd3cc0c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 -- 172.21.15.67:0/2847021480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc150ca0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3cbe00640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00d800 tx=0x7fd3bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.348+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00b660 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.349+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc019070 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.349+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00ba20 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.349+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc0c8f40 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.349+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc15f240 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.350+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc068ca0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.351+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 69) ==== 91159+0+0 (secure 0 0 0) 0x7fd3bc00bbc0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.352+0000 7fd3c3e00640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 0x7fd3ac06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.352+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7fd3bc08c0c0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.352+0000 7fd3cb400640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 msgr2=0x7fd3ac06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.352+0000 7fd3cb400640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 0x7fd3ac06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:38.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.357+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3cc150ca0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.553+0000 7fd3cb400640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 msgr2=0x7fd3ac06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.553+0000 7fd3cb400640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 0x7fd3ac06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:38.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.561+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd3cc0d2970 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.569+0000 7fd3c3e00640 1 -- 172.21.15.67:0/2847021480 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) ==== 133+0+0 (secure 0 0 0) 0x7fd3bc05a3b0 con 0x7fd3cc0cfa10 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 msgr2=0x7fd3ac06ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 0x7fd3ac06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 msgr2=0x7fd3cc14ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00d800 tx=0x7fd3bc00dcc0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 shutdown_connections 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3cc15eca0 0x7fd3cc0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7fd3ac06ca90 0x7fd3ac06ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3cc0d1090 0x7fd3cc0c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 --2- 172.21.15.67:0/2847021480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3cc0cfa10 0x7fd3cc14ef80 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.572+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 >> 172.21.15.67:0/2847021480 conn(0x7fd3cc067600 msgr2=0x7fd3cc0cc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.573+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 shutdown_connections 2024-04-17T07:33:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.573+0000 7fd3d3000640 1 -- 172.21.15.67:0/2847021480 wait complete. 2024-04-17T07:33:38.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:38.592 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-04-17T07:33:38.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-04-17T07:33:38.678 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.678+0000 7f993fbdb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:38.730 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.730+0000 7f993fbdb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:38.759 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.759+0000 7feaa8a32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:38.780 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.780+0000 7f993fbdb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:38.816 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.815+0000 7feaa8a32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:38.841 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.841+0000 7f993fbdb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.854+0000 7f66fbe00640 1 Processor -- start 2024-04-17T07:33:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.854+0000 7f66fbe00640 1 -- start start 2024-04-17T07:33:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.855+0000 7f66fbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66fc15ecb0 0x7f66fc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.855+0000 7f66fbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66fc0d00b0 con 0x7f66fc0d3210 2024-04-17T07:33:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.855+0000 7f66fbe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66fc0d0390 con 0x7f66fc0cfab0 2024-04-17T07:33:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.855+0000 7f66fbe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f66fc0d0c30 con 0x7f66fc15ecb0 2024-04-17T07:33:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.856+0000 7f66f2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66fc0d3210 0x7f66fc0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56970/0 (socket says 172.21.15.67:56970) 2024-04-17T07:33:38.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.856+0000 7f66f2000640 1 -- 172.21.15.67:0/3262011298 learned_addr learned my addr 172.21.15.67:0/3262011298 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.856+0000 7f66faa00640 1 --2- 172.21.15.67:0/3262011298 >> v2:172.21.15.187:3300/0 conn(0x7f66fc15ecb0 0x7f66fc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.856+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826946042 0 0) 0x7f66fc0d00b0 con 0x7f66fc0d3210 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.856+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66e0003520 con 0x7f66fc0d3210 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530919054 0 0) 0x7f66fc0d0390 con 0x7f66fc0cfab0 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66fc0d00b0 con 0x7f66fc0cfab0 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 206723220 0 0) 0x7f66e0003520 con 0x7f66fc0d3210 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 >> v2:172.21.15.187:3300/0 conn(0x7f66fc15ecb0 msgr2=0x7f66fc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 --2- 172.21.15.67:0/3262011298 >> v2:172.21.15.187:3300/0 conn(0x7f66fc15ecb0 0x7f66fc0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 >> v1:172.21.15.187:6789/0 conn(0x7f66fc0cfab0 legacy=0x7f66fc0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc0d0670 con 0x7f66fc0d3210 2024-04-17T07:33:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f66e40015f0 con 0x7f66fc0d3210 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 >> v1:172.21.15.67:6789/0 conn(0x7f66fc0d3210 legacy=0x7f66fc0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0003b90 0x7f66e0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66e0004800 0x7f66e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 0x7f66e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc0d0c30 con 0x7f66e0005210 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66e000c140 con 0x7f66e0004800 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66e000c340 con 0x7f66e0003b90 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.857+0000 7f66fb400640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 0x7f66e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66fb400640 1 -- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0003b90 msgr2=0x7f66e0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66faa00640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0003b90 0x7f66e0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66f2000640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66e0004800 0x7f66e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66fb400640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0003b90 0x7f66e0003f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66fb400640 1 -- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66e0004800 msgr2=0x7f66e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66fb400640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66e0004800 0x7f66e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66fb400640 1 -- 172.21.15.67:0/3262011298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e0003520 con 0x7f66e0005210 2024-04-17T07:33:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.858+0000 7f66f2000640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66e0004800 0x7f66e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.859+0000 7f66fb400640 1 -- 172.21.15.67:0/3262011298 reap_dead start 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.859+0000 7f66fb400640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 0x7f66e000b9c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f66f40018d0 tx=0x7f66e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a49dcf2e56b9a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.859+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40014f0 con 0x7f66e0005210 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.859+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f40052d0 con 0x7f66e0005210 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.859+0000 7f66fa000640 1 -- 172.21.15.67:0/3262011298 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4003830 con 0x7f66e0005210 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 -- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 msgr2=0x7f66e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 0x7f66e000b9c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f66f40018d0 tx=0x7f66e0005830 comp rx=0 tx=0).stop 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 -- 172.21.15.67:0/3262011298 shutdown_connections 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 --2- 172.21.15.67:0/3262011298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e0005210 0x7f66e000b9c0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 -- 172.21.15.67:0/3262011298 >> 172.21.15.67:0/3262011298 conn(0x7f66fc08bff0 msgr2=0x7f66fc0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 -- 172.21.15.67:0/3262011298 shutdown_connections 2024-04-17T07:33:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.860+0000 7f66fbe00640 1 -- 172.21.15.67:0/3262011298 wait complete. 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.862+0000 7f66fbe00640 1 Processor -- start 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.862+0000 7f66fbe00640 1 -- start start 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0005210 0x7f66fc0a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66fc15be10 0x7f66fc0ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 0x7f66fc09f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc15f210 con 0x7f66fc09ec90 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc15f090 con 0x7f66fc15be10 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66fc0d2cd0 con 0x7f66e0005210 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 0x7f66fc09f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 0x7f66fc09f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51484/0 (socket says 172.21.15.67:51484) 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 -- 172.21.15.67:0/1587942598 learned_addr learned my addr 172.21.15.67:0/1587942598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0005210 msgr2=0x7f66fc0a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66e0005210 0x7f66fc0a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66fc15be10 msgr2=0x7f66fc0ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66f2000640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66fc15be10 0x7f66fc0ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66fc15be10 0x7f66fc0ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.863+0000 7f66fb400640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc0a11b0 con 0x7f66fc09ec90 2024-04-17T07:33:38.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66fb400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 0x7f66fc09f110 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f66f402be70 tx=0x7f66f40012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40014f0 con 0x7f66fc09ec90 2024-04-17T07:33:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f4005480 con 0x7f66fc09ec90 2024-04-17T07:33:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4007680 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66fc0a1490 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.864+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66fc0a1a50 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.866+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66fc068a10 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.866+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 69) ==== 91159+0+0 (secure 0 0 0) 0x7f66f4044020 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.867+0000 7f66f3400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.867+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5737+0+0 (secure 0 0 0) 0x7f66f40ad160 con 0x7f66fc09ec90 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.867+0000 7f66faa00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 msgr2=0x7f66d806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:38.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.867+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:38.871 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.871+0000 7feaa8a32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:38.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:38.872+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f66f407b3c0 con 0x7f66fc09ec90 2024-04-17T07:33:38.891 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:38.891+0000 7f993fbdb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:38.938 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.937+0000 7feaa8a32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:38.990 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:38.990+0000 7feaa8a32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:39.019 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.019+0000 7f993fbdb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:39.068+0000 7f66faa00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 msgr2=0x7f66d806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:39.068+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:39.081 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.081+0000 7f993fbdb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:39.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:39.095+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f66fc0b5a70 con 0x7f66d806ca60 2024-04-17T07:33:39.126 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.126+0000 7feaa8a32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:39.132 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.132+0000 7f993fbdb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:39.189 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.188+0000 7f993fbdb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:39.192 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.192+0000 7feaa8a32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:39.239 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.239+0000 7f993fbdb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:39.244 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.244+0000 7feaa8a32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:39.304 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.304+0000 7feaa8a32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:39.357 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.357+0000 7feaa8a32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:39.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:39.469+0000 7f66faa00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 msgr2=0x7f66d806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:39.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:39.469+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:33:39.618 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.618+0000 7f993fbdb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:39.760 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.760+0000 7feaa8a32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:39.777 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.777+0000 7f993fbdb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:39.885 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.885+0000 7f993fbdb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:39.928 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:39.928+0000 7feaa8a32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:39.935 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:39.935+0000 7f993fbdb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:40.042 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.042+0000 7feaa8a32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:40.060 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:40.060+0000 7f993fbdb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:40.096 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.096+0000 7feaa8a32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:40.169 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:40.169+0000 7f993fbdb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:40.230 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.230+0000 7feaa8a32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:40.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:40.270+0000 7f66faa00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 msgr2=0x7f66d806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1274719646 2024-04-17T07:33:40.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:40.270+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:33:40.343 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.343+0000 7feaa8a32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:40.344 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:40.343+0000 7f993fbdb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:40.392 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:40.392+0000 7f993fbdb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:40.527 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.526+0000 7feaa8a32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:40.528 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:40.528+0000 7f993fbdb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:40.578 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.578+0000 7feaa8a32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:40.722 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:40.721+0000 7feaa8a32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:41.345 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:41.345+0000 7f993fbdb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:41.570 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:41.569+0000 7feaa8a32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:41.645 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:41.645+0000 7f993fbdb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:41.703 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:41.703+0000 7f993fbdb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:41.834 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:41.834+0000 7f993fbdb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:41.857+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 70) ==== 90768+0+0 (secure 0 0 0) 0x7f66f403b040 con 0x7f66fc09ec90 2024-04-17T07:33:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:41.858+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 msgr2=0x7f66d806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:41.858+0000 7f66f3400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/1274719646,v1:172.21.15.187:6833/1274719646] conn(0x7f66d806ca60 0x7f66d806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:41.896 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:41.895+0000 7feaa8a32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:41.960 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:41.960+0000 7feaa8a32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:42.106 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:42.105+0000 7feaa8a32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b3200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b3200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b3200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b3200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b3200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.129 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.129 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.129 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.129 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:42.127+0000 7f98b6e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:42.867+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 71) ==== 90895+0+0 (secure 0 0 0) 0x7f66f4077330 con 0x7f66fc09ec90 2024-04-17T07:33:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:42.868+0000 7f66f3400640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f66d8074970 0x7f66d8076d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:42.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:42.868+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f66fc0b5a70 con 0x7f66d8074970 2024-04-17T07:33:42.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:42.871+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f66d8074970 0x7f66d8076d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:42.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:42.876+0000 7f66faa00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f66d8074970 0x7f66d8076d60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0094c0 tx=0x7f66ec009750 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.050+0000 7f66f3400640 1 -- 172.21.15.67:0/1587942598 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f66fc0b5a70 con 0x7f66d8074970 2024-04-17T07:33:43.051 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/minutes 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f66d8074970 msgr2=0x7f66d8076d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f66d8074970 0x7f66d8076d60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0094c0 tx=0x7f66ec009750 comp rx=0 tx=0).stop 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 msgr2=0x7f66fc09f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 --2- 172.21.15.67:0/1587942598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66fc09ec90 0x7f66fc09f110 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f66f402be70 tx=0x7f66f40012d0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fb400640 1 -- 172.21.15.67:0/1587942598 reap_dead start 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 shutdown_connections 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 >> 172.21.15.67:0/1587942598 conn(0x7f66fc08bff0 msgr2=0x7f66fc0d3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 shutdown_connections 2024-04-17T07:33:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.054+0000 7f66fbe00640 1 -- 172.21.15.67:0/1587942598 wait complete. 2024-04-17T07:33:43.077 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-04-17T07:33:43.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:43.077 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-04-17T07:33:43.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-04-17T07:33:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.361+0000 7f57b3000640 1 Processor -- start 2024-04-17T07:33:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.361+0000 7f57b3000640 1 -- start start 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.363+0000 7f57b3000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57ac15b6b0 0x7f57ac06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.363+0000 7f57b3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac0cfc10 0x7f57ac06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.363+0000 7f57b3000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57ac06df40 con 0x7f57ac06f800 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.363+0000 7f57b3000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57ac06ed00 con 0x7f57ac15b6b0 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.363+0000 7f57b3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57ac06e870 con 0x7f57ac0cfc10 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57ac06f800 0x7f57ac15f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46062/0 (socket says 172.21.15.67:46062) 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1200640 1 -- 172.21.15.67:0/2234542741 learned_addr learned my addr 172.21.15.67:0/2234542741 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1c00640 1 --2- 172.21.15.67:0/2234542741 >> v2:172.21.15.67:3300/0 conn(0x7f57ac15b6b0 0x7f57ac06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b2600640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac0cfc10 0x7f57ac06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57abe00640 1 -- 172.21.15.67:0/2234542741 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401445392 0 0) 0x7f57ac06df40 con 0x7f57ac06f800 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57abe00640 1 -- 172.21.15.67:0/2234542741 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5794003520 con 0x7f57ac06f800 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1c00640 1 -- 172.21.15.67:0/2234542741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac0cfc10 msgr2=0x7f57ac06be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1c00640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac0cfc10 0x7f57ac06be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1c00640 1 -- 172.21.15.67:0/2234542741 >> v1:172.21.15.67:6789/0 conn(0x7f57ac06f800 legacy=0x7f57ac15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.364+0000 7f57b1c00640 1 -- 172.21.15.67:0/2234542741 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac06e220 con 0x7f57ac15b6b0 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57b1c00640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac15b6b0 0x7f57ac06b8c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f57a00097a0 tx=0x7f57a002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c016385ce81125 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57abe00640 1 -- 172.21.15.67:0/2234542741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0038070 con 0x7f57ac15b6b0 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57abe00640 1 -- 172.21.15.67:0/2234542741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a0031700 con 0x7f57ac15b6b0 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57abe00640 1 -- 172.21.15.67:0/2234542741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0031ab0 con 0x7f57ac15b6b0 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57b3000640 1 -- 172.21.15.67:0/2234542741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac15b6b0 msgr2=0x7f57ac06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.366+0000 7f57b3000640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac15b6b0 0x7f57ac06b8c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f57a00097a0 tx=0x7f57a002b7a0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 -- 172.21.15.67:0/2234542741 shutdown_connections 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac0cfc10 0x7f57ac06be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 --2- 172.21.15.67:0/2234542741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac15b6b0 0x7f57ac06b8c0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 -- 172.21.15.67:0/2234542741 >> 172.21.15.67:0/2234542741 conn(0x7f57ac0b2190 msgr2=0x7f57ac0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 -- 172.21.15.67:0/2234542741 shutdown_connections 2024-04-17T07:33:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.367+0000 7f57b3000640 1 -- 172.21.15.67:0/2234542741 wait complete. 2024-04-17T07:33:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.368+0000 7f57b3000640 1 Processor -- start 2024-04-17T07:33:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 -- start start 2024-04-17T07:33:43.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57ac0cfc10 0x7f57ac151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac15b6b0 0x7f57ac1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac0afee0 con 0x7f57ac06f800 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac0afd60 con 0x7f57ac0cfc10 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.369+0000 7f57b3000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57ac0b44a0 con 0x7f57ac15b6b0 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56602/0 (socket says 172.21.15.67:56602) 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 -- 172.21.15.67:0/3183946089 learned_addr learned my addr 172.21.15.67:0/3183946089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b2600640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac15b6b0 0x7f57ac1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1200640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57ac0cfc10 0x7f57ac151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 -- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac15b6b0 msgr2=0x7f57ac1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac15b6b0 0x7f57ac1524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 -- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57ac0cfc10 msgr2=0x7f57ac151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57ac0cfc10 0x7f57ac151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.370+0000 7f57b1c00640 1 -- 172.21.15.67:0/3183946089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac0ae380 con 0x7f57ac06f800 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57b1c00640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f57a0036040 tx=0x7f57a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0042020 con 0x7f57ac06f800 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a003d950 con 0x7f57ac06f800 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0038040 con 0x7f57ac06f800 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac0ae660 con 0x7f57ac06f800 2024-04-17T07:33:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.371+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac0aec20 con 0x7f57ac06f800 2024-04-17T07:33:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.372+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ac068a10 con 0x7f57ac06f800 2024-04-17T07:33:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.373+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 71) ==== 90895+0+0 (secure 0 0 0) 0x7f57a0045020 con 0x7f57ac06f800 2024-04-17T07:33:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.374+0000 7f57a9600640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 0x7f578c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.374+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f57a00ad460 con 0x7f57ac06f800 2024-04-17T07:33:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.375+0000 7f57b1200640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 0x7f578c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.376+0000 7f57b1200640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 0x7f578c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0ceda0 tx=0x7f5798009040 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.379+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f57a004c030 con 0x7f57ac06f800 2024-04-17T07:33:43.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.614+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f57ac06ee80 con 0x7f578c06c790 2024-04-17T07:33:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.796+0000 7f57a9600640 1 -- 172.21.15.67:0/3183946089 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7f57ac06ee80 con 0x7f578c06c790 2024-04-17T07:33:43.797 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/hourly 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 msgr2=0x7f578c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 0x7f578c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0ceda0 tx=0x7f5798009040 comp rx=0 tx=0).stop 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 msgr2=0x7f57ac150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f57a0036040 tx=0x7f57a00046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 shutdown_connections 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f578c06c790 0x7f578c06ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57ac15b6b0 0x7f57ac1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57ac0cfc10 0x7f57ac151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 --2- 172.21.15.67:0/3183946089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57ac06f800 0x7f57ac150fa0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.799+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 >> 172.21.15.67:0/3183946089 conn(0x7f57ac0b2190 msgr2=0x7f57ac0cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.800+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 shutdown_connections 2024-04-17T07:33:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:43.800+0000 7f57b3000640 1 -- 172.21.15.67:0/3183946089 wait complete. 2024-04-17T07:33:43.823 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-04-17T07:33:43.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:43.823 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-04-17T07:33:43.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-04-17T07:33:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.111+0000 7fd600a00640 1 Processor -- start 2024-04-17T07:33:44.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.111+0000 7fd600a00640 1 -- start start 2024-04-17T07:33:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd600a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5fc145f80 0x7fd5fc0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd600a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd5fc0cd750 0x7fd5fc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd600a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc06f1d0 con 0x7fd5fc06c0a0 2024-04-17T07:33:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd600a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd5fc0727a0 con 0x7fd5fc145f80 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd600a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd5fc146360 con 0x7fd5fc0cd750 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd5fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5fc145f80 0x7fd5fc0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd5fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5fc145f80 0x7fd5fc0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56616/0 (socket says 172.21.15.67:56616) 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.113+0000 7fd5fb400640 1 -- 172.21.15.67:0/2110744048 learned_addr learned my addr 172.21.15.67:0/2110744048 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.114+0000 7fd5faa00640 1 --2- 172.21.15.67:0/2110744048 >> v2:172.21.15.187:3300/0 conn(0x7fd5fc0cd750 0x7fd5fc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.114+0000 7fd5fb400640 1 -- 172.21.15.67:0/2110744048 >> v2:172.21.15.187:3300/0 conn(0x7fd5fc0cd750 msgr2=0x7fd5fc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.114+0000 7fd5fb400640 1 --2- 172.21.15.67:0/2110744048 >> v2:172.21.15.187:3300/0 conn(0x7fd5fc0cd750 0x7fd5fc06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.114+0000 7fd5fb400640 1 -- 172.21.15.67:0/2110744048 >> v1:172.21.15.187:6789/0 conn(0x7fd5fc06c0a0 legacy=0x7fd5fc160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.114+0000 7fd5fb400640 1 -- 172.21.15.67:0/2110744048 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc071e70 con 0x7fd5fc145f80 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.115+0000 7fd5fb400640 1 --2- 172.21.15.67:0/2110744048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc0cd210 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc06e140 tx=0x7fd5ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1733d57e3eae76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.115+0000 7fd5fa000640 1 -- 172.21.15.67:0/2110744048 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec02e070 con 0x7fd5fc145f80 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.115+0000 7fd5fa000640 1 -- 172.21.15.67:0/2110744048 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ec037070 con 0x7fd5fc145f80 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.115+0000 7fd5fa000640 1 -- 172.21.15.67:0/2110744048 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec031a00 con 0x7fd5fc145f80 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 -- 172.21.15.67:0/2110744048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 msgr2=0x7fd5fc0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 --2- 172.21.15.67:0/2110744048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc0cd210 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc06e140 tx=0x7fd5ec02b880 comp rx=0 tx=0).stop 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 -- 172.21.15.67:0/2110744048 shutdown_connections 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 --2- 172.21.15.67:0/2110744048 >> v2:172.21.15.187:3300/0 conn(0x7fd5fc0cd750 0x7fd5fc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 --2- 172.21.15.67:0/2110744048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc0cd210 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 -- 172.21.15.67:0/2110744048 >> 172.21.15.67:0/2110744048 conn(0x7fd5fc0b2190 msgr2=0x7fd5fc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 -- 172.21.15.67:0/2110744048 shutdown_connections 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.116+0000 7fd600a00640 1 -- 172.21.15.67:0/2110744048 wait complete. 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.118+0000 7fd600a00640 1 Processor -- start 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.118+0000 7fd600a00640 1 -- start start 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5fc06c0a0 0x7fd5fc0b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5fc0cd750 0x7fd5fc142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc072c30 con 0x7fd5fc145f80 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc072430 con 0x7fd5fc06c0a0 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd600a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc072db0 con 0x7fd5fc0cd750 2024-04-17T07:33:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd5fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56632/0 (socket says 172.21.15.67:56632) 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.119+0000 7fd5fbe00640 1 -- 172.21.15.67:0/415479493 learned_addr learned my addr 172.21.15.67:0/415479493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fb400640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5fc06c0a0 0x7fd5fc0b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5faa00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5fc0cd750 0x7fd5fc142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 -- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5fc0cd750 msgr2=0x7fd5fc142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5fc0cd750 0x7fd5fc142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 -- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5fc06c0a0 msgr2=0x7fd5fc0b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5fc06c0a0 0x7fd5fc0b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 -- 172.21.15.67:0/415479493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc1f5eb0 con 0x7fd5fc145f80 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5fbe00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400c7b0 tx=0x7fd5f400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4017070 con 0x7fd5fc145f80 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f4003da0 con 0x7fd5fc145f80 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.120+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40052b0 con 0x7fd5fc145f80 2024-04-17T07:33:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.121+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc0b0450 con 0x7fd5fc145f80 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.121+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc0b0a10 con 0x7fd5fc145f80 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.122+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc068a10 con 0x7fd5fc145f80 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.123+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 72) ==== 91160+0+0 (secure 0 0 0) 0x7fd5f4005450 con 0x7fd5fc145f80 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.124+0000 7fd5f3400640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 0x7fd5dc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.124+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7fd5f408b410 con 0x7fd5fc145f80 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.124+0000 7fd5fb400640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 0x7fd5dc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.126+0000 7fd5fb400640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 0x7fd5dc06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec036000 tx=0x7fd5ec03d040 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.129+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd5f40595b0 con 0x7fd5fc145f80 2024-04-17T07:33:44.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.358+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) -- 0x7fd5fc0b3ed0 con 0x7fd5dc06ca60 2024-04-17T07:33:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.538+0000 7fd5f3400640 1 -- 172.21.15.67:0/415479493 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd5fc0b3ed0 con 0x7fd5dc06ca60 2024-04-17T07:33:44.539 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/daily 2024-04-17T07:33:44.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 msgr2=0x7fd5dc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 0x7fd5dc06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec036000 tx=0x7fd5ec03d040 comp rx=0 tx=0).stop 2024-04-17T07:33:44.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 msgr2=0x7fd5fc1433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400c7b0 tx=0x7fd5f400cc70 comp rx=0 tx=0).stop 2024-04-17T07:33:44.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 shutdown_connections 2024-04-17T07:33:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fd5dc06ca60 0x7fd5dc06ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5fc145f80 0x7fd5fc1433e0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5fc0cd750 0x7fd5fc142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 --2- 172.21.15.67:0/415479493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5fc06c0a0 0x7fd5fc0b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.541+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 >> 172.21.15.67:0/415479493 conn(0x7fd5fc0b2190 msgr2=0x7fd5fc0b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:44.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.542+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 shutdown_connections 2024-04-17T07:33:44.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.542+0000 7fd600a00640 1 -- 172.21.15.67:0/415479493 wait complete. 2024-04-17T07:33:44.566 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-04-17T07:33:44.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:44.567 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-04-17T07:33:44.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-04-17T07:33:44.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.857+0000 7f3349a00640 1 Processor -- start 2024-04-17T07:33:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.857+0000 7f3349a00640 1 -- start start 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.858+0000 7f3349a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f33440cd920 0x7f334406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.858+0000 7f3349a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3344146360 con 0x7f3344145f80 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.858+0000 7f3349a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3344071e70 con 0x7f334406c0a0 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.858+0000 7f3349a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f33440727a0 con 0x7f33440cd920 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.859+0000 7f3343e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3344145f80 0x7f33440cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46078/0 (socket says 172.21.15.67:46078) 2024-04-17T07:33:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.859+0000 7f3343e00640 1 -- 172.21.15.67:0/2558984788 learned_addr learned my addr 172.21.15.67:0/2558984788 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.859+0000 7f3343400640 1 --2- 172.21.15.67:0/2558984788 >> v2:172.21.15.187:3300/0 conn(0x7f33440cd920 0x7f334406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.859+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686341455 0 0) 0x7f3344146360 con 0x7f3344145f80 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.859+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3328003520 con 0x7f3344145f80 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2510794029 0 0) 0x7f3328003520 con 0x7f3344145f80 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 >> v2:172.21.15.187:3300/0 conn(0x7f33440cd920 msgr2=0x7f334406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 --2- 172.21.15.67:0/2558984788 >> v2:172.21.15.187:3300/0 conn(0x7f33440cd920 0x7f334406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 >> v1:172.21.15.187:6789/0 conn(0x7f334406c0a0 legacy=0x7f334406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3344072150 con 0x7f3344145f80 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3334001670 con 0x7f3344145f80 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 >> v1:172.21.15.67:6789/0 conn(0x7f3344145f80 legacy=0x7f33440cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328003a00 0x7f3328003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 0x7f3328004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3328005210 0x7f332800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f33440727a0 con 0x7f3328004800 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f332800c160 con 0x7f3328005210 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f332800c360 con 0x7f3328003a00 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.860+0000 7f3343400640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 0x7f3328004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343e00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328003a00 0x7f3328003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343400640 1 -- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328003a00 msgr2=0x7f3328003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3349000640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3328005210 0x7f332800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343400640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328003a00 0x7f3328003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343400640 1 -- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3328005210 msgr2=0x7f332800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343400640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3328005210 0x7f332800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3343400640 1 -- 172.21.15.67:0/2558984788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3328003520 con 0x7f3328004800 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3349000640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3328005210 0x7f332800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:44.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.861+0000 7f3349000640 1 -- 172.21.15.67:0/2558984788 reap_dead start 2024-04-17T07:33:44.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.862+0000 7f3343400640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 0x7f3328004c80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f33440b3540 tx=0x7f3328005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1da6ad0cdcee91a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.864+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330039070 con 0x7f3328004800 2024-04-17T07:33:44.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.864+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f333002c660 con 0x7f3328004800 2024-04-17T07:33:44.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.864+0000 7f3342a00640 1 -- 172.21.15.67:0/2558984788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330009690 con 0x7f3328004800 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 -- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 msgr2=0x7f3328004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 0x7f3328004c80 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f33440b3540 tx=0x7f3328005830 comp rx=0 tx=0).stop 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 -- 172.21.15.67:0/2558984788 shutdown_connections 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 --2- 172.21.15.67:0/2558984788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004800 0x7f3328004c80 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 -- 172.21.15.67:0/2558984788 >> 172.21.15.67:0/2558984788 conn(0x7f33440b2190 msgr2=0x7f33440b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 -- 172.21.15.67:0/2558984788 shutdown_connections 2024-04-17T07:33:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.865+0000 7f3349a00640 1 -- 172.21.15.67:0/2558984788 wait complete. 2024-04-17T07:33:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.867+0000 7f3349a00640 1 Processor -- start 2024-04-17T07:33:44.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.867+0000 7f3349a00640 1 -- start start 2024-04-17T07:33:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328004800 0x7f33440ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33441358e0 0x7f3344140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3344072c30 con 0x7f33440ae820 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f334406c530 con 0x7f33441358e0 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3344072db0 con 0x7f3328004800 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56660/0 (socket says 172.21.15.67:56660) 2024-04-17T07:33:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.868+0000 7f3349000640 1 -- 172.21.15.67:0/2066121581 learned_addr learned my addr 172.21.15.67:0/2066121581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3343e00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328004800 0x7f33440ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3343400640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33441358e0 0x7f3344140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 -- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328004800 msgr2=0x7f33440ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328004800 0x7f33440ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 -- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33441358e0 msgr2=0x7f3344140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33441358e0 0x7f3344140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 -- 172.21.15.67:0/2066121581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3344138790 con 0x7f33440ae820 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.869+0000 7f3349000640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f33440ceed0 tx=0x7f333c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.870+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c018070 con 0x7f33440ae820 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.870+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f333c003da0 con 0x7f33440ae820 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.870+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333c0052b0 con 0x7f33440ae820 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.870+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33440b3710 con 0x7f33440ae820 2024-04-17T07:33:44.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.870+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33440b3cd0 con 0x7f33440ae820 2024-04-17T07:33:44.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.871+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33440b2770 con 0x7f33440ae820 2024-04-17T07:33:44.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.872+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 72) ==== 91160+0+0 (secure 0 0 0) 0x7f333c005450 con 0x7f33440ae820 2024-04-17T07:33:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.873+0000 7f333be00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 0x7f332406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.873+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f333c08b580 con 0x7f33440ae820 2024-04-17T07:33:44.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.874+0000 7f3343e00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 0x7f332406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:44.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.875+0000 7f3343e00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 0x7f332406ef50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3334001eb0 tx=0x7f3334009040 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:44.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:44.878+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f33440b2770 con 0x7f33440ae820 2024-04-17T07:33:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.115+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) -- 0x7f3344145ac0 con 0x7f332406ca90 2024-04-17T07:33:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.303+0000 7f333be00640 1 -- 172.21.15.67:0/2066121581 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7f3344145ac0 con 0x7f332406ca90 2024-04-17T07:33:45.303 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/weekly 2024-04-17T07:33:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.306+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 msgr2=0x7f332406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.306+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 0x7f332406ef50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3334001eb0 tx=0x7f3334009040 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.306+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 msgr2=0x7f3344140ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.306+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f33440ceed0 tx=0x7f333c00d9d0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 shutdown_connections 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f332406ca90 0x7f332406ef50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440ae820 0x7f3344140ea0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33441358e0 0x7f3344140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 --2- 172.21.15.67:0/2066121581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3328004800 0x7f33440ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 >> 172.21.15.67:0/2066121581 conn(0x7f33440b2190 msgr2=0x7f334415c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 shutdown_connections 2024-04-17T07:33:45.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.307+0000 7f3349a00640 1 -- 172.21.15.67:0/2066121581 wait complete. 2024-04-17T07:33:45.333 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-04-17T07:33:45.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:45.334 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-04-17T07:33:45.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-04-17T07:33:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.624+0000 7f6d04a00640 1 Processor -- start 2024-04-17T07:33:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.625+0000 7f6d04a00640 1 -- start start 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.626+0000 7f6d04a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d000cd920 0x7f6d0006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.626+0000 7f6d04a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 0x7f6d0006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.626+0000 7f6d04a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d00146360 con 0x7f6d00145f80 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.626+0000 7f6d04a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6d000727a0 con 0x7f6d000cd920 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.626+0000 7f6d04a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d00071e70 con 0x7f6d0006c0a0 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d000cd920 0x7f6d0006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d000cd920 0x7f6d0006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56668/0 (socket says 172.21.15.67:56668) 2024-04-17T07:33:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 -- 172.21.15.67:0/3935370385 learned_addr learned my addr 172.21.15.67:0/3935370385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cff200640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 0x7f6d0006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 -- 172.21.15.67:0/3935370385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 msgr2=0x7f6d0006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 0x7f6d0006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 -- 172.21.15.67:0/3935370385 >> v1:172.21.15.187:6789/0 conn(0x7f6d00145f80 legacy=0x7f6d000cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cf5e00640 1 -- 172.21.15.67:0/3935370385 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d00072150 con 0x7f6d000cd920 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.627+0000 7f6cff200640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 0x7f6d0006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.628+0000 7f6cf5e00640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000cd920 0x7f6d0006bb60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009870 tx=0x7f6ce802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e89652662a1f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.629+0000 7f6cfde00640 1 -- 172.21.15.67:0/3935370385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80317c0 con 0x7f6d000cd920 2024-04-17T07:33:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.629+0000 7f6cfde00640 1 -- 172.21.15.67:0/3935370385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce8037070 con 0x7f6d000cd920 2024-04-17T07:33:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.629+0000 7f6cfde00640 1 -- 172.21.15.67:0/3935370385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8031b80 con 0x7f6d000cd920 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.629+0000 7f6d04a00640 1 -- 172.21.15.67:0/3935370385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000cd920 msgr2=0x7f6d0006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.629+0000 7f6d04a00640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000cd920 0x7f6d0006bb60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009870 tx=0x7f6ce802b860 comp rx=0 tx=0).stop 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 -- 172.21.15.67:0/3935370385 shutdown_connections 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d0006c0a0 0x7f6d0006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 --2- 172.21.15.67:0/3935370385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000cd920 0x7f6d0006bb60 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 -- 172.21.15.67:0/3935370385 >> 172.21.15.67:0/3935370385 conn(0x7f6d000b2190 msgr2=0x7f6d000b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 -- 172.21.15.67:0/3935370385 shutdown_connections 2024-04-17T07:33:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.630+0000 7f6d04a00640 1 -- 172.21.15.67:0/3935370385 wait complete. 2024-04-17T07:33:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.632+0000 7f6d04a00640 1 Processor -- start 2024-04-17T07:33:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.632+0000 7f6d04a00640 1 -- start start 2024-04-17T07:33:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d000cd920 0x7f6d00140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 0x7f6d0015de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d00072c30 con 0x7f6d0006c0a0 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0006c530 con 0x7f6d00145f80 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.633+0000 7f6d04a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d00072db0 con 0x7f6d000cd920 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56680/0 (socket says 172.21.15.67:56680) 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 -- 172.21.15.67:0/817412007 learned_addr learned my addr 172.21.15.67:0/817412007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cff200640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 0x7f6d0015de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cf5e00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d000cd920 0x7f6d00140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 -- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d000cd920 msgr2=0x7f6d00140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d000cd920 0x7f6d00140750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 -- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 msgr2=0x7f6d0015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 0x7f6d0015de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cfe800640 1 -- 172.21.15.67:0/817412007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0013f3f0 con 0x7f6d0006c0a0 2024-04-17T07:33:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.634+0000 7f6cff200640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 0x7f6d0015de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6cfe800640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b3f0 tx=0x7f6cf000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0015020 con 0x7f6d0006c0a0 2024-04-17T07:33:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf0011830 con 0x7f6d0006c0a0 2024-04-17T07:33:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011c20 con 0x7f6d0006c0a0 2024-04-17T07:33:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0013f6b0 con 0x7f6d0006c0a0 2024-04-17T07:33:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.635+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0013fc70 con 0x7f6d0006c0a0 2024-04-17T07:33:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.636+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d000ce380 con 0x7f6d0006c0a0 2024-04-17T07:33:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.637+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 72) ==== 91160+0+0 (secure 0 0 0) 0x7f6cf0018020 con 0x7f6d0006c0a0 2024-04-17T07:33:45.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.638+0000 7f6cf7400640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 0x7f6cdc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:45.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.638+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f6cf008bc10 con 0x7f6d0006c0a0 2024-04-17T07:33:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.639+0000 7f6cf5e00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 0x7f6cdc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.640+0000 7f6cf5e00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 0x7f6cdc06ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8036000 tx=0x7f6ce8008030 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.645+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d000ce380 con 0x7f6d0006c0a0 2024-04-17T07:33:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.877+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f6d000cc680 con 0x7f6cdc06ca60 2024-04-17T07:33:45.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:45.940+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f6cf0055bf0 con 0x7f6d0006c0a0 2024-04-17T07:33:46.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.072+0000 7f6cf7400640 1 -- 172.21.15.67:0/817412007 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f6d000cc680 con 0x7f6cdc06ca60 2024-04-17T07:33:46.072 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/monthly 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.074+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 msgr2=0x7f6cdc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.074+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 0x7f6cdc06ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8036000 tx=0x7f6ce8008030 comp rx=0 tx=0).stop 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 msgr2=0x7f6d0015be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b3f0 tx=0x7f6cf000b8b0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 shutdown_connections 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06ca60 0x7f6cdc06ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d00145f80 0x7f6d0015de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d000cd920 0x7f6d00140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 --2- 172.21.15.67:0/817412007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0006c0a0 0x7f6d0015be20 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 >> 172.21.15.67:0/817412007 conn(0x7f6d000b2190 msgr2=0x7f6d0006eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 shutdown_connections 2024-04-17T07:33:46.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.075+0000 7f6d04a00640 1 -- 172.21.15.67:0/817412007 wait complete. 2024-04-17T07:33:46.102 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-04-17T07:33:46.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:46.102 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-04-17T07:33:46.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1y 2024-04-17T07:33:46.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.386+0000 7f6cf4e00640 1 Processor -- start 2024-04-17T07:33:46.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.387+0000 7f6cf4e00640 1 -- start start 2024-04-17T07:33:46.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.388+0000 7f6cf4e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6cf015ecb0 0x7f6cf00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:46.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.388+0000 7f6cf4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf00d00b0 con 0x7f6cf00d3210 2024-04-17T07:33:46.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.388+0000 7f6cf4e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf00d0390 con 0x7f6cf00cfab0 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.388+0000 7f6cf4e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf00d0c30 con 0x7f6cf015ecb0 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.389+0000 7f6ceea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6cf00d3210 0x7f6cf00cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46090/0 (socket says 172.21.15.67:46090) 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.389+0000 7f6ceea00640 1 -- 172.21.15.67:0/2233680027 learned_addr learned my addr 172.21.15.67:0/2233680027 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.389+0000 7f6cef400640 1 --2- 172.21.15.67:0/2233680027 >> v2:172.21.15.187:3300/0 conn(0x7f6cf015ecb0 0x7f6cf00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.389+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3812911993 0 0) 0x7f6cf00d00b0 con 0x7f6cf00d3210 2024-04-17T07:33:46.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.389+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cd0003520 con 0x7f6cf00d3210 2024-04-17T07:33:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.390+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2275453879 0 0) 0x7f6cd0003520 con 0x7f6cf00d3210 2024-04-17T07:33:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.390+0000 7f6cef400640 1 -- 172.21.15.67:0/2233680027 >> v1:172.21.15.187:6789/0 conn(0x7f6cf00cfab0 legacy=0x7f6cf00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.390+0000 7f6cef400640 1 -- 172.21.15.67:0/2233680027 >> v1:172.21.15.67:6789/0 conn(0x7f6cf00d3210 legacy=0x7f6cf00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.390+0000 7f6cef400640 1 -- 172.21.15.67:0/2233680027 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf00d0670 con 0x7f6cf015ecb0 2024-04-17T07:33:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.391+0000 7f6cef400640 1 --2- 172.21.15.67:0/2233680027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00d2cd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f81da91e66214ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0037070 con 0x7f6cf015ecb0 2024-04-17T07:33:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0030510 con 0x7f6cf015ecb0 2024-04-17T07:33:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cee000640 1 -- 172.21.15.67:0/2233680027 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80019f0 con 0x7f6cf015ecb0 2024-04-17T07:33:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 -- 172.21.15.67:0/2233680027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 msgr2=0x7f6cf00d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/2233680027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00d2cd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b790 comp rx=0 tx=0).stop 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 -- 172.21.15.67:0/2233680027 shutdown_connections 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/2233680027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00d2cd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 -- 172.21.15.67:0/2233680027 >> 172.21.15.67:0/2233680027 conn(0x7f6cf008bff0 msgr2=0x7f6cf00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 -- 172.21.15.67:0/2233680027 shutdown_connections 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.392+0000 7f6cf4e00640 1 -- 172.21.15.67:0/2233680027 wait complete. 2024-04-17T07:33:46.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.394+0000 7f6cf4e00640 1 Processor -- start 2024-04-17T07:33:46.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.394+0000 7f6cf4e00640 1 -- start start 2024-04-17T07:33:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00d3210 0x7f6cf00b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:46.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00b9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf015f210 con 0x7f6cf00cfab0 2024-04-17T07:33:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf015f090 con 0x7f6cf015ecb0 2024-04-17T07:33:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cf4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf00b46d0 con 0x7f6cf00d3210 2024-04-17T07:33:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.395+0000 7f6cef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56682/0 (socket says 172.21.15.67:56682) 2024-04-17T07:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 -- 172.21.15.67:0/365688494 learned_addr learned my addr 172.21.15.67:0/365688494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cefe00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00b9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 -- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00d3210 msgr2=0x7f6cf00b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00d3210 0x7f6cf00b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 -- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 msgr2=0x7f6cf00b9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00b9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 -- 172.21.15.67:0/365688494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf00cc800 con 0x7f6cf00cfab0 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.396+0000 7f6cef400640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00098c0 tx=0x7f6ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.397+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0042070 con 0x7f6cf00cfab0 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.397+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce003e870 con 0x7f6cf00cfab0 2024-04-17T07:33:46.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.397+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0037040 con 0x7f6cf00cfab0 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.397+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf00ccac0 con 0x7f6cf00cfab0 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.397+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf00cd080 con 0x7f6cf00cfab0 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.398+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf0068a10 con 0x7f6cf00cfab0 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.399+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f6ce0045070 con 0x7f6cf00cfab0 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.400+0000 7f6ce7400640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 0x7f6ccc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.400+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f6ce00add90 con 0x7f6cf00cfab0 2024-04-17T07:33:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.401+0000 7f6ceea00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 0x7f6ccc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.402+0000 7f6ceea00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 0x7f6ccc06efb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00caf90 tx=0x7f6cd8001420 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:46.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.405+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6ce007bc10 con 0x7f6cf00cfab0 2024-04-17T07:33:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.632+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1y", "target": ["mon-mgr", ""]}) -- 0x7f6cf00cb3a0 con 0x7f6ccc06caf0 2024-04-17T07:33:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.821+0000 7f6ce7400640 1 -- 172.21.15.67:0/365688494 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7f6cf00cb3a0 con 0x7f6ccc06caf0 2024-04-17T07:33:46.822 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1/yearly 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.824+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 msgr2=0x7f6ccc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.824+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 0x7f6ccc06efb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00caf90 tx=0x7f6cd8001420 comp rx=0 tx=0).stop 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.824+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 msgr2=0x7f6cf0153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.824+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00098c0 tx=0x7f6ce00046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 shutdown_connections 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6ccc06caf0 0x7f6ccc06efb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf015ecb0 0x7f6cf00b9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf00d3210 0x7f6cf00b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 --2- 172.21.15.67:0/365688494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf00cfab0 0x7f6cf0153a40 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:46.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 >> 172.21.15.67:0/365688494 conn(0x7f6cf008bff0 msgr2=0x7f6cf00d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:46.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 shutdown_connections 2024-04-17T07:33:46.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:46.825+0000 7f6cf4e00640 1 -- 172.21.15.67:0/365688494 wait complete. 2024-04-17T07:33:46.850 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-04-17T07:33:46.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:46.850 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-04-17T07:33:46.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-04-17T07:33:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.131+0000 7fc383200640 1 Processor -- start 2024-04-17T07:33:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.132+0000 7fc383200640 1 -- start start 2024-04-17T07:33:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc383200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc384145f80 0x7fc3840cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc383200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc38406c270 0x7fc38406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc383200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc384146360 con 0x7fc3840cd750 2024-04-17T07:33:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc383200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc3840727a0 con 0x7fc38406c270 2024-04-17T07:33:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc383200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc384071e70 con 0x7fc384145f80 2024-04-17T07:33:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.133+0000 7fc382800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc38406c270 0x7fc38406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc38406c270 0x7fc38406e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56694/0 (socket says 172.21.15.67:56694) 2024-04-17T07:33:47.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 -- 172.21.15.67:0/3470106028 learned_addr learned my addr 172.21.15.67:0/3470106028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc381e00640 1 --2- 172.21.15.67:0/3470106028 >> v2:172.21.15.187:3300/0 conn(0x7fc384145f80 0x7fc3840cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 -- 172.21.15.67:0/3470106028 >> v2:172.21.15.187:3300/0 conn(0x7fc384145f80 msgr2=0x7fc3840cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 --2- 172.21.15.67:0/3470106028 >> v2:172.21.15.187:3300/0 conn(0x7fc384145f80 0x7fc3840cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 -- 172.21.15.67:0/3470106028 >> v1:172.21.15.67:6789/0 conn(0x7fc3840cd750 legacy=0x7fc38406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc382800640 1 -- 172.21.15.67:0/3470106028 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc384072150 con 0x7fc38406c270 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.134+0000 7fc380a00640 1 -- 172.21.15.67:0/3470106028 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159573251 0 0) 0x7fc384146360 con 0x7fc3840cd750 2024-04-17T07:33:47.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.135+0000 7fc382800640 1 --2- 172.21.15.67:0/3470106028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc38406c270 0x7fc38406e1c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fc38406c9c0 tx=0x7fc37c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d53fc21f44dae28 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc380a00640 1 -- 172.21.15.67:0/3470106028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c02ca80 con 0x7fc38406c270 2024-04-17T07:33:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc380a00640 1 -- 172.21.15.67:0/3470106028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c02cc20 con 0x7fc38406c270 2024-04-17T07:33:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc380a00640 1 -- 172.21.15.67:0/3470106028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c0343f0 con 0x7fc38406c270 2024-04-17T07:33:47.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 -- 172.21.15.67:0/3470106028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc38406c270 msgr2=0x7fc38406e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 --2- 172.21.15.67:0/3470106028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc38406c270 0x7fc38406e1c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fc38406c9c0 tx=0x7fc37c02bee0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 -- 172.21.15.67:0/3470106028 shutdown_connections 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 --2- 172.21.15.67:0/3470106028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc38406c270 0x7fc38406e1c0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 --2- 172.21.15.67:0/3470106028 >> v2:172.21.15.187:3300/0 conn(0x7fc384145f80 0x7fc3840cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.136+0000 7fc383200640 1 -- 172.21.15.67:0/3470106028 >> 172.21.15.67:0/3470106028 conn(0x7fc3840b2190 msgr2=0x7fc3840b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.137+0000 7fc383200640 1 -- 172.21.15.67:0/3470106028 shutdown_connections 2024-04-17T07:33:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.137+0000 7fc383200640 1 -- 172.21.15.67:0/3470106028 wait complete. 2024-04-17T07:33:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.138+0000 7fc383200640 1 Processor -- start 2024-04-17T07:33:47.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.138+0000 7fc383200640 1 -- start start 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.139+0000 7fc383200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38406c270 0x7fc3840ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.139+0000 7fc383200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840cd750 0x7fc38415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc383200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc384145f80 0x7fc3840a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc383200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc384072db0 con 0x7fc384145f80 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc383200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc384072c30 con 0x7fc3840cd750 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc383200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc38406e700 con 0x7fc38406c270 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc384145f80 0x7fc3840a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc384145f80 0x7fc3840a9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56710/0 (socket says 172.21.15.67:56710) 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 -- 172.21.15.67:0/1678932291 learned_addr learned my addr 172.21.15.67:0/1678932291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc381400640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840cd750 0x7fc38415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc381e00640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38406c270 0x7fc3840ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 -- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38406c270 msgr2=0x7fc3840ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38406c270 0x7fc3840ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 -- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840cd750 msgr2=0x7fc38415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840cd750 0x7fc38415bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc382800640 1 -- 172.21.15.67:0/1678932291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3840aaa90 con 0x7fc384145f80 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.140+0000 7fc381400640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840cd750 0x7fc38415bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc382800640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc384145f80 0x7fc3840a9ed0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fc37c036040 tx=0x7fc37c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c043030 con 0x7fc384145f80 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c03e450 con 0x7fc384145f80 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc383200640 1 -- 172.21.15.67:0/1678932291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3841e5ca0 con 0x7fc384145f80 2024-04-17T07:33:47.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c009510 con 0x7fc384145f80 2024-04-17T07:33:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.141+0000 7fc383200640 1 -- 172.21.15.67:0/1678932291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3841e6260 con 0x7fc384145f80 2024-04-17T07:33:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.142+0000 7fc383200640 1 -- 172.21.15.67:0/1678932291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc38406e1c0 con 0x7fc384145f80 2024-04-17T07:33:47.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.143+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7fc37c005d10 con 0x7fc384145f80 2024-04-17T07:33:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.144+0000 7fc37a000640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc36406caf0 0x7fc36406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.144+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7fc37c0ae300 con 0x7fc384145f80 2024-04-17T07:33:47.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.145+0000 7fc381e00640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc36406caf0 0x7fc36406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.146+0000 7fc381e00640 1 --2- 172.21.15.67:0/1678932291 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc36406caf0 0x7fc36406efb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc374006fd0 tx=0x7fc374008040 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.149+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc38406e1c0 con 0x7fc384145f80 2024-04-17T07:33:47.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.383+0000 7fc383200640 1 -- 172.21.15.67:0/1678932291 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) -- 0x7fc384142290 con 0x7fc36406caf0 2024-04-17T07:33:47.386 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:47.386+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-04-17T07:33:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.387+0000 7fc37a000640 1 -- 172.21.15.67:0/1678932291 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,y) ==== 74+0+0 (secure 0 0 0) 0x7fc384142290 con 0x7fc36406caf0 2024-04-17T07:33:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-04-17T07:33:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:47.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-04-17T07:33:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.645+0000 7f69ce200640 1 Processor -- start 2024-04-17T07:33:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.645+0000 7f69ce200640 1 -- start start 2024-04-17T07:33:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69ce200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69c8145f80 0x7f69c80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69ce200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69c806c270 0x7f69c806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69ce200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c8146360 con 0x7f69c80cd750 2024-04-17T07:33:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69ce200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f69c80727a0 con 0x7f69c8145f80 2024-04-17T07:33:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69ce200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f69c8071e70 con 0x7f69c806c270 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69cce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69c8145f80 0x7f69c80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69cce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69c8145f80 0x7f69c80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56724/0 (socket says 172.21.15.67:56724) 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69cce00640 1 -- 172.21.15.67:0/1647607586 learned_addr learned my addr 172.21.15.67:0/1647607586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.647+0000 7f69cd800640 1 --2- 172.21.15.67:0/1647607586 >> v2:172.21.15.187:3300/0 conn(0x7f69c806c270 0x7f69c806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.648+0000 7f69cce00640 1 -- 172.21.15.67:0/1647607586 >> v2:172.21.15.187:3300/0 conn(0x7f69c806c270 msgr2=0x7f69c806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.648+0000 7f69cce00640 1 --2- 172.21.15.67:0/1647607586 >> v2:172.21.15.187:3300/0 conn(0x7f69c806c270 0x7f69c806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.648+0000 7f69cce00640 1 -- 172.21.15.67:0/1647607586 >> v1:172.21.15.187:6789/0 conn(0x7f69c80cd750 legacy=0x7f69c806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.648+0000 7f69cce00640 1 -- 172.21.15.67:0/1647607586 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c8072150 con 0x7f69c8145f80 2024-04-17T07:33:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.649+0000 7f69cce00640 1 --2- 172.21.15.67:0/1647607586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c8145f80 0x7f69c80cd210 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f69c80ceed0 tx=0x7f69bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afddb32d48e2bbc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.649+0000 7f69c7400640 1 -- 172.21.15.67:0/1647607586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc02e070 con 0x7f69c8145f80 2024-04-17T07:33:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.649+0000 7f69c7400640 1 -- 172.21.15.67:0/1647607586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69bc037070 con 0x7f69c8145f80 2024-04-17T07:33:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.649+0000 7f69c7400640 1 -- 172.21.15.67:0/1647607586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc031a00 con 0x7f69c8145f80 2024-04-17T07:33:47.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 -- 172.21.15.67:0/1647607586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c8145f80 msgr2=0x7f69c80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 --2- 172.21.15.67:0/1647607586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c8145f80 0x7f69c80cd210 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f69c80ceed0 tx=0x7f69bc02b880 comp rx=0 tx=0).stop 2024-04-17T07:33:47.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 -- 172.21.15.67:0/1647607586 shutdown_connections 2024-04-17T07:33:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 --2- 172.21.15.67:0/1647607586 >> v2:172.21.15.187:3300/0 conn(0x7f69c806c270 0x7f69c806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 --2- 172.21.15.67:0/1647607586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c8145f80 0x7f69c80cd210 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 -- 172.21.15.67:0/1647607586 >> 172.21.15.67:0/1647607586 conn(0x7f69c80b2190 msgr2=0x7f69c80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 -- 172.21.15.67:0/1647607586 shutdown_connections 2024-04-17T07:33:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.650+0000 7f69ce200640 1 -- 172.21.15.67:0/1647607586 wait complete. 2024-04-17T07:33:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.652+0000 7f69ce200640 1 Processor -- start 2024-04-17T07:33:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.652+0000 7f69ce200640 1 -- start start 2024-04-17T07:33:47.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c806c270 0x7f69c80cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69c80cd750 0x7f69c80cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69c8145f80 0x7f69c8140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69c8072db0 con 0x7f69c806c270 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69c8072c30 con 0x7f69c8145f80 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69ce200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69c806e700 con 0x7f69c80cd750 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69cce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c806c270 0x7f69c80cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69cce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c806c270 0x7f69c80cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56734/0 (socket says 172.21.15.67:56734) 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.653+0000 7f69cce00640 1 -- 172.21.15.67:0/3395759335 learned_addr learned my addr 172.21.15.67:0/3395759335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 -- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69c80cd750 msgr2=0x7f69c80cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69c80cd750 0x7f69c80cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 -- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69c8145f80 msgr2=0x7f69c8140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69c8145f80 0x7f69c8140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 -- 172.21.15.67:0/3395759335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c80b47b0 con 0x7f69c806c270 2024-04-17T07:33:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69cce00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69c806c270 0x7f69c80cb9c0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f69bc02b8c0 tx=0x7f69bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc032670 con 0x7f69c806c270 2024-04-17T07:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69bc032830 con 0x7f69c806c270 2024-04-17T07:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc032c20 con 0x7f69c806c270 2024-04-17T07:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.654+0000 7f69ce200640 1 -- 172.21.15.67:0/3395759335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c80b4a90 con 0x7f69c806c270 2024-04-17T07:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.655+0000 7f69ce200640 1 -- 172.21.15.67:0/3395759335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c815c060 con 0x7f69c806c270 2024-04-17T07:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.656+0000 7f69ce200640 1 -- 172.21.15.67:0/3395759335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c8068a10 con 0x7f69c806c270 2024-04-17T07:33:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.657+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f69bc04c060 con 0x7f69c806c270 2024-04-17T07:33:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.658+0000 7f69c4c00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f69a806caf0 0x7f69a806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.658+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f69bc0adbf0 con 0x7f69c806c270 2024-04-17T07:33:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.659+0000 7f69c7e00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f69a806caf0 0x7f69a806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:47.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.660+0000 7f69c7e00640 1 --2- 172.21.15.67:0/3395759335 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f69a806caf0 0x7f69a806efb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69c80cc440 tx=0x7f69b40079b0 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:47.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.662+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f69bc040070 con 0x7f69c806c270 2024-04-17T07:33:47.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.897+0000 7f69ce200640 1 -- 172.21.15.67:0/3395759335 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) -- 0x7f69c8145780 con 0x7f69a806caf0 2024-04-17T07:33:47.900 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:47.900+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:47.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:47.901+0000 7f69c4c00640 1 -- 172.21.15.67:0/3395759335 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f69c8145780 con 0x7f69a806caf0 2024-04-17T07:33:47.901 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:47.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-04-17T07:33:48.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.136+0000 7f91ef000640 1 Processor -- start 2024-04-17T07:33:48.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.136+0000 7f91ef000640 1 -- start start 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 0x7f91e80d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91e80d3210 0x7f91e80cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e80cf8e0 0x7f91e80d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91e80d0c30 con 0x7f91e80d3210 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f91e80d00b0 con 0x7f91e815ecb0 2024-04-17T07:33:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.137+0000 7f91ef000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91e80d0390 con 0x7f91e80cf8e0 2024-04-17T07:33:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91e80d3210 0x7f91e80cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91e80d3210 0x7f91e80cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56738/0 (socket says 172.21.15.67:56738) 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 -- 172.21.15.67:0/2448501713 learned_addr learned my addr 172.21.15.67:0/2448501713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ee600640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e80cf8e0 0x7f91e80d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91edc00640 1 --2- 172.21.15.67:0/2448501713 >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 0x7f91e80d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 -- 172.21.15.67:0/2448501713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e80cf8e0 msgr2=0x7f91e80d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e80cf8e0 0x7f91e80d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 -- 172.21.15.67:0/2448501713 >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 msgr2=0x7f91e80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 --2- 172.21.15.67:0/2448501713 >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 0x7f91e80d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91ed200640 1 -- 172.21.15.67:0/2448501713 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e80d0670 con 0x7f91e80d3210 2024-04-17T07:33:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.138+0000 7f91edc00640 1 --2- 172.21.15.67:0/2448501713 >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 0x7f91e80d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:48.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.140+0000 7f91ed200640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80d3210 0x7f91e80cf3a0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f91e80d05a0 tx=0x7f91d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92de4287304cb237 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.140+0000 7f91e7e00640 1 -- 172.21.15.67:0/2448501713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d80317c0 con 0x7f91e80d3210 2024-04-17T07:33:48.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.140+0000 7f91e7e00640 1 -- 172.21.15.67:0/2448501713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d8037070 con 0x7f91e80d3210 2024-04-17T07:33:48.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.140+0000 7f91e7e00640 1 -- 172.21.15.67:0/2448501713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8031b80 con 0x7f91e80d3210 2024-04-17T07:33:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 -- 172.21.15.67:0/2448501713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80d3210 msgr2=0x7f91e80cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80d3210 0x7f91e80cf3a0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f91e80d05a0 tx=0x7f91d802b860 comp rx=0 tx=0).stop 2024-04-17T07:33:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 -- 172.21.15.67:0/2448501713 shutdown_connections 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e80cf8e0 0x7f91e80d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 --2- 172.21.15.67:0/2448501713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80d3210 0x7f91e80cf3a0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 --2- 172.21.15.67:0/2448501713 >> v2:172.21.15.187:3300/0 conn(0x7f91e815ecb0 0x7f91e80d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 -- 172.21.15.67:0/2448501713 >> 172.21.15.67:0/2448501713 conn(0x7f91e808bff0 msgr2=0x7f91e80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 -- 172.21.15.67:0/2448501713 shutdown_connections 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.141+0000 7f91ef000640 1 -- 172.21.15.67:0/2448501713 wait complete. 2024-04-17T07:33:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.143+0000 7f91ef000640 1 Processor -- start 2024-04-17T07:33:48.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.143+0000 7f91ef000640 1 -- start start 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80cf8e0 0x7f91e8158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91e80d3210 0x7f91e815f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e815ecb0 0x7f91e815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91e80cfd00 con 0x7f91e80cf8e0 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f91e815f090 con 0x7f91e80d3210 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91ef000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91e8160e70 con 0x7f91e815ecb0 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80cf8e0 0x7f91e8158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80cf8e0 0x7f91e8158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56754/0 (socket says 172.21.15.67:56754) 2024-04-17T07:33:48.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 -- 172.21.15.67:0/933719956 learned_addr learned my addr 172.21.15.67:0/933719956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 -- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e815ecb0 msgr2=0x7f91e815f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91e815ecb0 0x7f91e815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 -- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91e80d3210 msgr2=0x7f91e815f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91e80d3210 0x7f91e815f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.144+0000 7f91edc00640 1 -- 172.21.15.67:0/933719956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e81610e0 con 0x7f91e80cf8e0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91edc00640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e80cf8e0 0x7f91e8158520 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00d7d0 tx=0x7f91dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc004030 con 0x7f91e80cf8e0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc0041d0 con 0x7f91e80cf8e0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91ef000640 1 -- 172.21.15.67:0/933719956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e81613c0 con 0x7f91e80cf8e0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc009620 con 0x7f91e80cf8e0 2024-04-17T07:33:48.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.145+0000 7f91ef000640 1 -- 172.21.15.67:0/933719956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e8161980 con 0x7f91e80cf8e0 2024-04-17T07:33:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.146+0000 7f91ef000640 1 -- 172.21.15.67:0/933719956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e8158b80 con 0x7f91e80cf8e0 2024-04-17T07:33:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.147+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f91dc0097c0 con 0x7f91e80cf8e0 2024-04-17T07:33:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.149+0000 7f91e5600640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f91cc06caf0 0x7f91cc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.149+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f91dc08c380 con 0x7f91e80cf8e0 2024-04-17T07:33:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.149+0000 7f91ed200640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f91cc06caf0 0x7f91cc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.151+0000 7f91ed200640 1 --2- 172.21.15.67:0/933719956 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f91cc06caf0 0x7f91cc06efb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91d8036000 tx=0x7f91d8030000 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.153+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f91dc05a310 con 0x7f91e80cf8e0 2024-04-17T07:33:48.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.380+0000 7f91ef000640 1 -- 172.21.15.67:0/933719956 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) -- 0x7f91e8068a10 con 0x7f91cc06caf0 2024-04-17T07:33:48.384 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:48.384+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.385+0000 7f91e5600640 1 -- 172.21.15.67:0/933719956 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f91e8068a10 con 0x7f91cc06caf0 2024-04-17T07:33:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:48.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-04-17T07:33:48.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.631+0000 7f1114600640 1 Processor -- start 2024-04-17T07:33:48.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.631+0000 7f1114600640 1 -- start start 2024-04-17T07:33:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.633+0000 7f1114600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f110c15b860 0x7f110c0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.633+0000 7f1114600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110c0cfcc0 con 0x7f110c0ce4d0 2024-04-17T07:33:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.633+0000 7f1114600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110c071f60 con 0x7f110c0b4f80 2024-04-17T07:33:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.633+0000 7f1114600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f110c071d10 con 0x7f110c15b860 2024-04-17T07:33:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.634+0000 7f1111200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f110c0ce4d0 0x7f110c0ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46110/0 (socket says 172.21.15.67:46110) 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.634+0000 7f1111200640 1 -- 172.21.15.67:0/3917880263 learned_addr learned my addr 172.21.15.67:0/3917880263 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.634+0000 7f1111c00640 1 --2- 172.21.15.67:0/3917880263 >> v2:172.21.15.187:3300/0 conn(0x7f110c15b860 0x7f110c0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.634+0000 7f110be00640 1 -- 172.21.15.67:0/3917880263 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3752134452 0 0) 0x7f110c0cfcc0 con 0x7f110c0ce4d0 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.634+0000 7f110be00640 1 -- 172.21.15.67:0/3917880263 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f4003520 con 0x7f110c0ce4d0 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.635+0000 7f1111c00640 1 -- 172.21.15.67:0/3917880263 >> v1:172.21.15.187:6789/0 conn(0x7f110c0b4f80 legacy=0x7f110c0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.635+0000 7f1111c00640 1 -- 172.21.15.67:0/3917880263 >> v1:172.21.15.67:6789/0 conn(0x7f110c0ce4d0 legacy=0x7f110c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.635+0000 7f1111c00640 1 -- 172.21.15.67:0/3917880263 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c072240 con 0x7f110c15b860 2024-04-17T07:33:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.636+0000 7f1111c00640 1 --2- 172.21.15.67:0/3917880263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c15b860 0x7f110c0d0420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb6db33b30ad071 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.636+0000 7f110be00640 1 -- 172.21.15.67:0/3917880263 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100034070 con 0x7f110c15b860 2024-04-17T07:33:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.636+0000 7f110be00640 1 -- 172.21.15.67:0/3917880263 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1100030610 con 0x7f110c15b860 2024-04-17T07:33:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f110be00640 1 -- 172.21.15.67:0/3917880263 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000309c0 con 0x7f110c15b860 2024-04-17T07:33:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 -- 172.21.15.67:0/3917880263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c15b860 msgr2=0x7f110c0d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 --2- 172.21.15.67:0/3917880263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c15b860 0x7f110c0d0420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b960 comp rx=0 tx=0).stop 2024-04-17T07:33:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 -- 172.21.15.67:0/3917880263 shutdown_connections 2024-04-17T07:33:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 --2- 172.21.15.67:0/3917880263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c15b860 0x7f110c0d0420 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 -- 172.21.15.67:0/3917880263 >> 172.21.15.67:0/3917880263 conn(0x7f110c067ee0 msgr2=0x7f110c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 -- 172.21.15.67:0/3917880263 shutdown_connections 2024-04-17T07:33:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.637+0000 7f1114600640 1 -- 172.21.15.67:0/3917880263 wait complete. 2024-04-17T07:33:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.639+0000 7f1114600640 1 Processor -- start 2024-04-17T07:33:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.639+0000 7f1114600640 1 -- start start 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f110c0b4f80 0x7f110c0cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c0ce4d0 0x7f110c0c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110c15b860 0x7f110c149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f110c072ae0 con 0x7f110c15b860 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f110c072800 con 0x7f110c0ce4d0 2024-04-17T07:33:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1114600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f110c15bc40 con 0x7f110c0b4f80 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.640+0000 7f1112600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110c15b860 0x7f110c149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110c15b860 0x7f110c149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56770/0 (socket says 172.21.15.67:56770) 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 -- 172.21.15.67:0/2006750195 learned_addr learned my addr 172.21.15.67:0/2006750195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1111c00640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f110c0b4f80 0x7f110c0cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1111200640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c0ce4d0 0x7f110c0c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 -- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f110c0b4f80 msgr2=0x7f110c0cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f110c0b4f80 0x7f110c0cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 -- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c0ce4d0 msgr2=0x7f110c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c0ce4d0 0x7f110c0c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 -- 172.21.15.67:0/2006750195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c0cde20 con 0x7f110c15b860 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1111200640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f110c0ce4d0 0x7f110c0c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.641+0000 7f1112600640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110c15b860 0x7f110c149e90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f110400c6b0 tx=0x7f110400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.642+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104003b70 con 0x7f110c15b860 2024-04-17T07:33:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.642+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1104003d30 con 0x7f110c15b860 2024-04-17T07:33:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.642+0000 7f1114600640 1 -- 172.21.15.67:0/2006750195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c0ab420 con 0x7f110c15b860 2024-04-17T07:33:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.642+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104005310 con 0x7f110c15b860 2024-04-17T07:33:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.642+0000 7f1114600640 1 -- 172.21.15.67:0/2006750195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c0ab9e0 con 0x7f110c15b860 2024-04-17T07:33:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.643+0000 7f1114600640 1 -- 172.21.15.67:0/2006750195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f110c068ca0 con 0x7f110c15b860 2024-04-17T07:33:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.644+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f11040054b0 con 0x7f110c15b860 2024-04-17T07:33:48.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.645+0000 7f1109600640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f10ec06caf0 0x7f10ec06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:48.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.645+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f110408bec0 con 0x7f110c15b860 2024-04-17T07:33:48.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.646+0000 7f1111c00640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f10ec06caf0 0x7f10ec06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.647+0000 7f1111c00640 1 --2- 172.21.15.67:0/2006750195 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f10ec06caf0 0x7f10ec06efb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1100036000 tx=0x7f1100008030 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.650+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1104059ee0 con 0x7f110c15b860 2024-04-17T07:33:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.882+0000 7f1114600640 1 -- 172.21.15.67:0/2006750195 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) -- 0x7f110c0b55e0 con 0x7f10ec06caf0 2024-04-17T07:33:48.885 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:48.885+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:48.887+0000 7f1109600640 1 -- 172.21.15.67:0/2006750195 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f110c0b55e0 con 0x7f10ec06caf0 2024-04-17T07:33:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:48.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-04-17T07:33:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.134+0000 7f6f2d400640 1 Processor -- start 2024-04-17T07:33:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.134+0000 7f6f2d400640 1 -- start start 2024-04-17T07:33:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.136+0000 7f6f2d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f2806e830 0x7f6f2815f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.136+0000 7f6f2d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f280cfe00 0x7f6f2815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.136+0000 7f6f2d400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2815beb0 con 0x7f6f280cce80 2024-04-17T07:33:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.136+0000 7f6f2d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2815c190 con 0x7f6f280cfe00 2024-04-17T07:33:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.136+0000 7f6f2d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2815ec80 con 0x7f6f2806e830 2024-04-17T07:33:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f280cfe00 0x7f6f2815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f280cfe00 0x7f6f2815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56774/0 (socket says 172.21.15.67:56774) 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 -- 172.21.15.67:0/382049737 learned_addr learned my addr 172.21.15.67:0/382049737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f26800640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f2806e830 0x7f6f2815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 -- 172.21.15.67:0/382049737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f2806e830 msgr2=0x7f6f2815f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f2806e830 0x7f6f2815f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 -- 172.21.15.67:0/382049737 >> v1:172.21.15.187:6789/0 conn(0x7f6f280cce80 legacy=0x7f6f280ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.137+0000 7f6f1de00640 1 -- 172.21.15.67:0/382049737 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2815ef60 con 0x7f6f280cfe00 2024-04-17T07:33:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.139+0000 7f6f1de00640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f2815b5f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000bb10 tx=0x7f6f1002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecc09e1500bae1d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.139+0000 7f6f25e00640 1 -- 172.21.15.67:0/382049737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1003b070 con 0x7f6f280cfe00 2024-04-17T07:33:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.139+0000 7f6f25e00640 1 -- 172.21.15.67:0/382049737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f10036440 con 0x7f6f280cfe00 2024-04-17T07:33:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.139+0000 7f6f25e00640 1 -- 172.21.15.67:0/382049737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10036810 con 0x7f6f280cfe00 2024-04-17T07:33:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 -- 172.21.15.67:0/382049737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 msgr2=0x7f6f2815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f2815b5f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000bb10 tx=0x7f6f1002e150 comp rx=0 tx=0).stop 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 -- 172.21.15.67:0/382049737 shutdown_connections 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f2815b5f0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 --2- 172.21.15.67:0/382049737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f2806e830 0x7f6f2815f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 -- 172.21.15.67:0/382049737 >> 172.21.15.67:0/382049737 conn(0x7f6f28067ee0 msgr2=0x7f6f2806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 -- 172.21.15.67:0/382049737 shutdown_connections 2024-04-17T07:33:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.140+0000 7f6f2d400640 1 -- 172.21.15.67:0/382049737 wait complete. 2024-04-17T07:33:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.142+0000 7f6f2d400640 1 Processor -- start 2024-04-17T07:33:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.142+0000 7f6f2d400640 1 -- start start 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f2806e830 0x7f6f281567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f280cce80 0x7f6f281478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f281e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2815bb30 con 0x7f6f280cfe00 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2815f9c0 con 0x7f6f2806e830 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.143+0000 7f6f2d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f280cecd0 con 0x7f6f280cce80 2024-04-17T07:33:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f281e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f281e6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56786/0 (socket says 172.21.15.67:56786) 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 -- 172.21.15.67:0/1305376801 learned_addr learned my addr 172.21.15.67:0/1305376801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f26800640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f2806e830 0x7f6f281567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f1de00640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f280cce80 0x7f6f281478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 -- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f280cce80 msgr2=0x7f6f281478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f280cce80 0x7f6f281478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 -- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f2806e830 msgr2=0x7f6f281567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f2806e830 0x7f6f281567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.144+0000 7f6f27200640 1 -- 172.21.15.67:0/1305376801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f281e7660 con 0x7f6f280cfe00 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f27200640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f280cfe00 0x7f6f281e6e10 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000c3c0 tx=0x7f6f2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20017070 con 0x7f6f280cfe00 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f200052d0 con 0x7f6f280cfe00 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f2d400640 1 -- 172.21.15.67:0/1305376801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f281e7940 con 0x7f6f280cfe00 2024-04-17T07:33:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20003850 con 0x7f6f280cfe00 2024-04-17T07:33:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.145+0000 7f6f2d400640 1 -- 172.21.15.67:0/1305376801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f281e7f00 con 0x7f6f280cfe00 2024-04-17T07:33:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.146+0000 7f6f2d400640 1 -- 172.21.15.67:0/1305376801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f28068ca0 con 0x7f6f280cfe00 2024-04-17T07:33:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.147+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f6f2001a070 con 0x7f6f280cfe00 2024-04-17T07:33:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.148+0000 7f6f1f400640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6f0406caf0 0x7f6f0406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.148+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f6f2008b3b0 con 0x7f6f280cfe00 2024-04-17T07:33:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.149+0000 7f6f26800640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6f0406caf0 0x7f6f0406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.150+0000 7f6f26800640 1 --2- 172.21.15.67:0/1305376801 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6f0406caf0 0x7f6f0406efb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009a60 tx=0x7f6f18003990 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.153+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6f200593d0 con 0x7f6f280cfe00 2024-04-17T07:33:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.389+0000 7f6f2d400640 1 -- 172.21.15.67:0/1305376801 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) -- 0x7f6f280afd70 con 0x7f6f0406caf0 2024-04-17T07:33:49.391 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:49.391+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-17T07:33:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.392+0000 7f6f1f400640 1 -- 172.21.15.67:0/1305376801 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) ==== 96+0+0 (secure 0 0 0) 0x7f6f280afd70 con 0x7f6f0406caf0 2024-04-17T07:33:49.392 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-04-17T07:33:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:49.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-04-17T07:33:49.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.631+0000 7f3967200640 1 Processor -- start 2024-04-17T07:33:49.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.631+0000 7f3967200640 1 -- start start 2024-04-17T07:33:49.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f3967200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39680af400 0x7f3968162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f3967200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f396815f780 con 0x7f3968159020 2024-04-17T07:33:49.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f3967200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f396815fa60 con 0x7f396806b8c0 2024-04-17T07:33:49.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f3967200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f396815f540 con 0x7f39680af400 2024-04-17T07:33:49.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f395d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39680af400 0x7f3968162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f395d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39680af400 0x7f3968162ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56798/0 (socket says 172.21.15.67:56798) 2024-04-17T07:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.633+0000 7f395d400640 1 -- 172.21.15.67:0/3183807925 learned_addr learned my addr 172.21.15.67:0/3183807925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.634+0000 7f395d400640 1 -- 172.21.15.67:0/3183807925 >> v1:172.21.15.187:6789/0 conn(0x7f396806b8c0 legacy=0x7f39680adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.634+0000 7f395d400640 1 -- 172.21.15.67:0/3183807925 >> v1:172.21.15.67:6789/0 conn(0x7f3968159020 legacy=0x7f396815ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.634+0000 7f395d400640 1 -- 172.21.15.67:0/3183807925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39681624a0 con 0x7f39680af400 2024-04-17T07:33:49.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.635+0000 7f395d400640 1 --2- 172.21.15.67:0/3183807925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39680af400 0x7f3968162ca0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f3950009870 tx=0x7f395002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a5696c72c4ad52 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.636+0000 7f3965400640 1 -- 172.21.15.67:0/3183807925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950035020 con 0x7f39680af400 2024-04-17T07:33:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.636+0000 7f3965400640 1 -- 172.21.15.67:0/3183807925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3950031870 con 0x7f39680af400 2024-04-17T07:33:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.636+0000 7f3965400640 1 -- 172.21.15.67:0/3183807925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950031c40 con 0x7f39680af400 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 -- 172.21.15.67:0/3183807925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39680af400 msgr2=0x7f3968162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 --2- 172.21.15.67:0/3183807925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39680af400 0x7f3968162ca0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f3950009870 tx=0x7f395002b910 comp rx=0 tx=0).stop 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 -- 172.21.15.67:0/3183807925 shutdown_connections 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 --2- 172.21.15.67:0/3183807925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39680af400 0x7f3968162ca0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 -- 172.21.15.67:0/3183807925 >> 172.21.15.67:0/3183807925 conn(0x7f39680b2190 msgr2=0x7f39680b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 -- 172.21.15.67:0/3183807925 shutdown_connections 2024-04-17T07:33:49.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.637+0000 7f3967200640 1 -- 172.21.15.67:0/3183807925 wait complete. 2024-04-17T07:33:49.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.639+0000 7f3967200640 1 Processor -- start 2024-04-17T07:33:49.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.639+0000 7f3967200640 1 -- start start 2024-04-17T07:33:49.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f396806b8c0 0x7f3968154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39680af400 0x7f39681e8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3968159020 0x7f39681e8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39681631e0 con 0x7f396806b8c0 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f396815fd40 con 0x7f39680af400 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3967200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3968163360 con 0x7f3968159020 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3965e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f396806b8c0 0x7f3968154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3965e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f396806b8c0 0x7f3968154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56810/0 (socket says 172.21.15.67:56810) 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3965e00640 1 -- 172.21.15.67:0/3672002085 learned_addr learned my addr 172.21.15.67:0/3672002085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f395d400640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39680af400 0x7f39681e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3965e00640 1 -- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3968159020 msgr2=0x7f39681e8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.640+0000 7f3965e00640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3968159020 0x7f39681e8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3965e00640 1 -- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39680af400 msgr2=0x7f39681e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3965e00640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39680af400 0x7f39681e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:49.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3965e00640 1 -- 172.21.15.67:0/3672002085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39680b04f0 con 0x7f396806b8c0 2024-04-17T07:33:49.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3965e00640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f396806b8c0 0x7f3968154160 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f395800c3c0 tx=0x7f395800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39580052d0 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3958005490 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3958007260 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3967200640 1 -- 172.21.15.67:0/3672002085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39680b07d0 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.641+0000 7f3967200640 1 -- 172.21.15.67:0/3672002085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39680b0d90 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.643+0000 7f3967200640 1 -- 172.21.15.67:0/3672002085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3968068a10 con 0x7f396806b8c0 2024-04-17T07:33:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.644+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f3958007400 con 0x7f396806b8c0 2024-04-17T07:33:49.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.644+0000 7f395ea00640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f394406caf0 0x7f394406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:49.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.645+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f395808b590 con 0x7f396806b8c0 2024-04-17T07:33:49.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.645+0000 7f395d400640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f394406caf0 0x7f394406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:49.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.646+0000 7f395d400640 1 --2- 172.21.15.67:0/3672002085 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f394406caf0 0x7f394406efb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3950036000 tx=0x7f395003c040 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.649+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3958059520 con 0x7f396806b8c0 2024-04-17T07:33:49.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.877+0000 7f3967200640 1 -- 172.21.15.67:0/3672002085 --> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) -- 0x7f396806c7b0 con 0x7f394406caf0 2024-04-17T07:33:49.879 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:49.879+0000 7f98bfa00640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:49.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:49.880+0000 7f395ea00640 1 -- 172.21.15.67:0/3672002085 <== mgr.6967 v2:172.21.15.187:6832/2946630767 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f396806c7b0 con 0x7f394406caf0 2024-04-17T07:33:49.880 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-04-17T07:33:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T07:33:49.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:49.895 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-04-17T07:33:49.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:49.953 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-04-17T07:33:50.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:50.009 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-04-17T07:33:50.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:50.066 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-04-17T07:33:50.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:50.121 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-04-17T07:33:50.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:50.171 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-04-17T07:33:50.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:50.226 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-04-17T07:33:50.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T07:33:50.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.505+0000 7f9081a00640 1 Processor -- start 2024-04-17T07:33:50.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.505+0000 7f9081a00640 1 -- start start 2024-04-17T07:33:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f9081a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f9081a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c163130 0x7f907c0c7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f9081a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907c0cf140 con 0x7f907c0d0d70 2024-04-17T07:33:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f9081a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f907c0c8240 con 0x7f907c163130 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f9081a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f907c0cf420 con 0x7f907c162d50 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f907b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c163130 0x7f907c0c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f907b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c163130 0x7f907c0c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56824/0 (socket says 172.21.15.67:56824) 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.506+0000 7f907b400640 1 -- 172.21.15.67:0/1440527869 learned_addr learned my addr 172.21.15.67:0/1440527869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.507+0000 7f907b400640 1 -- 172.21.15.67:0/1440527869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 msgr2=0x7f907c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.507+0000 7f907b400640 1 --2- 172.21.15.67:0/1440527869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.507+0000 7f907b400640 1 -- 172.21.15.67:0/1440527869 >> v1:172.21.15.187:6789/0 conn(0x7f907c0d0d70 legacy=0x7f907c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.507+0000 7f907b400640 1 -- 172.21.15.67:0/1440527869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c0cf700 con 0x7f907c163130 2024-04-17T07:33:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.508+0000 7f907b400640 1 --2- 172.21.15.67:0/1440527869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c163130 0x7f907c0c7910 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28466d08ae278a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.508+0000 7f907aa00640 1 -- 172.21.15.67:0/1440527869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640317c0 con 0x7f907c163130 2024-04-17T07:33:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.508+0000 7f907aa00640 1 -- 172.21.15.67:0/1440527869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9064037070 con 0x7f907c163130 2024-04-17T07:33:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.508+0000 7f907aa00640 1 -- 172.21.15.67:0/1440527869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064031b80 con 0x7f907c163130 2024-04-17T07:33:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 -- 172.21.15.67:0/1440527869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c163130 msgr2=0x7f907c0c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 --2- 172.21.15.67:0/1440527869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c163130 0x7f907c0c7910 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906402b860 comp rx=0 tx=0).stop 2024-04-17T07:33:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 -- 172.21.15.67:0/1440527869 shutdown_connections 2024-04-17T07:33:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 --2- 172.21.15.67:0/1440527869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c163130 0x7f907c0c7910 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 --2- 172.21.15.67:0/1440527869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 -- 172.21.15.67:0/1440527869 >> 172.21.15.67:0/1440527869 conn(0x7f907c0b2fe0 msgr2=0x7f907c0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 -- 172.21.15.67:0/1440527869 shutdown_connections 2024-04-17T07:33:50.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.509+0000 7f9081a00640 1 -- 172.21.15.67:0/1440527869 wait complete. 2024-04-17T07:33:50.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.511+0000 7f9081a00640 1 Processor -- start 2024-04-17T07:33:50.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.511+0000 7f9081a00640 1 -- start start 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c1655f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c163130 0x7f907c165b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f907c0c7fd0 con 0x7f907c0d0d70 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f907c0c7e50 con 0x7f907c163130 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f9081a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f907c0b2bc0 con 0x7f907c162d50 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f907be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:50.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f907be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56826/0 (socket says 172.21.15.67:56826) 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.512+0000 7f907be00640 1 -- 172.21.15.67:0/2213231858 learned_addr learned my addr 172.21.15.67:0/2213231858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 -- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 msgr2=0x7f907c1655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c1655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 -- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c163130 msgr2=0x7f907c165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c163130 0x7f907c165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 -- 172.21.15.67:0/2213231858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c0b6aa0 con 0x7f907c0d0d70 2024-04-17T07:33:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f907be00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f906c00ba50 tx=0x7f906c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.513+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c019070 con 0x7f907c0d0d70 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.514+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f906c00cc00 con 0x7f907c0d0d70 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.514+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c014440 con 0x7f907c0d0d70 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.514+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907c0b55c0 con 0x7f907c0d0d70 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.514+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907c163890 con 0x7f907c0d0d70 2024-04-17T07:33:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.515+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907c0755d0 con 0x7f907c0d0d70 2024-04-17T07:33:50.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.516+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 73) ==== 91216+0+0 (secure 0 0 0) 0x7f906c01c070 con 0x7f907c0d0d70 2024-04-17T07:33:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.517+0000 7f9073e00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 0x7f905806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:50.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.518+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f906c08bdf0 con 0x7f907c0d0d70 2024-04-17T07:33:50.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.518+0000 7f907b400640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 0x7f905806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.520+0000 7f907b400640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 0x7f905806efb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9064036000 tx=0x7f9064030000 comp rx=0 tx=0).ready entity=mgr.6967 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.523+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f907c0b6aa0 con 0x7f907c0d0d70 2024-04-17T07:33:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.782+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f907c06dd00 con 0x7f907c0d0d70 2024-04-17T07:33:50.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.960+0000 7f9073e00640 1 -- 172.21.15.67:0/2213231858 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) ==== 93+0+0 (secure 0 0 0) 0x7f906c059e10 con 0x7f907c0d0d70 2024-04-17T07:33:50.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.964+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 msgr2=0x7f905806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:50.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.964+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 0x7f905806efb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9064036000 tx=0x7f9064030000 comp rx=0 tx=0).stop 2024-04-17T07:33:50.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.964+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 msgr2=0x7f907c150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:50.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.964+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f906c00ba50 tx=0x7f906c00bf10 comp rx=0 tx=0).stop 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 shutdown_connections 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f905806caf0 0x7f905806efb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c163130 0x7f907c165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c162d50 0x7f907c1655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 --2- 172.21.15.67:0/2213231858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c0d0d70 0x7f907c150a80 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.965+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 >> 172.21.15.67:0/2213231858 conn(0x7f907c0b2fe0 msgr2=0x7f907c0cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.966+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 shutdown_connections 2024-04-17T07:33:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:50.966+0000 7f9081a00640 1 -- 172.21.15.67:0/2213231858 wait complete. 2024-04-17T07:33:50.989 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:33:50.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:51.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:33:51.194 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:33:51.194 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:51.194 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:51.194 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:51.246 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:51.247 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:51.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:51.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:51.513+0000 7f13bb612040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:51.534 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:51.535+0000 7f675740f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:33:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.543+0000 7f2582800640 1 Processor -- start 2024-04-17T07:33:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.544+0000 7f2582800640 1 -- start start 2024-04-17T07:33:51.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2582800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c15ec60 0x7f257c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2582800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2582800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257c0d2d30 con 0x7f257c0cf910 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2582800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f257c0d49c0 con 0x7f257c15ec60 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2582800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f257c0d3660 con 0x7f257c0b6b30 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2581400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c15ec60 0x7f257c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2581400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c15ec60 0x7f257c0b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56852/0 (socket says 172.21.15.67:56852) 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.545+0000 7f2581400640 1 -- 172.21.15.67:0/3900036251 learned_addr learned my addr 172.21.15.67:0/3900036251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.546+0000 7f2580a00640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.546+0000 7f2581400640 1 -- 172.21.15.67:0/3900036251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 msgr2=0x7f257c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.546+0000 7f2581400640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.546+0000 7f2581400640 1 -- 172.21.15.67:0/3900036251 >> v1:172.21.15.187:6789/0 conn(0x7f257c0cf910 legacy=0x7f257c06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.546+0000 7f2581400640 1 -- 172.21.15.67:0/3900036251 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257c0d3010 con 0x7f257c15ec60 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.547+0000 7f2581400640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c15ec60 0x7f257c0b65f0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2570009870 tx=0x7f257002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61a45c5754ef329 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.547+0000 7f257be00640 1 -- 172.21.15.67:0/3900036251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570035020 con 0x7f257c15ec60 2024-04-17T07:33:51.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.547+0000 7f257be00640 1 -- 172.21.15.67:0/3900036251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2570031640 con 0x7f257c15ec60 2024-04-17T07:33:51.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.547+0000 7f257be00640 1 -- 172.21.15.67:0/3900036251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257002d040 con 0x7f257c15ec60 2024-04-17T07:33:51.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 -- 172.21.15.67:0/3900036251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c15ec60 msgr2=0x7f257c0b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:51.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c15ec60 0x7f257c0b65f0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2570009870 tx=0x7f257002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 -- 172.21.15.67:0/3900036251 shutdown_connections 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 --2- 172.21.15.67:0/3900036251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c15ec60 0x7f257c0b65f0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 -- 172.21.15.67:0/3900036251 >> 172.21.15.67:0/3900036251 conn(0x7f257c0d15f0 msgr2=0x7f257c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 -- 172.21.15.67:0/3900036251 shutdown_connections 2024-04-17T07:33:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.548+0000 7f2582800640 1 -- 172.21.15.67:0/3900036251 wait complete. 2024-04-17T07:33:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.550+0000 7f2582800640 1 Processor -- start 2024-04-17T07:33:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.550+0000 7f2582800640 1 -- start start 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f257c15ec60 0x7f257c161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f257c0cfda0 con 0x7f257c0cf910 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f257c0b6f70 con 0x7f257c15ec60 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2582800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f257c15f040 con 0x7f257c0b6b30 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2580a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2580a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56868/0 (socket says 172.21.15.67:56868) 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.551+0000 7f2580a00640 1 -- 172.21.15.67:0/3741664948 learned_addr learned my addr 172.21.15.67:0/3741664948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2581e00640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f257c15ec60 0x7f257c161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 msgr2=0x7f257c0ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:51.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f257c15ec60 msgr2=0x7f257c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:51.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f257c15ec60 0x7f257c161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 -- 172.21.15.67:0/3741664948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257c0ca140 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2580a00640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f256800d800 tx=0x7f256800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800b6c0 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.552+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2568019070 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.553+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257c0ca420 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.553+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800ba80 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.553+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257c0ca9e0 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.554+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f257c068a10 con 0x7f257c0cf910 2024-04-17T07:33:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.555+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7f256800bc20 con 0x7f257c0cf910 2024-04-17T07:33:51.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.555+0000 7f2579600640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 0x7f255c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:51.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.556+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f256808c590 con 0x7f257c0cf910 2024-04-17T07:33:51.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.556+0000 7f2581400640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 msgr2=0x7f255c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:51.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.556+0000 7f2581400640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 0x7f255c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:51.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.560+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f256805a520 con 0x7f257c0cf910 2024-04-17T07:33:51.572 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:51.572+0000 7f13bb612040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:51.597 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:51.598+0000 7f675740f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:33:51.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.757+0000 7f2581400640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 msgr2=0x7f255c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:51.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.757+0000 7f2581400640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 0x7f255c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:51.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.767+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f257c0b4730 con 0x7f257c0cf910 2024-04-17T07:33:51.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.770+0000 7f2579600640 1 -- 172.21.15.67:0/3741664948 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+17131 (secure 0 0 0) 0x7f2568059e20 con 0x7f257c0cf910 2024-04-17T07:33:51.770 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:51.770 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:41.840313+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/231129067":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2363868131":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2637392297":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6833/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2619020255":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6832/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/4219847990":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2597865993":"2024-04-18T07:33:41.840202+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:51.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 msgr2=0x7f255c06ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:51.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 0x7f255c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 msgr2=0x7f257c0cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f256800d800 tx=0x7f256800dcc0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 shutdown_connections 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f255c06caa0 0x7f255c06ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f257c15ec60 0x7f257c161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0cf910 0x7f257c0cb1e0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 --2- 172.21.15.67:0/3741664948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f257c0b6b30 0x7f257c0ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 >> 172.21.15.67:0/3741664948 conn(0x7f257c0d15f0 msgr2=0x7f257c0b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 shutdown_connections 2024-04-17T07:33:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:51.773+0000 7f2582800640 1 -- 172.21.15.67:0/3741664948 wait complete. 2024-04-17T07:33:51.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:33:24.887328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:33:25.960254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:51.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.002+0000 7fb394a00640 1 Processor -- start 2024-04-17T07:33:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.002+0000 7fb394a00640 1 -- start start 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb394a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb3900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb394a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb3900b3b50 0x7fb39006ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb394a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900d4f70 con 0x7fb39006cf80 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb394a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb3900d5250 con 0x7fb3900b3b50 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb394a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900cf4d0 con 0x7fb39015ec20 2024-04-17T07:33:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb38fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb39006cf80 0x7fb3900cfcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46144/0 (socket says 172.21.15.67:46144) 2024-04-17T07:33:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.004+0000 7fb38fe00640 1 -- 172.21.15.67:0/4210870548 learned_addr learned my addr 172.21.15.67:0/4210870548 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38f400640 1 --2- 172.21.15.67:0/4210870548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb3900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb387e00640 1 --2- 172.21.15.67:0/4210870548 >> v2:172.21.15.187:3300/0 conn(0x7fb3900b3b50 0x7fb39006ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 809729853 0 0) 0x7fb3900d4f70 con 0x7fb39006cf80 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb374003520 con 0x7fb39006cf80 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38f400640 1 -- 172.21.15.67:0/4210870548 >> v2:172.21.15.187:3300/0 conn(0x7fb3900b3b50 msgr2=0x7fb39006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38f400640 1 --2- 172.21.15.67:0/4210870548 >> v2:172.21.15.187:3300/0 conn(0x7fb3900b3b50 0x7fb39006ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2271743275 0 0) 0x7fb374003520 con 0x7fb39006cf80 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38f400640 1 -- 172.21.15.67:0/4210870548 >> v1:172.21.15.67:6789/0 conn(0x7fb39006cf80 legacy=0x7fb3900cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.005+0000 7fb38f400640 1 -- 172.21.15.67:0/4210870548 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900cf7b0 con 0x7fb39015ec20 2024-04-17T07:33:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.007+0000 7fb38f400640 1 --2- 172.21.15.67:0/4210870548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb3900d22b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb3800097a0 tx=0x7fb38002b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d24dc05b93ebd528 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.007+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380037070 con 0x7fb39015ec20 2024-04-17T07:33:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.007+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb380030510 con 0x7fb39015ec20 2024-04-17T07:33:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.007+0000 7fb38ea00640 1 -- 172.21.15.67:0/4210870548 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38002be60 con 0x7fb39015ec20 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 -- 172.21.15.67:0/4210870548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 msgr2=0x7fb3900d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 --2- 172.21.15.67:0/4210870548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb3900d22b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb3800097a0 tx=0x7fb38002b660 comp rx=0 tx=0).stop 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 -- 172.21.15.67:0/4210870548 shutdown_connections 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 --2- 172.21.15.67:0/4210870548 >> v2:172.21.15.187:3300/0 conn(0x7fb3900b3b50 0x7fb39006ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 --2- 172.21.15.67:0/4210870548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb3900d22b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 -- 172.21.15.67:0/4210870548 >> 172.21.15.67:0/4210870548 conn(0x7fb39008bff0 msgr2=0x7fb3900d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 -- 172.21.15.67:0/4210870548 shutdown_connections 2024-04-17T07:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.008+0000 7fb394a00640 1 -- 172.21.15.67:0/4210870548 wait complete. 2024-04-17T07:33:52.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.010+0000 7fb394a00640 1 Processor -- start 2024-04-17T07:33:52.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.010+0000 7fb394a00640 1 -- start start 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3900b3b50 0x7fb3901504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb390159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900d0560 con 0x7fb39006cf80 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900d03e0 con 0x7fb3900b3b50 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.011+0000 7fb394a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900d10d0 con 0x7fb39015ec20 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56880/0 (socket says 172.21.15.67:56880) 2024-04-17T07:33:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 -- 172.21.15.67:0/2406673844 learned_addr learned my addr 172.21.15.67:0/2406673844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:52.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38fe00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb390159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb387e00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3900b3b50 0x7fb3901504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 msgr2=0x7fb390159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb390159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3900b3b50 msgr2=0x7fb3901504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3900b3b50 0x7fb3901504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.012+0000 7fb38f400640 1 -- 172.21.15.67:0/2406673844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900b5110 con 0x7fb39006cf80 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb38f400640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb3800098c0 tx=0x7fb3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380041020 con 0x7fb39006cf80 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38003c870 con 0x7fb39006cf80 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380037040 con 0x7fb39006cf80 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3900b4e30 con 0x7fb39006cf80 2024-04-17T07:33:52.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.013+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3900b9480 con 0x7fb39006cf80 2024-04-17T07:33:52.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.014+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb390068a10 con 0x7fb39006cf80 2024-04-17T07:33:52.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.015+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7fb380044020 con 0x7fb39006cf80 2024-04-17T07:33:52.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.016+0000 7fb387400640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 0x7fb36c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.016+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7fb3800ad7d0 con 0x7fb39006cf80 2024-04-17T07:33:52.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.017+0000 7fb387e00640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 msgr2=0x7fb36c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:52.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.017+0000 7fb387e00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 0x7fb36c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:52.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.021+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb38004b030 con 0x7fb39006cf80 2024-04-17T07:33:52.020 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.020+0000 7f13bb612040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:52.073 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.072+0000 7f13bb612040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:52.078 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.079+0000 7f675740f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:33:52.123 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.123+0000 7f13bb612040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:52.134 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.135+0000 7f675740f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:33:52.183 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.183+0000 7f13bb612040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:52.188 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.189+0000 7f675740f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:33:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.218+0000 7fb387e00640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 msgr2=0x7fb36c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.218+0000 7fb387e00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 0x7fb36c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.222+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb3900d55f0 con 0x7fb39006cf80 2024-04-17T07:33:52.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.223+0000 7fb387400640 1 -- 172.21.15.67:0/2406673844 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) ==== 93+0+10 (secure 0 0 0) 0x7fb38007b6f0 con 0x7fb39006cf80 2024-04-17T07:33:52.223 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 msgr2=0x7fb36c06ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 0x7fb36c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 msgr2=0x7fb3900b9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb3800098c0 tx=0x7fb3800046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 shutdown_connections 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fb36c06caa0 0x7fb36c06ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb39015ec20 0x7fb390159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3900b3b50 0x7fb3901504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 --2- 172.21.15.67:0/2406673844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb39006cf80 0x7fb3900b9c80 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 >> 172.21.15.67:0/2406673844 conn(0x7fb39008bff0 msgr2=0x7fb390074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 shutdown_connections 2024-04-17T07:33:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.226+0000 7fb394a00640 1 -- 172.21.15.67:0/2406673844 wait complete. 2024-04-17T07:33:52.232 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.232+0000 7f13bb612040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:52.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:33:52.256 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.257+0000 7f675740f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:33:52.310 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.310+0000 7f675740f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:33:52.360 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.360+0000 7f13bb612040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:52.423 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.423+0000 7f13bb612040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:52.450 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.451+0000 7f675740f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:33:52.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.452+0000 7ff3ec400640 1 Processor -- start 2024-04-17T07:33:52.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.452+0000 7ff3ec400640 1 -- start start 2024-04-17T07:33:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.453+0000 7ff3ec400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff3e40d0b30 0x7ff3e40b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.453+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e40ce4d0 0x7ff3e40b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.453+0000 7ff3ec400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e4071f60 con 0x7ff3e415b860 2024-04-17T07:33:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.453+0000 7ff3ec400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e4071d10 con 0x7ff3e40d0b30 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.453+0000 7ff3ec400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e4072240 con 0x7ff3e40ce4d0 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3e415b860 0x7ff3e40d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46154/0 (socket says 172.21.15.67:46154) 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9a00640 1 -- 172.21.15.67:0/3152770953 learned_addr learned my addr 172.21.15.67:0/3152770953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9000640 1 --2- 172.21.15.67:0/3152770953 >> v2:172.21.15.187:3300/0 conn(0x7ff3e40d0b30 0x7ff3e40b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3ea400640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e40ce4d0 0x7ff3e40b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e3e00640 1 -- 172.21.15.67:0/3152770953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106202571 0 0) 0x7ff3e4071f60 con 0x7ff3e415b860 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e3e00640 1 -- 172.21.15.67:0/3152770953 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3cc003520 con 0x7ff3e415b860 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9000640 1 -- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e40ce4d0 msgr2=0x7ff3e40b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9000640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e40ce4d0 0x7ff3e40b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9000640 1 -- 172.21.15.67:0/3152770953 >> v1:172.21.15.67:6789/0 conn(0x7ff3e415b860 legacy=0x7ff3e40d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.454+0000 7ff3e9000640 1 -- 172.21.15.67:0/3152770953 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e4072520 con 0x7ff3e40d0b30 2024-04-17T07:33:52.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.456+0000 7ff3e9000640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40d0b30 0x7ff3e40b3f70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4071e90 tx=0x7ff3d002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c4b827b51c7b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.456+0000 7ff3e3e00640 1 -- 172.21.15.67:0/3152770953 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0035020 con 0x7ff3e40d0b30 2024-04-17T07:33:52.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.456+0000 7ff3e3e00640 1 -- 172.21.15.67:0/3152770953 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d0031800 con 0x7ff3e40d0b30 2024-04-17T07:33:52.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.456+0000 7ff3e3e00640 1 -- 172.21.15.67:0/3152770953 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0031bb0 con 0x7ff3e40d0b30 2024-04-17T07:33:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 -- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40d0b30 msgr2=0x7ff3e40b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40d0b30 0x7ff3e40b3f70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4071e90 tx=0x7ff3d002b8a0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 -- 172.21.15.67:0/3152770953 shutdown_connections 2024-04-17T07:33:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e40ce4d0 0x7ff3e40b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 --2- 172.21.15.67:0/3152770953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40d0b30 0x7ff3e40b3f70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 -- 172.21.15.67:0/3152770953 >> 172.21.15.67:0/3152770953 conn(0x7ff3e4067ee0 msgr2=0x7ff3e406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 -- 172.21.15.67:0/3152770953 shutdown_connections 2024-04-17T07:33:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.457+0000 7ff3ec400640 1 -- 172.21.15.67:0/3152770953 wait complete. 2024-04-17T07:33:52.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.459+0000 7ff3ec400640 1 Processor -- start 2024-04-17T07:33:52.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.459+0000 7ff3ec400640 1 -- start start 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40ce4d0 0x7ff3e414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e415b860 0x7ff3e41572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e415bc40 con 0x7ff3e40d0b30 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e4072ae0 con 0x7ff3e40ce4d0 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ec400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e406f380 con 0x7ff3e415b860 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56890/0 (socket says 172.21.15.67:56890) 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 -- 172.21.15.67:0/2657503115 learned_addr learned my addr 172.21.15.67:0/2657503115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9a00640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40ce4d0 0x7ff3e414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3ea400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e415b860 0x7ff3e41572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e415b860 msgr2=0x7ff3e41572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e415b860 0x7ff3e41572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40ce4d0 msgr2=0x7ff3e414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40ce4d0 0x7ff3e414f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.460+0000 7ff3e9000640 1 -- 172.21.15.67:0/2657503115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e414ebb0 con 0x7ff3e40d0b30 2024-04-17T07:33:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3e9000640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000c00 tx=0x7ff3d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0035040 con 0x7ff3e40d0b30 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d0003820 con 0x7ff3e40d0b30 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0003c10 con 0x7ff3e40d0b30 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e414ee90 con 0x7ff3e40d0b30 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.461+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e415d510 con 0x7ff3e40d0b30 2024-04-17T07:33:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.463+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e4068ca0 con 0x7ff3e40d0b30 2024-04-17T07:33:52.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.463+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7ff3d0038020 con 0x7ff3e40d0b30 2024-04-17T07:33:52.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.464+0000 7ff3e1600640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 0x7ff3c406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.464+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7ff3d00ae440 con 0x7ff3e40d0b30 2024-04-17T07:33:52.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.465+0000 7ff3e9a00640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 msgr2=0x7ff3c406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:52.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.465+0000 7ff3e9a00640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 0x7ff3c406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.469+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff3d007c3e0 con 0x7ff3e40d0b30 2024-04-17T07:33:52.476 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.476+0000 7f13bb612040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:52.520 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.520+0000 7f675740f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:52.533 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.533+0000 7f13bb612040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:52.573 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.574+0000 7f675740f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:33:52.583 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.583+0000 7f13bb612040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:52.639 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.640+0000 7f675740f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:33:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.666+0000 7ff3e9a00640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 msgr2=0x7ff3c406ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.666+0000 7ff3e9a00640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 0x7ff3c406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.679+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff3e40cdb70 con 0x7ff3e40d0b30 2024-04-17T07:33:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.680+0000 7ff3e1600640 1 -- 172.21.15.67:0/2657503115 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) ==== 104+0+11 (secure 0 0 0) 0x7ff3d007bce0 con 0x7ff3e40d0b30 2024-04-17T07:33:52.679 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:33:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 msgr2=0x7ff3c406ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 0x7ff3c406ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 msgr2=0x7ff3e4147660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000c00 tx=0x7ff3d00046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 shutdown_connections 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7ff3c406caa0 0x7ff3c406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff3e415b860 0x7ff3e41572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e40d0b30 0x7ff3e4147660 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 --2- 172.21.15.67:0/2657503115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3e40ce4d0 0x7ff3e414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 >> 172.21.15.67:0/2657503115 conn(0x7ff3e4067ee0 msgr2=0x7ff3e40b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 shutdown_connections 2024-04-17T07:33:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.682+0000 7ff3ec400640 1 -- 172.21.15.67:0/2657503115 wait complete. 2024-04-17T07:33:52.694 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:52.695+0000 7f675740f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:33:52.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:33:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.952+0000 7f0d43e00640 1 Processor -- start 2024-04-17T07:33:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.953+0000 7f0d43e00640 1 -- start start 2024-04-17T07:33:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d440d2ff0 0x7f0d440b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440b4c70 0x7f0d44161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d440b5110 con 0x7f0d440d33d0 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0d440b53f0 con 0x7f0d440d2ff0 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d440b56d0 con 0x7f0d440b4c70 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d42a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d440d2ff0 0x7f0d440b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.954+0000 7f0d43400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0d440d33d0 0x7f0d440d1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46158/0 (socket says 172.21.15.67:46158) 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d440d2ff0 0x7f0d440b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56904/0 (socket says 172.21.15.67:56904) 2024-04-17T07:33:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d43400640 1 -- 172.21.15.67:0/521483990 learned_addr learned my addr 172.21.15.67:0/521483990 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d3a000640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440b4c70 0x7f0d44161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42a00640 1 -- 172.21.15.67:0/521483990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440b4c70 msgr2=0x7f0d44161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42a00640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440b4c70 0x7f0d44161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42a00640 1 -- 172.21.15.67:0/521483990 >> v1:172.21.15.67:6789/0 conn(0x7f0d440d33d0 legacy=0x7f0d440d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42a00640 1 -- 172.21.15.67:0/521483990 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d440cf340 con 0x7f0d440d2ff0 2024-04-17T07:33:52.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.955+0000 7f0d42000640 1 -- 172.21.15.67:0/521483990 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029766778 0 0) 0x7f0d440b5110 con 0x7f0d440d33d0 2024-04-17T07:33:52.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.956+0000 7f0d42a00640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d2ff0 0x7f0d440b4730 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009d90 tx=0x7f0d3402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdee691ee6b91691 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d42000640 1 -- 172.21.15.67:0/521483990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34039070 con 0x7f0d440d2ff0 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d42000640 1 -- 172.21.15.67:0/521483990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3402cc50 con 0x7f0d440d2ff0 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d42000640 1 -- 172.21.15.67:0/521483990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34034440 con 0x7f0d440d2ff0 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 -- 172.21.15.67:0/521483990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d2ff0 msgr2=0x7f0d440b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d2ff0 0x7f0d440b4730 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009d90 tx=0x7f0d3402bea0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 -- 172.21.15.67:0/521483990 shutdown_connections 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440b4c70 0x7f0d44161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 --2- 172.21.15.67:0/521483990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d2ff0 0x7f0d440b4730 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 -- 172.21.15.67:0/521483990 >> 172.21.15.67:0/521483990 conn(0x7f0d4409da80 msgr2=0x7f0d4408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 -- 172.21.15.67:0/521483990 shutdown_connections 2024-04-17T07:33:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.957+0000 7f0d43e00640 1 -- 172.21.15.67:0/521483990 wait complete. 2024-04-17T07:33:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.959+0000 7f0d43e00640 1 Processor -- start 2024-04-17T07:33:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.959+0000 7f0d43e00640 1 -- start start 2024-04-17T07:33:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d440b4c70 0x7f0d44155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 0x7f0d440cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d440d0dd0 con 0x7f0d440d33d0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0d440d0c50 con 0x7f0d440b4c70 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.960+0000 7f0d43e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d440d0f50 con 0x7f0d440d2ff0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56918/0 (socket says 172.21.15.67:56918) 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 -- 172.21.15.67:0/1037349678 learned_addr learned my addr 172.21.15.67:0/1037349678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d42a00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d440b4c70 0x7f0d44155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d3a000640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 0x7f0d440cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 msgr2=0x7f0d440cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 0x7f0d440cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d440b4c70 msgr2=0x7f0d44155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d440b4c70 0x7f0d44155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 -- 172.21.15.67:0/1037349678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d440c83c0 con 0x7f0d440d33d0 2024-04-17T07:33:52.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d3a000640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 0x7f0d440cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:52.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.961+0000 7f0d43400640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:52.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.962+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c018070 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.962+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c007500 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.962+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d440c86a0 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.962+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c004a20 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.962+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d440c8c60 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.963+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d440b4460 con 0x7f0d440d33d0 2024-04-17T07:33:52.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.964+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7f0d3c01b070 con 0x7f0d440d33d0 2024-04-17T07:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.966+0000 7f0d3b400640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 0x7f0d2006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.966+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f0d3c08b220 con 0x7f0d440d33d0 2024-04-17T07:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.966+0000 7f0d42a00640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 msgr2=0x7f0d2006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.966+0000 7f0d42a00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 0x7f0d2006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:52.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:52.969+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0d440b4460 con 0x7f0d440d33d0 2024-04-17T07:33:52.969 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:52.969+0000 7f13bb612040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:53.109 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.110+0000 7f675740f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:33:53.128 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.127+0000 7f13bb612040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.167+0000 7f0d42a00640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 msgr2=0x7f0d2006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.167+0000 7f0d42a00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 0x7f0d2006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.174+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0d440755d0 con 0x7f0d440d33d0 2024-04-17T07:33:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.175+0000 7f0d3b400640 1 -- 172.21.15.67:0/1037349678 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) ==== 100+0+11 (secure 0 0 0) 0x7f0d3c058bf0 con 0x7f0d440d33d0 2024-04-17T07:33:53.174 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:33:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 msgr2=0x7f0d2006ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 0x7f0d2006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 msgr2=0x7f0d440ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).stop 2024-04-17T07:33:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 shutdown_connections 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f0d2006cad0 0x7f0d2006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d440d33d0 0x7f0d440ce0e0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d440d2ff0 0x7f0d440cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 --2- 172.21.15.67:0/1037349678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d440b4c70 0x7f0d44155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 >> 172.21.15.67:0/1037349678 conn(0x7f0d4409da80 msgr2=0x7f0d44157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 shutdown_connections 2024-04-17T07:33:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.178+0000 7f0d43e00640 1 -- 172.21.15.67:0/1037349678 wait complete. 2024-04-17T07:33:53.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T07:33:53.237 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.236+0000 7f13bb612040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:53.282 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.283+0000 7f675740f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:33:53.288 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.288+0000 7f13bb612040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:53.399 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.400+0000 7f675740f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:33:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.406+0000 7f61a5600640 1 Processor -- start 2024-04-17T07:33:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.406+0000 7f61a5600640 1 -- start start 2024-04-17T07:33:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f61a5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f61a00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f61a5600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a015d060 con 0x7f61a0072570 2024-04-17T07:33:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f61a5600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a015d340 con 0x7f61a00cbc00 2024-04-17T07:33:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f61a5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61a00cfe90 con 0x7f61a00b63c0 2024-04-17T07:33:53.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f619f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61a0072570 0x7f61a00ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44152/0 (socket says 172.21.15.67:44152) 2024-04-17T07:33:53.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.407+0000 7f619f400640 1 -- 172.21.15.67:0/2663895235 learned_addr learned my addr 172.21.15.67:0/2663895235 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:53.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f61a4c00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f61a00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495874623 0 0) 0x7f61a015d060 con 0x7f61a0072570 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6184003520 con 0x7f61a0072570 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1455895747 0 0) 0x7f6184003520 con 0x7f61a0072570 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 msgr2=0x7f61a00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f61a00cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 >> v1:172.21.15.187:6789/0 conn(0x7f61a00cbc00 legacy=0x7f61a0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a015d620 con 0x7f61a0072570 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f618c001670 con 0x7f61a0072570 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 >> v1:172.21.15.67:6789/0 conn(0x7f61a0072570 legacy=0x7f61a00ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184003a50 0x7f6184003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f6184004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184004d40 0x7f61840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61a00cfe90 con 0x7f6184004d40 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f618400bb00 con 0x7f6184003a50 2024-04-17T07:33:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.408+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f618400bd00 con 0x7f61a00b63c0 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184004d40 0x7f61840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f619fe00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184003a50 0x7f6184003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f619f400640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f6184004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 -- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 msgr2=0x7f6184004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a00b63c0 0x7f6184004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 -- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184003a50 msgr2=0x7f6184003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184003a50 0x7f6184003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f61a4c00640 1 -- 172.21.15.67:0/2663895235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6184003520 con 0x7f6184004d40 2024-04-17T07:33:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.409+0000 7f619fe00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184003a50 0x7f6184003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.410+0000 7f61a4c00640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184004d40 0x7f61840051c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f61a00b6870 tx=0x7f6184005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0734cca01dabd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.410+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198039070 con 0x7f6184004d40 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.410+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619802cb90 con 0x7f6184004d40 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.410+0000 7f619ea00640 1 -- 172.21.15.67:0/2663895235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198009c10 con 0x7f6184004d40 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 -- 172.21.15.67:0/2663895235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184004d40 msgr2=0x7f61840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 --2- 172.21.15.67:0/2663895235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6184004d40 0x7f61840051c0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f61a00b6870 tx=0x7f6184005830 comp rx=0 tx=0).stop 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a4c00640 1 -- 172.21.15.67:0/2663895235 reap_dead start 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 -- 172.21.15.67:0/2663895235 shutdown_connections 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 -- 172.21.15.67:0/2663895235 >> 172.21.15.67:0/2663895235 conn(0x7f61a006bef0 msgr2=0x7f61a00b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 -- 172.21.15.67:0/2663895235 shutdown_connections 2024-04-17T07:33:53.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.411+0000 7f61a5600640 1 -- 172.21.15.67:0/2663895235 wait complete. 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.413+0000 7f61a5600640 1 Processor -- start 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.413+0000 7f61a5600640 1 -- start start 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184004d40 0x7f61a0159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a0142ce0 0x7f61a0143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61a00d02d0 con 0x7f61a015dfc0 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61a015dc70 con 0x7f61a0142ce0 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61a00cbfe0 con 0x7f6184004d40 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40514/0 (socket says 172.21.15.67:40514) 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.414+0000 7f61a4c00640 1 -- 172.21.15.67:0/1771197510 learned_addr learned my addr 172.21.15.67:0/1771197510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f619fe00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184004d40 0x7f61a0159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f619f400640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a0142ce0 0x7f61a0143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184004d40 msgr2=0x7f61a0159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184004d40 0x7f61a0159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a0142ce0 msgr2=0x7f61a0143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a0142ce0 0x7f61a0143140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 -- 172.21.15.67:0/1771197510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a015ecb0 con 0x7f61a015dfc0 2024-04-17T07:33:53.416 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.413+0000 7f13bb612040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:53.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.415+0000 7f61a4c00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f619800e230 tx=0x7f6198009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:53.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.416+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198039030 con 0x7f61a015dfc0 2024-04-17T07:33:53.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.416+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6198043070 con 0x7f61a015dfc0 2024-04-17T07:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.416+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619803d7f0 con 0x7f61a015dfc0 2024-04-17T07:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.416+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a015ef90 con 0x7f61a015dfc0 2024-04-17T07:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.416+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a015f550 con 0x7f61a015dfc0 2024-04-17T07:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.417+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a0068a10 con 0x7f61a015dfc0 2024-04-17T07:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.418+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7f6198005d10 con 0x7f61a015dfc0 2024-04-17T07:33:53.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.419+0000 7f6197e00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 0x7f618006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:53.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.419+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f61980ae340 con 0x7f61a015dfc0 2024-04-17T07:33:53.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.419+0000 7f619fe00640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 msgr2=0x7f618006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:53.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.419+0000 7f619fe00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 0x7f618006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:53.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.423+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f619807c2d0 con 0x7f61a015dfc0 2024-04-17T07:33:53.457 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.458+0000 7f675740f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:33:53.521 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.521+0000 7f13bb612040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:53.592 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.592+0000 7f675740f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:53.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.620+0000 7f619fe00640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 msgr2=0x7f618006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:53.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.620+0000 7f619fe00640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 0x7f618006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.642+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f61a015f930 con 0x7f61a015dfc0 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.643+0000 7f6197e00640 1 -- 172.21.15.67:0/1771197510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v143) ==== 80+0+477 (secure 0 0 0) 0x7f619807bbd0 con 0x7f61a015dfc0 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/231129067 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2363868131 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2637392297 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/1274719646 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2619020255 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/1274719646 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/4219847990 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.643 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2597865993 2024-04-18T07:33:41.840202+0000 2024-04-17T07:33:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T07:33:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 msgr2=0x7f618006ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 0x7f618006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 msgr2=0x7f61a015e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:53.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f619800e230 tx=0x7f6198009750 comp rx=0 tx=0).stop 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 shutdown_connections 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f618006cad0 0x7f618006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a015dfc0 0x7f61a015e440 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a0142ce0 0x7f61a0143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 --2- 172.21.15.67:0/1771197510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184004d40 0x7f61a0159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 >> 172.21.15.67:0/1771197510 conn(0x7f61a006bef0 msgr2=0x7f61a00cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 shutdown_connections 2024-04-17T07:33:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:53.646+0000 7f61a5600640 1 -- 172.21.15.67:0/1771197510 wait complete. 2024-04-17T07:33:53.663 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T07:33:53.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:53.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T07:33:53.696 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.695+0000 7f13bb612040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:53.712 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.713+0000 7f675740f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:33:53.744 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.744+0000 7f13bb612040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:53.764 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T07:33:53.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:33:53.764 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T07:33:53.812 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:53.812 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:53.812 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:53.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:53.863 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:53.863 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:33:53.863 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:53.885 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:53.885+0000 7f13bb612040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:53.906 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.907+0000 7f675740f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:33:53.913 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:33:53.914 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T07:33:53.914 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:33:53.914 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:33:53.959 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:53.960+0000 7f675740f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:33:53.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:33:53.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:54.111 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:54.112+0000 7f675740f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:33:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.179+0000 7fc04ec00640 1 Processor -- start 2024-04-17T07:33:54.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.180+0000 7fc04ec00640 1 -- start start 2024-04-17T07:33:54.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.181+0000 7fc04ec00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc04815ec60 0x7fc0480b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.181+0000 7fc04ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc0480cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.181+0000 7fc04ec00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0480d2d30 con 0x7fc0480cf910 2024-04-17T07:33:54.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.181+0000 7fc04ec00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc0480d49c0 con 0x7fc04815ec60 2024-04-17T07:33:54.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.181+0000 7fc04ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0480d3660 con 0x7fc0480b6b30 2024-04-17T07:33:54.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc04815ec60 0x7fc0480b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc04815ec60 0x7fc0480b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:52186/0 (socket says 172.21.15.67:52186) 2024-04-17T07:33:54.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 -- 172.21.15.67:0/1978096791 learned_addr learned my addr 172.21.15.67:0/1978096791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04ce00640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc0480cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 -- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 msgr2=0x7fc0480cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc0480cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 -- 172.21.15.67:0/1978096791 >> v1:172.21.15.187:6789/0 conn(0x7fc0480cf910 legacy=0x7fc04806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.182+0000 7fc04d800640 1 -- 172.21.15.67:0/1978096791 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0480d3010 con 0x7fc04815ec60 2024-04-17T07:33:54.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.184+0000 7fc04d800640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc04815ec60 0x7fc0480b65f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009870 tx=0x7fc03c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=390f05082deb8bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:54.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.184+0000 7fc047e00640 1 -- 172.21.15.67:0/1978096791 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c034070 con 0x7fc04815ec60 2024-04-17T07:33:54.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.184+0000 7fc047e00640 1 -- 172.21.15.67:0/1978096791 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc03c030840 con 0x7fc04815ec60 2024-04-17T07:33:54.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.184+0000 7fc047e00640 1 -- 172.21.15.67:0/1978096791 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c030c10 con 0x7fc04815ec60 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 -- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc04815ec60 msgr2=0x7fc0480b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc04815ec60 0x7fc0480b65f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009870 tx=0x7fc03c02b980 comp rx=0 tx=0).stop 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 -- 172.21.15.67:0/1978096791 shutdown_connections 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc0480cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1978096791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc04815ec60 0x7fc0480b65f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 -- 172.21.15.67:0/1978096791 >> 172.21.15.67:0/1978096791 conn(0x7fc0480d15f0 msgr2=0x7fc0480d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 -- 172.21.15.67:0/1978096791 shutdown_connections 2024-04-17T07:33:54.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.185+0000 7fc04ec00640 1 -- 172.21.15.67:0/1978096791 wait complete. 2024-04-17T07:33:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.187+0000 7fc04ec00640 1 Processor -- start 2024-04-17T07:33:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.187+0000 7fc04ec00640 1 -- start start 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc048147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0480cf910 0x7fc0480c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0480cfda0 con 0x7fc04815ec60 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0480b6f70 con 0x7fc0480cf910 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.188+0000 7fc04ec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc04815f040 con 0x7fc0480b6b30 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40528/0 (socket says 172.21.15.67:40528) 2024-04-17T07:33:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 -- 172.21.15.67:0/1160635633 learned_addr learned my addr 172.21.15.67:0/1160635633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04ce00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0480cf910 0x7fc0480c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04d800640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc048147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 msgr2=0x7fc048147980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc048147980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0480cf910 msgr2=0x7fc0480c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0480cf910 0x7fc0480c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.189+0000 7fc04e200640 1 -- 172.21.15.67:0/1160635633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc048147f60 con 0x7fc04815ec60 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc04e200640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c7d0 tx=0x7fc04000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc040003b70 con 0x7fc04815ec60 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc040003d30 con 0x7fc04815ec60 2024-04-17T07:33:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04000aa30 con 0x7fc04815ec60 2024-04-17T07:33:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc048148240 con 0x7fc04815ec60 2024-04-17T07:33:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.190+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc048148800 con 0x7fc04815ec60 2024-04-17T07:33:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.192+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc048068a10 con 0x7fc04815ec60 2024-04-17T07:33:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.192+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7fc04000abd0 con 0x7fc04815ec60 2024-04-17T07:33:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.194+0000 7fc045600640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 0x7fc02806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.194+0000 7fc04d800640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 msgr2=0x7fc02806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.194+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7fc04008bff0 con 0x7fc04815ec60 2024-04-17T07:33:54.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.195+0000 7fc04d800640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 0x7fc02806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.201+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc040059ef0 con 0x7fc04815ec60 2024-04-17T07:33:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.396+0000 7fc04d800640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 msgr2=0x7fc02806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.396+0000 7fc04d800640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 0x7fc02806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.443+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0480b91b0 con 0x7fc04815ec60 2024-04-17T07:33:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.444+0000 7fc045600640 1 -- 172.21.15.67:0/1160635633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7fc0400597f0 con 0x7fc04815ec60 2024-04-17T07:33:54.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:54.444 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.446+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 msgr2=0x7fc02806ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.446+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 0x7fc02806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.446+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 msgr2=0x7fc0481f4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.446+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc04000c7d0 tx=0x7fc04000cc90 comp rx=0 tx=0).stop 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.446+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 shutdown_connections 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7fc02806caa0 0x7fc02806ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc04815ec60 0x7fc0481f4450 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0480cf910 0x7fc0480c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 --2- 172.21.15.67:0/1160635633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0480b6b30 0x7fc048147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 >> 172.21.15.67:0/1160635633 conn(0x7fc0480d15f0 msgr2=0x7fc0480b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:54.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 shutdown_connections 2024-04-17T07:33:54.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.447+0000 7fc04ec00640 1 -- 172.21.15.67:0/1160635633 wait complete. 2024-04-17T07:33:54.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.668+0000 7f6d02000640 1 Processor -- start 2024-04-17T07:33:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.669+0000 7f6d02000640 1 -- start start 2024-04-17T07:33:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6d02000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 0x7f6cfc0cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6d02000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cfc0cff60 con 0x7f6cfc15ed10 2024-04-17T07:33:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6d02000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cfc0d3020 con 0x7f6cfc0d2ab0 2024-04-17T07:33:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6d02000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cfc0d3c30 con 0x7f6cfc0cf640 2024-04-17T07:33:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6cfaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6cfc15ed10 0x7f6cfc0d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44156/0 (socket says 172.21.15.67:44156) 2024-04-17T07:33:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.670+0000 7f6cfaa00640 1 -- 172.21.15.67:0/106361650 learned_addr learned my addr 172.21.15.67:0/106361650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfb400640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 0x7f6cfc0cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603974968 0 0) 0x7f6cfc0cff60 con 0x7f6cfc15ed10 2024-04-17T07:33:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ce0003520 con 0x7f6cfc15ed10 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3302959622 0 0) 0x7f6ce0003520 con 0x7f6cfc15ed10 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 msgr2=0x7f6cfc0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 0x7f6cfc0cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 >> v1:172.21.15.187:6789/0 conn(0x7f6cfc0d2ab0 legacy=0x7f6cfc0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc0d3300 con 0x7f6cfc15ed10 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.671+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6ce8001670 con 0x7f6cfc15ed10 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 >> v1:172.21.15.67:6789/0 conn(0x7f6cfc15ed10 legacy=0x7f6cfc0d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce00039e0 0x7f6ce0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 0x7f6ce00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc0d3c30 con 0x7f6ce0004800 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce000bb30 con 0x7f6ce00039e0 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce000bd30 con 0x7f6cfc0cf640 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfaa00640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.672+0000 7f6cfb400640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce00039e0 0x7f6ce0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.673+0000 7f6cfaa00640 1 -- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 msgr2=0x7f6ce00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.673+0000 7f6cfaa00640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0cf640 0x7f6ce00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.673+0000 7f6cfaa00640 1 -- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce00039e0 msgr2=0x7f6ce0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.673+0000 7f6cfaa00640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce00039e0 0x7f6ce0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.673+0000 7f6cfaa00640 1 -- 172.21.15.67:0/106361650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce0003520 con 0x7f6ce0004800 2024-04-17T07:33:54.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.674+0000 7f6cfaa00640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc15f110 tx=0x7f6ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f6686d50ab44de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.674+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8007c60 con 0x7f6ce0004800 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.674+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce8030070 con 0x7f6ce0004800 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.674+0000 7f6cfa000640 1 -- 172.21.15.67:0/106361650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8005d70 con 0x7f6ce0004800 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 -- 172.21.15.67:0/106361650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce0004800 msgr2=0x7f6ce0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 --2- 172.21.15.67:0/106361650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc15f110 tx=0x7f6ce0005830 comp rx=0 tx=0).stop 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6cfbe00640 1 -- 172.21.15.67:0/106361650 reap_dead start 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 -- 172.21.15.67:0/106361650 shutdown_connections 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 -- 172.21.15.67:0/106361650 >> 172.21.15.67:0/106361650 conn(0x7f6cfc06cbd0 msgr2=0x7f6cfc06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 -- 172.21.15.67:0/106361650 shutdown_connections 2024-04-17T07:33:54.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.675+0000 7f6d02000640 1 -- 172.21.15.67:0/106361650 wait complete. 2024-04-17T07:33:54.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.677+0000 7f6d02000640 1 Processor -- start 2024-04-17T07:33:54.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.677+0000 7f6d02000640 1 -- start start 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 0x7f6cfc1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0b5d40 0x7f6cfc0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc0d25b0 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc0d38c0 con 0x7f6ce0004800 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6d02000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cfc0ba300 con 0x7f6cfc0b5d40 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40552/0 (socket says 172.21.15.67:40552) 2024-04-17T07:33:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6cfbe00640 1 -- 172.21.15.67:0/2135836061 learned_addr learned my addr 172.21.15.67:0/2135836061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.678+0000 7f6cfb400640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 0x7f6cfc1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfaa00640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0b5d40 0x7f6cfc0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0b5d40 msgr2=0x7f6cfc0b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0b5d40 0x7f6cfc0b61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 msgr2=0x7f6cfc1f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 0x7f6cfc1f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 -- 172.21.15.67:0/2135836061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc148220 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfb400640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 0x7f6cfc1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cfbe00640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0d4920 tx=0x7f6cf400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4017070 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4005d60 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400a850 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc148500 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.679+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc148ac0 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.680+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc068b90 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.682+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7f6cf400a9f0 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.683+0000 7f6cf3400640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 0x7f6cdc06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:54.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.683+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f6cf408b360 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.683+0000 7f6cfb400640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 msgr2=0x7f6cdc06ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:54.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.683+0000 7f6cfb400640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 0x7f6cdc06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:54.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.687+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6cf40592f0 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.730 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:54.730+0000 7f13bb612040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.884+0000 7f6cfb400640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 msgr2=0x7f6cdc06ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:54.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.884+0000 7f6cfb400640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 0x7f6cdc06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:33:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.897+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cfc148c90 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.899+0000 7f6cf3400640 1 -- 172.21.15.67:0/2135836061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+17131 (secure 0 0 0) 0x7f6cf4058bf0 con 0x7f6cfc15f2d0 2024-04-17T07:33:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:54.899 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:41.840313+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/231129067":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2363868131":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2637392297":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6833/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2619020255":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6832/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/4219847990":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2597865993":"2024-04-18T07:33:41.840202+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 msgr2=0x7f6cdc06ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 0x7f6cdc06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 msgr2=0x7f6cfc15f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0d4920 tx=0x7f6cf400cb90 comp rx=0 tx=0).stop 2024-04-17T07:33:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 shutdown_connections 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f6cdc06cad0 0x7f6cdc06ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc15f2d0 0x7f6cfc15f750 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cfc0b5d40 0x7f6cfc0b61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 --2- 172.21.15.67:0/2135836061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ce0004800 0x7f6cfc1f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 >> 172.21.15.67:0/2135836061 conn(0x7f6cfc06cbd0 msgr2=0x7f6cfc0c9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 shutdown_connections 2024-04-17T07:33:54.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:54.902+0000 7f6d02000640 1 -- 172.21.15.67:0/2135836061 wait complete. 2024-04-17T07:33:54.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:33:24.887328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:33:25.960254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:33:54.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:33:54.999 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:55.000+0000 7f675740f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:33:55.064 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.064+0000 7f13bb612040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:55.124 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.124+0000 7f13bb612040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.130+0000 7f8434e00640 1 Processor -- start 2024-04-17T07:33:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.130+0000 7f8434e00640 1 -- start start 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.131+0000 7f8434e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430160110 0x7f8430161190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.131+0000 7f8434e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f843009b250 0x7f84300cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.131+0000 7f8434e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84301604f0 con 0x7f84300cd2b0 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.131+0000 7f8434e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f84300b41c0 con 0x7f8430160110 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.131+0000 7f8434e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f84300b2110 con 0x7f843009b250 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430160110 0x7f8430161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430160110 0x7f8430161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40556/0 (socket says 172.21.15.67:40556) 2024-04-17T07:33:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 -- 172.21.15.67:0/2980024197 learned_addr learned my addr 172.21.15.67:0/2980024197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842e800640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f843009b250 0x7f84300cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 -- 172.21.15.67:0/2980024197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f843009b250 msgr2=0x7f84300cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f843009b250 0x7f84300cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 -- 172.21.15.67:0/2980024197 >> v1:172.21.15.187:6789/0 conn(0x7f84300cd2b0 legacy=0x7f843009ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.132+0000 7f842de00640 1 -- 172.21.15.67:0/2980024197 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84300b23f0 con 0x7f8430160110 2024-04-17T07:33:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.134+0000 7f842de00640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430160110 0x7f8430161190 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8420009870 tx=0x7f842002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471d6a3e9be0011c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.134+0000 7f842d400640 1 -- 172.21.15.67:0/2980024197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420035020 con 0x7f8430160110 2024-04-17T07:33:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.134+0000 7f842d400640 1 -- 172.21.15.67:0/2980024197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8420031640 con 0x7f8430160110 2024-04-17T07:33:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.134+0000 7f842d400640 1 -- 172.21.15.67:0/2980024197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842002d040 con 0x7f8430160110 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 -- 172.21.15.67:0/2980024197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430160110 msgr2=0x7f8430161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430160110 0x7f8430161190 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8420009870 tx=0x7f842002b8f0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 -- 172.21.15.67:0/2980024197 shutdown_connections 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f843009b250 0x7f84300cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 --2- 172.21.15.67:0/2980024197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430160110 0x7f8430161190 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 -- 172.21.15.67:0/2980024197 >> 172.21.15.67:0/2980024197 conn(0x7f84300b4470 msgr2=0x7f84300b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 -- 172.21.15.67:0/2980024197 shutdown_connections 2024-04-17T07:33:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.135+0000 7f8434e00640 1 -- 172.21.15.67:0/2980024197 wait complete. 2024-04-17T07:33:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.137+0000 7f8434e00640 1 Processor -- start 2024-04-17T07:33:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.137+0000 7f8434e00640 1 -- start start 2024-04-17T07:33:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f843009b250 0x7f84300c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 0x7f84300cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84300b2c90 con 0x7f84300cd2b0 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f84300b29b0 con 0x7f843009b250 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8434e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f84300b31b0 con 0x7f8430160110 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40564/0 (socket says 172.21.15.67:40564) 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 -- 172.21.15.67:0/2553573535 learned_addr learned my addr 172.21.15.67:0/2553573535 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f842de00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f843009b250 0x7f84300c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f842e800640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 0x7f84300cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 -- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 msgr2=0x7f84300cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 0x7f84300cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 -- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f843009b250 msgr2=0x7f84300c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f843009b250 0x7f84300c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f8427e00640 1 -- 172.21.15.67:0/2553573535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84300c70d0 con 0x7f84300cd2b0 2024-04-17T07:33:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.138+0000 7f842e800640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 0x7f84300cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8427e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f841800c400 tx=0x7f841800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180052d0 con 0x7f84300cd2b0 2024-04-17T07:33:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8418005490 con 0x7f84300cd2b0 2024-04-17T07:33:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418007260 con 0x7f84300cd2b0 2024-04-17T07:33:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84300c73b0 con 0x7f84300cd2b0 2024-04-17T07:33:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.139+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84300c7970 con 0x7f84300cd2b0 2024-04-17T07:33:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.140+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430068b90 con 0x7f84300cd2b0 2024-04-17T07:33:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.141+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 74) ==== 91199+0+0 (secure 0 0 0) 0x7f8418007400 con 0x7f84300cd2b0 2024-04-17T07:33:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.142+0000 7f8426000640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 0x7f840c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.142+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 6081+0+0 (secure 0 0 0) 0x7f841808b590 con 0x7f84300cd2b0 2024-04-17T07:33:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.143+0000 7f842de00640 1 -- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 msgr2=0x7f840c06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2946630767 2024-04-17T07:33:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.143+0000 7f842de00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 0x7f840c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:33:55.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.147+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8418058ed0 con 0x7f84300cd2b0 2024-04-17T07:33:55.257 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.257+0000 7f13bb612040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:55.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.280+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 75) ==== 90751+0+0 (secure 0 0 0) 0x7f8418055540 con 0x7f84300cd2b0 2024-04-17T07:33:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.280+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 msgr2=0x7f840c06ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:33:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.280+0000 7f8426000640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 0x7f840c06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.336 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:55.337+0000 7f675740f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:33:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.354+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84300b5100 con 0x7f84300cd2b0 2024-04-17T07:33:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.355+0000 7f8426000640 1 -- 172.21.15.67:0/2553573535 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) ==== 93+0+10 (secure 0 0 0) 0x7f841805f930 con 0x7f84300cd2b0 2024-04-17T07:33:55.354 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:33:55.356 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.356+0000 7f1334e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.356 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.356+0000 7f1334e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.356 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.356+0000 7f1334e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.356 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.356+0000 7f1334e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.357 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.356+0000 7f1334e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 msgr2=0x7f84301f4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f841800c400 tx=0x7f841800c8c0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 shutdown_connections 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:6832/2946630767,v1:172.21.15.187:6833/2946630767] conn(0x7f840c06caa0 0x7f840c06ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8430160110 0x7f84300cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84300cd2b0 0x7f84301f4520 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 --2- 172.21.15.67:0/2553573535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f843009b250 0x7f84300c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 >> 172.21.15.67:0/2553573535 conn(0x7f84300b4470 msgr2=0x7f8430156c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 shutdown_connections 2024-04-17T07:33:55.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.357+0000 7f8434e00640 1 -- 172.21.15.67:0/2553573535 wait complete. 2024-04-17T07:33:55.358 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.357+0000 7f1338000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.359 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.357+0000 7f1338000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.359 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.357+0000 7f1338000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.359 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.357+0000 7f1338000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.359 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:33:55.357+0000 7f1338000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:33:55.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T07:33:55.403 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:55.404+0000 7f675740f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:33:55.549 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:33:55.549+0000 7f675740f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:33:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.599+0000 7f8f67e00640 1 Processor -- start 2024-04-17T07:33:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.599+0000 7f8f67e00640 1 -- start start 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.600+0000 7f8f67e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8f6809acd0 0x7f8f68161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.600+0000 7f8f67e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f6809b0b0 0x7f8f6815f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.600+0000 7f8f67e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f680cd510 con 0x7f8f680ce2d0 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.600+0000 7f8f67e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8f680cde40 con 0x7f8f6809b0b0 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.600+0000 7f8f67e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8f680cd7f0 con 0x7f8f6809acd0 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f5fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8f680ce2d0 0x7f8f6809b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44164/0 (socket says 172.21.15.67:44164) 2024-04-17T07:33:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f5fe00640 1 -- 172.21.15.67:0/2112383095 learned_addr learned my addr 172.21.15.67:0/2112383095 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f67400640 1 --2- 172.21.15.67:0/2112383095 >> v2:172.21.15.67:3300/0 conn(0x7f8f6809b0b0 0x7f8f6815f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f66a00640 1 --2- 172.21.15.67:0/2112383095 >> v2:172.21.15.187:3300/0 conn(0x7f8f6809acd0 0x7f8f68161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f67400640 1 -- 172.21.15.67:0/2112383095 >> v2:172.21.15.187:3300/0 conn(0x7f8f6809acd0 msgr2=0x7f8f68161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.601+0000 7f8f67400640 1 --2- 172.21.15.67:0/2112383095 >> v2:172.21.15.187:3300/0 conn(0x7f8f6809acd0 0x7f8f68161100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.602+0000 7f8f67400640 1 -- 172.21.15.67:0/2112383095 >> v1:172.21.15.67:6789/0 conn(0x7f8f680ce2d0 legacy=0x7f8f6809b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.602+0000 7f8f67400640 1 -- 172.21.15.67:0/2112383095 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f680cdad0 con 0x7f8f6809b0b0 2024-04-17T07:33:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.603+0000 7f8f67400640 1 --2- 172.21.15.67:0/2112383095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809b0b0 0x7f8f6815f990 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8f680730f0 tx=0x7f8f6002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4e4e2354a02818 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.603+0000 7f8f66000640 1 -- 172.21.15.67:0/2112383095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60039070 con 0x7f8f6809b0b0 2024-04-17T07:33:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.603+0000 7f8f66000640 1 -- 172.21.15.67:0/2112383095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6002cbe0 con 0x7f8f6809b0b0 2024-04-17T07:33:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.603+0000 7f8f66000640 1 -- 172.21.15.67:0/2112383095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60034440 con 0x7f8f6809b0b0 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 -- 172.21.15.67:0/2112383095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809b0b0 msgr2=0x7f8f6815f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 --2- 172.21.15.67:0/2112383095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809b0b0 0x7f8f6815f990 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f8f680730f0 tx=0x7f8f6002be30 comp rx=0 tx=0).stop 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 -- 172.21.15.67:0/2112383095 shutdown_connections 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 --2- 172.21.15.67:0/2112383095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809b0b0 0x7f8f6815f990 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 --2- 172.21.15.67:0/2112383095 >> v2:172.21.15.187:3300/0 conn(0x7f8f6809acd0 0x7f8f68161100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 -- 172.21.15.67:0/2112383095 >> 172.21.15.67:0/2112383095 conn(0x7f8f680b3ff0 msgr2=0x7f8f680b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 -- 172.21.15.67:0/2112383095 shutdown_connections 2024-04-17T07:33:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.604+0000 7f8f67e00640 1 -- 172.21.15.67:0/2112383095 wait complete. 2024-04-17T07:33:55.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.606+0000 7f8f67e00640 1 Processor -- start 2024-04-17T07:33:55.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.606+0000 7f8f67e00640 1 -- start start 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f6809b0b0 0x7f8f680c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f680ce2d0 0x7f8f680c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f68160050 con 0x7f8f6809acd0 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6815fed0 con 0x7f8f680ce2d0 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f67e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f681601d0 con 0x7f8f6809b0b0 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40584/0 (socket says 172.21.15.67:40584) 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 -- 172.21.15.67:0/3785448139 learned_addr learned my addr 172.21.15.67:0/3785448139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 -- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f6809b0b0 msgr2=0x7f8f680c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f6809b0b0 0x7f8f680c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 -- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f680ce2d0 msgr2=0x7f8f680c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.607+0000 7f8f66a00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f680ce2d0 0x7f8f680c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f66a00640 1 -- 172.21.15.67:0/3785448139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f680cc410 con 0x7f8f6809acd0 2024-04-17T07:33:55.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f66a00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800d5f0 tx=0x7f8f5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f580125f0 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f58019070 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f680c8e50 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f580129d0 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.608+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f681597f0 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.609+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f680755d0 con 0x7f8f6809acd0 2024-04-17T07:33:55.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.610+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 75) ==== 90751+0+0 (secure 0 0 0) 0x7f8f58012b70 con 0x7f8f6809acd0 2024-04-17T07:33:55.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.611+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f8f5808bde0 con 0x7f8f6809acd0 2024-04-17T07:33:55.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.615+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8f5805a270 con 0x7f8f6809acd0 2024-04-17T07:33:55.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.817+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8f680b4c50 con 0x7f8f6809acd0 2024-04-17T07:33:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.818+0000 7f8f5f400640 1 -- 172.21.15.67:0/3785448139 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) ==== 104+0+11 (secure 0 0 0) 0x7f8f58059b70 con 0x7f8f6809acd0 2024-04-17T07:33:55.817 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.820+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 msgr2=0x7f8f680cbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.820+0000 7f8f67e00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800d5f0 tx=0x7f8f5800dab0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 shutdown_connections 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f680ce2d0 0x7f8f680c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f6809b0b0 0x7f8f680c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 --2- 172.21.15.67:0/3785448139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6809acd0 0x7f8f680cbed0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 >> 172.21.15.67:0/3785448139 conn(0x7f8f680b3ff0 msgr2=0x7f8f680b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 shutdown_connections 2024-04-17T07:33:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:55.821+0000 7f8f67e00640 1 -- 172.21.15.67:0/3785448139 wait complete. 2024-04-17T07:33:55.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T07:33:56.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.108+0000 7f9533a00640 1 Processor -- start 2024-04-17T07:33:56.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.108+0000 7f9533a00640 1 -- start start 2024-04-17T07:33:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9533a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f952c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9533a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c06ded0 con 0x7f952c09aa60 2024-04-17T07:33:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9533a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c0d0990 con 0x7f952c0b37a0 2024-04-17T07:33:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9533a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f952c0d1280 con 0x7f952c162350 2024-04-17T07:33:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9531a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f952c09aa60 0x7f952c0d1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44170/0 (socket says 172.21.15.67:44170) 2024-04-17T07:33:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.110+0000 7f9531a00640 1 -- 172.21.15.67:0/2876988511 learned_addr learned my addr 172.21.15.67:0/2876988511 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f9531000640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f952c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568116370 0 0) 0x7f952c06ded0 con 0x7f952c09aa60 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9514003520 con 0x7f952c09aa60 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1419990229 0 0) 0x7f9514003520 con 0x7f952c09aa60 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 msgr2=0x7f952c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f952c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 >> v1:172.21.15.187:6789/0 conn(0x7f952c0b37a0 legacy=0x7f952c0ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c06d920 con 0x7f952c09aa60 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f95240019f0 con 0x7f952c09aa60 2024-04-17T07:33:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 >> v1:172.21.15.67:6789/0 conn(0x7f952c09aa60 legacy=0x7f952c0d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f95140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9514003f60 0x7f9514004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514004d40 0x7f95140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0d1280 con 0x7f9514003f60 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f951400baa0 con 0x7f9514004d40 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.111+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f951400bca0 con 0x7f952c162350 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9514003f60 0x7f9514004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f9531000640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f95140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 -- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 msgr2=0x7f95140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c162350 0x7f95140039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f9531a00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514004d40 0x7f95140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 -- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514004d40 msgr2=0x7f95140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514004d40 0x7f95140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.112+0000 7f952be00640 1 -- 172.21.15.67:0/2876988511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9514003520 con 0x7f9514003f60 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.113+0000 7f952be00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9514003f60 0x7f9514004800 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9518001750 tx=0x7f9514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dc03bab383d635 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.113+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518001370 con 0x7f9514003f60 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.113+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95180052d0 con 0x7f9514003f60 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.113+0000 7f952b400640 1 -- 172.21.15.67:0/2876988511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9518003830 con 0x7f9514003f60 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 -- 172.21.15.67:0/2876988511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9514003f60 msgr2=0x7f9514004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 --2- 172.21.15.67:0/2876988511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9514003f60 0x7f9514004800 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9518001750 tx=0x7f9514005830 comp rx=0 tx=0).stop 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9531a00640 1 -- 172.21.15.67:0/2876988511 reap_dead start 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 -- 172.21.15.67:0/2876988511 shutdown_connections 2024-04-17T07:33:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 -- 172.21.15.67:0/2876988511 >> 172.21.15.67:0/2876988511 conn(0x7f952c0b4240 msgr2=0x7f952c0b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 -- 172.21.15.67:0/2876988511 shutdown_connections 2024-04-17T07:33:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.114+0000 7f9533a00640 1 -- 172.21.15.67:0/2876988511 wait complete. 2024-04-17T07:33:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.116+0000 7f9533a00640 1 Processor -- start 2024-04-17T07:33:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.116+0000 7f9533a00640 1 -- start start 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514003f60 0x7f952c0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c160490 0x7f952c15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0d0f10 con 0x7f952c160910 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0d0c30 con 0x7f9514003f60 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9533a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f952c06d3e0 con 0x7f952c160490 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9531a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9531a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40596/0 (socket says 172.21.15.67:40596) 2024-04-17T07:33:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.117+0000 7f9531a00640 1 -- 172.21.15.67:0/583969450 learned_addr learned my addr 172.21.15.67:0/583969450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531000640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514003f60 0x7f952c0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f952be00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c160490 0x7f952c15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 -- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c160490 msgr2=0x7f952c15f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c160490 0x7f952c15f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 -- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514003f60 msgr2=0x7f952c0c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514003f60 0x7f952c0c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 -- 172.21.15.67:0/583969450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c150290 con 0x7f952c160910 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.118+0000 7f9531a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f9524007c40 tx=0x7f952400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.119+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952400bb80 con 0x7f952c160910 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.119+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f952400bd20 con 0x7f952c160910 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.119+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952c15e410 con 0x7f952c160910 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.119+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524007350 con 0x7f952c160910 2024-04-17T07:33:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.119+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f952c15e9d0 con 0x7f952c160910 2024-04-17T07:33:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.120+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f952c15f0f0 con 0x7f952c160910 2024-04-17T07:33:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.121+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 75) ==== 90751+0+0 (secure 0 0 0) 0x7f952401b060 con 0x7f952c160910 2024-04-17T07:33:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.122+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f952408b970 con 0x7f952c160910 2024-04-17T07:33:56.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.126+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f952408de20 con 0x7f952c160910 2024-04-17T07:33:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.285+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 76) ==== 90878+0+0 (secure 0 0 0) 0x7f9524055540 con 0x7f952c160910 2024-04-17T07:33:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.286+0000 7f9528c00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 0x7f950c0744a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.297+0000 7f9531000640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 0x7f950c0744a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.304+0000 7f9531000640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 0x7f950c0744a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9520009a10 tx=0x7f9520008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.340+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f952c09b200 con 0x7f952c160910 2024-04-17T07:33:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.341+0000 7f9528c00640 1 -- 172.21.15.67:0/583969450 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) ==== 100+0+11 (secure 0 0 0) 0x7f952c150290 con 0x7f952c160910 2024-04-17T07:33:56.340 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T07:33:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 msgr2=0x7f950c0744a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 0x7f950c0744a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9520009a10 tx=0x7f9520008040 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 msgr2=0x7f952c14d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f9524007c40 tx=0x7f952400b4a0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 shutdown_connections 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f950c0720b0 0x7f950c0744a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952c160910 0x7f952c14d3b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f952c160490 0x7f952c15f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 --2- 172.21.15.67:0/583969450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9514003f60 0x7f952c0c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 >> 172.21.15.67:0/583969450 conn(0x7f952c0b4240 msgr2=0x7f952c15bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 shutdown_connections 2024-04-17T07:33:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.344+0000 7f9533a00640 1 -- 172.21.15.67:0/583969450 wait complete. 2024-04-17T07:33:56.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:56.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.585+0000 7f642ce00640 1 Processor -- start 2024-04-17T07:33:56.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.585+0000 7f642ce00640 1 -- start start 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f642ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f642815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f642ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f642815f440 con 0x7f64280afb50 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f642ce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f642815f720 con 0x7f64280b4210 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f642ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f642815fa00 con 0x7f642815c5a0 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f6427e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64280afb50 0x7f64280ae090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44182/0 (socket says 172.21.15.67:44182) 2024-04-17T07:33:56.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.587+0000 7f6427e00640 1 -- 172.21.15.67:0/1978487137 learned_addr learned my addr 172.21.15.67:0/1978487137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6427400640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f642815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712696750 0 0) 0x7f642815f440 con 0x7f64280afb50 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f640c003520 con 0x7f64280afb50 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3882311117 0 0) 0x7f640c003520 con 0x7f64280afb50 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 msgr2=0x7f642815ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f642815ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 >> v1:172.21.15.187:6789/0 conn(0x7f64280b4210 legacy=0x7f64280af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642815fce0 con 0x7f64280afb50 2024-04-17T07:33:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f64200019f0 con 0x7f64280afb50 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 >> v1:172.21.15.67:6789/0 conn(0x7f64280afb50 legacy=0x7f64280ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c003a50 0x7f640c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c004800 0x7f640c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f640c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f642815fa00 con 0x7f640c003a50 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f640c005600 con 0x7f640c004800 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.588+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f640c00bf00 con 0x7f642815c5a0 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c003a50 0x7f640c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427e00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f640c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f641fe00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c004800 0x7f640c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 -- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 msgr2=0x7f640c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f642815c5a0 0x7f640c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 -- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c004800 msgr2=0x7f640c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c004800 0x7f640c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f6427400640 1 -- 172.21.15.67:0/1978487137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c003520 con 0x7f640c003a50 2024-04-17T07:33:56.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.589+0000 7f641fe00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c004800 0x7f640c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.590+0000 7f6427400640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c003a50 0x7f640c003e70 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f641800ba60 tx=0x7f640c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f387e544f38222 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418036020 con 0x7f640c003a50 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6418032630 con 0x7f640c003a50 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f6426a00640 1 -- 172.21.15.67:0/1978487137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641800b600 con 0x7f640c003a50 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 -- 172.21.15.67:0/1978487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c003a50 msgr2=0x7f640c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 --2- 172.21.15.67:0/1978487137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c003a50 0x7f640c003e70 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f641800ba60 tx=0x7f640c005830 comp rx=0 tx=0).stop 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f6427e00640 1 -- 172.21.15.67:0/1978487137 reap_dead start 2024-04-17T07:33:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 -- 172.21.15.67:0/1978487137 shutdown_connections 2024-04-17T07:33:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 -- 172.21.15.67:0/1978487137 >> 172.21.15.67:0/1978487137 conn(0x7f6428067d10 msgr2=0x7f64280b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 -- 172.21.15.67:0/1978487137 shutdown_connections 2024-04-17T07:33:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.591+0000 7f642ce00640 1 -- 172.21.15.67:0/1978487137 wait complete. 2024-04-17T07:33:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.593+0000 7f642ce00640 1 Processor -- start 2024-04-17T07:33:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.593+0000 7f642ce00640 1 -- start start 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 0x7f642814a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 0x7f642814add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f642806e7f0 con 0x7f642814ba20 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f642806e670 con 0x7f640c003a50 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.594+0000 7f642ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64280b4690 con 0x7f64280cf220 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40622/0 (socket says 172.21.15.67:40622) 2024-04-17T07:33:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 -- 172.21.15.67:0/119315544 learned_addr learned my addr 172.21.15.67:0/119315544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f6427e00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 0x7f642814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f6427400640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 0x7f642814a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 -- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 msgr2=0x7f642814add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 0x7f642814add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 -- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 msgr2=0x7f642814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 0x7f642814a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 -- 172.21.15.67:0/119315544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642814ab70 con 0x7f642814ba20 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f6427e00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 0x7f642814add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f6427400640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 0x7f642814a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.595+0000 7f641fe00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f641000d770 tx=0x7f641000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.596+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410005270 con 0x7f642814ba20 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.596+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6410017070 con 0x7f642814ba20 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.596+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641000b6b0 con 0x7f642814ba20 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.596+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6428149590 con 0x7f642814ba20 2024-04-17T07:33:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.596+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6428149b50 con 0x7f642814ba20 2024-04-17T07:33:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.597+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6428068a10 con 0x7f642814ba20 2024-04-17T07:33:56.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.598+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 76) ==== 90878+0+0 (secure 0 0 0) 0x7f641000b850 con 0x7f642814ba20 2024-04-17T07:33:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.599+0000 7f641f400640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 0x7f640406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.599+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f641008b8e0 con 0x7f642814ba20 2024-04-17T07:33:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.600+0000 7f6427400640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 0x7f640406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:56.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.601+0000 7f6427400640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 0x7f640406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f641800ba30 tx=0x7f6418030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:56.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.604+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6410059d50 con 0x7f642814ba20 2024-04-17T07:33:56.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.838+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64280aead0 con 0x7f642814ba20 2024-04-17T07:33:56.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.840+0000 7f641f400640 1 -- 172.21.15.67:0/119315544 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7f6410059650 con 0x7f642814ba20 2024-04-17T07:33:56.839 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:56.839 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:56.839 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 msgr2=0x7f640406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 0x7f640406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f641800ba30 tx=0x7f6418030000 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 msgr2=0x7f6428148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f641000d770 tx=0x7f641000dc30 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 shutdown_connections 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f640406c740 0x7f640406ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64280cf220 0x7f642814add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642814ba20 0x7f6428148d40 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 --2- 172.21.15.67:0/119315544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f640c003a50 0x7f642814a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 >> 172.21.15.67:0/119315544 conn(0x7f6428067d10 msgr2=0x7f64280af2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 shutdown_connections 2024-04-17T07:33:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:56.842+0000 7f642ce00640 1 -- 172.21.15.67:0/119315544 wait complete. 2024-04-17T07:33:56.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.073+0000 7f5ffa400640 1 Processor -- start 2024-04-17T07:33:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.073+0000 7f5ffa400640 1 -- start start 2024-04-17T07:33:57.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ffa400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff40d0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ffa400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ff40ce8a0 0x7f5ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ffa400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ff40d1060 con 0x7f5ff409aa60 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ffa400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5ff40cdc70 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ffa400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5ff40d1340 con 0x7f5ff40ce4c0 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ff3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ff40ce8a0 0x7f5ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ff9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ff409aa60 0x7f5ff415e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44184/0 (socket says 172.21.15.67:44184) 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.075+0000 7f5ff9a00640 1 -- 172.21.15.67:0/3715505832 learned_addr learned my addr 172.21.15.67:0/3715505832 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff9000640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff40d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff3e00640 1 -- 172.21.15.67:0/3715505832 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 msgr2=0x7f5ff40d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff3e00640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff40d0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff3e00640 1 -- 172.21.15.67:0/3715505832 >> v1:172.21.15.67:6789/0 conn(0x7f5ff409aa60 legacy=0x7f5ff415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff3e00640 1 -- 172.21.15.67:0/3715505832 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff40d1620 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.076+0000 7f5ff3400640 1 -- 172.21.15.67:0/3715505832 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1409865032 0 0) 0x7f5ff40d1060 con 0x7f5ff409aa60 2024-04-17T07:33:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.077+0000 7f5ff3e00640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff406d990 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00097a0 tx=0x7f5fe002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe518f47bfe919d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.077+0000 7f5ff3400640 1 -- 172.21.15.67:0/3715505832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0035020 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.077+0000 7f5ff3400640 1 -- 172.21.15.67:0/3715505832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fe0031830 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.077+0000 7f5ff3400640 1 -- 172.21.15.67:0/3715505832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0031c00 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 -- 172.21.15.67:0/3715505832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 msgr2=0x7f5ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff406d990 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00097a0 tx=0x7f5fe002b8d0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 -- 172.21.15.67:0/3715505832 shutdown_connections 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff406d990 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3715505832 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff40d0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 -- 172.21.15.67:0/3715505832 >> 172.21.15.67:0/3715505832 conn(0x7f5ff40b4100 msgr2=0x7f5ff40b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 -- 172.21.15.67:0/3715505832 shutdown_connections 2024-04-17T07:33:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.078+0000 7f5ffa400640 1 -- 172.21.15.67:0/3715505832 wait complete. 2024-04-17T07:33:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.080+0000 7f5ffa400640 1 Processor -- start 2024-04-17T07:33:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.080+0000 7f5ffa400640 1 -- start start 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ff409aa60 0x7f5ff414b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff41e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff40d2100 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5ff406ded0 con 0x7f5ff409aa60 2024-04-17T07:33:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ffa400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5ff40d2280 con 0x7f5ff40ce4c0 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ff9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ff9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40646/0 (socket says 172.21.15.67:40646) 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.081+0000 7f5ff9a00640 1 -- 172.21.15.67:0/3376648277 learned_addr learned my addr 172.21.15.67:0/3376648277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff3e00640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff41e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9000640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ff409aa60 0x7f5ff414b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 -- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 msgr2=0x7f5ff41e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff41e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 -- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ff409aa60 msgr2=0x7f5ff414b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ff409aa60 0x7f5ff414b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 -- 172.21.15.67:0/3376648277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ff41e7ed0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.082+0000 7f5ff9a00640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00c780 tx=0x7f5fec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.083+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec018070 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.083+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fec00a640 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.083+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00aa30 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.083+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ff4148b80 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.083+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ff4149140 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.084+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ff40755d0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.085+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 76) ==== 90878+0+0 (secure 0 0 0) 0x7f5fec00abd0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.086+0000 7f5ff0c00640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 0x7f5fd406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.086+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f5fec08b120 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.087+0000 7f5ff9000640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 0x7f5fd406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.088+0000 7f5ff9000640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 0x7f5fd406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8006fd0 tx=0x7f5fe8008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.090+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5fec0594c0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.299+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7f5fec055430 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.302+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ff40b4fa0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.304+0000 7f5ff0c00640 1 -- 172.21.15.67:0/3376648277 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17702 (secure 0 0 0) 0x7f5fec07a0b0 con 0x7f5ff40ce8a0 2024-04-17T07:33:57.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:57.304 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":144,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:55.262564+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4163913453":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1827135944":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6833/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6832/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1208304941":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/600549912":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/231129067":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/1184199060":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2363868131":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2637392297":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6833/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/156923885":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2619020255":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6832/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/3318285601":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/4219847990":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2597865993":"2024-04-18T07:33:41.840202+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.306+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 msgr2=0x7f5fd406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.306+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 0x7f5fd406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8006fd0 tx=0x7f5fe8008040 comp rx=0 tx=0).stop 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.306+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 msgr2=0x7f5ff40708c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.306+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00c780 tx=0x7f5fec00cc40 comp rx=0 tx=0).stop 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 shutdown_connections 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f5fd406c740 0x7f5fd406ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff40ce8a0 0x7f5ff40708c0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ff40ce4c0 0x7f5ff41e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 --2- 172.21.15.67:0/3376648277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ff409aa60 0x7f5ff414b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 >> 172.21.15.67:0/3376648277 conn(0x7f5ff40b4100 msgr2=0x7f5ff4161b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 shutdown_connections 2024-04-17T07:33:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.307+0000 7f5ffa400640 1 -- 172.21.15.67:0/3376648277 wait complete. 2024-04-17T07:33:57.325 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T07:33:57.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.541+0000 7fb833200640 1 Processor -- start 2024-04-17T07:33:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.542+0000 7fb833200640 1 -- start start 2024-04-17T07:33:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb833200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb8340cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb833200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8340cf910 0x7fb8340b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb833200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8340d2d30 con 0x7fb83415ec60 2024-04-17T07:33:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb833200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb8340d49c0 con 0x7fb8340cf910 2024-04-17T07:33:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb833200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8340d3660 con 0x7fb8340b6d00 2024-04-17T07:33:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb831e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb83415ec60 0x7fb8340b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44200/0 (socket says 172.21.15.67:44200) 2024-04-17T07:33:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb831e00640 1 -- 172.21.15.67:0/1764792211 learned_addr learned my addr 172.21.15.67:0/1764792211 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.543+0000 7fb832800640 1 --2- 172.21.15.67:0/1764792211 >> v2:172.21.15.67:3300/0 conn(0x7fb8340cf910 0x7fb8340b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb831400640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb8340cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb832800640 1 -- 172.21.15.67:0/1764792211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 msgr2=0x7fb8340cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb832800640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb8340cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb832800640 1 -- 172.21.15.67:0/1764792211 >> v1:172.21.15.67:6789/0 conn(0x7fb83415ec60 legacy=0x7fb8340b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb832800640 1 -- 172.21.15.67:0/1764792211 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8340d3010 con 0x7fb8340cf910 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.544+0000 7fb830a00640 1 -- 172.21.15.67:0/1764792211 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640350445 0 0) 0x7fb8340d2d30 con 0x7fb83415ec60 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.545+0000 7fb832800640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8340b7b70 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423939536ce15584 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.545+0000 7fb830a00640 1 -- 172.21.15.67:0/1764792211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c039070 con 0x7fb8340cf910 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.545+0000 7fb830a00640 1 -- 172.21.15.67:0/1764792211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82c02cc50 con 0x7fb8340cf910 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.545+0000 7fb830a00640 1 -- 172.21.15.67:0/1764792211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c034440 con 0x7fb8340cf910 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 -- 172.21.15.67:0/1764792211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 msgr2=0x7fb8340b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8340b7b70 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009d90 tx=0x7fb82c02bea0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 -- 172.21.15.67:0/1764792211 shutdown_connections 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8340b7b70 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 --2- 172.21.15.67:0/1764792211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb8340cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 -- 172.21.15.67:0/1764792211 >> 172.21.15.67:0/1764792211 conn(0x7fb8340d15f0 msgr2=0x7fb8340d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.546+0000 7fb833200640 1 -- 172.21.15.67:0/1764792211 shutdown_connections 2024-04-17T07:33:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.547+0000 7fb833200640 1 -- 172.21.15.67:0/1764792211 wait complete. 2024-04-17T07:33:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.549+0000 7fb833200640 1 Processor -- start 2024-04-17T07:33:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.549+0000 7fb833200640 1 -- start start 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb834147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb83415ec60 0x7fb834149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb83415f040 con 0x7fb8340cf910 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb8340cfdc0 con 0x7fb83415ec60 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.550+0000 7fb833200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8340d00c0 con 0x7fb8340b6d00 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40664/0 (socket says 172.21.15.67:40664) 2024-04-17T07:33:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 -- 172.21.15.67:0/1430885813 learned_addr learned my addr 172.21.15.67:0/1430885813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb832800640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb83415ec60 0x7fb834149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831e00640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb834147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 -- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 msgr2=0x7fb834147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb834147960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 -- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb83415ec60 msgr2=0x7fb834149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb83415ec60 0x7fb834149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.551+0000 7fb831400640 1 -- 172.21.15.67:0/1430885813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb834161070 con 0x7fb8340cf910 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb831400640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb82000c7f0 tx=0x7fb82000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820015070 con 0x7fb8340cf910 2024-04-17T07:33:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb820004240 con 0x7fb8340cf910 2024-04-17T07:33:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820007620 con 0x7fb8340cf910 2024-04-17T07:33:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb834159600 con 0x7fb8340cf910 2024-04-17T07:33:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.552+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb834159bc0 con 0x7fb8340cf910 2024-04-17T07:33:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.553+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb834068a10 con 0x7fb8340cf910 2024-04-17T07:33:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.554+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7fb8200079d0 con 0x7fb8340cf910 2024-04-17T07:33:57.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.555+0000 7fb82a000640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 0x7fb81406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:57.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.555+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7fb82008c070 con 0x7fb8340cf910 2024-04-17T07:33:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.556+0000 7fb831e00640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 0x7fb81406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:57.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.557+0000 7fb831e00640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 0x7fb81406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb824000f60 tx=0x7fb824007c00 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:57.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.560+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb82005a030 con 0x7fb8340cf910 2024-04-17T07:33:57.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.805+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8340ba0b0 con 0x7fb8340cf910 2024-04-17T07:33:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.806+0000 7fb82a000640 1 -- 172.21.15.67:0/1430885813 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7fb820059930 con 0x7fb8340cf910 2024-04-17T07:33:57.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:57.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 msgr2=0x7fb81406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 0x7fb81406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb824000f60 tx=0x7fb824007c00 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 msgr2=0x7fb8341499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb82000c7f0 tx=0x7fb82000ccb0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 shutdown_connections 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb81406c980 0x7fb81406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb83415ec60 0x7fb834149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8340cf910 0x7fb8341499c0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 --2- 172.21.15.67:0/1430885813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8340b6d00 0x7fb834147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 >> 172.21.15.67:0/1430885813 conn(0x7fb8340d15f0 msgr2=0x7fb8340b92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.809+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 shutdown_connections 2024-04-17T07:33:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:57.810+0000 7fb833200640 1 -- 172.21.15.67:0/1430885813 wait complete. 2024-04-17T07:33:57.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:33:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.037+0000 7f81ce200640 1 Processor -- start 2024-04-17T07:33:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.037+0000 7f81ce200640 1 -- start start 2024-04-17T07:33:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81ce200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f81c8161e70 0x7f81c80c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81ce200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 0x7f81c8162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81ce200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c80cc500 con 0x7f81c80ce4a0 2024-04-17T07:33:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81ce200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f81c80c71e0 con 0x7f81c8161e70 2024-04-17T07:33:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81ce200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81c80cc7a0 con 0x7f81c80cd5a0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81c6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81c80ce4a0 0x7f81c80cce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44212/0 (socket says 172.21.15.67:44212) 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81c6a00640 1 -- 172.21.15.67:0/2975614226 learned_addr learned my addr 172.21.15.67:0/2975614226 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.039+0000 7f81c7400640 1 --2- 172.21.15.67:0/2975614226 >> v2:172.21.15.187:3300/0 conn(0x7f81c8161e70 0x7f81c80c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c7e00640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 0x7f81c8162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872683082 0 0) 0x7f81c80cc500 con 0x7f81c80ce4a0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81ac003520 con 0x7f81c80ce4a0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1975091706 0 0) 0x7f81ac003520 con 0x7f81c80ce4a0 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 msgr2=0x7f81c8162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 0x7f81c8162fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 >> v2:172.21.15.187:3300/0 conn(0x7f81c8161e70 msgr2=0x7f81c80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 --2- 172.21.15.67:0/2975614226 >> v2:172.21.15.187:3300/0 conn(0x7f81c8161e70 0x7f81c80c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c80cca80 con 0x7f81c80ce4a0 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f81b4001670 con 0x7f81c80ce4a0 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 >> v1:172.21.15.67:6789/0 conn(0x7f81c80ce4a0 legacy=0x7f81c80cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac0039b0 0x7f81ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 0x7f81ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81ac004d40 0x7f81ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80c71e0 con 0x7f81ac0039b0 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80cc7a0 con 0x7f81ac004d40 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.040+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81ac00b9c0 con 0x7f81c80cd5a0 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac0039b0 0x7f81ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 -- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 msgr2=0x7f81ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81c80cd5a0 0x7f81ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 -- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81ac004d40 msgr2=0x7f81ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7e00640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81ac004d40 0x7f81ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81ac004d40 0x7f81ac0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.041+0000 7f81c7400640 1 -- 172.21.15.67:0/2975614226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ac003520 con 0x7f81ac0039b0 2024-04-17T07:33:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.042+0000 7f81c7400640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac0039b0 0x7f81ac003dd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009ac0 tx=0x7f81ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed7dd2f209601fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.042+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8039070 con 0x7f81ac0039b0 2024-04-17T07:33:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.042+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b802c710 con 0x7f81ac0039b0 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.042+0000 7f81c6000640 1 -- 172.21.15.67:0/2975614226 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8009690 con 0x7f81ac0039b0 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 -- 172.21.15.67:0/2975614226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac0039b0 msgr2=0x7f81ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 --2- 172.21.15.67:0/2975614226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81ac0039b0 0x7f81ac003dd0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f81b8009ac0 tx=0x7f81ac005830 comp rx=0 tx=0).stop 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81c7e00640 1 -- 172.21.15.67:0/2975614226 reap_dead start 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 -- 172.21.15.67:0/2975614226 shutdown_connections 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 -- 172.21.15.67:0/2975614226 >> 172.21.15.67:0/2975614226 conn(0x7f81c806bf50 msgr2=0x7f81c80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 -- 172.21.15.67:0/2975614226 shutdown_connections 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.043+0000 7f81ce200640 1 -- 172.21.15.67:0/2975614226 wait complete. 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.045+0000 7f81ce200640 1 Processor -- start 2024-04-17T07:33:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.045+0000 7f81ce200640 1 -- start start 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81ac0039b0 0x7f81c8155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81c80af370 0x7f81c80af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81c8163160 con 0x7f81c806ed80 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81c8162fe0 con 0x7f81c80af370 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81ce200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81c81632e0 con 0x7f81ac0039b0 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40682/0 (socket says 172.21.15.67:40682) 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.046+0000 7f81c7e00640 1 -- 172.21.15.67:0/2559947506 learned_addr learned my addr 172.21.15.67:0/2559947506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c6a00640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81c80af370 0x7f81c80af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 -- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81ac0039b0 msgr2=0x7f81c8155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81ac0039b0 0x7f81c8155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 -- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81c80af370 msgr2=0x7f81c80af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81c80af370 0x7f81c80af7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 -- 172.21.15.67:0/2559947506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c80adb20 con 0x7f81c806ed80 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81c7e00640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f81c80caa30 tx=0x7f81c000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c000fe90 con 0x7f81c806ed80 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c0003990 con 0x7f81c806ed80 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.047+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0003d60 con 0x7f81c806ed80 2024-04-17T07:33:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.048+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c80ade00 con 0x7f81c806ed80 2024-04-17T07:33:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.048+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c80ae3c0 con 0x7f81c806ed80 2024-04-17T07:33:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.049+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c8068b90 con 0x7f81c806ed80 2024-04-17T07:33:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.050+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7f81c0005d10 con 0x7f81c806ed80 2024-04-17T07:33:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.051+0000 7f81bf400640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 0x7f81a806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.051+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f81c008c320 con 0x7f81c806ed80 2024-04-17T07:33:58.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.052+0000 7f81c7400640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 0x7f81a806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.053+0000 7f81c7400640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 0x7f81a806f010 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81b8036000 tx=0x7f81b802e000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.056+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f81c0012070 con 0x7f81c806ed80 2024-04-17T07:33:58.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.301+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81c815bfd0 con 0x7f81c806ed80 2024-04-17T07:33:58.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.302+0000 7f81bf400640 1 -- 172.21.15.67:0/2559947506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5201 (secure 0 0 0) 0x7f81c005a2e0 con 0x7f81c806ed80 2024-04-17T07:33:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:58.301 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:33:26.988125+0000","modified":"2024-04-17T07:33:30.097371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8397},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8397":{"gid":8397,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/996743914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":996743914},{"type":"v1","addr":"172.21.15.67:6835","nonce":996743914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8397,"qdb_cluster":[8397]},"id":8}]} 2024-04-17T07:33:58.301 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 68 2024-04-17T07:33:58.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.304+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 msgr2=0x7f81a806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.304+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 0x7f81a806f010 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81b8036000 tx=0x7f81b802e000 comp rx=0 tx=0).stop 2024-04-17T07:33:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.304+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 msgr2=0x7f81c806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.304+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f81c80caa30 tx=0x7f81c000f4a0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 shutdown_connections 2024-04-17T07:33:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f81a806cb50 0x7f81a806f010 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81c806ed80 0x7f81c806f200 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81c80af370 0x7f81c80af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 --2- 172.21.15.67:0/2559947506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81ac0039b0 0x7f81c8155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 >> 172.21.15.67:0/2559947506 conn(0x7f81c806bf50 msgr2=0x7f81c80cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 shutdown_connections 2024-04-17T07:33:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.305+0000 7f81ce200640 1 -- 172.21.15.67:0/2559947506 wait complete. 2024-04-17T07:33:58.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:33:58.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.547+0000 7f6838a00640 1 Processor -- start 2024-04-17T07:33:58.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.547+0000 7f6838a00640 1 -- start start 2024-04-17T07:33:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6838a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f683415c5a0 0x7f683415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6838a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68340b4210 0x7f68340af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6838a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f683415f440 con 0x7f68340b46c0 2024-04-17T07:33:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6838a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f68340af0a0 con 0x7f683415c5a0 2024-04-17T07:33:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6838a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f683415f720 con 0x7f68340b4210 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6833400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f683415c5a0 0x7f683415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6833400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f683415c5a0 0x7f683415ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40690/0 (socket says 172.21.15.67:40690) 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.549+0000 7f6833400640 1 -- 172.21.15.67:0/3635558307 learned_addr learned my addr 172.21.15.67:0/3635558307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.550+0000 7f6833400640 1 -- 172.21.15.67:0/3635558307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68340b4210 msgr2=0x7f68340af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.550+0000 7f6833400640 1 --2- 172.21.15.67:0/3635558307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68340b4210 0x7f68340af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.550+0000 7f6833400640 1 -- 172.21.15.67:0/3635558307 >> v1:172.21.15.187:6789/0 conn(0x7f68340b46c0 legacy=0x7f6834071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.550+0000 7f6833400640 1 -- 172.21.15.67:0/3635558307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683415fa00 con 0x7f683415c5a0 2024-04-17T07:33:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.551+0000 7f6833400640 1 --2- 172.21.15.67:0/3635558307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f683415c5a0 0x7f683415ee60 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f682402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1a15af0ca580f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.551+0000 7f6832000640 1 -- 172.21.15.67:0/3635558307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824038070 con 0x7f683415c5a0 2024-04-17T07:33:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.551+0000 7f6832000640 1 -- 172.21.15.67:0/3635558307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6824031520 con 0x7f683415c5a0 2024-04-17T07:33:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.551+0000 7f6832000640 1 -- 172.21.15.67:0/3635558307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68240318f0 con 0x7f683415c5a0 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 -- 172.21.15.67:0/3635558307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f683415c5a0 msgr2=0x7f683415ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 --2- 172.21.15.67:0/3635558307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f683415c5a0 0x7f683415ee60 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f682402b7c0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 -- 172.21.15.67:0/3635558307 shutdown_connections 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 --2- 172.21.15.67:0/3635558307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68340b4210 0x7f68340af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 --2- 172.21.15.67:0/3635558307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f683415c5a0 0x7f683415ee60 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 -- 172.21.15.67:0/3635558307 >> 172.21.15.67:0/3635558307 conn(0x7f6834067d10 msgr2=0x7f68340b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 -- 172.21.15.67:0/3635558307 shutdown_connections 2024-04-17T07:33:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.552+0000 7f6838a00640 1 -- 172.21.15.67:0/3635558307 wait complete. 2024-04-17T07:33:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.554+0000 7f6838a00640 1 Processor -- start 2024-04-17T07:33:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.554+0000 7f6838a00640 1 -- start start 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683415c5a0 0x7f6834145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68341488e0 0x7f68340b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f68340adc30 con 0x7f68340b4210 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f68340adab0 con 0x7f683415c5a0 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6838a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f683406e670 con 0x7f68341488e0 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6833400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6833400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40698/0 (socket says 172.21.15.67:40698) 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.555+0000 7f6833400640 1 -- 172.21.15.67:0/2463114771 learned_addr learned my addr 172.21.15.67:0/2463114771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 -- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68341488e0 msgr2=0x7f68340b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68341488e0 0x7f68340b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 -- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683415c5a0 msgr2=0x7f6834145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6832a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683415c5a0 0x7f6834145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683415c5a0 0x7f6834145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 -- 172.21.15.67:0/2463114771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68341464c0 con 0x7f68340b4210 2024-04-17T07:33:58.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6833400640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f68240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824043070 con 0x7f68340b4210 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682403f870 con 0x7f68340b4210 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824038040 con 0x7f68340b4210 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.556+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68341467a0 con 0x7f68340b4210 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.557+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6834146d60 con 0x7f68340b4210 2024-04-17T07:33:58.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.558+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6834068a10 con 0x7f68340b4210 2024-04-17T07:33:58.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.559+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7f6824046070 con 0x7f68340b4210 2024-04-17T07:33:58.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.560+0000 7f682b400640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 0x7f681406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.560+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f68240adb80 con 0x7f68340b4210 2024-04-17T07:33:58.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.561+0000 7f6832a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 0x7f681406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:58.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.562+0000 7f6832a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 0x7f681406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68341e7690 tx=0x7f6820008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:58.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.565+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f682407bbd0 con 0x7f68340b4210 2024-04-17T07:33:58.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.763+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68340b31d0 con 0x7f68340b4210 2024-04-17T07:33:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.766+0000 7f682b400640 1 -- 172.21.15.67:0/2463114771 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) ==== 74+0+17702 (secure 0 0 0) 0x7f682407b4d0 con 0x7f68340b4210 2024-04-17T07:33:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:33:58.766 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":144,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:33:55.262564+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:33:24.887328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-17T07:33:25.960254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4163913453":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1827135944":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6833/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6832/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1208304941":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/600549912":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/231129067":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/1184199060":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2363868131":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2637392297":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6833/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/156923885":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2619020255":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6832/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/3318285601":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/4219847990":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2597865993":"2024-04-18T07:33:41.840202+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:33:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 msgr2=0x7f681406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 0x7f681406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68341e7690 tx=0x7f6820008040 comp rx=0 tx=0).stop 2024-04-17T07:33:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 msgr2=0x7f6834147560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f68240098a0 tx=0x7f68240046c0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 shutdown_connections 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f681406c980 0x7f681406ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68341488e0 0x7f68340b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683415c5a0 0x7f6834145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 --2- 172.21.15.67:0/2463114771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68340b4210 0x7f6834147560 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 >> 172.21.15.67:0/2463114771 conn(0x7f6834067d10 msgr2=0x7f68340b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 shutdown_connections 2024-04-17T07:33:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.769+0000 7f6838a00640 1 -- 172.21.15.67:0/2463114771 wait complete. 2024-04-17T07:33:58.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T07:33:58.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.997+0000 7f453d800640 1 Processor -- start 2024-04-17T07:33:58.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.998+0000 7f453d800640 1 -- start start 2024-04-17T07:33:58.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.999+0000 7f453d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45380cbc00 0x7f45380cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.999+0000 7f453d800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 0x7f45380ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.999+0000 7f453d800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f453815d060 con 0x7f45380b63c0 2024-04-17T07:33:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.999+0000 7f453d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f45380cfe90 con 0x7f45380cbc00 2024-04-17T07:33:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:58.999+0000 7f453d800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f453815dc70 con 0x7f4538072570 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45380cbc00 0x7f45380cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45380cbc00 0x7f45380cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40704/0 (socket says 172.21.15.67:40704) 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 -- 172.21.15.67:0/70231253 learned_addr learned my addr 172.21.15.67:0/70231253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f453ce00640 1 --2- 172.21.15.67:0/70231253 >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 0x7f45380ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 -- 172.21.15.67:0/70231253 >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 msgr2=0x7f45380ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 --2- 172.21.15.67:0/70231253 >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 0x7f45380ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 -- 172.21.15.67:0/70231253 >> v1:172.21.15.187:6789/0 conn(0x7f45380b63c0 legacy=0x7f4538071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f4537e00640 1 -- 172.21.15.67:0/70231253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453815d340 con 0x7f45380cbc00 2024-04-17T07:33:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.000+0000 7f453ce00640 1 --2- 172.21.15.67:0/70231253 >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 0x7f45380ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.002+0000 7f4537e00640 1 --2- 172.21.15.67:0/70231253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f45380cf950 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f4528009870 tx=0x7f452802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5cd4f407d17ca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f4536a00640 1 -- 172.21.15.67:0/70231253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4528035020 con 0x7f45380cbc00 2024-04-17T07:33:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f4536a00640 1 -- 172.21.15.67:0/70231253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f452802d040 con 0x7f45380cbc00 2024-04-17T07:33:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f4536a00640 1 -- 172.21.15.67:0/70231253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4528031a80 con 0x7f45380cbc00 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 -- 172.21.15.67:0/70231253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 msgr2=0x7f45380cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 --2- 172.21.15.67:0/70231253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f45380cf950 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f4528009870 tx=0x7f452802b900 comp rx=0 tx=0).stop 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 -- 172.21.15.67:0/70231253 shutdown_connections 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 --2- 172.21.15.67:0/70231253 >> v2:172.21.15.187:3300/0 conn(0x7f4538072570 0x7f45380ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 --2- 172.21.15.67:0/70231253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f45380cf950 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 -- 172.21.15.67:0/70231253 >> 172.21.15.67:0/70231253 conn(0x7f453806bef0 msgr2=0x7f45380b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.003+0000 7f453d800640 1 -- 172.21.15.67:0/70231253 shutdown_connections 2024-04-17T07:33:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.004+0000 7f453d800640 1 -- 172.21.15.67:0/70231253 wait complete. 2024-04-17T07:33:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.005+0000 7f453d800640 1 Processor -- start 2024-04-17T07:33:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.005+0000 7f453d800640 1 -- start start 2024-04-17T07:33:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 0x7f45380c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45380b63c0 0x7f453806df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45380d02d0 con 0x7f45380cbc00 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f453815d900 con 0x7f45380b63c0 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45380cbfe0 con 0x7f4538072570 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40718/0 (socket says 172.21.15.67:40718) 2024-04-17T07:33:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.006+0000 7f453ce00640 1 -- 172.21.15.67:0/3711617776 learned_addr learned my addr 172.21.15.67:0/3711617776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f4537e00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 0x7f45380c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 -- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 msgr2=0x7f45380c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f4537400640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45380b63c0 0x7f453806df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 0x7f45380c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 -- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45380b63c0 msgr2=0x7f453806df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45380b63c0 0x7f453806df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 -- 172.21.15.67:0/3711617776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453806eb80 con 0x7f45380cbc00 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f4537e00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 0x7f45380c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f453ce00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f453000ce80 tx=0x7f453000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.007+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453001a070 con 0x7f45380cbc00 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.008+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f453000d880 con 0x7f45380cbc00 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.008+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453000dc70 con 0x7f45380cbc00 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.008+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453806ee60 con 0x7f45380cbc00 2024-04-17T07:33:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.008+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453806f420 con 0x7f45380cbc00 2024-04-17T07:33:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.009+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45380692d0 con 0x7f45380cbc00 2024-04-17T07:33:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.010+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7f453001d070 con 0x7f45380cbc00 2024-04-17T07:33:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.011+0000 7f452fe00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 0x7f451806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.011+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 6468+0+0 (secure 0 0 0) 0x7f453008c660 con 0x7f45380cbc00 2024-04-17T07:33:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.011+0000 7f4537e00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 0x7f451806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.012+0000 7f4537e00640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 0x7f451806ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4528036000 tx=0x7f4528008030 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.015+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f45380692d0 con 0x7f45380cbc00 2024-04-17T07:33:59.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.255+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f45380b3020 con 0x7f45380cbc00 2024-04-17T07:33:59.255 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T07:33:59.256+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T07:33:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.260+0000 7f452fe00640 1 -- 172.21.15.67:0/3711617776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7f453005a0e0 con 0x7f45380cbc00 2024-04-17T07:33:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T07:33:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.262+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 msgr2=0x7f451806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.262+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 0x7f451806ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4528036000 tx=0x7f4528008030 comp rx=0 tx=0).stop 2024-04-17T07:33:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.262+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 msgr2=0x7f453806e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.262+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f453000ce80 tx=0x7f453000a710 comp rx=0 tx=0).stop 2024-04-17T07:33:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 shutdown_connections 2024-04-17T07:33:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f451806c980 0x7f451806ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45380cbc00 0x7f453806e4b0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45380b63c0 0x7f453806df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 --2- 172.21.15.67:0/3711617776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4538072570 0x7f45380c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 >> 172.21.15.67:0/3711617776 conn(0x7f453806bef0 msgr2=0x7f4538074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 shutdown_connections 2024-04-17T07:33:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.263+0000 7f453d800640 1 -- 172.21.15.67:0/3711617776 wait complete. 2024-04-17T07:33:59.280 INFO:tasks.ceph.mds.a.smithi067.stderr: -9250> 2024-04-17T07:33:31.920+0000 7f9775000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:33:59.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T07:33:59.343 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:33:59.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.473+0000 7fae4e600640 1 Processor -- start 2024-04-17T07:33:59.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.474+0000 7fae4e600640 1 -- start start 2024-04-17T07:33:59.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.475+0000 7fae4e600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae480d17a0 0x7fae480cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.475+0000 7fae4e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae480d4100 con 0x7fae480cfa10 2024-04-17T07:33:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.475+0000 7fae4e600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae48163860 con 0x7fae4815eca0 2024-04-17T07:33:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.475+0000 7fae4e600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fae480d03b0 con 0x7fae480d17a0 2024-04-17T07:33:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.476+0000 7fae46a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae480cfa10 0x7fae480d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44218/0 (socket says 172.21.15.67:44218) 2024-04-17T07:33:59.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.476+0000 7fae46a00640 1 -- 172.21.15.67:0/923566185 learned_addr learned my addr 172.21.15.67:0/923566185 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.476+0000 7fae47e00640 1 --2- 172.21.15.67:0/923566185 >> v2:172.21.15.187:3300/0 conn(0x7fae480d17a0 0x7fae480cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.476+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1734170282 0 0) 0x7fae480d4100 con 0x7fae480cfa10 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.476+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae28003520 con 0x7fae480cfa10 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1942966300 0 0) 0x7fae28003520 con 0x7fae480cfa10 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 >> v2:172.21.15.187:3300/0 conn(0x7fae480d17a0 msgr2=0x7fae480cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 --2- 172.21.15.67:0/923566185 >> v2:172.21.15.187:3300/0 conn(0x7fae480d17a0 0x7fae480cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 >> v1:172.21.15.187:6789/0 conn(0x7fae4815eca0 legacy=0x7fae480d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae480d1d40 con 0x7fae480cfa10 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fae300019f0 con 0x7fae480cfa10 2024-04-17T07:33:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 >> v1:172.21.15.67:6789/0 conn(0x7fae480cfa10 legacy=0x7fae480d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 0x7fae28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28004800 0x7fae28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae280051c0 0x7fae2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae2800c090 con 0x7fae28003a00 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae2800c240 con 0x7fae28004800 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae2800c440 con 0x7fae280051c0 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae47400640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 0x7fae28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae47400640 1 -- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae280051c0 msgr2=0x7fae2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae46a00640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28004800 0x7fae28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae47400640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae280051c0 0x7fae2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.477+0000 7fae47400640 1 -- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28004800 msgr2=0x7fae28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.478+0000 7fae47400640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28004800 0x7fae28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.478+0000 7fae47400640 1 -- 172.21.15.67:0/923566185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae28003520 con 0x7fae28003a00 2024-04-17T07:33:59.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.478+0000 7fae47e00640 1 -- 172.21.15.67:0/923566185 reap_dead start 2024-04-17T07:33:59.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.478+0000 7fae46a00640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28004800 0x7fae28004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae47400640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 0x7fae28003e20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fae38001750 tx=0x7fae28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05b9e3cfc7808c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38001370 con 0x7fae28003a00 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae380052d0 con 0x7fae28003a00 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae46000640 1 -- 172.21.15.67:0/923566185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38003830 con 0x7fae28003a00 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae4e600640 1 -- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 msgr2=0x7fae28003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.479+0000 7fae4e600640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 0x7fae28003e20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fae38001750 tx=0x7fae28005830 comp rx=0 tx=0).stop 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.480+0000 7fae4e600640 1 -- 172.21.15.67:0/923566185 shutdown_connections 2024-04-17T07:33:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.480+0000 7fae4e600640 1 --2- 172.21.15.67:0/923566185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae28003a00 0x7fae28003e20 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.480+0000 7fae4e600640 1 -- 172.21.15.67:0/923566185 >> 172.21.15.67:0/923566185 conn(0x7fae48067600 msgr2=0x7fae480679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.480+0000 7fae4e600640 1 -- 172.21.15.67:0/923566185 shutdown_connections 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.480+0000 7fae4e600640 1 -- 172.21.15.67:0/923566185 wait complete. 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.482+0000 7fae4e600640 1 Processor -- start 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.482+0000 7fae4e600640 1 -- start start 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 0x7fae48149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 0x7fae480b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae480d31f0 con 0x7fae480b6b20 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae480d2f10 con 0x7fae28003a00 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae4e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae480d34d0 con 0x7fae4814e020 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40764/0 (socket says 172.21.15.67:40764) 2024-04-17T07:33:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 -- 172.21.15.67:0/1335893799 learned_addr learned my addr 172.21.15.67:0/1335893799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae46a00640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 0x7fae480b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 -- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 msgr2=0x7fae480b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47400640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 0x7fae48149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 0x7fae480b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 -- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 msgr2=0x7fae48149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 0x7fae48149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47e00640 1 -- 172.21.15.67:0/1335893799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae480cc960 con 0x7fae480b6b20 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae46a00640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 0x7fae480b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.483+0000 7fae47400640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 0x7fae48149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae47e00640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fae480d0560 tx=0x7fae4000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4000c620 con 0x7fae480b6b20 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4001a070 con 0x7fae480b6b20 2024-04-17T07:33:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae480ccc40 con 0x7fae480b6b20 2024-04-17T07:33:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4000ca00 con 0x7fae480b6b20 2024-04-17T07:33:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.484+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae480c8eb0 con 0x7fae480b6b20 2024-04-17T07:33:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.485+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae480c8040 con 0x7fae480b6b20 2024-04-17T07:33:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.486+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 77) ==== 91086+0+0 (secure 0 0 0) 0x7fae4000cba0 con 0x7fae480b6b20 2024-04-17T07:33:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.488+0000 7fae3f400640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 0x7fae2406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:33:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.488+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6554+0+0 (secure 0 0 0) 0x7fae4008bd20 con 0x7fae480b6b20 2024-04-17T07:33:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.488+0000 7fae47400640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 0x7fae2406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:33:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.490+0000 7fae47400640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 0x7fae2406ee40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fae38033000 tx=0x7fae38009030 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:33:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.493+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fae480c8040 con 0x7fae480b6b20 2024-04-17T07:33:59.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:33:59.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:33:59.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:33:59.745+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fae48068ca0 con 0x7fae480b6b20 2024-04-17T07:34:00.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.277+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fae400c0b90 con 0x7fae480b6b20 2024-04-17T07:34:00.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.310+0000 7fae3f400640 1 -- 172.21.15.67:0/1335893799 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) ==== 104+0+0 (secure 0 0 0) 0x7fae40024020 con 0x7fae480b6b20 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 msgr2=0x7fae2406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 0x7fae2406ee40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fae38033000 tx=0x7fae38009030 comp rx=0 tx=0).stop 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 msgr2=0x7fae480c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fae480d0560 tx=0x7fae4000ec00 comp rx=0 tx=0).stop 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 shutdown_connections 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fae2406c980 0x7fae2406ee40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae480b6b20 0x7fae480c96b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae4814e020 0x7fae480b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 --2- 172.21.15.67:0/1335893799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae28003a00 0x7fae48149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.314+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 >> 172.21.15.67:0/1335893799 conn(0x7fae48067600 msgr2=0x7fae480cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.315+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 shutdown_connections 2024-04-17T07:34:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.315+0000 7fae4e600640 1 -- 172.21.15.67:0/1335893799 wait complete. 2024-04-17T07:34:00.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T07:34:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.563+0000 7f12bec00640 1 Processor -- start 2024-04-17T07:34:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.564+0000 7f12bec00640 1 -- start start 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.565+0000 7f12bec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12b815ee90 0x7f12b80cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.565+0000 7f12bec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b80cf7f0 0x7f12b806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.565+0000 7f12bec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b80cfe20 con 0x7f12b806d2d0 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.565+0000 7f12bec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f12b80d0100 con 0x7f12b815ee90 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.565+0000 7f12bec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12b80d2730 con 0x7f12b80cf7f0 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12b815ee90 0x7f12b80cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12bcc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f12b806d2d0 0x7f12b81628a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44232/0 (socket says 172.21.15.67:44232) 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12b815ee90 0x7f12b80cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40768/0 (socket says 172.21.15.67:40768) 2024-04-17T07:34:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12bcc00640 1 -- 172.21.15.67:0/3941691466 learned_addr learned my addr 172.21.15.67:0/3941691466 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12afe00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b80cf7f0 0x7f12b806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 -- 172.21.15.67:0/3941691466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b80cf7f0 msgr2=0x7f12b806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b80cf7f0 0x7f12b806cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 -- 172.21.15.67:0/3941691466 >> v1:172.21.15.67:6789/0 conn(0x7f12b806d2d0 legacy=0x7f12b81628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7400640 1 -- 172.21.15.67:0/3941691466 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923366825 0 0) 0x7f12b80cfe20 con 0x7f12b806d2d0 2024-04-17T07:34:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.566+0000 7f12b7e00640 1 -- 172.21.15.67:0/3941691466 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b80d2a10 con 0x7f12b815ee90 2024-04-17T07:34:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12b7e00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b815ee90 0x7f12b80cf2b0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f12a80097a0 tx=0x7f12a802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e187580973dce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12b7400640 1 -- 172.21.15.67:0/3941691466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031780 con 0x7f12b815ee90 2024-04-17T07:34:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12b7400640 1 -- 172.21.15.67:0/3941691466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a8037070 con 0x7f12b815ee90 2024-04-17T07:34:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12b7400640 1 -- 172.21.15.67:0/3941691466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8031b40 con 0x7f12b815ee90 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12bec00640 1 -- 172.21.15.67:0/3941691466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b815ee90 msgr2=0x7f12b80cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.568+0000 7f12bec00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b815ee90 0x7f12b80cf2b0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f12a80097a0 tx=0x7f12a802b820 comp rx=0 tx=0).stop 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 -- 172.21.15.67:0/3941691466 shutdown_connections 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b80cf7f0 0x7f12b806cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 --2- 172.21.15.67:0/3941691466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b815ee90 0x7f12b80cf2b0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 -- 172.21.15.67:0/3941691466 >> 172.21.15.67:0/3941691466 conn(0x7f12b80d1a70 msgr2=0x7f12b80d49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 -- 172.21.15.67:0/3941691466 shutdown_connections 2024-04-17T07:34:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.569+0000 7f12bec00640 1 -- 172.21.15.67:0/3941691466 wait complete. 2024-04-17T07:34:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.571+0000 7f12bec00640 1 Processor -- start 2024-04-17T07:34:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.571+0000 7f12bec00640 1 -- start start 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806d2d0 0x7f12b81602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 0x7f12b80cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12b80d40c0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f12b80d3f40 con 0x7f12b815ee90 2024-04-17T07:34:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12bec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12b80d4240 con 0x7f12b806d2d0 2024-04-17T07:34:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40776/0 (socket says 172.21.15.67:40776) 2024-04-17T07:34:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12afe00640 1 -- 172.21.15.67:0/1918834400 learned_addr learned my addr 172.21.15.67:0/1918834400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12afe00640 1 -- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806d2d0 msgr2=0x7f12b81602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.572+0000 7f12b7e00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806d2d0 0x7f12b81602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12bcc00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 0x7f12b80cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12afe00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806d2d0 0x7f12b81602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12afe00640 1 -- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 msgr2=0x7f12b80cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12afe00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 0x7f12b80cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12afe00640 1 -- 172.21.15.67:0/1918834400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b8161dd0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12bcc00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 0x7f12b80cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12afe00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d800 tx=0x7f12a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a000b6c0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a0019070 con 0x7f12b80cf7f0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b81608a0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a000ba80 con 0x7f12b80cf7f0 2024-04-17T07:34:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.573+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b8160e60 con 0x7f12b80cf7f0 2024-04-17T07:34:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.575+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b806e0c0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.576+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f12a000bc20 con 0x7f12b80cf7f0 2024-04-17T07:34:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.576+0000 7f12b4c00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 0x7f129406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:00.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.577+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 6554+0+0 (secure 0 0 0) 0x7f12a008c5c0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.577+0000 7f12b7e00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 0x7f129406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:00.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.578+0000 7f12b7e00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 0x7f129406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f12a8036000 tx=0x7f12a8008030 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.581+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f12b8161dd0 con 0x7f12b80cf7f0 2024-04-17T07:34:00.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:00.798+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f12b8068d60 con 0x7f12b80cf7f0 2024-04-17T07:34:01.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.313+0000 7f12b4c00640 1 -- 172.21.15.67:0/1918834400 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v146) ==== 181+0+0 (secure 0 0 0) 0x7f12a005a4f0 con 0x7f12b80cf7f0 2024-04-17T07:34:01.313 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 msgr2=0x7f129406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 0x7f129406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f12a8036000 tx=0x7f12a8008030 comp rx=0 tx=0).stop 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 msgr2=0x7f12b81eeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d800 tx=0x7f12a000dcc0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 shutdown_connections 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f129406ca10 0x7f129406eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b815ee90 0x7f12b80cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b80cf7f0 0x7f12b81eeee0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 --2- 172.21.15.67:0/1918834400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806d2d0 0x7f12b81602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.316+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 >> 172.21.15.67:0/1918834400 conn(0x7f12b80d1a70 msgr2=0x7f12b80b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.317+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 shutdown_connections 2024-04-17T07:34:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.317+0000 7f12bec00640 1 -- 172.21.15.67:0/1918834400 wait complete. 2024-04-17T07:34:01.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T07:34:01.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.555+0000 7fb7a0200640 1 Processor -- start 2024-04-17T07:34:01.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.555+0000 7fb7a0200640 1 -- start start 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb7a0200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb7980723e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb7a0200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7980cf040 con 0x7fb798146010 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb7a0200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb79806de30 con 0x7fb7980ce3f0 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb7a0200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb798072920 con 0x7fb798071f80 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb79d800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb798146010 0x7fb79806e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44236/0 (socket says 172.21.15.67:44236) 2024-04-17T07:34:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb79d800640 1 -- 172.21.15.67:0/3818983503 learned_addr learned my addr 172.21.15.67:0/3818983503 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.557+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238604316 0 0) 0x7fb7980cf040 con 0x7fb798146010 2024-04-17T07:34:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb780003520 con 0x7fb798146010 2024-04-17T07:34:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb79ce00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb7980723e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3197765093 0 0) 0x7fb780003520 con 0x7fb798146010 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 msgr2=0x7fb7980723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb7980723e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 >> v1:172.21.15.187:6789/0 conn(0x7fb7980ce3f0 legacy=0x7fb7980ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79806e110 con 0x7fb798146010 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb78c001670 con 0x7fb798146010 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 >> v1:172.21.15.67:6789/0 conn(0x7fb798146010 legacy=0x7fb79806e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb780003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb78000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7800054c0 con 0x7fb7800039e0 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb780005670 con 0x7fb780004800 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb78000bf80 con 0x7fb798071f80 2024-04-17T07:34:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.558+0000 7fb79d800640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb780003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79ce00640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79e200640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb78000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79d800640 1 -- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 msgr2=0x7fb78000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79d800640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb78000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79d800640 1 -- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb780004800 msgr2=0x7fb780004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79d800640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb780004800 0x7fb780004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79d800640 1 -- 172.21.15.67:0/3818983503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb780003520 con 0x7fb7800039e0 2024-04-17T07:34:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.559+0000 7fb79e200640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb798071f80 0x7fb78000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.560+0000 7fb79d800640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb780003e00 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c0e0 tx=0x7fb780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0991074bfbd390 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.560+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c007c20 con 0x7fb7800039e0 2024-04-17T07:34:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.560+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78c030070 con 0x7fb7800039e0 2024-04-17T07:34:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.560+0000 7fb797e00640 1 -- 172.21.15.67:0/3818983503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c005d30 con 0x7fb7800039e0 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 -- 172.21.15.67:0/3818983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 msgr2=0x7fb780003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 --2- 172.21.15.67:0/3818983503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb780003e00 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb78c00c0e0 tx=0x7fb780005830 comp rx=0 tx=0).stop 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb79e200640 1 -- 172.21.15.67:0/3818983503 reap_dead start 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 -- 172.21.15.67:0/3818983503 shutdown_connections 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 -- 172.21.15.67:0/3818983503 >> 172.21.15.67:0/3818983503 conn(0x7fb79806b8c0 msgr2=0x7fb79806bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 -- 172.21.15.67:0/3818983503 shutdown_connections 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.561+0000 7fb7a0200640 1 -- 172.21.15.67:0/3818983503 wait complete. 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.563+0000 7fb7a0200640 1 Processor -- start 2024-04-17T07:34:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.563+0000 7fb7a0200640 1 -- start start 2024-04-17T07:34:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7980c5f70 0x7fb7980c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7980ae220 0x7fb7980ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7981463f0 con 0x7fb7800039e0 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb7980cf320 con 0x7fb7980ae220 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb7a0200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb798146570 con 0x7fb7980c5f70 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb79d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb79d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40792/0 (socket says 172.21.15.67:40792) 2024-04-17T07:34:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.564+0000 7fb79d800640 1 -- 172.21.15.67:0/471180155 learned_addr learned my addr 172.21.15.67:0/471180155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 -- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7980c5f70 msgr2=0x7fb7980c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7980c5f70 0x7fb7980c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79e200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7980ae220 0x7fb7980ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 -- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7980ae220 msgr2=0x7fb7980ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7980ae220 0x7fb7980ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 -- 172.21.15.67:0/471180155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7980aefc0 con 0x7fb7800039e0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.565+0000 7fb79d800640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb78c02c460 tx=0x7fb79806e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.566+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c004bf0 con 0x7fb7800039e0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.566+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78c004d90 con 0x7fb7800039e0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.566+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78c0072b0 con 0x7fb7800039e0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.566+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7980af2a0 con 0x7fb7800039e0 2024-04-17T07:34:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.566+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7980af860 con 0x7fb7800039e0 2024-04-17T07:34:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.567+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb798073a50 con 0x7fb7800039e0 2024-04-17T07:34:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.568+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fb78c047020 con 0x7fb7800039e0 2024-04-17T07:34:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.569+0000 7fb795600640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 0x7fb77806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.569+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 6116+0+0 (secure 0 0 0) 0x7fb78c0ad1b0 con 0x7fb7800039e0 2024-04-17T07:34:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.570+0000 7fb79ce00640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 0x7fb77806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:01.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.571+0000 7fb79ce00640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 0x7fb77806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb784006a90 tx=0x7fb784006a20 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.574+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb78c07b2a0 con 0x7fb7800039e0 2024-04-17T07:34:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:01.783+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb798068d60 con 0x7fb7800039e0 2024-04-17T07:34:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.328+0000 7fb795600640 1 -- 172.21.15.67:0/471180155 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v147) ==== 169+0+0 (secure 0 0 0) 0x7fb78c034e80 con 0x7fb7800039e0 2024-04-17T07:34:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T07:34:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.331+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 msgr2=0x7fb77806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.331+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 0x7fb77806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb784006a90 tx=0x7fb784006a20 comp rx=0 tx=0).stop 2024-04-17T07:34:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.331+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 msgr2=0x7fb79813c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.331+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb78c02c460 tx=0x7fb79806e000 comp rx=0 tx=0).stop 2024-04-17T07:34:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 shutdown_connections 2024-04-17T07:34:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fb77806ca10 0x7fb77806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7980ae220 0x7fb7980ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7980c5f70 0x7fb7980c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 --2- 172.21.15.67:0/471180155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7800039e0 0x7fb79813c300 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.334+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 >> 172.21.15.67:0/471180155 conn(0x7fb79806b8c0 msgr2=0x7fb7980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.335+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 shutdown_connections 2024-04-17T07:34:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.336+0000 7fb7a0200640 1 -- 172.21.15.67:0/471180155 wait complete. 2024-04-17T07:34:02.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T07:34:02.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.568+0000 7fbf44c00640 1 Processor -- start 2024-04-17T07:34:02.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.568+0000 7fbf44c00640 1 -- start start 2024-04-17T07:34:02.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.569+0000 7fbf44c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf400cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.569+0000 7fbf44c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf400d00b0 con 0x7fbf400d33e0 2024-04-17T07:34:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.569+0000 7fbf44c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf400d0390 con 0x7fbf4015ecb0 2024-04-17T07:34:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.569+0000 7fbf44c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf400d0c30 con 0x7fbf400b4fd0 2024-04-17T07:34:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.570+0000 7fbf3ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf400d33e0 0x7fbf400cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44252/0 (socket says 172.21.15.67:44252) 2024-04-17T07:34:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.570+0000 7fbf3ea00640 1 -- 172.21.15.67:0/3668430098 learned_addr learned my addr 172.21.15.67:0/3668430098 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.570+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf400cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.570+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546382463 0 0) 0x7fbf400d00b0 con 0x7fbf400d33e0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.570+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf24003520 con 0x7fbf400d33e0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1013724742 0 0) 0x7fbf24003520 con 0x7fbf400d33e0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 msgr2=0x7fbf400cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf400cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 >> v1:172.21.15.187:6789/0 conn(0x7fbf4015ecb0 legacy=0x7fbf400d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf400d0670 con 0x7fbf400d33e0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbf280019f0 con 0x7fbf400d33e0 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 >> v1:172.21.15.67:6789/0 conn(0x7fbf400d33e0 legacy=0x7fbf400cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf400d0c30 con 0x7fbf24004d40 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2400bb50 con 0x7fbf240039e0 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf2400bd20 con 0x7fbf400b4fd0 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.571+0000 7fbf3f400640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3fe00640 1 -- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 msgr2=0x7fbf24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3ea00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf400b4fd0 0x7fbf24004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3fe00640 1 -- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf240039e0 msgr2=0x7fbf24003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.572+0000 7fbf3fe00640 1 -- 172.21.15.67:0/3668430098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf24003520 con 0x7fbf24004d40 2024-04-17T07:34:02.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.573+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009ac0 tx=0x7fbf24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3e8ade2127d3ab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:02.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.573+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38039070 con 0x7fbf24004d40 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.573+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3802c680 con 0x7fbf24004d40 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.573+0000 7fbf3e000640 1 -- 172.21.15.67:0/3668430098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38009690 con 0x7fbf24004d40 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 -- 172.21.15.67:0/3668430098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf24004d40 msgr2=0x7fbf240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 --2- 172.21.15.67:0/3668430098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009ac0 tx=0x7fbf24005830 comp rx=0 tx=0).stop 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf3fe00640 1 -- 172.21.15.67:0/3668430098 reap_dead start 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 -- 172.21.15.67:0/3668430098 shutdown_connections 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 -- 172.21.15.67:0/3668430098 >> 172.21.15.67:0/3668430098 conn(0x7fbf4008bff0 msgr2=0x7fbf400d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 -- 172.21.15.67:0/3668430098 shutdown_connections 2024-04-17T07:34:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.574+0000 7fbf44c00640 1 -- 172.21.15.67:0/3668430098 wait complete. 2024-04-17T07:34:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.576+0000 7fbf44c00640 1 Processor -- start 2024-04-17T07:34:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.576+0000 7fbf44c00640 1 -- start start 2024-04-17T07:34:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf24004d40 0x7fbf400ba170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf40150920 0x7fbf40150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4015f210 con 0x7fbf400b72d0 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4015f090 con 0x7fbf24004d40 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf44c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf400cfab0 con 0x7fbf40150920 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf3ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf3ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40810/0 (socket says 172.21.15.67:40810) 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.577+0000 7fbf3ea00640 1 -- 172.21.15.67:0/1895006907 learned_addr learned my addr 172.21.15.67:0/1895006907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3f400640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf24004d40 0x7fbf400ba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 -- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf40150920 msgr2=0x7fbf40150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf40150920 0x7fbf40150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 -- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf24004d40 msgr2=0x7fbf400ba170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf24004d40 0x7fbf400ba170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 -- 172.21.15.67:0/1895006907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf401512e0 con 0x7fbf400b72d0 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf3ea00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800dfc0 tx=0x7fbf2800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:02.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28019070 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf28004d80 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28007310 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf400b7750 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.578+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf400c9ce0 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.580+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf40068a10 con 0x7fbf400b72d0 2024-04-17T07:34:02.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.580+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fbf2801c070 con 0x7fbf400b72d0 2024-04-17T07:34:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.581+0000 7fbf37400640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 0x7fbf1c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.582+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7fbf2808bfa0 con 0x7fbf400b72d0 2024-04-17T07:34:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.582+0000 7fbf3f400640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 0x7fbf1c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.583+0000 7fbf3f400640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 0x7fbf1c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbf30007fd0 tx=0x7fbf30009040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:02.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.586+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbf2805a1b0 con 0x7fbf400b72d0 2024-04-17T07:34:02.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.807+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7fbf4006b930 con 0x7fbf400b72d0 2024-04-17T07:34:02.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.813+0000 7fbf37400640 1 -- 172.21.15.67:0/1895006907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) ==== 115+0+0 (secure 0 0 0) 0x7fbf28059ab0 con 0x7fbf400b72d0 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 msgr2=0x7fbf1c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 0x7fbf1c06ef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbf30007fd0 tx=0x7fbf30009040 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 msgr2=0x7fbf400ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800dfc0 tx=0x7fbf2800b4a0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 shutdown_connections 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fbf1c06ca40 0x7fbf1c06ef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf40150920 0x7fbf40150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf400b72d0 0x7fbf400ce040 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 --2- 172.21.15.67:0/1895006907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf24004d40 0x7fbf400ba170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.816+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 >> 172.21.15.67:0/1895006907 conn(0x7fbf4008bff0 msgr2=0x7fbf4006dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.817+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 shutdown_connections 2024-04-17T07:34:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:02.817+0000 7fbf44c00640 1 -- 172.21.15.67:0/1895006907 wait complete. 2024-04-17T07:34:02.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T07:34:03.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.038+0000 7f7545000640 1 Processor -- start 2024-04-17T07:34:03.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.039+0000 7f7545000640 1 -- start start 2024-04-17T07:34:03.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.040+0000 7f7545000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f754006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.040+0000 7f7545000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f754006f800 0x7f754006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.040+0000 7f7545000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754006df40 con 0x7f75400cfc10 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f7545000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f754006ed00 con 0x7f754006f800 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f7545000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f754006e870 con 0x7f754015b6b0 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f753ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f754006f800 0x7f754006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f753fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75400cfc10 0x7f754015f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44268/0 (socket says 172.21.15.67:44268) 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f753fe00640 1 -- 172.21.15.67:0/2555297436 learned_addr learned my addr 172.21.15.67:0/2555297436 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f753f400640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f754006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.041+0000 7f753ea00640 1 -- 172.21.15.67:0/2555297436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 msgr2=0x7f754006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.042+0000 7f753ea00640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f754006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.042+0000 7f753ea00640 1 -- 172.21.15.67:0/2555297436 >> v1:172.21.15.67:6789/0 conn(0x7f75400cfc10 legacy=0x7f754015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.042+0000 7f753ea00640 1 -- 172.21.15.67:0/2555297436 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754006e220 con 0x7f754006f800 2024-04-17T07:34:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.042+0000 7f753e000640 1 -- 172.21.15.67:0/2555297436 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280768006 0 0) 0x7f754006df40 con 0x7f75400cfc10 2024-04-17T07:34:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.043+0000 7f753ea00640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f754006f800 0x7f754006be00 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f75280097a0 tx=0x7f752802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ace8441cd857ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.043+0000 7f753e000640 1 -- 172.21.15.67:0/2555297436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528031780 con 0x7f754006f800 2024-04-17T07:34:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.043+0000 7f753e000640 1 -- 172.21.15.67:0/2555297436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7528037070 con 0x7f754006f800 2024-04-17T07:34:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.043+0000 7f753e000640 1 -- 172.21.15.67:0/2555297436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528031b40 con 0x7f754006f800 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 -- 172.21.15.67:0/2555297436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f754006f800 msgr2=0x7f754006be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f754006f800 0x7f754006be00 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f75280097a0 tx=0x7f752802b820 comp rx=0 tx=0).stop 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 -- 172.21.15.67:0/2555297436 shutdown_connections 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f754006f800 0x7f754006be00 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 --2- 172.21.15.67:0/2555297436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f754006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 -- 172.21.15.67:0/2555297436 >> 172.21.15.67:0/2555297436 conn(0x7f75400b2190 msgr2=0x7f75400b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 -- 172.21.15.67:0/2555297436 shutdown_connections 2024-04-17T07:34:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.044+0000 7f7545000640 1 -- 172.21.15.67:0/2555297436 wait complete. 2024-04-17T07:34:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.046+0000 7f7545000640 1 Processor -- start 2024-04-17T07:34:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.046+0000 7f7545000640 1 -- start start 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754006f800 0x7f7540146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f75401f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75400afee0 con 0x7f75400cfc10 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75400afd60 con 0x7f754006f800 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f7545000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75400b44a0 con 0x7f754015b6b0 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f753ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f753ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40822/0 (socket says 172.21.15.67:40822) 2024-04-17T07:34:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.047+0000 7f753ea00640 1 -- 172.21.15.67:0/1041124084 learned_addr learned my addr 172.21.15.67:0/1041124084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753f400640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754006f800 0x7f7540146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 -- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 msgr2=0x7f75401f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f75401f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 -- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754006f800 msgr2=0x7f7540146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754006f800 0x7f7540146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 -- 172.21.15.67:0/1041124084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754015d650 con 0x7f75400cfc10 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f753ea00640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7528034040 tx=0x7f75280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528004230 con 0x7f75400cfc10 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75280043d0 con 0x7f75400cfc10 2024-04-17T07:34:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528003850 con 0x7f75400cfc10 2024-04-17T07:34:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754015cac0 con 0x7f75400cfc10 2024-04-17T07:34:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.048+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7540151c80 con 0x7f75400cfc10 2024-04-17T07:34:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.050+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7540068a10 con 0x7f75400cfc10 2024-04-17T07:34:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.050+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f75280039f0 con 0x7f75400cfc10 2024-04-17T07:34:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.051+0000 7f7537400640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 0x7f751c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.051+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f75280adc60 con 0x7f75400cfc10 2024-04-17T07:34:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.052+0000 7f753f400640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 0x7f751c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.053+0000 7f753f400640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 0x7f751c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7530009f30 tx=0x7f753000b040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.056+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f754015d650 con 0x7f75400cfc10 2024-04-17T07:34:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.262+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f754015abd0 con 0x7f75400cfc10 2024-04-17T07:34:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.271+0000 7f7537400640 1 -- 172.21.15.67:0/1041124084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) ==== 104+0+0 (secure 0 0 0) 0x7f7528041070 con 0x7f75400cfc10 2024-04-17T07:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 msgr2=0x7f751c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 0x7f751c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7530009f30 tx=0x7f753000b040 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 msgr2=0x7f754015d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7528034040 tx=0x7f75280046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 shutdown_connections 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f751c06ca10 0x7f751c06eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f754015b6b0 0x7f75401f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75400cfc10 0x7f754015d110 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 --2- 172.21.15.67:0/1041124084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f754006f800 0x7f7540146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.274+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 >> 172.21.15.67:0/1041124084 conn(0x7f75400b2190 msgr2=0x7f75400d0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.275+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 shutdown_connections 2024-04-17T07:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.275+0000 7f7545000640 1 -- 172.21.15.67:0/1041124084 wait complete. 2024-04-17T07:34:03.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T07:34:03.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.521+0000 7f0085400640 1 Processor -- start 2024-04-17T07:34:03.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.521+0000 7f0085400640 1 -- start start 2024-04-17T07:34:03.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f0085400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 0x7f0080071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f0085400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00800cea70 con 0x7f00800ce3b0 2024-04-17T07:34:03.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f0085400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00800ced50 con 0x7f0080145ac0 2024-04-17T07:34:03.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f0085400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00800ce840 con 0x7f0080072d80 2024-04-17T07:34:03.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f007f200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00800ce3b0 0x7f00800cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50100/0 (socket says 172.21.15.67:50100) 2024-04-17T07:34:03.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.523+0000 7f007f200640 1 -- 172.21.15.67:0/3473928861 learned_addr learned my addr 172.21.15.67:0/3473928861 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:03.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007de00640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 0x7f0080071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379089512 0 0) 0x7f00800cea70 con 0x7f00800ce3b0 2024-04-17T07:34:03.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0064003520 con 0x7f00800ce3b0 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3723263482 0 0) 0x7f0064003520 con 0x7f00800ce3b0 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 msgr2=0x7f0080071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 0x7f0080071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 >> v1:172.21.15.187:6789/0 conn(0x7f0080145ac0 legacy=0x7f0080072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00800cf030 con 0x7f00800ce3b0 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f00780019f0 con 0x7f00800ce3b0 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.524+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 >> v1:172.21.15.67:6789/0 conn(0x7f00800ce3b0 legacy=0x7f00800cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00640039e0 0x7f0064003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f0064004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 0x7f00640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f006400ba70 con 0x7f0064004800 2024-04-17T07:34:03.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f006400bc20 con 0x7f00640039e0 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f006400be20 con 0x7f0080072d80 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f0064004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007e800640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00640039e0 0x7f0064003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 -- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 msgr2=0x7f00640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0080072d80 0x7f00640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 -- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00640039e0 msgr2=0x7f0064003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00640039e0 0x7f0064003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.525+0000 7f007de00640 1 -- 172.21.15.67:0/3473928861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0064003520 con 0x7f0064004800 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.526+0000 7f007de00640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f0064004c80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f006c009a90 tx=0x7f0064005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fbe9fb4b90733a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c039070 con 0x7f0064004800 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006c02c6b0 con 0x7f0064004800 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f007d400640 1 -- 172.21.15.67:0/3473928861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c009690 con 0x7f0064004800 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f0085400640 1 -- 172.21.15.67:0/3473928861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 msgr2=0x7f0064004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f0085400640 1 --2- 172.21.15.67:0/3473928861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f0064004c80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f006c009a90 tx=0x7f0064005830 comp rx=0 tx=0).stop 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f007f200640 1 -- 172.21.15.67:0/3473928861 reap_dead start 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f0085400640 1 -- 172.21.15.67:0/3473928861 shutdown_connections 2024-04-17T07:34:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f0085400640 1 -- 172.21.15.67:0/3473928861 >> 172.21.15.67:0/3473928861 conn(0x7f0080067ee0 msgr2=0x7f008006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.527+0000 7f0085400640 1 -- 172.21.15.67:0/3473928861 shutdown_connections 2024-04-17T07:34:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.528+0000 7f0085400640 1 -- 172.21.15.67:0/3473928861 wait complete. 2024-04-17T07:34:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.529+0000 7f0085400640 1 Processor -- start 2024-04-17T07:34:03.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.529+0000 7f0085400640 1 -- start start 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 0x7f00801393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 0x7f00801e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0080145ea0 con 0x7f0064004800 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00800cf5f0 con 0x7f008013e880 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.530+0000 7f0085400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0080071ea0 con 0x7f008013ac70 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38916/0 (socket says 172.21.15.67:38916) 2024-04-17T07:34:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 -- 172.21.15.67:0/496639814 learned_addr learned my addr 172.21.15.67:0/496639814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007de00640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 0x7f00801393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007f200640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 0x7f00801e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 -- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 msgr2=0x7f00801e50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 0x7f00801e50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 -- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 msgr2=0x7f00801393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 0x7f00801393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007e800640 1 -- 172.21.15.67:0/496639814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0080139930 con 0x7f0064004800 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007de00640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 0x7f00801393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.531+0000 7f007f200640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 0x7f00801e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f007e800640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f007000e550 tx=0x7f007000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00700052e0 con 0x7f0064004800 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0070018070 con 0x7f0064004800 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0080139bd0 con 0x7f0064004800 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070003810 con 0x7f0064004800 2024-04-17T07:34:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.532+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008013a190 con 0x7f0064004800 2024-04-17T07:34:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.533+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0080068ca0 con 0x7f0064004800 2024-04-17T07:34:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.534+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f00700039b0 con 0x7f0064004800 2024-04-17T07:34:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.535+0000 7f0076a00640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 0x7f006006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:03.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.535+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f007008ba50 con 0x7f0064004800 2024-04-17T07:34:03.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.536+0000 7f007de00640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 0x7f006006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:03.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.537+0000 7f007de00640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 0x7f006006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f006c036000 tx=0x7f006c030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.540+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0070059cd0 con 0x7f0064004800 2024-04-17T07:34:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.745+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f00800cc6c0 con 0x7f0064004800 2024-04-17T07:34:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.751+0000 7f0076a00640 1 -- 172.21.15.67:0/496639814 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v51) ==== 120+0+0 (secure 0 0 0) 0x7f00700595d0 con 0x7f0064004800 2024-04-17T07:34:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 msgr2=0x7f006006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 0x7f006006eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f006c036000 tx=0x7f006c030000 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 msgr2=0x7f008015c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f007000e550 tx=0x7f007000ea10 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 shutdown_connections 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f006006ca10 0x7f006006eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f008013ac70 0x7f00801e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f008013e880 0x7f00801393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 --2- 172.21.15.67:0/496639814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0064004800 0x7f008015c340 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 >> 172.21.15.67:0/496639814 conn(0x7f0080067ee0 msgr2=0x7f008015f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 shutdown_connections 2024-04-17T07:34:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.754+0000 7f0085400640 1 -- 172.21.15.67:0/496639814 wait complete. 2024-04-17T07:34:03.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-04-17T07:34:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.998+0000 7fd2d9600640 1 Processor -- start 2024-04-17T07:34:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:03.999+0000 7fd2d9600640 1 -- start start 2024-04-17T07:34:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.000+0000 7fd2d9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 0x7fd2d406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.000+0000 7fd2d9600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d4072b30 con 0x7fd2d406f030 2024-04-17T07:34:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.000+0000 7fd2d9600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d406ff90 con 0x7fd2d40cd020 2024-04-17T07:34:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.000+0000 7fd2d9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d40ceb00 con 0x7fd2d406f670 2024-04-17T07:34:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2d406f030 0x7fd2d4072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50108/0 (socket says 172.21.15.67:50108) 2024-04-17T07:34:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d3400640 1 -- 172.21.15.67:0/3650318565 learned_addr learned my addr 172.21.15.67:0/3650318565 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 0x7fd2d406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2710386710 0 0) 0x7fd2d4072b30 con 0x7fd2d406f030 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b8003520 con 0x7fd2d406f030 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 823486142 0 0) 0x7fd2b8003520 con 0x7fd2d406f030 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.001+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 msgr2=0x7fd2d406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 0x7fd2d406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 >> v1:172.21.15.187:6789/0 conn(0x7fd2d40cd020 legacy=0x7fd2d4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d40cede0 con 0x7fd2d406f030 2024-04-17T07:34:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd2cc001670 con 0x7fd2d406f030 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 >> v1:172.21.15.67:6789/0 conn(0x7fd2d406f030 legacy=0x7fd2d4072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b80039e0 0x7fd2b8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 0x7fd2b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d40ceb00 con 0x7fd2b80039e0 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b800bba0 con 0x7fd2b8004d40 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b800bda0 con 0x7fd2d406f670 2024-04-17T07:34:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b80039e0 0x7fd2b8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 -- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 msgr2=0x7fd2b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2d406f670 0x7fd2b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 -- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b8004d40 msgr2=0x7fd2b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.002+0000 7fd2d2a00640 1 -- 172.21.15.67:0/3650318565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b8003520 con 0x7fd2b80039e0 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b80039e0 0x7fd2b8003e00 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4007360 tx=0x7fd2b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a328805ce387b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4039070 con 0x7fd2b80039e0 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c402ccf0 con 0x7fd2b80039e0 2024-04-17T07:34:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d1600640 1 -- 172.21.15.67:0/3650318565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4009be0 con 0x7fd2b80039e0 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d9600640 1 -- 172.21.15.67:0/3650318565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b80039e0 msgr2=0x7fd2b8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d9600640 1 --2- 172.21.15.67:0/3650318565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2b80039e0 0x7fd2b8003e00 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4007360 tx=0x7fd2b8005830 comp rx=0 tx=0).stop 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.004+0000 7fd2d3400640 1 -- 172.21.15.67:0/3650318565 reap_dead start 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.005+0000 7fd2d9600640 1 -- 172.21.15.67:0/3650318565 shutdown_connections 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.005+0000 7fd2d9600640 1 -- 172.21.15.67:0/3650318565 >> 172.21.15.67:0/3650318565 conn(0x7fd2d406bf50 msgr2=0x7fd2d40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.005+0000 7fd2d9600640 1 -- 172.21.15.67:0/3650318565 shutdown_connections 2024-04-17T07:34:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.005+0000 7fd2d9600640 1 -- 172.21.15.67:0/3650318565 wait complete. 2024-04-17T07:34:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.007+0000 7fd2d9600640 1 Processor -- start 2024-04-17T07:34:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.007+0000 7fd2d9600640 1 -- start start 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2b80039e0 0x7fd2d415e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 0x7fd2d415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d40cf680 con 0x7fd2d415de30 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d40cf3a0 con 0x7fd2d4157970 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d40ce720 con 0x7fd2b80039e0 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38928/0 (socket says 172.21.15.67:38928) 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.008+0000 7fd2d2000640 1 -- 172.21.15.67:0/283680625 learned_addr learned my addr 172.21.15.67:0/283680625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:04.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d3400640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 0x7fd2d415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 -- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2b80039e0 msgr2=0x7fd2d415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2b80039e0 0x7fd2d415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2b80039e0 0x7fd2d415e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 -- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 msgr2=0x7fd2d415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 0x7fd2d415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 -- 172.21.15.67:0/283680625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d415f380 con 0x7fd2d415de30 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d3400640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 0x7fd2d415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d2000640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00c500 tx=0x7fd2bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0052d0 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2bc005470 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc012870 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.009+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d415f660 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.010+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d415d910 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.011+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d4068b90 con 0x7fd2d415de30 2024-04-17T07:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.012+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fd2bc012a10 con 0x7fd2d415de30 2024-04-17T07:34:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.013+0000 7fd2caa00640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 0x7fd2b006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.013+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7fd2bc08b980 con 0x7fd2d415de30 2024-04-17T07:34:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.013+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 0x7fd2b006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.015+0000 7fd2d2a00640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 0x7fd2b006ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4036000 tx=0x7fd2c4030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.017+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd2bc059c90 con 0x7fd2d415de30 2024-04-17T07:34:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.298+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7fd2d40c6580 con 0x7fd2d415de30 2024-04-17T07:34:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.351+0000 7fd2caa00640 1 -- 172.21.15.67:0/283680625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v795) ==== 160+0+0 (secure 0 0 0) 0x7fd2bc059590 con 0x7fd2d415de30 2024-04-17T07:34:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 msgr2=0x7fd2b006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 0x7fd2b006ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4036000 tx=0x7fd2c4030000 comp rx=0 tx=0).stop 2024-04-17T07:34:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 msgr2=0x7fd2d4157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00c500 tx=0x7fd2bc00c9c0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 shutdown_connections 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd2b006ca40 0x7fd2b006ef00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d4157970 0x7fd2d415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d415de30 0x7fd2d4157430 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 --2- 172.21.15.67:0/283680625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2b80039e0 0x7fd2d415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 >> 172.21.15.67:0/283680625 conn(0x7fd2d406bf50 msgr2=0x7fd2d41599e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 shutdown_connections 2024-04-17T07:34:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.354+0000 7fd2d9600640 1 -- 172.21.15.67:0/283680625 wait complete. 2024-04-17T07:34:04.374 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-04-17T07:34:04.374 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-04-17T07:34:04.374 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T07:34:04.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:34:04.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.596+0000 7f1e09400640 1 Processor -- start 2024-04-17T07:34:04.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.596+0000 7f1e09400640 1 -- start start 2024-04-17T07:34:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e09400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e0415b530 0x7f1e04071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e09400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e040c68c0 0x7f1e04072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e09400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e0415f3c0 con 0x7f1e0415b910 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e09400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e0406ed60 con 0x7f1e0415b530 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e09400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0415f660 con 0x7f1e040c68c0 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.598+0000 7f1e02800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e0415b530 0x7f1e04071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e0415b530 0x7f1e04071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38938/0 (socket says 172.21.15.67:38938) 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 -- 172.21.15.67:0/1059326711 learned_addr learned my addr 172.21.15.67:0/1059326711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e01e00640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e040c68c0 0x7f1e04072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 -- 172.21.15.67:0/1059326711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e040c68c0 msgr2=0x7f1e04072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e040c68c0 0x7f1e04072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 -- 172.21.15.67:0/1059326711 >> v1:172.21.15.187:6789/0 conn(0x7f1e0415b910 legacy=0x7f1e040c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.599+0000 7f1e02800640 1 -- 172.21.15.67:0/1059326711 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0415f940 con 0x7f1e0415b530 2024-04-17T07:34:04.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.601+0000 7f1e02800640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e04071d10 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009870 tx=0x7f1df402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac899e8bcbebf4fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.601+0000 7f1e01400640 1 -- 172.21.15.67:0/1059326711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4034070 con 0x7f1e0415b530 2024-04-17T07:34:04.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.601+0000 7f1e01400640 1 -- 172.21.15.67:0/1059326711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4030840 con 0x7f1e0415b530 2024-04-17T07:34:04.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.601+0000 7f1e01400640 1 -- 172.21.15.67:0/1059326711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4030c10 con 0x7f1e0415b530 2024-04-17T07:34:04.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 -- 172.21.15.67:0/1059326711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 msgr2=0x7f1e04071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e04071d10 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009870 tx=0x7f1df402b980 comp rx=0 tx=0).stop 2024-04-17T07:34:04.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 -- 172.21.15.67:0/1059326711 shutdown_connections 2024-04-17T07:34:04.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e040c68c0 0x7f1e04072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 --2- 172.21.15.67:0/1059326711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e04071d10 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 -- 172.21.15.67:0/1059326711 >> 172.21.15.67:0/1059326711 conn(0x7f1e0406bf50 msgr2=0x7f1e040b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:04.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 -- 172.21.15.67:0/1059326711 shutdown_connections 2024-04-17T07:34:04.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.602+0000 7f1e09400640 1 -- 172.21.15.67:0/1059326711 wait complete. 2024-04-17T07:34:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.604+0000 7f1e09400640 1 Processor -- start 2024-04-17T07:34:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.605+0000 7f1e09400640 1 -- start start 2024-04-17T07:34:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e040c68c0 0x7f1e04151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e0415b910 0x7f1e04136a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04160140 con 0x7f1e0415b530 2024-04-17T07:34:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e040cc950 con 0x7f1e040c68c0 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.606+0000 7f1e09400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e04152560 con 0x7f1e0415b910 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38954/0 (socket says 172.21.15.67:38954) 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 -- 172.21.15.67:0/518247768 learned_addr learned my addr 172.21.15.67:0/518247768 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e02800640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e040c68c0 0x7f1e04151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e03200640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e0415b910 0x7f1e04136a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 -- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e0415b910 msgr2=0x7f1e04136a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e0415b910 0x7f1e04136a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 -- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e040c68c0 msgr2=0x7f1e04151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e040c68c0 0x7f1e04151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.607+0000 7f1e01e00640 1 -- 172.21.15.67:0/518247768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e04152840 con 0x7f1e0415b530 2024-04-17T07:34:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1e01e00640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00d7d0 tx=0x7f1dec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00b6c0 con 0x7f1e0415b530 2024-04-17T07:34:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec019070 con 0x7f1e0415b530 2024-04-17T07:34:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e04152b20 con 0x7f1e0415b530 2024-04-17T07:34:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00ba80 con 0x7f1e0415b530 2024-04-17T07:34:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.608+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041530e0 con 0x7f1e0415b530 2024-04-17T07:34:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.609+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e04068b90 con 0x7f1e0415b530 2024-04-17T07:34:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.610+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f1dec00bc20 con 0x7f1e0415b530 2024-04-17T07:34:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.612+0000 7f1dfaa00640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 0x7f1de006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.612+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f1dec08c230 con 0x7f1e0415b530 2024-04-17T07:34:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.612+0000 7f1e02800640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 0x7f1de006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.613+0000 7f1e02800640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 0x7f1de006eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1df4036000 tx=0x7f1df403b040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.616+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1dec01c360 con 0x7f1e0415b530 2024-04-17T07:34:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.821+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e0414fda0 con 0x7f1e0415b530 2024-04-17T07:34:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.823+0000 7f1dfaa00640 1 -- 172.21.15.67:0/518247768 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) ==== 74+0+14104 (secure 0 0 0) 0x7f1dec05a4c0 con 0x7f1e0415b530 2024-04-17T07:34:04.823 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:04.824 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":147,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:34:02.324902+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/996743914":"2024-04-18T07:33:59.256874+0000","172.21.15.187:0/4163913453":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1827135944":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6833/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:6832/2946630767":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/1208304941":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/600549912":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/231129067":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/1184199060":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2363868131":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2637392297":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6833/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.67:6834/996743914":"2024-04-18T07:33:59.256874+0000","172.21.15.187:0/156923885":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/2619020255":"2024-04-18T07:33:41.840202+0000","172.21.15.187:6832/1274719646":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/3318285601":"2024-04-18T07:33:55.262461+0000","172.21.15.187:0/4219847990":"2024-04-18T07:33:41.840202+0000","172.21.15.187:0/2597865993":"2024-04-18T07:33:41.840202+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:34:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 msgr2=0x7f1de006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 0x7f1de006eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1df4036000 tx=0x7f1df403b040 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 msgr2=0x7f1e040ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00d7d0 tx=0x7f1dec00dc90 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 shutdown_connections 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f1de006ca10 0x7f1de006eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e0415b910 0x7f1e04136a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0415b530 0x7f1e040ccf10 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 --2- 172.21.15.67:0/518247768 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e040c68c0 0x7f1e04151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 >> 172.21.15.67:0/518247768 conn(0x7f1e0406bf50 msgr2=0x7f1e041505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 shutdown_connections 2024-04-17T07:34:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:04.826+0000 7f1e09400640 1 -- 172.21.15.67:0/518247768 wait complete. 2024-04-17T07:34:04.848 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:34:04.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:34:05.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.078+0000 7f96fd600640 1 Processor -- start 2024-04-17T07:34:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.078+0000 7f96fd600640 1 -- start start 2024-04-17T07:34:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96fd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96f80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96fd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f80d4f70 con 0x7f96f80b3b50 2024-04-17T07:34:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96fd600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f80d5250 con 0x7f96f80cfcd0 2024-04-17T07:34:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96fd600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96f80cf4d0 con 0x7f96f815ec20 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96f7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96f80b3b50 0x7f96f806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50124/0 (socket says 172.21.15.67:50124) 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.080+0000 7f96f7400640 1 -- 172.21.15.67:0/3430687915 learned_addr learned my addr 172.21.15.67:0/3430687915 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f7e00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96f80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155096725 0 0) 0x7f96f80d4f70 con 0x7f96f80b3b50 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96dc003520 con 0x7f96f80b3b50 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 543246066 0 0) 0x7f96dc003520 con 0x7f96f80b3b50 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 msgr2=0x7f96f80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96f80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 >> v1:172.21.15.187:6789/0 conn(0x7f96f80cfcd0 legacy=0x7f96f80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f80cf7b0 con 0x7f96f80b3b50 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f96e40017f0 con 0x7f96f80b3b50 2024-04-17T07:34:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 >> v1:172.21.15.67:6789/0 conn(0x7f96f80b3b50 legacy=0x7f96f806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc003f60 0x7f96dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004d40 0x7f96dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96f80cf4d0 con 0x7f96dc004d40 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc00bc00 con 0x7f96dc003f60 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.081+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc00be00 con 0x7f96f815ec20 2024-04-17T07:34:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004d40 0x7f96dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96f7400640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc003f60 0x7f96dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96f7e00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 -- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 msgr2=0x7f96dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 -- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc003f60 msgr2=0x7f96dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc003f60 0x7f96dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96fcc00640 1 -- 172.21.15.67:0/3430687915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc003520 con 0x7f96dc004d40 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.082+0000 7f96f7e00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f815ec20 0x7f96dc0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.083+0000 7f96fcc00640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004d40 0x7f96dc0051c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f96f0001e50 tx=0x7f96dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239e293dbad30c1c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f0001a70 con 0x7f96dc004d40 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f0003b70 con 0x7f96dc004d40 2024-04-17T07:34:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96f6a00640 1 -- 172.21.15.67:0/3430687915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f00050d0 con 0x7f96dc004d40 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96fd600640 1 -- 172.21.15.67:0/3430687915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004d40 msgr2=0x7f96dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96fd600640 1 --2- 172.21.15.67:0/3430687915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004d40 0x7f96dc0051c0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f96f0001e50 tx=0x7f96dc005830 comp rx=0 tx=0).stop 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96fcc00640 1 -- 172.21.15.67:0/3430687915 reap_dead start 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96fd600640 1 -- 172.21.15.67:0/3430687915 shutdown_connections 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.084+0000 7f96fd600640 1 -- 172.21.15.67:0/3430687915 >> 172.21.15.67:0/3430687915 conn(0x7f96f808bff0 msgr2=0x7f96f80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.085+0000 7f96fd600640 1 -- 172.21.15.67:0/3430687915 shutdown_connections 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.085+0000 7f96fd600640 1 -- 172.21.15.67:0/3430687915 wait complete. 2024-04-17T07:34:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.086+0000 7f96fd600640 1 Processor -- start 2024-04-17T07:34:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.087+0000 7f96fd600640 1 -- start start 2024-04-17T07:34:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc004d40 0x7f96f81520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 0x7f96f80b8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96f806d2d0 con 0x7f96f80ccc80 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f96f806d150 con 0x7f96dc004d40 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fd600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f96f80d17e0 con 0x7f96f80b7cd0 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38970/0 (socket says 172.21.15.67:38970) 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 -- 172.21.15.67:0/1365213171 learned_addr learned my addr 172.21.15.67:0/1365213171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7e00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc004d40 0x7f96f81520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fcc00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 0x7f96f80b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 -- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 msgr2=0x7f96f80b8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 0x7f96f80b8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 -- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc004d40 msgr2=0x7f96f81520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc004d40 0x7f96f81520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96f7400640 1 -- 172.21.15.67:0/1365213171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f8160260 con 0x7f96f80ccc80 2024-04-17T07:34:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.088+0000 7f96fcc00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 0x7f96f80b8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96f7400640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f96e4001380 tx=0x7f96e400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40070d0 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e4018070 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40074b0 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f8160540 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.089+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f8160b00 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.090+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f8068a10 con 0x7f96f80ccc80 2024-04-17T07:34:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.091+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f96e40057a0 con 0x7f96f80ccc80 2024-04-17T07:34:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.092+0000 7f96efe00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 0x7f96d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.093+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f96e408bc20 con 0x7f96f80ccc80 2024-04-17T07:34:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.093+0000 7f96f7e00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 0x7f96d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.094+0000 7f96f7e00640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 0x7f96d806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009aa0 tx=0x7f96e8008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.097+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f96e4059f30 con 0x7f96f80ccc80 2024-04-17T07:34:05.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.301+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96f80d55f0 con 0x7f96f80ccc80 2024-04-17T07:34:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.302+0000 7f96efe00640 1 -- 172.21.15.67:0/1365213171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) ==== 93+0+10 (secure 0 0 0) 0x7f96e401be40 con 0x7f96f80ccc80 2024-04-17T07:34:05.302 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:34:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 msgr2=0x7f96d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 0x7f96d806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009aa0 tx=0x7f96e8008040 comp rx=0 tx=0).stop 2024-04-17T07:34:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 msgr2=0x7f96f80cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f96e4001380 tx=0x7f96e400eb60 comp rx=0 tx=0).stop 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 shutdown_connections 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f96d806ca10 0x7f96d806eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f96f80b7cd0 0x7f96f80b8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96f80ccc80 0x7f96f80cd100 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 --2- 172.21.15.67:0/1365213171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96dc004d40 0x7f96f81520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 >> 172.21.15.67:0/1365213171 conn(0x7f96f808bff0 msgr2=0x7f96f8074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 shutdown_connections 2024-04-17T07:34:05.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.305+0000 7f96fd600640 1 -- 172.21.15.67:0/1365213171 wait complete. 2024-04-17T07:34:05.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-17T07:34:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.551+0000 7f395f200640 1 Processor -- start 2024-04-17T07:34:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.552+0000 7f395f200640 1 -- start start 2024-04-17T07:34:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f395806b8c0 0x7f39580cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3958141ed0 0x7f3958142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39580cd4a0 con 0x7f3958160280 2024-04-17T07:34:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f395815ce30 con 0x7f395806b8c0 2024-04-17T07:34:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f39581607f0 con 0x7f3958141ed0 2024-04-17T07:34:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f395806b8c0 0x7f39580cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3958160280 0x7f395806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50128/0 (socket says 172.21.15.67:50128) 2024-04-17T07:34:05.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.553+0000 7f395d400640 1 -- 172.21.15.67:0/2906453310 learned_addr learned my addr 172.21.15.67:0/2906453310 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395e800640 1 --2- 172.21.15.67:0/2906453310 >> v2:172.21.15.187:3300/0 conn(0x7f3958141ed0 0x7f3958142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395de00640 1 -- 172.21.15.67:0/2906453310 >> v2:172.21.15.187:3300/0 conn(0x7f3958141ed0 msgr2=0x7f3958142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395de00640 1 --2- 172.21.15.67:0/2906453310 >> v2:172.21.15.187:3300/0 conn(0x7f3958141ed0 0x7f3958142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395de00640 1 -- 172.21.15.67:0/2906453310 >> v1:172.21.15.67:6789/0 conn(0x7f3958160280 legacy=0x7f395806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395de00640 1 -- 172.21.15.67:0/2906453310 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3958160ad0 con 0x7f395806b8c0 2024-04-17T07:34:05.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.554+0000 7f395ca00640 1 -- 172.21.15.67:0/2906453310 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141533255 0 0) 0x7f39580cd4a0 con 0x7f3958160280 2024-04-17T07:34:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.556+0000 7f395de00640 1 --2- 172.21.15.67:0/2906453310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f39580cd8b0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f39580b2120 tx=0x7f394c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b2fce4a726a222 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.556+0000 7f395ca00640 1 -- 172.21.15.67:0/2906453310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c035020 con 0x7f395806b8c0 2024-04-17T07:34:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.556+0000 7f395ca00640 1 -- 172.21.15.67:0/2906453310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c031830 con 0x7f395806b8c0 2024-04-17T07:34:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.556+0000 7f395ca00640 1 -- 172.21.15.67:0/2906453310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c031c00 con 0x7f395806b8c0 2024-04-17T07:34:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 -- 172.21.15.67:0/2906453310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 msgr2=0x7f39580cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 --2- 172.21.15.67:0/2906453310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f39580cd8b0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f39580b2120 tx=0x7f394c02b8d0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 -- 172.21.15.67:0/2906453310 shutdown_connections 2024-04-17T07:34:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 --2- 172.21.15.67:0/2906453310 >> v2:172.21.15.187:3300/0 conn(0x7f3958141ed0 0x7f3958142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 --2- 172.21.15.67:0/2906453310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f39580cd8b0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 -- 172.21.15.67:0/2906453310 >> 172.21.15.67:0/2906453310 conn(0x7f3958067d10 msgr2=0x7f39580b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 -- 172.21.15.67:0/2906453310 shutdown_connections 2024-04-17T07:34:05.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.557+0000 7f395f200640 1 -- 172.21.15.67:0/2906453310 wait complete. 2024-04-17T07:34:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.559+0000 7f395f200640 1 Processor -- start 2024-04-17T07:34:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.559+0000 7f395f200640 1 -- start start 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3958141ed0 0x7f3958155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3958160280 0x7f3958143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3958161090 con 0x7f395806b8c0 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39580cddf0 con 0x7f3958141ed0 2024-04-17T07:34:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f395813f9b0 con 0x7f3958160280 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38994/0 (socket says 172.21.15.67:38994) 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 -- 172.21.15.67:0/1556817203 learned_addr learned my addr 172.21.15.67:0/1556817203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395d400640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3958141ed0 0x7f3958155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 -- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3958160280 msgr2=0x7f3958143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3958160280 0x7f3958143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 -- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3958141ed0 msgr2=0x7f3958155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3958141ed0 0x7f3958155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.560+0000 7f395de00640 1 -- 172.21.15.67:0/1556817203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395815dde0 con 0x7f395806b8c0 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.561+0000 7f395de00640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f394c02bee0 tx=0x7f394c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.561+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c041070 con 0x7f395806b8c0 2024-04-17T07:34:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.561+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c03f070 con 0x7f395806b8c0 2024-04-17T07:34:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.561+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c035040 con 0x7f395806b8c0 2024-04-17T07:34:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.561+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39581561b0 con 0x7f395806b8c0 2024-04-17T07:34:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.562+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39580b3490 con 0x7f395806b8c0 2024-04-17T07:34:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.563+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3958068a10 con 0x7f395806b8c0 2024-04-17T07:34:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.564+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f394c038020 con 0x7f395806b8c0 2024-04-17T07:34:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.565+0000 7f3952000640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 0x7f393c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.565+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f394c0ad440 con 0x7f395806b8c0 2024-04-17T07:34:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.566+0000 7f395d400640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 0x7f393c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.567+0000 7f395d400640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 0x7f393c06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3958136a30 tx=0x7f3948009040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.569+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f394c07b6c0 con 0x7f395806b8c0 2024-04-17T07:34:05.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:05.847+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7f395806c020 con 0x7f395806b8c0 2024-04-17T07:34:06.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.365+0000 7f3952000640 1 -- 172.21.15.67:0/1556817203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v797) ==== 181+0+0 (secure 0 0 0) 0x7f394c07afc0 con 0x7f395806b8c0 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 msgr2=0x7f393c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 0x7f393c06eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3958136a30 tx=0x7f3948009040 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 msgr2=0x7f395815d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f394c02bee0 tx=0x7f394c0046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 shutdown_connections 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f393c06ca10 0x7f393c06eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3958160280 0x7f3958143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3958141ed0 0x7f3958155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 --2- 172.21.15.67:0/1556817203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f395806b8c0 0x7f395815d8a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.368+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 >> 172.21.15.67:0/1556817203 conn(0x7f3958067d10 msgr2=0x7f3958074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.369+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 shutdown_connections 2024-04-17T07:34:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.369+0000 7f395f200640 1 -- 172.21.15.67:0/1556817203 wait complete. 2024-04-17T07:34:06.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T07:34:06.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.618+0000 7f4055400640 1 Processor -- start 2024-04-17T07:34:06.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.618+0000 7f4055400640 1 -- start start 2024-04-17T07:34:06.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.620+0000 7f4055400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f405015fc00 0x7f40501696d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:06.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.620+0000 7f4055400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405006f990 con 0x7f405006f4f0 2024-04-17T07:34:06.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.620+0000 7f4055400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405006fc70 con 0x7f405015f110 2024-04-17T07:34:06.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.620+0000 7f4055400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40500d0af0 con 0x7f405015fc00 2024-04-17T07:34:06.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f405006f4f0 0x7f405015f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50142/0 (socket says 172.21.15.67:50142) 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404de00640 1 -- 172.21.15.67:0/283358758 learned_addr learned my addr 172.21.15.67:0/283358758 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404f200640 1 --2- 172.21.15.67:0/283358758 >> v2:172.21.15.67:3300/0 conn(0x7f405015fc00 0x7f40501696d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404d400640 1 -- 172.21.15.67:0/283358758 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531353254 0 0) 0x7f405006f990 con 0x7f405006f4f0 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404d400640 1 -- 172.21.15.67:0/283358758 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4034003520 con 0x7f405006f4f0 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404f200640 1 -- 172.21.15.67:0/283358758 >> v1:172.21.15.187:6789/0 conn(0x7f405015f110 legacy=0x7f4050072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.621+0000 7f404f200640 1 -- 172.21.15.67:0/283358758 >> v1:172.21.15.67:6789/0 conn(0x7f405006f4f0 legacy=0x7f405015f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.622+0000 7f404f200640 1 -- 172.21.15.67:0/283358758 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40500d04a0 con 0x7f405015fc00 2024-04-17T07:34:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.623+0000 7f404f200640 1 --2- 172.21.15.67:0/283358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015fc00 0x7f40501696d0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4048009d90 tx=0x7f404802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ccf6634d56ee03 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.623+0000 7f404d400640 1 -- 172.21.15.67:0/283358758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048039070 con 0x7f405015fc00 2024-04-17T07:34:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.623+0000 7f404d400640 1 -- 172.21.15.67:0/283358758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404802cd00 con 0x7f405015fc00 2024-04-17T07:34:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.623+0000 7f404d400640 1 -- 172.21.15.67:0/283358758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048034660 con 0x7f405015fc00 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 -- 172.21.15.67:0/283358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015fc00 msgr2=0x7f40501696d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 --2- 172.21.15.67:0/283358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015fc00 0x7f40501696d0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4048009d90 tx=0x7f404802bf50 comp rx=0 tx=0).stop 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 -- 172.21.15.67:0/283358758 shutdown_connections 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 --2- 172.21.15.67:0/283358758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015fc00 0x7f40501696d0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 -- 172.21.15.67:0/283358758 >> 172.21.15.67:0/283358758 conn(0x7f4050067ee0 msgr2=0x7f405006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:06.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 -- 172.21.15.67:0/283358758 shutdown_connections 2024-04-17T07:34:06.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.624+0000 7f4055400640 1 -- 172.21.15.67:0/283358758 wait complete. 2024-04-17T07:34:06.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.626+0000 7f4055400640 1 Processor -- start 2024-04-17T07:34:06.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.626+0000 7f4055400640 1 -- start start 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 0x7f4050160b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f405015fc00 0x7f405015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40500cf3c0 con 0x7f405015f110 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40500afdc0 con 0x7f405015fc00 2024-04-17T07:34:06.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f4055400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40500cf540 con 0x7f405006f4f0 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f404de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.627+0000 7f404de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39000/0 (socket says 172.21.15.67:39000) 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 -- 172.21.15.67:0/1841140477 learned_addr learned my addr 172.21.15.67:0/1841140477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404f200640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f405015fc00 0x7f405015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404e800640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 0x7f4050160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 -- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 msgr2=0x7f4050160b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 0x7f4050160b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 -- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f405015fc00 msgr2=0x7f405015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f405015fc00 0x7f405015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 -- 172.21.15.67:0/1841140477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4050159af0 con 0x7f405015f110 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404e800640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 0x7f4050160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.628+0000 7f404de00640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c500 tx=0x7f403c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.629+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c017070 con 0x7f405015f110 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.629+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403c005ce0 con 0x7f405015f110 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.629+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c007330 con 0x7f405015f110 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.629+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4050159dd0 con 0x7f405015f110 2024-04-17T07:34:06.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.629+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405015a390 con 0x7f405015f110 2024-04-17T07:34:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.630+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4050068ca0 con 0x7f405015f110 2024-04-17T07:34:06.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.631+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f403c01a070 con 0x7f405015f110 2024-04-17T07:34:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.632+0000 7f4046a00640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 0x7f403006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:06.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.632+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f403c08b030 con 0x7f405015f110 2024-04-17T07:34:06.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.633+0000 7f404e800640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 0x7f403006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.634+0000 7f404e800640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 0x7f403006eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f40501f7e80 tx=0x7f4040009040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.636+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f403c058e50 con 0x7f405015f110 2024-04-17T07:34:06.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.844+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f405015c500 con 0x7f405015f110 2024-04-17T07:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.850+0000 7f4046a00640 1 -- 172.21.15.67:0/1841140477 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) ==== 122+0+0 (secure 0 0 0) 0x7f403c05d150 con 0x7f405015f110 2024-04-17T07:34:06.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 msgr2=0x7f403006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 0x7f403006eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f40501f7e80 tx=0x7f4040009040 comp rx=0 tx=0).stop 2024-04-17T07:34:06.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 msgr2=0x7f405014aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c500 tx=0x7f403c00c9c0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 shutdown_connections 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f403006ca10 0x7f403006eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f405015fc00 0x7f405015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f405015f110 0x7f405014aef0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 --2- 172.21.15.67:0/1841140477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405006f4f0 0x7f4050160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 >> 172.21.15.67:0/1841140477 conn(0x7f4050067ee0 msgr2=0x7f40500b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 shutdown_connections 2024-04-17T07:34:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:06.853+0000 7f4055400640 1 -- 172.21.15.67:0/1841140477 wait complete. 2024-04-17T07:34:06.872 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T07:34:06.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:34:06.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T07:34:07.114 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T07:34:07.115 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:34:07.115 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:34:07.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:34:07.176 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:34:07.176 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:34:07.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:34:07.234 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:34:07.234 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:34:07.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T07:34:07.294 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T07:34:07.294 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:34:07.352 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T07:34:07.352 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T07:34:07.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:34:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.581+0000 7f8d56600640 1 Processor -- start 2024-04-17T07:34:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.582+0000 7f8d56600640 1 -- start start 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.583+0000 7f8d56600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d5006b8c0 0x7f8d500cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.583+0000 7f8d56600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d50160280 0x7f8d5006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d56600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d500cd4a0 con 0x7f8d50141ed0 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d56600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8d5015ce30 con 0x7f8d50160280 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d56600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d501607f0 con 0x7f8d5006b8c0 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d55c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d50141ed0 0x7f8d5015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50150/0 (socket says 172.21.15.67:50150) 2024-04-17T07:34:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d55c00640 1 -- 172.21.15.67:0/2436335980 learned_addr learned my addr 172.21.15.67:0/2436335980 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d4fe00640 1 --2- 172.21.15.67:0/2436335980 >> v2:172.21.15.67:3300/0 conn(0x7f8d50160280 0x7f8d5006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d55200640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d5006b8c0 0x7f8d500cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.584+0000 7f8d4f400640 1 -- 172.21.15.67:0/2436335980 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702780834 0 0) 0x7f8d500cd4a0 con 0x7f8d50141ed0 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.585+0000 7f8d4f400640 1 -- 172.21.15.67:0/2436335980 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d38003520 con 0x7f8d50141ed0 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.585+0000 7f8d4fe00640 1 -- 172.21.15.67:0/2436335980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d5006b8c0 msgr2=0x7f8d500cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.585+0000 7f8d4fe00640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d5006b8c0 0x7f8d500cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.585+0000 7f8d4fe00640 1 -- 172.21.15.67:0/2436335980 >> v1:172.21.15.67:6789/0 conn(0x7f8d50141ed0 legacy=0x7f8d5015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.585+0000 7f8d4fe00640 1 -- 172.21.15.67:0/2436335980 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d50160ad0 con 0x7f8d50160280 2024-04-17T07:34:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.586+0000 7f8d4fe00640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d5006bca0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0097a0 tx=0x7f8d3c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb524279bdce729a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.586+0000 7f8d4f400640 1 -- 172.21.15.67:0/2436335980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c035020 con 0x7f8d50160280 2024-04-17T07:34:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.586+0000 7f8d4f400640 1 -- 172.21.15.67:0/2436335980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d3c031810 con 0x7f8d50160280 2024-04-17T07:34:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.586+0000 7f8d4f400640 1 -- 172.21.15.67:0/2436335980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c031be0 con 0x7f8d50160280 2024-04-17T07:34:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 -- 172.21.15.67:0/2436335980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 msgr2=0x7f8d5006bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d5006bca0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0097a0 tx=0x7f8d3c02b8b0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 -- 172.21.15.67:0/2436335980 shutdown_connections 2024-04-17T07:34:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d5006bca0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 --2- 172.21.15.67:0/2436335980 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d5006b8c0 0x7f8d500cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 -- 172.21.15.67:0/2436335980 >> 172.21.15.67:0/2436335980 conn(0x7f8d50067d10 msgr2=0x7f8d500b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 -- 172.21.15.67:0/2436335980 shutdown_connections 2024-04-17T07:34:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.587+0000 7f8d56600640 1 -- 172.21.15.67:0/2436335980 wait complete. 2024-04-17T07:34:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.589+0000 7f8d56600640 1 Processor -- start 2024-04-17T07:34:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.589+0000 7f8d56600640 1 -- start start 2024-04-17T07:34:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5006b8c0 0x7f8d50156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 0x7f8d500b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5006c1e0 con 0x7f8d50160280 2024-04-17T07:34:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8d500cddf0 con 0x7f8d5006b8c0 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d56600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d50161090 con 0x7f8d50141ed0 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.590+0000 7f8d55c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39028/0 (socket says 172.21.15.67:39028) 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 -- 172.21.15.67:0/3981466415 learned_addr learned my addr 172.21.15.67:0/3981466415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55200640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5006b8c0 0x7f8d50156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d4fe00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 0x7f8d500b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 -- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 msgr2=0x7f8d500b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 0x7f8d500b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 -- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5006b8c0 msgr2=0x7f8d50156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5006b8c0 0x7f8d50156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 -- 172.21.15.67:0/3981466415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d500b4410 con 0x7f8d50160280 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d4fe00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 0x7f8d500b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.591+0000 7f8d55c00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800cb40 tx=0x7f8d4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.592+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48017070 con 0x7f8d50160280 2024-04-17T07:34:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.592+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4800aca0 con 0x7f8d50160280 2024-04-17T07:34:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.592+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48012620 con 0x7f8d50160280 2024-04-17T07:34:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.592+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d500b46f0 con 0x7f8d50160280 2024-04-17T07:34:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.592+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d5015dd40 con 0x7f8d50160280 2024-04-17T07:34:07.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.593+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d50068a10 con 0x7f8d50160280 2024-04-17T07:34:07.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.594+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f8d4801a070 con 0x7f8d50160280 2024-04-17T07:34:07.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.595+0000 7f8d4cc00640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 0x7f8d3006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:07.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.595+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7f8d4808bba0 con 0x7f8d50160280 2024-04-17T07:34:07.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.596+0000 7f8d55200640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 0x7f8d3006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:07.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.597+0000 7f8d55200640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 0x7f8d3006eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009ad0 tx=0x7f8d44008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:07.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.600+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8d48059e20 con 0x7f8d50160280 2024-04-17T07:34:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.844+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d50140b90 con 0x7f8d50160280 2024-04-17T07:34:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.845+0000 7f8d4cc00640 1 -- 172.21.15.67:0/3981466415 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+3872 (secure 0 0 0) 0x7f8d48059720 con 0x7f8d50160280 2024-04-17T07:34:07.845 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:07.845 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1180603271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1180603271},{"type":"v1","addr":"172.21.15.187:6836","nonce":1180603271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6353,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1248219731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":1248219731},{"type":"v1","addr":"172.21.15.187:6837","nonce":1248219731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/268999170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":268999170},{"type":"v1","addr":"172.21.15.67:6833","nonce":268999170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":8850,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2646933231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2646933231},{"type":"v1","addr":"172.21.15.67:6835","nonce":2646933231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[]} 2024-04-17T07:34:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 70 2024-04-17T07:34:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.847+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 msgr2=0x7f8d3006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.847+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 0x7f8d3006eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009ad0 tx=0x7f8d44008040 comp rx=0 tx=0).stop 2024-04-17T07:34:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.847+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 msgr2=0x7f8d50143c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.847+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800cb40 tx=0x7f8d4800a060 comp rx=0 tx=0).stop 2024-04-17T07:34:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 shutdown_connections 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f8d3006ca10 0x7f8d3006eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d50160280 0x7f8d50143c20 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d50141ed0 0x7f8d500b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 --2- 172.21.15.67:0/3981466415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d5006b8c0 0x7f8d50156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 >> 172.21.15.67:0/3981466415 conn(0x7f8d50067d10 msgr2=0x7f8d5013f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 shutdown_connections 2024-04-17T07:34:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:07.848+0000 7f8d56600640 1 -- 172.21.15.67:0/3981466415 wait complete. 2024-04-17T07:34:07.870 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T07:34:07.870 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T07:34:07.870 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T07:34:07.870 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:34:07.870 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T07:34:07.870 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T07:34:07.871 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T07:34:07.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:34:07.871 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T07:34:07.871 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T07:34:07.871 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T07:34:07.871 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:34:07.871 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T07:34:07.871 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T07:34:07.872 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T07:34:07.872 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T07:34:07.893 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T07:34:07.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T07:34:07.894 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T07:34:07.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T07:34:07.897 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T07:34:07.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T07:34:07.902 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T07:34:07.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T07:34:07.943 INFO:tasks.ceph.mds.a:Started 2024-04-17T07:34:07.943 DEBUG:teuthology.parallel:result is None 2024-04-17T07:34:07.947 INFO:tasks.ceph.mds.b:Started 2024-04-17T07:34:07.947 DEBUG:teuthology.parallel:result is None 2024-04-17T07:34:07.949 INFO:tasks.ceph.mds.c:Started 2024-04-17T07:34:07.950 DEBUG:teuthology.parallel:result is None 2024-04-17T07:34:07.951 INFO:tasks.ceph.mds.d:Started 2024-04-17T07:34:07.951 DEBUG:teuthology.parallel:result is None 2024-04-17T07:34:07.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T07:34:08.104 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T07:34:08.104 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T07:34:08.107 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T07:34:08.112 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T07:34:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.144+0000 7fdcdfc00640 1 Processor -- start 2024-04-17T07:34:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.144+0000 7fdcdfc00640 1 -- start start 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcdfc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdcd806ce00 0x7fdcd809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcdfc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcd80d1520 con 0x7fdcd809afa0 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcdfc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcd80d1800 con 0x7fdcd806daf0 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcdfc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdcd80d1ae0 con 0x7fdcd806ce00 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcddc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdcd809afa0 0x7fdcd8070040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50172/0 (socket says 172.21.15.67:50172) 2024-04-17T07:34:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcddc00640 1 -- 172.21.15.67:0/671424447 learned_addr learned my addr 172.21.15.67:0/671424447 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.145+0000 7fdcd7e00640 1 --2- 172.21.15.67:0/671424447 >> v2:172.21.15.187:3300/0 conn(0x7fdcd806ce00 0x7fdcd809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.146+0000 7fdcd7400640 1 -- 172.21.15.67:0/671424447 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837929233 0 0) 0x7fdcd80d1520 con 0x7fdcd809afa0 2024-04-17T07:34:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.146+0000 7fdcd7400640 1 -- 172.21.15.67:0/671424447 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcc0003520 con 0x7fdcd809afa0 2024-04-17T07:34:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.146+0000 7fdcd7e00640 1 -- 172.21.15.67:0/671424447 >> v1:172.21.15.187:6789/0 conn(0x7fdcd806daf0 legacy=0x7fdcd80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.146+0000 7fdcd7e00640 1 -- 172.21.15.67:0/671424447 >> v1:172.21.15.67:6789/0 conn(0x7fdcd809afa0 legacy=0x7fdcd8070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.146+0000 7fdcd7e00640 1 -- 172.21.15.67:0/671424447 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd80d1dc0 con 0x7fdcd806ce00 2024-04-17T07:34:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.147+0000 7fdcd7e00640 1 --2- 172.21.15.67:0/671424447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd809aa60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40097a0 tx=0x7fdcc402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65658a0d45003f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.147+0000 7fdcd7400640 1 -- 172.21.15.67:0/671424447 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4034070 con 0x7fdcd806ce00 2024-04-17T07:34:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.147+0000 7fdcd7400640 1 -- 172.21.15.67:0/671424447 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcc4030610 con 0x7fdcd806ce00 2024-04-17T07:34:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.147+0000 7fdcd7400640 1 -- 172.21.15.67:0/671424447 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40309e0 con 0x7fdcd806ce00 2024-04-17T07:34:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 -- 172.21.15.67:0/671424447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 msgr2=0x7fdcd809aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/671424447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd809aa60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40097a0 tx=0x7fdcc402b960 comp rx=0 tx=0).stop 2024-04-17T07:34:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 -- 172.21.15.67:0/671424447 shutdown_connections 2024-04-17T07:34:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/671424447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd809aa60 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 -- 172.21.15.67:0/671424447 >> 172.21.15.67:0/671424447 conn(0x7fdcd80b41a0 msgr2=0x7fdcd80b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 -- 172.21.15.67:0/671424447 shutdown_connections 2024-04-17T07:34:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.148+0000 7fdcdfc00640 1 -- 172.21.15.67:0/671424447 wait complete. 2024-04-17T07:34:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.150+0000 7fdcdfc00640 1 Processor -- start 2024-04-17T07:34:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.150+0000 7fdcdfc00640 1 -- start start 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd80cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdcd809afa0 0x7fdcd81578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd806ded0 con 0x7fdcd806daf0 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd80d1230 con 0x7fdcd806ce00 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdfc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdcd806e050 con 0x7fdcd809afa0 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39044/0 (socket says 172.21.15.67:39044) 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcd7e00640 1 -- 172.21.15.67:0/57085497 learned_addr learned my addr 172.21.15.67:0/57085497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcdd200640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd80cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.151+0000 7fdcddc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdcd809afa0 0x7fdcd81578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 -- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdcd809afa0 msgr2=0x7fdcd81578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdcd809afa0 0x7fdcd81578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 -- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 msgr2=0x7fdcd80cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:08.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd80cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:08.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 -- 172.21.15.67:0/57085497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd80cc6d0 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd7e00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009890 tx=0x7fdcc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4034040 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcc4039990 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4039d60 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.152+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd80c9b20 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.153+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd80ca0e0 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.154+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd8068ca0 con 0x7fdcd806daf0 2024-04-17T07:34:08.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.155+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fdcc4037070 con 0x7fdcd806daf0 2024-04-17T07:34:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.155+0000 7fdcd4c00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 0x7fdcb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.156+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5702+0+0 (secure 0 0 0) 0x7fdcc40adbe0 con 0x7fdcd806daf0 2024-04-17T07:34:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.156+0000 7fdcdd200640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 0x7fdcb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.157+0000 7fdcdd200640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 0x7fdcb806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdcd81f7cc0 tx=0x7fdccc009040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.160+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdcd80cc6d0 con 0x7fdcd806daf0 2024-04-17T07:34:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:08.381+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fdcd806c990 con 0x7fdcd806daf0 2024-04-17T07:34:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.405+0000 7fdcd4c00640 1 -- 172.21.15.67:0/57085497 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v152) ==== 97+0+0 (secure 0 0 0) 0x7fdcc407b8c0 con 0x7fdcd806daf0 2024-04-17T07:34:09.406 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T07:34:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 msgr2=0x7fdcb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 0x7fdcb806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdcd81f7cc0 tx=0x7fdccc009040 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 msgr2=0x7fdcd8157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009890 tx=0x7fdcc40046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 shutdown_connections 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdcb806ca10 0x7fdcb806eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdcd809afa0 0x7fdcd81578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd806daf0 0x7fdcd8157380 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 --2- 172.21.15.67:0/57085497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdcd806ce00 0x7fdcd80cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 >> 172.21.15.67:0/57085497 conn(0x7fdcd80b41a0 msgr2=0x7fdcd80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 shutdown_connections 2024-04-17T07:34:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.410+0000 7fdcdfc00640 1 -- 172.21.15.67:0/57085497 wait complete. 2024-04-17T07:34:09.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T07:34:09.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.656+0000 7f85e9000640 1 Processor -- start 2024-04-17T07:34:09.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.656+0000 7f85e9000640 1 -- start start 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.657+0000 7f85e9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85e40d2d60 0x7f85e40d31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.657+0000 7f85e9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e40d01d0 con 0x7f85e40cfcd0 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e9000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e40d04b0 con 0x7f85e415e7a0 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f85e40d36f0 con 0x7f85e40d2d60 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85e40cfcd0 0x7f85e406ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50182/0 (socket says 172.21.15.67:50182) 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e2a00640 1 -- 172.21.15.67:0/3327602281 learned_addr learned my addr 172.21.15.67:0/3327602281 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e3e00640 1 --2- 172.21.15.67:0/3327602281 >> v2:172.21.15.67:3300/0 conn(0x7f85e40d2d60 0x7f85e40d31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:09.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.658+0000 7f85e2000640 1 -- 172.21.15.67:0/3327602281 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256295712 0 0) 0x7f85e40d01d0 con 0x7f85e40cfcd0 2024-04-17T07:34:09.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.659+0000 7f85e2000640 1 -- 172.21.15.67:0/3327602281 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c8003520 con 0x7f85e40cfcd0 2024-04-17T07:34:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.659+0000 7f85e3e00640 1 -- 172.21.15.67:0/3327602281 >> v1:172.21.15.187:6789/0 conn(0x7f85e415e7a0 legacy=0x7f85e40d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.659+0000 7f85e3e00640 1 -- 172.21.15.67:0/3327602281 >> v1:172.21.15.67:6789/0 conn(0x7f85e40cfcd0 legacy=0x7f85e406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.659+0000 7f85e3e00640 1 -- 172.21.15.67:0/3327602281 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e40d2280 con 0x7f85e40d2d60 2024-04-17T07:34:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.660+0000 7f85e3e00640 1 --2- 172.21.15.67:0/3327602281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40d2d60 0x7f85e40d31b0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009d90 tx=0x7f85dc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c26622b47560832 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.660+0000 7f85e2000640 1 -- 172.21.15.67:0/3327602281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc039070 con 0x7f85e40d2d60 2024-04-17T07:34:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.660+0000 7f85e2000640 1 -- 172.21.15.67:0/3327602281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85dc02cc60 con 0x7f85e40d2d60 2024-04-17T07:34:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.660+0000 7f85e2000640 1 -- 172.21.15.67:0/3327602281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc034640 con 0x7f85e40d2d60 2024-04-17T07:34:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 -- 172.21.15.67:0/3327602281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40d2d60 msgr2=0x7f85e40d31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 --2- 172.21.15.67:0/3327602281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40d2d60 0x7f85e40d31b0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009d90 tx=0x7f85dc02beb0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 -- 172.21.15.67:0/3327602281 shutdown_connections 2024-04-17T07:34:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 --2- 172.21.15.67:0/3327602281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40d2d60 0x7f85e40d31b0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 -- 172.21.15.67:0/3327602281 >> 172.21.15.67:0/3327602281 conn(0x7f85e408bff0 msgr2=0x7f85e40d46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 -- 172.21.15.67:0/3327602281 shutdown_connections 2024-04-17T07:34:09.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.661+0000 7f85e9000640 1 -- 172.21.15.67:0/3327602281 wait complete. 2024-04-17T07:34:09.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.663+0000 7f85e9000640 1 Processor -- start 2024-04-17T07:34:09.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.663+0000 7f85e9000640 1 -- start start 2024-04-17T07:34:09.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:09.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e40d2d60 0x7f85e40c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f85e415e7a0 0x7f85e415cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f85e415eb80 con 0x7f85e40cfcd0 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f85e40d2840 con 0x7f85e40d2d60 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e9000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f85e406d190 con 0x7f85e415e7a0 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39064/0 (socket says 172.21.15.67:39064) 2024-04-17T07:34:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.664+0000 7f85e3400640 1 -- 172.21.15.67:0/200699504 learned_addr learned my addr 172.21.15.67:0/200699504 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:09.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3e00640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f85e415e7a0 0x7f85e415cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:09.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 -- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f85e415e7a0 msgr2=0x7f85e415cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f85e415e7a0 0x7f85e415cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 -- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e40d2d60 msgr2=0x7f85e40c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e40d2d60 0x7f85e40c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 -- 172.21.15.67:0/200699504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e40cbc90 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85e3400640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c430 tx=0x7f85d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d40052d0 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.665+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d4005490 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.666+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e40cbf70 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.666+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d4007260 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.666+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e40cc530 con 0x7f85e40cfcd0 2024-04-17T07:34:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.667+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e40755d0 con 0x7f85e40cfcd0 2024-04-17T07:34:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.668+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f85d4007400 con 0x7f85e40cfcd0 2024-04-17T07:34:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.669+0000 7f85db400640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 0x7f85c006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.669+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4885+0+0 (secure 0 0 0) 0x7f85d408af30 con 0x7f85e40cfcd0 2024-04-17T07:34:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.670+0000 7f85e2a00640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 0x7f85c006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.671+0000 7f85e2a00640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 0x7f85c006ef00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f85e40b58a0 tx=0x7f85cc007930 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:09.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.674+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f85d4059570 con 0x7f85e40cfcd0 2024-04-17T07:34:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.951+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f85e40d1330 con 0x7f85e40cfcd0 2024-04-17T07:34:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.952+0000 7f85db400640 1 -- 172.21.15.67:0/200699504 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) ==== 80+0+5304 (secure 0 0 0) 0x7f85d4058e70 con 0x7f85e40cfcd0 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T07:34:09.954 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:34:09.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T07:34:09.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T07:34:09.961 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.963 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:34:09.965 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T07:34:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:34:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 msgr2=0x7f85c006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 0x7f85c006ef00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f85e40b58a0 tx=0x7f85cc007930 comp rx=0 tx=0).stop 2024-04-17T07:34:09.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 msgr2=0x7f85e4156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c430 tx=0x7f85d400c8f0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 shutdown_connections 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f85c006ca40 0x7f85c006ef00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f85e415e7a0 0x7f85e415cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85e40d2d60 0x7f85e40c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 --2- 172.21.15.67:0/200699504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e40cfcd0 0x7f85e4156020 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 >> 172.21.15.67:0/200699504 conn(0x7f85e408bff0 msgr2=0x7f85e40d4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 shutdown_connections 2024-04-17T07:34:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:09.955+0000 7f85e9000640 1 -- 172.21.15.67:0/200699504 wait complete. 2024-04-17T07:34:09.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:34:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.190+0000 7ff8b3c00640 1 Processor -- start 2024-04-17T07:34:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.190+0000 7ff8b3c00640 1 -- start start 2024-04-17T07:34:10.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 0x7ff8ac06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8ac0cd020 0x7ff8ac0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac071d10 0x7ff8ac0721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8ac06ff90 con 0x7ff8ac0cd020 2024-04-17T07:34:10.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff8ac0ceb00 con 0x7ff8ac06f670 2024-04-17T07:34:10.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.191+0000 7ff8b3c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ac0cede0 con 0x7ff8ac071d10 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8ac0cd020 0x7ff8ac0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8ac0cd020 0x7ff8ac0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39080/0 (socket says 172.21.15.67:39080) 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 -- 172.21.15.67:0/2016135418 learned_addr learned my addr 172.21.15.67:0/2016135418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8b1200640 1 --2- 172.21.15.67:0/2016135418 >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 0x7ff8ac06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8b1c00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac071d10 0x7ff8ac0721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 -- 172.21.15.67:0/2016135418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac071d10 msgr2=0x7ff8ac0721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac071d10 0x7ff8ac0721a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.192+0000 7ff8abe00640 1 -- 172.21.15.67:0/2016135418 >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 msgr2=0x7ff8ac06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.193+0000 7ff8abe00640 1 --2- 172.21.15.67:0/2016135418 >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 0x7ff8ac06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.193+0000 7ff8abe00640 1 -- 172.21.15.67:0/2016135418 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac0cf0c0 con 0x7ff8ac0cd020 2024-04-17T07:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.193+0000 7ff8b1200640 1 --2- 172.21.15.67:0/2016135418 >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 0x7ff8ac06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.194+0000 7ff8abe00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0cd480 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac06c910 tx=0x7ff89802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2834b92699fce52b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.194+0000 7ff8ab400640 1 -- 172.21.15.67:0/2016135418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898035020 con 0x7ff8ac0cd020 2024-04-17T07:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.194+0000 7ff8ab400640 1 -- 172.21.15.67:0/2016135418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898031870 con 0x7ff8ac0cd020 2024-04-17T07:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.194+0000 7ff8ab400640 1 -- 172.21.15.67:0/2016135418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898031c40 con 0x7ff8ac0cd020 2024-04-17T07:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 -- 172.21.15.67:0/2016135418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 msgr2=0x7ff8ac0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0cd480 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac06c910 tx=0x7ff89802b910 comp rx=0 tx=0).stop 2024-04-17T07:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 -- 172.21.15.67:0/2016135418 shutdown_connections 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac071d10 0x7ff8ac0721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/2016135418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0cd480 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/2016135418 >> v2:172.21.15.187:3300/0 conn(0x7ff8ac06f670 0x7ff8ac06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 -- 172.21.15.67:0/2016135418 >> 172.21.15.67:0/2016135418 conn(0x7ff8ac06bf50 msgr2=0x7ff8ac0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 -- 172.21.15.67:0/2016135418 shutdown_connections 2024-04-17T07:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.195+0000 7ff8b3c00640 1 -- 172.21.15.67:0/2016135418 wait complete. 2024-04-17T07:34:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.197+0000 7ff8b3c00640 1 Processor -- start 2024-04-17T07:34:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.197+0000 7ff8b3c00640 1 -- start start 2024-04-17T07:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 0x7ff8ac146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ac071d10 0x7ff8ac0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac0ce720 con 0x7ff8ac0cd020 2024-04-17T07:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac0cf680 con 0x7ff8ac071d10 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.198+0000 7ff8b3c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ac0ce8a0 con 0x7ff8ac06f670 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39084/0 (socket says 172.21.15.67:39084) 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 -- 172.21.15.67:0/1134372771 learned_addr learned my addr 172.21.15.67:0/1134372771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1200640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 0x7ff8ac146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8abe00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ac071d10 0x7ff8ac0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 -- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 msgr2=0x7ff8ac146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 0x7ff8ac146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 -- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ac071d10 msgr2=0x7ff8ac0ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ac071d10 0x7ff8ac0ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 -- 172.21.15.67:0/1134372771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac0b5c10 con 0x7ff8ac0cd020 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1200640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 0x7ff8ac146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.199+0000 7ff8b1c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4007e40 tx=0x7ff8a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.200+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4014620 con 0x7ff8ac0cd020 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.200+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a4019070 con 0x7ff8ac0cd020 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.200+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4014a20 con 0x7ff8ac0cd020 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.200+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac0c9e90 con 0x7ff8ac0cd020 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.200+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac0af120 con 0x7ff8ac0cd020 2024-04-17T07:34:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.201+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8ac0c9cc0 con 0x7ff8ac0cd020 2024-04-17T07:34:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.202+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7ff8a4014bc0 con 0x7ff8ac0cd020 2024-04-17T07:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.203+0000 7ff8a8c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 0x7ff88c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.203+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4885+0+0 (secure 0 0 0) 0x7ff8a408c140 con 0x7ff8ac0cd020 2024-04-17T07:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.204+0000 7ff8b1200640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 0x7ff88c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.205+0000 7ff8b1200640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 0x7ff88c06ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.207+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff8a405a900 con 0x7ff8ac0cd020 2024-04-17T07:34:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.407+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8ac068b90 con 0x7ff8ac0cd020 2024-04-17T07:34:10.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.416+0000 7ff8a8c00640 1 -- 172.21.15.67:0/1134372771 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+12897 (secure 0 0 0) 0x7ff8a405a200 con 0x7ff8ac0cd020 2024-04-17T07:34:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:10.416 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":152,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:34:09.402671+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.418+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 msgr2=0x7ff88c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.418+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 0x7ff88c06ef20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006fd0 tx=0x7ff8a0008040 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.418+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 msgr2=0x7ff8ac0aebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.418+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4007e40 tx=0x7ff8a400e4b0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 shutdown_connections 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff88c06ca60 0x7ff88c06ef20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac0cd020 0x7ff8ac0aebe0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ac071d10 0x7ff8ac0ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 --2- 172.21.15.67:0/1134372771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ac06f670 0x7ff8ac146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 >> 172.21.15.67:0/1134372771 conn(0x7ff8ac06bf50 msgr2=0x7ff8ac0aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:10.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 shutdown_connections 2024-04-17T07:34:10.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.419+0000 7ff8b3c00640 1 -- 172.21.15.67:0/1134372771 wait complete. 2024-04-17T07:34:10.436 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T07:34:10.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T07:34:10.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.650+0000 7f354b200640 1 Processor -- start 2024-04-17T07:34:10.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.651+0000 7f354b200640 1 -- start start 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.652+0000 7f354b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 0x7f354c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.652+0000 7f354b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f354c0b3b50 0x7f354c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.652+0000 7f354b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f354c0d4f70 con 0x7f354c06cf80 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.652+0000 7f354b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f354c0d5250 con 0x7f354c0b3b50 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.652+0000 7f354b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f354c0cf4d0 con 0x7f354c15ec20 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f354c0b3b50 0x7f354c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f354a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f354c06cf80 0x7f354c0cfcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50194/0 (socket says 172.21.15.67:50194) 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f354a800640 1 -- 172.21.15.67:0/1918106174 learned_addr learned my addr 172.21.15.67:0/1918106174 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549e00640 1 --2- 172.21.15.67:0/1918106174 >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 0x7f354c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549400640 1 -- 172.21.15.67:0/1918106174 >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 msgr2=0x7f354c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549400640 1 --2- 172.21.15.67:0/1918106174 >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 0x7f354c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549400640 1 -- 172.21.15.67:0/1918106174 >> v1:172.21.15.67:6789/0 conn(0x7f354c06cf80 legacy=0x7f354c0cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549400640 1 -- 172.21.15.67:0/1918106174 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c0cf7b0 con 0x7f354c0b3b50 2024-04-17T07:34:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.653+0000 7f3549e00640 1 --2- 172.21.15.67:0/1918106174 >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 0x7f354c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.654+0000 7f3549400640 1 --2- 172.21.15.67:0/1918106174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c0b3b50 0x7f354c06ca40 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f354c08ca90 tx=0x7f353402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3794e785c3853b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f3548a00640 1 -- 172.21.15.67:0/1918106174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534038070 con 0x7f354c0b3b50 2024-04-17T07:34:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f3548a00640 1 -- 172.21.15.67:0/1918106174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3534031700 con 0x7f354c0b3b50 2024-04-17T07:34:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f3548a00640 1 -- 172.21.15.67:0/1918106174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534031ad0 con 0x7f354c0b3b50 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f354b200640 1 -- 172.21.15.67:0/1918106174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c0b3b50 msgr2=0x7f354c06ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f354b200640 1 --2- 172.21.15.67:0/1918106174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c0b3b50 0x7f354c06ca40 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f354c08ca90 tx=0x7f353402b7a0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.655+0000 7f354b200640 1 -- 172.21.15.67:0/1918106174 shutdown_connections 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.656+0000 7f354b200640 1 --2- 172.21.15.67:0/1918106174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c0b3b50 0x7f354c06ca40 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.656+0000 7f354b200640 1 --2- 172.21.15.67:0/1918106174 >> v2:172.21.15.187:3300/0 conn(0x7f354c15ec20 0x7f354c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.656+0000 7f354b200640 1 -- 172.21.15.67:0/1918106174 >> 172.21.15.67:0/1918106174 conn(0x7f354c08bff0 msgr2=0x7f354c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.656+0000 7f354b200640 1 -- 172.21.15.67:0/1918106174 shutdown_connections 2024-04-17T07:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.656+0000 7f354b200640 1 -- 172.21.15.67:0/1918106174 wait complete. 2024-04-17T07:34:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.658+0000 7f354b200640 1 Processor -- start 2024-04-17T07:34:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.658+0000 7f354b200640 1 -- start start 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f354c0b3b50 0x7f354c160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f354c15ec20 0x7f354c159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f354c0d0560 con 0x7f354c06cf80 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f354c0d03e0 con 0x7f354c15ec20 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f354b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f354c0d10d0 con 0x7f354c0b3b50 2024-04-17T07:34:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f3549e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f3549e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39106/0 (socket says 172.21.15.67:39106) 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.659+0000 7f3549e00640 1 -- 172.21.15.67:0/3837346810 learned_addr learned my addr 172.21.15.67:0/3837346810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 -- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f354c0b3b50 msgr2=0x7f354c160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f354c0b3b50 0x7f354c160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 -- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f354c15ec20 msgr2=0x7f354c159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f354c15ec20 0x7f354c159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 -- 172.21.15.67:0/3837346810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c160c20 con 0x7f354c06cf80 2024-04-17T07:34:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3549e00640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f353c00d7d0 tx=0x7f353c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c004030 con 0x7f354c06cf80 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353c0041d0 con 0x7f354c06cf80 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0096a0 con 0x7f354c06cf80 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.660+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354c160f00 con 0x7f354c06cf80 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.661+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354c1614c0 con 0x7f354c06cf80 2024-04-17T07:34:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.662+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354c068a10 con 0x7f354c06cf80 2024-04-17T07:34:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.663+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f353c009840 con 0x7f354c06cf80 2024-04-17T07:34:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.664+0000 7f3542000640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 0x7f352806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.664+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4885+0+0 (secure 0 0 0) 0x7f353c08bcd0 con 0x7f354c06cf80 2024-04-17T07:34:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.664+0000 7f3549400640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 0x7f352806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.666+0000 7f3549400640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 0x7f352806eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3534035000 tx=0x7f3534030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.668+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f353c05a310 con 0x7f354c06cf80 2024-04-17T07:34:10.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.864+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f354c0d55f0 con 0x7f354c06cf80 2024-04-17T07:34:10.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.865+0000 7f3542000640 1 -- 172.21.15.67:0/3837346810 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) ==== 93+0+10 (secure 0 0 0) 0x7f353c059c10 con 0x7f354c06cf80 2024-04-17T07:34:10.865 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T07:34:10.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 msgr2=0x7f352806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 0x7f352806eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3534035000 tx=0x7f3534030000 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 msgr2=0x7f354c159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f353c00d7d0 tx=0x7f353c00dc90 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 shutdown_connections 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f352806ca10 0x7f352806eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f354c15ec20 0x7f354c159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f354c0b3b50 0x7f354c160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 --2- 172.21.15.67:0/3837346810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f354c06cf80 0x7f354c159e90 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 >> 172.21.15.67:0/3837346810 conn(0x7f354c08bff0 msgr2=0x7f354c15ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 shutdown_connections 2024-04-17T07:34:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:10.867+0000 7f354b200640 1 -- 172.21.15.67:0/3837346810 wait complete. 2024-04-17T07:34:10.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T07:34:11.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.111+0000 7f72a2200640 1 Processor -- start 2024-04-17T07:34:11.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.112+0000 7f72a2200640 1 -- start start 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.113+0000 7f72a2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c06b8c0 0x7f729c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.113+0000 7f72a2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f729c160280 0x7f729c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.113+0000 7f72a2200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c0cd4a0 con 0x7f729c141ed0 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.113+0000 7f72a2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f729c15ce30 con 0x7f729c160280 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.113+0000 7f72a2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f729c1607f0 con 0x7f729c06b8c0 2024-04-17T07:34:11.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f7293e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f729c160280 0x7f729c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f72a1800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f729c141ed0 0x7f729c15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50206/0 (socket says 172.21.15.67:50206) 2024-04-17T07:34:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f72a1800640 1 -- 172.21.15.67:0/213414144 learned_addr learned my addr 172.21.15.67:0/213414144 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f72a0e00640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c06b8c0 0x7f729c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f7293e00640 1 -- 172.21.15.67:0/213414144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c06b8c0 msgr2=0x7f729c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:11.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f7293e00640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c06b8c0 0x7f729c0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f7293e00640 1 -- 172.21.15.67:0/213414144 >> v1:172.21.15.67:6789/0 conn(0x7f729c141ed0 legacy=0x7f729c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.114+0000 7f7293e00640 1 -- 172.21.15.67:0/213414144 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c160ad0 con 0x7f729c160280 2024-04-17T07:34:11.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.115+0000 7f7293e00640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c160280 0x7f729c06bca0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f72840097a0 tx=0x7f728402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7483fab5379a135 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f729be00640 1 -- 172.21.15.67:0/213414144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284035020 con 0x7f729c160280 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f729be00640 1 -- 172.21.15.67:0/213414144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284031810 con 0x7f729c160280 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f729be00640 1 -- 172.21.15.67:0/213414144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284031be0 con 0x7f729c160280 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f72a2200640 1 -- 172.21.15.67:0/213414144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c160280 msgr2=0x7f729c06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f72a2200640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c160280 0x7f729c06bca0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f72840097a0 tx=0x7f728402b8b0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f72a2200640 1 -- 172.21.15.67:0/213414144 shutdown_connections 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.116+0000 7f72a2200640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c160280 0x7f729c06bca0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.117+0000 7f72a2200640 1 --2- 172.21.15.67:0/213414144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c06b8c0 0x7f729c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.117+0000 7f72a2200640 1 -- 172.21.15.67:0/213414144 >> 172.21.15.67:0/213414144 conn(0x7f729c067d10 msgr2=0x7f729c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:11.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.117+0000 7f72a2200640 1 -- 172.21.15.67:0/213414144 shutdown_connections 2024-04-17T07:34:11.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.117+0000 7f72a2200640 1 -- 172.21.15.67:0/213414144 wait complete. 2024-04-17T07:34:11.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.118+0000 7f72a2200640 1 Processor -- start 2024-04-17T07:34:11.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.119+0000 7f72a2200640 1 -- start start 2024-04-17T07:34:11.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.119+0000 7f72a2200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c06b8c0 0x7f729c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.119+0000 7f72a2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c160280 0x7f729c13ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f729c06c1e0 con 0x7f729c141ed0 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a2200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f729c0cddf0 con 0x7f729c06b8c0 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f729c161090 con 0x7f729c160280 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39122/0 (socket says 172.21.15.67:39122) 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 -- 172.21.15.67:0/606646739 learned_addr learned my addr 172.21.15.67:0/606646739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a1800640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c160280 0x7f729c13ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f72a0e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c06b8c0 0x7f729c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 -- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c160280 msgr2=0x7f729c13ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c160280 0x7f729c13ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 -- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c06b8c0 msgr2=0x7f729c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c06b8c0 0x7f729c15ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.120+0000 7f7293e00640 1 -- 172.21.15.67:0/606646739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c15f600 con 0x7f729c141ed0 2024-04-17T07:34:11.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f7293e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f72840098c0 tx=0x7f72840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72840443f0 con 0x7f729c141ed0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284044590 con 0x7f729c141ed0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284044980 con 0x7f729c141ed0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729c15f8c0 con 0x7f729c141ed0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.121+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c0b01a0 con 0x7f729c141ed0 2024-04-17T07:34:11.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.122+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729c0cd0a0 con 0x7f729c141ed0 2024-04-17T07:34:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.123+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f728404b060 con 0x7f729c141ed0 2024-04-17T07:34:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.124+0000 7f7299600640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 0x7f727806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.124+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4885+0+0 (secure 0 0 0) 0x7f72840adad0 con 0x7f729c141ed0 2024-04-17T07:34:11.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.125+0000 7f72a0e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 0x7f727806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.126+0000 7f72a0e00640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 0x7f727806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.129+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f729c0cd0a0 con 0x7f729c141ed0 2024-04-17T07:34:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.340+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f729c068a10 con 0x7f729c141ed0 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.341+0000 7f7299600640 1 -- 172.21.15.67:0/606646739 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) ==== 84+0+65 (secure 0 0 0) 0x7f728403f070 con 0x7f729c141ed0 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:34:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 msgr2=0x7f727806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 0x7f727806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).stop 2024-04-17T07:34:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 msgr2=0x7f729c13a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f72840098c0 tx=0x7f72840046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 shutdown_connections 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f727806ca40 0x7f727806ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f729c160280 0x7f729c13ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f729c141ed0 0x7f729c13a710 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 --2- 172.21.15.67:0/606646739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f729c06b8c0 0x7f729c15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 >> 172.21.15.67:0/606646739 conn(0x7f729c067d10 msgr2=0x7f729c1400b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 shutdown_connections 2024-04-17T07:34:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.344+0000 7f72a2200640 1 -- 172.21.15.67:0/606646739 wait complete. 2024-04-17T07:34:11.363 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T07:34:11.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T07:34:11.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.582+0000 7f4d8d000640 1 Processor -- start 2024-04-17T07:34:11.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.583+0000 7f4d8d000640 1 -- start start 2024-04-17T07:34:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.584+0000 7f4d8d000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4d880cd6d0 0x7f4d88145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.584+0000 7f4d8d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d88160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.584+0000 7f4d8d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d88161000 con 0x7f4d880cd2f0 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.584+0000 7f4d8d000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4d880cfa60 con 0x7f4d880cd6d0 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.584+0000 7f4d8d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d88146530 con 0x7f4d88146080 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d87400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d880cd2f0 0x7f4d880cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50208/0 (socket says 172.21.15.67:50208) 2024-04-17T07:34:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d87400640 1 -- 172.21.15.67:0/3523215104 learned_addr learned my addr 172.21.15.67:0/3523215104 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272600497 0 0) 0x7f4d88161000 con 0x7f4d880cd2f0 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d87e00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d88160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d6c003520 con 0x7f4d880cd2f0 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.585+0000 7f4d7fe00640 1 --2- 172.21.15.67:0/3523215104 >> v2:172.21.15.187:3300/0 conn(0x7f4d880cd6d0 0x7f4d88145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1150255146 0 0) 0x7f4d6c003520 con 0x7f4d880cd2f0 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 msgr2=0x7f4d88160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d88160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 >> v2:172.21.15.187:3300/0 conn(0x7f4d880cd6d0 msgr2=0x7f4d88145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 --2- 172.21.15.67:0/3523215104 >> v2:172.21.15.187:3300/0 conn(0x7f4d880cd6d0 0x7f4d88145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d880ce870 con 0x7f4d880cd2f0 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4d78001670 con 0x7f4d880cd2f0 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 >> v1:172.21.15.67:6789/0 conn(0x7f4d880cd2f0 legacy=0x7f4d880cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d6c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d6c003f30 0x7f4d6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6c004d40 0x7f4d6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d880cfa60 con 0x7f4d6c004d40 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88146530 con 0x7f4d6c003f30 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c00ba10 con 0x7f4d88146080 2024-04-17T07:34:11.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d7fe00640 1 --2- 172.21.15.67:0/3523215104 >> v2:172.21.15.187:3300/0 conn(0x7f4d880cd6d0 0x7f4d88145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d87e00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6c004d40 0x7f4d6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.586+0000 7f4d87400640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d6c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.587+0000 7f4d87e00640 1 -- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 msgr2=0x7f4d6c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.587+0000 7f4d87e00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d88146080 0x7f4d6c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.587+0000 7f4d87e00640 1 -- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d6c003f30 msgr2=0x7f4d6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.587+0000 7f4d87e00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d6c003f30 0x7f4d6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.587+0000 7f4d87e00640 1 -- 172.21.15.67:0/3523215104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c003520 con 0x7f4d6c004d40 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.588+0000 7f4d87e00640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6c004d40 0x7f4d6c0051c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009fc0 tx=0x7f4d6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7018ee4b295461b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.588+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80039070 con 0x7f4d6c004d40 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.588+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8002cc20 con 0x7f4d6c004d40 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.588+0000 7f4d86a00640 1 -- 172.21.15.67:0/3523215104 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80009c10 con 0x7f4d6c004d40 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 -- 172.21.15.67:0/3523215104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6c004d40 msgr2=0x7f4d6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 --2- 172.21.15.67:0/3523215104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6c004d40 0x7f4d6c0051c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009fc0 tx=0x7f4d6c005830 comp rx=0 tx=0).stop 2024-04-17T07:34:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d87e00640 1 -- 172.21.15.67:0/3523215104 reap_dead start 2024-04-17T07:34:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 -- 172.21.15.67:0/3523215104 shutdown_connections 2024-04-17T07:34:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 -- 172.21.15.67:0/3523215104 >> 172.21.15.67:0/3523215104 conn(0x7f4d8806b8c0 msgr2=0x7f4d8806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 -- 172.21.15.67:0/3523215104 shutdown_connections 2024-04-17T07:34:11.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.589+0000 7f4d8d000640 1 -- 172.21.15.67:0/3523215104 wait complete. 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.591+0000 7f4d8d000640 1 Processor -- start 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.591+0000 7f4d8d000640 1 -- start start 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d881f8080 0x7f4d8815f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d8815d4e0 0x7f4d880b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88160360 con 0x7f4d881f8460 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4d880cee30 con 0x7f4d881f8080 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d8d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4d881604e0 con 0x7f4d8815d4e0 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39146/0 (socket says 172.21.15.67:39146) 2024-04-17T07:34:11.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 -- 172.21.15.67:0/1083986554 learned_addr learned my addr 172.21.15.67:0/1083986554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 -- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d8815d4e0 msgr2=0x7f4d880b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d8815d4e0 0x7f4d880b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 -- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d881f8080 msgr2=0x7f4d8815f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d881f8080 0x7f4d8815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.592+0000 7f4d7fe00640 1 -- 172.21.15.67:0/1083986554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8813a970 con 0x7f4d881f8460 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d7fe00640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e5c0 tx=0x7f4d7000ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70017070 con 0x7f4d881f8460 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d70013610 con 0x7f4d881f8460 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7000f040 con 0x7f4d881f8460 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8813ac50 con 0x7f4d881f8460 2024-04-17T07:34:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.593+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8813af30 con 0x7f4d881f8460 2024-04-17T07:34:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.594+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d88068a10 con 0x7f4d881f8460 2024-04-17T07:34:11.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.595+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f4d7001a070 con 0x7f4d881f8460 2024-04-17T07:34:11.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.596+0000 7f4d7f400640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 0x7f4d6406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:11.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.596+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4885+0+0 (secure 0 0 0) 0x7f4d7008b5e0 con 0x7f4d881f8460 2024-04-17T07:34:11.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.597+0000 7f4d87400640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 0x7f4d6406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:11.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.598+0000 7f4d87400640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 0x7f4d6406eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d78001050 tx=0x7f4d78000fe0 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:11.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.600+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4d70059b90 con 0x7f4d881f8460 2024-04-17T07:34:11.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:11.808+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4d88068610 con 0x7f4d881f8460 2024-04-17T07:34:12.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.436+0000 7f4d7f400640 1 -- 172.21.15.67:0/1083986554 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v153) ==== 138+0+0 (secure 0 0 0) 0x7f4d70059490 con 0x7f4d881f8460 2024-04-17T07:34:12.436 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T07:34:12.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 msgr2=0x7f4d6406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 0x7f4d6406eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d78001050 tx=0x7f4d78000fe0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 msgr2=0x7f4d8815cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000e5c0 tx=0x7f4d7000ea80 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 shutdown_connections 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f4d6406ca10 0x7f4d6406eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4d8815d4e0 0x7f4d880b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d881f8460 0x7f4d8815cfa0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 --2- 172.21.15.67:0/1083986554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d881f8080 0x7f4d8815f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 >> 172.21.15.67:0/1083986554 conn(0x7f4d8806b8c0 msgr2=0x7f4d88074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 shutdown_connections 2024-04-17T07:34:12.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.441+0000 7f4d8d000640 1 -- 172.21.15.67:0/1083986554 wait complete. 2024-04-17T07:34:12.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T07:34:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.681+0000 7f7a00a00640 1 Processor -- start 2024-04-17T07:34:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.681+0000 7f7a00a00640 1 -- start start 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f7a00a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79fc071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f7a00a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79fc15cbe0 con 0x7f79fc0b27c0 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f7a00a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79fc15cea0 con 0x7f79fc0cd5d0 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f7a00a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79fc15d180 con 0x7f79fc13f6d0 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f79fb200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79fc0b27c0 0x7f79fc142020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50210/0 (socket says 172.21.15.67:50210) 2024-04-17T07:34:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.683+0000 7f79fb200640 1 -- 172.21.15.67:0/2020338383 learned_addr learned my addr 172.21.15.67:0/2020338383 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79fa800640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79fc071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437489088 0 0) 0x7f79fc15cbe0 con 0x7f79fc0b27c0 2024-04-17T07:34:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79e0003520 con 0x7f79fc0b27c0 2024-04-17T07:34:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 661227073 0 0) 0x7f79e0003520 con 0x7f79fc0b27c0 2024-04-17T07:34:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 msgr2=0x7f79fc071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79fc071d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 >> v1:172.21.15.187:6789/0 conn(0x7f79fc0cd5d0 legacy=0x7f79fc0b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc15d460 con 0x7f79fc0b27c0 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f79f40019f0 con 0x7f79fc0b27c0 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 >> v1:172.21.15.67:6789/0 conn(0x7f79fc0b27c0 legacy=0x7f79fc142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79e00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e0003f60 0x7f79e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0004d40 0x7f79e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc15d180 con 0x7f79e0003f60 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f79e000baa0 con 0x7f79e0004d40 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.684+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79e000bca0 con 0x7f79fc13f6d0 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e0003f60 0x7f79e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79fa800640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79e00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79fb200640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0004d40 0x7f79e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 -- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 msgr2=0x7f79e00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc13f6d0 0x7f79e00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 -- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0004d40 msgr2=0x7f79e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0004d40 0x7f79e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.685+0000 7f79f9e00640 1 -- 172.21.15.67:0/2020338383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e0003520 con 0x7f79e0003f60 2024-04-17T07:34:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.687+0000 7f79f9e00640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e0003f60 0x7f79e0004800 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f79e8001750 tx=0x7f79e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fd90b2525a9610 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.687+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8001370 con 0x7f79e0003f60 2024-04-17T07:34:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.687+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e80052d0 con 0x7f79e0003f60 2024-04-17T07:34:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.687+0000 7f79f9400640 1 -- 172.21.15.67:0/2020338383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8003830 con 0x7f79e0003f60 2024-04-17T07:34:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 -- 172.21.15.67:0/2020338383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e0003f60 msgr2=0x7f79e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 --2- 172.21.15.67:0/2020338383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e0003f60 0x7f79e0004800 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f79e8001750 tx=0x7f79e0005830 comp rx=0 tx=0).stop 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f79fb200640 1 -- 172.21.15.67:0/2020338383 reap_dead start 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 -- 172.21.15.67:0/2020338383 shutdown_connections 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 -- 172.21.15.67:0/2020338383 >> 172.21.15.67:0/2020338383 conn(0x7f79fc067d10 msgr2=0x7f79fc0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 -- 172.21.15.67:0/2020338383 shutdown_connections 2024-04-17T07:34:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.688+0000 7f7a00a00640 1 -- 172.21.15.67:0/2020338383 wait complete. 2024-04-17T07:34:12.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.690+0000 7f7a00a00640 1 Processor -- start 2024-04-17T07:34:12.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.690+0000 7f7a00a00640 1 -- start start 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0003f60 0x7f79fc13e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc0b1750 0x7f79fc144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc15da20 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc0d1120 con 0x7f79e0003f60 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f7a00a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79fc0cda30 con 0x7f79fc0b1750 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f79f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f79f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39172/0 (socket says 172.21.15.67:39172) 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.691+0000 7f79f9e00640 1 -- 172.21.15.67:0/1064352189 learned_addr learned my addr 172.21.15.67:0/1064352189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79fa800640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0003f60 0x7f79fc13e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79fb200640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc0b1750 0x7f79fc144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 -- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc0b1750 msgr2=0x7f79fc144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc0b1750 0x7f79fc144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 -- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0003f60 msgr2=0x7f79fc13e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0003f60 0x7f79fc13e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:12.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 -- 172.21.15.67:0/1064352189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc145640 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.692+0000 7f79f9e00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f79e8008fd0 tx=0x7f79e8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.693+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8041070 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.693+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e8005480 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.693+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8001370 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.693+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79fc145920 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.693+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79fc142b00 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.694+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79fc0b1b60 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.696+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f79e8044070 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.697+0000 7f79f2a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 0x7f79dc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:12.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.697+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5255+0+0 (secure 0 0 0) 0x7f79e80ad9d0 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.697+0000 7f79fa800640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 0x7f79dc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:12.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.699+0000 7f79fa800640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 0x7f79dc06ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009b00 tx=0x7f79ec008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.701+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f79fc0b1b60 con 0x7f79fc1f0ed0 2024-04-17T07:34:12.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:12.908+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f79fc068a10 con 0x7f79fc1f0ed0 2024-04-17T07:34:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.442+0000 7f79f2a00640 1 -- 172.21.15.67:0/1064352189 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v154) ==== 170+0+0 (secure 0 0 0) 0x7f79e80052a0 con 0x7f79fc1f0ed0 2024-04-17T07:34:13.443 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T07:34:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.448+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 msgr2=0x7f79dc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.448+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 0x7f79dc06ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009b00 tx=0x7f79ec008040 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.448+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 msgr2=0x7f79fc0b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.448+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f79e8008fd0 tx=0x7f79e8001120 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 shutdown_connections 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f79dc06ca40 0x7f79dc06ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79fc0b1750 0x7f79fc144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79fc1f0ed0 0x7f79fc0b1210 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 --2- 172.21.15.67:0/1064352189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e0003f60 0x7f79fc13e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 >> 172.21.15.67:0/1064352189 conn(0x7f79fc067d10 msgr2=0x7f79fc15f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 shutdown_connections 2024-04-17T07:34:13.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.449+0000 7f7a00a00640 1 -- 172.21.15.67:0/1064352189 wait complete. 2024-04-17T07:34:13.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T07:34:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.676+0000 7f0d03e00640 1 Processor -- start 2024-04-17T07:34:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.676+0000 7f0d03e00640 1 -- start start 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d04145f80 0x7f0d040cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 0x7f0d0406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0d0406c0a0 0x7f0d04160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0d040727a0 con 0x7f0d04145f80 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0d04146360 con 0x7f0d0406c0a0 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.677+0000 7f0d03e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d04071e70 con 0x7f0d040cd750 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d04145f80 0x7f0d040cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d04145f80 0x7f0d040cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60634/0 (socket says 172.21.15.67:60634) 2024-04-17T07:34:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 -- 172.21.15.67:0/488756210 learned_addr learned my addr 172.21.15.67:0/488756210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d03400640 1 --2- 172.21.15.67:0/488756210 >> v2:172.21.15.187:3300/0 conn(0x7f0d0406c0a0 0x7f0d04160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02000640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 0x7f0d0406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 -- 172.21.15.67:0/488756210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 msgr2=0x7f0d0406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 0x7f0d0406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 -- 172.21.15.67:0/488756210 >> v2:172.21.15.187:3300/0 conn(0x7f0d0406c0a0 msgr2=0x7f0d04160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 --2- 172.21.15.67:0/488756210 >> v2:172.21.15.187:3300/0 conn(0x7f0d0406c0a0 0x7f0d04160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02a00640 1 -- 172.21.15.67:0/488756210 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d04072150 con 0x7f0d04145f80 2024-04-17T07:34:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.678+0000 7f0d02000640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 0x7f0d0406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.680+0000 7f0d02a00640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d04145f80 0x7f0d040cd210 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f0d0406f7a0 tx=0x7f0cf402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612a6b48cb2ac85e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.680+0000 7f0d01600640 1 -- 172.21.15.67:0/488756210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf402e070 con 0x7f0d04145f80 2024-04-17T07:34:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.680+0000 7f0d01600640 1 -- 172.21.15.67:0/488756210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf4037070 con 0x7f0d04145f80 2024-04-17T07:34:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.680+0000 7f0d01600640 1 -- 172.21.15.67:0/488756210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4031a00 con 0x7f0d04145f80 2024-04-17T07:34:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 -- 172.21.15.67:0/488756210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d04145f80 msgr2=0x7f0d040cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d04145f80 0x7f0d040cd210 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f0d0406f7a0 tx=0x7f0cf402b880 comp rx=0 tx=0).stop 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 -- 172.21.15.67:0/488756210 shutdown_connections 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 --2- 172.21.15.67:0/488756210 >> v2:172.21.15.187:3300/0 conn(0x7f0d0406c0a0 0x7f0d04160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d040cd750 0x7f0d0406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 --2- 172.21.15.67:0/488756210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d04145f80 0x7f0d040cd210 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 -- 172.21.15.67:0/488756210 >> 172.21.15.67:0/488756210 conn(0x7f0d040b2190 msgr2=0x7f0d040b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 -- 172.21.15.67:0/488756210 shutdown_connections 2024-04-17T07:34:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.681+0000 7f0d03e00640 1 -- 172.21.15.67:0/488756210 wait complete. 2024-04-17T07:34:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.683+0000 7f0d03e00640 1 Processor -- start 2024-04-17T07:34:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.683+0000 7f0d03e00640 1 -- start start 2024-04-17T07:34:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d040cd750 0x7f0d0415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d04145f80 0x7f0d04144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d04072db0 con 0x7f0d0406c0a0 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0d04072c30 con 0x7f0d040cd750 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.684+0000 7f0d03e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d0406c4f0 con 0x7f0d04145f80 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60650/0 (socket says 172.21.15.67:60650) 2024-04-17T07:34:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 -- 172.21.15.67:0/2143707058 learned_addr learned my addr 172.21.15.67:0/2143707058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:13.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d03400640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d04145f80 0x7f0d04144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02000640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d040cd750 0x7f0d0415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 -- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d04145f80 msgr2=0x7f0d04144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d04145f80 0x7f0d04144590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 -- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d040cd750 msgr2=0x7f0d0415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d040cd750 0x7f0d0415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.685+0000 7f0d02a00640 1 -- 172.21.15.67:0/2143707058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d041452f0 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0d02a00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000c00 tx=0x7f0cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4032670 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf4032850 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4032c40 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d04136740 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.686+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d04140110 con 0x7f0d0406c0a0 2024-04-17T07:34:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.687+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d04068a10 con 0x7f0d0406c0a0 2024-04-17T07:34:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.688+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f0cf40453f0 con 0x7f0d0406c0a0 2024-04-17T07:34:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.689+0000 7f0cfb400640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 0x7f0ce406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.689+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5637+0+0 (secure 0 0 0) 0x7f0cf40adbc0 con 0x7f0d0406c0a0 2024-04-17T07:34:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.690+0000 7f0d02000640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 0x7f0ce406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:13.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.691+0000 7f0d02000640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 0x7f0ce406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d040d1380 tx=0x7f0cf0008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:13.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.695+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0cf4040070 con 0x7f0d0406c0a0 2024-04-17T07:34:13.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:13.941+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0d0406ccd0 con 0x7f0d0406c0a0 2024-04-17T07:34:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.461+0000 7f0cfb400640 1 -- 172.21.15.67:0/2143707058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v72) ==== 174+0+0 (secure 0 0 0) 0x7f0cf407be00 con 0x7f0d0406c0a0 2024-04-17T07:34:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 18 and data pool 19 2024-04-17T07:34:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.466+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 msgr2=0x7f0ce406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.466+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 0x7f0ce406eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d040d1380 tx=0x7f0cf0008040 comp rx=0 tx=0).stop 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.466+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 msgr2=0x7f0d0415bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.466+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000c00 tx=0x7f0cf40046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 shutdown_connections 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0ce406ca10 0x7f0ce406eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d04145f80 0x7f0d04144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d040cd750 0x7f0d0415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 --2- 172.21.15.67:0/2143707058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0406c0a0 0x7f0d0415bdd0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 >> 172.21.15.67:0/2143707058 conn(0x7f0d040b2190 msgr2=0x7f0d0413eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 shutdown_connections 2024-04-17T07:34:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.467+0000 7f0d03e00640 1 -- 172.21.15.67:0/2143707058 wait complete. 2024-04-17T07:34:14.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:34:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.699+0000 7ff6dca00640 1 Processor -- start 2024-04-17T07:34:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.699+0000 7ff6dca00640 1 -- start start 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.700+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.700+0000 7ff6dca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6d8141ed0 0x7ff6d8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.700+0000 7ff6dca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d80cd4a0 con 0x7ff6d806b8c0 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.700+0000 7ff6dca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d815ce30 con 0x7ff6d8141ed0 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.700+0000 7ff6dca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d81607f0 con 0x7ff6d8160280 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6d8141ed0 0x7ff6d8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6d8141ed0 0x7ff6d8142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60662/0 (socket says 172.21.15.67:60662) 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 -- 172.21.15.67:0/1342464075 learned_addr learned my addr 172.21.15.67:0/1342464075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6cfe00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 -- 172.21.15.67:0/1342464075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 msgr2=0x7ff6d806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 -- 172.21.15.67:0/1342464075 >> v1:172.21.15.187:6789/0 conn(0x7ff6d806b8c0 legacy=0x7ff6d80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.701+0000 7ff6d7e00640 1 -- 172.21.15.67:0/1342464075 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d8160ad0 con 0x7ff6d8141ed0 2024-04-17T07:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.703+0000 7ff6d7e00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d8141ed0 0x7ff6d8142360 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e50 tx=0x7ff6d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af748da95c4e6af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.703+0000 7ff6d6a00640 1 -- 172.21.15.67:0/1342464075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d002ca70 con 0x7ff6d8141ed0 2024-04-17T07:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.703+0000 7ff6d6a00640 1 -- 172.21.15.67:0/1342464075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d002cc10 con 0x7ff6d8141ed0 2024-04-17T07:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.703+0000 7ff6d6a00640 1 -- 172.21.15.67:0/1342464075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d00343f0 con 0x7ff6d8141ed0 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 -- 172.21.15.67:0/1342464075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d8141ed0 msgr2=0x7ff6d8142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d8141ed0 0x7ff6d8142360 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009e50 tx=0x7ff6d002bed0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 -- 172.21.15.67:0/1342464075 shutdown_connections 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d8141ed0 0x7ff6d8142360 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 --2- 172.21.15.67:0/1342464075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 -- 172.21.15.67:0/1342464075 >> 172.21.15.67:0/1342464075 conn(0x7ff6d8067d10 msgr2=0x7ff6d80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 -- 172.21.15.67:0/1342464075 shutdown_connections 2024-04-17T07:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.704+0000 7ff6dca00640 1 -- 172.21.15.67:0/1342464075 wait complete. 2024-04-17T07:34:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.706+0000 7ff6dca00640 1 Processor -- start 2024-04-17T07:34:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.706+0000 7ff6dca00640 1 -- start start 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6d8141ed0 0x7ff6d806f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d8145a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8161090 con 0x7ff6d806b8c0 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d806c1e0 con 0x7ff6d8141ed0 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6dca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d813f9b0 con 0x7ff6d8160280 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60676/0 (socket says 172.21.15.67:60676) 2024-04-17T07:34:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 -- 172.21.15.67:0/889963633 learned_addr learned my addr 172.21.15.67:0/889963633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7e00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d8145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 -- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 msgr2=0x7ff6d8145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d8145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 -- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6d8141ed0 msgr2=0x7ff6d806f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6d8141ed0 0x7ff6d806f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.708+0000 7ff6d7400640 1 -- 172.21.15.67:0/889963633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d806e620 con 0x7ff6d806b8c0 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6d7400640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c430 tx=0x7ff6c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8011790 con 0x7ff6d806b8c0 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c8017070 con 0x7ff6d806b8c0 2024-04-17T07:34:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8011bb0 con 0x7ff6d806b8c0 2024-04-17T07:34:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d806e900 con 0x7ff6d806b8c0 2024-04-17T07:34:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.709+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d806eec0 con 0x7ff6d806b8c0 2024-04-17T07:34:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.710+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d8068a10 con 0x7ff6d806b8c0 2024-04-17T07:34:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.711+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7ff6c801c460 con 0x7ff6d806b8c0 2024-04-17T07:34:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.712+0000 7ff6cf400640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 0x7ff6b406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.713+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 5737+0+0 (secure 0 0 0) 0x7ff6c808bb70 con 0x7ff6d806b8c0 2024-04-17T07:34:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.713+0000 7ff6cfe00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 0x7ff6b406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:14.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.715+0000 7ff6cfe00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 0x7ff6b406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0006fd0 tx=0x7ff6c0008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.717+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff6c8059e70 con 0x7ff6d806b8c0 2024-04-17T07:34:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.918+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6d80cf670 con 0x7ff6d806b8c0 2024-04-17T07:34:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.921+0000 7ff6cf400640 1 -- 172.21.15.67:0/889963633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16688 (secure 0 0 0) 0x7ff6c8059770 con 0x7ff6d806b8c0 2024-04-17T07:34:14.921 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:14.922 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":155,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:34:13.943601+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:34:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 msgr2=0x7ff6b406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 0x7ff6b406ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0006fd0 tx=0x7ff6c0008040 comp rx=0 tx=0).stop 2024-04-17T07:34:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 msgr2=0x7ff6d8155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c430 tx=0x7ff6c800c8f0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 shutdown_connections 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7ff6b406ca40 0x7ff6b406ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6d8160280 0x7ff6d8145a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6d8141ed0 0x7ff6d806f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 --2- 172.21.15.67:0/889963633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d806b8c0 0x7ff6d8155e30 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 >> 172.21.15.67:0/889963633 conn(0x7ff6d8067d10 msgr2=0x7ff6d80d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 shutdown_connections 2024-04-17T07:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:14.924+0000 7ff6dca00640 1 -- 172.21.15.67:0/889963633 wait complete. 2024-04-17T07:34:14.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:34:15.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.162+0000 7f7659000640 1 Processor -- start 2024-04-17T07:34:15.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.162+0000 7f7659000640 1 -- start start 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.163+0000 7f7659000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540b9f00 0x7f76540b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.163+0000 7f7659000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76540cf8f0 0x7f76541634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.163+0000 7f7659000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7654163a20 con 0x7f76540d12a0 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.163+0000 7f7659000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f76540cf640 con 0x7f76540cf8f0 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.163+0000 7f7659000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76540b84c0 con 0x7f76540b9f00 2024-04-17T07:34:15.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76540cf8f0 0x7f76541634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76540cf8f0 0x7f76541634e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60684/0 (socket says 172.21.15.67:60684) 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 -- 172.21.15.67:0/1280797477 learned_addr learned my addr 172.21.15.67:0/1280797477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7653400640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540b9f00 0x7f76540b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 -- 172.21.15.67:0/1280797477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540b9f00 msgr2=0x7f76540b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540b9f00 0x7f76540b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 -- 172.21.15.67:0/1280797477 >> v1:172.21.15.187:6789/0 conn(0x7f76540d12a0 legacy=0x7f76540d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.164+0000 7f7652a00640 1 -- 172.21.15.67:0/1280797477 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76540cfe20 con 0x7f76540cf8f0 2024-04-17T07:34:15.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.165+0000 7f7652a00640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541634e0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f764002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d32197e3393979 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7652000640 1 -- 172.21.15.67:0/1280797477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400317c0 con 0x7f76540cf8f0 2024-04-17T07:34:15.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7652000640 1 -- 172.21.15.67:0/1280797477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7640037070 con 0x7f76540cf8f0 2024-04-17T07:34:15.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7652000640 1 -- 172.21.15.67:0/1280797477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640031b80 con 0x7f76540cf8f0 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 -- 172.21.15.67:0/1280797477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 msgr2=0x7f76541634e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541634e0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f764002b860 comp rx=0 tx=0).stop 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 -- 172.21.15.67:0/1280797477 shutdown_connections 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541634e0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 --2- 172.21.15.67:0/1280797477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540b9f00 0x7f76540b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.166+0000 7f7659000640 1 -- 172.21.15.67:0/1280797477 >> 172.21.15.67:0/1280797477 conn(0x7f765406cb60 msgr2=0x7f765406cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.167+0000 7f7659000640 1 -- 172.21.15.67:0/1280797477 shutdown_connections 2024-04-17T07:34:15.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.167+0000 7f7659000640 1 -- 172.21.15.67:0/1280797477 wait complete. 2024-04-17T07:34:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.168+0000 7f7659000640 1 Processor -- start 2024-04-17T07:34:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.169+0000 7f7659000640 1 -- start start 2024-04-17T07:34:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.169+0000 7f7659000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76540b9f00 0x7f765415f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.169+0000 7f7659000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7659000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540d12a0 0x7f76541f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7659000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76540d3420 con 0x7f76540cf8f0 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7659000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f76540d3140 con 0x7f76540b9f00 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7659000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76540ba2e0 con 0x7f76540d12a0 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60688/0 (socket says 172.21.15.67:60688) 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 -- 172.21.15.67:0/3251843839 learned_addr learned my addr 172.21.15.67:0/3251843839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7653e00640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540d12a0 0x7f76541f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 -- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540d12a0 msgr2=0x7f76541f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540d12a0 0x7f76541f84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 -- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76540b9f00 msgr2=0x7f765415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76540b9f00 0x7f765415f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.170+0000 7f7652a00640 1 -- 172.21.15.67:0/3251843839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76540c9300 con 0x7f76540cf8f0 2024-04-17T07:34:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f7652a00640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f76400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640032650 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76400327f0 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640032be0 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76540c86f0 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.171+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76540c8cb0 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.172+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76540755d0 con 0x7f76540cf8f0 2024-04-17T07:34:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.173+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f764004c060 con 0x7f76540cf8f0 2024-04-17T07:34:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.175+0000 7f764b400640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 0x7f763406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.175+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 5737+0+0 (secure 0 0 0) 0x7f76400ad530 con 0x7f76540cf8f0 2024-04-17T07:34:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.176+0000 7f7653400640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 0x7f763406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.177+0000 7f7653400640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 0x7f763406ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f76440070b0 tx=0x7f7644007040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.179+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7640040070 con 0x7f76540cf8f0 2024-04-17T07:34:15.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.387+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7654162e70 con 0x7f76540cf8f0 2024-04-17T07:34:15.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.390+0000 7f764b400640 1 -- 172.21.15.67:0/3251843839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+16688 (secure 0 0 0) 0x7f7654163a20 con 0x7f76540cf8f0 2024-04-17T07:34:15.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:15.391 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":155,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:34:13.943601+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.392+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 msgr2=0x7f763406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.392+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 0x7f763406ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f76440070b0 tx=0x7f7644007040 comp rx=0 tx=0).stop 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.392+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 msgr2=0x7f76541ead50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.392+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f7640009870 tx=0x7f76400046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 shutdown_connections 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f763406ca40 0x7f763406ef00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76540d12a0 0x7f76541f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76540cf8f0 0x7f76541ead50 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 --2- 172.21.15.67:0/3251843839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76540b9f00 0x7f765415f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 >> 172.21.15.67:0/3251843839 conn(0x7f765406cb60 msgr2=0x7f76540b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 shutdown_connections 2024-04-17T07:34:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.393+0000 7f7659000640 1 -- 172.21.15.67:0/3251843839 wait complete. 2024-04-17T07:34:15.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T07:34:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.630+0000 7f2d3b800640 1 Processor -- start 2024-04-17T07:34:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.630+0000 7f2d3b800640 1 -- start start 2024-04-17T07:34:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d3b800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d34145ac0 0x7f2d34071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d3b800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d340cea70 con 0x7f2d340ce3b0 2024-04-17T07:34:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d3b800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d340ced50 con 0x7f2d34072d80 2024-04-17T07:34:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d3b800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2d340ce840 con 0x7f2d34145ac0 2024-04-17T07:34:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d33e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2d340ce3b0 0x7f2d34072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55544/0 (socket says 172.21.15.67:55544) 2024-04-17T07:34:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.632+0000 7f2d33e00640 1 -- 172.21.15.67:0/1267692743 learned_addr learned my addr 172.21.15.67:0/1267692743 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d38e00640 1 --2- 172.21.15.67:0/1267692743 >> v2:172.21.15.187:3300/0 conn(0x7f2d34145ac0 0x7f2d34071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860320571 0 0) 0x7f2d340cea70 con 0x7f2d340ce3b0 2024-04-17T07:34:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d1c003520 con 0x7f2d340ce3b0 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2082164333 0 0) 0x7f2d1c003520 con 0x7f2d340ce3b0 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 >> v2:172.21.15.187:3300/0 conn(0x7f2d34145ac0 msgr2=0x7f2d34071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 --2- 172.21.15.67:0/1267692743 >> v2:172.21.15.187:3300/0 conn(0x7f2d34145ac0 0x7f2d34071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 >> v1:172.21.15.187:6789/0 conn(0x7f2d34072d80 legacy=0x7f2d340cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d340cf030 con 0x7f2d340ce3b0 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2d20001670 con 0x7f2d340ce3b0 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 >> v1:172.21.15.67:6789/0 conn(0x7f2d340ce3b0 legacy=0x7f2d34072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1c003a00 0x7f2d1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c00c090 con 0x7f2d1c0051c0 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c00c240 con 0x7f2d1c003a00 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.633+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c00c440 con 0x7f2d1c004800 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d33e00640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 -- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c004800 msgr2=0x7f2d1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c004800 0x7f2d1c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 -- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1c003a00 msgr2=0x7f2d1c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d38e00640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1c003a00 0x7f2d1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1c003a00 0x7f2d1c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d39800640 1 -- 172.21.15.67:0/1267692743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c003520 con 0x7f2d1c0051c0 2024-04-17T07:34:15.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.634+0000 7f2d38e00640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d1c003a00 0x7f2d1c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:15.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.635+0000 7f2d39800640 1 -- 172.21.15.67:0/1267692743 reap_dead start 2024-04-17T07:34:15.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.635+0000 7f2d39800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c001e50 tx=0x7f2d1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e167f434da072e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c001a70 con 0x7f2d1c0051c0 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c003bb0 con 0x7f2d1c0051c0 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d33400640 1 -- 172.21.15.67:0/1267692743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c0050d0 con 0x7f2d1c0051c0 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 -- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 msgr2=0x7f2d1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c001e50 tx=0x7f2d1c005830 comp rx=0 tx=0).stop 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 -- 172.21.15.67:0/1267692743 shutdown_connections 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 --2- 172.21.15.67:0/1267692743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d1c0051c0 0x7f2d1c00b9c0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 -- 172.21.15.67:0/1267692743 >> 172.21.15.67:0/1267692743 conn(0x7f2d34067ee0 msgr2=0x7f2d3406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 -- 172.21.15.67:0/1267692743 shutdown_connections 2024-04-17T07:34:15.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.636+0000 7f2d3b800640 1 -- 172.21.15.67:0/1267692743 wait complete. 2024-04-17T07:34:15.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.638+0000 7f2d3b800640 1 Processor -- start 2024-04-17T07:34:15.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.638+0000 7f2d3b800640 1 -- start start 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c0051c0 0x7f2d3415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3415f3f0 0x7f2d3415db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d34145ea0 con 0x7f2d3415e050 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d340cf5f0 con 0x7f2d3415f3f0 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d3b800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d34071ea0 con 0x7f2d1c0051c0 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.639+0000 7f2d39800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60698/0 (socket says 172.21.15.67:60698) 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 -- 172.21.15.67:0/2544211117 learned_addr learned my addr 172.21.15.67:0/2544211117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d33e00640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3415f3f0 0x7f2d3415db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d38e00640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c0051c0 0x7f2d3415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 -- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c0051c0 msgr2=0x7f2d3415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c0051c0 0x7f2d3415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 -- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3415f3f0 msgr2=0x7f2d3415db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3415f3f0 0x7f2d3415db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 -- 172.21.15.67:0/2544211117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3415e4f0 con 0x7f2d3415e050 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.640+0000 7f2d39800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c02c060 tx=0x7f2d2c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.641+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c001250 con 0x7f2d3415e050 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.641+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c006d60 con 0x7f2d3415e050 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.641+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c039680 con 0x7f2d3415e050 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.641+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3415e7d0 con 0x7f2d3415e050 2024-04-17T07:34:15.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.641+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3415ed90 con 0x7f2d3415e050 2024-04-17T07:34:15.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.642+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d34068ca0 con 0x7f2d3415e050 2024-04-17T07:34:15.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.643+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f2d2c039820 con 0x7f2d3415e050 2024-04-17T07:34:15.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.644+0000 7f2d30c00640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 0x7f2d1406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:15.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.644+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f2d2c0addb0 con 0x7f2d3415e050 2024-04-17T07:34:15.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.645+0000 7f2d38e00640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 0x7f2d1406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:15.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.646+0000 7f2d38e00640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 0x7f2d1406f070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d280099c0 tx=0x7f2d28008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.649+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2d2c033030 con 0x7f2d3415e050 2024-04-17T07:34:15.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:15.897+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2d340cc6c0 con 0x7f2d3415e050 2024-04-17T07:34:16.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.501+0000 7f2d30c00640 1 -- 172.21.15.67:0/2544211117 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v75) ==== 118+0+0 (secure 0 0 0) 0x7f2d2c07bf00 con 0x7f2d3415e050 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 msgr2=0x7f2d1406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 0x7f2d1406f070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d280099c0 tx=0x7f2d28008040 comp rx=0 tx=0).stop 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 msgr2=0x7f2d340cb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c02c060 tx=0x7f2d2c0014a0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 shutdown_connections 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2d1406cbb0 0x7f2d1406f070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3415e050 0x7f2d340cb920 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d3415f3f0 0x7f2d3415db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 --2- 172.21.15.67:0/2544211117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d1c0051c0 0x7f2d3415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.505+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 >> 172.21.15.67:0/2544211117 conn(0x7f2d34067ee0 msgr2=0x7f2d340b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.506+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 shutdown_connections 2024-04-17T07:34:16.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.506+0000 7f2d3b800640 1 -- 172.21.15.67:0/2544211117 wait complete. 2024-04-17T07:34:16.525 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T07:34:16.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T07:34:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.753+0000 7f1233e00640 1 Processor -- start 2024-04-17T07:34:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.753+0000 7f1233e00640 1 -- start start 2024-04-17T07:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1234145f80 0x7f12340cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1234146360 con 0x7f123406c270 2024-04-17T07:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1234071e70 con 0x7f12340cd750 2024-04-17T07:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f12340727a0 con 0x7f1234145f80 2024-04-17T07:34:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f123406c270 0x7f123406e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55560/0 (socket says 172.21.15.67:55560) 2024-04-17T07:34:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.755+0000 7f1233400640 1 -- 172.21.15.67:0/1346295329 learned_addr learned my addr 172.21.15.67:0/1346295329 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1999445965 0 0) 0x7f1234146360 con 0x7f123406c270 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1218003520 con 0x7f123406c270 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1232a00640 1 --2- 172.21.15.67:0/1346295329 >> v2:172.21.15.187:3300/0 conn(0x7f1234145f80 0x7f12340cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 189706360 0 0) 0x7f1218003520 con 0x7f123406c270 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 >> v2:172.21.15.187:3300/0 conn(0x7f1234145f80 msgr2=0x7f12340cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 --2- 172.21.15.67:0/1346295329 >> v2:172.21.15.187:3300/0 conn(0x7f1234145f80 0x7f12340cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 >> v1:172.21.15.187:6789/0 conn(0x7f12340cd750 legacy=0x7f123406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1234072150 con 0x7f123406c270 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f122c0019f0 con 0x7f123406c270 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 >> v1:172.21.15.67:6789/0 conn(0x7f123406c270 legacy=0x7f123406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12180039b0 0x7f1218003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 0x7f1218004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.756+0000 7f1231600640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12180052e0 0x7f121800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12340727a0 con 0x7f1218004800 2024-04-17T07:34:16.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f121800c1b0 con 0x7f12180052e0 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f121800c3b0 con 0x7f12180039b0 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 0x7f1218004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232a00640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12180039b0 0x7f1218003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1233400640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12180052e0 0x7f121800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 -- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12180039b0 msgr2=0x7f1218003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12180039b0 0x7f1218003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 -- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12180052e0 msgr2=0x7f121800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12180052e0 0x7f121800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1232000640 1 -- 172.21.15.67:0/1346295329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1218003520 con 0x7f1218004800 2024-04-17T07:34:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.757+0000 7f1233400640 1 -- 172.21.15.67:0/1346295329 reap_dead start 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.758+0000 7f1232000640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 0x7f1218004c80 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1220001720 tx=0x7f1218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de82a6a92d174ac server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220001370 con 0x7f1218004800 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12200052d0 con 0x7f1218004800 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1231600640 1 -- 172.21.15.67:0/1346295329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220003830 con 0x7f1218004800 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1233e00640 1 -- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 msgr2=0x7f1218004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1233e00640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 0x7f1218004c80 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1220001720 tx=0x7f1218005830 comp rx=0 tx=0).stop 2024-04-17T07:34:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1233e00640 1 -- 172.21.15.67:0/1346295329 shutdown_connections 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1233e00640 1 --2- 172.21.15.67:0/1346295329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1218004800 0x7f1218004c80 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.759+0000 7f1233e00640 1 -- 172.21.15.67:0/1346295329 >> 172.21.15.67:0/1346295329 conn(0x7f12340b2190 msgr2=0x7f12340b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.760+0000 7f1233e00640 1 -- 172.21.15.67:0/1346295329 shutdown_connections 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.760+0000 7f1233e00640 1 -- 172.21.15.67:0/1346295329 wait complete. 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.761+0000 7f1233e00640 1 Processor -- start 2024-04-17T07:34:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.762+0000 7f1233e00640 1 -- start start 2024-04-17T07:34:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.762+0000 7f1233e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1218004800 0x7f12341f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.762+0000 7f1233e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1234144910 0x7f1234144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1234072db0 con 0x7f12340c5dd0 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1234072c30 con 0x7f1234144910 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12340cd210 con 0x7f1218004800 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60718/0 (socket says 172.21.15.67:60718) 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 -- 172.21.15.67:0/703306528 learned_addr learned my addr 172.21.15.67:0/703306528 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1232a00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1218004800 0x7f12341f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1232000640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1234144910 0x7f1234144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 -- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1218004800 msgr2=0x7f12341f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1218004800 0x7f12341f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 -- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1234144910 msgr2=0x7f1234144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1234144910 0x7f1234144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.763+0000 7f1233400640 1 -- 172.21.15.67:0/703306528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12341f4930 con 0x7f12340c5dd0 2024-04-17T07:34:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f1233400640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f12340d03d0 tx=0x7f122c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122c00ec40 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f122c00ede0 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12340c5a60 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122c014670 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.764+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12340c54a0 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.765+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1234068a10 con 0x7f12340c5dd0 2024-04-17T07:34:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.766+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f122c014810 con 0x7f12340c5dd0 2024-04-17T07:34:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.767+0000 7f122aa00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 0x7f121406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.767+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f122c08bc10 con 0x7f12340c5dd0 2024-04-17T07:34:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.768+0000 7f1232a00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 0x7f121406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:16.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.769+0000 7f1232a00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 0x7f121406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12340690c0 tx=0x7f1224008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:16.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:16.772+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f12341f4930 con 0x7f12340c5dd0 2024-04-17T07:34:17.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.015+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f123406ccd0 con 0x7f12340c5dd0 2024-04-17T07:34:17.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.515+0000 7f122aa00640 1 -- 172.21.15.67:0/703306528 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7f122c059f00 con 0x7f12340c5dd0 2024-04-17T07:34:17.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.518+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 msgr2=0x7f121406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.518+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 0x7f121406eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12340690c0 tx=0x7f1224008040 comp rx=0 tx=0).stop 2024-04-17T07:34:17.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.518+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 msgr2=0x7f12340b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.518+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f12340d03d0 tx=0x7f122c00e530 comp rx=0 tx=0).stop 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 shutdown_connections 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f121406ca10 0x7f121406eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12340c5dd0 0x7f12340b48d0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1234144910 0x7f1234144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 --2- 172.21.15.67:0/703306528 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1218004800 0x7f12341f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 >> 172.21.15.67:0/703306528 conn(0x7f12340b2190 msgr2=0x7f123415d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 shutdown_connections 2024-04-17T07:34:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.519+0000 7f1233e00640 1 -- 172.21.15.67:0/703306528 wait complete. 2024-04-17T07:34:17.543 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T07:34:17.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:34:17.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.768+0000 7f93c3e00640 1 Processor -- start 2024-04-17T07:34:17.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.768+0000 7f93c3e00640 1 -- start start 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 0x7f93c40cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c40cd4a0 con 0x7f93c4160280 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c41607f0 con 0x7f93c4141ed0 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93c415ce30 con 0x7f93c406b8c0 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f93c4160280 0x7f93c406bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55568/0 (socket says 172.21.15.67:55568) 2024-04-17T07:34:17.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c2000640 1 -- 172.21.15.67:0/1867119046 learned_addr learned my addr 172.21.15.67:0/1867119046 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.770+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729232024 0 0) 0x7f93c40cd4a0 con 0x7f93c4160280 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a8003520 con 0x7f93c4160280 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 0x7f93c40cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 788188928 0 0) 0x7f93a8003520 con 0x7f93c4160280 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 msgr2=0x7f93c40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:17.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 0x7f93c40cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 >> v1:172.21.15.187:6789/0 conn(0x7f93c4141ed0 legacy=0x7f93c415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c4160ad0 con 0x7f93c4160280 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f93b0001670 con 0x7f93c4160280 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 >> v1:172.21.15.67:6789/0 conn(0x7f93c4160280 legacy=0x7f93c406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a8003a50 0x7f93a8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a8004800 0x7f93a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 0x7f93a8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93c415ce30 con 0x7f93a8004800 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93a800bb80 con 0x7f93a8003a50 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93a800bd80 con 0x7f93c406b8c0 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.771+0000 7f93c2000640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a8004800 0x7f93a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a8003a50 0x7f93a8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2000640 1 -- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 msgr2=0x7f93a8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2000640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93c406b8c0 0x7f93a8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2000640 1 -- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a8003a50 msgr2=0x7f93a8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2000640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a8003a50 0x7f93a8003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.772+0000 7f93c2000640 1 -- 172.21.15.67:0/1867119046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a8003520 con 0x7f93a8004800 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.773+0000 7f93c2000640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a8004800 0x7f93a8004c80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c210 tx=0x7f93a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e286de647ecd710c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.773+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0007d30 con 0x7f93a8004800 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.773+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b0030070 con 0x7f93a8004800 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.773+0000 7f93c1600640 1 -- 172.21.15.67:0/1867119046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b002cde0 con 0x7f93a8004800 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 -- 172.21.15.67:0/1867119046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a8004800 msgr2=0x7f93a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1867119046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a8004800 0x7f93a8004c80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c210 tx=0x7f93a8005830 comp rx=0 tx=0).stop 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3400640 1 -- 172.21.15.67:0/1867119046 reap_dead start 2024-04-17T07:34:17.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 -- 172.21.15.67:0/1867119046 shutdown_connections 2024-04-17T07:34:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 -- 172.21.15.67:0/1867119046 >> 172.21.15.67:0/1867119046 conn(0x7f93c4067d10 msgr2=0x7f93c40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 -- 172.21.15.67:0/1867119046 shutdown_connections 2024-04-17T07:34:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.774+0000 7f93c3e00640 1 -- 172.21.15.67:0/1867119046 wait complete. 2024-04-17T07:34:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.776+0000 7f93c3e00640 1 Processor -- start 2024-04-17T07:34:17.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.776+0000 7f93c3e00640 1 -- start start 2024-04-17T07:34:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 0x7f93c41f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93c4143fb0 0x7f93c40b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93c413f9b0 con 0x7f93c41f42b0 2024-04-17T07:34:17.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4161090 con 0x7f93c4143fb0 2024-04-17T07:34:17.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93c413fb30 con 0x7f93a8004800 2024-04-17T07:34:17.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60736/0 (socket says 172.21.15.67:60736) 2024-04-17T07:34:17.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.777+0000 7f93c2000640 1 -- 172.21.15.67:0/1087579954 learned_addr learned my addr 172.21.15.67:0/1087579954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:17.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 0x7f93c41f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 -- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 msgr2=0x7f93c41f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 0x7f93c41f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c3400640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93c4143fb0 0x7f93c40b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 -- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93c4143fb0 msgr2=0x7f93c40b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93c4143fb0 0x7f93c40b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 -- 172.21.15.67:0/1087579954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c41f4710 con 0x7f93c41f42b0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 0x7f93c41f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93c2000640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c210 tx=0x7f93b0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.778+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0007d10 con 0x7f93c41f42b0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.779+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b00049f0 con 0x7f93c41f42b0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.779+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c415d3d0 con 0x7f93c41f42b0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.779+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0004ba0 con 0x7f93c41f42b0 2024-04-17T07:34:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.779+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c415d670 con 0x7f93c41f42b0 2024-04-17T07:34:17.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.780+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c4068a10 con 0x7f93c41f42b0 2024-04-17T07:34:17.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.781+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f93b0047030 con 0x7f93c41f42b0 2024-04-17T07:34:17.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.782+0000 7f93baa00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 0x7f93a406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:17.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.782+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f93b00ad190 con 0x7f93c41f42b0 2024-04-17T07:34:17.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.782+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 0x7f93a406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:17.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.784+0000 7f93c2a00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 0x7f93a406ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93b4006fd0 tx=0x7f93b4008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:17.786+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f93b004e020 con 0x7f93c41f42b0 2024-04-17T07:34:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.034+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93c4144390 con 0x7f93c41f42b0 2024-04-17T07:34:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.035+0000 7f93baa00640 1 -- 172.21.15.67:0/1087579954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7f93b007b480 con 0x7f93c41f42b0 2024-04-17T07:34:18.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:18.036 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T07:34:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T07:34:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 msgr2=0x7f93a406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 0x7f93a406ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93b4006fd0 tx=0x7f93b4008040 comp rx=0 tx=0).stop 2024-04-17T07:34:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 msgr2=0x7f93c4143a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c210 tx=0x7f93b0001a40 comp rx=0 tx=0).stop 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 shutdown_connections 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f93a406ca40 0x7f93a406ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93c4143fb0 0x7f93c40b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c41f42b0 0x7f93c4143a70 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 --2- 172.21.15.67:0/1087579954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93a8004800 0x7f93c41f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.038+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 >> 172.21.15.67:0/1087579954 conn(0x7f93c4067d10 msgr2=0x7f93c413fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.039+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 shutdown_connections 2024-04-17T07:34:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.039+0000 7f93c3e00640 1 -- 172.21.15.67:0/1087579954 wait complete. 2024-04-17T07:34:18.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T07:34:18.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.289+0000 7f45d9a00640 1 Processor -- start 2024-04-17T07:34:18.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.289+0000 7f45d9a00640 1 -- start start 2024-04-17T07:34:18.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45d415b570 0x7f45d406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d9a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45d406c040 0x7f45d415f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d9a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45d415bc80 con 0x7f45d406fce0 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d9a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f45d406c470 con 0x7f45d415b570 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d9a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f45d415bf40 con 0x7f45d406c040 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45d415b570 0x7f45d406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f45d415b570 0x7f45d406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60748/0 (socket says 172.21.15.67:60748) 2024-04-17T07:34:18.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.291+0000 7f45d3e00640 1 -- 172.21.15.67:0/2300206309 learned_addr learned my addr 172.21.15.67:0/2300206309 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.292+0000 7f45d3e00640 1 -- 172.21.15.67:0/2300206309 >> v2:172.21.15.187:3300/0 conn(0x7f45d406c040 msgr2=0x7f45d415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.292+0000 7f45d3e00640 1 --2- 172.21.15.67:0/2300206309 >> v2:172.21.15.187:3300/0 conn(0x7f45d406c040 0x7f45d415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.292+0000 7f45d3e00640 1 -- 172.21.15.67:0/2300206309 >> v1:172.21.15.187:6789/0 conn(0x7f45d406fce0 legacy=0x7f45d406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.292+0000 7f45d3e00640 1 -- 172.21.15.67:0/2300206309 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d415c220 con 0x7f45d415b570 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.293+0000 7f45d3e00640 1 --2- 172.21.15.67:0/2300206309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d406f7a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f45d406c5f0 tx=0x7f45c402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7df074ba76e1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.293+0000 7f45d2a00640 1 -- 172.21.15.67:0/2300206309 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c402c9e0 con 0x7f45d415b570 2024-04-17T07:34:18.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.293+0000 7f45d2a00640 1 -- 172.21.15.67:0/2300206309 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c402cb80 con 0x7f45d415b570 2024-04-17T07:34:18.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.293+0000 7f45d2a00640 1 -- 172.21.15.67:0/2300206309 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c40343f0 con 0x7f45d415b570 2024-04-17T07:34:18.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 -- 172.21.15.67:0/2300206309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 msgr2=0x7f45d406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 --2- 172.21.15.67:0/2300206309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d406f7a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f45d406c5f0 tx=0x7f45c402be40 comp rx=0 tx=0).stop 2024-04-17T07:34:18.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 -- 172.21.15.67:0/2300206309 shutdown_connections 2024-04-17T07:34:18.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 --2- 172.21.15.67:0/2300206309 >> v2:172.21.15.187:3300/0 conn(0x7f45d406c040 0x7f45d415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 --2- 172.21.15.67:0/2300206309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d406f7a0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 -- 172.21.15.67:0/2300206309 >> 172.21.15.67:0/2300206309 conn(0x7f45d4067d10 msgr2=0x7f45d40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:18.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 -- 172.21.15.67:0/2300206309 shutdown_connections 2024-04-17T07:34:18.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.294+0000 7f45d9a00640 1 -- 172.21.15.67:0/2300206309 wait complete. 2024-04-17T07:34:18.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.296+0000 7f45d9a00640 1 Processor -- start 2024-04-17T07:34:18.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.296+0000 7f45d9a00640 1 -- start start 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d406c040 0x7f45d4144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406fce0 0x7f45d40cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45d40b47b0 con 0x7f45d415b570 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45d415b950 con 0x7f45d406fce0 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45d40b4930 con 0x7f45d406c040 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.297+0000 7f45d9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60750/0 (socket says 172.21.15.67:60750) 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 -- 172.21.15.67:0/4209448475 learned_addr learned my addr 172.21.15.67:0/4209448475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d3e00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d406c040 0x7f45d4144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d3400640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406fce0 0x7f45d40cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 -- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d406c040 msgr2=0x7f45d4144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d406c040 0x7f45d4144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 -- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406fce0 msgr2=0x7f45d40cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406fce0 0x7f45d40cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 -- 172.21.15.67:0/4209448475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d41449d0 con 0x7f45d415b570 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.298+0000 7f45d9000640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009fd0 tx=0x7f45cc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.299+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc011830 con 0x7f45d415b570 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.299+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc016070 con 0x7f45d415b570 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.299+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41473c0 con 0x7f45d415b570 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.299+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc011bf0 con 0x7f45d415b570 2024-04-17T07:34:18.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.299+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d414ecf0 con 0x7f45d415b570 2024-04-17T07:34:18.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.300+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d4068a10 con 0x7f45d415b570 2024-04-17T07:34:18.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.301+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f45cc01b460 con 0x7f45d415b570 2024-04-17T07:34:18.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.303+0000 7f45cbe00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 0x7f45b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.303+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f45cc08bb20 con 0x7f45d415b570 2024-04-17T07:34:18.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.303+0000 7f45d3e00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 0x7f45b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.305+0000 7f45d3e00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 0x7f45b006eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45c4035000 tx=0x7f45c4030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.307+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f45d41449d0 con 0x7f45d415b570 2024-04-17T07:34:18.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.512+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45d414e610 con 0x7f45d415b570 2024-04-17T07:34:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.514+0000 7f45cbe00640 1 -- 172.21.15.67:0/4209448475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16675 (secure 0 0 0) 0x7f45cc059cf0 con 0x7f45d415b570 2024-04-17T07:34:18.514 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:18.515 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":156,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T07:34:15.479715+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 msgr2=0x7f45b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 0x7f45b006eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45c4035000 tx=0x7f45c4030000 comp rx=0 tx=0).stop 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 msgr2=0x7f45d40b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009fd0 tx=0x7f45cc00b8e0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 shutdown_connections 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f45b006ca10 0x7f45b006eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45d415b570 0x7f45d40b3250 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406fce0 0x7f45d40cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 --2- 172.21.15.67:0/4209448475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d406c040 0x7f45d4144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 >> 172.21.15.67:0/4209448475 conn(0x7f45d4067d10 msgr2=0x7f45d415ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 shutdown_connections 2024-04-17T07:34:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.517+0000 7f45d9a00640 1 -- 172.21.15.67:0/4209448475 wait complete. 2024-04-17T07:34:18.540 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T07:34:18.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:34:18.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.761+0000 7fe1ca000640 1 Processor -- start 2024-04-17T07:34:18.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.762+0000 7fe1ca000640 1 -- start start 2024-04-17T07:34:18.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.763+0000 7fe1ca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1c40d09f0 0x7fe1c409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.763+0000 7fe1ca000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1c409afa0 0x7fe1c415e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.763+0000 7fe1ca000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c406dc30 con 0x7fe1c40b3d10 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.763+0000 7fe1ca000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1c406df10 con 0x7fe1c40d09f0 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.763+0000 7fe1ca000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe1c406e1f0 con 0x7fe1c409afa0 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1c40d09f0 0x7fe1c409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1c40d09f0 0x7fe1c409aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60752/0 (socket says 172.21.15.67:60752) 2024-04-17T07:34:18.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 -- 172.21.15.67:0/3183488807 learned_addr learned my addr 172.21.15.67:0/3183488807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c3e00640 1 --2- 172.21.15.67:0/3183488807 >> v2:172.21.15.187:3300/0 conn(0x7fe1c409afa0 0x7fe1c415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 -- 172.21.15.67:0/3183488807 >> v2:172.21.15.187:3300/0 conn(0x7fe1c409afa0 msgr2=0x7fe1c415e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 --2- 172.21.15.67:0/3183488807 >> v2:172.21.15.187:3300/0 conn(0x7fe1c409afa0 0x7fe1c415e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 -- 172.21.15.67:0/3183488807 >> v1:172.21.15.187:6789/0 conn(0x7fe1c40b3d10 legacy=0x7fe1c406d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.764+0000 7fe1c2a00640 1 -- 172.21.15.67:0/3183488807 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c406e4d0 con 0x7fe1c40d09f0 2024-04-17T07:34:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.765+0000 7fe1c2a00640 1 --2- 172.21.15.67:0/3183488807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c40d09f0 0x7fe1c409aa60 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4073810 tx=0x7fe1ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1ea73fe472790c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.766+0000 7fe1c2000640 1 -- 172.21.15.67:0/3183488807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac035020 con 0x7fe1c40d09f0 2024-04-17T07:34:18.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.766+0000 7fe1c2000640 1 -- 172.21.15.67:0/3183488807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1ac031870 con 0x7fe1c40d09f0 2024-04-17T07:34:18.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.766+0000 7fe1c2000640 1 -- 172.21.15.67:0/3183488807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac031c40 con 0x7fe1c40d09f0 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.766+0000 7fe1ca000640 1 -- 172.21.15.67:0/3183488807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c40d09f0 msgr2=0x7fe1c409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.766+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3183488807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c40d09f0 0x7fe1c409aa60 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4073810 tx=0x7fe1ac02b910 comp rx=0 tx=0).stop 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 -- 172.21.15.67:0/3183488807 shutdown_connections 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3183488807 >> v2:172.21.15.187:3300/0 conn(0x7fe1c409afa0 0x7fe1c415e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3183488807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c40d09f0 0x7fe1c409aa60 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 -- 172.21.15.67:0/3183488807 >> 172.21.15.67:0/3183488807 conn(0x7fe1c40b41b0 msgr2=0x7fe1c40b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 -- 172.21.15.67:0/3183488807 shutdown_connections 2024-04-17T07:34:18.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.767+0000 7fe1ca000640 1 -- 172.21.15.67:0/3183488807 wait complete. 2024-04-17T07:34:18.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.768+0000 7fe1ca000640 1 Processor -- start 2024-04-17T07:34:18.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 -- start start 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1c40b3d10 0x7fe1c40cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c40d09f0 0x7fe1c40b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c415e870 con 0x7fe1c409afa0 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c406d8d0 con 0x7fe1c40d09f0 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.769+0000 7fe1ca000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c415e9f0 con 0x7fe1c40b3d10 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60762/0 (socket says 172.21.15.67:60762) 2024-04-17T07:34:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 -- 172.21.15.67:0/3115356653 learned_addr learned my addr 172.21.15.67:0/3115356653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:18.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c2a00640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1c40b3d10 0x7fe1c40cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3e00640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c40d09f0 0x7fe1c40b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 -- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1c40b3d10 msgr2=0x7fe1c40cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1c40b3d10 0x7fe1c40cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 -- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c40d09f0 msgr2=0x7fe1c40b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c40d09f0 0x7fe1c40b77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.770+0000 7fe1c3400640 1 -- 172.21.15.67:0/3115356653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c40b5730 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1c3400640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c7f0 tx=0x7fe1b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4016020 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b400aa50 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b400ae40 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c40b5a10 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.771+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c40b5fd0 con 0x7fe1c409afa0 2024-04-17T07:34:18.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.772+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c4068b90 con 0x7fe1c409afa0 2024-04-17T07:34:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.773+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fe1b4019020 con 0x7fe1c409afa0 2024-04-17T07:34:18.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.775+0000 7fe1bb400640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 0x7fe1a006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.775+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fe1b408bb20 con 0x7fe1c409afa0 2024-04-17T07:34:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.775+0000 7fe1c2a00640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 0x7fe1a006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.776+0000 7fe1c2a00640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 0x7fe1a006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac036000 tx=0x7fe1ac03c040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:18.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.780+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1b4020070 con 0x7fe1c409afa0 2024-04-17T07:34:18.874 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T07:34:18.874+0000 7f7538800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T07:34:18.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:18.982+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 --> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1c40cca50 con 0x7fe1a006ca10 2024-04-17T07:34:19.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.006+0000 7fe1bb400640 1 -- 172.21.15.67:0/3115356653 <== mgr.7087 v2:172.21.15.187:6832/115904466 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252454 (secure 0 0 0) 0x7fe1c40cca50 con 0x7fe1a006ca10 2024-04-17T07:34:19.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 msgr2=0x7fe1a006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 0x7fe1a006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac036000 tx=0x7fe1ac03c040 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 msgr2=0x7fe1c40b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c7f0 tx=0x7fe1b400ccb0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 shutdown_connections 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe1a006ca10 0x7fe1a006eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1c40d09f0 0x7fe1c40b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1c40b3d10 0x7fe1c40cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 --2- 172.21.15.67:0/3115356653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1c409afa0 0x7fe1c40b67d0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 >> 172.21.15.67:0/3115356653 conn(0x7fe1c40b41b0 msgr2=0x7fe1c40cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 shutdown_connections 2024-04-17T07:34:19.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.012+0000 7fe1ca000640 1 -- 172.21.15.67:0/3115356653 wait complete. 2024-04-17T07:34:19.033 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T07:34:17.310278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":451,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361496,"kb_used_data":16288,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613224,"statfs":{"total":773094113280,"available":772723941376,"internally_reserved":0,"allocated":16678912,"data_stored":8612028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.870517"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.506426+0000","last_change":"2024-04-17T07:34:14.502350+0000","last_active":"2024-04-17T07:34:14.506426+0000","last_peered":"2024-04-17T07:34:14.506426+0000","last_clean":"2024-04-17T07:34:14.506426+0000","last_became_active":"2024-04-17T07:34:14.500560+0000","last_became_peered":"2024-04-17T07:34:14.500560+0000","last_unstale":"2024-04-17T07:34:14.506426+0000","last_undegraded":"2024-04-17T07:34:14.506426+0000","last_fullsized":"2024-04-17T07:34:14.506426+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:00:15.119442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493549+0000","last_change":"2024-04-17T07:34:14.501572+0000","last_active":"2024-04-17T07:34:15.493549+0000","last_peered":"2024-04-17T07:34:15.493549+0000","last_clean":"2024-04-17T07:34:15.493549+0000","last_became_active":"2024-04-17T07:34:14.500910+0000","last_became_peered":"2024-04-17T07:34:14.500910+0000","last_unstale":"2024-04-17T07:34:15.493549+0000","last_undegraded":"2024-04-17T07:34:15.493549+0000","last_fullsized":"2024-04-17T07:34:15.493549+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:46.790600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162898+0000","last_change":"2024-04-17T07:34:14.507382+0000","last_active":"2024-04-17T07:34:16.162898+0000","last_peered":"2024-04-17T07:34:16.162898+0000","last_clean":"2024-04-17T07:34:16.162898+0000","last_became_active":"2024-04-17T07:34:14.502526+0000","last_became_peered":"2024-04-17T07:34:14.502526+0000","last_unstale":"2024-04-17T07:34:16.162898+0000","last_undegraded":"2024-04-17T07:34:16.162898+0000","last_fullsized":"2024-04-17T07:34:16.162898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:43.844557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.501137+0000","last_change":"2024-04-17T07:34:14.497350+0000","last_active":"2024-04-17T07:34:14.501137+0000","last_peered":"2024-04-17T07:34:14.501137+0000","last_clean":"2024-04-17T07:34:14.501137+0000","last_became_active":"2024-04-17T07:34:14.496296+0000","last_became_peered":"2024-04-17T07:34:14.496296+0000","last_unstale":"2024-04-17T07:34:14.501137+0000","last_undegraded":"2024-04-17T07:34:14.501137+0000","last_fullsized":"2024-04-17T07:34:14.501137+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:28.955882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497811+0000","last_change":"2024-04-17T07:34:14.510852+0000","last_active":"2024-04-17T07:34:15.497811+0000","last_peered":"2024-04-17T07:34:15.497811+0000","last_clean":"2024-04-17T07:34:15.497811+0000","last_became_active":"2024-04-17T07:34:14.510538+0000","last_became_peered":"2024-04-17T07:34:14.510538+0000","last_unstale":"2024-04-17T07:34:15.497811+0000","last_undegraded":"2024-04-17T07:34:15.497811+0000","last_fullsized":"2024-04-17T07:34:15.497811+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:07.609244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.499236+0000","last_change":"2024-04-17T07:34:14.487905+0000","last_active":"2024-04-17T07:34:14.499236+0000","last_peered":"2024-04-17T07:34:14.499236+0000","last_clean":"2024-04-17T07:34:14.499236+0000","last_became_active":"2024-04-17T07:34:14.487568+0000","last_became_peered":"2024-04-17T07:34:14.487568+0000","last_unstale":"2024-04-17T07:34:14.499236+0000","last_undegraded":"2024-04-17T07:34:14.499236+0000","last_fullsized":"2024-04-17T07:34:14.499236+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:20:01.858330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508441+0000","last_change":"2024-04-17T07:34:14.503113+0000","last_active":"2024-04-17T07:34:14.508441+0000","last_peered":"2024-04-17T07:34:14.508441+0000","last_clean":"2024-04-17T07:34:14.508441+0000","last_became_active":"2024-04-17T07:34:14.502798+0000","last_became_peered":"2024-04-17T07:34:14.502798+0000","last_unstale":"2024-04-17T07:34:14.508441+0000","last_undegraded":"2024-04-17T07:34:14.508441+0000","last_fullsized":"2024-04-17T07:34:14.508441+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:19.146614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163047+0000","last_change":"2024-04-17T07:34:14.503259+0000","last_active":"2024-04-17T07:34:16.163047+0000","last_peered":"2024-04-17T07:34:16.163047+0000","last_clean":"2024-04-17T07:34:16.163047+0000","last_became_active":"2024-04-17T07:34:14.502903+0000","last_became_peered":"2024-04-17T07:34:14.502903+0000","last_unstale":"2024-04-17T07:34:16.163047+0000","last_undegraded":"2024-04-17T07:34:16.163047+0000","last_fullsized":"2024-04-17T07:34:16.163047+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:13.115027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163452+0000","last_change":"2024-04-17T07:34:14.508623+0000","last_active":"2024-04-17T07:34:16.163452+0000","last_peered":"2024-04-17T07:34:16.163452+0000","last_clean":"2024-04-17T07:34:16.163452+0000","last_became_active":"2024-04-17T07:34:14.506735+0000","last_became_peered":"2024-04-17T07:34:14.506735+0000","last_unstale":"2024-04-17T07:34:16.163452+0000","last_undegraded":"2024-04-17T07:34:16.163452+0000","last_fullsized":"2024-04-17T07:34:16.163452+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:48.420144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493740+0000","last_change":"2024-04-17T07:34:14.502447+0000","last_active":"2024-04-17T07:34:15.493740+0000","last_peered":"2024-04-17T07:34:15.493740+0000","last_clean":"2024-04-17T07:34:15.493740+0000","last_became_active":"2024-04-17T07:34:14.501540+0000","last_became_peered":"2024-04-17T07:34:14.501540+0000","last_unstale":"2024-04-17T07:34:15.493740+0000","last_undegraded":"2024-04-17T07:34:15.493740+0000","last_fullsized":"2024-04-17T07:34:15.493740+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:33.909583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494772+0000","last_change":"2024-04-17T07:34:14.489911+0000","last_active":"2024-04-17T07:34:14.494772+0000","last_peered":"2024-04-17T07:34:14.494772+0000","last_clean":"2024-04-17T07:34:14.494772+0000","last_became_active":"2024-04-17T07:34:14.489627+0000","last_became_peered":"2024-04-17T07:34:14.489627+0000","last_unstale":"2024-04-17T07:34:14.494772+0000","last_undegraded":"2024-04-17T07:34:14.494772+0000","last_fullsized":"2024-04-17T07:34:14.494772+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:26.284679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162601+0000","last_change":"2024-04-17T07:34:14.509249+0000","last_active":"2024-04-17T07:34:16.162601+0000","last_peered":"2024-04-17T07:34:16.162601+0000","last_clean":"2024-04-17T07:34:16.162601+0000","last_became_active":"2024-04-17T07:34:14.506750+0000","last_became_peered":"2024-04-17T07:34:14.506750+0000","last_unstale":"2024-04-17T07:34:16.162601+0000","last_undegraded":"2024-04-17T07:34:16.162601+0000","last_fullsized":"2024-04-17T07:34:16.162601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:11.761214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.490448+0000","last_change":"2024-04-17T07:34:14.488978+0000","last_active":"2024-04-17T07:34:14.490448+0000","last_peered":"2024-04-17T07:34:14.490448+0000","last_clean":"2024-04-17T07:34:14.490448+0000","last_became_active":"2024-04-17T07:34:14.488706+0000","last_became_peered":"2024-04-17T07:34:14.488706+0000","last_unstale":"2024-04-17T07:34:14.490448+0000","last_undegraded":"2024-04-17T07:34:14.490448+0000","last_fullsized":"2024-04-17T07:34:14.490448+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:06.803534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492604+0000","last_change":"2024-04-17T07:34:14.486803+0000","last_active":"2024-04-17T07:34:15.492604+0000","last_peered":"2024-04-17T07:34:15.492604+0000","last_clean":"2024-04-17T07:34:15.492604+0000","last_became_active":"2024-04-17T07:34:14.486516+0000","last_became_peered":"2024-04-17T07:34:14.486516+0000","last_unstale":"2024-04-17T07:34:15.492604+0000","last_undegraded":"2024-04-17T07:34:15.492604+0000","last_fullsized":"2024-04-17T07:34:15.492604+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:58.611392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508097+0000","last_change":"2024-04-17T07:34:14.506224+0000","last_active":"2024-04-17T07:34:14.508097+0000","last_peered":"2024-04-17T07:34:14.508097+0000","last_clean":"2024-04-17T07:34:14.508097+0000","last_became_active":"2024-04-17T07:34:14.505109+0000","last_became_peered":"2024-04-17T07:34:14.505109+0000","last_unstale":"2024-04-17T07:34:14.508097+0000","last_undegraded":"2024-04-17T07:34:14.508097+0000","last_fullsized":"2024-04-17T07:34:14.508097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:59.260260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.496461+0000","last_change":"2024-04-17T07:34:14.492852+0000","last_active":"2024-04-17T07:34:14.496461+0000","last_peered":"2024-04-17T07:34:14.496461+0000","last_clean":"2024-04-17T07:34:14.496461+0000","last_became_active":"2024-04-17T07:34:14.492498+0000","last_became_peered":"2024-04-17T07:34:14.492498+0000","last_unstale":"2024-04-17T07:34:14.496461+0000","last_undegraded":"2024-04-17T07:34:14.496461+0000","last_fullsized":"2024-04-17T07:34:14.496461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:27.692060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.497635+0000","last_change":"2024-04-17T07:34:14.487588+0000","last_active":"2024-04-17T07:34:14.497635+0000","last_peered":"2024-04-17T07:34:14.497635+0000","last_clean":"2024-04-17T07:34:14.497635+0000","last_became_active":"2024-04-17T07:34:14.486738+0000","last_became_peered":"2024-04-17T07:34:14.486738+0000","last_unstale":"2024-04-17T07:34:14.497635+0000","last_undegraded":"2024-04-17T07:34:14.497635+0000","last_fullsized":"2024-04-17T07:34:14.497635+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:52:41.869677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.505394+0000","last_change":"2024-04-17T07:34:14.501661+0000","last_active":"2024-04-17T07:34:14.505394+0000","last_peered":"2024-04-17T07:34:14.505394+0000","last_clean":"2024-04-17T07:34:14.505394+0000","last_became_active":"2024-04-17T07:34:14.501148+0000","last_became_peered":"2024-04-17T07:34:14.501148+0000","last_unstale":"2024-04-17T07:34:14.505394+0000","last_undegraded":"2024-04-17T07:34:14.505394+0000","last_fullsized":"2024-04-17T07:34:14.505394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:31.545462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.496393+0000","last_change":"2024-04-17T07:34:14.493010+0000","last_active":"2024-04-17T07:34:14.496393+0000","last_peered":"2024-04-17T07:34:14.496393+0000","last_clean":"2024-04-17T07:34:14.496393+0000","last_became_active":"2024-04-17T07:34:14.492703+0000","last_became_peered":"2024-04-17T07:34:14.492703+0000","last_unstale":"2024-04-17T07:34:14.496393+0000","last_undegraded":"2024-04-17T07:34:14.496393+0000","last_fullsized":"2024-04-17T07:34:14.496393+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:05.976469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498776+0000","last_change":"2024-04-17T07:34:14.495259+0000","last_active":"2024-04-17T07:34:14.498776+0000","last_peered":"2024-04-17T07:34:14.498776+0000","last_clean":"2024-04-17T07:34:14.498776+0000","last_became_active":"2024-04-17T07:34:14.494917+0000","last_became_peered":"2024-04-17T07:34:14.494917+0000","last_unstale":"2024-04-17T07:34:14.498776+0000","last_undegraded":"2024-04-17T07:34:14.498776+0000","last_fullsized":"2024-04-17T07:34:14.498776+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:49.137506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497468+0000","last_change":"2024-04-17T07:34:14.507005+0000","last_active":"2024-04-17T07:34:15.497468+0000","last_peered":"2024-04-17T07:34:15.497468+0000","last_clean":"2024-04-17T07:34:15.497468+0000","last_became_active":"2024-04-17T07:34:14.506727+0000","last_became_peered":"2024-04-17T07:34:14.506727+0000","last_unstale":"2024-04-17T07:34:15.497468+0000","last_undegraded":"2024-04-17T07:34:15.497468+0000","last_fullsized":"2024-04-17T07:34:15.497468+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:25.841542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487221+0000","last_change":"2024-04-17T07:34:14.483773+0000","last_active":"2024-04-17T07:34:14.487221+0000","last_peered":"2024-04-17T07:34:14.487221+0000","last_clean":"2024-04-17T07:34:14.487221+0000","last_became_active":"2024-04-17T07:34:14.483471+0000","last_became_peered":"2024-04-17T07:34:14.483471+0000","last_unstale":"2024-04-17T07:34:14.487221+0000","last_undegraded":"2024-04-17T07:34:14.487221+0000","last_fullsized":"2024-04-17T07:34:14.487221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:06.146869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163002+0000","last_change":"2024-04-17T07:34:14.510582+0000","last_active":"2024-04-17T07:34:16.163002+0000","last_peered":"2024-04-17T07:34:16.163002+0000","last_clean":"2024-04-17T07:34:16.163002+0000","last_became_active":"2024-04-17T07:34:14.510042+0000","last_became_peered":"2024-04-17T07:34:14.510042+0000","last_unstale":"2024-04-17T07:34:16.163002+0000","last_undegraded":"2024-04-17T07:34:16.163002+0000","last_fullsized":"2024-04-17T07:34:16.163002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:18.855740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.493492+0000","last_change":"2024-04-17T07:34:14.486145+0000","last_active":"2024-04-17T07:34:14.493492+0000","last_peered":"2024-04-17T07:34:14.493492+0000","last_clean":"2024-04-17T07:34:14.493492+0000","last_became_active":"2024-04-17T07:34:14.485860+0000","last_became_peered":"2024-04-17T07:34:14.485860+0000","last_unstale":"2024-04-17T07:34:14.493492+0000","last_undegraded":"2024-04-17T07:34:14.493492+0000","last_fullsized":"2024-04-17T07:34:14.493492+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:22.319028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493557+0000","last_change":"2024-04-17T07:34:14.505331+0000","last_active":"2024-04-17T07:34:15.493557+0000","last_peered":"2024-04-17T07:34:15.493557+0000","last_clean":"2024-04-17T07:34:15.493557+0000","last_became_active":"2024-04-17T07:34:14.504964+0000","last_became_peered":"2024-04-17T07:34:14.504964+0000","last_unstale":"2024-04-17T07:34:15.493557+0000","last_undegraded":"2024-04-17T07:34:15.493557+0000","last_fullsized":"2024-04-17T07:34:15.493557+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:18.021895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498555+0000","last_change":"2024-04-17T07:34:14.495963+0000","last_active":"2024-04-17T07:34:14.498555+0000","last_peered":"2024-04-17T07:34:14.498555+0000","last_clean":"2024-04-17T07:34:14.498555+0000","last_became_active":"2024-04-17T07:34:14.495650+0000","last_became_peered":"2024-04-17T07:34:14.495650+0000","last_unstale":"2024-04-17T07:34:14.498555+0000","last_undegraded":"2024-04-17T07:34:14.498555+0000","last_fullsized":"2024-04-17T07:34:14.498555+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:06.385241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.506872+0000","last_change":"2024-04-17T07:34:14.502304+0000","last_active":"2024-04-17T07:34:14.506872+0000","last_peered":"2024-04-17T07:34:14.506872+0000","last_clean":"2024-04-17T07:34:14.506872+0000","last_became_active":"2024-04-17T07:34:14.502030+0000","last_became_peered":"2024-04-17T07:34:14.502030+0000","last_unstale":"2024-04-17T07:34:14.506872+0000","last_undegraded":"2024-04-17T07:34:14.506872+0000","last_fullsized":"2024-04-17T07:34:14.506872+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:40.792604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498430+0000","last_change":"2024-04-17T07:34:14.499430+0000","last_active":"2024-04-17T07:34:15.498430+0000","last_peered":"2024-04-17T07:34:15.498430+0000","last_clean":"2024-04-17T07:34:15.498430+0000","last_became_active":"2024-04-17T07:34:14.498886+0000","last_became_peered":"2024-04-17T07:34:14.498886+0000","last_unstale":"2024-04-17T07:34:15.498430+0000","last_undegraded":"2024-04-17T07:34:15.498430+0000","last_fullsized":"2024-04-17T07:34:15.498430+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:24.539675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508720+0000","last_change":"2024-04-17T07:34:14.507325+0000","last_active":"2024-04-17T07:34:14.508720+0000","last_peered":"2024-04-17T07:34:14.508720+0000","last_clean":"2024-04-17T07:34:14.508720+0000","last_became_active":"2024-04-17T07:34:14.504438+0000","last_became_peered":"2024-04-17T07:34:14.504438+0000","last_unstale":"2024-04-17T07:34:14.508720+0000","last_undegraded":"2024-04-17T07:34:14.508720+0000","last_fullsized":"2024-04-17T07:34:14.508720+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:42.441631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.502662+0000","last_change":"2024-04-17T07:34:14.496718+0000","last_active":"2024-04-17T07:34:14.502662+0000","last_peered":"2024-04-17T07:34:14.502662+0000","last_clean":"2024-04-17T07:34:14.502662+0000","last_became_active":"2024-04-17T07:34:14.496004+0000","last_became_peered":"2024-04-17T07:34:14.496004+0000","last_unstale":"2024-04-17T07:34:14.502662+0000","last_undegraded":"2024-04-17T07:34:14.502662+0000","last_fullsized":"2024-04-17T07:34:14.502662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:04.233979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498281+0000","last_change":"2024-04-17T07:34:14.503925+0000","last_active":"2024-04-17T07:34:15.498281+0000","last_peered":"2024-04-17T07:34:15.498281+0000","last_clean":"2024-04-17T07:34:15.498281+0000","last_became_active":"2024-04-17T07:34:14.503647+0000","last_became_peered":"2024-04-17T07:34:14.503647+0000","last_unstale":"2024-04-17T07:34:15.498281+0000","last_undegraded":"2024-04-17T07:34:15.498281+0000","last_fullsized":"2024-04-17T07:34:15.498281+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:51.973483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.500672+0000","last_change":"2024-04-17T07:34:14.497393+0000","last_active":"2024-04-17T07:34:14.500672+0000","last_peered":"2024-04-17T07:34:14.500672+0000","last_clean":"2024-04-17T07:34:14.500672+0000","last_became_active":"2024-04-17T07:34:14.497095+0000","last_became_peered":"2024-04-17T07:34:14.497095+0000","last_unstale":"2024-04-17T07:34:14.500672+0000","last_undegraded":"2024-04-17T07:34:14.500672+0000","last_fullsized":"2024-04-17T07:34:14.500672+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:19.042966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498533+0000","last_change":"2024-04-17T07:34:14.492626+0000","last_active":"2024-04-17T07:34:14.498533+0000","last_peered":"2024-04-17T07:34:14.498533+0000","last_clean":"2024-04-17T07:34:14.498533+0000","last_became_active":"2024-04-17T07:34:14.492297+0000","last_became_peered":"2024-04-17T07:34:14.492297+0000","last_unstale":"2024-04-17T07:34:14.498533+0000","last_undegraded":"2024-04-17T07:34:14.498533+0000","last_fullsized":"2024-04-17T07:34:14.498533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:29.645880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497723+0000","last_change":"2024-04-17T07:34:13.482108+0000","last_active":"2024-04-17T07:34:15.497723+0000","last_peered":"2024-04-17T07:34:15.497723+0000","last_clean":"2024-04-17T07:34:15.497723+0000","last_became_active":"2024-04-17T07:34:13.481638+0000","last_became_peered":"2024-04-17T07:34:13.481638+0000","last_unstale":"2024-04-17T07:34:15.497723+0000","last_undegraded":"2024-04-17T07:34:15.497723+0000","last_fullsized":"2024-04-17T07:34:15.497723+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:06.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163584+0000","last_change":"2024-04-17T07:34:14.502790+0000","last_active":"2024-04-17T07:34:16.163584+0000","last_peered":"2024-04-17T07:34:16.163584+0000","last_clean":"2024-04-17T07:34:16.163584+0000","last_became_active":"2024-04-17T07:34:14.502519+0000","last_became_peered":"2024-04-17T07:34:14.502519+0000","last_unstale":"2024-04-17T07:34:16.163584+0000","last_undegraded":"2024-04-17T07:34:16.163584+0000","last_fullsized":"2024-04-17T07:34:16.163584+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:35.702756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497945+0000","last_change":"2024-04-17T07:34:13.472917+0000","last_active":"2024-04-17T07:34:15.497945+0000","last_peered":"2024-04-17T07:34:15.497945+0000","last_clean":"2024-04-17T07:34:15.497945+0000","last_became_active":"2024-04-17T07:34:13.472542+0000","last_became_peered":"2024-04-17T07:34:13.472542+0000","last_unstale":"2024-04-17T07:34:15.497945+0000","last_undegraded":"2024-04-17T07:34:15.497945+0000","last_fullsized":"2024-04-17T07:34:15.497945+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:51.378724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499971+0000","last_change":"2024-04-17T07:34:14.501840+0000","last_active":"2024-04-17T07:34:15.499971+0000","last_peered":"2024-04-17T07:34:15.499971+0000","last_clean":"2024-04-17T07:34:15.499971+0000","last_became_active":"2024-04-17T07:34:14.501423+0000","last_became_peered":"2024-04-17T07:34:14.501423+0000","last_unstale":"2024-04-17T07:34:15.499971+0000","last_undegraded":"2024-04-17T07:34:15.499971+0000","last_fullsized":"2024-04-17T07:34:15.499971+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:57:36.741933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.499151+0000","last_change":"2024-04-17T07:34:13.476284+0000","last_active":"2024-04-17T07:34:14.499151+0000","last_peered":"2024-04-17T07:34:14.499151+0000","last_clean":"2024-04-17T07:34:14.499151+0000","last_became_active":"2024-04-17T07:34:13.476011+0000","last_became_peered":"2024-04-17T07:34:13.476011+0000","last_unstale":"2024-04-17T07:34:14.499151+0000","last_undegraded":"2024-04-17T07:34:14.499151+0000","last_fullsized":"2024-04-17T07:34:14.499151+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:41.129787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498733+0000","last_change":"2024-04-17T07:34:14.501318+0000","last_active":"2024-04-17T07:34:15.498733+0000","last_peered":"2024-04-17T07:34:15.498733+0000","last_clean":"2024-04-17T07:34:15.498733+0000","last_became_active":"2024-04-17T07:34:14.501013+0000","last_became_peered":"2024-04-17T07:34:14.501013+0000","last_unstale":"2024-04-17T07:34:15.498733+0000","last_undegraded":"2024-04-17T07:34:15.498733+0000","last_fullsized":"2024-04-17T07:34:15.498733+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:14:13.108917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493023+0000","last_change":"2024-04-17T07:34:13.477009+0000","last_active":"2024-04-17T07:34:15.493023+0000","last_peered":"2024-04-17T07:34:15.493023+0000","last_clean":"2024-04-17T07:34:15.493023+0000","last_became_active":"2024-04-17T07:34:13.476708+0000","last_became_peered":"2024-04-17T07:34:13.476708+0000","last_unstale":"2024-04-17T07:34:15.493023+0000","last_undegraded":"2024-04-17T07:34:15.493023+0000","last_fullsized":"2024-04-17T07:34:15.493023+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:27:26.955485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498744+0000","last_change":"2024-04-17T07:34:14.497392+0000","last_active":"2024-04-17T07:34:14.498744+0000","last_peered":"2024-04-17T07:34:14.498744+0000","last_clean":"2024-04-17T07:34:14.498744+0000","last_became_active":"2024-04-17T07:34:14.497045+0000","last_became_peered":"2024-04-17T07:34:14.497045+0000","last_unstale":"2024-04-17T07:34:14.498744+0000","last_undegraded":"2024-04-17T07:34:14.498744+0000","last_fullsized":"2024-04-17T07:34:14.498744+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:40:11.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492552+0000","last_change":"2024-04-17T07:34:13.472097+0000","last_active":"2024-04-17T07:34:15.492552+0000","last_peered":"2024-04-17T07:34:15.492552+0000","last_clean":"2024-04-17T07:34:15.492552+0000","last_became_active":"2024-04-17T07:34:13.471282+0000","last_became_peered":"2024-04-17T07:34:13.471282+0000","last_unstale":"2024-04-17T07:34:15.492552+0000","last_undegraded":"2024-04-17T07:34:15.492552+0000","last_fullsized":"2024-04-17T07:34:15.492552+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:49.774976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494633+0000","last_change":"2024-04-17T07:34:14.485764+0000","last_active":"2024-04-17T07:34:14.494633+0000","last_peered":"2024-04-17T07:34:14.494633+0000","last_clean":"2024-04-17T07:34:14.494633+0000","last_became_active":"2024-04-17T07:34:14.485423+0000","last_became_peered":"2024-04-17T07:34:14.485423+0000","last_unstale":"2024-04-17T07:34:14.494633+0000","last_undegraded":"2024-04-17T07:34:14.494633+0000","last_fullsized":"2024-04-17T07:34:14.494633+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:41.384791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494470+0000","last_change":"2024-04-17T07:34:13.470606+0000","last_active":"2024-04-17T07:34:14.494470+0000","last_peered":"2024-04-17T07:34:14.494470+0000","last_clean":"2024-04-17T07:34:14.494470+0000","last_became_active":"2024-04-17T07:34:13.470231+0000","last_became_peered":"2024-04-17T07:34:13.470231+0000","last_unstale":"2024-04-17T07:34:14.494470+0000","last_undegraded":"2024-04-17T07:34:14.494470+0000","last_fullsized":"2024-04-17T07:34:14.494470+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:27.702775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162600+0000","last_change":"2024-04-17T07:34:14.502091+0000","last_active":"2024-04-17T07:34:16.162600+0000","last_peered":"2024-04-17T07:34:16.162600+0000","last_clean":"2024-04-17T07:34:16.162600+0000","last_became_active":"2024-04-17T07:34:14.500408+0000","last_became_peered":"2024-04-17T07:34:14.500408+0000","last_unstale":"2024-04-17T07:34:16.162600+0000","last_undegraded":"2024-04-17T07:34:16.162600+0000","last_fullsized":"2024-04-17T07:34:16.162600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:27:27.972837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.472876+0000","last_change":"2024-04-17T07:34:13.483670+0000","last_active":"2024-04-17T07:34:14.472876+0000","last_peered":"2024-04-17T07:34:14.472876+0000","last_clean":"2024-04-17T07:34:14.472876+0000","last_became_active":"2024-04-17T07:34:13.483351+0000","last_became_peered":"2024-04-17T07:34:13.483351+0000","last_unstale":"2024-04-17T07:34:14.472876+0000","last_undegraded":"2024-04-17T07:34:14.472876+0000","last_fullsized":"2024-04-17T07:34:14.472876+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:53.738666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493149+0000","last_change":"2024-04-17T07:34:14.498031+0000","last_active":"2024-04-17T07:34:15.493149+0000","last_peered":"2024-04-17T07:34:15.493149+0000","last_clean":"2024-04-17T07:34:15.493149+0000","last_became_active":"2024-04-17T07:34:14.497439+0000","last_became_peered":"2024-04-17T07:34:14.497439+0000","last_unstale":"2024-04-17T07:34:15.493149+0000","last_undegraded":"2024-04-17T07:34:15.493149+0000","last_fullsized":"2024-04-17T07:34:15.493149+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:53:06.025412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492597+0000","last_change":"2024-04-17T07:34:13.479618+0000","last_active":"2024-04-17T07:34:15.492597+0000","last_peered":"2024-04-17T07:34:15.492597+0000","last_clean":"2024-04-17T07:34:15.492597+0000","last_became_active":"2024-04-17T07:34:13.479301+0000","last_became_peered":"2024-04-17T07:34:13.479301+0000","last_unstale":"2024-04-17T07:34:15.492597+0000","last_undegraded":"2024-04-17T07:34:15.492597+0000","last_fullsized":"2024-04-17T07:34:15.492597+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:20:44.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487460+0000","last_change":"2024-04-17T07:34:14.484315+0000","last_active":"2024-04-17T07:34:14.487460+0000","last_peered":"2024-04-17T07:34:14.487460+0000","last_clean":"2024-04-17T07:34:14.487460+0000","last_became_active":"2024-04-17T07:34:14.481914+0000","last_became_peered":"2024-04-17T07:34:14.481914+0000","last_unstale":"2024-04-17T07:34:14.487460+0000","last_undegraded":"2024-04-17T07:34:14.487460+0000","last_fullsized":"2024-04-17T07:34:14.487460+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:36.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.500193+0000","last_change":"2024-04-17T07:34:13.474704+0000","last_active":"2024-04-17T07:34:15.500193+0000","last_peered":"2024-04-17T07:34:15.500193+0000","last_clean":"2024-04-17T07:34:15.500193+0000","last_became_active":"2024-04-17T07:34:13.474436+0000","last_became_peered":"2024-04-17T07:34:13.474436+0000","last_unstale":"2024-04-17T07:34:15.500193+0000","last_undegraded":"2024-04-17T07:34:15.500193+0000","last_fullsized":"2024-04-17T07:34:15.500193+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:06.166341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499518+0000","last_change":"2024-04-17T07:34:14.497957+0000","last_active":"2024-04-17T07:34:15.499518+0000","last_peered":"2024-04-17T07:34:15.499518+0000","last_clean":"2024-04-17T07:34:15.499518+0000","last_became_active":"2024-04-17T07:34:14.497649+0000","last_became_peered":"2024-04-17T07:34:14.497649+0000","last_unstale":"2024-04-17T07:34:15.499518+0000","last_undegraded":"2024-04-17T07:34:15.499518+0000","last_fullsized":"2024-04-17T07:34:15.499518+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:35:34.025909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.488502+0000","last_change":"2024-04-17T07:34:13.483491+0000","last_active":"2024-04-17T07:34:14.488502+0000","last_peered":"2024-04-17T07:34:14.488502+0000","last_clean":"2024-04-17T07:34:14.488502+0000","last_became_active":"2024-04-17T07:34:13.483088+0000","last_became_peered":"2024-04-17T07:34:13.483088+0000","last_unstale":"2024-04-17T07:34:14.488502+0000","last_undegraded":"2024-04-17T07:34:14.488502+0000","last_fullsized":"2024-04-17T07:34:14.488502+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:52:58.615117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499857+0000","last_change":"2024-04-17T07:34:14.506961+0000","last_active":"2024-04-17T07:34:15.499857+0000","last_peered":"2024-04-17T07:34:15.499857+0000","last_clean":"2024-04-17T07:34:15.499857+0000","last_became_active":"2024-04-17T07:34:14.501792+0000","last_became_peered":"2024-04-17T07:34:14.501792+0000","last_unstale":"2024-04-17T07:34:15.499857+0000","last_undegraded":"2024-04-17T07:34:15.499857+0000","last_fullsized":"2024-04-17T07:34:15.499857+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:51.117385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498284+0000","last_change":"2024-04-17T07:34:13.473275+0000","last_active":"2024-04-17T07:34:15.498284+0000","last_peered":"2024-04-17T07:34:15.498284+0000","last_clean":"2024-04-17T07:34:15.498284+0000","last_became_active":"2024-04-17T07:34:13.471183+0000","last_became_peered":"2024-04-17T07:34:13.471183+0000","last_unstale":"2024-04-17T07:34:15.498284+0000","last_undegraded":"2024-04-17T07:34:15.498284+0000","last_fullsized":"2024-04-17T07:34:15.498284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:45.694228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498866+0000","last_change":"2024-04-17T07:34:14.505454+0000","last_active":"2024-04-17T07:34:15.498866+0000","last_peered":"2024-04-17T07:34:15.498866+0000","last_clean":"2024-04-17T07:34:15.498866+0000","last_became_active":"2024-04-17T07:34:14.505115+0000","last_became_peered":"2024-04-17T07:34:14.505115+0000","last_unstale":"2024-04-17T07:34:15.498866+0000","last_undegraded":"2024-04-17T07:34:15.498866+0000","last_fullsized":"2024-04-17T07:34:15.498866+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:21.459297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162829+0000","last_change":"2024-04-17T07:34:13.483643+0000","last_active":"2024-04-17T07:34:16.162829+0000","last_peered":"2024-04-17T07:34:16.162829+0000","last_clean":"2024-04-17T07:34:16.162829+0000","last_became_active":"2024-04-17T07:34:13.483376+0000","last_became_peered":"2024-04-17T07:34:13.483376+0000","last_unstale":"2024-04-17T07:34:16.162829+0000","last_undegraded":"2024-04-17T07:34:16.162829+0000","last_fullsized":"2024-04-17T07:34:16.162829+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:00:19.834138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487750+0000","last_change":"2024-04-17T07:34:14.483871+0000","last_active":"2024-04-17T07:34:14.487750+0000","last_peered":"2024-04-17T07:34:14.487750+0000","last_clean":"2024-04-17T07:34:14.487750+0000","last_became_active":"2024-04-17T07:34:14.483577+0000","last_became_peered":"2024-04-17T07:34:14.483577+0000","last_unstale":"2024-04-17T07:34:14.487750+0000","last_undegraded":"2024-04-17T07:34:14.487750+0000","last_fullsized":"2024-04-17T07:34:14.487750+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:03:41.153156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.474215+0000","last_change":"2024-04-17T07:34:13.479476+0000","last_active":"2024-04-17T07:34:14.474215+0000","last_peered":"2024-04-17T07:34:14.474215+0000","last_clean":"2024-04-17T07:34:14.474215+0000","last_became_active":"2024-04-17T07:34:13.479183+0000","last_became_peered":"2024-04-17T07:34:13.479183+0000","last_unstale":"2024-04-17T07:34:14.474215+0000","last_undegraded":"2024-04-17T07:34:14.474215+0000","last_fullsized":"2024-04-17T07:34:14.474215+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:13.519290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492461+0000","last_change":"2024-04-17T07:34:14.505725+0000","last_active":"2024-04-17T07:34:15.492461+0000","last_peered":"2024-04-17T07:34:15.492461+0000","last_clean":"2024-04-17T07:34:15.492461+0000","last_became_active":"2024-04-17T07:34:14.504587+0000","last_became_peered":"2024-04-17T07:34:14.504587+0000","last_unstale":"2024-04-17T07:34:15.492461+0000","last_undegraded":"2024-04-17T07:34:15.492461+0000","last_fullsized":"2024-04-17T07:34:15.492461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:00.551170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492718+0000","last_change":"2024-04-17T07:34:13.482454+0000","last_active":"2024-04-17T07:34:15.492718+0000","last_peered":"2024-04-17T07:34:15.492718+0000","last_clean":"2024-04-17T07:34:15.492718+0000","last_became_active":"2024-04-17T07:34:13.482132+0000","last_became_peered":"2024-04-17T07:34:13.482132+0000","last_unstale":"2024-04-17T07:34:15.492718+0000","last_undegraded":"2024-04-17T07:34:15.492718+0000","last_fullsized":"2024-04-17T07:34:15.492718+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:32:01.673108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162997+0000","last_change":"2024-04-17T07:34:14.510828+0000","last_active":"2024-04-17T07:34:16.162997+0000","last_peered":"2024-04-17T07:34:16.162997+0000","last_clean":"2024-04-17T07:34:16.162997+0000","last_became_active":"2024-04-17T07:34:14.508610+0000","last_became_peered":"2024-04-17T07:34:14.508610+0000","last_unstale":"2024-04-17T07:34:16.162997+0000","last_undegraded":"2024-04-17T07:34:16.162997+0000","last_fullsized":"2024-04-17T07:34:16.162997+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:55.174384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163251+0000","last_change":"2024-04-17T07:34:13.478067+0000","last_active":"2024-04-17T07:34:16.163251+0000","last_peered":"2024-04-17T07:34:16.163251+0000","last_clean":"2024-04-17T07:34:16.163251+0000","last_became_active":"2024-04-17T07:34:13.477615+0000","last_became_peered":"2024-04-17T07:34:13.477615+0000","last_unstale":"2024-04-17T07:34:16.163251+0000","last_undegraded":"2024-04-17T07:34:16.163251+0000","last_fullsized":"2024-04-17T07:34:16.163251+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:23.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162529+0000","last_change":"2024-04-17T07:34:14.508294+0000","last_active":"2024-04-17T07:34:16.162529+0000","last_peered":"2024-04-17T07:34:16.162529+0000","last_clean":"2024-04-17T07:34:16.162529+0000","last_became_active":"2024-04-17T07:34:14.507290+0000","last_became_peered":"2024-04-17T07:34:14.507290+0000","last_unstale":"2024-04-17T07:34:16.162529+0000","last_undegraded":"2024-04-17T07:34:16.162529+0000","last_fullsized":"2024-04-17T07:34:16.162529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:02.201102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.500424+0000","last_change":"2024-04-17T07:34:13.477688+0000","last_active":"2024-04-17T07:34:15.500424+0000","last_peered":"2024-04-17T07:34:15.500424+0000","last_clean":"2024-04-17T07:34:15.500424+0000","last_became_active":"2024-04-17T07:34:13.477392+0000","last_became_peered":"2024-04-17T07:34:13.477392+0000","last_unstale":"2024-04-17T07:34:15.500424+0000","last_undegraded":"2024-04-17T07:34:15.500424+0000","last_fullsized":"2024-04-17T07:34:15.500424+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:36.565251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499923+0000","last_change":"2024-04-17T07:34:14.480348+0000","last_active":"2024-04-17T07:34:15.499923+0000","last_peered":"2024-04-17T07:34:15.499923+0000","last_clean":"2024-04-17T07:34:15.499923+0000","last_became_active":"2024-04-17T07:34:14.477958+0000","last_became_peered":"2024-04-17T07:34:14.477958+0000","last_unstale":"2024-04-17T07:34:15.499923+0000","last_undegraded":"2024-04-17T07:34:15.499923+0000","last_fullsized":"2024-04-17T07:34:15.499923+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:12.161315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.507534+0000","last_change":"2024-04-17T07:34:13.481026+0000","last_active":"2024-04-17T07:34:14.507534+0000","last_peered":"2024-04-17T07:34:14.507534+0000","last_clean":"2024-04-17T07:34:14.507534+0000","last_became_active":"2024-04-17T07:34:13.480718+0000","last_became_peered":"2024-04-17T07:34:13.480718+0000","last_unstale":"2024-04-17T07:34:14.507534+0000","last_undegraded":"2024-04-17T07:34:14.507534+0000","last_fullsized":"2024-04-17T07:34:14.507534+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:08.693617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498797+0000","last_change":"2024-04-17T07:34:14.496241+0000","last_active":"2024-04-17T07:34:14.498797+0000","last_peered":"2024-04-17T07:34:14.498797+0000","last_clean":"2024-04-17T07:34:14.498797+0000","last_became_active":"2024-04-17T07:34:14.491459+0000","last_became_peered":"2024-04-17T07:34:14.491459+0000","last_unstale":"2024-04-17T07:34:14.498797+0000","last_undegraded":"2024-04-17T07:34:14.498797+0000","last_fullsized":"2024-04-17T07:34:14.498797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:07.074731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.505912+0000","last_change":"2024-04-17T07:34:13.484327+0000","last_active":"2024-04-17T07:34:14.505912+0000","last_peered":"2024-04-17T07:34:14.505912+0000","last_clean":"2024-04-17T07:34:14.505912+0000","last_became_active":"2024-04-17T07:34:13.483671+0000","last_became_peered":"2024-04-17T07:34:13.483671+0000","last_unstale":"2024-04-17T07:34:14.505912+0000","last_undegraded":"2024-04-17T07:34:14.505912+0000","last_fullsized":"2024-04-17T07:34:14.505912+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:48.233787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493476+0000","last_change":"2024-04-17T07:34:14.498280+0000","last_active":"2024-04-17T07:34:15.493476+0000","last_peered":"2024-04-17T07:34:15.493476+0000","last_clean":"2024-04-17T07:34:15.493476+0000","last_became_active":"2024-04-17T07:34:14.497962+0000","last_became_peered":"2024-04-17T07:34:14.497962+0000","last_unstale":"2024-04-17T07:34:15.493476+0000","last_undegraded":"2024-04-17T07:34:15.493476+0000","last_fullsized":"2024-04-17T07:34:15.493476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:26:21.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498679+0000","last_change":"2024-04-17T07:34:13.472996+0000","last_active":"2024-04-17T07:34:15.498679+0000","last_peered":"2024-04-17T07:34:15.498679+0000","last_clean":"2024-04-17T07:34:15.498679+0000","last_became_active":"2024-04-17T07:34:13.472698+0000","last_became_peered":"2024-04-17T07:34:13.472698+0000","last_unstale":"2024-04-17T07:34:15.498679+0000","last_undegraded":"2024-04-17T07:34:15.498679+0000","last_fullsized":"2024-04-17T07:34:15.498679+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:22.713668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.492878+0000","last_change":"2024-04-17T07:34:14.491035+0000","last_active":"2024-04-17T07:34:14.492878+0000","last_peered":"2024-04-17T07:34:14.492878+0000","last_clean":"2024-04-17T07:34:14.492878+0000","last_became_active":"2024-04-17T07:34:14.490682+0000","last_became_peered":"2024-04-17T07:34:14.490682+0000","last_unstale":"2024-04-17T07:34:14.492878+0000","last_undegraded":"2024-04-17T07:34:14.492878+0000","last_fullsized":"2024-04-17T07:34:14.492878+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:13.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498097+0000","last_change":"2024-04-17T07:34:13.476812+0000","last_active":"2024-04-17T07:34:15.498097+0000","last_peered":"2024-04-17T07:34:15.498097+0000","last_clean":"2024-04-17T07:34:15.498097+0000","last_became_active":"2024-04-17T07:34:13.476530+0000","last_became_peered":"2024-04-17T07:34:13.476530+0000","last_unstale":"2024-04-17T07:34:15.498097+0000","last_undegraded":"2024-04-17T07:34:15.498097+0000","last_fullsized":"2024-04-17T07:34:15.498097+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:18.129263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508438+0000","last_change":"2024-04-17T07:34:14.505410+0000","last_active":"2024-04-17T07:34:14.508438+0000","last_peered":"2024-04-17T07:34:14.508438+0000","last_clean":"2024-04-17T07:34:14.508438+0000","last_became_active":"2024-04-17T07:34:14.502610+0000","last_became_peered":"2024-04-17T07:34:14.502610+0000","last_unstale":"2024-04-17T07:34:14.508438+0000","last_undegraded":"2024-04-17T07:34:14.508438+0000","last_fullsized":"2024-04-17T07:34:14.508438+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:46.711580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.486433+0000","last_change":"2024-04-17T07:34:13.476281+0000","last_active":"2024-04-17T07:34:14.486433+0000","last_peered":"2024-04-17T07:34:14.486433+0000","last_clean":"2024-04-17T07:34:14.486433+0000","last_became_active":"2024-04-17T07:34:13.475920+0000","last_became_peered":"2024-04-17T07:34:13.475920+0000","last_unstale":"2024-04-17T07:34:14.486433+0000","last_undegraded":"2024-04-17T07:34:14.486433+0000","last_fullsized":"2024-04-17T07:34:14.486433+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:11.067689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492938+0000","last_change":"2024-04-17T07:34:14.484831+0000","last_active":"2024-04-17T07:34:15.492938+0000","last_peered":"2024-04-17T07:34:15.492938+0000","last_clean":"2024-04-17T07:34:15.492938+0000","last_became_active":"2024-04-17T07:34:14.482465+0000","last_became_peered":"2024-04-17T07:34:14.482465+0000","last_unstale":"2024-04-17T07:34:15.492938+0000","last_undegraded":"2024-04-17T07:34:15.492938+0000","last_fullsized":"2024-04-17T07:34:15.492938+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:47:10.347613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.485804+0000","last_change":"2024-04-17T07:34:13.465034+0000","last_active":"2024-04-17T07:34:14.485804+0000","last_peered":"2024-04-17T07:34:14.485804+0000","last_clean":"2024-04-17T07:34:14.485804+0000","last_became_active":"2024-04-17T07:34:13.464594+0000","last_became_peered":"2024-04-17T07:34:13.464594+0000","last_unstale":"2024-04-17T07:34:14.485804+0000","last_undegraded":"2024-04-17T07:34:14.485804+0000","last_fullsized":"2024-04-17T07:34:14.485804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:26.285361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497789+0000","last_change":"2024-04-17T07:34:14.502214+0000","last_active":"2024-04-17T07:34:15.497789+0000","last_peered":"2024-04-17T07:34:15.497789+0000","last_clean":"2024-04-17T07:34:15.497789+0000","last_became_active":"2024-04-17T07:34:14.501182+0000","last_became_peered":"2024-04-17T07:34:14.501182+0000","last_unstale":"2024-04-17T07:34:15.497789+0000","last_undegraded":"2024-04-17T07:34:15.497789+0000","last_fullsized":"2024-04-17T07:34:15.497789+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:04.044214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.467181+0000","last_change":"2024-04-17T07:34:13.481675+0000","last_active":"2024-04-17T07:34:14.467181+0000","last_peered":"2024-04-17T07:34:14.467181+0000","last_clean":"2024-04-17T07:34:14.467181+0000","last_became_active":"2024-04-17T07:34:13.480731+0000","last_became_peered":"2024-04-17T07:34:13.480731+0000","last_unstale":"2024-04-17T07:34:14.467181+0000","last_undegraded":"2024-04-17T07:34:14.467181+0000","last_fullsized":"2024-04-17T07:34:14.467181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:25.190166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.500644+0000","last_change":"2024-04-17T07:34:14.496746+0000","last_active":"2024-04-17T07:34:14.500644+0000","last_peered":"2024-04-17T07:34:14.500644+0000","last_clean":"2024-04-17T07:34:14.500644+0000","last_became_active":"2024-04-17T07:34:14.493956+0000","last_became_peered":"2024-04-17T07:34:14.493956+0000","last_unstale":"2024-04-17T07:34:14.500644+0000","last_undegraded":"2024-04-17T07:34:14.500644+0000","last_fullsized":"2024-04-17T07:34:14.500644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:38.779800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.503340+0000","last_change":"2024-04-17T07:34:13.477341+0000","last_active":"2024-04-17T07:34:14.503340+0000","last_peered":"2024-04-17T07:34:14.503340+0000","last_clean":"2024-04-17T07:34:14.503340+0000","last_became_active":"2024-04-17T07:34:13.476859+0000","last_became_peered":"2024-04-17T07:34:13.476859+0000","last_unstale":"2024-04-17T07:34:14.503340+0000","last_undegraded":"2024-04-17T07:34:14.503340+0000","last_fullsized":"2024-04-17T07:34:14.503340+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:55:24.471927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.507355+0000","last_change":"2024-04-17T07:34:14.504637+0000","last_active":"2024-04-17T07:34:14.507355+0000","last_peered":"2024-04-17T07:34:14.507355+0000","last_clean":"2024-04-17T07:34:14.507355+0000","last_became_active":"2024-04-17T07:34:14.504366+0000","last_became_peered":"2024-04-17T07:34:14.504366+0000","last_unstale":"2024-04-17T07:34:14.507355+0000","last_undegraded":"2024-04-17T07:34:14.507355+0000","last_fullsized":"2024-04-17T07:34:14.507355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:36.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498316+0000","last_change":"2024-04-17T07:34:13.480375+0000","last_active":"2024-04-17T07:34:15.498316+0000","last_peered":"2024-04-17T07:34:15.498316+0000","last_clean":"2024-04-17T07:34:15.498316+0000","last_became_active":"2024-04-17T07:34:13.479603+0000","last_became_peered":"2024-04-17T07:34:13.479603+0000","last_unstale":"2024-04-17T07:34:15.498316+0000","last_undegraded":"2024-04-17T07:34:15.498316+0000","last_fullsized":"2024-04-17T07:34:15.498316+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:30.528110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493045+0000","last_change":"2024-04-17T07:34:14.505811+0000","last_active":"2024-04-17T07:34:15.493045+0000","last_peered":"2024-04-17T07:34:15.493045+0000","last_clean":"2024-04-17T07:34:15.493045+0000","last_became_active":"2024-04-17T07:34:14.504223+0000","last_became_peered":"2024-04-17T07:34:14.504223+0000","last_unstale":"2024-04-17T07:34:15.493045+0000","last_undegraded":"2024-04-17T07:34:15.493045+0000","last_fullsized":"2024-04-17T07:34:15.493045+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:30.622147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493423+0000","last_change":"2024-04-17T07:34:13.479526+0000","last_active":"2024-04-17T07:34:15.493423+0000","last_peered":"2024-04-17T07:34:15.493423+0000","last_clean":"2024-04-17T07:34:15.493423+0000","last_became_active":"2024-04-17T07:34:13.479202+0000","last_became_peered":"2024-04-17T07:34:13.479202+0000","last_unstale":"2024-04-17T07:34:15.493423+0000","last_undegraded":"2024-04-17T07:34:15.493423+0000","last_fullsized":"2024-04-17T07:34:15.493423+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:10.743862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498653+0000","last_change":"2024-04-17T07:34:14.487588+0000","last_active":"2024-04-17T07:34:14.498653+0000","last_peered":"2024-04-17T07:34:14.498653+0000","last_clean":"2024-04-17T07:34:14.498653+0000","last_became_active":"2024-04-17T07:34:14.486041+0000","last_became_peered":"2024-04-17T07:34:14.486041+0000","last_unstale":"2024-04-17T07:34:14.498653+0000","last_undegraded":"2024-04-17T07:34:14.498653+0000","last_fullsized":"2024-04-17T07:34:14.498653+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:26.558713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.466967+0000","last_change":"2024-04-17T07:34:13.474614+0000","last_active":"2024-04-17T07:34:14.466967+0000","last_peered":"2024-04-17T07:34:14.466967+0000","last_clean":"2024-04-17T07:34:14.466967+0000","last_became_active":"2024-04-17T07:34:13.474090+0000","last_became_peered":"2024-04-17T07:34:13.474090+0000","last_unstale":"2024-04-17T07:34:14.466967+0000","last_undegraded":"2024-04-17T07:34:14.466967+0000","last_fullsized":"2024-04-17T07:34:14.466967+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:33.495237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499088+0000","last_change":"2024-04-17T07:34:14.505437+0000","last_active":"2024-04-17T07:34:15.499088+0000","last_peered":"2024-04-17T07:34:15.499088+0000","last_clean":"2024-04-17T07:34:15.499088+0000","last_became_active":"2024-04-17T07:34:14.504257+0000","last_became_peered":"2024-04-17T07:34:14.504257+0000","last_unstale":"2024-04-17T07:34:15.499088+0000","last_undegraded":"2024-04-17T07:34:15.499088+0000","last_fullsized":"2024-04-17T07:34:15.499088+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:27.512930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492701+0000","last_change":"2024-04-17T07:34:13.483214+0000","last_active":"2024-04-17T07:34:15.492701+0000","last_peered":"2024-04-17T07:34:15.492701+0000","last_clean":"2024-04-17T07:34:15.492701+0000","last_became_active":"2024-04-17T07:34:13.482919+0000","last_became_peered":"2024-04-17T07:34:13.482919+0000","last_unstale":"2024-04-17T07:34:15.492701+0000","last_undegraded":"2024-04-17T07:34:15.492701+0000","last_fullsized":"2024-04-17T07:34:15.492701+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:01.183750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497827+0000","last_change":"2024-04-17T07:34:14.483307+0000","last_active":"2024-04-17T07:34:15.497827+0000","last_peered":"2024-04-17T07:34:15.497827+0000","last_clean":"2024-04-17T07:34:15.497827+0000","last_became_active":"2024-04-17T07:34:14.482392+0000","last_became_peered":"2024-04-17T07:34:14.482392+0000","last_unstale":"2024-04-17T07:34:15.497827+0000","last_undegraded":"2024-04-17T07:34:15.497827+0000","last_fullsized":"2024-04-17T07:34:15.497827+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:48.360284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499457+0000","last_change":"2024-04-17T07:34:13.477016+0000","last_active":"2024-04-17T07:34:15.499457+0000","last_peered":"2024-04-17T07:34:15.499457+0000","last_clean":"2024-04-17T07:34:15.499457+0000","last_became_active":"2024-04-17T07:34:13.476702+0000","last_became_peered":"2024-04-17T07:34:13.476702+0000","last_unstale":"2024-04-17T07:34:15.499457+0000","last_undegraded":"2024-04-17T07:34:15.499457+0000","last_fullsized":"2024-04-17T07:34:15.499457+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:06.057379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508354+0000","last_change":"2024-04-17T07:34:14.506315+0000","last_active":"2024-04-17T07:34:14.508354+0000","last_peered":"2024-04-17T07:34:14.508354+0000","last_clean":"2024-04-17T07:34:14.508354+0000","last_became_active":"2024-04-17T07:34:14.505661+0000","last_became_peered":"2024-04-17T07:34:14.505661+0000","last_unstale":"2024-04-17T07:34:14.508354+0000","last_undegraded":"2024-04-17T07:34:14.508354+0000","last_fullsized":"2024-04-17T07:34:14.508354+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.921844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.485970+0000","last_change":"2024-04-17T07:34:13.476036+0000","last_active":"2024-04-17T07:34:14.485970+0000","last_peered":"2024-04-17T07:34:14.485970+0000","last_clean":"2024-04-17T07:34:14.485970+0000","last_became_active":"2024-04-17T07:34:13.475741+0000","last_became_peered":"2024-04-17T07:34:13.475741+0000","last_unstale":"2024-04-17T07:34:14.485970+0000","last_undegraded":"2024-04-17T07:34:14.485970+0000","last_fullsized":"2024-04-17T07:34:14.485970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:50.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492550+0000","last_change":"2024-04-17T07:34:14.495839+0000","last_active":"2024-04-17T07:34:15.492550+0000","last_peered":"2024-04-17T07:34:15.492550+0000","last_clean":"2024-04-17T07:34:15.492550+0000","last_became_active":"2024-04-17T07:34:14.495501+0000","last_became_peered":"2024-04-17T07:34:14.495501+0000","last_unstale":"2024-04-17T07:34:15.492550+0000","last_undegraded":"2024-04-17T07:34:15.492550+0000","last_fullsized":"2024-04-17T07:34:15.492550+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:26.044682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.467243+0000","last_change":"2024-04-17T07:34:13.481482+0000","last_active":"2024-04-17T07:34:14.467243+0000","last_peered":"2024-04-17T07:34:14.467243+0000","last_clean":"2024-04-17T07:34:14.467243+0000","last_became_active":"2024-04-17T07:34:13.480395+0000","last_became_peered":"2024-04-17T07:34:13.480395+0000","last_unstale":"2024-04-17T07:34:14.467243+0000","last_undegraded":"2024-04-17T07:34:14.467243+0000","last_fullsized":"2024-04-17T07:34:14.467243+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:54.475442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498726+0000","last_change":"2024-04-17T07:34:14.502324+0000","last_active":"2024-04-17T07:34:15.498726+0000","last_peered":"2024-04-17T07:34:15.498726+0000","last_clean":"2024-04-17T07:34:15.498726+0000","last_became_active":"2024-04-17T07:34:14.501644+0000","last_became_peered":"2024-04-17T07:34:14.501644+0000","last_unstale":"2024-04-17T07:34:15.498726+0000","last_undegraded":"2024-04-17T07:34:15.498726+0000","last_fullsized":"2024-04-17T07:34:15.498726+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:13.647727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498032+0000","last_change":"2024-04-17T07:34:13.479096+0000","last_active":"2024-04-17T07:34:15.498032+0000","last_peered":"2024-04-17T07:34:15.498032+0000","last_clean":"2024-04-17T07:34:15.498032+0000","last_became_active":"2024-04-17T07:34:13.478783+0000","last_became_peered":"2024-04-17T07:34:13.478783+0000","last_unstale":"2024-04-17T07:34:15.498032+0000","last_undegraded":"2024-04-17T07:34:15.498032+0000","last_fullsized":"2024-04-17T07:34:15.498032+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:49.336892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"144'255","reported_seq":920,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.496656+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:34:15.496656+0000","last_peered":"2024-04-17T07:34:15.496656+0000","last_clean":"2024-04-17T07:34:15.496656+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:34:15.496656+0000","last_undegraded":"2024-04-17T07:34:15.496656+0000","last_fullsized":"2024-04-17T07:34:15.496656+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":407,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":407,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":255,"ondisk_log_size":255,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542384,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37840,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334000,"statfs":{"total":96636764160,"available":96598016000,"internally_reserved":0,"allocated":2555904,"data_stored":1525673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542384,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":928339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542383,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1933312,"data_stored":926839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542383,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59440,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312400,"statfs":{"total":96636764160,"available":96575897600,"internally_reserved":0,"allocated":1929216,"data_stored":926797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542385,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":926887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542384,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59436,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312404,"statfs":{"total":96636764160,"available":96575901696,"internally_reserved":0,"allocated":1933312,"data_stored":927343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542385,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55284,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316556,"statfs":{"total":96636764160,"available":96580153344,"internally_reserved":0,"allocated":1937408,"data_stored":933055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542383,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37800,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334040,"statfs":{"total":96636764160,"available":96598056960,"internally_reserved":0,"allocated":2519040,"data_stored":1517095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:34:19.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T07:34:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.299+0000 7fd9e5000640 1 Processor -- start 2024-04-17T07:34:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.299+0000 7fd9e5000640 1 -- start start 2024-04-17T07:34:19.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.301+0000 7fd9e5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.301+0000 7fd9e5000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9e0160280 0x7fd9e006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.301+0000 7fd9e5000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e00cd4a0 con 0x7fd9e0141ed0 2024-04-17T07:34:19.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.301+0000 7fd9e5000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e015ce30 con 0x7fd9e0160280 2024-04-17T07:34:19.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.301+0000 7fd9e5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e01607f0 con 0x7fd9e006b8c0 2024-04-17T07:34:19.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:59596/0 (socket says 172.21.15.67:59596) 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 -- 172.21.15.67:0/2971693739 learned_addr learned my addr 172.21.15.67:0/2971693739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9dea00640 1 --2- 172.21.15.67:0/2971693739 >> v2:172.21.15.187:3300/0 conn(0x7fd9e0160280 0x7fd9e006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 -- 172.21.15.67:0/2971693739 >> v2:172.21.15.187:3300/0 conn(0x7fd9e0160280 msgr2=0x7fd9e006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 --2- 172.21.15.67:0/2971693739 >> v2:172.21.15.187:3300/0 conn(0x7fd9e0160280 0x7fd9e006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 -- 172.21.15.67:0/2971693739 >> v1:172.21.15.187:6789/0 conn(0x7fd9e0141ed0 legacy=0x7fd9e015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.302+0000 7fd9df400640 1 -- 172.21.15.67:0/2971693739 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e0160ad0 con 0x7fd9e006b8c0 2024-04-17T07:34:19.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.304+0000 7fd9df400640 1 --2- 172.21.15.67:0/2971693739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009870 tx=0x7fd9d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30602a48b1700760 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9de000640 1 -- 172.21.15.67:0/2971693739 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0034070 con 0x7fd9e006b8c0 2024-04-17T07:34:19.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9de000640 1 -- 172.21.15.67:0/2971693739 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0030840 con 0x7fd9e006b8c0 2024-04-17T07:34:19.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9de000640 1 -- 172.21.15.67:0/2971693739 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0030c10 con 0x7fd9e006b8c0 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 -- 172.21.15.67:0/2971693739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 msgr2=0x7fd9e00cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 --2- 172.21.15.67:0/2971693739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009870 tx=0x7fd9d002b980 comp rx=0 tx=0).stop 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 -- 172.21.15.67:0/2971693739 shutdown_connections 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 --2- 172.21.15.67:0/2971693739 >> v2:172.21.15.187:3300/0 conn(0x7fd9e0160280 0x7fd9e006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 --2- 172.21.15.67:0/2971693739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e006b8c0 0x7fd9e00cd8b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 -- 172.21.15.67:0/2971693739 >> 172.21.15.67:0/2971693739 conn(0x7fd9e0067d10 msgr2=0x7fd9e00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 -- 172.21.15.67:0/2971693739 shutdown_connections 2024-04-17T07:34:19.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.305+0000 7fd9e5000640 1 -- 172.21.15.67:0/2971693739 wait complete. 2024-04-17T07:34:19.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.307+0000 7fd9e5000640 1 Processor -- start 2024-04-17T07:34:19.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.307+0000 7fd9e5000640 1 -- start start 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e006b8c0 0x7fd9e00cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e0160280 0x7fd9e015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e006c1e0 con 0x7fd9e0141ed0 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e00cddf0 con 0x7fd9e006b8c0 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.308+0000 7fd9e5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e0161090 con 0x7fd9e0160280 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60770/0 (socket says 172.21.15.67:60770) 2024-04-17T07:34:19.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 -- 172.21.15.67:0/351689375 learned_addr learned my addr 172.21.15.67:0/351689375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:19.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9df400640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e006b8c0 0x7fd9e00cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dfe00640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e0160280 0x7fd9e015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 -- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e0160280 msgr2=0x7fd9e015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e0160280 0x7fd9e015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 -- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e006b8c0 msgr2=0x7fd9e00cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e006b8c0 0x7fd9e00cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.309+0000 7fd9dea00640 1 -- 172.21.15.67:0/351689375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e00cd060 con 0x7fd9e0141ed0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9dea00640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00e870 tx=0x7fd9cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc017070 con 0x7fd9e0141ed0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9cc00c940 con 0x7fd9e0141ed0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc00cd30 con 0x7fd9e0141ed0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e00ccd80 con 0x7fd9e0141ed0 2024-04-17T07:34:19.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.310+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e0144c90 con 0x7fd9e0141ed0 2024-04-17T07:34:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.311+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e0068a10 con 0x7fd9e0141ed0 2024-04-17T07:34:19.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.312+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fd9cc01a070 con 0x7fd9e0141ed0 2024-04-17T07:34:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.313+0000 7fd9d7400640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 0x7fd9c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.314+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fd9cc08bbc0 con 0x7fd9e0141ed0 2024-04-17T07:34:19.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.314+0000 7fd9df400640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 0x7fd9c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.315+0000 7fd9df400640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 0x7fd9c006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0036000 tx=0x7fd9d0008030 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.318+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd9cc0597d0 con 0x7fd9e0141ed0 2024-04-17T07:34:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.525+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 --> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9e015f4b0 con 0x7fd9c006ca10 2024-04-17T07:34:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.545+0000 7fd9d7400640 1 -- 172.21.15.67:0/351689375 <== mgr.7087 v2:172.21.15.187:6832/115904466 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252454 (secure 0 0 0) 0x7fd9d0030810 con 0x7fd9c006ca10 2024-04-17T07:34:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:19.551 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 msgr2=0x7fd9c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 0x7fd9c006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0036000 tx=0x7fd9d0008030 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 msgr2=0x7fd9e013cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00e870 tx=0x7fd9cc00ed30 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 shutdown_connections 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd9c006ca10 0x7fd9c006eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9e0160280 0x7fd9e015ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9e0141ed0 0x7fd9e013cf60 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 --2- 172.21.15.67:0/351689375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9e006b8c0 0x7fd9e00cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 >> 172.21.15.67:0/351689375 conn(0x7fd9e0067d10 msgr2=0x7fd9e013ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 shutdown_connections 2024-04-17T07:34:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.552+0000 7fd9e5000640 1 -- 172.21.15.67:0/351689375 wait complete. 2024-04-17T07:34:19.572 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-17T07:34:19.311189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":451,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":361496,"kb_used_data":16288,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754613224,"statfs":{"total":773094113280,"available":772723941376,"internally_reserved":0,"allocated":16678912,"data_stored":8612028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":353357360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002128"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.506426+0000","last_change":"2024-04-17T07:34:14.502350+0000","last_active":"2024-04-17T07:34:14.506426+0000","last_peered":"2024-04-17T07:34:14.506426+0000","last_clean":"2024-04-17T07:34:14.506426+0000","last_became_active":"2024-04-17T07:34:14.500560+0000","last_became_peered":"2024-04-17T07:34:14.500560+0000","last_unstale":"2024-04-17T07:34:14.506426+0000","last_undegraded":"2024-04-17T07:34:14.506426+0000","last_fullsized":"2024-04-17T07:34:14.506426+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:00:15.119442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493549+0000","last_change":"2024-04-17T07:34:14.501572+0000","last_active":"2024-04-17T07:34:15.493549+0000","last_peered":"2024-04-17T07:34:15.493549+0000","last_clean":"2024-04-17T07:34:15.493549+0000","last_became_active":"2024-04-17T07:34:14.500910+0000","last_became_peered":"2024-04-17T07:34:14.500910+0000","last_unstale":"2024-04-17T07:34:15.493549+0000","last_undegraded":"2024-04-17T07:34:15.493549+0000","last_fullsized":"2024-04-17T07:34:15.493549+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:46.790600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162898+0000","last_change":"2024-04-17T07:34:14.507382+0000","last_active":"2024-04-17T07:34:16.162898+0000","last_peered":"2024-04-17T07:34:16.162898+0000","last_clean":"2024-04-17T07:34:16.162898+0000","last_became_active":"2024-04-17T07:34:14.502526+0000","last_became_peered":"2024-04-17T07:34:14.502526+0000","last_unstale":"2024-04-17T07:34:16.162898+0000","last_undegraded":"2024-04-17T07:34:16.162898+0000","last_fullsized":"2024-04-17T07:34:16.162898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:43.844557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.501137+0000","last_change":"2024-04-17T07:34:14.497350+0000","last_active":"2024-04-17T07:34:14.501137+0000","last_peered":"2024-04-17T07:34:14.501137+0000","last_clean":"2024-04-17T07:34:14.501137+0000","last_became_active":"2024-04-17T07:34:14.496296+0000","last_became_peered":"2024-04-17T07:34:14.496296+0000","last_unstale":"2024-04-17T07:34:14.501137+0000","last_undegraded":"2024-04-17T07:34:14.501137+0000","last_fullsized":"2024-04-17T07:34:14.501137+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:28.955882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497811+0000","last_change":"2024-04-17T07:34:14.510852+0000","last_active":"2024-04-17T07:34:15.497811+0000","last_peered":"2024-04-17T07:34:15.497811+0000","last_clean":"2024-04-17T07:34:15.497811+0000","last_became_active":"2024-04-17T07:34:14.510538+0000","last_became_peered":"2024-04-17T07:34:14.510538+0000","last_unstale":"2024-04-17T07:34:15.497811+0000","last_undegraded":"2024-04-17T07:34:15.497811+0000","last_fullsized":"2024-04-17T07:34:15.497811+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:07.609244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.499236+0000","last_change":"2024-04-17T07:34:14.487905+0000","last_active":"2024-04-17T07:34:14.499236+0000","last_peered":"2024-04-17T07:34:14.499236+0000","last_clean":"2024-04-17T07:34:14.499236+0000","last_became_active":"2024-04-17T07:34:14.487568+0000","last_became_peered":"2024-04-17T07:34:14.487568+0000","last_unstale":"2024-04-17T07:34:14.499236+0000","last_undegraded":"2024-04-17T07:34:14.499236+0000","last_fullsized":"2024-04-17T07:34:14.499236+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:20:01.858330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508441+0000","last_change":"2024-04-17T07:34:14.503113+0000","last_active":"2024-04-17T07:34:14.508441+0000","last_peered":"2024-04-17T07:34:14.508441+0000","last_clean":"2024-04-17T07:34:14.508441+0000","last_became_active":"2024-04-17T07:34:14.502798+0000","last_became_peered":"2024-04-17T07:34:14.502798+0000","last_unstale":"2024-04-17T07:34:14.508441+0000","last_undegraded":"2024-04-17T07:34:14.508441+0000","last_fullsized":"2024-04-17T07:34:14.508441+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:19.146614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163047+0000","last_change":"2024-04-17T07:34:14.503259+0000","last_active":"2024-04-17T07:34:16.163047+0000","last_peered":"2024-04-17T07:34:16.163047+0000","last_clean":"2024-04-17T07:34:16.163047+0000","last_became_active":"2024-04-17T07:34:14.502903+0000","last_became_peered":"2024-04-17T07:34:14.502903+0000","last_unstale":"2024-04-17T07:34:16.163047+0000","last_undegraded":"2024-04-17T07:34:16.163047+0000","last_fullsized":"2024-04-17T07:34:16.163047+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:13.115027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163452+0000","last_change":"2024-04-17T07:34:14.508623+0000","last_active":"2024-04-17T07:34:16.163452+0000","last_peered":"2024-04-17T07:34:16.163452+0000","last_clean":"2024-04-17T07:34:16.163452+0000","last_became_active":"2024-04-17T07:34:14.506735+0000","last_became_peered":"2024-04-17T07:34:14.506735+0000","last_unstale":"2024-04-17T07:34:16.163452+0000","last_undegraded":"2024-04-17T07:34:16.163452+0000","last_fullsized":"2024-04-17T07:34:16.163452+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:48.420144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493740+0000","last_change":"2024-04-17T07:34:14.502447+0000","last_active":"2024-04-17T07:34:15.493740+0000","last_peered":"2024-04-17T07:34:15.493740+0000","last_clean":"2024-04-17T07:34:15.493740+0000","last_became_active":"2024-04-17T07:34:14.501540+0000","last_became_peered":"2024-04-17T07:34:14.501540+0000","last_unstale":"2024-04-17T07:34:15.493740+0000","last_undegraded":"2024-04-17T07:34:15.493740+0000","last_fullsized":"2024-04-17T07:34:15.493740+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:33.909583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494772+0000","last_change":"2024-04-17T07:34:14.489911+0000","last_active":"2024-04-17T07:34:14.494772+0000","last_peered":"2024-04-17T07:34:14.494772+0000","last_clean":"2024-04-17T07:34:14.494772+0000","last_became_active":"2024-04-17T07:34:14.489627+0000","last_became_peered":"2024-04-17T07:34:14.489627+0000","last_unstale":"2024-04-17T07:34:14.494772+0000","last_undegraded":"2024-04-17T07:34:14.494772+0000","last_fullsized":"2024-04-17T07:34:14.494772+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:26.284679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162601+0000","last_change":"2024-04-17T07:34:14.509249+0000","last_active":"2024-04-17T07:34:16.162601+0000","last_peered":"2024-04-17T07:34:16.162601+0000","last_clean":"2024-04-17T07:34:16.162601+0000","last_became_active":"2024-04-17T07:34:14.506750+0000","last_became_peered":"2024-04-17T07:34:14.506750+0000","last_unstale":"2024-04-17T07:34:16.162601+0000","last_undegraded":"2024-04-17T07:34:16.162601+0000","last_fullsized":"2024-04-17T07:34:16.162601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:11.761214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.490448+0000","last_change":"2024-04-17T07:34:14.488978+0000","last_active":"2024-04-17T07:34:14.490448+0000","last_peered":"2024-04-17T07:34:14.490448+0000","last_clean":"2024-04-17T07:34:14.490448+0000","last_became_active":"2024-04-17T07:34:14.488706+0000","last_became_peered":"2024-04-17T07:34:14.488706+0000","last_unstale":"2024-04-17T07:34:14.490448+0000","last_undegraded":"2024-04-17T07:34:14.490448+0000","last_fullsized":"2024-04-17T07:34:14.490448+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:06.803534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492604+0000","last_change":"2024-04-17T07:34:14.486803+0000","last_active":"2024-04-17T07:34:15.492604+0000","last_peered":"2024-04-17T07:34:15.492604+0000","last_clean":"2024-04-17T07:34:15.492604+0000","last_became_active":"2024-04-17T07:34:14.486516+0000","last_became_peered":"2024-04-17T07:34:14.486516+0000","last_unstale":"2024-04-17T07:34:15.492604+0000","last_undegraded":"2024-04-17T07:34:15.492604+0000","last_fullsized":"2024-04-17T07:34:15.492604+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:26:58.611392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508097+0000","last_change":"2024-04-17T07:34:14.506224+0000","last_active":"2024-04-17T07:34:14.508097+0000","last_peered":"2024-04-17T07:34:14.508097+0000","last_clean":"2024-04-17T07:34:14.508097+0000","last_became_active":"2024-04-17T07:34:14.505109+0000","last_became_peered":"2024-04-17T07:34:14.505109+0000","last_unstale":"2024-04-17T07:34:14.508097+0000","last_undegraded":"2024-04-17T07:34:14.508097+0000","last_fullsized":"2024-04-17T07:34:14.508097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:59.260260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.496461+0000","last_change":"2024-04-17T07:34:14.492852+0000","last_active":"2024-04-17T07:34:14.496461+0000","last_peered":"2024-04-17T07:34:14.496461+0000","last_clean":"2024-04-17T07:34:14.496461+0000","last_became_active":"2024-04-17T07:34:14.492498+0000","last_became_peered":"2024-04-17T07:34:14.492498+0000","last_unstale":"2024-04-17T07:34:14.496461+0000","last_undegraded":"2024-04-17T07:34:14.496461+0000","last_fullsized":"2024-04-17T07:34:14.496461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:27.692060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.497635+0000","last_change":"2024-04-17T07:34:14.487588+0000","last_active":"2024-04-17T07:34:14.497635+0000","last_peered":"2024-04-17T07:34:14.497635+0000","last_clean":"2024-04-17T07:34:14.497635+0000","last_became_active":"2024-04-17T07:34:14.486738+0000","last_became_peered":"2024-04-17T07:34:14.486738+0000","last_unstale":"2024-04-17T07:34:14.497635+0000","last_undegraded":"2024-04-17T07:34:14.497635+0000","last_fullsized":"2024-04-17T07:34:14.497635+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:52:41.869677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.505394+0000","last_change":"2024-04-17T07:34:14.501661+0000","last_active":"2024-04-17T07:34:14.505394+0000","last_peered":"2024-04-17T07:34:14.505394+0000","last_clean":"2024-04-17T07:34:14.505394+0000","last_became_active":"2024-04-17T07:34:14.501148+0000","last_became_peered":"2024-04-17T07:34:14.501148+0000","last_unstale":"2024-04-17T07:34:14.505394+0000","last_undegraded":"2024-04-17T07:34:14.505394+0000","last_fullsized":"2024-04-17T07:34:14.505394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:31.545462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.496393+0000","last_change":"2024-04-17T07:34:14.493010+0000","last_active":"2024-04-17T07:34:14.496393+0000","last_peered":"2024-04-17T07:34:14.496393+0000","last_clean":"2024-04-17T07:34:14.496393+0000","last_became_active":"2024-04-17T07:34:14.492703+0000","last_became_peered":"2024-04-17T07:34:14.492703+0000","last_unstale":"2024-04-17T07:34:14.496393+0000","last_undegraded":"2024-04-17T07:34:14.496393+0000","last_fullsized":"2024-04-17T07:34:14.496393+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:05.976469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498776+0000","last_change":"2024-04-17T07:34:14.495259+0000","last_active":"2024-04-17T07:34:14.498776+0000","last_peered":"2024-04-17T07:34:14.498776+0000","last_clean":"2024-04-17T07:34:14.498776+0000","last_became_active":"2024-04-17T07:34:14.494917+0000","last_became_peered":"2024-04-17T07:34:14.494917+0000","last_unstale":"2024-04-17T07:34:14.498776+0000","last_undegraded":"2024-04-17T07:34:14.498776+0000","last_fullsized":"2024-04-17T07:34:14.498776+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:25:49.137506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497468+0000","last_change":"2024-04-17T07:34:14.507005+0000","last_active":"2024-04-17T07:34:15.497468+0000","last_peered":"2024-04-17T07:34:15.497468+0000","last_clean":"2024-04-17T07:34:15.497468+0000","last_became_active":"2024-04-17T07:34:14.506727+0000","last_became_peered":"2024-04-17T07:34:14.506727+0000","last_unstale":"2024-04-17T07:34:15.497468+0000","last_undegraded":"2024-04-17T07:34:15.497468+0000","last_fullsized":"2024-04-17T07:34:15.497468+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:25.841542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487221+0000","last_change":"2024-04-17T07:34:14.483773+0000","last_active":"2024-04-17T07:34:14.487221+0000","last_peered":"2024-04-17T07:34:14.487221+0000","last_clean":"2024-04-17T07:34:14.487221+0000","last_became_active":"2024-04-17T07:34:14.483471+0000","last_became_peered":"2024-04-17T07:34:14.483471+0000","last_unstale":"2024-04-17T07:34:14.487221+0000","last_undegraded":"2024-04-17T07:34:14.487221+0000","last_fullsized":"2024-04-17T07:34:14.487221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:06.146869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163002+0000","last_change":"2024-04-17T07:34:14.510582+0000","last_active":"2024-04-17T07:34:16.163002+0000","last_peered":"2024-04-17T07:34:16.163002+0000","last_clean":"2024-04-17T07:34:16.163002+0000","last_became_active":"2024-04-17T07:34:14.510042+0000","last_became_peered":"2024-04-17T07:34:14.510042+0000","last_unstale":"2024-04-17T07:34:16.163002+0000","last_undegraded":"2024-04-17T07:34:16.163002+0000","last_fullsized":"2024-04-17T07:34:16.163002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:18.855740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.493492+0000","last_change":"2024-04-17T07:34:14.486145+0000","last_active":"2024-04-17T07:34:14.493492+0000","last_peered":"2024-04-17T07:34:14.493492+0000","last_clean":"2024-04-17T07:34:14.493492+0000","last_became_active":"2024-04-17T07:34:14.485860+0000","last_became_peered":"2024-04-17T07:34:14.485860+0000","last_unstale":"2024-04-17T07:34:14.493492+0000","last_undegraded":"2024-04-17T07:34:14.493492+0000","last_fullsized":"2024-04-17T07:34:14.493492+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:22.319028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493557+0000","last_change":"2024-04-17T07:34:14.505331+0000","last_active":"2024-04-17T07:34:15.493557+0000","last_peered":"2024-04-17T07:34:15.493557+0000","last_clean":"2024-04-17T07:34:15.493557+0000","last_became_active":"2024-04-17T07:34:14.504964+0000","last_became_peered":"2024-04-17T07:34:14.504964+0000","last_unstale":"2024-04-17T07:34:15.493557+0000","last_undegraded":"2024-04-17T07:34:15.493557+0000","last_fullsized":"2024-04-17T07:34:15.493557+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:18.021895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498555+0000","last_change":"2024-04-17T07:34:14.495963+0000","last_active":"2024-04-17T07:34:14.498555+0000","last_peered":"2024-04-17T07:34:14.498555+0000","last_clean":"2024-04-17T07:34:14.498555+0000","last_became_active":"2024-04-17T07:34:14.495650+0000","last_became_peered":"2024-04-17T07:34:14.495650+0000","last_unstale":"2024-04-17T07:34:14.498555+0000","last_undegraded":"2024-04-17T07:34:14.498555+0000","last_fullsized":"2024-04-17T07:34:14.498555+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:06.385241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.506872+0000","last_change":"2024-04-17T07:34:14.502304+0000","last_active":"2024-04-17T07:34:14.506872+0000","last_peered":"2024-04-17T07:34:14.506872+0000","last_clean":"2024-04-17T07:34:14.506872+0000","last_became_active":"2024-04-17T07:34:14.502030+0000","last_became_peered":"2024-04-17T07:34:14.502030+0000","last_unstale":"2024-04-17T07:34:14.506872+0000","last_undegraded":"2024-04-17T07:34:14.506872+0000","last_fullsized":"2024-04-17T07:34:14.506872+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:40.792604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498430+0000","last_change":"2024-04-17T07:34:14.499430+0000","last_active":"2024-04-17T07:34:15.498430+0000","last_peered":"2024-04-17T07:34:15.498430+0000","last_clean":"2024-04-17T07:34:15.498430+0000","last_became_active":"2024-04-17T07:34:14.498886+0000","last_became_peered":"2024-04-17T07:34:14.498886+0000","last_unstale":"2024-04-17T07:34:15.498430+0000","last_undegraded":"2024-04-17T07:34:15.498430+0000","last_fullsized":"2024-04-17T07:34:15.498430+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:24.539675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508720+0000","last_change":"2024-04-17T07:34:14.507325+0000","last_active":"2024-04-17T07:34:14.508720+0000","last_peered":"2024-04-17T07:34:14.508720+0000","last_clean":"2024-04-17T07:34:14.508720+0000","last_became_active":"2024-04-17T07:34:14.504438+0000","last_became_peered":"2024-04-17T07:34:14.504438+0000","last_unstale":"2024-04-17T07:34:14.508720+0000","last_undegraded":"2024-04-17T07:34:14.508720+0000","last_fullsized":"2024-04-17T07:34:14.508720+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:42.441631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.502662+0000","last_change":"2024-04-17T07:34:14.496718+0000","last_active":"2024-04-17T07:34:14.502662+0000","last_peered":"2024-04-17T07:34:14.502662+0000","last_clean":"2024-04-17T07:34:14.502662+0000","last_became_active":"2024-04-17T07:34:14.496004+0000","last_became_peered":"2024-04-17T07:34:14.496004+0000","last_unstale":"2024-04-17T07:34:14.502662+0000","last_undegraded":"2024-04-17T07:34:14.502662+0000","last_fullsized":"2024-04-17T07:34:14.502662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:04.233979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498281+0000","last_change":"2024-04-17T07:34:14.503925+0000","last_active":"2024-04-17T07:34:15.498281+0000","last_peered":"2024-04-17T07:34:15.498281+0000","last_clean":"2024-04-17T07:34:15.498281+0000","last_became_active":"2024-04-17T07:34:14.503647+0000","last_became_peered":"2024-04-17T07:34:14.503647+0000","last_unstale":"2024-04-17T07:34:15.498281+0000","last_undegraded":"2024-04-17T07:34:15.498281+0000","last_fullsized":"2024-04-17T07:34:15.498281+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:33:51.973483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.500672+0000","last_change":"2024-04-17T07:34:14.497393+0000","last_active":"2024-04-17T07:34:14.500672+0000","last_peered":"2024-04-17T07:34:14.500672+0000","last_clean":"2024-04-17T07:34:14.500672+0000","last_became_active":"2024-04-17T07:34:14.497095+0000","last_became_peered":"2024-04-17T07:34:14.497095+0000","last_unstale":"2024-04-17T07:34:14.500672+0000","last_undegraded":"2024-04-17T07:34:14.500672+0000","last_fullsized":"2024-04-17T07:34:14.500672+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:19.042966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498533+0000","last_change":"2024-04-17T07:34:14.492626+0000","last_active":"2024-04-17T07:34:14.498533+0000","last_peered":"2024-04-17T07:34:14.498533+0000","last_clean":"2024-04-17T07:34:14.498533+0000","last_became_active":"2024-04-17T07:34:14.492297+0000","last_became_peered":"2024-04-17T07:34:14.492297+0000","last_unstale":"2024-04-17T07:34:14.498533+0000","last_undegraded":"2024-04-17T07:34:14.498533+0000","last_fullsized":"2024-04-17T07:34:14.498533+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:29.645880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497723+0000","last_change":"2024-04-17T07:34:13.482108+0000","last_active":"2024-04-17T07:34:15.497723+0000","last_peered":"2024-04-17T07:34:15.497723+0000","last_clean":"2024-04-17T07:34:15.497723+0000","last_became_active":"2024-04-17T07:34:13.481638+0000","last_became_peered":"2024-04-17T07:34:13.481638+0000","last_unstale":"2024-04-17T07:34:15.497723+0000","last_undegraded":"2024-04-17T07:34:15.497723+0000","last_fullsized":"2024-04-17T07:34:15.497723+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:06.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163584+0000","last_change":"2024-04-17T07:34:14.502790+0000","last_active":"2024-04-17T07:34:16.163584+0000","last_peered":"2024-04-17T07:34:16.163584+0000","last_clean":"2024-04-17T07:34:16.163584+0000","last_became_active":"2024-04-17T07:34:14.502519+0000","last_became_peered":"2024-04-17T07:34:14.502519+0000","last_unstale":"2024-04-17T07:34:16.163584+0000","last_undegraded":"2024-04-17T07:34:16.163584+0000","last_fullsized":"2024-04-17T07:34:16.163584+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:35.702756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497945+0000","last_change":"2024-04-17T07:34:13.472917+0000","last_active":"2024-04-17T07:34:15.497945+0000","last_peered":"2024-04-17T07:34:15.497945+0000","last_clean":"2024-04-17T07:34:15.497945+0000","last_became_active":"2024-04-17T07:34:13.472542+0000","last_became_peered":"2024-04-17T07:34:13.472542+0000","last_unstale":"2024-04-17T07:34:15.497945+0000","last_undegraded":"2024-04-17T07:34:15.497945+0000","last_fullsized":"2024-04-17T07:34:15.497945+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:51.378724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499971+0000","last_change":"2024-04-17T07:34:14.501840+0000","last_active":"2024-04-17T07:34:15.499971+0000","last_peered":"2024-04-17T07:34:15.499971+0000","last_clean":"2024-04-17T07:34:15.499971+0000","last_became_active":"2024-04-17T07:34:14.501423+0000","last_became_peered":"2024-04-17T07:34:14.501423+0000","last_unstale":"2024-04-17T07:34:15.499971+0000","last_undegraded":"2024-04-17T07:34:15.499971+0000","last_fullsized":"2024-04-17T07:34:15.499971+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:57:36.741933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.499151+0000","last_change":"2024-04-17T07:34:13.476284+0000","last_active":"2024-04-17T07:34:14.499151+0000","last_peered":"2024-04-17T07:34:14.499151+0000","last_clean":"2024-04-17T07:34:14.499151+0000","last_became_active":"2024-04-17T07:34:13.476011+0000","last_became_peered":"2024-04-17T07:34:13.476011+0000","last_unstale":"2024-04-17T07:34:14.499151+0000","last_undegraded":"2024-04-17T07:34:14.499151+0000","last_fullsized":"2024-04-17T07:34:14.499151+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:41.129787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498733+0000","last_change":"2024-04-17T07:34:14.501318+0000","last_active":"2024-04-17T07:34:15.498733+0000","last_peered":"2024-04-17T07:34:15.498733+0000","last_clean":"2024-04-17T07:34:15.498733+0000","last_became_active":"2024-04-17T07:34:14.501013+0000","last_became_peered":"2024-04-17T07:34:14.501013+0000","last_unstale":"2024-04-17T07:34:15.498733+0000","last_undegraded":"2024-04-17T07:34:15.498733+0000","last_fullsized":"2024-04-17T07:34:15.498733+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:14:13.108917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493023+0000","last_change":"2024-04-17T07:34:13.477009+0000","last_active":"2024-04-17T07:34:15.493023+0000","last_peered":"2024-04-17T07:34:15.493023+0000","last_clean":"2024-04-17T07:34:15.493023+0000","last_became_active":"2024-04-17T07:34:13.476708+0000","last_became_peered":"2024-04-17T07:34:13.476708+0000","last_unstale":"2024-04-17T07:34:15.493023+0000","last_undegraded":"2024-04-17T07:34:15.493023+0000","last_fullsized":"2024-04-17T07:34:15.493023+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:27:26.955485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498744+0000","last_change":"2024-04-17T07:34:14.497392+0000","last_active":"2024-04-17T07:34:14.498744+0000","last_peered":"2024-04-17T07:34:14.498744+0000","last_clean":"2024-04-17T07:34:14.498744+0000","last_became_active":"2024-04-17T07:34:14.497045+0000","last_became_peered":"2024-04-17T07:34:14.497045+0000","last_unstale":"2024-04-17T07:34:14.498744+0000","last_undegraded":"2024-04-17T07:34:14.498744+0000","last_fullsized":"2024-04-17T07:34:14.498744+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:40:11.246341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492552+0000","last_change":"2024-04-17T07:34:13.472097+0000","last_active":"2024-04-17T07:34:15.492552+0000","last_peered":"2024-04-17T07:34:15.492552+0000","last_clean":"2024-04-17T07:34:15.492552+0000","last_became_active":"2024-04-17T07:34:13.471282+0000","last_became_peered":"2024-04-17T07:34:13.471282+0000","last_unstale":"2024-04-17T07:34:15.492552+0000","last_undegraded":"2024-04-17T07:34:15.492552+0000","last_fullsized":"2024-04-17T07:34:15.492552+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:49.774976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494633+0000","last_change":"2024-04-17T07:34:14.485764+0000","last_active":"2024-04-17T07:34:14.494633+0000","last_peered":"2024-04-17T07:34:14.494633+0000","last_clean":"2024-04-17T07:34:14.494633+0000","last_became_active":"2024-04-17T07:34:14.485423+0000","last_became_peered":"2024-04-17T07:34:14.485423+0000","last_unstale":"2024-04-17T07:34:14.494633+0000","last_undegraded":"2024-04-17T07:34:14.494633+0000","last_fullsized":"2024-04-17T07:34:14.494633+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:41.384791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.494470+0000","last_change":"2024-04-17T07:34:13.470606+0000","last_active":"2024-04-17T07:34:14.494470+0000","last_peered":"2024-04-17T07:34:14.494470+0000","last_clean":"2024-04-17T07:34:14.494470+0000","last_became_active":"2024-04-17T07:34:13.470231+0000","last_became_peered":"2024-04-17T07:34:13.470231+0000","last_unstale":"2024-04-17T07:34:14.494470+0000","last_undegraded":"2024-04-17T07:34:14.494470+0000","last_fullsized":"2024-04-17T07:34:14.494470+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:27.702775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162600+0000","last_change":"2024-04-17T07:34:14.502091+0000","last_active":"2024-04-17T07:34:16.162600+0000","last_peered":"2024-04-17T07:34:16.162600+0000","last_clean":"2024-04-17T07:34:16.162600+0000","last_became_active":"2024-04-17T07:34:14.500408+0000","last_became_peered":"2024-04-17T07:34:14.500408+0000","last_unstale":"2024-04-17T07:34:16.162600+0000","last_undegraded":"2024-04-17T07:34:16.162600+0000","last_fullsized":"2024-04-17T07:34:16.162600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:27:27.972837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.472876+0000","last_change":"2024-04-17T07:34:13.483670+0000","last_active":"2024-04-17T07:34:14.472876+0000","last_peered":"2024-04-17T07:34:14.472876+0000","last_clean":"2024-04-17T07:34:14.472876+0000","last_became_active":"2024-04-17T07:34:13.483351+0000","last_became_peered":"2024-04-17T07:34:13.483351+0000","last_unstale":"2024-04-17T07:34:14.472876+0000","last_undegraded":"2024-04-17T07:34:14.472876+0000","last_fullsized":"2024-04-17T07:34:14.472876+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:53.738666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493149+0000","last_change":"2024-04-17T07:34:14.498031+0000","last_active":"2024-04-17T07:34:15.493149+0000","last_peered":"2024-04-17T07:34:15.493149+0000","last_clean":"2024-04-17T07:34:15.493149+0000","last_became_active":"2024-04-17T07:34:14.497439+0000","last_became_peered":"2024-04-17T07:34:14.497439+0000","last_unstale":"2024-04-17T07:34:15.493149+0000","last_undegraded":"2024-04-17T07:34:15.493149+0000","last_fullsized":"2024-04-17T07:34:15.493149+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:53:06.025412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492597+0000","last_change":"2024-04-17T07:34:13.479618+0000","last_active":"2024-04-17T07:34:15.492597+0000","last_peered":"2024-04-17T07:34:15.492597+0000","last_clean":"2024-04-17T07:34:15.492597+0000","last_became_active":"2024-04-17T07:34:13.479301+0000","last_became_peered":"2024-04-17T07:34:13.479301+0000","last_unstale":"2024-04-17T07:34:15.492597+0000","last_undegraded":"2024-04-17T07:34:15.492597+0000","last_fullsized":"2024-04-17T07:34:15.492597+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:20:44.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487460+0000","last_change":"2024-04-17T07:34:14.484315+0000","last_active":"2024-04-17T07:34:14.487460+0000","last_peered":"2024-04-17T07:34:14.487460+0000","last_clean":"2024-04-17T07:34:14.487460+0000","last_became_active":"2024-04-17T07:34:14.481914+0000","last_became_peered":"2024-04-17T07:34:14.481914+0000","last_unstale":"2024-04-17T07:34:14.487460+0000","last_undegraded":"2024-04-17T07:34:14.487460+0000","last_fullsized":"2024-04-17T07:34:14.487460+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:36.731262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.500193+0000","last_change":"2024-04-17T07:34:13.474704+0000","last_active":"2024-04-17T07:34:15.500193+0000","last_peered":"2024-04-17T07:34:15.500193+0000","last_clean":"2024-04-17T07:34:15.500193+0000","last_became_active":"2024-04-17T07:34:13.474436+0000","last_became_peered":"2024-04-17T07:34:13.474436+0000","last_unstale":"2024-04-17T07:34:15.500193+0000","last_undegraded":"2024-04-17T07:34:15.500193+0000","last_fullsized":"2024-04-17T07:34:15.500193+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:06.166341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499518+0000","last_change":"2024-04-17T07:34:14.497957+0000","last_active":"2024-04-17T07:34:15.499518+0000","last_peered":"2024-04-17T07:34:15.499518+0000","last_clean":"2024-04-17T07:34:15.499518+0000","last_became_active":"2024-04-17T07:34:14.497649+0000","last_became_peered":"2024-04-17T07:34:14.497649+0000","last_unstale":"2024-04-17T07:34:15.499518+0000","last_undegraded":"2024-04-17T07:34:15.499518+0000","last_fullsized":"2024-04-17T07:34:15.499518+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:35:34.025909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.488502+0000","last_change":"2024-04-17T07:34:13.483491+0000","last_active":"2024-04-17T07:34:14.488502+0000","last_peered":"2024-04-17T07:34:14.488502+0000","last_clean":"2024-04-17T07:34:14.488502+0000","last_became_active":"2024-04-17T07:34:13.483088+0000","last_became_peered":"2024-04-17T07:34:13.483088+0000","last_unstale":"2024-04-17T07:34:14.488502+0000","last_undegraded":"2024-04-17T07:34:14.488502+0000","last_fullsized":"2024-04-17T07:34:14.488502+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:52:58.615117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499857+0000","last_change":"2024-04-17T07:34:14.506961+0000","last_active":"2024-04-17T07:34:15.499857+0000","last_peered":"2024-04-17T07:34:15.499857+0000","last_clean":"2024-04-17T07:34:15.499857+0000","last_became_active":"2024-04-17T07:34:14.501792+0000","last_became_peered":"2024-04-17T07:34:14.501792+0000","last_unstale":"2024-04-17T07:34:15.499857+0000","last_undegraded":"2024-04-17T07:34:15.499857+0000","last_fullsized":"2024-04-17T07:34:15.499857+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:51.117385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498284+0000","last_change":"2024-04-17T07:34:13.473275+0000","last_active":"2024-04-17T07:34:15.498284+0000","last_peered":"2024-04-17T07:34:15.498284+0000","last_clean":"2024-04-17T07:34:15.498284+0000","last_became_active":"2024-04-17T07:34:13.471183+0000","last_became_peered":"2024-04-17T07:34:13.471183+0000","last_unstale":"2024-04-17T07:34:15.498284+0000","last_undegraded":"2024-04-17T07:34:15.498284+0000","last_fullsized":"2024-04-17T07:34:15.498284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:45.694228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498866+0000","last_change":"2024-04-17T07:34:14.505454+0000","last_active":"2024-04-17T07:34:15.498866+0000","last_peered":"2024-04-17T07:34:15.498866+0000","last_clean":"2024-04-17T07:34:15.498866+0000","last_became_active":"2024-04-17T07:34:14.505115+0000","last_became_peered":"2024-04-17T07:34:14.505115+0000","last_unstale":"2024-04-17T07:34:15.498866+0000","last_undegraded":"2024-04-17T07:34:15.498866+0000","last_fullsized":"2024-04-17T07:34:15.498866+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:21.459297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162829+0000","last_change":"2024-04-17T07:34:13.483643+0000","last_active":"2024-04-17T07:34:16.162829+0000","last_peered":"2024-04-17T07:34:16.162829+0000","last_clean":"2024-04-17T07:34:16.162829+0000","last_became_active":"2024-04-17T07:34:13.483376+0000","last_became_peered":"2024-04-17T07:34:13.483376+0000","last_unstale":"2024-04-17T07:34:16.162829+0000","last_undegraded":"2024-04-17T07:34:16.162829+0000","last_fullsized":"2024-04-17T07:34:16.162829+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:00:19.834138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.487750+0000","last_change":"2024-04-17T07:34:14.483871+0000","last_active":"2024-04-17T07:34:14.487750+0000","last_peered":"2024-04-17T07:34:14.487750+0000","last_clean":"2024-04-17T07:34:14.487750+0000","last_became_active":"2024-04-17T07:34:14.483577+0000","last_became_peered":"2024-04-17T07:34:14.483577+0000","last_unstale":"2024-04-17T07:34:14.487750+0000","last_undegraded":"2024-04-17T07:34:14.487750+0000","last_fullsized":"2024-04-17T07:34:14.487750+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:03:41.153156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.474215+0000","last_change":"2024-04-17T07:34:13.479476+0000","last_active":"2024-04-17T07:34:14.474215+0000","last_peered":"2024-04-17T07:34:14.474215+0000","last_clean":"2024-04-17T07:34:14.474215+0000","last_became_active":"2024-04-17T07:34:13.479183+0000","last_became_peered":"2024-04-17T07:34:13.479183+0000","last_unstale":"2024-04-17T07:34:14.474215+0000","last_undegraded":"2024-04-17T07:34:14.474215+0000","last_fullsized":"2024-04-17T07:34:14.474215+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:13.519290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492461+0000","last_change":"2024-04-17T07:34:14.505725+0000","last_active":"2024-04-17T07:34:15.492461+0000","last_peered":"2024-04-17T07:34:15.492461+0000","last_clean":"2024-04-17T07:34:15.492461+0000","last_became_active":"2024-04-17T07:34:14.504587+0000","last_became_peered":"2024-04-17T07:34:14.504587+0000","last_unstale":"2024-04-17T07:34:15.492461+0000","last_undegraded":"2024-04-17T07:34:15.492461+0000","last_fullsized":"2024-04-17T07:34:15.492461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:00.551170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492718+0000","last_change":"2024-04-17T07:34:13.482454+0000","last_active":"2024-04-17T07:34:15.492718+0000","last_peered":"2024-04-17T07:34:15.492718+0000","last_clean":"2024-04-17T07:34:15.492718+0000","last_became_active":"2024-04-17T07:34:13.482132+0000","last_became_peered":"2024-04-17T07:34:13.482132+0000","last_unstale":"2024-04-17T07:34:15.492718+0000","last_undegraded":"2024-04-17T07:34:15.492718+0000","last_fullsized":"2024-04-17T07:34:15.492718+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:32:01.673108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162997+0000","last_change":"2024-04-17T07:34:14.510828+0000","last_active":"2024-04-17T07:34:16.162997+0000","last_peered":"2024-04-17T07:34:16.162997+0000","last_clean":"2024-04-17T07:34:16.162997+0000","last_became_active":"2024-04-17T07:34:14.508610+0000","last_became_peered":"2024-04-17T07:34:14.508610+0000","last_unstale":"2024-04-17T07:34:16.162997+0000","last_undegraded":"2024-04-17T07:34:16.162997+0000","last_fullsized":"2024-04-17T07:34:16.162997+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:55.174384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.163251+0000","last_change":"2024-04-17T07:34:13.478067+0000","last_active":"2024-04-17T07:34:16.163251+0000","last_peered":"2024-04-17T07:34:16.163251+0000","last_clean":"2024-04-17T07:34:16.163251+0000","last_became_active":"2024-04-17T07:34:13.477615+0000","last_became_peered":"2024-04-17T07:34:13.477615+0000","last_unstale":"2024-04-17T07:34:16.163251+0000","last_undegraded":"2024-04-17T07:34:16.163251+0000","last_fullsized":"2024-04-17T07:34:16.163251+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:23.441816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:16.162529+0000","last_change":"2024-04-17T07:34:14.508294+0000","last_active":"2024-04-17T07:34:16.162529+0000","last_peered":"2024-04-17T07:34:16.162529+0000","last_clean":"2024-04-17T07:34:16.162529+0000","last_became_active":"2024-04-17T07:34:14.507290+0000","last_became_peered":"2024-04-17T07:34:14.507290+0000","last_unstale":"2024-04-17T07:34:16.162529+0000","last_undegraded":"2024-04-17T07:34:16.162529+0000","last_fullsized":"2024-04-17T07:34:16.162529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:02.201102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.500424+0000","last_change":"2024-04-17T07:34:13.477688+0000","last_active":"2024-04-17T07:34:15.500424+0000","last_peered":"2024-04-17T07:34:15.500424+0000","last_clean":"2024-04-17T07:34:15.500424+0000","last_became_active":"2024-04-17T07:34:13.477392+0000","last_became_peered":"2024-04-17T07:34:13.477392+0000","last_unstale":"2024-04-17T07:34:15.500424+0000","last_undegraded":"2024-04-17T07:34:15.500424+0000","last_fullsized":"2024-04-17T07:34:15.500424+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:36.565251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499923+0000","last_change":"2024-04-17T07:34:14.480348+0000","last_active":"2024-04-17T07:34:15.499923+0000","last_peered":"2024-04-17T07:34:15.499923+0000","last_clean":"2024-04-17T07:34:15.499923+0000","last_became_active":"2024-04-17T07:34:14.477958+0000","last_became_peered":"2024-04-17T07:34:14.477958+0000","last_unstale":"2024-04-17T07:34:15.499923+0000","last_undegraded":"2024-04-17T07:34:15.499923+0000","last_fullsized":"2024-04-17T07:34:15.499923+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:12.161315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.507534+0000","last_change":"2024-04-17T07:34:13.481026+0000","last_active":"2024-04-17T07:34:14.507534+0000","last_peered":"2024-04-17T07:34:14.507534+0000","last_clean":"2024-04-17T07:34:14.507534+0000","last_became_active":"2024-04-17T07:34:13.480718+0000","last_became_peered":"2024-04-17T07:34:13.480718+0000","last_unstale":"2024-04-17T07:34:14.507534+0000","last_undegraded":"2024-04-17T07:34:14.507534+0000","last_fullsized":"2024-04-17T07:34:14.507534+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:08.693617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498797+0000","last_change":"2024-04-17T07:34:14.496241+0000","last_active":"2024-04-17T07:34:14.498797+0000","last_peered":"2024-04-17T07:34:14.498797+0000","last_clean":"2024-04-17T07:34:14.498797+0000","last_became_active":"2024-04-17T07:34:14.491459+0000","last_became_peered":"2024-04-17T07:34:14.491459+0000","last_unstale":"2024-04-17T07:34:14.498797+0000","last_undegraded":"2024-04-17T07:34:14.498797+0000","last_fullsized":"2024-04-17T07:34:14.498797+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:07.074731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"155'1","reported_seq":13,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.505912+0000","last_change":"2024-04-17T07:34:13.484327+0000","last_active":"2024-04-17T07:34:14.505912+0000","last_peered":"2024-04-17T07:34:14.505912+0000","last_clean":"2024-04-17T07:34:14.505912+0000","last_became_active":"2024-04-17T07:34:13.483671+0000","last_became_peered":"2024-04-17T07:34:13.483671+0000","last_unstale":"2024-04-17T07:34:14.505912+0000","last_undegraded":"2024-04-17T07:34:14.505912+0000","last_fullsized":"2024-04-17T07:34:14.505912+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:48.233787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493476+0000","last_change":"2024-04-17T07:34:14.498280+0000","last_active":"2024-04-17T07:34:15.493476+0000","last_peered":"2024-04-17T07:34:15.493476+0000","last_clean":"2024-04-17T07:34:15.493476+0000","last_became_active":"2024-04-17T07:34:14.497962+0000","last_became_peered":"2024-04-17T07:34:14.497962+0000","last_unstale":"2024-04-17T07:34:15.493476+0000","last_undegraded":"2024-04-17T07:34:15.493476+0000","last_fullsized":"2024-04-17T07:34:15.493476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:26:21.369290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498679+0000","last_change":"2024-04-17T07:34:13.472996+0000","last_active":"2024-04-17T07:34:15.498679+0000","last_peered":"2024-04-17T07:34:15.498679+0000","last_clean":"2024-04-17T07:34:15.498679+0000","last_became_active":"2024-04-17T07:34:13.472698+0000","last_became_peered":"2024-04-17T07:34:13.472698+0000","last_unstale":"2024-04-17T07:34:15.498679+0000","last_undegraded":"2024-04-17T07:34:15.498679+0000","last_fullsized":"2024-04-17T07:34:15.498679+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:22.713668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.492878+0000","last_change":"2024-04-17T07:34:14.491035+0000","last_active":"2024-04-17T07:34:14.492878+0000","last_peered":"2024-04-17T07:34:14.492878+0000","last_clean":"2024-04-17T07:34:14.492878+0000","last_became_active":"2024-04-17T07:34:14.490682+0000","last_became_peered":"2024-04-17T07:34:14.490682+0000","last_unstale":"2024-04-17T07:34:14.492878+0000","last_undegraded":"2024-04-17T07:34:14.492878+0000","last_fullsized":"2024-04-17T07:34:14.492878+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:13.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498097+0000","last_change":"2024-04-17T07:34:13.476812+0000","last_active":"2024-04-17T07:34:15.498097+0000","last_peered":"2024-04-17T07:34:15.498097+0000","last_clean":"2024-04-17T07:34:15.498097+0000","last_became_active":"2024-04-17T07:34:13.476530+0000","last_became_peered":"2024-04-17T07:34:13.476530+0000","last_unstale":"2024-04-17T07:34:15.498097+0000","last_undegraded":"2024-04-17T07:34:15.498097+0000","last_fullsized":"2024-04-17T07:34:15.498097+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:18.129263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508438+0000","last_change":"2024-04-17T07:34:14.505410+0000","last_active":"2024-04-17T07:34:14.508438+0000","last_peered":"2024-04-17T07:34:14.508438+0000","last_clean":"2024-04-17T07:34:14.508438+0000","last_became_active":"2024-04-17T07:34:14.502610+0000","last_became_peered":"2024-04-17T07:34:14.502610+0000","last_unstale":"2024-04-17T07:34:14.508438+0000","last_undegraded":"2024-04-17T07:34:14.508438+0000","last_fullsized":"2024-04-17T07:34:14.508438+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:46.711580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.486433+0000","last_change":"2024-04-17T07:34:13.476281+0000","last_active":"2024-04-17T07:34:14.486433+0000","last_peered":"2024-04-17T07:34:14.486433+0000","last_clean":"2024-04-17T07:34:14.486433+0000","last_became_active":"2024-04-17T07:34:13.475920+0000","last_became_peered":"2024-04-17T07:34:13.475920+0000","last_unstale":"2024-04-17T07:34:14.486433+0000","last_undegraded":"2024-04-17T07:34:14.486433+0000","last_fullsized":"2024-04-17T07:34:14.486433+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:11.067689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492938+0000","last_change":"2024-04-17T07:34:14.484831+0000","last_active":"2024-04-17T07:34:15.492938+0000","last_peered":"2024-04-17T07:34:15.492938+0000","last_clean":"2024-04-17T07:34:15.492938+0000","last_became_active":"2024-04-17T07:34:14.482465+0000","last_became_peered":"2024-04-17T07:34:14.482465+0000","last_unstale":"2024-04-17T07:34:15.492938+0000","last_undegraded":"2024-04-17T07:34:15.492938+0000","last_fullsized":"2024-04-17T07:34:15.492938+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:47:10.347613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.485804+0000","last_change":"2024-04-17T07:34:13.465034+0000","last_active":"2024-04-17T07:34:14.485804+0000","last_peered":"2024-04-17T07:34:14.485804+0000","last_clean":"2024-04-17T07:34:14.485804+0000","last_became_active":"2024-04-17T07:34:13.464594+0000","last_became_peered":"2024-04-17T07:34:13.464594+0000","last_unstale":"2024-04-17T07:34:14.485804+0000","last_undegraded":"2024-04-17T07:34:14.485804+0000","last_fullsized":"2024-04-17T07:34:14.485804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:57:26.285361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497789+0000","last_change":"2024-04-17T07:34:14.502214+0000","last_active":"2024-04-17T07:34:15.497789+0000","last_peered":"2024-04-17T07:34:15.497789+0000","last_clean":"2024-04-17T07:34:15.497789+0000","last_became_active":"2024-04-17T07:34:14.501182+0000","last_became_peered":"2024-04-17T07:34:14.501182+0000","last_unstale":"2024-04-17T07:34:15.497789+0000","last_undegraded":"2024-04-17T07:34:15.497789+0000","last_fullsized":"2024-04-17T07:34:15.497789+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:04.044214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.467181+0000","last_change":"2024-04-17T07:34:13.481675+0000","last_active":"2024-04-17T07:34:14.467181+0000","last_peered":"2024-04-17T07:34:14.467181+0000","last_clean":"2024-04-17T07:34:14.467181+0000","last_became_active":"2024-04-17T07:34:13.480731+0000","last_became_peered":"2024-04-17T07:34:13.480731+0000","last_unstale":"2024-04-17T07:34:14.467181+0000","last_undegraded":"2024-04-17T07:34:14.467181+0000","last_fullsized":"2024-04-17T07:34:14.467181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:25.190166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.500644+0000","last_change":"2024-04-17T07:34:14.496746+0000","last_active":"2024-04-17T07:34:14.500644+0000","last_peered":"2024-04-17T07:34:14.500644+0000","last_clean":"2024-04-17T07:34:14.500644+0000","last_became_active":"2024-04-17T07:34:14.493956+0000","last_became_peered":"2024-04-17T07:34:14.493956+0000","last_unstale":"2024-04-17T07:34:14.500644+0000","last_undegraded":"2024-04-17T07:34:14.500644+0000","last_fullsized":"2024-04-17T07:34:14.500644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:38.779800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"155'2","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.503340+0000","last_change":"2024-04-17T07:34:13.477341+0000","last_active":"2024-04-17T07:34:14.503340+0000","last_peered":"2024-04-17T07:34:14.503340+0000","last_clean":"2024-04-17T07:34:14.503340+0000","last_became_active":"2024-04-17T07:34:13.476859+0000","last_became_peered":"2024-04-17T07:34:13.476859+0000","last_unstale":"2024-04-17T07:34:14.503340+0000","last_undegraded":"2024-04-17T07:34:14.503340+0000","last_fullsized":"2024-04-17T07:34:14.503340+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:55:24.471927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.507355+0000","last_change":"2024-04-17T07:34:14.504637+0000","last_active":"2024-04-17T07:34:14.507355+0000","last_peered":"2024-04-17T07:34:14.507355+0000","last_clean":"2024-04-17T07:34:14.507355+0000","last_became_active":"2024-04-17T07:34:14.504366+0000","last_became_peered":"2024-04-17T07:34:14.504366+0000","last_unstale":"2024-04-17T07:34:14.507355+0000","last_undegraded":"2024-04-17T07:34:14.507355+0000","last_fullsized":"2024-04-17T07:34:14.507355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:36.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498316+0000","last_change":"2024-04-17T07:34:13.480375+0000","last_active":"2024-04-17T07:34:15.498316+0000","last_peered":"2024-04-17T07:34:15.498316+0000","last_clean":"2024-04-17T07:34:15.498316+0000","last_became_active":"2024-04-17T07:34:13.479603+0000","last_became_peered":"2024-04-17T07:34:13.479603+0000","last_unstale":"2024-04-17T07:34:15.498316+0000","last_undegraded":"2024-04-17T07:34:15.498316+0000","last_fullsized":"2024-04-17T07:34:15.498316+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:38:30.528110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493045+0000","last_change":"2024-04-17T07:34:14.505811+0000","last_active":"2024-04-17T07:34:15.493045+0000","last_peered":"2024-04-17T07:34:15.493045+0000","last_clean":"2024-04-17T07:34:15.493045+0000","last_became_active":"2024-04-17T07:34:14.504223+0000","last_became_peered":"2024-04-17T07:34:14.504223+0000","last_unstale":"2024-04-17T07:34:15.493045+0000","last_undegraded":"2024-04-17T07:34:15.493045+0000","last_fullsized":"2024-04-17T07:34:15.493045+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:30.622147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.493423+0000","last_change":"2024-04-17T07:34:13.479526+0000","last_active":"2024-04-17T07:34:15.493423+0000","last_peered":"2024-04-17T07:34:15.493423+0000","last_clean":"2024-04-17T07:34:15.493423+0000","last_became_active":"2024-04-17T07:34:13.479202+0000","last_became_peered":"2024-04-17T07:34:13.479202+0000","last_unstale":"2024-04-17T07:34:15.493423+0000","last_undegraded":"2024-04-17T07:34:15.493423+0000","last_fullsized":"2024-04-17T07:34:15.493423+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:10.743862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.498653+0000","last_change":"2024-04-17T07:34:14.487588+0000","last_active":"2024-04-17T07:34:14.498653+0000","last_peered":"2024-04-17T07:34:14.498653+0000","last_clean":"2024-04-17T07:34:14.498653+0000","last_became_active":"2024-04-17T07:34:14.486041+0000","last_became_peered":"2024-04-17T07:34:14.486041+0000","last_unstale":"2024-04-17T07:34:14.498653+0000","last_undegraded":"2024-04-17T07:34:14.498653+0000","last_fullsized":"2024-04-17T07:34:14.498653+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:26.558713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.466967+0000","last_change":"2024-04-17T07:34:13.474614+0000","last_active":"2024-04-17T07:34:14.466967+0000","last_peered":"2024-04-17T07:34:14.466967+0000","last_clean":"2024-04-17T07:34:14.466967+0000","last_became_active":"2024-04-17T07:34:13.474090+0000","last_became_peered":"2024-04-17T07:34:13.474090+0000","last_unstale":"2024-04-17T07:34:14.466967+0000","last_undegraded":"2024-04-17T07:34:14.466967+0000","last_fullsized":"2024-04-17T07:34:14.466967+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:33.495237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499088+0000","last_change":"2024-04-17T07:34:14.505437+0000","last_active":"2024-04-17T07:34:15.499088+0000","last_peered":"2024-04-17T07:34:15.499088+0000","last_clean":"2024-04-17T07:34:15.499088+0000","last_became_active":"2024-04-17T07:34:14.504257+0000","last_became_peered":"2024-04-17T07:34:14.504257+0000","last_unstale":"2024-04-17T07:34:15.499088+0000","last_undegraded":"2024-04-17T07:34:15.499088+0000","last_fullsized":"2024-04-17T07:34:15.499088+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T07:46:27.512930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492701+0000","last_change":"2024-04-17T07:34:13.483214+0000","last_active":"2024-04-17T07:34:15.492701+0000","last_peered":"2024-04-17T07:34:15.492701+0000","last_clean":"2024-04-17T07:34:15.492701+0000","last_became_active":"2024-04-17T07:34:13.482919+0000","last_became_peered":"2024-04-17T07:34:13.482919+0000","last_unstale":"2024-04-17T07:34:15.492701+0000","last_undegraded":"2024-04-17T07:34:15.492701+0000","last_fullsized":"2024-04-17T07:34:15.492701+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:01.183750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.497827+0000","last_change":"2024-04-17T07:34:14.483307+0000","last_active":"2024-04-17T07:34:15.497827+0000","last_peered":"2024-04-17T07:34:15.497827+0000","last_clean":"2024-04-17T07:34:15.497827+0000","last_became_active":"2024-04-17T07:34:14.482392+0000","last_became_peered":"2024-04-17T07:34:14.482392+0000","last_unstale":"2024-04-17T07:34:15.497827+0000","last_undegraded":"2024-04-17T07:34:15.497827+0000","last_fullsized":"2024-04-17T07:34:15.497827+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:48.360284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.499457+0000","last_change":"2024-04-17T07:34:13.477016+0000","last_active":"2024-04-17T07:34:15.499457+0000","last_peered":"2024-04-17T07:34:15.499457+0000","last_clean":"2024-04-17T07:34:15.499457+0000","last_became_active":"2024-04-17T07:34:13.476702+0000","last_became_peered":"2024-04-17T07:34:13.476702+0000","last_unstale":"2024-04-17T07:34:15.499457+0000","last_undegraded":"2024-04-17T07:34:15.499457+0000","last_fullsized":"2024-04-17T07:34:15.499457+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:06.057379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.508354+0000","last_change":"2024-04-17T07:34:14.506315+0000","last_active":"2024-04-17T07:34:14.508354+0000","last_peered":"2024-04-17T07:34:14.508354+0000","last_clean":"2024-04-17T07:34:14.508354+0000","last_became_active":"2024-04-17T07:34:14.505661+0000","last_became_peered":"2024-04-17T07:34:14.505661+0000","last_unstale":"2024-04-17T07:34:14.508354+0000","last_undegraded":"2024-04-17T07:34:14.508354+0000","last_fullsized":"2024-04-17T07:34:14.508354+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.921844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.485970+0000","last_change":"2024-04-17T07:34:13.476036+0000","last_active":"2024-04-17T07:34:14.485970+0000","last_peered":"2024-04-17T07:34:14.485970+0000","last_clean":"2024-04-17T07:34:14.485970+0000","last_became_active":"2024-04-17T07:34:13.475741+0000","last_became_peered":"2024-04-17T07:34:13.475741+0000","last_unstale":"2024-04-17T07:34:14.485970+0000","last_undegraded":"2024-04-17T07:34:14.485970+0000","last_fullsized":"2024-04-17T07:34:14.485970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:50.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.492550+0000","last_change":"2024-04-17T07:34:14.495839+0000","last_active":"2024-04-17T07:34:15.492550+0000","last_peered":"2024-04-17T07:34:15.492550+0000","last_clean":"2024-04-17T07:34:15.492550+0000","last_became_active":"2024-04-17T07:34:14.495501+0000","last_became_peered":"2024-04-17T07:34:14.495501+0000","last_unstale":"2024-04-17T07:34:15.492550+0000","last_undegraded":"2024-04-17T07:34:15.492550+0000","last_fullsized":"2024-04-17T07:34:15.492550+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:03:26.044682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-04-17T07:34:14.467243+0000","last_change":"2024-04-17T07:34:13.481482+0000","last_active":"2024-04-17T07:34:14.467243+0000","last_peered":"2024-04-17T07:34:14.467243+0000","last_clean":"2024-04-17T07:34:14.467243+0000","last_became_active":"2024-04-17T07:34:13.480395+0000","last_became_peered":"2024-04-17T07:34:13.480395+0000","last_unstale":"2024-04-17T07:34:14.467243+0000","last_undegraded":"2024-04-17T07:34:14.467243+0000","last_fullsized":"2024-04-17T07:34:14.467243+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:54.475442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498726+0000","last_change":"2024-04-17T07:34:14.502324+0000","last_active":"2024-04-17T07:34:15.498726+0000","last_peered":"2024-04-17T07:34:15.498726+0000","last_clean":"2024-04-17T07:34:15.498726+0000","last_became_active":"2024-04-17T07:34:14.501644+0000","last_became_peered":"2024-04-17T07:34:14.501644+0000","last_unstale":"2024-04-17T07:34:15.498726+0000","last_undegraded":"2024-04-17T07:34:15.498726+0000","last_fullsized":"2024-04-17T07:34:15.498726+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:13.437490+0000","last_clean_scrub_stamp":"2024-04-17T07:34:13.437490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:13.647727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.498032+0000","last_change":"2024-04-17T07:34:13.479096+0000","last_active":"2024-04-17T07:34:15.498032+0000","last_peered":"2024-04-17T07:34:15.498032+0000","last_clean":"2024-04-17T07:34:15.498032+0000","last_became_active":"2024-04-17T07:34:13.478783+0000","last_became_peered":"2024-04-17T07:34:13.478783+0000","last_unstale":"2024-04-17T07:34:15.498032+0000","last_undegraded":"2024-04-17T07:34:15.498032+0000","last_fullsized":"2024-04-17T07:34:15.498032+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:34:12.420139+0000","last_clean_scrub_stamp":"2024-04-17T07:34:12.420139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:49.336892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"144'255","reported_seq":920,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-04-17T07:34:15.496656+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T07:34:15.496656+0000","last_peered":"2024-04-17T07:34:15.496656+0000","last_clean":"2024-04-17T07:34:15.496656+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T07:34:15.496656+0000","last_undegraded":"2024-04-17T07:34:15.496656+0000","last_fullsized":"2024-04-17T07:34:15.496656+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":407,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1314,"num_read_kb":2736,"num_write":407,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":255,"ondisk_log_size":255,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542384,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37840,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334000,"statfs":{"total":96636764160,"available":96598016000,"internally_reserved":0,"allocated":2555904,"data_stored":1525673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542384,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":928339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542383,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1933312,"data_stored":926839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542383,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59440,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312400,"statfs":{"total":96636764160,"available":96575897600,"internally_reserved":0,"allocated":1929216,"data_stored":926797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542385,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1933312,"data_stored":926887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542384,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59436,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312404,"statfs":{"total":96636764160,"available":96575901696,"internally_reserved":0,"allocated":1933312,"data_stored":927343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542385,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55284,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316556,"statfs":{"total":96636764160,"available":96580153344,"internally_reserved":0,"allocated":1937408,"data_stored":933055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542383,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37800,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334040,"statfs":{"total":96636764160,"available":96598056960,"internally_reserved":0,"allocated":2519040,"data_stored":1517095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T07:34:19.576 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T07:34:19.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T07:34:19.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.849+0000 7fc949c00640 1 Processor -- start 2024-04-17T07:34:19.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.849+0000 7fc949c00640 1 -- start start 2024-04-17T07:34:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc949c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc944169710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc949c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9440d4730 con 0x7fc94415ec80 2024-04-17T07:34:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc949c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9440d3690 con 0x7fc9440b6b90 2024-04-17T07:34:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc949c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9440ba670 con 0x7fc9440d3190 2024-04-17T07:34:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc942a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc94415ec80 0x7fc9440b6480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55578/0 (socket says 172.21.15.67:55578) 2024-04-17T07:34:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.850+0000 7fc942a00640 1 -- 172.21.15.67:0/364670347 learned_addr learned my addr 172.21.15.67:0/364670347 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc943400640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc944169710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325649561 0 0) 0x7fc9440d4730 con 0x7fc94415ec80 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc928003520 con 0x7fc94415ec80 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3498721408 0 0) 0x7fc928003520 con 0x7fc94415ec80 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 msgr2=0x7fc944169710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc944169710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 >> v1:172.21.15.187:6789/0 conn(0x7fc9440b6b90 legacy=0x7fc9440d2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9440d3970 con 0x7fc94415ec80 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc9340019f0 con 0x7fc94415ec80 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 >> v1:172.21.15.67:6789/0 conn(0x7fc94415ec80 legacy=0x7fc9440b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.851+0000 7fc941600640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc9280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc941600640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928003fb0 0x7fc928004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc941600640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc928004d40 0x7fc9280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440ba670 con 0x7fc928004d40 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc92800bb90 con 0x7fc928003fb0 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc92800bd90 con 0x7fc9440d3190 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc928004d40 0x7fc9280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc942a00640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc9280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 -- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 msgr2=0x7fc9280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9440d3190 0x7fc9280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc942000640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928003fb0 0x7fc928004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 -- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928003fb0 msgr2=0x7fc928004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928003fb0 0x7fc928004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc943400640 1 -- 172.21.15.67:0/364670347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc928003520 con 0x7fc928004d40 2024-04-17T07:34:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.852+0000 7fc942000640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928003fb0 0x7fc928004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:19.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.853+0000 7fc943400640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc928004d40 0x7fc9280051c0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009ac0 tx=0x7fc928005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6354bea7144bff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c039070 con 0x7fc928004d40 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93c02c6b0 con 0x7fc928004d40 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc941600640 1 -- 172.21.15.67:0/364670347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c009690 con 0x7fc928004d40 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc949c00640 1 -- 172.21.15.67:0/364670347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc928004d40 msgr2=0x7fc9280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc949c00640 1 --2- 172.21.15.67:0/364670347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc928004d40 0x7fc9280051c0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009ac0 tx=0x7fc928005830 comp rx=0 tx=0).stop 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc943400640 1 -- 172.21.15.67:0/364670347 reap_dead start 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc949c00640 1 -- 172.21.15.67:0/364670347 shutdown_connections 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc949c00640 1 -- 172.21.15.67:0/364670347 >> 172.21.15.67:0/364670347 conn(0x7fc944067c30 msgr2=0x7fc9440d1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.854+0000 7fc949c00640 1 -- 172.21.15.67:0/364670347 shutdown_connections 2024-04-17T07:34:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.855+0000 7fc949c00640 1 -- 172.21.15.67:0/364670347 wait complete. 2024-04-17T07:34:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.856+0000 7fc949c00640 1 Processor -- start 2024-04-17T07:34:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.856+0000 7fc949c00640 1 -- start start 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 0x7fc944152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc94414f100 0x7fc9440d2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440d4210 con 0x7fc9440cce40 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9440d3f30 con 0x7fc928004d40 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.857+0000 7fc949c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9440b8b40 con 0x7fc94414f100 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60790/0 (socket says 172.21.15.67:60790) 2024-04-17T07:34:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 -- 172.21.15.67:0/562389378 learned_addr learned my addr 172.21.15.67:0/562389378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942a00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 0x7fc944152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc943400640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc94414f100 0x7fc9440d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 -- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc94414f100 msgr2=0x7fc9440d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc94414f100 0x7fc9440d2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 -- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 msgr2=0x7fc944152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 0x7fc944152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942000640 1 -- 172.21.15.67:0/562389378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9441f7bc0 con 0x7fc9440cce40 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.858+0000 7fc942a00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 0x7fc944152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc942000640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d7a0 tx=0x7fc92c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c005070 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92c005210 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c007030 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9441f7e60 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.859+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9441f8700 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.860+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9441f8030 con 0x7fc9440cce40 2024-04-17T07:34:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.861+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fc92c01a020 con 0x7fc9440cce40 2024-04-17T07:34:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.862+0000 7fc93aa00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 0x7fc92006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.862+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fc92c08b660 con 0x7fc9440cce40 2024-04-17T07:34:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.863+0000 7fc942a00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 0x7fc92006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.864+0000 7fc942a00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 0x7fc92006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc934001f70 tx=0x7fc934001310 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:19.867+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc92c059850 con 0x7fc9440cce40 2024-04-17T07:34:20.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.152+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc944068ca0 con 0x7fc9440cce40 2024-04-17T07:34:20.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.158+0000 7fc93aa00640 1 -- 172.21.15.67:0/562389378 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) ==== 190+0+147 (secure 0 0 0) 0x7fc92c059150 con 0x7fc9440cce40 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T07:34:20.159 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 msgr2=0x7fc92006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 0x7fc92006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc934001f70 tx=0x7fc934001310 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 msgr2=0x7fc9440d1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d7a0 tx=0x7fc92c00dc60 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 shutdown_connections 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fc92006ca10 0x7fc92006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc94414f100 0x7fc9440d2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9440cce40 0x7fc9440d1ee0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 --2- 172.21.15.67:0/562389378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc928004d40 0x7fc944152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.161+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 >> 172.21.15.67:0/562389378 conn(0x7fc944067c30 msgr2=0x7fc9440b9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:20.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.162+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 shutdown_connections 2024-04-17T07:34:20.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.162+0000 7fc949c00640 1 -- 172.21.15.67:0/562389378 wait complete. 2024-04-17T07:34:20.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T07:34:20.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.413+0000 7f2b98e00640 1 Processor -- start 2024-04-17T07:34:20.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.413+0000 7f2b98e00640 1 -- start start 2024-04-17T07:34:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b98e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b9415ac80 0x7f2b9406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b98e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b94072eb0 0x7f2b94071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b98e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b940cd920 con 0x7f2b94072330 2024-04-17T07:34:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b98e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2b94072790 con 0x7f2b94072eb0 2024-04-17T07:34:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b98e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b940cdc30 con 0x7f2b9415ac80 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b92a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b94072eb0 0x7f2b94071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b93e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b94072330 0x7f2b9406ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55588/0 (socket says 172.21.15.67:55588) 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.415+0000 7f2b93e00640 1 -- 172.21.15.67:0/1372397407 learned_addr learned my addr 172.21.15.67:0/1372397407 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b93400640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b9415ac80 0x7f2b9406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b92a00640 1 -- 172.21.15.67:0/1372397407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b9415ac80 msgr2=0x7f2b9406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b92a00640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b9415ac80 0x7f2b9406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b92a00640 1 -- 172.21.15.67:0/1372397407 >> v1:172.21.15.67:6789/0 conn(0x7f2b94072330 legacy=0x7f2b9406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b92a00640 1 -- 172.21.15.67:0/1372397407 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9406f980 con 0x7f2b94072eb0 2024-04-17T07:34:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.416+0000 7f2b92000640 1 -- 172.21.15.67:0/1372397407 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639796152 0 0) 0x7f2b8c001490 con 0x7f2b94072330 2024-04-17T07:34:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.417+0000 7f2b92a00640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b94072eb0 0x7f2b94071df0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad01908dcf88d69 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.417+0000 7f2b92000640 1 -- 172.21.15.67:0/1372397407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0315a0 con 0x7f2b94072eb0 2024-04-17T07:34:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.417+0000 7f2b92000640 1 -- 172.21.15.67:0/1372397407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b7c037070 con 0x7f2b94072eb0 2024-04-17T07:34:20.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.417+0000 7f2b92000640 1 -- 172.21.15.67:0/1372397407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c031960 con 0x7f2b94072eb0 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 -- 172.21.15.67:0/1372397407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b94072eb0 msgr2=0x7f2b94071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b94072eb0 0x7f2b94071df0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0097a0 tx=0x7f2b7c02b840 comp rx=0 tx=0).stop 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 -- 172.21.15.67:0/1372397407 shutdown_connections 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b94072eb0 0x7f2b94071df0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 --2- 172.21.15.67:0/1372397407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b9415ac80 0x7f2b9406f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 -- 172.21.15.67:0/1372397407 >> 172.21.15.67:0/1372397407 conn(0x7f2b94067d10 msgr2=0x7f2b940b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 -- 172.21.15.67:0/1372397407 shutdown_connections 2024-04-17T07:34:20.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.418+0000 7f2b98e00640 1 -- 172.21.15.67:0/1372397407 wait complete. 2024-04-17T07:34:20.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.420+0000 7f2b98e00640 1 Processor -- start 2024-04-17T07:34:20.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.420+0000 7f2b98e00640 1 -- start start 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 0x7f2b940ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b94072eb0 0x7f2b94155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9415b060 con 0x7f2b9415ac80 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9406ff40 con 0x7f2b94072330 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.421+0000 7f2b98e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b94156740 con 0x7f2b94072eb0 2024-04-17T07:34:20.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60808/0 (socket says 172.21.15.67:60808) 2024-04-17T07:34:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 -- 172.21.15.67:0/3351605047 learned_addr learned my addr 172.21.15.67:0/3351605047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93400640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 0x7f2b940ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b92a00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b94072eb0 0x7f2b94155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 -- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b94072eb0 msgr2=0x7f2b94155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b94072eb0 0x7f2b94155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 -- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 msgr2=0x7f2b940ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 0x7f2b940ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93e00640 1 -- 172.21.15.67:0/3351605047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b94156a20 con 0x7f2b9415ac80 2024-04-17T07:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.422+0000 7f2b93400640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 0x7f2b940ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b93e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c820 tx=0x7f2b8c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c018070 con 0x7f2b9415ac80 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8c00a5e0 con 0x7f2b9415ac80 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00a9d0 con 0x7f2b9415ac80 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b94156d00 con 0x7f2b9415ac80 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.423+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b941572c0 con 0x7f2b9415ac80 2024-04-17T07:34:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.424+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b94068a10 con 0x7f2b9415ac80 2024-04-17T07:34:20.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.425+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f2b8c00ab70 con 0x7f2b9415ac80 2024-04-17T07:34:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.427+0000 7f2b8b400640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 0x7f2b7006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.427+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f2b8c08b380 con 0x7f2b9415ac80 2024-04-17T07:34:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.427+0000 7f2b93400640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 0x7f2b7006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.429+0000 7f2b93400640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 0x7f2b7006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.431+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2b940cd920 con 0x7f2b9415ac80 2024-04-17T07:34:20.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.684+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b94157490 con 0x7f2b9415ac80 2024-04-17T07:34:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.686+0000 7f2b8b400640 1 -- 172.21.15.67:0/3351605047 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7f2b8c059680 con 0x7f2b9415ac80 2024-04-17T07:34:20.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:20.686 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T07:34:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 msgr2=0x7f2b7006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 0x7f2b7006eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84008040 comp rx=0 tx=0).stop 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 msgr2=0x7f2b940cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c820 tx=0x7f2b8c00cce0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 shutdown_connections 2024-04-17T07:34:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f2b7006ca10 0x7f2b7006eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b9415ac80 0x7f2b940cf890 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b94072eb0 0x7f2b94155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 --2- 172.21.15.67:0/3351605047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b94072330 0x7f2b940ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 >> 172.21.15.67:0/3351605047 conn(0x7f2b94067d10 msgr2=0x7f2b940cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 shutdown_connections 2024-04-17T07:34:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:20.689+0000 7f2b98e00640 1 -- 172.21.15.67:0/3351605047 wait complete. 2024-04-17T07:34:21.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T07:34:13.943539+0000', 'modified': '2024-04-17T07:34:17.018032+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/2582506112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 2582506112}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2582506112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8955, 'qdb_cluster': [8955]} 2024-04-17T07:34:21.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T07:34:21.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T07:34:21.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.340+0000 7fe088a00640 1 Processor -- start 2024-04-17T07:34:21.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.341+0000 7fe088a00640 1 -- start start 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe088a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe08415fdd0 0x7fe0840ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe088a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0840b3b70 0x7fe08409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe088a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe08406d830 con 0x7fe0840b3600 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe088a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe0841601b0 con 0x7fe0840b3b70 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe088a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe08406db10 con 0x7fe08415fdd0 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe082a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0840b3b70 0x7fe08409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe082a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0840b3b70 0x7fe08409aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60810/0 (socket says 172.21.15.67:60810) 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.342+0000 7fe082a00640 1 -- 172.21.15.67:0/748938633 learned_addr learned my addr 172.21.15.67:0/748938633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.343+0000 7fe082a00640 1 -- 172.21.15.67:0/748938633 >> v2:172.21.15.187:3300/0 conn(0x7fe08415fdd0 msgr2=0x7fe0840ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.343+0000 7fe082a00640 1 --2- 172.21.15.67:0/748938633 >> v2:172.21.15.187:3300/0 conn(0x7fe08415fdd0 0x7fe0840ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.343+0000 7fe082a00640 1 -- 172.21.15.67:0/748938633 >> v1:172.21.15.187:6789/0 conn(0x7fe0840b3600 legacy=0x7fe0840d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.343+0000 7fe082a00640 1 -- 172.21.15.67:0/748938633 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0840d0b40 con 0x7fe0840b3b70 2024-04-17T07:34:21.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.344+0000 7fe082a00640 1 --2- 172.21.15.67:0/748938633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3b70 0x7fe08409aa60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fe084073980 tx=0x7fe07002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a4c7419198ca6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.344+0000 7fe082000640 1 -- 172.21.15.67:0/748938633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070035020 con 0x7fe0840b3b70 2024-04-17T07:34:21.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.344+0000 7fe082000640 1 -- 172.21.15.67:0/748938633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe070031870 con 0x7fe0840b3b70 2024-04-17T07:34:21.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.344+0000 7fe082000640 1 -- 172.21.15.67:0/748938633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070031c40 con 0x7fe0840b3b70 2024-04-17T07:34:21.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 -- 172.21.15.67:0/748938633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3b70 msgr2=0x7fe08409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 --2- 172.21.15.67:0/748938633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3b70 0x7fe08409aa60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fe084073980 tx=0x7fe07002b910 comp rx=0 tx=0).stop 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 -- 172.21.15.67:0/748938633 shutdown_connections 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 --2- 172.21.15.67:0/748938633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3b70 0x7fe08409aa60 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 --2- 172.21.15.67:0/748938633 >> v2:172.21.15.187:3300/0 conn(0x7fe08415fdd0 0x7fe0840ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 -- 172.21.15.67:0/748938633 >> 172.21.15.67:0/748938633 conn(0x7fe0840b41c0 msgr2=0x7fe0840b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 -- 172.21.15.67:0/748938633 shutdown_connections 2024-04-17T07:34:21.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.345+0000 7fe088a00640 1 -- 172.21.15.67:0/748938633 wait complete. 2024-04-17T07:34:21.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.348+0000 7fe088a00640 1 Processor -- start 2024-04-17T07:34:21.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.348+0000 7fe088a00640 1 -- start start 2024-04-17T07:34:21.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0840b3b70 0x7fe0841f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08415fdd0 0x7fe084155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe08406df70 con 0x7fe0840b3600 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe08406ddf0 con 0x7fe08415fdd0 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe088a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe08406d3e0 con 0x7fe0840b3b70 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.349+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60820/0 (socket says 172.21.15.67:60820) 2024-04-17T07:34:21.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 -- 172.21.15.67:0/2726023394 learned_addr learned my addr 172.21.15.67:0/2726023394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe082a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0840b3b70 0x7fe0841f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083e00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08415fdd0 0x7fe084155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 -- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0840b3b70 msgr2=0x7fe0841f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0840b3b70 0x7fe0841f4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 -- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08415fdd0 msgr2=0x7fe084155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08415fdd0 0x7fe084155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 -- 172.21.15.67:0/2726023394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0841576c0 con 0x7fe0840b3600 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.350+0000 7fe083400640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe07400b410 tx=0x7fe07400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.351+0000 7fe07b400640 1 -- 172.21.15.67:0/2726023394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074015020 con 0x7fe0840b3600 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.351+0000 7fe07b400640 1 -- 172.21.15.67:0/2726023394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe074011850 con 0x7fe0840b3600 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.351+0000 7fe07b400640 1 -- 172.21.15.67:0/2726023394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074011c40 con 0x7fe0840b3600 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.351+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe084157960 con 0x7fe0840b3600 2024-04-17T07:34:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.351+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe084157f20 con 0x7fe0840b3600 2024-04-17T07:34:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.353+0000 7fe07b400640 1 -- 172.21.15.67:0/2726023394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fe074018020 con 0x7fe0840b3600 2024-04-17T07:34:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.354+0000 7fe07b400640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 0x7fe06406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.354+0000 7fe07b400640 1 -- 172.21.15.67:0/2726023394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fe07408bcb0 con 0x7fe0840b3600 2024-04-17T07:34:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.355+0000 7fe08cd98740 1 Processor -- start 2024-04-17T07:34:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.355+0000 7fe082a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 0x7fe06406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.355+0000 7fe08cd98740 1 -- start start 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796b36a0 0x5561796b5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796afb40 0x5561796df930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55617961b980 con 0x5561796fd620 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5561795c2220 con 0x5561796b36a0 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x556179617940 con 0x5561796afb40 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe082a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 0x7fe06406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe070036000 tx=0x7fe07003c040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60826/0 (socket says 172.21.15.67:60826) 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 -- 172.21.15.67:0/3742549765 learned_addr learned my addr 172.21.15.67:0/3742549765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe082a00640 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796b36a0 0x5561796b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083e00640 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796afb40 0x5561796df930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 -- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796afb40 msgr2=0x5561796df930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796afb40 0x5561796df930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 -- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796b36a0 msgr2=0x5561796b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796b36a0 0x5561796b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.356+0000 7fe083400640 1 -- 172.21.15.67:0/3742549765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55617932e2d0 con 0x5561796fd620 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.357+0000 7fe083400640 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe074061ec0 tx=0x7fe07405a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709d9cd3b4d14b65 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.357+0000 7fe06f400640 1 -- 172.21.15.67:0/3742549765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074088070 con 0x5561796fd620 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.357+0000 7fe06f400640 1 -- 172.21.15.67:0/3742549765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740578b0 con 0x5561796fd620 2024-04-17T07:34:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.357+0000 7fe06f400640 1 -- 172.21.15.67:0/3742549765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07408a110 con 0x5561796fd620 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- 172.21.15.67:0/3742549765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 msgr2=0x5561796b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fe074061ec0 tx=0x7fe07405a040 comp rx=0 tx=0).stop 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- 172.21.15.67:0/3742549765 shutdown_connections 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796afb40 0x5561796df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796b36a0 0x5561796b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 --2- 172.21.15.67:0/3742549765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796fd620 0x5561796b4c50 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- 172.21.15.67:0/3742549765 >> 172.21.15.67:0/3742549765 conn(0x5561796b1db0 msgr2=0x5561796fa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- 172.21.15.67:0/3742549765 shutdown_connections 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- 172.21.15.67:0/3742549765 wait complete. 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.358+0000 7fe08cd98740 1 -- start start 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.359+0000 7fe08cd98740 1 Processor -- start 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x5561796ff310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796ff850 0x5561796fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5561796b3bb0 con 0x5561796b36a0 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5561795d5590 con 0x5561796ff850 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5561796b3e90 con 0x5561796fd620 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe08cd98740 10 client.0 _ll_register_callbacks cb 0x55617932dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60832/0 (socket says 172.21.15.67:60832) 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 -- 172.21.15.67:0/1459625429 learned_addr learned my addr 172.21.15.67:0/1459625429 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083e00640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796ff850 0x5561796fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 -- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 msgr2=0x5561796ff310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x5561796ff310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 -- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796ff850 msgr2=0x5561796fda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796ff850 0x5561796fda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.360+0000 7fe083400640 1 -- 172.21.15.67:0/1459625429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561793f86b0 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe083400640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fe0740597a0 tx=0x7fe07401c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c68a7966403679 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074088070 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fe07405ab40 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0740578b0 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fe07405ad60 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe08cd98740 10 client.9090 fetch_fsmap learned FSMap version 76 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x5561796c2c00 con 0x5561796b36a0 2024-04-17T07:34:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740578b0 con 0x5561796b36a0 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 76) ==== 3495+0+0 (secure 0 0 0) 0x7fe074058c60 con 0x5561796b36a0 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.361+0000 7fe08cd98740 10 client.9090 fetch_fsmap finished waiting for FSMap version 76 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe08cd98740 10 client.9090 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe08cd98740 20 client.9090 populate_metadata read hostname 'smithi067' 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796c38a0 0x5561796c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe08cd98740 4 client.9090 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 --> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561796293a0 con 0x5561796c38a0 2024-04-17T07:34:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe082a00640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796c38a0 0x5561796c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe082a00640 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796c38a0 0x5561796c3c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68ccfd44b0dcf70b server_cookie=f8d2d333cf07b58 in_seq=0 out_seq=0 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.362+0000 7fe063e00640 10 client.9090 ms_handle_connect on v2:172.21.15.67:6833/2582506112 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.363+0000 7fe063e00640 1 -- 172.21.15.67:0/1459625429 <== mds.0 v2:172.21.15.67:6833/2582506112 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561796293a0 con 0x5561796c38a0 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.363+0000 7fe063e00640 10 client.9090 handle_command_reply: tid=0 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 client.9090 shutdown 2024-04-17T07:34:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 msgr2=0x5561796d6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fe0740597a0 tx=0x7fe07401c070 comp rx=0 tx=0).stop 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 shutdown_connections 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796c38a0 0x5561796c3c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796ff850 0x5561796fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x5561796ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1459625429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d6820 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe063e00640 0 client.9090 ms_handle_reset on v2:172.21.15.67:6833/2582506112 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 >> 172.21.15.67:0/1459625429 conn(0x5561796afb40 msgr2=0x5561796b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 shutdown_connections 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 1 -- 172.21.15.67:0/1459625429 wait complete. 2024-04-17T07:34:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.364+0000 7fe08cd98740 20 client.9090 trim_cache size 0 max 16384 2024-04-17T07:34:21.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.384+0000 7fe08cd98740 1 Processor -- start 2024-04-17T07:34:21.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.384+0000 7fe08cd98740 1 -- start start 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 0x5561796c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 0x5561796c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841fffd0 con 0x5561796b36a0 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841ff480 con 0x5561796fd620 2024-04-17T07:34:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0841ff980 con 0x5561796ff850 2024-04-17T07:34:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe083400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60848/0 (socket says 172.21.15.67:60848) 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.385+0000 7fe083400640 1 -- 172.21.15.67:0/112523745 learned_addr learned my addr 172.21.15.67:0/112523745 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe082a00640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 0x5561796c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083e00640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 0x5561796c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 -- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 msgr2=0x5561796c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 0x5561796c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 -- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 msgr2=0x5561796c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 0x5561796c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 -- 172.21.15.67:0/112523745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561796fdd70 con 0x5561796b36a0 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe082a00640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 0x5561796c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083e00640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 0x5561796c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe083400640 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe0740594c0 tx=0x7fe07405c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867c3a3ef1b968b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe06ea00640 1 -- 172.21.15.67:0/112523745 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074088070 con 0x5561796b36a0 2024-04-17T07:34:21.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe06ea00640 1 -- 172.21.15.67:0/112523745 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740172c0 con 0x5561796b36a0 2024-04-17T07:34:21.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.386+0000 7fe06ea00640 1 -- 172.21.15.67:0/112523745 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074059f10 con 0x5561796b36a0 2024-04-17T07:34:21.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 -- 172.21.15.67:0/112523745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 msgr2=0x5561796d5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe0740594c0 tx=0x7fe07405c040 comp rx=0 tx=0).stop 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 -- 172.21.15.67:0/112523745 shutdown_connections 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796ff850 0x5561796c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796fd620 0x5561796c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 --2- 172.21.15.67:0/112523745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796b36a0 0x5561796d5ee0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 -- 172.21.15.67:0/112523745 >> 172.21.15.67:0/112523745 conn(0x5561796afb40 msgr2=0x556179702820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 -- 172.21.15.67:0/112523745 shutdown_connections 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.387+0000 7fe08cd98740 1 -- 172.21.15.67:0/112523745 wait complete. 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.388+0000 7fe08cd98740 1 -- start start 2024-04-17T07:34:21.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.388+0000 7fe08cd98740 1 Processor -- start 2024-04-17T07:34:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x556179777cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796dec40 0x5561796aff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5561796b3bb0 con 0x5561796ff850 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5561796c29e0 con 0x5561796dec40 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5561795d5590 con 0x5561796fd620 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe08cd98740 10 client.0 _ll_register_callbacks cb 0x7fe074086070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe082a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe082a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60858/0 (socket says 172.21.15.67:60858) 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.389+0000 7fe082a00640 1 -- 172.21.15.67:0/1803266584 learned_addr learned my addr 172.21.15.67:0/1803266584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 -- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 msgr2=0x556179777cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x556179777cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 -- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796dec40 msgr2=0x5561796aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796dec40 0x5561796aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 -- 172.21.15.67:0/1803266584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561796fb210 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe082a00640 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe070031c20 tx=0x7fe070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a70c9989bc55711 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe063e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07004f600 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fe07004f7a0 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0740172c0 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.390+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fe07005e050 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 10 client.9096 fetch_fsmap learned FSMap version 76 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x5561796f9c60 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740172c0 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 76) ==== 3495+0+0 (secure 0 0 0) 0x7fe070061020 con 0x5561796ff850 2024-04-17T07:34:21.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 10 client.9096 fetch_fsmap finished waiting for FSMap version 76 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 10 client.9096 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 20 client.9096 populate_metadata read hostname 'smithi067' 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796f83c0 0x5561796f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 4 client.9096 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 --> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] -- command(tid 0: {"prefix": "status"}) -- 0x556179623210 con 0x5561796f83c0 2024-04-17T07:34:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.391+0000 7fe083400640 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796f83c0 0x5561796f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe083400640 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796f83c0 0x5561796f87a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f199b73fff388275 server_cookie=b49a9d5f533c228d in_seq=0 out_seq=0 2024-04-17T07:34:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe063e00640 10 client.9096 ms_handle_connect on v2:172.21.15.67:6833/2582506112 2024-04-17T07:34:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe063e00640 1 -- 172.21.15.67:0/1803266584 <== mds.0 v2:172.21.15.67:6833/2582506112 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556179623210 con 0x5561796f83c0 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe063e00640 10 client.9096 handle_command_reply: tid=0 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 client.9096 shutdown 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 msgr2=0x5561796b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe070031c20 tx=0x7fe070004730 comp rx=0 tx=0).stop 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 shutdown_connections 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:6833/2582506112,v1:172.21.15.67:6835/2582506112] conn(0x5561796f83c0 0x5561796f87a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5561796dec40 0x5561796aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5561796ff850 0x5561796b4ef0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 --2- 172.21.15.67:0/1803266584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5561796fd620 0x556179777cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 >> 172.21.15.67:0/1803266584 conn(0x5561796afb40 msgr2=0x556179702820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.392+0000 7fe063e00640 0 client.9096 ms_handle_reset on v2:172.21.15.67:6833/2582506112 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 shutdown_connections 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe08cd98740 1 -- 172.21.15.67:0/1803266584 wait complete. 2024-04-17T07:34:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe08cd98740 20 client.9096 trim_cache size 0 max 16384 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8955, 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.9296266170000003, 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 76, 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 156, 2024-04-17T07:34:21.397 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 156, 2024-04-17T07:34:21.398 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.281865414 2024-04-17T07:34:21.398 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T07:34:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 msgr2=0x7fe06406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 0x7fe06406eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe070036000 tx=0x7fe07003c040 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 msgr2=0x7fe0841f5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fe07400b410 tx=0x7fe07400b8d0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 shutdown_connections 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fe06406ca10 0x7fe06406eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe08415fdd0 0x7fe084155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0840b3b70 0x7fe0841f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 --2- 172.21.15.67:0/2726023394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0840b3600 0x7fe0841f5790 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 >> 172.21.15.67:0/2726023394 conn(0x7fe0840b41c0 msgr2=0x7fe084155700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.393+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 shutdown_connections 2024-04-17T07:34:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.394+0000 7fe088a00640 1 -- 172.21.15.67:0/2726023394 wait complete. 2024-04-17T07:34:21.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.619+0000 7fdc33c00640 1 Processor -- start 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.619+0000 7fdc33c00640 1 -- start start 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.620+0000 7fdc33c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc2c15b530 0x7fdc2c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc33c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc2c0c6140 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc33c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc2c15f3c0 con 0x7fdc2c15b910 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc33c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc2c06ed60 con 0x7fdc2c15b530 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc2be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdc2c0c68c0 0x7fdc2c0c5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55594/0 (socket says 172.21.15.67:55594) 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc2be00640 1 -- 172.21.15.67:0/3945843438 learned_addr learned my addr 172.21.15.67:0/3945843438 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308528038 0 0) 0x7fdc2c0c6140 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc14003520 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.621+0000 7fdc31200640 1 --2- 172.21.15.67:0/3945843438 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c15b530 0x7fdc2c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1466234732 0 0) 0x7fdc14003520 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c15b530 msgr2=0x7fdc2c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 --2- 172.21.15.67:0/3945843438 >> v2:172.21.15.187:3300/0 conn(0x7fdc2c15b530 0x7fdc2c071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 >> v1:172.21.15.187:6789/0 conn(0x7fdc2c15b910 legacy=0x7fdc2c0c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c15f660 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fdc18001670 con 0x7fdc2c0c68c0 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 >> v1:172.21.15.67:6789/0 conn(0x7fdc2c0c68c0 legacy=0x7fdc2c0c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc14003a00 0x7fdc14003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14004800 0x7fdc14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 0x7fdc1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c06ed60 con 0x7fdc14005230 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1400c190 con 0x7fdc14004800 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1400c390 con 0x7fdc14003a00 2024-04-17T07:34:21.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.622+0000 7fdc31c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 0x7fdc1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc2be00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14004800 0x7fdc14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31200640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc14003a00 0x7fdc14003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31c00640 1 -- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc14003a00 msgr2=0x7fdc14003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc14003a00 0x7fdc14003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31c00640 1 -- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14004800 msgr2=0x7fdc14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14004800 0x7fdc14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.623+0000 7fdc31c00640 1 -- 172.21.15.67:0/3945843438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc14003520 con 0x7fdc14005230 2024-04-17T07:34:21.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.624+0000 7fdc31c00640 1 -- 172.21.15.67:0/3945843438 reap_dead start 2024-04-17T07:34:21.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.624+0000 7fdc31c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 0x7fdc1400b9c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fdc24001e50 tx=0x7fdc14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed154da06ee55738 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.624+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24001a70 con 0x7fdc14005230 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.624+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc24003b90 con 0x7fdc14005230 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.624+0000 7fdc2b400640 1 -- 172.21.15.67:0/3945843438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc240050d0 con 0x7fdc14005230 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 -- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 msgr2=0x7fdc1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 0x7fdc1400b9c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fdc24001e50 tx=0x7fdc14005830 comp rx=0 tx=0).stop 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 -- 172.21.15.67:0/3945843438 shutdown_connections 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 --2- 172.21.15.67:0/3945843438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc14005230 0x7fdc1400b9c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 -- 172.21.15.67:0/3945843438 >> 172.21.15.67:0/3945843438 conn(0x7fdc2c06bf50 msgr2=0x7fdc2c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 -- 172.21.15.67:0/3945843438 shutdown_connections 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.625+0000 7fdc33c00640 1 -- 172.21.15.67:0/3945843438 wait complete. 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.627+0000 7fdc33c00640 1 Processor -- start 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.627+0000 7fdc33c00640 1 -- start start 2024-04-17T07:34:21.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14005230 0x7fdc2c0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c14dd70 0x7fdc2c0b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c0cc950 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c15fc20 con 0x7fdc14005230 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc33c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2c160140 con 0x7fdc2c14dd70 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc31c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc31c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60874/0 (socket says 172.21.15.67:60874) 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.628+0000 7fdc31c00640 1 -- 172.21.15.67:0/2076762256 learned_addr learned my addr 172.21.15.67:0/2076762256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31200640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14005230 0x7fdc2c0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc2be00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c14dd70 0x7fdc2c0b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 -- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c14dd70 msgr2=0x7fdc2c0b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c14dd70 0x7fdc2c0b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 -- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14005230 msgr2=0x7fdc2c0c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14005230 0x7fdc2c0c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 -- 172.21.15.67:0/2076762256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c0cee10 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc31c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fdc24001f70 tx=0x7fdc240014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.629+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24007db0 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.630+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc24006d50 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.630+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24039680 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.630+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c0cf0f0 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.630+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c1420e0 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.631+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c068b90 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.632+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7fdc24039820 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.633+0000 7fdc28c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 0x7fdc0c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:21.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.633+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fdc240ade10 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.633+0000 7fdc31200640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 0x7fdc0c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:21.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.635+0000 7fdc31200640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 0x7fdc0c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc20006fd0 tx=0x7fdc20008040 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:21.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.637+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdc240b1030 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.886+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdc2c0b0f40 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.887+0000 7fdc28c00640 1 -- 172.21.15.67:0/2076762256 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) ==== 53+0+73 (secure 0 0 0) 0x7fdc24033030 con 0x7fdc2c14e1f0 2024-04-17T07:34:21.887 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T07:34:21.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.889+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 msgr2=0x7fdc0c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.889+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 0x7fdc0c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc20006fd0 tx=0x7fdc20008040 comp rx=0 tx=0).stop 2024-04-17T07:34:21.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.889+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 msgr2=0x7fdc2c0b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:21.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.889+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fdc24001f70 tx=0x7fdc240014a0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 shutdown_connections 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fdc0c06ca40 0x7fdc0c06ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2c14e1f0 0x7fdc2c0b5830 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc2c14dd70 0x7fdc2c0b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 --2- 172.21.15.67:0/2076762256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc14005230 0x7fdc2c0c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 >> 172.21.15.67:0/2076762256 conn(0x7fdc2c06bf50 msgr2=0x7fdc2c14f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 shutdown_connections 2024-04-17T07:34:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:21.890+0000 7fdc33c00640 1 -- 172.21.15.67:0/2076762256 wait complete. 2024-04-17T07:34:21.909 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T07:34:21.909 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T07:34:21.909 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T07:34:21.909 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T07:34:21.910 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T07:34:21.910 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T07:34:21.910 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T07:34:21.910 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T07:34:21.910 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:34:21.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:21.910 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T07:34:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T07:34:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41382sec preferred_lft 41382sec 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T07:34:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.962 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T07:34:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T07:34:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T07:34:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T07:34:21.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:21.963 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T07:34:22.054 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T07:34:22.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:22.059 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T07:34:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T07:34:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T07:34:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T07:34:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T07:34:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T07:34:22.111 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T07:34:22.111 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T07:34:22.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T07:34:22.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T07:34:22.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T07:34:22.112 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T07:34:22.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T07:34:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:34:22.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:22.378 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T07:34:22.432 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T07:34:22.432 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T07:34:22.433 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:34:22.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:22.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T07:34:22.532 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:34:22.532 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T07:34:22.588 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T07:34:22.588 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T07:34:22.588 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:34:22.643 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T07:34:22.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:22.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:34:22.791 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:34:22.792 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:34:22.826 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T07:34:22.826 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T07:34:22.826 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T07:34:22.826 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T07:34:22.826 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T07:34:22.829 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T07:34:22.829 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T07:34:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T07:34:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T07:34:22.830 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T07:34:22.836 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T07:34:22.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:22.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T07:34:22.932 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T07:34:23.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T07:34:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T07:34:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:client7339 2024-04-17T07:34:23.074 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T07:34:23.074 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T07:34:23.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T07:34:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.337+0000 7f32f3200640 1 Processor -- start 2024-04-17T07:34:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.338+0000 7f32f3200640 1 -- start start 2024-04-17T07:34:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.339+0000 7f32f3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f32f415ee20 0x7f32f40d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.339+0000 7f32f3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32f40d4140 con 0x7f32f40d18e0 2024-04-17T07:34:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.339+0000 7f32f3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32f40d3b80 con 0x7f32f40d3520 2024-04-17T07:34:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.339+0000 7f32f3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f32f40b83b0 con 0x7f32f415ee20 2024-04-17T07:34:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.340+0000 7f32f1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32f40d18e0 0x7f32f40d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38476/0 (socket says 172.21.15.67:38476) 2024-04-17T07:34:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.340+0000 7f32f1400640 1 -- 172.21.15.67:0/3186541210 learned_addr learned my addr 172.21.15.67:0/3186541210 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.340+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3186541210 >> v2:172.21.15.187:3300/0 conn(0x7f32f415ee20 0x7f32f40d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.340+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609886651 0 0) 0x7f32f40d4140 con 0x7f32f40d18e0 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.340+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d8003520 con 0x7f32f40d18e0 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3283359581 0 0) 0x7f32d8003520 con 0x7f32f40d18e0 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 >> v2:172.21.15.187:3300/0 conn(0x7f32f415ee20 msgr2=0x7f32f40d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 --2- 172.21.15.67:0/3186541210 >> v2:172.21.15.187:3300/0 conn(0x7f32f415ee20 0x7f32f40d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 >> v1:172.21.15.187:6789/0 conn(0x7f32f40d3520 legacy=0x7f32f40b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f40d3e60 con 0x7f32f40d18e0 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f32dc0019f0 con 0x7f32f40d18e0 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 >> v1:172.21.15.67:6789/0 conn(0x7f32f40d18e0 legacy=0x7f32f40d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d8003a20 0x7f32d8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32d80051c0 0x7f32d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32f40b83b0 con 0x7f32d8004800 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f32d800c0f0 con 0x7f32d8003a20 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32d800c2f0 con 0x7f32d80051c0 2024-04-17T07:34:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f1400640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.341+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d8003a20 0x7f32d8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1400640 1 -- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32d80051c0 msgr2=0x7f32d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1400640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32d80051c0 0x7f32d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1400640 1 -- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d8003a20 msgr2=0x7f32d8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1400640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d8003a20 0x7f32d8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1400640 1 -- 172.21.15.67:0/3186541210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d8003520 con 0x7f32d8004800 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f2800640 1 -- 172.21.15.67:0/3186541210 reap_dead start 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.342+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32d8003a20 0x7f32d8003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.343+0000 7f32f1400640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32d8004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00a850 tx=0x7f32d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4fb364b7d9588c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.343+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00a4a0 con 0x7f32d8004800 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.343+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32dc034650 con 0x7f32d8004800 2024-04-17T07:34:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.343+0000 7f32f0a00640 1 -- 172.21.15.67:0/3186541210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc034a80 con 0x7f32d8004800 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 -- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 msgr2=0x7f32d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32d8004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00a850 tx=0x7f32d8005830 comp rx=0 tx=0).stop 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 -- 172.21.15.67:0/3186541210 shutdown_connections 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 --2- 172.21.15.67:0/3186541210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32d8004c80 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 -- 172.21.15.67:0/3186541210 >> 172.21.15.67:0/3186541210 conn(0x7f32f406d3c0 msgr2=0x7f32f40b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 -- 172.21.15.67:0/3186541210 shutdown_connections 2024-04-17T07:34:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.344+0000 7f32f3200640 1 -- 172.21.15.67:0/3186541210 wait complete. 2024-04-17T07:34:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.346+0000 7f32f3200640 1 Processor -- start 2024-04-17T07:34:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.346+0000 7f32f3200640 1 -- start start 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32f40ca3a0 0x7f32f415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32f40ca820 0x7f32f415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32f415f200 con 0x7f32d8004800 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f32f40d1d20 con 0x7f32f40ca820 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32f40d13a0 con 0x7f32f40ca3a0 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47460/0 (socket says 172.21.15.67:47460) 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.347+0000 7f32f1e00640 1 -- 172.21.15.67:0/3441422372 learned_addr learned my addr 172.21.15.67:0/3441422372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 -- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32f40ca3a0 msgr2=0x7f32f415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32f40ca3a0 0x7f32f415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 -- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32f40ca820 msgr2=0x7f32f415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32f40ca820 0x7f32f415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 -- 172.21.15.67:0/3441422372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f41550d0 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f1e00640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f32e40107c0 tx=0x7f32e4010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400f040 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e400eb60 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e401c400 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f40cae70 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.348+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f40c9270 con 0x7f32d8004800 2024-04-17T07:34:23.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.350+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32f4068a10 con 0x7f32d8004800 2024-04-17T07:34:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.351+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f32e401c5a0 con 0x7f32d8004800 2024-04-17T07:34:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.351+0000 7f32ea000640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 0x7f32d006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.352+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f32e408c2d0 con 0x7f32d8004800 2024-04-17T07:34:23.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.352+0000 7f32f1400640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 0x7f32d006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.354+0000 7f32f1400640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 0x7f32d006eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f32dc034000 tx=0x7f32dc009030 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.356+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f32e4018030 con 0x7f32d8004800 2024-04-17T07:34:23.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.589+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 --> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f32f40b5910 con 0x7f32d006ca10 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.593+0000 7f32ea000640 1 -- 172.21.15.67:0/3441422372 <== mgr.7087 v2:172.21.15.187:6832/115904466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f32f40b5910 con 0x7f32d006ca10 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T07:34:23.594 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 msgr2=0x7f32d006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 0x7f32d006eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f32dc034000 tx=0x7f32dc009030 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 msgr2=0x7f32f4154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f32e40107c0 tx=0x7f32e4010c80 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 shutdown_connections 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f32d006ca10 0x7f32d006eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32f40ca820 0x7f32f415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32f40ca3a0 0x7f32f415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 --2- 172.21.15.67:0/3441422372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004800 0x7f32f4154b90 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 >> 172.21.15.67:0/3441422372 conn(0x7f32f406d3c0 msgr2=0x7f32f415da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.596+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 shutdown_connections 2024-04-17T07:34:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.597+0000 7f32f3200640 1 -- 172.21.15.67:0/3441422372 wait complete. 2024-04-17T07:34:23.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T07:34:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.835+0000 7f497f200640 1 Processor -- start 2024-04-17T07:34:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.835+0000 7f497f200640 1 -- start start 2024-04-17T07:34:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.837+0000 7f497f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 0x7f49780d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f497f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49780b4f80 0x7f49780d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f497f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4978071f60 con 0x7f49780ce4d0 2024-04-17T07:34:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f497f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4978071d10 con 0x7f49780b4f80 2024-04-17T07:34:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f497f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4978072240 con 0x7f497815b860 2024-04-17T07:34:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f4977400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f49780ce4d0 0x7f49780ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:50306/0 (socket says 172.21.15.67:50306) 2024-04-17T07:34:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.838+0000 7f4977400640 1 -- 172.21.15.67:0/2425233663 learned_addr learned my addr 172.21.15.67:0/2425233663 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f497d200640 1 --2- 172.21.15.67:0/2425233663 >> v2:172.21.15.187:3300/0 conn(0x7f49780b4f80 0x7f49780d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4977e00640 1 --2- 172.21.15.67:0/2425233663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 0x7f49780d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4976a00640 1 -- 172.21.15.67:0/2425233663 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584788179 0 0) 0x7f4978071f60 con 0x7f49780ce4d0 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4976a00640 1 -- 172.21.15.67:0/2425233663 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f495c003520 con 0x7f49780ce4d0 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4977e00640 1 -- 172.21.15.67:0/2425233663 >> v2:172.21.15.187:3300/0 conn(0x7f49780b4f80 msgr2=0x7f49780d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4977e00640 1 --2- 172.21.15.67:0/2425233663 >> v2:172.21.15.187:3300/0 conn(0x7f49780b4f80 0x7f49780d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4977e00640 1 -- 172.21.15.67:0/2425233663 >> v1:172.21.15.187:6789/0 conn(0x7f49780ce4d0 legacy=0x7f49780ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.839+0000 7f4977e00640 1 -- 172.21.15.67:0/2425233663 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4978072520 con 0x7f497815b860 2024-04-17T07:34:23.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.841+0000 7f4977e00640 1 --2- 172.21.15.67:0/2425233663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 0x7f49780d0420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4968009d90 tx=0x7f496802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57d9a30bc39731d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.841+0000 7f4976a00640 1 -- 172.21.15.67:0/2425233663 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4968039070 con 0x7f497815b860 2024-04-17T07:34:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.841+0000 7f4976a00640 1 -- 172.21.15.67:0/2425233663 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496802cc50 con 0x7f497815b860 2024-04-17T07:34:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.841+0000 7f4976a00640 1 -- 172.21.15.67:0/2425233663 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4968034440 con 0x7f497815b860 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 -- 172.21.15.67:0/2425233663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 msgr2=0x7f49780d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 --2- 172.21.15.67:0/2425233663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 0x7f49780d0420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4968009d90 tx=0x7f496802bea0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 -- 172.21.15.67:0/2425233663 shutdown_connections 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 --2- 172.21.15.67:0/2425233663 >> v2:172.21.15.187:3300/0 conn(0x7f49780b4f80 0x7f49780d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 --2- 172.21.15.67:0/2425233663 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f497815b860 0x7f49780d0420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 -- 172.21.15.67:0/2425233663 >> 172.21.15.67:0/2425233663 conn(0x7f4978067ee0 msgr2=0x7f497806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 -- 172.21.15.67:0/2425233663 shutdown_connections 2024-04-17T07:34:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.842+0000 7f497f200640 1 -- 172.21.15.67:0/2425233663 wait complete. 2024-04-17T07:34:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.844+0000 7f497f200640 1 Processor -- start 2024-04-17T07:34:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.844+0000 7f497f200640 1 -- start start 2024-04-17T07:34:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 0x7f49780c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 0x7f497815d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f497815bc40 con 0x7f49780ce4d0 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4978072ae0 con 0x7f497815b860 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.845+0000 7f497f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f497806f380 con 0x7f49780b4f80 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47474/0 (socket says 172.21.15.67:47474) 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 -- 172.21.15.67:0/4265192737 learned_addr learned my addr 172.21.15.67:0/4265192737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977e00640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 0x7f49780c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f497d200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 0x7f497815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 -- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 msgr2=0x7f49780c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 0x7f49780c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 -- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 msgr2=0x7f497815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 0x7f497815d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977400640 1 -- 172.21.15.67:0/4265192737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49780ca440 con 0x7f49780ce4d0 2024-04-17T07:34:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f4977e00640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 0x7f49780c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.846+0000 7f497d200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 0x7f497815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f4977400640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f496000c430 tx=0x7f496000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960017070 con 0x7f49780ce4d0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4960005ce0 con 0x7f49780ce4d0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49600072f0 con 0x7f49780ce4d0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497814d4a0 con 0x7f49780ce4d0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.847+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497814da60 con 0x7f49780ce4d0 2024-04-17T07:34:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.848+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4978068ca0 con 0x7f49780ce4d0 2024-04-17T07:34:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.849+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 78) ==== 91142+0+0 (secure 0 0 0) 0x7f496001a070 con 0x7f49780ce4d0 2024-04-17T07:34:23.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.850+0000 7f496fe00640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 0x7f495406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:23.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.850+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f496008b2a0 con 0x7f49780ce4d0 2024-04-17T07:34:23.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.851+0000 7f4977e00640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 0x7f495406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:23.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.852+0000 7f4977e00640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 0x7f495406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4968036000 tx=0x7f4968030000 comp rx=0 tx=0).ready entity=mgr.7087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:23.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:23.855+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4960059500 con 0x7f49780ce4d0 2024-04-17T07:34:24.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.113+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f49780b2bb0 con 0x7f49780ce4d0 2024-04-17T07:34:24.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.555+0000 7f496fe00640 1 -- 172.21.15.67:0/4265192737 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) ==== 92+0+0 (secure 0 0 0) 0x7f4960058e00 con 0x7f49780ce4d0 2024-04-17T07:34:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.560+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 msgr2=0x7f495406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.560+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 0x7f495406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4968036000 tx=0x7f4968030000 comp rx=0 tx=0).stop 2024-04-17T07:34:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.560+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 msgr2=0x7f497815ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.560+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f496000c430 tx=0x7f496000c8f0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.561+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 shutdown_connections 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.561+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f495406ca10 0x7f495406eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.561+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f497815b860 0x7f497815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.561+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49780ce4d0 0x7f497815ce10 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.562+0000 7f497f200640 1 --2- 172.21.15.67:0/4265192737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49780b4f80 0x7f49780c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.562+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 >> 172.21.15.67:0/4265192737 conn(0x7f4978067ee0 msgr2=0x7f49780b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:24.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.563+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 shutdown_connections 2024-04-17T07:34:24.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.563+0000 7f497f200640 1 -- 172.21.15.67:0/4265192737 wait complete. 2024-04-17T07:34:24.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T07:34:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.794+0000 7f0ff1e00640 1 Processor -- start 2024-04-17T07:34:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.794+0000 7f0ff1e00640 1 -- start start 2024-04-17T07:34:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.796+0000 7f0ff1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0fec06daf0 0x7f0fec0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.796+0000 7f0ff1e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0fec0d1060 0x7f0fec09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.796+0000 7f0ff1e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fec0d1520 con 0x7f0fec06ce00 2024-04-17T07:34:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.796+0000 7f0ff1e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0fec0d1800 con 0x7f0fec06daf0 2024-04-17T07:34:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.796+0000 7f0ff1e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0fec0d1ae0 con 0x7f0fec0d1060 2024-04-17T07:34:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0fec06daf0 0x7f0fec0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0fec06daf0 0x7f0fec0d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47494/0 (socket says 172.21.15.67:47494) 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 -- 172.21.15.67:0/3734832869 learned_addr learned my addr 172.21.15.67:0/3734832869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0fe3e00640 1 --2- 172.21.15.67:0/3734832869 >> v2:172.21.15.187:3300/0 conn(0x7f0fec0d1060 0x7f0fec09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 -- 172.21.15.67:0/3734832869 >> v2:172.21.15.187:3300/0 conn(0x7f0fec0d1060 msgr2=0x7f0fec09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 --2- 172.21.15.67:0/3734832869 >> v2:172.21.15.187:3300/0 conn(0x7f0fec0d1060 0x7f0fec09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 -- 172.21.15.67:0/3734832869 >> v1:172.21.15.187:6789/0 conn(0x7f0fec06ce00 legacy=0x7f0fec1628a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.797+0000 7f0feaa00640 1 -- 172.21.15.67:0/3734832869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec0d1dc0 con 0x7f0fec06daf0 2024-04-17T07:34:24.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.798+0000 7f0feaa00640 1 --2- 172.21.15.67:0/3734832869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec06daf0 0x7f0fec0d0b20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0738f0 tx=0x7f0fdc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b1e98c2df3d6e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:24.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.799+0000 7f0fea000640 1 -- 172.21.15.67:0/3734832869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc034070 con 0x7f0fec06daf0 2024-04-17T07:34:24.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.799+0000 7f0fea000640 1 -- 172.21.15.67:0/3734832869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fdc030860 con 0x7f0fec06daf0 2024-04-17T07:34:24.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.799+0000 7f0fea000640 1 -- 172.21.15.67:0/3734832869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc030c30 con 0x7f0fec06daf0 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.799+0000 7f0ff1e00640 1 -- 172.21.15.67:0/3734832869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec06daf0 msgr2=0x7f0fec0d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.799+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/3734832869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec06daf0 0x7f0fec0d0b20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0738f0 tx=0x7f0fdc02b9a0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 -- 172.21.15.67:0/3734832869 shutdown_connections 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/3734832869 >> v2:172.21.15.187:3300/0 conn(0x7f0fec0d1060 0x7f0fec09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/3734832869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec06daf0 0x7f0fec0d0b20 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 -- 172.21.15.67:0/3734832869 >> 172.21.15.67:0/3734832869 conn(0x7f0fec0b41a0 msgr2=0x7f0fec0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 -- 172.21.15.67:0/3734832869 shutdown_connections 2024-04-17T07:34:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.800+0000 7f0ff1e00640 1 -- 172.21.15.67:0/3734832869 wait complete. 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.801+0000 7f0ff1e00640 1 Processor -- start 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 -- start start 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fec06ce00 0x7f0fec15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fec06daf0 0x7f0fec1f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec06e050 con 0x7f0fec0d1060 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec06ded0 con 0x7f0fec06ce00 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.802+0000 7f0ff1e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0fec071050 con 0x7f0fec06daf0 2024-04-17T07:34:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47496/0 (socket says 172.21.15.67:47496) 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 -- 172.21.15.67:0/125690847 learned_addr learned my addr 172.21.15.67:0/125690847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feaa00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fec06ce00 0x7f0fec15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0fe3e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fec06daf0 0x7f0fec1f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fec06daf0 msgr2=0x7f0fec1f4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fec06daf0 0x7f0fec1f4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fec06ce00 msgr2=0x7f0fec15dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fec06ce00 0x7f0fec15dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.803+0000 7f0feb400640 1 -- 172.21.15.67:0/125690847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec1f5300 con 0x7f0fec0d1060 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0feb400640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c7a0 tx=0x7f0fe400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4003b70 con 0x7f0fec0d1060 2024-04-17T07:34:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe4003d30 con 0x7f0fec0d1060 2024-04-17T07:34:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe400aa90 con 0x7f0fec0d1060 2024-04-17T07:34:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fec1f55e0 con 0x7f0fec0d1060 2024-04-17T07:34:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.804+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fec0cb140 con 0x7f0fec0d1060 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.806+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fec0cb340 con 0x7f0fec0d1060 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.807+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 79) ==== 91159+0+0 (secure 0 0 0) 0x7f0fe400ac30 con 0x7f0fec0d1060 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.808+0000 7f0fe3400640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 0x7f0fc806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.808+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f0fe408bd80 con 0x7f0fec0d1060 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.808+0000 7f0feaa00640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 msgr2=0x7f0fc806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.808+0000 7f0feaa00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 0x7f0fc806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:34:24.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:24.812+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0fec0cb340 con 0x7f0fec0d1060 2024-04-17T07:34:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.009+0000 7f0feaa00640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 msgr2=0x7f0fc806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.009+0000 7f0feaa00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 0x7f0fc806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:34:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.021+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f0fec1f57b0 con 0x7f0fec0d1060 2024-04-17T07:34:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.026+0000 7f0fe3400640 1 -- 172.21.15.67:0/125690847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) ==== 138+0+0 (secure 0 0 0) 0x7f0fe405ee20 con 0x7f0fec0d1060 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 msgr2=0x7f0fc806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 0x7f0fc806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 msgr2=0x7f0fec0ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f0fe400c7a0 tx=0x7f0fe400cc60 comp rx=0 tx=0).stop 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 shutdown_connections 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7f0fc806ca60 0x7f0fc806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fec0d1060 0x7f0fec0ccce0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0fec06daf0 0x7f0fec1f4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 --2- 172.21.15.67:0/125690847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fec06ce00 0x7f0fec15dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.028+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 >> 172.21.15.67:0/125690847 conn(0x7f0fec0b41a0 msgr2=0x7f0fec071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:25.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.029+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 shutdown_connections 2024-04-17T07:34:25.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.029+0000 7f0ff1e00640 1 -- 172.21.15.67:0/125690847 wait complete. 2024-04-17T07:34:25.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T07:34:25.108 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.108+0000 7f102de2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:34:25.143 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.142+0000 7f85a7e3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T07:34:25.162 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.163+0000 7f102de2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:34:25.201 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.201+0000 7f85a7e3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T07:34:25.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.262+0000 7fd6f9e00640 1 Processor -- start 2024-04-17T07:34:25.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.262+0000 7fd6f9e00640 1 -- start start 2024-04-17T07:34:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6f40aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd6f415bae0 0x7fd6f406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40b3cc0 con 0x7fd6f406ce10 2024-04-17T07:34:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f415c3a0 con 0x7fd6f415bae0 2024-04-17T07:34:25.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f415c680 con 0x7fd6f40ae790 2024-04-17T07:34:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6f406ce10 0x7fd6f406f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38490/0 (socket says 172.21.15.67:38490) 2024-04-17T07:34:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.264+0000 7fd6f9400640 1 -- 172.21.15.67:0/2249285797 learned_addr learned my addr 172.21.15.67:0/2249285797 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> v2:172.21.15.187:3300/0 conn(0x7fd6f415bae0 0x7fd6f406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6f40aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4072306614 0 0) 0x7fd6f40b3cc0 con 0x7fd6f406ce10 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6dc005730 con 0x7fd6f406ce10 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3964892913 0 0) 0x7fd6dc005730 con 0x7fd6f406ce10 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 msgr2=0x7fd6f40aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6f40aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 >> v2:172.21.15.187:3300/0 conn(0x7fd6f415bae0 msgr2=0x7fd6f406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 --2- 172.21.15.67:0/2249285797 >> v2:172.21.15.187:3300/0 conn(0x7fd6f415bae0 0x7fd6f406c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f415c960 con 0x7fd6f406ce10 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd6ec0019f0 con 0x7fd6f406ce10 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> v2:172.21.15.187:3300/0 conn(0x7fd6f415bae0 0x7fd6f406c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 >> v1:172.21.15.67:6789/0 conn(0x7fd6f406ce10 legacy=0x7fd6f406f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.265+0000 7fd6f3400640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6dc005bc0 0x7fd6dc005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3400640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6dc006a10 0x7fd6dc006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3400640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6dc0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f415c3a0 con 0x7fd6dc006a10 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f415c680 con 0x7fd6dc005bc0 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6dc00dbd0 con 0x7fd6f40ae790 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6dc006a10 0x7fd6dc006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6dc005bc0 0x7fd6dc005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f9400640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6dc0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 msgr2=0x7fd6dc0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6f40ae790 0x7fd6dc0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6dc005bc0 msgr2=0x7fd6dc005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6dc005bc0 0x7fd6dc005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2249285797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6dc005730 con 0x7fd6dc006a10 2024-04-17T07:34:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.266+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6dc005bc0 0x7fd6dc005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.267+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6dc006a10 0x7fd6dc006e90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009c90 tx=0x7fd6dc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6550c8ff203646e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0039070 con 0x7fd6dc006a10 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e002c980 con 0x7fd6dc006a10 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f3400640 1 -- 172.21.15.67:0/2249285797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0009860 con 0x7fd6dc006a10 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2249285797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6dc006a10 msgr2=0x7fd6dc006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2249285797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6dc006a10 0x7fd6dc006e90 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009c90 tx=0x7fd6dc007a40 comp rx=0 tx=0).stop 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.268+0000 7fd6f9400640 1 -- 172.21.15.67:0/2249285797 reap_dead start 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.269+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2249285797 shutdown_connections 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.269+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2249285797 >> 172.21.15.67:0/2249285797 conn(0x7fd6f4072460 msgr2=0x7fd6f4072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.269+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2249285797 shutdown_connections 2024-04-17T07:34:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.269+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2249285797 wait complete. 2024-04-17T07:34:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.270+0000 7fd6f9e00640 1 Processor -- start 2024-04-17T07:34:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.271+0000 7fd6f9e00640 1 -- start start 2024-04-17T07:34:25.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6dc006a10 0x7fd6f4141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6f40caf90 0x7fd6f41f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f415c0e0 con 0x7fd6f409cb90 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f415bf60 con 0x7fd6f40caf90 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f40b5490 con 0x7fd6dc006a10 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47516/0 (socket says 172.21.15.67:47516) 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2475156995 learned_addr learned my addr 172.21.15.67:0/2475156995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6dc006a10 msgr2=0x7fd6f4141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f9400640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6f40caf90 0x7fd6f41f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6dc006a10 0x7fd6f4141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6dc006a10 0x7fd6f4141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6f40caf90 msgr2=0x7fd6f41f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6f40caf90 0x7fd6f41f4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.272+0000 7fd6f3e00640 1 -- 172.21.15.67:0/2475156995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f409ac00 con 0x7fd6f409cb90 2024-04-17T07:34:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f3e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009d80 tx=0x7fd6e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0042020 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e003a040 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0039070 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f409aee0 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.273+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f409d010 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.275+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f40755d0 con 0x7fd6f409cb90 2024-04-17T07:34:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.275+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 79) ==== 91159+0+0 (secure 0 0 0) 0x7fd6e0045020 con 0x7fd6f409cb90 2024-04-17T07:34:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.277+0000 7fd6f0c00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 0x7fd6d406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.277+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fd6e00ad460 con 0x7fd6f409cb90 2024-04-17T07:34:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.277+0000 7fd6f8a00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 msgr2=0x7fd6d406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.277+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 0x7fd6d406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:34:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.281+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd6e004c070 con 0x7fd6f409cb90 2024-04-17T07:34:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.478+0000 7fd6f8a00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 msgr2=0x7fd6d406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.478+0000 7fd6f8a00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 0x7fd6d406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:34:25.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.482+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd6f40cb370 con 0x7fd6f409cb90 2024-04-17T07:34:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.486+0000 7fd6f0c00640 1 -- 172.21.15.67:0/2475156995 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) ==== 133+0+0 (secure 0 0 0) 0x7fd6e007b760 con 0x7fd6f409cb90 2024-04-17T07:34:25.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 msgr2=0x7fd6d406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:34:25.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 0x7fd6d406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 msgr2=0x7fd6f40caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009d80 tx=0x7fd6e0004730 comp rx=0 tx=0).stop 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 shutdown_connections 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7fd6d406ca60 0x7fd6d406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6f40caf90 0x7fd6f41f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6f409cb90 0x7fd6f40caa50 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 --2- 172.21.15.67:0/2475156995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6dc006a10 0x7fd6f4141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.489+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 >> 172.21.15.67:0/2475156995 conn(0x7fd6f4072460 msgr2=0x7fd6f40b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:25.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.490+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 shutdown_connections 2024-04-17T07:34:25.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.490+0000 7fd6f9e00640 1 -- 172.21.15.67:0/2475156995 wait complete. 2024-04-17T07:34:25.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T07:34:25.510 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T07:34:25.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-04-17T07:34:25.611 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.612+0000 7f102de2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:34:25.663 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.664+0000 7f102de2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:34:25.680 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.680+0000 7f85a7e3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T07:34:25.714 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.715+0000 7f102de2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:34:25.736 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.736+0000 7f85a7e3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T07:34:25.774 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.775+0000 7f102de2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:34:25.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.777+0000 7fab22200640 1 Processor -- start 2024-04-17T07:34:25.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.777+0000 7fab22200640 1 -- start start 2024-04-17T07:34:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab22200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab1c0d33e0 0x7fab1c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab22200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c0d00b0 con 0x7fab1c0cf8e0 2024-04-17T07:34:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab22200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c0d0390 con 0x7fab1c15ecb0 2024-04-17T07:34:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab22200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c0d0c30 con 0x7fab1c0d33e0 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab21800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab1c0cf8e0 0x7fab1c0b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38504/0 (socket says 172.21.15.67:38504) 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab1be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab1c0d33e0 0x7fab1c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.779+0000 7fab21800640 1 -- 172.21.15.67:0/2195598174 learned_addr learned my addr 172.21.15.67:0/2195598174 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.780+0000 7fab1b400640 1 -- 172.21.15.67:0/2195598174 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3426286782 0 0) 0x7fab1c0d00b0 con 0x7fab1c0cf8e0 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.780+0000 7fab1be00640 1 -- 172.21.15.67:0/2195598174 >> v1:172.21.15.187:6789/0 conn(0x7fab1c15ecb0 legacy=0x7fab1c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.780+0000 7fab1be00640 1 -- 172.21.15.67:0/2195598174 >> v1:172.21.15.67:6789/0 conn(0x7fab1c0cf8e0 legacy=0x7fab1c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.780+0000 7fab1be00640 1 -- 172.21.15.67:0/2195598174 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c0d0670 con 0x7fab1c0d33e0 2024-04-17T07:34:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.781+0000 7fab1be00640 1 --2- 172.21.15.67:0/2195598174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c0d33e0 0x7fab1c0cf3a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fab080097a0 tx=0x7fab0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f07e44b6255694 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.781+0000 7fab1b400640 1 -- 172.21.15.67:0/2195598174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08035020 con 0x7fab1c0d33e0 2024-04-17T07:34:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.781+0000 7fab1b400640 1 -- 172.21.15.67:0/2195598174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab08031830 con 0x7fab1c0d33e0 2024-04-17T07:34:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.781+0000 7fab1b400640 1 -- 172.21.15.67:0/2195598174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08031c00 con 0x7fab1c0d33e0 2024-04-17T07:34:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 -- 172.21.15.67:0/2195598174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c0d33e0 msgr2=0x7fab1c0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 --2- 172.21.15.67:0/2195598174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c0d33e0 0x7fab1c0cf3a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fab080097a0 tx=0x7fab0802b8d0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 -- 172.21.15.67:0/2195598174 shutdown_connections 2024-04-17T07:34:25.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 --2- 172.21.15.67:0/2195598174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c0d33e0 0x7fab1c0cf3a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 -- 172.21.15.67:0/2195598174 >> 172.21.15.67:0/2195598174 conn(0x7fab1c08bff0 msgr2=0x7fab1c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:25.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 -- 172.21.15.67:0/2195598174 shutdown_connections 2024-04-17T07:34:25.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.782+0000 7fab22200640 1 -- 172.21.15.67:0/2195598174 wait complete. 2024-04-17T07:34:25.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.784+0000 7fab22200640 1 Processor -- start 2024-04-17T07:34:25.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.784+0000 7fab22200640 1 -- start start 2024-04-17T07:34:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab1c0cf8e0 0x7fab1c0b9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab1c0d33e0 0x7fab1c1f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 0x7fab1c0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c15f090 con 0x7fab1c15ecb0 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0cfd30 con 0x7fab1c0d33e0 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab22200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c15f210 con 0x7fab1c0cf8e0 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab21800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 0x7fab1c0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab21800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 0x7fab1c0cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47544/0 (socket says 172.21.15.67:47544) 2024-04-17T07:34:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.785+0000 7fab21800640 1 -- 172.21.15.67:0/305940599 learned_addr learned my addr 172.21.15.67:0/305940599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:25.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab1be00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab1c0d33e0 0x7fab1c1f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab1c0cf8e0 msgr2=0x7fab1c0b9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab1c0cf8e0 0x7fab1c0b9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:25.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab1c0cf8e0 0x7fab1c0b9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab1c0d33e0 msgr2=0x7fab1c1f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab1c0d33e0 0x7fab1c1f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c1f6100 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab21800640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 0x7fab1c0cc800 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c810 tx=0x7fab1400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14018070 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1400a640 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1400aa30 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.786+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c1f63e0 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.787+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c1f69a0 con 0x7fab1c15ecb0 2024-04-17T07:34:25.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.788+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1c068a10 con 0x7fab1c15ecb0 2024-04-17T07:34:25.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.789+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 79) ==== 91159+0+0 (secure 0 0 0) 0x7fab1400abd0 con 0x7fab1c15ecb0 2024-04-17T07:34:25.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.789+0000 7fab18c00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:25.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.789+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fab1408b420 con 0x7fab1c15ecb0 2024-04-17T07:34:25.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.790+0000 7fab20e00640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 msgr2=0x7faafc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:25.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.790+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T07:34:25.792 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.792+0000 7f85a7e3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T07:34:25.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.794+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fab14059680 con 0x7fab1c15ecb0 2024-04-17T07:34:25.825 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.826+0000 7f102de2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:34:25.857 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.857+0000 7f85a7e3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T07:34:25.910 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:25.909+0000 7f85a7e3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T07:34:25.954 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:25.955+0000 7f102de2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:34:25.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.991+0000 7fab20e00640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 msgr2=0x7faafc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:25.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:25.991+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T07:34:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:26.016+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fab1c06e290 con 0x7faafc06ca60 2024-04-17T07:34:26.017 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.017+0000 7f102de2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:34:26.046 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.045+0000 7f85a7e3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T07:34:26.068 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.068+0000 7f102de2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:34:26.111 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.111+0000 7f85a7e3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T07:34:26.124 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.125+0000 7f102de2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:34:26.163 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.163+0000 7f85a7e3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T07:34:26.174 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.175+0000 7f102de2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:34:26.223 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.223+0000 7f85a7e3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T07:34:26.276 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.275+0000 7f85a7e3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T07:34:26.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:26.392+0000 7fab20e00640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 msgr2=0x7faafc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:26.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:26.392+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T07:34:26.560 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.560+0000 7f102de2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:34:26.676 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.675+0000 7f85a7e3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T07:34:26.718 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.718+0000 7f102de2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:34:26.825 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.826+0000 7f102de2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:34:26.842 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.841+0000 7f85a7e3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T07:34:26.876 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:26.877+0000 7f102de2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:34:26.955 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:26.955+0000 7f85a7e3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T07:34:27.001 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:27.002+0000 7f102de2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:34:27.008 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.008+0000 7f85a7e3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T07:34:27.108 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:27.109+0000 7f102de2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:34:27.140 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.140+0000 7f85a7e3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:34:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:27.193+0000 7fab20e00640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 msgr2=0x7faafc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/115904466 2024-04-17T07:34:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:27.193+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T07:34:27.251 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.251+0000 7f85a7e3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T07:34:27.282 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:27.283+0000 7f102de2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:34:27.331 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:27.332+0000 7f102de2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:34:27.435 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.434+0000 7f85a7e3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T07:34:27.469 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:27.469+0000 7f102de2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:34:27.486 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.486+0000 7f85a7e3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T07:34:27.628 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:27.628+0000 7f85a7e3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T07:34:28.277 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:28.277+0000 7f102de2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:34:28.510 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:28.510+0000 7f85a7e3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T07:34:28.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:28.579+0000 7f102de2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:34:28.635 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:28.636+0000 7f102de2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:34:28.766 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:28.767+0000 7f102de2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:34:28.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:28.791+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 80) ==== 90768+0+0 (secure 0 0 0) 0x7fab140555f0 con 0x7fab1c15ecb0 2024-04-17T07:34:28.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:28.792+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 msgr2=0x7faafc06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T07:34:28.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:28.792+0000 7fab18c00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/115904466,v1:172.21.15.187:6833/115904466] conn(0x7faafc06ca60 0x7faafc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:28.864 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:28.863+0000 7f85a7e3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T07:34:28.927 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:28.927+0000 7f85a7e3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.046+0000 7f0fa0600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa0600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa0600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa0600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa0600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.046 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.047 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.047 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.047 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.047 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:34:29.047+0000 7f0fa4200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:34:29.071 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T07:34:29.071+0000 7f85a7e3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T07:34:29.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:34:29.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:34:29.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.797+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 81) ==== 90895+0+0 (secure 0 0 0) 0x7fab140616d0 con 0x7fab1c15ecb0 2024-04-17T07:34:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.797+0000 7fab18c00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7faafc074970 0x7faafc076d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.798+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 --> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fab1408b210 con 0x7faafc074970 2024-04-17T07:34:29.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.801+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7faafc074970 0x7faafc076d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:29.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.814+0000 7fab20e00640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7faafc074970 0x7faafc076d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fab10000f60 tx=0x7fab10007c70 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.989+0000 7fab18c00640 1 -- 172.21.15.67:0/305940599 <== mgr.6737 v2:172.21.15.187:6832/2413623528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fab1408b210 con 0x7faafc074970 2024-04-17T07:34:29.989 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1 2024-04-17T07:34:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7faafc074970 msgr2=0x7faafc076d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7faafc074970 0x7faafc076d60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fab10000f60 tx=0x7fab10007c70 comp rx=0 tx=0).stop 2024-04-17T07:34:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 msgr2=0x7fab1c0cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 --2- 172.21.15.67:0/305940599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab1c15ecb0 0x7fab1c0cc800 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c810 tx=0x7fab1400ccd0 comp rx=0 tx=0).stop 2024-04-17T07:34:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab21800640 1 -- 172.21.15.67:0/305940599 reap_dead start 2024-04-17T07:34:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 shutdown_connections 2024-04-17T07:34:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 >> 172.21.15.67:0/305940599 conn(0x7fab1c08bff0 msgr2=0x7fab1c06d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 shutdown_connections 2024-04-17T07:34:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:29.992+0000 7fab22200640 1 -- 172.21.15.67:0/305940599 wait complete. 2024-04-17T07:34:30.017 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-04-17T07:34:30.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-04-17T07:34:30.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.248+0000 7fcb61000640 1 Processor -- start 2024-04-17T07:34:30.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.249+0000 7fcb61000640 1 -- start start 2024-04-17T07:34:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb61000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb5c0cfe30 0x7fcb5c1662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb61000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5c0b5860 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb61000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5c0b4060 con 0x7fcb5c0b4e20 2024-04-17T07:34:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb61000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5c0b4340 con 0x7fcb5c0cfe30 2024-04-17T07:34:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb59e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb5c0cd7c0 0x7fcb5c0cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38514/0 (socket says 172.21.15.67:38514) 2024-04-17T07:34:30.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.251+0000 7fcb59e00640 1 -- 172.21.15.67:0/2093403795 learned_addr learned my addr 172.21.15.67:0/2093403795 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb5a800640 1 --2- 172.21.15.67:0/2093403795 >> v2:172.21.15.187:3300/0 conn(0x7fcb5c0cfe30 0x7fcb5c1662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 347962629 0 0) 0x7fcb5c0b5860 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb40003520 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 426362926 0 0) 0x7fcb40003520 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 >> v2:172.21.15.187:3300/0 conn(0x7fcb5c0cfe30 msgr2=0x7fcb5c1662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 --2- 172.21.15.67:0/2093403795 >> v2:172.21.15.187:3300/0 conn(0x7fcb5c0cfe30 0x7fcb5c1662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 >> v1:172.21.15.187:6789/0 conn(0x7fcb5c0b4e20 legacy=0x7fcb5c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c0b4620 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcb4c001670 con 0x7fcb5c0cd7c0 2024-04-17T07:34:30.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 >> v1:172.21.15.67:6789/0 conn(0x7fcb5c0cd7c0 legacy=0x7fcb5c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 0x7fcb40003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb40004800 0x7fcb40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40005210 0x7fcb4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c0b4340 con 0x7fcb40003a00 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4000c160 con 0x7fcb40004800 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.252+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4000c360 con 0x7fcb40005210 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 0x7fcb40003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 -- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40005210 msgr2=0x7fcb4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40005210 0x7fcb4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 -- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb40004800 msgr2=0x7fcb40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59400640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb40004800 0x7fcb40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb40004800 0x7fcb40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59e00640 1 -- 172.21.15.67:0/2093403795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb40003520 con 0x7fcb40003a00 2024-04-17T07:34:30.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb5a800640 1 -- 172.21.15.67:0/2093403795 reap_dead start 2024-04-17T07:34:30.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.253+0000 7fcb59400640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb40004800 0x7fcb40004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.254+0000 7fcb59e00640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 0x7fcb40003e20 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00c160 tx=0x7fcb40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d9046869ff3351 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c007c60 con 0x7fcb40003a00 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb4c030070 con 0x7fcb40003a00 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb58a00640 1 -- 172.21.15.67:0/2093403795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c005db0 con 0x7fcb40003a00 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 -- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 msgr2=0x7fcb40003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 0x7fcb40003e20 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00c160 tx=0x7fcb40005830 comp rx=0 tx=0).stop 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 -- 172.21.15.67:0/2093403795 shutdown_connections 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 --2- 172.21.15.67:0/2093403795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb40003a00 0x7fcb40003e20 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 -- 172.21.15.67:0/2093403795 >> 172.21.15.67:0/2093403795 conn(0x7fcb5c06b8c0 msgr2=0x7fcb5c06bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 -- 172.21.15.67:0/2093403795 shutdown_connections 2024-04-17T07:34:30.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.255+0000 7fcb61000640 1 -- 172.21.15.67:0/2093403795 wait complete. 2024-04-17T07:34:30.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.257+0000 7fcb61000640 1 Processor -- start 2024-04-17T07:34:30.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.257+0000 7fcb61000640 1 -- start start 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 0x7fcb5c142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb5c131640 0x7fcb5c131ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c145510 con 0x7fcb5c1431c0 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c145390 con 0x7fcb5c131640 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.258+0000 7fcb61000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5c0cf720 con 0x7fcb40003a00 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47554/0 (socket says 172.21.15.67:47554) 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 -- 172.21.15.67:0/3562229693 learned_addr learned my addr 172.21.15.67:0/3562229693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59e00640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 0x7fcb5c142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb5a800640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb5c131640 0x7fcb5c131ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 -- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 msgr2=0x7fcb5c142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 0x7fcb5c142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 -- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb5c131640 msgr2=0x7fcb5c131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb5c131640 0x7fcb5c131ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59400640 1 -- 172.21.15.67:0/3562229693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c13ed00 con 0x7fcb5c1431c0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.259+0000 7fcb59e00640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 0x7fcb5c142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb59400640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400d4d0 tx=0x7fcb4400d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44005480 con 0x7fcb5c1431c0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb44017070 con 0x7fcb5c1431c0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44003830 con 0x7fcb5c1431c0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5c143860 con 0x7fcb5c1431c0 2024-04-17T07:34:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.260+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5c143e20 con 0x7fcb5c1431c0 2024-04-17T07:34:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.261+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb5c073a50 con 0x7fcb5c1431c0 2024-04-17T07:34:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.262+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 81) ==== 90895+0+0 (secure 0 0 0) 0x7fcb440039d0 con 0x7fcb5c1431c0 2024-04-17T07:34:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.263+0000 7fcb52000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 0x7fcb3806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.263+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6081+0+0 (secure 0 0 0) 0x7fcb4408b790 con 0x7fcb5c1431c0 2024-04-17T07:34:30.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.264+0000 7fcb59e00640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 0x7fcb3806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.265+0000 7fcb59e00640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 0x7fcb3806ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb40004780 tx=0x7fcb4c039040 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.268+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcb5c13ed00 con 0x7fcb5c1431c0 2024-04-17T07:34:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.504+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 --> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) -- 0x7fcb5c068d60 con 0x7fcb3806c7c0 2024-04-17T07:34:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.681+0000 7fcb52000640 1 -- 172.21.15.67:0/3562229693 <== mgr.6737 v2:172.21.15.187:6832/2413623528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+40 (secure 0 0 0) 0x7fcb5c068d60 con 0x7fcb3806c7c0 2024-04-17T07:34:30.681 INFO:teuthology.orchestra.run.smithi067.stdout:Retention added to path /snap_test_dir1 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.683+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 msgr2=0x7fcb3806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.683+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 0x7fcb3806ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb40004780 tx=0x7fcb4c039040 comp rx=0 tx=0).stop 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 msgr2=0x7fcb5c131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400d4d0 tx=0x7fcb4400d990 comp rx=0 tx=0).stop 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 shutdown_connections 2024-04-17T07:34:30.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcb3806c7c0 0x7fcb3806ec80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb5c131640 0x7fcb5c131ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c1431c0 0x7fcb5c131100 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 --2- 172.21.15.67:0/3562229693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb40003a00 0x7fcb5c142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 >> 172.21.15.67:0/3562229693 conn(0x7fcb5c06b8c0 msgr2=0x7fcb5c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 shutdown_connections 2024-04-17T07:34:30.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.684+0000 7fcb61000640 1 -- 172.21.15.67:0/3562229693 wait complete. 2024-04-17T07:34:30.710 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-04-17T07:34:30.710 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-04-17T07:34:30.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-04-17T07:34:30.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.944+0000 7f4252400640 1 Processor -- start 2024-04-17T07:34:30.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.944+0000 7f4252400640 1 -- start start 2024-04-17T07:34:30.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 0x7f424c15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424c0af400 0x7f424c162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f424c15f780 con 0x7f424c0af400 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f424c15fa60 con 0x7f424c159020 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.945+0000 7f4252400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f424c1624a0 con 0x7f424c06b8c0 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424c0af400 0x7f424c162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424c0af400 0x7f424c162ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47566/0 (socket says 172.21.15.67:47566) 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 -- 172.21.15.67:0/549631139 learned_addr learned my addr 172.21.15.67:0/549631139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f4251000640 1 --2- 172.21.15.67:0/549631139 >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 0x7f424c15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f4251a00640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 -- 172.21.15.67:0/549631139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 msgr2=0x7f424c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 -- 172.21.15.67:0/549631139 >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 msgr2=0x7f424c15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 --2- 172.21.15.67:0/549631139 >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 0x7f424c15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f424be00640 1 -- 172.21.15.67:0/549631139 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c162780 con 0x7f424c0af400 2024-04-17T07:34:30.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.946+0000 7f4251000640 1 --2- 172.21.15.67:0/549631139 >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 0x7f424c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.947+0000 7f424be00640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c162ca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f424c0736e0 tx=0x7f423802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef21f7bb81aa3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.948+0000 7f424b400640 1 -- 172.21.15.67:0/549631139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423802e070 con 0x7f424c0af400 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.948+0000 7f424b400640 1 -- 172.21.15.67:0/549631139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238037070 con 0x7f424c0af400 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.948+0000 7f424b400640 1 -- 172.21.15.67:0/549631139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238031a00 con 0x7f424c0af400 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.948+0000 7f4252400640 1 -- 172.21.15.67:0/549631139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 msgr2=0x7f424c162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.948+0000 7f4252400640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c162ca0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f424c0736e0 tx=0x7f423802b880 comp rx=0 tx=0).stop 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 -- 172.21.15.67:0/549631139 shutdown_connections 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 --2- 172.21.15.67:0/549631139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c162ca0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 --2- 172.21.15.67:0/549631139 >> v2:172.21.15.187:3300/0 conn(0x7f424c159020 0x7f424c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 -- 172.21.15.67:0/549631139 >> 172.21.15.67:0/549631139 conn(0x7f424c0b2190 msgr2=0x7f424c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 -- 172.21.15.67:0/549631139 shutdown_connections 2024-04-17T07:34:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.949+0000 7f4252400640 1 -- 172.21.15.67:0/549631139 wait complete. 2024-04-17T07:34:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.950+0000 7f4252400640 1 Processor -- start 2024-04-17T07:34:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.951+0000 7f4252400640 1 -- start start 2024-04-17T07:34:30.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.951+0000 7f4252400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.951+0000 7f4252400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4252400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 0x7f424c159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4252400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f424c15f4f0 con 0x7f424c0af400 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4252400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f424c15f370 con 0x7f424c159020 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4252400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f424c1631e0 con 0x7f424c06b8c0 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47574/0 (socket says 172.21.15.67:47574) 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 -- 172.21.15.67:0/251068269 learned_addr learned my addr 172.21.15.67:0/251068269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4251a00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 0x7f424c159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 -- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 msgr2=0x7f424c142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4251000640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 -- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 msgr2=0x7f424c159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 0x7f424c159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f424be00640 1 -- 172.21.15.67:0/251068269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c1562f0 con 0x7f424c0af400 2024-04-17T07:34:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.952+0000 7f4251a00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 0x7f424c159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T07:34:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f424be00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f423802b8c0 tx=0x7f42380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238032670 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4238032850 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424c155770 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238032c40 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.953+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424c155d30 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.955+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424c068a10 con 0x7f424c0af400 2024-04-17T07:34:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.955+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 82) ==== 91160+0+0 (secure 0 0 0) 0x7f42380453f0 con 0x7f424c0af400 2024-04-17T07:34:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.957+0000 7f4248c00640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 0x7f422c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T07:34:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.957+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6081+0+0 (secure 0 0 0) 0x7f42380add60 con 0x7f424c0af400 2024-04-17T07:34:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.957+0000 7f4251000640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 0x7f422c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T07:34:30.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.959+0000 7f4251000640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 0x7f422c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4240006fd0 tx=0x7f4240008040 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T07:34:30.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:30.961+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4238040070 con 0x7f424c0af400 2024-04-17T07:34:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.192+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 --> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f424c143700 con 0x7f422c06ca60 2024-04-17T07:34:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.210+0000 7f4248c00640 1 -- 172.21.15.67:0/251068269 <== mgr.6737 v2:172.21.15.187:6832/2413623528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7f424c143700 con 0x7f422c06ca60 2024-04-17T07:34:31.209 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T07:34:31.210 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.211+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 msgr2=0x7f422c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.211+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 0x7f422c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4240006fd0 tx=0x7f4240008040 comp rx=0 tx=0).stop 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.211+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 msgr2=0x7f424c15bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.211+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f423802b8c0 tx=0x7f42380046c0 comp rx=0 tx=0).stop 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 shutdown_connections 2024-04-17T07:34:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f422c06ca60 0x7f422c06ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f424c159020 0x7f424c159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424c0af400 0x7f424c15bbe0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 --2- 172.21.15.67:0/251068269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f424c06b8c0 0x7f424c142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 >> 172.21.15.67:0/251068269 conn(0x7f424c0b2190 msgr2=0x7f424c0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 shutdown_connections 2024-04-17T07:34:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T07:34:31.212+0000 7f4252400640 1 -- 172.21.15.67:0/251068269 wait complete. 2024-04-17T07:34:31.232 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-04-17T07:34:31.232 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-04-17T07:34:59.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:34:59.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:35:00.705 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:35:00.705+0000 7f0f8ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:35:00.705 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:35:00.705+0000 7f0f8ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:35:00.705 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:35:00.705+0000 7f0f8ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:35:00.706 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:35:00.705+0000 7f0f8ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:35:00.706 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T07:35:00.705+0000 7f0f8ee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T07:35:29.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:35:29.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:36:00.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:36:00.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:36:30.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:36:30.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:37:00.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:37:00.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:37:30.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:37:30.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:38:00.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:38:00.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:38:30.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:38:30.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:39:01.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:39:01.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:39:31.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:39:31.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:40:01.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:40:01.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:40:31.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:40:31.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:41:01.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:41:01.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:41:31.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:41:31.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:42:02.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:42:02.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:42:32.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:42:32.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:43:02.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:43:02.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:43:32.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:43:32.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:44:02.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:44:02.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:44:32.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:44:32.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:45:03.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:45:03.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:45:33.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:45:33.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:46:03.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:46:03.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:46:33.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:46:33.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:47:03.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:47:03.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:47:33.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:47:33.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:48:03.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:48:03.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:48:34.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:48:34.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:49:04.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:49:04.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:49:34.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:49:34.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:50:04.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:50:04.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:50:34.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:50:34.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:51:04.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:51:04.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:51:35.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:51:35.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:52:05.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:52:05.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:52:35.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:52:35.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:53:05.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:53:05.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:53:35.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:53:35.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:54:05.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:54:05.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:54:35.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:54:35.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:55:06.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:55:06.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:55:36.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:55:36.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:56:06.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:56:06.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:56:36.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:56:36.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:57:06.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:57:06.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:57:36.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:57:36.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:58:06.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:58:06.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:58:37.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:58:37.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:59:07.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:59:07.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:59:37.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T07:59:37.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:00:07.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:00:07.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:00:37.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:00:37.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:01:07.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:01:07.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:01:38.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:01:38.053 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:02:08.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:02:08.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:02:38.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:02:38.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:03:08.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:03:08.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:03:38.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:03:38.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:04:08.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:04:08.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:04:39.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:04:39.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:05:09.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:05:09.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:05:39.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:05:39.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:06:09.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:06:09.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:06:39.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:06:39.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:07:09.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:07:09.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:07:40.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:07:40.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:08:10.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:08:10.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:08:40.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:08:40.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:09:10.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:09:10.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:09:40.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:09:40.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:10:10.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:10:10.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:10:41.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:10:41.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:11:11.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:11:11.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:11:41.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:11:41.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:12:11.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:12:11.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:12:41.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:12:41.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:13:12.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:13:12.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:13:42.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:13:42.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:14:12.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:14:12.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:14:42.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:14:42.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:15:12.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:15:12.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:15:43.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:15:43.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:16:13.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:16:13.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:16:43.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:16:43.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:17:13.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:17:13.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:17:43.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:17:43.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:18:13.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:18:14.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:18:44.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:18:44.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:19:14.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:19:14.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:19:44.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:19:44.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:20:14.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:20:14.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:20:44.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:20:44.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:21:15.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:21:15.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:21:45.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:21:45.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:22:15.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:22:15.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:22:45.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:22:45.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:23:15.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:23:15.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:23:46.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:23:46.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:24:16.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:24:16.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:24:46.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:24:46.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:25:16.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:25:16.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:25:46.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:25:46.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:26:17.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:26:17.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:26:47.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:26:47.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:27:17.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:27:17.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:27:47.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:27:47.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:28:17.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:28:17.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:28:48.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:28:48.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:29:18.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:29:18.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:29:48.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:29:48.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:30:16.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:16.234 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:30:16.305 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_36_00_UTC 2024-04-17T08:30:16.305 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_37_00_UTC 2024-04-17T08:30:16.305 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_38_00_UTC 2024-04-17T08:30:16.305 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_39_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_40_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_41_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_42_01_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_43_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_44_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_45_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_46_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_47_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_48_00_UTC 2024-04-17T08:30:16.306 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_49_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_50_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_51_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_52_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_53_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_54_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_55_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_56_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_57_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_58_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_59_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_00_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_01_00_UTC 2024-04-17T08:30:16.307 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_02_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_03_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_04_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_05_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_06_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_07_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_08_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_09_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_10_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_11_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_12_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_13_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_14_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_15_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_16_00_UTC 2024-04-17T08:30:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_17_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_18_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_19_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_20_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_21_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_22_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_23_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_24_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_25_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_26_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_27_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_28_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_29_00_UTC 2024-04-17T08:30:16.309 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_30_00_UTC 2024-04-17T08:30:16.311 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-07_36_00_UTC', 'scheduled-2024-04-17-07_37_00_UTC', 'scheduled-2024-04-17-07_38_00_UTC', 'scheduled-2024-04-17-07_39_00_UTC', 'scheduled-2024-04-17-07_40_00_UTC', 'scheduled-2024-04-17-07_41_00_UTC', 'scheduled-2024-04-17-07_42_01_UTC', 'scheduled-2024-04-17-07_43_00_UTC', 'scheduled-2024-04-17-07_44_00_UTC', 'scheduled-2024-04-17-07_45_00_UTC', 'scheduled-2024-04-17-07_46_00_UTC', 'scheduled-2024-04-17-07_47_00_UTC', 'scheduled-2024-04-17-07_48_00_UTC', 'scheduled-2024-04-17-07_49_00_UTC', 'scheduled-2024-04-17-07_50_00_UTC', 'scheduled-2024-04-17-07_51_00_UTC', 'scheduled-2024-04-17-07_52_00_UTC', 'scheduled-2024-04-17-07_53_00_UTC', 'scheduled-2024-04-17-07_54_00_UTC', 'scheduled-2024-04-17-07_55_00_UTC', 'scheduled-2024-04-17-07_56_00_UTC', 'scheduled-2024-04-17-07_57_00_UTC', 'scheduled-2024-04-17-07_58_00_UTC', 'scheduled-2024-04-17-07_59_00_UTC', 'scheduled-2024-04-17-08_00_00_UTC', 'scheduled-2024-04-17-08_01_00_UTC', 'scheduled-2024-04-17-08_02_00_UTC', 'scheduled-2024-04-17-08_03_00_UTC', 'scheduled-2024-04-17-08_04_00_UTC', 'scheduled-2024-04-17-08_05_00_UTC', 'scheduled-2024-04-17-08_06_00_UTC', 'scheduled-2024-04-17-08_07_00_UTC', 'scheduled-2024-04-17-08_08_00_UTC', 'scheduled-2024-04-17-08_09_00_UTC', 'scheduled-2024-04-17-08_10_00_UTC', 'scheduled-2024-04-17-08_11_00_UTC', 'scheduled-2024-04-17-08_12_00_UTC', 'scheduled-2024-04-17-08_13_00_UTC', 'scheduled-2024-04-17-08_14_00_UTC', 'scheduled-2024-04-17-08_15_00_UTC', 'scheduled-2024-04-17-08_16_00_UTC', 'scheduled-2024-04-17-08_17_00_UTC', 'scheduled-2024-04-17-08_18_00_UTC', 'scheduled-2024-04-17-08_19_00_UTC', 'scheduled-2024-04-17-08_20_00_UTC', 'scheduled-2024-04-17-08_21_00_UTC', 'scheduled-2024-04-17-08_22_00_UTC', 'scheduled-2024-04-17-08_23_00_UTC', 'scheduled-2024-04-17-08_24_00_UTC', 'scheduled-2024-04-17-08_25_00_UTC', 'scheduled-2024-04-17-08_26_00_UTC', 'scheduled-2024-04-17-08_27_00_UTC', 'scheduled-2024-04-17-08_28_00_UTC', 'scheduled-2024-04-17-08_29_00_UTC', 'scheduled-2024-04-17-08_30_00_UTC'] 2024-04-17T08:30:16.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-04-17T08:30:16.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.531+0000 7f31a5400640 1 Processor -- start 2024-04-17T08:30:16.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.531+0000 7f31a5400640 1 -- start start 2024-04-17T08:30:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.533+0000 7f31a5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31a00cd920 0x7f31a006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.533+0000 7f31a5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a006c0a0 0x7f31a006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.533+0000 7f31a5400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a0146360 con 0x7f31a0145f80 2024-04-17T08:30:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.533+0000 7f31a5400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f31a00727a0 con 0x7f31a00cd920 2024-04-17T08:30:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.533+0000 7f31a5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0071e70 con 0x7f31a006c0a0 2024-04-17T08:30:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31a00cd920 0x7f31a006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31a00cd920 0x7f31a006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52786/0 (socket says 172.21.15.67:52786) 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 -- 172.21.15.67:0/408098927 learned_addr learned my addr 172.21.15.67:0/408098927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f31a4a00640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a006c0a0 0x7f31a006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 -- 172.21.15.67:0/408098927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a006c0a0 msgr2=0x7f31a006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a006c0a0 0x7f31a006e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 -- 172.21.15.67:0/408098927 >> v1:172.21.15.187:6789/0 conn(0x7f31a0145f80 legacy=0x7f31a00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.534+0000 7f319f400640 1 -- 172.21.15.67:0/408098927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a0072150 con 0x7f31a00cd920 2024-04-17T08:30:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.535+0000 7f319f400640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a00cd920 0x7f31a006bb60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f318c009870 tx=0x7f318c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b09e8f4e34b3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.536+0000 7f319ea00640 1 -- 172.21.15.67:0/408098927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c0317c0 con 0x7f31a00cd920 2024-04-17T08:30:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.536+0000 7f319ea00640 1 -- 172.21.15.67:0/408098927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318c037070 con 0x7f31a00cd920 2024-04-17T08:30:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.536+0000 7f319ea00640 1 -- 172.21.15.67:0/408098927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c031b80 con 0x7f31a00cd920 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 -- 172.21.15.67:0/408098927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a00cd920 msgr2=0x7f31a006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a00cd920 0x7f31a006bb60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f318c009870 tx=0x7f318c02b860 comp rx=0 tx=0).stop 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 -- 172.21.15.67:0/408098927 shutdown_connections 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a006c0a0 0x7f31a006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 --2- 172.21.15.67:0/408098927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a00cd920 0x7f31a006bb60 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 -- 172.21.15.67:0/408098927 >> 172.21.15.67:0/408098927 conn(0x7f31a00b2190 msgr2=0x7f31a00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 -- 172.21.15.67:0/408098927 shutdown_connections 2024-04-17T08:30:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.537+0000 7f31a5400640 1 -- 172.21.15.67:0/408098927 wait complete. 2024-04-17T08:30:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.540+0000 7f31a5400640 1 Processor -- start 2024-04-17T08:30:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.540+0000 7f31a5400640 1 -- start start 2024-04-17T08:30:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 0x7f31a015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a00cd920 0x7f31a01e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0072c30 con 0x7f31a0145f80 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f31a006c530 con 0x7f31a006c0a0 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a5400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0072db0 con 0x7f31a00cd920 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.541+0000 7f31a4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52790/0 (socket says 172.21.15.67:52790) 2024-04-17T08:30:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 -- 172.21.15.67:0/3103663057 learned_addr learned my addr 172.21.15.67:0/3103663057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f319fe00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 0x7f31a015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f319f400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a00cd920 0x7f31a01e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 -- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a00cd920 msgr2=0x7f31a01e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a00cd920 0x7f31a01e5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 -- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 msgr2=0x7f31a015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 0x7f31a015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 -- 172.21.15.67:0/3103663057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a015c570 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f319fe00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 0x7f31a015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.542+0000 7f31a4a00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f319800cd70 tx=0x7f319800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.543+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198003b90 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.543+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3198003d30 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.543+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198004240 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.543+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a00aa1b0 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.543+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a0131db0 con 0x7f31a0145f80 2024-04-17T08:30:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.544+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a0068a10 con 0x7f31a0145f80 2024-04-17T08:30:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.545+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 85) ==== 91171+0+0 (secure 0 0 0) 0x7f31980043e0 con 0x7f31a0145f80 2024-04-17T08:30:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.546+0000 7f3197e00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 0x7f318006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.546+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 6183+0+0 (secure 0 0 0) 0x7f319808cbd0 con 0x7f31a0145f80 2024-04-17T08:30:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.547+0000 7f319fe00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 0x7f318006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.548+0000 7f319fe00640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 0x7f318006ef20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f319000a830 tx=0x7f3190008040 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.551+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f319805ab60 con 0x7f31a0145f80 2024-04-17T08:30:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.784+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 --> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f31a00cec50 con 0x7f318006ca60 2024-04-17T08:30:16.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.803+0000 7f3197e00640 1 -- 172.21.15.67:0/3103663057 <== mgr.6737 v2:172.21.15.187:6832/2413623528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+373 (secure 0 0 0) 0x7f31a00cec50 con 0x7f318006ca60 2024-04-17T08:30:16.803 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:16.803 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:34:29", "first": "2024-04-17T07:35:00", "last": "2024-04-17T08:30:00", "last_pruned": "2024-04-17T08:30:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 msgr2=0x7f318006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 0x7f318006ef20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f319000a830 tx=0x7f3190008040 comp rx=0 tx=0).stop 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 msgr2=0x7f31a01e61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f319800cd70 tx=0x7f319800a660 comp rx=0 tx=0).stop 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 shutdown_connections 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f318006ca60 0x7f318006ef20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a0145f80 0x7f31a01e61e0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31a00cd920 0x7f31a01e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 --2- 172.21.15.67:0/3103663057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31a006c0a0 0x7f31a015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:16.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.806+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 >> 172.21.15.67:0/3103663057 conn(0x7f31a00b2190 msgr2=0x7f31a006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:16.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.807+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 shutdown_connections 2024-04-17T08:30:16.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:16.807+0000 7f31a5400640 1 -- 172.21.15.67:0/3103663057 wait complete. 2024-04-17T08:30:16.825 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T07:34:29", "first": "2024-04-17T07:35:00", "last": "2024-04-17T08:30:00", "last_pruned": "2024-04-17T08:30:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-04-17T08:30:16.825 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T07:34:29', 'first': '2024-04-17T07:35:00', 'last': '2024-04-17T08:30:00', 'last_pruned': '2024-04-17T08:30:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-04-17T08:30:16.825 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-04-17T08:30:16.826 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-04-17T08:30:16.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-04-17T08:30:17.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.043+0000 7f1563200640 1 Processor -- start 2024-04-17T08:30:17.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.043+0000 7f1563200640 1 -- start start 2024-04-17T08:30:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1563200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 0x7f15640cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1563200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1564146360 con 0x7f156406c270 2024-04-17T08:30:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1563200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1564071e70 con 0x7f15640cd750 2024-04-17T08:30:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1563200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15640cdbb0 con 0x7f1564145f80 2024-04-17T08:30:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1562800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f156406c270 0x7f156406e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45146/0 (socket says 172.21.15.67:45146) 2024-04-17T08:30:17.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.045+0000 7f1562800640 1 -- 172.21.15.67:0/2083468687 learned_addr learned my addr 172.21.15.67:0/2083468687 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:17.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1561e00640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 0x7f15640cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791953463 0 0) 0x7f1564146360 con 0x7f156406c270 2024-04-17T08:30:17.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1548003520 con 0x7f156406c270 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129190780 0 0) 0x7f1548003520 con 0x7f156406c270 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 msgr2=0x7f15640cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 0x7f15640cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 >> v1:172.21.15.187:6789/0 conn(0x7f15640cd750 legacy=0x7f156406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1564072150 con 0x7f156406c270 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f155c0019f0 con 0x7f156406c270 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 >> v1:172.21.15.67:6789/0 conn(0x7f156406c270 legacy=0x7f156406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 0x7f15480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1548003f60 0x7f1548004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004d40 0x7f15480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f154800bb50 con 0x7f1548003f60 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f154800bcd0 con 0x7f1548004d40 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.046+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f154800bed0 con 0x7f1564145f80 2024-04-17T08:30:17.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1548003f60 0x7f1548004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 -- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 msgr2=0x7f15480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:17.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1564145f80 0x7f15480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 -- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004d40 msgr2=0x7f15480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:17.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548004d40 0x7f15480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.047+0000 7f1561400640 1 -- 172.21.15.67:0/2083468687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1548003520 con 0x7f1548003f60 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.048+0000 7f1561400640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1548003f60 0x7f1548004800 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1550001750 tx=0x7f1548005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18dbc8e1a78927c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.048+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550001370 con 0x7f1548003f60 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.048+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15500052d0 con 0x7f1548003f60 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.048+0000 7f1560a00640 1 -- 172.21.15.67:0/2083468687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550003830 con 0x7f1548003f60 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 -- 172.21.15.67:0/2083468687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1548003f60 msgr2=0x7f1548004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 --2- 172.21.15.67:0/2083468687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1548003f60 0x7f1548004800 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1550001750 tx=0x7f1548005830 comp rx=0 tx=0).stop 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1562800640 1 -- 172.21.15.67:0/2083468687 reap_dead start 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 -- 172.21.15.67:0/2083468687 shutdown_connections 2024-04-17T08:30:17.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 -- 172.21.15.67:0/2083468687 >> 172.21.15.67:0/2083468687 conn(0x7f15640b2190 msgr2=0x7f15640b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:17.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 -- 172.21.15.67:0/2083468687 shutdown_connections 2024-04-17T08:30:17.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.049+0000 7f1563200640 1 -- 172.21.15.67:0/2083468687 wait complete. 2024-04-17T08:30:17.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.051+0000 7f1563200640 1 Processor -- start 2024-04-17T08:30:17.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.051+0000 7f1563200640 1 -- start start 2024-04-17T08:30:17.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548003f60 0x7f156415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156414f3f0 0x7f156414f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1564072c30 con 0x7f156413b370 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15640727a0 con 0x7f1548003f60 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.052+0000 7f1563200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1564072db0 con 0x7f156414f3f0 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52812/0 (socket says 172.21.15.67:52812) 2024-04-17T08:30:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 -- 172.21.15.67:0/4177764792 learned_addr learned my addr 172.21.15.67:0/4177764792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561e00640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548003f60 0x7f156415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1562800640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156414f3f0 0x7f156414f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 -- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156414f3f0 msgr2=0x7f156414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156414f3f0 0x7f156414f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 -- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548003f60 msgr2=0x7f156415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548003f60 0x7f156415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.053+0000 7f1561400640 1 -- 172.21.15.67:0/4177764792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156413be40 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f1561400640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1550008fd0 tx=0x7f1550001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550041070 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1550005480 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550001370 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156413c120 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.054+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156413c6e0 con 0x7f156413b370 2024-04-17T08:30:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.056+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1564068a10 con 0x7f156413b370 2024-04-17T08:30:17.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.056+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 85) ==== 91171+0+0 (secure 0 0 0) 0x7f1550044070 con 0x7f156413b370 2024-04-17T08:30:17.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.057+0000 7f155a000640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 0x7f154406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:17.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.058+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 6183+0+0 (secure 0 0 0) 0x7f15500adaf0 con 0x7f156413b370 2024-04-17T08:30:17.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.058+0000 7f1561e00640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 0x7f154406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:17.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.059+0000 7f1561e00640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 0x7f154406ef50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1554009a50 tx=0x7f1554008040 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.062+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f155007bb20 con 0x7f156413b370 2024-04-17T08:30:17.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.296+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 --> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f156415c7d0 con 0x7f154406ca90 2024-04-17T08:30:17.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.517+0000 7f155a000640 1 -- 172.21.15.67:0/4177764792 <== mgr.6737 v2:172.21.15.187:6832/2413623528 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+42 (secure 0 0 0) 0x7f156415c7d0 con 0x7f154406ca90 2024-04-17T08:30:17.517 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 msgr2=0x7f154406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 0x7f154406ef50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1554009a50 tx=0x7f1554008040 comp rx=0 tx=0).stop 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 msgr2=0x7f156413b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1550008fd0 tx=0x7f1550001120 comp rx=0 tx=0).stop 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 shutdown_connections 2024-04-17T08:30:17.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f154406ca90 0x7f154406ef50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f156414f3f0 0x7f156414f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f156413b370 0x7f156413b7d0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 --2- 172.21.15.67:0/4177764792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1548003f60 0x7f156415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 >> 172.21.15.67:0/4177764792 conn(0x7f15640b2190 msgr2=0x7f156406e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 shutdown_connections 2024-04-17T08:30:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:17.521+0000 7f1563200640 1 -- 172.21.15.67:0/4177764792 wait complete. 2024-04-17T08:30:17.544 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-04-17T08:30:17.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.545 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:30:17.605 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_36_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_37_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_38_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_39_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_40_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_41_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_42_01_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_43_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_44_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_45_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_46_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_47_00_UTC 2024-04-17T08:30:17.606 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_48_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_49_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_50_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_51_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_52_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_53_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_54_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_55_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_56_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_57_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_58_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-07_59_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_00_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_01_00_UTC 2024-04-17T08:30:17.607 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_02_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_03_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_04_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_05_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_06_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_07_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_08_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_09_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_10_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_11_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_12_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_13_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_14_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_15_00_UTC 2024-04-17T08:30:17.608 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_16_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_17_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_18_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_19_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_20_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_21_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_22_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_23_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_24_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_25_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_26_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_27_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_28_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_29_00_UTC 2024-04-17T08:30:17.609 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_30_00_UTC 2024-04-17T08:30:17.610 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_36_00_UTC 2024-04-17T08:30:17.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.611 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_36_00_UTC) 2024-04-17T08:30:17.681 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_37_00_UTC 2024-04-17T08:30:17.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.681 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_37_00_UTC) 2024-04-17T08:30:17.752 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_38_00_UTC 2024-04-17T08:30:17.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.752 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_38_00_UTC) 2024-04-17T08:30:17.825 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_39_00_UTC 2024-04-17T08:30:17.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.826 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_39_00_UTC) 2024-04-17T08:30:17.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_40_00_UTC 2024-04-17T08:30:17.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.899 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_40_00_UTC) 2024-04-17T08:30:17.974 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_41_00_UTC 2024-04-17T08:30:17.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:17.974 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_41_00_UTC) 2024-04-17T08:30:18.048 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_42_01_UTC 2024-04-17T08:30:18.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.049 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_42_01_UTC) 2024-04-17T08:30:18.117 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_43_00_UTC 2024-04-17T08:30:18.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.118 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_43_00_UTC) 2024-04-17T08:30:18.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_44_00_UTC 2024-04-17T08:30:18.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.188 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_44_00_UTC) 2024-04-17T08:30:18.263 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_45_00_UTC 2024-04-17T08:30:18.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.263 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_45_00_UTC) 2024-04-17T08:30:18.337 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_46_00_UTC 2024-04-17T08:30:18.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.337 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_46_00_UTC) 2024-04-17T08:30:18.413 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_47_00_UTC 2024-04-17T08:30:18.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.413 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_47_00_UTC) 2024-04-17T08:30:18.495 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_48_00_UTC 2024-04-17T08:30:18.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.496 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_48_00_UTC) 2024-04-17T08:30:18.555 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_49_00_UTC 2024-04-17T08:30:18.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.555 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_49_00_UTC) 2024-04-17T08:30:18.623 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_50_00_UTC 2024-04-17T08:30:18.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.623 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_50_00_UTC) 2024-04-17T08:30:18.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:30:18.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:30:18.678 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_51_00_UTC 2024-04-17T08:30:18.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.678 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_51_00_UTC) 2024-04-17T08:30:18.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_52_00_UTC 2024-04-17T08:30:18.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.771 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_52_00_UTC) 2024-04-17T08:30:18.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_53_00_UTC 2024-04-17T08:30:18.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.830 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_53_00_UTC) 2024-04-17T08:30:18.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_54_00_UTC 2024-04-17T08:30:18.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.901 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_54_00_UTC) 2024-04-17T08:30:18.976 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_55_00_UTC 2024-04-17T08:30:18.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:18.977 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_55_00_UTC) 2024-04-17T08:30:19.050 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_56_00_UTC 2024-04-17T08:30:19.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.050 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_56_00_UTC) 2024-04-17T08:30:19.133 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_57_00_UTC 2024-04-17T08:30:19.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.133 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_57_00_UTC) 2024-04-17T08:30:19.187 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_58_00_UTC 2024-04-17T08:30:19.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.187 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_58_00_UTC) 2024-04-17T08:30:19.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-07_59_00_UTC 2024-04-17T08:30:19.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.244 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-07_59_00_UTC) 2024-04-17T08:30:19.308 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_00_00_UTC 2024-04-17T08:30:19.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.308 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_00_00_UTC) 2024-04-17T08:30:19.379 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_01_00_UTC 2024-04-17T08:30:19.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.380 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_01_00_UTC) 2024-04-17T08:30:19.478 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_02_00_UTC 2024-04-17T08:30:19.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.478 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_02_00_UTC) 2024-04-17T08:30:19.543 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_03_00_UTC 2024-04-17T08:30:19.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.543 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_03_00_UTC) 2024-04-17T08:30:19.618 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_04_00_UTC 2024-04-17T08:30:19.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.618 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_04_00_UTC) 2024-04-17T08:30:19.682 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_05_00_UTC 2024-04-17T08:30:19.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.683 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_05_00_UTC) 2024-04-17T08:30:19.742 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_06_00_UTC 2024-04-17T08:30:19.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.742 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_06_00_UTC) 2024-04-17T08:30:19.808 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_07_00_UTC 2024-04-17T08:30:19.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.808 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_07_00_UTC) 2024-04-17T08:30:19.882 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_08_00_UTC 2024-04-17T08:30:19.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.883 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_08_00_UTC) 2024-04-17T08:30:19.954 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_09_00_UTC 2024-04-17T08:30:19.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:19.954 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_09_00_UTC) 2024-04-17T08:30:20.033 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_10_00_UTC 2024-04-17T08:30:20.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.034 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_10_00_UTC) 2024-04-17T08:30:20.096 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_11_00_UTC 2024-04-17T08:30:20.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.097 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_11_00_UTC) 2024-04-17T08:30:20.160 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_12_00_UTC 2024-04-17T08:30:20.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.160 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_12_00_UTC) 2024-04-17T08:30:20.230 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_13_00_UTC 2024-04-17T08:30:20.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.230 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_13_00_UTC) 2024-04-17T08:30:20.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_14_00_UTC 2024-04-17T08:30:20.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.299 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_14_00_UTC) 2024-04-17T08:30:20.372 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_15_00_UTC 2024-04-17T08:30:20.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.372 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_15_00_UTC) 2024-04-17T08:30:20.462 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_16_00_UTC 2024-04-17T08:30:20.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.462 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_16_00_UTC) 2024-04-17T08:30:20.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_17_00_UTC 2024-04-17T08:30:20.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.541 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_17_00_UTC) 2024-04-17T08:30:20.585 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_18_00_UTC 2024-04-17T08:30:20.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.585 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_18_00_UTC) 2024-04-17T08:30:20.697 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_19_00_UTC 2024-04-17T08:30:20.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.698 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_19_00_UTC) 2024-04-17T08:30:20.741 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_20_00_UTC 2024-04-17T08:30:20.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.741 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_20_00_UTC) 2024-04-17T08:30:20.835 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_21_00_UTC 2024-04-17T08:30:20.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.835 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_21_00_UTC) 2024-04-17T08:30:20.889 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_22_00_UTC 2024-04-17T08:30:20.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.889 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_22_00_UTC) 2024-04-17T08:30:20.949 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_23_00_UTC 2024-04-17T08:30:20.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:20.949 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_23_00_UTC) 2024-04-17T08:30:21.010 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_24_00_UTC 2024-04-17T08:30:21.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.010 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_24_00_UTC) 2024-04-17T08:30:21.073 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_25_00_UTC 2024-04-17T08:30:21.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.074 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_25_00_UTC) 2024-04-17T08:30:21.140 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_26_00_UTC 2024-04-17T08:30:21.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.141 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_26_00_UTC) 2024-04-17T08:30:21.208 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_27_00_UTC 2024-04-17T08:30:21.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.208 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_27_00_UTC) 2024-04-17T08:30:21.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_28_00_UTC 2024-04-17T08:30:21.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.275 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_28_00_UTC) 2024-04-17T08:30:21.343 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_29_00_UTC 2024-04-17T08:30:21.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.343 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_29_00_UTC) 2024-04-17T08:30:21.411 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_30_00_UTC 2024-04-17T08:30:21.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.411 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_30_00_UTC) 2024-04-17T08:30:21.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:21.495 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T08:30:21.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:30:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.795+0000 7f032ee00640 1 Processor -- start 2024-04-17T08:30:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.795+0000 7f032ee00640 1 -- start start 2024-04-17T08:30:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.796+0000 7f032ee00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03280d0b30 0x7f03280b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.796+0000 7f032ee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0328071f60 con 0x7f032815b860 2024-04-17T08:30:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.796+0000 7f032ee00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0328072240 con 0x7f03280ce4d0 2024-04-17T08:30:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.796+0000 7f032ee00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0328071d10 con 0x7f03280d0b30 2024-04-17T08:30:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.796+0000 7f0327e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f032815b860 0x7f03280d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45158/0 (socket says 172.21.15.67:45158) 2024-04-17T08:30:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.797+0000 7f0327e00640 1 -- 172.21.15.67:0/1878946709 learned_addr learned my addr 172.21.15.67:0/1878946709 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.797+0000 7f031fe00640 1 --2- 172.21.15.67:0/1878946709 >> v2:172.21.15.187:3300/0 conn(0x7f03280d0b30 0x7f03280b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.797+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137835304 0 0) 0x7f0328071f60 con 0x7f032815b860 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.797+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030c003520 con 0x7f032815b860 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1575161629 0 0) 0x7f030c003520 con 0x7f032815b860 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 >> v2:172.21.15.187:3300/0 conn(0x7f03280d0b30 msgr2=0x7f03280b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 --2- 172.21.15.67:0/1878946709 >> v2:172.21.15.187:3300/0 conn(0x7f03280d0b30 0x7f03280b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 >> v1:172.21.15.187:6789/0 conn(0x7f03280ce4d0 legacy=0x7f03280b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328072520 con 0x7f032815b860 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0318001670 con 0x7f032815b860 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 >> v1:172.21.15.67:6789/0 conn(0x7f032815b860 legacy=0x7f03280d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f030c003a00 0x7f030c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f030c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f030c0051c0 0x7f030c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0328071d10 con 0x7f030c004800 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f030c00c0c0 con 0x7f030c003a00 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f030c00c2c0 con 0x7f030c0051c0 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f030c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f0327e00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f030c003a00 0x7f030c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 -- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f030c0051c0 msgr2=0x7f030c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f030c0051c0 0x7f030c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 -- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f030c003a00 msgr2=0x7f030c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f030c003a00 0x7f030c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.798+0000 7f031fe00640 1 -- 172.21.15.67:0/1878946709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c003520 con 0x7f030c004800 2024-04-17T08:30:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.799+0000 7f032ce00640 1 -- 172.21.15.67:0/1878946709 reap_dead start 2024-04-17T08:30:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f031fe00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f030c004c80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0310009ac0 tx=0x7f030c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adb06c8b7c9e2d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310039070 con 0x7f030c004800 2024-04-17T08:30:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031002c6b0 con 0x7f030c004800 2024-04-17T08:30:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f0327400640 1 -- 172.21.15.67:0/1878946709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310009690 con 0x7f030c004800 2024-04-17T08:30:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f032ee00640 1 -- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 msgr2=0x7f030c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f032ee00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f030c004c80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0310009ac0 tx=0x7f030c005830 comp rx=0 tx=0).stop 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f032ee00640 1 -- 172.21.15.67:0/1878946709 shutdown_connections 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f032ee00640 1 --2- 172.21.15.67:0/1878946709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f030c004c80 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.800+0000 7f032ee00640 1 -- 172.21.15.67:0/1878946709 >> 172.21.15.67:0/1878946709 conn(0x7f0328067ee0 msgr2=0x7f032806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.801+0000 7f032ee00640 1 -- 172.21.15.67:0/1878946709 shutdown_connections 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.801+0000 7f032ee00640 1 -- 172.21.15.67:0/1878946709 wait complete. 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.802+0000 7f032ee00640 1 Processor -- start 2024-04-17T08:30:21.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.802+0000 7f032ee00640 1 -- start start 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 0x7f032814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 0x7f03280aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f032815bc40 con 0x7f030c004800 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0328072ae0 con 0x7f03280ae740 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.803+0000 7f032ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f032806f380 con 0x7f0328150e60 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52826/0 (socket says 172.21.15.67:52826) 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 -- 172.21.15.67:0/4144711508 learned_addr learned my addr 172.21.15.67:0/4144711508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f031fe00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 0x7f032814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f032ce00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 0x7f03280aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 -- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 msgr2=0x7f032814e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 0x7f032814e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 -- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 msgr2=0x7f03280aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 0x7f03280aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 -- 172.21.15.67:0/4144711508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328151320 con 0x7f030c004800 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f032ce00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 0x7f03280aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:21.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f031fe00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 0x7f032814e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.804+0000 7f0327e00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f031800e790 tx=0x7f031800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.805+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031800eea0 con 0x7f030c004800 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.805+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0318018070 con 0x7f030c004800 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.805+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031800cb10 con 0x7f030c004800 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.805+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032814ee40 con 0x7f030c004800 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.805+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032814f400 con 0x7f030c004800 2024-04-17T08:30:21.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.806+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0328068ca0 con 0x7f030c004800 2024-04-17T08:30:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.807+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 85) ==== 91171+0+0 (secure 0 0 0) 0x7f031800ccb0 con 0x7f030c004800 2024-04-17T08:30:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.808+0000 7f0324c00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 0x7f030406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.808+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 6227+0+0 (secure 0 0 0) 0x7f031808bb50 con 0x7f030c004800 2024-04-17T08:30:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.809+0000 7f031fe00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 0x7f030406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:21.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.810+0000 7f031fe00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 0x7f030406ef50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0310036000 tx=0x7f0310030000 comp rx=0 tx=0).ready entity=mgr.6737 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:21.813+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0318059c60 con 0x7f030c004800 2024-04-17T08:30:22.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.070+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f032815a910 con 0x7f030c004800 2024-04-17T08:30:22.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.458+0000 7f0324c00640 1 -- 172.21.15.67:0/4144711508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) ==== 93+0+0 (secure 0 0 0) 0x7f0318059560 con 0x7f030c004800 2024-04-17T08:30:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.464+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 msgr2=0x7f030406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.464+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 0x7f030406ef50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0310036000 tx=0x7f0310030000 comp rx=0 tx=0).stop 2024-04-17T08:30:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.464+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 msgr2=0x7f0328142460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.464+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f031800e790 tx=0x7f031800ec50 comp rx=0 tx=0).stop 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 shutdown_connections 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f030406ca90 0x7f030406ef50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03280ae740 0x7f03280aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0328150e60 0x7f032814e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 --2- 172.21.15.67:0/4144711508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f030c004800 0x7f0328142460 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.467+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 >> 172.21.15.67:0/4144711508 conn(0x7f0328067ee0 msgr2=0x7f03280b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.468+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 shutdown_connections 2024-04-17T08:30:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:22.469+0000 7f032ee00640 1 -- 172.21.15.67:0/4144711508 wait complete. 2024-04-17T08:30:22.499 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:30:22.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:30:22.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:30:22.724 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:30:22.724 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:30:22.724 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:30:22.724 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:22.792 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:30:22.792 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:22.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.062+0000 7fc3af200640 1 Processor -- start 2024-04-17T08:30:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.062+0000 7fc3af200640 1 -- start start 2024-04-17T08:30:23.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc3b006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3af200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b00d1310 con 0x7fc3b00d06b0 2024-04-17T08:30:23.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3af200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b006f060 con 0x7fc3b0161ec0 2024-04-17T08:30:23.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b00d0d50 con 0x7fc3b006b8c0 2024-04-17T08:30:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3ad400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3b00d06b0 0x7fc3b00ae330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45174/0 (socket says 172.21.15.67:45174) 2024-04-17T08:30:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3ad400640 1 -- 172.21.15.67:0/1370314555 learned_addr learned my addr 172.21.15.67:0/1370314555 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.064+0000 7fc3ae800640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc3b006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151333394 0 0) 0x7fc3b00d1310 con 0x7fc3b00d06b0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc394003520 con 0x7fc3b00d06b0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 156713329 0 0) 0x7fc394003520 con 0x7fc3b00d06b0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 msgr2=0x7fc3b006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc3b006bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 >> v1:172.21.15.187:6789/0 conn(0x7fc3b0161ec0 legacy=0x7fc3b00b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b00d1030 con 0x7fc3b00d06b0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc39c001670 con 0x7fc3b00d06b0 2024-04-17T08:30:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 >> v1:172.21.15.67:6789/0 conn(0x7fc3b00d06b0 legacy=0x7fc3b00ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc394003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc394004800 0x7fc394004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc39400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b00d0d50 con 0x7fc394003a50 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc394005520 con 0x7fc394004800 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc39400bf00 con 0x7fc3b006b8c0 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.065+0000 7fc3ade00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc394003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ae800640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc39400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ad400640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc394004800 0x7fc394004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ade00640 1 -- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 msgr2=0x7fc39400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ade00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b006b8c0 0x7fc39400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ade00640 1 -- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc394004800 msgr2=0x7fc394004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ade00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc394004800 0x7fc394004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ade00640 1 -- 172.21.15.67:0/1370314555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc394003520 con 0x7fc394003a50 2024-04-17T08:30:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.066+0000 7fc3ad400640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc394004800 0x7fc394004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.067+0000 7fc3ade00640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc394003e70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0001750 tx=0x7fc394005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927c87251129617f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.067+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0001370 con 0x7fc394003a50 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.067+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a00052d0 con 0x7fc394003a50 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.067+0000 7fc3aca00640 1 -- 172.21.15.67:0/1370314555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0003830 con 0x7fc394003a50 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 -- 172.21.15.67:0/1370314555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 msgr2=0x7fc394003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 --2- 172.21.15.67:0/1370314555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc394003e70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0001750 tx=0x7fc394005830 comp rx=0 tx=0).stop 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3ae800640 1 -- 172.21.15.67:0/1370314555 reap_dead start 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 -- 172.21.15.67:0/1370314555 shutdown_connections 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 -- 172.21.15.67:0/1370314555 >> 172.21.15.67:0/1370314555 conn(0x7fc3b009aa60 msgr2=0x7fc3b00b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 -- 172.21.15.67:0/1370314555 shutdown_connections 2024-04-17T08:30:23.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.068+0000 7fc3af200640 1 -- 172.21.15.67:0/1370314555 wait complete. 2024-04-17T08:30:23.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.071+0000 7fc3af200640 1 Processor -- start 2024-04-17T08:30:23.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.071+0000 7fc3af200640 1 -- start start 2024-04-17T08:30:23.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3b0150c40 0x7fc3b00c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b009e640 0x7fc3b00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b006bd50 con 0x7fc394003a50 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b01622a0 con 0x7fc3b0150c40 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3af200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b006bed0 con 0x7fc3b009e640 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52856/0 (socket says 172.21.15.67:52856) 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 -- 172.21.15.67:0/35324805 learned_addr learned my addr 172.21.15.67:0/35324805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ae800640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b009e640 0x7fc3b00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ad400640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3b0150c40 0x7fc3b00c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b009e640 msgr2=0x7fc3b00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b009e640 0x7fc3b00c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3b0150c40 msgr2=0x7fc3b00c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3b0150c40 0x7fc3b00c6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.073+0000 7fc3ade00640 1 -- 172.21.15.67:0/35324805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b009eb70 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3ade00640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0005040 tx=0x7fc3a0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0041070 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a0005480 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b009ee50 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0001370 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.074+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b015f2d0 con 0x7fc394003a50 2024-04-17T08:30:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.075+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b00755d0 con 0x7fc394003a50 2024-04-17T08:30:23.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.076+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7fc3a0044070 con 0x7fc394003a50 2024-04-17T08:30:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.078+0000 7fc3a6000640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 0x7fc39006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.078+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 6227+0+0 (secure 0 0 0) 0x7fc3a00adad0 con 0x7fc394003a50 2024-04-17T08:30:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.078+0000 7fc3ad400640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 msgr2=0x7fc39006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.078+0000 7fc3ad400640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 0x7fc39006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:23.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.082+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc3a0006e20 con 0x7fc394003a50 2024-04-17T08:30:23.112 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.112+0000 7f47cf2da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:30:23.132 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.133+0000 7f8291e15040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:30:23.166 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.167+0000 7f47cf2da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:30:23.190 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.191+0000 7f8291e15040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:30:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.279+0000 7fc3ad400640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 msgr2=0x7fc39006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.279+0000 7fc3ad400640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 0x7fc39006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.287+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3b00b7480 con 0x7fc394003a50 2024-04-17T08:30:23.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.289+0000 7fc3a6000640 1 -- 172.21.15.67:0/35324805 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) ==== 74+0+17445 (secure 0 0 0) 0x7fc3a007b590 con 0x7fc394003a50 2024-04-17T08:30:23.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:23.289 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":166,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:22.444873+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":111,"snap_epoch":166,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.e","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:6832/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/4218440392":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2263870823":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2708413279":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3224926034":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/791073325":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2286768646":"2024-04-18T07:34:28.771962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":53},{"begin":59,"length":53}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":41,"length":15},{"begin":97,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.291+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 msgr2=0x7fc39006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.291+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 0x7fc39006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.291+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 msgr2=0x7fc3b009bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.291+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0005040 tx=0x7fc3a0001120 comp rx=0 tx=0).stop 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 shutdown_connections 2024-04-17T08:30:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fc39006ca40 0x7fc39006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3b009e640 0x7fc3b00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3b0150c40 0x7fc3b00c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 --2- 172.21.15.67:0/35324805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc394003a50 0x7fc3b009bfc0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 >> 172.21.15.67:0/35324805 conn(0x7fc3b009aa60 msgr2=0x7fc3b00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 shutdown_connections 2024-04-17T08:30:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.292+0000 7fc3af200640 1 -- 172.21.15.67:0/35324805 wait complete. 2024-04-17T08:30:23.311 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:34:11.829435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:34:12.928331+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 111, 'snap_epoch': 166, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:30:23.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:30:23.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.524+0000 7f46ca600640 1 Processor -- start 2024-04-17T08:30:23.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.525+0000 7f46ca600640 1 -- start start 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46ca600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46c40d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46ca600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c40d4740 con 0x7f46c40d18b0 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46ca600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c40d2a00 con 0x7f46c415ec00 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46ca600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f46c40d3610 con 0x7f46c40b7b60 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46c3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46c40d18b0 0x7f46c40b6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46822/0 (socket says 172.21.15.67:46822) 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.526+0000 7f46c3e00640 1 -- 172.21.15.67:0/2629297557 learned_addr learned my addr 172.21.15.67:0/2629297557 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46bbe00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46c40d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634980606 0 0) 0x7f46c40d4740 con 0x7f46c40d18b0 2024-04-17T08:30:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a4003520 con 0x7f46c40d18b0 2024-04-17T08:30:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2971582747 0 0) 0x7f46a4003520 con 0x7f46c40d18b0 2024-04-17T08:30:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 msgr2=0x7f46c40d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46c40d1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 >> v1:172.21.15.187:6789/0 conn(0x7f46c415ec00 legacy=0x7f46c40cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c40d2ce0 con 0x7f46c40d18b0 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f46bc0019f0 con 0x7f46c40d18b0 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 >> v1:172.21.15.67:6789/0 conn(0x7f46c40d18b0 legacy=0x7f46c40b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46a40039e0 0x7f46a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.527+0000 7f46c2a00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46a4004800 0x7f46a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c2a00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46a4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40d3610 con 0x7f46a40039e0 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f46a400bb50 con 0x7f46a4004800 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f46a400bd50 con 0x7f46c40b7b60 2024-04-17T08:30:23.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46a40039e0 0x7f46a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46bbe00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46a4004800 0x7f46a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3e00640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46a4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 -- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 msgr2=0x7f46a4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46c40b7b60 0x7f46a4005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 -- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46a4004800 msgr2=0x7f46a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46a4004800 0x7f46a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.528+0000 7f46c3400640 1 -- 172.21.15.67:0/2629297557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a4003520 con 0x7f46a40039e0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.529+0000 7f46c3400640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46a40039e0 0x7f46a4003e00 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f46b4001750 tx=0x7f46a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a695ecd3df8967b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.529+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4001370 con 0x7f46a40039e0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.529+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b40052d0 con 0x7f46a40039e0 2024-04-17T08:30:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.529+0000 7f46c2a00640 1 -- 172.21.15.67:0/2629297557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4003830 con 0x7f46a40039e0 2024-04-17T08:30:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 -- 172.21.15.67:0/2629297557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46a40039e0 msgr2=0x7f46a4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 --2- 172.21.15.67:0/2629297557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46a40039e0 0x7f46a4003e00 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f46b4001750 tx=0x7f46a4005830 comp rx=0 tx=0).stop 2024-04-17T08:30:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46c3e00640 1 -- 172.21.15.67:0/2629297557 reap_dead start 2024-04-17T08:30:23.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 -- 172.21.15.67:0/2629297557 shutdown_connections 2024-04-17T08:30:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 -- 172.21.15.67:0/2629297557 >> 172.21.15.67:0/2629297557 conn(0x7f46c406ccb0 msgr2=0x7f46c406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 -- 172.21.15.67:0/2629297557 shutdown_connections 2024-04-17T08:30:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.530+0000 7f46ca600640 1 -- 172.21.15.67:0/2629297557 wait complete. 2024-04-17T08:30:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.532+0000 7f46ca600640 1 Processor -- start 2024-04-17T08:30:23.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.532+0000 7f46ca600640 1 -- start start 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46a40039e0 0x7f46c414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46c40c8030 0x7f46c40c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46c415efe0 con 0x7f46c4160c50 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40d32a0 con 0x7f46c40c8030 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46ca600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f46c415f160 con 0x7f46a40039e0 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56416/0 (socket says 172.21.15.67:56416) 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.533+0000 7f46bbe00640 1 -- 172.21.15.67:0/2464335249 learned_addr learned my addr 172.21.15.67:0/2464335249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46c3400640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46a40039e0 0x7f46c414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46c3e00640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46c40c8030 0x7f46c40c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46a40039e0 msgr2=0x7f46c414ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46a40039e0 0x7f46c414ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46c40c8030 msgr2=0x7f46c40c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46c40c8030 0x7f46c40c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 -- 172.21.15.67:0/2464335249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c40c8c40 con 0x7f46c4160c50 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bbe00640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f46ac011600 tx=0x7f46ac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.534+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00f920 con 0x7f46c4160c50 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.535+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ac00fac0 con 0x7f46c4160c50 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.535+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac00fe70 con 0x7f46c4160c50 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.535+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c40c8f20 con 0x7f46c4160c50 2024-04-17T08:30:23.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.535+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c4154d10 con 0x7f46c4160c50 2024-04-17T08:30:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.536+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c4068b90 con 0x7f46c4160c50 2024-04-17T08:30:23.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.537+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7f46ac015460 con 0x7f46c4160c50 2024-04-17T08:30:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.538+0000 7f46bb400640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 0x7f46a006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.538+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 6227+0+0 (secure 0 0 0) 0x7f46ac08c750 con 0x7f46c4160c50 2024-04-17T08:30:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.538+0000 7f46c3400640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 msgr2=0x7f46a006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:23.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.538+0000 7f46c3400640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 0x7f46a006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.542+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f46ac019030 con 0x7f46c4160c50 2024-04-17T08:30:23.622 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.623+0000 7f47cf2da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:30:23.677 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.678+0000 7f47cf2da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:30:23.680 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.681+0000 7f8291e15040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:30:23.728 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.729+0000 7f47cf2da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:30:23.735 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.737+0000 7f8291e15040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:30:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.739+0000 7f46c3400640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 msgr2=0x7f46a006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:23.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.739+0000 7f46c3400640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 0x7f46a006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.754+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46c415e3b0 con 0x7f46c4160c50 2024-04-17T08:30:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.755+0000 7f46bb400640 1 -- 172.21.15.67:0/2464335249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) ==== 93+0+10 (secure 0 0 0) 0x7f46ac05a850 con 0x7f46c4160c50 2024-04-17T08:30:23.754 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 msgr2=0x7f46a006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 0x7f46a006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 msgr2=0x7f46c41610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f46ac011600 tx=0x7f46ac011ac0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 shutdown_connections 2024-04-17T08:30:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f46a006ca40 0x7f46a006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f46c40c8030 0x7f46c40c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c4160c50 0x7f46c41610d0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 --2- 172.21.15.67:0/2464335249 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f46a40039e0 0x7f46c414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 >> 172.21.15.67:0/2464335249 conn(0x7f46c406ccb0 msgr2=0x7f46c40cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 shutdown_connections 2024-04-17T08:30:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.758+0000 7f46ca600640 1 -- 172.21.15.67:0/2464335249 wait complete. 2024-04-17T08:30:23.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:30:23.788 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.789+0000 7f47cf2da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:30:23.791 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.792+0000 7f8291e15040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:30:23.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.839+0000 7f47cf2da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:30:23.856 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.857+0000 7f8291e15040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:30:23.910 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:23.911+0000 7f8291e15040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:30:23.966 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:23.967+0000 7f47cf2da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:30:23.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.985+0000 7f6718a00640 1 Processor -- start 2024-04-17T08:30:23.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.985+0000 7f6718a00640 1 -- start start 2024-04-17T08:30:23.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6718a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f671415ecb0 0x7f67140d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6718a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 0x7f67140b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6718a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67140d00b0 con 0x7f67140d3210 2024-04-17T08:30:23.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6718a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f67140d0c30 con 0x7f671415ecb0 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6718a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67140d0390 con 0x7f67140cfab0 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6712800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f671415ecb0 0x7f67140d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6709e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67140d3210 0x7f67140cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46824/0 (socket says 172.21.15.67:46824) 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6709e00640 1 -- 172.21.15.67:0/3919701546 learned_addr learned my addr 172.21.15.67:0/3919701546 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.987+0000 7f6713200640 1 --2- 172.21.15.67:0/3919701546 >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 0x7f67140b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6711e00640 1 -- 172.21.15.67:0/3919701546 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076978971 0 0) 0x7f67140d00b0 con 0x7f67140d3210 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6711e00640 1 -- 172.21.15.67:0/3919701546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66f8003520 con 0x7f67140d3210 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6712800640 1 -- 172.21.15.67:0/3919701546 >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 msgr2=0x7f67140b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6712800640 1 --2- 172.21.15.67:0/3919701546 >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 0x7f67140b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6712800640 1 -- 172.21.15.67:0/3919701546 >> v1:172.21.15.67:6789/0 conn(0x7f67140d3210 legacy=0x7f67140cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6712800640 1 -- 172.21.15.67:0/3919701546 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67140d0670 con 0x7f671415ecb0 2024-04-17T08:30:23.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.988+0000 7f6713200640 1 --2- 172.21.15.67:0/3919701546 >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 0x7f67140b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.989+0000 7f6712800640 1 --2- 172.21.15.67:0/3919701546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f671415ecb0 0x7f67140d2cd0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f67140c81b0 tx=0x7f670402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0002a4155af31b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.989+0000 7f6711e00640 1 -- 172.21.15.67:0/3919701546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704039070 con 0x7f671415ecb0 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.989+0000 7f6711e00640 1 -- 172.21.15.67:0/3919701546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670402cbe0 con 0x7f671415ecb0 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.989+0000 7f6711e00640 1 -- 172.21.15.67:0/3919701546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704034440 con 0x7f671415ecb0 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 -- 172.21.15.67:0/3919701546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f671415ecb0 msgr2=0x7f67140d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 --2- 172.21.15.67:0/3919701546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f671415ecb0 0x7f67140d2cd0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f67140c81b0 tx=0x7f670402be30 comp rx=0 tx=0).stop 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 -- 172.21.15.67:0/3919701546 shutdown_connections 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 --2- 172.21.15.67:0/3919701546 >> v2:172.21.15.187:3300/0 conn(0x7f67140cfab0 0x7f67140b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 --2- 172.21.15.67:0/3919701546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f671415ecb0 0x7f67140d2cd0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 -- 172.21.15.67:0/3919701546 >> 172.21.15.67:0/3919701546 conn(0x7f671408bff0 msgr2=0x7f67140d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 -- 172.21.15.67:0/3919701546 shutdown_connections 2024-04-17T08:30:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.990+0000 7f6718a00640 1 -- 172.21.15.67:0/3919701546 wait complete. 2024-04-17T08:30:23.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.993+0000 7f6718a00640 1 Processor -- start 2024-04-17T08:30:23.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.993+0000 7f6718a00640 1 -- start start 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 0x7f671415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f671415ecb0 0x7f67140ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f671415f210 con 0x7f67140cfab0 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f671415f090 con 0x7f67140d3210 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6718a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67140b4500 con 0x7f671415ecb0 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6712800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6712800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56446/0 (socket says 172.21.15.67:56446) 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.994+0000 7f6712800640 1 -- 172.21.15.67:0/1087563008 learned_addr learned my addr 172.21.15.67:0/1087563008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6709e00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 0x7f671415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6713200640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f671415ecb0 0x7f67140ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f671415ecb0 msgr2=0x7f67140ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f671415ecb0 0x7f67140ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 msgr2=0x7f671415cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 0x7f671415cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 -- 172.21.15.67:0/1087563008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67140cd0e0 con 0x7f67140cfab0 2024-04-17T08:30:23.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6709e00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 0x7f671415cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.995+0000 7f6712800640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f6704000f80 tx=0x7f6704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.996+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704039050 con 0x7f67140cfab0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.996+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6704045070 con 0x7f67140cfab0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.996+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704003e10 con 0x7f67140cfab0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.996+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67140cbb00 con 0x7f67140cfab0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.996+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67140cc0c0 con 0x7f67140cfab0 2024-04-17T08:30:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.997+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6714068a10 con 0x7f67140cfab0 2024-04-17T08:30:23.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.998+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7f670404a460 con 0x7f67140cfab0 2024-04-17T08:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.999+0000 7f670b400640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.999+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 6227+0+0 (secure 0 0 0) 0x7f67040ae960 con 0x7f67140cfab0 2024-04-17T08:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.999+0000 7f6709e00640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 msgr2=0x7f66f006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:23.999+0000 7f6709e00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.003+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f670407cb00 con 0x7f67140cfab0 2024-04-17T08:30:24.030 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.031+0000 7f47cf2da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:30:24.051 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.052+0000 7f8291e15040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:30:24.084 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.085+0000 7f47cf2da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:30:24.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.120+0000 7f8291e15040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:30:24.141 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.142+0000 7f47cf2da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:30:24.174 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.175+0000 7f8291e15040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:30:24.193 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.194+0000 7f47cf2da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:30:24.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.200+0000 7f6709e00640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 msgr2=0x7f66f006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:24.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.200+0000 7f6709e00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.208+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f67140b5a70 con 0x7f67140cfab0 2024-04-17T08:30:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.209+0000 7f670b400640 1 -- 172.21.15.67:0/1087563008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v167) ==== 104+0+11 (secure 0 0 0) 0x7f670407c400 con 0x7f67140cfab0 2024-04-17T08:30:24.208 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.211+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 msgr2=0x7f66f006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.211+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.211+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 msgr2=0x7f67140ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.211+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f6704000f80 tx=0x7f6704004a40 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 shutdown_connections 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f66f006ca10 0x7f66f006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f671415ecb0 0x7f67140ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67140d3210 0x7f671415cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 --2- 172.21.15.67:0/1087563008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67140cfab0 0x7f67140ccba0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 >> 172.21.15.67:0/1087563008 conn(0x7f671408bff0 msgr2=0x7f67140d3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 shutdown_connections 2024-04-17T08:30:24.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.212+0000 7f6718a00640 1 -- 172.21.15.67:0/1087563008 wait complete. 2024-04-17T08:30:24.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:30:24.236 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.237+0000 7f8291e15040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:30:24.293 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.294+0000 7f8291e15040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:30:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.448+0000 7f924c600640 1 Processor -- start 2024-04-17T08:30:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.449+0000 7f924c600640 1 -- start start 2024-04-17T08:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f924c600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f924406bfd0 0x7f92440d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f924c600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92440b54f0 con 0x7f924415e900 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f924c600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92440b57d0 con 0x7f92440d1c60 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f924c600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f92440d17d0 con 0x7f924406bfd0 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f9249c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f924415e900 0x7f92440b0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46826/0 (socket says 172.21.15.67:46826) 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f9249c00640 1 -- 172.21.15.67:0/3154436783 learned_addr learned my addr 172.21.15.67:0/3154436783 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.450+0000 7f924a600640 1 --2- 172.21.15.67:0/3154436783 >> v2:172.21.15.67:3300/0 conn(0x7f924406bfd0 0x7f92440d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.451+0000 7f9243e00640 1 -- 172.21.15.67:0/3154436783 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1240752721 0 0) 0x7f92440b54f0 con 0x7f924415e900 2024-04-17T08:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.451+0000 7f924a600640 1 -- 172.21.15.67:0/3154436783 >> v1:172.21.15.187:6789/0 conn(0x7f92440d1c60 legacy=0x7f924406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.451+0000 7f924a600640 1 -- 172.21.15.67:0/3154436783 >> v1:172.21.15.67:6789/0 conn(0x7f924415e900 legacy=0x7f92440b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.451+0000 7f924a600640 1 -- 172.21.15.67:0/3154436783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92440b5ab0 con 0x7f924406bfd0 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.452+0000 7f924a600640 1 --2- 172.21.15.67:0/3154436783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92440d20f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f923c009d90 tx=0x7f923c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ab0bf3ea9207a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.452+0000 7f9243e00640 1 -- 172.21.15.67:0/3154436783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c039070 con 0x7f924406bfd0 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.452+0000 7f9243e00640 1 -- 172.21.15.67:0/3154436783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c02cc60 con 0x7f924406bfd0 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.452+0000 7f9243e00640 1 -- 172.21.15.67:0/3154436783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c034660 con 0x7f924406bfd0 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 -- 172.21.15.67:0/3154436783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 msgr2=0x7f92440d20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 --2- 172.21.15.67:0/3154436783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92440d20f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f923c009d90 tx=0x7f923c02beb0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 -- 172.21.15.67:0/3154436783 shutdown_connections 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 --2- 172.21.15.67:0/3154436783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92440d20f0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 -- 172.21.15.67:0/3154436783 >> 172.21.15.67:0/3154436783 conn(0x7f92440b3390 msgr2=0x7f92440b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 -- 172.21.15.67:0/3154436783 shutdown_connections 2024-04-17T08:30:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.453+0000 7f924c600640 1 -- 172.21.15.67:0/3154436783 wait complete. 2024-04-17T08:30:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.455+0000 7f924c600640 1 Processor -- start 2024-04-17T08:30:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.455+0000 7f924c600640 1 -- start start 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 0x7f92440cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924415e900 0x7f92440cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92440b1010 con 0x7f924406bfd0 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f92440b51a0 con 0x7f92440d1c60 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f924c600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f924415ece0 con 0x7f924415e900 2024-04-17T08:30:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f9249c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f9249c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56470/0 (socket says 172.21.15.67:56470) 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f9249c00640 1 -- 172.21.15.67:0/2590994629 learned_addr learned my addr 172.21.15.67:0/2590994629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.456+0000 7f9249200640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 0x7f92440cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924415e900 msgr2=0x7f92440cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f924a600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924415e900 0x7f92440cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924415e900 0x7f92440cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 msgr2=0x7f92440cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 0x7f92440cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 -- 172.21.15.67:0/2590994629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9244154d30 con 0x7f924406bfd0 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249200640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 0x7f92440cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9249c00640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f923800c3f0 tx=0x7f923800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92380074e0 con 0x7f924406bfd0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92380047e0 con 0x7f924406bfd0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.457+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238004c10 con 0x7f924406bfd0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.458+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9244155010 con 0x7f924406bfd0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.458+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92441555d0 con 0x7f924406bfd0 2024-04-17T08:30:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.459+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9244155900 con 0x7f924406bfd0 2024-04-17T08:30:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.461+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7f9238011070 con 0x7f924406bfd0 2024-04-17T08:30:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.462+0000 7f9241600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 0x7f922406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.462+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6271+0+0 (secure 0 0 0) 0x7f923808bb90 con 0x7f924406bfd0 2024-04-17T08:30:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.462+0000 7f9249200640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 msgr2=0x7f922406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.462+0000 7f9249200640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 0x7f922406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:24.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.467+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f923801a2b0 con 0x7f924406bfd0 2024-04-17T08:30:24.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.596+0000 7f47cf2da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:30:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.663+0000 7f9249200640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 msgr2=0x7f922406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.663+0000 7f9249200640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 0x7f922406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.670+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9244068b90 con 0x7f924406bfd0 2024-04-17T08:30:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.671+0000 7f9241600640 1 -- 172.21.15.67:0/2590994629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) ==== 100+0+11 (secure 0 0 0) 0x7f9238059be0 con 0x7f924406bfd0 2024-04-17T08:30:24.670 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 msgr2=0x7f922406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 0x7f922406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 msgr2=0x7f92441522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f923800c3f0 tx=0x7f923800c8b0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 shutdown_connections 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f922406ca10 0x7f922406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924415e900 0x7f92440cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92440d1c60 0x7f92440cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 --2- 172.21.15.67:0/2590994629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924406bfd0 0x7f92441522d0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.673+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 >> 172.21.15.67:0/2590994629 conn(0x7f92440b3390 msgr2=0x7f924406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.674+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 shutdown_connections 2024-04-17T08:30:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.674+0000 7f924c600640 1 -- 172.21.15.67:0/2590994629 wait complete. 2024-04-17T08:30:24.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:30:24.717 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.718+0000 7f8291e15040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:30:24.756 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.757+0000 7f47cf2da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:30:24.865 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.866+0000 7f47cf2da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:30:24.888 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:24.889+0000 7f8291e15040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:30:24.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.897+0000 7f5b21600640 1 Processor -- start 2024-04-17T08:30:24.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.897+0000 7f5b21600640 1 -- start start 2024-04-17T08:30:24.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b21600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c0afbf0 0x7f5b1c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b21600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5b1c15fad0 0x7f5b1c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b21600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b1c0cb420 con 0x7f5b1c06eec0 2024-04-17T08:30:24.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b21600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1c0cc030 con 0x7f5b1c15fad0 2024-04-17T08:30:24.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b21600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1c0cb700 con 0x7f5b1c0afbf0 2024-04-17T08:30:24.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.899+0000 7f5b1b400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5b1c06eec0 0x7f5b1c15f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:49702/0 (socket says 172.21.15.67:49702) 2024-04-17T08:30:24.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.900+0000 7f5b1b400640 1 -- 172.21.15.67:0/2005842126 learned_addr learned my addr 172.21.15.67:0/2005842126 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:24.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.900+0000 7f5b20c00640 1 --2- 172.21.15.67:0/2005842126 >> v2:172.21.15.187:3300/0 conn(0x7f5b1c15fad0 0x7f5b1c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.900+0000 7f5b1be00640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c0afbf0 0x7f5b1c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.900+0000 7f5b1aa00640 1 -- 172.21.15.67:0/2005842126 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798128530 0 0) 0x7f5b1c0cb420 con 0x7f5b1c06eec0 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.900+0000 7f5b1aa00640 1 -- 172.21.15.67:0/2005842126 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b00003520 con 0x7f5b1c06eec0 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.901+0000 7f5b20c00640 1 -- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c0afbf0 msgr2=0x7f5b1c06e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.901+0000 7f5b20c00640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c0afbf0 0x7f5b1c06e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.901+0000 7f5b20c00640 1 -- 172.21.15.67:0/2005842126 >> v1:172.21.15.187:6789/0 conn(0x7f5b1c06eec0 legacy=0x7f5b1c15f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.901+0000 7f5b20c00640 1 -- 172.21.15.67:0/2005842126 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c0cb9e0 con 0x7f5b1c15fad0 2024-04-17T08:30:24.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.903+0000 7f5b20c00640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c15fad0 0x7f5b1c0ce510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8880cfc55fb6ee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.903+0000 7f5b1aa00640 1 -- 172.21.15.67:0/2005842126 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b14039070 con 0x7f5b1c15fad0 2024-04-17T08:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.903+0000 7f5b1aa00640 1 -- 172.21.15.67:0/2005842126 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b1402cc50 con 0x7f5b1c15fad0 2024-04-17T08:30:24.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.903+0000 7f5b1aa00640 1 -- 172.21.15.67:0/2005842126 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b14034440 con 0x7f5b1c15fad0 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 -- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c15fad0 msgr2=0x7f5b1c0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c15fad0 0x7f5b1c0ce510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402bea0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 -- 172.21.15.67:0/2005842126 shutdown_connections 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c15fad0 0x7f5b1c0ce510 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 --2- 172.21.15.67:0/2005842126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c0afbf0 0x7f5b1c06e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 -- 172.21.15.67:0/2005842126 >> 172.21.15.67:0/2005842126 conn(0x7f5b1c06bef0 msgr2=0x7f5b1c0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:24.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 -- 172.21.15.67:0/2005842126 shutdown_connections 2024-04-17T08:30:24.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.904+0000 7f5b21600640 1 -- 172.21.15.67:0/2005842126 wait complete. 2024-04-17T08:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.907+0000 7f5b21600640 1 Processor -- start 2024-04-17T08:30:24.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.907+0000 7f5b21600640 1 -- start start 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 0x7f5b1c0c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c15fad0 0x7f5b1c0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c0affd0 con 0x7f5b1c06eec0 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c160110 con 0x7f5b1c0afbf0 2024-04-17T08:30:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b21600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1c0b0150 con 0x7f5b1c15fad0 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.908+0000 7f5b1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56472/0 (socket says 172.21.15.67:56472) 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 -- 172.21.15.67:0/3338307588 learned_addr learned my addr 172.21.15.67:0/3338307588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1b400640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 0x7f5b1c0c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b20c00640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c15fad0 0x7f5b1c0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c15fad0 msgr2=0x7f5b1c0c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c15fad0 0x7f5b1c0c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 msgr2=0x7f5b1c0c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 0x7f5b1c0c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 -- 172.21.15.67:0/3338307588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c15ae60 con 0x7f5b1c06eec0 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1b400640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 0x7f5b1c0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:24.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.909+0000 7f5b1be00640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9b0 tx=0x7f5b0c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:24.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.910+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c017070 con 0x7f5b1c06eec0 2024-04-17T08:30:24.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.910+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0c00c9a0 con 0x7f5b1c06eec0 2024-04-17T08:30:24.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.910+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c00cd90 con 0x7f5b1c06eec0 2024-04-17T08:30:24.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.910+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1c15a0a0 con 0x7f5b1c06eec0 2024-04-17T08:30:24.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.910+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1c15a660 con 0x7f5b1c06eec0 2024-04-17T08:30:24.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.911+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b1c068a10 con 0x7f5b1c06eec0 2024-04-17T08:30:24.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.912+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7f5b0c01a070 con 0x7f5b1c06eec0 2024-04-17T08:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.913+0000 7f5b13e00640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 0x7f5afc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.913+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6271+0+0 (secure 0 0 0) 0x7f5b0c08bfb0 con 0x7f5b1c06eec0 2024-04-17T08:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.913+0000 7f5b1b400640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 msgr2=0x7f5afc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:24.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.913+0000 7f5b1b400640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 0x7f5afc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:24.921+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5b0c05a000 con 0x7f5b1c06eec0 2024-04-17T08:30:24.924 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:24.925+0000 7f47cf2da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:30:25.015 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.015+0000 7f8291e15040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:30:25.055 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:25.056+0000 7f47cf2da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:30:25.072 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.073+0000 7f8291e15040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:30:25.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.114+0000 7f5b1b400640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 msgr2=0x7f5afc06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:25.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.114+0000 7f5b1b400640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 0x7f5afc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.155+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5b1c0b5d60 con 0x7f5b1c06eec0 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.156+0000 7f5b13e00640 1 -- 172.21.15.67:0/3338307588 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v168) ==== 80+0+475 (secure 0 0 0) 0x7f5b0c059900 con 0x7f5b1c06eec0 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/115904466 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/115904466 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/4218440392 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2263870823 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2708413279 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3224926034 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/791073325 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.155 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2286768646 2024-04-18T07:34:28.771962+0000 2024-04-17T08:30:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 msgr2=0x7f5afc06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 0x7f5afc06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 msgr2=0x7f5b1c14f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9b0 tx=0x7f5b0c00ee70 comp rx=0 tx=0).stop 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 shutdown_connections 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.158+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7f5afc06ca10 0x7f5afc06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c15fad0 0x7f5b1c0c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c0afbf0 0x7f5b1c0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 --2- 172.21.15.67:0/3338307588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c06eec0 0x7f5b1c14f170 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 >> 172.21.15.67:0/3338307588 conn(0x7f5b1c06bef0 msgr2=0x7f5b1c158fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 shutdown_connections 2024-04-17T08:30:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.159+0000 7f5b21600640 1 -- 172.21.15.67:0/3338307588 wait complete. 2024-04-17T08:30:25.164 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:25.165+0000 7f47cf2da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:30:25.176 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:30:25.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:25.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:30:25.206 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.207+0000 7f8291e15040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:30:25.301 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:30:25.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:25.302 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:30:25.326 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.327+0000 7f8291e15040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:30:25.340 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:25.341+0000 7f47cf2da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:30:25.349 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:30:25.350 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:30:25.350 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:25.392 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:25.393+0000 7f47cf2da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:30:25.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:30:25.403 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:30:25.404 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:30:25.404 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:25.457 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:30:25.457 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:30:25.457 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:30:25.457 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:25.508 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:30:25.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:25.514 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.513+0000 7f8291e15040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:30:25.539 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:25.540+0000 7f47cf2da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:30:25.565 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.566+0000 7f8291e15040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:30:25.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.712+0000 7ff4b5000640 1 Processor -- start 2024-04-17T08:30:25.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.713+0000 7ff4b5000640 1 -- start start 2024-04-17T08:30:25.712 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:25.713+0000 7f8291e15040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b015b530 0x7ff4b0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff4b00c68c0 0x7ff4b0072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b910 0x7ff4b00c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b015f3c0 con 0x7ff4b015b530 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b015f660 con 0x7ff4b00c68c0 2024-04-17T08:30:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.714+0000 7ff4b5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b015f940 con 0x7ff4b015b910 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b015b530 0x7ff4b0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b015b530 0x7ff4b0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56486/0 (socket says 172.21.15.67:56486) 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 -- 172.21.15.67:0/3159772789 learned_addr learned my addr 172.21.15.67:0/3159772789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ad400640 1 --2- 172.21.15.67:0/3159772789 >> v2:172.21.15.187:3300/0 conn(0x7ff4b00c68c0 0x7ff4b0072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ae800640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b910 0x7ff4b00c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 -- 172.21.15.67:0/3159772789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b910 msgr2=0x7ff4b00c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b910 0x7ff4b00c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 -- 172.21.15.67:0/3159772789 >> v2:172.21.15.187:3300/0 conn(0x7ff4b00c68c0 msgr2=0x7ff4b0072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 --2- 172.21.15.67:0/3159772789 >> v2:172.21.15.187:3300/0 conn(0x7ff4b00c68c0 0x7ff4b0072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.715+0000 7ff4ade00640 1 -- 172.21.15.67:0/3159772789 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b015fc20 con 0x7ff4b015b530 2024-04-17T08:30:25.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.716+0000 7ff4ade00640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b015b530 0x7ff4b0071d10 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0071cc0 tx=0x7ff4a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e9d4eadb8d2ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:25.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.717+0000 7ff4aca00640 1 -- 172.21.15.67:0/3159772789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0035020 con 0x7ff4b015b530 2024-04-17T08:30:25.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.717+0000 7ff4aca00640 1 -- 172.21.15.67:0/3159772789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a0031870 con 0x7ff4b015b530 2024-04-17T08:30:25.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.717+0000 7ff4aca00640 1 -- 172.21.15.67:0/3159772789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0031c40 con 0x7ff4b015b530 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.717+0000 7ff4b5000640 1 -- 172.21.15.67:0/3159772789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b015b530 msgr2=0x7ff4b0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.717+0000 7ff4b5000640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b015b530 0x7ff4b0071d10 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0071cc0 tx=0x7ff4a002b910 comp rx=0 tx=0).stop 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 -- 172.21.15.67:0/3159772789 shutdown_connections 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b910 0x7ff4b00c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 --2- 172.21.15.67:0/3159772789 >> v2:172.21.15.187:3300/0 conn(0x7ff4b00c68c0 0x7ff4b0072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 --2- 172.21.15.67:0/3159772789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b015b530 0x7ff4b0071d10 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 -- 172.21.15.67:0/3159772789 >> 172.21.15.67:0/3159772789 conn(0x7ff4b006bf50 msgr2=0x7ff4b00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 -- 172.21.15.67:0/3159772789 shutdown_connections 2024-04-17T08:30:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.718+0000 7ff4b5000640 1 -- 172.21.15.67:0/3159772789 wait complete. 2024-04-17T08:30:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.719+0000 7ff4b5000640 1 Processor -- start 2024-04-17T08:30:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.720+0000 7ff4b5000640 1 -- start start 2024-04-17T08:30:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.720+0000 7ff4b5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.720+0000 7ff4b5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b530 0x7ff4b01f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4b5000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4b015b910 0x7ff4b0155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4b5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b0160140 con 0x7ff4b00c68c0 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4b5000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b00cc950 con 0x7ff4b015b910 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4b5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b00b38d0 con 0x7ff4b015b530 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56502/0 (socket says 172.21.15.67:56502) 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 -- 172.21.15.67:0/2249217191 learned_addr learned my addr 172.21.15.67:0/2249217191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ae800640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4b015b910 0x7ff4b0155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ad400640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b530 0x7ff4b01f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b530 msgr2=0x7ff4b01f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b530 0x7ff4b01f5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4b015b910 msgr2=0x7ff4b0155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4b015b910 0x7ff4b0155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.721+0000 7ff4ade00640 1 -- 172.21.15.67:0/2249217191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b00b3bb0 con 0x7ff4b00c68c0 2024-04-17T08:30:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4ade00640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000c00 tx=0x7ff4a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0035040 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a0030d00 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00456a0 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b00b3e90 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.722+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b0156420 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.723+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b0068b90 con 0x7ff4b00c68c0 2024-04-17T08:30:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.724+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7ff4a0038020 con 0x7ff4b00c68c0 2024-04-17T08:30:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.725+0000 7ff4a6000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 0x7ff48c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.725+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6271+0+0 (secure 0 0 0) 0x7ff4a00ae2b0 con 0x7ff4b00c68c0 2024-04-17T08:30:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.725+0000 7ff4ad400640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 msgr2=0x7ff48c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.725+0000 7ff4ad400640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 0x7ff48c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:25.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.730+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff4a007c390 con 0x7ff4b00c68c0 2024-04-17T08:30:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.926+0000 7ff4ad400640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 msgr2=0x7ff48c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.926+0000 7ff4ad400640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 0x7ff48c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.969+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4b00c6390 con 0x7ff4b00c68c0 2024-04-17T08:30:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.970+0000 7ff4a6000640 1 -- 172.21.15.67:0/2249217191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7ff4a007bc90 con 0x7ff4b00c68c0 2024-04-17T08:30:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:25.970 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T08:30:25.970 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T08:30:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 msgr2=0x7ff48c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 0x7ff48c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 msgr2=0x7ff4b00b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0000c00 tx=0x7ff4a00046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 shutdown_connections 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7ff48c06ca10 0x7ff48c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4b015b910 0x7ff4b0155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4b015b530 0x7ff4b01f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 --2- 172.21.15.67:0/2249217191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00c68c0 0x7ff4b00b31a0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 >> 172.21.15.67:0/2249217191 conn(0x7ff4b006bf50 msgr2=0x7ff4b015e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 shutdown_connections 2024-04-17T08:30:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:25.973+0000 7ff4b5000640 1 -- 172.21.15.67:0/2249217191 wait complete. 2024-04-17T08:30:25.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:26.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.206+0000 7fcdb0400640 1 Processor -- start 2024-04-17T08:30:26.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.207+0000 7fcdb0400640 1 -- start start 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdb0400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcda80b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdb0400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda80d3020 con 0x7fcda80cf640 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdb0400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda80d3300 con 0x7fcda815ed10 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdb0400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcda80d3c30 con 0x7fcda80d2ab0 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdada00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcda80cf640 0x7fcda80cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46832/0 (socket says 172.21.15.67:46832) 2024-04-17T08:30:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.208+0000 7fcdada00640 1 -- 172.21.15.67:0/72444891 learned_addr learned my addr 172.21.15.67:0/72444891 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855411130 0 0) 0x7fcda80d3020 con 0x7fcda80cf640 2024-04-17T08:30:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd90003520 con 0x7fcda80cf640 2024-04-17T08:30:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcdae400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcda80b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 861233192 0 0) 0x7fcd90003520 con 0x7fcda80cf640 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 msgr2=0x7fcda80b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcda80b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 >> v1:172.21.15.187:6789/0 conn(0x7fcda815ed10 legacy=0x7fcda80d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda80d35e0 con 0x7fcda80cf640 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fcd9c001670 con 0x7fcda80cf640 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 >> v1:172.21.15.67:6789/0 conn(0x7fcda80cf640 legacy=0x7fcda80cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd900039e0 0x7fcd90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcd90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcd900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcda80d3c30 con 0x7fcd90004d40 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9000bbe0 con 0x7fcd900039e0 2024-04-17T08:30:26.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.209+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcd9000bde0 con 0x7fcda80d2ab0 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcd900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdada00640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd900039e0 0x7fcd90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdad000640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcd90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 -- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 msgr2=0x7fcd90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcd90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 -- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd900039e0 msgr2=0x7fcd90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd900039e0 0x7fcd90003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdae400640 1 -- 172.21.15.67:0/72444891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd90003520 con 0x7fcd90004d40 2024-04-17T08:30:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.210+0000 7fcdad000640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80d2ab0 0x7fcd90004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.211+0000 7fcdae400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcd900051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcda0007360 tx=0x7fcd90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8798d98e69af33b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.211+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0039070 con 0x7fcd90004d40 2024-04-17T08:30:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.211+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda002ccb0 con 0x7fcd90004d40 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.211+0000 7fcda7e00640 1 -- 172.21.15.67:0/72444891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0009be0 con 0x7fcd90004d40 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 -- 172.21.15.67:0/72444891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 msgr2=0x7fcd900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 --2- 172.21.15.67:0/72444891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcd900051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fcda0007360 tx=0x7fcd90005830 comp rx=0 tx=0).stop 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdae400640 1 -- 172.21.15.67:0/72444891 reap_dead start 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 -- 172.21.15.67:0/72444891 shutdown_connections 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 -- 172.21.15.67:0/72444891 >> 172.21.15.67:0/72444891 conn(0x7fcda806cbd0 msgr2=0x7fcda806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 -- 172.21.15.67:0/72444891 shutdown_connections 2024-04-17T08:30:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.212+0000 7fcdb0400640 1 -- 172.21.15.67:0/72444891 wait complete. 2024-04-17T08:30:26.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.214+0000 7fcdb0400640 1 Processor -- start 2024-04-17T08:30:26.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.214+0000 7fcdb0400640 1 -- start start 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 0x7fcda80b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 0x7fcda8150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcda80ba300 con 0x7fcd90004d40 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcda80d25b0 con 0x7fcda81509c0 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdb0400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcda80ba480 con 0x7fcda80b4310 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdada00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdada00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56528/0 (socket says 172.21.15.67:56528) 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.215+0000 7fcdada00640 1 -- 172.21.15.67:0/1490844186 learned_addr learned my addr 172.21.15.67:0/1490844186 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdad000640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 0x7fcda80b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdae400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 0x7fcda8150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 msgr2=0x7fcda80b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 0x7fcda80b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 msgr2=0x7fcda8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 0x7fcda8150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 -- 172.21.15.67:0/1490844186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda80b5390 con 0x7fcd90004d40 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdad000640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 0x7fcda80b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdae400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 0x7fcda8150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcdada00640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e840 tx=0x7fcd9c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.216+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00c870 con 0x7fcd90004d40 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.217+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd9c018070 con 0x7fcd90004d40 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.217+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00cc70 con 0x7fcd90004d40 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.217+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda80b4dd0 con 0x7fcd90004d40 2024-04-17T08:30:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.217+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda80b5690 con 0x7fcd90004d40 2024-04-17T08:30:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.218+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda8068b90 con 0x7fcd90004d40 2024-04-17T08:30:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.219+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7fcd9c013420 con 0x7fcd90004d40 2024-04-17T08:30:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.219+0000 7fcda5600640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 0x7fcd8806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.220+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6271+0+0 (secure 0 0 0) 0x7fcd9c08bc00 con 0x7fcd90004d40 2024-04-17T08:30:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.220+0000 7fcdad000640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 msgr2=0x7fcd8806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.220+0000 7fcdad000640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 0x7fcd8806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.224+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcd9c059c50 con 0x7fcd90004d40 2024-04-17T08:30:26.403 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:26.404+0000 7f47cf2da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:30:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.421+0000 7fcdad000640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 msgr2=0x7fcd8806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.421+0000 7fcdad000640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 0x7fcd8806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:26.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.429+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcda80b40b0 con 0x7fcd90004d40 2024-04-17T08:30:26.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.431+0000 7fcda5600640 1 -- 172.21.15.67:0/1490844186 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+17515 (secure 0 0 0) 0x7fcd9c059550 con 0x7fcd90004d40 2024-04-17T08:30:26.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:26.430 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":168,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:24.389957+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.e","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:6832/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/4218440392":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2263870823":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2708413279":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3224926034":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/791073325":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2286768646":"2024-04-18T07:34:28.771962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":13,"length":45},{"begin":69,"length":45}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":3,"length":10},{"begin":59,"length":10}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.433+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 msgr2=0x7fcd8806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.433+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 0x7fcd8806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.433+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 msgr2=0x7fcda8160400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.433+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00e840 tx=0x7fcd9c00ed00 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 shutdown_connections 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fcd8806ca40 0x7fcd8806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcda81509c0 0x7fcda8150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcda80b4310 0x7fcda80b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 --2- 172.21.15.67:0/1490844186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd90004d40 0x7fcda8160400 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 >> 172.21.15.67:0/1490844186 conn(0x7fcda806cbd0 msgr2=0x7fcda80c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 shutdown_connections 2024-04-17T08:30:26.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.434+0000 7fcdb0400640 1 -- 172.21.15.67:0/1490844186 wait complete. 2024-04-17T08:30:26.461 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T07:34:11.829435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T07:34:12.928331+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 168, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:30:26.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:30:26.594 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:26.595+0000 7f8291e15040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:30:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.669+0000 7fa0b7200640 1 Processor -- start 2024-04-17T08:30:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.669+0000 7fa0b7200640 1 -- start start 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa0b80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa0b81622b0 0x7fa0b80cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b80b5110 con 0x7fa0b80b4c70 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b80b53f0 con 0x7fa0b81622b0 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80b56d0 con 0x7fa0b80d2ff0 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0b80b4c70 0x7fa0b8161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46844/0 (socket says 172.21.15.67:46844) 2024-04-17T08:30:26.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b5400640 1 -- 172.21.15.67:0/2728768974 learned_addr learned my addr 172.21.15.67:0/2728768974 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b6800640 1 --2- 172.21.15.67:0/2728768974 >> v2:172.21.15.187:3300/0 conn(0x7fa0b81622b0 0x7fa0b80cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa0b80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.671+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155142706 0 0) 0x7fa0b80b5110 con 0x7fa0b80b4c70 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa09c003520 con 0x7fa0b80b4c70 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1117526423 0 0) 0x7fa09c003520 con 0x7fa0b80b4c70 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 msgr2=0x7fa0b80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa0b80b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 >> v2:172.21.15.187:3300/0 conn(0x7fa0b81622b0 msgr2=0x7fa0b80cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 --2- 172.21.15.67:0/2728768974 >> v2:172.21.15.187:3300/0 conn(0x7fa0b81622b0 0x7fa0b80cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80cf340 con 0x7fa0b80b4c70 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa0a4001670 con 0x7fa0b80b4c70 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 >> v1:172.21.15.67:6789/0 conn(0x7fa0b80b4c70 legacy=0x7fa0b8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa09c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa09c003f80 0x7fa09c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa09c004d40 0x7fa09c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80b53f0 con 0x7fa09c004d40 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80b56d0 con 0x7fa09c003f80 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa09c00b9c0 con 0x7fa0b80d2ff0 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.672+0000 7fa0b6800640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa09c004d40 0x7fa09c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa09c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b5400640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa09c003f80 0x7fa09c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b6800640 1 -- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 msgr2=0x7fa09c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b6800640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa09c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b6800640 1 -- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa09c003f80 msgr2=0x7fa09c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b6800640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa09c003f80 0x7fa09c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b6800640 1 -- 172.21.15.67:0/2728768974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09c003520 con 0x7fa09c004d40 2024-04-17T08:30:26.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b5400640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa09c003f80 0x7fa09c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:26.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.673+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80d2ff0 0x7fa09c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.674+0000 7fa0b6800640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa09c004d40 0x7fa09c0051c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009fc0 tx=0x7fa09c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f5bbbe0e1d45bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0039070 con 0x7fa09c004d40 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b002cbf0 con 0x7fa09c004d40 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b4a00640 1 -- 172.21.15.67:0/2728768974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0009c10 con 0x7fa09c004d40 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b7200640 1 -- 172.21.15.67:0/2728768974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa09c004d40 msgr2=0x7fa09c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b7200640 1 --2- 172.21.15.67:0/2728768974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa09c004d40 0x7fa09c0051c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009fc0 tx=0x7fa09c005830 comp rx=0 tx=0).stop 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b6800640 1 -- 172.21.15.67:0/2728768974 reap_dead start 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.675+0000 7fa0b7200640 1 -- 172.21.15.67:0/2728768974 shutdown_connections 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.676+0000 7fa0b7200640 1 -- 172.21.15.67:0/2728768974 >> 172.21.15.67:0/2728768974 conn(0x7fa0b809da80 msgr2=0x7fa0b808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:26.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.676+0000 7fa0b7200640 1 -- 172.21.15.67:0/2728768974 shutdown_connections 2024-04-17T08:30:26.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.676+0000 7fa0b7200640 1 -- 172.21.15.67:0/2728768974 wait complete. 2024-04-17T08:30:26.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.677+0000 7fa0b7200640 1 Processor -- start 2024-04-17T08:30:26.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 -- start start 2024-04-17T08:30:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80c9a50 0x7fa0b81f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0b81f7f20 0x7fa0b80b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80d3550 con 0x7fa0b81579b0 2024-04-17T08:30:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.678+0000 7fa0b7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80d33d0 con 0x7fa0b81f7f20 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80d36d0 con 0x7fa0b80c9a50 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56548/0 (socket says 172.21.15.67:56548) 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 -- 172.21.15.67:0/3332640197 learned_addr learned my addr 172.21.15.67:0/3332640197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5400640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80c9a50 0x7fa0b81f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b6800640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0b81f7f20 0x7fa0b80b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80c9a50 msgr2=0x7fa0b81f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80c9a50 0x7fa0b81f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0b81f7f20 msgr2=0x7fa0b80b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0b81f7f20 0x7fa0b80b84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.679+0000 7fa0b5e00640 1 -- 172.21.15.67:0/3332640197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80c9fc0 con 0x7fa0b81579b0 2024-04-17T08:30:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0b5e00640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80108b0 tx=0x7fa0a8010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a800f040 con 0x7fa0b81579b0 2024-04-17T08:30:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a800eb60 con 0x7fa0b81579b0 2024-04-17T08:30:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a801c400 con 0x7fa0b81579b0 2024-04-17T08:30:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b80ca2a0 con 0x7fa0b81579b0 2024-04-17T08:30:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.680+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b80ca860 con 0x7fa0b81579b0 2024-04-17T08:30:26.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.681+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b80cfee0 con 0x7fa0b81579b0 2024-04-17T08:30:26.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.682+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 86) ==== 91154+0+0 (secure 0 0 0) 0x7fa0a801c5a0 con 0x7fa0b81579b0 2024-04-17T08:30:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.683+0000 7fa0ae000640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 0x7fa09806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.683+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 6271+0+0 (secure 0 0 0) 0x7fa0a808c5e0 con 0x7fa0b81579b0 2024-04-17T08:30:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.683+0000 7fa0b5400640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 msgr2=0x7fa09806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.683+0000 7fa0b5400640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 0x7fa09806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:30:26.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.692+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa0b80cfee0 con 0x7fa0b81579b0 2024-04-17T08:30:26.740 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:26.741+0000 7f47cf2da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:30:26.798 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:26.799+0000 7f47cf2da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:30:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.884+0000 7fa0b5400640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 msgr2=0x7fa09806ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2413623528 2024-04-17T08:30:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.884+0000 7fa0b5400640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 0x7fa09806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:30:26.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.892+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa0b80755d0 con 0x7fa0b81579b0 2024-04-17T08:30:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.893+0000 7fa0ae000640 1 -- 172.21.15.67:0/3332640197 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) ==== 93+0+10 (secure 0 0 0) 0x7fa0a8018030 con 0x7fa0b81579b0 2024-04-17T08:30:26.892 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.895+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 msgr2=0x7fa09806ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.895+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 0x7fa09806ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.895+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 msgr2=0x7fa0b80c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.895+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80108b0 tx=0x7fa0a8010d70 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 shutdown_connections 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:6832/2413623528,v1:172.21.15.187:6833/2413623528] conn(0x7fa09806ca40 0x7fa09806ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0b81f7f20 0x7fa0b80b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0b80c9a50 0x7fa0b81f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 --2- 172.21.15.67:0/3332640197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b81579b0 0x7fa0b80c8fe0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 >> 172.21.15.67:0/3332640197 conn(0x7fa0b809da80 msgr2=0x7fa0b80b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 shutdown_connections 2024-04-17T08:30:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:26.896+0000 7fa0b7200640 1 -- 172.21.15.67:0/3332640197 wait complete. 2024-04-17T08:30:26.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:30:26.925 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:26.926+0000 7f8291e15040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:30:26.930 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:26.931+0000 7f47cf2da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:30:26.994 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:26.995+0000 7f8291e15040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.026+0000 7f4749800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.026+0000 7f4749800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.026+0000 7f4749800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.026+0000 7f4749800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.026+0000 7f4749800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.025 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.027+0000 7f474b600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.027+0000 7f474b600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.027+0000 7f474b600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.027+0000 7f474b600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:30:27.027+0000 7f474b600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:30:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.108+0000 7f45d1000640 1 Processor -- start 2024-04-17T08:30:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.109+0000 7f45d1000640 1 -- start start 2024-04-17T08:30:27.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45d1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45d1000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc0cea70 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45d1000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc0ced50 con 0x7f45cc145ac0 2024-04-17T08:30:27.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45d1000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc0ce840 con 0x7f45cc072d80 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45ca800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45cc0ce3b0 0x7f45cc0cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46850/0 (socket says 172.21.15.67:46850) 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45ca800640 1 -- 172.21.15.67:0/3331689730 learned_addr learned my addr 172.21.15.67:0/3331689730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.111+0000 7f45c9400640 1 --2- 172.21.15.67:0/3331689730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269166431 0 0) 0x7f45cc0cea70 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45b0003520 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186978664 0 0) 0x7f45cc0ced50 con 0x7f45cc145ac0 2024-04-17T08:30:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45cc0cea70 con 0x7f45cc145ac0 2024-04-17T08:30:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c9400640 1 -- 172.21.15.67:0/3331689730 >> v1:172.21.15.187:6789/0 conn(0x7f45cc145ac0 legacy=0x7f45cc072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c9400640 1 -- 172.21.15.67:0/3331689730 >> v1:172.21.15.67:6789/0 conn(0x7f45cc0ce3b0 legacy=0x7f45cc0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.112+0000 7f45c9400640 1 -- 172.21.15.67:0/3331689730 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc0cf030 con 0x7f45cc072d80 2024-04-17T08:30:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.114+0000 7f45c9400640 1 --2- 172.21.15.67:0/3331689730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc071ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45b40097d0 tx=0x7f45b402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee6007720b9aa64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.114+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4034070 con 0x7f45cc072d80 2024-04-17T08:30:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.114+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b4030800 con 0x7f45cc072d80 2024-04-17T08:30:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.114+0000 7f45c8a00640 1 -- 172.21.15.67:0/3331689730 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4030bd0 con 0x7f45cc072d80 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 -- 172.21.15.67:0/3331689730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 msgr2=0x7f45cc071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 --2- 172.21.15.67:0/3331689730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc071ea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45b40097d0 tx=0x7f45b402b940 comp rx=0 tx=0).stop 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 -- 172.21.15.67:0/3331689730 shutdown_connections 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 --2- 172.21.15.67:0/3331689730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc071ea0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 -- 172.21.15.67:0/3331689730 >> 172.21.15.67:0/3331689730 conn(0x7f45cc067ee0 msgr2=0x7f45cc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 -- 172.21.15.67:0/3331689730 shutdown_connections 2024-04-17T08:30:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.115+0000 7f45d1000640 1 -- 172.21.15.67:0/3331689730 wait complete. 2024-04-17T08:30:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.116+0000 7f45d1000640 1 Processor -- start 2024-04-17T08:30:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.117+0000 7f45d1000640 1 -- start start 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.117+0000 7f45d1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc1f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.117+0000 7f45d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.117+0000 7f45d1000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc145ac0 0x7f45cc15f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45d1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc145ea0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45d1000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc0cf5f0 con 0x7f45cc145ac0 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45d1000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc0d0220 con 0x7f45cc072d80 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56554/0 (socket says 172.21.15.67:56554) 2024-04-17T08:30:27.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 -- 172.21.15.67:0/2865774074 learned_addr learned my addr 172.21.15.67:0/2865774074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:27.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45ca800640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc145ac0 0x7f45cc15f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc1f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 -- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 msgr2=0x7f45cc1f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc1f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 -- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc145ac0 msgr2=0x7f45cc15f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc145ac0 0x7f45cc15f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.118+0000 7f45c9400640 1 -- 172.21.15.67:0/2865774074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc1435b0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45c9400640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f45b40098f0 tx=0x7f45b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4034040 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b4030e30 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc143890 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4039d10 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.119+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc143e50 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.120+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc0cc6c0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.122+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 87) ==== 90751+0+0 (secure 0 0 0) 0x7f45b4037070 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.123+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6527+0+0 (secure 0 0 0) 0x7f45b40adaa0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.128+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f45b407b8b0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.140 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:30:27.141+0000 7f8291e15040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:30:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.334+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45cc06bc40 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.335+0000 7f45c2000640 1 -- 172.21.15.67:0/2865774074 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v169) ==== 104+0+11 (secure 0 0 0) 0x7f45b407fbb0 con 0x7f45cc0ce3b0 2024-04-17T08:30:27.334 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:30:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 msgr2=0x7f45cc154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f45b40098f0 tx=0x7f45b40046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 shutdown_connections 2024-04-17T08:30:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc145ac0 0x7f45cc15f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0ce3b0 0x7f45cc154f10 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 --2- 172.21.15.67:0/2865774074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc072d80 0x7f45cc1f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 >> 172.21.15.67:0/2865774074 conn(0x7f45cc067ee0 msgr2=0x7f45cc140e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 shutdown_connections 2024-04-17T08:30:27.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.338+0000 7f45d1000640 1 -- 172.21.15.67:0/2865774074 wait complete. 2024-04-17T08:30:27.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:30:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.633+0000 7fb5c7200640 1 Processor -- start 2024-04-17T08:30:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.633+0000 7fb5c7200640 1 -- start start 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.634+0000 7fb5c7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb5c809aec0 0x7fb5c815f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.634+0000 7fb5c7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5c80d0350 con 0x7fb5c809aa60 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5c80d0630 con 0x7fb5c80b3c80 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb5c80b2ee0 con 0x7fb5c809aec0 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5c809aa60 0x7fb5c815f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46858/0 (socket says 172.21.15.67:46858) 2024-04-17T08:30:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c5400640 1 -- 172.21.15.67:0/3522107303 learned_addr learned my addr 172.21.15.67:0/3522107303 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c6800640 1 --2- 172.21.15.67:0/3522107303 >> v2:172.21.15.187:3300/0 conn(0x7fb5c809aec0 0x7fb5c815f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431883878 0 0) 0x7fb5c80d0350 con 0x7fb5c809aa60 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.635+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5a8003520 con 0x7fb5c809aa60 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1145599131 0 0) 0x7fb5a8003520 con 0x7fb5c809aa60 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 >> v2:172.21.15.187:3300/0 conn(0x7fb5c809aec0 msgr2=0x7fb5c815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 --2- 172.21.15.67:0/3522107303 >> v2:172.21.15.187:3300/0 conn(0x7fb5c809aec0 0x7fb5c815f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 >> v1:172.21.15.187:6789/0 conn(0x7fb5c80b3c80 legacy=0x7fb5c80cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c80d0910 con 0x7fb5c809aa60 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb5b0001670 con 0x7fb5c809aa60 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 >> v1:172.21.15.67:6789/0 conn(0x7fb5c809aa60 legacy=0x7fb5c815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 0x7fb5a8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5a8004800 0x7fb5a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a80051c0 0x7fb5a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a800c0c0 con 0x7fb5a8003a30 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a800c270 con 0x7fb5a8004800 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a800c470 con 0x7fb5a80051c0 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c5e00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 0x7fb5a8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.636+0000 7fb5c5400640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5a8004800 0x7fb5a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c5e00640 1 -- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a80051c0 msgr2=0x7fb5a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:27.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c6800640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a80051c0 0x7fb5a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c5e00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a80051c0 0x7fb5a800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c5e00640 1 -- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5a8004800 msgr2=0x7fb5a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c5e00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5a8004800 0x7fb5a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c5e00640 1 -- 172.21.15.67:0/3522107303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a8003520 con 0x7fb5a8003a30 2024-04-17T08:30:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.637+0000 7fb5c6800640 1 -- 172.21.15.67:0/3522107303 reap_dead start 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.638+0000 7fb5c5e00640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 0x7fb5a8003e50 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8001750 tx=0x7fb5a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7bc3d92f02c192 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.638+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8001370 con 0x7fb5a8003a30 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.638+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b80052d0 con 0x7fb5a8003a30 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.638+0000 7fb5c4a00640 1 -- 172.21.15.67:0/3522107303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8003830 con 0x7fb5a8003a30 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 -- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 msgr2=0x7fb5a8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 0x7fb5a8003e50 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8001750 tx=0x7fb5a8005830 comp rx=0 tx=0).stop 2024-04-17T08:30:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 -- 172.21.15.67:0/3522107303 shutdown_connections 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 --2- 172.21.15.67:0/3522107303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8003a30 0x7fb5a8003e50 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 -- 172.21.15.67:0/3522107303 >> 172.21.15.67:0/3522107303 conn(0x7fb5c80b4060 msgr2=0x7fb5c80b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 -- 172.21.15.67:0/3522107303 shutdown_connections 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.639+0000 7fb5c7200640 1 -- 172.21.15.67:0/3522107303 wait complete. 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.641+0000 7fb5c7200640 1 Processor -- start 2024-04-17T08:30:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.641+0000 7fb5c7200640 1 -- start start 2024-04-17T08:30:27.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a8003a30 0x7fb5c80c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 0x7fb5c80b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c809b350 con 0x7fb5c80c6080 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c80d0ed0 con 0x7fb5c80b1ed0 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5c80d13f0 con 0x7fb5a8003a30 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.642+0000 7fb5c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56572/0 (socket says 172.21.15.67:56572) 2024-04-17T08:30:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 -- 172.21.15.67:0/2624178972 learned_addr learned my addr 172.21.15.67:0/2624178972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c5e00640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a8003a30 0x7fb5c80c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c5400640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 0x7fb5c80b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 -- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a8003a30 msgr2=0x7fb5c80c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a8003a30 0x7fb5c80c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 -- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 msgr2=0x7fb5c80b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 0x7fb5c80b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 -- 172.21.15.67:0/2624178972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c81f8250 con 0x7fb5c80c6080 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c5400640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 0x7fb5c80b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.643+0000 7fb5c6800640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80d22d0 tx=0x7fb5c000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.644+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0016620 con 0x7fb5c80c6080 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.644+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5c001b070 con 0x7fb5c80c6080 2024-04-17T08:30:27.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.644+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0016a20 con 0x7fb5c80c6080 2024-04-17T08:30:27.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.644+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c81f8fd0 con 0x7fb5c80c6080 2024-04-17T08:30:27.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.644+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c81f84f0 con 0x7fb5c80c6080 2024-04-17T08:30:27.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.645+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c80755d0 con 0x7fb5c80c6080 2024-04-17T08:30:27.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.646+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 87) ==== 90751+0+0 (secure 0 0 0) 0x7fb5c0005d10 con 0x7fb5c80c6080 2024-04-17T08:30:27.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.647+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6527+0+0 (secure 0 0 0) 0x7fb5c008c3a0 con 0x7fb5c80c6080 2024-04-17T08:30:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.652+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb5c005a800 con 0x7fb5c80c6080 2024-04-17T08:30:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.857+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb5c80cedd0 con 0x7fb5c80c6080 2024-04-17T08:30:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.858+0000 7fb5be000640 1 -- 172.21.15.67:0/2624178972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v169) ==== 100+0+11 (secure 0 0 0) 0x7fb5c005a100 con 0x7fb5c80c6080 2024-04-17T08:30:27.857 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 msgr2=0x7fb5c80c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80d22d0 tx=0x7fb5c000f520 comp rx=0 tx=0).stop 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 shutdown_connections 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5c80c6080 0x7fb5c80c6510 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5c80b1ed0 0x7fb5c80b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 --2- 172.21.15.67:0/2624178972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb5a8003a30 0x7fb5c80c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.861+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 >> 172.21.15.67:0/2624178972 conn(0x7fb5c80b4060 msgr2=0x7fb5c81e3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.862+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 shutdown_connections 2024-04-17T08:30:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:27.862+0000 7fb5c7200640 1 -- 172.21.15.67:0/2624178972 wait complete. 2024-04-17T08:30:27.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.098+0000 7fc24d200640 1 Processor -- start 2024-04-17T08:30:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.098+0000 7fc24d200640 1 -- start start 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.099+0000 7fc24d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480cf640 0x7fc2480cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc24d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2480d2ab0 0x7fc2480b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc24d200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2480cff60 con 0x7fc24815ed10 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc24d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc2480d3c30 con 0x7fc2480d2ab0 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc24d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc2480d3020 con 0x7fc2480cf640 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc246800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2480d2ab0 0x7fc2480b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc246800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2480d2ab0 0x7fc2480b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56578/0 (socket says 172.21.15.67:56578) 2024-04-17T08:30:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc246800640 1 -- 172.21.15.67:0/3274873335 learned_addr learned my addr 172.21.15.67:0/3274873335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.100+0000 7fc245e00640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480cf640 0x7fc2480cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.101+0000 7fc246800640 1 -- 172.21.15.67:0/3274873335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480cf640 msgr2=0x7fc2480cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.101+0000 7fc246800640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480cf640 0x7fc2480cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.101+0000 7fc246800640 1 -- 172.21.15.67:0/3274873335 >> v1:172.21.15.187:6789/0 conn(0x7fc24815ed10 legacy=0x7fc2480d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.101+0000 7fc246800640 1 -- 172.21.15.67:0/3274873335 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2480d3300 con 0x7fc2480d2ab0 2024-04-17T08:30:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.103+0000 7fc246800640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480d2ab0 0x7fc2480b7b10 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1ef8fd185f7c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.103+0000 7fc245400640 1 -- 172.21.15.67:0/3274873335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24002f840 con 0x7fc2480d2ab0 2024-04-17T08:30:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc245400640 1 -- 172.21.15.67:0/3274873335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24002f9e0 con 0x7fc2480d2ab0 2024-04-17T08:30:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc245400640 1 -- 172.21.15.67:0/3274873335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24002fdb0 con 0x7fc2480d2ab0 2024-04-17T08:30:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 -- 172.21.15.67:0/3274873335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480d2ab0 msgr2=0x7fc2480b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480d2ab0 0x7fc2480b7b10 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24002ba20 comp rx=0 tx=0).stop 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 -- 172.21.15.67:0/3274873335 shutdown_connections 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480d2ab0 0x7fc2480b7b10 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 --2- 172.21.15.67:0/3274873335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480cf640 0x7fc2480cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.104+0000 7fc24d200640 1 -- 172.21.15.67:0/3274873335 >> 172.21.15.67:0/3274873335 conn(0x7fc24806cbd0 msgr2=0x7fc24806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.105+0000 7fc24d200640 1 -- 172.21.15.67:0/3274873335 shutdown_connections 2024-04-17T08:30:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.105+0000 7fc24d200640 1 -- 172.21.15.67:0/3274873335 wait complete. 2024-04-17T08:30:28.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.106+0000 7fc24d200640 1 Processor -- start 2024-04-17T08:30:28.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.106+0000 7fc24d200640 1 -- start start 2024-04-17T08:30:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 0x7fc2480b8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 0x7fc2480b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc2480d25b0 con 0x7fc2480cf640 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc2480d38c0 con 0x7fc24815ed10 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.107+0000 7fc24d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc2480ba300 con 0x7fc2480d2ab0 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56588/0 (socket says 172.21.15.67:56588) 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 -- 172.21.15.67:0/364042593 learned_addr learned my addr 172.21.15.67:0/364042593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc23d400640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 0x7fc2480b8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 -- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 msgr2=0x7fc2480b8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc246800640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 0x7fc2480b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 0x7fc2480b8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 -- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 msgr2=0x7fc2480b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 0x7fc2480b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 -- 172.21.15.67:0/364042593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2480cc730 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc23d400640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 0x7fc2480b8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc246800640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 0x7fc2480b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.108+0000 7fc245e00640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fc238007ef0 tx=0x7fc23800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.109+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238014620 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.109+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc238019070 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.109+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238014a00 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.109+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2480cca10 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.109+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2480ccfd0 con 0x7fc2480cf640 2024-04-17T08:30:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.110+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc248068b90 con 0x7fc2480cf640 2024-04-17T08:30:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.111+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 88) ==== 90878+0+0 (secure 0 0 0) 0x7fc238014ba0 con 0x7fc2480cf640 2024-04-17T08:30:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.112+0000 7fc23ea00640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 0x7fc22406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.112+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6527+0+0 (secure 0 0 0) 0x7fc23808c440 con 0x7fc2480cf640 2024-04-17T08:30:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.113+0000 7fc23d400640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 0x7fc22406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.114+0000 7fc23d400640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 0x7fc22406ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc2480b3970 tx=0x7fc230009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.117+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc23805a840 con 0x7fc2480cf640 2024-04-17T08:30:28.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.368+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc24806d200 con 0x7fc2480cf640 2024-04-17T08:30:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.370+0000 7fc23ea00640 1 -- 172.21.15.67:0/364042593 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7fc23805a140 con 0x7fc2480cf640 2024-04-17T08:30:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:28.369 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T08:30:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.372+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 msgr2=0x7fc22406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 0x7fc22406ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc2480b3970 tx=0x7fc230009040 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 msgr2=0x7fc24814ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fc238007ef0 tx=0x7fc23800e4b0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 shutdown_connections 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc22406c740 0x7fc22406ec00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc24815ed10 0x7fc2480b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2480d2ab0 0x7fc2480b8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 --2- 172.21.15.67:0/364042593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2480cf640 0x7fc24814ce90 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 >> 172.21.15.67:0/364042593 conn(0x7fc24806cbd0 msgr2=0x7fc248161150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:28.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 shutdown_connections 2024-04-17T08:30:28.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.373+0000 7fc24d200640 1 -- 172.21.15.67:0/364042593 wait complete. 2024-04-17T08:30:28.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:28.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.612+0000 7f4afd400640 1 Processor -- start 2024-04-17T08:30:28.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.612+0000 7f4afd400640 1 -- start start 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4afd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4af8141ed0 0x7f4af8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4afd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af80cd4a0 con 0x7f4af8160280 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4afd400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af81607f0 con 0x7f4af806b8c0 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4afd400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4af815ce30 con 0x7f4af8141ed0 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4af7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4af8160280 0x7f4af806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46864/0 (socket says 172.21.15.67:46864) 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4af7400640 1 -- 172.21.15.67:0/1999929650 learned_addr learned my addr 172.21.15.67:0/1999929650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:28.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.614+0000 7f4afca00640 1 --2- 172.21.15.67:0/1999929650 >> v2:172.21.15.67:3300/0 conn(0x7f4af8141ed0 0x7f4af8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.615+0000 7f4af6a00640 1 -- 172.21.15.67:0/1999929650 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647229744 0 0) 0x7f4af80cd4a0 con 0x7f4af8160280 2024-04-17T08:30:28.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.615+0000 7f4af6a00640 1 -- 172.21.15.67:0/1999929650 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4adc003520 con 0x7f4af8160280 2024-04-17T08:30:28.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.615+0000 7f4afca00640 1 -- 172.21.15.67:0/1999929650 >> v1:172.21.15.187:6789/0 conn(0x7f4af806b8c0 legacy=0x7f4af80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.615+0000 7f4afca00640 1 -- 172.21.15.67:0/1999929650 >> v1:172.21.15.67:6789/0 conn(0x7f4af8160280 legacy=0x7f4af806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.615+0000 7f4afca00640 1 -- 172.21.15.67:0/1999929650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af8160ad0 con 0x7f4af8141ed0 2024-04-17T08:30:28.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.616+0000 7f4afca00640 1 --2- 172.21.15.67:0/1999929650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8141ed0 0x7f4af8142360 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097a0 tx=0x7f4af002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75109e6ad7e04c41 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.616+0000 7f4af6a00640 1 -- 172.21.15.67:0/1999929650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0034070 con 0x7f4af8141ed0 2024-04-17T08:30:28.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.616+0000 7f4af6a00640 1 -- 172.21.15.67:0/1999929650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af0030610 con 0x7f4af8141ed0 2024-04-17T08:30:28.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.616+0000 7f4af6a00640 1 -- 172.21.15.67:0/1999929650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00309e0 con 0x7f4af8141ed0 2024-04-17T08:30:28.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 -- 172.21.15.67:0/1999929650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8141ed0 msgr2=0x7f4af8142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 --2- 172.21.15.67:0/1999929650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8141ed0 0x7f4af8142360 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097a0 tx=0x7f4af002b960 comp rx=0 tx=0).stop 2024-04-17T08:30:28.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 -- 172.21.15.67:0/1999929650 shutdown_connections 2024-04-17T08:30:28.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 --2- 172.21.15.67:0/1999929650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8141ed0 0x7f4af8142360 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 -- 172.21.15.67:0/1999929650 >> 172.21.15.67:0/1999929650 conn(0x7f4af8067d10 msgr2=0x7f4af80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:28.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 -- 172.21.15.67:0/1999929650 shutdown_connections 2024-04-17T08:30:28.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.617+0000 7f4afd400640 1 -- 172.21.15.67:0/1999929650 wait complete. 2024-04-17T08:30:28.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.619+0000 7f4afd400640 1 Processor -- start 2024-04-17T08:30:28.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.619+0000 7f4afd400640 1 -- start start 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4af806b8c0 0x7f4af80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4af8141ed0 0x7f4af813bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4af813f9b0 con 0x7f4af8160280 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8161090 con 0x7f4af8141ed0 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afd400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4af813fb30 con 0x7f4af806b8c0 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56620/0 (socket says 172.21.15.67:56620) 2024-04-17T08:30:28.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 -- 172.21.15.67:0/2719600743 learned_addr learned my addr 172.21.15.67:0/2719600743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:28.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 -- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4af806b8c0 msgr2=0x7f4af80b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4af806b8c0 0x7f4af80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 -- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4af8141ed0 msgr2=0x7f4af813bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4af8141ed0 0x7f4af813bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.620+0000 7f4afca00640 1 -- 172.21.15.67:0/2719600743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af80b4f80 con 0x7f4af8160280 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4afca00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097a0 tx=0x7f4af00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0034040 con 0x7f4af8160280 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af0030410 con 0x7f4af8160280 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0039bd0 con 0x7f4af8160280 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af80b5260 con 0x7f4af8160280 2024-04-17T08:30:28.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.621+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af813e8e0 con 0x7f4af8160280 2024-04-17T08:30:28.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.622+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af8068a10 con 0x7f4af8160280 2024-04-17T08:30:28.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.623+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 88) ==== 90878+0+0 (secure 0 0 0) 0x7f4af0037070 con 0x7f4af8160280 2024-04-17T08:30:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.624+0000 7f4aefe00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 0x7f4ad406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.624+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6527+0+0 (secure 0 0 0) 0x7f4af00ada80 con 0x7f4af8160280 2024-04-17T08:30:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.625+0000 7f4af7e00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 0x7f4ad406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.626+0000 7f4af7e00640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 0x7f4ad406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80012e0 tx=0x7f4ae8007d50 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.629+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4af00305e0 con 0x7f4af8160280 2024-04-17T08:30:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.834+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4af80af730 con 0x7f4af8160280 2024-04-17T08:30:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.836+0000 7f4aefe00640 1 -- 172.21.15.67:0/2719600743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+17884 (secure 0 0 0) 0x7f4af007bec0 con 0x7f4af8160280 2024-04-17T08:30:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:28.836 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":169,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:26.935574+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.e","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6832/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/4100169880":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/529736597":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2208412755":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1092697670":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1418137156":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6833/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:6832/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/4218440392":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2263870823":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2708413279":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3224926034":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/791073325":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3654447672":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2286768646":"2024-04-18T07:34:28.771962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":13,"length":45},{"begin":69,"length":45}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.838+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 msgr2=0x7f4ad406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.838+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 0x7f4ad406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80012e0 tx=0x7f4ae8007d50 comp rx=0 tx=0).stop 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.838+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 msgr2=0x7f4af813e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.838+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097a0 tx=0x7f4af00046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 shutdown_connections 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4ad406c740 0x7f4ad406ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4af8160280 0x7f4af813e3a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4af8141ed0 0x7f4af813bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 --2- 172.21.15.67:0/2719600743 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4af806b8c0 0x7f4af80b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:28.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 >> 172.21.15.67:0/2719600743 conn(0x7f4af8067d10 msgr2=0x7f4af80cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:28.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 shutdown_connections 2024-04-17T08:30:28.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:28.839+0000 7f4afd400640 1 -- 172.21.15.67:0/2719600743 wait complete. 2024-04-17T08:30:28.858 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:30:28.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:29.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.086+0000 7fb21aa00640 1 Processor -- start 2024-04-17T08:30:29.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.087+0000 7fb21aa00640 1 -- start start 2024-04-17T08:30:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.088+0000 7fb21aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb214160280 0x7fb21406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.088+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb214141ed0 0x7fb214142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.088+0000 7fb21aa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2140cd4a0 con 0x7fb21406b8c0 2024-04-17T08:30:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.088+0000 7fb21aa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb21415ce30 con 0x7fb214160280 2024-04-17T08:30:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.088+0000 7fb21aa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2141607f0 con 0x7fb214141ed0 2024-04-17T08:30:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb214160280 0x7fb21406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb214160280 0x7fb21406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:33692/0 (socket says 172.21.15.67:33692) 2024-04-17T08:30:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb219600640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb21406b8c0 0x7fb2140cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:49746/0 (socket says 172.21.15.67:49746) 2024-04-17T08:30:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 -- 172.21.15.67:0/1732436430 learned_addr learned my addr 172.21.15.67:0/1732436430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb21a000640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb214141ed0 0x7fb214142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 -- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb214141ed0 msgr2=0x7fb214142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb214141ed0 0x7fb214142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 -- 172.21.15.67:0/1732436430 >> v1:172.21.15.187:6789/0 conn(0x7fb21406b8c0 legacy=0x7fb2140cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.089+0000 7fb218c00640 1 -- 172.21.15.67:0/1732436430 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb214160ad0 con 0x7fb214160280 2024-04-17T08:30:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.091+0000 7fb218c00640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb21406bca0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb200009870 tx=0x7fb20002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95ecec30135b3b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.091+0000 7fb213e00640 1 -- 172.21.15.67:0/1732436430 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200034070 con 0x7fb214160280 2024-04-17T08:30:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.091+0000 7fb213e00640 1 -- 172.21.15.67:0/1732436430 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb200030840 con 0x7fb214160280 2024-04-17T08:30:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.091+0000 7fb213e00640 1 -- 172.21.15.67:0/1732436430 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200030c10 con 0x7fb214160280 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 -- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 msgr2=0x7fb21406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb21406bca0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb200009870 tx=0x7fb20002b980 comp rx=0 tx=0).stop 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 -- 172.21.15.67:0/1732436430 shutdown_connections 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb214141ed0 0x7fb214142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 --2- 172.21.15.67:0/1732436430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb21406bca0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 -- 172.21.15.67:0/1732436430 >> 172.21.15.67:0/1732436430 conn(0x7fb214067d10 msgr2=0x7fb2140b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 -- 172.21.15.67:0/1732436430 shutdown_connections 2024-04-17T08:30:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.092+0000 7fb21aa00640 1 -- 172.21.15.67:0/1732436430 wait complete. 2024-04-17T08:30:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.094+0000 7fb21aa00640 1 Processor -- start 2024-04-17T08:30:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.094+0000 7fb21aa00640 1 -- start start 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb21406b8c0 0x7fb214155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb214143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb214161090 con 0x7fb214141ed0 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb21406c1e0 con 0x7fb214160280 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21aa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb21413f9b0 con 0x7fb21406b8c0 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb218c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb218c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56634/0 (socket says 172.21.15.67:56634) 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb218c00640 1 -- 172.21.15.67:0/775998562 learned_addr learned my addr 172.21.15.67:0/775998562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.095+0000 7fb21a000640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb214143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 -- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb21406b8c0 msgr2=0x7fb214155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb219600640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb21406b8c0 0x7fb214155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb21406b8c0 0x7fb214155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 -- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 msgr2=0x7fb214143dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb214143dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 -- 172.21.15.67:0/775998562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2141f66e0 con 0x7fb214141ed0 2024-04-17T08:30:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb218c00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb2000090b0 tx=0x7fb2000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200034040 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20003ada0 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200044610 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.096+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2141f69c0 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.097+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2140b4120 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.098+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb214068a10 con 0x7fb214141ed0 2024-04-17T08:30:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.098+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 88) ==== 90878+0+0 (secure 0 0 0) 0x7fb20004b060 con 0x7fb214141ed0 2024-04-17T08:30:29.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.099+0000 7fb211600640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 0x7fb1f406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.099+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6527+0+0 (secure 0 0 0) 0x7fb2000adb30 con 0x7fb214141ed0 2024-04-17T08:30:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.100+0000 7fb219600640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 0x7fb1f406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.101+0000 7fb219600640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 0x7fb1f406ec30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb208007fd0 tx=0x7fb208009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.104+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb20003f070 con 0x7fb214141ed0 2024-04-17T08:30:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.351+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2141f6bc0 con 0x7fb214141ed0 2024-04-17T08:30:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.352+0000 7fb211600640 1 -- 172.21.15.67:0/775998562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7fb20007bea0 con 0x7fb214141ed0 2024-04-17T08:30:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:29.351 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T08:30:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 msgr2=0x7fb1f406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 0x7fb1f406ec30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb208007fd0 tx=0x7fb208009040 comp rx=0 tx=0).stop 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 msgr2=0x7fb2141f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fb2000090b0 tx=0x7fb2000046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 shutdown_connections 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb1f406c770 0x7fb1f406ec30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb214160280 0x7fb214143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb214141ed0 0x7fb2141f5d70 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 --2- 172.21.15.67:0/775998562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb21406b8c0 0x7fb214155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.355+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 >> 172.21.15.67:0/775998562 conn(0x7fb214067d10 msgr2=0x7fb2140d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:29.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.356+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 shutdown_connections 2024-04-17T08:30:29.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.356+0000 7fb21aa00640 1 -- 172.21.15.67:0/775998562 wait complete. 2024-04-17T08:30:29.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:29.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.591+0000 7f2a19600640 1 Processor -- start 2024-04-17T08:30:29.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.591+0000 7f2a19600640 1 -- start start 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a19600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2a1415ee60 0x7f2a140af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a19600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 0x7f2a140ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a19600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a1415f440 con 0x7f2a1415c5a0 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a19600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2a140b4920 con 0x7f2a1415ee60 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a19600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2a140afdd0 con 0x7f2a140af980 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a13e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a1415c5a0 0x7f2a140b4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46870/0 (socket says 172.21.15.67:46870) 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a13e00640 1 -- 172.21.15.67:0/4286832948 learned_addr learned my addr 172.21.15.67:0/4286832948 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.593+0000 7f2a13400640 1 --2- 172.21.15.67:0/4286832948 >> v2:172.21.15.67:3300/0 conn(0x7f2a1415ee60 0x7f2a140af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a12a00640 1 -- 172.21.15.67:0/4286832948 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156441266 0 0) 0x7f2a1415f440 con 0x7f2a1415c5a0 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a18c00640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 0x7f2a140ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a12a00640 1 -- 172.21.15.67:0/4286832948 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f8003520 con 0x7f2a1415c5a0 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a13400640 1 -- 172.21.15.67:0/4286832948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 msgr2=0x7f2a140ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a13400640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 0x7f2a140ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a13400640 1 -- 172.21.15.67:0/4286832948 >> v1:172.21.15.67:6789/0 conn(0x7f2a1415c5a0 legacy=0x7f2a140b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a13400640 1 -- 172.21.15.67:0/4286832948 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1415f720 con 0x7f2a1415ee60 2024-04-17T08:30:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.594+0000 7f2a18c00640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 0x7f2a140ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:29.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.595+0000 7f2a13400640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140af440 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1af150d94655590 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.595+0000 7f2a12a00640 1 -- 172.21.15.67:0/4286832948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00039070 con 0x7f2a1415ee60 2024-04-17T08:30:29.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.595+0000 7f2a12a00640 1 -- 172.21.15.67:0/4286832948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a000316f0 con 0x7f2a1415ee60 2024-04-17T08:30:29.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.595+0000 7f2a12a00640 1 -- 172.21.15.67:0/4286832948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00031ac0 con 0x7f2a1415ee60 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 -- 172.21.15.67:0/4286832948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 msgr2=0x7f2a140af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140af440 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b790 comp rx=0 tx=0).stop 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 -- 172.21.15.67:0/4286832948 shutdown_connections 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a140af980 0x7f2a140ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 --2- 172.21.15.67:0/4286832948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140af440 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 -- 172.21.15.67:0/4286832948 >> 172.21.15.67:0/4286832948 conn(0x7f2a14067d10 msgr2=0x7f2a140b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 -- 172.21.15.67:0/4286832948 shutdown_connections 2024-04-17T08:30:29.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.596+0000 7f2a19600640 1 -- 172.21.15.67:0/4286832948 wait complete. 2024-04-17T08:30:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.598+0000 7f2a19600640 1 Processor -- start 2024-04-17T08:30:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.598+0000 7f2a19600640 1 -- start start 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a140af980 0x7f2a14158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 0x7f2a141f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a140adab0 con 0x7f2a1415ee60 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1415fce0 con 0x7f2a140af980 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a19600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2a140adc30 con 0x7f2a1415c5a0 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a18c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a18c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56646/0 (socket says 172.21.15.67:56646) 2024-04-17T08:30:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.599+0000 7f2a18c00640 1 -- 172.21.15.67:0/1294950160 learned_addr learned my addr 172.21.15.67:0/1294950160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a13400640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 0x7f2a141f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a13e00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a140af980 0x7f2a14158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 -- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 msgr2=0x7f2a141f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 0x7f2a141f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 -- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a140af980 msgr2=0x7f2a14158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a140af980 0x7f2a14158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 -- 172.21.15.67:0/1294950160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a141f48d0 con 0x7f2a1415ee60 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a13400640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 0x7f2a141f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a18c00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00cf20 tx=0x7f2a0c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c01a070 con 0x7f2a1415ee60 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a0c00d840 con 0x7f2a1415ee60 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00dc30 con 0x7f2a1415ee60 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.600+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a141f4bb0 con 0x7f2a1415ee60 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.601+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a141585c0 con 0x7f2a1415ee60 2024-04-17T08:30:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.602+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a14068a10 con 0x7f2a1415ee60 2024-04-17T08:30:29.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.603+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 89) ==== 91086+0+0 (secure 0 0 0) 0x7f2a0c01d070 con 0x7f2a1415ee60 2024-04-17T08:30:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.604+0000 7f2a0be00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 0x7f29f406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.604+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6527+0+0 (secure 0 0 0) 0x7f2a0c08c840 con 0x7f2a1415ee60 2024-04-17T08:30:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.604+0000 7f2a13e00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 0x7f29f406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.606+0000 7f2a13e00640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 0x7f29f406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a04000f60 tx=0x7f2a04009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.608+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2a0c05a8e0 con 0x7f2a1415ee60 2024-04-17T08:30:29.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.860+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a140cfd20 con 0x7f2a1415ee60 2024-04-17T08:30:29.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.861+0000 7f2a0be00640 1 -- 172.21.15.67:0/1294950160 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5201 (secure 0 0 0) 0x7f2a0c05a1e0 con 0x7f2a1415ee60 2024-04-17T08:30:29.860 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:29.860 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T07:34:13.943539+0000","modified":"2024-04-17T07:34:17.018032+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2582506112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2582506112},{"type":"v1","addr":"172.21.15.67:6835","nonce":2582506112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":9}]} 2024-04-17T08:30:29.861 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 76 2024-04-17T08:30:29.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 msgr2=0x7f29f406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 0x7f29f406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a04000f60 tx=0x7f2a04009040 comp rx=0 tx=0).stop 2024-04-17T08:30:29.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 msgr2=0x7f2a140b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:29.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00cf20 tx=0x7f2a0c00a7d0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 shutdown_connections 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f29f406c980 0x7f29f406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a1415ee60 0x7f2a140b0f60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2a1415c5a0 0x7f2a141f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 --2- 172.21.15.67:0/1294950160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2a140af980 0x7f2a14158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 >> 172.21.15.67:0/1294950160 conn(0x7f2a14067d10 msgr2=0x7f2a140adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 shutdown_connections 2024-04-17T08:30:29.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:29.864+0000 7f2a19600640 1 -- 172.21.15.67:0/1294950160 wait complete. 2024-04-17T08:30:29.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:30.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.100+0000 7f42e6a00640 1 Processor -- start 2024-04-17T08:30:30.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.100+0000 7f42e6a00640 1 -- start start 2024-04-17T08:30:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f42e015ac80 0x7f42e006f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42e0072eb0 0x7f42e0071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e00cd920 con 0x7f42e0072330 2024-04-17T08:30:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f42e00727a0 con 0x7f42e0072eb0 2024-04-17T08:30:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f42e00cdc30 con 0x7f42e015ac80 2024-04-17T08:30:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f42e0072330 0x7f42e006ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46878/0 (socket says 172.21.15.67:46878) 2024-04-17T08:30:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42e0072eb0 0x7f42e0071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.102+0000 7f42e6000640 1 -- 172.21.15.67:0/385273059 learned_addr learned my addr 172.21.15.67:0/385273059 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.103+0000 7f42e5600640 1 --2- 172.21.15.67:0/385273059 >> v2:172.21.15.187:3300/0 conn(0x7f42e015ac80 0x7f42e006f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.103+0000 7f42e4c00640 1 -- 172.21.15.67:0/385273059 >> v2:172.21.15.187:3300/0 conn(0x7f42e015ac80 msgr2=0x7f42e006f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.103+0000 7f42e4c00640 1 --2- 172.21.15.67:0/385273059 >> v2:172.21.15.187:3300/0 conn(0x7f42e015ac80 0x7f42e006f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.103+0000 7f42e4c00640 1 -- 172.21.15.67:0/385273059 >> v1:172.21.15.67:6789/0 conn(0x7f42e0072330 legacy=0x7f42e006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.103+0000 7f42e4c00640 1 -- 172.21.15.67:0/385273059 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e006f980 con 0x7f42e0072eb0 2024-04-17T08:30:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.104+0000 7f42e4c00640 1 --2- 172.21.15.67:0/385273059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e0071df0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f42e006c9f0 tx=0x7f42cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d54b45bd2c79655 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.104+0000 7f42dfe00640 1 -- 172.21.15.67:0/385273059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0315a0 con 0x7f42e0072eb0 2024-04-17T08:30:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.104+0000 7f42dfe00640 1 -- 172.21.15.67:0/385273059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42cc037070 con 0x7f42e0072eb0 2024-04-17T08:30:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.104+0000 7f42dfe00640 1 -- 172.21.15.67:0/385273059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc031960 con 0x7f42e0072eb0 2024-04-17T08:30:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 -- 172.21.15.67:0/385273059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 msgr2=0x7f42e0071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 --2- 172.21.15.67:0/385273059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e0071df0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f42e006c9f0 tx=0x7f42cc02b840 comp rx=0 tx=0).stop 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 -- 172.21.15.67:0/385273059 shutdown_connections 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 --2- 172.21.15.67:0/385273059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e0071df0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 --2- 172.21.15.67:0/385273059 >> v2:172.21.15.187:3300/0 conn(0x7f42e015ac80 0x7f42e006f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 -- 172.21.15.67:0/385273059 >> 172.21.15.67:0/385273059 conn(0x7f42e0067d10 msgr2=0x7f42e00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 -- 172.21.15.67:0/385273059 shutdown_connections 2024-04-17T08:30:30.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.105+0000 7f42e6a00640 1 -- 172.21.15.67:0/385273059 wait complete. 2024-04-17T08:30:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.107+0000 7f42e6a00640 1 Processor -- start 2024-04-17T08:30:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.107+0000 7f42e6a00640 1 -- start start 2024-04-17T08:30:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 0x7f42e0159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42e015ac80 0x7f42e01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42e015b060 con 0x7f42e0072eb0 2024-04-17T08:30:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f42e006ff40 con 0x7f42e0072330 2024-04-17T08:30:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e6a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f42e0157590 con 0x7f42e015ac80 2024-04-17T08:30:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56672/0 (socket says 172.21.15.67:56672) 2024-04-17T08:30:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.108+0000 7f42e4c00640 1 -- 172.21.15.67:0/741159135 learned_addr learned my addr 172.21.15.67:0/741159135 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e5600640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 0x7f42e0159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 -- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42e015ac80 msgr2=0x7f42e01553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e6000640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42e015ac80 0x7f42e01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42e015ac80 0x7f42e01553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 -- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 msgr2=0x7f42e0159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 0x7f42e0159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 -- 172.21.15.67:0/741159135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e0156980 con 0x7f42e0072eb0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e5600640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 0x7f42e0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e4c00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000c00 tx=0x7f42cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc032840 con 0x7f42e0072eb0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42cc032a00 con 0x7f42e0072eb0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc032df0 con 0x7f42e0072eb0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.109+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e0156c60 con 0x7f42e0072eb0 2024-04-17T08:30:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.110+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e014bc30 con 0x7f42e0072eb0 2024-04-17T08:30:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.111+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e0157050 con 0x7f42e0072eb0 2024-04-17T08:30:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.112+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 89) ==== 91086+0+0 (secure 0 0 0) 0x7f42cc04d3f0 con 0x7f42e0072eb0 2024-04-17T08:30:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.113+0000 7f42dd600640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 0x7f42c006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.113+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6527+0+0 (secure 0 0 0) 0x7f42cc02f030 con 0x7f42e0072eb0 2024-04-17T08:30:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.113+0000 7f42e5600640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 0x7f42c006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.115+0000 7f42e5600640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 0x7f42c006ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42d4006fd0 tx=0x7f42d4008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.117+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f42cc040070 con 0x7f42e0072eb0 2024-04-17T08:30:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.327+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42e0156e30 con 0x7f42e0072eb0 2024-04-17T08:30:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.330+0000 7f42dd600640 1 -- 172.21.15.67:0/741159135 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) ==== 74+0+17883 (secure 0 0 0) 0x7f42cc07bcb0 con 0x7f42e0072eb0 2024-04-17T08:30:30.330 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:30.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":170,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:29.391836+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-04-17T07:34:11.829435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-04-17T07:34:12.928331+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.e","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6832/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/4100169880":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/529736597":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2208412755":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1092697670":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1418137156":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6833/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:6832/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/4218440392":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2263870823":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2708413279":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3224926034":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/791073325":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3654447672":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2286768646":"2024-04-18T07:34:28.771962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":13,"length":45},{"begin":69,"length":45}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 msgr2=0x7f42c006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 0x7f42c006ee70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42d4006fd0 tx=0x7f42d4008040 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 msgr2=0x7f42e014d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000c00 tx=0x7f42cc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 shutdown_connections 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42c006c9b0 0x7f42c006ee70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42e015ac80 0x7f42e01553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e0072eb0 0x7f42e014d230 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 --2- 172.21.15.67:0/741159135 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42e0072330 0x7f42e0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 >> 172.21.15.67:0/741159135 conn(0x7f42e0067d10 msgr2=0x7f42e015d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:30.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.333+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 shutdown_connections 2024-04-17T08:30:30.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.334+0000 7f42e6a00640 1 -- 172.21.15.67:0/741159135 wait complete. 2024-04-17T08:30:30.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:30:30.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.570+0000 7f7ad4c00640 1 Processor -- start 2024-04-17T08:30:30.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.571+0000 7f7ad4c00640 1 -- start start 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.572+0000 7f7ad4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ad00cbc00 0x7f7ad00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.572+0000 7f7ad4c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 0x7f7ad0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.572+0000 7f7ad4c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ad015d060 con 0x7f7ad00723a0 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.572+0000 7f7ad4c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ad00cfe90 con 0x7f7ad00cbc00 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.572+0000 7f7ad4c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7ad015dc70 con 0x7f7ad00b63c0 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ad00cbc00 0x7f7ad00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ad00cbc00 0x7f7ad00cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56684/0 (socket says 172.21.15.67:56684) 2024-04-17T08:30:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 -- 172.21.15.67:0/3572137025 learned_addr learned my addr 172.21.15.67:0/3572137025 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 -- 172.21.15.67:0/3572137025 >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 msgr2=0x7f7ad0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7ac7e00640 1 --2- 172.21.15.67:0/3572137025 >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 0x7f7ad0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 --2- 172.21.15.67:0/3572137025 >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 0x7f7ad0071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 -- 172.21.15.67:0/3572137025 >> v1:172.21.15.187:6789/0 conn(0x7f7ad00723a0 legacy=0x7f7ad0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7acf400640 1 -- 172.21.15.67:0/3572137025 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad015d340 con 0x7f7ad00cbc00 2024-04-17T08:30:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.573+0000 7f7ac7e00640 1 --2- 172.21.15.67:0/3572137025 >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 0x7f7ad0071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.574+0000 7f7acf400640 1 --2- 172.21.15.67:0/3572137025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00cbc00 0x7f7ad00cf950 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00caee0 tx=0x7f7ac002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f7ac4bf4c413d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.575+0000 7f7acea00640 1 -- 172.21.15.67:0/3572137025 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0035020 con 0x7f7ad00cbc00 2024-04-17T08:30:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.575+0000 7f7acea00640 1 -- 172.21.15.67:0/3572137025 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac0031870 con 0x7f7ad00cbc00 2024-04-17T08:30:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.575+0000 7f7acea00640 1 -- 172.21.15.67:0/3572137025 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0031c40 con 0x7f7ad00cbc00 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.575+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3572137025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00cbc00 msgr2=0x7f7ad00cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.575+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3572137025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00cbc00 0x7f7ad00cf950 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00caee0 tx=0x7f7ac002b910 comp rx=0 tx=0).stop 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3572137025 shutdown_connections 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3572137025 >> v2:172.21.15.187:3300/0 conn(0x7f7ad00b63c0 0x7f7ad0071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3572137025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00cbc00 0x7f7ad00cf950 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3572137025 >> 172.21.15.67:0/3572137025 conn(0x7f7ad006bef0 msgr2=0x7f7ad00b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3572137025 shutdown_connections 2024-04-17T08:30:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.576+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3572137025 wait complete. 2024-04-17T08:30:30.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.578+0000 7f7ad4c00640 1 Processor -- start 2024-04-17T08:30:30.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.578+0000 7f7ad4c00640 1 -- start start 2024-04-17T08:30:30.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ad00723a0 0x7f7ad0145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ad00cbc00 0x7f7ad0148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad00d02d0 con 0x7f7ad00b63c0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad015d900 con 0x7f7ad00723a0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ad4c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad00cbfe0 con 0x7f7ad00cbc00 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56698/0 (socket says 172.21.15.67:56698) 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7ac7e00640 1 -- 172.21.15.67:0/3457979774 learned_addr learned my addr 172.21.15.67:0/3457979774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7acf400640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ad00723a0 0x7f7ad0145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.579+0000 7f7acfe00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ad00cbc00 0x7f7ad0148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 -- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ad00cbc00 msgr2=0x7f7ad0148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ad00cbc00 0x7f7ad0148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 -- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ad00723a0 msgr2=0x7f7ad0145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ad00723a0 0x7f7ad0145e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 -- 172.21.15.67:0/3457979774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad01f4d50 con 0x7f7ad00b63c0 2024-04-17T08:30:30.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7e00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c7f0 tx=0x7f7ab800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8015070 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab8007a30 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad01463b0 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8007e00 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.580+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad01543b0 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.581+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad0068a10 con 0x7f7ad00b63c0 2024-04-17T08:30:30.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.582+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 89) ==== 91086+0+0 (secure 0 0 0) 0x7f7ab8018070 con 0x7f7ad00b63c0 2024-04-17T08:30:30.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.583+0000 7f7ac7400640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 0x7f7aac06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:30.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.584+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6527+0+0 (secure 0 0 0) 0x7f7ab808bfc0 con 0x7f7ad00b63c0 2024-04-17T08:30:30.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.584+0000 7f7acf400640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 0x7f7aac06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:30.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.585+0000 7f7acf400640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 0x7f7aac06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0036000 tx=0x7f7ac003c040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.588+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7ab8059f50 con 0x7f7ad00b63c0 2024-04-17T08:30:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.841+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7ad00cb4a0 con 0x7f7ad00b63c0 2024-04-17T08:30:30.842 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:30:30.843+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:30:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.847+0000 7f7ac7400640 1 -- 172.21.15.67:0/3457979774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v77) ==== 161+0+0 (secure 0 0 0) 0x7f7ab8059850 con 0x7f7ad00b63c0 2024-04-17T08:30:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:30:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 msgr2=0x7f7aac06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 0x7f7aac06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0036000 tx=0x7f7ac003c040 comp rx=0 tx=0).stop 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 msgr2=0x7f7ad0151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c7f0 tx=0x7f7ab800cbf0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 shutdown_connections 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7aac06c9b0 0x7f7aac06ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ad00cbc00 0x7f7ad0148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ad00b63c0 0x7f7ad0151ee0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 --2- 172.21.15.67:0/3457979774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ad00723a0 0x7f7ad0145e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.851+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 >> 172.21.15.67:0/3457979774 conn(0x7f7ad006bef0 msgr2=0x7f7ad00ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.852+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 shutdown_connections 2024-04-17T08:30:30.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:30.852+0000 7f7ad4c00640 1 -- 172.21.15.67:0/3457979774 wait complete. 2024-04-17T08:30:30.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:30:30.944 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:30:31.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.080+0000 7fe2e3e00640 1 Processor -- start 2024-04-17T08:30:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.081+0000 7fe2e3e00640 1 -- start start 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.082+0000 7fe2e3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2e415b6b0 0x7fe2e406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.082+0000 7fe2e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe2e40cfc10 0x7fe2e406be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.082+0000 7fe2e3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e406df40 con 0x7fe2e406f800 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.082+0000 7fe2e3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e406ed00 con 0x7fe2e40cfc10 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.082+0000 7fe2e3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e406e870 con 0x7fe2e415b6b0 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe2e40cfc10 0x7fe2e406be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2dbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe2e406f800 0x7fe2e415f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46886/0 (socket says 172.21.15.67:46886) 2024-04-17T08:30:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2dbe00640 1 -- 172.21.15.67:0/3659442328 learned_addr learned my addr 172.21.15.67:0/3659442328 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e2a00640 1 --2- 172.21.15.67:0/3659442328 >> v2:172.21.15.187:3300/0 conn(0x7fe2e415b6b0 0x7fe2e406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e3400640 1 -- 172.21.15.67:0/3659442328 >> v2:172.21.15.187:3300/0 conn(0x7fe2e415b6b0 msgr2=0x7fe2e406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e3400640 1 --2- 172.21.15.67:0/3659442328 >> v2:172.21.15.187:3300/0 conn(0x7fe2e415b6b0 0x7fe2e406b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e3400640 1 -- 172.21.15.67:0/3659442328 >> v1:172.21.15.67:6789/0 conn(0x7fe2e406f800 legacy=0x7fe2e415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e3400640 1 -- 172.21.15.67:0/3659442328 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e406e220 con 0x7fe2e40cfc10 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.083+0000 7fe2e2000640 1 -- 172.21.15.67:0/3659442328 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931614416 0 0) 0x7fe2e406df40 con 0x7fe2e406f800 2024-04-17T08:30:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.084+0000 7fe2e3400640 1 --2- 172.21.15.67:0/3659442328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e40cfc10 0x7fe2e406be00 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4072ea0 tx=0x7fe2dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73aa7a6fac1b469 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e2000640 1 -- 172.21.15.67:0/3659442328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc039070 con 0x7fe2e40cfc10 2024-04-17T08:30:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e2000640 1 -- 172.21.15.67:0/3659442328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc02cbf0 con 0x7fe2e40cfc10 2024-04-17T08:30:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e2000640 1 -- 172.21.15.67:0/3659442328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc034440 con 0x7fe2e40cfc10 2024-04-17T08:30:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 -- 172.21.15.67:0/3659442328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e40cfc10 msgr2=0x7fe2e406be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/3659442328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e40cfc10 0x7fe2e406be00 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4072ea0 tx=0x7fe2dc02be40 comp rx=0 tx=0).stop 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 -- 172.21.15.67:0/3659442328 shutdown_connections 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/3659442328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e40cfc10 0x7fe2e406be00 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/3659442328 >> v2:172.21.15.187:3300/0 conn(0x7fe2e415b6b0 0x7fe2e406b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.085+0000 7fe2e3e00640 1 -- 172.21.15.67:0/3659442328 >> 172.21.15.67:0/3659442328 conn(0x7fe2e40b2190 msgr2=0x7fe2e40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.086+0000 7fe2e3e00640 1 -- 172.21.15.67:0/3659442328 shutdown_connections 2024-04-17T08:30:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.086+0000 7fe2e3e00640 1 -- 172.21.15.67:0/3659442328 wait complete. 2024-04-17T08:30:31.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.087+0000 7fe2e3e00640 1 Processor -- start 2024-04-17T08:30:31.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.087+0000 7fe2e3e00640 1 -- start start 2024-04-17T08:30:31.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2e40cfc10 0x7fe2e415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2e415b6b0 0x7fe2e415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e40afee0 con 0x7fe2e406f800 2024-04-17T08:30:31.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e40afd60 con 0x7fe2e415b6b0 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.088+0000 7fe2e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e40b44a0 con 0x7fe2e40cfc10 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56734/0 (socket says 172.21.15.67:56734) 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 -- 172.21.15.67:0/505342847 learned_addr learned my addr 172.21.15.67:0/505342847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 -- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2e40cfc10 msgr2=0x7fe2e415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2e40cfc10 0x7fe2e415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 -- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2e415b6b0 msgr2=0x7fe2e415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2e415b6b0 0x7fe2e415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 -- 172.21.15.67:0/505342847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e40ae480 con 0x7fe2e406f800 2024-04-17T08:30:31.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.089+0000 7fe2e2a00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400d800 tx=0x7fe2d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:31.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.090+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d400b6c0 con 0x7fe2e406f800 2024-04-17T08:30:31.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.090+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d4019070 con 0x7fe2e406f800 2024-04-17T08:30:31.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.090+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d400ba80 con 0x7fe2e406f800 2024-04-17T08:30:31.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.090+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e40b1300 con 0x7fe2e406f800 2024-04-17T08:30:31.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.090+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e40aea00 con 0x7fe2e406f800 2024-04-17T08:30:31.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.091+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e4068a10 con 0x7fe2e406f800 2024-04-17T08:30:31.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.092+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 89) ==== 91086+0+0 (secure 0 0 0) 0x7fe2d400bc20 con 0x7fe2e406f800 2024-04-17T08:30:31.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.093+0000 7fe2db400640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 0x7fe2c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:31.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.094+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 6569+0+0 (secure 0 0 0) 0x7fe2d408c440 con 0x7fe2e406f800 2024-04-17T08:30:31.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.094+0000 7fe2dbe00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 0x7fe2c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:31.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.095+0000 7fe2dbe00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 0x7fe2c006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007fd0 tx=0x7fe2cc007960 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.098+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe2d405a4b0 con 0x7fe2e406f800 2024-04-17T08:30:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.349+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe2e406ee80 con 0x7fe2e406f800 2024-04-17T08:30:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.851+0000 7fe2db400640 1 -- 172.21.15.67:0/505342847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v78) ==== 104+0+0 (secure 0 0 0) 0x7fe2d4059db0 con 0x7fe2e406f800 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 msgr2=0x7fe2c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 0x7fe2c006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc007fd0 tx=0x7fe2cc007960 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 msgr2=0x7fe2e4151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400d800 tx=0x7fe2d400dcc0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 shutdown_connections 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe2c006c980 0x7fe2c006ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2e415b6b0 0x7fe2e415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2e40cfc10 0x7fe2e415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 --2- 172.21.15.67:0/505342847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2e406f800 0x7fe2e4151ef0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.855+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 >> 172.21.15.67:0/505342847 conn(0x7fe2e40b2190 msgr2=0x7fe2e40b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.856+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 shutdown_connections 2024-04-17T08:30:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:31.856+0000 7fe2e3e00640 1 -- 172.21.15.67:0/505342847 wait complete. 2024-04-17T08:30:31.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:30:32.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.110+0000 7f9b17600640 1 Processor -- start 2024-04-17T08:30:32.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.111+0000 7f9b17600640 1 -- start start 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b17600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b100cfe90 0x7f9b1006e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b17600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1015d210 con 0x7f9b100cd940 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b17600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1015d4f0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b17600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9b1015d7d0 con 0x7f9b100cfe90 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b0fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b100cd940 0x7f9b1006b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46896/0 (socket says 172.21.15.67:46896) 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b100cfe90 0x7f9b1006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.113+0000 7f9b0fe00640 1 -- 172.21.15.67:0/4091441965 learned_addr learned my addr 172.21.15.67:0/4091441965 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:32.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.114+0000 7f9b14c00640 1 -- 172.21.15.67:0/4091441965 >> v1:172.21.15.187:6789/0 conn(0x7f9b1006bfd0 legacy=0x7f9b100722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:32.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.114+0000 7f9b14c00640 1 -- 172.21.15.67:0/4091441965 >> v1:172.21.15.67:6789/0 conn(0x7f9b100cd940 legacy=0x7f9b1006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.114+0000 7f9b14c00640 1 -- 172.21.15.67:0/4091441965 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1015dab0 con 0x7f9b100cfe90 2024-04-17T08:30:32.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.115+0000 7f9b14c00640 1 --2- 172.21.15.67:0/4091441965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b100cfe90 0x7f9b1006e710 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9b000097a0 tx=0x7f9b0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58fe1eedc3807a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:32.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.115+0000 7f9b0f400640 1 -- 172.21.15.67:0/4091441965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00035020 con 0x7f9b100cfe90 2024-04-17T08:30:32.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.115+0000 7f9b0f400640 1 -- 172.21.15.67:0/4091441965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b00031830 con 0x7f9b100cfe90 2024-04-17T08:30:32.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.115+0000 7f9b0f400640 1 -- 172.21.15.67:0/4091441965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00031c00 con 0x7f9b100cfe90 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 -- 172.21.15.67:0/4091441965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b100cfe90 msgr2=0x7f9b1006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 --2- 172.21.15.67:0/4091441965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b100cfe90 0x7f9b1006e710 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9b000097a0 tx=0x7f9b0002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 -- 172.21.15.67:0/4091441965 shutdown_connections 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 --2- 172.21.15.67:0/4091441965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b100cfe90 0x7f9b1006e710 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 -- 172.21.15.67:0/4091441965 >> 172.21.15.67:0/4091441965 conn(0x7f9b100b2370 msgr2=0x7f9b100b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 -- 172.21.15.67:0/4091441965 shutdown_connections 2024-04-17T08:30:32.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.116+0000 7f9b17600640 1 -- 172.21.15.67:0/4091441965 wait complete. 2024-04-17T08:30:32.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.118+0000 7f9b17600640 1 Processor -- start 2024-04-17T08:30:32.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.118+0000 7f9b17600640 1 -- start start 2024-04-17T08:30:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b100cd940 0x7f9b101e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b100cfe90 0x7f9b100ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1015ce90 con 0x7f9b1006bfd0 2024-04-17T08:30:32.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9b100d0270 con 0x7f9b100cfe90 2024-04-17T08:30:32.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b17600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b100cddd0 con 0x7f9b100cd940 2024-04-17T08:30:32.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:32.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56752/0 (socket says 172.21.15.67:56752) 2024-04-17T08:30:32.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.119+0000 7f9b14c00640 1 -- 172.21.15.67:0/2397493066 learned_addr learned my addr 172.21.15.67:0/2397493066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 -- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b100cd940 msgr2=0x7f9b101e6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b100cd940 0x7f9b101e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 -- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b100cfe90 msgr2=0x7f9b100ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b100cfe90 0x7f9b100ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 -- 172.21.15.67:0/2397493066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b101429e0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b14c00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9b000098c0 tx=0x7f9b000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0003aa70 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0003ac10 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b10142cc0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.120+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b00044400 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.121+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b10143280 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.122+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b10073980 con 0x7f9b1006bfd0 2024-04-17T08:30:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.123+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f9b00038020 con 0x7f9b1006bfd0 2024-04-17T08:30:32.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.124+0000 7f9b0cc00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 0x7f9aec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:32.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.124+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 6569+0+0 (secure 0 0 0) 0x7f9b000adfe0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.124+0000 7f9b0fe00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 0x7f9aec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:32.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.126+0000 7f9b0fe00640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 0x7f9aec06ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9af80012e0 tx=0x7f9af8001630 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:32.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.128+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9b0003f070 con 0x7f9b1006bfd0 2024-04-17T08:30:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.336+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9b1006e2c0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.866+0000 7f9b0cc00640 1 -- 172.21.15.67:0/2397493066 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v172) ==== 181+0+0 (secure 0 0 0) 0x7f9b0002f2e0 con 0x7f9b1006bfd0 2024-04-17T08:30:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 msgr2=0x7f9aec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 0x7f9aec06ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9af80012e0 tx=0x7f9af8001630 comp rx=0 tx=0).stop 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 msgr2=0x7f9b1015bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9b000098c0 tx=0x7f9b000046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 shutdown_connections 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9aec06ca40 0x7f9aec06ef00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b100cfe90 0x7f9b100ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b100cd940 0x7f9b101e6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 --2- 172.21.15.67:0/2397493066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b1006bfd0 0x7f9b1015bff0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.870+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 >> 172.21.15.67:0/2397493066 conn(0x7f9b100b2370 msgr2=0x7f9b10144d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.871+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 shutdown_connections 2024-04-17T08:30:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:32.871+0000 7f9b17600640 1 -- 172.21.15.67:0/2397493066 wait complete. 2024-04-17T08:30:32.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:30:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.123+0000 7f230f000640 1 Processor -- start 2024-04-17T08:30:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.124+0000 7f230f000640 1 -- start start 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23080ce510 0x7f23080b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230f000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23080721c0 con 0x7f2308162f80 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230f000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23080724a0 con 0x7f23080cd750 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2308072780 con 0x7f23080ce510 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2308162f80 0x7f23080cd040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46910/0 (socket says 172.21.15.67:46910) 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230dc00640 1 -- 172.21.15.67:0/2396179906 learned_addr learned my addr 172.21.15.67:0/2396179906 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.125+0000 7f230e600640 1 --2- 172.21.15.67:0/2396179906 >> v2:172.21.15.67:3300/0 conn(0x7f23080ce510 0x7f23080b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.126+0000 7f2307e00640 1 -- 172.21.15.67:0/2396179906 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262658970 0 0) 0x7f23080721c0 con 0x7f2308162f80 2024-04-17T08:30:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.126+0000 7f2307e00640 1 -- 172.21.15.67:0/2396179906 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22f0003520 con 0x7f2308162f80 2024-04-17T08:30:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.126+0000 7f230e600640 1 -- 172.21.15.67:0/2396179906 >> v1:172.21.15.187:6789/0 conn(0x7f23080cd750 legacy=0x7f23080b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.126+0000 7f230e600640 1 -- 172.21.15.67:0/2396179906 >> v1:172.21.15.67:6789/0 conn(0x7f2308162f80 legacy=0x7f23080cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.126+0000 7f230e600640 1 -- 172.21.15.67:0/2396179906 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2308072a60 con 0x7f23080ce510 2024-04-17T08:30:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.127+0000 7f230e600640 1 --2- 172.21.15.67:0/2396179906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23080ce510 0x7f23080b4a90 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f2300009d90 tx=0x7f230002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4320f71b46292fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.127+0000 7f2307e00640 1 -- 172.21.15.67:0/2396179906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300039070 con 0x7f23080ce510 2024-04-17T08:30:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.127+0000 7f2307e00640 1 -- 172.21.15.67:0/2396179906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230002cc60 con 0x7f23080ce510 2024-04-17T08:30:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.127+0000 7f2307e00640 1 -- 172.21.15.67:0/2396179906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300034660 con 0x7f23080ce510 2024-04-17T08:30:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 -- 172.21.15.67:0/2396179906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23080ce510 msgr2=0x7f23080b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 --2- 172.21.15.67:0/2396179906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23080ce510 0x7f23080b4a90 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f2300009d90 tx=0x7f230002beb0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 -- 172.21.15.67:0/2396179906 shutdown_connections 2024-04-17T08:30:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 --2- 172.21.15.67:0/2396179906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23080ce510 0x7f23080b4a90 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 -- 172.21.15.67:0/2396179906 >> 172.21.15.67:0/2396179906 conn(0x7f230806b8c0 msgr2=0x7f230806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 -- 172.21.15.67:0/2396179906 shutdown_connections 2024-04-17T08:30:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.128+0000 7f230f000640 1 -- 172.21.15.67:0/2396179906 wait complete. 2024-04-17T08:30:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.130+0000 7f230f000640 1 Processor -- start 2024-04-17T08:30:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.130+0000 7f230f000640 1 -- start start 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 0x7f230815ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23080ce510 0x7f230815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23080d0260 con 0x7f2308162f80 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f23080d00e0 con 0x7f23080cd750 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23080d03e0 con 0x7f23080ce510 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56778/0 (socket says 172.21.15.67:56778) 2024-04-17T08:30:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.131+0000 7f230e600640 1 -- 172.21.15.67:0/1285775543 learned_addr learned my addr 172.21.15.67:0/1285775543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 -- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23080ce510 msgr2=0x7f230815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23080ce510 0x7f230815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 -- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 msgr2=0x7f230815ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230dc00640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 0x7f230815ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 0x7f230815ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 -- 172.21.15.67:0/1285775543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230814cb10 con 0x7f2308162f80 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230dc00640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 0x7f230815ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f230e600640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f2300009d90 tx=0x7f2300004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300039050 con 0x7f2308162f80 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.132+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2300045070 con 0x7f2308162f80 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.133+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300009450 con 0x7f2308162f80 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.133+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230815d480 con 0x7f2308162f80 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.133+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230815dcc0 con 0x7f2308162f80 2024-04-17T08:30:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.134+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308068a10 con 0x7f2308162f80 2024-04-17T08:30:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.135+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f230003e850 con 0x7f2308162f80 2024-04-17T08:30:33.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.136+0000 7f2305600640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:33.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.136+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 6131+0+0 (secure 0 0 0) 0x7f23000aeb00 con 0x7f2308162f80 2024-04-17T08:30:33.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.136+0000 7f230dc00640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:33.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.138+0000 7f230dc00640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 0x7f22e806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22fc005880 tx=0x7f22fc009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:33.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.140+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f230815d480 con 0x7f2308162f80 2024-04-17T08:30:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.348+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2308068610 con 0x7f2308162f80 2024-04-17T08:30:33.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.877+0000 7f2305600640 1 -- 172.21.15.67:0/1285775543 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v173) ==== 169+0+0 (secure 0 0 0) 0x7f2300032e20 con 0x7f2308162f80 2024-04-17T08:30:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.880+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 msgr2=0x7f22e806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.880+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 0x7f22e806eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22fc005880 tx=0x7f22fc009040 comp rx=0 tx=0).stop 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.880+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 msgr2=0x7f230814eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.880+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f2300009d90 tx=0x7f2300004a40 comp rx=0 tx=0).stop 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 shutdown_connections 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f22e806ca10 0x7f22e806eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308162f80 0x7f230814eba0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23080ce510 0x7f230815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 --2- 172.21.15.67:0/1285775543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23080cd750 0x7f230815ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 >> 172.21.15.67:0/1285775543 conn(0x7f230806b8c0 msgr2=0x7f23080b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 shutdown_connections 2024-04-17T08:30:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:33.881+0000 7f230f000640 1 -- 172.21.15.67:0/1285775543 wait complete. 2024-04-17T08:30:33.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:30:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.115+0000 7fddc9a00640 1 Processor -- start 2024-04-17T08:30:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.115+0000 7fddc9a00640 1 -- start start 2024-04-17T08:30:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.116+0000 7fddc9a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddc40ce480 0x7fddc406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.116+0000 7fddc9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.116+0000 7fddc9a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddc40d07a0 con 0x7fddc406db30 2024-04-17T08:30:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.116+0000 7fddc9a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fddc40d1650 con 0x7fddc40ce480 2024-04-17T08:30:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.116+0000 7fddc9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddc40d1ae0 con 0x7fddc409b170 2024-04-17T08:30:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.117+0000 7fddc3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddc40ce480 0x7fddc406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.117+0000 7fddc9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.117+0000 7fddc9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:42160/0 (socket says 172.21.15.67:42160) 2024-04-17T08:30:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.117+0000 7fddc9000640 1 -- 172.21.15.67:0/1676847366 learned_addr learned my addr 172.21.15.67:0/1676847366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.117+0000 7fddc3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddc40ce480 0x7fddc406d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:56582/0 (socket says 172.21.15.67:56582) 2024-04-17T08:30:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.118+0000 7fddc3e00640 1 -- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 msgr2=0x7fddc40ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.118+0000 7fddc3e00640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.118+0000 7fddc3e00640 1 -- 172.21.15.67:0/1676847366 >> v1:172.21.15.187:6789/0 conn(0x7fddc406db30 legacy=0x7fddc409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.118+0000 7fddc3e00640 1 -- 172.21.15.67:0/1676847366 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc409ba50 con 0x7fddc40ce480 2024-04-17T08:30:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.118+0000 7fddc9000640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:30:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.119+0000 7fddc3e00640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc406d5f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fddac0097a0 tx=0x7fddac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72fd10f0bcd645c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.120+0000 7fddc2a00640 1 -- 172.21.15.67:0/1676847366 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac035020 con 0x7fddc40ce480 2024-04-17T08:30:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.120+0000 7fddc2a00640 1 -- 172.21.15.67:0/1676847366 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddac031830 con 0x7fddc40ce480 2024-04-17T08:30:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.120+0000 7fddc2a00640 1 -- 172.21.15.67:0/1676847366 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac031c00 con 0x7fddc40ce480 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 -- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 msgr2=0x7fddc406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc406d5f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fddac0097a0 tx=0x7fddac02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 -- 172.21.15.67:0/1676847366 shutdown_connections 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1676847366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc406d5f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 -- 172.21.15.67:0/1676847366 >> 172.21.15.67:0/1676847366 conn(0x7fddc40b4050 msgr2=0x7fddc40b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 -- 172.21.15.67:0/1676847366 shutdown_connections 2024-04-17T08:30:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.121+0000 7fddc9a00640 1 -- 172.21.15.67:0/1676847366 wait complete. 2024-04-17T08:30:34.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.123+0000 7fddc9a00640 1 Processor -- start 2024-04-17T08:30:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.123+0000 7fddc9a00640 1 -- start start 2024-04-17T08:30:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc40cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddc40d1000 con 0x7fddc406db30 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddc40d0d20 con 0x7fddc40ce480 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddc40d12e0 con 0x7fddc409b170 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40320/0 (socket says 172.21.15.67:40320) 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc3e00640 1 -- 172.21.15.67:0/1508112448 learned_addr learned my addr 172.21.15.67:0/1508112448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc9000640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc40cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.124+0000 7fddc3400640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 -- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 msgr2=0x7fddc40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 -- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 msgr2=0x7fddc40cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc40cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 -- 172.21.15.67:0/1508112448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc40c8ba0 con 0x7fddc406db30 2024-04-17T08:30:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc3e00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fddac034040 tx=0x7fddac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac040070 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddac03a910 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac035040 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.125+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc40c6060 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.126+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc40c6620 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.127+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc40755d0 con 0x7fddc406db30 2024-04-17T08:30:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.128+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fddac038020 con 0x7fddc406db30 2024-04-17T08:30:34.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.129+0000 7fddbbe00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 0x7fdda006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.129+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7fddac0ad500 con 0x7fddc406db30 2024-04-17T08:30:34.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.130+0000 7fddc3400640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 0x7fdda006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.131+0000 7fddc3400640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 0x7fdda006eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fddc40c8340 tx=0x7fddb4009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.134+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fddac031800 con 0x7fddc406db30 2024-04-17T08:30:34.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.343+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7fddc40b3860 con 0x7fddc406db30 2024-04-17T08:30:34.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.350+0000 7fddbbe00640 1 -- 172.21.15.67:0/1508112448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) ==== 115+0+0 (secure 0 0 0) 0x7fddac07b8e0 con 0x7fddc406db30 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 msgr2=0x7fdda006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 0x7fdda006eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fddc40c8340 tx=0x7fddb4009040 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 msgr2=0x7fddc40ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fddac034040 tx=0x7fddac0046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 shutdown_connections 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fdda006ca10 0x7fdda006eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddc40ce480 0x7fddc40cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddc409b170 0x7fddc40c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.353+0000 7fddc9a00640 1 --2- 172.21.15.67:0/1508112448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc406db30 0x7fddc40ca040 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.354+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 >> 172.21.15.67:0/1508112448 conn(0x7fddc40b4050 msgr2=0x7fddc40b7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:34.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.354+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 shutdown_connections 2024-04-17T08:30:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.354+0000 7fddc9a00640 1 -- 172.21.15.67:0/1508112448 wait complete. 2024-04-17T08:30:34.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:30:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.600+0000 7ff87e200640 1 Processor -- start 2024-04-17T08:30:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.600+0000 7ff87e200640 1 -- start start 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.601+0000 7ff87e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8780cde80 0x7ff87815c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.601+0000 7ff87e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.601+0000 7ff87e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8780cc270 con 0x7ff8780cd030 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.601+0000 7ff87e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8780cc550 con 0x7ff8780cde80 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.601+0000 7ff87e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8780cc830 con 0x7ff87815b7f0 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8780cde80 0x7ff87815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87d800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8780cd030 0x7ff8780ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42052/0 (socket says 172.21.15.67:42052) 2024-04-17T08:30:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87d800640 1 -- 172.21.15.67:0/1035958891 learned_addr learned my addr 172.21.15.67:0/1035958891 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff877e00640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87ce00640 1 -- 172.21.15.67:0/1035958891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 msgr2=0x7ff8780b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87ce00640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87ce00640 1 -- 172.21.15.67:0/1035958891 >> v1:172.21.15.67:6789/0 conn(0x7ff8780cd030 legacy=0x7ff8780ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff87ce00640 1 -- 172.21.15.67:0/1035958891 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8780cf040 con 0x7ff8780cde80 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff877400640 1 -- 172.21.15.67:0/1035958891 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149752855 0 0) 0x7ff8780cc270 con 0x7ff8780cd030 2024-04-17T08:30:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.602+0000 7ff877e00640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff87ce00640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87815c640 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff8680097a0 tx=0x7ff86802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1894dd1a482cd407 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff877400640 1 -- 172.21.15.67:0/1035958891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868034070 con 0x7ff8780cde80 2024-04-17T08:30:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff877400640 1 -- 172.21.15.67:0/1035958891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff868030610 con 0x7ff8780cde80 2024-04-17T08:30:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff877400640 1 -- 172.21.15.67:0/1035958891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8680309e0 con 0x7ff8780cde80 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff87e200640 1 -- 172.21.15.67:0/1035958891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 msgr2=0x7ff87815c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.604+0000 7ff87e200640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87815c640 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff8680097a0 tx=0x7ff86802b960 comp rx=0 tx=0).stop 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 -- 172.21.15.67:0/1035958891 shutdown_connections 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 --2- 172.21.15.67:0/1035958891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87815c640 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 -- 172.21.15.67:0/1035958891 >> 172.21.15.67:0/1035958891 conn(0x7ff87809aa60 msgr2=0x7ff8780b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 -- 172.21.15.67:0/1035958891 shutdown_connections 2024-04-17T08:30:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.605+0000 7ff87e200640 1 -- 172.21.15.67:0/1035958891 wait complete. 2024-04-17T08:30:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.606+0000 7ff87e200640 1 Processor -- start 2024-04-17T08:30:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.607+0000 7ff87e200640 1 -- start start 2024-04-17T08:30:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8780cd030 0x7ff878150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780cf600 con 0x7ff8780cde80 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780cf480 con 0x7ff8780cd030 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8780cf780 con 0x7ff87815b7f0 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40344/0 (socket says 172.21.15.67:40344) 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 -- 172.21.15.67:0/174445210 learned_addr learned my addr 172.21.15.67:0/174445210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff87ce00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8780cd030 0x7ff878150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 -- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 msgr2=0x7ff8780b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 -- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8780cd030 msgr2=0x7ff878150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8780cd030 0x7ff878150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.608+0000 7ff877e00640 1 -- 172.21.15.67:0/174445210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8781f3620 con 0x7ff8780cde80 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff877e00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff86000c7f0 tx=0x7ff86000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff860015070 con 0x7ff8780cde80 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff860007820 con 0x7ff8780cde80 2024-04-17T08:30:34.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff878150070 con 0x7ff8780cde80 2024-04-17T08:30:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff860007bd0 con 0x7ff8780cde80 2024-04-17T08:30:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.609+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8781f3900 con 0x7ff8780cde80 2024-04-17T08:30:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.610+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8780755d0 con 0x7ff8780cde80 2024-04-17T08:30:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.611+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7ff860018070 con 0x7ff8780cde80 2024-04-17T08:30:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.612+0000 7ff874c00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 0x7ff85406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.612+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7ff86008bfc0 con 0x7ff8780cde80 2024-04-17T08:30:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.613+0000 7ff87ce00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 0x7ff85406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:34.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.614+0000 7ff87ce00640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 0x7ff85406eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff868036000 tx=0x7ff868008030 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:34.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.617+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff86005a160 con 0x7ff8780cde80 2024-04-17T08:30:34.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.826+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7ff87815ff60 con 0x7ff8780cde80 2024-04-17T08:30:34.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.833+0000 7ff874c00640 1 -- 172.21.15.67:0/174445210 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) ==== 104+0+0 (secure 0 0 0) 0x7ff860059a60 con 0x7ff8780cde80 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 msgr2=0x7ff85406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 0x7ff85406eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff868036000 tx=0x7ff868008030 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 msgr2=0x7ff87814bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff86000c7f0 tx=0x7ff86000ccb0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 shutdown_connections 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff85406ca10 0x7ff85406eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff87815b7f0 0x7ff8780b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780cde80 0x7ff87814bfd0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 --2- 172.21.15.67:0/174445210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8780cd030 0x7ff878150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.836+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 >> 172.21.15.67:0/174445210 conn(0x7ff87809aa60 msgr2=0x7ff8781515e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.837+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 shutdown_connections 2024-04-17T08:30:34.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:34.837+0000 7ff87e200640 1 -- 172.21.15.67:0/174445210 wait complete. 2024-04-17T08:30:34.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:30:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.072+0000 7f5d07c00640 1 Processor -- start 2024-04-17T08:30:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.073+0000 7f5d07c00640 1 -- start start 2024-04-17T08:30:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5d07c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5d00145ac0 0x7f5d00071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5d07c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5d000cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5d07c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d000cea70 con 0x7f5d000ce3b0 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5d07c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5d000ce840 con 0x7f5d00145ac0 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5d07c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d000ced50 con 0x7f5d00072d80 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.074+0000 7f5cffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5d000ce3b0 0x7f5d00072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42054/0 (socket says 172.21.15.67:42054) 2024-04-17T08:30:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cffe00640 1 -- 172.21.15.67:0/1424294003 learned_addr learned my addr 172.21.15.67:0/1424294003 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5d000cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133864127 0 0) 0x7f5d000cea70 con 0x7f5d000ce3b0 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce4003520 con 0x7f5d000ce3b0 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5d05200640 1 --2- 172.21.15.67:0/1424294003 >> v2:172.21.15.187:3300/0 conn(0x7f5d00145ac0 0x7f5d00071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2230388991 0 0) 0x7f5ce4003520 con 0x7f5d000ce3b0 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 msgr2=0x7f5d000cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5d000cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 >> v2:172.21.15.187:3300/0 conn(0x7f5d00145ac0 msgr2=0x7f5d00071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 --2- 172.21.15.67:0/1424294003 >> v2:172.21.15.187:3300/0 conn(0x7f5d00145ac0 0x7f5d00071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d000cf030 con 0x7f5d000ce3b0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5ce8001670 con 0x7f5d000ce3b0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5d000cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.075+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 >> v1:172.21.15.67:6789/0 conn(0x7f5d000ce3b0 legacy=0x7f5d00072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce40039b0 0x7f5ce4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5ce4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5ce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000ced50 con 0x7f5ce4004d40 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce400b9f0 con 0x7f5ce40039b0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce400bbf0 con 0x7f5d00072d80 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5ce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05200640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce40039b0 0x7f5ce4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5cffe00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5ce4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 -- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 msgr2=0x7f5ce4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d00072d80 0x7f5ce4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 -- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce40039b0 msgr2=0x7f5ce4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ce40039b0 0x7f5ce4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.076+0000 7f5d05c00640 1 -- 172.21.15.67:0/1424294003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce4003520 con 0x7f5ce4004d40 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.077+0000 7f5d05c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5ce40051c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f5d001467e0 tx=0x7f5ce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158c202e6ef970b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf800f040 con 0x7f5ce4004d40 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8004030 con 0x7f5ce4004d40 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5cff400640 1 -- 172.21.15.67:0/1424294003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8004400 con 0x7f5ce4004d40 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d07c00640 1 -- 172.21.15.67:0/1424294003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 msgr2=0x7f5ce40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d07c00640 1 --2- 172.21.15.67:0/1424294003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5ce40051c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f5d001467e0 tx=0x7f5ce4005830 comp rx=0 tx=0).stop 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d05c00640 1 -- 172.21.15.67:0/1424294003 reap_dead start 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d07c00640 1 -- 172.21.15.67:0/1424294003 shutdown_connections 2024-04-17T08:30:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d07c00640 1 -- 172.21.15.67:0/1424294003 >> 172.21.15.67:0/1424294003 conn(0x7f5d00067ee0 msgr2=0x7f5d0006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.078+0000 7f5d07c00640 1 -- 172.21.15.67:0/1424294003 shutdown_connections 2024-04-17T08:30:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.079+0000 7f5d07c00640 1 -- 172.21.15.67:0/1424294003 wait complete. 2024-04-17T08:30:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.080+0000 7f5d07c00640 1 Processor -- start 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.080+0000 7f5d07c00640 1 -- start start 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d001f5de0 0x7f5d001f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d001f2080 0x7f5d0006ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00145ea0 con 0x7f5ce4004d40 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000cf5f0 con 0x7f5d001f2080 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.081+0000 7f5d07c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d000cfb10 con 0x7f5d001f5de0 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40354/0 (socket says 172.21.15.67:40354) 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 -- 172.21.15.67:0/3802018133 learned_addr learned my addr 172.21.15.67:0/3802018133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5cffe00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d001f5de0 0x7f5d001f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 -- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d001f5de0 msgr2=0x7f5d001f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d001f5de0 0x7f5d001f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 -- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d001f2080 msgr2=0x7f5d0006ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d001f2080 0x7f5d0006ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 -- 172.21.15.67:0/3802018133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d001f5b90 con 0x7f5ce4004d40 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.082+0000 7f5d05200640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009ce0 tx=0x7f5cf0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.083+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0014610 con 0x7f5ce4004d40 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.083+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0019070 con 0x7f5ce4004d40 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.083+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00149d0 con 0x7f5ce4004d40 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.083+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001f62a0 con 0x7f5ce4004d40 2024-04-17T08:30:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.083+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d001f6860 con 0x7f5ce4004d40 2024-04-17T08:30:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.084+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00068ca0 con 0x7f5ce4004d40 2024-04-17T08:30:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.085+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f5cf0014b70 con 0x7f5ce4004d40 2024-04-17T08:30:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.086+0000 7f5cfcc00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 0x7f5cdc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.086+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7f5cf008c770 con 0x7f5ce4004d40 2024-04-17T08:30:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.086+0000 7f5cffe00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 0x7f5cdc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.087+0000 7f5cffe00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 0x7f5cdc06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5d00140970 tx=0x7f5ce8001db0 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.090+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5cf005a910 con 0x7f5ce4004d40 2024-04-17T08:30:35.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.293+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f5d000cc6c0 con 0x7f5ce4004d40 2024-04-17T08:30:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.298+0000 7f5cfcc00640 1 -- 172.21.15.67:0/3802018133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v57) ==== 120+0+0 (secure 0 0 0) 0x7f5cf005a210 con 0x7f5ce4004d40 2024-04-17T08:30:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 msgr2=0x7f5cdc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 0x7f5cdc06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5d00140970 tx=0x7f5ce8001db0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 msgr2=0x7f5d001f2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009ce0 tx=0x7f5cf0007700 comp rx=0 tx=0).stop 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 shutdown_connections 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5cdc06ca10 0x7f5cdc06eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d001f2080 0x7f5d0006ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d001f5de0 0x7f5d001f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 --2- 172.21.15.67:0/3802018133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ce4004d40 0x7f5d001f2860 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 >> 172.21.15.67:0/3802018133 conn(0x7f5d00067ee0 msgr2=0x7f5d0006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 shutdown_connections 2024-04-17T08:30:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.301+0000 7f5d07c00640 1 -- 172.21.15.67:0/3802018133 wait complete. 2024-04-17T08:30:35.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-04-17T08:30:35.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.524+0000 7f4315c00640 1 Processor -- start 2024-04-17T08:30:35.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.525+0000 7f4315c00640 1 -- start start 2024-04-17T08:30:35.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.526+0000 7f4315c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f431015b570 0x7f431006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.526+0000 7f4315c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f431015bc80 con 0x7f431006fce0 2024-04-17T08:30:35.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.526+0000 7f4315c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f431015bf40 con 0x7f431006c040 2024-04-17T08:30:35.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.526+0000 7f4315c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f431006c470 con 0x7f431015b570 2024-04-17T08:30:35.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.527+0000 7f4307400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f431006fce0 0x7f431006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42066/0 (socket says 172.21.15.67:42066) 2024-04-17T08:30:35.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.527+0000 7f4307400640 1 -- 172.21.15.67:0/628557067 learned_addr learned my addr 172.21.15.67:0/628557067 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.527+0000 7f430fe00640 1 --2- 172.21.15.67:0/628557067 >> v2:172.21.15.187:3300/0 conn(0x7f431015b570 0x7f431006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.527+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419863913 0 0) 0x7f431015bc80 con 0x7f431006fce0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.527+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f4003520 con 0x7f431006fce0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1017558696 0 0) 0x7f42f4003520 con 0x7f431006fce0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 >> v2:172.21.15.187:3300/0 conn(0x7f431015b570 msgr2=0x7f431006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 --2- 172.21.15.67:0/628557067 >> v2:172.21.15.187:3300/0 conn(0x7f431015b570 0x7f431006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 >> v1:172.21.15.187:6789/0 conn(0x7f431006c040 legacy=0x7f431015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431015c220 con 0x7f431006fce0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f42f8001670 con 0x7f431006fce0 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 >> v1:172.21.15.67:6789/0 conn(0x7f431006fce0 legacy=0x7f431006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 0x7f42f4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f4004800 0x7f42f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4005290 0x7f42f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f431006c470 con 0x7f42f4003a70 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f42f400e3f0 con 0x7f42f4004800 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f42f400e5f0 con 0x7f42f4005290 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f430fe00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 0x7f42f4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.528+0000 7f4307400640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f4004800 0x7f42f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f430fe00640 1 -- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4005290 msgr2=0x7f42f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f430fe00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4005290 0x7f42f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f430fe00640 1 -- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f4004800 msgr2=0x7f42f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f430fe00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42f4004800 0x7f42f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f430fe00640 1 -- 172.21.15.67:0/628557067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f4003520 con 0x7f42f4003a70 2024-04-17T08:30:35.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.529+0000 7f4315200640 1 -- 172.21.15.67:0/628557067 reap_dead start 2024-04-17T08:30:35.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.530+0000 7f430fe00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 0x7f42f4003eb0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f4300009ac0 tx=0x7f42f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2865da0dce5d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.530+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4300039070 con 0x7f42f4003a70 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.530+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f430002c710 con 0x7f42f4003a70 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.530+0000 7f430f400640 1 -- 172.21.15.67:0/628557067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4300009690 con 0x7f42f4003a70 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 -- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 msgr2=0x7f42f4003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 0x7f42f4003eb0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f4300009ac0 tx=0x7f42f4005830 comp rx=0 tx=0).stop 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 -- 172.21.15.67:0/628557067 shutdown_connections 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 --2- 172.21.15.67:0/628557067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42f4003a70 0x7f42f4003eb0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 -- 172.21.15.67:0/628557067 >> 172.21.15.67:0/628557067 conn(0x7f4310067d10 msgr2=0x7f43100b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 -- 172.21.15.67:0/628557067 shutdown_connections 2024-04-17T08:30:35.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.531+0000 7f4315c00640 1 -- 172.21.15.67:0/628557067 wait complete. 2024-04-17T08:30:35.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.533+0000 7f4315c00640 1 Processor -- start 2024-04-17T08:30:35.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.533+0000 7f4315c00640 1 -- start start 2024-04-17T08:30:35.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4003a70 0x7f4310141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f431015e0d0 0x7f431015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43100b47b0 con 0x7f43101575e0 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f431015b950 con 0x7f431015e0d0 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4315c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43100b4930 con 0x7f42f4003a70 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40384/0 (socket says 172.21.15.67:40384) 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 -- 172.21.15.67:0/2948269072 learned_addr learned my addr 172.21.15.67:0/2948269072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f430fe00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4003a70 0x7f4310141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 -- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4003a70 msgr2=0x7f4310141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4003a70 0x7f4310141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 -- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f431015e0d0 msgr2=0x7f431015e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f431015e0d0 0x7f431015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.534+0000 7f4307400640 1 -- 172.21.15.67:0/2948269072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4310157ae0 con 0x7f43101575e0 2024-04-17T08:30:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f4307400640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f42f800e740 tx=0x7f42f800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f800ee50 con 0x7f43101575e0 2024-04-17T08:30:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f8018070 con 0x7f43101575e0 2024-04-17T08:30:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f800cb10 con 0x7f43101575e0 2024-04-17T08:30:35.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4310157dc0 con 0x7f43101575e0 2024-04-17T08:30:35.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.535+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4310158380 con 0x7f43101575e0 2024-04-17T08:30:35.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.536+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4310068a10 con 0x7f43101575e0 2024-04-17T08:30:35.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.537+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f42f800ccb0 con 0x7f43101575e0 2024-04-17T08:30:35.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.538+0000 7f430cc00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 0x7f42ec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:35.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.538+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7f42f808bba0 con 0x7f43101575e0 2024-04-17T08:30:35.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.539+0000 7f430fe00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 0x7f42ec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.540+0000 7f430fe00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 0x7f42ec06ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4300036000 tx=0x7f4300030000 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.543+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f42f8059e50 con 0x7f43101575e0 2024-04-17T08:30:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.820+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7f43100757d0 con 0x7f43101575e0 2024-04-17T08:30:35.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.883+0000 7f430cc00640 1 -- 172.21.15.67:0/2948269072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2515) ==== 178+0+0 (secure 0 0 0) 0x7f42f8059750 con 0x7f43101575e0 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.886+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 msgr2=0x7f42ec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.886+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 0x7f42ec06ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4300036000 tx=0x7f4300030000 comp rx=0 tx=0).stop 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.886+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 msgr2=0x7f431014f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.886+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f42f800e740 tx=0x7f42f800ec00 comp rx=0 tx=0).stop 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 shutdown_connections 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f42ec06ca40 0x7f42ec06ef00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f431015e0d0 0x7f431015e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43101575e0 0x7f431014f620 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 --2- 172.21.15.67:0/2948269072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42f4003a70 0x7f4310141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 >> 172.21.15.67:0/2948269072 conn(0x7f4310067d10 msgr2=0x7f43100ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 shutdown_connections 2024-04-17T08:30:35.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:35.887+0000 7f4315c00640 1 -- 172.21.15.67:0/2948269072 wait complete. 2024-04-17T08:30:35.909 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-04-17T08:30:35.910 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-04-17T08:30:35.910 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T08:30:35.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:36.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.131+0000 7f7e85c00640 1 Processor -- start 2024-04-17T08:30:36.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.131+0000 7f7e85c00640 1 -- start start 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e85c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7e80160280 0x7f7e8006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e85c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e80142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e85c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e800cd4a0 con 0x7f7e8006b8c0 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e85c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7e8015ce30 con 0x7f7e80160280 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e85c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e801607f0 con 0x7f7e80141ed0 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e7fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e8006b8c0 0x7f7e800cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42078/0 (socket says 172.21.15.67:42078) 2024-04-17T08:30:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.133+0000 7f7e7fe00640 1 -- 172.21.15.67:0/892130537 learned_addr learned my addr 172.21.15.67:0/892130537 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030958163 0 0) 0x7f7e800cd4a0 con 0x7f7e8006b8c0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e85200640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e80142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> v2:172.21.15.187:3300/0 conn(0x7f7e80160280 0x7f7e8006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e64003520 con 0x7f7e8006b8c0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 399121704 0 0) 0x7f7e64003520 con 0x7f7e8006b8c0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 msgr2=0x7f7e80142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e80142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 >> v2:172.21.15.187:3300/0 conn(0x7f7e80160280 msgr2=0x7f7e8006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 --2- 172.21.15.67:0/892130537 >> v2:172.21.15.187:3300/0 conn(0x7f7e80160280 0x7f7e8006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e80160ad0 con 0x7f7e8006b8c0 2024-04-17T08:30:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7e70001670 con 0x7f7e8006b8c0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 >> v1:172.21.15.67:6789/0 conn(0x7f7e8006b8c0 legacy=0x7f7e800cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e64003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e64003ff0 0x7f7e64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8015ce30 con 0x7f7e64003ff0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e801607f0 con 0x7f7e64004d40 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6400baf0 con 0x7f7e80141ed0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.134+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> v2:172.21.15.187:3300/0 conn(0x7f7e80160280 0x7f7e8006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e64003ff0 0x7f7e64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e85200640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e7fe00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e64003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 -- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 msgr2=0x7f7e64003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e80141ed0 0x7f7e64003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 -- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64004d40 msgr2=0x7f7e640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64004d40 0x7f7e640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.135+0000 7f7e77e00640 1 -- 172.21.15.67:0/892130537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e64003520 con 0x7f7e64003ff0 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.136+0000 7f7e77e00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e64003ff0 0x7f7e64004800 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef866dfd072809f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800f040 con 0x7f7e64003ff0 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e68037600 con 0x7f7e64003ff0 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e7f400640 1 -- 172.21.15.67:0/892130537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680379b0 con 0x7f7e64003ff0 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e85c00640 1 -- 172.21.15.67:0/892130537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e64003ff0 msgr2=0x7f7e64004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e85c00640 1 --2- 172.21.15.67:0/892130537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e64003ff0 0x7f7e64004800 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e64005830 comp rx=0 tx=0).stop 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e85200640 1 -- 172.21.15.67:0/892130537 reap_dead start 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e85c00640 1 -- 172.21.15.67:0/892130537 shutdown_connections 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.137+0000 7f7e85c00640 1 -- 172.21.15.67:0/892130537 >> 172.21.15.67:0/892130537 conn(0x7f7e80067d10 msgr2=0x7f7e800b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.138+0000 7f7e85c00640 1 -- 172.21.15.67:0/892130537 shutdown_connections 2024-04-17T08:30:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.138+0000 7f7e85c00640 1 -- 172.21.15.67:0/892130537 wait complete. 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.139+0000 7f7e85c00640 1 Processor -- start 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.139+0000 7f7e85c00640 1 -- start start 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64003ff0 0x7f7e801f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e801450f0 0x7f7e80145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e80161090 con 0x7f7e800b3160 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8006c1e0 con 0x7f7e64003ff0 2024-04-17T08:30:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.140+0000 7f7e85c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8013f9b0 con 0x7f7e801450f0 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40406/0 (socket says 172.21.15.67:40406) 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 -- 172.21.15.67:0/3399675023 learned_addr learned my addr 172.21.15.67:0/3399675023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e7fe00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64003ff0 0x7f7e801f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e85200640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e801450f0 0x7f7e80145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 -- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e801450f0 msgr2=0x7f7e80145570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e801450f0 0x7f7e80145570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 -- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64003ff0 msgr2=0x7f7e801f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64003ff0 0x7f7e801f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.141+0000 7f7e77e00640 1 -- 172.21.15.67:0/3399675023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e801f3ca0 con 0x7f7e800b3160 2024-04-17T08:30:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e77e00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7e6802bff0 tx=0x7f7e68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800f000 con 0x7f7e800b3160 2024-04-17T08:30:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6802fa30 con 0x7f7e800b3160 2024-04-17T08:30:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6802fe00 con 0x7f7e800b3160 2024-04-17T08:30:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e801f3f80 con 0x7f7e800b3160 2024-04-17T08:30:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.142+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8006ee40 con 0x7f7e800b3160 2024-04-17T08:30:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.143+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e801449c0 con 0x7f7e800b3160 2024-04-17T08:30:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.144+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f7e68042440 con 0x7f7e800b3160 2024-04-17T08:30:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.145+0000 7f7e7cc00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 0x7f7e5c06eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.145+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7f7e680ae3c0 con 0x7f7e800b3160 2024-04-17T08:30:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.146+0000 7f7e7fe00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 0x7f7e5c06eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.147+0000 7f7e7fe00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 0x7f7e5c06eea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001050 tx=0x7f7e70009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.150+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7e68033030 con 0x7f7e800b3160 2024-04-17T08:30:36.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.357+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e80068a10 con 0x7f7e800b3160 2024-04-17T08:30:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.359+0000 7f7e7cc00640 1 -- 172.21.15.67:0/3399675023 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) ==== 74+0+14042 (secure 0 0 0) 0x7f7e6807c560 con 0x7f7e800b3160 2024-04-17T08:30:36.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:36.359 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":173,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:33.868259+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/2582506112":"2024-04-18T08:30:30.844067+0000","172.21.15.67:6833/2582506112":"2024-04-18T08:30:30.844067+0000","172.21.15.187:6833/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6832/2413623528":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/4100169880":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/529736597":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2208412755":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1092697670":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/1418137156":"2024-04-18T08:30:26.935522+0000","172.21.15.187:6833/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:6832/115904466":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/4218440392":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2263870823":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/2708413279":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3224926034":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/791073325":"2024-04-18T07:34:28.771962+0000","172.21.15.187:0/3654447672":"2024-04-18T08:30:26.935522+0000","172.21.15.187:0/2286768646":"2024-04-18T07:34:28.771962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 msgr2=0x7f7e5c06eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 0x7f7e5c06eea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e70001050 tx=0x7f7e70009040 comp rx=0 tx=0).stop 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 msgr2=0x7f7e800b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7e6802bff0 tx=0x7f7e68004730 comp rx=0 tx=0).stop 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 shutdown_connections 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f7e5c06c9e0 0x7f7e5c06eea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e801450f0 0x7f7e80145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e800b3160 0x7f7e800b35e0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 --2- 172.21.15.67:0/3399675023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64003ff0 0x7f7e801f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 >> 172.21.15.67:0/3399675023 conn(0x7f7e80067d10 msgr2=0x7f7e800d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.362+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 shutdown_connections 2024-04-17T08:30:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.363+0000 7f7e85c00640 1 -- 172.21.15.67:0/3399675023 wait complete. 2024-04-17T08:30:36.381 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:30:36.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:30:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.606+0000 7feb51e00640 1 Processor -- start 2024-04-17T08:30:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.606+0000 7feb51e00640 1 -- start start 2024-04-17T08:30:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.608+0000 7feb51e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb4c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.608+0000 7feb51e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4c160920 con 0x7feb4c146060 2024-04-17T08:30:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.608+0000 7feb51e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4c160c00 con 0x7feb4c0b4ac0 2024-04-17T08:30:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.608+0000 7feb51e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c160ee0 con 0x7feb4c145b00 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.609+0000 7feb4be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feb4c146060 0x7feb4c15cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42090/0 (socket says 172.21.15.67:42090) 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.609+0000 7feb4be00640 1 -- 172.21.15.67:0/3284462395 learned_addr learned my addr 172.21.15.67:0/3284462395 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.609+0000 7feb50a00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb4c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.609+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678400842 0 0) 0x7feb4c160920 con 0x7feb4c146060 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.609+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb34003520 con 0x7feb4c146060 2024-04-17T08:30:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 520292407 0 0) 0x7feb34003520 con 0x7feb4c146060 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 msgr2=0x7feb4c160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb4c160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 >> v1:172.21.15.187:6789/0 conn(0x7feb4c0b4ac0 legacy=0x7feb4c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c1611c0 con 0x7feb4c146060 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7feb38001670 con 0x7feb4c146060 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 >> v1:172.21.15.67:6789/0 conn(0x7feb4c146060 legacy=0x7feb4c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb340039e0 0x7feb34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb34004800 0x7feb34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb34005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c160ee0 con 0x7feb34004800 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feb3400bb50 con 0x7feb340039e0 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb3400bd50 con 0x7feb4c145b00 2024-04-17T08:30:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb4be00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb34004800 0x7feb34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.610+0000 7feb50a00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb340039e0 0x7feb34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb51400640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb34005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb4be00640 1 -- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 msgr2=0x7feb34005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb4be00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb4c145b00 0x7feb34005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb4be00640 1 -- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb340039e0 msgr2=0x7feb34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb4be00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb340039e0 0x7feb34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.611+0000 7feb4be00640 1 -- 172.21.15.67:0/3284462395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb34003520 con 0x7feb34004800 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.612+0000 7feb4be00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb34004800 0x7feb34004c80 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7feb3800c110 tx=0x7feb34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9da78f86549477 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38007c40 con 0x7feb34004800 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb38030070 con 0x7feb34004800 2024-04-17T08:30:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb4b400640 1 -- 172.21.15.67:0/3284462395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38005d30 con 0x7feb34004800 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 -- 172.21.15.67:0/3284462395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb34004800 msgr2=0x7feb34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 --2- 172.21.15.67:0/3284462395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb34004800 0x7feb34004c80 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7feb3800c110 tx=0x7feb34005830 comp rx=0 tx=0).stop 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51400640 1 -- 172.21.15.67:0/3284462395 reap_dead start 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 -- 172.21.15.67:0/3284462395 shutdown_connections 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 -- 172.21.15.67:0/3284462395 >> 172.21.15.67:0/3284462395 conn(0x7feb4c0b23c0 msgr2=0x7feb4c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 -- 172.21.15.67:0/3284462395 shutdown_connections 2024-04-17T08:30:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.613+0000 7feb51e00640 1 -- 172.21.15.67:0/3284462395 wait complete. 2024-04-17T08:30:36.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.615+0000 7feb51e00640 1 Processor -- start 2024-04-17T08:30:36.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.615+0000 7feb51e00640 1 -- start start 2024-04-17T08:30:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb34004800 0x7feb4c1f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 0x7feb4c13c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c160340 con 0x7feb4c141220 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c0cfba0 con 0x7feb4c1f8050 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.616+0000 7feb51e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c1604c0 con 0x7feb34004800 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40424/0 (socket says 172.21.15.67:40424) 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 -- 172.21.15.67:0/2083070325 learned_addr learned my addr 172.21.15.67:0/2083070325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb50a00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb34004800 0x7feb4c1f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb51400640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 0x7feb4c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 -- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb34004800 msgr2=0x7feb4c1f56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb34004800 0x7feb4c1f56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 -- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 msgr2=0x7feb4c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 0x7feb4c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 -- 172.21.15.67:0/2083070325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c141680 con 0x7feb4c141220 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb51400640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 0x7feb4c13c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.617+0000 7feb4be00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7feb3802c4a0 tx=0x7feb4c160af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.618+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38045070 con 0x7feb4c141220 2024-04-17T08:30:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.618+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb38000ed0 con 0x7feb4c141220 2024-04-17T08:30:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.618+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38007c40 con 0x7feb4c141220 2024-04-17T08:30:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.618+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c13d720 con 0x7feb4c141220 2024-04-17T08:30:36.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.618+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c0c9f70 con 0x7feb4c141220 2024-04-17T08:30:36.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.619+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c068a10 con 0x7feb4c141220 2024-04-17T08:30:36.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.620+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7feb38007070 con 0x7feb4c141220 2024-04-17T08:30:36.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.621+0000 7feb48c00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 0x7feb2c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:36.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.621+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7feb380acd50 con 0x7feb4c141220 2024-04-17T08:30:36.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.622+0000 7feb50a00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 0x7feb2c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.623+0000 7feb50a00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 0x7feb2c06ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb40009b80 tx=0x7feb40008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.625+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7feb3807aa30 con 0x7feb4c141220 2024-04-17T08:30:36.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.818+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feb4c068610 con 0x7feb4c141220 2024-04-17T08:30:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.819+0000 7feb48c00640 1 -- 172.21.15.67:0/2083070325 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) ==== 93+0+10 (secure 0 0 0) 0x7feb3807ed30 con 0x7feb4c141220 2024-04-17T08:30:36.819 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:30:36.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 msgr2=0x7feb2c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 0x7feb2c06ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb40009b80 tx=0x7feb40008040 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 msgr2=0x7feb4c1f7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7feb3802c4a0 tx=0x7feb4c160af0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 shutdown_connections 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7feb2c06ca40 0x7feb2c06ef00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb4c1f8050 0x7feb4c13c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb4c141220 0x7feb4c1f7b10 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 --2- 172.21.15.67:0/2083070325 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb34004800 0x7feb4c1f56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 >> 172.21.15.67:0/2083070325 conn(0x7feb4c0b23c0 msgr2=0x7feb4c0cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 shutdown_connections 2024-04-17T08:30:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:36.822+0000 7feb51e00640 1 -- 172.21.15.67:0/2083070325 wait complete. 2024-04-17T08:30:36.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-17T08:30:37.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.057+0000 7fba4b200640 1 Processor -- start 2024-04-17T08:30:37.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.057+0000 7fba4b200640 1 -- start start 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.058+0000 7fba4b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba4c0ce8a0 0x7fba4c0d0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba4b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4c0d1060 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba4b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4c0d1340 con 0x7fba4c09aa60 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba4b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fba4c0cdc70 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba49e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba4c0ce4c0 0x7fba4c06d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42096/0 (socket says 172.21.15.67:42096) 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba49e00640 1 -- 172.21.15.67:0/4164159094 learned_addr learned my addr 172.21.15.67:0/4164159094 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba49400640 1 --2- 172.21.15.67:0/4164159094 >> v2:172.21.15.67:3300/0 conn(0x7fba4c0ce8a0 0x7fba4c0d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba48a00640 1 -- 172.21.15.67:0/4164159094 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942746165 0 0) 0x7fba4c0d1060 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.059+0000 7fba48a00640 1 -- 172.21.15.67:0/4164159094 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba30003520 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.060+0000 7fba49400640 1 -- 172.21.15.67:0/4164159094 >> v1:172.21.15.187:6789/0 conn(0x7fba4c09aa60 legacy=0x7fba4c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.060+0000 7fba49400640 1 -- 172.21.15.67:0/4164159094 >> v1:172.21.15.67:6789/0 conn(0x7fba4c0ce4c0 legacy=0x7fba4c06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.060+0000 7fba49400640 1 -- 172.21.15.67:0/4164159094 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c0d1620 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.061+0000 7fba49400640 1 --2- 172.21.15.67:0/4164159094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce8a0 0x7fba4c0d0980 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fba380097a0 tx=0x7fba3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d28b3a761eed50 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.061+0000 7fba48a00640 1 -- 172.21.15.67:0/4164159094 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38035020 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba48a00640 1 -- 172.21.15.67:0/4164159094 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba38031830 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba48a00640 1 -- 172.21.15.67:0/4164159094 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38031be0 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba4b200640 1 -- 172.21.15.67:0/4164159094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce8a0 msgr2=0x7fba4c0d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba4b200640 1 --2- 172.21.15.67:0/4164159094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce8a0 0x7fba4c0d0980 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fba380097a0 tx=0x7fba3802b8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba4b200640 1 -- 172.21.15.67:0/4164159094 shutdown_connections 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba4b200640 1 --2- 172.21.15.67:0/4164159094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce8a0 0x7fba4c0d0980 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.062+0000 7fba4b200640 1 -- 172.21.15.67:0/4164159094 >> 172.21.15.67:0/4164159094 conn(0x7fba4c0b4100 msgr2=0x7fba4c0b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.063+0000 7fba4b200640 1 -- 172.21.15.67:0/4164159094 shutdown_connections 2024-04-17T08:30:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.063+0000 7fba4b200640 1 -- 172.21.15.67:0/4164159094 wait complete. 2024-04-17T08:30:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.065+0000 7fba4b200640 1 Processor -- start 2024-04-17T08:30:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.065+0000 7fba4b200640 1 -- start start 2024-04-17T08:30:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba4c09aa60 0x7fba4c1f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 0x7fba4c15e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c0d2280 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c0d2100 con 0x7fba4c09aa60 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.066+0000 7fba4b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba4c09ae90 con 0x7fba4c0ce8a0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40446/0 (socket says 172.21.15.67:40446) 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 -- 172.21.15.67:0/1821936035 learned_addr learned my addr 172.21.15.67:0/1821936035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49e00640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba4c09aa60 0x7fba4c1f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba4a800640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 0x7fba4c15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 -- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 msgr2=0x7fba4c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 0x7fba4c15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 -- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba4c09aa60 msgr2=0x7fba4c1f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba4c09aa60 0x7fba4c1f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba49400640 1 -- 172.21.15.67:0/1821936035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c1f5460 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.067+0000 7fba4a800640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 0x7fba4c15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba49400640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fba38000c00 tx=0x7fba380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38040070 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba3803a830 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38035040 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c09cd50 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.068+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c09f2c0 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.069+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba4c0755d0 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.070+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fba3804b060 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.071+0000 7fba42000640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 0x7fba2c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.071+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7fba380ad730 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.072+0000 7fba49e00640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 0x7fba2c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.073+0000 7fba49e00640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 0x7fba2c06ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005880 tx=0x7fba3c009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:37.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.076+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba4c1f5460 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.366+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fba4c161f30 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.896+0000 7fba42000640 1 -- 172.21.15.67:0/1821936035 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2517) ==== 177+0+0 (secure 0 0 0) 0x7fba38031800 con 0x7fba4c0ce4c0 2024-04-17T08:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 msgr2=0x7fba2c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 0x7fba2c06ef00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba3c005880 tx=0x7fba3c009040 comp rx=0 tx=0).stop 2024-04-17T08:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 msgr2=0x7fba4c152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fba38000c00 tx=0x7fba380046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 shutdown_connections 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fba2c06ca40 0x7fba2c06ef00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4c0ce8a0 0x7fba4c15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c0ce4c0 0x7fba4c152960 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 --2- 172.21.15.67:0/1821936035 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba4c09aa60 0x7fba4c1f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 >> 172.21.15.67:0/1821936035 conn(0x7fba4c0b4100 msgr2=0x7fba4c0b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 shutdown_connections 2024-04-17T08:30:37.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:37.900+0000 7fba4b200640 1 -- 172.21.15.67:0/1821936035 wait complete. 2024-04-17T08:30:37.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:30:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.155+0000 7f4221c00640 1 Processor -- start 2024-04-17T08:30:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.155+0000 7f4221c00640 1 -- start start 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.156+0000 7f4221c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f421c145f80 0x7f421c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f4221c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c146360 con 0x7f421c0cd750 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f4221c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c071e70 con 0x7f421c06c270 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f4221c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f421c0727a0 con 0x7f421c145f80 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f4213e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f421c0cd750 0x7f421c06bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42098/0 (socket says 172.21.15.67:42098) 2024-04-17T08:30:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f4213e00640 1 -- 172.21.15.67:0/1838317614 learned_addr learned my addr 172.21.15.67:0/1838317614 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.157+0000 7f421be00640 1 --2- 172.21.15.67:0/1838317614 >> v2:172.21.15.187:3300/0 conn(0x7f421c145f80 0x7f421c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648056373 0 0) 0x7f421c146360 con 0x7f421c0cd750 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4200003520 con 0x7f421c0cd750 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 127354548 0 0) 0x7f4200003520 con 0x7f421c0cd750 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 >> v2:172.21.15.187:3300/0 conn(0x7f421c145f80 msgr2=0x7f421c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 --2- 172.21.15.67:0/1838317614 >> v2:172.21.15.187:3300/0 conn(0x7f421c145f80 0x7f421c0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 >> v1:172.21.15.187:6789/0 conn(0x7f421c06c270 legacy=0x7f421c06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c072150 con 0x7f421c0cd750 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4204001670 con 0x7f421c0cd750 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 >> v1:172.21.15.67:6789/0 conn(0x7f421c0cd750 legacy=0x7f421c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004800 0x7f4200004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f420000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f421c0727a0 con 0x7f42000051c0 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f420000c160 con 0x7f4200004800 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.158+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f420000c360 con 0x7f42000039b0 2024-04-17T08:30:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f420000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f421be00640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 -- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42000039b0 msgr2=0x7f4200003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4213e00640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004800 0x7f4200004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 -- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004800 msgr2=0x7f4200004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004800 0x7f4200004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.159+0000 7f4221200640 1 -- 172.21.15.67:0/1838317614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4200003520 con 0x7f42000051c0 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.160+0000 7f4221200640 1 -- 172.21.15.67:0/1838317614 reap_dead start 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.160+0000 7f4221200640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f420000b9c0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4214001e50 tx=0x7f4200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06abcc0fd73c2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4214001a70 con 0x7f42000051c0 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4214003bb0 con 0x7f42000051c0 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f421b400640 1 -- 172.21.15.67:0/1838317614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42140050d0 con 0x7f42000051c0 2024-04-17T08:30:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 -- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 msgr2=0x7f420000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f420000b9c0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f4214001e50 tx=0x7f4200005830 comp rx=0 tx=0).stop 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 -- 172.21.15.67:0/1838317614 shutdown_connections 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 --2- 172.21.15.67:0/1838317614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f420000b9c0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 -- 172.21.15.67:0/1838317614 >> 172.21.15.67:0/1838317614 conn(0x7f421c0b2190 msgr2=0x7f421c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 -- 172.21.15.67:0/1838317614 shutdown_connections 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.161+0000 7f4221c00640 1 -- 172.21.15.67:0/1838317614 wait complete. 2024-04-17T08:30:38.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.163+0000 7f4221c00640 1 Processor -- start 2024-04-17T08:30:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.163+0000 7f4221c00640 1 -- start start 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 0x7f421c142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c0b1110 0x7f421c0b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f421c072db0 con 0x7f42000051c0 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f421c072c30 con 0x7f421c142180 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f4221c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f421c0cd210 con 0x7f421c0b1110 2024-04-17T08:30:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f421be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f421be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40460/0 (socket says 172.21.15.67:40460) 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.164+0000 7f421be00640 1 -- 172.21.15.67:0/1528184126 learned_addr learned my addr 172.21.15.67:0/1528184126 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f4221200640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c0b1110 0x7f421c0b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f4213e00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 0x7f421c142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 -- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c0b1110 msgr2=0x7f421c0b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c0b1110 0x7f421c0b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 -- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 msgr2=0x7f421c142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 0x7f421c142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 -- 172.21.15.67:0/1528184126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c142c80 con 0x7f42000051c0 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f4213e00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 0x7f421c142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:38.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.165+0000 7f421be00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f420c011540 tx=0x7f420c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.166+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c015930 con 0x7f42000051c0 2024-04-17T08:30:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.166+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f420c015ad0 con 0x7f42000051c0 2024-04-17T08:30:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.166+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c00f460 con 0x7f42000051c0 2024-04-17T08:30:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.166+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c142f60 con 0x7f42000051c0 2024-04-17T08:30:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.166+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f421c143520 con 0x7f42000051c0 2024-04-17T08:30:38.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.167+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421c068a10 con 0x7f42000051c0 2024-04-17T08:30:38.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.168+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f420c00f600 con 0x7f42000051c0 2024-04-17T08:30:38.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.169+0000 7f4218c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 0x7f41f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:38.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.169+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7f420c08c4c0 con 0x7f42000051c0 2024-04-17T08:30:38.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.170+0000 7f4213e00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 0x7f41f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:38.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.171+0000 7f4213e00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 0x7f41f806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f421c1f7bd0 tx=0x7f4204009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:38.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.174+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f420c019030 con 0x7f42000051c0 2024-04-17T08:30:38.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.377+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f421c15f130 con 0x7f42000051c0 2024-04-17T08:30:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.382+0000 7f4218c00640 1 -- 172.21.15.67:0/1528184126 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) ==== 122+0+0 (secure 0 0 0) 0x7f420c05a8a0 con 0x7f42000051c0 2024-04-17T08:30:38.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 msgr2=0x7f41f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 0x7f41f806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f421c1f7bd0 tx=0x7f4204009040 comp rx=0 tx=0).stop 2024-04-17T08:30:38.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 msgr2=0x7f421c15bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f420c011540 tx=0x7f420c011a00 comp rx=0 tx=0).stop 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 shutdown_connections 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f41f806ca10 0x7f41f806eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c0b1110 0x7f421c0b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f421c142180 0x7f421c142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 --2- 172.21.15.67:0/1528184126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42000051c0 0x7f421c15bf30 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 >> 172.21.15.67:0/1528184126 conn(0x7f421c0b2190 msgr2=0x7f421c0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 shutdown_connections 2024-04-17T08:30:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:38.385+0000 7f4221c00640 1 -- 172.21.15.67:0/1528184126 wait complete. 2024-04-17T08:30:38.404 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:30:38.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:30:38.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:30:38.649 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:30:38.649 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:30:38.649 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:38.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:30:38.710 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:30:38.710 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:38.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:30:38.767 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:30:38.767 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:38.829 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:30:38.829 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:30:38.829 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:30:38.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:30:38.887 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:30:38.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:39.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.118+0000 7fb440a00640 1 Processor -- start 2024-04-17T08:30:39.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.119+0000 7fb440a00640 1 -- start start 2024-04-17T08:30:39.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb440a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb43c0d09f0 0x7fb43c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb440a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 0x7fb43c15e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb440a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb43c06dc30 con 0x7fb43c0b3d10 2024-04-17T08:30:39.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb440a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb43c06df10 con 0x7fb43c0d09f0 2024-04-17T08:30:39.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb440a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb43c06e1f0 con 0x7fb43c09afa0 2024-04-17T08:30:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb439e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb43c0b3d10 0x7fb43c06d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42110/0 (socket says 172.21.15.67:42110) 2024-04-17T08:30:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb439e00640 1 -- 172.21.15.67:0/439833610 learned_addr learned my addr 172.21.15.67:0/439833610 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:39.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.120+0000 7fb439400640 1 --2- 172.21.15.67:0/439833610 >> v2:172.21.15.67:3300/0 conn(0x7fb43c0d09f0 0x7fb43c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb43a800640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 0x7fb43c15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb438a00640 1 -- 172.21.15.67:0/439833610 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041236067 0 0) 0x7fb43c06dc30 con 0x7fb43c0b3d10 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb439400640 1 -- 172.21.15.67:0/439833610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 msgr2=0x7fb43c15e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb439400640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 0x7fb43c15e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb439400640 1 -- 172.21.15.67:0/439833610 >> v1:172.21.15.67:6789/0 conn(0x7fb43c0b3d10 legacy=0x7fb43c06d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb439400640 1 -- 172.21.15.67:0/439833610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c06e4d0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.121+0000 7fb43a800640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 0x7fb43c15e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:39.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.122+0000 7fb439400640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c09aa60 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb4240097a0 tx=0x7fb42402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e74e14cbd14ff28 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.122+0000 7fb438a00640 1 -- 172.21.15.67:0/439833610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424035020 con 0x7fb43c0d09f0 2024-04-17T08:30:39.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.122+0000 7fb438a00640 1 -- 172.21.15.67:0/439833610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb424031810 con 0x7fb43c0d09f0 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.122+0000 7fb438a00640 1 -- 172.21.15.67:0/439833610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424031be0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 -- 172.21.15.67:0/439833610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 msgr2=0x7fb43c09aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c09aa60 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb4240097a0 tx=0x7fb42402b8b0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 -- 172.21.15.67:0/439833610 shutdown_connections 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c09afa0 0x7fb43c15e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 --2- 172.21.15.67:0/439833610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c09aa60 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 -- 172.21.15.67:0/439833610 >> 172.21.15.67:0/439833610 conn(0x7fb43c0b41b0 msgr2=0x7fb43c0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 -- 172.21.15.67:0/439833610 shutdown_connections 2024-04-17T08:30:39.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.123+0000 7fb440a00640 1 -- 172.21.15.67:0/439833610 wait complete. 2024-04-17T08:30:39.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.125+0000 7fb440a00640 1 Processor -- start 2024-04-17T08:30:39.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.125+0000 7fb440a00640 1 -- start start 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 0x7fb43c0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c0b3d10 0x7fb43c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c15e870 con 0x7fb43c0d09f0 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c06d8d0 con 0x7fb43c09afa0 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb440a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb43c15e9f0 con 0x7fb43c0b3d10 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb43a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb43a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40472/0 (socket says 172.21.15.67:40472) 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.126+0000 7fb43a800640 1 -- 172.21.15.67:0/1660832506 learned_addr learned my addr 172.21.15.67:0/1660832506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb439e00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 0x7fb43c0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 -- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c0b3d10 msgr2=0x7fb43c159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c0b3d10 0x7fb43c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 -- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 msgr2=0x7fb43c0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 0x7fb43c0c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 -- 172.21.15.67:0/1660832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c0c8cd0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb439e00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 0x7fb43c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb43a800640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb43400cef0 tx=0x7fb43400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43401a070 con 0x7fb43c0d09f0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb43400d860 con 0x7fb43c0d09f0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43400dc50 con 0x7fb43c0d09f0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.127+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c0c76f0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.128+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c0c7cb0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.129+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c068b90 con 0x7fb43c0d09f0 2024-04-17T08:30:39.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.130+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fb43401d070 con 0x7fb43c0d09f0 2024-04-17T08:30:39.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.131+0000 7fb432000640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 0x7fb41806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:39.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.131+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5659+0+0 (secure 0 0 0) 0x7fb43408c5c0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.132+0000 7fb439e00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 0x7fb41806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:39.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.133+0000 7fb439e00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 0x7fb41806eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb42c000f60 tx=0x7fb42c009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:39.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.135+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb43405a9c0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.380+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb43c15a820 con 0x7fb43c0d09f0 2024-04-17T08:30:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.381+0000 7fb432000640 1 -- 172.21.15.67:0/1660832506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+3869 (secure 0 0 0) 0x7fb43405a2c0 con 0x7fb43c0d09f0 2024-04-17T08:30:39.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:39.381 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":78,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6662,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/1818051769","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":1818051769},{"type":"v1","addr":"172.21.15.187:6837","nonce":1818051769}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":7255,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/475393777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":475393777},{"type":"v1","addr":"172.21.15.187:6835","nonce":475393777}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":8952,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/931929987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":931929987},{"type":"v1","addr":"172.21.15.67:6834","nonce":931929987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71},{"gid":9360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4125331836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":4125331836},{"type":"v1","addr":"172.21.15.67:6835","nonce":4125331836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78}],"filesystems":[]} 2024-04-17T08:30:39.381 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 78 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 msgr2=0x7fb41806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 0x7fb41806eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb42c000f60 tx=0x7fb42c009040 comp rx=0 tx=0).stop 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 msgr2=0x7fb43c15a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fb43400cef0 tx=0x7fb43400a710 comp rx=0 tx=0).stop 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 shutdown_connections 2024-04-17T08:30:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fb41806ca10 0x7fb41806eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb43c0d09f0 0x7fb43c15a1c0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb43c0b3d10 0x7fb43c159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 --2- 172.21.15.67:0/1660832506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb43c09afa0 0x7fb43c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 >> 172.21.15.67:0/1660832506 conn(0x7fb43c0b41b0 msgr2=0x7fb43c0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 shutdown_connections 2024-04-17T08:30:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:39.384+0000 7fb440a00640 1 -- 172.21.15.67:0/1660832506 wait complete. 2024-04-17T08:30:39.408 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:30:39.408 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:30:39.408 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:30:39.408 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:30:39.408 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:30:39.408 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:30:39.409 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:30:39.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:30:39.409 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:30:39.409 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:30:39.409 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:30:39.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:30:39.409 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:30:39.410 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:30:39.410 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:30:39.410 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:30:39.431 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:30:39.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:30:39.434 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:30:39.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:30:39.459 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:30:39.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:30:39.467 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:30:39.467 DEBUG:teuthology.parallel:result is None 2024-04-17T08:30:39.479 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:30:39.479 DEBUG:teuthology.parallel:result is None 2024-04-17T08:30:39.483 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:30:39.483 DEBUG:teuthology.parallel:result is None 2024-04-17T08:30:39.643 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:30:39.646 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:30:39.650 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:30:43.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-17T08:30:45.459 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:30:45.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:30:45.468 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:30:45.469 DEBUG:teuthology.parallel:result is None 2024-04-17T08:30:45.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:30:45.649 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:30:45.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.703+0000 7f589a400640 1 Processor -- start 2024-04-17T08:30:45.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.703+0000 7f589a400640 1 -- start start 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f589a400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 0x7f589415f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f589a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58940ce510 0x7f58940ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f589a400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58940cb420 con 0x7f58940afbf0 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f589a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f589415fd60 con 0x7f58940ce510 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f589a400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f58940cc030 con 0x7f589415f3c0 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f5899000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58940afbf0 0x7f589406e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42032/0 (socket says 172.21.15.67:42032) 2024-04-17T08:30:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f5899000640 1 -- 172.21.15.67:0/1326506631 learned_addr learned my addr 172.21.15.67:0/1326506631 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f5899a00640 1 --2- 172.21.15.67:0/1326506631 >> v2:172.21.15.67:3300/0 conn(0x7f58940ce510 0x7f58940ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.705+0000 7f5893e00640 1 --2- 172.21.15.67:0/1326506631 >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 0x7f589415f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.706+0000 7f5899a00640 1 -- 172.21.15.67:0/1326506631 >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 msgr2=0x7f589415f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.706+0000 7f5899a00640 1 --2- 172.21.15.67:0/1326506631 >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 0x7f589415f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.706+0000 7f5899a00640 1 -- 172.21.15.67:0/1326506631 >> v1:172.21.15.67:6789/0 conn(0x7f58940afbf0 legacy=0x7f589406e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.706+0000 7f5899a00640 1 -- 172.21.15.67:0/1326506631 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58940cb700 con 0x7f58940ce510 2024-04-17T08:30:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.706+0000 7f5893e00640 1 --2- 172.21.15.67:0/1326506631 >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 0x7f589415f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.707+0000 7f5899a00640 1 --2- 172.21.15.67:0/1326506631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940ce510 0x7f58940ce960 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f58940d0450 tx=0x7f588c02dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c991bc91d946a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.707+0000 7f5893400640 1 -- 172.21.15.67:0/1326506631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c02f8a0 con 0x7f58940ce510 2024-04-17T08:30:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.707+0000 7f5893400640 1 -- 172.21.15.67:0/1326506631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c02fa40 con 0x7f58940ce510 2024-04-17T08:30:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.707+0000 7f5893400640 1 -- 172.21.15.67:0/1326506631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c02fe10 con 0x7f58940ce510 2024-04-17T08:30:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 -- 172.21.15.67:0/1326506631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940ce510 msgr2=0x7f58940ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 --2- 172.21.15.67:0/1326506631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940ce510 0x7f58940ce960 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f58940d0450 tx=0x7f588c02dae0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 -- 172.21.15.67:0/1326506631 shutdown_connections 2024-04-17T08:30:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 --2- 172.21.15.67:0/1326506631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940ce510 0x7f58940ce960 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 --2- 172.21.15.67:0/1326506631 >> v2:172.21.15.187:3300/0 conn(0x7f589415f3c0 0x7f589415f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 -- 172.21.15.67:0/1326506631 >> 172.21.15.67:0/1326506631 conn(0x7f589406bef0 msgr2=0x7f58940b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 -- 172.21.15.67:0/1326506631 shutdown_connections 2024-04-17T08:30:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.708+0000 7f589a400640 1 -- 172.21.15.67:0/1326506631 wait complete. 2024-04-17T08:30:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.710+0000 7f589a400640 1 Processor -- start 2024-04-17T08:30:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.711+0000 7f589a400640 1 -- start start 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58940ce510 0x7f5894157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f589415f3c0 0x7f58940c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f589406f090 con 0x7f58940afbf0 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f58940cbcc0 con 0x7f58940ce510 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f589a400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5894160110 con 0x7f589415f3c0 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f5899000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f5899000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37434/0 (socket says 172.21.15.67:37434) 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.712+0000 7f5899000640 1 -- 172.21.15.67:0/1620383425 learned_addr learned my addr 172.21.15.67:0/1620383425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5893e00640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58940ce510 0x7f5894157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899a00640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f589415f3c0 0x7f58940c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 -- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f589415f3c0 msgr2=0x7f58940c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f589415f3c0 0x7f58940c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 -- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58940ce510 msgr2=0x7f5894157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58940ce510 0x7f5894157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:45.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 -- 172.21.15.67:0/1620383425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58941565f0 con 0x7f58940afbf0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.713+0000 7f5899000640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f588400c3c0 tx=0x7f588400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.714+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884017070 con 0x7f58940afbf0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.714+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5884005ce0 con 0x7f58940afbf0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.714+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58840072f0 con 0x7f58940afbf0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.714+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58941568d0 con 0x7f58940afbf0 2024-04-17T08:30:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.714+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5894156e90 con 0x7f58940afbf0 2024-04-17T08:30:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.715+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5894068a10 con 0x7f58940afbf0 2024-04-17T08:30:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.716+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f588401a070 con 0x7f58940afbf0 2024-04-17T08:30:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.717+0000 7f5890c00640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 0x7f587006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.717+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5659+0+0 (secure 0 0 0) 0x7f588408b120 con 0x7f58940afbf0 2024-04-17T08:30:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.718+0000 7f5893e00640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 0x7f587006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:45.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.719+0000 7f5893e00640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 0x7f587006eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f587c0075b0 tx=0x7f587c007540 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.722+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5884059390 con 0x7f58940afbf0 2024-04-17T08:30:45.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:45.940+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f58940b5d60 con 0x7f58940afbf0 2024-04-17T08:30:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.041+0000 7f5890c00640 1 -- 172.21.15.67:0/1620383425 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v178) ==== 97+0+0 (secure 0 0 0) 0x7f5884058c90 con 0x7f58940afbf0 2024-04-17T08:30:47.040 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.045+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 msgr2=0x7f587006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.045+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 0x7f587006eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f587c0075b0 tx=0x7f587c007540 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.045+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 msgr2=0x7f5894155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.045+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f588400c3c0 tx=0x7f588400c880 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 shutdown_connections 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f587006ca10 0x7f587006eed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f589415f3c0 0x7f58940c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58940ce510 0x7f5894157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 --2- 172.21.15.67:0/1620383425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58940afbf0 0x7f5894155f60 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 >> 172.21.15.67:0/1620383425 conn(0x7f589406bef0 msgr2=0x7f5894074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:47.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 shutdown_connections 2024-04-17T08:30:47.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.046+0000 7f589a400640 1 -- 172.21.15.67:0/1620383425 wait complete. 2024-04-17T08:30:47.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:30:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.291+0000 7ff4f4a00640 1 Processor -- start 2024-04-17T08:30:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.291+0000 7ff4f4a00640 1 -- start start 2024-04-17T08:30:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4f4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4f01675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4f4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f015cd40 con 0x7ff4f01452f0 2024-04-17T08:30:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4f4a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f015d020 con 0x7ff4f0160e50 2024-04-17T08:30:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4f4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f015d670 con 0x7ff4f0072470 2024-04-17T08:30:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4ef400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4f01452f0 0x7ff4f0145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42042/0 (socket says 172.21.15.67:42042) 2024-04-17T08:30:47.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.293+0000 7ff4ef400640 1 -- 172.21.15.67:0/3997594968 learned_addr learned my addr 172.21.15.67:0/3997594968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:47.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4efe00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4f01675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583914749 0 0) 0x7ff4f015cd40 con 0x7ff4f01452f0 2024-04-17T08:30:47.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d4003520 con 0x7ff4f01452f0 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 195999040 0 0) 0x7ff4d4003520 con 0x7ff4f01452f0 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 msgr2=0x7ff4f01675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4f01675b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 >> v1:172.21.15.187:6789/0 conn(0x7ff4f0160e50 legacy=0x7ff4f0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f015d300 con 0x7ff4f01452f0 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff4e0001690 con 0x7ff4f01452f0 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 >> v1:172.21.15.67:6789/0 conn(0x7ff4f01452f0 legacy=0x7ff4f0145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d40039e0 0x7ff4d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d4004800 0x7ff4d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4d4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f015d670 con 0x7ff4d4004800 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.294+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d400bb80 con 0x7ff4d40039e0 2024-04-17T08:30:47.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d400bd80 con 0x7ff4f0072470 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d4004800 0x7ff4d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4efe00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4d4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4ef400640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d40039e0 0x7ff4d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 -- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 msgr2=0x7ff4d4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f0072470 0x7ff4d4005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 -- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d40039e0 msgr2=0x7ff4d4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d40039e0 0x7ff4d4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4eea00640 1 -- 172.21.15.67:0/3997594968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d4003520 con 0x7ff4d4004800 2024-04-17T08:30:47.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.295+0000 7ff4ef400640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4d40039e0 0x7ff4d4003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d4004800 0x7ff4d4004c80 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc001750 tx=0x7ff4d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3db7a3a4d000a17 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc001370 con 0x7ff4d4004800 2024-04-17T08:30:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0052d0 con 0x7ff4d4004800 2024-04-17T08:30:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4ee000640 1 -- 172.21.15.67:0/3997594968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc003830 con 0x7ff4d4004800 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3997594968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d4004800 msgr2=0x7ff4d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3997594968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4d4004800 0x7ff4d4004c80 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc001750 tx=0x7ff4d4005830 comp rx=0 tx=0).stop 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4efe00640 1 -- 172.21.15.67:0/3997594968 reap_dead start 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3997594968 shutdown_connections 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.297+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3997594968 >> 172.21.15.67:0/3997594968 conn(0x7ff4f00b5f70 msgr2=0x7ff4f006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.298+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3997594968 shutdown_connections 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.298+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3997594968 wait complete. 2024-04-17T08:30:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.299+0000 7ff4f4a00640 1 Processor -- start 2024-04-17T08:30:47.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.299+0000 7ff4f4a00640 1 -- start start 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4f00b1590 0x7ff4f00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 0x7ff4f014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f015eb10 con 0x7ff4f014df00 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01456d0 con 0x7ff4f00b1590 2024-04-17T08:30:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4f4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f015ec90 con 0x7ff4f013b3b0 2024-04-17T08:30:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37454/0 (socket says 172.21.15.67:37454) 2024-04-17T08:30:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4efe00640 1 -- 172.21.15.67:0/3294385039 learned_addr learned my addr 172.21.15.67:0/3294385039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 0x7ff4f014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.301+0000 7ff4ef400640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4f00b1590 0x7ff4f00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 -- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 msgr2=0x7ff4f014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 0x7ff4f014d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 -- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4f00b1590 msgr2=0x7ff4f00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4f00b1590 0x7ff4f00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 -- 172.21.15.67:0/3294385039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f013d020 con 0x7ff4f014df00 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4eea00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 0x7ff4f014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:47.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4efe00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0072940 tx=0x7ff4e80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8011e40 con 0x7ff4f014df00 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e801b070 con 0x7ff4f014df00 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f013b920 con 0x7ff4f014df00 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8003990 con 0x7ff4f014df00 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.302+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f013d5a0 con 0x7ff4f014df00 2024-04-17T08:30:47.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.303+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f0068a10 con 0x7ff4f014df00 2024-04-17T08:30:47.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.305+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7ff4e8005d10 con 0x7ff4f014df00 2024-04-17T08:30:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.306+0000 7ff4e7400640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 0x7ff4d006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.306+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4885+0+0 (secure 0 0 0) 0x7ff4e808bff0 con 0x7ff4f014df00 2024-04-17T08:30:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.306+0000 7ff4ef400640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 0x7ff4d006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.308+0000 7ff4ef400640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 0x7ff4d006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0000fd0 tx=0x7ff4e0009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.311+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff4e801d310 con 0x7ff4f014df00 2024-04-17T08:30:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.595+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff4f0072e50 con 0x7ff4f014df00 2024-04-17T08:30:47.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.596+0000 7ff4e7400640 1 -- 172.21.15.67:0/3294385039 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) ==== 80+0+5304 (secure 0 0 0) 0x7ff4e805a490 con 0x7ff4f014df00 2024-04-17T08:30:47.594 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:47.594 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:30:47.595 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:30:47.596 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:30:47.597 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:30:47.603 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:30:47.604 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.605 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:30:47.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:30:47.607 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:30:47.608 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 msgr2=0x7ff4d006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 0x7ff4d006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0000fd0 tx=0x7ff4e0009040 comp rx=0 tx=0).stop 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 msgr2=0x7ff4f013dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0072940 tx=0x7ff4e80115a0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 shutdown_connections 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff4d006ca10 0x7ff4d006eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4f014df00 0x7ff4f013dda0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff4f013b3b0 0x7ff4f014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 --2- 172.21.15.67:0/3294385039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4f00b1590 0x7ff4f00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 >> 172.21.15.67:0/3294385039 conn(0x7ff4f00b5f70 msgr2=0x7ff4f015f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:47.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 shutdown_connections 2024-04-17T08:30:47.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.599+0000 7ff4f4a00640 1 -- 172.21.15.67:0/3294385039 wait complete. 2024-04-17T08:30:47.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.840+0000 7ff27d400640 1 Processor -- start 2024-04-17T08:30:47.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.840+0000 7ff27d400640 1 -- start start 2024-04-17T08:30:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff27d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff278160280 0x7ff27806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff27d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff278141ed0 0x7ff278142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff27d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2780cd4a0 con 0x7ff27806b8c0 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff27d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff27815ce30 con 0x7ff278160280 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff27d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff2781607f0 con 0x7ff278141ed0 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff26f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff278160280 0x7ff27806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff26f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff278160280 0x7ff27806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37460/0 (socket says 172.21.15.67:37460) 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff277e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff27806b8c0 0x7ff2780cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42050/0 (socket says 172.21.15.67:42050) 2024-04-17T08:30:47.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.842+0000 7ff26f400640 1 -- 172.21.15.67:0/2767669324 learned_addr learned my addr 172.21.15.67:0/2767669324 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.843+0000 7ff26f400640 1 -- 172.21.15.67:0/2767669324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff278141ed0 msgr2=0x7ff278142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.843+0000 7ff26f400640 1 --2- 172.21.15.67:0/2767669324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff278141ed0 0x7ff278142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.843+0000 7ff26f400640 1 -- 172.21.15.67:0/2767669324 >> v1:172.21.15.67:6789/0 conn(0x7ff27806b8c0 legacy=0x7ff2780cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.843+0000 7ff26f400640 1 -- 172.21.15.67:0/2767669324 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff278160ad0 con 0x7ff278160280 2024-04-17T08:30:47.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.844+0000 7ff26f400640 1 --2- 172.21.15.67:0/2767669324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278160280 0x7ff27806bca0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7ff260009870 tx=0x7ff26002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904150459ef7241a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.844+0000 7ff277400640 1 -- 172.21.15.67:0/2767669324 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260034070 con 0x7ff278160280 2024-04-17T08:30:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff277400640 1 -- 172.21.15.67:0/2767669324 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff260030840 con 0x7ff278160280 2024-04-17T08:30:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff277400640 1 -- 172.21.15.67:0/2767669324 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260030c10 con 0x7ff278160280 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 -- 172.21.15.67:0/2767669324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278160280 msgr2=0x7ff27806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 --2- 172.21.15.67:0/2767669324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278160280 0x7ff27806bca0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7ff260009870 tx=0x7ff26002b980 comp rx=0 tx=0).stop 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 -- 172.21.15.67:0/2767669324 shutdown_connections 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 --2- 172.21.15.67:0/2767669324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff278141ed0 0x7ff278142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 --2- 172.21.15.67:0/2767669324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278160280 0x7ff27806bca0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 -- 172.21.15.67:0/2767669324 >> 172.21.15.67:0/2767669324 conn(0x7ff278067d10 msgr2=0x7ff2780b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 -- 172.21.15.67:0/2767669324 shutdown_connections 2024-04-17T08:30:47.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.845+0000 7ff27d400640 1 -- 172.21.15.67:0/2767669324 wait complete. 2024-04-17T08:30:47.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.847+0000 7ff27d400640 1 Processor -- start 2024-04-17T08:30:47.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.847+0000 7ff27d400640 1 -- start start 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff27806b8c0 0x7ff278155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff278160280 0x7ff27815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff278161090 con 0x7ff278141ed0 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff27806c1e0 con 0x7ff278160280 2024-04-17T08:30:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff27d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff27813f9b0 con 0x7ff27806b8c0 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff26f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff26f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37474/0 (socket says 172.21.15.67:37474) 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.848+0000 7ff26f400640 1 -- 172.21.15.67:0/3340943389 learned_addr learned my addr 172.21.15.67:0/3340943389 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff277e00640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff27806b8c0 0x7ff278155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff27ca00640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff278160280 0x7ff27815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 -- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff27806b8c0 msgr2=0x7ff278155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff27806b8c0 0x7ff278155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 -- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff278160280 msgr2=0x7ff27815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff278160280 0x7ff27815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 -- 172.21.15.67:0/3340943389 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2780b4d30 con 0x7ff278141ed0 2024-04-17T08:30:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.849+0000 7ff26f400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff2600090b0 tx=0x7ff2600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.850+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260034040 con 0x7ff278141ed0 2024-04-17T08:30:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.850+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2600443f0 con 0x7ff278141ed0 2024-04-17T08:30:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.850+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2600447e0 con 0x7ff278141ed0 2024-04-17T08:30:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.850+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2780b5010 con 0x7ff278141ed0 2024-04-17T08:30:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.850+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27806df90 con 0x7ff278141ed0 2024-04-17T08:30:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.851+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27815f2d0 con 0x7ff278141ed0 2024-04-17T08:30:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.852+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7ff26004b060 con 0x7ff278141ed0 2024-04-17T08:30:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.853+0000 7ff274c00640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 0x7ff25406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.853+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4885+0+0 (secure 0 0 0) 0x7ff2600ade30 con 0x7ff278141ed0 2024-04-17T08:30:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.854+0000 7ff277e00640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 0x7ff25406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:47.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.855+0000 7ff277e00640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 0x7ff25406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff268000f60 tx=0x7ff268007cc0 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:47.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:47.858+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff27815f2d0 con 0x7ff278141ed0 2024-04-17T08:30:48.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.067+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff278160040 con 0x7ff278141ed0 2024-04-17T08:30:48.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.069+0000 7ff274c00640 1 -- 172.21.15.67:0/3340943389 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) ==== 74+0+12897 (secure 0 0 0) 0x7ff278160040 con 0x7ff278141ed0 2024-04-17T08:30:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:48.068 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":178,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:47.038356+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:48.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 msgr2=0x7ff25406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 0x7ff25406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff268000f60 tx=0x7ff268007cc0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 msgr2=0x7ff2780b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff2600090b0 tx=0x7ff2600046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 shutdown_connections 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7ff25406ca10 0x7ff25406eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff278160280 0x7ff27815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278141ed0 0x7ff2780b4290 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 --2- 172.21.15.67:0/3340943389 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff27806b8c0 0x7ff278155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.072+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 >> 172.21.15.67:0/3340943389 conn(0x7ff278067d10 msgr2=0x7ff278156840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.073+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 shutdown_connections 2024-04-17T08:30:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.073+0000 7ff27d400640 1 -- 172.21.15.67:0/3340943389 wait complete. 2024-04-17T08:30:48.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:30:48.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:30:48.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.316+0000 7fd14f200640 1 Processor -- start 2024-04-17T08:30:48.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.317+0000 7fd14f200640 1 -- start start 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1500b3b50 0x7fd1500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1500cfcd0 0x7fd1500d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1500d4f70 con 0x7fd15015ec20 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd1500d5250 con 0x7fd1500b3b50 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1500cf4d0 con 0x7fd1500cfcd0 2024-04-17T08:30:48.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.318+0000 7fd14d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1500b3b50 0x7fd1500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1500b3b50 0x7fd1500d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37486/0 (socket says 172.21.15.67:37486) 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 -- 172.21.15.67:0/1431726610 learned_addr learned my addr 172.21.15.67:0/1431726610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14e800640 1 --2- 172.21.15.67:0/1431726610 >> v2:172.21.15.187:3300/0 conn(0x7fd1500cfcd0 0x7fd1500d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 -- 172.21.15.67:0/1431726610 >> v2:172.21.15.187:3300/0 conn(0x7fd1500cfcd0 msgr2=0x7fd1500d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 --2- 172.21.15.67:0/1431726610 >> v2:172.21.15.187:3300/0 conn(0x7fd1500cfcd0 0x7fd1500d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 -- 172.21.15.67:0/1431726610 >> v1:172.21.15.187:6789/0 conn(0x7fd15015ec20 legacy=0x7fd15006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.319+0000 7fd14d400640 1 -- 172.21.15.67:0/1431726610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1500cf7b0 con 0x7fd1500b3b50 2024-04-17T08:30:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.320+0000 7fd14d400640 1 --2- 172.21.15.67:0/1431726610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1500b3b50 0x7fd1500d22b0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd15008c640 tx=0x7fd13c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62d710229fa5e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.321+0000 7fd14ca00640 1 -- 172.21.15.67:0/1431726610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c038070 con 0x7fd1500b3b50 2024-04-17T08:30:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.321+0000 7fd14ca00640 1 -- 172.21.15.67:0/1431726610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd13c004230 con 0x7fd1500b3b50 2024-04-17T08:30:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.321+0000 7fd14ca00640 1 -- 172.21.15.67:0/1431726610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c02f050 con 0x7fd1500b3b50 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.321+0000 7fd14f200640 1 -- 172.21.15.67:0/1431726610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1500b3b50 msgr2=0x7fd1500d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.321+0000 7fd14f200640 1 --2- 172.21.15.67:0/1431726610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1500b3b50 0x7fd1500d22b0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fd15008c640 tx=0x7fd13c02b7e0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 -- 172.21.15.67:0/1431726610 shutdown_connections 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 --2- 172.21.15.67:0/1431726610 >> v2:172.21.15.187:3300/0 conn(0x7fd1500cfcd0 0x7fd1500d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 --2- 172.21.15.67:0/1431726610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1500b3b50 0x7fd1500d22b0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 -- 172.21.15.67:0/1431726610 >> 172.21.15.67:0/1431726610 conn(0x7fd15008bff0 msgr2=0x7fd1500d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 -- 172.21.15.67:0/1431726610 shutdown_connections 2024-04-17T08:30:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.322+0000 7fd14f200640 1 -- 172.21.15.67:0/1431726610 wait complete. 2024-04-17T08:30:48.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.324+0000 7fd14f200640 1 Processor -- start 2024-04-17T08:30:48.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.324+0000 7fd14f200640 1 -- start start 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1500b3b50 0x7fd15015fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1500cfcd0 0x7fd1500cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd15006d2d0 con 0x7fd15015ec20 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd15006d150 con 0x7fd1500b3b50 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1500d10d0 con 0x7fd1500cfcd0 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.325+0000 7fd14e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37496/0 (socket says 172.21.15.67:37496) 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 -- 172.21.15.67:0/236594395 learned_addr learned my addr 172.21.15.67:0/236594395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14d400640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1500cfcd0 0x7fd1500cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14de00640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1500b3b50 0x7fd15015fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 -- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1500cfcd0 msgr2=0x7fd1500cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1500cfcd0 0x7fd1500cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 -- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1500b3b50 msgr2=0x7fd15015fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1500b3b50 0x7fd15015fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 -- 172.21.15.67:0/236594395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1501600a0 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.326+0000 7fd14e800640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd148007d90 tx=0x7fd14800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.327+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148014420 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.327+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd148019070 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.327+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd150160380 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.327+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148014820 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.327+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd150160940 con 0x7fd15015ec20 2024-04-17T08:30:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.328+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd150068a10 con 0x7fd15015ec20 2024-04-17T08:30:48.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.329+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fd1480149c0 con 0x7fd15015ec20 2024-04-17T08:30:48.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.330+0000 7fd146000640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 0x7fd13006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.330+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4885+0+0 (secure 0 0 0) 0x7fd14808c2b0 con 0x7fd15015ec20 2024-04-17T08:30:48.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.331+0000 7fd14de00640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 0x7fd13006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.332+0000 7fd14de00640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 0x7fd13006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1500b9d00 tx=0x7fd140008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.334+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd14805a860 con 0x7fd15015ec20 2024-04-17T08:30:48.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.541+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1500d55f0 con 0x7fd15015ec20 2024-04-17T08:30:48.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.542+0000 7fd146000640 1 -- 172.21.15.67:0/236594395 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) ==== 93+0+10 (secure 0 0 0) 0x7fd14805a160 con 0x7fd15015ec20 2024-04-17T08:30:48.540 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:30:48.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 msgr2=0x7fd13006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 0x7fd13006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1500b9d00 tx=0x7fd140008040 comp rx=0 tx=0).stop 2024-04-17T08:30:48.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 msgr2=0x7fd1501f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd148007d90 tx=0x7fd14800e4b0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 shutdown_connections 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd13006ca10 0x7fd13006eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd15015ec20 0x7fd1501f4440 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1500cfcd0 0x7fd1500cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 --2- 172.21.15.67:0/236594395 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1500b3b50 0x7fd15015fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 >> 172.21.15.67:0/236594395 conn(0x7fd15008bff0 msgr2=0x7fd1500d3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 shutdown_connections 2024-04-17T08:30:48.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.545+0000 7fd14f200640 1 -- 172.21.15.67:0/236594395 wait complete. 2024-04-17T08:30:48.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:30:48.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.792+0000 7fda17000640 1 Processor -- start 2024-04-17T08:30:48.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.792+0000 7fda17000640 1 -- start start 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda17000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda1015d530 0x7fda10164480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda17000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fda100b3490 0x7fda1006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda17000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1006ead0 con 0x7fda100b3910 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda17000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fda1006f6e0 con 0x7fda1015d530 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda17000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fda1006edb0 con 0x7fda100b3490 2024-04-17T08:30:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda15c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda1015d530 0x7fda10164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda15c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda1015d530 0x7fda10164480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37500/0 (socket says 172.21.15.67:37500) 2024-04-17T08:30:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.794+0000 7fda15c00640 1 -- 172.21.15.67:0/2201010846 learned_addr learned my addr 172.21.15.67:0/2201010846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.795+0000 7fda15200640 1 --2- 172.21.15.67:0/2201010846 >> v2:172.21.15.187:3300/0 conn(0x7fda100b3490 0x7fda1006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.795+0000 7fda15c00640 1 -- 172.21.15.67:0/2201010846 >> v2:172.21.15.187:3300/0 conn(0x7fda100b3490 msgr2=0x7fda1006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.795+0000 7fda15c00640 1 --2- 172.21.15.67:0/2201010846 >> v2:172.21.15.187:3300/0 conn(0x7fda100b3490 0x7fda1006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.795+0000 7fda15c00640 1 -- 172.21.15.67:0/2201010846 >> v1:172.21.15.187:6789/0 conn(0x7fda100b3910 legacy=0x7fda101634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.795+0000 7fda15c00640 1 -- 172.21.15.67:0/2201010846 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1006f090 con 0x7fda1015d530 2024-04-17T08:30:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.797+0000 7fda15c00640 1 --2- 172.21.15.67:0/2201010846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda10164480 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fda100b3010 tx=0x7fda0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c52ddd157781ad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.797+0000 7fda0fe00640 1 -- 172.21.15.67:0/2201010846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04035020 con 0x7fda1015d530 2024-04-17T08:30:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.797+0000 7fda0fe00640 1 -- 172.21.15.67:0/2201010846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda04031870 con 0x7fda1015d530 2024-04-17T08:30:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.797+0000 7fda0fe00640 1 -- 172.21.15.67:0/2201010846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04031c40 con 0x7fda1015d530 2024-04-17T08:30:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 -- 172.21.15.67:0/2201010846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 msgr2=0x7fda10164480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 --2- 172.21.15.67:0/2201010846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda10164480 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fda100b3010 tx=0x7fda0402b910 comp rx=0 tx=0).stop 2024-04-17T08:30:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 -- 172.21.15.67:0/2201010846 shutdown_connections 2024-04-17T08:30:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 --2- 172.21.15.67:0/2201010846 >> v2:172.21.15.187:3300/0 conn(0x7fda100b3490 0x7fda1006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 --2- 172.21.15.67:0/2201010846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda10164480 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 -- 172.21.15.67:0/2201010846 >> 172.21.15.67:0/2201010846 conn(0x7fda10067d10 msgr2=0x7fda100b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 -- 172.21.15.67:0/2201010846 shutdown_connections 2024-04-17T08:30:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.798+0000 7fda17000640 1 -- 172.21.15.67:0/2201010846 wait complete. 2024-04-17T08:30:48.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.800+0000 7fda17000640 1 Processor -- start 2024-04-17T08:30:48.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.800+0000 7fda17000640 1 -- start start 2024-04-17T08:30:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 0x7fda101f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda100b3910 0x7fda100a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda1006fee0 con 0x7fda1015d530 2024-04-17T08:30:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fda1006fd60 con 0x7fda100b3910 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.801+0000 7fda17000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fda1006be00 con 0x7fda100b3490 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37512/0 (socket says 172.21.15.67:37512) 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 -- 172.21.15.67:0/3738850654 learned_addr learned my addr 172.21.15.67:0/3738850654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda15c00640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 0x7fda101f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda15200640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda100b3910 0x7fda100a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 -- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 msgr2=0x7fda101f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 0x7fda101f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 -- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda100b3910 msgr2=0x7fda100a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda100b3910 0x7fda100a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda16600640 1 -- 172.21.15.67:0/3738850654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1015e5d0 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.802+0000 7fda15c00640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 0x7fda101f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda16600640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c700 tx=0x7fda0800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08016020 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0800aa50 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0800ce10 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1015e8b0 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.803+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1015ee70 con 0x7fda1015d530 2024-04-17T08:30:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.804+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda1015f250 con 0x7fda1015d530 2024-04-17T08:30:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.805+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fda08019020 con 0x7fda1015d530 2024-04-17T08:30:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.806+0000 7fda0d600640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 0x7fd9f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.806+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4885+0+0 (secure 0 0 0) 0x7fda0808bcf0 con 0x7fda1015d530 2024-04-17T08:30:48.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.807+0000 7fda15c00640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 0x7fd9f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:48.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.808+0000 7fda15c00640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 0x7fd9f006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda04036000 tx=0x7fda0403c040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:48.811+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fda08020020 con 0x7fda1015d530 2024-04-17T08:30:48.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:30:48.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:30:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.024+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fda10164b50 con 0x7fda1015d530 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.026+0000 7fda0d600640 1 -- 172.21.15.67:0/3738850654 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v178) ==== 84+0+65 (secure 0 0 0) 0x7fda0805a330 con 0x7fda1015d530 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:30:49.024 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.028+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 msgr2=0x7fd9f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.028+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 0x7fd9f006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda04036000 tx=0x7fda0403c040 comp rx=0 tx=0).stop 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.028+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 msgr2=0x7fda100a9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.028+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c700 tx=0x7fda0800cbc0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 shutdown_connections 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fd9f006ca10 0x7fd9f006eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda1015d530 0x7fda100a9a00 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda100b3910 0x7fda100a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 --2- 172.21.15.67:0/3738850654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fda100b3490 0x7fda101f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 >> 172.21.15.67:0/3738850654 conn(0x7fda10067d10 msgr2=0x7fda10074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 shutdown_connections 2024-04-17T08:30:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.029+0000 7fda17000640 1 -- 172.21.15.67:0/3738850654 wait complete. 2024-04-17T08:30:49.048 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:30:49.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:30:49.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.320+0000 7fa7c0c00640 1 Processor -- start 2024-04-17T08:30:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.320+0000 7fa7c0c00640 1 -- start start 2024-04-17T08:30:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7c0c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa7bc162ae0 0x7fa7bc160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7c0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7bc0b4a70 con 0x7fa7bc162690 2024-04-17T08:30:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7c0c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7bc15fc40 con 0x7fa7bc1484e0 2024-04-17T08:30:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7c0c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa7bc160290 con 0x7fa7bc162ae0 2024-04-17T08:30:49.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7b3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7bc162690 0x7fa7bc160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42056/0 (socket says 172.21.15.67:42056) 2024-04-17T08:30:49.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.322+0000 7fa7b3e00640 1 -- 172.21.15.67:0/1028684849 learned_addr learned my addr 172.21.15.67:0/1028684849 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7ba800640 1 --2- 172.21.15.67:0/1028684849 >> v2:172.21.15.187:3300/0 conn(0x7fa7bc162ae0 0x7fa7bc160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260830785 0 0) 0x7fa7bc0b4a70 con 0x7fa7bc162690 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a0003520 con 0x7fa7bc162690 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3152661020 0 0) 0x7fa7a0003520 con 0x7fa7bc162690 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 >> v2:172.21.15.187:3300/0 conn(0x7fa7bc162ae0 msgr2=0x7fa7bc160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 --2- 172.21.15.67:0/1028684849 >> v2:172.21.15.187:3300/0 conn(0x7fa7bc162ae0 0x7fa7bc160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 >> v1:172.21.15.187:6789/0 conn(0x7fa7bc1484e0 legacy=0x7fa7bc0b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc15ff20 con 0x7fa7bc162690 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa7a4001670 con 0x7fa7bc162690 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 >> v1:172.21.15.67:6789/0 conn(0x7fa7bc162690 legacy=0x7fa7bc160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0003a00 0x7fa7a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 0x7fa7a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7a0005210 0x7fa7a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc160290 con 0x7fa7a0004800 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a000c160 con 0x7fa7a0005210 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.323+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a000c360 con 0x7fa7a0003a00 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 0x7fa7a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 -- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0003a00 msgr2=0x7fa7a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b9e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0003a00 0x7fa7a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7ba800640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7a0005210 0x7fa7a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0003a00 0x7fa7a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 -- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7a0005210 msgr2=0x7fa7a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:49.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7a0005210 0x7fa7a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b3e00640 1 -- 172.21.15.67:0/1028684849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a0003520 con 0x7fa7a0004800 2024-04-17T08:30:49.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7b9e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0003a00 0x7fa7a0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:49.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7ba800640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7a0005210 0x7fa7a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:49.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.324+0000 7fa7ba800640 1 -- 172.21.15.67:0/1028684849 reap_dead start 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 0x7fa7a0004c80 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31dde1ccf4e746d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4007c60 con 0x7fa7a0004800 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a4030070 con 0x7fa7a0004800 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7b9400640 1 -- 172.21.15.67:0/1028684849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4005d90 con 0x7fa7a0004800 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7c0c00640 1 -- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 msgr2=0x7fa7a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.326+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 0x7fa7a0004c80 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c190 tx=0x7fa7a0005830 comp rx=0 tx=0).stop 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.327+0000 7fa7c0c00640 1 -- 172.21.15.67:0/1028684849 shutdown_connections 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.327+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/1028684849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a0004800 0x7fa7a0004c80 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.327+0000 7fa7c0c00640 1 -- 172.21.15.67:0/1028684849 >> 172.21.15.67:0/1028684849 conn(0x7fa7bc0b72d0 msgr2=0x7fa7bc09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.327+0000 7fa7c0c00640 1 -- 172.21.15.67:0/1028684849 shutdown_connections 2024-04-17T08:30:49.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.327+0000 7fa7c0c00640 1 -- 172.21.15.67:0/1028684849 wait complete. 2024-04-17T08:30:49.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.328+0000 7fa7c0c00640 1 Processor -- start 2024-04-17T08:30:49.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.329+0000 7fa7c0c00640 1 -- start start 2024-04-17T08:30:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0004800 0x7fa7bc151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 0x7fa7bc13ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b4ed0 con 0x7fa7bc15eea0 2024-04-17T08:30:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b4d50 con 0x7fa7bc13d280 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7c0c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa7bc0b4640 con 0x7fa7a0004800 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37532/0 (socket says 172.21.15.67:37532) 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 -- 172.21.15.67:0/382200302 learned_addr learned my addr 172.21.15.67:0/382200302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7b9e00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0004800 0x7fa7bc151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 0x7fa7bc13ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 -- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0004800 msgr2=0x7fa7bc151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0004800 0x7fa7bc151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 -- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 msgr2=0x7fa7bc13ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.330+0000 7fa7ba800640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 0x7fa7bc13ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7ba800640 1 -- 172.21.15.67:0/382200302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc13ee90 con 0x7fa7bc15eea0 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7b3e00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 0x7fa7bc13ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7ba800640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0b29c0 tx=0x7fa7b4009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4019070 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b4007d40 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc13f170 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4014640 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.331+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc13f730 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.332+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7bc09f930 con 0x7fa7bc15eea0 2024-04-17T08:30:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.333+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fa7b401c070 con 0x7fa7bc15eea0 2024-04-17T08:30:49.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.334+0000 7fa7b2000640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 0x7fa79806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:49.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.334+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4885+0+0 (secure 0 0 0) 0x7fa7b408b620 con 0x7fa7bc15eea0 2024-04-17T08:30:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.335+0000 7fa7b9e00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 0x7fa79806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:49.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.336+0000 7fa7b9e00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 0x7fa79806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0516a0 tx=0x7fa7ac009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:49.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.339+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa7b4059c60 con 0x7fa7bc15eea0 2024-04-17T08:30:49.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:49.545+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa7bc068d60 con 0x7fa7bc15eea0 2024-04-17T08:30:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.094+0000 7fa7b2000640 1 -- 172.21.15.67:0/382200302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v179) ==== 138+0+0 (secure 0 0 0) 0x7fa7b4059560 con 0x7fa7bc15eea0 2024-04-17T08:30:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:30:50.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.097+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 msgr2=0x7fa79806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.097+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 0x7fa79806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0516a0 tx=0x7fa7ac009040 comp rx=0 tx=0).stop 2024-04-17T08:30:50.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.097+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 msgr2=0x7fa7bc15f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.097+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0b29c0 tx=0x7fa7b4009cd0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 shutdown_connections 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fa79806ca10 0x7fa79806eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7bc15eea0 0x7fa7bc15f300 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7bc13d280 0x7fa7bc13ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 --2- 172.21.15.67:0/382200302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa7a0004800 0x7fa7bc151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 >> 172.21.15.67:0/382200302 conn(0x7fa7bc0b72d0 msgr2=0x7fa7bc06f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 shutdown_connections 2024-04-17T08:30:50.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.098+0000 7fa7c0c00640 1 -- 172.21.15.67:0/382200302 wait complete. 2024-04-17T08:30:50.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:30:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.358+0000 7fca12a00640 1 Processor -- start 2024-04-17T08:30:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.358+0000 7fca12a00640 1 -- start start 2024-04-17T08:30:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 0x7fca0c0b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca0c0b25f0 0x7fca0c142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c15cbe0 con 0x7fca0c13f6d0 2024-04-17T08:30:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fca0c0b2a60 con 0x7fca0c0b25f0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fca0c072190 con 0x7fca0c071d00 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca11600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fca0c13f6d0 0x7fca0c0cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42058/0 (socket says 172.21.15.67:42058) 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca11600640 1 -- 172.21.15.67:0/3255663012 learned_addr learned my addr 172.21.15.67:0/3255663012 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.360+0000 7fca12000640 1 --2- 172.21.15.67:0/3255663012 >> v2:172.21.15.67:3300/0 conn(0x7fca0c0b25f0 0x7fca0c142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca10c00640 1 --2- 172.21.15.67:0/3255663012 >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 0x7fca0c0b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca0be00640 1 -- 172.21.15.67:0/3255663012 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161609863 0 0) 0x7fca0c15cbe0 con 0x7fca0c13f6d0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca0be00640 1 -- 172.21.15.67:0/3255663012 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f4003520 con 0x7fca0c13f6d0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca12000640 1 -- 172.21.15.67:0/3255663012 >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 msgr2=0x7fca0c0b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca12000640 1 --2- 172.21.15.67:0/3255663012 >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 0x7fca0c0b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca12000640 1 -- 172.21.15.67:0/3255663012 >> v1:172.21.15.67:6789/0 conn(0x7fca0c13f6d0 legacy=0x7fca0c0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca12000640 1 -- 172.21.15.67:0/3255663012 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c15cea0 con 0x7fca0c0b25f0 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.361+0000 7fca10c00640 1 --2- 172.21.15.67:0/3255663012 >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 0x7fca0c0b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.362+0000 7fca12000640 1 --2- 172.21.15.67:0/3255663012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0b25f0 0x7fca0c142020 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0d02d0 tx=0x7fca0402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90a510cc478e659 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.362+0000 7fca0be00640 1 -- 172.21.15.67:0/3255663012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04039070 con 0x7fca0c0b25f0 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.362+0000 7fca0be00640 1 -- 172.21.15.67:0/3255663012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0402cc80 con 0x7fca0c0b25f0 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca0be00640 1 -- 172.21.15.67:0/3255663012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04034660 con 0x7fca0c0b25f0 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 -- 172.21.15.67:0/3255663012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0b25f0 msgr2=0x7fca0c142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 --2- 172.21.15.67:0/3255663012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0b25f0 0x7fca0c142020 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0d02d0 tx=0x7fca0402bed0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 -- 172.21.15.67:0/3255663012 shutdown_connections 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 --2- 172.21.15.67:0/3255663012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0b25f0 0x7fca0c142020 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 --2- 172.21.15.67:0/3255663012 >> v2:172.21.15.187:3300/0 conn(0x7fca0c071d00 0x7fca0c0b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 -- 172.21.15.67:0/3255663012 >> 172.21.15.67:0/3255663012 conn(0x7fca0c067d10 msgr2=0x7fca0c0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 -- 172.21.15.67:0/3255663012 shutdown_connections 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.363+0000 7fca12a00640 1 -- 172.21.15.67:0/3255663012 wait complete. 2024-04-17T08:30:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.365+0000 7fca12a00640 1 Processor -- start 2024-04-17T08:30:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.365+0000 7fca12a00640 1 -- start start 2024-04-17T08:30:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 0x7fca0c1f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca0c13f6d0 0x7fca0c0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c15d740 con 0x7fca0c071d00 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c15d460 con 0x7fca0c0b25f0 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.366+0000 7fca12a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fca0c0d1120 con 0x7fca0c13f6d0 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37556/0 (socket says 172.21.15.67:37556) 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 -- 172.21.15.67:0/3354880177 learned_addr learned my addr 172.21.15.67:0/3354880177 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca10c00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 0x7fca0c1f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca12000640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca0c13f6d0 0x7fca0c0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 -- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca0c13f6d0 msgr2=0x7fca0c0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca0c13f6d0 0x7fca0c0b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 -- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 msgr2=0x7fca0c1f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 0x7fca0c1f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca11600640 1 -- 172.21.15.67:0/3354880177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c0b3750 con 0x7fca0c071d00 2024-04-17T08:30:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.367+0000 7fca10c00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 0x7fca0c1f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca11600640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c3c0 tx=0x7fca0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00017070 con 0x7fca0c071d00 2024-04-17T08:30:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca00005ce0 con 0x7fca0c071d00 2024-04-17T08:30:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000072f0 con 0x7fca0c071d00 2024-04-17T08:30:50.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c0b39f0 con 0x7fca0c071d00 2024-04-17T08:30:50.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.368+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c0b3fb0 con 0x7fca0c071d00 2024-04-17T08:30:50.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.369+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0c068a10 con 0x7fca0c071d00 2024-04-17T08:30:50.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.371+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fca0001a070 con 0x7fca0c071d00 2024-04-17T08:30:50.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.372+0000 7fca09600640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 0x7fc9ec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:50.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.372+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 5255+0+0 (secure 0 0 0) 0x7fca0008b270 con 0x7fca0c071d00 2024-04-17T08:30:50.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-17T08:30:50.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.373+0000 7fca10c00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 0x7fc9ec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:50.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.374+0000 7fca10c00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 0x7fc9ec06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8006fd0 tx=0x7fc9f8008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:50.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.375+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fca00021020 con 0x7fca0c071d00 2024-04-17T08:30:50.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:50.581+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fca0c160b60 con 0x7fca0c071d00 2024-04-17T08:30:51.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.101+0000 7fca09600640 1 -- 172.21.15.67:0/3354880177 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v180) ==== 170+0+0 (secure 0 0 0) 0x7fca00059830 con 0x7fca0c071d00 2024-04-17T08:30:51.100 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:30:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 msgr2=0x7fc9ec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 0x7fc9ec06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8006fd0 tx=0x7fc9f8008040 comp rx=0 tx=0).stop 2024-04-17T08:30:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 msgr2=0x7fca0c1f41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:51.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c3c0 tx=0x7fca0000c880 comp rx=0 tx=0).stop 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 shutdown_connections 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9ec06ca10 0x7fc9ec06eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca0c13f6d0 0x7fca0c0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca0c0b25f0 0x7fca0c1f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 --2- 172.21.15.67:0/3354880177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c071d00 0x7fca0c1f41e0 secure :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fca0000c3c0 tx=0x7fca0000c880 comp rx=0 tx=0).stop 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 >> 172.21.15.67:0/3354880177 conn(0x7fca0c067d10 msgr2=0x7fca0c15bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 shutdown_connections 2024-04-17T08:30:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.106+0000 7fca12a00640 1 -- 172.21.15.67:0/3354880177 wait complete. 2024-04-17T08:30:51.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:30:51.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.328+0000 7f17d6c00640 1 Processor -- start 2024-04-17T08:30:51.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.329+0000 7f17d6c00640 1 -- start start 2024-04-17T08:30:51.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.330+0000 7f17d6c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f17d0145f80 0x7f17d00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.330+0000 7f17d6c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17d00cd750 0x7f17d006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.330+0000 7f17d6c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17d006f1d0 con 0x7f17d006c0a0 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.330+0000 7f17d6c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17d00727a0 con 0x7f17d00cd750 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.330+0000 7f17d6c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f17d0146360 con 0x7f17d0145f80 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17d00cd750 0x7f17d006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17d00cd750 0x7f17d006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37560/0 (socket says 172.21.15.67:37560) 2024-04-17T08:30:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 -- 172.21.15.67:0/141560920 learned_addr learned my addr 172.21.15.67:0/141560920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d5800640 1 --2- 172.21.15.67:0/141560920 >> v2:172.21.15.187:3300/0 conn(0x7f17d0145f80 0x7f17d00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 -- 172.21.15.67:0/141560920 >> v2:172.21.15.187:3300/0 conn(0x7f17d0145f80 msgr2=0x7f17d00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 --2- 172.21.15.67:0/141560920 >> v2:172.21.15.187:3300/0 conn(0x7f17d0145f80 0x7f17d00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 -- 172.21.15.67:0/141560920 >> v1:172.21.15.187:6789/0 conn(0x7f17d006c0a0 legacy=0x7f17d0160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.331+0000 7f17d4e00640 1 -- 172.21.15.67:0/141560920 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d0071e70 con 0x7f17d00cd750 2024-04-17T08:30:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.332+0000 7f17d4e00640 1 --2- 172.21.15.67:0/141560920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d006bb60 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f17d006c9c0 tx=0x7f17bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2077e91b3ef2d768 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17cfe00640 1 -- 172.21.15.67:0/141560920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc02e070 con 0x7f17d00cd750 2024-04-17T08:30:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17cfe00640 1 -- 172.21.15.67:0/141560920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc037070 con 0x7f17d00cd750 2024-04-17T08:30:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17cfe00640 1 -- 172.21.15.67:0/141560920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031a00 con 0x7f17d00cd750 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 -- 172.21.15.67:0/141560920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 msgr2=0x7f17d006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 --2- 172.21.15.67:0/141560920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d006bb60 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f17d006c9c0 tx=0x7f17bc02b880 comp rx=0 tx=0).stop 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 -- 172.21.15.67:0/141560920 shutdown_connections 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 --2- 172.21.15.67:0/141560920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d006bb60 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 --2- 172.21.15.67:0/141560920 >> v2:172.21.15.187:3300/0 conn(0x7f17d0145f80 0x7f17d00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.333+0000 7f17d6c00640 1 -- 172.21.15.67:0/141560920 >> 172.21.15.67:0/141560920 conn(0x7f17d00b2190 msgr2=0x7f17d00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.334+0000 7f17d6c00640 1 -- 172.21.15.67:0/141560920 shutdown_connections 2024-04-17T08:30:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.334+0000 7f17d6c00640 1 -- 172.21.15.67:0/141560920 wait complete. 2024-04-17T08:30:51.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.335+0000 7f17d6c00640 1 Processor -- start 2024-04-17T08:30:51.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.335+0000 7f17d6c00640 1 -- start start 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17d006c0a0 0x7f17d01f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17d0145f80 0x7f17d01f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0072c30 con 0x7f17d00cd750 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0072430 con 0x7f17d006c0a0 2024-04-17T08:30:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.336+0000 7f17d6c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17d0072db0 con 0x7f17d0145f80 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37562/0 (socket says 172.21.15.67:37562) 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 -- 172.21.15.67:0/119431622 learned_addr learned my addr 172.21.15.67:0/119431622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d5800640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17d006c0a0 0x7f17d01f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d6200640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17d0145f80 0x7f17d01f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 -- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17d0145f80 msgr2=0x7f17d01f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17d0145f80 0x7f17d01f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 -- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17d006c0a0 msgr2=0x7f17d01f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17d006c0a0 0x7f17d01f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.337+0000 7f17d4e00640 1 -- 172.21.15.67:0/119431622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d01f58f0 con 0x7f17d00cd750 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17d4e00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02b8c0 tx=0x7f17bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc032670 con 0x7f17d00cd750 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc032830 con 0x7f17d00cd750 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc032c20 con 0x7f17d00cd750 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d01f88e0 con 0x7f17d00cd750 2024-04-17T08:30:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.338+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d01f4bb0 con 0x7f17d00cd750 2024-04-17T08:30:51.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.339+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d00cec90 con 0x7f17d00cd750 2024-04-17T08:30:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.341+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f17bc04c060 con 0x7f17d00cd750 2024-04-17T08:30:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.342+0000 7f17cd600640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 0x7f17b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.342+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 5637+0+0 (secure 0 0 0) 0x7f17bc0ada40 con 0x7f17d00cd750 2024-04-17T08:30:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.342+0000 7f17d5800640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 0x7f17b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.344+0000 7f17d5800640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 0x7f17b006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.347+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f17d00cec90 con 0x7f17d00cd750 2024-04-17T08:30:51.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:51.592+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f17d0068a10 con 0x7f17d00cd750 2024-04-17T08:30:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 20 and data pool 21 2024-04-17T08:30:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.151+0000 7f17cd600640 1 -- 172.21.15.67:0/119431622 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v81) ==== 174+0+0 (secure 0 0 0) 0x7f17bc040070 con 0x7f17d00cd750 2024-04-17T08:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.160+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 msgr2=0x7f17b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.160+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 0x7f17b006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).stop 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 msgr2=0x7f17d015bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02b8c0 tx=0x7f17bc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 shutdown_connections 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f17b006ca10 0x7f17b006eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17d0145f80 0x7f17d01f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17d00cd750 0x7f17d015bcb0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 --2- 172.21.15.67:0/119431622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17d006c0a0 0x7f17d01f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 >> 172.21.15.67:0/119431622 conn(0x7f17d00b2190 msgr2=0x7f17d00b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.161+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 shutdown_connections 2024-04-17T08:30:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.162+0000 7f17d6c00640 1 -- 172.21.15.67:0/119431622 wait complete. 2024-04-17T08:30:52.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:52.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.373+0000 7f9138e00640 1 Processor -- start 2024-04-17T08:30:52.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.373+0000 7f9138e00640 1 -- start start 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9138e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f913415c5a0 0x7f913415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9138e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91340b46c0 0x7f91340ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9138e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913415f440 con 0x7f91340b4210 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9138e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f913415f720 con 0x7f913415c5a0 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9138e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f913415fa00 con 0x7f91340b46c0 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9132a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f91340b4210 0x7f91340af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42070/0 (socket says 172.21.15.67:42070) 2024-04-17T08:30:52.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.375+0000 7f9132a00640 1 -- 172.21.15.67:0/1183834480 learned_addr learned my addr 172.21.15.67:0/1183834480 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> v2:172.21.15.187:3300/0 conn(0x7f913415c5a0 0x7f913415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443718050 0 0) 0x7f913415f440 con 0x7f91340b4210 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9133e00640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91340b46c0 0x7f91340ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9118003520 con 0x7f91340b4210 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4107265130 0 0) 0x7f9118003520 con 0x7f91340b4210 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91340b46c0 msgr2=0x7f91340ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91340b46c0 0x7f91340ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 >> v2:172.21.15.187:3300/0 conn(0x7f913415c5a0 msgr2=0x7f913415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 --2- 172.21.15.67:0/1183834480 >> v2:172.21.15.187:3300/0 conn(0x7f913415c5a0 0x7f913415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913415fce0 con 0x7f91340b4210 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f911c001670 con 0x7f91340b4210 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> v2:172.21.15.187:3300/0 conn(0x7f913415c5a0 0x7f913415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 >> v1:172.21.15.67:6789/0 conn(0x7f91340b4210 legacy=0x7f91340af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f9118003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9118004800 0x7f9118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9118005290 0x7f911800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f913415f720 con 0x7f9118003a20 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f913415fa00 con 0x7f9118004800 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f911800c060 con 0x7f9118005290 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.376+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f9118003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9132a00640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9118004800 0x7f9118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9133400640 1 -- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9118005290 msgr2=0x7f911800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:52.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9118005290 0x7f911800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9133400640 1 -- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9118004800 msgr2=0x7f9118004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:52.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9118004800 0x7f9118004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.377+0000 7f9133400640 1 -- 172.21.15.67:0/1183834480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9118003520 con 0x7f9118003a20 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.378+0000 7f9133400640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f9118003e40 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f91240098a0 tx=0x7f9118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f04ccafac3d2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.378+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912400f040 con 0x7f9118003a20 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.378+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124004230 con 0x7f9118003a20 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.378+0000 7f9132000640 1 -- 172.21.15.67:0/1183834480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124003610 con 0x7f9118003a20 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 -- 172.21.15.67:0/1183834480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 msgr2=0x7f9118003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 --2- 172.21.15.67:0/1183834480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f9118003e40 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f91240098a0 tx=0x7f9118005830 comp rx=0 tx=0).stop 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9133e00640 1 -- 172.21.15.67:0/1183834480 reap_dead start 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 -- 172.21.15.67:0/1183834480 shutdown_connections 2024-04-17T08:30:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 -- 172.21.15.67:0/1183834480 >> 172.21.15.67:0/1183834480 conn(0x7f9134067d10 msgr2=0x7f91340b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 -- 172.21.15.67:0/1183834480 shutdown_connections 2024-04-17T08:30:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.379+0000 7f9138e00640 1 -- 172.21.15.67:0/1183834480 wait complete. 2024-04-17T08:30:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.381+0000 7f9138e00640 1 Processor -- start 2024-04-17T08:30:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.381+0000 7f9138e00640 1 -- start start 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913406f9b0 0x7f9134074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913406fe30 0x7f9134075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f913406e7f0 con 0x7f9118003a20 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f913406e670 con 0x7f913406f9b0 2024-04-17T08:30:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9138e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91340afb50 con 0x7f913406fe30 2024-04-17T08:30:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9133400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9133400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37570/0 (socket says 172.21.15.67:37570) 2024-04-17T08:30:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.382+0000 7f9133400640 1 -- 172.21.15.67:0/375334992 learned_addr learned my addr 172.21.15.67:0/375334992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 -- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913406fe30 msgr2=0x7f9134075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913406fe30 0x7f9134075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 -- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913406f9b0 msgr2=0x7f9134074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913406f9b0 0x7f9134074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 -- 172.21.15.67:0/375334992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9134156e60 con 0x7f9118003a20 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9133400640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9124009990 tx=0x7f9124004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124047070 con 0x7f9118003a20 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124042400 con 0x7f9118003a20 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91240427d0 con 0x7f9118003a20 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9134157140 con 0x7f9118003a20 2024-04-17T08:30:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.383+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9134157700 con 0x7f9118003a20 2024-04-17T08:30:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.384+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9134068a10 con 0x7f9118003a20 2024-04-17T08:30:52.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.386+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f912404a070 con 0x7f9118003a20 2024-04-17T08:30:52.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.386+0000 7f912b400640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 0x7f911006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.387+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5737+0+0 (secure 0 0 0) 0x7f91240ad630 con 0x7f9118003a20 2024-04-17T08:30:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.387+0000 7f9132a00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 0x7f911006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.389+0000 7f9132a00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 0x7f911006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f911c0017e0 tx=0x7f911c009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.391+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9124033070 con 0x7f9118003a20 2024-04-17T08:30:52.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.587+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f913406f030 con 0x7f9118003a20 2024-04-17T08:30:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.589+0000 7f912b400640 1 -- 172.21.15.67:0/375334992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) ==== 74+0+16688 (secure 0 0 0) 0x7f912407b920 con 0x7f9118003a20 2024-04-17T08:30:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:52.589 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":181,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:51.593473+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 msgr2=0x7f911006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 0x7f911006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f911c0017e0 tx=0x7f911c009040 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 msgr2=0x7f91341504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9124009990 tx=0x7f9124004730 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 shutdown_connections 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f911006ca10 0x7f911006eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f913406fe30 0x7f9134075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f913406f9b0 0x7f9134074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 --2- 172.21.15.67:0/375334992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118003a20 0x7f91341504e0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.591+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 >> 172.21.15.67:0/375334992 conn(0x7f9134067d10 msgr2=0x7f913415b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.592+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 shutdown_connections 2024-04-17T08:30:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.592+0000 7f9138e00640 1 -- 172.21.15.67:0/375334992 wait complete. 2024-04-17T08:30:52.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.820+0000 7f4a6b200640 1 Processor -- start 2024-04-17T08:30:52.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.820+0000 7f4a6b200640 1 -- start start 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a6b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a6c06feb0 0x7f4a6c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c06be70 0x7f4a6c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a6b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a6c15bc80 con 0x7f4a6c15b570 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a6b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a6c15bf40 con 0x7f4a6c06feb0 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a6b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c15c220 con 0x7f4a6c06be70 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a69400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a6c06feb0 0x7f4a6c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a69400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a6c06feb0 0x7f4a6c06b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37578/0 (socket says 172.21.15.67:37578) 2024-04-17T08:30:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.822+0000 7f4a69400640 1 -- 172.21.15.67:0/3154187483 learned_addr learned my addr 172.21.15.67:0/3154187483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.823+0000 7f4a69400640 1 -- 172.21.15.67:0/3154187483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c06be70 msgr2=0x7f4a6c15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.823+0000 7f4a69400640 1 --2- 172.21.15.67:0/3154187483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c06be70 0x7f4a6c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.823+0000 7f4a69400640 1 -- 172.21.15.67:0/3154187483 >> v1:172.21.15.67:6789/0 conn(0x7f4a6c15b570 legacy=0x7f4a6c06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.823+0000 7f4a69400640 1 -- 172.21.15.67:0/3154187483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c15c500 con 0x7f4a6c06feb0 2024-04-17T08:30:52.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.823+0000 7f4a68a00640 1 -- 172.21.15.67:0/3154187483 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733121009 0 0) 0x7f4a6c15bc80 con 0x7f4a6c15b570 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.824+0000 7f4a69400640 1 --2- 172.21.15.67:0/3154187483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c06b930 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009870 tx=0x7f4a5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12acc43cce5c4571 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.824+0000 7f4a68a00640 1 -- 172.21.15.67:0/3154187483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58035020 con 0x7f4a6c06feb0 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.824+0000 7f4a68a00640 1 -- 172.21.15.67:0/3154187483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a58031640 con 0x7f4a6c06feb0 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.824+0000 7f4a68a00640 1 -- 172.21.15.67:0/3154187483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5802d040 con 0x7f4a6c06feb0 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 -- 172.21.15.67:0/3154187483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06feb0 msgr2=0x7f4a6c06b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3154187483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c06b930 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009870 tx=0x7f4a5802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 -- 172.21.15.67:0/3154187483 shutdown_connections 2024-04-17T08:30:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3154187483 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c06be70 0x7f4a6c15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3154187483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c06b930 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 -- 172.21.15.67:0/3154187483 >> 172.21.15.67:0/3154187483 conn(0x7f4a6c067d10 msgr2=0x7f4a6c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 -- 172.21.15.67:0/3154187483 shutdown_connections 2024-04-17T08:30:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.825+0000 7f4a6b200640 1 -- 172.21.15.67:0/3154187483 wait complete. 2024-04-17T08:30:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.827+0000 7f4a6b200640 1 Processor -- start 2024-04-17T08:30:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.827+0000 7f4a6b200640 1 -- start start 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c1f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c15b570 0x7f4a6c150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c06c440 con 0x7f4a6c06be70 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c06c2c0 con 0x7f4a6c06feb0 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a6b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c0b47b0 con 0x7f4a6c15b570 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37588/0 (socket says 172.21.15.67:37588) 2024-04-17T08:30:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 -- 172.21.15.67:0/3912981403 learned_addr learned my addr 172.21.15.67:0/3912981403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:52.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 -- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c15b570 msgr2=0x7f4a6c150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c15b570 0x7f4a6c150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 -- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a6c06feb0 msgr2=0x7f4a6c1f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c1f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.828+0000 7f4a69e00640 1 -- 172.21.15.67:0/3912981403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c150f50 con 0x7f4a6c06be70 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a69e00640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c410 tx=0x7f4a5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c017070 con 0x7f4a6c06be70 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a5c0035f0 con 0x7f4a6c06be70 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0039e0 con 0x7f4a6c06be70 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6c0adfd0 con 0x7f4a6c06be70 2024-04-17T08:30:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.829+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a6c0ae590 con 0x7f4a6c06be70 2024-04-17T08:30:52.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.830+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6c068a10 con 0x7f4a6c06be70 2024-04-17T08:30:52.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.831+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f4a5c01a070 con 0x7f4a6c06be70 2024-04-17T08:30:52.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.832+0000 7f4a62000640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 0x7f4a4c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:52.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.832+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5737+0+0 (secure 0 0 0) 0x7f4a5c08b1e0 con 0x7f4a6c06be70 2024-04-17T08:30:52.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.833+0000 7f4a69400640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 0x7f4a4c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.834+0000 7f4a69400640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 0x7f4a4c06ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a58036000 tx=0x7f4a58030000 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:52.837+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a5c059440 con 0x7f4a6c06be70 2024-04-17T08:30:53.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.036+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a6c06f350 con 0x7f4a6c06be70 2024-04-17T08:30:53.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.039+0000 7f4a62000640 1 -- 172.21.15.67:0/3912981403 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) ==== 74+0+16688 (secure 0 0 0) 0x7f4a5c058d40 con 0x7f4a6c06be70 2024-04-17T08:30:53.038 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:53.039 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":181,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:51.593473+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 msgr2=0x7f4a4c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 0x7f4a4c06ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a58036000 tx=0x7f4a58030000 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 msgr2=0x7f4a6c0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00c410 tx=0x7f4a5c00c8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 shutdown_connections 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a4c06ca40 0x7f4a4c06ef00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a6c15b570 0x7f4a6c150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a6c06feb0 0x7f4a6c1f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.041+0000 7f4a6b200640 1 --2- 172.21.15.67:0/3912981403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a6c06be70 0x7f4a6c0ad8d0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.042+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 >> 172.21.15.67:0/3912981403 conn(0x7f4a6c067d10 msgr2=0x7f4a6c0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.042+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 shutdown_connections 2024-04-17T08:30:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.042+0000 7f4a6b200640 1 -- 172.21.15.67:0/3912981403 wait complete. 2024-04-17T08:30:53.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:30:53.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.282+0000 7f8118200640 1 Processor -- start 2024-04-17T08:30:53.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.282+0000 7f8118200640 1 -- start start 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8118200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8110145ac0 0x7f8110071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8118200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81100ce3b0 0x7f8110072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8118200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81100cea70 con 0x7f8110072bb0 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8118200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f81100ce840 con 0x7f81100ce3b0 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8118200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8110072ff0 con 0x7f8110145ac0 2024-04-17T08:30:53.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8114e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81100ce3b0 0x7f8110072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8114e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81100ce3b0 0x7f8110072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35424/0 (socket says 172.21.15.67:35424) 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.284+0000 7f8114e00640 1 -- 172.21.15.67:0/1192211632 learned_addr learned my addr 172.21.15.67:0/1192211632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.285+0000 7f8114e00640 1 -- 172.21.15.67:0/1192211632 >> v2:172.21.15.187:3300/0 conn(0x7f8110145ac0 msgr2=0x7f8110071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.285+0000 7f8114e00640 1 --2- 172.21.15.67:0/1192211632 >> v2:172.21.15.187:3300/0 conn(0x7f8110145ac0 0x7f8110071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.285+0000 7f8114e00640 1 -- 172.21.15.67:0/1192211632 >> v1:172.21.15.187:6789/0 conn(0x7f8110072bb0 legacy=0x7f81100ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.285+0000 7f8114e00640 1 -- 172.21.15.67:0/1192211632 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81100ced50 con 0x7f81100ce3b0 2024-04-17T08:30:53.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.286+0000 7f8114e00640 1 --2- 172.21.15.67:0/1192211632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f8110072670 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740d5b73e1aa8ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:53.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.286+0000 7f810fe00640 1 -- 172.21.15.67:0/1192211632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc035020 con 0x7f81100ce3b0 2024-04-17T08:30:53.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.286+0000 7f810fe00640 1 -- 172.21.15.67:0/1192211632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80fc02d040 con 0x7f81100ce3b0 2024-04-17T08:30:53.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.286+0000 7f810fe00640 1 -- 172.21.15.67:0/1192211632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc031a80 con 0x7f81100ce3b0 2024-04-17T08:30:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 -- 172.21.15.67:0/1192211632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 msgr2=0x7f8110072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 --2- 172.21.15.67:0/1192211632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f8110072670 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc02b900 comp rx=0 tx=0).stop 2024-04-17T08:30:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 -- 172.21.15.67:0/1192211632 shutdown_connections 2024-04-17T08:30:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 --2- 172.21.15.67:0/1192211632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f8110072670 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 --2- 172.21.15.67:0/1192211632 >> v2:172.21.15.187:3300/0 conn(0x7f8110145ac0 0x7f8110071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 -- 172.21.15.67:0/1192211632 >> 172.21.15.67:0/1192211632 conn(0x7f8110067ee0 msgr2=0x7f811006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 -- 172.21.15.67:0/1192211632 shutdown_connections 2024-04-17T08:30:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.287+0000 7f8118200640 1 -- 172.21.15.67:0/1192211632 wait complete. 2024-04-17T08:30:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.289+0000 7f8118200640 1 Processor -- start 2024-04-17T08:30:53.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.289+0000 7f8118200640 1 -- start start 2024-04-17T08:30:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8110072bb0 0x7f811015e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8110145ac0 0x7f81101440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81100cf5f0 con 0x7f81100ce3b0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81100cf310 con 0x7f8110072bb0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8118200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8110145ea0 con 0x7f8110145ac0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8114e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8114e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35426/0 (socket says 172.21.15.67:35426) 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8114e00640 1 -- 172.21.15.67:0/3744249150 learned_addr learned my addr 172.21.15.67:0/3744249150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8115800640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8110072bb0 0x7f811015e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.290+0000 7f8116200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8110145ac0 0x7f81101440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 -- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8110145ac0 msgr2=0x7f81101440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8110145ac0 0x7f81101440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 -- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8110072bb0 msgr2=0x7f811015e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8110072bb0 0x7f811015e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 -- 172.21.15.67:0/3744249150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811015d190 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8114e00640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc03ac60 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80fc03ae00 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc044630 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.291+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811015d470 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.292+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811015da30 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.293+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8110068ca0 con 0x7f81100ce3b0 2024-04-17T08:30:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.294+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f80fc04b060 con 0x7f81100ce3b0 2024-04-17T08:30:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.295+0000 7f810d600640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 0x7f80f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.295+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f80fc0ade50 con 0x7f81100ce3b0 2024-04-17T08:30:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.295+0000 7f8115800640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 0x7f80f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.296+0000 7f8115800640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 0x7f80f006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f810400b810 tx=0x7f8104008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.299+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f80fc03f070 con 0x7f81100ce3b0 2024-04-17T08:30:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:53.553+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f81100cc6c0 con 0x7f81100ce3b0 2024-04-17T08:30:54.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.161+0000 7f810d600640 1 -- 172.21.15.67:0/3744249150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v84) ==== 118+0+0 (secure 0 0 0) 0x7f80fc07c030 con 0x7f81100ce3b0 2024-04-17T08:30:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 msgr2=0x7f80f006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 0x7f80f006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f810400b810 tx=0x7f8104008040 comp rx=0 tx=0).stop 2024-04-17T08:30:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 msgr2=0x7f811015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009870 tx=0x7f80fc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 shutdown_connections 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f80f006ca10 0x7f80f006eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8110145ac0 0x7f81101440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81100ce3b0 0x7f811015c1c0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 --2- 172.21.15.67:0/3744249150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8110072bb0 0x7f811015e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 >> 172.21.15.67:0/3744249150 conn(0x7f8110067ee0 msgr2=0x7f81100b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 shutdown_connections 2024-04-17T08:30:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.165+0000 7f8118200640 1 -- 172.21.15.67:0/3744249150 wait complete. 2024-04-17T08:30:54.190 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:30:54.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:30:54.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.419+0000 7f79e7200640 1 Processor -- start 2024-04-17T08:30:54.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.419+0000 7f79e7200640 1 -- start start 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.420+0000 7f79e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e00d0280 0x7f79e00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e00cdac0 0x7f79e00cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e00d0e60 con 0x7f79e009aa60 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f79e015d350 con 0x7f79e00cdac0 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79e00d0c20 con 0x7f79e00d0280 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e00cdac0 0x7f79e00cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:54.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e00cdac0 0x7f79e00cdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35436/0 (socket says 172.21.15.67:35436) 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 -- 172.21.15.67:0/139057866 learned_addr learned my addr 172.21.15.67:0/139057866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79df400640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e00d0280 0x7f79e00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 -- 172.21.15.67:0/139057866 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e00d0280 msgr2=0x7f79e00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e00d0280 0x7f79e00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 -- 172.21.15.67:0/139057866 >> v1:172.21.15.187:6789/0 conn(0x7f79e009aa60 legacy=0x7f79e009ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.421+0000 7f79e5200640 1 -- 172.21.15.67:0/139057866 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00d1140 con 0x7f79e00cdac0 2024-04-17T08:30:54.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.423+0000 7f79e5200640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00cdf50 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f79d80098a0 tx=0x7f79d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b661131c661385 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:54.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.423+0000 7f79dea00640 1 -- 172.21.15.67:0/139057866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8035020 con 0x7f79e00cdac0 2024-04-17T08:30:54.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.423+0000 7f79dea00640 1 -- 172.21.15.67:0/139057866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d8031640 con 0x7f79e00cdac0 2024-04-17T08:30:54.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.423+0000 7f79dea00640 1 -- 172.21.15.67:0/139057866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d802d040 con 0x7f79e00cdac0 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 -- 172.21.15.67:0/139057866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 msgr2=0x7f79e00cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00cdf50 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f79d80098a0 tx=0x7f79d802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 -- 172.21.15.67:0/139057866 shutdown_connections 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00cdf50 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 --2- 172.21.15.67:0/139057866 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e00d0280 0x7f79e00d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 -- 172.21.15.67:0/139057866 >> 172.21.15.67:0/139057866 conn(0x7f79e00b41a0 msgr2=0x7f79e00b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 -- 172.21.15.67:0/139057866 shutdown_connections 2024-04-17T08:30:54.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.424+0000 7f79e7200640 1 -- 172.21.15.67:0/139057866 wait complete. 2024-04-17T08:30:54.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.426+0000 7f79e7200640 1 Processor -- start 2024-04-17T08:30:54.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.426+0000 7f79e7200640 1 -- start start 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e009aa60 0x7f79e0150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e00d0280 0x7f79e00b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00d1700 con 0x7f79e00cdac0 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00cd630 con 0x7f79e00d0280 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f79e015ccf0 con 0x7f79e009aa60 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35446/0 (socket says 172.21.15.67:35446) 2024-04-17T08:30:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 -- 172.21.15.67:0/137888311 learned_addr learned my addr 172.21.15.67:0/137888311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 -- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e009aa60 msgr2=0x7f79e0150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e009aa60 0x7f79e0150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79e5200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e00d0280 0x7f79e00b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 -- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e00d0280 msgr2=0x7f79e00b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.427+0000 7f79df400640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e00d0280 0x7f79e00b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79df400640 1 -- 172.21.15.67:0/137888311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e015f630 con 0x7f79e00cdac0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79df400640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f79c800d5f0 tx=0x7f79c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c80125f0 con 0x7f79e00cdac0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c8019070 con 0x7f79e00cdac0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c80129d0 con 0x7f79e00cdac0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e015f910 con 0x7f79e00cdac0 2024-04-17T08:30:54.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.428+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e015fed0 con 0x7f79e00cdac0 2024-04-17T08:30:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.429+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e0068ca0 con 0x7f79e00cdac0 2024-04-17T08:30:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.431+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f79c8012b70 con 0x7f79e00cdac0 2024-04-17T08:30:54.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.431+0000 7f79d7e00640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 0x7f79bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:54.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.432+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f79c808c170 con 0x7f79e00cdac0 2024-04-17T08:30:54.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.432+0000 7f79dfe00640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 0x7f79bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:54.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.433+0000 7f79dfe00640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 0x7f79bc06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f79d00043c0 tx=0x7f79d0007d30 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.436+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f79c805a3d0 con 0x7f79e00cdac0 2024-04-17T08:30:54.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:54.684+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f79e00c5920 con 0x7f79e00cdac0 2024-04-17T08:30:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.174+0000 7f79d7e00640 1 -- 172.21.15.67:0/137888311 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v85) ==== 122+0+0 (secure 0 0 0) 0x7f79c8059cd0 con 0x7f79e00cdac0 2024-04-17T08:30:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 msgr2=0x7f79bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 0x7f79bc06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f79d00043c0 tx=0x7f79d0007d30 comp rx=0 tx=0).stop 2024-04-17T08:30:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 msgr2=0x7f79e00c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f79c800d5f0 tx=0x7f79c800dab0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 shutdown_connections 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f79bc06ca10 0x7f79bc06eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79e00d0280 0x7f79e00b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e00cdac0 0x7f79e00c9780 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 --2- 172.21.15.67:0/137888311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f79e009aa60 0x7f79e0150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.177+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 >> 172.21.15.67:0/137888311 conn(0x7f79e00b41a0 msgr2=0x7f79e00cdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.178+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 shutdown_connections 2024-04-17T08:30:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.178+0000 7f79e7200640 1 -- 172.21.15.67:0/137888311 wait complete. 2024-04-17T08:30:55.200 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:30:55.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:55.421 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T08:30:55.422+0000 7f4dd9800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:30:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.429+0000 7f4f6ee00640 1 Processor -- start 2024-04-17T08:30:55.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.430+0000 7f4f6ee00640 1 -- start start 2024-04-17T08:30:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f680b6d00 0x7f4f680cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f680cf910 0x7f4f680b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6ee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f680d2d30 con 0x7f4f6815ec60 2024-04-17T08:30:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4f680d49c0 con 0x7f4f680cf910 2024-04-17T08:30:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680d3660 con 0x7f4f680b6d00 2024-04-17T08:30:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6da00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4f6815ec60 0x7f4f680b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57340/0 (socket says 172.21.15.67:57340) 2024-04-17T08:30:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6da00640 1 -- 172.21.15.67:0/2741273156 learned_addr learned my addr 172.21.15.67:0/2741273156 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.431+0000 7f4f6e400640 1 --2- 172.21.15.67:0/2741273156 >> v2:172.21.15.67:3300/0 conn(0x7f4f680cf910 0x7f4f680b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f6d000640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f680b6d00 0x7f4f680cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f6e400640 1 -- 172.21.15.67:0/2741273156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f680b6d00 msgr2=0x7f4f680cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f6e400640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f680b6d00 0x7f4f680cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f6e400640 1 -- 172.21.15.67:0/2741273156 >> v1:172.21.15.67:6789/0 conn(0x7f4f6815ec60 legacy=0x7f4f680b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f67e00640 1 -- 172.21.15.67:0/2741273156 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067400160 0 0) 0x7f4f680d2d30 con 0x7f4f6815ec60 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.432+0000 7f4f6e400640 1 -- 172.21.15.67:0/2741273156 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f680d3010 con 0x7f4f680cf910 2024-04-17T08:30:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.434+0000 7f4f6e400640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680cf910 0x7f4f680b7b70 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f600097a0 tx=0x7f4f6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3645b53d5be5973f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.434+0000 7f4f67e00640 1 -- 172.21.15.67:0/2741273156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60035020 con 0x7f4f680cf910 2024-04-17T08:30:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.434+0000 7f4f67e00640 1 -- 172.21.15.67:0/2741273156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f60031830 con 0x7f4f680cf910 2024-04-17T08:30:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.434+0000 7f4f67e00640 1 -- 172.21.15.67:0/2741273156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60031c00 con 0x7f4f680cf910 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 -- 172.21.15.67:0/2741273156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680cf910 msgr2=0x7f4f680b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680cf910 0x7f4f680b7b70 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f600097a0 tx=0x7f4f6002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 -- 172.21.15.67:0/2741273156 shutdown_connections 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680cf910 0x7f4f680b7b70 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/2741273156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f680b6d00 0x7f4f680cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 -- 172.21.15.67:0/2741273156 >> 172.21.15.67:0/2741273156 conn(0x7f4f680d15f0 msgr2=0x7f4f680d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 -- 172.21.15.67:0/2741273156 shutdown_connections 2024-04-17T08:30:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.435+0000 7f4f6ee00640 1 -- 172.21.15.67:0/2741273156 wait complete. 2024-04-17T08:30:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.437+0000 7f4f6ee00640 1 Processor -- start 2024-04-17T08:30:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.437+0000 7f4f6ee00640 1 -- start start 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f680cf910 0x7f4f6815f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f6815ec60 0x7f4f680ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6815f040 con 0x7f4f680b6d00 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4f680cfdc0 con 0x7f4f680cf910 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f680d00c0 con 0x7f4f6815ec60 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35466/0 (socket says 172.21.15.67:35466) 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 -- 172.21.15.67:0/534130551 learned_addr learned my addr 172.21.15.67:0/534130551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6e400640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f6815ec60 0x7f4f680ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 -- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f6815ec60 msgr2=0x7f4f680ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6d000640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f680cf910 0x7f4f6815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f6815ec60 0x7f4f680ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 -- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f680cf910 msgr2=0x7f4f6815f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.438+0000 7f4f6da00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f680cf910 0x7f4f6815f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f6da00640 1 -- 172.21.15.67:0/534130551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f681595d0 con 0x7f4f680b6d00 2024-04-17T08:30:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f6da00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c430 tx=0x7f4f5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0074e0 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0047e0 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f680cbf40 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c004bd0 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.439+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f68159b20 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.441+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f68068a10 con 0x7f4f680b6d00 2024-04-17T08:30:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.442+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f4f5c011070 con 0x7f4f680b6d00 2024-04-17T08:30:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.444+0000 7f4f65600640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 0x7f4f4c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.444+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f4f5c08b980 con 0x7f4f680b6d00 2024-04-17T08:30:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.444+0000 7f4f6d000640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 0x7f4f4c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.446+0000 7f4f6d000640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 0x7f4f4c06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f680cd880 tx=0x7f4f5800b040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.448+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4f5c00e280 con 0x7f4f680b6d00 2024-04-17T08:30:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.691+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f680ba020 con 0x7f4f680b6d00 2024-04-17T08:30:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.693+0000 7f4f65600640 1 -- 172.21.15.67:0/534130551 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f4f5c059be0 con 0x7f4f680b6d00 2024-04-17T08:30:55.692 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:30:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.695+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 msgr2=0x7f4f4c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.695+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 0x7f4f4c06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f680cd880 tx=0x7f4f5800b040 comp rx=0 tx=0).stop 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.695+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 msgr2=0x7f4f68157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.695+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c430 tx=0x7f4f5c00c8f0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.695+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 shutdown_connections 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4f4c06ca40 0x7f4f4c06ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f6815ec60 0x7f4f680ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f680cf910 0x7f4f6815f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 --2- 172.21.15.67:0/534130551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f680b6d00 0x7f4f68157250 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 >> 172.21.15.67:0/534130551 conn(0x7f4f680d15f0 msgr2=0x7f4f680b92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:55.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 shutdown_connections 2024-04-17T08:30:55.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.696+0000 7f4f6ee00640 1 -- 172.21.15.67:0/534130551 wait complete. 2024-04-17T08:30:55.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:30:55.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.937+0000 7fca9d200640 1 Processor -- start 2024-04-17T08:30:55.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.938+0000 7fca9d200640 1 -- start start 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.939+0000 7fca9d200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fca98141ed0 0x7fca98142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.939+0000 7fca9d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca980cd4a0 con 0x7fca9806b8c0 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca9d200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca981607f0 con 0x7fca98160280 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca9d200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fca9815ce30 con 0x7fca98141ed0 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca97400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fca9806b8c0 0x7fca980cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57348/0 (socket says 172.21.15.67:57348) 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca97400640 1 -- 172.21.15.67:0/293707376 learned_addr learned my addr 172.21.15.67:0/293707376 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca97e00640 1 --2- 172.21.15.67:0/293707376 >> v2:172.21.15.187:3300/0 conn(0x7fca98141ed0 0x7fca98142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005414675 0 0) 0x7fca980cd4a0 con 0x7fca9806b8c0 2024-04-17T08:30:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.940+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca7c003520 con 0x7fca9806b8c0 2024-04-17T08:30:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4132760526 0 0) 0x7fca7c003520 con 0x7fca9806b8c0 2024-04-17T08:30:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 >> v2:172.21.15.187:3300/0 conn(0x7fca98141ed0 msgr2=0x7fca98142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 --2- 172.21.15.67:0/293707376 >> v2:172.21.15.187:3300/0 conn(0x7fca98141ed0 0x7fca98142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 >> v1:172.21.15.187:6789/0 conn(0x7fca98160280 legacy=0x7fca9806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca98160ad0 con 0x7fca9806b8c0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fca88001670 con 0x7fca9806b8c0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 >> v1:172.21.15.67:6789/0 conn(0x7fca9806b8c0 legacy=0x7fca980cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 0x7fca7c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c004800 0x7fca7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca7c0051c0 0x7fca7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca9815ce30 con 0x7fca7c003a20 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c00c0f0 con 0x7fca7c0051c0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fca7c00c2f0 con 0x7fca7c004800 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca97400640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 0x7fca7c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca8fe00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c004800 0x7fca7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.941+0000 7fca97e00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca7c0051c0 0x7fca7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97400640 1 -- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c004800 msgr2=0x7fca7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97400640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c004800 0x7fca7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97400640 1 -- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca7c0051c0 msgr2=0x7fca7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97400640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca7c0051c0 0x7fca7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97400640 1 -- 172.21.15.67:0/293707376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c003520 con 0x7fca7c003a20 2024-04-17T08:30:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97e00640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca7c0051c0 0x7fca7c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.942+0000 7fca97e00640 1 -- 172.21.15.67:0/293707376 reap_dead start 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.943+0000 7fca97400640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 0x7fca7c003e40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fca8800c210 tx=0x7fca7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7098faecd7c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.943+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca88007d30 con 0x7fca7c003a20 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.943+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca88030070 con 0x7fca7c003a20 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.943+0000 7fca96a00640 1 -- 172.21.15.67:0/293707376 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca88005d70 con 0x7fca7c003a20 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 -- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 msgr2=0x7fca7c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 0x7fca7c003e40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fca8800c210 tx=0x7fca7c005830 comp rx=0 tx=0).stop 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 -- 172.21.15.67:0/293707376 shutdown_connections 2024-04-17T08:30:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 --2- 172.21.15.67:0/293707376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca7c003a20 0x7fca7c003e40 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 -- 172.21.15.67:0/293707376 >> 172.21.15.67:0/293707376 conn(0x7fca98067d10 msgr2=0x7fca980b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 -- 172.21.15.67:0/293707376 shutdown_connections 2024-04-17T08:30:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.944+0000 7fca9d200640 1 -- 172.21.15.67:0/293707376 wait complete. 2024-04-17T08:30:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.946+0000 7fca9d200640 1 Processor -- start 2024-04-17T08:30:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.946+0000 7fca9d200640 1 -- start start 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 0x7fca981438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca9806ef70 0x7fca9806f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca9813f9b0 con 0x7fca98132d20 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fca98161090 con 0x7fca9806ef70 2024-04-17T08:30:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca9d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fca9813fb30 con 0x7fca7c003a20 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35488/0 (socket says 172.21.15.67:35488) 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.947+0000 7fca8fe00640 1 -- 172.21.15.67:0/3592933143 learned_addr learned my addr 172.21.15.67:0/3592933143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca97400640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 0x7fca981438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 -- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 msgr2=0x7fca981438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca97e00640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca9806ef70 0x7fca9806f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 0x7fca981438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 -- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca9806ef70 msgr2=0x7fca9806f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca9806ef70 0x7fca9806f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 -- 172.21.15.67:0/3592933143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca981334c0 con 0x7fca98132d20 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca97400640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 0x7fca981438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8fe00640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fca8000e4d0 tx=0x7fca8000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.948+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca800052e0 con 0x7fca98132d20 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.949+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca80018070 con 0x7fca98132d20 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.949+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80003810 con 0x7fca98132d20 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.949+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca981337a0 con 0x7fca98132d20 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.949+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca98130e40 con 0x7fca98132d20 2024-04-17T08:30:55.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.950+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca98068a10 con 0x7fca98132d20 2024-04-17T08:30:55.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.951+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fca800039b0 con 0x7fca98132d20 2024-04-17T08:30:55.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.952+0000 7fca8f400640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 0x7fca7406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:55.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.952+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7fca8008baa0 con 0x7fca98132d20 2024-04-17T08:30:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.953+0000 7fca97400640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 0x7fca7406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.954+0000 7fca97400640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 0x7fca7406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca88034000 tx=0x7fca88039040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:55.956+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fca80059d00 con 0x7fca98132d20 2024-04-17T08:30:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.164+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca9806c3f0 con 0x7fca98132d20 2024-04-17T08:30:56.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.166+0000 7fca8f400640 1 -- 172.21.15.67:0/3592933143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) ==== 74+0+16675 (secure 0 0 0) 0x7fca80059600 con 0x7fca98132d20 2024-04-17T08:30:56.165 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:56.165 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":182,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:30:53.150014+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 msgr2=0x7fca7406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 0x7fca7406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca88034000 tx=0x7fca88039040 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 msgr2=0x7fca9813eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fca8000e4d0 tx=0x7fca8000e990 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 shutdown_connections 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fca7406ca40 0x7fca7406ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca9806ef70 0x7fca9806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca98132d20 0x7fca9813eb20 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 --2- 172.21.15.67:0/3592933143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fca7c003a20 0x7fca981438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.168+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 >> 172.21.15.67:0/3592933143 conn(0x7fca98067d10 msgr2=0x7fca98141720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.169+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 shutdown_connections 2024-04-17T08:30:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.169+0000 7fca9d200640 1 -- 172.21.15.67:0/3592933143 wait complete. 2024-04-17T08:30:56.190 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:30:56.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:30:56.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.415+0000 7f9b87000640 1 Processor -- start 2024-04-17T08:30:56.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.415+0000 7f9b87000640 1 -- start start 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b87000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9b8015b530 0x7f9b80071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b87000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b8015b910 0x7f9b80072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b87000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b8015f3c0 con 0x7f9b800c68c0 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b87000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9b8006ed60 con 0x7f9b8015b910 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b87000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9b8015f660 con 0x7f9b8015b530 2024-04-17T08:30:56.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b7f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b800c68c0 0x7f9b800c5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57350/0 (socket says 172.21.15.67:57350) 2024-04-17T08:30:56.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b7f400640 1 -- 172.21.15.67:0/4256260395 learned_addr learned my addr 172.21.15.67:0/4256260395 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:56.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b85000640 1 --2- 172.21.15.67:0/4256260395 >> v2:172.21.15.67:3300/0 conn(0x7f9b8015b910 0x7f9b80072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.417+0000 7f9b7fe00640 1 --2- 172.21.15.67:0/4256260395 >> v2:172.21.15.187:3300/0 conn(0x7f9b8015b530 0x7f9b80071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b7ea00640 1 -- 172.21.15.67:0/4256260395 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404682383 0 0) 0x7f9b8015f3c0 con 0x7f9b800c68c0 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b7ea00640 1 -- 172.21.15.67:0/4256260395 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b64003520 con 0x7f9b800c68c0 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b85000640 1 -- 172.21.15.67:0/4256260395 >> v2:172.21.15.187:3300/0 conn(0x7f9b8015b530 msgr2=0x7f9b80071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b85000640 1 --2- 172.21.15.67:0/4256260395 >> v2:172.21.15.187:3300/0 conn(0x7f9b8015b530 0x7f9b80071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b85000640 1 -- 172.21.15.67:0/4256260395 >> v1:172.21.15.67:6789/0 conn(0x7f9b800c68c0 legacy=0x7f9b800c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.418+0000 7f9b85000640 1 -- 172.21.15.67:0/4256260395 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8015f940 con 0x7f9b8015b910 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.419+0000 7f9b85000640 1 --2- 172.21.15.67:0/4256260395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b8015b910 0x7f9b80072250 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f9b8006c840 tx=0x7f9b7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebb2c1889ac9ff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.419+0000 7f9b7ea00640 1 -- 172.21.15.67:0/4256260395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78035020 con 0x7f9b8015b910 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.419+0000 7f9b7ea00640 1 -- 172.21.15.67:0/4256260395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b78031830 con 0x7f9b8015b910 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.419+0000 7f9b7ea00640 1 -- 172.21.15.67:0/4256260395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78031c00 con 0x7f9b8015b910 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 -- 172.21.15.67:0/4256260395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b8015b910 msgr2=0x7f9b80072250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 --2- 172.21.15.67:0/4256260395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b8015b910 0x7f9b80072250 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f9b8006c840 tx=0x7f9b7802b8d0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 -- 172.21.15.67:0/4256260395 shutdown_connections 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 --2- 172.21.15.67:0/4256260395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b8015b910 0x7f9b80072250 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 --2- 172.21.15.67:0/4256260395 >> v2:172.21.15.187:3300/0 conn(0x7f9b8015b530 0x7f9b80071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 -- 172.21.15.67:0/4256260395 >> 172.21.15.67:0/4256260395 conn(0x7f9b8006bf50 msgr2=0x7f9b800b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 -- 172.21.15.67:0/4256260395 shutdown_connections 2024-04-17T08:30:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.420+0000 7f9b87000640 1 -- 172.21.15.67:0/4256260395 wait complete. 2024-04-17T08:30:56.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.422+0000 7f9b87000640 1 Processor -- start 2024-04-17T08:30:56.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.422+0000 7f9b87000640 1 -- start start 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b8015b530 0x7f9b800c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b8015b910 0x7f9b801554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800cc950 con 0x7f9b800c68c0 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800ccca0 con 0x7f9b8015b910 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b87000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b80160140 con 0x7f9b8015b530 2024-04-17T08:30:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35516/0 (socket says 172.21.15.67:35516) 2024-04-17T08:30:56.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 -- 172.21.15.67:0/2287267266 learned_addr learned my addr 172.21.15.67:0/2287267266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 -- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b8015b530 msgr2=0x7f9b800c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b8015b530 0x7f9b800c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b7fe00640 1 -- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b8015b910 msgr2=0x7f9b801554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.423+0000 7f9b85000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b8015b910 0x7f9b801554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b7fe00640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b8015b910 0x7f9b801554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b7fe00640 1 -- 172.21.15.67:0/2287267266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b800c8c50 con 0x7f9b800c68c0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b7fe00640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000efc0 tx=0x7f9b7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70018020 con 0x7f9b800c68c0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b70014480 con 0x7f9b800c68c0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70014850 con 0x7f9b800c68c0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b800c8f30 con 0x7f9b800c68c0 2024-04-17T08:30:56.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.424+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b800c94f0 con 0x7f9b800c68c0 2024-04-17T08:30:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.426+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b80068b90 con 0x7f9b800c68c0 2024-04-17T08:30:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.427+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f9b70005d10 con 0x7f9b800c68c0 2024-04-17T08:30:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.428+0000 7f9b77e00640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 0x7f9b5c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.428+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f9b7008cce0 con 0x7f9b800c68c0 2024-04-17T08:30:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.429+0000 7f9b7f400640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 0x7f9b5c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.430+0000 7f9b7f400640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 0x7f9b5c06eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b800ca550 tx=0x7f9b680078f0 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.433+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9b70022030 con 0x7f9b800c68c0 2024-04-17T08:30:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.639+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 --> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b800c6390 con 0x7f9b5c06ca10 2024-04-17T08:30:56.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.663+0000 7f9b77e00640 1 -- 172.21.15.67:0/2287267266 <== mgr.7465 v2:172.21.15.187:6832/659531354 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252427 (secure 0 0 0) 0x7f9b68053a20 con 0x7f9b5c06ca10 2024-04-17T08:30:56.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:30:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 msgr2=0x7f9b5c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 0x7f9b5c06eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b800ca550 tx=0x7f9b680078f0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 msgr2=0x7f9b80150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000efc0 tx=0x7f9b7000c490 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 shutdown_connections 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f9b5c06ca10 0x7f9b5c06eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b8015b910 0x7f9b801554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b8015b530 0x7f9b800c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 --2- 172.21.15.67:0/2287267266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b800c68c0 0x7f9b80150330 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 >> 172.21.15.67:0/2287267266 conn(0x7f9b8006bf50 msgr2=0x7f9b80071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 shutdown_connections 2024-04-17T08:30:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.669+0000 7f9b87000640 1 -- 172.21.15.67:0/2287267266 wait complete. 2024-04-17T08:30:56.688 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-17T08:30:54.975056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":491,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413996,"kb_used_data":18796,"kb_used_omap":12,"kb_used_meta":395059,"kb_avail":754560724,"statfs":{"total":773094113280,"available":772670181376,"internally_reserved":0,"allocated":19247104,"data_stored":9942943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":404540976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.877429"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178156+0000","last_change":"2024-04-17T08:30:52.176535+0000","last_active":"2024-04-17T08:30:52.178156+0000","last_peered":"2024-04-17T08:30:52.178156+0000","last_clean":"2024-04-17T08:30:52.178156+0000","last_became_active":"2024-04-17T08:30:52.176142+0000","last_became_peered":"2024-04-17T08:30:52.176142+0000","last_unstale":"2024-04-17T08:30:52.178156+0000","last_undegraded":"2024-04-17T08:30:52.178156+0000","last_fullsized":"2024-04-17T08:30:52.178156+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:29:09.083064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165511+0000","last_change":"2024-04-17T08:30:52.160725+0000","last_active":"2024-04-17T08:30:52.165511+0000","last_peered":"2024-04-17T08:30:52.165511+0000","last_clean":"2024-04-17T08:30:52.165511+0000","last_became_active":"2024-04-17T08:30:52.160453+0000","last_became_peered":"2024-04-17T08:30:52.160453+0000","last_unstale":"2024-04-17T08:30:52.165511+0000","last_undegraded":"2024-04-17T08:30:52.165511+0000","last_fullsized":"2024-04-17T08:30:52.165511+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.912770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172933+0000","last_change":"2024-04-17T08:30:52.166841+0000","last_active":"2024-04-17T08:30:52.172933+0000","last_peered":"2024-04-17T08:30:52.172933+0000","last_clean":"2024-04-17T08:30:52.172933+0000","last_became_active":"2024-04-17T08:30:52.166419+0000","last_became_peered":"2024-04-17T08:30:52.166419+0000","last_unstale":"2024-04-17T08:30:52.172933+0000","last_undegraded":"2024-04-17T08:30:52.172933+0000","last_fullsized":"2024-04-17T08:30:52.172933+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:40.063359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.157436+0000","last_change":"2024-04-17T08:30:52.154568+0000","last_active":"2024-04-17T08:30:52.157436+0000","last_peered":"2024-04-17T08:30:52.157436+0000","last_clean":"2024-04-17T08:30:52.157436+0000","last_became_active":"2024-04-17T08:30:52.154278+0000","last_became_peered":"2024-04-17T08:30:52.154278+0000","last_unstale":"2024-04-17T08:30:52.157436+0000","last_undegraded":"2024-04-17T08:30:52.157436+0000","last_fullsized":"2024-04-17T08:30:52.157436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:35.540898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172879+0000","last_change":"2024-04-17T08:30:52.169806+0000","last_active":"2024-04-17T08:30:52.172879+0000","last_peered":"2024-04-17T08:30:52.172879+0000","last_clean":"2024-04-17T08:30:52.172879+0000","last_became_active":"2024-04-17T08:30:52.169306+0000","last_became_peered":"2024-04-17T08:30:52.169306+0000","last_unstale":"2024-04-17T08:30:52.172879+0000","last_undegraded":"2024-04-17T08:30:52.172879+0000","last_fullsized":"2024-04-17T08:30:52.172879+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:49.361756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166018+0000","last_change":"2024-04-17T08:30:52.162622+0000","last_active":"2024-04-17T08:30:52.166018+0000","last_peered":"2024-04-17T08:30:52.166018+0000","last_clean":"2024-04-17T08:30:52.166018+0000","last_became_active":"2024-04-17T08:30:52.162304+0000","last_became_peered":"2024-04-17T08:30:52.162304+0000","last_unstale":"2024-04-17T08:30:52.166018+0000","last_undegraded":"2024-04-17T08:30:52.166018+0000","last_fullsized":"2024-04-17T08:30:52.166018+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:03.337757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173571+0000","last_change":"2024-04-17T08:30:52.171688+0000","last_active":"2024-04-17T08:30:52.173571+0000","last_peered":"2024-04-17T08:30:52.173571+0000","last_clean":"2024-04-17T08:30:52.173571+0000","last_became_active":"2024-04-17T08:30:52.169274+0000","last_became_peered":"2024-04-17T08:30:52.169274+0000","last_unstale":"2024-04-17T08:30:52.173571+0000","last_undegraded":"2024-04-17T08:30:52.173571+0000","last_fullsized":"2024-04-17T08:30:52.173571+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:15.928453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175328+0000","last_change":"2024-04-17T08:30:52.169816+0000","last_active":"2024-04-17T08:30:52.175328+0000","last_peered":"2024-04-17T08:30:52.175328+0000","last_clean":"2024-04-17T08:30:52.175328+0000","last_became_active":"2024-04-17T08:30:52.169275+0000","last_became_peered":"2024-04-17T08:30:52.169275+0000","last_unstale":"2024-04-17T08:30:52.175328+0000","last_undegraded":"2024-04-17T08:30:52.175328+0000","last_fullsized":"2024-04-17T08:30:52.175328+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:49.498787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171496+0000","last_change":"2024-04-17T08:30:52.169241+0000","last_active":"2024-04-17T08:30:52.171496+0000","last_peered":"2024-04-17T08:30:52.171496+0000","last_clean":"2024-04-17T08:30:52.171496+0000","last_became_active":"2024-04-17T08:30:52.168892+0000","last_became_peered":"2024-04-17T08:30:52.168892+0000","last_unstale":"2024-04-17T08:30:52.171496+0000","last_undegraded":"2024-04-17T08:30:52.171496+0000","last_fullsized":"2024-04-17T08:30:52.171496+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:44.498403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177808+0000","last_change":"2024-04-17T08:30:52.175749+0000","last_active":"2024-04-17T08:30:52.177808+0000","last_peered":"2024-04-17T08:30:52.177808+0000","last_clean":"2024-04-17T08:30:52.177808+0000","last_became_active":"2024-04-17T08:30:52.174864+0000","last_became_peered":"2024-04-17T08:30:52.174864+0000","last_unstale":"2024-04-17T08:30:52.177808+0000","last_undegraded":"2024-04-17T08:30:52.177808+0000","last_fullsized":"2024-04-17T08:30:52.177808+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:41.111737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178193+0000","last_change":"2024-04-17T08:30:52.173534+0000","last_active":"2024-04-17T08:30:52.178193+0000","last_peered":"2024-04-17T08:30:52.178193+0000","last_clean":"2024-04-17T08:30:52.178193+0000","last_became_active":"2024-04-17T08:30:52.172387+0000","last_became_peered":"2024-04-17T08:30:52.172387+0000","last_unstale":"2024-04-17T08:30:52.178193+0000","last_undegraded":"2024-04-17T08:30:52.178193+0000","last_fullsized":"2024-04-17T08:30:52.178193+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.200418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.159801+0000","last_change":"2024-04-17T08:30:52.157740+0000","last_active":"2024-04-17T08:30:52.159801+0000","last_peered":"2024-04-17T08:30:52.159801+0000","last_clean":"2024-04-17T08:30:52.159801+0000","last_became_active":"2024-04-17T08:30:52.157419+0000","last_became_peered":"2024-04-17T08:30:52.157419+0000","last_unstale":"2024-04-17T08:30:52.159801+0000","last_undegraded":"2024-04-17T08:30:52.159801+0000","last_fullsized":"2024-04-17T08:30:52.159801+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:26:46.934348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171716+0000","last_change":"2024-04-17T08:30:52.169340+0000","last_active":"2024-04-17T08:30:52.171716+0000","last_peered":"2024-04-17T08:30:52.171716+0000","last_clean":"2024-04-17T08:30:52.171716+0000","last_became_active":"2024-04-17T08:30:52.167959+0000","last_became_peered":"2024-04-17T08:30:52.167959+0000","last_unstale":"2024-04-17T08:30:52.171716+0000","last_undegraded":"2024-04-17T08:30:52.171716+0000","last_fullsized":"2024-04-17T08:30:52.171716+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:35.307775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.174064+0000","last_change":"2024-04-17T08:30:52.170444+0000","last_active":"2024-04-17T08:30:52.174064+0000","last_peered":"2024-04-17T08:30:52.174064+0000","last_clean":"2024-04-17T08:30:52.174064+0000","last_became_active":"2024-04-17T08:30:52.170050+0000","last_became_peered":"2024-04-17T08:30:52.170050+0000","last_unstale":"2024-04-17T08:30:52.174064+0000","last_undegraded":"2024-04-17T08:30:52.174064+0000","last_fullsized":"2024-04-17T08:30:52.174064+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:51.504928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167553+0000","last_change":"2024-04-17T08:30:52.164866+0000","last_active":"2024-04-17T08:30:52.167553+0000","last_peered":"2024-04-17T08:30:52.167553+0000","last_clean":"2024-04-17T08:30:52.167553+0000","last_became_active":"2024-04-17T08:30:52.164568+0000","last_became_peered":"2024-04-17T08:30:52.164568+0000","last_unstale":"2024-04-17T08:30:52.167553+0000","last_undegraded":"2024-04-17T08:30:52.167553+0000","last_fullsized":"2024-04-17T08:30:52.167553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:58.754144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171895+0000","last_change":"2024-04-17T08:30:52.166413+0000","last_active":"2024-04-17T08:30:52.171895+0000","last_peered":"2024-04-17T08:30:52.171895+0000","last_clean":"2024-04-17T08:30:52.171895+0000","last_became_active":"2024-04-17T08:30:52.166105+0000","last_became_peered":"2024-04-17T08:30:52.166105+0000","last_unstale":"2024-04-17T08:30:52.171895+0000","last_undegraded":"2024-04-17T08:30:52.171895+0000","last_fullsized":"2024-04-17T08:30:52.171895+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:08:14.428867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177871+0000","last_change":"2024-04-17T08:30:52.175393+0000","last_active":"2024-04-17T08:30:52.177871+0000","last_peered":"2024-04-17T08:30:52.177871+0000","last_clean":"2024-04-17T08:30:52.177871+0000","last_became_active":"2024-04-17T08:30:52.174996+0000","last_became_peered":"2024-04-17T08:30:52.174996+0000","last_unstale":"2024-04-17T08:30:52.177871+0000","last_undegraded":"2024-04-17T08:30:52.177871+0000","last_fullsized":"2024-04-17T08:30:52.177871+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:45:52.336402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.168520+0000","last_change":"2024-04-17T08:30:52.165274+0000","last_active":"2024-04-17T08:30:52.168520+0000","last_peered":"2024-04-17T08:30:52.168520+0000","last_clean":"2024-04-17T08:30:52.168520+0000","last_became_active":"2024-04-17T08:30:52.164317+0000","last_became_peered":"2024-04-17T08:30:52.164317+0000","last_unstale":"2024-04-17T08:30:52.168520+0000","last_undegraded":"2024-04-17T08:30:52.168520+0000","last_fullsized":"2024-04-17T08:30:52.168520+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:35.976583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178211+0000","last_change":"2024-04-17T08:30:52.176010+0000","last_active":"2024-04-17T08:30:52.178211+0000","last_peered":"2024-04-17T08:30:52.178211+0000","last_clean":"2024-04-17T08:30:52.178211+0000","last_became_active":"2024-04-17T08:30:52.175736+0000","last_became_peered":"2024-04-17T08:30:52.175736+0000","last_unstale":"2024-04-17T08:30:52.178211+0000","last_undegraded":"2024-04-17T08:30:52.178211+0000","last_fullsized":"2024-04-17T08:30:52.178211+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:19.635104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171966+0000","last_change":"2024-04-17T08:30:52.167129+0000","last_active":"2024-04-17T08:30:52.171966+0000","last_peered":"2024-04-17T08:30:52.171966+0000","last_clean":"2024-04-17T08:30:52.171966+0000","last_became_active":"2024-04-17T08:30:52.166156+0000","last_became_peered":"2024-04-17T08:30:52.166156+0000","last_unstale":"2024-04-17T08:30:52.171966+0000","last_undegraded":"2024-04-17T08:30:52.171966+0000","last_fullsized":"2024-04-17T08:30:52.171966+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:27.321036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179766+0000","last_change":"2024-04-17T08:30:52.176742+0000","last_active":"2024-04-17T08:30:52.179766+0000","last_peered":"2024-04-17T08:30:52.179766+0000","last_clean":"2024-04-17T08:30:52.179766+0000","last_became_active":"2024-04-17T08:30:52.176438+0000","last_became_peered":"2024-04-17T08:30:52.176438+0000","last_unstale":"2024-04-17T08:30:52.179766+0000","last_undegraded":"2024-04-17T08:30:52.179766+0000","last_fullsized":"2024-04-17T08:30:52.179766+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:47:35.642668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158622+0000","last_change":"2024-04-17T08:30:52.156525+0000","last_active":"2024-04-17T08:30:52.158622+0000","last_peered":"2024-04-17T08:30:52.158622+0000","last_clean":"2024-04-17T08:30:52.158622+0000","last_became_active":"2024-04-17T08:30:52.151328+0000","last_became_peered":"2024-04-17T08:30:52.151328+0000","last_unstale":"2024-04-17T08:30:52.158622+0000","last_undegraded":"2024-04-17T08:30:52.158622+0000","last_fullsized":"2024-04-17T08:30:52.158622+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:12.798025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170902+0000","last_change":"2024-04-17T08:30:52.165119+0000","last_active":"2024-04-17T08:30:52.170902+0000","last_peered":"2024-04-17T08:30:52.170902+0000","last_clean":"2024-04-17T08:30:52.170902+0000","last_became_active":"2024-04-17T08:30:52.164802+0000","last_became_peered":"2024-04-17T08:30:52.164802+0000","last_unstale":"2024-04-17T08:30:52.170902+0000","last_undegraded":"2024-04-17T08:30:52.170902+0000","last_fullsized":"2024-04-17T08:30:52.170902+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:50.778911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173029+0000","last_change":"2024-04-17T08:30:52.170419+0000","last_active":"2024-04-17T08:30:52.173029+0000","last_peered":"2024-04-17T08:30:52.173029+0000","last_clean":"2024-04-17T08:30:52.173029+0000","last_became_active":"2024-04-17T08:30:52.170133+0000","last_became_peered":"2024-04-17T08:30:52.170133+0000","last_unstale":"2024-04-17T08:30:52.173029+0000","last_undegraded":"2024-04-17T08:30:52.173029+0000","last_fullsized":"2024-04-17T08:30:52.173029+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:05.058041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179632+0000","last_change":"2024-04-17T08:30:52.174805+0000","last_active":"2024-04-17T08:30:52.179632+0000","last_peered":"2024-04-17T08:30:52.179632+0000","last_clean":"2024-04-17T08:30:52.179632+0000","last_became_active":"2024-04-17T08:30:52.174504+0000","last_became_peered":"2024-04-17T08:30:52.174504+0000","last_unstale":"2024-04-17T08:30:52.179632+0000","last_undegraded":"2024-04-17T08:30:52.179632+0000","last_fullsized":"2024-04-17T08:30:52.179632+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:16.951240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178091+0000","last_change":"2024-04-17T08:30:52.174969+0000","last_active":"2024-04-17T08:30:52.178091+0000","last_peered":"2024-04-17T08:30:52.178091+0000","last_clean":"2024-04-17T08:30:52.178091+0000","last_became_active":"2024-04-17T08:30:52.174684+0000","last_became_peered":"2024-04-17T08:30:52.174684+0000","last_unstale":"2024-04-17T08:30:52.178091+0000","last_undegraded":"2024-04-17T08:30:52.178091+0000","last_fullsized":"2024-04-17T08:30:52.178091+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:08.905330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160851+0000","last_change":"2024-04-17T08:30:52.156336+0000","last_active":"2024-04-17T08:30:52.160851+0000","last_peered":"2024-04-17T08:30:52.160851+0000","last_clean":"2024-04-17T08:30:52.160851+0000","last_became_active":"2024-04-17T08:30:52.155626+0000","last_became_peered":"2024-04-17T08:30:52.155626+0000","last_unstale":"2024-04-17T08:30:52.160851+0000","last_undegraded":"2024-04-17T08:30:52.160851+0000","last_fullsized":"2024-04-17T08:30:52.160851+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:21.631248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166022+0000","last_change":"2024-04-17T08:30:52.163152+0000","last_active":"2024-04-17T08:30:52.166022+0000","last_peered":"2024-04-17T08:30:52.166022+0000","last_clean":"2024-04-17T08:30:52.166022+0000","last_became_active":"2024-04-17T08:30:52.159167+0000","last_became_peered":"2024-04-17T08:30:52.159167+0000","last_unstale":"2024-04-17T08:30:52.166022+0000","last_undegraded":"2024-04-17T08:30:52.166022+0000","last_fullsized":"2024-04-17T08:30:52.166022+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:20:01.467968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175321+0000","last_change":"2024-04-17T08:30:52.171195+0000","last_active":"2024-04-17T08:30:52.175321+0000","last_peered":"2024-04-17T08:30:52.175321+0000","last_clean":"2024-04-17T08:30:52.175321+0000","last_became_active":"2024-04-17T08:30:52.166955+0000","last_became_peered":"2024-04-17T08:30:52.166955+0000","last_unstale":"2024-04-17T08:30:52.175321+0000","last_undegraded":"2024-04-17T08:30:52.175321+0000","last_fullsized":"2024-04-17T08:30:52.175321+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:53.772266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170257+0000","last_change":"2024-04-17T08:30:52.165446+0000","last_active":"2024-04-17T08:30:52.170257+0000","last_peered":"2024-04-17T08:30:52.170257+0000","last_clean":"2024-04-17T08:30:52.170257+0000","last_became_active":"2024-04-17T08:30:52.164177+0000","last_became_peered":"2024-04-17T08:30:52.164177+0000","last_unstale":"2024-04-17T08:30:52.170257+0000","last_undegraded":"2024-04-17T08:30:52.170257+0000","last_fullsized":"2024-04-17T08:30:52.170257+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:11.065845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166232+0000","last_change":"2024-04-17T08:30:52.162598+0000","last_active":"2024-04-17T08:30:52.166232+0000","last_peered":"2024-04-17T08:30:52.166232+0000","last_clean":"2024-04-17T08:30:52.166232+0000","last_became_active":"2024-04-17T08:30:52.162302+0000","last_became_peered":"2024-04-17T08:30:52.162302+0000","last_unstale":"2024-04-17T08:30:52.166232+0000","last_undegraded":"2024-04-17T08:30:52.166232+0000","last_fullsized":"2024-04-17T08:30:52.166232+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:17.734042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160413+0000","last_change":"2024-04-17T08:30:52.153683+0000","last_active":"2024-04-17T08:30:52.160413+0000","last_peered":"2024-04-17T08:30:52.160413+0000","last_clean":"2024-04-17T08:30:52.160413+0000","last_became_active":"2024-04-17T08:30:52.153251+0000","last_became_peered":"2024-04-17T08:30:52.153251+0000","last_unstale":"2024-04-17T08:30:52.160413+0000","last_undegraded":"2024-04-17T08:30:52.160413+0000","last_fullsized":"2024-04-17T08:30:52.160413+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:43.322648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179691+0000","last_change":"2024-04-17T08:30:52.176624+0000","last_active":"2024-04-17T08:30:52.179691+0000","last_peered":"2024-04-17T08:30:52.179691+0000","last_clean":"2024-04-17T08:30:52.179691+0000","last_became_active":"2024-04-17T08:30:52.175772+0000","last_became_peered":"2024-04-17T08:30:52.175772+0000","last_unstale":"2024-04-17T08:30:52.179691+0000","last_undegraded":"2024-04-17T08:30:52.179691+0000","last_fullsized":"2024-04-17T08:30:52.179691+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:26.416829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"181'4","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186629+0000","last_change":"2024-04-17T08:30:51.143969+0000","last_active":"2024-04-17T08:30:52.186629+0000","last_peered":"2024-04-17T08:30:52.186629+0000","last_clean":"2024-04-17T08:30:52.186629+0000","last_became_active":"2024-04-17T08:30:51.143210+0000","last_became_peered":"2024-04-17T08:30:51.143210+0000","last_unstale":"2024-04-17T08:30:52.186629+0000","last_undegraded":"2024-04-17T08:30:52.186629+0000","last_fullsized":"2024-04-17T08:30:52.186629+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:06.446696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165031+0000","last_change":"2024-04-17T08:30:52.160627+0000","last_active":"2024-04-17T08:30:52.165031+0000","last_peered":"2024-04-17T08:30:52.165031+0000","last_clean":"2024-04-17T08:30:52.165031+0000","last_became_active":"2024-04-17T08:30:52.160343+0000","last_became_peered":"2024-04-17T08:30:52.160343+0000","last_unstale":"2024-04-17T08:30:52.165031+0000","last_undegraded":"2024-04-17T08:30:52.165031+0000","last_fullsized":"2024-04-17T08:30:52.165031+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:08:14.121360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.184606+0000","last_change":"2024-04-17T08:30:51.143415+0000","last_active":"2024-04-17T08:30:52.184606+0000","last_peered":"2024-04-17T08:30:52.184606+0000","last_clean":"2024-04-17T08:30:52.184606+0000","last_became_active":"2024-04-17T08:30:51.142223+0000","last_became_peered":"2024-04-17T08:30:51.142223+0000","last_unstale":"2024-04-17T08:30:52.184606+0000","last_undegraded":"2024-04-17T08:30:52.184606+0000","last_fullsized":"2024-04-17T08:30:52.184606+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:28.985117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177185+0000","last_change":"2024-04-17T08:30:52.172416+0000","last_active":"2024-04-17T08:30:52.177185+0000","last_peered":"2024-04-17T08:30:52.177185+0000","last_clean":"2024-04-17T08:30:52.177185+0000","last_became_active":"2024-04-17T08:30:52.170458+0000","last_became_peered":"2024-04-17T08:30:52.170458+0000","last_unstale":"2024-04-17T08:30:52.177185+0000","last_undegraded":"2024-04-17T08:30:52.177185+0000","last_fullsized":"2024-04-17T08:30:52.177185+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:45.550828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186011+0000","last_change":"2024-04-17T08:30:51.131325+0000","last_active":"2024-04-17T08:30:52.186011+0000","last_peered":"2024-04-17T08:30:52.186011+0000","last_clean":"2024-04-17T08:30:52.186011+0000","last_became_active":"2024-04-17T08:30:51.131009+0000","last_became_peered":"2024-04-17T08:30:51.131009+0000","last_unstale":"2024-04-17T08:30:52.186011+0000","last_undegraded":"2024-04-17T08:30:52.186011+0000","last_fullsized":"2024-04-17T08:30:52.186011+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:16.023338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173003+0000","last_change":"2024-04-17T08:30:52.168619+0000","last_active":"2024-04-17T08:30:52.173003+0000","last_peered":"2024-04-17T08:30:52.173003+0000","last_clean":"2024-04-17T08:30:52.173003+0000","last_became_active":"2024-04-17T08:30:52.168309+0000","last_became_peered":"2024-04-17T08:30:52.168309+0000","last_unstale":"2024-04-17T08:30:52.173003+0000","last_undegraded":"2024-04-17T08:30:52.173003+0000","last_fullsized":"2024-04-17T08:30:52.173003+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:36.171565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186470+0000","last_change":"2024-04-17T08:30:51.138295+0000","last_active":"2024-04-17T08:30:52.186470+0000","last_peered":"2024-04-17T08:30:52.186470+0000","last_clean":"2024-04-17T08:30:52.186470+0000","last_became_active":"2024-04-17T08:30:51.137747+0000","last_became_peered":"2024-04-17T08:30:51.137747+0000","last_unstale":"2024-04-17T08:30:52.186470+0000","last_undegraded":"2024-04-17T08:30:52.186470+0000","last_fullsized":"2024-04-17T08:30:52.186470+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:27.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166420+0000","last_change":"2024-04-17T08:30:52.163852+0000","last_active":"2024-04-17T08:30:52.166420+0000","last_peered":"2024-04-17T08:30:52.166420+0000","last_clean":"2024-04-17T08:30:52.166420+0000","last_became_active":"2024-04-17T08:30:52.162957+0000","last_became_peered":"2024-04-17T08:30:52.162957+0000","last_unstale":"2024-04-17T08:30:52.166420+0000","last_undegraded":"2024-04-17T08:30:52.166420+0000","last_fullsized":"2024-04-17T08:30:52.166420+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:00.962539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.181381+0000","last_change":"2024-04-17T08:30:51.140186+0000","last_active":"2024-04-17T08:30:52.181381+0000","last_peered":"2024-04-17T08:30:52.181381+0000","last_clean":"2024-04-17T08:30:52.181381+0000","last_became_active":"2024-04-17T08:30:51.139852+0000","last_became_peered":"2024-04-17T08:30:51.139852+0000","last_unstale":"2024-04-17T08:30:52.181381+0000","last_undegraded":"2024-04-17T08:30:52.181381+0000","last_fullsized":"2024-04-17T08:30:52.181381+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:01.210967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173089+0000","last_change":"2024-04-17T08:30:52.168030+0000","last_active":"2024-04-17T08:30:52.173089+0000","last_peered":"2024-04-17T08:30:52.173089+0000","last_clean":"2024-04-17T08:30:52.173089+0000","last_became_active":"2024-04-17T08:30:52.167189+0000","last_became_peered":"2024-04-17T08:30:52.167189+0000","last_unstale":"2024-04-17T08:30:52.173089+0000","last_undegraded":"2024-04-17T08:30:52.173089+0000","last_fullsized":"2024-04-17T08:30:52.173089+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:16.884795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.183740+0000","last_change":"2024-04-17T08:30:51.142844+0000","last_active":"2024-04-17T08:30:52.183740+0000","last_peered":"2024-04-17T08:30:52.183740+0000","last_clean":"2024-04-17T08:30:52.183740+0000","last_became_active":"2024-04-17T08:30:51.142524+0000","last_became_peered":"2024-04-17T08:30:51.142524+0000","last_unstale":"2024-04-17T08:30:52.183740+0000","last_undegraded":"2024-04-17T08:30:52.183740+0000","last_fullsized":"2024-04-17T08:30:52.183740+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:03.644316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.159396+0000","last_change":"2024-04-17T08:30:52.151491+0000","last_active":"2024-04-17T08:30:52.159396+0000","last_peered":"2024-04-17T08:30:52.159396+0000","last_clean":"2024-04-17T08:30:52.159396+0000","last_became_active":"2024-04-17T08:30:52.151190+0000","last_became_peered":"2024-04-17T08:30:52.151190+0000","last_unstale":"2024-04-17T08:30:52.159396+0000","last_undegraded":"2024-04-17T08:30:52.159396+0000","last_fullsized":"2024-04-17T08:30:52.159396+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:00.296364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.131428+0000","last_change":"2024-04-17T08:30:51.134498+0000","last_active":"2024-04-17T08:30:52.131428+0000","last_peered":"2024-04-17T08:30:52.131428+0000","last_clean":"2024-04-17T08:30:52.131428+0000","last_became_active":"2024-04-17T08:30:51.134207+0000","last_became_peered":"2024-04-17T08:30:51.134207+0000","last_unstale":"2024-04-17T08:30:52.131428+0000","last_undegraded":"2024-04-17T08:30:52.131428+0000","last_fullsized":"2024-04-17T08:30:52.131428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:07.771377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170427+0000","last_change":"2024-04-17T08:30:52.166488+0000","last_active":"2024-04-17T08:30:52.170427+0000","last_peered":"2024-04-17T08:30:52.170427+0000","last_clean":"2024-04-17T08:30:52.170427+0000","last_became_active":"2024-04-17T08:30:52.165289+0000","last_became_peered":"2024-04-17T08:30:52.165289+0000","last_unstale":"2024-04-17T08:30:52.170427+0000","last_undegraded":"2024-04-17T08:30:52.170427+0000","last_fullsized":"2024-04-17T08:30:52.170427+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:14.995324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.137948+0000","last_change":"2024-04-17T08:30:51.136258+0000","last_active":"2024-04-17T08:30:52.137948+0000","last_peered":"2024-04-17T08:30:52.137948+0000","last_clean":"2024-04-17T08:30:52.137948+0000","last_became_active":"2024-04-17T08:30:51.135949+0000","last_became_peered":"2024-04-17T08:30:51.135949+0000","last_unstale":"2024-04-17T08:30:52.137948+0000","last_undegraded":"2024-04-17T08:30:52.137948+0000","last_fullsized":"2024-04-17T08:30:52.137948+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:25.132852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.169244+0000","last_change":"2024-04-17T08:30:52.160217+0000","last_active":"2024-04-17T08:30:52.169244+0000","last_peered":"2024-04-17T08:30:52.169244+0000","last_clean":"2024-04-17T08:30:52.169244+0000","last_became_active":"2024-04-17T08:30:52.159912+0000","last_became_peered":"2024-04-17T08:30:52.159912+0000","last_unstale":"2024-04-17T08:30:52.169244+0000","last_undegraded":"2024-04-17T08:30:52.169244+0000","last_fullsized":"2024-04-17T08:30:52.169244+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:39:08.638199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.150576+0000","last_change":"2024-04-17T08:30:51.143277+0000","last_active":"2024-04-17T08:30:52.150576+0000","last_peered":"2024-04-17T08:30:52.150576+0000","last_clean":"2024-04-17T08:30:52.150576+0000","last_became_active":"2024-04-17T08:30:51.142069+0000","last_became_peered":"2024-04-17T08:30:51.142069+0000","last_unstale":"2024-04-17T08:30:52.150576+0000","last_undegraded":"2024-04-17T08:30:52.150576+0000","last_fullsized":"2024-04-17T08:30:52.150576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:48.094315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177883+0000","last_change":"2024-04-17T08:30:52.175579+0000","last_active":"2024-04-17T08:30:52.177883+0000","last_peered":"2024-04-17T08:30:52.177883+0000","last_clean":"2024-04-17T08:30:52.177883+0000","last_became_active":"2024-04-17T08:30:52.174298+0000","last_became_peered":"2024-04-17T08:30:52.174298+0000","last_unstale":"2024-04-17T08:30:52.177883+0000","last_undegraded":"2024-04-17T08:30:52.177883+0000","last_fullsized":"2024-04-17T08:30:52.177883+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:34.355725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.162779+0000","last_change":"2024-04-17T08:30:51.142822+0000","last_active":"2024-04-17T08:30:52.162779+0000","last_peered":"2024-04-17T08:30:52.162779+0000","last_clean":"2024-04-17T08:30:52.162779+0000","last_became_active":"2024-04-17T08:30:51.142316+0000","last_became_peered":"2024-04-17T08:30:51.142316+0000","last_unstale":"2024-04-17T08:30:52.162779+0000","last_undegraded":"2024-04-17T08:30:52.162779+0000","last_fullsized":"2024-04-17T08:30:52.162779+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:55.787013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160876+0000","last_change":"2024-04-17T08:30:52.156578+0000","last_active":"2024-04-17T08:30:52.160876+0000","last_peered":"2024-04-17T08:30:52.160876+0000","last_clean":"2024-04-17T08:30:52.160876+0000","last_became_active":"2024-04-17T08:30:52.156025+0000","last_became_peered":"2024-04-17T08:30:52.156025+0000","last_unstale":"2024-04-17T08:30:52.160876+0000","last_undegraded":"2024-04-17T08:30:52.160876+0000","last_fullsized":"2024-04-17T08:30:52.160876+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:38.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"181'6","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.188758+0000","last_change":"2024-04-17T08:30:51.130242+0000","last_active":"2024-04-17T08:30:52.188758+0000","last_peered":"2024-04-17T08:30:52.188758+0000","last_clean":"2024-04-17T08:30:52.188758+0000","last_became_active":"2024-04-17T08:30:51.129868+0000","last_became_peered":"2024-04-17T08:30:51.129868+0000","last_unstale":"2024-04-17T08:30:52.188758+0000","last_undegraded":"2024-04-17T08:30:52.188758+0000","last_fullsized":"2024-04-17T08:30:52.188758+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:01.224299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175320+0000","last_change":"2024-04-17T08:30:52.169255+0000","last_active":"2024-04-17T08:30:52.175320+0000","last_peered":"2024-04-17T08:30:52.175320+0000","last_clean":"2024-04-17T08:30:52.175320+0000","last_became_active":"2024-04-17T08:30:52.168033+0000","last_became_peered":"2024-04-17T08:30:52.168033+0000","last_unstale":"2024-04-17T08:30:52.175320+0000","last_undegraded":"2024-04-17T08:30:52.175320+0000","last_fullsized":"2024-04-17T08:30:52.175320+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:54.749428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186033+0000","last_change":"2024-04-17T08:30:51.124880+0000","last_active":"2024-04-17T08:30:52.186033+0000","last_peered":"2024-04-17T08:30:52.186033+0000","last_clean":"2024-04-17T08:30:52.186033+0000","last_became_active":"2024-04-17T08:30:51.122582+0000","last_became_peered":"2024-04-17T08:30:51.122582+0000","last_unstale":"2024-04-17T08:30:52.186033+0000","last_undegraded":"2024-04-17T08:30:52.186033+0000","last_fullsized":"2024-04-17T08:30:52.186033+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:32:20.506153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166254+0000","last_change":"2024-04-17T08:30:52.163388+0000","last_active":"2024-04-17T08:30:52.166254+0000","last_peered":"2024-04-17T08:30:52.166254+0000","last_clean":"2024-04-17T08:30:52.166254+0000","last_became_active":"2024-04-17T08:30:52.163073+0000","last_became_peered":"2024-04-17T08:30:52.163073+0000","last_unstale":"2024-04-17T08:30:52.166254+0000","last_undegraded":"2024-04-17T08:30:52.166254+0000","last_fullsized":"2024-04-17T08:30:52.166254+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:42:59.104395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.155522+0000","last_change":"2024-04-17T08:30:51.139507+0000","last_active":"2024-04-17T08:30:52.155522+0000","last_peered":"2024-04-17T08:30:52.155522+0000","last_clean":"2024-04-17T08:30:52.155522+0000","last_became_active":"2024-04-17T08:30:51.139184+0000","last_became_peered":"2024-04-17T08:30:51.139184+0000","last_unstale":"2024-04-17T08:30:52.155522+0000","last_undegraded":"2024-04-17T08:30:52.155522+0000","last_fullsized":"2024-04-17T08:30:52.155522+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:13.432503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172690+0000","last_change":"2024-04-17T08:30:52.169710+0000","last_active":"2024-04-17T08:30:52.172690+0000","last_peered":"2024-04-17T08:30:52.172690+0000","last_clean":"2024-04-17T08:30:52.172690+0000","last_became_active":"2024-04-17T08:30:52.168312+0000","last_became_peered":"2024-04-17T08:30:52.168312+0000","last_unstale":"2024-04-17T08:30:52.172690+0000","last_undegraded":"2024-04-17T08:30:52.172690+0000","last_fullsized":"2024-04-17T08:30:52.172690+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:55.339072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.149209+0000","last_change":"2024-04-17T08:30:51.139897+0000","last_active":"2024-04-17T08:30:52.149209+0000","last_peered":"2024-04-17T08:30:52.149209+0000","last_clean":"2024-04-17T08:30:52.149209+0000","last_became_active":"2024-04-17T08:30:51.138510+0000","last_became_peered":"2024-04-17T08:30:51.138510+0000","last_unstale":"2024-04-17T08:30:52.149209+0000","last_undegraded":"2024-04-17T08:30:52.149209+0000","last_fullsized":"2024-04-17T08:30:52.149209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:17.007012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165073+0000","last_change":"2024-04-17T08:30:52.162443+0000","last_active":"2024-04-17T08:30:52.165073+0000","last_peered":"2024-04-17T08:30:52.165073+0000","last_clean":"2024-04-17T08:30:52.165073+0000","last_became_active":"2024-04-17T08:30:52.162098+0000","last_became_peered":"2024-04-17T08:30:52.162098+0000","last_unstale":"2024-04-17T08:30:52.165073+0000","last_undegraded":"2024-04-17T08:30:52.165073+0000","last_fullsized":"2024-04-17T08:30:52.165073+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:27:05.273119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"181'8","reported_seq":20,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.183879+0000","last_change":"2024-04-17T08:30:51.137868+0000","last_active":"2024-04-17T08:30:52.183879+0000","last_peered":"2024-04-17T08:30:52.183879+0000","last_clean":"2024-04-17T08:30:52.183879+0000","last_became_active":"2024-04-17T08:30:51.136385+0000","last_became_peered":"2024-04-17T08:30:51.136385+0000","last_unstale":"2024-04-17T08:30:52.183879+0000","last_undegraded":"2024-04-17T08:30:52.183879+0000","last_fullsized":"2024-04-17T08:30:52.183879+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:28.940365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177787+0000","last_change":"2024-04-17T08:30:52.175472+0000","last_active":"2024-04-17T08:30:52.177787+0000","last_peered":"2024-04-17T08:30:52.177787+0000","last_clean":"2024-04-17T08:30:52.177787+0000","last_became_active":"2024-04-17T08:30:52.174354+0000","last_became_peered":"2024-04-17T08:30:52.174354+0000","last_unstale":"2024-04-17T08:30:52.177787+0000","last_undegraded":"2024-04-17T08:30:52.177787+0000","last_fullsized":"2024-04-17T08:30:52.177787+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:47.591301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.184632+0000","last_change":"2024-04-17T08:30:51.140559+0000","last_active":"2024-04-17T08:30:52.184632+0000","last_peered":"2024-04-17T08:30:52.184632+0000","last_clean":"2024-04-17T08:30:52.184632+0000","last_became_active":"2024-04-17T08:30:51.140275+0000","last_became_peered":"2024-04-17T08:30:51.140275+0000","last_unstale":"2024-04-17T08:30:52.184632+0000","last_undegraded":"2024-04-17T08:30:52.184632+0000","last_fullsized":"2024-04-17T08:30:52.184632+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:53.733492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171821+0000","last_change":"2024-04-17T08:30:52.164139+0000","last_active":"2024-04-17T08:30:52.171821+0000","last_peered":"2024-04-17T08:30:52.171821+0000","last_clean":"2024-04-17T08:30:52.171821+0000","last_became_active":"2024-04-17T08:30:52.162245+0000","last_became_peered":"2024-04-17T08:30:52.162245+0000","last_unstale":"2024-04-17T08:30:52.171821+0000","last_undegraded":"2024-04-17T08:30:52.171821+0000","last_fullsized":"2024-04-17T08:30:52.171821+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:19:37.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186352+0000","last_change":"2024-04-17T08:30:51.136065+0000","last_active":"2024-04-17T08:30:52.186352+0000","last_peered":"2024-04-17T08:30:52.186352+0000","last_clean":"2024-04-17T08:30:52.186352+0000","last_became_active":"2024-04-17T08:30:51.135734+0000","last_became_peered":"2024-04-17T08:30:51.135734+0000","last_unstale":"2024-04-17T08:30:52.186352+0000","last_undegraded":"2024-04-17T08:30:52.186352+0000","last_fullsized":"2024-04-17T08:30:52.186352+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:30.534866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.174920+0000","last_change":"2024-04-17T08:30:52.167748+0000","last_active":"2024-04-17T08:30:52.174920+0000","last_peered":"2024-04-17T08:30:52.174920+0000","last_clean":"2024-04-17T08:30:52.174920+0000","last_became_active":"2024-04-17T08:30:52.166360+0000","last_became_peered":"2024-04-17T08:30:52.166360+0000","last_unstale":"2024-04-17T08:30:52.174920+0000","last_undegraded":"2024-04-17T08:30:52.174920+0000","last_fullsized":"2024-04-17T08:30:52.174920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.660252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.180964+0000","last_change":"2024-04-17T08:30:51.134068+0000","last_active":"2024-04-17T08:30:52.180964+0000","last_peered":"2024-04-17T08:30:52.180964+0000","last_clean":"2024-04-17T08:30:52.180964+0000","last_became_active":"2024-04-17T08:30:51.133169+0000","last_became_peered":"2024-04-17T08:30:51.133169+0000","last_unstale":"2024-04-17T08:30:52.180964+0000","last_undegraded":"2024-04-17T08:30:52.180964+0000","last_fullsized":"2024-04-17T08:30:52.180964+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:11:38.141287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178171+0000","last_change":"2024-04-17T08:30:52.173357+0000","last_active":"2024-04-17T08:30:52.178171+0000","last_peered":"2024-04-17T08:30:52.178171+0000","last_clean":"2024-04-17T08:30:52.178171+0000","last_became_active":"2024-04-17T08:30:52.173047+0000","last_became_peered":"2024-04-17T08:30:52.173047+0000","last_unstale":"2024-04-17T08:30:52.178171+0000","last_undegraded":"2024-04-17T08:30:52.178171+0000","last_fullsized":"2024-04-17T08:30:52.178171+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:05.455922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.151283+0000","last_change":"2024-04-17T08:30:51.140780+0000","last_active":"2024-04-17T08:30:52.151283+0000","last_peered":"2024-04-17T08:30:52.151283+0000","last_clean":"2024-04-17T08:30:52.151283+0000","last_became_active":"2024-04-17T08:30:51.140513+0000","last_became_peered":"2024-04-17T08:30:51.140513+0000","last_unstale":"2024-04-17T08:30:52.151283+0000","last_undegraded":"2024-04-17T08:30:52.151283+0000","last_fullsized":"2024-04-17T08:30:52.151283+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:02.985822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158643+0000","last_change":"2024-04-17T08:30:52.156813+0000","last_active":"2024-04-17T08:30:52.158643+0000","last_peered":"2024-04-17T08:30:52.158643+0000","last_clean":"2024-04-17T08:30:52.158643+0000","last_became_active":"2024-04-17T08:30:52.153109+0000","last_became_peered":"2024-04-17T08:30:52.153109+0000","last_unstale":"2024-04-17T08:30:52.158643+0000","last_undegraded":"2024-04-17T08:30:52.158643+0000","last_fullsized":"2024-04-17T08:30:52.158643+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:37.004223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.185557+0000","last_change":"2024-04-17T08:30:51.125597+0000","last_active":"2024-04-17T08:30:52.185557+0000","last_peered":"2024-04-17T08:30:52.185557+0000","last_clean":"2024-04-17T08:30:52.185557+0000","last_became_active":"2024-04-17T08:30:51.124747+0000","last_became_peered":"2024-04-17T08:30:51.124747+0000","last_unstale":"2024-04-17T08:30:52.185557+0000","last_undegraded":"2024-04-17T08:30:52.185557+0000","last_fullsized":"2024-04-17T08:30:52.185557+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:31:27.197036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.164281+0000","last_change":"2024-04-17T08:30:52.155554+0000","last_active":"2024-04-17T08:30:52.164281+0000","last_peered":"2024-04-17T08:30:52.164281+0000","last_clean":"2024-04-17T08:30:52.164281+0000","last_became_active":"2024-04-17T08:30:52.155149+0000","last_became_peered":"2024-04-17T08:30:52.155149+0000","last_unstale":"2024-04-17T08:30:52.164281+0000","last_undegraded":"2024-04-17T08:30:52.164281+0000","last_fullsized":"2024-04-17T08:30:52.164281+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:50.472795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.154968+0000","last_change":"2024-04-17T08:30:51.143092+0000","last_active":"2024-04-17T08:30:52.154968+0000","last_peered":"2024-04-17T08:30:52.154968+0000","last_clean":"2024-04-17T08:30:52.154968+0000","last_became_active":"2024-04-17T08:30:51.142666+0000","last_became_peered":"2024-04-17T08:30:51.142666+0000","last_unstale":"2024-04-17T08:30:52.154968+0000","last_undegraded":"2024-04-17T08:30:52.154968+0000","last_fullsized":"2024-04-17T08:30:52.154968+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:56.488726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166919+0000","last_change":"2024-04-17T08:30:52.163125+0000","last_active":"2024-04-17T08:30:52.166919+0000","last_peered":"2024-04-17T08:30:52.166919+0000","last_clean":"2024-04-17T08:30:52.166919+0000","last_became_active":"2024-04-17T08:30:52.162834+0000","last_became_peered":"2024-04-17T08:30:52.162834+0000","last_unstale":"2024-04-17T08:30:52.166919+0000","last_undegraded":"2024-04-17T08:30:52.166919+0000","last_fullsized":"2024-04-17T08:30:52.166919+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:41:39.786033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.151826+0000","last_change":"2024-04-17T08:30:51.140770+0000","last_active":"2024-04-17T08:30:52.151826+0000","last_peered":"2024-04-17T08:30:52.151826+0000","last_clean":"2024-04-17T08:30:52.151826+0000","last_became_active":"2024-04-17T08:30:51.140454+0000","last_became_peered":"2024-04-17T08:30:51.140454+0000","last_unstale":"2024-04-17T08:30:52.151826+0000","last_undegraded":"2024-04-17T08:30:52.151826+0000","last_fullsized":"2024-04-17T08:30:52.151826+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:16:19.278306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.168533+0000","last_change":"2024-04-17T08:30:52.158816+0000","last_active":"2024-04-17T08:30:52.168533+0000","last_peered":"2024-04-17T08:30:52.168533+0000","last_clean":"2024-04-17T08:30:52.168533+0000","last_became_active":"2024-04-17T08:30:52.155384+0000","last_became_peered":"2024-04-17T08:30:52.155384+0000","last_unstale":"2024-04-17T08:30:52.168533+0000","last_undegraded":"2024-04-17T08:30:52.168533+0000","last_fullsized":"2024-04-17T08:30:52.168533+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:47.375416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.152252+0000","last_change":"2024-04-17T08:30:51.138858+0000","last_active":"2024-04-17T08:30:52.152252+0000","last_peered":"2024-04-17T08:30:52.152252+0000","last_clean":"2024-04-17T08:30:52.152252+0000","last_became_active":"2024-04-17T08:30:51.138495+0000","last_became_peered":"2024-04-17T08:30:51.138495+0000","last_unstale":"2024-04-17T08:30:52.152252+0000","last_undegraded":"2024-04-17T08:30:52.152252+0000","last_fullsized":"2024-04-17T08:30:52.152252+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:40.512510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171344+0000","last_change":"2024-04-17T08:30:52.169075+0000","last_active":"2024-04-17T08:30:52.171344+0000","last_peered":"2024-04-17T08:30:52.171344+0000","last_clean":"2024-04-17T08:30:52.171344+0000","last_became_active":"2024-04-17T08:30:52.168731+0000","last_became_peered":"2024-04-17T08:30:52.168731+0000","last_unstale":"2024-04-17T08:30:52.171344+0000","last_undegraded":"2024-04-17T08:30:52.171344+0000","last_fullsized":"2024-04-17T08:30:52.171344+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:32:00.486067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.185227+0000","last_change":"2024-04-17T08:30:51.145555+0000","last_active":"2024-04-17T08:30:52.185227+0000","last_peered":"2024-04-17T08:30:52.185227+0000","last_clean":"2024-04-17T08:30:52.185227+0000","last_became_active":"2024-04-17T08:30:51.145244+0000","last_became_peered":"2024-04-17T08:30:51.145244+0000","last_unstale":"2024-04-17T08:30:52.185227+0000","last_undegraded":"2024-04-17T08:30:52.185227+0000","last_fullsized":"2024-04-17T08:30:52.185227+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:58.861494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167921+0000","last_change":"2024-04-17T08:30:52.166234+0000","last_active":"2024-04-17T08:30:52.167921+0000","last_peered":"2024-04-17T08:30:52.167921+0000","last_clean":"2024-04-17T08:30:52.167921+0000","last_became_active":"2024-04-17T08:30:52.165861+0000","last_became_peered":"2024-04-17T08:30:52.165861+0000","last_unstale":"2024-04-17T08:30:52.167921+0000","last_undegraded":"2024-04-17T08:30:52.167921+0000","last_fullsized":"2024-04-17T08:30:52.167921+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:21:31.671024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158671+0000","last_change":"2024-04-17T08:30:51.144619+0000","last_active":"2024-04-17T08:30:52.158671+0000","last_peered":"2024-04-17T08:30:52.158671+0000","last_clean":"2024-04-17T08:30:52.158671+0000","last_became_active":"2024-04-17T08:30:51.144303+0000","last_became_peered":"2024-04-17T08:30:51.144303+0000","last_unstale":"2024-04-17T08:30:52.158671+0000","last_undegraded":"2024-04-17T08:30:52.158671+0000","last_fullsized":"2024-04-17T08:30:52.158671+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:33.163237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166384+0000","last_change":"2024-04-17T08:30:52.163463+0000","last_active":"2024-04-17T08:30:52.166384+0000","last_peered":"2024-04-17T08:30:52.166384+0000","last_clean":"2024-04-17T08:30:52.166384+0000","last_became_active":"2024-04-17T08:30:52.163136+0000","last_became_peered":"2024-04-17T08:30:52.163136+0000","last_unstale":"2024-04-17T08:30:52.166384+0000","last_undegraded":"2024-04-17T08:30:52.166384+0000","last_fullsized":"2024-04-17T08:30:52.166384+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:43.508755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158612+0000","last_change":"2024-04-17T08:30:51.145359+0000","last_active":"2024-04-17T08:30:52.158612+0000","last_peered":"2024-04-17T08:30:52.158612+0000","last_clean":"2024-04-17T08:30:52.158612+0000","last_became_active":"2024-04-17T08:30:51.142679+0000","last_became_peered":"2024-04-17T08:30:51.142679+0000","last_unstale":"2024-04-17T08:30:52.158612+0000","last_undegraded":"2024-04-17T08:30:52.158612+0000","last_fullsized":"2024-04-17T08:30:52.158612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:06.357590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175002+0000","last_change":"2024-04-17T08:30:52.168823+0000","last_active":"2024-04-17T08:30:52.175002+0000","last_peered":"2024-04-17T08:30:52.175002+0000","last_clean":"2024-04-17T08:30:52.175002+0000","last_became_active":"2024-04-17T08:30:52.168529+0000","last_became_peered":"2024-04-17T08:30:52.168529+0000","last_unstale":"2024-04-17T08:30:52.175002+0000","last_undegraded":"2024-04-17T08:30:52.175002+0000","last_fullsized":"2024-04-17T08:30:52.175002+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:43.897523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.155841+0000","last_change":"2024-04-17T08:30:51.143759+0000","last_active":"2024-04-17T08:30:52.155841+0000","last_peered":"2024-04-17T08:30:52.155841+0000","last_clean":"2024-04-17T08:30:52.155841+0000","last_became_active":"2024-04-17T08:30:51.142591+0000","last_became_peered":"2024-04-17T08:30:51.142591+0000","last_unstale":"2024-04-17T08:30:52.155841+0000","last_undegraded":"2024-04-17T08:30:52.155841+0000","last_fullsized":"2024-04-17T08:30:52.155841+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:51.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.156203+0000","last_change":"2024-04-17T08:30:52.148094+0000","last_active":"2024-04-17T08:30:52.156203+0000","last_peered":"2024-04-17T08:30:52.156203+0000","last_clean":"2024-04-17T08:30:52.156203+0000","last_became_active":"2024-04-17T08:30:52.147483+0000","last_became_peered":"2024-04-17T08:30:52.147483+0000","last_unstale":"2024-04-17T08:30:52.156203+0000","last_undegraded":"2024-04-17T08:30:52.156203+0000","last_fullsized":"2024-04-17T08:30:52.156203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:40.756138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.144208+0000","last_change":"2024-04-17T08:30:51.125300+0000","last_active":"2024-04-17T08:30:52.144208+0000","last_peered":"2024-04-17T08:30:52.144208+0000","last_clean":"2024-04-17T08:30:52.144208+0000","last_became_active":"2024-04-17T08:30:51.123820+0000","last_became_peered":"2024-04-17T08:30:51.123820+0000","last_unstale":"2024-04-17T08:30:52.144208+0000","last_undegraded":"2024-04-17T08:30:52.144208+0000","last_fullsized":"2024-04-17T08:30:52.144208+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:36.069030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172848+0000","last_change":"2024-04-17T08:30:52.169570+0000","last_active":"2024-04-17T08:30:52.172848+0000","last_peered":"2024-04-17T08:30:52.172848+0000","last_clean":"2024-04-17T08:30:52.172848+0000","last_became_active":"2024-04-17T08:30:52.167958+0000","last_became_peered":"2024-04-17T08:30:52.167958+0000","last_unstale":"2024-04-17T08:30:52.172848+0000","last_undegraded":"2024-04-17T08:30:52.172848+0000","last_fullsized":"2024-04-17T08:30:52.172848+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:11.058906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158607+0000","last_change":"2024-04-17T08:30:51.139748+0000","last_active":"2024-04-17T08:30:52.158607+0000","last_peered":"2024-04-17T08:30:52.158607+0000","last_clean":"2024-04-17T08:30:52.158607+0000","last_became_active":"2024-04-17T08:30:51.138377+0000","last_became_peered":"2024-04-17T08:30:51.138377+0000","last_unstale":"2024-04-17T08:30:52.158607+0000","last_undegraded":"2024-04-17T08:30:52.158607+0000","last_fullsized":"2024-04-17T08:30:52.158607+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:04.499164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167048+0000","last_change":"2024-04-17T08:30:52.162843+0000","last_active":"2024-04-17T08:30:52.167048+0000","last_peered":"2024-04-17T08:30:52.167048+0000","last_clean":"2024-04-17T08:30:52.167048+0000","last_became_active":"2024-04-17T08:30:52.162563+0000","last_became_peered":"2024-04-17T08:30:52.162563+0000","last_unstale":"2024-04-17T08:30:52.167048+0000","last_undegraded":"2024-04-17T08:30:52.167048+0000","last_fullsized":"2024-04-17T08:30:52.167048+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:58.919566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.154467+0000","last_change":"2024-04-17T08:30:51.137838+0000","last_active":"2024-04-17T08:30:52.154467+0000","last_peered":"2024-04-17T08:30:52.154467+0000","last_clean":"2024-04-17T08:30:52.154467+0000","last_became_active":"2024-04-17T08:30:51.135840+0000","last_became_peered":"2024-04-17T08:30:51.135840+0000","last_unstale":"2024-04-17T08:30:52.154467+0000","last_undegraded":"2024-04-17T08:30:52.154467+0000","last_fullsized":"2024-04-17T08:30:52.154467+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.035116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170405+0000","last_change":"2024-04-17T08:30:52.161904+0000","last_active":"2024-04-17T08:30:52.170405+0000","last_peered":"2024-04-17T08:30:52.170405+0000","last_clean":"2024-04-17T08:30:52.170405+0000","last_became_active":"2024-04-17T08:30:52.159362+0000","last_became_peered":"2024-04-17T08:30:52.159362+0000","last_unstale":"2024-04-17T08:30:52.170405+0000","last_undegraded":"2024-04-17T08:30:52.170405+0000","last_fullsized":"2024-04-17T08:30:52.170405+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:57.028786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.142935+0000","last_change":"2024-04-17T08:30:51.140395+0000","last_active":"2024-04-17T08:30:52.142935+0000","last_peered":"2024-04-17T08:30:52.142935+0000","last_clean":"2024-04-17T08:30:52.142935+0000","last_became_active":"2024-04-17T08:30:51.140082+0000","last_became_peered":"2024-04-17T08:30:51.140082+0000","last_unstale":"2024-04-17T08:30:52.142935+0000","last_undegraded":"2024-04-17T08:30:52.142935+0000","last_fullsized":"2024-04-17T08:30:52.142935+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:44.787255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.163500+0000","last_change":"2024-04-17T08:30:52.157090+0000","last_active":"2024-04-17T08:30:52.163500+0000","last_peered":"2024-04-17T08:30:52.163500+0000","last_clean":"2024-04-17T08:30:52.163500+0000","last_became_active":"2024-04-17T08:30:52.156777+0000","last_became_peered":"2024-04-17T08:30:52.156777+0000","last_unstale":"2024-04-17T08:30:52.163500+0000","last_undegraded":"2024-04-17T08:30:52.163500+0000","last_fullsized":"2024-04-17T08:30:52.163500+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:24.431021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"181'3","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186868+0000","last_change":"2024-04-17T08:30:51.138889+0000","last_active":"2024-04-17T08:30:52.186868+0000","last_peered":"2024-04-17T08:30:52.186868+0000","last_clean":"2024-04-17T08:30:52.186868+0000","last_became_active":"2024-04-17T08:30:51.138425+0000","last_became_peered":"2024-04-17T08:30:51.138425+0000","last_unstale":"2024-04-17T08:30:52.186868+0000","last_undegraded":"2024-04-17T08:30:52.186868+0000","last_fullsized":"2024-04-17T08:30:52.186868+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"169'277","reported_seq":1073,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.153802+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:30:52.153802+0000","last_peered":"2024-04-17T08:30:52.153802+0000","last_clean":"2024-04-17T08:30:52.153802+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:30:52.153802+0000","last_undegraded":"2024-04-17T08:30:52.153802+0000","last_fullsized":"2024-04-17T08:30:52.153802+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":447,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":447,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47860,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323980,"statfs":{"total":96636764160,"available":96587755520,"internally_reserved":0,"allocated":2854912,"data_stored":1686754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543062,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37556,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334284,"statfs":{"total":96636764160,"available":96598306816,"internally_reserved":0,"allocated":2265088,"data_stored":1095880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2256896,"data_stored":1095756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55592,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316248,"statfs":{"total":96636764160,"available":96579837952,"internally_reserved":0,"allocated":2252800,"data_stored":1095734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63920,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307920,"statfs":{"total":96636764160,"available":96571310080,"internally_reserved":0,"allocated":2256896,"data_stored":1095824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543062,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47260,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324580,"statfs":{"total":96636764160,"available":96588369920,"internally_reserved":0,"allocated":2240512,"data_stored":1088931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543063,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1096316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60352,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311488,"statfs":{"total":96636764160,"available":96574963712,"internally_reserved":0,"allocated":2863104,"data_stored":1687748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:30:56.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:30:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.956+0000 7fc9d9800640 1 Processor -- start 2024-04-17T08:30:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.957+0000 7fc9d9800640 1 -- start start 2024-04-17T08:30:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9d415b570 0x7fc9d406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc9d406fce0 0x7fc9d406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406be70 0x7fc9d406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d415bc80 con 0x7fc9d415b570 2024-04-17T08:30:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d415bf40 con 0x7fc9d406fce0 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.958+0000 7fc9d9800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d415c220 con 0x7fc9d406be70 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9d415b570 0x7fc9d406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9d415b570 0x7fc9d406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35528/0 (socket says 172.21.15.67:35528) 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3457047586 learned_addr learned my addr 172.21.15.67:0/3457047586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d8e00640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406be70 0x7fc9d406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/3457047586 >> v2:172.21.15.187:3300/0 conn(0x7fc9d406fce0 0x7fc9d406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3457047586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406be70 msgr2=0x7fc9d406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406be70 0x7fc9d406ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3457047586 >> v2:172.21.15.187:3300/0 conn(0x7fc9d406fce0 msgr2=0x7fc9d406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3457047586 >> v2:172.21.15.187:3300/0 conn(0x7fc9d406fce0 0x7fc9d406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.959+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3457047586 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d415c500 con 0x7fc9d415b570 2024-04-17T08:30:56.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.960+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d415b570 0x7fc9d406f7a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4073040 tx=0x7fc9c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f39e6d5a813c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.961+0000 7fc9d3400640 1 -- 172.21.15.67:0/3457047586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c402e070 con 0x7fc9d415b570 2024-04-17T08:30:56.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.961+0000 7fc9d3400640 1 -- 172.21.15.67:0/3457047586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c4037070 con 0x7fc9d415b570 2024-04-17T08:30:56.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.961+0000 7fc9d3400640 1 -- 172.21.15.67:0/3457047586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4031a00 con 0x7fc9d415b570 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 -- 172.21.15.67:0/3457047586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d415b570 msgr2=0x7fc9d406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d415b570 0x7fc9d406f7a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4073040 tx=0x7fc9c402b880 comp rx=0 tx=0).stop 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 -- 172.21.15.67:0/3457047586 shutdown_connections 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406be70 0x7fc9d406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3457047586 >> v2:172.21.15.187:3300/0 conn(0x7fc9d406fce0 0x7fc9d406b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3457047586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d415b570 0x7fc9d406f7a0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 -- 172.21.15.67:0/3457047586 >> 172.21.15.67:0/3457047586 conn(0x7fc9d4067d10 msgr2=0x7fc9d40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:56.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 -- 172.21.15.67:0/3457047586 shutdown_connections 2024-04-17T08:30:56.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.962+0000 7fc9d9800640 1 -- 172.21.15.67:0/3457047586 wait complete. 2024-04-17T08:30:56.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.964+0000 7fc9d9800640 1 Processor -- start 2024-04-17T08:30:56.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.964+0000 7fc9d9800640 1 -- start start 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 0x7fc9d414f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9d415b570 0x7fc9d414ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d406c460 con 0x7fc9d406be70 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d406c2e0 con 0x7fc9d415b570 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d9800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d40b47b0 con 0x7fc9d406fce0 2024-04-17T08:30:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35530/0 (socket says 172.21.15.67:35530) 2024-04-17T08:30:56.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3713025017 learned_addr learned my addr 172.21.15.67:0/3713025017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.965+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 0x7fc9d414f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d8e00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9d415b570 0x7fc9d414ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 msgr2=0x7fc9d414f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 0x7fc9d414f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9d415b570 msgr2=0x7fc9d414ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9d415b570 0x7fc9d414ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 -- 172.21.15.67:0/3713025017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d41f0f80 con 0x7fc9d406be70 2024-04-17T08:30:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 0x7fc9d414f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d3e00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4000c00 tx=0x7fc9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4032670 con 0x7fc9d406be70 2024-04-17T08:30:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c4032850 con 0x7fc9d406be70 2024-04-17T08:30:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4032c40 con 0x7fc9d406be70 2024-04-17T08:30:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.966+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d414d560 con 0x7fc9d406be70 2024-04-17T08:30:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.967+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d414db20 con 0x7fc9d406be70 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.968+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d4068a10 con 0x7fc9d406be70 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.969+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fc9c404c060 con 0x7fc9d406be70 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.970+0000 7fc9d0c00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 0x7fc9b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.970+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7fc9c40ad8e0 con 0x7fc9d406be70 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.970+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 0x7fc9b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.971+0000 7fc9cbe00640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 0x7fc9b006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40b41c0 tx=0x7fc9bc008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:56.975+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc9c4040070 con 0x7fc9d406be70 2024-04-17T08:30:57.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.182+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 --> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc9d406f2f0 con 0x7fc9b006ca10 2024-04-17T08:30:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.202+0000 7fc9d0c00640 1 -- 172.21.15.67:0/3713025017 <== mgr.7465 v2:172.21.15.187:6832/659531354 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252427 (secure 0 0 0) 0x7fc9d406f2f0 con 0x7fc9b006ca10 2024-04-17T08:30:57.205 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:30:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.208+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 msgr2=0x7fc9b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.208+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 0x7fc9b006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40b41c0 tx=0x7fc9bc008040 comp rx=0 tx=0).stop 2024-04-17T08:30:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.208+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 msgr2=0x7fc9d41f0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.208+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4000c00 tx=0x7fc9c40046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 shutdown_connections 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fc9b006ca10 0x7fc9b006eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9d415b570 0x7fc9d414ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9d406fce0 0x7fc9d414f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 --2- 172.21.15.67:0/3713025017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9d406be70 0x7fc9d41f0a40 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 >> 172.21.15.67:0/3713025017 conn(0x7fc9d4067d10 msgr2=0x7fc9d40b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 shutdown_connections 2024-04-17T08:30:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.209+0000 7fc9d9800640 1 -- 172.21.15.67:0/3713025017 wait complete. 2024-04-17T08:30:57.229 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-17T08:30:56.975662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":491,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413996,"kb_used_data":18796,"kb_used_omap":12,"kb_used_meta":395059,"kb_avail":754560724,"statfs":{"total":773094113280,"available":772670181376,"internally_reserved":0,"allocated":19247104,"data_stored":9942943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":404540976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001911"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178156+0000","last_change":"2024-04-17T08:30:52.176535+0000","last_active":"2024-04-17T08:30:52.178156+0000","last_peered":"2024-04-17T08:30:52.178156+0000","last_clean":"2024-04-17T08:30:52.178156+0000","last_became_active":"2024-04-17T08:30:52.176142+0000","last_became_peered":"2024-04-17T08:30:52.176142+0000","last_unstale":"2024-04-17T08:30:52.178156+0000","last_undegraded":"2024-04-17T08:30:52.178156+0000","last_fullsized":"2024-04-17T08:30:52.178156+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:29:09.083064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165511+0000","last_change":"2024-04-17T08:30:52.160725+0000","last_active":"2024-04-17T08:30:52.165511+0000","last_peered":"2024-04-17T08:30:52.165511+0000","last_clean":"2024-04-17T08:30:52.165511+0000","last_became_active":"2024-04-17T08:30:52.160453+0000","last_became_peered":"2024-04-17T08:30:52.160453+0000","last_unstale":"2024-04-17T08:30:52.165511+0000","last_undegraded":"2024-04-17T08:30:52.165511+0000","last_fullsized":"2024-04-17T08:30:52.165511+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.912770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172933+0000","last_change":"2024-04-17T08:30:52.166841+0000","last_active":"2024-04-17T08:30:52.172933+0000","last_peered":"2024-04-17T08:30:52.172933+0000","last_clean":"2024-04-17T08:30:52.172933+0000","last_became_active":"2024-04-17T08:30:52.166419+0000","last_became_peered":"2024-04-17T08:30:52.166419+0000","last_unstale":"2024-04-17T08:30:52.172933+0000","last_undegraded":"2024-04-17T08:30:52.172933+0000","last_fullsized":"2024-04-17T08:30:52.172933+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:40.063359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.157436+0000","last_change":"2024-04-17T08:30:52.154568+0000","last_active":"2024-04-17T08:30:52.157436+0000","last_peered":"2024-04-17T08:30:52.157436+0000","last_clean":"2024-04-17T08:30:52.157436+0000","last_became_active":"2024-04-17T08:30:52.154278+0000","last_became_peered":"2024-04-17T08:30:52.154278+0000","last_unstale":"2024-04-17T08:30:52.157436+0000","last_undegraded":"2024-04-17T08:30:52.157436+0000","last_fullsized":"2024-04-17T08:30:52.157436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:35.540898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172879+0000","last_change":"2024-04-17T08:30:52.169806+0000","last_active":"2024-04-17T08:30:52.172879+0000","last_peered":"2024-04-17T08:30:52.172879+0000","last_clean":"2024-04-17T08:30:52.172879+0000","last_became_active":"2024-04-17T08:30:52.169306+0000","last_became_peered":"2024-04-17T08:30:52.169306+0000","last_unstale":"2024-04-17T08:30:52.172879+0000","last_undegraded":"2024-04-17T08:30:52.172879+0000","last_fullsized":"2024-04-17T08:30:52.172879+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:49.361756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166018+0000","last_change":"2024-04-17T08:30:52.162622+0000","last_active":"2024-04-17T08:30:52.166018+0000","last_peered":"2024-04-17T08:30:52.166018+0000","last_clean":"2024-04-17T08:30:52.166018+0000","last_became_active":"2024-04-17T08:30:52.162304+0000","last_became_peered":"2024-04-17T08:30:52.162304+0000","last_unstale":"2024-04-17T08:30:52.166018+0000","last_undegraded":"2024-04-17T08:30:52.166018+0000","last_fullsized":"2024-04-17T08:30:52.166018+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:03.337757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173571+0000","last_change":"2024-04-17T08:30:52.171688+0000","last_active":"2024-04-17T08:30:52.173571+0000","last_peered":"2024-04-17T08:30:52.173571+0000","last_clean":"2024-04-17T08:30:52.173571+0000","last_became_active":"2024-04-17T08:30:52.169274+0000","last_became_peered":"2024-04-17T08:30:52.169274+0000","last_unstale":"2024-04-17T08:30:52.173571+0000","last_undegraded":"2024-04-17T08:30:52.173571+0000","last_fullsized":"2024-04-17T08:30:52.173571+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:15.928453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175328+0000","last_change":"2024-04-17T08:30:52.169816+0000","last_active":"2024-04-17T08:30:52.175328+0000","last_peered":"2024-04-17T08:30:52.175328+0000","last_clean":"2024-04-17T08:30:52.175328+0000","last_became_active":"2024-04-17T08:30:52.169275+0000","last_became_peered":"2024-04-17T08:30:52.169275+0000","last_unstale":"2024-04-17T08:30:52.175328+0000","last_undegraded":"2024-04-17T08:30:52.175328+0000","last_fullsized":"2024-04-17T08:30:52.175328+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:49.498787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171496+0000","last_change":"2024-04-17T08:30:52.169241+0000","last_active":"2024-04-17T08:30:52.171496+0000","last_peered":"2024-04-17T08:30:52.171496+0000","last_clean":"2024-04-17T08:30:52.171496+0000","last_became_active":"2024-04-17T08:30:52.168892+0000","last_became_peered":"2024-04-17T08:30:52.168892+0000","last_unstale":"2024-04-17T08:30:52.171496+0000","last_undegraded":"2024-04-17T08:30:52.171496+0000","last_fullsized":"2024-04-17T08:30:52.171496+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:44.498403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177808+0000","last_change":"2024-04-17T08:30:52.175749+0000","last_active":"2024-04-17T08:30:52.177808+0000","last_peered":"2024-04-17T08:30:52.177808+0000","last_clean":"2024-04-17T08:30:52.177808+0000","last_became_active":"2024-04-17T08:30:52.174864+0000","last_became_peered":"2024-04-17T08:30:52.174864+0000","last_unstale":"2024-04-17T08:30:52.177808+0000","last_undegraded":"2024-04-17T08:30:52.177808+0000","last_fullsized":"2024-04-17T08:30:52.177808+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:41.111737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178193+0000","last_change":"2024-04-17T08:30:52.173534+0000","last_active":"2024-04-17T08:30:52.178193+0000","last_peered":"2024-04-17T08:30:52.178193+0000","last_clean":"2024-04-17T08:30:52.178193+0000","last_became_active":"2024-04-17T08:30:52.172387+0000","last_became_peered":"2024-04-17T08:30:52.172387+0000","last_unstale":"2024-04-17T08:30:52.178193+0000","last_undegraded":"2024-04-17T08:30:52.178193+0000","last_fullsized":"2024-04-17T08:30:52.178193+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:24.200418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.159801+0000","last_change":"2024-04-17T08:30:52.157740+0000","last_active":"2024-04-17T08:30:52.159801+0000","last_peered":"2024-04-17T08:30:52.159801+0000","last_clean":"2024-04-17T08:30:52.159801+0000","last_became_active":"2024-04-17T08:30:52.157419+0000","last_became_peered":"2024-04-17T08:30:52.157419+0000","last_unstale":"2024-04-17T08:30:52.159801+0000","last_undegraded":"2024-04-17T08:30:52.159801+0000","last_fullsized":"2024-04-17T08:30:52.159801+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:26:46.934348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171716+0000","last_change":"2024-04-17T08:30:52.169340+0000","last_active":"2024-04-17T08:30:52.171716+0000","last_peered":"2024-04-17T08:30:52.171716+0000","last_clean":"2024-04-17T08:30:52.171716+0000","last_became_active":"2024-04-17T08:30:52.167959+0000","last_became_peered":"2024-04-17T08:30:52.167959+0000","last_unstale":"2024-04-17T08:30:52.171716+0000","last_undegraded":"2024-04-17T08:30:52.171716+0000","last_fullsized":"2024-04-17T08:30:52.171716+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:35.307775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.174064+0000","last_change":"2024-04-17T08:30:52.170444+0000","last_active":"2024-04-17T08:30:52.174064+0000","last_peered":"2024-04-17T08:30:52.174064+0000","last_clean":"2024-04-17T08:30:52.174064+0000","last_became_active":"2024-04-17T08:30:52.170050+0000","last_became_peered":"2024-04-17T08:30:52.170050+0000","last_unstale":"2024-04-17T08:30:52.174064+0000","last_undegraded":"2024-04-17T08:30:52.174064+0000","last_fullsized":"2024-04-17T08:30:52.174064+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:51.504928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167553+0000","last_change":"2024-04-17T08:30:52.164866+0000","last_active":"2024-04-17T08:30:52.167553+0000","last_peered":"2024-04-17T08:30:52.167553+0000","last_clean":"2024-04-17T08:30:52.167553+0000","last_became_active":"2024-04-17T08:30:52.164568+0000","last_became_peered":"2024-04-17T08:30:52.164568+0000","last_unstale":"2024-04-17T08:30:52.167553+0000","last_undegraded":"2024-04-17T08:30:52.167553+0000","last_fullsized":"2024-04-17T08:30:52.167553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:58.754144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171895+0000","last_change":"2024-04-17T08:30:52.166413+0000","last_active":"2024-04-17T08:30:52.171895+0000","last_peered":"2024-04-17T08:30:52.171895+0000","last_clean":"2024-04-17T08:30:52.171895+0000","last_became_active":"2024-04-17T08:30:52.166105+0000","last_became_peered":"2024-04-17T08:30:52.166105+0000","last_unstale":"2024-04-17T08:30:52.171895+0000","last_undegraded":"2024-04-17T08:30:52.171895+0000","last_fullsized":"2024-04-17T08:30:52.171895+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:08:14.428867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177871+0000","last_change":"2024-04-17T08:30:52.175393+0000","last_active":"2024-04-17T08:30:52.177871+0000","last_peered":"2024-04-17T08:30:52.177871+0000","last_clean":"2024-04-17T08:30:52.177871+0000","last_became_active":"2024-04-17T08:30:52.174996+0000","last_became_peered":"2024-04-17T08:30:52.174996+0000","last_unstale":"2024-04-17T08:30:52.177871+0000","last_undegraded":"2024-04-17T08:30:52.177871+0000","last_fullsized":"2024-04-17T08:30:52.177871+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:45:52.336402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.168520+0000","last_change":"2024-04-17T08:30:52.165274+0000","last_active":"2024-04-17T08:30:52.168520+0000","last_peered":"2024-04-17T08:30:52.168520+0000","last_clean":"2024-04-17T08:30:52.168520+0000","last_became_active":"2024-04-17T08:30:52.164317+0000","last_became_peered":"2024-04-17T08:30:52.164317+0000","last_unstale":"2024-04-17T08:30:52.168520+0000","last_undegraded":"2024-04-17T08:30:52.168520+0000","last_fullsized":"2024-04-17T08:30:52.168520+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:35.976583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178211+0000","last_change":"2024-04-17T08:30:52.176010+0000","last_active":"2024-04-17T08:30:52.178211+0000","last_peered":"2024-04-17T08:30:52.178211+0000","last_clean":"2024-04-17T08:30:52.178211+0000","last_became_active":"2024-04-17T08:30:52.175736+0000","last_became_peered":"2024-04-17T08:30:52.175736+0000","last_unstale":"2024-04-17T08:30:52.178211+0000","last_undegraded":"2024-04-17T08:30:52.178211+0000","last_fullsized":"2024-04-17T08:30:52.178211+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:19.635104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171966+0000","last_change":"2024-04-17T08:30:52.167129+0000","last_active":"2024-04-17T08:30:52.171966+0000","last_peered":"2024-04-17T08:30:52.171966+0000","last_clean":"2024-04-17T08:30:52.171966+0000","last_became_active":"2024-04-17T08:30:52.166156+0000","last_became_peered":"2024-04-17T08:30:52.166156+0000","last_unstale":"2024-04-17T08:30:52.171966+0000","last_undegraded":"2024-04-17T08:30:52.171966+0000","last_fullsized":"2024-04-17T08:30:52.171966+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:27.321036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179766+0000","last_change":"2024-04-17T08:30:52.176742+0000","last_active":"2024-04-17T08:30:52.179766+0000","last_peered":"2024-04-17T08:30:52.179766+0000","last_clean":"2024-04-17T08:30:52.179766+0000","last_became_active":"2024-04-17T08:30:52.176438+0000","last_became_peered":"2024-04-17T08:30:52.176438+0000","last_unstale":"2024-04-17T08:30:52.179766+0000","last_undegraded":"2024-04-17T08:30:52.179766+0000","last_fullsized":"2024-04-17T08:30:52.179766+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:47:35.642668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158622+0000","last_change":"2024-04-17T08:30:52.156525+0000","last_active":"2024-04-17T08:30:52.158622+0000","last_peered":"2024-04-17T08:30:52.158622+0000","last_clean":"2024-04-17T08:30:52.158622+0000","last_became_active":"2024-04-17T08:30:52.151328+0000","last_became_peered":"2024-04-17T08:30:52.151328+0000","last_unstale":"2024-04-17T08:30:52.158622+0000","last_undegraded":"2024-04-17T08:30:52.158622+0000","last_fullsized":"2024-04-17T08:30:52.158622+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:12.798025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170902+0000","last_change":"2024-04-17T08:30:52.165119+0000","last_active":"2024-04-17T08:30:52.170902+0000","last_peered":"2024-04-17T08:30:52.170902+0000","last_clean":"2024-04-17T08:30:52.170902+0000","last_became_active":"2024-04-17T08:30:52.164802+0000","last_became_peered":"2024-04-17T08:30:52.164802+0000","last_unstale":"2024-04-17T08:30:52.170902+0000","last_undegraded":"2024-04-17T08:30:52.170902+0000","last_fullsized":"2024-04-17T08:30:52.170902+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:50.778911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173029+0000","last_change":"2024-04-17T08:30:52.170419+0000","last_active":"2024-04-17T08:30:52.173029+0000","last_peered":"2024-04-17T08:30:52.173029+0000","last_clean":"2024-04-17T08:30:52.173029+0000","last_became_active":"2024-04-17T08:30:52.170133+0000","last_became_peered":"2024-04-17T08:30:52.170133+0000","last_unstale":"2024-04-17T08:30:52.173029+0000","last_undegraded":"2024-04-17T08:30:52.173029+0000","last_fullsized":"2024-04-17T08:30:52.173029+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:00:05.058041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179632+0000","last_change":"2024-04-17T08:30:52.174805+0000","last_active":"2024-04-17T08:30:52.179632+0000","last_peered":"2024-04-17T08:30:52.179632+0000","last_clean":"2024-04-17T08:30:52.179632+0000","last_became_active":"2024-04-17T08:30:52.174504+0000","last_became_peered":"2024-04-17T08:30:52.174504+0000","last_unstale":"2024-04-17T08:30:52.179632+0000","last_undegraded":"2024-04-17T08:30:52.179632+0000","last_fullsized":"2024-04-17T08:30:52.179632+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:16.951240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178091+0000","last_change":"2024-04-17T08:30:52.174969+0000","last_active":"2024-04-17T08:30:52.178091+0000","last_peered":"2024-04-17T08:30:52.178091+0000","last_clean":"2024-04-17T08:30:52.178091+0000","last_became_active":"2024-04-17T08:30:52.174684+0000","last_became_peered":"2024-04-17T08:30:52.174684+0000","last_unstale":"2024-04-17T08:30:52.178091+0000","last_undegraded":"2024-04-17T08:30:52.178091+0000","last_fullsized":"2024-04-17T08:30:52.178091+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:08.905330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160851+0000","last_change":"2024-04-17T08:30:52.156336+0000","last_active":"2024-04-17T08:30:52.160851+0000","last_peered":"2024-04-17T08:30:52.160851+0000","last_clean":"2024-04-17T08:30:52.160851+0000","last_became_active":"2024-04-17T08:30:52.155626+0000","last_became_peered":"2024-04-17T08:30:52.155626+0000","last_unstale":"2024-04-17T08:30:52.160851+0000","last_undegraded":"2024-04-17T08:30:52.160851+0000","last_fullsized":"2024-04-17T08:30:52.160851+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:21.631248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166022+0000","last_change":"2024-04-17T08:30:52.163152+0000","last_active":"2024-04-17T08:30:52.166022+0000","last_peered":"2024-04-17T08:30:52.166022+0000","last_clean":"2024-04-17T08:30:52.166022+0000","last_became_active":"2024-04-17T08:30:52.159167+0000","last_became_peered":"2024-04-17T08:30:52.159167+0000","last_unstale":"2024-04-17T08:30:52.166022+0000","last_undegraded":"2024-04-17T08:30:52.166022+0000","last_fullsized":"2024-04-17T08:30:52.166022+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:20:01.467968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175321+0000","last_change":"2024-04-17T08:30:52.171195+0000","last_active":"2024-04-17T08:30:52.175321+0000","last_peered":"2024-04-17T08:30:52.175321+0000","last_clean":"2024-04-17T08:30:52.175321+0000","last_became_active":"2024-04-17T08:30:52.166955+0000","last_became_peered":"2024-04-17T08:30:52.166955+0000","last_unstale":"2024-04-17T08:30:52.175321+0000","last_undegraded":"2024-04-17T08:30:52.175321+0000","last_fullsized":"2024-04-17T08:30:52.175321+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:53.772266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170257+0000","last_change":"2024-04-17T08:30:52.165446+0000","last_active":"2024-04-17T08:30:52.170257+0000","last_peered":"2024-04-17T08:30:52.170257+0000","last_clean":"2024-04-17T08:30:52.170257+0000","last_became_active":"2024-04-17T08:30:52.164177+0000","last_became_peered":"2024-04-17T08:30:52.164177+0000","last_unstale":"2024-04-17T08:30:52.170257+0000","last_undegraded":"2024-04-17T08:30:52.170257+0000","last_fullsized":"2024-04-17T08:30:52.170257+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:11.065845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166232+0000","last_change":"2024-04-17T08:30:52.162598+0000","last_active":"2024-04-17T08:30:52.166232+0000","last_peered":"2024-04-17T08:30:52.166232+0000","last_clean":"2024-04-17T08:30:52.166232+0000","last_became_active":"2024-04-17T08:30:52.162302+0000","last_became_peered":"2024-04-17T08:30:52.162302+0000","last_unstale":"2024-04-17T08:30:52.166232+0000","last_undegraded":"2024-04-17T08:30:52.166232+0000","last_fullsized":"2024-04-17T08:30:52.166232+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:17.734042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160413+0000","last_change":"2024-04-17T08:30:52.153683+0000","last_active":"2024-04-17T08:30:52.160413+0000","last_peered":"2024-04-17T08:30:52.160413+0000","last_clean":"2024-04-17T08:30:52.160413+0000","last_became_active":"2024-04-17T08:30:52.153251+0000","last_became_peered":"2024-04-17T08:30:52.153251+0000","last_unstale":"2024-04-17T08:30:52.160413+0000","last_undegraded":"2024-04-17T08:30:52.160413+0000","last_fullsized":"2024-04-17T08:30:52.160413+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:43.322648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.179691+0000","last_change":"2024-04-17T08:30:52.176624+0000","last_active":"2024-04-17T08:30:52.179691+0000","last_peered":"2024-04-17T08:30:52.179691+0000","last_clean":"2024-04-17T08:30:52.179691+0000","last_became_active":"2024-04-17T08:30:52.175772+0000","last_became_peered":"2024-04-17T08:30:52.175772+0000","last_unstale":"2024-04-17T08:30:52.179691+0000","last_undegraded":"2024-04-17T08:30:52.179691+0000","last_fullsized":"2024-04-17T08:30:52.179691+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:26.416829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"181'4","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186629+0000","last_change":"2024-04-17T08:30:51.143969+0000","last_active":"2024-04-17T08:30:52.186629+0000","last_peered":"2024-04-17T08:30:52.186629+0000","last_clean":"2024-04-17T08:30:52.186629+0000","last_became_active":"2024-04-17T08:30:51.143210+0000","last_became_peered":"2024-04-17T08:30:51.143210+0000","last_unstale":"2024-04-17T08:30:52.186629+0000","last_undegraded":"2024-04-17T08:30:52.186629+0000","last_fullsized":"2024-04-17T08:30:52.186629+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:06.446696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165031+0000","last_change":"2024-04-17T08:30:52.160627+0000","last_active":"2024-04-17T08:30:52.165031+0000","last_peered":"2024-04-17T08:30:52.165031+0000","last_clean":"2024-04-17T08:30:52.165031+0000","last_became_active":"2024-04-17T08:30:52.160343+0000","last_became_peered":"2024-04-17T08:30:52.160343+0000","last_unstale":"2024-04-17T08:30:52.165031+0000","last_undegraded":"2024-04-17T08:30:52.165031+0000","last_fullsized":"2024-04-17T08:30:52.165031+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:08:14.121360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.184606+0000","last_change":"2024-04-17T08:30:51.143415+0000","last_active":"2024-04-17T08:30:52.184606+0000","last_peered":"2024-04-17T08:30:52.184606+0000","last_clean":"2024-04-17T08:30:52.184606+0000","last_became_active":"2024-04-17T08:30:51.142223+0000","last_became_peered":"2024-04-17T08:30:51.142223+0000","last_unstale":"2024-04-17T08:30:52.184606+0000","last_undegraded":"2024-04-17T08:30:52.184606+0000","last_fullsized":"2024-04-17T08:30:52.184606+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:28.985117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177185+0000","last_change":"2024-04-17T08:30:52.172416+0000","last_active":"2024-04-17T08:30:52.177185+0000","last_peered":"2024-04-17T08:30:52.177185+0000","last_clean":"2024-04-17T08:30:52.177185+0000","last_became_active":"2024-04-17T08:30:52.170458+0000","last_became_peered":"2024-04-17T08:30:52.170458+0000","last_unstale":"2024-04-17T08:30:52.177185+0000","last_undegraded":"2024-04-17T08:30:52.177185+0000","last_fullsized":"2024-04-17T08:30:52.177185+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:45.550828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186011+0000","last_change":"2024-04-17T08:30:51.131325+0000","last_active":"2024-04-17T08:30:52.186011+0000","last_peered":"2024-04-17T08:30:52.186011+0000","last_clean":"2024-04-17T08:30:52.186011+0000","last_became_active":"2024-04-17T08:30:51.131009+0000","last_became_peered":"2024-04-17T08:30:51.131009+0000","last_unstale":"2024-04-17T08:30:52.186011+0000","last_undegraded":"2024-04-17T08:30:52.186011+0000","last_fullsized":"2024-04-17T08:30:52.186011+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:16.023338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173003+0000","last_change":"2024-04-17T08:30:52.168619+0000","last_active":"2024-04-17T08:30:52.173003+0000","last_peered":"2024-04-17T08:30:52.173003+0000","last_clean":"2024-04-17T08:30:52.173003+0000","last_became_active":"2024-04-17T08:30:52.168309+0000","last_became_peered":"2024-04-17T08:30:52.168309+0000","last_unstale":"2024-04-17T08:30:52.173003+0000","last_undegraded":"2024-04-17T08:30:52.173003+0000","last_fullsized":"2024-04-17T08:30:52.173003+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:36.171565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186470+0000","last_change":"2024-04-17T08:30:51.138295+0000","last_active":"2024-04-17T08:30:52.186470+0000","last_peered":"2024-04-17T08:30:52.186470+0000","last_clean":"2024-04-17T08:30:52.186470+0000","last_became_active":"2024-04-17T08:30:51.137747+0000","last_became_peered":"2024-04-17T08:30:51.137747+0000","last_unstale":"2024-04-17T08:30:52.186470+0000","last_undegraded":"2024-04-17T08:30:52.186470+0000","last_fullsized":"2024-04-17T08:30:52.186470+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:27.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166420+0000","last_change":"2024-04-17T08:30:52.163852+0000","last_active":"2024-04-17T08:30:52.166420+0000","last_peered":"2024-04-17T08:30:52.166420+0000","last_clean":"2024-04-17T08:30:52.166420+0000","last_became_active":"2024-04-17T08:30:52.162957+0000","last_became_peered":"2024-04-17T08:30:52.162957+0000","last_unstale":"2024-04-17T08:30:52.166420+0000","last_undegraded":"2024-04-17T08:30:52.166420+0000","last_fullsized":"2024-04-17T08:30:52.166420+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:00.962539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.181381+0000","last_change":"2024-04-17T08:30:51.140186+0000","last_active":"2024-04-17T08:30:52.181381+0000","last_peered":"2024-04-17T08:30:52.181381+0000","last_clean":"2024-04-17T08:30:52.181381+0000","last_became_active":"2024-04-17T08:30:51.139852+0000","last_became_peered":"2024-04-17T08:30:51.139852+0000","last_unstale":"2024-04-17T08:30:52.181381+0000","last_undegraded":"2024-04-17T08:30:52.181381+0000","last_fullsized":"2024-04-17T08:30:52.181381+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:01.210967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.173089+0000","last_change":"2024-04-17T08:30:52.168030+0000","last_active":"2024-04-17T08:30:52.173089+0000","last_peered":"2024-04-17T08:30:52.173089+0000","last_clean":"2024-04-17T08:30:52.173089+0000","last_became_active":"2024-04-17T08:30:52.167189+0000","last_became_peered":"2024-04-17T08:30:52.167189+0000","last_unstale":"2024-04-17T08:30:52.173089+0000","last_undegraded":"2024-04-17T08:30:52.173089+0000","last_fullsized":"2024-04-17T08:30:52.173089+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:16.884795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.183740+0000","last_change":"2024-04-17T08:30:51.142844+0000","last_active":"2024-04-17T08:30:52.183740+0000","last_peered":"2024-04-17T08:30:52.183740+0000","last_clean":"2024-04-17T08:30:52.183740+0000","last_became_active":"2024-04-17T08:30:51.142524+0000","last_became_peered":"2024-04-17T08:30:51.142524+0000","last_unstale":"2024-04-17T08:30:52.183740+0000","last_undegraded":"2024-04-17T08:30:52.183740+0000","last_fullsized":"2024-04-17T08:30:52.183740+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:03.644316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.159396+0000","last_change":"2024-04-17T08:30:52.151491+0000","last_active":"2024-04-17T08:30:52.159396+0000","last_peered":"2024-04-17T08:30:52.159396+0000","last_clean":"2024-04-17T08:30:52.159396+0000","last_became_active":"2024-04-17T08:30:52.151190+0000","last_became_peered":"2024-04-17T08:30:52.151190+0000","last_unstale":"2024-04-17T08:30:52.159396+0000","last_undegraded":"2024-04-17T08:30:52.159396+0000","last_fullsized":"2024-04-17T08:30:52.159396+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:00.296364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.131428+0000","last_change":"2024-04-17T08:30:51.134498+0000","last_active":"2024-04-17T08:30:52.131428+0000","last_peered":"2024-04-17T08:30:52.131428+0000","last_clean":"2024-04-17T08:30:52.131428+0000","last_became_active":"2024-04-17T08:30:51.134207+0000","last_became_peered":"2024-04-17T08:30:51.134207+0000","last_unstale":"2024-04-17T08:30:52.131428+0000","last_undegraded":"2024-04-17T08:30:52.131428+0000","last_fullsized":"2024-04-17T08:30:52.131428+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:07.771377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170427+0000","last_change":"2024-04-17T08:30:52.166488+0000","last_active":"2024-04-17T08:30:52.170427+0000","last_peered":"2024-04-17T08:30:52.170427+0000","last_clean":"2024-04-17T08:30:52.170427+0000","last_became_active":"2024-04-17T08:30:52.165289+0000","last_became_peered":"2024-04-17T08:30:52.165289+0000","last_unstale":"2024-04-17T08:30:52.170427+0000","last_undegraded":"2024-04-17T08:30:52.170427+0000","last_fullsized":"2024-04-17T08:30:52.170427+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:14.995324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.137948+0000","last_change":"2024-04-17T08:30:51.136258+0000","last_active":"2024-04-17T08:30:52.137948+0000","last_peered":"2024-04-17T08:30:52.137948+0000","last_clean":"2024-04-17T08:30:52.137948+0000","last_became_active":"2024-04-17T08:30:51.135949+0000","last_became_peered":"2024-04-17T08:30:51.135949+0000","last_unstale":"2024-04-17T08:30:52.137948+0000","last_undegraded":"2024-04-17T08:30:52.137948+0000","last_fullsized":"2024-04-17T08:30:52.137948+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:56:25.132852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.169244+0000","last_change":"2024-04-17T08:30:52.160217+0000","last_active":"2024-04-17T08:30:52.169244+0000","last_peered":"2024-04-17T08:30:52.169244+0000","last_clean":"2024-04-17T08:30:52.169244+0000","last_became_active":"2024-04-17T08:30:52.159912+0000","last_became_peered":"2024-04-17T08:30:52.159912+0000","last_unstale":"2024-04-17T08:30:52.169244+0000","last_undegraded":"2024-04-17T08:30:52.169244+0000","last_fullsized":"2024-04-17T08:30:52.169244+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:39:08.638199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.150576+0000","last_change":"2024-04-17T08:30:51.143277+0000","last_active":"2024-04-17T08:30:52.150576+0000","last_peered":"2024-04-17T08:30:52.150576+0000","last_clean":"2024-04-17T08:30:52.150576+0000","last_became_active":"2024-04-17T08:30:51.142069+0000","last_became_peered":"2024-04-17T08:30:51.142069+0000","last_unstale":"2024-04-17T08:30:52.150576+0000","last_undegraded":"2024-04-17T08:30:52.150576+0000","last_fullsized":"2024-04-17T08:30:52.150576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:48.094315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177883+0000","last_change":"2024-04-17T08:30:52.175579+0000","last_active":"2024-04-17T08:30:52.177883+0000","last_peered":"2024-04-17T08:30:52.177883+0000","last_clean":"2024-04-17T08:30:52.177883+0000","last_became_active":"2024-04-17T08:30:52.174298+0000","last_became_peered":"2024-04-17T08:30:52.174298+0000","last_unstale":"2024-04-17T08:30:52.177883+0000","last_undegraded":"2024-04-17T08:30:52.177883+0000","last_fullsized":"2024-04-17T08:30:52.177883+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:34.355725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.162779+0000","last_change":"2024-04-17T08:30:51.142822+0000","last_active":"2024-04-17T08:30:52.162779+0000","last_peered":"2024-04-17T08:30:52.162779+0000","last_clean":"2024-04-17T08:30:52.162779+0000","last_became_active":"2024-04-17T08:30:51.142316+0000","last_became_peered":"2024-04-17T08:30:51.142316+0000","last_unstale":"2024-04-17T08:30:52.162779+0000","last_undegraded":"2024-04-17T08:30:52.162779+0000","last_fullsized":"2024-04-17T08:30:52.162779+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:55.787013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.160876+0000","last_change":"2024-04-17T08:30:52.156578+0000","last_active":"2024-04-17T08:30:52.160876+0000","last_peered":"2024-04-17T08:30:52.160876+0000","last_clean":"2024-04-17T08:30:52.160876+0000","last_became_active":"2024-04-17T08:30:52.156025+0000","last_became_peered":"2024-04-17T08:30:52.156025+0000","last_unstale":"2024-04-17T08:30:52.160876+0000","last_undegraded":"2024-04-17T08:30:52.160876+0000","last_fullsized":"2024-04-17T08:30:52.160876+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:38.117269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"181'6","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.188758+0000","last_change":"2024-04-17T08:30:51.130242+0000","last_active":"2024-04-17T08:30:52.188758+0000","last_peered":"2024-04-17T08:30:52.188758+0000","last_clean":"2024-04-17T08:30:52.188758+0000","last_became_active":"2024-04-17T08:30:51.129868+0000","last_became_peered":"2024-04-17T08:30:51.129868+0000","last_unstale":"2024-04-17T08:30:52.188758+0000","last_undegraded":"2024-04-17T08:30:52.188758+0000","last_fullsized":"2024-04-17T08:30:52.188758+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:01.224299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175320+0000","last_change":"2024-04-17T08:30:52.169255+0000","last_active":"2024-04-17T08:30:52.175320+0000","last_peered":"2024-04-17T08:30:52.175320+0000","last_clean":"2024-04-17T08:30:52.175320+0000","last_became_active":"2024-04-17T08:30:52.168033+0000","last_became_peered":"2024-04-17T08:30:52.168033+0000","last_unstale":"2024-04-17T08:30:52.175320+0000","last_undegraded":"2024-04-17T08:30:52.175320+0000","last_fullsized":"2024-04-17T08:30:52.175320+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:54.749428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186033+0000","last_change":"2024-04-17T08:30:51.124880+0000","last_active":"2024-04-17T08:30:52.186033+0000","last_peered":"2024-04-17T08:30:52.186033+0000","last_clean":"2024-04-17T08:30:52.186033+0000","last_became_active":"2024-04-17T08:30:51.122582+0000","last_became_peered":"2024-04-17T08:30:51.122582+0000","last_unstale":"2024-04-17T08:30:52.186033+0000","last_undegraded":"2024-04-17T08:30:52.186033+0000","last_fullsized":"2024-04-17T08:30:52.186033+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:32:20.506153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166254+0000","last_change":"2024-04-17T08:30:52.163388+0000","last_active":"2024-04-17T08:30:52.166254+0000","last_peered":"2024-04-17T08:30:52.166254+0000","last_clean":"2024-04-17T08:30:52.166254+0000","last_became_active":"2024-04-17T08:30:52.163073+0000","last_became_peered":"2024-04-17T08:30:52.163073+0000","last_unstale":"2024-04-17T08:30:52.166254+0000","last_undegraded":"2024-04-17T08:30:52.166254+0000","last_fullsized":"2024-04-17T08:30:52.166254+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:42:59.104395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.155522+0000","last_change":"2024-04-17T08:30:51.139507+0000","last_active":"2024-04-17T08:30:52.155522+0000","last_peered":"2024-04-17T08:30:52.155522+0000","last_clean":"2024-04-17T08:30:52.155522+0000","last_became_active":"2024-04-17T08:30:51.139184+0000","last_became_peered":"2024-04-17T08:30:51.139184+0000","last_unstale":"2024-04-17T08:30:52.155522+0000","last_undegraded":"2024-04-17T08:30:52.155522+0000","last_fullsized":"2024-04-17T08:30:52.155522+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:13.432503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172690+0000","last_change":"2024-04-17T08:30:52.169710+0000","last_active":"2024-04-17T08:30:52.172690+0000","last_peered":"2024-04-17T08:30:52.172690+0000","last_clean":"2024-04-17T08:30:52.172690+0000","last_became_active":"2024-04-17T08:30:52.168312+0000","last_became_peered":"2024-04-17T08:30:52.168312+0000","last_unstale":"2024-04-17T08:30:52.172690+0000","last_undegraded":"2024-04-17T08:30:52.172690+0000","last_fullsized":"2024-04-17T08:30:52.172690+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:55.339072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.149209+0000","last_change":"2024-04-17T08:30:51.139897+0000","last_active":"2024-04-17T08:30:52.149209+0000","last_peered":"2024-04-17T08:30:52.149209+0000","last_clean":"2024-04-17T08:30:52.149209+0000","last_became_active":"2024-04-17T08:30:51.138510+0000","last_became_peered":"2024-04-17T08:30:51.138510+0000","last_unstale":"2024-04-17T08:30:52.149209+0000","last_undegraded":"2024-04-17T08:30:52.149209+0000","last_fullsized":"2024-04-17T08:30:52.149209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:17.007012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.165073+0000","last_change":"2024-04-17T08:30:52.162443+0000","last_active":"2024-04-17T08:30:52.165073+0000","last_peered":"2024-04-17T08:30:52.165073+0000","last_clean":"2024-04-17T08:30:52.165073+0000","last_became_active":"2024-04-17T08:30:52.162098+0000","last_became_peered":"2024-04-17T08:30:52.162098+0000","last_unstale":"2024-04-17T08:30:52.165073+0000","last_undegraded":"2024-04-17T08:30:52.165073+0000","last_fullsized":"2024-04-17T08:30:52.165073+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:27:05.273119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"181'8","reported_seq":20,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.183879+0000","last_change":"2024-04-17T08:30:51.137868+0000","last_active":"2024-04-17T08:30:52.183879+0000","last_peered":"2024-04-17T08:30:52.183879+0000","last_clean":"2024-04-17T08:30:52.183879+0000","last_became_active":"2024-04-17T08:30:51.136385+0000","last_became_peered":"2024-04-17T08:30:51.136385+0000","last_unstale":"2024-04-17T08:30:52.183879+0000","last_undegraded":"2024-04-17T08:30:52.183879+0000","last_fullsized":"2024-04-17T08:30:52.183879+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:28.940365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.177787+0000","last_change":"2024-04-17T08:30:52.175472+0000","last_active":"2024-04-17T08:30:52.177787+0000","last_peered":"2024-04-17T08:30:52.177787+0000","last_clean":"2024-04-17T08:30:52.177787+0000","last_became_active":"2024-04-17T08:30:52.174354+0000","last_became_peered":"2024-04-17T08:30:52.174354+0000","last_unstale":"2024-04-17T08:30:52.177787+0000","last_undegraded":"2024-04-17T08:30:52.177787+0000","last_fullsized":"2024-04-17T08:30:52.177787+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:47.591301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.184632+0000","last_change":"2024-04-17T08:30:51.140559+0000","last_active":"2024-04-17T08:30:52.184632+0000","last_peered":"2024-04-17T08:30:52.184632+0000","last_clean":"2024-04-17T08:30:52.184632+0000","last_became_active":"2024-04-17T08:30:51.140275+0000","last_became_peered":"2024-04-17T08:30:51.140275+0000","last_unstale":"2024-04-17T08:30:52.184632+0000","last_undegraded":"2024-04-17T08:30:52.184632+0000","last_fullsized":"2024-04-17T08:30:52.184632+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:53.733492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171821+0000","last_change":"2024-04-17T08:30:52.164139+0000","last_active":"2024-04-17T08:30:52.171821+0000","last_peered":"2024-04-17T08:30:52.171821+0000","last_clean":"2024-04-17T08:30:52.171821+0000","last_became_active":"2024-04-17T08:30:52.162245+0000","last_became_peered":"2024-04-17T08:30:52.162245+0000","last_unstale":"2024-04-17T08:30:52.171821+0000","last_undegraded":"2024-04-17T08:30:52.171821+0000","last_fullsized":"2024-04-17T08:30:52.171821+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:19:37.947112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186352+0000","last_change":"2024-04-17T08:30:51.136065+0000","last_active":"2024-04-17T08:30:52.186352+0000","last_peered":"2024-04-17T08:30:52.186352+0000","last_clean":"2024-04-17T08:30:52.186352+0000","last_became_active":"2024-04-17T08:30:51.135734+0000","last_became_peered":"2024-04-17T08:30:51.135734+0000","last_unstale":"2024-04-17T08:30:52.186352+0000","last_undegraded":"2024-04-17T08:30:52.186352+0000","last_fullsized":"2024-04-17T08:30:52.186352+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:30.534866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.174920+0000","last_change":"2024-04-17T08:30:52.167748+0000","last_active":"2024-04-17T08:30:52.174920+0000","last_peered":"2024-04-17T08:30:52.174920+0000","last_clean":"2024-04-17T08:30:52.174920+0000","last_became_active":"2024-04-17T08:30:52.166360+0000","last_became_peered":"2024-04-17T08:30:52.166360+0000","last_unstale":"2024-04-17T08:30:52.174920+0000","last_undegraded":"2024-04-17T08:30:52.174920+0000","last_fullsized":"2024-04-17T08:30:52.174920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.660252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"181'1","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.180964+0000","last_change":"2024-04-17T08:30:51.134068+0000","last_active":"2024-04-17T08:30:52.180964+0000","last_peered":"2024-04-17T08:30:52.180964+0000","last_clean":"2024-04-17T08:30:52.180964+0000","last_became_active":"2024-04-17T08:30:51.133169+0000","last_became_peered":"2024-04-17T08:30:51.133169+0000","last_unstale":"2024-04-17T08:30:52.180964+0000","last_undegraded":"2024-04-17T08:30:52.180964+0000","last_fullsized":"2024-04-17T08:30:52.180964+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:11:38.141287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.178171+0000","last_change":"2024-04-17T08:30:52.173357+0000","last_active":"2024-04-17T08:30:52.178171+0000","last_peered":"2024-04-17T08:30:52.178171+0000","last_clean":"2024-04-17T08:30:52.178171+0000","last_became_active":"2024-04-17T08:30:52.173047+0000","last_became_peered":"2024-04-17T08:30:52.173047+0000","last_unstale":"2024-04-17T08:30:52.178171+0000","last_undegraded":"2024-04-17T08:30:52.178171+0000","last_fullsized":"2024-04-17T08:30:52.178171+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:05.455922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.151283+0000","last_change":"2024-04-17T08:30:51.140780+0000","last_active":"2024-04-17T08:30:52.151283+0000","last_peered":"2024-04-17T08:30:52.151283+0000","last_clean":"2024-04-17T08:30:52.151283+0000","last_became_active":"2024-04-17T08:30:51.140513+0000","last_became_peered":"2024-04-17T08:30:51.140513+0000","last_unstale":"2024-04-17T08:30:52.151283+0000","last_undegraded":"2024-04-17T08:30:52.151283+0000","last_fullsized":"2024-04-17T08:30:52.151283+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:02.985822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158643+0000","last_change":"2024-04-17T08:30:52.156813+0000","last_active":"2024-04-17T08:30:52.158643+0000","last_peered":"2024-04-17T08:30:52.158643+0000","last_clean":"2024-04-17T08:30:52.158643+0000","last_became_active":"2024-04-17T08:30:52.153109+0000","last_became_peered":"2024-04-17T08:30:52.153109+0000","last_unstale":"2024-04-17T08:30:52.158643+0000","last_undegraded":"2024-04-17T08:30:52.158643+0000","last_fullsized":"2024-04-17T08:30:52.158643+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:37.004223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.185557+0000","last_change":"2024-04-17T08:30:51.125597+0000","last_active":"2024-04-17T08:30:52.185557+0000","last_peered":"2024-04-17T08:30:52.185557+0000","last_clean":"2024-04-17T08:30:52.185557+0000","last_became_active":"2024-04-17T08:30:51.124747+0000","last_became_peered":"2024-04-17T08:30:51.124747+0000","last_unstale":"2024-04-17T08:30:52.185557+0000","last_undegraded":"2024-04-17T08:30:52.185557+0000","last_fullsized":"2024-04-17T08:30:52.185557+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:31:27.197036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.164281+0000","last_change":"2024-04-17T08:30:52.155554+0000","last_active":"2024-04-17T08:30:52.164281+0000","last_peered":"2024-04-17T08:30:52.164281+0000","last_clean":"2024-04-17T08:30:52.164281+0000","last_became_active":"2024-04-17T08:30:52.155149+0000","last_became_peered":"2024-04-17T08:30:52.155149+0000","last_unstale":"2024-04-17T08:30:52.164281+0000","last_undegraded":"2024-04-17T08:30:52.164281+0000","last_fullsized":"2024-04-17T08:30:52.164281+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:50.472795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.154968+0000","last_change":"2024-04-17T08:30:51.143092+0000","last_active":"2024-04-17T08:30:52.154968+0000","last_peered":"2024-04-17T08:30:52.154968+0000","last_clean":"2024-04-17T08:30:52.154968+0000","last_became_active":"2024-04-17T08:30:51.142666+0000","last_became_peered":"2024-04-17T08:30:51.142666+0000","last_unstale":"2024-04-17T08:30:52.154968+0000","last_undegraded":"2024-04-17T08:30:52.154968+0000","last_fullsized":"2024-04-17T08:30:52.154968+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:56.488726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166919+0000","last_change":"2024-04-17T08:30:52.163125+0000","last_active":"2024-04-17T08:30:52.166919+0000","last_peered":"2024-04-17T08:30:52.166919+0000","last_clean":"2024-04-17T08:30:52.166919+0000","last_became_active":"2024-04-17T08:30:52.162834+0000","last_became_peered":"2024-04-17T08:30:52.162834+0000","last_unstale":"2024-04-17T08:30:52.166919+0000","last_undegraded":"2024-04-17T08:30:52.166919+0000","last_fullsized":"2024-04-17T08:30:52.166919+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:41:39.786033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.151826+0000","last_change":"2024-04-17T08:30:51.140770+0000","last_active":"2024-04-17T08:30:52.151826+0000","last_peered":"2024-04-17T08:30:52.151826+0000","last_clean":"2024-04-17T08:30:52.151826+0000","last_became_active":"2024-04-17T08:30:51.140454+0000","last_became_peered":"2024-04-17T08:30:51.140454+0000","last_unstale":"2024-04-17T08:30:52.151826+0000","last_undegraded":"2024-04-17T08:30:52.151826+0000","last_fullsized":"2024-04-17T08:30:52.151826+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:16:19.278306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.168533+0000","last_change":"2024-04-17T08:30:52.158816+0000","last_active":"2024-04-17T08:30:52.168533+0000","last_peered":"2024-04-17T08:30:52.168533+0000","last_clean":"2024-04-17T08:30:52.168533+0000","last_became_active":"2024-04-17T08:30:52.155384+0000","last_became_peered":"2024-04-17T08:30:52.155384+0000","last_unstale":"2024-04-17T08:30:52.168533+0000","last_undegraded":"2024-04-17T08:30:52.168533+0000","last_fullsized":"2024-04-17T08:30:52.168533+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:47.375416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.152252+0000","last_change":"2024-04-17T08:30:51.138858+0000","last_active":"2024-04-17T08:30:52.152252+0000","last_peered":"2024-04-17T08:30:52.152252+0000","last_clean":"2024-04-17T08:30:52.152252+0000","last_became_active":"2024-04-17T08:30:51.138495+0000","last_became_peered":"2024-04-17T08:30:51.138495+0000","last_unstale":"2024-04-17T08:30:52.152252+0000","last_undegraded":"2024-04-17T08:30:52.152252+0000","last_fullsized":"2024-04-17T08:30:52.152252+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:40.512510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.171344+0000","last_change":"2024-04-17T08:30:52.169075+0000","last_active":"2024-04-17T08:30:52.171344+0000","last_peered":"2024-04-17T08:30:52.171344+0000","last_clean":"2024-04-17T08:30:52.171344+0000","last_became_active":"2024-04-17T08:30:52.168731+0000","last_became_peered":"2024-04-17T08:30:52.168731+0000","last_unstale":"2024-04-17T08:30:52.171344+0000","last_undegraded":"2024-04-17T08:30:52.171344+0000","last_fullsized":"2024-04-17T08:30:52.171344+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:32:00.486067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"181'2","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.185227+0000","last_change":"2024-04-17T08:30:51.145555+0000","last_active":"2024-04-17T08:30:52.185227+0000","last_peered":"2024-04-17T08:30:52.185227+0000","last_clean":"2024-04-17T08:30:52.185227+0000","last_became_active":"2024-04-17T08:30:51.145244+0000","last_became_peered":"2024-04-17T08:30:51.145244+0000","last_unstale":"2024-04-17T08:30:52.185227+0000","last_undegraded":"2024-04-17T08:30:52.185227+0000","last_fullsized":"2024-04-17T08:30:52.185227+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:58.861494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167921+0000","last_change":"2024-04-17T08:30:52.166234+0000","last_active":"2024-04-17T08:30:52.167921+0000","last_peered":"2024-04-17T08:30:52.167921+0000","last_clean":"2024-04-17T08:30:52.167921+0000","last_became_active":"2024-04-17T08:30:52.165861+0000","last_became_peered":"2024-04-17T08:30:52.165861+0000","last_unstale":"2024-04-17T08:30:52.167921+0000","last_undegraded":"2024-04-17T08:30:52.167921+0000","last_fullsized":"2024-04-17T08:30:52.167921+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:21:31.671024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158671+0000","last_change":"2024-04-17T08:30:51.144619+0000","last_active":"2024-04-17T08:30:52.158671+0000","last_peered":"2024-04-17T08:30:52.158671+0000","last_clean":"2024-04-17T08:30:52.158671+0000","last_became_active":"2024-04-17T08:30:51.144303+0000","last_became_peered":"2024-04-17T08:30:51.144303+0000","last_unstale":"2024-04-17T08:30:52.158671+0000","last_undegraded":"2024-04-17T08:30:52.158671+0000","last_fullsized":"2024-04-17T08:30:52.158671+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:33.163237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.166384+0000","last_change":"2024-04-17T08:30:52.163463+0000","last_active":"2024-04-17T08:30:52.166384+0000","last_peered":"2024-04-17T08:30:52.166384+0000","last_clean":"2024-04-17T08:30:52.166384+0000","last_became_active":"2024-04-17T08:30:52.163136+0000","last_became_peered":"2024-04-17T08:30:52.163136+0000","last_unstale":"2024-04-17T08:30:52.166384+0000","last_undegraded":"2024-04-17T08:30:52.166384+0000","last_fullsized":"2024-04-17T08:30:52.166384+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:43.508755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158612+0000","last_change":"2024-04-17T08:30:51.145359+0000","last_active":"2024-04-17T08:30:52.158612+0000","last_peered":"2024-04-17T08:30:52.158612+0000","last_clean":"2024-04-17T08:30:52.158612+0000","last_became_active":"2024-04-17T08:30:51.142679+0000","last_became_peered":"2024-04-17T08:30:51.142679+0000","last_unstale":"2024-04-17T08:30:52.158612+0000","last_undegraded":"2024-04-17T08:30:52.158612+0000","last_fullsized":"2024-04-17T08:30:52.158612+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:06.357590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.175002+0000","last_change":"2024-04-17T08:30:52.168823+0000","last_active":"2024-04-17T08:30:52.175002+0000","last_peered":"2024-04-17T08:30:52.175002+0000","last_clean":"2024-04-17T08:30:52.175002+0000","last_became_active":"2024-04-17T08:30:52.168529+0000","last_became_peered":"2024-04-17T08:30:52.168529+0000","last_unstale":"2024-04-17T08:30:52.175002+0000","last_undegraded":"2024-04-17T08:30:52.175002+0000","last_fullsized":"2024-04-17T08:30:52.175002+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:43.897523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.155841+0000","last_change":"2024-04-17T08:30:51.143759+0000","last_active":"2024-04-17T08:30:52.155841+0000","last_peered":"2024-04-17T08:30:52.155841+0000","last_clean":"2024-04-17T08:30:52.155841+0000","last_became_active":"2024-04-17T08:30:51.142591+0000","last_became_peered":"2024-04-17T08:30:51.142591+0000","last_unstale":"2024-04-17T08:30:52.155841+0000","last_undegraded":"2024-04-17T08:30:52.155841+0000","last_fullsized":"2024-04-17T08:30:52.155841+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:51.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.156203+0000","last_change":"2024-04-17T08:30:52.148094+0000","last_active":"2024-04-17T08:30:52.156203+0000","last_peered":"2024-04-17T08:30:52.156203+0000","last_clean":"2024-04-17T08:30:52.156203+0000","last_became_active":"2024-04-17T08:30:52.147483+0000","last_became_peered":"2024-04-17T08:30:52.147483+0000","last_unstale":"2024-04-17T08:30:52.156203+0000","last_undegraded":"2024-04-17T08:30:52.156203+0000","last_fullsized":"2024-04-17T08:30:52.156203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:40.756138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.144208+0000","last_change":"2024-04-17T08:30:51.125300+0000","last_active":"2024-04-17T08:30:52.144208+0000","last_peered":"2024-04-17T08:30:52.144208+0000","last_clean":"2024-04-17T08:30:52.144208+0000","last_became_active":"2024-04-17T08:30:51.123820+0000","last_became_peered":"2024-04-17T08:30:51.123820+0000","last_unstale":"2024-04-17T08:30:52.144208+0000","last_undegraded":"2024-04-17T08:30:52.144208+0000","last_fullsized":"2024-04-17T08:30:52.144208+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:36.069030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.172848+0000","last_change":"2024-04-17T08:30:52.169570+0000","last_active":"2024-04-17T08:30:52.172848+0000","last_peered":"2024-04-17T08:30:52.172848+0000","last_clean":"2024-04-17T08:30:52.172848+0000","last_became_active":"2024-04-17T08:30:52.167958+0000","last_became_peered":"2024-04-17T08:30:52.167958+0000","last_unstale":"2024-04-17T08:30:52.172848+0000","last_undegraded":"2024-04-17T08:30:52.172848+0000","last_fullsized":"2024-04-17T08:30:52.172848+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:11.058906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.158607+0000","last_change":"2024-04-17T08:30:51.139748+0000","last_active":"2024-04-17T08:30:52.158607+0000","last_peered":"2024-04-17T08:30:52.158607+0000","last_clean":"2024-04-17T08:30:52.158607+0000","last_became_active":"2024-04-17T08:30:51.138377+0000","last_became_peered":"2024-04-17T08:30:51.138377+0000","last_unstale":"2024-04-17T08:30:52.158607+0000","last_undegraded":"2024-04-17T08:30:52.158607+0000","last_fullsized":"2024-04-17T08:30:52.158607+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:04.499164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.167048+0000","last_change":"2024-04-17T08:30:52.162843+0000","last_active":"2024-04-17T08:30:52.167048+0000","last_peered":"2024-04-17T08:30:52.167048+0000","last_clean":"2024-04-17T08:30:52.167048+0000","last_became_active":"2024-04-17T08:30:52.162563+0000","last_became_peered":"2024-04-17T08:30:52.162563+0000","last_unstale":"2024-04-17T08:30:52.167048+0000","last_undegraded":"2024-04-17T08:30:52.167048+0000","last_fullsized":"2024-04-17T08:30:52.167048+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:52:58.919566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.154467+0000","last_change":"2024-04-17T08:30:51.137838+0000","last_active":"2024-04-17T08:30:52.154467+0000","last_peered":"2024-04-17T08:30:52.154467+0000","last_clean":"2024-04-17T08:30:52.154467+0000","last_became_active":"2024-04-17T08:30:51.135840+0000","last_became_peered":"2024-04-17T08:30:51.135840+0000","last_unstale":"2024-04-17T08:30:52.154467+0000","last_undegraded":"2024-04-17T08:30:52.154467+0000","last_fullsized":"2024-04-17T08:30:52.154467+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:06.035116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.170405+0000","last_change":"2024-04-17T08:30:52.161904+0000","last_active":"2024-04-17T08:30:52.170405+0000","last_peered":"2024-04-17T08:30:52.170405+0000","last_clean":"2024-04-17T08:30:52.170405+0000","last_became_active":"2024-04-17T08:30:52.159362+0000","last_became_peered":"2024-04-17T08:30:52.159362+0000","last_unstale":"2024-04-17T08:30:52.170405+0000","last_undegraded":"2024-04-17T08:30:52.170405+0000","last_fullsized":"2024-04-17T08:30:52.170405+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:57.028786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.142935+0000","last_change":"2024-04-17T08:30:51.140395+0000","last_active":"2024-04-17T08:30:52.142935+0000","last_peered":"2024-04-17T08:30:52.142935+0000","last_clean":"2024-04-17T08:30:52.142935+0000","last_became_active":"2024-04-17T08:30:51.140082+0000","last_became_peered":"2024-04-17T08:30:51.140082+0000","last_unstale":"2024-04-17T08:30:52.142935+0000","last_undegraded":"2024-04-17T08:30:52.142935+0000","last_fullsized":"2024-04-17T08:30:52.142935+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:44.787255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":9,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.163500+0000","last_change":"2024-04-17T08:30:52.157090+0000","last_active":"2024-04-17T08:30:52.163500+0000","last_peered":"2024-04-17T08:30:52.163500+0000","last_clean":"2024-04-17T08:30:52.163500+0000","last_became_active":"2024-04-17T08:30:52.156777+0000","last_became_peered":"2024-04-17T08:30:52.156777+0000","last_unstale":"2024-04-17T08:30:52.163500+0000","last_undegraded":"2024-04-17T08:30:52.163500+0000","last_fullsized":"2024-04-17T08:30:52.163500+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:51.093937+0000","last_clean_scrub_stamp":"2024-04-17T08:30:51.093937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:24.431021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"181'3","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.186868+0000","last_change":"2024-04-17T08:30:51.138889+0000","last_active":"2024-04-17T08:30:52.186868+0000","last_peered":"2024-04-17T08:30:52.186868+0000","last_clean":"2024-04-17T08:30:52.186868+0000","last_became_active":"2024-04-17T08:30:51.138425+0000","last_became_peered":"2024-04-17T08:30:51.138425+0000","last_unstale":"2024-04-17T08:30:52.186868+0000","last_undegraded":"2024-04-17T08:30:52.186868+0000","last_fullsized":"2024-04-17T08:30:52.186868+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:30:50.078001+0000","last_clean_scrub_stamp":"2024-04-17T08:30:50.078001+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:51.731306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"169'277","reported_seq":1073,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-04-17T08:30:52.153802+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:30:52.153802+0000","last_peered":"2024-04-17T08:30:52.153802+0000","last_clean":"2024-04-17T08:30:52.153802+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:30:52.153802+0000","last_undegraded":"2024-04-17T08:30:52.153802+0000","last_fullsized":"2024-04-17T08:30:52.153802+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":447,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1506,"num_read_kb":3148,"num_write":447,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47860,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323980,"statfs":{"total":96636764160,"available":96587755520,"internally_reserved":0,"allocated":2854912,"data_stored":1686754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543062,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37556,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334284,"statfs":{"total":96636764160,"available":96598306816,"internally_reserved":0,"allocated":2265088,"data_stored":1095880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2256896,"data_stored":1095756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55592,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316248,"statfs":{"total":96636764160,"available":96579837952,"internally_reserved":0,"allocated":2252800,"data_stored":1095734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543063,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63920,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307920,"statfs":{"total":96636764160,"available":96571310080,"internally_reserved":0,"allocated":2256896,"data_stored":1095824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543062,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47260,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324580,"statfs":{"total":96636764160,"available":96588369920,"internally_reserved":0,"allocated":2240512,"data_stored":1088931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543063,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1096316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543061,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60352,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311488,"statfs":{"total":96636764160,"available":96574963712,"internally_reserved":0,"allocated":2863104,"data_stored":1687748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:30:57.275 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:30:57.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:30:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.517+0000 7f31e8c00640 1 Processor -- start 2024-04-17T08:30:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.517+0000 7f31e8c00640 1 -- start start 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31e8c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f31e40b27c0 0x7f31e4142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31e8c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e415cbe0 con 0x7f31e4071d00 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31e8c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e415cea0 con 0x7f31e413f6d0 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31e8c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f31e4072190 con 0x7f31e40b27c0 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f31e4071d00 0x7f31e40b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57364/0 (socket says 172.21.15.67:57364) 2024-04-17T08:30:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.519+0000 7f31daa00640 1 -- 172.21.15.67:0/4276176686 learned_addr learned my addr 172.21.15.67:0/4276176686 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e3e00640 1 --2- 172.21.15.67:0/4276176686 >> v2:172.21.15.187:3300/0 conn(0x7f31e40b27c0 0x7f31e4142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992660752 0 0) 0x7f31e415cbe0 con 0x7f31e4071d00 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31c8003520 con 0x7f31e4071d00 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3109334777 0 0) 0x7f31c8003520 con 0x7f31e4071d00 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 >> v2:172.21.15.187:3300/0 conn(0x7f31e40b27c0 msgr2=0x7f31e4142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 --2- 172.21.15.67:0/4276176686 >> v2:172.21.15.187:3300/0 conn(0x7f31e40b27c0 0x7f31e4142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 >> v1:172.21.15.187:6789/0 conn(0x7f31e413f6d0 legacy=0x7f31e40cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e415d180 con 0x7f31e4071d00 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f31cc001670 con 0x7f31e4071d00 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 >> v1:172.21.15.67:6789/0 conn(0x7f31e4071d00 legacy=0x7f31e40b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c8003a00 0x7f31c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31c80051c0 0x7f31c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31e4072190 con 0x7f31c8004800 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f31c800c0c0 con 0x7f31c8003a00 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.520+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31c800c2c0 con 0x7f31c80051c0 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31e3400640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c8003a00 0x7f31c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31e3e00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31c80051c0 0x7f31c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 -- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31c80051c0 msgr2=0x7f31c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31c80051c0 0x7f31c800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 -- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c8003a00 msgr2=0x7f31c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c8003a00 0x7f31c8003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31daa00640 1 -- 172.21.15.67:0/4276176686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c8003520 con 0x7f31c8004800 2024-04-17T08:30:57.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.521+0000 7f31e3e00640 1 -- 172.21.15.67:0/4276176686 reap_dead start 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.522+0000 7f31daa00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31c8004c80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c0e0 tx=0x7f31c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53c7859b51453a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.522+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc007c20 con 0x7f31c8004800 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc030070 con 0x7f31c8004800 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e2a00640 1 -- 172.21.15.67:0/4276176686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc005d70 con 0x7f31c8004800 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 -- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 msgr2=0x7f31c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31c8004c80 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c0e0 tx=0x7f31c8005830 comp rx=0 tx=0).stop 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 -- 172.21.15.67:0/4276176686 shutdown_connections 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 --2- 172.21.15.67:0/4276176686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31c8004c80 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 -- 172.21.15.67:0/4276176686 >> 172.21.15.67:0/4276176686 conn(0x7f31e4067d10 msgr2=0x7f31e40d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:57.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 -- 172.21.15.67:0/4276176686 shutdown_connections 2024-04-17T08:30:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.523+0000 7f31e8c00640 1 -- 172.21.15.67:0/4276176686 wait complete. 2024-04-17T08:30:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.525+0000 7f31e8c00640 1 Processor -- start 2024-04-17T08:30:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.525+0000 7f31e8c00640 1 -- start start 2024-04-17T08:30:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 0x7f31e4140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 0x7f31e41f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31e40d1120 con 0x7f31c8004800 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f31e415d740 con 0x7f31e4135440 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e8c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31e415da20 con 0x7f31e4141170 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.526+0000 7f31e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35538/0 (socket says 172.21.15.67:35538) 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 -- 172.21.15.67:0/3921531461 learned_addr learned my addr 172.21.15.67:0/3921531461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3e00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 0x7f31e41f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31daa00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 0x7f31e4140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 -- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 msgr2=0x7f31e41f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 0x7f31e41f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 -- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 msgr2=0x7f31e4140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 0x7f31e4140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 -- 172.21.15.67:0/3921531461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e4156b30 con 0x7f31c8004800 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3e00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 0x7f31e41f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31daa00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 0x7f31e4140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.527+0000 7f31e3400640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d400 tx=0x7f31d400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.528+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4018070 con 0x7f31c8004800 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.528+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d4003610 con 0x7f31c8004800 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.528+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4003a40 con 0x7f31c8004800 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.528+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e4135bb0 con 0x7f31c8004800 2024-04-17T08:30:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.528+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e4136170 con 0x7f31c8004800 2024-04-17T08:30:57.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.529+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e4068a10 con 0x7f31c8004800 2024-04-17T08:30:57.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.530+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f31d401b070 con 0x7f31c8004800 2024-04-17T08:30:57.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.531+0000 7f31dbe00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 0x7f31c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:57.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.531+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f31d408b630 con 0x7f31c8004800 2024-04-17T08:30:57.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.532+0000 7f31daa00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 0x7f31c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:57.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.533+0000 7f31daa00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 0x7f31c006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31cc034000 tx=0x7f31cc039040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:57.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.535+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f31d4059130 con 0x7f31c8004800 2024-04-17T08:30:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.815+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f31e406e760 con 0x7f31c8004800 2024-04-17T08:30:57.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.819+0000 7f31dbe00640 1 -- 172.21.15.67:0/3921531461 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v41) ==== 190+0+147 (secure 0 0 0) 0x7f31d405d430 con 0x7f31c8004800 2024-04-17T08:30:57.818 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:30:57.818 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:30:57.819 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:30:57.819 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:30:57.819 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:30:57.819 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 msgr2=0x7f31c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 0x7f31c006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31cc034000 tx=0x7f31cc039040 comp rx=0 tx=0).stop 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 msgr2=0x7f31e41f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d400 tx=0x7f31d400d8c0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 shutdown_connections 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f31c006ca10 0x7f31c006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31e4141170 0x7f31e41f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31e4135440 0x7f31e4140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 --2- 172.21.15.67:0/3921531461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31c8004800 0x7f31e41f8fb0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:57.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 >> 172.21.15.67:0/3921531461 conn(0x7f31e4067d10 msgr2=0x7f31e406dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:57.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 shutdown_connections 2024-04-17T08:30:57.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:57.822+0000 7f31e8c00640 1 -- 172.21.15.67:0/3921531461 wait complete. 2024-04-17T08:30:57.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:30:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.065+0000 7f8cfb200640 1 Processor -- start 2024-04-17T08:30:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.065+0000 7f8cfb200640 1 -- start start 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfb200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8cfc06feb0 0x7f8cfc06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc06be70 0x7f8cfc15f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cfc15bc80 con 0x7f8cfc15b570 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfb200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8cfc15bf40 con 0x7f8cfc06feb0 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cfc15c220 con 0x7f8cfc06be70 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cf9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8cfc15b570 0x7f8cfc06f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57378/0 (socket says 172.21.15.67:57378) 2024-04-17T08:30:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cf9e00640 1 -- 172.21.15.67:0/2916169217 learned_addr learned my addr 172.21.15.67:0/2916169217 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954950333 0 0) 0x7f8cfc15bc80 con 0x7f8cfc15b570 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cfa800640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc06be70 0x7f8cfc15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ce0005730 con 0x7f8cfc15b570 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.067+0000 7f8cf1400640 1 --2- 172.21.15.67:0/2916169217 >> v2:172.21.15.187:3300/0 conn(0x7f8cfc06feb0 0x7f8cfc06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3409552867 0 0) 0x7f8ce0005730 con 0x7f8cfc15b570 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc06be70 msgr2=0x7f8cfc15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc06be70 0x7f8cfc15f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 >> v2:172.21.15.187:3300/0 conn(0x7f8cfc06feb0 msgr2=0x7f8cfc06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 --2- 172.21.15.67:0/2916169217 >> v2:172.21.15.187:3300/0 conn(0x7f8cfc06feb0 0x7f8cfc06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cfc15c500 con 0x7f8cfc15b570 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8cec001670 con 0x7f8cfc15b570 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 >> v1:172.21.15.67:6789/0 conn(0x7f8cfc15b570 legacy=0x7f8cfc06f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ce0005c30 0x7f8ce0006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ce0006a10 0x7f8ce0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce00073d0 0x7f8ce000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc15bf40 con 0x7f8ce0006a10 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc15c220 con 0x7f8ce00073d0 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce000e320 con 0x7f8ce0005c30 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.068+0000 7f8cf1400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ce0006a10 0x7f8ce0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.069+0000 7f8cf1400640 1 -- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ce0005c30 msgr2=0x7f8ce0006050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.069+0000 7f8cf1400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ce0005c30 0x7f8ce0006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.069+0000 7f8cf1400640 1 -- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce00073d0 msgr2=0x7f8ce000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.069+0000 7f8cf1400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce00073d0 0x7f8ce000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.069+0000 7f8cf1400640 1 -- 172.21.15.67:0/2916169217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce0005730 con 0x7f8ce0006a10 2024-04-17T08:30:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.070+0000 7f8cf1400640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ce0006a10 0x7f8ce0006e90 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400ba30 tx=0x7f8ce0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274e1f9fd6c46e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.070+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4036020 con 0x7f8ce0006a10 2024-04-17T08:30:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.070+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce4032680 con 0x7f8ce0006a10 2024-04-17T08:30:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.070+0000 7f8cf9400640 1 -- 172.21.15.67:0/2916169217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce400b600 con 0x7f8ce0006a10 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 -- 172.21.15.67:0/2916169217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ce0006a10 msgr2=0x7f8ce0006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 --2- 172.21.15.67:0/2916169217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ce0006a10 0x7f8ce0006e90 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f8ce400ba30 tx=0x7f8ce0007a40 comp rx=0 tx=0).stop 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfa800640 1 -- 172.21.15.67:0/2916169217 reap_dead start 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 -- 172.21.15.67:0/2916169217 shutdown_connections 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 -- 172.21.15.67:0/2916169217 >> 172.21.15.67:0/2916169217 conn(0x7f8cfc067d10 msgr2=0x7f8cfc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 -- 172.21.15.67:0/2916169217 shutdown_connections 2024-04-17T08:30:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.071+0000 7f8cfb200640 1 -- 172.21.15.67:0/2916169217 wait complete. 2024-04-17T08:30:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.073+0000 7f8cfb200640 1 Processor -- start 2024-04-17T08:30:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.073+0000 7f8cfb200640 1 -- start start 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce0006a10 0x7f8cfc0b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 0x7f8cfc140bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc06c440 con 0x7f8cfc142640 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8cfc06c2c0 con 0x7f8ce0006a10 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cfb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cfc0b47b0 con 0x7f8cfc140730 2024-04-17T08:30:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35556/0 (socket says 172.21.15.67:35556) 2024-04-17T08:30:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.074+0000 7f8cf1400640 1 -- 172.21.15.67:0/3298909144 learned_addr learned my addr 172.21.15.67:0/3298909144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cfa800640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 0x7f8cfc140bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 -- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 msgr2=0x7f8cfc140bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf9e00640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce0006a10 0x7f8cfc0b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 0x7f8cfc140bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 -- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce0006a10 msgr2=0x7f8cfc0b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce0006a10 0x7f8cfc0b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 -- 172.21.15.67:0/3298909144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cfc13ad00 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cfa800640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 0x7f8cfc140bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf1400640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4035040 tx=0x7f8ce4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.075+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4041070 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.076+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce4030a90 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.076+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4036040 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.076+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cfc13afe0 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.076+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cfc13b5a0 con 0x7f8cfc142640 2024-04-17T08:30:58.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.077+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cfc0b56d0 con 0x7f8cfc142640 2024-04-17T08:30:58.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.078+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f8ce4039020 con 0x7f8cfc142640 2024-04-17T08:30:58.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.079+0000 7f8cf2a00640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 0x7f8cd806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:58.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.079+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f8ce40ae210 con 0x7f8cfc142640 2024-04-17T08:30:58.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.080+0000 7f8cf9e00640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 0x7f8cd806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.081+0000 7f8cf9e00640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 0x7f8cd806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8cec000ff0 tx=0x7f8cec009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.083+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8cfc0b56d0 con 0x7f8cfc142640 2024-04-17T08:30:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.340+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cfc068a10 con 0x7f8cfc142640 2024-04-17T08:30:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.341+0000 7f8cf2a00640 1 -- 172.21.15.67:0/3298909144 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f8ce407c470 con 0x7f8cfc142640 2024-04-17T08:30:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:30:58.340 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:30:58.341 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:30:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 msgr2=0x7f8cd806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 0x7f8cd806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8cec000ff0 tx=0x7f8cec009040 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 msgr2=0x7f8cfc13f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4035040 tx=0x7f8ce4004730 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 shutdown_connections 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f8cd806ca10 0x7f8cd806eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cfc140730 0x7f8cfc140bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cfc142640 0x7f8cfc13f520 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 --2- 172.21.15.67:0/3298909144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ce0006a10 0x7f8cfc0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 >> 172.21.15.67:0/3298909144 conn(0x7f8cfc067d10 msgr2=0x7f8cfc0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 shutdown_connections 2024-04-17T08:30:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:58.344+0000 7f8cfb200640 1 -- 172.21.15.67:0/3298909144 wait complete. 2024-04-17T08:30:58.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:30:51.593402+0000', 'modified': '2024-04-17T08:30:54.685999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9462}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9462': {'gid': 9462, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.67:6833/3206760811', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3206760811}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3206760811}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9462, 'qdb_cluster': [9462]} 2024-04-17T08:30:58.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:30:58.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T08:30:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.003+0000 7f5af4800640 1 Processor -- start 2024-04-17T08:30:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.003+0000 7f5af4800640 1 -- start start 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.004+0000 7f5af4800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5aec15b860 0x7f5aec0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af4800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aec0ce4d0 0x7f5aec0d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af4800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec071f60 con 0x7f5aec06dd10 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af4800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec071d10 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af4800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec072240 con 0x7f5aec15b860 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aec0ce4d0 0x7f5aec0d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5aec0ce4d0 0x7f5aec0d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35568/0 (socket says 172.21.15.67:35568) 2024-04-17T08:30:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 -- 172.21.15.67:0/1439454885 learned_addr learned my addr 172.21.15.67:0/1439454885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 -- 172.21.15.67:0/1439454885 >> v2:172.21.15.187:3300/0 conn(0x7f5aec15b860 msgr2=0x7f5aec0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 --2- 172.21.15.67:0/1439454885 >> v2:172.21.15.187:3300/0 conn(0x7f5aec15b860 0x7f5aec0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 -- 172.21.15.67:0/1439454885 >> v1:172.21.15.187:6789/0 conn(0x7f5aec06dd10 legacy=0x7f5aec0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.005+0000 7f5af1400640 1 -- 172.21.15.67:0/1439454885 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec072520 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.006+0000 7f5af1400640 1 --2- 172.21.15.67:0/1439454885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec0ce4d0 0x7f5aec0d0960 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0ce240 tx=0x7f5adc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4704872fd964bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.007+0000 7f5af0a00640 1 -- 172.21.15.67:0/1439454885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc035020 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.007+0000 7f5af0a00640 1 -- 172.21.15.67:0/1439454885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5adc031870 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.007+0000 7f5af0a00640 1 -- 172.21.15.67:0/1439454885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc031c40 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 -- 172.21.15.67:0/1439454885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec0ce4d0 msgr2=0x7f5aec0d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 --2- 172.21.15.67:0/1439454885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec0ce4d0 0x7f5aec0d0960 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0ce240 tx=0x7f5adc02b910 comp rx=0 tx=0).stop 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 -- 172.21.15.67:0/1439454885 shutdown_connections 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 --2- 172.21.15.67:0/1439454885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec0ce4d0 0x7f5aec0d0960 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 --2- 172.21.15.67:0/1439454885 >> v2:172.21.15.187:3300/0 conn(0x7f5aec15b860 0x7f5aec0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 -- 172.21.15.67:0/1439454885 >> 172.21.15.67:0/1439454885 conn(0x7f5aec067ee0 msgr2=0x7f5aec06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 -- 172.21.15.67:0/1439454885 shutdown_connections 2024-04-17T08:30:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.008+0000 7f5af4800640 1 -- 172.21.15.67:0/1439454885 wait complete. 2024-04-17T08:30:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.010+0000 7f5af4800640 1 Processor -- start 2024-04-17T08:30:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.010+0000 7f5af4800640 1 -- start start 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aec06dd10 0x7f5aec1f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aec0ce4d0 0x7f5aec155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec15bc40 con 0x7f5aec15b860 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec072ae0 con 0x7f5aec06dd10 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af4800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec06f380 con 0x7f5aec0ce4d0 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35570/0 (socket says 172.21.15.67:35570) 2024-04-17T08:30:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af2800640 1 -- 172.21.15.67:0/2883577301 learned_addr learned my addr 172.21.15.67:0/2883577301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af1e00640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aec06dd10 0x7f5aec1f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af1400640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aec0ce4d0 0x7f5aec155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.012+0000 7f5af2800640 1 -- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aec0ce4d0 msgr2=0x7f5aec155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af2800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aec0ce4d0 0x7f5aec155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af2800640 1 -- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aec06dd10 msgr2=0x7f5aec1f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af2800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aec06dd10 0x7f5aec1f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af2800640 1 -- 172.21.15.67:0/2883577301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec155960 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af2800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c3c0 tx=0x7f5ae800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5ae2000640 1 -- 172.21.15.67:0/2883577301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8017070 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5ae2000640 1 -- 172.21.15.67:0/2883577301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae80035f0 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5ae2000640 1 -- 172.21.15.67:0/2883577301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80039e0 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec1f46f0 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.013+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec1f8a10 con 0x7f5aec15b860 2024-04-17T08:30:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.016+0000 7f5ae2000640 1 -- 172.21.15.67:0/2883577301 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f5ae801a070 con 0x7f5aec15b860 2024-04-17T08:30:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.017+0000 7f5ae2000640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 0x7f5ad006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.017+0000 7f5ae2000640 1 -- 172.21.15.67:0/2883577301 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f5ae808b160 con 0x7f5aec15b860 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.017+0000 7f5af75ce740 1 Processor -- start 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.017+0000 7f5af75ce740 1 -- start start 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af1e00640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 0x7f5ad006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa683a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa683f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625aa5ea980 con 0x5625aa67e8e0 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5625aa591220 con 0x5625aa6cc410 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.018+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5625aa5e6940 con 0x5625aa682490 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35586/0 (socket says 172.21.15.67:35586) 2024-04-17T08:30:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 -- 172.21.15.67:0/2533095388 learned_addr learned my addr 172.21.15.67:0/2533095388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af1e00640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa683a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af1400640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa683f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 -- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 msgr2=0x5625aa683f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af1e00640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 0x7f5ad006eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f70 tx=0x7f5ae40094f0 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa683f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 -- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 msgr2=0x5625aa683a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa683a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 -- 172.21.15.67:0/2533095388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625aa2fd2d0 con 0x5625aa67e8e0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af1400640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa683f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.019+0000 7f5af2800640 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8001060 tx=0x7f5ae80594f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea63a91cc05f4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5ad6600640 1 -- 172.21.15.67:0/2533095388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8071020 con 0x5625aa67e8e0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5ad6600640 1 -- 172.21.15.67:0/2533095388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae80711c0 con 0x5625aa67e8e0 2024-04-17T08:30:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5ad6600640 1 -- 172.21.15.67:0/2533095388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8055330 con 0x5625aa67e8e0 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 -- 172.21.15.67:0/2533095388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 msgr2=0x5625aa6ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8001060 tx=0x7f5ae80594f0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 -- 172.21.15.67:0/2533095388 shutdown_connections 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa67e8e0 0x5625aa6ae720 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa683f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 --2- 172.21.15.67:0/2533095388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa683a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 -- 172.21.15.67:0/2533095388 >> 172.21.15.67:0/2533095388 conn(0x5625aa680ba0 msgr2=0x5625aa6c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 -- 172.21.15.67:0/2533095388 shutdown_connections 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.020+0000 7f5af75ce740 1 -- 172.21.15.67:0/2533095388 wait complete. 2024-04-17T08:30:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.021+0000 7f5af75ce740 1 -- start start 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.021+0000 7f5af75ce740 1 Processor -- start 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa6cc410 0x5625aa6c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa6ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625aa6829a0 con 0x5625aa682490 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5625aa5a4590 con 0x5625aa6ce6c0 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.022+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5625aa682c80 con 0x5625aa6cc410 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af75ce740 10 client.0 _ll_register_callbacks cb 0x5625aa2fcac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35602/0 (socket says 172.21.15.67:35602) 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 -- 172.21.15.67:0/1001614024 learned_addr learned my addr 172.21.15.67:0/1001614024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1400640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa6cc410 0x5625aa6c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af2800640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa6ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 -- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa6cc410 msgr2=0x5625aa6c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa6cc410 0x5625aa6c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 -- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 msgr2=0x5625aa6ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa6ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.023+0000 7f5af1e00640 1 -- 172.21.15.67:0/1001614024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625aa3c76b0 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4017da0 tx=0x7f5ae4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d582caa7a5a8550 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5acaa00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4027060 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f5ae4016b30 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ae80711c0 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f5ae4016d30 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5af75ce740 10 client.9612 fetch_fsmap learned FSMap version 85 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) -- 0x5625aa6deaa0 con 0x5625aa682490 2024-04-17T08:30:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.024+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae80711c0 con 0x5625aa682490 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 85) ==== 3495+0+0 (secure 0 0 0) 0x7f5ae4030080 con 0x5625aa682490 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 10 client.9612 fetch_fsmap finished waiting for FSMap version 85 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 10 client.9612 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 20 client.9612 populate_metadata read hostname 'smithi067' 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6cb040 0x5625aa6cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 4 client.9612 mds_command: new command op to 9462 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 --> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625aa5f83a0 con 0x5625aa6cb040 2024-04-17T08:30:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.025+0000 7f5af1400640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6cb040 0x5625aa6cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.026+0000 7f5af1400640 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6cb040 0x5625aa6cb420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76daf15264c8bb82 server_cookie=a285c0f36c68d323 in_seq=0 out_seq=0 2024-04-17T08:30:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.026+0000 7f5acaa00640 10 client.9612 ms_handle_connect on v2:172.21.15.67:6832/3206760811 2024-04-17T08:30:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5acaa00640 1 -- 172.21.15.67:0/1001614024 <== mds.0 v2:172.21.15.67:6832/3206760811 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625aa5f83a0 con 0x5625aa6cb040 2024-04-17T08:30:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5acaa00640 10 client.9612 handle_command_reply: tid=0 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 client.9612 shutdown 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 msgr2=0x5625aa6dde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4017da0 tx=0x7f5ae4004a40 comp rx=0 tx=0).stop 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 shutdown_connections 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6cb040 0x5625aa6cb420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa6ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa6cc410 0x5625aa6c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1001614024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa682490 0x5625aa6dde50 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5acaa00640 0 client.9612 ms_handle_reset on v2:172.21.15.67:6832/3206760811 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 >> 172.21.15.67:0/1001614024 conn(0x5625aa67e8e0 msgr2=0x5625aa67f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 shutdown_connections 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 1 -- 172.21.15.67:0/1001614024 wait complete. 2024-04-17T08:30:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.027+0000 7f5af75ce740 20 client.9612 trim_cache size 0 max 16384 2024-04-17T08:30:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.047+0000 7f5af75ce740 1 Processor -- start 2024-04-17T08:30:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 -- start start 2024-04-17T08:30:59.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa67d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa6837a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec200130 con 0x5625aa6ce6c0 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec15cb30 con 0x5625aa6cc410 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.048+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec15e930 con 0x5625aa682490 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35616/0 (socket says 172.21.15.67:35616) 2024-04-17T08:30:59.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 -- 172.21.15.67:0/1639669311 learned_addr learned my addr 172.21.15.67:0/1639669311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 -- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 msgr2=0x5625aa67d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa67d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 -- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 msgr2=0x5625aa6837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa6837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 -- 172.21.15.67:0/1639669311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625aa6cb880 con 0x5625aa6ce6c0 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5af2800640 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80589c0 tx=0x7f5ae808b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e862a911a90c9410 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.049+0000 7f5ad5c00640 1 -- 172.21.15.67:0/1639669311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8071040 con 0x5625aa6ce6c0 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5ad5c00640 1 -- 172.21.15.67:0/1639669311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae80895c0 con 0x5625aa6ce6c0 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5ad5c00640 1 -- 172.21.15.67:0/1639669311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8003b80 con 0x5625aa6ce6c0 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 -- 172.21.15.67:0/1639669311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 msgr2=0x5625aa6bf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80589c0 tx=0x7f5ae808b000 comp rx=0 tx=0).stop 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 -- 172.21.15.67:0/1639669311 shutdown_connections 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6ce6c0 0x5625aa6bf0b0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6cc410 0x5625aa6837a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1639669311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa67d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 -- 172.21.15.67:0/1639669311 >> 172.21.15.67:0/1639669311 conn(0x5625aa67e8e0 msgr2=0x5625aa684550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 -- 172.21.15.67:0/1639669311 shutdown_connections 2024-04-17T08:30:59.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.050+0000 7f5af75ce740 1 -- 172.21.15.67:0/1639669311 wait complete. 2024-04-17T08:30:59.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.051+0000 7f5af75ce740 1 -- start start 2024-04-17T08:30:59.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.051+0000 7f5af75ce740 1 Processor -- start 2024-04-17T08:30:59.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa6ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa73c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625aa6a4d20 con 0x5625aa6cc410 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5625aa6d0700 con 0x5625aa6ce6c0 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5625aa6d17a0 con 0x5625aa682490 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af75ce740 10 client.0 _ll_register_callbacks cb 0x7f5ae4017a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35618/0 (socket says 172.21.15.67:35618) 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.052+0000 7f5af1400640 1 -- 172.21.15.67:0/1117233732 learned_addr learned my addr 172.21.15.67:0/1117233732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa6ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 -- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 msgr2=0x5625aa6ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa6ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 -- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 msgr2=0x5625aa73c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af2800640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa73c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa73c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 -- 172.21.15.67:0/1117233732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625aa6cd130 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5af1400640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009930 tx=0x7f5adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a932e97ff65aa09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.053+0000 7f5acaa00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc049050 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ae80895c0 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f5adc0491f0 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f5adc005990 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 10 client.9618 fetch_fsmap learned FSMap version 85 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) -- 0x5625aa6b2260 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae80895c0 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 85) ==== 3495+0+0 (secure 0 0 0) 0x7f5adc044040 con 0x5625aa6cc410 2024-04-17T08:30:59.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 10 client.9618 fetch_fsmap finished waiting for FSMap version 85 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 10 client.9618 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 20 client.9618 populate_metadata read hostname 'smithi067' 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6ee3b0 0x5625aa7487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 4 client.9618 mds_command: new command op to 9462 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.054+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 --> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] -- command(tid 0: {"prefix": "status"}) -- 0x5625aa5f2210 con 0x5625aa6ee3b0 2024-04-17T08:30:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6ee3b0 0x5625aa7487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5af1e00640 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6ee3b0 0x5625aa7487e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f1bbec090bca1c server_cookie=ac650cc50ac13df8 in_seq=0 out_seq=0 2024-04-17T08:30:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5acaa00640 10 client.9618 ms_handle_connect on v2:172.21.15.67:6832/3206760811 2024-04-17T08:30:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5acaa00640 1 -- 172.21.15.67:0/1117233732 <== mds.0 v2:172.21.15.67:6832/3206760811 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x5625aa5f2210 con 0x5625aa6ee3b0 2024-04-17T08:30:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5acaa00640 10 client.9618 handle_command_reply: tid=0 2024-04-17T08:30:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.055+0000 7f5af75ce740 1 client.9618 shutdown 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 msgr2=0x5625aa73bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009930 tx=0x7f5adc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 shutdown_connections 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:6832/3206760811,v1:172.21.15.67:6833/3206760811] conn(0x5625aa6ee3b0 0x5625aa7487e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5625aa6ce6c0 0x5625aa73c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625aa6cc410 0x5625aa73bd00 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 --2- 172.21.15.67:0/1117233732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5625aa682490 0x5625aa6ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5acaa00640 0 client.9618 ms_handle_reset on v2:172.21.15.67:6832/3206760811 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 >> 172.21.15.67:0/1117233732 conn(0x5625aa67e8e0 msgr2=0x5625aa684550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 shutdown_connections 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 1 -- 172.21.15.67:0/1117233732 wait complete. 2024-04-17T08:30:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.056+0000 7f5af75ce740 20 client.9618 trim_cache size 0 max 16384 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9462, 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.9031962729999998, 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 85, 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 182, 2024-04-17T08:30:59.059 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 182, 2024-04-17T08:30:59.060 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 19.407185512000002 2024-04-17T08:30:59.060 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:30:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 msgr2=0x7f5ad006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 0x7f5ad006eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009f70 tx=0x7f5ae40094f0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 msgr2=0x7f5aec1f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c3c0 tx=0x7f5ae800c880 comp rx=0 tx=0).stop 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 shutdown_connections 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f5ad006ca10 0x7f5ad006eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5aec15b860 0x7f5aec1f7c70 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5aec0ce4d0 0x7f5aec155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 --2- 172.21.15.67:0/2883577301 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aec06dd10 0x7f5aec1f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 >> 172.21.15.67:0/2883577301 conn(0x7f5aec067ee0 msgr2=0x7f5aec06e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 shutdown_connections 2024-04-17T08:30:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.057+0000 7f5af4800640 1 -- 172.21.15.67:0/2883577301 wait complete. 2024-04-17T08:30:59.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:30:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.283+0000 7fe681e00640 1 Processor -- start 2024-04-17T08:30:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.283+0000 7fe681e00640 1 -- start start 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.284+0000 7fe681e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.284+0000 7fe681e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67c141ed0 0x7fe67c142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe67c0cd4a0 con 0x7fe67c06b8c0 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe67c15ce30 con 0x7fe67c141ed0 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe67c1607f0 con 0x7fe67c160280 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67c141ed0 0x7fe67c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67c141ed0 0x7fe67c142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35634/0 (socket says 172.21.15.67:35634) 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681400640 1 -- 172.21.15.67:0/1696014699 learned_addr learned my addr 172.21.15.67:0/1696014699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe67be00640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681400640 1 -- 172.21.15.67:0/1696014699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 msgr2=0x7fe67c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:30:59.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.285+0000 7fe681400640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.286+0000 7fe681400640 1 -- 172.21.15.67:0/1696014699 >> v1:172.21.15.187:6789/0 conn(0x7fe67c06b8c0 legacy=0x7fe67c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.286+0000 7fe681400640 1 -- 172.21.15.67:0/1696014699 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67c160ad0 con 0x7fe67c141ed0 2024-04-17T08:30:59.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.287+0000 7fe681400640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c142360 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe674009e50 tx=0x7fe67402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e05a59eb168522 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.287+0000 7fe67b400640 1 -- 172.21.15.67:0/1696014699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67402cb00 con 0x7fe67c141ed0 2024-04-17T08:30:59.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.287+0000 7fe67b400640 1 -- 172.21.15.67:0/1696014699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe67402cca0 con 0x7fe67c141ed0 2024-04-17T08:30:59.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.287+0000 7fe67b400640 1 -- 172.21.15.67:0/1696014699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674034610 con 0x7fe67c141ed0 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 -- 172.21.15.67:0/1696014699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 msgr2=0x7fe67c142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c142360 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe674009e50 tx=0x7fe67402bf60 comp rx=0 tx=0).stop 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 -- 172.21.15.67:0/1696014699 shutdown_connections 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c142360 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 --2- 172.21.15.67:0/1696014699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 -- 172.21.15.67:0/1696014699 >> 172.21.15.67:0/1696014699 conn(0x7fe67c067d10 msgr2=0x7fe67c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 -- 172.21.15.67:0/1696014699 shutdown_connections 2024-04-17T08:30:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.288+0000 7fe681e00640 1 -- 172.21.15.67:0/1696014699 wait complete. 2024-04-17T08:30:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.290+0000 7fe681e00640 1 Processor -- start 2024-04-17T08:30:59.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.290+0000 7fe681e00640 1 -- start start 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe67c06b8c0 0x7fe67c144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c161090 con 0x7fe67c141ed0 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c06c1e0 con 0x7fe67c06b8c0 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe67c13f9b0 con 0x7fe67c160280 2024-04-17T08:30:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35648/0 (socket says 172.21.15.67:35648) 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 -- 172.21.15.67:0/1184928575 learned_addr learned my addr 172.21.15.67:0/1184928575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681400640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe680a00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe67c06b8c0 0x7fe67c144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 -- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 msgr2=0x7fe67c158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 -- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe67c06b8c0 msgr2=0x7fe67c144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe67c06b8c0 0x7fe67c144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe67be00640 1 -- 172.21.15.67:0/1184928575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67c144b40 con 0x7fe67c141ed0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.291+0000 7fe681400640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe67be00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d5d0 tx=0x7fe66c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0125f0 con 0x7fe67c141ed0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe66c019070 con 0x7fe67c141ed0 2024-04-17T08:30:59.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe67c144e20 con 0x7fe67c141ed0 2024-04-17T08:30:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0129f0 con 0x7fe67c141ed0 2024-04-17T08:30:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.292+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67c1453e0 con 0x7fe67c141ed0 2024-04-17T08:30:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.293+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe67c068a10 con 0x7fe67c141ed0 2024-04-17T08:30:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.294+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7fe66c012b90 con 0x7fe67c141ed0 2024-04-17T08:30:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.295+0000 7fe678c00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 0x7fe66006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:30:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.295+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7fe66c08c1d0 con 0x7fe67c141ed0 2024-04-17T08:30:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.296+0000 7fe680a00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 0x7fe66006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:30:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.297+0000 7fe680a00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 0x7fe66006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe6700015a0 tx=0x7fe670009040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:30:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.300+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe66c05a430 con 0x7fe67c141ed0 2024-04-17T08:30:59.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.546+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe67c146220 con 0x7fe67c141ed0 2024-04-17T08:30:59.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.546+0000 7fe678c00640 1 -- 172.21.15.67:0/1184928575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v85) ==== 53+0+73 (secure 0 0 0) 0x7fe66c059d30 con 0x7fe67c141ed0 2024-04-17T08:30:59.546 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 msgr2=0x7fe66006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 0x7fe66006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe6700015a0 tx=0x7fe670009040 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 msgr2=0x7fe67c157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d5d0 tx=0x7fe66c00da90 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 shutdown_connections 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fe66006ca40 0x7fe66006ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe67c160280 0x7fe67c158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c141ed0 0x7fe67c157bf0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 --2- 172.21.15.67:0/1184928575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe67c06b8c0 0x7fe67c144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 >> 172.21.15.67:0/1184928575 conn(0x7fe67c067d10 msgr2=0x7fe67c0d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 shutdown_connections 2024-04-17T08:30:59.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59.549+0000 7fe681e00640 1 -- 172.21.15.67:0/1184928575 wait complete. 2024-04-17T08:30:59.569 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:30:59.570 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:30:59.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:59.570 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:30:59.626 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:30:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:30:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:30:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.626 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:30:59.627 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 37984sec preferred_lft 37984sec 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.628 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:30:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:30:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:30:59.629 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:30:59.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:59.630 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:30:59.721 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:30:59.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:30:59.726 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:30:59.779 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:30:59.779 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:30:59.780 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:30:59.780 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:30:59.780 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:30:59.780 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:30:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:30:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:31:00.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:31:00.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:00.033 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:31:00.083 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:31:00.083 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:31:00.084 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:31:00.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:00.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:31:00.179 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:31:00.180 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:00.233 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:31:00.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:31:00.233 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:00.284 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:31:00.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:00.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:00.429 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:00.430 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:31:00.464 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:00.464 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:31:00.464 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:31:00.464 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:31:00.464 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:31:00.466 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:31:00.466 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:31:00.466 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:31:00.466 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:31:00.466 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:31:00.470 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:31:00.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:00.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:00.553 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:31:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:31:00.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:31:00.685 INFO:teuthology.orchestra.run.smithi067.stdout:client7714 2024-04-17T08:31:00.685 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:00.685 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:00.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:31:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.956+0000 7f10baa00640 1 Processor -- start 2024-04-17T08:31:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.957+0000 7f10baa00640 1 -- start start 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10b414c1d0 0x7f10b40ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b406f550 0x7f10b4167400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b4071e60 0x7f10b40722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10b4166f70 con 0x7f10b406f550 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10b406fb10 con 0x7f10b414c1d0 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.958+0000 7f10baa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10b406fdf0 con 0x7f10b4071e60 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b406f550 0x7f10b4167400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b406f550 0x7f10b4167400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35654/0 (socket says 172.21.15.67:35654) 2024-04-17T08:31:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 -- 172.21.15.67:0/1646043835 learned_addr learned my addr 172.21.15.67:0/1646043835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b9600640 1 --2- 172.21.15.67:0/1646043835 >> v2:172.21.15.187:3300/0 conn(0x7f10b414c1d0 0x7f10b40ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10ba000640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b4071e60 0x7f10b40722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 -- 172.21.15.67:0/1646043835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b4071e60 msgr2=0x7f10b40722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b4071e60 0x7f10b40722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 -- 172.21.15.67:0/1646043835 >> v2:172.21.15.187:3300/0 conn(0x7f10b414c1d0 msgr2=0x7f10b40ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1646043835 >> v2:172.21.15.187:3300/0 conn(0x7f10b414c1d0 0x7f10b40ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.959+0000 7f10b8c00640 1 -- 172.21.15.67:0/1646043835 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b4166c00 con 0x7f10b406f550 2024-04-17T08:31:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.961+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b406f550 0x7f10b4167400 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f10b4072fc0 tx=0x7f10a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be82810e59c68a4c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.961+0000 7f10b3e00640 1 -- 172.21.15.67:0/1646043835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a802c9e0 con 0x7f10b406f550 2024-04-17T08:31:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.961+0000 7f10b3e00640 1 -- 172.21.15.67:0/1646043835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a802cb80 con 0x7f10b406f550 2024-04-17T08:31:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.961+0000 7f10b3e00640 1 -- 172.21.15.67:0/1646043835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a80343f0 con 0x7f10b406f550 2024-04-17T08:31:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 -- 172.21.15.67:0/1646043835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b406f550 msgr2=0x7f10b4167400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b406f550 0x7f10b4167400 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f10b4072fc0 tx=0x7f10a802be40 comp rx=0 tx=0).stop 2024-04-17T08:31:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 -- 172.21.15.67:0/1646043835 shutdown_connections 2024-04-17T08:31:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b4071e60 0x7f10b40722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 --2- 172.21.15.67:0/1646043835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b406f550 0x7f10b4167400 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 --2- 172.21.15.67:0/1646043835 >> v2:172.21.15.187:3300/0 conn(0x7f10b414c1d0 0x7f10b40ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 -- 172.21.15.67:0/1646043835 >> 172.21.15.67:0/1646043835 conn(0x7f10b406bef0 msgr2=0x7f10b40b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 -- 172.21.15.67:0/1646043835 shutdown_connections 2024-04-17T08:31:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.962+0000 7f10baa00640 1 -- 172.21.15.67:0/1646043835 wait complete. 2024-04-17T08:31:00.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.964+0000 7f10baa00640 1 Processor -- start 2024-04-17T08:31:00.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.964+0000 7f10baa00640 1 -- start start 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b406f550 0x7f10b40b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10b414c1d0 0x7f10b4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10b40700d0 con 0x7f10b4071e60 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10b414ca80 con 0x7f10b414c1d0 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10baa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10b4167940 con 0x7f10b406f550 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10b8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10b8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35668/0 (socket says 172.21.15.67:35668) 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.965+0000 7f10b8c00640 1 -- 172.21.15.67:0/1938988417 learned_addr learned my addr 172.21.15.67:0/1938988417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b9600640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b406f550 0x7f10b40b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10ba000640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10b414c1d0 0x7f10b4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 -- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b406f550 msgr2=0x7f10b40b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:00.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b406f550 0x7f10b40b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 -- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10b414c1d0 msgr2=0x7f10b4158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:00.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10b414c1d0 0x7f10b4158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:00.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 -- 172.21.15.67:0/1938988417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b40b5810 con 0x7f10b4071e60 2024-04-17T08:31:00.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.966+0000 7f10b8c00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f10a8000f80 tx=0x7f10a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:00.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.967+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8043020 con 0x7f10b4071e60 2024-04-17T08:31:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.967+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a803e450 con 0x7f10b4071e60 2024-04-17T08:31:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.967+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8009510 con 0x7f10b4071e60 2024-04-17T08:31:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.967+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b40b5af0 con 0x7f10b4071e60 2024-04-17T08:31:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.967+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b4164250 con 0x7f10b4071e60 2024-04-17T08:31:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.968+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b4068a10 con 0x7f10b4071e60 2024-04-17T08:31:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.969+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f10a8005d10 con 0x7f10b4071e60 2024-04-17T08:31:00.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.971+0000 7f10b1600640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 0x7f109806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:00.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.971+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f10a80ae880 con 0x7f10b4071e60 2024-04-17T08:31:00.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.972+0000 7f10b9600640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 0x7f109806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:00.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.973+0000 7f10b9600640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 0x7f109806ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10a4006fd0 tx=0x7f10a4008040 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:00.976+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f10a807cb80 con 0x7f10b4071e60 2024-04-17T08:31:01.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.214+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 --> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f10b4147160 con 0x7f109806ca40 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.219+0000 7f10b1600640 1 -- 172.21.15.67:0/1938988417 <== mgr.7465 v2:172.21.15.187:6832/659531354 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f10b4147160 con 0x7f109806ca40 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:01.218 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 msgr2=0x7f109806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 0x7f109806ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10a4006fd0 tx=0x7f10a4008040 comp rx=0 tx=0).stop 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 msgr2=0x7f10b4143a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f10a8000f80 tx=0x7f10a8004a40 comp rx=0 tx=0).stop 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 shutdown_connections 2024-04-17T08:31:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f109806ca40 0x7f109806ef00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10b414c1d0 0x7f10b4158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b4071e60 0x7f10b4143a40 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 --2- 172.21.15.67:0/1938988417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10b406f550 0x7f10b40b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 >> 172.21.15.67:0/1938988417 conn(0x7f10b406bef0 msgr2=0x7f10b40b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 shutdown_connections 2024-04-17T08:31:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.222+0000 7f10baa00640 1 -- 172.21.15.67:0/1938988417 wait complete. 2024-04-17T08:31:01.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:31:01.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.464+0000 7f5104a00640 1 Processor -- start 2024-04-17T08:31:01.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.464+0000 7f5104a00640 1 -- start start 2024-04-17T08:31:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f5104a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000cd920 0x7f510006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f5104a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f510006c0a0 0x7f510006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f5104a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5100146360 con 0x7f5100145f80 2024-04-17T08:31:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f5104a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f51000727a0 con 0x7f51000cd920 2024-04-17T08:31:01.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f5104a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5100071e70 con 0x7f510006c0a0 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f50fea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000cd920 0x7f510006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f50ff400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5100145f80 0x7f51000cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57394/0 (socket says 172.21.15.67:57394) 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f50ff400640 1 -- 172.21.15.67:0/3432328969 learned_addr learned my addr 172.21.15.67:0/3432328969 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.466+0000 7f50fea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51000cd920 0x7f510006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35684/0 (socket says 172.21.15.67:35684) 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.467+0000 7f50ffe00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f510006c0a0 0x7f510006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.467+0000 7f50fea00640 1 -- 172.21.15.67:0/3432328969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f510006c0a0 msgr2=0x7f510006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.467+0000 7f50fea00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f510006c0a0 0x7f510006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.467+0000 7f50fea00640 1 -- 172.21.15.67:0/3432328969 >> v1:172.21.15.67:6789/0 conn(0x7f5100145f80 legacy=0x7f51000cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.467+0000 7f50fea00640 1 -- 172.21.15.67:0/3432328969 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5100072150 con 0x7f51000cd920 2024-04-17T08:31:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.468+0000 7f50fea00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000cd920 0x7f510006bb60 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0097a0 tx=0x7f50ec02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78d69f16c1b045f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.468+0000 7f50fe000640 1 -- 172.21.15.67:0/3432328969 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec031780 con 0x7f51000cd920 2024-04-17T08:31:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.468+0000 7f50fe000640 1 -- 172.21.15.67:0/3432328969 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ec037070 con 0x7f51000cd920 2024-04-17T08:31:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.468+0000 7f50fe000640 1 -- 172.21.15.67:0/3432328969 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec031b40 con 0x7f51000cd920 2024-04-17T08:31:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 -- 172.21.15.67:0/3432328969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000cd920 msgr2=0x7f510006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000cd920 0x7f510006bb60 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0097a0 tx=0x7f50ec02b820 comp rx=0 tx=0).stop 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 -- 172.21.15.67:0/3432328969 shutdown_connections 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f510006c0a0 0x7f510006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 --2- 172.21.15.67:0/3432328969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51000cd920 0x7f510006bb60 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 -- 172.21.15.67:0/3432328969 >> 172.21.15.67:0/3432328969 conn(0x7f51000b2190 msgr2=0x7f51000b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 -- 172.21.15.67:0/3432328969 shutdown_connections 2024-04-17T08:31:01.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.469+0000 7f5104a00640 1 -- 172.21.15.67:0/3432328969 wait complete. 2024-04-17T08:31:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.471+0000 7f5104a00640 1 Processor -- start 2024-04-17T08:31:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.471+0000 7f5104a00640 1 -- start start 2024-04-17T08:31:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51000cd920 0x7f51000aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5100145f80 0x7f51001f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5100072c30 con 0x7f510006c0a0 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f510006c530 con 0x7f51000cd920 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f5104a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5100072db0 con 0x7f5100145f80 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f50ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f50ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35698/0 (socket says 172.21.15.67:35698) 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.472+0000 7f50ff400640 1 -- 172.21.15.67:0/3078047054 learned_addr learned my addr 172.21.15.67:0/3078047054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 -- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5100145f80 msgr2=0x7f51001f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5100145f80 0x7f51001f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 -- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51000cd920 msgr2=0x7f51000aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:01.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51000cd920 0x7f51000aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 -- 172.21.15.67:0/3078047054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51001f5140 con 0x7f510006c0a0 2024-04-17T08:31:01.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50ff400640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f50f000c3c0 tx=0x7f50f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0017070 con 0x7f510006c0a0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f0005ce0 con 0x7f510006c0a0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f00072f0 con 0x7f510006c0a0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.473+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51001f5420 con 0x7f510006c0a0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.474+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51001f81a0 con 0x7f510006c0a0 2024-04-17T08:31:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.475+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5100068a10 con 0x7f510006c0a0 2024-04-17T08:31:01.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.476+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 90) ==== 91142+0+0 (secure 0 0 0) 0x7f50f001a070 con 0x7f510006c0a0 2024-04-17T08:31:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.477+0000 7f50f7400640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 0x7f50e006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.477+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f50f008b080 con 0x7f510006c0a0 2024-04-17T08:31:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.478+0000 7f50fea00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 0x7f50e006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.479+0000 7f50fea00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 0x7f50e006ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f50ec036000 tx=0x7f50ec030000 comp rx=0 tx=0).ready entity=mgr.7465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:01.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.481+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f50f0059370 con 0x7f510006c0a0 2024-04-17T08:31:01.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:01.741+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f51000b1f70 con 0x7f510006c0a0 2024-04-17T08:31:02.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.215+0000 7f50f7400640 1 -- 172.21.15.67:0/3078047054 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) ==== 92+0+0 (secure 0 0 0) 0x7f50f005f070 con 0x7f510006c0a0 2024-04-17T08:31:02.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.218+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 msgr2=0x7f50e006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.218+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 0x7f50e006ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f50ec036000 tx=0x7f50ec030000 comp rx=0 tx=0).stop 2024-04-17T08:31:02.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.218+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 msgr2=0x7f51001f86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.218+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f50f000c3c0 tx=0x7f50f000c880 comp rx=0 tx=0).stop 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 shutdown_connections 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f50e006ca40 0x7f50e006ef00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5100145f80 0x7f51001f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51000cd920 0x7f51000aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 --2- 172.21.15.67:0/3078047054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f510006c0a0 0x7f51001f86c0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.219+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 >> 172.21.15.67:0/3078047054 conn(0x7f51000b2190 msgr2=0x7f51000b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.220+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 shutdown_connections 2024-04-17T08:31:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.220+0000 7f5104a00640 1 -- 172.21.15.67:0/3078047054 wait complete. 2024-04-17T08:31:02.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:31:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.446+0000 7f6411600640 1 Processor -- start 2024-04-17T08:31:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.446+0000 7f6411600640 1 -- start start 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6411600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f640c0aff10 0x7f640c06c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6411600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f640c0b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6411600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f640c0b3cc0 con 0x7f640c0ae790 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6411600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f640c15c3a0 con 0x7f640c0aff10 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6411600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f640c15c680 con 0x7f640c06ce10 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f640be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f640c0ae790 0x7f640c15bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57396/0 (socket says 172.21.15.67:57396) 2024-04-17T08:31:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f640be00640 1 -- 172.21.15.67:0/3189920514 learned_addr learned my addr 172.21.15.67:0/3189920514 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f6410c00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f640c0b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.448+0000 7f640b400640 1 --2- 172.21.15.67:0/3189920514 >> v2:172.21.15.187:3300/0 conn(0x7f640c0aff10 0x7f640c06c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572885436 0 0) 0x7f640c0b3cc0 con 0x7f640c0ae790 2024-04-17T08:31:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63f0003420 con 0x7f640c0ae790 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 719151060 0 0) 0x7f63f0003420 con 0x7f640c0ae790 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 msgr2=0x7f640c0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f640c0b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 >> v2:172.21.15.187:3300/0 conn(0x7f640c0aff10 msgr2=0x7f640c06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 --2- 172.21.15.67:0/3189920514 >> v2:172.21.15.187:3300/0 conn(0x7f640c0aff10 0x7f640c06c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c15c960 con 0x7f640c0ae790 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f63fc001670 con 0x7f640c0ae790 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 >> v1:172.21.15.67:6789/0 conn(0x7f640c0ae790 legacy=0x7f640c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f0004a90 0x7f63f0004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f00036a0 0x7f63f0003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f63f0004110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f640c15c3a0 con 0x7f63f00036a0 2024-04-17T08:31:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f640c15c680 con 0x7f63f0004a90 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.449+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63f000fdb0 con 0x7f640c06ce10 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f00036a0 0x7f63f0003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640be00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f0004a90 0x7f63f0004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f6410c00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f63f0004110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 -- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 msgr2=0x7f63f0004110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f63f0004110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 -- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f0004a90 msgr2=0x7f63f0004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f0004a90 0x7f63f0004eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f640b400640 1 -- 172.21.15.67:0/3189920514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f0003420 con 0x7f63f00036a0 2024-04-17T08:31:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.450+0000 7f6410c00640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c06ce10 0x7f63f0004110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.451+0000 7f640b400640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f00036a0 0x7f63f0003b20 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009ac0 tx=0x7f63f0004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3c45f8613bb821 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8039070 con 0x7f63f00036a0 2024-04-17T08:31:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f802c710 con 0x7f63f00036a0 2024-04-17T08:31:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f640aa00640 1 -- 172.21.15.67:0/3189920514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8009690 con 0x7f63f00036a0 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6411600640 1 -- 172.21.15.67:0/3189920514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f00036a0 msgr2=0x7f63f0003b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6411600640 1 --2- 172.21.15.67:0/3189920514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f00036a0 0x7f63f0003b20 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009ac0 tx=0x7f63f0004720 comp rx=0 tx=0).stop 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6410c00640 1 -- 172.21.15.67:0/3189920514 reap_dead start 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6411600640 1 -- 172.21.15.67:0/3189920514 shutdown_connections 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6411600640 1 -- 172.21.15.67:0/3189920514 >> 172.21.15.67:0/3189920514 conn(0x7f640c072460 msgr2=0x7f640c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.452+0000 7f6411600640 1 -- 172.21.15.67:0/3189920514 shutdown_connections 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.453+0000 7f6411600640 1 -- 172.21.15.67:0/3189920514 wait complete. 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.454+0000 7f6411600640 1 Processor -- start 2024-04-17T08:31:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.454+0000 7f6411600640 1 -- start start 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f00036a0 0x7f640c139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 0x7f640c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f640c06e480 con 0x7f640c09cd10 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f640c06e300 con 0x7f63f00036a0 2024-04-17T08:31:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.455+0000 7f6411600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f640c06e600 con 0x7f640c158980 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35722/0 (socket says 172.21.15.67:35722) 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 -- 172.21.15.67:0/3153104605 learned_addr learned my addr 172.21.15.67:0/3153104605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f640be00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f00036a0 0x7f640c139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f640b400640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 0x7f640c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 msgr2=0x7f640c158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 0x7f640c158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f00036a0 msgr2=0x7f640c139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f00036a0 0x7f640c139340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f6410c00640 1 -- 172.21.15.67:0/3153104605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640c09da80 con 0x7f640c09cd10 2024-04-17T08:31:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.456+0000 7f640b400640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 0x7f640c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6410c00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005600 tx=0x7f640400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640400fd80 con 0x7f640c09cd10 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6404003990 con 0x7f640c09cd10 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404003d60 con 0x7f640c09cd10 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640c09dd60 con 0x7f640c09cd10 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.457+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640c09e320 con 0x7f640c09cd10 2024-04-17T08:31:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.458+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640c0755d0 con 0x7f640c09cd10 2024-04-17T08:31:02.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.459+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 91) ==== 91159+0+0 (secure 0 0 0) 0x7f6404003f00 con 0x7f640c09cd10 2024-04-17T08:31:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.460+0000 7f6403e00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 0x7f63ec06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.460+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f640408c080 con 0x7f640c09cd10 2024-04-17T08:31:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.460+0000 7f640be00640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 msgr2=0x7f63ec06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:02.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.460+0000 7f640be00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 0x7f63ec06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:02.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.465+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6404012070 con 0x7f640c09cd10 2024-04-17T08:31:02.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.661+0000 7f640be00640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 msgr2=0x7f63ec06ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:02.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.661+0000 7f640be00640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 0x7f63ec06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.665+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f640c072b20 con 0x7f640c09cd10 2024-04-17T08:31:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.670+0000 7f6403e00640 1 -- 172.21.15.67:0/3153104605 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) ==== 138+0+0 (secure 0 0 0) 0x7f640405a1d0 con 0x7f640c09cd10 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.672+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 msgr2=0x7f63ec06ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 0x7f63ec06ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 msgr2=0x7f640c09d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005600 tx=0x7f640400f4a0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 shutdown_connections 2024-04-17T08:31:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f63ec06ca90 0x7f63ec06ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640c09cd10 0x7f640c09d190 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f640c158980 0x7f640c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 --2- 172.21.15.67:0/3153104605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f00036a0 0x7f640c139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 >> 172.21.15.67:0/3153104605 conn(0x7f640c072460 msgr2=0x7f640c15cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 shutdown_connections 2024-04-17T08:31:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.673+0000 7f6411600640 1 -- 172.21.15.67:0/3153104605 wait complete. 2024-04-17T08:31:02.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:31:02.800 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:02.801+0000 7f8a0062c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:02.815 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:02.816+0000 7fcef6a2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:02.859 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:02.860+0000 7f8a0062c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:02.873 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:02.874+0000 7fcef6a2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:02.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.900+0000 7f4a4f400640 1 Processor -- start 2024-04-17T08:31:02.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.901+0000 7f4a4f400640 1 -- start start 2024-04-17T08:31:02.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.902+0000 7f4a4f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a4806b8c0 0x7f4a4806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.902+0000 7f4a4f400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4a481627b0 0x7f4a480cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.902+0000 7f4a4f400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a48163710 con 0x7f4a480d0b50 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.902+0000 7f4a4f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4815f360 con 0x7f4a4806b8c0 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.902+0000 7f4a4f400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4806c1e0 con 0x7f4a481627b0 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a4806b8c0 0x7f4a4806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a4806b8c0 0x7f4a4806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35734/0 (socket says 172.21.15.67:35734) 2024-04-17T08:31:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 -- 172.21.15.67:0/3705490548 learned_addr learned my addr 172.21.15.67:0/3705490548 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4d400640 1 --2- 172.21.15.67:0/3705490548 >> v2:172.21.15.187:3300/0 conn(0x7f4a481627b0 0x7f4a480cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 -- 172.21.15.67:0/3705490548 >> v2:172.21.15.187:3300/0 conn(0x7f4a481627b0 msgr2=0x7f4a480cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 --2- 172.21.15.67:0/3705490548 >> v2:172.21.15.187:3300/0 conn(0x7f4a481627b0 0x7f4a480cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 -- 172.21.15.67:0/3705490548 >> v1:172.21.15.187:6789/0 conn(0x7f4a480d0b50 legacy=0x7f4a4815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.903+0000 7f4a4ca00640 1 -- 172.21.15.67:0/3705490548 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a48163150 con 0x7f4a4806b8c0 2024-04-17T08:31:02.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.905+0000 7f4a4ca00640 1 --2- 172.21.15.67:0/3705490548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a4806b8c0 0x7f4a4806bca0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f4a480d01e0 tx=0x7f4a3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645b3826b01b9dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:02.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.905+0000 7f4a47400640 1 -- 172.21.15.67:0/3705490548 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c035020 con 0x7f4a4806b8c0 2024-04-17T08:31:02.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.905+0000 7f4a47400640 1 -- 172.21.15.67:0/3705490548 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c031870 con 0x7f4a4806b8c0 2024-04-17T08:31:02.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.905+0000 7f4a47400640 1 -- 172.21.15.67:0/3705490548 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c031c40 con 0x7f4a4806b8c0 2024-04-17T08:31:02.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 -- 172.21.15.67:0/3705490548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a4806b8c0 msgr2=0x7f4a4806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 --2- 172.21.15.67:0/3705490548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a4806b8c0 0x7f4a4806bca0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f4a480d01e0 tx=0x7f4a3c02b910 comp rx=0 tx=0).stop 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 -- 172.21.15.67:0/3705490548 shutdown_connections 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 --2- 172.21.15.67:0/3705490548 >> v2:172.21.15.187:3300/0 conn(0x7f4a481627b0 0x7f4a480cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 --2- 172.21.15.67:0/3705490548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a4806b8c0 0x7f4a4806bca0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 -- 172.21.15.67:0/3705490548 >> 172.21.15.67:0/3705490548 conn(0x7f4a48073360 msgr2=0x7f4a4809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 -- 172.21.15.67:0/3705490548 shutdown_connections 2024-04-17T08:31:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.906+0000 7f4a4f400640 1 -- 172.21.15.67:0/3705490548 wait complete. 2024-04-17T08:31:02.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.908+0000 7f4a4f400640 1 Processor -- start 2024-04-17T08:31:02.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.909+0000 7f4a4f400640 1 -- start start 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4806b8c0 0x7f4a480b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 0x7f4a480c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4815eef0 con 0x7f4a481627b0 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4815f600 con 0x7f4a4806b8c0 2024-04-17T08:31:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4f400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4815f070 con 0x7f4a480d0b50 2024-04-17T08:31:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35746/0 (socket says 172.21.15.67:35746) 2024-04-17T08:31:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4d400640 1 -- 172.21.15.67:0/788873099 learned_addr learned my addr 172.21.15.67:0/788873099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a4ca00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4806b8c0 0x7f4a480b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.910+0000 7f4a47e00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 0x7f4a480c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 msgr2=0x7f4a480c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 0x7f4a480c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4806b8c0 msgr2=0x7f4a480b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4806b8c0 0x7f4a480b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 -- 172.21.15.67:0/788873099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a480c70b0 con 0x7f4a481627b0 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a47e00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 0x7f4a480c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a4d400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f4a40007e90 tx=0x7f4a4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40014420 con 0x7f4a481627b0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.911+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a40019070 con 0x7f4a481627b0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.912+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40014800 con 0x7f4a481627b0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.912+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a480c7390 con 0x7f4a481627b0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.912+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a480c7950 con 0x7f4a481627b0 2024-04-17T08:31:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.913+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a48068ca0 con 0x7f4a481627b0 2024-04-17T08:31:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.914+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 91) ==== 91159+0+0 (secure 0 0 0) 0x7f4a400149a0 con 0x7f4a481627b0 2024-04-17T08:31:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.915+0000 7f4a44c00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 0x7f4a2806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.915+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7f4a4008c450 con 0x7f4a481627b0 2024-04-17T08:31:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.915+0000 7f4a4ca00640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 msgr2=0x7f4a2806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.915+0000 7f4a4ca00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 0x7f4a2806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:02.919+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a4005a7e0 con 0x7f4a481627b0 2024-04-17T08:31:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.116+0000 7f4a4ca00640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 msgr2=0x7f4a2806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.116+0000 7f4a4ca00640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 0x7f4a2806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.125+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f4a4806c700 con 0x7f4a481627b0 2024-04-17T08:31:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.130+0000 7f4a44c00640 1 -- 172.21.15.67:0/788873099 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) ==== 133+0+0 (secure 0 0 0) 0x7f4a4005a0e0 con 0x7f4a481627b0 2024-04-17T08:31:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.132+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 msgr2=0x7f4a2806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.132+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 0x7f4a2806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.132+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 msgr2=0x7f4a481f5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.132+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f4a40007e90 tx=0x7f4a4000e4b0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 shutdown_connections 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7f4a2806ca60 0x7f4a2806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a481627b0 0x7f4a481f5c40 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a480d0b50 0x7f4a480c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 --2- 172.21.15.67:0/788873099 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a4806b8c0 0x7f4a480b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 >> 172.21.15.67:0/788873099 conn(0x7f4a48073360 msgr2=0x7f4a480ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 shutdown_connections 2024-04-17T08:31:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.133+0000 7f4a4f400640 1 -- 172.21.15.67:0/788873099 wait complete. 2024-04-17T08:31:03.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:03.150 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T08:31:03.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-04-17T08:31:03.318 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.319+0000 7fcef6a2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:03.337 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.338+0000 7f8a0062c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:03.371 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.371+0000 7fcef6a2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:03.393 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.394+0000 7f8a0062c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:03.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.408+0000 7fcc5a800640 1 Processor -- start 2024-04-17T08:31:03.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.408+0000 7fcc5a800640 1 -- start start 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 0x7fcc54161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc5a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc540d0980 0x7fcc540d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc5a800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc5415d800 con 0x7fcc5415cfc0 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc5a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcc5406b8c0 con 0x7fcc540d0980 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc5a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5406bba0 con 0x7fcc54160c30 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc540d0980 0x7fcc540d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:03.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc540d0980 0x7fcc540d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48692/0 (socket says 172.21.15.67:48692) 2024-04-17T08:31:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc53e00640 1 -- 172.21.15.67:0/3217695191 learned_addr learned my addr 172.21.15.67:0/3217695191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.410+0000 7fcc53400640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 0x7fcc54161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:03.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.411+0000 7fcc53e00640 1 -- 172.21.15.67:0/3217695191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 msgr2=0x7fcc54161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:03.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.411+0000 7fcc53e00640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 0x7fcc54161010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.411+0000 7fcc53e00640 1 -- 172.21.15.67:0/3217695191 >> v1:172.21.15.187:6789/0 conn(0x7fcc5415cfc0 legacy=0x7fcc5406c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:03.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.411+0000 7fcc53e00640 1 -- 172.21.15.67:0/3217695191 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5415dac0 con 0x7fcc540d0980 2024-04-17T08:31:03.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.411+0000 7fcc53400640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 0x7fcc54161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.412+0000 7fcc53e00640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc540d0980 0x7fcc540d0e10 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009e50 tx=0x7fcc4c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a4a0f10df43893 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.412+0000 7fcc52a00640 1 -- 172.21.15.67:0/3217695191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c02cb90 con 0x7fcc540d0980 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.412+0000 7fcc52a00640 1 -- 172.21.15.67:0/3217695191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c02cd30 con 0x7fcc540d0980 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.412+0000 7fcc52a00640 1 -- 172.21.15.67:0/3217695191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c034610 con 0x7fcc540d0980 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 -- 172.21.15.67:0/3217695191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc540d0980 msgr2=0x7fcc540d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc540d0980 0x7fcc540d0e10 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009e50 tx=0x7fcc4c02bff0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 -- 172.21.15.67:0/3217695191 shutdown_connections 2024-04-17T08:31:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc540d0980 0x7fcc540d0e10 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 --2- 172.21.15.67:0/3217695191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc54160c30 0x7fcc54161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 -- 172.21.15.67:0/3217695191 >> 172.21.15.67:0/3217695191 conn(0x7fcc54073360 msgr2=0x7fcc5409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 -- 172.21.15.67:0/3217695191 shutdown_connections 2024-04-17T08:31:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.413+0000 7fcc5a800640 1 -- 172.21.15.67:0/3217695191 wait complete. 2024-04-17T08:31:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.415+0000 7fcc5a800640 1 Processor -- start 2024-04-17T08:31:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.415+0000 7fcc5a800640 1 -- start start 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc540d0980 0x7fcc5414d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc5415cfc0 0x7fcc5415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 0x7fcc5415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5415d3a0 con 0x7fcc54160c30 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5406c740 con 0x7fcc540d0980 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc5a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5415d520 con 0x7fcc5415cfc0 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 0x7fcc5415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 0x7fcc5415bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48704/0 (socket says 172.21.15.67:48704) 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.416+0000 7fcc53e00640 1 -- 172.21.15.67:0/2168428784 learned_addr learned my addr 172.21.15.67:0/2168428784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc540d0980 0x7fcc5414d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc4be00640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc5415cfc0 0x7fcc5415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc5415cfc0 msgr2=0x7fcc5415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcc5415cfc0 0x7fcc5415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc540d0980 msgr2=0x7fcc5414d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc540d0980 0x7fcc5414d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc540af660 con 0x7fcc54160c30 2024-04-17T08:31:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcc540d0980 0x7fcc5414d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc53e00640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 0x7fcc5415bf90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009e50 tx=0x7fcc4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.417+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c043020 con 0x7fcc54160c30 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.418+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c03e450 con 0x7fcc54160c30 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.418+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc540af940 con 0x7fcc54160c30 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.418+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c009510 con 0x7fcc54160c30 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.418+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc540aff00 con 0x7fcc54160c30 2024-04-17T08:31:03.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.419+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc54068ca0 con 0x7fcc54160c30 2024-04-17T08:31:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.420+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 91) ==== 91159+0+0 (secure 0 0 0) 0x7fcc4c046020 con 0x7fcc54160c30 2024-04-17T08:31:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.420+0000 7fcc4b400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.421+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 5737+0+0 (secure 0 0 0) 0x7fcc4c0ae640 con 0x7fcc54160c30 2024-04-17T08:31:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.421+0000 7fcc53400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.421+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:03.422 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.423+0000 7fcef6a2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:03.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.425+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcc4c04d030 con 0x7fcc54160c30 2024-04-17T08:31:03.449 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.450+0000 7f8a0062c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:03.484 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.485+0000 7fcef6a2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:03.515 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.516+0000 7f8a0062c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:03.534 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.535+0000 7fcef6a2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:03.567 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.568+0000 7f8a0062c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:03.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.622+0000 7fcc53400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:03.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.622+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:03.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:03.644+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7fcc540af410 con 0x7fcc3006ca60 2024-04-17T08:31:03.663 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.664+0000 7fcef6a2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:03.703 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.704+0000 7f8a0062c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:03.726 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.727+0000 7fcef6a2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:03.769 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.770+0000 7f8a0062c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:03.778 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.778+0000 7fcef6a2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:03.821 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.822+0000 7f8a0062c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:03.835 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.835+0000 7fcef6a2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:03.881 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.882+0000 7f8a0062c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:03.886 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:03.886+0000 7fcef6a2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:03.934 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:03.935+0000 7f8a0062c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:04.023+0000 7fcc53400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:04.023+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:31:04.270 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.270+0000 7fcef6a2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:04.339 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.339+0000 7f8a0062c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:04.428 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.429+0000 7fcef6a2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:04.508 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.508+0000 7f8a0062c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:04.538 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.539+0000 7fcef6a2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:04.589 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.589+0000 7fcef6a2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:04.621 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.622+0000 7f8a0062c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:04.674 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.675+0000 7f8a0062c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:04.714 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.715+0000 7fcef6a2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:04.806 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.807+0000 7f8a0062c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:04.820 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.821+0000 7fcef6a2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:04.824+0000 7fcc53400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:04.824+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:31:04.919 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:04.919+0000 7f8a0062c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:04.996 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:04.996+0000 7fcef6a2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:05.045 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:05.045+0000 7fcef6a2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:05.102 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:05.102+0000 7f8a0062c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:05.153 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:05.154+0000 7f8a0062c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:05.180 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:05.181+0000 7fcef6a2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:05.297 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:05.297+0000 7f8a0062c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:05.989 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:05.989+0000 7fcef6a2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:06.195 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:06.196+0000 7f8a0062c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:06.290 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.290+0000 7fcef6a2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:06.347 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.348+0000 7fcef6a2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:06.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:06.426+0000 7fcc53400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/659531354 2024-04-17T08:31:06.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:06.426+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T08:31:06.478 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.479+0000 7fcef6a2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:06.500+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 92) ==== 90768+0+0 (secure 0 0 0) 0x7fcc4c078810 con 0x7fcc54160c30 2024-04-17T08:31:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:06.501+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 msgr2=0x7fcc3006ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:06.501+0000 7fcc4b400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/659531354,v1:172.21.15.187:6833/659531354] conn(0x7fcc3006ca60 0x7fcc3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:06.522 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:06.523+0000 7f8a0062c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:06.592 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:06.592+0000 7f8a0062c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:06.755 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:06.756+0000 7f8a0062c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.836+0000 7fce69200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.836+0000 7fce69200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.836+0000 7fce69200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.836+0000 7fce69200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.836+0000 7fce69200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.837+0000 7fce6ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.837+0000 7fce6ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.837+0000 7fce6ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.837+0000 7fce6ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:06.836 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:06.837+0000 7fce6ce00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.514+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 93) ==== 90895+0+0 (secure 0 0 0) 0x7fcc4c0a6300 con 0x7fcc54160c30 2024-04-17T08:31:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.515+0000 7fcc4b400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fcc30074970 0x7fcc30076d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.515+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 --> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7fcc4c008ea0 con 0x7fcc30074970 2024-04-17T08:31:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.527+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fcc30074970 0x7fcc30076d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.537+0000 7fcc53400640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fcc30074970 0x7fcc30076d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc44004690 tx=0x7fcc44008040 comp rx=0 tx=0).ready entity=mgr.7729 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:07.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.743+0000 7fcc4b400640 1 -- 172.21.15.67:0/2168428784 <== mgr.7729 v2:172.21.15.187:6832/2551862940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc4c008ea0 con 0x7fcc30074970 2024-04-17T08:31:07.742 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T08:31:07.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fcc30074970 msgr2=0x7fcc30076d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fcc30074970 0x7fcc30076d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcc44004690 tx=0x7fcc44008040 comp rx=0 tx=0).stop 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 msgr2=0x7fcc5415bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 --2- 172.21.15.67:0/2168428784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc54160c30 0x7fcc5415bf90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009e50 tx=0x7fcc4c004a40 comp rx=0 tx=0).stop 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc53e00640 1 -- 172.21.15.67:0/2168428784 reap_dead start 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 shutdown_connections 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.746+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 >> 172.21.15.67:0/2168428784 conn(0x7fcc54073360 msgr2=0x7fcc541557a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.747+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 shutdown_connections 2024-04-17T08:31:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.747+0000 7fcc5a800640 1 -- 172.21.15.67:0/2168428784 wait complete. 2024-04-17T08:31:07.772 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T08:31:07.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-17T08:31:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.999+0000 7f025fc00640 1 Processor -- start 2024-04-17T08:31:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:07.999+0000 7f025fc00640 1 -- start start 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.000+0000 7f025fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f0258071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025fc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02580ce3b0 0x7f02580cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025fc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02580cea70 con 0x7f0258145ac0 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025fc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02580ce840 con 0x7f02580ce3b0 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02580ced50 con 0x7f0258072d80 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025d200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0258145ac0 0x7f0258072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40764/0 (socket says 172.21.15.67:40764) 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025d200640 1 -- 172.21.15.67:0/375467087 learned_addr learned my addr 172.21.15.67:0/375467087 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:08.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025dc00640 1 --2- 172.21.15.67:0/375467087 >> v2:172.21.15.67:3300/0 conn(0x7f02580ce3b0 0x7f02580cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025dc00640 1 -- 172.21.15.67:0/375467087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 msgr2=0x7f0258071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025dc00640 1 --2- 172.21.15.67:0/375467087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f0258071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025dc00640 1 -- 172.21.15.67:0/375467087 >> v1:172.21.15.67:6789/0 conn(0x7f0258145ac0 legacy=0x7f0258072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.001+0000 7f025dc00640 1 -- 172.21.15.67:0/375467087 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02580cf030 con 0x7f02580ce3b0 2024-04-17T08:31:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f025dc00640 1 --2- 172.21.15.67:0/375467087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cfb10 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0250009d90 tx=0x7f025002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1e8cccc2e1f8c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f0257400640 1 -- 172.21.15.67:0/375467087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250039070 con 0x7f02580ce3b0 2024-04-17T08:31:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f0257400640 1 -- 172.21.15.67:0/375467087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025002cc60 con 0x7f02580ce3b0 2024-04-17T08:31:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f0257400640 1 -- 172.21.15.67:0/375467087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250034660 con 0x7f02580ce3b0 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f025fc00640 1 -- 172.21.15.67:0/375467087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 msgr2=0x7f02580cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.003+0000 7f025fc00640 1 --2- 172.21.15.67:0/375467087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cfb10 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0250009d90 tx=0x7f025002beb0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 -- 172.21.15.67:0/375467087 shutdown_connections 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 --2- 172.21.15.67:0/375467087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cfb10 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 --2- 172.21.15.67:0/375467087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f0258071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 -- 172.21.15.67:0/375467087 >> 172.21.15.67:0/375467087 conn(0x7f0258067ee0 msgr2=0x7f025806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 -- 172.21.15.67:0/375467087 shutdown_connections 2024-04-17T08:31:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.004+0000 7f025fc00640 1 -- 172.21.15.67:0/375467087 wait complete. 2024-04-17T08:31:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.005+0000 7f025fc00640 1 Processor -- start 2024-04-17T08:31:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.006+0000 7f025fc00640 1 -- start start 2024-04-17T08:31:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f02581534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0258145ac0 0x7f02580cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0258145ea0 con 0x7f02580ce3b0 2024-04-17T08:31:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f02580cf5f0 con 0x7f0258145ac0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02580d0050 con 0x7f0258072d80 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48734/0 (socket says 172.21.15.67:48734) 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 -- 172.21.15.67:0/1434070397 learned_addr learned my addr 172.21.15.67:0/1434070397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025d200640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f02581534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 -- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 msgr2=0x7f02581534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f025dc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0258145ac0 0x7f02580cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f02581534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.007+0000 7f0257e00640 1 -- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0258145ac0 msgr2=0x7f02580cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0257e00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0258145ac0 0x7f02580cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0257e00640 1 -- 172.21.15.67:0/1434070397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02580c98d0 con 0x7f02580ce3b0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0257e00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f024c00b4d0 tx=0x7f024c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c014070 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f024c010870 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02580c9bb0 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c010c60 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.008+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02580ca170 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.010+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0258068ca0 con 0x7f02580ce3b0 2024-04-17T08:31:08.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.011+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 93) ==== 90895+0+0 (secure 0 0 0) 0x7f024c017070 con 0x7f02580ce3b0 2024-04-17T08:31:08.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.012+0000 7f0254c00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 0x7f023c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.012+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f024c08b750 con 0x7f02580ce3b0 2024-04-17T08:31:08.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.013+0000 7f025d200640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 0x7f023c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.014+0000 7f025d200640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 0x7f023c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f60 tx=0x7f0248009040 comp rx=0 tx=0).ready entity=mgr.7729 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.017+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f024c059d40 con 0x7f02580ce3b0 2024-04-17T08:31:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.254+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 --> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f0258141330 con 0x7f023c06c790 2024-04-17T08:31:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.438+0000 7f0254c00640 1 -- 172.21.15.67:0/1434070397 <== mgr.7729 v2:172.21.15.187:6832/2551862940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f0258141330 con 0x7f023c06c790 2024-04-17T08:31:08.438 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path snap_test_dir1 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 msgr2=0x7f023c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 0x7f023c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0248000f60 tx=0x7f0248009040 comp rx=0 tx=0).stop 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 msgr2=0x7f02580cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f024c00b4d0 tx=0x7f024c00b990 comp rx=0 tx=0).stop 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 shutdown_connections 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0258145ac0 0x7f02580cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.442+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02580ce3b0 0x7f02580cd110 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.443+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f023c06c790 0x7f023c06ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.443+0000 7f025fc00640 1 --2- 172.21.15.67:0/1434070397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0258072d80 0x7f02581534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.443+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 >> 172.21.15.67:0/1434070397 conn(0x7f0258067ee0 msgr2=0x7f02581440c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.443+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 shutdown_connections 2024-04-17T08:31:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.443+0000 7f025fc00640 1 -- 172.21.15.67:0/1434070397 wait complete. 2024-04-17T08:31:08.465 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-17T08:31:08.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-04-17T08:31:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.697+0000 7f72efe00640 1 Processor -- start 2024-04-17T08:31:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.697+0000 7f72efe00640 1 -- start start 2024-04-17T08:31:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.698+0000 7f72efe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72e80b17c0 0x7f72e80b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.698+0000 7f72efe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.698+0000 7f72efe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e80b20e0 con 0x7f72e80b4fa0 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.698+0000 7f72efe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f72e80ce920 con 0x7f72e80b17c0 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.698+0000 7f72efe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72e80b5560 con 0x7f72e80cfd00 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.699+0000 7f72ed400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72e80b17c0 0x7f72e80b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.699+0000 7f72ede00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.699+0000 7f72ed400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72e80b17c0 0x7f72e80b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:46042/0 (socket says 172.21.15.67:46042) 2024-04-17T08:31:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.699+0000 7f72ed400640 1 -- 172.21.15.67:0/850756588 learned_addr learned my addr 172.21.15.67:0/850756588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.700+0000 7f72ed400640 1 -- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 msgr2=0x7f72e80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.700+0000 7f72ed400640 1 --2- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e80cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.700+0000 7f72ed400640 1 -- 172.21.15.67:0/850756588 >> v1:172.21.15.187:6789/0 conn(0x7f72e80b4fa0 legacy=0x7f72e815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.700+0000 7f72ed400640 1 -- 172.21.15.67:0/850756588 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e80b5840 con 0x7f72e80b17c0 2024-04-17T08:31:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.702+0000 7f72ed400640 1 --2- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e80b1ba0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009870 tx=0x7f72dc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b911f869b152d7ab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.702+0000 7f72e7e00640 1 -- 172.21.15.67:0/850756588 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc035020 con 0x7f72e80b17c0 2024-04-17T08:31:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.702+0000 7f72e7e00640 1 -- 172.21.15.67:0/850756588 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72dc031640 con 0x7f72e80b17c0 2024-04-17T08:31:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.702+0000 7f72e7e00640 1 -- 172.21.15.67:0/850756588 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc02d040 con 0x7f72e80b17c0 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 -- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 msgr2=0x7f72e80b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 --2- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e80b1ba0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009870 tx=0x7f72dc02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 -- 172.21.15.67:0/850756588 shutdown_connections 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 --2- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e80cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 --2- 172.21.15.67:0/850756588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e80b1ba0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 -- 172.21.15.67:0/850756588 >> 172.21.15.67:0/850756588 conn(0x7f72e806b8c0 msgr2=0x7f72e806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 -- 172.21.15.67:0/850756588 shutdown_connections 2024-04-17T08:31:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.703+0000 7f72efe00640 1 -- 172.21.15.67:0/850756588 wait complete. 2024-04-17T08:31:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.705+0000 7f72efe00640 1 Processor -- start 2024-04-17T08:31:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.705+0000 7f72efe00640 1 -- start start 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e812dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80b4fa0 0x7f72e8143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e81442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8145490 con 0x7f72e80b4fa0 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f72e80b5b20 con 0x7f72e80b17c0 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72efe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72e8145610 con 0x7f72e80cfd00 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72eca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80b4fa0 0x7f72e8143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72eca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80b4fa0 0x7f72e8143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48738/0 (socket says 172.21.15.67:48738) 2024-04-17T08:31:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.706+0000 7f72eca00640 1 -- 172.21.15.67:0/2736058440 learned_addr learned my addr 172.21.15.67:0/2736058440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72ed400640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e812dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72ede00640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e81442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 -- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 msgr2=0x7f72e81442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cfd00 0x7f72e81442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 -- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 msgr2=0x7f72e812dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e812dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 -- 172.21.15.67:0/2736058440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e812e280 con 0x7f72e80b4fa0 2024-04-17T08:31:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72ed400640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e80b17c0 0x7f72e812dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.707+0000 7f72eca00640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80b4fa0 0x7f72e8143da0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c430 tx=0x7f72d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.708+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4011790 con 0x7f72e80b4fa0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.708+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d4017070 con 0x7f72e80b4fa0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.708+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4011bb0 con 0x7f72e80b4fa0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.708+0000 7f72efe00640 1 -- 172.21.15.67:0/2736058440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e80c65d0 con 0x7f72e80b4fa0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.708+0000 7f72efe00640 1 -- 172.21.15.67:0/2736058440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e80c6b90 con 0x7f72e80b4fa0 2024-04-17T08:31:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.709+0000 7f72efe00640 1 -- 172.21.15.67:0/2736058440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e80d0190 con 0x7f72e80b4fa0 2024-04-17T08:31:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.710+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 94) ==== 91160+0+0 (secure 0 0 0) 0x7f72d401c460 con 0x7f72e80b4fa0 2024-04-17T08:31:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.711+0000 7f72e5600640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f72c806ca90 0x7f72c806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.711+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f72d408bd10 con 0x7f72e80b4fa0 2024-04-17T08:31:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.712+0000 7f72ed400640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f72c806ca90 0x7f72c806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.713+0000 7f72ed400640 1 --2- 172.21.15.67:0/2736058440 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f72c806ca90 0x7f72c806ef50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72dc036000 tx=0x7f72dc03c040 comp rx=0 tx=0).ready entity=mgr.7729 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.716+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f72e80d0190 con 0x7f72e80b4fa0 2024-04-17T08:31:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.942+0000 7f72efe00640 1 -- 172.21.15.67:0/2736058440 --> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f72e806be20 con 0x7f72c806ca90 2024-04-17T08:31:08.957 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:08.958+0000 7fce75a00640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-04-17T08:31:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:08.958+0000 7f72e5600640 1 -- 172.21.15.67:0/2736058440 <== mgr.7729 v2:172.21.15.187:6832/2551862940 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f72e806be20 con 0x7f72c806ca90 2024-04-17T08:31:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-04-17T08:31:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T08:31:08.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:08.972 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T08:31:09.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:31:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.251+0000 7f8d66e00640 1 Processor -- start 2024-04-17T08:31:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.251+0000 7f8d66e00640 1 -- start start 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.252+0000 7f8d66e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8d6015b6b0 0x7f8d6006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.252+0000 7f8d66e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d600cfc10 0x7f8d6006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d6006df40 con 0x7f8d6006f800 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8d6006ed00 con 0x7f8d600cfc10 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8d6006e870 con 0x7f8d6015b6b0 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d600cfc10 0x7f8d6006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d600cfc10 0x7f8d6006be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48746/0 (socket says 172.21.15.67:48746) 2024-04-17T08:31:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 -- 172.21.15.67:0/3175951058 learned_addr learned my addr 172.21.15.67:0/3175951058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d65000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d6006f800 0x7f8d6015f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40774/0 (socket says 172.21.15.67:40774) 2024-04-17T08:31:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d65a00640 1 --2- 172.21.15.67:0/3175951058 >> v2:172.21.15.187:3300/0 conn(0x7f8d6015b6b0 0x7f8d6006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 -- 172.21.15.67:0/3175951058 >> v2:172.21.15.187:3300/0 conn(0x7f8d6015b6b0 msgr2=0x7f8d6006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 --2- 172.21.15.67:0/3175951058 >> v2:172.21.15.187:3300/0 conn(0x7f8d6015b6b0 0x7f8d6006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.253+0000 7f8d66400640 1 -- 172.21.15.67:0/3175951058 >> v1:172.21.15.67:6789/0 conn(0x7f8d6006f800 legacy=0x7f8d6015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.254+0000 7f8d66400640 1 -- 172.21.15.67:0/3175951058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6006e220 con 0x7f8d600cfc10 2024-04-17T08:31:09.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.254+0000 7f8d5fe00640 1 -- 172.21.15.67:0/3175951058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886530357 0 0) 0x7f8d6006df40 con 0x7f8d6006f800 2024-04-17T08:31:09.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d66400640 1 --2- 172.21.15.67:0/3175951058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6006be00 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8d60072ea0 tx=0x7f8d5802e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fc60754e50b803 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d5fe00640 1 -- 172.21.15.67:0/3175951058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5802ea90 con 0x7f8d600cfc10 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d5fe00640 1 -- 172.21.15.67:0/3175951058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5802ec30 con 0x7f8d600cfc10 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d5fe00640 1 -- 172.21.15.67:0/3175951058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58009600 con 0x7f8d600cfc10 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d66e00640 1 -- 172.21.15.67:0/3175951058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 msgr2=0x7f8d6006be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.255+0000 7f8d66e00640 1 --2- 172.21.15.67:0/3175951058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6006be00 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8d60072ea0 tx=0x7f8d5802e0f0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 -- 172.21.15.67:0/3175951058 shutdown_connections 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 --2- 172.21.15.67:0/3175951058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6006be00 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 --2- 172.21.15.67:0/3175951058 >> v2:172.21.15.187:3300/0 conn(0x7f8d6015b6b0 0x7f8d6006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 -- 172.21.15.67:0/3175951058 >> 172.21.15.67:0/3175951058 conn(0x7f8d600b2190 msgr2=0x7f8d600b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 -- 172.21.15.67:0/3175951058 shutdown_connections 2024-04-17T08:31:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.256+0000 7f8d66e00640 1 -- 172.21.15.67:0/3175951058 wait complete. 2024-04-17T08:31:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.257+0000 7f8d66e00640 1 Processor -- start 2024-04-17T08:31:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 -- start start 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 0x7f8d6014b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d6015b6b0 0x7f8d601f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d600afee0 con 0x7f8d600cfc10 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8d600afd60 con 0x7f8d6006f800 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.258+0000 7f8d66e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d600b44a0 con 0x7f8d6015b6b0 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48754/0 (socket says 172.21.15.67:48754) 2024-04-17T08:31:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 -- 172.21.15.67:0/834260453 learned_addr learned my addr 172.21.15.67:0/834260453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65a00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 0x7f8d6014b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 -- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d6015b6b0 msgr2=0x7f8d601f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d6015b6b0 0x7f8d601f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 -- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 msgr2=0x7f8d6014b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 0x7f8d6014b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65000640 1 -- 172.21.15.67:0/834260453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d601f5500 con 0x7f8d600cfc10 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.259+0000 7f8d65a00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 0x7f8d6014b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d65000640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00c460 tx=0x7f8d4c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c018070 con 0x7f8d600cfc10 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4c007500 con 0x7f8d600cfc10 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c004a20 con 0x7f8d600cfc10 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d601f57e0 con 0x7f8d600cfc10 2024-04-17T08:31:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.260+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d6014f810 con 0x7f8d600cfc10 2024-04-17T08:31:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.261+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d6014fa10 con 0x7f8d600cfc10 2024-04-17T08:31:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.262+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 94) ==== 91160+0+0 (secure 0 0 0) 0x7f8d4c01b070 con 0x7f8d600cfc10 2024-04-17T08:31:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.263+0000 7f8d5d600640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 0x7f8d4006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.263+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f8d4c08b100 con 0x7f8d600cfc10 2024-04-17T08:31:09.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.264+0000 7f8d65a00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 0x7f8d4006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.265+0000 7f8d65a00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 0x7f8d4006ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d54006fd0 tx=0x7f8d54008040 comp rx=0 tx=0).ready entity=mgr.7729 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:09.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.268+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8d6014fa10 con 0x7f8d600cfc10 2024-04-17T08:31:09.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.530+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f8d6014f340 con 0x7f8d600cfc10 2024-04-17T08:31:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.571+0000 7f8d5d600640 1 -- 172.21.15.67:0/834260453 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) ==== 93+0+0 (secure 0 0 0) 0x7f8d4c059330 con 0x7f8d600cfc10 2024-04-17T08:31:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.576+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 msgr2=0x7f8d4006ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.576+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 0x7f8d4006ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d54006fd0 tx=0x7f8d54008040 comp rx=0 tx=0).stop 2024-04-17T08:31:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.576+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 msgr2=0x7f8d6014bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.576+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00c460 tx=0x7f8d4c00c920 comp rx=0 tx=0).stop 2024-04-17T08:31:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 shutdown_connections 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f8d4006ca60 0x7f8d4006ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d6015b6b0 0x7f8d601f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d600cfc10 0x7f8d6014bd60 secure :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00c460 tx=0x7f8d4c00c920 comp rx=0 tx=0).stop 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 --2- 172.21.15.67:0/834260453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d6006f800 0x7f8d6014b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.577+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 >> 172.21.15.67:0/834260453 conn(0x7f8d600b2190 msgr2=0x7f8d600b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.578+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 shutdown_connections 2024-04-17T08:31:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:09.579+0000 7f8d66e00640 1 -- 172.21.15.67:0/834260453 wait complete. 2024-04-17T08:31:09.599 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:31:09.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:31:09.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:31:09.793 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:31:09.793 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:31:09.793 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:09.793 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:09.845 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:09.845 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:09.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:10.133 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.134+0000 7f14ece45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:10.138 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.139+0000 7fec01042040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.146+0000 7fb576400640 1 Processor -- start 2024-04-17T08:31:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.147+0000 7fb576400640 1 -- start start 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.148+0000 7fb576400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb5700b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.148+0000 7fb576400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5700cfae0 0x7fb5700b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.148+0000 7fb576400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5700d2d30 con 0x7fb5700b6b30 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.148+0000 7fb576400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb5700d49c0 con 0x7fb5700cfae0 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.148+0000 7fb576400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5700d3660 con 0x7fb57015ec60 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5700cfae0 0x7fb5700b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb56fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5700b6b30 0x7fb5700cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40800/0 (socket says 172.21.15.67:40800) 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb56fe00640 1 -- 172.21.15.67:0/2659003275 learned_addr learned my addr 172.21.15.67:0/2659003275 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575000640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb5700b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575a00640 1 -- 172.21.15.67:0/2659003275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 msgr2=0x7fb5700b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575a00640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb5700b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575a00640 1 -- 172.21.15.67:0/2659003275 >> v1:172.21.15.67:6789/0 conn(0x7fb5700b6b30 legacy=0x7fb5700cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb575a00640 1 -- 172.21.15.67:0/2659003275 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5700d3010 con 0x7fb5700cfae0 2024-04-17T08:31:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.149+0000 7fb56f400640 1 -- 172.21.15.67:0/2659003275 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807026255 0 0) 0x7fb5700d2d30 con 0x7fb5700b6b30 2024-04-17T08:31:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.151+0000 7fb575a00640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700cfae0 0x7fb5700b7b70 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb568009d90 tx=0x7fb56802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51ab6510506ed5f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.151+0000 7fb56f400640 1 -- 172.21.15.67:0/2659003275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568039070 con 0x7fb5700cfae0 2024-04-17T08:31:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.151+0000 7fb56f400640 1 -- 172.21.15.67:0/2659003275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56802cc50 con 0x7fb5700cfae0 2024-04-17T08:31:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.151+0000 7fb56f400640 1 -- 172.21.15.67:0/2659003275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568034440 con 0x7fb5700cfae0 2024-04-17T08:31:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 -- 172.21.15.67:0/2659003275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700cfae0 msgr2=0x7fb5700b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700cfae0 0x7fb5700b7b70 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb568009d90 tx=0x7fb56802bea0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 -- 172.21.15.67:0/2659003275 shutdown_connections 2024-04-17T08:31:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700cfae0 0x7fb5700b7b70 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 --2- 172.21.15.67:0/2659003275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb5700b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 -- 172.21.15.67:0/2659003275 >> 172.21.15.67:0/2659003275 conn(0x7fb5700d15f0 msgr2=0x7fb5700d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 -- 172.21.15.67:0/2659003275 shutdown_connections 2024-04-17T08:31:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.152+0000 7fb576400640 1 -- 172.21.15.67:0/2659003275 wait complete. 2024-04-17T08:31:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.154+0000 7fb576400640 1 Processor -- start 2024-04-17T08:31:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.154+0000 7fb576400640 1 -- start start 2024-04-17T08:31:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5700cfae0 0x7fb5700b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb570156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb57015f040 con 0x7fb5700b6b30 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb5700b6f70 con 0x7fb5700cfae0 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb576400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5700d00c0 con 0x7fb57015ec60 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb575000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb575000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48776/0 (socket says 172.21.15.67:48776) 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.155+0000 7fb575000640 1 -- 172.21.15.67:0/2174037275 learned_addr learned my addr 172.21.15.67:0/2174037275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 msgr2=0x7fb570156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb570156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5700cfae0 msgr2=0x7fb5700b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5700cfae0 0x7fb5700b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 -- 172.21.15.67:0/2174037275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5700b9ba0 con 0x7fb5700b6b30 2024-04-17T08:31:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb575000640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d620 tx=0x7fb56400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56400b620 con 0x7fb5700b6b30 2024-04-17T08:31:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb564019070 con 0x7fb5700b6b30 2024-04-17T08:31:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56400ba00 con 0x7fb5700b6b30 2024-04-17T08:31:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5700b9e80 con 0x7fb5700b6b30 2024-04-17T08:31:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.156+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5701f5b80 con 0x7fb5700b6b30 2024-04-17T08:31:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.158+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5700d2d30 con 0x7fb5700b6b30 2024-04-17T08:31:10.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.160+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7fb56400bba0 con 0x7fb5700b6b30 2024-04-17T08:31:10.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.160+0000 7fb56cc00640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 0x7fb55006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.160+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7fb56408c210 con 0x7fb5700b6b30 2024-04-17T08:31:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.161+0000 7fb56fe00640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 msgr2=0x7fb55006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.161+0000 7fb56fe00640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 0x7fb55006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:10.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.166+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb5700d2d30 con 0x7fb5700b6b30 2024-04-17T08:31:10.189 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.190+0000 7f14ece45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:10.201 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.201+0000 7fec01042040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.362+0000 7fb56fe00640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 msgr2=0x7fb55006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.362+0000 7fb56fe00640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 0x7fb55006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:10.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.369+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb570068a10 con 0x7fb5700b6b30 2024-04-17T08:31:10.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.371+0000 7fb56cc00640 1 -- 172.21.15.67:0/2174037275 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) ==== 74+0+17182 (secure 0 0 0) 0x7fb56405a440 con 0x7fb5700b6b30 2024-04-17T08:31:10.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:10.370 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":183,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:06.484407+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1428143598":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/1712983456":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2676684388":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6833/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6832/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2751617585":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2559339550":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2642787856":"2024-04-18T08:31:06.484301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:10.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 msgr2=0x7fb55006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:10.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 0x7fb55006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 msgr2=0x7fb570153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d620 tx=0x7fb56400dae0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 shutdown_connections 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fb55006ca10 0x7fb55006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb57015ec60 0x7fb570156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5700cfae0 0x7fb5700b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 --2- 172.21.15.67:0/2174037275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5700b6b30 0x7fb570153dc0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 >> 172.21.15.67:0/2174037275 conn(0x7fb5700d15f0 msgr2=0x7fb57015e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 shutdown_connections 2024-04-17T08:31:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.373+0000 7fb576400640 1 -- 172.21.15.67:0/2174037275 wait complete. 2024-04-17T08:31:10.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:30:49.565353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:30:50.604764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:31:10.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:31:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.590+0000 7f9cd7200640 1 Processor -- start 2024-04-17T08:31:10.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.591+0000 7f9cd7200640 1 -- start start 2024-04-17T08:31:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cd00cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00cf3e0 con 0x7f9cd00d0200 2024-04-17T08:31:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00b2700 con 0x7f9cd015ba80 2024-04-17T08:31:10.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd00b29e0 con 0x7f9cd00cf890 2024-04-17T08:31:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9cd00d0200 0x7f9cd015be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40814/0 (socket says 172.21.15.67:40814) 2024-04-17T08:31:10.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.592+0000 7f9cd6800640 1 -- 172.21.15.67:0/1933438677 learned_addr learned my addr 172.21.15.67:0/1933438677 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd5400640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cd00cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282201902 0 0) 0x7f9cd00cf3e0 con 0x7f9cd00d0200 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb8003520 con 0x7f9cd00d0200 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 433360026 0 0) 0x7f9cd00b2700 con 0x7f9cd015ba80 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cd00cf3e0 con 0x7f9cd015ba80 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1925981362 0 0) 0x7f9cb8003520 con 0x7f9cd00d0200 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 msgr2=0x7f9cd00cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cd00cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 >> v1:172.21.15.187:6789/0 conn(0x7f9cd015ba80 legacy=0x7f9cd00cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd00b2cc0 con 0x7f9cd00d0200 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9ccc001970 con 0x7f9cd00d0200 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 >> v1:172.21.15.67:6789/0 conn(0x7f9cd00d0200 legacy=0x7f9cd015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.593+0000 7f9cd4a00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8003b70 0x7f9cb8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd4a00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd4a00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb8004d40 0x7f9cb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd00b29e0 con 0x7f9cb8003b70 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb800dcb0 con 0x7f9cb8004d40 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb800deb0 con 0x7f9cd00cf890 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8003b70 0x7f9cb8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd6800640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb8004d40 0x7f9cb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5400640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 -- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 msgr2=0x7f9cb8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cd00cf890 0x7f9cb8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 -- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb8004d40 msgr2=0x7f9cb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb8004d40 0x7f9cb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.594+0000 7f9cd5e00640 1 -- 172.21.15.67:0/1933438677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb8003520 con 0x7f9cb8003b70 2024-04-17T08:31:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.596+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8003b70 0x7f9cb8003f70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40018d0 tx=0x7f9cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d171084f66180 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.596+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4037070 con 0x7f9cb8003b70 2024-04-17T08:31:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.596+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc40052d0 con 0x7f9cb8003b70 2024-04-17T08:31:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.596+0000 7f9cd4a00640 1 -- 172.21.15.67:0/1933438677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40014f0 con 0x7f9cb8003b70 2024-04-17T08:31:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 -- 172.21.15.67:0/1933438677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8003b70 msgr2=0x7f9cb8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 --2- 172.21.15.67:0/1933438677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8003b70 0x7f9cb8003f70 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40018d0 tx=0x7f9cb8005830 comp rx=0 tx=0).stop 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd6800640 1 -- 172.21.15.67:0/1933438677 reap_dead start 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 -- 172.21.15.67:0/1933438677 shutdown_connections 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 -- 172.21.15.67:0/1933438677 >> 172.21.15.67:0/1933438677 conn(0x7f9cd009aa60 msgr2=0x7f9cd00b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 -- 172.21.15.67:0/1933438677 shutdown_connections 2024-04-17T08:31:10.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.597+0000 7f9cd7200640 1 -- 172.21.15.67:0/1933438677 wait complete. 2024-04-17T08:31:10.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.599+0000 7f9cd7200640 1 Processor -- start 2024-04-17T08:31:10.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.599+0000 7f9cd7200640 1 -- start start 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cb8003b70 0x7f9cd006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cd00c8dc0 0x7f9cd0155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd00b07d0 con 0x7f9cd0155ac0 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd00b0650 con 0x7f9cd00c8dc0 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd00ce840 con 0x7f9cb8003b70 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48792/0 (socket says 172.21.15.67:48792) 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 -- 172.21.15.67:0/2554832369 learned_addr learned my addr 172.21.15.67:0/2554832369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cb8003b70 0x7f9cd006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cb8003b70 msgr2=0x7f9cd006fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cb8003b70 0x7f9cd006fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cd00c8dc0 msgr2=0x7f9cd0155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cd00c8dc0 0x7f9cd0155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.600+0000 7f9cd6800640 1 -- 172.21.15.67:0/2554832369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd0155f50 con 0x7f9cd0155ac0 2024-04-17T08:31:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cd6800640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f9cd006eae0 tx=0x7f9ccc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00ead0 con 0x7f9cd0155ac0 2024-04-17T08:31:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00ec90 con 0x7f9cd0155ac0 2024-04-17T08:31:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc014430 con 0x7f9cd0155ac0 2024-04-17T08:31:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd01f5070 con 0x7f9cd0155ac0 2024-04-17T08:31:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.601+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd01f5630 con 0x7f9cd0155ac0 2024-04-17T08:31:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.602+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd00755d0 con 0x7f9cd0155ac0 2024-04-17T08:31:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.604+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7f9ccc0145f0 con 0x7f9cd0155ac0 2024-04-17T08:31:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.605+0000 7f9cca000640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 0x7f9cb406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.605+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f9ccc08bb50 con 0x7f9cd0155ac0 2024-04-17T08:31:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.605+0000 7f9cd5e00640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 msgr2=0x7f9cb406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.605+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 0x7f9cb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:10.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.610+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9ccc059d80 con 0x7f9cd0155ac0 2024-04-17T08:31:10.634 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.635+0000 7f14ece45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:10.677 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.678+0000 7fec01042040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:10.686 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.687+0000 7f14ece45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:10.732 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.733+0000 7fec01042040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:10.736 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.737+0000 7f14ece45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:10.786 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.787+0000 7fec01042040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:10.797 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.798+0000 7f14ece45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:10.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.806+0000 7f9cd5e00640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 msgr2=0x7f9cb406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:10.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.806+0000 7f9cd5e00640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 0x7f9cb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.817+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9cd00b73c0 con 0x7f9cd0155ac0 2024-04-17T08:31:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.818+0000 7f9cca000640 1 -- 172.21.15.67:0/2554832369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v183) ==== 93+0+10 (secure 0 0 0) 0x7f9ccc059680 con 0x7f9cd0155ac0 2024-04-17T08:31:10.817 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 msgr2=0x7f9cb406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 0x7f9cb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 msgr2=0x7f9cd00b6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f9cd006eae0 tx=0x7f9ccc007bc0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 shutdown_connections 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd0155ac0 0x7f9cd00b6520 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cd00c8dc0 0x7f9cd0155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f9cb406ca10 0x7f9cb406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 --2- 172.21.15.67:0/2554832369 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9cb8003b70 0x7f9cd006fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 >> 172.21.15.67:0/2554832369 conn(0x7f9cd009aa60 msgr2=0x7f9cd00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 shutdown_connections 2024-04-17T08:31:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:10.820+0000 7f9cd7200640 1 -- 172.21.15.67:0/2554832369 wait complete. 2024-04-17T08:31:10.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:31:10.846 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.847+0000 7f14ece45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:10.852 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.853+0000 7fec01042040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:10.906 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:10.906+0000 7fec01042040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:10.973 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:10.974+0000 7f14ece45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:11.035 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.036+0000 7f14ece45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:11.044 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.045+0000 7fec01042040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:11.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.050+0000 7ff1d4e00640 1 Processor -- start 2024-04-17T08:31:11.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.051+0000 7ff1d4e00640 1 -- start start 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.052+0000 7ff1d4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d015e9b0 0x7ff1d00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.052+0000 7ff1d4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1d00d4510 0x7ff1d00d4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.052+0000 7ff1d4e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d00b51d0 con 0x7ff1d00cf910 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.052+0000 7ff1d4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff1d00b5790 con 0x7ff1d00d4510 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.052+0000 7ff1d4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d00d2830 con 0x7ff1d015e9b0 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1d00d4510 0x7ff1d00d4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1d00d4510 0x7ff1d00d4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48798/0 (socket says 172.21.15.67:48798) 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 -- 172.21.15.67:0/1081746240 learned_addr learned my addr 172.21.15.67:0/1081746240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cf400640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d015e9b0 0x7ff1d00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 -- 172.21.15.67:0/1081746240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d015e9b0 msgr2=0x7ff1d00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d015e9b0 0x7ff1d00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 -- 172.21.15.67:0/1081746240 >> v1:172.21.15.187:6789/0 conn(0x7ff1d00cf910 legacy=0x7ff1d00b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.053+0000 7ff1cfe00640 1 -- 172.21.15.67:0/1081746240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d00b54b0 con 0x7ff1d00d4510 2024-04-17T08:31:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.054+0000 7ff1cfe00640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00d4510 0x7ff1d00d4940 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009e50 tx=0x7ff1c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c5e5443d87bd2a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1ce000640 1 -- 172.21.15.67:0/1081746240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c802ca70 con 0x7ff1d00d4510 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1ce000640 1 -- 172.21.15.67:0/1081746240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c802cc10 con 0x7ff1d00d4510 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1ce000640 1 -- 172.21.15.67:0/1081746240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80343f0 con 0x7ff1d00d4510 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 -- 172.21.15.67:0/1081746240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00d4510 msgr2=0x7ff1d00d4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00d4510 0x7ff1d00d4940 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009e50 tx=0x7ff1c802bed0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 -- 172.21.15.67:0/1081746240 shutdown_connections 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00d4510 0x7ff1d00d4940 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/1081746240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d015e9b0 0x7ff1d00cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 -- 172.21.15.67:0/1081746240 >> 172.21.15.67:0/1081746240 conn(0x7ff1d009daa0 msgr2=0x7ff1d008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.055+0000 7ff1d4e00640 1 -- 172.21.15.67:0/1081746240 shutdown_connections 2024-04-17T08:31:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.056+0000 7ff1d4e00640 1 -- 172.21.15.67:0/1081746240 wait complete. 2024-04-17T08:31:11.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.057+0000 7ff1d4e00640 1 Processor -- start 2024-04-17T08:31:11.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.057+0000 7ff1d4e00640 1 -- start start 2024-04-17T08:31:11.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 0x7ff1d0153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d015e9b0 0x7ff1d0152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0162830 con 0x7ff1d00cf910 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d00d4e80 con 0x7ff1d015e9b0 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1d4e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d00b4e00 con 0x7ff1d00d4510 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48806/0 (socket says 172.21.15.67:48806) 2024-04-17T08:31:11.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.058+0000 7ff1cf400640 1 -- 172.21.15.67:0/2018263756 learned_addr learned my addr 172.21.15.67:0/2018263756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:11.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cea00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 0x7ff1d0153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 msgr2=0x7ff1d0153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 0x7ff1d0153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d015e9b0 msgr2=0x7ff1d0152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d015e9b0 0x7ff1d0152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 -- 172.21.15.67:0/2018263756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d015ed90 con 0x7ff1d00cf910 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cea00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 0x7ff1d0153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.059+0000 7ff1cf400640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000d5f0 tx=0x7ff1c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.060+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00125f0 con 0x7ff1d00cf910 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.060+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c0019070 con 0x7ff1d00cf910 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.060+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d015c180 con 0x7ff1d00cf910 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.060+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00129d0 con 0x7ff1d00cf910 2024-04-17T08:31:11.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.060+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d015c740 con 0x7ff1d00cf910 2024-04-17T08:31:11.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.061+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d00755d0 con 0x7ff1d00cf910 2024-04-17T08:31:11.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.062+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7ff1c0012b70 con 0x7ff1d00cf910 2024-04-17T08:31:11.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.063+0000 7ff1c7400640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 0x7ff1b006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.063+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7ff1c008c2e0 con 0x7ff1d00cf910 2024-04-17T08:31:11.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.063+0000 7ff1cea00640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 msgr2=0x7ff1b006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:11.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.063+0000 7ff1cea00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 0x7ff1b006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:11.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.068+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff1c005a3f0 con 0x7ff1d00cf910 2024-04-17T08:31:11.085 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.086+0000 7f14ece45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:11.113 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.114+0000 7fec01042040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:11.142 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.143+0000 7f14ece45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:11.166 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.167+0000 7fec01042040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:11.193 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.194+0000 7f14ece45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:11.231 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.232+0000 7fec01042040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:11.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.264+0000 7ff1cea00640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 msgr2=0x7ff1b006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:11.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.264+0000 7ff1cea00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 0x7ff1b006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:11.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.272+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1d01e8000 con 0x7ff1d00cf910 2024-04-17T08:31:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.273+0000 7ff1c7400640 1 -- 172.21.15.67:0/2018263756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v183) ==== 104+0+11 (secure 0 0 0) 0x7ff1c0059cf0 con 0x7ff1d00cf910 2024-04-17T08:31:11.272 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:31:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 msgr2=0x7ff1b006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 0x7ff1b006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 msgr2=0x7ff1d015baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000d5f0 tx=0x7ff1c000dab0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 shutdown_connections 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1d015e9b0 0x7ff1d0152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7ff1b006ca40 0x7ff1b006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1d00d4510 0x7ff1d0153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 --2- 172.21.15.67:0/2018263756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1d00cf910 0x7ff1d015baa0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.275+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 >> 172.21.15.67:0/2018263756 conn(0x7ff1d009daa0 msgr2=0x7ff1d006de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.276+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 shutdown_connections 2024-04-17T08:31:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.276+0000 7ff1d4e00640 1 -- 172.21.15.67:0/2018263756 wait complete. 2024-04-17T08:31:11.286 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.287+0000 7fec01042040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:11.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:31:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.552+0000 7f1b86c00640 1 Processor -- start 2024-04-17T08:31:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.553+0000 7f1b86c00640 1 -- start start 2024-04-17T08:31:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b86c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b800d01a0 0x7f1b800b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b86c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1b800d0580 0x7f1b80162ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b86c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b8015f520 con 0x7f1b800ce3a0 2024-04-17T08:31:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b86c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1b800cfc90 con 0x7f1b800d01a0 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b86c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1b8015fe50 con 0x7f1b800d0580 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b7fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b800d01a0 0x7f1b800b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b7fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b800d01a0 0x7f1b800b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48814/0 (socket says 172.21.15.67:48814) 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.555+0000 7f1b7fe00640 1 -- 172.21.15.67:0/1293890106 learned_addr learned my addr 172.21.15.67:0/1293890106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.556+0000 7f1b7fe00640 1 -- 172.21.15.67:0/1293890106 >> v2:172.21.15.187:3300/0 conn(0x7f1b800d0580 msgr2=0x7f1b80162ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.556+0000 7f1b84c00640 1 --2- 172.21.15.67:0/1293890106 >> v2:172.21.15.187:3300/0 conn(0x7f1b800d0580 0x7f1b80162ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.556+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/1293890106 >> v2:172.21.15.187:3300/0 conn(0x7f1b800d0580 0x7f1b80162ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.556+0000 7f1b7fe00640 1 -- 172.21.15.67:0/1293890106 >> v1:172.21.15.187:6789/0 conn(0x7f1b800ce3a0 legacy=0x7f1b801627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.556+0000 7f1b7fe00640 1 -- 172.21.15.67:0/1293890106 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8015f800 con 0x7f1b800d01a0 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.557+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/1293890106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d01a0 0x7f1b800b02f0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f1b800b0a30 tx=0x7f1b7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab78027779f69a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.557+0000 7f1b7ea00640 1 -- 172.21.15.67:0/1293890106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70035020 con 0x7f1b800d01a0 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.557+0000 7f1b7ea00640 1 -- 172.21.15.67:0/1293890106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b70031870 con 0x7f1b800d01a0 2024-04-17T08:31:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.557+0000 7f1b7ea00640 1 -- 172.21.15.67:0/1293890106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70031c40 con 0x7f1b800d01a0 2024-04-17T08:31:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 -- 172.21.15.67:0/1293890106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d01a0 msgr2=0x7f1b800b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 --2- 172.21.15.67:0/1293890106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d01a0 0x7f1b800b02f0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f1b800b0a30 tx=0x7f1b7002b910 comp rx=0 tx=0).stop 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 -- 172.21.15.67:0/1293890106 shutdown_connections 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 --2- 172.21.15.67:0/1293890106 >> v2:172.21.15.187:3300/0 conn(0x7f1b800d0580 0x7f1b80162ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 --2- 172.21.15.67:0/1293890106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d01a0 0x7f1b800b02f0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 -- 172.21.15.67:0/1293890106 >> 172.21.15.67:0/1293890106 conn(0x7f1b800cd880 msgr2=0x7f1b800cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 -- 172.21.15.67:0/1293890106 shutdown_connections 2024-04-17T08:31:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.558+0000 7f1b86c00640 1 -- 172.21.15.67:0/1293890106 wait complete. 2024-04-17T08:31:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.561+0000 7f1b86c00640 1 Processor -- start 2024-04-17T08:31:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.561+0000 7f1b86c00640 1 -- start start 2024-04-17T08:31:11.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b800ce3a0 0x7f1b80148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b800d01a0 0x7f1b8014b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b800cd5f0 con 0x7f1b800d0580 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8015cc30 con 0x7f1b800ce3a0 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b86c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b8015f1f0 con 0x7f1b800d01a0 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b84c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b84c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48830/0 (socket says 172.21.15.67:48830) 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.562+0000 7f1b84c00640 1 -- 172.21.15.67:0/4022328097 learned_addr learned my addr 172.21.15.67:0/4022328097 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b800ce3a0 0x7f1b80148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b800d01a0 msgr2=0x7f1b8014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b800d01a0 0x7f1b8014b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b800ce3a0 msgr2=0x7f1b80148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b800ce3a0 0x7f1b80148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 -- 172.21.15.67:0/4022328097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8014bf30 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.563+0000 7f1b84c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800cc60 tx=0x7f1b7800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.564+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78003990 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.564+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b78003b30 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.564+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78004020 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.564+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8015a500 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.564+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8015aac0 con 0x7f1b800d0580 2024-04-17T08:31:11.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.565+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b80068b90 con 0x7f1b800d0580 2024-04-17T08:31:11.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.566+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7f1b780041c0 con 0x7f1b800d0580 2024-04-17T08:31:11.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.567+0000 7f1b77e00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 0x7f1b6006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:11.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.567+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f1b7808c490 con 0x7f1b800d0580 2024-04-17T08:31:11.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.567+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 msgr2=0x7f1b6006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:11.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.567+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 0x7f1b6006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:11.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.572+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1b8014bf30 con 0x7f1b800d0580 2024-04-17T08:31:11.581 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.582+0000 7f14ece45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:11.699 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.700+0000 7fec01042040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:11.741 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.742+0000 7f14ece45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.768+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 msgr2=0x7f1b6006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.768+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 0x7f1b6006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:11.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.778+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1b800cb460 con 0x7f1b800d0580 2024-04-17T08:31:11.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.779+0000 7f1b77e00640 1 -- 172.21.15.67:0/4022328097 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v183) ==== 100+0+11 (secure 0 0 0) 0x7f1b78014030 con 0x7f1b800d0580 2024-04-17T08:31:11.778 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 msgr2=0x7f1b6006ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 0x7f1b6006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 msgr2=0x7f1b800b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800cc60 tx=0x7f1b7800a5c0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 shutdown_connections 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.781+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b800d0580 0x7f1b800b1870 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f1b6006ca60 0x7f1b6006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b800d01a0 0x7f1b8014b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 --2- 172.21.15.67:0/4022328097 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b800ce3a0 0x7f1b80148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 >> 172.21.15.67:0/4022328097 conn(0x7f1b800cd880 msgr2=0x7f1b8015c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 shutdown_connections 2024-04-17T08:31:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:11.782+0000 7f1b86c00640 1 -- 172.21.15.67:0/4022328097 wait complete. 2024-04-17T08:31:11.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:31:11.849 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.850+0000 7f14ece45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:11.870 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.871+0000 7fec01042040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:11.899 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:11.900+0000 7f14ece45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:11.986 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:11.987+0000 7fec01042040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.002+0000 7f1752c00640 1 Processor -- start 2024-04-17T08:31:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.002+0000 7f1752c00640 1 -- start start 2024-04-17T08:31:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1752c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c15ee20 0x7f174c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1752c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f174c0d18e0 0x7f174c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1752c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174c0d4140 con 0x7f174c0d3350 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1752c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f174c0b83b0 con 0x7f174c0d18e0 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1752c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f174c0d1d20 con 0x7f174c15ee20 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1750e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f174c0d18e0 0x7f174c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1750e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f174c0d18e0 0x7f174c0d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48842/0 (socket says 172.21.15.67:48842) 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.004+0000 7f1750e00640 1 -- 172.21.15.67:0/1413330936 learned_addr learned my addr 172.21.15.67:0/1413330936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.005+0000 7f1751800640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c15ee20 0x7f174c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.005+0000 7f1750e00640 1 -- 172.21.15.67:0/1413330936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c15ee20 msgr2=0x7f174c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.005+0000 7f1750e00640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c15ee20 0x7f174c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.005+0000 7f1750e00640 1 -- 172.21.15.67:0/1413330936 >> v1:172.21.15.187:6789/0 conn(0x7f174c0d3350 legacy=0x7f174c067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.005+0000 7f1750e00640 1 -- 172.21.15.67:0/1413330936 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0d3b80 con 0x7f174c0d18e0 2024-04-17T08:31:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.006+0000 7f1750e00640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c0d18e0 0x7f174c0d2e10 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f1738009870 tx=0x7f173802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c504ca52670f22a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.006+0000 7f174be00640 1 -- 172.21.15.67:0/1413330936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738035020 con 0x7f174c0d18e0 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.006+0000 7f174be00640 1 -- 172.21.15.67:0/1413330936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1738031640 con 0x7f174c0d18e0 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.006+0000 7f174be00640 1 -- 172.21.15.67:0/1413330936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173802d040 con 0x7f174c0d18e0 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 -- 172.21.15.67:0/1413330936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c0d18e0 msgr2=0x7f174c0d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c0d18e0 0x7f174c0d2e10 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f1738009870 tx=0x7f173802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 -- 172.21.15.67:0/1413330936 shutdown_connections 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c0d18e0 0x7f174c0d2e10 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 --2- 172.21.15.67:0/1413330936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c15ee20 0x7f174c0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 -- 172.21.15.67:0/1413330936 >> 172.21.15.67:0/1413330936 conn(0x7f174c06d3c0 msgr2=0x7f174c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 -- 172.21.15.67:0/1413330936 shutdown_connections 2024-04-17T08:31:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.007+0000 7f1752c00640 1 -- 172.21.15.67:0/1413330936 wait complete. 2024-04-17T08:31:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.009+0000 7f1752c00640 1 Processor -- start 2024-04-17T08:31:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.009+0000 7f1752c00640 1 -- start start 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c0d18e0 0x7f174c1f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f174c0d3350 0x7f174c1f8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f174c15f200 con 0x7f174c15ee20 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f174c0b8040 con 0x7f174c0d3350 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f174c0d37e0 con 0x7f174c0d18e0 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48844/0 (socket says 172.21.15.67:48844) 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1752200640 1 -- 172.21.15.67:0/3701346189 learned_addr learned my addr 172.21.15.67:0/3701346189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1750e00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f174c0d3350 0x7f174c1f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.010+0000 7f1751800640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c0d18e0 0x7f174c1f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c0d18e0 msgr2=0x7f174c1f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c0d18e0 0x7f174c1f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f174c0d3350 msgr2=0x7f174c1f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f174c0d3350 0x7f174c1f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 -- 172.21.15.67:0/3701346189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0cbc00 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752200640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f174400c780 tx=0x7f174400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744004b60 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1744004d00 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174400aa90 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.011+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c0cbee0 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.012+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c0cc4a0 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.013+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174c068a10 con 0x7f174c15ee20 2024-04-17T08:31:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.014+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7f174400ac30 con 0x7f174c15ee20 2024-04-17T08:31:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.015+0000 7f1749600640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 0x7f172c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.015+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f174408c0e0 con 0x7f174c15ee20 2024-04-17T08:31:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.015+0000 7f1751800640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 msgr2=0x7f172c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.015+0000 7f1751800640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 0x7f172c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.020+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1744020070 con 0x7f174c15ee20 2024-04-17T08:31:12.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:12.027+0000 7f14ece45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:12.043 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.044+0000 7fec01042040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:12.132 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:12.133+0000 7f14ece45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:12.177 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.178+0000 7fec01042040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:12.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.216+0000 7f1751800640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 msgr2=0x7f172c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:12.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.216+0000 7f1751800640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 0x7f172c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.236+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f174c0b58a0 con 0x7f174c15ee20 2024-04-17T08:31:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.236+0000 7f1749600640 1 -- 172.21.15.67:0/3701346189 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v183) ==== 80+0+476 (secure 0 0 0) 0x7f174405a270 con 0x7f174c15ee20 2024-04-17T08:31:12.235 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1428143598 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.235 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1712983456 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2676684388 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/659531354 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/659531354 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2751617585 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2559339550 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2642787856 2024-04-18T08:31:06.484301+0000 2024-04-17T08:31:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T08:31:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 msgr2=0x7f172c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 0x7f172c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 msgr2=0x7f174c1f86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f174400c780 tx=0x7f174400cc40 comp rx=0 tx=0).stop 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 shutdown_connections 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c15ee20 0x7f174c1f86b0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f174c0d3350 0x7f174c1f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f172c06ca10 0x7f172c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 --2- 172.21.15.67:0/3701346189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f174c0d18e0 0x7f174c1f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 >> 172.21.15.67:0/3701346189 conn(0x7f174c06d3c0 msgr2=0x7f174c0b9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 shutdown_connections 2024-04-17T08:31:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.239+0000 7f1752c00640 1 -- 172.21.15.67:0/3701346189 wait complete. 2024-04-17T08:31:12.258 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:31:12.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:12.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:12.295 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.296+0000 7fec01042040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:12.307 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:12.308+0000 7f14ece45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:12.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:31:12.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:12.352 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:31:12.356 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:12.357+0000 7f14ece45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:12.442 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:31:12.442 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:31:12.443 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:12.487 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.488+0000 7fec01042040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:12.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:31:12.495 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:31:12.496 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:12.496 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:12.496 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:12.496+0000 7f14ece45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:12.543 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.544+0000 7fec01042040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:12.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:31:12.548 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:31:12.548 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:31:12.548 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:12.600 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:31:12.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:12.693 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:12.693+0000 7fec01042040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.812+0000 7f3564a00640 1 Processor -- start 2024-04-17T08:31:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.812+0000 7f3564a00640 1 -- start start 2024-04-17T08:31:12.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f3564a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35600cfae0 0x7f3560169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f3564a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35600d2d30 con 0x7f35600b6d00 2024-04-17T08:31:12.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f3564a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35600d3010 con 0x7f356015ec60 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f3564a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f35600d49c0 con 0x7f35600cfae0 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f355f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35600cfae0 0x7f3560169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f355f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35600cfae0 0x7f3560169810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48846/0 (socket says 172.21.15.67:48846) 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f355f200640 1 -- 172.21.15.67:0/180196395 learned_addr learned my addr 172.21.15.67:0/180196395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.814+0000 7f3555e00640 1 --1- 172.21.15.67:0/180196395 >> v1:172.21.15.67:6789/0 conn(0x7f35600b6d00 0x7f35600cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40818/0 (socket says 172.21.15.67:40818) 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.815+0000 7f355f200640 1 -- 172.21.15.67:0/180196395 >> v1:172.21.15.187:6789/0 conn(0x7f356015ec60 legacy=0x7f35600b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.815+0000 7f355f200640 1 -- 172.21.15.67:0/180196395 >> v1:172.21.15.67:6789/0 conn(0x7f35600b6d00 legacy=0x7f35600cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.815+0000 7f355f200640 1 -- 172.21.15.67:0/180196395 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35600d32f0 con 0x7f35600cfae0 2024-04-17T08:31:12.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.816+0000 7f355f200640 1 --2- 172.21.15.67:0/180196395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600cfae0 0x7f3560169810 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3558009870 tx=0x7f355802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa2eb9283a5ba31 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:12.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.816+0000 7f355de00640 1 -- 172.21.15.67:0/180196395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355802f850 con 0x7f35600cfae0 2024-04-17T08:31:12.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.816+0000 7f355de00640 1 -- 172.21.15.67:0/180196395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f355802f9f0 con 0x7f35600cfae0 2024-04-17T08:31:12.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.816+0000 7f355de00640 1 -- 172.21.15.67:0/180196395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355802fdc0 con 0x7f35600cfae0 2024-04-17T08:31:12.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 -- 172.21.15.67:0/180196395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600cfae0 msgr2=0x7f3560169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 --2- 172.21.15.67:0/180196395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600cfae0 0x7f3560169810 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3558009870 tx=0x7f355802ba30 comp rx=0 tx=0).stop 2024-04-17T08:31:12.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 -- 172.21.15.67:0/180196395 shutdown_connections 2024-04-17T08:31:12.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 --2- 172.21.15.67:0/180196395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600cfae0 0x7f3560169810 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 -- 172.21.15.67:0/180196395 >> 172.21.15.67:0/180196395 conn(0x7f35600d15f0 msgr2=0x7f35600d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 -- 172.21.15.67:0/180196395 shutdown_connections 2024-04-17T08:31:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.817+0000 7f3564a00640 1 -- 172.21.15.67:0/180196395 wait complete. 2024-04-17T08:31:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.819+0000 7f3564a00640 1 Processor -- start 2024-04-17T08:31:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.819+0000 7f3564a00640 1 -- start start 2024-04-17T08:31:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35600cfae0 0x7f35601f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f356015ec60 0x7f35600cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35600b8b80 con 0x7f35600b6d00 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f35600d3660 con 0x7f35600cfae0 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3564a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f35600d00c0 con 0x7f356015ec60 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48860/0 (socket says 172.21.15.67:48860) 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 -- 172.21.15.67:0/1221330954 learned_addr learned my addr 172.21.15.67:0/1221330954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f3555e00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35600cfae0 0x7f35601f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f356015ec60 msgr2=0x7f35600cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f356015ec60 0x7f35600cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35600cfae0 msgr2=0x7f35601f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35600cfae0 0x7f35601f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.820+0000 7f355e800640 1 -- 172.21.15.67:0/1221330954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35600cce60 con 0x7f35600b6d00 2024-04-17T08:31:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f355e800640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f355000c730 tx=0x7f355000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:12.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550003b70 con 0x7f35600b6d00 2024-04-17T08:31:12.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3550003d30 con 0x7f35600b6d00 2024-04-17T08:31:12.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355000aa90 con 0x7f35600b6d00 2024-04-17T08:31:12.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35600cd100 con 0x7f35600b6d00 2024-04-17T08:31:12.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.821+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35600cd6c0 con 0x7f35600b6d00 2024-04-17T08:31:12.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.822+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3560068a10 con 0x7f35600b6d00 2024-04-17T08:31:12.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.823+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7f355000ac30 con 0x7f35600b6d00 2024-04-17T08:31:12.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.824+0000 7f3557400640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.824+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f355008bda0 con 0x7f35600b6d00 2024-04-17T08:31:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.824+0000 7f3555e00640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 msgr2=0x7f353c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.824+0000 7f3555e00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:12.829+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3550059f50 con 0x7f35600b6d00 2024-04-17T08:31:13.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.025+0000 7f3555e00640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 msgr2=0x7f353c06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:13.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.025+0000 7f3555e00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:13.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.071+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35600cda90 con 0x7f35600b6d00 2024-04-17T08:31:13.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.072+0000 7f3557400640 1 -- 172.21.15.67:0/1221330954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f3550059850 con 0x7f35600b6d00 2024-04-17T08:31:13.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:13.071 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:31:13.071 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.074+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 msgr2=0x7f353c06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.074+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.074+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 msgr2=0x7f356014d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.074+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f355000c730 tx=0x7f355000cbf0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 shutdown_connections 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f353c06ca10 0x7f353c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f356015ec60 0x7f35600cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35600cfae0 0x7f35601f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 --2- 172.21.15.67:0/1221330954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35600b6d00 0x7f356014d240 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 >> 172.21.15.67:0/1221330954 conn(0x7f35600d15f0 msgr2=0x7f356015e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 shutdown_connections 2024-04-17T08:31:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.075+0000 7f3564a00640 1 -- 172.21.15.67:0/1221330954 wait complete. 2024-04-17T08:31:13.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:13.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.291+0000 7fc0ea600640 1 Processor -- start 2024-04-17T08:31:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.291+0000 7fc0ea600640 1 -- start start 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0ea600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0e40cf640 0x7fc0e40cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0ea600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e40cff60 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0ea600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e40d3020 con 0x7fc0e415ed10 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0ea600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e40d3c30 con 0x7fc0e40cf640 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0e3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc0e40d2ab0 0x7fc0e40b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48324/0 (socket says 172.21.15.67:48324) 2024-04-17T08:31:13.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.293+0000 7fc0e3e00640 1 -- 172.21.15.67:0/1698137239 learned_addr learned my addr 172.21.15.67:0/1698137239 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e3400640 1 --2- 172.21.15.67:0/1698137239 >> v2:172.21.15.187:3300/0 conn(0x7fc0e40cf640 0x7fc0e40cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722548231 0 0) 0x7fc0e40cff60 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0c8003520 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 483233171 0 0) 0x7fc0c8003520 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 >> v2:172.21.15.187:3300/0 conn(0x7fc0e40cf640 msgr2=0x7fc0e40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 --2- 172.21.15.67:0/1698137239 >> v2:172.21.15.187:3300/0 conn(0x7fc0e40cf640 0x7fc0e40cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 >> v1:172.21.15.187:6789/0 conn(0x7fc0e415ed10 legacy=0x7fc0e40d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e40d3300 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc0dc0019f0 con 0x7fc0e40d2ab0 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 >> v1:172.21.15.67:6789/0 conn(0x7fc0e40d2ab0 legacy=0x7fc0e40b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c80039b0 0x7fc0c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.294+0000 7fc0e2a00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c80051c0 0x7fc0c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e40d3c30 con 0x7fc0c8004800 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c800c070 con 0x7fc0c80039b0 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c800c270 con 0x7fc0c80051c0 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e3400640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c80039b0 0x7fc0c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e3e00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c80051c0 0x7fc0c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 -- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c80051c0 msgr2=0x7fc0c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0c80051c0 0x7fc0c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 -- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c80039b0 msgr2=0x7fc0c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0c80039b0 0x7fc0c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0dbe00640 1 -- 172.21.15.67:0/1698137239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c8003520 con 0x7fc0c8004800 2024-04-17T08:31:13.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.295+0000 7fc0e3e00640 1 -- 172.21.15.67:0/1698137239 reap_dead start 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.296+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0c8004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc001720 tx=0x7fc0c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b01fc48d2aca575 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.297+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc001370 con 0x7fc0c8004800 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.297+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc0052d0 con 0x7fc0c8004800 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.297+0000 7fc0e2a00640 1 -- 172.21.15.67:0/1698137239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc003830 con 0x7fc0c8004800 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 -- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 msgr2=0x7fc0c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0c8004c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc001720 tx=0x7fc0c8005830 comp rx=0 tx=0).stop 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 -- 172.21.15.67:0/1698137239 shutdown_connections 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 --2- 172.21.15.67:0/1698137239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0c8004c80 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 -- 172.21.15.67:0/1698137239 >> 172.21.15.67:0/1698137239 conn(0x7fc0e406cbd0 msgr2=0x7fc0e406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 -- 172.21.15.67:0/1698137239 shutdown_connections 2024-04-17T08:31:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.298+0000 7fc0ea600640 1 -- 172.21.15.67:0/1698137239 wait complete. 2024-04-17T08:31:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.301+0000 7fc0ea600640 1 Processor -- start 2024-04-17T08:31:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.301+0000 7fc0ea600640 1 -- start start 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.302+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0e40b6800 0x7fc0e40b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e40b42d0 0x7fc0e40b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0ea600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e40d25b0 con 0x7fc0c8004800 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0ea600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e40d38c0 con 0x7fc0e40b6800 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0ea600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e40ba300 con 0x7fc0e40b42d0 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35596/0 (socket says 172.21.15.67:35596) 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 -- 172.21.15.67:0/517728623 learned_addr learned my addr 172.21.15.67:0/517728623 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3e00640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e40b42d0 0x7fc0e40b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0e40b6800 0x7fc0e40b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e40b42d0 msgr2=0x7fc0e40b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e40b42d0 0x7fc0e40b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0e40b6800 msgr2=0x7fc0e40b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0e40b6800 0x7fc0e40b6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.303+0000 7fc0e3400640 1 -- 172.21.15.67:0/517728623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e40b4c90 con 0x7fc0c8004800 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0e3400640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40114e0 tx=0x7fc0d40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4015870 con 0x7fc0c8004800 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e40b4f50 con 0x7fc0c8004800 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4015a10 con 0x7fc0c8004800 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4015dc0 con 0x7fc0c8004800 2024-04-17T08:31:13.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.304+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e40b5510 con 0x7fc0c8004800 2024-04-17T08:31:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.305+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e415a760 con 0x7fc0c8004800 2024-04-17T08:31:13.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.307+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7fc0d4013400 con 0x7fc0c8004800 2024-04-17T08:31:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.307+0000 7fc0db400640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 0x7fc0c006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.308+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7fc0d408c630 con 0x7fc0c8004800 2024-04-17T08:31:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.308+0000 7fc0dbe00640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 msgr2=0x7fc0c006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.308+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 0x7fc0c006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:13.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.312+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc0e415a760 con 0x7fc0c8004800 2024-04-17T08:31:13.315 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.316+0000 7f14ece45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:13.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.509+0000 7fc0dbe00640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 msgr2=0x7fc0c006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:13.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.509+0000 7fc0dbe00640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 0x7fc0c006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:13.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.516+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0e4068b90 con 0x7fc0c8004800 2024-04-17T08:31:13.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.518+0000 7fc0db400640 1 -- 172.21.15.67:0/517728623 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) ==== 74+0+17182 (secure 0 0 0) 0x7fc0d4019030 con 0x7fc0c8004800 2024-04-17T08:31:13.517 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:13.517 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":183,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:06.484407+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1428143598":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/1712983456":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2676684388":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6833/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6832/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2751617585":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2559339550":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2642787856":"2024-04-18T08:31:06.484301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.520+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 msgr2=0x7fc0c006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.520+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 0x7fc0c006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.520+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 msgr2=0x7fc0e414cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.520+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40114e0 tx=0x7fc0d40119a0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 shutdown_connections 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7fc0c006ca40 0x7fc0c006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0e40b42d0 0x7fc0e40b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0e40b6800 0x7fc0e40b6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 --2- 172.21.15.67:0/517728623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c8004800 0x7fc0e414cd00 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 >> 172.21.15.67:0/517728623 conn(0x7fc0e406cbd0 msgr2=0x7fc0e40d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 shutdown_connections 2024-04-17T08:31:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.521+0000 7fc0ea600640 1 -- 172.21.15.67:0/517728623 wait complete. 2024-04-17T08:31:13.536 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:30:49.565353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:30:50.604764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:31:13.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:31:13.564 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:13.565+0000 7fec01042040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:13.617 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.618+0000 7f14ece45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:13.675 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.676+0000 7f14ece45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.742+0000 7f8967e00640 1 Processor -- start 2024-04-17T08:31:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.742+0000 7f8967e00640 1 -- start start 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.743+0000 7f8967e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89680d2ff0 0x7f89680b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.743+0000 7f8967e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89680b5110 con 0x7f89681622b0 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.743+0000 7f8967e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89680b53f0 con 0x7f89680b4c70 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.743+0000 7f8967e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89680b56d0 con 0x7f89680d2ff0 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89680d2ff0 0x7f89680b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89680d2ff0 0x7f89680b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35608/0 (socket says 172.21.15.67:35608) 2024-04-17T08:31:13.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 -- 172.21.15.67:0/145597276 learned_addr learned my addr 172.21.15.67:0/145597276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:13.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8967400640 1 --1- 172.21.15.67:0/145597276 >> v1:172.21.15.67:6789/0 conn(0x7f89681622b0 0x7f89680cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48330/0 (socket says 172.21.15.67:48330) 2024-04-17T08:31:13.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 -- 172.21.15.67:0/145597276 >> v1:172.21.15.187:6789/0 conn(0x7f89680b4c70 legacy=0x7f8968161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:13.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 -- 172.21.15.67:0/145597276 >> v1:172.21.15.67:6789/0 conn(0x7f89681622b0 legacy=0x7f89680cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.744+0000 7f8966a00640 1 -- 172.21.15.67:0/145597276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89680cf340 con 0x7f89680d2ff0 2024-04-17T08:31:13.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.745+0000 7f8966a00640 1 --2- 172.21.15.67:0/145597276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f89680b4730 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f8958009870 tx=0x7f895802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346a43a18756bc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8965600640 1 -- 172.21.15.67:0/145597276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958034070 con 0x7f89680d2ff0 2024-04-17T08:31:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8965600640 1 -- 172.21.15.67:0/145597276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8958030860 con 0x7f89680d2ff0 2024-04-17T08:31:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8965600640 1 -- 172.21.15.67:0/145597276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958030c30 con 0x7f89680d2ff0 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8967e00640 1 -- 172.21.15.67:0/145597276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 msgr2=0x7f89680b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8967e00640 1 --2- 172.21.15.67:0/145597276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f89680b4730 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f8958009870 tx=0x7f895802b9a0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8967e00640 1 -- 172.21.15.67:0/145597276 shutdown_connections 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8967e00640 1 --2- 172.21.15.67:0/145597276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f89680b4730 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.746+0000 7f8967e00640 1 -- 172.21.15.67:0/145597276 >> 172.21.15.67:0/145597276 conn(0x7f896809da80 msgr2=0x7f896808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.747+0000 7f8967e00640 1 -- 172.21.15.67:0/145597276 shutdown_connections 2024-04-17T08:31:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.747+0000 7f8967e00640 1 -- 172.21.15.67:0/145597276 wait complete. 2024-04-17T08:31:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.748+0000 7f8967e00640 1 Processor -- start 2024-04-17T08:31:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.748+0000 7f8967e00640 1 -- start start 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89680b4c70 0x7f89680b7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89681622b0 0x7f89680c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89680d3550 con 0x7f89680d2ff0 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89680d33d0 con 0x7f89681622b0 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8967e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89680d36d0 con 0x7f89680b4c70 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.749+0000 7f8966000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35614/0 (socket says 172.21.15.67:35614) 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 -- 172.21.15.67:0/1856870201 learned_addr learned my addr 172.21.15.67:0/1856870201 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966a00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89680b4c70 0x7f89680b7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8967400640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89681622b0 0x7f89680c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:13.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 -- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89680b4c70 msgr2=0x7f89680b7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89680b4c70 0x7f89680b7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 -- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89681622b0 msgr2=0x7f89680c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89681622b0 0x7f89680c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 -- 172.21.15.67:0/1856870201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89680cc960 con 0x7f89680d2ff0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.750+0000 7f8966000640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f895000b420 tx=0x7f895000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.751+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950011830 con 0x7f89680d2ff0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.751+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8950016070 con 0x7f89680d2ff0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.751+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950011c10 con 0x7f89680d2ff0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.751+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896806e4a0 con 0x7f89680d2ff0 2024-04-17T08:31:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.751+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896806e780 con 0x7f89680d2ff0 2024-04-17T08:31:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.752+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89680755d0 con 0x7f89680d2ff0 2024-04-17T08:31:13.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.753+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 95) ==== 91143+0+0 (secure 0 0 0) 0x7f895001b460 con 0x7f89680d2ff0 2024-04-17T08:31:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.754+0000 7f895f400640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 0x7f894406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.754+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6081+0+0 (secure 0 0 0) 0x7f895008b770 con 0x7f89680d2ff0 2024-04-17T08:31:13.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.754+0000 7f8966a00640 1 -- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 msgr2=0x7f894406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2551862940 2024-04-17T08:31:13.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.754+0000 7f8966a00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 0x7f894406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:13.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.758+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8950020020 con 0x7f89680d2ff0 2024-04-17T08:31:13.807 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.808+0000 7f14ece45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.828+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 96) ==== 90751+0+0 (secure 0 0 0) 0x7f8950055910 con 0x7f89680d2ff0 2024-04-17T08:31:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.828+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 msgr2=0x7f894406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.828+0000 7f895f400640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 0x7f894406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.901 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:13.901+0000 7fec01042040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f146a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f146a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f146a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f146a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f146a800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f1466c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.904 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f1466c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.905 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f1466c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.905 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f1466c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.905 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:13.905+0000 7f1466c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:13.964 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:13.964+0000 7fec01042040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:13.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.970+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8968161ba0 con 0x7f89680d2ff0 2024-04-17T08:31:13.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.971+0000 7f895f400640 1 -- 172.21.15.67:0/1856870201 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) ==== 93+0+10 (secure 0 0 0) 0x7f8950010380 con 0x7f89680d2ff0 2024-04-17T08:31:13.971 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:31:13.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 msgr2=0x7f8968156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:13.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f895000b420 tx=0x7f895000b8e0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 shutdown_connections 2024-04-17T08:31:13.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89681622b0 0x7f89680c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89680d2ff0 0x7f8968156f10 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:6832/2551862940,v1:172.21.15.187:6833/2551862940] conn(0x7f894406ca10 0x7f894406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 --2- 172.21.15.67:0/1856870201 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89680b4c70 0x7f89680b7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 >> 172.21.15.67:0/1856870201 conn(0x7f896809da80 msgr2=0x7f89680b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 shutdown_connections 2024-04-17T08:31:13.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:13.974+0000 7f8967e00640 1 -- 172.21.15.67:0/1856870201 wait complete. 2024-04-17T08:31:13.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:31:14.104 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:14.105+0000 7fec01042040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:14.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.237+0000 7f9b88a00640 1 Processor -- start 2024-04-17T08:31:14.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.237+0000 7f9b88a00640 1 -- start start 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.238+0000 7f9b88a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9b84145f80 0x7f9b840cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b88a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b84146360 con 0x7f9b8406c270 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b88a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b84071e70 con 0x7f9b840cd750 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b88a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9b840727a0 con 0x7f9b84145f80 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b83200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b8406c270 0x7f9b8406e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48346/0 (socket says 172.21.15.67:48346) 2024-04-17T08:31:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b83200640 1 -- 172.21.15.67:0/3011995742 learned_addr learned my addr 172.21.15.67:0/3011995742 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b82800640 1 --2- 172.21.15.67:0/3011995742 >> v2:172.21.15.187:3300/0 conn(0x7f9b84145f80 0x7f9b840cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033111108 0 0) 0x7f9b84146360 con 0x7f9b8406c270 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.239+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b68003520 con 0x7f9b8406c270 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 18102258 0 0) 0x7f9b68003520 con 0x7f9b8406c270 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 >> v2:172.21.15.187:3300/0 conn(0x7f9b84145f80 msgr2=0x7f9b840cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 --2- 172.21.15.67:0/3011995742 >> v2:172.21.15.187:3300/0 conn(0x7f9b84145f80 0x7f9b840cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 >> v1:172.21.15.187:6789/0 conn(0x7f9b840cd750 legacy=0x7f9b8406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b84072150 con 0x7f9b8406c270 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9b7c0019f0 con 0x7f9b8406c270 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 >> v1:172.21.15.67:6789/0 conn(0x7f9b8406c270 legacy=0x7f9b8406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 0x7f9b68003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68004800 0x7f9b68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b68005210 0x7f9b6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b840727a0 con 0x7f9b68003a00 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6800c160 con 0x7f9b68005210 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6800c360 con 0x7f9b68004800 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.240+0000 7f9b82800640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 0x7f9b68003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b82800640 1 -- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68004800 msgr2=0x7f9b68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b82800640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68004800 0x7f9b68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b82800640 1 -- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b68005210 msgr2=0x7f9b6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b82800640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b68005210 0x7f9b6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b82800640 1 -- 172.21.15.67:0/3011995742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b68003520 con 0x7f9b68003a00 2024-04-17T08:31:14.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.241+0000 7f9b83200640 1 -- 172.21.15.67:0/3011995742 reap_dead start 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b82800640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 0x7f9b68003e20 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009a90 tx=0x7f9b68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b979f65306b99d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74039070 con 0x7f9b68003a00 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7402c6b0 con 0x7f9b68003a00 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b81e00640 1 -- 172.21.15.67:0/3011995742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74009690 con 0x7f9b68003a00 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b88a00640 1 -- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 msgr2=0x7f9b68003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.242+0000 7f9b88a00640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 0x7f9b68003e20 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009a90 tx=0x7f9b68005830 comp rx=0 tx=0).stop 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.243+0000 7f9b88a00640 1 -- 172.21.15.67:0/3011995742 shutdown_connections 2024-04-17T08:31:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.243+0000 7f9b88a00640 1 --2- 172.21.15.67:0/3011995742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b68003a00 0x7f9b68003e20 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.243+0000 7f9b88a00640 1 -- 172.21.15.67:0/3011995742 >> 172.21.15.67:0/3011995742 conn(0x7f9b840b2190 msgr2=0x7f9b840b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.243+0000 7f9b88a00640 1 -- 172.21.15.67:0/3011995742 shutdown_connections 2024-04-17T08:31:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.243+0000 7f9b88a00640 1 -- 172.21.15.67:0/3011995742 wait complete. 2024-04-17T08:31:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.245+0000 7f9b88a00640 1 Processor -- start 2024-04-17T08:31:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.245+0000 7f9b88a00640 1 -- start start 2024-04-17T08:31:14.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 0x7f9b841e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b840b38a0 0x7f9b840b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b84072db0 con 0x7f9b840ad8d0 2024-04-17T08:31:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9b84072c30 con 0x7f9b840b38a0 2024-04-17T08:31:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b88a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9b840cd210 con 0x7f9b68003a00 2024-04-17T08:31:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b83200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b83200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35632/0 (socket says 172.21.15.67:35632) 2024-04-17T08:31:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.246+0000 7f9b83200640 1 -- 172.21.15.67:0/1658543785 learned_addr learned my addr 172.21.15.67:0/1658543785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 -- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 msgr2=0x7f9b841e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b82800640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 0x7f9b841e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 0x7f9b841e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 -- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b840b38a0 msgr2=0x7f9b840b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b840b38a0 0x7f9b840b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 -- 172.21.15.67:0/1658543785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b840b4240 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b82800640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 0x7f9b841e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b83200640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9b840d1470 tx=0x7f9b7c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c00ce70 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c00a610 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c00aa40 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.247+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b840b4500 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.248+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b840b4ac0 con 0x7f9b840ad8d0 2024-04-17T08:31:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.249+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b84068a10 con 0x7f9b840ad8d0 2024-04-17T08:31:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.250+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 96) ==== 90751+0+0 (secure 0 0 0) 0x7f9b7c011070 con 0x7f9b840ad8d0 2024-04-17T08:31:14.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.251+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f9b7c08b0d0 con 0x7f9b840ad8d0 2024-04-17T08:31:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.255+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9b7c059590 con 0x7f9b840ad8d0 2024-04-17T08:31:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.458+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b840cec90 con 0x7f9b840ad8d0 2024-04-17T08:31:14.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.459+0000 7f9b7b400640 1 -- 172.21.15.67:0/1658543785 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v184) ==== 104+0+11 (secure 0 0 0) 0x7f9b7c058e90 con 0x7f9b840ad8d0 2024-04-17T08:31:14.458 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.461+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 msgr2=0x7f9b840add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.461+0000 7f9b88a00640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9b840d1470 tx=0x7f9b7c00cc20 comp rx=0 tx=0).stop 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 shutdown_connections 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b840ad8d0 0x7f9b840add50 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9b840b38a0 0x7f9b840b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 --2- 172.21.15.67:0/1658543785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9b68003a00 0x7f9b841e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 >> 172.21.15.67:0/1658543785 conn(0x7f9b840b2190 msgr2=0x7f9b840b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 shutdown_connections 2024-04-17T08:31:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.462+0000 7f9b88a00640 1 -- 172.21.15.67:0/1658543785 wait complete. 2024-04-17T08:31:14.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:31:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.700+0000 7f421be00640 1 Processor -- start 2024-04-17T08:31:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.701+0000 7f421be00640 1 -- start start 2024-04-17T08:31:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.703+0000 7f421be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f421c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.703+0000 7f421be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f421c0cfcd0 0x7f421c0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.703+0000 7f421be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c0d4f70 con 0x7f421c0b3b50 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.703+0000 7f421be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f421c0d5250 con 0x7f421c0cfcd0 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.703+0000 7f421be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f421c0cf4d0 con 0x7f421c15ec20 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f421a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f421c0b3b50 0x7f421c06ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48362/0 (socket says 172.21.15.67:48362) 2024-04-17T08:31:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f421a000640 1 -- 172.21.15.67:0/2184569884 learned_addr learned my addr 172.21.15.67:0/2184569884 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f421aa00640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f421c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> v2:172.21.15.187:3300/0 conn(0x7f421c0cfcd0 0x7f421c0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138404221 0 0) 0x7f421c0d4f70 con 0x7f421c0b3b50 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.704+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4200003520 con 0x7f421c0b3b50 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4160364313 0 0) 0x7f4200003520 con 0x7f421c0b3b50 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 msgr2=0x7f421c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f421c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 >> v2:172.21.15.187:3300/0 conn(0x7f421c0cfcd0 msgr2=0x7f421c0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 --2- 172.21.15.67:0/2184569884 >> v2:172.21.15.187:3300/0 conn(0x7f421c0cfcd0 0x7f421c0d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c0cf7b0 con 0x7f421c0b3b50 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f42080016c0 con 0x7f421c0b3b50 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 >> v1:172.21.15.67:6789/0 conn(0x7f421c0b3b50 legacy=0x7f421c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f4200004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200004d40 0x7f42000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f421c0d5250 con 0x7f4200004d40 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f421c0cf4d0 con 0x7f42000039b0 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f420000baf0 con 0x7f421c15ec20 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> v2:172.21.15.187:3300/0 conn(0x7f421c0cfcd0 0x7f421c0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200004d40 0x7f42000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.705+0000 7f421a000640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f4200004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421aa00640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421b400640 1 -- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 msgr2=0x7f4200004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c15ec20 0x7f4200004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421b400640 1 -- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42000039b0 msgr2=0x7f4200003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42000039b0 0x7f4200003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.706+0000 7f421b400640 1 -- 172.21.15.67:0/2184569884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4200003520 con 0x7f4200004d40 2024-04-17T08:31:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f421b400640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200004d40 0x7f42000051c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1391ceb3c4ec31c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421400f040 con 0x7f4200004d40 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4214037600 con 0x7f4200004d40 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f4219600640 1 -- 172.21.15.67:0/2184569884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42140379b0 con 0x7f4200004d40 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f421be00640 1 -- 172.21.15.67:0/2184569884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200004d40 msgr2=0x7f42000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f421be00640 1 --2- 172.21.15.67:0/2184569884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200004d40 0x7f42000051c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f42140098a0 tx=0x7f4200005830 comp rx=0 tx=0).stop 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.707+0000 7f421b400640 1 -- 172.21.15.67:0/2184569884 reap_dead start 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.708+0000 7f421be00640 1 -- 172.21.15.67:0/2184569884 shutdown_connections 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.708+0000 7f421be00640 1 -- 172.21.15.67:0/2184569884 >> 172.21.15.67:0/2184569884 conn(0x7f421c08bff0 msgr2=0x7f421c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.708+0000 7f421be00640 1 -- 172.21.15.67:0/2184569884 shutdown_connections 2024-04-17T08:31:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.708+0000 7f421be00640 1 -- 172.21.15.67:0/2184569884 wait complete. 2024-04-17T08:31:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.710+0000 7f421be00640 1 Processor -- start 2024-04-17T08:31:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.710+0000 7f421be00640 1 -- start start 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 0x7f421c161290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c150b90 0x7f421c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f421c06d2d0 con 0x7f421c0b5980 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f421c06d150 con 0x7f4200004d40 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f421c0d22b0 con 0x7f421c150b90 2024-04-17T08:31:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35638/0 (socket says 172.21.15.67:35638) 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421a000640 1 -- 172.21.15.67:0/369503263 learned_addr learned my addr 172.21.15.67:0/369503263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.711+0000 7f421b400640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c150b90 0x7f421c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421aa00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 0x7f421c161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 -- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c150b90 msgr2=0x7f421c0b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c150b90 0x7f421c0b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 -- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 msgr2=0x7f421c161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 0x7f421c161290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 -- 172.21.15.67:0/369503263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c0b48e0 con 0x7f421c0b5980 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421aa00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 0x7f421c161290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f421a000640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f420800d770 tx=0x7f420800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208019070 con 0x7f421c0b5980 2024-04-17T08:31:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.712+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42080074e0 con 0x7f421c0b5980 2024-04-17T08:31:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.713+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208004a00 con 0x7f421c0b5980 2024-04-17T08:31:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.713+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c0b4bc0 con 0x7f421c0b5980 2024-04-17T08:31:14.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.713+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f421c0b5180 con 0x7f421c0b5980 2024-04-17T08:31:14.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.714+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421c068a10 con 0x7f421c0b5980 2024-04-17T08:31:14.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.715+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 96) ==== 90751+0+0 (secure 0 0 0) 0x7f4208004ba0 con 0x7f421c0b5980 2024-04-17T08:31:14.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.716+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f420808b130 con 0x7f421c0b5980 2024-04-17T08:31:14.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.721+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f420800de20 con 0x7f421c0b5980 2024-04-17T08:31:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.838+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 97) ==== 90878+0+0 (secure 0 0 0) 0x7f4208054d50 con 0x7f421c0b5980 2024-04-17T08:31:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.839+0000 7f4213400640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 0x7f41fc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.846+0000 7f421aa00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 0x7f41fc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:14.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.857+0000 7f421aa00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 0x7f41fc074490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f420c007460 tx=0x7f420c00d040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:14.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.932+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f421c0d55f0 con 0x7f421c0b5980 2024-04-17T08:31:14.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.933+0000 7f4213400640 1 -- 172.21.15.67:0/369503263 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v184) ==== 100+0+11 (secure 0 0 0) 0x7f420805faf0 con 0x7f421c0b5980 2024-04-17T08:31:14.932 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.935+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 msgr2=0x7f41fc074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.935+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 0x7f41fc074490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f420c007460 tx=0x7f420c00d040 comp rx=0 tx=0).stop 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 msgr2=0x7f421c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f420800d770 tx=0x7f420800dc30 comp rx=0 tx=0).stop 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 shutdown_connections 2024-04-17T08:31:14.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f421c150b90 0x7f421c0b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0b5980 0x7f421c0b4140 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f41fc0720a0 0x7f41fc074490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 --2- 172.21.15.67:0/369503263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4200004d40 0x7f421c161290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 >> 172.21.15.67:0/369503263 conn(0x7f421c08bff0 msgr2=0x7f421c0d2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 shutdown_connections 2024-04-17T08:31:14.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:14.936+0000 7f421be00640 1 -- 172.21.15.67:0/369503263 wait complete. 2024-04-17T08:31:14.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.174+0000 7f1707e00640 1 Processor -- start 2024-04-17T08:31:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.174+0000 7f1707e00640 1 -- start start 2024-04-17T08:31:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1707e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f170806c040 0x7f17081664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1707e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170806c450 con 0x7f170815b570 2024-04-17T08:31:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1707e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170815bc80 con 0x7f170806feb0 2024-04-17T08:31:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1707e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f170815bf40 con 0x7f170806c040 2024-04-17T08:31:15.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1707400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f170806c040 0x7f17081664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1706a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f170815b570 0x7f170806f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48378/0 (socket says 172.21.15.67:48378) 2024-04-17T08:31:15.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.176+0000 7f1706a00640 1 -- 172.21.15.67:0/4282196908 learned_addr learned my addr 172.21.15.67:0/4282196908 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:15.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.177+0000 7f1707400640 1 -- 172.21.15.67:0/4282196908 >> v1:172.21.15.187:6789/0 conn(0x7f170806feb0 legacy=0x7f170806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:15.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.177+0000 7f1707400640 1 -- 172.21.15.67:0/4282196908 >> v1:172.21.15.67:6789/0 conn(0x7f170815b570 legacy=0x7f170806f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.177+0000 7f1707400640 1 -- 172.21.15.67:0/4282196908 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170815c220 con 0x7f170806c040 2024-04-17T08:31:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.178+0000 7f1707400640 1 --2- 172.21.15.67:0/4282196908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17081664d0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f17000097a0 tx=0x7f170002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946ff4323254d1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.178+0000 7f1706000640 1 -- 172.21.15.67:0/4282196908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17000315a0 con 0x7f170806c040 2024-04-17T08:31:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.178+0000 7f1706000640 1 -- 172.21.15.67:0/4282196908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1700037070 con 0x7f170806c040 2024-04-17T08:31:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.178+0000 7f1706000640 1 -- 172.21.15.67:0/4282196908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1700031960 con 0x7f170806c040 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 -- 172.21.15.67:0/4282196908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 msgr2=0x7f17081664d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 --2- 172.21.15.67:0/4282196908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17081664d0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f17000097a0 tx=0x7f170002b840 comp rx=0 tx=0).stop 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 -- 172.21.15.67:0/4282196908 shutdown_connections 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 --2- 172.21.15.67:0/4282196908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17081664d0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 -- 172.21.15.67:0/4282196908 >> 172.21.15.67:0/4282196908 conn(0x7f1708067d10 msgr2=0x7f17080b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 -- 172.21.15.67:0/4282196908 shutdown_connections 2024-04-17T08:31:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.179+0000 7f1707e00640 1 -- 172.21.15.67:0/4282196908 wait complete. 2024-04-17T08:31:15.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.181+0000 7f1707e00640 1 Processor -- start 2024-04-17T08:31:15.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.181+0000 7f1707e00640 1 -- start start 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 0x7f170814cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f170815b570 0x7f17080aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17080b47b0 con 0x7f170806c040 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f170815b950 con 0x7f170815b570 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17080b4930 con 0x7f170806feb0 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1706a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1706a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35646/0 (socket says 172.21.15.67:35646) 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1706a00640 1 -- 172.21.15.67:0/3152409486 learned_addr learned my addr 172.21.15.67:0/3152409486 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f1707400640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f170815b570 0x7f17080aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.182+0000 7f16fe000640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 0x7f170814cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 -- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 msgr2=0x7f170814cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 0x7f170814cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 -- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f170815b570 msgr2=0x7f17080aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f170815b570 0x7f17080aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 -- 172.21.15.67:0/3152409486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17080aebe0 con 0x7f170806c040 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f16fe000640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 0x7f170814cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:15.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1706a00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f16f800c3c0 tx=0x7f16f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8017070 con 0x7f170806c040 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f8005ce0 con 0x7f170806c040 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80072f0 con 0x7f170806c040 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17080cf960 con 0x7f170806c040 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.183+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17080cff20 con 0x7f170806c040 2024-04-17T08:31:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.184+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1708068a10 con 0x7f170806c040 2024-04-17T08:31:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.186+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 97) ==== 90878+0+0 (secure 0 0 0) 0x7f16f801a070 con 0x7f170806c040 2024-04-17T08:31:15.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.187+0000 7f16ff400640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 0x7f16e406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.187+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f16f808aeb0 con 0x7f170806c040 2024-04-17T08:31:15.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.187+0000 7f16fe000640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 0x7f16e406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.188+0000 7f16fe000640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 0x7f16e406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17080b35e0 tx=0x7f16f0009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.192+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f16f8059280 con 0x7f170806c040 2024-04-17T08:31:15.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.439+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17080b2210 con 0x7f170806c040 2024-04-17T08:31:15.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.440+0000 7f16ff400640 1 -- 172.21.15.67:0/3152409486 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f16f8058b80 con 0x7f170806c040 2024-04-17T08:31:15.439 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:15.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:31:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:31:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 msgr2=0x7f16e406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 0x7f16e406ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17080b35e0 tx=0x7f16f0009040 comp rx=0 tx=0).stop 2024-04-17T08:31:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 msgr2=0x7f17080cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f16f800c3c0 tx=0x7f16f800c880 comp rx=0 tx=0).stop 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 shutdown_connections 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f16e406c740 0x7f16e406ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f170815b570 0x7f17080aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f170806feb0 0x7f170814cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 --2- 172.21.15.67:0/3152409486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170806c040 0x7f17080cee30 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 >> 172.21.15.67:0/3152409486 conn(0x7f1708067d10 msgr2=0x7f1708071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 shutdown_connections 2024-04-17T08:31:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.443+0000 7f1707e00640 1 -- 172.21.15.67:0/3152409486 wait complete. 2024-04-17T08:31:15.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:15.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.684+0000 7f8b6da00640 1 Processor -- start 2024-04-17T08:31:15.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.684+0000 7f8b6da00640 1 -- start start 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b680cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b680cd4a0 con 0x7f8b68141ed0 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b681607f0 con 0x7f8b68160280 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6815ce30 con 0x7f8b6806b8c0 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b68141ed0 0x7f8b6815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48394/0 (socket says 172.21.15.67:48394) 2024-04-17T08:31:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.686+0000 7f8b6d000640 1 -- 172.21.15.67:0/1259357733 learned_addr learned my addr 172.21.15.67:0/1259357733 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b67e00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b680cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101156300 0 0) 0x7f8b680cd4a0 con 0x7f8b68141ed0 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4c003520 con 0x7f8b68141ed0 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4205454904 0 0) 0x7f8b4c003520 con 0x7f8b68141ed0 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 msgr2=0x7f8b680cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b680cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 >> v1:172.21.15.187:6789/0 conn(0x7f8b68160280 legacy=0x7f8b6806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b68160ad0 con 0x7f8b68141ed0 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8b600019f0 con 0x7f8b68141ed0 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 >> v1:172.21.15.67:6789/0 conn(0x7f8b68141ed0 legacy=0x7f8b6815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b4c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b4c004020 0x7f8b4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b4c004d40 0x7f8b4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6815ce30 con 0x7f8b4c004020 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c00bb80 con 0x7f8b4c004d40 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.687+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4c00bd80 con 0x7f8b6806b8c0 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b4c004020 0x7f8b4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67e00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b4c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b6d000640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b4c004d40 0x7f8b4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 -- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 msgr2=0x7f8b4c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b6806b8c0 0x7f8b4c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 -- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b4c004d40 msgr2=0x7f8b4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b4c004d40 0x7f8b4c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.688+0000 7f8b67400640 1 -- 172.21.15.67:0/1259357733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c003520 con 0x7f8b4c004020 2024-04-17T08:31:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.689+0000 7f8b67400640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b4c004020 0x7f8b4c004800 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8b54001750 tx=0x7f8b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802f43bc3d55175f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54001370 con 0x7f8b4c004020 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b540052d0 con 0x7f8b4c004020 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b66a00640 1 -- 172.21.15.67:0/1259357733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54003830 con 0x7f8b4c004020 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 -- 172.21.15.67:0/1259357733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b4c004020 msgr2=0x7f8b4c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 --2- 172.21.15.67:0/1259357733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b4c004020 0x7f8b4c004800 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8b54001750 tx=0x7f8b4c005830 comp rx=0 tx=0).stop 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6d000640 1 -- 172.21.15.67:0/1259357733 reap_dead start 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 -- 172.21.15.67:0/1259357733 shutdown_connections 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 -- 172.21.15.67:0/1259357733 >> 172.21.15.67:0/1259357733 conn(0x7f8b68067d10 msgr2=0x7f8b680b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 -- 172.21.15.67:0/1259357733 shutdown_connections 2024-04-17T08:31:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.690+0000 7f8b6da00640 1 -- 172.21.15.67:0/1259357733 wait complete. 2024-04-17T08:31:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.692+0000 7f8b6da00640 1 Processor -- start 2024-04-17T08:31:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.692+0000 7f8b6da00640 1 -- start start 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b4c004020 0x7f8b681f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b680b4220 0x7f8b68143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6813f9b0 con 0x7f8b680b46a0 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68161090 con 0x7f8b680b4220 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.693+0000 7f8b6da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6813fb30 con 0x7f8b4c004020 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35656/0 (socket says 172.21.15.67:35656) 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 -- 172.21.15.67:0/147402629 learned_addr learned my addr 172.21.15.67:0/147402629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b67400640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b680b4220 0x7f8b68143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b67e00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b4c004020 0x7f8b681f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 -- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b4c004020 msgr2=0x7f8b681f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b4c004020 0x7f8b681f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 -- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b680b4220 msgr2=0x7f8b68143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b680b4220 0x7f8b68143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.694+0000 7f8b6d000640 1 -- 172.21.15.67:0/147402629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b680b4c40 con 0x7f8b680b46a0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b6d000640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f8b60001370 tx=0x7f8b6000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60004d60 con 0x7f8b680b46a0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6000bb80 con 0x7f8b680b46a0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b680b4f20 con 0x7f8b680b46a0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60007070 con 0x7f8b680b46a0 2024-04-17T08:31:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.695+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b680b54e0 con 0x7f8b680b46a0 2024-04-17T08:31:15.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.696+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b68068a10 con 0x7f8b680b46a0 2024-04-17T08:31:15.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.697+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 97) ==== 90878+0+0 (secure 0 0 0) 0x7f8b6001b020 con 0x7f8b680b46a0 2024-04-17T08:31:15.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.698+0000 7f8b5fe00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 0x7f8b4806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:15.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.698+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f8b6008bd90 con 0x7f8b680b46a0 2024-04-17T08:31:15.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.699+0000 7f8b67e00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 0x7f8b4806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:15.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.700+0000 7f8b67e00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 0x7f8b4806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009950 tx=0x7f8b58008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:15.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.703+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8b6005a220 con 0x7f8b680b46a0 2024-04-17T08:31:15.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.851+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 98) ==== 91086+0+0 (secure 0 0 0) 0x7f8b60056190 con 0x7f8b680b46a0 2024-04-17T08:31:15.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.914+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b680cd310 con 0x7f8b680b46a0 2024-04-17T08:31:15.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.916+0000 7f8b5fe00640 1 -- 172.21.15.67:0/147402629 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) ==== 74+0+17691 (secure 0 0 0) 0x7f8b6007df30 con 0x7f8b680b46a0 2024-04-17T08:31:15.916 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:15.916 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":184,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:13.813303+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2335112946":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/3177304055":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6832/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2645268054":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1428143598":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/3079350952":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1712983456":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2676684388":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/863702549":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6832/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2751617585":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2559339550":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/1037073318":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2642787856":"2024-04-18T08:31:06.484301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:15.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 msgr2=0x7f8b4806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 0x7f8b4806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009950 tx=0x7f8b58008040 comp rx=0 tx=0).stop 2024-04-17T08:31:15.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 msgr2=0x7f8b68144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f8b60001370 tx=0x7f8b6000b4a0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 shutdown_connections 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b680b46a0 0x7f8b68144350 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8b4806c740 0x7f8b4806ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b680b4220 0x7f8b68143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 --2- 172.21.15.67:0/147402629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b4c004020 0x7f8b681f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 >> 172.21.15.67:0/147402629 conn(0x7f8b68067d10 msgr2=0x7f8b6806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 shutdown_connections 2024-04-17T08:31:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:15.919+0000 7f8b6da00640 1 -- 172.21.15.67:0/147402629 wait complete. 2024-04-17T08:31:15.938 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:31:15.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:16.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.156+0000 7f81f3200640 1 Processor -- start 2024-04-17T08:31:16.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.156+0000 7f81f3200640 1 -- start start 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f81f40b6d00 0x7f81f40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81f40cf910 0x7f81f40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81f40d2d30 con 0x7f81f415ec60 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f81f40d49c0 con 0x7f81f40b6d00 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81f40d3660 con 0x7f81f40cf910 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81f415ec60 0x7f81f40b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48408/0 (socket says 172.21.15.67:48408) 2024-04-17T08:31:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f1e00640 1 -- 172.21.15.67:0/2823611859 learned_addr learned my addr 172.21.15.67:0/2823611859 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81f2800640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81f40cf910 0x7f81f40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.158+0000 7f81ebe00640 1 --2- 172.21.15.67:0/2823611859 >> v2:172.21.15.187:3300/0 conn(0x7f81f40b6d00 0x7f81f40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720037453 0 0) 0x7f81f40d2d30 con 0x7f81f415ec60 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d8003520 con 0x7f81f415ec60 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3193358214 0 0) 0x7f81d8003520 con 0x7f81f415ec60 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81f40cf910 msgr2=0x7f81f40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81f40cf910 0x7f81f40b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 >> v2:172.21.15.187:3300/0 conn(0x7f81f40b6d00 msgr2=0x7f81f40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 --2- 172.21.15.67:0/2823611859 >> v2:172.21.15.187:3300/0 conn(0x7f81f40b6d00 0x7f81f40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f40d3010 con 0x7f81f415ec60 2024-04-17T08:31:16.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f81e4001670 con 0x7f81f415ec60 2024-04-17T08:31:16.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 >> v1:172.21.15.67:6789/0 conn(0x7f81f415ec60 legacy=0x7f81f40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d8003a20 0x7f81d8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8004800 0x7f81d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d80051c0 0x7f81d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81f40d49c0 con 0x7f81d8003a20 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81f40d3660 con 0x7f81d80051c0 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.159+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81d800bf50 con 0x7f81d8004800 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d8003a20 0x7f81d8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f2800640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d80051c0 0x7f81d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81ebe00640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8004800 0x7f81d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 -- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8004800 msgr2=0x7f81d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8004800 0x7f81d8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 -- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d80051c0 msgr2=0x7f81d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81d80051c0 0x7f81d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.160+0000 7f81f1e00640 1 -- 172.21.15.67:0/2823611859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d8003520 con 0x7f81d8003a20 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.162+0000 7f81f1e00640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d8003a20 0x7f81d8003e40 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c190 tx=0x7f81d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa782748c2e023a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.162+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4007c80 con 0x7f81d8003a20 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.162+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e4030070 con 0x7f81d8003a20 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.162+0000 7f81f1400640 1 -- 172.21.15.67:0/2823611859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4005d30 con 0x7f81d8003a20 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 -- 172.21.15.67:0/2823611859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d8003a20 msgr2=0x7f81d8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 --2- 172.21.15.67:0/2823611859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81d8003a20 0x7f81d8003e40 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c190 tx=0x7f81d8005830 comp rx=0 tx=0).stop 2024-04-17T08:31:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f2800640 1 -- 172.21.15.67:0/2823611859 reap_dead start 2024-04-17T08:31:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 -- 172.21.15.67:0/2823611859 shutdown_connections 2024-04-17T08:31:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 -- 172.21.15.67:0/2823611859 >> 172.21.15.67:0/2823611859 conn(0x7f81f40d15f0 msgr2=0x7f81f40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 -- 172.21.15.67:0/2823611859 shutdown_connections 2024-04-17T08:31:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.163+0000 7f81f3200640 1 -- 172.21.15.67:0/2823611859 wait complete. 2024-04-17T08:31:16.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.165+0000 7f81f3200640 1 Processor -- start 2024-04-17T08:31:16.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.166+0000 7f81f3200640 1 -- start start 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8003a20 0x7f81f4157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f40cbc40 0x7f81f40cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81f415f040 con 0x7f81f40cd610 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81f40cfdc0 con 0x7f81f40cbc40 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81f3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81f40d00c0 con 0x7f81d8003a20 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35672/0 (socket says 172.21.15.67:35672) 2024-04-17T08:31:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 -- 172.21.15.67:0/960067062 learned_addr learned my addr 172.21.15.67:0/960067062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 -- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8003a20 msgr2=0x7f81f4157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8003a20 0x7f81f4157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 -- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f40cbc40 msgr2=0x7f81f40cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f40cbc40 0x7f81f40cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.167+0000 7f81ebe00640 1 -- 172.21.15.67:0/960067062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f41558d0 con 0x7f81f40cd610 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81ebe00640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f81dc010890 tx=0x7f81dc010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00f040 con 0x7f81f40cd610 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc00eae0 con 0x7f81f40cd610 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc01c400 con 0x7f81f40cd610 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f4155bb0 con 0x7f81f40cd610 2024-04-17T08:31:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.168+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f4156170 con 0x7f81f40cd610 2024-04-17T08:31:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.170+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f4068a10 con 0x7f81f40cd610 2024-04-17T08:31:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.171+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 98) ==== 91086+0+0 (secure 0 0 0) 0x7f81dc01c5a0 con 0x7f81f40cd610 2024-04-17T08:31:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.171+0000 7f81ea000640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 0x7f81d006eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.172+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f81dc08c2f0 con 0x7f81f40cd610 2024-04-17T08:31:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.172+0000 7f81f1e00640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 0x7f81d006eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.173+0000 7f81f1e00640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 0x7f81d006eec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c190 tx=0x7f81e4039040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.177+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f81dc018030 con 0x7f81f40cd610 2024-04-17T08:31:16.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.425+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81f40b5330 con 0x7f81f40cd610 2024-04-17T08:31:16.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.426+0000 7f81ea000640 1 -- 172.21.15.67:0/960067062 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f81dc05a470 con 0x7f81f40cd610 2024-04-17T08:31:16.426 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:16.426 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:31:16.426 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 msgr2=0x7f81d006eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 0x7f81d006eec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c190 tx=0x7f81e4039040 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 msgr2=0x7f81f41606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f81dc010890 tx=0x7f81dc010d50 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 shutdown_connections 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f81d006ca00 0x7f81d006eec0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f40cbc40 0x7f81f40cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81f40cd610 0x7f81f41606e0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 --2- 172.21.15.67:0/960067062 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81d8003a20 0x7f81f4157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 >> 172.21.15.67:0/960067062 conn(0x7f81f40d15f0 msgr2=0x7f81f4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 shutdown_connections 2024-04-17T08:31:16.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.429+0000 7f81f3200640 1 -- 172.21.15.67:0/960067062 wait complete. 2024-04-17T08:31:16.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.673+0000 7f8ea1400640 1 Processor -- start 2024-04-17T08:31:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.673+0000 7f8ea1400640 1 -- start start 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8ea1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e9c15b570 0x7f8e9c06f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8ea1400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e9c06fce0 0x7f8e9c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8ea1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e9c15bc80 con 0x7f8e9c06be70 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8ea1400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9c15bf40 con 0x7f8e9c15b570 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8ea1400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9c15c220 con 0x7f8e9c06fce0 2024-04-17T08:31:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8e9be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e9c15b570 0x7f8e9c06f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8e9be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e9c15b570 0x7f8e9c06f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35686/0 (socket says 172.21.15.67:35686) 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.675+0000 7f8e9be00640 1 -- 172.21.15.67:0/2255921889 learned_addr learned my addr 172.21.15.67:0/2255921889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.676+0000 7f8e9be00640 1 -- 172.21.15.67:0/2255921889 >> v2:172.21.15.187:3300/0 conn(0x7f8e9c06fce0 msgr2=0x7f8e9c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.676+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2255921889 >> v2:172.21.15.187:3300/0 conn(0x7f8e9c06fce0 0x7f8e9c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.676+0000 7f8e9be00640 1 -- 172.21.15.67:0/2255921889 >> v1:172.21.15.187:6789/0 conn(0x7f8e9c06be70 legacy=0x7f8e9c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:16.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.676+0000 7f8e9be00640 1 -- 172.21.15.67:0/2255921889 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c15c500 con 0x7f8e9c15b570 2024-04-17T08:31:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.677+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2255921889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c06f7a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c073040 tx=0x7f8e8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73fe7738becc13a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.677+0000 7f8e9aa00640 1 -- 172.21.15.67:0/2255921889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c035020 con 0x7f8e9c15b570 2024-04-17T08:31:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.677+0000 7f8e9aa00640 1 -- 172.21.15.67:0/2255921889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e8c031870 con 0x7f8e9c15b570 2024-04-17T08:31:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8e9aa00640 1 -- 172.21.15.67:0/2255921889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c031c40 con 0x7f8e9c15b570 2024-04-17T08:31:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 -- 172.21.15.67:0/2255921889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 msgr2=0x7f8e9c06f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2255921889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c06f7a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c073040 tx=0x7f8e8c02b910 comp rx=0 tx=0).stop 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 -- 172.21.15.67:0/2255921889 shutdown_connections 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2255921889 >> v2:172.21.15.187:3300/0 conn(0x7f8e9c06fce0 0x7f8e9c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2255921889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c06f7a0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 -- 172.21.15.67:0/2255921889 >> 172.21.15.67:0/2255921889 conn(0x7f8e9c067d10 msgr2=0x7f8e9c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 -- 172.21.15.67:0/2255921889 shutdown_connections 2024-04-17T08:31:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.678+0000 7f8ea1400640 1 -- 172.21.15.67:0/2255921889 wait complete. 2024-04-17T08:31:16.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.680+0000 7f8ea1400640 1 Processor -- start 2024-04-17T08:31:16.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.680+0000 7f8ea1400640 1 -- start start 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 0x7f8e9c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e9c06fce0 0x7f8e9c15a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c0b4930 con 0x7f8e9c15b570 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c0b47b0 con 0x7f8e9c06be70 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.681+0000 7f8ea1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e9c06c2a0 con 0x7f8e9c06fce0 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35700/0 (socket says 172.21.15.67:35700) 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 -- 172.21.15.67:0/2590291511 learned_addr learned my addr 172.21.15.67:0/2590291511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8e9b400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e9c06fce0 0x7f8e9c15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 0x7f8e9c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 -- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e9c06fce0 msgr2=0x7f8e9c15a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e9c06fce0 0x7f8e9c15a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 -- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 msgr2=0x7f8e9c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 0x7f8e9c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8ea0a00640 1 -- 172.21.15.67:0/2590291511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c147f70 con 0x7f8e9c15b570 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.682+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 0x7f8e9c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8ea0a00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c7d0 tx=0x7f8e9400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94003b70 con 0x7f8e9c15b570 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e94003d30 con 0x7f8e9c15b570 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9c0adfd0 con 0x7f8e9c15b570 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400aa90 con 0x7f8e9c15b570 2024-04-17T08:31:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.683+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9c0ae590 con 0x7f8e9c15b570 2024-04-17T08:31:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.684+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e9c068a10 con 0x7f8e9c15b570 2024-04-17T08:31:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.685+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 98) ==== 91086+0+0 (secure 0 0 0) 0x7f8e9400ac30 con 0x7f8e9c15b570 2024-04-17T08:31:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.686+0000 7f8e93e00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 0x7f8e7c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:16.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.686+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f8e9408be60 con 0x7f8e9c15b570 2024-04-17T08:31:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.687+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 0x7f8e7c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.688+0000 7f8e9be00640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 0x7f8e7c06ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c036000 tx=0x7f8e8c03c040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:16.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.691+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8e9c147f70 con 0x7f8e9c15b570 2024-04-17T08:31:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.935+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e9c06f350 con 0x7f8e9c15b570 2024-04-17T08:31:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.937+0000 7f8e93e00640 1 -- 172.21.15.67:0/2590291511 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5209 (secure 0 0 0) 0x7f8e94059ed0 con 0x7f8e9c15b570 2024-04-17T08:31:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:16.936 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:30:51.593402+0000","modified":"2024-04-17T08:30:54.685999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6833/3206760811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3206760811},{"type":"v1","addr":"172.21.15.67:6833","nonce":3206760811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9462,"qdb_cluster":[9462]},"id":10}]} 2024-04-17T08:31:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 85 2024-04-17T08:31:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 msgr2=0x7f8e7c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 0x7f8e7c06ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c036000 tx=0x7f8e8c03c040 comp rx=0 tx=0).stop 2024-04-17T08:31:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 msgr2=0x7f8e9c147a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400c7d0 tx=0x7f8e9400cc90 comp rx=0 tx=0).stop 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 shutdown_connections 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e9c15b570 0x7f8e9c147a30 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e9c06fce0 0x7f8e9c15a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f8e7c06c980 0x7f8e7c06ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 --2- 172.21.15.67:0/2590291511 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e9c06be70 0x7f8e9c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.939+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 >> 172.21.15.67:0/2590291511 conn(0x7f8e9c067d10 msgr2=0x7f8e9c0cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.940+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 shutdown_connections 2024-04-17T08:31:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:16.940+0000 7f8ea1400640 1 -- 172.21.15.67:0/2590291511 wait complete. 2024-04-17T08:31:16.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:17.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.171+0000 7f592aa00640 1 Processor -- start 2024-04-17T08:31:17.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.171+0000 7f592aa00640 1 -- start start 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f592406b8c0 0x7f59240cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5924141ed0 0x7f5924142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592aa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59240cd4a0 con 0x7f5924160280 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592aa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f592415ce30 con 0x7f5924141ed0 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592aa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f59241607f0 con 0x7f592406b8c0 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f592a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5924141ed0 0x7f5924142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f5928c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5924160280 0x7f592406bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48416/0 (socket says 172.21.15.67:48416) 2024-04-17T08:31:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f5928c00640 1 -- 172.21.15.67:0/1652304343 learned_addr learned my addr 172.21.15.67:0/1652304343 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:17.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.173+0000 7f5929600640 1 --2- 172.21.15.67:0/1652304343 >> v2:172.21.15.187:3300/0 conn(0x7f592406b8c0 0x7f59240cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f5923e00640 1 -- 172.21.15.67:0/1652304343 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719747445 0 0) 0x7f59240cd4a0 con 0x7f5924160280 2024-04-17T08:31:17.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f5923e00640 1 -- 172.21.15.67:0/1652304343 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f590c003520 con 0x7f5924160280 2024-04-17T08:31:17.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f592a000640 1 -- 172.21.15.67:0/1652304343 >> v2:172.21.15.187:3300/0 conn(0x7f592406b8c0 msgr2=0x7f59240cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:17.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f592a000640 1 --2- 172.21.15.67:0/1652304343 >> v2:172.21.15.187:3300/0 conn(0x7f592406b8c0 0x7f59240cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f592a000640 1 -- 172.21.15.67:0/1652304343 >> v1:172.21.15.67:6789/0 conn(0x7f5924160280 legacy=0x7f592406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.174+0000 7f592a000640 1 -- 172.21.15.67:0/1652304343 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5924160ad0 con 0x7f5924141ed0 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.175+0000 7f592a000640 1 --2- 172.21.15.67:0/1652304343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5924141ed0 0x7f5924142360 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f59240b3210 tx=0x7f591c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a523622ed77ae385 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.175+0000 7f5923e00640 1 -- 172.21.15.67:0/1652304343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c039070 con 0x7f5924141ed0 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.175+0000 7f5923e00640 1 -- 172.21.15.67:0/1652304343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f591c02cc80 con 0x7f5924141ed0 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.175+0000 7f5923e00640 1 -- 172.21.15.67:0/1652304343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c034640 con 0x7f5924141ed0 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 -- 172.21.15.67:0/1652304343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5924141ed0 msgr2=0x7f5924142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 --2- 172.21.15.67:0/1652304343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5924141ed0 0x7f5924142360 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f59240b3210 tx=0x7f591c02bed0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 -- 172.21.15.67:0/1652304343 shutdown_connections 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 --2- 172.21.15.67:0/1652304343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5924141ed0 0x7f5924142360 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 --2- 172.21.15.67:0/1652304343 >> v2:172.21.15.187:3300/0 conn(0x7f592406b8c0 0x7f59240cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 -- 172.21.15.67:0/1652304343 >> 172.21.15.67:0/1652304343 conn(0x7f5924067d10 msgr2=0x7f59240b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 -- 172.21.15.67:0/1652304343 shutdown_connections 2024-04-17T08:31:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.176+0000 7f592aa00640 1 -- 172.21.15.67:0/1652304343 wait complete. 2024-04-17T08:31:17.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.178+0000 7f592aa00640 1 Processor -- start 2024-04-17T08:31:17.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.178+0000 7f592aa00640 1 -- start start 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5924141ed0 0x7f592415f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924160280 0x7f592415e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5924161090 con 0x7f592406b8c0 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f59240cddf0 con 0x7f5924160280 2024-04-17T08:31:17.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f592aa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f592413f9b0 con 0x7f5924141ed0 2024-04-17T08:31:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f5929600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f5929600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35720/0 (socket says 172.21.15.67:35720) 2024-04-17T08:31:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.179+0000 7f5929600640 1 -- 172.21.15.67:0/1982418965 learned_addr learned my addr 172.21.15.67:0/1982418965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 -- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5924141ed0 msgr2=0x7f592415f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5924141ed0 0x7f592415f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f592a000640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924160280 0x7f592415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 -- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924160280 msgr2=0x7f592415e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924160280 0x7f592415e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 -- 172.21.15.67:0/1982418965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5924144740 con 0x7f592406b8c0 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5929600640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f591800d830 tx=0x7f591800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591800b6c0 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918019070 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591800bac0 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.180+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5924144a20 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.181+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5924144fe0 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.182+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5924068a10 con 0x7f592406b8c0 2024-04-17T08:31:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.183+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 98) ==== 91086+0+0 (secure 0 0 0) 0x7f591801e460 con 0x7f592406b8c0 2024-04-17T08:31:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.183+0000 7f5921600640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 0x7f590406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.184+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f591808c6e0 con 0x7f592406b8c0 2024-04-17T08:31:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.184+0000 7f5928c00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 0x7f590406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.185+0000 7f5928c00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 0x7f590406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5910000f60 tx=0x7f59100078f0 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.188+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f591805a750 con 0x7f592406b8c0 2024-04-17T08:31:17.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.391+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5924159390 con 0x7f592406b8c0 2024-04-17T08:31:17.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.394+0000 7f5921600640 1 -- 172.21.15.67:0/1982418965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) ==== 74+0+17691 (secure 0 0 0) 0x7f5918089530 con 0x7f592406b8c0 2024-04-17T08:31:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:17.394 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":184,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:13.813303+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:30:49.565353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-04-17T08:30:50.604764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2335112946":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/3177304055":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6832/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2645268054":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1428143598":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/3079350952":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1712983456":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2676684388":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/863702549":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6832/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2751617585":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2559339550":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/1037073318":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2642787856":"2024-04-18T08:31:06.484301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:17.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 msgr2=0x7f590406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 0x7f590406ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5910000f60 tx=0x7f59100078f0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 msgr2=0x7f592413b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f591800d830 tx=0x7f591800dcf0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 shutdown_connections 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f590406c980 0x7f590406ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5924160280 0x7f592415e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5924141ed0 0x7f592415f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 --2- 172.21.15.67:0/1982418965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f592406b8c0 0x7f592413b8e0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 >> 172.21.15.67:0/1982418965 conn(0x7f5924067d10 msgr2=0x7f5924074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 shutdown_connections 2024-04-17T08:31:17.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.397+0000 7f592aa00640 1 -- 172.21.15.67:0/1982418965 wait complete. 2024-04-17T08:31:17.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:31:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.637+0000 7f1585400640 1 Processor -- start 2024-04-17T08:31:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.637+0000 7f1585400640 1 -- start start 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f1585400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1580145ac0 0x7f1580071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f1585400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1580072d80 0x7f15800cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f1585400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15800cea70 con 0x7f15800ce3b0 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f1585400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f15800ce810 con 0x7f1580072d80 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f1585400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15800ced50 con 0x7f1580145ac0 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f157f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1580072d80 0x7f15800cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f157f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1580072d80 0x7f15800cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35736/0 (socket says 172.21.15.67:35736) 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.639+0000 7f157f200640 1 -- 172.21.15.67:0/960977484 learned_addr learned my addr 172.21.15.67:0/960977484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.640+0000 7f157e800640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1580145ac0 0x7f1580071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.640+0000 7f157f200640 1 -- 172.21.15.67:0/960977484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1580145ac0 msgr2=0x7f1580071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.640+0000 7f157f200640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1580145ac0 0x7f1580071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.640+0000 7f157f200640 1 -- 172.21.15.67:0/960977484 >> v1:172.21.15.187:6789/0 conn(0x7f15800ce3b0 legacy=0x7f1580072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:17.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.640+0000 7f157f200640 1 -- 172.21.15.67:0/960977484 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15800cf030 con 0x7f1580072d80 2024-04-17T08:31:17.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.641+0000 7f157f200640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f15800cfb10 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f1578009e50 tx=0x7f157802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddb73162d548ac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.641+0000 7f157d400640 1 -- 172.21.15.67:0/960977484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157802ca70 con 0x7f1580072d80 2024-04-17T08:31:17.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.641+0000 7f157d400640 1 -- 172.21.15.67:0/960977484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f157802cc10 con 0x7f1580072d80 2024-04-17T08:31:17.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.641+0000 7f157d400640 1 -- 172.21.15.67:0/960977484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780343f0 con 0x7f1580072d80 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 -- 172.21.15.67:0/960977484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 msgr2=0x7f15800cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f15800cfb10 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f1578009e50 tx=0x7f157802bed0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 -- 172.21.15.67:0/960977484 shutdown_connections 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f15800cfb10 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 --2- 172.21.15.67:0/960977484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1580145ac0 0x7f1580071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 -- 172.21.15.67:0/960977484 >> 172.21.15.67:0/960977484 conn(0x7f1580067ee0 msgr2=0x7f158006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 -- 172.21.15.67:0/960977484 shutdown_connections 2024-04-17T08:31:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.642+0000 7f1585400640 1 -- 172.21.15.67:0/960977484 wait complete. 2024-04-17T08:31:17.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.644+0000 7f1585400640 1 Processor -- start 2024-04-17T08:31:17.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.644+0000 7f1585400640 1 -- start start 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15800ce3b0 0x7f158013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1580145ac0 0x7f158013fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1580145ea0 con 0x7f1580072d80 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15800cf5f0 con 0x7f1580145ac0 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f1585400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15800d0050 con 0x7f15800ce3b0 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f157e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f157e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35742/0 (socket says 172.21.15.67:35742) 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.645+0000 7f157e800640 1 -- 172.21.15.67:0/3954629534 learned_addr learned my addr 172.21.15.67:0/3954629534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 -- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15800ce3b0 msgr2=0x7f158013f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15800ce3b0 0x7f158013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 -- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1580145ac0 msgr2=0x7f158013fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1580145ac0 0x7f158013fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 -- 172.21.15.67:0/3954629534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158015d5a0 con 0x7f1580072d80 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f157e800640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f157000d5f0 tx=0x7f157000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700125f0 con 0x7f1580072d80 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1570019070 con 0x7f1580072d80 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.646+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700129d0 con 0x7f1580072d80 2024-04-17T08:31:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.647+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158013b680 con 0x7f1580072d80 2024-04-17T08:31:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.647+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158015d860 con 0x7f1580072d80 2024-04-17T08:31:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.648+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15800b4340 con 0x7f1580072d80 2024-04-17T08:31:17.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.649+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 98) ==== 91086+0+0 (secure 0 0 0) 0x7f1570012b70 con 0x7f1580072d80 2024-04-17T08:31:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.650+0000 7f1576a00640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 0x7f156006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.650+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6425+0+0 (secure 0 0 0) 0x7f157008c290 con 0x7f1580072d80 2024-04-17T08:31:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.651+0000 7f157de00640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 0x7f156006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.652+0000 7f157de00640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 0x7f156006ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1580140610 tx=0x7f156c0079b0 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:17.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.655+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f15800b4340 con 0x7f1580072d80 2024-04-17T08:31:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.869+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f1570056370 con 0x7f1580072d80 2024-04-17T08:31:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.910+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f15800cc6c0 con 0x7f1580072d80 2024-04-17T08:31:17.911 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:31:17.912+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:31:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.917+0000 7f1576a00640 1 -- 172.21.15.67:0/3954629534 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v86) ==== 161+0+0 (secure 0 0 0) 0x7f1570075220 con 0x7f1580072d80 2024-04-17T08:31:17.916 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 msgr2=0x7f156006ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 0x7f156006ee90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1580140610 tx=0x7f156c0079b0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 msgr2=0x7f1580138d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f157000d5f0 tx=0x7f157000dab0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 shutdown_connections 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f156006c9d0 0x7f156006ee90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1580145ac0 0x7f158013fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15800ce3b0 0x7f158013f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 --2- 172.21.15.67:0/3954629534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1580072d80 0x7f1580138d30 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.920+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 >> 172.21.15.67:0/3954629534 conn(0x7f1580067ee0 msgr2=0x7f15800b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.921+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 shutdown_connections 2024-04-17T08:31:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:17.921+0000 7f1585400640 1 -- 172.21.15.67:0/3954629534 wait complete. 2024-04-17T08:31:17.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:31:17.955 INFO:tasks.ceph.mds.a.smithi067.stderr: -4504> 2024-04-17T08:30:55.422+0000 7f4dd9800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:31:18.030 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:31:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.153+0000 7fe464c00640 1 Processor -- start 2024-04-17T08:31:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.153+0000 7fe464c00640 1 -- start start 2024-04-17T08:31:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe464c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe460072d80 0x7fe460071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe464c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4600cea70 con 0x7fe460145ac0 2024-04-17T08:31:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe464c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4600ced50 con 0x7fe4600ce3b0 2024-04-17T08:31:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe464c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe4600ce820 con 0x7fe460072d80 2024-04-17T08:31:18.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe45de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe460145ac0 0x7fe460072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48442/0 (socket says 172.21.15.67:48442) 2024-04-17T08:31:18.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.154+0000 7fe45de00640 1 -- 172.21.15.67:0/1481855573 learned_addr learned my addr 172.21.15.67:0/1481855573 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883840714 0 0) 0x7fe4600cea70 con 0x7fe460145ac0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe444003520 con 0x7fe460145ac0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe455400640 1 --2- 172.21.15.67:0/1481855573 >> v2:172.21.15.187:3300/0 conn(0x7fe460072d80 0x7fe460071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1456546028 0 0) 0x7fe444003520 con 0x7fe460145ac0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 >> v2:172.21.15.187:3300/0 conn(0x7fe460072d80 msgr2=0x7fe460071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 --2- 172.21.15.67:0/1481855573 >> v2:172.21.15.187:3300/0 conn(0x7fe460072d80 0x7fe460071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 >> v1:172.21.15.187:6789/0 conn(0x7fe4600ce3b0 legacy=0x7fe4600cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4600cf030 con 0x7fe460145ac0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe450001670 con 0x7fe460145ac0 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 >> v1:172.21.15.67:6789/0 conn(0x7fe460145ac0 legacy=0x7fe460072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4440039b0 0x7fe444003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.155+0000 7fe45d400640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe444004800 0x7fe444004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45d400640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 0x7fe44400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4600ce820 con 0x7fe4440051c0 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe44400c0f0 con 0x7fe4440039b0 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe44400c2f0 con 0x7fe444004800 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 0x7fe44400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45de00640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4440039b0 0x7fe444003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe455400640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe444004800 0x7fe444004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 -- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe444004800 msgr2=0x7fe444004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe444004800 0x7fe444004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 -- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4440039b0 msgr2=0x7fe444003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4440039b0 0x7fe444003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.156+0000 7fe45e800640 1 -- 172.21.15.67:0/1481855573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe444003520 con 0x7fe4440051c0 2024-04-17T08:31:18.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.157+0000 7fe45e800640 1 -- 172.21.15.67:0/1481855573 reap_dead start 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.157+0000 7fe45e800640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 0x7fe44400b9c0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe458004480 tx=0x7fe444005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c29d88fee068b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458034070 con 0x7fe4440051c0 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe458006b50 con 0x7fe4440051c0 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe45d400640 1 -- 172.21.15.67:0/1481855573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458004080 con 0x7fe4440051c0 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 -- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 msgr2=0x7fe44400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 0x7fe44400b9c0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe458004480 tx=0x7fe444005830 comp rx=0 tx=0).stop 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 -- 172.21.15.67:0/1481855573 shutdown_connections 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 --2- 172.21.15.67:0/1481855573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4440051c0 0x7fe44400b9c0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 -- 172.21.15.67:0/1481855573 >> 172.21.15.67:0/1481855573 conn(0x7fe460067ee0 msgr2=0x7fe46006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.158+0000 7fe464c00640 1 -- 172.21.15.67:0/1481855573 shutdown_connections 2024-04-17T08:31:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.159+0000 7fe464c00640 1 -- 172.21.15.67:0/1481855573 wait complete. 2024-04-17T08:31:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.160+0000 7fe464c00640 1 Processor -- start 2024-04-17T08:31:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.160+0000 7fe464c00640 1 -- start start 2024-04-17T08:31:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.161+0000 7fe464c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4440051c0 0x7fe46013a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.161+0000 7fe464c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe46015c5e0 0x7fe46006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.161+0000 7fe464c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe464c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe460145ea0 con 0x7fe46013ea80 2024-04-17T08:31:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe464c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe4600cf5f0 con 0x7fe46015c5e0 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe464c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe460071ea0 con 0x7fe4440051c0 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35790/0 (socket says 172.21.15.67:35790) 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 -- 172.21.15.67:0/845965387 learned_addr learned my addr 172.21.15.67:0/845965387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45de00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4440051c0 0x7fe46013a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe455400640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe46015c5e0 0x7fe46006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 -- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4440051c0 msgr2=0x7fe46013a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4440051c0 0x7fe46013a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 -- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe46015c5e0 msgr2=0x7fe46006e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe46015c5e0 0x7fe46006e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.162+0000 7fe45e800640 1 -- 172.21.15.67:0/845965387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46013aa50 con 0x7fe46013ea80 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe45e800640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fe458004480 tx=0x7fe458003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458034040 con 0x7fe46013ea80 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe458038600 con 0x7fe46013ea80 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458007df0 con 0x7fe46013ea80 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46013ad30 con 0x7fe46013ea80 2024-04-17T08:31:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.163+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46013b2f0 con 0x7fe46013ea80 2024-04-17T08:31:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.164+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe460068ca0 con 0x7fe46013ea80 2024-04-17T08:31:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.165+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fe4580389f0 con 0x7fe46013ea80 2024-04-17T08:31:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.166+0000 7fe456a00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 0x7fe43c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.166+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6511+0+0 (secure 0 0 0) 0x7fe4580ad450 con 0x7fe46013ea80 2024-04-17T08:31:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.167+0000 7fe45de00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 0x7fe43c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.168+0000 7fe45de00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 0x7fe43c06ef00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe450005a70 tx=0x7fe450009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.171+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe45804a030 con 0x7fe46013ea80 2024-04-17T08:31:18.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.417+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe4600cc6c0 con 0x7fe46013ea80 2024-04-17T08:31:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.921+0000 7fe456a00640 1 -- 172.21.15.67:0/845965387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v87) ==== 104+0+0 (secure 0 0 0) 0x7fe45807b330 con 0x7fe46013ea80 2024-04-17T08:31:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 msgr2=0x7fe43c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 0x7fe43c06ef00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe450005a70 tx=0x7fe450009040 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 msgr2=0x7fe46013ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fe458004480 tx=0x7fe458003a70 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 shutdown_connections 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe46013ea80 0x7fe46013ef30 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe43c06ca40 0x7fe43c06ef00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe46015c5e0 0x7fe46006e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 --2- 172.21.15.67:0/845965387 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4440051c0 0x7fe46013a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 >> 172.21.15.67:0/845965387 conn(0x7fe460067ee0 msgr2=0x7fe4600b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 shutdown_connections 2024-04-17T08:31:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:18.925+0000 7fe464c00640 1 -- 172.21.15.67:0/845965387 wait complete. 2024-04-17T08:31:18.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:31:19.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:31:19.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:31:19.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.215+0000 7f3714200640 1 Processor -- start 2024-04-17T08:31:19.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.215+0000 7f3714200640 1 -- start start 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.216+0000 7f3714200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f370c06e710 0x7f370c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.216+0000 7f3714200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c06c210 con 0x7f370c0cfe90 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.216+0000 7f3714200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c15d210 con 0x7f370c06be00 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.216+0000 7f3714200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f370c15d4f0 con 0x7f370c06e710 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3711800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f370c0cfe90 0x7f370c0cd940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48456/0 (socket says 172.21.15.67:48456) 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3711800640 1 -- 172.21.15.67:0/2888157869 learned_addr learned my addr 172.21.15.67:0/2888157869 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3710e00640 1 --2- 172.21.15.67:0/2888157869 >> v2:172.21.15.67:3300/0 conn(0x7f370c06e710 0x7f370c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3710e00640 1 -- 172.21.15.67:0/2888157869 >> v1:172.21.15.187:6789/0 conn(0x7f370c06be00 legacy=0x7f370c0722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3710e00640 1 -- 172.21.15.67:0/2888157869 >> v1:172.21.15.67:6789/0 conn(0x7f370c0cfe90 legacy=0x7f370c0cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.217+0000 7f3710e00640 1 -- 172.21.15.67:0/2888157869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c15d7d0 con 0x7f370c06e710 2024-04-17T08:31:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.218+0000 7f3710e00640 1 --2- 172.21.15.67:0/2888157869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c06e710 0x7f370c06b8c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f36f80097a0 tx=0x7f36f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185a639d4b1abea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.219+0000 7f370be00640 1 -- 172.21.15.67:0/2888157869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8035020 con 0x7f370c06e710 2024-04-17T08:31:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.219+0000 7f370be00640 1 -- 172.21.15.67:0/2888157869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f8031830 con 0x7f370c06e710 2024-04-17T08:31:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.219+0000 7f370be00640 1 -- 172.21.15.67:0/2888157869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8031c00 con 0x7f370c06e710 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.219+0000 7f3714200640 1 -- 172.21.15.67:0/2888157869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c06e710 msgr2=0x7f370c06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.219+0000 7f3714200640 1 --2- 172.21.15.67:0/2888157869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c06e710 0x7f370c06b8c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f36f80097a0 tx=0x7f36f802b8d0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.220+0000 7f3714200640 1 -- 172.21.15.67:0/2888157869 shutdown_connections 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.220+0000 7f3714200640 1 --2- 172.21.15.67:0/2888157869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c06e710 0x7f370c06b8c0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.220+0000 7f3714200640 1 -- 172.21.15.67:0/2888157869 >> 172.21.15.67:0/2888157869 conn(0x7f370c0b2370 msgr2=0x7f370c0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.220+0000 7f3714200640 1 -- 172.21.15.67:0/2888157869 shutdown_connections 2024-04-17T08:31:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.220+0000 7f3714200640 1 -- 172.21.15.67:0/2888157869 wait complete. 2024-04-17T08:31:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.221+0000 7f3714200640 1 Processor -- start 2024-04-17T08:31:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.222+0000 7f3714200640 1 -- start start 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.222+0000 7f3714200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c06be00 0x7f370c1f8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3714200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c06e710 0x7f370c1f8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3714200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3714200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f370c0d0270 con 0x7f370c0cfe90 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3714200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f370c15dd90 con 0x7f370c06be00 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3714200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f370c15ce90 con 0x7f370c06e710 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35810/0 (socket says 172.21.15.67:35810) 2024-04-17T08:31:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 -- 172.21.15.67:0/1861070136 learned_addr learned my addr 172.21.15.67:0/1861070136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3711800640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c06be00 0x7f370c1f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3710e00640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c06e710 0x7f370c1f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 -- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c06e710 msgr2=0x7f370c1f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c06e710 0x7f370c1f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 -- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c06be00 msgr2=0x7f370c1f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c06be00 0x7f370c1f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.223+0000 7f3712200640 1 -- 172.21.15.67:0/1861070136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c0af240 con 0x7f370c0cfe90 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3712200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f370400c780 tx=0x7f370400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704004b60 con 0x7f370c0cfe90 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3704004d00 con 0x7f370c0cfe90 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c15cb30 con 0x7f370c0cfe90 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370400aa90 con 0x7f370c0cfe90 2024-04-17T08:31:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.224+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c0b3b60 con 0x7f370c0cfe90 2024-04-17T08:31:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.226+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370c073980 con 0x7f370c0cfe90 2024-04-17T08:31:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.227+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f370400ac30 con 0x7f370c0cfe90 2024-04-17T08:31:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.228+0000 7f3709600640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 0x7f36ec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.228+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6511+0+0 (secure 0 0 0) 0x7f370408c640 con 0x7f370c0cfe90 2024-04-17T08:31:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.229+0000 7f3711800640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 0x7f36ec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:19.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.230+0000 7f3711800640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 0x7f36ec06ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f370c145ca0 tx=0x7f3700009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:19.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.233+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f370405a5a0 con 0x7f370c0cfe90 2024-04-17T08:31:19.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.440+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f370c068d60 con 0x7f370c0cfe90 2024-04-17T08:31:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.925+0000 7f3709600640 1 -- 172.21.15.67:0/1861070136 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v186) ==== 181+0+0 (secure 0 0 0) 0x7f3704059ea0 con 0x7f370c0cfe90 2024-04-17T08:31:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:31:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.930+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 msgr2=0x7f36ec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.930+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 0x7f36ec06ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f370c145ca0 tx=0x7f3700009040 comp rx=0 tx=0).stop 2024-04-17T08:31:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.930+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 msgr2=0x7f370c0ae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.930+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f370400c780 tx=0x7f370400cc40 comp rx=0 tx=0).stop 2024-04-17T08:31:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 shutdown_connections 2024-04-17T08:31:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f370c0cfe90 0x7f370c0ae4e0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f370c06e710 0x7f370c1f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f36ec06ca40 0x7f36ec06ef00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 --2- 172.21.15.67:0/1861070136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f370c06be00 0x7f370c1f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.931+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 >> 172.21.15.67:0/1861070136 conn(0x7f370c0b2370 msgr2=0x7f370c15e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.932+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 shutdown_connections 2024-04-17T08:31:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:19.932+0000 7f3714200640 1 -- 172.21.15.67:0/1861070136 wait complete. 2024-04-17T08:31:19.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:31:20.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.232+0000 7f2f9f200640 1 Processor -- start 2024-04-17T08:31:20.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.232+0000 7f2f9f200640 1 -- start start 2024-04-17T08:31:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2fa00cd2f0 0x7f2fa00cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fa00cd6d0 0x7f2fa0145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa0146510 con 0x7f2fa0146080 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa0161000 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa00ce870 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fa00cd6d0 0x7f2fa0145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fa00cd6d0 0x7f2fa0145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35818/0 (socket says 172.21.15.67:35818) 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.234+0000 7f2f9d400640 1 -- 172.21.15.67:0/715655723 learned_addr learned my addr 172.21.15.67:0/715655723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.235+0000 7f2f9d400640 1 -- 172.21.15.67:0/715655723 >> v2:172.21.15.187:3300/0 conn(0x7f2fa00cd2f0 msgr2=0x7f2fa00cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:20.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.235+0000 7f2f9d400640 1 --2- 172.21.15.67:0/715655723 >> v2:172.21.15.187:3300/0 conn(0x7f2fa00cd2f0 0x7f2fa00cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.235+0000 7f2f9d400640 1 -- 172.21.15.67:0/715655723 >> v1:172.21.15.187:6789/0 conn(0x7f2fa0146080 legacy=0x7f2fa0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:20.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.235+0000 7f2f9d400640 1 -- 172.21.15.67:0/715655723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa00ceb50 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.236+0000 7f2f9d400640 1 --2- 172.21.15.67:0/715655723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd6d0 0x7f2fa0145b40 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009870 tx=0x7f2f8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b106def1b26d1d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.236+0000 7f2f9ca00640 1 -- 172.21.15.67:0/715655723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c035020 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9ca00640 1 -- 172.21.15.67:0/715655723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c031870 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9ca00640 1 -- 172.21.15.67:0/715655723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c031c40 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 -- 172.21.15.67:0/715655723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd6d0 msgr2=0x7f2fa0145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 --2- 172.21.15.67:0/715655723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd6d0 0x7f2fa0145b40 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009870 tx=0x7f2f8c02b910 comp rx=0 tx=0).stop 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 -- 172.21.15.67:0/715655723 shutdown_connections 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 --2- 172.21.15.67:0/715655723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd6d0 0x7f2fa0145b40 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 --2- 172.21.15.67:0/715655723 >> v2:172.21.15.187:3300/0 conn(0x7f2fa00cd2f0 0x7f2fa00cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.237+0000 7f2f9f200640 1 -- 172.21.15.67:0/715655723 >> 172.21.15.67:0/715655723 conn(0x7f2fa006b8c0 msgr2=0x7f2fa006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.238+0000 7f2f9f200640 1 -- 172.21.15.67:0/715655723 shutdown_connections 2024-04-17T08:31:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.238+0000 7f2f9f200640 1 -- 172.21.15.67:0/715655723 wait complete. 2024-04-17T08:31:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.240+0000 7f2f9f200640 1 Processor -- start 2024-04-17T08:31:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.240+0000 7f2f9f200640 1 -- start start 2024-04-17T08:31:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.241+0000 7f2f9f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.241+0000 7f2f9f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fa00cd6d0 0x7f2fa015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.241+0000 7f2f9f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fa0146080 0x7f2fa013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa00cfa10 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa00cf890 con 0x7f2fa0146080 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa0160a70 con 0x7f2fa00cd6d0 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35820/0 (socket says 172.21.15.67:35820) 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 -- 172.21.15.67:0/2915609670 learned_addr learned my addr 172.21.15.67:0/2915609670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9e800640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fa0146080 0x7f2fa013b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9d400640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fa00cd6d0 0x7f2fa015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 -- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fa00cd6d0 msgr2=0x7f2fa015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fa00cd6d0 0x7f2fa015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 -- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fa0146080 msgr2=0x7f2fa013b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fa0146080 0x7f2fa013b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.242+0000 7f2f9de00640 1 -- 172.21.15.67:0/2915609670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa015ec20 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f9de00640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000b4a0 tx=0x7f2f9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90014070 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f90010610 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90010a00 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa015ef00 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.243+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa015ce70 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.245+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa0068a10 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.246+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f2f90017070 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.247+0000 7f2f96000640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 0x7f2f8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:20.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.247+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 6073+0+0 (secure 0 0 0) 0x7f2f9008b950 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.248+0000 7f2f9d400640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 0x7f2f8006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.249+0000 7f2f9d400640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 0x7f2f8006eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c036000 tx=0x7f2f8c03c040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.252+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2f90059a60 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.458+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2fa0068610 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.940+0000 7f2f96000640 1 -- 172.21.15.67:0/2915609670 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v187) ==== 169+0+0 (secure 0 0 0) 0x7f2f90059360 con 0x7f2fa00cd2f0 2024-04-17T08:31:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:31:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.944+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 msgr2=0x7f2f8006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.944+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 0x7f2f8006eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c036000 tx=0x7f2f8c03c040 comp rx=0 tx=0).stop 2024-04-17T08:31:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.944+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 msgr2=0x7f2fa015e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.944+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000b4a0 tx=0x7f2f9000b960 comp rx=0 tx=0).stop 2024-04-17T08:31:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 shutdown_connections 2024-04-17T08:31:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f2f8006ca10 0x7f2f8006eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fa0146080 0x7f2fa013b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2fa00cd6d0 0x7f2fa015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 --2- 172.21.15.67:0/2915609670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa00cd2f0 0x7f2fa015e560 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.946+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 >> 172.21.15.67:0/2915609670 conn(0x7f2fa006b8c0 msgr2=0x7f2fa00b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.948+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 shutdown_connections 2024-04-17T08:31:20.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:20.948+0000 7f2f9f200640 1 -- 172.21.15.67:0/2915609670 wait complete. 2024-04-17T08:31:20.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:31:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.181+0000 7efebaa00640 1 Processor -- start 2024-04-17T08:31:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.182+0000 7efebaa00640 1 -- start start 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efeb415ecb0 0x7efeb40d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb40d3210 0x7efeb40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efeb40cf8e0 0x7efeb40d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efeb40d0c30 con 0x7efeb415ecb0 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efeb40d00b0 con 0x7efeb40cf8e0 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efebaa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efeb40d0390 con 0x7efeb40d3210 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.183+0000 7efeb9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efeb415ecb0 0x7efeb40d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efeb415ecb0 0x7efeb40d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35832/0 (socket says 172.21.15.67:35832) 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 -- 172.21.15.67:0/4012605080 learned_addr learned my addr 172.21.15.67:0/4012605080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 -- 172.21.15.67:0/4012605080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb40d3210 msgr2=0x7efeb40cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 --2- 172.21.15.67:0/4012605080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb40d3210 0x7efeb40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 -- 172.21.15.67:0/4012605080 >> v2:172.21.15.187:3300/0 conn(0x7efeb40cf8e0 msgr2=0x7efeb40d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 --2- 172.21.15.67:0/4012605080 >> v2:172.21.15.187:3300/0 conn(0x7efeb40cf8e0 0x7efeb40d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.184+0000 7efeb9600640 1 -- 172.21.15.67:0/4012605080 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb40d0670 con 0x7efeb415ecb0 2024-04-17T08:31:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.185+0000 7efeb9600640 1 --2- 172.21.15.67:0/4012605080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb415ecb0 0x7efeb40d2cd0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7efeb40b61a0 tx=0x7efea802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202d51517db698b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.185+0000 7efeb3e00640 1 -- 172.21.15.67:0/4012605080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea802c9f0 con 0x7efeb415ecb0 2024-04-17T08:31:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.185+0000 7efeb3e00640 1 -- 172.21.15.67:0/4012605080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea802cb90 con 0x7efeb415ecb0 2024-04-17T08:31:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efeb3e00640 1 -- 172.21.15.67:0/4012605080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea80343f0 con 0x7efeb415ecb0 2024-04-17T08:31:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 -- 172.21.15.67:0/4012605080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb415ecb0 msgr2=0x7efeb40d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 --2- 172.21.15.67:0/4012605080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb415ecb0 0x7efeb40d2cd0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7efeb40b61a0 tx=0x7efea802be50 comp rx=0 tx=0).stop 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 -- 172.21.15.67:0/4012605080 shutdown_connections 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 --2- 172.21.15.67:0/4012605080 >> v2:172.21.15.187:3300/0 conn(0x7efeb40cf8e0 0x7efeb40d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 --2- 172.21.15.67:0/4012605080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb40d3210 0x7efeb40cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 --2- 172.21.15.67:0/4012605080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb415ecb0 0x7efeb40d2cd0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 -- 172.21.15.67:0/4012605080 >> 172.21.15.67:0/4012605080 conn(0x7efeb408bff0 msgr2=0x7efeb40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 -- 172.21.15.67:0/4012605080 shutdown_connections 2024-04-17T08:31:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.186+0000 7efebaa00640 1 -- 172.21.15.67:0/4012605080 wait complete. 2024-04-17T08:31:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.188+0000 7efebaa00640 1 Processor -- start 2024-04-17T08:31:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.188+0000 7efebaa00640 1 -- start start 2024-04-17T08:31:21.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 0x7efeb40b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 0x7efeb414d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efeb415f090 con 0x7efeb40d3210 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40cfd70 con 0x7efeb40cf8e0 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.189+0000 7efebaa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efeb415f210 con 0x7efeb415ecb0 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35836/0 (socket says 172.21.15.67:35836) 2024-04-17T08:31:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 -- 172.21.15.67:0/2727270953 learned_addr learned my addr 172.21.15.67:0/2727270953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb9600640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 0x7efeb40b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeba000640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 0x7efeb414d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 -- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 msgr2=0x7efeb414d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 0x7efeb414d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 -- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 msgr2=0x7efeb40b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 0x7efeb40b8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 -- 172.21.15.67:0/2727270953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb40b8b70 con 0x7efeb40d3210 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb9600640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 0x7efeb40b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeba000640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 0x7efeb414d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.190+0000 7efeb8c00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7efea000b4a0 tx=0x7efea000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.191+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0014070 con 0x7efeb40d3210 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.191+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea0010610 con 0x7efeb40d3210 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.191+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0010a00 con 0x7efeb40d3210 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.191+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb40b8e50 con 0x7efeb40d3210 2024-04-17T08:31:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.191+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb40b9410 con 0x7efeb40d3210 2024-04-17T08:31:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.192+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb4068a10 con 0x7efeb40d3210 2024-04-17T08:31:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.193+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7efea0017070 con 0x7efeb40d3210 2024-04-17T08:31:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.194+0000 7efeb1600640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 0x7efe9406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.194+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7efea008b740 con 0x7efeb40d3210 2024-04-17T08:31:21.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.195+0000 7efeb9600640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 0x7efe9406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.196+0000 7efeb9600640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 0x7efe9406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efea8035000 tx=0x7efea8030000 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.199+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efea0059a90 con 0x7efeb40d3210 2024-04-17T08:31:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.412+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7efeb406e400 con 0x7efeb40d3210 2024-04-17T08:31:21.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.420+0000 7efeb1600640 1 -- 172.21.15.67:0/2727270953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) ==== 115+0+0 (secure 0 0 0) 0x7efea0059390 con 0x7efeb40d3210 2024-04-17T08:31:21.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 msgr2=0x7efe9406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 0x7efe9406eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efea8035000 tx=0x7efea8030000 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 msgr2=0x7efeb40cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7efea000b4a0 tx=0x7efea000b960 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 shutdown_connections 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efeb415ecb0 0x7efeb414d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb40d3210 0x7efeb40cba20 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7efe9406ca10 0x7efe9406eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 --2- 172.21.15.67:0/2727270953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efeb40cf8e0 0x7efeb40b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 >> 172.21.15.67:0/2727270953 conn(0x7efeb408bff0 msgr2=0x7efeb406d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 shutdown_connections 2024-04-17T08:31:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.423+0000 7efebaa00640 1 -- 172.21.15.67:0/2727270953 wait complete. 2024-04-17T08:31:21.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:31:21.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.677+0000 7fa28a200640 1 Processor -- start 2024-04-17T08:31:21.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.677+0000 7fa28a200640 1 -- start start 2024-04-17T08:31:21.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.678+0000 7fa28a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2840d0b50 0x7fa28415fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.678+0000 7fa28a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa284163710 con 0x7fa2841627b0 2024-04-17T08:31:21.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.678+0000 7fa28a200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa28415f360 con 0x7fa28406b8c0 2024-04-17T08:31:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.678+0000 7fa28a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa284163150 con 0x7fa2840d0b50 2024-04-17T08:31:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa283e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2841627b0 0x7fa2840cdf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48458/0 (socket says 172.21.15.67:48458) 2024-04-17T08:31:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa283e00640 1 -- 172.21.15.67:0/265864959 learned_addr learned my addr 172.21.15.67:0/265864959 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:21.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282a00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2840d0b50 0x7fa28415fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219157418 0 0) 0x7fa284163710 con 0x7fa2841627b0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa268003520 con 0x7fa2841627b0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 569781704 0 0) 0x7fa268003520 con 0x7fa2841627b0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2840d0b50 msgr2=0x7fa28415fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2840d0b50 0x7fa28415fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 >> v1:172.21.15.187:6789/0 conn(0x7fa28406b8c0 legacy=0x7fa28406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa284163430 con 0x7fa2841627b0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa27c001670 con 0x7fa2841627b0 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.679+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 >> v1:172.21.15.67:6789/0 conn(0x7fa2841627b0 legacy=0x7fa2840cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680039e0 0x7fa268003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa268004800 0x7fa268004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680051c0 0x7fa26800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa284163150 con 0x7fa2680051c0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa26800c090 con 0x7fa268004800 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa26800c290 con 0x7fa2680039e0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282a00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2840d0b50 0x7fa28415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa283e00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680051c0 0x7fa26800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa283e00640 1 -- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680039e0 msgr2=0x7fa268003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa283400640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680039e0 0x7fa268003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa283e00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680039e0 0x7fa268003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa282a00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa268004800 0x7fa268004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.680+0000 7fa283e00640 1 -- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa268004800 msgr2=0x7fa268004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.681+0000 7fa283e00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa268004800 0x7fa268004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.681+0000 7fa283e00640 1 -- 172.21.15.67:0/265864959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa268003520 con 0x7fa2680051c0 2024-04-17T08:31:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.681+0000 7fa283400640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680039e0 0x7fa268003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:21.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa283e00640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680051c0 0x7fa26800b9c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00c210 tx=0x7fa268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fae4ef8bae77dd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c007d30 con 0x7fa2680051c0 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa27c030070 con 0x7fa2680051c0 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa282000640 1 -- 172.21.15.67:0/265864959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c02cd60 con 0x7fa2680051c0 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa28a200640 1 -- 172.21.15.67:0/265864959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680051c0 msgr2=0x7fa26800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa28a200640 1 --2- 172.21.15.67:0/265864959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680051c0 0x7fa26800b9c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00c210 tx=0x7fa268005830 comp rx=0 tx=0).stop 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.682+0000 7fa283e00640 1 -- 172.21.15.67:0/265864959 reap_dead start 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.683+0000 7fa28a200640 1 -- 172.21.15.67:0/265864959 shutdown_connections 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.683+0000 7fa28a200640 1 -- 172.21.15.67:0/265864959 >> 172.21.15.67:0/265864959 conn(0x7fa284073360 msgr2=0x7fa28409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.683+0000 7fa28a200640 1 -- 172.21.15.67:0/265864959 shutdown_connections 2024-04-17T08:31:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.683+0000 7fa28a200640 1 -- 172.21.15.67:0/265864959 wait complete. 2024-04-17T08:31:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.685+0000 7fa28a200640 1 Processor -- start 2024-04-17T08:31:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.685+0000 7fa28a200640 1 -- start start 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680051c0 0x7fa284153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 0x7fa2840c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa28415f070 con 0x7fa2840c76d0 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa28415eef0 con 0x7fa2840c5a60 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa28a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa284162c40 con 0x7fa2680051c0 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35846/0 (socket says 172.21.15.67:35846) 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 -- 172.21.15.67:0/2539818413 learned_addr learned my addr 172.21.15.67:0/2539818413 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa283400640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680051c0 0x7fa284153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 -- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680051c0 msgr2=0x7fa284153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680051c0 0x7fa284153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 -- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 msgr2=0x7fa2840c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa283e00640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 0x7fa2840c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 0x7fa2840c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa282a00640 1 -- 172.21.15.67:0/2539818413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2840c7bb0 con 0x7fa2840c76d0 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.686+0000 7fa283e00640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 0x7fa2840c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa282a00640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00f490 tx=0x7fa26c00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c018020 con 0x7fa2840c76d0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26c014890 con 0x7fa2840c76d0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c014c60 con 0x7fa2840c76d0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2840c7e90 con 0x7fa2840c76d0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.687+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28415c870 con 0x7fa2840c76d0 2024-04-17T08:31:21.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.689+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa284070d30 con 0x7fa2840c76d0 2024-04-17T08:31:21.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.690+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fa26c01b020 con 0x7fa2840c76d0 2024-04-17T08:31:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.691+0000 7fa27b400640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 0x7fa26006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.691+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7fa26c08c310 con 0x7fa2840c76d0 2024-04-17T08:31:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.692+0000 7fa283400640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 0x7fa26006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.693+0000 7fa283400640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 0x7fa26006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa274001d20 tx=0x7fa274009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:21.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.696+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa284070d30 con 0x7fa2840c76d0 2024-04-17T08:31:21.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.911+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fa284068ca0 con 0x7fa2840c76d0 2024-04-17T08:31:21.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.919+0000 7fa27b400640 1 -- 172.21.15.67:0/2539818413 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) ==== 104+0+0 (secure 0 0 0) 0x7fa26c022020 con 0x7fa2840c76d0 2024-04-17T08:31:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 msgr2=0x7fa26006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 0x7fa26006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa274001d20 tx=0x7fa274009040 comp rx=0 tx=0).stop 2024-04-17T08:31:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 msgr2=0x7fa28415d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00f490 tx=0x7fa26c00f950 comp rx=0 tx=0).stop 2024-04-17T08:31:21.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 shutdown_connections 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa26006ca40 0x7fa26006ef00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2840c5a60 0x7fa2840c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2840c76d0 0x7fa28415d070 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 --2- 172.21.15.67:0/2539818413 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2680051c0 0x7fa284153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 >> 172.21.15.67:0/2539818413 conn(0x7fa284073360 msgr2=0x7fa284161d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 shutdown_connections 2024-04-17T08:31:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:21.922+0000 7fa28a200640 1 -- 172.21.15.67:0/2539818413 wait complete. 2024-04-17T08:31:21.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:31:22.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.160+0000 7f6dc7200640 1 Processor -- start 2024-04-17T08:31:22.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.160+0000 7f6dc7200640 1 -- start start 2024-04-17T08:31:22.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.161+0000 7f6dc7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dc80d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.161+0000 7f6dc7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc80d00b0 con 0x7f6dc80cfab0 2024-04-17T08:31:22.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.161+0000 7f6dc7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc80d0390 con 0x7f6dc80d3210 2024-04-17T08:31:22.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.161+0000 7f6dc7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc80d3670 con 0x7f6dc815ecb0 2024-04-17T08:31:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.162+0000 7f6dc6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6dc80cfab0 0x7f6dc80b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48466/0 (socket says 172.21.15.67:48466) 2024-04-17T08:31:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.162+0000 7f6dc6800640 1 -- 172.21.15.67:0/2787770493 learned_addr learned my addr 172.21.15.67:0/2787770493 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.162+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dc80d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.162+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225281648 0 0) 0x7f6dc80d00b0 con 0x7f6dc80cfab0 2024-04-17T08:31:22.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.162+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dac003520 con 0x7f6dc80cfab0 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3395667576 0 0) 0x7f6dac003520 con 0x7f6dc80cfab0 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 msgr2=0x7f6dc80d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dc80d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 >> v1:172.21.15.187:6789/0 conn(0x7f6dc80d3210 legacy=0x7f6dc80cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc80d0670 con 0x7f6dc80cfab0 2024-04-17T08:31:22.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6dc0001690 con 0x7f6dc80cfab0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 >> v1:172.21.15.67:6789/0 conn(0x7f6dc80cfab0 legacy=0x7f6dc80b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dac0039e0 0x7f6dac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dac004d40 0x7f6dac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac00bb50 con 0x7f6dac004d40 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac00bcd0 con 0x7f6dac0039e0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6dac00bed0 con 0x7f6dc815ecb0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.163+0000 7f6dc6800640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dac004d40 0x7f6dac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dac0039e0 0x7f6dac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc6800640 1 -- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 msgr2=0x7f6dac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc5400640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc6800640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc6800640 1 -- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dac0039e0 msgr2=0x7f6dac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:22.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc6800640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dac0039e0 0x7f6dac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc6800640 1 -- 172.21.15.67:0/2787770493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac003520 con 0x7f6dac004d40 2024-04-17T08:31:22.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.164+0000 7f6dc5400640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc815ecb0 0x7f6dac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.165+0000 7f6dc6800640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dac004d40 0x7f6dac0051c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000c0e0 tx=0x7f6dac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43650f2a31fa2bde server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.165+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0007c40 con 0x7f6dac004d40 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.165+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc0030070 con 0x7f6dac004d40 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.165+0000 7f6dc4a00640 1 -- 172.21.15.67:0/2787770493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0005d10 con 0x7f6dac004d40 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 -- 172.21.15.67:0/2787770493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dac004d40 msgr2=0x7f6dac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 --2- 172.21.15.67:0/2787770493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dac004d40 0x7f6dac0051c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000c0e0 tx=0x7f6dac005830 comp rx=0 tx=0).stop 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc6800640 1 -- 172.21.15.67:0/2787770493 reap_dead start 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 -- 172.21.15.67:0/2787770493 shutdown_connections 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 -- 172.21.15.67:0/2787770493 >> 172.21.15.67:0/2787770493 conn(0x7f6dc808bff0 msgr2=0x7f6dc80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 -- 172.21.15.67:0/2787770493 shutdown_connections 2024-04-17T08:31:22.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.166+0000 7f6dc7200640 1 -- 172.21.15.67:0/2787770493 wait complete. 2024-04-17T08:31:22.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.168+0000 7f6dc7200640 1 Processor -- start 2024-04-17T08:31:22.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.168+0000 7f6dc7200640 1 -- start start 2024-04-17T08:31:22.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 0x7f6dc80ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc80cbc50 0x7f6dc815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc815f090 con 0x7f6dc8158f10 2024-04-17T08:31:22.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc80d0c30 con 0x7f6dc8158b30 2024-04-17T08:31:22.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc815f210 con 0x7f6dc80cbc50 2024-04-17T08:31:22.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35868/0 (socket says 172.21.15.67:35868) 2024-04-17T08:31:22.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.169+0000 7f6dc5400640 1 -- 172.21.15.67:0/76106157 learned_addr learned my addr 172.21.15.67:0/76106157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 0x7f6dc80ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc6800640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc80cbc50 0x7f6dc815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 -- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc80cbc50 msgr2=0x7f6dc815d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc80cbc50 0x7f6dc815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 -- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 msgr2=0x7f6dc80ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 0x7f6dc80ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 -- 172.21.15.67:0/76106157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc8160280 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 0x7f6dc80ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc5400640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f6db000d770 tx=0x7f6db000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0005270 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db0017070 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db000b6b0 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.170+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc815f4c0 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.171+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc815fa80 con 0x7f6dc8158f10 2024-04-17T08:31:22.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.172+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc8068a10 con 0x7f6dc8158f10 2024-04-17T08:31:22.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.173+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f6db000b850 con 0x7f6dc8158f10 2024-04-17T08:31:22.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.174+0000 7f6dbe000640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 0x7f6da406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.174+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f6db008bcc0 con 0x7f6dc8158f10 2024-04-17T08:31:22.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.175+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 0x7f6da406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.176+0000 7f6dc5e00640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 0x7f6da406ef00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6dc81604e0 tx=0x7f6db800b040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.179+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6db0092080 con 0x7f6dc8158f10 2024-04-17T08:31:22.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.390+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f6dc80b2ee0 con 0x7f6dc8158f10 2024-04-17T08:31:22.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.396+0000 7f6dbe000640 1 -- 172.21.15.67:0/76106157 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v63) ==== 120+0+0 (secure 0 0 0) 0x7f6db0059ef0 con 0x7f6dc8158f10 2024-04-17T08:31:22.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.399+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 msgr2=0x7f6da406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.399+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 0x7f6da406ef00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6dc81604e0 tx=0x7f6db800b040 comp rx=0 tx=0).stop 2024-04-17T08:31:22.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.399+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 msgr2=0x7f6dc815ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.399+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f6db000d770 tx=0x7f6db000dc30 comp rx=0 tx=0).stop 2024-04-17T08:31:22.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 shutdown_connections 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dc80cbc50 0x7f6dc815d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc8158f10 0x7f6dc815ce90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6da406ca40 0x7f6da406ef00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 --2- 172.21.15.67:0/76106157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dc8158b30 0x7f6dc80ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 >> 172.21.15.67:0/76106157 conn(0x7f6dc808bff0 msgr2=0x7f6dc806deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 shutdown_connections 2024-04-17T08:31:22.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.400+0000 7f6dc7200640 1 -- 172.21.15.67:0/76106157 wait complete. 2024-04-17T08:31:22.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-04-17T08:31:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.643+0000 7fe1d7400640 1 Processor -- start 2024-04-17T08:31:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.643+0000 7fe1d7400640 1 -- start start 2024-04-17T08:31:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1d006f670 0x7fe1d006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d7400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1d006f030 0x7fe1d00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d7400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d006ff90 con 0x7fe1d00cd020 2024-04-17T08:31:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d7400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1d00ceb00 con 0x7fe1d006f670 2024-04-17T08:31:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d7400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe1d00cede0 con 0x7fe1d006f030 2024-04-17T08:31:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1d006f670 0x7fe1d006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1d006f670 0x7fe1d006fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35876/0 (socket says 172.21.15.67:35876) 2024-04-17T08:31:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.645+0000 7fe1d4a00640 1 -- 172.21.15.67:0/2624309861 learned_addr learned my addr 172.21.15.67:0/2624309861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.646+0000 7fe1d5400640 1 --2- 172.21.15.67:0/2624309861 >> v2:172.21.15.187:3300/0 conn(0x7fe1d006f030 0x7fe1d00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.646+0000 7fe1d4a00640 1 -- 172.21.15.67:0/2624309861 >> v2:172.21.15.187:3300/0 conn(0x7fe1d006f030 msgr2=0x7fe1d00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.646+0000 7fe1d4a00640 1 --2- 172.21.15.67:0/2624309861 >> v2:172.21.15.187:3300/0 conn(0x7fe1d006f030 0x7fe1d00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.646+0000 7fe1d4a00640 1 -- 172.21.15.67:0/2624309861 >> v1:172.21.15.187:6789/0 conn(0x7fe1d00cd020 legacy=0x7fe1d0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.646+0000 7fe1d4a00640 1 -- 172.21.15.67:0/2624309861 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d00cf0c0 con 0x7fe1d006f670 2024-04-17T08:31:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.648+0000 7fe1d4a00640 1 --2- 172.21.15.67:0/2624309861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f670 0x7fe1d006fa50 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00cdf80 tx=0x7fe1c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24c516fb7bef085 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.648+0000 7fe1cf400640 1 -- 172.21.15.67:0/2624309861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0035020 con 0x7fe1d006f670 2024-04-17T08:31:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.648+0000 7fe1cf400640 1 -- 172.21.15.67:0/2624309861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c0031870 con 0x7fe1d006f670 2024-04-17T08:31:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.648+0000 7fe1cf400640 1 -- 172.21.15.67:0/2624309861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0031c40 con 0x7fe1d006f670 2024-04-17T08:31:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 -- 172.21.15.67:0/2624309861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f670 msgr2=0x7fe1d006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 --2- 172.21.15.67:0/2624309861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f670 0x7fe1d006fa50 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00cdf80 tx=0x7fe1c002b910 comp rx=0 tx=0).stop 2024-04-17T08:31:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 -- 172.21.15.67:0/2624309861 shutdown_connections 2024-04-17T08:31:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 --2- 172.21.15.67:0/2624309861 >> v2:172.21.15.187:3300/0 conn(0x7fe1d006f030 0x7fe1d00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 --2- 172.21.15.67:0/2624309861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f670 0x7fe1d006fa50 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 -- 172.21.15.67:0/2624309861 >> 172.21.15.67:0/2624309861 conn(0x7fe1d006bf50 msgr2=0x7fe1d00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 -- 172.21.15.67:0/2624309861 shutdown_connections 2024-04-17T08:31:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.649+0000 7fe1d7400640 1 -- 172.21.15.67:0/2624309861 wait complete. 2024-04-17T08:31:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.651+0000 7fe1d7400640 1 Processor -- start 2024-04-17T08:31:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.651+0000 7fe1d7400640 1 -- start start 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 0x7fe1d015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1d00cd020 0x7fe1d0154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00ce720 con 0x7fe1d006f030 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00cf680 con 0x7fe1d006f670 2024-04-17T08:31:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d7400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d00ce8a0 con 0x7fe1d00cd020 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35880/0 (socket says 172.21.15.67:35880) 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.652+0000 7fe1d4a00640 1 -- 172.21.15.67:0/1083308108 learned_addr learned my addr 172.21.15.67:0/1083308108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d5400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1d00cd020 0x7fe1d0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1cfe00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 0x7fe1d015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 -- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1d00cd020 msgr2=0x7fe1d0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1d00cd020 0x7fe1d0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 -- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 msgr2=0x7fe1d015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 0x7fe1d015a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 -- 172.21.15.67:0/1083308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d01f46f0 con 0x7fe1d006f030 2024-04-17T08:31:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1cfe00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 0x7fe1d015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1d4a00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000c00 tx=0x7fe1c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.653+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0040070 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.654+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c003a420 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.654+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0035040 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.654+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d01f5720 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.654+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d01f4c40 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.655+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d0068b90 con 0x7fe1d006f030 2024-04-17T08:31:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.656+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fe1c004b060 con 0x7fe1d006f030 2024-04-17T08:31:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.658+0000 7fe1ccc00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 0x7fe1b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:22.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.658+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7fe1c00adcd0 con 0x7fe1d006f030 2024-04-17T08:31:22.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.658+0000 7fe1cfe00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 0x7fe1b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.660+0000 7fe1cfe00640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 0x7fe1b006eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe1d015fd10 tx=0x7fe1bc008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.662+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1c007c010 con 0x7fe1d006f030 2024-04-17T08:31:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.941+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fe1d00d10f0 con 0x7fe1d006f030 2024-04-17T08:31:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.950+0000 7fe1ccc00640 1 -- 172.21.15.67:0/1083308108 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2555) ==== 174+0+0 (secure 0 0 0) 0x7fe1c007b910 con 0x7fe1d006f030 2024-04-17T08:31:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 msgr2=0x7fe1b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 0x7fe1b006eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe1d015fd10 tx=0x7fe1bc008040 comp rx=0 tx=0).stop 2024-04-17T08:31:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 msgr2=0x7fe1d015d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000c00 tx=0x7fe1c00046c0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 shutdown_connections 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1d00cd020 0x7fe1d0154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fe1b006ca10 0x7fe1b006eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1d006f670 0x7fe1d015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 --2- 172.21.15.67:0/1083308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1d006f030 0x7fe1d015d470 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 >> 172.21.15.67:0/1083308108 conn(0x7fe1d006bf50 msgr2=0x7fe1d00d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 shutdown_connections 2024-04-17T08:31:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:22.953+0000 7fe1d7400640 1 -- 172.21.15.67:0/1083308108 wait complete. 2024-04-17T08:31:22.970 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-04-17T08:31:22.970 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-04-17T08:31:22.970 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T08:31:22.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:23.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.186+0000 7f7f47200640 1 Processor -- start 2024-04-17T08:31:23.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.186+0000 7f7f47200640 1 -- start start 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.188+0000 7f7f47200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f400cfae0 0x7f7f40169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.188+0000 7f7f47200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f400d2d30 con 0x7f7f400b6d00 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f47200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f400d3010 con 0x7f7f4015ec60 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f47200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7f400d49c0 con 0x7f7f400cfae0 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f45400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7f400b6d00 0x7f7f400cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48482/0 (socket says 172.21.15.67:48482) 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f45400640 1 -- 172.21.15.67:0/1720851577 learned_addr learned my addr 172.21.15.67:0/1720851577 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f46800640 1 --2- 172.21.15.67:0/1720851577 >> v2:172.21.15.67:3300/0 conn(0x7f7f400cfae0 0x7f7f40169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.189+0000 7f7f44a00640 1 -- 172.21.15.67:0/1720851577 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662086762 0 0) 0x7f7f400d2d30 con 0x7f7f400b6d00 2024-04-17T08:31:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.190+0000 7f7f44a00640 1 -- 172.21.15.67:0/1720851577 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f28003520 con 0x7f7f400b6d00 2024-04-17T08:31:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.190+0000 7f7f46800640 1 -- 172.21.15.67:0/1720851577 >> v1:172.21.15.187:6789/0 conn(0x7f7f4015ec60 legacy=0x7f7f400b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.190+0000 7f7f46800640 1 -- 172.21.15.67:0/1720851577 >> v1:172.21.15.67:6789/0 conn(0x7f7f400b6d00 legacy=0x7f7f400cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.190+0000 7f7f46800640 1 -- 172.21.15.67:0/1720851577 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f400d32f0 con 0x7f7f400cfae0 2024-04-17T08:31:23.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f46800640 1 --2- 172.21.15.67:0/1720851577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400cfae0 0x7f7f40169810 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009d90 tx=0x7f7f3c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca8a7b9d6977879 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f44a00640 1 -- 172.21.15.67:0/1720851577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c039070 con 0x7f7f400cfae0 2024-04-17T08:31:23.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f44a00640 1 -- 172.21.15.67:0/1720851577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f3c02cd00 con 0x7f7f400cfae0 2024-04-17T08:31:23.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f44a00640 1 -- 172.21.15.67:0/1720851577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c034660 con 0x7f7f400cfae0 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f47200640 1 -- 172.21.15.67:0/1720851577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400cfae0 msgr2=0x7f7f40169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.191+0000 7f7f47200640 1 --2- 172.21.15.67:0/1720851577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400cfae0 0x7f7f40169810 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009d90 tx=0x7f7f3c02bf50 comp rx=0 tx=0).stop 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.192+0000 7f7f47200640 1 -- 172.21.15.67:0/1720851577 shutdown_connections 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.192+0000 7f7f47200640 1 --2- 172.21.15.67:0/1720851577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400cfae0 0x7f7f40169810 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.192+0000 7f7f47200640 1 -- 172.21.15.67:0/1720851577 >> 172.21.15.67:0/1720851577 conn(0x7f7f400d15f0 msgr2=0x7f7f400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.192+0000 7f7f47200640 1 -- 172.21.15.67:0/1720851577 shutdown_connections 2024-04-17T08:31:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.192+0000 7f7f47200640 1 -- 172.21.15.67:0/1720851577 wait complete. 2024-04-17T08:31:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.193+0000 7f7f47200640 1 Processor -- start 2024-04-17T08:31:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 -- start start 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f400cfae0 0x7f7f40153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f4015ec60 0x7f7f40159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f400b8b80 con 0x7f7f400b6d00 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7f400d3660 con 0x7f7f4015ec60 2024-04-17T08:31:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.194+0000 7f7f47200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7f400d00c0 con 0x7f7f400cfae0 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35900/0 (socket says 172.21.15.67:35900) 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 -- 172.21.15.67:0/1748535285 learned_addr learned my addr 172.21.15.67:0/1748535285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45400640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f400cfae0 0x7f7f40153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 -- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f400cfae0 msgr2=0x7f7f40153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f400cfae0 0x7f7f40153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 -- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f4015ec60 msgr2=0x7f7f40159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f46800640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f4015ec60 0x7f7f40159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f4015ec60 0x7f7f40159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.195+0000 7f7f45e00640 1 -- 172.21.15.67:0/1748535285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f40158a70 con 0x7f7f400b6d00 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f45e00640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400c400 tx=0x7f7f3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34017070 con 0x7f7f400b6d00 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f340052d0 con 0x7f7f400b6d00 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f34003830 con 0x7f7f400b6d00 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f40158d50 con 0x7f7f400b6d00 2024-04-17T08:31:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.196+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f40159310 con 0x7f7f400b6d00 2024-04-17T08:31:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.197+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f40068a10 con 0x7f7f400b6d00 2024-04-17T08:31:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.198+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f7f3401a070 con 0x7f7f400b6d00 2024-04-17T08:31:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.199+0000 7f7f3a000640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 0x7f7f2406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.199+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f7f3408b0d0 con 0x7f7f400b6d00 2024-04-17T08:31:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.200+0000 7f7f45400640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 0x7f7f2406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.201+0000 7f7f45400640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 0x7f7f2406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f400ce6a0 tx=0x7f7f30009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.203+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7f34059380 con 0x7f7f400b6d00 2024-04-17T08:31:23.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.402+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f400cddb0 con 0x7f7f400b6d00 2024-04-17T08:31:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.405+0000 7f7f3a000640 1 -- 172.21.15.67:0/1748535285 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) ==== 74+0+14043 (secure 0 0 0) 0x7f7f34058c80 con 0x7f7f400b6d00 2024-04-17T08:31:23.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:23.404 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":187,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:20.937760+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3206760811":"2024-04-18T08:31:17.913169+0000","172.21.15.67:6832/3206760811":"2024-04-18T08:31:17.913169+0000","172.21.15.187:0/2335112946":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/3177304055":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6832/2551862940":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2645268054":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1428143598":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/3079350952":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/1712983456":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2676684388":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/863702549":"2024-04-18T08:31:13.813177+0000","172.21.15.187:6833/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:6832/659531354":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2751617585":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/2559339550":"2024-04-18T08:31:06.484301+0000","172.21.15.187:0/1037073318":"2024-04-18T08:31:13.813177+0000","172.21.15.187:0/2642787856":"2024-04-18T08:31:06.484301+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 msgr2=0x7f7f2406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 0x7f7f2406eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f400ce6a0 tx=0x7f7f30009040 comp rx=0 tx=0).stop 2024-04-17T08:31:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 msgr2=0x7f7f400cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400c400 tx=0x7f7f3400c8c0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 shutdown_connections 2024-04-17T08:31:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f7f2406ca10 0x7f7f2406eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7f4015ec60 0x7f7f40159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7f400cfae0 0x7f7f40153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 --2- 172.21.15.67:0/1748535285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400b6d00 0x7f7f400cd640 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 >> 172.21.15.67:0/1748535285 conn(0x7f7f400d15f0 msgr2=0x7f7f400d53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.407+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 shutdown_connections 2024-04-17T08:31:23.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.408+0000 7f7f47200640 1 -- 172.21.15.67:0/1748535285 wait complete. 2024-04-17T08:31:23.427 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:31:23.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:31:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.639+0000 7fcedec00640 1 Processor -- start 2024-04-17T08:31:23.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.640+0000 7fcedec00640 1 -- start start 2024-04-17T08:31:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced80aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fced8162fe0 0x7fced8071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fced8072380 0x7fced806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fced80b0150 con 0x7fced8072380 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fced8145420 con 0x7fced8162fe0 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.641+0000 7fcedec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fced80af0b0 con 0x7fced8145040 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fced8072380 0x7fced806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fced8072380 0x7fced806e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53726/0 (socket says 172.21.15.67:53726) 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 -- 172.21.15.67:0/937816272 learned_addr learned my addr 172.21.15.67:0/937816272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 -- 172.21.15.67:0/937816272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 msgr2=0x7fced80aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 --2- 172.21.15.67:0/937816272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced80aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 -- 172.21.15.67:0/937816272 >> v2:172.21.15.187:3300/0 conn(0x7fced8162fe0 msgr2=0x7fced8071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 --2- 172.21.15.67:0/937816272 >> v2:172.21.15.187:3300/0 conn(0x7fced8162fe0 0x7fced8071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.642+0000 7fcede200640 1 -- 172.21.15.67:0/937816272 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced80af390 con 0x7fced8072380 2024-04-17T08:31:23.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.643+0000 7fcede200640 1 --2- 172.21.15.67:0/937816272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced806e0c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fced8073580 tx=0x7fced002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ba13df6df831a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.643+0000 7fced7e00640 1 -- 172.21.15.67:0/937816272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced002c9f0 con 0x7fced8072380 2024-04-17T08:31:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fced7e00640 1 -- 172.21.15.67:0/937816272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fced002cb90 con 0x7fced8072380 2024-04-17T08:31:23.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fced7e00640 1 -- 172.21.15.67:0/937816272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced00343f0 con 0x7fced8072380 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 -- 172.21.15.67:0/937816272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 msgr2=0x7fced806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 --2- 172.21.15.67:0/937816272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced806e0c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fced8073580 tx=0x7fced002be50 comp rx=0 tx=0).stop 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 -- 172.21.15.67:0/937816272 shutdown_connections 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 --2- 172.21.15.67:0/937816272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced806e0c0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 --2- 172.21.15.67:0/937816272 >> v2:172.21.15.187:3300/0 conn(0x7fced8162fe0 0x7fced8071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 --2- 172.21.15.67:0/937816272 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced80aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 -- 172.21.15.67:0/937816272 >> 172.21.15.67:0/937816272 conn(0x7fced80b23c0 msgr2=0x7fced80b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 -- 172.21.15.67:0/937816272 shutdown_connections 2024-04-17T08:31:23.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.644+0000 7fcedec00640 1 -- 172.21.15.67:0/937816272 wait complete. 2024-04-17T08:31:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.646+0000 7fcedec00640 1 Processor -- start 2024-04-17T08:31:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.646+0000 7fcedec00640 1 -- start start 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced81f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced8162fe0 0x7fced8141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fced80afc30 con 0x7fced8072380 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fced80af950 con 0x7fced8162fe0 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.647+0000 7fcedec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fced8163440 con 0x7fced8145040 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53732/0 (socket says 172.21.15.67:53732) 2024-04-17T08:31:23.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 -- 172.21.15.67:0/3358357388 learned_addr learned my addr 172.21.15.67:0/3358357388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 -- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 msgr2=0x7fced81f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedce00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced81f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced81f48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 -- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced8162fe0 msgr2=0x7fced8141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced8162fe0 0x7fced8141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 -- 172.21.15.67:0/3358357388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced81f4e10 con 0x7fced8072380 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.648+0000 7fcedd800640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00b410 tx=0x7fcecc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.649+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc015020 con 0x7fced8072380 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.649+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcecc011850 con 0x7fced8072380 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.649+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc011c40 con 0x7fced8072380 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.649+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced81f50b0 con 0x7fced8072380 2024-04-17T08:31:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.649+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced81f5670 con 0x7fced8072380 2024-04-17T08:31:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.650+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fced8068a10 con 0x7fced8072380 2024-04-17T08:31:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.651+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fcecc018020 con 0x7fced8072380 2024-04-17T08:31:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.652+0000 7fced5600640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 0x7fceb806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.652+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7fcecc08bc00 con 0x7fced8072380 2024-04-17T08:31:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.652+0000 7fcedce00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 0x7fceb806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.654+0000 7fcedce00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 0x7fceb806ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fced80c7380 tx=0x7fcec4008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.656+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcecc01f020 con 0x7fced8072380 2024-04-17T08:31:23.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.859+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fced8068610 con 0x7fced8072380 2024-04-17T08:31:23.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.860+0000 7fced5600640 1 -- 172.21.15.67:0/3358357388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) ==== 93+0+10 (secure 0 0 0) 0x7fcecc0172c0 con 0x7fced8072380 2024-04-17T08:31:23.859 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:31:23.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 msgr2=0x7fceb806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 0x7fceb806ef20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fced80c7380 tx=0x7fcec4008040 comp rx=0 tx=0).stop 2024-04-17T08:31:23.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 msgr2=0x7fced8141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:23.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00b410 tx=0x7fcecc00b8d0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 shutdown_connections 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fceb806ca60 0x7fceb806ef20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fced8162fe0 0x7fced8141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fced8145040 0x7fced81f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 --2- 172.21.15.67:0/3358357388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8072380 0x7fced8141d50 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 >> 172.21.15.67:0/3358357388 conn(0x7fced80b23c0 msgr2=0x7fced8160f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 shutdown_connections 2024-04-17T08:31:23.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:23.862+0000 7fcedec00640 1 -- 172.21.15.67:0/3358357388 wait complete. 2024-04-17T08:31:23.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-17T08:31:24.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.096+0000 7f72ee800640 1 Processor -- start 2024-04-17T08:31:24.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.097+0000 7f72ee800640 1 -- start start 2024-04-17T08:31:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.098+0000 7f72ee800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f72e80cd020 0x7f72e80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.098+0000 7f72ee800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 0x7f72e806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.098+0000 7f72ee800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e80ceb00 con 0x7f72e806f670 2024-04-17T08:31:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.098+0000 7f72ee800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f72e80cede0 con 0x7f72e80cd020 2024-04-17T08:31:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.098+0000 7f72ee800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72e80cf0c0 con 0x7f72e806e020 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72dfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f72e80cd020 0x7f72e80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72e7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f72e806f670 0x7f72e806fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44152/0 (socket says 172.21.15.67:44152) 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72e7400640 1 -- 172.21.15.67:0/3643299346 learned_addr learned my addr 172.21.15.67:0/3643299346 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72e7e00640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 0x7f72e806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72dfe00640 1 -- 172.21.15.67:0/3643299346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 msgr2=0x7f72e806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72dfe00640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 0x7f72e806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72dfe00640 1 -- 172.21.15.67:0/3643299346 >> v1:172.21.15.67:6789/0 conn(0x7f72e806f670 legacy=0x7f72e806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72dfe00640 1 -- 172.21.15.67:0/3643299346 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e80cf3a0 con 0x7f72e80cd020 2024-04-17T08:31:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.099+0000 7f72e7e00640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 0x7f72e806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.100+0000 7f72dfe00640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80cd020 0x7f72e80cd480 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f72d00097a0 tx=0x7f72d002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfd046ef06b76d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72e6a00640 1 -- 172.21.15.67:0/3643299346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0035020 con 0x7f72e80cd020 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72e6a00640 1 -- 172.21.15.67:0/3643299346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d0031810 con 0x7f72e80cd020 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72e6a00640 1 -- 172.21.15.67:0/3643299346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0031be0 con 0x7f72e80cd020 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 -- 172.21.15.67:0/3643299346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80cd020 msgr2=0x7f72e80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80cd020 0x7f72e80cd480 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f72d00097a0 tx=0x7f72d002b8b0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 -- 172.21.15.67:0/3643299346 shutdown_connections 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e806e020 0x7f72e806e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 --2- 172.21.15.67:0/3643299346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e80cd020 0x7f72e80cd480 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.101+0000 7f72ee800640 1 -- 172.21.15.67:0/3643299346 >> 172.21.15.67:0/3643299346 conn(0x7f72e806bf50 msgr2=0x7f72e80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.102+0000 7f72ee800640 1 -- 172.21.15.67:0/3643299346 shutdown_connections 2024-04-17T08:31:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.102+0000 7f72ee800640 1 -- 172.21.15.67:0/3643299346 wait complete. 2024-04-17T08:31:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.103+0000 7f72ee800640 1 Processor -- start 2024-04-17T08:31:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.103+0000 7f72ee800640 1 -- start start 2024-04-17T08:31:24.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.104+0000 7f72ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.104+0000 7f72ee800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e806f670 0x7f72e81f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72ee800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 0x7f72e81f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72ee800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72e80ce8a0 con 0x7f72e806e020 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72ee800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f72e80ce720 con 0x7f72e806f670 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72ee800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72e815cb00 con 0x7f72e80cd020 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53742/0 (socket says 172.21.15.67:53742) 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 -- 172.21.15.67:0/1832241666 learned_addr learned my addr 172.21.15.67:0/1832241666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7e00640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 0x7f72e81f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 -- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 msgr2=0x7f72e81f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72dfe00640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e806f670 0x7f72e81f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 0x7f72e81f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 -- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e806f670 msgr2=0x7f72e81f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e806f670 0x7f72e81f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7400640 1 -- 172.21.15.67:0/1832241666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e80ae620 con 0x7f72e806e020 2024-04-17T08:31:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.105+0000 7f72e7e00640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 0x7f72e81f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72e7400640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f72d800c3f0 tx=0x7f72d800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8017070 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d8005ce0 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80072f0 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e80ae900 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.106+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e80aeec0 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.107+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e8068b90 con 0x7f72e806e020 2024-04-17T08:31:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.108+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f72d801a070 con 0x7f72e806e020 2024-04-17T08:31:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.109+0000 7f72df400640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 0x7f72c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.109+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f72d808afe0 con 0x7f72e806e020 2024-04-17T08:31:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.110+0000 7f72dfe00640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 0x7f72c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:24.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.111+0000 7f72dfe00640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 0x7f72c406eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72d0036000 tx=0x7f72d0008030 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:24.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.114+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f72d8059380 con 0x7f72e806e020 2024-04-17T08:31:24.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.393+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7f72e806fa50 con 0x7f72e806e020 2024-04-17T08:31:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.966+0000 7f72df400640 1 -- 172.21.15.67:0/1832241666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2557) ==== 168+0+0 (secure 0 0 0) 0x7f72d8058c80 con 0x7f72e806e020 2024-04-17T08:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 msgr2=0x7f72c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 0x7f72c406eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f72d0036000 tx=0x7f72d0008030 comp rx=0 tx=0).stop 2024-04-17T08:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 msgr2=0x7f72e80af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f72d800c3f0 tx=0x7f72d800c8b0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 shutdown_connections 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72e80cd020 0x7f72e81f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f72c406ca10 0x7f72c406eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72e806f670 0x7f72e81f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 --2- 172.21.15.67:0/1832241666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e806e020 0x7f72e80af6c0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.971+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 >> 172.21.15.67:0/1832241666 conn(0x7f72e806bf50 msgr2=0x7f72e806ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.972+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 shutdown_connections 2024-04-17T08:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:24.972+0000 7f72ee800640 1 -- 172.21.15.67:0/1832241666 wait complete. 2024-04-17T08:31:24.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:31:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.223+0000 7f71db000640 1 Processor -- start 2024-04-17T08:31:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.224+0000 7f71db000640 1 -- start start 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71db000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d415ec40 0x7f71d40d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71db000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71d40d33f0 0x7f71d415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71db000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d40d51d0 con 0x7f71d40b3660 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71db000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f71d40d54b0 con 0x7f71d40d33f0 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71db000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71d415f3a0 con 0x7f71d415ec40 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.225+0000 7f71da600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71d40d33f0 0x7f71d415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71d40d33f0 0x7f71d415fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53748/0 (socket says 172.21.15.67:53748) 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 -- 172.21.15.67:0/3170953949 learned_addr learned my addr 172.21.15.67:0/3170953949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71d9c00640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d415ec40 0x7f71d40d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 -- 172.21.15.67:0/3170953949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d415ec40 msgr2=0x7f71d40d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d415ec40 0x7f71d40d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 -- 172.21.15.67:0/3170953949 >> v1:172.21.15.187:6789/0 conn(0x7f71d40b3660 legacy=0x7f71d40d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.226+0000 7f71da600640 1 -- 172.21.15.67:0/3170953949 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d415f680 con 0x7f71d40d33f0 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.228+0000 7f71da600640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d415fba0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009870 tx=0x7f71cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c742c88445ecdf09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.228+0000 7f71d3e00640 1 -- 172.21.15.67:0/3170953949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc034070 con 0x7f71d40d33f0 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.228+0000 7f71d3e00640 1 -- 172.21.15.67:0/3170953949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71cc030840 con 0x7f71d40d33f0 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.228+0000 7f71d3e00640 1 -- 172.21.15.67:0/3170953949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc030c10 con 0x7f71d40d33f0 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 -- 172.21.15.67:0/3170953949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 msgr2=0x7f71d415fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d415fba0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009870 tx=0x7f71cc02b980 comp rx=0 tx=0).stop 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 -- 172.21.15.67:0/3170953949 shutdown_connections 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d415fba0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 --2- 172.21.15.67:0/3170953949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d415ec40 0x7f71d40d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 -- 172.21.15.67:0/3170953949 >> 172.21.15.67:0/3170953949 conn(0x7f71d408bff0 msgr2=0x7f71d40d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 -- 172.21.15.67:0/3170953949 shutdown_connections 2024-04-17T08:31:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.229+0000 7f71db000640 1 -- 172.21.15.67:0/3170953949 wait complete. 2024-04-17T08:31:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.231+0000 7f71db000640 1 Processor -- start 2024-04-17T08:31:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.231+0000 7f71db000640 1 -- start start 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d40b3660 0x7f71d4157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71d415ec40 0x7f71d41e8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71d40cf400 con 0x7f71d40d33f0 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71d40cf280 con 0x7f71d415ec40 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71db000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71d40cf580 con 0x7f71d40b3660 2024-04-17T08:31:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71d9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71d9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53762/0 (socket says 172.21.15.67:53762) 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.232+0000 7f71d9200640 1 -- 172.21.15.67:0/22593047 learned_addr learned my addr 172.21.15.67:0/22593047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9c00640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d40b3660 0x7f71d4157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71da600640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71d415ec40 0x7f71d41e8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 -- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d40b3660 msgr2=0x7f71d4157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d40b3660 0x7f71d4157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 -- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71d415ec40 msgr2=0x7f71d41e8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71d415ec40 0x7f71d41e8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 -- 172.21.15.67:0/22593047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d40ca160 con 0x7f71d40d33f0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d9200640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c430 tx=0x7f71c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c00052d0 con 0x7f71d40d33f0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c0005490 con 0x7f71d40d33f0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0007260 con 0x7f71d40d33f0 2024-04-17T08:31:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.233+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d40ca440 con 0x7f71d40d33f0 2024-04-17T08:31:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.234+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d40caa00 con 0x7f71d40d33f0 2024-04-17T08:31:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.235+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d4068a10 con 0x7f71d40d33f0 2024-04-17T08:31:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.236+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f71c0007400 con 0x7f71d40d33f0 2024-04-17T08:31:25.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.237+0000 7f71d1600640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 0x7f71b406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:25.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.237+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f71c008b200 con 0x7f71d40d33f0 2024-04-17T08:31:25.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.238+0000 7f71d9c00640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 0x7f71b406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.239+0000 7f71d9c00640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 0x7f71b406ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71c40099c0 tx=0x7f71c4003990 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.242+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f71c0059550 con 0x7f71d40d33f0 2024-04-17T08:31:25.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.449+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f71d40cac00 con 0x7f71d40d33f0 2024-04-17T08:31:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.456+0000 7f71d1600640 1 -- 172.21.15.67:0/22593047 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) ==== 122+0+0 (secure 0 0 0) 0x7f71c0058e50 con 0x7f71d40d33f0 2024-04-17T08:31:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 msgr2=0x7f71b406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 0x7f71b406ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71c40099c0 tx=0x7f71c4003990 comp rx=0 tx=0).stop 2024-04-17T08:31:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 msgr2=0x7f71d41e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:25.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c430 tx=0x7f71c000c8f0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 shutdown_connections 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f71b406ca40 0x7f71b406ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71d415ec40 0x7f71d41e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71d40d33f0 0x7f71d41e8450 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 --2- 172.21.15.67:0/22593047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71d40b3660 0x7f71d4157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 >> 172.21.15.67:0/22593047 conn(0x7f71d408bff0 msgr2=0x7f71d40cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 shutdown_connections 2024-04-17T08:31:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:25.459+0000 7f71db000640 1 -- 172.21.15.67:0/22593047 wait complete. 2024-04-17T08:31:25.477 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:31:25.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:31:25.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:31:25.742 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:31:25.743 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:25.743 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:25.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:31:25.803 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:31:25.803 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:25.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:31:25.861 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:25.861 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:25.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:31:25.923 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:31:25.923 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:25.981 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:31:25.981 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:31:25.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.215+0000 7f201da00640 1 Processor -- start 2024-04-17T08:31:26.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.215+0000 7f201da00640 1 -- start start 2024-04-17T08:31:26.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f201da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201815b530 0x7f2018071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f201da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 0x7f2018072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f201da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201815f3c0 con 0x7f201815b910 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f201da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f201806ed60 con 0x7f201815b530 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f201da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f201815f660 con 0x7f20180c68c0 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f2016a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201815b530 0x7f2018071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.217+0000 7f2016a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f201815b530 0x7f2018071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53778/0 (socket says 172.21.15.67:53778) 2024-04-17T08:31:26.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f2016a00640 1 -- 172.21.15.67:0/2483905785 learned_addr learned my addr 172.21.15.67:0/2483905785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f2016a00640 1 -- 172.21.15.67:0/2483905785 >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 msgr2=0x7f2018072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f200e000640 1 --2- 172.21.15.67:0/2483905785 >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 0x7f2018072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f2016a00640 1 --2- 172.21.15.67:0/2483905785 >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 0x7f2018072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f2016a00640 1 -- 172.21.15.67:0/2483905785 >> v1:172.21.15.187:6789/0 conn(0x7f201815b910 legacy=0x7f20180c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f2016a00640 1 -- 172.21.15.67:0/2483905785 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201815f940 con 0x7f201815b530 2024-04-17T08:31:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.218+0000 7f200e000640 1 --2- 172.21.15.67:0/2483905785 >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 0x7f2018072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.219+0000 7f2016a00640 1 --2- 172.21.15.67:0/2483905785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201815b530 0x7f2018071d10 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f201806c840 tx=0x7f200802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e166e2e1cddab306 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f2016000640 1 -- 172.21.15.67:0/2483905785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008035020 con 0x7f201815b530 2024-04-17T08:31:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f2016000640 1 -- 172.21.15.67:0/2483905785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2008031640 con 0x7f201815b530 2024-04-17T08:31:26.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f2016000640 1 -- 172.21.15.67:0/2483905785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200802d040 con 0x7f201815b530 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 -- 172.21.15.67:0/2483905785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201815b530 msgr2=0x7f2018071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 --2- 172.21.15.67:0/2483905785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201815b530 0x7f2018071d10 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f201806c840 tx=0x7f200802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 -- 172.21.15.67:0/2483905785 shutdown_connections 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 --2- 172.21.15.67:0/2483905785 >> v2:172.21.15.187:3300/0 conn(0x7f20180c68c0 0x7f2018072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 --2- 172.21.15.67:0/2483905785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f201815b530 0x7f2018071d10 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.220+0000 7f201da00640 1 -- 172.21.15.67:0/2483905785 >> 172.21.15.67:0/2483905785 conn(0x7f201806bf50 msgr2=0x7f20180b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.221+0000 7f201da00640 1 -- 172.21.15.67:0/2483905785 shutdown_connections 2024-04-17T08:31:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.221+0000 7f201da00640 1 -- 172.21.15.67:0/2483905785 wait complete. 2024-04-17T08:31:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.223+0000 7f201da00640 1 Processor -- start 2024-04-17T08:31:26.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.223+0000 7f201da00640 1 -- start start 2024-04-17T08:31:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f201815b530 0x7f201814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 0x7f20180b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20180cc950 con 0x7f20180c68c0 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20180ccca0 con 0x7f201815b910 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f201da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2018160140 con 0x7f201815b530 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.224+0000 7f2016a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53790/0 (socket says 172.21.15.67:53790) 2024-04-17T08:31:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 -- 172.21.15.67:0/3051528545 learned_addr learned my addr 172.21.15.67:0/3051528545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2017400640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 0x7f20180b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f200e000640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f201815b530 0x7f201814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 -- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f201815b530 msgr2=0x7f201814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f201815b530 0x7f201814bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 -- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 msgr2=0x7f20180b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 0x7f20180b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 -- 172.21.15.67:0/3051528545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180b4670 con 0x7f20180c68c0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2017400640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 0x7f20180b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.225+0000 7f2016a00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f20080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.226+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200803ac60 con 0x7f20180c68c0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.226+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200803ae20 con 0x7f20180c68c0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.226+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008044630 con 0x7f20180c68c0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.226+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20181591b0 con 0x7f20180c68c0 2024-04-17T08:31:26.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.226+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2018159770 con 0x7f20180c68c0 2024-04-17T08:31:26.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.227+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2018068b90 con 0x7f20180c68c0 2024-04-17T08:31:26.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.228+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f200804b060 con 0x7f20180c68c0 2024-04-17T08:31:26.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.229+0000 7f200f400640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 0x7f1ff406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.230+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f20080add80 con 0x7f20180c68c0 2024-04-17T08:31:26.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.230+0000 7f200e000640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 0x7f1ff406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.231+0000 7f200e000640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 0x7f1ff406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20180b3930 tx=0x7f2000008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.235+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f200803f070 con 0x7f20180c68c0 2024-04-17T08:31:26.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.481+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201814c410 con 0x7f20180c68c0 2024-04-17T08:31:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.482+0000 7f200f400640 1 -- 172.21.15.67:0/3051528545 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+3872 (secure 0 0 0) 0x7f200804a2c0 con 0x7f20180c68c0 2024-04-17T08:31:26.480 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:26.480 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":87,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6932,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4012443384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4012443384},{"type":"v1","addr":"172.21.15.187:6835","nonce":4012443384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7636,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3364845833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3364845833},{"type":"v1","addr":"172.21.15.67:6835","nonce":3364845833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":7645,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3658716409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3658716409},{"type":"v1","addr":"172.21.15.187:6837","nonce":3658716409}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9879,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/728022648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":728022648},{"type":"v1","addr":"172.21.15.67:6833","nonce":728022648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[]} 2024-04-17T08:31:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 87 2024-04-17T08:31:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.484+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 msgr2=0x7f1ff406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 0x7f1ff406eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f20180b3930 tx=0x7f2000008040 comp rx=0 tx=0).stop 2024-04-17T08:31:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 msgr2=0x7f201814c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f20080046c0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 shutdown_connections 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f1ff406ca10 0x7f1ff406eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f201815b910 0x7f20180b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f201815b530 0x7f201814bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 --2- 172.21.15.67:0/3051528545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20180c68c0 0x7f201814c780 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 >> 172.21.15.67:0/3051528545 conn(0x7f201806bf50 msgr2=0x7f20180cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 shutdown_connections 2024-04-17T08:31:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.485+0000 7f201da00640 1 -- 172.21.15.67:0/3051528545 wait complete. 2024-04-17T08:31:26.505 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:31:26.505 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:31:26.505 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:31:26.506 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:31:26.506 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:31:26.506 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:31:26.506 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:31:26.506 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:31:26.506 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:31:26.506 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:31:26.507 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:31:26.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:31:26.507 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:31:26.507 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:31:26.507 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:31:26.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:31:26.523 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:31:26.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:31:26.529 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:31:26.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:31:26.531 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:31:26.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:31:26.536 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:31:26.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:31:26.577 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:31:26.578 DEBUG:teuthology.parallel:result is None 2024-04-17T08:31:26.582 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:31:26.582 DEBUG:teuthology.parallel:result is None 2024-04-17T08:31:26.583 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:31:26.583 DEBUG:teuthology.parallel:result is None 2024-04-17T08:31:26.590 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:31:26.591 DEBUG:teuthology.parallel:result is None 2024-04-17T08:31:26.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:31:26.736 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:31:26.738 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:31:26.741 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:31:26.741 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:31:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.783+0000 7f51e7200640 1 Processor -- start 2024-04-17T08:31:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.783+0000 7f51e7200640 1 -- start start 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.784+0000 7f51e7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51e80b9850 0x7f51e80d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.784+0000 7f51e7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e80d4140 con 0x7f51e815ee20 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.784+0000 7f51e7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e80d3b80 con 0x7f51e80d1ab0 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.784+0000 7f51e7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f51e80b83b0 con 0x7f51e80b9850 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f51e815ee20 0x7f51e80d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44160/0 (socket says 172.21.15.67:44160) 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e5e00640 1 -- 172.21.15.67:0/3887220021 learned_addr learned my addr 172.21.15.67:0/3887220021 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:26.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e6800640 1 --2- 172.21.15.67:0/3887220021 >> v2:172.21.15.67:3300/0 conn(0x7f51e80b9850 0x7f51e80d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e6800640 1 -- 172.21.15.67:0/3887220021 >> v1:172.21.15.187:6789/0 conn(0x7f51e80d1ab0 legacy=0x7f51e80d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e6800640 1 -- 172.21.15.67:0/3887220021 >> v1:172.21.15.67:6789/0 conn(0x7f51e815ee20 legacy=0x7f51e80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.785+0000 7f51e6800640 1 -- 172.21.15.67:0/3887220021 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e80d3e60 con 0x7f51e80b9850 2024-04-17T08:31:26.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.786+0000 7f51e6800640 1 --2- 172.21.15.67:0/3887220021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80b9850 0x7f51e80d3520 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009d90 tx=0x7f51e002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e965c1ac899d86d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e4a00640 1 -- 172.21.15.67:0/3887220021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0039070 con 0x7f51e80b9850 2024-04-17T08:31:26.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e4a00640 1 -- 172.21.15.67:0/3887220021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e002cd00 con 0x7f51e80b9850 2024-04-17T08:31:26.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e4a00640 1 -- 172.21.15.67:0/3887220021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0034660 con 0x7f51e80b9850 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 -- 172.21.15.67:0/3887220021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80b9850 msgr2=0x7f51e80d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 --2- 172.21.15.67:0/3887220021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80b9850 0x7f51e80d3520 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009d90 tx=0x7f51e002bf50 comp rx=0 tx=0).stop 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 -- 172.21.15.67:0/3887220021 shutdown_connections 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 --2- 172.21.15.67:0/3887220021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80b9850 0x7f51e80d3520 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 -- 172.21.15.67:0/3887220021 >> 172.21.15.67:0/3887220021 conn(0x7f51e806d3c0 msgr2=0x7f51e80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 -- 172.21.15.67:0/3887220021 shutdown_connections 2024-04-17T08:31:26.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.787+0000 7f51e7200640 1 -- 172.21.15.67:0/3887220021 wait complete. 2024-04-17T08:31:26.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.789+0000 7f51e7200640 1 Processor -- start 2024-04-17T08:31:26.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.789+0000 7f51e7200640 1 -- start start 2024-04-17T08:31:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51e80b9850 0x7f51e80cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51e815ee20 0x7f51e80c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51e8169a00 con 0x7f51e80d1ab0 2024-04-17T08:31:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f51e815f200 con 0x7f51e815ee20 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.790+0000 7f51e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f51e8169b80 con 0x7f51e80b9850 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51e80b9850 0x7f51e80cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53844/0 (socket says 172.21.15.67:53844) 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 -- 172.21.15.67:0/785648849 learned_addr learned my addr 172.21.15.67:0/785648849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e6800640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51e815ee20 0x7f51e80c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 -- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51e80b9850 msgr2=0x7f51e80cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51e80b9850 0x7f51e80cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 -- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51e815ee20 msgr2=0x7f51e80c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:26.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51e815ee20 0x7f51e80c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 -- 172.21.15.67:0/785648849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e80ce8c0 con 0x7f51e80d1ab0 2024-04-17T08:31:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.791+0000 7f51e5400640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c400 tx=0x7f51d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.792+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d40052d0 con 0x7f51e80d1ab0 2024-04-17T08:31:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.792+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d4005490 con 0x7f51e80d1ab0 2024-04-17T08:31:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.792+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4007260 con 0x7f51e80d1ab0 2024-04-17T08:31:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.792+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e80cdd40 con 0x7f51e80d1ab0 2024-04-17T08:31:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.792+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e80ce300 con 0x7f51e80d1ab0 2024-04-17T08:31:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.793+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e8068a10 con 0x7f51e80d1ab0 2024-04-17T08:31:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.794+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f51d4007400 con 0x7f51e80d1ab0 2024-04-17T08:31:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.795+0000 7f51de000640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 0x7f51c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.795+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 5659+0+0 (secure 0 0 0) 0x7f51d408b230 con 0x7f51e80d1ab0 2024-04-17T08:31:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.796+0000 7f51e5e00640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 0x7f51c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.796+0000 7f51e5e00640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 0x7f51c806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005880 tx=0x7f51d8009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:26.799+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f51d4058ea0 con 0x7f51e80d1ab0 2024-04-17T08:31:27.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.019+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f51e8151d60 con 0x7f51e80d1ab0 2024-04-17T08:31:27.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.482+0000 7f51de000640 1 -- 172.21.15.67:0/785648849 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v189) ==== 97+0+0 (secure 0 0 0) 0x7f51d405d1a0 con 0x7f51e80d1ab0 2024-04-17T08:31:27.481 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:31:27.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 msgr2=0x7f51c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 0x7f51c806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005880 tx=0x7f51d8009040 comp rx=0 tx=0).stop 2024-04-17T08:31:27.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 msgr2=0x7f51e80c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f51d400c400 tx=0x7f51d400c8c0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 shutdown_connections 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f51c806ca10 0x7f51c806eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51e815ee20 0x7f51e80c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e80d1ab0 0x7f51e80c65f0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 --2- 172.21.15.67:0/785648849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51e80b9850 0x7f51e80cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 >> 172.21.15.67:0/785648849 conn(0x7f51e806d3c0 msgr2=0x7f51e815e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 shutdown_connections 2024-04-17T08:31:27.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.486+0000 7f51e7200640 1 -- 172.21.15.67:0/785648849 wait complete. 2024-04-17T08:31:27.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:31:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.728+0000 7fd1cae00640 1 Processor -- start 2024-04-17T08:31:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.728+0000 7fd1cae00640 1 -- start start 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1cae00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1c40cd040 0x7fd1c40cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1cae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1c40b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1cae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c40cd960 con 0x7fd1c40ce610 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1cae00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c4071f70 con 0x7fd1c40cd040 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1cae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c4072250 con 0x7fd1c415ed00 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1c3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1c40ce610 0x7fd1c40b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44174/0 (socket says 172.21.15.67:44174) 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.730+0000 7fd1c3400640 1 -- 172.21.15.67:0/1958391908 learned_addr learned my addr 172.21.15.67:0/1958391908 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c8e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1c40b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111264742 0 0) 0x7fd1c40cd960 con 0x7fd1c40ce610 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> v2:172.21.15.187:3300/0 conn(0x7fd1c40cd040 0x7fd1c40cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a8003520 con 0x7fd1c40ce610 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3060145302 0 0) 0x7fd1a8003520 con 0x7fd1c40ce610 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 msgr2=0x7fd1c40b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1c40b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 >> v2:172.21.15.187:3300/0 conn(0x7fd1c40cd040 msgr2=0x7fd1c40cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 --2- 172.21.15.67:0/1958391908 >> v2:172.21.15.187:3300/0 conn(0x7fd1c40cd040 0x7fd1c40cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c4072530 con 0x7fd1c40ce610 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd1b0001690 con 0x7fd1c40ce610 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 >> v1:172.21.15.67:6789/0 conn(0x7fd1c40ce610 legacy=0x7fd1c40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a80039b0 0x7fd1a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a8004800 0x7fd1a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c4071f70 con 0x7fd1a80039b0 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c4072250 con 0x7fd1a8004800 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1a800ba40 con 0x7fd1c415ed00 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.731+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> v2:172.21.15.187:3300/0 conn(0x7fd1c40cd040 0x7fd1c40cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a80039b0 0x7fd1a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c8e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3400640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a8004800 0x7fd1a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 -- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 msgr2=0x7fd1a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415ed00 0x7fd1a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 -- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a8004800 msgr2=0x7fd1a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a8004800 0x7fd1a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.732+0000 7fd1c3e00640 1 -- 172.21.15.67:0/1958391908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a8003520 con 0x7fd1a80039b0 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.734+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a80039b0 0x7fd1a8003dd0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40098a0 tx=0x7fd1a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04a37bb0ffc22c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.734+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400f040 con 0x7fd1a80039b0 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.734+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b4037600 con 0x7fd1a80039b0 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.734+0000 7fd1c2a00640 1 -- 172.21.15.67:0/1958391908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40379b0 con 0x7fd1a80039b0 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.734+0000 7fd1cae00640 1 -- 172.21.15.67:0/1958391908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a80039b0 msgr2=0x7fd1a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1cae00640 1 --2- 172.21.15.67:0/1958391908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1a80039b0 0x7fd1a8003dd0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40098a0 tx=0x7fd1a8005830 comp rx=0 tx=0).stop 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1c8e00640 1 -- 172.21.15.67:0/1958391908 reap_dead start 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1cae00640 1 -- 172.21.15.67:0/1958391908 shutdown_connections 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1cae00640 1 -- 172.21.15.67:0/1958391908 >> 172.21.15.67:0/1958391908 conn(0x7fd1c406b8c0 msgr2=0x7fd1c406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1cae00640 1 -- 172.21.15.67:0/1958391908 shutdown_connections 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.735+0000 7fd1cae00640 1 -- 172.21.15.67:0/1958391908 wait complete. 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.736+0000 7fd1cae00640 1 Processor -- start 2024-04-17T08:31:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.737+0000 7fd1cae00640 1 -- start start 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 0x7fd1c4145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415d000 0x7fd1c415d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40b40c0 con 0x7fd1c41527c0 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c4072af0 con 0x7fd1a80039b0 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1cae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c40b4240 con 0x7fd1c415d000 2024-04-17T08:31:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53858/0 (socket says 172.21.15.67:53858) 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 -- 172.21.15.67:0/155547932 learned_addr learned my addr 172.21.15.67:0/155547932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c8e00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415d000 0x7fd1c415d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 0x7fd1c4145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 -- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415d000 msgr2=0x7fd1c415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415d000 0x7fd1c415d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 -- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 msgr2=0x7fd1c4145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 0x7fd1c4145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3400640 1 -- 172.21.15.67:0/155547932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c415f210 con 0x7fd1c41527c0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.738+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 0x7fd1c4145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1c3400640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0001050 tx=0x7fd1b000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000c6c0 con 0x7fd1c41527c0 2024-04-17T08:31:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b0018070 con 0x7fd1c41527c0 2024-04-17T08:31:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000cac0 con 0x7fd1c41527c0 2024-04-17T08:31:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c415f4f0 con 0x7fd1c41527c0 2024-04-17T08:31:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.739+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c415fab0 con 0x7fd1c41527c0 2024-04-17T08:31:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.740+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c4068b90 con 0x7fd1c41527c0 2024-04-17T08:31:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.741+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fd1b000cc60 con 0x7fd1c41527c0 2024-04-17T08:31:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.742+0000 7fd1bbe00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 0x7fd1a406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.742+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4885+0+0 (secure 0 0 0) 0x7fd1b008b4b0 con 0x7fd1c41527c0 2024-04-17T08:31:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.743+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 0x7fd1a406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.744+0000 7fd1c3e00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 0x7fd1a406ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4036000 tx=0x7fd1b4031000 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:27.747+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd1b0059af0 con 0x7fd1c41527c0 2024-04-17T08:31:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.031+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fd1c40b2a30 con 0x7fd1c41527c0 2024-04-17T08:31:28.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.032+0000 7fd1bbe00640 1 -- 172.21.15.67:0/155547932 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) ==== 80+0+5304 (secure 0 0 0) 0x7fd1b00593f0 con 0x7fd1c41527c0 2024-04-17T08:31:28.030 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:31:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:31:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.034 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:31:28.035 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:31:28.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.041 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.042 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:31:28.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:31:28.044 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:31:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 msgr2=0x7fd1a406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 0x7fd1a406ef00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4036000 tx=0x7fd1b4031000 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 msgr2=0x7fd1c4152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0001050 tx=0x7fd1b000ec90 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 shutdown_connections 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c415d000 0x7fd1c415d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c41527c0 0x7fd1c4152c40 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fd1a406ca40 0x7fd1a406ef00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 --2- 172.21.15.67:0/155547932 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1a80039b0 0x7fd1c4145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 >> 172.21.15.67:0/155547932 conn(0x7fd1c406b8c0 msgr2=0x7fd1c40cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 shutdown_connections 2024-04-17T08:31:28.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.035+0000 7fd1cae00640 1 -- 172.21.15.67:0/155547932 wait complete. 2024-04-17T08:31:28.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:28.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.275+0000 7f4511400640 1 Processor -- start 2024-04-17T08:31:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.276+0000 7f4511400640 1 -- start start 2024-04-17T08:31:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.277+0000 7f4511400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.277+0000 7f4511400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.277+0000 7f4511400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450c0cc500 con 0x7f450c0ce4a0 2024-04-17T08:31:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.277+0000 7f4511400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f450c0c71e0 con 0x7f450c161e70 2024-04-17T08:31:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.277+0000 7f4511400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f450c0cc7a0 con 0x7f450c0cd5a0 2024-04-17T08:31:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.278+0000 7f450a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.278+0000 7f450b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.278+0000 7f450a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:44052/0 (socket says 172.21.15.67:44052) 2024-04-17T08:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.278+0000 7f450a800640 1 -- 172.21.15.67:0/2219777235 learned_addr learned my addr 172.21.15.67:0/2219777235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.279+0000 7f450b200640 1 -- 172.21.15.67:0/2219777235 >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 msgr2=0x7f450c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.279+0000 7f450b200640 1 --2- 172.21.15.67:0/2219777235 >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.279+0000 7f450b200640 1 -- 172.21.15.67:0/2219777235 >> v1:172.21.15.187:6789/0 conn(0x7f450c0ce4a0 legacy=0x7f450c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.279+0000 7f450b200640 1 -- 172.21.15.67:0/2219777235 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c0cca80 con 0x7f450c0cd5a0 2024-04-17T08:31:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.279+0000 7f450a800640 1 --2- 172.21.15.67:0/2219777235 >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:31:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.281+0000 7f450b200640 1 --2- 172.21.15.67:0/2219777235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c162fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4504009d90 tx=0x7f450402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d514c34337ef6712 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.281+0000 7f4509e00640 1 -- 172.21.15.67:0/2219777235 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504039070 con 0x7f450c0cd5a0 2024-04-17T08:31:28.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4509e00640 1 -- 172.21.15.67:0/2219777235 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f450402cc50 con 0x7f450c0cd5a0 2024-04-17T08:31:28.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4509e00640 1 -- 172.21.15.67:0/2219777235 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504034440 con 0x7f450c0cd5a0 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 -- 172.21.15.67:0/2219777235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 msgr2=0x7f450c162fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 --2- 172.21.15.67:0/2219777235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c162fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4504009d90 tx=0x7f450402bea0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 -- 172.21.15.67:0/2219777235 shutdown_connections 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 --2- 172.21.15.67:0/2219777235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c162fe0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 --2- 172.21.15.67:0/2219777235 >> v2:172.21.15.187:3300/0 conn(0x7f450c161e70 0x7f450c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 -- 172.21.15.67:0/2219777235 >> 172.21.15.67:0/2219777235 conn(0x7f450c06bf50 msgr2=0x7f450c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 -- 172.21.15.67:0/2219777235 shutdown_connections 2024-04-17T08:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.282+0000 7f4511400640 1 -- 172.21.15.67:0/2219777235 wait complete. 2024-04-17T08:31:28.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.284+0000 7f4511400640 1 Processor -- start 2024-04-17T08:31:28.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.284+0000 7f4511400640 1 -- start start 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c1f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f450c161e70 0x7f450c1f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f450c0c5630 con 0x7f450c0ce4a0 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f450c0c54b0 con 0x7f450c161e70 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4511400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f450c0c57b0 con 0x7f450c0cd5a0 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.285+0000 7f4503e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53860/0 (socket says 172.21.15.67:53860) 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 -- 172.21.15.67:0/1115177856 learned_addr learned my addr 172.21.15.67:0/1115177856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f450a800640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c1f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f450b200640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f450c161e70 0x7f450c1f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 -- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 msgr2=0x7f450c1f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c1f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 -- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f450c161e70 msgr2=0x7f450c1f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f450c161e70 0x7f450c1f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.286+0000 7f4503e00640 1 -- 172.21.15.67:0/1115177856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c06e270 con 0x7f450c0ce4a0 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4503e00640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c400 tx=0x7f44f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f40052d0 con 0x7f450c0ce4a0 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f4005490 con 0x7f450c0ce4a0 2024-04-17T08:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4007260 con 0x7f450c0ce4a0 2024-04-17T08:31:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450c06e540 con 0x7f450c0ce4a0 2024-04-17T08:31:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.287+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450c06eb00 con 0x7f450c0ce4a0 2024-04-17T08:31:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.288+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f450c163470 con 0x7f450c0ce4a0 2024-04-17T08:31:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.289+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f44f4007400 con 0x7f450c0ce4a0 2024-04-17T08:31:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.290+0000 7f4502a00640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 0x7f44e806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.290+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4885+0+0 (secure 0 0 0) 0x7f44f408aff0 con 0x7f450c0ce4a0 2024-04-17T08:31:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.291+0000 7f450a800640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 0x7f44e806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.293+0000 7f450a800640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 0x7f44e806eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009ac0 tx=0x7f44fc008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.295+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f450c163470 con 0x7f450c0ce4a0 2024-04-17T08:31:28.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.501+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f450c15aec0 con 0x7f450c0ce4a0 2024-04-17T08:31:28.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.502+0000 7f4502a00640 1 -- 172.21.15.67:0/1115177856 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) ==== 74+0+12897 (secure 0 0 0) 0x7f44f40595a0 con 0x7f450c0ce4a0 2024-04-17T08:31:28.501 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:28.501 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":191,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:27.496492+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 msgr2=0x7f44e806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 0x7f44e806eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009ac0 tx=0x7f44fc008040 comp rx=0 tx=0).stop 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 msgr2=0x7f450c06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c400 tx=0x7f44f400c8c0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 shutdown_connections 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f44e806ca10 0x7f44e806eed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f450c161e70 0x7f450c1f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f450c0ce4a0 0x7f450c06dd30 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 --2- 172.21.15.67:0/1115177856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f450c0cd5a0 0x7f450c1f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.505+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 >> 172.21.15.67:0/1115177856 conn(0x7f450c06bf50 msgr2=0x7f450c0b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.506+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 shutdown_connections 2024-04-17T08:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.506+0000 7f4511400640 1 -- 172.21.15.67:0/1115177856 wait complete. 2024-04-17T08:31:28.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:31:28.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:31:28.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.753+0000 7fc8d9e00640 1 Processor -- start 2024-04-17T08:31:28.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.753+0000 7fc8d9e00640 1 -- start start 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8d9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d40b4260 0x7fc8d415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8d9e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d415f240 con 0x7fc8d40cf620 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8d9e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d40b6420 con 0x7fc8d4144fb0 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8d9e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d40b61c0 con 0x7fc8d40b4260 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8cbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d40b4260 0x7fc8d415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8cbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d40b4260 0x7fc8d415fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53876/0 (socket says 172.21.15.67:53876) 2024-04-17T08:31:28.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.755+0000 7fc8cbe00640 1 -- 172.21.15.67:0/2807294533 learned_addr learned my addr 172.21.15.67:0/2807294533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:28.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.756+0000 7fc8cbe00640 1 -- 172.21.15.67:0/2807294533 >> v1:172.21.15.187:6789/0 conn(0x7fc8d4144fb0 legacy=0x7fc8d40b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:28.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.756+0000 7fc8cbe00640 1 -- 172.21.15.67:0/2807294533 >> v1:172.21.15.67:6789/0 conn(0x7fc8d40cf620 legacy=0x7fc8d406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.756+0000 7fc8cbe00640 1 -- 172.21.15.67:0/2807294533 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d415f4b0 con 0x7fc8d40b4260 2024-04-17T08:31:28.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.757+0000 7fc8cbe00640 1 --2- 172.21.15.67:0/2807294533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d40b4260 0x7fc8d415fcb0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009870 tx=0x7fc8bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adb1728002fdf6a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.757+0000 7fc8d3e00640 1 -- 172.21.15.67:0/2807294533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc034070 con 0x7fc8d40b4260 2024-04-17T08:31:28.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.757+0000 7fc8d3e00640 1 -- 172.21.15.67:0/2807294533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8bc030860 con 0x7fc8d40b4260 2024-04-17T08:31:28.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.757+0000 7fc8d3e00640 1 -- 172.21.15.67:0/2807294533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc030c30 con 0x7fc8d40b4260 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 -- 172.21.15.67:0/2807294533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d40b4260 msgr2=0x7fc8d415fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/2807294533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d40b4260 0x7fc8d415fcb0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009870 tx=0x7fc8bc02b9a0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 -- 172.21.15.67:0/2807294533 shutdown_connections 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/2807294533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d40b4260 0x7fc8d415fcb0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 -- 172.21.15.67:0/2807294533 >> 172.21.15.67:0/2807294533 conn(0x7fc8d40b23c0 msgr2=0x7fc8d40b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:28.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 -- 172.21.15.67:0/2807294533 shutdown_connections 2024-04-17T08:31:28.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.758+0000 7fc8d9e00640 1 -- 172.21.15.67:0/2807294533 wait complete. 2024-04-17T08:31:28.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.760+0000 7fc8d9e00640 1 Processor -- start 2024-04-17T08:31:28.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.760+0000 7fc8d9e00640 1 -- start start 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 0x7fc8d413f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8d40cf620 0x7fc8d40ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d4145390 con 0x7fc8d4144fb0 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d40b6700 con 0x7fc8d40cf620 2024-04-17T08:31:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d40cfa70 con 0x7fc8d40b4260 2024-04-17T08:31:28.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53880/0 (socket says 172.21.15.67:53880) 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.761+0000 7fc8d9400640 1 -- 172.21.15.67:0/3944936851 learned_addr learned my addr 172.21.15.67:0/3944936851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8cbe00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8d40cf620 0x7fc8d40ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d8a00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 0x7fc8d413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 -- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 msgr2=0x7fc8d413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 0x7fc8d413f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 -- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8d40cf620 msgr2=0x7fc8d40ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8d40cf620 0x7fc8d40ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 -- 172.21.15.67:0/3944936851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d413f6b0 con 0x7fc8d4144fb0 2024-04-17T08:31:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d8a00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 0x7fc8d413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9400640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00c7d0 tx=0x7fc8cc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc003b70 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8cc003d30 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00aa90 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.762+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d40ae820 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.763+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d41e5270 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.764+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d4068a10 con 0x7fc8d4144fb0 2024-04-17T08:31:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.765+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fc8cc00ac30 con 0x7fc8d4144fb0 2024-04-17T08:31:28.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.766+0000 7fc8d1600640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 0x7fc8b006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:28.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.766+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4885+0+0 (secure 0 0 0) 0x7fc8cc08b9d0 con 0x7fc8d4144fb0 2024-04-17T08:31:28.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.766+0000 7fc8d8a00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 0x7fc8b006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:28.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.767+0000 7fc8d8a00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 0x7fc8b006ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a830 tx=0x7fc8c4008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:28.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.770+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc8cc059f80 con 0x7fc8d4144fb0 2024-04-17T08:31:28.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.978+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8d4068610 con 0x7fc8d4144fb0 2024-04-17T08:31:28.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.980+0000 7fc8d1600640 1 -- 172.21.15.67:0/3944936851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v191) ==== 93+0+10 (secure 0 0 0) 0x7fc8cc059880 con 0x7fc8d4144fb0 2024-04-17T08:31:28.978 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:31:28.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 msgr2=0x7fc8b006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 0x7fc8b006ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a830 tx=0x7fc8c4008040 comp rx=0 tx=0).stop 2024-04-17T08:31:28.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 msgr2=0x7fc8d41f44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:28.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc00c7d0 tx=0x7fc8cc00cc90 comp rx=0 tx=0).stop 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 shutdown_connections 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d4144fb0 0x7fc8d41f44a0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc8b006ca40 0x7fc8b006ef00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8d40cf620 0x7fc8d40ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.982+0000 7fc8d9e00640 1 --2- 172.21.15.67:0/3944936851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8d40b4260 0x7fc8d413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.983+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 >> 172.21.15.67:0/3944936851 conn(0x7fc8d40b23c0 msgr2=0x7fc8d40b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.983+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 shutdown_connections 2024-04-17T08:31:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:28.983+0000 7fc8d9e00640 1 -- 172.21.15.67:0/3944936851 wait complete. 2024-04-17T08:31:29.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:31:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.231+0000 7f21e4c00640 1 Processor -- start 2024-04-17T08:31:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.231+0000 7f21e4c00640 1 -- start start 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21e4c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21e006b8c0 0x7f21e00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21e4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21e0160280 0x7f21e006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21e4c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e00cd4a0 con 0x7f21e0141ed0 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21e4c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21e015ce30 con 0x7f21e0160280 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21e4c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f21e01607f0 con 0x7f21e006b8c0 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21dea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21e0160280 0x7f21e006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21dfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21e0141ed0 0x7f21e015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44196/0 (socket says 172.21.15.67:44196) 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.233+0000 7f21dfe00640 1 -- 172.21.15.67:0/2765312795 learned_addr learned my addr 172.21.15.67:0/2765312795 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.234+0000 7f21df400640 1 --2- 172.21.15.67:0/2765312795 >> v2:172.21.15.187:3300/0 conn(0x7f21e006b8c0 0x7f21e00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.234+0000 7f21dea00640 1 -- 172.21.15.67:0/2765312795 >> v2:172.21.15.187:3300/0 conn(0x7f21e006b8c0 msgr2=0x7f21e00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.234+0000 7f21dea00640 1 --2- 172.21.15.67:0/2765312795 >> v2:172.21.15.187:3300/0 conn(0x7f21e006b8c0 0x7f21e00cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.234+0000 7f21dea00640 1 -- 172.21.15.67:0/2765312795 >> v1:172.21.15.67:6789/0 conn(0x7f21e0141ed0 legacy=0x7f21e015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.234+0000 7f21dea00640 1 -- 172.21.15.67:0/2765312795 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e0160ad0 con 0x7f21e0160280 2024-04-17T08:31:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.235+0000 7f21dea00640 1 --2- 172.21.15.67:0/2765312795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0160280 0x7f21e006bca0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f21e00b3210 tx=0x7f21cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1a972c070d8fec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.235+0000 7f21de000640 1 -- 172.21.15.67:0/2765312795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc035020 con 0x7f21e0160280 2024-04-17T08:31:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.235+0000 7f21de000640 1 -- 172.21.15.67:0/2765312795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21cc031830 con 0x7f21e0160280 2024-04-17T08:31:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.235+0000 7f21de000640 1 -- 172.21.15.67:0/2765312795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc031c00 con 0x7f21e0160280 2024-04-17T08:31:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 -- 172.21.15.67:0/2765312795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0160280 msgr2=0x7f21e006bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 --2- 172.21.15.67:0/2765312795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0160280 0x7f21e006bca0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f21e00b3210 tx=0x7f21cc02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 -- 172.21.15.67:0/2765312795 shutdown_connections 2024-04-17T08:31:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 --2- 172.21.15.67:0/2765312795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0160280 0x7f21e006bca0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 --2- 172.21.15.67:0/2765312795 >> v2:172.21.15.187:3300/0 conn(0x7f21e006b8c0 0x7f21e00cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 -- 172.21.15.67:0/2765312795 >> 172.21.15.67:0/2765312795 conn(0x7f21e0067d10 msgr2=0x7f21e00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 -- 172.21.15.67:0/2765312795 shutdown_connections 2024-04-17T08:31:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.236+0000 7f21e4c00640 1 -- 172.21.15.67:0/2765312795 wait complete. 2024-04-17T08:31:29.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.238+0000 7f21e4c00640 1 Processor -- start 2024-04-17T08:31:29.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.238+0000 7f21e4c00640 1 -- start start 2024-04-17T08:31:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21e006b8c0 0x7f21e013b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21e0160280 0x7f21e015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21e006c1e0 con 0x7f21e0141ed0 2024-04-17T08:31:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f21e00cddf0 con 0x7f21e0160280 2024-04-17T08:31:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21e4c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21e0161090 con 0x7f21e006b8c0 2024-04-17T08:31:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53892/0 (socket says 172.21.15.67:53892) 2024-04-17T08:31:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.239+0000 7f21dea00640 1 -- 172.21.15.67:0/4085353431 learned_addr learned my addr 172.21.15.67:0/4085353431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 -- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21e006b8c0 msgr2=0x7f21e013b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dfe00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21e0160280 0x7f21e015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21df400640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21e006b8c0 0x7f21e013b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21e006b8c0 0x7f21e013b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 -- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21e0160280 msgr2=0x7f21e015b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21e0160280 0x7f21e015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 -- 172.21.15.67:0/4085353431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e01457d0 con 0x7f21e0141ed0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21dea00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009890 tx=0x7f21cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc040070 con 0x7f21e0141ed0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21cc03e070 con 0x7f21e0141ed0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21cc035040 con 0x7f21e0141ed0 2024-04-17T08:31:29.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.240+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e0145a70 con 0x7f21e0141ed0 2024-04-17T08:31:29.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.241+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e00b3cc0 con 0x7f21e0141ed0 2024-04-17T08:31:29.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.242+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e006fe90 con 0x7f21e0141ed0 2024-04-17T08:31:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.243+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f21cc038020 con 0x7f21e0141ed0 2024-04-17T08:31:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.244+0000 7f21d7400640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 0x7f21c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.244+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4885+0+0 (secure 0 0 0) 0x7f21cc0ad300 con 0x7f21e0141ed0 2024-04-17T08:31:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.244+0000 7f21df400640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 0x7f21c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.246+0000 7f21df400640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 0x7f21c006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006fd0 tx=0x7f21d0008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.248+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f21cc07b9e0 con 0x7f21e0141ed0 2024-04-17T08:31:29.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.455+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f21e0068a10 con 0x7f21e0141ed0 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.456+0000 7f21d7400640 1 -- 172.21.15.67:0/4085353431 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v191) ==== 84+0+65 (secure 0 0 0) 0x7f21cc07b2e0 con 0x7f21e0141ed0 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:29.454 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.458+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 msgr2=0x7f21c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.458+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 0x7f21c006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006fd0 tx=0x7f21d0008040 comp rx=0 tx=0).stop 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.458+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 msgr2=0x7f21e01f89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.458+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009890 tx=0x7f21cc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 shutdown_connections 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f21c006ca10 0x7f21c006eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21e0160280 0x7f21e015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0141ed0 0x7f21e01f89c0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 --2- 172.21.15.67:0/4085353431 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21e006b8c0 0x7f21e013b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 >> 172.21.15.67:0/4085353431 conn(0x7f21e0067d10 msgr2=0x7f21e013f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 shutdown_connections 2024-04-17T08:31:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.459+0000 7f21e4c00640 1 -- 172.21.15.67:0/4085353431 wait complete. 2024-04-17T08:31:29.477 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:31:29.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:31:29.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.696+0000 7f6bf7000640 1 Processor -- start 2024-04-17T08:31:29.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.696+0000 7f6bf7000640 1 -- start start 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 0x7f6bf00cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf7000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6bf00cd6d0 0x7f6bf0145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf7000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf0146510 con 0x7f6bf0146080 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf7000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf0161000 con 0x7f6bf00cd6d0 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf00ce870 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf6600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6bf0146080 0x7f6bf0160360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44200/0 (socket says 172.21.15.67:44200) 2024-04-17T08:31:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf6600640 1 -- 172.21.15.67:0/91507673 learned_addr learned my addr 172.21.15.67:0/91507673 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.698+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/91507673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 0x7f6bf00cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6bf5200640 1 --2- 172.21.15.67:0/91507673 >> v2:172.21.15.187:3300/0 conn(0x7f6bf00cd6d0 0x7f6bf0145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3340930685 0 0) 0x7f6bf0146510 con 0x7f6bf0146080 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd4003520 con 0x7f6bf0146080 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3142481668 0 0) 0x7f6bd4003520 con 0x7f6bf0146080 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6bf5c00640 1 -- 172.21.15.67:0/91507673 >> v2:172.21.15.187:3300/0 conn(0x7f6bf00cd6d0 msgr2=0x7f6bf0145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/91507673 >> v2:172.21.15.187:3300/0 conn(0x7f6bf00cd6d0 0x7f6bf0145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6bf5c00640 1 -- 172.21.15.67:0/91507673 >> v1:172.21.15.67:6789/0 conn(0x7f6bf0146080 legacy=0x7f6bf0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.699+0000 7f6bf5c00640 1 -- 172.21.15.67:0/91507673 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf00ceb50 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.701+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/91507673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 0x7f6bf00cf350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6be00097a0 tx=0x7f6be002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988b0aec3eca95e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.701+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0037070 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.701+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be0030510 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.701+0000 7f6befe00640 1 -- 172.21.15.67:0/91507673 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80019f0 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 -- 172.21.15.67:0/91507673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 msgr2=0x7f6bf00cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 --2- 172.21.15.67:0/91507673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 0x7f6bf00cf350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6be00097a0 tx=0x7f6be002b790 comp rx=0 tx=0).stop 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 -- 172.21.15.67:0/91507673 shutdown_connections 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 --2- 172.21.15.67:0/91507673 >> v2:172.21.15.187:3300/0 conn(0x7f6bf00cd6d0 0x7f6bf0145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 --2- 172.21.15.67:0/91507673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd2f0 0x7f6bf00cf350 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 -- 172.21.15.67:0/91507673 >> 172.21.15.67:0/91507673 conn(0x7f6bf006b8c0 msgr2=0x7f6bf006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 -- 172.21.15.67:0/91507673 shutdown_connections 2024-04-17T08:31:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.702+0000 7f6bf7000640 1 -- 172.21.15.67:0/91507673 wait complete. 2024-04-17T08:31:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.704+0000 7f6bf7000640 1 Processor -- start 2024-04-17T08:31:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.704+0000 7f6bf7000640 1 -- start start 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf00cd2f0 0x7f6bf013f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd6d0 0x7f6bf015d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf00cfa10 con 0x7f6bf0146080 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf00cf890 con 0x7f6bf00cd2f0 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf0160a70 con 0x7f6bf00cd6d0 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53908/0 (socket says 172.21.15.67:53908) 2024-04-17T08:31:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 -- 172.21.15.67:0/920892961 learned_addr learned my addr 172.21.15.67:0/920892961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf5200640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd6d0 0x7f6bf015d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf00cd2f0 0x7f6bf013f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 -- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd6d0 msgr2=0x7f6bf015d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd6d0 0x7f6bf015d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 -- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf00cd2f0 msgr2=0x7f6bf013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.705+0000 7f6bf6600640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf00cd2f0 0x7f6bf013f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bf6600640 1 -- 172.21.15.67:0/920892961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf006eb20 con 0x7f6bf0146080 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bf6600640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f6be800cc40 tx=0x7f6be800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800ce50 con 0x7f6bf0146080 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be8012400 con 0x7f6bf0146080 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80127f0 con 0x7f6bf0146080 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf006ee00 con 0x7f6bf0146080 2024-04-17T08:31:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.706+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf006f3c0 con 0x7f6bf0146080 2024-04-17T08:31:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.707+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf0068a10 con 0x7f6bf0146080 2024-04-17T08:31:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.708+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f6be8012990 con 0x7f6bf0146080 2024-04-17T08:31:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.709+0000 7f6bed600640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 0x7f6bd006f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.709+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(191..191 src has 1..191) ==== 4885+0+0 (secure 0 0 0) 0x7f6be808c280 con 0x7f6bf0146080 2024-04-17T08:31:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.710+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 0x7f6bd006f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.711+0000 7f6bf5c00640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 0x7f6bd006f0a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be002f000 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.714+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6be80182b0 con 0x7f6bf0146080 2024-04-17T08:31:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:29.920+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f6bf0068610 con 0x7f6bf0146080 2024-04-17T08:31:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.507+0000 7f6bed600640 1 -- 172.21.15.67:0/920892961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v192) ==== 138+0+0 (secure 0 0 0) 0x7f6be805a830 con 0x7f6bf0146080 2024-04-17T08:31:30.506 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 msgr2=0x7f6bd006f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 0x7f6bd006f0a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6be0034000 tx=0x7f6be002f000 comp rx=0 tx=0).stop 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 msgr2=0x7f6bf015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f6be800cc40 tx=0x7f6be800a430 comp rx=0 tx=0).stop 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 shutdown_connections 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf0146080 0x7f6bf015e240 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6bf00cd6d0 0x7f6bf015d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6bd006cbe0 0x7f6bd006f0a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 --2- 172.21.15.67:0/920892961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bf00cd2f0 0x7f6bf013f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 >> 172.21.15.67:0/920892961 conn(0x7f6bf006b8c0 msgr2=0x7f6bf00cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 shutdown_connections 2024-04-17T08:31:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.511+0000 7f6bf7000640 1 -- 172.21.15.67:0/920892961 wait complete. 2024-04-17T08:31:30.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:31:30.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.747+0000 7f550ea00640 1 Processor -- start 2024-04-17T08:31:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.747+0000 7f550ea00640 1 -- start start 2024-04-17T08:31:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.749+0000 7f550ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55080ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.749+0000 7f550ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55080b20b0 0x7f55080b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.749+0000 7f550ea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55080b29f0 con 0x7f550815c580 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.749+0000 7f550ea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f55080b3770 con 0x7f55080b20b0 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.749+0000 7f550ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f550815f050 con 0x7f550815c1a0 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550cc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f550815c580 0x7f55080af470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44202/0 (socket says 172.21.15.67:44202) 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550cc00640 1 -- 172.21.15.67:0/2141169209 learned_addr learned my addr 172.21.15.67:0/2141169209 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550e000640 1 --2- 172.21.15.67:0/2141169209 >> v2:172.21.15.67:3300/0 conn(0x7f55080b20b0 0x7f55080b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550d600640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55080ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550e000640 1 -- 172.21.15.67:0/2141169209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 msgr2=0x7f55080ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550e000640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55080ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550e000640 1 -- 172.21.15.67:0/2141169209 >> v1:172.21.15.67:6789/0 conn(0x7f550815c580 legacy=0x7f55080af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.750+0000 7f550e000640 1 -- 172.21.15.67:0/2141169209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550815f330 con 0x7f55080b20b0 2024-04-17T08:31:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.752+0000 7f550e000640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f55080b24b0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38093ac2ebf7fa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.752+0000 7f5507e00640 1 -- 172.21.15.67:0/2141169209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500039070 con 0x7f55080b20b0 2024-04-17T08:31:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.752+0000 7f5507e00640 1 -- 172.21.15.67:0/2141169209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550002cc60 con 0x7f55080b20b0 2024-04-17T08:31:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.752+0000 7f5507e00640 1 -- 172.21.15.67:0/2141169209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5500034660 con 0x7f55080b20b0 2024-04-17T08:31:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 -- 172.21.15.67:0/2141169209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 msgr2=0x7f55080b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f55080b24b0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f5500009d90 tx=0x7f550002beb0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 -- 172.21.15.67:0/2141169209 shutdown_connections 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f55080b24b0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 --2- 172.21.15.67:0/2141169209 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55080ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 -- 172.21.15.67:0/2141169209 >> 172.21.15.67:0/2141169209 conn(0x7f5508067d10 msgr2=0x7f55080d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 -- 172.21.15.67:0/2141169209 shutdown_connections 2024-04-17T08:31:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.753+0000 7f550ea00640 1 -- 172.21.15.67:0/2141169209 wait complete. 2024-04-17T08:31:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.755+0000 7f550ea00640 1 Processor -- start 2024-04-17T08:31:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.755+0000 7f550ea00640 1 -- start start 2024-04-17T08:31:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55081f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f550815c580 0x7f55080aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55080afb80 con 0x7f55080b20b0 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f550815f8f0 con 0x7f550815c580 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55080afd00 con 0x7f550815c1a0 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.756+0000 7f550d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53926/0 (socket says 172.21.15.67:53926) 2024-04-17T08:31:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 -- 172.21.15.67:0/3760871332 learned_addr learned my addr 172.21.15.67:0/3760871332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:30.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550e000640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f550815c580 0x7f55080aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550cc00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55081f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 -- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 msgr2=0x7f55081f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55081f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 -- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f550815c580 msgr2=0x7f55080aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f550815c580 0x7f55080aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 -- 172.21.15.67:0/3760871332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55080cfce0 con 0x7f55080b20b0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550cc00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55081f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.757+0000 7f550d600640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00d800 tx=0x7f54fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.758+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00b6c0 con 0x7f55080b20b0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.758+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54fc019070 con 0x7f55080b20b0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.758+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00ba80 con 0x7f55080b20b0 2024-04-17T08:31:30.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.758+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55080ccf80 con 0x7f55080b20b0 2024-04-17T08:31:30.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.758+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55080cd240 con 0x7f55080b20b0 2024-04-17T08:31:30.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.759+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5508068a10 con 0x7f55080b20b0 2024-04-17T08:31:30.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.761+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f54fc00bc20 con 0x7f55080b20b0 2024-04-17T08:31:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.762+0000 7f5505600640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 0x7f54e806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.762+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5255+0+0 (secure 0 0 0) 0x7f54fc08bf80 con 0x7f55080b20b0 2024-04-17T08:31:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.762+0000 7f550cc00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 0x7f54e806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:30.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.764+0000 7f550cc00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 0x7f54e806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55080d09a0 tx=0x7f54f4009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:30.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.767+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f55080cfce0 con 0x7f55080b20b0 2024-04-17T08:31:30.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:30.972+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f55080d06f0 con 0x7f55080b20b0 2024-04-17T08:31:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.514+0000 7f5505600640 1 -- 172.21.15.67:0/3760871332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v193) ==== 170+0+0 (secure 0 0 0) 0x7f54fc05a4f0 con 0x7f55080b20b0 2024-04-17T08:31:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:31:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 msgr2=0x7f54e806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 0x7f54e806eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55080d09a0 tx=0x7f54f4009040 comp rx=0 tx=0).stop 2024-04-17T08:31:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 msgr2=0x7f5508162040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00d800 tx=0x7f54fc00dcc0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 shutdown_connections 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f54e806ca10 0x7f54e806eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f550815c580 0x7f55080aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f550815c1a0 0x7f55081f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 --2- 172.21.15.67:0/3760871332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55080b20b0 0x7f5508162040 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.518+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 >> 172.21.15.67:0/3760871332 conn(0x7f5508067d10 msgr2=0x7f55080b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.519+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 shutdown_connections 2024-04-17T08:31:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.519+0000 7f550ea00640 1 -- 172.21.15.67:0/3760871332 wait complete. 2024-04-17T08:31:31.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:31:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.742+0000 7fc47e200640 1 Processor -- start 2024-04-17T08:31:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.742+0000 7fc47e200640 1 -- start start 2024-04-17T08:31:31.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.743+0000 7fc47e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc478145f80 0x7fc4780cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.743+0000 7fc47e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc47806c270 0x7fc47806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.743+0000 7fc47e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc478146360 con 0x7fc4780cd750 2024-04-17T08:31:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.743+0000 7fc47e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc4780cdbb0 con 0x7fc47806c270 2024-04-17T08:31:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.743+0000 7fc47e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc4780727a0 con 0x7fc478145f80 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.744+0000 7fc47d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc47806c270 0x7fc47806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.744+0000 7fc47d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc47806c270 0x7fc47806e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53942/0 (socket says 172.21.15.67:53942) 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.744+0000 7fc47d800640 1 -- 172.21.15.67:0/4139954478 learned_addr learned my addr 172.21.15.67:0/4139954478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.744+0000 7fc47ce00640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc478145f80 0x7fc4780cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.745+0000 7fc47d800640 1 -- 172.21.15.67:0/4139954478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc478145f80 msgr2=0x7fc4780cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.745+0000 7fc47d800640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc478145f80 0x7fc4780cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.745+0000 7fc47d800640 1 -- 172.21.15.67:0/4139954478 >> v1:172.21.15.67:6789/0 conn(0x7fc4780cd750 legacy=0x7fc47806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.745+0000 7fc47d800640 1 -- 172.21.15.67:0/4139954478 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc478071e70 con 0x7fc47806c270 2024-04-17T08:31:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.746+0000 7fc47d800640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc47806c270 0x7fc47806e1c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fc470009e50 tx=0x7fc47002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a90ea9975f2126d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.746+0000 7fc477400640 1 -- 172.21.15.67:0/4139954478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47002ca70 con 0x7fc47806c270 2024-04-17T08:31:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.746+0000 7fc477400640 1 -- 172.21.15.67:0/4139954478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc47002cc10 con 0x7fc47806c270 2024-04-17T08:31:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.746+0000 7fc477400640 1 -- 172.21.15.67:0/4139954478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4700343f0 con 0x7fc47806c270 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 -- 172.21.15.67:0/4139954478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc47806c270 msgr2=0x7fc47806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc47806c270 0x7fc47806e1c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fc470009e50 tx=0x7fc47002bed0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 -- 172.21.15.67:0/4139954478 shutdown_connections 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc47806c270 0x7fc47806e1c0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 --2- 172.21.15.67:0/4139954478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc478145f80 0x7fc4780cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 -- 172.21.15.67:0/4139954478 >> 172.21.15.67:0/4139954478 conn(0x7fc4780b2190 msgr2=0x7fc4780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 -- 172.21.15.67:0/4139954478 shutdown_connections 2024-04-17T08:31:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.747+0000 7fc47e200640 1 -- 172.21.15.67:0/4139954478 wait complete. 2024-04-17T08:31:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.749+0000 7fc47e200640 1 Processor -- start 2024-04-17T08:31:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.749+0000 7fc47e200640 1 -- start start 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47806c270 0x7fc47815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4780cd750 0x7fc47815e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc478072c30 con 0x7fc478145f80 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc478072430 con 0x7fc47806c270 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47e200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc478072db0 con 0x7fc4780cd750 2024-04-17T08:31:31.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53952/0 (socket says 172.21.15.67:53952) 2024-04-17T08:31:31.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.750+0000 7fc47d800640 1 -- 172.21.15.67:0/428507406 learned_addr learned my addr 172.21.15.67:0/428507406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc477e00640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4780cd750 0x7fc47815e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 -- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4780cd750 msgr2=0x7fc47815e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4780cd750 0x7fc47815e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 -- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47806c270 msgr2=0x7fc47815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47ce00640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47806c270 0x7fc47815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47806c270 0x7fc47815cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 -- 172.21.15.67:0/428507406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47815d2e0 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47d800640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fc470009f70 tx=0x7fc470004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470043020 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc47003e450 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470009510 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.751+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47815d5c0 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.752+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47815db80 con 0x7fc478145f80 2024-04-17T08:31:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.753+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc478068a10 con 0x7fc478145f80 2024-04-17T08:31:31.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.754+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fc470046020 con 0x7fc478145f80 2024-04-17T08:31:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.755+0000 7fc474c00640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 0x7fc45806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:31.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.755+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 5637+0+0 (secure 0 0 0) 0x7fc4700ae900 con 0x7fc478145f80 2024-04-17T08:31:31.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.756+0000 7fc47ce00640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 0x7fc45806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:31.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.757+0000 7fc47ce00640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 0x7fc45806eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:31.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:31.760+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc47004d030 con 0x7fc478145f80 2024-04-17T08:31:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.001+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc47806ccd0 con 0x7fc478145f80 2024-04-17T08:31:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.568+0000 7fc474c00640 1 -- 172.21.15.67:0/428507406 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v90) ==== 174+0+0 (secure 0 0 0) 0x7fc47007ce60 con 0x7fc478145f80 2024-04-17T08:31:32.570 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 22 and data pool 23 2024-04-17T08:31:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.579+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 msgr2=0x7fc45806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.579+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 0x7fc45806eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).stop 2024-04-17T08:31:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.579+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 msgr2=0x7fc478142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.579+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fc470009f70 tx=0x7fc470004a40 comp rx=0 tx=0).stop 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 shutdown_connections 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc478145f80 0x7fc478142240 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4780cd750 0x7fc47815e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc45806ca10 0x7fc45806eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 --2- 172.21.15.67:0/428507406 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc47806c270 0x7fc47815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.580+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 >> 172.21.15.67:0/428507406 conn(0x7fc4780b2190 msgr2=0x7fc4780b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.582+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 shutdown_connections 2024-04-17T08:31:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.582+0000 7fc47e200640 1 -- 172.21.15.67:0/428507406 wait complete. 2024-04-17T08:31:32.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.793+0000 7f66c4a00640 1 Processor -- start 2024-04-17T08:31:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.793+0000 7f66c4a00640 1 -- start start 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.794+0000 7f66c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 0x7f66c00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66c4a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66c0141ed0 0x7f66c0142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66c4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c00cd4a0 con 0x7f66c0160280 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66c4a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f66c015ce30 con 0x7f66c0141ed0 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66c01607f0 con 0x7f66c006b8c0 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66b5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66c0160280 0x7f66c006bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44216/0 (socket says 172.21.15.67:44216) 2024-04-17T08:31:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66b5e00640 1 -- 172.21.15.67:0/2004598185 learned_addr learned my addr 172.21.15.67:0/2004598185 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66be800640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 0x7f66c00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66bf200640 1 --2- 172.21.15.67:0/2004598185 >> v2:172.21.15.187:3300/0 conn(0x7f66c0141ed0 0x7f66c0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449539128 0 0) 0x7f66c00cd4a0 con 0x7f66c0160280 2024-04-17T08:31:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.795+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a4003520 con 0x7f66c0160280 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2739128093 0 0) 0x7f66a4003520 con 0x7f66c0160280 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 msgr2=0x7f66c00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 0x7f66c00cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 >> v2:172.21.15.187:3300/0 conn(0x7f66c0141ed0 msgr2=0x7f66c0142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 --2- 172.21.15.67:0/2004598185 >> v2:172.21.15.187:3300/0 conn(0x7f66c0141ed0 0x7f66c0142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c0160ad0 con 0x7f66c0160280 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f66a8001670 con 0x7f66c0160280 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 >> v1:172.21.15.67:6789/0 conn(0x7f66c0160280 legacy=0x7f66c006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a4003a20 0x7f66a4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66a4004800 0x7f66a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 0x7f66a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66c015ce30 con 0x7f66a4003a20 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66c01607f0 con 0x7f66a4004800 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66a400bb40 con 0x7f66c006b8c0 2024-04-17T08:31:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.796+0000 7f66be800640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a4003a20 0x7f66a4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66b5e00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66a4004800 0x7f66a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66be800640 1 -- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 msgr2=0x7f66a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66be800640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66c006b8c0 0x7f66a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66be800640 1 -- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66a4004800 msgr2=0x7f66a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66be800640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66a4004800 0x7f66a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66be800640 1 -- 172.21.15.67:0/2004598185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a4003520 con 0x7f66a4003a20 2024-04-17T08:31:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.797+0000 7f66b5e00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66a4004800 0x7f66a4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.798+0000 7f66be800640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a4003a20 0x7f66a4003e40 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f66b0007360 tx=0x7f66a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6e404902c75249 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.798+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0039070 con 0x7f66a4003a20 2024-04-17T08:31:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.798+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b002ccf0 con 0x7f66a4003a20 2024-04-17T08:31:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.798+0000 7f66bde00640 1 -- 172.21.15.67:0/2004598185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0009be0 con 0x7f66a4003a20 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 -- 172.21.15.67:0/2004598185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a4003a20 msgr2=0x7f66a4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 --2- 172.21.15.67:0/2004598185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a4003a20 0x7f66a4003e40 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f66b0007360 tx=0x7f66a4005830 comp rx=0 tx=0).stop 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66bf200640 1 -- 172.21.15.67:0/2004598185 reap_dead start 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 -- 172.21.15.67:0/2004598185 shutdown_connections 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 -- 172.21.15.67:0/2004598185 >> 172.21.15.67:0/2004598185 conn(0x7f66c0067d10 msgr2=0x7f66c00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 -- 172.21.15.67:0/2004598185 shutdown_connections 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.799+0000 7f66c4a00640 1 -- 172.21.15.67:0/2004598185 wait complete. 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.800+0000 7f66c4a00640 1 Processor -- start 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.801+0000 7f66c4a00640 1 -- start start 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.801+0000 7f66c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66a4003a20 0x7f66c0155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.801+0000 7f66c4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 0x7f66c015e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66c4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66c4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66c013f9b0 con 0x7f66c00b5510 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66c4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0161090 con 0x7f66c00b5090 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66c013fb30 con 0x7f66a4003a20 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53978/0 (socket says 172.21.15.67:53978) 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 -- 172.21.15.67:0/4112156082 learned_addr learned my addr 172.21.15.67:0/4112156082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66be800640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66a4003a20 0x7f66c0155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66b5e00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 0x7f66c015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 -- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66a4003a20 msgr2=0x7f66c0155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66a4003a20 0x7f66c0155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 -- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 msgr2=0x7f66c015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 0x7f66c015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66bf200640 1 -- 172.21.15.67:0/4112156082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c00b3ff0 con 0x7f66c00b5510 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.802+0000 7f66b5e00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 0x7f66c015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66bf200640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f66c00b00d0 tx=0x7f66b800d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8004020 con 0x7f66c00b5510 2024-04-17T08:31:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b80041c0 con 0x7f66c00b5510 2024-04-17T08:31:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c00b42d0 con 0x7f66c00b5510 2024-04-17T08:31:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80043f0 con 0x7f66c00b5510 2024-04-17T08:31:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.803+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c00b4890 con 0x7f66c00b5510 2024-04-17T08:31:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.804+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c0142360 con 0x7f66c00b5510 2024-04-17T08:31:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.805+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f66b801a020 con 0x7f66c00b5510 2024-04-17T08:31:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.806+0000 7f66b7400640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 0x7f669c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.806+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 5737+0+0 (secure 0 0 0) 0x7f66b808b480 con 0x7f66c00b5510 2024-04-17T08:31:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.807+0000 7f66be800640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 0x7f669c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.808+0000 7f66be800640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 0x7f669c06eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66b0036000 tx=0x7f66b0030000 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:32.811+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f66c0142360 con 0x7f66c00b5510 2024-04-17T08:31:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.015+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66c00ce5e0 con 0x7f66c00b5510 2024-04-17T08:31:33.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.017+0000 7f66b7400640 1 -- 172.21.15.67:0/4112156082 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) ==== 74+0+16680 (secure 0 0 0) 0x7f66c00ce5e0 con 0x7f66c00b5510 2024-04-17T08:31:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:33.017 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":194,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:32.003105+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 msgr2=0x7f669c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 0x7f669c06eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66b0036000 tx=0x7f66b0030000 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 msgr2=0x7f66c0142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f66c00b00d0 tx=0x7f66b800d820 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 shutdown_connections 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c00b5510 0x7f66c0142fc0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f669c06ca10 0x7f669c06eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66c00b5090 0x7f66c015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 --2- 172.21.15.67:0/4112156082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66a4003a20 0x7f66c0155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 >> 172.21.15.67:0/4112156082 conn(0x7f66c0067d10 msgr2=0x7f66c00d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 shutdown_connections 2024-04-17T08:31:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.020+0000 7f66c4a00640 1 -- 172.21.15.67:0/4112156082 wait complete. 2024-04-17T08:31:33.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.246+0000 7f9e67e00640 1 Processor -- start 2024-04-17T08:31:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.246+0000 7f9e67e00640 1 -- start start 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e6815c5a0 0x7f9e6815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9e680b46c0 0x7f9e680ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6815f440 con 0x7f9e680b4210 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9e6815f720 con 0x7f9e6815c5a0 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9e6815fa00 con 0x7f9e680b46c0 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e66a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e6815c5a0 0x7f9e6815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e66a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e6815c5a0 0x7f9e6815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53982/0 (socket says 172.21.15.67:53982) 2024-04-17T08:31:33.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e66a00640 1 -- 172.21.15.67:0/1480712291 learned_addr learned my addr 172.21.15.67:0/1480712291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:33.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e66a00640 1 -- 172.21.15.67:0/1480712291 >> v2:172.21.15.187:3300/0 conn(0x7f9e680b46c0 msgr2=0x7f9e680ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:33.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.248+0000 7f9e67400640 1 --2- 172.21.15.67:0/1480712291 >> v2:172.21.15.187:3300/0 conn(0x7f9e680b46c0 0x7f9e680ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.249+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1480712291 >> v2:172.21.15.187:3300/0 conn(0x7f9e680b46c0 0x7f9e680ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.249+0000 7f9e66a00640 1 -- 172.21.15.67:0/1480712291 >> v1:172.21.15.187:6789/0 conn(0x7f9e680b4210 legacy=0x7f9e680af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:33.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.249+0000 7f9e66a00640 1 -- 172.21.15.67:0/1480712291 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6815fce0 con 0x7f9e6815c5a0 2024-04-17T08:31:33.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.250+0000 7f9e66a00640 1 --2- 172.21.15.67:0/1480712291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e6815c5a0 0x7f9e6815ee60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9e680b5b60 tx=0x7f9e5802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64346c274a80e74d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.250+0000 7f9e66000640 1 -- 172.21.15.67:0/1480712291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58038070 con 0x7f9e6815c5a0 2024-04-17T08:31:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.250+0000 7f9e66000640 1 -- 172.21.15.67:0/1480712291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e5802f050 con 0x7f9e6815c5a0 2024-04-17T08:31:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.250+0000 7f9e66000640 1 -- 172.21.15.67:0/1480712291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58031970 con 0x7f9e6815c5a0 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 -- 172.21.15.67:0/1480712291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e6815c5a0 msgr2=0x7f9e6815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1480712291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e6815c5a0 0x7f9e6815ee60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9e680b5b60 tx=0x7f9e5802b7f0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 -- 172.21.15.67:0/1480712291 shutdown_connections 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1480712291 >> v2:172.21.15.187:3300/0 conn(0x7f9e680b46c0 0x7f9e680ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 --2- 172.21.15.67:0/1480712291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e6815c5a0 0x7f9e6815ee60 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 -- 172.21.15.67:0/1480712291 >> 172.21.15.67:0/1480712291 conn(0x7f9e68067d10 msgr2=0x7f9e680b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 -- 172.21.15.67:0/1480712291 shutdown_connections 2024-04-17T08:31:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.251+0000 7f9e67e00640 1 -- 172.21.15.67:0/1480712291 wait complete. 2024-04-17T08:31:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.253+0000 7f9e67e00640 1 Processor -- start 2024-04-17T08:31:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.253+0000 7f9e67e00640 1 -- start start 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9e680b4210 0x7f9e68151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 0x7f9e680c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6806e7f0 con 0x7f9e680ca750 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6806e670 con 0x7f9e6815c5a0 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9e680afb50 con 0x7f9e680b4210 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53996/0 (socket says 172.21.15.67:53996) 2024-04-17T08:31:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67400640 1 -- 172.21.15.67:0/2900480810 learned_addr learned my addr 172.21.15.67:0/2900480810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e5fe00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 0x7f9e680c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e66a00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9e680b4210 0x7f9e68151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67400640 1 -- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9e680b4210 msgr2=0x7f9e68151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.255+0000 7f9e67400640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9e680b4210 0x7f9e68151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67400640 1 -- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 msgr2=0x7f9e680c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67400640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 0x7f9e680c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67400640 1 -- 172.21.15.67:0/2900480810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e680c8330 con 0x7f9e680ca750 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e5fe00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 0x7f9e680c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67400640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000cd60 tx=0x7f9e6000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60003b90 con 0x7f9e680ca750 2024-04-17T08:31:33.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e60003d50 con 0x7f9e680ca750 2024-04-17T08:31:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60004260 con 0x7f9e680ca750 2024-04-17T08:31:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e680c9600 con 0x7f9e680ca750 2024-04-17T08:31:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.256+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e680c9bc0 con 0x7f9e680ca750 2024-04-17T08:31:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.258+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e68068a10 con 0x7f9e680ca750 2024-04-17T08:31:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.259+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f9e60004400 con 0x7f9e680ca750 2024-04-17T08:31:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.260+0000 7f9e5f400640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 0x7f9e4406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.261+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 5737+0+0 (secure 0 0 0) 0x7f9e6008c8d0 con 0x7f9e680ca750 2024-04-17T08:31:33.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.261+0000 7f9e66a00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 0x7f9e4406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.262+0000 7f9e66a00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 0x7f9e4406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e58035000 tx=0x7f9e58008030 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.265+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9e6005aac0 con 0x7f9e680ca750 2024-04-17T08:31:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.471+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e6806fbc0 con 0x7f9e680ca750 2024-04-17T08:31:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.473+0000 7f9e5f400640 1 -- 172.21.15.67:0/2900480810 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) ==== 74+0+16680 (secure 0 0 0) 0x7f9e6005a3c0 con 0x7f9e680ca750 2024-04-17T08:31:33.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:33.472 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":194,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:32.003105+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 msgr2=0x7f9e4406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 0x7f9e4406eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e58035000 tx=0x7f9e58008030 comp rx=0 tx=0).stop 2024-04-17T08:31:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 msgr2=0x7f9e680cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000cd60 tx=0x7f9e6000a660 comp rx=0 tx=0).stop 2024-04-17T08:31:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 shutdown_connections 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e680ca750 0x7f9e680cabe0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9e4406ca10 0x7f9e4406eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e6815c5a0 0x7f9e680c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 --2- 172.21.15.67:0/2900480810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9e680b4210 0x7f9e68151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 >> 172.21.15.67:0/2900480810 conn(0x7f9e68067d10 msgr2=0x7f9e6815b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 shutdown_connections 2024-04-17T08:31:33.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.476+0000 7f9e67e00640 1 -- 172.21.15.67:0/2900480810 wait complete. 2024-04-17T08:31:33.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:31:33.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.727+0000 7f0ad3200640 1 Processor -- start 2024-04-17T08:31:33.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.727+0000 7f0ad3200640 1 -- start start 2024-04-17T08:31:33.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.728+0000 7f0ad3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ad40b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad409b9f0 con 0x7f0ad40ce320 2024-04-17T08:31:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad409bcd0 con 0x7f0ad40b58f0 2024-04-17T08:31:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad40cd410 con 0x7f0ad415fdf0 2024-04-17T08:31:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad2800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0ad40ce320 0x7f0ad4160570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55228/0 (socket says 172.21.15.67:55228) 2024-04-17T08:31:33.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad2800640 1 -- 172.21.15.67:0/3125212682 learned_addr learned my addr 172.21.15.67:0/3125212682 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:33.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.729+0000 7f0ad1e00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ad40b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917424053 0 0) 0x7f0ad409b9f0 con 0x7f0ad40ce320 2024-04-17T08:31:33.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ab8003520 con 0x7f0ad40ce320 2024-04-17T08:31:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582049721 0 0) 0x7f0ab8003520 con 0x7f0ad40ce320 2024-04-17T08:31:33.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 msgr2=0x7f0ad40b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ad40b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 >> v1:172.21.15.187:6789/0 conn(0x7f0ad40b58f0 legacy=0x7f0ad40cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad40cd6f0 con 0x7f0ad40ce320 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0acc0019f0 con 0x7f0ad40ce320 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 >> v1:172.21.15.67:6789/0 conn(0x7f0ad40ce320 legacy=0x7f0ad4160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ab80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ab8003f60 0x7f0ab8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ab80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad40cd410 con 0x7f0ab8004d40 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab800baa0 con 0x7f0ab8003f60 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab80100b0 con 0x7f0ad415fdf0 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.730+0000 7f0ad2800640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ab80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad1e00640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ab80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad1400640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ab8003f60 0x7f0ab8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad2800640 1 -- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 msgr2=0x7f0ab80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad2800640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad415fdf0 0x7f0ab80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad2800640 1 -- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ab8003f60 msgr2=0x7f0ab8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad2800640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ab8003f60 0x7f0ab8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.731+0000 7f0ad2800640 1 -- 172.21.15.67:0/3125212682 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab8003520 con 0x7f0ab8004d40 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.732+0000 7f0ad2800640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ab80051c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0ad406c3a0 tx=0x7f0ab8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de23fe20a17cfb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.732+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc00a4a0 con 0x7f0ab8004d40 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.732+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0acc034650 con 0x7f0ab8004d40 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.732+0000 7f0ad0a00640 1 -- 172.21.15.67:0/3125212682 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc034a80 con 0x7f0ab8004d40 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 -- 172.21.15.67:0/3125212682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 msgr2=0x7f0ab80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 --2- 172.21.15.67:0/3125212682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ab80051c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0ad406c3a0 tx=0x7f0ab8005830 comp rx=0 tx=0).stop 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad2800640 1 -- 172.21.15.67:0/3125212682 reap_dead start 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 -- 172.21.15.67:0/3125212682 shutdown_connections 2024-04-17T08:31:33.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 -- 172.21.15.67:0/3125212682 >> 172.21.15.67:0/3125212682 conn(0x7f0ad40b4050 msgr2=0x7f0ad40b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:33.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 -- 172.21.15.67:0/3125212682 shutdown_connections 2024-04-17T08:31:33.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.733+0000 7f0ad3200640 1 -- 172.21.15.67:0/3125212682 wait complete. 2024-04-17T08:31:33.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.735+0000 7f0ad3200640 1 Processor -- start 2024-04-17T08:31:33.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.735+0000 7f0ad3200640 1 -- start start 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad40c9b70 0x7f0ad40cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad40cbb60 0x7f0ad40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad4160350 con 0x7f0ab8004d40 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad41601d0 con 0x7f0ad40cbb60 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad40cd030 con 0x7f0ad40c9b70 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46670/0 (socket says 172.21.15.67:46670) 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.736+0000 7f0ad1e00640 1 -- 172.21.15.67:0/523552118 learned_addr learned my addr 172.21.15.67:0/523552118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 -- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad40c9b70 msgr2=0x7f0ad40cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad40c9b70 0x7f0ad40cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 -- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad40cbb60 msgr2=0x7f0ad40cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:33.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad2800640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad40cbb60 0x7f0ad40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad40cbb60 0x7f0ad40cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:33.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 -- 172.21.15.67:0/523552118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad40cc520 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0ad1e00640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4011600 tx=0x7f0ac4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4015870 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac4015a10 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.737+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4015dc0 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.738+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad41589d0 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.738+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad40c6800 con 0x7f0ab8004d40 2024-04-17T08:31:33.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.739+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad40755d0 con 0x7f0ab8004d40 2024-04-17T08:31:33.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.740+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f0ac400f450 con 0x7f0ab8004d40 2024-04-17T08:31:33.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.741+0000 7f0aca000640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 0x7f0ab006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:33.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.741+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f0ac408c220 con 0x7f0ab8004d40 2024-04-17T08:31:33.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.741+0000 7f0ad1400640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 0x7f0ab006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:33.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.743+0000 7f0ad1400640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 0x7f0ab006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0abc007fd0 tx=0x7f0abc009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:33.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.745+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0ac4019030 con 0x7f0ab8004d40 2024-04-17T08:31:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:33.997+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0ad415f610 con 0x7f0ab8004d40 2024-04-17T08:31:34.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.587+0000 7f0aca000640 1 -- 172.21.15.67:0/523552118 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) ==== 118+0+0 (secure 0 0 0) 0x7f0ac405a520 con 0x7f0ab8004d40 2024-04-17T08:31:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.590+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 msgr2=0x7f0ab006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.590+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 0x7f0ab006eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0abc007fd0 tx=0x7f0abc009040 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.590+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 msgr2=0x7f0ad40c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.590+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4011600 tx=0x7f0ac4011ac0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.590+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 shutdown_connections 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f0ab006ca10 0x7f0ab006eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad40cbb60 0x7f0ad40cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0ad40c9b70 0x7f0ad40cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 --2- 172.21.15.67:0/523552118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ab8004d40 0x7f0ad40c9630 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 >> 172.21.15.67:0/523552118 conn(0x7f0ad40b4050 msgr2=0x7f0ad406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 shutdown_connections 2024-04-17T08:31:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.591+0000 7f0ad3200640 1 -- 172.21.15.67:0/523552118 wait complete. 2024-04-17T08:31:34.610 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:31:34.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:31:34.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.840+0000 7f9f9fe00640 1 Processor -- start 2024-04-17T08:31:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.840+0000 7f9f9fe00640 1 -- start start 2024-04-17T08:31:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa0145f80 0x7f9fa00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9fa00cd750 0x7f9fa006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa006f1d0 con 0x7f9fa006c0a0 2024-04-17T08:31:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa00cdbb0 con 0x7f9fa00cd750 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa00727a0 con 0x7f9fa0145f80 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9fa00cd750 0x7f9fa006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa0145f80 0x7f9fa00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9fa00cd750 0x7f9fa006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:41904/0 (socket says 172.21.15.67:41904) 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.842+0000 7f9f9e000640 1 -- 172.21.15.67:0/1313862989 learned_addr learned my addr 172.21.15.67:0/1313862989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.843+0000 7f9f9e000640 1 -- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa0145f80 msgr2=0x7f9fa00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.843+0000 7f9f9e000640 1 --2- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa0145f80 0x7f9fa00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.843+0000 7f9f9e000640 1 -- 172.21.15.67:0/1313862989 >> v1:172.21.15.187:6789/0 conn(0x7f9fa006c0a0 legacy=0x7f9fa0160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.843+0000 7f9f9e000640 1 -- 172.21.15.67:0/1313862989 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa0146360 con 0x7f9fa00cd750 2024-04-17T08:31:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.845+0000 7f9f9e000640 1 --2- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa006bb60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009870 tx=0x7f9f8c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e7f9f7f9f850920 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.845+0000 7f9f9d600640 1 -- 172.21.15.67:0/1313862989 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c035020 con 0x7f9fa00cd750 2024-04-17T08:31:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.845+0000 7f9f9d600640 1 -- 172.21.15.67:0/1313862989 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c031640 con 0x7f9fa00cd750 2024-04-17T08:31:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.845+0000 7f9f9d600640 1 -- 172.21.15.67:0/1313862989 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c02d040 con 0x7f9fa00cd750 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 msgr2=0x7f9fa006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa006bb60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009870 tx=0x7f9f8c02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1313862989 shutdown_connections 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa006bb60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1313862989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa0145f80 0x7f9fa00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1313862989 >> 172.21.15.67:0/1313862989 conn(0x7f9fa00b2190 msgr2=0x7f9fa00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1313862989 shutdown_connections 2024-04-17T08:31:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.846+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1313862989 wait complete. 2024-04-17T08:31:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.847+0000 7f9f9fe00640 1 Processor -- start 2024-04-17T08:31:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.848+0000 7f9f9fe00640 1 -- start start 2024-04-17T08:31:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.848+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa006c0a0 0x7f9fa00b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.848+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa00b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0072430 con 0x7f9fa0145f80 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa0072150 con 0x7f9fa00cd750 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa0072c30 con 0x7f9fa006c0a0 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46682/0 (socket says 172.21.15.67:46682) 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 -- 172.21.15.67:0/1218527701 learned_addr learned my addr 172.21.15.67:0/1218527701 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9e000640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa00b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9ea00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa006c0a0 0x7f9fa00b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 -- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa006c0a0 msgr2=0x7f9fa00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa006c0a0 0x7f9fa00b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 -- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 msgr2=0x7f9fa00b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa00b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.849+0000 7f9f9f400640 1 -- 172.21.15.67:0/1218527701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa01f2cd0 con 0x7f9fa0145f80 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f9f400640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800c7a0 tx=0x7f9f9800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98003b70 con 0x7f9fa0145f80 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f98003d30 con 0x7f9fa0145f80 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9800aa90 con 0x7f9fa0145f80 2024-04-17T08:31:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa01f2fb0 con 0x7f9fa0145f80 2024-04-17T08:31:34.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.850+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa01f3570 con 0x7f9fa0145f80 2024-04-17T08:31:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.851+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa006c4d0 con 0x7f9fa0145f80 2024-04-17T08:31:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.852+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f9f9800ac30 con 0x7f9fa0145f80 2024-04-17T08:31:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.853+0000 7f9f96a00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 0x7f9f8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.853+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f9f9808bcb0 con 0x7f9fa0145f80 2024-04-17T08:31:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.854+0000 7f9f9ea00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 0x7f9f8006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.855+0000 7f9f9ea00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 0x7f9f8006eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009a30 tx=0x7f9f90008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:34.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:34.858+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9fa006c4d0 con 0x7f9fa0145f80 2024-04-17T08:31:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.100+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9fa0068a10 con 0x7f9fa0145f80 2024-04-17T08:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.590+0000 7f9f96a00640 1 -- 172.21.15.67:0/1218527701 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) ==== 122+0+0 (secure 0 0 0) 0x7f9fa006f1d0 con 0x7f9fa0145f80 2024-04-17T08:31:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.593+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 msgr2=0x7f9f8006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.593+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 0x7f9f8006eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009a30 tx=0x7f9f90008040 comp rx=0 tx=0).stop 2024-04-17T08:31:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.593+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 msgr2=0x7f9fa0134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.593+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800c7a0 tx=0x7f9f9800cc60 comp rx=0 tx=0).stop 2024-04-17T08:31:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 shutdown_connections 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa0145f80 0x7f9fa0134480 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f9f8006ca10 0x7f9f8006eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fa00cd750 0x7f9fa00b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 --2- 172.21.15.67:0/1218527701 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fa006c0a0 0x7f9fa00b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 >> 172.21.15.67:0/1218527701 conn(0x7f9fa00b2190 msgr2=0x7f9fa015ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 shutdown_connections 2024-04-17T08:31:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.594+0000 7f9f9fe00640 1 -- 172.21.15.67:0/1218527701 wait complete. 2024-04-17T08:31:35.619 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:31:35.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.850+0000 7f292d600640 1 Processor -- start 2024-04-17T08:31:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.850+0000 7f292d600640 1 -- start start 2024-04-17T08:31:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.851+0000 7f292d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29280ce0a0 0x7f29280d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.851+0000 7f292d600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292809b8a0 con 0x7f29280cdc80 2024-04-17T08:31:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.851+0000 7f292d600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292809bb80 con 0x7f292815fe10 2024-04-17T08:31:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.851+0000 7f292d600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f29280cd7f0 con 0x7f29280ce0a0 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f2927400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f29280cdc80 0x7f29280d0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55242/0 (socket says 172.21.15.67:55242) 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f2927400640 1 -- 172.21.15.67:0/1508831871 learned_addr learned my addr 172.21.15.67:0/1508831871 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f292cc00640 1 --2- 172.21.15.67:0/1508831871 >> v2:172.21.15.67:3300/0 conn(0x7f29280ce0a0 0x7f29280d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f2926a00640 1 -- 172.21.15.67:0/1508831871 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371847183 0 0) 0x7f292809b8a0 con 0x7f29280cdc80 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f2926a00640 1 -- 172.21.15.67:0/1508831871 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f290c003520 con 0x7f29280cdc80 2024-04-17T08:31:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f292cc00640 1 -- 172.21.15.67:0/1508831871 >> v1:172.21.15.187:6789/0 conn(0x7f292815fe10 legacy=0x7f29280c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f292cc00640 1 -- 172.21.15.67:0/1508831871 >> v1:172.21.15.67:6789/0 conn(0x7f29280cdc80 legacy=0x7f29280d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.852+0000 7f292cc00640 1 -- 172.21.15.67:0/1508831871 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29280cd480 con 0x7f29280ce0a0 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.854+0000 7f292cc00640 1 --2- 172.21.15.67:0/1508831871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280ce0a0 0x7f29280d0e10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2920009d90 tx=0x7f292002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfec8ee2c7a9e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.854+0000 7f2926a00640 1 -- 172.21.15.67:0/1508831871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920039070 con 0x7f29280ce0a0 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.854+0000 7f2926a00640 1 -- 172.21.15.67:0/1508831871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f292002cc60 con 0x7f29280ce0a0 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.854+0000 7f2926a00640 1 -- 172.21.15.67:0/1508831871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920034640 con 0x7f29280ce0a0 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 -- 172.21.15.67:0/1508831871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280ce0a0 msgr2=0x7f29280d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 --2- 172.21.15.67:0/1508831871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280ce0a0 0x7f29280d0e10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f2920009d90 tx=0x7f292002beb0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 -- 172.21.15.67:0/1508831871 shutdown_connections 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 --2- 172.21.15.67:0/1508831871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280ce0a0 0x7f29280d0e10 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 -- 172.21.15.67:0/1508831871 >> 172.21.15.67:0/1508831871 conn(0x7f29280b4170 msgr2=0x7f29280b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 -- 172.21.15.67:0/1508831871 shutdown_connections 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.855+0000 7f292d600640 1 -- 172.21.15.67:0/1508831871 wait complete. 2024-04-17T08:31:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.857+0000 7f292d600640 1 Processor -- start 2024-04-17T08:31:35.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.857+0000 7f292d600640 1 -- start start 2024-04-17T08:31:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29280ce0a0 0x7f29280b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 0x7f29280b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29280c68e0 con 0x7f29280cdc80 2024-04-17T08:31:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f29280cd170 con 0x7f292815fe10 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f292d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29280ca440 con 0x7f29280ce0a0 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f2927e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.858+0000 7f2927e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46700/0 (socket says 172.21.15.67:46700) 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 -- 172.21.15.67:0/530564191 learned_addr learned my addr 172.21.15.67:0/530564191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927400640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29280ce0a0 0x7f29280b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f292cc00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 0x7f29280b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 -- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29280ce0a0 msgr2=0x7f29280b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29280ce0a0 0x7f29280b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 -- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 msgr2=0x7f29280b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 0x7f29280b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 -- 172.21.15.67:0/530564191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29280ca720 con 0x7f29280cdc80 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f292cc00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 0x7f29280b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.859+0000 7f2927e00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f291800b3f0 tx=0x7f291800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.860+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918015020 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.860+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2918011830 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.860+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918011c20 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.860+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29280caa00 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.860+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29280cafc0 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.861+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29280755d0 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.862+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f2918018020 con 0x7f29280cdc80 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.863+0000 7f291fe00640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 0x7f290806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:35.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.863+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f291808b970 con 0x7f29280cdc80 2024-04-17T08:31:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.864+0000 7f2927400640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 0x7f290806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.865+0000 7f2927400640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 0x7f290806ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29280c5b80 tx=0x7f2914009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:35.868+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f291801f030 con 0x7f29280cdc80 2024-04-17T08:31:36.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.111+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29280b4ed0 con 0x7f29280cdc80 2024-04-17T08:31:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.112+0000 7f291fe00640 1 -- 172.21.15.67:0/530564191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7f29180172c0 con 0x7f29280cdc80 2024-04-17T08:31:36.110 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:36.110 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:31:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.114+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 msgr2=0x7f290806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.114+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 0x7f290806ef20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29280c5b80 tx=0x7f2914009040 comp rx=0 tx=0).stop 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 msgr2=0x7f29281590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f291800b3f0 tx=0x7f291800b8b0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 shutdown_connections 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f290806ca60 0x7f290806ef20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f292815fe10 0x7f29280b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29280ce0a0 0x7f29280b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 --2- 172.21.15.67:0/530564191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29280cdc80 0x7f29281590d0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 >> 172.21.15.67:0/530564191 conn(0x7f29280b4170 msgr2=0x7f292806e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 shutdown_connections 2024-04-17T08:31:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.115+0000 7f292d600640 1 -- 172.21.15.67:0/530564191 wait complete. 2024-04-17T08:31:36.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:31:36.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.358+0000 7ffbdb000640 1 Processor -- start 2024-04-17T08:31:36.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.358+0000 7ffbdb000640 1 -- start start 2024-04-17T08:31:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbdb000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffbd40afb50 0x7ffbd4169490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbdb000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd415f440 con 0x7ffbd415c5a0 2024-04-17T08:31:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbdb000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd415f720 con 0x7ffbd415ee60 2024-04-17T08:31:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbdb000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd40b4920 con 0x7ffbd40afb50 2024-04-17T08:31:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbd9c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffbd415c5a0 0x7ffbd40b4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55244/0 (socket says 172.21.15.67:55244) 2024-04-17T08:31:36.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.360+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4093853114 learned_addr learned my addr 172.21.15.67:0/4093853114 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbda600640 1 --2- 172.21.15.67:0/4093853114 >> v2:172.21.15.187:3300/0 conn(0x7ffbd40afb50 0x7ffbd4169490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097453881 0 0) 0x7ffbd415f440 con 0x7ffbd415c5a0 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbbc003520 con 0x7ffbd415c5a0 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 117329466 0 0) 0x7ffbbc003520 con 0x7ffbd415c5a0 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 >> v2:172.21.15.187:3300/0 conn(0x7ffbd40afb50 msgr2=0x7ffbd4169490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 --2- 172.21.15.67:0/4093853114 >> v2:172.21.15.187:3300/0 conn(0x7ffbd40afb50 0x7ffbd4169490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 >> v1:172.21.15.187:6789/0 conn(0x7ffbd415ee60 legacy=0x7ffbd40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd415fa00 con 0x7ffbd415c5a0 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ffbc8001670 con 0x7ffbd415c5a0 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 >> v1:172.21.15.67:6789/0 conn(0x7ffbd415c5a0 legacy=0x7ffbd40b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.361+0000 7ffbd3e00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbbc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd3e00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbbc004800 0x7ffbbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd3e00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbbc0052c0 0x7ffbbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc00c1d0 con 0x7ffbbc003a20 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc00c380 con 0x7ffbbc004800 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffbbc00c580 con 0x7ffbbc0052c0 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbbc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbd9200640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbbc004800 0x7ffbbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.362+0000 7ffbda600640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbbc0052c0 0x7ffbbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbbc0052c0 msgr2=0x7ffbbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbbc0052c0 0x7ffbbc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbbc004800 msgr2=0x7ffbbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbbc004800 0x7ffbbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4093853114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbbc003520 con 0x7ffbbc003a20 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbda600640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbbc0052c0 0x7ffbbc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbd9200640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbbc004800 0x7ffbbc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.363+0000 7ffbda600640 1 -- 172.21.15.67:0/4093853114 reap_dead start 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.364+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbbc003e40 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c240 tx=0x7ffbbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bfff8d3746b13e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.364+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8007d10 con 0x7ffbbc003a20 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.364+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc8030070 con 0x7ffbbc003a20 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.364+0000 7ffbd3e00640 1 -- 172.21.15.67:0/4093853114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8005d70 con 0x7ffbbc003a20 2024-04-17T08:31:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 -- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 msgr2=0x7ffbbc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbbc003e40 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c240 tx=0x7ffbbc005830 comp rx=0 tx=0).stop 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 -- 172.21.15.67:0/4093853114 shutdown_connections 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4093853114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbbc003e40 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 -- 172.21.15.67:0/4093853114 >> 172.21.15.67:0/4093853114 conn(0x7ffbd4067d10 msgr2=0x7ffbd40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 -- 172.21.15.67:0/4093853114 shutdown_connections 2024-04-17T08:31:36.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.365+0000 7ffbdb000640 1 -- 172.21.15.67:0/4093853114 wait complete. 2024-04-17T08:31:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.367+0000 7ffbdb000640 1 Processor -- start 2024-04-17T08:31:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.367+0000 7ffbdb000640 1 -- start start 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbd41460a0 0x7ffbd406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbd406f270 0x7ffbd415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd40adc30 con 0x7ffbbc003a20 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd40adab0 con 0x7ffbd41460a0 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbdb000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd406e670 con 0x7ffbd406f270 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbd9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbd9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46712/0 (socket says 172.21.15.67:46712) 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.368+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4152960305 learned_addr learned my addr 172.21.15.67:0/4152960305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbda600640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbd406f270 0x7ffbd415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9200640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbd41460a0 0x7ffbd406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbd406f270 msgr2=0x7ffbd415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbd406f270 0x7ffbd415d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbd41460a0 msgr2=0x7ffbd406ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbd41460a0 0x7ffbd406ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 -- 172.21.15.67:0/4152960305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd41520d0 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.369+0000 7ffbd9c00640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c360 tx=0x7ffbc802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.370+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8045070 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.370+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc8004bf0 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.370+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd41523b0 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.370+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8007d10 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.370+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd4152970 con 0x7ffbbc003a20 2024-04-17T08:31:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.371+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd4068a10 con 0x7ffbbc003a20 2024-04-17T08:31:36.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.372+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7ffbc8007070 con 0x7ffbbc003a20 2024-04-17T08:31:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.373+0000 7ffbd1600640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 0x7ffbb406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.373+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7ffbc80aca20 con 0x7ffbbc003a20 2024-04-17T08:31:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.374+0000 7ffbd9200640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 0x7ffbb406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.375+0000 7ffbd9200640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 0x7ffbb406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0007fd0 tx=0x7ffbc0009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.378+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ffbc807a6e0 con 0x7ffbbc003a20 2024-04-17T08:31:36.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.576+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbd406ff50 con 0x7ffbbc003a20 2024-04-17T08:31:36.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.579+0000 7ffbd1600640 1 -- 172.21.15.67:0/4152960305 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) ==== 74+0+16667 (secure 0 0 0) 0x7ffbc807e9e0 con 0x7ffbbc003a20 2024-04-17T08:31:36.577 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:36.577 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":195,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:31:33.565243+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:31:36.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 msgr2=0x7ffbb406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 0x7ffbb406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0007fd0 tx=0x7ffbc0009040 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 msgr2=0x7ffbd4142ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c360 tx=0x7ffbc802e070 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 shutdown_connections 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffbd406f270 0x7ffbd415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ffbb406ca40 0x7ffbb406ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffbd41460a0 0x7ffbd406ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 --2- 172.21.15.67:0/4152960305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbbc003a20 0x7ffbd4142ea0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 >> 172.21.15.67:0/4152960305 conn(0x7ffbd4067d10 msgr2=0x7ffbd40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.581+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 shutdown_connections 2024-04-17T08:31:36.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.582+0000 7ffbdb000640 1 -- 172.21.15.67:0/4152960305 wait complete. 2024-04-17T08:31:36.599 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:31:36.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:31:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.815+0000 7faab3200640 1 Processor -- start 2024-04-17T08:31:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.815+0000 7faab3200640 1 -- start start 2024-04-17T08:31:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.816+0000 7faab3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faab415fe10 0x7faab40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.816+0000 7faab3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faab40c6710 0x7faab40cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.816+0000 7faab3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab409b8a0 con 0x7faab40ce1c0 2024-04-17T08:31:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.816+0000 7faab3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faab40cd7f0 con 0x7faab40c6710 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.816+0000 7faab3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faab409bb80 con 0x7faab415fe10 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faab40c6710 0x7faab40cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faab40c6710 0x7faab40cdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46720/0 (socket says 172.21.15.67:46720) 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 -- 172.21.15.67:0/2765231896 learned_addr learned my addr 172.21.15.67:0/2765231896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1e00640 1 --2- 172.21.15.67:0/2765231896 >> v2:172.21.15.187:3300/0 conn(0x7faab415fe10 0x7faab40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 -- 172.21.15.67:0/2765231896 >> v2:172.21.15.187:3300/0 conn(0x7faab415fe10 msgr2=0x7faab40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 --2- 172.21.15.67:0/2765231896 >> v2:172.21.15.187:3300/0 conn(0x7faab415fe10 0x7faab40c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 -- 172.21.15.67:0/2765231896 >> v1:172.21.15.187:6789/0 conn(0x7faab40ce1c0 legacy=0x7faab4160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.817+0000 7faab1400640 1 -- 172.21.15.67:0/2765231896 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab40cd480 con 0x7faab40c6710 2024-04-17T08:31:36.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.818+0000 7faab1400640 1 --2- 172.21.15.67:0/2765231896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40c6710 0x7faab40cdc80 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7faab40d01c0 tx=0x7faa9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe458aa81612252a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.819+0000 7faab0a00640 1 -- 172.21.15.67:0/2765231896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c02ca80 con 0x7faab40c6710 2024-04-17T08:31:36.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.819+0000 7faab0a00640 1 -- 172.21.15.67:0/2765231896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c02cc20 con 0x7faab40c6710 2024-04-17T08:31:36.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.819+0000 7faab0a00640 1 -- 172.21.15.67:0/2765231896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c0343f0 con 0x7faab40c6710 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.819+0000 7faab3200640 1 -- 172.21.15.67:0/2765231896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40c6710 msgr2=0x7faab40cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.819+0000 7faab3200640 1 --2- 172.21.15.67:0/2765231896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40c6710 0x7faab40cdc80 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7faab40d01c0 tx=0x7faa9c02bee0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 -- 172.21.15.67:0/2765231896 shutdown_connections 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 --2- 172.21.15.67:0/2765231896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab40c6710 0x7faab40cdc80 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 --2- 172.21.15.67:0/2765231896 >> v2:172.21.15.187:3300/0 conn(0x7faab415fe10 0x7faab40c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 -- 172.21.15.67:0/2765231896 >> 172.21.15.67:0/2765231896 conn(0x7faab40b4170 msgr2=0x7faab40b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:36.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 -- 172.21.15.67:0/2765231896 shutdown_connections 2024-04-17T08:31:36.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.820+0000 7faab3200640 1 -- 172.21.15.67:0/2765231896 wait complete. 2024-04-17T08:31:36.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.822+0000 7faab3200640 1 Processor -- start 2024-04-17T08:31:36.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.822+0000 7faab3200640 1 -- start start 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faab40c6710 0x7faab4152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faab40ce1c0 0x7faab40ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faab4160370 con 0x7faab415fe10 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faab41601f0 con 0x7faab40c6710 2024-04-17T08:31:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faab409b470 con 0x7faab40ce1c0 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46732/0 (socket says 172.21.15.67:46732) 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.823+0000 7faab2800640 1 -- 172.21.15.67:0/3026672203 learned_addr learned my addr 172.21.15.67:0/3026672203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 -- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faab40ce1c0 msgr2=0x7faab40ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab1e00640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faab40c6710 0x7faab4152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faab40ce1c0 0x7faab40ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 -- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faab40c6710 msgr2=0x7faab4152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faab40c6710 0x7faab4152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 -- 172.21.15.67:0/3026672203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab40b81b0 con 0x7faab415fe10 2024-04-17T08:31:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faab2800640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c410 tx=0x7faaac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.824+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac011790 con 0x7faab415fe10 2024-04-17T08:31:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.825+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaac017070 con 0x7faab415fe10 2024-04-17T08:31:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.825+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac011b70 con 0x7faab415fe10 2024-04-17T08:31:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.825+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab40b8490 con 0x7faab415fe10 2024-04-17T08:31:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.825+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab40b5930 con 0x7faab415fe10 2024-04-17T08:31:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.826+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab40755d0 con 0x7faab415fe10 2024-04-17T08:31:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.827+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7faaac011d10 con 0x7faab415fe10 2024-04-17T08:31:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.828+0000 7faaaa000640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 0x7faa9006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.828+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7faaac08b8a0 con 0x7faab415fe10 2024-04-17T08:31:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.828+0000 7faab1e00640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 0x7faa9006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.830+0000 7faab1e00640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 0x7faa9006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:36.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:36.833+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faaac021020 con 0x7faab415fe10 2024-04-17T08:31:37.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.028+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 --> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faab40b4ed0 con 0x7faa9006ca10 2024-04-17T08:31:37.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.051+0000 7faaaa000640 1 -- 172.21.15.67:0/3026672203 <== mgr.7807 v2:172.21.15.187:6832/3300567104 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252496 (secure 0 0 0) 0x7faab40b4ed0 con 0x7faa9006ca10 2024-04-17T08:31:37.053 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:37.056 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 msgr2=0x7faa9006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 0x7faa9006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4008040 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 msgr2=0x7faab40cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c410 tx=0x7faaac00c8d0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 shutdown_connections 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab415fe10 0x7faab40cace0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faab40ce1c0 0x7faab40ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7faa9006ca10 0x7faa9006eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 --2- 172.21.15.67:0/3026672203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faab40c6710 0x7faab4152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 >> 172.21.15.67:0/3026672203 conn(0x7faab40b4170 msgr2=0x7faab40b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 shutdown_connections 2024-04-17T08:31:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.058+0000 7faab3200640 1 -- 172.21.15.67:0/3026672203 wait complete. 2024-04-17T08:31:37.077 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-17T08:31:35.858686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":532,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":338,"ondisk_log_size":338,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419584,"kb_used_data":20232,"kb_used_omap":12,"kb_used_meta":399219,"kb_avail":754555136,"statfs":{"total":773094113280,"available":772664459264,"internally_reserved":0,"allocated":20717568,"data_stored":10645364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":408800816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.349708"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579935+0000","last_change":"2024-04-17T08:31:32.569758+0000","last_active":"2024-04-17T08:31:33.579935+0000","last_peered":"2024-04-17T08:31:33.579935+0000","last_clean":"2024-04-17T08:31:33.579935+0000","last_became_active":"2024-04-17T08:31:32.569415+0000","last_became_peered":"2024-04-17T08:31:32.569415+0000","last_unstale":"2024-04-17T08:31:33.579935+0000","last_undegraded":"2024-04-17T08:31:33.579935+0000","last_fullsized":"2024-04-17T08:31:33.579935+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:50.692018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721910+0000","last_change":"2024-04-17T08:31:32.581312+0000","last_active":"2024-04-17T08:31:33.721910+0000","last_peered":"2024-04-17T08:31:33.721910+0000","last_clean":"2024-04-17T08:31:33.721910+0000","last_became_active":"2024-04-17T08:31:32.581012+0000","last_became_peered":"2024-04-17T08:31:32.581012+0000","last_unstale":"2024-04-17T08:31:33.721910+0000","last_undegraded":"2024-04-17T08:31:33.721910+0000","last_fullsized":"2024-04-17T08:31:33.721910+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:51.478954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582046+0000","last_change":"2024-04-17T08:31:32.573303+0000","last_active":"2024-04-17T08:31:33.582046+0000","last_peered":"2024-04-17T08:31:33.582046+0000","last_clean":"2024-04-17T08:31:33.582046+0000","last_became_active":"2024-04-17T08:31:32.572644+0000","last_became_peered":"2024-04-17T08:31:32.572644+0000","last_unstale":"2024-04-17T08:31:33.582046+0000","last_undegraded":"2024-04-17T08:31:33.582046+0000","last_fullsized":"2024-04-17T08:31:33.582046+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:23.240248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579908+0000","last_change":"2024-04-17T08:31:32.582020+0000","last_active":"2024-04-17T08:31:33.579908+0000","last_peered":"2024-04-17T08:31:33.579908+0000","last_clean":"2024-04-17T08:31:33.579908+0000","last_became_active":"2024-04-17T08:31:32.581337+0000","last_became_peered":"2024-04-17T08:31:32.581337+0000","last_unstale":"2024-04-17T08:31:33.579908+0000","last_undegraded":"2024-04-17T08:31:33.579908+0000","last_fullsized":"2024-04-17T08:31:33.579908+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:26.212613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721734+0000","last_change":"2024-04-17T08:31:32.582417+0000","last_active":"2024-04-17T08:31:33.721734+0000","last_peered":"2024-04-17T08:31:33.721734+0000","last_clean":"2024-04-17T08:31:33.721734+0000","last_became_active":"2024-04-17T08:31:32.582025+0000","last_became_peered":"2024-04-17T08:31:32.582025+0000","last_unstale":"2024-04-17T08:31:33.721734+0000","last_undegraded":"2024-04-17T08:31:33.721734+0000","last_fullsized":"2024-04-17T08:31:33.721734+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:17.417333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722474+0000","last_change":"2024-04-17T08:31:32.578286+0000","last_active":"2024-04-17T08:31:33.722474+0000","last_peered":"2024-04-17T08:31:33.722474+0000","last_clean":"2024-04-17T08:31:33.722474+0000","last_became_active":"2024-04-17T08:31:32.577947+0000","last_became_peered":"2024-04-17T08:31:32.577947+0000","last_unstale":"2024-04-17T08:31:33.722474+0000","last_undegraded":"2024-04-17T08:31:33.722474+0000","last_fullsized":"2024-04-17T08:31:33.722474+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:38.450228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720125+0000","last_change":"2024-04-17T08:31:32.590058+0000","last_active":"2024-04-17T08:31:33.720125+0000","last_peered":"2024-04-17T08:31:33.720125+0000","last_clean":"2024-04-17T08:31:33.720125+0000","last_became_active":"2024-04-17T08:31:32.589724+0000","last_became_peered":"2024-04-17T08:31:32.589724+0000","last_unstale":"2024-04-17T08:31:33.720125+0000","last_undegraded":"2024-04-17T08:31:33.720125+0000","last_fullsized":"2024-04-17T08:31:33.720125+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:21:43.178564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721444+0000","last_change":"2024-04-17T08:31:32.581939+0000","last_active":"2024-04-17T08:31:33.721444+0000","last_peered":"2024-04-17T08:31:33.721444+0000","last_clean":"2024-04-17T08:31:33.721444+0000","last_became_active":"2024-04-17T08:31:32.580851+0000","last_became_peered":"2024-04-17T08:31:32.580851+0000","last_unstale":"2024-04-17T08:31:33.721444+0000","last_undegraded":"2024-04-17T08:31:33.721444+0000","last_fullsized":"2024-04-17T08:31:33.721444+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:15.954008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722557+0000","last_change":"2024-04-17T08:31:32.582031+0000","last_active":"2024-04-17T08:31:33.722557+0000","last_peered":"2024-04-17T08:31:33.722557+0000","last_clean":"2024-04-17T08:31:33.722557+0000","last_became_active":"2024-04-17T08:31:32.581730+0000","last_became_peered":"2024-04-17T08:31:32.581730+0000","last_unstale":"2024-04-17T08:31:33.722557+0000","last_undegraded":"2024-04-17T08:31:33.722557+0000","last_fullsized":"2024-04-17T08:31:33.722557+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:18.372976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722514+0000","last_change":"2024-04-17T08:31:32.585759+0000","last_active":"2024-04-17T08:31:33.722514+0000","last_peered":"2024-04-17T08:31:33.722514+0000","last_clean":"2024-04-17T08:31:33.722514+0000","last_became_active":"2024-04-17T08:31:32.585441+0000","last_became_peered":"2024-04-17T08:31:32.585441+0000","last_unstale":"2024-04-17T08:31:33.722514+0000","last_undegraded":"2024-04-17T08:31:33.722514+0000","last_fullsized":"2024-04-17T08:31:33.722514+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:40:31.918038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722417+0000","last_change":"2024-04-17T08:31:32.589047+0000","last_active":"2024-04-17T08:31:33.722417+0000","last_peered":"2024-04-17T08:31:33.722417+0000","last_clean":"2024-04-17T08:31:33.722417+0000","last_became_active":"2024-04-17T08:31:32.588738+0000","last_became_peered":"2024-04-17T08:31:32.588738+0000","last_unstale":"2024-04-17T08:31:33.722417+0000","last_undegraded":"2024-04-17T08:31:33.722417+0000","last_fullsized":"2024-04-17T08:31:33.722417+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:57.297905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722000+0000","last_change":"2024-04-17T08:31:32.582010+0000","last_active":"2024-04-17T08:31:33.722000+0000","last_peered":"2024-04-17T08:31:33.722000+0000","last_clean":"2024-04-17T08:31:33.722000+0000","last_became_active":"2024-04-17T08:31:32.581632+0000","last_became_peered":"2024-04-17T08:31:32.581632+0000","last_unstale":"2024-04-17T08:31:33.722000+0000","last_undegraded":"2024-04-17T08:31:33.722000+0000","last_fullsized":"2024-04-17T08:31:33.722000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:31.192021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581323+0000","last_change":"2024-04-17T08:31:32.578019+0000","last_active":"2024-04-17T08:31:33.581323+0000","last_peered":"2024-04-17T08:31:33.581323+0000","last_clean":"2024-04-17T08:31:33.581323+0000","last_became_active":"2024-04-17T08:31:32.577733+0000","last_became_peered":"2024-04-17T08:31:32.577733+0000","last_unstale":"2024-04-17T08:31:33.581323+0000","last_undegraded":"2024-04-17T08:31:33.581323+0000","last_fullsized":"2024-04-17T08:31:33.581323+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:54.699101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580783+0000","last_change":"2024-04-17T08:31:32.576812+0000","last_active":"2024-04-17T08:31:33.580783+0000","last_peered":"2024-04-17T08:31:33.580783+0000","last_clean":"2024-04-17T08:31:33.580783+0000","last_became_active":"2024-04-17T08:31:32.576535+0000","last_became_peered":"2024-04-17T08:31:32.576535+0000","last_unstale":"2024-04-17T08:31:33.580783+0000","last_undegraded":"2024-04-17T08:31:33.580783+0000","last_fullsized":"2024-04-17T08:31:33.580783+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:34.672847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579277+0000","last_change":"2024-04-17T08:31:32.585287+0000","last_active":"2024-04-17T08:31:33.579277+0000","last_peered":"2024-04-17T08:31:33.579277+0000","last_clean":"2024-04-17T08:31:33.579277+0000","last_became_active":"2024-04-17T08:31:32.584983+0000","last_became_peered":"2024-04-17T08:31:32.584983+0000","last_unstale":"2024-04-17T08:31:33.579277+0000","last_undegraded":"2024-04-17T08:31:33.579277+0000","last_fullsized":"2024-04-17T08:31:33.579277+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:05.681152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719087+0000","last_change":"2024-04-17T08:31:32.593519+0000","last_active":"2024-04-17T08:31:33.719087+0000","last_peered":"2024-04-17T08:31:33.719087+0000","last_clean":"2024-04-17T08:31:33.719087+0000","last_became_active":"2024-04-17T08:31:32.593198+0000","last_became_peered":"2024-04-17T08:31:32.593198+0000","last_unstale":"2024-04-17T08:31:33.719087+0000","last_undegraded":"2024-04-17T08:31:33.719087+0000","last_fullsized":"2024-04-17T08:31:33.719087+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:00.330621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722625+0000","last_change":"2024-04-17T08:31:32.585581+0000","last_active":"2024-04-17T08:31:33.722625+0000","last_peered":"2024-04-17T08:31:33.722625+0000","last_clean":"2024-04-17T08:31:33.722625+0000","last_became_active":"2024-04-17T08:31:32.585302+0000","last_became_peered":"2024-04-17T08:31:32.585302+0000","last_unstale":"2024-04-17T08:31:33.722625+0000","last_undegraded":"2024-04-17T08:31:33.722625+0000","last_fullsized":"2024-04-17T08:31:33.722625+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:54.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720180+0000","last_change":"2024-04-17T08:31:32.579030+0000","last_active":"2024-04-17T08:31:33.720180+0000","last_peered":"2024-04-17T08:31:33.720180+0000","last_clean":"2024-04-17T08:31:33.720180+0000","last_became_active":"2024-04-17T08:31:32.578659+0000","last_became_peered":"2024-04-17T08:31:32.578659+0000","last_unstale":"2024-04-17T08:31:33.720180+0000","last_undegraded":"2024-04-17T08:31:33.720180+0000","last_fullsized":"2024-04-17T08:31:33.720180+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:02.872844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722357+0000","last_change":"2024-04-17T08:31:32.575634+0000","last_active":"2024-04-17T08:31:33.722357+0000","last_peered":"2024-04-17T08:31:33.722357+0000","last_clean":"2024-04-17T08:31:33.722357+0000","last_became_active":"2024-04-17T08:31:32.575278+0000","last_became_peered":"2024-04-17T08:31:32.575278+0000","last_unstale":"2024-04-17T08:31:33.722357+0000","last_undegraded":"2024-04-17T08:31:33.722357+0000","last_fullsized":"2024-04-17T08:31:33.722357+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:55.088760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581762+0000","last_change":"2024-04-17T08:31:32.579079+0000","last_active":"2024-04-17T08:31:33.581762+0000","last_peered":"2024-04-17T08:31:33.581762+0000","last_clean":"2024-04-17T08:31:33.581762+0000","last_became_active":"2024-04-17T08:31:32.577773+0000","last_became_peered":"2024-04-17T08:31:32.577773+0000","last_unstale":"2024-04-17T08:31:33.581762+0000","last_undegraded":"2024-04-17T08:31:33.581762+0000","last_fullsized":"2024-04-17T08:31:33.581762+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:39.130003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722118+0000","last_change":"2024-04-17T08:31:32.586437+0000","last_active":"2024-04-17T08:31:33.722118+0000","last_peered":"2024-04-17T08:31:33.722118+0000","last_clean":"2024-04-17T08:31:33.722118+0000","last_became_active":"2024-04-17T08:31:32.586158+0000","last_became_peered":"2024-04-17T08:31:32.586158+0000","last_unstale":"2024-04-17T08:31:33.722118+0000","last_undegraded":"2024-04-17T08:31:33.722118+0000","last_fullsized":"2024-04-17T08:31:33.722118+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:32.055531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721839+0000","last_change":"2024-04-17T08:31:32.586521+0000","last_active":"2024-04-17T08:31:33.721839+0000","last_peered":"2024-04-17T08:31:33.721839+0000","last_clean":"2024-04-17T08:31:33.721839+0000","last_became_active":"2024-04-17T08:31:32.586184+0000","last_became_peered":"2024-04-17T08:31:32.586184+0000","last_unstale":"2024-04-17T08:31:33.721839+0000","last_undegraded":"2024-04-17T08:31:33.721839+0000","last_fullsized":"2024-04-17T08:31:33.721839+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:20.199143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722115+0000","last_change":"2024-04-17T08:31:32.579058+0000","last_active":"2024-04-17T08:31:33.722115+0000","last_peered":"2024-04-17T08:31:33.722115+0000","last_clean":"2024-04-17T08:31:33.722115+0000","last_became_active":"2024-04-17T08:31:32.578517+0000","last_became_peered":"2024-04-17T08:31:32.578517+0000","last_unstale":"2024-04-17T08:31:33.722115+0000","last_undegraded":"2024-04-17T08:31:33.722115+0000","last_fullsized":"2024-04-17T08:31:33.722115+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:31:02.773610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721520+0000","last_change":"2024-04-17T08:31:32.582609+0000","last_active":"2024-04-17T08:31:33.721520+0000","last_peered":"2024-04-17T08:31:33.721520+0000","last_clean":"2024-04-17T08:31:33.721520+0000","last_became_active":"2024-04-17T08:31:32.582299+0000","last_became_peered":"2024-04-17T08:31:32.582299+0000","last_unstale":"2024-04-17T08:31:33.721520+0000","last_undegraded":"2024-04-17T08:31:33.721520+0000","last_fullsized":"2024-04-17T08:31:33.721520+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:53.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582953+0000","last_change":"2024-04-17T08:31:32.587300+0000","last_active":"2024-04-17T08:31:33.582953+0000","last_peered":"2024-04-17T08:31:33.582953+0000","last_clean":"2024-04-17T08:31:33.582953+0000","last_became_active":"2024-04-17T08:31:32.586924+0000","last_became_peered":"2024-04-17T08:31:32.586924+0000","last_unstale":"2024-04-17T08:31:33.582953+0000","last_undegraded":"2024-04-17T08:31:33.582953+0000","last_fullsized":"2024-04-17T08:31:33.582953+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:40.750797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720182+0000","last_change":"2024-04-17T08:31:32.589698+0000","last_active":"2024-04-17T08:31:33.720182+0000","last_peered":"2024-04-17T08:31:33.720182+0000","last_clean":"2024-04-17T08:31:33.720182+0000","last_became_active":"2024-04-17T08:31:32.589385+0000","last_became_peered":"2024-04-17T08:31:32.589385+0000","last_unstale":"2024-04-17T08:31:33.720182+0000","last_undegraded":"2024-04-17T08:31:33.720182+0000","last_fullsized":"2024-04-17T08:31:33.720182+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:20.820558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582336+0000","last_change":"2024-04-17T08:31:32.580879+0000","last_active":"2024-04-17T08:31:33.582336+0000","last_peered":"2024-04-17T08:31:33.582336+0000","last_clean":"2024-04-17T08:31:33.582336+0000","last_became_active":"2024-04-17T08:31:32.580496+0000","last_became_peered":"2024-04-17T08:31:32.580496+0000","last_unstale":"2024-04-17T08:31:33.582336+0000","last_undegraded":"2024-04-17T08:31:33.582336+0000","last_fullsized":"2024-04-17T08:31:33.582336+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:25.768435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579422+0000","last_change":"2024-04-17T08:31:32.576746+0000","last_active":"2024-04-17T08:31:33.579422+0000","last_peered":"2024-04-17T08:31:33.579422+0000","last_clean":"2024-04-17T08:31:33.579422+0000","last_became_active":"2024-04-17T08:31:32.576430+0000","last_became_peered":"2024-04-17T08:31:32.576430+0000","last_unstale":"2024-04-17T08:31:33.579422+0000","last_undegraded":"2024-04-17T08:31:33.579422+0000","last_fullsized":"2024-04-17T08:31:33.579422+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:37.968961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581350+0000","last_change":"2024-04-17T08:31:32.586999+0000","last_active":"2024-04-17T08:31:33.581350+0000","last_peered":"2024-04-17T08:31:33.581350+0000","last_clean":"2024-04-17T08:31:33.581350+0000","last_became_active":"2024-04-17T08:31:32.585585+0000","last_became_peered":"2024-04-17T08:31:32.585585+0000","last_unstale":"2024-04-17T08:31:33.581350+0000","last_undegraded":"2024-04-17T08:31:33.581350+0000","last_fullsized":"2024-04-17T08:31:33.581350+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:20:07.779308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586299+0000","last_change":"2024-04-17T08:31:32.584526+0000","last_active":"2024-04-17T08:31:33.586299+0000","last_peered":"2024-04-17T08:31:33.586299+0000","last_clean":"2024-04-17T08:31:33.586299+0000","last_became_active":"2024-04-17T08:31:32.581798+0000","last_became_peered":"2024-04-17T08:31:32.581798+0000","last_unstale":"2024-04-17T08:31:33.586299+0000","last_undegraded":"2024-04-17T08:31:33.586299+0000","last_fullsized":"2024-04-17T08:31:33.586299+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:26.700104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586205+0000","last_change":"2024-04-17T08:31:32.587319+0000","last_active":"2024-04-17T08:31:33.586205+0000","last_peered":"2024-04-17T08:31:33.586205+0000","last_clean":"2024-04-17T08:31:33.586205+0000","last_became_active":"2024-04-17T08:31:32.587046+0000","last_became_peered":"2024-04-17T08:31:32.587046+0000","last_unstale":"2024-04-17T08:31:33.586205+0000","last_undegraded":"2024-04-17T08:31:33.586205+0000","last_fullsized":"2024-04-17T08:31:33.586205+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:46:54.666276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.718985+0000","last_change":"2024-04-17T08:31:32.572171+0000","last_active":"2024-04-17T08:31:33.718985+0000","last_peered":"2024-04-17T08:31:33.718985+0000","last_clean":"2024-04-17T08:31:33.718985+0000","last_became_active":"2024-04-17T08:31:32.571857+0000","last_became_peered":"2024-04-17T08:31:32.571857+0000","last_unstale":"2024-04-17T08:31:33.718985+0000","last_undegraded":"2024-04-17T08:31:33.718985+0000","last_fullsized":"2024-04-17T08:31:33.718985+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:14.768835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720409+0000","last_change":"2024-04-17T08:31:32.587840+0000","last_active":"2024-04-17T08:31:33.720409+0000","last_peered":"2024-04-17T08:31:33.720409+0000","last_clean":"2024-04-17T08:31:33.720409+0000","last_became_active":"2024-04-17T08:31:32.586391+0000","last_became_peered":"2024-04-17T08:31:32.586391+0000","last_unstale":"2024-04-17T08:31:33.720409+0000","last_undegraded":"2024-04-17T08:31:33.720409+0000","last_fullsized":"2024-04-17T08:31:33.720409+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:38.102957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"194'4","reported_seq":18,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580048+0000","last_change":"2024-04-17T08:31:31.552503+0000","last_active":"2024-04-17T08:31:33.580048+0000","last_peered":"2024-04-17T08:31:33.580048+0000","last_clean":"2024-04-17T08:31:33.580048+0000","last_became_active":"2024-04-17T08:31:31.551312+0000","last_became_peered":"2024-04-17T08:31:31.551312+0000","last_unstale":"2024-04-17T08:31:33.580048+0000","last_undegraded":"2024-04-17T08:31:33.580048+0000","last_fullsized":"2024-04-17T08:31:33.580048+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:59.247863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586592+0000","last_change":"2024-04-17T08:31:32.585074+0000","last_active":"2024-04-17T08:31:33.586592+0000","last_peered":"2024-04-17T08:31:33.586592+0000","last_clean":"2024-04-17T08:31:33.586592+0000","last_became_active":"2024-04-17T08:31:32.581521+0000","last_became_peered":"2024-04-17T08:31:32.581521+0000","last_unstale":"2024-04-17T08:31:33.586592+0000","last_undegraded":"2024-04-17T08:31:33.586592+0000","last_fullsized":"2024-04-17T08:31:33.586592+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:29.865883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586055+0000","last_change":"2024-04-17T08:31:31.555944+0000","last_active":"2024-04-17T08:31:33.586055+0000","last_peered":"2024-04-17T08:31:33.586055+0000","last_clean":"2024-04-17T08:31:33.586055+0000","last_became_active":"2024-04-17T08:31:31.555218+0000","last_became_peered":"2024-04-17T08:31:31.555218+0000","last_unstale":"2024-04-17T08:31:33.586055+0000","last_undegraded":"2024-04-17T08:31:33.586055+0000","last_fullsized":"2024-04-17T08:31:33.586055+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:55.501648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722084+0000","last_change":"2024-04-17T08:31:32.582741+0000","last_active":"2024-04-17T08:31:33.722084+0000","last_peered":"2024-04-17T08:31:33.722084+0000","last_clean":"2024-04-17T08:31:33.722084+0000","last_became_active":"2024-04-17T08:31:32.582421+0000","last_became_peered":"2024-04-17T08:31:32.582421+0000","last_unstale":"2024-04-17T08:31:33.722084+0000","last_undegraded":"2024-04-17T08:31:33.722084+0000","last_fullsized":"2024-04-17T08:31:33.722084+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:12.223066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579581+0000","last_change":"2024-04-17T08:31:31.555455+0000","last_active":"2024-04-17T08:31:33.579581+0000","last_peered":"2024-04-17T08:31:33.579581+0000","last_clean":"2024-04-17T08:31:33.579581+0000","last_became_active":"2024-04-17T08:31:31.554972+0000","last_became_peered":"2024-04-17T08:31:31.554972+0000","last_unstale":"2024-04-17T08:31:33.579581+0000","last_undegraded":"2024-04-17T08:31:33.579581+0000","last_fullsized":"2024-04-17T08:31:33.579581+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:24.751908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582316+0000","last_change":"2024-04-17T08:31:32.573529+0000","last_active":"2024-04-17T08:31:33.582316+0000","last_peered":"2024-04-17T08:31:33.582316+0000","last_clean":"2024-04-17T08:31:33.582316+0000","last_became_active":"2024-04-17T08:31:32.573025+0000","last_became_peered":"2024-04-17T08:31:32.573025+0000","last_unstale":"2024-04-17T08:31:33.582316+0000","last_undegraded":"2024-04-17T08:31:33.582316+0000","last_fullsized":"2024-04-17T08:31:33.582316+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:37.518463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579790+0000","last_change":"2024-04-17T08:31:31.551040+0000","last_active":"2024-04-17T08:31:33.579790+0000","last_peered":"2024-04-17T08:31:33.579790+0000","last_clean":"2024-04-17T08:31:33.579790+0000","last_became_active":"2024-04-17T08:31:31.550364+0000","last_became_peered":"2024-04-17T08:31:31.550364+0000","last_unstale":"2024-04-17T08:31:33.579790+0000","last_undegraded":"2024-04-17T08:31:33.579790+0000","last_fullsized":"2024-04-17T08:31:33.579790+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:14.789121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719918+0000","last_change":"2024-04-17T08:31:32.580542+0000","last_active":"2024-04-17T08:31:33.719918+0000","last_peered":"2024-04-17T08:31:33.719918+0000","last_clean":"2024-04-17T08:31:33.719918+0000","last_became_active":"2024-04-17T08:31:32.580248+0000","last_became_peered":"2024-04-17T08:31:32.580248+0000","last_unstale":"2024-04-17T08:31:33.719918+0000","last_undegraded":"2024-04-17T08:31:33.719918+0000","last_fullsized":"2024-04-17T08:31:33.719918+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:56.318745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581474+0000","last_change":"2024-04-17T08:31:31.541315+0000","last_active":"2024-04-17T08:31:33.581474+0000","last_peered":"2024-04-17T08:31:33.581474+0000","last_clean":"2024-04-17T08:31:33.581474+0000","last_became_active":"2024-04-17T08:31:31.540855+0000","last_became_peered":"2024-04-17T08:31:31.540855+0000","last_unstale":"2024-04-17T08:31:33.581474+0000","last_undegraded":"2024-04-17T08:31:33.581474+0000","last_fullsized":"2024-04-17T08:31:33.581474+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:52.194608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579635+0000","last_change":"2024-04-17T08:31:32.582252+0000","last_active":"2024-04-17T08:31:33.579635+0000","last_peered":"2024-04-17T08:31:33.579635+0000","last_clean":"2024-04-17T08:31:33.579635+0000","last_became_active":"2024-04-17T08:31:32.581900+0000","last_became_peered":"2024-04-17T08:31:32.581900+0000","last_unstale":"2024-04-17T08:31:33.579635+0000","last_undegraded":"2024-04-17T08:31:33.579635+0000","last_fullsized":"2024-04-17T08:31:33.579635+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:01.007081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578673+0000","last_change":"2024-04-17T08:31:31.552258+0000","last_active":"2024-04-17T08:31:33.578673+0000","last_peered":"2024-04-17T08:31:33.578673+0000","last_clean":"2024-04-17T08:31:33.578673+0000","last_became_active":"2024-04-17T08:31:31.549114+0000","last_became_peered":"2024-04-17T08:31:31.549114+0000","last_unstale":"2024-04-17T08:31:33.578673+0000","last_undegraded":"2024-04-17T08:31:33.578673+0000","last_fullsized":"2024-04-17T08:31:33.578673+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:39.734270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578787+0000","last_change":"2024-04-17T08:31:32.576723+0000","last_active":"2024-04-17T08:31:33.578787+0000","last_peered":"2024-04-17T08:31:33.578787+0000","last_clean":"2024-04-17T08:31:33.578787+0000","last_became_active":"2024-04-17T08:31:32.576182+0000","last_became_peered":"2024-04-17T08:31:32.576182+0000","last_unstale":"2024-04-17T08:31:33.578787+0000","last_undegraded":"2024-04-17T08:31:33.578787+0000","last_fullsized":"2024-04-17T08:31:33.578787+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:14.735563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721567+0000","last_change":"2024-04-17T08:31:31.556646+0000","last_active":"2024-04-17T08:31:33.721567+0000","last_peered":"2024-04-17T08:31:33.721567+0000","last_clean":"2024-04-17T08:31:33.721567+0000","last_became_active":"2024-04-17T08:31:31.556379+0000","last_became_peered":"2024-04-17T08:31:31.556379+0000","last_unstale":"2024-04-17T08:31:33.721567+0000","last_undegraded":"2024-04-17T08:31:33.721567+0000","last_fullsized":"2024-04-17T08:31:33.721567+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:48.301913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722315+0000","last_change":"2024-04-17T08:31:32.583229+0000","last_active":"2024-04-17T08:31:33.722315+0000","last_peered":"2024-04-17T08:31:33.722315+0000","last_clean":"2024-04-17T08:31:33.722315+0000","last_became_active":"2024-04-17T08:31:32.582925+0000","last_became_peered":"2024-04-17T08:31:32.582925+0000","last_unstale":"2024-04-17T08:31:33.722315+0000","last_undegraded":"2024-04-17T08:31:33.722315+0000","last_fullsized":"2024-04-17T08:31:33.722315+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:16.578895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719387+0000","last_change":"2024-04-17T08:31:31.548659+0000","last_active":"2024-04-17T08:31:33.719387+0000","last_peered":"2024-04-17T08:31:33.719387+0000","last_clean":"2024-04-17T08:31:33.719387+0000","last_became_active":"2024-04-17T08:31:31.548287+0000","last_became_peered":"2024-04-17T08:31:31.548287+0000","last_unstale":"2024-04-17T08:31:33.719387+0000","last_undegraded":"2024-04-17T08:31:33.719387+0000","last_fullsized":"2024-04-17T08:31:33.719387+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:09.494648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720618+0000","last_change":"2024-04-17T08:31:32.570936+0000","last_active":"2024-04-17T08:31:33.720618+0000","last_peered":"2024-04-17T08:31:33.720618+0000","last_clean":"2024-04-17T08:31:33.720618+0000","last_became_active":"2024-04-17T08:31:32.570620+0000","last_became_peered":"2024-04-17T08:31:32.570620+0000","last_unstale":"2024-04-17T08:31:33.720618+0000","last_undegraded":"2024-04-17T08:31:33.720618+0000","last_fullsized":"2024-04-17T08:31:33.720618+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:23.756041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722569+0000","last_change":"2024-04-17T08:31:31.550450+0000","last_active":"2024-04-17T08:31:33.722569+0000","last_peered":"2024-04-17T08:31:33.722569+0000","last_clean":"2024-04-17T08:31:33.722569+0000","last_became_active":"2024-04-17T08:31:31.550156+0000","last_became_peered":"2024-04-17T08:31:31.550156+0000","last_unstale":"2024-04-17T08:31:33.722569+0000","last_undegraded":"2024-04-17T08:31:33.722569+0000","last_fullsized":"2024-04-17T08:31:33.722569+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:04.252568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719637+0000","last_change":"2024-04-17T08:31:32.587147+0000","last_active":"2024-04-17T08:31:33.719637+0000","last_peered":"2024-04-17T08:31:33.719637+0000","last_clean":"2024-04-17T08:31:33.719637+0000","last_became_active":"2024-04-17T08:31:32.586846+0000","last_became_peered":"2024-04-17T08:31:32.586846+0000","last_unstale":"2024-04-17T08:31:33.719637+0000","last_undegraded":"2024-04-17T08:31:33.719637+0000","last_fullsized":"2024-04-17T08:31:33.719637+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:16.055778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"194'3","reported_seq":17,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722363+0000","last_change":"2024-04-17T08:31:31.549648+0000","last_active":"2024-04-17T08:31:33.722363+0000","last_peered":"2024-04-17T08:31:33.722363+0000","last_clean":"2024-04-17T08:31:33.722363+0000","last_became_active":"2024-04-17T08:31:31.549347+0000","last_became_peered":"2024-04-17T08:31:31.549347+0000","last_unstale":"2024-04-17T08:31:33.722363+0000","last_undegraded":"2024-04-17T08:31:33.722363+0000","last_fullsized":"2024-04-17T08:31:33.722363+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:32:46.823813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586415+0000","last_change":"2024-04-17T08:31:32.587678+0000","last_active":"2024-04-17T08:31:33.586415+0000","last_peered":"2024-04-17T08:31:33.586415+0000","last_clean":"2024-04-17T08:31:33.586415+0000","last_became_active":"2024-04-17T08:31:32.581010+0000","last_became_peered":"2024-04-17T08:31:32.581010+0000","last_unstale":"2024-04-17T08:31:33.586415+0000","last_undegraded":"2024-04-17T08:31:33.586415+0000","last_fullsized":"2024-04-17T08:31:33.586415+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:04:58.912860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"194'6","reported_seq":20,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586099+0000","last_change":"2024-04-17T08:31:31.549787+0000","last_active":"2024-04-17T08:31:33.586099+0000","last_peered":"2024-04-17T08:31:33.586099+0000","last_clean":"2024-04-17T08:31:33.586099+0000","last_became_active":"2024-04-17T08:31:31.549474+0000","last_became_peered":"2024-04-17T08:31:31.549474+0000","last_unstale":"2024-04-17T08:31:33.586099+0000","last_undegraded":"2024-04-17T08:31:33.586099+0000","last_fullsized":"2024-04-17T08:31:33.586099+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:47:50.401531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719132+0000","last_change":"2024-04-17T08:31:32.580964+0000","last_active":"2024-04-17T08:31:33.719132+0000","last_peered":"2024-04-17T08:31:33.719132+0000","last_clean":"2024-04-17T08:31:33.719132+0000","last_became_active":"2024-04-17T08:31:32.578743+0000","last_became_peered":"2024-04-17T08:31:32.578743+0000","last_unstale":"2024-04-17T08:31:33.719132+0000","last_undegraded":"2024-04-17T08:31:33.719132+0000","last_fullsized":"2024-04-17T08:31:33.719132+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:21.524847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579178+0000","last_change":"2024-04-17T08:31:31.540943+0000","last_active":"2024-04-17T08:31:33.579178+0000","last_peered":"2024-04-17T08:31:33.579178+0000","last_clean":"2024-04-17T08:31:33.579178+0000","last_became_active":"2024-04-17T08:31:31.540643+0000","last_became_peered":"2024-04-17T08:31:31.540643+0000","last_unstale":"2024-04-17T08:31:33.579178+0000","last_undegraded":"2024-04-17T08:31:33.579178+0000","last_fullsized":"2024-04-17T08:31:33.579178+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:36.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719988+0000","last_change":"2024-04-17T08:31:32.584370+0000","last_active":"2024-04-17T08:31:33.719988+0000","last_peered":"2024-04-17T08:31:33.719988+0000","last_clean":"2024-04-17T08:31:33.719988+0000","last_became_active":"2024-04-17T08:31:32.583337+0000","last_became_peered":"2024-04-17T08:31:32.583337+0000","last_unstale":"2024-04-17T08:31:33.719988+0000","last_undegraded":"2024-04-17T08:31:33.719988+0000","last_fullsized":"2024-04-17T08:31:33.719988+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:35:23.577873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719370+0000","last_change":"2024-04-17T08:31:31.551462+0000","last_active":"2024-04-17T08:31:33.719370+0000","last_peered":"2024-04-17T08:31:33.719370+0000","last_clean":"2024-04-17T08:31:33.719370+0000","last_became_active":"2024-04-17T08:31:31.551172+0000","last_became_peered":"2024-04-17T08:31:31.551172+0000","last_unstale":"2024-04-17T08:31:33.719370+0000","last_undegraded":"2024-04-17T08:31:33.719370+0000","last_fullsized":"2024-04-17T08:31:33.719370+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:36.901245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578927+0000","last_change":"2024-04-17T08:31:32.576496+0000","last_active":"2024-04-17T08:31:33.578927+0000","last_peered":"2024-04-17T08:31:33.578927+0000","last_clean":"2024-04-17T08:31:33.578927+0000","last_became_active":"2024-04-17T08:31:32.575800+0000","last_became_peered":"2024-04-17T08:31:32.575800+0000","last_unstale":"2024-04-17T08:31:33.578927+0000","last_undegraded":"2024-04-17T08:31:33.578927+0000","last_fullsized":"2024-04-17T08:31:33.578927+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:16.901074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578747+0000","last_change":"2024-04-17T08:31:31.552383+0000","last_active":"2024-04-17T08:31:33.578747+0000","last_peered":"2024-04-17T08:31:33.578747+0000","last_clean":"2024-04-17T08:31:33.578747+0000","last_became_active":"2024-04-17T08:31:31.551698+0000","last_became_peered":"2024-04-17T08:31:31.551698+0000","last_unstale":"2024-04-17T08:31:33.578747+0000","last_undegraded":"2024-04-17T08:31:33.578747+0000","last_fullsized":"2024-04-17T08:31:33.578747+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:38.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580543+0000","last_change":"2024-04-17T08:31:32.579764+0000","last_active":"2024-04-17T08:31:33.580543+0000","last_peered":"2024-04-17T08:31:33.580543+0000","last_clean":"2024-04-17T08:31:33.580543+0000","last_became_active":"2024-04-17T08:31:32.579451+0000","last_became_peered":"2024-04-17T08:31:32.579451+0000","last_unstale":"2024-04-17T08:31:33.580543+0000","last_undegraded":"2024-04-17T08:31:33.580543+0000","last_fullsized":"2024-04-17T08:31:33.580543+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:26.861739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"194'8","reported_seq":22,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586134+0000","last_change":"2024-04-17T08:31:31.547977+0000","last_active":"2024-04-17T08:31:33.586134+0000","last_peered":"2024-04-17T08:31:33.586134+0000","last_clean":"2024-04-17T08:31:33.586134+0000","last_became_active":"2024-04-17T08:31:31.547687+0000","last_became_peered":"2024-04-17T08:31:31.547687+0000","last_unstale":"2024-04-17T08:31:33.586134+0000","last_undegraded":"2024-04-17T08:31:33.586134+0000","last_fullsized":"2024-04-17T08:31:33.586134+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:47.928198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580245+0000","last_change":"2024-04-17T08:31:32.582193+0000","last_active":"2024-04-17T08:31:33.580245+0000","last_peered":"2024-04-17T08:31:33.580245+0000","last_clean":"2024-04-17T08:31:33.580245+0000","last_became_active":"2024-04-17T08:31:32.576489+0000","last_became_peered":"2024-04-17T08:31:32.576489+0000","last_unstale":"2024-04-17T08:31:33.580245+0000","last_undegraded":"2024-04-17T08:31:33.580245+0000","last_fullsized":"2024-04-17T08:31:33.580245+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:03.558167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582806+0000","last_change":"2024-04-17T08:31:31.553466+0000","last_active":"2024-04-17T08:31:33.582806+0000","last_peered":"2024-04-17T08:31:33.582806+0000","last_clean":"2024-04-17T08:31:33.582806+0000","last_became_active":"2024-04-17T08:31:31.552436+0000","last_became_peered":"2024-04-17T08:31:31.552436+0000","last_unstale":"2024-04-17T08:31:33.582806+0000","last_undegraded":"2024-04-17T08:31:33.582806+0000","last_fullsized":"2024-04-17T08:31:33.582806+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:19.182616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720768+0000","last_change":"2024-04-17T08:31:32.584297+0000","last_active":"2024-04-17T08:31:33.720768+0000","last_peered":"2024-04-17T08:31:33.720768+0000","last_clean":"2024-04-17T08:31:33.720768+0000","last_became_active":"2024-04-17T08:31:32.583850+0000","last_became_peered":"2024-04-17T08:31:32.583850+0000","last_unstale":"2024-04-17T08:31:33.720768+0000","last_undegraded":"2024-04-17T08:31:33.720768+0000","last_fullsized":"2024-04-17T08:31:33.720768+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:55.906822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581901+0000","last_change":"2024-04-17T08:31:31.538815+0000","last_active":"2024-04-17T08:31:33.581901+0000","last_peered":"2024-04-17T08:31:33.581901+0000","last_clean":"2024-04-17T08:31:33.581901+0000","last_became_active":"2024-04-17T08:31:31.538506+0000","last_became_peered":"2024-04-17T08:31:31.538506+0000","last_unstale":"2024-04-17T08:31:33.581901+0000","last_undegraded":"2024-04-17T08:31:33.581901+0000","last_fullsized":"2024-04-17T08:31:33.581901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:14.937481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579634+0000","last_change":"2024-04-17T08:31:32.582255+0000","last_active":"2024-04-17T08:31:33.579634+0000","last_peered":"2024-04-17T08:31:33.579634+0000","last_clean":"2024-04-17T08:31:33.579634+0000","last_became_active":"2024-04-17T08:31:32.581724+0000","last_became_peered":"2024-04-17T08:31:32.581724+0000","last_unstale":"2024-04-17T08:31:33.579634+0000","last_undegraded":"2024-04-17T08:31:33.579634+0000","last_fullsized":"2024-04-17T08:31:33.579634+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:44.241037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579830+0000","last_change":"2024-04-17T08:31:31.555888+0000","last_active":"2024-04-17T08:31:33.579830+0000","last_peered":"2024-04-17T08:31:33.579830+0000","last_clean":"2024-04-17T08:31:33.579830+0000","last_became_active":"2024-04-17T08:31:31.555599+0000","last_became_peered":"2024-04-17T08:31:31.555599+0000","last_unstale":"2024-04-17T08:31:33.579830+0000","last_undegraded":"2024-04-17T08:31:33.579830+0000","last_fullsized":"2024-04-17T08:31:33.579830+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:13.442598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719954+0000","last_change":"2024-04-17T08:31:32.589808+0000","last_active":"2024-04-17T08:31:33.719954+0000","last_peered":"2024-04-17T08:31:33.719954+0000","last_clean":"2024-04-17T08:31:33.719954+0000","last_became_active":"2024-04-17T08:31:32.589523+0000","last_became_peered":"2024-04-17T08:31:32.589523+0000","last_unstale":"2024-04-17T08:31:33.719954+0000","last_undegraded":"2024-04-17T08:31:33.719954+0000","last_fullsized":"2024-04-17T08:31:33.719954+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:43.489588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579882+0000","last_change":"2024-04-17T08:31:31.553573+0000","last_active":"2024-04-17T08:31:33.579882+0000","last_peered":"2024-04-17T08:31:33.579882+0000","last_clean":"2024-04-17T08:31:33.579882+0000","last_became_active":"2024-04-17T08:31:31.553133+0000","last_became_peered":"2024-04-17T08:31:31.553133+0000","last_unstale":"2024-04-17T08:31:33.579882+0000","last_undegraded":"2024-04-17T08:31:33.579882+0000","last_fullsized":"2024-04-17T08:31:33.579882+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:52.738963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581007+0000","last_change":"2024-04-17T08:31:32.562089+0000","last_active":"2024-04-17T08:31:33.581007+0000","last_peered":"2024-04-17T08:31:33.581007+0000","last_clean":"2024-04-17T08:31:33.581007+0000","last_became_active":"2024-04-17T08:31:32.561659+0000","last_became_peered":"2024-04-17T08:31:32.561659+0000","last_unstale":"2024-04-17T08:31:33.581007+0000","last_undegraded":"2024-04-17T08:31:33.581007+0000","last_fullsized":"2024-04-17T08:31:33.581007+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:21.318610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579344+0000","last_change":"2024-04-17T08:31:31.552886+0000","last_active":"2024-04-17T08:31:33.579344+0000","last_peered":"2024-04-17T08:31:33.579344+0000","last_clean":"2024-04-17T08:31:33.579344+0000","last_became_active":"2024-04-17T08:31:31.551839+0000","last_became_peered":"2024-04-17T08:31:31.551839+0000","last_unstale":"2024-04-17T08:31:33.579344+0000","last_undegraded":"2024-04-17T08:31:33.579344+0000","last_fullsized":"2024-04-17T08:31:33.579344+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:56.307309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720133+0000","last_change":"2024-04-17T08:31:32.590670+0000","last_active":"2024-04-17T08:31:33.720133+0000","last_peered":"2024-04-17T08:31:33.720133+0000","last_clean":"2024-04-17T08:31:33.720133+0000","last_became_active":"2024-04-17T08:31:32.590394+0000","last_became_peered":"2024-04-17T08:31:32.590394+0000","last_unstale":"2024-04-17T08:31:33.720133+0000","last_undegraded":"2024-04-17T08:31:33.720133+0000","last_fullsized":"2024-04-17T08:31:33.720133+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:53.906218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579557+0000","last_change":"2024-04-17T08:31:31.554438+0000","last_active":"2024-04-17T08:31:33.579557+0000","last_peered":"2024-04-17T08:31:33.579557+0000","last_clean":"2024-04-17T08:31:33.579557+0000","last_became_active":"2024-04-17T08:31:31.552942+0000","last_became_peered":"2024-04-17T08:31:31.552942+0000","last_unstale":"2024-04-17T08:31:33.579557+0000","last_undegraded":"2024-04-17T08:31:33.579557+0000","last_fullsized":"2024-04-17T08:31:33.579557+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:42.076821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579280+0000","last_change":"2024-04-17T08:31:32.576687+0000","last_active":"2024-04-17T08:31:33.579280+0000","last_peered":"2024-04-17T08:31:33.579280+0000","last_clean":"2024-04-17T08:31:33.579280+0000","last_became_active":"2024-04-17T08:31:32.576372+0000","last_became_peered":"2024-04-17T08:31:32.576372+0000","last_unstale":"2024-04-17T08:31:33.579280+0000","last_undegraded":"2024-04-17T08:31:33.579280+0000","last_fullsized":"2024-04-17T08:31:33.579280+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:36:19.399635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579037+0000","last_change":"2024-04-17T08:31:31.544808+0000","last_active":"2024-04-17T08:31:33.579037+0000","last_peered":"2024-04-17T08:31:33.579037+0000","last_clean":"2024-04-17T08:31:33.579037+0000","last_became_active":"2024-04-17T08:31:31.544565+0000","last_became_peered":"2024-04-17T08:31:31.544565+0000","last_unstale":"2024-04-17T08:31:33.579037+0000","last_undegraded":"2024-04-17T08:31:33.579037+0000","last_fullsized":"2024-04-17T08:31:33.579037+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:22.223721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722346+0000","last_change":"2024-04-17T08:31:32.582664+0000","last_active":"2024-04-17T08:31:33.722346+0000","last_peered":"2024-04-17T08:31:33.722346+0000","last_clean":"2024-04-17T08:31:33.722346+0000","last_became_active":"2024-04-17T08:31:32.582200+0000","last_became_peered":"2024-04-17T08:31:32.582200+0000","last_unstale":"2024-04-17T08:31:33.722346+0000","last_undegraded":"2024-04-17T08:31:33.722346+0000","last_fullsized":"2024-04-17T08:31:33.722346+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:23.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579347+0000","last_change":"2024-04-17T08:31:31.555126+0000","last_active":"2024-04-17T08:31:33.579347+0000","last_peered":"2024-04-17T08:31:33.579347+0000","last_clean":"2024-04-17T08:31:33.579347+0000","last_became_active":"2024-04-17T08:31:31.554836+0000","last_became_peered":"2024-04-17T08:31:31.554836+0000","last_unstale":"2024-04-17T08:31:33.579347+0000","last_undegraded":"2024-04-17T08:31:33.579347+0000","last_fullsized":"2024-04-17T08:31:33.579347+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:29.890915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720606+0000","last_change":"2024-04-17T08:31:32.584094+0000","last_active":"2024-04-17T08:31:33.720606+0000","last_peered":"2024-04-17T08:31:33.720606+0000","last_clean":"2024-04-17T08:31:33.720606+0000","last_became_active":"2024-04-17T08:31:32.581409+0000","last_became_peered":"2024-04-17T08:31:32.581409+0000","last_unstale":"2024-04-17T08:31:33.720606+0000","last_undegraded":"2024-04-17T08:31:33.720606+0000","last_fullsized":"2024-04-17T08:31:33.720606+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:30.424287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720871+0000","last_change":"2024-04-17T08:31:31.552940+0000","last_active":"2024-04-17T08:31:33.720871+0000","last_peered":"2024-04-17T08:31:33.720871+0000","last_clean":"2024-04-17T08:31:33.720871+0000","last_became_active":"2024-04-17T08:31:31.552640+0000","last_became_peered":"2024-04-17T08:31:31.552640+0000","last_unstale":"2024-04-17T08:31:33.720871+0000","last_undegraded":"2024-04-17T08:31:33.720871+0000","last_fullsized":"2024-04-17T08:31:33.720871+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:46.859215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720816+0000","last_change":"2024-04-17T08:31:32.586133+0000","last_active":"2024-04-17T08:31:33.720816+0000","last_peered":"2024-04-17T08:31:33.720816+0000","last_clean":"2024-04-17T08:31:33.720816+0000","last_became_active":"2024-04-17T08:31:32.585843+0000","last_became_peered":"2024-04-17T08:31:32.585843+0000","last_unstale":"2024-04-17T08:31:33.720816+0000","last_undegraded":"2024-04-17T08:31:33.720816+0000","last_fullsized":"2024-04-17T08:31:33.720816+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:49.200424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"195'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.585247+0000","last_change":"2024-04-17T08:31:31.546420+0000","last_active":"2024-04-17T08:31:33.585247+0000","last_peered":"2024-04-17T08:31:33.585247+0000","last_clean":"2024-04-17T08:31:33.585247+0000","last_became_active":"2024-04-17T08:31:31.540406+0000","last_became_peered":"2024-04-17T08:31:31.540406+0000","last_unstale":"2024-04-17T08:31:33.585247+0000","last_undegraded":"2024-04-17T08:31:33.585247+0000","last_fullsized":"2024-04-17T08:31:33.585247+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:16.400806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722119+0000","last_change":"2024-04-17T08:31:32.585635+0000","last_active":"2024-04-17T08:31:33.722119+0000","last_peered":"2024-04-17T08:31:33.722119+0000","last_clean":"2024-04-17T08:31:33.722119+0000","last_became_active":"2024-04-17T08:31:32.585340+0000","last_became_peered":"2024-04-17T08:31:32.585340+0000","last_unstale":"2024-04-17T08:31:33.722119+0000","last_undegraded":"2024-04-17T08:31:33.722119+0000","last_fullsized":"2024-04-17T08:31:33.722119+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:52.188712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722310+0000","last_change":"2024-04-17T08:31:31.553569+0000","last_active":"2024-04-17T08:31:33.722310+0000","last_peered":"2024-04-17T08:31:33.722310+0000","last_clean":"2024-04-17T08:31:33.722310+0000","last_became_active":"2024-04-17T08:31:31.551799+0000","last_became_peered":"2024-04-17T08:31:31.551799+0000","last_unstale":"2024-04-17T08:31:33.722310+0000","last_undegraded":"2024-04-17T08:31:33.722310+0000","last_fullsized":"2024-04-17T08:31:33.722310+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:44.852505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720181+0000","last_change":"2024-04-17T08:31:32.565160+0000","last_active":"2024-04-17T08:31:33.720181+0000","last_peered":"2024-04-17T08:31:33.720181+0000","last_clean":"2024-04-17T08:31:33.720181+0000","last_became_active":"2024-04-17T08:31:32.562882+0000","last_became_peered":"2024-04-17T08:31:32.562882+0000","last_unstale":"2024-04-17T08:31:33.720181+0000","last_undegraded":"2024-04-17T08:31:33.720181+0000","last_fullsized":"2024-04-17T08:31:33.720181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:08.659018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720435+0000","last_change":"2024-04-17T08:31:31.539426+0000","last_active":"2024-04-17T08:31:33.720435+0000","last_peered":"2024-04-17T08:31:33.720435+0000","last_clean":"2024-04-17T08:31:33.720435+0000","last_became_active":"2024-04-17T08:31:31.539004+0000","last_became_peered":"2024-04-17T08:31:31.539004+0000","last_unstale":"2024-04-17T08:31:33.720435+0000","last_undegraded":"2024-04-17T08:31:33.720435+0000","last_fullsized":"2024-04-17T08:31:33.720435+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:16:59.690825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719573+0000","last_change":"2024-04-17T08:31:32.572245+0000","last_active":"2024-04-17T08:31:33.719573+0000","last_peered":"2024-04-17T08:31:33.719573+0000","last_clean":"2024-04-17T08:31:33.719573+0000","last_became_active":"2024-04-17T08:31:32.571929+0000","last_became_peered":"2024-04-17T08:31:32.571929+0000","last_unstale":"2024-04-17T08:31:33.719573+0000","last_undegraded":"2024-04-17T08:31:33.719573+0000","last_fullsized":"2024-04-17T08:31:33.719573+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:06.829939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.585929+0000","last_change":"2024-04-17T08:31:31.543062+0000","last_active":"2024-04-17T08:31:33.585929+0000","last_peered":"2024-04-17T08:31:33.585929+0000","last_clean":"2024-04-17T08:31:33.585929+0000","last_became_active":"2024-04-17T08:31:31.540387+0000","last_became_peered":"2024-04-17T08:31:31.540387+0000","last_unstale":"2024-04-17T08:31:33.585929+0000","last_undegraded":"2024-04-17T08:31:33.585929+0000","last_fullsized":"2024-04-17T08:31:33.585929+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:50.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586482+0000","last_change":"2024-04-17T08:31:32.585039+0000","last_active":"2024-04-17T08:31:33.586482+0000","last_peered":"2024-04-17T08:31:33.586482+0000","last_clean":"2024-04-17T08:31:33.586482+0000","last_became_active":"2024-04-17T08:31:32.584748+0000","last_became_peered":"2024-04-17T08:31:32.584748+0000","last_unstale":"2024-04-17T08:31:33.586482+0000","last_undegraded":"2024-04-17T08:31:33.586482+0000","last_fullsized":"2024-04-17T08:31:33.586482+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:34:03.496325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578686+0000","last_change":"2024-04-17T08:31:31.548504+0000","last_active":"2024-04-17T08:31:33.578686+0000","last_peered":"2024-04-17T08:31:33.578686+0000","last_clean":"2024-04-17T08:31:33.578686+0000","last_became_active":"2024-04-17T08:31:31.546576+0000","last_became_peered":"2024-04-17T08:31:31.546576+0000","last_unstale":"2024-04-17T08:31:33.578686+0000","last_undegraded":"2024-04-17T08:31:33.578686+0000","last_fullsized":"2024-04-17T08:31:33.578686+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:38.545887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719808+0000","last_change":"2024-04-17T08:31:32.592976+0000","last_active":"2024-04-17T08:31:33.719808+0000","last_peered":"2024-04-17T08:31:33.719808+0000","last_clean":"2024-04-17T08:31:33.719808+0000","last_became_active":"2024-04-17T08:31:32.592664+0000","last_became_peered":"2024-04-17T08:31:32.592664+0000","last_unstale":"2024-04-17T08:31:33.719808+0000","last_undegraded":"2024-04-17T08:31:33.719808+0000","last_fullsized":"2024-04-17T08:31:33.719808+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:44.800190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721865+0000","last_change":"2024-04-17T08:31:31.545445+0000","last_active":"2024-04-17T08:31:33.721865+0000","last_peered":"2024-04-17T08:31:33.721865+0000","last_clean":"2024-04-17T08:31:33.721865+0000","last_became_active":"2024-04-17T08:31:31.545158+0000","last_became_peered":"2024-04-17T08:31:31.545158+0000","last_unstale":"2024-04-17T08:31:33.721865+0000","last_undegraded":"2024-04-17T08:31:33.721865+0000","last_fullsized":"2024-04-17T08:31:33.721865+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:03.597001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580707+0000","last_change":"2024-04-17T08:31:32.570700+0000","last_active":"2024-04-17T08:31:33.580707+0000","last_peered":"2024-04-17T08:31:33.580707+0000","last_clean":"2024-04-17T08:31:33.580707+0000","last_became_active":"2024-04-17T08:31:32.570362+0000","last_became_peered":"2024-04-17T08:31:32.570362+0000","last_unstale":"2024-04-17T08:31:33.580707+0000","last_undegraded":"2024-04-17T08:31:33.580707+0000","last_fullsized":"2024-04-17T08:31:33.580707+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:40.337116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586292+0000","last_change":"2024-04-17T08:31:31.532073+0000","last_active":"2024-04-17T08:31:33.586292+0000","last_peered":"2024-04-17T08:31:33.586292+0000","last_clean":"2024-04-17T08:31:33.586292+0000","last_became_active":"2024-04-17T08:31:31.531233+0000","last_became_peered":"2024-04-17T08:31:31.531233+0000","last_unstale":"2024-04-17T08:31:33.586292+0000","last_undegraded":"2024-04-17T08:31:33.586292+0000","last_fullsized":"2024-04-17T08:31:33.586292+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:15.473555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720410+0000","last_change":"2024-04-17T08:31:32.585600+0000","last_active":"2024-04-17T08:31:33.720410+0000","last_peered":"2024-04-17T08:31:33.720410+0000","last_clean":"2024-04-17T08:31:33.720410+0000","last_became_active":"2024-04-17T08:31:32.583740+0000","last_became_peered":"2024-04-17T08:31:32.583740+0000","last_unstale":"2024-04-17T08:31:33.720410+0000","last_undegraded":"2024-04-17T08:31:33.720410+0000","last_fullsized":"2024-04-17T08:31:33.720410+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:12.524053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586352+0000","last_change":"2024-04-17T08:31:31.548777+0000","last_active":"2024-04-17T08:31:33.586352+0000","last_peered":"2024-04-17T08:31:33.586352+0000","last_clean":"2024-04-17T08:31:33.586352+0000","last_became_active":"2024-04-17T08:31:31.548505+0000","last_became_peered":"2024-04-17T08:31:31.548505+0000","last_unstale":"2024-04-17T08:31:33.586352+0000","last_undegraded":"2024-04-17T08:31:33.586352+0000","last_fullsized":"2024-04-17T08:31:33.586352+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:13.719036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"184'299","reported_seq":1176,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579411+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:31:33.579411+0000","last_peered":"2024-04-17T08:31:33.579411+0000","last_clean":"2024-04-17T08:31:33.579411+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:31:33.579411+0000","last_undegraded":"2024-04-17T08:31:33.579411+0000","last_fullsized":"2024-04-17T08:31:33.579411+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":487,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":487,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48048,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323792,"statfs":{"total":96636764160,"available":96587563008,"internally_reserved":0,"allocated":3051520,"data_stored":1775198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2437120,"data_stored":1182498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543068,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41892,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329948,"statfs":{"total":96636764160,"available":96593866752,"internally_reserved":0,"allocated":2441216,"data_stored":1183080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543068,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55780,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316060,"statfs":{"total":96636764160,"available":96579645440,"internally_reserved":0,"allocated":2449408,"data_stored":1184158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543070,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64092,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307748,"statfs":{"total":96636764160,"available":96571133952,"internally_reserved":0,"allocated":2437120,"data_stored":1182510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47448,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324392,"statfs":{"total":96636764160,"available":96588177408,"internally_reserved":0,"allocated":2437120,"data_stored":1182566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543070,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59936,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311904,"statfs":{"total":96636764160,"available":96575389696,"internally_reserved":0,"allocated":2437120,"data_stored":1182510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3026944,"data_stored":1772844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:31:37.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:31:37.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.349+0000 7ff7e6400640 1 Processor -- start 2024-04-17T08:31:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.349+0000 7ff7e6400640 1 -- start start 2024-04-17T08:31:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7e6400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7e015b530 0x7ff7e0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7e6400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e00c6140 con 0x7ff7e00c68c0 2024-04-17T08:31:37.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7e6400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e015f3c0 con 0x7ff7e015b910 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7e6400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff7e006ed60 con 0x7ff7e015b530 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7e015b530 0x7ff7e0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7e015b530 0x7ff7e0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46744/0 (socket says 172.21.15.67:46744) 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7df400640 1 -- 172.21.15.67:0/1866977136 learned_addr learned my addr 172.21.15.67:0/1866977136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.351+0000 7ff7dea00640 1 --1- 172.21.15.67:0/1866977136 >> v1:172.21.15.67:6789/0 conn(0x7ff7e00c68c0 0x7ff7e00c5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55254/0 (socket says 172.21.15.67:55254) 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.352+0000 7ff7df400640 1 -- 172.21.15.67:0/1866977136 >> v1:172.21.15.187:6789/0 conn(0x7ff7e015b910 legacy=0x7ff7e00c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:37.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.352+0000 7ff7df400640 1 -- 172.21.15.67:0/1866977136 >> v1:172.21.15.67:6789/0 conn(0x7ff7e00c68c0 legacy=0x7ff7e00c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.352+0000 7ff7df400640 1 -- 172.21.15.67:0/1866977136 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e015f660 con 0x7ff7e015b530 2024-04-17T08:31:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.353+0000 7ff7df400640 1 --2- 172.21.15.67:0/1866977136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e0071d10 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7ff7e006eee0 tx=0x7ff7d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703e0e353336f5b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.353+0000 7ff7de000640 1 -- 172.21.15.67:0/1866977136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0034070 con 0x7ff7e015b530 2024-04-17T08:31:37.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.353+0000 7ff7de000640 1 -- 172.21.15.67:0/1866977136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d0030840 con 0x7ff7e015b530 2024-04-17T08:31:37.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.353+0000 7ff7de000640 1 -- 172.21.15.67:0/1866977136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0030c10 con 0x7ff7e015b530 2024-04-17T08:31:37.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 -- 172.21.15.67:0/1866977136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 msgr2=0x7ff7e0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 --2- 172.21.15.67:0/1866977136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e0071d10 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7ff7e006eee0 tx=0x7ff7d002b980 comp rx=0 tx=0).stop 2024-04-17T08:31:37.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 -- 172.21.15.67:0/1866977136 shutdown_connections 2024-04-17T08:31:37.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 --2- 172.21.15.67:0/1866977136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e0071d10 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 -- 172.21.15.67:0/1866977136 >> 172.21.15.67:0/1866977136 conn(0x7ff7e006bf50 msgr2=0x7ff7e00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:37.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 -- 172.21.15.67:0/1866977136 shutdown_connections 2024-04-17T08:31:37.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.354+0000 7ff7e6400640 1 -- 172.21.15.67:0/1866977136 wait complete. 2024-04-17T08:31:37.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.356+0000 7ff7e6400640 1 Processor -- start 2024-04-17T08:31:37.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.356+0000 7ff7e6400640 1 -- start start 2024-04-17T08:31:37.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 0x7ff7e014c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 0x7ff7e00ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e00ccca0 con 0x7ff7e015b530 2024-04-17T08:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e015fc20 con 0x7ff7e015b910 2024-04-17T08:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7e6400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e00cc950 con 0x7ff7e00c68c0 2024-04-17T08:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46758/0 (socket says 172.21.15.67:46758) 2024-04-17T08:31:37.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.357+0000 7ff7dea00640 1 -- 172.21.15.67:0/142553886 learned_addr learned my addr 172.21.15.67:0/142553886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dfe00640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 0x7ff7e00ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7df400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 0x7ff7e014c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 -- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 msgr2=0x7ff7e014c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 0x7ff7e014c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 -- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 msgr2=0x7ff7e00ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 0x7ff7e00ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 -- 172.21.15.67:0/142553886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e014c5a0 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7df400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 0x7ff7e014c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dfe00640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 0x7ff7e00ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7dea00640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c470 tx=0x7ff7cc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0074e0 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.358+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7cc007680 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.359+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e0154540 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.359+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc004a20 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.359+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e015c1b0 con 0x7ff7e015b530 2024-04-17T08:31:37.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.360+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e0068b90 con 0x7ff7e015b530 2024-04-17T08:31:37.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.361+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7ff7cc01a020 con 0x7ff7e015b530 2024-04-17T08:31:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.362+0000 7ff7d7400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 0x7ff7c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.362+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7ff7cc08b500 con 0x7ff7e015b530 2024-04-17T08:31:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.363+0000 7ff7df400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 0x7ff7c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.364+0000 7ff7df400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 0x7ff7c006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0036000 tx=0x7ff7d003b040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.367+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff7cc0597f0 con 0x7ff7e015b530 2024-04-17T08:31:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.562+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 --> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7e00c6390 con 0x7ff7c006ca10 2024-04-17T08:31:37.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.575+0000 7ff7d7400640 1 -- 172.21.15.67:0/142553886 <== mgr.7807 v2:172.21.15.187:6832/3300567104 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252496 (secure 0 0 0) 0x7ff7e00c6390 con 0x7ff7c006ca10 2024-04-17T08:31:37.578 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:37.579 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 msgr2=0x7ff7c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 0x7ff7c006eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0036000 tx=0x7ff7d003b040 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 msgr2=0x7ff7e00ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c470 tx=0x7ff7cc001e70 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 shutdown_connections 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7ff7c006ca10 0x7ff7c006eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7e015b910 0x7ff7e00ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7e015b530 0x7ff7e00ce1f0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 --2- 172.21.15.67:0/142553886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7e00c68c0 0x7ff7e014c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 >> 172.21.15.67:0/142553886 conn(0x7ff7e006bf50 msgr2=0x7ff7e00cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:37.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 shutdown_connections 2024-04-17T08:31:37.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.581+0000 7ff7e6400640 1 -- 172.21.15.67:0/142553886 wait complete. 2024-04-17T08:31:37.598 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-17T08:31:35.858686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":532,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":338,"ondisk_log_size":338,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419584,"kb_used_data":20232,"kb_used_omap":12,"kb_used_meta":399219,"kb_avail":754555136,"statfs":{"total":773094113280,"available":772664459264,"internally_reserved":0,"allocated":20717568,"data_stored":10645364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":408800816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.349708"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579935+0000","last_change":"2024-04-17T08:31:32.569758+0000","last_active":"2024-04-17T08:31:33.579935+0000","last_peered":"2024-04-17T08:31:33.579935+0000","last_clean":"2024-04-17T08:31:33.579935+0000","last_became_active":"2024-04-17T08:31:32.569415+0000","last_became_peered":"2024-04-17T08:31:32.569415+0000","last_unstale":"2024-04-17T08:31:33.579935+0000","last_undegraded":"2024-04-17T08:31:33.579935+0000","last_fullsized":"2024-04-17T08:31:33.579935+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:50.692018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721910+0000","last_change":"2024-04-17T08:31:32.581312+0000","last_active":"2024-04-17T08:31:33.721910+0000","last_peered":"2024-04-17T08:31:33.721910+0000","last_clean":"2024-04-17T08:31:33.721910+0000","last_became_active":"2024-04-17T08:31:32.581012+0000","last_became_peered":"2024-04-17T08:31:32.581012+0000","last_unstale":"2024-04-17T08:31:33.721910+0000","last_undegraded":"2024-04-17T08:31:33.721910+0000","last_fullsized":"2024-04-17T08:31:33.721910+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:51.478954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582046+0000","last_change":"2024-04-17T08:31:32.573303+0000","last_active":"2024-04-17T08:31:33.582046+0000","last_peered":"2024-04-17T08:31:33.582046+0000","last_clean":"2024-04-17T08:31:33.582046+0000","last_became_active":"2024-04-17T08:31:32.572644+0000","last_became_peered":"2024-04-17T08:31:32.572644+0000","last_unstale":"2024-04-17T08:31:33.582046+0000","last_undegraded":"2024-04-17T08:31:33.582046+0000","last_fullsized":"2024-04-17T08:31:33.582046+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:23.240248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579908+0000","last_change":"2024-04-17T08:31:32.582020+0000","last_active":"2024-04-17T08:31:33.579908+0000","last_peered":"2024-04-17T08:31:33.579908+0000","last_clean":"2024-04-17T08:31:33.579908+0000","last_became_active":"2024-04-17T08:31:32.581337+0000","last_became_peered":"2024-04-17T08:31:32.581337+0000","last_unstale":"2024-04-17T08:31:33.579908+0000","last_undegraded":"2024-04-17T08:31:33.579908+0000","last_fullsized":"2024-04-17T08:31:33.579908+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:26.212613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721734+0000","last_change":"2024-04-17T08:31:32.582417+0000","last_active":"2024-04-17T08:31:33.721734+0000","last_peered":"2024-04-17T08:31:33.721734+0000","last_clean":"2024-04-17T08:31:33.721734+0000","last_became_active":"2024-04-17T08:31:32.582025+0000","last_became_peered":"2024-04-17T08:31:32.582025+0000","last_unstale":"2024-04-17T08:31:33.721734+0000","last_undegraded":"2024-04-17T08:31:33.721734+0000","last_fullsized":"2024-04-17T08:31:33.721734+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:17.417333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722474+0000","last_change":"2024-04-17T08:31:32.578286+0000","last_active":"2024-04-17T08:31:33.722474+0000","last_peered":"2024-04-17T08:31:33.722474+0000","last_clean":"2024-04-17T08:31:33.722474+0000","last_became_active":"2024-04-17T08:31:32.577947+0000","last_became_peered":"2024-04-17T08:31:32.577947+0000","last_unstale":"2024-04-17T08:31:33.722474+0000","last_undegraded":"2024-04-17T08:31:33.722474+0000","last_fullsized":"2024-04-17T08:31:33.722474+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:38.450228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720125+0000","last_change":"2024-04-17T08:31:32.590058+0000","last_active":"2024-04-17T08:31:33.720125+0000","last_peered":"2024-04-17T08:31:33.720125+0000","last_clean":"2024-04-17T08:31:33.720125+0000","last_became_active":"2024-04-17T08:31:32.589724+0000","last_became_peered":"2024-04-17T08:31:32.589724+0000","last_unstale":"2024-04-17T08:31:33.720125+0000","last_undegraded":"2024-04-17T08:31:33.720125+0000","last_fullsized":"2024-04-17T08:31:33.720125+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:21:43.178564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721444+0000","last_change":"2024-04-17T08:31:32.581939+0000","last_active":"2024-04-17T08:31:33.721444+0000","last_peered":"2024-04-17T08:31:33.721444+0000","last_clean":"2024-04-17T08:31:33.721444+0000","last_became_active":"2024-04-17T08:31:32.580851+0000","last_became_peered":"2024-04-17T08:31:32.580851+0000","last_unstale":"2024-04-17T08:31:33.721444+0000","last_undegraded":"2024-04-17T08:31:33.721444+0000","last_fullsized":"2024-04-17T08:31:33.721444+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:15.954008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722557+0000","last_change":"2024-04-17T08:31:32.582031+0000","last_active":"2024-04-17T08:31:33.722557+0000","last_peered":"2024-04-17T08:31:33.722557+0000","last_clean":"2024-04-17T08:31:33.722557+0000","last_became_active":"2024-04-17T08:31:32.581730+0000","last_became_peered":"2024-04-17T08:31:32.581730+0000","last_unstale":"2024-04-17T08:31:33.722557+0000","last_undegraded":"2024-04-17T08:31:33.722557+0000","last_fullsized":"2024-04-17T08:31:33.722557+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:10:18.372976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722514+0000","last_change":"2024-04-17T08:31:32.585759+0000","last_active":"2024-04-17T08:31:33.722514+0000","last_peered":"2024-04-17T08:31:33.722514+0000","last_clean":"2024-04-17T08:31:33.722514+0000","last_became_active":"2024-04-17T08:31:32.585441+0000","last_became_peered":"2024-04-17T08:31:32.585441+0000","last_unstale":"2024-04-17T08:31:33.722514+0000","last_undegraded":"2024-04-17T08:31:33.722514+0000","last_fullsized":"2024-04-17T08:31:33.722514+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:40:31.918038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722417+0000","last_change":"2024-04-17T08:31:32.589047+0000","last_active":"2024-04-17T08:31:33.722417+0000","last_peered":"2024-04-17T08:31:33.722417+0000","last_clean":"2024-04-17T08:31:33.722417+0000","last_became_active":"2024-04-17T08:31:32.588738+0000","last_became_peered":"2024-04-17T08:31:32.588738+0000","last_unstale":"2024-04-17T08:31:33.722417+0000","last_undegraded":"2024-04-17T08:31:33.722417+0000","last_fullsized":"2024-04-17T08:31:33.722417+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:57.297905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722000+0000","last_change":"2024-04-17T08:31:32.582010+0000","last_active":"2024-04-17T08:31:33.722000+0000","last_peered":"2024-04-17T08:31:33.722000+0000","last_clean":"2024-04-17T08:31:33.722000+0000","last_became_active":"2024-04-17T08:31:32.581632+0000","last_became_peered":"2024-04-17T08:31:32.581632+0000","last_unstale":"2024-04-17T08:31:33.722000+0000","last_undegraded":"2024-04-17T08:31:33.722000+0000","last_fullsized":"2024-04-17T08:31:33.722000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:31.192021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581323+0000","last_change":"2024-04-17T08:31:32.578019+0000","last_active":"2024-04-17T08:31:33.581323+0000","last_peered":"2024-04-17T08:31:33.581323+0000","last_clean":"2024-04-17T08:31:33.581323+0000","last_became_active":"2024-04-17T08:31:32.577733+0000","last_became_peered":"2024-04-17T08:31:32.577733+0000","last_unstale":"2024-04-17T08:31:33.581323+0000","last_undegraded":"2024-04-17T08:31:33.581323+0000","last_fullsized":"2024-04-17T08:31:33.581323+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:54.699101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580783+0000","last_change":"2024-04-17T08:31:32.576812+0000","last_active":"2024-04-17T08:31:33.580783+0000","last_peered":"2024-04-17T08:31:33.580783+0000","last_clean":"2024-04-17T08:31:33.580783+0000","last_became_active":"2024-04-17T08:31:32.576535+0000","last_became_peered":"2024-04-17T08:31:32.576535+0000","last_unstale":"2024-04-17T08:31:33.580783+0000","last_undegraded":"2024-04-17T08:31:33.580783+0000","last_fullsized":"2024-04-17T08:31:33.580783+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:34.672847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579277+0000","last_change":"2024-04-17T08:31:32.585287+0000","last_active":"2024-04-17T08:31:33.579277+0000","last_peered":"2024-04-17T08:31:33.579277+0000","last_clean":"2024-04-17T08:31:33.579277+0000","last_became_active":"2024-04-17T08:31:32.584983+0000","last_became_peered":"2024-04-17T08:31:32.584983+0000","last_unstale":"2024-04-17T08:31:33.579277+0000","last_undegraded":"2024-04-17T08:31:33.579277+0000","last_fullsized":"2024-04-17T08:31:33.579277+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:05.681152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719087+0000","last_change":"2024-04-17T08:31:32.593519+0000","last_active":"2024-04-17T08:31:33.719087+0000","last_peered":"2024-04-17T08:31:33.719087+0000","last_clean":"2024-04-17T08:31:33.719087+0000","last_became_active":"2024-04-17T08:31:32.593198+0000","last_became_peered":"2024-04-17T08:31:32.593198+0000","last_unstale":"2024-04-17T08:31:33.719087+0000","last_undegraded":"2024-04-17T08:31:33.719087+0000","last_fullsized":"2024-04-17T08:31:33.719087+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:00.330621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722625+0000","last_change":"2024-04-17T08:31:32.585581+0000","last_active":"2024-04-17T08:31:33.722625+0000","last_peered":"2024-04-17T08:31:33.722625+0000","last_clean":"2024-04-17T08:31:33.722625+0000","last_became_active":"2024-04-17T08:31:32.585302+0000","last_became_peered":"2024-04-17T08:31:32.585302+0000","last_unstale":"2024-04-17T08:31:33.722625+0000","last_undegraded":"2024-04-17T08:31:33.722625+0000","last_fullsized":"2024-04-17T08:31:33.722625+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:54.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720180+0000","last_change":"2024-04-17T08:31:32.579030+0000","last_active":"2024-04-17T08:31:33.720180+0000","last_peered":"2024-04-17T08:31:33.720180+0000","last_clean":"2024-04-17T08:31:33.720180+0000","last_became_active":"2024-04-17T08:31:32.578659+0000","last_became_peered":"2024-04-17T08:31:32.578659+0000","last_unstale":"2024-04-17T08:31:33.720180+0000","last_undegraded":"2024-04-17T08:31:33.720180+0000","last_fullsized":"2024-04-17T08:31:33.720180+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:02.872844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722357+0000","last_change":"2024-04-17T08:31:32.575634+0000","last_active":"2024-04-17T08:31:33.722357+0000","last_peered":"2024-04-17T08:31:33.722357+0000","last_clean":"2024-04-17T08:31:33.722357+0000","last_became_active":"2024-04-17T08:31:32.575278+0000","last_became_peered":"2024-04-17T08:31:32.575278+0000","last_unstale":"2024-04-17T08:31:33.722357+0000","last_undegraded":"2024-04-17T08:31:33.722357+0000","last_fullsized":"2024-04-17T08:31:33.722357+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:41:55.088760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581762+0000","last_change":"2024-04-17T08:31:32.579079+0000","last_active":"2024-04-17T08:31:33.581762+0000","last_peered":"2024-04-17T08:31:33.581762+0000","last_clean":"2024-04-17T08:31:33.581762+0000","last_became_active":"2024-04-17T08:31:32.577773+0000","last_became_peered":"2024-04-17T08:31:32.577773+0000","last_unstale":"2024-04-17T08:31:33.581762+0000","last_undegraded":"2024-04-17T08:31:33.581762+0000","last_fullsized":"2024-04-17T08:31:33.581762+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:39.130003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722118+0000","last_change":"2024-04-17T08:31:32.586437+0000","last_active":"2024-04-17T08:31:33.722118+0000","last_peered":"2024-04-17T08:31:33.722118+0000","last_clean":"2024-04-17T08:31:33.722118+0000","last_became_active":"2024-04-17T08:31:32.586158+0000","last_became_peered":"2024-04-17T08:31:32.586158+0000","last_unstale":"2024-04-17T08:31:33.722118+0000","last_undegraded":"2024-04-17T08:31:33.722118+0000","last_fullsized":"2024-04-17T08:31:33.722118+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:32.055531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721839+0000","last_change":"2024-04-17T08:31:32.586521+0000","last_active":"2024-04-17T08:31:33.721839+0000","last_peered":"2024-04-17T08:31:33.721839+0000","last_clean":"2024-04-17T08:31:33.721839+0000","last_became_active":"2024-04-17T08:31:32.586184+0000","last_became_peered":"2024-04-17T08:31:32.586184+0000","last_unstale":"2024-04-17T08:31:33.721839+0000","last_undegraded":"2024-04-17T08:31:33.721839+0000","last_fullsized":"2024-04-17T08:31:33.721839+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:20.199143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722115+0000","last_change":"2024-04-17T08:31:32.579058+0000","last_active":"2024-04-17T08:31:33.722115+0000","last_peered":"2024-04-17T08:31:33.722115+0000","last_clean":"2024-04-17T08:31:33.722115+0000","last_became_active":"2024-04-17T08:31:32.578517+0000","last_became_peered":"2024-04-17T08:31:32.578517+0000","last_unstale":"2024-04-17T08:31:33.722115+0000","last_undegraded":"2024-04-17T08:31:33.722115+0000","last_fullsized":"2024-04-17T08:31:33.722115+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:31:02.773610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721520+0000","last_change":"2024-04-17T08:31:32.582609+0000","last_active":"2024-04-17T08:31:33.721520+0000","last_peered":"2024-04-17T08:31:33.721520+0000","last_clean":"2024-04-17T08:31:33.721520+0000","last_became_active":"2024-04-17T08:31:32.582299+0000","last_became_peered":"2024-04-17T08:31:32.582299+0000","last_unstale":"2024-04-17T08:31:33.721520+0000","last_undegraded":"2024-04-17T08:31:33.721520+0000","last_fullsized":"2024-04-17T08:31:33.721520+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:53.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582953+0000","last_change":"2024-04-17T08:31:32.587300+0000","last_active":"2024-04-17T08:31:33.582953+0000","last_peered":"2024-04-17T08:31:33.582953+0000","last_clean":"2024-04-17T08:31:33.582953+0000","last_became_active":"2024-04-17T08:31:32.586924+0000","last_became_peered":"2024-04-17T08:31:32.586924+0000","last_unstale":"2024-04-17T08:31:33.582953+0000","last_undegraded":"2024-04-17T08:31:33.582953+0000","last_fullsized":"2024-04-17T08:31:33.582953+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:40.750797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720182+0000","last_change":"2024-04-17T08:31:32.589698+0000","last_active":"2024-04-17T08:31:33.720182+0000","last_peered":"2024-04-17T08:31:33.720182+0000","last_clean":"2024-04-17T08:31:33.720182+0000","last_became_active":"2024-04-17T08:31:32.589385+0000","last_became_peered":"2024-04-17T08:31:32.589385+0000","last_unstale":"2024-04-17T08:31:33.720182+0000","last_undegraded":"2024-04-17T08:31:33.720182+0000","last_fullsized":"2024-04-17T08:31:33.720182+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:20.820558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582336+0000","last_change":"2024-04-17T08:31:32.580879+0000","last_active":"2024-04-17T08:31:33.582336+0000","last_peered":"2024-04-17T08:31:33.582336+0000","last_clean":"2024-04-17T08:31:33.582336+0000","last_became_active":"2024-04-17T08:31:32.580496+0000","last_became_peered":"2024-04-17T08:31:32.580496+0000","last_unstale":"2024-04-17T08:31:33.582336+0000","last_undegraded":"2024-04-17T08:31:33.582336+0000","last_fullsized":"2024-04-17T08:31:33.582336+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:25.768435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579422+0000","last_change":"2024-04-17T08:31:32.576746+0000","last_active":"2024-04-17T08:31:33.579422+0000","last_peered":"2024-04-17T08:31:33.579422+0000","last_clean":"2024-04-17T08:31:33.579422+0000","last_became_active":"2024-04-17T08:31:32.576430+0000","last_became_peered":"2024-04-17T08:31:32.576430+0000","last_unstale":"2024-04-17T08:31:33.579422+0000","last_undegraded":"2024-04-17T08:31:33.579422+0000","last_fullsized":"2024-04-17T08:31:33.579422+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:37.968961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581350+0000","last_change":"2024-04-17T08:31:32.586999+0000","last_active":"2024-04-17T08:31:33.581350+0000","last_peered":"2024-04-17T08:31:33.581350+0000","last_clean":"2024-04-17T08:31:33.581350+0000","last_became_active":"2024-04-17T08:31:32.585585+0000","last_became_peered":"2024-04-17T08:31:32.585585+0000","last_unstale":"2024-04-17T08:31:33.581350+0000","last_undegraded":"2024-04-17T08:31:33.581350+0000","last_fullsized":"2024-04-17T08:31:33.581350+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:20:07.779308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586299+0000","last_change":"2024-04-17T08:31:32.584526+0000","last_active":"2024-04-17T08:31:33.586299+0000","last_peered":"2024-04-17T08:31:33.586299+0000","last_clean":"2024-04-17T08:31:33.586299+0000","last_became_active":"2024-04-17T08:31:32.581798+0000","last_became_peered":"2024-04-17T08:31:32.581798+0000","last_unstale":"2024-04-17T08:31:33.586299+0000","last_undegraded":"2024-04-17T08:31:33.586299+0000","last_fullsized":"2024-04-17T08:31:33.586299+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:26.700104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586205+0000","last_change":"2024-04-17T08:31:32.587319+0000","last_active":"2024-04-17T08:31:33.586205+0000","last_peered":"2024-04-17T08:31:33.586205+0000","last_clean":"2024-04-17T08:31:33.586205+0000","last_became_active":"2024-04-17T08:31:32.587046+0000","last_became_peered":"2024-04-17T08:31:32.587046+0000","last_unstale":"2024-04-17T08:31:33.586205+0000","last_undegraded":"2024-04-17T08:31:33.586205+0000","last_fullsized":"2024-04-17T08:31:33.586205+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:46:54.666276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.718985+0000","last_change":"2024-04-17T08:31:32.572171+0000","last_active":"2024-04-17T08:31:33.718985+0000","last_peered":"2024-04-17T08:31:33.718985+0000","last_clean":"2024-04-17T08:31:33.718985+0000","last_became_active":"2024-04-17T08:31:32.571857+0000","last_became_peered":"2024-04-17T08:31:32.571857+0000","last_unstale":"2024-04-17T08:31:33.718985+0000","last_undegraded":"2024-04-17T08:31:33.718985+0000","last_fullsized":"2024-04-17T08:31:33.718985+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:14.768835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720409+0000","last_change":"2024-04-17T08:31:32.587840+0000","last_active":"2024-04-17T08:31:33.720409+0000","last_peered":"2024-04-17T08:31:33.720409+0000","last_clean":"2024-04-17T08:31:33.720409+0000","last_became_active":"2024-04-17T08:31:32.586391+0000","last_became_peered":"2024-04-17T08:31:32.586391+0000","last_unstale":"2024-04-17T08:31:33.720409+0000","last_undegraded":"2024-04-17T08:31:33.720409+0000","last_fullsized":"2024-04-17T08:31:33.720409+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:38.102957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"194'4","reported_seq":18,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580048+0000","last_change":"2024-04-17T08:31:31.552503+0000","last_active":"2024-04-17T08:31:33.580048+0000","last_peered":"2024-04-17T08:31:33.580048+0000","last_clean":"2024-04-17T08:31:33.580048+0000","last_became_active":"2024-04-17T08:31:31.551312+0000","last_became_peered":"2024-04-17T08:31:31.551312+0000","last_unstale":"2024-04-17T08:31:33.580048+0000","last_undegraded":"2024-04-17T08:31:33.580048+0000","last_fullsized":"2024-04-17T08:31:33.580048+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:59.247863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586592+0000","last_change":"2024-04-17T08:31:32.585074+0000","last_active":"2024-04-17T08:31:33.586592+0000","last_peered":"2024-04-17T08:31:33.586592+0000","last_clean":"2024-04-17T08:31:33.586592+0000","last_became_active":"2024-04-17T08:31:32.581521+0000","last_became_peered":"2024-04-17T08:31:32.581521+0000","last_unstale":"2024-04-17T08:31:33.586592+0000","last_undegraded":"2024-04-17T08:31:33.586592+0000","last_fullsized":"2024-04-17T08:31:33.586592+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:29.865883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586055+0000","last_change":"2024-04-17T08:31:31.555944+0000","last_active":"2024-04-17T08:31:33.586055+0000","last_peered":"2024-04-17T08:31:33.586055+0000","last_clean":"2024-04-17T08:31:33.586055+0000","last_became_active":"2024-04-17T08:31:31.555218+0000","last_became_peered":"2024-04-17T08:31:31.555218+0000","last_unstale":"2024-04-17T08:31:33.586055+0000","last_undegraded":"2024-04-17T08:31:33.586055+0000","last_fullsized":"2024-04-17T08:31:33.586055+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:55.501648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722084+0000","last_change":"2024-04-17T08:31:32.582741+0000","last_active":"2024-04-17T08:31:33.722084+0000","last_peered":"2024-04-17T08:31:33.722084+0000","last_clean":"2024-04-17T08:31:33.722084+0000","last_became_active":"2024-04-17T08:31:32.582421+0000","last_became_peered":"2024-04-17T08:31:32.582421+0000","last_unstale":"2024-04-17T08:31:33.722084+0000","last_undegraded":"2024-04-17T08:31:33.722084+0000","last_fullsized":"2024-04-17T08:31:33.722084+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:12.223066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579581+0000","last_change":"2024-04-17T08:31:31.555455+0000","last_active":"2024-04-17T08:31:33.579581+0000","last_peered":"2024-04-17T08:31:33.579581+0000","last_clean":"2024-04-17T08:31:33.579581+0000","last_became_active":"2024-04-17T08:31:31.554972+0000","last_became_peered":"2024-04-17T08:31:31.554972+0000","last_unstale":"2024-04-17T08:31:33.579581+0000","last_undegraded":"2024-04-17T08:31:33.579581+0000","last_fullsized":"2024-04-17T08:31:33.579581+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:24.751908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582316+0000","last_change":"2024-04-17T08:31:32.573529+0000","last_active":"2024-04-17T08:31:33.582316+0000","last_peered":"2024-04-17T08:31:33.582316+0000","last_clean":"2024-04-17T08:31:33.582316+0000","last_became_active":"2024-04-17T08:31:32.573025+0000","last_became_peered":"2024-04-17T08:31:32.573025+0000","last_unstale":"2024-04-17T08:31:33.582316+0000","last_undegraded":"2024-04-17T08:31:33.582316+0000","last_fullsized":"2024-04-17T08:31:33.582316+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:37.518463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579790+0000","last_change":"2024-04-17T08:31:31.551040+0000","last_active":"2024-04-17T08:31:33.579790+0000","last_peered":"2024-04-17T08:31:33.579790+0000","last_clean":"2024-04-17T08:31:33.579790+0000","last_became_active":"2024-04-17T08:31:31.550364+0000","last_became_peered":"2024-04-17T08:31:31.550364+0000","last_unstale":"2024-04-17T08:31:33.579790+0000","last_undegraded":"2024-04-17T08:31:33.579790+0000","last_fullsized":"2024-04-17T08:31:33.579790+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:14.789121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719918+0000","last_change":"2024-04-17T08:31:32.580542+0000","last_active":"2024-04-17T08:31:33.719918+0000","last_peered":"2024-04-17T08:31:33.719918+0000","last_clean":"2024-04-17T08:31:33.719918+0000","last_became_active":"2024-04-17T08:31:32.580248+0000","last_became_peered":"2024-04-17T08:31:32.580248+0000","last_unstale":"2024-04-17T08:31:33.719918+0000","last_undegraded":"2024-04-17T08:31:33.719918+0000","last_fullsized":"2024-04-17T08:31:33.719918+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:56.318745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581474+0000","last_change":"2024-04-17T08:31:31.541315+0000","last_active":"2024-04-17T08:31:33.581474+0000","last_peered":"2024-04-17T08:31:33.581474+0000","last_clean":"2024-04-17T08:31:33.581474+0000","last_became_active":"2024-04-17T08:31:31.540855+0000","last_became_peered":"2024-04-17T08:31:31.540855+0000","last_unstale":"2024-04-17T08:31:33.581474+0000","last_undegraded":"2024-04-17T08:31:33.581474+0000","last_fullsized":"2024-04-17T08:31:33.581474+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:52.194608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579635+0000","last_change":"2024-04-17T08:31:32.582252+0000","last_active":"2024-04-17T08:31:33.579635+0000","last_peered":"2024-04-17T08:31:33.579635+0000","last_clean":"2024-04-17T08:31:33.579635+0000","last_became_active":"2024-04-17T08:31:32.581900+0000","last_became_peered":"2024-04-17T08:31:32.581900+0000","last_unstale":"2024-04-17T08:31:33.579635+0000","last_undegraded":"2024-04-17T08:31:33.579635+0000","last_fullsized":"2024-04-17T08:31:33.579635+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:01.007081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578673+0000","last_change":"2024-04-17T08:31:31.552258+0000","last_active":"2024-04-17T08:31:33.578673+0000","last_peered":"2024-04-17T08:31:33.578673+0000","last_clean":"2024-04-17T08:31:33.578673+0000","last_became_active":"2024-04-17T08:31:31.549114+0000","last_became_peered":"2024-04-17T08:31:31.549114+0000","last_unstale":"2024-04-17T08:31:33.578673+0000","last_undegraded":"2024-04-17T08:31:33.578673+0000","last_fullsized":"2024-04-17T08:31:33.578673+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:39.734270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578787+0000","last_change":"2024-04-17T08:31:32.576723+0000","last_active":"2024-04-17T08:31:33.578787+0000","last_peered":"2024-04-17T08:31:33.578787+0000","last_clean":"2024-04-17T08:31:33.578787+0000","last_became_active":"2024-04-17T08:31:32.576182+0000","last_became_peered":"2024-04-17T08:31:32.576182+0000","last_unstale":"2024-04-17T08:31:33.578787+0000","last_undegraded":"2024-04-17T08:31:33.578787+0000","last_fullsized":"2024-04-17T08:31:33.578787+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:14.735563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721567+0000","last_change":"2024-04-17T08:31:31.556646+0000","last_active":"2024-04-17T08:31:33.721567+0000","last_peered":"2024-04-17T08:31:33.721567+0000","last_clean":"2024-04-17T08:31:33.721567+0000","last_became_active":"2024-04-17T08:31:31.556379+0000","last_became_peered":"2024-04-17T08:31:31.556379+0000","last_unstale":"2024-04-17T08:31:33.721567+0000","last_undegraded":"2024-04-17T08:31:33.721567+0000","last_fullsized":"2024-04-17T08:31:33.721567+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:48.301913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722315+0000","last_change":"2024-04-17T08:31:32.583229+0000","last_active":"2024-04-17T08:31:33.722315+0000","last_peered":"2024-04-17T08:31:33.722315+0000","last_clean":"2024-04-17T08:31:33.722315+0000","last_became_active":"2024-04-17T08:31:32.582925+0000","last_became_peered":"2024-04-17T08:31:32.582925+0000","last_unstale":"2024-04-17T08:31:33.722315+0000","last_undegraded":"2024-04-17T08:31:33.722315+0000","last_fullsized":"2024-04-17T08:31:33.722315+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:16.578895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719387+0000","last_change":"2024-04-17T08:31:31.548659+0000","last_active":"2024-04-17T08:31:33.719387+0000","last_peered":"2024-04-17T08:31:33.719387+0000","last_clean":"2024-04-17T08:31:33.719387+0000","last_became_active":"2024-04-17T08:31:31.548287+0000","last_became_peered":"2024-04-17T08:31:31.548287+0000","last_unstale":"2024-04-17T08:31:33.719387+0000","last_undegraded":"2024-04-17T08:31:33.719387+0000","last_fullsized":"2024-04-17T08:31:33.719387+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:09.494648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720618+0000","last_change":"2024-04-17T08:31:32.570936+0000","last_active":"2024-04-17T08:31:33.720618+0000","last_peered":"2024-04-17T08:31:33.720618+0000","last_clean":"2024-04-17T08:31:33.720618+0000","last_became_active":"2024-04-17T08:31:32.570620+0000","last_became_peered":"2024-04-17T08:31:32.570620+0000","last_unstale":"2024-04-17T08:31:33.720618+0000","last_undegraded":"2024-04-17T08:31:33.720618+0000","last_fullsized":"2024-04-17T08:31:33.720618+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:23.756041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722569+0000","last_change":"2024-04-17T08:31:31.550450+0000","last_active":"2024-04-17T08:31:33.722569+0000","last_peered":"2024-04-17T08:31:33.722569+0000","last_clean":"2024-04-17T08:31:33.722569+0000","last_became_active":"2024-04-17T08:31:31.550156+0000","last_became_peered":"2024-04-17T08:31:31.550156+0000","last_unstale":"2024-04-17T08:31:33.722569+0000","last_undegraded":"2024-04-17T08:31:33.722569+0000","last_fullsized":"2024-04-17T08:31:33.722569+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:04.252568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719637+0000","last_change":"2024-04-17T08:31:32.587147+0000","last_active":"2024-04-17T08:31:33.719637+0000","last_peered":"2024-04-17T08:31:33.719637+0000","last_clean":"2024-04-17T08:31:33.719637+0000","last_became_active":"2024-04-17T08:31:32.586846+0000","last_became_peered":"2024-04-17T08:31:32.586846+0000","last_unstale":"2024-04-17T08:31:33.719637+0000","last_undegraded":"2024-04-17T08:31:33.719637+0000","last_fullsized":"2024-04-17T08:31:33.719637+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:16.055778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"194'3","reported_seq":17,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722363+0000","last_change":"2024-04-17T08:31:31.549648+0000","last_active":"2024-04-17T08:31:33.722363+0000","last_peered":"2024-04-17T08:31:33.722363+0000","last_clean":"2024-04-17T08:31:33.722363+0000","last_became_active":"2024-04-17T08:31:31.549347+0000","last_became_peered":"2024-04-17T08:31:31.549347+0000","last_unstale":"2024-04-17T08:31:33.722363+0000","last_undegraded":"2024-04-17T08:31:33.722363+0000","last_fullsized":"2024-04-17T08:31:33.722363+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:32:46.823813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586415+0000","last_change":"2024-04-17T08:31:32.587678+0000","last_active":"2024-04-17T08:31:33.586415+0000","last_peered":"2024-04-17T08:31:33.586415+0000","last_clean":"2024-04-17T08:31:33.586415+0000","last_became_active":"2024-04-17T08:31:32.581010+0000","last_became_peered":"2024-04-17T08:31:32.581010+0000","last_unstale":"2024-04-17T08:31:33.586415+0000","last_undegraded":"2024-04-17T08:31:33.586415+0000","last_fullsized":"2024-04-17T08:31:33.586415+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:04:58.912860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"194'6","reported_seq":20,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586099+0000","last_change":"2024-04-17T08:31:31.549787+0000","last_active":"2024-04-17T08:31:33.586099+0000","last_peered":"2024-04-17T08:31:33.586099+0000","last_clean":"2024-04-17T08:31:33.586099+0000","last_became_active":"2024-04-17T08:31:31.549474+0000","last_became_peered":"2024-04-17T08:31:31.549474+0000","last_unstale":"2024-04-17T08:31:33.586099+0000","last_undegraded":"2024-04-17T08:31:33.586099+0000","last_fullsized":"2024-04-17T08:31:33.586099+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:47:50.401531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719132+0000","last_change":"2024-04-17T08:31:32.580964+0000","last_active":"2024-04-17T08:31:33.719132+0000","last_peered":"2024-04-17T08:31:33.719132+0000","last_clean":"2024-04-17T08:31:33.719132+0000","last_became_active":"2024-04-17T08:31:32.578743+0000","last_became_peered":"2024-04-17T08:31:32.578743+0000","last_unstale":"2024-04-17T08:31:33.719132+0000","last_undegraded":"2024-04-17T08:31:33.719132+0000","last_fullsized":"2024-04-17T08:31:33.719132+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:21.524847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579178+0000","last_change":"2024-04-17T08:31:31.540943+0000","last_active":"2024-04-17T08:31:33.579178+0000","last_peered":"2024-04-17T08:31:33.579178+0000","last_clean":"2024-04-17T08:31:33.579178+0000","last_became_active":"2024-04-17T08:31:31.540643+0000","last_became_peered":"2024-04-17T08:31:31.540643+0000","last_unstale":"2024-04-17T08:31:33.579178+0000","last_undegraded":"2024-04-17T08:31:33.579178+0000","last_fullsized":"2024-04-17T08:31:33.579178+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:36.501936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719988+0000","last_change":"2024-04-17T08:31:32.584370+0000","last_active":"2024-04-17T08:31:33.719988+0000","last_peered":"2024-04-17T08:31:33.719988+0000","last_clean":"2024-04-17T08:31:33.719988+0000","last_became_active":"2024-04-17T08:31:32.583337+0000","last_became_peered":"2024-04-17T08:31:32.583337+0000","last_unstale":"2024-04-17T08:31:33.719988+0000","last_undegraded":"2024-04-17T08:31:33.719988+0000","last_fullsized":"2024-04-17T08:31:33.719988+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:35:23.577873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719370+0000","last_change":"2024-04-17T08:31:31.551462+0000","last_active":"2024-04-17T08:31:33.719370+0000","last_peered":"2024-04-17T08:31:33.719370+0000","last_clean":"2024-04-17T08:31:33.719370+0000","last_became_active":"2024-04-17T08:31:31.551172+0000","last_became_peered":"2024-04-17T08:31:31.551172+0000","last_unstale":"2024-04-17T08:31:33.719370+0000","last_undegraded":"2024-04-17T08:31:33.719370+0000","last_fullsized":"2024-04-17T08:31:33.719370+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:36.901245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578927+0000","last_change":"2024-04-17T08:31:32.576496+0000","last_active":"2024-04-17T08:31:33.578927+0000","last_peered":"2024-04-17T08:31:33.578927+0000","last_clean":"2024-04-17T08:31:33.578927+0000","last_became_active":"2024-04-17T08:31:32.575800+0000","last_became_peered":"2024-04-17T08:31:32.575800+0000","last_unstale":"2024-04-17T08:31:33.578927+0000","last_undegraded":"2024-04-17T08:31:33.578927+0000","last_fullsized":"2024-04-17T08:31:33.578927+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:16.901074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578747+0000","last_change":"2024-04-17T08:31:31.552383+0000","last_active":"2024-04-17T08:31:33.578747+0000","last_peered":"2024-04-17T08:31:33.578747+0000","last_clean":"2024-04-17T08:31:33.578747+0000","last_became_active":"2024-04-17T08:31:31.551698+0000","last_became_peered":"2024-04-17T08:31:31.551698+0000","last_unstale":"2024-04-17T08:31:33.578747+0000","last_undegraded":"2024-04-17T08:31:33.578747+0000","last_fullsized":"2024-04-17T08:31:33.578747+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:38.113476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580543+0000","last_change":"2024-04-17T08:31:32.579764+0000","last_active":"2024-04-17T08:31:33.580543+0000","last_peered":"2024-04-17T08:31:33.580543+0000","last_clean":"2024-04-17T08:31:33.580543+0000","last_became_active":"2024-04-17T08:31:32.579451+0000","last_became_peered":"2024-04-17T08:31:32.579451+0000","last_unstale":"2024-04-17T08:31:33.580543+0000","last_undegraded":"2024-04-17T08:31:33.580543+0000","last_fullsized":"2024-04-17T08:31:33.580543+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:26.861739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"194'8","reported_seq":22,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586134+0000","last_change":"2024-04-17T08:31:31.547977+0000","last_active":"2024-04-17T08:31:33.586134+0000","last_peered":"2024-04-17T08:31:33.586134+0000","last_clean":"2024-04-17T08:31:33.586134+0000","last_became_active":"2024-04-17T08:31:31.547687+0000","last_became_peered":"2024-04-17T08:31:31.547687+0000","last_unstale":"2024-04-17T08:31:33.586134+0000","last_undegraded":"2024-04-17T08:31:33.586134+0000","last_fullsized":"2024-04-17T08:31:33.586134+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:47.928198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580245+0000","last_change":"2024-04-17T08:31:32.582193+0000","last_active":"2024-04-17T08:31:33.580245+0000","last_peered":"2024-04-17T08:31:33.580245+0000","last_clean":"2024-04-17T08:31:33.580245+0000","last_became_active":"2024-04-17T08:31:32.576489+0000","last_became_peered":"2024-04-17T08:31:32.576489+0000","last_unstale":"2024-04-17T08:31:33.580245+0000","last_undegraded":"2024-04-17T08:31:33.580245+0000","last_fullsized":"2024-04-17T08:31:33.580245+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:49:03.558167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.582806+0000","last_change":"2024-04-17T08:31:31.553466+0000","last_active":"2024-04-17T08:31:33.582806+0000","last_peered":"2024-04-17T08:31:33.582806+0000","last_clean":"2024-04-17T08:31:33.582806+0000","last_became_active":"2024-04-17T08:31:31.552436+0000","last_became_peered":"2024-04-17T08:31:31.552436+0000","last_unstale":"2024-04-17T08:31:33.582806+0000","last_undegraded":"2024-04-17T08:31:33.582806+0000","last_fullsized":"2024-04-17T08:31:33.582806+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:19.182616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720768+0000","last_change":"2024-04-17T08:31:32.584297+0000","last_active":"2024-04-17T08:31:33.720768+0000","last_peered":"2024-04-17T08:31:33.720768+0000","last_clean":"2024-04-17T08:31:33.720768+0000","last_became_active":"2024-04-17T08:31:32.583850+0000","last_became_peered":"2024-04-17T08:31:32.583850+0000","last_unstale":"2024-04-17T08:31:33.720768+0000","last_undegraded":"2024-04-17T08:31:33.720768+0000","last_fullsized":"2024-04-17T08:31:33.720768+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:55.906822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581901+0000","last_change":"2024-04-17T08:31:31.538815+0000","last_active":"2024-04-17T08:31:33.581901+0000","last_peered":"2024-04-17T08:31:33.581901+0000","last_clean":"2024-04-17T08:31:33.581901+0000","last_became_active":"2024-04-17T08:31:31.538506+0000","last_became_peered":"2024-04-17T08:31:31.538506+0000","last_unstale":"2024-04-17T08:31:33.581901+0000","last_undegraded":"2024-04-17T08:31:33.581901+0000","last_fullsized":"2024-04-17T08:31:33.581901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:14.937481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579634+0000","last_change":"2024-04-17T08:31:32.582255+0000","last_active":"2024-04-17T08:31:33.579634+0000","last_peered":"2024-04-17T08:31:33.579634+0000","last_clean":"2024-04-17T08:31:33.579634+0000","last_became_active":"2024-04-17T08:31:32.581724+0000","last_became_peered":"2024-04-17T08:31:32.581724+0000","last_unstale":"2024-04-17T08:31:33.579634+0000","last_undegraded":"2024-04-17T08:31:33.579634+0000","last_fullsized":"2024-04-17T08:31:33.579634+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:44.241037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"194'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579830+0000","last_change":"2024-04-17T08:31:31.555888+0000","last_active":"2024-04-17T08:31:33.579830+0000","last_peered":"2024-04-17T08:31:33.579830+0000","last_clean":"2024-04-17T08:31:33.579830+0000","last_became_active":"2024-04-17T08:31:31.555599+0000","last_became_peered":"2024-04-17T08:31:31.555599+0000","last_unstale":"2024-04-17T08:31:33.579830+0000","last_undegraded":"2024-04-17T08:31:33.579830+0000","last_fullsized":"2024-04-17T08:31:33.579830+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:13.442598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719954+0000","last_change":"2024-04-17T08:31:32.589808+0000","last_active":"2024-04-17T08:31:33.719954+0000","last_peered":"2024-04-17T08:31:33.719954+0000","last_clean":"2024-04-17T08:31:33.719954+0000","last_became_active":"2024-04-17T08:31:32.589523+0000","last_became_peered":"2024-04-17T08:31:32.589523+0000","last_unstale":"2024-04-17T08:31:33.719954+0000","last_undegraded":"2024-04-17T08:31:33.719954+0000","last_fullsized":"2024-04-17T08:31:33.719954+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:43.489588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579882+0000","last_change":"2024-04-17T08:31:31.553573+0000","last_active":"2024-04-17T08:31:33.579882+0000","last_peered":"2024-04-17T08:31:33.579882+0000","last_clean":"2024-04-17T08:31:33.579882+0000","last_became_active":"2024-04-17T08:31:31.553133+0000","last_became_peered":"2024-04-17T08:31:31.553133+0000","last_unstale":"2024-04-17T08:31:33.579882+0000","last_undegraded":"2024-04-17T08:31:33.579882+0000","last_fullsized":"2024-04-17T08:31:33.579882+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:52.738963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.581007+0000","last_change":"2024-04-17T08:31:32.562089+0000","last_active":"2024-04-17T08:31:33.581007+0000","last_peered":"2024-04-17T08:31:33.581007+0000","last_clean":"2024-04-17T08:31:33.581007+0000","last_became_active":"2024-04-17T08:31:32.561659+0000","last_became_peered":"2024-04-17T08:31:32.561659+0000","last_unstale":"2024-04-17T08:31:33.581007+0000","last_undegraded":"2024-04-17T08:31:33.581007+0000","last_fullsized":"2024-04-17T08:31:33.581007+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:42:21.318610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579344+0000","last_change":"2024-04-17T08:31:31.552886+0000","last_active":"2024-04-17T08:31:33.579344+0000","last_peered":"2024-04-17T08:31:33.579344+0000","last_clean":"2024-04-17T08:31:33.579344+0000","last_became_active":"2024-04-17T08:31:31.551839+0000","last_became_peered":"2024-04-17T08:31:31.551839+0000","last_unstale":"2024-04-17T08:31:33.579344+0000","last_undegraded":"2024-04-17T08:31:33.579344+0000","last_fullsized":"2024-04-17T08:31:33.579344+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:56.307309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720133+0000","last_change":"2024-04-17T08:31:32.590670+0000","last_active":"2024-04-17T08:31:33.720133+0000","last_peered":"2024-04-17T08:31:33.720133+0000","last_clean":"2024-04-17T08:31:33.720133+0000","last_became_active":"2024-04-17T08:31:32.590394+0000","last_became_peered":"2024-04-17T08:31:32.590394+0000","last_unstale":"2024-04-17T08:31:33.720133+0000","last_undegraded":"2024-04-17T08:31:33.720133+0000","last_fullsized":"2024-04-17T08:31:33.720133+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:53.906218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579557+0000","last_change":"2024-04-17T08:31:31.554438+0000","last_active":"2024-04-17T08:31:33.579557+0000","last_peered":"2024-04-17T08:31:33.579557+0000","last_clean":"2024-04-17T08:31:33.579557+0000","last_became_active":"2024-04-17T08:31:31.552942+0000","last_became_peered":"2024-04-17T08:31:31.552942+0000","last_unstale":"2024-04-17T08:31:33.579557+0000","last_undegraded":"2024-04-17T08:31:33.579557+0000","last_fullsized":"2024-04-17T08:31:33.579557+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:42.076821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579280+0000","last_change":"2024-04-17T08:31:32.576687+0000","last_active":"2024-04-17T08:31:33.579280+0000","last_peered":"2024-04-17T08:31:33.579280+0000","last_clean":"2024-04-17T08:31:33.579280+0000","last_became_active":"2024-04-17T08:31:32.576372+0000","last_became_peered":"2024-04-17T08:31:32.576372+0000","last_unstale":"2024-04-17T08:31:33.579280+0000","last_undegraded":"2024-04-17T08:31:33.579280+0000","last_fullsized":"2024-04-17T08:31:33.579280+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:36:19.399635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579037+0000","last_change":"2024-04-17T08:31:31.544808+0000","last_active":"2024-04-17T08:31:33.579037+0000","last_peered":"2024-04-17T08:31:33.579037+0000","last_clean":"2024-04-17T08:31:33.579037+0000","last_became_active":"2024-04-17T08:31:31.544565+0000","last_became_peered":"2024-04-17T08:31:31.544565+0000","last_unstale":"2024-04-17T08:31:33.579037+0000","last_undegraded":"2024-04-17T08:31:33.579037+0000","last_fullsized":"2024-04-17T08:31:33.579037+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:22.223721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722346+0000","last_change":"2024-04-17T08:31:32.582664+0000","last_active":"2024-04-17T08:31:33.722346+0000","last_peered":"2024-04-17T08:31:33.722346+0000","last_clean":"2024-04-17T08:31:33.722346+0000","last_became_active":"2024-04-17T08:31:32.582200+0000","last_became_peered":"2024-04-17T08:31:32.582200+0000","last_unstale":"2024-04-17T08:31:33.722346+0000","last_undegraded":"2024-04-17T08:31:33.722346+0000","last_fullsized":"2024-04-17T08:31:33.722346+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:23.629367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579347+0000","last_change":"2024-04-17T08:31:31.555126+0000","last_active":"2024-04-17T08:31:33.579347+0000","last_peered":"2024-04-17T08:31:33.579347+0000","last_clean":"2024-04-17T08:31:33.579347+0000","last_became_active":"2024-04-17T08:31:31.554836+0000","last_became_peered":"2024-04-17T08:31:31.554836+0000","last_unstale":"2024-04-17T08:31:33.579347+0000","last_undegraded":"2024-04-17T08:31:33.579347+0000","last_fullsized":"2024-04-17T08:31:33.579347+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:42:29.890915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720606+0000","last_change":"2024-04-17T08:31:32.584094+0000","last_active":"2024-04-17T08:31:33.720606+0000","last_peered":"2024-04-17T08:31:33.720606+0000","last_clean":"2024-04-17T08:31:33.720606+0000","last_became_active":"2024-04-17T08:31:32.581409+0000","last_became_peered":"2024-04-17T08:31:32.581409+0000","last_unstale":"2024-04-17T08:31:33.720606+0000","last_undegraded":"2024-04-17T08:31:33.720606+0000","last_fullsized":"2024-04-17T08:31:33.720606+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:30.424287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"194'2","reported_seq":16,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720871+0000","last_change":"2024-04-17T08:31:31.552940+0000","last_active":"2024-04-17T08:31:33.720871+0000","last_peered":"2024-04-17T08:31:33.720871+0000","last_clean":"2024-04-17T08:31:33.720871+0000","last_became_active":"2024-04-17T08:31:31.552640+0000","last_became_peered":"2024-04-17T08:31:31.552640+0000","last_unstale":"2024-04-17T08:31:33.720871+0000","last_undegraded":"2024-04-17T08:31:33.720871+0000","last_fullsized":"2024-04-17T08:31:33.720871+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:46.859215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720816+0000","last_change":"2024-04-17T08:31:32.586133+0000","last_active":"2024-04-17T08:31:33.720816+0000","last_peered":"2024-04-17T08:31:33.720816+0000","last_clean":"2024-04-17T08:31:33.720816+0000","last_became_active":"2024-04-17T08:31:32.585843+0000","last_became_peered":"2024-04-17T08:31:32.585843+0000","last_unstale":"2024-04-17T08:31:33.720816+0000","last_undegraded":"2024-04-17T08:31:33.720816+0000","last_fullsized":"2024-04-17T08:31:33.720816+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:49.200424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"195'1","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.585247+0000","last_change":"2024-04-17T08:31:31.546420+0000","last_active":"2024-04-17T08:31:33.585247+0000","last_peered":"2024-04-17T08:31:33.585247+0000","last_clean":"2024-04-17T08:31:33.585247+0000","last_became_active":"2024-04-17T08:31:31.540406+0000","last_became_peered":"2024-04-17T08:31:31.540406+0000","last_unstale":"2024-04-17T08:31:33.585247+0000","last_undegraded":"2024-04-17T08:31:33.585247+0000","last_fullsized":"2024-04-17T08:31:33.585247+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:16.400806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722119+0000","last_change":"2024-04-17T08:31:32.585635+0000","last_active":"2024-04-17T08:31:33.722119+0000","last_peered":"2024-04-17T08:31:33.722119+0000","last_clean":"2024-04-17T08:31:33.722119+0000","last_became_active":"2024-04-17T08:31:32.585340+0000","last_became_peered":"2024-04-17T08:31:32.585340+0000","last_unstale":"2024-04-17T08:31:33.722119+0000","last_undegraded":"2024-04-17T08:31:33.722119+0000","last_fullsized":"2024-04-17T08:31:33.722119+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:52.188712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.722310+0000","last_change":"2024-04-17T08:31:31.553569+0000","last_active":"2024-04-17T08:31:33.722310+0000","last_peered":"2024-04-17T08:31:33.722310+0000","last_clean":"2024-04-17T08:31:33.722310+0000","last_became_active":"2024-04-17T08:31:31.551799+0000","last_became_peered":"2024-04-17T08:31:31.551799+0000","last_unstale":"2024-04-17T08:31:33.722310+0000","last_undegraded":"2024-04-17T08:31:33.722310+0000","last_fullsized":"2024-04-17T08:31:33.722310+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:44.852505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720181+0000","last_change":"2024-04-17T08:31:32.565160+0000","last_active":"2024-04-17T08:31:33.720181+0000","last_peered":"2024-04-17T08:31:33.720181+0000","last_clean":"2024-04-17T08:31:33.720181+0000","last_became_active":"2024-04-17T08:31:32.562882+0000","last_became_peered":"2024-04-17T08:31:32.562882+0000","last_unstale":"2024-04-17T08:31:33.720181+0000","last_undegraded":"2024-04-17T08:31:33.720181+0000","last_fullsized":"2024-04-17T08:31:33.720181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:08.659018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720435+0000","last_change":"2024-04-17T08:31:31.539426+0000","last_active":"2024-04-17T08:31:33.720435+0000","last_peered":"2024-04-17T08:31:33.720435+0000","last_clean":"2024-04-17T08:31:33.720435+0000","last_became_active":"2024-04-17T08:31:31.539004+0000","last_became_peered":"2024-04-17T08:31:31.539004+0000","last_unstale":"2024-04-17T08:31:33.720435+0000","last_undegraded":"2024-04-17T08:31:33.720435+0000","last_fullsized":"2024-04-17T08:31:33.720435+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:16:59.690825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719573+0000","last_change":"2024-04-17T08:31:32.572245+0000","last_active":"2024-04-17T08:31:33.719573+0000","last_peered":"2024-04-17T08:31:33.719573+0000","last_clean":"2024-04-17T08:31:33.719573+0000","last_became_active":"2024-04-17T08:31:32.571929+0000","last_became_peered":"2024-04-17T08:31:32.571929+0000","last_unstale":"2024-04-17T08:31:33.719573+0000","last_undegraded":"2024-04-17T08:31:33.719573+0000","last_fullsized":"2024-04-17T08:31:33.719573+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:06.829939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.585929+0000","last_change":"2024-04-17T08:31:31.543062+0000","last_active":"2024-04-17T08:31:33.585929+0000","last_peered":"2024-04-17T08:31:33.585929+0000","last_clean":"2024-04-17T08:31:33.585929+0000","last_became_active":"2024-04-17T08:31:31.540387+0000","last_became_peered":"2024-04-17T08:31:31.540387+0000","last_unstale":"2024-04-17T08:31:33.585929+0000","last_undegraded":"2024-04-17T08:31:33.585929+0000","last_fullsized":"2024-04-17T08:31:33.585929+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:50.086505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586482+0000","last_change":"2024-04-17T08:31:32.585039+0000","last_active":"2024-04-17T08:31:33.586482+0000","last_peered":"2024-04-17T08:31:33.586482+0000","last_clean":"2024-04-17T08:31:33.586482+0000","last_became_active":"2024-04-17T08:31:32.584748+0000","last_became_peered":"2024-04-17T08:31:32.584748+0000","last_unstale":"2024-04-17T08:31:33.586482+0000","last_undegraded":"2024-04-17T08:31:33.586482+0000","last_fullsized":"2024-04-17T08:31:33.586482+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:34:03.496325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.578686+0000","last_change":"2024-04-17T08:31:31.548504+0000","last_active":"2024-04-17T08:31:33.578686+0000","last_peered":"2024-04-17T08:31:33.578686+0000","last_clean":"2024-04-17T08:31:33.578686+0000","last_became_active":"2024-04-17T08:31:31.546576+0000","last_became_peered":"2024-04-17T08:31:31.546576+0000","last_unstale":"2024-04-17T08:31:33.578686+0000","last_undegraded":"2024-04-17T08:31:33.578686+0000","last_fullsized":"2024-04-17T08:31:33.578686+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:38.545887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.719808+0000","last_change":"2024-04-17T08:31:32.592976+0000","last_active":"2024-04-17T08:31:33.719808+0000","last_peered":"2024-04-17T08:31:33.719808+0000","last_clean":"2024-04-17T08:31:33.719808+0000","last_became_active":"2024-04-17T08:31:32.592664+0000","last_became_peered":"2024-04-17T08:31:32.592664+0000","last_unstale":"2024-04-17T08:31:33.719808+0000","last_undegraded":"2024-04-17T08:31:33.719808+0000","last_fullsized":"2024-04-17T08:31:33.719808+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:44.800190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.721865+0000","last_change":"2024-04-17T08:31:31.545445+0000","last_active":"2024-04-17T08:31:33.721865+0000","last_peered":"2024-04-17T08:31:33.721865+0000","last_clean":"2024-04-17T08:31:33.721865+0000","last_became_active":"2024-04-17T08:31:31.545158+0000","last_became_peered":"2024-04-17T08:31:31.545158+0000","last_unstale":"2024-04-17T08:31:33.721865+0000","last_undegraded":"2024-04-17T08:31:33.721865+0000","last_fullsized":"2024-04-17T08:31:33.721865+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:03.597001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.580707+0000","last_change":"2024-04-17T08:31:32.570700+0000","last_active":"2024-04-17T08:31:33.580707+0000","last_peered":"2024-04-17T08:31:33.580707+0000","last_clean":"2024-04-17T08:31:33.580707+0000","last_became_active":"2024-04-17T08:31:32.570362+0000","last_became_peered":"2024-04-17T08:31:32.570362+0000","last_unstale":"2024-04-17T08:31:33.580707+0000","last_undegraded":"2024-04-17T08:31:33.580707+0000","last_fullsized":"2024-04-17T08:31:33.580707+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:40.337116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586292+0000","last_change":"2024-04-17T08:31:31.532073+0000","last_active":"2024-04-17T08:31:33.586292+0000","last_peered":"2024-04-17T08:31:33.586292+0000","last_clean":"2024-04-17T08:31:33.586292+0000","last_became_active":"2024-04-17T08:31:31.531233+0000","last_became_peered":"2024-04-17T08:31:31.531233+0000","last_unstale":"2024-04-17T08:31:33.586292+0000","last_undegraded":"2024-04-17T08:31:33.586292+0000","last_fullsized":"2024-04-17T08:31:33.586292+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:15.473555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.720410+0000","last_change":"2024-04-17T08:31:32.585600+0000","last_active":"2024-04-17T08:31:33.720410+0000","last_peered":"2024-04-17T08:31:33.720410+0000","last_clean":"2024-04-17T08:31:33.720410+0000","last_became_active":"2024-04-17T08:31:32.583740+0000","last_became_peered":"2024-04-17T08:31:32.583740+0000","last_unstale":"2024-04-17T08:31:33.720410+0000","last_undegraded":"2024-04-17T08:31:33.720410+0000","last_fullsized":"2024-04-17T08:31:33.720410+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:31.507047+0000","last_clean_scrub_stamp":"2024-04-17T08:31:31.507047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:12.524053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.586352+0000","last_change":"2024-04-17T08:31:31.548777+0000","last_active":"2024-04-17T08:31:33.586352+0000","last_peered":"2024-04-17T08:31:33.586352+0000","last_clean":"2024-04-17T08:31:33.586352+0000","last_became_active":"2024-04-17T08:31:31.548505+0000","last_became_peered":"2024-04-17T08:31:31.548505+0000","last_unstale":"2024-04-17T08:31:33.586352+0000","last_undegraded":"2024-04-17T08:31:33.586352+0000","last_fullsized":"2024-04-17T08:31:33.586352+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:31:30.490520+0000","last_clean_scrub_stamp":"2024-04-17T08:31:30.490520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:13.719036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"184'299","reported_seq":1176,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-04-17T08:31:33.579411+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:31:33.579411+0000","last_peered":"2024-04-17T08:31:33.579411+0000","last_clean":"2024-04-17T08:31:33.579411+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:31:33.579411+0000","last_undegraded":"2024-04-17T08:31:33.579411+0000","last_fullsized":"2024-04-17T08:31:33.579411+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":487,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1698,"num_read_kb":3560,"num_write":487,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543069,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48048,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323792,"statfs":{"total":96636764160,"available":96587563008,"internally_reserved":0,"allocated":3051520,"data_stored":1775198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2437120,"data_stored":1182498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543068,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41892,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329948,"statfs":{"total":96636764160,"available":96593866752,"internally_reserved":0,"allocated":2441216,"data_stored":1183080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543068,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55780,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316060,"statfs":{"total":96636764160,"available":96579645440,"internally_reserved":0,"allocated":2449408,"data_stored":1184158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543070,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64092,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307748,"statfs":{"total":96636764160,"available":96571133952,"internally_reserved":0,"allocated":2437120,"data_stored":1182510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543069,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47448,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324392,"statfs":{"total":96636764160,"available":96588177408,"internally_reserved":0,"allocated":2437120,"data_stored":1182566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543070,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59936,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311904,"statfs":{"total":96636764160,"available":96575389696,"internally_reserved":0,"allocated":2437120,"data_stored":1182510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3026944,"data_stored":1772844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:31:37.602 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:31:37.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:31:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.870+0000 7f707e600640 1 Processor -- start 2024-04-17T08:31:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.870+0000 7f707e600640 1 -- start start 2024-04-17T08:31:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078160d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70780cd060 0x7f7078071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70780cd4a0 0x7f7078072460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7078146530 con 0x7f70780cd4a0 2024-04-17T08:31:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7078161240 con 0x7f70780cd060 2024-04-17T08:31:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f707e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70780cda40 con 0x7f7078146150 2024-04-17T08:31:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f7077e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70780cd4a0 0x7f7078072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f7077e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70780cd4a0 0x7f7078072460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46762/0 (socket says 172.21.15.67:46762) 2024-04-17T08:31:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.872+0000 7f7077e00640 1 -- 172.21.15.67:0/152974392 learned_addr learned my addr 172.21.15.67:0/152974392 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077400640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7076a00640 1 --2- 172.21.15.67:0/152974392 >> v2:172.21.15.187:3300/0 conn(0x7f70780cd060 0x7f7078071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077e00640 1 -- 172.21.15.67:0/152974392 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 msgr2=0x7f7078160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077e00640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077e00640 1 -- 172.21.15.67:0/152974392 >> v2:172.21.15.187:3300/0 conn(0x7f70780cd060 msgr2=0x7f7078071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077e00640 1 --2- 172.21.15.67:0/152974392 >> v2:172.21.15.187:3300/0 conn(0x7f70780cd060 0x7f7078071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.873+0000 7f7077e00640 1 -- 172.21.15.67:0/152974392 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70780cdd20 con 0x7f70780cd4a0 2024-04-17T08:31:37.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.874+0000 7f7077e00640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd4a0 0x7f7078072460 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f70780b5be0 tx=0x7f707002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b379fceb4f75fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.874+0000 7f7076000640 1 -- 172.21.15.67:0/152974392 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707002c9e0 con 0x7f70780cd4a0 2024-04-17T08:31:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.874+0000 7f7076000640 1 -- 172.21.15.67:0/152974392 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707002cb80 con 0x7f70780cd4a0 2024-04-17T08:31:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.874+0000 7f7076000640 1 -- 172.21.15.67:0/152974392 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700343f0 con 0x7f70780cd4a0 2024-04-17T08:31:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 -- 172.21.15.67:0/152974392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd4a0 msgr2=0x7f7078072460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd4a0 0x7f7078072460 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f70780b5be0 tx=0x7f707002be40 comp rx=0 tx=0).stop 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 -- 172.21.15.67:0/152974392 shutdown_connections 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd4a0 0x7f7078072460 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 --2- 172.21.15.67:0/152974392 >> v2:172.21.15.187:3300/0 conn(0x7f70780cd060 0x7f7078071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 --2- 172.21.15.67:0/152974392 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 -- 172.21.15.67:0/152974392 >> 172.21.15.67:0/152974392 conn(0x7f7078067c30 msgr2=0x7f707806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 -- 172.21.15.67:0/152974392 shutdown_connections 2024-04-17T08:31:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.875+0000 7f707e600640 1 -- 172.21.15.67:0/152974392 wait complete. 2024-04-17T08:31:37.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.877+0000 7f707e600640 1 Processor -- start 2024-04-17T08:31:37.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.877+0000 7f707e600640 1 -- start start 2024-04-17T08:31:37.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70780cd4a0 0x7f7078143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f707806f310 con 0x7f70780cd060 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70781607e0 con 0x7f70780cd4a0 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.878+0000 7f707e600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f707806f490 con 0x7f7078146150 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46774/0 (socket says 172.21.15.67:46774) 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 -- 172.21.15.67:0/378547266 learned_addr learned my addr 172.21.15.67:0/378547266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 -- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 msgr2=0x7f7078143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 -- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70780cd4a0 msgr2=0x7f7078143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70780cd4a0 0x7f7078143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 -- 172.21.15.67:0/378547266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70780c9140 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.879+0000 7f7077400640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f706800d830 tx=0x7f706800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.880+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706800b6c0 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.880+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7068019070 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.880+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706800ba60 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.880+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70780c9420 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.880+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70780b4fc0 con 0x7f70780cd060 2024-04-17T08:31:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.881+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7078068ca0 con 0x7f70780cd060 2024-04-17T08:31:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.882+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f706801e460 con 0x7f70780cd060 2024-04-17T08:31:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.883+0000 7f706f400640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 0x7f705806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.883+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f706808c3d0 con 0x7f70780cd060 2024-04-17T08:31:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.884+0000 7f7076a00640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 0x7f705806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.885+0000 7f7076a00640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 0x7f705806ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7064006a90 tx=0x7f7064008040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:37.888+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f706805a6c0 con 0x7f70780cd060 2024-04-17T08:31:38.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.172+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f70780b4b80 con 0x7f70780cd060 2024-04-17T08:31:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.177+0000 7f706f400640 1 -- 172.21.15.67:0/378547266 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) ==== 190+0+147 (secure 0 0 0) 0x7f7068059fc0 con 0x7f70780cd060 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:31:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 msgr2=0x7f705806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 0x7f705806ef00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7064006a90 tx=0x7f7064008040 comp rx=0 tx=0).stop 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 msgr2=0x7f70780b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f706800d830 tx=0x7f706800dcf0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 shutdown_connections 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7078146150 0x7f7078143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f705806ca40 0x7f705806ef00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70780cd4a0 0x7f7078143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 --2- 172.21.15.67:0/378547266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70780cd060 0x7f70780b4610 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 >> 172.21.15.67:0/378547266 conn(0x7f7078067c30 msgr2=0x7f7078145340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 shutdown_connections 2024-04-17T08:31:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.180+0000 7f707e600640 1 -- 172.21.15.67:0/378547266 wait complete. 2024-04-17T08:31:38.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:31:38.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.421+0000 7f6dbf000640 1 Processor -- start 2024-04-17T08:31:38.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.421+0000 7f6dbf000640 1 -- start start 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbf000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6db8160280 0x7f6db806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbf000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6db8141ed0 0x7f6db8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbf000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db80cd4a0 con 0x7f6db806b8c0 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbf000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6db815ce30 con 0x7f6db8160280 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbf000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6db81607f0 con 0x7f6db8141ed0 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6db8160280 0x7f6db806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6db8160280 0x7f6db806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46790/0 (socket says 172.21.15.67:46790) 2024-04-17T08:31:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbd200640 1 -- 172.21.15.67:0/3111990978 learned_addr learned my addr 172.21.15.67:0/3111990978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.423+0000 7f6dbe600640 1 --2- 172.21.15.67:0/3111990978 >> v2:172.21.15.187:3300/0 conn(0x7f6db8141ed0 0x7f6db8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.424+0000 7f6dbd200640 1 -- 172.21.15.67:0/3111990978 >> v2:172.21.15.187:3300/0 conn(0x7f6db8141ed0 msgr2=0x7f6db8142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.424+0000 7f6dbd200640 1 --2- 172.21.15.67:0/3111990978 >> v2:172.21.15.187:3300/0 conn(0x7f6db8141ed0 0x7f6db8142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.424+0000 7f6dbd200640 1 -- 172.21.15.67:0/3111990978 >> v1:172.21.15.187:6789/0 conn(0x7f6db806b8c0 legacy=0x7f6db80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.424+0000 7f6dbd200640 1 -- 172.21.15.67:0/3111990978 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db8160ad0 con 0x7f6db8160280 2024-04-17T08:31:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.425+0000 7f6dbd200640 1 --2- 172.21.15.67:0/3111990978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8160280 0x7f6db806bca0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f6db80b2120 tx=0x7f6da402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b25b6e5457fc358 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.425+0000 7f6db7e00640 1 -- 172.21.15.67:0/3111990978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4035020 con 0x7f6db8160280 2024-04-17T08:31:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.425+0000 7f6db7e00640 1 -- 172.21.15.67:0/3111990978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da4031870 con 0x7f6db8160280 2024-04-17T08:31:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.425+0000 7f6db7e00640 1 -- 172.21.15.67:0/3111990978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4031c40 con 0x7f6db8160280 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 -- 172.21.15.67:0/3111990978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8160280 msgr2=0x7f6db806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 --2- 172.21.15.67:0/3111990978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8160280 0x7f6db806bca0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f6db80b2120 tx=0x7f6da402b910 comp rx=0 tx=0).stop 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 -- 172.21.15.67:0/3111990978 shutdown_connections 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 --2- 172.21.15.67:0/3111990978 >> v2:172.21.15.187:3300/0 conn(0x7f6db8141ed0 0x7f6db8142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 --2- 172.21.15.67:0/3111990978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8160280 0x7f6db806bca0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 -- 172.21.15.67:0/3111990978 >> 172.21.15.67:0/3111990978 conn(0x7f6db8067d10 msgr2=0x7f6db80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 -- 172.21.15.67:0/3111990978 shutdown_connections 2024-04-17T08:31:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.426+0000 7f6dbf000640 1 -- 172.21.15.67:0/3111990978 wait complete. 2024-04-17T08:31:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.428+0000 7f6dbf000640 1 Processor -- start 2024-04-17T08:31:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.428+0000 7f6dbf000640 1 -- start start 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 0x7f6db8155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6db8160280 0x7f6db815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8161090 con 0x7f6db8141ed0 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6db806c1e0 con 0x7f6db806b8c0 2024-04-17T08:31:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbf000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6db813f9b0 con 0x7f6db8160280 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46798/0 (socket says 172.21.15.67:46798) 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 -- 172.21.15.67:0/2337570130 learned_addr learned my addr 172.21.15.67:0/2337570130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbdc00640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 0x7f6db8155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 -- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6db8160280 msgr2=0x7f6db815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbe600640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6db8160280 0x7f6db815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6db8160280 0x7f6db815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.429+0000 7f6dbd200640 1 -- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 msgr2=0x7f6db8155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbd200640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 0x7f6db8155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbd200640 1 -- 172.21.15.67:0/2337570130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db815dbc0 con 0x7f6db8141ed0 2024-04-17T08:31:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbdc00640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 0x7f6db8155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbd200640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f6da402b950 tx=0x7f6da40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4040070 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da4030c80 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4035040 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db815de30 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.430+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db815e3f0 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.432+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db8141cf0 con 0x7f6db8141ed0 2024-04-17T08:31:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.433+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f6da4038020 con 0x7f6db8141ed0 2024-04-17T08:31:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.434+0000 7f6db5600640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 0x7f6d9806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.434+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f6da40adb70 con 0x7f6db8141ed0 2024-04-17T08:31:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.435+0000 7f6dbdc00640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 0x7f6d9806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:38.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.436+0000 7f6dbdc00640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 0x7f6d9806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6dac003eb0 tx=0x7f6dac009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:38.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.439+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6db8141cf0 con 0x7f6db8141ed0 2024-04-17T08:31:38.499 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-17T08:31:38.501+0000 7f678f000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:31:38.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.682+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db80cd0a0 con 0x7f6db8141ed0 2024-04-17T08:31:38.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.683+0000 7f6db5600640 1 -- 172.21.15.67:0/2337570130 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7f6db80cd0a0 con 0x7f6db8141ed0 2024-04-17T08:31:38.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:31:38.682 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:31:38.682 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:31:38.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 msgr2=0x7f6d9806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 0x7f6d9806eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6dac003eb0 tx=0x7f6dac009040 comp rx=0 tx=0).stop 2024-04-17T08:31:38.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 msgr2=0x7f6db8145840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:38.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f6da402b950 tx=0x7f6da40046c0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 shutdown_connections 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6db8160280 0x7f6db815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6db8141ed0 0x7f6db8145840 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f6d9806ca10 0x7f6d9806eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 --2- 172.21.15.67:0/2337570130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6db806b8c0 0x7f6db8155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 >> 172.21.15.67:0/2337570130 conn(0x7f6db8067d10 msgr2=0x7f6db81415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 shutdown_connections 2024-04-17T08:31:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:38.686+0000 7f6dbf000640 1 -- 172.21.15.67:0/2337570130 wait complete. 2024-04-17T08:31:39.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:31:32.003013+0000', 'modified': '2024-04-17T08:31:35.101530+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9987': {'gid': 9987, 'name': 'd', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6837/3432159492', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6835', 'nonce': 3432159492}, {'type': 'v1', 'addr': '172.21.15.187:6837', 'nonce': 3432159492}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9987, 'qdb_cluster': [9987]} 2024-04-17T08:31:39.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:31:39.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T08:31:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.339+0000 7f4dbdc00640 1 Processor -- start 2024-04-17T08:31:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.340+0000 7f4dbdc00640 1 -- start start 2024-04-17T08:31:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db80d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbdc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4db80d3350 0x7f4db80b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbdc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db80d4140 con 0x7f4db815ee20 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbdc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4db80b83b0 con 0x7f4db80d3350 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4db80d3b80 con 0x7f4db80d1ab0 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.341+0000 7f4dbd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4db80d3350 0x7f4db80b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4db80d3350 0x7f4db80b8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46808/0 (socket says 172.21.15.67:46808) 2024-04-17T08:31:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 -- 172.21.15.67:0/2119957213 learned_addr learned my addr 172.21.15.67:0/2119957213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4db7400640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db80d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 -- 172.21.15.67:0/2119957213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 msgr2=0x7f4db80d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db80d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 -- 172.21.15.67:0/2119957213 >> v1:172.21.15.67:6789/0 conn(0x7f4db815ee20 legacy=0x7f4db80d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.342+0000 7f4dbd200640 1 -- 172.21.15.67:0/2119957213 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db80d3e60 con 0x7f4db80d3350 2024-04-17T08:31:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.343+0000 7f4dbd200640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80d3350 0x7f4db80b8840 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009870 tx=0x7f4db002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5845243ee5bff29 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4db6a00640 1 -- 172.21.15.67:0/2119957213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0034070 con 0x7f4db80d3350 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4db6a00640 1 -- 172.21.15.67:0/2119957213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db0030840 con 0x7f4db80d3350 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4db6a00640 1 -- 172.21.15.67:0/2119957213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0030c10 con 0x7f4db80d3350 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 -- 172.21.15.67:0/2119957213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80d3350 msgr2=0x7f4db80b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80d3350 0x7f4db80b8840 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009870 tx=0x7f4db002b980 comp rx=0 tx=0).stop 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 -- 172.21.15.67:0/2119957213 shutdown_connections 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db80d3350 0x7f4db80b8840 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/2119957213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db80d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 -- 172.21.15.67:0/2119957213 >> 172.21.15.67:0/2119957213 conn(0x7f4db806d3c0 msgr2=0x7f4db80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.344+0000 7f4dbdc00640 1 -- 172.21.15.67:0/2119957213 shutdown_connections 2024-04-17T08:31:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.345+0000 7f4dbdc00640 1 -- 172.21.15.67:0/2119957213 wait complete. 2024-04-17T08:31:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.346+0000 7f4dbdc00640 1 Processor -- start 2024-04-17T08:31:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.346+0000 7f4dbdc00640 1 -- start start 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db814f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4db80d3350 0x7f4db80cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4db80d3800 con 0x7f4db815ee20 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4db815f200 con 0x7f4db80d3350 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.347+0000 7f4dbdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4db80b8d80 con 0x7f4db80d1ab0 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46812/0 (socket says 172.21.15.67:46812) 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 -- 172.21.15.67:0/3129611756 learned_addr learned my addr 172.21.15.67:0/3129611756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4db7400640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4db80d3350 0x7f4db80cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db814f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 -- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 msgr2=0x7f4db814f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db814f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 -- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4db80d3350 msgr2=0x7f4db80cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4db80d3350 0x7f4db80cdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 -- 172.21.15.67:0/3129611756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db814fb70 con 0x7f4db815ee20 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.348+0000 7f4dbd200640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4db002b9c0 tx=0x7f4db0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.349+0000 7f4dafe00640 1 -- 172.21.15.67:0/3129611756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0048070 con 0x7f4db815ee20 2024-04-17T08:31:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.349+0000 7f4dafe00640 1 -- 172.21.15.67:0/3129611756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db0030640 con 0x7f4db815ee20 2024-04-17T08:31:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.349+0000 7f4dafe00640 1 -- 172.21.15.67:0/3129611756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0034040 con 0x7f4db815ee20 2024-04-17T08:31:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.349+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db814fe50 con 0x7f4db815ee20 2024-04-17T08:31:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.349+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db8150410 con 0x7f4db815ee20 2024-04-17T08:31:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.351+0000 7f4dafe00640 1 -- 172.21.15.67:0/3129611756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f4db004b070 con 0x7f4db815ee20 2024-04-17T08:31:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.353+0000 7f4dafe00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 0x7f4d9806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.353+0000 7f4dafe00640 1 -- 172.21.15.67:0/3129611756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f4db00adec0 con 0x7f4db815ee20 2024-04-17T08:31:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.354+0000 7f4dc1e1d740 1 Processor -- start 2024-04-17T08:31:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.354+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 0x7f4d9806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.354+0000 7f4dc1e1d740 1 -- start start 2024-04-17T08:31:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 0x559242ec1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559242dfd980 con 0x559242e956a0 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x559242da4220 con 0x559242e91b40 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x559242df9940 con 0x559242edf620 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46822/0 (socket says 172.21.15.67:46822) 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4db7400640 1 -- 172.21.15.67:0/1203765218 learned_addr learned my addr 172.21.15.67:0/1203765218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 0x7f4d9806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4da80015e0 tx=0x7f4da8009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4db7e00640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.355+0000 7f4dbd200640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 0x559242ec1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 -- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 msgr2=0x559242e96c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 -- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 msgr2=0x559242ec1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 0x559242ec1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 -- 172.21.15.67:0/1203765218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559242b102d0 con 0x559242e956a0 2024-04-17T08:31:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4dbd200640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 0x559242ec1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4db7400640 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009fc0 tx=0x7f4da40077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaa442b9e1cf92 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4dacc00640 1 -- 172.21.15.67:0/1203765218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4013840 con 0x559242e956a0 2024-04-17T08:31:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4dacc00640 1 -- 172.21.15.67:0/1203765218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da4019070 con 0x559242e956a0 2024-04-17T08:31:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.356+0000 7f4dacc00640 1 -- 172.21.15.67:0/1203765218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4013c20 con 0x559242e956a0 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1203765218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 msgr2=0x559242e97190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009fc0 tx=0x7f4da40077c0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1203765218 shutdown_connections 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242e91b40 0x559242ec1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242e97190 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1203765218 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1203765218 >> 172.21.15.67:0/1203765218 conn(0x559242e93db0 msgr2=0x559242edc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1203765218 shutdown_connections 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.357+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1203765218 wait complete. 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.358+0000 7f4dc1e1d740 1 -- start start 2024-04-17T08:31:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.358+0000 7f4dc1e1d740 1 Processor -- start 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242ede5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242ea9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559242e95bb0 con 0x559242edf620 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x559242db7590 con 0x559242eb7df0 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x559242e95e90 con 0x559242e956a0 2024-04-17T08:31:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.359+0000 7f4dc1e1d740 10 client.0 _ll_register_callbacks cb 0x559242b0fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46828/0 (socket says 172.21.15.67:46828) 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 -- 172.21.15.67:0/3374116053 learned_addr learned my addr 172.21.15.67:0/3374116053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 -- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 msgr2=0x559242ede5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242ede5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 -- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 msgr2=0x559242ea9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242ea9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 -- 172.21.15.67:0/3374116053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559242bda6b0 con 0x559242edf620 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4db7400640 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x559242e91ff0 tx=0x7f4da4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7274beead37476e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.360+0000 7f4da1600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4013a50 con 0x559242edf620 2024-04-17T08:31:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f4da4013bf0 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da4019070 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f4da4031050 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 10 client.10122 fetch_fsmap learned FSMap version 94 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) -- 0x559242ee2040 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da4019070 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 94) ==== 3495+0+0 (secure 0 0 0) 0x7f4da4034020 con 0x559242edf620 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 10 client.10122 fetch_fsmap finished waiting for FSMap version 94 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 10 client.10122 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 20 client.10122 populate_metadata read hostname 'smithi067' 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ee4120 0x559242ee4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 4 client.10122 mds_command: new command op to 9987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.361+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 --> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559242e0b3a0 con 0x559242ee4120 2024-04-17T08:31:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.362+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ee4120 0x559242ee4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.363+0000 7f4db7e00640 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ee4120 0x559242ee4500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=277c5c1b0d151c9b server_cookie=df564a18b6dfabce in_seq=0 out_seq=0 2024-04-17T08:31:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.363+0000 7f4da1600640 10 client.10122 ms_handle_connect on v2:172.21.15.187:6835/3432159492 2024-04-17T08:31:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4da1600640 1 -- 172.21.15.67:0/3374116053 <== mds.0 v2:172.21.15.187:6835/3432159492 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559242e0b3a0 con 0x559242ee4120 2024-04-17T08:31:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4da1600640 10 client.10122 handle_command_reply: tid=0 2024-04-17T08:31:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 client.10122 shutdown 2024-04-17T08:31:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 msgr2=0x559242ee2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x559242e91ff0 tx=0x7f4da4004ab0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 shutdown_connections 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ee4120 0x559242ee4500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242ea9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242edf620 0x559242ee2e00 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/3374116053 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242ede5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 >> 172.21.15.67:0/3374116053 conn(0x559242e91b40 msgr2=0x559242e96a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4da1600640 0 client.10122 ms_handle_reset on v2:172.21.15.187:6835/3432159492 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 shutdown_connections 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/3374116053 wait complete. 2024-04-17T08:31:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.366+0000 7f4dc1e1d740 20 client.10122 trim_cache size 0 max 16384 2024-04-17T08:31:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.387+0000 7f4dc1e1d740 1 Processor -- start 2024-04-17T08:31:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.387+0000 7f4dc1e1d740 1 -- start start 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.387+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242e95dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242edf620 0x559242ef7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4db81fffb0 con 0x559242eb7df0 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4db81619b0 con 0x559242edf620 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4db8162000 con 0x559242e956a0 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46842/0 (socket says 172.21.15.67:46842) 2024-04-17T08:31:39.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 -- 172.21.15.67:0/2294068564 learned_addr learned my addr 172.21.15.67:0/2294068564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 -- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 msgr2=0x559242e95dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242e95dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 -- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242edf620 msgr2=0x559242ef7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242edf620 0x559242ef7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.388+0000 7f4db7400640 1 -- 172.21.15.67:0/2294068564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559242e8dad0 con 0x559242eb7df0 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4db7400640 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f4da4000f80 tx=0x7f4da4022070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a87d3646c7047e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4da3e00640 1 -- 172.21.15.67:0/2294068564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4003fc0 con 0x559242eb7df0 2024-04-17T08:31:39.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4da3e00640 1 -- 172.21.15.67:0/2294068564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da4019070 con 0x559242eb7df0 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4da3e00640 1 -- 172.21.15.67:0/2294068564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4013a90 con 0x559242eb7df0 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/2294068564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 msgr2=0x559242ef71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f4da4000f80 tx=0x7f4da4022070 comp rx=0 tx=0).stop 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/2294068564 shutdown_connections 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242edf620 0x559242ef7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242eb7df0 0x559242ef71f0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/2294068564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242e956a0 0x559242e95dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.389+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/2294068564 >> 172.21.15.67:0/2294068564 conn(0x559242e91b40 msgr2=0x559242ea9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.390+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/2294068564 shutdown_connections 2024-04-17T08:31:39.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.390+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/2294068564 wait complete. 2024-04-17T08:31:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.390+0000 7f4dc1e1d740 1 -- start start 2024-04-17T08:31:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.390+0000 7f4dc1e1d740 1 Processor -- start 2024-04-17T08:31:39.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242e969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559242ee2970 con 0x559242e956a0 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x559242eaa310 con 0x559242eb7df0 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x559242eedc40 con 0x559242edf620 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.391+0000 7f4dc1e1d740 10 client.0 _ll_register_callbacks cb 0x7f4da4009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46854/0 (socket says 172.21.15.67:46854) 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 -- 172.21.15.67:0/1991805183 learned_addr learned my addr 172.21.15.67:0/1991805183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 -- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 msgr2=0x559242e96f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 -- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 msgr2=0x559242e969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242e969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 -- 172.21.15.67:0/1991805183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559242ebd200 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4db7e00640 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f4da8001e70 tx=0x7f4da800de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e51f5581451861 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.392+0000 7f4da1600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8029020 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f4da8007040 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da4019070 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f4da8033050 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 10 client.10128 fetch_fsmap learned FSMap version 94 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) -- 0x559242eb9860 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da4019070 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 94) ==== 3495+0+0 (secure 0 0 0) 0x7f4da8036020 con 0x559242e956a0 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 10 client.10128 fetch_fsmap finished waiting for FSMap version 94 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 10 client.10128 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 20 client.10128 populate_metadata read hostname 'smithi067' 2024-04-17T08:31:39.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ef78d0 0x559242e90b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 4 client.10128 mds_command: new command op to 9987 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:31:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.393+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 --> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] -- command(tid 0: {"prefix": "status"}) -- 0x559242e05210 con 0x559242ef78d0 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.394+0000 7f4db7400640 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ef78d0 0x559242e90b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.395+0000 7f4db7400640 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ef78d0 0x559242e90b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cb34c410bd9ace1 server_cookie=3787abd3fa25e38a in_seq=0 out_seq=0 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.395+0000 7f4da1600640 10 client.10128 ms_handle_connect on v2:172.21.15.187:6835/3432159492 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4da1600640 1 -- 172.21.15.67:0/1991805183 <== mds.0 v2:172.21.15.187:6835/3432159492 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559242e05210 con 0x559242ef78d0 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4da1600640 10 client.10128 handle_command_reply: tid=0 2024-04-17T08:31:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 client.10128 shutdown 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 msgr2=0x559242ebccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f4da8001e70 tx=0x7f4da800de80 comp rx=0 tx=0).stop 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 shutdown_connections 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:6835/3432159492,v1:172.21.15.187:6837/3432159492] conn(0x559242ef78d0 0x559242e90b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x559242edf620 0x559242e96f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x559242eb7df0 0x559242e969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 --2- 172.21.15.67:0/1991805183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559242e956a0 0x559242ebccc0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 >> 172.21.15.67:0/1991805183 conn(0x559242e91b40 msgr2=0x559242e920b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4da1600640 0 client.10128 ms_handle_reset on v2:172.21.15.187:6835/3432159492 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 shutdown_connections 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 1 -- 172.21.15.67:0/1991805183 wait complete. 2024-04-17T08:31:39.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.396+0000 7f4dc1e1d740 20 client.10128 trim_cache size 0 max 16384 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9987, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8300232850000002, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 94, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 195, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 195, 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 12.648462936 2024-04-17T08:31:39.398 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:31:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 msgr2=0x7f4d9806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 0x7f4d9806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4da80015e0 tx=0x7f4da8009040 comp rx=0 tx=0).stop 2024-04-17T08:31:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 msgr2=0x7f4db80ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4db002b9c0 tx=0x7f4db0004730 comp rx=0 tx=0).stop 2024-04-17T08:31:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 shutdown_connections 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4db815ee20 0x7f4db80ce170 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f4d9806ca10 0x7f4d9806eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4db80d3350 0x7f4db80cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 --2- 172.21.15.67:0/3129611756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4db80d1ab0 0x7f4db814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 >> 172.21.15.67:0/3129611756 conn(0x7f4db806d3c0 msgr2=0x7f4db80b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 shutdown_connections 2024-04-17T08:31:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.397+0000 7f4dbdc00640 1 -- 172.21.15.67:0/3129611756 wait complete. 2024-04-17T08:31:39.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:31:39.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.635+0000 7f71a9c00640 1 Processor -- start 2024-04-17T08:31:39.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.636+0000 7f71a9c00640 1 -- start start 2024-04-17T08:31:39.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.637+0000 7f71a9c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f71a415ec60 0x7f71a40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.637+0000 7f71a9c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 0x7f71a40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.637+0000 7f71a9c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71a40d2d30 con 0x7f71a40cf910 2024-04-17T08:31:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.637+0000 7f71a9c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f71a40d49c0 con 0x7f71a415ec60 2024-04-17T08:31:39.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.637+0000 7f71a9c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40d3660 con 0x7f71a40b6b30 2024-04-17T08:31:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a9200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f71a40cf910 0x7f71a406cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55266/0 (socket says 172.21.15.67:55266) 2024-04-17T08:31:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a9200640 1 -- 172.21.15.67:0/2727961179 learned_addr learned my addr 172.21.15.67:0/2727961179 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a3e00640 1 --2- 172.21.15.67:0/2727961179 >> v2:172.21.15.187:3300/0 conn(0x7f71a415ec60 0x7f71a40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a3400640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 0x7f71a40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71622671 0 0) 0x7f71a40d2d30 con 0x7f71a40cf910 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.638+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7188003520 con 0x7f71a40cf910 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1831774491 0 0) 0x7f7188003520 con 0x7f71a40cf910 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 msgr2=0x7f71a40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 0x7f71a40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 >> v2:172.21.15.187:3300/0 conn(0x7f71a415ec60 msgr2=0x7f71a40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 --2- 172.21.15.67:0/2727961179 >> v2:172.21.15.187:3300/0 conn(0x7f71a415ec60 0x7f71a40b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40d3010 con 0x7f71a40cf910 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f719c001670 con 0x7f71a40cf910 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 >> v1:172.21.15.67:6789/0 conn(0x7f71a40cf910 legacy=0x7f71a406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7188003a20 0x7f7188003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f7188004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 0x7f71880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40d49c0 con 0x7f7188004800 2024-04-17T08:31:39.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40d3660 con 0x7f7188003a20 2024-04-17T08:31:39.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f718800b9c0 con 0x7f71a40b6b30 2024-04-17T08:31:39.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.639+0000 7f71a3400640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f7188004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3e00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7188003a20 0x7f7188003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3400640 1 -- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 msgr2=0x7f71880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3400640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40b6b30 0x7f71880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3400640 1 -- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7188003a20 msgr2=0x7f7188003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3400640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7188003a20 0x7f7188003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.640+0000 7f71a3400640 1 -- 172.21.15.67:0/2727961179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7188003520 con 0x7f7188004800 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.641+0000 7f71a3400640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f7188004c80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f718c00e170 tx=0x7f7188005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54111ad40dad9a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.641+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c039070 con 0x7f7188004800 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.641+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718c02cc20 con 0x7f7188004800 2024-04-17T08:31:39.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.641+0000 7f71a2a00640 1 -- 172.21.15.67:0/2727961179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c009c10 con 0x7f7188004800 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 -- 172.21.15.67:0/2727961179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 msgr2=0x7f7188004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 --2- 172.21.15.67:0/2727961179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f7188004c80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f718c00e170 tx=0x7f7188005830 comp rx=0 tx=0).stop 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9200640 1 -- 172.21.15.67:0/2727961179 reap_dead start 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 -- 172.21.15.67:0/2727961179 shutdown_connections 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 -- 172.21.15.67:0/2727961179 >> 172.21.15.67:0/2727961179 conn(0x7f71a40d15f0 msgr2=0x7f71a40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 -- 172.21.15.67:0/2727961179 shutdown_connections 2024-04-17T08:31:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.642+0000 7f71a9c00640 1 -- 172.21.15.67:0/2727961179 wait complete. 2024-04-17T08:31:39.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.644+0000 7f71a9c00640 1 Processor -- start 2024-04-17T08:31:39.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.644+0000 7f71a9c00640 1 -- start start 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40cb020 0x7f71a40cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 0x7f71a40cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40cfda0 con 0x7f7188004800 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40b6f70 con 0x7f71a40cbc40 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a9c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71a415f040 con 0x7f71a40cb020 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46874/0 (socket says 172.21.15.67:46874) 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.645+0000 7f71a3e00640 1 -- 172.21.15.67:0/129290928 learned_addr learned my addr 172.21.15.67:0/129290928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a9200640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 0x7f71a40cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3400640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40cb020 0x7f71a40cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 -- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40cb020 msgr2=0x7f71a40cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40cb020 0x7f71a40cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 -- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 msgr2=0x7f71a40cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 0x7f71a40cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 -- 172.21.15.67:0/129290928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40ccf30 con 0x7f7188004800 2024-04-17T08:31:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a9200640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 0x7f71a40cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.646+0000 7f71a3e00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f71940108e0 tx=0x7f7194010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.647+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719400f040 con 0x7f7188004800 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.647+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719400eb00 con 0x7f7188004800 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.647+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719401c400 con 0x7f7188004800 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.647+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a40cc600 con 0x7f7188004800 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.647+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a40cd1a0 con 0x7f7188004800 2024-04-17T08:31:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.648+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a4068a10 con 0x7f7188004800 2024-04-17T08:31:39.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.649+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f719401c5a0 con 0x7f7188004800 2024-04-17T08:31:39.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.651+0000 7f719be00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 0x7f718006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:39.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.651+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f719408c460 con 0x7f7188004800 2024-04-17T08:31:39.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.651+0000 7f71a3400640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 0x7f718006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:39.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.653+0000 7f71a3400640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 0x7f718006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718c036000 tx=0x7f718c02e000 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.656+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7194018030 con 0x7f7188004800 2024-04-17T08:31:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.902+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71a40b8e00 con 0x7f7188004800 2024-04-17T08:31:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.902+0000 7f719be00640 1 -- 172.21.15.67:0/129290928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) ==== 53+0+73 (secure 0 0 0) 0x7f719405a100 con 0x7f7188004800 2024-04-17T08:31:39.902 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:31:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 msgr2=0x7f718006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 0x7f718006ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f718c036000 tx=0x7f718c02e000 comp rx=0 tx=0).stop 2024-04-17T08:31:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 msgr2=0x7f71a4153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f71940108e0 tx=0x7f7194010da0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 shutdown_connections 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f718006ca40 0x7f718006ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a40cbc40 0x7f71a40cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71a40cb020 0x7f71a40cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 --2- 172.21.15.67:0/129290928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7188004800 0x7f71a4153e40 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 >> 172.21.15.67:0/129290928 conn(0x7f71a40d15f0 msgr2=0x7f71a40b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.905+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 shutdown_connections 2024-04-17T08:31:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:39.906+0000 7f71a9c00640 1 -- 172.21.15.67:0/129290928 wait complete. 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:31:39.925 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:31:39.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:39.926 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.978 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 37944sec preferred_lft 37944sec 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:31:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:31:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:31:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:31:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:31:39.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:39.982 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:31:40.067 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:31:40.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:40.072 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:31:40.124 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:31:40.124 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:31:40.125 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:31:40.125 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:31:40.125 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:31:40.126 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:31:40.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:31:40.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:31:40.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:40.389 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:31:40.439 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:31:40.439 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:31:40.439 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:31:40.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:40.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:31:40.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:31:40.541 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:40.598 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:31:40.598 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:31:40.598 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:40.647 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:31:40.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:40.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:40.793 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:40.793 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:31:40.828 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:31:40.828 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:31:40.828 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:31:40.829 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:31:40.829 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:31:40.831 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:31:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:31:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:31:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:31:40.831 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:31:40.835 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:31:40.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:40.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:31:40.919 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:31:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:31:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:31:41.069 INFO:teuthology.orchestra.run.smithi067.stdout:client8014 2024-04-17T08:31:41.070 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:31:41.070 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:31:41.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:31:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.355+0000 7fa139c00640 1 Processor -- start 2024-04-17T08:31:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.355+0000 7fa139c00640 1 -- start start 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa139c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1340b9850 0x7fa1340d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa139c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1340d4140 con 0x7fa1340d1ab0 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa139c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1340d3b80 con 0x7fa13415ee20 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa139c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa1340b83b0 con 0x7fa1340b9850 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa12b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa1340d1ab0 0x7fa1340d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55278/0 (socket says 172.21.15.67:55278) 2024-04-17T08:31:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.357+0000 7fa12b400640 1 -- 172.21.15.67:0/2843681105 learned_addr learned my addr 172.21.15.67:0/2843681105 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258201620 0 0) 0x7fa1340d4140 con 0x7fa1340d1ab0 2024-04-17T08:31:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa118003520 con 0x7fa1340d1ab0 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa139200640 1 --2- 172.21.15.67:0/2843681105 >> v2:172.21.15.187:3300/0 conn(0x7fa1340b9850 0x7fa1340d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4051616092 0 0) 0x7fa118003520 con 0x7fa1340d1ab0 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 >> v2:172.21.15.187:3300/0 conn(0x7fa1340b9850 msgr2=0x7fa1340d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 --2- 172.21.15.67:0/2843681105 >> v2:172.21.15.187:3300/0 conn(0x7fa1340b9850 0x7fa1340d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 >> v1:172.21.15.187:6789/0 conn(0x7fa13415ee20 legacy=0x7fa1340d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1340d3e60 con 0x7fa1340d1ab0 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa11c001670 con 0x7fa1340d1ab0 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 >> v1:172.21.15.67:6789/0 conn(0x7fa1340d1ab0 legacy=0x7fa1340d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa118003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa118004800 0x7fa118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa118005230 0x7fa11800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1340b83b0 con 0x7fa118003a20 2024-04-17T08:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa11800c190 con 0x7fa118004800 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.358+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa11800c390 con 0x7fa118005230 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa118003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa139200640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa118005230 0x7fa11800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa12b400640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa118004800 0x7fa118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 -- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa118005230 msgr2=0x7fa11800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa118005230 0x7fa11800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 -- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa118004800 msgr2=0x7fa118004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa118004800 0x7fa118004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa133e00640 1 -- 172.21.15.67:0/2843681105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa118003520 con 0x7fa118003a20 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa139200640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa118005230 0x7fa11800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa12b400640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa118004800 0x7fa118004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.359+0000 7fa139200640 1 -- 172.21.15.67:0/2843681105 reap_dead start 2024-04-17T08:31:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.360+0000 7fa133e00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa118003e40 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa124001e50 tx=0x7fa118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213f3c24f80eb573 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124001a70 con 0x7fa118003a20 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa124003b70 con 0x7fa118003a20 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa133400640 1 -- 172.21.15.67:0/2843681105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1240050d0 con 0x7fa118003a20 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 -- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 msgr2=0x7fa118003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa118003e40 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa124001e50 tx=0x7fa118005830 comp rx=0 tx=0).stop 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 -- 172.21.15.67:0/2843681105 shutdown_connections 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 --2- 172.21.15.67:0/2843681105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa118003e40 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 -- 172.21.15.67:0/2843681105 >> 172.21.15.67:0/2843681105 conn(0x7fa13406d3c0 msgr2=0x7fa1340b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 -- 172.21.15.67:0/2843681105 shutdown_connections 2024-04-17T08:31:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.361+0000 7fa139c00640 1 -- 172.21.15.67:0/2843681105 wait complete. 2024-04-17T08:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.363+0000 7fa139c00640 1 Processor -- start 2024-04-17T08:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.363+0000 7fa139c00640 1 -- start start 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1340b56c0 0x7fa1340b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 0x7fa134161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1340d13a0 con 0x7fa118003a20 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa13415f200 con 0x7fa134160ca0 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.364+0000 7fa139c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1340d1520 con 0x7fa1340b56c0 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46898/0 (socket says 172.21.15.67:46898) 2024-04-17T08:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 -- 172.21.15.67:0/4151715943 learned_addr learned my addr 172.21.15.67:0/4151715943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa12b400640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1340b56c0 0x7fa1340b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa139200640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 0x7fa134161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 -- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1340b56c0 msgr2=0x7fa1340b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1340b56c0 0x7fa1340b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 -- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 msgr2=0x7fa134161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 0x7fa134161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa133e00640 1 -- 172.21.15.67:0/4151715943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13415f520 con 0x7fa118003a20 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.365+0000 7fa139200640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 0x7fa134161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa133e00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fa124005040 tx=0x7fa1240014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124006b50 con 0x7fa118003a20 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa124006cf0 con 0x7fa118003a20 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124007270 con 0x7fa118003a20 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13415f800 con 0x7fa118003a20 2024-04-17T08:31:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.366+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13415fdc0 con 0x7fa118003a20 2024-04-17T08:31:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.367+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa134068a10 con 0x7fa118003a20 2024-04-17T08:31:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.368+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7fa124003a30 con 0x7fa118003a20 2024-04-17T08:31:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.369+0000 7fa130c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 0x7fa11006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.369+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7fa1240ada10 con 0x7fa118003a20 2024-04-17T08:31:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.370+0000 7fa12b400640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 0x7fa11006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.371+0000 7fa12b400640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 0x7fa11006ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0017e0 tx=0x7fa11c009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.374+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa124033020 con 0x7fa118003a20 2024-04-17T08:31:41.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.606+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 --> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa1341626f0 con 0x7fa11006ca40 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.610+0000 7fa130c00640 1 -- 172.21.15.67:0/4151715943 <== mgr.7807 v2:172.21.15.187:6832/3300567104 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa1341626f0 con 0x7fa11006ca40 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:31:41.609 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:31:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 msgr2=0x7fa11006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 0x7fa11006ef00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0017e0 tx=0x7fa11c009040 comp rx=0 tx=0).stop 2024-04-17T08:31:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 msgr2=0x7fa134148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fa124005040 tx=0x7fa1240014a0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 shutdown_connections 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fa11006ca40 0x7fa11006ef00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa134160ca0 0x7fa134161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1340b56c0 0x7fa1340b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 --2- 172.21.15.67:0/4151715943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118003a20 0x7fa134148bd0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.612+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 >> 172.21.15.67:0/4151715943 conn(0x7fa13406d3c0 msgr2=0x7fa1340d51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.613+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 shutdown_connections 2024-04-17T08:31:41.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.613+0000 7fa139c00640 1 -- 172.21.15.67:0/4151715943 wait complete. 2024-04-17T08:31:41.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:31:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.859+0000 7f59f7e00640 1 Processor -- start 2024-04-17T08:31:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.859+0000 7f59f7e00640 1 -- start start 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59f80cd920 0x7f59f806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f8146360 con 0x7f59f8145f80 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f8071e70 con 0x7f59f806c0a0 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f59f80727a0 con 0x7f59f80cd920 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f59f8145f80 0x7f59f80cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55284/0 (socket says 172.21.15.67:55284) 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f6a00640 1 -- 172.21.15.67:0/3470721747 learned_addr learned my addr 172.21.15.67:0/3470721747 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:41.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.861+0000 7f59f6000640 1 --2- 172.21.15.67:0/3470721747 >> v2:172.21.15.67:3300/0 conn(0x7f59f80cd920 0x7f59f806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.862+0000 7f59f6000640 1 -- 172.21.15.67:0/3470721747 >> v1:172.21.15.187:6789/0 conn(0x7f59f806c0a0 legacy=0x7f59f806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:41.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.862+0000 7f59f6000640 1 -- 172.21.15.67:0/3470721747 >> v1:172.21.15.67:6789/0 conn(0x7f59f8145f80 legacy=0x7f59f80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.862+0000 7f59f6000640 1 -- 172.21.15.67:0/3470721747 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f8072150 con 0x7f59f80cd920 2024-04-17T08:31:41.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.862+0000 7f59f5600640 1 -- 172.21.15.67:0/3470721747 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3646005265 0 0) 0x7f59f8146360 con 0x7f59f8145f80 2024-04-17T08:31:41.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.863+0000 7f59f6000640 1 --2- 172.21.15.67:0/3470721747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f806bb60 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097a0 tx=0x7f59e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231e02c382177e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.863+0000 7f59f5600640 1 -- 172.21.15.67:0/3470721747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40315a0 con 0x7f59f80cd920 2024-04-17T08:31:41.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.863+0000 7f59f5600640 1 -- 172.21.15.67:0/3470721747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e4037070 con 0x7f59f80cd920 2024-04-17T08:31:41.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.863+0000 7f59f5600640 1 -- 172.21.15.67:0/3470721747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4031960 con 0x7f59f80cd920 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 -- 172.21.15.67:0/3470721747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 msgr2=0x7f59f806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3470721747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f806bb60 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097a0 tx=0x7f59e402b840 comp rx=0 tx=0).stop 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 -- 172.21.15.67:0/3470721747 shutdown_connections 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3470721747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f806bb60 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 -- 172.21.15.67:0/3470721747 >> 172.21.15.67:0/3470721747 conn(0x7f59f80b2190 msgr2=0x7f59f80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:41.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 -- 172.21.15.67:0/3470721747 shutdown_connections 2024-04-17T08:31:41.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.864+0000 7f59f7e00640 1 -- 172.21.15.67:0/3470721747 wait complete. 2024-04-17T08:31:41.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.866+0000 7f59f7e00640 1 Processor -- start 2024-04-17T08:31:41.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.866+0000 7f59f7e00640 1 -- start start 2024-04-17T08:31:41.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59f806c0a0 0x7f59f813b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59f8145f80 0x7f59f8141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8072c30 con 0x7f59f80cd920 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f59f806c530 con 0x7f59f806c0a0 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f59f8072db0 con 0x7f59f8145f80 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46922/0 (socket says 172.21.15.67:46922) 2024-04-17T08:31:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.867+0000 7f59f6000640 1 -- 172.21.15.67:0/3868817300 learned_addr learned my addr 172.21.15.67:0/3868817300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6a00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59f806c0a0 0x7f59f813b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 -- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59f8145f80 msgr2=0x7f59f8141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59f8145f80 0x7f59f8141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 -- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59f806c0a0 msgr2=0x7f59f813b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59f806c0a0 0x7f59f813b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 -- 172.21.15.67:0/3868817300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f81422a0 con 0x7f59f80cd920 2024-04-17T08:31:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59f6000640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f59e4034040 tx=0x7f59e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.868+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4032840 con 0x7f59f80cd920 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.869+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e40329e0 con 0x7f59f80cd920 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.869+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f8142580 con 0x7f59f80cd920 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.869+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4032dd0 con 0x7f59f80cd920 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.869+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f8142b40 con 0x7f59f80cd920 2024-04-17T08:31:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.870+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f8068a10 con 0x7f59f80cd920 2024-04-17T08:31:41.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.871+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 99) ==== 91142+0+0 (secure 0 0 0) 0x7f59e403b3f0 con 0x7f59f80cd920 2024-04-17T08:31:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.872+0000 7f59ef400640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 0x7f59d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.872+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f59e402f030 con 0x7f59f80cd920 2024-04-17T08:31:41.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.873+0000 7f59f6a00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 0x7f59d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.874+0000 7f59f6a00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 0x7f59d806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59e8000f60 tx=0x7f59e8009040 comp rx=0 tx=0).ready entity=mgr.7807 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:41.876+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f59e4040070 con 0x7f59f80cd920 2024-04-17T08:31:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.134+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f59f806f830 con 0x7f59f80cd920 2024-04-17T08:31:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.651+0000 7f59ef400640 1 -- 172.21.15.67:0/3868817300 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) ==== 92+0+0 (secure 0 0 0) 0x7f59e407bfa0 con 0x7f59f80cd920 2024-04-17T08:31:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 msgr2=0x7f59d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 0x7f59d806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f59e8000f60 tx=0x7f59e8009040 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 msgr2=0x7f59f8141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f59e4034040 tx=0x7f59e40046c0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 shutdown_connections 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59f8145f80 0x7f59f8141cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80cd920 0x7f59f8141770 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f59d806ca10 0x7f59d806eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 --2- 172.21.15.67:0/3868817300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59f806c0a0 0x7f59f813b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.654+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 >> 172.21.15.67:0/3868817300 conn(0x7f59f80b2190 msgr2=0x7f59f80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.655+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 shutdown_connections 2024-04-17T08:31:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.655+0000 7f59f7e00640 1 -- 172.21.15.67:0/3868817300 wait complete. 2024-04-17T08:31:42.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:31:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.891+0000 7f9915200640 1 Processor -- start 2024-04-17T08:31:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.891+0000 7f9915200640 1 -- start start 2024-04-17T08:31:42.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f9915200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99100d0b50 0x7f991015fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f9915200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9910163710 con 0x7f99101627b0 2024-04-17T08:31:42.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f9915200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f991015f360 con 0x7f991006b8c0 2024-04-17T08:31:42.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f9915200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9910163150 con 0x7f99100d0b50 2024-04-17T08:31:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f990e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f99101627b0 0x7f99100cdf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55288/0 (socket says 172.21.15.67:55288) 2024-04-17T08:31:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f990e800640 1 -- 172.21.15.67:0/420142527 learned_addr learned my addr 172.21.15.67:0/420142527 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.893+0000 7f9905400640 1 --2- 172.21.15.67:0/420142527 >> v2:172.21.15.187:3300/0 conn(0x7f99100d0b50 0x7f991015fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129237088 0 0) 0x7f9910163710 con 0x7f99101627b0 2024-04-17T08:31:42.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98f4003520 con 0x7f99101627b0 2024-04-17T08:31:42.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3409574787 0 0) 0x7f98f4003520 con 0x7f99101627b0 2024-04-17T08:31:42.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 >> v2:172.21.15.187:3300/0 conn(0x7f99100d0b50 msgr2=0x7f991015fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 --2- 172.21.15.67:0/420142527 >> v2:172.21.15.187:3300/0 conn(0x7f99100d0b50 0x7f991015fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 >> v1:172.21.15.187:6789/0 conn(0x7f991006b8c0 legacy=0x7f991006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9910163430 con 0x7f99101627b0 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9908001670 con 0x7f99101627b0 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 >> v1:172.21.15.67:6789/0 conn(0x7f99101627b0 legacy=0x7f99100cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98f4003a00 0x7f98f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98f4004800 0x7f98f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f98f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9910163150 con 0x7f98f40051c0 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.894+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f98f400c0c0 con 0x7f98f4004800 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f98f400c2c0 con 0x7f98f4003a00 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f98f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990de00640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98f4003a00 0x7f98f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 -- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98f4003a00 msgr2=0x7f98f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98f4003a00 0x7f98f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 -- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98f4004800 msgr2=0x7f98f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98f4004800 0x7f98f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990e800640 1 -- 172.21.15.67:0/420142527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f4003520 con 0x7f98f40051c0 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.895+0000 7f990de00640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98f4003a00 0x7f98f4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:42.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.897+0000 7f990e800640 1 -- 172.21.15.67:0/420142527 reap_dead start 2024-04-17T08:31:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.897+0000 7f990e800640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f98f400b9c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f990800c190 tx=0x7f98f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f55bb988dfd900 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9908007c60 con 0x7f98f40051c0 2024-04-17T08:31:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9908030070 con 0x7f98f40051c0 2024-04-17T08:31:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f990d400640 1 -- 172.21.15.67:0/420142527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9908005d30 con 0x7f98f40051c0 2024-04-17T08:31:42.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f9915200640 1 -- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 msgr2=0x7f98f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f9915200640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f98f400b9c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f990800c190 tx=0x7f98f4005830 comp rx=0 tx=0).stop 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f9915200640 1 -- 172.21.15.67:0/420142527 shutdown_connections 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f9915200640 1 --2- 172.21.15.67:0/420142527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f98f400b9c0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.898+0000 7f9915200640 1 -- 172.21.15.67:0/420142527 >> 172.21.15.67:0/420142527 conn(0x7f9910073360 msgr2=0x7f991009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.899+0000 7f9915200640 1 -- 172.21.15.67:0/420142527 shutdown_connections 2024-04-17T08:31:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.899+0000 7f9915200640 1 -- 172.21.15.67:0/420142527 wait complete. 2024-04-17T08:31:42.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.900+0000 7f9915200640 1 Processor -- start 2024-04-17T08:31:42.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.900+0000 7f9915200640 1 -- start start 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99101f44a0 0x7f99100c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 0x7f9910160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f991015f070 con 0x7f98f40051c0 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f991015eef0 con 0x7f99101f44a0 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.901+0000 7f9915200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9910162c00 con 0x7f99100c5590 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46964/0 (socket says 172.21.15.67:46964) 2024-04-17T08:31:42.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 -- 172.21.15.67:0/3658547290 learned_addr learned my addr 172.21.15.67:0/3658547290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:42.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 msgr2=0x7f9910160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f9905400640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99101f44a0 0x7f99100c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990e800640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 0x7f9910160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 0x7f9910160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99101f44a0 msgr2=0x7f99100c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99101f44a0 0x7f99100c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990de00640 1 -- 172.21.15.67:0/3658547290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99100c5a30 con 0x7f98f40051c0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.902+0000 7f990e800640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 0x7f9910160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f990de00640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f990000e8f0 tx=0x7f990000edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900003b80 con 0x7f98f40051c0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9900018070 con 0x7f98f40051c0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000050d0 con 0x7f98f40051c0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99100c5d10 con 0x7f98f40051c0 2024-04-17T08:31:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.903+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991014b340 con 0x7f98f40051c0 2024-04-17T08:31:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.905+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9910068ca0 con 0x7f98f40051c0 2024-04-17T08:31:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.905+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 100) ==== 91159+0+0 (secure 0 0 0) 0x7f9900005270 con 0x7f98f40051c0 2024-04-17T08:31:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.907+0000 7f9906a00640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 0x7f98ec06ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.907+0000 7f9905400640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 msgr2=0x7f98ec06ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.907+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f990008bc50 con 0x7f98f40051c0 2024-04-17T08:31:42.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.907+0000 7f9905400640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 0x7f98ec06ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:42.911+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f990001a310 con 0x7f98f40051c0 2024-04-17T08:31:43.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.108+0000 7f9905400640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 msgr2=0x7f98ec06ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:43.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.108+0000 7f9905400640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 0x7f98ec06ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:43.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.156+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f99100b07a0 con 0x7f98f40051c0 2024-04-17T08:31:43.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.162+0000 7f9906a00640 1 -- 172.21.15.67:0/3658547290 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) ==== 138+0+0 (secure 0 0 0) 0x7f990005a0c0 con 0x7f98f40051c0 2024-04-17T08:31:43.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 msgr2=0x7f98ec06ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 0x7f98ec06ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 msgr2=0x7f99101485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f990000e8f0 tx=0x7f990000edb0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 shutdown_connections 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99100c5590 0x7f9910160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f98ec06cab0 0x7f98ec06ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99101f44a0 0x7f99100c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 --2- 172.21.15.67:0/3658547290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f40051c0 0x7f99101485d0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 >> 172.21.15.67:0/3658547290 conn(0x7f9910073360 msgr2=0x7f99100ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 shutdown_connections 2024-04-17T08:31:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.165+0000 7f9915200640 1 -- 172.21.15.67:0/3658547290 wait complete. 2024-04-17T08:31:43.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:31:43.205 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.206+0000 7f8b66efa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:43.259 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.259+0000 7f8b66efa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:43.284 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:43.285+0000 7f4ead649040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:31:43.346 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:43.347+0000 7f4ead649040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:31:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.400+0000 7f67a7200640 1 Processor -- start 2024-04-17T08:31:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.400+0000 7f67a7200640 1 -- start start 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a815e9b0 0x7f67a80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67a80d4510 0x7f67a80d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67a80b51d0 con 0x7f67a80cf910 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f67a80b5790 con 0x7f67a815e9b0 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67a80d2830 con 0x7f67a80d4510 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a815e9b0 0x7f67a80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a815e9b0 0x7f67a80cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40732/0 (socket says 172.21.15.67:40732) 2024-04-17T08:31:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a5e00640 1 -- 172.21.15.67:0/2216772328 learned_addr learned my addr 172.21.15.67:0/2216772328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a5e00640 1 -- 172.21.15.67:0/2216772328 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d4510 msgr2=0x7f67a80d4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.402+0000 7f67a5e00640 1 --2- 172.21.15.67:0/2216772328 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d4510 0x7f67a80d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.403+0000 7f67a5e00640 1 -- 172.21.15.67:0/2216772328 >> v1:172.21.15.187:6789/0 conn(0x7f67a80cf910 legacy=0x7f67a80b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.403+0000 7f67a5e00640 1 -- 172.21.15.67:0/2216772328 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a80b54b0 con 0x7f67a815e9b0 2024-04-17T08:31:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.404+0000 7f67a5e00640 1 --2- 172.21.15.67:0/2216772328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a815e9b0 0x7f67a80cf3d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f67a80cb840 tx=0x7f679802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753dc45eff5920f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.404+0000 7f67a4a00640 1 -- 172.21.15.67:0/2216772328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798035020 con 0x7f67a815e9b0 2024-04-17T08:31:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.404+0000 7f67a4a00640 1 -- 172.21.15.67:0/2216772328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6798031800 con 0x7f67a815e9b0 2024-04-17T08:31:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.404+0000 7f67a4a00640 1 -- 172.21.15.67:0/2216772328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798031bd0 con 0x7f67a815e9b0 2024-04-17T08:31:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 -- 172.21.15.67:0/2216772328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a815e9b0 msgr2=0x7f67a80cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 --2- 172.21.15.67:0/2216772328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a815e9b0 0x7f67a80cf3d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f67a80cb840 tx=0x7f679802b8a0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 -- 172.21.15.67:0/2216772328 shutdown_connections 2024-04-17T08:31:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 --2- 172.21.15.67:0/2216772328 >> v2:172.21.15.187:3300/0 conn(0x7f67a80d4510 0x7f67a80d4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 --2- 172.21.15.67:0/2216772328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a815e9b0 0x7f67a80cf3d0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 -- 172.21.15.67:0/2216772328 >> 172.21.15.67:0/2216772328 conn(0x7f67a809daa0 msgr2=0x7f67a808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 -- 172.21.15.67:0/2216772328 shutdown_connections 2024-04-17T08:31:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.405+0000 7f67a7200640 1 -- 172.21.15.67:0/2216772328 wait complete. 2024-04-17T08:31:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.407+0000 7f67a7200640 1 Processor -- start 2024-04-17T08:31:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.407+0000 7f67a7200640 1 -- start start 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a80cf910 0x7f67a80b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 0x7f67a8153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67a8162830 con 0x7f67a80d4510 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f67a80d4ea0 con 0x7f67a815e9b0 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67a80b4e00 con 0x7f67a80cf910 2024-04-17T08:31:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40748/0 (socket says 172.21.15.67:40748) 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a5400640 1 -- 172.21.15.67:0/3072903928 learned_addr learned my addr 172.21.15.67:0/3072903928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.408+0000 7f67a5e00640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a80cf910 0x7f67a80b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a80cf910 msgr2=0x7f67a80b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a6800640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 0x7f67a8153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a80cf910 0x7f67a80b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 msgr2=0x7f67a8153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 0x7f67a8153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 -- 172.21.15.67:0/3072903928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a80cd800 con 0x7f67a80d4510 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a6800640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 0x7f67a8153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a5400640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f679400b3f0 tx=0x7f679400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794015020 con 0x7f67a80d4510 2024-04-17T08:31:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6794011830 con 0x7f67a80d4510 2024-04-17T08:31:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794011c20 con 0x7f67a80d4510 2024-04-17T08:31:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.409+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a80cdd90 con 0x7f67a80d4510 2024-04-17T08:31:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.410+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a80ce350 con 0x7f67a80d4510 2024-04-17T08:31:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.411+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a80ce550 con 0x7f67a80d4510 2024-04-17T08:31:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.412+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 100) ==== 91159+0+0 (secure 0 0 0) 0x7f6794018020 con 0x7f67a80d4510 2024-04-17T08:31:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.413+0000 7f679e000640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 0x7f678806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.413+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7f679408b8e0 con 0x7f67a80d4510 2024-04-17T08:31:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.414+0000 7f67a5e00640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 msgr2=0x7f678806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:43.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.414+0000 7f67a5e00640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 0x7f678806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.418+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f67a80ce550 con 0x7f67a80d4510 2024-04-17T08:31:43.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.615+0000 7f67a5e00640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 msgr2=0x7f678806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:43.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.615+0000 7f67a5e00640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 0x7f678806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.623+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f67a80755d0 con 0x7f67a80d4510 2024-04-17T08:31:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.630+0000 7f679e000640 1 -- 172.21.15.67:0/3072903928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) ==== 133+0+0 (secure 0 0 0) 0x7f679401f030 con 0x7f67a80d4510 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.632+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 msgr2=0x7f678806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 0x7f678806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 msgr2=0x7f67a80ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f679400b3f0 tx=0x7f679400b8b0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 shutdown_connections 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7f678806ca60 0x7f678806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a815e9b0 0x7f67a8153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a80d4510 0x7f67a80ccdd0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 --2- 172.21.15.67:0/3072903928 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67a80cf910 0x7f67a80b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 >> 172.21.15.67:0/3072903928 conn(0x7f67a809daa0 msgr2=0x7f67a806d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 shutdown_connections 2024-04-17T08:31:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.633+0000 7f67a7200640 1 -- 172.21.15.67:0/3072903928 wait complete. 2024-04-17T08:31:43.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:43.651 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T08:31:43.704 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.705+0000 7f8b66efa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:43.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:31:43.710 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-04-17T08:31:43.758 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.758+0000 7f8b66efa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:43.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-04-17T08:31:43.808 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.809+0000 7f8b66efa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:43.827 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:43.824+0000 7f4ead649040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:31:43.869 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.869+0000 7f8b66efa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:43.879 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:43.880+0000 7f4ead649040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:31:43.917 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:43.918+0000 7f8b66efa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:43.933 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:43.934+0000 7f4ead649040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:31:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.972+0000 7fc107e00640 1 Processor -- start 2024-04-17T08:31:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.972+0000 7fc107e00640 1 -- start start 2024-04-17T08:31:43.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.973+0000 7fc107e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc1080cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.973+0000 7fc107e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1080cf910 0x7fc1080b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.973+0000 7fc107e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1080b51d0 con 0x7fc1080b4c30 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.973+0000 7fc107e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc1080b5790 con 0x7fc1080cf910 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.973+0000 7fc107e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1080d2830 con 0x7fc10815e9b0 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc107400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1080b4c30 0x7fc1080d4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46822/0 (socket says 172.21.15.67:46822) 2024-04-17T08:31:43.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc107400640 1 -- 172.21.15.67:0/705750574 learned_addr learned my addr 172.21.15.67:0/705750574 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:43.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc106000640 1 --2- 172.21.15.67:0/705750574 >> v2:172.21.15.187:3300/0 conn(0x7fc1080cf910 0x7fc1080b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc106a00640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc1080cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573842952 0 0) 0x7fc1080b51d0 con 0x7fc1080b4c30 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.974+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0ec003520 con 0x7fc1080b4c30 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1466053748 0 0) 0x7fc0ec003520 con 0x7fc1080b4c30 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 msgr2=0x7fc1080cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc1080cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 >> v2:172.21.15.187:3300/0 conn(0x7fc1080cf910 msgr2=0x7fc1080b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 --2- 172.21.15.67:0/705750574 >> v2:172.21.15.187:3300/0 conn(0x7fc1080cf910 0x7fc1080b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1080b54b0 con 0x7fc1080b4c30 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc1000019f0 con 0x7fc1080b4c30 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 >> v1:172.21.15.67:6789/0 conn(0x7fc1080b4c30 legacy=0x7fc1080d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec0039b0 0x7fc0ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc0ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0ec004d40 0x7fc0ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080b5790 con 0x7fc0ec004d40 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080d2830 con 0x7fc0ec0039b0 2024-04-17T08:31:43.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ec00b9c0 con 0x7fc10815e9b0 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc107400640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0ec004d40 0x7fc0ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.975+0000 7fc106a00640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec0039b0 0x7fc0ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc106000640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc0ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc107400640 1 -- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 msgr2=0x7fc0ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc107400640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc10815e9b0 0x7fc0ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc107400640 1 -- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec0039b0 msgr2=0x7fc0ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc107400640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec0039b0 0x7fc0ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc107400640 1 -- 172.21.15.67:0/705750574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec003520 con 0x7fc0ec004d40 2024-04-17T08:31:43.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.976+0000 7fc106a00640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec0039b0 0x7fc0ec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:31:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.977+0000 7fc107400640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0ec004d40 0x7fc0ec0051c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc10000a880 tx=0x7fc0ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de21fc0719ce2503 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10000a4a0 con 0x7fc0ec004d40 2024-04-17T08:31:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc100034650 con 0x7fc0ec004d40 2024-04-17T08:31:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc105600640 1 -- 172.21.15.67:0/705750574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100034a40 con 0x7fc0ec004d40 2024-04-17T08:31:43.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc107e00640 1 -- 172.21.15.67:0/705750574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0ec004d40 msgr2=0x7fc0ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc107e00640 1 --2- 172.21.15.67:0/705750574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0ec004d40 0x7fc0ec0051c0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc10000a880 tx=0x7fc0ec005830 comp rx=0 tx=0).stop 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc107400640 1 -- 172.21.15.67:0/705750574 reap_dead start 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc107e00640 1 -- 172.21.15.67:0/705750574 shutdown_connections 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.978+0000 7fc107e00640 1 -- 172.21.15.67:0/705750574 >> 172.21.15.67:0/705750574 conn(0x7fc10809daa0 msgr2=0x7fc10808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.979+0000 7fc107e00640 1 -- 172.21.15.67:0/705750574 shutdown_connections 2024-04-17T08:31:43.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.979+0000 7fc107e00640 1 -- 172.21.15.67:0/705750574 wait complete. 2024-04-17T08:31:43.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.980+0000 7fc107e00640 1 Processor -- start 2024-04-17T08:31:43.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.980+0000 7fc107e00640 1 -- start start 2024-04-17T08:31:43.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec004d40 0x7fc1081535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1080cd2b0 0x7fc1080cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 0x7fc1080b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc10815ed90 con 0x7fc1080b73e0 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc108162830 con 0x7fc0ec004d40 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.981+0000 7fc107e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc10815ef10 con 0x7fc1080cd2b0 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 0x7fc1080b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 0x7fc1080b7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40758/0 (socket says 172.21.15.67:40758) 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 -- 172.21.15.67:0/693567874 learned_addr learned my addr 172.21.15.67:0/693567874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec004d40 0x7fc1081535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc106000640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1080cd2b0 0x7fc1080cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1080cd2b0 msgr2=0x7fc1080cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1080cd2b0 0x7fc1080cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec004d40 msgr2=0x7fc1081535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0ec004d40 0x7fc1081535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.982+0000 7fc107400640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1080cbc00 con 0x7fc1080b73e0 2024-04-17T08:31:43.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc107400640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 0x7fc1080b7860 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc10000a9a0 tx=0x7fc1080b53a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:43.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100045070 con 0x7fc1080b73e0 2024-04-17T08:31:43.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10003a040 con 0x7fc1080b73e0 2024-04-17T08:31:43.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1080cbee0 con 0x7fc1080b73e0 2024-04-17T08:31:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100001ff0 con 0x7fc1080b73e0 2024-04-17T08:31:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.983+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1080cc4a0 con 0x7fc1080b73e0 2024-04-17T08:31:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.984+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1080755d0 con 0x7fc1080b73e0 2024-04-17T08:31:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.985+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 100) ==== 91159+0+0 (secure 0 0 0) 0x7fc100048070 con 0x7fc1080b73e0 2024-04-17T08:31:43.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.986+0000 7fc0fea00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:43.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.986+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5737+0+0 (secure 0 0 0) 0x7fc1000ada00 con 0x7fc1080b73e0 2024-04-17T08:31:43.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.986+0000 7fc106a00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 msgr2=0x7fc0e806efa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:43.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.987+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:31:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:43.991+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc10007bcd0 con 0x7fc1080b73e0 2024-04-17T08:31:44.002 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.003+0000 7f4ead649040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:31:44.048 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.048+0000 7f8b66efa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:44.055 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.056+0000 7f4ead649040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:31:44.110 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.110+0000 7f8b66efa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:44.160 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.161+0000 7f8b66efa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:44.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:44.188+0000 7fc106a00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 msgr2=0x7fc0e806efa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:44.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:44.188+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:31:44.192 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.193+0000 7f4ead649040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:31:44.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:44.215+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc1080b6ba0 con 0x7fc0e806cae0 2024-04-17T08:31:44.218 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.219+0000 7f8b66efa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:44.258 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.259+0000 7f4ead649040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:44.268 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.269+0000 7f8b66efa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:44.310 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.311+0000 7f4ead649040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:31:44.370 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.371+0000 7f4ead649040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:31:44.424 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.424+0000 7f4ead649040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:31:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:44.589+0000 7fc106a00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 msgr2=0x7fc0e806efa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:44.589+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:31:44.650 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.650+0000 7f8b66efa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:44.809 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.810+0000 7f8b66efa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:44.828 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.828+0000 7f4ead649040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:31:44.918 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.919+0000 7f8b66efa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:44.968 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:44.969+0000 7f8b66efa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:44.995 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:44.996+0000 7f4ead649040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:31:45.093 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:45.094+0000 7f8b66efa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:45.109 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.110+0000 7f4ead649040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:31:45.163 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.164+0000 7f4ead649040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:31:45.200 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:45.201+0000 7f8b66efa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:45.296 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.297+0000 7f4ead649040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:45.373 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:45.373+0000 7f8b66efa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:45.390+0000 7fc106a00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 msgr2=0x7fc0e806efa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3300567104 2024-04-17T08:31:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:45.390+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:31:45.409 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.410+0000 7f4ead649040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:31:45.422 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:45.423+0000 7f8b66efa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:45.558 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:45.559+0000 7f8b66efa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:45.592 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.593+0000 7f4ead649040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:31:45.644 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.645+0000 7f4ead649040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:31:45.788 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:45.788+0000 7f4ead649040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:31:46.365 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:46.366+0000 7f8b66efa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:46.637 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:46.637+0000 7f4ead649040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:31:46.667 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:46.669+0000 7f8b66efa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:46.725 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:46.726+0000 7f8b66efa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:46.859 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:46.860+0000 7f8b66efa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:46.884+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 101) ==== 90768+0+0 (secure 0 0 0) 0x7fc100077c40 con 0x7fc1080b73e0 2024-04-17T08:31:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:46.885+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 msgr2=0x7fc0e806efa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:31:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:46.885+0000 7fc0fea00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/3300567104,v1:172.21.15.187:6833/3300567104] conn(0x7fc0e806cae0 0x7fc0e806efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:46.968 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:46.968+0000 7f4ead649040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:31:47.028 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:47.028+0000 7f4ead649040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ade200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ade200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ade200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ade200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ade200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.091 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ada600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.092 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ada600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.092 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ada600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.092 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ada600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.092 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:31:47.093+0000 7f8ada600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:31:47.166 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:31:47.166+0000 7f4ead649040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:31:47.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:47.893+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 102) ==== 90895+0+0 (secure 0 0 0) 0x7fc100098bd0 con 0x7fc1080b73e0 2024-04-17T08:31:47.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:47.894+0000 7fc0fea00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc0e80749f0 0x7fc0e8076de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:47.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:47.894+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 --> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc1000ad830 con 0x7fc0e80749f0 2024-04-17T08:31:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:47.905+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc0e80749f0 0x7fc0e8076de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:47.913+0000 7fc106a00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc0e80749f0 0x7fc0e8076de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8006a20 tx=0x7fc0f8008040 comp rx=0 tx=0).ready entity=mgr.7247 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:48.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.079+0000 7fc0fea00640 1 -- 172.21.15.67:0/693567874 <== mgr.7247 v2:172.21.15.187:6832/1430869695 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc1000ad830 con 0x7fc0e80749f0 2024-04-17T08:31:48.079 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc0e80749f0 msgr2=0x7fc0e8076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc0e80749f0 0x7fc0e8076de0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8006a20 tx=0x7fc0f8008040 comp rx=0 tx=0).stop 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 msgr2=0x7fc1080b7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 --2- 172.21.15.67:0/693567874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1080b73e0 0x7fc1080b7860 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc10000a9a0 tx=0x7fc1080b53a0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107400640 1 -- 172.21.15.67:0/693567874 reap_dead start 2024-04-17T08:31:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 shutdown_connections 2024-04-17T08:31:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 >> 172.21.15.67:0/693567874 conn(0x7fc10809daa0 msgr2=0x7fc10806e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 shutdown_connections 2024-04-17T08:31:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.083+0000 7fc107e00640 1 -- 172.21.15.67:0/693567874 wait complete. 2024-04-17T08:31:48.107 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-04-17T08:31:48.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.339+0000 7f3ad3200640 1 Processor -- start 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.339+0000 7f3ad3200640 1 -- start start 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.340+0000 7f3ad3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3acc0cd920 0x7f3acc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3acc146360 con 0x7f3acc145f80 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3acc071e70 con 0x7f3acc06c0a0 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3acc0727a0 con 0x7f3acc0cd920 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3acc145f80 0x7f3acc0cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46826/0 (socket says 172.21.15.67:46826) 2024-04-17T08:31:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad1e00640 1 -- 172.21.15.67:0/1702051148 learned_addr learned my addr 172.21.15.67:0/1702051148 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:31:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649693765 0 0) 0x7f3acc146360 con 0x7f3acc145f80 2024-04-17T08:31:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab4003520 con 0x7f3acc145f80 2024-04-17T08:31:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.341+0000 7f3ad1400640 1 --2- 172.21.15.67:0/1702051148 >> v2:172.21.15.187:3300/0 conn(0x7f3acc0cd920 0x7f3acc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 848914688 0 0) 0x7f3ab4003520 con 0x7f3acc145f80 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 >> v2:172.21.15.187:3300/0 conn(0x7f3acc0cd920 msgr2=0x7f3acc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 --2- 172.21.15.67:0/1702051148 >> v2:172.21.15.187:3300/0 conn(0x7f3acc0cd920 0x7f3acc06bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 >> v1:172.21.15.187:6789/0 conn(0x7f3acc06c0a0 legacy=0x7f3acc06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc072150 con 0x7f3acc145f80 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f3ac0001670 con 0x7f3acc145f80 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 >> v1:172.21.15.67:6789/0 conn(0x7f3acc145f80 legacy=0x7f3acc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab40039b0 0x7f3ab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 0x7f3ab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ab40051c0 0x7f3ab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc0727a0 con 0x7f3ab4004800 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab400c160 con 0x7f3ab40039b0 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab400c360 con 0x7f3ab40051c0 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.342+0000 7f3ad1400640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 0x7f3ab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1400640 1 -- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ab40051c0 msgr2=0x7f3ab400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1400640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ab40051c0 0x7f3ab400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1400640 1 -- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab40039b0 msgr2=0x7f3ab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab40039b0 0x7f3ab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1400640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab40039b0 0x7f3ab4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1400640 1 -- 172.21.15.67:0/1702051148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab4003520 con 0x7f3ab4004800 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad2800640 1 -- 172.21.15.67:0/1702051148 reap_dead start 2024-04-17T08:31:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.343+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab40039b0 0x7f3ab4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.344+0000 7f3ad1400640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 0x7f3ab4004c80 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009a90 tx=0x7f3ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131fb446c06b088 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.344+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc039070 con 0x7f3ab4004800 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.344+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc02c6b0 con 0x7f3ab4004800 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.344+0000 7f3ad0a00640 1 -- 172.21.15.67:0/1702051148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc009690 con 0x7f3ab4004800 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 -- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 msgr2=0x7f3ab4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 0x7f3ab4004c80 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009a90 tx=0x7f3ab4005830 comp rx=0 tx=0).stop 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 -- 172.21.15.67:0/1702051148 shutdown_connections 2024-04-17T08:31:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 --2- 172.21.15.67:0/1702051148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ab4004800 0x7f3ab4004c80 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 -- 172.21.15.67:0/1702051148 >> 172.21.15.67:0/1702051148 conn(0x7f3acc0b2190 msgr2=0x7f3acc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 -- 172.21.15.67:0/1702051148 shutdown_connections 2024-04-17T08:31:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.345+0000 7f3ad3200640 1 -- 172.21.15.67:0/1702051148 wait complete. 2024-04-17T08:31:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.347+0000 7f3ad3200640 1 Processor -- start 2024-04-17T08:31:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.347+0000 7f3ad3200640 1 -- start start 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 0x7f3acc1e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3acc1e4d30 0x7f3acc1e51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc072c30 con 0x7f3acc13ea70 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc06c530 con 0x7f3ab4004800 2024-04-17T08:31:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3acc072db0 con 0x7f3acc1e4d30 2024-04-17T08:31:48.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40776/0 (socket says 172.21.15.67:40776) 2024-04-17T08:31:48.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.348+0000 7f3ad1400640 1 -- 172.21.15.67:0/3742819949 learned_addr learned my addr 172.21.15.67:0/3742819949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 -- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3acc1e4d30 msgr2=0x7f3acc1e51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3acc1e4d30 0x7f3acc1e51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 -- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 msgr2=0x7f3acc1e75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 0x7f3acc1e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 0x7f3acc1e75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 -- 172.21.15.67:0/3742819949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc1e6b40 con 0x7f3acc13ea70 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 0x7f3acc1e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad1400640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3abc037040 tx=0x7f3abc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc039050 con 0x7f3acc13ea70 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3abc045070 con 0x7f3acc13ea70 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc003a70 con 0x7f3acc13ea70 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.349+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc13f410 con 0x7f3acc13ea70 2024-04-17T08:31:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.350+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc1e5d80 con 0x7f3acc13ea70 2024-04-17T08:31:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.351+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3acc068a10 con 0x7f3acc13ea70 2024-04-17T08:31:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.352+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 102) ==== 90895+0+0 (secure 0 0 0) 0x7f3abc003c10 con 0x7f3acc13ea70 2024-04-17T08:31:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.353+0000 7f3ac6000640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 0x7f3ab006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:31:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.353+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6081+0+0 (secure 0 0 0) 0x7f3abc0ad670 con 0x7f3acc13ea70 2024-04-17T08:31:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.353+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 0x7f3ab006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:31:48.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.355+0000 7f3ad1e00640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 0x7f3ab006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00017e0 tx=0x7f3ac0009040 comp rx=0 tx=0).ready entity=mgr.7247 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:31:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.358+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3abc04e020 con 0x7f3acc13ea70 2024-04-17T08:31:48.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.599+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 --> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3acc0afd00 con 0x7f3ab006c790 2024-04-17T08:31:48.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.741+0000 7f3ac6000640 1 -- 172.21.15.67:0/3742819949 <== mgr.7247 v2:172.21.15.187:6832/1430869695 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7f3acc0afd00 con 0x7f3ab006c790 2024-04-17T08:31:48.743 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 msgr2=0x7f3ab006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 0x7f3ab006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00017e0 tx=0x7f3ac0009040 comp rx=0 tx=0).stop 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 msgr2=0x7f3acc13eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3abc037040 tx=0x7f3abc004730 comp rx=0 tx=0).stop 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 shutdown_connections 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3acc1e4d30 0x7f3acc1e51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7f3ab006c790 0x7f3ab006ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3acc13ea70 0x7f3acc13eed0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 --2- 172.21.15.67:0/3742819949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ab4004800 0x7f3acc1e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:31:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.744+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 >> 172.21.15.67:0/3742819949 conn(0x7f3acc0b2190 msgr2=0x7f3acc0b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:31:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.745+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 shutdown_connections 2024-04-17T08:31:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:31:48.745+0000 7f3ad3200640 1 -- 172.21.15.67:0/3742819949 wait complete. 2024-04-17T08:31:48.769 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-04-17T08:31:49.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:31:49.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:32:00.764 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:32:00.764+0000 7f8ac8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:32:00.765 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:32:00.764+0000 7f8ac8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:32:00.765 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:32:00.764+0000 7f8ac8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:32:00.765 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:32:00.764+0000 7f8ac8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:32:00.765 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:32:00.764+0000 7f8ac8e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:32:19.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:32:19.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:32:49.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:32:49.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:33:20.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:33:20.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:33:50.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:33:50.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:34:20.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:34:20.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:34:50.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:34:50.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:35:20.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:35:20.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:35:51.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:35:51.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:36:21.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:36:21.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:36:51.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:36:51.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:37:21.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:37:21.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:37:51.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:37:51.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:38:22.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:38:22.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:38:52.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:38:52.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:39:22.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:39:22.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:39:52.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:39:52.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:40:23.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:40:23.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:40:53.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:40:53.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:41:23.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:41:23.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:41:53.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:41:53.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:42:15.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:42:15.770 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-17T08:42:15.837 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_32_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_33_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_34_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_35_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_36_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_37_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_38_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_39_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_40_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_41_00_UTC 2024-04-17T08:42:15.838 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_42_00_UTC 2024-04-17T08:42:15.839 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-08_32_00_UTC', 'scheduled-2024-04-17-08_33_00_UTC', 'scheduled-2024-04-17-08_34_00_UTC', 'scheduled-2024-04-17-08_35_00_UTC', 'scheduled-2024-04-17-08_36_00_UTC', 'scheduled-2024-04-17-08_37_00_UTC', 'scheduled-2024-04-17-08_38_00_UTC', 'scheduled-2024-04-17-08_39_00_UTC', 'scheduled-2024-04-17-08_40_00_UTC', 'scheduled-2024-04-17-08_41_00_UTC', 'scheduled-2024-04-17-08_42_00_UTC'] 2024-04-17T08:42:15.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-17T08:42:16.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.075+0000 7fc517e00640 1 Processor -- start 2024-04-17T08:42:16.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.075+0000 7fc517e00640 1 -- start start 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.077+0000 7fc517e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc51806ee50 0x7fc51806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.077+0000 7fc517e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 0x7fc5180af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.077+0000 7fc517e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc51806c200 con 0x7fc518161fd0 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.077+0000 7fc517e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc5180b0d90 con 0x7fc51806ee50 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.077+0000 7fc517e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc5180ae440 con 0x7fc51806be00 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc51806ee50 0x7fc51806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc51806ee50 0x7fc51806b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42224/0 (socket says 172.21.15.67:42224) 2024-04-17T08:42:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 -- 172.21.15.67:0/324087076 learned_addr learned my addr 172.21.15.67:0/324087076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 -- 172.21.15.67:0/324087076 >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 msgr2=0x7fc5180af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc517400640 1 --2- 172.21.15.67:0/324087076 >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 0x7fc5180af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 --2- 172.21.15.67:0/324087076 >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 0x7fc5180af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 -- 172.21.15.67:0/324087076 >> v1:172.21.15.187:6789/0 conn(0x7fc518161fd0 legacy=0x7fc5180b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc50fe00640 1 -- 172.21.15.67:0/324087076 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5180ae6e0 con 0x7fc51806ee50 2024-04-17T08:42:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.078+0000 7fc517400640 1 --2- 172.21.15.67:0/324087076 >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 0x7fc5180af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:42:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.079+0000 7fc50fe00640 1 --2- 172.21.15.67:0/324087076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806ee50 0x7fc51806b8c0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fc5180c84c0 tx=0x7fc50002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475a9c127e0fe4a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.080+0000 7fc516000640 1 -- 172.21.15.67:0/324087076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50002e070 con 0x7fc51806ee50 2024-04-17T08:42:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.080+0000 7fc516000640 1 -- 172.21.15.67:0/324087076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500037070 con 0x7fc51806ee50 2024-04-17T08:42:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.080+0000 7fc516000640 1 -- 172.21.15.67:0/324087076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500031a00 con 0x7fc51806ee50 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.080+0000 7fc517e00640 1 -- 172.21.15.67:0/324087076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806ee50 msgr2=0x7fc51806b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.080+0000 7fc517e00640 1 --2- 172.21.15.67:0/324087076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806ee50 0x7fc51806b8c0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fc5180c84c0 tx=0x7fc50002b880 comp rx=0 tx=0).stop 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 -- 172.21.15.67:0/324087076 shutdown_connections 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 --2- 172.21.15.67:0/324087076 >> v2:172.21.15.187:3300/0 conn(0x7fc51806be00 0x7fc5180af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 --2- 172.21.15.67:0/324087076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806ee50 0x7fc51806b8c0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 -- 172.21.15.67:0/324087076 >> 172.21.15.67:0/324087076 conn(0x7fc51809aa60 msgr2=0x7fc5180b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:42:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 -- 172.21.15.67:0/324087076 shutdown_connections 2024-04-17T08:42:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.081+0000 7fc517e00640 1 -- 172.21.15.67:0/324087076 wait complete. 2024-04-17T08:42:16.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.082+0000 7fc517e00640 1 Processor -- start 2024-04-17T08:42:16.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.083+0000 7fc517e00640 1 -- start start 2024-04-17T08:42:16.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.083+0000 7fc517e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc517e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc51806ee50 0x7fc5180ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc517e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc518161fd0 0x7fc51815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc517e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5180aef80 con 0x7fc51806be00 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc517e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5180aeca0 con 0x7fc51806ee50 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc517e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5180b3760 con 0x7fc518161fd0 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42228/0 (socket says 172.21.15.67:42228) 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 -- 172.21.15.67:0/4132130063 learned_addr learned my addr 172.21.15.67:0/4132130063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 -- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc518161fd0 msgr2=0x7fc51815f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc518161fd0 0x7fc51815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 -- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc51806ee50 msgr2=0x7fc5180ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc51806ee50 0x7fc5180ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.084+0000 7fc516a00640 1 -- 172.21.15.67:0/4132130063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5180c88e0 con 0x7fc51806be00 2024-04-17T08:42:16.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc516a00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fc50800b420 tx=0x7fc50800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508011830 con 0x7fc51806be00 2024-04-17T08:42:16.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc508016070 con 0x7fc51806be00 2024-04-17T08:42:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508011c10 con 0x7fc51806be00 2024-04-17T08:42:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5180c8bc0 con 0x7fc51806be00 2024-04-17T08:42:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.085+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5180c9180 con 0x7fc51806be00 2024-04-17T08:42:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.086+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5180cf400 con 0x7fc51806be00 2024-04-17T08:42:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.087+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 106) ==== 91171+0+0 (secure 0 0 0) 0x7fc50801b460 con 0x7fc51806be00 2024-04-17T08:42:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.088+0000 7fc50f400640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 0x7fc4f406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.088+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6081+0+0 (secure 0 0 0) 0x7fc50808bb20 con 0x7fc51806be00 2024-04-17T08:42:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.089+0000 7fc50fe00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 0x7fc4f406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.090+0000 7fc50fe00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 0x7fc4f406ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc500036000 tx=0x7fc500030000 comp rx=0 tx=0).ready entity=mgr.7247 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.093+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc5180cf400 con 0x7fc51806be00 2024-04-17T08:42:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.319+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 --> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc518068a10 con 0x7fc4f406ca60 2024-04-17T08:42:16.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.338+0000 7fc50f400640 1 -- 172.21.15.67:0/4132130063 <== mgr.7247 v2:172.21.15.187:6832/1430869695 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+375 (secure 0 0 0) 0x7fc518068a10 con 0x7fc4f406ca60 2024-04-17T08:42:16.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:42:16.337 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:31:47", "first": "2024-04-17T08:32:00", "last": "2024-04-17T08:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 msgr2=0x7fc4f406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 0x7fc4f406ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc500036000 tx=0x7fc500030000 comp rx=0 tx=0).stop 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 msgr2=0x7fc518154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fc50800b420 tx=0x7fc50800b8e0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 shutdown_connections 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc518161fd0 0x7fc51815f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc51806ee50 0x7fc5180ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fc4f406ca60 0x7fc4f406ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 --2- 172.21.15.67:0/4132130063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc51806be00 0x7fc518154ad0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.340+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 >> 172.21.15.67:0/4132130063 conn(0x7fc51809aa60 msgr2=0x7fc51806f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:42:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.341+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 shutdown_connections 2024-04-17T08:42:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.341+0000 7fc517e00640 1 -- 172.21.15.67:0/4132130063 wait complete. 2024-04-17T08:42:16.358 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:31:47", "first": "2024-04-17T08:32:00", "last": "2024-04-17T08:42:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-04-17T08:42:16.359 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T08:31:47', 'first': '2024-04-17T08:32:00', 'last': '2024-04-17T08:42:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-04-17T08:42:16.359 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-04-17T08:42:16.359 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-04-17T08:42:16.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-17T08:42:16.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.578+0000 7fa835e00640 1 Processor -- start 2024-04-17T08:42:16.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.578+0000 7fa835e00640 1 -- start start 2024-04-17T08:42:16.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa83006b8c0 0x7fa8300cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa83006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8300cd4a0 con 0x7fa830141ed0 2024-04-17T08:42:16.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa83015ce30 con 0x7fa83006b8c0 2024-04-17T08:42:16.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8301607f0 con 0x7fa830160280 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.580+0000 7fa835400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa830141ed0 0x7fa83015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50842/0 (socket says 172.21.15.67:50842) 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa835400640 1 -- 172.21.15.67:0/218703836 learned_addr learned my addr 172.21.15.67:0/218703836 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa834a00640 1 --2- 172.21.15.67:0/218703836 >> v2:172.21.15.67:3300/0 conn(0x7fa83006b8c0 0x7fa8300cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa827e00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa83006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa82fe00640 1 -- 172.21.15.67:0/218703836 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 311800281 0 0) 0x7fa8300cd4a0 con 0x7fa830141ed0 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa82fe00640 1 -- 172.21.15.67:0/218703836 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa814003520 con 0x7fa830141ed0 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa834a00640 1 -- 172.21.15.67:0/218703836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 msgr2=0x7fa83006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa834a00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa83006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa834a00640 1 -- 172.21.15.67:0/218703836 >> v1:172.21.15.67:6789/0 conn(0x7fa830141ed0 legacy=0x7fa83015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.581+0000 7fa834a00640 1 -- 172.21.15.67:0/218703836 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa830160ad0 con 0x7fa83006b8c0 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.582+0000 7fa834a00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa8300cd8b0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fa820009d90 tx=0x7fa82002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431f37886071f0c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa82fe00640 1 -- 172.21.15.67:0/218703836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820039070 con 0x7fa83006b8c0 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa82fe00640 1 -- 172.21.15.67:0/218703836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82002cc60 con 0x7fa83006b8c0 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa82fe00640 1 -- 172.21.15.67:0/218703836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820034660 con 0x7fa83006b8c0 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 -- 172.21.15.67:0/218703836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 msgr2=0x7fa8300cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa8300cd8b0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fa820009d90 tx=0x7fa82002beb0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 -- 172.21.15.67:0/218703836 shutdown_connections 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa83006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 --2- 172.21.15.67:0/218703836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa8300cd8b0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 -- 172.21.15.67:0/218703836 >> 172.21.15.67:0/218703836 conn(0x7fa830067d10 msgr2=0x7fa8300b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.583+0000 7fa835e00640 1 -- 172.21.15.67:0/218703836 shutdown_connections 2024-04-17T08:42:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.584+0000 7fa835e00640 1 -- 172.21.15.67:0/218703836 wait complete. 2024-04-17T08:42:16.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.585+0000 7fa835e00640 1 Processor -- start 2024-04-17T08:42:16.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.585+0000 7fa835e00640 1 -- start start 2024-04-17T08:42:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa830141ed0 0x7fa8301f4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa8300b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa83006c1e0 con 0x7fa83006b8c0 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300cddf0 con 0x7fa830141ed0 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa835e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa830161090 con 0x7fa830160280 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.586+0000 7fa834a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42246/0 (socket says 172.21.15.67:42246) 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 -- 172.21.15.67:0/1954245523 learned_addr learned my addr 172.21.15.67:0/1954245523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 -- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 msgr2=0x7fa8300b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa8300b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 -- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa830141ed0 msgr2=0x7fa8301f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa830141ed0 0x7fa8301f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 -- 172.21.15.67:0/1954245523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83013eec0 con 0x7fa83006b8c0 2024-04-17T08:42:16.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.587+0000 7fa834a00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fa820037040 tx=0x7fa820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.588+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820044020 con 0x7fa83006b8c0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.588+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82003f470 con 0x7fa83006b8c0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.588+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820039040 con 0x7fa83006b8c0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.588+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83015d3f0 con 0x7fa83006b8c0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.588+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83015d9b0 con 0x7fa83006b8c0 2024-04-17T08:42:16.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.589+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa830068a10 con 0x7fa83006b8c0 2024-04-17T08:42:16.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.590+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 106) ==== 91171+0+0 (secure 0 0 0) 0x7fa820047020 con 0x7fa83006b8c0 2024-04-17T08:42:16.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.591+0000 7fa82d600640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 0x7fa80c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:42:16.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.592+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 6081+0+0 (secure 0 0 0) 0x7fa8200aeac0 con 0x7fa83006b8c0 2024-04-17T08:42:16.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.592+0000 7fa827e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 0x7fa80c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:42:16.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.593+0000 7fa827e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 0x7fa80c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa818006fd0 tx=0x7fa818008040 comp rx=0 tx=0).ready entity=mgr.7247 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:42:16.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.596+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa82004e030 con 0x7fa83006b8c0 2024-04-17T08:42:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.855+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa8300cfbf0 con 0x7fa83006b8c0 2024-04-17T08:42:16.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.864+0000 7fa82d600640 1 -- 172.21.15.67:0/1954245523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) ==== 74+0+174354 (secure 0 0 0) 0x7fa82007cc50 con 0x7fa83006b8c0 2024-04-17T08:42:16.867 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:42:16.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.868+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 msgr2=0x7fa80c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.868+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 0x7fa80c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa818006fd0 tx=0x7fa818008040 comp rx=0 tx=0).stop 2024-04-17T08:42:16.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.868+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 msgr2=0x7fa83013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:42:16.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.868+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fa820037040 tx=0x7fa820004a40 comp rx=0 tx=0).stop 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 shutdown_connections 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa830160280 0x7fa8300b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa830141ed0 0x7fa8301f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.187:6832/1430869695,v1:172.21.15.187:6833/1430869695] conn(0x7fa80c06ca60 0x7fa80c06ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 --2- 172.21.15.67:0/1954245523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa83006b8c0 0x7fa83013e980 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 >> 172.21.15.67:0/1954245523 conn(0x7fa830067d10 msgr2=0x7fa8300d03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 shutdown_connections 2024-04-17T08:42:16.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:42:16.869+0000 7fa835e00640 1 -- 172.21.15.67:0/1954245523 wait complete. 2024-04-17T08:42:16.887 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"flags":0,"active_gid":7247,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1430869695},{"type":"v1","addr":"172.21.15.187:6833","nonce":1430869695}]},"active_addr":"172.21.15.187:6833/1430869695","active_change":"2024-04-17T08:31:46.865472+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":10161,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":196,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1228337607}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":3491826676}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":521765659}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":1964974751}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":2568338216}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.187:0","nonce":2365426680}]}]} 2024-04-17T08:42:16.891 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-04-17T08:42:16.892 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-17T08:42:16.892 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-04-17T08:42:16.892 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-17T08:42:16.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:42:16.974 INFO:tasks.ceph.mgr.x:Stopped 2024-04-17T08:42:16.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-17T08:42:16.980 INFO:tasks.ceph.mgr.x:Started 2024-04-17T08:42:17.588 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:17.589+0000 7eff58e40040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:42:17.642 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:17.643+0000 7eff58e40040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:42:18.083 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.084+0000 7eff58e40040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:42:18.135 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.136+0000 7eff58e40040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:42:18.185 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.185+0000 7eff58e40040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:42:18.245 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.246+0000 7eff58e40040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:42:18.294 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.295+0000 7eff58e40040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:42:18.424 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.424+0000 7eff58e40040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:42:18.486 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.487+0000 7eff58e40040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:42:18.536 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.536+0000 7eff58e40040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:42:18.592 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.593+0000 7eff58e40040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:42:18.642 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:18.643+0000 7eff58e40040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:42:19.024 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.024+0000 7eff58e40040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:42:19.181 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.182+0000 7eff58e40040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:42:19.290 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.291+0000 7eff58e40040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:42:19.340 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.341+0000 7eff58e40040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:42:19.466 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.466+0000 7eff58e40040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:42:19.572 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.573+0000 7eff58e40040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:42:19.750 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.750+0000 7eff58e40040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:42:19.799 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.799+0000 7eff58e40040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:42:19.935 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:19.936+0000 7eff58e40040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:42:20.749 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:20.749+0000 7eff58e40040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:42:21.050 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.051+0000 7eff58e40040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:42:21.107 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.108+0000 7eff58e40040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:42:21.237 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.238+0000 7eff58e40040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:42:21.509 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efecb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.509 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efecb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.509 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efecb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.509 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efecb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efecb400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efece600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efece600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efece600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efece600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:21.510 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:42:21.510+0000 7efece600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:42:23.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:42:23.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:42:54.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:42:54.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:43:00.511 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:43:00.511+0000 7efed9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:43:00.511 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:43:00.511+0000 7efed9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:43:00.511 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:43:00.511+0000 7efed9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:43:00.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:43:00.511+0000 7efed9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:43:00.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:43:00.511+0000 7efed9000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:43:24.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:43:24.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:43:54.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:43:54.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:44:24.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:44:24.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:44:54.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:44:54.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:45:25.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:45:25.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:45:55.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:45:55.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:46:25.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:46:25.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:46:55.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:46:55.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:47:16.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-04-17T08:47:17.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.217+0000 7fdd90a00640 1 Processor -- start 2024-04-17T08:47:17.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.217+0000 7fdd90a00640 1 -- start start 2024-04-17T08:47:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd90a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd8c15b860 0x7fdd8c0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd90a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd90a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c071f60 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd90a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8c071d10 con 0x7fdd8c15b860 2024-04-17T08:47:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd90a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c072240 con 0x7fdd8c0b4f80 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd89e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd8c15b860 0x7fdd8c0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd89e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd8c15b860 0x7fdd8c0d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58470/0 (socket says 172.21.15.67:58470) 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.219+0000 7fdd89e00640 1 -- 172.21.15.67:0/4160071660 learned_addr learned my addr 172.21.15.67:0/4160071660 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.220+0000 7fdd8a800640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.220+0000 7fdd89e00640 1 -- 172.21.15.67:0/4160071660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 msgr2=0x7fdd8c0d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.220+0000 7fdd89e00640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.220+0000 7fdd89e00640 1 -- 172.21.15.67:0/4160071660 >> v1:172.21.15.187:6789/0 conn(0x7fdd8c0ce4d0 legacy=0x7fdd8c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.220+0000 7fdd89e00640 1 -- 172.21.15.67:0/4160071660 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c072520 con 0x7fdd8c15b860 2024-04-17T08:47:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.221+0000 7fdd89e00640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c15b860 0x7fdd8c0d0420 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c071e90 tx=0x7fdd7c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe9e17147fa9e79 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.221+0000 7fdd88a00640 1 -- 172.21.15.67:0/4160071660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c0317c0 con 0x7fdd8c15b860 2024-04-17T08:47:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.221+0000 7fdd88a00640 1 -- 172.21.15.67:0/4160071660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd7c037070 con 0x7fdd8c15b860 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.221+0000 7fdd88a00640 1 -- 172.21.15.67:0/4160071660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c031b80 con 0x7fdd8c15b860 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 -- 172.21.15.67:0/4160071660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c15b860 msgr2=0x7fdd8c0d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c15b860 0x7fdd8c0d0420 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c071e90 tx=0x7fdd7c02b860 comp rx=0 tx=0).stop 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 -- 172.21.15.67:0/4160071660 shutdown_connections 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 --2- 172.21.15.67:0/4160071660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c15b860 0x7fdd8c0d0420 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 -- 172.21.15.67:0/4160071660 >> 172.21.15.67:0/4160071660 conn(0x7fdd8c067ee0 msgr2=0x7fdd8c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:17.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 -- 172.21.15.67:0/4160071660 shutdown_connections 2024-04-17T08:47:17.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.222+0000 7fdd90a00640 1 -- 172.21.15.67:0/4160071660 wait complete. 2024-04-17T08:47:17.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.224+0000 7fdd90a00640 1 Processor -- start 2024-04-17T08:47:17.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.224+0000 7fdd90a00640 1 -- start start 2024-04-17T08:47:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd8c15b860 0x7fdd8c15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c15bc40 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c072ae0 con 0x7fdd8c15b860 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd90a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c06f380 con 0x7fdd8c0b4f80 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58484/0 (socket says 172.21.15.67:58484) 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.225+0000 7fdd89400640 1 -- 172.21.15.67:0/1448793166 learned_addr learned my addr 172.21.15.67:0/1448793166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89e00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 -- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 msgr2=0x7fdd8c0af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd8a800640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd8c15b860 0x7fdd8c15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 -- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd8c15b860 msgr2=0x7fdd8c15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd8c15b860 0x7fdd8c15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 -- 172.21.15.67:0/1448793166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c15d5b0 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89e00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd89400640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400c780 tx=0x7fdd7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74016020 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd7400aa50 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c15d890 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.226+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74012420 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.227+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8c15de50 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.228+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8c068ca0 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.229+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 112) ==== 91171+0+0 (secure 0 0 0) 0x7fdd74019020 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.229+0000 7fdd82000640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 0x7fdd6806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.230+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7fdd7408c0e0 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.230+0000 7fdd89e00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 0x7fdd6806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.231+0000 7fdd89e00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 0x7fdd6806ef20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c036000 tx=0x7fdd7c008030 comp rx=0 tx=0).ready entity=mgr.8104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.235+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdd74020030 con 0x7fdd8c0ce4d0 2024-04-17T08:47:17.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.466+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 --> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fdd8c15d350 con 0x7fdd6806ca60 2024-04-17T08:47:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.648+0000 7fdd82000640 1 -- 172.21.15.67:0/1448793166 <== mgr.8104 v2:172.21.15.187:6832/219728900 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7fdd7c037040 con 0x7fdd6806ca60 2024-04-17T08:47:17.647 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 msgr2=0x7fdd6806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 0x7fdd6806ef20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c036000 tx=0x7fdd7c008030 comp rx=0 tx=0).stop 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 msgr2=0x7fdd8c147fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400c780 tx=0x7fdd7400cc40 comp rx=0 tx=0).stop 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 shutdown_connections 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fdd6806ca60 0x7fdd6806ef20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd8c15b860 0x7fdd8c15ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd8c0ce4d0 0x7fdd8c147fc0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 --2- 172.21.15.67:0/1448793166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd8c0b4f80 0x7fdd8c0af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.650+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 >> 172.21.15.67:0/1448793166 conn(0x7fdd8c067ee0 msgr2=0x7fdd8c0b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.651+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 shutdown_connections 2024-04-17T08:47:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.651+0000 7fdd90a00640 1 -- 172.21.15.67:0/1448793166 wait complete. 2024-04-17T08:47:17.672 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-04-17T08:47:17.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:17.672 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-17T08:47:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_32_00_UTC 2024-04-17T08:47:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_33_00_UTC 2024-04-17T08:47:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_34_00_UTC 2024-04-17T08:47:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_35_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_36_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_37_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_38_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_39_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_40_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_41_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_42_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_43_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_44_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_45_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_46_00_UTC 2024-04-17T08:47:17.733 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_47_00_UTC 2024-04-17T08:47:17.734 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-04-17-08_32_00_UTC', 'scheduled-2024-04-17-08_33_00_UTC', 'scheduled-2024-04-17-08_34_00_UTC', 'scheduled-2024-04-17-08_35_00_UTC', 'scheduled-2024-04-17-08_36_00_UTC', 'scheduled-2024-04-17-08_37_00_UTC', 'scheduled-2024-04-17-08_38_00_UTC', 'scheduled-2024-04-17-08_39_00_UTC', 'scheduled-2024-04-17-08_40_00_UTC', 'scheduled-2024-04-17-08_41_00_UTC', 'scheduled-2024-04-17-08_42_00_UTC', 'scheduled-2024-04-17-08_43_00_UTC', 'scheduled-2024-04-17-08_44_00_UTC', 'scheduled-2024-04-17-08_45_00_UTC', 'scheduled-2024-04-17-08_46_00_UTC', 'scheduled-2024-04-17-08_47_00_UTC'] 2024-04-17T08:47:17.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-04-17T08:47:17.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.958+0000 7f545d600640 1 Processor -- start 2024-04-17T08:47:17.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.958+0000 7f545d600640 1 -- start start 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.959+0000 7f545d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f5458072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f545d600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f545806f990 con 0x7f545806f4f0 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f545d600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f545806fc70 con 0x7f545815fc00 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f545d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54580d0af0 con 0x7f545815f110 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f544e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f545806f4f0 0x7f545815f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49192/0 (socket says 172.21.15.67:49192) 2024-04-17T08:47:17.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f544e000640 1 -- 172.21.15.67:0/2644453855 learned_addr learned my addr 172.21.15.67:0/2644453855 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:17.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.960+0000 7f5456a00640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f5458072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659812284 0 0) 0x7f545806f990 con 0x7f545806f4f0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f543c003520 con 0x7f545806f4f0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4162371789 0 0) 0x7f543c003520 con 0x7f545806f4f0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 msgr2=0x7f5458072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f5458072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 >> v1:172.21.15.187:6789/0 conn(0x7f545815fc00 legacy=0x7f54581686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54580d04a0 con 0x7f545806f4f0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5440001670 con 0x7f545806f4f0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 >> v1:172.21.15.67:6789/0 conn(0x7f545806f4f0 legacy=0x7f545815f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f543c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c003fb0 0x7f543c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c004d40 0x7f543c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f543c00bad0 con 0x7f543c004d40 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f543c00bc80 con 0x7f543c003fb0 2024-04-17T08:47:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.961+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f543c00be80 con 0x7f545815f110 2024-04-17T08:47:17.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c004d40 0x7f543c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5456a00640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f543c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f544e000640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c003fb0 0x7f543c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 -- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 msgr2=0x7f543c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f545815f110 0x7f543c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 -- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c003fb0 msgr2=0x7f543c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c003fb0 0x7f543c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.962+0000 7f5457400640 1 -- 172.21.15.67:0/2644453855 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c003520 con 0x7f543c004d40 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.963+0000 7f5457400640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c004d40 0x7f543c0051c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f5450001750 tx=0x7f543c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ba8c14679e57c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.963+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450001370 con 0x7f543c004d40 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.963+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54500052d0 con 0x7f543c004d40 2024-04-17T08:47:17.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.963+0000 7f5456000640 1 -- 172.21.15.67:0/2644453855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450003830 con 0x7f543c004d40 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 -- 172.21.15.67:0/2644453855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c004d40 msgr2=0x7f543c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 --2- 172.21.15.67:0/2644453855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c004d40 0x7f543c0051c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f5450001750 tx=0x7f543c005830 comp rx=0 tx=0).stop 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f5457400640 1 -- 172.21.15.67:0/2644453855 reap_dead start 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 -- 172.21.15.67:0/2644453855 shutdown_connections 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 -- 172.21.15.67:0/2644453855 >> 172.21.15.67:0/2644453855 conn(0x7f5458067ee0 msgr2=0x7f545806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 -- 172.21.15.67:0/2644453855 shutdown_connections 2024-04-17T08:47:17.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.964+0000 7f545d600640 1 -- 172.21.15.67:0/2644453855 wait complete. 2024-04-17T08:47:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.966+0000 7f545d600640 1 Processor -- start 2024-04-17T08:47:17.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.966+0000 7f545d600640 1 -- start start 2024-04-17T08:47:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 0x7f5458147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 0x7f5458156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54580cf3c0 con 0x7f5458154dd0 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54580afdc0 con 0x7f543c004d40 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.967+0000 7f545d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54580cf540 con 0x7f5458154950 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58498/0 (socket says 172.21.15.67:58498) 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 -- 172.21.15.67:0/416632038 learned_addr learned my addr 172.21.15.67:0/416632038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5456a00640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 0x7f5458147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f544e000640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 0x7f5458156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 -- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 msgr2=0x7f5458156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 0x7f5458156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 -- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 msgr2=0x7f5458147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 0x7f5458147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 -- 172.21.15.67:0/416632038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5458148470 con 0x7f5458154dd0 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f544e000640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 0x7f5458156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5456a00640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 0x7f5458147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:17.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.968+0000 7f5457400640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5450005040 tx=0x7f5450001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.969+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450041070 con 0x7f5458154dd0 2024-04-17T08:47:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.969+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5450005480 con 0x7f5458154dd0 2024-04-17T08:47:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.969+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450001370 con 0x7f5458154dd0 2024-04-17T08:47:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.969+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5458157520 con 0x7f5458154dd0 2024-04-17T08:47:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.969+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5458157ae0 con 0x7f5458154dd0 2024-04-17T08:47:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.970+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5458068ca0 con 0x7f5458154dd0 2024-04-17T08:47:17.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.971+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 112) ==== 91171+0+0 (secure 0 0 0) 0x7f5450044070 con 0x7f5458154dd0 2024-04-17T08:47:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.972+0000 7f544f400640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 0x7f543406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:17.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.972+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f54500ad8f0 con 0x7f5458154dd0 2024-04-17T08:47:17.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.973+0000 7f5456a00640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 0x7f543406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.974+0000 7f5456a00640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 0x7f543406ef20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5448007440 tx=0x7f544800b040 comp rx=0 tx=0).ready entity=mgr.8104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:17.977+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f545007b940 con 0x7f5458154dd0 2024-04-17T08:47:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.209+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 --> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f54580b55d0 con 0x7f543406ca60 2024-04-17T08:47:18.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.228+0000 7f544f400640 1 -- 172.21.15.67:0/416632038 <== mgr.8104 v2:172.21.15.187:6832/219728900 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+376 (secure 0 0 0) 0x7f54580b55d0 con 0x7f543406ca60 2024-04-17T08:47:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:18.227 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:31:47", "first": "2024-04-17T08:32:00", "last": "2024-04-17T08:47:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-17T08:47:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 msgr2=0x7f543406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 0x7f543406ef20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5448007440 tx=0x7f544800b040 comp rx=0 tx=0).stop 2024-04-17T08:47:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 msgr2=0x7f5458156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5450005040 tx=0x7f5450001120 comp rx=0 tx=0).stop 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 shutdown_connections 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f543406ca60 0x7f543406ef20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5458154dd0 0x7f5458156a90 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5458154950 0x7f5458156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 --2- 172.21.15.67:0/416632038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f543c004d40 0x7f5458147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 >> 172.21.15.67:0/416632038 conn(0x7f5458067ee0 msgr2=0x7f54580aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 shutdown_connections 2024-04-17T08:47:18.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.231+0000 7f545d600640 1 -- 172.21.15.67:0/416632038 wait complete. 2024-04-17T08:47:18.250 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:31:47", "first": "2024-04-17T08:32:00", "last": "2024-04-17T08:47:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-04-17T08:47:18.250 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-04-17T00:00:00', 'created': '2024-04-17T08:31:47', 'first': '2024-04-17T08:32:00', 'last': '2024-04-17T08:47:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-04-17T08:47:18.250 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-04-17T08:47:18.251 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-04-17T08:47:18.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-04-17T08:47:18.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.476+0000 7f644be00640 1 Processor -- start 2024-04-17T08:47:18.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.476+0000 7f644be00640 1 -- start start 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 0x7f644c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f644c15f780 con 0x7f644c0af400 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f644c15fa60 con 0x7f644c159020 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f644c15f540 con 0x7f644c06b8c0 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f644c0af400 0x7f644c162ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49196/0 (socket says 172.21.15.67:49196) 2024-04-17T08:47:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644a000640 1 -- 172.21.15.67:0/2700610929 learned_addr learned my addr 172.21.15.67:0/2700610929 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.478+0000 7f644b400640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 0x7f644c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890332610 0 0) 0x7f644c15f780 con 0x7f644c0af400 2024-04-17T08:47:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6430003520 con 0x7f644c0af400 2024-04-17T08:47:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 342297850 0 0) 0x7f6430003520 con 0x7f644c0af400 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 msgr2=0x7f644c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 0x7f644c06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 >> v1:172.21.15.187:6789/0 conn(0x7f644c159020 legacy=0x7f644c15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644c1624a0 con 0x7f644c0af400 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6438001670 con 0x7f644c0af400 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 >> v1:172.21.15.67:6789/0 conn(0x7f644c0af400 legacy=0x7f644c162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f6430003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 0x7f6430004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.479+0000 7f6449600640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6430004d40 0x7f64300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f644c15f540 con 0x7f64300039e0 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f643000ba70 con 0x7f6430004d40 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f643000bc70 con 0x7f644c06b8c0 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f6430003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644b400640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6430004d40 0x7f64300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 -- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 msgr2=0x7f6430004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c06b8c0 0x7f6430004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 -- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6430004d40 msgr2=0x7f64300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6430004d40 0x7f64300051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.480+0000 7f644aa00640 1 -- 172.21.15.67:0/2700610929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6430003520 con 0x7f64300039e0 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.481+0000 7f644aa00640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f6430003dc0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f643c001860 tx=0x7f6430005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dd90e8cb36e895 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c02f050 con 0x7f64300039e0 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f643c033650 con 0x7f64300039e0 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f6449600640 1 -- 172.21.15.67:0/2700610929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c001430 con 0x7f64300039e0 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 -- 172.21.15.67:0/2700610929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 msgr2=0x7f6430003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 --2- 172.21.15.67:0/2700610929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f6430003dc0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f643c001860 tx=0x7f6430005830 comp rx=0 tx=0).stop 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644b400640 1 -- 172.21.15.67:0/2700610929 reap_dead start 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 -- 172.21.15.67:0/2700610929 shutdown_connections 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 -- 172.21.15.67:0/2700610929 >> 172.21.15.67:0/2700610929 conn(0x7f644c0b2190 msgr2=0x7f644c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 -- 172.21.15.67:0/2700610929 shutdown_connections 2024-04-17T08:47:18.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.482+0000 7f644be00640 1 -- 172.21.15.67:0/2700610929 wait complete. 2024-04-17T08:47:18.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.484+0000 7f644be00640 1 Processor -- start 2024-04-17T08:47:18.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.484+0000 7f644be00640 1 -- start start 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 0x7f644c145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f644c146200 0x7f644c071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f644c15fd40 con 0x7f64300039e0 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f644c1633b0 con 0x7f644c146200 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f644c06bd50 con 0x7f644c1454c0 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.485+0000 7f644aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58514/0 (socket says 172.21.15.67:58514) 2024-04-17T08:47:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 -- 172.21.15.67:0/1758390674 learned_addr learned my addr 172.21.15.67:0/1758390674 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644b400640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f644c146200 0x7f644c071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644a000640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 0x7f644c145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 -- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 msgr2=0x7f644c145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 0x7f644c145cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 -- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f644c146200 msgr2=0x7f644c071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f644c146200 0x7f644c071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644aa00640 1 -- 172.21.15.67:0/1758390674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644c146680 con 0x7f64300039e0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.486+0000 7f644a000640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 0x7f644c145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f644aa00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f643c001980 tx=0x7f643c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c0035f0 con 0x7f64300039e0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f643c003790 con 0x7f64300039e0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c003b40 con 0x7f64300039e0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644c146960 con 0x7f64300039e0 2024-04-17T08:47:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.487+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f644c146f20 con 0x7f64300039e0 2024-04-17T08:47:18.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.488+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f644c068a10 con 0x7f64300039e0 2024-04-17T08:47:18.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.489+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 112) ==== 91171+0+0 (secure 0 0 0) 0x7f643c003ce0 con 0x7f64300039e0 2024-04-17T08:47:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.490+0000 7f6442a00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 0x7f642c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.490+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f643c0adf30 con 0x7f64300039e0 2024-04-17T08:47:18.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.491+0000 7f644a000640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 0x7f642c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.492+0000 7f644a000640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 0x7f642c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64380011e0 tx=0x7f6438009040 comp rx=0 tx=0).ready entity=mgr.8104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.495+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f643c041030 con 0x7f64300039e0 2024-04-17T08:47:18.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.725+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 --> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f644c0d0d60 con 0x7f642c06ca60 2024-04-17T08:47:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.923+0000 7f6442a00640 1 -- 172.21.15.67:0/1758390674 <== mgr.8104 v2:172.21.15.187:6832/219728900 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7f644c0d0d60 con 0x7f642c06ca60 2024-04-17T08:47:18.923 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-04-17T08:47:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 msgr2=0x7f642c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 0x7f642c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64380011e0 tx=0x7f6438009040 comp rx=0 tx=0).stop 2024-04-17T08:47:18.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 msgr2=0x7f644c147720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:18.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f643c001980 tx=0x7f643c001230 comp rx=0 tx=0).stop 2024-04-17T08:47:18.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 shutdown_connections 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f642c06ca60 0x7f642c06ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f644c146200 0x7f644c071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f644c1454c0 0x7f644c145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 --2- 172.21.15.67:0/1758390674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64300039e0 0x7f644c147720 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.926+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 >> 172.21.15.67:0/1758390674 conn(0x7f644c0b2190 msgr2=0x7f644c15d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.927+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 shutdown_connections 2024-04-17T08:47:18.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:18.927+0000 7f644be00640 1 -- 172.21.15.67:0/1758390674 wait complete. 2024-04-17T08:47:18.949 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-04-17T08:47:18.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:18.950 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_32_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_33_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_34_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_35_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_36_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_37_00_UTC 2024-04-17T08:47:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_38_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_39_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_40_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_41_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_42_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_43_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_44_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_45_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_46_00_UTC 2024-04-17T08:47:19.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_47_00_UTC 2024-04-17T08:47:19.009 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_32_00_UTC 2024-04-17T08:47:19.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.009 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_32_00_UTC) 2024-04-17T08:47:19.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_33_00_UTC 2024-04-17T08:47:19.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.082 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_33_00_UTC) 2024-04-17T08:47:19.158 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_34_00_UTC 2024-04-17T08:47:19.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.158 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_34_00_UTC) 2024-04-17T08:47:19.231 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_35_00_UTC 2024-04-17T08:47:19.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.231 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_35_00_UTC) 2024-04-17T08:47:19.299 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_36_00_UTC 2024-04-17T08:47:19.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.300 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_36_00_UTC) 2024-04-17T08:47:19.370 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_37_00_UTC 2024-04-17T08:47:19.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.370 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_37_00_UTC) 2024-04-17T08:47:19.442 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_38_00_UTC 2024-04-17T08:47:19.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.442 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_38_00_UTC) 2024-04-17T08:47:19.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_39_00_UTC 2024-04-17T08:47:19.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.514 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_39_00_UTC) 2024-04-17T08:47:19.585 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_40_00_UTC 2024-04-17T08:47:19.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.586 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_40_00_UTC) 2024-04-17T08:47:19.658 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_41_00_UTC 2024-04-17T08:47:19.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.659 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_41_00_UTC) 2024-04-17T08:47:19.731 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_42_00_UTC 2024-04-17T08:47:19.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.731 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_42_00_UTC) 2024-04-17T08:47:19.807 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_43_00_UTC 2024-04-17T08:47:19.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.807 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_43_00_UTC) 2024-04-17T08:47:19.876 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_44_00_UTC 2024-04-17T08:47:19.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.877 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_44_00_UTC) 2024-04-17T08:47:19.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_45_00_UTC 2024-04-17T08:47:19.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:19.947 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_45_00_UTC) 2024-04-17T08:47:20.015 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_46_00_UTC 2024-04-17T08:47:20.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:20.015 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_46_00_UTC) 2024-04-17T08:47:20.089 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_47_00_UTC 2024-04-17T08:47:20.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:20.089 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-04-17-08_47_00_UTC) 2024-04-17T08:47:20.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:20.162 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-04-17T08:47:20.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:47:20.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.450+0000 7f8f91a00640 1 Processor -- start 2024-04-17T08:47:20.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.450+0000 7f8f91a00640 1 -- start start 2024-04-17T08:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 0x7f8f8c15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f8c0af400 0x7f8f8c162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 0x7f8f8c06bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8f8c15f780 con 0x7f8f8c0af400 2024-04-17T08:47:20.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8f8c15fa60 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f91a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8c1624a0 con 0x7f8f8c159020 2024-04-17T08:47:20.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f8c0af400 0x7f8f8c162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:20.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f8c0af400 0x7f8f8c162ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58520/0 (socket says 172.21.15.67:58520) 2024-04-17T08:47:20.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.452+0000 7f8f8b400640 1 -- 172.21.15.67:0/1209380862 learned_addr learned my addr 172.21.15.67:0/1209380862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8be00640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 0x7f8f8c15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8b400640 1 -- 172.21.15.67:0/1209380862 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 msgr2=0x7f8f8c15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f91000640 1 --2- 172.21.15.67:0/1209380862 >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 0x7f8f8c06bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8b400640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 0x7f8f8c15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8b400640 1 -- 172.21.15.67:0/1209380862 >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 msgr2=0x7f8f8c06bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8b400640 1 --2- 172.21.15.67:0/1209380862 >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 0x7f8f8c06bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8b400640 1 -- 172.21.15.67:0/1209380862 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8c162780 con 0x7f8f8c0af400 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f8be00640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 0x7f8f8c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:20.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.453+0000 7f8f91000640 1 --2- 172.21.15.67:0/1209380862 >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 0x7f8f8c06bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.454+0000 7f8f8b400640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c0af400 0x7f8f8c162ca0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0af0b0 tx=0x7f8f7802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d309713fd5b1e576 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.454+0000 7f8f8aa00640 1 -- 172.21.15.67:0/1209380862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7802e070 con 0x7f8f8c0af400 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.454+0000 7f8f8aa00640 1 -- 172.21.15.67:0/1209380862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f78037070 con 0x7f8f8c0af400 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f8aa00640 1 -- 172.21.15.67:0/1209380862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78031a00 con 0x7f8f8c0af400 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 -- 172.21.15.67:0/1209380862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c0af400 msgr2=0x7f8f8c162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c0af400 0x7f8f8c162ca0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0af0b0 tx=0x7f8f7802b880 comp rx=0 tx=0).stop 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 -- 172.21.15.67:0/1209380862 shutdown_connections 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 --2- 172.21.15.67:0/1209380862 >> v2:172.21.15.187:3300/0 conn(0x7f8f8c06b8c0 0x7f8f8c06bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c0af400 0x7f8f8c162ca0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 --2- 172.21.15.67:0/1209380862 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c159020 0x7f8f8c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 -- 172.21.15.67:0/1209380862 >> 172.21.15.67:0/1209380862 conn(0x7f8f8c0b2190 msgr2=0x7f8f8c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 -- 172.21.15.67:0/1209380862 shutdown_connections 2024-04-17T08:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.455+0000 7f8f91a00640 1 -- 172.21.15.67:0/1209380862 wait complete. 2024-04-17T08:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.457+0000 7f8f91a00640 1 Processor -- start 2024-04-17T08:47:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.457+0000 7f8f91a00640 1 -- start start 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c0af400 0x7f8f8c1485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f8c159020 0x7f8f8c15bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c15f4f0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c15f370 con 0x7f8f8c159020 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.458+0000 7f8f91a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8c1631e0 con 0x7f8f8c0af400 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58532/0 (socket says 172.21.15.67:58532) 2024-04-17T08:47:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 -- 172.21.15.67:0/664737479 learned_addr learned my addr 172.21.15.67:0/664737479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 -- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c0af400 msgr2=0x7f8f8c1485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c0af400 0x7f8f8c1485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 -- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f8c159020 msgr2=0x7f8f8c15bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f8c159020 0x7f8f8c15bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 -- 172.21.15.67:0/664737479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8c15c350 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.459+0000 7f8f8be00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00c7f0 tx=0x7f8f7c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.460+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c015070 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.460+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7c0079d0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.460+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c007da0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.460+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8c145b10 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.460+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8c1460d0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.461+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8c068a10 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.462+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 112) ==== 91171+0+0 (secure 0 0 0) 0x7f8f7c018070 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.463+0000 7f8f83e00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 0x7f8f6c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.463+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f8f7c08c0f0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.464+0000 7f8f8b400640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 0x7f8f6c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.465+0000 7f8f8b400640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 0x7f8f6c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8f78036000 tx=0x7f8f78030000 comp rx=0 tx=0).ready entity=mgr.8104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.469+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8f7c059f90 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.725+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f8f8c161fd0 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.749+0000 7f8f83e00640 1 -- 172.21.15.67:0/664737479 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) ==== 93+0+0 (secure 0 0 0) 0x7f8f7c059890 con 0x7f8f8c06b8c0 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 msgr2=0x7f8f6c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 0x7f8f6c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8f78036000 tx=0x7f8f78030000 comp rx=0 tx=0).stop 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 msgr2=0x7f8f8c145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00c7f0 tx=0x7f8f7c00cbf0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 shutdown_connections 2024-04-17T08:47:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f8f6c06ca60 0x7f8f6c06ef20 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8f78036000 tx=0x7f8f78030000 comp rx=0 tx=0).stop 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f8c159020 0x7f8f8c15bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f8c0af400 0x7f8f8c1485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 --2- 172.21.15.67:0/664737479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f8c06b8c0 0x7f8f8c145330 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.753+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 >> 172.21.15.67:0/664737479 conn(0x7f8f8c0b2190 msgr2=0x7f8f8c0aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.754+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 shutdown_connections 2024-04-17T08:47:20.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:20.754+0000 7f8f91a00640 1 -- 172.21.15.67:0/664737479 wait complete. 2024-04-17T08:47:20.776 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:47:20.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:47:20.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:47:20.968 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:47:20.968 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:47:20.968 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:20.968 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:21.020 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:21.021 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:21.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.279+0000 7f34db200640 1 Processor -- start 2024-04-17T08:47:21.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.279+0000 7f34db200640 1 -- start start 2024-04-17T08:47:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34db200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34d40cf640 0x7f34d40cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34db200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34d415ed10 0x7f34d40d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34db200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34d40cff60 con 0x7f34d40d2ab0 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34db200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f34d40d3c30 con 0x7f34d40cf640 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34db200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f34d40d3020 con 0x7f34d415ed10 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.281+0000 7f34d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34d40cf640 0x7f34d40cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34d40cf640 0x7f34d40cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58552/0 (socket says 172.21.15.67:58552) 2024-04-17T08:47:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 -- 172.21.15.67:0/76789724 learned_addr learned my addr 172.21.15.67:0/76789724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3400640 1 --2- 172.21.15.67:0/76789724 >> v2:172.21.15.187:3300/0 conn(0x7f34d415ed10 0x7f34d40d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 -- 172.21.15.67:0/76789724 >> v2:172.21.15.187:3300/0 conn(0x7f34d415ed10 msgr2=0x7f34d40d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 --2- 172.21.15.67:0/76789724 >> v2:172.21.15.187:3300/0 conn(0x7f34d415ed10 0x7f34d40d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 -- 172.21.15.67:0/76789724 >> v1:172.21.15.187:6789/0 conn(0x7f34d40d2ab0 legacy=0x7f34d40b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.282+0000 7f34d3e00640 1 -- 172.21.15.67:0/76789724 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d40d3300 con 0x7f34d40cf640 2024-04-17T08:47:21.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.283+0000 7f34d3e00640 1 --2- 172.21.15.67:0/76789724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d40cf640 0x7f34d40cfa20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f34d408ddd0 tx=0x7f34c402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921700157fb3681c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:21.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.284+0000 7f34d2a00640 1 -- 172.21.15.67:0/76789724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c402cbb0 con 0x7f34d40cf640 2024-04-17T08:47:21.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.284+0000 7f34d2a00640 1 -- 172.21.15.67:0/76789724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c402cd50 con 0x7f34d40cf640 2024-04-17T08:47:21.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.284+0000 7f34d2a00640 1 -- 172.21.15.67:0/76789724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4034610 con 0x7f34d40cf640 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.284+0000 7f34db200640 1 -- 172.21.15.67:0/76789724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d40cf640 msgr2=0x7f34d40cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.284+0000 7f34db200640 1 --2- 172.21.15.67:0/76789724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d40cf640 0x7f34d40cfa20 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f34d408ddd0 tx=0x7f34c402c010 comp rx=0 tx=0).stop 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 -- 172.21.15.67:0/76789724 shutdown_connections 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 --2- 172.21.15.67:0/76789724 >> v2:172.21.15.187:3300/0 conn(0x7f34d415ed10 0x7f34d40d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 --2- 172.21.15.67:0/76789724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d40cf640 0x7f34d40cfa20 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 -- 172.21.15.67:0/76789724 >> 172.21.15.67:0/76789724 conn(0x7f34d406cbd0 msgr2=0x7f34d406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 -- 172.21.15.67:0/76789724 shutdown_connections 2024-04-17T08:47:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.285+0000 7f34db200640 1 -- 172.21.15.67:0/76789724 wait complete. 2024-04-17T08:47:21.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.286+0000 7f34db200640 1 Processor -- start 2024-04-17T08:47:21.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.287+0000 7f34db200640 1 -- start start 2024-04-17T08:47:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 0x7f34d4155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34d40d2ab0 0x7f34d415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34d40d25b0 con 0x7f34d415ed10 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34d40d38c0 con 0x7f34d40cf640 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34db200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34d40d4600 con 0x7f34d40d2ab0 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58566/0 (socket says 172.21.15.67:58566) 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 -- 172.21.15.67:0/690240955 learned_addr learned my addr 172.21.15.67:0/690240955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d3e00640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 0x7f34d4155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d3400640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34d40d2ab0 0x7f34d415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34d40d2ab0 msgr2=0x7f34d415b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34d40d2ab0 0x7f34d415b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.288+0000 7f34d9200640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 msgr2=0x7f34d4155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34d9200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 0x7f34d4155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34d9200640 1 -- 172.21.15.67:0/690240955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d4159df0 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34d3e00640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 0x7f34d4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34d9200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00c400 tx=0x7f34cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc017070 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34cc0052d0 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc003810 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d40b6dc0 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.289+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d40b4640 con 0x7f34d415ed10 2024-04-17T08:47:21.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.290+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d4068b90 con 0x7f34d415ed10 2024-04-17T08:47:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.292+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7f34cc01a070 con 0x7f34d415ed10 2024-04-17T08:47:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.293+0000 7f34cbe00640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 0x7f34b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.293+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f34cc08b390 con 0x7f34d415ed10 2024-04-17T08:47:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.294+0000 7f34d3e00640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 msgr2=0x7f34b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.294+0000 7f34d3e00640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 0x7f34b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:21.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.298+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f34cc0593e0 con 0x7f34d415ed10 2024-04-17T08:47:21.315 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.316+0000 7fb301e3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:47:21.343 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:21.343+0000 7f253083f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:47:21.370 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.371+0000 7fb301e3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:47:21.405 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:21.406+0000 7f253083f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:47:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.495+0000 7f34d3e00640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 msgr2=0x7f34b006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.495+0000 7f34d3e00640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 0x7f34b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:21.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.505+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34d40c98b0 con 0x7f34d415ed10 2024-04-17T08:47:21.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.508+0000 7f34cbe00640 1 -- 172.21.15.67:0/690240955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) ==== 74+0+17682 (secure 0 0 0) 0x7f34cc058ce0 con 0x7f34d415ed10 2024-04-17T08:47:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:21.508 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":197,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:42:21.244593+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2365426680":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1228337607":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6833/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6832/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/521765659":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1964974751":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3275456489":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/630850508":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/74160311":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/3491826676":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1329453466":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2568338216":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1472858533":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2385834843":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6833/3300567104":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6832/3300567104":"2024-04-18T08:31:46.865019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 msgr2=0x7f34b006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 0x7f34b006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 msgr2=0x7f34d40b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00c400 tx=0x7f34cc00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 shutdown_connections 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f34b006ca10 0x7f34b006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d415ed10 0x7f34d40b6270 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34d40d2ab0 0x7f34d415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 --2- 172.21.15.67:0/690240955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34d40cf640 0x7f34d4155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 >> 172.21.15.67:0/690240955 conn(0x7f34d406cbd0 msgr2=0x7f34d40ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 shutdown_connections 2024-04-17T08:47:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.511+0000 7f34db200640 1 -- 172.21.15.67:0/690240955 wait complete. 2024-04-17T08:47:21.529 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:31:29.943141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:31:30.995201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:47:21.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:47:21.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.741+0000 7ff8ecc00640 1 Processor -- start 2024-04-17T08:47:21.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.741+0000 7ff8ecc00640 1 -- start start 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.742+0000 7ff8ecc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 0x7ff8e80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.742+0000 7ff8ecc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e80ce610 0x7ff8e80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.742+0000 7ff8ecc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e80cd960 con 0x7ff8e815ed00 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.742+0000 7ff8ecc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8e8071f70 con 0x7ff8e80ce610 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.742+0000 7ff8ecc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff8e8072250 con 0x7ff8e80cd040 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e80ce610 0x7ff8e80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e80ce610 0x7ff8e80b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58580/0 (socket says 172.21.15.67:58580) 2024-04-17T08:47:21.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 -- 172.21.15.67:0/538116750 learned_addr learned my addr 172.21.15.67:0/538116750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8e5e00640 1 --2- 172.21.15.67:0/538116750 >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 0x7ff8e80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 -- 172.21.15.67:0/538116750 >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 msgr2=0x7ff8e80cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 --2- 172.21.15.67:0/538116750 >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 0x7ff8e80cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 -- 172.21.15.67:0/538116750 >> v1:172.21.15.187:6789/0 conn(0x7ff8e815ed00 legacy=0x7ff8e80b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:21.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8dfe00640 1 -- 172.21.15.67:0/538116750 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e8072530 con 0x7ff8e80ce610 2024-04-17T08:47:21.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.743+0000 7ff8e5e00640 1 --2- 172.21.15.67:0/538116750 >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 0x7ff8e80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.744+0000 7ff8dfe00640 1 --2- 172.21.15.67:0/538116750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80ce610 0x7ff8e80b1bc0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff8e806c660 tx=0x7ff8d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ec3b42969f562 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.745+0000 7ff8e5400640 1 -- 172.21.15.67:0/538116750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0035020 con 0x7ff8e80ce610 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.745+0000 7ff8e5400640 1 -- 172.21.15.67:0/538116750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d0031870 con 0x7ff8e80ce610 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.745+0000 7ff8e5400640 1 -- 172.21.15.67:0/538116750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0031c40 con 0x7ff8e80ce610 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.745+0000 7ff8ecc00640 1 -- 172.21.15.67:0/538116750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80ce610 msgr2=0x7ff8e80b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.745+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/538116750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80ce610 0x7ff8e80b1bc0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7ff8e806c660 tx=0x7ff8d002b910 comp rx=0 tx=0).stop 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 -- 172.21.15.67:0/538116750 shutdown_connections 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/538116750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80ce610 0x7ff8e80b1bc0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/538116750 >> v2:172.21.15.187:3300/0 conn(0x7ff8e80cd040 0x7ff8e80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 -- 172.21.15.67:0/538116750 >> 172.21.15.67:0/538116750 conn(0x7ff8e806b8c0 msgr2=0x7ff8e806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 -- 172.21.15.67:0/538116750 shutdown_connections 2024-04-17T08:47:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.746+0000 7ff8ecc00640 1 -- 172.21.15.67:0/538116750 wait complete. 2024-04-17T08:47:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.747+0000 7ff8ecc00640 1 Processor -- start 2024-04-17T08:47:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.748+0000 7ff8ecc00640 1 -- start start 2024-04-17T08:47:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.748+0000 7ff8ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.748+0000 7ff8ecc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8e80ce610 0x7ff8e814e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8ecc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8e815ed00 0x7ff8e81550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8ecc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e80b2100 con 0x7ff8e80cd040 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8ecc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e8072af0 con 0x7ff8e815ed00 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8ecc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e80b50d0 con 0x7ff8e80ce610 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58588/0 (socket says 172.21.15.67:58588) 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 -- 172.21.15.67:0/1564790189 learned_addr learned my addr 172.21.15.67:0/1564790189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8dfe00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8e80ce610 0x7ff8e814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e6800640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8e815ed00 0x7ff8e81550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8e80ce610 msgr2=0x7ff8e814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8e80ce610 0x7ff8e814e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8e815ed00 msgr2=0x7ff8e81550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8e815ed00 0x7ff8e81550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.749+0000 7ff8e5e00640 1 -- 172.21.15.67:0/1564790189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e815a100 con 0x7ff8e80cd040 2024-04-17T08:47:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8e5e00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c7f0 tx=0x7ff8d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8015070 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8007a30 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e815a3e0 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8007e00 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.750+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e815a9a0 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.751+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e8068b90 con 0x7ff8e80cd040 2024-04-17T08:47:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.752+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7ff8d8018070 con 0x7ff8e80cd040 2024-04-17T08:47:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.753+0000 7ff8de000640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 0x7ff8c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.753+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7ff8d808c0b0 con 0x7ff8e80cd040 2024-04-17T08:47:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.753+0000 7ff8dfe00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 msgr2=0x7ff8c406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:21.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.753+0000 7ff8dfe00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 0x7ff8c406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.757+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff8d8059fe0 con 0x7ff8e80cd040 2024-04-17T08:47:21.817 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.817+0000 7fb301e3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:47:21.870 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.870+0000 7fb301e3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:47:21.885 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:21.886+0000 7f253083f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:47:21.920 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.920+0000 7fb301e3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:47:21.941 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:21.942+0000 7f253083f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:47:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.954+0000 7ff8dfe00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 msgr2=0x7ff8c406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.954+0000 7ff8dfe00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 0x7ff8c406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:21.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.962+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8e815ad10 con 0x7ff8e80cd040 2024-04-17T08:47:21.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.963+0000 7ff8de000640 1 -- 172.21.15.67:0/1564790189 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) ==== 93+0+10 (secure 0 0 0) 0x7ff8d805ee20 con 0x7ff8e80cd040 2024-04-17T08:47:21.962 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 msgr2=0x7ff8c406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 0x7ff8c406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 msgr2=0x7ff8e8156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c7f0 tx=0x7ff8d800cbf0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 shutdown_connections 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7ff8c406ca10 0x7ff8c406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8e815ed00 0x7ff8e81550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8e80ce610 0x7ff8e814e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 --2- 172.21.15.67:0/1564790189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e80cd040 0x7ff8e8156800 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.965+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 >> 172.21.15.67:0/1564790189 conn(0x7ff8e806b8c0 msgr2=0x7ff8e815dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:21.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.966+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 shutdown_connections 2024-04-17T08:47:21.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:21.966+0000 7ff8ecc00640 1 -- 172.21.15.67:0/1564790189 wait complete. 2024-04-17T08:47:21.980 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:21.980+0000 7fb301e3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:47:21.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:47:21.998 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:21.999+0000 7f253083f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:47:22.028 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.029+0000 7fb301e3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:47:22.064 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.065+0000 7f253083f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:47:22.118 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.119+0000 7f253083f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:47:22.156 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.157+0000 7fb301e3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:47:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.187+0000 7f7419c00640 1 Processor -- start 2024-04-17T08:47:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.187+0000 7f7419c00640 1 -- start start 2024-04-17T08:47:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f7419c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f74140cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f7419c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f741406c270 0x7f741406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f7419c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7414146360 con 0x7f74140cd750 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f7419c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f74140cdbb0 con 0x7f741406c270 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f7419c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74140727a0 con 0x7f7414145f80 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f740be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74140cd750 0x7f741406bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49222/0 (socket says 172.21.15.67:49222) 2024-04-17T08:47:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.189+0000 7f740be00640 1 -- 172.21.15.67:0/705255529 learned_addr learned my addr 172.21.15.67:0/705255529 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7419200640 1 --2- 172.21.15.67:0/705255529 >> v2:172.21.15.187:3300/0 conn(0x7f741406c270 0x7f741406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413e00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f74140cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626668323 0 0) 0x7f7414146360 con 0x7f74140cd750 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f8003520 con 0x7f74140cd750 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1548562263 0 0) 0x7f73f8003520 con 0x7f74140cd750 2024-04-17T08:47:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 msgr2=0x7f74140cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f74140cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 >> v2:172.21.15.187:3300/0 conn(0x7f741406c270 msgr2=0x7f741406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 --2- 172.21.15.67:0/705255529 >> v2:172.21.15.187:3300/0 conn(0x7f741406c270 0x7f741406e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7414071e70 con 0x7f74140cd750 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f73fc001670 con 0x7f74140cd750 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 >> v1:172.21.15.67:6789/0 conn(0x7f74140cd750 legacy=0x7f741406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73f80039b0 0x7f73f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f73f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73f8004d40 0x7f73f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74140727a0 con 0x7f73f80039b0 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f73f800b9c0 con 0x7f73f8004d40 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.190+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73f800bb70 con 0x7f7414145f80 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73f80039b0 0x7f73f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7419200640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73f8004d40 0x7f73f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f740be00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f73f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 -- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 msgr2=0x7f73f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7414145f80 0x7f73f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 -- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73f8004d40 msgr2=0x7f73f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73f8004d40 0x7f73f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.191+0000 7f7413e00640 1 -- 172.21.15.67:0/705255529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f8003520 con 0x7f73f80039b0 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.192+0000 7f7413e00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73f80039b0 0x7f73f8003dd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f7404009ac0 tx=0x7f73f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a33e31200332bea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404039070 con 0x7f73f80039b0 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f740402c6b0 con 0x7f73f80039b0 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7413400640 1 -- 172.21.15.67:0/705255529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404009690 con 0x7f73f80039b0 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7419c00640 1 -- 172.21.15.67:0/705255529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73f80039b0 msgr2=0x7f73f8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7419c00640 1 --2- 172.21.15.67:0/705255529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73f80039b0 0x7f73f8003dd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f7404009ac0 tx=0x7f73f8005830 comp rx=0 tx=0).stop 2024-04-17T08:47:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7419200640 1 -- 172.21.15.67:0/705255529 reap_dead start 2024-04-17T08:47:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7419c00640 1 -- 172.21.15.67:0/705255529 shutdown_connections 2024-04-17T08:47:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.193+0000 7f7419c00640 1 -- 172.21.15.67:0/705255529 >> 172.21.15.67:0/705255529 conn(0x7f74140b2190 msgr2=0x7f74140b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.194+0000 7f7419c00640 1 -- 172.21.15.67:0/705255529 shutdown_connections 2024-04-17T08:47:22.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.194+0000 7f7419c00640 1 -- 172.21.15.67:0/705255529 wait complete. 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.195+0000 7f7419c00640 1 Processor -- start 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.195+0000 7f7419c00640 1 -- start start 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 0x7f741413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 0x7f741413c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7414072c30 con 0x7f74140b3a10 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7414072430 con 0x7f74140b3590 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.196+0000 7f7419c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7414072db0 con 0x7f73f80039b0 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58602/0 (socket says 172.21.15.67:58602) 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 -- 172.21.15.67:0/2839447468 learned_addr learned my addr 172.21.15.67:0/2839447468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7413e00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 0x7f741413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f740be00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 0x7f741413c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 msgr2=0x7f741413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 0x7f741413f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 msgr2=0x7f741413c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 0x7f741413c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 -- 172.21.15.67:0/2839447468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741413fb20 con 0x7f74140b3a10 2024-04-17T08:47:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f740be00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 0x7f741413c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7413e00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 0x7f741413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.197+0000 7f7419200640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f741406c9c0 tx=0x7f740c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.198+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c011db0 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.198+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f740c01b070 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.198+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c003990 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.198+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74140b3f70 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.198+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74140b4530 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.199+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7414068a10 con 0x7f74140b3a10 2024-04-17T08:47:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.200+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7f740c003b30 con 0x7f74140b3a10 2024-04-17T08:47:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.201+0000 7f7410c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 0x7f73f006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.201+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f740c08c770 con 0x7f74140b3a10 2024-04-17T08:47:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.201+0000 7f7413e00640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 msgr2=0x7f73f006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.201+0000 7f7413e00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 0x7f73f006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:22.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.205+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f740c003e20 con 0x7f74140b3a10 2024-04-17T08:47:22.220 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.220+0000 7fb301e3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:22.260 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.260+0000 7f253083f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:47:22.271 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.272+0000 7fb301e3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:47:22.327 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.328+0000 7f253083f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:22.328 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.328+0000 7fb301e3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:47:22.378 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.378+0000 7fb301e3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:47:22.380 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.381+0000 7f253083f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:47:22.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.402+0000 7f7413e00640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 msgr2=0x7f73f006ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:22.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.402+0000 7f7413e00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 0x7f73f006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:22.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.405+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f741406ccd0 con 0x7f74140b3a10 2024-04-17T08:47:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.406+0000 7f7410c00640 1 -- 172.21.15.67:0/2839447468 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) ==== 104+0+11 (secure 0 0 0) 0x7f740c05a6b0 con 0x7f74140b3a10 2024-04-17T08:47:22.405 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 msgr2=0x7f73f006ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 0x7f73f006ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 msgr2=0x7f741413ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f741406c9c0 tx=0x7f740c0115a0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 shutdown_connections 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f73f006ca40 0x7f73f006ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74140b3a10 0x7f741413ccd0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74140b3590 0x7f741413c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 --2- 172.21.15.67:0/2839447468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73f80039b0 0x7f741413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.408+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 >> 172.21.15.67:0/2839447468 conn(0x7f74140b2190 msgr2=0x7f741415f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.409+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 shutdown_connections 2024-04-17T08:47:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.409+0000 7f7419c00640 1 -- 172.21.15.67:0/2839447468 wait complete. 2024-04-17T08:47:22.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:47:22.443 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.444+0000 7f253083f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:47:22.498 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:22.499+0000 7f253083f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:47:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.637+0000 7f3ec3200640 1 Processor -- start 2024-04-17T08:47:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.638+0000 7f3ec3200640 1 -- start start 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ec3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0ce3a0 0x7f3ebc0b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ec3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ebc0d0580 0x7f3ebc162ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ec3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ebc15f520 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ec3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3ebc0cfc90 con 0x7f3ebc0d0580 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ec3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3ebc15fe50 con 0x7f3ebc0ce3a0 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ebbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ebc0d01a0 0x7f3ebc1627c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49226/0 (socket says 172.21.15.67:49226) 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.639+0000 7f3ebbe00640 1 -- 172.21.15.67:0/2058776200 learned_addr learned my addr 172.21.15.67:0/2058776200 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ec1200640 1 --2- 172.21.15.67:0/2058776200 >> v2:172.21.15.67:3300/0 conn(0x7f3ebc0d0580 0x7f3ebc162ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3eb3e00640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0ce3a0 0x7f3ebc0b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ebb400640 1 -- 172.21.15.67:0/2058776200 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340840045 0 0) 0x7f3ebc15f520 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ebb400640 1 -- 172.21.15.67:0/2058776200 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ea0003520 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ec1200640 1 -- 172.21.15.67:0/2058776200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0ce3a0 msgr2=0x7f3ebc0b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ec1200640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0ce3a0 0x7f3ebc0b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ec1200640 1 -- 172.21.15.67:0/2058776200 >> v1:172.21.15.67:6789/0 conn(0x7f3ebc0d01a0 legacy=0x7f3ebc1627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.640+0000 7f3ec1200640 1 -- 172.21.15.67:0/2058776200 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ebc15f800 con 0x7f3ebc0d0580 2024-04-17T08:47:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.641+0000 7f3ec1200640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d0580 0x7f3ebc162ed0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009d90 tx=0x7f3eb402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610dba359442dfc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ebb400640 1 -- 172.21.15.67:0/2058776200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4039070 con 0x7f3ebc0d0580 2024-04-17T08:47:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ebb400640 1 -- 172.21.15.67:0/2058776200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb402cbc0 con 0x7f3ebc0d0580 2024-04-17T08:47:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ebb400640 1 -- 172.21.15.67:0/2058776200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4034440 con 0x7f3ebc0d0580 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 -- 172.21.15.67:0/2058776200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d0580 msgr2=0x7f3ebc162ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d0580 0x7f3ebc162ed0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009d90 tx=0x7f3eb402be10 comp rx=0 tx=0).stop 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 -- 172.21.15.67:0/2058776200 shutdown_connections 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d0580 0x7f3ebc162ed0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 --2- 172.21.15.67:0/2058776200 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0ce3a0 0x7f3ebc0b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 -- 172.21.15.67:0/2058776200 >> 172.21.15.67:0/2058776200 conn(0x7f3ebc0cd880 msgr2=0x7f3ebc0cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 -- 172.21.15.67:0/2058776200 shutdown_connections 2024-04-17T08:47:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.642+0000 7f3ec3200640 1 -- 172.21.15.67:0/2058776200 wait complete. 2024-04-17T08:47:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.644+0000 7f3ec3200640 1 Processor -- start 2024-04-17T08:47:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.644+0000 7f3ec3200640 1 -- start start 2024-04-17T08:47:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ebc0ce3a0 0x7f3ebc1f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0d0580 0x7f3ebc150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc0cd5f0 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc15cc30 con 0x7f3ebc0ce3a0 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.645+0000 7f3ec3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3ebc15f1f0 con 0x7f3ebc0d0580 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58620/0 (socket says 172.21.15.67:58620) 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 -- 172.21.15.67:0/799750717 learned_addr learned my addr 172.21.15.67:0/799750717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3ec1200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0d0580 0x7f3ebc150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3ebbe00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ebc0ce3a0 0x7f3ebc1f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0d0580 msgr2=0x7f3ebc150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0d0580 0x7f3ebc150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ebc0ce3a0 msgr2=0x7f3ebc1f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ebc0ce3a0 0x7f3ebc1f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 -- 172.21.15.67:0/799750717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ebc15dfa0 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.646+0000 7f3eb3e00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400d5f0 tx=0x7f3ea400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.647+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40125f0 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.647+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea4019070 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.647+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40129d0 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.647+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ebc15e280 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.647+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ebc1f8e00 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.648+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ebc15e670 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.649+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7f3ea4012b70 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.650+0000 7f3eb8c00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 0x7f3e9806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.650+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6425+0+0 (secure 0 0 0) 0x7f3ea408c410 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.650+0000 7f3ebbe00640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 msgr2=0x7f3e9806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.650+0000 7f3ebbe00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 0x7f3e9806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.654+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3ea405a460 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.757 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:22.758+0000 7fb301e3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:47:22.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.854+0000 7f3ebbe00640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 msgr2=0x7f3e9806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:22.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.854+0000 7f3ebbe00640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 0x7f3e9806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.944+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ebc15e450 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.946+0000 7f3eb8c00640 1 -- 172.21.15.67:0/799750717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v198) ==== 100+0+11 (secure 0 0 0) 0x7f3ea4059d60 con 0x7f3ebc0d01a0 2024-04-17T08:47:22.945 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:47:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 msgr2=0x7f3e9806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 0x7f3e9806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 msgr2=0x7f3ebc15d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400d5f0 tx=0x7f3ea400dab0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 shutdown_connections 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f3e9806ca10 0x7f3e9806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3ebc0d0580 0x7f3ebc150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ebc0d01a0 0x7f3ebc15d550 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 --2- 172.21.15.67:0/799750717 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3ebc0ce3a0 0x7f3ebc1f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.952+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 >> 172.21.15.67:0/799750717 conn(0x7f3ebc0cd880 msgr2=0x7f3ebc0cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.953+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 shutdown_connections 2024-04-17T08:47:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:22.953+0000 7f3ec3200640 1 -- 172.21.15.67:0/799750717 wait complete. 2024-04-17T08:47:22.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:47:23.006 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.007+0000 7fb301e3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:47:23.011 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.012+0000 7f253083f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:47:23.115 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.115+0000 7fb301e3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:47:23.164 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.165+0000 7fb301e3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:47:23.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.168+0000 7fe98da00640 1 Processor -- start 2024-04-17T08:47:23.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.168+0000 7fe98da00640 1 -- start start 2024-04-17T08:47:23.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe988145ac0 0x7fe988071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9880ce3b0 0x7fe988072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 0x7fe9880ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe9880ce840 con 0x7fe988145ac0 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe9880cea70 con 0x7fe9880ce3b0 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe98da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe9880ced50 con 0x7fe988072bb0 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe986a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe988145ac0 0x7fe988071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe986a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe988145ac0 0x7fe988071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58636/0 (socket says 172.21.15.67:58636) 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe986a00640 1 -- 172.21.15.67:0/3070667567 learned_addr learned my addr 172.21.15.67:0/3070667567 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.170+0000 7fe987400640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 0x7fe9880ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe97fe00640 1 --2- 172.21.15.67:0/3070667567 >> v2:172.21.15.187:3300/0 conn(0x7fe9880ce3b0 0x7fe988072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:23.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe986a00640 1 -- 172.21.15.67:0/3070667567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 msgr2=0x7fe9880ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:23.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe986a00640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 0x7fe9880ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe986a00640 1 -- 172.21.15.67:0/3070667567 >> v2:172.21.15.187:3300/0 conn(0x7fe9880ce3b0 msgr2=0x7fe988072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe986a00640 1 --2- 172.21.15.67:0/3070667567 >> v2:172.21.15.187:3300/0 conn(0x7fe9880ce3b0 0x7fe988072670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe986a00640 1 -- 172.21.15.67:0/3070667567 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9880cf030 con 0x7fe988145ac0 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.171+0000 7fe987400640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 0x7fe9880ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.172+0000 7fe986a00640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988145ac0 0x7fe988071ea0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fe978009870 tx=0x7fe97802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4057cf6588c0f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.172+0000 7fe986000640 1 -- 172.21.15.67:0/3070667567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978035020 con 0x7fe988145ac0 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.172+0000 7fe986000640 1 -- 172.21.15.67:0/3070667567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe978031640 con 0x7fe988145ac0 2024-04-17T08:47:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.172+0000 7fe986000640 1 -- 172.21.15.67:0/3070667567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97802d040 con 0x7fe988145ac0 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 -- 172.21.15.67:0/3070667567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988145ac0 msgr2=0x7fe988071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988145ac0 0x7fe988071ea0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fe978009870 tx=0x7fe97802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 -- 172.21.15.67:0/3070667567 shutdown_connections 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe988072bb0 0x7fe9880ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 --2- 172.21.15.67:0/3070667567 >> v2:172.21.15.187:3300/0 conn(0x7fe9880ce3b0 0x7fe988072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 --2- 172.21.15.67:0/3070667567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988145ac0 0x7fe988071ea0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 -- 172.21.15.67:0/3070667567 >> 172.21.15.67:0/3070667567 conn(0x7fe988067ee0 msgr2=0x7fe98806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 -- 172.21.15.67:0/3070667567 shutdown_connections 2024-04-17T08:47:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.173+0000 7fe98da00640 1 -- 172.21.15.67:0/3070667567 wait complete. 2024-04-17T08:47:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.175+0000 7fe98da00640 1 Processor -- start 2024-04-17T08:47:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.175+0000 7fe98da00640 1 -- start start 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9880ce3b0 0x7fe9881f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe988145ac0 0x7fe988154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe988145ea0 con 0x7fe988072bb0 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe9880cf5f0 con 0x7fe988145ac0 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe98da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe9880cd3e0 con 0x7fe9880ce3b0 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe986a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe986a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58650/0 (socket says 172.21.15.67:58650) 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe986a00640 1 -- 172.21.15.67:0/3877632058 learned_addr learned my addr 172.21.15.67:0/3877632058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.176+0000 7fe97fe00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9880ce3b0 0x7fe9881f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9880ce3b0 msgr2=0x7fe9881f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9880ce3b0 0x7fe9881f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe988145ac0 msgr2=0x7fe988154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe988145ac0 0x7fe988154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 -- 172.21.15.67:0/3877632058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9880c9f70 con 0x7fe988072bb0 2024-04-17T08:47:23.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe986a00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fe978000c00 tx=0x7fe9780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978040070 con 0x7fe988072bb0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9780399f0 con 0x7fe988072bb0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978035040 con 0x7fe988072bb0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.177+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9880ca250 con 0x7fe988072bb0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.178+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9880ca810 con 0x7fe988072bb0 2024-04-17T08:47:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.179+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe988068ca0 con 0x7fe988072bb0 2024-04-17T08:47:23.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.180+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7fe97804b060 con 0x7fe988072bb0 2024-04-17T08:47:23.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.181+0000 7fe97ea00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 0x7fe96406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:23.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.181+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(198..198 src has 1..198) ==== 6513+0+0 (secure 0 0 0) 0x7fe9780ad9d0 con 0x7fe988072bb0 2024-04-17T08:47:23.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.181+0000 7fe97fe00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 msgr2=0x7fe96406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:23.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.181+0000 7fe97fe00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 0x7fe96406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:23.188 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.189+0000 7f253083f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:47:23.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.190+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe97807b920 con 0x7fe988072bb0 2024-04-17T08:47:23.291 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.292+0000 7fb301e3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:23.306 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.307+0000 7f253083f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:47:23.360 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.361+0000 7f253083f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:47:23.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.382+0000 7fe97fe00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 msgr2=0x7fe96406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:23.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.382+0000 7fe97fe00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 0x7fe96406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:23.397 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.398+0000 7fb301e3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:47:23.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.404+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe9880cc6c0 con 0x7fe988072bb0 2024-04-17T08:47:23.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.405+0000 7fe97ea00640 1 -- 172.21.15.67:0/3877632058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v198) ==== 81+0+952 (secure 0 0 0) 0x7fe97807b220 con 0x7fe988072bb0 2024-04-17T08:47:23.404 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2365426680 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.404 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1228337607 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.404 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/1430869695 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/1430869695 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/521765659 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1964974751 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3275456489 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/630850508 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/74160311 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3491826676 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1329453466 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2568338216 2024-04-18T08:42:21.244475+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1472858533 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2385834843 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/3300567104 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.405 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/3300567104 2024-04-18T08:31:46.865019+0000 2024-04-17T08:47:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:listed 16 entries 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 msgr2=0x7fe96406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 0x7fe96406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 msgr2=0x7fe9880cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7fe978000c00 tx=0x7fe9780046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 shutdown_connections 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fe96406ca10 0x7fe96406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe988145ac0 0x7fe988154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9880ce3b0 0x7fe9881f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 --2- 172.21.15.67:0/3877632058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe988072bb0 0x7fe9880cb010 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 >> 172.21.15.67:0/3877632058 conn(0x7fe988067ee0 msgr2=0x7fe9880d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 shutdown_connections 2024-04-17T08:47:23.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:23.407+0000 7fe98da00640 1 -- 172.21.15.67:0/3877632058 wait complete. 2024-04-17T08:47:23.423 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:47:23.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:23.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:47:23.496 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.497+0000 7f253083f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:23.564 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:47:23.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:23.564 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:47:23.572 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.572+0000 7fb301e3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:23.615 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.615+0000 7f253083f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:47:23.623 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.624+0000 7fb301e3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:23.655 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:47:23.655 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:47:23.655 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:23.706 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:47:23.706 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:47:23.706 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:23.706 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:23.759 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:47:23.759 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:47:23.759 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:47:23.759 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:23.767 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:23.765+0000 7fb301e3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:47:23.806 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.807+0000 7f253083f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:23.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:47:23.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:23.861 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:23.862+0000 7f253083f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:24.009 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:24.010+0000 7f253083f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:47:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.020+0000 7fa0ab000640 1 Processor -- start 2024-04-17T08:47:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.021+0000 7fa0ab000640 1 -- start start 2024-04-17T08:47:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0ab000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa0a415ec60 0x7fa0a40b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0ab000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a40d2d30 con 0x7fa0a40b6b30 2024-04-17T08:47:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0ab000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a40d3010 con 0x7fa0a40cfae0 2024-04-17T08:47:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0ab000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a40d49c0 con 0x7fa0a415ec60 2024-04-17T08:47:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0a9200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0a40b6b30 0x7fa0a40cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60018/0 (socket says 172.21.15.67:60018) 2024-04-17T08:47:24.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.022+0000 7fa0a9200640 1 -- 172.21.15.67:0/688688709 learned_addr learned my addr 172.21.15.67:0/688688709 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a9c00640 1 --2- 172.21.15.67:0/688688709 >> v2:172.21.15.187:3300/0 conn(0x7fa0a415ec60 0x7fa0a40b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946336164 0 0) 0x7fa0a40d2d30 con 0x7fa0a40b6b30 2024-04-17T08:47:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c003520 con 0x7fa0a40b6b30 2024-04-17T08:47:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3408321138 0 0) 0x7fa08c003520 con 0x7fa0a40b6b30 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 >> v2:172.21.15.187:3300/0 conn(0x7fa0a415ec60 msgr2=0x7fa0a40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 --2- 172.21.15.67:0/688688709 >> v2:172.21.15.187:3300/0 conn(0x7fa0a415ec60 0x7fa0a40b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 >> v1:172.21.15.187:6789/0 conn(0x7fa0a40cfae0 legacy=0x7fa0a40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40d32f0 con 0x7fa0a40b6b30 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa090001670 con 0x7fa0a40b6b30 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 >> v1:172.21.15.67:6789/0 conn(0x7fa0a40b6b30 legacy=0x7fa0a40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08c003a70 0x7fa08c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa08c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa08c005210 0x7fa08c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40d49c0 con 0x7fa08c004800 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c00c140 con 0x7fa08c005210 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.023+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c00c340 con 0x7fa08c003a70 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa08c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9c00640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08c003a70 0x7fa08c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0aa600640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa08c005210 0x7fa08c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 -- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08c003a70 msgr2=0x7fa08c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08c003a70 0x7fa08c003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 -- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa08c005210 msgr2=0x7fa08c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa08c005210 0x7fa08c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0a9200640 1 -- 172.21.15.67:0/688688709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c003520 con 0x7fa08c004800 2024-04-17T08:47:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.024+0000 7fa0aa600640 1 -- 172.21.15.67:0/688688709 reap_dead start 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.025+0000 7fa0a9200640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa08c004c80 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c0e0 tx=0x7fa08c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a0c2902c0991ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa090007c20 con 0x7fa08c004800 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa090030070 con 0x7fa08c004800 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0a3e00640 1 -- 172.21.15.67:0/688688709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa090005d70 con 0x7fa08c004800 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 -- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 msgr2=0x7fa08c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa08c004c80 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c0e0 tx=0x7fa08c005830 comp rx=0 tx=0).stop 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 -- 172.21.15.67:0/688688709 shutdown_connections 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 --2- 172.21.15.67:0/688688709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa08c004c80 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 -- 172.21.15.67:0/688688709 >> 172.21.15.67:0/688688709 conn(0x7fa0a40d15f0 msgr2=0x7fa0a40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 -- 172.21.15.67:0/688688709 shutdown_connections 2024-04-17T08:47:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.026+0000 7fa0ab000640 1 -- 172.21.15.67:0/688688709 wait complete. 2024-04-17T08:47:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.028+0000 7fa0ab000640 1 Processor -- start 2024-04-17T08:47:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.028+0000 7fa0ab000640 1 -- start start 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 0x7fa0a41471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 0x7fa0a4147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40b6f70 con 0x7fa08c004800 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40d3660 con 0x7fa0a41552c0 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0ab000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40d00c0 con 0x7fa0a4155740 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.029+0000 7fa0a9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58192/0 (socket says 172.21.15.67:58192) 2024-04-17T08:47:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 -- 172.21.15.67:0/3316504898 learned_addr learned my addr 172.21.15.67:0/3316504898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9200640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 0x7fa0a41471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0aa600640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 0x7fa0a4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 msgr2=0x7fa0a4147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 0x7fa0a4147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 msgr2=0x7fa0a41471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 0x7fa0a41471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 -- 172.21.15.67:0/3316504898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40cd930 con 0x7fa08c004800 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9200640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 0x7fa0a41471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0aa600640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 0x7fa0a4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.030+0000 7fa0a9c00640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa0980114e0 tx=0x7fa0980119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.031+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098015870 con 0x7fa08c004800 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.031+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa098015a10 con 0x7fa08c004800 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.031+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098015dc0 con 0x7fa08c004800 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.031+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a40cdc10 con 0x7fa08c004800 2024-04-17T08:47:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.031+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a40ce1d0 con 0x7fa08c004800 2024-04-17T08:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.032+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a4068a10 con 0x7fa08c004800 2024-04-17T08:47:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.033+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7fa098013400 con 0x7fa08c004800 2024-04-17T08:47:24.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.034+0000 7fa0a1600640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 0x7fa08406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.034+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 6497+0+0 (secure 0 0 0) 0x7fa09808c8a0 con 0x7fa08c004800 2024-04-17T08:47:24.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.034+0000 7fa0a9200640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 msgr2=0x7fa08406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:24.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.034+0000 7fa0a9200640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 0x7fa08406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:24.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.038+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa098019030 con 0x7fa08c004800 2024-04-17T08:47:24.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.235+0000 7fa0a9200640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 msgr2=0x7fa08406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:24.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.235+0000 7fa0a9200640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 0x7fa08406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:24.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.296+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0a415c490 con 0x7fa08c004800 2024-04-17T08:47:24.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.297+0000 7fa0a1600640 1 -- 172.21.15.67:0/3316504898 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7fa09805a930 con 0x7fa08c004800 2024-04-17T08:47:24.297 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:24.297 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:47:24.297 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:47:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 msgr2=0x7fa08406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 0x7fa08406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 msgr2=0x7fa0a40cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fa0980114e0 tx=0x7fa0980119a0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 shutdown_connections 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fa08406ca10 0x7fa08406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0a4155740 0x7fa0a4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0a41552c0 0x7fa0a41471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 --2- 172.21.15.67:0/3316504898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa08c004800 0x7fa0a40cd230 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 >> 172.21.15.67:0/3316504898 conn(0x7fa0a40d15f0 msgr2=0x7fa0a4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 shutdown_connections 2024-04-17T08:47:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.300+0000 7fa0ab000640 1 -- 172.21.15.67:0/3316504898 wait complete. 2024-04-17T08:47:24.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:24.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.568+0000 7fbfde400640 1 Processor -- start 2024-04-17T08:47:24.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.569+0000 7fbfde400640 1 -- start start 2024-04-17T08:47:24.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.570+0000 7fbfde400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfd80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.570+0000 7fbfde400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbfd806e020 0x7fbfd806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.570+0000 7fbfde400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd80ceb00 con 0x7fbfd806f670 2024-04-17T08:47:24.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.570+0000 7fbfde400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd80cede0 con 0x7fbfd806e020 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.570+0000 7fbfde400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd80cf0c0 con 0x7fbfd80cd020 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbfd806f670 0x7fbfd806fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60022/0 (socket says 172.21.15.67:60022) 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd7400640 1 -- 172.21.15.67:0/430117977 learned_addr learned my addr 172.21.15.67:0/430117977 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/430117977 >> v2:172.21.15.187:3300/0 conn(0x7fbfd806e020 0x7fbfd806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfd80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2950227515 0 0) 0x7fbfd80ceb00 con 0x7fbfd806f670 2024-04-17T08:47:24.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfbc003520 con 0x7fbfd806f670 2024-04-17T08:47:24.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1328427443 0 0) 0x7fbfbc003520 con 0x7fbfd806f670 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 msgr2=0x7fbfd80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfd80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 >> v2:172.21.15.187:3300/0 conn(0x7fbfd806e020 msgr2=0x7fbfd806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 --2- 172.21.15.67:0/430117977 >> v2:172.21.15.187:3300/0 conn(0x7fbfd806e020 0x7fbfd806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.571+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd80cf3a0 con 0x7fbfd806f670 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbfc8001670 con 0x7fbfd806f670 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 >> v1:172.21.15.67:6789/0 conn(0x7fbfd806f670 legacy=0x7fbfd806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfbc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfbc003f30 0x7fbfbc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfbc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80cede0 con 0x7fbfbc004d40 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80cf0c0 con 0x7fbfbc003f30 2024-04-17T08:47:24.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbfbc00b9c0 con 0x7fbfd80cd020 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfbc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7400640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfbc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfbc003f30 0x7fbfbc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7e00640 1 -- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 msgr2=0x7fbfbc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80cd020 0x7fbfbc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7e00640 1 -- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfbc003f30 msgr2=0x7fbfbc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.572+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfbc003f30 0x7fbfbc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.573+0000 7fbfd7e00640 1 -- 172.21.15.67:0/430117977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc003520 con 0x7fbfbc004d40 2024-04-17T08:47:24.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfbc0051c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009fc0 tx=0x7fbfbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2a38795d1c4a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0039070 con 0x7fbfbc004d40 2024-04-17T08:47:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd002cc40 con 0x7fbfbc004d40 2024-04-17T08:47:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfd6a00640 1 -- 172.21.15.67:0/430117977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0009c10 con 0x7fbfbc004d40 2024-04-17T08:47:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfde400640 1 -- 172.21.15.67:0/430117977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 msgr2=0x7fbfbc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfde400640 1 --2- 172.21.15.67:0/430117977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfbc0051c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009fc0 tx=0x7fbfbc005830 comp rx=0 tx=0).stop 2024-04-17T08:47:24.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.574+0000 7fbfd7e00640 1 -- 172.21.15.67:0/430117977 reap_dead start 2024-04-17T08:47:24.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.575+0000 7fbfde400640 1 -- 172.21.15.67:0/430117977 shutdown_connections 2024-04-17T08:47:24.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.575+0000 7fbfde400640 1 -- 172.21.15.67:0/430117977 >> 172.21.15.67:0/430117977 conn(0x7fbfd806bf50 msgr2=0x7fbfd80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:24.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.575+0000 7fbfde400640 1 -- 172.21.15.67:0/430117977 shutdown_connections 2024-04-17T08:47:24.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.575+0000 7fbfde400640 1 -- 172.21.15.67:0/430117977 wait complete. 2024-04-17T08:47:24.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.576+0000 7fbfde400640 1 Processor -- start 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.577+0000 7fbfde400640 1 -- start start 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.577+0000 7fbfde400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfde400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 0x7fbfd80caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfde400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfd81524e0 0x7fbfd8152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfde400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80ce8a0 con 0x7fbfbc004d40 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfde400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd80ce720 con 0x7fbfd81524e0 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfde400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd815cb00 con 0x7fbfd80caab0 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58210/0 (socket says 172.21.15.67:58210) 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 -- 172.21.15.67:0/2226742273 learned_addr learned my addr 172.21.15.67:0/2226742273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7e00640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfd81524e0 0x7fbfd8152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 0x7fbfd80caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:24.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 msgr2=0x7fbfd80caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 0x7fbfd80caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfd81524e0 msgr2=0x7fbfd8152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfd81524e0 0x7fbfd8152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfd7400640 1 -- 172.21.15.67:0/2226742273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd81f4510 con 0x7fbfbc004d40 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.578+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 0x7fbfd80caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:24.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfd7400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8001eb0 tx=0x7fbfc800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8019070 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc8005260 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd81f47d0 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8005650 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.579+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd81f4d90 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.580+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd8068b90 con 0x7fbfbc004d40 2024-04-17T08:47:24.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.581+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7fbfc801c070 con 0x7fbfbc004d40 2024-04-17T08:47:24.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.582+0000 7fbfcf400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 0x7fbfb406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:24.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.582+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 6497+0+0 (secure 0 0 0) 0x7fbfc808bb20 con 0x7fbfbc004d40 2024-04-17T08:47:24.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.582+0000 7fbfcfe00640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 msgr2=0x7fbfb406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:24.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.582+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 0x7fbfb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:24.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.586+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbfc8059a90 con 0x7fbfbc004d40 2024-04-17T08:47:24.632 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:24.633+0000 7fb301e3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.783+0000 7fbfcfe00640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 msgr2=0x7fbfb406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.783+0000 7fbfcfe00640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 0x7fbfb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.796+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfd8072aa0 con 0x7fbfbc004d40 2024-04-17T08:47:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.797+0000 7fbfcf400640 1 -- 172.21.15.67:0/2226742273 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) ==== 74+0+17821 (secure 0 0 0) 0x7fbfc8059390 con 0x7fbfbc004d40 2024-04-17T08:47:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:24.797 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":199,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:23.774358+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":199,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2365426680":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1228337607":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6833/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6832/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/521765659":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1964974751":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3275456489":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/630850508":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/74160311":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/3491826676":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1329453466":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2568338216":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1472858533":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2385834843":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6833/3300567104":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6832/3300567104":"2024-04-18T08:31:46.865019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 msgr2=0x7fbfb406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 0x7fbfb406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 msgr2=0x7fbfd814f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8001eb0 tx=0x7fbfc800ed70 comp rx=0 tx=0).stop 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 shutdown_connections 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7fbfb406ca10 0x7fbfb406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbfd81524e0 0x7fbfd8152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbfd80caab0 0x7fbfd80caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 --2- 172.21.15.67:0/2226742273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfbc004d40 0x7fbfd814f240 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:24.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.800+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 >> 172.21.15.67:0/2226742273 conn(0x7fbfd806bf50 msgr2=0x7fbfd80b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:24.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.801+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 shutdown_connections 2024-04-17T08:47:24.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:24.801+0000 7fbfde400640 1 -- 172.21.15.67:0/2226742273 wait complete. 2024-04-17T08:47:24.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:31:29.943141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:31:30.995201+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 199, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:47:24.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:47:24.916 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:24.917+0000 7f253083f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:47:24.940 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:24.940+0000 7fb301e3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:47:24.996 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:24.997+0000 7fb301e3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:47:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.021+0000 7f547fc00640 1 Processor -- start 2024-04-17T08:47:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.022+0000 7f547fc00640 1 -- start start 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.023+0000 7f547fc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f547815ec00 0x7f54780b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.023+0000 7f547fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f54780cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.023+0000 7f547fc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54780d4740 con 0x7f54780d1370 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.023+0000 7f547fc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f54780d3610 con 0x7f547815ec00 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.023+0000 7f547fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54780d2a00 con 0x7f54780cf870 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f547815ec00 0x7f54780b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f547815ec00 0x7f54780b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58218/0 (socket says 172.21.15.67:58218) 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 -- 172.21.15.67:0/107532349 learned_addr learned my addr 172.21.15.67:0/107532349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f5477e00640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f54780cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 -- 172.21.15.67:0/107532349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 msgr2=0x7f54780cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f54780cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 -- 172.21.15.67:0/107532349 >> v1:172.21.15.187:6789/0 conn(0x7f54780d1370 legacy=0x7f54781628f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f547d200640 1 -- 172.21.15.67:0/107532349 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54780d2ce0 con 0x7f547815ec00 2024-04-17T08:47:25.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.024+0000 7f5477e00640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f54780cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.025+0000 7f547d200640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f54780b7b60 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f546c009870 tx=0x7f546c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47e47092adde01d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f5477400640 1 -- 172.21.15.67:0/107532349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c0317c0 con 0x7f547815ec00 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f5477400640 1 -- 172.21.15.67:0/107532349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546c037070 con 0x7f547815ec00 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f5477400640 1 -- 172.21.15.67:0/107532349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c031b80 con 0x7f547815ec00 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 -- 172.21.15.67:0/107532349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 msgr2=0x7f54780b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f54780b7b60 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f546c009870 tx=0x7f546c02b860 comp rx=0 tx=0).stop 2024-04-17T08:47:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 -- 172.21.15.67:0/107532349 shutdown_connections 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f54780cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 --2- 172.21.15.67:0/107532349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f54780b7b60 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.026+0000 7f547fc00640 1 -- 172.21.15.67:0/107532349 >> 172.21.15.67:0/107532349 conn(0x7f547806ccb0 msgr2=0x7f547806d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.027+0000 7f547fc00640 1 -- 172.21.15.67:0/107532349 shutdown_connections 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.027+0000 7f547fc00640 1 -- 172.21.15.67:0/107532349 wait complete. 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.028+0000 7f547fc00640 1 Processor -- start 2024-04-17T08:47:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.028+0000 7f547fc00640 1 -- start start 2024-04-17T08:47:25.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f5478157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780d1370 0x7f54780ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f547815efe0 con 0x7f547815ec00 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f54780d32a0 con 0x7f54780d1370 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547fc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f547815f160 con 0x7f54780cf870 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.029+0000 7f547dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58230/0 (socket says 172.21.15.67:58230) 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 -- 172.21.15.67:0/1361631872 learned_addr learned my addr 172.21.15.67:0/1361631872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547d200640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f5478157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f5477e00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780d1370 0x7f54780ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 -- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 msgr2=0x7f5478157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f5478157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 -- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780d1370 msgr2=0x7f54780ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780d1370 0x7f54780ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 -- 172.21.15.67:0/1361631872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54781542f0 con 0x7f547815ec00 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.030+0000 7f547dc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f547000c7b0 tx=0x7f547000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.031+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470004b60 con 0x7f547815ec00 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.031+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5470004d00 con 0x7f547815ec00 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.031+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547000aa90 con 0x7f547815ec00 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.031+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54780ce800 con 0x7f547815ec00 2024-04-17T08:47:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.031+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547815b700 con 0x7f547815ec00 2024-04-17T08:47:25.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.032+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5478068b90 con 0x7f547815ec00 2024-04-17T08:47:25.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.033+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 113) ==== 91154+0+0 (secure 0 0 0) 0x7f547000ac30 con 0x7f547815ec00 2024-04-17T08:47:25.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.034+0000 7f5474c00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 0x7f545806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.034+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 6497+0+0 (secure 0 0 0) 0x7f547008c2f0 con 0x7f547815ec00 2024-04-17T08:47:25.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.034+0000 7f547d200640 1 -- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 msgr2=0x7f545806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/219728900 2024-04-17T08:47:25.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.034+0000 7f547d200640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 0x7f545806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:25.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.039+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5470020070 con 0x7f547815ec00 2024-04-17T08:47:25.127 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.128+0000 7fb301e3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:47:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.152+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 114) ==== 90751+0+0 (secure 0 0 0) 0x7f547008a4f0 con 0x7f547815ec00 2024-04-17T08:47:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.153+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 msgr2=0x7f545806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.153+0000 7f5474c00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 0x7f545806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27b000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27b000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27b000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27b000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27b000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.221 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:25.221+0000 7fb27e200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.245+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5478158df0 con 0x7f547815ec00 2024-04-17T08:47:25.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.246+0000 7f5474c00640 1 -- 172.21.15.67:0/1361631872 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) ==== 93+0+10 (secure 0 0 0) 0x7f5470004eb0 con 0x7f547815ec00 2024-04-17T08:47:25.246 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 msgr2=0x7f5478154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f547000c7b0 tx=0x7f547000cc70 comp rx=0 tx=0).stop 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 shutdown_connections 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:6832/219728900,v1:172.21.15.187:6833/219728900] conn(0x7f545806ca10 0x7f545806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f547815ec00 0x7f5478154930 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54780d1370 0x7f54780ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 --2- 172.21.15.67:0/1361631872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54780cf870 0x7f5478157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 >> 172.21.15.67:0/1361631872 conn(0x7f547806ccb0 msgr2=0x7f54780d38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 shutdown_connections 2024-04-17T08:47:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.249+0000 7f547fc00640 1 -- 172.21.15.67:0/1361631872 wait complete. 2024-04-17T08:47:25.255 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:25.256+0000 7f253083f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:47:25.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:47:25.318 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:25.318+0000 7f253083f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:47:25.458 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:25.459+0000 7f253083f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:47:25.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.473+0000 7feb92c00640 1 Processor -- start 2024-04-17T08:47:25.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.474+0000 7feb92c00640 1 -- start start 2024-04-17T08:47:25.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb92c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feb8c142000 0x7feb8c1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb92c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb8c0aef70 con 0x7feb8c1417f0 2024-04-17T08:47:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb92c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb8c0af250 con 0x7feb8c0afd30 2024-04-17T08:47:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb92c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7feb8c163270 con 0x7feb8c142000 2024-04-17T08:47:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb91800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feb8c1417f0 0x7feb8c162b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60024/0 (socket says 172.21.15.67:60024) 2024-04-17T08:47:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.475+0000 7feb91800640 1 -- 172.21.15.67:0/3638584711 learned_addr learned my addr 172.21.15.67:0/3638584711 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb90e00640 1 --2- 172.21.15.67:0/3638584711 >> v2:172.21.15.187:3300/0 conn(0x7feb8c142000 0x7feb8c1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573725979 0 0) 0x7feb8c0aef70 con 0x7feb8c1417f0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb74003520 con 0x7feb8c1417f0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1133948186 0 0) 0x7feb74003520 con 0x7feb8c1417f0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 >> v2:172.21.15.187:3300/0 conn(0x7feb8c142000 msgr2=0x7feb8c1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 --2- 172.21.15.67:0/3638584711 >> v2:172.21.15.187:3300/0 conn(0x7feb8c142000 0x7feb8c1424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 >> v1:172.21.15.187:6789/0 conn(0x7feb8c0afd30 legacy=0x7feb8c06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8c0af530 con 0x7feb8c1417f0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7feb80001670 con 0x7feb8c1417f0 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.476+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 >> v1:172.21.15.67:6789/0 conn(0x7feb8c1417f0 legacy=0x7feb8c162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740039b0 0x7feb74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb74004800 0x7feb74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 0x7feb7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb8c163270 con 0x7feb740051c0 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feb7400c060 con 0x7feb74004800 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb7400c260 con 0x7feb740039b0 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 0x7feb7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb90e00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb74004800 0x7feb74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 -- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740039b0 msgr2=0x7feb74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740039b0 0x7feb74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 -- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb74004800 msgr2=0x7feb74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb74004800 0x7feb74004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.477+0000 7feb92200640 1 -- 172.21.15.67:0/3638584711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb74003520 con 0x7feb740051c0 2024-04-17T08:47:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.478+0000 7feb92200640 1 -- 172.21.15.67:0/3638584711 reap_dead start 2024-04-17T08:47:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.478+0000 7feb92200640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 0x7feb7400b9c0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7feb84001e50 tx=0x7feb74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187d7362464f1cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84001a70 con 0x7feb740051c0 2024-04-17T08:47:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb84003bb0 con 0x7feb740051c0 2024-04-17T08:47:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb8be00640 1 -- 172.21.15.67:0/3638584711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb840050d0 con 0x7feb740051c0 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 -- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 msgr2=0x7feb7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 0x7feb7400b9c0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7feb84001e50 tx=0x7feb74005830 comp rx=0 tx=0).stop 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 -- 172.21.15.67:0/3638584711 shutdown_connections 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 --2- 172.21.15.67:0/3638584711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb740051c0 0x7feb7400b9c0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 -- 172.21.15.67:0/3638584711 >> 172.21.15.67:0/3638584711 conn(0x7feb8c072460 msgr2=0x7feb8c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 -- 172.21.15.67:0/3638584711 shutdown_connections 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.479+0000 7feb92c00640 1 -- 172.21.15.67:0/3638584711 wait complete. 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.481+0000 7feb92c00640 1 Processor -- start 2024-04-17T08:47:25.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.482+0000 7feb92c00640 1 -- start start 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.482+0000 7feb92c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740051c0 0x7feb8c161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 0x7feb8c15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb8c141d50 con 0x7feb8c153e80 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feb8c141bd0 con 0x7feb8c137430 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feb8c0aeb90 con 0x7feb740051c0 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58252/0 (socket says 172.21.15.67:58252) 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 -- 172.21.15.67:0/3912313621 learned_addr learned my addr 172.21.15.67:0/3912313621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92200640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 0x7feb8c15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb91800640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740051c0 0x7feb8c161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 -- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740051c0 msgr2=0x7feb8c161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740051c0 0x7feb8c161820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 -- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 msgr2=0x7feb8c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 0x7feb8c15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb90e00640 1 -- 172.21.15.67:0/3912313621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8c151880 con 0x7feb8c153e80 2024-04-17T08:47:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.483+0000 7feb92200640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 0x7feb8c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb90e00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7feb780107f0 tx=0x7feb78010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7800f040 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7800eb60 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7801c400 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb8c0b0a90 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.484+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb8c0b1050 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.485+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb8c0755d0 con 0x7feb8c153e80 2024-04-17T08:47:25.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.486+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 114) ==== 90751+0+0 (secure 0 0 0) 0x7feb7801c5a0 con 0x7feb8c153e80 2024-04-17T08:47:25.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.487+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7feb7808c1b0 con 0x7feb8c153e80 2024-04-17T08:47:25.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.491+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7feb78018030 con 0x7feb8c153e80 2024-04-17T08:47:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.692+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feb8c073760 con 0x7feb8c153e80 2024-04-17T08:47:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.693+0000 7feb89600640 1 -- 172.21.15.67:0/3912313621 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v200) ==== 104+0+11 (secure 0 0 0) 0x7feb7805a500 con 0x7feb8c153e80 2024-04-17T08:47:25.693 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 msgr2=0x7feb8c151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7feb780107f0 tx=0x7feb78010cb0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 shutdown_connections 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb8c137430 0x7feb8c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb8c153e80 0x7feb8c151340 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 --2- 172.21.15.67:0/3912313621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feb740051c0 0x7feb8c161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 >> 172.21.15.67:0/3912313621 conn(0x7feb8c072460 msgr2=0x7feb8c06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 shutdown_connections 2024-04-17T08:47:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.695+0000 7feb92c00640 1 -- 172.21.15.67:0/3912313621 wait complete. 2024-04-17T08:47:25.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:47:25.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:47:25.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:47:25.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.933+0000 7f89d3200640 1 Processor -- start 2024-04-17T08:47:25.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.933+0000 7f89d3200640 1 -- start start 2024-04-17T08:47:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.934+0000 7f89d3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89cc15ec00 0x7f89cc0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.934+0000 7f89d3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0cf870 0x7f89cc0cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89d3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89cc0d4740 con 0x7f89cc0d1370 2024-04-17T08:47:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89d3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89cc0d3610 con 0x7f89cc0cf870 2024-04-17T08:47:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89d3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89cc0d2a00 con 0x7f89cc15ec00 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0cf870 0x7f89cc0cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0cf870 0x7f89cc0cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58262/0 (socket says 172.21.15.67:58262) 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 -- 172.21.15.67:0/2981253962 learned_addr learned my addr 172.21.15.67:0/2981253962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cbe00640 1 --2- 172.21.15.67:0/2981253962 >> v2:172.21.15.187:3300/0 conn(0x7f89cc15ec00 0x7f89cc0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 -- 172.21.15.67:0/2981253962 >> v2:172.21.15.187:3300/0 conn(0x7f89cc15ec00 msgr2=0x7f89cc0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 --2- 172.21.15.67:0/2981253962 >> v2:172.21.15.187:3300/0 conn(0x7f89cc15ec00 0x7f89cc0b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.935+0000 7f89cb400640 1 -- 172.21.15.67:0/2981253962 >> v1:172.21.15.187:6789/0 conn(0x7f89cc0d1370 legacy=0x7f89cc1628f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.936+0000 7f89cb400640 1 -- 172.21.15.67:0/2981253962 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89cc0d2ce0 con 0x7f89cc0cf870 2024-04-17T08:47:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.937+0000 7f89cb400640 1 --2- 172.21.15.67:0/2981253962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0cf870 0x7f89cc0cfcd0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0d59e0 tx=0x7f89b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15dad32b4b8b1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.937+0000 7f89caa00640 1 -- 172.21.15.67:0/2981253962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8035020 con 0x7f89cc0cf870 2024-04-17T08:47:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.937+0000 7f89caa00640 1 -- 172.21.15.67:0/2981253962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8031870 con 0x7f89cc0cf870 2024-04-17T08:47:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.937+0000 7f89caa00640 1 -- 172.21.15.67:0/2981253962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8031c40 con 0x7f89cc0cf870 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 -- 172.21.15.67:0/2981253962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0cf870 msgr2=0x7f89cc0cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 --2- 172.21.15.67:0/2981253962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0cf870 0x7f89cc0cfcd0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0d59e0 tx=0x7f89b802b910 comp rx=0 tx=0).stop 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 -- 172.21.15.67:0/2981253962 shutdown_connections 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 --2- 172.21.15.67:0/2981253962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0cf870 0x7f89cc0cfcd0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 --2- 172.21.15.67:0/2981253962 >> v2:172.21.15.187:3300/0 conn(0x7f89cc15ec00 0x7f89cc0b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 -- 172.21.15.67:0/2981253962 >> 172.21.15.67:0/2981253962 conn(0x7f89cc06ccb0 msgr2=0x7f89cc06d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 -- 172.21.15.67:0/2981253962 shutdown_connections 2024-04-17T08:47:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.938+0000 7f89d3200640 1 -- 172.21.15.67:0/2981253962 wait complete. 2024-04-17T08:47:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.940+0000 7f89d3200640 1 Processor -- start 2024-04-17T08:47:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.940+0000 7f89d3200640 1 -- start start 2024-04-17T08:47:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 0x7f89cc0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89cc15ec00 0x7f89cc155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc15efe0 con 0x7f89cc0d1370 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89cc0d32a0 con 0x7f89cc0cf870 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89cc15f160 con 0x7f89cc15ec00 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58268/0 (socket says 172.21.15.67:58268) 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89cb400640 1 -- 172.21.15.67:0/1983748416 learned_addr learned my addr 172.21.15.67:0/1983748416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89cbe00640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 0x7f89cc0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.941+0000 7f89d1200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89cc15ec00 0x7f89cc155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 -- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89cc15ec00 msgr2=0x7f89cc155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89cc15ec00 0x7f89cc155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 -- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 msgr2=0x7f89cc0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 0x7f89cc0c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 -- 172.21.15.67:0/1983748416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89cc15cd90 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cbe00640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 0x7f89cc0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89cb400640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f89b8000c00 tx=0x7f89b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8035040 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8030d00 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.942+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80456a0 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.943+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89cc147e60 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.943+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89cc15d310 con 0x7f89cc0d1370 2024-04-17T08:47:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.944+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89cc0d05e0 con 0x7f89cc0d1370 2024-04-17T08:47:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.945+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 114) ==== 90751+0+0 (secure 0 0 0) 0x7f89b8038020 con 0x7f89cc0d1370 2024-04-17T08:47:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.946+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7f89b80adf60 con 0x7f89cc0d1370 2024-04-17T08:47:25.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:25.951+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f89cc0d05e0 con 0x7f89cc0d1370 2024-04-17T08:47:26.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.156+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f89cc068b90 con 0x7f89cc0d1370 2024-04-17T08:47:26.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.161+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 115) ==== 90878+0+0 (secure 0 0 0) 0x7f89b80af020 con 0x7f89cc0d1370 2024-04-17T08:47:26.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.162+0000 7f89c3e00640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f89ac072380 0x7f89ac074770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.162+0000 7f89c3e00640 1 -- 172.21.15.67:0/1983748416 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v200) ==== 100+0+11 (secure 0 0 0) 0x7f89b8075020 con 0x7f89cc0d1370 2024-04-17T08:47:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:47:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f89ac072380 msgr2=0x7f89ac074770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f89ac072380 0x7f89ac074770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 msgr2=0x7f89cc15db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f89b8000c00 tx=0x7f89b80046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 shutdown_connections 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f89ac072380 0x7f89ac074770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89cc15ec00 0x7f89cc155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0d1370 0x7f89cc15db10 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 --2- 172.21.15.67:0/1983748416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89cc0cf870 0x7f89cc0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 >> 172.21.15.67:0/1983748416 conn(0x7f89cc06ccb0 msgr2=0x7f89cc0cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 shutdown_connections 2024-04-17T08:47:26.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.165+0000 7f89d3200640 1 -- 172.21.15.67:0/1983748416 wait complete. 2024-04-17T08:47:26.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.397+0000 7fc261a00640 1 Processor -- start 2024-04-17T08:47:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.397+0000 7fc261a00640 1 -- start start 2024-04-17T08:47:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc261a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc25c0d18d0 0x7fc25c0d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc261a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25c0d3170 con 0x7fc25c0d14f0 2024-04-17T08:47:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc261a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25c0d3450 con 0x7fc25c0ba470 2024-04-17T08:47:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc261a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc25c0d3730 con 0x7fc25c0d18d0 2024-04-17T08:47:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc25be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc25c0d14f0 0x7fc25c0d2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60034/0 (socket says 172.21.15.67:60034) 2024-04-17T08:47:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc25be00640 1 -- 172.21.15.67:0/3815458671 learned_addr learned my addr 172.21.15.67:0/3815458671 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.398+0000 7fc25b400640 1 --2- 172.21.15.67:0/3815458671 >> v2:172.21.15.67:3300/0 conn(0x7fc25c0d18d0 0x7fc25c0d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.399+0000 7fc25b400640 1 -- 172.21.15.67:0/3815458671 >> v1:172.21.15.187:6789/0 conn(0x7fc25c0ba470 legacy=0x7fc25c15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:26.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.399+0000 7fc25b400640 1 -- 172.21.15.67:0/3815458671 >> v1:172.21.15.67:6789/0 conn(0x7fc25c0d14f0 legacy=0x7fc25c0d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.399+0000 7fc25b400640 1 -- 172.21.15.67:0/3815458671 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25c0d3a10 con 0x7fc25c0d18d0 2024-04-17T08:47:26.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.400+0000 7fc25b400640 1 --2- 172.21.15.67:0/3815458671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d18d0 0x7fc25c0d4210 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc2480097a0 tx=0x7fc24802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a8e2e6b8f19aec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.400+0000 7fc25aa00640 1 -- 172.21.15.67:0/3815458671 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248037070 con 0x7fc25c0d18d0 2024-04-17T08:47:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.400+0000 7fc25aa00640 1 -- 172.21.15.67:0/3815458671 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc248030710 con 0x7fc25c0d18d0 2024-04-17T08:47:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.400+0000 7fc25aa00640 1 -- 172.21.15.67:0/3815458671 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248030ae0 con 0x7fc25c0d18d0 2024-04-17T08:47:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 -- 172.21.15.67:0/3815458671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d18d0 msgr2=0x7fc25c0d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 --2- 172.21.15.67:0/3815458671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d18d0 0x7fc25c0d4210 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc2480097a0 tx=0x7fc24802b710 comp rx=0 tx=0).stop 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 -- 172.21.15.67:0/3815458671 shutdown_connections 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 --2- 172.21.15.67:0/3815458671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d18d0 0x7fc25c0d4210 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 -- 172.21.15.67:0/3815458671 >> 172.21.15.67:0/3815458671 conn(0x7fc25c06cbd0 msgr2=0x7fc25c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 -- 172.21.15.67:0/3815458671 shutdown_connections 2024-04-17T08:47:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.401+0000 7fc261a00640 1 -- 172.21.15.67:0/3815458671 wait complete. 2024-04-17T08:47:26.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.403+0000 7fc261a00640 1 Processor -- start 2024-04-17T08:47:26.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.403+0000 7fc261a00640 1 -- start start 2024-04-17T08:47:26.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc25c0ba470 0x7fc25c0cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc25c0d18d0 0x7fc25c1f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c15ed30 con 0x7fc25c0d14f0 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc25c15ebb0 con 0x7fc25c0ba470 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc261a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc25c146af0 con 0x7fc25c0d18d0 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc25b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc25b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58298/0 (socket says 172.21.15.67:58298) 2024-04-17T08:47:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.404+0000 7fc25b400640 1 -- 172.21.15.67:0/2386132947 learned_addr learned my addr 172.21.15.67:0/2386132947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 -- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc25c0d18d0 msgr2=0x7fc25c1f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc25c0d18d0 0x7fc25c1f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25be00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc25c0ba470 0x7fc25c0cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 -- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc25c0ba470 msgr2=0x7fc25c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc25c0ba470 0x7fc25c0cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 -- 172.21.15.67:0/2386132947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25c0cd9f0 con 0x7fc25c0d14f0 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc25b400640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc2480097a0 tx=0x7fc2480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248042070 con 0x7fc25c0d14f0 2024-04-17T08:47:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc248030ec0 con 0x7fc25c0d14f0 2024-04-17T08:47:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248037040 con 0x7fc25c0d14f0 2024-04-17T08:47:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.405+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25c0cdcd0 con 0x7fc25c0d14f0 2024-04-17T08:47:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.406+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25c0ce290 con 0x7fc25c0d14f0 2024-04-17T08:47:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.407+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc25c068a10 con 0x7fc25c0d14f0 2024-04-17T08:47:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.408+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 115) ==== 90878+0+0 (secure 0 0 0) 0x7fc248045070 con 0x7fc25c0d14f0 2024-04-17T08:47:26.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.408+0000 7fc253e00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 0x7fc23c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.409+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7fc2480adb10 con 0x7fc25c0d14f0 2024-04-17T08:47:26.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.409+0000 7fc25be00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 0x7fc23c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.410+0000 7fc25be00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 0x7fc23c06ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc24c007fd0 tx=0x7fc24c009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.413+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc24807be90 con 0x7fc25c0d14f0 2024-04-17T08:47:26.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.664+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc25c0c8140 con 0x7fc25c0d14f0 2024-04-17T08:47:26.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.665+0000 7fc253e00640 1 -- 172.21.15.67:0/2386132947 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7fc24807b790 con 0x7fc25c0d14f0 2024-04-17T08:47:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:26.665 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:47:26.665 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 msgr2=0x7fc23c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 0x7fc23c06ec00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc24c007fd0 tx=0x7fc24c009040 comp rx=0 tx=0).stop 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 msgr2=0x7fc25c1465b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc2480097a0 tx=0x7fc2480046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 shutdown_connections 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc23c06c740 0x7fc23c06ec00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc25c0d18d0 0x7fc25c1f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc25c0d14f0 0x7fc25c1465b0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 --2- 172.21.15.67:0/2386132947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc25c0ba470 0x7fc25c0cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 >> 172.21.15.67:0/2386132947 conn(0x7fc25c06cbd0 msgr2=0x7fc25c0b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 shutdown_connections 2024-04-17T08:47:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.668+0000 7fc261a00640 1 -- 172.21.15.67:0/2386132947 wait complete. 2024-04-17T08:47:26.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:26.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.891+0000 7fc450a00640 1 Processor -- start 2024-04-17T08:47:26.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.891+0000 7fc450a00640 1 -- start start 2024-04-17T08:47:26.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc450a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc44c160280 0x7fc44c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc450a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44c0cd4a0 con 0x7fc44c06b8c0 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc450a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44c1607f0 con 0x7fc44c141ed0 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc450a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc44c15ce30 con 0x7fc44c160280 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc44a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc44c06b8c0 0x7fc44c0cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60050/0 (socket says 172.21.15.67:60050) 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc44a800640 1 -- 172.21.15.67:0/502860765 learned_addr learned my addr 172.21.15.67:0/502860765 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:26.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.892+0000 7fc441e00640 1 --2- 172.21.15.67:0/502860765 >> v2:172.21.15.67:3300/0 conn(0x7fc44c160280 0x7fc44c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.893+0000 7fc449e00640 1 -- 172.21.15.67:0/502860765 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395750120 0 0) 0x7fc44c0cd4a0 con 0x7fc44c06b8c0 2024-04-17T08:47:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.893+0000 7fc449e00640 1 -- 172.21.15.67:0/502860765 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc430003520 con 0x7fc44c06b8c0 2024-04-17T08:47:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.893+0000 7fc441e00640 1 -- 172.21.15.67:0/502860765 >> v1:172.21.15.187:6789/0 conn(0x7fc44c141ed0 legacy=0x7fc44c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.893+0000 7fc441e00640 1 -- 172.21.15.67:0/502860765 >> v1:172.21.15.67:6789/0 conn(0x7fc44c06b8c0 legacy=0x7fc44c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.893+0000 7fc441e00640 1 -- 172.21.15.67:0/502860765 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c160ad0 con 0x7fc44c160280 2024-04-17T08:47:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.894+0000 7fc441e00640 1 --2- 172.21.15.67:0/502860765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c06bca0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fc4340097a0 tx=0x7fc43402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1ca8815486cfb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc449e00640 1 -- 172.21.15.67:0/502860765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434035020 con 0x7fc44c160280 2024-04-17T08:47:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc449e00640 1 -- 172.21.15.67:0/502860765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc434031830 con 0x7fc44c160280 2024-04-17T08:47:26.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc449e00640 1 -- 172.21.15.67:0/502860765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434031c00 con 0x7fc44c160280 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 -- 172.21.15.67:0/502860765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 msgr2=0x7fc44c06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 --2- 172.21.15.67:0/502860765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c06bca0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fc4340097a0 tx=0x7fc43402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 -- 172.21.15.67:0/502860765 shutdown_connections 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 --2- 172.21.15.67:0/502860765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c06bca0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 -- 172.21.15.67:0/502860765 >> 172.21.15.67:0/502860765 conn(0x7fc44c067d10 msgr2=0x7fc44c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 -- 172.21.15.67:0/502860765 shutdown_connections 2024-04-17T08:47:26.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.895+0000 7fc450a00640 1 -- 172.21.15.67:0/502860765 wait complete. 2024-04-17T08:47:26.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.897+0000 7fc450a00640 1 Processor -- start 2024-04-17T08:47:26.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.897+0000 7fc450a00640 1 -- start start 2024-04-17T08:47:26.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc44c06b8c0 0x7fc44c0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc44c141ed0 0x7fc44c1439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c161090 con 0x7fc44c160280 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c06c1e0 con 0x7fc44c06b8c0 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.898+0000 7fc450a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c13f9b0 con 0x7fc44c141ed0 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58328/0 (socket says 172.21.15.67:58328) 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 -- 172.21.15.67:0/1087669464 learned_addr learned my addr 172.21.15.67:0/1087669464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc441e00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc44c141ed0 0x7fc44c1439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44a800640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc44c06b8c0 0x7fc44c0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 -- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc44c141ed0 msgr2=0x7fc44c1439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc44c141ed0 0x7fc44c1439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 -- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc44c06b8c0 msgr2=0x7fc44c0b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:26.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc44c06b8c0 0x7fc44c0b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:26.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.899+0000 7fc44b200640 1 -- 172.21.15.67:0/1087669464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c144150 con 0x7fc44c160280 2024-04-17T08:47:26.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc44b200640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c780 tx=0x7fc44400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444004b60 con 0x7fc44c160280 2024-04-17T08:47:26.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc444004d00 con 0x7fc44c160280 2024-04-17T08:47:26.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44400aa90 con 0x7fc44c160280 2024-04-17T08:47:26.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44c144430 con 0x7fc44c160280 2024-04-17T08:47:26.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.900+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44c1449f0 con 0x7fc44c160280 2024-04-17T08:47:26.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.901+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44c068a10 con 0x7fc44c160280 2024-04-17T08:47:26.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.903+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 115) ==== 90878+0+0 (secure 0 0 0) 0x7fc44400ac30 con 0x7fc44c160280 2024-04-17T08:47:26.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.903+0000 7fc443400640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 0x7fc42806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:26.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.904+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7fc44408be50 con 0x7fc44c160280 2024-04-17T08:47:26.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.904+0000 7fc44a800640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 0x7fc42806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:26.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.905+0000 7fc44a800640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 0x7fc42806ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005880 tx=0x7fc43c009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:26.908+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc444020070 con 0x7fc44c160280 2024-04-17T08:47:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.115+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc44c1419d0 con 0x7fc44c160280 2024-04-17T08:47:27.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.117+0000 7fc443400640 1 -- 172.21.15.67:0/1087669464 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) ==== 74+0+18259 (secure 0 0 0) 0x7fc44405a180 con 0x7fc44c160280 2024-04-17T08:47:27.117 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:27.117 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":200,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:25.133942+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":199,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1001338088":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2867744332":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2365426680":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3395595927":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6832/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/1228337607":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6833/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6832/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/521765659":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1964974751":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3275456489":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/630850508":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/74160311":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2788069147":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/3491826676":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1329453466":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2568338216":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1472858533":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2385834843":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2798850698":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/293550442":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/3300567104":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6832/3300567104":"2024-04-18T08:31:46.865019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 msgr2=0x7fc42806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 0x7fc42806ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc43c005880 tx=0x7fc43c009040 comp rx=0 tx=0).stop 2024-04-17T08:47:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 msgr2=0x7fc44c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc44400c780 tx=0x7fc44400cc40 comp rx=0 tx=0).stop 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 shutdown_connections 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc42806c770 0x7fc42806ec30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c160280 0x7fc44c1390b0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc44c141ed0 0x7fc44c1439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 --2- 172.21.15.67:0/1087669464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc44c06b8c0 0x7fc44c0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 >> 172.21.15.67:0/1087669464 conn(0x7fc44c067d10 msgr2=0x7fc44c0cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 shutdown_connections 2024-04-17T08:47:27.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.120+0000 7fc450a00640 1 -- 172.21.15.67:0/1087669464 wait complete. 2024-04-17T08:47:27.138 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:47:27.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.360+0000 7ff299a00640 1 Processor -- start 2024-04-17T08:47:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.360+0000 7ff299a00640 1 -- start start 2024-04-17T08:47:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff299a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff29406b8c0 0x7ff2940cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff299a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 0x7ff294142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff299a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2940cd4a0 con 0x7ff294160280 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff299a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff29415ce30 con 0x7ff29406b8c0 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff299a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff2941607f0 con 0x7ff294141ed0 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff293e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff29406b8c0 0x7ff2940cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff293e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff29406b8c0 0x7ff2940cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58342/0 (socket says 172.21.15.67:58342) 2024-04-17T08:47:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.362+0000 7ff293e00640 1 -- 172.21.15.67:0/3961487547 learned_addr learned my addr 172.21.15.67:0/3961487547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff293e00640 1 -- 172.21.15.67:0/3961487547 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 msgr2=0x7ff294142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff299000640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 0x7ff294142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff293e00640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 0x7ff294142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff293e00640 1 -- 172.21.15.67:0/3961487547 >> v1:172.21.15.187:6789/0 conn(0x7ff294160280 legacy=0x7ff29406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff293e00640 1 -- 172.21.15.67:0/3961487547 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff294160ad0 con 0x7ff29406b8c0 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.363+0000 7ff299000640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 0x7ff294142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.364+0000 7ff293e00640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29406b8c0 0x7ff2940cd8b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff284009e50 tx=0x7ff28402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df40250c5b9e7db server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.365+0000 7ff293400640 1 -- 172.21.15.67:0/3961487547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28402ca70 con 0x7ff29406b8c0 2024-04-17T08:47:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.365+0000 7ff293400640 1 -- 172.21.15.67:0/3961487547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff28402cc10 con 0x7ff29406b8c0 2024-04-17T08:47:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.365+0000 7ff293400640 1 -- 172.21.15.67:0/3961487547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2840343f0 con 0x7ff29406b8c0 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.365+0000 7ff299a00640 1 -- 172.21.15.67:0/3961487547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29406b8c0 msgr2=0x7ff2940cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.365+0000 7ff299a00640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29406b8c0 0x7ff2940cd8b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff284009e50 tx=0x7ff28402bed0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 -- 172.21.15.67:0/3961487547 shutdown_connections 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff294141ed0 0x7ff294142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 --2- 172.21.15.67:0/3961487547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff29406b8c0 0x7ff2940cd8b0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 -- 172.21.15.67:0/3961487547 >> 172.21.15.67:0/3961487547 conn(0x7ff294067d10 msgr2=0x7ff2940b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 -- 172.21.15.67:0/3961487547 shutdown_connections 2024-04-17T08:47:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.366+0000 7ff299a00640 1 -- 172.21.15.67:0/3961487547 wait complete. 2024-04-17T08:47:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.368+0000 7ff299a00640 1 Processor -- start 2024-04-17T08:47:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.368+0000 7ff299a00640 1 -- start start 2024-04-17T08:47:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff29406b8c0 0x7ff2941e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff294141ed0 0x7ff29413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff294161090 con 0x7ff294160280 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff2940cddf0 con 0x7ff294141ed0 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff29413f9b0 con 0x7ff29406b8c0 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58350/0 (socket says 172.21.15.67:58350) 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff299000640 1 -- 172.21.15.67:0/500066654 learned_addr learned my addr 172.21.15.67:0/500066654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.370+0000 7ff293e00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff29406b8c0 0x7ff2941e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff28b400640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff294141ed0 0x7ff29413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 -- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff29406b8c0 msgr2=0x7ff2941e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff29406b8c0 0x7ff2941e8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 -- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff294141ed0 msgr2=0x7ff29413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff294141ed0 0x7ff29413dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 -- 172.21.15.67:0/500066654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29413fd80 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff299000640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c820 tx=0x7ff28c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c015070 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff28c004240 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.371+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c00ce10 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.372+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff294140060 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.372+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff294140620 con 0x7ff294160280 2024-04-17T08:47:27.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.373+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff294068a10 con 0x7ff294160280 2024-04-17T08:47:27.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.374+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 116) ==== 91086+0+0 (secure 0 0 0) 0x7ff28c007830 con 0x7ff294160280 2024-04-17T08:47:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.375+0000 7ff290c00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 0x7ff27006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.375+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7ff28c08c0b0 con 0x7ff294160280 2024-04-17T08:47:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.375+0000 7ff293e00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 0x7ff27006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.377+0000 7ff293e00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 0x7ff27006ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff284035000 tx=0x7ff284030000 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.381+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff28c059ea0 con 0x7ff294160280 2024-04-17T08:47:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.631+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2941e8e60 con 0x7ff294160280 2024-04-17T08:47:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.632+0000 7ff290c00640 1 -- 172.21.15.67:0/500066654 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7ff28c0597a0 con 0x7ff294160280 2024-04-17T08:47:27.631 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:27.631 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:47:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:47:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 msgr2=0x7ff27006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 0x7ff27006ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff284035000 tx=0x7ff284030000 comp rx=0 tx=0).stop 2024-04-17T08:47:27.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 msgr2=0x7ff2940b3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c820 tx=0x7ff28c00cce0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 shutdown_connections 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff27006c9b0 0x7ff27006ee70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff294160280 0x7ff2940b3b70 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff294141ed0 0x7ff29413dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 --2- 172.21.15.67:0/500066654 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff29406b8c0 0x7ff2941e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 >> 172.21.15.67:0/500066654 conn(0x7ff294067d10 msgr2=0x7ff2940cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 shutdown_connections 2024-04-17T08:47:27.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.635+0000 7ff299a00640 1 -- 172.21.15.67:0/500066654 wait complete. 2024-04-17T08:47:27.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:27.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.875+0000 7fbe34200640 1 Processor -- start 2024-04-17T08:47:27.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.875+0000 7fbe34200640 1 -- start start 2024-04-17T08:47:27.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe34200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 0x7fbe2c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe34200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2c06f030 0x7fbe2c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe34200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2c06ff90 con 0x7fbe2c0cd020 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe34200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2c0ceb00 con 0x7fbe2c06f030 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe34200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c0cede0 con 0x7fbe2c06f670 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe32200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2c06f030 0x7fbe2c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe32200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2c06f030 0x7fbe2c0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58352/0 (socket says 172.21.15.67:58352) 2024-04-17T08:47:27.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.877+0000 7fbe32200640 1 -- 172.21.15.67:0/1153947925 learned_addr learned my addr 172.21.15.67:0/1153947925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe31800640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 0x7fbe2c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe32200640 1 -- 172.21.15.67:0/1153947925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 msgr2=0x7fbe2c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe32200640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 0x7fbe2c06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe32200640 1 -- 172.21.15.67:0/1153947925 >> v1:172.21.15.187:6789/0 conn(0x7fbe2c0cd020 legacy=0x7fbe2c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe32200640 1 -- 172.21.15.67:0/1153947925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c0cf0c0 con 0x7fbe2c06f030 2024-04-17T08:47:27.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.878+0000 7fbe31800640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 0x7fbe2c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.879+0000 7fbe32200640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06f030 0x7fbe2c0cd480 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009e50 tx=0x7fbe2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cca214221b28c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.879+0000 7fbe2be00640 1 -- 172.21.15.67:0/1153947925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2402ca70 con 0x7fbe2c06f030 2024-04-17T08:47:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.879+0000 7fbe2be00640 1 -- 172.21.15.67:0/1153947925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe2402cc10 con 0x7fbe2c06f030 2024-04-17T08:47:27.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.879+0000 7fbe2be00640 1 -- 172.21.15.67:0/1153947925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240343f0 con 0x7fbe2c06f030 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 -- 172.21.15.67:0/1153947925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06f030 msgr2=0x7fbe2c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06f030 0x7fbe2c0cd480 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009e50 tx=0x7fbe2402bed0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 -- 172.21.15.67:0/1153947925 shutdown_connections 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c06f030 0x7fbe2c0cd480 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 --2- 172.21.15.67:0/1153947925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f670 0x7fbe2c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 -- 172.21.15.67:0/1153947925 >> 172.21.15.67:0/1153947925 conn(0x7fbe2c06bf50 msgr2=0x7fbe2c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 -- 172.21.15.67:0/1153947925 shutdown_connections 2024-04-17T08:47:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.880+0000 7fbe34200640 1 -- 172.21.15.67:0/1153947925 wait complete. 2024-04-17T08:47:27.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.882+0000 7fbe34200640 1 Processor -- start 2024-04-17T08:47:27.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.882+0000 7fbe34200640 1 -- start start 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f030 0x7fbe2c15d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 0x7fbe2c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c0ce720 con 0x7fbe2c0cd020 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c0cf680 con 0x7fbe2c06f670 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe34200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c0ce8a0 con 0x7fbe2c06f030 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.883+0000 7fbe32200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58356/0 (socket says 172.21.15.67:58356) 2024-04-17T08:47:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 -- 172.21.15.67:0/2335260759 learned_addr learned my addr 172.21.15.67:0/2335260759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe30e00640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 0x7fbe2c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe31800640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f030 0x7fbe2c15d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 -- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f030 msgr2=0x7fbe2c15d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f030 0x7fbe2c15d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 -- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 msgr2=0x7fbe2c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 0x7fbe2c0b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 -- 172.21.15.67:0/2335260759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c0c90e0 con 0x7fbe2c0cd020 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe30e00640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 0x7fbe2c0b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.884+0000 7fbe32200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009e50 tx=0x7fbe24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.885+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24043020 con 0x7fbe2c0cd020 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.885+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe2403e450 con 0x7fbe2c0cd020 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.885+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c0c9e60 con 0x7fbe2c0cd020 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.885+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2403e820 con 0x7fbe2c0cd020 2024-04-17T08:47:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.885+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c0c9380 con 0x7fbe2c0cd020 2024-04-17T08:47:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.886+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2c068b90 con 0x7fbe2c0cd020 2024-04-17T08:47:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.887+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 116) ==== 91086+0+0 (secure 0 0 0) 0x7fbe24046020 con 0x7fbe2c0cd020 2024-04-17T08:47:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.888+0000 7fbe29600640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 0x7fbe0c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.888+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7fbe240ae870 con 0x7fbe2c0cd020 2024-04-17T08:47:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.888+0000 7fbe31800640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 0x7fbe0c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.890+0000 7fbe31800640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 0x7fbe0c06ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbe20008850 tx=0x7fbe20008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:27.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:27.892+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbe2404d030 con 0x7fbe2c0cd020 2024-04-17T08:47:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.136+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe2c0cbdd0 con 0x7fbe2c0cd020 2024-04-17T08:47:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.137+0000 7fbe29600640 1 -- 172.21.15.67:0/2335260759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5206 (secure 0 0 0) 0x7fbe2407c900 con 0x7fbe2c0cd020 2024-04-17T08:47:28.137 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:28.137 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:31:32.003013+0000","modified":"2024-04-17T08:31:35.101530+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9987":{"gid":9987,"name":"d","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/3432159492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3432159492},{"type":"v1","addr":"172.21.15.187:6837","nonce":3432159492}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9987,"qdb_cluster":[9987]},"id":11}]} 2024-04-17T08:47:28.137 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T08:47:28.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 msgr2=0x7fbe0c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 0x7fbe0c06ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbe20008850 tx=0x7fbe20008040 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 msgr2=0x7fbe2c14c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009e50 tx=0x7fbe24004a40 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 shutdown_connections 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbe0c06c980 0x7fbe0c06ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2c0cd020 0x7fbe2c14c8f0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe2c06f670 0x7fbe2c0b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 --2- 172.21.15.67:0/2335260759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbe2c06f030 0x7fbe2c15d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 >> 172.21.15.67:0/2335260759 conn(0x7fbe2c06bf50 msgr2=0x7fbe2c15b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 shutdown_connections 2024-04-17T08:47:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.140+0000 7fbe34200640 1 -- 172.21.15.67:0/2335260759 wait complete. 2024-04-17T08:47:28.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:28.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.379+0000 7fe547000640 1 Processor -- start 2024-04-17T08:47:28.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.379+0000 7fe547000640 1 -- start start 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe547000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 0x7fe5401664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe547000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54015bc80 con 0x7fe54015b570 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe547000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54015bf40 con 0x7fe54006feb0 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe547000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe54006c490 con 0x7fe54006c040 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe545c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe54015b570 0x7fe54006f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60058/0 (socket says 172.21.15.67:60058) 2024-04-17T08:47:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.381+0000 7fe545c00640 1 -- 172.21.15.67:0/2741356300 learned_addr learned my addr 172.21.15.67:0/2741356300 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe546600640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 0x7fe5401664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556051744 0 0) 0x7fe54015bc80 con 0x7fe54015b570 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe528003520 con 0x7fe54015b570 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2411407513 0 0) 0x7fe528003520 con 0x7fe54015b570 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 msgr2=0x7fe5401664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 0x7fe5401664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 >> v1:172.21.15.187:6789/0 conn(0x7fe54006feb0 legacy=0x7fe54006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54015c220 con 0x7fe54015b570 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe534001670 con 0x7fe54015b570 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 >> v1:172.21.15.67:6789/0 conn(0x7fe54015b570 legacy=0x7fe54006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 0x7fe528003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528004020 0x7fe528004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004d40 0x7fe5280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe52800bbd0 con 0x7fe528004020 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe52800bd80 con 0x7fe528004d40 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.382+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe52800bf80 con 0x7fe54006c040 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528004020 0x7fe528004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 -- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 msgr2=0x7fe528003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe54006c040 0x7fe528003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 -- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004d40 msgr2=0x7fe5280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe546600640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004d40 0x7fe5280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004d40 0x7fe5280051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.383+0000 7fe545200640 1 -- 172.21.15.67:0/2741356300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe528003520 con 0x7fe528004020 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.384+0000 7fe545200640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528004020 0x7fe528004800 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001750 tx=0x7fe528005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d119952988bcc178 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.384+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c001370 con 0x7fe528004020 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.384+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52c0052d0 con 0x7fe528004020 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.384+0000 7fe53fe00640 1 -- 172.21.15.67:0/2741356300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c003830 con 0x7fe528004020 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 -- 172.21.15.67:0/2741356300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528004020 msgr2=0x7fe528004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 --2- 172.21.15.67:0/2741356300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528004020 0x7fe528004800 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001750 tx=0x7fe528005830 comp rx=0 tx=0).stop 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe546600640 1 -- 172.21.15.67:0/2741356300 reap_dead start 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 -- 172.21.15.67:0/2741356300 shutdown_connections 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 -- 172.21.15.67:0/2741356300 >> 172.21.15.67:0/2741356300 conn(0x7fe540067d10 msgr2=0x7fe5400b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 -- 172.21.15.67:0/2741356300 shutdown_connections 2024-04-17T08:47:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.385+0000 7fe547000640 1 -- 172.21.15.67:0/2741356300 wait complete. 2024-04-17T08:47:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.387+0000 7fe547000640 1 Processor -- start 2024-04-17T08:47:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.387+0000 7fe547000640 1 -- start start 2024-04-17T08:47:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004020 0x7fe540072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 0x7fe5401e63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400b47b0 con 0x7fe54014e1a0 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe54015b950 con 0x7fe528004020 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe547000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe5400b4930 con 0x7fe5401e5f40 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.388+0000 7fe545200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58370/0 (socket says 172.21.15.67:58370) 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 -- 172.21.15.67:0/2101966313 learned_addr learned my addr 172.21.15.67:0/2101966313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545c00640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004020 0x7fe540072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe546600640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 0x7fe5401e63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 -- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 msgr2=0x7fe5401e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 0x7fe5401e63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 -- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004020 msgr2=0x7fe540072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004020 0x7fe540072830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 -- 172.21.15.67:0/2101966313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54015d830 con 0x7fe54014e1a0 2024-04-17T08:47:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe546600640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 0x7fe5401e63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.389+0000 7fe545200640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005040 tx=0x7fe52c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.390+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c041070 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.390+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52c000ed0 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.390+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54015dad0 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.390+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c007a30 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.390+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54015e090 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.391+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe540068a10 con 0x7fe54014e1a0 2024-04-17T08:47:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.392+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 116) ==== 91086+0+0 (secure 0 0 0) 0x7fe52c044070 con 0x7fe54014e1a0 2024-04-17T08:47:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.393+0000 7fe53d600640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 0x7fe52006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.393+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7fe52c0adbc0 con 0x7fe54014e1a0 2024-04-17T08:47:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.394+0000 7fe545c00640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 0x7fe52006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.395+0000 7fe545c00640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 0x7fe52006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe534000ff0 tx=0x7fe5340015c0 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.398+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe52c07b9f0 con 0x7fe54014e1a0 2024-04-17T08:47:28.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.603+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe540072d70 con 0x7fe54014e1a0 2024-04-17T08:47:28.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.606+0000 7fe53d600640 1 -- 172.21.15.67:0/2101966313 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) ==== 74+0+18259 (secure 0 0 0) 0x7fe52c07b2f0 con 0x7fe54014e1a0 2024-04-17T08:47:28.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:28.606 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":200,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:25.133942+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:31:29.943141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-04-17T08:31:30.995201+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":199,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1001338088":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2867744332":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2365426680":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3395595927":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6832/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/1228337607":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6833/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6832/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/521765659":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1964974751":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3275456489":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/630850508":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/74160311":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2788069147":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/3491826676":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1329453466":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2568338216":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1472858533":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2385834843":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2798850698":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/293550442":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/3300567104":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6832/3300567104":"2024-04-18T08:31:46.865019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:28.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 msgr2=0x7fe52006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 0x7fe52006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe534000ff0 tx=0x7fe5340015c0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 msgr2=0x7fe54014da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005040 tx=0x7fe52c001120 comp rx=0 tx=0).stop 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 shutdown_connections 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fe52006c980 0x7fe52006ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5401e5f40 0x7fe5401e63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54014e1a0 0x7fe54014da00 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 --2- 172.21.15.67:0/2101966313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe528004020 0x7fe540072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 >> 172.21.15.67:0/2101966313 conn(0x7fe540067d10 msgr2=0x7fe540155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 shutdown_connections 2024-04-17T08:47:28.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.609+0000 7fe547000640 1 -- 172.21.15.67:0/2101966313 wait complete. 2024-04-17T08:47:28.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:47:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.849+0000 7f575ca00640 1 Processor -- start 2024-04-17T08:47:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.849+0000 7f575ca00640 1 -- start start 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f575ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f57580c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f575ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5758162ce0 con 0x7f57580c9f80 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f575ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5758162fc0 con 0x7f57580afc70 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f575ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f575806e1b0 con 0x7f5758162740 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f5757200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57580c9f80 0x7f575806e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60060/0 (socket says 172.21.15.67:60060) 2024-04-17T08:47:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f5757200640 1 -- 172.21.15.67:0/3050528552 learned_addr learned my addr 172.21.15.67:0/3050528552 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.851+0000 7f5755e00640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f57580c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097770178 0 0) 0x7f5758162ce0 con 0x7f57580c9f80 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f573c003520 con 0x7f57580c9f80 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3286422048 0 0) 0x7f573c003520 con 0x7f57580c9f80 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 msgr2=0x7f57580c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f57580c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 >> v1:172.21.15.187:6789/0 conn(0x7f57580afc70 legacy=0x7f575815fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57581632a0 con 0x7f57580c9f80 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f57500019f0 con 0x7f57580c9f80 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 >> v1:172.21.15.67:6789/0 conn(0x7f57580c9f80 legacy=0x7f575806e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f573c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f573c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f573c004d40 0x7f573c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f575806e1b0 con 0x7f573c003a50 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00bbd0 con 0x7f573c004d40 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f573c00bdd0 con 0x7f5758162740 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.852+0000 7f5756800640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f573c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5757200640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f573c004d40 0x7f573c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5755e00640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f573c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5756800640 1 -- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 msgr2=0x7f573c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5756800640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f573c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5756800640 1 -- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f573c004d40 msgr2=0x7f573c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5756800640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f573c004d40 0x7f573c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5756800640 1 -- 172.21.15.67:0/3050528552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573c003520 con 0x7f573c003a50 2024-04-17T08:47:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.853+0000 7f5755e00640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5758162740 0x7f573c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.854+0000 7f5756800640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f573c003e70 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f5748001750 tx=0x7f573c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10207d3f49b2629 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.854+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748001370 con 0x7f573c003a50 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.854+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57480052d0 con 0x7f573c003a50 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.854+0000 7f5755400640 1 -- 172.21.15.67:0/3050528552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748003830 con 0x7f573c003a50 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 -- 172.21.15.67:0/3050528552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 msgr2=0x7f573c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 --2- 172.21.15.67:0/3050528552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f573c003e70 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f5748001750 tx=0x7f573c005830 comp rx=0 tx=0).stop 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f5757200640 1 -- 172.21.15.67:0/3050528552 reap_dead start 2024-04-17T08:47:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 -- 172.21.15.67:0/3050528552 shutdown_connections 2024-04-17T08:47:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 -- 172.21.15.67:0/3050528552 >> 172.21.15.67:0/3050528552 conn(0x7f575806bef0 msgr2=0x7f57580b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 -- 172.21.15.67:0/3050528552 shutdown_connections 2024-04-17T08:47:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.855+0000 7f575ca00640 1 -- 172.21.15.67:0/3050528552 wait complete. 2024-04-17T08:47:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.857+0000 7f575ca00640 1 Processor -- start 2024-04-17T08:47:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.858+0000 7f575ca00640 1 -- start start 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f575813ebe0 0x7f57580b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758147a50 0x7f5758147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57580c9a40 con 0x7f573c003a50 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f57580b0050 con 0x7f5758147a50 2024-04-17T08:47:28.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f575ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57580c9bc0 con 0x7f575813ebe0 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f5756800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f5756800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58386/0 (socket says 172.21.15.67:58386) 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f5756800640 1 -- 172.21.15.67:0/284095362 learned_addr learned my addr 172.21.15.67:0/284095362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.859+0000 7f5755e00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f575813ebe0 0x7f57580b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 -- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f575813ebe0 msgr2=0x7f57580b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5757200640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758147a50 0x7f5758147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f575813ebe0 0x7f57580b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 -- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758147a50 msgr2=0x7f5758147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758147a50 0x7f5758147ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 -- 172.21.15.67:0/284095362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57580b3d70 con 0x7f573c003a50 2024-04-17T08:47:28.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f5756800640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5748008fd0 tx=0x7f5748001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748041070 con 0x7f573c003a50 2024-04-17T08:47:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5748005480 con 0x7f573c003a50 2024-04-17T08:47:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748001370 con 0x7f573c003a50 2024-04-17T08:47:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.860+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57580b4050 con 0x7f573c003a50 2024-04-17T08:47:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.861+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57580b4610 con 0x7f573c003a50 2024-04-17T08:47:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.862+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57580b4850 con 0x7f573c003a50 2024-04-17T08:47:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.863+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 116) ==== 91086+0+0 (secure 0 0 0) 0x7f5748044070 con 0x7f573c003a50 2024-04-17T08:47:28.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.864+0000 7f574ea00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 0x7f573806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:28.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.864+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6797+0+0 (secure 0 0 0) 0x7f57480adcd0 con 0x7f573c003a50 2024-04-17T08:47:28.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.865+0000 7f5755e00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 0x7f573806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:28.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.866+0000 7f5755e00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 0x7f573806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575815faf0 tx=0x7f5744008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:28.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:28.869+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f57580b4850 con 0x7f573c003a50 2024-04-17T08:47:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.125+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5758068a10 con 0x7f573c003a50 2024-04-17T08:47:29.125 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:47:29.126+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:47:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.130+0000 7f574ea00640 1 -- 172.21.15.67:0/284095362 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) ==== 161+0+0 (secure 0 0 0) 0x7f57480abe20 con 0x7f573c003a50 2024-04-17T08:47:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:47:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 msgr2=0x7f573806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 0x7f573806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575815faf0 tx=0x7f5744008040 comp rx=0 tx=0).stop 2024-04-17T08:47:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 msgr2=0x7f5758145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5748008fd0 tx=0x7f5748001120 comp rx=0 tx=0).stop 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 shutdown_connections 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f573806c9b0 0x7f573806ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5758147a50 0x7f5758147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f575813ebe0 0x7f57580b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 --2- 172.21.15.67:0/284095362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f573c003a50 0x7f5758145110 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 >> 172.21.15.67:0/284095362 conn(0x7f575806bef0 msgr2=0x7f5758161480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:29.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 shutdown_connections 2024-04-17T08:47:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.134+0000 7f575ca00640 1 -- 172.21.15.67:0/284095362 wait complete. 2024-04-17T08:47:29.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:47:29.264 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:47:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.366+0000 7fa356200640 1 Processor -- start 2024-04-17T08:47:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.366+0000 7fa356200640 1 -- start start 2024-04-17T08:47:29.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa356200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa35015ecb0 0x7fa3500d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa356200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa3500d3210 0x7fa3500cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa356200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3500d00b0 con 0x7fa3500cf8e0 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa356200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa3500d0c30 con 0x7fa35015ecb0 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa356200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa3500d0390 con 0x7fa3500d3210 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa354e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa35015ecb0 0x7fa3500d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa355800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3500cf8e0 0x7fa3500d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60076/0 (socket says 172.21.15.67:60076) 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa354e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa35015ecb0 0x7fa3500d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58396/0 (socket says 172.21.15.67:58396) 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa355800640 1 -- 172.21.15.67:0/3659311331 learned_addr learned my addr 172.21.15.67:0/3659311331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.368+0000 7fa34fe00640 1 --2- 172.21.15.67:0/3659311331 >> v2:172.21.15.187:3300/0 conn(0x7fa3500d3210 0x7fa3500cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.369+0000 7fa34f400640 1 -- 172.21.15.67:0/3659311331 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736486364 0 0) 0x7fa3500d00b0 con 0x7fa3500cf8e0 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.369+0000 7fa354e00640 1 -- 172.21.15.67:0/3659311331 >> v2:172.21.15.187:3300/0 conn(0x7fa3500d3210 msgr2=0x7fa3500cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.369+0000 7fa354e00640 1 --2- 172.21.15.67:0/3659311331 >> v2:172.21.15.187:3300/0 conn(0x7fa3500d3210 0x7fa3500cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.369+0000 7fa354e00640 1 -- 172.21.15.67:0/3659311331 >> v1:172.21.15.67:6789/0 conn(0x7fa3500cf8e0 legacy=0x7fa3500d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.369+0000 7fa354e00640 1 -- 172.21.15.67:0/3659311331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3500d0670 con 0x7fa35015ecb0 2024-04-17T08:47:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.370+0000 7fa354e00640 1 --2- 172.21.15.67:0/3659311331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa35015ecb0 0x7fa3500d2cd0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa3501627f0 tx=0x7fa34002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2a80f0f0485aff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa34f400640 1 -- 172.21.15.67:0/3659311331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3400315a0 con 0x7fa35015ecb0 2024-04-17T08:47:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa34f400640 1 -- 172.21.15.67:0/3659311331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa340037070 con 0x7fa35015ecb0 2024-04-17T08:47:29.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa34f400640 1 -- 172.21.15.67:0/3659311331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340031960 con 0x7fa35015ecb0 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 -- 172.21.15.67:0/3659311331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa35015ecb0 msgr2=0x7fa3500d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 --2- 172.21.15.67:0/3659311331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa35015ecb0 0x7fa3500d2cd0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fa3501627f0 tx=0x7fa34002b840 comp rx=0 tx=0).stop 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 -- 172.21.15.67:0/3659311331 shutdown_connections 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 --2- 172.21.15.67:0/3659311331 >> v2:172.21.15.187:3300/0 conn(0x7fa3500d3210 0x7fa3500cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 --2- 172.21.15.67:0/3659311331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa35015ecb0 0x7fa3500d2cd0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 -- 172.21.15.67:0/3659311331 >> 172.21.15.67:0/3659311331 conn(0x7fa35008bff0 msgr2=0x7fa3500d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 -- 172.21.15.67:0/3659311331 shutdown_connections 2024-04-17T08:47:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.371+0000 7fa356200640 1 -- 172.21.15.67:0/3659311331 wait complete. 2024-04-17T08:47:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.373+0000 7fa356200640 1 Processor -- start 2024-04-17T08:47:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.373+0000 7fa356200640 1 -- start start 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 0x7fa3500c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa35015ecb0 0x7fa3500c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3500cfd00 con 0x7fa3500d3210 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa35015f090 con 0x7fa35015ecb0 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa356200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa3500cb5f0 con 0x7fa3500cf8e0 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.374+0000 7fa34fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58412/0 (socket says 172.21.15.67:58412) 2024-04-17T08:47:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 -- 172.21.15.67:0/11542545 learned_addr learned my addr 172.21.15.67:0/11542545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:29.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa355800640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa35015ecb0 0x7fa3500c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa354e00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 0x7fa3500c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 -- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 msgr2=0x7fa3500c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 0x7fa3500c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 -- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa35015ecb0 msgr2=0x7fa3500c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa35015ecb0 0x7fa3500c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 -- 172.21.15.67:0/11542545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3500c8720 con 0x7fa3500d3210 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa354e00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 0x7fa3500c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.375+0000 7fa34fe00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fa33800d800 tx=0x7fa33800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.376+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33800b6c0 con 0x7fa3500d3210 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.376+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa338019070 con 0x7fa3500d3210 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.376+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33800ba80 con 0x7fa3500d3210 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.376+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa350155130 con 0x7fa3500d3210 2024-04-17T08:47:29.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.376+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3500caf30 con 0x7fa3500d3210 2024-04-17T08:47:29.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.377+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa350068a10 con 0x7fa3500d3210 2024-04-17T08:47:29.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.378+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 116) ==== 91086+0+0 (secure 0 0 0) 0x7fa33800bc20 con 0x7fa3500d3210 2024-04-17T08:47:29.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.379+0000 7fa34cc00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 0x7fa32c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:29.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.379+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6883+0+0 (secure 0 0 0) 0x7fa33808c540 con 0x7fa3500d3210 2024-04-17T08:47:29.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.380+0000 7fa354e00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 0x7fa32c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.381+0000 7fa354e00640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 0x7fa32c06ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa340036000 tx=0x7fa340008030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.383+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa33805a470 con 0x7fa3500d3210 2024-04-17T08:47:29.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:29.636+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa3500b5a70 con 0x7fa3500d3210 2024-04-17T08:47:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.152+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fa3380563e0 con 0x7fa3500d3210 2024-04-17T08:47:30.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.188+0000 7fa34cc00640 1 -- 172.21.15.67:0/11542545 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) ==== 104+0+0 (secure 0 0 0) 0x7fa338076250 con 0x7fa3500d3210 2024-04-17T08:47:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 msgr2=0x7fa32c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 0x7fa32c06ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa340036000 tx=0x7fa340008030 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 msgr2=0x7fa350154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fa33800d800 tx=0x7fa33800dcc0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 shutdown_connections 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa32c06c980 0x7fa32c06ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa35015ecb0 0x7fa3500c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3500d3210 0x7fa350154bf0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 --2- 172.21.15.67:0/11542545 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3500cf8e0 0x7fa3500c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 >> 172.21.15.67:0/11542545 conn(0x7fa35008bff0 msgr2=0x7fa35015dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 shutdown_connections 2024-04-17T08:47:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.192+0000 7fa356200640 1 -- 172.21.15.67:0/11542545 wait complete. 2024-04-17T08:47:30.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:47:30.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.444+0000 7f69d2800640 1 Processor -- start 2024-04-17T08:47:30.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.444+0000 7f69d2800640 1 -- start start 2024-04-17T08:47:30.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.446+0000 7f69d2800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f69cc06f220 0x7f69cc1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.446+0000 7f69d2800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69cc1612d0 con 0x7f69cc142e10 2024-04-17T08:47:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.446+0000 7f69d2800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69cc160510 con 0x7f69cc072430 2024-04-17T08:47:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.446+0000 7f69d2800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f69cc1431f0 con 0x7f69cc06f220 2024-04-17T08:47:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.447+0000 7f69d1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69cc142e10 0x7f69cc071d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60092/0 (socket says 172.21.15.67:60092) 2024-04-17T08:47:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.447+0000 7f69d1400640 1 -- 172.21.15.67:0/2198499184 learned_addr learned my addr 172.21.15.67:0/2198499184 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.447+0000 7f69d0a00640 1 --2- 172.21.15.67:0/2198499184 >> v2:172.21.15.187:3300/0 conn(0x7f69cc06f220 0x7f69cc1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.447+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862616874 0 0) 0x7f69cc1612d0 con 0x7f69cc142e10 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.447+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4003520 con 0x7f69cc142e10 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3097959978 0 0) 0x7f69b4003520 con 0x7f69cc142e10 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 >> v2:172.21.15.187:3300/0 conn(0x7f69cc06f220 msgr2=0x7f69cc1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 --2- 172.21.15.67:0/2198499184 >> v2:172.21.15.187:3300/0 conn(0x7f69cc06f220 0x7f69cc1434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 >> v1:172.21.15.187:6789/0 conn(0x7f69cc072430 legacy=0x7f69cc1439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69cc1607f0 con 0x7f69cc142e10 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f69c0001670 con 0x7f69cc142e10 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 >> v1:172.21.15.67:6789/0 conn(0x7f69cc142e10 legacy=0x7f69cc071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b4003a70 0x7f69b4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 0x7f69b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc1431f0 con 0x7f69b40052e0 2024-04-17T08:47:30.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69b400c1b0 con 0x7f69b4003a70 2024-04-17T08:47:30.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69b400c3b0 con 0x7f69b4004800 2024-04-17T08:47:30.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.448+0000 7f69d1e00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 0x7f69b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1400640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b4003a70 0x7f69b4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d0a00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1e00640 1 -- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b4004800 msgr2=0x7f69b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1e00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1e00640 1 -- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b4003a70 msgr2=0x7f69b4003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1e00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69b4003a70 0x7f69b4003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d1e00640 1 -- 172.21.15.67:0/2198499184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b4003520 con 0x7f69b40052e0 2024-04-17T08:47:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.449+0000 7f69d0a00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.450+0000 7f69d1e00640 1 -- 172.21.15.67:0/2198499184 reap_dead start 2024-04-17T08:47:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.450+0000 7f69d1e00640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 0x7f69b400b9c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f69c4001e50 tx=0x7f69b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bdd6df57d6f1af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.451+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4001a70 con 0x7f69b40052e0 2024-04-17T08:47:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.451+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c4003bb0 con 0x7f69b40052e0 2024-04-17T08:47:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.451+0000 7f69cbe00640 1 -- 172.21.15.67:0/2198499184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c40050d0 con 0x7f69b40052e0 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 -- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 msgr2=0x7f69b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 0x7f69b400b9c0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f69c4001e50 tx=0x7f69b4005830 comp rx=0 tx=0).stop 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 -- 172.21.15.67:0/2198499184 shutdown_connections 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 --2- 172.21.15.67:0/2198499184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69b40052e0 0x7f69b400b9c0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 -- 172.21.15.67:0/2198499184 >> 172.21.15.67:0/2198499184 conn(0x7f69cc06b8c0 msgr2=0x7f69cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 -- 172.21.15.67:0/2198499184 shutdown_connections 2024-04-17T08:47:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.452+0000 7f69d2800640 1 -- 172.21.15.67:0/2198499184 wait complete. 2024-04-17T08:47:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.454+0000 7f69d2800640 1 Processor -- start 2024-04-17T08:47:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.454+0000 7f69d2800640 1 -- start start 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b40052e0 0x7f69cc1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 0x7f69cc15f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc1460a0 con 0x7f69cc13a180 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69cc160e40 con 0x7f69cc15d800 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d2800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69cc146220 con 0x7f69b40052e0 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58436/0 (socket says 172.21.15.67:58436) 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.455+0000 7f69d1e00640 1 -- 172.21.15.67:0/1585284688 learned_addr learned my addr 172.21.15.67:0/1585284688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1400640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b40052e0 0x7f69cc1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d0a00640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 0x7f69cc15f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 -- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b40052e0 msgr2=0x7f69cc1567d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b40052e0 0x7f69cc1567d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 -- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 msgr2=0x7f69cc15f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 0x7f69cc15f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 -- 172.21.15.67:0/1585284688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69cc13ab40 con 0x7f69cc13a180 2024-04-17T08:47:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d0a00640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 0x7f69cc15f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69d1e00640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f69c4001f40 tx=0x7f69c40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.456+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4006d60 con 0x7f69cc13a180 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.457+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c4007030 con 0x7f69cc13a180 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.457+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c40073e0 con 0x7f69cc13a180 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.457+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69cc156f40 con 0x7f69cc13a180 2024-04-17T08:47:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.457+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69cc157500 con 0x7f69cc13a180 2024-04-17T08:47:30.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.458+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69cc068a10 con 0x7f69cc13a180 2024-04-17T08:47:30.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.459+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f69c4007580 con 0x7f69cc13a180 2024-04-17T08:47:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.460+0000 7f69c9600640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 0x7f69ac06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.460+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6883+0+0 (secure 0 0 0) 0x7f69c40ade60 con 0x7f69cc13a180 2024-04-17T08:47:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.461+0000 7f69d1400640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 0x7f69ac06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.462+0000 7f69d1400640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 0x7f69ac06ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69cc1f5d00 tx=0x7f69c0009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:30.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.465+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f69c4033030 con 0x7f69cc13a180 2024-04-17T08:47:30.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:30.671+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f69cc072e30 con 0x7f69cc13a180 2024-04-17T08:47:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.200+0000 7f69c9600640 1 -- 172.21.15.67:0/1585284688 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v203) ==== 181+0+0 (secure 0 0 0) 0x7f69c407bb30 con 0x7f69cc13a180 2024-04-17T08:47:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:47:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 msgr2=0x7f69ac06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 0x7f69ac06ef00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69cc1f5d00 tx=0x7f69c0009040 comp rx=0 tx=0).stop 2024-04-17T08:47:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 msgr2=0x7f69cc13a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f69c4001f40 tx=0x7f69c40014a0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 shutdown_connections 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f69ac06ca40 0x7f69ac06ef00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69cc13a180 0x7f69cc13a600 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69cc15d800 0x7f69cc15f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 --2- 172.21.15.67:0/1585284688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69b40052e0 0x7f69cc1567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 >> 172.21.15.67:0/1585284688 conn(0x7f69cc06b8c0 msgr2=0x7f69cc1449b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 shutdown_connections 2024-04-17T08:47:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.204+0000 7f69d2800640 1 -- 172.21.15.67:0/1585284688 wait complete. 2024-04-17T08:47:31.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:47:31.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.445+0000 7fc8b6e00640 1 Processor -- start 2024-04-17T08:47:31.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.445+0000 7fc8b6e00640 1 -- start start 2024-04-17T08:47:31.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc8b01484e0 0x7fc8b00b6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 0x7fc8b0162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b0162bd0 0x7fc8b006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b0160290 con 0x7fc8b0162bd0 2024-04-17T08:47:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b0163000 con 0x7fc8b01484e0 2024-04-17T08:47:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.447+0000 7fc8b6e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b00b4a70 con 0x7fc8b00b6840 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b0162bd0 0x7fc8b006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8b0162bd0 0x7fc8b006b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58442/0 (socket says 172.21.15.67:58442) 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 -- 172.21.15.67:0/1402379461 learned_addr learned my addr 172.21.15.67:0/1402379461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8afe00640 1 --2- 172.21.15.67:0/1402379461 >> v2:172.21.15.187:3300/0 conn(0x7fc8b01484e0 0x7fc8b00b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8a7e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 0x7fc8b0162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 -- 172.21.15.67:0/1402379461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 msgr2=0x7fc8b0162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 0x7fc8b0162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 -- 172.21.15.67:0/1402379461 >> v2:172.21.15.187:3300/0 conn(0x7fc8b01484e0 msgr2=0x7fc8b00b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 --2- 172.21.15.67:0/1402379461 >> v2:172.21.15.187:3300/0 conn(0x7fc8b01484e0 0x7fc8b00b6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8b4e00640 1 -- 172.21.15.67:0/1402379461 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b015fc40 con 0x7fc8b0162bd0 2024-04-17T08:47:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.448+0000 7fc8a7e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 0x7fc8b0162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.449+0000 7fc8b4e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b0162bd0 0x7fc8b006b8c0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fc8b006eef0 tx=0x7fc8a802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb61ebffe628932 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8af400640 1 -- 172.21.15.67:0/1402379461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a802ca80 con 0x7fc8b0162bd0 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8af400640 1 -- 172.21.15.67:0/1402379461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a802cc20 con 0x7fc8b0162bd0 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8af400640 1 -- 172.21.15.67:0/1402379461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80343f0 con 0x7fc8b0162bd0 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 -- 172.21.15.67:0/1402379461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b0162bd0 msgr2=0x7fc8b006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b0162bd0 0x7fc8b006b8c0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fc8b006eef0 tx=0x7fc8a802bee0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 -- 172.21.15.67:0/1402379461 shutdown_connections 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b0162bd0 0x7fc8b006b8c0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/1402379461 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b00b6840 0x7fc8b0162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/1402379461 >> v2:172.21.15.187:3300/0 conn(0x7fc8b01484e0 0x7fc8b00b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 -- 172.21.15.67:0/1402379461 >> 172.21.15.67:0/1402379461 conn(0x7fc8b00b72d0 msgr2=0x7fc8b009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 -- 172.21.15.67:0/1402379461 shutdown_connections 2024-04-17T08:47:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.450+0000 7fc8b6e00640 1 -- 172.21.15.67:0/1402379461 wait complete. 2024-04-17T08:47:31.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.452+0000 7fc8b6e00640 1 Processor -- start 2024-04-17T08:47:31.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.452+0000 7fc8b6e00640 1 -- start start 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 0x7fc8b015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b0162bd0 0x7fc8b013cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b00b4ed0 con 0x7fc8b01484e0 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b00b4d50 con 0x7fc8b00b6840 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8b6e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b00b4640 con 0x7fc8b0162bd0 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.453+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58446/0 (socket says 172.21.15.67:58446) 2024-04-17T08:47:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 -- 172.21.15.67:0/4059017959 learned_addr learned my addr 172.21.15.67:0/4059017959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:31.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8b4e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b0162bd0 0x7fc8b013cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8afe00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 0x7fc8b015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 -- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b0162bd0 msgr2=0x7fc8b013cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b0162bd0 0x7fc8b013cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 -- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 msgr2=0x7fc8b015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 0x7fc8b015e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 -- 172.21.15.67:0/4059017959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b00cbf30 con 0x7fc8b01484e0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8afe00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 0x7fc8b015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.454+0000 7fc8a7e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c820 tx=0x7fc89800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.455+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898015070 con 0x7fc8b01484e0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.455+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc898007a30 con 0x7fc8b01484e0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.455+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898007e00 con 0x7fc8b01484e0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.455+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b015e820 con 0x7fc8b01484e0 2024-04-17T08:47:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.455+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b0140670 con 0x7fc8b01484e0 2024-04-17T08:47:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.456+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8b009f930 con 0x7fc8b01484e0 2024-04-17T08:47:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.457+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fc898018070 con 0x7fc8b01484e0 2024-04-17T08:47:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.458+0000 7fc8acc00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 0x7fc88c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.458+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6417+0+0 (secure 0 0 0) 0x7fc89808c120 con 0x7fc8b01484e0 2024-04-17T08:47:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.459+0000 7fc8afe00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 0x7fc88c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:31.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.460+0000 7fc8afe00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 0x7fc88c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000b840 tx=0x7fc8a0008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:31.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.463+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc898059fc0 con 0x7fc8b01484e0 2024-04-17T08:47:31.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:31.709+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc8b0068d60 con 0x7fc8b01484e0 2024-04-17T08:47:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.215+0000 7fc8acc00640 1 -- 172.21.15.67:0/4059017959 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v204) ==== 169+0+0 (secure 0 0 0) 0x7fc8980598c0 con 0x7fc8b01484e0 2024-04-17T08:47:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:47:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 msgr2=0x7fc88c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 0x7fc88c06eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000b840 tx=0x7fc8a0008040 comp rx=0 tx=0).stop 2024-04-17T08:47:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 msgr2=0x7fc8b01473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c820 tx=0x7fc89800cc20 comp rx=0 tx=0).stop 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 shutdown_connections 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fc88c06ca10 0x7fc88c06eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8b0162bd0 0x7fc8b013cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8b01484e0 0x7fc8b01473d0 secure :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c820 tx=0x7fc89800cc20 comp rx=0 tx=0).stop 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 --2- 172.21.15.67:0/4059017959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8b00b6840 0x7fc8b015e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.219+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 >> 172.21.15.67:0/4059017959 conn(0x7fc8b00b72d0 msgr2=0x7fc8b0070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.222+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 shutdown_connections 2024-04-17T08:47:32.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.223+0000 7fc8b6e00640 1 -- 172.21.15.67:0/4059017959 wait complete. 2024-04-17T08:47:32.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:47:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.473+0000 7f6990000640 1 Processor -- start 2024-04-17T08:47:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.473+0000 7f6990000640 1 -- start start 2024-04-17T08:47:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f6990000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 0x7f6988071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f6990000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69880ce3b0 0x7f6988072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f6990000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69880cea70 con 0x7f6988072bb0 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f6990000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f69880ce810 con 0x7f69880ce3b0 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f6990000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69880ced50 con 0x7f6988145ac0 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f698cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69880ce3b0 0x7f6988072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f698cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69880ce3b0 0x7f6988072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58452/0 (socket says 172.21.15.67:58452) 2024-04-17T08:47:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.475+0000 7f698cc00640 1 -- 172.21.15.67:0/3965989162 learned_addr learned my addr 172.21.15.67:0/3965989162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698d600640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 0x7f6988071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698cc00640 1 -- 172.21.15.67:0/3965989162 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 msgr2=0x7f6988071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698cc00640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 0x7f6988071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698cc00640 1 -- 172.21.15.67:0/3965989162 >> v1:172.21.15.187:6789/0 conn(0x7f6988072bb0 legacy=0x7f69880ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698cc00640 1 -- 172.21.15.67:0/3965989162 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880cf030 con 0x7f69880ce3b0 2024-04-17T08:47:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.476+0000 7f698d600640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 0x7f6988071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.477+0000 7f698cc00640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880ce3b0 0x7f6988072670 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f6978009870 tx=0x7f697802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe6a3939e622249 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.477+0000 7f6987e00640 1 -- 172.21.15.67:0/3965989162 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6978035020 con 0x7f69880ce3b0 2024-04-17T08:47:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6987e00640 1 -- 172.21.15.67:0/3965989162 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6978031640 con 0x7f69880ce3b0 2024-04-17T08:47:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6987e00640 1 -- 172.21.15.67:0/3965989162 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697802d040 con 0x7f69880ce3b0 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 -- 172.21.15.67:0/3965989162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880ce3b0 msgr2=0x7f6988072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880ce3b0 0x7f6988072670 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f6978009870 tx=0x7f697802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 -- 172.21.15.67:0/3965989162 shutdown_connections 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69880ce3b0 0x7f6988072670 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 --2- 172.21.15.67:0/3965989162 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6988145ac0 0x7f6988071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 -- 172.21.15.67:0/3965989162 >> 172.21.15.67:0/3965989162 conn(0x7f6988067ee0 msgr2=0x7f698806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 -- 172.21.15.67:0/3965989162 shutdown_connections 2024-04-17T08:47:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.478+0000 7f6990000640 1 -- 172.21.15.67:0/3965989162 wait complete. 2024-04-17T08:47:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.480+0000 7f6990000640 1 Processor -- start 2024-04-17T08:47:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.480+0000 7f6990000640 1 -- start start 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6988072bb0 0x7f6988139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880ce3b0 0x7f698806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6988145ea0 con 0x7f6988145ac0 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69880cf5f0 con 0x7f6988072bb0 2024-04-17T08:47:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f6990000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69880d0b20 con 0x7f69880ce3b0 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.481+0000 7f698e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58464/0 (socket says 172.21.15.67:58464) 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 -- 172.21.15.67:0/2664898359 learned_addr learned my addr 172.21.15.67:0/2664898359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698d600640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6988072bb0 0x7f6988139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698cc00640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880ce3b0 0x7f698806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 -- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880ce3b0 msgr2=0x7f698806ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880ce3b0 0x7f698806ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 -- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6988072bb0 msgr2=0x7f6988139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6988072bb0 0x7f6988139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 -- 172.21.15.67:0/2664898359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880c9f30 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.482+0000 7f698e000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f698000c780 tx=0x7f698000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.483+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980004b60 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.483+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6980004d00 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.483+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698000aa90 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.483+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69880ca210 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.483+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69880ca7d0 con 0x7f6988145ac0 2024-04-17T08:47:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.484+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6988142d00 con 0x7f6988145ac0 2024-04-17T08:47:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.485+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f698000ac30 con 0x7f6988145ac0 2024-04-17T08:47:32.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.486+0000 7f6985600640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 0x7f696c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.486+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f698008c1e0 con 0x7f6988145ac0 2024-04-17T08:47:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.487+0000 7f698d600640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 0x7f696c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.488+0000 7f698d600640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 0x7f696c06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f69880c98c0 tx=0x7f697c008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.491+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6980020070 con 0x7f6988145ac0 2024-04-17T08:47:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.725+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f6988143c10 con 0x7f6988145ac0 2024-04-17T08:47:32.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.731+0000 7f6985600640 1 -- 172.21.15.67:0/2664898359 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) ==== 115+0+0 (secure 0 0 0) 0x7f698005a330 con 0x7f6988145ac0 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 msgr2=0x7f696c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 0x7f696c06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f69880c98c0 tx=0x7f697c008040 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 msgr2=0x7f698815cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f698000c780 tx=0x7f698000cc40 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 shutdown_connections 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f696c06ca10 0x7f696c06eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988145ac0 0x7f698815cdb0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69880ce3b0 0x7f698806ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 --2- 172.21.15.67:0/2664898359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6988072bb0 0x7f6988139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.733+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 >> 172.21.15.67:0/2664898359 conn(0x7f6988067ee0 msgr2=0x7f69880b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:32.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.734+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 shutdown_connections 2024-04-17T08:47:32.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.734+0000 7f6990000640 1 -- 172.21.15.67:0/2664898359 wait complete. 2024-04-17T08:47:32.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:47:32.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.961+0000 7f4f52800640 1 Processor -- start 2024-04-17T08:47:32.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.961+0000 7f4f52800640 1 -- start start 2024-04-17T08:47:32.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f52800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f4c09b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f52800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4c0b1970 con 0x7f4f4c0d0170 2024-04-17T08:47:32.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f52800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4c09b7e0 con 0x7f4f4c15ff30 2024-04-17T08:47:32.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f52800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4c09bac0 con 0x7f4f4c0b1400 2024-04-17T08:47:32.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f51e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4f4c0d0170 0x7f4f4c0af450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60108/0 (socket says 172.21.15.67:60108) 2024-04-17T08:47:32.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.963+0000 7f4f51e00640 1 -- 172.21.15.67:0/4143145789 learned_addr learned my addr 172.21.15.67:0/4143145789 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:32.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f50a00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f4c09b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236745241 0 0) 0x7f4f4c0b1970 con 0x7f4f4c0d0170 2024-04-17T08:47:32.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f34003520 con 0x7f4f4c0d0170 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 170776469 0 0) 0x7f4f34003520 con 0x7f4f4c0d0170 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 msgr2=0x7f4f4c09b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f4c09b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 >> v1:172.21.15.187:6789/0 conn(0x7f4f4c15ff30 legacy=0x7f4f4c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4c160310 con 0x7f4f4c0d0170 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4f440019f0 con 0x7f4f4c0d0170 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 >> v1:172.21.15.67:6789/0 conn(0x7f4f4c0d0170 legacy=0x7f4f4c0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340039e0 0x7f4f34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f34004800 0x7f4f34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c09bac0 con 0x7f4f340039e0 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34005520 con 0x7f4f34004800 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.964+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3400bf00 con 0x7f4f4c0b1400 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340039e0 0x7f4f34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f50a00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f34004800 0x7f4f34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51e00640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 -- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 msgr2=0x7f4f3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c0b1400 0x7f4f3400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 -- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f34004800 msgr2=0x7f4f34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f34004800 0x7f4f34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.965+0000 7f4f51400640 1 -- 172.21.15.67:0/4143145789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f34003520 con 0x7f4f340039e0 2024-04-17T08:47:32.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.966+0000 7f4f51400640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340039e0 0x7f4f34003e00 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f4f40001750 tx=0x7f4f34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d416701849dc5a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40001370 con 0x7f4f340039e0 2024-04-17T08:47:32.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f400052d0 con 0x7f4f340039e0 2024-04-17T08:47:32.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f4be00640 1 -- 172.21.15.67:0/4143145789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40003830 con 0x7f4f340039e0 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f52800640 1 -- 172.21.15.67:0/4143145789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340039e0 msgr2=0x7f4f34003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f52800640 1 --2- 172.21.15.67:0/4143145789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f340039e0 0x7f4f34003e00 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f4f40001750 tx=0x7f4f34005830 comp rx=0 tx=0).stop 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f51e00640 1 -- 172.21.15.67:0/4143145789 reap_dead start 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f52800640 1 -- 172.21.15.67:0/4143145789 shutdown_connections 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.967+0000 7f4f52800640 1 -- 172.21.15.67:0/4143145789 >> 172.21.15.67:0/4143145789 conn(0x7f4f4c0b4090 msgr2=0x7f4f4c0b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.968+0000 7f4f52800640 1 -- 172.21.15.67:0/4143145789 shutdown_connections 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.968+0000 7f4f52800640 1 -- 172.21.15.67:0/4143145789 wait complete. 2024-04-17T08:47:32.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.969+0000 7f4f52800640 1 Processor -- start 2024-04-17T08:47:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.969+0000 7f4f52800640 1 -- start start 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 0x7f4f4c0b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 0x7f4f4c1487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c09b2f0 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c09b170 con 0x7f4f340039e0 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.970+0000 7f4f52800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4c09b470 con 0x7f4f4c148350 2024-04-17T08:47:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58474/0 (socket says 172.21.15.67:58474) 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 -- 172.21.15.67:0/1045140807 learned_addr learned my addr 172.21.15.67:0/1045140807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f51400640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 0x7f4f4c0b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f51e00640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 0x7f4f4c1487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 -- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 msgr2=0x7f4f4c1487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 0x7f4f4c1487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 -- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 msgr2=0x7f4f4c0b7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 0x7f4f4c0b7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f50a00640 1 -- 172.21.15.67:0/1045140807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4c15c490 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f51400640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 0x7f4f4c0b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.971+0000 7f4f51e00640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 0x7f4f4c1487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f50a00640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f4f38011660 tx=0x7f4f38011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3800f9c0 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3800fb60 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38015460 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f4c15c730 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.972+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f4c15ccf0 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.973+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f4c068a10 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.974+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f4f38015600 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.975+0000 7f4f49600640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 0x7f4f2c06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.975+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f4f3808c700 con 0x7f4f4c1ef7d0 2024-04-17T08:47:32.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.976+0000 7f4f51400640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 0x7f4f2c06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:32.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.977+0000 7f4f51400640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 0x7f4f2c06ee10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4f40033000 tx=0x7f4f40009030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:32.980+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4f38019030 con 0x7f4f4c1ef7d0 2024-04-17T08:47:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.185+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f4f4c06ee50 con 0x7f4f4c1ef7d0 2024-04-17T08:47:33.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.191+0000 7f4f49600640 1 -- 172.21.15.67:0/1045140807 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v68) ==== 104+0+0 (secure 0 0 0) 0x7f4f3805a8e0 con 0x7f4f4c1ef7d0 2024-04-17T08:47:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 msgr2=0x7f4f2c06ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 0x7f4f2c06ee10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4f40033000 tx=0x7f4f40009030 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 msgr2=0x7f4f4c0b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f4f38011660 tx=0x7f4f38011b20 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 shutdown_connections 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f4f2c06c950 0x7f4f2c06ee10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f4c148350 0x7f4f4c1487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f4c1ef7d0 0x7f4f4c0b51c0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 --2- 172.21.15.67:0/1045140807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f340039e0 0x7f4f4c0b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 >> 172.21.15.67:0/1045140807 conn(0x7f4f4c0b4090 msgr2=0x7f4f4c070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 shutdown_connections 2024-04-17T08:47:33.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.194+0000 7f4f52800640 1 -- 172.21.15.67:0/1045140807 wait complete. 2024-04-17T08:47:33.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:47:33.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.431+0000 7f4066c00640 1 Processor -- start 2024-04-17T08:47:33.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.432+0000 7f4066c00640 1 -- start start 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.433+0000 7f4066c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4060145ac0 0x7f4060071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.433+0000 7f4066c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060072d80 0x7f40600cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.433+0000 7f4066c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40600cea70 con 0x7f40600ce3b0 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.433+0000 7f4066c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40600ce840 con 0x7f4060145ac0 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.433+0000 7f4066c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40600ced50 con 0x7f4060072d80 2024-04-17T08:47:33.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4060145ac0 0x7f4060071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4060145ac0 0x7f4060071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52414/0 (socket says 172.21.15.67:52414) 2024-04-17T08:47:33.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 -- 172.21.15.67:0/1435675404 learned_addr learned my addr 172.21.15.67:0/1435675404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:33.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 -- 172.21.15.67:0/1435675404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060072d80 msgr2=0x7f40600cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:33.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 --2- 172.21.15.67:0/1435675404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060072d80 0x7f40600cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 -- 172.21.15.67:0/1435675404 >> v1:172.21.15.187:6789/0 conn(0x7f40600ce3b0 legacy=0x7f4060072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:33.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.434+0000 7f405fe00640 1 -- 172.21.15.67:0/1435675404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40600cf030 con 0x7f4060145ac0 2024-04-17T08:47:33.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.435+0000 7f405fe00640 1 --2- 172.21.15.67:0/1435675404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4060145ac0 0x7f4060071ea0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4050009e80 tx=0x7f405002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba071a4a6884850b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.436+0000 7f405ea00640 1 -- 172.21.15.67:0/1435675404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405002ca50 con 0x7f4060145ac0 2024-04-17T08:47:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.436+0000 7f405ea00640 1 -- 172.21.15.67:0/1435675404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405002cbf0 con 0x7f4060145ac0 2024-04-17T08:47:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.436+0000 7f405ea00640 1 -- 172.21.15.67:0/1435675404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500343f0 con 0x7f4060145ac0 2024-04-17T08:47:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.436+0000 7f4066c00640 1 -- 172.21.15.67:0/1435675404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4060145ac0 msgr2=0x7f4060071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.436+0000 7f4066c00640 1 --2- 172.21.15.67:0/1435675404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4060145ac0 0x7f4060071ea0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4050009e80 tx=0x7f405002beb0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 -- 172.21.15.67:0/1435675404 shutdown_connections 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 --2- 172.21.15.67:0/1435675404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060072d80 0x7f40600cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 --2- 172.21.15.67:0/1435675404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4060145ac0 0x7f4060071ea0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 -- 172.21.15.67:0/1435675404 >> 172.21.15.67:0/1435675404 conn(0x7f4060067ee0 msgr2=0x7f406006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 -- 172.21.15.67:0/1435675404 shutdown_connections 2024-04-17T08:47:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.437+0000 7f4066c00640 1 -- 172.21.15.67:0/1435675404 wait complete. 2024-04-17T08:47:33.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.439+0000 7f4066c00640 1 Processor -- start 2024-04-17T08:47:33.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.439+0000 7f4066c00640 1 -- start start 2024-04-17T08:47:33.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4060072d80 0x7f406015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060145ac0 0x7f406006e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4060145ea0 con 0x7f40600ce3b0 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40600cf5f0 con 0x7f4060072d80 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4066c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40600d0050 con 0x7f4060145ac0 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f405f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f405f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52416/0 (socket says 172.21.15.67:52416) 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f405f400640 1 -- 172.21.15.67:0/1493991179 learned_addr learned my addr 172.21.15.67:0/1493991179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f405fe00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4060072d80 0x7f406015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.440+0000 7f4064c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060145ac0 0x7f406006e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 -- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060145ac0 msgr2=0x7f406006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060145ac0 0x7f406006e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 -- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4060072d80 msgr2=0x7f406015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4060072d80 0x7f406015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 -- 172.21.15.67:0/1493991179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406013a850 con 0x7f40600ce3b0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f405f400640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f404800c430 tx=0x7f404800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048011790 con 0x7f40600ce3b0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4048017070 con 0x7f40600ce3b0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048011b70 con 0x7f40600ce3b0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406013ab30 con 0x7f40600ce3b0 2024-04-17T08:47:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.441+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4060140bb0 con 0x7f40600ce3b0 2024-04-17T08:47:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.442+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40600cc6c0 con 0x7f40600ce3b0 2024-04-17T08:47:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.443+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f4048011d10 con 0x7f40600ce3b0 2024-04-17T08:47:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.444+0000 7f4057e00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 0x7f403c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.444+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f404808bc10 con 0x7f40600ce3b0 2024-04-17T08:47:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.445+0000 7f405fe00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 0x7f403c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.446+0000 7f405fe00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 0x7f403c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4050035000 tx=0x7f4050030000 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.449+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4048059e00 con 0x7f40600ce3b0 2024-04-17T08:47:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.652+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f40600685d0 con 0x7f40600ce3b0 2024-04-17T08:47:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.657+0000 7f4057e00640 1 -- 172.21.15.67:0/1493991179 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v69) ==== 120+0+0 (secure 0 0 0) 0x7f4048059700 con 0x7f40600ce3b0 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 msgr2=0x7f403c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 0x7f403c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4050035000 tx=0x7f4050030000 comp rx=0 tx=0).stop 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 msgr2=0x7f4060140210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f404800c430 tx=0x7f404800c8f0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 shutdown_connections 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f403c06ca10 0x7f403c06eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4060145ac0 0x7f406006e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40600ce3b0 0x7f4060140210 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 --2- 172.21.15.67:0/1493991179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4060072d80 0x7f406015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 >> 172.21.15.67:0/1493991179 conn(0x7f4060067ee0 msgr2=0x7f40600b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.660+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 shutdown_connections 2024-04-17T08:47:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.661+0000 7f4066c00640 1 -- 172.21.15.67:0/1493991179 wait complete. 2024-04-17T08:47:33.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-04-17T08:47:33.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.905+0000 7f0818a00640 1 Processor -- start 2024-04-17T08:47:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.905+0000 7f0818a00640 1 -- start start 2024-04-17T08:47:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.906+0000 7f0818a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f081415ac80 0x7f081406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.906+0000 7f0818a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0814072eb0 0x7f0814071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.906+0000 7f0818a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08140cd920 con 0x7f0814072330 2024-04-17T08:47:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.906+0000 7f0818a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f08140727a0 con 0x7f081415ac80 2024-04-17T08:47:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.906+0000 7f0818a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f08140cdc30 con 0x7f0814072eb0 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f081415ac80 0x7f081406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f081415ac80 0x7f081406f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52430/0 (socket says 172.21.15.67:52430) 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 -- 172.21.15.67:0/3034886201 learned_addr learned my addr 172.21.15.67:0/3034886201 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0811e00640 1 --2- 172.21.15.67:0/3034886201 >> v2:172.21.15.187:3300/0 conn(0x7f0814072eb0 0x7f0814071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 -- 172.21.15.67:0/3034886201 >> v2:172.21.15.187:3300/0 conn(0x7f0814072eb0 msgr2=0x7f0814071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 --2- 172.21.15.67:0/3034886201 >> v2:172.21.15.187:3300/0 conn(0x7f0814072eb0 0x7f0814071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 -- 172.21.15.67:0/3034886201 >> v1:172.21.15.187:6789/0 conn(0x7f0814072330 legacy=0x7f081406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.907+0000 7f0812800640 1 -- 172.21.15.67:0/3034886201 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081406f980 con 0x7f081415ac80 2024-04-17T08:47:33.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.908+0000 7f0812800640 1 --2- 172.21.15.67:0/3034886201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f081415ac80 0x7f081406f010 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f08140d03b0 tx=0x7f080402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb96407ffc7c5f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0811400640 1 -- 172.21.15.67:0/3034886201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080402c9f0 con 0x7f081415ac80 2024-04-17T08:47:33.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0811400640 1 -- 172.21.15.67:0/3034886201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080402cb90 con 0x7f081415ac80 2024-04-17T08:47:33.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0811400640 1 -- 172.21.15.67:0/3034886201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08040343f0 con 0x7f081415ac80 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 -- 172.21.15.67:0/3034886201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f081415ac80 msgr2=0x7f081406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 --2- 172.21.15.67:0/3034886201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f081415ac80 0x7f081406f010 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f08140d03b0 tx=0x7f080402be50 comp rx=0 tx=0).stop 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 -- 172.21.15.67:0/3034886201 shutdown_connections 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 --2- 172.21.15.67:0/3034886201 >> v2:172.21.15.187:3300/0 conn(0x7f0814072eb0 0x7f0814071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 --2- 172.21.15.67:0/3034886201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f081415ac80 0x7f081406f010 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.909+0000 7f0818a00640 1 -- 172.21.15.67:0/3034886201 >> 172.21.15.67:0/3034886201 conn(0x7f0814067d10 msgr2=0x7f08140b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.910+0000 7f0818a00640 1 -- 172.21.15.67:0/3034886201 shutdown_connections 2024-04-17T08:47:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.910+0000 7f0818a00640 1 -- 172.21.15.67:0/3034886201 wait complete. 2024-04-17T08:47:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.911+0000 7f0818a00640 1 Processor -- start 2024-04-17T08:47:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.911+0000 7f0818a00640 1 -- start start 2024-04-17T08:47:33.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0814072eb0 0x7f081414edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f081415ac80 0x7f08141542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f081415b060 con 0x7f0814072330 2024-04-17T08:47:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f081406ff40 con 0x7f081415ac80 2024-04-17T08:47:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.912+0000 7f0818a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08140a9ea0 con 0x7f0814072eb0 2024-04-17T08:47:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52444/0 (socket says 172.21.15.67:52444) 2024-04-17T08:47:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 -- 172.21.15.67:0/3146169646 learned_addr learned my addr 172.21.15.67:0/3146169646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0811e00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0814072eb0 0x7f081414edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0813200640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f081415ac80 0x7f08141542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 -- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0814072eb0 msgr2=0x7f081414edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0814072eb0 0x7f081414edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 -- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f081415ac80 msgr2=0x7f08141542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f081415ac80 0x7f08141542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.913+0000 7f0812800640 1 -- 172.21.15.67:0/3146169646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08140ab3e0 con 0x7f0814072330 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f0812800640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f0804000f80 tx=0x7f0804004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0804043020 con 0x7f0814072330 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080403e450 con 0x7f0814072330 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080403e840 con 0x7f0814072330 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08140aa0e0 con 0x7f0814072330 2024-04-17T08:47:33.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.914+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08140aa6a0 con 0x7f0814072330 2024-04-17T08:47:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.915+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0814068a10 con 0x7f0814072330 2024-04-17T08:47:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.916+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f0804046020 con 0x7f0814072330 2024-04-17T08:47:33.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.918+0000 7f080aa00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 0x7f07f406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:33.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.918+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f08040ae780 con 0x7f0814072330 2024-04-17T08:47:33.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.918+0000 7f0811e00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 0x7f07f406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:33.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.920+0000 7f0811e00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 0x7f07f406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f08141f5e00 tx=0x7f0800008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:33.923+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f080404d030 con 0x7f0814072330 2024-04-17T08:47:34.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.197+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7f0814152040 con 0x7f0814072330 2024-04-17T08:47:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.220+0000 7f080aa00640 1 -- 172.21.15.67:0/3146169646 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3059) ==== 165+0+0 (secure 0 0 0) 0x7f08140cd920 con 0x7f0814072330 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 msgr2=0x7f07f406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 0x7f07f406eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f08141f5e00 tx=0x7f0800008040 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 msgr2=0x7f08140aaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f0804000f80 tx=0x7f0804004a40 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 shutdown_connections 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f07f406ca10 0x7f07f406eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f081415ac80 0x7f08141542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0814072eb0 0x7f081414edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 --2- 172.21.15.67:0/3146169646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814072330 0x7f08140aaea0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 >> 172.21.15.67:0/3146169646 conn(0x7f0814067d10 msgr2=0x7f08140b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 shutdown_connections 2024-04-17T08:47:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.224+0000 7f0818a00640 1 -- 172.21.15.67:0/3146169646 wait complete. 2024-04-17T08:47:34.243 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-04-17T08:47:34.243 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-04-17T08:47:34.243 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-04-17T08:47:34.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.470+0000 7fd51be00640 1 Processor -- start 2024-04-17T08:47:34.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.470+0000 7fd51be00640 1 -- start start 2024-04-17T08:47:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd51c0b6d00 0x7fd51c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd51c0cf910 0x7fd51c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51c0d2d30 con 0x7fd51c15ec60 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd51c0d49c0 con 0x7fd51c0cf910 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd51c0d3660 con 0x7fd51c0b6d00 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd51c15ec60 0x7fd51c0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43912/0 (socket says 172.21.15.67:43912) 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51aa00640 1 -- 172.21.15.67:0/2448427115 learned_addr learned my addr 172.21.15.67:0/2448427115 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.472+0000 7fd51b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd51c0cf910 0x7fd51c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd512000640 1 --2- 172.21.15.67:0/2448427115 >> v2:172.21.15.187:3300/0 conn(0x7fd51c0b6d00 0x7fd51c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd51b400640 1 -- 172.21.15.67:0/2448427115 >> v2:172.21.15.187:3300/0 conn(0x7fd51c0b6d00 msgr2=0x7fd51c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd51b400640 1 --2- 172.21.15.67:0/2448427115 >> v2:172.21.15.187:3300/0 conn(0x7fd51c0b6d00 0x7fd51c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd51a000640 1 -- 172.21.15.67:0/2448427115 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370948459 0 0) 0x7fd51c0d2d30 con 0x7fd51c15ec60 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd51b400640 1 -- 172.21.15.67:0/2448427115 >> v1:172.21.15.67:6789/0 conn(0x7fd51c15ec60 legacy=0x7fd51c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.473+0000 7fd51b400640 1 -- 172.21.15.67:0/2448427115 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c0d3010 con 0x7fd51c0cf910 2024-04-17T08:47:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.474+0000 7fd51b400640 1 --2- 172.21.15.67:0/2448427115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c0b7b70 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0ba590 tx=0x7fd51402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c333d72f741cdaf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.474+0000 7fd51a000640 1 -- 172.21.15.67:0/2448427115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514039070 con 0x7fd51c0cf910 2024-04-17T08:47:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51a000640 1 -- 172.21.15.67:0/2448427115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd51402cc80 con 0x7fd51c0cf910 2024-04-17T08:47:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51a000640 1 -- 172.21.15.67:0/2448427115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514034660 con 0x7fd51c0cf910 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 -- 172.21.15.67:0/2448427115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 msgr2=0x7fd51c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 --2- 172.21.15.67:0/2448427115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c0b7b70 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0ba590 tx=0x7fd51402bed0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 -- 172.21.15.67:0/2448427115 shutdown_connections 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 --2- 172.21.15.67:0/2448427115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c0b7b70 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 --2- 172.21.15.67:0/2448427115 >> v2:172.21.15.187:3300/0 conn(0x7fd51c0b6d00 0x7fd51c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 -- 172.21.15.67:0/2448427115 >> 172.21.15.67:0/2448427115 conn(0x7fd51c0d15f0 msgr2=0x7fd51c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 -- 172.21.15.67:0/2448427115 shutdown_connections 2024-04-17T08:47:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.475+0000 7fd51be00640 1 -- 172.21.15.67:0/2448427115 wait complete. 2024-04-17T08:47:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.477+0000 7fd51be00640 1 Processor -- start 2024-04-17T08:47:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.477+0000 7fd51be00640 1 -- start start 2024-04-17T08:47:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd51c0b6d00 0x7fd51c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd51c15ec60 0x7fd51c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c15f040 con 0x7fd51c0cf910 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c0cfdc0 con 0x7fd51c15ec60 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd51be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd51c0d00c0 con 0x7fd51c0b6d00 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd512000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd512000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52476/0 (socket says 172.21.15.67:52476) 2024-04-17T08:47:34.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.478+0000 7fd512000640 1 -- 172.21.15.67:0/1790603458 learned_addr learned my addr 172.21.15.67:0/1790603458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 -- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd51c0b6d00 msgr2=0x7fd51c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd51c0b6d00 0x7fd51c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 -- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd51c15ec60 msgr2=0x7fd51c15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd51b400640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd51c15ec60 0x7fd51c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd51c15ec60 0x7fd51c15c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 -- 172.21.15.67:0/1790603458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c15f600 con 0x7fd51c0cf910 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd512000640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fd50400d830 tx=0x7fd50400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50400b6c0 con 0x7fd51c0cf910 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd504019070 con 0x7fd51c0cf910 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd51c15f8e0 con 0x7fd51c0cf910 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.479+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50400ba80 con 0x7fd51c0cf910 2024-04-17T08:47:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.480+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51c15fea0 con 0x7fd51c0cf910 2024-04-17T08:47:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.481+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd51c068a10 con 0x7fd51c0cf910 2024-04-17T08:47:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.482+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fd50401e460 con 0x7fd51c0cf910 2024-04-17T08:47:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.482+0000 7fd513400640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 0x7fd4f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.483+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7fd50408c570 con 0x7fd51c0cf910 2024-04-17T08:47:34.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.483+0000 7fd51aa00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 0x7fd4f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.484+0000 7fd51aa00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 0x7fd4f806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd50c000c00 tx=0x7fd50c0079c0 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.487+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd50405a6c0 con 0x7fd51c0cf910 2024-04-17T08:47:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.694+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd51c0ba300 con 0x7fd51c0cf910 2024-04-17T08:47:34.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.696+0000 7fd513400640 1 -- 172.21.15.67:0/1790603458 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) ==== 74+0+14551 (secure 0 0 0) 0x7fd504059fc0 con 0x7fd51c0cf910 2024-04-17T08:47:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:34.696 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":204,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:32.202542+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6837/3432159492":"2024-04-18T08:47:29.127221+0000","172.21.15.187:0/1001338088":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2867744332":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/2365426680":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3395595927":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6832/219728900":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/1228337607":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6833/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:6832/1430869695":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/521765659":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1964974751":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/3275456489":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/630850508":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6835/3432159492":"2024-04-18T08:47:29.127221+0000","172.21.15.187:0/74160311":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2788069147":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/3491826676":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1329453466":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2568338216":"2024-04-18T08:42:21.244475+0000","172.21.15.187:0/1472858533":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2385834843":"2024-04-18T08:31:46.865019+0000","172.21.15.187:0/2798850698":"2024-04-18T08:47:25.133801+0000","172.21.15.187:0/293550442":"2024-04-18T08:47:25.133801+0000","172.21.15.187:6833/3300567104":"2024-04-18T08:31:46.865019+0000","172.21.15.187:6832/3300567104":"2024-04-18T08:31:46.865019+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:34.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.699+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 msgr2=0x7fd4f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.699+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 0x7fd4f806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd50c000c00 tx=0x7fd50c0079c0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.699+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 msgr2=0x7fd51c15c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.699+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fd50400d830 tx=0x7fd50400dcf0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 shutdown_connections 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fd4f806ca10 0x7fd4f806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd51c15ec60 0x7fd51c15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd51c0cf910 0x7fd51c15c400 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 --2- 172.21.15.67:0/1790603458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd51c0b6d00 0x7fd51c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 >> 172.21.15.67:0/1790603458 conn(0x7fd51c0d15f0 msgr2=0x7fd51c0b83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:34.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 shutdown_connections 2024-04-17T08:47:34.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.700+0000 7fd51be00640 1 -- 172.21.15.67:0/1790603458 wait complete. 2024-04-17T08:47:34.720 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:47:34.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:47:34.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.942+0000 7f53d5a00640 1 Processor -- start 2024-04-17T08:47:34.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.943+0000 7f53d5a00640 1 -- start start 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.944+0000 7f53d5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53d0072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.944+0000 7f53d5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d00b08f0 con 0x7f53d0155840 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.944+0000 7f53d5a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d00cfb30 con 0x7f53d00b5220 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.944+0000 7f53d5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53d00cfe10 con 0x7f53d0071df0 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.945+0000 7f53cea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53d0155840 0x7f53d00b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43924/0 (socket says 172.21.15.67:43924) 2024-04-17T08:47:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.945+0000 7f53cea00640 1 -- 172.21.15.67:0/3263551759 learned_addr learned my addr 172.21.15.67:0/3263551759 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.945+0000 7f53ce000640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53d0072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.945+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4213723749 0 0) 0x7f53d00b08f0 con 0x7f53d0155840 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.945+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b4003520 con 0x7f53d0155840 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 433121824 0 0) 0x7f53b4003520 con 0x7f53d0155840 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 msgr2=0x7f53d0072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53d0072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 >> v1:172.21.15.187:6789/0 conn(0x7f53d00b5220 legacy=0x7f53d00b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00b4a20 con 0x7f53d0155840 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f53c0001670 con 0x7f53d0155840 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 >> v1:172.21.15.67:6789/0 conn(0x7f53d0155840 legacy=0x7f53d00b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b40039e0 0x7f53b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53d00cfe10 con 0x7f53b4004800 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53b400baa0 con 0x7f53b40039e0 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53b400deb0 con 0x7f53d0071df0 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.946+0000 7f53ce000640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53cf400640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53cea00640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b40039e0 0x7f53b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53ce000640 1 -- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 msgr2=0x7f53b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53ce000640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d0071df0 0x7f53b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53ce000640 1 -- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b40039e0 msgr2=0x7f53b4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53ce000640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53b40039e0 0x7f53b4003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.947+0000 7f53ce000640 1 -- 172.21.15.67:0/3263551759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b4003520 con 0x7f53b4004800 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.948+0000 7f53ce000640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53b4004c80 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f53b800aa10 tx=0x7f53b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8360ae6ef3378b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.948+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8039070 con 0x7f53b4004800 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.948+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b802cb70 con 0x7f53b4004800 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.948+0000 7f53cd600640 1 -- 172.21.15.67:0/3263551759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b800b040 con 0x7f53b4004800 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 -- 172.21.15.67:0/3263551759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 msgr2=0x7f53b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 --2- 172.21.15.67:0/3263551759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53b4004c80 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f53b800aa10 tx=0x7f53b4005830 comp rx=0 tx=0).stop 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53cf400640 1 -- 172.21.15.67:0/3263551759 reap_dead start 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 -- 172.21.15.67:0/3263551759 shutdown_connections 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 -- 172.21.15.67:0/3263551759 >> 172.21.15.67:0/3263551759 conn(0x7f53d006b8c0 msgr2=0x7f53d006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:34.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 -- 172.21.15.67:0/3263551759 shutdown_connections 2024-04-17T08:47:34.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.949+0000 7f53d5a00640 1 -- 172.21.15.67:0/3263551759 wait complete. 2024-04-17T08:47:34.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.951+0000 7f53d5a00640 1 Processor -- start 2024-04-17T08:47:34.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.951+0000 7f53d5a00640 1 -- start start 2024-04-17T08:47:34.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d01567e0 0x7f53d0158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 0x7f53d0154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0072270 con 0x7f53b4004800 2024-04-17T08:47:34.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53d00b46d0 con 0x7f53d0153e80 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53d5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53d00723f0 con 0x7f53d01567e0 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53cea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52490/0 (socket says 172.21.15.67:52490) 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.952+0000 7f53cea00640 1 -- 172.21.15.67:0/2346225669 learned_addr learned my addr 172.21.15.67:0/2346225669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 -- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d01567e0 msgr2=0x7f53d0158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53ce000640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d01567e0 0x7f53d0158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cf400640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 0x7f53d0154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d01567e0 0x7f53d0158610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 -- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 msgr2=0x7f53d0154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 0x7f53d0154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 -- 172.21.15.67:0/2346225669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d01f9190 con 0x7f53b4004800 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cf400640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 0x7f53d0154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:34.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.953+0000 7f53cea00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f53c000d810 tx=0x7f53c000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.954+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0019070 con 0x7f53b4004800 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.954+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c00074e0 con 0x7f53b4004800 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.954+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0004a00 con 0x7f53b4004800 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.954+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d015e2a0 con 0x7f53b4004800 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.954+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d015e860 con 0x7f53b4004800 2024-04-17T08:47:34.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.955+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d0068b90 con 0x7f53b4004800 2024-04-17T08:47:34.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.956+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f53c001c070 con 0x7f53b4004800 2024-04-17T08:47:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.957+0000 7f53c6a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 0x7f53ac06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.957+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f53c008ba20 con 0x7f53b4004800 2024-04-17T08:47:34.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.957+0000 7f53ce000640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 0x7f53ac06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:34.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.958+0000 7f53ce000640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 0x7f53ac06ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f53b8036000 tx=0x7f53b8030000 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:34.962+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f53d01f9190 con 0x7f53b4004800 2024-04-17T08:47:35.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.166+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53d00b2160 con 0x7f53b4004800 2024-04-17T08:47:35.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.166+0000 7f53c6a00640 1 -- 172.21.15.67:0/2346225669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) ==== 93+0+10 (secure 0 0 0) 0x7f53c0059a60 con 0x7f53b4004800 2024-04-17T08:47:35.166 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 msgr2=0x7f53ac06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 0x7f53ac06ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f53b8036000 tx=0x7f53b8030000 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 msgr2=0x7f53d01f8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f53c000d810 tx=0x7f53c000dcd0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 shutdown_connections 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f53ac06ca40 0x7f53ac06ef00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53d0153e80 0x7f53d0154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53d01567e0 0x7f53d0158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 --2- 172.21.15.67:0/2346225669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b4004800 0x7f53d01f8c50 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 >> 172.21.15.67:0/2346225669 conn(0x7f53d006b8c0 msgr2=0x7f53d015b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.169+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 shutdown_connections 2024-04-17T08:47:35.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.170+0000 7f53d5a00640 1 -- 172.21.15.67:0/2346225669 wait complete. 2024-04-17T08:47:35.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-17T08:47:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.413+0000 7f38ca800640 1 Processor -- start 2024-04-17T08:47:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.413+0000 7f38ca800640 1 -- start start 2024-04-17T08:47:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.415+0000 7f38ca800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38c4160280 0x7f38c406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.415+0000 7f38ca800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c40cd4a0 con 0x7f38c4141ed0 2024-04-17T08:47:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.415+0000 7f38ca800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c41607f0 con 0x7f38c406b8c0 2024-04-17T08:47:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.415+0000 7f38ca800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f38c415ce30 con 0x7f38c4160280 2024-04-17T08:47:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.415+0000 7f38c9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f38c4141ed0 0x7f38c415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43938/0 (socket says 172.21.15.67:43938) 2024-04-17T08:47:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c9e00640 1 -- 172.21.15.67:0/1954945818 learned_addr learned my addr 172.21.15.67:0/1954945818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297433773 0 0) 0x7f38c40cd4a0 con 0x7f38c4141ed0 2024-04-17T08:47:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ac003520 con 0x7f38c4141ed0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c8a00640 1 --2- 172.21.15.67:0/1954945818 >> v2:172.21.15.187:3300/0 conn(0x7f38c4160280 0x7f38c406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4204398450 0 0) 0x7f38ac003520 con 0x7f38c4141ed0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 >> v2:172.21.15.187:3300/0 conn(0x7f38c4160280 msgr2=0x7f38c406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 --2- 172.21.15.67:0/1954945818 >> v2:172.21.15.187:3300/0 conn(0x7f38c4160280 0x7f38c406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 >> v1:172.21.15.187:6789/0 conn(0x7f38c406b8c0 legacy=0x7f38c40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c4160ad0 con 0x7f38c4141ed0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f38bc0019f0 con 0x7f38c4141ed0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 >> v1:172.21.15.67:6789/0 conn(0x7f38c4141ed0 legacy=0x7f38c415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.416+0000 7f38c3e00640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 0x7f38ac003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c3e00640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac004800 0x7f38ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c3e00640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38ac0052e0 0x7f38ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38c415ce30 con 0x7f38ac003a70 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac00c1e0 con 0x7f38ac0052e0 2024-04-17T08:47:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac00c3e0 con 0x7f38ac004800 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 0x7f38ac003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9e00640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38ac0052e0 0x7f38ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c8a00640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac004800 0x7f38ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 -- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac004800 msgr2=0x7f38ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac004800 0x7f38ac004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 -- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38ac0052e0 msgr2=0x7f38ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38ac0052e0 0x7f38ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9400640 1 -- 172.21.15.67:0/1954945818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac003520 con 0x7f38ac003a70 2024-04-17T08:47:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.417+0000 7f38c9e00640 1 -- 172.21.15.67:0/1954945818 reap_dead start 2024-04-17T08:47:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38c9400640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 0x7f38ac003e90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f38b8001720 tx=0x7f38ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faeb82518bc73393 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8001370 con 0x7f38ac003a70 2024-04-17T08:47:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b80052d0 con 0x7f38ac003a70 2024-04-17T08:47:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38c3e00640 1 -- 172.21.15.67:0/1954945818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8003830 con 0x7f38ac003a70 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38ca800640 1 -- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 msgr2=0x7f38ac003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.419+0000 7f38ca800640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 0x7f38ac003e90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f38b8001720 tx=0x7f38ac005830 comp rx=0 tx=0).stop 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.420+0000 7f38ca800640 1 -- 172.21.15.67:0/1954945818 shutdown_connections 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.420+0000 7f38ca800640 1 --2- 172.21.15.67:0/1954945818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38ac003a70 0x7f38ac003e90 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.420+0000 7f38ca800640 1 -- 172.21.15.67:0/1954945818 >> 172.21.15.67:0/1954945818 conn(0x7f38c4067d10 msgr2=0x7f38c40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.420+0000 7f38ca800640 1 -- 172.21.15.67:0/1954945818 shutdown_connections 2024-04-17T08:47:35.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.420+0000 7f38ca800640 1 -- 172.21.15.67:0/1954945818 wait complete. 2024-04-17T08:47:35.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.421+0000 7f38ca800640 1 Processor -- start 2024-04-17T08:47:35.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.422+0000 7f38ca800640 1 -- start start 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac003a70 0x7f38c4142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c4143e80 0x7f38c41f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38c413f9b0 con 0x7f38c4155d70 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38c4161090 con 0x7f38c4143e80 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38ca800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38c413fb30 con 0x7f38ac003a70 2024-04-17T08:47:35.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52516/0 (socket says 172.21.15.67:52516) 2024-04-17T08:47:35.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 -- 172.21.15.67:0/322317665 learned_addr learned my addr 172.21.15.67:0/322317665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c9400640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac003a70 0x7f38c4142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c9e00640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c4143e80 0x7f38c41f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 -- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac003a70 msgr2=0x7f38c4142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac003a70 0x7f38c4142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 -- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c4143e80 msgr2=0x7f38c41f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c4143e80 0x7f38c41f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.423+0000 7f38c8a00640 1 -- 172.21.15.67:0/322317665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c415ea10 con 0x7f38c4155d70 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38c8a00640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f38b000e810 tx=0x7f38b000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0017070 con 0x7f38c4155d70 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b000c6d0 con 0x7f38c4155d70 2024-04-17T08:47:35.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000f040 con 0x7f38c4155d70 2024-04-17T08:47:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c415ecf0 con 0x7f38c4155d70 2024-04-17T08:47:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.424+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c415f2b0 con 0x7f38c4155d70 2024-04-17T08:47:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.425+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4068a10 con 0x7f38c4155d70 2024-04-17T08:47:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.426+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f38b001a070 con 0x7f38c4155d70 2024-04-17T08:47:35.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.428+0000 7f38c1600640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 0x7f38a406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:35.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.428+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f38b008bbe0 con 0x7f38c4155d70 2024-04-17T08:47:35.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.428+0000 7f38c9400640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 0x7f38a406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:35.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.429+0000 7f38c9400640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 0x7f38a406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38b8033000 tx=0x7f38b8009030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:35.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.432+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f38b0059c20 con 0x7f38c4155d70 2024-04-17T08:47:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:35.711+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7f38c4144260 con 0x7f38c4155d70 2024-04-17T08:47:36.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.235+0000 7f38c1600640 1 -- 172.21.15.67:0/322317665 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3061) ==== 166+0+0 (secure 0 0 0) 0x7f38b0059520 con 0x7f38c4155d70 2024-04-17T08:47:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.238+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 msgr2=0x7f38a406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.238+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 0x7f38a406eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f38b8033000 tx=0x7f38b8009030 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.238+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 msgr2=0x7f38c4143940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.238+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f38b000e810 tx=0x7f38b000ecd0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 shutdown_connections 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f38a406ca10 0x7f38a406eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38c4143e80 0x7f38c41f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38c4155d70 0x7f38c4143940 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 --2- 172.21.15.67:0/322317665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38ac003a70 0x7f38c4142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 >> 172.21.15.67:0/322317665 conn(0x7f38c4067d10 msgr2=0x7f38c41415c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 shutdown_connections 2024-04-17T08:47:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.239+0000 7f38ca800640 1 -- 172.21.15.67:0/322317665 wait complete. 2024-04-17T08:47:36.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:47:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.494+0000 7f713ce00640 1 Processor -- start 2024-04-17T08:47:36.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.494+0000 7f713ce00640 1 -- start start 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.495+0000 7f713ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.495+0000 7f713ce00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f713806f800 0x7f71380b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.495+0000 7f713ce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713806df40 con 0x7f713815b6b0 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.495+0000 7f713ce00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f713806ed00 con 0x7f713806f800 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.495+0000 7f713ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f713806e870 con 0x7f713806bfd0 2024-04-17T08:47:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.496+0000 7f712ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.496+0000 7f712ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:35636/0 (socket says 172.21.15.67:35636) 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.496+0000 7f712ea00640 1 -- 172.21.15.67:0/1427743605 learned_addr learned my addr 172.21.15.67:0/1427743605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.496+0000 7f7137e00640 1 --2- 172.21.15.67:0/1427743605 >> v2:172.21.15.187:3300/0 conn(0x7f713806f800 0x7f71380b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.497+0000 7f7137e00640 1 -- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 msgr2=0x7f71380b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.497+0000 7f7137e00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.497+0000 7f7137e00640 1 -- 172.21.15.67:0/1427743605 >> v1:172.21.15.187:6789/0 conn(0x7f713815b6b0 legacy=0x7f713806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.497+0000 7f7137e00640 1 -- 172.21.15.67:0/1427743605 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713806e220 con 0x7f713806f800 2024-04-17T08:47:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.497+0000 7f712ea00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:47:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.499+0000 7f7137e00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806f800 0x7f71380b39d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f71300097a0 tx=0x7f713002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f698e0d183400024 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.499+0000 7f7136a00640 1 -- 172.21.15.67:0/1427743605 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130031780 con 0x7f713806f800 2024-04-17T08:47:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.499+0000 7f7136a00640 1 -- 172.21.15.67:0/1427743605 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7130037070 con 0x7f713806f800 2024-04-17T08:47:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.499+0000 7f7136a00640 1 -- 172.21.15.67:0/1427743605 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130031b40 con 0x7f713806f800 2024-04-17T08:47:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 -- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806f800 msgr2=0x7f71380b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806f800 0x7f71380b39d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f71300097a0 tx=0x7f713002b820 comp rx=0 tx=0).stop 2024-04-17T08:47:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 -- 172.21.15.67:0/1427743605 shutdown_connections 2024-04-17T08:47:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806f800 0x7f71380b39d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 --2- 172.21.15.67:0/1427743605 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713806bfd0 0x7f71380b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 -- 172.21.15.67:0/1427743605 >> 172.21.15.67:0/1427743605 conn(0x7f71380b2190 msgr2=0x7f71380b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 -- 172.21.15.67:0/1427743605 shutdown_connections 2024-04-17T08:47:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.500+0000 7f713ce00640 1 -- 172.21.15.67:0/1427743605 wait complete. 2024-04-17T08:47:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.502+0000 7f713ce00640 1 Processor -- start 2024-04-17T08:47:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.502+0000 7f713ce00640 1 -- start start 2024-04-17T08:47:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806bfd0 0x7f7138138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713815b6b0 0x7f7138150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71380afee0 con 0x7f713806f800 2024-04-17T08:47:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71380afd60 con 0x7f713806bfd0 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f713ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71380cfc10 con 0x7f713815b6b0 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.503+0000 7f712ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52530/0 (socket says 172.21.15.67:52530) 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 -- 172.21.15.67:0/288388426 learned_addr learned my addr 172.21.15.67:0/288388426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f7137400640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806bfd0 0x7f7138138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f7137e00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713815b6b0 0x7f7138150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 -- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713815b6b0 msgr2=0x7f7138150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713815b6b0 0x7f7138150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 -- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806bfd0 msgr2=0x7f7138138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806bfd0 0x7f7138138f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 -- 172.21.15.67:0/288388426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713814b8f0 con 0x7f713806f800 2024-04-17T08:47:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.504+0000 7f712ea00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f712000e9d0 tx=0x7f712000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.505+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120017070 con 0x7f713806f800 2024-04-17T08:47:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.505+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f712000ca60 con 0x7f713806f800 2024-04-17T08:47:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.505+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712000ce10 con 0x7f713806f800 2024-04-17T08:47:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.505+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713814bbd0 con 0x7f713806f800 2024-04-17T08:47:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.505+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713814c190 con 0x7f713806f800 2024-04-17T08:47:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.506+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7138068a10 con 0x7f713806f800 2024-04-17T08:47:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.507+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f712001a070 con 0x7f713806f800 2024-04-17T08:47:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.508+0000 7f712fe00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 0x7f711406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.508+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f712008c180 con 0x7f713806f800 2024-04-17T08:47:36.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.509+0000 7f7137400640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 0x7f711406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.510+0000 7f7137400640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 0x7f711406eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7128001ca0 tx=0x7f7128009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.512+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f712005a360 con 0x7f713806f800 2024-04-17T08:47:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.719+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f713806ee80 con 0x7f713806f800 2024-04-17T08:47:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.726+0000 7f712fe00640 1 -- 172.21.15.67:0/288388426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) ==== 122+0+0 (secure 0 0 0) 0x7f7120059c60 con 0x7f713806f800 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 msgr2=0x7f711406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 0x7f711406eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7128001ca0 tx=0x7f7128009040 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 msgr2=0x7f71381508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f712000e9d0 tx=0x7f712000ee90 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 shutdown_connections 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f711406ca10 0x7f711406eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f713815b6b0 0x7f7138150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f713806f800 0x7f71381508a0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 --2- 172.21.15.67:0/288388426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f713806bfd0 0x7f7138138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 >> 172.21.15.67:0/288388426 conn(0x7f71380b2190 msgr2=0x7f71380cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:36.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.729+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 shutdown_connections 2024-04-17T08:47:36.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:36.730+0000 7f713ce00640 1 -- 172.21.15.67:0/288388426 wait complete. 2024-04-17T08:47:36.751 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:47:36.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:47:36.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:47:37.012 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:47:37.012 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:37.012 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:37.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:47:37.076 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:47:37.076 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:37.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:47:37.134 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:37.134 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:37.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:47:37.194 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:47:37.194 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:37.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:47:37.252 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:47:37.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.483+0000 7f2f20a00640 1 Processor -- start 2024-04-17T08:47:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.483+0000 7f2f20a00640 1 -- start start 2024-04-17T08:47:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c15c5a0 0x7f2f1c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f1c0b4210 0x7f2f1c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2f1c0b4690 0x7f2f1c0af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1c15f440 con 0x7f2f1c0b4210 2024-04-17T08:47:37.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1c15f720 con 0x7f2f1c0b4690 2024-04-17T08:47:37.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.485+0000 7f2f20a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1c15fa00 con 0x7f2f1c15c5a0 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f1c0b4210 0x7f2f1c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f1c0b4210 0x7f2f1c0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52534/0 (socket says 172.21.15.67:52534) 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 -- 172.21.15.67:0/129833195 learned_addr learned my addr 172.21.15.67:0/129833195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f1b400640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c15c5a0 0x7f2f1c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f1be00640 1 --2- 172.21.15.67:0/129833195 >> v2:172.21.15.187:3300/0 conn(0x7f2f1c0b4690 0x7f2f1c0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 -- 172.21.15.67:0/129833195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c15c5a0 msgr2=0x7f2f1c15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c15c5a0 0x7f2f1c15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 -- 172.21.15.67:0/129833195 >> v2:172.21.15.187:3300/0 conn(0x7f2f1c0b4690 msgr2=0x7f2f1c0af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 --2- 172.21.15.67:0/129833195 >> v2:172.21.15.187:3300/0 conn(0x7f2f1c0b4690 0x7f2f1c0af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.486+0000 7f2f12a00640 1 -- 172.21.15.67:0/129833195 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c15fce0 con 0x7f2f1c0b4210 2024-04-17T08:47:37.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.488+0000 7f2f12a00640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c0af440 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009870 tx=0x7f2f0402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f54b839e8ddfc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:37.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.488+0000 7f2f1aa00640 1 -- 172.21.15.67:0/129833195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04038070 con 0x7f2f1c0b4210 2024-04-17T08:47:37.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.488+0000 7f2f1aa00640 1 -- 172.21.15.67:0/129833195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f04031740 con 0x7f2f1c0b4210 2024-04-17T08:47:37.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.488+0000 7f2f1aa00640 1 -- 172.21.15.67:0/129833195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0402f050 con 0x7f2f1c0b4210 2024-04-17T08:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 -- 172.21.15.67:0/129833195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 msgr2=0x7f2f1c0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c0af440 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009870 tx=0x7f2f0402b7e0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 -- 172.21.15.67:0/129833195 shutdown_connections 2024-04-17T08:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 --2- 172.21.15.67:0/129833195 >> v2:172.21.15.187:3300/0 conn(0x7f2f1c0b4690 0x7f2f1c0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c0af440 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 --2- 172.21.15.67:0/129833195 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c15c5a0 0x7f2f1c15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 -- 172.21.15.67:0/129833195 >> 172.21.15.67:0/129833195 conn(0x7f2f1c067d10 msgr2=0x7f2f1c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 -- 172.21.15.67:0/129833195 shutdown_connections 2024-04-17T08:47:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.489+0000 7f2f20a00640 1 -- 172.21.15.67:0/129833195 wait complete. 2024-04-17T08:47:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.491+0000 7f2f20a00640 1 Processor -- start 2024-04-17T08:47:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.492+0000 7f2f20a00640 1 -- start start 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c0b4690 0x7f2f1c143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f1c15c5a0 0x7f2f1c1e8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c06e7f0 con 0x7f2f1c0b4210 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1c06e670 con 0x7f2f1c15c5a0 2024-04-17T08:47:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f20a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1c071df0 con 0x7f2f1c0b4690 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f1b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f1b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52540/0 (socket says 172.21.15.67:52540) 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f1b400640 1 -- 172.21.15.67:0/1301275554 learned_addr learned my addr 172.21.15.67:0/1301275554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f12a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c0b4690 0x7f2f1c143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.493+0000 7f2f1be00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f1c15c5a0 0x7f2f1c1e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 -- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c0b4690 msgr2=0x7f2f1c143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c0b4690 0x7f2f1c143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 -- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f1c15c5a0 msgr2=0x7f2f1c1e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f1c15c5a0 0x7f2f1c1e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 -- 172.21.15.67:0/1301275554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1c144570 con 0x7f2f1c0b4210 2024-04-17T08:47:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f1b400640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00d620 tx=0x7f2f0c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00b620 con 0x7f2f1c0b4210 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0c019070 con 0x7f2f1c0b4210 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00ba00 con 0x7f2f1c0b4210 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.494+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1c1447e0 con 0x7f2f1c0b4210 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.495+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1c144da0 con 0x7f2f1c0b4210 2024-04-17T08:47:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.496+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1c068a10 con 0x7f2f1c0b4210 2024-04-17T08:47:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.497+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f2f0c00bba0 con 0x7f2f1c0b4210 2024-04-17T08:47:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.498+0000 7f2f13e00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 0x7f2ef806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.498+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f2f0c08c240 con 0x7f2f1c0b4210 2024-04-17T08:47:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.498+0000 7f2f12a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 0x7f2ef806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.499+0000 7f2f12a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 0x7f2ef806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f04035000 tx=0x7f2f04008030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.502+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2f0c05a390 con 0x7f2f1c0b4210 2024-04-17T08:47:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.747+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f1c0cf750 con 0x7f2f1c0b4210 2024-04-17T08:47:37.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.748+0000 7f2f13e00640 1 -- 172.21.15.67:0/1301275554 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+3872 (secure 0 0 0) 0x7f2f0c059c90 con 0x7f2f1c0b4210 2024-04-17T08:47:37.748 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:37.748 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1968875608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1968875608},{"type":"v1","addr":"172.21.15.67:6835","nonce":1968875608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":7948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/503387587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":503387587},{"type":"v1","addr":"172.21.15.187:6836","nonce":503387587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":8269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/2292452703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2292452703},{"type":"v1","addr":"172.21.15.187:6837","nonce":2292452703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":9978,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2479868856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2479868856},{"type":"v1","addr":"172.21.15.67:6834","nonce":2479868856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[]} 2024-04-17T08:47:37.748 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 96 2024-04-17T08:47:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 msgr2=0x7f2ef806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 0x7f2ef806ef00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f04035000 tx=0x7f2f04008030 comp rx=0 tx=0).stop 2024-04-17T08:47:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 msgr2=0x7f2f1c1e5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00d620 tx=0x7f2f0c00dae0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 shutdown_connections 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f2ef806ca40 0x7f2ef806ef00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f1c15c5a0 0x7f2f1c1e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f1c0b4690 0x7f2f1c143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 --2- 172.21.15.67:0/1301275554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f1c0b4210 0x7f2f1c1e5930 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 >> 172.21.15.67:0/1301275554 conn(0x7f2f1c067d10 msgr2=0x7f2f1c15b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 shutdown_connections 2024-04-17T08:47:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:37.751+0000 7f2f20a00640 1 -- 172.21.15.67:0/1301275554 wait complete. 2024-04-17T08:47:37.770 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:47:37.770 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:47:37.770 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:47:37.770 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:47:37.771 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:47:37.771 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:47:37.771 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:47:37.771 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:47:37.771 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:47:37.771 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:47:37.771 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:47:37.771 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:47:37.772 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:47:37.772 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:47:37.772 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:47:37.772 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:47:37.790 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:47:37.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:47:37.794 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:47:37.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:47:37.798 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:47:37.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:47:37.799 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:47:37.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:47:37.841 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:47:37.842 DEBUG:teuthology.parallel:result is None 2024-04-17T08:47:37.845 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:47:37.845 DEBUG:teuthology.parallel:result is None 2024-04-17T08:47:37.848 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:47:37.848 DEBUG:teuthology.parallel:result is None 2024-04-17T08:47:37.850 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:47:37.850 DEBUG:teuthology.parallel:result is None 2024-04-17T08:47:37.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:47:38.001 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:47:38.003 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:47:38.009 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:47:38.010 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:47:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.040+0000 7f6389400640 1 Processor -- start 2024-04-17T08:47:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.040+0000 7f6389400640 1 -- start start 2024-04-17T08:47:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6389400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 0x7f63840d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6389400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63840d17a0 0x7f63840cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6389400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6384163860 con 0x7f63840cfa10 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6389400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f63840d03b0 con 0x7f63840d17a0 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6389400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63840d1d40 con 0x7f638415eca0 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6379e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63840cfa10 0x7f63840d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43968/0 (socket says 172.21.15.67:43968) 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6379e00640 1 -- 172.21.15.67:0/2694418859 learned_addr learned my addr 172.21.15.67:0/2694418859 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.041+0000 7f6382800640 1 --2- 172.21.15.67:0/2694418859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 0x7f63840d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6383200640 1 --2- 172.21.15.67:0/2694418859 >> v2:172.21.15.187:3300/0 conn(0x7f63840d17a0 0x7f63840cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6381e00640 1 -- 172.21.15.67:0/2694418859 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 46252207 0 0) 0x7f6384163860 con 0x7f63840cfa10 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6381e00640 1 -- 172.21.15.67:0/2694418859 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6368003520 con 0x7f63840cfa10 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6382800640 1 -- 172.21.15.67:0/2694418859 >> v2:172.21.15.187:3300/0 conn(0x7f63840d17a0 msgr2=0x7f63840cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6382800640 1 --2- 172.21.15.67:0/2694418859 >> v2:172.21.15.187:3300/0 conn(0x7f63840d17a0 0x7f63840cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6382800640 1 -- 172.21.15.67:0/2694418859 >> v1:172.21.15.67:6789/0 conn(0x7f63840cfa10 legacy=0x7f63840d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.042+0000 7f6382800640 1 -- 172.21.15.67:0/2694418859 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63840d2c30 con 0x7f638415eca0 2024-04-17T08:47:38.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.043+0000 7f6382800640 1 --2- 172.21.15.67:0/2694418859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 0x7f63840d39f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f63740097a0 tx=0x7f637402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f56519cbab95d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6381e00640 1 -- 172.21.15.67:0/2694418859 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374035020 con 0x7f638415eca0 2024-04-17T08:47:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6381e00640 1 -- 172.21.15.67:0/2694418859 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6374031810 con 0x7f638415eca0 2024-04-17T08:47:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6381e00640 1 -- 172.21.15.67:0/2694418859 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374031be0 con 0x7f638415eca0 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6389400640 1 -- 172.21.15.67:0/2694418859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 msgr2=0x7f63840d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6389400640 1 --2- 172.21.15.67:0/2694418859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 0x7f63840d39f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f63740097a0 tx=0x7f637402b8b0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6389400640 1 -- 172.21.15.67:0/2694418859 shutdown_connections 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.044+0000 7f6389400640 1 --2- 172.21.15.67:0/2694418859 >> v2:172.21.15.187:3300/0 conn(0x7f63840d17a0 0x7f63840cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.045+0000 7f6389400640 1 --2- 172.21.15.67:0/2694418859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f638415eca0 0x7f63840d39f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.045+0000 7f6389400640 1 -- 172.21.15.67:0/2694418859 >> 172.21.15.67:0/2694418859 conn(0x7f6384067600 msgr2=0x7f63840679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.045+0000 7f6389400640 1 -- 172.21.15.67:0/2694418859 shutdown_connections 2024-04-17T08:47:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.045+0000 7f6389400640 1 -- 172.21.15.67:0/2694418859 wait complete. 2024-04-17T08:47:38.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.046+0000 7f6389400640 1 Processor -- start 2024-04-17T08:47:38.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.046+0000 7f6389400640 1 -- start start 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63840d17a0 0x7f63840ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638415eca0 0x7f63840caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f63840d34d0 con 0x7f63840cfa10 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f63840d31f0 con 0x7f638415eca0 2024-04-17T08:47:38.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.047+0000 7f6389400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f638415f080 con 0x7f63840d17a0 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52562/0 (socket says 172.21.15.67:52562) 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 -- 172.21.15.67:0/3279240994 learned_addr learned my addr 172.21.15.67:0/3279240994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6383200640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638415eca0 0x7f63840caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6379e00640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63840d17a0 0x7f63840ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 -- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63840d17a0 msgr2=0x7f63840ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63840d17a0 0x7f63840ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 -- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638415eca0 msgr2=0x7f63840caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638415eca0 0x7f63840caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.048+0000 7f6382800640 1 -- 172.21.15.67:0/3279240994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638414d670 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f6382800640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6374034040 tx=0x7f63740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374040070 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6374039c70 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374035040 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638414ccb0 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.049+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638414d270 con 0x7f63840cfa10 2024-04-17T08:47:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.050+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6384069120 con 0x7f63840cfa10 2024-04-17T08:47:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.051+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f6374038020 con 0x7f63840cfa10 2024-04-17T08:47:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.052+0000 7f637b400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 0x7f636006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.052+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6003+0+0 (secure 0 0 0) 0x7f63740ad7d0 con 0x7f63840cfa10 2024-04-17T08:47:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.053+0000 7f6379e00640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 0x7f636006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:38.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.054+0000 7f6379e00640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 0x7f636006eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6384153af0 tx=0x7f636c009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:38.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.057+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6384069120 con 0x7f63840cfa10 2024-04-17T08:47:38.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.270+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f638414caa0 con 0x7f63840cfa10 2024-04-17T08:47:38.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.761+0000 7f637b400640 1 -- 172.21.15.67:0/3279240994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v206) ==== 97+0+0 (secure 0 0 0) 0x7f637407b2d0 con 0x7f63840cfa10 2024-04-17T08:47:38.761 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.765+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 msgr2=0x7f636006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.765+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 0x7f636006eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6384153af0 tx=0x7f636c009040 comp rx=0 tx=0).stop 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.765+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 msgr2=0x7f63840b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.765+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6374034040 tx=0x7f63740046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 shutdown_connections 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f636006ca10 0x7f636006eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f638415eca0 0x7f63840caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63840d17a0 0x7f63840ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 --2- 172.21.15.67:0/3279240994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63840cfa10 0x7f63840b5ff0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 >> 172.21.15.67:0/3279240994 conn(0x7f6384067600 msgr2=0x7f63840ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 shutdown_connections 2024-04-17T08:47:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:38.766+0000 7f6389400640 1 -- 172.21.15.67:0/3279240994 wait complete. 2024-04-17T08:47:38.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:47:39.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.009+0000 7f7a3dc00640 1 Processor -- start 2024-04-17T08:47:39.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.009+0000 7f7a3dc00640 1 -- start start 2024-04-17T08:47:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.010+0000 7f7a3dc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7a38072030 0x7f7a380b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a38072410 0x7f7a380b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a381629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7a380b4a90 con 0x7f7a38072410 2024-04-17T08:47:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3dc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7a380ce1c0 con 0x7f7a38072030 2024-04-17T08:47:39.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a38162ef0 con 0x7f7a38162500 2024-04-17T08:47:39.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a2f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a38072410 0x7f7a380b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a2f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a38072410 0x7f7a380b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52578/0 (socket says 172.21.15.67:52578) 2024-04-17T08:47:39.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a2f400640 1 -- 172.21.15.67:0/1691750767 learned_addr learned my addr 172.21.15.67:0/1691750767 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.011+0000 7f7a3d200640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a381629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a37e00640 1 --2- 172.21.15.67:0/1691750767 >> v2:172.21.15.187:3300/0 conn(0x7f7a38072030 0x7f7a380b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a2f400640 1 -- 172.21.15.67:0/1691750767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 msgr2=0x7f7a381629b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a2f400640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a381629b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a2f400640 1 -- 172.21.15.67:0/1691750767 >> v2:172.21.15.187:3300/0 conn(0x7f7a38072030 msgr2=0x7f7a380b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a2f400640 1 --2- 172.21.15.67:0/1691750767 >> v2:172.21.15.187:3300/0 conn(0x7f7a38072030 0x7f7a380b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.012+0000 7f7a2f400640 1 -- 172.21.15.67:0/1691750767 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a380b3fa0 con 0x7f7a38072410 2024-04-17T08:47:39.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.014+0000 7f7a2f400640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380b3960 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7a3806fed0 tx=0x7f7a2002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a3bf5f6e43e759 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.014+0000 7f7a37400640 1 -- 172.21.15.67:0/1691750767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20035020 con 0x7f7a38072410 2024-04-17T08:47:39.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.014+0000 7f7a37400640 1 -- 172.21.15.67:0/1691750767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2002d040 con 0x7f7a38072410 2024-04-17T08:47:39.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.014+0000 7f7a37400640 1 -- 172.21.15.67:0/1691750767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20031a80 con 0x7f7a38072410 2024-04-17T08:47:39.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 -- 172.21.15.67:0/1691750767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 msgr2=0x7f7a380b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380b3960 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f7a3806fed0 tx=0x7f7a2002b900 comp rx=0 tx=0).stop 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 -- 172.21.15.67:0/1691750767 shutdown_connections 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a381629b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/1691750767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380b3960 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/1691750767 >> v2:172.21.15.187:3300/0 conn(0x7f7a38072030 0x7f7a380b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 -- 172.21.15.67:0/1691750767 >> 172.21.15.67:0/1691750767 conn(0x7f7a380b5f70 msgr2=0x7f7a3806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 -- 172.21.15.67:0/1691750767 shutdown_connections 2024-04-17T08:47:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.015+0000 7f7a3dc00640 1 -- 172.21.15.67:0/1691750767 wait complete. 2024-04-17T08:47:39.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.017+0000 7f7a3dc00640 1 Processor -- start 2024-04-17T08:47:39.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.017+0000 7f7a3dc00640 1 -- start start 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a38072030 0x7f7a3814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a3814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380b4280 con 0x7f7a38072410 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a380ce710 con 0x7f7a38072030 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3806dd80 con 0x7f7a38162500 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52592/0 (socket says 172.21.15.67:52592) 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a2f400640 1 -- 172.21.15.67:0/3997581492 learned_addr learned my addr 172.21.15.67:0/3997581492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a37e00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a38072030 0x7f7a3814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.018+0000 7f7a3d200640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a3814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 -- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 msgr2=0x7f7a3814f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a3814f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 -- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a38072030 msgr2=0x7f7a3814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a38072030 0x7f7a3814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 -- 172.21.15.67:0/3997581492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a380a9f40 con 0x7f7a38072410 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a3d200640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a3814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a2f400640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7a2002b940 tx=0x7f7a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20040070 con 0x7f7a38072410 2024-04-17T08:47:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a20039c10 con 0x7f7a38072410 2024-04-17T08:47:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20035040 con 0x7f7a38072410 2024-04-17T08:47:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.019+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a380aa220 con 0x7f7a38072410 2024-04-17T08:47:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.020+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3815b370 con 0x7f7a38072410 2024-04-17T08:47:39.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.021+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a38068a10 con 0x7f7a38072410 2024-04-17T08:47:39.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.022+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f7a20038020 con 0x7f7a38072410 2024-04-17T08:47:39.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.022+0000 7f7a34c00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 0x7f7a1406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.023+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 4885+0+0 (secure 0 0 0) 0x7f7a200ad620 con 0x7f7a38072410 2024-04-17T08:47:39.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.023+0000 7f7a37e00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 0x7f7a1406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.025+0000 7f7a37e00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 0x7f7a1406eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.027+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7a2007bc60 con 0x7f7a38072410 2024-04-17T08:47:39.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.305+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7a380008d0 con 0x7f7a38072410 2024-04-17T08:47:39.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.306+0000 7f7a34c00640 1 -- 172.21.15.67:0/3997581492 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) ==== 80+0+5304 (secure 0 0 0) 0x7f7a2007b560 con 0x7f7a38072410 2024-04-17T08:47:39.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:39.305 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:47:39.305 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.309 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.312 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:47:39.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:47:39.314 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:47:39.315 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:47:39.316 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:39.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:47:39.318 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:47:39.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.308+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 msgr2=0x7f7a1406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.308+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 0x7f7a1406eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28008040 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.308+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 msgr2=0x7f7a380a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.308+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f7a2002b940 tx=0x7f7a200046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 shutdown_connections 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f7a1406ca10 0x7f7a1406eed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a38162500 0x7f7a3814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a38072410 0x7f7a380a94c0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 --2- 172.21.15.67:0/3997581492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a38072030 0x7f7a3814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 >> 172.21.15.67:0/3997581492 conn(0x7f7a380b5f70 msgr2=0x7f7a380cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 shutdown_connections 2024-04-17T08:47:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.309+0000 7f7a3dc00640 1 -- 172.21.15.67:0/3997581492 wait complete. 2024-04-17T08:47:39.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:39.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.535+0000 7f3aa2e00640 1 Processor -- start 2024-04-17T08:47:39.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.535+0000 7f3aa2e00640 1 -- start start 2024-04-17T08:47:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.537+0000 7f3aa2e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 0x7f3a9c06f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.537+0000 7f3aa2e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 0x7f3a9c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.537+0000 7f3aa2e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a9c0cd920 con 0x7f3a9c072330 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.537+0000 7f3aa2e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9c072790 con 0x7f3a9c072eb0 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.537+0000 7f3aa2e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c0cdc30 con 0x7f3a9c15ac80 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.538+0000 7f3aa1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 0x7f3a9c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.538+0000 7f3aa1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 0x7f3a9c06f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.538+0000 7f3aa2400640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f3a9c072330 0x7f3a9c06ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:32894/0 (socket says 172.21.15.67:32894) 2024-04-17T08:47:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.538+0000 7f3aa2400640 1 -- 172.21.15.67:0/2348064322 learned_addr learned my addr 172.21.15.67:0/2348064322 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3a9be00640 1 -- 172.21.15.67:0/2348064322 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263084293 0 0) 0x7f3a94001490 con 0x7f3a9c072330 2024-04-17T08:47:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3a9be00640 1 -- 172.21.15.67:0/2348064322 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a84005730 con 0x7f3a9c072330 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3aa1a00640 1 -- 172.21.15.67:0/2348064322 >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 msgr2=0x7f3a9c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3aa1a00640 1 --2- 172.21.15.67:0/2348064322 >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 0x7f3a9c071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3aa1a00640 1 -- 172.21.15.67:0/2348064322 >> v1:172.21.15.187:6789/0 conn(0x7f3a9c072330 legacy=0x7f3a9c06ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3aa1a00640 1 -- 172.21.15.67:0/2348064322 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c06f980 con 0x7f3a9c15ac80 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.539+0000 7f3aa1000640 1 --2- 172.21.15.67:0/2348064322 >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 0x7f3a9c071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:47:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.541+0000 7f3aa1a00640 1 --2- 172.21.15.67:0/2348064322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 0x7f3a9c06f010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000a940 tx=0x7f3a9002bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7ae5a0f50a80a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.541+0000 7f3a9be00640 1 -- 172.21.15.67:0/2348064322 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90039070 con 0x7f3a9c15ac80 2024-04-17T08:47:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.541+0000 7f3a9be00640 1 -- 172.21.15.67:0/2348064322 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9002c9a0 con 0x7f3a9c15ac80 2024-04-17T08:47:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.541+0000 7f3a9be00640 1 -- 172.21.15.67:0/2348064322 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000b040 con 0x7f3a9c15ac80 2024-04-17T08:47:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2348064322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 msgr2=0x7f3a9c06f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2348064322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 0x7f3a9c06f010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000a940 tx=0x7f3a9002bbf0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2348064322 shutdown_connections 2024-04-17T08:47:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2348064322 >> v2:172.21.15.187:3300/0 conn(0x7f3a9c072eb0 0x7f3a9c071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2348064322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c15ac80 0x7f3a9c06f010 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2348064322 >> 172.21.15.67:0/2348064322 conn(0x7f3a9c067d10 msgr2=0x7f3a9c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2348064322 shutdown_connections 2024-04-17T08:47:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.542+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2348064322 wait complete. 2024-04-17T08:47:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.544+0000 7f3aa2e00640 1 Processor -- start 2024-04-17T08:47:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.545+0000 7f3aa2e00640 1 -- start start 2024-04-17T08:47:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a9c072330 0x7f3a9c155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c072eb0 0x7f3a9c141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c15b060 con 0x7f3a9c15ac80 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c06ff40 con 0x7f3a9c072330 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c148500 con 0x7f3a9c072eb0 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52594/0 (socket says 172.21.15.67:52594) 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.546+0000 7f3aa2400640 1 -- 172.21.15.67:0/2727808881 learned_addr learned my addr 172.21.15.67:0/2727808881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa1a00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a9c072330 0x7f3a9c155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa1000640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c072eb0 0x7f3a9c141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 -- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c072eb0 msgr2=0x7f3a9c141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c072eb0 0x7f3a9c141530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 -- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a9c072330 msgr2=0x7f3a9c155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a9c072330 0x7f3a9c155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 -- 172.21.15.67:0/2727808881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c155bc0 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2400640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400cb50 tx=0x7f3a9400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94017070 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9400ad40 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94012620 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.547+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c155ea0 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.548+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c156460 con 0x7f3a9c15ac80 2024-04-17T08:47:39.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.549+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a9c068a10 con 0x7f3a9c15ac80 2024-04-17T08:47:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.550+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f3a9401a070 con 0x7f3a9c15ac80 2024-04-17T08:47:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.551+0000 7f3a99600640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 0x7f3a7c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.551+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 4885+0+0 (secure 0 0 0) 0x7f3a9408b850 con 0x7f3a9c15ac80 2024-04-17T08:47:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.551+0000 7f3aa1a00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 0x7f3a7c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.552+0000 7f3aa1a00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 0x7f3a7c06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a90036000 tx=0x7f3a9000c040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.555+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3a9c0cd920 con 0x7f3a9c15ac80 2024-04-17T08:47:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.767+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a9c15db50 con 0x7f3a9c15ac80 2024-04-17T08:47:39.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.768+0000 7f3a99600640 1 -- 172.21.15.67:0/2727808881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) ==== 74+0+12897 (secure 0 0 0) 0x7f3a94059e90 con 0x7f3a9c15ac80 2024-04-17T08:47:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:39.768 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":208,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:38.779661+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 msgr2=0x7f3a7c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 0x7f3a7c06eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a90036000 tx=0x7f3a9000c040 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 msgr2=0x7f3a9c141a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400cb50 tx=0x7f3a9400a430 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 shutdown_connections 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f3a7c06ca10 0x7f3a7c06eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a9c15ac80 0x7f3a9c141a70 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a9c072eb0 0x7f3a9c141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 --2- 172.21.15.67:0/2727808881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a9c072330 0x7f3a9c155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 >> 172.21.15.67:0/2727808881 conn(0x7f3a9c067d10 msgr2=0x7f3a9c15e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 shutdown_connections 2024-04-17T08:47:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:39.772+0000 7f3aa2e00640 1 -- 172.21.15.67:0/2727808881 wait complete. 2024-04-17T08:47:39.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:47:39.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:47:40.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.013+0000 7f8256000640 1 Processor -- start 2024-04-17T08:47:40.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.014+0000 7f8256000640 1 -- start start 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.015+0000 7f8256000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f82500b3b50 0x7f82500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.015+0000 7f8256000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500cfcd0 0x7f82500d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.015+0000 7f8256000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82500d4f70 con 0x7f825015ec20 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.015+0000 7f8256000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f82500d5250 con 0x7f82500b3b50 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.015+0000 7f8256000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82500cf4d0 con 0x7f82500cfcd0 2024-04-17T08:47:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.016+0000 7f8255600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500cfcd0 0x7f82500d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.016+0000 7f8254c00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f825015ec20 0x7f825006ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:32896/0 (socket says 172.21.15.67:32896) 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.016+0000 7f8254c00640 1 -- 172.21.15.67:0/3002244078 learned_addr learned my addr 172.21.15.67:0/3002244078 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.016+0000 7f824fe00640 1 --2- 172.21.15.67:0/3002244078 >> v2:172.21.15.187:3300/0 conn(0x7f82500b3b50 0x7f82500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.017+0000 7f824fe00640 1 -- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500cfcd0 msgr2=0x7f82500d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.017+0000 7f824fe00640 1 --2- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500cfcd0 0x7f82500d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.017+0000 7f824fe00640 1 -- 172.21.15.67:0/3002244078 >> v1:172.21.15.187:6789/0 conn(0x7f825015ec20 legacy=0x7f825006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.017+0000 7f824fe00640 1 -- 172.21.15.67:0/3002244078 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82500cf7b0 con 0x7f82500b3b50 2024-04-17T08:47:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.019+0000 7f824fe00640 1 --2- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82500b3b50 0x7f82500d22b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f823c009790 tx=0x7f823c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4455855383f4d9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.019+0000 7f824f400640 1 -- 172.21.15.67:0/3002244078 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c039070 con 0x7f82500b3b50 2024-04-17T08:47:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.019+0000 7f824f400640 1 -- 172.21.15.67:0/3002244078 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823c0316f0 con 0x7f82500b3b50 2024-04-17T08:47:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.019+0000 7f824f400640 1 -- 172.21.15.67:0/3002244078 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c031ac0 con 0x7f82500b3b50 2024-04-17T08:47:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 -- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82500b3b50 msgr2=0x7f82500d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 --2- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82500b3b50 0x7f82500d22b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f823c009790 tx=0x7f823c02b790 comp rx=0 tx=0).stop 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 -- 172.21.15.67:0/3002244078 shutdown_connections 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 --2- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500cfcd0 0x7f82500d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 --2- 172.21.15.67:0/3002244078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82500b3b50 0x7f82500d22b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 -- 172.21.15.67:0/3002244078 >> 172.21.15.67:0/3002244078 conn(0x7f825008bff0 msgr2=0x7f82500d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 -- 172.21.15.67:0/3002244078 shutdown_connections 2024-04-17T08:47:40.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.020+0000 7f8256000640 1 -- 172.21.15.67:0/3002244078 wait complete. 2024-04-17T08:47:40.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.023+0000 7f8256000640 1 Processor -- start 2024-04-17T08:47:40.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.023+0000 7f8256000640 1 -- start start 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500b3b50 0x7f82501f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f825015ec20 0x7f825014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f825006d2d0 con 0x7f82500cfcd0 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f825006d150 con 0x7f825015ec20 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.024+0000 7f8256000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82500d10d0 con 0x7f82500b3b50 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52598/0 (socket says 172.21.15.67:52598) 2024-04-17T08:47:40.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 -- 172.21.15.67:0/1237129283 learned_addr learned my addr 172.21.15.67:0/1237129283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:40.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f8254c00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500b3b50 0x7f82501f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f8255600640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f825015ec20 0x7f825014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 -- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500b3b50 msgr2=0x7f82501f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500b3b50 0x7f82501f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 -- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f825015ec20 msgr2=0x7f825014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f825015ec20 0x7f825014faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.025+0000 7f824fe00640 1 -- 172.21.15.67:0/1237129283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8250160300 con 0x7f82500cfcd0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.026+0000 7f824fe00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f823c000c00 tx=0x7f823c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.026+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c043070 con 0x7f82500cfcd0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.026+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823c034480 con 0x7f82500cfcd0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.026+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c039040 con 0x7f82500cfcd0 2024-04-17T08:47:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.026+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82500cea60 con 0x7f82500cfcd0 2024-04-17T08:47:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.027+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82501f8340 con 0x7f82500cfcd0 2024-04-17T08:47:40.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.028+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8250068a10 con 0x7f82500cfcd0 2024-04-17T08:47:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.029+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f823c046070 con 0x7f82500cfcd0 2024-04-17T08:47:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.030+0000 7f824cc00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 0x7f823006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.030+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 4885+0+0 (secure 0 0 0) 0x7f823c0ad5c0 con 0x7f82500cfcd0 2024-04-17T08:47:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.031+0000 7f8254c00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 0x7f823006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.032+0000 7f8254c00640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 0x7f823006eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8244007fd0 tx=0x7f8244007e80 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.035+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f823c0316c0 con 0x7f82500cfcd0 2024-04-17T08:47:40.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.243+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f82500d55f0 con 0x7f82500cfcd0 2024-04-17T08:47:40.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.244+0000 7f824cc00640 1 -- 172.21.15.67:0/1237129283 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v208) ==== 93+0+10 (secure 0 0 0) 0x7f823c07b4b0 con 0x7f82500cfcd0 2024-04-17T08:47:40.244 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 msgr2=0x7f823006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 0x7f823006eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8244007fd0 tx=0x7f8244007e80 comp rx=0 tx=0).stop 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 msgr2=0x7f82500ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f823c000c00 tx=0x7f823c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 shutdown_connections 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f823006ca10 0x7f823006eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f825015ec20 0x7f825014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82500cfcd0 0x7f82500ce520 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 --2- 172.21.15.67:0/1237129283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82500b3b50 0x7f82501f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.247+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 >> 172.21.15.67:0/1237129283 conn(0x7f825008bff0 msgr2=0x7f82500d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.248+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 shutdown_connections 2024-04-17T08:47:40.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.248+0000 7f8256000640 1 -- 172.21.15.67:0/1237129283 wait complete. 2024-04-17T08:47:40.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:47:40.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.483+0000 7f40d3a00640 1 Processor -- start 2024-04-17T08:47:40.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.483+0000 7f40d3a00640 1 -- start start 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d3a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f40cc0cd480 0x7f40cc06e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d3a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc0ceb00 con 0x7f40cc06f670 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d3a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc0cede0 con 0x7f40cc0cd020 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d3a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc0cf0c0 con 0x7f40cc0cd480 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d1000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f40cc06f670 0x7f40cc06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43972/0 (socket says 172.21.15.67:43972) 2024-04-17T08:47:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d1000640 1 -- 172.21.15.67:0/328218464 learned_addr learned my addr 172.21.15.67:0/328218464 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819296322 0 0) 0x7f40cc0ceb00 con 0x7f40cc06f670 2024-04-17T08:47:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b4003520 con 0x7f40cc06f670 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.484+0000 7f40d1a00640 1 --2- 172.21.15.67:0/328218464 >> v2:172.21.15.187:3300/0 conn(0x7f40cc0cd480 0x7f40cc06e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3515236459 0 0) 0x7f40b4003520 con 0x7f40cc06f670 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 >> v2:172.21.15.187:3300/0 conn(0x7f40cc0cd480 msgr2=0x7f40cc06e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 --2- 172.21.15.67:0/328218464 >> v2:172.21.15.187:3300/0 conn(0x7f40cc0cd480 0x7f40cc06e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 >> v1:172.21.15.187:6789/0 conn(0x7f40cc0cd020 legacy=0x7f40cc06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0cf3a0 con 0x7f40cc06f670 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f40c0001670 con 0x7f40cc06f670 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 >> v1:172.21.15.67:6789/0 conn(0x7f40cc06f670 legacy=0x7f40cc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40b40051c0 0x7f40b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0cf0c0 con 0x7f40b4004800 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40b400c060 con 0x7f40b40039b0 2024-04-17T08:47:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40b400c260 con 0x7f40b40051c0 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.485+0000 7f40cbe00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40d1000640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40d1a00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40b40051c0 0x7f40b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40cbe00640 1 -- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40b40051c0 msgr2=0x7f40b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40cbe00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40b40051c0 0x7f40b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40cbe00640 1 -- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b40039b0 msgr2=0x7f40b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40cbe00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40b40039b0 0x7f40b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40cbe00640 1 -- 172.21.15.67:0/328218464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4003520 con 0x7f40b4004800 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40d1a00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40b40051c0 0x7f40b400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.486+0000 7f40d1a00640 1 -- 172.21.15.67:0/328218464 reap_dead start 2024-04-17T08:47:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.487+0000 7f40cbe00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f40b8001750 tx=0x7f40b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71327f128fa0ef34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.487+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8001370 con 0x7f40b4004800 2024-04-17T08:47:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.487+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b80052d0 con 0x7f40b4004800 2024-04-17T08:47:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.487+0000 7f40cb400640 1 -- 172.21.15.67:0/328218464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8003830 con 0x7f40b4004800 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 -- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 msgr2=0x7f40b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f40b8001750 tx=0x7f40b4005830 comp rx=0 tx=0).stop 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 -- 172.21.15.67:0/328218464 shutdown_connections 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 --2- 172.21.15.67:0/328218464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40b4004c80 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 -- 172.21.15.67:0/328218464 >> 172.21.15.67:0/328218464 conn(0x7f40cc06bf50 msgr2=0x7f40cc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 -- 172.21.15.67:0/328218464 shutdown_connections 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.488+0000 7f40d3a00640 1 -- 172.21.15.67:0/328218464 wait complete. 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.490+0000 7f40d3a00640 1 Processor -- start 2024-04-17T08:47:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.490+0000 7f40d3a00640 1 -- start start 2024-04-17T08:47:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 0x7f40cc15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 0x7f40cc14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0ce8a0 con 0x7f40b4004800 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0ce720 con 0x7f40cc0c8e80 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc15cb00 con 0x7f40cc14ece0 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.491+0000 7f40d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52618/0 (socket says 172.21.15.67:52618) 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 -- 172.21.15.67:0/994397250 learned_addr learned my addr 172.21.15.67:0/994397250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40cbe00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 0x7f40cc15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 0x7f40cc14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 -- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 msgr2=0x7f40cc14f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 0x7f40cc14f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 -- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 msgr2=0x7f40cc15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 0x7f40cc15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 -- 172.21.15.67:0/994397250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc14f6a0 con 0x7f40b4004800 2024-04-17T08:47:40.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40cbe00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 0x7f40cc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:40.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 0x7f40cc14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:40.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.492+0000 7f40d1000640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f40c000e7e0 tx=0x7f40c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.493+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c000c6c0 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.493+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c0018070 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.493+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c000cac0 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.493+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc0c66e0 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.493+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc0c6ca0 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.494+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40cc068b90 con 0x7f40b4004800 2024-04-17T08:47:40.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.495+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f40c000cc60 con 0x7f40b4004800 2024-04-17T08:47:40.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.496+0000 7f40c8c00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 0x7f40ac06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.496+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 4885+0+0 (secure 0 0 0) 0x7f40c008b7c0 con 0x7f40b4004800 2024-04-17T08:47:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.497+0000 7f40cbe00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 0x7f40ac06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.498+0000 7f40cbe00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 0x7f40ac06ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40b8033000 tx=0x7f40b8009030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.501+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f40c0059e00 con 0x7f40b4004800 2024-04-17T08:47:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.714+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f40cc072920 con 0x7f40b4004800 2024-04-17T08:47:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.715+0000 7f40c8c00640 1 -- 172.21.15.67:0/994397250 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v208) ==== 84+0+65 (secure 0 0 0) 0x7f40c0059700 con 0x7f40b4004800 2024-04-17T08:47:40.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:40.715 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:47:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 msgr2=0x7f40ac06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 0x7f40ac06ef00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40b8033000 tx=0x7f40b8009030 comp rx=0 tx=0).stop 2024-04-17T08:47:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 msgr2=0x7f40cc1579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f40c000e7e0 tx=0x7f40c000eca0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 shutdown_connections 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f40ac06ca40 0x7f40ac06ef00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40cc14ece0 0x7f40cc14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40cc0c8e80 0x7f40cc15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 --2- 172.21.15.67:0/994397250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40b4004800 0x7f40cc1579d0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 >> 172.21.15.67:0/994397250 conn(0x7f40cc06bf50 msgr2=0x7f40cc159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 shutdown_connections 2024-04-17T08:47:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.718+0000 7f40d3a00640 1 -- 172.21.15.67:0/994397250 wait complete. 2024-04-17T08:47:40.738 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:47:40.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:47:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.966+0000 7f1b8f200640 1 Processor -- start 2024-04-17T08:47:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.966+0000 7f1b8f200640 1 -- start start 2024-04-17T08:47:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.968+0000 7f1b8f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1b88148210 0x7f1b8806e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.968+0000 7f1b8f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b880b4260 con 0x7f1b880b3780 2024-04-17T08:47:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.968+0000 7f1b8f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b880b4540 con 0x7f1b88072690 2024-04-17T08:47:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.968+0000 7f1b8f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1b8806dea0 con 0x7f1b88148210 2024-04-17T08:47:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.969+0000 7f1b87400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b880b3780 0x7f1b88071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43976/0 (socket says 172.21.15.67:43976) 2024-04-17T08:47:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.969+0000 7f1b87400640 1 -- 172.21.15.67:0/3905734546 learned_addr learned my addr 172.21.15.67:0/3905734546 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.969+0000 7f1b87e00640 1 --2- 172.21.15.67:0/3905734546 >> v2:172.21.15.187:3300/0 conn(0x7f1b88148210 0x7f1b8806e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.969+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200292236 0 0) 0x7f1b880b4260 con 0x7f1b880b3780 2024-04-17T08:47:40.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.969+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b6c003520 con 0x7f1b880b3780 2024-04-17T08:47:40.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.970+0000 7f1b87e00640 1 -- 172.21.15.67:0/3905734546 >> v1:172.21.15.187:6789/0 conn(0x7f1b88072690 legacy=0x7f1b880b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.970+0000 7f1b87e00640 1 -- 172.21.15.67:0/3905734546 >> v1:172.21.15.67:6789/0 conn(0x7f1b880b3780 legacy=0x7f1b88071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.970+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3792446692 0 0) 0x7f1b6c003520 con 0x7f1b880b3780 2024-04-17T08:47:40.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.970+0000 7f1b87e00640 1 -- 172.21.15.67:0/3905734546 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8806e180 con 0x7f1b88148210 2024-04-17T08:47:40.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.971+0000 7f1b87e00640 1 --2- 172.21.15.67:0/3905734546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b88148210 0x7f1b8806e6a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1b780097a0 tx=0x7f1b7802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d91b9093dd31f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78037070 con 0x7f1b88148210 2024-04-17T08:47:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b78030510 con 0x7f1b88148210 2024-04-17T08:47:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b86a00640 1 -- 172.21.15.67:0/3905734546 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74001670 con 0x7f1b88148210 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 -- 172.21.15.67:0/3905734546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b88148210 msgr2=0x7f1b8806e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 --2- 172.21.15.67:0/3905734546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b88148210 0x7f1b8806e6a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1b780097a0 tx=0x7f1b7802b710 comp rx=0 tx=0).stop 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 -- 172.21.15.67:0/3905734546 shutdown_connections 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 --2- 172.21.15.67:0/3905734546 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b88148210 0x7f1b8806e6a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 -- 172.21.15.67:0/3905734546 >> 172.21.15.67:0/3905734546 conn(0x7f1b8806b8c0 msgr2=0x7f1b8806bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.972+0000 7f1b8f200640 1 -- 172.21.15.67:0/3905734546 shutdown_connections 2024-04-17T08:47:40.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.973+0000 7f1b8f200640 1 -- 172.21.15.67:0/3905734546 wait complete. 2024-04-17T08:47:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.975+0000 7f1b8f200640 1 Processor -- start 2024-04-17T08:47:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.975+0000 7f1b8f200640 1 -- start start 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 0x7f1b88145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b88148210 0x7f1b8813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b88148770 con 0x7f1b88072690 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b881485f0 con 0x7f1b880b3780 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b8f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b8806ebe0 con 0x7f1b88148210 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.976+0000 7f1b87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52630/0 (socket says 172.21.15.67:52630) 2024-04-17T08:47:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 -- 172.21.15.67:0/4281565388 learned_addr learned my addr 172.21.15.67:0/4281565388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:40.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87400640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 0x7f1b88145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b8d200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b88148210 0x7f1b8813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 -- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b88148210 msgr2=0x7f1b8813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b88148210 0x7f1b8813be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 -- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 msgr2=0x7f1b88145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 0x7f1b88145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87e00640 1 -- 172.21.15.67:0/4281565388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8813f180 con 0x7f1b88072690 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.977+0000 7f1b87400640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 0x7f1b88145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b87e00640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1b78000c00 tx=0x7f1b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78042070 con 0x7f1b88072690 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7803e870 con 0x7f1b88072690 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78037040 con 0x7f1b88072690 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b88144d30 con 0x7f1b88072690 2024-04-17T08:47:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.978+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b881452f0 con 0x7f1b88072690 2024-04-17T08:47:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.979+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b88073a50 con 0x7f1b88072690 2024-04-17T08:47:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.981+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f1b78045070 con 0x7f1b88072690 2024-04-17T08:47:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.982+0000 7f1b7fe00640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 0x7f1b6806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.982+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 4885+0+0 (secure 0 0 0) 0x7f1b780ad440 con 0x7f1b88072690 2024-04-17T08:47:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.983+0000 7f1b87400640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 0x7f1b6806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.984+0000 7f1b87400640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 0x7f1b6806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b740011e0 tx=0x7f1b74009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:40.987+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1b7804c070 con 0x7f1b88072690 2024-04-17T08:47:41.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.194+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1b88068d60 con 0x7f1b88072690 2024-04-17T08:47:41.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.774+0000 7f1b7fe00640 1 -- 172.21.15.67:0/4281565388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v209) ==== 138+0+0 (secure 0 0 0) 0x7f1b7807ba10 con 0x7f1b88072690 2024-04-17T08:47:41.773 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.777+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 msgr2=0x7f1b6806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 0x7f1b6806ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b740011e0 tx=0x7f1b74009040 comp rx=0 tx=0).stop 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 msgr2=0x7f1b8813f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f1b78000c00 tx=0x7f1b780046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 shutdown_connections 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1b6806ca40 0x7f1b6806ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b88148210 0x7f1b8813be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:41.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b880b3780 0x7f1b88145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:41.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 --2- 172.21.15.67:0/4281565388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b88072690 0x7f1b8813f5c0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:41.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 >> 172.21.15.67:0/4281565388 conn(0x7f1b8806b8c0 msgr2=0x7f1b88142c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:41.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 shutdown_connections 2024-04-17T08:47:41.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:41.778+0000 7f1b8f200640 1 -- 172.21.15.67:0/4281565388 wait complete. 2024-04-17T08:47:41.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:47:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.021+0000 7fa1c4a00640 1 Processor -- start 2024-04-17T08:47:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.021+0000 7fa1c4a00640 1 -- start start 2024-04-17T08:47:42.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.023+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.023+0000 7fa1c4a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 0x7fa1c006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.023+0000 7fa1c4a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c015f0a0 con 0x7fa1c0073650 2024-04-17T08:47:42.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.023+0000 7fa1c4a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c015f380 con 0x7fa1c0070b10 2024-04-17T08:47:42.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.023+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c015f660 con 0x7fa1c015eb90 2024-04-17T08:47:42.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.024+0000 7fa1be800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.024+0000 7fa1be800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:35778/0 (socket says 172.21.15.67:35778) 2024-04-17T08:47:42.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.024+0000 7fa1be800640 1 -- 172.21.15.67:0/2966482193 learned_addr learned my addr 172.21.15.67:0/2966482193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:42.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.024+0000 7fa1bde00640 1 --2- 172.21.15.67:0/2966482193 >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 0x7fa1c006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.025+0000 7fa1be800640 1 -- 172.21.15.67:0/2966482193 >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 msgr2=0x7fa1c006c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:42.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.025+0000 7fa1be800640 1 --2- 172.21.15.67:0/2966482193 >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 0x7fa1c006c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.025+0000 7fa1be800640 1 -- 172.21.15.67:0/2966482193 >> v1:172.21.15.187:6789/0 conn(0x7fa1c0073650 legacy=0x7fa1c0162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.025+0000 7fa1be800640 1 -- 172.21.15.67:0/2966482193 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c015f940 con 0x7fa1c015eb90 2024-04-17T08:47:42.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.025+0000 7fa1bde00640 1 --2- 172.21.15.67:0/2966482193 >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 0x7fa1c006c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:47:42.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.026+0000 7fa1be800640 1 --2- 172.21.15.67:0/2966482193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009870 tx=0x7fa1b002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbad78fe851b3e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:42.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.026+0000 7fa1bd400640 1 -- 172.21.15.67:0/2966482193 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00317c0 con 0x7fa1c015eb90 2024-04-17T08:47:42.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1bd400640 1 -- 172.21.15.67:0/2966482193 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b0037070 con 0x7fa1c015eb90 2024-04-17T08:47:42.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1bd400640 1 -- 172.21.15.67:0/2966482193 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0031b80 con 0x7fa1c015eb90 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 -- 172.21.15.67:0/2966482193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 msgr2=0x7fa1c0073110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/2966482193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009870 tx=0x7fa1b002b860 comp rx=0 tx=0).stop 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 -- 172.21.15.67:0/2966482193 shutdown_connections 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/2966482193 >> v2:172.21.15.187:3300/0 conn(0x7fa1c0070b10 0x7fa1c006c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/2966482193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c0073110 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 -- 172.21.15.67:0/2966482193 >> 172.21.15.67:0/2966482193 conn(0x7fa1c00725f0 msgr2=0x7fa1c0072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.027+0000 7fa1c4a00640 1 -- 172.21.15.67:0/2966482193 shutdown_connections 2024-04-17T08:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.028+0000 7fa1c4a00640 1 -- 172.21.15.67:0/2966482193 wait complete. 2024-04-17T08:47:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.029+0000 7fa1c4a00640 1 Processor -- start 2024-04-17T08:47:42.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.029+0000 7fa1c4a00640 1 -- start start 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.030+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1c0070b10 0x7fa1c0154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.030+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.030+0000 7fa1c4a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c01f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c006ce10 con 0x7fa1c0073650 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c00cfd30 con 0x7fa1c0070b10 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1c4a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c006cf90 con 0x7fa1c015eb90 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52634/0 (socket says 172.21.15.67:52634) 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 -- 172.21.15.67:0/1490271650 learned_addr learned my addr 172.21.15.67:0/1490271650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:42.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1be800640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1c0070b10 0x7fa1c0154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bf200640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c01f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 -- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 msgr2=0x7fa1c01f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c01f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 -- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1c0070b10 msgr2=0x7fa1c0154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1c0070b10 0x7fa1c0154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.031+0000 7fa1bde00640 1 -- 172.21.15.67:0/1490271650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c01f5820 con 0x7fa1c0073650 2024-04-17T08:47:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1bde00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ebe0 tx=0x7fa1ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac017070 con 0x7fa1c0073650 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ac00cc30 con 0x7fa1c0073650 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c01f5b00 con 0x7fa1c0073650 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac013470 con 0x7fa1c0073650 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.032+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c00b3c00 con 0x7fa1c0073650 2024-04-17T08:47:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.033+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c00b3a30 con 0x7fa1c0073650 2024-04-17T08:47:42.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.035+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fa1ac01a070 con 0x7fa1c0073650 2024-04-17T08:47:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.036+0000 7fa1b6a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 0x7fa1a006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.036+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 5255+0+0 (secure 0 0 0) 0x7fa1ac08beb0 con 0x7fa1c0073650 2024-04-17T08:47:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.036+0000 7fa1be800640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 0x7fa1a006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:42.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.038+0000 7fa1be800640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 0x7fa1a006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0036000 tx=0x7fa1b003d040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.042+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa1ac05a2f0 con 0x7fa1c0073650 2024-04-17T08:47:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.249+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa1c00755d0 con 0x7fa1c0073650 2024-04-17T08:47:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.781+0000 7fa1b6a00640 1 -- 172.21.15.67:0/1490271650 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v210) ==== 170+0+0 (secure 0 0 0) 0x7fa1ac059bf0 con 0x7fa1c0073650 2024-04-17T08:47:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:47:42.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.785+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 msgr2=0x7fa1a006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.785+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 0x7fa1a006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0036000 tx=0x7fa1b003d040 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.785+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 msgr2=0x7fa1c01f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.785+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ebe0 tx=0x7fa1ac00c520 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 shutdown_connections 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fa1a006ca10 0x7fa1a006eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c015eb90 0x7fa1c01f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1c0073650 0x7fa1c01f9160 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 --2- 172.21.15.67:0/1490271650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1c0070b10 0x7fa1c0154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.786+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 >> 172.21.15.67:0/1490271650 conn(0x7fa1c00725f0 msgr2=0x7fa1c006ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.787+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 shutdown_connections 2024-04-17T08:47:42.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:42.787+0000 7fa1c4a00640 1 -- 172.21.15.67:0/1490271650 wait complete. 2024-04-17T08:47:42.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:47:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.009+0000 7f925ea00640 1 Processor -- start 2024-04-17T08:47:43.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.010+0000 7f925ea00640 1 -- start start 2024-04-17T08:47:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.011+0000 7f925ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9258159020 0x7f925815ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.011+0000 7f925ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92580af400 0x7f9258162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.011+0000 7f925ea00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925815f780 con 0x7f925806b8c0 2024-04-17T08:47:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.011+0000 7f925ea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f925815fa60 con 0x7f9258159020 2024-04-17T08:47:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.011+0000 7f925ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92581624a0 con 0x7f92580af400 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9258159020 0x7f925815ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9258159020 0x7f925815ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52642/0 (socket says 172.21.15.67:52642) 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 -- 172.21.15.67:0/385303820 learned_addr learned my addr 172.21.15.67:0/385303820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925cc00640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92580af400 0x7f9258162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 -- 172.21.15.67:0/385303820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92580af400 msgr2=0x7f9258162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92580af400 0x7f9258162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 -- 172.21.15.67:0/385303820 >> v1:172.21.15.187:6789/0 conn(0x7f925806b8c0 legacy=0x7f92580adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.012+0000 7f925d600640 1 -- 172.21.15.67:0/385303820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9258162780 con 0x7f9258159020 2024-04-17T08:47:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.013+0000 7f925d600640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9258159020 0x7f925815ee30 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ac0712a15c3f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.014+0000 7f9257e00640 1 -- 172.21.15.67:0/385303820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c0317c0 con 0x7f9258159020 2024-04-17T08:47:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.014+0000 7f9257e00640 1 -- 172.21.15.67:0/385303820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c037070 con 0x7f9258159020 2024-04-17T08:47:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.014+0000 7f9257e00640 1 -- 172.21.15.67:0/385303820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c031b80 con 0x7f9258159020 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.014+0000 7f925ea00640 1 -- 172.21.15.67:0/385303820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9258159020 msgr2=0x7f925815ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.014+0000 7f925ea00640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9258159020 0x7f925815ee30 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c02b860 comp rx=0 tx=0).stop 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 -- 172.21.15.67:0/385303820 shutdown_connections 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92580af400 0x7f9258162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 --2- 172.21.15.67:0/385303820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9258159020 0x7f925815ee30 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 -- 172.21.15.67:0/385303820 >> 172.21.15.67:0/385303820 conn(0x7f92580b2190 msgr2=0x7f92580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 -- 172.21.15.67:0/385303820 shutdown_connections 2024-04-17T08:47:43.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.015+0000 7f925ea00640 1 -- 172.21.15.67:0/385303820 wait complete. 2024-04-17T08:47:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.017+0000 7f925ea00640 1 Processor -- start 2024-04-17T08:47:43.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.017+0000 7f925ea00640 1 -- start start 2024-04-17T08:47:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92580af400 0x7f92581616b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9258159020 0x7f9258148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f925815f4f0 con 0x7f925806b8c0 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f925815f370 con 0x7f92580af400 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925ea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92581631e0 con 0x7f9258159020 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52646/0 (socket says 172.21.15.67:52646) 2024-04-17T08:47:43.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.018+0000 7f925d600640 1 -- 172.21.15.67:0/2788446885 learned_addr learned my addr 172.21.15.67:0/2788446885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 -- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9258159020 msgr2=0x7f9258148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9258159020 0x7f9258148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 -- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92580af400 msgr2=0x7f92581616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92580af400 0x7f92581616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 -- 172.21.15.67:0/2788446885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925815a1b0 con 0x7f925806b8c0 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f925d600640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:43.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.019+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c032650 con 0x7f925806b8c0 2024-04-17T08:47:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.020+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c0327f0 con 0x7f925806b8c0 2024-04-17T08:47:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.020+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925815a490 con 0x7f925806b8c0 2024-04-17T08:47:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.020+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c032be0 con 0x7f925806b8c0 2024-04-17T08:47:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.020+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925815aa50 con 0x7f925806b8c0 2024-04-17T08:47:43.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.021+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258068a10 con 0x7f925806b8c0 2024-04-17T08:47:43.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.023+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f924c04c060 con 0x7f925806b8c0 2024-04-17T08:47:43.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.024+0000 7f9255600640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 0x7f923c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:43.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.024+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 5637+0+0 (secure 0 0 0) 0x7f924c0ad6c0 con 0x7f925806b8c0 2024-04-17T08:47:43.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.024+0000 7f925cc00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 0x7f923c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:43.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.025+0000 7f925cc00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 0x7f923c06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f92581547c0 tx=0x7f9248008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:43.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.029+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f924c040070 con 0x7f925806b8c0 2024-04-17T08:47:43.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.270+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f92580d0550 con 0x7f925806b8c0 2024-04-17T08:47:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.835+0000 7f9255600640 1 -- 172.21.15.67:0/2788446885 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v99) ==== 174+0+0 (secure 0 0 0) 0x7f924c07ba20 con 0x7f925806b8c0 2024-04-17T08:47:43.843 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 24 and data pool 25 2024-04-17T08:47:43.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.849+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 msgr2=0x7f923c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:43.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.849+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 0x7f923c06eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f92581547c0 tx=0x7f9248008040 comp rx=0 tx=0).stop 2024-04-17T08:47:43.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.849+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 msgr2=0x7f9258155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:43.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.849+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 shutdown_connections 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f923c06ca10 0x7f923c06eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9258159020 0x7f9258148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92580af400 0x7f92581616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 --2- 172.21.15.67:0/2788446885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f925806b8c0 0x7f9258155a20 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.851+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 >> 172.21.15.67:0/2788446885 conn(0x7f92580b2190 msgr2=0x7f92580b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.852+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 shutdown_connections 2024-04-17T08:47:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:43.852+0000 7f925ea00640 1 -- 172.21.15.67:0/2788446885 wait complete. 2024-04-17T08:47:43.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:44.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.066+0000 7f725ee00640 1 Processor -- start 2024-04-17T08:47:44.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.066+0000 7f725ee00640 1 -- start start 2024-04-17T08:47:44.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f725815ec60 0x7f72580b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725ee00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72580b6b30 0x7f72580cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725ee00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72580d2d30 con 0x7f72580cf910 2024-04-17T08:47:44.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d49c0 con 0x7f725815ec60 2024-04-17T08:47:44.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725ee00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d3660 con 0x7f72580b6b30 2024-04-17T08:47:44.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f725815ec60 0x7f72580b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f725815ec60 0x7f72580b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58160/0 (socket says 172.21.15.67:58160) 2024-04-17T08:47:44.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.068+0000 7f725da00640 1 -- 172.21.15.67:0/1067915586 learned_addr learned my addr 172.21.15.67:0/1067915586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.069+0000 7f725d000640 1 --2- 172.21.15.67:0/1067915586 >> v2:172.21.15.187:3300/0 conn(0x7f72580b6b30 0x7f72580cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.069+0000 7f725da00640 1 -- 172.21.15.67:0/1067915586 >> v2:172.21.15.187:3300/0 conn(0x7f72580b6b30 msgr2=0x7f72580cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.069+0000 7f725da00640 1 --2- 172.21.15.67:0/1067915586 >> v2:172.21.15.187:3300/0 conn(0x7f72580b6b30 0x7f72580cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.069+0000 7f725da00640 1 -- 172.21.15.67:0/1067915586 >> v1:172.21.15.187:6789/0 conn(0x7f72580cf910 legacy=0x7f725806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.069+0000 7f725da00640 1 -- 172.21.15.67:0/1067915586 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72580d3010 con 0x7f725815ec60 2024-04-17T08:47:44.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.070+0000 7f725da00640 1 --2- 172.21.15.67:0/1067915586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725815ec60 0x7f72580b65f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f72580ba590 tx=0x7f724802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce0bb6bf8df228d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.070+0000 7f7257e00640 1 -- 172.21.15.67:0/1067915586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248035020 con 0x7f725815ec60 2024-04-17T08:47:44.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.070+0000 7f7257e00640 1 -- 172.21.15.67:0/1067915586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7248031870 con 0x7f725815ec60 2024-04-17T08:47:44.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.070+0000 7f7257e00640 1 -- 172.21.15.67:0/1067915586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248031c40 con 0x7f725815ec60 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 -- 172.21.15.67:0/1067915586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725815ec60 msgr2=0x7f72580b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 --2- 172.21.15.67:0/1067915586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725815ec60 0x7f72580b65f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f72580ba590 tx=0x7f724802b910 comp rx=0 tx=0).stop 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 -- 172.21.15.67:0/1067915586 shutdown_connections 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 --2- 172.21.15.67:0/1067915586 >> v2:172.21.15.187:3300/0 conn(0x7f72580b6b30 0x7f72580cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 --2- 172.21.15.67:0/1067915586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725815ec60 0x7f72580b65f0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 -- 172.21.15.67:0/1067915586 >> 172.21.15.67:0/1067915586 conn(0x7f72580d15f0 msgr2=0x7f72580d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 -- 172.21.15.67:0/1067915586 shutdown_connections 2024-04-17T08:47:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.071+0000 7f725ee00640 1 -- 172.21.15.67:0/1067915586 wait complete. 2024-04-17T08:47:44.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.073+0000 7f725ee00640 1 Processor -- start 2024-04-17T08:47:44.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.073+0000 7f725ee00640 1 -- start start 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72580cf910 0x7f725815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f725815ec60 0x7f72580b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72580cfda0 con 0x7f72580b6b30 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f72580b6f70 con 0x7f725815ec60 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725ee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f725815f040 con 0x7f72580cf910 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58166/0 (socket says 172.21.15.67:58166) 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.074+0000 7f725da00640 1 -- 172.21.15.67:0/4134801685 learned_addr learned my addr 172.21.15.67:0/4134801685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 -- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72580cf910 msgr2=0x7f725815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72580cf910 0x7f725815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 -- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f725815ec60 msgr2=0x7f72580b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f725815ec60 0x7f72580b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 -- 172.21.15.67:0/4134801685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725815b9e0 con 0x7f72580b6b30 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f725da00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f724802b950 tx=0x7f72480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.075+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248040070 con 0x7f72580b6b30 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.076+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724803a420 con 0x7f72580b6b30 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.076+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725815bca0 con 0x7f72580b6b30 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.076+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248035040 con 0x7f72580b6b30 2024-04-17T08:47:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.076+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72581f80b0 con 0x7f72580b6b30 2024-04-17T08:47:44.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.077+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7258068a10 con 0x7f72580b6b30 2024-04-17T08:47:44.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.078+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f7248038020 con 0x7f72580b6b30 2024-04-17T08:47:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.079+0000 7f7255600640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 0x7f723806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.079+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5737+0+0 (secure 0 0 0) 0x7f72480add50 con 0x7f72580b6b30 2024-04-17T08:47:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.079+0000 7f725d000640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 0x7f723806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.081+0000 7f725d000640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 0x7f723806eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7244006fd0 tx=0x7f7244008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.084+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f724807bfb0 con 0x7f72580b6b30 2024-04-17T08:47:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.291+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72580b3a50 con 0x7f72580b6b30 2024-04-17T08:47:44.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.294+0000 7f7255600640 1 -- 172.21.15.67:0/4134801685 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) ==== 74+0+16688 (secure 0 0 0) 0x7f724807b8b0 con 0x7f72580b6b30 2024-04-17T08:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:44.293 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":211,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:43.271903+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 msgr2=0x7f723806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 0x7f723806eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7244006fd0 tx=0x7f7244008040 comp rx=0 tx=0).stop 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 msgr2=0x7f725814efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f724802b950 tx=0x7f72480046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 shutdown_connections 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f723806ca10 0x7f723806eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f725815ec60 0x7f72580b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72580cf910 0x7f725815b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 --2- 172.21.15.67:0/4134801685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72580b6b30 0x7f725814efb0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 >> 172.21.15.67:0/4134801685 conn(0x7f72580d15f0 msgr2=0x7f7258074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 shutdown_connections 2024-04-17T08:47:44.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.297+0000 7f725ee00640 1 -- 172.21.15.67:0/4134801685 wait complete. 2024-04-17T08:47:44.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:44.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.524+0000 7f142e000640 1 Processor -- start 2024-04-17T08:47:44.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.524+0000 7f142e000640 1 -- start start 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f142815c5a0 0x7f142815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f14280af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14280b46c0 0x7f14280af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f142815f440 con 0x7f14280b46c0 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f142815f720 con 0x7f142815c5a0 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142e000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f142815fa00 con 0x7f14280b4210 2024-04-17T08:47:44.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.526+0000 7f142d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14280b46c0 0x7f14280af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14280b46c0 0x7f14280af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58170/0 (socket says 172.21.15.67:58170) 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 -- 172.21.15.67:0/2484737506 learned_addr learned my addr 172.21.15.67:0/2484737506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142cc00640 1 --2- 172.21.15.67:0/2484737506 >> v2:172.21.15.187:3300/0 conn(0x7f142815c5a0 0x7f142815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f141fe00640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f14280af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 -- 172.21.15.67:0/2484737506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 msgr2=0x7f14280af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f14280af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 -- 172.21.15.67:0/2484737506 >> v2:172.21.15.187:3300/0 conn(0x7f142815c5a0 msgr2=0x7f142815ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 --2- 172.21.15.67:0/2484737506 >> v2:172.21.15.187:3300/0 conn(0x7f142815c5a0 0x7f142815ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.527+0000 7f142d600640 1 -- 172.21.15.67:0/2484737506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142815fce0 con 0x7f14280b46c0 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.528+0000 7f142d600640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14280b46c0 0x7f14280af980 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793ba64e915a02ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.528+0000 7f1427e00640 1 -- 172.21.15.67:0/2484737506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420038070 con 0x7f14280b46c0 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f1427e00640 1 -- 172.21.15.67:0/2484737506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1420031740 con 0x7f14280b46c0 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f1427e00640 1 -- 172.21.15.67:0/2484737506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142002f050 con 0x7f14280b46c0 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 -- 172.21.15.67:0/2484737506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14280b46c0 msgr2=0x7f14280af980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14280b46c0 0x7f14280af980 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142002b7e0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 -- 172.21.15.67:0/2484737506 shutdown_connections 2024-04-17T08:47:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14280b46c0 0x7f14280af980 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 --2- 172.21.15.67:0/2484737506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f14280af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 --2- 172.21.15.67:0/2484737506 >> v2:172.21.15.187:3300/0 conn(0x7f142815c5a0 0x7f142815ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 -- 172.21.15.67:0/2484737506 >> 172.21.15.67:0/2484737506 conn(0x7f1428067d10 msgr2=0x7f14280b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.529+0000 7f142e000640 1 -- 172.21.15.67:0/2484737506 shutdown_connections 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.530+0000 7f142e000640 1 -- 172.21.15.67:0/2484737506 wait complete. 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.531+0000 7f142e000640 1 Processor -- start 2024-04-17T08:47:44.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.531+0000 7f142e000640 1 -- start start 2024-04-17T08:47:44.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f1428142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f142815c5a0 0x7f14281f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f142806e7f0 con 0x7f14281f8620 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f142806e670 con 0x7f142815c5a0 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142e000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1428071df0 con 0x7f14280b4210 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.532+0000 7f142d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58180/0 (socket says 172.21.15.67:58180) 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 -- 172.21.15.67:0/3273618370 learned_addr learned my addr 172.21.15.67:0/3273618370 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142cc00640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f1428142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 -- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 msgr2=0x7f1428142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f1428142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 -- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f142815c5a0 msgr2=0x7f14281f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f142815c5a0 0x7f14281f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 -- 172.21.15.67:0/3273618370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1428156340 con 0x7f14281f8620 2024-04-17T08:47:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f142d600640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f14200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.533+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420042070 con 0x7f14281f8620 2024-04-17T08:47:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.534+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1420031e80 con 0x7f14281f8620 2024-04-17T08:47:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.534+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420038040 con 0x7f14281f8620 2024-04-17T08:47:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.534+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1428156620 con 0x7f14281f8620 2024-04-17T08:47:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.534+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1428156be0 con 0x7f14281f8620 2024-04-17T08:47:44.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.535+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1428068a10 con 0x7f14281f8620 2024-04-17T08:47:44.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.536+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f1420045070 con 0x7f14281f8620 2024-04-17T08:47:44.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.537+0000 7f1425600640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 0x7f140406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.537+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5737+0+0 (secure 0 0 0) 0x7f14200ada50 con 0x7f14281f8620 2024-04-17T08:47:44.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.537+0000 7f142cc00640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 0x7f140406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.539+0000 7f142cc00640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 0x7f140406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e90 tx=0x7f141800b040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.542+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f142004c030 con 0x7f14281f8620 2024-04-17T08:47:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.738+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14280b23c0 con 0x7f14281f8620 2024-04-17T08:47:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.741+0000 7f1425600640 1 -- 172.21.15.67:0/3273618370 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) ==== 74+0+16688 (secure 0 0 0) 0x7f142007bcd0 con 0x7f14281f8620 2024-04-17T08:47:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":211,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:43.271903+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 msgr2=0x7f140406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 0x7f140406eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e90 tx=0x7f141800b040 comp rx=0 tx=0).stop 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 msgr2=0x7f14281f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f14200046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 shutdown_connections 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f140406ca10 0x7f140406eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281f8620 0x7f14281f7ee0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f142815c5a0 0x7f14281f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 --2- 172.21.15.67:0/3273618370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14280b4210 0x7f1428142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 >> 172.21.15.67:0/3273618370 conn(0x7f1428067d10 msgr2=0x7f142815b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.743+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 shutdown_connections 2024-04-17T08:47:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.744+0000 7f142e000640 1 -- 172.21.15.67:0/3273618370 wait complete. 2024-04-17T08:47:44.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:47:44.765 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T08:47:44.767+0000 7f80a1e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:47:44.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.981+0000 7febe1e00640 1 Processor -- start 2024-04-17T08:47:44.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.981+0000 7febe1e00640 1 -- start start 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc09ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7febdc0d0280 0x7febdc0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febdc1605a0 0x7febdc160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7febdc15d350 con 0x7febdc1605a0 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7febdc0d0c20 con 0x7febdc0d0280 2024-04-17T08:47:44.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.982+0000 7febe1e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0cd630 con 0x7febdc09aa60 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.983+0000 7febdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febdc1605a0 0x7febdc160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.983+0000 7febdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febdc1605a0 0x7febdc160a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58184/0 (socket says 172.21.15.67:58184) 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.983+0000 7febdb400640 1 -- 172.21.15.67:0/2628590000 learned_addr learned my addr 172.21.15.67:0/2628590000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.983+0000 7febdaa00640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc09ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.983+0000 7febda000640 1 --2- 172.21.15.67:0/2628590000 >> v2:172.21.15.187:3300/0 conn(0x7febdc0d0280 0x7febdc0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.984+0000 7febdb400640 1 -- 172.21.15.67:0/2628590000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 msgr2=0x7febdc09ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.984+0000 7febdb400640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc09ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.984+0000 7febdb400640 1 -- 172.21.15.67:0/2628590000 >> v2:172.21.15.187:3300/0 conn(0x7febdc0d0280 msgr2=0x7febdc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.984+0000 7febdb400640 1 --2- 172.21.15.67:0/2628590000 >> v2:172.21.15.187:3300/0 conn(0x7febdc0d0280 0x7febdc0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.984+0000 7febdb400640 1 -- 172.21.15.67:0/2628590000 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc09b380 con 0x7febdc1605a0 2024-04-17T08:47:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.985+0000 7febdb400640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc160a30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7febdc073c10 tx=0x7febd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7a3af5318328f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.986+0000 7febd9600640 1 -- 172.21.15.67:0/2628590000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd402ca70 con 0x7febdc1605a0 2024-04-17T08:47:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.986+0000 7febd9600640 1 -- 172.21.15.67:0/2628590000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd402cc10 con 0x7febdc1605a0 2024-04-17T08:47:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.986+0000 7febd9600640 1 -- 172.21.15.67:0/2628590000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd40343f0 con 0x7febdc1605a0 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.986+0000 7febe1e00640 1 -- 172.21.15.67:0/2628590000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 msgr2=0x7febdc160a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.986+0000 7febe1e00640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc160a30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7febdc073c10 tx=0x7febd402bed0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 -- 172.21.15.67:0/2628590000 shutdown_connections 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc160a30 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 --2- 172.21.15.67:0/2628590000 >> v2:172.21.15.187:3300/0 conn(0x7febdc0d0280 0x7febdc0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 --2- 172.21.15.67:0/2628590000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc09ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 -- 172.21.15.67:0/2628590000 >> 172.21.15.67:0/2628590000 conn(0x7febdc0b41a0 msgr2=0x7febdc0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 -- 172.21.15.67:0/2628590000 shutdown_connections 2024-04-17T08:47:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.987+0000 7febe1e00640 1 -- 172.21.15.67:0/2628590000 wait complete. 2024-04-17T08:47:44.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.988+0000 7febe1e00640 1 Processor -- start 2024-04-17T08:47:44.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.989+0000 7febe1e00640 1 -- start start 2024-04-17T08:47:44.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.989+0000 7febe1e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc0b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febe1e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 0x7febdc0cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febe1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febe1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7febdc0d1420 con 0x7febdc1605a0 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febe1e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7febdc0d1140 con 0x7febdc0d0280 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febe1e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0d1700 con 0x7febdc09aa60 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58190/0 (socket says 172.21.15.67:58190) 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 -- 172.21.15.67:0/3343597364 learned_addr learned my addr 172.21.15.67:0/3343597364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdaa00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc0b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febda000640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 0x7febdc0cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 -- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 msgr2=0x7febdc0b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc0b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 -- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 msgr2=0x7febdc0cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 0x7febdc0cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.990+0000 7febdb400640 1 -- 172.21.15.67:0/3343597364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc0cc170 con 0x7febdc1605a0 2024-04-17T08:47:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febda000640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 0x7febdc0cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febdb400640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7febd4009f70 tx=0x7febd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4044020 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd4040690 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4009510 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc0c7090 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.991+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc0c7650 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.992+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febdc068ca0 con 0x7febdc1605a0 2024-04-17T08:47:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.993+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7febd4047020 con 0x7febdc1605a0 2024-04-17T08:47:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.995+0000 7febd3400640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 0x7febbc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.995+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7febd40aea10 con 0x7febdc1605a0 2024-04-17T08:47:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.995+0000 7febdaa00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 0x7febbc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:44.997+0000 7febdaa00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 0x7febbc06ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7febcc006fd0 tx=0x7febcc008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.000+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7febd404e020 con 0x7febdc1605a0 2024-04-17T08:47:45.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.246+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7febdc09bfa0 con 0x7febdc1605a0 2024-04-17T08:47:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.849+0000 7febd3400640 1 -- 172.21.15.67:0/3343597364 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v101) ==== 118+0+0 (secure 0 0 0) 0x7febd407ce30 con 0x7febdc1605a0 2024-04-17T08:47:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 msgr2=0x7febbc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 0x7febbc06ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7febcc006fd0 tx=0x7febcc008040 comp rx=0 tx=0).stop 2024-04-17T08:47:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 msgr2=0x7febdc156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7febd4009f70 tx=0x7febd4004a40 comp rx=0 tx=0).stop 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 shutdown_connections 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7febbc06ca40 0x7febbc06ef00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febdc1605a0 0x7febdc156640 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7febdc0d0280 0x7febdc0cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 --2- 172.21.15.67:0/3343597364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7febdc09aa60 0x7febdc0b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.853+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 >> 172.21.15.67:0/3343597364 conn(0x7febdc0b41a0 msgr2=0x7febdc06d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.854+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 shutdown_connections 2024-04-17T08:47:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:45.854+0000 7febe1e00640 1 -- 172.21.15.67:0/3343597364 wait complete. 2024-04-17T08:47:45.877 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:47:45.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:47:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.097+0000 7f5961800640 1 Processor -- start 2024-04-17T08:47:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.098+0000 7f5961800640 1 -- start start 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.099+0000 7f5961800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.099+0000 7f5961800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f595c06f800 0x7f595c06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.099+0000 7f5961800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f595c06df40 con 0x7f595c0cfc10 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.099+0000 7f5961800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f595c06ed00 con 0x7f595c06f800 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.099+0000 7f5961800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f595c06e870 con 0x7f595c15b6b0 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f595c06f800 0x7f595c06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f595c06f800 0x7f595c06be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58200/0 (socket says 172.21.15.67:58200) 2024-04-17T08:47:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 -- 172.21.15.67:0/408827555 learned_addr learned my addr 172.21.15.67:0/408827555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f595be00640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 -- 172.21.15.67:0/408827555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 msgr2=0x7f595c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c06b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 -- 172.21.15.67:0/408827555 >> v1:172.21.15.187:6789/0 conn(0x7f595c0cfc10 legacy=0x7f595c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.100+0000 7f5953e00640 1 -- 172.21.15.67:0/408827555 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c06e220 con 0x7f595c06f800 2024-04-17T08:47:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.102+0000 7f5953e00640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c06f800 0x7f595c06be00 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f5944009870 tx=0x7f594402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d48864853afb336 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.102+0000 7f595b400640 1 -- 172.21.15.67:0/408827555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5944035020 con 0x7f595c06f800 2024-04-17T08:47:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.102+0000 7f595b400640 1 -- 172.21.15.67:0/408827555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5944031640 con 0x7f595c06f800 2024-04-17T08:47:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.102+0000 7f595b400640 1 -- 172.21.15.67:0/408827555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594402d040 con 0x7f595c06f800 2024-04-17T08:47:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 -- 172.21.15.67:0/408827555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c06f800 msgr2=0x7f595c06be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c06f800 0x7f595c06be00 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f5944009870 tx=0x7f594402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 -- 172.21.15.67:0/408827555 shutdown_connections 2024-04-17T08:47:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c06f800 0x7f595c06be00 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 --2- 172.21.15.67:0/408827555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 -- 172.21.15.67:0/408827555 >> 172.21.15.67:0/408827555 conn(0x7f595c0b2190 msgr2=0x7f595c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 -- 172.21.15.67:0/408827555 shutdown_connections 2024-04-17T08:47:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.103+0000 7f5961800640 1 -- 172.21.15.67:0/408827555 wait complete. 2024-04-17T08:47:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.105+0000 7f5961800640 1 Processor -- start 2024-04-17T08:47:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.105+0000 7f5961800640 1 -- start start 2024-04-17T08:47:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f595c06f800 0x7f595c149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f595c0afee0 con 0x7f595c0cfc10 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f595c0afd60 con 0x7f595c06f800 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5961800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f595c0b44a0 con 0x7f595c15b6b0 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5953e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5953e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58206/0 (socket says 172.21.15.67:58206) 2024-04-17T08:47:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.106+0000 7f5953e00640 1 -- 172.21.15.67:0/2375812977 learned_addr learned my addr 172.21.15.67:0/2375812977 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f595be00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f595c06f800 0x7f595c149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 -- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 msgr2=0x7f595c157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 -- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f595c06f800 msgr2=0x7f595c149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f595c06f800 0x7f595c149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 -- 172.21.15.67:0/2375812977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c158080 con 0x7f595c0cfc10 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5953e00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f594402b930 tx=0x7f59440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5944040070 con 0x7f595c0cfc10 2024-04-17T08:47:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f594403e070 con 0x7f595c0cfc10 2024-04-17T08:47:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5944035040 con 0x7f595c0cfc10 2024-04-17T08:47:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.107+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595c158360 con 0x7f595c0cfc10 2024-04-17T08:47:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.108+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595c1498c0 con 0x7f595c0cfc10 2024-04-17T08:47:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.109+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f595c068a10 con 0x7f595c0cfc10 2024-04-17T08:47:46.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.110+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f5944038020 con 0x7f595c0cfc10 2024-04-17T08:47:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.111+0000 7f5958c00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 0x7f593806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.111+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f59440ad700 con 0x7f595c0cfc10 2024-04-17T08:47:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.112+0000 7f595be00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 0x7f593806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:46.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.113+0000 7f595be00640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 0x7f593806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f594c006fd0 tx=0x7f594c008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.116+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f594407b9f0 con 0x7f595c0cfc10 2024-04-17T08:47:46.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.354+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f595c06ee80 con 0x7f595c0cfc10 2024-04-17T08:47:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.863+0000 7f5958c00640 1 -- 172.21.15.67:0/2375812977 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v102) ==== 122+0+0 (secure 0 0 0) 0x7f5944043e40 con 0x7f595c0cfc10 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 msgr2=0x7f593806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 0x7f593806eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f594c006fd0 tx=0x7f594c008040 comp rx=0 tx=0).stop 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 msgr2=0x7f595c146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f594402b930 tx=0x7f59440046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 shutdown_connections 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f593806ca10 0x7f593806eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f595c15b6b0 0x7f595c157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cfc10 0x7f595c146a40 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 --2- 172.21.15.67:0/2375812977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f595c06f800 0x7f595c149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 >> 172.21.15.67:0/2375812977 conn(0x7f595c0b2190 msgr2=0x7f595c0b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 shutdown_connections 2024-04-17T08:47:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:46.867+0000 7f5961800640 1 -- 172.21.15.67:0/2375812977 wait complete. 2024-04-17T08:47:46.887 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:47:46.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.113+0000 7f5bd2e00640 1 Processor -- start 2024-04-17T08:47:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.113+0000 7f5bd2e00640 1 -- start start 2024-04-17T08:47:47.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5bcc160280 0x7f5bcc06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bcc0cd4a0 con 0x7f5bcc141ed0 2024-04-17T08:47:47.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bcc1607f0 con 0x7f5bcc06b8c0 2024-04-17T08:47:47.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5bcc15ce30 con 0x7f5bcc160280 2024-04-17T08:47:47.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5bcc141ed0 0x7f5bcc15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41272/0 (socket says 172.21.15.67:41272) 2024-04-17T08:47:47.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.115+0000 7f5bd2400640 1 -- 172.21.15.67:0/2157942802 learned_addr learned my addr 172.21.15.67:0/2157942802 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:47.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bd1000640 1 --2- 172.21.15.67:0/2157942802 >> v2:172.21.15.187:3300/0 conn(0x7f5bcc160280 0x7f5bcc06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238210300 0 0) 0x7f5bcc0cd4a0 con 0x7f5bcc141ed0 2024-04-17T08:47:47.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bb4003520 con 0x7f5bcc141ed0 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 141636056 0 0) 0x7f5bb4003520 con 0x7f5bcc141ed0 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 >> v2:172.21.15.187:3300/0 conn(0x7f5bcc160280 msgr2=0x7f5bcc06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 --2- 172.21.15.67:0/2157942802 >> v2:172.21.15.187:3300/0 conn(0x7f5bcc160280 0x7f5bcc06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 >> v1:172.21.15.187:6789/0 conn(0x7f5bcc06b8c0 legacy=0x7f5bcc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc160ad0 con 0x7f5bcc141ed0 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5bc40019f0 con 0x7f5bcc141ed0 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 >> v1:172.21.15.67:6789/0 conn(0x7f5bcc141ed0 legacy=0x7f5bcc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bb4003a70 0x7f5bb4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bb4004800 0x7f5bb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc15ce30 con 0x7f5bb4005210 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.116+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb400c140 con 0x7f5bb4003a70 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb400c340 con 0x7f5bb4004800 2024-04-17T08:47:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd1000640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bb4004800 0x7f5bb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd1a00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bb4003a70 0x7f5bb4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 -- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bb4004800 msgr2=0x7f5bb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bb4004800 0x7f5bb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 -- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bb4003a70 msgr2=0x7f5bb4003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bb4003a70 0x7f5bb4003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.117+0000 7f5bd2400640 1 -- 172.21.15.67:0/2157942802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb4003520 con 0x7f5bb4005210 2024-04-17T08:47:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.118+0000 7f5bd2400640 1 -- 172.21.15.67:0/2157942802 reap_dead start 2024-04-17T08:47:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.118+0000 7f5bd2400640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bb400b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400a790 tx=0x7f5bb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a6535434bbcef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400a3b0 con 0x7f5bb4005210 2024-04-17T08:47:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc4034670 con 0x7f5bb4005210 2024-04-17T08:47:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bcbe00640 1 -- 172.21.15.67:0/2157942802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4034a60 con 0x7f5bb4005210 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 -- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 msgr2=0x7f5bb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bb400b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400a790 tx=0x7f5bb4005830 comp rx=0 tx=0).stop 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 -- 172.21.15.67:0/2157942802 shutdown_connections 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/2157942802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bb400b9c0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 -- 172.21.15.67:0/2157942802 >> 172.21.15.67:0/2157942802 conn(0x7f5bcc067d10 msgr2=0x7f5bcc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 -- 172.21.15.67:0/2157942802 shutdown_connections 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.119+0000 7f5bd2e00640 1 -- 172.21.15.67:0/2157942802 wait complete. 2024-04-17T08:47:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.121+0000 7f5bd2e00640 1 Processor -- start 2024-04-17T08:47:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.121+0000 7f5bd2e00640 1 -- start start 2024-04-17T08:47:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bcc144590 0x7f5bcc13c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc1e9450 0x7f5bcc0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc13f9b0 con 0x7f5bb4005210 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc161090 con 0x7f5bcc1e9450 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.122+0000 7f5bd2e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5bcc13fb30 con 0x7f5bcc144590 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58220/0 (socket says 172.21.15.67:58220) 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 -- 172.21.15.67:0/1395930643 learned_addr learned my addr 172.21.15.67:0/1395930643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd2400640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc1e9450 0x7f5bcc0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 -- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bcc144590 msgr2=0x7f5bcc13c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1000640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bcc144590 0x7f5bcc13c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bcc144590 0x7f5bcc13c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 -- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc1e9450 msgr2=0x7f5bcc0d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc1e9450 0x7f5bcc0d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 -- 172.21.15.67:0/1395930643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc1e9ea0 con 0x7f5bb4005210 2024-04-17T08:47:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.123+0000 7f5bd1a00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000e480 tx=0x7f5bc000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.124+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00052e0 con 0x7f5bb4005210 2024-04-17T08:47:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.124+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc0018070 con 0x7f5bb4005210 2024-04-17T08:47:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.124+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0003830 con 0x7f5bb4005210 2024-04-17T08:47:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.124+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc0b3d70 con 0x7f5bb4005210 2024-04-17T08:47:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.124+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc0b4330 con 0x7f5bb4005210 2024-04-17T08:47:47.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.125+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc068a10 con 0x7f5bb4005210 2024-04-17T08:47:47.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.126+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f5bc00039d0 con 0x7f5bb4005210 2024-04-17T08:47:47.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.127+0000 7f5bc9600640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 0x7f5bac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.127+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f5bc008b9d0 con 0x7f5bb4005210 2024-04-17T08:47:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.128+0000 7f5bd1000640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 0x7f5bac06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.129+0000 7f5bd1000640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 0x7f5bac06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc1435e0 tx=0x7f5bb8008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.132+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5bc0059cc0 con 0x7f5bb4005210 2024-04-17T08:47:47.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.387+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bcc15f120 con 0x7f5bb4005210 2024-04-17T08:47:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.389+0000 7f5bc9600640 1 -- 172.21.15.67:0/1395930643 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7f5bc00595c0 con 0x7f5bb4005210 2024-04-17T08:47:47.387 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:47.387 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:47:47.388 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 msgr2=0x7f5bac06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 0x7f5bac06eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc1435e0 tx=0x7f5bb8008040 comp rx=0 tx=0).stop 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 msgr2=0x7f5bcc1e60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000e480 tx=0x7f5bc000e940 comp rx=0 tx=0).stop 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 shutdown_connections 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5bac06ca10 0x7f5bac06eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc1e9450 0x7f5bcc0d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5bcc144590 0x7f5bcc13c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 --2- 172.21.15.67:0/1395930643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bb4005210 0x7f5bcc1e60a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.391+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 >> 172.21.15.67:0/1395930643 conn(0x7f5bcc067d10 msgr2=0x7f5bcc074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.392+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 shutdown_connections 2024-04-17T08:47:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.392+0000 7f5bd2e00640 1 -- 172.21.15.67:0/1395930643 wait complete. 2024-04-17T08:47:47.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:47:47.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.636+0000 7f4256800640 1 Processor -- start 2024-04-17T08:47:47.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.636+0000 7f4256800640 1 -- start start 2024-04-17T08:47:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4256800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f425006b8c0 0x7f42500cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4256800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4256800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42500cd4a0 con 0x7f4250141ed0 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4256800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f425015ce30 con 0x7f425006b8c0 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4256800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f42501607f0 con 0x7f4250160280 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4255400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f425006b8c0 0x7f42500cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4255e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4250141ed0 0x7f425015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41282/0 (socket says 172.21.15.67:41282) 2024-04-17T08:47:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.638+0000 7f4255e00640 1 -- 172.21.15.67:0/3869330307 learned_addr learned my addr 172.21.15.67:0/3869330307 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f4254a00640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f424fe00640 1 -- 172.21.15.67:0/3869330307 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520719087 0 0) 0x7f42500cd4a0 con 0x7f4250141ed0 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f424fe00640 1 -- 172.21.15.67:0/3869330307 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4238003520 con 0x7f4250141ed0 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f4255400640 1 -- 172.21.15.67:0/3869330307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 msgr2=0x7f425006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f4255400640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f4255400640 1 -- 172.21.15.67:0/3869330307 >> v1:172.21.15.67:6789/0 conn(0x7f4250141ed0 legacy=0x7f425015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.639+0000 7f4255400640 1 -- 172.21.15.67:0/3869330307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4250160ad0 con 0x7f425006b8c0 2024-04-17T08:47:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f4255400640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f42500cd8b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f42440097a0 tx=0x7f424402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d877da37e8d03d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f424fe00640 1 -- 172.21.15.67:0/3869330307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244035020 con 0x7f425006b8c0 2024-04-17T08:47:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f424fe00640 1 -- 172.21.15.67:0/3869330307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4244031810 con 0x7f425006b8c0 2024-04-17T08:47:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f424fe00640 1 -- 172.21.15.67:0/3869330307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244031be0 con 0x7f425006b8c0 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f4256800640 1 -- 172.21.15.67:0/3869330307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 msgr2=0x7f42500cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.641+0000 7f4256800640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f42500cd8b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f42440097a0 tx=0x7f424402b8b0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 -- 172.21.15.67:0/3869330307 shutdown_connections 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 --2- 172.21.15.67:0/3869330307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f42500cd8b0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 -- 172.21.15.67:0/3869330307 >> 172.21.15.67:0/3869330307 conn(0x7f4250067d10 msgr2=0x7f42500b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 -- 172.21.15.67:0/3869330307 shutdown_connections 2024-04-17T08:47:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.642+0000 7f4256800640 1 -- 172.21.15.67:0/3869330307 wait complete. 2024-04-17T08:47:47.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.643+0000 7f4256800640 1 Processor -- start 2024-04-17T08:47:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.644+0000 7f4256800640 1 -- start start 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.644+0000 7f4256800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4256800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4250141ed0 0x7f425015b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4256800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4256800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f425006c1e0 con 0x7f425006b8c0 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4256800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f42500cddf0 con 0x7f4250141ed0 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4256800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4250161090 con 0x7f4250160280 2024-04-17T08:47:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58238/0 (socket says 172.21.15.67:58238) 2024-04-17T08:47:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 -- 172.21.15.67:0/451093287 learned_addr learned my addr 172.21.15.67:0/451093287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255e00640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 -- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 msgr2=0x7f425006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4254a00640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4250141ed0 0x7f425015b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 -- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4250141ed0 msgr2=0x7f425015b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4250141ed0 0x7f425015b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255400640 1 -- 172.21.15.67:0/451093287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425006e920 con 0x7f425006b8c0 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.645+0000 7f4255e00640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f4255400640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f42440098c0 tx=0x7f42440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244006e10 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424403bd70 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244045630 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425006ec00 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.646+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425006f1c0 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.647+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4250068a10 con 0x7f425006b8c0 2024-04-17T08:47:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.648+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f424404c060 con 0x7f425006b8c0 2024-04-17T08:47:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.650+0000 7f424d600640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 0x7f423006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.650+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f42440adcf0 con 0x7f425006b8c0 2024-04-17T08:47:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.650+0000 7f4254a00640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 0x7f423006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.651+0000 7f4254a00640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 0x7f423006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f423c006fd0 tx=0x7f423c008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.654+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f42440b2070 con 0x7f425006b8c0 2024-04-17T08:47:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.858+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42500af850 con 0x7f425006b8c0 2024-04-17T08:47:47.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.862+0000 7f424d600640 1 -- 172.21.15.67:0/451093287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) ==== 74+0+16675 (secure 0 0 0) 0x7f4244040070 con 0x7f425006b8c0 2024-04-17T08:47:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:47.861 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":212,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:47:44.835157+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 msgr2=0x7f423006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 0x7f423006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f423c006fd0 tx=0x7f423c008040 comp rx=0 tx=0).stop 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 msgr2=0x7f425006f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f42440098c0 tx=0x7f42440046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 shutdown_connections 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f423006ca10 0x7f423006eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4250160280 0x7f425006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4250141ed0 0x7f425015b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 --2- 172.21.15.67:0/451093287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f425006b8c0 0x7f425006f9c0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.864+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 >> 172.21.15.67:0/451093287 conn(0x7f4250067d10 msgr2=0x7f42500b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.865+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 shutdown_connections 2024-04-17T08:47:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:47.865+0000 7f4256800640 1 -- 172.21.15.67:0/451093287 wait complete. 2024-04-17T08:47:47.887 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:47:47.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:47:48.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.109+0000 7f1e2b200640 1 Processor -- start 2024-04-17T08:47:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.109+0000 7f1e2b200640 1 -- start start 2024-04-17T08:47:48.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c0d2d30 con 0x7f1e2c0cf910 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e2c0d3010 con 0x7f1e2c15ec60 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e2c0d49c0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e2c0cf910 0x7f1e2c0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41288/0 (socket says 172.21.15.67:41288) 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e2a800640 1 -- 172.21.15.67:0/2793050137 learned_addr learned my addr 172.21.15.67:0/2793050137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:48.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.111+0000 7f1e23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58254/0 (socket says 172.21.15.67:58254) 2024-04-17T08:47:48.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.112+0000 7f1e23e00640 1 -- 172.21.15.67:0/2793050137 >> v1:172.21.15.187:6789/0 conn(0x7f1e2c15ec60 legacy=0x7f1e2c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:48.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.112+0000 7f1e23e00640 1 -- 172.21.15.67:0/2793050137 >> v1:172.21.15.67:6789/0 conn(0x7f1e2c0cf910 legacy=0x7f1e2c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.112+0000 7f1e23e00640 1 -- 172.21.15.67:0/2793050137 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c0d32f0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.112+0000 7f1e29400640 1 -- 172.21.15.67:0/2793050137 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1304538119 0 0) 0x7f1e2c0d2d30 con 0x7f1e2c0cf910 2024-04-17T08:47:48.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.113+0000 7f1e23e00640 1 --2- 172.21.15.67:0/2793050137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f1e140097a0 tx=0x7f1e1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92d3f4eee7d2918 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.113+0000 7f1e29400640 1 -- 172.21.15.67:0/2793050137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14035020 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.113+0000 7f1e29400640 1 -- 172.21.15.67:0/2793050137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e14031830 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.113+0000 7f1e29400640 1 -- 172.21.15.67:0/2793050137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14031c00 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 -- 172.21.15.67:0/2793050137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 msgr2=0x7f1e2c0cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 --2- 172.21.15.67:0/2793050137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f1e140097a0 tx=0x7f1e1402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 -- 172.21.15.67:0/2793050137 shutdown_connections 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 --2- 172.21.15.67:0/2793050137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c0cf3d0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 -- 172.21.15.67:0/2793050137 >> 172.21.15.67:0/2793050137 conn(0x7f1e2c0d15f0 msgr2=0x7f1e2c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 -- 172.21.15.67:0/2793050137 shutdown_connections 2024-04-17T08:47:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.114+0000 7f1e2b200640 1 -- 172.21.15.67:0/2793050137 wait complete. 2024-04-17T08:47:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.116+0000 7f1e2b200640 1 Processor -- start 2024-04-17T08:47:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.117+0000 7f1e2b200640 1 -- start start 2024-04-17T08:47:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cf910 0x7f1e2c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 0x7f1e2c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c15f040 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c0d3660 con 0x7f1e2c0cf910 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e2b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2c0d00c0 con 0x7f1e2c15ec60 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58270/0 (socket says 172.21.15.67:58270) 2024-04-17T08:47:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.118+0000 7f1e29e00640 1 -- 172.21.15.67:0/1304602394 learned_addr learned my addr 172.21.15.67:0/1304602394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e2a800640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 0x7f1e2c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e23e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cf910 0x7f1e2c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 -- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 msgr2=0x7f1e2c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 0x7f1e2c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 -- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cf910 msgr2=0x7f1e2c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cf910 0x7f1e2c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 -- 172.21.15.67:0/1304602394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c0ce8c0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e2a800640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 0x7f1e2c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e29e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b420 tx=0x7f1e1c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.119+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c011830 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.120+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c016070 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.120+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2c0ceba0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.120+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c011c10 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.120+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e2c1576e0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.121+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e2c068a10 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.122+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f1e1c01b460 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.123+0000 7f1e22000640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 0x7f1e0806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.123+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f1e1c08b7c0 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.123+0000 7f1e23e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 0x7f1e0806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.124+0000 7f1e23e00640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 0x7f1e0806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1e14036000 tx=0x7f1e14008030 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.127+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e1c020020 con 0x7f1e2c0b6d00 2024-04-17T08:47:48.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.329+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 --> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e2c15d6e0 con 0x7f1e0806ca10 2024-04-17T08:47:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.352+0000 7f1e22000640 1 -- 172.21.15.67:0/1304602394 <== mgr.7379 v2:172.21.15.187:6832/2105214300 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f1e2c15d6e0 con 0x7f1e0806ca10 2024-04-17T08:47:48.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 msgr2=0x7f1e0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 0x7f1e0806eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1e14036000 tx=0x7f1e14008030 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 msgr2=0x7f1e2c153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b420 tx=0x7f1e1c00b8e0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 shutdown_connections 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e0806ca10 0x7f1e0806eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e2c15ec60 0x7f1e2c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e2c0cf910 0x7f1e2c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 --2- 172.21.15.67:0/1304602394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e2c0b6d00 0x7f1e2c153cb0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 >> 172.21.15.67:0/1304602394 conn(0x7f1e2c0d15f0 msgr2=0x7f1e2c0b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 shutdown_connections 2024-04-17T08:47:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.357+0000 7f1e2b200640 1 -- 172.21.15.67:0/1304602394 wait complete. 2024-04-17T08:47:48.376 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T08:47:47.179028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":592,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400444,"kb_used_data":21892,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754574276,"statfs":{"total":773094113280,"available":772684058624,"internally_reserved":0,"allocated":22417408,"data_stored":11513020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":387501616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.402555"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966585+0000","last_change":"2024-04-17T08:47:43.853279+0000","last_active":"2024-04-17T08:47:45.966585+0000","last_peered":"2024-04-17T08:47:45.966585+0000","last_clean":"2024-04-17T08:47:45.966585+0000","last_became_active":"2024-04-17T08:47:43.852929+0000","last_became_peered":"2024-04-17T08:47:43.852929+0000","last_unstale":"2024-04-17T08:47:45.966585+0000","last_undegraded":"2024-04-17T08:47:45.966585+0000","last_fullsized":"2024-04-17T08:47:45.966585+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:47.546488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967871+0000","last_change":"2024-04-17T08:47:43.844037+0000","last_active":"2024-04-17T08:47:45.967871+0000","last_peered":"2024-04-17T08:47:45.967871+0000","last_clean":"2024-04-17T08:47:45.967871+0000","last_became_active":"2024-04-17T08:47:43.843776+0000","last_became_peered":"2024-04-17T08:47:43.843776+0000","last_unstale":"2024-04-17T08:47:45.967871+0000","last_undegraded":"2024-04-17T08:47:45.967871+0000","last_fullsized":"2024-04-17T08:47:45.967871+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:17.623695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843305+0000","last_change":"2024-04-17T08:47:43.854189+0000","last_active":"2024-04-17T08:47:44.843305+0000","last_peered":"2024-04-17T08:47:44.843305+0000","last_clean":"2024-04-17T08:47:44.843305+0000","last_became_active":"2024-04-17T08:47:43.852791+0000","last_became_peered":"2024-04-17T08:47:43.852791+0000","last_unstale":"2024-04-17T08:47:44.843305+0000","last_undegraded":"2024-04-17T08:47:44.843305+0000","last_fullsized":"2024-04-17T08:47:44.843305+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:44:28.530312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967796+0000","last_change":"2024-04-17T08:47:43.845515+0000","last_active":"2024-04-17T08:47:45.967796+0000","last_peered":"2024-04-17T08:47:45.967796+0000","last_clean":"2024-04-17T08:47:45.967796+0000","last_became_active":"2024-04-17T08:47:43.844523+0000","last_became_peered":"2024-04-17T08:47:43.844523+0000","last_unstale":"2024-04-17T08:47:45.967796+0000","last_undegraded":"2024-04-17T08:47:45.967796+0000","last_fullsized":"2024-04-17T08:47:45.967796+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:16.247078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965906+0000","last_change":"2024-04-17T08:47:43.854577+0000","last_active":"2024-04-17T08:47:45.965906+0000","last_peered":"2024-04-17T08:47:45.965906+0000","last_clean":"2024-04-17T08:47:45.965906+0000","last_became_active":"2024-04-17T08:47:43.854235+0000","last_became_peered":"2024-04-17T08:47:43.854235+0000","last_unstale":"2024-04-17T08:47:45.965906+0000","last_undegraded":"2024-04-17T08:47:45.965906+0000","last_fullsized":"2024-04-17T08:47:45.965906+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:29.954534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843042+0000","last_change":"2024-04-17T08:47:43.850791+0000","last_active":"2024-04-17T08:47:44.843042+0000","last_peered":"2024-04-17T08:47:44.843042+0000","last_clean":"2024-04-17T08:47:44.843042+0000","last_became_active":"2024-04-17T08:47:43.850524+0000","last_became_peered":"2024-04-17T08:47:43.850524+0000","last_unstale":"2024-04-17T08:47:44.843042+0000","last_undegraded":"2024-04-17T08:47:44.843042+0000","last_fullsized":"2024-04-17T08:47:44.843042+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:12.297226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.847405+0000","last_change":"2024-04-17T08:47:43.843029+0000","last_active":"2024-04-17T08:47:44.847405+0000","last_peered":"2024-04-17T08:47:44.847405+0000","last_clean":"2024-04-17T08:47:44.847405+0000","last_became_active":"2024-04-17T08:47:43.842735+0000","last_became_peered":"2024-04-17T08:47:43.842735+0000","last_unstale":"2024-04-17T08:47:44.847405+0000","last_undegraded":"2024-04-17T08:47:44.847405+0000","last_fullsized":"2024-04-17T08:47:44.847405+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:28.771576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964097+0000","last_change":"2024-04-17T08:47:43.846810+0000","last_active":"2024-04-17T08:47:45.964097+0000","last_peered":"2024-04-17T08:47:45.964097+0000","last_clean":"2024-04-17T08:47:45.964097+0000","last_became_active":"2024-04-17T08:47:43.846478+0000","last_became_peered":"2024-04-17T08:47:43.846478+0000","last_unstale":"2024-04-17T08:47:45.964097+0000","last_undegraded":"2024-04-17T08:47:45.964097+0000","last_fullsized":"2024-04-17T08:47:45.964097+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:01.526257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849235+0000","last_change":"2024-04-17T08:47:43.839332+0000","last_active":"2024-04-17T08:47:44.849235+0000","last_peered":"2024-04-17T08:47:44.849235+0000","last_clean":"2024-04-17T08:47:44.849235+0000","last_became_active":"2024-04-17T08:47:43.838943+0000","last_became_peered":"2024-04-17T08:47:43.838943+0000","last_unstale":"2024-04-17T08:47:44.849235+0000","last_undegraded":"2024-04-17T08:47:44.849235+0000","last_fullsized":"2024-04-17T08:47:44.849235+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:09.970222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860932+0000","last_change":"2024-04-17T08:47:43.852299+0000","last_active":"2024-04-17T08:47:44.860932+0000","last_peered":"2024-04-17T08:47:44.860932+0000","last_clean":"2024-04-17T08:47:44.860932+0000","last_became_active":"2024-04-17T08:47:43.851087+0000","last_became_peered":"2024-04-17T08:47:43.851087+0000","last_unstale":"2024-04-17T08:47:44.860932+0000","last_undegraded":"2024-04-17T08:47:44.860932+0000","last_fullsized":"2024-04-17T08:47:44.860932+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:44:25.624806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848161+0000","last_change":"2024-04-17T08:47:43.834717+0000","last_active":"2024-04-17T08:47:44.848161+0000","last_peered":"2024-04-17T08:47:44.848161+0000","last_clean":"2024-04-17T08:47:44.848161+0000","last_became_active":"2024-04-17T08:47:43.834115+0000","last_became_peered":"2024-04-17T08:47:43.834115+0000","last_unstale":"2024-04-17T08:47:44.848161+0000","last_undegraded":"2024-04-17T08:47:44.848161+0000","last_fullsized":"2024-04-17T08:47:44.848161+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:24.640878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968217+0000","last_change":"2024-04-17T08:47:43.847876+0000","last_active":"2024-04-17T08:47:45.968217+0000","last_peered":"2024-04-17T08:47:45.968217+0000","last_clean":"2024-04-17T08:47:45.968217+0000","last_became_active":"2024-04-17T08:47:43.847521+0000","last_became_peered":"2024-04-17T08:47:43.847521+0000","last_unstale":"2024-04-17T08:47:45.968217+0000","last_undegraded":"2024-04-17T08:47:45.968217+0000","last_fullsized":"2024-04-17T08:47:45.968217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:05.032689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967679+0000","last_change":"2024-04-17T08:47:43.843660+0000","last_active":"2024-04-17T08:47:45.967679+0000","last_peered":"2024-04-17T08:47:45.967679+0000","last_clean":"2024-04-17T08:47:45.967679+0000","last_became_active":"2024-04-17T08:47:43.843371+0000","last_became_peered":"2024-04-17T08:47:43.843371+0000","last_unstale":"2024-04-17T08:47:45.967679+0000","last_undegraded":"2024-04-17T08:47:45.967679+0000","last_fullsized":"2024-04-17T08:47:45.967679+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:45.521002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849941+0000","last_change":"2024-04-17T08:47:43.847563+0000","last_active":"2024-04-17T08:47:44.849941+0000","last_peered":"2024-04-17T08:47:44.849941+0000","last_clean":"2024-04-17T08:47:44.849941+0000","last_became_active":"2024-04-17T08:47:43.847239+0000","last_became_peered":"2024-04-17T08:47:43.847239+0000","last_unstale":"2024-04-17T08:47:44.849941+0000","last_undegraded":"2024-04-17T08:47:44.849941+0000","last_fullsized":"2024-04-17T08:47:44.849941+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:21:55.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967766+0000","last_change":"2024-04-17T08:47:43.849404+0000","last_active":"2024-04-17T08:47:45.967766+0000","last_peered":"2024-04-17T08:47:45.967766+0000","last_clean":"2024-04-17T08:47:45.967766+0000","last_became_active":"2024-04-17T08:47:43.848895+0000","last_became_peered":"2024-04-17T08:47:43.848895+0000","last_unstale":"2024-04-17T08:47:45.967766+0000","last_undegraded":"2024-04-17T08:47:45.967766+0000","last_fullsized":"2024-04-17T08:47:45.967766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:08.366575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965032+0000","last_change":"2024-04-17T08:47:43.856933+0000","last_active":"2024-04-17T08:47:45.965032+0000","last_peered":"2024-04-17T08:47:45.965032+0000","last_clean":"2024-04-17T08:47:45.965032+0000","last_became_active":"2024-04-17T08:47:43.856580+0000","last_became_peered":"2024-04-17T08:47:43.856580+0000","last_unstale":"2024-04-17T08:47:45.965032+0000","last_undegraded":"2024-04-17T08:47:45.965032+0000","last_fullsized":"2024-04-17T08:47:45.965032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:13.389562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966486+0000","last_change":"2024-04-17T08:47:43.855856+0000","last_active":"2024-04-17T08:47:45.966486+0000","last_peered":"2024-04-17T08:47:45.966486+0000","last_clean":"2024-04-17T08:47:45.966486+0000","last_became_active":"2024-04-17T08:47:43.853234+0000","last_became_peered":"2024-04-17T08:47:43.853234+0000","last_unstale":"2024-04-17T08:47:45.966486+0000","last_undegraded":"2024-04-17T08:47:45.966486+0000","last_fullsized":"2024-04-17T08:47:45.966486+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:13:25.564151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966180+0000","last_change":"2024-04-17T08:47:43.850753+0000","last_active":"2024-04-17T08:47:45.966180+0000","last_peered":"2024-04-17T08:47:45.966180+0000","last_clean":"2024-04-17T08:47:45.966180+0000","last_became_active":"2024-04-17T08:47:43.850333+0000","last_became_peered":"2024-04-17T08:47:43.850333+0000","last_unstale":"2024-04-17T08:47:45.966180+0000","last_undegraded":"2024-04-17T08:47:45.966180+0000","last_fullsized":"2024-04-17T08:47:45.966180+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:38.547787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860792+0000","last_change":"2024-04-17T08:47:43.837722+0000","last_active":"2024-04-17T08:47:44.860792+0000","last_peered":"2024-04-17T08:47:44.860792+0000","last_clean":"2024-04-17T08:47:44.860792+0000","last_became_active":"2024-04-17T08:47:43.837365+0000","last_became_peered":"2024-04-17T08:47:43.837365+0000","last_unstale":"2024-04-17T08:47:44.860792+0000","last_undegraded":"2024-04-17T08:47:44.860792+0000","last_fullsized":"2024-04-17T08:47:44.860792+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:51.373438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965715+0000","last_change":"2024-04-17T08:47:43.853921+0000","last_active":"2024-04-17T08:47:45.965715+0000","last_peered":"2024-04-17T08:47:45.965715+0000","last_clean":"2024-04-17T08:47:45.965715+0000","last_became_active":"2024-04-17T08:47:43.853359+0000","last_became_peered":"2024-04-17T08:47:43.853359+0000","last_unstale":"2024-04-17T08:47:45.965715+0000","last_undegraded":"2024-04-17T08:47:45.965715+0000","last_fullsized":"2024-04-17T08:47:45.965715+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:42.733226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966099+0000","last_change":"2024-04-17T08:47:43.854643+0000","last_active":"2024-04-17T08:47:45.966099+0000","last_peered":"2024-04-17T08:47:45.966099+0000","last_clean":"2024-04-17T08:47:45.966099+0000","last_became_active":"2024-04-17T08:47:43.854331+0000","last_became_peered":"2024-04-17T08:47:43.854331+0000","last_unstale":"2024-04-17T08:47:45.966099+0000","last_undegraded":"2024-04-17T08:47:45.966099+0000","last_fullsized":"2024-04-17T08:47:45.966099+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:15:11.176172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860332+0000","last_change":"2024-04-17T08:47:43.845007+0000","last_active":"2024-04-17T08:47:44.860332+0000","last_peered":"2024-04-17T08:47:44.860332+0000","last_clean":"2024-04-17T08:47:44.860332+0000","last_became_active":"2024-04-17T08:47:43.844438+0000","last_became_peered":"2024-04-17T08:47:43.844438+0000","last_unstale":"2024-04-17T08:47:44.860332+0000","last_undegraded":"2024-04-17T08:47:44.860332+0000","last_fullsized":"2024-04-17T08:47:44.860332+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:33.785820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843148+0000","last_change":"2024-04-17T08:47:43.859763+0000","last_active":"2024-04-17T08:47:44.843148+0000","last_peered":"2024-04-17T08:47:44.843148+0000","last_clean":"2024-04-17T08:47:44.843148+0000","last_became_active":"2024-04-17T08:47:43.859431+0000","last_became_peered":"2024-04-17T08:47:43.859431+0000","last_unstale":"2024-04-17T08:47:44.843148+0000","last_undegraded":"2024-04-17T08:47:44.843148+0000","last_fullsized":"2024-04-17T08:47:44.843148+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:35.081905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964430+0000","last_change":"2024-04-17T08:47:43.859249+0000","last_active":"2024-04-17T08:47:45.964430+0000","last_peered":"2024-04-17T08:47:45.964430+0000","last_clean":"2024-04-17T08:47:45.964430+0000","last_became_active":"2024-04-17T08:47:43.858962+0000","last_became_peered":"2024-04-17T08:47:43.858962+0000","last_unstale":"2024-04-17T08:47:45.964430+0000","last_undegraded":"2024-04-17T08:47:45.964430+0000","last_fullsized":"2024-04-17T08:47:45.964430+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.047336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968653+0000","last_change":"2024-04-17T08:47:43.853286+0000","last_active":"2024-04-17T08:47:45.968653+0000","last_peered":"2024-04-17T08:47:45.968653+0000","last_clean":"2024-04-17T08:47:45.968653+0000","last_became_active":"2024-04-17T08:47:43.852929+0000","last_became_peered":"2024-04-17T08:47:43.852929+0000","last_unstale":"2024-04-17T08:47:45.968653+0000","last_undegraded":"2024-04-17T08:47:45.968653+0000","last_fullsized":"2024-04-17T08:47:45.968653+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:52.489657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848521+0000","last_change":"2024-04-17T08:47:43.831730+0000","last_active":"2024-04-17T08:47:44.848521+0000","last_peered":"2024-04-17T08:47:44.848521+0000","last_clean":"2024-04-17T08:47:44.848521+0000","last_became_active":"2024-04-17T08:47:43.831153+0000","last_became_peered":"2024-04-17T08:47:43.831153+0000","last_unstale":"2024-04-17T08:47:44.848521+0000","last_undegraded":"2024-04-17T08:47:44.848521+0000","last_fullsized":"2024-04-17T08:47:44.848521+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:09.489346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843393+0000","last_change":"2024-04-17T08:47:43.845063+0000","last_active":"2024-04-17T08:47:44.843393+0000","last_peered":"2024-04-17T08:47:44.843393+0000","last_clean":"2024-04-17T08:47:44.843393+0000","last_became_active":"2024-04-17T08:47:43.844147+0000","last_became_peered":"2024-04-17T08:47:43.844147+0000","last_unstale":"2024-04-17T08:47:44.843393+0000","last_undegraded":"2024-04-17T08:47:44.843393+0000","last_fullsized":"2024-04-17T08:47:44.843393+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:22.012868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966023+0000","last_change":"2024-04-17T08:47:43.855551+0000","last_active":"2024-04-17T08:47:45.966023+0000","last_peered":"2024-04-17T08:47:45.966023+0000","last_clean":"2024-04-17T08:47:45.966023+0000","last_became_active":"2024-04-17T08:47:43.855261+0000","last_became_peered":"2024-04-17T08:47:43.855261+0000","last_unstale":"2024-04-17T08:47:45.966023+0000","last_undegraded":"2024-04-17T08:47:45.966023+0000","last_fullsized":"2024-04-17T08:47:45.966023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:59.848987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968075+0000","last_change":"2024-04-17T08:47:43.848415+0000","last_active":"2024-04-17T08:47:45.968075+0000","last_peered":"2024-04-17T08:47:45.968075+0000","last_clean":"2024-04-17T08:47:45.968075+0000","last_became_active":"2024-04-17T08:47:43.847926+0000","last_became_peered":"2024-04-17T08:47:43.847926+0000","last_unstale":"2024-04-17T08:47:45.968075+0000","last_undegraded":"2024-04-17T08:47:45.968075+0000","last_fullsized":"2024-04-17T08:47:45.968075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:19.989335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849327+0000","last_change":"2024-04-17T08:47:43.837262+0000","last_active":"2024-04-17T08:47:44.849327+0000","last_peered":"2024-04-17T08:47:44.849327+0000","last_clean":"2024-04-17T08:47:44.849327+0000","last_became_active":"2024-04-17T08:47:43.836831+0000","last_became_peered":"2024-04-17T08:47:43.836831+0000","last_unstale":"2024-04-17T08:47:44.849327+0000","last_undegraded":"2024-04-17T08:47:44.849327+0000","last_fullsized":"2024-04-17T08:47:44.849327+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:26.885336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842883+0000","last_change":"2024-04-17T08:47:43.856823+0000","last_active":"2024-04-17T08:47:44.842883+0000","last_peered":"2024-04-17T08:47:44.842883+0000","last_clean":"2024-04-17T08:47:44.842883+0000","last_became_active":"2024-04-17T08:47:43.856452+0000","last_became_peered":"2024-04-17T08:47:43.856452+0000","last_unstale":"2024-04-17T08:47:44.842883+0000","last_undegraded":"2024-04-17T08:47:44.842883+0000","last_fullsized":"2024-04-17T08:47:44.842883+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:29.564289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965175+0000","last_change":"2024-04-17T08:47:43.848798+0000","last_active":"2024-04-17T08:47:45.965175+0000","last_peered":"2024-04-17T08:47:45.965175+0000","last_clean":"2024-04-17T08:47:45.965175+0000","last_became_active":"2024-04-17T08:47:43.844231+0000","last_became_peered":"2024-04-17T08:47:43.844231+0000","last_unstale":"2024-04-17T08:47:45.965175+0000","last_undegraded":"2024-04-17T08:47:45.965175+0000","last_fullsized":"2024-04-17T08:47:45.965175+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.959616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965217+0000","last_change":"2024-04-17T08:47:43.851228+0000","last_active":"2024-04-17T08:47:45.965217+0000","last_peered":"2024-04-17T08:47:45.965217+0000","last_clean":"2024-04-17T08:47:45.965217+0000","last_became_active":"2024-04-17T08:47:43.849853+0000","last_became_peered":"2024-04-17T08:47:43.849853+0000","last_unstale":"2024-04-17T08:47:45.965217+0000","last_undegraded":"2024-04-17T08:47:45.965217+0000","last_fullsized":"2024-04-17T08:47:45.965217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:26:56.580759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"211'4","reported_seq":18,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965980+0000","last_change":"2024-04-17T08:47:42.813949+0000","last_active":"2024-04-17T08:47:45.965980+0000","last_peered":"2024-04-17T08:47:45.965980+0000","last_clean":"2024-04-17T08:47:45.965980+0000","last_became_active":"2024-04-17T08:47:42.813312+0000","last_became_peered":"2024-04-17T08:47:42.813312+0000","last_unstale":"2024-04-17T08:47:45.965980+0000","last_undegraded":"2024-04-17T08:47:45.965980+0000","last_fullsized":"2024-04-17T08:47:45.965980+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:51.655582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861544+0000","last_change":"2024-04-17T08:47:43.852046+0000","last_active":"2024-04-17T08:47:44.861544+0000","last_peered":"2024-04-17T08:47:44.861544+0000","last_clean":"2024-04-17T08:47:44.861544+0000","last_became_active":"2024-04-17T08:47:43.849718+0000","last_became_peered":"2024-04-17T08:47:43.849718+0000","last_unstale":"2024-04-17T08:47:44.861544+0000","last_undegraded":"2024-04-17T08:47:44.861544+0000","last_fullsized":"2024-04-17T08:47:44.861544+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:54.241566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849147+0000","last_change":"2024-04-17T08:47:42.819004+0000","last_active":"2024-04-17T08:47:44.849147+0000","last_peered":"2024-04-17T08:47:44.849147+0000","last_clean":"2024-04-17T08:47:44.849147+0000","last_became_active":"2024-04-17T08:47:42.818150+0000","last_became_peered":"2024-04-17T08:47:42.818150+0000","last_unstale":"2024-04-17T08:47:44.849147+0000","last_undegraded":"2024-04-17T08:47:44.849147+0000","last_fullsized":"2024-04-17T08:47:44.849147+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:36.111988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848259+0000","last_change":"2024-04-17T08:47:43.844060+0000","last_active":"2024-04-17T08:47:44.848259+0000","last_peered":"2024-04-17T08:47:44.848259+0000","last_clean":"2024-04-17T08:47:44.848259+0000","last_became_active":"2024-04-17T08:47:43.842814+0000","last_became_peered":"2024-04-17T08:47:43.842814+0000","last_unstale":"2024-04-17T08:47:44.848259+0000","last_undegraded":"2024-04-17T08:47:44.848259+0000","last_fullsized":"2024-04-17T08:47:44.848259+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:31.382958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843386+0000","last_change":"2024-04-17T08:47:42.814575+0000","last_active":"2024-04-17T08:47:44.843386+0000","last_peered":"2024-04-17T08:47:44.843386+0000","last_clean":"2024-04-17T08:47:44.843386+0000","last_became_active":"2024-04-17T08:47:42.814291+0000","last_became_peered":"2024-04-17T08:47:42.814291+0000","last_unstale":"2024-04-17T08:47:44.843386+0000","last_undegraded":"2024-04-17T08:47:44.843386+0000","last_fullsized":"2024-04-17T08:47:44.843386+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:08.956721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850609+0000","last_change":"2024-04-17T08:47:43.832507+0000","last_active":"2024-04-17T08:47:44.850609+0000","last_peered":"2024-04-17T08:47:44.850609+0000","last_clean":"2024-04-17T08:47:44.850609+0000","last_became_active":"2024-04-17T08:47:43.832035+0000","last_became_peered":"2024-04-17T08:47:43.832035+0000","last_unstale":"2024-04-17T08:47:44.850609+0000","last_undegraded":"2024-04-17T08:47:44.850609+0000","last_fullsized":"2024-04-17T08:47:44.850609+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:24.640878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968084+0000","last_change":"2024-04-17T08:47:42.819118+0000","last_active":"2024-04-17T08:47:45.968084+0000","last_peered":"2024-04-17T08:47:45.968084+0000","last_clean":"2024-04-17T08:47:45.968084+0000","last_became_active":"2024-04-17T08:47:42.818837+0000","last_became_peered":"2024-04-17T08:47:42.818837+0000","last_unstale":"2024-04-17T08:47:45.968084+0000","last_undegraded":"2024-04-17T08:47:45.968084+0000","last_fullsized":"2024-04-17T08:47:45.968084+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:34:16.764062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860536+0000","last_change":"2024-04-17T08:47:43.847128+0000","last_active":"2024-04-17T08:47:44.860536+0000","last_peered":"2024-04-17T08:47:44.860536+0000","last_clean":"2024-04-17T08:47:44.860536+0000","last_became_active":"2024-04-17T08:47:43.841146+0000","last_became_peered":"2024-04-17T08:47:43.841146+0000","last_unstale":"2024-04-17T08:47:44.860536+0000","last_undegraded":"2024-04-17T08:47:44.860536+0000","last_fullsized":"2024-04-17T08:47:44.860536+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:10.499515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966250+0000","last_change":"2024-04-17T08:47:42.814949+0000","last_active":"2024-04-17T08:47:45.966250+0000","last_peered":"2024-04-17T08:47:45.966250+0000","last_clean":"2024-04-17T08:47:45.966250+0000","last_became_active":"2024-04-17T08:47:42.814215+0000","last_became_peered":"2024-04-17T08:47:42.814215+0000","last_unstale":"2024-04-17T08:47:45.966250+0000","last_undegraded":"2024-04-17T08:47:45.966250+0000","last_fullsized":"2024-04-17T08:47:45.966250+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:06.281860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964724+0000","last_change":"2024-04-17T08:47:43.854761+0000","last_active":"2024-04-17T08:47:45.964724+0000","last_peered":"2024-04-17T08:47:45.964724+0000","last_clean":"2024-04-17T08:47:45.964724+0000","last_became_active":"2024-04-17T08:47:43.854501+0000","last_became_peered":"2024-04-17T08:47:43.854501+0000","last_unstale":"2024-04-17T08:47:45.964724+0000","last_undegraded":"2024-04-17T08:47:45.964724+0000","last_fullsized":"2024-04-17T08:47:45.964724+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:03.781249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964476+0000","last_change":"2024-04-17T08:47:42.816347+0000","last_active":"2024-04-17T08:47:45.964476+0000","last_peered":"2024-04-17T08:47:45.964476+0000","last_clean":"2024-04-17T08:47:45.964476+0000","last_became_active":"2024-04-17T08:47:42.816044+0000","last_became_peered":"2024-04-17T08:47:42.816044+0000","last_unstale":"2024-04-17T08:47:45.964476+0000","last_undegraded":"2024-04-17T08:47:45.964476+0000","last_fullsized":"2024-04-17T08:47:45.964476+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:16:24.607522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.847835+0000","last_change":"2024-04-17T08:47:43.834894+0000","last_active":"2024-04-17T08:47:44.847835+0000","last_peered":"2024-04-17T08:47:44.847835+0000","last_clean":"2024-04-17T08:47:44.847835+0000","last_became_active":"2024-04-17T08:47:43.833320+0000","last_became_peered":"2024-04-17T08:47:43.833320+0000","last_unstale":"2024-04-17T08:47:44.847835+0000","last_undegraded":"2024-04-17T08:47:44.847835+0000","last_fullsized":"2024-04-17T08:47:44.847835+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:21:55.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965727+0000","last_change":"2024-04-17T08:47:42.817918+0000","last_active":"2024-04-17T08:47:45.965727+0000","last_peered":"2024-04-17T08:47:45.965727+0000","last_clean":"2024-04-17T08:47:45.965727+0000","last_became_active":"2024-04-17T08:47:42.817551+0000","last_became_peered":"2024-04-17T08:47:42.817551+0000","last_unstale":"2024-04-17T08:47:45.965727+0000","last_undegraded":"2024-04-17T08:47:45.965727+0000","last_fullsized":"2024-04-17T08:47:45.965727+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:52.191803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842894+0000","last_change":"2024-04-17T08:47:43.847468+0000","last_active":"2024-04-17T08:47:44.842894+0000","last_peered":"2024-04-17T08:47:44.842894+0000","last_clean":"2024-04-17T08:47:44.842894+0000","last_became_active":"2024-04-17T08:47:43.847156+0000","last_became_peered":"2024-04-17T08:47:43.847156+0000","last_unstale":"2024-04-17T08:47:44.842894+0000","last_undegraded":"2024-04-17T08:47:44.842894+0000","last_fullsized":"2024-04-17T08:47:44.842894+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:08.979625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849481+0000","last_change":"2024-04-17T08:47:42.810707+0000","last_active":"2024-04-17T08:47:44.849481+0000","last_peered":"2024-04-17T08:47:44.849481+0000","last_clean":"2024-04-17T08:47:44.849481+0000","last_became_active":"2024-04-17T08:47:42.810366+0000","last_became_peered":"2024-04-17T08:47:42.810366+0000","last_unstale":"2024-04-17T08:47:44.849481+0000","last_undegraded":"2024-04-17T08:47:44.849481+0000","last_fullsized":"2024-04-17T08:47:44.849481+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:16.773889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964824+0000","last_change":"2024-04-17T08:47:43.853611+0000","last_active":"2024-04-17T08:47:45.964824+0000","last_peered":"2024-04-17T08:47:45.964824+0000","last_clean":"2024-04-17T08:47:45.964824+0000","last_became_active":"2024-04-17T08:47:43.853221+0000","last_became_peered":"2024-04-17T08:47:43.853221+0000","last_unstale":"2024-04-17T08:47:45.964824+0000","last_undegraded":"2024-04-17T08:47:45.964824+0000","last_fullsized":"2024-04-17T08:47:45.964824+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:45.091029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966250+0000","last_change":"2024-04-17T08:47:42.821446+0000","last_active":"2024-04-17T08:47:45.966250+0000","last_peered":"2024-04-17T08:47:45.966250+0000","last_clean":"2024-04-17T08:47:45.966250+0000","last_became_active":"2024-04-17T08:47:42.821159+0000","last_became_peered":"2024-04-17T08:47:42.821159+0000","last_unstale":"2024-04-17T08:47:45.966250+0000","last_undegraded":"2024-04-17T08:47:45.966250+0000","last_fullsized":"2024-04-17T08:47:45.966250+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:04:16.800633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849882+0000","last_change":"2024-04-17T08:47:43.832633+0000","last_active":"2024-04-17T08:47:44.849882+0000","last_peered":"2024-04-17T08:47:44.849882+0000","last_clean":"2024-04-17T08:47:44.849882+0000","last_became_active":"2024-04-17T08:47:43.832195+0000","last_became_peered":"2024-04-17T08:47:43.832195+0000","last_unstale":"2024-04-17T08:47:44.849882+0000","last_undegraded":"2024-04-17T08:47:44.849882+0000","last_fullsized":"2024-04-17T08:47:44.849882+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:35.081905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"211'3","reported_seq":17,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966037+0000","last_change":"2024-04-17T08:47:42.813363+0000","last_active":"2024-04-17T08:47:45.966037+0000","last_peered":"2024-04-17T08:47:45.966037+0000","last_clean":"2024-04-17T08:47:45.966037+0000","last_became_active":"2024-04-17T08:47:42.813055+0000","last_became_peered":"2024-04-17T08:47:42.813055+0000","last_unstale":"2024-04-17T08:47:45.966037+0000","last_undegraded":"2024-04-17T08:47:45.966037+0000","last_fullsized":"2024-04-17T08:47:45.966037+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:43.551895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861057+0000","last_change":"2024-04-17T08:47:43.844847+0000","last_active":"2024-04-17T08:47:44.861057+0000","last_peered":"2024-04-17T08:47:44.861057+0000","last_clean":"2024-04-17T08:47:44.861057+0000","last_became_active":"2024-04-17T08:47:43.844568+0000","last_became_peered":"2024-04-17T08:47:43.844568+0000","last_unstale":"2024-04-17T08:47:44.861057+0000","last_undegraded":"2024-04-17T08:47:44.861057+0000","last_fullsized":"2024-04-17T08:47:44.861057+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:16.529905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"211'6","reported_seq":20,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842627+0000","last_change":"2024-04-17T08:47:42.811120+0000","last_active":"2024-04-17T08:47:44.842627+0000","last_peered":"2024-04-17T08:47:44.842627+0000","last_clean":"2024-04-17T08:47:44.842627+0000","last_became_active":"2024-04-17T08:47:42.810828+0000","last_became_peered":"2024-04-17T08:47:42.810828+0000","last_unstale":"2024-04-17T08:47:44.842627+0000","last_undegraded":"2024-04-17T08:47:44.842627+0000","last_fullsized":"2024-04-17T08:47:44.842627+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:04.411604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843110+0000","last_change":"2024-04-17T08:47:43.857598+0000","last_active":"2024-04-17T08:47:44.843110+0000","last_peered":"2024-04-17T08:47:44.843110+0000","last_clean":"2024-04-17T08:47:44.843110+0000","last_became_active":"2024-04-17T08:47:43.857287+0000","last_became_peered":"2024-04-17T08:47:43.857287+0000","last_unstale":"2024-04-17T08:47:44.843110+0000","last_undegraded":"2024-04-17T08:47:44.843110+0000","last_fullsized":"2024-04-17T08:47:44.843110+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:51.822943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965921+0000","last_change":"2024-04-17T08:47:42.821363+0000","last_active":"2024-04-17T08:47:45.965921+0000","last_peered":"2024-04-17T08:47:45.965921+0000","last_clean":"2024-04-17T08:47:45.965921+0000","last_became_active":"2024-04-17T08:47:42.821002+0000","last_became_peered":"2024-04-17T08:47:42.821002+0000","last_unstale":"2024-04-17T08:47:45.965921+0000","last_undegraded":"2024-04-17T08:47:45.965921+0000","last_fullsized":"2024-04-17T08:47:45.965921+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:26.163096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849440+0000","last_change":"2024-04-17T08:47:43.830583+0000","last_active":"2024-04-17T08:47:44.849440+0000","last_peered":"2024-04-17T08:47:44.849440+0000","last_clean":"2024-04-17T08:47:44.849440+0000","last_became_active":"2024-04-17T08:47:43.830071+0000","last_became_peered":"2024-04-17T08:47:43.830071+0000","last_unstale":"2024-04-17T08:47:44.849440+0000","last_undegraded":"2024-04-17T08:47:44.849440+0000","last_fullsized":"2024-04-17T08:47:44.849440+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:56.091246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860666+0000","last_change":"2024-04-17T08:47:42.820473+0000","last_active":"2024-04-17T08:47:44.860666+0000","last_peered":"2024-04-17T08:47:44.860666+0000","last_clean":"2024-04-17T08:47:44.860666+0000","last_became_active":"2024-04-17T08:47:42.819120+0000","last_became_peered":"2024-04-17T08:47:42.819120+0000","last_unstale":"2024-04-17T08:47:44.860666+0000","last_undegraded":"2024-04-17T08:47:44.860666+0000","last_fullsized":"2024-04-17T08:47:44.860666+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:47.059315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848372+0000","last_change":"2024-04-17T08:47:43.840829+0000","last_active":"2024-04-17T08:47:44.848372+0000","last_peered":"2024-04-17T08:47:44.848372+0000","last_clean":"2024-04-17T08:47:44.848372+0000","last_became_active":"2024-04-17T08:47:43.840526+0000","last_became_peered":"2024-04-17T08:47:43.840526+0000","last_unstale":"2024-04-17T08:47:44.848372+0000","last_undegraded":"2024-04-17T08:47:44.848372+0000","last_fullsized":"2024-04-17T08:47:44.848372+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:29.438541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.963899+0000","last_change":"2024-04-17T08:47:42.819248+0000","last_active":"2024-04-17T08:47:45.963899+0000","last_peered":"2024-04-17T08:47:45.963899+0000","last_clean":"2024-04-17T08:47:45.963899+0000","last_became_active":"2024-04-17T08:47:42.818977+0000","last_became_peered":"2024-04-17T08:47:42.818977+0000","last_unstale":"2024-04-17T08:47:45.963899+0000","last_undegraded":"2024-04-17T08:47:45.963899+0000","last_fullsized":"2024-04-17T08:47:45.963899+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:47:18.357684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860305+0000","last_change":"2024-04-17T08:47:43.833701+0000","last_active":"2024-04-17T08:47:44.860305+0000","last_peered":"2024-04-17T08:47:44.860305+0000","last_clean":"2024-04-17T08:47:44.860305+0000","last_became_active":"2024-04-17T08:47:43.829594+0000","last_became_peered":"2024-04-17T08:47:43.829594+0000","last_unstale":"2024-04-17T08:47:44.860305+0000","last_undegraded":"2024-04-17T08:47:44.860305+0000","last_fullsized":"2024-04-17T08:47:44.860305+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:53.462519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"211'8","reported_seq":22,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965340+0000","last_change":"2024-04-17T08:47:42.821650+0000","last_active":"2024-04-17T08:47:45.965340+0000","last_peered":"2024-04-17T08:47:45.965340+0000","last_clean":"2024-04-17T08:47:45.965340+0000","last_became_active":"2024-04-17T08:47:42.821122+0000","last_became_peered":"2024-04-17T08:47:42.821122+0000","last_unstale":"2024-04-17T08:47:45.965340+0000","last_undegraded":"2024-04-17T08:47:45.965340+0000","last_fullsized":"2024-04-17T08:47:45.965340+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:43.316088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861170+0000","last_change":"2024-04-17T08:47:43.830514+0000","last_active":"2024-04-17T08:47:44.861170+0000","last_peered":"2024-04-17T08:47:44.861170+0000","last_clean":"2024-04-17T08:47:44.861170+0000","last_became_active":"2024-04-17T08:47:43.829670+0000","last_became_peered":"2024-04-17T08:47:43.829670+0000","last_unstale":"2024-04-17T08:47:44.861170+0000","last_undegraded":"2024-04-17T08:47:44.861170+0000","last_fullsized":"2024-04-17T08:47:44.861170+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:43:11.314195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966188+0000","last_change":"2024-04-17T08:47:42.814185+0000","last_active":"2024-04-17T08:47:45.966188+0000","last_peered":"2024-04-17T08:47:45.966188+0000","last_clean":"2024-04-17T08:47:45.966188+0000","last_became_active":"2024-04-17T08:47:42.813673+0000","last_became_peered":"2024-04-17T08:47:42.813673+0000","last_unstale":"2024-04-17T08:47:45.966188+0000","last_undegraded":"2024-04-17T08:47:45.966188+0000","last_fullsized":"2024-04-17T08:47:45.966188+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:30.746785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849640+0000","last_change":"2024-04-17T08:47:43.841100+0000","last_active":"2024-04-17T08:47:44.849640+0000","last_peered":"2024-04-17T08:47:44.849640+0000","last_clean":"2024-04-17T08:47:44.849640+0000","last_became_active":"2024-04-17T08:47:43.840812+0000","last_became_peered":"2024-04-17T08:47:43.840812+0000","last_unstale":"2024-04-17T08:47:44.849640+0000","last_undegraded":"2024-04-17T08:47:44.849640+0000","last_fullsized":"2024-04-17T08:47:44.849640+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:28.771576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964948+0000","last_change":"2024-04-17T08:47:42.806681+0000","last_active":"2024-04-17T08:47:45.964948+0000","last_peered":"2024-04-17T08:47:45.964948+0000","last_clean":"2024-04-17T08:47:45.964948+0000","last_became_active":"2024-04-17T08:47:42.805677+0000","last_became_peered":"2024-04-17T08:47:42.805677+0000","last_unstale":"2024-04-17T08:47:45.964948+0000","last_undegraded":"2024-04-17T08:47:45.964948+0000","last_fullsized":"2024-04-17T08:47:45.964948+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:18.113143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848947+0000","last_change":"2024-04-17T08:47:43.837785+0000","last_active":"2024-04-17T08:47:44.848947+0000","last_peered":"2024-04-17T08:47:44.848947+0000","last_clean":"2024-04-17T08:47:44.848947+0000","last_became_active":"2024-04-17T08:47:43.837507+0000","last_became_peered":"2024-04-17T08:47:43.837507+0000","last_unstale":"2024-04-17T08:47:44.848947+0000","last_undegraded":"2024-04-17T08:47:44.848947+0000","last_fullsized":"2024-04-17T08:47:44.848947+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:05.425104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966382+0000","last_change":"2024-04-17T08:47:42.821659+0000","last_active":"2024-04-17T08:47:45.966382+0000","last_peered":"2024-04-17T08:47:45.966382+0000","last_clean":"2024-04-17T08:47:45.966382+0000","last_became_active":"2024-04-17T08:47:42.821150+0000","last_became_peered":"2024-04-17T08:47:42.821150+0000","last_unstale":"2024-04-17T08:47:45.966382+0000","last_undegraded":"2024-04-17T08:47:45.966382+0000","last_fullsized":"2024-04-17T08:47:45.966382+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:28.788973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968433+0000","last_change":"2024-04-17T08:47:43.845043+0000","last_active":"2024-04-17T08:47:45.968433+0000","last_peered":"2024-04-17T08:47:45.968433+0000","last_clean":"2024-04-17T08:47:45.968433+0000","last_became_active":"2024-04-17T08:47:43.844783+0000","last_became_peered":"2024-04-17T08:47:43.844783+0000","last_unstale":"2024-04-17T08:47:45.968433+0000","last_undegraded":"2024-04-17T08:47:45.968433+0000","last_fullsized":"2024-04-17T08:47:45.968433+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:02.324702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850255+0000","last_change":"2024-04-17T08:47:42.810555+0000","last_active":"2024-04-17T08:47:44.850255+0000","last_peered":"2024-04-17T08:47:44.850255+0000","last_clean":"2024-04-17T08:47:44.850255+0000","last_became_active":"2024-04-17T08:47:42.808744+0000","last_became_peered":"2024-04-17T08:47:42.808744+0000","last_unstale":"2024-04-17T08:47:44.850255+0000","last_undegraded":"2024-04-17T08:47:44.850255+0000","last_fullsized":"2024-04-17T08:47:44.850255+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:17:04.555542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860770+0000","last_change":"2024-04-17T08:47:43.844713+0000","last_active":"2024-04-17T08:47:44.860770+0000","last_peered":"2024-04-17T08:47:44.860770+0000","last_clean":"2024-04-17T08:47:44.860770+0000","last_became_active":"2024-04-17T08:47:43.843988+0000","last_became_peered":"2024-04-17T08:47:43.843988+0000","last_unstale":"2024-04-17T08:47:44.860770+0000","last_undegraded":"2024-04-17T08:47:44.860770+0000","last_fullsized":"2024-04-17T08:47:44.860770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.934805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842587+0000","last_change":"2024-04-17T08:47:42.823486+0000","last_active":"2024-04-17T08:47:44.842587+0000","last_peered":"2024-04-17T08:47:44.842587+0000","last_clean":"2024-04-17T08:47:44.842587+0000","last_became_active":"2024-04-17T08:47:42.822869+0000","last_became_peered":"2024-04-17T08:47:42.822869+0000","last_unstale":"2024-04-17T08:47:44.842587+0000","last_undegraded":"2024-04-17T08:47:44.842587+0000","last_fullsized":"2024-04-17T08:47:44.842587+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.747666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849825+0000","last_change":"2024-04-17T08:47:43.847071+0000","last_active":"2024-04-17T08:47:44.849825+0000","last_peered":"2024-04-17T08:47:44.849825+0000","last_clean":"2024-04-17T08:47:44.849825+0000","last_became_active":"2024-04-17T08:47:43.846785+0000","last_became_peered":"2024-04-17T08:47:43.846785+0000","last_unstale":"2024-04-17T08:47:44.849825+0000","last_undegraded":"2024-04-17T08:47:44.849825+0000","last_fullsized":"2024-04-17T08:47:44.849825+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:06:34.129966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966101+0000","last_change":"2024-04-17T08:47:42.819774+0000","last_active":"2024-04-17T08:47:45.966101+0000","last_peered":"2024-04-17T08:47:45.966101+0000","last_clean":"2024-04-17T08:47:45.966101+0000","last_became_active":"2024-04-17T08:47:42.819464+0000","last_became_peered":"2024-04-17T08:47:42.819464+0000","last_unstale":"2024-04-17T08:47:45.966101+0000","last_undegraded":"2024-04-17T08:47:45.966101+0000","last_fullsized":"2024-04-17T08:47:45.966101+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:45.564858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966325+0000","last_change":"2024-04-17T08:47:43.847533+0000","last_active":"2024-04-17T08:47:45.966325+0000","last_peered":"2024-04-17T08:47:45.966325+0000","last_clean":"2024-04-17T08:47:45.966325+0000","last_became_active":"2024-04-17T08:47:43.847267+0000","last_became_peered":"2024-04-17T08:47:43.847267+0000","last_unstale":"2024-04-17T08:47:45.966325+0000","last_undegraded":"2024-04-17T08:47:45.966325+0000","last_fullsized":"2024-04-17T08:47:45.966325+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:57.075429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843023+0000","last_change":"2024-04-17T08:47:42.819212+0000","last_active":"2024-04-17T08:47:44.843023+0000","last_peered":"2024-04-17T08:47:44.843023+0000","last_clean":"2024-04-17T08:47:44.843023+0000","last_became_active":"2024-04-17T08:47:42.818642+0000","last_became_peered":"2024-04-17T08:47:42.818642+0000","last_unstale":"2024-04-17T08:47:44.843023+0000","last_undegraded":"2024-04-17T08:47:44.843023+0000","last_fullsized":"2024-04-17T08:47:44.843023+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:28.425040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965392+0000","last_change":"2024-04-17T08:47:43.847550+0000","last_active":"2024-04-17T08:47:45.965392+0000","last_peered":"2024-04-17T08:47:45.965392+0000","last_clean":"2024-04-17T08:47:45.965392+0000","last_became_active":"2024-04-17T08:47:43.847214+0000","last_became_peered":"2024-04-17T08:47:43.847214+0000","last_unstale":"2024-04-17T08:47:45.965392+0000","last_undegraded":"2024-04-17T08:47:45.965392+0000","last_fullsized":"2024-04-17T08:47:45.965392+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:16.817223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850873+0000","last_change":"2024-04-17T08:47:42.819212+0000","last_active":"2024-04-17T08:47:44.850873+0000","last_peered":"2024-04-17T08:47:44.850873+0000","last_clean":"2024-04-17T08:47:44.850873+0000","last_became_active":"2024-04-17T08:47:42.818890+0000","last_became_peered":"2024-04-17T08:47:42.818890+0000","last_unstale":"2024-04-17T08:47:44.850873+0000","last_undegraded":"2024-04-17T08:47:44.850873+0000","last_fullsized":"2024-04-17T08:47:44.850873+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:20.401612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966436+0000","last_change":"2024-04-17T08:47:43.851605+0000","last_active":"2024-04-17T08:47:45.966436+0000","last_peered":"2024-04-17T08:47:45.966436+0000","last_clean":"2024-04-17T08:47:45.966436+0000","last_became_active":"2024-04-17T08:47:43.851258+0000","last_became_peered":"2024-04-17T08:47:43.851258+0000","last_unstale":"2024-04-17T08:47:45.966436+0000","last_undegraded":"2024-04-17T08:47:45.966436+0000","last_fullsized":"2024-04-17T08:47:45.966436+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:34.252594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966558+0000","last_change":"2024-04-17T08:47:42.821818+0000","last_active":"2024-04-17T08:47:45.966558+0000","last_peered":"2024-04-17T08:47:45.966558+0000","last_clean":"2024-04-17T08:47:45.966558+0000","last_became_active":"2024-04-17T08:47:42.821354+0000","last_became_peered":"2024-04-17T08:47:42.821354+0000","last_unstale":"2024-04-17T08:47:45.966558+0000","last_undegraded":"2024-04-17T08:47:45.966558+0000","last_fullsized":"2024-04-17T08:47:45.966558+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:09.804299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861545+0000","last_change":"2024-04-17T08:47:43.842711+0000","last_active":"2024-04-17T08:47:44.861545+0000","last_peered":"2024-04-17T08:47:44.861545+0000","last_clean":"2024-04-17T08:47:44.861545+0000","last_became_active":"2024-04-17T08:47:43.841929+0000","last_became_peered":"2024-04-17T08:47:43.841929+0000","last_unstale":"2024-04-17T08:47:44.861545+0000","last_undegraded":"2024-04-17T08:47:44.861545+0000","last_fullsized":"2024-04-17T08:47:44.861545+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:49.372829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"212'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.863562+0000","last_change":"2024-04-17T08:47:42.819460+0000","last_active":"2024-04-17T08:47:44.863562+0000","last_peered":"2024-04-17T08:47:44.863562+0000","last_clean":"2024-04-17T08:47:44.863562+0000","last_became_active":"2024-04-17T08:47:42.816560+0000","last_became_peered":"2024-04-17T08:47:42.816560+0000","last_unstale":"2024-04-17T08:47:44.863562+0000","last_undegraded":"2024-04-17T08:47:44.863562+0000","last_fullsized":"2024-04-17T08:47:44.863562+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:32.501537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964846+0000","last_change":"2024-04-17T08:47:43.849159+0000","last_active":"2024-04-17T08:47:45.964846+0000","last_peered":"2024-04-17T08:47:45.964846+0000","last_clean":"2024-04-17T08:47:45.964846+0000","last_became_active":"2024-04-17T08:47:43.845427+0000","last_became_peered":"2024-04-17T08:47:43.845427+0000","last_unstale":"2024-04-17T08:47:45.964846+0000","last_undegraded":"2024-04-17T08:47:45.964846+0000","last_fullsized":"2024-04-17T08:47:45.964846+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:00.999483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860919+0000","last_change":"2024-04-17T08:47:42.818676+0000","last_active":"2024-04-17T08:47:44.860919+0000","last_peered":"2024-04-17T08:47:44.860919+0000","last_clean":"2024-04-17T08:47:44.860919+0000","last_became_active":"2024-04-17T08:47:42.816807+0000","last_became_peered":"2024-04-17T08:47:42.816807+0000","last_unstale":"2024-04-17T08:47:44.860919+0000","last_undegraded":"2024-04-17T08:47:44.860919+0000","last_fullsized":"2024-04-17T08:47:44.860919+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:50.255072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965531+0000","last_change":"2024-04-17T08:47:43.852436+0000","last_active":"2024-04-17T08:47:45.965531+0000","last_peered":"2024-04-17T08:47:45.965531+0000","last_clean":"2024-04-17T08:47:45.965531+0000","last_became_active":"2024-04-17T08:47:43.850247+0000","last_became_peered":"2024-04-17T08:47:43.850247+0000","last_unstale":"2024-04-17T08:47:45.965531+0000","last_undegraded":"2024-04-17T08:47:45.965531+0000","last_fullsized":"2024-04-17T08:47:45.965531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:52.331396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964738+0000","last_change":"2024-04-17T08:47:42.802811+0000","last_active":"2024-04-17T08:47:45.964738+0000","last_peered":"2024-04-17T08:47:45.964738+0000","last_clean":"2024-04-17T08:47:45.964738+0000","last_became_active":"2024-04-17T08:47:42.802428+0000","last_became_peered":"2024-04-17T08:47:42.802428+0000","last_unstale":"2024-04-17T08:47:45.964738+0000","last_undegraded":"2024-04-17T08:47:45.964738+0000","last_fullsized":"2024-04-17T08:47:45.964738+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:07.507003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965795+0000","last_change":"2024-04-17T08:47:43.846971+0000","last_active":"2024-04-17T08:47:45.965795+0000","last_peered":"2024-04-17T08:47:45.965795+0000","last_clean":"2024-04-17T08:47:45.965795+0000","last_became_active":"2024-04-17T08:47:43.846355+0000","last_became_peered":"2024-04-17T08:47:43.846355+0000","last_unstale":"2024-04-17T08:47:45.965795+0000","last_undegraded":"2024-04-17T08:47:45.965795+0000","last_fullsized":"2024-04-17T08:47:45.965795+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:56.967670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850726+0000","last_change":"2024-04-17T08:47:42.809920+0000","last_active":"2024-04-17T08:47:44.850726+0000","last_peered":"2024-04-17T08:47:44.850726+0000","last_clean":"2024-04-17T08:47:44.850726+0000","last_became_active":"2024-04-17T08:47:42.809138+0000","last_became_peered":"2024-04-17T08:47:42.809138+0000","last_unstale":"2024-04-17T08:47:44.850726+0000","last_undegraded":"2024-04-17T08:47:44.850726+0000","last_fullsized":"2024-04-17T08:47:44.850726+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:35.570254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965619+0000","last_change":"2024-04-17T08:47:43.851022+0000","last_active":"2024-04-17T08:47:45.965619+0000","last_peered":"2024-04-17T08:47:45.965619+0000","last_clean":"2024-04-17T08:47:45.965619+0000","last_became_active":"2024-04-17T08:47:43.849991+0000","last_became_peered":"2024-04-17T08:47:43.849991+0000","last_unstale":"2024-04-17T08:47:45.965619+0000","last_undegraded":"2024-04-17T08:47:45.965619+0000","last_fullsized":"2024-04-17T08:47:45.965619+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:29.053465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849590+0000","last_change":"2024-04-17T08:47:42.810444+0000","last_active":"2024-04-17T08:47:44.849590+0000","last_peered":"2024-04-17T08:47:44.849590+0000","last_clean":"2024-04-17T08:47:44.849590+0000","last_became_active":"2024-04-17T08:47:42.810104+0000","last_became_peered":"2024-04-17T08:47:42.810104+0000","last_unstale":"2024-04-17T08:47:44.849590+0000","last_undegraded":"2024-04-17T08:47:44.849590+0000","last_fullsized":"2024-04-17T08:47:44.849590+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:41.770362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966796+0000","last_change":"2024-04-17T08:47:43.846925+0000","last_active":"2024-04-17T08:47:45.966796+0000","last_peered":"2024-04-17T08:47:45.966796+0000","last_clean":"2024-04-17T08:47:45.966796+0000","last_became_active":"2024-04-17T08:47:43.846665+0000","last_became_peered":"2024-04-17T08:47:43.846665+0000","last_unstale":"2024-04-17T08:47:45.966796+0000","last_undegraded":"2024-04-17T08:47:45.966796+0000","last_fullsized":"2024-04-17T08:47:45.966796+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:14.224403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967814+0000","last_change":"2024-04-17T08:47:42.816009+0000","last_active":"2024-04-17T08:47:45.967814+0000","last_peered":"2024-04-17T08:47:45.967814+0000","last_clean":"2024-04-17T08:47:45.967814+0000","last_became_active":"2024-04-17T08:47:42.815517+0000","last_became_peered":"2024-04-17T08:47:42.815517+0000","last_unstale":"2024-04-17T08:47:45.967814+0000","last_undegraded":"2024-04-17T08:47:45.967814+0000","last_fullsized":"2024-04-17T08:47:45.967814+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:33:50.251489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965759+0000","last_change":"2024-04-17T08:47:43.852122+0000","last_active":"2024-04-17T08:47:45.965759+0000","last_peered":"2024-04-17T08:47:45.965759+0000","last_clean":"2024-04-17T08:47:45.965759+0000","last_became_active":"2024-04-17T08:47:43.851799+0000","last_became_peered":"2024-04-17T08:47:43.851799+0000","last_unstale":"2024-04-17T08:47:45.965759+0000","last_undegraded":"2024-04-17T08:47:45.965759+0000","last_fullsized":"2024-04-17T08:47:45.965759+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:48.705154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842341+0000","last_change":"2024-04-17T08:47:42.815922+0000","last_active":"2024-04-17T08:47:44.842341+0000","last_peered":"2024-04-17T08:47:44.842341+0000","last_clean":"2024-04-17T08:47:44.842341+0000","last_became_active":"2024-04-17T08:47:42.815619+0000","last_became_peered":"2024-04-17T08:47:42.815619+0000","last_unstale":"2024-04-17T08:47:44.842341+0000","last_undegraded":"2024-04-17T08:47:44.842341+0000","last_fullsized":"2024-04-17T08:47:44.842341+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:17.163964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860304+0000","last_change":"2024-04-17T08:47:43.840571+0000","last_active":"2024-04-17T08:47:44.860304+0000","last_peered":"2024-04-17T08:47:44.860304+0000","last_clean":"2024-04-17T08:47:44.860304+0000","last_became_active":"2024-04-17T08:47:43.840254+0000","last_became_peered":"2024-04-17T08:47:43.840254+0000","last_unstale":"2024-04-17T08:47:44.860304+0000","last_undegraded":"2024-04-17T08:47:44.860304+0000","last_fullsized":"2024-04-17T08:47:44.860304+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:55.854257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966113+0000","last_change":"2024-04-17T08:47:42.820712+0000","last_active":"2024-04-17T08:47:45.966113+0000","last_peered":"2024-04-17T08:47:45.966113+0000","last_clean":"2024-04-17T08:47:45.966113+0000","last_became_active":"2024-04-17T08:47:42.820393+0000","last_became_peered":"2024-04-17T08:47:42.820393+0000","last_unstale":"2024-04-17T08:47:45.966113+0000","last_undegraded":"2024-04-17T08:47:45.966113+0000","last_fullsized":"2024-04-17T08:47:45.966113+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:45.405730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"200'332","reported_seq":1333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964774+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:47:45.964774+0000","last_peered":"2024-04-17T08:47:45.964774+0000","last_clean":"2024-04-17T08:47:45.964774+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:47:45.964774+0000","last_undegraded":"2024-04-17T08:47:45.964774+0000","last_fullsized":"2024-04-17T08:47:45.964774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":547,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":547,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543261,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52404,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319436,"statfs":{"total":96636764160,"available":96583102464,"internally_reserved":0,"allocated":3252224,"data_stored":1883778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42080,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329760,"statfs":{"total":96636764160,"available":96593674240,"internally_reserved":0,"allocated":2637824,"data_stored":1286978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29612,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342228,"statfs":{"total":96636764160,"available":96606441472,"internally_reserved":0,"allocated":2650112,"data_stored":1293306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60152,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311688,"statfs":{"total":96636764160,"available":96575168512,"internally_reserved":0,"allocated":2666496,"data_stored":1295500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47664,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324176,"statfs":{"total":96636764160,"available":96587956224,"internally_reserved":0,"allocated":2650112,"data_stored":1287172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47660,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324180,"statfs":{"total":96636764160,"available":96587960320,"internally_reserved":0,"allocated":2650112,"data_stored":1287672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543262,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60152,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311688,"statfs":{"total":96636764160,"available":96575168512,"internally_reserved":0,"allocated":2662400,"data_stored":1294918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60720,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311120,"statfs":{"total":96636764160,"available":96574586880,"internally_reserved":0,"allocated":3248128,"data_stored":1883696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:47:48.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:47:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.649+0000 7f93b6600640 1 Processor -- start 2024-04-17T08:47:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.649+0000 7f93b6600640 1 -- start start 2024-04-17T08:47:48.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.650+0000 7f93b6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93b006b8c0 0x7f93b00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.650+0000 7f93b6600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f93b0160280 0x7f93b006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.650+0000 7f93b6600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b00cd4a0 con 0x7f93b0141ed0 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.650+0000 7f93b6600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f93b015ce30 con 0x7f93b006b8c0 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.650+0000 7f93b6600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f93b01607f0 con 0x7f93b0160280 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93b006b8c0 0x7f93b00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93b006b8c0 0x7f93b00cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58286/0 (socket says 172.21.15.67:58286) 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 -- 172.21.15.67:0/2271827334 learned_addr learned my addr 172.21.15.67:0/2271827334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 -- 172.21.15.67:0/2271827334 >> v2:172.21.15.187:3300/0 conn(0x7f93b0160280 msgr2=0x7f93b006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 --2- 172.21.15.67:0/2271827334 >> v2:172.21.15.187:3300/0 conn(0x7f93b0160280 0x7f93b006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 -- 172.21.15.67:0/2271827334 >> v1:172.21.15.187:6789/0 conn(0x7f93b0141ed0 legacy=0x7f93b015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.651+0000 7f93b5200640 1 -- 172.21.15.67:0/2271827334 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b0160ad0 con 0x7f93b006b8c0 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.652+0000 7f93b5200640 1 --2- 172.21.15.67:0/2271827334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b00cd8b0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f93b00aff50 tx=0x7f93a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6d00339c9e75fc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93af400640 1 -- 172.21.15.67:0/2271827334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a4035020 con 0x7f93b006b8c0 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93af400640 1 -- 172.21.15.67:0/2271827334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a4031870 con 0x7f93b006b8c0 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93af400640 1 -- 172.21.15.67:0/2271827334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a4031c40 con 0x7f93b006b8c0 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 -- 172.21.15.67:0/2271827334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 msgr2=0x7f93b00cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 --2- 172.21.15.67:0/2271827334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b00cd8b0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f93b00aff50 tx=0x7f93a402b910 comp rx=0 tx=0).stop 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 -- 172.21.15.67:0/2271827334 shutdown_connections 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 --2- 172.21.15.67:0/2271827334 >> v2:172.21.15.187:3300/0 conn(0x7f93b0160280 0x7f93b006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 --2- 172.21.15.67:0/2271827334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b00cd8b0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 -- 172.21.15.67:0/2271827334 >> 172.21.15.67:0/2271827334 conn(0x7f93b0067d10 msgr2=0x7f93b00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 -- 172.21.15.67:0/2271827334 shutdown_connections 2024-04-17T08:47:48.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.653+0000 7f93b6600640 1 -- 172.21.15.67:0/2271827334 wait complete. 2024-04-17T08:47:48.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.655+0000 7f93b6600640 1 Processor -- start 2024-04-17T08:47:48.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.655+0000 7f93b6600640 1 -- start start 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 0x7f93b013d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 0x7f93b00b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93b006c1e0 con 0x7f93b006b8c0 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93b00cddf0 con 0x7f93b0160280 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.656+0000 7f93b6600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93b0161090 con 0x7f93b0141ed0 2024-04-17T08:47:48.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58296/0 (socket says 172.21.15.67:58296) 2024-04-17T08:47:48.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 -- 172.21.15.67:0/739751279 learned_addr learned my addr 172.21.15.67:0/739751279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93afe00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 0x7f93b013d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 -- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 msgr2=0x7f93b013d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5c00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 0x7f93b00b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 0x7f93b013d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 -- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 msgr2=0x7f93b00b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 0x7f93b00b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 -- 172.21.15.67:0/739751279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b0145df0 con 0x7f93b006b8c0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5c00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 0x7f93b00b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93afe00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 0x7f93b013d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.657+0000 7f93b5200640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f93a4000c00 tx=0x7f93a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.658+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a403ad60 con 0x7f93b006b8c0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.658+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a40443f0 con 0x7f93b006b8c0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.658+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a40447e0 con 0x7f93b006b8c0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.658+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b01460d0 con 0x7f93b006b8c0 2024-04-17T08:47:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.658+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b00b4520 con 0x7f93b006b8c0 2024-04-17T08:47:48.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.659+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b0068a10 con 0x7f93b006b8c0 2024-04-17T08:47:48.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.661+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f93a404b060 con 0x7f93b006b8c0 2024-04-17T08:47:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.661+0000 7f93acc00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 0x7f939006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.662+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f93a40ae090 con 0x7f93b006b8c0 2024-04-17T08:47:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.662+0000 7f93afe00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 0x7f939006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:48.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.663+0000 7f93afe00640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 0x7f939006ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f93b006dff0 tx=0x7f939c008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.667+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f93a403f070 con 0x7f93b006b8c0 2024-04-17T08:47:48.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.869+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 --> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93b00cc740 con 0x7f939006ca40 2024-04-17T08:47:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.891+0000 7f93acc00640 1 -- 172.21.15.67:0/739751279 <== mgr.7379 v2:172.21.15.187:6832/2105214300 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252497 (secure 0 0 0) 0x7f93b00cc740 con 0x7f939006ca40 2024-04-17T08:47:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:48.894 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.896+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 msgr2=0x7f939006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.896+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 0x7f939006ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f93b006dff0 tx=0x7f939c008040 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.896+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 msgr2=0x7f93b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.896+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f93a4000c00 tx=0x7f93a40046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 shutdown_connections 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f939006ca40 0x7f939006ef00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b0160280 0x7f93b00b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93b0141ed0 0x7f93b013d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 --2- 172.21.15.67:0/739751279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93b006b8c0 0x7f93b006ede0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 >> 172.21.15.67:0/739751279 conn(0x7f93b0067d10 msgr2=0x7f93b00b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 shutdown_connections 2024-04-17T08:47:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:48.897+0000 7f93b6600640 1 -- 172.21.15.67:0/739751279 wait complete. 2024-04-17T08:47:48.916 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T08:47:47.179028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":592,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":371,"ondisk_log_size":371,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400444,"kb_used_data":21892,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754574276,"statfs":{"total":773094113280,"available":772684058624,"internally_reserved":0,"allocated":22417408,"data_stored":11513020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":387501616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.402555"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966585+0000","last_change":"2024-04-17T08:47:43.853279+0000","last_active":"2024-04-17T08:47:45.966585+0000","last_peered":"2024-04-17T08:47:45.966585+0000","last_clean":"2024-04-17T08:47:45.966585+0000","last_became_active":"2024-04-17T08:47:43.852929+0000","last_became_peered":"2024-04-17T08:47:43.852929+0000","last_unstale":"2024-04-17T08:47:45.966585+0000","last_undegraded":"2024-04-17T08:47:45.966585+0000","last_fullsized":"2024-04-17T08:47:45.966585+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:47.546488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967871+0000","last_change":"2024-04-17T08:47:43.844037+0000","last_active":"2024-04-17T08:47:45.967871+0000","last_peered":"2024-04-17T08:47:45.967871+0000","last_clean":"2024-04-17T08:47:45.967871+0000","last_became_active":"2024-04-17T08:47:43.843776+0000","last_became_peered":"2024-04-17T08:47:43.843776+0000","last_unstale":"2024-04-17T08:47:45.967871+0000","last_undegraded":"2024-04-17T08:47:45.967871+0000","last_fullsized":"2024-04-17T08:47:45.967871+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:16:17.623695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843305+0000","last_change":"2024-04-17T08:47:43.854189+0000","last_active":"2024-04-17T08:47:44.843305+0000","last_peered":"2024-04-17T08:47:44.843305+0000","last_clean":"2024-04-17T08:47:44.843305+0000","last_became_active":"2024-04-17T08:47:43.852791+0000","last_became_peered":"2024-04-17T08:47:43.852791+0000","last_unstale":"2024-04-17T08:47:44.843305+0000","last_undegraded":"2024-04-17T08:47:44.843305+0000","last_fullsized":"2024-04-17T08:47:44.843305+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:44:28.530312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967796+0000","last_change":"2024-04-17T08:47:43.845515+0000","last_active":"2024-04-17T08:47:45.967796+0000","last_peered":"2024-04-17T08:47:45.967796+0000","last_clean":"2024-04-17T08:47:45.967796+0000","last_became_active":"2024-04-17T08:47:43.844523+0000","last_became_peered":"2024-04-17T08:47:43.844523+0000","last_unstale":"2024-04-17T08:47:45.967796+0000","last_undegraded":"2024-04-17T08:47:45.967796+0000","last_fullsized":"2024-04-17T08:47:45.967796+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:16.247078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965906+0000","last_change":"2024-04-17T08:47:43.854577+0000","last_active":"2024-04-17T08:47:45.965906+0000","last_peered":"2024-04-17T08:47:45.965906+0000","last_clean":"2024-04-17T08:47:45.965906+0000","last_became_active":"2024-04-17T08:47:43.854235+0000","last_became_peered":"2024-04-17T08:47:43.854235+0000","last_unstale":"2024-04-17T08:47:45.965906+0000","last_undegraded":"2024-04-17T08:47:45.965906+0000","last_fullsized":"2024-04-17T08:47:45.965906+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:29.954534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843042+0000","last_change":"2024-04-17T08:47:43.850791+0000","last_active":"2024-04-17T08:47:44.843042+0000","last_peered":"2024-04-17T08:47:44.843042+0000","last_clean":"2024-04-17T08:47:44.843042+0000","last_became_active":"2024-04-17T08:47:43.850524+0000","last_became_peered":"2024-04-17T08:47:43.850524+0000","last_unstale":"2024-04-17T08:47:44.843042+0000","last_undegraded":"2024-04-17T08:47:44.843042+0000","last_fullsized":"2024-04-17T08:47:44.843042+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:37:12.297226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.847405+0000","last_change":"2024-04-17T08:47:43.843029+0000","last_active":"2024-04-17T08:47:44.847405+0000","last_peered":"2024-04-17T08:47:44.847405+0000","last_clean":"2024-04-17T08:47:44.847405+0000","last_became_active":"2024-04-17T08:47:43.842735+0000","last_became_peered":"2024-04-17T08:47:43.842735+0000","last_unstale":"2024-04-17T08:47:44.847405+0000","last_undegraded":"2024-04-17T08:47:44.847405+0000","last_fullsized":"2024-04-17T08:47:44.847405+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:28.771576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964097+0000","last_change":"2024-04-17T08:47:43.846810+0000","last_active":"2024-04-17T08:47:45.964097+0000","last_peered":"2024-04-17T08:47:45.964097+0000","last_clean":"2024-04-17T08:47:45.964097+0000","last_became_active":"2024-04-17T08:47:43.846478+0000","last_became_peered":"2024-04-17T08:47:43.846478+0000","last_unstale":"2024-04-17T08:47:45.964097+0000","last_undegraded":"2024-04-17T08:47:45.964097+0000","last_fullsized":"2024-04-17T08:47:45.964097+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:24:01.526257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849235+0000","last_change":"2024-04-17T08:47:43.839332+0000","last_active":"2024-04-17T08:47:44.849235+0000","last_peered":"2024-04-17T08:47:44.849235+0000","last_clean":"2024-04-17T08:47:44.849235+0000","last_became_active":"2024-04-17T08:47:43.838943+0000","last_became_peered":"2024-04-17T08:47:43.838943+0000","last_unstale":"2024-04-17T08:47:44.849235+0000","last_undegraded":"2024-04-17T08:47:44.849235+0000","last_fullsized":"2024-04-17T08:47:44.849235+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:09.970222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860932+0000","last_change":"2024-04-17T08:47:43.852299+0000","last_active":"2024-04-17T08:47:44.860932+0000","last_peered":"2024-04-17T08:47:44.860932+0000","last_clean":"2024-04-17T08:47:44.860932+0000","last_became_active":"2024-04-17T08:47:43.851087+0000","last_became_peered":"2024-04-17T08:47:43.851087+0000","last_unstale":"2024-04-17T08:47:44.860932+0000","last_undegraded":"2024-04-17T08:47:44.860932+0000","last_fullsized":"2024-04-17T08:47:44.860932+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:44:25.624806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848161+0000","last_change":"2024-04-17T08:47:43.834717+0000","last_active":"2024-04-17T08:47:44.848161+0000","last_peered":"2024-04-17T08:47:44.848161+0000","last_clean":"2024-04-17T08:47:44.848161+0000","last_became_active":"2024-04-17T08:47:43.834115+0000","last_became_peered":"2024-04-17T08:47:43.834115+0000","last_unstale":"2024-04-17T08:47:44.848161+0000","last_undegraded":"2024-04-17T08:47:44.848161+0000","last_fullsized":"2024-04-17T08:47:44.848161+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:24.640878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968217+0000","last_change":"2024-04-17T08:47:43.847876+0000","last_active":"2024-04-17T08:47:45.968217+0000","last_peered":"2024-04-17T08:47:45.968217+0000","last_clean":"2024-04-17T08:47:45.968217+0000","last_became_active":"2024-04-17T08:47:43.847521+0000","last_became_peered":"2024-04-17T08:47:43.847521+0000","last_unstale":"2024-04-17T08:47:45.968217+0000","last_undegraded":"2024-04-17T08:47:45.968217+0000","last_fullsized":"2024-04-17T08:47:45.968217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:05.032689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967679+0000","last_change":"2024-04-17T08:47:43.843660+0000","last_active":"2024-04-17T08:47:45.967679+0000","last_peered":"2024-04-17T08:47:45.967679+0000","last_clean":"2024-04-17T08:47:45.967679+0000","last_became_active":"2024-04-17T08:47:43.843371+0000","last_became_peered":"2024-04-17T08:47:43.843371+0000","last_unstale":"2024-04-17T08:47:45.967679+0000","last_undegraded":"2024-04-17T08:47:45.967679+0000","last_fullsized":"2024-04-17T08:47:45.967679+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:45.521002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849941+0000","last_change":"2024-04-17T08:47:43.847563+0000","last_active":"2024-04-17T08:47:44.849941+0000","last_peered":"2024-04-17T08:47:44.849941+0000","last_clean":"2024-04-17T08:47:44.849941+0000","last_became_active":"2024-04-17T08:47:43.847239+0000","last_became_peered":"2024-04-17T08:47:43.847239+0000","last_unstale":"2024-04-17T08:47:44.849941+0000","last_undegraded":"2024-04-17T08:47:44.849941+0000","last_fullsized":"2024-04-17T08:47:44.849941+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:21:55.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967766+0000","last_change":"2024-04-17T08:47:43.849404+0000","last_active":"2024-04-17T08:47:45.967766+0000","last_peered":"2024-04-17T08:47:45.967766+0000","last_clean":"2024-04-17T08:47:45.967766+0000","last_became_active":"2024-04-17T08:47:43.848895+0000","last_became_peered":"2024-04-17T08:47:43.848895+0000","last_unstale":"2024-04-17T08:47:45.967766+0000","last_undegraded":"2024-04-17T08:47:45.967766+0000","last_fullsized":"2024-04-17T08:47:45.967766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:08.366575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965032+0000","last_change":"2024-04-17T08:47:43.856933+0000","last_active":"2024-04-17T08:47:45.965032+0000","last_peered":"2024-04-17T08:47:45.965032+0000","last_clean":"2024-04-17T08:47:45.965032+0000","last_became_active":"2024-04-17T08:47:43.856580+0000","last_became_peered":"2024-04-17T08:47:43.856580+0000","last_unstale":"2024-04-17T08:47:45.965032+0000","last_undegraded":"2024-04-17T08:47:45.965032+0000","last_fullsized":"2024-04-17T08:47:45.965032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:13.389562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966486+0000","last_change":"2024-04-17T08:47:43.855856+0000","last_active":"2024-04-17T08:47:45.966486+0000","last_peered":"2024-04-17T08:47:45.966486+0000","last_clean":"2024-04-17T08:47:45.966486+0000","last_became_active":"2024-04-17T08:47:43.853234+0000","last_became_peered":"2024-04-17T08:47:43.853234+0000","last_unstale":"2024-04-17T08:47:45.966486+0000","last_undegraded":"2024-04-17T08:47:45.966486+0000","last_fullsized":"2024-04-17T08:47:45.966486+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:13:25.564151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966180+0000","last_change":"2024-04-17T08:47:43.850753+0000","last_active":"2024-04-17T08:47:45.966180+0000","last_peered":"2024-04-17T08:47:45.966180+0000","last_clean":"2024-04-17T08:47:45.966180+0000","last_became_active":"2024-04-17T08:47:43.850333+0000","last_became_peered":"2024-04-17T08:47:43.850333+0000","last_unstale":"2024-04-17T08:47:45.966180+0000","last_undegraded":"2024-04-17T08:47:45.966180+0000","last_fullsized":"2024-04-17T08:47:45.966180+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:38.547787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860792+0000","last_change":"2024-04-17T08:47:43.837722+0000","last_active":"2024-04-17T08:47:44.860792+0000","last_peered":"2024-04-17T08:47:44.860792+0000","last_clean":"2024-04-17T08:47:44.860792+0000","last_became_active":"2024-04-17T08:47:43.837365+0000","last_became_peered":"2024-04-17T08:47:43.837365+0000","last_unstale":"2024-04-17T08:47:44.860792+0000","last_undegraded":"2024-04-17T08:47:44.860792+0000","last_fullsized":"2024-04-17T08:47:44.860792+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:51.373438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965715+0000","last_change":"2024-04-17T08:47:43.853921+0000","last_active":"2024-04-17T08:47:45.965715+0000","last_peered":"2024-04-17T08:47:45.965715+0000","last_clean":"2024-04-17T08:47:45.965715+0000","last_became_active":"2024-04-17T08:47:43.853359+0000","last_became_peered":"2024-04-17T08:47:43.853359+0000","last_unstale":"2024-04-17T08:47:45.965715+0000","last_undegraded":"2024-04-17T08:47:45.965715+0000","last_fullsized":"2024-04-17T08:47:45.965715+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:42.733226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966099+0000","last_change":"2024-04-17T08:47:43.854643+0000","last_active":"2024-04-17T08:47:45.966099+0000","last_peered":"2024-04-17T08:47:45.966099+0000","last_clean":"2024-04-17T08:47:45.966099+0000","last_became_active":"2024-04-17T08:47:43.854331+0000","last_became_peered":"2024-04-17T08:47:43.854331+0000","last_unstale":"2024-04-17T08:47:45.966099+0000","last_undegraded":"2024-04-17T08:47:45.966099+0000","last_fullsized":"2024-04-17T08:47:45.966099+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:15:11.176172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860332+0000","last_change":"2024-04-17T08:47:43.845007+0000","last_active":"2024-04-17T08:47:44.860332+0000","last_peered":"2024-04-17T08:47:44.860332+0000","last_clean":"2024-04-17T08:47:44.860332+0000","last_became_active":"2024-04-17T08:47:43.844438+0000","last_became_peered":"2024-04-17T08:47:43.844438+0000","last_unstale":"2024-04-17T08:47:44.860332+0000","last_undegraded":"2024-04-17T08:47:44.860332+0000","last_fullsized":"2024-04-17T08:47:44.860332+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:33.785820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843148+0000","last_change":"2024-04-17T08:47:43.859763+0000","last_active":"2024-04-17T08:47:44.843148+0000","last_peered":"2024-04-17T08:47:44.843148+0000","last_clean":"2024-04-17T08:47:44.843148+0000","last_became_active":"2024-04-17T08:47:43.859431+0000","last_became_peered":"2024-04-17T08:47:43.859431+0000","last_unstale":"2024-04-17T08:47:44.843148+0000","last_undegraded":"2024-04-17T08:47:44.843148+0000","last_fullsized":"2024-04-17T08:47:44.843148+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:35.081905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964430+0000","last_change":"2024-04-17T08:47:43.859249+0000","last_active":"2024-04-17T08:47:45.964430+0000","last_peered":"2024-04-17T08:47:45.964430+0000","last_clean":"2024-04-17T08:47:45.964430+0000","last_became_active":"2024-04-17T08:47:43.858962+0000","last_became_peered":"2024-04-17T08:47:43.858962+0000","last_unstale":"2024-04-17T08:47:45.964430+0000","last_undegraded":"2024-04-17T08:47:45.964430+0000","last_fullsized":"2024-04-17T08:47:45.964430+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.047336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968653+0000","last_change":"2024-04-17T08:47:43.853286+0000","last_active":"2024-04-17T08:47:45.968653+0000","last_peered":"2024-04-17T08:47:45.968653+0000","last_clean":"2024-04-17T08:47:45.968653+0000","last_became_active":"2024-04-17T08:47:43.852929+0000","last_became_peered":"2024-04-17T08:47:43.852929+0000","last_unstale":"2024-04-17T08:47:45.968653+0000","last_undegraded":"2024-04-17T08:47:45.968653+0000","last_fullsized":"2024-04-17T08:47:45.968653+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:52.489657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848521+0000","last_change":"2024-04-17T08:47:43.831730+0000","last_active":"2024-04-17T08:47:44.848521+0000","last_peered":"2024-04-17T08:47:44.848521+0000","last_clean":"2024-04-17T08:47:44.848521+0000","last_became_active":"2024-04-17T08:47:43.831153+0000","last_became_peered":"2024-04-17T08:47:43.831153+0000","last_unstale":"2024-04-17T08:47:44.848521+0000","last_undegraded":"2024-04-17T08:47:44.848521+0000","last_fullsized":"2024-04-17T08:47:44.848521+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:43:09.489346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843393+0000","last_change":"2024-04-17T08:47:43.845063+0000","last_active":"2024-04-17T08:47:44.843393+0000","last_peered":"2024-04-17T08:47:44.843393+0000","last_clean":"2024-04-17T08:47:44.843393+0000","last_became_active":"2024-04-17T08:47:43.844147+0000","last_became_peered":"2024-04-17T08:47:43.844147+0000","last_unstale":"2024-04-17T08:47:44.843393+0000","last_undegraded":"2024-04-17T08:47:44.843393+0000","last_fullsized":"2024-04-17T08:47:44.843393+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:22.012868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966023+0000","last_change":"2024-04-17T08:47:43.855551+0000","last_active":"2024-04-17T08:47:45.966023+0000","last_peered":"2024-04-17T08:47:45.966023+0000","last_clean":"2024-04-17T08:47:45.966023+0000","last_became_active":"2024-04-17T08:47:43.855261+0000","last_became_peered":"2024-04-17T08:47:43.855261+0000","last_unstale":"2024-04-17T08:47:45.966023+0000","last_undegraded":"2024-04-17T08:47:45.966023+0000","last_fullsized":"2024-04-17T08:47:45.966023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:59.848987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968075+0000","last_change":"2024-04-17T08:47:43.848415+0000","last_active":"2024-04-17T08:47:45.968075+0000","last_peered":"2024-04-17T08:47:45.968075+0000","last_clean":"2024-04-17T08:47:45.968075+0000","last_became_active":"2024-04-17T08:47:43.847926+0000","last_became_peered":"2024-04-17T08:47:43.847926+0000","last_unstale":"2024-04-17T08:47:45.968075+0000","last_undegraded":"2024-04-17T08:47:45.968075+0000","last_fullsized":"2024-04-17T08:47:45.968075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:19.989335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849327+0000","last_change":"2024-04-17T08:47:43.837262+0000","last_active":"2024-04-17T08:47:44.849327+0000","last_peered":"2024-04-17T08:47:44.849327+0000","last_clean":"2024-04-17T08:47:44.849327+0000","last_became_active":"2024-04-17T08:47:43.836831+0000","last_became_peered":"2024-04-17T08:47:43.836831+0000","last_unstale":"2024-04-17T08:47:44.849327+0000","last_undegraded":"2024-04-17T08:47:44.849327+0000","last_fullsized":"2024-04-17T08:47:44.849327+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:26.885336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842883+0000","last_change":"2024-04-17T08:47:43.856823+0000","last_active":"2024-04-17T08:47:44.842883+0000","last_peered":"2024-04-17T08:47:44.842883+0000","last_clean":"2024-04-17T08:47:44.842883+0000","last_became_active":"2024-04-17T08:47:43.856452+0000","last_became_peered":"2024-04-17T08:47:43.856452+0000","last_unstale":"2024-04-17T08:47:44.842883+0000","last_undegraded":"2024-04-17T08:47:44.842883+0000","last_fullsized":"2024-04-17T08:47:44.842883+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:29.564289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965175+0000","last_change":"2024-04-17T08:47:43.848798+0000","last_active":"2024-04-17T08:47:45.965175+0000","last_peered":"2024-04-17T08:47:45.965175+0000","last_clean":"2024-04-17T08:47:45.965175+0000","last_became_active":"2024-04-17T08:47:43.844231+0000","last_became_peered":"2024-04-17T08:47:43.844231+0000","last_unstale":"2024-04-17T08:47:45.965175+0000","last_undegraded":"2024-04-17T08:47:45.965175+0000","last_fullsized":"2024-04-17T08:47:45.965175+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.959616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965217+0000","last_change":"2024-04-17T08:47:43.851228+0000","last_active":"2024-04-17T08:47:45.965217+0000","last_peered":"2024-04-17T08:47:45.965217+0000","last_clean":"2024-04-17T08:47:45.965217+0000","last_became_active":"2024-04-17T08:47:43.849853+0000","last_became_peered":"2024-04-17T08:47:43.849853+0000","last_unstale":"2024-04-17T08:47:45.965217+0000","last_undegraded":"2024-04-17T08:47:45.965217+0000","last_fullsized":"2024-04-17T08:47:45.965217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:26:56.580759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"211'4","reported_seq":18,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965980+0000","last_change":"2024-04-17T08:47:42.813949+0000","last_active":"2024-04-17T08:47:45.965980+0000","last_peered":"2024-04-17T08:47:45.965980+0000","last_clean":"2024-04-17T08:47:45.965980+0000","last_became_active":"2024-04-17T08:47:42.813312+0000","last_became_peered":"2024-04-17T08:47:42.813312+0000","last_unstale":"2024-04-17T08:47:45.965980+0000","last_undegraded":"2024-04-17T08:47:45.965980+0000","last_fullsized":"2024-04-17T08:47:45.965980+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:41:51.655582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861544+0000","last_change":"2024-04-17T08:47:43.852046+0000","last_active":"2024-04-17T08:47:44.861544+0000","last_peered":"2024-04-17T08:47:44.861544+0000","last_clean":"2024-04-17T08:47:44.861544+0000","last_became_active":"2024-04-17T08:47:43.849718+0000","last_became_peered":"2024-04-17T08:47:43.849718+0000","last_unstale":"2024-04-17T08:47:44.861544+0000","last_undegraded":"2024-04-17T08:47:44.861544+0000","last_fullsized":"2024-04-17T08:47:44.861544+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:54.241566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849147+0000","last_change":"2024-04-17T08:47:42.819004+0000","last_active":"2024-04-17T08:47:44.849147+0000","last_peered":"2024-04-17T08:47:44.849147+0000","last_clean":"2024-04-17T08:47:44.849147+0000","last_became_active":"2024-04-17T08:47:42.818150+0000","last_became_peered":"2024-04-17T08:47:42.818150+0000","last_unstale":"2024-04-17T08:47:44.849147+0000","last_undegraded":"2024-04-17T08:47:44.849147+0000","last_fullsized":"2024-04-17T08:47:44.849147+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:36.111988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848259+0000","last_change":"2024-04-17T08:47:43.844060+0000","last_active":"2024-04-17T08:47:44.848259+0000","last_peered":"2024-04-17T08:47:44.848259+0000","last_clean":"2024-04-17T08:47:44.848259+0000","last_became_active":"2024-04-17T08:47:43.842814+0000","last_became_peered":"2024-04-17T08:47:43.842814+0000","last_unstale":"2024-04-17T08:47:44.848259+0000","last_undegraded":"2024-04-17T08:47:44.848259+0000","last_fullsized":"2024-04-17T08:47:44.848259+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:31.382958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843386+0000","last_change":"2024-04-17T08:47:42.814575+0000","last_active":"2024-04-17T08:47:44.843386+0000","last_peered":"2024-04-17T08:47:44.843386+0000","last_clean":"2024-04-17T08:47:44.843386+0000","last_became_active":"2024-04-17T08:47:42.814291+0000","last_became_peered":"2024-04-17T08:47:42.814291+0000","last_unstale":"2024-04-17T08:47:44.843386+0000","last_undegraded":"2024-04-17T08:47:44.843386+0000","last_fullsized":"2024-04-17T08:47:44.843386+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:08.956721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850609+0000","last_change":"2024-04-17T08:47:43.832507+0000","last_active":"2024-04-17T08:47:44.850609+0000","last_peered":"2024-04-17T08:47:44.850609+0000","last_clean":"2024-04-17T08:47:44.850609+0000","last_became_active":"2024-04-17T08:47:43.832035+0000","last_became_peered":"2024-04-17T08:47:43.832035+0000","last_unstale":"2024-04-17T08:47:44.850609+0000","last_undegraded":"2024-04-17T08:47:44.850609+0000","last_fullsized":"2024-04-17T08:47:44.850609+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:24.640878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968084+0000","last_change":"2024-04-17T08:47:42.819118+0000","last_active":"2024-04-17T08:47:45.968084+0000","last_peered":"2024-04-17T08:47:45.968084+0000","last_clean":"2024-04-17T08:47:45.968084+0000","last_became_active":"2024-04-17T08:47:42.818837+0000","last_became_peered":"2024-04-17T08:47:42.818837+0000","last_unstale":"2024-04-17T08:47:45.968084+0000","last_undegraded":"2024-04-17T08:47:45.968084+0000","last_fullsized":"2024-04-17T08:47:45.968084+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:34:16.764062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860536+0000","last_change":"2024-04-17T08:47:43.847128+0000","last_active":"2024-04-17T08:47:44.860536+0000","last_peered":"2024-04-17T08:47:44.860536+0000","last_clean":"2024-04-17T08:47:44.860536+0000","last_became_active":"2024-04-17T08:47:43.841146+0000","last_became_peered":"2024-04-17T08:47:43.841146+0000","last_unstale":"2024-04-17T08:47:44.860536+0000","last_undegraded":"2024-04-17T08:47:44.860536+0000","last_fullsized":"2024-04-17T08:47:44.860536+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:10.499515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966250+0000","last_change":"2024-04-17T08:47:42.814949+0000","last_active":"2024-04-17T08:47:45.966250+0000","last_peered":"2024-04-17T08:47:45.966250+0000","last_clean":"2024-04-17T08:47:45.966250+0000","last_became_active":"2024-04-17T08:47:42.814215+0000","last_became_peered":"2024-04-17T08:47:42.814215+0000","last_unstale":"2024-04-17T08:47:45.966250+0000","last_undegraded":"2024-04-17T08:47:45.966250+0000","last_fullsized":"2024-04-17T08:47:45.966250+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:06.281860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964724+0000","last_change":"2024-04-17T08:47:43.854761+0000","last_active":"2024-04-17T08:47:45.964724+0000","last_peered":"2024-04-17T08:47:45.964724+0000","last_clean":"2024-04-17T08:47:45.964724+0000","last_became_active":"2024-04-17T08:47:43.854501+0000","last_became_peered":"2024-04-17T08:47:43.854501+0000","last_unstale":"2024-04-17T08:47:45.964724+0000","last_undegraded":"2024-04-17T08:47:45.964724+0000","last_fullsized":"2024-04-17T08:47:45.964724+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:03.781249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964476+0000","last_change":"2024-04-17T08:47:42.816347+0000","last_active":"2024-04-17T08:47:45.964476+0000","last_peered":"2024-04-17T08:47:45.964476+0000","last_clean":"2024-04-17T08:47:45.964476+0000","last_became_active":"2024-04-17T08:47:42.816044+0000","last_became_peered":"2024-04-17T08:47:42.816044+0000","last_unstale":"2024-04-17T08:47:45.964476+0000","last_undegraded":"2024-04-17T08:47:45.964476+0000","last_fullsized":"2024-04-17T08:47:45.964476+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:16:24.607522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.847835+0000","last_change":"2024-04-17T08:47:43.834894+0000","last_active":"2024-04-17T08:47:44.847835+0000","last_peered":"2024-04-17T08:47:44.847835+0000","last_clean":"2024-04-17T08:47:44.847835+0000","last_became_active":"2024-04-17T08:47:43.833320+0000","last_became_peered":"2024-04-17T08:47:43.833320+0000","last_unstale":"2024-04-17T08:47:44.847835+0000","last_undegraded":"2024-04-17T08:47:44.847835+0000","last_fullsized":"2024-04-17T08:47:44.847835+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:21:55.187952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965727+0000","last_change":"2024-04-17T08:47:42.817918+0000","last_active":"2024-04-17T08:47:45.965727+0000","last_peered":"2024-04-17T08:47:45.965727+0000","last_clean":"2024-04-17T08:47:45.965727+0000","last_became_active":"2024-04-17T08:47:42.817551+0000","last_became_peered":"2024-04-17T08:47:42.817551+0000","last_unstale":"2024-04-17T08:47:45.965727+0000","last_undegraded":"2024-04-17T08:47:45.965727+0000","last_fullsized":"2024-04-17T08:47:45.965727+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:52.191803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842894+0000","last_change":"2024-04-17T08:47:43.847468+0000","last_active":"2024-04-17T08:47:44.842894+0000","last_peered":"2024-04-17T08:47:44.842894+0000","last_clean":"2024-04-17T08:47:44.842894+0000","last_became_active":"2024-04-17T08:47:43.847156+0000","last_became_peered":"2024-04-17T08:47:43.847156+0000","last_unstale":"2024-04-17T08:47:44.842894+0000","last_undegraded":"2024-04-17T08:47:44.842894+0000","last_fullsized":"2024-04-17T08:47:44.842894+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:08.979625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849481+0000","last_change":"2024-04-17T08:47:42.810707+0000","last_active":"2024-04-17T08:47:44.849481+0000","last_peered":"2024-04-17T08:47:44.849481+0000","last_clean":"2024-04-17T08:47:44.849481+0000","last_became_active":"2024-04-17T08:47:42.810366+0000","last_became_peered":"2024-04-17T08:47:42.810366+0000","last_unstale":"2024-04-17T08:47:44.849481+0000","last_undegraded":"2024-04-17T08:47:44.849481+0000","last_fullsized":"2024-04-17T08:47:44.849481+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:59:16.773889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964824+0000","last_change":"2024-04-17T08:47:43.853611+0000","last_active":"2024-04-17T08:47:45.964824+0000","last_peered":"2024-04-17T08:47:45.964824+0000","last_clean":"2024-04-17T08:47:45.964824+0000","last_became_active":"2024-04-17T08:47:43.853221+0000","last_became_peered":"2024-04-17T08:47:43.853221+0000","last_unstale":"2024-04-17T08:47:45.964824+0000","last_undegraded":"2024-04-17T08:47:45.964824+0000","last_fullsized":"2024-04-17T08:47:45.964824+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:06:45.091029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966250+0000","last_change":"2024-04-17T08:47:42.821446+0000","last_active":"2024-04-17T08:47:45.966250+0000","last_peered":"2024-04-17T08:47:45.966250+0000","last_clean":"2024-04-17T08:47:45.966250+0000","last_became_active":"2024-04-17T08:47:42.821159+0000","last_became_peered":"2024-04-17T08:47:42.821159+0000","last_unstale":"2024-04-17T08:47:45.966250+0000","last_undegraded":"2024-04-17T08:47:45.966250+0000","last_fullsized":"2024-04-17T08:47:45.966250+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:04:16.800633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849882+0000","last_change":"2024-04-17T08:47:43.832633+0000","last_active":"2024-04-17T08:47:44.849882+0000","last_peered":"2024-04-17T08:47:44.849882+0000","last_clean":"2024-04-17T08:47:44.849882+0000","last_became_active":"2024-04-17T08:47:43.832195+0000","last_became_peered":"2024-04-17T08:47:43.832195+0000","last_unstale":"2024-04-17T08:47:44.849882+0000","last_undegraded":"2024-04-17T08:47:44.849882+0000","last_fullsized":"2024-04-17T08:47:44.849882+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:35.081905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"211'3","reported_seq":17,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966037+0000","last_change":"2024-04-17T08:47:42.813363+0000","last_active":"2024-04-17T08:47:45.966037+0000","last_peered":"2024-04-17T08:47:45.966037+0000","last_clean":"2024-04-17T08:47:45.966037+0000","last_became_active":"2024-04-17T08:47:42.813055+0000","last_became_peered":"2024-04-17T08:47:42.813055+0000","last_unstale":"2024-04-17T08:47:45.966037+0000","last_undegraded":"2024-04-17T08:47:45.966037+0000","last_fullsized":"2024-04-17T08:47:45.966037+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:43.551895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861057+0000","last_change":"2024-04-17T08:47:43.844847+0000","last_active":"2024-04-17T08:47:44.861057+0000","last_peered":"2024-04-17T08:47:44.861057+0000","last_clean":"2024-04-17T08:47:44.861057+0000","last_became_active":"2024-04-17T08:47:43.844568+0000","last_became_peered":"2024-04-17T08:47:43.844568+0000","last_unstale":"2024-04-17T08:47:44.861057+0000","last_undegraded":"2024-04-17T08:47:44.861057+0000","last_fullsized":"2024-04-17T08:47:44.861057+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:16.529905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"211'6","reported_seq":20,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842627+0000","last_change":"2024-04-17T08:47:42.811120+0000","last_active":"2024-04-17T08:47:44.842627+0000","last_peered":"2024-04-17T08:47:44.842627+0000","last_clean":"2024-04-17T08:47:44.842627+0000","last_became_active":"2024-04-17T08:47:42.810828+0000","last_became_peered":"2024-04-17T08:47:42.810828+0000","last_unstale":"2024-04-17T08:47:44.842627+0000","last_undegraded":"2024-04-17T08:47:44.842627+0000","last_fullsized":"2024-04-17T08:47:44.842627+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:04.411604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843110+0000","last_change":"2024-04-17T08:47:43.857598+0000","last_active":"2024-04-17T08:47:44.843110+0000","last_peered":"2024-04-17T08:47:44.843110+0000","last_clean":"2024-04-17T08:47:44.843110+0000","last_became_active":"2024-04-17T08:47:43.857287+0000","last_became_peered":"2024-04-17T08:47:43.857287+0000","last_unstale":"2024-04-17T08:47:44.843110+0000","last_undegraded":"2024-04-17T08:47:44.843110+0000","last_fullsized":"2024-04-17T08:47:44.843110+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:51.822943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965921+0000","last_change":"2024-04-17T08:47:42.821363+0000","last_active":"2024-04-17T08:47:45.965921+0000","last_peered":"2024-04-17T08:47:45.965921+0000","last_clean":"2024-04-17T08:47:45.965921+0000","last_became_active":"2024-04-17T08:47:42.821002+0000","last_became_peered":"2024-04-17T08:47:42.821002+0000","last_unstale":"2024-04-17T08:47:45.965921+0000","last_undegraded":"2024-04-17T08:47:45.965921+0000","last_fullsized":"2024-04-17T08:47:45.965921+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:26.163096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849440+0000","last_change":"2024-04-17T08:47:43.830583+0000","last_active":"2024-04-17T08:47:44.849440+0000","last_peered":"2024-04-17T08:47:44.849440+0000","last_clean":"2024-04-17T08:47:44.849440+0000","last_became_active":"2024-04-17T08:47:43.830071+0000","last_became_peered":"2024-04-17T08:47:43.830071+0000","last_unstale":"2024-04-17T08:47:44.849440+0000","last_undegraded":"2024-04-17T08:47:44.849440+0000","last_fullsized":"2024-04-17T08:47:44.849440+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:56.091246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860666+0000","last_change":"2024-04-17T08:47:42.820473+0000","last_active":"2024-04-17T08:47:44.860666+0000","last_peered":"2024-04-17T08:47:44.860666+0000","last_clean":"2024-04-17T08:47:44.860666+0000","last_became_active":"2024-04-17T08:47:42.819120+0000","last_became_peered":"2024-04-17T08:47:42.819120+0000","last_unstale":"2024-04-17T08:47:44.860666+0000","last_undegraded":"2024-04-17T08:47:44.860666+0000","last_fullsized":"2024-04-17T08:47:44.860666+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:47.059315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848372+0000","last_change":"2024-04-17T08:47:43.840829+0000","last_active":"2024-04-17T08:47:44.848372+0000","last_peered":"2024-04-17T08:47:44.848372+0000","last_clean":"2024-04-17T08:47:44.848372+0000","last_became_active":"2024-04-17T08:47:43.840526+0000","last_became_peered":"2024-04-17T08:47:43.840526+0000","last_unstale":"2024-04-17T08:47:44.848372+0000","last_undegraded":"2024-04-17T08:47:44.848372+0000","last_fullsized":"2024-04-17T08:47:44.848372+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:29.438541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.963899+0000","last_change":"2024-04-17T08:47:42.819248+0000","last_active":"2024-04-17T08:47:45.963899+0000","last_peered":"2024-04-17T08:47:45.963899+0000","last_clean":"2024-04-17T08:47:45.963899+0000","last_became_active":"2024-04-17T08:47:42.818977+0000","last_became_peered":"2024-04-17T08:47:42.818977+0000","last_unstale":"2024-04-17T08:47:45.963899+0000","last_undegraded":"2024-04-17T08:47:45.963899+0000","last_fullsized":"2024-04-17T08:47:45.963899+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:47:18.357684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860305+0000","last_change":"2024-04-17T08:47:43.833701+0000","last_active":"2024-04-17T08:47:44.860305+0000","last_peered":"2024-04-17T08:47:44.860305+0000","last_clean":"2024-04-17T08:47:44.860305+0000","last_became_active":"2024-04-17T08:47:43.829594+0000","last_became_peered":"2024-04-17T08:47:43.829594+0000","last_unstale":"2024-04-17T08:47:44.860305+0000","last_undegraded":"2024-04-17T08:47:44.860305+0000","last_fullsized":"2024-04-17T08:47:44.860305+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:58:53.462519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"211'8","reported_seq":22,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965340+0000","last_change":"2024-04-17T08:47:42.821650+0000","last_active":"2024-04-17T08:47:45.965340+0000","last_peered":"2024-04-17T08:47:45.965340+0000","last_clean":"2024-04-17T08:47:45.965340+0000","last_became_active":"2024-04-17T08:47:42.821122+0000","last_became_peered":"2024-04-17T08:47:42.821122+0000","last_unstale":"2024-04-17T08:47:45.965340+0000","last_undegraded":"2024-04-17T08:47:45.965340+0000","last_fullsized":"2024-04-17T08:47:45.965340+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:43.316088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861170+0000","last_change":"2024-04-17T08:47:43.830514+0000","last_active":"2024-04-17T08:47:44.861170+0000","last_peered":"2024-04-17T08:47:44.861170+0000","last_clean":"2024-04-17T08:47:44.861170+0000","last_became_active":"2024-04-17T08:47:43.829670+0000","last_became_peered":"2024-04-17T08:47:43.829670+0000","last_unstale":"2024-04-17T08:47:44.861170+0000","last_undegraded":"2024-04-17T08:47:44.861170+0000","last_fullsized":"2024-04-17T08:47:44.861170+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:43:11.314195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966188+0000","last_change":"2024-04-17T08:47:42.814185+0000","last_active":"2024-04-17T08:47:45.966188+0000","last_peered":"2024-04-17T08:47:45.966188+0000","last_clean":"2024-04-17T08:47:45.966188+0000","last_became_active":"2024-04-17T08:47:42.813673+0000","last_became_peered":"2024-04-17T08:47:42.813673+0000","last_unstale":"2024-04-17T08:47:45.966188+0000","last_undegraded":"2024-04-17T08:47:45.966188+0000","last_fullsized":"2024-04-17T08:47:45.966188+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:30.746785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849640+0000","last_change":"2024-04-17T08:47:43.841100+0000","last_active":"2024-04-17T08:47:44.849640+0000","last_peered":"2024-04-17T08:47:44.849640+0000","last_clean":"2024-04-17T08:47:44.849640+0000","last_became_active":"2024-04-17T08:47:43.840812+0000","last_became_peered":"2024-04-17T08:47:43.840812+0000","last_unstale":"2024-04-17T08:47:44.849640+0000","last_undegraded":"2024-04-17T08:47:44.849640+0000","last_fullsized":"2024-04-17T08:47:44.849640+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:28.771576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964948+0000","last_change":"2024-04-17T08:47:42.806681+0000","last_active":"2024-04-17T08:47:45.964948+0000","last_peered":"2024-04-17T08:47:45.964948+0000","last_clean":"2024-04-17T08:47:45.964948+0000","last_became_active":"2024-04-17T08:47:42.805677+0000","last_became_peered":"2024-04-17T08:47:42.805677+0000","last_unstale":"2024-04-17T08:47:45.964948+0000","last_undegraded":"2024-04-17T08:47:45.964948+0000","last_fullsized":"2024-04-17T08:47:45.964948+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:18.113143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.848947+0000","last_change":"2024-04-17T08:47:43.837785+0000","last_active":"2024-04-17T08:47:44.848947+0000","last_peered":"2024-04-17T08:47:44.848947+0000","last_clean":"2024-04-17T08:47:44.848947+0000","last_became_active":"2024-04-17T08:47:43.837507+0000","last_became_peered":"2024-04-17T08:47:43.837507+0000","last_unstale":"2024-04-17T08:47:44.848947+0000","last_undegraded":"2024-04-17T08:47:44.848947+0000","last_fullsized":"2024-04-17T08:47:44.848947+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:05.425104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"211'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966382+0000","last_change":"2024-04-17T08:47:42.821659+0000","last_active":"2024-04-17T08:47:45.966382+0000","last_peered":"2024-04-17T08:47:45.966382+0000","last_clean":"2024-04-17T08:47:45.966382+0000","last_became_active":"2024-04-17T08:47:42.821150+0000","last_became_peered":"2024-04-17T08:47:42.821150+0000","last_unstale":"2024-04-17T08:47:45.966382+0000","last_undegraded":"2024-04-17T08:47:45.966382+0000","last_fullsized":"2024-04-17T08:47:45.966382+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:28.788973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.968433+0000","last_change":"2024-04-17T08:47:43.845043+0000","last_active":"2024-04-17T08:47:45.968433+0000","last_peered":"2024-04-17T08:47:45.968433+0000","last_clean":"2024-04-17T08:47:45.968433+0000","last_became_active":"2024-04-17T08:47:43.844783+0000","last_became_peered":"2024-04-17T08:47:43.844783+0000","last_unstale":"2024-04-17T08:47:45.968433+0000","last_undegraded":"2024-04-17T08:47:45.968433+0000","last_fullsized":"2024-04-17T08:47:45.968433+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:02.324702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850255+0000","last_change":"2024-04-17T08:47:42.810555+0000","last_active":"2024-04-17T08:47:44.850255+0000","last_peered":"2024-04-17T08:47:44.850255+0000","last_clean":"2024-04-17T08:47:44.850255+0000","last_became_active":"2024-04-17T08:47:42.808744+0000","last_became_peered":"2024-04-17T08:47:42.808744+0000","last_unstale":"2024-04-17T08:47:44.850255+0000","last_undegraded":"2024-04-17T08:47:44.850255+0000","last_fullsized":"2024-04-17T08:47:44.850255+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:17:04.555542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860770+0000","last_change":"2024-04-17T08:47:43.844713+0000","last_active":"2024-04-17T08:47:44.860770+0000","last_peered":"2024-04-17T08:47:44.860770+0000","last_clean":"2024-04-17T08:47:44.860770+0000","last_became_active":"2024-04-17T08:47:43.843988+0000","last_became_peered":"2024-04-17T08:47:43.843988+0000","last_unstale":"2024-04-17T08:47:44.860770+0000","last_undegraded":"2024-04-17T08:47:44.860770+0000","last_fullsized":"2024-04-17T08:47:44.860770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.934805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842587+0000","last_change":"2024-04-17T08:47:42.823486+0000","last_active":"2024-04-17T08:47:44.842587+0000","last_peered":"2024-04-17T08:47:44.842587+0000","last_clean":"2024-04-17T08:47:44.842587+0000","last_became_active":"2024-04-17T08:47:42.822869+0000","last_became_peered":"2024-04-17T08:47:42.822869+0000","last_unstale":"2024-04-17T08:47:44.842587+0000","last_undegraded":"2024-04-17T08:47:44.842587+0000","last_fullsized":"2024-04-17T08:47:44.842587+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.747666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849825+0000","last_change":"2024-04-17T08:47:43.847071+0000","last_active":"2024-04-17T08:47:44.849825+0000","last_peered":"2024-04-17T08:47:44.849825+0000","last_clean":"2024-04-17T08:47:44.849825+0000","last_became_active":"2024-04-17T08:47:43.846785+0000","last_became_peered":"2024-04-17T08:47:43.846785+0000","last_unstale":"2024-04-17T08:47:44.849825+0000","last_undegraded":"2024-04-17T08:47:44.849825+0000","last_fullsized":"2024-04-17T08:47:44.849825+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:06:34.129966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966101+0000","last_change":"2024-04-17T08:47:42.819774+0000","last_active":"2024-04-17T08:47:45.966101+0000","last_peered":"2024-04-17T08:47:45.966101+0000","last_clean":"2024-04-17T08:47:45.966101+0000","last_became_active":"2024-04-17T08:47:42.819464+0000","last_became_peered":"2024-04-17T08:47:42.819464+0000","last_unstale":"2024-04-17T08:47:45.966101+0000","last_undegraded":"2024-04-17T08:47:45.966101+0000","last_fullsized":"2024-04-17T08:47:45.966101+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:16:45.564858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966325+0000","last_change":"2024-04-17T08:47:43.847533+0000","last_active":"2024-04-17T08:47:45.966325+0000","last_peered":"2024-04-17T08:47:45.966325+0000","last_clean":"2024-04-17T08:47:45.966325+0000","last_became_active":"2024-04-17T08:47:43.847267+0000","last_became_peered":"2024-04-17T08:47:43.847267+0000","last_unstale":"2024-04-17T08:47:45.966325+0000","last_undegraded":"2024-04-17T08:47:45.966325+0000","last_fullsized":"2024-04-17T08:47:45.966325+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:57.075429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.843023+0000","last_change":"2024-04-17T08:47:42.819212+0000","last_active":"2024-04-17T08:47:44.843023+0000","last_peered":"2024-04-17T08:47:44.843023+0000","last_clean":"2024-04-17T08:47:44.843023+0000","last_became_active":"2024-04-17T08:47:42.818642+0000","last_became_peered":"2024-04-17T08:47:42.818642+0000","last_unstale":"2024-04-17T08:47:44.843023+0000","last_undegraded":"2024-04-17T08:47:44.843023+0000","last_fullsized":"2024-04-17T08:47:44.843023+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:28.425040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965392+0000","last_change":"2024-04-17T08:47:43.847550+0000","last_active":"2024-04-17T08:47:45.965392+0000","last_peered":"2024-04-17T08:47:45.965392+0000","last_clean":"2024-04-17T08:47:45.965392+0000","last_became_active":"2024-04-17T08:47:43.847214+0000","last_became_peered":"2024-04-17T08:47:43.847214+0000","last_unstale":"2024-04-17T08:47:45.965392+0000","last_undegraded":"2024-04-17T08:47:45.965392+0000","last_fullsized":"2024-04-17T08:47:45.965392+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:07:16.817223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850873+0000","last_change":"2024-04-17T08:47:42.819212+0000","last_active":"2024-04-17T08:47:44.850873+0000","last_peered":"2024-04-17T08:47:44.850873+0000","last_clean":"2024-04-17T08:47:44.850873+0000","last_became_active":"2024-04-17T08:47:42.818890+0000","last_became_peered":"2024-04-17T08:47:42.818890+0000","last_unstale":"2024-04-17T08:47:44.850873+0000","last_undegraded":"2024-04-17T08:47:44.850873+0000","last_fullsized":"2024-04-17T08:47:44.850873+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:20.401612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966436+0000","last_change":"2024-04-17T08:47:43.851605+0000","last_active":"2024-04-17T08:47:45.966436+0000","last_peered":"2024-04-17T08:47:45.966436+0000","last_clean":"2024-04-17T08:47:45.966436+0000","last_became_active":"2024-04-17T08:47:43.851258+0000","last_became_peered":"2024-04-17T08:47:43.851258+0000","last_unstale":"2024-04-17T08:47:45.966436+0000","last_undegraded":"2024-04-17T08:47:45.966436+0000","last_fullsized":"2024-04-17T08:47:45.966436+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:34.252594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"211'2","reported_seq":16,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966558+0000","last_change":"2024-04-17T08:47:42.821818+0000","last_active":"2024-04-17T08:47:45.966558+0000","last_peered":"2024-04-17T08:47:45.966558+0000","last_clean":"2024-04-17T08:47:45.966558+0000","last_became_active":"2024-04-17T08:47:42.821354+0000","last_became_peered":"2024-04-17T08:47:42.821354+0000","last_unstale":"2024-04-17T08:47:45.966558+0000","last_undegraded":"2024-04-17T08:47:45.966558+0000","last_fullsized":"2024-04-17T08:47:45.966558+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:09.804299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.861545+0000","last_change":"2024-04-17T08:47:43.842711+0000","last_active":"2024-04-17T08:47:44.861545+0000","last_peered":"2024-04-17T08:47:44.861545+0000","last_clean":"2024-04-17T08:47:44.861545+0000","last_became_active":"2024-04-17T08:47:43.841929+0000","last_became_peered":"2024-04-17T08:47:43.841929+0000","last_unstale":"2024-04-17T08:47:44.861545+0000","last_undegraded":"2024-04-17T08:47:44.861545+0000","last_fullsized":"2024-04-17T08:47:44.861545+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:22:49.372829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"212'1","reported_seq":15,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.863562+0000","last_change":"2024-04-17T08:47:42.819460+0000","last_active":"2024-04-17T08:47:44.863562+0000","last_peered":"2024-04-17T08:47:44.863562+0000","last_clean":"2024-04-17T08:47:44.863562+0000","last_became_active":"2024-04-17T08:47:42.816560+0000","last_became_peered":"2024-04-17T08:47:42.816560+0000","last_unstale":"2024-04-17T08:47:44.863562+0000","last_undegraded":"2024-04-17T08:47:44.863562+0000","last_fullsized":"2024-04-17T08:47:44.863562+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:32.501537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964846+0000","last_change":"2024-04-17T08:47:43.849159+0000","last_active":"2024-04-17T08:47:45.964846+0000","last_peered":"2024-04-17T08:47:45.964846+0000","last_clean":"2024-04-17T08:47:45.964846+0000","last_became_active":"2024-04-17T08:47:43.845427+0000","last_became_peered":"2024-04-17T08:47:43.845427+0000","last_unstale":"2024-04-17T08:47:45.964846+0000","last_undegraded":"2024-04-17T08:47:45.964846+0000","last_fullsized":"2024-04-17T08:47:45.964846+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:00.999483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860919+0000","last_change":"2024-04-17T08:47:42.818676+0000","last_active":"2024-04-17T08:47:44.860919+0000","last_peered":"2024-04-17T08:47:44.860919+0000","last_clean":"2024-04-17T08:47:44.860919+0000","last_became_active":"2024-04-17T08:47:42.816807+0000","last_became_peered":"2024-04-17T08:47:42.816807+0000","last_unstale":"2024-04-17T08:47:44.860919+0000","last_undegraded":"2024-04-17T08:47:44.860919+0000","last_fullsized":"2024-04-17T08:47:44.860919+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:50.255072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965531+0000","last_change":"2024-04-17T08:47:43.852436+0000","last_active":"2024-04-17T08:47:45.965531+0000","last_peered":"2024-04-17T08:47:45.965531+0000","last_clean":"2024-04-17T08:47:45.965531+0000","last_became_active":"2024-04-17T08:47:43.850247+0000","last_became_peered":"2024-04-17T08:47:43.850247+0000","last_unstale":"2024-04-17T08:47:45.965531+0000","last_undegraded":"2024-04-17T08:47:45.965531+0000","last_fullsized":"2024-04-17T08:47:45.965531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:52.331396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964738+0000","last_change":"2024-04-17T08:47:42.802811+0000","last_active":"2024-04-17T08:47:45.964738+0000","last_peered":"2024-04-17T08:47:45.964738+0000","last_clean":"2024-04-17T08:47:45.964738+0000","last_became_active":"2024-04-17T08:47:42.802428+0000","last_became_peered":"2024-04-17T08:47:42.802428+0000","last_unstale":"2024-04-17T08:47:45.964738+0000","last_undegraded":"2024-04-17T08:47:45.964738+0000","last_fullsized":"2024-04-17T08:47:45.964738+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:50:07.507003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965795+0000","last_change":"2024-04-17T08:47:43.846971+0000","last_active":"2024-04-17T08:47:45.965795+0000","last_peered":"2024-04-17T08:47:45.965795+0000","last_clean":"2024-04-17T08:47:45.965795+0000","last_became_active":"2024-04-17T08:47:43.846355+0000","last_became_peered":"2024-04-17T08:47:43.846355+0000","last_unstale":"2024-04-17T08:47:45.965795+0000","last_undegraded":"2024-04-17T08:47:45.965795+0000","last_fullsized":"2024-04-17T08:47:45.965795+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:56.967670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.850726+0000","last_change":"2024-04-17T08:47:42.809920+0000","last_active":"2024-04-17T08:47:44.850726+0000","last_peered":"2024-04-17T08:47:44.850726+0000","last_clean":"2024-04-17T08:47:44.850726+0000","last_became_active":"2024-04-17T08:47:42.809138+0000","last_became_peered":"2024-04-17T08:47:42.809138+0000","last_unstale":"2024-04-17T08:47:44.850726+0000","last_undegraded":"2024-04-17T08:47:44.850726+0000","last_fullsized":"2024-04-17T08:47:44.850726+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:35.570254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965619+0000","last_change":"2024-04-17T08:47:43.851022+0000","last_active":"2024-04-17T08:47:45.965619+0000","last_peered":"2024-04-17T08:47:45.965619+0000","last_clean":"2024-04-17T08:47:45.965619+0000","last_became_active":"2024-04-17T08:47:43.849991+0000","last_became_peered":"2024-04-17T08:47:43.849991+0000","last_unstale":"2024-04-17T08:47:45.965619+0000","last_undegraded":"2024-04-17T08:47:45.965619+0000","last_fullsized":"2024-04-17T08:47:45.965619+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:29.053465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.849590+0000","last_change":"2024-04-17T08:47:42.810444+0000","last_active":"2024-04-17T08:47:44.849590+0000","last_peered":"2024-04-17T08:47:44.849590+0000","last_clean":"2024-04-17T08:47:44.849590+0000","last_became_active":"2024-04-17T08:47:42.810104+0000","last_became_peered":"2024-04-17T08:47:42.810104+0000","last_unstale":"2024-04-17T08:47:44.849590+0000","last_undegraded":"2024-04-17T08:47:44.849590+0000","last_fullsized":"2024-04-17T08:47:44.849590+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:41.770362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966796+0000","last_change":"2024-04-17T08:47:43.846925+0000","last_active":"2024-04-17T08:47:45.966796+0000","last_peered":"2024-04-17T08:47:45.966796+0000","last_clean":"2024-04-17T08:47:45.966796+0000","last_became_active":"2024-04-17T08:47:43.846665+0000","last_became_peered":"2024-04-17T08:47:43.846665+0000","last_unstale":"2024-04-17T08:47:45.966796+0000","last_undegraded":"2024-04-17T08:47:45.966796+0000","last_fullsized":"2024-04-17T08:47:45.966796+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:14.224403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.967814+0000","last_change":"2024-04-17T08:47:42.816009+0000","last_active":"2024-04-17T08:47:45.967814+0000","last_peered":"2024-04-17T08:47:45.967814+0000","last_clean":"2024-04-17T08:47:45.967814+0000","last_became_active":"2024-04-17T08:47:42.815517+0000","last_became_peered":"2024-04-17T08:47:42.815517+0000","last_unstale":"2024-04-17T08:47:45.967814+0000","last_undegraded":"2024-04-17T08:47:45.967814+0000","last_fullsized":"2024-04-17T08:47:45.967814+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:33:50.251489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.965759+0000","last_change":"2024-04-17T08:47:43.852122+0000","last_active":"2024-04-17T08:47:45.965759+0000","last_peered":"2024-04-17T08:47:45.965759+0000","last_clean":"2024-04-17T08:47:45.965759+0000","last_became_active":"2024-04-17T08:47:43.851799+0000","last_became_peered":"2024-04-17T08:47:43.851799+0000","last_unstale":"2024-04-17T08:47:45.965759+0000","last_undegraded":"2024-04-17T08:47:45.965759+0000","last_fullsized":"2024-04-17T08:47:45.965759+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:48.705154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.842341+0000","last_change":"2024-04-17T08:47:42.815922+0000","last_active":"2024-04-17T08:47:44.842341+0000","last_peered":"2024-04-17T08:47:44.842341+0000","last_clean":"2024-04-17T08:47:44.842341+0000","last_became_active":"2024-04-17T08:47:42.815619+0000","last_became_peered":"2024-04-17T08:47:42.815619+0000","last_unstale":"2024-04-17T08:47:44.842341+0000","last_undegraded":"2024-04-17T08:47:44.842341+0000","last_fullsized":"2024-04-17T08:47:44.842341+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:17.163964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:44.860304+0000","last_change":"2024-04-17T08:47:43.840571+0000","last_active":"2024-04-17T08:47:44.860304+0000","last_peered":"2024-04-17T08:47:44.860304+0000","last_clean":"2024-04-17T08:47:44.860304+0000","last_became_active":"2024-04-17T08:47:43.840254+0000","last_became_peered":"2024-04-17T08:47:43.840254+0000","last_unstale":"2024-04-17T08:47:44.860304+0000","last_undegraded":"2024-04-17T08:47:44.860304+0000","last_fullsized":"2024-04-17T08:47:44.860304+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:42.774015+0000","last_clean_scrub_stamp":"2024-04-17T08:47:42.774015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:17:55.854257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":14,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.966113+0000","last_change":"2024-04-17T08:47:42.820712+0000","last_active":"2024-04-17T08:47:45.966113+0000","last_peered":"2024-04-17T08:47:45.966113+0000","last_clean":"2024-04-17T08:47:45.966113+0000","last_became_active":"2024-04-17T08:47:42.820393+0000","last_became_peered":"2024-04-17T08:47:42.820393+0000","last_unstale":"2024-04-17T08:47:45.966113+0000","last_undegraded":"2024-04-17T08:47:45.966113+0000","last_fullsized":"2024-04-17T08:47:45.966113+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":209,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:47:41.760514+0000","last_clean_scrub_stamp":"2024-04-17T08:47:41.760514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:45.405730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"200'332","reported_seq":1333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-17T08:47:45.964774+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:47:45.964774+0000","last_peered":"2024-04-17T08:47:45.964774+0000","last_clean":"2024-04-17T08:47:45.964774+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:47:45.964774+0000","last_undegraded":"2024-04-17T08:47:45.964774+0000","last_fullsized":"2024-04-17T08:47:45.964774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":547,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1986,"num_read_kb":4178,"num_write":547,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543261,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52404,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319436,"statfs":{"total":96636764160,"available":96583102464,"internally_reserved":0,"allocated":3252224,"data_stored":1883778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42080,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329760,"statfs":{"total":96636764160,"available":96593674240,"internally_reserved":0,"allocated":2637824,"data_stored":1286978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29612,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342228,"statfs":{"total":96636764160,"available":96606441472,"internally_reserved":0,"allocated":2650112,"data_stored":1293306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60152,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311688,"statfs":{"total":96636764160,"available":96575168512,"internally_reserved":0,"allocated":2666496,"data_stored":1295500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543262,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47664,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324176,"statfs":{"total":96636764160,"available":96587956224,"internally_reserved":0,"allocated":2650112,"data_stored":1287172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47660,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324180,"statfs":{"total":96636764160,"available":96587960320,"internally_reserved":0,"allocated":2650112,"data_stored":1287672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543262,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60152,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311688,"statfs":{"total":96636764160,"available":96575168512,"internally_reserved":0,"allocated":2662400,"data_stored":1294918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60720,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311120,"statfs":{"total":96636764160,"available":96574586880,"internally_reserved":0,"allocated":3248128,"data_stored":1883696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:47:48.920 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:47:48.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:47:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.195+0000 7f5d29600640 1 Processor -- start 2024-04-17T08:47:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.196+0000 7f5d29600640 1 -- start start 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.197+0000 7f5d29600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d240b6bf0 0x7f5d240d4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.197+0000 7f5d29600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d240b8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.197+0000 7f5d29600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d240b79b0 con 0x7f5d2415ebe0 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.197+0000 7f5d29600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5d240d3bd0 con 0x7f5d240b6bf0 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.197+0000 7f5d29600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240d3330 con 0x7f5d240d2bc0 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d240b6bf0 0x7f5d240d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d240b6bf0 0x7f5d240d4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58308/0 (socket says 172.21.15.67:58308) 2024-04-17T08:47:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 -- 172.21.15.67:0/2274336009 learned_addr learned my addr 172.21.15.67:0/2274336009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d28c00640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d240b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 -- 172.21.15.67:0/2274336009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 msgr2=0x7f5d240b8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d240b8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 -- 172.21.15.67:0/2274336009 >> v1:172.21.15.187:6789/0 conn(0x7f5d2415ebe0 legacy=0x7f5d240b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d23400640 1 -- 172.21.15.67:0/2274336009 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d240d3610 con 0x7f5d240b6bf0 2024-04-17T08:47:49.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.198+0000 7f5d28c00640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d240b8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.200+0000 7f5d23400640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d240b6bf0 0x7f5d240d4370 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009870 tx=0x7f5d1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50abae592c0e1db server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.200+0000 7f5d22a00640 1 -- 172.21.15.67:0/2274336009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10034070 con 0x7f5d240b6bf0 2024-04-17T08:47:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.200+0000 7f5d22a00640 1 -- 172.21.15.67:0/2274336009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d10030840 con 0x7f5d240b6bf0 2024-04-17T08:47:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.200+0000 7f5d22a00640 1 -- 172.21.15.67:0/2274336009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10030c10 con 0x7f5d240b6bf0 2024-04-17T08:47:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 -- 172.21.15.67:0/2274336009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d240b6bf0 msgr2=0x7f5d240d4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d240b6bf0 0x7f5d240d4370 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009870 tx=0x7f5d1002b980 comp rx=0 tx=0).stop 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 -- 172.21.15.67:0/2274336009 shutdown_connections 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d240b8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 --2- 172.21.15.67:0/2274336009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d240b6bf0 0x7f5d240d4370 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 -- 172.21.15.67:0/2274336009 >> 172.21.15.67:0/2274336009 conn(0x7f5d240d16c0 msgr2=0x7f5d240d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 -- 172.21.15.67:0/2274336009 shutdown_connections 2024-04-17T08:47:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.201+0000 7f5d29600640 1 -- 172.21.15.67:0/2274336009 wait complete. 2024-04-17T08:47:49.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.203+0000 7f5d29600640 1 Processor -- start 2024-04-17T08:47:49.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.203+0000 7f5d29600640 1 -- start start 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d240b6bf0 0x7f5d240b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d2415f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2415efc0 con 0x7f5d2415ebe0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d240b7c30 con 0x7f5d240b6bf0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d29600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240d3010 con 0x7f5d240d2bc0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d28c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d28c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58320/0 (socket says 172.21.15.67:58320) 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d28c00640 1 -- 172.21.15.67:0/1129858061 learned_addr learned my addr 172.21.15.67:0/1129858061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.204+0000 7f5d23e00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d240b6bf0 0x7f5d240b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d23400640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d2415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 -- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 msgr2=0x7f5d2415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d2415f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 -- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d240b6bf0 msgr2=0x7f5d240b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d240b6bf0 0x7f5d240b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 -- 172.21.15.67:0/1129858061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d24160130 con 0x7f5d2415ebe0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d23400640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d2415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d28c00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00c7f0 tx=0x7f5d1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c015070 con 0x7f5d2415ebe0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1c007820 con 0x7f5d2415ebe0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c007c50 con 0x7f5d2415ebe0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.205+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2415f890 con 0x7f5d2415ebe0 2024-04-17T08:47:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.206+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2415fe50 con 0x7f5d2415ebe0 2024-04-17T08:47:49.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.207+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d24068a10 con 0x7f5d2415ebe0 2024-04-17T08:47:49.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.208+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f5d1c018070 con 0x7f5d2415ebe0 2024-04-17T08:47:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.209+0000 7f5d1be00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 0x7f5d0406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.209+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f5d1c08bf90 con 0x7f5d2415ebe0 2024-04-17T08:47:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.210+0000 7f5d23e00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 0x7f5d0406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.211+0000 7f5d23e00640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 0x7f5d0406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d140012e0 tx=0x7f5d14009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.214+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5d1c05a180 con 0x7f5d2415ebe0 2024-04-17T08:47:49.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.492+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5d240d1cb0 con 0x7f5d2415ebe0 2024-04-17T08:47:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.497+0000 7f5d1be00640 1 -- 172.21.15.67:0/1129858061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v47) ==== 190+0+147 (secure 0 0 0) 0x7f5d1c059a80 con 0x7f5d2415ebe0 2024-04-17T08:47:49.495 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:47:49.496 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:47:49.496 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:47:49.496 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:47:49.496 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:47:49.496 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:47:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.500+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 msgr2=0x7f5d0406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.500+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 0x7f5d0406eed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d140012e0 tx=0x7f5d14009040 comp rx=0 tx=0).stop 2024-04-17T08:47:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.500+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 msgr2=0x7f5d240ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.500+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00c7f0 tx=0x7f5d1c00ccb0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 shutdown_connections 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f5d0406ca10 0x7f5d0406eed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d2415ebe0 0x7f5d240ca9d0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d240d2bc0 0x7f5d2415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 --2- 172.21.15.67:0/1129858061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d240b6bf0 0x7f5d240b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 >> 172.21.15.67:0/1129858061 conn(0x7f5d240d16c0 msgr2=0x7f5d240cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 shutdown_connections 2024-04-17T08:47:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.501+0000 7f5d29600640 1 -- 172.21.15.67:0/1129858061 wait complete. 2024-04-17T08:47:49.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:47:49.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.728+0000 7f8d03200640 1 Processor -- start 2024-04-17T08:47:49.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.729+0000 7f8d03200640 1 -- start start 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.730+0000 7f8d03200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d040b9f00 0x7f8d040b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.730+0000 7f8d03200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040cfe20 con 0x7f8d040d12a0 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.730+0000 7f8d03200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040d0100 con 0x7f8d040cf8f0 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.730+0000 7f8d03200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8d040cf640 con 0x7f8d040b9f00 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d01e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d040b9f00 0x7f8d040b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d02800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d040d12a0 0x7f8d040d3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41300/0 (socket says 172.21.15.67:41300) 2024-04-17T08:47:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d02800640 1 -- 172.21.15.67:0/778808966 learned_addr learned my addr 172.21.15.67:0/778808966 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d01e00640 1 -- 172.21.15.67:0/778808966 >> v1:172.21.15.187:6789/0 conn(0x7f8d040cf8f0 legacy=0x7f8d041634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d01e00640 1 -- 172.21.15.67:0/778808966 >> v1:172.21.15.67:6789/0 conn(0x7f8d040d12a0 legacy=0x7f8d040d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d01e00640 1 -- 172.21.15.67:0/778808966 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040d3140 con 0x7f8d040b9f00 2024-04-17T08:47:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.731+0000 7f8d00a00640 1 -- 172.21.15.67:0/778808966 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2647701906 0 0) 0x7f8d040cfe20 con 0x7f8d040d12a0 2024-04-17T08:47:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.732+0000 7f8d01e00640 1 --2- 172.21.15.67:0/778808966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040b7f80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794ea5104f34ecf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.733+0000 7f8d00a00640 1 -- 172.21.15.67:0/778808966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40315a0 con 0x7f8d040b9f00 2024-04-17T08:47:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.733+0000 7f8d00a00640 1 -- 172.21.15.67:0/778808966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf4037070 con 0x7f8d040b9f00 2024-04-17T08:47:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.733+0000 7f8d00a00640 1 -- 172.21.15.67:0/778808966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4031960 con 0x7f8d040b9f00 2024-04-17T08:47:49.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 -- 172.21.15.67:0/778808966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 msgr2=0x7f8d040b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 --2- 172.21.15.67:0/778808966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040b7f80 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf402b840 comp rx=0 tx=0).stop 2024-04-17T08:47:49.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 -- 172.21.15.67:0/778808966 shutdown_connections 2024-04-17T08:47:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 --2- 172.21.15.67:0/778808966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040b7f80 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 -- 172.21.15.67:0/778808966 >> 172.21.15.67:0/778808966 conn(0x7f8d0406cb60 msgr2=0x7f8d0406cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 -- 172.21.15.67:0/778808966 shutdown_connections 2024-04-17T08:47:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.734+0000 7f8d03200640 1 -- 172.21.15.67:0/778808966 wait complete. 2024-04-17T08:47:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.736+0000 7f8d03200640 1 Processor -- start 2024-04-17T08:47:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.736+0000 7f8d03200640 1 -- start start 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040cf8f0 0x7f8d041576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040d12a0 0x7f8d0415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d040ba2e0 con 0x7f8d040b9f00 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8d040b84c0 con 0x7f8d040cf8f0 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d03200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8d040d03e0 con 0x7f8d040d12a0 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58336/0 (socket says 172.21.15.67:58336) 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d01e00640 1 -- 172.21.15.67:0/845192503 learned_addr learned my addr 172.21.15.67:0/845192503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d01400640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040cf8f0 0x7f8d041576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.738+0000 7f8d02800640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040d12a0 0x7f8d0415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 -- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040d12a0 msgr2=0x7f8d0415b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040d12a0 0x7f8d0415b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 -- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040cf8f0 msgr2=0x7f8d041576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040cf8f0 0x7f8d041576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 -- 172.21.15.67:0/845192503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d04157c00 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d01e00640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4004030 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf40041f0 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4032640 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.739+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d041598b0 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.740+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d04159e70 con 0x7f8d040b9f00 2024-04-17T08:47:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.741+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d040755d0 con 0x7f8d040b9f00 2024-04-17T08:47:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.742+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f8cf40327e0 con 0x7f8d040b9f00 2024-04-17T08:47:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.742+0000 7f8cfa000640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 0x7f8ce006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.743+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f8cf402f030 con 0x7f8d040b9f00 2024-04-17T08:47:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.743+0000 7f8d01400640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 0x7f8ce006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:49.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.744+0000 7f8d01400640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 0x7f8ce006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d0415b060 tx=0x7f8cec009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:49.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.747+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8cf4041070 con 0x7f8d040b9f00 2024-04-17T08:47:49.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.992+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d040d2a20 con 0x7f8d040b9f00 2024-04-17T08:47:49.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.994+0000 7f8cfa000640 1 -- 172.21.15.67:0/845192503 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7f8cf407c140 con 0x7f8d040b9f00 2024-04-17T08:47:49.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:47:49.992 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:47:49.992 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 msgr2=0x7f8ce006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 0x7f8ce006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d0415b060 tx=0x7f8cec009040 comp rx=0 tx=0).stop 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 msgr2=0x7f8d040cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097a0 tx=0x7f8cf40046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 shutdown_connections 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8ce006ca10 0x7f8ce006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8d040d12a0 0x7f8d0415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d040cf8f0 0x7f8d041576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 --2- 172.21.15.67:0/845192503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d040b9f00 0x7f8d040cb150 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.996+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 >> 172.21.15.67:0/845192503 conn(0x7f8d0406cb60 msgr2=0x7f8d040b69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.997+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 shutdown_connections 2024-04-17T08:47:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:49.997+0000 7f8d03200640 1 -- 172.21.15.67:0/845192503 wait complete. 2024-04-17T08:47:50.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:47:43.271811+0000', 'modified': '2024-04-17T08:47:46.355432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10518': {'gid': 10518, 'name': 'c', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/2414893402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 2414893402}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2414893402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10518, 'qdb_cluster': [10518]} 2024-04-17T08:47:50.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:47:50.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T08:47:50.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.650+0000 7f8e4f000640 1 Processor -- start 2024-04-17T08:47:50.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.650+0000 7f8e4f000640 1 -- start start 2024-04-17T08:47:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e480cbc00 0x7f8e480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e480b63c0 0x7f8e48071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e48160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8e480cfe90 con 0x7f8e480b63c0 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4815dc70 con 0x7f8e480cbc00 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4815d060 con 0x7f8e480723a0 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e480b63c0 0x7f8e48071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e480b63c0 0x7f8e48071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58344/0 (socket says 172.21.15.67:58344) 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4d200640 1 -- 172.21.15.67:0/2250711890 learned_addr learned my addr 172.21.15.67:0/2250711890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.652+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/2250711890 >> v2:172.21.15.187:3300/0 conn(0x7f8e480cbc00 0x7f8e480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4e600640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e48160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4d200640 1 -- 172.21.15.67:0/2250711890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 msgr2=0x7f8e48160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e48160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4d200640 1 -- 172.21.15.67:0/2250711890 >> v2:172.21.15.187:3300/0 conn(0x7f8e480cbc00 msgr2=0x7f8e480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2250711890 >> v2:172.21.15.187:3300/0 conn(0x7f8e480cbc00 0x7f8e480cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.653+0000 7f8e4d200640 1 -- 172.21.15.67:0/2250711890 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4815d340 con 0x7f8e480b63c0 2024-04-17T08:47:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.654+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480b63c0 0x7f8e48071e60 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f8e480702d0 tx=0x7f8e3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38247061600b1d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.654+0000 7f8e47e00640 1 -- 172.21.15.67:0/2250711890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34035020 con 0x7f8e480b63c0 2024-04-17T08:47:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e47e00640 1 -- 172.21.15.67:0/2250711890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e34031870 con 0x7f8e480b63c0 2024-04-17T08:47:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e47e00640 1 -- 172.21.15.67:0/2250711890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031c40 con 0x7f8e480b63c0 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 -- 172.21.15.67:0/2250711890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480b63c0 msgr2=0x7f8e48071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480b63c0 0x7f8e48071e60 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f8e480702d0 tx=0x7f8e3402b910 comp rx=0 tx=0).stop 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 -- 172.21.15.67:0/2250711890 shutdown_connections 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e48160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 --2- 172.21.15.67:0/2250711890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480b63c0 0x7f8e48071e60 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 --2- 172.21.15.67:0/2250711890 >> v2:172.21.15.187:3300/0 conn(0x7f8e480cbc00 0x7f8e480cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 -- 172.21.15.67:0/2250711890 >> 172.21.15.67:0/2250711890 conn(0x7f8e4806bef0 msgr2=0x7f8e480b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.655+0000 7f8e4f000640 1 -- 172.21.15.67:0/2250711890 shutdown_connections 2024-04-17T08:47:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.656+0000 7f8e4f000640 1 -- 172.21.15.67:0/2250711890 wait complete. 2024-04-17T08:47:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.657+0000 7f8e4f000640 1 Processor -- start 2024-04-17T08:47:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.657+0000 7f8e4f000640 1 -- start start 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e4815a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 0x7f8e48140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e480d02d0 con 0x7f8e480cbc00 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4815d900 con 0x7f8e480b63c0 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.658+0000 7f8e4f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e480cbfe0 con 0x7f8e480723a0 2024-04-17T08:47:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58360/0 (socket says 172.21.15.67:58360) 2024-04-17T08:47:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 -- 172.21.15.67:0/1073706696 learned_addr learned my addr 172.21.15.67:0/1073706696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4d200640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 0x7f8e48140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e4815a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 -- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 msgr2=0x7f8e4815a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e4815a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 -- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 msgr2=0x7f8e48140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 0x7f8e48140d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4e600640 1 -- 172.21.15.67:0/1073706696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e480b2160 con 0x7f8e480cbc00 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.659+0000 7f8e4d200640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 0x7f8e48140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e4e600640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f8e40007e60 tx=0x7f8e4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e45600640 1 -- 172.21.15.67:0/1073706696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40014620 con 0x7f8e480cbc00 2024-04-17T08:47:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e45600640 1 -- 172.21.15.67:0/1073706696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e40019070 con 0x7f8e480cbc00 2024-04-17T08:47:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e45600640 1 -- 172.21.15.67:0/1073706696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40014a20 con 0x7f8e480cbc00 2024-04-17T08:47:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4814d290 con 0x7f8e480cbc00 2024-04-17T08:47:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.660+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e480b1680 con 0x7f8e480cbc00 2024-04-17T08:47:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.662+0000 7f8e45600640 1 -- 172.21.15.67:0/1073706696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7f8e40014bc0 con 0x7f8e480cbc00 2024-04-17T08:47:50.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.663+0000 7f8e45600640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 0x7f8e2806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.663+0000 7f8e45600640 1 -- 172.21.15.67:0/1073706696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f8e4008c5e0 con 0x7f8e480cbc00 2024-04-17T08:47:50.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.664+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 0x7f8e2806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.664+0000 7f8e53255740 1 Processor -- start 2024-04-17T08:47:50.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.664+0000 7f8e53255740 1 -- start start 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 0x7f8e2806ef00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009a30 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec382c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec37db40 0x5592ec3ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5592ec2e9980 con 0x5592ec3816a0 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5592ec290220 con 0x5592ec3cb620 2024-04-17T08:47:50.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5592ec2e5940 con 0x5592ec37db40 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.665+0000 7f8e4d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58376/0 (socket says 172.21.15.67:58376) 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 -- 172.21.15.67:0/4027115698 learned_addr learned my addr 172.21.15.67:0/4027115698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec382c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 -- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec37db40 msgr2=0x5592ec3ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec37db40 0x5592ec3ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 -- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 msgr2=0x5592ec382c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec382c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 -- 172.21.15.67:0/4027115698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ebffc2d0 con 0x5592ec3816a0 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.666+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f8e3402b950 tx=0x7f8e34008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4154694e144aa428 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e3a000640 1 -- 172.21.15.67:0/4027115698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34035040 con 0x5592ec3816a0 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e3a000640 1 -- 172.21.15.67:0/4027115698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e34031830 con 0x5592ec3816a0 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e3a000640 1 -- 172.21.15.67:0/4027115698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031c20 con 0x5592ec3816a0 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 -- 172.21.15.67:0/4027115698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 msgr2=0x5592ec383190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f8e3402b950 tx=0x7f8e34008030 comp rx=0 tx=0).stop 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 -- 172.21.15.67:0/4027115698 shutdown_connections 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec37db40 0x5592ec3ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec383190 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 --2- 172.21.15.67:0/4027115698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec382c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.667+0000 7f8e53255740 1 -- 172.21.15.67:0/4027115698 >> 172.21.15.67:0/4027115698 conn(0x5592ec37fdb0 msgr2=0x5592ec3c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.668+0000 7f8e53255740 1 -- 172.21.15.67:0/4027115698 shutdown_connections 2024-04-17T08:47:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.668+0000 7f8e53255740 1 -- 172.21.15.67:0/4027115698 wait complete. 2024-04-17T08:47:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.668+0000 7f8e53255740 1 -- start start 2024-04-17T08:47:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.668+0000 7f8e53255740 1 Processor -- start 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec3a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3a8340 0x5592ec3a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5592ec381bb0 con 0x5592ec3816a0 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5592ec2a3590 con 0x5592ec3cb620 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5592ec381e90 con 0x5592ec3a8340 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.669+0000 7f8e53255740 10 client.0 _ll_register_callbacks cb 0x5592ebffbac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58380/0 (socket says 172.21.15.67:58380) 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 -- 172.21.15.67:0/4183111120 learned_addr learned my addr 172.21.15.67:0/4183111120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 -- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3a8340 msgr2=0x5592ec3a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec3a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4e600640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3a8340 0x5592ec3a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3a8340 0x5592ec3a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 -- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 msgr2=0x5592ec3a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec3a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4dc00640 1 -- 172.21.15.67:0/4183111120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592ec0c66b0 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.670+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec3a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0156c0 tx=0x7f8e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca0fde28877ad5c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c026070 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f8e3c0147b0 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e34031830 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f8e3c031050 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e53255740 10 client.10644 fetch_fsmap learned FSMap version 102 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x5592ec3d0b60 con 0x5592ec3816a0 2024-04-17T08:47:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.671+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e34031830 con 0x5592ec3816a0 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 102) ==== 3495+0+0 (secure 0 0 0) 0x7f8e3c034020 con 0x5592ec3816a0 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 10 client.10644 fetch_fsmap finished waiting for FSMap version 102 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 10 client.10644 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 20 client.10644 populate_metadata read hostname 'smithi067' 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3cdfb0 0x5592ec3a7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 4 client.10644 mds_command: new command op to 10518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 --> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592ec2f73a0 con 0x5592ec3cdfb0 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.672+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3cdfb0 0x5592ec3a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.673+0000 7f8e4d200640 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3cdfb0 0x5592ec3a7490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0dc12ac7d2f1217 server_cookie=6e2f9ef5b81d4cbb in_seq=0 out_seq=0 2024-04-17T08:47:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.673+0000 7f8e2ea00640 10 client.10644 ms_handle_connect on v2:172.21.15.67:6833/2414893402 2024-04-17T08:47:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e2ea00640 1 -- 172.21.15.67:0/4183111120 <== mds.0 v2:172.21.15.67:6833/2414893402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592ec2f73a0 con 0x5592ec3cdfb0 2024-04-17T08:47:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e2ea00640 10 client.10644 handle_command_reply: tid=0 2024-04-17T08:47:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 client.10644 shutdown 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 msgr2=0x5592ec3a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0156c0 tx=0x7f8e3c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 shutdown_connections 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3cdfb0 0x5592ec3a7490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3a8340 0x5592ec3a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec3a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 --2- 172.21.15.67:0/4183111120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3816a0 0x5592ec3a1040 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e2ea00640 0 client.10644 ms_handle_reset on v2:172.21.15.67:6833/2414893402 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 >> 172.21.15.67:0/4183111120 conn(0x5592ec37db40 msgr2=0x5592ec37e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 shutdown_connections 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 1 -- 172.21.15.67:0/4183111120 wait complete. 2024-04-17T08:47:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.674+0000 7f8e53255740 20 client.10644 trim_cache size 0 max 16384 2024-04-17T08:47:50.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.694+0000 7f8e53255740 1 Processor -- start 2024-04-17T08:47:50.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 -- start start 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec3828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec37e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e481fb8c0 con 0x5592ec3a8340 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e48156280 con 0x5592ec3cb620 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.695+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e481fb2a0 con 0x5592ec3816a0 2024-04-17T08:47:50.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58384/0 (socket says 172.21.15.67:58384) 2024-04-17T08:47:50.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 -- 172.21.15.67:0/2777915607 learned_addr learned my addr 172.21.15.67:0/2777915607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec3828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4e600640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec37e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 -- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 msgr2=0x5592ec3828c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec3828c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 -- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 msgr2=0x5592ec37e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec37e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4d200640 1 -- 172.21.15.67:0/2777915607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ec3c9500 con 0x5592ec3a8340 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.696+0000 7f8e4e600640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec37e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e4d200640 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009930 tx=0x7f8e34030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbe151049b9e661 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e39600640 1 -- 172.21.15.67:0/2777915607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34035040 con 0x5592ec3a8340 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e39600640 1 -- 172.21.15.67:0/2777915607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3404f420 con 0x5592ec3a8340 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e39600640 1 -- 172.21.15.67:0/2777915607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031e60 con 0x5592ec3a8340 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 -- 172.21.15.67:0/2777915607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 msgr2=0x5592ec3a7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009930 tx=0x7f8e34030000 comp rx=0 tx=0).stop 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 -- 172.21.15.67:0/2777915607 shutdown_connections 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3cb620 0x5592ec37e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3a8340 0x5592ec3a7680 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 --2- 172.21.15.67:0/2777915607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec3828c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.697+0000 7f8e53255740 1 -- 172.21.15.67:0/2777915607 >> 172.21.15.67:0/2777915607 conn(0x5592ec37db40 msgr2=0x5592ec3cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.698+0000 7f8e53255740 1 -- 172.21.15.67:0/2777915607 shutdown_connections 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.698+0000 7f8e53255740 1 -- 172.21.15.67:0/2777915607 wait complete. 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.698+0000 7f8e53255740 1 -- start start 2024-04-17T08:47:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.698+0000 7f8e53255740 1 Processor -- start 2024-04-17T08:47:50.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec447010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 0x5592ec3827c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5592ec3cde30 con 0x5592ec3cb620 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5592ec3cd640 con 0x5592ec3a8340 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.699+0000 7f8e53255740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5592ec3ce450 con 0x5592ec3816a0 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e53255740 10 client.0 _ll_register_callbacks cb 0x7f8e3c009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58398/0 (socket says 172.21.15.67:58398) 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 -- 172.21.15.67:0/3750871510 learned_addr learned my addr 172.21.15.67:0/3750871510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec447010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4d200640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 0x5592ec3827c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 -- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 msgr2=0x5592ec447010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec447010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 -- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 msgr2=0x5592ec3827c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 0x5592ec3827c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4e600640 1 -- 172.21.15.67:0/3750871510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592ec3ad820 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.700+0000 7f8e4d200640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 0x5592ec3827c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e4e600640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f8e40059e40 tx=0x7f8e4005a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32218eb7a9bbde43 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40062d20 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f8e400567b0 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e3404f420 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f8e40062890 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e53255740 10 client.10650 fetch_fsmap learned FSMap version 102 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) -- 0x5592ec382440 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.701+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3404f420 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 102) ==== 3495+0+0 (secure 0 0 0) 0x7f8e4006eb50 con 0x5592ec3cb620 2024-04-17T08:47:50.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 10 client.10650 fetch_fsmap finished waiting for FSMap version 102 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 10 client.10650 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 20 client.10650 populate_metadata read hostname 'smithi067' 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3c9ba0 0x5592ec3c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 4 client.10650 mds_command: new command op to 10518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 --> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] -- command(tid 0: {"prefix": "status"}) -- 0x5592ec2f1210 con 0x5592ec3c9ba0 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3c9ba0 0x5592ec3c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e4dc00640 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3c9ba0 0x5592ec3c9f80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca29a4223f0a37af server_cookie=46fe1a948d0efbd8 in_seq=0 out_seq=0 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.702+0000 7f8e2ea00640 10 client.10650 ms_handle_connect on v2:172.21.15.67:6833/2414893402 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e2ea00640 1 -- 172.21.15.67:0/3750871510 <== mds.0 v2:172.21.15.67:6833/2414893402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x5592ec2f1210 con 0x5592ec3c9ba0 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e2ea00640 10 client.10650 handle_command_reply: tid=0 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 client.10650 shutdown 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 msgr2=0x5592ec382d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f8e40059e40 tx=0x7f8e4005a900 comp rx=0 tx=0).stop 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 shutdown_connections 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5592ec3cb620 0x5592ec382d00 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5592ec3a8340 0x5592ec3827c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.67:6833/2414893402,v1:172.21.15.67:6835/2414893402] conn(0x5592ec3c9ba0 0x5592ec3c9f80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 --2- 172.21.15.67:0/3750871510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5592ec3816a0 0x5592ec447010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 >> 172.21.15.67:0/3750871510 conn(0x5592ec37db40 msgr2=0x5592ec3a0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e2ea00640 0 client.10650 ms_handle_reset on v2:172.21.15.67:6833/2414893402 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 shutdown_connections 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 1 -- 172.21.15.67:0/3750871510 wait complete. 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.703+0000 7f8e53255740 20 client.10650 trim_cache size 0 max 16384 2024-04-17T08:47:50.705 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 10518, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8640675570000003, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 102, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 212, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 212, 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 12.692813205 2024-04-17T08:47:50.706 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:47:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 msgr2=0x7f8e2806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 0x7f8e2806ef00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009a30 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2024-04-17T08:47:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 msgr2=0x7f8e48160330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f8e40007e60 tx=0x7f8e4000e4b0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 shutdown_connections 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f8e2806ca40 0x7f8e2806ef00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e480cbc00 0x7f8e48160330 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e480b63c0 0x7f8e48140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 --2- 172.21.15.67:0/1073706696 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e480723a0 0x7f8e4815a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 >> 172.21.15.67:0/1073706696 conn(0x7f8e4806bef0 msgr2=0x7f8e481570a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 shutdown_connections 2024-04-17T08:47:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.704+0000 7f8e4f000640 1 -- 172.21.15.67:0/1073706696 wait complete. 2024-04-17T08:47:50.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:47:50.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.943+0000 7fcf8a000640 1 Processor -- start 2024-04-17T08:47:50.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.943+0000 7fcf8a000640 1 -- start start 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.944+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf8406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.944+0000 7fcf8a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf8406be70 0x7fcf8415f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.944+0000 7fcf8a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8415bc80 con 0x7fcf8415b570 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf8a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8415bf40 con 0x7fcf8406be70 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf8a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf8415c220 con 0x7fcf8406feb0 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf88c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf8415b570 0x7fcf8406f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41304/0 (socket says 172.21.15.67:41304) 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf88c00640 1 -- 172.21.15.67:0/2497033494 learned_addr learned my addr 172.21.15.67:0/2497033494 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:50.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf89600640 1 --2- 172.21.15.67:0/2497033494 >> v2:172.21.15.67:3300/0 conn(0x7fcf8406be70 0x7fcf8415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf7be00640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf8406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf83e00640 1 -- 172.21.15.67:0/2497033494 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503209727 0 0) 0x7fcf8415bc80 con 0x7fcf8415b570 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf83e00640 1 -- 172.21.15.67:0/2497033494 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf68003520 con 0x7fcf8415b570 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf89600640 1 -- 172.21.15.67:0/2497033494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 msgr2=0x7fcf8406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf89600640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf8406b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf89600640 1 -- 172.21.15.67:0/2497033494 >> v1:172.21.15.67:6789/0 conn(0x7fcf8415b570 legacy=0x7fcf8406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.945+0000 7fcf89600640 1 -- 172.21.15.67:0/2497033494 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8415c500 con 0x7fcf8406be70 2024-04-17T08:47:50.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf89600640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8406be70 0x7fcf8415f140 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90af483384c50b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf83e00640 1 -- 172.21.15.67:0/2497033494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c039070 con 0x7fcf8406be70 2024-04-17T08:47:50.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf83e00640 1 -- 172.21.15.67:0/2497033494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf7c02cbd0 con 0x7fcf8406be70 2024-04-17T08:47:50.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf83e00640 1 -- 172.21.15.67:0/2497033494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c034440 con 0x7fcf8406be70 2024-04-17T08:47:50.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf8a000640 1 -- 172.21.15.67:0/2497033494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8406be70 msgr2=0x7fcf8415f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.947+0000 7fcf8a000640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8406be70 0x7fcf8415f140 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c02be20 comp rx=0 tx=0).stop 2024-04-17T08:47:50.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 -- 172.21.15.67:0/2497033494 shutdown_connections 2024-04-17T08:47:50.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8406be70 0x7fcf8415f140 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 --2- 172.21.15.67:0/2497033494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf8406b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 -- 172.21.15.67:0/2497033494 >> 172.21.15.67:0/2497033494 conn(0x7fcf84067d10 msgr2=0x7fcf840b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:50.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 -- 172.21.15.67:0/2497033494 shutdown_connections 2024-04-17T08:47:50.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.948+0000 7fcf8a000640 1 -- 172.21.15.67:0/2497033494 wait complete. 2024-04-17T08:47:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.949+0000 7fcf8a000640 1 Processor -- start 2024-04-17T08:47:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.950+0000 7fcf8a000640 1 -- start start 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.950+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 0x7fcf84146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf84155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf8a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8406c460 con 0x7fcf8415b570 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf8a000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8406c2e0 con 0x7fcf8406be70 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf8a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf840b47b0 con 0x7fcf8406feb0 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58414/0 (socket says 172.21.15.67:58414) 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 -- 172.21.15.67:0/3074294074 learned_addr learned my addr 172.21.15.67:0/3074294074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf7be00640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf84155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf88c00640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 0x7fcf84146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 -- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 msgr2=0x7fcf84155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf84155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 -- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 msgr2=0x7fcf84146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 0x7fcf84146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf89600640 1 -- 172.21.15.67:0/3074294074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf84147490 con 0x7fcf8415b570 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.951+0000 7fcf88c00640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 0x7fcf84146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf89600640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c039050 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf7c045070 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c009450 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf840b1480 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.952+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf840b1a40 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.953+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf840cff00 con 0x7fcf8415b570 2024-04-17T08:47:50.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.954+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fcf7c03e850 con 0x7fcf8415b570 2024-04-17T08:47:50.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.955+0000 7fcf81600640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 0x7fcf6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.955+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7fcf7c0aeab0 con 0x7fcf8415b570 2024-04-17T08:47:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.956+0000 7fcf88c00640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 0x7fcf6006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:50.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.957+0000 7fcf88c00640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 0x7fcf6006eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf74005880 tx=0x7fcf74009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:50.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:50.960+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcf840cff00 con 0x7fcf8415b570 2024-04-17T08:47:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.207+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcf84071f20 con 0x7fcf8415b570 2024-04-17T08:47:51.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.208+0000 7fcf81600640 1 -- 172.21.15.67:0/3074294074 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v102) ==== 53+0+73 (secure 0 0 0) 0x7fcf7c032e20 con 0x7fcf8415b570 2024-04-17T08:47:51.206 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 msgr2=0x7fcf6006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 0x7fcf6006eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf74005880 tx=0x7fcf74009040 comp rx=0 tx=0).stop 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 msgr2=0x7fcf84155ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c004a40 comp rx=0 tx=0).stop 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 shutdown_connections 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fcf6006ca10 0x7fcf6006eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8415b570 0x7fcf84155ea0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf8406feb0 0x7fcf84155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 --2- 172.21.15.67:0/3074294074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf8406be70 0x7fcf84146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 >> 172.21.15.67:0/3074294074 conn(0x7fcf84067d10 msgr2=0x7fcf840b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 shutdown_connections 2024-04-17T08:47:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51.210+0000 7fcf8a000640 1 -- 172.21.15.67:0/3074294074 wait complete. 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:47:51.230 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:47:51.231 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:47:51.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.231 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:47:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 36973sec preferred_lft 36973sec 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.285 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:47:51.286 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:47:51.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.287 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:47:51.383 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:47:51.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.388 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:47:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:47:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:47:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:47:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:47:51.446 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:47:51.447 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:47:51.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:47:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:47:51.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.719 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:47:51.773 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:47:51.774 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:47:51.775 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:47:51.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:47:51.881 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:47:51.881 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:47:51.935 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:47:51.935 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:47:51.936 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:47:51.984 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:47:51.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:51.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:47:52.120 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:47:52.121 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:47:52.156 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:47:52.156 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:47:52.156 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:47:52.156 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:47:52.156 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:47:52.158 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:47:52.159 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:47:52.159 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:47:52.159 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:47:52.159 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:47:52.164 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:47:52.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:52.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:47:52.250 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:47:52.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:47:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:47:52.393 INFO:teuthology.orchestra.run.smithi067.stdout:client7586 2024-04-17T08:47:52.393 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:47:52.393 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:47:52.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:47:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.715+0000 7ff993000640 1 Processor -- start 2024-04-17T08:47:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.715+0000 7ff993000640 1 -- start start 2024-04-17T08:47:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.717+0000 7ff993000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 0x7ff98c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.717+0000 7ff993000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98c15d060 con 0x7ff98c072570 2024-04-17T08:47:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.717+0000 7ff993000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98c15d340 con 0x7ff98c0cbc00 2024-04-17T08:47:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.717+0000 7ff993000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff98c0cfe90 con 0x7ff98c0b63c0 2024-04-17T08:47:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff991200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff98c072570 0x7ff98c0ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41310/0 (socket says 172.21.15.67:41310) 2024-04-17T08:47:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff991200640 1 -- 172.21.15.67:0/1248562117 learned_addr learned my addr 172.21.15.67:0/1248562117 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff992600640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 0x7ff98c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172346138 0 0) 0x7ff98c15d060 con 0x7ff98c072570 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff974003520 con 0x7ff98c072570 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4261504896 0 0) 0x7ff974003520 con 0x7ff98c072570 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 msgr2=0x7ff98c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 0x7ff98c0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 >> v1:172.21.15.187:6789/0 conn(0x7ff98c0cbc00 legacy=0x7ff98c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c15d620 con 0x7ff98c072570 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff978001670 con 0x7ff98c072570 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.718+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 >> v1:172.21.15.67:6789/0 conn(0x7ff98c072570 legacy=0x7ff98c0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974003a50 0x7ff974003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 0x7ff974004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c0cfe90 con 0x7ff974004d40 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff97400bb50 con 0x7ff974003a50 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff97400bd50 con 0x7ff98c0b63c0 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff991c00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974003a50 0x7ff974003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 -- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 msgr2=0x7ff974004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0b63c0 0x7ff974004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 -- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974003a50 msgr2=0x7ff974003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974003a50 0x7ff974003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff992600640 1 -- 172.21.15.67:0/1248562117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff974003520 con 0x7ff974004d40 2024-04-17T08:47:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.719+0000 7ff991c00640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974003a50 0x7ff974003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff992600640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0b6870 tx=0x7ff974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ee7ca256bafbd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984039070 con 0x7ff974004d40 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff98402cbd0 con 0x7ff974004d40 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff98be00640 1 -- 172.21.15.67:0/1248562117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff984009c10 con 0x7ff974004d40 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff993000640 1 -- 172.21.15.67:0/1248562117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff974004d40 msgr2=0x7ff9740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff993000640 1 --2- 172.21.15.67:0/1248562117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff974004d40 0x7ff9740051c0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0b6870 tx=0x7ff974005830 comp rx=0 tx=0).stop 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.721+0000 7ff992600640 1 -- 172.21.15.67:0/1248562117 reap_dead start 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.722+0000 7ff993000640 1 -- 172.21.15.67:0/1248562117 shutdown_connections 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.722+0000 7ff993000640 1 -- 172.21.15.67:0/1248562117 >> 172.21.15.67:0/1248562117 conn(0x7ff98c06bef0 msgr2=0x7ff98c0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.722+0000 7ff993000640 1 -- 172.21.15.67:0/1248562117 shutdown_connections 2024-04-17T08:47:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.722+0000 7ff993000640 1 -- 172.21.15.67:0/1248562117 wait complete. 2024-04-17T08:47:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.723+0000 7ff993000640 1 Processor -- start 2024-04-17T08:47:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.724+0000 7ff993000640 1 -- start start 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974004d40 0x7ff98c15b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0c9a40 0x7ff98c0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c0d02d0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c15dc70 con 0x7ff974004d40 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff993000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff98c0cbfe0 con 0x7ff98c0c9a40 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58424/0 (socket says 172.21.15.67:58424) 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 -- 172.21.15.67:0/4190041233 learned_addr learned my addr 172.21.15.67:0/4190041233 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991c00640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974004d40 0x7ff98c15b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff992600640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0c9a40 0x7ff98c0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 -- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0c9a40 msgr2=0x7ff98c0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0c9a40 0x7ff98c0c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 -- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974004d40 msgr2=0x7ff98c15b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974004d40 0x7ff98c15b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.725+0000 7ff991200640 1 -- 172.21.15.67:0/4190041233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c157d30 con 0x7ff98c15a9b0 2024-04-17T08:47:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff991200640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c730 tx=0x7ff97800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978018070 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9780072d0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98c158010 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9780047e0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.726+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98c157a00 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.727+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98c158ad0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.728+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7ff97801b070 con 0x7ff98c15a9b0 2024-04-17T08:47:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.729+0000 7ff989600640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 0x7ff96c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.729+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7ff97808b2b0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.730+0000 7ff991c00640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 0x7ff96c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.731+0000 7ff991c00640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 0x7ff96c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff98c156ad0 tx=0x7ff980009040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:52.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.734+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff98c158ad0 con 0x7ff98c15a9b0 2024-04-17T08:47:52.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.964+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 --> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7ff98c068a10 con 0x7ff96c06ca40 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.969+0000 7ff989600640 1 -- 172.21.15.67:0/4190041233 <== mgr.7379 v2:172.21.15.187:6832/2105214300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7ff98c068a10 con 0x7ff96c06ca40 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:47:52.968 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:47:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 msgr2=0x7ff96c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 0x7ff96c06ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff98c156ad0 tx=0x7ff980009040 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 msgr2=0x7ff98c1f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7ff97800c730 tx=0x7ff97800cbf0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 shutdown_connections 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7ff96c06ca40 0x7ff96c06ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98c0c9a40 0x7ff98c0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c15a9b0 0x7ff98c1f4a70 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 --2- 172.21.15.67:0/4190041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff974004d40 0x7ff98c15b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 >> 172.21.15.67:0/4190041233 conn(0x7ff98c06bef0 msgr2=0x7ff98c159490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.971+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 shutdown_connections 2024-04-17T08:47:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:52.972+0000 7ff993000640 1 -- 172.21.15.67:0/4190041233 wait complete. 2024-04-17T08:47:52.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:47:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.206+0000 7fbf3fe00640 1 Processor -- start 2024-04-17T08:47:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.207+0000 7fbf3fe00640 1 -- start start 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf3809aa60 0x7fbf3809ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf380d0280 0x7fbf380d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf381605a0 0x7fbf38160a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3815d350 con 0x7fbf380d0280 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbf380d0c20 con 0x7fbf3809aa60 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf380cd630 con 0x7fbf381605a0 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.208+0000 7fbf3ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf380d0280 0x7fbf380d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf380d0280 0x7fbf380d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58432/0 (socket says 172.21.15.67:58432) 2024-04-17T08:47:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 -- 172.21.15.67:0/623996597 learned_addr learned my addr 172.21.15.67:0/623996597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3d400640 1 --2- 172.21.15.67:0/623996597 >> v2:172.21.15.187:3300/0 conn(0x7fbf3809aa60 0x7fbf3809ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3de00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf381605a0 0x7fbf38160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 -- 172.21.15.67:0/623996597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf381605a0 msgr2=0x7fbf38160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf381605a0 0x7fbf38160a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 -- 172.21.15.67:0/623996597 >> v2:172.21.15.187:3300/0 conn(0x7fbf3809aa60 msgr2=0x7fbf3809ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/623996597 >> v2:172.21.15.187:3300/0 conn(0x7fbf3809aa60 0x7fbf3809ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.209+0000 7fbf3ca00640 1 -- 172.21.15.67:0/623996597 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3809b380 con 0x7fbf380d0280 2024-04-17T08:47:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.210+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380d0280 0x7fbf380d06e0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fbf380cf100 tx=0x7fbf2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0781c89595612fc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:53.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.211+0000 7fbf37e00640 1 -- 172.21.15.67:0/623996597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf24035020 con 0x7fbf380d0280 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.211+0000 7fbf37e00640 1 -- 172.21.15.67:0/623996597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf24031640 con 0x7fbf380d0280 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.211+0000 7fbf37e00640 1 -- 172.21.15.67:0/623996597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2402d040 con 0x7fbf380d0280 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.211+0000 7fbf3fe00640 1 -- 172.21.15.67:0/623996597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380d0280 msgr2=0x7fbf380d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.211+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380d0280 0x7fbf380d06e0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fbf380cf100 tx=0x7fbf2402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 -- 172.21.15.67:0/623996597 shutdown_connections 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf381605a0 0x7fbf38160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/623996597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf380d0280 0x7fbf380d06e0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/623996597 >> v2:172.21.15.187:3300/0 conn(0x7fbf3809aa60 0x7fbf3809ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 -- 172.21.15.67:0/623996597 >> 172.21.15.67:0/623996597 conn(0x7fbf380b41a0 msgr2=0x7fbf380b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 -- 172.21.15.67:0/623996597 shutdown_connections 2024-04-17T08:47:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.212+0000 7fbf3fe00640 1 -- 172.21.15.67:0/623996597 wait complete. 2024-04-17T08:47:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.213+0000 7fbf3fe00640 1 Processor -- start 2024-04-17T08:47:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.214+0000 7fbf3fe00640 1 -- start start 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.214+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf380d0280 0x7fbf38155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf381605a0 0x7fbf3815f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf380d1420 con 0x7fbf3809aa60 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbf380d1140 con 0x7fbf381605a0 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf380d1700 con 0x7fbf380d0280 2024-04-17T08:47:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58440/0 (socket says 172.21.15.67:58440) 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 -- 172.21.15.67:0/4132491704 learned_addr learned my addr 172.21.15.67:0/4132491704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3de00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf381605a0 0x7fbf3815f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf380d0280 0x7fbf38155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 -- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf380d0280 msgr2=0x7fbf38155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf380d0280 0x7fbf38155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 -- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf381605a0 msgr2=0x7fbf3815f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf381605a0 0x7fbf3815f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.215+0000 7fbf3d400640 1 -- 172.21.15.67:0/4132491704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf381f3b60 con 0x7fbf3809aa60 2024-04-17T08:47:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf3d400640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800d800 tx=0x7fbf2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28018020 con 0x7fbf3809aa60 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf280096a0 con 0x7fbf3809aa60 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28009a70 con 0x7fbf3809aa60 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf380b5830 con 0x7fbf3809aa60 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.216+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf380b5df0 con 0x7fbf3809aa60 2024-04-17T08:47:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.217+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf38068ca0 con 0x7fbf3809aa60 2024-04-17T08:47:53.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.219+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 117) ==== 91142+0+0 (secure 0 0 0) 0x7fbf2801b020 con 0x7fbf3809aa60 2024-04-17T08:47:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.220+0000 7fbf35600640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 0x7fbf1806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.220+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7fbf2808bfa0 con 0x7fbf3809aa60 2024-04-17T08:47:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.220+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 0x7fbf1806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.221+0000 7fbf3ca00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 0x7fbf1806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf24036000 tx=0x7fbf2403c040 comp rx=0 tx=0).ready entity=mgr.7379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.224+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbf28022030 con 0x7fbf3809aa60 2024-04-17T08:47:53.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:53.482+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fbf38160380 con 0x7fbf3809aa60 2024-04-17T08:47:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.251+0000 7fbf35600640 1 -- 172.21.15.67:0/4132491704 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) ==== 92+0+0 (secure 0 0 0) 0x7fbf381f3b60 con 0x7fbf3809aa60 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.254+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 msgr2=0x7fbf1806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.254+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 0x7fbf1806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf24036000 tx=0x7fbf2403c040 comp rx=0 tx=0).stop 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.254+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 msgr2=0x7fbf381f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.254+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800d800 tx=0x7fbf2800dcc0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 shutdown_connections 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fbf1806ca10 0x7fbf1806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf381605a0 0x7fbf3815f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf380d0280 0x7fbf38155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 --2- 172.21.15.67:0/4132491704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf3809aa60 0x7fbf381f3620 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.255+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 >> 172.21.15.67:0/4132491704 conn(0x7fbf380b41a0 msgr2=0x7fbf380ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.257+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 shutdown_connections 2024-04-17T08:47:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.257+0000 7fbf3fe00640 1 -- 172.21.15.67:0/4132491704 wait complete. 2024-04-17T08:47:54.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:47:54.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.501+0000 7fb580800640 1 Processor -- start 2024-04-17T08:47:54.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.501+0000 7fb580800640 1 -- start start 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb580800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578145ac0 0x7fb578071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb580800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb578072d80 0x7fb5780cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb580800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5780cea70 con 0x7fb5780ce3b0 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb580800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb5780ce810 con 0x7fb578072d80 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb580800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5780ced50 con 0x7fb578145ac0 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb57d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5780ce3b0 0x7fb578072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43174/0 (socket says 172.21.15.67:43174) 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb57d400640 1 -- 172.21.15.67:0/2069142566 learned_addr learned my addr 172.21.15.67:0/2069142566 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.503+0000 7fb57e800640 1 --2- 172.21.15.67:0/2069142566 >> v2:172.21.15.67:3300/0 conn(0x7fb578072d80 0x7fb5780cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57de00640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578145ac0 0x7fb578071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57e800640 1 -- 172.21.15.67:0/2069142566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578145ac0 msgr2=0x7fb578071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57e800640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578145ac0 0x7fb578071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57e800640 1 -- 172.21.15.67:0/2069142566 >> v1:172.21.15.67:6789/0 conn(0x7fb5780ce3b0 legacy=0x7fb578072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57ca00640 1 -- 172.21.15.67:0/2069142566 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742202905 0 0) 0x7fb5780cea70 con 0x7fb5780ce3b0 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.504+0000 7fb57e800640 1 -- 172.21.15.67:0/2069142566 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5780cf030 con 0x7fb578072d80 2024-04-17T08:47:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.505+0000 7fb57e800640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb578072d80 0x7fb5780cfb10 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d3eb68aebad565 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:54.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb57ca00640 1 -- 172.21.15.67:0/2069142566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574039070 con 0x7fb578072d80 2024-04-17T08:47:54.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb57ca00640 1 -- 172.21.15.67:0/2069142566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57402cc50 con 0x7fb578072d80 2024-04-17T08:47:54.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb57ca00640 1 -- 172.21.15.67:0/2069142566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574034440 con 0x7fb578072d80 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb580800640 1 -- 172.21.15.67:0/2069142566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb578072d80 msgr2=0x7fb5780cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb580800640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb578072d80 0x7fb5780cfb10 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402bea0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb580800640 1 -- 172.21.15.67:0/2069142566 shutdown_connections 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb580800640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb578072d80 0x7fb5780cfb10 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.506+0000 7fb580800640 1 --2- 172.21.15.67:0/2069142566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578145ac0 0x7fb578071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.507+0000 7fb580800640 1 -- 172.21.15.67:0/2069142566 >> 172.21.15.67:0/2069142566 conn(0x7fb578067ee0 msgr2=0x7fb57806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.507+0000 7fb580800640 1 -- 172.21.15.67:0/2069142566 shutdown_connections 2024-04-17T08:47:54.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.507+0000 7fb580800640 1 -- 172.21.15.67:0/2069142566 wait complete. 2024-04-17T08:47:54.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.508+0000 7fb580800640 1 Processor -- start 2024-04-17T08:47:54.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.509+0000 7fb580800640 1 -- start start 2024-04-17T08:47:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.509+0000 7fb580800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 0x7fb578141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb580800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb580800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb578145ac0 0x7fb57815c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb580800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb578145ea0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb580800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb5780cf5f0 con 0x7fb578145ac0 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb580800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5780d0050 con 0x7fb578072d80 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57144/0 (socket says 172.21.15.67:57144) 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 -- 172.21.15.67:0/3589332719 learned_addr learned my addr 172.21.15.67:0/3589332719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57e800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb578145ac0 0x7fb57815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57de00640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 0x7fb578141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 msgr2=0x7fb578141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 0x7fb578141f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb578145ac0 msgr2=0x7fb57815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.510+0000 7fb57d400640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb578145ac0 0x7fb57815c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb57d400640 1 -- 172.21.15.67:0/3589332719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57815eae0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb57de00640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 0x7fb578141f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb57d400640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c4d0 tx=0x7fb56c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c018070 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c007500 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c004a20 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57815edc0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.511+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57815f380 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.512+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5780cc6c0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.513+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 118) ==== 91159+0+0 (secure 0 0 0) 0x7fb56c004bc0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.514+0000 7fb572000640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 0x7fb55c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.515+0000 7fb57de00640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 msgr2=0x7fb55c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.515+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7fb56c08aff0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.515+0000 7fb57de00640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 0x7fb55c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.519+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb56c058c40 con 0x7fb5780ce3b0 2024-04-17T08:47:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.716+0000 7fb57de00640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 msgr2=0x7fb55c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.716+0000 7fb57de00640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 0x7fb55c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.723+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fb5780685d0 con 0x7fb5780ce3b0 2024-04-17T08:47:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.730+0000 7fb572000640 1 -- 172.21.15.67:0/3589332719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) ==== 138+0+0 (secure 0 0 0) 0x7fb56c05cf40 con 0x7fb5780ce3b0 2024-04-17T08:47:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 msgr2=0x7fb55c06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 0x7fb55c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 msgr2=0x7fb57815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00c4d0 tx=0x7fb56c00c990 comp rx=0 tx=0).stop 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 shutdown_connections 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7fb55c06ca60 0x7fb55c06ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb578145ac0 0x7fb57815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5780ce3b0 0x7fb57815bd10 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 --2- 172.21.15.67:0/3589332719 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb578072d80 0x7fb578141f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 >> 172.21.15.67:0/3589332719 conn(0x7fb578067ee0 msgr2=0x7fb57806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 shutdown_connections 2024-04-17T08:47:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.733+0000 7fb580800640 1 -- 172.21.15.67:0/3589332719 wait complete. 2024-04-17T08:47:54.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:47:54.800 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:54.800+0000 7fbb62e3d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:47:54.829 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:54.830+0000 7f5a74830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:47:54.854 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:54.855+0000 7fbb62e3d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:47:54.887 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:54.888+0000 7f5a74830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:47:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.957+0000 7f1e9b200640 1 Processor -- start 2024-04-17T08:47:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.958+0000 7f1e9b200640 1 -- start start 2024-04-17T08:47:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e9c09acd0 0x7f1e9c161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c09b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e9c09b0b0 0x7f1e9c15f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9c0cde40 con 0x7f1e9c09acd0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1e9c09bce0 con 0x7f1e9c09b0b0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.959+0000 7f1e9b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c0cd510 con 0x7f1e9c0ce2d0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e9c09acd0 0x7f1e9c161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e9c09acd0 0x7f1e9c161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57160/0 (socket says 172.21.15.67:57160) 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 -- 172.21.15.67:0/219175128 learned_addr learned my addr 172.21.15.67:0/219175128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e9a800640 1 --2- 172.21.15.67:0/219175128 >> v2:172.21.15.187:3300/0 conn(0x7f1e9c09b0b0 0x7f1e9c15f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e93e00640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c09b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 -- 172.21.15.67:0/219175128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 msgr2=0x7f1e9c09b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c09b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 -- 172.21.15.67:0/219175128 >> v2:172.21.15.187:3300/0 conn(0x7f1e9c09b0b0 msgr2=0x7f1e9c15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 --2- 172.21.15.67:0/219175128 >> v2:172.21.15.187:3300/0 conn(0x7f1e9c09b0b0 0x7f1e9c15f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.960+0000 7f1e99e00640 1 -- 172.21.15.67:0/219175128 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c0cd7f0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.961+0000 7f1e99e00640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c161100 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0d06b0 tx=0x7f1e8c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c893f769b26acbc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e99400640 1 -- 172.21.15.67:0/219175128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0317d0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e99400640 1 -- 172.21.15.67:0/219175128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c037070 con 0x7f1e9c09acd0 2024-04-17T08:47:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e99400640 1 -- 172.21.15.67:0/219175128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c031b90 con 0x7f1e9c09acd0 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 -- 172.21.15.67:0/219175128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 msgr2=0x7f1e9c161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c161100 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0d06b0 tx=0x7f1e8c02b870 comp rx=0 tx=0).stop 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 -- 172.21.15.67:0/219175128 shutdown_connections 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 --2- 172.21.15.67:0/219175128 >> v2:172.21.15.187:3300/0 conn(0x7f1e9c09b0b0 0x7f1e9c15f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c09b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 --2- 172.21.15.67:0/219175128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c161100 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.962+0000 7f1e9b200640 1 -- 172.21.15.67:0/219175128 >> 172.21.15.67:0/219175128 conn(0x7f1e9c0b3ff0 msgr2=0x7f1e9c0b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.963+0000 7f1e9b200640 1 -- 172.21.15.67:0/219175128 shutdown_connections 2024-04-17T08:47:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.963+0000 7f1e9b200640 1 -- 172.21.15.67:0/219175128 wait complete. 2024-04-17T08:47:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.964+0000 7f1e9b200640 1 Processor -- start 2024-04-17T08:47:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.964+0000 7f1e9b200640 1 -- start start 2024-04-17T08:47:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e9c09b0b0 0x7f1e9c15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c0c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c15fed0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c0cd1c0 con 0x7f1e9c09b0b0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.965+0000 7f1e9b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c160050 con 0x7f1e9c0ce2d0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57170/0 (socket says 172.21.15.67:57170) 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 -- 172.21.15.67:0/1403433021 learned_addr learned my addr 172.21.15.67:0/1403433021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e93e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e9c09b0b0 0x7f1e9c15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 msgr2=0x7f1e9c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e9a800640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c0c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c0c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e9c09b0b0 msgr2=0x7f1e9c15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e9c09b0b0 0x7f1e9c15d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.966+0000 7f1e99e00640 1 -- 172.21.15.67:0/1403433021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c0c7600 con 0x7f1e9c09acd0 2024-04-17T08:47:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e99e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c032660 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c032840 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c032c30 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c0c78e0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.967+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c0c7ea0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.968+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9c0755d0 con 0x7f1e9c09acd0 2024-04-17T08:47:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.969+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 118) ==== 91159+0+0 (secure 0 0 0) 0x7f1e8c04c060 con 0x7f1e9c09acd0 2024-04-17T08:47:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.971+0000 7f1e92000640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 0x7f1e7806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.971+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f1e8c0ad820 con 0x7f1e9c09acd0 2024-04-17T08:47:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.971+0000 7f1e93e00640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 msgr2=0x7f1e7806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.971+0000 7f1e93e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 0x7f1e7806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:54.976+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e8c040070 con 0x7f1e9c09acd0 2024-04-17T08:47:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.172+0000 7f1e93e00640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 msgr2=0x7f1e7806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.172+0000 7f1e93e00640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 0x7f1e7806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.178+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1e9c15da70 con 0x7f1e9c09acd0 2024-04-17T08:47:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.186+0000 7f1e92000640 1 -- 172.21.15.67:0/1403433021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) ==== 133+0+0 (secure 0 0 0) 0x7f1e8c07ba00 con 0x7f1e9c09acd0 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 msgr2=0x7f1e7806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 0x7f1e7806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 msgr2=0x7f1e9c06e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 shutdown_connections 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f1e7806ca60 0x7f1e7806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e9c0ce2d0 0x7f1e9c0c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e9c09b0b0 0x7f1e9c15d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 --2- 172.21.15.67:0/1403433021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e9c09acd0 0x7f1e9c06e5c0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.188+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 >> 172.21.15.67:0/1403433021 conn(0x7f1e9c0b3ff0 msgr2=0x7f1e9c0b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.189+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 shutdown_connections 2024-04-17T08:47:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.189+0000 7f1e9b200640 1 -- 172.21.15.67:0/1403433021 wait complete. 2024-04-17T08:47:55.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:47:55.207 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T08:47:55.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-17T08:47:55.299 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.299+0000 7fbb62e3d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:47:55.350 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.351+0000 7fbb62e3d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:47:55.365 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.366+0000 7f5a74830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:47:55.401 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.402+0000 7fbb62e3d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:47:55.421 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.422+0000 7f5a74830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:47:55.461 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.462+0000 7fbb62e3d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:47:55.474 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.475+0000 7f5a74830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:47:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.479+0000 7f1872600640 1 Processor -- start 2024-04-17T08:47:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.479+0000 7f1872600640 1 -- start start 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f186c15e9b0 0x7f186c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c0cf910 0x7f186c0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186c0b4c30 0x7f186c0d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f186c0b5790 con 0x7f186c0b4c30 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f186c0d2830 con 0x7f186c15e9b0 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.480+0000 7f1872600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f186c0b51d0 con 0x7f186c0cf910 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186c0b4c30 0x7f186c0d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f186c0b4c30 0x7f186c0d4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57174/0 (socket says 172.21.15.67:57174) 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 -- 172.21.15.67:0/1862287695 learned_addr learned my addr 172.21.15.67:0/1862287695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871200640 1 --2- 172.21.15.67:0/1862287695 >> v2:172.21.15.187:3300/0 conn(0x7f186c15e9b0 0x7f186c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f186be00640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c0cf910 0x7f186c0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 -- 172.21.15.67:0/1862287695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c0cf910 msgr2=0x7f186c0b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c0cf910 0x7f186c0b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 -- 172.21.15.67:0/1862287695 >> v2:172.21.15.187:3300/0 conn(0x7f186c15e9b0 msgr2=0x7f186c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.481+0000 7f1871c00640 1 --2- 172.21.15.67:0/1862287695 >> v2:172.21.15.187:3300/0 conn(0x7f186c15e9b0 0x7f186c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.482+0000 7f1871c00640 1 -- 172.21.15.67:0/1862287695 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186c0b54b0 con 0x7f186c0b4c30 2024-04-17T08:47:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.483+0000 7f1871c00640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0b4c30 0x7f186c0d4510 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f186c09e080 tx=0x7f186402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db9d559efe57ca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f186b400640 1 -- 172.21.15.67:0/1862287695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186402e070 con 0x7f186c0b4c30 2024-04-17T08:47:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f186b400640 1 -- 172.21.15.67:0/1862287695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1864037070 con 0x7f186c0b4c30 2024-04-17T08:47:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f186b400640 1 -- 172.21.15.67:0/1862287695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864031a00 con 0x7f186c0b4c30 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 -- 172.21.15.67:0/1862287695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0b4c30 msgr2=0x7f186c0d4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0b4c30 0x7f186c0d4510 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f186c09e080 tx=0x7f186402b880 comp rx=0 tx=0).stop 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 -- 172.21.15.67:0/1862287695 shutdown_connections 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0b4c30 0x7f186c0d4510 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 --2- 172.21.15.67:0/1862287695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c0cf910 0x7f186c0b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 --2- 172.21.15.67:0/1862287695 >> v2:172.21.15.187:3300/0 conn(0x7f186c15e9b0 0x7f186c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.484+0000 7f1872600640 1 -- 172.21.15.67:0/1862287695 >> 172.21.15.67:0/1862287695 conn(0x7f186c09daa0 msgr2=0x7f186c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.485+0000 7f1872600640 1 -- 172.21.15.67:0/1862287695 shutdown_connections 2024-04-17T08:47:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.485+0000 7f1872600640 1 -- 172.21.15.67:0/1862287695 wait complete. 2024-04-17T08:47:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.486+0000 7f1872600640 1 Processor -- start 2024-04-17T08:47:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 -- start start 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186c0b4c30 0x7f186c0c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 0x7f186c0c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c15e9b0 0x7f186c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f186c15ed90 con 0x7f186c0cf910 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f186c162830 con 0x7f186c0b4c30 2024-04-17T08:47:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.487+0000 7f1872600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f186c15ef10 con 0x7f186c15e9b0 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 0x7f186c0c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 0x7f186c0c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57188/0 (socket says 172.21.15.67:57188) 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 -- 172.21.15.67:0/1301057906 learned_addr learned my addr 172.21.15.67:0/1301057906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c15e9b0 msgr2=0x7f186c0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f186c15e9b0 0x7f186c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186c0b4c30 msgr2=0x7f186c0c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f186c0b4c30 0x7f186c0c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186c160070 con 0x7f186c0cf910 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.488+0000 7f186be00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 0x7f186c0c9190 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f185800d7d0 tx=0x7f185800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.489+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18580096a0 con 0x7f186c0cf910 2024-04-17T08:47:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.489+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1858019070 con 0x7f186c0cf910 2024-04-17T08:47:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.489+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858009a60 con 0x7f186c0cf910 2024-04-17T08:47:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.489+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186c160350 con 0x7f186c0cf910 2024-04-17T08:47:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.489+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186c160910 con 0x7f186c0cf910 2024-04-17T08:47:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.490+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186c0755d0 con 0x7f186c0cf910 2024-04-17T08:47:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.491+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 118) ==== 91159+0+0 (secure 0 0 0) 0x7f1858009c00 con 0x7f186c0cf910 2024-04-17T08:47:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.492+0000 7f1868c00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.492+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5737+0+0 (secure 0 0 0) 0x7f185808c1d0 con 0x7f186c0cf910 2024-04-17T08:47:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.492+0000 7f1871200640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:55.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.492+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:47:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.497+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f185805a4c0 con 0x7f186c0cf910 2024-04-17T08:47:55.511 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.512+0000 7fbb62e3d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:47:55.542 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.542+0000 7f5a74830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:47:55.594 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.595+0000 7f5a74830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:47:55.639 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.640+0000 7fbb62e3d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:47:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.693+0000 7f1871200640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.693+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:47:55.701 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.702+0000 7fbb62e3d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:55.727+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f186c0b60f0 con 0x7f184c06ca60 2024-04-17T08:47:55.729 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.730+0000 7f5a74830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:47:55.752 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.752+0000 7fbb62e3d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:47:55.795 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.795+0000 7f5a74830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:55.808 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.809+0000 7fbb62e3d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:47:55.847 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.847+0000 7f5a74830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:47:55.859 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:55.860+0000 7fbb62e3d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:47:55.907 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.908+0000 7f5a74830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:47:55.960 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:55.960+0000 7f5a74830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:47:56.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:56.094+0000 7f1871200640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:56.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:56.094+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:47:56.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:47:56.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:47:56.249 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.250+0000 7fbb62e3d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:47:56.364 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.364+0000 7f5a74830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:47:56.411 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.412+0000 7fbb62e3d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:47:56.520 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.521+0000 7fbb62e3d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:47:56.531 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.532+0000 7f5a74830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:47:56.571 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.572+0000 7fbb62e3d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:47:56.645 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.646+0000 7f5a74830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:47:56.697 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.698+0000 7fbb62e3d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:56.698 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.699+0000 7f5a74830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:47:56.804 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.805+0000 7fbb62e3d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:47:56.830 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.830+0000 7f5a74830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:56.895+0000 7f1871200640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:56.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:56.895+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:47:56.942 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:56.942+0000 7f5a74830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:47:56.978 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:56.979+0000 7fbb62e3d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:57.027 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:57.028+0000 7fbb62e3d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:57.125 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:57.125+0000 7f5a74830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:47:57.163 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:57.164+0000 7fbb62e3d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:47:57.176 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:57.177+0000 7f5a74830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:47:57.319 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:57.319+0000 7f5a74830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:47:58.003 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.003+0000 7fbb62e3d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:47:58.164 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:58.165+0000 7f5a74830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:47:58.308 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.308+0000 7fbb62e3d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:47:58.365 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.365+0000 7fbb62e3d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:47:58.482 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:58.482+0000 7f5a74830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:47:58.497 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.497+0000 7fbb62e3d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:47:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:58.497+0000 7f1871200640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2105214300 2024-04-17T08:47:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:58.498+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T08:47:58.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:58.520+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 119) ==== 90768+0+0 (secure 0 0 0) 0x7f1858056430 con 0x7f186c0cf910 2024-04-17T08:47:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:58.521+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 msgr2=0x7f184c06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:47:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:58.521+0000 7f1868c00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/2105214300,v1:172.21.15.187:6833/2105214300] conn(0x7f184c06ca60 0x7f184c06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:58.549 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:58.550+0000 7f5a74830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:47:58.713 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:47:58.713+0000 7f5a74830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:47:58.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.837+0000 7fbad9800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.838+0000 7fbad9800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.838+0000 7fbad9800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.838+0000 7fbad9800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.838+0000 7fbad9800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.839+0000 7fbad5c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.839+0000 7fbad5c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.839+0000 7fbad5c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.839+0000 7fbad5c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:58.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:47:58.839+0000 7fbad5c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:47:59.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.531+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 120) ==== 90895+0+0 (secure 0 0 0) 0x7f185801c360 con 0x7f186c0cf910 2024-04-17T08:47:59.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.532+0000 7f1868c00640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f184c074970 0x7f184c076d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:59.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.533+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 --> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f185808bfc0 con 0x7f184c074970 2024-04-17T08:47:59.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.535+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f184c074970 0x7f184c076d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.542+0000 7f1871200640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f184c074970 0x7f184c076d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f186c0d4fb0 tx=0x7f185c0073b0 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:59.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.730+0000 7f1868c00640 1 -- 172.21.15.67:0/1301057906 <== mgr.8449 v2:172.21.15.187:6832/953837164 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f185808bfc0 con 0x7f184c074970 2024-04-17T08:47:59.730 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T08:47:59.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f184c074970 msgr2=0x7f184c076d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:59.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f184c074970 0x7f184c076d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f186c0d4fb0 tx=0x7f185c0073b0 comp rx=0 tx=0).stop 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 msgr2=0x7f186c0c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 --2- 172.21.15.67:0/1301057906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f186c0cf910 0x7f186c0c9190 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f185800d7d0 tx=0x7f185800dc90 comp rx=0 tx=0).stop 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1871c00640 1 -- 172.21.15.67:0/1301057906 reap_dead start 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 shutdown_connections 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 >> 172.21.15.67:0/1301057906 conn(0x7f186c09daa0 msgr2=0x7f186c0b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 shutdown_connections 2024-04-17T08:47:59.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.734+0000 7f1872600640 1 -- 172.21.15.67:0/1301057906 wait complete. 2024-04-17T08:47:59.759 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T08:47:59.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-17T08:47:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.993+0000 7fd3e7800640 1 Processor -- start 2024-04-17T08:47:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.993+0000 7fd3e7800640 1 -- start start 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.994+0000 7fd3e7800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00723e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e7800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3e00ce3f0 0x7fd3e00ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e7800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e00cedc0 con 0x7fd3e0146010 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e7800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e0072920 con 0x7fd3e00ce3f0 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e7800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e00cf320 con 0x7fd3e0071f80 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd3e0146010 0x7fd3e006e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43190/0 (socket says 172.21.15.67:43190) 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e4e00640 1 -- 172.21.15.67:0/553836279 learned_addr learned my addr 172.21.15.67:0/553836279 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:47:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3e5800640 1 --2- 172.21.15.67:0/553836279 >> v2:172.21.15.67:3300/0 conn(0x7fd3e00ce3f0 0x7fd3e00ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:59.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3dfe00640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00723e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.995+0000 7fd3df400640 1 -- 172.21.15.67:0/553836279 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429692727 0 0) 0x7fd3d0001110 con 0x7fd3e0146010 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.996+0000 7fd3df400640 1 -- 172.21.15.67:0/553836279 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c4003520 con 0x7fd3e0146010 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.996+0000 7fd3e5800640 1 -- 172.21.15.67:0/553836279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 msgr2=0x7fd3e00723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.996+0000 7fd3e5800640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00723e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.996+0000 7fd3e5800640 1 -- 172.21.15.67:0/553836279 >> v1:172.21.15.67:6789/0 conn(0x7fd3e0146010 legacy=0x7fd3e006e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:59.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.996+0000 7fd3e5800640 1 -- 172.21.15.67:0/553836279 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e00cf040 con 0x7fd3e00ce3f0 2024-04-17T08:47:59.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.997+0000 7fd3e5800640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ce880 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009d90 tx=0x7fd3d802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d698aa8c27858bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:47:59.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.997+0000 7fd3df400640 1 -- 172.21.15.67:0/553836279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8039070 con 0x7fd3e00ce3f0 2024-04-17T08:47:59.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.997+0000 7fd3df400640 1 -- 172.21.15.67:0/553836279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d802cbc0 con 0x7fd3e00ce3f0 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.997+0000 7fd3df400640 1 -- 172.21.15.67:0/553836279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8034440 con 0x7fd3e00ce3f0 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 -- 172.21.15.67:0/553836279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e00ce3f0 msgr2=0x7fd3e00ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ce880 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009d90 tx=0x7fd3d802be10 comp rx=0 tx=0).stop 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 -- 172.21.15.67:0/553836279 shutdown_connections 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ce880 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 --2- 172.21.15.67:0/553836279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00723e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 -- 172.21.15.67:0/553836279 >> 172.21.15.67:0/553836279 conn(0x7fd3e006b8c0 msgr2=0x7fd3e006bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 -- 172.21.15.67:0/553836279 shutdown_connections 2024-04-17T08:47:59.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:47:59.998+0000 7fd3e7800640 1 -- 172.21.15.67:0/553836279 wait complete. 2024-04-17T08:47:59.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.000+0000 7fd3e7800640 1 Processor -- start 2024-04-17T08:47:59.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.000+0000 7fd3e7800640 1 -- start start 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e006e3f0 con 0x7fd3e0146010 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e006e110 con 0x7fd3e00ce3f0 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e7800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e01463f0 con 0x7fd3e0071f80 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57212/0 (socket says 172.21.15.67:57212) 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 -- 172.21.15.67:0/1736666068 learned_addr learned my addr 172.21.15.67:0/1736666068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e4e00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3dfe00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 -- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 msgr2=0x7fd3e00adbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00adbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 -- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 msgr2=0x7fd3e00ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e5800640 1 -- 172.21.15.67:0/1736666068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e0142cc0 con 0x7fd3e0146010 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3dfe00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.001+0000 7fd3e4e00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3e5800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009eb0 tx=0x7fd3d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8039050 con 0x7fd3e0146010 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d8045070 con 0x7fd3e0146010 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d803e830 con 0x7fd3e0146010 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e01361d0 con 0x7fd3e0146010 2024-04-17T08:48:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.002+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e0143240 con 0x7fd3e0146010 2024-04-17T08:48:00.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.003+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e0073a50 con 0x7fd3e0146010 2024-04-17T08:48:00.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.004+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 120) ==== 90895+0+0 (secure 0 0 0) 0x7fd3d803e9d0 con 0x7fd3e0146010 2024-04-17T08:48:00.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.005+0000 7fd3dcc00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 0x7fd3c006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.005+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 6081+0+0 (secure 0 0 0) 0x7fd3d80ae7f0 con 0x7fd3e0146010 2024-04-17T08:48:00.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.006+0000 7fd3e4e00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 0x7fd3c006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.007+0000 7fd3e4e00640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 0x7fd3c006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00012b0 tx=0x7fd3d0009040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:48:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.010+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3d807cd10 con 0x7fd3e0146010 2024-04-17T08:48:00.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.244+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 --> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd3e0068d60 con 0x7fd3c006c790 2024-04-17T08:48:00.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.420+0000 7fd3dcc00640 1 -- 172.21.15.67:0/1736666068 <== mgr.8449 v2:172.21.15.187:6832/953837164 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7fd3e0068d60 con 0x7fd3c006c790 2024-04-17T08:48:00.420 INFO:teuthology.orchestra.run.smithi067.stdout:Retention added to path snap_test_dir1 2024-04-17T08:48:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 msgr2=0x7fd3c006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 0x7fd3c006ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00012b0 tx=0x7fd3d0009040 comp rx=0 tx=0).stop 2024-04-17T08:48:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 msgr2=0x7fd3e01367a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009eb0 tx=0x7fd3d8004a40 comp rx=0 tx=0).stop 2024-04-17T08:48:00.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 shutdown_connections 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd3c006c790 0x7fd3c006ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e0146010 0x7fd3e01367a0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3e00ce3f0 0x7fd3e00ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 --2- 172.21.15.67:0/1736666068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3e0071f80 0x7fd3e00adbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 >> 172.21.15.67:0/1736666068 conn(0x7fd3e006b8c0 msgr2=0x7fd3e015aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 shutdown_connections 2024-04-17T08:48:00.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.424+0000 7fd3e7800640 1 -- 172.21.15.67:0/1736666068 wait complete. 2024-04-17T08:48:00.447 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-17T08:48:00.447 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-17-08_49_00 in ~60s... 2024-04-17T08:48:00.447 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-17T08:48:00.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-17T08:48:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.681+0000 7fd2b2a00640 1 Processor -- start 2024-04-17T08:48:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.681+0000 7fd2b2a00640 1 -- start start 2024-04-17T08:48:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2ac145ac0 0x7fd2ac071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 0x7fd2ac072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b2a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ac0cea70 con 0x7fd2ac072bb0 2024-04-17T08:48:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b2a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd2ac0ce810 con 0x7fd2ac145ac0 2024-04-17T08:48:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b2a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac0ced50 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2abe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2ac145ac0 0x7fd2ac071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b0a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2ac072bb0 0x7fd2ac0ccea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43194/0 (socket says 172.21.15.67:43194) 2024-04-17T08:48:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2b0a00640 1 -- 172.21.15.67:0/4067078725 learned_addr learned my addr 172.21.15.67:0/4067078725 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.683+0000 7fd2ab400640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 0x7fd2ac072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.684+0000 7fd2abe00640 1 -- 172.21.15.67:0/4067078725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 msgr2=0x7fd2ac072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.684+0000 7fd2abe00640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 0x7fd2ac072670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.684+0000 7fd2abe00640 1 -- 172.21.15.67:0/4067078725 >> v1:172.21.15.67:6789/0 conn(0x7fd2ac072bb0 legacy=0x7fd2ac0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.684+0000 7fd2abe00640 1 -- 172.21.15.67:0/4067078725 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac0cf030 con 0x7fd2ac145ac0 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.684+0000 7fd2ab400640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 0x7fd2ac072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:48:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.685+0000 7fd2abe00640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac145ac0 0x7fd2ac071ea0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097a0 tx=0x7fd29c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efcc17c4c148ee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:48:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.685+0000 7fd2aaa00640 1 -- 172.21.15.67:0/4067078725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c035020 con 0x7fd2ac145ac0 2024-04-17T08:48:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.686+0000 7fd2aaa00640 1 -- 172.21.15.67:0/4067078725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd29c031810 con 0x7fd2ac145ac0 2024-04-17T08:48:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.686+0000 7fd2aaa00640 1 -- 172.21.15.67:0/4067078725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c031be0 con 0x7fd2ac145ac0 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.686+0000 7fd2b2a00640 1 -- 172.21.15.67:0/4067078725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac145ac0 msgr2=0x7fd2ac071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.686+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac145ac0 0x7fd2ac071ea0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097a0 tx=0x7fd29c02b8b0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 -- 172.21.15.67:0/4067078725 shutdown_connections 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac0ce3b0 0x7fd2ac072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/4067078725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac145ac0 0x7fd2ac071ea0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 -- 172.21.15.67:0/4067078725 >> 172.21.15.67:0/4067078725 conn(0x7fd2ac067ee0 msgr2=0x7fd2ac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 -- 172.21.15.67:0/4067078725 shutdown_connections 2024-04-17T08:48:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.687+0000 7fd2b2a00640 1 -- 172.21.15.67:0/4067078725 wait complete. 2024-04-17T08:48:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.689+0000 7fd2b2a00640 1 Processor -- start 2024-04-17T08:48:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.689+0000 7fd2b2a00640 1 -- start start 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 0x7fd2ac13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac145ac0 0x7fd2ac13f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac145ea0 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac0cf5f0 con 0x7fd2ac072bb0 2024-04-17T08:48:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2b2a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac0d0b20 con 0x7fd2ac145ac0 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2ab400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2ab400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57228/0 (socket says 172.21.15.67:57228) 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2ab400640 1 -- 172.21.15.67:0/802662073 learned_addr learned my addr 172.21.15.67:0/802662073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2abe00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 0x7fd2ac13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.690+0000 7fd2ab400640 1 -- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac145ac0 msgr2=0x7fd2ac13f960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2ab400640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac145ac0 0x7fd2ac13f960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2ab400640 1 -- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 msgr2=0x7fd2ac13cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2ab400640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 0x7fd2ac13cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2ab400640 1 -- 172.21.15.67:0/802662073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac1f5720 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2abe00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 0x7fd2ac13cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2ab400640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fd29800e730 tx=0x7fd29800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.691+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29800f040 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.692+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd298004230 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.692+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29800ee40 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.692+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ac1f5a00 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.692+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ac06dc10 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.693+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ac068ca0 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.694+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 121) ==== 91160+0+0 (secure 0 0 0) 0x7fd29800c6d0 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.695+0000 7fd2a3e00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 0x7fd28c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:48:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.695+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 6081+0+0 (secure 0 0 0) 0x7fd29808bde0 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.696+0000 7fd2abe00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 0x7fd28c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:48:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.697+0000 7fd2abe00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 0x7fd28c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd29c036000 tx=0x7fd29c008030 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:48:00.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.700+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd298017020 con 0x7fd2ac0ce3b0 2024-04-17T08:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.931+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 --> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2ac0cc6c0 con 0x7fd28c06ca60 2024-04-17T08:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.947+0000 7fd2a3e00640 1 -- 172.21.15.67:0/802662073 <== mgr.8449 v2:172.21.15.187:6832/953837164 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7fd2ac0cc6c0 con 0x7fd28c06ca60 2024-04-17T08:48:00.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:48:00.947 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 msgr2=0x7fd28c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 0x7fd28c06ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd29c036000 tx=0x7fd29c008030 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 msgr2=0x7fd2ac13f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fd29800e730 tx=0x7fd29800ebf0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 shutdown_connections 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd28c06ca60 0x7fd28c06ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2ac145ac0 0x7fd2ac13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2ac0ce3b0 0x7fd2ac13f420 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 --2- 172.21.15.67:0/802662073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2ac072bb0 0x7fd2ac13cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 >> 172.21.15.67:0/802662073 conn(0x7fd2ac067ee0 msgr2=0x7fd2ac0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 shutdown_connections 2024-04-17T08:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:48:00.950+0000 7fd2b2a00640 1 -- 172.21.15.67:0/802662073 wait complete. 2024-04-17T08:48:00.971 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-17T08:48:00.971 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-17T08:48:00.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:00.971 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:01.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:01.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:01.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:02.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:02.032 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:02.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:02.094 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:02.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:03.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:03.096 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:03.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:03.158 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:03.158 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:04.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:04.159 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:04.224 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:04.224 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:04.224 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:05.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:05.225 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:05.290 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:05.290 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:05.290 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:06.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:06.291 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:06.353 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:06.353 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:06.354 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:07.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:07.354 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:07.417 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:07.417 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:07.417 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:08.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:08.418 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:08.483 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:08.483 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:08.483 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:09.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:09.485 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:09.549 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:09.549 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:09.549 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:10.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:10.551 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:10.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:10.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:10.615 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:11.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:11.616 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:11.681 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:11.681 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:11.681 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:12.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:12.683 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:12.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:12.747 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:12.747 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:13.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:13.748 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:13.812 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:13.812 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:13.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:14.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:14.813 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:14.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:14.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:14.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:15.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:15.879 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:15.939 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:15.939 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:15.939 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:16.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:16.940 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:17.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:17.005 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:17.005 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:18.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:18.006 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:18.073 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:18.073 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:18.073 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:19.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:19.074 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:19.139 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:19.139 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:19.139 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:20.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:20.140 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:20.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:20.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:20.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:21.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:21.205 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:21.265 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:21.265 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:21.265 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:22.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:22.267 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:22.331 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:22.331 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:22.331 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:23.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:23.332 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:23.396 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:23.396 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:23.396 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:24.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:24.398 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:24.460 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:24.460 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:24.460 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:25.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:25.461 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:25.525 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:25.525 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:25.525 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:26.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:48:26.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:48:26.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:26.526 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:26.590 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:26.590 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:26.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:27.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:27.591 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:27.656 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:27.656 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:27.656 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:28.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:28.657 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:28.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:28.720 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:28.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:29.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:29.721 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:29.784 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:29.784 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:29.784 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:30.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:30.785 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:30.849 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:30.849 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:30.849 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:31.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:31.850 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:31.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:31.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:31.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:32.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:32.911 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:32.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:32.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:32.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:33.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:33.976 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:34.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:34.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:34.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:35.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:35.041 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:35.104 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:35.104 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:35.104 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:36.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:36.105 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:36.168 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:36.168 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:36.168 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:37.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:37.170 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:37.233 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:37.234 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:37.234 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:38.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:38.235 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:38.300 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:38.300 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:38.300 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:39.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:39.301 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:39.366 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:39.366 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:39.366 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:40.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:40.367 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:40.431 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:40.431 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:40.431 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:41.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:41.432 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:41.496 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:41.496 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:41.496 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:42.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:42.498 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:42.561 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:42.561 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:42.562 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:43.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:43.563 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:43.627 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:43.628 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:43.628 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:44.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:44.629 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:44.694 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:44.694 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:44.694 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:45.695 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:45.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:45.758 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:45.758 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:46.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:46.759 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:46.825 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:46.825 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:46.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:47.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:47.826 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:47.889 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:47.889 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:47.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:48.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:48.890 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:48.954 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:48.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:48.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:49.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:49.956 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:50.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:50.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:50.020 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:51.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:51.022 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:51.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:51.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:51.087 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:52.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:52.088 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:52.153 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:52.153 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:52.153 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:53.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:53.154 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:53.218 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:53.218 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:53.218 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:54.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:54.220 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:54.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:54.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:54.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:55.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:55.284 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:55.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:55.350 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:55.350 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:56.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:56.351 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:56.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:56.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:56.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:56.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:48:56.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:48:57.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:57.416 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:57.478 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:57.478 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:57.478 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:58.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:58.480 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:58.543 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:58.543 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:58.543 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:48:59.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:48:59.544 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:48:59.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-04-17T08:48:59.607 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:48:59.607 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:00.445 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:49:00.446+0000 7fbac4400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:49:00.446 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:49:00.446+0000 7fbac4400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:49:00.446 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:49:00.446+0000 7fbac4400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:49:00.446 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:49:00.446+0000 7fbac4400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:49:00.446 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:49:00.446+0000 7fbac4400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:49:00.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:00.608 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:00.666 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:00.668 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:00.668 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:00.668 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:00.668 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-04-17-08_49_00_UTC'] 2024-04-17T08:49:00.668 DEBUG:tasks.cephfs.test_snap_schedules:exec=1713343680.4472518, now = 1713343740.668515, timo = 60 2024-04-17T08:49:01.669 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-04-17-08_49_00 in ~120s... 2024-04-17T08:49:01.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:01.669 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:01.732 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:01.734 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:01.734 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:01.734 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:02.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:02.735 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:02.799 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:02.800 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:02.800 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:02.800 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:03.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:03.801 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:03.862 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:03.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:03.863 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:03.864 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:04.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:04.865 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:04.927 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:04.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:04.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:04.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:05.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:05.931 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:05.993 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:05.995 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:05.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:05.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:06.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:06.996 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:07.056 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:07.058 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:07.058 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:07.058 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:08.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:08.059 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:08.121 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:08.123 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:08.123 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:08.123 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:09.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:09.124 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:09.187 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:09.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:09.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:09.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:10.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:10.190 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:10.252 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:10.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:10.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:10.254 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:11.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:11.254 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:11.318 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:11.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:11.319 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:11.319 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:12.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:12.321 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:12.383 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:12.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:12.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:12.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:13.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:13.386 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:13.446 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:13.448 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:13.448 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:13.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:14.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:14.449 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:14.511 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:14.513 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:14.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:14.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:15.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:15.514 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:15.577 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:15.579 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:15.579 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:15.579 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:16.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:16.580 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:16.642 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:16.644 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:16.644 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:16.645 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:17.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:17.645 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:17.707 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:17.709 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:17.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:17.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:18.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:18.710 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:18.774 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:18.774 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:18.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:19.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:19.775 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:19.837 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:19.838 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:19.838 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:19.838 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:20.840 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:20.902 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:20.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:20.903 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:20.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:21.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:21.905 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:21.967 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:21.968 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:21.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:21.969 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:22.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:22.970 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:23.032 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:23.033 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:23.033 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:23.033 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:24.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:24.035 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:24.097 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:24.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:24.098 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:24.098 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:25.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:25.100 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:25.159 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:25.161 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:25.161 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:25.162 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:26.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:26.163 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:26.225 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:26.227 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:26.227 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:26.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:26.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:49:26.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:49:27.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:27.229 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:27.290 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:27.292 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:27.292 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:27.292 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:28.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:28.293 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:28.355 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:28.356 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:28.356 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:28.356 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:29.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:29.357 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:29.420 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:29.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:29.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:29.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:30.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:30.424 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:30.485 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:30.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:30.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:30.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:31.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:31.488 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:31.550 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:31.551 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:31.551 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:31.551 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:32.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:32.552 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:32.614 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:32.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:32.615 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:32.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:33.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:33.617 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:33.680 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:33.682 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:33.682 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:33.682 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:34.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:34.683 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:34.746 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:34.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:34.748 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:34.748 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:35.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:35.749 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:35.812 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:35.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:35.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:35.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:36.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:36.814 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:36.877 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:36.879 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:36.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:36.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:37.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:37.880 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:37.942 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:37.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:37.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:37.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:38.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:38.945 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:39.008 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:39.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:39.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:39.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:40.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:40.011 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:40.074 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:40.075 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:40.075 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:40.075 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:41.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:41.076 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:41.136 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:41.137 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:41.137 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:41.137 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:42.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:42.138 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:42.202 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:42.204 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:42.204 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:42.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:43.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:43.205 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:43.267 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:43.268 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:43.268 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:43.269 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:44.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:44.270 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:44.332 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:44.334 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:44.334 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:44.334 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:45.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:45.335 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:45.398 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:45.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:45.399 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:45.399 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:46.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:46.400 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:46.463 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:46.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:46.465 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:46.465 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:47.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:47.466 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:47.528 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:47.529 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:47.529 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:47.529 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:48.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:48.531 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:48.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:48.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:48.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:49.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:49.598 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:49.662 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:49.663 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:49.664 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:49.664 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:50.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:50.665 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:50.727 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:50.729 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:50.729 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:50.729 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:51.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:51.730 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:51.793 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:51.794 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:51.794 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:51.794 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:52.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:52.795 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:52.857 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:52.859 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:52.859 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:52.859 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:53.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:53.860 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:53.923 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:53.925 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:53.925 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:53.925 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:54.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:54.926 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:54.988 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:54.990 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:54.990 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:54.990 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:55.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:55.991 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:56.055 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:56.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:56.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:56.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:56.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:49:56.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:49:57.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:57.058 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:57.120 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:57.121 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:57.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:57.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:58.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:58.123 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:58.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:58.187 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:58.187 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:49:59.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:49:59.189 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:49:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:49:59.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:49:59.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:49:59.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:50:00.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:00.254 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:50:00.318 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_49_00_UTC 2024-04-17T08:50:00.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:50:00.319 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-04-17T08:50:00.320 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-04-17T08:50:01.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:01.320 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:50:01.381 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_50_00_UTC 2024-04-17T08:50:01.383 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-04-17-08_50_00_UTC'} 2024-04-17T08:50:01.383 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-04-17-08_50_00_UTC'} 2024-04-17T08:50:01.383 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-04-17-08_49_00_UTC'} 2024-04-17T08:50:01.383 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-04-17-08_49_00_UTC'] 2024-04-17T08:50:01.383 DEBUG:tasks.cephfs.test_snap_schedules:exec=1713343680.4472518, now = 1713343801.3833976, timo = 120 2024-04-17T08:50:02.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-17T08:50:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.619+0000 7fb35ac00640 1 Processor -- start 2024-04-17T08:50:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.619+0000 7fb35ac00640 1 -- start start 2024-04-17T08:50:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb35ac00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb354145f80 0x7fb3540cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb35ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb354146360 con 0x7fb3540cd750 2024-04-17T08:50:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb35ac00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb354071e70 con 0x7fb35406c270 2024-04-17T08:50:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb35ac00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb3540727a0 con 0x7fb354145f80 2024-04-17T08:50:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb358e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb3540cd750 0x7fb35406bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53340/0 (socket says 172.21.15.67:53340) 2024-04-17T08:50:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.621+0000 7fb358e00640 1 -- 172.21.15.67:0/407972837 learned_addr learned my addr 172.21.15.67:0/407972837 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb359800640 1 --2- 172.21.15.67:0/407972837 >> v2:172.21.15.187:3300/0 conn(0x7fb354145f80 0x7fb3540cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689018360 0 0) 0x7fb354146360 con 0x7fb3540cd750 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb33c003520 con 0x7fb3540cd750 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1042059994 0 0) 0x7fb33c003520 con 0x7fb3540cd750 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 >> v2:172.21.15.187:3300/0 conn(0x7fb354145f80 msgr2=0x7fb3540cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 --2- 172.21.15.67:0/407972837 >> v2:172.21.15.187:3300/0 conn(0x7fb354145f80 0x7fb3540cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 >> v1:172.21.15.187:6789/0 conn(0x7fb35406c270 legacy=0x7fb35406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb354072150 con 0x7fb3540cd750 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb340001670 con 0x7fb3540cd750 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 >> v1:172.21.15.67:6789/0 conn(0x7fb3540cd750 legacy=0x7fb35406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 0x7fb33c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb33c0051c0 0x7fb33c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3540727a0 con 0x7fb33c0039b0 2024-04-17T08:50:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c00c0f0 con 0x7fb33c004800 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb33c00c2f0 con 0x7fb33c0051c0 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.622+0000 7fb359800640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 0x7fb33c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb359800640 1 -- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb33c0051c0 msgr2=0x7fb33c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb359800640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb33c0051c0 0x7fb33c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb359800640 1 -- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c004800 msgr2=0x7fb33c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb359800640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c004800 0x7fb33c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb359800640 1 -- 172.21.15.67:0/407972837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33c003520 con 0x7fb33c0039b0 2024-04-17T08:50:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.623+0000 7fb35a200640 1 -- 172.21.15.67:0/407972837 reap_dead start 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.624+0000 7fb359800640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 0x7fb33c003db0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb348009ac0 tx=0x7fb33c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102aa747ffee850c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.624+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348039070 con 0x7fb33c0039b0 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.624+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34802c710 con 0x7fb33c0039b0 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.624+0000 7fb353e00640 1 -- 172.21.15.67:0/407972837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348009690 con 0x7fb33c0039b0 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 -- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 msgr2=0x7fb33c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 0x7fb33c003db0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fb348009ac0 tx=0x7fb33c005830 comp rx=0 tx=0).stop 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 -- 172.21.15.67:0/407972837 shutdown_connections 2024-04-17T08:50:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 --2- 172.21.15.67:0/407972837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb33c0039b0 0x7fb33c003db0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 -- 172.21.15.67:0/407972837 >> 172.21.15.67:0/407972837 conn(0x7fb3540b2190 msgr2=0x7fb3540b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 -- 172.21.15.67:0/407972837 shutdown_connections 2024-04-17T08:50:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.625+0000 7fb35ac00640 1 -- 172.21.15.67:0/407972837 wait complete. 2024-04-17T08:50:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.627+0000 7fb35ac00640 1 Processor -- start 2024-04-17T08:50:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.627+0000 7fb35ac00640 1 -- start start 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c0039b0 0x7fb35415f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb35413f180 0x7fb354143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb354072db0 con 0x7fb35413e7c0 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb354072c30 con 0x7fb33c0039b0 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.628+0000 7fb35ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb3540cd210 con 0x7fb35413f180 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47370/0 (socket says 172.21.15.67:47370) 2024-04-17T08:50:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 -- 172.21.15.67:0/2464793417 learned_addr learned my addr 172.21.15.67:0/2464793417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb35a200640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb35413f180 0x7fb354143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb359800640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c0039b0 0x7fb35415f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 -- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb35413f180 msgr2=0x7fb354143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb35413f180 0x7fb354143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 -- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c0039b0 msgr2=0x7fb35415f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c0039b0 0x7fb35415f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.629+0000 7fb358e00640 1 -- 172.21.15.67:0/2464793417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb354141810 con 0x7fb35413e7c0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb358e00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c820 tx=0x7fb34000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340018070 con 0x7fb35413e7c0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3400074e0 con 0x7fb35413e7c0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340007690 con 0x7fb35413e7c0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb354141af0 con 0x7fb35413e7c0 2024-04-17T08:50:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.630+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35415ec70 con 0x7fb35413e7c0 2024-04-17T08:50:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.631+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb354071e70 con 0x7fb35413e7c0 2024-04-17T08:50:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.632+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 124) ==== 91171+0+0 (secure 0 0 0) 0x7fb340004a20 con 0x7fb35413e7c0 2024-04-17T08:50:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.633+0000 7fb351600640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 0x7fb33406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.633+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7fb34008b500 con 0x7fb35413e7c0 2024-04-17T08:50:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.634+0000 7fb359800640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 0x7fb33406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:02.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.635+0000 7fb359800640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 0x7fb33406ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb348036000 tx=0x7fb348030000 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:02.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.638+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb354071e70 con 0x7fb35413e7c0 2024-04-17T08:50:02.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:02.870+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 --> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7fb35415f6d0 con 0x7fb33406ca60 2024-04-17T08:50:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.067+0000 7fb351600640 1 -- 172.21.15.67:0/2464793417 <== mgr.8449 v2:172.21.15.187:6832/953837164 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fb35415f6d0 con 0x7fb33406ca60 2024-04-17T08:50:03.065 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path snap_test_dir1 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 msgr2=0x7fb33406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 0x7fb33406ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb348036000 tx=0x7fb348030000 comp rx=0 tx=0).stop 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 msgr2=0x7fb35413ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c820 tx=0x7fb34000cce0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 shutdown_connections 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb33406ca60 0x7fb33406ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb35413f180 0x7fb354143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35413e7c0 0x7fb35413ec40 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 --2- 172.21.15.67:0/2464793417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb33c0039b0 0x7fb35415f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 >> 172.21.15.67:0/2464793417 conn(0x7fb3540b2190 msgr2=0x7fb3540af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 shutdown_connections 2024-04-17T08:50:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.071+0000 7fb35ac00640 1 -- 172.21.15.67:0/2464793417 wait complete. 2024-04-17T08:50:03.095 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-17T08:50:03.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:03.096 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:50:03.152 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_50_00_UTC 2024-04-17T08:50:03.154 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-04-17-08_50_00_UTC 2024-04-17T08:50:03.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:03.154 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-04-17-08_50_00_UTC) 2024-04-17T08:50:03.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:03.218 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T08:50:03.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:50:03.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.499+0000 7f20da000640 1 Processor -- start 2024-04-17T08:50:03.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.499+0000 7f20da000640 1 -- start start 2024-04-17T08:50:03.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20da000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20d4162510 0x7f20d4162950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20da000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d406cb50 con 0x7f20d415e210 2024-04-17T08:50:03.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20da000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d406ce30 con 0x7f20d40ce570 2024-04-17T08:50:03.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20da000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20d406d760 con 0x7f20d4162510 2024-04-17T08:50:03.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20d9600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20d415e210 0x7f20d415e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46858/0 (socket says 172.21.15.67:46858) 2024-04-17T08:50:03.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.501+0000 7f20d9600640 1 -- 172.21.15.67:0/1906463138 learned_addr learned my addr 172.21.15.67:0/1906463138 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:03.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3e00640 1 --2- 172.21.15.67:0/1906463138 >> v2:172.21.15.187:3300/0 conn(0x7f20d4162510 0x7f20d4162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975615633 0 0) 0x7f20d406cb50 con 0x7f20d415e210 2024-04-17T08:50:03.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc003520 con 0x7f20d415e210 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 904039619 0 0) 0x7f20bc003520 con 0x7f20d415e210 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 >> v2:172.21.15.187:3300/0 conn(0x7f20d4162510 msgr2=0x7f20d4162950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 --2- 172.21.15.67:0/1906463138 >> v2:172.21.15.187:3300/0 conn(0x7f20d4162510 0x7f20d4162950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 >> v1:172.21.15.187:6789/0 conn(0x7f20d40ce570 legacy=0x7f20d40d12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d406d110 con 0x7f20d415e210 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f20cc0019f0 con 0x7f20d415e210 2024-04-17T08:50:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 >> v1:172.21.15.67:6789/0 conn(0x7f20d415e210 legacy=0x7f20d415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.502+0000 7f20d3400640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d3400640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d406d760 con 0x7f20bc0051c0 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc00c160 con 0x7f20bc0039b0 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20bc00c360 con 0x7f20bc004800 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d8c00640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d3e00640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 -- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc004800 msgr2=0x7f20bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 -- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc0039b0 msgr2=0x7f20bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.503+0000 7f20d9600640 1 -- 172.21.15.67:0/1906463138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc003520 con 0x7f20bc0051c0 2024-04-17T08:50:03.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.505+0000 7f20d9600640 1 -- 172.21.15.67:0/1906463138 reap_dead start 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.505+0000 7f20d9600640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20bc00b9c0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00a850 tx=0x7f20bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffdb42652868b91 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.505+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00a4a0 con 0x7f20bc0051c0 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.505+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc034650 con 0x7f20bc0051c0 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.505+0000 7f20d3400640 1 -- 172.21.15.67:0/1906463138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc034a40 con 0x7f20bc0051c0 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 -- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 msgr2=0x7f20bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20bc00b9c0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00a850 tx=0x7f20bc005830 comp rx=0 tx=0).stop 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 -- 172.21.15.67:0/1906463138 shutdown_connections 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 --2- 172.21.15.67:0/1906463138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20bc00b9c0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 -- 172.21.15.67:0/1906463138 >> 172.21.15.67:0/1906463138 conn(0x7f20d40b4090 msgr2=0x7f20d40b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 -- 172.21.15.67:0/1906463138 shutdown_connections 2024-04-17T08:50:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.506+0000 7f20da000640 1 -- 172.21.15.67:0/1906463138 wait complete. 2024-04-17T08:50:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.508+0000 7f20da000640 1 Processor -- start 2024-04-17T08:50:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.508+0000 7f20da000640 1 -- start start 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d40c80b0 0x7f20d40c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d415f550 0x7f20d415f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d406dbf0 con 0x7f20bc0051c0 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40ce950 con 0x7f20d40c80b0 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20da000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20d4161750 con 0x7f20d415f550 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59522/0 (socket says 172.21.15.67:59522) 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20d8c00640 1 -- 172.21.15.67:0/2946363041 learned_addr learned my addr 172.21.15.67:0/2946363041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20d8c00640 1 -- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d415f550 msgr2=0x7f20d415f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.509+0000 7f20d8c00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d415f550 0x7f20d415f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d8c00640 1 -- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d40c80b0 msgr2=0x7f20d40c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d8c00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d40c80b0 0x7f20d40c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d8c00640 1 -- 172.21.15.67:0/2946363041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d4160230 con 0x7f20bc0051c0 2024-04-17T08:50:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d8c00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f20c800e4b0 tx=0x7f20c800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c80052e0 con 0x7f20bc0051c0 2024-04-17T08:50:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c8018070 con 0x7f20bc0051c0 2024-04-17T08:50:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d4160510 con 0x7f20bc0051c0 2024-04-17T08:50:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8003830 con 0x7f20bc0051c0 2024-04-17T08:50:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.510+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41e8100 con 0x7f20bc0051c0 2024-04-17T08:50:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.511+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d4160a30 con 0x7f20bc0051c0 2024-04-17T08:50:03.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.512+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 124) ==== 91171+0+0 (secure 0 0 0) 0x7f20c80039d0 con 0x7f20bc0051c0 2024-04-17T08:50:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.513+0000 7f20d0c00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 0x7f20b406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.513+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7f20c808bb70 con 0x7f20bc0051c0 2024-04-17T08:50:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.514+0000 7f20d3e00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 0x7f20b406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.515+0000 7f20d3e00640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 0x7f20b406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f20d41f24d0 tx=0x7f20c0008040 comp rx=0 tx=0).ready entity=mgr.8449 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:03.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.519+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f20d4160a30 con 0x7f20bc0051c0 2024-04-17T08:50:03.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:03.779+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f20d406cb50 con 0x7f20bc0051c0 2024-04-17T08:50:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.079+0000 7f20d0c00640 1 -- 172.21.15.67:0/2946363041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v125) ==== 93+0+0 (secure 0 0 0) 0x7f20d406cb50 con 0x7f20bc0051c0 2024-04-17T08:50:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.082+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 msgr2=0x7f20b406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.082+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 0x7f20b406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f20d41f24d0 tx=0x7f20c0008040 comp rx=0 tx=0).stop 2024-04-17T08:50:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.082+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 msgr2=0x7f20d4157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.082+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f20c800e4b0 tx=0x7f20c800e970 comp rx=0 tx=0).stop 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 shutdown_connections 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f20b406ca60 0x7f20b406ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d415f550 0x7f20d415f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d40c80b0 0x7f20d40c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 --2- 172.21.15.67:0/2946363041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc0051c0 0x7f20d4157b00 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.083+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 >> 172.21.15.67:0/2946363041 conn(0x7f20d40b4090 msgr2=0x7f20d4071020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.084+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 shutdown_connections 2024-04-17T08:50:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.084+0000 7f20da000640 1 -- 172.21.15.67:0/2946363041 wait complete. 2024-04-17T08:50:04.105 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:50:04.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:50:04.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:50:04.300 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:50:04.300 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:50:04.300 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:04.301 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:04.351 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:04.351 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:04.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:04.638 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:04.638+0000 7efe4f22b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:50:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.653+0000 7fddc1400640 1 Processor -- start 2024-04-17T08:50:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.653+0000 7fddc1400640 1 -- start start 2024-04-17T08:50:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc0b6d00 0x7fddbc0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc1400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddbc0cf910 0x7fddbc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc1400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddbc0d2d30 con 0x7fddbc15ec60 2024-04-17T08:50:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc1400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fddbc0d49c0 con 0x7fddbc0cf910 2024-04-17T08:50:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddbc0d3660 con 0x7fddbc0b6d00 2024-04-17T08:50:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc0a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddbc0cf910 0x7fddbc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddbb400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc0b6d00 0x7fddbc0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc0a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fddbc0cf910 0x7fddbc0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:46144/0 (socket says 172.21.15.67:46144) 2024-04-17T08:50:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.655+0000 7fddc0a00640 1 -- 172.21.15.67:0/1335427653 learned_addr learned my addr 172.21.15.67:0/1335427653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.656+0000 7fddc0a00640 1 -- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc0b6d00 msgr2=0x7fddbc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.656+0000 7fddc0a00640 1 --2- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc0b6d00 0x7fddbc0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.656+0000 7fddc0a00640 1 -- 172.21.15.67:0/1335427653 >> v1:172.21.15.187:6789/0 conn(0x7fddbc15ec60 legacy=0x7fddbc0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.656+0000 7fddc0a00640 1 -- 172.21.15.67:0/1335427653 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddbc0d3010 con 0x7fddbc0cf910 2024-04-17T08:50:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.657+0000 7fddc0a00640 1 --2- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0cf910 0x7fddbc0b7b70 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009e50 tx=0x7fddb402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be62840b720f200 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.658+0000 7fddbaa00640 1 -- 172.21.15.67:0/1335427653 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb402cb00 con 0x7fddbc0cf910 2024-04-17T08:50:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.658+0000 7fddbaa00640 1 -- 172.21.15.67:0/1335427653 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb402cca0 con 0x7fddbc0cf910 2024-04-17T08:50:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.658+0000 7fddbaa00640 1 -- 172.21.15.67:0/1335427653 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4034610 con 0x7fddbc0cf910 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 -- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0cf910 msgr2=0x7fddbc0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 --2- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0cf910 0x7fddbc0b7b70 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009e50 tx=0x7fddb402bf60 comp rx=0 tx=0).stop 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 -- 172.21.15.67:0/1335427653 shutdown_connections 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 --2- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0cf910 0x7fddbc0b7b70 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 --2- 172.21.15.67:0/1335427653 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc0b6d00 0x7fddbc0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 -- 172.21.15.67:0/1335427653 >> 172.21.15.67:0/1335427653 conn(0x7fddbc0d15f0 msgr2=0x7fddbc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 -- 172.21.15.67:0/1335427653 shutdown_connections 2024-04-17T08:50:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.659+0000 7fddc1400640 1 -- 172.21.15.67:0/1335427653 wait complete. 2024-04-17T08:50:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.661+0000 7fddc1400640 1 Processor -- start 2024-04-17T08:50:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.661+0000 7fddc1400640 1 -- start start 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 0x7fddbc1f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc15ec60 0x7fddbc1f7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc15f040 con 0x7fddbc0cf910 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fddbc0cfdc0 con 0x7fddbc0b6d00 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc1400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fddbc0d00c0 con 0x7fddbc15ec60 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddbb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddbb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59558/0 (socket says 172.21.15.67:59558) 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddbb400640 1 -- 172.21.15.67:0/2556263265 learned_addr learned my addr 172.21.15.67:0/2556263265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddbbe00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 0x7fddbc1f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.663+0000 7fddc0a00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc15ec60 0x7fddbc1f7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc15ec60 msgr2=0x7fddbc1f7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc15ec60 0x7fddbc1f7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 msgr2=0x7fddbc1f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 0x7fddbc1f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 -- 172.21.15.67:0/2556263265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddbc15fc70 con 0x7fddbc0cf910 2024-04-17T08:50:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbbe00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 0x7fddbc1f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddbb400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fdda800e9b0 tx=0x7fdda800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda8017070 con 0x7fddbc0cf910 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda800c9a0 con 0x7fddbc0cf910 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda800cd90 con 0x7fddbc0cf910 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.664+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddbc15ff50 con 0x7fddbc0cf910 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.665+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddbc160510 con 0x7fddbc0cf910 2024-04-17T08:50:04.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.666+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddbc068a10 con 0x7fddbc0cf910 2024-04-17T08:50:04.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.668+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7fdda801a070 con 0x7fddbc0cf910 2024-04-17T08:50:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.669+0000 7fddb3e00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 0x7fdd9c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.669+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7fdda808be00 con 0x7fddbc0cf910 2024-04-17T08:50:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.669+0000 7fddbbe00640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 msgr2=0x7fdd9c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.669+0000 7fddbbe00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 0x7fdd9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.674+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdda805a000 con 0x7fddbc0cf910 2024-04-17T08:50:04.675 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:04.677+0000 7eff7ac0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:50:04.695 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:04.696+0000 7efe4f22b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:50:04.733 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:04.735+0000 7eff7ac0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:50:04.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.870+0000 7fddbbe00640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 msgr2=0x7fdd9c06ef00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:04.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.870+0000 7fddbbe00640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 0x7fdd9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:04.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.879+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddbc1ea630 con 0x7fddbc0cf910 2024-04-17T08:50:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.881+0000 7fddb3e00640 1 -- 172.21.15.67:0/2556263265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) ==== 74+0+17223 (secure 0 0 0) 0x7fdda8059900 con 0x7fddbc0cf910 2024-04-17T08:50:04.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:04.880 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":215,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:48:59.795260+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.7","mappings":[{"from":0,"to":2}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3145453289":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/910136246":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/4251121774":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6833/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6832/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2605475186":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2381197913":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/713566291":"2024-04-18T08:47:58.503956+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 msgr2=0x7fdd9c06ef00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 0x7fdd9c06ef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 msgr2=0x7fddbc161980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fdda800e9b0 tx=0x7fdda800ee70 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 shutdown_connections 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fdd9c06ca40 0x7fdd9c06ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fddbc15ec60 0x7fddbc1f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddbc0cf910 0x7fddbc161980 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 --2- 172.21.15.67:0/2556263265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fddbc0b6d00 0x7fddbc1f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 >> 172.21.15.67:0/2556263265 conn(0x7fddbc0d15f0 msgr2=0x7fddbc15df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 shutdown_connections 2024-04-17T08:50:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:04.883+0000 7fddc1400640 1 -- 172.21.15.67:0/2556263265 wait complete. 2024-04-17T08:50:04.900 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:47:41.219096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:47:42.272552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:50:04.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:50:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.119+0000 7fd879c00640 1 Processor -- start 2024-04-17T08:50:05.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.120+0000 7fd879c00640 1 -- start start 2024-04-17T08:50:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd879c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 0x7fd8740d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd879c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8740d4f70 con 0x7fd8740b3b50 2024-04-17T08:50:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd879c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8740d5250 con 0x7fd87415ec20 2024-04-17T08:50:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd879c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd8740d2760 con 0x7fd8740d22b0 2024-04-17T08:50:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd86be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8740b3b50 0x7fd8740cfcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46880/0 (socket says 172.21.15.67:46880) 2024-04-17T08:50:05.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.121+0000 7fd86be00640 1 -- 172.21.15.67:0/1825540584 learned_addr learned my addr 172.21.15.67:0/1825540584 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd879200640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 0x7fd8740d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802041537 0 0) 0x7fd8740d4f70 con 0x7fd8740b3b50 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd858003520 con 0x7fd8740b3b50 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 528616635 0 0) 0x7fd858003520 con 0x7fd8740b3b50 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 msgr2=0x7fd8740d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 0x7fd8740d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 >> v1:172.21.15.187:6789/0 conn(0x7fd87415ec20 legacy=0x7fd87406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8740cf4d0 con 0x7fd8740b3b50 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd85c0017f0 con 0x7fd8740b3b50 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 >> v1:172.21.15.67:6789/0 conn(0x7fd8740b3b50 legacy=0x7fd8740cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.122+0000 7fd873400640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8580039e0 0x7fd858003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873400640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 0x7fd858004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873400640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd858004d40 0x7fd8580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd8740d2760 con 0x7fd858004d40 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd85800bb50 con 0x7fd8580039e0 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd85800bd50 con 0x7fd8740d22b0 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd858004d40 0x7fd8580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd873e00640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8580039e0 0x7fd858003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 -- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 msgr2=0x7fd858004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740d22b0 0x7fd858004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 -- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8580039e0 msgr2=0x7fd858003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8580039e0 0x7fd858003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.123+0000 7fd879200640 1 -- 172.21.15.67:0/1825540584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd858003520 con 0x7fd858004d40 2024-04-17T08:50:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.124+0000 7fd879200640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd858004d40 0x7fd8580051c0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fc0 tx=0x7fd858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1024ea860eaeaef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.124+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c039070 con 0x7fd858004d40 2024-04-17T08:50:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.124+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c02cc40 con 0x7fd858004d40 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.124+0000 7fd873400640 1 -- 172.21.15.67:0/1825540584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c009c10 con 0x7fd858004d40 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 -- 172.21.15.67:0/1825540584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd858004d40 msgr2=0x7fd8580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 --2- 172.21.15.67:0/1825540584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd858004d40 0x7fd8580051c0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fc0 tx=0x7fd858005830 comp rx=0 tx=0).stop 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879200640 1 -- 172.21.15.67:0/1825540584 reap_dead start 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 -- 172.21.15.67:0/1825540584 shutdown_connections 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 -- 172.21.15.67:0/1825540584 >> 172.21.15.67:0/1825540584 conn(0x7fd87408bff0 msgr2=0x7fd8740d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 -- 172.21.15.67:0/1825540584 shutdown_connections 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.125+0000 7fd879c00640 1 -- 172.21.15.67:0/1825540584 wait complete. 2024-04-17T08:50:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.127+0000 7fd879c00640 1 Processor -- start 2024-04-17T08:50:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.127+0000 7fd879c00640 1 -- start start 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd858004d40 0x7fd8740b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740b4140 0x7fd8740b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd87406d150 con 0x7fd8741e9d90 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd8740cf180 con 0x7fd858004d40 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd87406d2d0 con 0x7fd8740b4140 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59574/0 (socket says 172.21.15.67:59574) 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.128+0000 7fd879200640 1 -- 172.21.15.67:0/1282854981 learned_addr learned my addr 172.21.15.67:0/1282854981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd873e00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd858004d40 0x7fd8740b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd86be00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740b4140 0x7fd8740b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740b4140 msgr2=0x7fd8740b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740b4140 0x7fd8740b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd858004d40 msgr2=0x7fd8740b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd858004d40 0x7fd8740b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 -- 172.21.15.67:0/1282854981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8740b5b20 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879200640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fd86c002230 tx=0x7fd86c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c039040 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c02ca40 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8740b5e00 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.129+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003990 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.130+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8740b63c0 con 0x7fd8741e9d90 2024-04-17T08:50:05.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.131+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd874068a10 con 0x7fd8741e9d90 2024-04-17T08:50:05.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.133+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7fd86c047030 con 0x7fd8741e9d90 2024-04-17T08:50:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.133+0000 7fd870c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 0x7fd85006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.134+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7fd86c0adfe0 con 0x7fd8741e9d90 2024-04-17T08:50:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.134+0000 7fd873e00640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 msgr2=0x7fd85006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:05.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.134+0000 7fd873e00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 0x7fd85006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:05.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.139+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd86c04e020 con 0x7fd8741e9d90 2024-04-17T08:50:05.144 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.145+0000 7efe4f22b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:50:05.197 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.197+0000 7efe4f22b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:50:05.207 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.209+0000 7eff7ac0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:50:05.247 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.248+0000 7efe4f22b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:50:05.263 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.264+0000 7eff7ac0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:50:05.309 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.310+0000 7efe4f22b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:50:05.317 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.318+0000 7eff7ac0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:50:05.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.335+0000 7fd873e00640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 msgr2=0x7fd85006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:05.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.335+0000 7fd873e00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 0x7fd85006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:05.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.340+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8740d0050 con 0x7fd8741e9d90 2024-04-17T08:50:05.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.341+0000 7fd870c00640 1 -- 172.21.15.67:0/1282854981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v215) ==== 93+0+10 (secure 0 0 0) 0x7fd86c02cc10 con 0x7fd8741e9d90 2024-04-17T08:50:05.339 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:50:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.343+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 msgr2=0x7fd85006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.343+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 0x7fd85006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.343+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 msgr2=0x7fd874152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.343+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fd86c002230 tx=0x7fd86c009750 comp rx=0 tx=0).stop 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 shutdown_connections 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fd85006ca10 0x7fd85006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8741e9d90 0x7fd874152ca0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8740b4140 0x7fd8740b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 --2- 172.21.15.67:0/1282854981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd858004d40 0x7fd8740b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 >> 172.21.15.67:0/1282854981 conn(0x7fd87408bff0 msgr2=0x7fd87415c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 shutdown_connections 2024-04-17T08:50:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.344+0000 7fd879c00640 1 -- 172.21.15.67:0/1282854981 wait complete. 2024-04-17T08:50:05.359 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.360+0000 7efe4f22b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:50:05.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:50:05.383 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.385+0000 7eff7ac0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:50:05.436 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.438+0000 7eff7ac0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:50:05.488 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.488+0000 7efe4f22b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:50:05.551 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.551+0000 7efe4f22b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.565+0000 7fec88e00640 1 Processor -- start 2024-04-17T08:50:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.566+0000 7fec88e00640 1 -- start start 2024-04-17T08:50:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec88e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec840b46f0 0x7fec840b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec88e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec840b51d0 con 0x7fec840d4510 2024-04-17T08:50:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec88e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec840b54b0 con 0x7fec8415e9b0 2024-04-17T08:50:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec88e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fec840b5790 con 0x7fec840b46f0 2024-04-17T08:50:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec83e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fec840d4510 0x7fec840d4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46888/0 (socket says 172.21.15.67:46888) 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.567+0000 7fec83e00640 1 -- 172.21.15.67:0/3453338549 learned_addr learned my addr 172.21.15.67:0/3453338549 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063215616 0 0) 0x7fec840b51d0 con 0x7fec840d4510 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec68003520 con 0x7fec840d4510 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec7be00640 1 --2- 172.21.15.67:0/3453338549 >> v2:172.21.15.187:3300/0 conn(0x7fec840b46f0 0x7fec840b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464485869 0 0) 0x7fec840b54b0 con 0x7fec8415e9b0 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec840b51d0 con 0x7fec8415e9b0 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3350342708 0 0) 0x7fec68003520 con 0x7fec840d4510 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 >> v2:172.21.15.187:3300/0 conn(0x7fec840b46f0 msgr2=0x7fec840b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 --2- 172.21.15.67:0/3453338549 >> v2:172.21.15.187:3300/0 conn(0x7fec840b46f0 0x7fec840b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 >> v1:172.21.15.187:6789/0 conn(0x7fec8415e9b0 legacy=0x7fec840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec840d24c0 con 0x7fec840d4510 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fec7c0019f0 con 0x7fec840d4510 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.568+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 >> v1:172.21.15.67:6789/0 conn(0x7fec840d4510 legacy=0x7fec840d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec68003b90 0x7fec68003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec68004800 0x7fec68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 0x7fec6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec840b5790 con 0x7fec680052e0 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec6800c200 con 0x7fec68003b90 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec6800c400 con 0x7fec68004800 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 0x7fec6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83400640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec68003b90 0x7fec68003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec7be00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec68004800 0x7fec68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 -- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec68004800 msgr2=0x7fec68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec68004800 0x7fec68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 -- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec68003b90 msgr2=0x7fec68003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec68003b90 0x7fec68003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.569+0000 7fec83e00640 1 -- 172.21.15.67:0/3453338549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec68003520 con 0x7fec680052e0 2024-04-17T08:50:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.570+0000 7fec83400640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec68003b90 0x7fec68003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.571+0000 7fec83e00640 1 -- 172.21.15.67:0/3453338549 reap_dead start 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.571+0000 7fec83e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 0x7fec6800b9c0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00a730 tx=0x7fec68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a799697773535c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.571+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c00a350 con 0x7fec680052e0 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.571+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec7c034650 con 0x7fec680052e0 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.571+0000 7fec82a00640 1 -- 172.21.15.67:0/3453338549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c034a40 con 0x7fec680052e0 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 -- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 msgr2=0x7fec6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 0x7fec6800b9c0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00a730 tx=0x7fec68005830 comp rx=0 tx=0).stop 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 -- 172.21.15.67:0/3453338549 shutdown_connections 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 --2- 172.21.15.67:0/3453338549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec680052e0 0x7fec6800b9c0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 -- 172.21.15.67:0/3453338549 >> 172.21.15.67:0/3453338549 conn(0x7fec8409daa0 msgr2=0x7fec8408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 -- 172.21.15.67:0/3453338549 shutdown_connections 2024-04-17T08:50:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.572+0000 7fec88e00640 1 -- 172.21.15.67:0/3453338549 wait complete. 2024-04-17T08:50:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.574+0000 7fec88e00640 1 Processor -- start 2024-04-17T08:50:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.575+0000 7fec88e00640 1 -- start start 2024-04-17T08:50:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 0x7fec8414dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 0x7fec840c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec8415ed90 con 0x7fec84156c90 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec840d2830 con 0x7fec840c8ad0 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec88e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec8415ef10 con 0x7fec680052e0 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59586/0 (socket says 172.21.15.67:59586) 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 -- 172.21.15.67:0/1929320084 learned_addr learned my addr 172.21.15.67:0/1929320084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec83400640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 0x7fec8414dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec83e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 0x7fec840c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 msgr2=0x7fec8414dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 0x7fec8414dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.576+0000 7fec7be00640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 msgr2=0x7fec840c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7be00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 0x7fec840c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7be00640 1 -- 172.21.15.67:0/1929320084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec840c9490 con 0x7fec84156c90 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec83e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 0x7fec840c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec83400640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 0x7fec8414dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7be00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00f5b0 tx=0x7fec6c00fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c013810 con 0x7fec84156c90 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c01a070 con 0x7fec84156c90 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec840c9740 con 0x7fec84156c90 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c013bb0 con 0x7fec84156c90 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.577+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec840c9d00 con 0x7fec84156c90 2024-04-17T08:50:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.579+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec840755d0 con 0x7fec84156c90 2024-04-17T08:50:05.578 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.580+0000 7eff7ac0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:50:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.580+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7fec6c012070 con 0x7fec84156c90 2024-04-17T08:50:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.582+0000 7fec7b400640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 0x7fec6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.582+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7fec6c08bdb0 con 0x7fec84156c90 2024-04-17T08:50:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.582+0000 7fec83400640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 msgr2=0x7fec6006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.582+0000 7fec83400640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 0x7fec6006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:05.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.587+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fec6c01c310 con 0x7fec84156c90 2024-04-17T08:50:05.604 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.604+0000 7efe4f22b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:50:05.647 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.648+0000 7eff7ac0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:05.661 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.661+0000 7efe4f22b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:50:05.699 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.701+0000 7eff7ac0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:50:05.712 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:05.712+0000 7efe4f22b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:50:05.759 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.761+0000 7eff7ac0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:50:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.783+0000 7fec83400640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 msgr2=0x7fec6006eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.783+0000 7fec83400640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 0x7fec6006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:05.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.787+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fec8406d2a0 con 0x7fec84156c90 2024-04-17T08:50:05.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.788+0000 7fec7b400640 1 -- 172.21.15.67:0/1929320084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v215) ==== 104+0+11 (secure 0 0 0) 0x7fec6c059fb0 con 0x7fec84156c90 2024-04-17T08:50:05.786 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:50:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 msgr2=0x7fec6006eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 0x7fec6006eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 msgr2=0x7fec840c8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00f5b0 tx=0x7fec6c00fa70 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 shutdown_connections 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fec6006ca10 0x7fec6006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec840c8ad0 0x7fec840c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec84156c90 0x7fec840c8590 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 --2- 172.21.15.67:0/1929320084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec680052e0 0x7fec8414dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 >> 172.21.15.67:0/1929320084 conn(0x7fec8409daa0 msgr2=0x7fec8406ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 shutdown_connections 2024-04-17T08:50:05.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:05.791+0000 7fec88e00640 1 -- 172.21.15.67:0/1929320084 wait complete. 2024-04-17T08:50:05.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:50:05.814 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:05.816+0000 7eff7ac0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:50:06.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.023+0000 7f8aad800640 1 Processor -- start 2024-04-17T08:50:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.023+0000 7f8aad800640 1 -- start start 2024-04-17T08:50:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aad800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aa8143200 0x7f8aa80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aad800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8aa80cd580 0x7f8aa80b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aad800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aa81607a0 con 0x7f8aa8142e20 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aad800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa8160a60 con 0x7f8aa8143200 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aad800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa8160d40 con 0x7f8aa80cd580 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aa6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aa8143200 0x7f8aa80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.025+0000 7f8aa6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aa8143200 0x7f8aa80cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59592/0 (socket says 172.21.15.67:59592) 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa6000640 1 -- 172.21.15.67:0/736506462 learned_addr learned my addr 172.21.15.67:0/736506462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa7400640 1 --2- 172.21.15.67:0/736506462 >> v2:172.21.15.187:3300/0 conn(0x7f8aa80cd580 0x7f8aa80b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa6000640 1 -- 172.21.15.67:0/736506462 >> v2:172.21.15.187:3300/0 conn(0x7f8aa80cd580 msgr2=0x7f8aa80b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa6000640 1 --2- 172.21.15.67:0/736506462 >> v2:172.21.15.187:3300/0 conn(0x7f8aa80cd580 0x7f8aa80b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa6000640 1 -- 172.21.15.67:0/736506462 >> v1:172.21.15.187:6789/0 conn(0x7f8aa8142e20 legacy=0x7f8aa80cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.026+0000 7f8aa6000640 1 -- 172.21.15.67:0/736506462 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa8161020 con 0x7f8aa8143200 2024-04-17T08:50:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.028+0000 7f8aa6000640 1 --2- 172.21.15.67:0/736506462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8143200 0x7f8aa80cd040 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80cfad0 tx=0x7f8a9402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2a1045b47f5576 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.028+0000 7f8aa5600640 1 -- 172.21.15.67:0/736506462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9402e070 con 0x7f8aa8143200 2024-04-17T08:50:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.028+0000 7f8aa5600640 1 -- 172.21.15.67:0/736506462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a94037070 con 0x7f8aa8143200 2024-04-17T08:50:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.028+0000 7f8aa5600640 1 -- 172.21.15.67:0/736506462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94031a00 con 0x7f8aa8143200 2024-04-17T08:50:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 -- 172.21.15.67:0/736506462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8143200 msgr2=0x7f8aa80cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 --2- 172.21.15.67:0/736506462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8143200 0x7f8aa80cd040 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80cfad0 tx=0x7f8a9402b880 comp rx=0 tx=0).stop 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 -- 172.21.15.67:0/736506462 shutdown_connections 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 --2- 172.21.15.67:0/736506462 >> v2:172.21.15.187:3300/0 conn(0x7f8aa80cd580 0x7f8aa80b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 --2- 172.21.15.67:0/736506462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8143200 0x7f8aa80cd040 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 -- 172.21.15.67:0/736506462 >> 172.21.15.67:0/736506462 conn(0x7f8aa806b8c0 msgr2=0x7f8aa806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 -- 172.21.15.67:0/736506462 shutdown_connections 2024-04-17T08:50:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.029+0000 7f8aad800640 1 -- 172.21.15.67:0/736506462 wait complete. 2024-04-17T08:50:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.031+0000 7f8aad800640 1 Processor -- start 2024-04-17T08:50:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.031+0000 7f8aad800640 1 -- start start 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 0x7f8aa8134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 0x7f8aa80c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa80ce730 con 0x7f8aa8142e20 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa80cf330 con 0x7f8aa80cd580 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aad800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa80b0e20 con 0x7f8aa8143200 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aa6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aa6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59598/0 (socket says 172.21.15.67:59598) 2024-04-17T08:50:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aa6000640 1 -- 172.21.15.67:0/927959043 learned_addr learned my addr 172.21.15.67:0/927959043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aa7400640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 0x7f8aa80c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.032+0000 7f8aa6a00640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 0x7f8aa8134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 msgr2=0x7f8aa80c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 0x7f8aa80c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 msgr2=0x7f8aa8134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 0x7f8aa8134d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 -- 172.21.15.67:0/927959043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa8132330 con 0x7f8aa8142e20 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa7400640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 0x7f8aa80c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6a00640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 0x7f8aa8134d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aa6000640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000c00 tx=0x7f8a940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94032880 con 0x7f8aa8142e20 2024-04-17T08:50:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a94032a20 con 0x7f8aa8142e20 2024-04-17T08:50:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94032e10 con 0x7f8aa8142e20 2024-04-17T08:50:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa8132610 con 0x7f8aa8142e20 2024-04-17T08:50:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.033+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa80c5b70 con 0x7f8aa8142e20 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.035+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa8068b90 con 0x7f8aa8142e20 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.036+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7f8a940453f0 con 0x7f8aa8142e20 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.036+0000 7f8a9ea00640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 0x7f8a8806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.036+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7f8a940adf90 con 0x7f8aa8142e20 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.037+0000 7f8aa6a00640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 msgr2=0x7f8a8806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.037+0000 7f8aa6a00640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 0x7f8a8806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.041+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8a94040070 con 0x7f8aa8142e20 2024-04-17T08:50:06.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.099+0000 7efe4f22b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:50:06.226 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.227+0000 7eff7ac0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:50:06.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.238+0000 7f8aa6a00640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 msgr2=0x7f8a8806eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:06.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.238+0000 7f8aa6a00640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 0x7f8a8806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.245+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8aa80cde80 con 0x7f8aa8142e20 2024-04-17T08:50:06.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.246+0000 7f8a9ea00640 1 -- 172.21.15.67:0/927959043 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v215) ==== 100+0+11 (secure 0 0 0) 0x7f8a9407bff0 con 0x7f8aa8142e20 2024-04-17T08:50:06.245 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:50:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 msgr2=0x7f8a8806eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 0x7f8a8806eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 msgr2=0x7f8aa80cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000c00 tx=0x7f8a940046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 shutdown_connections 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f8a8806ca10 0x7f8a8806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8aa8143200 0x7f8aa80c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8142e20 0x7f8aa80cc800 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 --2- 172.21.15.67:0/927959043 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8aa80cd580 0x7f8aa8134d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 >> 172.21.15.67:0/927959043 conn(0x7f8aa806b8c0 msgr2=0x7f8aa8140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 shutdown_connections 2024-04-17T08:50:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.249+0000 7f8aad800640 1 -- 172.21.15.67:0/927959043 wait complete. 2024-04-17T08:50:06.259 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.260+0000 7efe4f22b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:50:06.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:50:06.370 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.371+0000 7efe4f22b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:50:06.400 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.401+0000 7eff7ac0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:50:06.421 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.421+0000 7efe4f22b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:50:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.475+0000 7f861b200640 1 Processor -- start 2024-04-17T08:50:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.476+0000 7f861b200640 1 -- start start 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c15ee20 0x7f861c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f861c0d18e0 0x7f861c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f861c0d4140 con 0x7f861c0d3350 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f861c0b83b0 con 0x7f861c0d18e0 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0d1d20 con 0x7f861c15ee20 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f8611400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f861c0d18e0 0x7f861c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f861c0d3350 0x7f861c067600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46904/0 (socket says 172.21.15.67:46904) 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.477+0000 7f861a800640 1 -- 172.21.15.67:0/1144780520 learned_addr learned my addr 172.21.15.67:0/1144780520 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8619e00640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c15ee20 0x7f861c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8611400640 1 -- 172.21.15.67:0/1144780520 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c15ee20 msgr2=0x7f861c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8611400640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c15ee20 0x7f861c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8611400640 1 -- 172.21.15.67:0/1144780520 >> v1:172.21.15.67:6789/0 conn(0x7f861c0d3350 legacy=0x7f861c067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8611400640 1 -- 172.21.15.67:0/1144780520 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c0d3b80 con 0x7f861c0d18e0 2024-04-17T08:50:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.478+0000 7f8619400640 1 -- 172.21.15.67:0/1144780520 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447356265 0 0) 0x7f861c0d4140 con 0x7f861c0d3350 2024-04-17T08:50:06.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.479+0000 7f8611400640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0d18e0 0x7f861c0d2e10 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f86040097a0 tx=0x7f860402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53c921562470867 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.479+0000 7f8619400640 1 -- 172.21.15.67:0/1144780520 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604035020 con 0x7f861c0d18e0 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.479+0000 7f8619400640 1 -- 172.21.15.67:0/1144780520 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8604031830 con 0x7f861c0d18e0 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.479+0000 7f8619400640 1 -- 172.21.15.67:0/1144780520 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604031c00 con 0x7f861c0d18e0 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 -- 172.21.15.67:0/1144780520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0d18e0 msgr2=0x7f861c0d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0d18e0 0x7f861c0d2e10 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f86040097a0 tx=0x7f860402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 -- 172.21.15.67:0/1144780520 shutdown_connections 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0d18e0 0x7f861c0d2e10 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 --2- 172.21.15.67:0/1144780520 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c15ee20 0x7f861c0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 -- 172.21.15.67:0/1144780520 >> 172.21.15.67:0/1144780520 conn(0x7f861c06d3c0 msgr2=0x7f861c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 -- 172.21.15.67:0/1144780520 shutdown_connections 2024-04-17T08:50:06.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.480+0000 7f861b200640 1 -- 172.21.15.67:0/1144780520 wait complete. 2024-04-17T08:50:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.482+0000 7f861b200640 1 Processor -- start 2024-04-17T08:50:06.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.482+0000 7f861b200640 1 -- start start 2024-04-17T08:50:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f861c0d18e0 0x7f861c15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c0d3350 0x7f861c161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f861c15f200 con 0x7f861c15ee20 2024-04-17T08:50:06.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f861c0b8040 con 0x7f861c0d18e0 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0d37e0 con 0x7f861c0d3350 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.483+0000 7f861a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59610/0 (socket says 172.21.15.67:59610) 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 -- 172.21.15.67:0/2345391391 learned_addr learned my addr 172.21.15.67:0/2345391391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f8611400640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c0d3350 0x7f861c161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f8619e00640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f861c0d18e0 0x7f861c15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c0d3350 msgr2=0x7f861c161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c0d3350 0x7f861c161980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f861c0d18e0 msgr2=0x7f861c15bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f861c0d18e0 0x7f861c15bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 -- 172.21.15.67:0/2345391391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c0ca920 con 0x7f861c15ee20 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.484+0000 7f861a800640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f861400c7a0 tx=0x7f861400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.485+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614018070 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.485+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861400a640 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.485+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861400aab0 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.485+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c0cac00 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.485+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c0cb1c0 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.486+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861c068a10 con 0x7f861c15ee20 2024-04-17T08:50:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.487+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7f861401b070 con 0x7f861c15ee20 2024-04-17T08:50:06.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.488+0000 7f8612a00640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 0x7f85f806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:06.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.488+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7f861408b460 con 0x7f861c15ee20 2024-04-17T08:50:06.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.488+0000 7f8619e00640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 msgr2=0x7f85f806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:06.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.488+0000 7f8619e00640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 0x7f85f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:06.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.492+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8614058ff0 con 0x7f861c15ee20 2024-04-17T08:50:06.518 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.519+0000 7eff7ac0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:50:06.548 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.549+0000 7efe4f22b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:06.572 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.573+0000 7eff7ac0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:50:06.655 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.655+0000 7efe4f22b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:50:06.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.689+0000 7f8619e00640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 msgr2=0x7f85f806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:06.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.689+0000 7f8619e00640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 0x7f85f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:06.706 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.707+0000 7eff7ac0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:06.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.707+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f861c0b99f0 con 0x7f861c15ee20 2024-04-17T08:50:06.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.708+0000 7f8612a00640 1 -- 172.21.15.67:0/2345391391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v215) ==== 80+0+476 (secure 0 0 0) 0x7f861405d2f0 con 0x7f861c15ee20 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3145453289 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/910136246 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/4251121774 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/2105214300 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/2105214300 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2605475186 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2381197913 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/713566291 2024-04-18T08:47:58.503956+0000 2024-04-17T08:50:06.708 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T08:50:06.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 msgr2=0x7f85f806ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 0x7f85f806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 msgr2=0x7f861c14fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f861400c7a0 tx=0x7f861400cc60 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 shutdown_connections 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f85f806ca90 0x7f85f806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c15ee20 0x7f861c14fbe0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f861c0d3350 0x7f861c161980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 --2- 172.21.15.67:0/2345391391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f861c0d18e0 0x7f861c15bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 >> 172.21.15.67:0/2345391391 conn(0x7f861c06d3c0 msgr2=0x7f861c15dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 shutdown_connections 2024-04-17T08:50:06.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:06.711+0000 7f861b200640 1 -- 172.21.15.67:0/2345391391 wait complete. 2024-04-17T08:50:06.729 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:50:06.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:06.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:50:06.828 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:06.828+0000 7eff7ac0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:50:06.829 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:50:06.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:06.830 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:50:06.834 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.834+0000 7efe4f22b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:06.872 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:50:06.872 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:50:06.873 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:06.885 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:06.885+0000 7efe4f22b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:06.921 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:50:06.921 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:50:06.922 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:06.922 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:06.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:50:06.975 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:50:06.975 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:50:06.975 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:07.016 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:07.017+0000 7eff7ac0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:07.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:50:07.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:07.029 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:07.030+0000 7efe4f22b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:50:07.069 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:07.070+0000 7eff7ac0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:07.218 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:07.218+0000 7eff7ac0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:50:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.242+0000 7f7ceee00640 1 Processor -- start 2024-04-17T08:50:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.243+0000 7f7ceee00640 1 -- start start 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.244+0000 7f7ceee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ce815ed10 0x7f7ce80b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.244+0000 7f7ceee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce80b48a0 con 0x7f7ce80b43d0 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.244+0000 7f7ceee00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce80b4b80 con 0x7f7ce806c8d0 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.244+0000 7f7ceee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce815f0f0 con 0x7f7ce815ed10 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7ce7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ce815ed10 0x7f7ce80b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7cece00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7ce80b43d0 0x7f7ce806e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46906/0 (socket says 172.21.15.67:46906) 2024-04-17T08:50:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7cece00640 1 -- 172.21.15.67:0/1501121276 learned_addr learned my addr 172.21.15.67:0/1501121276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7ce7e00640 1 -- 172.21.15.67:0/1501121276 >> v1:172.21.15.187:6789/0 conn(0x7f7ce806c8d0 legacy=0x7f7ce80b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7ce7e00640 1 -- 172.21.15.67:0/1501121276 >> v1:172.21.15.67:6789/0 conn(0x7f7ce80b43d0 legacy=0x7f7ce806e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.245+0000 7f7ce7e00640 1 -- 172.21.15.67:0/1501121276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce80b4e60 con 0x7f7ce815ed10 2024-04-17T08:50:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.247+0000 7f7ce7e00640 1 --2- 172.21.15.67:0/1501121276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce80b5940 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80097a0 tx=0x7f7cd802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f11bb41adfd595 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.247+0000 7f7ce7400640 1 -- 172.21.15.67:0/1501121276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8034070 con 0x7f7ce815ed10 2024-04-17T08:50:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.247+0000 7f7ce7400640 1 -- 172.21.15.67:0/1501121276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd8030610 con 0x7f7ce815ed10 2024-04-17T08:50:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.247+0000 7f7ce7400640 1 -- 172.21.15.67:0/1501121276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80309e0 con 0x7f7ce815ed10 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 -- 172.21.15.67:0/1501121276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 msgr2=0x7f7ce80b5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 --2- 172.21.15.67:0/1501121276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce80b5940 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80097a0 tx=0x7f7cd802b960 comp rx=0 tx=0).stop 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 -- 172.21.15.67:0/1501121276 shutdown_connections 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 --2- 172.21.15.67:0/1501121276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce80b5940 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 -- 172.21.15.67:0/1501121276 >> 172.21.15.67:0/1501121276 conn(0x7f7ce80725d0 msgr2=0x7f7ce80729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 -- 172.21.15.67:0/1501121276 shutdown_connections 2024-04-17T08:50:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.248+0000 7f7ceee00640 1 -- 172.21.15.67:0/1501121276 wait complete. 2024-04-17T08:50:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.251+0000 7f7ceee00640 1 Processor -- start 2024-04-17T08:50:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.251+0000 7f7ceee00640 1 -- start start 2024-04-17T08:50:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ce806c8d0 0x7f7ce815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce80b43d0 0x7f7ce815df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80b5420 con 0x7f7ce815ed10 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce80b5e80 con 0x7f7ce80b43d0 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ceee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce806cd60 con 0x7f7ce806c8d0 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7cece00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7cece00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59626/0 (socket says 172.21.15.67:59626) 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7cece00640 1 -- 172.21.15.67:0/2511731071 learned_addr learned my addr 172.21.15.67:0/2511731071 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7ce7e00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ce806c8d0 0x7f7ce815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.253+0000 7f7cdfe00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce80b43d0 0x7f7ce815df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ce806c8d0 msgr2=0x7f7ce815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ce806c8d0 0x7f7ce815bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce80b43d0 msgr2=0x7f7ce815df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce80b43d0 0x7f7ce815df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 -- 172.21.15.67:0/2511731071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce8153060 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7cece00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000cb20 tx=0x7f7ce000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0017070 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce000aca0 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce815f320 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.254+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0012600 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.255+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce815f8e0 con 0x7f7ce815ed10 2024-04-17T08:50:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.256+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce8068b90 con 0x7f7ce815ed10 2024-04-17T08:50:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.257+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7f7ce001a070 con 0x7f7ce815ed10 2024-04-17T08:50:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.258+0000 7f7ce4c00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 0x7f7cc406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.258+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7f7ce008bab0 con 0x7f7ce815ed10 2024-04-17T08:50:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.258+0000 7f7ce7e00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 msgr2=0x7f7cc406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.258+0000 7f7ce7e00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 0x7f7cc406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.262+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7ce0059660 con 0x7f7ce815ed10 2024-04-17T08:50:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.459+0000 7f7ce7e00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 msgr2=0x7f7cc406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:07.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.459+0000 7f7ce7e00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 0x7f7cc406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.504+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ce80b2560 con 0x7f7ce815ed10 2024-04-17T08:50:07.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.505+0000 7f7ce4c00640 1 -- 172.21.15.67:0/2511731071 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7f7ce005d960 con 0x7f7ce815ed10 2024-04-17T08:50:07.507 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:07.507 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 msgr2=0x7f7cc406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 0x7f7cc406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 msgr2=0x7f7ce8161490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000cb20 tx=0x7f7ce000a060 comp rx=0 tx=0).stop 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 shutdown_connections 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7f7cc406ca10 0x7f7cc406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce815ed10 0x7f7ce8161490 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ce80b43d0 0x7f7ce815df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 --2- 172.21.15.67:0/2511731071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ce806c8d0 0x7f7ce815bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 >> 172.21.15.67:0/2511731071 conn(0x7f7ce80725d0 msgr2=0x7f7ce806fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:07.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 shutdown_connections 2024-04-17T08:50:07.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.508+0000 7f7ceee00640 1 -- 172.21.15.67:0/2511731071 wait complete. 2024-04-17T08:50:07.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.776+0000 7fb7d3a00640 1 Processor -- start 2024-04-17T08:50:07.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.776+0000 7fb7d3a00640 1 -- start start 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7cc15ed10 0x7fb7cc0d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d3a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7cc0cff60 con 0x7fb7cc0d2ab0 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7cc0d3c30 con 0x7fb7cc15ed10 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d3020 con 0x7fb7cc0cf640 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7cbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7cc15ed10 0x7fb7cc0d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d1a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7cc0d2ab0 0x7fb7cc0b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46918/0 (socket says 172.21.15.67:46918) 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.777+0000 7fb7d1a00640 1 -- 172.21.15.67:0/231153885 learned_addr learned my addr 172.21.15.67:0/231153885 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7d1000640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7cb400640 1 -- 172.21.15.67:0/231153885 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2063803232 0 0) 0x7fb7cc0cff60 con 0x7fb7cc0d2ab0 2024-04-17T08:50:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7cbe00640 1 -- 172.21.15.67:0/231153885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 msgr2=0x7fb7cc0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7cbe00640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7cbe00640 1 -- 172.21.15.67:0/231153885 >> v1:172.21.15.67:6789/0 conn(0x7fb7cc0d2ab0 legacy=0x7fb7cc0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.778+0000 7fb7cbe00640 1 -- 172.21.15.67:0/231153885 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc0d3300 con 0x7fb7cc15ed10 2024-04-17T08:50:07.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.780+0000 7fb7cbe00640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc0d40c0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80097a0 tx=0x7fb7b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfc47e8f9ba6a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:07.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.780+0000 7fb7cb400640 1 -- 172.21.15.67:0/231153885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8034070 con 0x7fb7cc15ed10 2024-04-17T08:50:07.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.780+0000 7fb7cb400640 1 -- 172.21.15.67:0/231153885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b8030610 con 0x7fb7cc15ed10 2024-04-17T08:50:07.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.780+0000 7fb7cb400640 1 -- 172.21.15.67:0/231153885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80309e0 con 0x7fb7cc15ed10 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 -- 172.21.15.67:0/231153885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 msgr2=0x7fb7cc0d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc0d40c0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80097a0 tx=0x7fb7b802b960 comp rx=0 tx=0).stop 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 -- 172.21.15.67:0/231153885 shutdown_connections 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc0d40c0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/231153885 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.781+0000 7fb7d3a00640 1 -- 172.21.15.67:0/231153885 >> 172.21.15.67:0/231153885 conn(0x7fb7cc06cbd0 msgr2=0x7fb7cc06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.782+0000 7fb7d3a00640 1 -- 172.21.15.67:0/231153885 shutdown_connections 2024-04-17T08:50:07.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.782+0000 7fb7d3a00640 1 -- 172.21.15.67:0/231153885 wait complete. 2024-04-17T08:50:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.783+0000 7fb7d3a00640 1 Processor -- start 2024-04-17T08:50:07.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.783+0000 7fb7d3a00640 1 -- start start 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7cc0d2ab0 0x7fb7cc14f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d25b0 con 0x7fb7cc15ed10 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d38c0 con 0x7fb7cc0d2ab0 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.784+0000 7fb7d3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d4600 con 0x7fb7cc0cf640 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59638/0 (socket says 172.21.15.67:59638) 2024-04-17T08:50:07.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 -- 172.21.15.67:0/3189534400 learned_addr learned my addr 172.21.15.67:0/3189534400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:07.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1000640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7cbe00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7cc0d2ab0 0x7fb7cc14f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 msgr2=0x7fb7cc0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7cc0d2ab0 msgr2=0x7fb7cc14f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7cc0d2ab0 0x7fb7cc14f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.785+0000 7fb7d1a00640 1 -- 172.21.15.67:0/3189534400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc0b5200 con 0x7fb7cc15ed10 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7d1a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400c7d0 tx=0x7fb7c400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4018070 con 0x7fb7cc15ed10 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c400a640 con 0x7fb7cc15ed10 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c400aa70 con 0x7fb7cc15ed10 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7cc0b54e0 con 0x7fb7cc15ed10 2024-04-17T08:50:07.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.786+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7cc0b5aa0 con 0x7fb7cc15ed10 2024-04-17T08:50:07.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.787+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7cc068b90 con 0x7fb7cc15ed10 2024-04-17T08:50:07.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.788+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7fb7c400ac10 con 0x7fb7cc15ed10 2024-04-17T08:50:07.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.790+0000 7fb7c8c00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 0x7fb7ac06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:07.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.790+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6139+0+0 (secure 0 0 0) 0x7fb7c408b470 con 0x7fb7cc15ed10 2024-04-17T08:50:07.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.790+0000 7fb7d1000640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 msgr2=0x7fb7ac06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:07.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.790+0000 7fb7d1000640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 0x7fb7ac06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:07.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.795+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb7c4058f90 con 0x7fb7cc15ed10 2024-04-17T08:50:07.853 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:07.854+0000 7efe4f22b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:50:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.991+0000 7fb7d1000640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 msgr2=0x7fb7ac06eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:07.991+0000 7fb7d1000640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 0x7fb7ac06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:08.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.013+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7cc0d2730 con 0x7fb7cc15ed10 2024-04-17T08:50:08.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.014+0000 7fb7c8c00640 1 -- 172.21.15.67:0/3189534400 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) ==== 74+0+17223 (secure 0 0 0) 0x7fb7c405d290 con 0x7fb7cc15ed10 2024-04-17T08:50:08.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:08.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":215,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:48:59.795260+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.7","mappings":[{"from":0,"to":2}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3145453289":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/910136246":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/4251121774":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6833/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6832/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2605475186":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2381197913":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/713566291":"2024-04-18T08:47:58.503956+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 msgr2=0x7fb7ac06eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 0x7fb7ac06eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 msgr2=0x7fb7cc14fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400c7d0 tx=0x7fb7c400cc90 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 shutdown_connections 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fb7ac06ca10 0x7fb7ac06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7cc15ed10 0x7fb7cc14fe60 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7cc0d2ab0 0x7fb7cc14f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 --2- 172.21.15.67:0/3189534400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7cc0cf640 0x7fb7cc0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 >> 172.21.15.67:0/3189534400 conn(0x7fb7cc06cbd0 msgr2=0x7fb7cc0c95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 shutdown_connections 2024-04-17T08:50:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.017+0000 7fb7d3a00640 1 -- 172.21.15.67:0/3189534400 wait complete. 2024-04-17T08:50:08.035 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:47:41.219096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:47:42.272552+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:50:08.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:50:08.086 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:08.086+0000 7eff7ac0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:50:08.172 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.172+0000 7efe4f22b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:50:08.230 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.230+0000 7efe4f22b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:50:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.244+0000 7fa233200640 1 Processor -- start 2024-04-17T08:50:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.244+0000 7fa233200640 1 -- start start 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.246+0000 7fa233200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa234161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.246+0000 7fa233200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2340b4e40 con 0x7fa2341620e0 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.246+0000 7fa233200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2340b5110 con 0x7fa2340d2ff0 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.246+0000 7fa233200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa234162530 con 0x7fa2340d33d0 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.247+0000 7fa232800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2341620e0 0x7fa2340cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46928/0 (socket says 172.21.15.67:46928) 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.247+0000 7fa232800640 1 -- 172.21.15.67:0/2961527188 learned_addr learned my addr 172.21.15.67:0/2961527188 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:08.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.247+0000 7fa231400640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa234161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.247+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277941363 0 0) 0x7fa2340b4e40 con 0x7fa2341620e0 2024-04-17T08:50:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.247+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa218003520 con 0x7fa2341620e0 2024-04-17T08:50:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3833636819 0 0) 0x7fa2340b5110 con 0x7fa2340d2ff0 2024-04-17T08:50:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2340b4e40 con 0x7fa2340d2ff0 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 192307154 0 0) 0x7fa218003520 con 0x7fa2341620e0 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 msgr2=0x7fa234161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa234161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 >> v1:172.21.15.187:6789/0 conn(0x7fa2340d2ff0 legacy=0x7fa2340b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2340b53f0 con 0x7fa2341620e0 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa22c0019f0 con 0x7fa2341620e0 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 >> v1:172.21.15.67:6789/0 conn(0x7fa2341620e0 legacy=0x7fa2340cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa218003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa218004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa218004d40 0x7fa2180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa234162530 con 0x7fa218004120 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa21800bb00 con 0x7fa218004d40 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa21800bd00 con 0x7fa2340d33d0 2024-04-17T08:50:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa231400640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa218004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa232800640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa218004d40 0x7fa2180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.248+0000 7fa231e00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa218003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231400640 1 -- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 msgr2=0x7fa218003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231400640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa218003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231400640 1 -- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa218004d40 msgr2=0x7fa2180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231400640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa218004d40 0x7fa2180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231400640 1 -- 172.21.15.67:0/2961527188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa218003520 con 0x7fa218004120 2024-04-17T08:50:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.249+0000 7fa231e00640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340d33d0 0x7fa218003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.250+0000 7fa231400640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa218004800 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fa220009ac0 tx=0x7fa218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c317114932a680 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.250+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220039070 con 0x7fa218004120 2024-04-17T08:50:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.250+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa22002c600 con 0x7fa218004120 2024-04-17T08:50:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.250+0000 7fa230a00640 1 -- 172.21.15.67:0/2961527188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220009690 con 0x7fa218004120 2024-04-17T08:50:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 -- 172.21.15.67:0/2961527188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 msgr2=0x7fa218004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 --2- 172.21.15.67:0/2961527188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa218004800 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fa220009ac0 tx=0x7fa218005830 comp rx=0 tx=0).stop 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa232800640 1 -- 172.21.15.67:0/2961527188 reap_dead start 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 -- 172.21.15.67:0/2961527188 shutdown_connections 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 -- 172.21.15.67:0/2961527188 >> 172.21.15.67:0/2961527188 conn(0x7fa23409da80 msgr2=0x7fa23408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 -- 172.21.15.67:0/2961527188 shutdown_connections 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.251+0000 7fa233200640 1 -- 172.21.15.67:0/2961527188 wait complete. 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.253+0000 7fa233200640 1 Processor -- start 2024-04-17T08:50:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.253+0000 7fa233200640 1 -- start start 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2340c9570 0x7fa2340c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340b8130 0x7fa2340b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2340cf4c0 con 0x7fa218004120 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2340cf340 con 0x7fa2340c9570 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa233200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2340d26d0 con 0x7fa2340b8130 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa231e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa231e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59648/0 (socket says 172.21.15.67:59648) 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa231e00640 1 -- 172.21.15.67:0/1031396345 learned_addr learned my addr 172.21.15.67:0/1031396345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.254+0000 7fa231400640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2340c9570 0x7fa2340c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa232800640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340b8130 0x7fa2340b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 -- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340b8130 msgr2=0x7fa2340b85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340b8130 0x7fa2340b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 -- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2340c9570 msgr2=0x7fa2340c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2340c9570 0x7fa2340c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 -- 172.21.15.67:0/1031396345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2340c92f0 con 0x7fa218004120 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa231e00640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e7b0 tx=0x7fa22400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22400ee60 con 0x7fa218004120 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2240052e0 con 0x7fa218004120 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224007020 con 0x7fa218004120 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.255+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2340b7370 con 0x7fa218004120 2024-04-17T08:50:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.256+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2340b7930 con 0x7fa218004120 2024-04-17T08:50:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.257+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2340755d0 con 0x7fa218004120 2024-04-17T08:50:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.258+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 125) ==== 91154+0+0 (secure 0 0 0) 0x7fa22401a020 con 0x7fa218004120 2024-04-17T08:50:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.259+0000 7fa22a000640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 0x7fa21406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.259+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6195+0+0 (secure 0 0 0) 0x7fa22408b660 con 0x7fa218004120 2024-04-17T08:50:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.259+0000 7fa231400640 1 -- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 msgr2=0x7fa21406eed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/953837164 2024-04-17T08:50:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.259+0000 7fa231400640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 0x7fa21406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:08.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.263+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa224059820 con 0x7fa218004120 2024-04-17T08:50:08.377 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.378+0000 7efe4f22b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:50:08.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.392+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 126) ==== 90751+0+0 (secure 0 0 0) 0x7fa224055790 con 0x7fa218004120 2024-04-17T08:50:08.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.393+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 msgr2=0x7fa21406eed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:08.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.393+0000 7fa22a000640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 0x7fa21406eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.441 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:08.442+0000 7eff7ac0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:50:08.463 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.464+0000 7efdcb800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.464+0000 7efdcb800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.464+0000 7efdcb800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.464+0000 7efdcb800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.464+0000 7efdcb800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.464 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.465+0000 7efdc7c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.465 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.465+0000 7efdc7c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.465 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.465+0000 7efdc7c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.465 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.465+0000 7efdc7c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.465 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:08.465+0000 7efdc7c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:50:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.489+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2340b1410 con 0x7fa218004120 2024-04-17T08:50:08.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.490+0000 7fa22a000640 1 -- 172.21.15.67:0/1031396345 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) ==== 93+0+10 (secure 0 0 0) 0x7fa224076ee0 con 0x7fa218004120 2024-04-17T08:50:08.490 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:50:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 msgr2=0x7fa23414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e7b0 tx=0x7fa22400ec70 comp rx=0 tx=0).stop 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 shutdown_connections 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:6832/953837164,v1:172.21.15.187:6833/953837164] conn(0x7fa21406ca10 0x7fa21406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2340b8130 0x7fa2340b85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2340c9570 0x7fa2340c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 --2- 172.21.15.67:0/1031396345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa218004120 0x7fa23414fb40 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 >> 172.21.15.67:0/1031396345 conn(0x7fa23409da80 msgr2=0x7fa2340d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 shutdown_connections 2024-04-17T08:50:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.493+0000 7fa233200640 1 -- 172.21.15.67:0/1031396345 wait complete. 2024-04-17T08:50:08.505 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:08.506+0000 7eff7ac0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:50:08.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:50:08.645 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:08.645+0000 7eff7ac0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:50:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.707+0000 7ff7c2c00640 1 Processor -- start 2024-04-17T08:50:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.708+0000 7ff7c2c00640 1 -- start start 2024-04-17T08:50:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.709+0000 7ff7c2c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7bc0ce4d0 0x7ff7bc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.709+0000 7ff7c2c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7bc071f60 con 0x7ff7bc15b860 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.709+0000 7ff7c2c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7bc072240 con 0x7ff7bc0d0b30 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.709+0000 7ff7c2c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff7bc071d10 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7bbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7bc15b860 0x7ff7bc0d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46936/0 (socket says 172.21.15.67:46936) 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7bbe00640 1 -- 172.21.15.67:0/2842442412 learned_addr learned my addr 172.21.15.67:0/2842442412 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7c0c00640 1 --2- 172.21.15.67:0/2842442412 >> v2:172.21.15.67:3300/0 conn(0x7ff7bc0ce4d0 0x7ff7bc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7bb400640 1 -- 172.21.15.67:0/2842442412 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921237593 0 0) 0x7ff7bc071f60 con 0x7ff7bc15b860 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7bb400640 1 -- 172.21.15.67:0/2842442412 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a0005730 con 0x7ff7bc15b860 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2842442412 >> v1:172.21.15.187:6789/0 conn(0x7ff7bc0d0b30 legacy=0x7ff7bc0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2842442412 >> v1:172.21.15.67:6789/0 conn(0x7ff7bc15b860 legacy=0x7ff7bc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.710+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2842442412 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc072520 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.711+0000 7ff7c0c00640 1 --2- 172.21.15.67:0/2842442412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc0b4680 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009d90 tx=0x7ff7b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca99b221b698027d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7bb400640 1 -- 172.21.15.67:0/2842442412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4039070 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7bb400640 1 -- 172.21.15.67:0/2842442412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7b402cc60 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7bb400640 1 -- 172.21.15.67:0/2842442412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4034640 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2842442412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc0ce4d0 msgr2=0x7ff7bc0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2842442412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc0b4680 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009d90 tx=0x7ff7b402beb0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2842442412 shutdown_connections 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2842442412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc0b4680 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2842442412 >> 172.21.15.67:0/2842442412 conn(0x7ff7bc067ee0 msgr2=0x7ff7bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2842442412 shutdown_connections 2024-04-17T08:50:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.712+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2842442412 wait complete. 2024-04-17T08:50:08.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.714+0000 7ff7c2c00640 1 Processor -- start 2024-04-17T08:50:08.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.714+0000 7ff7c2c00640 1 -- start start 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc14a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 0x7ff7bc151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc15bc40 con 0x7ff7bc15b860 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc072ae0 con 0x7ff7bc0ce4d0 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c2c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7bc06f380 con 0x7ff7bc0d0b30 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59660/0 (socket says 172.21.15.67:59660) 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.715+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2638030845 learned_addr learned my addr 172.21.15.67:0/2638030845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7b3e00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 0x7ff7bc151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7bbe00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc14a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 msgr2=0x7ff7bc151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 0x7ff7bc151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 msgr2=0x7ff7bc14a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc14a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 -- 172.21.15.67:0/2638030845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc0ae380 con 0x7ff7bc15b860 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7b3e00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 0x7ff7bc151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7bbe00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc14a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c0c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009eb0 tx=0x7ff7b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4039050 con 0x7ff7bc15b860 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7b4045070 con 0x7ff7bc15b860 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4009450 con 0x7ff7bc15b860 2024-04-17T08:50:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.716+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc0ae660 con 0x7ff7bc15b860 2024-04-17T08:50:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.717+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc0aec20 con 0x7ff7bc15b860 2024-04-17T08:50:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.718+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7bc0d0ff0 con 0x7ff7bc15b860 2024-04-17T08:50:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.719+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 126) ==== 90751+0+0 (secure 0 0 0) 0x7ff7b403e830 con 0x7ff7bc15b860 2024-04-17T08:50:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.720+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7ff7b40ae4b0 con 0x7ff7bc15b860 2024-04-17T08:50:08.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.724+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff7b404f020 con 0x7ff7bc15b860 2024-04-17T08:50:08.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.934+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff7bc0685d0 con 0x7ff7bc15b860 2024-04-17T08:50:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.935+0000 7ff7b8c00640 1 -- 172.21.15.67:0/2638030845 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v217) ==== 104+0+11 (secure 0 0 0) 0x7ff7b4032e20 con 0x7ff7bc15b860 2024-04-17T08:50:08.934 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:50:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.937+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 msgr2=0x7ff7bc152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.937+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009eb0 tx=0x7ff7b4004a40 comp rx=0 tx=0).stop 2024-04-17T08:50:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 shutdown_connections 2024-04-17T08:50:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc15b860 0x7ff7bc152350 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff7bc0d0b30 0x7ff7bc151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 --2- 172.21.15.67:0/2638030845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7bc0ce4d0 0x7ff7bc14a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 >> 172.21.15.67:0/2638030845 conn(0x7ff7bc067ee0 msgr2=0x7ff7bc0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 shutdown_connections 2024-04-17T08:50:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:08.938+0000 7ff7c2c00640 1 -- 172.21.15.67:0/2638030845 wait complete. 2024-04-17T08:50:08.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:50:09.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.179+0000 7f530fe00640 1 Processor -- start 2024-04-17T08:50:09.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.180+0000 7f530fe00640 1 -- start start 2024-04-17T08:50:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53100b3b50 0x7f53100d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53100cfcd0 0x7f53100d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53100d4f70 con 0x7f531015ec20 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f53100d5250 con 0x7f53100b3b50 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53100cf4d0 con 0x7f53100cfcd0 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53100b3b50 0x7f53100d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.181+0000 7f530ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f531015ec20 0x7f531006ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46952/0 (socket says 172.21.15.67:46952) 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530ea00640 1 -- 172.21.15.67:0/2698446840 learned_addr learned my addr 172.21.15.67:0/2698446840 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530e000640 1 -- 172.21.15.67:0/2698446840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53100cfcd0 msgr2=0x7f53100d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530e000640 1 --2- 172.21.15.67:0/2698446840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53100cfcd0 0x7f53100d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530e000640 1 -- 172.21.15.67:0/2698446840 >> v1:172.21.15.67:6789/0 conn(0x7f531015ec20 legacy=0x7f531006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530e000640 1 -- 172.21.15.67:0/2698446840 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53100cf7b0 con 0x7f53100b3b50 2024-04-17T08:50:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.182+0000 7f530d600640 1 -- 172.21.15.67:0/2698446840 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729815737 0 0) 0x7f53100d4f70 con 0x7f531015ec20 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.183+0000 7f530e000640 1 --2- 172.21.15.67:0/2698446840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100d22b0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009790 tx=0x7f52fc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e147909abdb7b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530d600640 1 -- 172.21.15.67:0/2698446840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc039070 con 0x7f53100b3b50 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530d600640 1 -- 172.21.15.67:0/2698446840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52fc0316f0 con 0x7f53100b3b50 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530d600640 1 -- 172.21.15.67:0/2698446840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc031ac0 con 0x7f53100b3b50 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 -- 172.21.15.67:0/2698446840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 msgr2=0x7f53100d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 --2- 172.21.15.67:0/2698446840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100d22b0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009790 tx=0x7f52fc02b790 comp rx=0 tx=0).stop 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 -- 172.21.15.67:0/2698446840 shutdown_connections 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 --2- 172.21.15.67:0/2698446840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53100cfcd0 0x7f53100d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 --2- 172.21.15.67:0/2698446840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100d22b0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 -- 172.21.15.67:0/2698446840 >> 172.21.15.67:0/2698446840 conn(0x7f531008bff0 msgr2=0x7f53100d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:09.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 -- 172.21.15.67:0/2698446840 shutdown_connections 2024-04-17T08:50:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.184+0000 7f530fe00640 1 -- 172.21.15.67:0/2698446840 wait complete. 2024-04-17T08:50:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.186+0000 7f530fe00640 1 Processor -- start 2024-04-17T08:50:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.186+0000 7f530fe00640 1 -- start start 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 0x7f5310145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 0x7f5310146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f531006d2d0 con 0x7f53100b3b50 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f531006d150 con 0x7f53100cfcd0 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53100d10d0 con 0x7f531015ec20 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59666/0 (socket says 172.21.15.67:59666) 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.187+0000 7f530ea00640 1 -- 172.21.15.67:0/653625793 learned_addr learned my addr 172.21.15.67:0/653625793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530e000640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 0x7f5310145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 -- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 msgr2=0x7f5310146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530f400640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 0x7f5310146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 0x7f5310146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 -- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 msgr2=0x7f5310145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 0x7f5310145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 -- 172.21.15.67:0/653625793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53100b48e0 con 0x7f53100b3b50 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530e000640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 0x7f5310145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530f400640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 0x7f5310146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.188+0000 7f530ea00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f530000c460 tx=0x7f530000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.189+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300018070 con 0x7f53100b3b50 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.189+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5300007500 con 0x7f53100b3b50 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.189+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300004a20 con 0x7f53100b3b50 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.189+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53100ba180 con 0x7f53100b3b50 2024-04-17T08:50:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.189+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53100cd3b0 con 0x7f53100b3b50 2024-04-17T08:50:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.190+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5310068a10 con 0x7f53100b3b50 2024-04-17T08:50:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.191+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 126) ==== 90751+0+0 (secure 0 0 0) 0x7f530001b070 con 0x7f53100b3b50 2024-04-17T08:50:09.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.192+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7f530008acd0 con 0x7f53100b3b50 2024-04-17T08:50:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.196+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f53100b48e0 con 0x7f53100b3b50 2024-04-17T08:50:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.395+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f53100d55f0 con 0x7f53100b3b50 2024-04-17T08:50:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.403+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 127) ==== 90878+0+0 (secure 0 0 0) 0x7f5300054790 con 0x7f53100b3b50 2024-04-17T08:50:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.403+0000 7f5306a00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f52f00721b0 0x7f52f00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.404+0000 7f530e000640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f52f00721b0 0x7f52f00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.407+0000 7f5306a00640 1 -- 172.21.15.67:0/653625793 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v217) ==== 100+0+11 (secure 0 0 0) 0x7f5300056090 con 0x7f53100b3b50 2024-04-17T08:50:09.407 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:50:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.410+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f52f00721b0 msgr2=0x7f52f00745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f52f00721b0 0x7f52f00745a0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f52fc036000 tx=0x7f52fc008030 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 msgr2=0x7f53100b9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f530000c460 tx=0x7f530000c920 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 shutdown_connections 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f52f00721b0 0x7f52f00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f531015ec20 0x7f5310146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53100cfcd0 0x7f5310145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 --2- 172.21.15.67:0/653625793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53100b3b50 0x7f53100b9c40 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 >> 172.21.15.67:0/653625793 conn(0x7f531008bff0 msgr2=0x7f5310074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 shutdown_connections 2024-04-17T08:50:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.411+0000 7f530fe00640 1 -- 172.21.15.67:0/653625793 wait complete. 2024-04-17T08:50:09.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:09.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.640+0000 7f5696000640 1 Processor -- start 2024-04-17T08:50:09.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.641+0000 7f5696000640 1 -- start start 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.642+0000 7f5696000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f569015ec60 0x7f56900b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.642+0000 7f5696000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56900b6b30 0x7f56900cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.642+0000 7f5696000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56900d2d30 con 0x7f56900cf910 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.642+0000 7f5696000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f56900d49c0 con 0x7f56900b6b30 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.642+0000 7f5696000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f56900d3660 con 0x7f569015ec60 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56900b6b30 0x7f56900cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56900b6b30 0x7f56900cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59682/0 (socket says 172.21.15.67:59682) 2024-04-17T08:50:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 -- 172.21.15.67:0/2151714577 learned_addr learned my addr 172.21.15.67:0/2151714577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f5694c00640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f569015ec60 0x7f56900b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 -- 172.21.15.67:0/2151714577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f569015ec60 msgr2=0x7f56900b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f569015ec60 0x7f56900b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 -- 172.21.15.67:0/2151714577 >> v1:172.21.15.67:6789/0 conn(0x7f56900cf910 legacy=0x7f569006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568fe00640 1 -- 172.21.15.67:0/2151714577 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56900d3010 con 0x7f56900b6b30 2024-04-17T08:50:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.643+0000 7f568f400640 1 -- 172.21.15.67:0/2151714577 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054874249 0 0) 0x7f56900d2d30 con 0x7f56900cf910 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f568fe00640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56900cf3d0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f567c009870 tx=0x7f567c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc118ca7f8325f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f568f400640 1 -- 172.21.15.67:0/2151714577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c034070 con 0x7f56900b6b30 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f568f400640 1 -- 172.21.15.67:0/2151714577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c030840 con 0x7f56900b6b30 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f568f400640 1 -- 172.21.15.67:0/2151714577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c030c10 con 0x7f56900b6b30 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f5696000640 1 -- 172.21.15.67:0/2151714577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 msgr2=0x7f56900cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.645+0000 7f5696000640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56900cf3d0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f567c009870 tx=0x7f567c02b980 comp rx=0 tx=0).stop 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 -- 172.21.15.67:0/2151714577 shutdown_connections 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56900cf3d0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 --2- 172.21.15.67:0/2151714577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f569015ec60 0x7f56900b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 -- 172.21.15.67:0/2151714577 >> 172.21.15.67:0/2151714577 conn(0x7f56900d15f0 msgr2=0x7f56900d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 -- 172.21.15.67:0/2151714577 shutdown_connections 2024-04-17T08:50:09.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.646+0000 7f5696000640 1 -- 172.21.15.67:0/2151714577 wait complete. 2024-04-17T08:50:09.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.647+0000 7f5696000640 1 Processor -- start 2024-04-17T08:50:09.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.647+0000 7f5696000640 1 -- start start 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56900cf910 0x7f56900ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f569015ec60 0x7f56901f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56900cfda0 con 0x7f56900b6b30 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f56900b6f70 con 0x7f569015ec60 2024-04-17T08:50:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5696000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f569015f040 con 0x7f56900cf910 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.648+0000 7f5694c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59684/0 (socket says 172.21.15.67:59684) 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 -- 172.21.15.67:0/3633977631 learned_addr learned my addr 172.21.15.67:0/3633977631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f568fe00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56900cf910 0x7f56900ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 -- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56900cf910 msgr2=0x7f56900ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56900cf910 0x7f56900ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 -- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f569015ec60 msgr2=0x7f56901f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f569015ec60 0x7f56901f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 -- 172.21.15.67:0/3633977631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56901f4c30 con 0x7f56900b6b30 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.649+0000 7f5694c00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f568000d800 tx=0x7f568000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.650+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568000b6c0 con 0x7f56900b6b30 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.650+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5680019070 con 0x7f56900b6b30 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.650+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568000ba80 con 0x7f56900b6b30 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.650+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56901f4f10 con 0x7f56900b6b30 2024-04-17T08:50:09.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.650+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56901f57d0 con 0x7f56900b6b30 2024-04-17T08:50:09.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.651+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5690068a10 con 0x7f56900b6b30 2024-04-17T08:50:09.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.652+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 127) ==== 90878+0+0 (secure 0 0 0) 0x7f568000bc20 con 0x7f56900b6b30 2024-04-17T08:50:09.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.653+0000 7f568cc00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 0x7f567006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:09.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.653+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7f568008bfe0 con 0x7f56900b6b30 2024-04-17T08:50:09.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.654+0000 7f568fe00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 0x7f567006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:09.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.655+0000 7f568fe00640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 0x7f567006ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f567c036000 tx=0x7f567c03b040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:09.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.658+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f568005a3a0 con 0x7f56900b6b30 2024-04-17T08:50:09.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.905+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56900b9d90 con 0x7f56900b6b30 2024-04-17T08:50:09.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.906+0000 7f568cc00640 1 -- 172.21.15.67:0/3633977631 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7f5680059ca0 con 0x7f56900b6b30 2024-04-17T08:50:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:09.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:50:09.906 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 msgr2=0x7f567006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 0x7f567006ec00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f567c036000 tx=0x7f567c03b040 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 msgr2=0x7f56901510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f568000d800 tx=0x7f568000dcc0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 shutdown_connections 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f567006c740 0x7f567006ec00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f569015ec60 0x7f56901f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56900cf910 0x7f56900ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 --2- 172.21.15.67:0/3633977631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56900b6b30 0x7f56901510d0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:09.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 >> 172.21.15.67:0/3633977631 conn(0x7f56900d15f0 msgr2=0x7f56900b6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:09.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.909+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 shutdown_connections 2024-04-17T08:50:09.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:09.910+0000 7f5696000640 1 -- 172.21.15.67:0/3633977631 wait complete. 2024-04-17T08:50:09.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.149+0000 7f0cb7e00640 1 Processor -- start 2024-04-17T08:50:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.150+0000 7f0cb7e00640 1 -- start start 2024-04-17T08:50:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0cb80b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0cb80cfae0 0x7f0cb80b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb80d2d30 con 0x7f0cb80b6b30 2024-04-17T08:50:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0cb80d49c0 con 0x7f0cb80cfae0 2024-04-17T08:50:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb80d3660 con 0x7f0cb815ec60 2024-04-17T08:50:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb6000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cb80b6b30 0x7f0cb80cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46968/0 (socket says 172.21.15.67:46968) 2024-04-17T08:50:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.151+0000 7f0cb6000640 1 -- 172.21.15.67:0/2459776168 learned_addr learned my addr 172.21.15.67:0/2459776168 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb6a00640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0cb80b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb7400640 1 --2- 172.21.15.67:0/2459776168 >> v2:172.21.15.187:3300/0 conn(0x7f0cb80cfae0 0x7f0cb80b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367917036 0 0) 0x7f0cb80d2d30 con 0x7f0cb80b6b30 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c9c003520 con 0x7f0cb80b6b30 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4242979696 0 0) 0x7f0c9c003520 con 0x7f0cb80b6b30 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 msgr2=0x7f0cb80b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0cb80b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 >> v2:172.21.15.187:3300/0 conn(0x7f0cb80cfae0 msgr2=0x7f0cb80b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 --2- 172.21.15.67:0/2459776168 >> v2:172.21.15.187:3300/0 conn(0x7f0cb80cfae0 0x7f0cb80b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb80d3010 con 0x7f0cb80b6b30 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0ca00019f0 con 0x7f0cb80b6b30 2024-04-17T08:50:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 >> v1:172.21.15.67:6789/0 conn(0x7f0cb80b6b30 legacy=0x7f0cb80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0c9c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.152+0000 7f0cb5600640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c003fa0 0x7f0c9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb5600640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb80d49c0 con 0x7f0c9c003fa0 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb80d3660 con 0x7f0c9c004d40 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c00b9f0 con 0x7f0cb815ec60 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c003fa0 0x7f0c9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6a00640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0c9c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb7400640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 -- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 msgr2=0x7f0c9c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb815ec60 0x7f0c9c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 -- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c9c004d40 msgr2=0x7f0c9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c9c004d40 0x7f0c9c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.153+0000 7f0cb6000640 1 -- 172.21.15.67:0/2459776168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb80d3010 con 0x7f0c9c003fa0 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.155+0000 7f0cb6000640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c003fa0 0x7f0c9c004800 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000a8e0 tx=0x7f0c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad76c4a7eb1d768 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.155+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca000a4e0 con 0x7f0c9c003fa0 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.155+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca0034650 con 0x7f0c9c003fa0 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.155+0000 7f0cb5600640 1 -- 172.21.15.67:0/2459776168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0034a40 con 0x7f0c9c003fa0 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 -- 172.21.15.67:0/2459776168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c003fa0 msgr2=0x7f0c9c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/2459776168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c9c003fa0 0x7f0c9c004800 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000a8e0 tx=0x7f0c9c005830 comp rx=0 tx=0).stop 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7400640 1 -- 172.21.15.67:0/2459776168 reap_dead start 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 -- 172.21.15.67:0/2459776168 shutdown_connections 2024-04-17T08:50:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 -- 172.21.15.67:0/2459776168 >> 172.21.15.67:0/2459776168 conn(0x7f0cb80d15f0 msgr2=0x7f0cb80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 -- 172.21.15.67:0/2459776168 shutdown_connections 2024-04-17T08:50:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.156+0000 7f0cb7e00640 1 -- 172.21.15.67:0/2459776168 wait complete. 2024-04-17T08:50:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.158+0000 7f0cb7e00640 1 Processor -- start 2024-04-17T08:50:10.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.158+0000 7f0cb7e00640 1 -- start start 2024-04-17T08:50:10.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb81f4c60 0x7f0cb815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 0x7f0cb80ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb815f040 con 0x7f0cb8151fe0 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb80b6f70 con 0x7f0cb815af40 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb80d00c0 con 0x7f0cb81f4c60 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59694/0 (socket says 172.21.15.67:59694) 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.159+0000 7f0cb6a00640 1 -- 172.21.15.67:0/940798713 learned_addr learned my addr 172.21.15.67:0/940798713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 -- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb81f4c60 msgr2=0x7f0cb815aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6000640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb81f4c60 0x7f0cb815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb7400640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 0x7f0cb80ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb81f4c60 0x7f0cb815aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 -- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 msgr2=0x7f0cb80ac800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 0x7f0cb80ac800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 -- 172.21.15.67:0/940798713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb80ab480 con 0x7f0cb8151fe0 2024-04-17T08:50:10.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb7400640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 0x7f0cb80ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:10.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.160+0000 7f0cb6a00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8011600 tx=0x7f0ca8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.161+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800f8c0 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.161+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca800fa60 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.161+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800fe10 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.161+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb80ab760 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.161+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb80abd20 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.162+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb808c750 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.163+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 127) ==== 90878+0+0 (secure 0 0 0) 0x7f0ca8015460 con 0x7f0cb8151fe0 2024-04-17T08:50:10.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.164+0000 7f0caea00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 0x7f0c9406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.164+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7f0ca808c2f0 con 0x7f0cb8151fe0 2024-04-17T08:50:10.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.165+0000 7f0cb6000640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 0x7f0c9406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.166+0000 7f0cb6000640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 0x7f0c9406ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0034000 tx=0x7f0ca0009030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.169+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0cb808c750 con 0x7f0cb8151fe0 2024-04-17T08:50:10.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.376+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cb8068a10 con 0x7f0cb8151fe0 2024-04-17T08:50:10.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.378+0000 7f0caea00640 1 -- 172.21.15.67:0/940798713 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) ==== 74+0+17799 (secure 0 0 0) 0x7f0ca8019030 con 0x7f0cb8151fe0 2024-04-17T08:50:10.378 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:10.378 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":217,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:08.382660+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":216,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.7","mappings":[{"from":0,"to":2}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1783845869":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/2158603650":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/1545043147":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/3057701238":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6832/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/2473833231":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/3145453289":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/910136246":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/4251121774":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2270739884":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6832/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2605475186":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2381197913":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/713566291":"2024-04-18T08:47:58.503956+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 msgr2=0x7f0c9406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 0x7f0c9406ec30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0034000 tx=0x7f0ca0009030 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 msgr2=0x7f0cb81f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8011600 tx=0x7f0ca8011ac0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 shutdown_connections 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c9406c770 0x7f0c9406ec30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cb815af40 0x7f0cb80ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cb81f4c60 0x7f0cb815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 --2- 172.21.15.67:0/940798713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb8151fe0 0x7f0cb81f4720 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 >> 172.21.15.67:0/940798713 conn(0x7f0cb80d15f0 msgr2=0x7f0cb8160d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 shutdown_connections 2024-04-17T08:50:10.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.381+0000 7f0cb7e00640 1 -- 172.21.15.67:0/940798713 wait complete. 2024-04-17T08:50:10.401 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:50:10.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:10.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.615+0000 7fed5d400640 1 Processor -- start 2024-04-17T08:50:10.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.615+0000 7fed5d400640 1 -- start start 2024-04-17T08:50:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.616+0000 7fed5d400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fed5806b8c0 0x7fed580cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.616+0000 7fed5d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed580cd4a0 con 0x7fed58141ed0 2024-04-17T08:50:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed5d400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed581607f0 con 0x7fed58160280 2024-04-17T08:50:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed5d400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fed5815ce30 con 0x7fed5806b8c0 2024-04-17T08:50:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed5ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed58141ed0 0x7fed5815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46972/0 (socket says 172.21.15.67:46972) 2024-04-17T08:50:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed5ca00640 1 -- 172.21.15.67:0/1306572042 learned_addr learned my addr 172.21.15.67:0/1306572042 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838752347 0 0) 0x7fed580cd4a0 con 0x7fed58141ed0 2024-04-17T08:50:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed57e00640 1 --2- 172.21.15.67:0/1306572042 >> v2:172.21.15.187:3300/0 conn(0x7fed5806b8c0 0x7fed580cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.617+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed3c003520 con 0x7fed58141ed0 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3887894627 0 0) 0x7fed3c003520 con 0x7fed58141ed0 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 >> v2:172.21.15.187:3300/0 conn(0x7fed5806b8c0 msgr2=0x7fed580cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 --2- 172.21.15.67:0/1306572042 >> v2:172.21.15.187:3300/0 conn(0x7fed5806b8c0 0x7fed580cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 >> v1:172.21.15.187:6789/0 conn(0x7fed58160280 legacy=0x7fed5806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed58160ad0 con 0x7fed58141ed0 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fed500019f0 con 0x7fed58141ed0 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 >> v1:172.21.15.67:6789/0 conn(0x7fed58141ed0 legacy=0x7fed5815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed3c003a20 0x7fed3c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c004800 0x7fed3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 0x7fed3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fed5815ce30 con 0x7fed3c0051c0 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c00c0f0 con 0x7fed3c003a20 2024-04-17T08:50:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fed3c00c2f0 con 0x7fed3c004800 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.618+0000 7fed5ca00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 0x7fed3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed57e00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed3c003a20 0x7fed3c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed57400640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c004800 0x7fed3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed5ca00640 1 -- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c004800 msgr2=0x7fed3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed5ca00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c004800 0x7fed3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed5ca00640 1 -- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed3c003a20 msgr2=0x7fed3c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed5ca00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed3c003a20 0x7fed3c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed5ca00640 1 -- 172.21.15.67:0/1306572042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c003520 con 0x7fed3c0051c0 2024-04-17T08:50:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.619+0000 7fed57e00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed3c003a20 0x7fed3c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.620+0000 7fed5ca00640 1 -- 172.21.15.67:0/1306572042 reap_dead start 2024-04-17T08:50:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.620+0000 7fed5ca00640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 0x7fed3c00b9c0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fed5000a860 tx=0x7fed3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13257ba0b71974d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.620+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5000a480 con 0x7fed3c0051c0 2024-04-17T08:50:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.620+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed50034630 con 0x7fed3c0051c0 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.620+0000 7fed56a00640 1 -- 172.21.15.67:0/1306572042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50034a20 con 0x7fed3c0051c0 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 -- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 msgr2=0x7fed3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 0x7fed3c00b9c0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fed5000a860 tx=0x7fed3c005830 comp rx=0 tx=0).stop 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 -- 172.21.15.67:0/1306572042 shutdown_connections 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 --2- 172.21.15.67:0/1306572042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed3c0051c0 0x7fed3c00b9c0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 -- 172.21.15.67:0/1306572042 >> 172.21.15.67:0/1306572042 conn(0x7fed58067d10 msgr2=0x7fed580b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 -- 172.21.15.67:0/1306572042 shutdown_connections 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.621+0000 7fed5d400640 1 -- 172.21.15.67:0/1306572042 wait complete. 2024-04-17T08:50:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.623+0000 7fed5d400640 1 Processor -- start 2024-04-17T08:50:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.623+0000 7fed5d400640 1 -- start start 2024-04-17T08:50:10.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 0x7fed58155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed58133210 0x7fed5806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fed5813f9b0 con 0x7fed58145d60 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fed58161090 con 0x7fed58133210 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed5d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fed5813fb30 con 0x7fed3c0051c0 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59708/0 (socket says 172.21.15.67:59708) 2024-04-17T08:50:10.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.624+0000 7fed57400640 1 -- 172.21.15.67:0/3646459494 learned_addr learned my addr 172.21.15.67:0/3646459494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57e00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 0x7fed58155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 -- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 msgr2=0x7fed58155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed5ca00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed58133210 0x7fed5806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 0x7fed58155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 -- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed58133210 msgr2=0x7fed5806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed58133210 0x7fed5806ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 -- 172.21.15.67:0/3646459494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed58144cc0 con 0x7fed58145d60 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57e00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 0x7fed58155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed57400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fed4400e480 tx=0x7fed4400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed440052e0 con 0x7fed58145d60 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed44018070 con 0x7fed58145d60 2024-04-17T08:50:10.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed58144fa0 con 0x7fed58145d60 2024-04-17T08:50:10.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.625+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed44003830 con 0x7fed58145d60 2024-04-17T08:50:10.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.626+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed58145560 con 0x7fed58145d60 2024-04-17T08:50:10.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.627+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed58068a10 con 0x7fed58145d60 2024-04-17T08:50:10.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.628+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 128) ==== 91086+0+0 (secure 0 0 0) 0x7fed440039d0 con 0x7fed58145d60 2024-04-17T08:50:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.629+0000 7fed4fe00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 0x7fed3806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.629+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7fed4408bc60 con 0x7fed58145d60 2024-04-17T08:50:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.629+0000 7fed57e00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 0x7fed3806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.630+0000 7fed57e00640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 0x7fed3806ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed48009990 tx=0x7fed48008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.633+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fed44059d10 con 0x7fed58145d60 2024-04-17T08:50:10.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.881+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed5813ff60 con 0x7fed58145d60 2024-04-17T08:50:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.883+0000 7fed4fe00640 1 -- 172.21.15.67:0/3646459494 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7fed44059610 con 0x7fed58145d60 2024-04-17T08:50:10.883 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:10.883 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:50:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 msgr2=0x7fed3806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 0x7fed3806ee70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed48009990 tx=0x7fed48008040 comp rx=0 tx=0).stop 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 msgr2=0x7fed58142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fed4400e480 tx=0x7fed4400e940 comp rx=0 tx=0).stop 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 shutdown_connections 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fed3806c9b0 0x7fed3806ee70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed58133210 0x7fed5806ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed58145d60 0x7fed58142fc0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 --2- 172.21.15.67:0/3646459494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fed3c0051c0 0x7fed58155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.885+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 >> 172.21.15.67:0/3646459494 conn(0x7fed58067d10 msgr2=0x7fed58074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.886+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 shutdown_connections 2024-04-17T08:50:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:10.886+0000 7fed5d400640 1 -- 172.21.15.67:0/3646459494 wait complete. 2024-04-17T08:50:10.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.127+0000 7f808a400640 1 Processor -- start 2024-04-17T08:50:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.128+0000 7f808a400640 1 -- start start 2024-04-17T08:50:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f808a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80840cfae0 0x7f8084169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f808a400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80840d2d30 con 0x7f80840b6d00 2024-04-17T08:50:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f808a400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80840d3010 con 0x7f808415ec60 2024-04-17T08:50:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f808a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f80840d49c0 con 0x7f80840cfae0 2024-04-17T08:50:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f8083e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80840b6d00 0x7f80840cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46980/0 (socket says 172.21.15.67:46980) 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f8083e00640 1 -- 172.21.15.67:0/4212766317 learned_addr learned my addr 172.21.15.67:0/4212766317 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.129+0000 7f8089a00640 1 --2- 172.21.15.67:0/4212766317 >> v2:172.21.15.67:3300/0 conn(0x7f80840cfae0 0x7f8084169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.130+0000 7f8083400640 1 -- 172.21.15.67:0/4212766317 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308960603 0 0) 0x7f80840d2d30 con 0x7f80840b6d00 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.130+0000 7f8083400640 1 -- 172.21.15.67:0/4212766317 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f806c003520 con 0x7f80840b6d00 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.130+0000 7f8089a00640 1 -- 172.21.15.67:0/4212766317 >> v1:172.21.15.187:6789/0 conn(0x7f808415ec60 legacy=0x7f80840b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.130+0000 7f8089a00640 1 -- 172.21.15.67:0/4212766317 >> v1:172.21.15.67:6789/0 conn(0x7f80840b6d00 legacy=0x7f80840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.130+0000 7f8089a00640 1 -- 172.21.15.67:0/4212766317 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80840d32f0 con 0x7f80840cfae0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.131+0000 7f8089a00640 1 --2- 172.21.15.67:0/4212766317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084169810 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f807c009d90 tx=0x7f807c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295522d4836f7c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.131+0000 7f8083400640 1 -- 172.21.15.67:0/4212766317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c039070 con 0x7f80840cfae0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.131+0000 7f8083400640 1 -- 172.21.15.67:0/4212766317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807c02cd00 con 0x7f80840cfae0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.131+0000 7f8083400640 1 -- 172.21.15.67:0/4212766317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c034660 con 0x7f80840cfae0 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 -- 172.21.15.67:0/4212766317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 msgr2=0x7f8084169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 --2- 172.21.15.67:0/4212766317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084169810 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f807c009d90 tx=0x7f807c02bf50 comp rx=0 tx=0).stop 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 -- 172.21.15.67:0/4212766317 shutdown_connections 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 --2- 172.21.15.67:0/4212766317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084169810 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 -- 172.21.15.67:0/4212766317 >> 172.21.15.67:0/4212766317 conn(0x7f80840d15f0 msgr2=0x7f80840d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 -- 172.21.15.67:0/4212766317 shutdown_connections 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.132+0000 7f808a400640 1 -- 172.21.15.67:0/4212766317 wait complete. 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.134+0000 7f808a400640 1 Processor -- start 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.134+0000 7f808a400640 1 -- start start 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 0x7f80840cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f808415ec60 0x7f8084159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80840b8b80 con 0x7f80840cfae0 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f80840d3660 con 0x7f808415ec60 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f808a400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f80840d00c0 con 0x7f80840b6d00 2024-04-17T08:50:11.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f8083e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f8083e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59728/0 (socket says 172.21.15.67:59728) 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.135+0000 7f8083e00640 1 -- 172.21.15.67:0/1526193280 learned_addr learned my addr 172.21.15.67:0/1526193280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8089000640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 0x7f80840cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 -- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 msgr2=0x7f80840cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8089a00640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f808415ec60 0x7f8084159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 0x7f80840cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 -- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f808415ec60 msgr2=0x7f8084159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f808415ec60 0x7f8084159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 -- 172.21.15.67:0/1526193280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8084158a70 con 0x7f80840cfae0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8089000640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 0x7f80840cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8083e00640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f807000c500 tx=0x7f807000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070017070 con 0x7f80840cfae0 2024-04-17T08:50:11.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070005ce0 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8084158d50 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.136+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070007330 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.137+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8084159310 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.138+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8084068a10 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.139+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 128) ==== 91086+0+0 (secure 0 0 0) 0x7f807001a070 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.140+0000 7f8080c00640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 0x7f806406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.140+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7f807008b3e0 con 0x7f80840cfae0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.140+0000 7f8089000640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 0x7f806406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.141+0000 7f8089000640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 0x7f806406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8078000f60 tx=0x7f8078009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.144+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8070059550 con 0x7f80840cfae0 2024-04-17T08:50:11.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.389+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80840cddb0 con 0x7f80840cfae0 2024-04-17T08:50:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.390+0000 7f8080c00640 1 -- 172.21.15.67:0/1526193280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5217 (secure 0 0 0) 0x7f8070058e50 con 0x7f80840cfae0 2024-04-17T08:50:11.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:11.390 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":102,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:47:43.271811+0000","modified":"2024-04-17T08:47:46.355432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10518":{"gid":10518,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2414893402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2414893402},{"type":"v1","addr":"172.21.15.67:6835","nonce":2414893402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10518,"qdb_cluster":[10518]},"id":12}]} 2024-04-17T08:50:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 102 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 msgr2=0x7f806406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 0x7f806406ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8078000f60 tx=0x7f8078009040 comp rx=0 tx=0).stop 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 msgr2=0x7f8084153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f807000c500 tx=0x7f807000c9c0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 shutdown_connections 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f806406c980 0x7f806406ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f808415ec60 0x7f8084159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80840cfae0 0x7f8084153390 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 --2- 172.21.15.67:0/1526193280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80840b6d00 0x7f80840cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.393+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 >> 172.21.15.67:0/1526193280 conn(0x7f80840d15f0 msgr2=0x7f80840d53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.394+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 shutdown_connections 2024-04-17T08:50:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.394+0000 7f808a400640 1 -- 172.21.15.67:0/1526193280 wait complete. 2024-04-17T08:50:11.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:11.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.629+0000 7fa2d2200640 1 Processor -- start 2024-04-17T08:50:11.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.629+0000 7fa2d2200640 1 -- start start 2024-04-17T08:50:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.630+0000 7fa2d2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2cc166440 0x7fa2cc0ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.630+0000 7fa2d2200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2cc072420 0x7fa2cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.630+0000 7fa2d2200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2cc072b20 con 0x7fa2cc0c6f90 2024-04-17T08:50:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.630+0000 7fa2d2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa2cc0ce960 con 0x7fa2cc166440 2024-04-17T08:50:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.630+0000 7fa2d2200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa2cc0c5eb0 con 0x7fa2cc072420 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2c3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2cc0c6f90 0x7fa2cc071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46988/0 (socket says 172.21.15.67:46988) 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2c3e00640 1 -- 172.21.15.67:0/523068012 learned_addr learned my addr 172.21.15.67:0/523068012 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cb400640 1 --2- 172.21.15.67:0/523068012 >> v2:172.21.15.67:3300/0 conn(0x7fa2cc166440 0x7fa2cc0ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cbe00640 1 --2- 172.21.15.67:0/523068012 >> v2:172.21.15.187:3300/0 conn(0x7fa2cc072420 0x7fa2cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cb400640 1 -- 172.21.15.67:0/523068012 >> v2:172.21.15.187:3300/0 conn(0x7fa2cc072420 msgr2=0x7fa2cc06dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cb400640 1 --2- 172.21.15.67:0/523068012 >> v2:172.21.15.187:3300/0 conn(0x7fa2cc072420 0x7fa2cc06dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cb400640 1 -- 172.21.15.67:0/523068012 >> v1:172.21.15.67:6789/0 conn(0x7fa2cc0c6f90 legacy=0x7fa2cc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.631+0000 7fa2cb400640 1 -- 172.21.15.67:0/523068012 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc0c6190 con 0x7fa2cc166440 2024-04-17T08:50:11.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.633+0000 7fa2cb400640 1 --2- 172.21.15.67:0/523068012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc166440 0x7fa2cc0ce050 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0cdf10 tx=0x7fa2bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4311d51d84dddd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.633+0000 7fa2caa00640 1 -- 172.21.15.67:0/523068012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0315a0 con 0x7fa2cc166440 2024-04-17T08:50:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.633+0000 7fa2caa00640 1 -- 172.21.15.67:0/523068012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2bc037070 con 0x7fa2cc166440 2024-04-17T08:50:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.633+0000 7fa2caa00640 1 -- 172.21.15.67:0/523068012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc031960 con 0x7fa2cc166440 2024-04-17T08:50:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 -- 172.21.15.67:0/523068012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc166440 msgr2=0x7fa2cc0ce050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 --2- 172.21.15.67:0/523068012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc166440 0x7fa2cc0ce050 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0cdf10 tx=0x7fa2bc02b840 comp rx=0 tx=0).stop 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 -- 172.21.15.67:0/523068012 shutdown_connections 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 --2- 172.21.15.67:0/523068012 >> v2:172.21.15.187:3300/0 conn(0x7fa2cc072420 0x7fa2cc06dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 --2- 172.21.15.67:0/523068012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc166440 0x7fa2cc0ce050 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 -- 172.21.15.67:0/523068012 >> 172.21.15.67:0/523068012 conn(0x7fa2cc06bf50 msgr2=0x7fa2cc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 -- 172.21.15.67:0/523068012 shutdown_connections 2024-04-17T08:50:11.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.634+0000 7fa2d2200640 1 -- 172.21.15.67:0/523068012 wait complete. 2024-04-17T08:50:11.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.636+0000 7fa2d2200640 1 Processor -- start 2024-04-17T08:50:11.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.636+0000 7fa2d2200640 1 -- start start 2024-04-17T08:50:11.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2cc0c6f90 0x7fa2cc0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2cc166440 0x7fa2cc0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc0cf0d0 con 0x7fa2cc072420 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc06e290 con 0x7fa2cc166440 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2d2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2cc14fc60 con 0x7fa2cc0c6f90 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59746/0 (socket says 172.21.15.67:59746) 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cb400640 1 -- 172.21.15.67:0/3157573069 learned_addr learned my addr 172.21.15.67:0/3157573069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2c3e00640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2cc0c6f90 0x7fa2cc0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cbe00640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2cc166440 0x7fa2cc0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cb400640 1 -- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2cc0c6f90 msgr2=0x7fa2cc0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.637+0000 7fa2cb400640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2cc0c6f90 0x7fa2cc0b56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2cb400640 1 -- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2cc166440 msgr2=0x7fa2cc0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2cb400640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2cc166440 0x7fa2cc0b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2cb400640 1 -- 172.21.15.67:0/3157573069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc1619c0 con 0x7fa2cc072420 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2cb400640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc032630 con 0x7fa2cc072420 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2bc0327f0 con 0x7fa2cc072420 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2cc161ca0 con 0x7fa2cc072420 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc032be0 con 0x7fa2cc072420 2024-04-17T08:50:11.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.638+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2cc162260 con 0x7fa2cc072420 2024-04-17T08:50:11.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.639+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2cc068b90 con 0x7fa2cc072420 2024-04-17T08:50:11.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.641+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 128) ==== 91086+0+0 (secure 0 0 0) 0x7fa2bc04c020 con 0x7fa2cc072420 2024-04-17T08:50:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.642+0000 7fa2c3400640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 0x7fa2a806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.642+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7fa2bc02f020 con 0x7fa2cc072420 2024-04-17T08:50:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.642+0000 7fa2c3e00640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 0x7fa2a806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:11.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.643+0000 7fa2c3e00640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 0x7fa2a806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0b2a80 tx=0x7fa2b4009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:11.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.646+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa2bc040070 con 0x7fa2cc072420 2024-04-17T08:50:11.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.853+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2cc0ce030 con 0x7fa2cc072420 2024-04-17T08:50:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.855+0000 7fa2c3400640 1 -- 172.21.15.67:0/3157573069 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) ==== 74+0+17799 (secure 0 0 0) 0x7fa2bc039310 con 0x7fa2cc072420 2024-04-17T08:50:11.855 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:11.855 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":217,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:08.382660+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:47:41.219096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-04-17T08:47:42.272552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":216,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.7","mappings":[{"from":0,"to":2}]},{"pgid":"25.12","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1783845869":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/2158603650":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/1545043147":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/3057701238":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6832/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/2473833231":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/3145453289":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/910136246":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/4251121774":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2270739884":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6832/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2605475186":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2381197913":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/713566291":"2024-04-18T08:47:58.503956+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 msgr2=0x7fa2a806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 0x7fa2a806ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0b2a80 tx=0x7fa2b4009040 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 msgr2=0x7fa2cc14f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 shutdown_connections 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa2a806c980 0x7fa2a806ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2cc166440 0x7fa2cc0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2cc0c6f90 0x7fa2cc0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 --2- 172.21.15.67:0/3157573069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc072420 0x7fa2cc14f720 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 >> 172.21.15.67:0/3157573069 conn(0x7fa2cc06bf50 msgr2=0x7fa2cc0cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 shutdown_connections 2024-04-17T08:50:11.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:11.858+0000 7fa2d2200640 1 -- 172.21.15.67:0/3157573069 wait complete. 2024-04-17T08:50:11.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:50:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.107+0000 7fdc1d000640 1 Processor -- start 2024-04-17T08:50:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.107+0000 7fdc1d000640 1 -- start start 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.109+0000 7fdc1d000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc1815b860 0x7fdc180d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.109+0000 7fdc1d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdc180d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc1d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc18071f60 con 0x7fdc180ce4d0 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc1d000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc18071d10 con 0x7fdc1815b860 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc1d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc18072240 con 0x7fdc180b4f80 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc15400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdc180ce4d0 0x7fdc180ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46994/0 (socket says 172.21.15.67:46994) 2024-04-17T08:50:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc15400640 1 -- 172.21.15.67:0/1973406906 learned_addr learned my addr 172.21.15.67:0/1973406906 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc16800640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdc180d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1973406906 >> v2:172.21.15.187:3300/0 conn(0x7fdc1815b860 0x7fdc180d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327715303 0 0) 0x7fdc18071f60 con 0x7fdc180ce4d0 2024-04-17T08:50:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.110+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbfc003520 con 0x7fdc180ce4d0 2024-04-17T08:50:12.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3000843948 0 0) 0x7fdbfc003520 con 0x7fdc180ce4d0 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 msgr2=0x7fdc180d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdc180d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 >> v2:172.21.15.187:3300/0 conn(0x7fdc1815b860 msgr2=0x7fdc180d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 --2- 172.21.15.67:0/1973406906 >> v2:172.21.15.187:3300/0 conn(0x7fdc1815b860 0x7fdc180d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc18072520 con 0x7fdc180ce4d0 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fdc04001670 con 0x7fdc180ce4d0 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 >> v1:172.21.15.67:6789/0 conn(0x7fdc180ce4d0 legacy=0x7fdc180ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0039b0 0x7fdbfc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc004800 0x7fdbfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdbfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc18071d10 con 0x7fdbfc004800 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc18072240 con 0x7fdbfc0039b0 2024-04-17T08:50:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdbfc00ba10 con 0x7fdc180b4f80 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.111+0000 7fdc15400640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc004800 0x7fdbfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0039b0 0x7fdbfc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc16800640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdbfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15400640 1 -- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 msgr2=0x7fdbfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15400640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdbfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15400640 1 -- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0039b0 msgr2=0x7fdbfc003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15400640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0039b0 0x7fdbfc003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc15400640 1 -- 172.21.15.67:0/1973406906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc003520 con 0x7fdbfc004800 2024-04-17T08:50:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.112+0000 7fdc16800640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc180b4f80 0x7fdbfc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.113+0000 7fdc15400640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc004800 0x7fdbfc004c80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fdc04007cf0 tx=0x7fdbfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed28927050aeb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.113+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007d30 con 0x7fdbfc004800 2024-04-17T08:50:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.113+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc04032070 con 0x7fdbfc004800 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc14a00640 1 -- 172.21.15.67:0/1973406906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc040058d0 con 0x7fdbfc004800 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 -- 172.21.15.67:0/1973406906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc004800 msgr2=0x7fdbfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1973406906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc004800 0x7fdbfc004c80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fdc04007cf0 tx=0x7fdbfc005830 comp rx=0 tx=0).stop 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc16800640 1 -- 172.21.15.67:0/1973406906 reap_dead start 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 -- 172.21.15.67:0/1973406906 shutdown_connections 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 -- 172.21.15.67:0/1973406906 >> 172.21.15.67:0/1973406906 conn(0x7fdc18067ee0 msgr2=0x7fdc1806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 -- 172.21.15.67:0/1973406906 shutdown_connections 2024-04-17T08:50:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.114+0000 7fdc1d000640 1 -- 172.21.15.67:0/1973406906 wait complete. 2024-04-17T08:50:12.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.116+0000 7fdc1d000640 1 Processor -- start 2024-04-17T08:50:12.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.116+0000 7fdc1d000640 1 -- start start 2024-04-17T08:50:12.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 0x7fdc180c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 0x7fdc180c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1815bc40 con 0x7fdc1815e470 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc18072ae0 con 0x7fdc1815dff0 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.117+0000 7fdc1d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1806f380 con 0x7fdbfc004800 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59770/0 (socket says 172.21.15.67:59770) 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 -- 172.21.15.67:0/1638997508 learned_addr learned my addr 172.21.15.67:0/1638997508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 0x7fdc180c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc15400640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 0x7fdc180c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 -- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 msgr2=0x7fdc180c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 0x7fdc180c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 -- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 msgr2=0x7fdc180c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 0x7fdc180c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc16800640 1 -- 172.21.15.67:0/1638997508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc180c5940 con 0x7fdc1815e470 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 0x7fdc180c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.118+0000 7fdc15400640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 0x7fdc180c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc16800640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009fd0 tx=0x7fdc1000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1001c070 con 0x7fdc1815e470 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc10017400 con 0x7fdc1815e470 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100177d0 con 0x7fdc1815e470 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc180c5c20 con 0x7fdc1815e470 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.119+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc180c61e0 con 0x7fdc1815e470 2024-04-17T08:50:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.120+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc18068ca0 con 0x7fdc1815e470 2024-04-17T08:50:12.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.122+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 128) ==== 91086+0+0 (secure 0 0 0) 0x7fdc1001f070 con 0x7fdc1815e470 2024-04-17T08:50:12.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.123+0000 7fdc0e000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 0x7fdbf806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.123+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6511+0+0 (secure 0 0 0) 0x7fdc1008c0c0 con 0x7fdc1815e470 2024-04-17T08:50:12.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.124+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 0x7fdbf806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.125+0000 7fdc15e00640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 0x7fdbf806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009a00 tx=0x7fdc08008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.128+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdc10012070 con 0x7fdc1815e470 2024-04-17T08:50:12.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.373+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdc1806ef60 con 0x7fdc1815e470 2024-04-17T08:50:12.373 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:50:12.374+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:50:12.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.378+0000 7fdc0e000640 1 -- 172.21.15.67:0/1638997508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v103) ==== 161+0+0 (secure 0 0 0) 0x7fdc10059fd0 con 0x7fdc1815e470 2024-04-17T08:50:12.378 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 msgr2=0x7fdbf806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 0x7fdbf806ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009a00 tx=0x7fdc08008040 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 msgr2=0x7fdc180c6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009fd0 tx=0x7fdc1000f4a0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 shutdown_connections 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fdbf806c9b0 0x7fdbf806ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1815e470 0x7fdc180c6f20 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1815dff0 0x7fdc180c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 --2- 172.21.15.67:0/1638997508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc004800 0x7fdc180c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 >> 172.21.15.67:0/1638997508 conn(0x7fdc18067ee0 msgr2=0x7fdc180d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 shutdown_connections 2024-04-17T08:50:12.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.382+0000 7fdc1d000640 1 -- 172.21.15.67:0/1638997508 wait complete. 2024-04-17T08:50:12.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:50:12.450 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:50:12.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.597+0000 7f020de00640 1 Processor -- start 2024-04-17T08:50:12.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.598+0000 7f020de00640 1 -- start start 2024-04-17T08:50:12.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f020de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0208071ea0 0x7f02080ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f020de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080cea70 con 0x7f0208072d80 2024-04-17T08:50:12.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f020de00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02080ced50 con 0x7f0208145ac0 2024-04-17T08:50:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f020de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02080cf030 con 0x7f0208071ea0 2024-04-17T08:50:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f01ffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0208072d80 0x7f02080cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46998/0 (socket says 172.21.15.67:46998) 2024-04-17T08:50:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f01ffe00640 1 -- 172.21.15.67:0/2758219128 learned_addr learned my addr 172.21.15.67:0/2758219128 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.599+0000 7f0207400640 1 --2- 172.21.15.67:0/2758219128 >> v2:172.21.15.67:3300/0 conn(0x7f0208071ea0 0x7f02080ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.600+0000 7f0206000640 1 -- 172.21.15.67:0/2758219128 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664760587 0 0) 0x7f02080cea70 con 0x7f0208072d80 2024-04-17T08:50:12.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.600+0000 7f0206000640 1 -- 172.21.15.67:0/2758219128 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01ec003520 con 0x7f0208072d80 2024-04-17T08:50:12.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.600+0000 7f0207400640 1 -- 172.21.15.67:0/2758219128 >> v1:172.21.15.187:6789/0 conn(0x7f0208145ac0 legacy=0x7f0208072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:12.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.600+0000 7f0207400640 1 -- 172.21.15.67:0/2758219128 >> v1:172.21.15.67:6789/0 conn(0x7f0208072d80 legacy=0x7f02080cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.600+0000 7f0207400640 1 -- 172.21.15.67:0/2758219128 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02080cf310 con 0x7f0208071ea0 2024-04-17T08:50:12.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.601+0000 7f0207400640 1 --2- 172.21.15.67:0/2758219128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208071ea0 0x7f02080ce3b0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0200009d90 tx=0x7f020002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d0e953d7cbd06c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.601+0000 7f0206000640 1 -- 172.21.15.67:0/2758219128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200039070 con 0x7f0208071ea0 2024-04-17T08:50:12.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f0206000640 1 -- 172.21.15.67:0/2758219128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020002cc60 con 0x7f0208071ea0 2024-04-17T08:50:12.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f0206000640 1 -- 172.21.15.67:0/2758219128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200034660 con 0x7f0208071ea0 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 -- 172.21.15.67:0/2758219128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208071ea0 msgr2=0x7f02080ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 --2- 172.21.15.67:0/2758219128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208071ea0 0x7f02080ce3b0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0200009d90 tx=0x7f020002beb0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 -- 172.21.15.67:0/2758219128 shutdown_connections 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 --2- 172.21.15.67:0/2758219128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208071ea0 0x7f02080ce3b0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 -- 172.21.15.67:0/2758219128 >> 172.21.15.67:0/2758219128 conn(0x7f0208067ee0 msgr2=0x7f020806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 -- 172.21.15.67:0/2758219128 shutdown_connections 2024-04-17T08:50:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.602+0000 7f020de00640 1 -- 172.21.15.67:0/2758219128 wait complete. 2024-04-17T08:50:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.604+0000 7f020de00640 1 Processor -- start 2024-04-17T08:50:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.605+0000 7f020de00640 1 -- start start 2024-04-17T08:50:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0208071ea0 0x7f020813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0208145ac0 0x7f02081440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02080d0220 con 0x7f0208072d80 2024-04-17T08:50:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0208145ea0 con 0x7f0208071ea0 2024-04-17T08:50:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f020de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02080d03a0 con 0x7f0208145ac0 2024-04-17T08:50:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f01ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f01ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59798/0 (socket says 172.21.15.67:59798) 2024-04-17T08:50:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f01ffe00640 1 -- 172.21.15.67:0/3404373727 learned_addr learned my addr 172.21.15.67:0/3404373727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f0206a00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0208071ea0 0x7f020813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.606+0000 7f0207400640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0208145ac0 0x7f02081440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 -- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0208145ac0 msgr2=0x7f02081440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0208145ac0 0x7f02081440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 -- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0208071ea0 msgr2=0x7f020813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0208071ea0 0x7f020813cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 -- 172.21.15.67:0/3404373727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020813d3e0 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ffe00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c430 tx=0x7f01f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0017070 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f0005ce0 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f00072f0 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.607+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020806de40 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.608+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020806e400 con 0x7f0208072d80 2024-04-17T08:50:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.609+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02080cc6c0 con 0x7f0208072d80 2024-04-17T08:50:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.610+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 128) ==== 91086+0+0 (secure 0 0 0) 0x7f01f001a070 con 0x7f0208072d80 2024-04-17T08:50:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.611+0000 7f01ff400640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 0x7f01e406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.611+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 6597+0+0 (secure 0 0 0) 0x7f01f008b420 con 0x7f0208072d80 2024-04-17T08:50:12.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.612+0000 7f0206a00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 0x7f01e406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:12.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.613+0000 7f0206a00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 0x7f01e406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:12.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.616+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f020813d3e0 con 0x7f0208072d80 2024-04-17T08:50:12.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:12.863+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f02080685d0 con 0x7f0208072d80 2024-04-17T08:50:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.403+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f01f00553e0 con 0x7f0208072d80 2024-04-17T08:50:13.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.429+0000 7f01ff400640 1 -- 172.21.15.67:0/3404373727 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v104) ==== 104+0+0 (secure 0 0 0) 0x7f01f0059470 con 0x7f0208072d80 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 msgr2=0x7f01e406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 0x7f01e406ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01f8006fd0 tx=0x7f01f8008040 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 msgr2=0x7f02080cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c430 tx=0x7f01f000c8f0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 shutdown_connections 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f01e406c980 0x7f01e406ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0208145ac0 0x7f02081440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0208072d80 0x7f02080cb630 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 --2- 172.21.15.67:0/3404373727 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0208071ea0 0x7f020813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 >> 172.21.15.67:0/3404373727 conn(0x7f0208067ee0 msgr2=0x7f020806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.434+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 shutdown_connections 2024-04-17T08:50:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.435+0000 7f020de00640 1 -- 172.21.15.67:0/3404373727 wait complete. 2024-04-17T08:50:13.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:50:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.724+0000 7f0aaa200640 1 Processor -- start 2024-04-17T08:50:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.724+0000 7f0aaa200640 1 -- start start 2024-04-17T08:50:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aaa200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 0x7f0aa40b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aaa200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0aa4071d20 0x7f0aa40721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aaa200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa415fb70 con 0x7f0aa415ee60 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aaa200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa4145210 con 0x7f0aa4071d20 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aaa200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa40b35c0 con 0x7f0aa4144e30 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aa3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0aa4071d20 0x7f0aa40721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aa3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0aa4071d20 0x7f0aa40721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58960/0 (socket says 172.21.15.67:58960) 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aa9800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0aa415ee60 0x7f0aa415f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45862/0 (socket says 172.21.15.67:45862) 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aa3e00640 1 -- 172.21.15.67:0/968648493 learned_addr learned my addr 172.21.15.67:0/968648493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.726+0000 7f0aa8e00640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 0x7f0aa40b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa3e00640 1 -- 172.21.15.67:0/968648493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 msgr2=0x7f0aa40b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa3e00640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 0x7f0aa40b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa3e00640 1 -- 172.21.15.67:0/968648493 >> v1:172.21.15.67:6789/0 conn(0x7f0aa415ee60 legacy=0x7f0aa415f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa3e00640 1 -- 172.21.15.67:0/968648493 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa40b38a0 con 0x7f0aa4071d20 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa8e00640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 0x7f0aa40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.727+0000 7f0aa3400640 1 -- 172.21.15.67:0/968648493 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775460371 0 0) 0x7f0aa415fb70 con 0x7f0aa415ee60 2024-04-17T08:50:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.728+0000 7f0aa3e00640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4071d20 0x7f0aa40721a0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bffc450f9659761 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aa3400640 1 -- 172.21.15.67:0/968648493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c034070 con 0x7f0aa4071d20 2024-04-17T08:50:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aa3400640 1 -- 172.21.15.67:0/968648493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c030840 con 0x7f0aa4071d20 2024-04-17T08:50:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aa3400640 1 -- 172.21.15.67:0/968648493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c030c10 con 0x7f0aa4071d20 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 -- 172.21.15.67:0/968648493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4071d20 msgr2=0x7f0aa40721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4071d20 0x7f0aa40721a0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c02b980 comp rx=0 tx=0).stop 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 -- 172.21.15.67:0/968648493 shutdown_connections 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4071d20 0x7f0aa40721a0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 --2- 172.21.15.67:0/968648493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa4144e30 0x7f0aa40b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.729+0000 7f0aaa200640 1 -- 172.21.15.67:0/968648493 >> 172.21.15.67:0/968648493 conn(0x7f0aa406b8c0 msgr2=0x7f0aa406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.730+0000 7f0aaa200640 1 -- 172.21.15.67:0/968648493 shutdown_connections 2024-04-17T08:50:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.730+0000 7f0aaa200640 1 -- 172.21.15.67:0/968648493 wait complete. 2024-04-17T08:50:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.731+0000 7f0aaa200640 1 Processor -- start 2024-04-17T08:50:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.732+0000 7f0aaa200640 1 -- start start 2024-04-17T08:50:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.732+0000 7f0aaa200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0aa4071d20 0x7f0aa4141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aaa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aaa200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa415ee60 0x7f0aa413f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aaa200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa406e0d0 con 0x7f0aa4144e30 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aaa200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa406df50 con 0x7f0aa4071d20 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aaa200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa406e250 con 0x7f0aa415ee60 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58972/0 (socket says 172.21.15.67:58972) 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 -- 172.21.15.67:0/3965436979 learned_addr learned my addr 172.21.15.67:0/3965436979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa9800640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa415ee60 0x7f0aa413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa8e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0aa4071d20 0x7f0aa4141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 -- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa415ee60 msgr2=0x7f0aa413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa415ee60 0x7f0aa413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 -- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0aa4071d20 msgr2=0x7f0aa4141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0aa4071d20 0x7f0aa4141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.733+0000 7f0aa3e00640 1 -- 172.21.15.67:0/3965436979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa41400c0 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aa3e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c00 tx=0x7f0a8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c048070 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c030640 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c034040 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa41403a0 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.734+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa4140960 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.735+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa4068a10 con 0x7f0aa4144e30 2024-04-17T08:50:13.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.736+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f0a8c04b070 con 0x7f0aa4144e30 2024-04-17T08:50:13.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.737+0000 7f0aa0c00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 0x7f0a8006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:13.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.737+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 6597+0+0 (secure 0 0 0) 0x7f0a8c0ae130 con 0x7f0aa4144e30 2024-04-17T08:50:13.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.738+0000 7f0aa8e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 0x7f0a8006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.739+0000 7f0aa8e00640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 0x7f0a8006eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a94003fd0 tx=0x7f0a940039f0 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.742+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0a8c07bfc0 con 0x7f0aa4144e30 2024-04-17T08:50:13.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:13.949+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0aa406f970 con 0x7f0aa4144e30 2024-04-17T08:50:14.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.450+0000 7f0aa0c00640 1 -- 172.21.15.67:0/3965436979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v219) ==== 181+0+0 (secure 0 0 0) 0x7f0a8c07b8c0 con 0x7f0aa4144e30 2024-04-17T08:50:14.450 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:50:14.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 msgr2=0x7f0a8006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:14.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 0x7f0a8006eed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a94003fd0 tx=0x7f0a940039f0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 msgr2=0x7f0aa40b13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c00 tx=0x7f0a8c004730 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 shutdown_connections 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0a8006ca10 0x7f0a8006eed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0aa415ee60 0x7f0aa413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aa4144e30 0x7f0aa40b13b0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 --2- 172.21.15.67:0/3965436979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0aa4071d20 0x7f0aa4141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.454+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 >> 172.21.15.67:0/3965436979 conn(0x7f0aa406b8c0 msgr2=0x7f0aa40af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:14.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.455+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 shutdown_connections 2024-04-17T08:50:14.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.455+0000 7f0aaa200640 1 -- 172.21.15.67:0/3965436979 wait complete. 2024-04-17T08:50:14.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:50:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.689+0000 7f59e2200640 1 Processor -- start 2024-04-17T08:50:14.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.690+0000 7f59e2200640 1 -- start start 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.691+0000 7f59e2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59dc15c670 0x7f59dc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.691+0000 7f59e2200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f59dc072930 0x7f59dc0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.691+0000 7f59e2200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59dc06de70 con 0x7f59dc0b3e30 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.691+0000 7f59e2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f59dc15f850 con 0x7f59dc15c670 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.691+0000 7f59e2200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f59dc15fb30 con 0x7f59dc072930 2024-04-17T08:50:14.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59dc15c670 0x7f59dc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59dc15c670 0x7f59dc15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58978/0 (socket says 172.21.15.67:58978) 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 -- 172.21.15.67:0/1160885064 learned_addr learned my addr 172.21.15.67:0/1160885064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 -- 172.21.15.67:0/1160885064 >> v2:172.21.15.187:3300/0 conn(0x7f59dc072930 msgr2=0x7f59dc0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 --2- 172.21.15.67:0/1160885064 >> v2:172.21.15.187:3300/0 conn(0x7f59dc072930 0x7f59dc0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 -- 172.21.15.67:0/1160885064 >> v1:172.21.15.187:6789/0 conn(0x7f59dc0b3e30 legacy=0x7f59dc0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:14.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.692+0000 7f59e0e00640 1 -- 172.21.15.67:0/1160885064 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc15fe10 con 0x7f59dc15c670 2024-04-17T08:50:14.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.693+0000 7f59e0e00640 1 --2- 172.21.15.67:0/1160885064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc15c670 0x7f59dc15ef90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f59dc070310 tx=0x7f59d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba2f4b1205356e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.693+0000 7f59db400640 1 -- 172.21.15.67:0/1160885064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0035020 con 0x7f59dc15c670 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59db400640 1 -- 172.21.15.67:0/1160885064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d0031870 con 0x7f59dc15c670 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59db400640 1 -- 172.21.15.67:0/1160885064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0031c40 con 0x7f59dc15c670 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 -- 172.21.15.67:0/1160885064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc15c670 msgr2=0x7f59dc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 --2- 172.21.15.67:0/1160885064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc15c670 0x7f59dc15ef90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f59dc070310 tx=0x7f59d002b910 comp rx=0 tx=0).stop 2024-04-17T08:50:14.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 -- 172.21.15.67:0/1160885064 shutdown_connections 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 --2- 172.21.15.67:0/1160885064 >> v2:172.21.15.187:3300/0 conn(0x7f59dc072930 0x7f59dc0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 --2- 172.21.15.67:0/1160885064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc15c670 0x7f59dc15ef90 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 -- 172.21.15.67:0/1160885064 >> 172.21.15.67:0/1160885064 conn(0x7f59dc06b8c0 msgr2=0x7f59dc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 -- 172.21.15.67:0/1160885064 shutdown_connections 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.694+0000 7f59e2200640 1 -- 172.21.15.67:0/1160885064 wait complete. 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.696+0000 7f59e2200640 1 Processor -- start 2024-04-17T08:50:14.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.696+0000 7f59e2200640 1 -- start start 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59dc0b3e30 0x7f59dc146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59dc15c670 0x7f59dc1f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc0cd5a0 con 0x7f59dc072930 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc15f4d0 con 0x7f59dc0b3e30 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e2200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc0cd720 con 0x7f59dc15c670 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58992/0 (socket says 172.21.15.67:58992) 2024-04-17T08:50:14.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.697+0000 7f59e0e00640 1 -- 172.21.15.67:0/1177477752 learned_addr learned my addr 172.21.15.67:0/1177477752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 -- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59dc15c670 msgr2=0x7f59dc1f7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59dc15c670 0x7f59dc1f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 -- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59dc0b3e30 msgr2=0x7f59dc146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59dc0b3e30 0x7f59dc146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 -- 172.21.15.67:0/1177477752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc147510 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e0e00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f59d0000c00 tx=0x7f59d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d003ad60 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00443f0 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00447e0 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.698+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc1f85f0 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.699+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc156160 con 0x7f59dc072930 2024-04-17T08:50:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.700+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc068a10 con 0x7f59dc072930 2024-04-17T08:50:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.701+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f59d004b060 con 0x7f59dc072930 2024-04-17T08:50:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.702+0000 7f59d8c00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 0x7f59bc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.702+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 6159+0+0 (secure 0 0 0) 0x7f59d00ae260 con 0x7f59dc072930 2024-04-17T08:50:14.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.703+0000 7f59dbe00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 0x7f59bc06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.704+0000 7f59dbe00640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 0x7f59bc06ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0b2320 tx=0x7f59c8006c90 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.706+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f59d003f070 con 0x7f59dc072930 2024-04-17T08:50:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:14.925+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f59dc068610 con 0x7f59dc072930 2024-04-17T08:50:15.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.756+0000 7f59d8c00640 1 -- 172.21.15.67:0/1177477752 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v221) ==== 169+0+0 (secure 0 0 0) 0x7f59d002f320 con 0x7f59dc072930 2024-04-17T08:50:15.756 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.761+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 msgr2=0x7f59bc06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.761+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 0x7f59bc06ef20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0b2320 tx=0x7f59c8006c90 comp rx=0 tx=0).stop 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.761+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 msgr2=0x7f59dc1540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.761+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f59d0000c00 tx=0x7f59d00046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.761+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 shutdown_connections 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f59bc06ca60 0x7f59bc06ef20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59dc15c670 0x7f59dc1f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59dc0b3e30 0x7f59dc146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 --2- 172.21.15.67:0/1177477752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59dc072930 0x7f59dc1540d0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 >> 172.21.15.67:0/1177477752 conn(0x7f59dc06b8c0 msgr2=0x7f59dc0cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 shutdown_connections 2024-04-17T08:50:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:15.762+0000 7f59e2200640 1 -- 172.21.15.67:0/1177477752 wait complete. 2024-04-17T08:50:15.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:50:16.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.005+0000 7f3772600640 1 Processor -- start 2024-04-17T08:50:16.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.006+0000 7f3772600640 1 -- start start 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.007+0000 7f3772600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f376c06bfd0 0x7f376c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.007+0000 7f3772600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c06df40 con 0x7f376c06f800 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f3772600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c06e220 con 0x7f376c15b6b0 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f3772600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f376c06ed00 con 0x7f376c06bfd0 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f3771c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f376c06f800 0x7f376c0b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45864/0 (socket says 172.21.15.67:45864) 2024-04-17T08:50:16.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f3771c00640 1 -- 172.21.15.67:0/196680283 learned_addr learned my addr 172.21.15.67:0/196680283 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f376be00640 1 --2- 172.21.15.67:0/196680283 >> v2:172.21.15.187:3300/0 conn(0x7f376c06bfd0 0x7f376c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737743213 0 0) 0x7f376c06df40 con 0x7f376c06f800 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.008+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3754005730 con 0x7f376c06f800 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2562835712 0 0) 0x7f3754005730 con 0x7f376c06f800 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 >> v2:172.21.15.187:3300/0 conn(0x7f376c06bfd0 msgr2=0x7f376c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 --2- 172.21.15.67:0/196680283 >> v2:172.21.15.187:3300/0 conn(0x7f376c06bfd0 0x7f376c0b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 >> v1:172.21.15.187:6789/0 conn(0x7f376c15b6b0 legacy=0x7f376c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c06e500 con 0x7f376c06f800 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f37640019f0 con 0x7f376c06f800 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 >> v1:172.21.15.67:6789/0 conn(0x7f376c06f800 legacy=0x7f376c0b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3754005bc0 0x7f3754005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f3754006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37540073d0 0x7f375400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06ed00 con 0x7f3754006a10 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f375400e370 con 0x7f37540073d0 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f375400e570 con 0x7f3754005bc0 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.009+0000 7f376be00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f3754006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f376be00640 1 -- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3754005bc0 msgr2=0x7f3754005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f3771200640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3754005bc0 0x7f3754005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f376be00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3754005bc0 0x7f3754005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f3771c00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37540073d0 0x7f375400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f376be00640 1 -- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37540073d0 msgr2=0x7f375400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f376be00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37540073d0 0x7f375400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f376be00640 1 -- 172.21.15.67:0/196680283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3754005730 con 0x7f3754006a10 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f3771200640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3754005bc0 0x7f3754005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f3771c00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37540073d0 0x7f375400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:16.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.010+0000 7f3771c00640 1 -- 172.21.15.67:0/196680283 reap_dead start 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.011+0000 7f376be00640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f3754006e90 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f3758009ac0 tx=0x7f3754007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba68541345311489 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.011+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758039070 con 0x7f3754006a10 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.011+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375802c6b0 con 0x7f3754006a10 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.011+0000 7f376b400640 1 -- 172.21.15.67:0/196680283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758009690 con 0x7f3754006a10 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 -- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 msgr2=0x7f3754006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f3754006e90 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f3758009ac0 tx=0x7f3754007a40 comp rx=0 tx=0).stop 2024-04-17T08:50:16.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 -- 172.21.15.67:0/196680283 shutdown_connections 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 --2- 172.21.15.67:0/196680283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f3754006e90 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 -- 172.21.15.67:0/196680283 >> 172.21.15.67:0/196680283 conn(0x7f376c0b2190 msgr2=0x7f376c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 -- 172.21.15.67:0/196680283 shutdown_connections 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.012+0000 7f3772600640 1 -- 172.21.15.67:0/196680283 wait complete. 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.014+0000 7f3772600640 1 Processor -- start 2024-04-17T08:50:16.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.014+0000 7f3772600640 1 -- start start 2024-04-17T08:50:16.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f376c1f5550 0x7f376c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f376c1f7900 0x7f376c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0cfc10 con 0x7f3754006a10 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06e870 con 0x7f376c1f5550 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.015+0000 7f3772600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f376c0cfd90 con 0x7f376c1f7900 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59000/0 (socket says 172.21.15.67:59000) 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 -- 172.21.15.67:0/3924738344 learned_addr learned my addr 172.21.15.67:0/3924738344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f376be00640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f376c1f5550 0x7f376c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 -- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f376c1f7900 msgr2=0x7f376c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f376c1f7900 0x7f376c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 -- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f376c1f5550 msgr2=0x7f376c155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f376c1f5550 0x7f376c155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.016+0000 7f3771200640 1 -- 172.21.15.67:0/3924738344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c1565e0 con 0x7f3754006a10 2024-04-17T08:50:16.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3771200640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f376000d7a0 tx=0x7f376000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760005070 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3760005210 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760007030 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c0adb20 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.017+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c156220 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.018+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c068a10 con 0x7f3754006a10 2024-04-17T08:50:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.019+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f376001a030 con 0x7f3754006a10 2024-04-17T08:50:16.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.020+0000 7f3768c00640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 0x7f374c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.020+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f376008b450 con 0x7f3754006a10 2024-04-17T08:50:16.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.021+0000 7f376be00640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 0x7f374c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.022+0000 7f376be00640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 0x7f374c06ef00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758030000 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.025+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3760021020 con 0x7f3754006a10 2024-04-17T08:50:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.242+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f376c06ee80 con 0x7f3754006a10 2024-04-17T08:50:16.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.249+0000 7f3768c00640 1 -- 172.21.15.67:0/3924738344 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) ==== 115+0+0 (secure 0 0 0) 0x7f3760059720 con 0x7f3754006a10 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 msgr2=0x7f374c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 0x7f374c06ef00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758030000 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 msgr2=0x7f376c1f5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f376000d7a0 tx=0x7f376000dc60 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 shutdown_connections 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f374c06ca40 0x7f374c06ef00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f376c1f7900 0x7f376c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f376c1f5550 0x7f376c155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 --2- 172.21.15.67:0/3924738344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3754006a10 0x7f376c1f5a40 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 >> 172.21.15.67:0/3924738344 conn(0x7f376c0b2190 msgr2=0x7f376c071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 shutdown_connections 2024-04-17T08:50:16.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.252+0000 7f3772600640 1 -- 172.21.15.67:0/3924738344 wait complete. 2024-04-17T08:50:16.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:50:16.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.477+0000 7f38e5e00640 1 Processor -- start 2024-04-17T08:50:16.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.477+0000 7f38e5e00640 1 -- start start 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.478+0000 7f38e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38e009acd0 0x7f38e0161100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.478+0000 7f38e5e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38e00ce2d0 0x7f38e009b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.478+0000 7f38e5e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e009bce0 con 0x7f38e009b0b0 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.478+0000 7f38e5e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f38e00cde40 con 0x7f38e009acd0 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.478+0000 7f38e5e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f38e00cd510 con 0x7f38e00ce2d0 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38e009acd0 0x7f38e0161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38e009acd0 0x7f38e0161100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59010/0 (socket says 172.21.15.67:59010) 2024-04-17T08:50:16.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 -- 172.21.15.67:0/3416273211 learned_addr learned my addr 172.21.15.67:0/3416273211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 -- 172.21.15.67:0/3416273211 >> v2:172.21.15.187:3300/0 conn(0x7f38e00ce2d0 msgr2=0x7f38e009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 --2- 172.21.15.67:0/3416273211 >> v2:172.21.15.187:3300/0 conn(0x7f38e00ce2d0 0x7f38e009b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 -- 172.21.15.67:0/3416273211 >> v1:172.21.15.187:6789/0 conn(0x7f38e009b0b0 legacy=0x7f38e015f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.479+0000 7f38e4a00640 1 -- 172.21.15.67:0/3416273211 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e00cd7f0 con 0x7f38e009acd0 2024-04-17T08:50:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.480+0000 7f38e4a00640 1 --2- 172.21.15.67:0/3416273211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009acd0 0x7f38e0161100 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f38e00d1cc0 tx=0x7f38c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f067954dff5bdb47 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.481+0000 7f38df400640 1 -- 172.21.15.67:0/3416273211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c802e070 con 0x7f38e009acd0 2024-04-17T08:50:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.481+0000 7f38df400640 1 -- 172.21.15.67:0/3416273211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c8037070 con 0x7f38e009acd0 2024-04-17T08:50:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.481+0000 7f38df400640 1 -- 172.21.15.67:0/3416273211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8031a00 con 0x7f38e009acd0 2024-04-17T08:50:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 -- 172.21.15.67:0/3416273211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009acd0 msgr2=0x7f38e0161100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 --2- 172.21.15.67:0/3416273211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009acd0 0x7f38e0161100 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f38e00d1cc0 tx=0x7f38c802b880 comp rx=0 tx=0).stop 2024-04-17T08:50:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 -- 172.21.15.67:0/3416273211 shutdown_connections 2024-04-17T08:50:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 --2- 172.21.15.67:0/3416273211 >> v2:172.21.15.187:3300/0 conn(0x7f38e00ce2d0 0x7f38e009b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 --2- 172.21.15.67:0/3416273211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009acd0 0x7f38e0161100 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 -- 172.21.15.67:0/3416273211 >> 172.21.15.67:0/3416273211 conn(0x7f38e00b3ff0 msgr2=0x7f38e00b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 -- 172.21.15.67:0/3416273211 shutdown_connections 2024-04-17T08:50:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.482+0000 7f38e5e00640 1 -- 172.21.15.67:0/3416273211 wait complete. 2024-04-17T08:50:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.484+0000 7f38e5e00640 1 Processor -- start 2024-04-17T08:50:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.484+0000 7f38e5e00640 1 -- start start 2024-04-17T08:50:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38e009acd0 0x7f38e006e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38e00ce2d0 0x7f38e00b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01600a0 con 0x7f38e009b0b0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00cd1c0 con 0x7f38e009acd0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38e5e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f38e0160220 con 0x7f38e00ce2d0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59014/0 (socket says 172.21.15.67:59014) 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.485+0000 7f38dfe00640 1 -- 172.21.15.67:0/4278339737 learned_addr learned my addr 172.21.15.67:0/4278339737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38e4a00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38e009acd0 0x7f38e006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38e5400640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38e00ce2d0 0x7f38e00b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 -- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38e00ce2d0 msgr2=0x7f38e00b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38e00ce2d0 0x7f38e00b18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 -- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38e009acd0 msgr2=0x7f38e006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38e009acd0 0x7f38e006e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 -- 172.21.15.67:0/4278339737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e006e9a0 con 0x7f38e009b0b0 2024-04-17T08:50:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dfe00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f38d000b450 tx=0x7f38d000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.486+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0015020 con 0x7f38e009b0b0 2024-04-17T08:50:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.487+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d0011830 con 0x7f38e009b0b0 2024-04-17T08:50:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.487+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0011c20 con 0x7f38e009b0b0 2024-04-17T08:50:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.487+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e015dab0 con 0x7f38e009b0b0 2024-04-17T08:50:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.487+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e015e070 con 0x7f38e009b0b0 2024-04-17T08:50:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.488+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e00755d0 con 0x7f38e009b0b0 2024-04-17T08:50:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.489+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f38d0018020 con 0x7f38e009b0b0 2024-04-17T08:50:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.490+0000 7f38dcc00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 0x7f38bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.490+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f38d008bcf0 con 0x7f38e009b0b0 2024-04-17T08:50:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.490+0000 7f38e4a00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 0x7f38bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.492+0000 7f38e4a00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 0x7f38bc06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38c8036000 tx=0x7f38c8008030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.495+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f38d001f020 con 0x7f38e009b0b0 2024-04-17T08:50:16.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.700+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f38e00b4c50 con 0x7f38e009b0b0 2024-04-17T08:50:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.707+0000 7f38dcc00640 1 -- 172.21.15.67:0/4278339737 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v74) ==== 104+0+0 (secure 0 0 0) 0x7f38d00172c0 con 0x7f38e009b0b0 2024-04-17T08:50:16.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 msgr2=0x7f38bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 0x7f38bc06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f38c8036000 tx=0x7f38c8008030 comp rx=0 tx=0).stop 2024-04-17T08:50:16.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 msgr2=0x7f38e00b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f38d000b450 tx=0x7f38d000b910 comp rx=0 tx=0).stop 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 shutdown_connections 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f38bc06ca10 0x7f38bc06eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38e00ce2d0 0x7f38e00b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38e009b0b0 0x7f38e00b13a0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 --2- 172.21.15.67:0/4278339737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38e009acd0 0x7f38e006e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 >> 172.21.15.67:0/4278339737 conn(0x7f38e00b3ff0 msgr2=0x7f38e00d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 shutdown_connections 2024-04-17T08:50:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.710+0000 7f38e5e00640 1 -- 172.21.15.67:0/4278339737 wait complete. 2024-04-17T08:50:16.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:50:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.948+0000 7f1470a00640 1 Processor -- start 2024-04-17T08:50:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.948+0000 7f1470a00640 1 -- start start 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.949+0000 7f1470a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f146c0d33e0 0x7f146c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.949+0000 7f1470a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0cf8e0 0x7f146c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.949+0000 7f1470a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146c0d00b0 con 0x7f146c15ecb0 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.949+0000 7f1470a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f146c0d0c30 con 0x7f146c0cf8e0 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.949+0000 7f1470a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f146c0d0390 con 0x7f146c0d33e0 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0cf8e0 0x7f146c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0cf8e0 0x7f146c0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59030/0 (socket says 172.21.15.67:59030) 2024-04-17T08:50:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 -- 172.21.15.67:0/2757956477 learned_addr learned my addr 172.21.15.67:0/2757956477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 -- 172.21.15.67:0/2757956477 >> v2:172.21.15.187:3300/0 conn(0x7f146c0d33e0 msgr2=0x7f146c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 --2- 172.21.15.67:0/2757956477 >> v2:172.21.15.187:3300/0 conn(0x7f146c0d33e0 0x7f146c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 -- 172.21.15.67:0/2757956477 >> v1:172.21.15.187:6789/0 conn(0x7f146c15ecb0 legacy=0x7f146c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.950+0000 7f146be00640 1 -- 172.21.15.67:0/2757956477 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c0d0670 con 0x7f146c0cf8e0 2024-04-17T08:50:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.952+0000 7f146be00640 1 --2- 172.21.15.67:0/2757956477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0cf8e0 0x7f146c0b3fc0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f146c0b61a0 tx=0x7f146402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a9b02df837d28a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.952+0000 7f146a000640 1 -- 172.21.15.67:0/2757956477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146402e070 con 0x7f146c0cf8e0 2024-04-17T08:50:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.952+0000 7f146a000640 1 -- 172.21.15.67:0/2757956477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1464037070 con 0x7f146c0cf8e0 2024-04-17T08:50:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.952+0000 7f146a000640 1 -- 172.21.15.67:0/2757956477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464031a00 con 0x7f146c0cf8e0 2024-04-17T08:50:16.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 -- 172.21.15.67:0/2757956477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0cf8e0 msgr2=0x7f146c0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 --2- 172.21.15.67:0/2757956477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0cf8e0 0x7f146c0b3fc0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f146c0b61a0 tx=0x7f146402b880 comp rx=0 tx=0).stop 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 -- 172.21.15.67:0/2757956477 shutdown_connections 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 --2- 172.21.15.67:0/2757956477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0cf8e0 0x7f146c0b3fc0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 --2- 172.21.15.67:0/2757956477 >> v2:172.21.15.187:3300/0 conn(0x7f146c0d33e0 0x7f146c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 -- 172.21.15.67:0/2757956477 >> 172.21.15.67:0/2757956477 conn(0x7f146c08bff0 msgr2=0x7f146c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 -- 172.21.15.67:0/2757956477 shutdown_connections 2024-04-17T08:50:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.953+0000 7f1470a00640 1 -- 172.21.15.67:0/2757956477 wait complete. 2024-04-17T08:50:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.955+0000 7f1470a00640 1 Processor -- start 2024-04-17T08:50:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.955+0000 7f1470a00640 1 -- start start 2024-04-17T08:50:16.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f146c0cf8e0 0x7f146c154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 0x7f146c0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f146c15f090 con 0x7f146c15ecb0 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f146c0cfd30 con 0x7f146c0cf8e0 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f1470a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f146c15f210 con 0x7f146c0d33e0 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f146be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f146be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59036/0 (socket says 172.21.15.67:59036) 2024-04-17T08:50:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.956+0000 7f146be00640 1 -- 172.21.15.67:0/149000824 learned_addr learned my addr 172.21.15.67:0/149000824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:16.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146b400640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f146c0cf8e0 0x7f146c154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146aa00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 0x7f146c0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 -- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 msgr2=0x7f146c0cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 0x7f146c0cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 -- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f146c0cf8e0 msgr2=0x7f146c154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f146c0cf8e0 0x7f146c154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 -- 172.21.15.67:0/149000824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c1550c0 con 0x7f146c15ecb0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146aa00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 0x7f146c0cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.957+0000 7f146be00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f14640090b0 tx=0x7f14640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.958+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464032880 con 0x7f146c15ecb0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.958+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1464032a40 con 0x7f146c15ecb0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.958+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146c0cd3b0 con 0x7f146c15ecb0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.958+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464032e30 con 0x7f146c15ecb0 2024-04-17T08:50:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.958+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146c1f4fe0 con 0x7f146c15ecb0 2024-04-17T08:50:16.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.959+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146c068a10 con 0x7f146c15ecb0 2024-04-17T08:50:16.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.960+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f146404c060 con 0x7f146c15ecb0 2024-04-17T08:50:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.961+0000 7f1463400640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 0x7f144806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.961+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f14640ad8f0 con 0x7f146c15ecb0 2024-04-17T08:50:16.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.962+0000 7f146b400640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 0x7f144806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:16.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.963+0000 7f146b400640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 0x7f144806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f145c006fd0 tx=0x7f145c008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:16.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:16.966+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f146c1550c0 con 0x7f146c15ecb0 2024-04-17T08:50:17.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.171+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f146c0b37a0 con 0x7f146c15ecb0 2024-04-17T08:50:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.177+0000 7f1463400640 1 -- 172.21.15.67:0/149000824 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v75) ==== 120+0+0 (secure 0 0 0) 0x7f1464040070 con 0x7f146c15ecb0 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 msgr2=0x7f144806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 0x7f144806eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f145c006fd0 tx=0x7f145c008040 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 msgr2=0x7f146c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f14640090b0 tx=0x7f14640046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 shutdown_connections 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f144806ca10 0x7f144806eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c15ecb0 0x7f146c06d800 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f146c0d33e0 0x7f146c0cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 --2- 172.21.15.67:0/149000824 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f146c0cf8e0 0x7f146c154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 >> 172.21.15.67:0/149000824 conn(0x7f146c08bff0 msgr2=0x7f146c0b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.180+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 shutdown_connections 2024-04-17T08:50:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.181+0000 7f1470a00640 1 -- 172.21.15.67:0/149000824 wait complete. 2024-04-17T08:50:17.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-04-17T08:50:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.423+0000 7f3db2800640 1 Processor -- start 2024-04-17T08:50:17.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.423+0000 7f3db2800640 1 -- start start 2024-04-17T08:50:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 0x7f3dac15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dac0b4210 0x7f3dac0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 0x7f3dac0af980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3dac15f440 con 0x7f3dac0b4210 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3dac15f720 con 0x7f3dac0b4690 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db2800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3dac15fa00 con 0x7f3dac15c5a0 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dac0b4210 0x7f3dac0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dac0b4210 0x7f3dac0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59042/0 (socket says 172.21.15.67:59042) 2024-04-17T08:50:17.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.425+0000 7f3db0a00640 1 -- 172.21.15.67:0/2274594110 learned_addr learned my addr 172.21.15.67:0/2274594110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db1e00640 1 --2- 172.21.15.67:0/2274594110 >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 0x7f3dac0af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db1400640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 0x7f3dac15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db0a00640 1 -- 172.21.15.67:0/2274594110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 msgr2=0x7f3dac15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db0a00640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 0x7f3dac15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db0a00640 1 -- 172.21.15.67:0/2274594110 >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 msgr2=0x7f3dac0af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db0a00640 1 --2- 172.21.15.67:0/2274594110 >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 0x7f3dac0af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db0a00640 1 -- 172.21.15.67:0/2274594110 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac15fce0 con 0x7f3dac0b4210 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db1e00640 1 --2- 172.21.15.67:0/2274594110 >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 0x7f3dac0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:17.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.426+0000 7f3db1400640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 0x7f3dac15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:17.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.427+0000 7f3db0a00640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac0b4210 0x7f3dac0af440 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009870 tx=0x7f3d9802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae618dfe53e1d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.427+0000 7f3dabe00640 1 -- 172.21.15.67:0/2274594110 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98038070 con 0x7f3dac0b4210 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.427+0000 7f3dabe00640 1 -- 172.21.15.67:0/2274594110 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d98031740 con 0x7f3dac0b4210 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.427+0000 7f3dabe00640 1 -- 172.21.15.67:0/2274594110 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9802f050 con 0x7f3dac0b4210 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 -- 172.21.15.67:0/2274594110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac0b4210 msgr2=0x7f3dac0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac0b4210 0x7f3dac0af440 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009870 tx=0x7f3d9802b7e0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 -- 172.21.15.67:0/2274594110 shutdown_connections 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 --2- 172.21.15.67:0/2274594110 >> v2:172.21.15.187:3300/0 conn(0x7f3dac0b4690 0x7f3dac0af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac0b4210 0x7f3dac0af440 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 --2- 172.21.15.67:0/2274594110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac15c5a0 0x7f3dac15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 -- 172.21.15.67:0/2274594110 >> 172.21.15.67:0/2274594110 conn(0x7f3dac067d10 msgr2=0x7f3dac0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 -- 172.21.15.67:0/2274594110 shutdown_connections 2024-04-17T08:50:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.428+0000 7f3db2800640 1 -- 172.21.15.67:0/2274594110 wait complete. 2024-04-17T08:50:17.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.431+0000 7f3db2800640 1 Processor -- start 2024-04-17T08:50:17.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.431+0000 7f3db2800640 1 -- start start 2024-04-17T08:50:17.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dac0b4210 0x7f3dac1e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 0x7f3dac1f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac06e7f0 con 0x7f3dac15c5a0 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac06e670 con 0x7f3dac0b4210 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db2800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3dac071df0 con 0x7f3dac0b4690 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59046/0 (socket says 172.21.15.67:59046) 2024-04-17T08:50:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.432+0000 7f3db1e00640 1 -- 172.21.15.67:0/2194917974 learned_addr learned my addr 172.21.15.67:0/2194917974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:17.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db0a00640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 0x7f3dac1f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1400640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dac0b4210 0x7f3dac1e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 -- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 msgr2=0x7f3dac1f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 0x7f3dac1f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 -- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dac0b4210 msgr2=0x7f3dac1e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dac0b4210 0x7f3dac1e4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 -- 172.21.15.67:0/2194917974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac1e53d0 con 0x7f3dac15c5a0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db0a00640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 0x7f3dac1f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.433+0000 7f3db1e00640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f3da400cdd0 tx=0x7f3da400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.434+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da401a070 con 0x7f3dac15c5a0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.434+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da400d840 con 0x7f3dac15c5a0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.434+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da400dc30 con 0x7f3dac15c5a0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.434+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac1e56b0 con 0x7f3dac15c5a0 2024-04-17T08:50:17.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.434+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac1e5c70 con 0x7f3dac15c5a0 2024-04-17T08:50:17.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.435+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dac068a10 con 0x7f3dac15c5a0 2024-04-17T08:50:17.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.436+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f3da401d070 con 0x7f3dac15c5a0 2024-04-17T08:50:17.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.437+0000 7f3da9600640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 0x7f3d8c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:17.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.438+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f3da408c4b0 con 0x7f3dac15c5a0 2024-04-17T08:50:17.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.438+0000 7f3db1400640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 0x7f3d8c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:17.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.440+0000 7f3db1400640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 0x7f3d8c06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3da000a950 tx=0x7f3da0008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.443+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3da405a800 con 0x7f3dac15c5a0 2024-04-17T08:50:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.724+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7f3dac0b2450 con 0x7f3dac15c5a0 2024-04-17T08:50:17.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.762+0000 7f3da9600640 1 -- 172.21.15.67:0/2194917974 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3159) ==== 163+0+0 (secure 0 0 0) 0x7f3da405a100 con 0x7f3dac15c5a0 2024-04-17T08:50:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.764+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 msgr2=0x7f3d8c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 0x7f3d8c06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3da000a950 tx=0x7f3da0008040 comp rx=0 tx=0).stop 2024-04-17T08:50:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 msgr2=0x7f3dac142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:17.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f3da400cdd0 tx=0x7f3da400a710 comp rx=0 tx=0).stop 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 shutdown_connections 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f3d8c06ca10 0x7f3d8c06eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dac15c5a0 0x7f3dac142fc0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dac0b4690 0x7f3dac1f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 --2- 172.21.15.67:0/2194917974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dac0b4210 0x7f3dac1e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 >> 172.21.15.67:0/2194917974 conn(0x7f3dac067d10 msgr2=0x7f3dac0b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 shutdown_connections 2024-04-17T08:50:17.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:17.765+0000 7f3db2800640 1 -- 172.21.15.67:0/2194917974 wait complete. 2024-04-17T08:50:17.785 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-04-17T08:50:17.785 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-04-17T08:50:17.786 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-17T08:50:17.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:18.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.009+0000 7f8be1a00640 1 Processor -- start 2024-04-17T08:50:18.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.009+0000 7f8be1a00640 1 -- start start 2024-04-17T08:50:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8be1a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8bdc0cd480 0x7f8bdc06e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8be1a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bdc0ceb00 con 0x7f8bdc0cd020 2024-04-17T08:50:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8be1a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bdc0cede0 con 0x7f8bdc06f670 2024-04-17T08:50:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8be1a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8bdc0cf0c0 con 0x7f8bdc0cd480 2024-04-17T08:50:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8bda000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8bdc0cd020 0x7f8bdc06e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45880/0 (socket says 172.21.15.67:45880) 2024-04-17T08:50:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.011+0000 7f8bda000640 1 -- 172.21.15.67:0/3102648211 learned_addr learned my addr 172.21.15.67:0/3102648211 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bdb400640 1 --2- 172.21.15.67:0/3102648211 >> v2:172.21.15.187:3300/0 conn(0x7f8bdc0cd480 0x7f8bdc06e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018704415 0 0) 0x7f8bdc0ceb00 con 0x7f8bdc0cd020 2024-04-17T08:50:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc0003520 con 0x7f8bdc0cd020 2024-04-17T08:50:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2934186700 0 0) 0x7f8bc0003520 con 0x7f8bdc0cd020 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 >> v2:172.21.15.187:3300/0 conn(0x7f8bdc0cd480 msgr2=0x7f8bdc06e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 --2- 172.21.15.67:0/3102648211 >> v2:172.21.15.187:3300/0 conn(0x7f8bdc0cd480 0x7f8bdc06e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 >> v1:172.21.15.187:6789/0 conn(0x7f8bdc06f670 legacy=0x7f8bdc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc0cf3a0 con 0x7f8bdc0cd020 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8bc8001670 con 0x7f8bdc0cd020 2024-04-17T08:50:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 >> v1:172.21.15.67:6789/0 conn(0x7f8bdc0cd020 legacy=0x7f8bdc06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 0x7f8bc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc0004800 0x7f8bc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.012+0000 7f8bd9600640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bc00051c0 0x7f8bc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0cf0c0 con 0x7f8bc00039b0 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc000c060 con 0x7f8bc00051c0 2024-04-17T08:50:18.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc000c260 con 0x7f8bc0004800 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 0x7f8bc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdb400640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bc00051c0 0x7f8bc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bda000640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc0004800 0x7f8bc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 -- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc0004800 msgr2=0x7f8bc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc0004800 0x7f8bc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 -- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bc00051c0 msgr2=0x7f8bc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bc00051c0 0x7f8bc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdaa00640 1 -- 172.21.15.67:0/3102648211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc0003520 con 0x7f8bc00039b0 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdb400640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bc00051c0 0x7f8bc000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.013+0000 7f8bdb400640 1 -- 172.21.15.67:0/3102648211 reap_dead start 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.015+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 0x7f8bc0003dd0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc001750 tx=0x7f8bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d00904fac27b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.015+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc001370 con 0x7f8bc00039b0 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.015+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bcc0052d0 con 0x7f8bc00039b0 2024-04-17T08:50:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.015+0000 7f8bd9600640 1 -- 172.21.15.67:0/3102648211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc003830 con 0x7f8bc00039b0 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 -- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 msgr2=0x7f8bc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 0x7f8bc0003dd0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc001750 tx=0x7f8bc0005830 comp rx=0 tx=0).stop 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 -- 172.21.15.67:0/3102648211 shutdown_connections 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 --2- 172.21.15.67:0/3102648211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc00039b0 0x7f8bc0003dd0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 -- 172.21.15.67:0/3102648211 >> 172.21.15.67:0/3102648211 conn(0x7f8bdc06bf50 msgr2=0x7f8bdc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:18.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 -- 172.21.15.67:0/3102648211 shutdown_connections 2024-04-17T08:50:18.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.016+0000 7f8be1a00640 1 -- 172.21.15.67:0/3102648211 wait complete. 2024-04-17T08:50:18.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.019+0000 7f8be1a00640 1 Processor -- start 2024-04-17T08:50:18.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.019+0000 7f8be1a00640 1 -- start start 2024-04-17T08:50:18.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 0x7f8bdc1f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bdc0c9680 0x7f8bdc14f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0ce8a0 con 0x7f8bdc0c9200 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8bdc0ce720 con 0x7f8bdc0c9680 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.020+0000 7f8be1a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8bdc15cb00 con 0x7f8bc00039b0 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59062/0 (socket says 172.21.15.67:59062) 2024-04-17T08:50:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 -- 172.21.15.67:0/1806594629 learned_addr learned my addr 172.21.15.67:0/1806594629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bdb400640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bdc0c9680 0x7f8bdc14f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 0x7f8bdc1f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 -- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 msgr2=0x7f8bdc1f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 0x7f8bdc1f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 -- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bdc0c9680 msgr2=0x7f8bdc14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bdc0c9680 0x7f8bdc14f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bda000640 1 -- 172.21.15.67:0/1806594629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bdc1f5af0 con 0x7f8bdc0c9200 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.021+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 0x7f8bdc1f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8bda000640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e7d0 tx=0x7f8bc800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800c870 con 0x7f8bdc0c9200 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc8018070 con 0x7f8bdc0c9200 2024-04-17T08:50:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800cc10 con 0x7f8bdc0c9200 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bdc1f7e70 con 0x7f8bdc0c9200 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.022+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bdc1f8430 con 0x7f8bdc0c9200 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.023+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bdc068b90 con 0x7f8bdc0c9200 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.024+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f8bc801d460 con 0x7f8bdc0c9200 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.025+0000 7f8bd2a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 0x7f8bbc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.025+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f8bc808bbe0 con 0x7f8bdc0c9200 2024-04-17T08:50:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.026+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 0x7f8bbc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.027+0000 7f8bdaa00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 0x7f8bbc06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc033000 tx=0x7f8bcc009030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.030+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8bc8059f30 con 0x7f8bdc0c9200 2024-04-17T08:50:18.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.235+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8bdc0c53c0 con 0x7f8bdc0c9200 2024-04-17T08:50:18.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.237+0000 7f8bd2a00640 1 -- 172.21.15.67:0/1806594629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) ==== 74+0+14042 (secure 0 0 0) 0x7f8bc8059830 con 0x7f8bdc0c9200 2024-04-17T08:50:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:18.237 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":221,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:15.753729+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1783845869":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/2158603650":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/1545043147":"2024-04-18T08:50:08.382579+0000","172.21.15.187:0/3057701238":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6832/953837164":"2024-04-18T08:50:08.382579+0000","172.21.15.67:6833/2414893402":"2024-04-18T08:50:12.375506+0000","172.21.15.187:0/2473833231":"2024-04-18T08:50:08.382579+0000","172.21.15.67:6835/2414893402":"2024-04-18T08:50:12.375506+0000","172.21.15.187:0/3145453289":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/910136246":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/4251121774":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2270739884":"2024-04-18T08:50:08.382579+0000","172.21.15.187:6833/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:6832/2105214300":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2605475186":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/2381197913":"2024-04-18T08:47:58.503956+0000","172.21.15.187:0/713566291":"2024-04-18T08:47:58.503956+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 msgr2=0x7f8bbc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 0x7f8bbc06eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc033000 tx=0x7f8bcc009030 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 msgr2=0x7f8bdc0af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800e7d0 tx=0x7f8bc800ec90 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 shutdown_connections 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8bbc06ca10 0x7f8bbc06eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8bdc0c9680 0x7f8bdc14f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bdc0c9200 0x7f8bdc0af230 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 --2- 172.21.15.67:0/1806594629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8bc00039b0 0x7f8bdc1f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 >> 172.21.15.67:0/1806594629 conn(0x7f8bdc06bf50 msgr2=0x7f8bdc0b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 shutdown_connections 2024-04-17T08:50:18.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.240+0000 7f8be1a00640 1 -- 172.21.15.67:0/1806594629 wait complete. 2024-04-17T08:50:18.259 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:50:18.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:50:18.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.487+0000 7fd6d9000640 1 Processor -- start 2024-04-17T08:50:18.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.487+0000 7fd6d9000640 1 -- start start 2024-04-17T08:50:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6d40d22b0 0x7fd6d40d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d40d4f70 con 0x7fd6d415ec20 2024-04-17T08:50:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d9000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d40d5250 con 0x7fd6d40b3b50 2024-04-17T08:50:18.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d40d2760 con 0x7fd6d40d22b0 2024-04-17T08:50:18.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6d40d22b0 0x7fd6d40d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6d415ec20 0x7fd6d406ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45888/0 (socket says 172.21.15.67:45888) 2024-04-17T08:50:18.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.488+0000 7fd6d3400640 1 -- 172.21.15.67:0/233000132 learned_addr learned my addr 172.21.15.67:0/233000132 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.489+0000 7fd6d3e00640 1 -- 172.21.15.67:0/233000132 >> v1:172.21.15.187:6789/0 conn(0x7fd6d40b3b50 legacy=0x7fd6d40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.489+0000 7fd6d3e00640 1 -- 172.21.15.67:0/233000132 >> v1:172.21.15.67:6789/0 conn(0x7fd6d415ec20 legacy=0x7fd6d406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.489+0000 7fd6d3e00640 1 -- 172.21.15.67:0/233000132 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d40cf4d0 con 0x7fd6d40d22b0 2024-04-17T08:50:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.489+0000 7fd6d2000640 1 -- 172.21.15.67:0/233000132 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785688943 0 0) 0x7fd6d40d4f70 con 0x7fd6d415ec20 2024-04-17T08:50:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.490+0000 7fd6d3e00640 1 --2- 172.21.15.67:0/233000132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40d10d0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0097a0 tx=0x7fd6cc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165ee022041e886d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.490+0000 7fd6d2000640 1 -- 172.21.15.67:0/233000132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc038070 con 0x7fd6d40d22b0 2024-04-17T08:50:18.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.490+0000 7fd6d2000640 1 -- 172.21.15.67:0/233000132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6cc031700 con 0x7fd6d40d22b0 2024-04-17T08:50:18.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.490+0000 7fd6d2000640 1 -- 172.21.15.67:0/233000132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc031ad0 con 0x7fd6d40d22b0 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 -- 172.21.15.67:0/233000132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 msgr2=0x7fd6d40d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 --2- 172.21.15.67:0/233000132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40d10d0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0097a0 tx=0x7fd6cc02b7a0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 -- 172.21.15.67:0/233000132 shutdown_connections 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 --2- 172.21.15.67:0/233000132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40d10d0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 -- 172.21.15.67:0/233000132 >> 172.21.15.67:0/233000132 conn(0x7fd6d408bff0 msgr2=0x7fd6d40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 -- 172.21.15.67:0/233000132 shutdown_connections 2024-04-17T08:50:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.491+0000 7fd6d9000640 1 -- 172.21.15.67:0/233000132 wait complete. 2024-04-17T08:50:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.493+0000 7fd6d9000640 1 Processor -- start 2024-04-17T08:50:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.493+0000 7fd6d9000640 1 -- start start 2024-04-17T08:50:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6d40b3b50 0x7fd6d41e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6d415ec20 0x7fd6d40cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d406d150 con 0x7fd6d40d22b0 2024-04-17T08:50:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d40cf180 con 0x7fd6d415ec20 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d9000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d406d2d0 con 0x7fd6d40b3b50 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59070/0 (socket says 172.21.15.67:59070) 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.494+0000 7fd6d2a00640 1 -- 172.21.15.67:0/644534210 learned_addr learned my addr 172.21.15.67:0/644534210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 -- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6d40b3b50 msgr2=0x7fd6d41e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6d40b3b50 0x7fd6d41e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 -- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6d415ec20 msgr2=0x7fd6d40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6d415ec20 0x7fd6d40cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 -- 172.21.15.67:0/644534210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41619d0 con 0x7fd6d40d22b0 2024-04-17T08:50:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d2a00640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00c7a0 tx=0x7fd6bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc016020 con 0x7fd6d40d22b0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6bc00a9f0 con 0x7fd6d40d22b0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d4161c70 con 0x7fd6d40d22b0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.495+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00ae20 con 0x7fd6d40d22b0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.496+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d4161f50 con 0x7fd6d40d22b0 2024-04-17T08:50:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.497+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d4068a10 con 0x7fd6d40d22b0 2024-04-17T08:50:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.498+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fd6bc019020 con 0x7fd6d40d22b0 2024-04-17T08:50:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.499+0000 7fd6cb400640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 0x7fd6b006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.499+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7fd6bc08bf50 con 0x7fd6d40d22b0 2024-04-17T08:50:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.499+0000 7fd6d3400640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 0x7fd6b006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.500+0000 7fd6d3400640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 0x7fd6b006ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4007fd0 tx=0x7fd6c4007960 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.503+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd6bc05a290 con 0x7fd6d40d22b0 2024-04-17T08:50:18.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.709+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6d40d55f0 con 0x7fd6d40d22b0 2024-04-17T08:50:18.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.710+0000 7fd6cb400640 1 -- 172.21.15.67:0/644534210 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v221) ==== 93+0+10 (secure 0 0 0) 0x7fd6bc059b90 con 0x7fd6d40d22b0 2024-04-17T08:50:18.709 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.712+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 msgr2=0x7fd6b006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.712+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 0x7fd6b006ef00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4007fd0 tx=0x7fd6c4007960 comp rx=0 tx=0).stop 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.712+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 msgr2=0x7fd6d40b95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.712+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00c7a0 tx=0x7fd6bc00cc60 comp rx=0 tx=0).stop 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 shutdown_connections 2024-04-17T08:50:18.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd6b006ca40 0x7fd6b006ef00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6d415ec20 0x7fd6d40cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6d40d22b0 0x7fd6d40b95e0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 --2- 172.21.15.67:0/644534210 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6d40b3b50 0x7fd6d41e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 >> 172.21.15.67:0/644534210 conn(0x7fd6d408bff0 msgr2=0x7fd6d40b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 shutdown_connections 2024-04-17T08:50:18.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.713+0000 7fd6d9000640 1 -- 172.21.15.67:0/644534210 wait complete. 2024-04-17T08:50:18.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-17T08:50:18.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.929+0000 7f9535600640 1 Processor -- start 2024-04-17T08:50:18.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.930+0000 7f9535600640 1 -- start start 2024-04-17T08:50:18.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f953015e9a0 0x7f95300d4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95300d04f0 0x7f95300d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953015fb20 0x7f95300d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f953015ed80 con 0x7f953015fb20 2024-04-17T08:50:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f95300cf730 con 0x7f953015e9a0 2024-04-17T08:50:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.931+0000 7f9535600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95300cfa10 con 0x7f95300d04f0 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953015fb20 0x7f95300d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953015fb20 0x7f95300d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59078/0 (socket says 172.21.15.67:59078) 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 -- 172.21.15.67:0/2200113776 learned_addr learned my addr 172.21.15.67:0/2200113776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f952f400640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95300d04f0 0x7f95300d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f952fe00640 1 --2- 172.21.15.67:0/2200113776 >> v2:172.21.15.187:3300/0 conn(0x7f953015e9a0 0x7f95300d4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 -- 172.21.15.67:0/2200113776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95300d04f0 msgr2=0x7f95300d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95300d04f0 0x7f95300d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 -- 172.21.15.67:0/2200113776 >> v2:172.21.15.187:3300/0 conn(0x7f953015e9a0 msgr2=0x7f95300d4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 --2- 172.21.15.67:0/2200113776 >> v2:172.21.15.187:3300/0 conn(0x7f953015e9a0 0x7f95300d4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.932+0000 7f9534c00640 1 -- 172.21.15.67:0/2200113776 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95300cfcf0 con 0x7f953015fb20 2024-04-17T08:50:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.934+0000 7f9534c00640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953015fb20 0x7f95300d1550 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f95300cf940 tx=0x7f952802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f51725f5b13a889 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.934+0000 7f952ea00640 1 -- 172.21.15.67:0/2200113776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952802ca60 con 0x7f953015fb20 2024-04-17T08:50:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.934+0000 7f952ea00640 1 -- 172.21.15.67:0/2200113776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f952802cc00 con 0x7f953015fb20 2024-04-17T08:50:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.934+0000 7f952ea00640 1 -- 172.21.15.67:0/2200113776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95280343f0 con 0x7f953015fb20 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 -- 172.21.15.67:0/2200113776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953015fb20 msgr2=0x7f95300d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953015fb20 0x7f95300d1550 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f95300cf940 tx=0x7f952802bec0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 -- 172.21.15.67:0/2200113776 shutdown_connections 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953015fb20 0x7f95300d1550 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 --2- 172.21.15.67:0/2200113776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95300d04f0 0x7f95300d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 --2- 172.21.15.67:0/2200113776 >> v2:172.21.15.187:3300/0 conn(0x7f953015e9a0 0x7f95300d4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 -- 172.21.15.67:0/2200113776 >> 172.21.15.67:0/2200113776 conn(0x7f953008bff0 msgr2=0x7f95300d4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 -- 172.21.15.67:0/2200113776 shutdown_connections 2024-04-17T08:50:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.935+0000 7f9535600640 1 -- 172.21.15.67:0/2200113776 wait complete. 2024-04-17T08:50:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.936+0000 7f9535600640 1 Processor -- start 2024-04-17T08:50:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.937+0000 7f9535600640 1 -- start start 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 0x7f95300cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953015fb20 0x7f95300cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f95300cf3e0 con 0x7f95300d04f0 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f95300d5370 con 0x7f953015e9a0 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9535600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9530162490 con 0x7f953015fb20 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f952fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f952fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59086/0 (socket says 172.21.15.67:59086) 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f952fe00640 1 -- 172.21.15.67:0/2199362430 learned_addr learned my addr 172.21.15.67:0/2199362430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f952f400640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 0x7f95300cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.938+0000 7f9534c00640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953015fb20 0x7f95300cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 -- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953015fb20 msgr2=0x7f95300cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953015fb20 0x7f95300cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 -- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 msgr2=0x7f95300cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 0x7f95300cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 -- 172.21.15.67:0/2199362430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95300b4b40 con 0x7f95300d04f0 2024-04-17T08:50:18.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952f400640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 0x7f95300cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f952fe00640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f952000d800 tx=0x7f952000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952000b6c0 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9520019070 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952000ba80 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95300b4e20 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.939+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95300b53e0 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.940+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9530068a10 con 0x7f95300d04f0 2024-04-17T08:50:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.941+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f952000bc20 con 0x7f95300d04f0 2024-04-17T08:50:18.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.942+0000 7f9527e00640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 0x7f950c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.942+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f952008c1b0 con 0x7f95300d04f0 2024-04-17T08:50:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.943+0000 7f952f400640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 0x7f950c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:18.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.944+0000 7f952f400640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 0x7f950c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f953015a5e0 tx=0x7f9518008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:18.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:18.947+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f952005a4f0 con 0x7f95300d04f0 2024-04-17T08:50:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.225+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7f95300d2080 con 0x7f95300d04f0 2024-04-17T08:50:19.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.777+0000 7f9527e00640 1 -- 172.21.15.67:0/2199362430 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3161) ==== 189+0+0 (secure 0 0 0) 0x7f9520059df0 con 0x7f95300d04f0 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.780+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 msgr2=0x7f950c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.780+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 0x7f950c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f953015a5e0 tx=0x7f9518008040 comp rx=0 tx=0).stop 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.780+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 msgr2=0x7f953006dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.780+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f952000d800 tx=0x7f952000dcc0 comp rx=0 tx=0).stop 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 shutdown_connections 2024-04-17T08:50:19.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f950c06ca10 0x7f950c06eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953015fb20 0x7f95300cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f953015e9a0 0x7f95300cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 --2- 172.21.15.67:0/2199362430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95300d04f0 0x7f953006dec0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 >> 172.21.15.67:0/2199362430 conn(0x7f953008bff0 msgr2=0x7f95300d22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 shutdown_connections 2024-04-17T08:50:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:19.781+0000 7f9535600640 1 -- 172.21.15.67:0/2199362430 wait complete. 2024-04-17T08:50:19.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:50:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.029+0000 7f3822400640 1 Processor -- start 2024-04-17T08:50:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.029+0000 7f3822400640 1 -- start start 2024-04-17T08:50:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f3822400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f381c14cbe0 0x7f381c0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f3822400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c0c5fd0 con 0x7f381c06d090 2024-04-17T08:50:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f3822400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c09b360 con 0x7f381c0c5bc0 2024-04-17T08:50:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f3822400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f381c06ff40 con 0x7f381c14cbe0 2024-04-17T08:50:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f381b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f381c06d090 0x7f381c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45890/0 (socket says 172.21.15.67:45890) 2024-04-17T08:50:20.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.031+0000 7f381b400640 1 -- 172.21.15.67:0/4277609798 learned_addr learned my addr 172.21.15.67:0/4277609798 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:20.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 739365757 0 0) 0x7f380c001490 con 0x7f381c06d090 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3800003520 con 0x7f381c06d090 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381aa00640 1 --2- 172.21.15.67:0/4277609798 >> v2:172.21.15.187:3300/0 conn(0x7f381c14cbe0 0x7f381c0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3337598894 0 0) 0x7f3800003520 con 0x7f381c06d090 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 >> v2:172.21.15.187:3300/0 conn(0x7f381c14cbe0 msgr2=0x7f381c0c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 --2- 172.21.15.67:0/4277609798 >> v2:172.21.15.187:3300/0 conn(0x7f381c14cbe0 0x7f381c0c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 >> v1:172.21.15.187:6789/0 conn(0x7f381c0c5bc0 legacy=0x7f381c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c09b640 con 0x7f381c06d090 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f380c001bc0 con 0x7f381c06d090 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 >> v1:172.21.15.67:6789/0 conn(0x7f381c06d090 legacy=0x7f381c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38000039b0 0x7f3800003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 0x7f3800004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000051c0 0x7f380000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f381c06ff40 con 0x7f3800004800 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f380000c060 con 0x7f38000051c0 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.032+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f380000c260 con 0x7f38000039b0 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 0x7f3800004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381be00640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000051c0 0x7f380000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381b400640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38000039b0 0x7f3800003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 -- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38000039b0 msgr2=0x7f3800003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f38000039b0 0x7f3800003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 -- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000051c0 msgr2=0x7f380000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000051c0 0x7f380000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381aa00640 1 -- 172.21.15.67:0/4277609798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800003520 con 0x7f3800004800 2024-04-17T08:50:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.033+0000 7f381be00640 1 -- 172.21.15.67:0/4277609798 reap_dead start 2024-04-17T08:50:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.034+0000 7f381aa00640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 0x7f3800004c80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f3804009a90 tx=0x7f3800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcdbf60e66baac server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.034+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804039070 con 0x7f3800004800 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.034+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f380402c680 con 0x7f3800004800 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.034+0000 7f381a000640 1 -- 172.21.15.67:0/4277609798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804009690 con 0x7f3800004800 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 -- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 msgr2=0x7f3800004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 0x7f3800004c80 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f3804009a90 tx=0x7f3800005830 comp rx=0 tx=0).stop 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 -- 172.21.15.67:0/4277609798 shutdown_connections 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 --2- 172.21.15.67:0/4277609798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800004800 0x7f3800004c80 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 -- 172.21.15.67:0/4277609798 >> 172.21.15.67:0/4277609798 conn(0x7f381c0b41a0 msgr2=0x7f381c0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 -- 172.21.15.67:0/4277609798 shutdown_connections 2024-04-17T08:50:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.035+0000 7f3822400640 1 -- 172.21.15.67:0/4277609798 wait complete. 2024-04-17T08:50:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.037+0000 7f3822400640 1 Processor -- start 2024-04-17T08:50:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.037+0000 7f3822400640 1 -- start start 2024-04-17T08:50:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 0x7f381c14b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 0x7f381c0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f381c0c5680 con 0x7f381c0c7c00 2024-04-17T08:50:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f381c09bc00 con 0x7f3800004800 2024-04-17T08:50:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.038+0000 7f3822400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f381c0c5800 con 0x7f381c0b6f70 2024-04-17T08:50:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59100/0 (socket says 172.21.15.67:59100) 2024-04-17T08:50:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 -- 172.21.15.67:0/662157615 learned_addr learned my addr 172.21.15.67:0/662157615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381aa00640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 0x7f381c0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 -- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 msgr2=0x7f381c0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 0x7f381c0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381b400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 0x7f381c14b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 -- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 msgr2=0x7f381c14b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 0x7f381c14b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 -- 172.21.15.67:0/662157615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c0c6b60 con 0x7f381c0c7c00 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381aa00640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 0x7f381c0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381b400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 0x7f381c14b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.039+0000 7f381be00640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f381c09c190 tx=0x7f381400e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.040+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814019070 con 0x7f381c0c7c00 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.040+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3814005110 con 0x7f381c0c7c00 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.040+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c0c6e40 con 0x7f381c0c7c00 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.040+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3814007030 con 0x7f381c0c7c00 2024-04-17T08:50:20.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.040+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c0c7400 con 0x7f381c0c7c00 2024-04-17T08:50:20.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.041+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c068ca0 con 0x7f381c0c7c00 2024-04-17T08:50:20.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.042+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f381401c070 con 0x7f381c0c7c00 2024-04-17T08:50:20.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.043+0000 7f3813400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 0x7f37f806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:20.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.043+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f381408b0f0 con 0x7f381c0c7c00 2024-04-17T08:50:20.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.044+0000 7f381b400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 0x7f37f806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:20.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.045+0000 7f381b400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 0x7f37f806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f380c002230 tx=0x7f380c001490 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.048+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f38140593a0 con 0x7f381c0c7c00 2024-04-17T08:50:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.251+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f381c0c9300 con 0x7f381c0c7c00 2024-04-17T08:50:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.256+0000 7f3813400640 1 -- 172.21.15.67:0/662157615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7f3814058ca0 con 0x7f381c0c7c00 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 msgr2=0x7f37f806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 0x7f37f806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f380c002230 tx=0x7f380c001490 comp rx=0 tx=0).stop 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 msgr2=0x7f381c0d12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f381c09c190 tx=0x7f381400e5f0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 shutdown_connections 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f37f806ca10 0x7f37f806eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c0c7c00 0x7f381c0d12b0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f381c0b6f70 0x7f381c0c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 --2- 172.21.15.67:0/662157615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3800004800 0x7f381c14b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.259+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 >> 172.21.15.67:0/662157615 conn(0x7f381c0b41a0 msgr2=0x7f381c0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.260+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 shutdown_connections 2024-04-17T08:50:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:20.260+0000 7f3822400640 1 -- 172.21.15.67:0/662157615 wait complete. 2024-04-17T08:50:20.280 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:50:20.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:50:20.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:50:20.537 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:50:20.537 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:20.537 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:20.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:50:20.599 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:50:20.599 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:20.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:50:20.656 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:20.656 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:20.717 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:50:20.717 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:50:20.718 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:20.775 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:50:20.775 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:50:20.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.009+0000 7f2de6000640 1 Processor -- start 2024-04-17T08:50:21.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.009+0000 7f2de6000640 1 -- start start 2024-04-17T08:50:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2de6000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2de00cd480 0x7f2de006e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2de6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de00ceb00 con 0x7f2de00cd020 2024-04-17T08:50:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2de6000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de00cede0 con 0x7f2de006f670 2024-04-17T08:50:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2de6000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2de00cf0c0 con 0x7f2de00cd480 2024-04-17T08:50:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2dd6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2de00cd020 0x7f2de006e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45906/0 (socket says 172.21.15.67:45906) 2024-04-17T08:50:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.011+0000 7f2dd6a00640 1 -- 172.21.15.67:0/676327266 learned_addr learned my addr 172.21.15.67:0/676327266 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.012+0000 7f2ddfe00640 1 --2- 172.21.15.67:0/676327266 >> v2:172.21.15.187:3300/0 conn(0x7f2de00cd480 0x7f2de006e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.012+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779722084 0 0) 0x7f2de00ceb00 con 0x7f2de00cd020 2024-04-17T08:50:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.012+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dc4003520 con 0x7f2de00cd020 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3639128352 0 0) 0x7f2dc4003520 con 0x7f2de00cd020 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 >> v2:172.21.15.187:3300/0 conn(0x7f2de00cd480 msgr2=0x7f2de006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 --2- 172.21.15.67:0/676327266 >> v2:172.21.15.187:3300/0 conn(0x7f2de00cd480 0x7f2de006e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 >> v1:172.21.15.187:6789/0 conn(0x7f2de006f670 legacy=0x7f2de006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de00cf3a0 con 0x7f2de00cd020 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2dc8001670 con 0x7f2de00cd020 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 >> v1:172.21.15.67:6789/0 conn(0x7f2de00cd020 legacy=0x7f2de006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 0x7f2dc4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4004800 0x7f2dc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2dc40051c0 0x7f2dc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2de00cf0c0 con 0x7f2dc4003a00 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc400c0c0 con 0x7f2dc40051c0 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc400c2c0 con 0x7f2dc4004800 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddf400640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 0x7f2dc4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2dd6a00640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4004800 0x7f2dc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.013+0000 7f2ddfe00640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2dc40051c0 0x7f2dc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddf400640 1 -- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4004800 msgr2=0x7f2dc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddf400640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4004800 0x7f2dc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddf400640 1 -- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2dc40051c0 msgr2=0x7f2dc400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddf400640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2dc40051c0 0x7f2dc400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddf400640 1 -- 172.21.15.67:0/676327266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc4003520 con 0x7f2dc4003a00 2024-04-17T08:50:21.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.014+0000 7f2ddfe00640 1 -- 172.21.15.67:0/676327266 reap_dead start 2024-04-17T08:50:21.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.015+0000 7f2ddf400640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 0x7f2dc4003e20 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0001750 tx=0x7f2dc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1302fc840614df4f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0001370 con 0x7f2dc4003a00 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd00052d0 con 0x7f2dc4003a00 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2ddea00640 1 -- 172.21.15.67:0/676327266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0003830 con 0x7f2dc4003a00 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 -- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 msgr2=0x7f2dc4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 0x7f2dc4003e20 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0001750 tx=0x7f2dc4005830 comp rx=0 tx=0).stop 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 -- 172.21.15.67:0/676327266 shutdown_connections 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 --2- 172.21.15.67:0/676327266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dc4003a00 0x7f2dc4003e20 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 -- 172.21.15.67:0/676327266 >> 172.21.15.67:0/676327266 conn(0x7f2de006bf50 msgr2=0x7f2de00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 -- 172.21.15.67:0/676327266 shutdown_connections 2024-04-17T08:50:21.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.016+0000 7f2de6000640 1 -- 172.21.15.67:0/676327266 wait complete. 2024-04-17T08:50:21.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.019+0000 7f2de6000640 1 Processor -- start 2024-04-17T08:50:21.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.019+0000 7f2de6000640 1 -- start start 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4003a00 0x7f2de014f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2de015bdf0 0x7f2de00c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2de00ce8a0 con 0x7f2de00ad8d0 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2de00ce720 con 0x7f2de015bdf0 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2de6000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2de015cb00 con 0x7f2dc4003a00 2024-04-17T08:50:21.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2dd6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59114/0 (socket says 172.21.15.67:59114) 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2dd6a00640 1 -- 172.21.15.67:0/1005881300 learned_addr learned my addr 172.21.15.67:0/1005881300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.021+0000 7f2ddfe00640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2de015bdf0 0x7f2de00c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2ddf400640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4003a00 0x7f2de014f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 -- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4003a00 msgr2=0x7f2de014f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4003a00 0x7f2de014f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 -- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2de015bdf0 msgr2=0x7f2de00c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2de015bdf0 0x7f2de00c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 -- 172.21.15.67:0/1005881300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de01f4870 con 0x7f2de00ad8d0 2024-04-17T08:50:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd6a00640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800e7e0 tx=0x7f2dc800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800c660 con 0x7f2de00ad8d0 2024-04-17T08:50:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.022+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc8018070 con 0x7f2de00ad8d0 2024-04-17T08:50:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.023+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de01f4b50 con 0x7f2de00ad8d0 2024-04-17T08:50:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.023+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800ca60 con 0x7f2de00ad8d0 2024-04-17T08:50:21.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.023+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de01f5110 con 0x7f2de00ad8d0 2024-04-17T08:50:21.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.024+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de0068b90 con 0x7f2de00ad8d0 2024-04-17T08:50:21.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.025+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f2dc800cc00 con 0x7f2de00ad8d0 2024-04-17T08:50:21.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.026+0000 7f2dd7e00640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 0x7f2dbc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:21.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.027+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 5659+0+0 (secure 0 0 0) 0x7f2dc808b9e0 con 0x7f2de00ad8d0 2024-04-17T08:50:21.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.027+0000 7f2ddf400640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 0x7f2dbc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:21.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.028+0000 7f2ddf400640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 0x7f2dbc06eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0033000 tx=0x7f2dd0009030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:21.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.031+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2dc8059d20 con 0x7f2de00ad8d0 2024-04-17T08:50:21.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.281+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2de00d10b0 con 0x7f2de00ad8d0 2024-04-17T08:50:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.282+0000 7f2dd7e00640 1 -- 172.21.15.67:0/1005881300 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3876 (secure 0 0 0) 0x7f2dc8059620 con 0x7f2de00ad8d0 2024-04-17T08:50:21.281 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:21.281 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":104,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8341,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1646972298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1646972298},{"type":"v1","addr":"172.21.15.187:6836","nonce":1646972298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":8344,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3341466800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3341466800},{"type":"v1","addr":"172.21.15.187:6837","nonce":3341466800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10515,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3419835617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3419835617},{"type":"v1","addr":"172.21.15.67:6834","nonce":3419835617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97},{"gid":10935,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/124706274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":124706274},{"type":"v1","addr":"172.21.15.67:6835","nonce":124706274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":104}],"filesystems":[]} 2024-04-17T08:50:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 104 2024-04-17T08:50:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.284+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 msgr2=0x7f2dbc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.284+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 0x7f2dbc06eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0033000 tx=0x7f2dd0009030 comp rx=0 tx=0).stop 2024-04-17T08:50:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 msgr2=0x7f2de015b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:21.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800e7e0 tx=0x7f2dc800eca0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 shutdown_connections 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2dbc06ca10 0x7f2dbc06eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2de015bdf0 0x7f2de00c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2de00ad8d0 0x7f2de015b8b0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 --2- 172.21.15.67:0/1005881300 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2dc4003a00 0x7f2de014f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 >> 172.21.15.67:0/1005881300 conn(0x7f2de006bf50 msgr2=0x7f2de00b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 shutdown_connections 2024-04-17T08:50:21.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:21.285+0000 7f2de6000640 1 -- 172.21.15.67:0/1005881300 wait complete. 2024-04-17T08:50:21.303 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:50:21.303 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:50:21.304 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:50:21.304 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:50:21.304 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:50:21.304 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:50:21.304 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:50:21.304 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:50:21.304 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:50:21.304 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:50:21.305 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:50:21.305 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:50:21.305 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:50:21.305 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:50:21.305 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:50:21.305 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:50:21.321 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:50:21.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:50:21.322 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:50:21.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:50:21.331 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:50:21.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:50:21.368 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:50:21.368 DEBUG:teuthology.parallel:result is None 2024-04-17T08:50:21.373 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:50:21.373 DEBUG:teuthology.parallel:result is None 2024-04-17T08:50:21.380 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:50:21.380 DEBUG:teuthology.parallel:result is None 2024-04-17T08:50:21.533 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:50:21.543 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:50:21.565 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:50:25.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-17T08:50:27.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:50:27.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:50:27.331 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:50:27.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:50:27.339 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:50:27.340 DEBUG:teuthology.parallel:result is None 2024-04-17T08:50:27.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:50:27.520 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:50:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.575+0000 7fb1a1600640 1 Processor -- start 2024-04-17T08:50:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.576+0000 7fb1a1600640 1 -- start start 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.577+0000 7fb1a1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb19c0b9850 0x7fb19c0d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.577+0000 7fb1a1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0d4140 con 0x7fb19c15ee20 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.577+0000 7fb1a1600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19c0d3b80 con 0x7fb19c0d1ab0 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.577+0000 7fb1a1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb19c0b83b0 con 0x7fb19c0b9850 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb19be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb19c15ee20 0x7fb19c0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32868/0 (socket says 172.21.15.67:32868) 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb19be00640 1 -- 172.21.15.67:0/3592632663 learned_addr learned my addr 172.21.15.67:0/3592632663 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb1a0c00640 1 --2- 172.21.15.67:0/3592632663 >> v2:172.21.15.67:3300/0 conn(0x7fb19c0b9850 0x7fb19c0d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb1a0c00640 1 -- 172.21.15.67:0/3592632663 >> v1:172.21.15.187:6789/0 conn(0x7fb19c0d1ab0 legacy=0x7fb19c0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:27.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb1a0c00640 1 -- 172.21.15.67:0/3592632663 >> v1:172.21.15.67:6789/0 conn(0x7fb19c15ee20 legacy=0x7fb19c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:27.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.578+0000 7fb1a0c00640 1 -- 172.21.15.67:0/3592632663 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c0d3e60 con 0x7fb19c0b9850 2024-04-17T08:50:27.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.579+0000 7fb1a0c00640 1 --2- 172.21.15.67:0/3592632663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c0b9850 0x7fb19c0d3520 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcb27165dbc2af1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:27.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb19aa00640 1 -- 172.21.15.67:0/3592632663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194034070 con 0x7fb19c0b9850 2024-04-17T08:50:27.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb19aa00640 1 -- 172.21.15.67:0/3592632663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194030610 con 0x7fb19c0b9850 2024-04-17T08:50:27.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb19aa00640 1 -- 172.21.15.67:0/3592632663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940309e0 con 0x7fb19c0b9850 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb1a1600640 1 -- 172.21.15.67:0/3592632663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c0b9850 msgr2=0x7fb19c0d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb1a1600640 1 --2- 172.21.15.67:0/3592632663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c0b9850 0x7fb19c0d3520 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402b960 comp rx=0 tx=0).stop 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.580+0000 7fb1a1600640 1 -- 172.21.15.67:0/3592632663 shutdown_connections 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.581+0000 7fb1a1600640 1 --2- 172.21.15.67:0/3592632663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c0b9850 0x7fb19c0d3520 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.581+0000 7fb1a1600640 1 -- 172.21.15.67:0/3592632663 >> 172.21.15.67:0/3592632663 conn(0x7fb19c06d3c0 msgr2=0x7fb19c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.581+0000 7fb1a1600640 1 -- 172.21.15.67:0/3592632663 shutdown_connections 2024-04-17T08:50:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.581+0000 7fb1a1600640 1 -- 172.21.15.67:0/3592632663 wait complete. 2024-04-17T08:50:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.582+0000 7fb1a1600640 1 Processor -- start 2024-04-17T08:50:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.582+0000 7fb1a1600640 1 -- start start 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb19c0b9850 0x7fb19c0c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19c0d1ab0 0x7fb19c15f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c169a00 con 0x7fb19c15ee20 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c15f200 con 0x7fb19c0b9850 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.583+0000 7fb1a1600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb19c169b80 con 0x7fb19c0d1ab0 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43928/0 (socket says 172.21.15.67:43928) 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 -- 172.21.15.67:0/1262074564 learned_addr learned my addr 172.21.15.67:0/1262074564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb19be00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb19c0b9850 0x7fb19c0c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb19b400640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19c0d1ab0 0x7fb19c15f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 -- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19c0d1ab0 msgr2=0x7fb19c15f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19c0d1ab0 0x7fb19c15f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 -- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb19c0b9850 msgr2=0x7fb19c0c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb19c0b9850 0x7fb19c0c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 -- 172.21.15.67:0/1262074564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c0cc6c0 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.584+0000 7fb1a0c00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.585+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194034040 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.585+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194039990 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.585+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194039d80 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.585+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19c0cc9a0 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.585+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb19c0ccf60 con 0x7fb19c15ee20 2024-04-17T08:50:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.586+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19c068a10 con 0x7fb19c15ee20 2024-04-17T08:50:27.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.587+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fb194037070 con 0x7fb19c15ee20 2024-04-17T08:50:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.588+0000 7fb193e00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 0x7fb17806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.588+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(224..224 src has 1..224) ==== 5659+0+0 (secure 0 0 0) 0x7fb1940adf70 con 0x7fb19c15ee20 2024-04-17T08:50:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.589+0000 7fb19be00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 0x7fb17806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.590+0000 7fb19be00640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 0x7fb17806eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb18c005880 tx=0x7fb18c009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.593+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb19407c110 con 0x7fb19c15ee20 2024-04-17T08:50:27.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:27.813+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb19c0c8a00 con 0x7fb19c15ee20 2024-04-17T08:50:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.809+0000 7fb193e00640 1 -- 172.21.15.67:0/1262074564 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v226) ==== 97+0+0 (secure 0 0 0) 0x7fb19407ba10 con 0x7fb19c15ee20 2024-04-17T08:50:28.809 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.812+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 msgr2=0x7fb17806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.812+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 0x7fb17806eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb18c005880 tx=0x7fb18c009040 comp rx=0 tx=0).stop 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.812+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 msgr2=0x7fb19c0b5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb1940098c0 tx=0x7fb1940046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 shutdown_connections 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb17806ca10 0x7fb17806eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb19c15ee20 0x7fb19c0b5810 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb19c0d1ab0 0x7fb19c15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 --2- 172.21.15.67:0/1262074564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb19c0b9850 0x7fb19c0c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 >> 172.21.15.67:0/1262074564 conn(0x7fb19c06d3c0 msgr2=0x7fb19c15e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 shutdown_connections 2024-04-17T08:50:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:28.813+0000 7fb1a1600640 1 -- 172.21.15.67:0/1262074564 wait complete. 2024-04-17T08:50:28.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:50:29.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.060+0000 7f1081200640 1 Processor -- start 2024-04-17T08:50:29.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.061+0000 7f1081200640 1 -- start start 2024-04-17T08:50:29.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f1081200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c147d60 0x7f107c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f1081200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107c0b3eb0 0x7f107c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f1081200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f107c0aedd0 con 0x7f107c0afb90 2024-04-17T08:50:29.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f1081200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f107c0af0b0 con 0x7f107c0b3eb0 2024-04-17T08:50:29.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f1081200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f107c0af390 con 0x7f107c147d60 2024-04-17T08:50:29.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.062+0000 7f107be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107c0b3eb0 0x7f107c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f107c0afb90 0x7f107c0b37a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32870/0 (socket says 172.21.15.67:32870) 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107aa00640 1 -- 172.21.15.67:0/737798681 learned_addr learned my addr 172.21.15.67:0/737798681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107b400640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c147d60 0x7f107c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107be00640 1 -- 172.21.15.67:0/737798681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c147d60 msgr2=0x7f107c071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107be00640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c147d60 0x7f107c071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107be00640 1 -- 172.21.15.67:0/737798681 >> v1:172.21.15.67:6789/0 conn(0x7f107c0afb90 legacy=0x7f107c0b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.063+0000 7f107be00640 1 -- 172.21.15.67:0/737798681 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c0af670 con 0x7f107c0b3eb0 2024-04-17T08:50:29.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.064+0000 7f107be00640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c0b3eb0 0x7f107c0722a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa0f8d931dfa8c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f107a000640 1 -- 172.21.15.67:0/737798681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074039070 con 0x7f107c0b3eb0 2024-04-17T08:50:29.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f107a000640 1 -- 172.21.15.67:0/737798681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107402cbc0 con 0x7f107c0b3eb0 2024-04-17T08:50:29.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f107a000640 1 -- 172.21.15.67:0/737798681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074034440 con 0x7f107c0b3eb0 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 -- 172.21.15.67:0/737798681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c0b3eb0 msgr2=0x7f107c0722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c0b3eb0 0x7f107c0722a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f107402be10 comp rx=0 tx=0).stop 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 -- 172.21.15.67:0/737798681 shutdown_connections 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c0b3eb0 0x7f107c0722a0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 --2- 172.21.15.67:0/737798681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c147d60 0x7f107c071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.065+0000 7f1081200640 1 -- 172.21.15.67:0/737798681 >> 172.21.15.67:0/737798681 conn(0x7f107c0b5f70 msgr2=0x7f107c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.066+0000 7f1081200640 1 -- 172.21.15.67:0/737798681 shutdown_connections 2024-04-17T08:50:29.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.066+0000 7f1081200640 1 -- 172.21.15.67:0/737798681 wait complete. 2024-04-17T08:50:29.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.067+0000 7f1081200640 1 Processor -- start 2024-04-17T08:50:29.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 -- start start 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 0x7f107c142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c0b3eb0 0x7f107c1e9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f107c1482c0 con 0x7f107c147d60 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f107c148140 con 0x7f107c0afb90 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.068+0000 7f1081200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f107c06dd80 con 0x7f107c0b3eb0 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43946/0 (socket says 172.21.15.67:43946) 2024-04-17T08:50:29.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 -- 172.21.15.67:0/1108149491 learned_addr learned my addr 172.21.15.67:0/1108149491 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107b400640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 0x7f107c142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 -- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c0b3eb0 msgr2=0x7f107c1e9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107aa00640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c0b3eb0 0x7f107c1e9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c0b3eb0 0x7f107c1e9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 -- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 msgr2=0x7f107c142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 0x7f107c142750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107be00640 1 -- 172.21.15.67:0/1108149491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c1ea000 con 0x7f107c147d60 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.069+0000 7f107b400640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 0x7f107c142750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:29.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f107be00640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f1074004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074039050 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1074045070 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107403e830 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107c1f5430 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.070+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107c13fd30 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.071+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107c068a10 con 0x7f107c147d60 2024-04-17T08:50:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.072+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f107403e9d0 con 0x7f107c147d60 2024-04-17T08:50:29.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.073+0000 7f1073400640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 0x7f105c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.073+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4885+0+0 (secure 0 0 0) 0x7f10740ae6a0 con 0x7f107c147d60 2024-04-17T08:50:29.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.074+0000 7f107b400640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 0x7f105c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.075+0000 7f107b400640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 0x7f105c06ef00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f106c006fd0 tx=0x7f106c008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.078+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1074032e20 con 0x7f107c147d60 2024-04-17T08:50:29.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.359+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f107c147230 con 0x7f107c147d60 2024-04-17T08:50:29.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.360+0000 7f1073400640 1 -- 172.21.15.67:0/1108149491 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) ==== 80+0+5304 (secure 0 0 0) 0x7f107407cce0 con 0x7f107c147d60 2024-04-17T08:50:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:29.359 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:50:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:50:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:50:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:50:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:50:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:50:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:50:29.372 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 msgr2=0x7f105c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 0x7f105c06ef00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f106c006fd0 tx=0x7f106c008040 comp rx=0 tx=0).stop 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 msgr2=0x7f107c1f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d90 tx=0x7f1074004a40 comp rx=0 tx=0).stop 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 shutdown_connections 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f105c06ca40 0x7f105c06ef00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c147d60 0x7f107c1f4ef0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107c0b3eb0 0x7f107c1e9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 --2- 172.21.15.67:0/1108149491 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0afb90 0x7f107c142750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 >> 172.21.15.67:0/1108149491 conn(0x7f107c0b5f70 msgr2=0x7f107c0b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 shutdown_connections 2024-04-17T08:50:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.363+0000 7f1081200640 1 -- 172.21.15.67:0/1108149491 wait complete. 2024-04-17T08:50:29.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.601+0000 7f8f83000640 1 Processor -- start 2024-04-17T08:50:29.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.602+0000 7f8f83000640 1 -- start start 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.603+0000 7f8f83000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 0x7f8f7c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.603+0000 7f8f83000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.603+0000 7f8f83000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f7c0d2d30 con 0x7f8f7c15ec60 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.603+0000 7f8f83000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8f7c0d49c0 con 0x7f8f7c0cf910 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.603+0000 7f8f83000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f7c0d3660 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43958/0 (socket says 172.21.15.67:43958) 2024-04-17T08:50:29.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 -- 172.21.15.67:0/2896636609 learned_addr learned my addr 172.21.15.67:0/2896636609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f81200640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 0x7f8f7c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 -- 172.21.15.67:0/2896636609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 msgr2=0x7f8f7c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 0x7f8f7c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 -- 172.21.15.67:0/2896636609 >> v1:172.21.15.187:6789/0 conn(0x7f8f7c15ec60 legacy=0x7f8f7c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f82600640 1 -- 172.21.15.67:0/2896636609 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7c0d3010 con 0x7f8f7c0cf910 2024-04-17T08:50:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.604+0000 7f8f81200640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 0x7f8f7c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.605+0000 7f8f82600640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009e50 tx=0x7f8f7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67912708b6f0546f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.606+0000 7f8f7be00640 1 -- 172.21.15.67:0/2896636609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7402ca80 con 0x7f8f7c0cf910 2024-04-17T08:50:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.606+0000 7f8f7be00640 1 -- 172.21.15.67:0/2896636609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7402cc20 con 0x7f8f7c0cf910 2024-04-17T08:50:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.606+0000 7f8f7be00640 1 -- 172.21.15.67:0/2896636609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f740343f0 con 0x7f8f7c0cf910 2024-04-17T08:50:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 -- 172.21.15.67:0/2896636609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0cf910 msgr2=0x7f8f7c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009e50 tx=0x7f8f7402bee0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 -- 172.21.15.67:0/2896636609 shutdown_connections 2024-04-17T08:50:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c0b7b70 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 --2- 172.21.15.67:0/2896636609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c0b6d00 0x7f8f7c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 -- 172.21.15.67:0/2896636609 >> 172.21.15.67:0/2896636609 conn(0x7f8f7c0d15f0 msgr2=0x7f8f7c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 -- 172.21.15.67:0/2896636609 shutdown_connections 2024-04-17T08:50:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.607+0000 7f8f83000640 1 -- 172.21.15.67:0/2896636609 wait complete. 2024-04-17T08:50:29.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.609+0000 7f8f83000640 1 Processor -- start 2024-04-17T08:50:29.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.609+0000 7f8f83000640 1 -- start start 2024-04-17T08:50:29.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c15ec60 0x7f8f7c154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7c15f040 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7c0cfdc0 con 0x7f8f7c0cf910 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f83000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f7c0d00c0 con 0x7f8f7c15ec60 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f81c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f81c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43968/0 (socket says 172.21.15.67:43968) 2024-04-17T08:50:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.610+0000 7f8f81c00640 1 -- 172.21.15.67:0/4205970248 learned_addr learned my addr 172.21.15.67:0/4205970248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 -- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c15ec60 msgr2=0x7f8f7c154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c15ec60 0x7f8f7c154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 -- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f7c0cf910 msgr2=0x7f8f7c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 -- 172.21.15.67:0/4205970248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7c161b60 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f81c00640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000b3f0 tx=0x7f8f7000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70015020 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f70011830 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70011c20 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f7c160580 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.611+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f7c160b40 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.613+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f7c068a10 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.614+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f8f70018020 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.615+0000 7f8f79600640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 0x7f8f6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:29.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.615+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4885+0+0 (secure 0 0 0) 0x7f8f7008b670 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.616+0000 7f8f81200640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 0x7f8f6006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:29.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.617+0000 7f8f81200640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 0x7f8f6006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c006fd0 tx=0x7f8f6c008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:29.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.620+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8f7001f030 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.825+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f7c15b790 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.827+0000 7f8f79600640 1 -- 172.21.15.67:0/4205970248 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) ==== 74+0+12897 (secure 0 0 0) 0x7f8f700172c0 con 0x7f8f7c0b6d00 2024-04-17T08:50:29.826 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:29.827 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":226,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:28.806194+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":214,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 msgr2=0x7f8f6006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 0x7f8f6006eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c006fd0 tx=0x7f8f6c008040 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 msgr2=0x7f8f7c161620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000b3f0 tx=0x7f8f7000b8b0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 shutdown_connections 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8f6006ca10 0x7f8f6006eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f7c15ec60 0x7f8f7c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f7c0cf910 0x7f8f7c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 --2- 172.21.15.67:0/4205970248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f7c0b6d00 0x7f8f7c161620 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.829+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 >> 172.21.15.67:0/4205970248 conn(0x7f8f7c0d15f0 msgr2=0x7f8f7c0b8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:29.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.830+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 shutdown_connections 2024-04-17T08:50:29.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:29.830+0000 7f8f83000640 1 -- 172.21.15.67:0/4205970248 wait complete. 2024-04-17T08:50:29.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:50:29.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:50:30.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.065+0000 7f5804a00640 1 Processor -- start 2024-04-17T08:50:30.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.066+0000 7f5804a00640 1 -- start start 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.067+0000 7f5804a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f5804a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 0x7f58000d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f5804a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58000d4f70 con 0x7f58000b3b50 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f5804a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f58000d5250 con 0x7f58000cfcd0 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f5804a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f58000cf4d0 con 0x7f580015ec20 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f57ff400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f57ffe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 0x7f58000d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f57ff400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:55598/0 (socket says 172.21.15.67:55598) 2024-04-17T08:50:30.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.068+0000 7f57ff400640 1 -- 172.21.15.67:0/1113745925 learned_addr learned my addr 172.21.15.67:0/1113745925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:30.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.069+0000 7f57ff400640 1 -- 172.21.15.67:0/1113745925 >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 msgr2=0x7f58000d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:30.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.069+0000 7f57ff400640 1 --2- 172.21.15.67:0/1113745925 >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 0x7f58000d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.069+0000 7f57ff400640 1 -- 172.21.15.67:0/1113745925 >> v1:172.21.15.187:6789/0 conn(0x7f58000b3b50 legacy=0x7f580006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.069+0000 7f57ff400640 1 -- 172.21.15.67:0/1113745925 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58000cf7b0 con 0x7f580015ec20 2024-04-17T08:50:30.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.069+0000 7f57ffe00640 1 --2- 172.21.15.67:0/1113745925 >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 0x7f58000d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:50:30.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.071+0000 7f57ff400640 1 --2- 172.21.15.67:0/1113745925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009870 tx=0x7f57e802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e9d07f23477d1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.071+0000 7f57fe000640 1 -- 172.21.15.67:0/1113745925 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e80317c0 con 0x7f580015ec20 2024-04-17T08:50:30.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.071+0000 7f57fe000640 1 -- 172.21.15.67:0/1113745925 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e8037070 con 0x7f580015ec20 2024-04-17T08:50:30.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.071+0000 7f57fe000640 1 -- 172.21.15.67:0/1113745925 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8031b80 con 0x7f580015ec20 2024-04-17T08:50:30.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 -- 172.21.15.67:0/1113745925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 msgr2=0x7f58000d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 --2- 172.21.15.67:0/1113745925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009870 tx=0x7f57e802b860 comp rx=0 tx=0).stop 2024-04-17T08:50:30.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 -- 172.21.15.67:0/1113745925 shutdown_connections 2024-04-17T08:50:30.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 --2- 172.21.15.67:0/1113745925 >> v2:172.21.15.187:3300/0 conn(0x7f58000cfcd0 0x7f58000d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 --2- 172.21.15.67:0/1113745925 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000d22b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 -- 172.21.15.67:0/1113745925 >> 172.21.15.67:0/1113745925 conn(0x7f580008bff0 msgr2=0x7f58000d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 -- 172.21.15.67:0/1113745925 shutdown_connections 2024-04-17T08:50:30.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.072+0000 7f5804a00640 1 -- 172.21.15.67:0/1113745925 wait complete. 2024-04-17T08:50:30.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.074+0000 7f5804a00640 1 Processor -- start 2024-04-17T08:50:30.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.075+0000 7f5804a00640 1 -- start start 2024-04-17T08:50:30.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 0x7f580014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000ba110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f580006d2d0 con 0x7f58000b3b50 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f580006d150 con 0x7f58000cfcd0 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f5804a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f58000d10d0 con 0x7f580015ec20 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f57ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f57ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43982/0 (socket says 172.21.15.67:43982) 2024-04-17T08:50:30.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.076+0000 7f57ff400640 1 -- 172.21.15.67:0/4277400168 learned_addr learned my addr 172.21.15.67:0/4277400168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57fea00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 0x7f580014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 -- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 msgr2=0x7f58000ba110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000ba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 -- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 msgr2=0x7f580014ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 0x7f580014ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 -- 172.21.15.67:0/4277400168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58000cd5d0 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57fea00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 0x7f580014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57ff400640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f57e802b8a0 tx=0x7f57e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.077+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8032650 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.078+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580015d610 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.078+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e80327f0 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.078+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8032be0 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.078+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58000b84a0 con 0x7f58000b3b50 2024-04-17T08:50:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.079+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5800068a10 con 0x7f58000b3b50 2024-04-17T08:50:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.080+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f57e804c060 con 0x7f58000b3b50 2024-04-17T08:50:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.081+0000 7f57f7400640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 0x7f57dc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.081+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4885+0+0 (secure 0 0 0) 0x7f57e80ad070 con 0x7f58000b3b50 2024-04-17T08:50:30.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.082+0000 7f57fea00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 0x7f57dc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.083+0000 7f57fea00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 0x7f57dc06ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58000cce70 tx=0x7f57f0009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.087+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f57e8040070 con 0x7f58000b3b50 2024-04-17T08:50:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.296+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f58000d55f0 con 0x7f58000b3b50 2024-04-17T08:50:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.297+0000 7f57f7400640 1 -- 172.21.15.67:0/4277400168 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v226) ==== 93+0+10 (secure 0 0 0) 0x7f57e807b630 con 0x7f58000b3b50 2024-04-17T08:50:30.297 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:50:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 msgr2=0x7f57dc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 0x7f57dc06ef00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58000cce70 tx=0x7f57f0009040 comp rx=0 tx=0).stop 2024-04-17T08:50:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 msgr2=0x7f58000ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f57e802b8a0 tx=0x7f57e80046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 shutdown_connections 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f57dc06ca40 0x7f57dc06ef00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f580015ec20 0x7f58000ba110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58000cfcd0 0x7f580014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 --2- 172.21.15.67:0/4277400168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000b3b50 0x7f58000ccf40 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 >> 172.21.15.67:0/4277400168 conn(0x7f580008bff0 msgr2=0x7f58000b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.300+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 shutdown_connections 2024-04-17T08:50:30.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.301+0000 7f5804a00640 1 -- 172.21.15.67:0/4277400168 wait complete. 2024-04-17T08:50:30.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:50:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.547+0000 7f6cf3e00640 1 Processor -- start 2024-04-17T08:50:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.547+0000 7f6cf3e00640 1 -- start start 2024-04-17T08:50:30.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf4160280 0x7f6cf406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf40cd4a0 con 0x7f6cf406b8c0 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf41607f0 con 0x7f6cf4141ed0 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf415ce30 con 0x7f6cf4160280 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf4160280 0x7f6cf406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6cf406b8c0 0x7f6cf40cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32872/0 (socket says 172.21.15.67:32872) 2024-04-17T08:50:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.549+0000 7f6cf2a00640 1 -- 172.21.15.67:0/2629621029 learned_addr learned my addr 172.21.15.67:0/2629621029 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.550+0000 7f6cf1600640 1 -- 172.21.15.67:0/2629621029 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493941264 0 0) 0x7f6cf40cd4a0 con 0x7f6cf406b8c0 2024-04-17T08:50:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.550+0000 7f6cf1600640 1 -- 172.21.15.67:0/2629621029 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cd8003520 con 0x7f6cf406b8c0 2024-04-17T08:50:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.550+0000 7f6cf2000640 1 -- 172.21.15.67:0/2629621029 >> v1:172.21.15.187:6789/0 conn(0x7f6cf4141ed0 legacy=0x7f6cf415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.550+0000 7f6cf2000640 1 -- 172.21.15.67:0/2629621029 >> v1:172.21.15.67:6789/0 conn(0x7f6cf406b8c0 legacy=0x7f6cf40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.550+0000 7f6cf2000640 1 -- 172.21.15.67:0/2629621029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf4160ad0 con 0x7f6cf4160280 2024-04-17T08:50:30.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.553+0000 7f6cf2000640 1 --2- 172.21.15.67:0/2629621029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4160280 0x7f6cf406bca0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac0c8cd627c9797 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.553+0000 7f6cf1600640 1 -- 172.21.15.67:0/2629621029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0034070 con 0x7f6cf4160280 2024-04-17T08:50:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.553+0000 7f6cf1600640 1 -- 172.21.15.67:0/2629621029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0030610 con 0x7f6cf4160280 2024-04-17T08:50:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.553+0000 7f6cf1600640 1 -- 172.21.15.67:0/2629621029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00309e0 con 0x7f6cf4160280 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 -- 172.21.15.67:0/2629621029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4160280 msgr2=0x7f6cf406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/2629621029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4160280 0x7f6cf406bca0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b960 comp rx=0 tx=0).stop 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 -- 172.21.15.67:0/2629621029 shutdown_connections 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/2629621029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4160280 0x7f6cf406bca0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 -- 172.21.15.67:0/2629621029 >> 172.21.15.67:0/2629621029 conn(0x7f6cf4067d10 msgr2=0x7f6cf40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 -- 172.21.15.67:0/2629621029 shutdown_connections 2024-04-17T08:50:30.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.554+0000 7f6cf3e00640 1 -- 172.21.15.67:0/2629621029 wait complete. 2024-04-17T08:50:30.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.556+0000 7f6cf3e00640 1 Processor -- start 2024-04-17T08:50:30.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.556+0000 7f6cf3e00640 1 -- start start 2024-04-17T08:50:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf406b8c0 0x7f6cf40b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf40b23e0 0x7f6cf41f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4161090 con 0x7f6cf4141ed0 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf406c1e0 con 0x7f6cf406b8c0 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf413f9b0 con 0x7f6cf40b23e0 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.557+0000 7f6cf2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44006/0 (socket says 172.21.15.67:44006) 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 -- 172.21.15.67:0/3343033477 learned_addr learned my addr 172.21.15.67:0/3343033477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 -- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf40b23e0 msgr2=0x7f6cf41f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf40b23e0 0x7f6cf41f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 -- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf406b8c0 msgr2=0x7f6cf40b13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf406b8c0 0x7f6cf40b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 -- 172.21.15.67:0/3343033477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf4159c70 con 0x7f6cf4141ed0 2024-04-17T08:50:30.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.558+0000 7f6cf2000640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f6ce002bef0 tx=0x7f6ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.559+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0034040 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.559+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0030410 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.559+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0006e10 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.559+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf4159f50 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.559+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf415a510 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.560+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf4068a10 con 0x7f6cf4141ed0 2024-04-17T08:50:30.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.561+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f6ce0037070 con 0x7f6cf4141ed0 2024-04-17T08:50:30.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.562+0000 7f6ceaa00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 0x7f6cd406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:30.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.562+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4885+0+0 (secure 0 0 0) 0x7f6ce00ad510 con 0x7f6cf4141ed0 2024-04-17T08:50:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.563+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 0x7f6cd406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.564+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 0x7f6cd406eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4001610 tx=0x7f6ce4007930 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.566+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6ce007ba40 con 0x7f6cf4141ed0 2024-04-17T08:50:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.778+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6cf406cdb0 con 0x7f6cf4141ed0 2024-04-17T08:50:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.779+0000 7f6ceaa00640 1 -- 172.21.15.67:0/3343033477 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v226) ==== 84+0+65 (secure 0 0 0) 0x7f6ce007b340 con 0x7f6cf4141ed0 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:30.779 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:50:30.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 msgr2=0x7f6cd406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 0x7f6cd406eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4001610 tx=0x7f6ce4007930 comp rx=0 tx=0).stop 2024-04-17T08:50:30.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 msgr2=0x7f6cf41f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:30.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f6ce002bef0 tx=0x7f6ce00046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 shutdown_connections 2024-04-17T08:50:30.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6cd406ca10 0x7f6cd406eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cf40b23e0 0x7f6cf41f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf4141ed0 0x7f6cf41f1480 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3343033477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cf406b8c0 0x7f6cf40b13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.782+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 >> 172.21.15.67:0/3343033477 conn(0x7f6cf4067d10 msgr2=0x7f6cf40b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.783+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 shutdown_connections 2024-04-17T08:50:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:30.783+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3343033477 wait complete. 2024-04-17T08:50:30.803 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:50:30.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:50:31.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.034+0000 7f83d3e00640 1 Processor -- start 2024-04-17T08:50:31.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.034+0000 7f83d3e00640 1 -- start start 2024-04-17T08:50:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83d40b1b50 0x7f83d4160a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f83d40ceab0 0x7f83d40cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d4160f60 con 0x7f83d415fd30 2024-04-17T08:50:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f83d4160110 con 0x7f83d40b1b50 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f83d40d14c0 con 0x7f83d40ceab0 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83d40b1b50 0x7f83d4160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83d40b1b50 0x7f83d4160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44018/0 (socket says 172.21.15.67:44018) 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d2000640 1 -- 172.21.15.67:0/2028893174 learned_addr learned my addr 172.21.15.67:0/2028893174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.036+0000 7f83d3400640 1 --2- 172.21.15.67:0/2028893174 >> v2:172.21.15.187:3300/0 conn(0x7f83d40ceab0 0x7f83d40cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.037+0000 7f83d2000640 1 -- 172.21.15.67:0/2028893174 >> v2:172.21.15.187:3300/0 conn(0x7f83d40ceab0 msgr2=0x7f83d40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.037+0000 7f83d2000640 1 --2- 172.21.15.67:0/2028893174 >> v2:172.21.15.187:3300/0 conn(0x7f83d40ceab0 0x7f83d40cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.037+0000 7f83d2000640 1 -- 172.21.15.67:0/2028893174 >> v1:172.21.15.187:6789/0 conn(0x7f83d415fd30 legacy=0x7f83d4160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.037+0000 7f83d2000640 1 -- 172.21.15.67:0/2028893174 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d40d0c20 con 0x7f83d40b1b50 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.038+0000 7f83d2000640 1 --2- 172.21.15.67:0/2028893174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d4160a20 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f83d4069710 tx=0x7f83bc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5f8affebbff147 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.038+0000 7f83d1600640 1 -- 172.21.15.67:0/2028893174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0317c0 con 0x7f83d40b1b50 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.038+0000 7f83d1600640 1 -- 172.21.15.67:0/2028893174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc037070 con 0x7f83d40b1b50 2024-04-17T08:50:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.038+0000 7f83d1600640 1 -- 172.21.15.67:0/2028893174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc031b80 con 0x7f83d40b1b50 2024-04-17T08:50:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 -- 172.21.15.67:0/2028893174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 msgr2=0x7f83d4160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 --2- 172.21.15.67:0/2028893174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d4160a20 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f83d4069710 tx=0x7f83bc02b860 comp rx=0 tx=0).stop 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 -- 172.21.15.67:0/2028893174 shutdown_connections 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 --2- 172.21.15.67:0/2028893174 >> v2:172.21.15.187:3300/0 conn(0x7f83d40ceab0 0x7f83d40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 --2- 172.21.15.67:0/2028893174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d4160a20 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 -- 172.21.15.67:0/2028893174 >> 172.21.15.67:0/2028893174 conn(0x7f83d40b4000 msgr2=0x7f83d40b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 -- 172.21.15.67:0/2028893174 shutdown_connections 2024-04-17T08:50:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.039+0000 7f83d3e00640 1 -- 172.21.15.67:0/2028893174 wait complete. 2024-04-17T08:50:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.041+0000 7f83d3e00640 1 Processor -- start 2024-04-17T08:50:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.041+0000 7f83d3e00640 1 -- start start 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83d40ceab0 0x7f83d40b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83d415fd30 0x7f83d414f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83d40d11e0 con 0x7f83d40b1b50 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f83d4161200 con 0x7f83d415fd30 2024-04-17T08:50:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f83d40cd020 con 0x7f83d40ceab0 2024-04-17T08:50:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44022/0 (socket says 172.21.15.67:44022) 2024-04-17T08:50:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.042+0000 7f83d2a00640 1 -- 172.21.15.67:0/1849196248 learned_addr learned my addr 172.21.15.67:0/1849196248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d3400640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83d415fd30 0x7f83d414f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 -- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83d40ceab0 msgr2=0x7f83d40b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83d40ceab0 0x7f83d40b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 -- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83d415fd30 msgr2=0x7f83d414f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83d415fd30 0x7f83d414f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 -- 172.21.15.67:0/1849196248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d40b75b0 con 0x7f83d40b1b50 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d2a00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b3f0 tx=0x7f83c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4015020 con 0x7f83d40b1b50 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c4011830 con 0x7f83d40b1b50 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4011c20 con 0x7f83d40b1b50 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d40b7890 con 0x7f83d40b1b50 2024-04-17T08:50:31.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.043+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d40b8130 con 0x7f83d40b1b50 2024-04-17T08:50:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.045+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d4068a10 con 0x7f83d40b1b50 2024-04-17T08:50:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.046+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f83c4018020 con 0x7f83d40b1b50 2024-04-17T08:50:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.046+0000 7f83caa00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 0x7f83b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.047+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(226..226 src has 1..226) ==== 4885+0+0 (secure 0 0 0) 0x7f83c408b6c0 con 0x7f83d40b1b50 2024-04-17T08:50:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.047+0000 7f83d2000640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 0x7f83b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.049+0000 7f83d2000640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 0x7f83b006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83bc036000 tx=0x7f83bc030000 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.051+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f83c401f030 con 0x7f83d40b1b50 2024-04-17T08:50:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.255+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f83d414d670 con 0x7f83d40b1b50 2024-04-17T08:50:31.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-17T08:50:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.849+0000 7f83caa00640 1 -- 172.21.15.67:0/1849196248 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v227) ==== 138+0+0 (secure 0 0 0) 0x7f83d414d670 con 0x7f83d40b1b50 2024-04-17T08:50:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:50:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 msgr2=0x7f83b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 0x7f83b006eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83bc036000 tx=0x7f83bc030000 comp rx=0 tx=0).stop 2024-04-17T08:50:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 msgr2=0x7f83d415eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f83c400b3f0 tx=0x7f83c400b8b0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 shutdown_connections 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f83b006ca10 0x7f83b006eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.852+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83d415fd30 0x7f83d414f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.853+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f83d40ceab0 0x7f83d40b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.853+0000 7f83d3e00640 1 --2- 172.21.15.67:0/1849196248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d40b1b50 0x7f83d415eeb0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.853+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 >> 172.21.15.67:0/1849196248 conn(0x7f83d40b4000 msgr2=0x7f83d40cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.853+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 shutdown_connections 2024-04-17T08:50:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:31.853+0000 7f83d3e00640 1 -- 172.21.15.67:0/1849196248 wait complete. 2024-04-17T08:50:31.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:50:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.098+0000 7f4400400640 1 Processor -- start 2024-04-17T08:50:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.099+0000 7f4400400640 1 -- start start 2024-04-17T08:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f4400400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f80d0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f4400400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43f80b3e20 0x7f43f80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f4400400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f80d1220 con 0x7f43f815da30 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f4400400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f43f80cddc0 con 0x7f43f80b3e20 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f4400400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43f80cdae0 con 0x7f43f80d0900 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f43fd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43f80b3e20 0x7f43f80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f43fe400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43f815da30 0x7f43f80cdff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32888/0 (socket says 172.21.15.67:32888) 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f43fd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43f80b3e20 0x7f43f80cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44026/0 (socket says 172.21.15.67:44026) 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f43fe400640 1 -- 172.21.15.67:0/242019973 learned_addr learned my addr 172.21.15.67:0/242019973 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.101+0000 7f43fda00640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f80d0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.102+0000 7f43fd000640 1 -- 172.21.15.67:0/242019973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 msgr2=0x7f43f80d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.102+0000 7f43fd000640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f80d0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.102+0000 7f43fd000640 1 -- 172.21.15.67:0/242019973 >> v1:172.21.15.67:6789/0 conn(0x7f43f815da30 legacy=0x7f43f80cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.102+0000 7f43fd000640 1 -- 172.21.15.67:0/242019973 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f815cf50 con 0x7f43f80b3e20 2024-04-17T08:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.103+0000 7f43fd000640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f80b3e20 0x7f43f80cd1e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f43e40097a0 tx=0x7f43e402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2707fc745e808ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.103+0000 7f43f7e00640 1 -- 172.21.15.67:0/242019973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4035020 con 0x7f43f80b3e20 2024-04-17T08:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.103+0000 7f43f7e00640 1 -- 172.21.15.67:0/242019973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e4031810 con 0x7f43f80b3e20 2024-04-17T08:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.103+0000 7f43f7e00640 1 -- 172.21.15.67:0/242019973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4031be0 con 0x7f43f80b3e20 2024-04-17T08:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 -- 172.21.15.67:0/242019973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f80b3e20 msgr2=0x7f43f80cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f80b3e20 0x7f43f80cd1e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f43e40097a0 tx=0x7f43e402b8b0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 -- 172.21.15.67:0/242019973 shutdown_connections 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f80b3e20 0x7f43f80cd1e0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 --2- 172.21.15.67:0/242019973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f80d0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 -- 172.21.15.67:0/242019973 >> 172.21.15.67:0/242019973 conn(0x7f43f809aa60 msgr2=0x7f43f80b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 -- 172.21.15.67:0/242019973 shutdown_connections 2024-04-17T08:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.104+0000 7f4400400640 1 -- 172.21.15.67:0/242019973 wait complete. 2024-04-17T08:50:32.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.106+0000 7f4400400640 1 Processor -- start 2024-04-17T08:50:32.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.106+0000 7f4400400640 1 -- start start 2024-04-17T08:50:32.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 0x7f43f8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f8151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43f815a6c0 con 0x7f43f815da30 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43f815d510 con 0x7f43f80b3e20 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f4400400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43f80cd720 con 0x7f43f80d0900 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f43fe400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f43fe400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44040/0 (socket says 172.21.15.67:44040) 2024-04-17T08:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.107+0000 7f43fe400640 1 -- 172.21.15.67:0/306373572 learned_addr learned my addr 172.21.15.67:0/306373572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fd000640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f8151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fda00640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 0x7f43f8159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 -- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 msgr2=0x7f43f8151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f8151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 -- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 msgr2=0x7f43f8159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 0x7f43f8159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 -- 172.21.15.67:0/306373572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f8160c90 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fda00640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 0x7f43f8159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.108+0000 7f43fe400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f43f000cab0 tx=0x7f43f000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.109+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0017070 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.109+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f000ac30 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.109+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0012400 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.109+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f80c7300 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.109+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f80c78c0 con 0x7f43f815da30 2024-04-17T08:50:32.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.110+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f8068ca0 con 0x7f43f815da30 2024-04-17T08:50:32.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.112+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f43f001a070 con 0x7f43f815da30 2024-04-17T08:50:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.112+0000 7f43f5600640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 0x7f43d806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.113+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(227..227 src has 1..227) ==== 5255+0+0 (secure 0 0 0) 0x7f43f008bba0 con 0x7f43f815da30 2024-04-17T08:50:32.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.113+0000 7f43fda00640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 0x7f43d806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:32.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.115+0000 7f43fda00640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 0x7f43d806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43ec006fd0 tx=0x7f43ec008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:32.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.117+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f43f0059b10 con 0x7f43f815da30 2024-04-17T08:50:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.324+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f43f809b190 con 0x7f43f815da30 2024-04-17T08:50:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.867+0000 7f43f5600640 1 -- 172.21.15.67:0/306373572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v228) ==== 170+0+0 (secure 0 0 0) 0x7f43f005de10 con 0x7f43f815da30 2024-04-17T08:50:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:50:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.872+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 msgr2=0x7f43d806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.872+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 0x7f43d806eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f43ec006fd0 tx=0x7f43ec008040 comp rx=0 tx=0).stop 2024-04-17T08:50:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.872+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 msgr2=0x7f43f8160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.872+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f43f000cab0 tx=0x7f43f000cf70 comp rx=0 tx=0).stop 2024-04-17T08:50:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 shutdown_connections 2024-04-17T08:50:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f43d806ca10 0x7f43d806eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43f815da30 0x7f43f8160530 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43f80d0900 0x7f43f8151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 --2- 172.21.15.67:0/306373572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43f80b3e20 0x7f43f8159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:32.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.873+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 >> 172.21.15.67:0/306373572 conn(0x7f43f809aa60 msgr2=0x7f43f8070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:32.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.877+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 shutdown_connections 2024-04-17T08:50:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:32.878+0000 7f4400400640 1 -- 172.21.15.67:0/306373572 wait complete. 2024-04-17T08:50:32.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:50:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.104+0000 7fd99f200640 1 Processor -- start 2024-04-17T08:50:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.105+0000 7fd99f200640 1 -- start start 2024-04-17T08:50:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a015ecb0 0x7fd9a00d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9a00d3210 0x7fd9a00cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 0x7fd9a00d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a00d3670 con 0x7fd9a00d3210 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a00d0c30 con 0x7fd9a00cf8e0 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.106+0000 7fd99f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a00d00b0 con 0x7fd9a015ecb0 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9a00d3210 0x7fd9a00cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9a00d3210 0x7fd9a00cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44042/0 (socket says 172.21.15.67:44042) 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 -- 172.21.15.67:0/1909252466 learned_addr learned my addr 172.21.15.67:0/1909252466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99de00640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a015ecb0 0x7fd9a00d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99e800640 1 --2- 172.21.15.67:0/1909252466 >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 0x7fd9a00d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 -- 172.21.15.67:0/1909252466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a015ecb0 msgr2=0x7fd9a00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a015ecb0 0x7fd9a00d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 -- 172.21.15.67:0/1909252466 >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 msgr2=0x7fd9a00d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 --2- 172.21.15.67:0/1909252466 >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 0x7fd9a00d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99d400640 1 -- 172.21.15.67:0/1909252466 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a00d0390 con 0x7fd9a00d3210 2024-04-17T08:50:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.107+0000 7fd99e800640 1 --2- 172.21.15.67:0/1909252466 >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 0x7fd9a00d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.108+0000 7fd99d400640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a00d3210 0x7fd9a00cf3a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00d2200 tx=0x7fd98802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af717f8dcb30ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.109+0000 7fd99ca00640 1 -- 172.21.15.67:0/1909252466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98802e070 con 0x7fd9a00d3210 2024-04-17T08:50:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.109+0000 7fd99ca00640 1 -- 172.21.15.67:0/1909252466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd988037070 con 0x7fd9a00d3210 2024-04-17T08:50:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.109+0000 7fd99ca00640 1 -- 172.21.15.67:0/1909252466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988031a00 con 0x7fd9a00d3210 2024-04-17T08:50:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 -- 172.21.15.67:0/1909252466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a00d3210 msgr2=0x7fd9a00cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a00d3210 0x7fd9a00cf3a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00d2200 tx=0x7fd98802b880 comp rx=0 tx=0).stop 2024-04-17T08:50:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 -- 172.21.15.67:0/1909252466 shutdown_connections 2024-04-17T08:50:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 --2- 172.21.15.67:0/1909252466 >> v2:172.21.15.187:3300/0 conn(0x7fd9a00cf8e0 0x7fd9a00d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a00d3210 0x7fd9a00cf3a0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 --2- 172.21.15.67:0/1909252466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a015ecb0 0x7fd9a00d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 -- 172.21.15.67:0/1909252466 >> 172.21.15.67:0/1909252466 conn(0x7fd9a008bff0 msgr2=0x7fd9a00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 -- 172.21.15.67:0/1909252466 shutdown_connections 2024-04-17T08:50:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.110+0000 7fd99f200640 1 -- 172.21.15.67:0/1909252466 wait complete. 2024-04-17T08:50:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.111+0000 7fd99f200640 1 Processor -- start 2024-04-17T08:50:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.112+0000 7fd99f200640 1 -- start start 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a00cf8e0 0x7fd9a014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a00d3210 0x7fd9a00b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a00cfd50 con 0x7fd9a015ecb0 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a00d0950 con 0x7fd9a00d3210 2024-04-17T08:50:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a015f090 con 0x7fd9a00cf8e0 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44044/0 (socket says 172.21.15.67:44044) 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 -- 172.21.15.67:0/4129919917 learned_addr learned my addr 172.21.15.67:0/4129919917 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99de00640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a00cf8e0 0x7fd9a014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99d400640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a00d3210 0x7fd9a00b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 -- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a00cf8e0 msgr2=0x7fd9a014cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a00cf8e0 0x7fd9a014cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 -- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a00d3210 msgr2=0x7fd9a00b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a00d3210 0x7fd9a00b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.113+0000 7fd99e800640 1 -- 172.21.15.67:0/4129919917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a014f990 con 0x7fd9a015ecb0 2024-04-17T08:50:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd99e800640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cd10 tx=0x7fd99800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99800f680 con 0x7fd9a015ecb0 2024-04-17T08:50:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99800f840 con 0x7fd9a015ecb0 2024-04-17T08:50:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99800fc30 con 0x7fd9a015ecb0 2024-04-17T08:50:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a014fc70 con 0x7fd9a015ecb0 2024-04-17T08:50:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.114+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a0150230 con 0x7fd9a015ecb0 2024-04-17T08:50:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.116+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a0068a10 con 0x7fd9a015ecb0 2024-04-17T08:50:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.117+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fd99800e3e0 con 0x7fd9a015ecb0 2024-04-17T08:50:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.118+0000 7fd996000640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 0x7fd97c06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.118+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(228..228 src has 1..228) ==== 5637+0+0 (secure 0 0 0) 0x7fd99808cb80 con 0x7fd9a015ecb0 2024-04-17T08:50:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.119+0000 7fd99de00640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 0x7fd97c06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:33.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.120+0000 7fd99de00640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 0x7fd97c06ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd990006fd0 tx=0x7fd990008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.123+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd998014030 con 0x7fd9a015ecb0 2024-04-17T08:50:33.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.369+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd9a00b5a70 con 0x7fd9a015ecb0 2024-04-17T08:50:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.886+0000 7fd996000640 1 -- 172.21.15.67:0/4129919917 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v107) ==== 174+0+0 (secure 0 0 0) 0x7fd99805ad40 con 0x7fd9a015ecb0 2024-04-17T08:50:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 26 and data pool 27 2024-04-17T08:50:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.894+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 msgr2=0x7fd97c06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.894+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 0x7fd97c06ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd990006fd0 tx=0x7fd990008040 comp rx=0 tx=0).stop 2024-04-17T08:50:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.894+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 msgr2=0x7fd9a00b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.894+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd99800cd10 tx=0x7fd99800a650 comp rx=0 tx=0).stop 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 shutdown_connections 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd97c06ca40 0x7fd97c06ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9a015ecb0 0x7fd9a00b2b70 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a00d3210 0x7fd9a00b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 --2- 172.21.15.67:0/4129919917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a00cf8e0 0x7fd9a014cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.895+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 >> 172.21.15.67:0/4129919917 conn(0x7fd9a008bff0 msgr2=0x7fd9a00d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.896+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 shutdown_connections 2024-04-17T08:50:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:33.896+0000 7fd99f200640 1 -- 172.21.15.67:0/4129919917 wait complete. 2024-04-17T08:50:33.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:34.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.112+0000 7fbcf3200640 1 Processor -- start 2024-04-17T08:50:34.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.113+0000 7fbcf3200640 1 -- start start 2024-04-17T08:50:34.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcf406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf3200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbcf4141ed0 0x7fbcf4142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf40cd4a0 con 0x7fbcf406b8c0 2024-04-17T08:50:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf3200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf415ce30 con 0x7fbcf4141ed0 2024-04-17T08:50:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf41607f0 con 0x7fbcf4160280 2024-04-17T08:50:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcf406b8c0 0x7fbcf40cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47918/0 (socket says 172.21.15.67:47918) 2024-04-17T08:50:34.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.114+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1871029720 learned_addr learned my addr 172.21.15.67:0/1871029720 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.115+0000 7fbce9400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcf406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.115+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1871029720 >> v2:172.21.15.187:3300/0 conn(0x7fbcf4141ed0 0x7fbcf4142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.115+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3589099838 0 0) 0x7fbcf40cd4a0 con 0x7fbcf406b8c0 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.115+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd8003520 con 0x7fbcf406b8c0 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4016098927 0 0) 0x7fbcd8003520 con 0x7fbcf406b8c0 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 msgr2=0x7fbcf406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcf406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 >> v2:172.21.15.187:3300/0 conn(0x7fbcf4141ed0 msgr2=0x7fbcf4142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1871029720 >> v2:172.21.15.187:3300/0 conn(0x7fbcf4141ed0 0x7fbcf4142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf4160ad0 con 0x7fbcf406b8c0 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbce4001670 con 0x7fbcf406b8c0 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 >> v1:172.21.15.67:6789/0 conn(0x7fbcf406b8c0 legacy=0x7fbcf40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8003a20 0x7fbcd8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8004d40 0x7fbcd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf415ce30 con 0x7fbcd8003a20 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf41607f0 con 0x7fbcd8004d40 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd800fdd0 con 0x7fbcf4160280 2024-04-17T08:50:34.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8003a20 0x7fbcd8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbce9400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.116+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8004d40 0x7fbcd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 msgr2=0x7fbcd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8004d40 msgr2=0x7fbcd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8004d40 0x7fbcd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1871029720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd8003520 con 0x7fbcd8003a20 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8004d40 0x7fbcd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.117+0000 7fbce9400640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf4160280 0x7fbcd8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.118+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8003a20 0x7fbcd8003e40 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c190 tx=0x7fbcd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e23711d9921cfb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.118+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4007c80 con 0x7fbcd8003a20 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.118+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce4030070 con 0x7fbcd8003a20 2024-04-17T08:50:34.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.118+0000 7fbcf1400640 1 -- 172.21.15.67:0/1871029720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4005d30 con 0x7fbcd8003a20 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 -- 172.21.15.67:0/1871029720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8003a20 msgr2=0x7fbcd8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1871029720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8003a20 0x7fbcd8003e40 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c190 tx=0x7fbcd8005830 comp rx=0 tx=0).stop 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf2800640 1 -- 172.21.15.67:0/1871029720 reap_dead start 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 -- 172.21.15.67:0/1871029720 shutdown_connections 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 -- 172.21.15.67:0/1871029720 >> 172.21.15.67:0/1871029720 conn(0x7fbcf4067d10 msgr2=0x7fbcf40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 -- 172.21.15.67:0/1871029720 shutdown_connections 2024-04-17T08:50:34.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.119+0000 7fbcf3200640 1 -- 172.21.15.67:0/1871029720 wait complete. 2024-04-17T08:50:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.121+0000 7fbcf3200640 1 Processor -- start 2024-04-17T08:50:34.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.121+0000 7fbcf3200640 1 -- start start 2024-04-17T08:50:34.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 0x7fbcf4144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf40b1870 0x7fbcf415e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf413f9b0 con 0x7fbcf4142fc0 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4161090 con 0x7fbcd8003a20 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf413fb30 con 0x7fbcf40b1870 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbce9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbce9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52696/0 (socket says 172.21.15.67:52696) 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbce9400640 1 -- 172.21.15.67:0/1420512082 learned_addr learned my addr 172.21.15.67:0/1420512082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:34.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.122+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 0x7fbcf4144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 -- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf40b1870 msgr2=0x7fbcf415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf40b1870 0x7fbcf415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 -- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 msgr2=0x7fbcf4144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 0x7fbcf4144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 -- 172.21.15.67:0/1420512082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf4143810 con 0x7fbcf4142fc0 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 0x7fbcf4144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbce9400640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0108b0 tx=0x7fbcdc010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc00f040 con 0x7fbcf4142fc0 2024-04-17T08:50:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc00eae0 con 0x7fbcf4142fc0 2024-04-17T08:50:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc00ee90 con 0x7fbcf4142fc0 2024-04-17T08:50:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.123+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf4143af0 con 0x7fbcf4142fc0 2024-04-17T08:50:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.124+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf413d1d0 con 0x7fbcf4142fc0 2024-04-17T08:50:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.125+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf4068a10 con 0x7fbcf4142fc0 2024-04-17T08:50:34.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.126+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fbcdc01c400 con 0x7fbcf4142fc0 2024-04-17T08:50:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.127+0000 7fbceaa00640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 0x7fbcd006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.127+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 5737+0+0 (secure 0 0 0) 0x7fbcdc08c440 con 0x7fbcf4142fc0 2024-04-17T08:50:34.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.128+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 0x7fbcd006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.129+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 0x7fbcd006f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4007000 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.132+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbcdc018030 con 0x7fbcf4142fc0 2024-04-17T08:50:34.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.341+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbcf40b3520 con 0x7fbcf4142fc0 2024-04-17T08:50:34.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.342+0000 7fbceaa00640 1 -- 172.21.15.67:0/1420512082 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) ==== 74+0+16692 (secure 0 0 0) 0x7fbcdc05a730 con 0x7fbcf4142fc0 2024-04-17T08:50:34.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:34.342 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":229,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:33.371190+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 msgr2=0x7fbcd006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 0x7fbcd006f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4007000 comp rx=0 tx=0).stop 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 msgr2=0x7fbcf40b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0108b0 tx=0x7fbcdc010d70 comp rx=0 tx=0).stop 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 shutdown_connections 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fbcd006cbb0 0x7fbcd006f070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcf40b1870 0x7fbcf415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4142fc0 0x7fbcf40b1330 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 --2- 172.21.15.67:0/1420512082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd8003a20 0x7fbcf4144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.345+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 >> 172.21.15.67:0/1420512082 conn(0x7fbcf4067d10 msgr2=0x7fbcf406e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:34.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.346+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 shutdown_connections 2024-04-17T08:50:34.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.346+0000 7fbcf3200640 1 -- 172.21.15.67:0/1420512082 wait complete. 2024-04-17T08:50:34.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:34.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.580+0000 7f8703e00640 1 Processor -- start 2024-04-17T08:50:34.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.580+0000 7f8703e00640 1 -- start start 2024-04-17T08:50:34.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.581+0000 7f8703e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.581+0000 7f8703e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 0x7f8704142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.581+0000 7f8703e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87040cd4a0 con 0x7f870406b8c0 2024-04-17T08:50:34.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.581+0000 7f8703e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f870415ce30 con 0x7f8704160280 2024-04-17T08:50:34.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.581+0000 7f8703e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f87041607f0 con 0x7f8704141ed0 2024-04-17T08:50:34.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.582+0000 7f8702000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.582+0000 7f8703400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 0x7f8704142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.582+0000 7f8702000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:60918/0 (socket says 172.21.15.67:60918) 2024-04-17T08:50:34.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.582+0000 7f8702000640 1 -- 172.21.15.67:0/3266469886 learned_addr learned my addr 172.21.15.67:0/3266469886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:34.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.583+0000 7f8703400640 1 -- 172.21.15.67:0/3266469886 >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 msgr2=0x7f870406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.583+0000 7f8703400640 1 --2- 172.21.15.67:0/3266469886 >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.583+0000 7f8703400640 1 -- 172.21.15.67:0/3266469886 >> v1:172.21.15.187:6789/0 conn(0x7f870406b8c0 legacy=0x7f87040cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:34.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.583+0000 7f8703400640 1 -- 172.21.15.67:0/3266469886 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8704160ad0 con 0x7f8704141ed0 2024-04-17T08:50:34.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.583+0000 7f8702000640 1 --2- 172.21.15.67:0/3266469886 >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:50:34.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.584+0000 7f8703400640 1 --2- 172.21.15.67:0/3266469886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 0x7f8704142380 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097a0 tx=0x7f86fc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2fb6c11a52a7af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.585+0000 7f8701600640 1 -- 172.21.15.67:0/3266469886 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc0315a0 con 0x7f8704141ed0 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8701600640 1 -- 172.21.15.67:0/3266469886 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86fc037070 con 0x7f8704141ed0 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8701600640 1 -- 172.21.15.67:0/3266469886 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc031960 con 0x7f8704141ed0 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 -- 172.21.15.67:0/3266469886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 msgr2=0x7f8704142380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 --2- 172.21.15.67:0/3266469886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 0x7f8704142380 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097a0 tx=0x7f86fc02b840 comp rx=0 tx=0).stop 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 -- 172.21.15.67:0/3266469886 shutdown_connections 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 --2- 172.21.15.67:0/3266469886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8704141ed0 0x7f8704142380 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 --2- 172.21.15.67:0/3266469886 >> v2:172.21.15.187:3300/0 conn(0x7f8704160280 0x7f870406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 -- 172.21.15.67:0/3266469886 >> 172.21.15.67:0/3266469886 conn(0x7f8704067d10 msgr2=0x7f87040b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 -- 172.21.15.67:0/3266469886 shutdown_connections 2024-04-17T08:50:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.586+0000 7f8703e00640 1 -- 172.21.15.67:0/3266469886 wait complete. 2024-04-17T08:50:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.588+0000 7f8703e00640 1 Processor -- start 2024-04-17T08:50:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.589+0000 7f8703e00640 1 -- start start 2024-04-17T08:50:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f870406b8c0 0x7f8704155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 0x7f8704145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8704161090 con 0x7f8704141ed0 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f870406c1e0 con 0x7f8704160280 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f870413f9b0 con 0x7f870406b8c0 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52706/0 (socket says 172.21.15.67:52706) 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 -- 172.21.15.67:0/1790430003 learned_addr learned my addr 172.21.15.67:0/1790430003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702a00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f870406b8c0 0x7f8704155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703400640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 0x7f8704145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 -- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f870406b8c0 msgr2=0x7f8704155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f870406b8c0 0x7f8704155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 -- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 msgr2=0x7f8704145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 0x7f8704145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8702000640 1 -- 172.21.15.67:0/1790430003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87040b3a20 con 0x7f8704141ed0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.590+0000 7f8703400640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 0x7f8704145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f8702000640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e9b0 tx=0x7f86ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec017070 con 0x7f8704141ed0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86ec00c9a0 con 0x7f8704141ed0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00cd50 con 0x7f8704141ed0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87040b3d00 con 0x7f8704141ed0 2024-04-17T08:50:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.591+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87040b42c0 con 0x7f8704141ed0 2024-04-17T08:50:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.592+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8704068a10 con 0x7f8704141ed0 2024-04-17T08:50:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.593+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f86ec01a070 con 0x7f8704141ed0 2024-04-17T08:50:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.594+0000 7f86faa00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 0x7f86e006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.594+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 5737+0+0 (secure 0 0 0) 0x7f86ec08bfe0 con 0x7f8704141ed0 2024-04-17T08:50:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.595+0000 7f8702a00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 0x7f86e006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.596+0000 7f8702a00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 0x7f86e006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86f4005520 tx=0x7f86f4009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:34.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.599+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f86ec05a280 con 0x7f8704141ed0 2024-04-17T08:50:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.805+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87040b0d60 con 0x7f8704141ed0 2024-04-17T08:50:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.808+0000 7f86faa00640 1 -- 172.21.15.67:0/1790430003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) ==== 74+0+16692 (secure 0 0 0) 0x7f86ec059b80 con 0x7f8704141ed0 2024-04-17T08:50:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:34.808 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":229,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:33.371190+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:34.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 msgr2=0x7f86e006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 0x7f86e006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86f4005520 tx=0x7f86f4009040 comp rx=0 tx=0).stop 2024-04-17T08:50:34.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 msgr2=0x7f87041455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e9b0 tx=0x7f86ec00ee70 comp rx=0 tx=0).stop 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 shutdown_connections 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f86e006ca10 0x7f86e006eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8704160280 0x7f8704145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8704141ed0 0x7f87041455b0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 --2- 172.21.15.67:0/1790430003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f870406b8c0 0x7f8704155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 >> 172.21.15.67:0/1790430003 conn(0x7f8704067d10 msgr2=0x7f87040cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 shutdown_connections 2024-04-17T08:50:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:34.811+0000 7f8703e00640 1 -- 172.21.15.67:0/1790430003 wait complete. 2024-04-17T08:50:34.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:50:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.035+0000 7fa527000640 1 Processor -- start 2024-04-17T08:50:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.035+0000 7fa527000640 1 -- start start 2024-04-17T08:50:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa527000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 0x7fa52006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa527000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa52006c0a0 0x7fa52006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa527000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa520146360 con 0x7fa520145f80 2024-04-17T08:50:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa527000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa5200727a0 con 0x7fa52006c0a0 2024-04-17T08:50:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa527000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa520071e70 con 0x7fa5200cd920 2024-04-17T08:50:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa525c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa520145f80 0x7fa5200cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47924/0 (socket says 172.21.15.67:47924) 2024-04-17T08:50:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.037+0000 7fa525c00640 1 -- 172.21.15.67:0/957599156 learned_addr learned my addr 172.21.15.67:0/957599156 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.038+0000 7fa526600640 1 --2- 172.21.15.67:0/957599156 >> v2:172.21.15.187:3300/0 conn(0x7fa52006c0a0 0x7fa52006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.038+0000 7fa525200640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 0x7fa52006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.038+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 623688188 0 0) 0x7fa520146360 con 0x7fa520145f80 2024-04-17T08:50:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.038+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa508003520 con 0x7fa520145f80 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3740539601 0 0) 0x7fa508003520 con 0x7fa520145f80 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 msgr2=0x7fa52006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 0x7fa52006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 >> v2:172.21.15.187:3300/0 conn(0x7fa52006c0a0 msgr2=0x7fa52006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 --2- 172.21.15.67:0/957599156 >> v2:172.21.15.187:3300/0 conn(0x7fa52006c0a0 0x7fa52006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa520072150 con 0x7fa520145f80 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa514001670 con 0x7fa520145f80 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 >> v1:172.21.15.67:6789/0 conn(0x7fa520145f80 legacy=0x7fa5200cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 0x7fa5080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa508003f30 0x7fa508004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508004d40 0x7fa5080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5200727a0 con 0x7fa508003f30 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa520071e70 con 0x7fa508004d40 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa50800fe20 con 0x7fa5200cd920 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa508003f30 0x7fa508004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 -- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 msgr2=0x7fa5080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5200cd920 0x7fa5080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 -- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508004d40 msgr2=0x7fa5080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508004d40 0x7fa5080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.039+0000 7fa525200640 1 -- 172.21.15.67:0/957599156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa508003520 con 0x7fa508003f30 2024-04-17T08:50:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.040+0000 7fa525200640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa508003f30 0x7fa508004800 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009af0 tx=0x7fa508005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83020793e878fd0f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c039070 con 0x7fa508003f30 2024-04-17T08:50:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa50c02c710 con 0x7fa508003f30 2024-04-17T08:50:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa51fe00640 1 -- 172.21.15.67:0/957599156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c009690 con 0x7fa508003f30 2024-04-17T08:50:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa527000640 1 -- 172.21.15.67:0/957599156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa508003f30 msgr2=0x7fa508004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa527000640 1 --2- 172.21.15.67:0/957599156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa508003f30 0x7fa508004800 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fa50c009af0 tx=0x7fa508005830 comp rx=0 tx=0).stop 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa526600640 1 -- 172.21.15.67:0/957599156 reap_dead start 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa527000640 1 -- 172.21.15.67:0/957599156 shutdown_connections 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa527000640 1 -- 172.21.15.67:0/957599156 >> 172.21.15.67:0/957599156 conn(0x7fa5200b2190 msgr2=0x7fa5200b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.041+0000 7fa527000640 1 -- 172.21.15.67:0/957599156 shutdown_connections 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.042+0000 7fa527000640 1 -- 172.21.15.67:0/957599156 wait complete. 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.043+0000 7fa527000640 1 Processor -- start 2024-04-17T08:50:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.043+0000 7fa527000640 1 -- start start 2024-04-17T08:50:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508003f30 0x7fa5200ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa520141d80 0x7fa520142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa520072db0 con 0x7fa5201404e0 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa520072c30 con 0x7fa508003f30 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa527000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa52006c4a0 con 0x7fa520141d80 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa526600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa526600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52722/0 (socket says 172.21.15.67:52722) 2024-04-17T08:50:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.044+0000 7fa526600640 1 -- 172.21.15.67:0/2774250894 learned_addr learned my addr 172.21.15.67:0/2774250894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 -- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa520141d80 msgr2=0x7fa520142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa520141d80 0x7fa520142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 -- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508003f30 msgr2=0x7fa5200ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508003f30 0x7fa5200ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 -- 172.21.15.67:0/2774250894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5200ae4f0 con 0x7fa5201404e0 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa526600640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fa5200d1470 tx=0x7fa51800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51800bae0 con 0x7fa5201404e0 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa51801a070 con 0x7fa5201404e0 2024-04-17T08:50:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518014400 con 0x7fa5201404e0 2024-04-17T08:50:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.045+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5200ad6b0 con 0x7fa5201404e0 2024-04-17T08:50:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.046+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5200adc50 con 0x7fa5201404e0 2024-04-17T08:50:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.047+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa520068a10 con 0x7fa5201404e0 2024-04-17T08:50:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.048+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fa518005d10 con 0x7fa5201404e0 2024-04-17T08:50:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.048+0000 7fa51d600640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 0x7fa50006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.048+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fa51808bef0 con 0x7fa5201404e0 2024-04-17T08:50:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.049+0000 7fa525c00640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 0x7fa50006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.050+0000 7fa525c00640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 0x7fa50006ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f60 tx=0x7fa5140015c0 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.054+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa5180122c0 con 0x7fa5201404e0 2024-04-17T08:50:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.301+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa5200cf630 con 0x7fa5201404e0 2024-04-17T08:50:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.890+0000 7fa51d600640 1 -- 172.21.15.67:0/2774250894 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v109) ==== 118+0+0 (secure 0 0 0) 0x7fa51805a040 con 0x7fa5201404e0 2024-04-17T08:50:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.893+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 msgr2=0x7fa50006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.893+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 0x7fa50006ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa514000f60 tx=0x7fa5140015c0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.893+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 msgr2=0x7fa520140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.893+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fa5200d1470 tx=0x7fa51800b480 comp rx=0 tx=0).stop 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 shutdown_connections 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa50006ca40 0x7fa50006ef00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5201404e0 0x7fa520140990 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa520141d80 0x7fa520142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 --2- 172.21.15.67:0/2774250894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508003f30 0x7fa5200ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 >> 172.21.15.67:0/2774250894 conn(0x7fa5200b2190 msgr2=0x7fa5200b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 shutdown_connections 2024-04-17T08:50:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:35.894+0000 7fa527000640 1 -- 172.21.15.67:0/2774250894 wait complete. 2024-04-17T08:50:35.915 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:50:35.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:50:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.133+0000 7f15b5e00640 1 Processor -- start 2024-04-17T08:50:36.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.133+0000 7f15b5e00640 1 -- start start 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.134+0000 7f15b5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f15b00d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15b5e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b015d800 con 0x7f15b0160c30 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15b5e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b015dac0 con 0x7f15b0161010 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15b5e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15b006b8c0 con 0x7f15b00d0980 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15aea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15b0160c30 0x7f15b015cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47930/0 (socket says 172.21.15.67:47930) 2024-04-17T08:50:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15aea00640 1 -- 172.21.15.67:0/105155430 learned_addr learned my addr 172.21.15.67:0/105155430 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.135+0000 7f15af400640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f15b00d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994575470 0 0) 0x7f15b015d800 con 0x7f15b0160c30 2024-04-17T08:50:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1594003520 con 0x7f15b0160c30 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3255270676 0 0) 0x7f1594003520 con 0x7f15b0160c30 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 msgr2=0x7f15b00d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f15b00d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 >> v1:172.21.15.187:6789/0 conn(0x7f15b0161010 legacy=0x7f15b006c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b015dda0 con 0x7f15b0160c30 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f15a0001670 con 0x7f15b0160c30 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 >> v1:172.21.15.67:6789/0 conn(0x7f15b0160c30 legacy=0x7f15b015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15940039e0 0x7f1594003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f1594004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1594004d40 0x7f15940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15b006b8c0 con 0x7f1594004d40 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f159400ba70 con 0x7f15940039e0 2024-04-17T08:50:36.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.136+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f159400bc70 con 0x7f15b00d0980 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1594004d40 0x7f15940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15aea00640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15940039e0 0x7f1594003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15ae000640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f1594004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 -- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 msgr2=0x7f1594004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f1594004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 -- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15940039e0 msgr2=0x7f1594003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15940039e0 0x7f1594003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15af400640 1 -- 172.21.15.67:0/105155430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1594003520 con 0x7f1594004d40 2024-04-17T08:50:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.137+0000 7f15ae000640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00d0980 0x7f1594004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.138+0000 7f15af400640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1594004d40 0x7f15940051c0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009f90 tx=0x7f1594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbaa4c12e461a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.138+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8039070 con 0x7f1594004d40 2024-04-17T08:50:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a802cbd0 con 0x7f1594004d40 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15ad600640 1 -- 172.21.15.67:0/105155430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009c10 con 0x7f1594004d40 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 -- 172.21.15.67:0/105155430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1594004d40 msgr2=0x7f15940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 --2- 172.21.15.67:0/105155430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1594004d40 0x7f15940051c0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009f90 tx=0x7f1594005830 comp rx=0 tx=0).stop 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15af400640 1 -- 172.21.15.67:0/105155430 reap_dead start 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 -- 172.21.15.67:0/105155430 shutdown_connections 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 -- 172.21.15.67:0/105155430 >> 172.21.15.67:0/105155430 conn(0x7f15b0073360 msgr2=0x7f15b009aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 -- 172.21.15.67:0/105155430 shutdown_connections 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.139+0000 7f15b5e00640 1 -- 172.21.15.67:0/105155430 wait complete. 2024-04-17T08:50:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.141+0000 7f15b5e00640 1 Processor -- start 2024-04-17T08:50:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.141+0000 7f15b5e00640 1 -- start start 2024-04-17T08:50:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1594004d40 0x7f15b00c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00c74a0 0x7f15b00c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01613f0 con 0x7f15b01f2a90 2024-04-17T08:50:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15b006bba0 con 0x7f1594004d40 2024-04-17T08:50:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.142+0000 7f15b5e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15b00d0e10 con 0x7f15b00c74a0 2024-04-17T08:50:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52744/0 (socket says 172.21.15.67:52744) 2024-04-17T08:50:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 -- 172.21.15.67:0/311564679 learned_addr learned my addr 172.21.15.67:0/311564679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:36.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15aea00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1594004d40 0x7f15b00c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15af400640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00c74a0 0x7f15b00c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 -- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00c74a0 msgr2=0x7f15b00c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00c74a0 0x7f15b00c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 -- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1594004d40 msgr2=0x7f15b00c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1594004d40 0x7f15b00c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 -- 172.21.15.67:0/311564679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b00c6230 con 0x7f15b01f2a90 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.143+0000 7f15ae000640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d7a0 tx=0x7f159c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.144+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c005070 con 0x7f15b01f2a90 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.144+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f159c005210 con 0x7f15b01f2a90 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.144+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c007030 con 0x7f15b01f2a90 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.144+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b00c6510 con 0x7f15b01f2a90 2024-04-17T08:50:36.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.144+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b0159120 con 0x7f15b01f2a90 2024-04-17T08:50:36.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.145+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b0068ca0 con 0x7f15b01f2a90 2024-04-17T08:50:36.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.146+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f159c01a020 con 0x7f15b01f2a90 2024-04-17T08:50:36.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.147+0000 7f15a7400640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 0x7f159006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:36.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.147+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f159c08b600 con 0x7f15b01f2a90 2024-04-17T08:50:36.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.148+0000 7f15aea00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 0x7f159006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:36.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.149+0000 7f15aea00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 0x7f159006ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a0001cd0 tx=0x7f15a0007040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:36.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.152+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f159c0591a0 con 0x7f15b01f2a90 2024-04-17T08:50:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.399+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f15b0152190 con 0x7f15b01f2a90 2024-04-17T08:50:36.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.904+0000 7f15a7400640 1 -- 172.21.15.67:0/311564679 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v110) ==== 122+0+0 (secure 0 0 0) 0x7f159c05d4a0 con 0x7f15b01f2a90 2024-04-17T08:50:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.907+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 msgr2=0x7f159006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.907+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 0x7f159006ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a0001cd0 tx=0x7f15a0007040 comp rx=0 tx=0).stop 2024-04-17T08:50:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.907+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 msgr2=0x7f15b00b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.907+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d7a0 tx=0x7f159c00dc60 comp rx=0 tx=0).stop 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 shutdown_connections 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f159006ca40 0x7f159006ef00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15b00c74a0 0x7f15b00c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15b01f2a90 0x7f15b00b4370 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 --2- 172.21.15.67:0/311564679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1594004d40 0x7f15b00c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 >> 172.21.15.67:0/311564679 conn(0x7f15b0073360 msgr2=0x7f15b00d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 shutdown_connections 2024-04-17T08:50:36.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:36.908+0000 7f15b5e00640 1 -- 172.21.15.67:0/311564679 wait complete. 2024-04-17T08:50:36.931 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:50:36.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:37.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.156+0000 7f8a4ca00640 1 Processor -- start 2024-04-17T08:50:37.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.156+0000 7f8a4ca00640 1 -- start start 2024-04-17T08:50:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a4806f670 0x7f8a4806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a4ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a480cd020 0x7f8a480cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a4ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4806ff90 con 0x7f8a48071d10 2024-04-17T08:50:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a4ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8a480ceb00 con 0x7f8a480cd020 2024-04-17T08:50:37.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480cede0 con 0x7f8a4806f670 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a45400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a480cd020 0x7f8a480cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a46800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a48071d10 0x7f8a480721c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47942/0 (socket says 172.21.15.67:47942) 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.158+0000 7f8a46800640 1 -- 172.21.15.67:0/2116357953 learned_addr learned my addr 172.21.15.67:0/2116357953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a45e00640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a4806f670 0x7f8a4806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a45400640 1 -- 172.21.15.67:0/2116357953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a4806f670 msgr2=0x7f8a4806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:37.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a45400640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a4806f670 0x7f8a4806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a45400640 1 -- 172.21.15.67:0/2116357953 >> v1:172.21.15.67:6789/0 conn(0x7f8a48071d10 legacy=0x7f8a480721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a45400640 1 -- 172.21.15.67:0/2116357953 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a480cf0c0 con 0x7f8a480cd020 2024-04-17T08:50:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.159+0000 7f8a44a00640 1 -- 172.21.15.67:0/2116357953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240317145 0 0) 0x7f8a4806ff90 con 0x7f8a48071d10 2024-04-17T08:50:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.160+0000 7f8a45400640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a480cd480 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f8a340097a0 tx=0x7f8a3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65d83081118c728 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.160+0000 7f8a44a00640 1 -- 172.21.15.67:0/2116357953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34035020 con 0x7f8a480cd020 2024-04-17T08:50:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.160+0000 7f8a44a00640 1 -- 172.21.15.67:0/2116357953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a34031830 con 0x7f8a480cd020 2024-04-17T08:50:37.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.160+0000 7f8a44a00640 1 -- 172.21.15.67:0/2116357953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34031c00 con 0x7f8a480cd020 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 -- 172.21.15.67:0/2116357953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 msgr2=0x7f8a480cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a480cd480 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f8a340097a0 tx=0x7f8a3402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 -- 172.21.15.67:0/2116357953 shutdown_connections 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a480cd480 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/2116357953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a4806f670 0x7f8a4806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 -- 172.21.15.67:0/2116357953 >> 172.21.15.67:0/2116357953 conn(0x7f8a4806bf50 msgr2=0x7f8a480b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 -- 172.21.15.67:0/2116357953 shutdown_connections 2024-04-17T08:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.161+0000 7f8a4ca00640 1 -- 172.21.15.67:0/2116357953 wait complete. 2024-04-17T08:50:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.163+0000 7f8a4ca00640 1 Processor -- start 2024-04-17T08:50:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.163+0000 7f8a4ca00640 1 -- start start 2024-04-17T08:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 0x7f8a480c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a48071d10 0x7f8a480c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480ce720 con 0x7f8a480cd020 2024-04-17T08:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480cf680 con 0x7f8a4806f670 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a4ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480ce8a0 con 0x7f8a48071d10 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.164+0000 7f8a46800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52772/0 (socket says 172.21.15.67:52772) 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 -- 172.21.15.67:0/1151785992 learned_addr learned my addr 172.21.15.67:0/1151785992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a45400640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a48071d10 0x7f8a480c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a45e00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 0x7f8a480c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 -- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a48071d10 msgr2=0x7f8a480c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a48071d10 0x7f8a480c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 -- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 msgr2=0x7f8a480c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 0x7f8a480c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 -- 172.21.15.67:0/1151785992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a480c9fe0 con 0x7f8a480cd020 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a45e00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 0x7f8a480c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.165+0000 7f8a46800640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c7a0 tx=0x7f8a4000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.166+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40018070 con 0x7f8a480cd020 2024-04-17T08:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.166+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a4000a640 con 0x7f8a480cd020 2024-04-17T08:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.166+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4000aa70 con 0x7f8a480cd020 2024-04-17T08:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.166+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a480c85f0 con 0x7f8a480cd020 2024-04-17T08:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.166+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a480c8bb0 con 0x7f8a480cd020 2024-04-17T08:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.167+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a48068b90 con 0x7f8a480cd020 2024-04-17T08:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.168+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f8a4000ac10 con 0x7f8a480cd020 2024-04-17T08:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.169+0000 7f8a3e000640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 0x7f8a2806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.169+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f8a4008b3b0 con 0x7f8a480cd020 2024-04-17T08:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.170+0000 7f8a45e00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 0x7f8a2806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.171+0000 7f8a45e00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 0x7f8a2806eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006fd0 tx=0x7f8a38008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.173+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8a400596a0 con 0x7f8a480cd020 2024-04-17T08:50:37.278 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T08:50:37.279+0000 7fb05a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:50:37.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.419+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a4806ccd0 con 0x7f8a480cd020 2024-04-17T08:50:37.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.421+0000 7f8a3e000640 1 -- 172.21.15.67:0/1151785992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5221 (secure 0 0 0) 0x7f8a40058fa0 con 0x7f8a480cd020 2024-04-17T08:50:37.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:37.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11025,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13}]} 2024-04-17T08:50:37.421 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 110 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.423+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 msgr2=0x7f8a2806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.423+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 0x7f8a2806eed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006fd0 tx=0x7f8a38008040 comp rx=0 tx=0).stop 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.423+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 msgr2=0x7f8a4814ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.423+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c7a0 tx=0x7f8a4000cc60 comp rx=0 tx=0).stop 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 shutdown_connections 2024-04-17T08:50:37.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8a2806ca10 0x7f8a2806eed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a480cd020 0x7f8a4814ef90 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a48071d10 0x7f8a480c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 --2- 172.21.15.67:0/1151785992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a4806f670 0x7f8a480c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 >> 172.21.15.67:0/1151785992 conn(0x7f8a4806bf50 msgr2=0x7f8a48074eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 shutdown_connections 2024-04-17T08:50:37.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.424+0000 7f8a4ca00640 1 -- 172.21.15.67:0/1151785992 wait complete. 2024-04-17T08:50:37.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.667+0000 7f6b93200640 1 Processor -- start 2024-04-17T08:50:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.667+0000 7f6b93200640 1 -- start start 2024-04-17T08:50:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b93200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b9415d530 0x7f6b94164480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b93200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b940b3490 0x7f6b9406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b93200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9406ead0 con 0x7f6b940b3910 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b93200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9406f6e0 con 0x7f6b940b3490 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b93200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9406edb0 con 0x7f6b9415d530 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b8be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b940b3490 0x7f6b9406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b92800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b940b3910 0x7f6b941634e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47944/0 (socket says 172.21.15.67:47944) 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b92800640 1 -- 172.21.15.67:0/1161660337 learned_addr learned my addr 172.21.15.67:0/1161660337 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b8be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b940b3490 0x7f6b9406b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52774/0 (socket says 172.21.15.67:52774) 2024-04-17T08:50:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.669+0000 7f6b91e00640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b9415d530 0x7f6b94164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.670+0000 7f6b8be00640 1 -- 172.21.15.67:0/1161660337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b9415d530 msgr2=0x7f6b94164480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.670+0000 7f6b8be00640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b9415d530 0x7f6b94164480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.670+0000 7f6b8be00640 1 -- 172.21.15.67:0/1161660337 >> v1:172.21.15.67:6789/0 conn(0x7f6b940b3910 legacy=0x7f6b941634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.670+0000 7f6b8be00640 1 -- 172.21.15.67:0/1161660337 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9406f090 con 0x7f6b940b3490 2024-04-17T08:50:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.671+0000 7f6b8be00640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b940b3490 0x7f6b9406b8c0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097a0 tx=0x7f6b7c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b1e311cea0efc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.671+0000 7f6b91400640 1 -- 172.21.15.67:0/1161660337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c035020 con 0x7f6b940b3490 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.671+0000 7f6b91400640 1 -- 172.21.15.67:0/1161660337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c031810 con 0x7f6b940b3490 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.671+0000 7f6b91400640 1 -- 172.21.15.67:0/1161660337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c031be0 con 0x7f6b940b3490 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 -- 172.21.15.67:0/1161660337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b940b3490 msgr2=0x7f6b9406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b940b3490 0x7f6b9406b8c0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097a0 tx=0x7f6b7c02b8b0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 -- 172.21.15.67:0/1161660337 shutdown_connections 2024-04-17T08:50:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b940b3490 0x7f6b9406b8c0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 --2- 172.21.15.67:0/1161660337 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b9415d530 0x7f6b94164480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 -- 172.21.15.67:0/1161660337 >> 172.21.15.67:0/1161660337 conn(0x7f6b94067d10 msgr2=0x7f6b940b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 -- 172.21.15.67:0/1161660337 shutdown_connections 2024-04-17T08:50:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.672+0000 7f6b93200640 1 -- 172.21.15.67:0/1161660337 wait complete. 2024-04-17T08:50:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.674+0000 7f6b93200640 1 Processor -- start 2024-04-17T08:50:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.674+0000 7f6b93200640 1 -- start start 2024-04-17T08:50:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b940b3490 0x7f6b94163650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940b3910 0x7f6b941572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9406fee0 con 0x7f6b9415d530 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9406fd60 con 0x7f6b940b3910 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b93200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9406be00 con 0x7f6b940b3490 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52786/0 (socket says 172.21.15.67:52786) 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.675+0000 7f6b92800640 1 -- 172.21.15.67:0/3099866710 learned_addr learned my addr 172.21.15.67:0/3099866710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b91e00640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b940b3490 0x7f6b94163650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b8be00640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940b3910 0x7f6b941572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 -- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b940b3490 msgr2=0x7f6b94163650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b940b3490 0x7f6b94163650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 -- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940b3910 msgr2=0x7f6b941572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940b3910 0x7f6b941572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 -- 172.21.15.67:0/3099866710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b94159640 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b92800640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00cbc0 tx=0x7f6b8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c017070 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c00ad40 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.676+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c012600 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.677+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b940d0ba0 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.677+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b94159900 con 0x7f6b9415d530 2024-04-17T08:50:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.678+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b94068a10 con 0x7f6b9415d530 2024-04-17T08:50:37.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.679+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f6b8c01a070 con 0x7f6b9415d530 2024-04-17T08:50:37.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.680+0000 7f6b8a000640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 0x7f6b7006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:37.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.680+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f6b8c08ba40 con 0x7f6b9415d530 2024-04-17T08:50:37.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.681+0000 7f6b91e00640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 0x7f6b7006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:37.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.682+0000 7f6b91e00640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 0x7f6b7006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6b84006fd0 tx=0x7f6b84008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:37.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.685+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6b8c059650 con 0x7f6b9415d530 2024-04-17T08:50:37.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.896+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b94159b40 con 0x7f6b9415d530 2024-04-17T08:50:37.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.899+0000 7f6b8a000640 1 -- 172.21.15.67:0/3099866710 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) ==== 74+0+16679 (secure 0 0 0) 0x7f6b8c05d950 con 0x7f6b9415d530 2024-04-17T08:50:37.899 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:37.899 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":230,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:34.879148+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:37.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.901+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 msgr2=0x7f6b7006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.901+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 0x7f6b7006eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6b84006fd0 tx=0x7f6b84008040 comp rx=0 tx=0).stop 2024-04-17T08:50:37.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.901+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 msgr2=0x7f6b941577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:37.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.901+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00cbc0 tx=0x7f6b8c00a430 comp rx=0 tx=0).stop 2024-04-17T08:50:37.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 shutdown_connections 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f6b7006ca10 0x7f6b7006eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9415d530 0x7f6b941577f0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940b3910 0x7f6b941572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 --2- 172.21.15.67:0/3099866710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b940b3490 0x7f6b94163650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 >> 172.21.15.67:0/3099866710 conn(0x7f6b94067d10 msgr2=0x7f6b940b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 shutdown_connections 2024-04-17T08:50:37.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:37.902+0000 7f6b93200640 1 -- 172.21.15.67:0/3099866710 wait complete. 2024-04-17T08:50:37.924 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:50:37.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:50:38.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.147+0000 7f5f85400640 1 Processor -- start 2024-04-17T08:50:38.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.147+0000 7f5f85400640 1 -- start start 2024-04-17T08:50:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.148+0000 7f5f85400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.148+0000 7f5f85400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f800d2ab0 0x7f5f800b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.148+0000 7f5f85400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f800cff60 con 0x7f5f8015ed10 2024-04-17T08:50:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.148+0000 7f5f85400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5f800d3c30 con 0x7f5f800d2ab0 2024-04-17T08:50:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.148+0000 7f5f85400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800d3020 con 0x7f5f800cf640 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5f8015ed10 0x7f5f800d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47952/0 (socket says 172.21.15.67:47952) 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7de00640 1 -- 172.21.15.67:0/4223157860 learned_addr learned my addr 172.21.15.67:0/4223157860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7f200640 1 --2- 172.21.15.67:0/4223157860 >> v2:172.21.15.67:3300/0 conn(0x7f5f800d2ab0 0x7f5f800b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7e800640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7d400640 1 -- 172.21.15.67:0/4223157860 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401662611 0 0) 0x7f5f800cff60 con 0x7f5f8015ed10 2024-04-17T08:50:38.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7d400640 1 -- 172.21.15.67:0/4223157860 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f64003520 con 0x7f5f8015ed10 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7f200640 1 -- 172.21.15.67:0/4223157860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 msgr2=0x7f5f800cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7f200640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7f200640 1 -- 172.21.15.67:0/4223157860 >> v1:172.21.15.67:6789/0 conn(0x7f5f8015ed10 legacy=0x7f5f800d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.149+0000 7f5f7f200640 1 -- 172.21.15.67:0/4223157860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f800d3300 con 0x7f5f800d2ab0 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.151+0000 7f5f7f200640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b7b10 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d96432c3208e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.151+0000 7f5f7d400640 1 -- 172.21.15.67:0/4223157860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78039070 con 0x7f5f800d2ab0 2024-04-17T08:50:38.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.151+0000 7f5f7d400640 1 -- 172.21.15.67:0/4223157860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7802cce0 con 0x7f5f800d2ab0 2024-04-17T08:50:38.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.151+0000 7f5f7d400640 1 -- 172.21.15.67:0/4223157860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034640 con 0x7f5f800d2ab0 2024-04-17T08:50:38.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 -- 172.21.15.67:0/4223157860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 msgr2=0x7f5f800b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b7b10 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802bf30 comp rx=0 tx=0).stop 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 -- 172.21.15.67:0/4223157860 shutdown_connections 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b7b10 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 --2- 172.21.15.67:0/4223157860 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 -- 172.21.15.67:0/4223157860 >> 172.21.15.67:0/4223157860 conn(0x7f5f8006cbd0 msgr2=0x7f5f8006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 -- 172.21.15.67:0/4223157860 shutdown_connections 2024-04-17T08:50:38.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.152+0000 7f5f85400640 1 -- 172.21.15.67:0/4223157860 wait complete. 2024-04-17T08:50:38.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.154+0000 7f5f85400640 1 Processor -- start 2024-04-17T08:50:38.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.154+0000 7f5f85400640 1 -- start start 2024-04-17T08:50:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f8015ed10 0x7f5f80158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d25b0 con 0x7f5f800d2ab0 2024-04-17T08:50:38.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d38c0 con 0x7f5f8015ed10 2024-04-17T08:50:38.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f85400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800ba300 con 0x7f5f800cf640 2024-04-17T08:50:38.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52812/0 (socket says 172.21.15.67:52812) 2024-04-17T08:50:38.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 -- 172.21.15.67:0/470236772 learned_addr learned my addr 172.21.15.67:0/470236772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 -- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 msgr2=0x7f5f800b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 -- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f8015ed10 msgr2=0x7f5f80158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f8015ed10 0x7f5f80158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.155+0000 7f5f7de00640 1 -- 172.21.15.67:0/470236772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f800b6570 con 0x7f5f800d2ab0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f7de00640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f5f68001eb0 tx=0x7f5f6800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68017070 con 0x7f5f800d2ab0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f68005d00 con 0x7f5f800d2ab0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68007310 con 0x7f5f800d2ab0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f800b6850 con 0x7f5f800d2ab0 2024-04-17T08:50:38.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.156+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f800b6e10 con 0x7f5f800d2ab0 2024-04-17T08:50:38.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.157+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f80068b90 con 0x7f5f800d2ab0 2024-04-17T08:50:38.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.158+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f5f6801a070 con 0x7f5f800d2ab0 2024-04-17T08:50:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.159+0000 7f5f76a00640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 0x7f5f5c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.160+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f5f6808b160 con 0x7f5f800d2ab0 2024-04-17T08:50:38.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.160+0000 7f5f7e800640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 0x7f5f5c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.161+0000 7f5f7e800640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 0x7f5f5c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006a90 tx=0x7f5f70006a20 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.164+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5f680593c0 con 0x7f5f800d2ab0 2024-04-17T08:50:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.367+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5f800b40b0 con 0x7f5f5c06ca10 2024-04-17T08:50:38.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.390+0000 7f5f76a00640 1 -- 172.21.15.67:0/470236772 <== mgr.7679 v2:172.21.15.187:6832/150239855 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252431 (secure 0 0 0) 0x7f5f800b40b0 con 0x7f5f5c06ca10 2024-04-17T08:50:38.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:50:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 msgr2=0x7f5f5c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 0x7f5f5c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006a90 tx=0x7f5f70006a20 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 msgr2=0x7f5f800b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f5f68001eb0 tx=0x7f5f6800c820 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 shutdown_connections 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f5f5c06ca10 0x7f5f5c06eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f8015ed10 0x7f5f80158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f800d2ab0 0x7f5f800b5270 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 --2- 172.21.15.67:0/470236772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f800cf640 0x7f5f800b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 >> 172.21.15.67:0/470236772 conn(0x7f5f8006cbd0 msgr2=0x7f5f800b9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 shutdown_connections 2024-04-17T08:50:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.396+0000 7f5f85400640 1 -- 172.21.15.67:0/470236772 wait complete. 2024-04-17T08:50:38.417 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-17T08:50:36.420690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":631,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":392,"ondisk_log_size":392,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":410420,"kb_used_data":23540,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754564300,"statfs":{"total":773094113280,"available":772673843200,"internally_reserved":0,"allocated":24104960,"data_stored":12364252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":396021296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.568358"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894449+0000","last_change":"2024-04-17T08:50:33.892568+0000","last_active":"2024-04-17T08:50:33.894449+0000","last_peered":"2024-04-17T08:50:33.894449+0000","last_clean":"2024-04-17T08:50:33.894449+0000","last_became_active":"2024-04-17T08:50:33.892228+0000","last_became_peered":"2024-04-17T08:50:33.892228+0000","last_unstale":"2024-04-17T08:50:33.894449+0000","last_undegraded":"2024-04-17T08:50:33.894449+0000","last_fullsized":"2024-04-17T08:50:33.894449+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:29.148122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.893489+0000","last_change":"2024-04-17T08:50:33.883908+0000","last_active":"2024-04-17T08:50:33.893489+0000","last_peered":"2024-04-17T08:50:33.893489+0000","last_clean":"2024-04-17T08:50:33.893489+0000","last_became_active":"2024-04-17T08:50:33.883336+0000","last_became_peered":"2024-04-17T08:50:33.883336+0000","last_unstale":"2024-04-17T08:50:33.893489+0000","last_undegraded":"2024-04-17T08:50:33.893489+0000","last_fullsized":"2024-04-17T08:50:33.893489+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:47.164759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906392+0000","last_change":"2024-04-17T08:50:33.900448+0000","last_active":"2024-04-17T08:50:33.906392+0000","last_peered":"2024-04-17T08:50:33.906392+0000","last_clean":"2024-04-17T08:50:33.906392+0000","last_became_active":"2024-04-17T08:50:33.900142+0000","last_became_peered":"2024-04-17T08:50:33.900142+0000","last_unstale":"2024-04-17T08:50:33.906392+0000","last_undegraded":"2024-04-17T08:50:33.906392+0000","last_fullsized":"2024-04-17T08:50:33.906392+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:13.541875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898864+0000","last_change":"2024-04-17T08:50:33.896077+0000","last_active":"2024-04-17T08:50:33.898864+0000","last_peered":"2024-04-17T08:50:33.898864+0000","last_clean":"2024-04-17T08:50:33.898864+0000","last_became_active":"2024-04-17T08:50:33.895408+0000","last_became_peered":"2024-04-17T08:50:33.895408+0000","last_unstale":"2024-04-17T08:50:33.898864+0000","last_undegraded":"2024-04-17T08:50:33.898864+0000","last_fullsized":"2024-04-17T08:50:33.898864+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:04.665259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911516+0000","last_change":"2024-04-17T08:50:33.908217+0000","last_active":"2024-04-17T08:50:33.911516+0000","last_peered":"2024-04-17T08:50:33.911516+0000","last_clean":"2024-04-17T08:50:33.911516+0000","last_became_active":"2024-04-17T08:50:33.907185+0000","last_became_peered":"2024-04-17T08:50:33.907185+0000","last_unstale":"2024-04-17T08:50:33.911516+0000","last_undegraded":"2024-04-17T08:50:33.911516+0000","last_fullsized":"2024-04-17T08:50:33.911516+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:40:56.232264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.903581+0000","last_change":"2024-04-17T08:50:33.900220+0000","last_active":"2024-04-17T08:50:33.903581+0000","last_peered":"2024-04-17T08:50:33.903581+0000","last_clean":"2024-04-17T08:50:33.903581+0000","last_became_active":"2024-04-17T08:50:33.898746+0000","last_became_peered":"2024-04-17T08:50:33.898746+0000","last_unstale":"2024-04-17T08:50:33.903581+0000","last_undegraded":"2024-04-17T08:50:33.903581+0000","last_fullsized":"2024-04-17T08:50:33.903581+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:17.253778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.890589+0000","last_change":"2024-04-17T08:50:33.880798+0000","last_active":"2024-04-17T08:50:33.890589+0000","last_peered":"2024-04-17T08:50:33.890589+0000","last_clean":"2024-04-17T08:50:33.890589+0000","last_became_active":"2024-04-17T08:50:33.879829+0000","last_became_peered":"2024-04-17T08:50:33.879829+0000","last_unstale":"2024-04-17T08:50:33.890589+0000","last_undegraded":"2024-04-17T08:50:33.890589+0000","last_fullsized":"2024-04-17T08:50:33.890589+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:14.610912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896792+0000","last_change":"2024-04-17T08:50:33.895419+0000","last_active":"2024-04-17T08:50:33.896792+0000","last_peered":"2024-04-17T08:50:33.896792+0000","last_clean":"2024-04-17T08:50:33.896792+0000","last_became_active":"2024-04-17T08:50:33.894089+0000","last_became_peered":"2024-04-17T08:50:33.894089+0000","last_unstale":"2024-04-17T08:50:33.896792+0000","last_undegraded":"2024-04-17T08:50:33.896792+0000","last_fullsized":"2024-04-17T08:50:33.896792+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:19.879655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897381+0000","last_change":"2024-04-17T08:50:33.894702+0000","last_active":"2024-04-17T08:50:33.897381+0000","last_peered":"2024-04-17T08:50:33.897381+0000","last_clean":"2024-04-17T08:50:33.897381+0000","last_became_active":"2024-04-17T08:50:33.894398+0000","last_became_peered":"2024-04-17T08:50:33.894398+0000","last_unstale":"2024-04-17T08:50:33.897381+0000","last_undegraded":"2024-04-17T08:50:33.897381+0000","last_fullsized":"2024-04-17T08:50:33.897381+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:15.124518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910127+0000","last_change":"2024-04-17T08:50:33.895456+0000","last_active":"2024-04-17T08:50:33.910127+0000","last_peered":"2024-04-17T08:50:33.910127+0000","last_clean":"2024-04-17T08:50:33.910127+0000","last_became_active":"2024-04-17T08:50:33.895142+0000","last_became_peered":"2024-04-17T08:50:33.895142+0000","last_unstale":"2024-04-17T08:50:33.910127+0000","last_undegraded":"2024-04-17T08:50:33.910127+0000","last_fullsized":"2024-04-17T08:50:33.910127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:35.646922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904098+0000","last_change":"2024-04-17T08:50:33.897057+0000","last_active":"2024-04-17T08:50:33.904098+0000","last_peered":"2024-04-17T08:50:33.904098+0000","last_clean":"2024-04-17T08:50:33.904098+0000","last_became_active":"2024-04-17T08:50:33.894722+0000","last_became_peered":"2024-04-17T08:50:33.894722+0000","last_unstale":"2024-04-17T08:50:33.904098+0000","last_undegraded":"2024-04-17T08:50:33.904098+0000","last_fullsized":"2024-04-17T08:50:33.904098+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:27.999528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901918+0000","last_change":"2024-04-17T08:50:33.897953+0000","last_active":"2024-04-17T08:50:33.901918+0000","last_peered":"2024-04-17T08:50:33.901918+0000","last_clean":"2024-04-17T08:50:33.901918+0000","last_became_active":"2024-04-17T08:50:33.897591+0000","last_became_peered":"2024-04-17T08:50:33.897591+0000","last_unstale":"2024-04-17T08:50:33.901918+0000","last_undegraded":"2024-04-17T08:50:33.901918+0000","last_fullsized":"2024-04-17T08:50:33.901918+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:08.803343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.900067+0000","last_change":"2024-04-17T08:50:33.894398+0000","last_active":"2024-04-17T08:50:33.900067+0000","last_peered":"2024-04-17T08:50:33.900067+0000","last_clean":"2024-04-17T08:50:33.900067+0000","last_became_active":"2024-04-17T08:50:33.894105+0000","last_became_peered":"2024-04-17T08:50:33.894105+0000","last_unstale":"2024-04-17T08:50:33.900067+0000","last_undegraded":"2024-04-17T08:50:33.900067+0000","last_fullsized":"2024-04-17T08:50:33.900067+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:08.859357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892403+0000","last_change":"2024-04-17T08:50:33.887226+0000","last_active":"2024-04-17T08:50:33.892403+0000","last_peered":"2024-04-17T08:50:33.892403+0000","last_clean":"2024-04-17T08:50:33.892403+0000","last_became_active":"2024-04-17T08:50:33.886912+0000","last_became_peered":"2024-04-17T08:50:33.886912+0000","last_unstale":"2024-04-17T08:50:33.892403+0000","last_undegraded":"2024-04-17T08:50:33.892403+0000","last_fullsized":"2024-04-17T08:50:33.892403+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:36.180669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901135+0000","last_change":"2024-04-17T08:50:33.898186+0000","last_active":"2024-04-17T08:50:33.901135+0000","last_peered":"2024-04-17T08:50:33.901135+0000","last_clean":"2024-04-17T08:50:33.901135+0000","last_became_active":"2024-04-17T08:50:33.897746+0000","last_became_peered":"2024-04-17T08:50:33.897746+0000","last_unstale":"2024-04-17T08:50:33.901135+0000","last_undegraded":"2024-04-17T08:50:33.901135+0000","last_fullsized":"2024-04-17T08:50:33.901135+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:52.169578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910192+0000","last_change":"2024-04-17T08:50:33.895366+0000","last_active":"2024-04-17T08:50:33.910192+0000","last_peered":"2024-04-17T08:50:33.910192+0000","last_clean":"2024-04-17T08:50:33.910192+0000","last_became_active":"2024-04-17T08:50:33.892102+0000","last_became_peered":"2024-04-17T08:50:33.892102+0000","last_unstale":"2024-04-17T08:50:33.910192+0000","last_undegraded":"2024-04-17T08:50:33.910192+0000","last_fullsized":"2024-04-17T08:50:33.910192+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:28.242561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892444+0000","last_change":"2024-04-17T08:50:33.888622+0000","last_active":"2024-04-17T08:50:33.892444+0000","last_peered":"2024-04-17T08:50:33.892444+0000","last_clean":"2024-04-17T08:50:33.892444+0000","last_became_active":"2024-04-17T08:50:33.888298+0000","last_became_peered":"2024-04-17T08:50:33.888298+0000","last_unstale":"2024-04-17T08:50:33.892444+0000","last_undegraded":"2024-04-17T08:50:33.892444+0000","last_fullsized":"2024-04-17T08:50:33.892444+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:59.627816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911857+0000","last_change":"2024-04-17T08:50:33.905082+0000","last_active":"2024-04-17T08:50:33.911857+0000","last_peered":"2024-04-17T08:50:33.911857+0000","last_clean":"2024-04-17T08:50:33.911857+0000","last_became_active":"2024-04-17T08:50:33.904723+0000","last_became_peered":"2024-04-17T08:50:33.904723+0000","last_unstale":"2024-04-17T08:50:33.911857+0000","last_undegraded":"2024-04-17T08:50:33.911857+0000","last_fullsized":"2024-04-17T08:50:33.911857+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:47.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.913597+0000","last_change":"2024-04-17T08:50:33.907077+0000","last_active":"2024-04-17T08:50:33.913597+0000","last_peered":"2024-04-17T08:50:33.913597+0000","last_clean":"2024-04-17T08:50:33.913597+0000","last_became_active":"2024-04-17T08:50:33.906771+0000","last_became_peered":"2024-04-17T08:50:33.906771+0000","last_unstale":"2024-04-17T08:50:33.913597+0000","last_undegraded":"2024-04-17T08:50:33.913597+0000","last_fullsized":"2024-04-17T08:50:33.913597+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:14.899784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911180+0000","last_change":"2024-04-17T08:50:33.907557+0000","last_active":"2024-04-17T08:50:33.911180+0000","last_peered":"2024-04-17T08:50:33.911180+0000","last_clean":"2024-04-17T08:50:33.911180+0000","last_became_active":"2024-04-17T08:50:33.907258+0000","last_became_peered":"2024-04-17T08:50:33.907258+0000","last_unstale":"2024-04-17T08:50:33.911180+0000","last_undegraded":"2024-04-17T08:50:33.911180+0000","last_fullsized":"2024-04-17T08:50:33.911180+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:26.641470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.905091+0000","last_change":"2024-04-17T08:50:33.904610+0000","last_active":"2024-04-17T08:50:33.905091+0000","last_peered":"2024-04-17T08:50:33.905091+0000","last_clean":"2024-04-17T08:50:33.905091+0000","last_became_active":"2024-04-17T08:50:33.904287+0000","last_became_peered":"2024-04-17T08:50:33.904287+0000","last_unstale":"2024-04-17T08:50:33.905091+0000","last_undegraded":"2024-04-17T08:50:33.905091+0000","last_fullsized":"2024-04-17T08:50:33.905091+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:02.874584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906484+0000","last_change":"2024-04-17T08:50:33.905971+0000","last_active":"2024-04-17T08:50:33.906484+0000","last_peered":"2024-04-17T08:50:33.906484+0000","last_clean":"2024-04-17T08:50:33.906484+0000","last_became_active":"2024-04-17T08:50:33.905591+0000","last_became_peered":"2024-04-17T08:50:33.905591+0000","last_unstale":"2024-04-17T08:50:33.906484+0000","last_undegraded":"2024-04-17T08:50:33.906484+0000","last_fullsized":"2024-04-17T08:50:33.906484+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:46.730352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.891914+0000","last_change":"2024-04-17T08:50:33.890185+0000","last_active":"2024-04-17T08:50:33.891914+0000","last_peered":"2024-04-17T08:50:33.891914+0000","last_clean":"2024-04-17T08:50:33.891914+0000","last_became_active":"2024-04-17T08:50:33.889133+0000","last_became_peered":"2024-04-17T08:50:33.889133+0000","last_unstale":"2024-04-17T08:50:33.891914+0000","last_undegraded":"2024-04-17T08:50:33.891914+0000","last_fullsized":"2024-04-17T08:50:33.891914+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:21.688586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901783+0000","last_change":"2024-04-17T08:50:33.900938+0000","last_active":"2024-04-17T08:50:33.901783+0000","last_peered":"2024-04-17T08:50:33.901783+0000","last_clean":"2024-04-17T08:50:33.901783+0000","last_became_active":"2024-04-17T08:50:33.900085+0000","last_became_peered":"2024-04-17T08:50:33.900085+0000","last_unstale":"2024-04-17T08:50:33.901783+0000","last_undegraded":"2024-04-17T08:50:33.901783+0000","last_fullsized":"2024-04-17T08:50:33.901783+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:56.712749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897352+0000","last_change":"2024-04-17T08:50:33.893295+0000","last_active":"2024-04-17T08:50:33.897352+0000","last_peered":"2024-04-17T08:50:33.897352+0000","last_clean":"2024-04-17T08:50:33.897352+0000","last_became_active":"2024-04-17T08:50:33.892999+0000","last_became_peered":"2024-04-17T08:50:33.892999+0000","last_unstale":"2024-04-17T08:50:33.897352+0000","last_undegraded":"2024-04-17T08:50:33.897352+0000","last_fullsized":"2024-04-17T08:50:33.897352+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:51.603438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886137+0000","last_change":"2024-04-17T08:50:33.879179+0000","last_active":"2024-04-17T08:50:33.886137+0000","last_peered":"2024-04-17T08:50:33.886137+0000","last_clean":"2024-04-17T08:50:33.886137+0000","last_became_active":"2024-04-17T08:50:33.878412+0000","last_became_peered":"2024-04-17T08:50:33.878412+0000","last_unstale":"2024-04-17T08:50:33.886137+0000","last_undegraded":"2024-04-17T08:50:33.886137+0000","last_fullsized":"2024-04-17T08:50:33.886137+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:15.166595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901627+0000","last_change":"2024-04-17T08:50:33.897365+0000","last_active":"2024-04-17T08:50:33.901627+0000","last_peered":"2024-04-17T08:50:33.901627+0000","last_clean":"2024-04-17T08:50:33.901627+0000","last_became_active":"2024-04-17T08:50:33.897012+0000","last_became_peered":"2024-04-17T08:50:33.897012+0000","last_unstale":"2024-04-17T08:50:33.901627+0000","last_undegraded":"2024-04-17T08:50:33.901627+0000","last_fullsized":"2024-04-17T08:50:33.901627+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:33:16.415969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906022+0000","last_change":"2024-04-17T08:50:33.897631+0000","last_active":"2024-04-17T08:50:33.906022+0000","last_peered":"2024-04-17T08:50:33.906022+0000","last_clean":"2024-04-17T08:50:33.906022+0000","last_became_active":"2024-04-17T08:50:33.897328+0000","last_became_peered":"2024-04-17T08:50:33.897328+0000","last_unstale":"2024-04-17T08:50:33.906022+0000","last_undegraded":"2024-04-17T08:50:33.906022+0000","last_fullsized":"2024-04-17T08:50:33.906022+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:18:26.586972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.902255+0000","last_change":"2024-04-17T08:50:33.899268+0000","last_active":"2024-04-17T08:50:33.902255+0000","last_peered":"2024-04-17T08:50:33.902255+0000","last_clean":"2024-04-17T08:50:33.902255+0000","last_became_active":"2024-04-17T08:50:33.898733+0000","last_became_peered":"2024-04-17T08:50:33.898733+0000","last_unstale":"2024-04-17T08:50:33.902255+0000","last_undegraded":"2024-04-17T08:50:33.902255+0000","last_fullsized":"2024-04-17T08:50:33.902255+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:42.459510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897415+0000","last_change":"2024-04-17T08:50:33.895564+0000","last_active":"2024-04-17T08:50:33.897415+0000","last_peered":"2024-04-17T08:50:33.897415+0000","last_clean":"2024-04-17T08:50:33.897415+0000","last_became_active":"2024-04-17T08:50:33.893775+0000","last_became_peered":"2024-04-17T08:50:33.893775+0000","last_unstale":"2024-04-17T08:50:33.897415+0000","last_undegraded":"2024-04-17T08:50:33.897415+0000","last_fullsized":"2024-04-17T08:50:33.897415+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:53.858430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904060+0000","last_change":"2024-04-17T08:50:33.890500+0000","last_active":"2024-04-17T08:50:33.904060+0000","last_peered":"2024-04-17T08:50:33.904060+0000","last_clean":"2024-04-17T08:50:33.904060+0000","last_became_active":"2024-04-17T08:50:33.890183+0000","last_became_peered":"2024-04-17T08:50:33.890183+0000","last_unstale":"2024-04-17T08:50:33.904060+0000","last_undegraded":"2024-04-17T08:50:33.904060+0000","last_fullsized":"2024-04-17T08:50:33.904060+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:57.852009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901387+0000","last_change":"2024-04-17T08:50:33.897001+0000","last_active":"2024-04-17T08:50:33.901387+0000","last_peered":"2024-04-17T08:50:33.901387+0000","last_clean":"2024-04-17T08:50:33.901387+0000","last_became_active":"2024-04-17T08:50:33.896545+0000","last_became_peered":"2024-04-17T08:50:33.896545+0000","last_unstale":"2024-04-17T08:50:33.901387+0000","last_undegraded":"2024-04-17T08:50:33.901387+0000","last_fullsized":"2024-04-17T08:50:33.901387+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:40.333703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899286+0000","last_change":"2024-04-17T08:50:33.897991+0000","last_active":"2024-04-17T08:50:33.899286+0000","last_peered":"2024-04-17T08:50:33.899286+0000","last_clean":"2024-04-17T08:50:33.899286+0000","last_became_active":"2024-04-17T08:50:33.897605+0000","last_became_peered":"2024-04-17T08:50:33.897605+0000","last_unstale":"2024-04-17T08:50:33.899286+0000","last_undegraded":"2024-04-17T08:50:33.899286+0000","last_fullsized":"2024-04-17T08:50:33.899286+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:35.168211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"229'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.918131+0000","last_change":"2024-04-17T08:50:32.901470+0000","last_active":"2024-04-17T08:50:33.918131+0000","last_peered":"2024-04-17T08:50:33.918131+0000","last_clean":"2024-04-17T08:50:33.918131+0000","last_became_active":"2024-04-17T08:50:32.900718+0000","last_became_peered":"2024-04-17T08:50:32.900718+0000","last_unstale":"2024-04-17T08:50:33.918131+0000","last_undegraded":"2024-04-17T08:50:33.918131+0000","last_fullsized":"2024-04-17T08:50:33.918131+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:32.091268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898276+0000","last_change":"2024-04-17T08:50:33.895246+0000","last_active":"2024-04-17T08:50:33.898276+0000","last_peered":"2024-04-17T08:50:33.898276+0000","last_clean":"2024-04-17T08:50:33.898276+0000","last_became_active":"2024-04-17T08:50:33.891622+0000","last_became_peered":"2024-04-17T08:50:33.891622+0000","last_unstale":"2024-04-17T08:50:33.898276+0000","last_undegraded":"2024-04-17T08:50:33.898276+0000","last_fullsized":"2024-04-17T08:50:33.898276+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:23.927426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919238+0000","last_change":"2024-04-17T08:50:32.904503+0000","last_active":"2024-04-17T08:50:33.919238+0000","last_peered":"2024-04-17T08:50:33.919238+0000","last_clean":"2024-04-17T08:50:33.919238+0000","last_became_active":"2024-04-17T08:50:32.904221+0000","last_became_peered":"2024-04-17T08:50:32.904221+0000","last_unstale":"2024-04-17T08:50:33.919238+0000","last_undegraded":"2024-04-17T08:50:33.919238+0000","last_fullsized":"2024-04-17T08:50:33.919238+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:54.730453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.887255+0000","last_change":"2024-04-17T08:50:33.882411+0000","last_active":"2024-04-17T08:50:33.887255+0000","last_peered":"2024-04-17T08:50:33.887255+0000","last_clean":"2024-04-17T08:50:33.887255+0000","last_became_active":"2024-04-17T08:50:33.881740+0000","last_became_peered":"2024-04-17T08:50:33.881740+0000","last_unstale":"2024-04-17T08:50:33.887255+0000","last_undegraded":"2024-04-17T08:50:33.887255+0000","last_fullsized":"2024-04-17T08:50:33.887255+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:30.084987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919797+0000","last_change":"2024-04-17T08:50:32.903736+0000","last_active":"2024-04-17T08:50:33.919797+0000","last_peered":"2024-04-17T08:50:33.919797+0000","last_clean":"2024-04-17T08:50:33.919797+0000","last_became_active":"2024-04-17T08:50:32.903485+0000","last_became_peered":"2024-04-17T08:50:32.903485+0000","last_unstale":"2024-04-17T08:50:33.919797+0000","last_undegraded":"2024-04-17T08:50:33.919797+0000","last_fullsized":"2024-04-17T08:50:33.919797+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:27.212635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911192+0000","last_change":"2024-04-17T08:50:33.904994+0000","last_active":"2024-04-17T08:50:33.911192+0000","last_peered":"2024-04-17T08:50:33.911192+0000","last_clean":"2024-04-17T08:50:33.911192+0000","last_became_active":"2024-04-17T08:50:33.904350+0000","last_became_peered":"2024-04-17T08:50:33.904350+0000","last_unstale":"2024-04-17T08:50:33.911192+0000","last_undegraded":"2024-04-17T08:50:33.911192+0000","last_fullsized":"2024-04-17T08:50:33.911192+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:47:46.355538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920660+0000","last_change":"2024-04-17T08:50:32.903006+0000","last_active":"2024-04-17T08:50:33.920660+0000","last_peered":"2024-04-17T08:50:33.920660+0000","last_clean":"2024-04-17T08:50:33.920660+0000","last_became_active":"2024-04-17T08:50:32.902660+0000","last_became_peered":"2024-04-17T08:50:32.902660+0000","last_unstale":"2024-04-17T08:50:33.920660+0000","last_undegraded":"2024-04-17T08:50:33.920660+0000","last_fullsized":"2024-04-17T08:50:33.920660+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:57.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898806+0000","last_change":"2024-04-17T08:50:33.896060+0000","last_active":"2024-04-17T08:50:33.898806+0000","last_peered":"2024-04-17T08:50:33.898806+0000","last_clean":"2024-04-17T08:50:33.898806+0000","last_became_active":"2024-04-17T08:50:33.895774+0000","last_became_peered":"2024-04-17T08:50:33.895774+0000","last_unstale":"2024-04-17T08:50:33.898806+0000","last_undegraded":"2024-04-17T08:50:33.898806+0000","last_fullsized":"2024-04-17T08:50:33.898806+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:10.938929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919207+0000","last_change":"2024-04-17T08:50:32.893112+0000","last_active":"2024-04-17T08:50:33.919207+0000","last_peered":"2024-04-17T08:50:33.919207+0000","last_clean":"2024-04-17T08:50:33.919207+0000","last_became_active":"2024-04-17T08:50:32.892793+0000","last_became_peered":"2024-04-17T08:50:32.892793+0000","last_unstale":"2024-04-17T08:50:33.919207+0000","last_undegraded":"2024-04-17T08:50:33.919207+0000","last_fullsized":"2024-04-17T08:50:33.919207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:52.104458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.895598+0000","last_change":"2024-04-17T08:50:33.890178+0000","last_active":"2024-04-17T08:50:33.895598+0000","last_peered":"2024-04-17T08:50:33.895598+0000","last_clean":"2024-04-17T08:50:33.895598+0000","last_became_active":"2024-04-17T08:50:33.889004+0000","last_became_peered":"2024-04-17T08:50:33.889004+0000","last_unstale":"2024-04-17T08:50:33.895598+0000","last_undegraded":"2024-04-17T08:50:33.895598+0000","last_fullsized":"2024-04-17T08:50:33.895598+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:00.894981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.883455+0000","last_change":"2024-04-17T08:50:32.900590+0000","last_active":"2024-04-17T08:50:33.883455+0000","last_peered":"2024-04-17T08:50:33.883455+0000","last_clean":"2024-04-17T08:50:33.883455+0000","last_became_active":"2024-04-17T08:50:32.900307+0000","last_became_peered":"2024-04-17T08:50:32.900307+0000","last_unstale":"2024-04-17T08:50:33.883455+0000","last_undegraded":"2024-04-17T08:50:33.883455+0000","last_fullsized":"2024-04-17T08:50:33.883455+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:45.711226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899753+0000","last_change":"2024-04-17T08:50:33.897386+0000","last_active":"2024-04-17T08:50:33.899753+0000","last_peered":"2024-04-17T08:50:33.899753+0000","last_clean":"2024-04-17T08:50:33.899753+0000","last_became_active":"2024-04-17T08:50:33.897087+0000","last_became_peered":"2024-04-17T08:50:33.897087+0000","last_unstale":"2024-04-17T08:50:33.899753+0000","last_undegraded":"2024-04-17T08:50:33.899753+0000","last_fullsized":"2024-04-17T08:50:33.899753+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:35.916088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.866092+0000","last_change":"2024-04-17T08:50:32.891454+0000","last_active":"2024-04-17T08:50:33.866092+0000","last_peered":"2024-04-17T08:50:33.866092+0000","last_clean":"2024-04-17T08:50:33.866092+0000","last_became_active":"2024-04-17T08:50:32.890756+0000","last_became_peered":"2024-04-17T08:50:32.890756+0000","last_unstale":"2024-04-17T08:50:33.866092+0000","last_undegraded":"2024-04-17T08:50:33.866092+0000","last_fullsized":"2024-04-17T08:50:33.866092+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:34.149084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901768+0000","last_change":"2024-04-17T08:50:33.899408+0000","last_active":"2024-04-17T08:50:33.901768+0000","last_peered":"2024-04-17T08:50:33.901768+0000","last_clean":"2024-04-17T08:50:33.901768+0000","last_became_active":"2024-04-17T08:50:33.898922+0000","last_became_peered":"2024-04-17T08:50:33.898922+0000","last_unstale":"2024-04-17T08:50:33.901768+0000","last_undegraded":"2024-04-17T08:50:33.901768+0000","last_fullsized":"2024-04-17T08:50:33.901768+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:06.016727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886747+0000","last_change":"2024-04-17T08:50:32.901844+0000","last_active":"2024-04-17T08:50:33.886747+0000","last_peered":"2024-04-17T08:50:33.886747+0000","last_clean":"2024-04-17T08:50:33.886747+0000","last_became_active":"2024-04-17T08:50:32.901571+0000","last_became_peered":"2024-04-17T08:50:32.901571+0000","last_unstale":"2024-04-17T08:50:33.886747+0000","last_undegraded":"2024-04-17T08:50:33.886747+0000","last_fullsized":"2024-04-17T08:50:33.886747+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:42.263359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906124+0000","last_change":"2024-04-17T08:50:33.903308+0000","last_active":"2024-04-17T08:50:33.906124+0000","last_peered":"2024-04-17T08:50:33.906124+0000","last_clean":"2024-04-17T08:50:33.906124+0000","last_became_active":"2024-04-17T08:50:33.902988+0000","last_became_peered":"2024-04-17T08:50:33.902988+0000","last_unstale":"2024-04-17T08:50:33.906124+0000","last_undegraded":"2024-04-17T08:50:33.906124+0000","last_fullsized":"2024-04-17T08:50:33.906124+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:09.971560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.872432+0000","last_change":"2024-04-17T08:50:32.903765+0000","last_active":"2024-04-17T08:50:33.872432+0000","last_peered":"2024-04-17T08:50:33.872432+0000","last_clean":"2024-04-17T08:50:33.872432+0000","last_became_active":"2024-04-17T08:50:32.902284+0000","last_became_peered":"2024-04-17T08:50:32.902284+0000","last_unstale":"2024-04-17T08:50:33.872432+0000","last_undegraded":"2024-04-17T08:50:33.872432+0000","last_fullsized":"2024-04-17T08:50:33.872432+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:41.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901732+0000","last_change":"2024-04-17T08:50:33.900099+0000","last_active":"2024-04-17T08:50:33.901732+0000","last_peered":"2024-04-17T08:50:33.901732+0000","last_clean":"2024-04-17T08:50:33.901732+0000","last_became_active":"2024-04-17T08:50:33.899785+0000","last_became_peered":"2024-04-17T08:50:33.899785+0000","last_unstale":"2024-04-17T08:50:33.901732+0000","last_undegraded":"2024-04-17T08:50:33.901732+0000","last_fullsized":"2024-04-17T08:50:33.901732+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:58.195690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"229'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920831+0000","last_change":"2024-04-17T08:50:32.892710+0000","last_active":"2024-04-17T08:50:33.920831+0000","last_peered":"2024-04-17T08:50:33.920831+0000","last_clean":"2024-04-17T08:50:33.920831+0000","last_became_active":"2024-04-17T08:50:32.892204+0000","last_became_peered":"2024-04-17T08:50:32.892204+0000","last_unstale":"2024-04-17T08:50:33.920831+0000","last_undegraded":"2024-04-17T08:50:33.920831+0000","last_fullsized":"2024-04-17T08:50:33.920831+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:01.085554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906193+0000","last_change":"2024-04-17T08:50:33.904088+0000","last_active":"2024-04-17T08:50:33.906193+0000","last_peered":"2024-04-17T08:50:33.906193+0000","last_clean":"2024-04-17T08:50:33.906193+0000","last_became_active":"2024-04-17T08:50:33.903730+0000","last_became_peered":"2024-04-17T08:50:33.903730+0000","last_unstale":"2024-04-17T08:50:33.906193+0000","last_undegraded":"2024-04-17T08:50:33.906193+0000","last_fullsized":"2024-04-17T08:50:33.906193+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.527681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"229'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.922108+0000","last_change":"2024-04-17T08:50:32.906064+0000","last_active":"2024-04-17T08:50:33.922108+0000","last_peered":"2024-04-17T08:50:33.922108+0000","last_clean":"2024-04-17T08:50:33.922108+0000","last_became_active":"2024-04-17T08:50:32.904959+0000","last_became_peered":"2024-04-17T08:50:32.904959+0000","last_unstale":"2024-04-17T08:50:33.922108+0000","last_undegraded":"2024-04-17T08:50:33.922108+0000","last_fullsized":"2024-04-17T08:50:33.922108+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:55.614891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911584+0000","last_change":"2024-04-17T08:50:33.908242+0000","last_active":"2024-04-17T08:50:33.911584+0000","last_peered":"2024-04-17T08:50:33.911584+0000","last_clean":"2024-04-17T08:50:33.911584+0000","last_became_active":"2024-04-17T08:50:33.907288+0000","last_became_peered":"2024-04-17T08:50:33.907288+0000","last_unstale":"2024-04-17T08:50:33.911584+0000","last_undegraded":"2024-04-17T08:50:33.911584+0000","last_fullsized":"2024-04-17T08:50:33.911584+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:32.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919180+0000","last_change":"2024-04-17T08:50:32.891648+0000","last_active":"2024-04-17T08:50:33.919180+0000","last_peered":"2024-04-17T08:50:33.919180+0000","last_clean":"2024-04-17T08:50:33.919180+0000","last_became_active":"2024-04-17T08:50:32.890804+0000","last_became_peered":"2024-04-17T08:50:32.890804+0000","last_unstale":"2024-04-17T08:50:33.919180+0000","last_undegraded":"2024-04-17T08:50:33.919180+0000","last_fullsized":"2024-04-17T08:50:33.919180+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:50.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.913658+0000","last_change":"2024-04-17T08:50:33.909416+0000","last_active":"2024-04-17T08:50:33.913658+0000","last_peered":"2024-04-17T08:50:33.913658+0000","last_clean":"2024-04-17T08:50:33.913658+0000","last_became_active":"2024-04-17T08:50:33.906483+0000","last_became_peered":"2024-04-17T08:50:33.906483+0000","last_unstale":"2024-04-17T08:50:33.913658+0000","last_undegraded":"2024-04-17T08:50:33.913658+0000","last_fullsized":"2024-04-17T08:50:33.913658+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:01.787161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.871340+0000","last_change":"2024-04-17T08:50:32.896932+0000","last_active":"2024-04-17T08:50:33.871340+0000","last_peered":"2024-04-17T08:50:33.871340+0000","last_clean":"2024-04-17T08:50:33.871340+0000","last_became_active":"2024-04-17T08:50:32.895134+0000","last_became_peered":"2024-04-17T08:50:32.895134+0000","last_unstale":"2024-04-17T08:50:33.871340+0000","last_undegraded":"2024-04-17T08:50:33.871340+0000","last_fullsized":"2024-04-17T08:50:33.871340+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:02.447617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904700+0000","last_change":"2024-04-17T08:50:33.896023+0000","last_active":"2024-04-17T08:50:33.904700+0000","last_peered":"2024-04-17T08:50:33.904700+0000","last_clean":"2024-04-17T08:50:33.904700+0000","last_became_active":"2024-04-17T08:50:33.895404+0000","last_became_peered":"2024-04-17T08:50:33.895404+0000","last_unstale":"2024-04-17T08:50:33.904700+0000","last_undegraded":"2024-04-17T08:50:33.904700+0000","last_fullsized":"2024-04-17T08:50:33.904700+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:28.665531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.889472+0000","last_change":"2024-04-17T08:50:32.895429+0000","last_active":"2024-04-17T08:50:33.889472+0000","last_peered":"2024-04-17T08:50:33.889472+0000","last_clean":"2024-04-17T08:50:33.889472+0000","last_became_active":"2024-04-17T08:50:32.894971+0000","last_became_peered":"2024-04-17T08:50:32.894971+0000","last_unstale":"2024-04-17T08:50:33.889472+0000","last_undegraded":"2024-04-17T08:50:33.889472+0000","last_fullsized":"2024-04-17T08:50:33.889472+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:22.318570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892729+0000","last_change":"2024-04-17T08:50:33.889134+0000","last_active":"2024-04-17T08:50:33.892729+0000","last_peered":"2024-04-17T08:50:33.892729+0000","last_clean":"2024-04-17T08:50:33.892729+0000","last_became_active":"2024-04-17T08:50:33.886115+0000","last_became_peered":"2024-04-17T08:50:33.886115+0000","last_unstale":"2024-04-17T08:50:33.892729+0000","last_undegraded":"2024-04-17T08:50:33.892729+0000","last_fullsized":"2024-04-17T08:50:33.892729+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:39:06.890071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"229'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920704+0000","last_change":"2024-04-17T08:50:32.903713+0000","last_active":"2024-04-17T08:50:33.920704+0000","last_peered":"2024-04-17T08:50:33.920704+0000","last_clean":"2024-04-17T08:50:33.920704+0000","last_became_active":"2024-04-17T08:50:32.903403+0000","last_became_peered":"2024-04-17T08:50:32.903403+0000","last_unstale":"2024-04-17T08:50:33.920704+0000","last_undegraded":"2024-04-17T08:50:33.920704+0000","last_fullsized":"2024-04-17T08:50:33.920704+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:09.595683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897870+0000","last_change":"2024-04-17T08:50:33.887711+0000","last_active":"2024-04-17T08:50:33.897870+0000","last_peered":"2024-04-17T08:50:33.897870+0000","last_clean":"2024-04-17T08:50:33.897870+0000","last_became_active":"2024-04-17T08:50:33.887362+0000","last_became_peered":"2024-04-17T08:50:33.887362+0000","last_unstale":"2024-04-17T08:50:33.897870+0000","last_undegraded":"2024-04-17T08:50:33.897870+0000","last_fullsized":"2024-04-17T08:50:33.897870+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:17.850270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.918596+0000","last_change":"2024-04-17T08:50:32.891171+0000","last_active":"2024-04-17T08:50:33.918596+0000","last_peered":"2024-04-17T08:50:33.918596+0000","last_clean":"2024-04-17T08:50:33.918596+0000","last_became_active":"2024-04-17T08:50:32.888285+0000","last_became_peered":"2024-04-17T08:50:32.888285+0000","last_unstale":"2024-04-17T08:50:33.918596+0000","last_undegraded":"2024-04-17T08:50:33.918596+0000","last_fullsized":"2024-04-17T08:50:33.918596+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.387644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.907850+0000","last_change":"2024-04-17T08:50:33.907099+0000","last_active":"2024-04-17T08:50:33.907850+0000","last_peered":"2024-04-17T08:50:33.907850+0000","last_clean":"2024-04-17T08:50:33.907850+0000","last_became_active":"2024-04-17T08:50:33.906802+0000","last_became_peered":"2024-04-17T08:50:33.906802+0000","last_unstale":"2024-04-17T08:50:33.907850+0000","last_undegraded":"2024-04-17T08:50:33.907850+0000","last_fullsized":"2024-04-17T08:50:33.907850+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:51.076665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919571+0000","last_change":"2024-04-17T08:50:32.901050+0000","last_active":"2024-04-17T08:50:33.919571+0000","last_peered":"2024-04-17T08:50:33.919571+0000","last_clean":"2024-04-17T08:50:33.919571+0000","last_became_active":"2024-04-17T08:50:32.900689+0000","last_became_peered":"2024-04-17T08:50:32.900689+0000","last_unstale":"2024-04-17T08:50:33.919571+0000","last_undegraded":"2024-04-17T08:50:33.919571+0000","last_fullsized":"2024-04-17T08:50:33.919571+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:39:51.234349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906044+0000","last_change":"2024-04-17T08:50:33.899313+0000","last_active":"2024-04-17T08:50:33.906044+0000","last_peered":"2024-04-17T08:50:33.906044+0000","last_clean":"2024-04-17T08:50:33.906044+0000","last_became_active":"2024-04-17T08:50:33.899005+0000","last_became_peered":"2024-04-17T08:50:33.899005+0000","last_unstale":"2024-04-17T08:50:33.906044+0000","last_undegraded":"2024-04-17T08:50:33.906044+0000","last_fullsized":"2024-04-17T08:50:33.906044+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:54.543895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920521+0000","last_change":"2024-04-17T08:50:32.896384+0000","last_active":"2024-04-17T08:50:33.920521+0000","last_peered":"2024-04-17T08:50:33.920521+0000","last_clean":"2024-04-17T08:50:33.920521+0000","last_became_active":"2024-04-17T08:50:32.893495+0000","last_became_peered":"2024-04-17T08:50:32.893495+0000","last_unstale":"2024-04-17T08:50:33.920521+0000","last_undegraded":"2024-04-17T08:50:33.920521+0000","last_fullsized":"2024-04-17T08:50:33.920521+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.852683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.912596+0000","last_change":"2024-04-17T08:50:33.905253+0000","last_active":"2024-04-17T08:50:33.912596+0000","last_peered":"2024-04-17T08:50:33.912596+0000","last_clean":"2024-04-17T08:50:33.912596+0000","last_became_active":"2024-04-17T08:50:33.904731+0000","last_became_peered":"2024-04-17T08:50:33.904731+0000","last_unstale":"2024-04-17T08:50:33.912596+0000","last_undegraded":"2024-04-17T08:50:33.912596+0000","last_fullsized":"2024-04-17T08:50:33.912596+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:21.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886095+0000","last_change":"2024-04-17T08:50:32.903243+0000","last_active":"2024-04-17T08:50:33.886095+0000","last_peered":"2024-04-17T08:50:33.886095+0000","last_clean":"2024-04-17T08:50:33.886095+0000","last_became_active":"2024-04-17T08:50:32.902864+0000","last_became_peered":"2024-04-17T08:50:32.902864+0000","last_unstale":"2024-04-17T08:50:33.886095+0000","last_undegraded":"2024-04-17T08:50:33.886095+0000","last_fullsized":"2024-04-17T08:50:33.886095+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:20.222164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910822+0000","last_change":"2024-04-17T08:50:33.897197+0000","last_active":"2024-04-17T08:50:33.910822+0000","last_peered":"2024-04-17T08:50:33.910822+0000","last_clean":"2024-04-17T08:50:33.910822+0000","last_became_active":"2024-04-17T08:50:33.895389+0000","last_became_peered":"2024-04-17T08:50:33.895389+0000","last_unstale":"2024-04-17T08:50:33.910822+0000","last_undegraded":"2024-04-17T08:50:33.910822+0000","last_fullsized":"2024-04-17T08:50:33.910822+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:37.711365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920894+0000","last_change":"2024-04-17T08:50:32.898032+0000","last_active":"2024-04-17T08:50:33.920894+0000","last_peered":"2024-04-17T08:50:33.920894+0000","last_clean":"2024-04-17T08:50:33.920894+0000","last_became_active":"2024-04-17T08:50:32.897755+0000","last_became_peered":"2024-04-17T08:50:32.897755+0000","last_unstale":"2024-04-17T08:50:33.920894+0000","last_undegraded":"2024-04-17T08:50:33.920894+0000","last_fullsized":"2024-04-17T08:50:33.920894+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:29.382845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.903922+0000","last_change":"2024-04-17T08:50:33.897335+0000","last_active":"2024-04-17T08:50:33.903922+0000","last_peered":"2024-04-17T08:50:33.903922+0000","last_clean":"2024-04-17T08:50:33.903922+0000","last_became_active":"2024-04-17T08:50:33.897009+0000","last_became_peered":"2024-04-17T08:50:33.897009+0000","last_unstale":"2024-04-17T08:50:33.903922+0000","last_undegraded":"2024-04-17T08:50:33.903922+0000","last_fullsized":"2024-04-17T08:50:33.903922+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:15.768477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.866878+0000","last_change":"2024-04-17T08:50:32.902321+0000","last_active":"2024-04-17T08:50:33.866878+0000","last_peered":"2024-04-17T08:50:33.866878+0000","last_clean":"2024-04-17T08:50:33.866878+0000","last_became_active":"2024-04-17T08:50:32.901781+0000","last_became_peered":"2024-04-17T08:50:32.901781+0000","last_unstale":"2024-04-17T08:50:33.866878+0000","last_undegraded":"2024-04-17T08:50:33.866878+0000","last_fullsized":"2024-04-17T08:50:33.866878+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:04.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897220+0000","last_change":"2024-04-17T08:50:33.884383+0000","last_active":"2024-04-17T08:50:33.897220+0000","last_peered":"2024-04-17T08:50:33.897220+0000","last_clean":"2024-04-17T08:50:33.897220+0000","last_became_active":"2024-04-17T08:50:33.883473+0000","last_became_peered":"2024-04-17T08:50:33.883473+0000","last_unstale":"2024-04-17T08:50:33.897220+0000","last_undegraded":"2024-04-17T08:50:33.897220+0000","last_fullsized":"2024-04-17T08:50:33.897220+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:15.432457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.864807+0000","last_change":"2024-04-17T08:50:32.896102+0000","last_active":"2024-04-17T08:50:33.864807+0000","last_peered":"2024-04-17T08:50:33.864807+0000","last_clean":"2024-04-17T08:50:33.864807+0000","last_became_active":"2024-04-17T08:50:32.895808+0000","last_became_peered":"2024-04-17T08:50:32.895808+0000","last_unstale":"2024-04-17T08:50:33.864807+0000","last_undegraded":"2024-04-17T08:50:33.864807+0000","last_fullsized":"2024-04-17T08:50:33.864807+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:38.551687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.895038+0000","last_change":"2024-04-17T08:50:33.892798+0000","last_active":"2024-04-17T08:50:33.895038+0000","last_peered":"2024-04-17T08:50:33.895038+0000","last_clean":"2024-04-17T08:50:33.895038+0000","last_became_active":"2024-04-17T08:50:33.891130+0000","last_became_peered":"2024-04-17T08:50:33.891130+0000","last_unstale":"2024-04-17T08:50:33.895038+0000","last_undegraded":"2024-04-17T08:50:33.895038+0000","last_fullsized":"2024-04-17T08:50:33.895038+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:07.536049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.877050+0000","last_change":"2024-04-17T08:50:32.894904+0000","last_active":"2024-04-17T08:50:33.877050+0000","last_peered":"2024-04-17T08:50:33.877050+0000","last_clean":"2024-04-17T08:50:33.877050+0000","last_became_active":"2024-04-17T08:50:32.894624+0000","last_became_peered":"2024-04-17T08:50:32.894624+0000","last_unstale":"2024-04-17T08:50:33.877050+0000","last_undegraded":"2024-04-17T08:50:33.877050+0000","last_fullsized":"2024-04-17T08:50:33.877050+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:22.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.902378+0000","last_change":"2024-04-17T08:50:33.899749+0000","last_active":"2024-04-17T08:50:33.902378+0000","last_peered":"2024-04-17T08:50:33.902378+0000","last_clean":"2024-04-17T08:50:33.902378+0000","last_became_active":"2024-04-17T08:50:33.898270+0000","last_became_peered":"2024-04-17T08:50:33.898270+0000","last_unstale":"2024-04-17T08:50:33.902378+0000","last_undegraded":"2024-04-17T08:50:33.902378+0000","last_fullsized":"2024-04-17T08:50:33.902378+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:10:44.340926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919805+0000","last_change":"2024-04-17T08:50:32.906744+0000","last_active":"2024-04-17T08:50:33.919805+0000","last_peered":"2024-04-17T08:50:33.919805+0000","last_clean":"2024-04-17T08:50:33.919805+0000","last_became_active":"2024-04-17T08:50:32.906445+0000","last_became_peered":"2024-04-17T08:50:32.906445+0000","last_unstale":"2024-04-17T08:50:33.919805+0000","last_undegraded":"2024-04-17T08:50:33.919805+0000","last_fullsized":"2024-04-17T08:50:33.919805+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:33.054437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894426+0000","last_change":"2024-04-17T08:50:33.892244+0000","last_active":"2024-04-17T08:50:33.894426+0000","last_peered":"2024-04-17T08:50:33.894426+0000","last_clean":"2024-04-17T08:50:33.894426+0000","last_became_active":"2024-04-17T08:50:33.891771+0000","last_became_peered":"2024-04-17T08:50:33.891771+0000","last_unstale":"2024-04-17T08:50:33.894426+0000","last_undegraded":"2024-04-17T08:50:33.894426+0000","last_fullsized":"2024-04-17T08:50:33.894426+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:18.314343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.888613+0000","last_change":"2024-04-17T08:50:32.904455+0000","last_active":"2024-04-17T08:50:33.888613+0000","last_peered":"2024-04-17T08:50:33.888613+0000","last_clean":"2024-04-17T08:50:33.888613+0000","last_became_active":"2024-04-17T08:50:32.904127+0000","last_became_peered":"2024-04-17T08:50:32.904127+0000","last_unstale":"2024-04-17T08:50:33.888613+0000","last_undegraded":"2024-04-17T08:50:33.888613+0000","last_fullsized":"2024-04-17T08:50:33.888613+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:09.473209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896574+0000","last_change":"2024-04-17T08:50:33.887506+0000","last_active":"2024-04-17T08:50:33.896574+0000","last_peered":"2024-04-17T08:50:33.896574+0000","last_clean":"2024-04-17T08:50:33.896574+0000","last_became_active":"2024-04-17T08:50:33.887071+0000","last_became_peered":"2024-04-17T08:50:33.887071+0000","last_unstale":"2024-04-17T08:50:33.896574+0000","last_undegraded":"2024-04-17T08:50:33.896574+0000","last_fullsized":"2024-04-17T08:50:33.896574+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:46:31.757149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886467+0000","last_change":"2024-04-17T08:50:32.901213+0000","last_active":"2024-04-17T08:50:33.886467+0000","last_peered":"2024-04-17T08:50:33.886467+0000","last_clean":"2024-04-17T08:50:33.886467+0000","last_became_active":"2024-04-17T08:50:32.900942+0000","last_became_peered":"2024-04-17T08:50:32.900942+0000","last_unstale":"2024-04-17T08:50:33.886467+0000","last_undegraded":"2024-04-17T08:50:33.886467+0000","last_fullsized":"2024-04-17T08:50:33.886467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:35.477286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894001+0000","last_change":"2024-04-17T08:50:33.892430+0000","last_active":"2024-04-17T08:50:33.894001+0000","last_peered":"2024-04-17T08:50:33.894001+0000","last_clean":"2024-04-17T08:50:33.894001+0000","last_became_active":"2024-04-17T08:50:33.892133+0000","last_became_peered":"2024-04-17T08:50:33.892133+0000","last_unstale":"2024-04-17T08:50:33.894001+0000","last_undegraded":"2024-04-17T08:50:33.894001+0000","last_fullsized":"2024-04-17T08:50:33.894001+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:19.182944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.887490+0000","last_change":"2024-04-17T08:50:32.906189+0000","last_active":"2024-04-17T08:50:33.887490+0000","last_peered":"2024-04-17T08:50:33.887490+0000","last_clean":"2024-04-17T08:50:33.887490+0000","last_became_active":"2024-04-17T08:50:32.905898+0000","last_became_peered":"2024-04-17T08:50:32.905898+0000","last_unstale":"2024-04-17T08:50:33.887490+0000","last_undegraded":"2024-04-17T08:50:33.887490+0000","last_fullsized":"2024-04-17T08:50:33.887490+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:24.678638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.884296+0000","last_change":"2024-04-17T08:50:33.879546+0000","last_active":"2024-04-17T08:50:33.884296+0000","last_peered":"2024-04-17T08:50:33.884296+0000","last_clean":"2024-04-17T08:50:33.884296+0000","last_became_active":"2024-04-17T08:50:33.878778+0000","last_became_peered":"2024-04-17T08:50:33.878778+0000","last_unstale":"2024-04-17T08:50:33.884296+0000","last_undegraded":"2024-04-17T08:50:33.884296+0000","last_fullsized":"2024-04-17T08:50:33.884296+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:28.759494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886708+0000","last_change":"2024-04-17T08:50:32.903970+0000","last_active":"2024-04-17T08:50:33.886708+0000","last_peered":"2024-04-17T08:50:33.886708+0000","last_clean":"2024-04-17T08:50:33.886708+0000","last_became_active":"2024-04-17T08:50:32.903620+0000","last_became_peered":"2024-04-17T08:50:32.903620+0000","last_unstale":"2024-04-17T08:50:33.886708+0000","last_undegraded":"2024-04-17T08:50:33.886708+0000","last_fullsized":"2024-04-17T08:50:33.886708+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:33.640417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899956+0000","last_change":"2024-04-17T08:50:33.892519+0000","last_active":"2024-04-17T08:50:33.899956+0000","last_peered":"2024-04-17T08:50:33.899956+0000","last_clean":"2024-04-17T08:50:33.899956+0000","last_became_active":"2024-04-17T08:50:33.892264+0000","last_became_peered":"2024-04-17T08:50:33.892264+0000","last_unstale":"2024-04-17T08:50:33.899956+0000","last_undegraded":"2024-04-17T08:50:33.899956+0000","last_fullsized":"2024-04-17T08:50:33.899956+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:16.218585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.884636+0000","last_change":"2024-04-17T08:50:32.891370+0000","last_active":"2024-04-17T08:50:33.884636+0000","last_peered":"2024-04-17T08:50:33.884636+0000","last_clean":"2024-04-17T08:50:33.884636+0000","last_became_active":"2024-04-17T08:50:32.888996+0000","last_became_peered":"2024-04-17T08:50:32.888996+0000","last_unstale":"2024-04-17T08:50:33.884636+0000","last_undegraded":"2024-04-17T08:50:33.884636+0000","last_fullsized":"2024-04-17T08:50:33.884636+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:31.815168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904679+0000","last_change":"2024-04-17T08:50:33.895678+0000","last_active":"2024-04-17T08:50:33.904679+0000","last_peered":"2024-04-17T08:50:33.904679+0000","last_clean":"2024-04-17T08:50:33.904679+0000","last_became_active":"2024-04-17T08:50:33.895010+0000","last_became_peered":"2024-04-17T08:50:33.895010+0000","last_unstale":"2024-04-17T08:50:33.904679+0000","last_undegraded":"2024-04-17T08:50:33.904679+0000","last_fullsized":"2024-04-17T08:50:33.904679+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:52:52.033934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.872055+0000","last_change":"2024-04-17T08:50:32.902675+0000","last_active":"2024-04-17T08:50:33.872055+0000","last_peered":"2024-04-17T08:50:33.872055+0000","last_clean":"2024-04-17T08:50:33.872055+0000","last_became_active":"2024-04-17T08:50:32.900858+0000","last_became_peered":"2024-04-17T08:50:32.900858+0000","last_unstale":"2024-04-17T08:50:33.872055+0000","last_undegraded":"2024-04-17T08:50:33.872055+0000","last_fullsized":"2024-04-17T08:50:33.872055+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:45.638814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896638+0000","last_change":"2024-04-17T08:50:33.884434+0000","last_active":"2024-04-17T08:50:33.896638+0000","last_peered":"2024-04-17T08:50:33.896638+0000","last_clean":"2024-04-17T08:50:33.896638+0000","last_became_active":"2024-04-17T08:50:33.884075+0000","last_became_peered":"2024-04-17T08:50:33.884075+0000","last_unstale":"2024-04-17T08:50:33.896638+0000","last_undegraded":"2024-04-17T08:50:33.896638+0000","last_fullsized":"2024-04-17T08:50:33.896638+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:23:51.344240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.879492+0000","last_change":"2024-04-17T08:50:32.894323+0000","last_active":"2024-04-17T08:50:33.879492+0000","last_peered":"2024-04-17T08:50:33.879492+0000","last_clean":"2024-04-17T08:50:33.879492+0000","last_became_active":"2024-04-17T08:50:32.893659+0000","last_became_peered":"2024-04-17T08:50:32.893659+0000","last_unstale":"2024-04-17T08:50:33.879492+0000","last_undegraded":"2024-04-17T08:50:33.879492+0000","last_fullsized":"2024-04-17T08:50:33.879492+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:16.424440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899445+0000","last_change":"2024-04-17T08:50:33.896994+0000","last_active":"2024-04-17T08:50:33.899445+0000","last_peered":"2024-04-17T08:50:33.899445+0000","last_clean":"2024-04-17T08:50:33.899445+0000","last_became_active":"2024-04-17T08:50:33.896640+0000","last_became_peered":"2024-04-17T08:50:33.896640+0000","last_unstale":"2024-04-17T08:50:33.899445+0000","last_undegraded":"2024-04-17T08:50:33.899445+0000","last_fullsized":"2024-04-17T08:50:33.899445+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:38.484472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.915027+0000","last_change":"2024-04-17T08:50:32.893722+0000","last_active":"2024-04-17T08:50:33.915027+0000","last_peered":"2024-04-17T08:50:33.915027+0000","last_clean":"2024-04-17T08:50:33.915027+0000","last_became_active":"2024-04-17T08:50:32.893404+0000","last_became_peered":"2024-04-17T08:50:32.893404+0000","last_unstale":"2024-04-17T08:50:33.915027+0000","last_undegraded":"2024-04-17T08:50:33.915027+0000","last_fullsized":"2024-04-17T08:50:33.915027+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:34.249904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"217'354","reported_seq":1446,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.885648+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:50:33.885648+0000","last_peered":"2024-04-17T08:50:33.885648+0000","last_clean":"2024-04-17T08:50:33.885648+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:50:33.885648+0000","last_undegraded":"2024-04-17T08:50:33.885648+0000","last_fullsized":"2024-04-17T08:50:33.885648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543293,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52628,"kb_used_data":3392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319212,"statfs":{"total":96636764160,"available":96582873088,"internally_reserved":0,"allocated":3473408,"data_stored":1993024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543293,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42292,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329548,"statfs":{"total":96636764160,"available":96593457152,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543292,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29828,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342012,"statfs":{"total":96636764160,"available":96606220288,"internally_reserved":0,"allocated":2871296,"data_stored":1401626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543292,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64500,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307340,"statfs":{"total":96636764160,"available":96570716160,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47880,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323960,"statfs":{"total":96636764160,"available":96587735040,"internally_reserved":0,"allocated":2879488,"data_stored":1401226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543293,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52036,"kb_used_data":2800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319804,"statfs":{"total":96636764160,"available":96583479296,"internally_reserved":0,"allocated":2867200,"data_stored":1401044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60336,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311504,"statfs":{"total":96636764160,"available":96574980096,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543292,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60920,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310920,"statfs":{"total":96636764160,"available":96574382080,"internally_reserved":0,"allocated":3448832,"data_stored":1984609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:50:38.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:50:38.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 Processor -- start 2024-04-17T08:50:38.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 -- start start 2024-04-17T08:50:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e6015c5a0 0x7f8e6015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6015f440 con 0x7f8e600b46c0 2024-04-17T08:50:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6015f720 con 0x7f8e600b4210 2024-04-17T08:50:38.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.685+0000 7f8e66c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8e6015fa00 con 0x7f8e6015c5a0 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e6015c5a0 0x7f8e6015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e6015c5a0 0x7f8e6015ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52816/0 (socket says 172.21.15.67:52816) 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 -- 172.21.15.67:0/1992320263 learned_addr learned my addr 172.21.15.67:0/1992320263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e66200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e600b46c0 0x7f8e600ae090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47968/0 (socket says 172.21.15.67:47968) 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 -- 172.21.15.67:0/1992320263 >> v1:172.21.15.187:6789/0 conn(0x7f8e600b4210 legacy=0x7f8e600af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e5fe00640 1 -- 172.21.15.67:0/1992320263 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589706478 0 0) 0x7f8e6015f440 con 0x7f8e600b46c0 2024-04-17T08:50:38.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 -- 172.21.15.67:0/1992320263 >> v1:172.21.15.67:6789/0 conn(0x7f8e600b46c0 legacy=0x7f8e600ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.686+0000 7f8e65800640 1 -- 172.21.15.67:0/1992320263 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6015fce0 con 0x7f8e6015c5a0 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e65800640 1 --2- 172.21.15.67:0/1992320263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e6015ee60 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009870 tx=0x7f8e5402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1e0b5e20842781 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e5fe00640 1 -- 172.21.15.67:0/1992320263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54035020 con 0x7f8e6015c5a0 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e5fe00640 1 -- 172.21.15.67:0/1992320263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5402d040 con 0x7f8e6015c5a0 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e5fe00640 1 -- 172.21.15.67:0/1992320263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54031a80 con 0x7f8e6015c5a0 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e66c00640 1 -- 172.21.15.67:0/1992320263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 msgr2=0x7f8e6015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.688+0000 7f8e66c00640 1 --2- 172.21.15.67:0/1992320263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e6015ee60 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009870 tx=0x7f8e5402b900 comp rx=0 tx=0).stop 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.689+0000 7f8e66c00640 1 -- 172.21.15.67:0/1992320263 shutdown_connections 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.689+0000 7f8e66c00640 1 --2- 172.21.15.67:0/1992320263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e6015ee60 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.689+0000 7f8e66c00640 1 -- 172.21.15.67:0/1992320263 >> 172.21.15.67:0/1992320263 conn(0x7f8e60067d10 msgr2=0x7f8e600b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.689+0000 7f8e66c00640 1 -- 172.21.15.67:0/1992320263 shutdown_connections 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.689+0000 7f8e66c00640 1 -- 172.21.15.67:0/1992320263 wait complete. 2024-04-17T08:50:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.690+0000 7f8e66c00640 1 Processor -- start 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.690+0000 7f8e66c00640 1 -- start start 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600b4210 0x7f8e601e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e6015a820 0x7f8e6015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6006e7f0 con 0x7f8e6015c5a0 2024-04-17T08:50:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6006e670 con 0x7f8e600b4210 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.691+0000 7f8e66c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e600afb50 con 0x7f8e6015a820 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52826/0 (socket says 172.21.15.67:52826) 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 -- 172.21.15.67:0/3663985935 learned_addr learned my addr 172.21.15.67:0/3663985935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e66200640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e6015a820 0x7f8e6015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e65800640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600b4210 0x7f8e601e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 -- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e6015a820 msgr2=0x7f8e6015acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e6015a820 0x7f8e6015acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 -- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600b4210 msgr2=0x7f8e601e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600b4210 0x7f8e601e51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.692+0000 7f8e64e00640 1 -- 172.21.15.67:0/3663985935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e601f5b70 con 0x7f8e6015c5a0 2024-04-17T08:50:38.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e64e00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c400 tx=0x7f8e4c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0052d0 con 0x7f8e6015c5a0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e4c005490 con 0x7f8e6015c5a0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c007260 con 0x7f8e6015c5a0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e601f5e50 con 0x7f8e6015c5a0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.693+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e601f6410 con 0x7f8e6015c5a0 2024-04-17T08:50:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.695+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e600b5630 con 0x7f8e6015c5a0 2024-04-17T08:50:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.695+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f8e4c007400 con 0x7f8e6015c5a0 2024-04-17T08:50:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.696+0000 7f8e5d600640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 0x7f8e4006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.698+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f8e4c08b190 con 0x7f8e6015c5a0 2024-04-17T08:50:38.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.698+0000 7f8e65800640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 0x7f8e4006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:38.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.698+0000 7f8e65800640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 0x7f8e4006ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036000 tx=0x7f8e54008030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:38.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.702+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8e600b5630 con 0x7f8e6015c5a0 2024-04-17T08:50:38.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.903+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e60068a10 con 0x7f8e4006ca40 2024-04-17T08:50:38.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.927+0000 7f8e5d600640 1 -- 172.21.15.67:0/3663985935 <== mgr.7679 v2:172.21.15.187:6832/150239855 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252431 (secure 0 0 0) 0x7f8e60068a10 con 0x7f8e4006ca40 2024-04-17T08:50:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:38.931 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:50:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 msgr2=0x7f8e4006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 0x7f8e4006ef00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036000 tx=0x7f8e54008030 comp rx=0 tx=0).stop 2024-04-17T08:50:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 msgr2=0x7f8e60145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:38.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c400 tx=0x7f8e4c00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 shutdown_connections 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f8e4006ca40 0x7f8e4006ef00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e6015a820 0x7f8e6015acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6015c5a0 0x7f8e60145fd0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 --2- 172.21.15.67:0/3663985935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e600b4210 0x7f8e601e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 >> 172.21.15.67:0/3663985935 conn(0x7f8e60067d10 msgr2=0x7f8e600ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.934+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 shutdown_connections 2024-04-17T08:50:38.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:38.935+0000 7f8e66c00640 1 -- 172.21.15.67:0/3663985935 wait complete. 2024-04-17T08:50:38.953 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-17T08:50:38.421324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":631,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":392,"ondisk_log_size":392,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":410420,"kb_used_data":23540,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754564300,"statfs":{"total":773094113280,"available":772673843200,"internally_reserved":0,"allocated":24104960,"data_stored":12364252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":396021296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002305"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894449+0000","last_change":"2024-04-17T08:50:33.892568+0000","last_active":"2024-04-17T08:50:33.894449+0000","last_peered":"2024-04-17T08:50:33.894449+0000","last_clean":"2024-04-17T08:50:33.894449+0000","last_became_active":"2024-04-17T08:50:33.892228+0000","last_became_peered":"2024-04-17T08:50:33.892228+0000","last_unstale":"2024-04-17T08:50:33.894449+0000","last_undegraded":"2024-04-17T08:50:33.894449+0000","last_fullsized":"2024-04-17T08:50:33.894449+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:29.148122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.893489+0000","last_change":"2024-04-17T08:50:33.883908+0000","last_active":"2024-04-17T08:50:33.893489+0000","last_peered":"2024-04-17T08:50:33.893489+0000","last_clean":"2024-04-17T08:50:33.893489+0000","last_became_active":"2024-04-17T08:50:33.883336+0000","last_became_peered":"2024-04-17T08:50:33.883336+0000","last_unstale":"2024-04-17T08:50:33.893489+0000","last_undegraded":"2024-04-17T08:50:33.893489+0000","last_fullsized":"2024-04-17T08:50:33.893489+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:47.164759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906392+0000","last_change":"2024-04-17T08:50:33.900448+0000","last_active":"2024-04-17T08:50:33.906392+0000","last_peered":"2024-04-17T08:50:33.906392+0000","last_clean":"2024-04-17T08:50:33.906392+0000","last_became_active":"2024-04-17T08:50:33.900142+0000","last_became_peered":"2024-04-17T08:50:33.900142+0000","last_unstale":"2024-04-17T08:50:33.906392+0000","last_undegraded":"2024-04-17T08:50:33.906392+0000","last_fullsized":"2024-04-17T08:50:33.906392+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:13.541875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898864+0000","last_change":"2024-04-17T08:50:33.896077+0000","last_active":"2024-04-17T08:50:33.898864+0000","last_peered":"2024-04-17T08:50:33.898864+0000","last_clean":"2024-04-17T08:50:33.898864+0000","last_became_active":"2024-04-17T08:50:33.895408+0000","last_became_peered":"2024-04-17T08:50:33.895408+0000","last_unstale":"2024-04-17T08:50:33.898864+0000","last_undegraded":"2024-04-17T08:50:33.898864+0000","last_fullsized":"2024-04-17T08:50:33.898864+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:04.665259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911516+0000","last_change":"2024-04-17T08:50:33.908217+0000","last_active":"2024-04-17T08:50:33.911516+0000","last_peered":"2024-04-17T08:50:33.911516+0000","last_clean":"2024-04-17T08:50:33.911516+0000","last_became_active":"2024-04-17T08:50:33.907185+0000","last_became_peered":"2024-04-17T08:50:33.907185+0000","last_unstale":"2024-04-17T08:50:33.911516+0000","last_undegraded":"2024-04-17T08:50:33.911516+0000","last_fullsized":"2024-04-17T08:50:33.911516+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:40:56.232264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.903581+0000","last_change":"2024-04-17T08:50:33.900220+0000","last_active":"2024-04-17T08:50:33.903581+0000","last_peered":"2024-04-17T08:50:33.903581+0000","last_clean":"2024-04-17T08:50:33.903581+0000","last_became_active":"2024-04-17T08:50:33.898746+0000","last_became_peered":"2024-04-17T08:50:33.898746+0000","last_unstale":"2024-04-17T08:50:33.903581+0000","last_undegraded":"2024-04-17T08:50:33.903581+0000","last_fullsized":"2024-04-17T08:50:33.903581+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:17.253778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.890589+0000","last_change":"2024-04-17T08:50:33.880798+0000","last_active":"2024-04-17T08:50:33.890589+0000","last_peered":"2024-04-17T08:50:33.890589+0000","last_clean":"2024-04-17T08:50:33.890589+0000","last_became_active":"2024-04-17T08:50:33.879829+0000","last_became_peered":"2024-04-17T08:50:33.879829+0000","last_unstale":"2024-04-17T08:50:33.890589+0000","last_undegraded":"2024-04-17T08:50:33.890589+0000","last_fullsized":"2024-04-17T08:50:33.890589+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:14.610912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896792+0000","last_change":"2024-04-17T08:50:33.895419+0000","last_active":"2024-04-17T08:50:33.896792+0000","last_peered":"2024-04-17T08:50:33.896792+0000","last_clean":"2024-04-17T08:50:33.896792+0000","last_became_active":"2024-04-17T08:50:33.894089+0000","last_became_peered":"2024-04-17T08:50:33.894089+0000","last_unstale":"2024-04-17T08:50:33.896792+0000","last_undegraded":"2024-04-17T08:50:33.896792+0000","last_fullsized":"2024-04-17T08:50:33.896792+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:19.879655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897381+0000","last_change":"2024-04-17T08:50:33.894702+0000","last_active":"2024-04-17T08:50:33.897381+0000","last_peered":"2024-04-17T08:50:33.897381+0000","last_clean":"2024-04-17T08:50:33.897381+0000","last_became_active":"2024-04-17T08:50:33.894398+0000","last_became_peered":"2024-04-17T08:50:33.894398+0000","last_unstale":"2024-04-17T08:50:33.897381+0000","last_undegraded":"2024-04-17T08:50:33.897381+0000","last_fullsized":"2024-04-17T08:50:33.897381+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:15.124518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910127+0000","last_change":"2024-04-17T08:50:33.895456+0000","last_active":"2024-04-17T08:50:33.910127+0000","last_peered":"2024-04-17T08:50:33.910127+0000","last_clean":"2024-04-17T08:50:33.910127+0000","last_became_active":"2024-04-17T08:50:33.895142+0000","last_became_peered":"2024-04-17T08:50:33.895142+0000","last_unstale":"2024-04-17T08:50:33.910127+0000","last_undegraded":"2024-04-17T08:50:33.910127+0000","last_fullsized":"2024-04-17T08:50:33.910127+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:35.646922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904098+0000","last_change":"2024-04-17T08:50:33.897057+0000","last_active":"2024-04-17T08:50:33.904098+0000","last_peered":"2024-04-17T08:50:33.904098+0000","last_clean":"2024-04-17T08:50:33.904098+0000","last_became_active":"2024-04-17T08:50:33.894722+0000","last_became_peered":"2024-04-17T08:50:33.894722+0000","last_unstale":"2024-04-17T08:50:33.904098+0000","last_undegraded":"2024-04-17T08:50:33.904098+0000","last_fullsized":"2024-04-17T08:50:33.904098+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:27.999528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901918+0000","last_change":"2024-04-17T08:50:33.897953+0000","last_active":"2024-04-17T08:50:33.901918+0000","last_peered":"2024-04-17T08:50:33.901918+0000","last_clean":"2024-04-17T08:50:33.901918+0000","last_became_active":"2024-04-17T08:50:33.897591+0000","last_became_peered":"2024-04-17T08:50:33.897591+0000","last_unstale":"2024-04-17T08:50:33.901918+0000","last_undegraded":"2024-04-17T08:50:33.901918+0000","last_fullsized":"2024-04-17T08:50:33.901918+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:08.803343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.900067+0000","last_change":"2024-04-17T08:50:33.894398+0000","last_active":"2024-04-17T08:50:33.900067+0000","last_peered":"2024-04-17T08:50:33.900067+0000","last_clean":"2024-04-17T08:50:33.900067+0000","last_became_active":"2024-04-17T08:50:33.894105+0000","last_became_peered":"2024-04-17T08:50:33.894105+0000","last_unstale":"2024-04-17T08:50:33.900067+0000","last_undegraded":"2024-04-17T08:50:33.900067+0000","last_fullsized":"2024-04-17T08:50:33.900067+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:08.859357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892403+0000","last_change":"2024-04-17T08:50:33.887226+0000","last_active":"2024-04-17T08:50:33.892403+0000","last_peered":"2024-04-17T08:50:33.892403+0000","last_clean":"2024-04-17T08:50:33.892403+0000","last_became_active":"2024-04-17T08:50:33.886912+0000","last_became_peered":"2024-04-17T08:50:33.886912+0000","last_unstale":"2024-04-17T08:50:33.892403+0000","last_undegraded":"2024-04-17T08:50:33.892403+0000","last_fullsized":"2024-04-17T08:50:33.892403+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:09:36.180669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901135+0000","last_change":"2024-04-17T08:50:33.898186+0000","last_active":"2024-04-17T08:50:33.901135+0000","last_peered":"2024-04-17T08:50:33.901135+0000","last_clean":"2024-04-17T08:50:33.901135+0000","last_became_active":"2024-04-17T08:50:33.897746+0000","last_became_peered":"2024-04-17T08:50:33.897746+0000","last_unstale":"2024-04-17T08:50:33.901135+0000","last_undegraded":"2024-04-17T08:50:33.901135+0000","last_fullsized":"2024-04-17T08:50:33.901135+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:52.169578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910192+0000","last_change":"2024-04-17T08:50:33.895366+0000","last_active":"2024-04-17T08:50:33.910192+0000","last_peered":"2024-04-17T08:50:33.910192+0000","last_clean":"2024-04-17T08:50:33.910192+0000","last_became_active":"2024-04-17T08:50:33.892102+0000","last_became_peered":"2024-04-17T08:50:33.892102+0000","last_unstale":"2024-04-17T08:50:33.910192+0000","last_undegraded":"2024-04-17T08:50:33.910192+0000","last_fullsized":"2024-04-17T08:50:33.910192+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:28.242561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892444+0000","last_change":"2024-04-17T08:50:33.888622+0000","last_active":"2024-04-17T08:50:33.892444+0000","last_peered":"2024-04-17T08:50:33.892444+0000","last_clean":"2024-04-17T08:50:33.892444+0000","last_became_active":"2024-04-17T08:50:33.888298+0000","last_became_peered":"2024-04-17T08:50:33.888298+0000","last_unstale":"2024-04-17T08:50:33.892444+0000","last_undegraded":"2024-04-17T08:50:33.892444+0000","last_fullsized":"2024-04-17T08:50:33.892444+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:59.627816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911857+0000","last_change":"2024-04-17T08:50:33.905082+0000","last_active":"2024-04-17T08:50:33.911857+0000","last_peered":"2024-04-17T08:50:33.911857+0000","last_clean":"2024-04-17T08:50:33.911857+0000","last_became_active":"2024-04-17T08:50:33.904723+0000","last_became_peered":"2024-04-17T08:50:33.904723+0000","last_unstale":"2024-04-17T08:50:33.911857+0000","last_undegraded":"2024-04-17T08:50:33.911857+0000","last_fullsized":"2024-04-17T08:50:33.911857+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:47.939265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.913597+0000","last_change":"2024-04-17T08:50:33.907077+0000","last_active":"2024-04-17T08:50:33.913597+0000","last_peered":"2024-04-17T08:50:33.913597+0000","last_clean":"2024-04-17T08:50:33.913597+0000","last_became_active":"2024-04-17T08:50:33.906771+0000","last_became_peered":"2024-04-17T08:50:33.906771+0000","last_unstale":"2024-04-17T08:50:33.913597+0000","last_undegraded":"2024-04-17T08:50:33.913597+0000","last_fullsized":"2024-04-17T08:50:33.913597+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:14.899784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911180+0000","last_change":"2024-04-17T08:50:33.907557+0000","last_active":"2024-04-17T08:50:33.911180+0000","last_peered":"2024-04-17T08:50:33.911180+0000","last_clean":"2024-04-17T08:50:33.911180+0000","last_became_active":"2024-04-17T08:50:33.907258+0000","last_became_peered":"2024-04-17T08:50:33.907258+0000","last_unstale":"2024-04-17T08:50:33.911180+0000","last_undegraded":"2024-04-17T08:50:33.911180+0000","last_fullsized":"2024-04-17T08:50:33.911180+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:26.641470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.905091+0000","last_change":"2024-04-17T08:50:33.904610+0000","last_active":"2024-04-17T08:50:33.905091+0000","last_peered":"2024-04-17T08:50:33.905091+0000","last_clean":"2024-04-17T08:50:33.905091+0000","last_became_active":"2024-04-17T08:50:33.904287+0000","last_became_peered":"2024-04-17T08:50:33.904287+0000","last_unstale":"2024-04-17T08:50:33.905091+0000","last_undegraded":"2024-04-17T08:50:33.905091+0000","last_fullsized":"2024-04-17T08:50:33.905091+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:02.874584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906484+0000","last_change":"2024-04-17T08:50:33.905971+0000","last_active":"2024-04-17T08:50:33.906484+0000","last_peered":"2024-04-17T08:50:33.906484+0000","last_clean":"2024-04-17T08:50:33.906484+0000","last_became_active":"2024-04-17T08:50:33.905591+0000","last_became_peered":"2024-04-17T08:50:33.905591+0000","last_unstale":"2024-04-17T08:50:33.906484+0000","last_undegraded":"2024-04-17T08:50:33.906484+0000","last_fullsized":"2024-04-17T08:50:33.906484+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:46.730352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.891914+0000","last_change":"2024-04-17T08:50:33.890185+0000","last_active":"2024-04-17T08:50:33.891914+0000","last_peered":"2024-04-17T08:50:33.891914+0000","last_clean":"2024-04-17T08:50:33.891914+0000","last_became_active":"2024-04-17T08:50:33.889133+0000","last_became_peered":"2024-04-17T08:50:33.889133+0000","last_unstale":"2024-04-17T08:50:33.891914+0000","last_undegraded":"2024-04-17T08:50:33.891914+0000","last_fullsized":"2024-04-17T08:50:33.891914+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:21.688586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901783+0000","last_change":"2024-04-17T08:50:33.900938+0000","last_active":"2024-04-17T08:50:33.901783+0000","last_peered":"2024-04-17T08:50:33.901783+0000","last_clean":"2024-04-17T08:50:33.901783+0000","last_became_active":"2024-04-17T08:50:33.900085+0000","last_became_peered":"2024-04-17T08:50:33.900085+0000","last_unstale":"2024-04-17T08:50:33.901783+0000","last_undegraded":"2024-04-17T08:50:33.901783+0000","last_fullsized":"2024-04-17T08:50:33.901783+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:56.712749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897352+0000","last_change":"2024-04-17T08:50:33.893295+0000","last_active":"2024-04-17T08:50:33.897352+0000","last_peered":"2024-04-17T08:50:33.897352+0000","last_clean":"2024-04-17T08:50:33.897352+0000","last_became_active":"2024-04-17T08:50:33.892999+0000","last_became_peered":"2024-04-17T08:50:33.892999+0000","last_unstale":"2024-04-17T08:50:33.897352+0000","last_undegraded":"2024-04-17T08:50:33.897352+0000","last_fullsized":"2024-04-17T08:50:33.897352+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:26:51.603438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886137+0000","last_change":"2024-04-17T08:50:33.879179+0000","last_active":"2024-04-17T08:50:33.886137+0000","last_peered":"2024-04-17T08:50:33.886137+0000","last_clean":"2024-04-17T08:50:33.886137+0000","last_became_active":"2024-04-17T08:50:33.878412+0000","last_became_peered":"2024-04-17T08:50:33.878412+0000","last_unstale":"2024-04-17T08:50:33.886137+0000","last_undegraded":"2024-04-17T08:50:33.886137+0000","last_fullsized":"2024-04-17T08:50:33.886137+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:15.166595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901627+0000","last_change":"2024-04-17T08:50:33.897365+0000","last_active":"2024-04-17T08:50:33.901627+0000","last_peered":"2024-04-17T08:50:33.901627+0000","last_clean":"2024-04-17T08:50:33.901627+0000","last_became_active":"2024-04-17T08:50:33.897012+0000","last_became_peered":"2024-04-17T08:50:33.897012+0000","last_unstale":"2024-04-17T08:50:33.901627+0000","last_undegraded":"2024-04-17T08:50:33.901627+0000","last_fullsized":"2024-04-17T08:50:33.901627+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:33:16.415969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906022+0000","last_change":"2024-04-17T08:50:33.897631+0000","last_active":"2024-04-17T08:50:33.906022+0000","last_peered":"2024-04-17T08:50:33.906022+0000","last_clean":"2024-04-17T08:50:33.906022+0000","last_became_active":"2024-04-17T08:50:33.897328+0000","last_became_peered":"2024-04-17T08:50:33.897328+0000","last_unstale":"2024-04-17T08:50:33.906022+0000","last_undegraded":"2024-04-17T08:50:33.906022+0000","last_fullsized":"2024-04-17T08:50:33.906022+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:18:26.586972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.902255+0000","last_change":"2024-04-17T08:50:33.899268+0000","last_active":"2024-04-17T08:50:33.902255+0000","last_peered":"2024-04-17T08:50:33.902255+0000","last_clean":"2024-04-17T08:50:33.902255+0000","last_became_active":"2024-04-17T08:50:33.898733+0000","last_became_peered":"2024-04-17T08:50:33.898733+0000","last_unstale":"2024-04-17T08:50:33.902255+0000","last_undegraded":"2024-04-17T08:50:33.902255+0000","last_fullsized":"2024-04-17T08:50:33.902255+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:42.459510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897415+0000","last_change":"2024-04-17T08:50:33.895564+0000","last_active":"2024-04-17T08:50:33.897415+0000","last_peered":"2024-04-17T08:50:33.897415+0000","last_clean":"2024-04-17T08:50:33.897415+0000","last_became_active":"2024-04-17T08:50:33.893775+0000","last_became_peered":"2024-04-17T08:50:33.893775+0000","last_unstale":"2024-04-17T08:50:33.897415+0000","last_undegraded":"2024-04-17T08:50:33.897415+0000","last_fullsized":"2024-04-17T08:50:33.897415+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:53.858430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904060+0000","last_change":"2024-04-17T08:50:33.890500+0000","last_active":"2024-04-17T08:50:33.904060+0000","last_peered":"2024-04-17T08:50:33.904060+0000","last_clean":"2024-04-17T08:50:33.904060+0000","last_became_active":"2024-04-17T08:50:33.890183+0000","last_became_peered":"2024-04-17T08:50:33.890183+0000","last_unstale":"2024-04-17T08:50:33.904060+0000","last_undegraded":"2024-04-17T08:50:33.904060+0000","last_fullsized":"2024-04-17T08:50:33.904060+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:57.852009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901387+0000","last_change":"2024-04-17T08:50:33.897001+0000","last_active":"2024-04-17T08:50:33.901387+0000","last_peered":"2024-04-17T08:50:33.901387+0000","last_clean":"2024-04-17T08:50:33.901387+0000","last_became_active":"2024-04-17T08:50:33.896545+0000","last_became_peered":"2024-04-17T08:50:33.896545+0000","last_unstale":"2024-04-17T08:50:33.901387+0000","last_undegraded":"2024-04-17T08:50:33.901387+0000","last_fullsized":"2024-04-17T08:50:33.901387+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:40.333703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899286+0000","last_change":"2024-04-17T08:50:33.897991+0000","last_active":"2024-04-17T08:50:33.899286+0000","last_peered":"2024-04-17T08:50:33.899286+0000","last_clean":"2024-04-17T08:50:33.899286+0000","last_became_active":"2024-04-17T08:50:33.897605+0000","last_became_peered":"2024-04-17T08:50:33.897605+0000","last_unstale":"2024-04-17T08:50:33.899286+0000","last_undegraded":"2024-04-17T08:50:33.899286+0000","last_fullsized":"2024-04-17T08:50:33.899286+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:35.168211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"229'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.918131+0000","last_change":"2024-04-17T08:50:32.901470+0000","last_active":"2024-04-17T08:50:33.918131+0000","last_peered":"2024-04-17T08:50:33.918131+0000","last_clean":"2024-04-17T08:50:33.918131+0000","last_became_active":"2024-04-17T08:50:32.900718+0000","last_became_peered":"2024-04-17T08:50:32.900718+0000","last_unstale":"2024-04-17T08:50:33.918131+0000","last_undegraded":"2024-04-17T08:50:33.918131+0000","last_fullsized":"2024-04-17T08:50:33.918131+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:32.091268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898276+0000","last_change":"2024-04-17T08:50:33.895246+0000","last_active":"2024-04-17T08:50:33.898276+0000","last_peered":"2024-04-17T08:50:33.898276+0000","last_clean":"2024-04-17T08:50:33.898276+0000","last_became_active":"2024-04-17T08:50:33.891622+0000","last_became_peered":"2024-04-17T08:50:33.891622+0000","last_unstale":"2024-04-17T08:50:33.898276+0000","last_undegraded":"2024-04-17T08:50:33.898276+0000","last_fullsized":"2024-04-17T08:50:33.898276+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:23.927426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919238+0000","last_change":"2024-04-17T08:50:32.904503+0000","last_active":"2024-04-17T08:50:33.919238+0000","last_peered":"2024-04-17T08:50:33.919238+0000","last_clean":"2024-04-17T08:50:33.919238+0000","last_became_active":"2024-04-17T08:50:32.904221+0000","last_became_peered":"2024-04-17T08:50:32.904221+0000","last_unstale":"2024-04-17T08:50:33.919238+0000","last_undegraded":"2024-04-17T08:50:33.919238+0000","last_fullsized":"2024-04-17T08:50:33.919238+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:54.730453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.887255+0000","last_change":"2024-04-17T08:50:33.882411+0000","last_active":"2024-04-17T08:50:33.887255+0000","last_peered":"2024-04-17T08:50:33.887255+0000","last_clean":"2024-04-17T08:50:33.887255+0000","last_became_active":"2024-04-17T08:50:33.881740+0000","last_became_peered":"2024-04-17T08:50:33.881740+0000","last_unstale":"2024-04-17T08:50:33.887255+0000","last_undegraded":"2024-04-17T08:50:33.887255+0000","last_fullsized":"2024-04-17T08:50:33.887255+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:30.084987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919797+0000","last_change":"2024-04-17T08:50:32.903736+0000","last_active":"2024-04-17T08:50:33.919797+0000","last_peered":"2024-04-17T08:50:33.919797+0000","last_clean":"2024-04-17T08:50:33.919797+0000","last_became_active":"2024-04-17T08:50:32.903485+0000","last_became_peered":"2024-04-17T08:50:32.903485+0000","last_unstale":"2024-04-17T08:50:33.919797+0000","last_undegraded":"2024-04-17T08:50:33.919797+0000","last_fullsized":"2024-04-17T08:50:33.919797+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:27.212635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911192+0000","last_change":"2024-04-17T08:50:33.904994+0000","last_active":"2024-04-17T08:50:33.911192+0000","last_peered":"2024-04-17T08:50:33.911192+0000","last_clean":"2024-04-17T08:50:33.911192+0000","last_became_active":"2024-04-17T08:50:33.904350+0000","last_became_peered":"2024-04-17T08:50:33.904350+0000","last_unstale":"2024-04-17T08:50:33.911192+0000","last_undegraded":"2024-04-17T08:50:33.911192+0000","last_fullsized":"2024-04-17T08:50:33.911192+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:47:46.355538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920660+0000","last_change":"2024-04-17T08:50:32.903006+0000","last_active":"2024-04-17T08:50:33.920660+0000","last_peered":"2024-04-17T08:50:33.920660+0000","last_clean":"2024-04-17T08:50:33.920660+0000","last_became_active":"2024-04-17T08:50:32.902660+0000","last_became_peered":"2024-04-17T08:50:32.902660+0000","last_unstale":"2024-04-17T08:50:33.920660+0000","last_undegraded":"2024-04-17T08:50:33.920660+0000","last_fullsized":"2024-04-17T08:50:33.920660+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:57.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.898806+0000","last_change":"2024-04-17T08:50:33.896060+0000","last_active":"2024-04-17T08:50:33.898806+0000","last_peered":"2024-04-17T08:50:33.898806+0000","last_clean":"2024-04-17T08:50:33.898806+0000","last_became_active":"2024-04-17T08:50:33.895774+0000","last_became_peered":"2024-04-17T08:50:33.895774+0000","last_unstale":"2024-04-17T08:50:33.898806+0000","last_undegraded":"2024-04-17T08:50:33.898806+0000","last_fullsized":"2024-04-17T08:50:33.898806+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:10.938929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919207+0000","last_change":"2024-04-17T08:50:32.893112+0000","last_active":"2024-04-17T08:50:33.919207+0000","last_peered":"2024-04-17T08:50:33.919207+0000","last_clean":"2024-04-17T08:50:33.919207+0000","last_became_active":"2024-04-17T08:50:32.892793+0000","last_became_peered":"2024-04-17T08:50:32.892793+0000","last_unstale":"2024-04-17T08:50:33.919207+0000","last_undegraded":"2024-04-17T08:50:33.919207+0000","last_fullsized":"2024-04-17T08:50:33.919207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:52.104458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.895598+0000","last_change":"2024-04-17T08:50:33.890178+0000","last_active":"2024-04-17T08:50:33.895598+0000","last_peered":"2024-04-17T08:50:33.895598+0000","last_clean":"2024-04-17T08:50:33.895598+0000","last_became_active":"2024-04-17T08:50:33.889004+0000","last_became_peered":"2024-04-17T08:50:33.889004+0000","last_unstale":"2024-04-17T08:50:33.895598+0000","last_undegraded":"2024-04-17T08:50:33.895598+0000","last_fullsized":"2024-04-17T08:50:33.895598+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:00.894981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.883455+0000","last_change":"2024-04-17T08:50:32.900590+0000","last_active":"2024-04-17T08:50:33.883455+0000","last_peered":"2024-04-17T08:50:33.883455+0000","last_clean":"2024-04-17T08:50:33.883455+0000","last_became_active":"2024-04-17T08:50:32.900307+0000","last_became_peered":"2024-04-17T08:50:32.900307+0000","last_unstale":"2024-04-17T08:50:33.883455+0000","last_undegraded":"2024-04-17T08:50:33.883455+0000","last_fullsized":"2024-04-17T08:50:33.883455+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:45.711226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899753+0000","last_change":"2024-04-17T08:50:33.897386+0000","last_active":"2024-04-17T08:50:33.899753+0000","last_peered":"2024-04-17T08:50:33.899753+0000","last_clean":"2024-04-17T08:50:33.899753+0000","last_became_active":"2024-04-17T08:50:33.897087+0000","last_became_peered":"2024-04-17T08:50:33.897087+0000","last_unstale":"2024-04-17T08:50:33.899753+0000","last_undegraded":"2024-04-17T08:50:33.899753+0000","last_fullsized":"2024-04-17T08:50:33.899753+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:35.916088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.866092+0000","last_change":"2024-04-17T08:50:32.891454+0000","last_active":"2024-04-17T08:50:33.866092+0000","last_peered":"2024-04-17T08:50:33.866092+0000","last_clean":"2024-04-17T08:50:33.866092+0000","last_became_active":"2024-04-17T08:50:32.890756+0000","last_became_peered":"2024-04-17T08:50:32.890756+0000","last_unstale":"2024-04-17T08:50:33.866092+0000","last_undegraded":"2024-04-17T08:50:33.866092+0000","last_fullsized":"2024-04-17T08:50:33.866092+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:34.149084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901768+0000","last_change":"2024-04-17T08:50:33.899408+0000","last_active":"2024-04-17T08:50:33.901768+0000","last_peered":"2024-04-17T08:50:33.901768+0000","last_clean":"2024-04-17T08:50:33.901768+0000","last_became_active":"2024-04-17T08:50:33.898922+0000","last_became_peered":"2024-04-17T08:50:33.898922+0000","last_unstale":"2024-04-17T08:50:33.901768+0000","last_undegraded":"2024-04-17T08:50:33.901768+0000","last_fullsized":"2024-04-17T08:50:33.901768+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:06.016727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886747+0000","last_change":"2024-04-17T08:50:32.901844+0000","last_active":"2024-04-17T08:50:33.886747+0000","last_peered":"2024-04-17T08:50:33.886747+0000","last_clean":"2024-04-17T08:50:33.886747+0000","last_became_active":"2024-04-17T08:50:32.901571+0000","last_became_peered":"2024-04-17T08:50:32.901571+0000","last_unstale":"2024-04-17T08:50:33.886747+0000","last_undegraded":"2024-04-17T08:50:33.886747+0000","last_fullsized":"2024-04-17T08:50:33.886747+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:42.263359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906124+0000","last_change":"2024-04-17T08:50:33.903308+0000","last_active":"2024-04-17T08:50:33.906124+0000","last_peered":"2024-04-17T08:50:33.906124+0000","last_clean":"2024-04-17T08:50:33.906124+0000","last_became_active":"2024-04-17T08:50:33.902988+0000","last_became_peered":"2024-04-17T08:50:33.902988+0000","last_unstale":"2024-04-17T08:50:33.906124+0000","last_undegraded":"2024-04-17T08:50:33.906124+0000","last_fullsized":"2024-04-17T08:50:33.906124+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:09.971560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.872432+0000","last_change":"2024-04-17T08:50:32.903765+0000","last_active":"2024-04-17T08:50:33.872432+0000","last_peered":"2024-04-17T08:50:33.872432+0000","last_clean":"2024-04-17T08:50:33.872432+0000","last_became_active":"2024-04-17T08:50:32.902284+0000","last_became_peered":"2024-04-17T08:50:32.902284+0000","last_unstale":"2024-04-17T08:50:33.872432+0000","last_undegraded":"2024-04-17T08:50:33.872432+0000","last_fullsized":"2024-04-17T08:50:33.872432+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:41.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.901732+0000","last_change":"2024-04-17T08:50:33.900099+0000","last_active":"2024-04-17T08:50:33.901732+0000","last_peered":"2024-04-17T08:50:33.901732+0000","last_clean":"2024-04-17T08:50:33.901732+0000","last_became_active":"2024-04-17T08:50:33.899785+0000","last_became_peered":"2024-04-17T08:50:33.899785+0000","last_unstale":"2024-04-17T08:50:33.901732+0000","last_undegraded":"2024-04-17T08:50:33.901732+0000","last_fullsized":"2024-04-17T08:50:33.901732+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:58:58.195690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"229'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920831+0000","last_change":"2024-04-17T08:50:32.892710+0000","last_active":"2024-04-17T08:50:33.920831+0000","last_peered":"2024-04-17T08:50:33.920831+0000","last_clean":"2024-04-17T08:50:33.920831+0000","last_became_active":"2024-04-17T08:50:32.892204+0000","last_became_peered":"2024-04-17T08:50:32.892204+0000","last_unstale":"2024-04-17T08:50:33.920831+0000","last_undegraded":"2024-04-17T08:50:33.920831+0000","last_fullsized":"2024-04-17T08:50:33.920831+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:01.085554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906193+0000","last_change":"2024-04-17T08:50:33.904088+0000","last_active":"2024-04-17T08:50:33.906193+0000","last_peered":"2024-04-17T08:50:33.906193+0000","last_clean":"2024-04-17T08:50:33.906193+0000","last_became_active":"2024-04-17T08:50:33.903730+0000","last_became_peered":"2024-04-17T08:50:33.903730+0000","last_unstale":"2024-04-17T08:50:33.906193+0000","last_undegraded":"2024-04-17T08:50:33.906193+0000","last_fullsized":"2024-04-17T08:50:33.906193+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.527681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"229'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.922108+0000","last_change":"2024-04-17T08:50:32.906064+0000","last_active":"2024-04-17T08:50:33.922108+0000","last_peered":"2024-04-17T08:50:33.922108+0000","last_clean":"2024-04-17T08:50:33.922108+0000","last_became_active":"2024-04-17T08:50:32.904959+0000","last_became_peered":"2024-04-17T08:50:32.904959+0000","last_unstale":"2024-04-17T08:50:33.922108+0000","last_undegraded":"2024-04-17T08:50:33.922108+0000","last_fullsized":"2024-04-17T08:50:33.922108+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:55.614891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.911584+0000","last_change":"2024-04-17T08:50:33.908242+0000","last_active":"2024-04-17T08:50:33.911584+0000","last_peered":"2024-04-17T08:50:33.911584+0000","last_clean":"2024-04-17T08:50:33.911584+0000","last_became_active":"2024-04-17T08:50:33.907288+0000","last_became_peered":"2024-04-17T08:50:33.907288+0000","last_unstale":"2024-04-17T08:50:33.911584+0000","last_undegraded":"2024-04-17T08:50:33.911584+0000","last_fullsized":"2024-04-17T08:50:33.911584+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:32.731268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919180+0000","last_change":"2024-04-17T08:50:32.891648+0000","last_active":"2024-04-17T08:50:33.919180+0000","last_peered":"2024-04-17T08:50:33.919180+0000","last_clean":"2024-04-17T08:50:33.919180+0000","last_became_active":"2024-04-17T08:50:32.890804+0000","last_became_peered":"2024-04-17T08:50:32.890804+0000","last_unstale":"2024-04-17T08:50:33.919180+0000","last_undegraded":"2024-04-17T08:50:33.919180+0000","last_fullsized":"2024-04-17T08:50:33.919180+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:50.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.913658+0000","last_change":"2024-04-17T08:50:33.909416+0000","last_active":"2024-04-17T08:50:33.913658+0000","last_peered":"2024-04-17T08:50:33.913658+0000","last_clean":"2024-04-17T08:50:33.913658+0000","last_became_active":"2024-04-17T08:50:33.906483+0000","last_became_peered":"2024-04-17T08:50:33.906483+0000","last_unstale":"2024-04-17T08:50:33.913658+0000","last_undegraded":"2024-04-17T08:50:33.913658+0000","last_fullsized":"2024-04-17T08:50:33.913658+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:01.787161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.871340+0000","last_change":"2024-04-17T08:50:32.896932+0000","last_active":"2024-04-17T08:50:33.871340+0000","last_peered":"2024-04-17T08:50:33.871340+0000","last_clean":"2024-04-17T08:50:33.871340+0000","last_became_active":"2024-04-17T08:50:32.895134+0000","last_became_peered":"2024-04-17T08:50:32.895134+0000","last_unstale":"2024-04-17T08:50:33.871340+0000","last_undegraded":"2024-04-17T08:50:33.871340+0000","last_fullsized":"2024-04-17T08:50:33.871340+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:02.447617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904700+0000","last_change":"2024-04-17T08:50:33.896023+0000","last_active":"2024-04-17T08:50:33.904700+0000","last_peered":"2024-04-17T08:50:33.904700+0000","last_clean":"2024-04-17T08:50:33.904700+0000","last_became_active":"2024-04-17T08:50:33.895404+0000","last_became_peered":"2024-04-17T08:50:33.895404+0000","last_unstale":"2024-04-17T08:50:33.904700+0000","last_undegraded":"2024-04-17T08:50:33.904700+0000","last_fullsized":"2024-04-17T08:50:33.904700+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:28.665531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.889472+0000","last_change":"2024-04-17T08:50:32.895429+0000","last_active":"2024-04-17T08:50:33.889472+0000","last_peered":"2024-04-17T08:50:33.889472+0000","last_clean":"2024-04-17T08:50:33.889472+0000","last_became_active":"2024-04-17T08:50:32.894971+0000","last_became_peered":"2024-04-17T08:50:32.894971+0000","last_unstale":"2024-04-17T08:50:33.889472+0000","last_undegraded":"2024-04-17T08:50:33.889472+0000","last_fullsized":"2024-04-17T08:50:33.889472+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:22.318570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.892729+0000","last_change":"2024-04-17T08:50:33.889134+0000","last_active":"2024-04-17T08:50:33.892729+0000","last_peered":"2024-04-17T08:50:33.892729+0000","last_clean":"2024-04-17T08:50:33.892729+0000","last_became_active":"2024-04-17T08:50:33.886115+0000","last_became_peered":"2024-04-17T08:50:33.886115+0000","last_unstale":"2024-04-17T08:50:33.892729+0000","last_undegraded":"2024-04-17T08:50:33.892729+0000","last_fullsized":"2024-04-17T08:50:33.892729+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:39:06.890071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"229'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920704+0000","last_change":"2024-04-17T08:50:32.903713+0000","last_active":"2024-04-17T08:50:33.920704+0000","last_peered":"2024-04-17T08:50:33.920704+0000","last_clean":"2024-04-17T08:50:33.920704+0000","last_became_active":"2024-04-17T08:50:32.903403+0000","last_became_peered":"2024-04-17T08:50:32.903403+0000","last_unstale":"2024-04-17T08:50:33.920704+0000","last_undegraded":"2024-04-17T08:50:33.920704+0000","last_fullsized":"2024-04-17T08:50:33.920704+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:09.595683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897870+0000","last_change":"2024-04-17T08:50:33.887711+0000","last_active":"2024-04-17T08:50:33.897870+0000","last_peered":"2024-04-17T08:50:33.897870+0000","last_clean":"2024-04-17T08:50:33.897870+0000","last_became_active":"2024-04-17T08:50:33.887362+0000","last_became_peered":"2024-04-17T08:50:33.887362+0000","last_unstale":"2024-04-17T08:50:33.897870+0000","last_undegraded":"2024-04-17T08:50:33.897870+0000","last_fullsized":"2024-04-17T08:50:33.897870+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:17.850270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.918596+0000","last_change":"2024-04-17T08:50:32.891171+0000","last_active":"2024-04-17T08:50:33.918596+0000","last_peered":"2024-04-17T08:50:33.918596+0000","last_clean":"2024-04-17T08:50:33.918596+0000","last_became_active":"2024-04-17T08:50:32.888285+0000","last_became_peered":"2024-04-17T08:50:32.888285+0000","last_unstale":"2024-04-17T08:50:33.918596+0000","last_undegraded":"2024-04-17T08:50:33.918596+0000","last_fullsized":"2024-04-17T08:50:33.918596+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.387644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.907850+0000","last_change":"2024-04-17T08:50:33.907099+0000","last_active":"2024-04-17T08:50:33.907850+0000","last_peered":"2024-04-17T08:50:33.907850+0000","last_clean":"2024-04-17T08:50:33.907850+0000","last_became_active":"2024-04-17T08:50:33.906802+0000","last_became_peered":"2024-04-17T08:50:33.906802+0000","last_unstale":"2024-04-17T08:50:33.907850+0000","last_undegraded":"2024-04-17T08:50:33.907850+0000","last_fullsized":"2024-04-17T08:50:33.907850+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:51.076665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919571+0000","last_change":"2024-04-17T08:50:32.901050+0000","last_active":"2024-04-17T08:50:33.919571+0000","last_peered":"2024-04-17T08:50:33.919571+0000","last_clean":"2024-04-17T08:50:33.919571+0000","last_became_active":"2024-04-17T08:50:32.900689+0000","last_became_peered":"2024-04-17T08:50:32.900689+0000","last_unstale":"2024-04-17T08:50:33.919571+0000","last_undegraded":"2024-04-17T08:50:33.919571+0000","last_fullsized":"2024-04-17T08:50:33.919571+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:39:51.234349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.906044+0000","last_change":"2024-04-17T08:50:33.899313+0000","last_active":"2024-04-17T08:50:33.906044+0000","last_peered":"2024-04-17T08:50:33.906044+0000","last_clean":"2024-04-17T08:50:33.906044+0000","last_became_active":"2024-04-17T08:50:33.899005+0000","last_became_peered":"2024-04-17T08:50:33.899005+0000","last_unstale":"2024-04-17T08:50:33.906044+0000","last_undegraded":"2024-04-17T08:50:33.906044+0000","last_fullsized":"2024-04-17T08:50:33.906044+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:54.543895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920521+0000","last_change":"2024-04-17T08:50:32.896384+0000","last_active":"2024-04-17T08:50:33.920521+0000","last_peered":"2024-04-17T08:50:33.920521+0000","last_clean":"2024-04-17T08:50:33.920521+0000","last_became_active":"2024-04-17T08:50:32.893495+0000","last_became_peered":"2024-04-17T08:50:32.893495+0000","last_unstale":"2024-04-17T08:50:33.920521+0000","last_undegraded":"2024-04-17T08:50:33.920521+0000","last_fullsized":"2024-04-17T08:50:33.920521+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.852683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.912596+0000","last_change":"2024-04-17T08:50:33.905253+0000","last_active":"2024-04-17T08:50:33.912596+0000","last_peered":"2024-04-17T08:50:33.912596+0000","last_clean":"2024-04-17T08:50:33.912596+0000","last_became_active":"2024-04-17T08:50:33.904731+0000","last_became_peered":"2024-04-17T08:50:33.904731+0000","last_unstale":"2024-04-17T08:50:33.912596+0000","last_undegraded":"2024-04-17T08:50:33.912596+0000","last_fullsized":"2024-04-17T08:50:33.912596+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:30:21.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886095+0000","last_change":"2024-04-17T08:50:32.903243+0000","last_active":"2024-04-17T08:50:33.886095+0000","last_peered":"2024-04-17T08:50:33.886095+0000","last_clean":"2024-04-17T08:50:33.886095+0000","last_became_active":"2024-04-17T08:50:32.902864+0000","last_became_peered":"2024-04-17T08:50:32.902864+0000","last_unstale":"2024-04-17T08:50:33.886095+0000","last_undegraded":"2024-04-17T08:50:33.886095+0000","last_fullsized":"2024-04-17T08:50:33.886095+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:20.222164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.910822+0000","last_change":"2024-04-17T08:50:33.897197+0000","last_active":"2024-04-17T08:50:33.910822+0000","last_peered":"2024-04-17T08:50:33.910822+0000","last_clean":"2024-04-17T08:50:33.910822+0000","last_became_active":"2024-04-17T08:50:33.895389+0000","last_became_peered":"2024-04-17T08:50:33.895389+0000","last_unstale":"2024-04-17T08:50:33.910822+0000","last_undegraded":"2024-04-17T08:50:33.910822+0000","last_fullsized":"2024-04-17T08:50:33.910822+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:57:37.711365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.920894+0000","last_change":"2024-04-17T08:50:32.898032+0000","last_active":"2024-04-17T08:50:33.920894+0000","last_peered":"2024-04-17T08:50:33.920894+0000","last_clean":"2024-04-17T08:50:33.920894+0000","last_became_active":"2024-04-17T08:50:32.897755+0000","last_became_peered":"2024-04-17T08:50:32.897755+0000","last_unstale":"2024-04-17T08:50:33.920894+0000","last_undegraded":"2024-04-17T08:50:33.920894+0000","last_fullsized":"2024-04-17T08:50:33.920894+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:29.382845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.903922+0000","last_change":"2024-04-17T08:50:33.897335+0000","last_active":"2024-04-17T08:50:33.903922+0000","last_peered":"2024-04-17T08:50:33.903922+0000","last_clean":"2024-04-17T08:50:33.903922+0000","last_became_active":"2024-04-17T08:50:33.897009+0000","last_became_peered":"2024-04-17T08:50:33.897009+0000","last_unstale":"2024-04-17T08:50:33.903922+0000","last_undegraded":"2024-04-17T08:50:33.903922+0000","last_fullsized":"2024-04-17T08:50:33.903922+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:15.768477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.866878+0000","last_change":"2024-04-17T08:50:32.902321+0000","last_active":"2024-04-17T08:50:33.866878+0000","last_peered":"2024-04-17T08:50:33.866878+0000","last_clean":"2024-04-17T08:50:33.866878+0000","last_became_active":"2024-04-17T08:50:32.901781+0000","last_became_peered":"2024-04-17T08:50:32.901781+0000","last_unstale":"2024-04-17T08:50:33.866878+0000","last_undegraded":"2024-04-17T08:50:33.866878+0000","last_fullsized":"2024-04-17T08:50:33.866878+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:04.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.897220+0000","last_change":"2024-04-17T08:50:33.884383+0000","last_active":"2024-04-17T08:50:33.897220+0000","last_peered":"2024-04-17T08:50:33.897220+0000","last_clean":"2024-04-17T08:50:33.897220+0000","last_became_active":"2024-04-17T08:50:33.883473+0000","last_became_peered":"2024-04-17T08:50:33.883473+0000","last_unstale":"2024-04-17T08:50:33.897220+0000","last_undegraded":"2024-04-17T08:50:33.897220+0000","last_fullsized":"2024-04-17T08:50:33.897220+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:15.432457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.864807+0000","last_change":"2024-04-17T08:50:32.896102+0000","last_active":"2024-04-17T08:50:33.864807+0000","last_peered":"2024-04-17T08:50:33.864807+0000","last_clean":"2024-04-17T08:50:33.864807+0000","last_became_active":"2024-04-17T08:50:32.895808+0000","last_became_peered":"2024-04-17T08:50:32.895808+0000","last_unstale":"2024-04-17T08:50:33.864807+0000","last_undegraded":"2024-04-17T08:50:33.864807+0000","last_fullsized":"2024-04-17T08:50:33.864807+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:38.551687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.895038+0000","last_change":"2024-04-17T08:50:33.892798+0000","last_active":"2024-04-17T08:50:33.895038+0000","last_peered":"2024-04-17T08:50:33.895038+0000","last_clean":"2024-04-17T08:50:33.895038+0000","last_became_active":"2024-04-17T08:50:33.891130+0000","last_became_peered":"2024-04-17T08:50:33.891130+0000","last_unstale":"2024-04-17T08:50:33.895038+0000","last_undegraded":"2024-04-17T08:50:33.895038+0000","last_fullsized":"2024-04-17T08:50:33.895038+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:07.536049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.877050+0000","last_change":"2024-04-17T08:50:32.894904+0000","last_active":"2024-04-17T08:50:33.877050+0000","last_peered":"2024-04-17T08:50:33.877050+0000","last_clean":"2024-04-17T08:50:33.877050+0000","last_became_active":"2024-04-17T08:50:32.894624+0000","last_became_peered":"2024-04-17T08:50:32.894624+0000","last_unstale":"2024-04-17T08:50:33.877050+0000","last_undegraded":"2024-04-17T08:50:33.877050+0000","last_fullsized":"2024-04-17T08:50:33.877050+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:22.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.902378+0000","last_change":"2024-04-17T08:50:33.899749+0000","last_active":"2024-04-17T08:50:33.902378+0000","last_peered":"2024-04-17T08:50:33.902378+0000","last_clean":"2024-04-17T08:50:33.902378+0000","last_became_active":"2024-04-17T08:50:33.898270+0000","last_became_peered":"2024-04-17T08:50:33.898270+0000","last_unstale":"2024-04-17T08:50:33.902378+0000","last_undegraded":"2024-04-17T08:50:33.902378+0000","last_fullsized":"2024-04-17T08:50:33.902378+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:10:44.340926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"229'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.919805+0000","last_change":"2024-04-17T08:50:32.906744+0000","last_active":"2024-04-17T08:50:33.919805+0000","last_peered":"2024-04-17T08:50:33.919805+0000","last_clean":"2024-04-17T08:50:33.919805+0000","last_became_active":"2024-04-17T08:50:32.906445+0000","last_became_peered":"2024-04-17T08:50:32.906445+0000","last_unstale":"2024-04-17T08:50:33.919805+0000","last_undegraded":"2024-04-17T08:50:33.919805+0000","last_fullsized":"2024-04-17T08:50:33.919805+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:33.054437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894426+0000","last_change":"2024-04-17T08:50:33.892244+0000","last_active":"2024-04-17T08:50:33.894426+0000","last_peered":"2024-04-17T08:50:33.894426+0000","last_clean":"2024-04-17T08:50:33.894426+0000","last_became_active":"2024-04-17T08:50:33.891771+0000","last_became_peered":"2024-04-17T08:50:33.891771+0000","last_unstale":"2024-04-17T08:50:33.894426+0000","last_undegraded":"2024-04-17T08:50:33.894426+0000","last_fullsized":"2024-04-17T08:50:33.894426+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:18.314343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.888613+0000","last_change":"2024-04-17T08:50:32.904455+0000","last_active":"2024-04-17T08:50:33.888613+0000","last_peered":"2024-04-17T08:50:33.888613+0000","last_clean":"2024-04-17T08:50:33.888613+0000","last_became_active":"2024-04-17T08:50:32.904127+0000","last_became_peered":"2024-04-17T08:50:32.904127+0000","last_unstale":"2024-04-17T08:50:33.888613+0000","last_undegraded":"2024-04-17T08:50:33.888613+0000","last_fullsized":"2024-04-17T08:50:33.888613+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:09.473209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896574+0000","last_change":"2024-04-17T08:50:33.887506+0000","last_active":"2024-04-17T08:50:33.896574+0000","last_peered":"2024-04-17T08:50:33.896574+0000","last_clean":"2024-04-17T08:50:33.896574+0000","last_became_active":"2024-04-17T08:50:33.887071+0000","last_became_peered":"2024-04-17T08:50:33.887071+0000","last_unstale":"2024-04-17T08:50:33.896574+0000","last_undegraded":"2024-04-17T08:50:33.896574+0000","last_fullsized":"2024-04-17T08:50:33.896574+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:46:31.757149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886467+0000","last_change":"2024-04-17T08:50:32.901213+0000","last_active":"2024-04-17T08:50:33.886467+0000","last_peered":"2024-04-17T08:50:33.886467+0000","last_clean":"2024-04-17T08:50:33.886467+0000","last_became_active":"2024-04-17T08:50:32.900942+0000","last_became_peered":"2024-04-17T08:50:32.900942+0000","last_unstale":"2024-04-17T08:50:33.886467+0000","last_undegraded":"2024-04-17T08:50:33.886467+0000","last_fullsized":"2024-04-17T08:50:33.886467+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:35.477286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.894001+0000","last_change":"2024-04-17T08:50:33.892430+0000","last_active":"2024-04-17T08:50:33.894001+0000","last_peered":"2024-04-17T08:50:33.894001+0000","last_clean":"2024-04-17T08:50:33.894001+0000","last_became_active":"2024-04-17T08:50:33.892133+0000","last_became_peered":"2024-04-17T08:50:33.892133+0000","last_unstale":"2024-04-17T08:50:33.894001+0000","last_undegraded":"2024-04-17T08:50:33.894001+0000","last_fullsized":"2024-04-17T08:50:33.894001+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:19.182944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.887490+0000","last_change":"2024-04-17T08:50:32.906189+0000","last_active":"2024-04-17T08:50:33.887490+0000","last_peered":"2024-04-17T08:50:33.887490+0000","last_clean":"2024-04-17T08:50:33.887490+0000","last_became_active":"2024-04-17T08:50:32.905898+0000","last_became_peered":"2024-04-17T08:50:32.905898+0000","last_unstale":"2024-04-17T08:50:33.887490+0000","last_undegraded":"2024-04-17T08:50:33.887490+0000","last_fullsized":"2024-04-17T08:50:33.887490+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:24.678638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.884296+0000","last_change":"2024-04-17T08:50:33.879546+0000","last_active":"2024-04-17T08:50:33.884296+0000","last_peered":"2024-04-17T08:50:33.884296+0000","last_clean":"2024-04-17T08:50:33.884296+0000","last_became_active":"2024-04-17T08:50:33.878778+0000","last_became_peered":"2024-04-17T08:50:33.878778+0000","last_unstale":"2024-04-17T08:50:33.884296+0000","last_undegraded":"2024-04-17T08:50:33.884296+0000","last_fullsized":"2024-04-17T08:50:33.884296+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:28.759494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.886708+0000","last_change":"2024-04-17T08:50:32.903970+0000","last_active":"2024-04-17T08:50:33.886708+0000","last_peered":"2024-04-17T08:50:33.886708+0000","last_clean":"2024-04-17T08:50:33.886708+0000","last_became_active":"2024-04-17T08:50:32.903620+0000","last_became_peered":"2024-04-17T08:50:32.903620+0000","last_unstale":"2024-04-17T08:50:33.886708+0000","last_undegraded":"2024-04-17T08:50:33.886708+0000","last_fullsized":"2024-04-17T08:50:33.886708+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:33.640417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899956+0000","last_change":"2024-04-17T08:50:33.892519+0000","last_active":"2024-04-17T08:50:33.899956+0000","last_peered":"2024-04-17T08:50:33.899956+0000","last_clean":"2024-04-17T08:50:33.899956+0000","last_became_active":"2024-04-17T08:50:33.892264+0000","last_became_peered":"2024-04-17T08:50:33.892264+0000","last_unstale":"2024-04-17T08:50:33.899956+0000","last_undegraded":"2024-04-17T08:50:33.899956+0000","last_fullsized":"2024-04-17T08:50:33.899956+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:18:16.218585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.884636+0000","last_change":"2024-04-17T08:50:32.891370+0000","last_active":"2024-04-17T08:50:33.884636+0000","last_peered":"2024-04-17T08:50:33.884636+0000","last_clean":"2024-04-17T08:50:33.884636+0000","last_became_active":"2024-04-17T08:50:32.888996+0000","last_became_peered":"2024-04-17T08:50:32.888996+0000","last_unstale":"2024-04-17T08:50:33.884636+0000","last_undegraded":"2024-04-17T08:50:33.884636+0000","last_fullsized":"2024-04-17T08:50:33.884636+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:31.815168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.904679+0000","last_change":"2024-04-17T08:50:33.895678+0000","last_active":"2024-04-17T08:50:33.904679+0000","last_peered":"2024-04-17T08:50:33.904679+0000","last_clean":"2024-04-17T08:50:33.904679+0000","last_became_active":"2024-04-17T08:50:33.895010+0000","last_became_peered":"2024-04-17T08:50:33.895010+0000","last_unstale":"2024-04-17T08:50:33.904679+0000","last_undegraded":"2024-04-17T08:50:33.904679+0000","last_fullsized":"2024-04-17T08:50:33.904679+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:52:52.033934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.872055+0000","last_change":"2024-04-17T08:50:32.902675+0000","last_active":"2024-04-17T08:50:33.872055+0000","last_peered":"2024-04-17T08:50:33.872055+0000","last_clean":"2024-04-17T08:50:33.872055+0000","last_became_active":"2024-04-17T08:50:32.900858+0000","last_became_peered":"2024-04-17T08:50:32.900858+0000","last_unstale":"2024-04-17T08:50:33.872055+0000","last_undegraded":"2024-04-17T08:50:33.872055+0000","last_fullsized":"2024-04-17T08:50:33.872055+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:45.638814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.896638+0000","last_change":"2024-04-17T08:50:33.884434+0000","last_active":"2024-04-17T08:50:33.896638+0000","last_peered":"2024-04-17T08:50:33.896638+0000","last_clean":"2024-04-17T08:50:33.896638+0000","last_became_active":"2024-04-17T08:50:33.884075+0000","last_became_peered":"2024-04-17T08:50:33.884075+0000","last_unstale":"2024-04-17T08:50:33.896638+0000","last_undegraded":"2024-04-17T08:50:33.896638+0000","last_fullsized":"2024-04-17T08:50:33.896638+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:23:51.344240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.879492+0000","last_change":"2024-04-17T08:50:32.894323+0000","last_active":"2024-04-17T08:50:33.879492+0000","last_peered":"2024-04-17T08:50:33.879492+0000","last_clean":"2024-04-17T08:50:33.879492+0000","last_became_active":"2024-04-17T08:50:32.893659+0000","last_became_peered":"2024-04-17T08:50:32.893659+0000","last_unstale":"2024-04-17T08:50:33.879492+0000","last_undegraded":"2024-04-17T08:50:33.879492+0000","last_fullsized":"2024-04-17T08:50:33.879492+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:16.424440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":9,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.899445+0000","last_change":"2024-04-17T08:50:33.896994+0000","last_active":"2024-04-17T08:50:33.899445+0000","last_peered":"2024-04-17T08:50:33.899445+0000","last_clean":"2024-04-17T08:50:33.899445+0000","last_became_active":"2024-04-17T08:50:33.896640+0000","last_became_peered":"2024-04-17T08:50:33.896640+0000","last_unstale":"2024-04-17T08:50:33.899445+0000","last_undegraded":"2024-04-17T08:50:33.899445+0000","last_fullsized":"2024-04-17T08:50:33.899445+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:38.484472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"229'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.915027+0000","last_change":"2024-04-17T08:50:32.893722+0000","last_active":"2024-04-17T08:50:33.915027+0000","last_peered":"2024-04-17T08:50:33.915027+0000","last_clean":"2024-04-17T08:50:33.915027+0000","last_became_active":"2024-04-17T08:50:32.893404+0000","last_became_peered":"2024-04-17T08:50:32.893404+0000","last_unstale":"2024-04-17T08:50:33.915027+0000","last_undegraded":"2024-04-17T08:50:33.915027+0000","last_fullsized":"2024-04-17T08:50:33.915027+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:34.249904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"217'354","reported_seq":1446,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-17T08:50:33.885648+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:50:33.885648+0000","last_peered":"2024-04-17T08:50:33.885648+0000","last_clean":"2024-04-17T08:50:33.885648+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:50:33.885648+0000","last_undegraded":"2024-04-17T08:50:33.885648+0000","last_fullsized":"2024-04-17T08:50:33.885648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543293,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52628,"kb_used_data":3392,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319212,"statfs":{"total":96636764160,"available":96582873088,"internally_reserved":0,"allocated":3473408,"data_stored":1993024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543293,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42292,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329548,"statfs":{"total":96636764160,"available":96593457152,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543292,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29828,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342012,"statfs":{"total":96636764160,"available":96606220288,"internally_reserved":0,"allocated":2871296,"data_stored":1401626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543292,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64500,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307340,"statfs":{"total":96636764160,"available":96570716160,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47880,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323960,"statfs":{"total":96636764160,"available":96587735040,"internally_reserved":0,"allocated":2879488,"data_stored":1401226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543293,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52036,"kb_used_data":2800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319804,"statfs":{"total":96636764160,"available":96583479296,"internally_reserved":0,"allocated":2867200,"data_stored":1401044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543294,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60336,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311504,"statfs":{"total":96636764160,"available":96574980096,"internally_reserved":0,"allocated":2854912,"data_stored":1394241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543292,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60920,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310920,"statfs":{"total":96636764160,"available":96574382080,"internally_reserved":0,"allocated":3448832,"data_stored":1984609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:50:38.956 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:50:38.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.235+0000 7f0b75600640 1 Processor -- start 2024-04-17T08:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.235+0000 7f0b75600640 1 -- start start 2024-04-17T08:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b75600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 0x7f0b700b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b75600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0b700b25f0 0x7f0b70142020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b75600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b70072160 con 0x7f0b7013f6d0 2024-04-17T08:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b75600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0b7015cbe0 con 0x7f0b700b25f0 2024-04-17T08:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b75600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7015cea0 con 0x7f0b70071d00 2024-04-17T08:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b6fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0b7013f6d0 0x7f0b700cd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47972/0 (socket says 172.21.15.67:47972) 2024-04-17T08:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.237+0000 7f0b6fe00640 1 -- 172.21.15.67:0/1735269048 learned_addr learned my addr 172.21.15.67:0/1735269048 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278979006 0 0) 0x7f0b60001110 con 0x7f0b7013f6d0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b54003520 con 0x7f0b7013f6d0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 0x7f0b700b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b74c00640 1 --2- 172.21.15.67:0/1735269048 >> v2:172.21.15.187:3300/0 conn(0x7f0b700b25f0 0x7f0b70142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3472957891 0 0) 0x7f0b54003520 con 0x7f0b7013f6d0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 msgr2=0x7f0b700b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 0x7f0b700b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 >> v2:172.21.15.187:3300/0 conn(0x7f0b700b25f0 msgr2=0x7f0b70142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 --2- 172.21.15.67:0/1735269048 >> v2:172.21.15.187:3300/0 conn(0x7f0b700b25f0 0x7f0b70142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7015d180 con 0x7f0b7013f6d0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0b60001840 con 0x7f0b7013f6d0 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 >> v1:172.21.15.67:6789/0 conn(0x7f0b7013f6d0 legacy=0x7f0b700cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b540039b0 0x7f0b54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54004800 0x7f0b54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 0x7f0b540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7015cbe0 con 0x7f0b54004800 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7015cea0 con 0x7f0b540039b0 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.238+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b5400bac0 con 0x7f0b70071d00 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54004800 0x7f0b54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 -- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 msgr2=0x7f0b540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b70071d00 0x7f0b540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 -- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b540039b0 msgr2=0x7f0b54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b540039b0 0x7f0b54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.239+0000 7f0b6f400640 1 -- 172.21.15.67:0/1735269048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b54003520 con 0x7f0b54004800 2024-04-17T08:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.240+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54004800 0x7f0b54004c80 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00ba00 tx=0x7f0b54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8726a7e459188a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.240+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c036020 con 0x7f0b54004800 2024-04-17T08:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.240+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5c032630 con 0x7f0b54004800 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.240+0000 7f0b6ea00640 1 -- 172.21.15.67:0/1735269048 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c00b600 con 0x7f0b54004800 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 -- 172.21.15.67:0/1735269048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54004800 msgr2=0x7f0b54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 --2- 172.21.15.67:0/1735269048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54004800 0x7f0b54004c80 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00ba00 tx=0x7f0b54005830 comp rx=0 tx=0).stop 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b74c00640 1 -- 172.21.15.67:0/1735269048 reap_dead start 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 -- 172.21.15.67:0/1735269048 shutdown_connections 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 -- 172.21.15.67:0/1735269048 >> 172.21.15.67:0/1735269048 conn(0x7f0b70067d10 msgr2=0x7f0b700d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 -- 172.21.15.67:0/1735269048 shutdown_connections 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.241+0000 7f0b75600640 1 -- 172.21.15.67:0/1735269048 wait complete. 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.243+0000 7f0b75600640 1 Processor -- start 2024-04-17T08:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.243+0000 7f0b75600640 1 -- start start 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b54004800 0x7f0b70154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b701f15d0 0x7f0b70141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b700d1120 con 0x7f0b700b10c0 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7015d740 con 0x7f0b701f15d0 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.244+0000 7f0b75600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7015da20 con 0x7f0b54004800 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52840/0 (socket says 172.21.15.67:52840) 2024-04-17T08:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 -- 172.21.15.67:0/1187677348 learned_addr learned my addr 172.21.15.67:0/1187677348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b54004800 0x7f0b70154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b6f400640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b701f15d0 0x7f0b70141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 -- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b54004800 msgr2=0x7f0b70154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b54004800 0x7f0b70154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 -- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b701f15d0 msgr2=0x7f0b70141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b701f15d0 0x7f0b70141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.245+0000 7f0b74c00640 1 -- 172.21.15.67:0/1187677348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b701f4660 con 0x7f0b700b10c0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b74c00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0b700afa70 tx=0x7f0b6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6800ed90 con 0x7f0b700b10c0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b68019070 con 0x7f0b700b10c0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68014660 con 0x7f0b700b10c0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b701f4940 con 0x7f0b700b10c0 2024-04-17T08:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.246+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b701f4f00 con 0x7f0b700b10c0 2024-04-17T08:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.247+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b70068a10 con 0x7f0b700b10c0 2024-04-17T08:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.248+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f0b68014800 con 0x7f0b700b10c0 2024-04-17T08:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.249+0000 7f0b67e00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 0x7f0b5006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.249+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f0b6808bf80 con 0x7f0b700b10c0 2024-04-17T08:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.250+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 0x7f0b5006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.251+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 0x7f0b5006ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b1cb0 tx=0x7f0b60001c00 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.253+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0b68023020 con 0x7f0b700b10c0 2024-04-17T08:50:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.535+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0b70154640 con 0x7f0b700b10c0 2024-04-17T08:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.539+0000 7f0b67e00640 1 -- 172.21.15.67:0/1187677348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) ==== 190+0+147 (secure 0 0 0) 0x7f0b6805a270 con 0x7f0b700b10c0 2024-04-17T08:50:39.537 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:50:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:50:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:50:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:50:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:50:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.541+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 msgr2=0x7f0b5006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.541+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 0x7f0b5006ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b1cb0 tx=0x7f0b60001c00 comp rx=0 tx=0).stop 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.541+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 msgr2=0x7f0b700b1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.541+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0b700afa70 tx=0x7f0b6800e4b0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 shutdown_connections 2024-04-17T08:50:39.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0b5006ca40 0x7f0b5006ef00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b700b10c0 0x7f0b700b1540 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b701f15d0 0x7f0b70141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 --2- 172.21.15.67:0/1187677348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b54004800 0x7f0b70154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 >> 172.21.15.67:0/1187677348 conn(0x7f0b70067d10 msgr2=0x7f0b7006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 shutdown_connections 2024-04-17T08:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.542+0000 7f0b75600640 1 -- 172.21.15.67:0/1187677348 wait complete. 2024-04-17T08:50:39.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:39.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.784+0000 7ff833200640 1 Processor -- start 2024-04-17T08:50:39.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.784+0000 7ff833200640 1 -- start start 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.786+0000 7ff833200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff82c164480 0x7ff82c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.786+0000 7ff833200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 0x7ff82c1634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.786+0000 7ff833200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82c0b3ba0 con 0x7ff82c15d530 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.786+0000 7ff833200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff82c06c270 con 0x7ff82c164480 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.786+0000 7ff833200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff82c06f6e0 con 0x7ff82c06be00 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff82c164480 0x7ff82c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff82c164480 0x7ff82c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52846/0 (socket says 172.21.15.67:52846) 2024-04-17T08:50:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 -- 172.21.15.67:0/2592631874 learned_addr learned my addr 172.21.15.67:0/2592631874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 -- 172.21.15.67:0/2592631874 >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 msgr2=0x7ff82c1634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff832800640 1 --2- 172.21.15.67:0/2592631874 >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 0x7ff82c1634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 --2- 172.21.15.67:0/2592631874 >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 0x7ff82c1634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 -- 172.21.15.67:0/2592631874 >> v1:172.21.15.67:6789/0 conn(0x7ff82c15d530 legacy=0x7ff82c0b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff831400640 1 -- 172.21.15.67:0/2592631874 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c06ead0 con 0x7ff82c164480 2024-04-17T08:50:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.787+0000 7ff832800640 1 --2- 172.21.15.67:0/2592631874 >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 0x7ff82c1634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.788+0000 7ff831400640 1 --2- 172.21.15.67:0/2592631874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c06b8c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7ff82c06e430 tx=0x7ff81c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ee59b5ecd0db7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.789+0000 7ff830a00640 1 -- 172.21.15.67:0/2592631874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c034070 con 0x7ff82c164480 2024-04-17T08:50:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.789+0000 7ff830a00640 1 -- 172.21.15.67:0/2592631874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c030880 con 0x7ff82c164480 2024-04-17T08:50:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.789+0000 7ff830a00640 1 -- 172.21.15.67:0/2592631874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c030c50 con 0x7ff82c164480 2024-04-17T08:50:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.789+0000 7ff833200640 1 -- 172.21.15.67:0/2592631874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 msgr2=0x7ff82c06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.789+0000 7ff833200640 1 --2- 172.21.15.67:0/2592631874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c06b8c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7ff82c06e430 tx=0x7ff81c02b9c0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 -- 172.21.15.67:0/2592631874 shutdown_connections 2024-04-17T08:50:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 --2- 172.21.15.67:0/2592631874 >> v2:172.21.15.187:3300/0 conn(0x7ff82c06be00 0x7ff82c1634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 --2- 172.21.15.67:0/2592631874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c06b8c0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 -- 172.21.15.67:0/2592631874 >> 172.21.15.67:0/2592631874 conn(0x7ff82c067d10 msgr2=0x7ff82c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 -- 172.21.15.67:0/2592631874 shutdown_connections 2024-04-17T08:50:39.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.790+0000 7ff833200640 1 -- 172.21.15.67:0/2592631874 wait complete. 2024-04-17T08:50:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.792+0000 7ff833200640 1 Processor -- start 2024-04-17T08:50:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.792+0000 7ff833200640 1 -- start start 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 0x7ff82c156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 0x7ff82c0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c06f370 con 0x7ff82c164480 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c06f090 con 0x7ff82c15d530 2024-04-17T08:50:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff833200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c06fd60 con 0x7ff82c06be00 2024-04-17T08:50:39.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff832800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff832800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52862/0 (socket says 172.21.15.67:52862) 2024-04-17T08:50:39.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff832800640 1 -- 172.21.15.67:0/4242723638 learned_addr learned my addr 172.21.15.67:0/4242723638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff831400640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 0x7ff82c0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff832800640 1 -- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 msgr2=0x7ff82c156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.793+0000 7ff831e00640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 0x7ff82c156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff832800640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 0x7ff82c156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff832800640 1 -- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 msgr2=0x7ff82c0cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff832800640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 0x7ff82c0cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff832800640 1 -- 172.21.15.67:0/4242723638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c15ef40 con 0x7ff82c164480 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff831400640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 0x7ff82c0cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff831e00640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 0x7ff82c156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff832800640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7ff82800cef0 tx=0x7ff82800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82801a070 con 0x7ff82c164480 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff82800d860 con 0x7ff82c164480 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82800dc50 con 0x7ff82c164480 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c15f220 con 0x7ff82c164480 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.794+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c15f7e0 con 0x7ff82c164480 2024-04-17T08:50:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.795+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82c068a10 con 0x7ff82c164480 2024-04-17T08:50:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.797+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7ff82801d070 con 0x7ff82c164480 2024-04-17T08:50:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.797+0000 7ff826000640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 0x7ff81006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.798+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7ff82808c5e0 con 0x7ff82c164480 2024-04-17T08:50:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.798+0000 7ff831e00640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 0x7ff81006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:39.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.799+0000 7ff831e00640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 0x7ff81006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff820001680 tx=0x7ff820009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:39.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:39.802+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff82805a9a0 con 0x7ff82c164480 2024-04-17T08:50:40.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.049+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff82c06bb80 con 0x7ff82c164480 2024-04-17T08:50:40.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.050+0000 7ff826000640 1 -- 172.21.15.67:0/4242723638 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5221 (secure 0 0 0) 0x7ff82805a2a0 con 0x7ff82c164480 2024-04-17T08:50:40.048 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:40.048 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11025,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13}]} 2024-04-17T08:50:40.049 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 110 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.052+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 msgr2=0x7ff81006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.052+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 0x7ff81006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff820001680 tx=0x7ff820009040 comp rx=0 tx=0).stop 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.052+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 msgr2=0x7ff82c164f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.052+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7ff82800cef0 tx=0x7ff82800a710 comp rx=0 tx=0).stop 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 shutdown_connections 2024-04-17T08:50:40.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7ff81006ca10 0x7ff81006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff82c164480 0x7ff82c164f80 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff82c15d530 0x7ff82c0cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 --2- 172.21.15.67:0/4242723638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff82c06be00 0x7ff82c156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 >> 172.21.15.67:0/4242723638 conn(0x7ff82c067d10 msgr2=0x7ff82c0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 shutdown_connections 2024-04-17T08:50:40.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.053+0000 7ff833200640 1 -- 172.21.15.67:0/4242723638 wait complete. 2024-04-17T08:50:40.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:50:33.371129+0000', 'modified': '2024-04-17T08:50:36.401776+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11028}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11028': {'gid': 11028, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.67:6835/3533295594', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 3533295594}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3533295594}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11028, 'qdb_cluster': [11028]} 2024-04-17T08:50:40.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:50:40.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T08:50:40.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.704+0000 7fb782a00640 1 Processor -- start 2024-04-17T08:50:40.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.705+0000 7fb782a00640 1 -- start start 2024-04-17T08:50:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.706+0000 7fb782a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb77c15ee20 0x7fb77c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.706+0000 7fb782a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb77c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.706+0000 7fb782a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb77c0d4140 con 0x7fb77c0d3350 2024-04-17T08:50:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.706+0000 7fb782a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb77c0b83b0 con 0x7fb77c15ee20 2024-04-17T08:50:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.706+0000 7fb782a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c0d1d20 con 0x7fb77c0d18e0 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb782000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb77c0d3350 0x7fb77c067600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48000/0 (socket says 172.21.15.67:48000) 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb782000640 1 -- 172.21.15.67:0/180050503 learned_addr learned my addr 172.21.15.67:0/180050503 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb781600640 1 --2- 172.21.15.67:0/180050503 >> v2:172.21.15.187:3300/0 conn(0x7fb77c15ee20 0x7fb77c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb780c00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb77c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239690253 0 0) 0x7fb77c0d4140 con 0x7fb77c0d3350 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.707+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb764003520 con 0x7fb77c0d3350 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2620095143 0 0) 0x7fb764003520 con 0x7fb77c0d3350 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 msgr2=0x7fb77c0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb77c0d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 >> v2:172.21.15.187:3300/0 conn(0x7fb77c15ee20 msgr2=0x7fb77c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 --2- 172.21.15.67:0/180050503 >> v2:172.21.15.187:3300/0 conn(0x7fb77c15ee20 0x7fb77c0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c0d3b80 con 0x7fb77c0d3350 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb7740019f0 con 0x7fb77c0d3350 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 >> v1:172.21.15.67:6789/0 conn(0x7fb77c0d3350 legacy=0x7fb77c067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb764003a20 0x7fb764003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764004800 0x7fb764004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb7640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c0b83b0 con 0x7fb764004800 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c0d1d20 con 0x7fb764003a20 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb76400b9c0 con 0x7fb77c0d18e0 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb780c00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764004800 0x7fb764004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb781600640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb764003a20 0x7fb764003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb782000640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb7640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.708+0000 7fb780c00640 1 -- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 msgr2=0x7fb7640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.709+0000 7fb780c00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb7640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.709+0000 7fb780c00640 1 -- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb764003a20 msgr2=0x7fb764003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.709+0000 7fb780c00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb764003a20 0x7fb764003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.709+0000 7fb780c00640 1 -- 172.21.15.67:0/180050503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb764003520 con 0x7fb764004800 2024-04-17T08:50:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.709+0000 7fb782000640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb77c0d18e0 0x7fb7640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.710+0000 7fb780c00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764004800 0x7fb764004c80 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fb768009af0 tx=0x7fb764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525d4344086298bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.710+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768039070 con 0x7fb764004800 2024-04-17T08:50:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.710+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76802c6b0 con 0x7fb764004800 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.710+0000 7fb77be00640 1 -- 172.21.15.67:0/180050503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768009690 con 0x7fb764004800 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 -- 172.21.15.67:0/180050503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764004800 msgr2=0x7fb764004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 --2- 172.21.15.67:0/180050503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764004800 0x7fb764004c80 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fb768009af0 tx=0x7fb764005830 comp rx=0 tx=0).stop 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782000640 1 -- 172.21.15.67:0/180050503 reap_dead start 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 -- 172.21.15.67:0/180050503 shutdown_connections 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 -- 172.21.15.67:0/180050503 >> 172.21.15.67:0/180050503 conn(0x7fb77c06d3c0 msgr2=0x7fb77c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 -- 172.21.15.67:0/180050503 shutdown_connections 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.711+0000 7fb782a00640 1 -- 172.21.15.67:0/180050503 wait complete. 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.713+0000 7fb782a00640 1 Processor -- start 2024-04-17T08:50:40.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.713+0000 7fb782a00640 1 -- start start 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb764004800 0x7fb77c0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 0x7fb77c157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c15f200 con 0x7fb77c154140 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c0b8040 con 0x7fb77c1549e0 2024-04-17T08:50:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c0d37e0 con 0x7fb764004800 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52872/0 (socket says 172.21.15.67:52872) 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb782000640 1 -- 172.21.15.67:0/3334325258 learned_addr learned my addr 172.21.15.67:0/3334325258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.714+0000 7fb780c00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 0x7fb77c157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 -- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb764004800 msgr2=0x7fb77c0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb781600640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb764004800 0x7fb77c0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb764004800 0x7fb77c0cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 -- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 msgr2=0x7fb77c157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 0x7fb77c157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 -- 172.21.15.67:0/3334325258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c1563b0 con 0x7fb77c154140 2024-04-17T08:50:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb780c00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 0x7fb77c157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782000640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0cb9d0 tx=0x7fb77400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb779600640 1 -- 172.21.15.67:0/3334325258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774004d60 con 0x7fb77c154140 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb779600640 1 -- 172.21.15.67:0/3334325258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77400bb80 con 0x7fb77c154140 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c156690 con 0x7fb77c154140 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb779600640 1 -- 172.21.15.67:0/3334325258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774007070 con 0x7fb77c154140 2024-04-17T08:50:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.715+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c1576b0 con 0x7fb77c154140 2024-04-17T08:50:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.718+0000 7fb779600640 1 -- 172.21.15.67:0/3334325258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fb77401b020 con 0x7fb77c154140 2024-04-17T08:50:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.719+0000 7fb779600640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 0x7fb75c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.719+0000 7fb779600640 1 -- 172.21.15.67:0/3334325258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fb77408c0c0 con 0x7fb77c154140 2024-04-17T08:50:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.719+0000 7fb781600640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 0x7fb75c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.719+0000 7fb786e34740 1 Processor -- start 2024-04-17T08:50:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.719+0000 7fb786e34740 1 -- start start 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de055c620 0x560de0513c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de0514190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560de047a980 con 0x560de050eb40 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560de0421220 con 0x560de05126a0 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560de0476940 con 0x560de055c620 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.720+0000 7fb781600640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 0x7fb75c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb7700099b0 tx=0x7fb770008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52884/0 (socket says 172.21.15.67:52884) 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 -- 172.21.15.67:0/3286032956 learned_addr learned my addr 172.21.15.67:0/3286032956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb780c00640 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de0514190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 -- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de055c620 msgr2=0x560de0513c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de055c620 0x560de0513c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 -- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 msgr2=0x560de0514190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de0514190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 -- 172.21.15.67:0/3286032956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de018d2d0 con 0x560de050eb40 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.721+0000 7fb782000640 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fb774004fd0 tx=0x7fb774001e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570aac439f3efe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb76dc00640 1 -- 172.21.15.67:0/3286032956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77408a520 con 0x560de050eb40 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb76dc00640 1 -- 172.21.15.67:0/3286032956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774056180 con 0x560de050eb40 2024-04-17T08:50:40.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb76dc00640 1 -- 172.21.15.67:0/3286032956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774062460 con 0x560de050eb40 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 -- 172.21.15.67:0/3286032956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 msgr2=0x560de053e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fb774004fd0 tx=0x7fb774001e20 comp rx=0 tx=0).stop 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 -- 172.21.15.67:0/3286032956 shutdown_connections 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de050eb40 0x560de053e930 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de0514190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 --2- 172.21.15.67:0/3286032956 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de055c620 0x560de0513c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 -- 172.21.15.67:0/3286032956 >> 172.21.15.67:0/3286032956 conn(0x560de0510db0 msgr2=0x560de0559930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 -- 172.21.15.67:0/3286032956 shutdown_connections 2024-04-17T08:50:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.722+0000 7fb786e34740 1 -- 172.21.15.67:0/3286032956 wait complete. 2024-04-17T08:50:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.723+0000 7fb786e34740 1 -- start start 2024-04-17T08:50:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.723+0000 7fb786e34740 1 Processor -- start 2024-04-17T08:50:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.724+0000 7fb786e34740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.724+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 0x560de055e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0528d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb786e34740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560de0512bb0 con 0x560de05126a0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560de0434590 con 0x560de055c620 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560de0512e90 con 0x560de05288f0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb786e34740 10 client.0 _ll_register_callbacks cb 0x560de018cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52898/0 (socket says 172.21.15.67:52898) 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 -- 172.21.15.67:0/4189025057 learned_addr learned my addr 172.21.15.67:0/4189025057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 -- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 msgr2=0x560de0528d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0528d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 -- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 msgr2=0x560de055e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb780c00640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 0x560de055e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 0x560de055e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb781600640 1 -- 172.21.15.67:0/4189025057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de02576b0 con 0x560de05126a0 2024-04-17T08:50:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.725+0000 7fb780c00640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 0x560de055e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb781600640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb770015800 tx=0x7fb770009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91a7922cda75c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb762000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77001d050 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fb770014ab0 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb774056180 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7fb77002f050 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb786e34740 10 client.11178 fetch_fsmap learned FSMap version 110 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) -- 0x560de053de50 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.726+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774056180 con 0x560de05126a0 2024-04-17T08:50:40.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 110) ==== 3495+0+0 (secure 0 0 0) 0x7fb770032020 con 0x560de05126a0 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 10 client.11178 fetch_fsmap finished waiting for FSMap version 110 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 10 client.11178 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 20 client.11178 populate_metadata read hostname 'smithi067' 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de0560ac0 0x560de0578ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 4 client.11178 mds_command: new command op to 11028 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 --> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560de04883a0 con 0x560de0560ac0 2024-04-17T08:50:40.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.727+0000 7fb780c00640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de0560ac0 0x560de0578ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.728+0000 7fb780c00640 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de0560ac0 0x560de0578ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f873157ab5561f7 server_cookie=19cec03cc22324c2 in_seq=0 out_seq=0 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.728+0000 7fb762000640 10 client.11178 ms_handle_connect on v2:172.21.15.67:6834/3533295594 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.728+0000 7fb762000640 1 -- 172.21.15.67:0/4189025057 <== mds.0 v2:172.21.15.67:6834/3533295594 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560de04883a0 con 0x560de0560ac0 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.728+0000 7fb762000640 10 client.11178 handle_command_reply: tid=0 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.728+0000 7fb786e34740 1 client.11178 shutdown 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 msgr2=0x560de053d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fb770015800 tx=0x7fb770009190 comp rx=0 tx=0).stop 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 shutdown_connections 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de0560ac0 0x560de0578ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0528d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de055c620 0x560de055e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb762000640 0 client.11178 ms_handle_reset on v2:172.21.15.67:6834/3533295594 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 --2- 172.21.15.67:0/4189025057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de05126a0 0x560de053d910 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 >> 172.21.15.67:0/4189025057 conn(0x560de050eb40 msgr2=0x560de0559730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 shutdown_connections 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 1 -- 172.21.15.67:0/4189025057 wait complete. 2024-04-17T08:50:40.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.729+0000 7fb786e34740 20 client.11178 trim_cache size 0 max 16384 2024-04-17T08:50:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.749+0000 7fb786e34740 1 Processor -- start 2024-04-17T08:50:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.749+0000 7fb786e34740 1 -- start start 2024-04-17T08:50:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05126a0 0x560de0514760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05288f0 0x560de0514ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c200690 con 0x560de055c620 2024-04-17T08:50:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c0cc820 con 0x560de05288f0 2024-04-17T08:50:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.750+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c0c9310 con 0x560de05126a0 2024-04-17T08:50:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52900/0 (socket says 172.21.15.67:52900) 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 -- 172.21.15.67:0/3983863791 learned_addr learned my addr 172.21.15.67:0/3983863791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb781600640 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05126a0 0x560de0514760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 -- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05126a0 msgr2=0x560de0514760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb780c00640 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05288f0 0x560de0514ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05126a0 0x560de0514760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 -- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05288f0 msgr2=0x560de0514ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05288f0 0x560de0514ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 -- 172.21.15.67:0/3983863791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de0513ae0 con 0x560de055c620 2024-04-17T08:50:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.751+0000 7fb782000640 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fb7740013d0 tx=0x7fb77401f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13cae6bc8cf78ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb76d200640 1 -- 172.21.15.67:0/3983863791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774057e10 con 0x560de055c620 2024-04-17T08:50:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb76d200640 1 -- 172.21.15.67:0/3983863791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774087020 con 0x560de055c620 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb76d200640 1 -- 172.21.15.67:0/3983863791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774056180 con 0x560de055c620 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 -- 172.21.15.67:0/3983863791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 msgr2=0x560de0539c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fb7740013d0 tx=0x7fb77401f070 comp rx=0 tx=0).stop 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 -- 172.21.15.67:0/3983863791 shutdown_connections 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de0539c30 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05288f0 0x560de0514ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 --2- 172.21.15.67:0/3983863791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05126a0 0x560de0514760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.752+0000 7fb786e34740 1 -- 172.21.15.67:0/3983863791 >> 172.21.15.67:0/3983863791 conn(0x560de050eb40 msgr2=0x560de050a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.753+0000 7fb786e34740 1 -- 172.21.15.67:0/3983863791 shutdown_connections 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.753+0000 7fb786e34740 1 -- 172.21.15.67:0/3983863791 wait complete. 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.753+0000 7fb786e34740 1 -- start start 2024-04-17T08:50:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.754+0000 7fb786e34740 1 Processor -- start 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.754+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de05d4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0573d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560de055eef0 con 0x560de055c620 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x560de05292b0 con 0x560de05126a0 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x560de055f740 con 0x560de05288f0 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb786e34740 10 client.0 _ll_register_callbacks cb 0x7fb770009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52904/0 (socket says 172.21.15.67:52904) 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 -- 172.21.15.67:0/1200743558 learned_addr learned my addr 172.21.15.67:0/1200743558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb781600640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de05d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 -- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 msgr2=0x560de0573d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0573d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 -- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 msgr2=0x560de05d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de05d4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb782000640 1 -- 172.21.15.67:0/1200743558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de05428e0 con 0x560de055c620 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.755+0000 7fb781600640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de05d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb782000640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fb774001b60 tx=0x7fb774004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97020fabb4d6915 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774079060 con 0x560de055c620 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fb774062260 con 0x560de055c620 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb774087020 con 0x560de055c620 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7fb77407f050 con 0x560de055c620 2024-04-17T08:50:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 10 client.11184 fetch_fsmap learned FSMap version 110 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) -- 0x560de0539fd0 con 0x560de055c620 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774087020 con 0x560de055c620 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 110) ==== 3495+0+0 (secure 0 0 0) 0x7fb774082020 con 0x560de055c620 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 10 client.11184 fetch_fsmap finished waiting for FSMap version 110 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 10 client.11184 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 20 client.11184 populate_metadata read hostname 'smithi067' 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de057c570 0x560de055b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 4 client.11184 mds_command: new command op to 11028 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.756+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 --> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] -- command(tid 0: {"prefix": "status"}) -- 0x560de053b210 con 0x560de057c570 2024-04-17T08:50:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.757+0000 7fb781600640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de057c570 0x560de055b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.757+0000 7fb781600640 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de057c570 0x560de055b0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107af6cbafd5cbe1 server_cookie=d8e0deb942fcbd13 in_seq=0 out_seq=0 2024-04-17T08:50:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.757+0000 7fb762000640 10 client.11184 ms_handle_connect on v2:172.21.15.67:6834/3533295594 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb762000640 1 -- 172.21.15.67:0/1200743558 <== mds.0 v2:172.21.15.67:6834/3533295594 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x560de053b210 con 0x560de057c570 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb762000640 10 client.11184 handle_command_reply: tid=0 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 client.11184 shutdown 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 msgr2=0x560de055aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fb774001b60 tx=0x7fb774004f40 comp rx=0 tx=0).stop 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 shutdown_connections 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:6834/3533295594,v1:172.21.15.67:6835/3533295594] conn(0x560de057c570 0x560de055b0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560de055c620 0x560de055aba0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x560de05288f0 0x560de0573d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 --2- 172.21.15.67:0/1200743558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560de05126a0 0x560de05d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb762000640 0 client.11184 ms_handle_reset on v2:172.21.15.67:6834/3533295594 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 >> 172.21.15.67:0/1200743558 conn(0x560de050eb40 msgr2=0x560de0514760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 shutdown_connections 2024-04-17T08:50:40.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 1 -- 172.21.15.67:0/1200743558 wait complete. 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.758+0000 7fb786e34740 20 client.11184 trim_cache size 0 max 16384 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 11028, 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8739682540000002, 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 110, 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 230, 2024-04-17T08:50:40.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 230, 2024-04-17T08:50:40.761 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 19.208140385 2024-04-17T08:50:40.761 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:50:40.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 msgr2=0x7fb75c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 0x7fb75c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb7700099b0 tx=0x7fb770008040 comp rx=0 tx=0).stop 2024-04-17T08:50:40.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 msgr2=0x7fb77c0b9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0cb9d0 tx=0x7fb77400b4a0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 shutdown_connections 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fb75c06ca10 0x7fb75c06eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c154140 0x7fb77c0b9bd0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb77c1549e0 0x7fb77c157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 --2- 172.21.15.67:0/3334325258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb764004800 0x7fb77c0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 >> 172.21.15.67:0/3334325258 conn(0x7fb77c06d3c0 msgr2=0x7fb77c0b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 shutdown_connections 2024-04-17T08:50:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.759+0000 7fb782a00640 1 -- 172.21.15.67:0/3334325258 wait complete. 2024-04-17T08:50:40.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:50:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.986+0000 7f910f200640 1 Processor -- start 2024-04-17T08:50:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.987+0000 7f910f200640 1 -- start start 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.988+0000 7f910f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91100b6d00 0x7f91100cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.988+0000 7f910f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 0x7f91100b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.988+0000 7f910f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91100d2d30 con 0x7f911015ec60 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.988+0000 7f910f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91100d49c0 con 0x7f91100b6d00 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.988+0000 7f910f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91100d3660 con 0x7f91100cf910 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f910de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f911015ec60 0x7f91100b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48010/0 (socket says 172.21.15.67:48010) 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f910de00640 1 -- 172.21.15.67:0/1500028283 learned_addr learned my addr 172.21.15.67:0/1500028283 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f9107e00640 1 --2- 172.21.15.67:0/1500028283 >> v2:172.21.15.67:3300/0 conn(0x7f91100b6d00 0x7f91100cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f910e800640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 0x7f91100b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f9107e00640 1 -- 172.21.15.67:0/1500028283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 msgr2=0x7f91100b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f910d400640 1 -- 172.21.15.67:0/1500028283 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326164305 0 0) 0x7f91100d2d30 con 0x7f911015ec60 2024-04-17T08:50:40.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f9107e00640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 0x7f91100b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f9107e00640 1 -- 172.21.15.67:0/1500028283 >> v1:172.21.15.67:6789/0 conn(0x7f911015ec60 legacy=0x7f91100b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f9107e00640 1 -- 172.21.15.67:0/1500028283 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91100d3010 con 0x7f91100b6d00 2024-04-17T08:50:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.989+0000 7f910e800640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 0x7f91100b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.990+0000 7f9107e00640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100b6d00 0x7f91100cf3d0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f90f80097a0 tx=0x7f90f802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d18ba9e90aa4c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910d400640 1 -- 172.21.15.67:0/1500028283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8035020 con 0x7f91100b6d00 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910d400640 1 -- 172.21.15.67:0/1500028283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f8031810 con 0x7f91100b6d00 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910d400640 1 -- 172.21.15.67:0/1500028283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8031be0 con 0x7f91100b6d00 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 -- 172.21.15.67:0/1500028283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100b6d00 msgr2=0x7f91100cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100b6d00 0x7f91100cf3d0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f90f80097a0 tx=0x7f90f802b8b0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 -- 172.21.15.67:0/1500028283 shutdown_connections 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91100cf910 0x7f91100b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 --2- 172.21.15.67:0/1500028283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100b6d00 0x7f91100cf3d0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 -- 172.21.15.67:0/1500028283 >> 172.21.15.67:0/1500028283 conn(0x7f91100d15f0 msgr2=0x7f91100d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 -- 172.21.15.67:0/1500028283 shutdown_connections 2024-04-17T08:50:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.991+0000 7f910f200640 1 -- 172.21.15.67:0/1500028283 wait complete. 2024-04-17T08:50:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.993+0000 7f910f200640 1 Processor -- start 2024-04-17T08:50:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.993+0000 7f910f200640 1 -- start start 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 0x7f91100ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f911015ec60 0x7f91100cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f911015f040 con 0x7f91100cf910 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f91100cfdc0 con 0x7f91100b6d00 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.994+0000 7f910f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91100d00c0 con 0x7f911015ec60 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52926/0 (socket says 172.21.15.67:52926) 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 -- 172.21.15.67:0/3268092725 learned_addr learned my addr 172.21.15.67:0/3268092725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 -- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f911015ec60 msgr2=0x7f91100cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f910de00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 0x7f91100ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f910e800640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f911015ec60 0x7f91100cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f911015ec60 0x7f91100cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 -- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 msgr2=0x7f91100ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 0x7f91100ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 -- 172.21.15.67:0/3268092725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91100ad030 con 0x7f91100cf910 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f910de00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 0x7f91100ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.995+0000 7f9107e00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009890 tx=0x7f90f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.996+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8041070 con 0x7f91100cf910 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.996+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f803f070 con 0x7f91100cf910 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.996+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8035040 con 0x7f91100cf910 2024-04-17T08:50:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.996+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91100ad310 con 0x7f91100cf910 2024-04-17T08:50:40.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.996+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91100ad8d0 con 0x7f91100cf910 2024-04-17T08:50:40.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.997+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9110068a10 con 0x7f91100cf910 2024-04-17T08:50:40.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.998+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f90f8038020 con 0x7f91100cf910 2024-04-17T08:50:40.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.999+0000 7f9106000640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 0x7f90ec06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:40.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:40.999+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f90f80adba0 con 0x7f91100cf910 2024-04-17T08:50:40.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.000+0000 7f910de00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 0x7f90ec06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.001+0000 7f910de00640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 0x7f90ec06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9100000f60 tx=0x7f9100009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.004+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f90f807bea0 con 0x7f91100cf910 2024-04-17T08:50:41.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.249+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f91100b7790 con 0x7f91100cf910 2024-04-17T08:50:41.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.250+0000 7f9106000640 1 -- 172.21.15.67:0/3268092725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v110) ==== 53+0+73 (secure 0 0 0) 0x7f90f807b7a0 con 0x7f91100cf910 2024-04-17T08:50:41.248 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 msgr2=0x7f90ec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 0x7f90ec06eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9100000f60 tx=0x7f9100009040 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 msgr2=0x7f91101f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009890 tx=0x7f90f80046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 shutdown_connections 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f90ec06ca10 0x7f90ec06eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f911015ec60 0x7f91100cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91100cf910 0x7f91101f5280 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 --2- 172.21.15.67:0/3268092725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91100b6d00 0x7f91100ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 >> 172.21.15.67:0/3268092725 conn(0x7f91100d15f0 msgr2=0x7f91100b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.252+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 shutdown_connections 2024-04-17T08:50:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41.253+0000 7f910f200640 1 -- 172.21.15.67:0/3268092725 wait complete. 2024-04-17T08:50:41.270 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:50:41.270 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:50:41.271 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:50:41.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:41.271 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.328 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 36803sec preferred_lft 36803sec 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:50:41.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:50:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:50:41.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:41.331 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:50:41.423 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:50:41.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:41.428 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:50:41.486 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:50:41.486 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:50:41.486 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:50:41.486 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:50:41.486 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:50:41.487 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:50:41.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:50:41.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:50:41.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:41.748 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:50:41.803 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:50:41.803 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:50:41.804 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:50:41.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:41.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:50:41.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:50:41.922 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:50:41.983 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:50:41.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:50:41.983 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:50:42.037 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:50:42.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:42.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:50:42.188 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:50:42.188 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:50:42.224 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:50:42.224 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:50:42.224 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:50:42.224 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:50:42.225 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:50:42.226 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:50:42.226 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:50:42.226 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:50:42.227 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:50:42.227 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:50:42.230 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:50:42.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:42.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:50:42.313 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:50:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:50:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:50:42.449 INFO:teuthology.orchestra.run.smithi067.stdout:client7868 2024-04-17T08:50:42.450 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:50:42.450 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:50:42.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-17T08:50:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.738+0000 7fd5f0e00640 1 Processor -- start 2024-04-17T08:50:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.738+0000 7fd5f0e00640 1 -- start start 2024-04-17T08:50:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cfe90 0x7fd5ec06e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd5ec06b8c0 0x7fd5ec06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd5ec15d210 con 0x7fd5ec0cd940 2024-04-17T08:50:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd5ec15d4f0 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5f0e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ec15d7d0 con 0x7fd5ec0cfe90 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52930/0 (socket says 172.21.15.67:52930) 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.740+0000 7fd5e3e00640 1 -- 172.21.15.67:0/2731567610 learned_addr learned my addr 172.21.15.67:0/2731567610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e9e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cfe90 0x7fd5ec06e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5ea800640 1 --2- 172.21.15.67:0/2731567610 >> v2:172.21.15.187:3300/0 conn(0x7fd5ec06b8c0 0x7fd5ec06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e3e00640 1 -- 172.21.15.67:0/2731567610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cfe90 msgr2=0x7fd5ec06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e3e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cfe90 0x7fd5ec06e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e3e00640 1 -- 172.21.15.67:0/2731567610 >> v2:172.21.15.187:3300/0 conn(0x7fd5ec06b8c0 msgr2=0x7fd5ec06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e3e00640 1 --2- 172.21.15.67:0/2731567610 >> v2:172.21.15.187:3300/0 conn(0x7fd5ec06b8c0 0x7fd5ec06bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.741+0000 7fd5e3e00640 1 -- 172.21.15.67:0/2731567610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec15dab0 con 0x7fd5ec0cd940 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.742+0000 7fd5e3e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec1603f0 tx=0x7fd5d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9646dc18331e8e46 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.742+0000 7fd5e9400640 1 -- 172.21.15.67:0/2731567610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4035020 con 0x7fd5ec0cd940 2024-04-17T08:50:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.742+0000 7fd5e9400640 1 -- 172.21.15.67:0/2731567610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d4031870 con 0x7fd5ec0cd940 2024-04-17T08:50:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.742+0000 7fd5e9400640 1 -- 172.21.15.67:0/2731567610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4031c40 con 0x7fd5ec0cd940 2024-04-17T08:50:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 -- 172.21.15.67:0/2731567610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0cd940 msgr2=0x7fd5ec0cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec1603f0 tx=0x7fd5d402b910 comp rx=0 tx=0).stop 2024-04-17T08:50:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 -- 172.21.15.67:0/2731567610 shutdown_connections 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/2731567610 >> v2:172.21.15.187:3300/0 conn(0x7fd5ec06b8c0 0x7fd5ec06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0cd940 0x7fd5ec0cdda0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/2731567610 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cfe90 0x7fd5ec06e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 -- 172.21.15.67:0/2731567610 >> 172.21.15.67:0/2731567610 conn(0x7fd5ec0b2370 msgr2=0x7fd5ec0b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 -- 172.21.15.67:0/2731567610 shutdown_connections 2024-04-17T08:50:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.743+0000 7fd5f0e00640 1 -- 172.21.15.67:0/2731567610 wait complete. 2024-04-17T08:50:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.745+0000 7fd5f0e00640 1 Processor -- start 2024-04-17T08:50:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.745+0000 7fd5f0e00640 1 -- start start 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cd940 0x7fd5ec0aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0cfe90 0x7fd5ec1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec15ce90 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec0d0270 con 0x7fd5ec0cfe90 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5f0e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ec0722d0 con 0x7fd5ec0cd940 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52944/0 (socket says 172.21.15.67:52944) 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.746+0000 7fd5e9e00640 1 -- 172.21.15.67:0/3784574022 learned_addr learned my addr 172.21.15.67:0/3784574022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 -- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cd940 msgr2=0x7fd5ec0aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cd940 0x7fd5ec0aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 -- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0cfe90 msgr2=0x7fd5ec1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0cfe90 0x7fd5ec1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 -- 172.21.15.67:0/3784574022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec0ae570 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e9e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c820 tx=0x7fd5dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc015070 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5dc004020 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ec143470 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.747+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0043f0 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.748+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ec0ae830 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.749+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ec073980 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.750+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fd5dc007830 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.751+0000 7fd5e2000640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 0x7fd5c806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.751+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fd5dc08bc80 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.751+0000 7fd5e3e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 0x7fd5c806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:42.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.753+0000 7fd5e3e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 0x7fd5c806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4036000 tx=0x7fd5d4030000 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:42.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:42.755+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd5dc059dd0 con 0x7fd5ec06b8c0 2024-04-17T08:50:42.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.001+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fd5ec068d60 con 0x7fd5ec06b8c0 2024-04-17T08:50:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.943+0000 7fd5e2000640 1 -- 172.21.15.67:0/3784574022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v111) ==== 136+0+0 (secure 0 0 0) 0x7fd5dc0596d0 con 0x7fd5ec06b8c0 2024-04-17T08:50:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.948+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 msgr2=0x7fd5c806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.948+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 0x7fd5c806eed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4036000 tx=0x7fd5d4030000 comp rx=0 tx=0).stop 2024-04-17T08:50:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.948+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 msgr2=0x7fd5ec142c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.948+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c820 tx=0x7fd5dc00cce0 comp rx=0 tx=0).stop 2024-04-17T08:50:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 shutdown_connections 2024-04-17T08:50:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5c806ca10 0x7fd5c806eed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0cfe90 0x7fd5ec1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec0cd940 0x7fd5ec0aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 --2- 172.21.15.67:0/3784574022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec06b8c0 0x7fd5ec142c50 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 >> 172.21.15.67:0/3784574022 conn(0x7fd5ec0b2370 msgr2=0x7fd5ec0cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 shutdown_connections 2024-04-17T08:50:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:43.949+0000 7fd5f0e00640 1 -- 172.21.15.67:0/3784574022 wait complete. 2024-04-17T08:50:43.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.197+0000 7fd0d7200640 1 Processor -- start 2024-04-17T08:50:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.197+0000 7fd0d7200640 1 -- start start 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.198+0000 7fd0d7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d80d1f00 0x7fd0d8162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.198+0000 7fd0d7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 0x7fd0d806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.198+0000 7fd0d7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0d80d0e60 con 0x7fd0d80ce770 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.198+0000 7fd0d7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d80ceb50 con 0x7fd0d80d1f00 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.198+0000 7fd0d7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd0d80d1a70 con 0x7fd0d806c9d0 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d80d1f00 0x7fd0d8162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0d80ce770 0x7fd0d81625a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52436/0 (socket says 172.21.15.67:52436) 2024-04-17T08:50:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d80d1f00 0x7fd0d8162cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35440/0 (socket says 172.21.15.67:35440) 2024-04-17T08:50:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 -- 172.21.15.67:0/402682954 learned_addr learned my addr 172.21.15.67:0/402682954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 -- 172.21.15.67:0/402682954 >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 msgr2=0x7fd0d806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d6800640 1 --2- 172.21.15.67:0/402682954 >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 0x7fd0d806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 --2- 172.21.15.67:0/402682954 >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 0x7fd0d806cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 -- 172.21.15.67:0/402682954 >> v1:172.21.15.67:6789/0 conn(0x7fd0d80ce770 legacy=0x7fd0d81625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d5400640 1 -- 172.21.15.67:0/402682954 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d80d1140 con 0x7fd0d80d1f00 2024-04-17T08:50:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d4a00640 1 -- 172.21.15.67:0/402682954 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79219001 0 0) 0x7fd0d80d0e60 con 0x7fd0d80ce770 2024-04-17T08:50:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.199+0000 7fd0d6800640 1 --2- 172.21.15.67:0/402682954 >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 0x7fd0d806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d5400640 1 --2- 172.21.15.67:0/402682954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8162cb0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80b8700 tx=0x7fd0c402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb9da7948866746 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d4a00640 1 -- 172.21.15.67:0/402682954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c402cb10 con 0x7fd0d80d1f00 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d4a00640 1 -- 172.21.15.67:0/402682954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c402ccb0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d4a00640 1 -- 172.21.15.67:0/402682954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c402ce60 con 0x7fd0d80d1f00 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d7200640 1 -- 172.21.15.67:0/402682954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 msgr2=0x7fd0d8162cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.201+0000 7fd0d7200640 1 --2- 172.21.15.67:0/402682954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8162cb0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80b8700 tx=0x7fd0c402bf70 comp rx=0 tx=0).stop 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 -- 172.21.15.67:0/402682954 shutdown_connections 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 --2- 172.21.15.67:0/402682954 >> v2:172.21.15.187:3300/0 conn(0x7fd0d806c9d0 0x7fd0d806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 --2- 172.21.15.67:0/402682954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8162cb0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 -- 172.21.15.67:0/402682954 >> 172.21.15.67:0/402682954 conn(0x7fd0d80b4170 msgr2=0x7fd0d80b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 -- 172.21.15.67:0/402682954 shutdown_connections 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.202+0000 7fd0d7200640 1 -- 172.21.15.67:0/402682954 wait complete. 2024-04-17T08:50:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.203+0000 7fd0d7200640 1 Processor -- start 2024-04-17T08:50:44.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.204+0000 7fd0d7200640 1 -- start start 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d806c9d0 0x7fd0d80cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d80ce770 0x7fd0d80cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81631f0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d80d1700 con 0x7fd0d80ce770 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d8163370 con 0x7fd0d806c9d0 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35448/0 (socket says 172.21.15.67:35448) 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 -- 172.21.15.67:0/1348360410 learned_addr learned my addr 172.21.15.67:0/1348360410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d5e00640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d806c9d0 0x7fd0d80cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 -- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d806c9d0 msgr2=0x7fd0d80cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d806c9d0 0x7fd0d80cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 -- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d80ce770 msgr2=0x7fd0d80cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d80ce770 0x7fd0d80cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.205+0000 7fd0d6800640 1 -- 172.21.15.67:0/1348360410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d8156cf0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0d6800640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c7c0 tx=0x7fd0d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0015020 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d0004230 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0007620 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d815a390 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.206+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d8143db0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.207+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d80755d0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.208+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fd0d0018020 con 0x7fd0d80d1f00 2024-04-17T08:50:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.209+0000 7fd0ce000640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 0x7fd0b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.209+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7fd0d008be90 con 0x7fd0d80d1f00 2024-04-17T08:50:44.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.210+0000 7fd0d5e00640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 0x7fd0b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.211+0000 7fd0d5e00640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 0x7fd0b806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8007fd0 tx=0x7fd0c8009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.214+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd0d001f020 con 0x7fd0d80d1f00 2024-04-17T08:50:44.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.422+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0d80b4ff0 con 0x7fd0d80d1f00 2024-04-17T08:50:44.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.424+0000 7fd0ce000640 1 -- 172.21.15.67:0/1348360410 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v230) ==== 74+0+16679 (secure 0 0 0) 0x7fd0d005a140 con 0x7fd0d80d1f00 2024-04-17T08:50:44.423 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:44.423 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":230,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:34.879148+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":228,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 msgr2=0x7fd0b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 0x7fd0b806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8007fd0 tx=0x7fd0c8009040 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 msgr2=0x7fd0d8145b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c7c0 tx=0x7fd0d000cc80 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 shutdown_connections 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd0b806ca10 0x7fd0b806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d80d1f00 0x7fd0d8145b20 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0d80ce770 0x7fd0d80cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 --2- 172.21.15.67:0/1348360410 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0d806c9d0 0x7fd0d80cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 >> 172.21.15.67:0/1348360410 conn(0x7fd0d80b4170 msgr2=0x7fd0d80ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 shutdown_connections 2024-04-17T08:50:44.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.427+0000 7fd0d7200640 1 -- 172.21.15.67:0/1348360410 wait complete. 2024-04-17T08:50:44.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:50:31.281467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:50:32.343386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:50:44.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:50:44.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.674+0000 7efcdbe00640 1 Processor -- start 2024-04-17T08:50:44.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.674+0000 7efcdbe00640 1 -- start start 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.675+0000 7efcdbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efcdc161fa0 0x7efcdc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.675+0000 7efcdbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcdc0d0780 0x7efcdc0d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.675+0000 7efcdbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc0b3970 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.675+0000 7efcdbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efcdc0d2570 con 0x7efcdc0d0780 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.675+0000 7efcdbe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efcdc0b1840 con 0x7efcdc161fa0 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcdc0d0780 0x7efcdc0d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcd3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcdc0d1ff0 0x7efcdc06cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52438/0 (socket says 172.21.15.67:52438) 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcd3e00640 1 -- 172.21.15.67:0/228426202 learned_addr learned my addr 172.21.15.67:0/228426202 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcdc0d0780 0x7efcdc0d0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35462/0 (socket says 172.21.15.67:35462) 2024-04-17T08:50:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdaa00640 1 --2- 172.21.15.67:0/228426202 >> v2:172.21.15.187:3300/0 conn(0x7efcdc161fa0 0x7efcdc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 -- 172.21.15.67:0/228426202 >> v2:172.21.15.187:3300/0 conn(0x7efcdc161fa0 msgr2=0x7efcdc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 --2- 172.21.15.67:0/228426202 >> v2:172.21.15.187:3300/0 conn(0x7efcdc161fa0 0x7efcdc06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 -- 172.21.15.67:0/228426202 >> v1:172.21.15.67:6789/0 conn(0x7efcdc0d1ff0 legacy=0x7efcdc06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcdb400640 1 -- 172.21.15.67:0/228426202 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc0b3c10 con 0x7efcdc0d0780 2024-04-17T08:50:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.676+0000 7efcda000640 1 -- 172.21.15.67:0/228426202 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365437054 0 0) 0x7efcdc0b3970 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.677+0000 7efcdb400640 1 --2- 172.21.15.67:0/228426202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d0780 0x7efcdc0d0bb0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7efcdc073690 tx=0x7efcd402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e8241327927f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.678+0000 7efcda000640 1 -- 172.21.15.67:0/228426202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4039070 con 0x7efcdc0d0780 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.678+0000 7efcda000640 1 -- 172.21.15.67:0/228426202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd402cb60 con 0x7efcdc0d0780 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.678+0000 7efcda000640 1 -- 172.21.15.67:0/228426202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4034440 con 0x7efcdc0d0780 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.678+0000 7efcdbe00640 1 -- 172.21.15.67:0/228426202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d0780 msgr2=0x7efcdc0d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.678+0000 7efcdbe00640 1 --2- 172.21.15.67:0/228426202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d0780 0x7efcdc0d0bb0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7efcdc073690 tx=0x7efcd402bdb0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 -- 172.21.15.67:0/228426202 shutdown_connections 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 --2- 172.21.15.67:0/228426202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d0780 0x7efcdc0d0bb0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 --2- 172.21.15.67:0/228426202 >> v2:172.21.15.187:3300/0 conn(0x7efcdc161fa0 0x7efcdc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 -- 172.21.15.67:0/228426202 >> 172.21.15.67:0/228426202 conn(0x7efcdc0b41e0 msgr2=0x7efcdc0b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 -- 172.21.15.67:0/228426202 shutdown_connections 2024-04-17T08:50:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.679+0000 7efcdbe00640 1 -- 172.21.15.67:0/228426202 wait complete. 2024-04-17T08:50:44.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.680+0000 7efcdbe00640 1 Processor -- start 2024-04-17T08:50:44.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.681+0000 7efcdbe00640 1 -- start start 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.681+0000 7efcdbe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 0x7efcdc0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.681+0000 7efcdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdbe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcdc161fa0 0x7efcdc0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc162380 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdbe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc0b3ef0 con 0x7efcdc0d0780 2024-04-17T08:50:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdbe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcdc162500 con 0x7efcdc161fa0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35478/0 (socket says 172.21.15.67:35478) 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 -- 172.21.15.67:0/438929391 learned_addr learned my addr 172.21.15.67:0/438929391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdaa00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 0x7efcdc0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdb400640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcdc161fa0 0x7efcdc0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 -- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcdc161fa0 msgr2=0x7efcdc0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcdc161fa0 0x7efcdc0ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 -- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 msgr2=0x7efcdc0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 0x7efcdc0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcd3e00640 1 -- 172.21.15.67:0/438929391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc0cc600 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.682+0000 7efcdaa00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 0x7efcdc0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcd3e00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c4a0 tx=0x7efcc400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4017070 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc4005ce0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcdc15b110 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc40072b0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.683+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcdc15b690 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.684+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcdc068a10 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.685+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7efcc401a070 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.686+0000 7efcd2a00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 0x7efcb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:44.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.686+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7efcc408b0f0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.687+0000 7efcdaa00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 0x7efcb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:44.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.688+0000 7efcdaa00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 0x7efcb806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efccc006fd0 tx=0x7efccc008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:44.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.691+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efcc40593e0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.901+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efcdc0b4dd0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.902+0000 7efcd2a00640 1 -- 172.21.15.67:0/438929391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v230) ==== 93+0+10 (secure 0 0 0) 0x7efcc4058ce0 con 0x7efcdc0d1ff0 2024-04-17T08:50:44.900 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:50:44.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.904+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 msgr2=0x7efcb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.904+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 0x7efcb806eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efccc006fd0 tx=0x7efccc008040 comp rx=0 tx=0).stop 2024-04-17T08:50:44.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.904+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 msgr2=0x7efcdc0cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:44.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.904+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c4a0 tx=0x7efcc400c960 comp rx=0 tx=0).stop 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 shutdown_connections 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7efcb806ca10 0x7efcb806eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcdc161fa0 0x7efcdc0ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcdc0d1ff0 0x7efcdc0cdea0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 --2- 172.21.15.67:0/438929391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0d0780 0x7efcdc0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 >> 172.21.15.67:0/438929391 conn(0x7efcdc0b41e0 msgr2=0x7efcdc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 shutdown_connections 2024-04-17T08:50:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:44.905+0000 7efcdbe00640 1 -- 172.21.15.67:0/438929391 wait complete. 2024-04-17T08:50:44.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:50:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.147+0000 7f7557200640 1 Processor -- start 2024-04-17T08:50:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.147+0000 7f7557200640 1 -- start start 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7557200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75580cd920 0x7f755806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7557200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f755806c0a0 0x7f755806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7557200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7558146360 con 0x7f7558145f80 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7557200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75580727a0 con 0x7f75580cd920 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7557200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7558071e70 con 0x7f755806c0a0 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7555e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7558145f80 0x7f75580cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52454/0 (socket says 172.21.15.67:52454) 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7555e00640 1 -- 172.21.15.67:0/1089628239 learned_addr learned my addr 172.21.15.67:0/1089628239 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.149+0000 7f7555400640 1 --2- 172.21.15.67:0/1089628239 >> v2:172.21.15.67:3300/0 conn(0x7f75580cd920 0x7f755806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7556800640 1 --2- 172.21.15.67:0/1089628239 >> v2:172.21.15.187:3300/0 conn(0x7f755806c0a0 0x7f755806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7554a00640 1 -- 172.21.15.67:0/1089628239 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444802186 0 0) 0x7f7558146360 con 0x7f7558145f80 2024-04-17T08:50:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7554a00640 1 -- 172.21.15.67:0/1089628239 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753c003520 con 0x7f7558145f80 2024-04-17T08:50:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7555400640 1 -- 172.21.15.67:0/1089628239 >> v2:172.21.15.187:3300/0 conn(0x7f755806c0a0 msgr2=0x7f755806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7555400640 1 --2- 172.21.15.67:0/1089628239 >> v2:172.21.15.187:3300/0 conn(0x7f755806c0a0 0x7f755806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7555400640 1 -- 172.21.15.67:0/1089628239 >> v1:172.21.15.67:6789/0 conn(0x7f7558145f80 legacy=0x7f75580cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.150+0000 7f7555400640 1 -- 172.21.15.67:0/1089628239 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7558072150 con 0x7f75580cd920 2024-04-17T08:50:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.151+0000 7f7555400640 1 --2- 172.21.15.67:0/1089628239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f755806bb60 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f755815cc90 tx=0x7f754402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c494d5687c3d83d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7554a00640 1 -- 172.21.15.67:0/1089628239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75440315a0 con 0x7f75580cd920 2024-04-17T08:50:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7554a00640 1 -- 172.21.15.67:0/1089628239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7544037070 con 0x7f75580cd920 2024-04-17T08:50:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7554a00640 1 -- 172.21.15.67:0/1089628239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544031940 con 0x7f75580cd920 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 -- 172.21.15.67:0/1089628239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 msgr2=0x7f755806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 --2- 172.21.15.67:0/1089628239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f755806bb60 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f755815cc90 tx=0x7f754402b840 comp rx=0 tx=0).stop 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 -- 172.21.15.67:0/1089628239 shutdown_connections 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 --2- 172.21.15.67:0/1089628239 >> v2:172.21.15.187:3300/0 conn(0x7f755806c0a0 0x7f755806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 --2- 172.21.15.67:0/1089628239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f755806bb60 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.152+0000 7f7557200640 1 -- 172.21.15.67:0/1089628239 >> 172.21.15.67:0/1089628239 conn(0x7f75580b2190 msgr2=0x7f75580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.153+0000 7f7557200640 1 -- 172.21.15.67:0/1089628239 shutdown_connections 2024-04-17T08:50:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.153+0000 7f7557200640 1 -- 172.21.15.67:0/1089628239 wait complete. 2024-04-17T08:50:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.154+0000 7f7557200640 1 Processor -- start 2024-04-17T08:50:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.155+0000 7f7557200640 1 -- start start 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.155+0000 7f7557200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 0x7f75580af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7557200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7557200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7558145f80 0x7f7558142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7557200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7558072c30 con 0x7f75580cd920 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7557200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f755806c530 con 0x7f755806c0a0 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7557200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7558072db0 con 0x7f7558145f80 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35494/0 (socket says 172.21.15.67:35494) 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 -- 172.21.15.67:0/1517859289 learned_addr learned my addr 172.21.15.67:0/1517859289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7556800640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7558145f80 0x7f7558142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555e00640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 0x7f75580af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 -- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7558145f80 msgr2=0x7f7558142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7558145f80 0x7f7558142410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 -- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 msgr2=0x7f75580af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 0x7f75580af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555400640 1 -- 172.21.15.67:0/1517859289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75580adbb0 con 0x7f75580cd920 2024-04-17T08:50:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.156+0000 7f7555e00640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 0x7f75580af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f7555400640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f7544000c00 tx=0x7f75440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544032630 con 0x7f75580cd920 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75440327d0 con 0x7f75580cd920 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544032b80 con 0x7f75580cd920 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7558142a40 con 0x7f75580cd920 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.157+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7558143000 con 0x7f75580cd920 2024-04-17T08:50:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.159+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7558068a10 con 0x7f75580cd920 2024-04-17T08:50:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.159+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f754404d3f0 con 0x7f75580cd920 2024-04-17T08:50:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.160+0000 7f754e000640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 0x7f753806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.160+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f754402f030 con 0x7f75580cd920 2024-04-17T08:50:45.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.161+0000 7f7555e00640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 0x7f753806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.162+0000 7f7555e00640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 0x7f753806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7548005880 tx=0x7f7548009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.165+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7544040070 con 0x7f75580cd920 2024-04-17T08:50:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.376+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7558143720 con 0x7f75580cd920 2024-04-17T08:50:45.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.377+0000 7f754e000640 1 -- 172.21.15.67:0/1517859289 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v230) ==== 104+0+11 (secure 0 0 0) 0x7f754407b380 con 0x7f75580cd920 2024-04-17T08:50:45.375 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.379+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 msgr2=0x7f753806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.379+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 0x7f753806eed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7548005880 tx=0x7f7548009040 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 msgr2=0x7f7558141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f7544000c00 tx=0x7f75440046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 shutdown_connections 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f753806ca10 0x7f753806eed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7558145f80 0x7f7558142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75580cd920 0x7f7558141ed0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 --2- 172.21.15.67:0/1517859289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f755806c0a0 0x7f75580af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 >> 172.21.15.67:0/1517859289 conn(0x7f75580b2190 msgr2=0x7f755815cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 shutdown_connections 2024-04-17T08:50:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.380+0000 7f7557200640 1 -- 172.21.15.67:0/1517859289 wait complete. 2024-04-17T08:50:45.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:50:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.632+0000 7f13a5800640 1 Processor -- start 2024-04-17T08:50:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.632+0000 7f13a5800640 1 -- start start 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.633+0000 7f13a5800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13a00cfd30 0x7f13a015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.633+0000 7f13a5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a015f240 con 0x7f13a00b4260 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.633+0000 7f13a5800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a00b6420 con 0x7f13a0144fb0 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.633+0000 7f13a5800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f13a00b61c0 con 0x7f13a00cfd30 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.634+0000 7f139f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f13a00b4260 0x7f13a00cf620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52470/0 (socket says 172.21.15.67:52470) 2024-04-17T08:50:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.634+0000 7f139f400640 1 -- 172.21.15.67:0/3481942320 learned_addr learned my addr 172.21.15.67:0/3481942320 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.634+0000 7f13a4e00640 1 --2- 172.21.15.67:0/3481942320 >> v2:172.21.15.187:3300/0 conn(0x7f13a00cfd30 0x7f13a015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.634+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838867155 0 0) 0x7f13a015f240 con 0x7f13a00b4260 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.634+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1384003520 con 0x7f13a00b4260 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2849947141 0 0) 0x7f1384003520 con 0x7f13a00b4260 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 >> v2:172.21.15.187:3300/0 conn(0x7f13a00cfd30 msgr2=0x7f13a015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 --2- 172.21.15.67:0/3481942320 >> v2:172.21.15.187:3300/0 conn(0x7f13a00cfd30 0x7f13a015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 >> v1:172.21.15.187:6789/0 conn(0x7f13a0144fb0 legacy=0x7f13a00b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a015f4b0 con 0x7f13a00b4260 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1388001670 con 0x7f13a00b4260 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 >> v1:172.21.15.67:6789/0 conn(0x7f13a00b4260 legacy=0x7f13a00cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840039b0 0x7f1384003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840051c0 0x7f138400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13a00b61c0 con 0x7f1384004800 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f138400c060 con 0x7f13840039b0 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f138400c260 con 0x7f13840051c0 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.635+0000 7f139f400640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139fe00640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840039b0 0x7f1384003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f13a4e00640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840051c0 0x7f138400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139f400640 1 -- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840051c0 msgr2=0x7f138400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139f400640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840051c0 0x7f138400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139f400640 1 -- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840039b0 msgr2=0x7f1384003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139f400640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840039b0 0x7f1384003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f139f400640 1 -- 172.21.15.67:0/3481942320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1384003520 con 0x7f1384004800 2024-04-17T08:50:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.636+0000 7f13a4e00640 1 -- 172.21.15.67:0/3481942320 reap_dead start 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.637+0000 7f139f400640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 0x7f1384004c80 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f138800c240 tx=0x7f1384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f299b146c7a198 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.637+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388007cf0 con 0x7f1384004800 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.637+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1388030070 con 0x7f1384004800 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.637+0000 7f139ea00640 1 -- 172.21.15.67:0/3481942320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388005d70 con 0x7f1384004800 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 -- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 msgr2=0x7f1384004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 0x7f1384004c80 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f138800c240 tx=0x7f1384005830 comp rx=0 tx=0).stop 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 -- 172.21.15.67:0/3481942320 shutdown_connections 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 --2- 172.21.15.67:0/3481942320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1384004800 0x7f1384004c80 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 -- 172.21.15.67:0/3481942320 >> 172.21.15.67:0/3481942320 conn(0x7f13a00b23c0 msgr2=0x7f13a00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 -- 172.21.15.67:0/3481942320 shutdown_connections 2024-04-17T08:50:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.638+0000 7f13a5800640 1 -- 172.21.15.67:0/3481942320 wait complete. 2024-04-17T08:50:45.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.640+0000 7f13a5800640 1 Processor -- start 2024-04-17T08:50:45.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.640+0000 7f13a5800640 1 -- start start 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 0x7f13a0137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13a00af080 0x7f13a015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13a0145390 con 0x7f13a00b0d80 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f13a00b6700 con 0x7f1384004800 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a5800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13a015fcb0 con 0x7f13a00af080 2024-04-17T08:50:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35516/0 (socket says 172.21.15.67:35516) 2024-04-17T08:50:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f13a4e00640 1 -- 172.21.15.67:0/2221549412 learned_addr learned my addr 172.21.15.67:0/2221549412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f139f400640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13a00af080 0x7f13a015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.641+0000 7f139fe00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 0x7f13a0137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 -- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13a00af080 msgr2=0x7f13a015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13a00af080 0x7f13a015bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 -- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 msgr2=0x7f13a0137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 0x7f13a0137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 -- 172.21.15.67:0/2221549412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a01383b0 con 0x7f13a00b0d80 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f139fe00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 0x7f13a0137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a4e00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f1398010790 tx=0x7f1398010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139800f040 con 0x7f13a00b0d80 2024-04-17T08:50:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139801c070 con 0x7f13a00b0d80 2024-04-17T08:50:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139800eab0 con 0x7f13a00b0d80 2024-04-17T08:50:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.642+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a015c4b0 con 0x7f13a00b0d80 2024-04-17T08:50:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.643+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a015ca70 con 0x7f13a00b0d80 2024-04-17T08:50:45.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.644+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a0068a10 con 0x7f13a00b0d80 2024-04-17T08:50:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.645+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f139800ec50 con 0x7f13a00b0d80 2024-04-17T08:50:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.646+0000 7f1397e00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 0x7f137c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.646+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f139808b4c0 con 0x7f13a00b0d80 2024-04-17T08:50:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.646+0000 7f139fe00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 0x7f137c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.647+0000 7f139fe00640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 0x7f137c06eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1390001d40 tx=0x7f1390009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.650+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1398025020 con 0x7f13a00b0d80 2024-04-17T08:50:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.857+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f13a0068610 con 0x7f13a00b0d80 2024-04-17T08:50:45.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.859+0000 7f1397e00640 1 -- 172.21.15.67:0/2221549412 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v230) ==== 100+0+11 (secure 0 0 0) 0x7f13a01383b0 con 0x7f13a00b0d80 2024-04-17T08:50:45.857 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 msgr2=0x7f137c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 0x7f137c06eed0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1390001d40 tx=0x7f1390009040 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 msgr2=0x7f13a00b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f1398010790 tx=0x7f1398010c50 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 shutdown_connections 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f137c06ca10 0x7f137c06eed0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a00b0d80 0x7f13a00b1200 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13a00af080 0x7f13a015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 --2- 172.21.15.67:0/2221549412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1384004800 0x7f13a0137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.861+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 >> 172.21.15.67:0/2221549412 conn(0x7f13a00b23c0 msgr2=0x7f13a00cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.862+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 shutdown_connections 2024-04-17T08:50:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:45.862+0000 7f13a5800640 1 -- 172.21.15.67:0/2221549412 wait complete. 2024-04-17T08:50:45.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:50:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.095+0000 7f19bfe00640 1 Processor -- start 2024-04-17T08:50:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.096+0000 7f19bfe00640 1 -- start start 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19c015ec60 0x7f19c00b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19c00cf910 0x7f19c006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f19c00d49c0 con 0x7f19c00cf910 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f19c00d3660 con 0x7f19c015ec60 2024-04-17T08:50:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.097+0000 7f19bfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19c00b6f70 con 0x7f19c00b6b30 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19c00cf910 0x7f19c006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19c00cf910 0x7f19c006cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35530/0 (socket says 172.21.15.67:35530) 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 -- 172.21.15.67:0/3686541058 learned_addr learned my addr 172.21.15.67:0/3686541058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19b7e00640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bea00640 1 --2- 172.21.15.67:0/3686541058 >> v2:172.21.15.187:3300/0 conn(0x7f19c015ec60 0x7f19c00b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 -- 172.21.15.67:0/3686541058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 msgr2=0x7f19c00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 -- 172.21.15.67:0/3686541058 >> v2:172.21.15.187:3300/0 conn(0x7f19c015ec60 msgr2=0x7f19c00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 --2- 172.21.15.67:0/3686541058 >> v2:172.21.15.187:3300/0 conn(0x7f19c015ec60 0x7f19c00b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.098+0000 7f19bf400640 1 -- 172.21.15.67:0/3686541058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c00d2d30 con 0x7f19c00cf910 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.099+0000 7f19bf400640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c006cdb0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f19c00d1550 tx=0x7f19b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893778980cc0fc3b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.100+0000 7f19be000640 1 -- 172.21.15.67:0/3686541058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b802ca70 con 0x7f19c00cf910 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.100+0000 7f19be000640 1 -- 172.21.15.67:0/3686541058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b802cc10 con 0x7f19c00cf910 2024-04-17T08:50:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.100+0000 7f19be000640 1 -- 172.21.15.67:0/3686541058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b80343f0 con 0x7f19c00cf910 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.100+0000 7f19bfe00640 1 -- 172.21.15.67:0/3686541058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 msgr2=0x7f19c006cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.100+0000 7f19bfe00640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c006cdb0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f19c00d1550 tx=0x7f19b802bed0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 -- 172.21.15.67:0/3686541058 shutdown_connections 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c006cdb0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 --2- 172.21.15.67:0/3686541058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c00cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 --2- 172.21.15.67:0/3686541058 >> v2:172.21.15.187:3300/0 conn(0x7f19c015ec60 0x7f19c00b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 -- 172.21.15.67:0/3686541058 >> 172.21.15.67:0/3686541058 conn(0x7f19c00d15f0 msgr2=0x7f19c00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 -- 172.21.15.67:0/3686541058 shutdown_connections 2024-04-17T08:50:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.101+0000 7f19bfe00640 1 -- 172.21.15.67:0/3686541058 wait complete. 2024-04-17T08:50:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.103+0000 7f19bfe00640 1 Processor -- start 2024-04-17T08:50:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.103+0000 7f19bfe00640 1 -- start start 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c0161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 0x7f19c01f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19c00cfda0 con 0x7f19c00cf910 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19c00d32f0 con 0x7f19c015ec60 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19c015f040 con 0x7f19c00b6b30 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35536/0 (socket says 172.21.15.67:35536) 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19b7e00640 1 -- 172.21.15.67:0/1821162371 learned_addr learned my addr 172.21.15.67:0/1821162371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bea00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c0161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.104+0000 7f19bf400640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 0x7f19c01f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 -- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 msgr2=0x7f19c0161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c0161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 -- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 msgr2=0x7f19c01f4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 0x7f19c01f4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 -- 172.21.15.67:0/1821162371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c01f7f00 con 0x7f19c00cf910 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19bea00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c0161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19bf400640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 0x7f19c01f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b7e00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f19a800d830 tx=0x7f19a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8018020 con 0x7f19c00cf910 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a80096a0 con 0x7f19c00cf910 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8009a70 con 0x7f19c00cf910 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c01f81e0 con 0x7f19c00cf910 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.105+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c01f4120 con 0x7f19c00cf910 2024-04-17T08:50:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.107+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19c0068a10 con 0x7f19c00cf910 2024-04-17T08:50:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.108+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f19a801b020 con 0x7f19c00cf910 2024-04-17T08:50:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.109+0000 7f19b6a00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 0x7f199c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.109+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f19a808c0e0 con 0x7f19c00cf910 2024-04-17T08:50:46.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.110+0000 7f19bea00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 0x7f199c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.111+0000 7f19bea00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 0x7f199c06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f19b0006fd0 tx=0x7f19b0008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.116+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f19a8022030 con 0x7f19c00cf910 2024-04-17T08:50:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.325+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f19c0161800 con 0x7f19c00cf910 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.326+0000 7f19b6a00640 1 -- 172.21.15.67:0/1821162371 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v230) ==== 84+0+209 (secure 0 0 0) 0x7f19a80102e0 con 0x7f19c00cf910 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:46.324 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 26, 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": "cephfs_metadata" 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 27, 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": "cephfs_data" 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:46.325 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.328+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 msgr2=0x7f199c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.328+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 0x7f199c06eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f19b0006fd0 tx=0x7f19b0008040 comp rx=0 tx=0).stop 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.328+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 msgr2=0x7f19c01f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.328+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f19a800d830 tx=0x7f19a800dcf0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 shutdown_connections 2024-04-17T08:50:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f199c06ca10 0x7f199c06eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c015ec60 0x7f19c01f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c00cf910 0x7f19c01f7790 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 --2- 172.21.15.67:0/1821162371 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19c00b6b30 0x7f19c0161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 >> 172.21.15.67:0/1821162371 conn(0x7f19c00d15f0 msgr2=0x7f19c00b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 shutdown_connections 2024-04-17T08:50:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.329+0000 7f19bfe00640 1 -- 172.21.15.67:0/1821162371 wait complete. 2024-04-17T08:50:46.347 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-17T08:50:46.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-17T08:50:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.576+0000 7f4620c00640 1 Processor -- start 2024-04-17T08:50:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.576+0000 7f4620c00640 1 -- start start 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.578+0000 7f4620c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f461c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.578+0000 7f4620c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c0ce0e0 con 0x7f461c0b4e20 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.578+0000 7f4620c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c0b5860 con 0x7f461c0cfe30 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.578+0000 7f4620c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f461c0b4060 con 0x7f461c0cd7c0 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4619400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f461c0b4e20 0x7f461c0cf720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52486/0 (socket says 172.21.15.67:52486) 2024-04-17T08:50:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4619400640 1 -- 172.21.15.67:0/2196213555 learned_addr learned my addr 172.21.15.67:0/2196213555 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4619e00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f461c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686438145 0 0) 0x7f461c0ce0e0 con 0x7f461c0b4e20 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4600005730 con 0x7f461c0b4e20 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3049414874 0 0) 0x7f4600005730 con 0x7f461c0b4e20 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 msgr2=0x7f461c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f461c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 >> v1:172.21.15.187:6789/0 conn(0x7f461c0cfe30 legacy=0x7f461c15bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.579+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c0b4340 con 0x7f461c0b4e20 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4608001670 con 0x7f461c0b4e20 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 >> v1:172.21.15.67:6789/0 conn(0x7f461c0b4e20 legacy=0x7f461c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f4600005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f4600006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4600006f50 0x7f46000073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f461c0b4060 con 0x7f4600006170 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f460000dd60 con 0x7f4600006f50 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f460000df60 con 0x7f461c0cd7c0 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f4600006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619e00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f4600005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 -- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 msgr2=0x7f4600005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0cd7c0 0x7f4600005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 -- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4600006f50 msgr2=0x7f46000073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f461a800640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4600006f50 0x7f46000073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4600006f50 0x7f46000073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.580+0000 7f4619400640 1 -- 172.21.15.67:0/2196213555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4600005730 con 0x7f4600006170 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4619400640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f4600006a10 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f460800c210 tx=0x7f4600007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9d90dcc4ae5f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608007d30 con 0x7f4600006170 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4608030070 con 0x7f4600006170 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4618a00640 1 -- 172.21.15.67:0/2196213555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460802cda0 con 0x7f4600006170 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4620c00640 1 -- 172.21.15.67:0/2196213555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 msgr2=0x7f4600006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f4620c00640 1 --2- 172.21.15.67:0/2196213555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f4600006a10 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f460800c210 tx=0x7f4600007a40 comp rx=0 tx=0).stop 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.582+0000 7f461a800640 1 -- 172.21.15.67:0/2196213555 reap_dead start 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.583+0000 7f4620c00640 1 -- 172.21.15.67:0/2196213555 shutdown_connections 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.583+0000 7f4620c00640 1 -- 172.21.15.67:0/2196213555 >> 172.21.15.67:0/2196213555 conn(0x7f461c06b8c0 msgr2=0x7f461c06bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.583+0000 7f4620c00640 1 -- 172.21.15.67:0/2196213555 shutdown_connections 2024-04-17T08:50:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.583+0000 7f4620c00640 1 -- 172.21.15.67:0/2196213555 wait complete. 2024-04-17T08:50:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.584+0000 7f4620c00640 1 Processor -- start 2024-04-17T08:50:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.585+0000 7f4620c00640 1 -- start start 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.585+0000 7f4620c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.585+0000 7f4620c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0c69c0 0x7f461c143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.585+0000 7f4620c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 0x7f461c0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4620c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f461c145390 con 0x7f4600006170 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4620c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f461c0d0350 con 0x7f461c0c7130 2024-04-17T08:50:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4620c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f461c145510 con 0x7f461c0c69c0 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35550/0 (socket says 172.21.15.67:35550) 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 -- 172.21.15.67:0/3163244709 learned_addr learned my addr 172.21.15.67:0/3163244709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619400640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0c69c0 0x7f461c143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 -- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0c69c0 msgr2=0x7f461c143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f461a800640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 0x7f461c0c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0c69c0 0x7f461c143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 -- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 msgr2=0x7f461c0c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 0x7f461c0c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f4619e00640 1 -- 172.21.15.67:0/3163244709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c1f0820 con 0x7f4600006170 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.586+0000 7f461a800640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 0x7f461c0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4619e00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f460c007f30 tx=0x7f460c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c01a020 con 0x7f4600006170 2024-04-17T08:50:46.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460c016470 con 0x7f4600006170 2024-04-17T08:50:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c0c6e40 con 0x7f4600006170 2024-04-17T08:50:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c016860 con 0x7f4600006170 2024-04-17T08:50:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.587+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c0acf00 con 0x7f4600006170 2024-04-17T08:50:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.588+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c073a50 con 0x7f4600006170 2024-04-17T08:50:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.589+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f460c005d10 con 0x7f4600006170 2024-04-17T08:50:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.590+0000 7f4612000640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 0x7f45fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.590+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5737+0+0 (secure 0 0 0) 0x7f460c08cdf0 con 0x7f4600006170 2024-04-17T08:50:46.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.591+0000 7f4619400640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 0x7f45fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.592+0000 7f4619400640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 0x7f45fc06eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f460800c210 tx=0x7f4608009030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.595+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f460c024030 con 0x7f4600006170 2024-04-17T08:50:46.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.800+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f461c0b35e0 con 0x7f4600006170 2024-04-17T08:50:46.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.973+0000 7f4612000640 1 -- 172.21.15.67:0/3163244709 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v231) ==== 144+0+0 (secure 0 0 0) 0x7f460c05af40 con 0x7f4600006170 2024-04-17T08:50:46.972 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_metadata' created 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 msgr2=0x7f45fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 0x7f45fc06eed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f460800c210 tx=0x7f4608009030 comp rx=0 tx=0).stop 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 msgr2=0x7f461c0ac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f460c007f30 tx=0x7f460c011660 comp rx=0 tx=0).stop 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 shutdown_connections 2024-04-17T08:50:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f45fc06ca10 0x7f45fc06eed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f461c0c7130 0x7f461c0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f461c0c69c0 0x7f461c143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 --2- 172.21.15.67:0/3163244709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4600006170 0x7f461c0ac520 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 >> 172.21.15.67:0/3163244709 conn(0x7f461c06b8c0 msgr2=0x7f461c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 shutdown_connections 2024-04-17T08:50:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:46.977+0000 7f4620c00640 1 -- 172.21.15.67:0/3163244709 wait complete. 2024-04-17T08:50:47.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:50:47.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.224+0000 7f4b50a00640 1 Processor -- start 2024-04-17T08:50:47.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.225+0000 7f4b50a00640 1 -- start start 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.226+0000 7f4b50a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.226+0000 7f4b50a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 0x7f4b4c0af470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.226+0000 7f4b50a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c15f050 con 0x7f4b4c0af9b0 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.226+0000 7f4b50a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4b4c0b3770 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.226+0000 7f4b50a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b4c15f330 con 0x7f4b4c15c580 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35552/0 (socket says 172.21.15.67:35552) 2024-04-17T08:50:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 -- 172.21.15.67:0/1588793759 learned_addr learned my addr 172.21.15.67:0/1588793759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:47.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b43e00640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 0x7f4b4c0af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 -- 172.21.15.67:0/1588793759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 msgr2=0x7f4b4c0af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:47.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 0x7f4b4c0af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 -- 172.21.15.67:0/1588793759 >> v1:172.21.15.187:6789/0 conn(0x7f4b4c0af9b0 legacy=0x7f4b4c0b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b4b400640 1 -- 172.21.15.67:0/1588793759 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c15f610 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.227+0000 7f4b43e00640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 0x7f4b4c0af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.229+0000 7f4b4b400640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009870 tx=0x7f4b3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315bcfd0bfe2bb58 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.229+0000 7f4b4aa00640 1 -- 172.21.15.67:0/1588793759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c035020 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.229+0000 7f4b4aa00640 1 -- 172.21.15.67:0/1588793759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3c031640 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.229+0000 7f4b4aa00640 1 -- 172.21.15.67:0/1588793759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c02d040 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 -- 172.21.15.67:0/1588793759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c1a0 msgr2=0x7f4b4c0ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009870 tx=0x7f4b3c02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 -- 172.21.15.67:0/1588793759 shutdown_connections 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c15c580 0x7f4b4c0af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 --2- 172.21.15.67:0/1588793759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c0ad9e0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 -- 172.21.15.67:0/1588793759 >> 172.21.15.67:0/1588793759 conn(0x7f4b4c067d10 msgr2=0x7f4b4c0d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 -- 172.21.15.67:0/1588793759 shutdown_connections 2024-04-17T08:50:47.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.230+0000 7f4b50a00640 1 -- 172.21.15.67:0/1588793759 wait complete. 2024-04-17T08:50:47.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.232+0000 7f4b50a00640 1 Processor -- start 2024-04-17T08:50:47.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.232+0000 7f4b50a00640 1 -- start start 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c0af9b0 0x7f4b4c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c1f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c0b2940 con 0x7f4b4c15c580 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c0b27c0 con 0x7f4b4c15c1a0 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b50a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b4c1625a0 con 0x7f4b4c0af9b0 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.233+0000 7f4b4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35554/0 (socket says 172.21.15.67:35554) 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 -- 172.21.15.67:0/733117513 learned_addr learned my addr 172.21.15.67:0/733117513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4b400640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c0af9b0 0x7f4b4c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b43e00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c1f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 -- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c0af9b0 msgr2=0x7f4b4c158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c0af9b0 0x7f4b4c158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 -- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b4c15c1a0 msgr2=0x7f4b4c1f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c1f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.234+0000 7f4b4be00640 1 -- 172.21.15.67:0/733117513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c155a40 con 0x7f4b4c15c580 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b4be00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400c780 tx=0x7f4b4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44004b60 con 0x7f4b4c15c580 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44004d00 con 0x7f4b4c15c580 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4c0ccfa0 con 0x7f4b4c15c580 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4400aa90 con 0x7f4b4c15c580 2024-04-17T08:50:47.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.235+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b4c0cece0 con 0x7f4b4c15c580 2024-04-17T08:50:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.236+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b4c0b4d20 con 0x7f4b4c15c580 2024-04-17T08:50:47.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.238+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f4b4400ac30 con 0x7f4b4c15c580 2024-04-17T08:50:47.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.239+0000 7f4b43400640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 0x7f4b2806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:47.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.240+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 6110+0+0 (secure 0 0 0) 0x7f4b4408c820 con 0x7f4b4c15c580 2024-04-17T08:50:47.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.240+0000 7f4b4b400640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 0x7f4b2806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.241+0000 7f4b4b400640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 0x7f4b2806eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c036000 tx=0x7f4b3c03c040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.244+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4b4c0b4d20 con 0x7f4b4c15c580 2024-04-17T08:50:47.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.456+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4b4c068a10 con 0x7f4b4c15c580 2024-04-17T08:50:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.993+0000 7f4b43400640 1 -- 172.21.15.67:0/733117513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v232) ==== 176+0+0 (secure 0 0 0) 0x7f4b44020070 con 0x7f4b4c15c580 2024-04-17T08:50:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_data' created 2024-04-17T08:50:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 msgr2=0x7f4b2806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 0x7f4b2806eed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c036000 tx=0x7f4b3c03c040 comp rx=0 tx=0).stop 2024-04-17T08:50:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 msgr2=0x7f4b4c155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400c780 tx=0x7f4b4400cc40 comp rx=0 tx=0).stop 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 shutdown_connections 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f4b2806ca10 0x7f4b2806eed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b4c15c580 0x7f4b4c155500 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b4c15c1a0 0x7f4b4c1f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 --2- 172.21.15.67:0/733117513 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b4c0af9b0 0x7f4b4c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:47.997+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 >> 172.21.15.67:0/733117513 conn(0x7f4b4c067d10 msgr2=0x7f4b4c074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.000+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 shutdown_connections 2024-04-17T08:50:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.000+0000 7f4b50a00640 1 -- 172.21.15.67:0/733117513 wait complete. 2024-04-17T08:50:48.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-17T08:50:48.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.228+0000 7f01b5a00640 1 Processor -- start 2024-04-17T08:50:48.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.229+0000 7f01b5a00640 1 -- start start 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.230+0000 7f01b5a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f01b00cfa10 0x7f01b00cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.230+0000 7f01b5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01b00d4100 con 0x7f01b00d1090 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.230+0000 7f01b5a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01b0163860 con 0x7f01b015eca0 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.230+0000 7f01b5a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f01b00d03b0 con 0x7f01b00cfa10 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.231+0000 7f01af400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f01b00d1090 0x7f01b00d14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52492/0 (socket says 172.21.15.67:52492) 2024-04-17T08:50:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.231+0000 7f01af400640 1 -- 172.21.15.67:0/53346993 learned_addr learned my addr 172.21.15.67:0/53346993 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.231+0000 7f01ae000640 1 --2- 172.21.15.67:0/53346993 >> v2:172.21.15.187:3300/0 conn(0x7f01b00cfa10 0x7f01b00cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.231+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052834810 0 0) 0x7f01b00d4100 con 0x7f01b00d1090 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.231+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0194003520 con 0x7f01b00d1090 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4159081042 0 0) 0x7f0194003520 con 0x7f01b00d1090 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 >> v2:172.21.15.187:3300/0 conn(0x7f01b00cfa10 msgr2=0x7f01b00cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 --2- 172.21.15.67:0/53346993 >> v2:172.21.15.187:3300/0 conn(0x7f01b00cfa10 0x7f01b00cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 >> v1:172.21.15.187:6789/0 conn(0x7f01b015eca0 legacy=0x7f01b00d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b00d1d40 con 0x7f01b00d1090 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f01a80019f0 con 0x7f01b00d1090 2024-04-17T08:50:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 >> v1:172.21.15.67:6789/0 conn(0x7f01b00d1090 legacy=0x7f01b00d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01940039b0 0x7f0194003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 0x7f0194004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01940051c0 0x7f019400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f019400c030 con 0x7f0194004800 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f019400c1e0 con 0x7f01940039b0 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f019400c3e0 con 0x7f01940051c0 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01ae000640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 0x7f0194004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.232+0000 7f01aea00640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01940039b0 0x7f0194003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01af400640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01940051c0 0x7f019400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01ae000640 1 -- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01940051c0 msgr2=0x7f019400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01ae000640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01940051c0 0x7f019400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01ae000640 1 -- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01940039b0 msgr2=0x7f0194003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01ae000640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01940039b0 0x7f0194003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01ae000640 1 -- 172.21.15.67:0/53346993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0194003520 con 0x7f0194004800 2024-04-17T08:50:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.233+0000 7f01af400640 1 -- 172.21.15.67:0/53346993 reap_dead start 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.234+0000 7f01ae000640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 0x7f0194004c80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ac0 tx=0x7f0194005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39ad2d624a73947 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.234+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198039070 con 0x7f0194004800 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.234+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f019802c6b0 con 0x7f0194004800 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.234+0000 7f01ad600640 1 -- 172.21.15.67:0/53346993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198009690 con 0x7f0194004800 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 -- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 msgr2=0x7f0194004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 0x7f0194004c80 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ac0 tx=0x7f0194005830 comp rx=0 tx=0).stop 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 -- 172.21.15.67:0/53346993 shutdown_connections 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 --2- 172.21.15.67:0/53346993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0194004800 0x7f0194004c80 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 -- 172.21.15.67:0/53346993 >> 172.21.15.67:0/53346993 conn(0x7f01b0067600 msgr2=0x7f01b00679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 -- 172.21.15.67:0/53346993 shutdown_connections 2024-04-17T08:50:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.235+0000 7f01b5a00640 1 -- 172.21.15.67:0/53346993 wait complete. 2024-04-17T08:50:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.237+0000 7f01b5a00640 1 Processor -- start 2024-04-17T08:50:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.237+0000 7f01b5a00640 1 -- start start 2024-04-17T08:50:48.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 0x7f01b0149d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01b00d5270 0x7f01b00cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01b00d31f0 con 0x7f01b00b9c20 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f01b00d2f10 con 0x7f0194004800 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01b5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01b00d34d0 con 0x7f01b00d5270 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35574/0 (socket says 172.21.15.67:35574) 2024-04-17T08:50:48.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.238+0000 7f01ae000640 1 -- 172.21.15.67:0/2744157601 learned_addr learned my addr 172.21.15.67:0/2744157601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01aea00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 0x7f01b0149d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01af400640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01b00d5270 0x7f01b00cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 -- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01b00d5270 msgr2=0x7f01b00cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01b00d5270 0x7f01b00cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 -- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 msgr2=0x7f01b0149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 0x7f01b0149d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 -- 172.21.15.67:0/2744157601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b0150860 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01aea00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 0x7f01b0149d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.239+0000 7f01ae000640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f0198009be0 tx=0x7f0198004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.240+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198042020 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.240+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f019803a040 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.240+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198039070 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.240+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b0150b00 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.240+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b014a250 con 0x7f01b00b9c20 2024-04-17T08:50:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.241+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b0068ca0 con 0x7f01b00b9c20 2024-04-17T08:50:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.243+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f0198045020 con 0x7f01b00b9c20 2024-04-17T08:50:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.244+0000 7f01a6a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 0x7f018c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.244+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 6495+0+0 (secure 0 0 0) 0x7f01980ad7e0 con 0x7f01b00b9c20 2024-04-17T08:50:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.244+0000 7f01aea00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 0x7f018c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:48.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.246+0000 7f01aea00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 0x7f018c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01a00016b0 tx=0x7f01a0009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:48.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.248+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f019804c070 con 0x7f01b00b9c20 2024-04-17T08:50:48.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:48.497+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f01b00d2970 con 0x7f01b00b9c20 2024-04-17T08:50:49.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.020+0000 7f01a6a00640 1 -- 172.21.15.67:0/2744157601 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v112) ==== 183+0+0 (secure 0 0 0) 0x7f019807b960 con 0x7f01b00b9c20 2024-04-17T08:50:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 28 and data pool 29 2024-04-17T08:50:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.030+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 msgr2=0x7f018c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.030+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 0x7f018c06eed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01a00016b0 tx=0x7f01a0009040 comp rx=0 tx=0).stop 2024-04-17T08:50:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.030+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 msgr2=0x7f01b00ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.030+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f0198009be0 tx=0x7f0198004730 comp rx=0 tx=0).stop 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 shutdown_connections 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f018c06ca10 0x7f018c06eed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01b00d5270 0x7f01b00cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01b00b9c20 0x7f01b00ca670 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 --2- 172.21.15.67:0/2744157601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0194004800 0x7f01b0149d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.032+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 >> 172.21.15.67:0/2744157601 conn(0x7f01b0067600 msgr2=0x7f01b00cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.034+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 shutdown_connections 2024-04-17T08:50:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.035+0000 7f01b5a00640 1 -- 172.21.15.67:0/2744157601 wait complete. 2024-04-17T08:50:49.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.250+0000 7f2ab3200640 1 Processor -- start 2024-04-17T08:50:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.250+0000 7f2ab3200640 1 -- start start 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.251+0000 7f2ab3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab415b570 0x7f2ab406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.251+0000 7f2ab3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406c040 0x7f2ab415f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.251+0000 7f2ab3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab415bc80 con 0x7f2ab406fce0 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.251+0000 7f2ab3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab415bf40 con 0x7f2ab415b570 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.251+0000 7f2ab3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab415c220 con 0x7f2ab406c040 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab415b570 0x7f2ab406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ab415b570 0x7f2ab406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35588/0 (socket says 172.21.15.67:35588) 2024-04-17T08:50:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 -- 172.21.15.67:0/1205313995 learned_addr learned my addr 172.21.15.67:0/1205313995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:49.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab2800640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406c040 0x7f2ab415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 -- 172.21.15.67:0/1205313995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406c040 msgr2=0x7f2ab415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:49.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406c040 0x7f2ab415f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 -- 172.21.15.67:0/1205313995 >> v1:172.21.15.187:6789/0 conn(0x7f2ab406fce0 legacy=0x7f2ab406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:49.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.252+0000 7f2ab1e00640 1 -- 172.21.15.67:0/1205313995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab415c500 con 0x7f2ab415b570 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.253+0000 7f2ab1e00640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab415b570 0x7f2ab406f7a0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009e50 tx=0x7f2aa402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0261509a7cf3d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab0a00640 1 -- 172.21.15.67:0/1205313995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa402c9e0 con 0x7f2ab415b570 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab0a00640 1 -- 172.21.15.67:0/1205313995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa402cb80 con 0x7f2ab415b570 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab0a00640 1 -- 172.21.15.67:0/1205313995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40343f0 con 0x7f2ab415b570 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 -- 172.21.15.67:0/1205313995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab415b570 msgr2=0x7f2ab406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab415b570 0x7f2ab406f7a0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009e50 tx=0x7f2aa402be40 comp rx=0 tx=0).stop 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 -- 172.21.15.67:0/1205313995 shutdown_connections 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406c040 0x7f2ab415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 --2- 172.21.15.67:0/1205313995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab415b570 0x7f2ab406f7a0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.254+0000 7f2ab3200640 1 -- 172.21.15.67:0/1205313995 >> 172.21.15.67:0/1205313995 conn(0x7f2ab4067d10 msgr2=0x7f2ab40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.255+0000 7f2ab3200640 1 -- 172.21.15.67:0/1205313995 shutdown_connections 2024-04-17T08:50:49.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.255+0000 7f2ab3200640 1 -- 172.21.15.67:0/1205313995 wait complete. 2024-04-17T08:50:49.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.256+0000 7f2ab3200640 1 Processor -- start 2024-04-17T08:50:49.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.256+0000 7f2ab3200640 1 -- start start 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 0x7f2ab40ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 0x7f2ab4143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40b4930 con 0x7f2ab406c040 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40b47b0 con 0x7f2ab415b570 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab415f680 con 0x7f2ab406fce0 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.257+0000 7f2ab1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35600/0 (socket says 172.21.15.67:35600) 2024-04-17T08:50:49.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 -- 172.21.15.67:0/3683553963 learned_addr learned my addr 172.21.15.67:0/3683553963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1400640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 0x7f2ab40ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 -- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 msgr2=0x7f2ab40ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab2800640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 0x7f2ab4143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 0x7f2ab40ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 -- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 msgr2=0x7f2ab4143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 0x7f2ab4143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 -- 172.21.15.67:0/3683553963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab415d110 con 0x7f2ab406c040 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1400640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 0x7f2ab40ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab2800640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 0x7f2ab4143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.258+0000 7f2ab1e00640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009e50 tx=0x7f2aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.259+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4043020 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.259+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa403e450 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.259+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa403e840 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.259+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab414ec60 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.259+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab414f220 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.260+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab40cc830 con 0x7f2ab406c040 2024-04-17T08:50:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.261+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f2aa4046020 con 0x7f2ab406c040 2024-04-17T08:50:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.262+0000 7f2aaa000640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 0x7f2a9006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.262+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 6601+0+0 (secure 0 0 0) 0x7f2aa40ae800 con 0x7f2ab406c040 2024-04-17T08:50:49.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.263+0000 7f2ab1400640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 0x7f2a9006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.264+0000 7f2ab1400640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 0x7f2a9006ef00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4153210 tx=0x7f2a9c009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.267+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2ab40cc830 con 0x7f2ab406c040 2024-04-17T08:50:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.474+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ab406f350 con 0x7f2ab406c040 2024-04-17T08:50:49.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.476+0000 7f2aaa000640 1 -- 172.21.15.67:0/3683553963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) ==== 74+0+20482 (secure 0 0 0) 0x7f2aa404d070 con 0x7f2ab406c040 2024-04-17T08:50:49.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:49.475 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":233,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:48.498982+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 msgr2=0x7f2a9006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 0x7f2a9006ef00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4153210 tx=0x7f2a9c009040 comp rx=0 tx=0).stop 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 msgr2=0x7f2ab414e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009e50 tx=0x7f2aa4004a40 comp rx=0 tx=0).stop 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 shutdown_connections 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2a9006ca40 0x7f2a9006ef00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab415b570 0x7f2ab4143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ab406fce0 0x7f2ab40ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 --2- 172.21.15.67:0/3683553963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab406c040 0x7f2ab414e3c0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 >> 172.21.15.67:0/3683553963 conn(0x7f2ab4067d10 msgr2=0x7f2ab41536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:49.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.479+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 shutdown_connections 2024-04-17T08:50:49.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.480+0000 7f2ab3200640 1 -- 172.21.15.67:0/3683553963 wait complete. 2024-04-17T08:50:49.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:49.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.725+0000 7f50b6e00640 1 Processor -- start 2024-04-17T08:50:49.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.725+0000 7f50b6e00640 1 -- start start 2024-04-17T08:50:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b6e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 0x7f50b01625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b6e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50b00d1f00 0x7f50b0162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b6e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50b00d0e60 con 0x7f50b0163020 2024-04-17T08:50:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b6e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f50b00ceb50 con 0x7f50b00d1f00 2024-04-17T08:50:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b6e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f50b00d1a70 con 0x7f50b00ce770 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50b00d1f00 0x7f50b0162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50b00d1f00 0x7f50b0162ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35610/0 (socket says 172.21.15.67:35610) 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.727+0000 7f50b5000640 1 -- 172.21.15.67:0/2896273577 learned_addr learned my addr 172.21.15.67:0/2896273577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5a00640 1 --2- 172.21.15.67:0/2896273577 >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 0x7f50b01625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5000640 1 -- 172.21.15.67:0/2896273577 >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 msgr2=0x7f50b01625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5000640 1 --2- 172.21.15.67:0/2896273577 >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 0x7f50b01625a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5000640 1 -- 172.21.15.67:0/2896273577 >> v1:172.21.15.187:6789/0 conn(0x7f50b0163020 legacy=0x7f50b00cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:49.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5000640 1 -- 172.21.15.67:0/2896273577 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b00d1140 con 0x7f50b00d1f00 2024-04-17T08:50:49.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.728+0000 7f50b5a00640 1 --2- 172.21.15.67:0/2896273577 >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 0x7f50b01625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:49.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.729+0000 7f50b5000640 1 --2- 172.21.15.67:0/2896273577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00d1f00 0x7f50b0162ae0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f50b0072a70 tx=0x7f509c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38f222da5dfef06 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.730+0000 7f50afe00640 1 -- 172.21.15.67:0/2896273577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c035020 con 0x7f50b00d1f00 2024-04-17T08:50:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.730+0000 7f50afe00640 1 -- 172.21.15.67:0/2896273577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f509c031870 con 0x7f50b00d1f00 2024-04-17T08:50:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.730+0000 7f50afe00640 1 -- 172.21.15.67:0/2896273577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c031c40 con 0x7f50b00d1f00 2024-04-17T08:50:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 -- 172.21.15.67:0/2896273577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00d1f00 msgr2=0x7f50b0162ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 --2- 172.21.15.67:0/2896273577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00d1f00 0x7f50b0162ae0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f50b0072a70 tx=0x7f509c02b910 comp rx=0 tx=0).stop 2024-04-17T08:50:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 -- 172.21.15.67:0/2896273577 shutdown_connections 2024-04-17T08:50:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 --2- 172.21.15.67:0/2896273577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00d1f00 0x7f50b0162ae0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 --2- 172.21.15.67:0/2896273577 >> v2:172.21.15.187:3300/0 conn(0x7f50b00ce770 0x7f50b01625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 -- 172.21.15.67:0/2896273577 >> 172.21.15.67:0/2896273577 conn(0x7f50b00b4170 msgr2=0x7f50b00b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 -- 172.21.15.67:0/2896273577 shutdown_connections 2024-04-17T08:50:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.731+0000 7f50b6e00640 1 -- 172.21.15.67:0/2896273577 wait complete. 2024-04-17T08:50:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.733+0000 7f50b6e00640 1 Processor -- start 2024-04-17T08:50:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.733+0000 7f50b6e00640 1 -- start start 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b00d1f00 0x7f50b00b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50b0163020 0x7f50b00b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50b00d09f0 con 0x7f50b00ce770 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f50b00d1700 con 0x7f50b00d1f00 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b6e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50b00d0b70 con 0x7f50b0163020 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35620/0 (socket says 172.21.15.67:35620) 2024-04-17T08:50:49.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.734+0000 7f50b5a00640 1 -- 172.21.15.67:0/3823410906 learned_addr learned my addr 172.21.15.67:0/3823410906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5000640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b00d1f00 0x7f50b00b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 -- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50b0163020 msgr2=0x7f50b00b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50b0163020 0x7f50b00b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 -- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b00d1f00 msgr2=0x7f50b00b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b00d1f00 0x7f50b00b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 -- 172.21.15.67:0/3823410906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b01e83e0 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.735+0000 7f50b5a00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ce00 tx=0x7f50a400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.736+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a400f890 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.736+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a400fa30 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.736+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a400fe40 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.736+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b01e8960 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.736+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b01e8f20 con 0x7f50b00ce770 2024-04-17T08:50:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.737+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b00755d0 con 0x7f50b00ce770 2024-04-17T08:50:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.738+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f50a4018400 con 0x7f50b00ce770 2024-04-17T08:50:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.739+0000 7f50ad600640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 0x7f509006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.739+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 6601+0+0 (secure 0 0 0) 0x7f50a408ced0 con 0x7f50b00ce770 2024-04-17T08:50:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.740+0000 7f50b5000640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 0x7f509006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.741+0000 7f50b5000640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 0x7f509006eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f509c036000 tx=0x7f509c03c040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:49.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.744+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f50a4014030 con 0x7f50b00ce770 2024-04-17T08:50:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.957+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50b0161780 con 0x7f50b00ce770 2024-04-17T08:50:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.961+0000 7f50ad600640 1 -- 172.21.15.67:0/3823410906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) ==== 74+0+20482 (secure 0 0 0) 0x7f50a400ae20 con 0x7f50b00ce770 2024-04-17T08:50:49.960 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:49.961 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":233,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:48.498982+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 msgr2=0x7f509006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 0x7f509006eed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f509c036000 tx=0x7f509c03c040 comp rx=0 tx=0).stop 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 msgr2=0x7f50b00b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ce00 tx=0x7f50a400a650 comp rx=0 tx=0).stop 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 shutdown_connections 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f509006ca10 0x7f509006eed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50b0163020 0x7f50b00b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50b00d1f00 0x7f50b00b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 --2- 172.21.15.67:0/3823410906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b00ce770 0x7f50b00b6260 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.964+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 >> 172.21.15.67:0/3823410906 conn(0x7f50b00b4170 msgr2=0x7f50b0160960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.965+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 shutdown_connections 2024-04-17T08:50:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:49.965+0000 7f50b6e00640 1 -- 172.21.15.67:0/3823410906 wait complete. 2024-04-17T08:50:49.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-17T08:50:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.198+0000 7fec0ac00640 1 Processor -- start 2024-04-17T08:50:50.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.198+0000 7fec0ac00640 1 -- start start 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec0415ecb0 0x7fec040d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec040d3210 0x7fec040cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec040cf8e0 0x7fec040d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fec040d0c30 con 0x7fec0415ecb0 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fec040d00b0 con 0x7fec040d3210 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec0ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec040d0390 con 0x7fec040cf8e0 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec0415ecb0 0x7fec040d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec0415ecb0 0x7fec040d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35626/0 (socket says 172.21.15.67:35626) 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 -- 172.21.15.67:0/2897306149 learned_addr learned my addr 172.21.15.67:0/2897306149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 -- 172.21.15.67:0/2897306149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec040cf8e0 msgr2=0x7fec040d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 --2- 172.21.15.67:0/2897306149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec040cf8e0 0x7fec040d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 -- 172.21.15.67:0/2897306149 >> v2:172.21.15.187:3300/0 conn(0x7fec040d3210 msgr2=0x7fec040cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 --2- 172.21.15.67:0/2897306149 >> v2:172.21.15.187:3300/0 conn(0x7fec040d3210 0x7fec040cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.200+0000 7fec09800640 1 -- 172.21.15.67:0/2897306149 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec040d0670 con 0x7fec0415ecb0 2024-04-17T08:50:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.202+0000 7fec09800640 1 --2- 172.21.15.67:0/2897306149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0415ecb0 0x7fec040d2cd0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fec040d05a0 tx=0x7febf802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a649aeb845b760f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.202+0000 7fec03e00640 1 -- 172.21.15.67:0/2897306149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf80317c0 con 0x7fec0415ecb0 2024-04-17T08:50:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.202+0000 7fec03e00640 1 -- 172.21.15.67:0/2897306149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf8037070 con 0x7fec0415ecb0 2024-04-17T08:50:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.202+0000 7fec03e00640 1 -- 172.21.15.67:0/2897306149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8031b80 con 0x7fec0415ecb0 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 -- 172.21.15.67:0/2897306149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0415ecb0 msgr2=0x7fec040d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 --2- 172.21.15.67:0/2897306149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0415ecb0 0x7fec040d2cd0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fec040d05a0 tx=0x7febf802b860 comp rx=0 tx=0).stop 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 -- 172.21.15.67:0/2897306149 shutdown_connections 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 --2- 172.21.15.67:0/2897306149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec040cf8e0 0x7fec040d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 --2- 172.21.15.67:0/2897306149 >> v2:172.21.15.187:3300/0 conn(0x7fec040d3210 0x7fec040cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 --2- 172.21.15.67:0/2897306149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0415ecb0 0x7fec040d2cd0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 -- 172.21.15.67:0/2897306149 >> 172.21.15.67:0/2897306149 conn(0x7fec0408bff0 msgr2=0x7fec040d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 -- 172.21.15.67:0/2897306149 shutdown_connections 2024-04-17T08:50:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.203+0000 7fec0ac00640 1 -- 172.21.15.67:0/2897306149 wait complete. 2024-04-17T08:50:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.205+0000 7fec0ac00640 1 Processor -- start 2024-04-17T08:50:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.205+0000 7fec0ac00640 1 -- start start 2024-04-17T08:50:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec040cf8e0 0x7fec04153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0415ecb0 0x7fec040c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec040cfd00 con 0x7fec040d3210 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec0415f090 con 0x7fec040cf8e0 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec040cb950 con 0x7fec0415ecb0 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35636/0 (socket says 172.21.15.67:35636) 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 -- 172.21.15.67:0/733836009 learned_addr learned my addr 172.21.15.67:0/733836009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec09800640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec040cf8e0 0x7fec04153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec0a200640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0415ecb0 0x7fec040c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 -- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0415ecb0 msgr2=0x7fec040c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0415ecb0 0x7fec040c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 -- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec040cf8e0 msgr2=0x7fec04153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec040cf8e0 0x7fec04153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.206+0000 7fec08e00640 1 -- 172.21.15.67:0/733836009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec040cbc30 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec08e00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7febf000b4a0 tx=0x7febf000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0014070 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf0010610 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0010a00 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec040cbf10 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.207+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec04153d00 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.208+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec04068a10 con 0x7fec040d3210 2024-04-17T08:50:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.209+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7febf0017070 con 0x7fec040d3210 2024-04-17T08:50:50.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.211+0000 7fec01600640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 0x7febe406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:50.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.211+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7febf008bb10 con 0x7fec040d3210 2024-04-17T08:50:50.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.211+0000 7fec09800640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 0x7febe406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.213+0000 7fec09800640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 0x7febe406eed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febf8036000 tx=0x7febf8008030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.216+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7febf0059aa0 con 0x7fec040d3210 2024-04-17T08:50:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:50.463+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fec040b1cd0 con 0x7fec040d3210 2024-04-17T08:50:51.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.049+0000 7fec01600640 1 -- 172.21.15.67:0/733836009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v114) ==== 121+0+0 (secure 0 0 0) 0x7febf00593a0 con 0x7fec040d3210 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 msgr2=0x7febe406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 0x7febe406eed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febf8036000 tx=0x7febf8008030 comp rx=0 tx=0).stop 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 msgr2=0x7fec041537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7febf000b4a0 tx=0x7febf000b960 comp rx=0 tx=0).stop 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 shutdown_connections 2024-04-17T08:50:51.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7febe406ca10 0x7febe406eed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0415ecb0 0x7fec040c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec040d3210 0x7fec041537c0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 --2- 172.21.15.67:0/733836009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec040cf8e0 0x7fec04153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 >> 172.21.15.67:0/733836009 conn(0x7fec0408bff0 msgr2=0x7fec040b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 shutdown_connections 2024-04-17T08:50:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.053+0000 7fec0ac00640 1 -- 172.21.15.67:0/733836009 wait complete. 2024-04-17T08:50:51.076 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:50:51.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-17T08:50:51.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.296+0000 7f0f0b200640 1 Processor -- start 2024-04-17T08:50:51.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.296+0000 7f0f0b200640 1 -- start start 2024-04-17T08:50:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.298+0000 7f0f0b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.298+0000 7f0f0b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0f0c0cfc10 0x7f0f0c06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.298+0000 7f0f0b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0c06df40 con 0x7f0f0c06f800 2024-04-17T08:50:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.298+0000 7f0f0b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0c06ed00 con 0x7f0f0c0cfc10 2024-04-17T08:50:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.298+0000 7f0f0b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c06e870 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:36084/0 (socket says 172.21.15.67:36084) 2024-04-17T08:50:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f0a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0f0c0cfc10 0x7f0f0c06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 -- 172.21.15.67:0/2943636185 learned_addr learned my addr 172.21.15.67:0/2943636185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:51.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 -- 172.21.15.67:0/2943636185 >> v2:172.21.15.187:3300/0 conn(0x7f0f0c0cfc10 msgr2=0x7f0f0c06be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:51.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 --2- 172.21.15.67:0/2943636185 >> v2:172.21.15.187:3300/0 conn(0x7f0f0c0cfc10 0x7f0f0c06be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 -- 172.21.15.67:0/2943636185 >> v1:172.21.15.187:6789/0 conn(0x7f0f0c06f800 legacy=0x7f0f0c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.299+0000 7f0f09e00640 1 -- 172.21.15.67:0/2943636185 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c06e220 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.301+0000 7f0f09e00640 1 --2- 172.21.15.67:0/2943636185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009870 tx=0x7f0efc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=158bc2473ed116ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.301+0000 7f0f08a00640 1 -- 172.21.15.67:0/2943636185 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc035020 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.301+0000 7f0f08a00640 1 -- 172.21.15.67:0/2943636185 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0efc031640 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.301+0000 7f0f08a00640 1 -- 172.21.15.67:0/2943636185 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc02d040 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 -- 172.21.15.67:0/2943636185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 msgr2=0x7f0f0c06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 --2- 172.21.15.67:0/2943636185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009870 tx=0x7f0efc02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 -- 172.21.15.67:0/2943636185 shutdown_connections 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 --2- 172.21.15.67:0/2943636185 >> v2:172.21.15.187:3300/0 conn(0x7f0f0c0cfc10 0x7f0f0c06be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 --2- 172.21.15.67:0/2943636185 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c06b8c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 -- 172.21.15.67:0/2943636185 >> 172.21.15.67:0/2943636185 conn(0x7f0f0c0b2190 msgr2=0x7f0f0c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 -- 172.21.15.67:0/2943636185 shutdown_connections 2024-04-17T08:50:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.302+0000 7f0f0b200640 1 -- 172.21.15.67:0/2943636185 wait complete. 2024-04-17T08:50:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.305+0000 7f0f0b200640 1 Processor -- start 2024-04-17T08:50:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.305+0000 7f0f0b200640 1 -- start start 2024-04-17T08:50:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 0x7f0f0c145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0afee0 con 0x7f0f0c06f800 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0afd60 con 0x7f0f0c0cfc10 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f0b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b44a0 con 0x7f0f0c15b6b0 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35642/0 (socket says 172.21.15.67:35642) 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.306+0000 7f0f09e00640 1 -- 172.21.15.67:0/1032236970 learned_addr learned my addr 172.21.15.67:0/1032236970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 -- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 msgr2=0x7f0f0c157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 -- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 msgr2=0x7f0f0c145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09400640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 0x7f0f0c145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 0x7f0f0c145be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 -- 172.21.15.67:0/1032236970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c15d020 con 0x7f0f0c06f800 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09400640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 0x7f0f0c145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.307+0000 7f0f09e00640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f0efc02b930 tx=0x7f0efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.308+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc040070 con 0x7f0f0c06f800 2024-04-17T08:50:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.308+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0efc03e070 con 0x7f0f0c06f800 2024-04-17T08:50:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.308+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc035040 con 0x7f0f0c06f800 2024-04-17T08:50:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.308+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c15d2c0 con 0x7f0f0c06f800 2024-04-17T08:50:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.308+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c15db60 con 0x7f0f0c06f800 2024-04-17T08:50:51.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.309+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c068a10 con 0x7f0f0c06f800 2024-04-17T08:50:51.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.310+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f0efc038020 con 0x7f0f0c06f800 2024-04-17T08:50:51.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.311+0000 7f0f02000640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 0x7f0ee806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:51.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.311+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f0efc0ad960 con 0x7f0f0c06f800 2024-04-17T08:50:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.312+0000 7f0f09400640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 0x7f0ee806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.313+0000 7f0f09400640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 0x7f0ee806eed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c15ecf0 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:51.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.317+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0efc07b8f0 con 0x7f0f0c06f800 2024-04-17T08:50:51.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:51.567+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0f0c06ee80 con 0x7f0f0c06f800 2024-04-17T08:50:52.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.053+0000 7f0f02000640 1 -- 172.21.15.67:0/1032236970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v115) ==== 125+0+0 (secure 0 0 0) 0x7f0efc043e40 con 0x7f0f0c06f800 2024-04-17T08:50:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 msgr2=0x7f0ee806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 0x7f0ee806eed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c15ecf0 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2024-04-17T08:50:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 msgr2=0x7f0f0c15cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f0efc02b930 tx=0x7f0efc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 shutdown_connections 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0ee806ca10 0x7f0ee806eed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f0c15b6b0 0x7f0f0c157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f0c0cfc10 0x7f0f0c145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 --2- 172.21.15.67:0/1032236970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0c06f800 0x7f0f0c15cae0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.057+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 >> 172.21.15.67:0/1032236970 conn(0x7f0f0c0b2190 msgr2=0x7f0f0c0cd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:52.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.058+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 shutdown_connections 2024-04-17T08:50:52.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.058+0000 7f0f0b200640 1 -- 172.21.15.67:0/1032236970 wait complete. 2024-04-17T08:50:52.080 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-17T08:50:52.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:52.280 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T08:50:52.281+0000 7f4790e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:50:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.306+0000 7f2af1200640 1 Processor -- start 2024-04-17T08:50:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.306+0000 7f2af1200640 1 -- start start 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2af1200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 0x7f2aec0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2af1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2af1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aec0d2d30 con 0x7f2aec0b6b30 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2af1200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2aec0d49c0 con 0x7f2aec0cfae0 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2af1200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2aec0d3660 con 0x7f2aec15ec60 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2aeaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2aec0b6b30 0x7f2aec0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52506/0 (socket says 172.21.15.67:52506) 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.308+0000 7f2aeaa00640 1 -- 172.21.15.67:0/3810862718 learned_addr learned my addr 172.21.15.67:0/3810862718 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aebe00640 1 --2- 172.21.15.67:0/3810862718 >> v2:172.21.15.67:3300/0 conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3810862718 >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 0x7f2aec0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aea000640 1 -- 172.21.15.67:0/3810862718 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 846780770 0 0) 0x7f2aec0d2d30 con 0x7f2aec0b6b30 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aebe00640 1 -- 172.21.15.67:0/3810862718 >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 msgr2=0x7f2aec0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aebe00640 1 --2- 172.21.15.67:0/3810862718 >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 0x7f2aec0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aebe00640 1 -- 172.21.15.67:0/3810862718 >> v1:172.21.15.67:6789/0 conn(0x7f2aec0b6b30 legacy=0x7f2aec0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aebe00640 1 -- 172.21.15.67:0/3810862718 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec0d3010 con 0x7f2aec0cfae0 2024-04-17T08:50:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.309+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3810862718 >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 0x7f2aec0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.310+0000 7f2aebe00640 1 --2- 172.21.15.67:0/3810862718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0d1550 tx=0x7f2ae402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfc81964d4dd75d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.310+0000 7f2aea000640 1 -- 172.21.15.67:0/3810862718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4039070 con 0x7f2aec0cfae0 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.310+0000 7f2aea000640 1 -- 172.21.15.67:0/3810862718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae402cc70 con 0x7f2aec0cfae0 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2aea000640 1 -- 172.21.15.67:0/3810862718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4034660 con 0x7f2aec0cfae0 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 -- 172.21.15.67:0/3810862718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 msgr2=0x7f2aec0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 --2- 172.21.15.67:0/3810862718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0d1550 tx=0x7f2ae402bec0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 -- 172.21.15.67:0/3810862718 shutdown_connections 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 --2- 172.21.15.67:0/3810862718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 --2- 172.21.15.67:0/3810862718 >> v2:172.21.15.187:3300/0 conn(0x7f2aec15ec60 0x7f2aec0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 -- 172.21.15.67:0/3810862718 >> 172.21.15.67:0/3810862718 conn(0x7f2aec0d15f0 msgr2=0x7f2aec0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 -- 172.21.15.67:0/3810862718 shutdown_connections 2024-04-17T08:50:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.311+0000 7f2af1200640 1 -- 172.21.15.67:0/3810862718 wait complete. 2024-04-17T08:50:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.313+0000 7f2af1200640 1 Processor -- start 2024-04-17T08:50:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.313+0000 7f2af1200640 1 -- start start 2024-04-17T08:50:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 0x7f2aec15d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2aec15ec60 0x7f2aec0b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec15f040 con 0x7f2aec0cfae0 2024-04-17T08:50:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0b6f70 con 0x7f2aec0b6b30 2024-04-17T08:50:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2af1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec0d00c0 con 0x7f2aec15ec60 2024-04-17T08:50:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2aeaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2aeaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35672/0 (socket says 172.21.15.67:35672) 2024-04-17T08:50:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.314+0000 7f2aeaa00640 1 -- 172.21.15.67:0/3979306028 learned_addr learned my addr 172.21.15.67:0/3979306028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 0x7f2aec15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 -- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2aec15ec60 msgr2=0x7f2aec0b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2aec15ec60 0x7f2aec0b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 -- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 msgr2=0x7f2aec15d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 0x7f2aec15d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 -- 172.21.15.67:0/3979306028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec15d6b0 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 0x7f2aec15d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2aeaa00640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c500 tx=0x7f2ad800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8018070 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad8007580 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.315+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8004a20 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.316+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aec0ccc50 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.316+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aec0cd210 con 0x7f2aec0cfae0 2024-04-17T08:50:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.317+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aec068a10 con 0x7f2aec0cfae0 2024-04-17T08:50:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.318+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f2ad8004bc0 con 0x7f2aec0cfae0 2024-04-17T08:50:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.319+0000 7f2ae3400640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 0x7f2acc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.319+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f2ad808b2e0 con 0x7f2aec0cfae0 2024-04-17T08:50:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.320+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 0x7f2acc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.321+0000 7f2aeb400640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 0x7f2acc06ef00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.323+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2ad8059270 con 0x7f2aec0cfae0 2024-04-17T08:50:52.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.568+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2aec1e9d10 con 0x7f2aec0cfae0 2024-04-17T08:50:52.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.569+0000 7f2ae3400640 1 -- 172.21.15.67:0/3979306028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7f2ad8058b70 con 0x7f2aec0cfae0 2024-04-17T08:50:52.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:52.569 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:50:52.569 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:50:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.571+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 msgr2=0x7f2acc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.571+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 0x7f2acc06ef00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc008040 comp rx=0 tx=0).stop 2024-04-17T08:50:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.571+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 msgr2=0x7f2aec0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.571+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c500 tx=0x7f2ad800c9c0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 shutdown_connections 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f2acc06ca40 0x7f2acc06ef00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2aec15ec60 0x7f2aec0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aec0cfae0 0x7f2aec0b7b70 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 --2- 172.21.15.67:0/3979306028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2aec0b6b30 0x7f2aec15d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 >> 172.21.15.67:0/3979306028 conn(0x7f2aec0d15f0 msgr2=0x7f2aec0b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 shutdown_connections 2024-04-17T08:50:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.572+0000 7f2af1200640 1 -- 172.21.15.67:0/3979306028 wait complete. 2024-04-17T08:50:52.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:50:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.807+0000 7f0639a00640 1 Processor -- start 2024-04-17T08:50:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.807+0000 7f0639a00640 1 -- start start 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0639a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0634160280 0x7f063406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0639a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0634141ed0 0x7f0634142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0639a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06340cd4a0 con 0x7f063406b8c0 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0639a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f063415ce30 con 0x7f0634160280 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0639a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06341607f0 con 0x7f0634141ed0 2024-04-17T08:50:52.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0633e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f063406b8c0 0x7f06340cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52522/0 (socket says 172.21.15.67:52522) 2024-04-17T08:50:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.809+0000 7f0633e00640 1 -- 172.21.15.67:0/1081684995 learned_addr learned my addr 172.21.15.67:0/1081684995 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0633400640 1 --2- 172.21.15.67:0/1081684995 >> v2:172.21.15.187:3300/0 conn(0x7f0634160280 0x7f063406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330320951 0 0) 0x7f06340cd4a0 con 0x7f063406b8c0 2024-04-17T08:50:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0639000640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0634141ed0 0x7f0634142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0618003520 con 0x7f063406b8c0 2024-04-17T08:50:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 332763855 0 0) 0x7f0618003520 con 0x7f063406b8c0 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0634141ed0 msgr2=0x7f0634142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0634141ed0 0x7f0634142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 >> v2:172.21.15.187:3300/0 conn(0x7f0634160280 msgr2=0x7f063406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 --2- 172.21.15.67:0/1081684995 >> v2:172.21.15.187:3300/0 conn(0x7f0634160280 0x7f063406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0634160ad0 con 0x7f063406b8c0 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0624001670 con 0x7f063406b8c0 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 >> v1:172.21.15.67:6789/0 conn(0x7f063406b8c0 legacy=0x7f06340cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0618003a20 0x7f0618003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f063415ce30 con 0x7f06180051c0 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06341607f0 con 0x7f0618004800 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f061800c090 con 0x7f0618003a20 2024-04-17T08:50:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.810+0000 7f0639000640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0633400640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0633e00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0618003a20 0x7f0618003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0639000640 1 -- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0618003a20 msgr2=0x7f0618003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0639000640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0618003a20 0x7f0618003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0639000640 1 -- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0618004800 msgr2=0x7f0618004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0639000640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.811+0000 7f0639000640 1 -- 172.21.15.67:0/1081684995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0618003520 con 0x7f06180051c0 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.812+0000 7f0639000640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f06341428c0 tx=0x7f0618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c48ec469e1945f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.812+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c039070 con 0x7f06180051c0 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.812+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062c02cb60 con 0x7f06180051c0 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.812+0000 7f0632a00640 1 -- 172.21.15.67:0/1081684995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c009b20 con 0x7f06180051c0 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 -- 172.21.15.67:0/1081684995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180051c0 msgr2=0x7f061800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 --2- 172.21.15.67:0/1081684995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f06341428c0 tx=0x7f0618005830 comp rx=0 tx=0).stop 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639000640 1 -- 172.21.15.67:0/1081684995 reap_dead start 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 -- 172.21.15.67:0/1081684995 shutdown_connections 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 -- 172.21.15.67:0/1081684995 >> 172.21.15.67:0/1081684995 conn(0x7f0634067d10 msgr2=0x7f06340b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 -- 172.21.15.67:0/1081684995 shutdown_connections 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.813+0000 7f0639a00640 1 -- 172.21.15.67:0/1081684995 wait complete. 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.815+0000 7f0639a00640 1 Processor -- start 2024-04-17T08:50:52.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.815+0000 7f0639a00640 1 -- start start 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06180051c0 0x7f063415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0634143310 0x7f0634143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0634161090 con 0x7f0634145470 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f063406c1e0 con 0x7f0634143310 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f063413f9b0 con 0x7f06180051c0 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35692/0 (socket says 172.21.15.67:35692) 2024-04-17T08:50:52.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.816+0000 7f0639000640 1 -- 172.21.15.67:0/1774740774 learned_addr learned my addr 172.21.15.67:0/1774740774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0633e00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06180051c0 0x7f063415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0633400640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0634143310 0x7f0634143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 -- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06180051c0 msgr2=0x7f063415b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06180051c0 0x7f063415b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 -- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0634143310 msgr2=0x7f0634143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0634143310 0x7f0634143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 -- 172.21.15.67:0/1774740774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063406df20 con 0x7f0634145470 2024-04-17T08:50:52.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639000640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f062c02bc10 tx=0x7f062c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c039070 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062c044070 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c009b20 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.817+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063406e1c0 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.818+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f063406e780 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.819+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0634068a10 con 0x7f0634145470 2024-04-17T08:50:52.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.820+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f062c02cd10 con 0x7f0634145470 2024-04-17T08:50:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.821+0000 7f062be00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 0x7f061406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.821+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f062c0adf30 con 0x7f0634145470 2024-04-17T08:50:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.821+0000 7f0633e00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 0x7f061406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:52.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.822+0000 7f0633e00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 0x7f061406ef00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0624001790 tx=0x7f0624000f90 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:52.825+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f062c07be30 con 0x7f0634145470 2024-04-17T08:50:53.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.024+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06340ccd30 con 0x7f0634145470 2024-04-17T08:50:53.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.027+0000 7f062be00640 1 -- 172.21.15.67:0/1774740774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) ==== 74+0+20469 (secure 0 0 0) 0x7f062c07b730 con 0x7f0634145470 2024-04-17T08:50:53.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:53.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":234,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:50:50.019771+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.029+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 msgr2=0x7f061406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.029+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 0x7f061406ef00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0624001790 tx=0x7f0624000f90 comp rx=0 tx=0).stop 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.029+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 msgr2=0x7f06341458f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.029+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f062c02bc10 tx=0x7f062c004ab0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 shutdown_connections 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f061406ca40 0x7f061406ef00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0634145470 0x7f06341458f0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0634143310 0x7f0634143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 --2- 172.21.15.67:0/1774740774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06180051c0 0x7f063415b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 >> 172.21.15.67:0/1774740774 conn(0x7f0634067d10 msgr2=0x7f06341422e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:53.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 shutdown_connections 2024-04-17T08:50:53.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.030+0000 7f0639a00640 1 -- 172.21.15.67:0/1774740774 wait complete. 2024-04-17T08:50:53.048 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:50:53.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:50:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.253+0000 7fa5e5000640 1 Processor -- start 2024-04-17T08:50:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.253+0000 7fa5e5000640 1 -- start start 2024-04-17T08:50:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.255+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.255+0000 7fa5e5000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5e006fce0 0x7fa5e006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.255+0000 7fa5e5000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5e015bc80 con 0x7fa5e006be70 2024-04-17T08:50:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.255+0000 7fa5e5000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa5e015bf40 con 0x7fa5e006fce0 2024-04-17T08:50:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.255+0000 7fa5e5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e015c220 con 0x7fa5e015b570 2024-04-17T08:50:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.256+0000 7fa5d7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5e006fce0 0x7fa5e006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.256+0000 7fa5df400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.256+0000 7fa5df400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:36128/0 (socket says 172.21.15.67:36128) 2024-04-17T08:50:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.256+0000 7fa5df400640 1 -- 172.21.15.67:0/3476777283 learned_addr learned my addr 172.21.15.67:0/3476777283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.257+0000 7fa5df400640 1 -- 172.21.15.67:0/3476777283 >> v2:172.21.15.187:3300/0 conn(0x7fa5e006fce0 msgr2=0x7fa5e006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.257+0000 7fa5df400640 1 --2- 172.21.15.67:0/3476777283 >> v2:172.21.15.187:3300/0 conn(0x7fa5e006fce0 0x7fa5e006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.257+0000 7fa5df400640 1 -- 172.21.15.67:0/3476777283 >> v1:172.21.15.187:6789/0 conn(0x7fa5e006be70 legacy=0x7fa5e006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.257+0000 7fa5df400640 1 -- 172.21.15.67:0/3476777283 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e015c500 con 0x7fa5e015b570 2024-04-17T08:50:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.258+0000 7fa5df400640 1 --2- 172.21.15.67:0/3476777283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009870 tx=0x7fa5d002db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7457a4117a4c2124 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5dea00640 1 -- 172.21.15.67:0/3476777283 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d002f8c0 con 0x7fa5e015b570 2024-04-17T08:50:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5dea00640 1 -- 172.21.15.67:0/3476777283 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d002fa60 con 0x7fa5e015b570 2024-04-17T08:50:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5dea00640 1 -- 172.21.15.67:0/3476777283 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d002fe30 con 0x7fa5e015b570 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 -- 172.21.15.67:0/3476777283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 msgr2=0x7fa5e006f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 --2- 172.21.15.67:0/3476777283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009870 tx=0x7fa5d002db00 comp rx=0 tx=0).stop 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 -- 172.21.15.67:0/3476777283 shutdown_connections 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 --2- 172.21.15.67:0/3476777283 >> v2:172.21.15.187:3300/0 conn(0x7fa5e006fce0 0x7fa5e006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 --2- 172.21.15.67:0/3476777283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e015b570 0x7fa5e006f7a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 -- 172.21.15.67:0/3476777283 >> 172.21.15.67:0/3476777283 conn(0x7fa5e0067d10 msgr2=0x7fa5e00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.259+0000 7fa5e5000640 1 -- 172.21.15.67:0/3476777283 shutdown_connections 2024-04-17T08:50:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.260+0000 7fa5e5000640 1 -- 172.21.15.67:0/3476777283 wait complete. 2024-04-17T08:50:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.261+0000 7fa5e5000640 1 Processor -- start 2024-04-17T08:50:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.261+0000 7fa5e5000640 1 -- start start 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e006be70 0x7fa5e00b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e006fce0 0x7fa5e0157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e006c460 con 0x7fa5e015b570 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e006c2e0 con 0x7fa5e006be70 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.262+0000 7fa5e5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e00b47b0 con 0x7fa5e006fce0 2024-04-17T08:50:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35696/0 (socket says 172.21.15.67:35696) 2024-04-17T08:50:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 -- 172.21.15.67:0/2264142171 learned_addr learned my addr 172.21.15.67:0/2264142171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5d7e00640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e006fce0 0x7fa5e0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5df400640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e006be70 0x7fa5e00b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 -- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e006fce0 msgr2=0x7fa5e0157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e006fce0 0x7fa5e0157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 -- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e006be70 msgr2=0x7fa5e00b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e006be70 0x7fa5e00b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.263+0000 7fa5dfe00640 1 -- 172.21.15.67:0/2264142171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e0150560 con 0x7fa5e015b570 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5dfe00640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c7d0 tx=0x7fa5d800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8003b70 con 0x7fa5e015b570 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d8003d30 con 0x7fa5e015b570 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d800aa90 con 0x7fa5e015b570 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e01473b0 con 0x7fa5e015b570 2024-04-17T08:50:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.264+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e00b0ab0 con 0x7fa5e015b570 2024-04-17T08:50:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.265+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e0068a10 con 0x7fa5e015b570 2024-04-17T08:50:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.266+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fa5d800ac30 con 0x7fa5e015b570 2024-04-17T08:50:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.268+0000 7fa5d7400640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 0x7fa5bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.268+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7fa5d808bf50 con 0x7fa5e015b570 2024-04-17T08:50:53.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.269+0000 7fa5df400640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 0x7fa5bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.270+0000 7fa5df400640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 0x7fa5bc06eed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0038000 tx=0x7fa5d0008030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.273+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa5e01473b0 con 0x7fa5e015b570 2024-04-17T08:50:53.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.472+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5e00cfd30 con 0x7fa5bc06ca10 2024-04-17T08:50:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.516+0000 7fa5d7400640 1 -- 172.21.15.67:0/2264142171 <== mgr.7679 v2:172.21.15.187:6832/150239855 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491414 (secure 0 0 0) 0x7fa5e00cfd30 con 0x7fa5bc06ca10 2024-04-17T08:50:53.517 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:53.526 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.527+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 msgr2=0x7fa5bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.527+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 0x7fa5bc06eed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0038000 tx=0x7fa5d0008030 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.527+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 msgr2=0x7fa5e0146e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.527+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c7d0 tx=0x7fa5d800cc90 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 shutdown_connections 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa5bc06ca10 0x7fa5bc06eed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5e015b570 0x7fa5e0146e70 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e006fce0 0x7fa5e0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 --2- 172.21.15.67:0/2264142171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e006be70 0x7fa5e00b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 >> 172.21.15.67:0/2264142171 conn(0x7fa5e0067d10 msgr2=0x7fa5e0150940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 shutdown_connections 2024-04-17T08:50:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.528+0000 7fa5e5000640 1 -- 172.21.15.67:0/2264142171 wait complete. 2024-04-17T08:50:53.554 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-17T08:50:52.426871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":678,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":240,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":410864,"kb_used_data":23992,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754563856,"statfs":{"total":773094113280,"available":772673388544,"internally_reserved":0,"allocated":24567808,"data_stored":12584999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":396021296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.451002"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044759+0000","last_change":"2024-04-17T08:50:49.041591+0000","last_active":"2024-04-17T08:50:49.044759+0000","last_peered":"2024-04-17T08:50:49.044759+0000","last_clean":"2024-04-17T08:50:49.044759+0000","last_became_active":"2024-04-17T08:50:49.041258+0000","last_became_peered":"2024-04-17T08:50:49.041258+0000","last_unstale":"2024-04-17T08:50:49.044759+0000","last_undegraded":"2024-04-17T08:50:49.044759+0000","last_fullsized":"2024-04-17T08:50:49.044759+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:56.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"217'354","reported_seq":1454,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028409+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:50:49.028409+0000","last_peered":"2024-04-17T08:50:49.028409+0000","last_clean":"2024-04-17T08:50:49.028409+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:50:49.028409+0000","last_undegraded":"2024-04-17T08:50:49.028409+0000","last_fullsized":"2024-04-17T08:50:49.028409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.051724+0000","last_change":"2024-04-17T08:50:32.893722+0000","last_active":"2024-04-17T08:50:49.051724+0000","last_peered":"2024-04-17T08:50:49.051724+0000","last_clean":"2024-04-17T08:50:49.051724+0000","last_became_active":"2024-04-17T08:50:32.893404+0000","last_became_peered":"2024-04-17T08:50:32.893404+0000","last_unstale":"2024-04-17T08:50:49.051724+0000","last_undegraded":"2024-04-17T08:50:49.051724+0000","last_fullsized":"2024-04-17T08:50:49.051724+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:34.249904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019210+0000","last_change":"2024-04-17T08:50:33.896994+0000","last_active":"2024-04-17T08:50:49.019210+0000","last_peered":"2024-04-17T08:50:49.019210+0000","last_clean":"2024-04-17T08:50:49.019210+0000","last_became_active":"2024-04-17T08:50:33.896640+0000","last_became_peered":"2024-04-17T08:50:33.896640+0000","last_unstale":"2024-04-17T08:50:49.019210+0000","last_undegraded":"2024-04-17T08:50:49.019210+0000","last_fullsized":"2024-04-17T08:50:49.019210+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:30:09.203741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057967+0000","last_change":"2024-04-17T08:50:48.034745+0000","last_active":"2024-04-17T08:50:49.057967+0000","last_peered":"2024-04-17T08:50:49.057967+0000","last_clean":"2024-04-17T08:50:49.057967+0000","last_became_active":"2024-04-17T08:50:48.034093+0000","last_became_peered":"2024-04-17T08:50:48.034093+0000","last_unstale":"2024-04-17T08:50:49.057967+0000","last_undegraded":"2024-04-17T08:50:49.057967+0000","last_fullsized":"2024-04-17T08:50:49.057967+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:42.731156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054609+0000","last_change":"2024-04-17T08:50:49.047477+0000","last_active":"2024-04-17T08:50:49.054609+0000","last_peered":"2024-04-17T08:50:49.054609+0000","last_clean":"2024-04-17T08:50:49.054609+0000","last_became_active":"2024-04-17T08:50:49.046829+0000","last_became_peered":"2024-04-17T08:50:49.046829+0000","last_unstale":"2024-04-17T08:50:49.054609+0000","last_undegraded":"2024-04-17T08:50:49.054609+0000","last_fullsized":"2024-04-17T08:50:49.054609+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:57.531837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045911+0000","last_change":"2024-04-17T08:50:32.894323+0000","last_active":"2024-04-17T08:50:49.045911+0000","last_peered":"2024-04-17T08:50:49.045911+0000","last_clean":"2024-04-17T08:50:49.045911+0000","last_became_active":"2024-04-17T08:50:32.893659+0000","last_became_peered":"2024-04-17T08:50:32.893659+0000","last_unstale":"2024-04-17T08:50:49.045911+0000","last_undegraded":"2024-04-17T08:50:49.045911+0000","last_fullsized":"2024-04-17T08:50:49.045911+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:16.424440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017404+0000","last_change":"2024-04-17T08:50:33.884434+0000","last_active":"2024-04-17T08:50:49.017404+0000","last_peered":"2024-04-17T08:50:49.017404+0000","last_clean":"2024-04-17T08:50:49.017404+0000","last_became_active":"2024-04-17T08:50:33.884075+0000","last_became_peered":"2024-04-17T08:50:33.884075+0000","last_unstale":"2024-04-17T08:50:49.017404+0000","last_undegraded":"2024-04-17T08:50:49.017404+0000","last_fullsized":"2024-04-17T08:50:49.017404+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:46.657940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023894+0000","last_change":"2024-04-17T08:50:48.021195+0000","last_active":"2024-04-17T08:50:49.023894+0000","last_peered":"2024-04-17T08:50:49.023894+0000","last_clean":"2024-04-17T08:50:49.023894+0000","last_became_active":"2024-04-17T08:50:48.020868+0000","last_became_peered":"2024-04-17T08:50:48.020868+0000","last_unstale":"2024-04-17T08:50:49.023894+0000","last_undegraded":"2024-04-17T08:50:49.023894+0000","last_fullsized":"2024-04-17T08:50:49.023894+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:31.330314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045649+0000","last_change":"2024-04-17T08:50:49.040734+0000","last_active":"2024-04-17T08:50:49.045649+0000","last_peered":"2024-04-17T08:50:49.045649+0000","last_clean":"2024-04-17T08:50:49.045649+0000","last_became_active":"2024-04-17T08:50:49.040450+0000","last_became_peered":"2024-04-17T08:50:49.040450+0000","last_unstale":"2024-04-17T08:50:49.045649+0000","last_undegraded":"2024-04-17T08:50:49.045649+0000","last_fullsized":"2024-04-17T08:50:49.045649+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:16.518722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011628+0000","last_change":"2024-04-17T08:50:32.902675+0000","last_active":"2024-04-17T08:50:49.011628+0000","last_peered":"2024-04-17T08:50:49.011628+0000","last_clean":"2024-04-17T08:50:49.011628+0000","last_became_active":"2024-04-17T08:50:32.900858+0000","last_became_peered":"2024-04-17T08:50:32.900858+0000","last_unstale":"2024-04-17T08:50:49.011628+0000","last_undegraded":"2024-04-17T08:50:49.011628+0000","last_fullsized":"2024-04-17T08:50:49.011628+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:45.638814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004205+0000","last_change":"2024-04-17T08:50:33.895678+0000","last_active":"2024-04-17T08:50:49.004205+0000","last_peered":"2024-04-17T08:50:49.004205+0000","last_clean":"2024-04-17T08:50:49.004205+0000","last_became_active":"2024-04-17T08:50:33.895010+0000","last_became_peered":"2024-04-17T08:50:33.895010+0000","last_unstale":"2024-04-17T08:50:49.004205+0000","last_undegraded":"2024-04-17T08:50:49.004205+0000","last_fullsized":"2024-04-17T08:50:49.004205+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:46:16.948548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.055450+0000","last_change":"2024-04-17T08:50:48.034319+0000","last_active":"2024-04-17T08:50:49.055450+0000","last_peered":"2024-04-17T08:50:49.055450+0000","last_clean":"2024-04-17T08:50:49.055450+0000","last_became_active":"2024-04-17T08:50:48.033896+0000","last_became_peered":"2024-04-17T08:50:48.033896+0000","last_unstale":"2024-04-17T08:50:49.055450+0000","last_undegraded":"2024-04-17T08:50:49.055450+0000","last_fullsized":"2024-04-17T08:50:49.055450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:44.797993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034984+0000","last_change":"2024-04-17T08:50:49.032527+0000","last_active":"2024-04-17T08:50:49.034984+0000","last_peered":"2024-04-17T08:50:49.034984+0000","last_clean":"2024-04-17T08:50:49.034984+0000","last_became_active":"2024-04-17T08:50:49.031441+0000","last_became_peered":"2024-04-17T08:50:49.031441+0000","last_unstale":"2024-04-17T08:50:49.034984+0000","last_undegraded":"2024-04-17T08:50:49.034984+0000","last_fullsized":"2024-04-17T08:50:49.034984+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:06.087433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014462+0000","last_change":"2024-04-17T08:50:32.891370+0000","last_active":"2024-04-17T08:50:49.014462+0000","last_peered":"2024-04-17T08:50:49.014462+0000","last_clean":"2024-04-17T08:50:49.014462+0000","last_became_active":"2024-04-17T08:50:32.888996+0000","last_became_peered":"2024-04-17T08:50:32.888996+0000","last_unstale":"2024-04-17T08:50:49.014462+0000","last_undegraded":"2024-04-17T08:50:49.014462+0000","last_fullsized":"2024-04-17T08:50:49.014462+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:31.815168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048769+0000","last_change":"2024-04-17T08:50:33.892519+0000","last_active":"2024-04-17T08:50:49.048769+0000","last_peered":"2024-04-17T08:50:49.048769+0000","last_clean":"2024-04-17T08:50:49.048769+0000","last_became_active":"2024-04-17T08:50:33.892264+0000","last_became_peered":"2024-04-17T08:50:33.892264+0000","last_unstale":"2024-04-17T08:50:49.048769+0000","last_undegraded":"2024-04-17T08:50:49.048769+0000","last_fullsized":"2024-04-17T08:50:49.048769+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:05.376200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048842+0000","last_change":"2024-04-17T08:50:48.029938+0000","last_active":"2024-04-17T08:50:49.048842+0000","last_peered":"2024-04-17T08:50:49.048842+0000","last_clean":"2024-04-17T08:50:49.048842+0000","last_became_active":"2024-04-17T08:50:48.029123+0000","last_became_peered":"2024-04-17T08:50:48.029123+0000","last_unstale":"2024-04-17T08:50:49.048842+0000","last_undegraded":"2024-04-17T08:50:49.048842+0000","last_fullsized":"2024-04-17T08:50:49.048842+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:21.642236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054616+0000","last_change":"2024-04-17T08:50:49.046714+0000","last_active":"2024-04-17T08:50:49.054616+0000","last_peered":"2024-04-17T08:50:49.054616+0000","last_clean":"2024-04-17T08:50:49.054616+0000","last_became_active":"2024-04-17T08:50:49.046392+0000","last_became_peered":"2024-04-17T08:50:49.046392+0000","last_unstale":"2024-04-17T08:50:49.054616+0000","last_undegraded":"2024-04-17T08:50:49.054616+0000","last_fullsized":"2024-04-17T08:50:49.054616+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:08.981690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021782+0000","last_change":"2024-04-17T08:50:32.903970+0000","last_active":"2024-04-17T08:50:49.021782+0000","last_peered":"2024-04-17T08:50:49.021782+0000","last_clean":"2024-04-17T08:50:49.021782+0000","last_became_active":"2024-04-17T08:50:32.903620+0000","last_became_peered":"2024-04-17T08:50:32.903620+0000","last_unstale":"2024-04-17T08:50:49.021782+0000","last_undegraded":"2024-04-17T08:50:49.021782+0000","last_fullsized":"2024-04-17T08:50:49.021782+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:33.640417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019406+0000","last_change":"2024-04-17T08:50:33.879546+0000","last_active":"2024-04-17T08:50:49.019406+0000","last_peered":"2024-04-17T08:50:49.019406+0000","last_clean":"2024-04-17T08:50:49.019406+0000","last_became_active":"2024-04-17T08:50:33.878778+0000","last_became_peered":"2024-04-17T08:50:33.878778+0000","last_unstale":"2024-04-17T08:50:49.019406+0000","last_undegraded":"2024-04-17T08:50:49.019406+0000","last_fullsized":"2024-04-17T08:50:49.019406+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.667020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028625+0000","last_change":"2024-04-17T08:50:48.023810+0000","last_active":"2024-04-17T08:50:49.028625+0000","last_peered":"2024-04-17T08:50:49.028625+0000","last_clean":"2024-04-17T08:50:49.028625+0000","last_became_active":"2024-04-17T08:50:48.023485+0000","last_became_peered":"2024-04-17T08:50:48.023485+0000","last_unstale":"2024-04-17T08:50:49.028625+0000","last_undegraded":"2024-04-17T08:50:49.028625+0000","last_fullsized":"2024-04-17T08:50:49.028625+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:54.085024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.056371+0000","last_change":"2024-04-17T08:50:49.046526+0000","last_active":"2024-04-17T08:50:49.056371+0000","last_peered":"2024-04-17T08:50:49.056371+0000","last_clean":"2024-04-17T08:50:49.056371+0000","last_became_active":"2024-04-17T08:50:49.046205+0000","last_became_peered":"2024-04-17T08:50:49.046205+0000","last_unstale":"2024-04-17T08:50:49.056371+0000","last_undegraded":"2024-04-17T08:50:49.056371+0000","last_fullsized":"2024-04-17T08:50:49.056371+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:30.247778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009992+0000","last_change":"2024-04-17T08:50:32.906189+0000","last_active":"2024-04-17T08:50:49.009992+0000","last_peered":"2024-04-17T08:50:49.009992+0000","last_clean":"2024-04-17T08:50:49.009992+0000","last_became_active":"2024-04-17T08:50:32.905898+0000","last_became_peered":"2024-04-17T08:50:32.905898+0000","last_unstale":"2024-04-17T08:50:49.009992+0000","last_undegraded":"2024-04-17T08:50:49.009992+0000","last_fullsized":"2024-04-17T08:50:49.009992+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:24.678638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014219+0000","last_change":"2024-04-17T08:50:33.892430+0000","last_active":"2024-04-17T08:50:49.014219+0000","last_peered":"2024-04-17T08:50:49.014219+0000","last_clean":"2024-04-17T08:50:49.014219+0000","last_became_active":"2024-04-17T08:50:33.892133+0000","last_became_peered":"2024-04-17T08:50:33.892133+0000","last_unstale":"2024-04-17T08:50:49.014219+0000","last_undegraded":"2024-04-17T08:50:49.014219+0000","last_fullsized":"2024-04-17T08:50:49.014219+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:34.406770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016943+0000","last_change":"2024-04-17T08:50:48.017579+0000","last_active":"2024-04-17T08:50:49.016943+0000","last_peered":"2024-04-17T08:50:49.016943+0000","last_clean":"2024-04-17T08:50:49.016943+0000","last_became_active":"2024-04-17T08:50:48.016441+0000","last_became_peered":"2024-04-17T08:50:48.016441+0000","last_unstale":"2024-04-17T08:50:49.016943+0000","last_undegraded":"2024-04-17T08:50:49.016943+0000","last_fullsized":"2024-04-17T08:50:49.016943+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:30.341972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034323+0000","last_change":"2024-04-17T08:50:49.026143+0000","last_active":"2024-04-17T08:50:49.034323+0000","last_peered":"2024-04-17T08:50:49.034323+0000","last_clean":"2024-04-17T08:50:49.034323+0000","last_became_active":"2024-04-17T08:50:49.025856+0000","last_became_peered":"2024-04-17T08:50:49.025856+0000","last_unstale":"2024-04-17T08:50:49.034323+0000","last_undegraded":"2024-04-17T08:50:49.034323+0000","last_fullsized":"2024-04-17T08:50:49.034323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.074044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019705+0000","last_change":"2024-04-17T08:50:32.901213+0000","last_active":"2024-04-17T08:50:49.019705+0000","last_peered":"2024-04-17T08:50:49.019705+0000","last_clean":"2024-04-17T08:50:49.019705+0000","last_became_active":"2024-04-17T08:50:32.900942+0000","last_became_peered":"2024-04-17T08:50:32.900942+0000","last_unstale":"2024-04-17T08:50:49.019705+0000","last_undegraded":"2024-04-17T08:50:49.019705+0000","last_fullsized":"2024-04-17T08:50:49.019705+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:35.477286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018049+0000","last_change":"2024-04-17T08:50:33.887506+0000","last_active":"2024-04-17T08:50:49.018049+0000","last_peered":"2024-04-17T08:50:49.018049+0000","last_clean":"2024-04-17T08:50:49.018049+0000","last_became_active":"2024-04-17T08:50:33.887071+0000","last_became_peered":"2024-04-17T08:50:33.887071+0000","last_unstale":"2024-04-17T08:50:49.018049+0000","last_undegraded":"2024-04-17T08:50:49.018049+0000","last_fullsized":"2024-04-17T08:50:49.018049+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:23.927426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020363+0000","last_change":"2024-04-17T08:50:48.018181+0000","last_active":"2024-04-17T08:50:49.020363+0000","last_peered":"2024-04-17T08:50:49.020363+0000","last_clean":"2024-04-17T08:50:49.020363+0000","last_became_active":"2024-04-17T08:50:48.016820+0000","last_became_peered":"2024-04-17T08:50:48.016820+0000","last_unstale":"2024-04-17T08:50:49.020363+0000","last_undegraded":"2024-04-17T08:50:49.020363+0000","last_fullsized":"2024-04-17T08:50:49.020363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:51.729857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.043228+0000","last_change":"2024-04-17T08:50:49.039958+0000","last_active":"2024-04-17T08:50:49.043228+0000","last_peered":"2024-04-17T08:50:49.043228+0000","last_clean":"2024-04-17T08:50:49.043228+0000","last_became_active":"2024-04-17T08:50:49.039667+0000","last_became_peered":"2024-04-17T08:50:49.039667+0000","last_unstale":"2024-04-17T08:50:49.043228+0000","last_undegraded":"2024-04-17T08:50:49.043228+0000","last_fullsized":"2024-04-17T08:50:49.043228+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:10.730836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"230'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024098+0000","last_change":"2024-04-17T08:50:32.904455+0000","last_active":"2024-04-17T08:50:49.024098+0000","last_peered":"2024-04-17T08:50:49.024098+0000","last_clean":"2024-04-17T08:50:49.024098+0000","last_became_active":"2024-04-17T08:50:32.904127+0000","last_became_peered":"2024-04-17T08:50:32.904127+0000","last_unstale":"2024-04-17T08:50:49.024098+0000","last_undegraded":"2024-04-17T08:50:49.024098+0000","last_fullsized":"2024-04-17T08:50:49.024098+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:09.473209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017287+0000","last_change":"2024-04-17T08:50:33.892244+0000","last_active":"2024-04-17T08:50:49.017287+0000","last_peered":"2024-04-17T08:50:49.017287+0000","last_clean":"2024-04-17T08:50:49.017287+0000","last_became_active":"2024-04-17T08:50:33.891771+0000","last_became_peered":"2024-04-17T08:50:33.891771+0000","last_unstale":"2024-04-17T08:50:49.017287+0000","last_undegraded":"2024-04-17T08:50:49.017287+0000","last_fullsized":"2024-04-17T08:50:49.017287+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:29.039336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045634+0000","last_change":"2024-04-17T08:50:48.027200+0000","last_active":"2024-04-17T08:50:49.045634+0000","last_peered":"2024-04-17T08:50:49.045634+0000","last_clean":"2024-04-17T08:50:49.045634+0000","last_became_active":"2024-04-17T08:50:48.026771+0000","last_became_peered":"2024-04-17T08:50:48.026771+0000","last_unstale":"2024-04-17T08:50:49.045634+0000","last_undegraded":"2024-04-17T08:50:49.045634+0000","last_fullsized":"2024-04-17T08:50:49.045634+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:31.956458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054790+0000","last_change":"2024-04-17T08:50:49.047452+0000","last_active":"2024-04-17T08:50:49.054790+0000","last_peered":"2024-04-17T08:50:49.054790+0000","last_clean":"2024-04-17T08:50:49.054790+0000","last_became_active":"2024-04-17T08:50:49.047142+0000","last_became_peered":"2024-04-17T08:50:49.047142+0000","last_unstale":"2024-04-17T08:50:49.054790+0000","last_undegraded":"2024-04-17T08:50:49.054790+0000","last_fullsized":"2024-04-17T08:50:49.054790+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:01.781200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.006696+0000","last_change":"2024-04-17T08:50:32.906744+0000","last_active":"2024-04-17T08:50:49.006696+0000","last_peered":"2024-04-17T08:50:49.006696+0000","last_clean":"2024-04-17T08:50:49.006696+0000","last_became_active":"2024-04-17T08:50:32.906445+0000","last_became_peered":"2024-04-17T08:50:32.906445+0000","last_unstale":"2024-04-17T08:50:49.006696+0000","last_undegraded":"2024-04-17T08:50:49.006696+0000","last_fullsized":"2024-04-17T08:50:49.006696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:33.054437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054335+0000","last_change":"2024-04-17T08:50:33.899749+0000","last_active":"2024-04-17T08:50:49.054335+0000","last_peered":"2024-04-17T08:50:49.054335+0000","last_clean":"2024-04-17T08:50:49.054335+0000","last_became_active":"2024-04-17T08:50:33.898270+0000","last_became_peered":"2024-04-17T08:50:33.898270+0000","last_unstale":"2024-04-17T08:50:49.054335+0000","last_undegraded":"2024-04-17T08:50:49.054335+0000","last_fullsized":"2024-04-17T08:50:49.054335+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:51.073059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014937+0000","last_change":"2024-04-17T08:50:48.023544+0000","last_active":"2024-04-17T08:50:49.014937+0000","last_peered":"2024-04-17T08:50:49.014937+0000","last_clean":"2024-04-17T08:50:49.014937+0000","last_became_active":"2024-04-17T08:50:48.023218+0000","last_became_peered":"2024-04-17T08:50:48.023218+0000","last_unstale":"2024-04-17T08:50:49.014937+0000","last_undegraded":"2024-04-17T08:50:49.014937+0000","last_fullsized":"2024-04-17T08:50:49.014937+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:12.937584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.042867+0000","last_change":"2024-04-17T08:50:49.040211+0000","last_active":"2024-04-17T08:50:49.042867+0000","last_peered":"2024-04-17T08:50:49.042867+0000","last_clean":"2024-04-17T08:50:49.042867+0000","last_became_active":"2024-04-17T08:50:49.039900+0000","last_became_peered":"2024-04-17T08:50:49.039900+0000","last_unstale":"2024-04-17T08:50:49.042867+0000","last_undegraded":"2024-04-17T08:50:49.042867+0000","last_fullsized":"2024-04-17T08:50:49.042867+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:55.621600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027356+0000","last_change":"2024-04-17T08:50:32.894904+0000","last_active":"2024-04-17T08:50:49.027356+0000","last_peered":"2024-04-17T08:50:49.027356+0000","last_clean":"2024-04-17T08:50:49.027356+0000","last_became_active":"2024-04-17T08:50:32.894624+0000","last_became_peered":"2024-04-17T08:50:32.894624+0000","last_unstale":"2024-04-17T08:50:49.027356+0000","last_undegraded":"2024-04-17T08:50:49.027356+0000","last_fullsized":"2024-04-17T08:50:49.027356+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:22.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011363+0000","last_change":"2024-04-17T08:50:33.892798+0000","last_active":"2024-04-17T08:50:49.011363+0000","last_peered":"2024-04-17T08:50:49.011363+0000","last_clean":"2024-04-17T08:50:49.011363+0000","last_became_active":"2024-04-17T08:50:33.891130+0000","last_became_peered":"2024-04-17T08:50:33.891130+0000","last_unstale":"2024-04-17T08:50:49.011363+0000","last_undegraded":"2024-04-17T08:50:49.011363+0000","last_fullsized":"2024-04-17T08:50:49.011363+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:43.282486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052741+0000","last_change":"2024-04-17T08:50:48.021836+0000","last_active":"2024-04-17T08:50:49.052741+0000","last_peered":"2024-04-17T08:50:49.052741+0000","last_clean":"2024-04-17T08:50:49.052741+0000","last_became_active":"2024-04-17T08:50:48.019867+0000","last_became_peered":"2024-04-17T08:50:48.019867+0000","last_unstale":"2024-04-17T08:50:49.052741+0000","last_undegraded":"2024-04-17T08:50:49.052741+0000","last_fullsized":"2024-04-17T08:50:49.052741+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:13.501771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.035124+0000","last_change":"2024-04-17T08:50:49.030997+0000","last_active":"2024-04-17T08:50:49.035124+0000","last_peered":"2024-04-17T08:50:49.035124+0000","last_clean":"2024-04-17T08:50:49.035124+0000","last_became_active":"2024-04-17T08:50:49.030558+0000","last_became_peered":"2024-04-17T08:50:49.030558+0000","last_unstale":"2024-04-17T08:50:49.035124+0000","last_undegraded":"2024-04-17T08:50:49.035124+0000","last_fullsized":"2024-04-17T08:50:49.035124+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:11.503895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019846+0000","last_change":"2024-04-17T08:50:32.896102+0000","last_active":"2024-04-17T08:50:49.019846+0000","last_peered":"2024-04-17T08:50:49.019846+0000","last_clean":"2024-04-17T08:50:49.019846+0000","last_became_active":"2024-04-17T08:50:32.895808+0000","last_became_peered":"2024-04-17T08:50:32.895808+0000","last_unstale":"2024-04-17T08:50:49.019846+0000","last_undegraded":"2024-04-17T08:50:49.019846+0000","last_fullsized":"2024-04-17T08:50:49.019846+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:38.551687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016137+0000","last_change":"2024-04-17T08:50:33.884383+0000","last_active":"2024-04-17T08:50:49.016137+0000","last_peered":"2024-04-17T08:50:49.016137+0000","last_clean":"2024-04-17T08:50:49.016137+0000","last_became_active":"2024-04-17T08:50:33.883473+0000","last_became_peered":"2024-04-17T08:50:33.883473+0000","last_unstale":"2024-04-17T08:50:49.016137+0000","last_undegraded":"2024-04-17T08:50:49.016137+0000","last_fullsized":"2024-04-17T08:50:49.016137+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:03.466744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060403+0000","last_change":"2024-04-17T08:50:48.032503+0000","last_active":"2024-04-17T08:50:49.060403+0000","last_peered":"2024-04-17T08:50:49.060403+0000","last_clean":"2024-04-17T08:50:49.060403+0000","last_became_active":"2024-04-17T08:50:48.032090+0000","last_became_peered":"2024-04-17T08:50:48.032090+0000","last_unstale":"2024-04-17T08:50:49.060403+0000","last_undegraded":"2024-04-17T08:50:49.060403+0000","last_fullsized":"2024-04-17T08:50:49.060403+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:38.033613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045756+0000","last_change":"2024-04-17T08:50:49.037319+0000","last_active":"2024-04-17T08:50:49.045756+0000","last_peered":"2024-04-17T08:50:49.045756+0000","last_clean":"2024-04-17T08:50:49.045756+0000","last_became_active":"2024-04-17T08:50:49.037012+0000","last_became_peered":"2024-04-17T08:50:49.037012+0000","last_unstale":"2024-04-17T08:50:49.045756+0000","last_undegraded":"2024-04-17T08:50:49.045756+0000","last_fullsized":"2024-04-17T08:50:49.045756+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:10.872839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024506+0000","last_change":"2024-04-17T08:50:32.902321+0000","last_active":"2024-04-17T08:50:49.024506+0000","last_peered":"2024-04-17T08:50:49.024506+0000","last_clean":"2024-04-17T08:50:49.024506+0000","last_became_active":"2024-04-17T08:50:32.901781+0000","last_became_peered":"2024-04-17T08:50:32.901781+0000","last_unstale":"2024-04-17T08:50:49.024506+0000","last_undegraded":"2024-04-17T08:50:49.024506+0000","last_fullsized":"2024-04-17T08:50:49.024506+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:04.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004920+0000","last_change":"2024-04-17T08:50:33.897335+0000","last_active":"2024-04-17T08:50:49.004920+0000","last_peered":"2024-04-17T08:50:49.004920+0000","last_clean":"2024-04-17T08:50:49.004920+0000","last_became_active":"2024-04-17T08:50:33.897009+0000","last_became_peered":"2024-04-17T08:50:33.897009+0000","last_unstale":"2024-04-17T08:50:49.004920+0000","last_undegraded":"2024-04-17T08:50:49.004920+0000","last_fullsized":"2024-04-17T08:50:49.004920+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:12.674509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023238+0000","last_change":"2024-04-17T08:50:48.019566+0000","last_active":"2024-04-17T08:50:49.023238+0000","last_peered":"2024-04-17T08:50:49.023238+0000","last_clean":"2024-04-17T08:50:49.023238+0000","last_became_active":"2024-04-17T08:50:48.018570+0000","last_became_peered":"2024-04-17T08:50:48.018570+0000","last_unstale":"2024-04-17T08:50:49.023238+0000","last_undegraded":"2024-04-17T08:50:49.023238+0000","last_fullsized":"2024-04-17T08:50:49.023238+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:36.720735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046848+0000","last_change":"2024-04-17T08:50:49.037873+0000","last_active":"2024-04-17T08:50:49.046848+0000","last_peered":"2024-04-17T08:50:49.046848+0000","last_clean":"2024-04-17T08:50:49.046848+0000","last_became_active":"2024-04-17T08:50:49.037589+0000","last_became_peered":"2024-04-17T08:50:49.037589+0000","last_unstale":"2024-04-17T08:50:49.046848+0000","last_undegraded":"2024-04-17T08:50:49.046848+0000","last_fullsized":"2024-04-17T08:50:49.046848+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:02.288019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026728+0000","last_change":"2024-04-17T08:50:32.898032+0000","last_active":"2024-04-17T08:50:49.026728+0000","last_peered":"2024-04-17T08:50:49.026728+0000","last_clean":"2024-04-17T08:50:49.026728+0000","last_became_active":"2024-04-17T08:50:32.897755+0000","last_became_peered":"2024-04-17T08:50:32.897755+0000","last_unstale":"2024-04-17T08:50:49.026728+0000","last_undegraded":"2024-04-17T08:50:49.026728+0000","last_fullsized":"2024-04-17T08:50:49.026728+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:29.382845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023428+0000","last_change":"2024-04-17T08:50:33.897197+0000","last_active":"2024-04-17T08:50:49.023428+0000","last_peered":"2024-04-17T08:50:49.023428+0000","last_clean":"2024-04-17T08:50:49.023428+0000","last_became_active":"2024-04-17T08:50:33.895389+0000","last_became_peered":"2024-04-17T08:50:33.895389+0000","last_unstale":"2024-04-17T08:50:49.023428+0000","last_undegraded":"2024-04-17T08:50:49.023428+0000","last_fullsized":"2024-04-17T08:50:49.023428+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:09.426321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048980+0000","last_change":"2024-04-17T08:50:48.019845+0000","last_active":"2024-04-17T08:50:49.048980+0000","last_peered":"2024-04-17T08:50:49.048980+0000","last_clean":"2024-04-17T08:50:49.048980+0000","last_became_active":"2024-04-17T08:50:48.019186+0000","last_became_peered":"2024-04-17T08:50:48.019186+0000","last_unstale":"2024-04-17T08:50:49.048980+0000","last_undegraded":"2024-04-17T08:50:49.048980+0000","last_fullsized":"2024-04-17T08:50:49.048980+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:50.726916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059762+0000","last_change":"2024-04-17T08:50:49.055993+0000","last_active":"2024-04-17T08:50:49.059762+0000","last_peered":"2024-04-17T08:50:49.059762+0000","last_clean":"2024-04-17T08:50:49.059762+0000","last_became_active":"2024-04-17T08:50:49.055537+0000","last_became_peered":"2024-04-17T08:50:49.055537+0000","last_unstale":"2024-04-17T08:50:49.059762+0000","last_undegraded":"2024-04-17T08:50:49.059762+0000","last_fullsized":"2024-04-17T08:50:49.059762+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:01.853612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009754+0000","last_change":"2024-04-17T08:50:32.903243+0000","last_active":"2024-04-17T08:50:49.009754+0000","last_peered":"2024-04-17T08:50:49.009754+0000","last_clean":"2024-04-17T08:50:49.009754+0000","last_became_active":"2024-04-17T08:50:32.902864+0000","last_became_peered":"2024-04-17T08:50:32.902864+0000","last_unstale":"2024-04-17T08:50:49.009754+0000","last_undegraded":"2024-04-17T08:50:49.009754+0000","last_fullsized":"2024-04-17T08:50:49.009754+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:20.222164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023609+0000","last_change":"2024-04-17T08:50:33.905253+0000","last_active":"2024-04-17T08:50:49.023609+0000","last_peered":"2024-04-17T08:50:49.023609+0000","last_clean":"2024-04-17T08:50:49.023609+0000","last_became_active":"2024-04-17T08:50:33.904731+0000","last_became_peered":"2024-04-17T08:50:33.904731+0000","last_unstale":"2024-04-17T08:50:49.023609+0000","last_undegraded":"2024-04-17T08:50:49.023609+0000","last_fullsized":"2024-04-17T08:50:49.023609+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:56.634018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016476+0000","last_change":"2024-04-17T08:50:48.018547+0000","last_active":"2024-04-17T08:50:49.016476+0000","last_peered":"2024-04-17T08:50:49.016476+0000","last_clean":"2024-04-17T08:50:49.016476+0000","last_became_active":"2024-04-17T08:50:48.018220+0000","last_became_peered":"2024-04-17T08:50:48.018220+0000","last_unstale":"2024-04-17T08:50:49.016476+0000","last_undegraded":"2024-04-17T08:50:49.016476+0000","last_fullsized":"2024-04-17T08:50:49.016476+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:00.149916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046188+0000","last_change":"2024-04-17T08:50:49.044469+0000","last_active":"2024-04-17T08:50:49.046188+0000","last_peered":"2024-04-17T08:50:49.046188+0000","last_clean":"2024-04-17T08:50:49.046188+0000","last_became_active":"2024-04-17T08:50:49.044170+0000","last_became_peered":"2024-04-17T08:50:49.044170+0000","last_unstale":"2024-04-17T08:50:49.046188+0000","last_undegraded":"2024-04-17T08:50:49.046188+0000","last_fullsized":"2024-04-17T08:50:49.046188+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:53.313084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026292+0000","last_change":"2024-04-17T08:50:32.896384+0000","last_active":"2024-04-17T08:50:49.026292+0000","last_peered":"2024-04-17T08:50:49.026292+0000","last_clean":"2024-04-17T08:50:49.026292+0000","last_became_active":"2024-04-17T08:50:32.893495+0000","last_became_peered":"2024-04-17T08:50:32.893495+0000","last_unstale":"2024-04-17T08:50:49.026292+0000","last_undegraded":"2024-04-17T08:50:49.026292+0000","last_fullsized":"2024-04-17T08:50:49.026292+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.852683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009473+0000","last_change":"2024-04-17T08:50:33.899313+0000","last_active":"2024-04-17T08:50:49.009473+0000","last_peered":"2024-04-17T08:50:49.009473+0000","last_clean":"2024-04-17T08:50:49.009473+0000","last_became_active":"2024-04-17T08:50:33.899005+0000","last_became_peered":"2024-04-17T08:50:33.899005+0000","last_unstale":"2024-04-17T08:50:49.009473+0000","last_undegraded":"2024-04-17T08:50:49.009473+0000","last_fullsized":"2024-04-17T08:50:49.009473+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:15:34.655071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060016+0000","last_change":"2024-04-17T08:50:48.033453+0000","last_active":"2024-04-17T08:50:49.060016+0000","last_peered":"2024-04-17T08:50:49.060016+0000","last_clean":"2024-04-17T08:50:49.060016+0000","last_became_active":"2024-04-17T08:50:48.032899+0000","last_became_peered":"2024-04-17T08:50:48.032899+0000","last_unstale":"2024-04-17T08:50:49.060016+0000","last_undegraded":"2024-04-17T08:50:49.060016+0000","last_fullsized":"2024-04-17T08:50:49.060016+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:33.289131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045946+0000","last_change":"2024-04-17T08:50:49.042134+0000","last_active":"2024-04-17T08:50:49.045946+0000","last_peered":"2024-04-17T08:50:49.045946+0000","last_clean":"2024-04-17T08:50:49.045946+0000","last_became_active":"2024-04-17T08:50:49.041627+0000","last_became_peered":"2024-04-17T08:50:49.041627+0000","last_unstale":"2024-04-17T08:50:49.045946+0000","last_undegraded":"2024-04-17T08:50:49.045946+0000","last_fullsized":"2024-04-17T08:50:49.045946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:59.522791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020619+0000","last_change":"2024-04-17T08:50:32.901050+0000","last_active":"2024-04-17T08:50:49.020619+0000","last_peered":"2024-04-17T08:50:49.020619+0000","last_clean":"2024-04-17T08:50:49.020619+0000","last_became_active":"2024-04-17T08:50:32.900689+0000","last_became_peered":"2024-04-17T08:50:32.900689+0000","last_unstale":"2024-04-17T08:50:49.020619+0000","last_undegraded":"2024-04-17T08:50:49.020619+0000","last_fullsized":"2024-04-17T08:50:49.020619+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:39:51.234349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019043+0000","last_change":"2024-04-17T08:50:33.907099+0000","last_active":"2024-04-17T08:50:49.019043+0000","last_peered":"2024-04-17T08:50:49.019043+0000","last_clean":"2024-04-17T08:50:49.019043+0000","last_became_active":"2024-04-17T08:50:33.906802+0000","last_became_peered":"2024-04-17T08:50:33.906802+0000","last_unstale":"2024-04-17T08:50:49.019043+0000","last_undegraded":"2024-04-17T08:50:49.019043+0000","last_fullsized":"2024-04-17T08:50:49.019043+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:53.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047133+0000","last_change":"2024-04-17T08:50:48.028554+0000","last_active":"2024-04-17T08:50:49.047133+0000","last_peered":"2024-04-17T08:50:49.047133+0000","last_clean":"2024-04-17T08:50:49.047133+0000","last_became_active":"2024-04-17T08:50:48.028241+0000","last_became_peered":"2024-04-17T08:50:48.028241+0000","last_unstale":"2024-04-17T08:50:49.047133+0000","last_undegraded":"2024-04-17T08:50:49.047133+0000","last_fullsized":"2024-04-17T08:50:49.047133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:49.274398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.035802+0000","last_change":"2024-04-17T08:50:49.032405+0000","last_active":"2024-04-17T08:50:49.035802+0000","last_peered":"2024-04-17T08:50:49.035802+0000","last_clean":"2024-04-17T08:50:49.035802+0000","last_became_active":"2024-04-17T08:50:49.031813+0000","last_became_peered":"2024-04-17T08:50:49.031813+0000","last_unstale":"2024-04-17T08:50:49.035802+0000","last_undegraded":"2024-04-17T08:50:49.035802+0000","last_fullsized":"2024-04-17T08:50:49.035802+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:23.883528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017595+0000","last_change":"2024-04-17T08:50:32.891171+0000","last_active":"2024-04-17T08:50:49.017595+0000","last_peered":"2024-04-17T08:50:49.017595+0000","last_clean":"2024-04-17T08:50:49.017595+0000","last_became_active":"2024-04-17T08:50:32.888285+0000","last_became_peered":"2024-04-17T08:50:32.888285+0000","last_unstale":"2024-04-17T08:50:49.017595+0000","last_undegraded":"2024-04-17T08:50:49.017595+0000","last_fullsized":"2024-04-17T08:50:49.017595+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.387644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019578+0000","last_change":"2024-04-17T08:50:33.887711+0000","last_active":"2024-04-17T08:50:49.019578+0000","last_peered":"2024-04-17T08:50:49.019578+0000","last_clean":"2024-04-17T08:50:49.019578+0000","last_became_active":"2024-04-17T08:50:33.887362+0000","last_became_peered":"2024-04-17T08:50:33.887362+0000","last_unstale":"2024-04-17T08:50:49.019578+0000","last_undegraded":"2024-04-17T08:50:49.019578+0000","last_fullsized":"2024-04-17T08:50:49.019578+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:07.536049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017490+0000","last_change":"2024-04-17T08:50:48.023932+0000","last_active":"2024-04-17T08:50:49.017490+0000","last_peered":"2024-04-17T08:50:49.017490+0000","last_clean":"2024-04-17T08:50:49.017490+0000","last_became_active":"2024-04-17T08:50:48.023640+0000","last_became_peered":"2024-04-17T08:50:48.023640+0000","last_unstale":"2024-04-17T08:50:49.017490+0000","last_undegraded":"2024-04-17T08:50:49.017490+0000","last_fullsized":"2024-04-17T08:50:49.017490+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:01.151039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045189+0000","last_change":"2024-04-17T08:50:49.042247+0000","last_active":"2024-04-17T08:50:49.045189+0000","last_peered":"2024-04-17T08:50:49.045189+0000","last_clean":"2024-04-17T08:50:49.045189+0000","last_became_active":"2024-04-17T08:50:49.041744+0000","last_became_peered":"2024-04-17T08:50:49.041744+0000","last_unstale":"2024-04-17T08:50:49.045189+0000","last_undegraded":"2024-04-17T08:50:49.045189+0000","last_fullsized":"2024-04-17T08:50:49.045189+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:57.766992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"229'8","reported_seq":28,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005497+0000","last_change":"2024-04-17T08:50:32.903713+0000","last_active":"2024-04-17T08:50:49.005497+0000","last_peered":"2024-04-17T08:50:49.005497+0000","last_clean":"2024-04-17T08:50:49.005497+0000","last_became_active":"2024-04-17T08:50:32.903403+0000","last_became_peered":"2024-04-17T08:50:32.903403+0000","last_unstale":"2024-04-17T08:50:49.005497+0000","last_undegraded":"2024-04-17T08:50:49.005497+0000","last_fullsized":"2024-04-17T08:50:49.005497+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:09.595683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019655+0000","last_change":"2024-04-17T08:50:33.889134+0000","last_active":"2024-04-17T08:50:49.019655+0000","last_peered":"2024-04-17T08:50:49.019655+0000","last_clean":"2024-04-17T08:50:49.019655+0000","last_became_active":"2024-04-17T08:50:33.886115+0000","last_became_peered":"2024-04-17T08:50:33.886115+0000","last_unstale":"2024-04-17T08:50:49.019655+0000","last_undegraded":"2024-04-17T08:50:49.019655+0000","last_fullsized":"2024-04-17T08:50:49.019655+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:20.719327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.015688+0000","last_change":"2024-04-17T08:50:48.017932+0000","last_active":"2024-04-17T08:50:49.015688+0000","last_peered":"2024-04-17T08:50:49.015688+0000","last_clean":"2024-04-17T08:50:49.015688+0000","last_became_active":"2024-04-17T08:50:48.017662+0000","last_became_peered":"2024-04-17T08:50:48.017662+0000","last_unstale":"2024-04-17T08:50:49.015688+0000","last_undegraded":"2024-04-17T08:50:49.015688+0000","last_fullsized":"2024-04-17T08:50:49.015688+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:44.878825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047027+0000","last_change":"2024-04-17T08:50:49.045258+0000","last_active":"2024-04-17T08:50:49.047027+0000","last_peered":"2024-04-17T08:50:49.047027+0000","last_clean":"2024-04-17T08:50:49.047027+0000","last_became_active":"2024-04-17T08:50:49.044518+0000","last_became_peered":"2024-04-17T08:50:49.044518+0000","last_unstale":"2024-04-17T08:50:49.047027+0000","last_undegraded":"2024-04-17T08:50:49.047027+0000","last_fullsized":"2024-04-17T08:50:49.047027+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:13.816186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025562+0000","last_change":"2024-04-17T08:50:32.895429+0000","last_active":"2024-04-17T08:50:49.025562+0000","last_peered":"2024-04-17T08:50:49.025562+0000","last_clean":"2024-04-17T08:50:49.025562+0000","last_became_active":"2024-04-17T08:50:32.894971+0000","last_became_peered":"2024-04-17T08:50:32.894971+0000","last_unstale":"2024-04-17T08:50:49.025562+0000","last_undegraded":"2024-04-17T08:50:49.025562+0000","last_fullsized":"2024-04-17T08:50:49.025562+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:22.318570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004172+0000","last_change":"2024-04-17T08:50:33.896023+0000","last_active":"2024-04-17T08:50:49.004172+0000","last_peered":"2024-04-17T08:50:49.004172+0000","last_clean":"2024-04-17T08:50:49.004172+0000","last_became_active":"2024-04-17T08:50:33.895404+0000","last_became_peered":"2024-04-17T08:50:33.895404+0000","last_unstale":"2024-04-17T08:50:49.004172+0000","last_undegraded":"2024-04-17T08:50:49.004172+0000","last_fullsized":"2024-04-17T08:50:49.004172+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:17.385187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.032923+0000","last_change":"2024-04-17T08:50:48.027804+0000","last_active":"2024-04-17T08:50:49.032923+0000","last_peered":"2024-04-17T08:50:49.032923+0000","last_clean":"2024-04-17T08:50:49.032923+0000","last_became_active":"2024-04-17T08:50:48.027446+0000","last_became_peered":"2024-04-17T08:50:48.027446+0000","last_unstale":"2024-04-17T08:50:49.032923+0000","last_undegraded":"2024-04-17T08:50:49.032923+0000","last_fullsized":"2024-04-17T08:50:49.032923+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:07.905438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045017+0000","last_change":"2024-04-17T08:50:49.041513+0000","last_active":"2024-04-17T08:50:49.045017+0000","last_peered":"2024-04-17T08:50:49.045017+0000","last_clean":"2024-04-17T08:50:49.045017+0000","last_became_active":"2024-04-17T08:50:49.041235+0000","last_became_peered":"2024-04-17T08:50:49.041235+0000","last_unstale":"2024-04-17T08:50:49.045017+0000","last_undegraded":"2024-04-17T08:50:49.045017+0000","last_fullsized":"2024-04-17T08:50:49.045017+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:54.702081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012191+0000","last_change":"2024-04-17T08:50:32.896932+0000","last_active":"2024-04-17T08:50:49.012191+0000","last_peered":"2024-04-17T08:50:49.012191+0000","last_clean":"2024-04-17T08:50:49.012191+0000","last_became_active":"2024-04-17T08:50:32.895134+0000","last_became_peered":"2024-04-17T08:50:32.895134+0000","last_unstale":"2024-04-17T08:50:49.012191+0000","last_undegraded":"2024-04-17T08:50:49.012191+0000","last_fullsized":"2024-04-17T08:50:49.012191+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:02.447617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025672+0000","last_change":"2024-04-17T08:50:33.909416+0000","last_active":"2024-04-17T08:50:49.025672+0000","last_peered":"2024-04-17T08:50:49.025672+0000","last_clean":"2024-04-17T08:50:49.025672+0000","last_became_active":"2024-04-17T08:50:33.906483+0000","last_became_peered":"2024-04-17T08:50:33.906483+0000","last_unstale":"2024-04-17T08:50:49.025672+0000","last_undegraded":"2024-04-17T08:50:49.025672+0000","last_fullsized":"2024-04-17T08:50:49.025672+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:07:00.037620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"233'6","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059068+0000","last_change":"2024-04-17T08:50:48.025812+0000","last_active":"2024-04-17T08:50:49.059068+0000","last_peered":"2024-04-17T08:50:49.059068+0000","last_clean":"2024-04-17T08:50:49.059068+0000","last_became_active":"2024-04-17T08:50:48.024873+0000","last_became_peered":"2024-04-17T08:50:48.024873+0000","last_unstale":"2024-04-17T08:50:49.059068+0000","last_undegraded":"2024-04-17T08:50:49.059068+0000","last_fullsized":"2024-04-17T08:50:49.059068+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:42.764709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046944+0000","last_change":"2024-04-17T08:50:49.039961+0000","last_active":"2024-04-17T08:50:49.046944+0000","last_peered":"2024-04-17T08:50:49.046944+0000","last_clean":"2024-04-17T08:50:49.046944+0000","last_became_active":"2024-04-17T08:50:49.039326+0000","last_became_peered":"2024-04-17T08:50:49.039326+0000","last_unstale":"2024-04-17T08:50:49.046944+0000","last_undegraded":"2024-04-17T08:50:49.046944+0000","last_fullsized":"2024-04-17T08:50:49.046944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:43.347365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011949+0000","last_change":"2024-04-17T08:50:32.891648+0000","last_active":"2024-04-17T08:50:49.011949+0000","last_peered":"2024-04-17T08:50:49.011949+0000","last_clean":"2024-04-17T08:50:49.011949+0000","last_became_active":"2024-04-17T08:50:32.890804+0000","last_became_peered":"2024-04-17T08:50:32.890804+0000","last_unstale":"2024-04-17T08:50:49.011949+0000","last_undegraded":"2024-04-17T08:50:49.011949+0000","last_fullsized":"2024-04-17T08:50:49.011949+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:50.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013873+0000","last_change":"2024-04-17T08:50:33.908242+0000","last_active":"2024-04-17T08:50:49.013873+0000","last_peered":"2024-04-17T08:50:49.013873+0000","last_clean":"2024-04-17T08:50:49.013873+0000","last_became_active":"2024-04-17T08:50:33.907288+0000","last_became_peered":"2024-04-17T08:50:33.907288+0000","last_unstale":"2024-04-17T08:50:49.013873+0000","last_undegraded":"2024-04-17T08:50:49.013873+0000","last_fullsized":"2024-04-17T08:50:49.013873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:45.950784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"233'3","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052186+0000","last_change":"2024-04-17T08:50:48.018930+0000","last_active":"2024-04-17T08:50:49.052186+0000","last_peered":"2024-04-17T08:50:49.052186+0000","last_clean":"2024-04-17T08:50:49.052186+0000","last_became_active":"2024-04-17T08:50:48.018088+0000","last_became_peered":"2024-04-17T08:50:48.018088+0000","last_unstale":"2024-04-17T08:50:49.052186+0000","last_undegraded":"2024-04-17T08:50:49.052186+0000","last_fullsized":"2024-04-17T08:50:49.052186+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:59.266629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.055303+0000","last_change":"2024-04-17T08:50:49.045041+0000","last_active":"2024-04-17T08:50:49.055303+0000","last_peered":"2024-04-17T08:50:49.055303+0000","last_clean":"2024-04-17T08:50:49.055303+0000","last_became_active":"2024-04-17T08:50:49.044767+0000","last_became_peered":"2024-04-17T08:50:49.044767+0000","last_unstale":"2024-04-17T08:50:49.055303+0000","last_undegraded":"2024-04-17T08:50:49.055303+0000","last_fullsized":"2024-04-17T08:50:49.055303+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:46:46.880409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"230'8","reported_seq":28,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028849+0000","last_change":"2024-04-17T08:50:32.906064+0000","last_active":"2024-04-17T08:50:49.028849+0000","last_peered":"2024-04-17T08:50:49.028849+0000","last_clean":"2024-04-17T08:50:49.028849+0000","last_became_active":"2024-04-17T08:50:32.904959+0000","last_became_peered":"2024-04-17T08:50:32.904959+0000","last_unstale":"2024-04-17T08:50:49.028849+0000","last_undegraded":"2024-04-17T08:50:49.028849+0000","last_fullsized":"2024-04-17T08:50:49.028849+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:55.614891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013403+0000","last_change":"2024-04-17T08:50:33.904088+0000","last_active":"2024-04-17T08:50:49.013403+0000","last_peered":"2024-04-17T08:50:49.013403+0000","last_clean":"2024-04-17T08:50:49.013403+0000","last_became_active":"2024-04-17T08:50:33.903730+0000","last_became_peered":"2024-04-17T08:50:33.903730+0000","last_unstale":"2024-04-17T08:50:49.013403+0000","last_undegraded":"2024-04-17T08:50:49.013403+0000","last_fullsized":"2024-04-17T08:50:49.013403+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:04:51.186798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019121+0000","last_change":"2024-04-17T08:50:48.021343+0000","last_active":"2024-04-17T08:50:49.019121+0000","last_peered":"2024-04-17T08:50:49.019121+0000","last_clean":"2024-04-17T08:50:49.019121+0000","last_became_active":"2024-04-17T08:50:48.020747+0000","last_became_peered":"2024-04-17T08:50:48.020747+0000","last_unstale":"2024-04-17T08:50:49.019121+0000","last_undegraded":"2024-04-17T08:50:49.019121+0000","last_fullsized":"2024-04-17T08:50:49.019121+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:55:09.798676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.030348+0000","last_change":"2024-04-17T08:50:49.028141+0000","last_active":"2024-04-17T08:50:49.030348+0000","last_peered":"2024-04-17T08:50:49.030348+0000","last_clean":"2024-04-17T08:50:49.030348+0000","last_became_active":"2024-04-17T08:50:49.027849+0000","last_became_peered":"2024-04-17T08:50:49.027849+0000","last_unstale":"2024-04-17T08:50:49.030348+0000","last_undegraded":"2024-04-17T08:50:49.030348+0000","last_fullsized":"2024-04-17T08:50:49.030348+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:50.677426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"229'3","reported_seq":23,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026153+0000","last_change":"2024-04-17T08:50:32.892710+0000","last_active":"2024-04-17T08:50:49.026153+0000","last_peered":"2024-04-17T08:50:49.026153+0000","last_clean":"2024-04-17T08:50:49.026153+0000","last_became_active":"2024-04-17T08:50:32.892204+0000","last_became_peered":"2024-04-17T08:50:32.892204+0000","last_unstale":"2024-04-17T08:50:49.026153+0000","last_undegraded":"2024-04-17T08:50:49.026153+0000","last_fullsized":"2024-04-17T08:50:49.026153+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:01.085554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024767+0000","last_change":"2024-04-17T08:50:33.900099+0000","last_active":"2024-04-17T08:50:49.024767+0000","last_peered":"2024-04-17T08:50:49.024767+0000","last_clean":"2024-04-17T08:50:49.024767+0000","last_became_active":"2024-04-17T08:50:33.899785+0000","last_became_peered":"2024-04-17T08:50:33.899785+0000","last_unstale":"2024-04-17T08:50:49.024767+0000","last_undegraded":"2024-04-17T08:50:49.024767+0000","last_fullsized":"2024-04-17T08:50:49.024767+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:22.614548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060742+0000","last_change":"2024-04-17T08:50:48.033102+0000","last_active":"2024-04-17T08:50:49.060742+0000","last_peered":"2024-04-17T08:50:49.060742+0000","last_clean":"2024-04-17T08:50:49.060742+0000","last_became_active":"2024-04-17T08:50:48.032743+0000","last_became_peered":"2024-04-17T08:50:48.032743+0000","last_unstale":"2024-04-17T08:50:49.060742+0000","last_undegraded":"2024-04-17T08:50:49.060742+0000","last_fullsized":"2024-04-17T08:50:49.060742+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:08.497645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.036336+0000","last_change":"2024-04-17T08:50:49.032687+0000","last_active":"2024-04-17T08:50:49.036336+0000","last_peered":"2024-04-17T08:50:49.036336+0000","last_clean":"2024-04-17T08:50:49.036336+0000","last_became_active":"2024-04-17T08:50:49.032374+0000","last_became_peered":"2024-04-17T08:50:49.032374+0000","last_unstale":"2024-04-17T08:50:49.036336+0000","last_undegraded":"2024-04-17T08:50:49.036336+0000","last_fullsized":"2024-04-17T08:50:49.036336+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:11.499171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014273+0000","last_change":"2024-04-17T08:50:32.903765+0000","last_active":"2024-04-17T08:50:49.014273+0000","last_peered":"2024-04-17T08:50:49.014273+0000","last_clean":"2024-04-17T08:50:49.014273+0000","last_became_active":"2024-04-17T08:50:32.902284+0000","last_became_peered":"2024-04-17T08:50:32.902284+0000","last_unstale":"2024-04-17T08:50:49.014273+0000","last_undegraded":"2024-04-17T08:50:49.014273+0000","last_fullsized":"2024-04-17T08:50:49.014273+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:41.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.006177+0000","last_change":"2024-04-17T08:50:33.903308+0000","last_active":"2024-04-17T08:50:49.006177+0000","last_peered":"2024-04-17T08:50:49.006177+0000","last_clean":"2024-04-17T08:50:49.006177+0000","last_became_active":"2024-04-17T08:50:33.902988+0000","last_became_peered":"2024-04-17T08:50:33.902988+0000","last_unstale":"2024-04-17T08:50:49.006177+0000","last_undegraded":"2024-04-17T08:50:49.006177+0000","last_fullsized":"2024-04-17T08:50:49.006177+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:00.031705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"233'8","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059511+0000","last_change":"2024-04-17T08:50:48.025753+0000","last_active":"2024-04-17T08:50:49.059511+0000","last_peered":"2024-04-17T08:50:49.059511+0000","last_clean":"2024-04-17T08:50:49.059511+0000","last_became_active":"2024-04-17T08:50:48.025431+0000","last_became_peered":"2024-04-17T08:50:48.025431+0000","last_unstale":"2024-04-17T08:50:49.059511+0000","last_undegraded":"2024-04-17T08:50:49.059511+0000","last_fullsized":"2024-04-17T08:50:49.059511+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:51.729857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054770+0000","last_change":"2024-04-17T08:50:49.047279+0000","last_active":"2024-04-17T08:50:49.054770+0000","last_peered":"2024-04-17T08:50:49.054770+0000","last_clean":"2024-04-17T08:50:49.054770+0000","last_became_active":"2024-04-17T08:50:49.046698+0000","last_became_peered":"2024-04-17T08:50:49.046698+0000","last_unstale":"2024-04-17T08:50:49.054770+0000","last_undegraded":"2024-04-17T08:50:49.054770+0000","last_fullsized":"2024-04-17T08:50:49.054770+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:18.590004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012956+0000","last_change":"2024-04-17T08:50:32.901844+0000","last_active":"2024-04-17T08:50:49.012956+0000","last_peered":"2024-04-17T08:50:49.012956+0000","last_clean":"2024-04-17T08:50:49.012956+0000","last_became_active":"2024-04-17T08:50:32.901571+0000","last_became_peered":"2024-04-17T08:50:32.901571+0000","last_unstale":"2024-04-17T08:50:49.012956+0000","last_undegraded":"2024-04-17T08:50:49.012956+0000","last_fullsized":"2024-04-17T08:50:49.012956+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:42.263359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023967+0000","last_change":"2024-04-17T08:50:33.899408+0000","last_active":"2024-04-17T08:50:49.023967+0000","last_peered":"2024-04-17T08:50:49.023967+0000","last_clean":"2024-04-17T08:50:49.023967+0000","last_became_active":"2024-04-17T08:50:33.898922+0000","last_became_peered":"2024-04-17T08:50:33.898922+0000","last_unstale":"2024-04-17T08:50:49.023967+0000","last_undegraded":"2024-04-17T08:50:49.023967+0000","last_fullsized":"2024-04-17T08:50:49.023967+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:24.049886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014507+0000","last_change":"2024-04-17T08:50:48.019963+0000","last_active":"2024-04-17T08:50:49.014507+0000","last_peered":"2024-04-17T08:50:49.014507+0000","last_clean":"2024-04-17T08:50:49.014507+0000","last_became_active":"2024-04-17T08:50:48.017979+0000","last_became_peered":"2024-04-17T08:50:48.017979+0000","last_unstale":"2024-04-17T08:50:49.014507+0000","last_undegraded":"2024-04-17T08:50:49.014507+0000","last_fullsized":"2024-04-17T08:50:49.014507+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:08.190400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044790+0000","last_change":"2024-04-17T08:50:49.041753+0000","last_active":"2024-04-17T08:50:49.044790+0000","last_peered":"2024-04-17T08:50:49.044790+0000","last_clean":"2024-04-17T08:50:49.044790+0000","last_became_active":"2024-04-17T08:50:49.039959+0000","last_became_peered":"2024-04-17T08:50:49.039959+0000","last_unstale":"2024-04-17T08:50:49.044790+0000","last_undegraded":"2024-04-17T08:50:49.044790+0000","last_fullsized":"2024-04-17T08:50:49.044790+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.427318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054202+0000","last_change":"2024-04-17T08:50:48.029348+0000","last_active":"2024-04-17T08:50:49.054202+0000","last_peered":"2024-04-17T08:50:49.054202+0000","last_clean":"2024-04-17T08:50:49.054202+0000","last_became_active":"2024-04-17T08:50:48.028932+0000","last_became_peered":"2024-04-17T08:50:48.028932+0000","last_unstale":"2024-04-17T08:50:49.054202+0000","last_undegraded":"2024-04-17T08:50:49.054202+0000","last_fullsized":"2024-04-17T08:50:49.054202+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:01.151039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016020+0000","last_change":"2024-04-17T08:50:32.891454+0000","last_active":"2024-04-17T08:50:49.016020+0000","last_peered":"2024-04-17T08:50:49.016020+0000","last_clean":"2024-04-17T08:50:49.016020+0000","last_became_active":"2024-04-17T08:50:32.890756+0000","last_became_peered":"2024-04-17T08:50:32.890756+0000","last_unstale":"2024-04-17T08:50:49.016020+0000","last_undegraded":"2024-04-17T08:50:49.016020+0000","last_fullsized":"2024-04-17T08:50:49.016020+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:34.149084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023628+0000","last_change":"2024-04-17T08:50:33.897386+0000","last_active":"2024-04-17T08:50:49.023628+0000","last_peered":"2024-04-17T08:50:49.023628+0000","last_clean":"2024-04-17T08:50:49.023628+0000","last_became_active":"2024-04-17T08:50:33.897087+0000","last_became_peered":"2024-04-17T08:50:33.897087+0000","last_unstale":"2024-04-17T08:50:49.023628+0000","last_undegraded":"2024-04-17T08:50:49.023628+0000","last_fullsized":"2024-04-17T08:50:49.023628+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:32.128048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054719+0000","last_change":"2024-04-17T08:50:49.045991+0000","last_active":"2024-04-17T08:50:49.054719+0000","last_peered":"2024-04-17T08:50:49.054719+0000","last_clean":"2024-04-17T08:50:49.054719+0000","last_became_active":"2024-04-17T08:50:49.045696+0000","last_became_peered":"2024-04-17T08:50:49.045696+0000","last_unstale":"2024-04-17T08:50:49.054719+0000","last_undegraded":"2024-04-17T08:50:49.054719+0000","last_fullsized":"2024-04-17T08:50:49.054719+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:06.726698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"233'4","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054134+0000","last_change":"2024-04-17T08:50:48.027678+0000","last_active":"2024-04-17T08:50:49.054134+0000","last_peered":"2024-04-17T08:50:49.054134+0000","last_clean":"2024-04-17T08:50:49.054134+0000","last_became_active":"2024-04-17T08:50:48.027351+0000","last_became_peered":"2024-04-17T08:50:48.027351+0000","last_unstale":"2024-04-17T08:50:49.054134+0000","last_undegraded":"2024-04-17T08:50:49.054134+0000","last_fullsized":"2024-04-17T08:50:49.054134+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:35:44.730781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.015085+0000","last_change":"2024-04-17T08:50:32.900590+0000","last_active":"2024-04-17T08:50:49.015085+0000","last_peered":"2024-04-17T08:50:49.015085+0000","last_clean":"2024-04-17T08:50:49.015085+0000","last_became_active":"2024-04-17T08:50:32.900307+0000","last_became_peered":"2024-04-17T08:50:32.900307+0000","last_unstale":"2024-04-17T08:50:49.015085+0000","last_undegraded":"2024-04-17T08:50:49.015085+0000","last_fullsized":"2024-04-17T08:50:49.015085+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:45.711226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027242+0000","last_change":"2024-04-17T08:50:33.890178+0000","last_active":"2024-04-17T08:50:49.027242+0000","last_peered":"2024-04-17T08:50:49.027242+0000","last_clean":"2024-04-17T08:50:49.027242+0000","last_became_active":"2024-04-17T08:50:33.889004+0000","last_became_peered":"2024-04-17T08:50:33.889004+0000","last_unstale":"2024-04-17T08:50:49.027242+0000","last_undegraded":"2024-04-17T08:50:49.027242+0000","last_fullsized":"2024-04-17T08:50:49.027242+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:47.044851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047775+0000","last_change":"2024-04-17T08:50:49.040011+0000","last_active":"2024-04-17T08:50:49.047775+0000","last_peered":"2024-04-17T08:50:49.047775+0000","last_clean":"2024-04-17T08:50:49.047775+0000","last_became_active":"2024-04-17T08:50:49.039622+0000","last_became_peered":"2024-04-17T08:50:49.039622+0000","last_unstale":"2024-04-17T08:50:49.047775+0000","last_undegraded":"2024-04-17T08:50:49.047775+0000","last_fullsized":"2024-04-17T08:50:49.047775+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:49.754556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014140+0000","last_change":"2024-04-17T08:50:32.893112+0000","last_active":"2024-04-17T08:50:49.014140+0000","last_peered":"2024-04-17T08:50:49.014140+0000","last_clean":"2024-04-17T08:50:49.014140+0000","last_became_active":"2024-04-17T08:50:32.892793+0000","last_became_peered":"2024-04-17T08:50:32.892793+0000","last_unstale":"2024-04-17T08:50:49.014140+0000","last_undegraded":"2024-04-17T08:50:49.014140+0000","last_fullsized":"2024-04-17T08:50:49.014140+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:52.104458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021579+0000","last_change":"2024-04-17T08:50:33.896060+0000","last_active":"2024-04-17T08:50:49.021579+0000","last_peered":"2024-04-17T08:50:49.021579+0000","last_clean":"2024-04-17T08:50:49.021579+0000","last_became_active":"2024-04-17T08:50:33.895774+0000","last_became_peered":"2024-04-17T08:50:33.895774+0000","last_unstale":"2024-04-17T08:50:49.021579+0000","last_undegraded":"2024-04-17T08:50:49.021579+0000","last_fullsized":"2024-04-17T08:50:49.021579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:41.589255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057315+0000","last_change":"2024-04-17T08:50:48.025447+0000","last_active":"2024-04-17T08:50:49.057315+0000","last_peered":"2024-04-17T08:50:49.057315+0000","last_clean":"2024-04-17T08:50:49.057315+0000","last_became_active":"2024-04-17T08:50:48.024909+0000","last_became_peered":"2024-04-17T08:50:48.024909+0000","last_unstale":"2024-04-17T08:50:49.057315+0000","last_undegraded":"2024-04-17T08:50:49.057315+0000","last_fullsized":"2024-04-17T08:50:49.057315+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:42.731156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057791+0000","last_change":"2024-04-17T08:50:49.056907+0000","last_active":"2024-04-17T08:50:49.057791+0000","last_peered":"2024-04-17T08:50:49.057791+0000","last_clean":"2024-04-17T08:50:49.057791+0000","last_became_active":"2024-04-17T08:50:49.056553+0000","last_became_peered":"2024-04-17T08:50:49.056553+0000","last_unstale":"2024-04-17T08:50:49.057791+0000","last_undegraded":"2024-04-17T08:50:49.057791+0000","last_fullsized":"2024-04-17T08:50:49.057791+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:26.928269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027364+0000","last_change":"2024-04-17T08:50:32.903006+0000","last_active":"2024-04-17T08:50:49.027364+0000","last_peered":"2024-04-17T08:50:49.027364+0000","last_clean":"2024-04-17T08:50:49.027364+0000","last_became_active":"2024-04-17T08:50:32.902660+0000","last_became_peered":"2024-04-17T08:50:32.902660+0000","last_unstale":"2024-04-17T08:50:49.027364+0000","last_undegraded":"2024-04-17T08:50:49.027364+0000","last_fullsized":"2024-04-17T08:50:49.027364+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:57.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023498+0000","last_change":"2024-04-17T08:50:33.904994+0000","last_active":"2024-04-17T08:50:49.023498+0000","last_peered":"2024-04-17T08:50:49.023498+0000","last_clean":"2024-04-17T08:50:49.023498+0000","last_became_active":"2024-04-17T08:50:33.904350+0000","last_became_peered":"2024-04-17T08:50:33.904350+0000","last_unstale":"2024-04-17T08:50:49.023498+0000","last_undegraded":"2024-04-17T08:50:49.023498+0000","last_fullsized":"2024-04-17T08:50:49.023498+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:38.102858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.058298+0000","last_change":"2024-04-17T08:50:48.031037+0000","last_active":"2024-04-17T08:50:49.058298+0000","last_peered":"2024-04-17T08:50:49.058298+0000","last_clean":"2024-04-17T08:50:49.058298+0000","last_became_active":"2024-04-17T08:50:48.029862+0000","last_became_peered":"2024-04-17T08:50:48.029862+0000","last_unstale":"2024-04-17T08:50:49.058298+0000","last_undegraded":"2024-04-17T08:50:49.058298+0000","last_fullsized":"2024-04-17T08:50:49.058298+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:35.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059551+0000","last_change":"2024-04-17T08:50:49.056531+0000","last_active":"2024-04-17T08:50:49.059551+0000","last_peered":"2024-04-17T08:50:49.059551+0000","last_clean":"2024-04-17T08:50:49.059551+0000","last_became_active":"2024-04-17T08:50:49.055468+0000","last_became_peered":"2024-04-17T08:50:49.055468+0000","last_unstale":"2024-04-17T08:50:49.059551+0000","last_undegraded":"2024-04-17T08:50:49.059551+0000","last_fullsized":"2024-04-17T08:50:49.059551+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:00.161509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005988+0000","last_change":"2024-04-17T08:50:32.903736+0000","last_active":"2024-04-17T08:50:49.005988+0000","last_peered":"2024-04-17T08:50:49.005988+0000","last_clean":"2024-04-17T08:50:49.005988+0000","last_became_active":"2024-04-17T08:50:32.903485+0000","last_became_peered":"2024-04-17T08:50:32.903485+0000","last_unstale":"2024-04-17T08:50:49.005988+0000","last_undegraded":"2024-04-17T08:50:49.005988+0000","last_fullsized":"2024-04-17T08:50:49.005988+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:27.212635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012159+0000","last_change":"2024-04-17T08:50:33.882411+0000","last_active":"2024-04-17T08:50:49.012159+0000","last_peered":"2024-04-17T08:50:49.012159+0000","last_clean":"2024-04-17T08:50:49.012159+0000","last_became_active":"2024-04-17T08:50:33.881740+0000","last_became_peered":"2024-04-17T08:50:33.881740+0000","last_unstale":"2024-04-17T08:50:49.012159+0000","last_undegraded":"2024-04-17T08:50:49.012159+0000","last_fullsized":"2024-04-17T08:50:49.012159+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:06:34.035597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054474+0000","last_change":"2024-04-17T08:50:48.024472+0000","last_active":"2024-04-17T08:50:49.054474+0000","last_peered":"2024-04-17T08:50:49.054474+0000","last_clean":"2024-04-17T08:50:49.054474+0000","last_became_active":"2024-04-17T08:50:48.024188+0000","last_became_peered":"2024-04-17T08:50:48.024188+0000","last_unstale":"2024-04-17T08:50:49.054474+0000","last_undegraded":"2024-04-17T08:50:49.054474+0000","last_fullsized":"2024-04-17T08:50:49.054474+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:18.234057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046591+0000","last_change":"2024-04-17T08:50:49.044786+0000","last_active":"2024-04-17T08:50:49.046591+0000","last_peered":"2024-04-17T08:50:49.046591+0000","last_clean":"2024-04-17T08:50:49.046591+0000","last_became_active":"2024-04-17T08:50:49.044485+0000","last_became_peered":"2024-04-17T08:50:49.044485+0000","last_unstale":"2024-04-17T08:50:49.046591+0000","last_undegraded":"2024-04-17T08:50:49.046591+0000","last_fullsized":"2024-04-17T08:50:49.046591+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:49.754556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020329+0000","last_change":"2024-04-17T08:50:32.904503+0000","last_active":"2024-04-17T08:50:49.020329+0000","last_peered":"2024-04-17T08:50:49.020329+0000","last_clean":"2024-04-17T08:50:49.020329+0000","last_became_active":"2024-04-17T08:50:32.904221+0000","last_became_peered":"2024-04-17T08:50:32.904221+0000","last_unstale":"2024-04-17T08:50:49.020329+0000","last_undegraded":"2024-04-17T08:50:49.020329+0000","last_fullsized":"2024-04-17T08:50:49.020329+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:54.730453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014697+0000","last_change":"2024-04-17T08:50:33.895246+0000","last_active":"2024-04-17T08:50:49.014697+0000","last_peered":"2024-04-17T08:50:49.014697+0000","last_clean":"2024-04-17T08:50:49.014697+0000","last_became_active":"2024-04-17T08:50:33.891622+0000","last_became_peered":"2024-04-17T08:50:33.891622+0000","last_unstale":"2024-04-17T08:50:49.014697+0000","last_undegraded":"2024-04-17T08:50:49.014697+0000","last_fullsized":"2024-04-17T08:50:49.014697+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:00.894981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.022931+0000","last_change":"2024-04-17T08:50:48.022192+0000","last_active":"2024-04-17T08:50:49.022931+0000","last_peered":"2024-04-17T08:50:49.022931+0000","last_clean":"2024-04-17T08:50:49.022931+0000","last_became_active":"2024-04-17T08:50:48.020823+0000","last_became_peered":"2024-04-17T08:50:48.020823+0000","last_unstale":"2024-04-17T08:50:49.022931+0000","last_undegraded":"2024-04-17T08:50:49.022931+0000","last_fullsized":"2024-04-17T08:50:49.022931+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:30.694630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.038683+0000","last_change":"2024-04-17T08:50:49.036829+0000","last_active":"2024-04-17T08:50:49.038683+0000","last_peered":"2024-04-17T08:50:49.038683+0000","last_clean":"2024-04-17T08:50:49.038683+0000","last_became_active":"2024-04-17T08:50:49.036519+0000","last_became_peered":"2024-04-17T08:50:49.036519+0000","last_unstale":"2024-04-17T08:50:49.038683+0000","last_undegraded":"2024-04-17T08:50:49.038683+0000","last_fullsized":"2024-04-17T08:50:49.038683+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:01.572683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"229'4","reported_seq":24,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054487+0000","last_change":"2024-04-17T08:50:32.901470+0000","last_active":"2024-04-17T08:50:49.054487+0000","last_peered":"2024-04-17T08:50:49.054487+0000","last_clean":"2024-04-17T08:50:49.054487+0000","last_became_active":"2024-04-17T08:50:32.900718+0000","last_became_peered":"2024-04-17T08:50:32.900718+0000","last_unstale":"2024-04-17T08:50:49.054487+0000","last_undegraded":"2024-04-17T08:50:49.054487+0000","last_fullsized":"2024-04-17T08:50:49.054487+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:32.091268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018665+0000","last_change":"2024-04-17T08:50:33.897991+0000","last_active":"2024-04-17T08:50:49.018665+0000","last_peered":"2024-04-17T08:50:49.018665+0000","last_clean":"2024-04-17T08:50:49.018665+0000","last_became_active":"2024-04-17T08:50:33.897605+0000","last_became_peered":"2024-04-17T08:50:33.897605+0000","last_unstale":"2024-04-17T08:50:49.018665+0000","last_undegraded":"2024-04-17T08:50:49.018665+0000","last_fullsized":"2024-04-17T08:50:49.018665+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:19.182944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017046+0000","last_change":"2024-04-17T08:50:48.026134+0000","last_active":"2024-04-17T08:50:49.017046+0000","last_peered":"2024-04-17T08:50:49.017046+0000","last_clean":"2024-04-17T08:50:49.017046+0000","last_became_active":"2024-04-17T08:50:48.025857+0000","last_became_peered":"2024-04-17T08:50:48.025857+0000","last_unstale":"2024-04-17T08:50:49.017046+0000","last_undegraded":"2024-04-17T08:50:49.017046+0000","last_fullsized":"2024-04-17T08:50:49.017046+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:40:33.236834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028676+0000","last_change":"2024-04-17T08:50:49.024224+0000","last_active":"2024-04-17T08:50:49.028676+0000","last_peered":"2024-04-17T08:50:49.028676+0000","last_clean":"2024-04-17T08:50:49.028676+0000","last_became_active":"2024-04-17T08:50:49.023932+0000","last_became_peered":"2024-04-17T08:50:49.023932+0000","last_unstale":"2024-04-17T08:50:49.028676+0000","last_undegraded":"2024-04-17T08:50:49.028676+0000","last_fullsized":"2024-04-17T08:50:49.028676+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:32.580750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018304+0000","last_change":"2024-04-17T08:50:33.897001+0000","last_active":"2024-04-17T08:50:49.018304+0000","last_peered":"2024-04-17T08:50:49.018304+0000","last_clean":"2024-04-17T08:50:49.018304+0000","last_became_active":"2024-04-17T08:50:33.896545+0000","last_became_peered":"2024-04-17T08:50:33.896545+0000","last_unstale":"2024-04-17T08:50:49.018304+0000","last_undegraded":"2024-04-17T08:50:49.018304+0000","last_fullsized":"2024-04-17T08:50:49.018304+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:52.128789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045100+0000","last_change":"2024-04-17T08:50:49.042008+0000","last_active":"2024-04-17T08:50:49.045100+0000","last_peered":"2024-04-17T08:50:49.045100+0000","last_clean":"2024-04-17T08:50:49.045100+0000","last_became_active":"2024-04-17T08:50:49.041386+0000","last_became_peered":"2024-04-17T08:50:49.041386+0000","last_unstale":"2024-04-17T08:50:49.045100+0000","last_undegraded":"2024-04-17T08:50:49.045100+0000","last_fullsized":"2024-04-17T08:50:49.045100+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:42.544919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025232+0000","last_change":"2024-04-17T08:50:33.890500+0000","last_active":"2024-04-17T08:50:49.025232+0000","last_peered":"2024-04-17T08:50:49.025232+0000","last_clean":"2024-04-17T08:50:49.025232+0000","last_became_active":"2024-04-17T08:50:33.890183+0000","last_became_peered":"2024-04-17T08:50:33.890183+0000","last_unstale":"2024-04-17T08:50:49.025232+0000","last_undegraded":"2024-04-17T08:50:49.025232+0000","last_fullsized":"2024-04-17T08:50:49.025232+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:30.472528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.037845+0000","last_change":"2024-04-17T08:50:49.035117+0000","last_active":"2024-04-17T08:50:49.037845+0000","last_peered":"2024-04-17T08:50:49.037845+0000","last_clean":"2024-04-17T08:50:49.037845+0000","last_became_active":"2024-04-17T08:50:49.034818+0000","last_became_peered":"2024-04-17T08:50:49.034818+0000","last_unstale":"2024-04-17T08:50:49.037845+0000","last_undegraded":"2024-04-17T08:50:49.037845+0000","last_fullsized":"2024-04-17T08:50:49.037845+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:33:40.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016895+0000","last_change":"2024-04-17T08:50:33.895564+0000","last_active":"2024-04-17T08:50:49.016895+0000","last_peered":"2024-04-17T08:50:49.016895+0000","last_clean":"2024-04-17T08:50:49.016895+0000","last_became_active":"2024-04-17T08:50:33.893775+0000","last_became_peered":"2024-04-17T08:50:33.893775+0000","last_unstale":"2024-04-17T08:50:49.016895+0000","last_undegraded":"2024-04-17T08:50:49.016895+0000","last_fullsized":"2024-04-17T08:50:49.016895+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:42.408577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052152+0000","last_change":"2024-04-17T08:50:49.049627+0000","last_active":"2024-04-17T08:50:49.052152+0000","last_peered":"2024-04-17T08:50:49.052152+0000","last_clean":"2024-04-17T08:50:49.052152+0000","last_became_active":"2024-04-17T08:50:49.049314+0000","last_became_peered":"2024-04-17T08:50:49.049314+0000","last_unstale":"2024-04-17T08:50:49.052152+0000","last_undegraded":"2024-04-17T08:50:49.052152+0000","last_fullsized":"2024-04-17T08:50:49.052152+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:07.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048859+0000","last_change":"2024-04-17T08:50:33.899268+0000","last_active":"2024-04-17T08:50:49.048859+0000","last_peered":"2024-04-17T08:50:49.048859+0000","last_clean":"2024-04-17T08:50:49.048859+0000","last_became_active":"2024-04-17T08:50:33.898733+0000","last_became_peered":"2024-04-17T08:50:33.898733+0000","last_unstale":"2024-04-17T08:50:49.048859+0000","last_undegraded":"2024-04-17T08:50:49.048859+0000","last_fullsized":"2024-04-17T08:50:49.048859+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:24.183129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059529+0000","last_change":"2024-04-17T08:50:49.056280+0000","last_active":"2024-04-17T08:50:49.059529+0000","last_peered":"2024-04-17T08:50:49.059529+0000","last_clean":"2024-04-17T08:50:49.059529+0000","last_became_active":"2024-04-17T08:50:49.055717+0000","last_became_peered":"2024-04-17T08:50:49.055717+0000","last_unstale":"2024-04-17T08:50:49.059529+0000","last_undegraded":"2024-04-17T08:50:49.059529+0000","last_fullsized":"2024-04-17T08:50:49.059529+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:40:21.628524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004125+0000","last_change":"2024-04-17T08:50:33.897631+0000","last_active":"2024-04-17T08:50:49.004125+0000","last_peered":"2024-04-17T08:50:49.004125+0000","last_clean":"2024-04-17T08:50:49.004125+0000","last_became_active":"2024-04-17T08:50:33.897328+0000","last_became_peered":"2024-04-17T08:50:33.897328+0000","last_unstale":"2024-04-17T08:50:49.004125+0000","last_undegraded":"2024-04-17T08:50:49.004125+0000","last_fullsized":"2024-04-17T08:50:49.004125+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:10:47.708966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059847+0000","last_change":"2024-04-17T08:50:49.057574+0000","last_active":"2024-04-17T08:50:49.059847+0000","last_peered":"2024-04-17T08:50:49.059847+0000","last_clean":"2024-04-17T08:50:49.059847+0000","last_became_active":"2024-04-17T08:50:49.057111+0000","last_became_peered":"2024-04-17T08:50:49.057111+0000","last_unstale":"2024-04-17T08:50:49.059847+0000","last_undegraded":"2024-04-17T08:50:49.059847+0000","last_fullsized":"2024-04-17T08:50:49.059847+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:13.816186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053498+0000","last_change":"2024-04-17T08:50:33.897365+0000","last_active":"2024-04-17T08:50:49.053498+0000","last_peered":"2024-04-17T08:50:49.053498+0000","last_clean":"2024-04-17T08:50:49.053498+0000","last_became_active":"2024-04-17T08:50:33.897012+0000","last_became_peered":"2024-04-17T08:50:33.897012+0000","last_unstale":"2024-04-17T08:50:49.053498+0000","last_undegraded":"2024-04-17T08:50:49.053498+0000","last_fullsized":"2024-04-17T08:50:49.053498+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:35.090017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044615+0000","last_change":"2024-04-17T08:50:49.043131+0000","last_active":"2024-04-17T08:50:49.044615+0000","last_peered":"2024-04-17T08:50:49.044615+0000","last_clean":"2024-04-17T08:50:49.044615+0000","last_became_active":"2024-04-17T08:50:49.042828+0000","last_became_peered":"2024-04-17T08:50:49.042828+0000","last_unstale":"2024-04-17T08:50:49.044615+0000","last_undegraded":"2024-04-17T08:50:49.044615+0000","last_fullsized":"2024-04-17T08:50:49.044615+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:18.298255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016238+0000","last_change":"2024-04-17T08:50:33.879179+0000","last_active":"2024-04-17T08:50:49.016238+0000","last_peered":"2024-04-17T08:50:49.016238+0000","last_clean":"2024-04-17T08:50:49.016238+0000","last_became_active":"2024-04-17T08:50:33.878412+0000","last_became_peered":"2024-04-17T08:50:33.878412+0000","last_unstale":"2024-04-17T08:50:49.016238+0000","last_undegraded":"2024-04-17T08:50:49.016238+0000","last_fullsized":"2024-04-17T08:50:49.016238+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:39.773357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045034+0000","last_change":"2024-04-17T08:50:49.038692+0000","last_active":"2024-04-17T08:50:49.045034+0000","last_peered":"2024-04-17T08:50:49.045034+0000","last_clean":"2024-04-17T08:50:49.045034+0000","last_became_active":"2024-04-17T08:50:49.038427+0000","last_became_peered":"2024-04-17T08:50:49.038427+0000","last_unstale":"2024-04-17T08:50:49.045034+0000","last_undegraded":"2024-04-17T08:50:49.045034+0000","last_fullsized":"2024-04-17T08:50:49.045034+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:31.479027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018877+0000","last_change":"2024-04-17T08:50:33.893295+0000","last_active":"2024-04-17T08:50:49.018877+0000","last_peered":"2024-04-17T08:50:49.018877+0000","last_clean":"2024-04-17T08:50:49.018877+0000","last_became_active":"2024-04-17T08:50:33.892999+0000","last_became_peered":"2024-04-17T08:50:33.892999+0000","last_unstale":"2024-04-17T08:50:49.018877+0000","last_undegraded":"2024-04-17T08:50:49.018877+0000","last_fullsized":"2024-04-17T08:50:49.018877+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:46.730352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046175+0000","last_change":"2024-04-17T08:50:49.045161+0000","last_active":"2024-04-17T08:50:49.046175+0000","last_peered":"2024-04-17T08:50:49.046175+0000","last_clean":"2024-04-17T08:50:49.046175+0000","last_became_active":"2024-04-17T08:50:49.043011+0000","last_became_peered":"2024-04-17T08:50:49.043011+0000","last_unstale":"2024-04-17T08:50:49.046175+0000","last_undegraded":"2024-04-17T08:50:49.046175+0000","last_fullsized":"2024-04-17T08:50:49.046175+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:49.937382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028246+0000","last_change":"2024-04-17T08:50:33.900938+0000","last_active":"2024-04-17T08:50:49.028246+0000","last_peered":"2024-04-17T08:50:49.028246+0000","last_clean":"2024-04-17T08:50:49.028246+0000","last_became_active":"2024-04-17T08:50:33.900085+0000","last_became_peered":"2024-04-17T08:50:33.900085+0000","last_unstale":"2024-04-17T08:50:49.028246+0000","last_undegraded":"2024-04-17T08:50:49.028246+0000","last_fullsized":"2024-04-17T08:50:49.028246+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:31.107767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034247+0000","last_change":"2024-04-17T08:50:49.026386+0000","last_active":"2024-04-17T08:50:49.034247+0000","last_peered":"2024-04-17T08:50:49.034247+0000","last_clean":"2024-04-17T08:50:49.034247+0000","last_became_active":"2024-04-17T08:50:49.025867+0000","last_became_peered":"2024-04-17T08:50:49.025867+0000","last_unstale":"2024-04-17T08:50:49.034247+0000","last_undegraded":"2024-04-17T08:50:49.034247+0000","last_fullsized":"2024-04-17T08:50:49.034247+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:28.747322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.010867+0000","last_change":"2024-04-17T08:50:33.890185+0000","last_active":"2024-04-17T08:50:49.010867+0000","last_peered":"2024-04-17T08:50:49.010867+0000","last_clean":"2024-04-17T08:50:49.010867+0000","last_became_active":"2024-04-17T08:50:33.889133+0000","last_became_peered":"2024-04-17T08:50:33.889133+0000","last_unstale":"2024-04-17T08:50:49.010867+0000","last_undegraded":"2024-04-17T08:50:49.010867+0000","last_fullsized":"2024-04-17T08:50:49.010867+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:36.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.033596+0000","last_change":"2024-04-17T08:50:49.029148+0000","last_active":"2024-04-17T08:50:49.033596+0000","last_peered":"2024-04-17T08:50:49.033596+0000","last_clean":"2024-04-17T08:50:49.033596+0000","last_became_active":"2024-04-17T08:50:49.028844+0000","last_became_peered":"2024-04-17T08:50:49.028844+0000","last_unstale":"2024-04-17T08:50:49.033596+0000","last_undegraded":"2024-04-17T08:50:49.033596+0000","last_fullsized":"2024-04-17T08:50:49.033596+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:56.712515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011837+0000","last_change":"2024-04-17T08:50:33.905971+0000","last_active":"2024-04-17T08:50:49.011837+0000","last_peered":"2024-04-17T08:50:49.011837+0000","last_clean":"2024-04-17T08:50:49.011837+0000","last_became_active":"2024-04-17T08:50:33.905591+0000","last_became_peered":"2024-04-17T08:50:33.905591+0000","last_unstale":"2024-04-17T08:50:49.011837+0000","last_undegraded":"2024-04-17T08:50:49.011837+0000","last_fullsized":"2024-04-17T08:50:49.011837+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:32.834294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047893+0000","last_change":"2024-04-17T08:50:49.042802+0000","last_active":"2024-04-17T08:50:49.047893+0000","last_peered":"2024-04-17T08:50:49.047893+0000","last_clean":"2024-04-17T08:50:49.047893+0000","last_became_active":"2024-04-17T08:50:49.041010+0000","last_became_peered":"2024-04-17T08:50:49.041010+0000","last_unstale":"2024-04-17T08:50:49.047893+0000","last_undegraded":"2024-04-17T08:50:49.047893+0000","last_fullsized":"2024-04-17T08:50:49.047893+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:46.224935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021294+0000","last_change":"2024-04-17T08:50:33.904610+0000","last_active":"2024-04-17T08:50:49.021294+0000","last_peered":"2024-04-17T08:50:49.021294+0000","last_clean":"2024-04-17T08:50:49.021294+0000","last_became_active":"2024-04-17T08:50:33.904287+0000","last_became_peered":"2024-04-17T08:50:33.904287+0000","last_unstale":"2024-04-17T08:50:49.021294+0000","last_undegraded":"2024-04-17T08:50:49.021294+0000","last_fullsized":"2024-04-17T08:50:49.021294+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:28.658522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.030321+0000","last_change":"2024-04-17T08:50:49.027296+0000","last_active":"2024-04-17T08:50:49.030321+0000","last_peered":"2024-04-17T08:50:49.030321+0000","last_clean":"2024-04-17T08:50:49.030321+0000","last_became_active":"2024-04-17T08:50:49.025787+0000","last_became_peered":"2024-04-17T08:50:49.025787+0000","last_unstale":"2024-04-17T08:50:49.030321+0000","last_undegraded":"2024-04-17T08:50:49.030321+0000","last_fullsized":"2024-04-17T08:50:49.030321+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:51.381129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024543+0000","last_change":"2024-04-17T08:50:33.907557+0000","last_active":"2024-04-17T08:50:49.024543+0000","last_peered":"2024-04-17T08:50:49.024543+0000","last_clean":"2024-04-17T08:50:49.024543+0000","last_became_active":"2024-04-17T08:50:33.907258+0000","last_became_peered":"2024-04-17T08:50:33.907258+0000","last_unstale":"2024-04-17T08:50:49.024543+0000","last_undegraded":"2024-04-17T08:50:49.024543+0000","last_fullsized":"2024-04-17T08:50:49.024543+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:40:19.130647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046245+0000","last_change":"2024-04-17T08:50:49.042577+0000","last_active":"2024-04-17T08:50:49.046245+0000","last_peered":"2024-04-17T08:50:49.046245+0000","last_clean":"2024-04-17T08:50:49.046245+0000","last_became_active":"2024-04-17T08:50:49.039823+0000","last_became_peered":"2024-04-17T08:50:49.039823+0000","last_unstale":"2024-04-17T08:50:49.046245+0000","last_undegraded":"2024-04-17T08:50:49.046245+0000","last_fullsized":"2024-04-17T08:50:49.046245+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:03.547181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026027+0000","last_change":"2024-04-17T08:50:33.907077+0000","last_active":"2024-04-17T08:50:49.026027+0000","last_peered":"2024-04-17T08:50:49.026027+0000","last_clean":"2024-04-17T08:50:49.026027+0000","last_became_active":"2024-04-17T08:50:33.906771+0000","last_became_peered":"2024-04-17T08:50:33.906771+0000","last_unstale":"2024-04-17T08:50:49.026027+0000","last_undegraded":"2024-04-17T08:50:49.026027+0000","last_fullsized":"2024-04-17T08:50:49.026027+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:34.659543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034592+0000","last_change":"2024-04-17T08:50:49.031610+0000","last_active":"2024-04-17T08:50:49.034592+0000","last_peered":"2024-04-17T08:50:49.034592+0000","last_clean":"2024-04-17T08:50:49.034592+0000","last_became_active":"2024-04-17T08:50:49.031310+0000","last_became_peered":"2024-04-17T08:50:49.031310+0000","last_unstale":"2024-04-17T08:50:49.034592+0000","last_undegraded":"2024-04-17T08:50:49.034592+0000","last_fullsized":"2024-04-17T08:50:49.034592+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:27.833812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026990+0000","last_change":"2024-04-17T08:50:33.905082+0000","last_active":"2024-04-17T08:50:49.026990+0000","last_peered":"2024-04-17T08:50:49.026990+0000","last_clean":"2024-04-17T08:50:49.026990+0000","last_became_active":"2024-04-17T08:50:33.904723+0000","last_became_peered":"2024-04-17T08:50:33.904723+0000","last_unstale":"2024-04-17T08:50:49.026990+0000","last_undegraded":"2024-04-17T08:50:49.026990+0000","last_fullsized":"2024-04-17T08:50:49.026990+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:10.492335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.039376+0000","last_change":"2024-04-17T08:50:49.034923+0000","last_active":"2024-04-17T08:50:49.039376+0000","last_peered":"2024-04-17T08:50:49.039376+0000","last_clean":"2024-04-17T08:50:49.039376+0000","last_became_active":"2024-04-17T08:50:49.034509+0000","last_became_peered":"2024-04-17T08:50:49.034509+0000","last_unstale":"2024-04-17T08:50:49.039376+0000","last_undegraded":"2024-04-17T08:50:49.039376+0000","last_fullsized":"2024-04-17T08:50:49.039376+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:17.060962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017084+0000","last_change":"2024-04-17T08:50:33.888622+0000","last_active":"2024-04-17T08:50:49.017084+0000","last_peered":"2024-04-17T08:50:49.017084+0000","last_clean":"2024-04-17T08:50:49.017084+0000","last_became_active":"2024-04-17T08:50:33.888298+0000","last_became_peered":"2024-04-17T08:50:33.888298+0000","last_unstale":"2024-04-17T08:50:49.017084+0000","last_undegraded":"2024-04-17T08:50:49.017084+0000","last_fullsized":"2024-04-17T08:50:49.017084+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:48.390987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.036139+0000","last_change":"2024-04-17T08:50:49.030470+0000","last_active":"2024-04-17T08:50:49.036139+0000","last_peered":"2024-04-17T08:50:49.036139+0000","last_clean":"2024-04-17T08:50:49.036139+0000","last_became_active":"2024-04-17T08:50:49.030193+0000","last_became_peered":"2024-04-17T08:50:49.030193+0000","last_unstale":"2024-04-17T08:50:49.036139+0000","last_undegraded":"2024-04-17T08:50:49.036139+0000","last_fullsized":"2024-04-17T08:50:49.036139+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:39.173146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.022998+0000","last_change":"2024-04-17T08:50:33.895366+0000","last_active":"2024-04-17T08:50:49.022998+0000","last_peered":"2024-04-17T08:50:49.022998+0000","last_clean":"2024-04-17T08:50:49.022998+0000","last_became_active":"2024-04-17T08:50:33.892102+0000","last_became_peered":"2024-04-17T08:50:33.892102+0000","last_unstale":"2024-04-17T08:50:49.022998+0000","last_undegraded":"2024-04-17T08:50:49.022998+0000","last_fullsized":"2024-04-17T08:50:49.022998+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:32.764250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054586+0000","last_change":"2024-04-17T08:50:49.047250+0000","last_active":"2024-04-17T08:50:49.054586+0000","last_peered":"2024-04-17T08:50:49.054586+0000","last_clean":"2024-04-17T08:50:49.054586+0000","last_became_active":"2024-04-17T08:50:49.046458+0000","last_became_peered":"2024-04-17T08:50:49.046458+0000","last_unstale":"2024-04-17T08:50:49.054586+0000","last_undegraded":"2024-04-17T08:50:49.054586+0000","last_fullsized":"2024-04-17T08:50:49.054586+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:06.199924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019502+0000","last_change":"2024-04-17T08:50:33.898186+0000","last_active":"2024-04-17T08:50:49.019502+0000","last_peered":"2024-04-17T08:50:49.019502+0000","last_clean":"2024-04-17T08:50:49.019502+0000","last_became_active":"2024-04-17T08:50:33.897746+0000","last_became_peered":"2024-04-17T08:50:33.897746+0000","last_unstale":"2024-04-17T08:50:49.019502+0000","last_undegraded":"2024-04-17T08:50:49.019502+0000","last_fullsized":"2024-04-17T08:50:49.019502+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:01:11.440017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040121+0000","last_change":"2024-04-17T08:50:49.037933+0000","last_active":"2024-04-17T08:50:49.040121+0000","last_peered":"2024-04-17T08:50:49.040121+0000","last_clean":"2024-04-17T08:50:49.040121+0000","last_became_active":"2024-04-17T08:50:49.035919+0000","last_became_peered":"2024-04-17T08:50:49.035919+0000","last_unstale":"2024-04-17T08:50:49.040121+0000","last_undegraded":"2024-04-17T08:50:49.040121+0000","last_fullsized":"2024-04-17T08:50:49.040121+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:43.347365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018729+0000","last_change":"2024-04-17T08:50:33.887226+0000","last_active":"2024-04-17T08:50:49.018729+0000","last_peered":"2024-04-17T08:50:49.018729+0000","last_clean":"2024-04-17T08:50:49.018729+0000","last_became_active":"2024-04-17T08:50:33.886912+0000","last_became_peered":"2024-04-17T08:50:33.886912+0000","last_unstale":"2024-04-17T08:50:49.018729+0000","last_undegraded":"2024-04-17T08:50:49.018729+0000","last_fullsized":"2024-04-17T08:50:49.018729+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:59.708199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.050946+0000","last_change":"2024-04-17T08:50:49.049142+0000","last_active":"2024-04-17T08:50:49.050946+0000","last_peered":"2024-04-17T08:50:49.050946+0000","last_clean":"2024-04-17T08:50:49.050946+0000","last_became_active":"2024-04-17T08:50:49.048696+0000","last_became_peered":"2024-04-17T08:50:49.048696+0000","last_unstale":"2024-04-17T08:50:49.050946+0000","last_undegraded":"2024-04-17T08:50:49.050946+0000","last_fullsized":"2024-04-17T08:50:49.050946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:30.138141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048893+0000","last_change":"2024-04-17T08:50:33.894398+0000","last_active":"2024-04-17T08:50:49.048893+0000","last_peered":"2024-04-17T08:50:49.048893+0000","last_clean":"2024-04-17T08:50:49.048893+0000","last_became_active":"2024-04-17T08:50:33.894105+0000","last_became_peered":"2024-04-17T08:50:33.894105+0000","last_unstale":"2024-04-17T08:50:49.048893+0000","last_undegraded":"2024-04-17T08:50:49.048893+0000","last_fullsized":"2024-04-17T08:50:49.048893+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:00.398625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034049+0000","last_change":"2024-04-17T08:50:49.025092+0000","last_active":"2024-04-17T08:50:49.034049+0000","last_peered":"2024-04-17T08:50:49.034049+0000","last_clean":"2024-04-17T08:50:49.034049+0000","last_became_active":"2024-04-17T08:50:49.023955+0000","last_became_peered":"2024-04-17T08:50:49.023955+0000","last_unstale":"2024-04-17T08:50:49.034049+0000","last_undegraded":"2024-04-17T08:50:49.034049+0000","last_fullsized":"2024-04-17T08:50:49.034049+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.074044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053008+0000","last_change":"2024-04-17T08:50:33.897953+0000","last_active":"2024-04-17T08:50:49.053008+0000","last_peered":"2024-04-17T08:50:49.053008+0000","last_clean":"2024-04-17T08:50:49.053008+0000","last_became_active":"2024-04-17T08:50:33.897591+0000","last_became_peered":"2024-04-17T08:50:33.897591+0000","last_unstale":"2024-04-17T08:50:49.053008+0000","last_undegraded":"2024-04-17T08:50:49.053008+0000","last_fullsized":"2024-04-17T08:50:49.053008+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:57.978668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040118+0000","last_change":"2024-04-17T08:50:49.031090+0000","last_active":"2024-04-17T08:50:49.040118+0000","last_peered":"2024-04-17T08:50:49.040118+0000","last_clean":"2024-04-17T08:50:49.040118+0000","last_became_active":"2024-04-17T08:50:49.028551+0000","last_became_peered":"2024-04-17T08:50:49.028551+0000","last_unstale":"2024-04-17T08:50:49.040118+0000","last_undegraded":"2024-04-17T08:50:49.040118+0000","last_fullsized":"2024-04-17T08:50:49.040118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:28.747322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005717+0000","last_change":"2024-04-17T08:50:33.897057+0000","last_active":"2024-04-17T08:50:49.005717+0000","last_peered":"2024-04-17T08:50:49.005717+0000","last_clean":"2024-04-17T08:50:49.005717+0000","last_became_active":"2024-04-17T08:50:33.894722+0000","last_became_peered":"2024-04-17T08:50:33.894722+0000","last_unstale":"2024-04-17T08:50:49.005717+0000","last_undegraded":"2024-04-17T08:50:49.005717+0000","last_fullsized":"2024-04-17T08:50:49.005717+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:09.971845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040250+0000","last_change":"2024-04-17T08:50:49.036087+0000","last_active":"2024-04-17T08:50:49.040250+0000","last_peered":"2024-04-17T08:50:49.040250+0000","last_clean":"2024-04-17T08:50:49.040250+0000","last_became_active":"2024-04-17T08:50:49.035555+0000","last_became_peered":"2024-04-17T08:50:49.035555+0000","last_unstale":"2024-04-17T08:50:49.040250+0000","last_undegraded":"2024-04-17T08:50:49.040250+0000","last_fullsized":"2024-04-17T08:50:49.040250+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:11.499171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026883+0000","last_change":"2024-04-17T08:50:33.895456+0000","last_active":"2024-04-17T08:50:49.026883+0000","last_peered":"2024-04-17T08:50:49.026883+0000","last_clean":"2024-04-17T08:50:49.026883+0000","last_became_active":"2024-04-17T08:50:33.895142+0000","last_became_peered":"2024-04-17T08:50:33.895142+0000","last_unstale":"2024-04-17T08:50:49.026883+0000","last_undegraded":"2024-04-17T08:50:49.026883+0000","last_fullsized":"2024-04-17T08:50:49.026883+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:58.168170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.051397+0000","last_change":"2024-04-17T08:50:49.048928+0000","last_active":"2024-04-17T08:50:49.051397+0000","last_peered":"2024-04-17T08:50:49.051397+0000","last_clean":"2024-04-17T08:50:49.051397+0000","last_became_active":"2024-04-17T08:50:49.045972+0000","last_became_peered":"2024-04-17T08:50:49.045972+0000","last_unstale":"2024-04-17T08:50:49.051397+0000","last_undegraded":"2024-04-17T08:50:49.051397+0000","last_fullsized":"2024-04-17T08:50:49.051397+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:31.057634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019312+0000","last_change":"2024-04-17T08:50:33.894702+0000","last_active":"2024-04-17T08:50:49.019312+0000","last_peered":"2024-04-17T08:50:49.019312+0000","last_clean":"2024-04-17T08:50:49.019312+0000","last_became_active":"2024-04-17T08:50:33.894398+0000","last_became_peered":"2024-04-17T08:50:33.894398+0000","last_unstale":"2024-04-17T08:50:49.019312+0000","last_undegraded":"2024-04-17T08:50:49.019312+0000","last_fullsized":"2024-04-17T08:50:49.019312+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:32.834294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047619+0000","last_change":"2024-04-17T08:50:49.045976+0000","last_active":"2024-04-17T08:50:49.047619+0000","last_peered":"2024-04-17T08:50:49.047619+0000","last_clean":"2024-04-17T08:50:49.047619+0000","last_became_active":"2024-04-17T08:50:49.044919+0000","last_became_peered":"2024-04-17T08:50:49.044919+0000","last_unstale":"2024-04-17T08:50:49.047619+0000","last_undegraded":"2024-04-17T08:50:49.047619+0000","last_fullsized":"2024-04-17T08:50:49.047619+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:01.572833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027129+0000","last_change":"2024-04-17T08:50:33.895419+0000","last_active":"2024-04-17T08:50:49.027129+0000","last_peered":"2024-04-17T08:50:49.027129+0000","last_clean":"2024-04-17T08:50:49.027129+0000","last_became_active":"2024-04-17T08:50:33.894089+0000","last_became_peered":"2024-04-17T08:50:33.894089+0000","last_unstale":"2024-04-17T08:50:49.027129+0000","last_undegraded":"2024-04-17T08:50:49.027129+0000","last_fullsized":"2024-04-17T08:50:49.027129+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:00:46.043729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.038318+0000","last_change":"2024-04-17T08:50:49.037732+0000","last_active":"2024-04-17T08:50:49.038318+0000","last_peered":"2024-04-17T08:50:49.038318+0000","last_clean":"2024-04-17T08:50:49.038318+0000","last_became_active":"2024-04-17T08:50:49.037397+0000","last_became_peered":"2024-04-17T08:50:49.037397+0000","last_unstale":"2024-04-17T08:50:49.038318+0000","last_undegraded":"2024-04-17T08:50:49.038318+0000","last_fullsized":"2024-04-17T08:50:49.038318+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:18.884898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018150+0000","last_change":"2024-04-17T08:50:33.880798+0000","last_active":"2024-04-17T08:50:49.018150+0000","last_peered":"2024-04-17T08:50:49.018150+0000","last_clean":"2024-04-17T08:50:49.018150+0000","last_became_active":"2024-04-17T08:50:33.879829+0000","last_became_peered":"2024-04-17T08:50:33.879829+0000","last_unstale":"2024-04-17T08:50:49.018150+0000","last_undegraded":"2024-04-17T08:50:49.018150+0000","last_fullsized":"2024-04-17T08:50:49.018150+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:46.264842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045908+0000","last_change":"2024-04-17T08:50:49.042026+0000","last_active":"2024-04-17T08:50:49.045908+0000","last_peered":"2024-04-17T08:50:49.045908+0000","last_clean":"2024-04-17T08:50:49.045908+0000","last_became_active":"2024-04-17T08:50:49.041698+0000","last_became_peered":"2024-04-17T08:50:49.041698+0000","last_unstale":"2024-04-17T08:50:49.045908+0000","last_undegraded":"2024-04-17T08:50:49.045908+0000","last_fullsized":"2024-04-17T08:50:49.045908+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:01.822353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013614+0000","last_change":"2024-04-17T08:50:33.900220+0000","last_active":"2024-04-17T08:50:49.013614+0000","last_peered":"2024-04-17T08:50:49.013614+0000","last_clean":"2024-04-17T08:50:49.013614+0000","last_became_active":"2024-04-17T08:50:33.898746+0000","last_became_peered":"2024-04-17T08:50:33.898746+0000","last_unstale":"2024-04-17T08:50:49.013614+0000","last_undegraded":"2024-04-17T08:50:49.013614+0000","last_fullsized":"2024-04-17T08:50:49.013614+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:37.623670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053243+0000","last_change":"2024-04-17T08:50:49.042525+0000","last_active":"2024-04-17T08:50:49.053243+0000","last_peered":"2024-04-17T08:50:49.053243+0000","last_clean":"2024-04-17T08:50:49.053243+0000","last_became_active":"2024-04-17T08:50:49.042215+0000","last_became_peered":"2024-04-17T08:50:49.042215+0000","last_unstale":"2024-04-17T08:50:49.053243+0000","last_undegraded":"2024-04-17T08:50:49.053243+0000","last_fullsized":"2024-04-17T08:50:49.053243+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:06.467500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024343+0000","last_change":"2024-04-17T08:50:33.908217+0000","last_active":"2024-04-17T08:50:49.024343+0000","last_peered":"2024-04-17T08:50:49.024343+0000","last_clean":"2024-04-17T08:50:49.024343+0000","last_became_active":"2024-04-17T08:50:33.907185+0000","last_became_peered":"2024-04-17T08:50:33.907185+0000","last_unstale":"2024-04-17T08:50:49.024343+0000","last_undegraded":"2024-04-17T08:50:49.024343+0000","last_fullsized":"2024-04-17T08:50:49.024343+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:28.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.039577+0000","last_change":"2024-04-17T08:50:49.037576+0000","last_active":"2024-04-17T08:50:49.039577+0000","last_peered":"2024-04-17T08:50:49.039577+0000","last_clean":"2024-04-17T08:50:49.039577+0000","last_became_active":"2024-04-17T08:50:49.037277+0000","last_became_peered":"2024-04-17T08:50:49.037277+0000","last_unstale":"2024-04-17T08:50:49.039577+0000","last_undegraded":"2024-04-17T08:50:49.039577+0000","last_fullsized":"2024-04-17T08:50:49.039577+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:30:52.677467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024562+0000","last_change":"2024-04-17T08:50:33.896077+0000","last_active":"2024-04-17T08:50:49.024562+0000","last_peered":"2024-04-17T08:50:49.024562+0000","last_clean":"2024-04-17T08:50:49.024562+0000","last_became_active":"2024-04-17T08:50:33.895408+0000","last_became_peered":"2024-04-17T08:50:33.895408+0000","last_unstale":"2024-04-17T08:50:49.024562+0000","last_undegraded":"2024-04-17T08:50:49.024562+0000","last_fullsized":"2024-04-17T08:50:49.024562+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:16.588443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.042763+0000","last_change":"2024-04-17T08:50:49.035395+0000","last_active":"2024-04-17T08:50:49.042763+0000","last_peered":"2024-04-17T08:50:49.042763+0000","last_clean":"2024-04-17T08:50:49.042763+0000","last_became_active":"2024-04-17T08:50:49.033291+0000","last_became_peered":"2024-04-17T08:50:49.033291+0000","last_unstale":"2024-04-17T08:50:49.042763+0000","last_undegraded":"2024-04-17T08:50:49.042763+0000","last_fullsized":"2024-04-17T08:50:49.042763+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:06:00.266799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005413+0000","last_change":"2024-04-17T08:50:33.900448+0000","last_active":"2024-04-17T08:50:49.005413+0000","last_peered":"2024-04-17T08:50:49.005413+0000","last_clean":"2024-04-17T08:50:49.005413+0000","last_became_active":"2024-04-17T08:50:33.900142+0000","last_became_peered":"2024-04-17T08:50:33.900142+0000","last_unstale":"2024-04-17T08:50:49.005413+0000","last_undegraded":"2024-04-17T08:50:49.005413+0000","last_fullsized":"2024-04-17T08:50:49.005413+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:37:31.446411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028607+0000","last_change":"2024-04-17T08:50:49.023214+0000","last_active":"2024-04-17T08:50:49.028607+0000","last_peered":"2024-04-17T08:50:49.028607+0000","last_clean":"2024-04-17T08:50:49.028607+0000","last_became_active":"2024-04-17T08:50:49.022785+0000","last_became_peered":"2024-04-17T08:50:49.022785+0000","last_unstale":"2024-04-17T08:50:49.028607+0000","last_undegraded":"2024-04-17T08:50:49.028607+0000","last_fullsized":"2024-04-17T08:50:49.028607+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:35.493434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018542+0000","last_change":"2024-04-17T08:50:33.883908+0000","last_active":"2024-04-17T08:50:49.018542+0000","last_peered":"2024-04-17T08:50:49.018542+0000","last_clean":"2024-04-17T08:50:49.018542+0000","last_became_active":"2024-04-17T08:50:33.883336+0000","last_became_peered":"2024-04-17T08:50:33.883336+0000","last_unstale":"2024-04-17T08:50:49.018542+0000","last_undegraded":"2024-04-17T08:50:49.018542+0000","last_fullsized":"2024-04-17T08:50:49.018542+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:21.688586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052894+0000","last_change":"2024-04-17T08:50:49.040978+0000","last_active":"2024-04-17T08:50:49.052894+0000","last_peered":"2024-04-17T08:50:49.052894+0000","last_clean":"2024-04-17T08:50:49.052894+0000","last_became_active":"2024-04-17T08:50:49.040638+0000","last_became_peered":"2024-04-17T08:50:49.040638+0000","last_unstale":"2024-04-17T08:50:49.052894+0000","last_undegraded":"2024-04-17T08:50:49.052894+0000","last_fullsized":"2024-04-17T08:50:49.052894+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:30.555717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016884+0000","last_change":"2024-04-17T08:50:33.892568+0000","last_active":"2024-04-17T08:50:49.016884+0000","last_peered":"2024-04-17T08:50:49.016884+0000","last_clean":"2024-04-17T08:50:49.016884+0000","last_became_active":"2024-04-17T08:50:33.892228+0000","last_became_peered":"2024-04-17T08:50:33.892228+0000","last_unstale":"2024-04-17T08:50:49.016884+0000","last_undegraded":"2024-04-17T08:50:49.016884+0000","last_fullsized":"2024-04-17T08:50:49.016884+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:51.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543296,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52660,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319180,"statfs":{"total":96636764160,"available":96582840320,"internally_reserved":0,"allocated":3510272,"data_stored":2014464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543296,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42344,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329496,"statfs":{"total":96636764160,"available":96593403904,"internally_reserved":0,"allocated":2908160,"data_stored":1421821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543295,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29860,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341980,"statfs":{"total":96636764160,"available":96606187520,"internally_reserved":0,"allocated":2908160,"data_stored":1421821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543295,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64568,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307272,"statfs":{"total":96636764160,"available":96570646528,"internally_reserved":0,"allocated":2924544,"data_stored":1424200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543297,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":2945024,"data_stored":1430776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543296,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52104,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319736,"statfs":{"total":96636764160,"available":96583409664,"internally_reserved":0,"allocated":2936832,"data_stored":1430580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543297,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60396,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311444,"statfs":{"total":96636764160,"available":96574918656,"internally_reserved":0,"allocated":2912256,"data_stored":1421295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543295,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":3522560,"data_stored":2020042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:50:53.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:50:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.823+0000 7f904d400640 1 Processor -- start 2024-04-17T08:50:53.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.823+0000 7f904d400640 1 -- start start 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.825+0000 7f904d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f904806b8c0 0x7f90480cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904d400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9048160280 0x7f904806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9048141ed0 0x7f9048142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f904815ce30 con 0x7f9048141ed0 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904d400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f90480cd4a0 con 0x7f9048160280 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f90481607f0 con 0x7f904806b8c0 2024-04-17T08:50:53.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9048141ed0 0x7f9048142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9048141ed0 0x7f9048142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48338/0 (socket says 172.21.15.67:48338) 2024-04-17T08:50:53.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f904ca00640 1 -- 172.21.15.67:0/960892349 learned_addr learned my addr 172.21.15.67:0/960892349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:53.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f9047e00640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f904806b8c0 0x7f90480cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.826+0000 7f903fe00640 1 --2- 172.21.15.67:0/960892349 >> v2:172.21.15.187:3300/0 conn(0x7f9048160280 0x7f904806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.827+0000 7f904ca00640 1 -- 172.21.15.67:0/960892349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f904806b8c0 msgr2=0x7f90480cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.827+0000 7f904ca00640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f904806b8c0 0x7f90480cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.827+0000 7f904ca00640 1 -- 172.21.15.67:0/960892349 >> v2:172.21.15.187:3300/0 conn(0x7f9048160280 msgr2=0x7f904806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.827+0000 7f904ca00640 1 --2- 172.21.15.67:0/960892349 >> v2:172.21.15.187:3300/0 conn(0x7f9048160280 0x7f904806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.827+0000 7f904ca00640 1 -- 172.21.15.67:0/960892349 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9048160ad0 con 0x7f9048141ed0 2024-04-17T08:50:53.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.828+0000 7f904ca00640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9048141ed0 0x7f9048142360 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f90480701c0 tx=0x7f904002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff96d1f00f11c78 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f9047400640 1 -- 172.21.15.67:0/960892349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904002ca70 con 0x7f9048141ed0 2024-04-17T08:50:53.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f9047400640 1 -- 172.21.15.67:0/960892349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904002cc10 con 0x7f9048141ed0 2024-04-17T08:50:53.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f9047400640 1 -- 172.21.15.67:0/960892349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400343f0 con 0x7f9048141ed0 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 -- 172.21.15.67:0/960892349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9048141ed0 msgr2=0x7f9048142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9048141ed0 0x7f9048142360 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f90480701c0 tx=0x7f904002bed0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 -- 172.21.15.67:0/960892349 shutdown_connections 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9048141ed0 0x7f9048142360 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 --2- 172.21.15.67:0/960892349 >> v2:172.21.15.187:3300/0 conn(0x7f9048160280 0x7f904806bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 --2- 172.21.15.67:0/960892349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f904806b8c0 0x7f90480cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 -- 172.21.15.67:0/960892349 >> 172.21.15.67:0/960892349 conn(0x7f9048067d10 msgr2=0x7f90480b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 -- 172.21.15.67:0/960892349 shutdown_connections 2024-04-17T08:50:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.829+0000 7f904d400640 1 -- 172.21.15.67:0/960892349 wait complete. 2024-04-17T08:50:53.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.831+0000 7f904d400640 1 Processor -- start 2024-04-17T08:50:53.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.831+0000 7f904d400640 1 -- start start 2024-04-17T08:50:53.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9048141ed0 0x7f90481439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9048160280 0x7f90481f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f904806c1e0 con 0x7f904806b8c0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f90480cddf0 con 0x7f9048160280 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f904d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9048161090 con 0x7f9048141ed0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.832+0000 7f9047e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48348/0 (socket says 172.21.15.67:48348) 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 -- 172.21.15.67:0/3797778974 learned_addr learned my addr 172.21.15.67:0/3797778974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f903fe00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9048141ed0 0x7f90481439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f904ca00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9048160280 0x7f90481f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 -- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9048141ed0 msgr2=0x7f90481439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9048141ed0 0x7f90481439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 -- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9048160280 msgr2=0x7f90481f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9048160280 0x7f90481f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 -- 172.21.15.67:0/3797778974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904815e7f0 con 0x7f904806b8c0 2024-04-17T08:50:53.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.833+0000 7f9047e00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f903800d830 tx=0x7f903800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.834+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038018030 con 0x7f904806b8c0 2024-04-17T08:50:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.834+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903800be10 con 0x7f904806b8c0 2024-04-17T08:50:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.834+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90380098b0 con 0x7f904806b8c0 2024-04-17T08:50:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.834+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904815ead0 con 0x7f904806b8c0 2024-04-17T08:50:53.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.834+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904815f090 con 0x7f904806b8c0 2024-04-17T08:50:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.835+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048068a10 con 0x7f904806b8c0 2024-04-17T08:50:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.836+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f903801b070 con 0x7f904806b8c0 2024-04-17T08:50:53.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.837+0000 7f9044c00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 0x7f902406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:53.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.837+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f903808c0c0 con 0x7f904806b8c0 2024-04-17T08:50:53.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.837+0000 7f903fe00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 0x7f902406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.838+0000 7f903fe00640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 0x7f902406ef20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90480d13a0 tx=0x7f9030008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:53.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:53.841+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f903801be40 con 0x7f904806b8c0 2024-04-17T08:50:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.043+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9048146220 con 0x7f902406ca60 2024-04-17T08:50:54.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.079+0000 7f9044c00640 1 -- 172.21.15.67:0/3797778974 <== mgr.7679 v2:172.21.15.187:6832/150239855 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491414 (secure 0 0 0) 0x7f9048146220 con 0x7f902406ca60 2024-04-17T08:50:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 msgr2=0x7f902406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 0x7f902406ef20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90480d13a0 tx=0x7f9030008040 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 msgr2=0x7f904815e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f903800d830 tx=0x7f903800dcf0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 shutdown_connections 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f902406ca60 0x7f902406ef20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9048160280 0x7f90481f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9048141ed0 0x7f90481439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 --2- 172.21.15.67:0/3797778974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904806b8c0 0x7f904815e160 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 >> 172.21.15.67:0/3797778974 conn(0x7f9048067d10 msgr2=0x7f90480d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 shutdown_connections 2024-04-17T08:50:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.090+0000 7f904d400640 1 -- 172.21.15.67:0/3797778974 wait complete. 2024-04-17T08:50:54.114 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-17T08:50:52.426871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":678,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":240,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":410864,"kb_used_data":23992,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754563856,"statfs":{"total":773094113280,"available":772673388544,"internally_reserved":0,"allocated":24567808,"data_stored":12584999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":396021296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.451002"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044759+0000","last_change":"2024-04-17T08:50:49.041591+0000","last_active":"2024-04-17T08:50:49.044759+0000","last_peered":"2024-04-17T08:50:49.044759+0000","last_clean":"2024-04-17T08:50:49.044759+0000","last_became_active":"2024-04-17T08:50:49.041258+0000","last_became_peered":"2024-04-17T08:50:49.041258+0000","last_unstale":"2024-04-17T08:50:49.044759+0000","last_undegraded":"2024-04-17T08:50:49.044759+0000","last_fullsized":"2024-04-17T08:50:49.044759+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:56.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"217'354","reported_seq":1454,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028409+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:50:49.028409+0000","last_peered":"2024-04-17T08:50:49.028409+0000","last_clean":"2024-04-17T08:50:49.028409+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:50:49.028409+0000","last_undegraded":"2024-04-17T08:50:49.028409+0000","last_fullsized":"2024-04-17T08:50:49.028409+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.051724+0000","last_change":"2024-04-17T08:50:32.893722+0000","last_active":"2024-04-17T08:50:49.051724+0000","last_peered":"2024-04-17T08:50:49.051724+0000","last_clean":"2024-04-17T08:50:49.051724+0000","last_became_active":"2024-04-17T08:50:32.893404+0000","last_became_peered":"2024-04-17T08:50:32.893404+0000","last_unstale":"2024-04-17T08:50:49.051724+0000","last_undegraded":"2024-04-17T08:50:49.051724+0000","last_fullsized":"2024-04-17T08:50:49.051724+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:34.249904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019210+0000","last_change":"2024-04-17T08:50:33.896994+0000","last_active":"2024-04-17T08:50:49.019210+0000","last_peered":"2024-04-17T08:50:49.019210+0000","last_clean":"2024-04-17T08:50:49.019210+0000","last_became_active":"2024-04-17T08:50:33.896640+0000","last_became_peered":"2024-04-17T08:50:33.896640+0000","last_unstale":"2024-04-17T08:50:49.019210+0000","last_undegraded":"2024-04-17T08:50:49.019210+0000","last_fullsized":"2024-04-17T08:50:49.019210+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:30:09.203741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057967+0000","last_change":"2024-04-17T08:50:48.034745+0000","last_active":"2024-04-17T08:50:49.057967+0000","last_peered":"2024-04-17T08:50:49.057967+0000","last_clean":"2024-04-17T08:50:49.057967+0000","last_became_active":"2024-04-17T08:50:48.034093+0000","last_became_peered":"2024-04-17T08:50:48.034093+0000","last_unstale":"2024-04-17T08:50:49.057967+0000","last_undegraded":"2024-04-17T08:50:49.057967+0000","last_fullsized":"2024-04-17T08:50:49.057967+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:42.731156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054609+0000","last_change":"2024-04-17T08:50:49.047477+0000","last_active":"2024-04-17T08:50:49.054609+0000","last_peered":"2024-04-17T08:50:49.054609+0000","last_clean":"2024-04-17T08:50:49.054609+0000","last_became_active":"2024-04-17T08:50:49.046829+0000","last_became_peered":"2024-04-17T08:50:49.046829+0000","last_unstale":"2024-04-17T08:50:49.054609+0000","last_undegraded":"2024-04-17T08:50:49.054609+0000","last_fullsized":"2024-04-17T08:50:49.054609+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:57.531837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045911+0000","last_change":"2024-04-17T08:50:32.894323+0000","last_active":"2024-04-17T08:50:49.045911+0000","last_peered":"2024-04-17T08:50:49.045911+0000","last_clean":"2024-04-17T08:50:49.045911+0000","last_became_active":"2024-04-17T08:50:32.893659+0000","last_became_peered":"2024-04-17T08:50:32.893659+0000","last_unstale":"2024-04-17T08:50:49.045911+0000","last_undegraded":"2024-04-17T08:50:49.045911+0000","last_fullsized":"2024-04-17T08:50:49.045911+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:16.424440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017404+0000","last_change":"2024-04-17T08:50:33.884434+0000","last_active":"2024-04-17T08:50:49.017404+0000","last_peered":"2024-04-17T08:50:49.017404+0000","last_clean":"2024-04-17T08:50:49.017404+0000","last_became_active":"2024-04-17T08:50:33.884075+0000","last_became_peered":"2024-04-17T08:50:33.884075+0000","last_unstale":"2024-04-17T08:50:49.017404+0000","last_undegraded":"2024-04-17T08:50:49.017404+0000","last_fullsized":"2024-04-17T08:50:49.017404+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:46.657940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023894+0000","last_change":"2024-04-17T08:50:48.021195+0000","last_active":"2024-04-17T08:50:49.023894+0000","last_peered":"2024-04-17T08:50:49.023894+0000","last_clean":"2024-04-17T08:50:49.023894+0000","last_became_active":"2024-04-17T08:50:48.020868+0000","last_became_peered":"2024-04-17T08:50:48.020868+0000","last_unstale":"2024-04-17T08:50:49.023894+0000","last_undegraded":"2024-04-17T08:50:49.023894+0000","last_fullsized":"2024-04-17T08:50:49.023894+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:31.330314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045649+0000","last_change":"2024-04-17T08:50:49.040734+0000","last_active":"2024-04-17T08:50:49.045649+0000","last_peered":"2024-04-17T08:50:49.045649+0000","last_clean":"2024-04-17T08:50:49.045649+0000","last_became_active":"2024-04-17T08:50:49.040450+0000","last_became_peered":"2024-04-17T08:50:49.040450+0000","last_unstale":"2024-04-17T08:50:49.045649+0000","last_undegraded":"2024-04-17T08:50:49.045649+0000","last_fullsized":"2024-04-17T08:50:49.045649+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:16.518722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011628+0000","last_change":"2024-04-17T08:50:32.902675+0000","last_active":"2024-04-17T08:50:49.011628+0000","last_peered":"2024-04-17T08:50:49.011628+0000","last_clean":"2024-04-17T08:50:49.011628+0000","last_became_active":"2024-04-17T08:50:32.900858+0000","last_became_peered":"2024-04-17T08:50:32.900858+0000","last_unstale":"2024-04-17T08:50:49.011628+0000","last_undegraded":"2024-04-17T08:50:49.011628+0000","last_fullsized":"2024-04-17T08:50:49.011628+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:45.638814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004205+0000","last_change":"2024-04-17T08:50:33.895678+0000","last_active":"2024-04-17T08:50:49.004205+0000","last_peered":"2024-04-17T08:50:49.004205+0000","last_clean":"2024-04-17T08:50:49.004205+0000","last_became_active":"2024-04-17T08:50:33.895010+0000","last_became_peered":"2024-04-17T08:50:33.895010+0000","last_unstale":"2024-04-17T08:50:49.004205+0000","last_undegraded":"2024-04-17T08:50:49.004205+0000","last_fullsized":"2024-04-17T08:50:49.004205+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:46:16.948548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.055450+0000","last_change":"2024-04-17T08:50:48.034319+0000","last_active":"2024-04-17T08:50:49.055450+0000","last_peered":"2024-04-17T08:50:49.055450+0000","last_clean":"2024-04-17T08:50:49.055450+0000","last_became_active":"2024-04-17T08:50:48.033896+0000","last_became_peered":"2024-04-17T08:50:48.033896+0000","last_unstale":"2024-04-17T08:50:49.055450+0000","last_undegraded":"2024-04-17T08:50:49.055450+0000","last_fullsized":"2024-04-17T08:50:49.055450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:44.797993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034984+0000","last_change":"2024-04-17T08:50:49.032527+0000","last_active":"2024-04-17T08:50:49.034984+0000","last_peered":"2024-04-17T08:50:49.034984+0000","last_clean":"2024-04-17T08:50:49.034984+0000","last_became_active":"2024-04-17T08:50:49.031441+0000","last_became_peered":"2024-04-17T08:50:49.031441+0000","last_unstale":"2024-04-17T08:50:49.034984+0000","last_undegraded":"2024-04-17T08:50:49.034984+0000","last_fullsized":"2024-04-17T08:50:49.034984+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:06.087433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014462+0000","last_change":"2024-04-17T08:50:32.891370+0000","last_active":"2024-04-17T08:50:49.014462+0000","last_peered":"2024-04-17T08:50:49.014462+0000","last_clean":"2024-04-17T08:50:49.014462+0000","last_became_active":"2024-04-17T08:50:32.888996+0000","last_became_peered":"2024-04-17T08:50:32.888996+0000","last_unstale":"2024-04-17T08:50:49.014462+0000","last_undegraded":"2024-04-17T08:50:49.014462+0000","last_fullsized":"2024-04-17T08:50:49.014462+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:31.815168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048769+0000","last_change":"2024-04-17T08:50:33.892519+0000","last_active":"2024-04-17T08:50:49.048769+0000","last_peered":"2024-04-17T08:50:49.048769+0000","last_clean":"2024-04-17T08:50:49.048769+0000","last_became_active":"2024-04-17T08:50:33.892264+0000","last_became_peered":"2024-04-17T08:50:33.892264+0000","last_unstale":"2024-04-17T08:50:49.048769+0000","last_undegraded":"2024-04-17T08:50:49.048769+0000","last_fullsized":"2024-04-17T08:50:49.048769+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:05.376200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048842+0000","last_change":"2024-04-17T08:50:48.029938+0000","last_active":"2024-04-17T08:50:49.048842+0000","last_peered":"2024-04-17T08:50:49.048842+0000","last_clean":"2024-04-17T08:50:49.048842+0000","last_became_active":"2024-04-17T08:50:48.029123+0000","last_became_peered":"2024-04-17T08:50:48.029123+0000","last_unstale":"2024-04-17T08:50:49.048842+0000","last_undegraded":"2024-04-17T08:50:49.048842+0000","last_fullsized":"2024-04-17T08:50:49.048842+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:21.642236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054616+0000","last_change":"2024-04-17T08:50:49.046714+0000","last_active":"2024-04-17T08:50:49.054616+0000","last_peered":"2024-04-17T08:50:49.054616+0000","last_clean":"2024-04-17T08:50:49.054616+0000","last_became_active":"2024-04-17T08:50:49.046392+0000","last_became_peered":"2024-04-17T08:50:49.046392+0000","last_unstale":"2024-04-17T08:50:49.054616+0000","last_undegraded":"2024-04-17T08:50:49.054616+0000","last_fullsized":"2024-04-17T08:50:49.054616+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:08.981690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021782+0000","last_change":"2024-04-17T08:50:32.903970+0000","last_active":"2024-04-17T08:50:49.021782+0000","last_peered":"2024-04-17T08:50:49.021782+0000","last_clean":"2024-04-17T08:50:49.021782+0000","last_became_active":"2024-04-17T08:50:32.903620+0000","last_became_peered":"2024-04-17T08:50:32.903620+0000","last_unstale":"2024-04-17T08:50:49.021782+0000","last_undegraded":"2024-04-17T08:50:49.021782+0000","last_fullsized":"2024-04-17T08:50:49.021782+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:33.640417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019406+0000","last_change":"2024-04-17T08:50:33.879546+0000","last_active":"2024-04-17T08:50:49.019406+0000","last_peered":"2024-04-17T08:50:49.019406+0000","last_clean":"2024-04-17T08:50:49.019406+0000","last_became_active":"2024-04-17T08:50:33.878778+0000","last_became_peered":"2024-04-17T08:50:33.878778+0000","last_unstale":"2024-04-17T08:50:49.019406+0000","last_undegraded":"2024-04-17T08:50:49.019406+0000","last_fullsized":"2024-04-17T08:50:49.019406+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.667020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028625+0000","last_change":"2024-04-17T08:50:48.023810+0000","last_active":"2024-04-17T08:50:49.028625+0000","last_peered":"2024-04-17T08:50:49.028625+0000","last_clean":"2024-04-17T08:50:49.028625+0000","last_became_active":"2024-04-17T08:50:48.023485+0000","last_became_peered":"2024-04-17T08:50:48.023485+0000","last_unstale":"2024-04-17T08:50:49.028625+0000","last_undegraded":"2024-04-17T08:50:49.028625+0000","last_fullsized":"2024-04-17T08:50:49.028625+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:54.085024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.056371+0000","last_change":"2024-04-17T08:50:49.046526+0000","last_active":"2024-04-17T08:50:49.056371+0000","last_peered":"2024-04-17T08:50:49.056371+0000","last_clean":"2024-04-17T08:50:49.056371+0000","last_became_active":"2024-04-17T08:50:49.046205+0000","last_became_peered":"2024-04-17T08:50:49.046205+0000","last_unstale":"2024-04-17T08:50:49.056371+0000","last_undegraded":"2024-04-17T08:50:49.056371+0000","last_fullsized":"2024-04-17T08:50:49.056371+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:30.247778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009992+0000","last_change":"2024-04-17T08:50:32.906189+0000","last_active":"2024-04-17T08:50:49.009992+0000","last_peered":"2024-04-17T08:50:49.009992+0000","last_clean":"2024-04-17T08:50:49.009992+0000","last_became_active":"2024-04-17T08:50:32.905898+0000","last_became_peered":"2024-04-17T08:50:32.905898+0000","last_unstale":"2024-04-17T08:50:49.009992+0000","last_undegraded":"2024-04-17T08:50:49.009992+0000","last_fullsized":"2024-04-17T08:50:49.009992+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:24.678638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014219+0000","last_change":"2024-04-17T08:50:33.892430+0000","last_active":"2024-04-17T08:50:49.014219+0000","last_peered":"2024-04-17T08:50:49.014219+0000","last_clean":"2024-04-17T08:50:49.014219+0000","last_became_active":"2024-04-17T08:50:33.892133+0000","last_became_peered":"2024-04-17T08:50:33.892133+0000","last_unstale":"2024-04-17T08:50:49.014219+0000","last_undegraded":"2024-04-17T08:50:49.014219+0000","last_fullsized":"2024-04-17T08:50:49.014219+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:34.406770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016943+0000","last_change":"2024-04-17T08:50:48.017579+0000","last_active":"2024-04-17T08:50:49.016943+0000","last_peered":"2024-04-17T08:50:49.016943+0000","last_clean":"2024-04-17T08:50:49.016943+0000","last_became_active":"2024-04-17T08:50:48.016441+0000","last_became_peered":"2024-04-17T08:50:48.016441+0000","last_unstale":"2024-04-17T08:50:49.016943+0000","last_undegraded":"2024-04-17T08:50:49.016943+0000","last_fullsized":"2024-04-17T08:50:49.016943+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:30.341972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034323+0000","last_change":"2024-04-17T08:50:49.026143+0000","last_active":"2024-04-17T08:50:49.034323+0000","last_peered":"2024-04-17T08:50:49.034323+0000","last_clean":"2024-04-17T08:50:49.034323+0000","last_became_active":"2024-04-17T08:50:49.025856+0000","last_became_peered":"2024-04-17T08:50:49.025856+0000","last_unstale":"2024-04-17T08:50:49.034323+0000","last_undegraded":"2024-04-17T08:50:49.034323+0000","last_fullsized":"2024-04-17T08:50:49.034323+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.074044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019705+0000","last_change":"2024-04-17T08:50:32.901213+0000","last_active":"2024-04-17T08:50:49.019705+0000","last_peered":"2024-04-17T08:50:49.019705+0000","last_clean":"2024-04-17T08:50:49.019705+0000","last_became_active":"2024-04-17T08:50:32.900942+0000","last_became_peered":"2024-04-17T08:50:32.900942+0000","last_unstale":"2024-04-17T08:50:49.019705+0000","last_undegraded":"2024-04-17T08:50:49.019705+0000","last_fullsized":"2024-04-17T08:50:49.019705+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:35.477286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018049+0000","last_change":"2024-04-17T08:50:33.887506+0000","last_active":"2024-04-17T08:50:49.018049+0000","last_peered":"2024-04-17T08:50:49.018049+0000","last_clean":"2024-04-17T08:50:49.018049+0000","last_became_active":"2024-04-17T08:50:33.887071+0000","last_became_peered":"2024-04-17T08:50:33.887071+0000","last_unstale":"2024-04-17T08:50:49.018049+0000","last_undegraded":"2024-04-17T08:50:49.018049+0000","last_fullsized":"2024-04-17T08:50:49.018049+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:23.927426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020363+0000","last_change":"2024-04-17T08:50:48.018181+0000","last_active":"2024-04-17T08:50:49.020363+0000","last_peered":"2024-04-17T08:50:49.020363+0000","last_clean":"2024-04-17T08:50:49.020363+0000","last_became_active":"2024-04-17T08:50:48.016820+0000","last_became_peered":"2024-04-17T08:50:48.016820+0000","last_unstale":"2024-04-17T08:50:49.020363+0000","last_undegraded":"2024-04-17T08:50:49.020363+0000","last_fullsized":"2024-04-17T08:50:49.020363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:51.729857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.043228+0000","last_change":"2024-04-17T08:50:49.039958+0000","last_active":"2024-04-17T08:50:49.043228+0000","last_peered":"2024-04-17T08:50:49.043228+0000","last_clean":"2024-04-17T08:50:49.043228+0000","last_became_active":"2024-04-17T08:50:49.039667+0000","last_became_peered":"2024-04-17T08:50:49.039667+0000","last_unstale":"2024-04-17T08:50:49.043228+0000","last_undegraded":"2024-04-17T08:50:49.043228+0000","last_fullsized":"2024-04-17T08:50:49.043228+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:10.730836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"230'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024098+0000","last_change":"2024-04-17T08:50:32.904455+0000","last_active":"2024-04-17T08:50:49.024098+0000","last_peered":"2024-04-17T08:50:49.024098+0000","last_clean":"2024-04-17T08:50:49.024098+0000","last_became_active":"2024-04-17T08:50:32.904127+0000","last_became_peered":"2024-04-17T08:50:32.904127+0000","last_unstale":"2024-04-17T08:50:49.024098+0000","last_undegraded":"2024-04-17T08:50:49.024098+0000","last_fullsized":"2024-04-17T08:50:49.024098+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:09.473209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017287+0000","last_change":"2024-04-17T08:50:33.892244+0000","last_active":"2024-04-17T08:50:49.017287+0000","last_peered":"2024-04-17T08:50:49.017287+0000","last_clean":"2024-04-17T08:50:49.017287+0000","last_became_active":"2024-04-17T08:50:33.891771+0000","last_became_peered":"2024-04-17T08:50:33.891771+0000","last_unstale":"2024-04-17T08:50:49.017287+0000","last_undegraded":"2024-04-17T08:50:49.017287+0000","last_fullsized":"2024-04-17T08:50:49.017287+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:29.039336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045634+0000","last_change":"2024-04-17T08:50:48.027200+0000","last_active":"2024-04-17T08:50:49.045634+0000","last_peered":"2024-04-17T08:50:49.045634+0000","last_clean":"2024-04-17T08:50:49.045634+0000","last_became_active":"2024-04-17T08:50:48.026771+0000","last_became_peered":"2024-04-17T08:50:48.026771+0000","last_unstale":"2024-04-17T08:50:49.045634+0000","last_undegraded":"2024-04-17T08:50:49.045634+0000","last_fullsized":"2024-04-17T08:50:49.045634+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:31.956458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054790+0000","last_change":"2024-04-17T08:50:49.047452+0000","last_active":"2024-04-17T08:50:49.054790+0000","last_peered":"2024-04-17T08:50:49.054790+0000","last_clean":"2024-04-17T08:50:49.054790+0000","last_became_active":"2024-04-17T08:50:49.047142+0000","last_became_peered":"2024-04-17T08:50:49.047142+0000","last_unstale":"2024-04-17T08:50:49.054790+0000","last_undegraded":"2024-04-17T08:50:49.054790+0000","last_fullsized":"2024-04-17T08:50:49.054790+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:01.781200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.006696+0000","last_change":"2024-04-17T08:50:32.906744+0000","last_active":"2024-04-17T08:50:49.006696+0000","last_peered":"2024-04-17T08:50:49.006696+0000","last_clean":"2024-04-17T08:50:49.006696+0000","last_became_active":"2024-04-17T08:50:32.906445+0000","last_became_peered":"2024-04-17T08:50:32.906445+0000","last_unstale":"2024-04-17T08:50:49.006696+0000","last_undegraded":"2024-04-17T08:50:49.006696+0000","last_fullsized":"2024-04-17T08:50:49.006696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:33.054437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054335+0000","last_change":"2024-04-17T08:50:33.899749+0000","last_active":"2024-04-17T08:50:49.054335+0000","last_peered":"2024-04-17T08:50:49.054335+0000","last_clean":"2024-04-17T08:50:49.054335+0000","last_became_active":"2024-04-17T08:50:33.898270+0000","last_became_peered":"2024-04-17T08:50:33.898270+0000","last_unstale":"2024-04-17T08:50:49.054335+0000","last_undegraded":"2024-04-17T08:50:49.054335+0000","last_fullsized":"2024-04-17T08:50:49.054335+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:51.073059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014937+0000","last_change":"2024-04-17T08:50:48.023544+0000","last_active":"2024-04-17T08:50:49.014937+0000","last_peered":"2024-04-17T08:50:49.014937+0000","last_clean":"2024-04-17T08:50:49.014937+0000","last_became_active":"2024-04-17T08:50:48.023218+0000","last_became_peered":"2024-04-17T08:50:48.023218+0000","last_unstale":"2024-04-17T08:50:49.014937+0000","last_undegraded":"2024-04-17T08:50:49.014937+0000","last_fullsized":"2024-04-17T08:50:49.014937+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:09:12.937584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.042867+0000","last_change":"2024-04-17T08:50:49.040211+0000","last_active":"2024-04-17T08:50:49.042867+0000","last_peered":"2024-04-17T08:50:49.042867+0000","last_clean":"2024-04-17T08:50:49.042867+0000","last_became_active":"2024-04-17T08:50:49.039900+0000","last_became_peered":"2024-04-17T08:50:49.039900+0000","last_unstale":"2024-04-17T08:50:49.042867+0000","last_undegraded":"2024-04-17T08:50:49.042867+0000","last_fullsized":"2024-04-17T08:50:49.042867+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:23:55.621600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027356+0000","last_change":"2024-04-17T08:50:32.894904+0000","last_active":"2024-04-17T08:50:49.027356+0000","last_peered":"2024-04-17T08:50:49.027356+0000","last_clean":"2024-04-17T08:50:49.027356+0000","last_became_active":"2024-04-17T08:50:32.894624+0000","last_became_peered":"2024-04-17T08:50:32.894624+0000","last_unstale":"2024-04-17T08:50:49.027356+0000","last_undegraded":"2024-04-17T08:50:49.027356+0000","last_fullsized":"2024-04-17T08:50:49.027356+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:22.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011363+0000","last_change":"2024-04-17T08:50:33.892798+0000","last_active":"2024-04-17T08:50:49.011363+0000","last_peered":"2024-04-17T08:50:49.011363+0000","last_clean":"2024-04-17T08:50:49.011363+0000","last_became_active":"2024-04-17T08:50:33.891130+0000","last_became_peered":"2024-04-17T08:50:33.891130+0000","last_unstale":"2024-04-17T08:50:49.011363+0000","last_undegraded":"2024-04-17T08:50:49.011363+0000","last_fullsized":"2024-04-17T08:50:49.011363+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:43.282486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052741+0000","last_change":"2024-04-17T08:50:48.021836+0000","last_active":"2024-04-17T08:50:49.052741+0000","last_peered":"2024-04-17T08:50:49.052741+0000","last_clean":"2024-04-17T08:50:49.052741+0000","last_became_active":"2024-04-17T08:50:48.019867+0000","last_became_peered":"2024-04-17T08:50:48.019867+0000","last_unstale":"2024-04-17T08:50:49.052741+0000","last_undegraded":"2024-04-17T08:50:49.052741+0000","last_fullsized":"2024-04-17T08:50:49.052741+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:13.501771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.035124+0000","last_change":"2024-04-17T08:50:49.030997+0000","last_active":"2024-04-17T08:50:49.035124+0000","last_peered":"2024-04-17T08:50:49.035124+0000","last_clean":"2024-04-17T08:50:49.035124+0000","last_became_active":"2024-04-17T08:50:49.030558+0000","last_became_peered":"2024-04-17T08:50:49.030558+0000","last_unstale":"2024-04-17T08:50:49.035124+0000","last_undegraded":"2024-04-17T08:50:49.035124+0000","last_fullsized":"2024-04-17T08:50:49.035124+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:11.503895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019846+0000","last_change":"2024-04-17T08:50:32.896102+0000","last_active":"2024-04-17T08:50:49.019846+0000","last_peered":"2024-04-17T08:50:49.019846+0000","last_clean":"2024-04-17T08:50:49.019846+0000","last_became_active":"2024-04-17T08:50:32.895808+0000","last_became_peered":"2024-04-17T08:50:32.895808+0000","last_unstale":"2024-04-17T08:50:49.019846+0000","last_undegraded":"2024-04-17T08:50:49.019846+0000","last_fullsized":"2024-04-17T08:50:49.019846+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:38.551687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016137+0000","last_change":"2024-04-17T08:50:33.884383+0000","last_active":"2024-04-17T08:50:49.016137+0000","last_peered":"2024-04-17T08:50:49.016137+0000","last_clean":"2024-04-17T08:50:49.016137+0000","last_became_active":"2024-04-17T08:50:33.883473+0000","last_became_peered":"2024-04-17T08:50:33.883473+0000","last_unstale":"2024-04-17T08:50:49.016137+0000","last_undegraded":"2024-04-17T08:50:49.016137+0000","last_fullsized":"2024-04-17T08:50:49.016137+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:03.466744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060403+0000","last_change":"2024-04-17T08:50:48.032503+0000","last_active":"2024-04-17T08:50:49.060403+0000","last_peered":"2024-04-17T08:50:49.060403+0000","last_clean":"2024-04-17T08:50:49.060403+0000","last_became_active":"2024-04-17T08:50:48.032090+0000","last_became_peered":"2024-04-17T08:50:48.032090+0000","last_unstale":"2024-04-17T08:50:49.060403+0000","last_undegraded":"2024-04-17T08:50:49.060403+0000","last_fullsized":"2024-04-17T08:50:49.060403+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:38.033613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045756+0000","last_change":"2024-04-17T08:50:49.037319+0000","last_active":"2024-04-17T08:50:49.045756+0000","last_peered":"2024-04-17T08:50:49.045756+0000","last_clean":"2024-04-17T08:50:49.045756+0000","last_became_active":"2024-04-17T08:50:49.037012+0000","last_became_peered":"2024-04-17T08:50:49.037012+0000","last_unstale":"2024-04-17T08:50:49.045756+0000","last_undegraded":"2024-04-17T08:50:49.045756+0000","last_fullsized":"2024-04-17T08:50:49.045756+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:10.872839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024506+0000","last_change":"2024-04-17T08:50:32.902321+0000","last_active":"2024-04-17T08:50:49.024506+0000","last_peered":"2024-04-17T08:50:49.024506+0000","last_clean":"2024-04-17T08:50:49.024506+0000","last_became_active":"2024-04-17T08:50:32.901781+0000","last_became_peered":"2024-04-17T08:50:32.901781+0000","last_unstale":"2024-04-17T08:50:49.024506+0000","last_undegraded":"2024-04-17T08:50:49.024506+0000","last_fullsized":"2024-04-17T08:50:49.024506+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:04.585512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004920+0000","last_change":"2024-04-17T08:50:33.897335+0000","last_active":"2024-04-17T08:50:49.004920+0000","last_peered":"2024-04-17T08:50:49.004920+0000","last_clean":"2024-04-17T08:50:49.004920+0000","last_became_active":"2024-04-17T08:50:33.897009+0000","last_became_peered":"2024-04-17T08:50:33.897009+0000","last_unstale":"2024-04-17T08:50:49.004920+0000","last_undegraded":"2024-04-17T08:50:49.004920+0000","last_fullsized":"2024-04-17T08:50:49.004920+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:12.674509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023238+0000","last_change":"2024-04-17T08:50:48.019566+0000","last_active":"2024-04-17T08:50:49.023238+0000","last_peered":"2024-04-17T08:50:49.023238+0000","last_clean":"2024-04-17T08:50:49.023238+0000","last_became_active":"2024-04-17T08:50:48.018570+0000","last_became_peered":"2024-04-17T08:50:48.018570+0000","last_unstale":"2024-04-17T08:50:49.023238+0000","last_undegraded":"2024-04-17T08:50:49.023238+0000","last_fullsized":"2024-04-17T08:50:49.023238+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:36.720735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046848+0000","last_change":"2024-04-17T08:50:49.037873+0000","last_active":"2024-04-17T08:50:49.046848+0000","last_peered":"2024-04-17T08:50:49.046848+0000","last_clean":"2024-04-17T08:50:49.046848+0000","last_became_active":"2024-04-17T08:50:49.037589+0000","last_became_peered":"2024-04-17T08:50:49.037589+0000","last_unstale":"2024-04-17T08:50:49.046848+0000","last_undegraded":"2024-04-17T08:50:49.046848+0000","last_fullsized":"2024-04-17T08:50:49.046848+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:02.288019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026728+0000","last_change":"2024-04-17T08:50:32.898032+0000","last_active":"2024-04-17T08:50:49.026728+0000","last_peered":"2024-04-17T08:50:49.026728+0000","last_clean":"2024-04-17T08:50:49.026728+0000","last_became_active":"2024-04-17T08:50:32.897755+0000","last_became_peered":"2024-04-17T08:50:32.897755+0000","last_unstale":"2024-04-17T08:50:49.026728+0000","last_undegraded":"2024-04-17T08:50:49.026728+0000","last_fullsized":"2024-04-17T08:50:49.026728+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:29.382845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023428+0000","last_change":"2024-04-17T08:50:33.897197+0000","last_active":"2024-04-17T08:50:49.023428+0000","last_peered":"2024-04-17T08:50:49.023428+0000","last_clean":"2024-04-17T08:50:49.023428+0000","last_became_active":"2024-04-17T08:50:33.895389+0000","last_became_peered":"2024-04-17T08:50:33.895389+0000","last_unstale":"2024-04-17T08:50:49.023428+0000","last_undegraded":"2024-04-17T08:50:49.023428+0000","last_fullsized":"2024-04-17T08:50:49.023428+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:48:09.426321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048980+0000","last_change":"2024-04-17T08:50:48.019845+0000","last_active":"2024-04-17T08:50:49.048980+0000","last_peered":"2024-04-17T08:50:49.048980+0000","last_clean":"2024-04-17T08:50:49.048980+0000","last_became_active":"2024-04-17T08:50:48.019186+0000","last_became_peered":"2024-04-17T08:50:48.019186+0000","last_unstale":"2024-04-17T08:50:49.048980+0000","last_undegraded":"2024-04-17T08:50:49.048980+0000","last_fullsized":"2024-04-17T08:50:49.048980+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:50.726916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059762+0000","last_change":"2024-04-17T08:50:49.055993+0000","last_active":"2024-04-17T08:50:49.059762+0000","last_peered":"2024-04-17T08:50:49.059762+0000","last_clean":"2024-04-17T08:50:49.059762+0000","last_became_active":"2024-04-17T08:50:49.055537+0000","last_became_peered":"2024-04-17T08:50:49.055537+0000","last_unstale":"2024-04-17T08:50:49.059762+0000","last_undegraded":"2024-04-17T08:50:49.059762+0000","last_fullsized":"2024-04-17T08:50:49.059762+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:01.853612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009754+0000","last_change":"2024-04-17T08:50:32.903243+0000","last_active":"2024-04-17T08:50:49.009754+0000","last_peered":"2024-04-17T08:50:49.009754+0000","last_clean":"2024-04-17T08:50:49.009754+0000","last_became_active":"2024-04-17T08:50:32.902864+0000","last_became_peered":"2024-04-17T08:50:32.902864+0000","last_unstale":"2024-04-17T08:50:49.009754+0000","last_undegraded":"2024-04-17T08:50:49.009754+0000","last_fullsized":"2024-04-17T08:50:49.009754+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:14:20.222164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023609+0000","last_change":"2024-04-17T08:50:33.905253+0000","last_active":"2024-04-17T08:50:49.023609+0000","last_peered":"2024-04-17T08:50:49.023609+0000","last_clean":"2024-04-17T08:50:49.023609+0000","last_became_active":"2024-04-17T08:50:33.904731+0000","last_became_peered":"2024-04-17T08:50:33.904731+0000","last_unstale":"2024-04-17T08:50:49.023609+0000","last_undegraded":"2024-04-17T08:50:49.023609+0000","last_fullsized":"2024-04-17T08:50:49.023609+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:56.634018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016476+0000","last_change":"2024-04-17T08:50:48.018547+0000","last_active":"2024-04-17T08:50:49.016476+0000","last_peered":"2024-04-17T08:50:49.016476+0000","last_clean":"2024-04-17T08:50:49.016476+0000","last_became_active":"2024-04-17T08:50:48.018220+0000","last_became_peered":"2024-04-17T08:50:48.018220+0000","last_unstale":"2024-04-17T08:50:49.016476+0000","last_undegraded":"2024-04-17T08:50:49.016476+0000","last_fullsized":"2024-04-17T08:50:49.016476+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:00.149916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046188+0000","last_change":"2024-04-17T08:50:49.044469+0000","last_active":"2024-04-17T08:50:49.046188+0000","last_peered":"2024-04-17T08:50:49.046188+0000","last_clean":"2024-04-17T08:50:49.046188+0000","last_became_active":"2024-04-17T08:50:49.044170+0000","last_became_peered":"2024-04-17T08:50:49.044170+0000","last_unstale":"2024-04-17T08:50:49.046188+0000","last_undegraded":"2024-04-17T08:50:49.046188+0000","last_fullsized":"2024-04-17T08:50:49.046188+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:53.313084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026292+0000","last_change":"2024-04-17T08:50:32.896384+0000","last_active":"2024-04-17T08:50:49.026292+0000","last_peered":"2024-04-17T08:50:49.026292+0000","last_clean":"2024-04-17T08:50:49.026292+0000","last_became_active":"2024-04-17T08:50:32.893495+0000","last_became_peered":"2024-04-17T08:50:32.893495+0000","last_unstale":"2024-04-17T08:50:49.026292+0000","last_undegraded":"2024-04-17T08:50:49.026292+0000","last_fullsized":"2024-04-17T08:50:49.026292+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.852683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.009473+0000","last_change":"2024-04-17T08:50:33.899313+0000","last_active":"2024-04-17T08:50:49.009473+0000","last_peered":"2024-04-17T08:50:49.009473+0000","last_clean":"2024-04-17T08:50:49.009473+0000","last_became_active":"2024-04-17T08:50:33.899005+0000","last_became_peered":"2024-04-17T08:50:33.899005+0000","last_unstale":"2024-04-17T08:50:49.009473+0000","last_undegraded":"2024-04-17T08:50:49.009473+0000","last_fullsized":"2024-04-17T08:50:49.009473+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:15:34.655071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060016+0000","last_change":"2024-04-17T08:50:48.033453+0000","last_active":"2024-04-17T08:50:49.060016+0000","last_peered":"2024-04-17T08:50:49.060016+0000","last_clean":"2024-04-17T08:50:49.060016+0000","last_became_active":"2024-04-17T08:50:48.032899+0000","last_became_peered":"2024-04-17T08:50:48.032899+0000","last_unstale":"2024-04-17T08:50:49.060016+0000","last_undegraded":"2024-04-17T08:50:49.060016+0000","last_fullsized":"2024-04-17T08:50:49.060016+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:33.289131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045946+0000","last_change":"2024-04-17T08:50:49.042134+0000","last_active":"2024-04-17T08:50:49.045946+0000","last_peered":"2024-04-17T08:50:49.045946+0000","last_clean":"2024-04-17T08:50:49.045946+0000","last_became_active":"2024-04-17T08:50:49.041627+0000","last_became_peered":"2024-04-17T08:50:49.041627+0000","last_unstale":"2024-04-17T08:50:49.045946+0000","last_undegraded":"2024-04-17T08:50:49.045946+0000","last_fullsized":"2024-04-17T08:50:49.045946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:59.522791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020619+0000","last_change":"2024-04-17T08:50:32.901050+0000","last_active":"2024-04-17T08:50:49.020619+0000","last_peered":"2024-04-17T08:50:49.020619+0000","last_clean":"2024-04-17T08:50:49.020619+0000","last_became_active":"2024-04-17T08:50:32.900689+0000","last_became_peered":"2024-04-17T08:50:32.900689+0000","last_unstale":"2024-04-17T08:50:49.020619+0000","last_undegraded":"2024-04-17T08:50:49.020619+0000","last_fullsized":"2024-04-17T08:50:49.020619+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:39:51.234349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019043+0000","last_change":"2024-04-17T08:50:33.907099+0000","last_active":"2024-04-17T08:50:49.019043+0000","last_peered":"2024-04-17T08:50:49.019043+0000","last_clean":"2024-04-17T08:50:49.019043+0000","last_became_active":"2024-04-17T08:50:33.906802+0000","last_became_peered":"2024-04-17T08:50:33.906802+0000","last_unstale":"2024-04-17T08:50:49.019043+0000","last_undegraded":"2024-04-17T08:50:49.019043+0000","last_fullsized":"2024-04-17T08:50:49.019043+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:53.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047133+0000","last_change":"2024-04-17T08:50:48.028554+0000","last_active":"2024-04-17T08:50:49.047133+0000","last_peered":"2024-04-17T08:50:49.047133+0000","last_clean":"2024-04-17T08:50:49.047133+0000","last_became_active":"2024-04-17T08:50:48.028241+0000","last_became_peered":"2024-04-17T08:50:48.028241+0000","last_unstale":"2024-04-17T08:50:49.047133+0000","last_undegraded":"2024-04-17T08:50:49.047133+0000","last_fullsized":"2024-04-17T08:50:49.047133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:49.274398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.035802+0000","last_change":"2024-04-17T08:50:49.032405+0000","last_active":"2024-04-17T08:50:49.035802+0000","last_peered":"2024-04-17T08:50:49.035802+0000","last_clean":"2024-04-17T08:50:49.035802+0000","last_became_active":"2024-04-17T08:50:49.031813+0000","last_became_peered":"2024-04-17T08:50:49.031813+0000","last_unstale":"2024-04-17T08:50:49.035802+0000","last_undegraded":"2024-04-17T08:50:49.035802+0000","last_fullsized":"2024-04-17T08:50:49.035802+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:23.883528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017595+0000","last_change":"2024-04-17T08:50:32.891171+0000","last_active":"2024-04-17T08:50:49.017595+0000","last_peered":"2024-04-17T08:50:49.017595+0000","last_clean":"2024-04-17T08:50:49.017595+0000","last_became_active":"2024-04-17T08:50:32.888285+0000","last_became_peered":"2024-04-17T08:50:32.888285+0000","last_unstale":"2024-04-17T08:50:49.017595+0000","last_undegraded":"2024-04-17T08:50:49.017595+0000","last_fullsized":"2024-04-17T08:50:49.017595+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:33.387644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019578+0000","last_change":"2024-04-17T08:50:33.887711+0000","last_active":"2024-04-17T08:50:49.019578+0000","last_peered":"2024-04-17T08:50:49.019578+0000","last_clean":"2024-04-17T08:50:49.019578+0000","last_became_active":"2024-04-17T08:50:33.887362+0000","last_became_peered":"2024-04-17T08:50:33.887362+0000","last_unstale":"2024-04-17T08:50:49.019578+0000","last_undegraded":"2024-04-17T08:50:49.019578+0000","last_fullsized":"2024-04-17T08:50:49.019578+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:07.536049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017490+0000","last_change":"2024-04-17T08:50:48.023932+0000","last_active":"2024-04-17T08:50:49.017490+0000","last_peered":"2024-04-17T08:50:49.017490+0000","last_clean":"2024-04-17T08:50:49.017490+0000","last_became_active":"2024-04-17T08:50:48.023640+0000","last_became_peered":"2024-04-17T08:50:48.023640+0000","last_unstale":"2024-04-17T08:50:49.017490+0000","last_undegraded":"2024-04-17T08:50:49.017490+0000","last_fullsized":"2024-04-17T08:50:49.017490+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:01.151039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045189+0000","last_change":"2024-04-17T08:50:49.042247+0000","last_active":"2024-04-17T08:50:49.045189+0000","last_peered":"2024-04-17T08:50:49.045189+0000","last_clean":"2024-04-17T08:50:49.045189+0000","last_became_active":"2024-04-17T08:50:49.041744+0000","last_became_peered":"2024-04-17T08:50:49.041744+0000","last_unstale":"2024-04-17T08:50:49.045189+0000","last_undegraded":"2024-04-17T08:50:49.045189+0000","last_fullsized":"2024-04-17T08:50:49.045189+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:34:57.766992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"229'8","reported_seq":28,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005497+0000","last_change":"2024-04-17T08:50:32.903713+0000","last_active":"2024-04-17T08:50:49.005497+0000","last_peered":"2024-04-17T08:50:49.005497+0000","last_clean":"2024-04-17T08:50:49.005497+0000","last_became_active":"2024-04-17T08:50:32.903403+0000","last_became_peered":"2024-04-17T08:50:32.903403+0000","last_unstale":"2024-04-17T08:50:49.005497+0000","last_undegraded":"2024-04-17T08:50:49.005497+0000","last_fullsized":"2024-04-17T08:50:49.005497+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:09.595683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019655+0000","last_change":"2024-04-17T08:50:33.889134+0000","last_active":"2024-04-17T08:50:49.019655+0000","last_peered":"2024-04-17T08:50:49.019655+0000","last_clean":"2024-04-17T08:50:49.019655+0000","last_became_active":"2024-04-17T08:50:33.886115+0000","last_became_peered":"2024-04-17T08:50:33.886115+0000","last_unstale":"2024-04-17T08:50:49.019655+0000","last_undegraded":"2024-04-17T08:50:49.019655+0000","last_fullsized":"2024-04-17T08:50:49.019655+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:20.719327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.015688+0000","last_change":"2024-04-17T08:50:48.017932+0000","last_active":"2024-04-17T08:50:49.015688+0000","last_peered":"2024-04-17T08:50:49.015688+0000","last_clean":"2024-04-17T08:50:49.015688+0000","last_became_active":"2024-04-17T08:50:48.017662+0000","last_became_peered":"2024-04-17T08:50:48.017662+0000","last_unstale":"2024-04-17T08:50:49.015688+0000","last_undegraded":"2024-04-17T08:50:49.015688+0000","last_fullsized":"2024-04-17T08:50:49.015688+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:44.878825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047027+0000","last_change":"2024-04-17T08:50:49.045258+0000","last_active":"2024-04-17T08:50:49.047027+0000","last_peered":"2024-04-17T08:50:49.047027+0000","last_clean":"2024-04-17T08:50:49.047027+0000","last_became_active":"2024-04-17T08:50:49.044518+0000","last_became_peered":"2024-04-17T08:50:49.044518+0000","last_unstale":"2024-04-17T08:50:49.047027+0000","last_undegraded":"2024-04-17T08:50:49.047027+0000","last_fullsized":"2024-04-17T08:50:49.047027+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:13.816186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025562+0000","last_change":"2024-04-17T08:50:32.895429+0000","last_active":"2024-04-17T08:50:49.025562+0000","last_peered":"2024-04-17T08:50:49.025562+0000","last_clean":"2024-04-17T08:50:49.025562+0000","last_became_active":"2024-04-17T08:50:32.894971+0000","last_became_peered":"2024-04-17T08:50:32.894971+0000","last_unstale":"2024-04-17T08:50:49.025562+0000","last_undegraded":"2024-04-17T08:50:49.025562+0000","last_fullsized":"2024-04-17T08:50:49.025562+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:22.318570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004172+0000","last_change":"2024-04-17T08:50:33.896023+0000","last_active":"2024-04-17T08:50:49.004172+0000","last_peered":"2024-04-17T08:50:49.004172+0000","last_clean":"2024-04-17T08:50:49.004172+0000","last_became_active":"2024-04-17T08:50:33.895404+0000","last_became_peered":"2024-04-17T08:50:33.895404+0000","last_unstale":"2024-04-17T08:50:49.004172+0000","last_undegraded":"2024-04-17T08:50:49.004172+0000","last_fullsized":"2024-04-17T08:50:49.004172+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:17.385187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.032923+0000","last_change":"2024-04-17T08:50:48.027804+0000","last_active":"2024-04-17T08:50:49.032923+0000","last_peered":"2024-04-17T08:50:49.032923+0000","last_clean":"2024-04-17T08:50:49.032923+0000","last_became_active":"2024-04-17T08:50:48.027446+0000","last_became_peered":"2024-04-17T08:50:48.027446+0000","last_unstale":"2024-04-17T08:50:49.032923+0000","last_undegraded":"2024-04-17T08:50:49.032923+0000","last_fullsized":"2024-04-17T08:50:49.032923+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:07.905438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045017+0000","last_change":"2024-04-17T08:50:49.041513+0000","last_active":"2024-04-17T08:50:49.045017+0000","last_peered":"2024-04-17T08:50:49.045017+0000","last_clean":"2024-04-17T08:50:49.045017+0000","last_became_active":"2024-04-17T08:50:49.041235+0000","last_became_peered":"2024-04-17T08:50:49.041235+0000","last_unstale":"2024-04-17T08:50:49.045017+0000","last_undegraded":"2024-04-17T08:50:49.045017+0000","last_fullsized":"2024-04-17T08:50:49.045017+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:54.702081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012191+0000","last_change":"2024-04-17T08:50:32.896932+0000","last_active":"2024-04-17T08:50:49.012191+0000","last_peered":"2024-04-17T08:50:49.012191+0000","last_clean":"2024-04-17T08:50:49.012191+0000","last_became_active":"2024-04-17T08:50:32.895134+0000","last_became_peered":"2024-04-17T08:50:32.895134+0000","last_unstale":"2024-04-17T08:50:49.012191+0000","last_undegraded":"2024-04-17T08:50:49.012191+0000","last_fullsized":"2024-04-17T08:50:49.012191+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:02.447617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025672+0000","last_change":"2024-04-17T08:50:33.909416+0000","last_active":"2024-04-17T08:50:49.025672+0000","last_peered":"2024-04-17T08:50:49.025672+0000","last_clean":"2024-04-17T08:50:49.025672+0000","last_became_active":"2024-04-17T08:50:33.906483+0000","last_became_peered":"2024-04-17T08:50:33.906483+0000","last_unstale":"2024-04-17T08:50:49.025672+0000","last_undegraded":"2024-04-17T08:50:49.025672+0000","last_fullsized":"2024-04-17T08:50:49.025672+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:07:00.037620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"233'6","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059068+0000","last_change":"2024-04-17T08:50:48.025812+0000","last_active":"2024-04-17T08:50:49.059068+0000","last_peered":"2024-04-17T08:50:49.059068+0000","last_clean":"2024-04-17T08:50:49.059068+0000","last_became_active":"2024-04-17T08:50:48.024873+0000","last_became_peered":"2024-04-17T08:50:48.024873+0000","last_unstale":"2024-04-17T08:50:49.059068+0000","last_undegraded":"2024-04-17T08:50:49.059068+0000","last_fullsized":"2024-04-17T08:50:49.059068+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:42.764709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046944+0000","last_change":"2024-04-17T08:50:49.039961+0000","last_active":"2024-04-17T08:50:49.046944+0000","last_peered":"2024-04-17T08:50:49.046944+0000","last_clean":"2024-04-17T08:50:49.046944+0000","last_became_active":"2024-04-17T08:50:49.039326+0000","last_became_peered":"2024-04-17T08:50:49.039326+0000","last_unstale":"2024-04-17T08:50:49.046944+0000","last_undegraded":"2024-04-17T08:50:49.046944+0000","last_fullsized":"2024-04-17T08:50:49.046944+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:43.347365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011949+0000","last_change":"2024-04-17T08:50:32.891648+0000","last_active":"2024-04-17T08:50:49.011949+0000","last_peered":"2024-04-17T08:50:49.011949+0000","last_clean":"2024-04-17T08:50:49.011949+0000","last_became_active":"2024-04-17T08:50:32.890804+0000","last_became_peered":"2024-04-17T08:50:32.890804+0000","last_unstale":"2024-04-17T08:50:49.011949+0000","last_undegraded":"2024-04-17T08:50:49.011949+0000","last_fullsized":"2024-04-17T08:50:49.011949+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:50.057538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013873+0000","last_change":"2024-04-17T08:50:33.908242+0000","last_active":"2024-04-17T08:50:49.013873+0000","last_peered":"2024-04-17T08:50:49.013873+0000","last_clean":"2024-04-17T08:50:49.013873+0000","last_became_active":"2024-04-17T08:50:33.907288+0000","last_became_peered":"2024-04-17T08:50:33.907288+0000","last_unstale":"2024-04-17T08:50:49.013873+0000","last_undegraded":"2024-04-17T08:50:49.013873+0000","last_fullsized":"2024-04-17T08:50:49.013873+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:45.950784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"233'3","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052186+0000","last_change":"2024-04-17T08:50:48.018930+0000","last_active":"2024-04-17T08:50:49.052186+0000","last_peered":"2024-04-17T08:50:49.052186+0000","last_clean":"2024-04-17T08:50:49.052186+0000","last_became_active":"2024-04-17T08:50:48.018088+0000","last_became_peered":"2024-04-17T08:50:48.018088+0000","last_unstale":"2024-04-17T08:50:49.052186+0000","last_undegraded":"2024-04-17T08:50:49.052186+0000","last_fullsized":"2024-04-17T08:50:49.052186+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:59.266629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.055303+0000","last_change":"2024-04-17T08:50:49.045041+0000","last_active":"2024-04-17T08:50:49.055303+0000","last_peered":"2024-04-17T08:50:49.055303+0000","last_clean":"2024-04-17T08:50:49.055303+0000","last_became_active":"2024-04-17T08:50:49.044767+0000","last_became_peered":"2024-04-17T08:50:49.044767+0000","last_unstale":"2024-04-17T08:50:49.055303+0000","last_undegraded":"2024-04-17T08:50:49.055303+0000","last_fullsized":"2024-04-17T08:50:49.055303+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:46:46.880409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"230'8","reported_seq":28,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028849+0000","last_change":"2024-04-17T08:50:32.906064+0000","last_active":"2024-04-17T08:50:49.028849+0000","last_peered":"2024-04-17T08:50:49.028849+0000","last_clean":"2024-04-17T08:50:49.028849+0000","last_became_active":"2024-04-17T08:50:32.904959+0000","last_became_peered":"2024-04-17T08:50:32.904959+0000","last_unstale":"2024-04-17T08:50:49.028849+0000","last_undegraded":"2024-04-17T08:50:49.028849+0000","last_fullsized":"2024-04-17T08:50:49.028849+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:55.614891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013403+0000","last_change":"2024-04-17T08:50:33.904088+0000","last_active":"2024-04-17T08:50:49.013403+0000","last_peered":"2024-04-17T08:50:49.013403+0000","last_clean":"2024-04-17T08:50:49.013403+0000","last_became_active":"2024-04-17T08:50:33.903730+0000","last_became_peered":"2024-04-17T08:50:33.903730+0000","last_unstale":"2024-04-17T08:50:49.013403+0000","last_undegraded":"2024-04-17T08:50:49.013403+0000","last_fullsized":"2024-04-17T08:50:49.013403+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:04:51.186798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019121+0000","last_change":"2024-04-17T08:50:48.021343+0000","last_active":"2024-04-17T08:50:49.019121+0000","last_peered":"2024-04-17T08:50:49.019121+0000","last_clean":"2024-04-17T08:50:49.019121+0000","last_became_active":"2024-04-17T08:50:48.020747+0000","last_became_peered":"2024-04-17T08:50:48.020747+0000","last_unstale":"2024-04-17T08:50:49.019121+0000","last_undegraded":"2024-04-17T08:50:49.019121+0000","last_fullsized":"2024-04-17T08:50:49.019121+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:55:09.798676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.030348+0000","last_change":"2024-04-17T08:50:49.028141+0000","last_active":"2024-04-17T08:50:49.030348+0000","last_peered":"2024-04-17T08:50:49.030348+0000","last_clean":"2024-04-17T08:50:49.030348+0000","last_became_active":"2024-04-17T08:50:49.027849+0000","last_became_peered":"2024-04-17T08:50:49.027849+0000","last_unstale":"2024-04-17T08:50:49.030348+0000","last_undegraded":"2024-04-17T08:50:49.030348+0000","last_fullsized":"2024-04-17T08:50:49.030348+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:50.677426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"229'3","reported_seq":23,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026153+0000","last_change":"2024-04-17T08:50:32.892710+0000","last_active":"2024-04-17T08:50:49.026153+0000","last_peered":"2024-04-17T08:50:49.026153+0000","last_clean":"2024-04-17T08:50:49.026153+0000","last_became_active":"2024-04-17T08:50:32.892204+0000","last_became_peered":"2024-04-17T08:50:32.892204+0000","last_unstale":"2024-04-17T08:50:49.026153+0000","last_undegraded":"2024-04-17T08:50:49.026153+0000","last_fullsized":"2024-04-17T08:50:49.026153+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:01.085554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024767+0000","last_change":"2024-04-17T08:50:33.900099+0000","last_active":"2024-04-17T08:50:49.024767+0000","last_peered":"2024-04-17T08:50:49.024767+0000","last_clean":"2024-04-17T08:50:49.024767+0000","last_became_active":"2024-04-17T08:50:33.899785+0000","last_became_peered":"2024-04-17T08:50:33.899785+0000","last_unstale":"2024-04-17T08:50:49.024767+0000","last_undegraded":"2024-04-17T08:50:49.024767+0000","last_fullsized":"2024-04-17T08:50:49.024767+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:22.614548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.060742+0000","last_change":"2024-04-17T08:50:48.033102+0000","last_active":"2024-04-17T08:50:49.060742+0000","last_peered":"2024-04-17T08:50:49.060742+0000","last_clean":"2024-04-17T08:50:49.060742+0000","last_became_active":"2024-04-17T08:50:48.032743+0000","last_became_peered":"2024-04-17T08:50:48.032743+0000","last_unstale":"2024-04-17T08:50:49.060742+0000","last_undegraded":"2024-04-17T08:50:49.060742+0000","last_fullsized":"2024-04-17T08:50:49.060742+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:57:08.497645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.036336+0000","last_change":"2024-04-17T08:50:49.032687+0000","last_active":"2024-04-17T08:50:49.036336+0000","last_peered":"2024-04-17T08:50:49.036336+0000","last_clean":"2024-04-17T08:50:49.036336+0000","last_became_active":"2024-04-17T08:50:49.032374+0000","last_became_peered":"2024-04-17T08:50:49.032374+0000","last_unstale":"2024-04-17T08:50:49.036336+0000","last_undegraded":"2024-04-17T08:50:49.036336+0000","last_fullsized":"2024-04-17T08:50:49.036336+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:11.499171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014273+0000","last_change":"2024-04-17T08:50:32.903765+0000","last_active":"2024-04-17T08:50:49.014273+0000","last_peered":"2024-04-17T08:50:49.014273+0000","last_clean":"2024-04-17T08:50:49.014273+0000","last_became_active":"2024-04-17T08:50:32.902284+0000","last_became_peered":"2024-04-17T08:50:32.902284+0000","last_unstale":"2024-04-17T08:50:49.014273+0000","last_undegraded":"2024-04-17T08:50:49.014273+0000","last_fullsized":"2024-04-17T08:50:49.014273+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:41.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.006177+0000","last_change":"2024-04-17T08:50:33.903308+0000","last_active":"2024-04-17T08:50:49.006177+0000","last_peered":"2024-04-17T08:50:49.006177+0000","last_clean":"2024-04-17T08:50:49.006177+0000","last_became_active":"2024-04-17T08:50:33.902988+0000","last_became_peered":"2024-04-17T08:50:33.902988+0000","last_unstale":"2024-04-17T08:50:49.006177+0000","last_undegraded":"2024-04-17T08:50:49.006177+0000","last_fullsized":"2024-04-17T08:50:49.006177+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:00.031705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"233'8","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059511+0000","last_change":"2024-04-17T08:50:48.025753+0000","last_active":"2024-04-17T08:50:49.059511+0000","last_peered":"2024-04-17T08:50:49.059511+0000","last_clean":"2024-04-17T08:50:49.059511+0000","last_became_active":"2024-04-17T08:50:48.025431+0000","last_became_peered":"2024-04-17T08:50:48.025431+0000","last_unstale":"2024-04-17T08:50:49.059511+0000","last_undegraded":"2024-04-17T08:50:49.059511+0000","last_fullsized":"2024-04-17T08:50:49.059511+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:51.729857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054770+0000","last_change":"2024-04-17T08:50:49.047279+0000","last_active":"2024-04-17T08:50:49.054770+0000","last_peered":"2024-04-17T08:50:49.054770+0000","last_clean":"2024-04-17T08:50:49.054770+0000","last_became_active":"2024-04-17T08:50:49.046698+0000","last_became_peered":"2024-04-17T08:50:49.046698+0000","last_unstale":"2024-04-17T08:50:49.054770+0000","last_undegraded":"2024-04-17T08:50:49.054770+0000","last_fullsized":"2024-04-17T08:50:49.054770+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:40:18.590004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012956+0000","last_change":"2024-04-17T08:50:32.901844+0000","last_active":"2024-04-17T08:50:49.012956+0000","last_peered":"2024-04-17T08:50:49.012956+0000","last_clean":"2024-04-17T08:50:49.012956+0000","last_became_active":"2024-04-17T08:50:32.901571+0000","last_became_peered":"2024-04-17T08:50:32.901571+0000","last_unstale":"2024-04-17T08:50:49.012956+0000","last_undegraded":"2024-04-17T08:50:49.012956+0000","last_fullsized":"2024-04-17T08:50:49.012956+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:42.263359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023967+0000","last_change":"2024-04-17T08:50:33.899408+0000","last_active":"2024-04-17T08:50:49.023967+0000","last_peered":"2024-04-17T08:50:49.023967+0000","last_clean":"2024-04-17T08:50:49.023967+0000","last_became_active":"2024-04-17T08:50:33.898922+0000","last_became_peered":"2024-04-17T08:50:33.898922+0000","last_unstale":"2024-04-17T08:50:49.023967+0000","last_undegraded":"2024-04-17T08:50:49.023967+0000","last_fullsized":"2024-04-17T08:50:49.023967+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:24.049886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014507+0000","last_change":"2024-04-17T08:50:48.019963+0000","last_active":"2024-04-17T08:50:49.014507+0000","last_peered":"2024-04-17T08:50:49.014507+0000","last_clean":"2024-04-17T08:50:49.014507+0000","last_became_active":"2024-04-17T08:50:48.017979+0000","last_became_peered":"2024-04-17T08:50:48.017979+0000","last_unstale":"2024-04-17T08:50:49.014507+0000","last_undegraded":"2024-04-17T08:50:49.014507+0000","last_fullsized":"2024-04-17T08:50:49.014507+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:57:08.190400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044790+0000","last_change":"2024-04-17T08:50:49.041753+0000","last_active":"2024-04-17T08:50:49.044790+0000","last_peered":"2024-04-17T08:50:49.044790+0000","last_clean":"2024-04-17T08:50:49.044790+0000","last_became_active":"2024-04-17T08:50:49.039959+0000","last_became_peered":"2024-04-17T08:50:49.039959+0000","last_unstale":"2024-04-17T08:50:49.044790+0000","last_undegraded":"2024-04-17T08:50:49.044790+0000","last_fullsized":"2024-04-17T08:50:49.044790+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:17:25.427318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054202+0000","last_change":"2024-04-17T08:50:48.029348+0000","last_active":"2024-04-17T08:50:49.054202+0000","last_peered":"2024-04-17T08:50:49.054202+0000","last_clean":"2024-04-17T08:50:49.054202+0000","last_became_active":"2024-04-17T08:50:48.028932+0000","last_became_peered":"2024-04-17T08:50:48.028932+0000","last_unstale":"2024-04-17T08:50:49.054202+0000","last_undegraded":"2024-04-17T08:50:49.054202+0000","last_fullsized":"2024-04-17T08:50:49.054202+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:01.151039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016020+0000","last_change":"2024-04-17T08:50:32.891454+0000","last_active":"2024-04-17T08:50:49.016020+0000","last_peered":"2024-04-17T08:50:49.016020+0000","last_clean":"2024-04-17T08:50:49.016020+0000","last_became_active":"2024-04-17T08:50:32.890756+0000","last_became_peered":"2024-04-17T08:50:32.890756+0000","last_unstale":"2024-04-17T08:50:49.016020+0000","last_undegraded":"2024-04-17T08:50:49.016020+0000","last_fullsized":"2024-04-17T08:50:49.016020+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:34.149084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023628+0000","last_change":"2024-04-17T08:50:33.897386+0000","last_active":"2024-04-17T08:50:49.023628+0000","last_peered":"2024-04-17T08:50:49.023628+0000","last_clean":"2024-04-17T08:50:49.023628+0000","last_became_active":"2024-04-17T08:50:33.897087+0000","last_became_peered":"2024-04-17T08:50:33.897087+0000","last_unstale":"2024-04-17T08:50:49.023628+0000","last_undegraded":"2024-04-17T08:50:49.023628+0000","last_fullsized":"2024-04-17T08:50:49.023628+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:32.128048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054719+0000","last_change":"2024-04-17T08:50:49.045991+0000","last_active":"2024-04-17T08:50:49.054719+0000","last_peered":"2024-04-17T08:50:49.054719+0000","last_clean":"2024-04-17T08:50:49.054719+0000","last_became_active":"2024-04-17T08:50:49.045696+0000","last_became_peered":"2024-04-17T08:50:49.045696+0000","last_unstale":"2024-04-17T08:50:49.054719+0000","last_undegraded":"2024-04-17T08:50:49.054719+0000","last_fullsized":"2024-04-17T08:50:49.054719+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:06.726698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"233'4","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054134+0000","last_change":"2024-04-17T08:50:48.027678+0000","last_active":"2024-04-17T08:50:49.054134+0000","last_peered":"2024-04-17T08:50:49.054134+0000","last_clean":"2024-04-17T08:50:49.054134+0000","last_became_active":"2024-04-17T08:50:48.027351+0000","last_became_peered":"2024-04-17T08:50:48.027351+0000","last_unstale":"2024-04-17T08:50:49.054134+0000","last_undegraded":"2024-04-17T08:50:49.054134+0000","last_fullsized":"2024-04-17T08:50:49.054134+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:35:44.730781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.015085+0000","last_change":"2024-04-17T08:50:32.900590+0000","last_active":"2024-04-17T08:50:49.015085+0000","last_peered":"2024-04-17T08:50:49.015085+0000","last_clean":"2024-04-17T08:50:49.015085+0000","last_became_active":"2024-04-17T08:50:32.900307+0000","last_became_peered":"2024-04-17T08:50:32.900307+0000","last_unstale":"2024-04-17T08:50:49.015085+0000","last_undegraded":"2024-04-17T08:50:49.015085+0000","last_fullsized":"2024-04-17T08:50:49.015085+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:45.711226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027242+0000","last_change":"2024-04-17T08:50:33.890178+0000","last_active":"2024-04-17T08:50:49.027242+0000","last_peered":"2024-04-17T08:50:49.027242+0000","last_clean":"2024-04-17T08:50:49.027242+0000","last_became_active":"2024-04-17T08:50:33.889004+0000","last_became_peered":"2024-04-17T08:50:33.889004+0000","last_unstale":"2024-04-17T08:50:49.027242+0000","last_undegraded":"2024-04-17T08:50:49.027242+0000","last_fullsized":"2024-04-17T08:50:49.027242+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:47.044851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047775+0000","last_change":"2024-04-17T08:50:49.040011+0000","last_active":"2024-04-17T08:50:49.047775+0000","last_peered":"2024-04-17T08:50:49.047775+0000","last_clean":"2024-04-17T08:50:49.047775+0000","last_became_active":"2024-04-17T08:50:49.039622+0000","last_became_peered":"2024-04-17T08:50:49.039622+0000","last_unstale":"2024-04-17T08:50:49.047775+0000","last_undegraded":"2024-04-17T08:50:49.047775+0000","last_fullsized":"2024-04-17T08:50:49.047775+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:49.754556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014140+0000","last_change":"2024-04-17T08:50:32.893112+0000","last_active":"2024-04-17T08:50:49.014140+0000","last_peered":"2024-04-17T08:50:49.014140+0000","last_clean":"2024-04-17T08:50:49.014140+0000","last_became_active":"2024-04-17T08:50:32.892793+0000","last_became_peered":"2024-04-17T08:50:32.892793+0000","last_unstale":"2024-04-17T08:50:49.014140+0000","last_undegraded":"2024-04-17T08:50:49.014140+0000","last_fullsized":"2024-04-17T08:50:49.014140+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:52.104458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021579+0000","last_change":"2024-04-17T08:50:33.896060+0000","last_active":"2024-04-17T08:50:49.021579+0000","last_peered":"2024-04-17T08:50:49.021579+0000","last_clean":"2024-04-17T08:50:49.021579+0000","last_became_active":"2024-04-17T08:50:33.895774+0000","last_became_peered":"2024-04-17T08:50:33.895774+0000","last_unstale":"2024-04-17T08:50:49.021579+0000","last_undegraded":"2024-04-17T08:50:49.021579+0000","last_fullsized":"2024-04-17T08:50:49.021579+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:41.589255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"233'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057315+0000","last_change":"2024-04-17T08:50:48.025447+0000","last_active":"2024-04-17T08:50:49.057315+0000","last_peered":"2024-04-17T08:50:49.057315+0000","last_clean":"2024-04-17T08:50:49.057315+0000","last_became_active":"2024-04-17T08:50:48.024909+0000","last_became_peered":"2024-04-17T08:50:48.024909+0000","last_unstale":"2024-04-17T08:50:49.057315+0000","last_undegraded":"2024-04-17T08:50:49.057315+0000","last_fullsized":"2024-04-17T08:50:49.057315+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:42.731156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.057791+0000","last_change":"2024-04-17T08:50:49.056907+0000","last_active":"2024-04-17T08:50:49.057791+0000","last_peered":"2024-04-17T08:50:49.057791+0000","last_clean":"2024-04-17T08:50:49.057791+0000","last_became_active":"2024-04-17T08:50:49.056553+0000","last_became_peered":"2024-04-17T08:50:49.056553+0000","last_unstale":"2024-04-17T08:50:49.057791+0000","last_undegraded":"2024-04-17T08:50:49.057791+0000","last_fullsized":"2024-04-17T08:50:49.057791+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:26.928269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027364+0000","last_change":"2024-04-17T08:50:32.903006+0000","last_active":"2024-04-17T08:50:49.027364+0000","last_peered":"2024-04-17T08:50:49.027364+0000","last_clean":"2024-04-17T08:50:49.027364+0000","last_became_active":"2024-04-17T08:50:32.902660+0000","last_became_peered":"2024-04-17T08:50:32.902660+0000","last_unstale":"2024-04-17T08:50:49.027364+0000","last_undegraded":"2024-04-17T08:50:49.027364+0000","last_fullsized":"2024-04-17T08:50:49.027364+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:57.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.023498+0000","last_change":"2024-04-17T08:50:33.904994+0000","last_active":"2024-04-17T08:50:49.023498+0000","last_peered":"2024-04-17T08:50:49.023498+0000","last_clean":"2024-04-17T08:50:49.023498+0000","last_became_active":"2024-04-17T08:50:33.904350+0000","last_became_peered":"2024-04-17T08:50:33.904350+0000","last_unstale":"2024-04-17T08:50:49.023498+0000","last_undegraded":"2024-04-17T08:50:49.023498+0000","last_fullsized":"2024-04-17T08:50:49.023498+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:38.102858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.058298+0000","last_change":"2024-04-17T08:50:48.031037+0000","last_active":"2024-04-17T08:50:49.058298+0000","last_peered":"2024-04-17T08:50:49.058298+0000","last_clean":"2024-04-17T08:50:49.058298+0000","last_became_active":"2024-04-17T08:50:48.029862+0000","last_became_peered":"2024-04-17T08:50:48.029862+0000","last_unstale":"2024-04-17T08:50:49.058298+0000","last_undegraded":"2024-04-17T08:50:49.058298+0000","last_fullsized":"2024-04-17T08:50:49.058298+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:35.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059551+0000","last_change":"2024-04-17T08:50:49.056531+0000","last_active":"2024-04-17T08:50:49.059551+0000","last_peered":"2024-04-17T08:50:49.059551+0000","last_clean":"2024-04-17T08:50:49.059551+0000","last_became_active":"2024-04-17T08:50:49.055468+0000","last_became_peered":"2024-04-17T08:50:49.055468+0000","last_unstale":"2024-04-17T08:50:49.059551+0000","last_undegraded":"2024-04-17T08:50:49.059551+0000","last_fullsized":"2024-04-17T08:50:49.059551+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:00.161509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"229'2","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005988+0000","last_change":"2024-04-17T08:50:32.903736+0000","last_active":"2024-04-17T08:50:49.005988+0000","last_peered":"2024-04-17T08:50:49.005988+0000","last_clean":"2024-04-17T08:50:49.005988+0000","last_became_active":"2024-04-17T08:50:32.903485+0000","last_became_peered":"2024-04-17T08:50:32.903485+0000","last_unstale":"2024-04-17T08:50:49.005988+0000","last_undegraded":"2024-04-17T08:50:49.005988+0000","last_fullsized":"2024-04-17T08:50:49.005988+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:27.212635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.012159+0000","last_change":"2024-04-17T08:50:33.882411+0000","last_active":"2024-04-17T08:50:49.012159+0000","last_peered":"2024-04-17T08:50:49.012159+0000","last_clean":"2024-04-17T08:50:49.012159+0000","last_became_active":"2024-04-17T08:50:33.881740+0000","last_became_peered":"2024-04-17T08:50:33.881740+0000","last_unstale":"2024-04-17T08:50:49.012159+0000","last_undegraded":"2024-04-17T08:50:49.012159+0000","last_fullsized":"2024-04-17T08:50:49.012159+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:06:34.035597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"233'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054474+0000","last_change":"2024-04-17T08:50:48.024472+0000","last_active":"2024-04-17T08:50:49.054474+0000","last_peered":"2024-04-17T08:50:49.054474+0000","last_clean":"2024-04-17T08:50:49.054474+0000","last_became_active":"2024-04-17T08:50:48.024188+0000","last_became_peered":"2024-04-17T08:50:48.024188+0000","last_unstale":"2024-04-17T08:50:49.054474+0000","last_undegraded":"2024-04-17T08:50:49.054474+0000","last_fullsized":"2024-04-17T08:50:49.054474+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:18.234057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046591+0000","last_change":"2024-04-17T08:50:49.044786+0000","last_active":"2024-04-17T08:50:49.046591+0000","last_peered":"2024-04-17T08:50:49.046591+0000","last_clean":"2024-04-17T08:50:49.046591+0000","last_became_active":"2024-04-17T08:50:49.044485+0000","last_became_peered":"2024-04-17T08:50:49.044485+0000","last_unstale":"2024-04-17T08:50:49.046591+0000","last_undegraded":"2024-04-17T08:50:49.046591+0000","last_fullsized":"2024-04-17T08:50:49.046591+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:09:49.754556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"229'1","reported_seq":21,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.020329+0000","last_change":"2024-04-17T08:50:32.904503+0000","last_active":"2024-04-17T08:50:49.020329+0000","last_peered":"2024-04-17T08:50:49.020329+0000","last_clean":"2024-04-17T08:50:49.020329+0000","last_became_active":"2024-04-17T08:50:32.904221+0000","last_became_peered":"2024-04-17T08:50:32.904221+0000","last_unstale":"2024-04-17T08:50:49.020329+0000","last_undegraded":"2024-04-17T08:50:49.020329+0000","last_fullsized":"2024-04-17T08:50:49.020329+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:54.730453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.014697+0000","last_change":"2024-04-17T08:50:33.895246+0000","last_active":"2024-04-17T08:50:49.014697+0000","last_peered":"2024-04-17T08:50:49.014697+0000","last_clean":"2024-04-17T08:50:49.014697+0000","last_became_active":"2024-04-17T08:50:33.891622+0000","last_became_peered":"2024-04-17T08:50:33.891622+0000","last_unstale":"2024-04-17T08:50:49.014697+0000","last_undegraded":"2024-04-17T08:50:49.014697+0000","last_fullsized":"2024-04-17T08:50:49.014697+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:00.894981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.022931+0000","last_change":"2024-04-17T08:50:48.022192+0000","last_active":"2024-04-17T08:50:49.022931+0000","last_peered":"2024-04-17T08:50:49.022931+0000","last_clean":"2024-04-17T08:50:49.022931+0000","last_became_active":"2024-04-17T08:50:48.020823+0000","last_became_peered":"2024-04-17T08:50:48.020823+0000","last_unstale":"2024-04-17T08:50:49.022931+0000","last_undegraded":"2024-04-17T08:50:49.022931+0000","last_fullsized":"2024-04-17T08:50:49.022931+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:30.694630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.038683+0000","last_change":"2024-04-17T08:50:49.036829+0000","last_active":"2024-04-17T08:50:49.038683+0000","last_peered":"2024-04-17T08:50:49.038683+0000","last_clean":"2024-04-17T08:50:49.038683+0000","last_became_active":"2024-04-17T08:50:49.036519+0000","last_became_peered":"2024-04-17T08:50:49.036519+0000","last_unstale":"2024-04-17T08:50:49.038683+0000","last_undegraded":"2024-04-17T08:50:49.038683+0000","last_fullsized":"2024-04-17T08:50:49.038683+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:01.572683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"229'4","reported_seq":24,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054487+0000","last_change":"2024-04-17T08:50:32.901470+0000","last_active":"2024-04-17T08:50:49.054487+0000","last_peered":"2024-04-17T08:50:49.054487+0000","last_clean":"2024-04-17T08:50:49.054487+0000","last_became_active":"2024-04-17T08:50:32.900718+0000","last_became_peered":"2024-04-17T08:50:32.900718+0000","last_unstale":"2024-04-17T08:50:49.054487+0000","last_undegraded":"2024-04-17T08:50:49.054487+0000","last_fullsized":"2024-04-17T08:50:49.054487+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:31.832070+0000","last_clean_scrub_stamp":"2024-04-17T08:50:31.832070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:32.091268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018665+0000","last_change":"2024-04-17T08:50:33.897991+0000","last_active":"2024-04-17T08:50:49.018665+0000","last_peered":"2024-04-17T08:50:49.018665+0000","last_clean":"2024-04-17T08:50:49.018665+0000","last_became_active":"2024-04-17T08:50:33.897605+0000","last_became_peered":"2024-04-17T08:50:33.897605+0000","last_unstale":"2024-04-17T08:50:49.018665+0000","last_undegraded":"2024-04-17T08:50:49.018665+0000","last_fullsized":"2024-04-17T08:50:49.018665+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:19.182944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017046+0000","last_change":"2024-04-17T08:50:48.026134+0000","last_active":"2024-04-17T08:50:49.017046+0000","last_peered":"2024-04-17T08:50:49.017046+0000","last_clean":"2024-04-17T08:50:49.017046+0000","last_became_active":"2024-04-17T08:50:48.025857+0000","last_became_peered":"2024-04-17T08:50:48.025857+0000","last_unstale":"2024-04-17T08:50:49.017046+0000","last_undegraded":"2024-04-17T08:50:49.017046+0000","last_fullsized":"2024-04-17T08:50:49.017046+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:46.957384+0000","last_clean_scrub_stamp":"2024-04-17T08:50:46.957384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:40:33.236834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028676+0000","last_change":"2024-04-17T08:50:49.024224+0000","last_active":"2024-04-17T08:50:49.028676+0000","last_peered":"2024-04-17T08:50:49.028676+0000","last_clean":"2024-04-17T08:50:49.028676+0000","last_became_active":"2024-04-17T08:50:49.023932+0000","last_became_peered":"2024-04-17T08:50:49.023932+0000","last_unstale":"2024-04-17T08:50:49.028676+0000","last_undegraded":"2024-04-17T08:50:49.028676+0000","last_fullsized":"2024-04-17T08:50:49.028676+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:32.580750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018304+0000","last_change":"2024-04-17T08:50:33.897001+0000","last_active":"2024-04-17T08:50:49.018304+0000","last_peered":"2024-04-17T08:50:49.018304+0000","last_clean":"2024-04-17T08:50:49.018304+0000","last_became_active":"2024-04-17T08:50:33.896545+0000","last_became_peered":"2024-04-17T08:50:33.896545+0000","last_unstale":"2024-04-17T08:50:49.018304+0000","last_undegraded":"2024-04-17T08:50:49.018304+0000","last_fullsized":"2024-04-17T08:50:49.018304+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:59:52.128789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045100+0000","last_change":"2024-04-17T08:50:49.042008+0000","last_active":"2024-04-17T08:50:49.045100+0000","last_peered":"2024-04-17T08:50:49.045100+0000","last_clean":"2024-04-17T08:50:49.045100+0000","last_became_active":"2024-04-17T08:50:49.041386+0000","last_became_peered":"2024-04-17T08:50:49.041386+0000","last_unstale":"2024-04-17T08:50:49.045100+0000","last_undegraded":"2024-04-17T08:50:49.045100+0000","last_fullsized":"2024-04-17T08:50:49.045100+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:42.544919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.025232+0000","last_change":"2024-04-17T08:50:33.890500+0000","last_active":"2024-04-17T08:50:49.025232+0000","last_peered":"2024-04-17T08:50:49.025232+0000","last_clean":"2024-04-17T08:50:49.025232+0000","last_became_active":"2024-04-17T08:50:33.890183+0000","last_became_peered":"2024-04-17T08:50:33.890183+0000","last_unstale":"2024-04-17T08:50:49.025232+0000","last_undegraded":"2024-04-17T08:50:49.025232+0000","last_fullsized":"2024-04-17T08:50:49.025232+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:30.472528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.037845+0000","last_change":"2024-04-17T08:50:49.035117+0000","last_active":"2024-04-17T08:50:49.037845+0000","last_peered":"2024-04-17T08:50:49.037845+0000","last_clean":"2024-04-17T08:50:49.037845+0000","last_became_active":"2024-04-17T08:50:49.034818+0000","last_became_peered":"2024-04-17T08:50:49.034818+0000","last_unstale":"2024-04-17T08:50:49.037845+0000","last_undegraded":"2024-04-17T08:50:49.037845+0000","last_fullsized":"2024-04-17T08:50:49.037845+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:33:40.533532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016895+0000","last_change":"2024-04-17T08:50:33.895564+0000","last_active":"2024-04-17T08:50:49.016895+0000","last_peered":"2024-04-17T08:50:49.016895+0000","last_clean":"2024-04-17T08:50:49.016895+0000","last_became_active":"2024-04-17T08:50:33.893775+0000","last_became_peered":"2024-04-17T08:50:33.893775+0000","last_unstale":"2024-04-17T08:50:49.016895+0000","last_undegraded":"2024-04-17T08:50:49.016895+0000","last_fullsized":"2024-04-17T08:50:49.016895+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:02:42.408577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052152+0000","last_change":"2024-04-17T08:50:49.049627+0000","last_active":"2024-04-17T08:50:49.052152+0000","last_peered":"2024-04-17T08:50:49.052152+0000","last_clean":"2024-04-17T08:50:49.052152+0000","last_became_active":"2024-04-17T08:50:49.049314+0000","last_became_peered":"2024-04-17T08:50:49.049314+0000","last_unstale":"2024-04-17T08:50:49.052152+0000","last_undegraded":"2024-04-17T08:50:49.052152+0000","last_fullsized":"2024-04-17T08:50:49.052152+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:07.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048859+0000","last_change":"2024-04-17T08:50:33.899268+0000","last_active":"2024-04-17T08:50:49.048859+0000","last_peered":"2024-04-17T08:50:49.048859+0000","last_clean":"2024-04-17T08:50:49.048859+0000","last_became_active":"2024-04-17T08:50:33.898733+0000","last_became_peered":"2024-04-17T08:50:33.898733+0000","last_unstale":"2024-04-17T08:50:49.048859+0000","last_undegraded":"2024-04-17T08:50:49.048859+0000","last_fullsized":"2024-04-17T08:50:49.048859+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:24.183129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059529+0000","last_change":"2024-04-17T08:50:49.056280+0000","last_active":"2024-04-17T08:50:49.059529+0000","last_peered":"2024-04-17T08:50:49.059529+0000","last_clean":"2024-04-17T08:50:49.059529+0000","last_became_active":"2024-04-17T08:50:49.055717+0000","last_became_peered":"2024-04-17T08:50:49.055717+0000","last_unstale":"2024-04-17T08:50:49.059529+0000","last_undegraded":"2024-04-17T08:50:49.059529+0000","last_fullsized":"2024-04-17T08:50:49.059529+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:40:21.628524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.004125+0000","last_change":"2024-04-17T08:50:33.897631+0000","last_active":"2024-04-17T08:50:49.004125+0000","last_peered":"2024-04-17T08:50:49.004125+0000","last_clean":"2024-04-17T08:50:49.004125+0000","last_became_active":"2024-04-17T08:50:33.897328+0000","last_became_peered":"2024-04-17T08:50:33.897328+0000","last_unstale":"2024-04-17T08:50:49.004125+0000","last_undegraded":"2024-04-17T08:50:49.004125+0000","last_fullsized":"2024-04-17T08:50:49.004125+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:10:47.708966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.059847+0000","last_change":"2024-04-17T08:50:49.057574+0000","last_active":"2024-04-17T08:50:49.059847+0000","last_peered":"2024-04-17T08:50:49.059847+0000","last_clean":"2024-04-17T08:50:49.059847+0000","last_became_active":"2024-04-17T08:50:49.057111+0000","last_became_peered":"2024-04-17T08:50:49.057111+0000","last_unstale":"2024-04-17T08:50:49.059847+0000","last_undegraded":"2024-04-17T08:50:49.059847+0000","last_fullsized":"2024-04-17T08:50:49.059847+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:13.816186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053498+0000","last_change":"2024-04-17T08:50:33.897365+0000","last_active":"2024-04-17T08:50:49.053498+0000","last_peered":"2024-04-17T08:50:49.053498+0000","last_clean":"2024-04-17T08:50:49.053498+0000","last_became_active":"2024-04-17T08:50:33.897012+0000","last_became_peered":"2024-04-17T08:50:33.897012+0000","last_unstale":"2024-04-17T08:50:49.053498+0000","last_undegraded":"2024-04-17T08:50:49.053498+0000","last_fullsized":"2024-04-17T08:50:49.053498+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:35.090017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.044615+0000","last_change":"2024-04-17T08:50:49.043131+0000","last_active":"2024-04-17T08:50:49.044615+0000","last_peered":"2024-04-17T08:50:49.044615+0000","last_clean":"2024-04-17T08:50:49.044615+0000","last_became_active":"2024-04-17T08:50:49.042828+0000","last_became_peered":"2024-04-17T08:50:49.042828+0000","last_unstale":"2024-04-17T08:50:49.044615+0000","last_undegraded":"2024-04-17T08:50:49.044615+0000","last_fullsized":"2024-04-17T08:50:49.044615+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:29:18.298255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016238+0000","last_change":"2024-04-17T08:50:33.879179+0000","last_active":"2024-04-17T08:50:49.016238+0000","last_peered":"2024-04-17T08:50:49.016238+0000","last_clean":"2024-04-17T08:50:49.016238+0000","last_became_active":"2024-04-17T08:50:33.878412+0000","last_became_peered":"2024-04-17T08:50:33.878412+0000","last_unstale":"2024-04-17T08:50:49.016238+0000","last_undegraded":"2024-04-17T08:50:49.016238+0000","last_fullsized":"2024-04-17T08:50:49.016238+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:44:39.773357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045034+0000","last_change":"2024-04-17T08:50:49.038692+0000","last_active":"2024-04-17T08:50:49.045034+0000","last_peered":"2024-04-17T08:50:49.045034+0000","last_clean":"2024-04-17T08:50:49.045034+0000","last_became_active":"2024-04-17T08:50:49.038427+0000","last_became_peered":"2024-04-17T08:50:49.038427+0000","last_unstale":"2024-04-17T08:50:49.045034+0000","last_undegraded":"2024-04-17T08:50:49.045034+0000","last_fullsized":"2024-04-17T08:50:49.045034+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:31.479027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018877+0000","last_change":"2024-04-17T08:50:33.893295+0000","last_active":"2024-04-17T08:50:49.018877+0000","last_peered":"2024-04-17T08:50:49.018877+0000","last_clean":"2024-04-17T08:50:49.018877+0000","last_became_active":"2024-04-17T08:50:33.892999+0000","last_became_peered":"2024-04-17T08:50:33.892999+0000","last_unstale":"2024-04-17T08:50:49.018877+0000","last_undegraded":"2024-04-17T08:50:49.018877+0000","last_fullsized":"2024-04-17T08:50:49.018877+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:46.730352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046175+0000","last_change":"2024-04-17T08:50:49.045161+0000","last_active":"2024-04-17T08:50:49.046175+0000","last_peered":"2024-04-17T08:50:49.046175+0000","last_clean":"2024-04-17T08:50:49.046175+0000","last_became_active":"2024-04-17T08:50:49.043011+0000","last_became_peered":"2024-04-17T08:50:49.043011+0000","last_unstale":"2024-04-17T08:50:49.046175+0000","last_undegraded":"2024-04-17T08:50:49.046175+0000","last_fullsized":"2024-04-17T08:50:49.046175+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:49.937382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028246+0000","last_change":"2024-04-17T08:50:33.900938+0000","last_active":"2024-04-17T08:50:49.028246+0000","last_peered":"2024-04-17T08:50:49.028246+0000","last_clean":"2024-04-17T08:50:49.028246+0000","last_became_active":"2024-04-17T08:50:33.900085+0000","last_became_peered":"2024-04-17T08:50:33.900085+0000","last_unstale":"2024-04-17T08:50:49.028246+0000","last_undegraded":"2024-04-17T08:50:49.028246+0000","last_fullsized":"2024-04-17T08:50:49.028246+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:10:31.107767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034247+0000","last_change":"2024-04-17T08:50:49.026386+0000","last_active":"2024-04-17T08:50:49.034247+0000","last_peered":"2024-04-17T08:50:49.034247+0000","last_clean":"2024-04-17T08:50:49.034247+0000","last_became_active":"2024-04-17T08:50:49.025867+0000","last_became_peered":"2024-04-17T08:50:49.025867+0000","last_unstale":"2024-04-17T08:50:49.034247+0000","last_undegraded":"2024-04-17T08:50:49.034247+0000","last_fullsized":"2024-04-17T08:50:49.034247+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:28.747322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.010867+0000","last_change":"2024-04-17T08:50:33.890185+0000","last_active":"2024-04-17T08:50:49.010867+0000","last_peered":"2024-04-17T08:50:49.010867+0000","last_clean":"2024-04-17T08:50:49.010867+0000","last_became_active":"2024-04-17T08:50:33.889133+0000","last_became_peered":"2024-04-17T08:50:33.889133+0000","last_unstale":"2024-04-17T08:50:49.010867+0000","last_undegraded":"2024-04-17T08:50:49.010867+0000","last_fullsized":"2024-04-17T08:50:49.010867+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:36.496412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.033596+0000","last_change":"2024-04-17T08:50:49.029148+0000","last_active":"2024-04-17T08:50:49.033596+0000","last_peered":"2024-04-17T08:50:49.033596+0000","last_clean":"2024-04-17T08:50:49.033596+0000","last_became_active":"2024-04-17T08:50:49.028844+0000","last_became_peered":"2024-04-17T08:50:49.028844+0000","last_unstale":"2024-04-17T08:50:49.033596+0000","last_undegraded":"2024-04-17T08:50:49.033596+0000","last_fullsized":"2024-04-17T08:50:49.033596+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:56.712515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.011837+0000","last_change":"2024-04-17T08:50:33.905971+0000","last_active":"2024-04-17T08:50:49.011837+0000","last_peered":"2024-04-17T08:50:49.011837+0000","last_clean":"2024-04-17T08:50:49.011837+0000","last_became_active":"2024-04-17T08:50:33.905591+0000","last_became_peered":"2024-04-17T08:50:33.905591+0000","last_unstale":"2024-04-17T08:50:49.011837+0000","last_undegraded":"2024-04-17T08:50:49.011837+0000","last_fullsized":"2024-04-17T08:50:49.011837+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:32.834294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047893+0000","last_change":"2024-04-17T08:50:49.042802+0000","last_active":"2024-04-17T08:50:49.047893+0000","last_peered":"2024-04-17T08:50:49.047893+0000","last_clean":"2024-04-17T08:50:49.047893+0000","last_became_active":"2024-04-17T08:50:49.041010+0000","last_became_peered":"2024-04-17T08:50:49.041010+0000","last_unstale":"2024-04-17T08:50:49.047893+0000","last_undegraded":"2024-04-17T08:50:49.047893+0000","last_fullsized":"2024-04-17T08:50:49.047893+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:46.224935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.021294+0000","last_change":"2024-04-17T08:50:33.904610+0000","last_active":"2024-04-17T08:50:49.021294+0000","last_peered":"2024-04-17T08:50:49.021294+0000","last_clean":"2024-04-17T08:50:49.021294+0000","last_became_active":"2024-04-17T08:50:33.904287+0000","last_became_peered":"2024-04-17T08:50:33.904287+0000","last_unstale":"2024-04-17T08:50:49.021294+0000","last_undegraded":"2024-04-17T08:50:49.021294+0000","last_fullsized":"2024-04-17T08:50:49.021294+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:14:28.658522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.030321+0000","last_change":"2024-04-17T08:50:49.027296+0000","last_active":"2024-04-17T08:50:49.030321+0000","last_peered":"2024-04-17T08:50:49.030321+0000","last_clean":"2024-04-17T08:50:49.030321+0000","last_became_active":"2024-04-17T08:50:49.025787+0000","last_became_peered":"2024-04-17T08:50:49.025787+0000","last_unstale":"2024-04-17T08:50:49.030321+0000","last_undegraded":"2024-04-17T08:50:49.030321+0000","last_fullsized":"2024-04-17T08:50:49.030321+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:51.381129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024543+0000","last_change":"2024-04-17T08:50:33.907557+0000","last_active":"2024-04-17T08:50:49.024543+0000","last_peered":"2024-04-17T08:50:49.024543+0000","last_clean":"2024-04-17T08:50:49.024543+0000","last_became_active":"2024-04-17T08:50:33.907258+0000","last_became_peered":"2024-04-17T08:50:33.907258+0000","last_unstale":"2024-04-17T08:50:49.024543+0000","last_undegraded":"2024-04-17T08:50:49.024543+0000","last_fullsized":"2024-04-17T08:50:49.024543+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:40:19.130647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.046245+0000","last_change":"2024-04-17T08:50:49.042577+0000","last_active":"2024-04-17T08:50:49.046245+0000","last_peered":"2024-04-17T08:50:49.046245+0000","last_clean":"2024-04-17T08:50:49.046245+0000","last_became_active":"2024-04-17T08:50:49.039823+0000","last_became_peered":"2024-04-17T08:50:49.039823+0000","last_unstale":"2024-04-17T08:50:49.046245+0000","last_undegraded":"2024-04-17T08:50:49.046245+0000","last_fullsized":"2024-04-17T08:50:49.046245+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:03.547181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026027+0000","last_change":"2024-04-17T08:50:33.907077+0000","last_active":"2024-04-17T08:50:49.026027+0000","last_peered":"2024-04-17T08:50:49.026027+0000","last_clean":"2024-04-17T08:50:49.026027+0000","last_became_active":"2024-04-17T08:50:33.906771+0000","last_became_peered":"2024-04-17T08:50:33.906771+0000","last_unstale":"2024-04-17T08:50:49.026027+0000","last_undegraded":"2024-04-17T08:50:49.026027+0000","last_fullsized":"2024-04-17T08:50:49.026027+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:34.659543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034592+0000","last_change":"2024-04-17T08:50:49.031610+0000","last_active":"2024-04-17T08:50:49.034592+0000","last_peered":"2024-04-17T08:50:49.034592+0000","last_clean":"2024-04-17T08:50:49.034592+0000","last_became_active":"2024-04-17T08:50:49.031310+0000","last_became_peered":"2024-04-17T08:50:49.031310+0000","last_unstale":"2024-04-17T08:50:49.034592+0000","last_undegraded":"2024-04-17T08:50:49.034592+0000","last_fullsized":"2024-04-17T08:50:49.034592+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:27.833812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026990+0000","last_change":"2024-04-17T08:50:33.905082+0000","last_active":"2024-04-17T08:50:49.026990+0000","last_peered":"2024-04-17T08:50:49.026990+0000","last_clean":"2024-04-17T08:50:49.026990+0000","last_became_active":"2024-04-17T08:50:33.904723+0000","last_became_peered":"2024-04-17T08:50:33.904723+0000","last_unstale":"2024-04-17T08:50:49.026990+0000","last_undegraded":"2024-04-17T08:50:49.026990+0000","last_fullsized":"2024-04-17T08:50:49.026990+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:10.492335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.039376+0000","last_change":"2024-04-17T08:50:49.034923+0000","last_active":"2024-04-17T08:50:49.039376+0000","last_peered":"2024-04-17T08:50:49.039376+0000","last_clean":"2024-04-17T08:50:49.039376+0000","last_became_active":"2024-04-17T08:50:49.034509+0000","last_became_peered":"2024-04-17T08:50:49.034509+0000","last_unstale":"2024-04-17T08:50:49.039376+0000","last_undegraded":"2024-04-17T08:50:49.039376+0000","last_fullsized":"2024-04-17T08:50:49.039376+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:17.060962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.017084+0000","last_change":"2024-04-17T08:50:33.888622+0000","last_active":"2024-04-17T08:50:49.017084+0000","last_peered":"2024-04-17T08:50:49.017084+0000","last_clean":"2024-04-17T08:50:49.017084+0000","last_became_active":"2024-04-17T08:50:33.888298+0000","last_became_peered":"2024-04-17T08:50:33.888298+0000","last_unstale":"2024-04-17T08:50:49.017084+0000","last_undegraded":"2024-04-17T08:50:49.017084+0000","last_fullsized":"2024-04-17T08:50:49.017084+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:48.390987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.036139+0000","last_change":"2024-04-17T08:50:49.030470+0000","last_active":"2024-04-17T08:50:49.036139+0000","last_peered":"2024-04-17T08:50:49.036139+0000","last_clean":"2024-04-17T08:50:49.036139+0000","last_became_active":"2024-04-17T08:50:49.030193+0000","last_became_peered":"2024-04-17T08:50:49.030193+0000","last_unstale":"2024-04-17T08:50:49.036139+0000","last_undegraded":"2024-04-17T08:50:49.036139+0000","last_fullsized":"2024-04-17T08:50:49.036139+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:33:39.173146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.022998+0000","last_change":"2024-04-17T08:50:33.895366+0000","last_active":"2024-04-17T08:50:49.022998+0000","last_peered":"2024-04-17T08:50:49.022998+0000","last_clean":"2024-04-17T08:50:49.022998+0000","last_became_active":"2024-04-17T08:50:33.892102+0000","last_became_peered":"2024-04-17T08:50:33.892102+0000","last_unstale":"2024-04-17T08:50:49.022998+0000","last_undegraded":"2024-04-17T08:50:49.022998+0000","last_fullsized":"2024-04-17T08:50:49.022998+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:21:32.764250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.054586+0000","last_change":"2024-04-17T08:50:49.047250+0000","last_active":"2024-04-17T08:50:49.054586+0000","last_peered":"2024-04-17T08:50:49.054586+0000","last_clean":"2024-04-17T08:50:49.054586+0000","last_became_active":"2024-04-17T08:50:49.046458+0000","last_became_peered":"2024-04-17T08:50:49.046458+0000","last_unstale":"2024-04-17T08:50:49.054586+0000","last_undegraded":"2024-04-17T08:50:49.054586+0000","last_fullsized":"2024-04-17T08:50:49.054586+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:06.199924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019502+0000","last_change":"2024-04-17T08:50:33.898186+0000","last_active":"2024-04-17T08:50:49.019502+0000","last_peered":"2024-04-17T08:50:49.019502+0000","last_clean":"2024-04-17T08:50:49.019502+0000","last_became_active":"2024-04-17T08:50:33.897746+0000","last_became_peered":"2024-04-17T08:50:33.897746+0000","last_unstale":"2024-04-17T08:50:49.019502+0000","last_undegraded":"2024-04-17T08:50:49.019502+0000","last_fullsized":"2024-04-17T08:50:49.019502+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:01:11.440017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040121+0000","last_change":"2024-04-17T08:50:49.037933+0000","last_active":"2024-04-17T08:50:49.040121+0000","last_peered":"2024-04-17T08:50:49.040121+0000","last_clean":"2024-04-17T08:50:49.040121+0000","last_became_active":"2024-04-17T08:50:49.035919+0000","last_became_peered":"2024-04-17T08:50:49.035919+0000","last_unstale":"2024-04-17T08:50:49.040121+0000","last_undegraded":"2024-04-17T08:50:49.040121+0000","last_fullsized":"2024-04-17T08:50:49.040121+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:28:43.347365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018729+0000","last_change":"2024-04-17T08:50:33.887226+0000","last_active":"2024-04-17T08:50:49.018729+0000","last_peered":"2024-04-17T08:50:49.018729+0000","last_clean":"2024-04-17T08:50:49.018729+0000","last_became_active":"2024-04-17T08:50:33.886912+0000","last_became_peered":"2024-04-17T08:50:33.886912+0000","last_unstale":"2024-04-17T08:50:49.018729+0000","last_undegraded":"2024-04-17T08:50:49.018729+0000","last_fullsized":"2024-04-17T08:50:49.018729+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:59.708199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.050946+0000","last_change":"2024-04-17T08:50:49.049142+0000","last_active":"2024-04-17T08:50:49.050946+0000","last_peered":"2024-04-17T08:50:49.050946+0000","last_clean":"2024-04-17T08:50:49.050946+0000","last_became_active":"2024-04-17T08:50:49.048696+0000","last_became_peered":"2024-04-17T08:50:49.048696+0000","last_unstale":"2024-04-17T08:50:49.050946+0000","last_undegraded":"2024-04-17T08:50:49.050946+0000","last_fullsized":"2024-04-17T08:50:49.050946+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:30.138141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.048893+0000","last_change":"2024-04-17T08:50:33.894398+0000","last_active":"2024-04-17T08:50:49.048893+0000","last_peered":"2024-04-17T08:50:49.048893+0000","last_clean":"2024-04-17T08:50:49.048893+0000","last_became_active":"2024-04-17T08:50:33.894105+0000","last_became_peered":"2024-04-17T08:50:33.894105+0000","last_unstale":"2024-04-17T08:50:49.048893+0000","last_undegraded":"2024-04-17T08:50:49.048893+0000","last_fullsized":"2024-04-17T08:50:49.048893+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:00.398625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.034049+0000","last_change":"2024-04-17T08:50:49.025092+0000","last_active":"2024-04-17T08:50:49.034049+0000","last_peered":"2024-04-17T08:50:49.034049+0000","last_clean":"2024-04-17T08:50:49.034049+0000","last_became_active":"2024-04-17T08:50:49.023955+0000","last_became_peered":"2024-04-17T08:50:49.023955+0000","last_unstale":"2024-04-17T08:50:49.034049+0000","last_undegraded":"2024-04-17T08:50:49.034049+0000","last_fullsized":"2024-04-17T08:50:49.034049+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:01.074044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053008+0000","last_change":"2024-04-17T08:50:33.897953+0000","last_active":"2024-04-17T08:50:49.053008+0000","last_peered":"2024-04-17T08:50:49.053008+0000","last_clean":"2024-04-17T08:50:49.053008+0000","last_became_active":"2024-04-17T08:50:33.897591+0000","last_became_peered":"2024-04-17T08:50:33.897591+0000","last_unstale":"2024-04-17T08:50:49.053008+0000","last_undegraded":"2024-04-17T08:50:49.053008+0000","last_fullsized":"2024-04-17T08:50:49.053008+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:57.978668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040118+0000","last_change":"2024-04-17T08:50:49.031090+0000","last_active":"2024-04-17T08:50:49.040118+0000","last_peered":"2024-04-17T08:50:49.040118+0000","last_clean":"2024-04-17T08:50:49.040118+0000","last_became_active":"2024-04-17T08:50:49.028551+0000","last_became_peered":"2024-04-17T08:50:49.028551+0000","last_unstale":"2024-04-17T08:50:49.040118+0000","last_undegraded":"2024-04-17T08:50:49.040118+0000","last_fullsized":"2024-04-17T08:50:49.040118+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:28.747322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005717+0000","last_change":"2024-04-17T08:50:33.897057+0000","last_active":"2024-04-17T08:50:49.005717+0000","last_peered":"2024-04-17T08:50:49.005717+0000","last_clean":"2024-04-17T08:50:49.005717+0000","last_became_active":"2024-04-17T08:50:33.894722+0000","last_became_peered":"2024-04-17T08:50:33.894722+0000","last_unstale":"2024-04-17T08:50:49.005717+0000","last_undegraded":"2024-04-17T08:50:49.005717+0000","last_fullsized":"2024-04-17T08:50:49.005717+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:09.971845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.040250+0000","last_change":"2024-04-17T08:50:49.036087+0000","last_active":"2024-04-17T08:50:49.040250+0000","last_peered":"2024-04-17T08:50:49.040250+0000","last_clean":"2024-04-17T08:50:49.040250+0000","last_became_active":"2024-04-17T08:50:49.035555+0000","last_became_peered":"2024-04-17T08:50:49.035555+0000","last_unstale":"2024-04-17T08:50:49.040250+0000","last_undegraded":"2024-04-17T08:50:49.040250+0000","last_fullsized":"2024-04-17T08:50:49.040250+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:11.499171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.026883+0000","last_change":"2024-04-17T08:50:33.895456+0000","last_active":"2024-04-17T08:50:49.026883+0000","last_peered":"2024-04-17T08:50:49.026883+0000","last_clean":"2024-04-17T08:50:49.026883+0000","last_became_active":"2024-04-17T08:50:33.895142+0000","last_became_peered":"2024-04-17T08:50:33.895142+0000","last_unstale":"2024-04-17T08:50:49.026883+0000","last_undegraded":"2024-04-17T08:50:49.026883+0000","last_fullsized":"2024-04-17T08:50:49.026883+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:33:58.168170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.051397+0000","last_change":"2024-04-17T08:50:49.048928+0000","last_active":"2024-04-17T08:50:49.051397+0000","last_peered":"2024-04-17T08:50:49.051397+0000","last_clean":"2024-04-17T08:50:49.051397+0000","last_became_active":"2024-04-17T08:50:49.045972+0000","last_became_peered":"2024-04-17T08:50:49.045972+0000","last_unstale":"2024-04-17T08:50:49.051397+0000","last_undegraded":"2024-04-17T08:50:49.051397+0000","last_fullsized":"2024-04-17T08:50:49.051397+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:31.057634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.019312+0000","last_change":"2024-04-17T08:50:33.894702+0000","last_active":"2024-04-17T08:50:49.019312+0000","last_peered":"2024-04-17T08:50:49.019312+0000","last_clean":"2024-04-17T08:50:49.019312+0000","last_became_active":"2024-04-17T08:50:33.894398+0000","last_became_peered":"2024-04-17T08:50:33.894398+0000","last_unstale":"2024-04-17T08:50:49.019312+0000","last_undegraded":"2024-04-17T08:50:49.019312+0000","last_fullsized":"2024-04-17T08:50:49.019312+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:32.834294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.047619+0000","last_change":"2024-04-17T08:50:49.045976+0000","last_active":"2024-04-17T08:50:49.047619+0000","last_peered":"2024-04-17T08:50:49.047619+0000","last_clean":"2024-04-17T08:50:49.047619+0000","last_became_active":"2024-04-17T08:50:49.044919+0000","last_became_peered":"2024-04-17T08:50:49.044919+0000","last_unstale":"2024-04-17T08:50:49.047619+0000","last_undegraded":"2024-04-17T08:50:49.047619+0000","last_fullsized":"2024-04-17T08:50:49.047619+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:01.572833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.027129+0000","last_change":"2024-04-17T08:50:33.895419+0000","last_active":"2024-04-17T08:50:49.027129+0000","last_peered":"2024-04-17T08:50:49.027129+0000","last_clean":"2024-04-17T08:50:49.027129+0000","last_became_active":"2024-04-17T08:50:33.894089+0000","last_became_peered":"2024-04-17T08:50:33.894089+0000","last_unstale":"2024-04-17T08:50:49.027129+0000","last_undegraded":"2024-04-17T08:50:49.027129+0000","last_fullsized":"2024-04-17T08:50:49.027129+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:00:46.043729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.038318+0000","last_change":"2024-04-17T08:50:49.037732+0000","last_active":"2024-04-17T08:50:49.038318+0000","last_peered":"2024-04-17T08:50:49.038318+0000","last_clean":"2024-04-17T08:50:49.038318+0000","last_became_active":"2024-04-17T08:50:49.037397+0000","last_became_peered":"2024-04-17T08:50:49.037397+0000","last_unstale":"2024-04-17T08:50:49.038318+0000","last_undegraded":"2024-04-17T08:50:49.038318+0000","last_fullsized":"2024-04-17T08:50:49.038318+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:18.884898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018150+0000","last_change":"2024-04-17T08:50:33.880798+0000","last_active":"2024-04-17T08:50:49.018150+0000","last_peered":"2024-04-17T08:50:49.018150+0000","last_clean":"2024-04-17T08:50:49.018150+0000","last_became_active":"2024-04-17T08:50:33.879829+0000","last_became_peered":"2024-04-17T08:50:33.879829+0000","last_unstale":"2024-04-17T08:50:49.018150+0000","last_undegraded":"2024-04-17T08:50:49.018150+0000","last_fullsized":"2024-04-17T08:50:49.018150+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:38:46.264842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.045908+0000","last_change":"2024-04-17T08:50:49.042026+0000","last_active":"2024-04-17T08:50:49.045908+0000","last_peered":"2024-04-17T08:50:49.045908+0000","last_clean":"2024-04-17T08:50:49.045908+0000","last_became_active":"2024-04-17T08:50:49.041698+0000","last_became_peered":"2024-04-17T08:50:49.041698+0000","last_unstale":"2024-04-17T08:50:49.045908+0000","last_undegraded":"2024-04-17T08:50:49.045908+0000","last_fullsized":"2024-04-17T08:50:49.045908+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:01.822353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.013614+0000","last_change":"2024-04-17T08:50:33.900220+0000","last_active":"2024-04-17T08:50:49.013614+0000","last_peered":"2024-04-17T08:50:49.013614+0000","last_clean":"2024-04-17T08:50:49.013614+0000","last_became_active":"2024-04-17T08:50:33.898746+0000","last_became_peered":"2024-04-17T08:50:33.898746+0000","last_unstale":"2024-04-17T08:50:49.013614+0000","last_undegraded":"2024-04-17T08:50:49.013614+0000","last_fullsized":"2024-04-17T08:50:49.013614+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:37.623670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.053243+0000","last_change":"2024-04-17T08:50:49.042525+0000","last_active":"2024-04-17T08:50:49.053243+0000","last_peered":"2024-04-17T08:50:49.053243+0000","last_clean":"2024-04-17T08:50:49.053243+0000","last_became_active":"2024-04-17T08:50:49.042215+0000","last_became_peered":"2024-04-17T08:50:49.042215+0000","last_unstale":"2024-04-17T08:50:49.053243+0000","last_undegraded":"2024-04-17T08:50:49.053243+0000","last_fullsized":"2024-04-17T08:50:49.053243+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:24:06.467500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024343+0000","last_change":"2024-04-17T08:50:33.908217+0000","last_active":"2024-04-17T08:50:49.024343+0000","last_peered":"2024-04-17T08:50:49.024343+0000","last_clean":"2024-04-17T08:50:49.024343+0000","last_became_active":"2024-04-17T08:50:33.907185+0000","last_became_peered":"2024-04-17T08:50:33.907185+0000","last_unstale":"2024-04-17T08:50:49.024343+0000","last_undegraded":"2024-04-17T08:50:49.024343+0000","last_fullsized":"2024-04-17T08:50:49.024343+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:28.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.039577+0000","last_change":"2024-04-17T08:50:49.037576+0000","last_active":"2024-04-17T08:50:49.039577+0000","last_peered":"2024-04-17T08:50:49.039577+0000","last_clean":"2024-04-17T08:50:49.039577+0000","last_became_active":"2024-04-17T08:50:49.037277+0000","last_became_peered":"2024-04-17T08:50:49.037277+0000","last_unstale":"2024-04-17T08:50:49.039577+0000","last_undegraded":"2024-04-17T08:50:49.039577+0000","last_fullsized":"2024-04-17T08:50:49.039577+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:30:52.677467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.024562+0000","last_change":"2024-04-17T08:50:33.896077+0000","last_active":"2024-04-17T08:50:49.024562+0000","last_peered":"2024-04-17T08:50:49.024562+0000","last_clean":"2024-04-17T08:50:49.024562+0000","last_became_active":"2024-04-17T08:50:33.895408+0000","last_became_peered":"2024-04-17T08:50:33.895408+0000","last_unstale":"2024-04-17T08:50:49.024562+0000","last_undegraded":"2024-04-17T08:50:49.024562+0000","last_fullsized":"2024-04-17T08:50:49.024562+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:16.588443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.042763+0000","last_change":"2024-04-17T08:50:49.035395+0000","last_active":"2024-04-17T08:50:49.042763+0000","last_peered":"2024-04-17T08:50:49.042763+0000","last_clean":"2024-04-17T08:50:49.042763+0000","last_became_active":"2024-04-17T08:50:49.033291+0000","last_became_peered":"2024-04-17T08:50:49.033291+0000","last_unstale":"2024-04-17T08:50:49.042763+0000","last_undegraded":"2024-04-17T08:50:49.042763+0000","last_fullsized":"2024-04-17T08:50:49.042763+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:06:00.266799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.005413+0000","last_change":"2024-04-17T08:50:33.900448+0000","last_active":"2024-04-17T08:50:49.005413+0000","last_peered":"2024-04-17T08:50:49.005413+0000","last_clean":"2024-04-17T08:50:49.005413+0000","last_became_active":"2024-04-17T08:50:33.900142+0000","last_became_peered":"2024-04-17T08:50:33.900142+0000","last_unstale":"2024-04-17T08:50:49.005413+0000","last_undegraded":"2024-04-17T08:50:49.005413+0000","last_fullsized":"2024-04-17T08:50:49.005413+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:37:31.446411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.028607+0000","last_change":"2024-04-17T08:50:49.023214+0000","last_active":"2024-04-17T08:50:49.028607+0000","last_peered":"2024-04-17T08:50:49.028607+0000","last_clean":"2024-04-17T08:50:49.028607+0000","last_became_active":"2024-04-17T08:50:49.022785+0000","last_became_peered":"2024-04-17T08:50:49.022785+0000","last_unstale":"2024-04-17T08:50:49.028607+0000","last_undegraded":"2024-04-17T08:50:49.028607+0000","last_fullsized":"2024-04-17T08:50:49.028607+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:35.493434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.018542+0000","last_change":"2024-04-17T08:50:33.883908+0000","last_active":"2024-04-17T08:50:49.018542+0000","last_peered":"2024-04-17T08:50:49.018542+0000","last_clean":"2024-04-17T08:50:49.018542+0000","last_became_active":"2024-04-17T08:50:33.883336+0000","last_became_peered":"2024-04-17T08:50:33.883336+0000","last_unstale":"2024-04-17T08:50:49.018542+0000","last_undegraded":"2024-04-17T08:50:49.018542+0000","last_fullsized":"2024-04-17T08:50:49.018542+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:48:21.688586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":9,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.052894+0000","last_change":"2024-04-17T08:50:49.040978+0000","last_active":"2024-04-17T08:50:49.052894+0000","last_peered":"2024-04-17T08:50:49.052894+0000","last_clean":"2024-04-17T08:50:49.052894+0000","last_became_active":"2024-04-17T08:50:49.040638+0000","last_became_peered":"2024-04-17T08:50:49.040638+0000","last_unstale":"2024-04-17T08:50:49.052894+0000","last_undegraded":"2024-04-17T08:50:49.052894+0000","last_fullsized":"2024-04-17T08:50:49.052894+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":232,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:47.974456+0000","last_clean_scrub_stamp":"2024-04-17T08:50:47.974456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:16:30.555717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-17T08:50:49.016884+0000","last_change":"2024-04-17T08:50:33.892568+0000","last_active":"2024-04-17T08:50:49.016884+0000","last_peered":"2024-04-17T08:50:49.016884+0000","last_clean":"2024-04-17T08:50:49.016884+0000","last_became_active":"2024-04-17T08:50:33.892228+0000","last_became_peered":"2024-04-17T08:50:33.892228+0000","last_unstale":"2024-04-17T08:50:49.016884+0000","last_undegraded":"2024-04-17T08:50:49.016884+0000","last_fullsized":"2024-04-17T08:50:49.016884+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":228,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:50:32.851197+0000","last_clean_scrub_stamp":"2024-04-17T08:50:32.851197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:51.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2178,"num_read_kb":4590,"num_write":587,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543296,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52660,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319180,"statfs":{"total":96636764160,"available":96582840320,"internally_reserved":0,"allocated":3510272,"data_stored":2014464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543296,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42344,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329496,"statfs":{"total":96636764160,"available":96593403904,"internally_reserved":0,"allocated":2908160,"data_stored":1421821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543295,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29860,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341980,"statfs":{"total":96636764160,"available":96606187520,"internally_reserved":0,"allocated":2908160,"data_stored":1421821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543295,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64568,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307272,"statfs":{"total":96636764160,"available":96570646528,"internally_reserved":0,"allocated":2924544,"data_stored":1424200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543297,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":2945024,"data_stored":1430776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543296,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52104,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319736,"statfs":{"total":96636764160,"available":96583409664,"internally_reserved":0,"allocated":2936832,"data_stored":1430580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543297,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60396,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311444,"statfs":{"total":96636764160,"available":96574918656,"internally_reserved":0,"allocated":2912256,"data_stored":1421295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543295,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":3522560,"data_stored":2020042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:50:54.121 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:50:54.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:50:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.376+0000 7fd5d3e00640 1 Processor -- start 2024-04-17T08:50:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.376+0000 7fd5d3e00640 1 -- start start 2024-04-17T08:50:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd5d406bfd0 0x7fd5d41634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d40b3ba0 con 0x7fd5d415d530 2024-04-17T08:50:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d406ead0 con 0x7fd5d4164480 2024-04-17T08:50:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd5d406f6e0 con 0x7fd5d406bfd0 2024-04-17T08:50:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd5d415d530 0x7fd5d40b3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46810/0 (socket says 172.21.15.67:46810) 2024-04-17T08:50:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d2a00640 1 -- 172.21.15.67:0/2407996037 learned_addr learned my addr 172.21.15.67:0/2407996037 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d3400640 1 --2- 172.21.15.67:0/2407996037 >> v2:172.21.15.187:3300/0 conn(0x7fd5d406bfd0 0x7fd5d41634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086051372 0 0) 0x7fd5c4001110 con 0x7fd5d415d530 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.378+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5b8003520 con 0x7fd5d415d530 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2848378789 0 0) 0x7fd5b8003520 con 0x7fd5d415d530 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 >> v2:172.21.15.187:3300/0 conn(0x7fd5d406bfd0 msgr2=0x7fd5d41634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 --2- 172.21.15.67:0/2407996037 >> v2:172.21.15.187:3300/0 conn(0x7fd5d406bfd0 0x7fd5d41634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 >> v1:172.21.15.187:6789/0 conn(0x7fd5d4164480 legacy=0x7fd5d406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d406edb0 con 0x7fd5d415d530 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd5c4001840 con 0x7fd5d415d530 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 >> v1:172.21.15.67:6789/0 conn(0x7fd5d415d530 legacy=0x7fd5d40b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5b8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5b8004800 0x7fd5b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5b8005210 0x7fd5b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d406f6e0 con 0x7fd5b8003a70 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b800c140 con 0x7fd5b8004800 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b800c340 con 0x7fd5b8005210 2024-04-17T08:50:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.379+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5b8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5ca000640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5b8004800 0x7fd5b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d2a00640 1 -- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5b8005210 msgr2=0x7fd5b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5b8005210 0x7fd5b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d2a00640 1 -- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5b8004800 msgr2=0x7fd5b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5b8004800 0x7fd5b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d2a00640 1 -- 172.21.15.67:0/2407996037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b8003520 con 0x7fd5b8003a70 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5ca000640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5b8004800 0x7fd5b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.380+0000 7fd5d3400640 1 -- 172.21.15.67:0/2407996037 reap_dead start 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.381+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5b8003eb0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c2f0 tx=0x7fd5b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ebb389bf03951b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.381+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4007e10 con 0x7fd5b8003a70 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.381+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5c4030070 con 0x7fd5b8003a70 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.381+0000 7fd5d2000640 1 -- 172.21.15.67:0/2407996037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c402cdf0 con 0x7fd5b8003a70 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 -- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 msgr2=0x7fd5b8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5b8003eb0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c2f0 tx=0x7fd5b8005830 comp rx=0 tx=0).stop 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 -- 172.21.15.67:0/2407996037 shutdown_connections 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/2407996037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5b8003eb0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 -- 172.21.15.67:0/2407996037 >> 172.21.15.67:0/2407996037 conn(0x7fd5d4067d10 msgr2=0x7fd5d40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 -- 172.21.15.67:0/2407996037 shutdown_connections 2024-04-17T08:50:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.382+0000 7fd5d3e00640 1 -- 172.21.15.67:0/2407996037 wait complete. 2024-04-17T08:50:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.384+0000 7fd5d3e00640 1 Processor -- start 2024-04-17T08:50:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.384+0000 7fd5d3e00640 1 -- start start 2024-04-17T08:50:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5d4159490 0x7fd5d415ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5d40d0390 0x7fd5d41f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d406fd60 con 0x7fd5b8003a70 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d406f370 con 0x7fd5d4159490 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d406fee0 con 0x7fd5d40d0390 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48372/0 (socket says 172.21.15.67:48372) 2024-04-17T08:50:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.385+0000 7fd5d2a00640 1 -- 172.21.15.67:0/4022667013 learned_addr learned my addr 172.21.15.67:0/4022667013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 -- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5d40d0390 msgr2=0x7fd5d41f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5d40d0390 0x7fd5d41f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 -- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5d4159490 msgr2=0x7fd5d415ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5ca000640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5d4159490 0x7fd5d415ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5d4159490 0x7fd5d415ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 -- 172.21.15.67:0/4022667013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d415f740 con 0x7fd5b8003a70 2024-04-17T08:50:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d2a00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c2f0 tx=0x7fd5c402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4046070 con 0x7fd5b8003a70 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5c4004bf0 con 0x7fd5b8003a70 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d415fa20 con 0x7fd5b8003a70 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.386+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4007e10 con 0x7fd5b8003a70 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.387+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d415ffe0 con 0x7fd5b8003a70 2024-04-17T08:50:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.388+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d4068a10 con 0x7fd5b8003a70 2024-04-17T08:50:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.389+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fd5c4007070 con 0x7fd5b8003a70 2024-04-17T08:50:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.390+0000 7fd5cbe00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 0x7fd5b006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.390+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7fd5c40ad280 con 0x7fd5b8003a70 2024-04-17T08:50:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.391+0000 7fd5ca000640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 0x7fd5b006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.392+0000 7fd5ca000640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 0x7fd5b006eed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40ccad0 tx=0x7fd5bc009040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.395+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd5d40b3ba0 con 0x7fd5b8003a70 2024-04-17T08:50:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.640+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5d4072b70 con 0x7fd5b8003a70 2024-04-17T08:50:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.642+0000 7fd5cbe00640 1 -- 172.21.15.67:0/4022667013 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7fd5c407b220 con 0x7fd5b8003a70 2024-04-17T08:50:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:50:54.641 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:50:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 msgr2=0x7fd5b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 0x7fd5b006eed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40ccad0 tx=0x7fd5bc009040 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 msgr2=0x7fd5d41607e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c2f0 tx=0x7fd5c402e070 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 shutdown_connections 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fd5b006ca10 0x7fd5b006eed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5d40d0390 0x7fd5d41f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5d4159490 0x7fd5d415ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 --2- 172.21.15.67:0/4022667013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5b8003a70 0x7fd5d41607e0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 >> 172.21.15.67:0/4022667013 conn(0x7fd5d4067d10 msgr2=0x7fd5d40b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 shutdown_connections 2024-04-17T08:50:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:54.644+0000 7fd5d3e00640 1 -- 172.21.15.67:0/4022667013 wait complete. 2024-04-17T08:50:55.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:50:48.498917+0000', 'modified': '2024-04-17T08:50:51.569805+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11025}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11025': {'gid': 11025, 'name': 'c', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.67:6833/4206695657', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 4206695657}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 4206695657}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11025, 'qdb_cluster': [11025]} 2024-04-17T08:50:55.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:50:55.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T08:50:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.295+0000 7f9747200640 1 Processor -- start 2024-04-17T08:50:55.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.296+0000 7f9747200640 1 -- start start 2024-04-17T08:50:55.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.297+0000 7f9747200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480afbf0 0x7f974806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.297+0000 7f9747200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f974815fad0 0x7f97480ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9747200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97480cb420 con 0x7f974806eec0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9745e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480afbf0 0x7f974806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f973fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f974806eec0 0x7f974815f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46824/0 (socket says 172.21.15.67:46824) 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9745e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480afbf0 0x7f974806e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48382/0 (socket says 172.21.15.67:48382) 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f973fe00640 1 -- 172.21.15.67:0/3696367377 learned_addr learned my addr 172.21.15.67:0/3696367377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f97480cc030 con 0x7f97480afbf0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f97480cb700 con 0x7f974815fad0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9746800640 1 --2- 172.21.15.67:0/3696367377 >> v2:172.21.15.187:3300/0 conn(0x7f974815fad0 0x7f97480ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9745400640 1 -- 172.21.15.67:0/3696367377 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2805146170 0 0) 0x7f97480cb420 con 0x7f974806eec0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.298+0000 7f9745400640 1 -- 172.21.15.67:0/3696367377 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f972c005730 con 0x7f974806eec0 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.299+0000 7f9745e00640 1 -- 172.21.15.67:0/3696367377 >> v2:172.21.15.187:3300/0 conn(0x7f974815fad0 msgr2=0x7f97480ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.299+0000 7f9745e00640 1 --2- 172.21.15.67:0/3696367377 >> v2:172.21.15.187:3300/0 conn(0x7f974815fad0 0x7f97480ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.299+0000 7f9745e00640 1 -- 172.21.15.67:0/3696367377 >> v1:172.21.15.67:6789/0 conn(0x7f974806eec0 legacy=0x7f974815f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.299+0000 7f9745e00640 1 -- 172.21.15.67:0/3696367377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97480cb9e0 con 0x7f97480afbf0 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.300+0000 7f9745e00640 1 --2- 172.21.15.67:0/3696367377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f974806e980 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f97480cae60 tx=0x7f973802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5241ab06e1815fff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.300+0000 7f9745400640 1 -- 172.21.15.67:0/3696367377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738035020 con 0x7f97480afbf0 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.300+0000 7f9745400640 1 -- 172.21.15.67:0/3696367377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9738031830 con 0x7f97480afbf0 2024-04-17T08:50:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.300+0000 7f9745400640 1 -- 172.21.15.67:0/3696367377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738031be0 con 0x7f97480afbf0 2024-04-17T08:50:55.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.300+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 msgr2=0x7f974806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 --2- 172.21.15.67:0/3696367377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f974806e980 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f97480cae60 tx=0x7f973802b8d0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 shutdown_connections 2024-04-17T08:50:55.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 --2- 172.21.15.67:0/3696367377 >> v2:172.21.15.187:3300/0 conn(0x7f974815fad0 0x7f97480ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 --2- 172.21.15.67:0/3696367377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f974806e980 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 >> 172.21.15.67:0/3696367377 conn(0x7f974806bef0 msgr2=0x7f97480b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 shutdown_connections 2024-04-17T08:50:55.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.301+0000 7f9747200640 1 -- 172.21.15.67:0/3696367377 wait complete. 2024-04-17T08:50:55.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.303+0000 7f9747200640 1 Processor -- start 2024-04-17T08:50:55.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.303+0000 7f9747200640 1 -- start start 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f974806eec0 0x7f9748156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 0x7f974813b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97480affd0 con 0x7f97480afbf0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9748160110 con 0x7f974815fad0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9747200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97480b0150 con 0x7f974806eec0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48388/0 (socket says 172.21.15.67:48388) 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 -- 172.21.15.67:0/3398347536 learned_addr learned my addr 172.21.15.67:0/3398347536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9746800640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 0x7f974813b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9745e00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f974806eec0 0x7f9748156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 -- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f974806eec0 msgr2=0x7f9748156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f974806eec0 0x7f9748156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 -- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 msgr2=0x7f974813b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 0x7f974813b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f973fe00640 1 -- 172.21.15.67:0/3398347536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9748155630 con 0x7f97480afbf0 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.304+0000 7f9746800640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 0x7f974813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f973fe00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f973000c500 tx=0x7f973000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f973e000640 1 -- 172.21.15.67:0/3398347536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730017070 con 0x7f97480afbf0 2024-04-17T08:50:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f973e000640 1 -- 172.21.15.67:0/3398347536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9730005ce0 con 0x7f97480afbf0 2024-04-17T08:50:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f973e000640 1 -- 172.21.15.67:0/3398347536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730007330 con 0x7f97480afbf0 2024-04-17T08:50:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9748155df0 con 0x7f97480afbf0 2024-04-17T08:50:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.305+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9748155350 con 0x7f97480afbf0 2024-04-17T08:50:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.308+0000 7f973e000640 1 -- 172.21.15.67:0/3398347536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f973001a070 con 0x7f97480afbf0 2024-04-17T08:50:55.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.309+0000 7f973e000640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 0x7f972406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.309+0000 7f973e000640 1 -- 172.21.15.67:0/3398347536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f973008b520 con 0x7f97480afbf0 2024-04-17T08:50:55.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.309+0000 7f9745e00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 0x7f972406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.309+0000 7f974fc30740 1 Processor -- start 2024-04-17T08:50:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.310+0000 7f974fc30740 1 -- start start 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bea180 0x55a045c08c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c08370 0x55a045bc2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a045b28980 con 0x55a045bec930 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a045acf220 con 0x55a045bea180 2024-04-17T08:50:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a045b24940 con 0x55a045c08370 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9745e00640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 0x7f972406ef00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9738036000 tx=0x7f9738008030 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48404/0 (socket says 172.21.15.67:48404) 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 -- 172.21.15.67:0/3372817067 learned_addr learned my addr 172.21.15.67:0/3372817067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9745e00640 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bea180 0x55a045c08c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f973fe00640 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c08370 0x55a045bc2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 -- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c08370 msgr2=0x55a045bc2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c08370 0x55a045bc2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 -- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bea180 msgr2=0x55a045c08c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.311+0000 7f9746800640 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bea180 0x55a045c08c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.312+0000 7f9746800640 1 -- 172.21.15.67:0/3372817067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a04583b2d0 con 0x55a045bec930 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.312+0000 7f9746800640 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f9740007390 tx=0x7f9740007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc53655c1cfc709b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.312+0000 7f9736a00640 1 -- 172.21.15.67:0/3372817067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974001a070 con 0x55a045bec930 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.312+0000 7f9736a00640 1 -- 172.21.15.67:0/3372817067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9740003da0 con 0x55a045bec930 2024-04-17T08:50:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.312+0000 7f9736a00640 1 -- 172.21.15.67:0/3372817067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740004230 con 0x55a045bec930 2024-04-17T08:50:55.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 -- 172.21.15.67:0/3372817067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 msgr2=0x55a045becdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f9740007390 tx=0x7f9740007850 comp rx=0 tx=0).stop 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 -- 172.21.15.67:0/3372817067 shutdown_connections 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045bec930 0x55a045becdb0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c08370 0x55a045bc2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 --2- 172.21.15.67:0/3372817067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bea180 0x55a045c08c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 -- 172.21.15.67:0/3372817067 >> 172.21.15.67:0/3372817067 conn(0x55a045bbedb0 msgr2=0x55a045c07930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 -- 172.21.15.67:0/3372817067 shutdown_connections 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.313+0000 7f974fc30740 1 -- 172.21.15.67:0/3372817067 wait complete. 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.314+0000 7f974fc30740 1 -- start start 2024-04-17T08:50:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.314+0000 7f974fc30740 1 Processor -- start 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045bc1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c08370 0x55a045c0a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a045bea560 con 0x55a045c0a850 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a045ae2590 con 0x55a045c08370 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a045bbcb40 con 0x55a045bec930 2024-04-17T08:50:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f974fc30740 10 client.0 _ll_register_callbacks cb 0x55a04583aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48406/0 (socket says 172.21.15.67:48406) 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f9746800640 1 -- 172.21.15.67:0/2872366517 learned_addr learned my addr 172.21.15.67:0/2872366517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.315+0000 7f973fe00640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c08370 0x55a045c0a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9745e00640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045bc1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 -- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 msgr2=0x55a045bc1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045bc1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 -- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c08370 msgr2=0x55a045c0a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c08370 0x55a045c0a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 -- 172.21.15.67:0/2872366517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0459056b0 con 0x55a045c0a850 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f9746800640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f974000b010 tx=0x7f9740004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfab456eac47bc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f972b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740024020 con 0x55a045c0a850 2024-04-17T08:50:55.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f9740004720 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.316+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9740003da0 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f974002e050 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 10 client.11322 fetch_fsmap learned FSMap version 115 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x55a045bc2240 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9740003da0 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 115) ==== 4143+0+0 (secure 0 0 0) 0x7f9740031020 con 0x55a045c0a850 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 10 client.11322 fetch_fsmap finished waiting for FSMap version 115 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 10 client.11322 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 20 client.11322 populate_metadata read hostname 'smithi067' 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c1f9b0 0x55a045bedea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 4 client.11322 mds_command: new command op to 11025 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 --> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a045b363a0 con 0x55a045c1f9b0 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.317+0000 7f9745e00640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c1f9b0 0x55a045bedea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.318+0000 7f9745e00640 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c1f9b0 0x55a045bedea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba2a4fd8f719ebe0 server_cookie=a23c41cb3b189948 in_seq=0 out_seq=0 2024-04-17T08:50:55.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.318+0000 7f972b400640 10 client.11322 ms_handle_connect on v2:172.21.15.67:6832/4206695657 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f972b400640 1 -- 172.21.15.67:0/2872366517 <== mds.0 v2:172.21.15.67:6832/4206695657 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a045b363a0 con 0x55a045c1f9b0 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f972b400640 10 client.11322 handle_command_reply: tid=0 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 client.11322 shutdown 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 msgr2=0x55a045be6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f974000b010 tx=0x7f9740004ab0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 shutdown_connections 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c1f9b0 0x55a045bedea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c0a850 0x55a045be6d20 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c08370 0x55a045c0a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 --2- 172.21.15.67:0/2872366517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045bc1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f972b400640 0 client.11322 ms_handle_reset on v2:172.21.15.67:6832/4206695657 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 >> 172.21.15.67:0/2872366517 conn(0x55a045bea180 msgr2=0x55a045bebca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.319+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 shutdown_connections 2024-04-17T08:50:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.320+0000 7f974fc30740 1 -- 172.21.15.67:0/2872366517 wait complete. 2024-04-17T08:50:55.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.320+0000 7f974fc30740 20 client.11322 trim_cache size 0 max 16384 2024-04-17T08:50:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.340+0000 7f974fc30740 1 Processor -- start 2024-04-17T08:50:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.340+0000 7f974fc30740 1 -- start start 2024-04-17T08:50:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045c7ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c0a850 0x55a045bbbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97481ff820 con 0x55a045c08370 2024-04-17T08:50:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f974813e5c0 con 0x55a045c0a850 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97481fd8a0 con 0x55a045bec930 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48412/0 (socket says 172.21.15.67:48412) 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 -- 172.21.15.67:0/61846974 learned_addr learned my addr 172.21.15.67:0/61846974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 -- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 msgr2=0x55a045c7ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045c7ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 -- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c0a850 msgr2=0x55a045bbbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c0a850 0x55a045bbbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 -- 172.21.15.67:0/61846974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a045bc09d0 con 0x55a045c08370 2024-04-17T08:50:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.341+0000 7f973fe00640 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f97300895a0 tx=0x7f9730061a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f4dad832df8de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f9736000640 1 -- 172.21.15.67:0/61846974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730089620 con 0x55a045c08370 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f9736000640 1 -- 172.21.15.67:0/61846974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97300897c0 con 0x55a045c08370 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f9736000640 1 -- 172.21.15.67:0/61846974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300592d0 con 0x55a045c08370 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 -- 172.21.15.67:0/61846974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 msgr2=0x55a045c1b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f97300895a0 tx=0x7f9730061a50 comp rx=0 tx=0).stop 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 -- 172.21.15.67:0/61846974 shutdown_connections 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045c0a850 0x55a045bbbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c1b480 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 --2- 172.21.15.67:0/61846974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045bec930 0x55a045c7ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 -- 172.21.15.67:0/61846974 >> 172.21.15.67:0/61846974 conn(0x55a045bea180 msgr2=0x55a045bbc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 -- 172.21.15.67:0/61846974 shutdown_connections 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.342+0000 7f974fc30740 1 -- 172.21.15.67:0/61846974 wait complete. 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.343+0000 7f974fc30740 1 -- start start 2024-04-17T08:50:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.343+0000 7f974fc30740 1 Processor -- start 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bec930 0x55a045bee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c0a850 0x55a045c711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a045bf00b0 con 0x55a045c08370 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a045bf03d0 con 0x55a045bec930 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a045bed970 con 0x55a045c0a850 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.344+0000 7f974fc30740 10 client.0 _ll_register_callbacks cb 0x7f9740018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48414/0 (socket says 172.21.15.67:48414) 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 -- 172.21.15.67:0/2062556089 learned_addr learned my addr 172.21.15.67:0/2062556089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 -- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c0a850 msgr2=0x55a045c711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f9746800640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c0a850 0x55a045c711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c0a850 0x55a045c711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 -- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bec930 msgr2=0x55a045bee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bec930 0x55a045bee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 -- 172.21.15.67:0/2062556089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a045beb050 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f973fe00640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f9730000f60 tx=0x7f9730001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221adb0619d80804 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.345+0000 7f972b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973007d060 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f9730078210 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97300897c0 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f9730078070 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 10 client.11328 fetch_fsmap learned FSMap version 115 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x55a045c05c70 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97300897c0 con 0x55a045c08370 2024-04-17T08:50:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 115) ==== 4143+0+0 (secure 0 0 0) 0x7f97300074d0 con 0x55a045c08370 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 10 client.11328 fetch_fsmap finished waiting for FSMap version 115 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 10 client.11328 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 20 client.11328 populate_metadata read hostname 'smithi067' 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c06270 0x55a045c06650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 4 client.11328 mds_command: new command op to 11025 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.346+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 --> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] -- command(tid 0: {"prefix": "status"}) -- 0x55a045b30210 con 0x55a045c06270 2024-04-17T08:50:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f9745e00640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c06270 0x55a045c06650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f972b400640 10 client.11328 ms_handle_connect on v2:172.21.15.67:6832/4206695657 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f9745e00640 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c06270 0x55a045c06650 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2c0e1ccdf5af7b server_cookie=e6d9db06361120aa in_seq=0 out_seq=0 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f972b400640 1 -- 172.21.15.67:0/2062556089 <== mds.0 v2:172.21.15.67:6832/4206695657 1 ==== command_reply(tid 0: 0 ) ==== 8+0+338 (crc 0 0 0) 0x55a045b30210 con 0x55a045c06270 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f972b400640 10 client.11328 handle_command_reply: tid=0 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 client.11328 shutdown 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 msgr2=0x55a045c70c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f9730000f60 tx=0x7f9730001150 comp rx=0 tx=0).stop 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 shutdown_connections 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a045c0a850 0x55a045c711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:6832/4206695657,v1:172.21.15.67:6833/4206695657] conn(0x55a045c06270 0x55a045c06650 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a045c08370 0x55a045c70c70 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 --2- 172.21.15.67:0/2062556089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a045bec930 0x55a045bee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.347+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 >> 172.21.15.67:0/2062556089 conn(0x55a045bea180 msgr2=0x55a045bbbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f972b400640 0 client.11328 ms_handle_reset on v2:172.21.15.67:6832/4206695657 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 shutdown_connections 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f974fc30740 1 -- 172.21.15.67:0/2062556089 wait complete. 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f974fc30740 20 client.11328 trim_cache size 0 max 16384 2024-04-17T08:50:55.350 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 11025, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "backup_fs", 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.3273865320000002, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 115, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 234, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 234, 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 33.809224020999999 2024-04-17T08:50:55.351 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 msgr2=0x7f972406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 0x7f972406ef00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9738036000 tx=0x7f9738008030 comp rx=0 tx=0).stop 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 msgr2=0x7f9748157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f973000c500 tx=0x7f973000c9c0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 shutdown_connections 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f972406ca40 0x7f972406ef00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f974815fad0 0x7f974813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480afbf0 0x7f9748157220 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.348+0000 7f9747200640 1 --2- 172.21.15.67:0/3398347536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f974806eec0 0x7f9748156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.349+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 >> 172.21.15.67:0/3398347536 conn(0x7f974806bef0 msgr2=0x7f97480cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.349+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 shutdown_connections 2024-04-17T08:50:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.349+0000 7f9747200640 1 -- 172.21.15.67:0/3398347536 wait complete. 2024-04-17T08:50:55.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:50:55.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.568+0000 7fe72d400640 1 Processor -- start 2024-04-17T08:50:55.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.568+0000 7fe72d400640 1 -- start start 2024-04-17T08:50:55.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.570+0000 7fe72d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe7280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.570+0000 7fe72d400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe728072570 0x7fe7280ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.570+0000 7fe72d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72815d060 con 0x7fe7280b63c0 2024-04-17T08:50:55.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.570+0000 7fe72d400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe7280cfe90 con 0x7fe728072570 2024-04-17T08:50:55.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.570+0000 7fe72d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe72815dc70 con 0x7fe7280cbc00 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe71fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7280b63c0 0x7fe728071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46828/0 (socket says 172.21.15.67:46828) 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe71fe00640 1 -- 172.21.15.67:0/188659334 learned_addr learned my addr 172.21.15.67:0/188659334 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe727e00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe7280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe72ca00640 1 --2- 172.21.15.67:0/188659334 >> v2:172.21.15.187:3300/0 conn(0x7fe728072570 0x7fe7280ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2512316693 0 0) 0x7fe72815d060 con 0x7fe7280b63c0 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.571+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe70c003520 con 0x7fe7280b63c0 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 435850816 0 0) 0x7fe70c003520 con 0x7fe7280b63c0 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 msgr2=0x7fe7280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe7280cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 >> v2:172.21.15.187:3300/0 conn(0x7fe728072570 msgr2=0x7fe7280ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 --2- 172.21.15.67:0/188659334 >> v2:172.21.15.187:3300/0 conn(0x7fe728072570 0x7fe7280ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72815d340 con 0x7fe7280b63c0 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe710001670 con 0x7fe7280b63c0 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 >> v1:172.21.15.67:6789/0 conn(0x7fe7280b63c0 legacy=0x7fe728071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe70c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7280cfe90 con 0x7fe70c003a20 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe72815dc70 con 0x7fe70c004d40 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe70c00b9c0 con 0x7fe7280cbc00 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe71fe00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe70c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 -- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 msgr2=0x7fe70c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280cbc00 0x7fe70c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe72ca00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 -- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c004d40 msgr2=0x7fe70c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c004d40 0x7fe70c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.572+0000 7fe727e00640 1 -- 172.21.15.67:0/188659334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c003520 con 0x7fe70c003a20 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.573+0000 7fe727e00640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ac0 tx=0x7fe70c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5587bb8e2e378106 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718039070 con 0x7fe70c003a20 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71802c6b0 con 0x7fe70c003a20 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe727400640 1 -- 172.21.15.67:0/188659334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718009690 con 0x7fe70c003a20 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe72d400640 1 -- 172.21.15.67:0/188659334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe70c003a20 msgr2=0x7fe70c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe72d400640 1 --2- 172.21.15.67:0/188659334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe70c003a20 0x7fe70c003e40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ac0 tx=0x7fe70c005830 comp rx=0 tx=0).stop 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe72ca00640 1 -- 172.21.15.67:0/188659334 reap_dead start 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe72d400640 1 -- 172.21.15.67:0/188659334 shutdown_connections 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.574+0000 7fe72d400640 1 -- 172.21.15.67:0/188659334 >> 172.21.15.67:0/188659334 conn(0x7fe72806bef0 msgr2=0x7fe7280b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.575+0000 7fe72d400640 1 -- 172.21.15.67:0/188659334 shutdown_connections 2024-04-17T08:50:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.575+0000 7fe72d400640 1 -- 172.21.15.67:0/188659334 wait complete. 2024-04-17T08:50:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.576+0000 7fe72d400640 1 Processor -- start 2024-04-17T08:50:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.576+0000 7fe72d400640 1 -- start start 2024-04-17T08:50:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 0x7fe7281f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280aed70 0x7fe72806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7280d02d0 con 0x7fe72806fcb0 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe72815d900 con 0x7fe70c003a20 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.577+0000 7fe72d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe7280cbfe0 con 0x7fe7280aed70 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48444/0 (socket says 172.21.15.67:48444) 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 -- 172.21.15.67:0/1112591007 learned_addr learned my addr 172.21.15.67:0/1112591007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe727e00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 0x7fe7281f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe71fe00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280aed70 0x7fe72806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 -- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280aed70 msgr2=0x7fe72806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280aed70 0x7fe72806f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 -- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 msgr2=0x7fe7281f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 0x7fe7281f4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 -- 172.21.15.67:0/1112591007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7280ca1f0 con 0x7fe72806fcb0 2024-04-17T08:50:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe727e00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 0x7fe7281f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.578+0000 7fe72ca00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fe728070350 tx=0x7fe72000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.579+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72001c070 con 0x7fe72806fcb0 2024-04-17T08:50:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.579+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe720017400 con 0x7fe72806fcb0 2024-04-17T08:50:55.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.579+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7200177f0 con 0x7fe72806fcb0 2024-04-17T08:50:55.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.579+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7280ca4d0 con 0x7fe72806fcb0 2024-04-17T08:50:55.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.579+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7280caa90 con 0x7fe72806fcb0 2024-04-17T08:50:55.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.580+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe728068a10 con 0x7fe72806fcb0 2024-04-17T08:50:55.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.581+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7fe72001f070 con 0x7fe72806fcb0 2024-04-17T08:50:55.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.582+0000 7fe724c00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 0x7fe70406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:55.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.582+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7fe72008c160 con 0x7fe72806fcb0 2024-04-17T08:50:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.583+0000 7fe727e00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 0x7fe70406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.584+0000 7fe727e00640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 0x7fe70406ef00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe718030000 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.587+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe720012070 con 0x7fe72806fcb0 2024-04-17T08:50:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.820+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fe7281f4db0 con 0x7fe70406ca40 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.825+0000 7fe724c00640 1 -- 172.21.15.67:0/1112591007 <== mgr.7679 v2:172.21.15.187:6832/150239855 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7fe718004140 con 0x7fe70406ca40 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "backup_fs" 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:50:55.824 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:50:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.827+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 msgr2=0x7fe70406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.827+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 0x7fe70406ef00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe718036000 tx=0x7fe718030000 comp rx=0 tx=0).stop 2024-04-17T08:50:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.827+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 msgr2=0x7fe7280cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.827+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fe728070350 tx=0x7fe72000f4a0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 shutdown_connections 2024-04-17T08:50:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fe70406ca40 0x7fe70406ef00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72806fcb0 0x7fe7280cef20 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe7280aed70 0x7fe72806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 --2- 172.21.15.67:0/1112591007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe70c003a20 0x7fe7281f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 >> 172.21.15.67:0/1112591007 conn(0x7fe72806bef0 msgr2=0x7fe72815a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 shutdown_connections 2024-04-17T08:50:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:55.828+0000 7fe72d400640 1 -- 172.21.15.67:0/1112591007 wait complete. 2024-04-17T08:50:55.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:50:56.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.063+0000 7f183f200640 1 Processor -- start 2024-04-17T08:50:56.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.063+0000 7f183f200640 1 -- start start 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1838145f80 0x7f18380cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18380cd750 0x7f183806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f183806f1d0 con 0x7f183806c0a0 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f18380727a0 con 0x7f1838145f80 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1838146360 con 0x7f18380cd750 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f183806c0a0 0x7f1838160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46830/0 (socket says 172.21.15.67:46830) 2024-04-17T08:50:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183e800640 1 -- 172.21.15.67:0/2787430180 learned_addr learned my addr 172.21.15.67:0/2787430180 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:50:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183de00640 1 --2- 172.21.15.67:0/2787430180 >> v2:172.21.15.187:3300/0 conn(0x7f1838145f80 0x7f18380cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.065+0000 7f183d400640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18380cd750 0x7f183806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1890755050 0 0) 0x7f1834001490 con 0x7f183806c0a0 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1820003520 con 0x7f183806c0a0 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1756985034 0 0) 0x7f1820003520 con 0x7f183806c0a0 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183de00640 1 -- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18380cd750 msgr2=0x7f183806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183de00640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18380cd750 0x7f183806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183de00640 1 -- 172.21.15.67:0/2787430180 >> v1:172.21.15.67:6789/0 conn(0x7f183806c0a0 legacy=0x7f1838160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.066+0000 7f183de00640 1 -- 172.21.15.67:0/2787430180 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1838071e70 con 0x7f1838145f80 2024-04-17T08:50:56.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.067+0000 7f183de00640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1838145f80 0x7f18380cd210 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f182c0097a0 tx=0x7f182c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31160c4ef8acb17c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:56.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.068+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c037070 con 0x7f1838145f80 2024-04-17T08:50:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.068+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c030510 con 0x7f1838145f80 2024-04-17T08:50:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.068+0000 7f183ca00640 1 -- 172.21.15.67:0/2787430180 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834001bc0 con 0x7f1838145f80 2024-04-17T08:50:56.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 -- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1838145f80 msgr2=0x7f18380cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1838145f80 0x7f18380cd210 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f182c0097a0 tx=0x7f182c02b710 comp rx=0 tx=0).stop 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 -- 172.21.15.67:0/2787430180 shutdown_connections 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18380cd750 0x7f183806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 --2- 172.21.15.67:0/2787430180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1838145f80 0x7f18380cd210 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 -- 172.21.15.67:0/2787430180 >> 172.21.15.67:0/2787430180 conn(0x7f18380b2190 msgr2=0x7f18380b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 -- 172.21.15.67:0/2787430180 shutdown_connections 2024-04-17T08:50:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.069+0000 7f183f200640 1 -- 172.21.15.67:0/2787430180 wait complete. 2024-04-17T08:50:56.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.071+0000 7f183f200640 1 Processor -- start 2024-04-17T08:50:56.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.071+0000 7f183f200640 1 -- start start 2024-04-17T08:50:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 0x7f18381e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1838145f80 0x7f18381e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1838072c30 con 0x7f183806c0a0 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1838072430 con 0x7f18380cd750 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1838072db0 con 0x7f1838145f80 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48460/0 (socket says 172.21.15.67:48460) 2024-04-17T08:50:56.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.072+0000 7f183de00640 1 -- 172.21.15.67:0/3600438776 learned_addr learned my addr 172.21.15.67:0/3600438776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:56.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183e800640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1838145f80 0x7f18381e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183d400640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 0x7f18381e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 -- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1838145f80 msgr2=0x7f18381e5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1838145f80 0x7f18381e5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 -- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 msgr2=0x7f18381e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 0x7f18381e4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 -- 172.21.15.67:0/3600438776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183813b3c0 con 0x7f183806c0a0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183d400640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 0x7f18381e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183de00640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f182c000c00 tx=0x7f182c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c041020 con 0x7f183806c0a0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c03c870 con 0x7f183806c0a0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c037040 con 0x7f183806c0a0 2024-04-17T08:50:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.073+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1838139c80 con 0x7f183806c0a0 2024-04-17T08:50:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.074+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183813b960 con 0x7f183806c0a0 2024-04-17T08:50:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.075+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1838068a10 con 0x7f183806c0a0 2024-04-17T08:50:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.076+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 129) ==== 91142+0+0 (secure 0 0 0) 0x7f182c044020 con 0x7f183806c0a0 2024-04-17T08:50:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.077+0000 7f1832000640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 0x7f181c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.077+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f182c0adbe0 con 0x7f183806c0a0 2024-04-17T08:50:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.077+0000 7f183d400640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 0x7f181c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:56.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.079+0000 7f183d400640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 0x7f181c06eed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1838140e90 tx=0x7f1828008040 comp rx=0 tx=0).ready entity=mgr.7679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:56.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.081+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f182c04b030 con 0x7f183806c0a0 2024-04-17T08:50:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:56.343+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f183815e0e0 con 0x7f183806c0a0 2024-04-17T08:50:57.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.099+0000 7f1832000640 1 -- 172.21.15.67:0/3600438776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v130) ==== 92+0+0 (secure 0 0 0) 0x7f182c0ad070 con 0x7f183806c0a0 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.103+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 msgr2=0x7f181c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.103+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 0x7f181c06eed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1838140e90 tx=0x7f1828008040 comp rx=0 tx=0).stop 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.103+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 msgr2=0x7f1838139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.103+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f182c000c00 tx=0x7f182c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 shutdown_connections 2024-04-17T08:50:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f181c06ca10 0x7f181c06eed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1838145f80 0x7f18381e5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18380cd750 0x7f18381e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 --2- 172.21.15.67:0/3600438776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f183806c0a0 0x7f1838139740 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 >> 172.21.15.67:0/3600438776 conn(0x7f18380b2190 msgr2=0x7f183815cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 shutdown_connections 2024-04-17T08:50:57.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.104+0000 7f183f200640 1 -- 172.21.15.67:0/3600438776 wait complete. 2024-04-17T08:50:57.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:50:57.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.340+0000 7f61ac800640 1 Processor -- start 2024-04-17T08:50:57.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.340+0000 7f61ac800640 1 -- start start 2024-04-17T08:50:57.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.342+0000 7f61ac800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f61a406e830 0x7f61a415f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.342+0000 7f61ac800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a40cce80 0x7f61a40cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.342+0000 7f61ac800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a415beb0 con 0x7f61a40cfe00 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.342+0000 7f61ac800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f61a40cd810 con 0x7f61a40cce80 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.342+0000 7f61ac800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f61a415c190 con 0x7f61a406e830 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a40cce80 0x7f61a40cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a40cce80 0x7f61a40cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48480/0 (socket says 172.21.15.67:48480) 2024-04-17T08:50:57.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 -- 172.21.15.67:0/2317761437 learned_addr learned my addr 172.21.15.67:0/2317761437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61a9e00640 1 --2- 172.21.15.67:0/2317761437 >> v2:172.21.15.187:3300/0 conn(0x7f61a406e830 0x7f61a415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 -- 172.21.15.67:0/2317761437 >> v2:172.21.15.187:3300/0 conn(0x7f61a406e830 msgr2=0x7f61a415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 --2- 172.21.15.67:0/2317761437 >> v2:172.21.15.187:3300/0 conn(0x7f61a406e830 0x7f61a415f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 -- 172.21.15.67:0/2317761437 >> v1:172.21.15.187:6789/0 conn(0x7f61a40cfe00 legacy=0x7f61a415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.343+0000 7f61aa800640 1 -- 172.21.15.67:0/2317761437 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a415ec80 con 0x7f61a40cce80 2024-04-17T08:50:57.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.345+0000 7f61aa800640 1 --2- 172.21.15.67:0/2317761437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a40cd2d0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f61a406dea0 tx=0x7f61a002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea867c1b9c65b526 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:57.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.345+0000 7f61a8a00640 1 -- 172.21.15.67:0/2317761437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a002cb20 con 0x7f61a40cce80 2024-04-17T08:50:57.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.345+0000 7f61a8a00640 1 -- 172.21.15.67:0/2317761437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a002ccc0 con 0x7f61a40cce80 2024-04-17T08:50:57.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.345+0000 7f61a8a00640 1 -- 172.21.15.67:0/2317761437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0034610 con 0x7f61a40cce80 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 -- 172.21.15.67:0/2317761437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 msgr2=0x7f61a40cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 --2- 172.21.15.67:0/2317761437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a40cd2d0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f61a406dea0 tx=0x7f61a002bf80 comp rx=0 tx=0).stop 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 -- 172.21.15.67:0/2317761437 shutdown_connections 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 --2- 172.21.15.67:0/2317761437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a40cd2d0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 --2- 172.21.15.67:0/2317761437 >> v2:172.21.15.187:3300/0 conn(0x7f61a406e830 0x7f61a415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 -- 172.21.15.67:0/2317761437 >> 172.21.15.67:0/2317761437 conn(0x7f61a4067ee0 msgr2=0x7f61a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:57.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 -- 172.21.15.67:0/2317761437 shutdown_connections 2024-04-17T08:50:57.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.346+0000 7f61ac800640 1 -- 172.21.15.67:0/2317761437 wait complete. 2024-04-17T08:50:57.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.348+0000 7f61ac800640 1 Processor -- start 2024-04-17T08:50:57.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.348+0000 7f61ac800640 1 -- start start 2024-04-17T08:50:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a406e830 0x7f61a416ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 0x7f61a414ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61a415f9c0 con 0x7f61a40cce80 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61a415c470 con 0x7f61a40cfe00 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61ac800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61a40d04b0 con 0x7f61a406e830 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48486/0 (socket says 172.21.15.67:48486) 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.349+0000 7f61a9400640 1 -- 172.21.15.67:0/3859315232 learned_addr learned my addr 172.21.15.67:0/3859315232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61aa800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 0x7f61a414ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9e00640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a406e830 0x7f61a416ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a406e830 msgr2=0x7f61a416ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a406e830 0x7f61a416ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 msgr2=0x7f61a414ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 0x7f61a414ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 -- 172.21.15.67:0/3859315232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a416b050 con 0x7f61a40cce80 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61aa800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 0x7f61a414ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.350+0000 7f61a9400640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f619400c3c0 tx=0x7f619400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:57.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.351+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61940052d0 con 0x7f61a40cce80 2024-04-17T08:50:57.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.351+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6194005490 con 0x7f61a40cce80 2024-04-17T08:50:57.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.351+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6194007260 con 0x7f61a40cce80 2024-04-17T08:50:57.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.351+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a414c200 con 0x7f61a40cce80 2024-04-17T08:50:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.351+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a414c4c0 con 0x7f61a40cce80 2024-04-17T08:50:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.352+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a40afd70 con 0x7f61a40cce80 2024-04-17T08:50:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.353+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 130) ==== 91159+0+0 (secure 0 0 0) 0x7f6194007400 con 0x7f61a40cce80 2024-04-17T08:50:57.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.354+0000 7f619e000640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 0x7f618806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.354+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f619408b580 con 0x7f61a40cce80 2024-04-17T08:50:57.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.354+0000 7f61a9e00640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 msgr2=0x7f618806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:57.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.354+0000 7f61a9e00640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 0x7f618806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:57.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.359+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6194059520 con 0x7f61a40cce80 2024-04-17T08:50:57.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:50:57.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:50:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.556+0000 7f61a9e00640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 msgr2=0x7f618806ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.556+0000 7f61a9e00640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 0x7f618806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:57.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.568+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f61a40d09e0 con 0x7f61a40cce80 2024-04-17T08:50:57.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.575+0000 7f619e000640 1 -- 172.21.15.67:0/3859315232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) ==== 138+0+0 (secure 0 0 0) 0x7f6194058e20 con 0x7f61a40cce80 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 msgr2=0x7f618806ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 0x7f618806ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 msgr2=0x7f61a4159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f619400c3c0 tx=0x7f619400c880 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 shutdown_connections 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f618806ca60 0x7f618806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61a40cfe00 0x7f61a414ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cce80 0x7f61a4159a30 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 --2- 172.21.15.67:0/3859315232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61a406e830 0x7f61a416ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 >> 172.21.15.67:0/3859315232 conn(0x7f61a4067ee0 msgr2=0x7f61a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 shutdown_connections 2024-04-17T08:50:57.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.578+0000 7f61ac800640 1 -- 172.21.15.67:0/3859315232 wait complete. 2024-04-17T08:50:57.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:50:57.664 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:57.665+0000 7f4a6c9d9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:50:57.714 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:57.714+0000 7f9104fb8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:50:57.719 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:57.719+0000 7f4a6c9d9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:50:57.776 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:57.777+0000 7f9104fb8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:50:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.855+0000 7f0c53e00640 1 Processor -- start 2024-04-17T08:50:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.855+0000 7f0c53e00640 1 -- start start 2024-04-17T08:50:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c5415b690 0x7f0c540ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 0x7f0c540ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cf580 0x7f0c540cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0c540d04e0 con 0x7f0c5415b690 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0c540cebf0 con 0x7f0c540cefc0 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.856+0000 7f0c53e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0c540cfed0 con 0x7f0c540cf580 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c5415b690 0x7f0c540ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c5415b690 0x7f0c540ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48498/0 (socket says 172.21.15.67:48498) 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 -- 172.21.15.67:0/1730898385 learned_addr learned my addr 172.21.15.67:0/1730898385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52000640 1 --2- 172.21.15.67:0/1730898385 >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 0x7f0c540ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 -- 172.21.15.67:0/1730898385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cf580 msgr2=0x7f0c540cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 --2- 172.21.15.67:0/1730898385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cf580 0x7f0c540cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 -- 172.21.15.67:0/1730898385 >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 msgr2=0x7f0c540ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 --2- 172.21.15.67:0/1730898385 >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 0x7f0c540ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52a00640 1 -- 172.21.15.67:0/1730898385 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c540cbf70 con 0x7f0c5415b690 2024-04-17T08:50:57.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.857+0000 7f0c52000640 1 --2- 172.21.15.67:0/1730898385 >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 0x7f0c540ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.858+0000 7f0c52a00640 1 --2- 172.21.15.67:0/1730898385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c540ccfd0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f0c540d0080 tx=0x7f0c4c02e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cff0e48a645fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c51600640 1 -- 172.21.15.67:0/1730898385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c02ea70 con 0x7f0c5415b690 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c51600640 1 -- 172.21.15.67:0/1730898385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c02ec10 con 0x7f0c5415b690 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c51600640 1 -- 172.21.15.67:0/1730898385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c009600 con 0x7f0c5415b690 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 -- 172.21.15.67:0/1730898385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 msgr2=0x7f0c540ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 --2- 172.21.15.67:0/1730898385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c540ccfd0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f0c540d0080 tx=0x7f0c4c02e0d0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 -- 172.21.15.67:0/1730898385 shutdown_connections 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 --2- 172.21.15.67:0/1730898385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cf580 0x7f0c540cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 --2- 172.21.15.67:0/1730898385 >> v2:172.21.15.187:3300/0 conn(0x7f0c540cefc0 0x7f0c540ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 --2- 172.21.15.67:0/1730898385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c540ccfd0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.859+0000 7f0c53e00640 1 -- 172.21.15.67:0/1730898385 >> 172.21.15.67:0/1730898385 conn(0x7f0c540b2fe0 msgr2=0x7f0c540b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:57.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.860+0000 7f0c53e00640 1 -- 172.21.15.67:0/1730898385 shutdown_connections 2024-04-17T08:50:57.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.860+0000 7f0c53e00640 1 -- 172.21.15.67:0/1730898385 wait complete. 2024-04-17T08:50:57.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.861+0000 7f0c53e00640 1 Processor -- start 2024-04-17T08:50:57.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.861+0000 7f0c53e00640 1 -- start start 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 0x7f0c5414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c540cf580 0x7f0c5406fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0c540cc7d0 con 0x7f0c5415b690 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0c540cc4f0 con 0x7f0c540cf580 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.862+0000 7f0c53e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0c540ccab0 con 0x7f0c540cefc0 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48514/0 (socket says 172.21.15.67:48514) 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 -- 172.21.15.67:0/2949734198 learned_addr learned my addr 172.21.15.67:0/2949734198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:57.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c52000640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c540cf580 0x7f0c5406fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c52a00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 0x7f0c5414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 msgr2=0x7f0c5414d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 0x7f0c5414d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c540cf580 msgr2=0x7f0c5406fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c540cf580 0x7f0c5406fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 -- 172.21.15.67:0/2949734198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c540c7580 con 0x7f0c5415b690 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c52a00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 0x7f0c5414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.863+0000 7f0c53400640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009fd0 tx=0x7f0c4400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.864+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44011830 con 0x7f0c5415b690 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.864+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c44016070 con 0x7f0c5415b690 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.864+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c44011bf0 con 0x7f0c5415b690 2024-04-17T08:50:57.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.864+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c540c7860 con 0x7f0c5415b690 2024-04-17T08:50:57.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.864+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5414a5a0 con 0x7f0c5415b690 2024-04-17T08:50:57.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.865+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c540755d0 con 0x7f0c5415b690 2024-04-17T08:50:57.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.866+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 130) ==== 91159+0+0 (secure 0 0 0) 0x7f0c4401b460 con 0x7f0c5415b690 2024-04-17T08:50:57.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.867+0000 7f0c4aa00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 0x7f0c3006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:57.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.868+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7f0c4408bf00 con 0x7f0c5415b690 2024-04-17T08:50:57.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.868+0000 7f0c52a00640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 msgr2=0x7f0c3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:57.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.868+0000 7f0c52a00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 0x7f0c3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:57.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:57.872+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0c44059e00 con 0x7f0c5415b690 2024-04-17T08:50:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.069+0000 7f0c52a00640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 msgr2=0x7f0c3006ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.069+0000 7f0c52a00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 0x7f0c3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:58.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.078+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0c540b3b00 con 0x7f0c5415b690 2024-04-17T08:50:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.085+0000 7f0c4aa00640 1 -- 172.21.15.67:0/2949734198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) ==== 133+0+0 (secure 0 0 0) 0x7f0c44059700 con 0x7f0c5415b690 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.087+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 msgr2=0x7f0c3006ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.087+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 0x7f0c3006ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.087+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 msgr2=0x7f0c54070370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.087+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009fd0 tx=0x7f0c4400b8e0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 shutdown_connections 2024-04-17T08:50:58.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7f0c3006ca60 0x7f0c3006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5415b690 0x7f0c54070370 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c540cf580 0x7f0c5406fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 --2- 172.21.15.67:0/2949734198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c540cefc0 0x7f0c5414d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 >> 172.21.15.67:0/2949734198 conn(0x7f0c540b2fe0 msgr2=0x7f0c540d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 shutdown_connections 2024-04-17T08:50:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.088+0000 7f0c53e00640 1 -- 172.21.15.67:0/2949734198 wait complete. 2024-04-17T08:50:58.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:50:58.105 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-17T08:50:58.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:50:58.172 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.173+0000 7f4a6c9d9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:50:58.226 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.227+0000 7f4a6c9d9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:50:58.262 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.263+0000 7f9104fb8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:50:58.276 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.277+0000 7f4a6c9d9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:50:58.319 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.319+0000 7f9104fb8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:50:58.338 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.339+0000 7f4a6c9d9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:50:58.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.372+0000 7fa8bcc00640 1 Processor -- start 2024-04-17T08:50:58.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.372+0000 7fa8bcc00640 1 -- start start 2024-04-17T08:50:58.373 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.374+0000 7f9104fb8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:50:58.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8bcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b80b3800 0x7fa8b80cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8bcc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 0x7fa8b8155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8bcc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b815f890 con 0x7fa8b80b3be0 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8bcc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b81629b0 con 0x7fa8b80b3800 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8bcc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b815fb70 con 0x7fa8b8162e40 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8b7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b80b3800 0x7fa8b80cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8b7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b80b3800 0x7fa8b80cfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48516/0 (socket says 172.21.15.67:48516) 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.374+0000 7fa8b7400640 1 -- 172.21.15.67:0/3695435529 learned_addr learned my addr 172.21.15.67:0/3695435529 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:58.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 0x7fa8b8155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7400640 1 -- 172.21.15.67:0/3695435529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 msgr2=0x7fa8b8155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7400640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 0x7fa8b8155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7400640 1 -- 172.21.15.67:0/3695435529 >> v1:172.21.15.187:6789/0 conn(0x7fa8b80b3be0 legacy=0x7fa8b815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7400640 1 -- 172.21.15.67:0/3695435529 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b815fe50 con 0x7fa8b80b3800 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.375+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 0x7fa8b8155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.376+0000 7fa8b7400640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b80cfc80 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009870 tx=0x7fa8a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0f94c7b5a6dc71 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.376+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3695435529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8035020 con 0x7fa8b80b3800 2024-04-17T08:50:58.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.376+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3695435529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a8031640 con 0x7fa8b80b3800 2024-04-17T08:50:58.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.376+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3695435529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a802d040 con 0x7fa8b80b3800 2024-04-17T08:50:58.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 -- 172.21.15.67:0/3695435529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 msgr2=0x7fa8b80cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:58.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b80cfc80 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009870 tx=0x7fa8a802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 -- 172.21.15.67:0/3695435529 shutdown_connections 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b8162e40 0x7fa8b8155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 --2- 172.21.15.67:0/3695435529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b80cfc80 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 -- 172.21.15.67:0/3695435529 >> 172.21.15.67:0/3695435529 conn(0x7fa8b8067600 msgr2=0x7fa8b80679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 -- 172.21.15.67:0/3695435529 shutdown_connections 2024-04-17T08:50:58.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.377+0000 7fa8bcc00640 1 -- 172.21.15.67:0/3695435529 wait complete. 2024-04-17T08:50:58.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.379+0000 7fa8bcc00640 1 Processor -- start 2024-04-17T08:50:58.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.379+0000 7fa8bcc00640 1 -- start start 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b80b3be0 0x7fa8b81e8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b8162e40 0x7fa8b80cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b81632d0 con 0x7fa8b80b3800 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b80b4040 con 0x7fa8b8162e40 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8bcc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b8163450 con 0x7fa8b80b3be0 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8b7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:58.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8b7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48518/0 (socket says 172.21.15.67:48518) 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.380+0000 7fa8b7400640 1 -- 172.21.15.67:0/444396454 learned_addr learned my addr 172.21.15.67:0/444396454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b80b3be0 0x7fa8b81e8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b80b3be0 msgr2=0x7fa8b81e8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b80b3be0 0x7fa8b81e8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b8162e40 0x7fa8b80cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b8162e40 msgr2=0x7fa8b80cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8b8162e40 0x7fa8b80cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b80c9a10 con 0x7fa8b80b3800 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8b80b3be0 0x7fa8b81e8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8b7400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b815d230 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802b930 tx=0x7fa8a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.381+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8040070 con 0x7fa8b80b3800 2024-04-17T08:50:58.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.382+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a8039c10 con 0x7fa8b80b3800 2024-04-17T08:50:58.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.382+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b81e9830 con 0x7fa8b80b3800 2024-04-17T08:50:58.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.382+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8035040 con 0x7fa8b80b3800 2024-04-17T08:50:58.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.382+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b80c9cb0 con 0x7fa8b80b3800 2024-04-17T08:50:58.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.383+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b8068a10 con 0x7fa8b80b3800 2024-04-17T08:50:58.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.384+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 130) ==== 91159+0+0 (secure 0 0 0) 0x7fa8a804b060 con 0x7fa8b80b3800 2024-04-17T08:50:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.385+0000 7fa8af400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:50:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.385+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6601+0+0 (secure 0 0 0) 0x7fa8a80adb90 con 0x7fa8b80b3800 2024-04-17T08:50:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.385+0000 7fa8afe00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 msgr2=0x7fa89406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.385+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:50:58.388 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.389+0000 7f4a6c9d9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:50:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.390+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa8a802f300 con 0x7fa8b80b3800 2024-04-17T08:50:58.440 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.441+0000 7f9104fb8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:50:58.493 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.494+0000 7f9104fb8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:50:58.517 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.517+0000 7f4a6c9d9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:50:58.579 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.579+0000 7f4a6c9d9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.590+0000 7fa8afe00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 msgr2=0x7fa89406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.590+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:50:58.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.617+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7fa8b80d5780 con 0x7fa89406ca60 2024-04-17T08:50:58.629 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.630+0000 7f4a6c9d9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:50:58.630 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.631+0000 7f9104fb8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:50:58.686 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.687+0000 7f4a6c9d9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:50:58.696 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.697+0000 7f9104fb8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:50:58.737 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:58.737+0000 7f4a6c9d9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:50:58.749 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.750+0000 7f9104fb8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:50:58.809 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.809+0000 7f9104fb8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:50:58.862 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:58.863+0000 7f9104fb8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:50:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.991+0000 7fa8afe00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 msgr2=0x7fa89406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:58.991+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:50:59.118 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.118+0000 7f4a6c9d9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:50:59.267 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.267+0000 7f9104fb8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:50:59.276 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.277+0000 7f4a6c9d9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:50:59.385 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.386+0000 7f4a6c9d9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:50:59.436 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.436+0000 7f9104fb8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:50:59.436 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.437+0000 7f4a6c9d9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:50:59.550 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.550+0000 7f9104fb8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:50:59.562 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.563+0000 7f4a6c9d9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:59.603 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.604+0000 7f9104fb8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:50:59.669 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.670+0000 7f4a6c9d9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:50:59.735 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.736+0000 7f9104fb8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:59.792+0000 7fa8afe00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 msgr2=0x7fa89406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/150239855 2024-04-17T08:50:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:50:59.792+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:50:59.844 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.844+0000 7f4a6c9d9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:50:59.849 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:50:59.849+0000 7f9104fb8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:50:59.893 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:50:59.893+0000 7f4a6c9d9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:00.028 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:00.029+0000 7f4a6c9d9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:51:00.033 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:00.033+0000 7f9104fb8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:51:00.085 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:00.086+0000 7f9104fb8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:00.229 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:00.230+0000 7f9104fb8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:51:00.855 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:00.855+0000 7f4a6c9d9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:51:01.105 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:01.105+0000 7f9104fb8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:51:01.158 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.159+0000 7f4a6c9d9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:51:01.215 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.216+0000 7f4a6c9d9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:51:01.348 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.348+0000 7f4a6c9d9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:51:01.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:01.369+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 131) ==== 90768+0+0 (secure 0 0 0) 0x7fa8a8038020 con 0x7fa8b80b3800 2024-04-17T08:51:01.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:01.370+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 msgr2=0x7fa89406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:01.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:01.370+0000 7fa8af400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/150239855,v1:172.21.15.187:6833/150239855] conn(0x7fa89406ca60 0x7fa89406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:01.462 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:01.463+0000 7f9104fb8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:51:01.527 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:01.528+0000 7f9104fb8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:51:01.673 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:01.674+0000 7f9104fb8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:51:01.907 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.907+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.907 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.907+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.907 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.907+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.907 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.907+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.907 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.907+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.908 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.908+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.908 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.908+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.908 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.908+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.908 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.908+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.908 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.908+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.925 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.926+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.925 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.926+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.925 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.926+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.926+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.926+0000 7f49df600640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.927+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.927+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.927+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.927 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.927+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:01.927 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:01.927+0000 7f49e2800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:02.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.383+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 132) ==== 90895+0+0 (secure 0 0 0) 0x7fa8a8077b30 con 0x7fa8b80b3800 2024-04-17T08:51:02.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.384+0000 7fa8af400640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fa894076fa0 0x7fa894079390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.384+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7fa8a804aeb0 con 0x7fa894076fa0 2024-04-17T08:51:02.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.395+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fa894076fa0 0x7fa894079390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.404+0000 7fa8afe00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fa894076fa0 0x7fa894079390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80ce7f0 tx=0x7fa8a0007db0 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:02.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.598+0000 7fa8af400640 1 -- 172.21.15.67:0/444396454 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+53 (secure 0 0 0) 0x7fa8a804aeb0 con 0x7fa894076fa0 2024-04-17T08:51:02.598 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fa894076fa0 msgr2=0x7fa894079390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fa894076fa0 0x7fa894079390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80ce7f0 tx=0x7fa8a0007db0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 msgr2=0x7fa8b815d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 --2- 172.21.15.67:0/444396454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80b3800 0x7fa8b815d230 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802b930 tx=0x7fa8a80046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8b7e00640 1 -- 172.21.15.67:0/444396454 reap_dead start 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 shutdown_connections 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.601+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 >> 172.21.15.67:0/444396454 conn(0x7fa8b8067600 msgr2=0x7fa8b80d49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.602+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 shutdown_connections 2024-04-17T08:51:02.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.602+0000 7fa8bcc00640 1 -- 172.21.15.67:0/444396454 wait complete. 2024-04-17T08:51:02.623 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-04-17T08:51:02.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.854+0000 7f017be00640 1 Processor -- start 2024-04-17T08:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.854+0000 7f017be00640 1 -- start start 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.855+0000 7f017be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c142e10 0x7f017c071d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.855+0000 7f017be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f017c06f220 0x7f017c072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.855+0000 7f017be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017c1612d0 con 0x7f017c1460a0 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.855+0000 7f017be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f017c1431f0 con 0x7f017c06f220 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.855+0000 7f017be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f017c160e40 con 0x7f017c142e10 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f017c06f220 0x7f017c072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f017c06f220 0x7f017c072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48530/0 (socket says 172.21.15.67:48530) 2024-04-17T08:51:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 -- 172.21.15.67:0/1258013240 learned_addr learned my addr 172.21.15.67:0/1258013240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 -- 172.21.15.67:0/1258013240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c142e10 msgr2=0x7f017c071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 --2- 172.21.15.67:0/1258013240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c142e10 0x7f017c071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 -- 172.21.15.67:0/1258013240 >> v1:172.21.15.187:6789/0 conn(0x7f017c1460a0 legacy=0x7f017c06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.856+0000 7f017a000640 1 -- 172.21.15.67:0/1258013240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c160510 con 0x7f017c06f220 2024-04-17T08:51:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.857+0000 7f017a000640 1 --2- 172.21.15.67:0/1258013240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c06f220 0x7f017c072260 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f0164009870 tx=0x7f016402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcf277d1f1a16ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f0179600640 1 -- 172.21.15.67:0/1258013240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640317c0 con 0x7f017c06f220 2024-04-17T08:51:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f0179600640 1 -- 172.21.15.67:0/1258013240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0164037070 con 0x7f017c06f220 2024-04-17T08:51:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f0179600640 1 -- 172.21.15.67:0/1258013240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164031b80 con 0x7f017c06f220 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 -- 172.21.15.67:0/1258013240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c06f220 msgr2=0x7f017c072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 --2- 172.21.15.67:0/1258013240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c06f220 0x7f017c072260 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f0164009870 tx=0x7f016402b860 comp rx=0 tx=0).stop 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 -- 172.21.15.67:0/1258013240 shutdown_connections 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 --2- 172.21.15.67:0/1258013240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c06f220 0x7f017c072260 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 --2- 172.21.15.67:0/1258013240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c142e10 0x7f017c071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 -- 172.21.15.67:0/1258013240 >> 172.21.15.67:0/1258013240 conn(0x7f017c06b8c0 msgr2=0x7f017c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 -- 172.21.15.67:0/1258013240 shutdown_connections 2024-04-17T08:51:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.858+0000 7f017be00640 1 -- 172.21.15.67:0/1258013240 wait complete. 2024-04-17T08:51:02.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.860+0000 7f017be00640 1 Processor -- start 2024-04-17T08:51:02.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.860+0000 7f017be00640 1 -- start start 2024-04-17T08:51:02.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c1460a0 0x7f017c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f017c15e4b0 0x7f017c0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f017c1444b0 con 0x7f017c142e10 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f017c160ad0 con 0x7f017c15e4b0 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.861+0000 7f017be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f017c144630 con 0x7f017c1460a0 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48546/0 (socket says 172.21.15.67:48546) 2024-04-17T08:51:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 -- 172.21.15.67:0/1675493247 learned_addr learned my addr 172.21.15.67:0/1675493247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:02.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017a000640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c1460a0 0x7f017c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017b400640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f017c15e4b0 0x7f017c0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 -- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c1460a0 msgr2=0x7f017c15df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c1460a0 0x7f017c15df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 -- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f017c15e4b0 msgr2=0x7f017c0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f017c15e4b0 0x7f017c0b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 -- 172.21.15.67:0/1675493247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c0ad970 con 0x7f017c142e10 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.862+0000 7f017aa00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b4a0 tx=0x7f016c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.863+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c014070 con 0x7f017c142e10 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.863+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c010610 con 0x7f017c142e10 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.863+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c010a00 con 0x7f017c142e10 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.863+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017c0adc50 con 0x7f017c142e10 2024-04-17T08:51:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.863+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017c0b5050 con 0x7f017c142e10 2024-04-17T08:51:02.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.864+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017c15e8c0 con 0x7f017c142e10 2024-04-17T08:51:02.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.865+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 132) ==== 90895+0+0 (secure 0 0 0) 0x7f016c017070 con 0x7f017c142e10 2024-04-17T08:51:02.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.866+0000 7f0172a00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 0x7f015806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:02.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.866+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f016c08b750 con 0x7f017c142e10 2024-04-17T08:51:02.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.866+0000 7f017a000640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 0x7f015806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:02.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.868+0000 7f017a000640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 0x7f015806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0164036000 tx=0x7f016403d040 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:02.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:02.871+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f017c15e8c0 con 0x7f017c142e10 2024-04-17T08:51:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.099+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f017c068a10 con 0x7f015806c790 2024-04-17T08:51:03.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.235+0000 7f0172a00640 1 -- 172.21.15.67:0/1675493247 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f017c068a10 con 0x7f015806c790 2024-04-17T08:51:03.234 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-04-17T08:51:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 msgr2=0x7f015806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 0x7f015806ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0164036000 tx=0x7f016403d040 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 msgr2=0x7f017c13ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b4a0 tx=0x7f016c00b960 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 shutdown_connections 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f015806c790 0x7f015806ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f017c15e4b0 0x7f017c0b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f017c1460a0 0x7f017c15df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 --2- 172.21.15.67:0/1675493247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017c142e10 0x7f017c13ad90 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 >> 172.21.15.67:0/1675493247 conn(0x7f017c06b8c0 msgr2=0x7f017c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 shutdown_connections 2024-04-17T08:51:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.238+0000 7f017be00640 1 -- 172.21.15.67:0/1675493247 wait complete. 2024-04-17T08:51:03.262 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-04-17T08:51:03.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:51:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.503+0000 7f4fcb200640 1 Processor -- start 2024-04-17T08:51:03.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.504+0000 7f4fcb200640 1 -- start start 2024-04-17T08:51:03.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fcb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fcc0b63c0 0x7f4fcc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:03.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fcb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fcc15d060 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fcb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fcc15d340 con 0x7f4fcc072570 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fcb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4fcc0cfe90 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fc9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4fcc0cbc00 0x7f4fcc071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47128/0 (socket says 172.21.15.67:47128) 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fc9e00640 1 -- 172.21.15.67:0/1572799526 learned_addr learned my addr 172.21.15.67:0/1572799526 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.505+0000 7f4fca800640 1 --2- 172.21.15.67:0/1572799526 >> v2:172.21.15.67:3300/0 conn(0x7f4fcc0b63c0 0x7f4fcc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372993775 0 0) 0x7f4fcc15d060 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fb0003520 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fca800640 1 -- 172.21.15.67:0/1572799526 >> v1:172.21.15.187:6789/0 conn(0x7f4fcc072570 legacy=0x7f4fcc0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985915136 0 0) 0x7f4fcc15d340 con 0x7f4fcc072570 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fca800640 1 -- 172.21.15.67:0/1572799526 >> v1:172.21.15.67:6789/0 conn(0x7f4fcc0cbc00 legacy=0x7f4fcc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.506+0000 7f4fca800640 1 -- 172.21.15.67:0/1572799526 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc15d620 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.507+0000 7f4fca800640 1 --2- 172.21.15.67:0/1572799526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc0cf950 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009d90 tx=0x7f4fc402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2befd65dee06c18a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.507+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4039070 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.507+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc402ccf0 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.507+0000 7f4fc9400640 1 -- 172.21.15.67:0/1572799526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4034660 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 -- 172.21.15.67:0/1572799526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0b63c0 msgr2=0x7f4fcc0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 --2- 172.21.15.67:0/1572799526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc0cf950 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009d90 tx=0x7f4fc402bf40 comp rx=0 tx=0).stop 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 -- 172.21.15.67:0/1572799526 shutdown_connections 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 --2- 172.21.15.67:0/1572799526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc0cf950 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 -- 172.21.15.67:0/1572799526 >> 172.21.15.67:0/1572799526 conn(0x7f4fcc06bef0 msgr2=0x7f4fcc0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 -- 172.21.15.67:0/1572799526 shutdown_connections 2024-04-17T08:51:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.508+0000 7f4fcb200640 1 -- 172.21.15.67:0/1572799526 wait complete. 2024-04-17T08:51:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.510+0000 7f4fcb200640 1 Processor -- start 2024-04-17T08:51:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.510+0000 7f4fcb200640 1 -- start start 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fcc072570 0x7f4fcc06e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc0d02d0 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc15dc70 con 0x7f4fcc0b63c0 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fcb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fcc0cbfe0 con 0x7f4fcc072570 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38638/0 (socket says 172.21.15.67:38638) 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.511+0000 7f4fca800640 1 -- 172.21.15.67:0/2844577412 learned_addr learned my addr 172.21.15.67:0/2844577412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fc3e00640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fc9e00640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fcc072570 0x7f4fcc06e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 -- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fcc072570 msgr2=0x7f4fcc06e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fcc072570 0x7f4fcc06e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 -- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fcc0b63c0 msgr2=0x7f4fcc14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 -- 172.21.15.67:0/2844577412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc06eb50 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.512+0000 7f4fca800640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4037040 tx=0x7f4fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.513+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4039050 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.513+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc403a040 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.513+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc06ee30 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.513+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4009450 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.513+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc06f3f0 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.514+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc068a10 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.515+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 133) ==== 91262+0+0 (secure 0 0 0) 0x7f4fc4047020 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.516+0000 7f4fc2000640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 0x7f4fa806f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.516+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f4fc40ae720 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.516+0000 7f4fc9e00640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 0x7f4fa806f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:03.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.517+0000 7f4fc9e00640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 0x7f4fa806f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc005880 tx=0x7f4fbc009040 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:03.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.521+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4fc404e020 con 0x7f4fcc0cbc00 2024-04-17T08:51:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.751+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f4fcc074ec0 con 0x7f4fa806cb80 2024-04-17T08:51:03.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.914+0000 7f4fc2000640 1 -- 172.21.15.67:0/2844577412 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7f4fcc074ec0 con 0x7f4fa806cb80 2024-04-17T08:51:03.913 INFO:teuthology.orchestra.run.smithi067.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-04-17T08:51:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.918+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 msgr2=0x7f4fa806f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.918+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 0x7f4fa806f040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc005880 tx=0x7f4fbc009040 comp rx=0 tx=0).stop 2024-04-17T08:51:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.918+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 msgr2=0x7f4fcc0a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.918+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4037040 tx=0x7f4fc4004a40 comp rx=0 tx=0).stop 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 shutdown_connections 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f4fa806cb80 0x7f4fa806f040 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc0cbc00 0x7f4fcc0a94c0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fcc0b63c0 0x7f4fcc14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 --2- 172.21.15.67:0/2844577412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fcc072570 0x7f4fcc06e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 >> 172.21.15.67:0/2844577412 conn(0x7f4fcc06bef0 msgr2=0x7f4fcc160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 shutdown_connections 2024-04-17T08:51:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:03.919+0000 7f4fcb200640 1 -- 172.21.15.67:0/2844577412 wait complete. 2024-04-17T08:51:03.943 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-04-17T08:51:03.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-17T08:51:04.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.175+0000 7f921de00640 1 Processor -- start 2024-04-17T08:51:04.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.175+0000 7f921de00640 1 -- start start 2024-04-17T08:51:04.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92180b3800 0x7f92180cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 0x7f9218155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921815f890 con 0x7f92180b3be0 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f92181629b0 con 0x7f92180b3800 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f921815fb70 con 0x7f9218162e40 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f921ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92180b3800 0x7f92180cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f9217e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92180b3be0 0x7f921815f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47138/0 (socket says 172.21.15.67:47138) 2024-04-17T08:51:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.177+0000 7f9217e00640 1 -- 172.21.15.67:0/614405524 learned_addr learned my addr 172.21.15.67:0/614405524 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921d400640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 0x7f9218155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921ca00640 1 -- 172.21.15.67:0/614405524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 msgr2=0x7f9218155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921ca00640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 0x7f9218155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921ca00640 1 -- 172.21.15.67:0/614405524 >> v1:172.21.15.67:6789/0 conn(0x7f92180b3be0 legacy=0x7f921815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921ca00640 1 -- 172.21.15.67:0/614405524 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921815fe50 con 0x7f92180b3800 2024-04-17T08:51:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.178+0000 7f921d400640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 0x7f9218155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:04.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.179+0000 7f921ca00640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cfc80 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f920c0097a0 tx=0x7f920c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd813a56883883ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.179+0000 7f9217400640 1 -- 172.21.15.67:0/614405524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c035020 con 0x7f92180b3800 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.179+0000 7f9217400640 1 -- 172.21.15.67:0/614405524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c031810 con 0x7f92180b3800 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.179+0000 7f9217400640 1 -- 172.21.15.67:0/614405524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c031be0 con 0x7f92180b3800 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 -- 172.21.15.67:0/614405524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 msgr2=0x7f92180cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cfc80 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f920c0097a0 tx=0x7f920c02b8b0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 -- 172.21.15.67:0/614405524 shutdown_connections 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9218162e40 0x7f9218155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 --2- 172.21.15.67:0/614405524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cfc80 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 -- 172.21.15.67:0/614405524 >> 172.21.15.67:0/614405524 conn(0x7f9218067600 msgr2=0x7f92180679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 -- 172.21.15.67:0/614405524 shutdown_connections 2024-04-17T08:51:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.180+0000 7f921de00640 1 -- 172.21.15.67:0/614405524 wait complete. 2024-04-17T08:51:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.182+0000 7f921de00640 1 Processor -- start 2024-04-17T08:51:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.182+0000 7f921de00640 1 -- start start 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92180b3be0 0x7f92181e8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9218162e40 0x7f92181e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92181632d0 con 0x7f92180b3800 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f92180b4040 con 0x7f9218162e40 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9218163450 con 0x7f92180b3be0 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38652/0 (socket says 172.21.15.67:38652) 2024-04-17T08:51:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.183+0000 7f921ca00640 1 -- 172.21.15.67:0/353954487 learned_addr learned my addr 172.21.15.67:0/353954487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:04.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 -- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92180b3be0 msgr2=0x7f92181e8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92180b3be0 0x7f92181e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 -- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9218162e40 msgr2=0x7f92181e8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9218162e40 0x7f92181e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 -- 172.21.15.67:0/353954487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92181e98d0 con 0x7f92180b3800 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f921ca00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f920c034040 tx=0x7f920c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.184+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c040070 con 0x7f92180b3800 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.185+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c039cf0 con 0x7f92180b3800 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.185+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c035040 con 0x7f92180b3800 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.185+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92181e9bb0 con 0x7f92180b3800 2024-04-17T08:51:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.185+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92180cd210 con 0x7f92180b3800 2024-04-17T08:51:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.186+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9218068a10 con 0x7f92180b3800 2024-04-17T08:51:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.187+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 133) ==== 91262+0+0 (secure 0 0 0) 0x7f920c038020 con 0x7f92180b3800 2024-04-17T08:51:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.188+0000 7f9214c00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 0x7f91fc06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.188+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f920c0ae2b0 con 0x7f92180b3800 2024-04-17T08:51:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.188+0000 7f9217e00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 0x7f91fc06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.190+0000 7f9217e00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 0x7f91fc06f040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9208000f60 tx=0x7f9208007a80 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.192+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f920c07be00 con 0x7f92180b3800 2024-04-17T08:51:04.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.421+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92181e8710 con 0x7f91fc06cb80 2024-04-17T08:51:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.438+0000 7f9214c00640 1 -- 172.21.15.67:0/353954487 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+88 (secure 0 0 0) 0x7f92181e8710 con 0x7f91fc06cb80 2024-04-17T08:51:04.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:04.438 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-17T08:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 msgr2=0x7f91fc06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 0x7f91fc06f040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9208000f60 tx=0x7f9208007a80 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 msgr2=0x7f92180cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f920c034040 tx=0x7f920c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 shutdown_connections 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f91fc06cb80 0x7f91fc06f040 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9218162e40 0x7f92181e8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92180b3be0 0x7f92181e8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 --2- 172.21.15.67:0/353954487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92180b3800 0x7f92180cb080 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 >> 172.21.15.67:0/353954487 conn(0x7f9218067600 msgr2=0x7f92180b7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 shutdown_connections 2024-04-17T08:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.441+0000 7f921de00640 1 -- 172.21.15.67:0/353954487 wait complete. 2024-04-17T08:51:04.459 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-04-17T08:51:04.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-04-17T08:51:04.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.674+0000 7f3dd1600640 1 Processor -- start 2024-04-17T08:51:04.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.675+0000 7f3dd1600640 1 -- start start 2024-04-17T08:51:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.676+0000 7f3dd1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dcc0b3800 0x7f3dcc0cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.676+0000 7f3dd1600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3dcc0b3be0 0x7f3dcc15f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.676+0000 7f3dd1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc15f890 con 0x7f3dcc162e40 2024-04-17T08:51:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.676+0000 7f3dd1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3dcc1629b0 con 0x7f3dcc0b3800 2024-04-17T08:51:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.676+0000 7f3dd1600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3dcc15fb70 con 0x7f3dcc0b3be0 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dcc0b3800 0x7f3dcc0cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dd0c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3dcc162e40 0x7f3dcc155560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47144/0 (socket says 172.21.15.67:47144) 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dd0c00640 1 -- 172.21.15.67:0/1500959708 learned_addr learned my addr 172.21.15.67:0/1500959708 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcb400640 1 --2- 172.21.15.67:0/1500959708 >> v2:172.21.15.187:3300/0 conn(0x7f3dcc0b3be0 0x7f3dcc15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcbe00640 1 -- 172.21.15.67:0/1500959708 >> v2:172.21.15.187:3300/0 conn(0x7f3dcc0b3be0 msgr2=0x7f3dcc15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcbe00640 1 --2- 172.21.15.67:0/1500959708 >> v2:172.21.15.187:3300/0 conn(0x7f3dcc0b3be0 0x7f3dcc15f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcbe00640 1 -- 172.21.15.67:0/1500959708 >> v1:172.21.15.67:6789/0 conn(0x7f3dcc162e40 legacy=0x7f3dcc155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.677+0000 7f3dcbe00640 1 -- 172.21.15.67:0/1500959708 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc15fe50 con 0x7f3dcc0b3800 2024-04-17T08:51:04.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.679+0000 7f3dcbe00640 1 --2- 172.21.15.67:0/1500959708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc0b3800 0x7f3dcc0cfc80 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06d040 tx=0x7f3dbc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125ba74f351c1e51 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.679+0000 7f3dcaa00640 1 -- 172.21.15.67:0/1500959708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc039070 con 0x7f3dcc0b3800 2024-04-17T08:51:04.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.679+0000 7f3dcaa00640 1 -- 172.21.15.67:0/1500959708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc02cc70 con 0x7f3dcc0b3800 2024-04-17T08:51:04.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.679+0000 7f3dcaa00640 1 -- 172.21.15.67:0/1500959708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc034660 con 0x7f3dcc0b3800 2024-04-17T08:51:04.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 -- 172.21.15.67:0/1500959708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc0b3800 msgr2=0x7f3dcc0cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 --2- 172.21.15.67:0/1500959708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc0b3800 0x7f3dcc0cfc80 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06d040 tx=0x7f3dbc02bec0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 -- 172.21.15.67:0/1500959708 shutdown_connections 2024-04-17T08:51:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 --2- 172.21.15.67:0/1500959708 >> v2:172.21.15.187:3300/0 conn(0x7f3dcc0b3be0 0x7f3dcc15f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 --2- 172.21.15.67:0/1500959708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc0b3800 0x7f3dcc0cfc80 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 -- 172.21.15.67:0/1500959708 >> 172.21.15.67:0/1500959708 conn(0x7f3dcc067600 msgr2=0x7f3dcc0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 -- 172.21.15.67:0/1500959708 shutdown_connections 2024-04-17T08:51:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.680+0000 7f3dd1600640 1 -- 172.21.15.67:0/1500959708 wait complete. 2024-04-17T08:51:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.682+0000 7f3dd1600640 1 Processor -- start 2024-04-17T08:51:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.682+0000 7f3dd1600640 1 -- start start 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dcc0b3800 0x7f3dcc152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 0x7f3dcc0c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc15f560 con 0x7f3dcc162e40 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc0b4040 con 0x7f3dcc0b3be0 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd1600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3dcc1632d0 con 0x7f3dcc0b3800 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38672/0 (socket says 172.21.15.67:38672) 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dd0c00640 1 -- 172.21.15.67:0/2596414620 learned_addr learned my addr 172.21.15.67:0/2596414620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dcbe00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dcc0b3800 0x7f3dcc152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.683+0000 7f3dcb400640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 0x7f3dcc0c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 -- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dcc0b3800 msgr2=0x7f3dcc152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dcc0b3800 0x7f3dcc152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 -- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 msgr2=0x7f3dcc0c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 0x7f3dcc0c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 -- 172.21.15.67:0/2596414620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc0cbde0 con 0x7f3dcc162e40 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dcb400640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 0x7f3dcc0c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd0c00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c450 tx=0x7f3dc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4017070 con 0x7f3dcc162e40 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc4005ce0 con 0x7f3dcc162e40 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4007270 con 0x7f3dcc162e40 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc0cc0c0 con 0x7f3dcc162e40 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.684+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc0cc680 con 0x7f3dcc162e40 2024-04-17T08:51:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.686+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc068a10 con 0x7f3dcc162e40 2024-04-17T08:51:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.687+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 133) ==== 91262+0+0 (secure 0 0 0) 0x7f3dc401a070 con 0x7f3dcc162e40 2024-04-17T08:51:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.688+0000 7f3dc3e00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 0x7f3da806f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.688+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f3dc408b6b0 con 0x7f3dcc162e40 2024-04-17T08:51:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.688+0000 7f3dcbe00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 0x7f3da806f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.689+0000 7f3dcbe00640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 0x7f3da806f070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc036000 tx=0x7f3dbc030000 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.692+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3dc4058c40 con 0x7f3dcc162e40 2024-04-17T08:51:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.929+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3dcc0cb6b0 con 0x7f3da806cbb0 2024-04-17T08:51:04.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.946+0000 7f3dc3e00640 1 -- 172.21.15.67:0/2596414620 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+353 (secure 0 0 0) 0x7f3dcc0cb6b0 con 0x7f3da806cbb0 2024-04-17T08:51:04.945 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:04.945 INFO:teuthology.orchestra.run.smithi067.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:51:02", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 msgr2=0x7f3da806f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 0x7f3da806f070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc036000 tx=0x7f3dbc030000 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 msgr2=0x7f3dcc0cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c450 tx=0x7f3dc400c910 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 shutdown_connections 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f3da806cbb0 0x7f3da806f070 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc162e40 0x7f3dcc0cd880 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.948+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3dcc0b3be0 0x7f3dcc0c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.949+0000 7f3dd1600640 1 --2- 172.21.15.67:0/2596414620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3dcc0b3800 0x7f3dcc152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.949+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 >> 172.21.15.67:0/2596414620 conn(0x7f3dcc067600 msgr2=0x7f3dcc074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.949+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 shutdown_connections 2024-04-17T08:51:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:04.949+0000 7f3dd1600640 1 -- 172.21.15.67:0/2596414620 wait complete. 2024-04-17T08:51:04.968 DEBUG:tasks.cephfs.test_snap_schedules:res=[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-04-17T00:00:00", "created": "2024-04-17T08:51:02", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-04-17T08:51:04.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:51:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.201+0000 7f0d46800640 1 Processor -- start 2024-04-17T08:51:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.201+0000 7f0d46800640 1 -- start start 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.202+0000 7f0d46800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40072d80 0x7f0d40071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.202+0000 7f0d46800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d400ce3b0 0x7f0d400cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.202+0000 7f0d46800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d400cea70 con 0x7f0d40145ac0 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.202+0000 7f0d46800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0d400ce840 con 0x7f0d400ce3b0 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.202+0000 7f0d46800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d400ced50 con 0x7f0d40072d80 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0d40145ac0 0x7f0d40072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47150/0 (socket says 172.21.15.67:47150) 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3f400640 1 -- 172.21.15.67:0/2197869318 learned_addr learned my addr 172.21.15.67:0/2197869318 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3fe00640 1 --2- 172.21.15.67:0/2197869318 >> v2:172.21.15.67:3300/0 conn(0x7f0d400ce3b0 0x7f0d400cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3ea00640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40072d80 0x7f0d40071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3e000640 1 -- 172.21.15.67:0/2197869318 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980317054 0 0) 0x7f0d400cea70 con 0x7f0d40145ac0 2024-04-17T08:51:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3fe00640 1 -- 172.21.15.67:0/2197869318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40072d80 msgr2=0x7f0d40071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:05.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3fe00640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40072d80 0x7f0d40071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3fe00640 1 -- 172.21.15.67:0/2197869318 >> v1:172.21.15.67:6789/0 conn(0x7f0d40145ac0 legacy=0x7f0d40072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.203+0000 7f0d3fe00640 1 -- 172.21.15.67:0/2197869318 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d400cf030 con 0x7f0d400ce3b0 2024-04-17T08:51:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d3fe00640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400cfb10 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009d90 tx=0x7f0d3802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76dc86f3350ceab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d3e000640 1 -- 172.21.15.67:0/2197869318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38039070 con 0x7f0d400ce3b0 2024-04-17T08:51:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d3e000640 1 -- 172.21.15.67:0/2197869318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3802cc50 con 0x7f0d400ce3b0 2024-04-17T08:51:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d3e000640 1 -- 172.21.15.67:0/2197869318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38034440 con 0x7f0d400ce3b0 2024-04-17T08:51:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d46800640 1 -- 172.21.15.67:0/2197869318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 msgr2=0x7f0d400cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.205+0000 7f0d46800640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400cfb10 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009d90 tx=0x7f0d3802bea0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 -- 172.21.15.67:0/2197869318 shutdown_connections 2024-04-17T08:51:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400cfb10 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 --2- 172.21.15.67:0/2197869318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40072d80 0x7f0d40071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 -- 172.21.15.67:0/2197869318 >> 172.21.15.67:0/2197869318 conn(0x7f0d40067ee0 msgr2=0x7f0d4006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 -- 172.21.15.67:0/2197869318 shutdown_connections 2024-04-17T08:51:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.206+0000 7f0d46800640 1 -- 172.21.15.67:0/2197869318 wait complete. 2024-04-17T08:51:05.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.207+0000 7f0d46800640 1 Processor -- start 2024-04-17T08:51:05.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.208+0000 7f0d46800640 1 -- start start 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d40072d80 0x7f0d4015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40145ac0 0x7f0d400cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d40145ea0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0d400cf5f0 con 0x7f0d40072d80 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d46800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0d400d0050 con 0x7f0d40145ac0 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d3ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d3ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38694/0 (socket says 172.21.15.67:38694) 2024-04-17T08:51:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d3ea00640 1 -- 172.21.15.67:0/2416188327 learned_addr learned my addr 172.21.15.67:0/2416188327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d3fe00640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40145ac0 0x7f0d400cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.209+0000 7f0d3f400640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d40072d80 0x7f0d4015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 -- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40145ac0 msgr2=0x7f0d400cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40145ac0 0x7f0d400cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 -- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d40072d80 msgr2=0x7f0d4015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d40072d80 0x7f0d4015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 -- 172.21.15.67:0/2416188327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d400ca710 con 0x7f0d400ce3b0 2024-04-17T08:51:05.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d3ea00640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00d800 tx=0x7f0d2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00b6c0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2c019070 con 0x7f0d400ce3b0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d400ca9f0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c00ba80 con 0x7f0d400ce3b0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.210+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d400cafb0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.211+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d40068ca0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.213+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 133) ==== 91262+0+0 (secure 0 0 0) 0x7f0d2c00bc20 con 0x7f0d400ce3b0 2024-04-17T08:51:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.214+0000 7f0d37400640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 0x7f0d2006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.214+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f0d2c08ca30 con 0x7f0d400ce3b0 2024-04-17T08:51:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.214+0000 7f0d3f400640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 0x7f0d2006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.215+0000 7f0d3f400640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 0x7f0d2006f070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d30000f60 tx=0x7f0d30007c00 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.219+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0d2c01c360 con 0x7f0d400ce3b0 2024-04-17T08:51:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.451+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f0d400cc6c0 con 0x7f0d2006cbb0 2024-04-17T08:51:05.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.454+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 134) ==== 91318+0+0 (secure 0 0 0) 0x7f0d2c0564f0 con 0x7f0d400ce3b0 2024-04-17T08:51:05.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.626+0000 7f0d37400640 1 -- 172.21.15.67:0/2416188327 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f0d400cc6c0 con 0x7f0d2006cbb0 2024-04-17T08:51:05.625 INFO:teuthology.orchestra.run.smithi067.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-04-17T08:51:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 msgr2=0x7f0d2006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 0x7f0d2006f070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d30000f60 tx=0x7f0d30007c00 comp rx=0 tx=0).stop 2024-04-17T08:51:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 msgr2=0x7f0d400c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00d800 tx=0x7f0d2c00dcc0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 shutdown_connections 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f0d2006cbb0 0x7f0d2006f070 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0d40145ac0 0x7f0d400cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d400ce3b0 0x7f0d400c9c90 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 --2- 172.21.15.67:0/2416188327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d40072d80 0x7f0d4015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 >> 172.21.15.67:0/2416188327 conn(0x7f0d40067ee0 msgr2=0x7f0d40144cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.629+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 shutdown_connections 2024-04-17T08:51:05.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.630+0000 7f0d46800640 1 -- 172.21.15.67:0/2416188327 wait complete. 2024-04-17T08:51:05.650 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-04-17T08:51:05.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:51:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.876+0000 7f621e000640 1 Processor -- start 2024-04-17T08:51:05.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.876+0000 7f621e000640 1 -- start start 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.877+0000 7f621e000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f62180cfc80 0x7f62180b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.877+0000 7f621e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f6218155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621e000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621815f890 con 0x7f62180b3800 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621e000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f62181629b0 con 0x7f62180cfc80 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621e000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f621815fb70 con 0x7f6218162e40 2024-04-17T08:51:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621cc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62180b3800 0x7f621815f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47152/0 (socket says 172.21.15.67:47152) 2024-04-17T08:51:05.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621cc00640 1 -- 172.21.15.67:0/88305992 learned_addr learned my addr 172.21.15.67:0/88305992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:05.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517694695 0 0) 0x7f621815f890 con 0x7f62180b3800 2024-04-17T08:51:05.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc003520 con 0x7f62180b3800 2024-04-17T08:51:05.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f621d600640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f6218155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.878+0000 7f620fe00640 1 --2- 172.21.15.67:0/88305992 >> v2:172.21.15.187:3300/0 conn(0x7f62180cfc80 0x7f62180b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1038991962 0 0) 0x7f61fc003520 con 0x7f62180b3800 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 msgr2=0x7f6218155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f6218155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 >> v2:172.21.15.187:3300/0 conn(0x7f62180cfc80 msgr2=0x7f62180b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 --2- 172.21.15.67:0/88305992 >> v2:172.21.15.187:3300/0 conn(0x7f62180cfc80 0x7f62180b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621815fe50 con 0x7f62180b3800 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6208001670 con 0x7f62180b3800 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 >> v1:172.21.15.67:6789/0 conn(0x7f62180b3800 legacy=0x7f621815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f61fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004030 0x7f61fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc004d40 0x7f61fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f62181629b0 con 0x7f61fc004d40 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f621815fb70 con 0x7f61fc004030 2024-04-17T08:51:05.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61fc00bbb0 con 0x7f6218162e40 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.879+0000 7f621d600640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc004d40 0x7f61fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f620fe00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004030 0x7f61fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621cc00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f61fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621d600640 1 -- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 msgr2=0x7f61fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621d600640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6218162e40 0x7f61fc0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621d600640 1 -- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004030 msgr2=0x7f61fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621d600640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004030 0x7f61fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f621d600640 1 -- 172.21.15.67:0/88305992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc003520 con 0x7f61fc004d40 2024-04-17T08:51:05.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.880+0000 7f620fe00640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004030 0x7f61fc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:05.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.881+0000 7f621d600640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc004d40 0x7f61fc0051c0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f6210007360 tx=0x7f61fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69770db668a4e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.881+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210039070 con 0x7f61fc004d40 2024-04-17T08:51:05.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.881+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621002ccd0 con 0x7f61fc004d40 2024-04-17T08:51:05.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.881+0000 7f6217e00640 1 -- 172.21.15.67:0/88305992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210009be0 con 0x7f61fc004d40 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 -- 172.21.15.67:0/88305992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc004d40 msgr2=0x7f61fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 --2- 172.21.15.67:0/88305992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc004d40 0x7f61fc0051c0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f6210007360 tx=0x7f61fc005830 comp rx=0 tx=0).stop 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621d600640 1 -- 172.21.15.67:0/88305992 reap_dead start 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 -- 172.21.15.67:0/88305992 shutdown_connections 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 -- 172.21.15.67:0/88305992 >> 172.21.15.67:0/88305992 conn(0x7f6218067600 msgr2=0x7f62180679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 -- 172.21.15.67:0/88305992 shutdown_connections 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.882+0000 7f621e000640 1 -- 172.21.15.67:0/88305992 wait complete. 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.884+0000 7f621e000640 1 Processor -- start 2024-04-17T08:51:05.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.884+0000 7f621e000640 1 -- start start 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 0x7f62180c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62180c84d0 0x7f62181493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6218163450 con 0x7f6218149c80 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f62181632d0 con 0x7f61fc004d40 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621e000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6218155560 con 0x7f62180c84d0 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38708/0 (socket says 172.21.15.67:38708) 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621d600640 1 -- 172.21.15.67:0/3680878976 learned_addr learned my addr 172.21.15.67:0/3680878976 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f621cc00640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 0x7f62180c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.885+0000 7f620fe00640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62180c84d0 0x7f62181493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 -- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62180c84d0 msgr2=0x7f62181493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62180c84d0 0x7f62181493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 -- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 msgr2=0x7f62180c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 0x7f62180c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 -- 172.21.15.67:0/3680878976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62180c8a40 con 0x7f6218149c80 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621cc00640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 0x7f62180c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:05.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621d600640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f6210037040 tx=0x7f6210004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210039030 con 0x7f6218149c80 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6210043070 con 0x7f6218149c80 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210009be0 con 0x7f6218149c80 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62180c8d20 con 0x7f6218149c80 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.886+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62180c92e0 con 0x7f6218149c80 2024-04-17T08:51:05.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.888+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6218068a10 con 0x7f6218149c80 2024-04-17T08:51:05.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.889+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 134) ==== 91318+0+0 (secure 0 0 0) 0x7f621003d7f0 con 0x7f6218149c80 2024-04-17T08:51:05.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.890+0000 7f6215600640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 0x7f61f406f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:05.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.890+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f62100ae730 con 0x7f6218149c80 2024-04-17T08:51:05.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.891+0000 7f621cc00640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 0x7f61f406f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:05.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.892+0000 7f621cc00640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 0x7f61f406f100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6208001050 tx=0x7f6208009040 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:05.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:05.895+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f621007c330 con 0x7f6218149c80 2024-04-17T08:51:06.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.130+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f621806da00 con 0x7f61f406cc40 2024-04-17T08:51:06.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.299+0000 7f6215600640 1 -- 172.21.15.67:0/3680878976 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+61 (secure 0 0 0) 0x7f621806da00 con 0x7f61f406cc40 2024-04-17T08:51:06.298 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 msgr2=0x7f61f406f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 0x7f61f406f100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6208001050 tx=0x7f6208009040 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 msgr2=0x7f62180ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f6210037040 tx=0x7f6210004ab0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 shutdown_connections 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f61f406cc40 0x7f61f406f100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218149c80 0x7f62180ca080 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f62180c84d0 0x7f62181493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 --2- 172.21.15.67:0/3680878976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61fc004d40 0x7f62180c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.302+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 >> 172.21.15.67:0/3680878976 conn(0x7f6218067600 msgr2=0x7f62180d46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.303+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 shutdown_connections 2024-04-17T08:51:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.303+0000 7f621e000640 1 -- 172.21.15.67:0/3680878976 wait complete. 2024-04-17T08:51:06.324 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-04-17T08:51:06.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-04-17T08:51:06.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.548+0000 7fb563200640 1 Processor -- start 2024-04-17T08:51:06.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.548+0000 7fb563200640 1 -- start start 2024-04-17T08:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb563200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 0x7fb56415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb563200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb56406f560 0x7fb5640b4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb563200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5641450d0 con 0x7fb56415c030 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb563200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb5640b34c0 con 0x7fb56406f560 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb563200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5640b37a0 con 0x7fb564144cf0 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb562800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb56406f560 0x7fb5640b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.550+0000 7fb562800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb56406f560 0x7fb5640b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38710/0 (socket says 172.21.15.67:38710) 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb562800640 1 -- 172.21.15.67:0/2902747084 learned_addr learned my addr 172.21.15.67:0/2902747084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb562800640 1 -- 172.21.15.67:0/2902747084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 msgr2=0x7fb56415baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb561e00640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 0x7fb56415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb562800640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 0x7fb56415baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb562800640 1 -- 172.21.15.67:0/2902747084 >> v1:172.21.15.187:6789/0 conn(0x7fb56415c030 legacy=0x7fb56406ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb562800640 1 -- 172.21.15.67:0/2902747084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5640b3a80 con 0x7fb56406f560 2024-04-17T08:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.551+0000 7fb561e00640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 0x7fb56415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.552+0000 7fb562800640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb5640b4560 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009e50 tx=0x7fb55c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c3d6662d795b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb561400640 1 -- 172.21.15.67:0/2902747084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c02ca80 con 0x7fb56406f560 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb561400640 1 -- 172.21.15.67:0/2902747084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb55c02cc20 con 0x7fb56406f560 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb561400640 1 -- 172.21.15.67:0/2902747084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0343f0 con 0x7fb56406f560 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 -- 172.21.15.67:0/2902747084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 msgr2=0x7fb5640b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb5640b4560 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009e50 tx=0x7fb55c02bee0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 -- 172.21.15.67:0/2902747084 shutdown_connections 2024-04-17T08:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb5640b4560 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 --2- 172.21.15.67:0/2902747084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb564144cf0 0x7fb56415baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 -- 172.21.15.67:0/2902747084 >> 172.21.15.67:0/2902747084 conn(0x7fb56406b8c0 msgr2=0x7fb56406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.553+0000 7fb563200640 1 -- 172.21.15.67:0/2902747084 shutdown_connections 2024-04-17T08:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.554+0000 7fb563200640 1 -- 172.21.15.67:0/2902747084 wait complete. 2024-04-17T08:51:06.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.556+0000 7fb563200640 1 Processor -- start 2024-04-17T08:51:06.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.556+0000 7fb563200640 1 -- start start 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb564144cf0 0x7fb56413fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb56415c030 0x7fb56413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5640b4aa0 con 0x7fb56406f560 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb5640b4040 con 0x7fb564144cf0 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb563200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb5640b4c20 con 0x7fb56415c030 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb561e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb561e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38716/0 (socket says 172.21.15.67:38716) 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb561e00640 1 -- 172.21.15.67:0/2363219769 learned_addr learned my addr 172.21.15.67:0/2363219769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb55be00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb564144cf0 0x7fb56413fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.558+0000 7fb562800640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb56415c030 0x7fb56413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 -- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb56415c030 msgr2=0x7fb56413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb56415c030 0x7fb56413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 -- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb564144cf0 msgr2=0x7fb56413fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb564144cf0 0x7fb56413fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 -- 172.21.15.67:0/2363219769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb564140450 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb561e00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fb55400c7f0 tx=0x7fb55400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb554015070 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb554007a30 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb564140730 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.559+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb554007e00 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.560+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb564140cf0 con 0x7fb56406f560 2024-04-17T08:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.561+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb564068a10 con 0x7fb56406f560 2024-04-17T08:51:06.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.562+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 134) ==== 91318+0+0 (secure 0 0 0) 0x7fb554018070 con 0x7fb56406f560 2024-04-17T08:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.563+0000 7fb55a000640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 0x7fb54006f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.563+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7fb55408c5c0 con 0x7fb56406f560 2024-04-17T08:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.563+0000 7fb55be00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 0x7fb54006f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.565+0000 7fb55be00640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 0x7fb54006f0d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb54c006fd0 tx=0x7fb54c008040 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:06.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.567+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb55405a100 con 0x7fb56406f560 2024-04-17T08:51:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.796+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 --> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7fb564143da0 con 0x7fb54006cc10 2024-04-17T08:51:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.976+0000 7fb55a000640 1 -- 172.21.15.67:0/2363219769 <== mgr.8821 v2:172.21.15.187:6832/3872755981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7fb564143da0 con 0x7fb54006cc10 2024-04-17T08:51:06.976 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-04-17T08:51:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.979+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 msgr2=0x7fb54006f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.979+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 0x7fb54006f0d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb54c006fd0 tx=0x7fb54c008040 comp rx=0 tx=0).stop 2024-04-17T08:51:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.979+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 msgr2=0x7fb564138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.979+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fb55400c7f0 tx=0x7fb55400cbf0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 shutdown_connections 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb54006cc10 0x7fb54006f0d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb56415c030 0x7fb56413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb564144cf0 0x7fb56413fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 --2- 172.21.15.67:0/2363219769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb56406f560 0x7fb564138f00 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 >> 172.21.15.67:0/2363219769 conn(0x7fb56406b8c0 msgr2=0x7fb5640cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 shutdown_connections 2024-04-17T08:51:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:06.980+0000 7fb563200640 1 -- 172.21.15.67:0/2363219769 wait complete. 2024-04-17T08:51:07.004 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-04-17T08:51:07.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:07.004 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-17T08:51:07.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:51:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.294+0000 7fb0a5e00640 1 Processor -- start 2024-04-17T08:51:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.295+0000 7fb0a5e00640 1 -- start start 2024-04-17T08:51:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0a00d33e0 0x7fb0a00cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a00d00b0 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a00d0390 con 0x7fb0a015ecb0 2024-04-17T08:51:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb0a00d0c30 con 0x7fb0a00d33e0 2024-04-17T08:51:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb0a00cf8e0 0x7fb0a00b3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47160/0 (socket says 172.21.15.67:47160) 2024-04-17T08:51:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.296+0000 7fb0a5400640 1 -- 172.21.15.67:0/2933331912 learned_addr learned my addr 172.21.15.67:0/2933331912 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09fe00640 1 --2- 172.21.15.67:0/2933331912 >> v2:172.21.15.187:3300/0 conn(0x7fb0a00d33e0 0x7fb0a00cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652859150 0 0) 0x7fb0a00d00b0 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb088003520 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3794168178 0 0) 0x7fb088003520 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 >> v2:172.21.15.187:3300/0 conn(0x7fb0a00d33e0 msgr2=0x7fb0a00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 --2- 172.21.15.67:0/2933331912 >> v2:172.21.15.187:3300/0 conn(0x7fb0a00d33e0 0x7fb0a00cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 >> v1:172.21.15.187:6789/0 conn(0x7fb0a015ecb0 legacy=0x7fb0a00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a00d0670 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.297+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb0980019f0 con 0x7fb0a00cf8e0 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 >> v1:172.21.15.67:6789/0 conn(0x7fb0a00cf8e0 legacy=0x7fb0a00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb088003a00 0x7fb088003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 0x7fb088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088005210 0x7fb08800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a00d0c30 con 0x7fb088004800 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb08800c160 con 0x7fb088005210 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb08800c360 con 0x7fb088003a00 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 0x7fb088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb0a4a00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb088003a00 0x7fb088003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb0a5400640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088005210 0x7fb08800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 -- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb088003a00 msgr2=0x7fb088003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb088003a00 0x7fb088003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 -- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088005210 msgr2=0x7fb08800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088005210 0x7fb08800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.298+0000 7fb09fe00640 1 -- 172.21.15.67:0/2933331912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb088003520 con 0x7fb088004800 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.299+0000 7fb0a5400640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088005210 0x7fb08800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.299+0000 7fb0a5400640 1 -- 172.21.15.67:0/2933331912 reap_dead start 2024-04-17T08:51:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.300+0000 7fb09fe00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 0x7fb088004c80 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009a90 tx=0x7fb088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e153a958f1f1877 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.300+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c039070 con 0x7fb088004800 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.300+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08c02c680 con 0x7fb088004800 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.300+0000 7fb09f400640 1 -- 172.21.15.67:0/2933331912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c009690 con 0x7fb088004800 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 -- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 msgr2=0x7fb088004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 0x7fb088004c80 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009a90 tx=0x7fb088005830 comp rx=0 tx=0).stop 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 -- 172.21.15.67:0/2933331912 shutdown_connections 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/2933331912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb088004800 0x7fb088004c80 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 -- 172.21.15.67:0/2933331912 >> 172.21.15.67:0/2933331912 conn(0x7fb0a008bff0 msgr2=0x7fb0a00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 -- 172.21.15.67:0/2933331912 shutdown_connections 2024-04-17T08:51:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.301+0000 7fb0a5e00640 1 -- 172.21.15.67:0/2933331912 wait complete. 2024-04-17T08:51:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.303+0000 7fb0a5e00640 1 Processor -- start 2024-04-17T08:51:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.303+0000 7fb0a5e00640 1 -- start start 2024-04-17T08:51:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 0x7fb0a00caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0a00c8650 0x7fb0a00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a015f090 con 0x7fb0a00b99e0 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a00cfd30 con 0x7fb088004800 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb0a015f210 con 0x7fb0a00c8650 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38736/0 (socket says 172.21.15.67:38736) 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 -- 172.21.15.67:0/440055665 learned_addr learned my addr 172.21.15.67:0/440055665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb09fe00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0a00c8650 0x7fb0a00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a4a00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 0x7fb0a00caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 -- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0a00c8650 msgr2=0x7fb0a00c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0a00c8650 0x7fb0a00c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 -- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 msgr2=0x7fb0a00caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 0x7fb0a00caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a5400640 1 -- 172.21.15.67:0/440055665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a00cd6c0 con 0x7fb0a00b99e0 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.304+0000 7fb0a4a00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 0x7fb0a00caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb0a5400640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fb098001b60 tx=0x7fb09800df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09800b860 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb09800ba00 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09800bdd0 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a00cd9a0 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.305+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a00cdf60 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.307+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a0068a10 con 0x7fb0a00b99e0 2024-04-17T08:51:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.307+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 134) ==== 91318+0+0 (secure 0 0 0) 0x7fb09801b020 con 0x7fb0a00b99e0 2024-04-17T08:51:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.309+0000 7fb09cc00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 0x7fb08006f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.309+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7fb09808c620 con 0x7fb0a00b99e0 2024-04-17T08:51:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.310+0000 7fb0a4a00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 0x7fb08006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.311+0000 7fb0a4a00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 0x7fb08006f100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0940016b0 tx=0x7fb094009040 comp rx=0 tx=0).ready entity=mgr.8821 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.313+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb09805a110 con 0x7fb0a00b99e0 2024-04-17T08:51:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:07.578+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fb0a00b5a70 con 0x7fb0a00b99e0 2024-04-17T08:51:08.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.470+0000 7fb09cc00640 1 -- 172.21.15.67:0/440055665 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v135) ==== 93+0+0 (secure 0 0 0) 0x7fb098059a10 con 0x7fb0a00b99e0 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 msgr2=0x7fb08006f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 0x7fb08006f100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0940016b0 tx=0x7fb094009040 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 msgr2=0x7fb0a00c9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fb098001b60 tx=0x7fb09800df70 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 shutdown_connections 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb08006cc40 0x7fb08006f100 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a00b99e0 0x7fb0a00c9050 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0a00c8650 0x7fb0a00c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 --2- 172.21.15.67:0/440055665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb088004800 0x7fb0a00caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.477+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 >> 172.21.15.67:0/440055665 conn(0x7fb0a008bff0 msgr2=0x7fb0a00b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.479+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 shutdown_connections 2024-04-17T08:51:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:08.479+0000 7fb0a5e00640 1 -- 172.21.15.67:0/440055665 wait complete. 2024-04-17T08:51:08.500 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:51:08.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:51:08.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:51:08.706 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:51:08.706 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:51:08.706 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:08.706 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:08.754 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:08.754 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:08.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:09.026 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.026+0000 7fd63ca48040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:51:09.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.047+0000 7f750b200640 1 Processor -- start 2024-04-17T08:51:09.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.047+0000 7f750b200640 1 -- start start 2024-04-17T08:51:09.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.048+0000 7f750b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 0x7f750c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f750b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f750c0cfae0 0x7f750c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f750b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f750c0d2d30 con 0x7f750c0b6b30 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f750b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f750c0d49c0 con 0x7f750c0cfae0 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f750b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f750c0d3660 con 0x7f750c15ec60 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f7509400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f750c0b6b30 0x7f750c0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47168/0 (socket says 172.21.15.67:47168) 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f7509400640 1 -- 172.21.15.67:0/3144210936 learned_addr learned my addr 172.21.15.67:0/3144210936 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f750a800640 1 --2- 172.21.15.67:0/3144210936 >> v2:172.21.15.187:3300/0 conn(0x7f750c0cfae0 0x7f750c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f7509e00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 0x7f750c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.049+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64212626 0 0) 0x7f750c0d2d30 con 0x7f750c0b6b30 2024-04-17T08:51:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74ec003520 con 0x7f750c0b6b30 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1304647561 0 0) 0x7f74ec003520 con 0x7f750c0b6b30 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 msgr2=0x7f750c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 0x7f750c0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 >> v2:172.21.15.187:3300/0 conn(0x7f750c0cfae0 msgr2=0x7f750c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 --2- 172.21.15.67:0/3144210936 >> v2:172.21.15.187:3300/0 conn(0x7f750c0cfae0 0x7f750c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c0d3010 con 0x7f750c0b6b30 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f74f40019f0 con 0x7f750c0b6b30 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 >> v1:172.21.15.67:6789/0 conn(0x7f750c0b6b30 legacy=0x7f750c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec003a20 0x7f74ec003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74ec004800 0x7f74ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 0x7f74ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f750c0d49c0 con 0x7f74ec003a20 2024-04-17T08:51:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f750c0d3660 con 0x7f74ec004800 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f74ec00b9f0 con 0x7f750c15ec60 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.050+0000 7f7509e00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec003a20 0x7f74ec003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.051+0000 7f7509e00640 1 -- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 msgr2=0x7f74ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.051+0000 7f7509e00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f750c15ec60 0x7f74ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.051+0000 7f7509e00640 1 -- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74ec004800 msgr2=0x7f74ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.051+0000 7f7509e00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74ec004800 0x7f74ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.051+0000 7f7509e00640 1 -- 172.21.15.67:0/3144210936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec003520 con 0x7f74ec003a20 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.052+0000 7f7509e00640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec003a20 0x7f74ec003e40 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009ac0 tx=0x7f74ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62665cd7531a8853 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.052+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc039070 con 0x7f74ec003a20 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.052+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74fc02c710 con 0x7f74ec003a20 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.052+0000 7f7508a00640 1 -- 172.21.15.67:0/3144210936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc009690 con 0x7f74ec003a20 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 -- 172.21.15.67:0/3144210936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec003a20 msgr2=0x7f74ec003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 --2- 172.21.15.67:0/3144210936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec003a20 0x7f74ec003e40 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009ac0 tx=0x7f74ec005830 comp rx=0 tx=0).stop 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750a800640 1 -- 172.21.15.67:0/3144210936 reap_dead start 2024-04-17T08:51:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 -- 172.21.15.67:0/3144210936 shutdown_connections 2024-04-17T08:51:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 -- 172.21.15.67:0/3144210936 >> 172.21.15.67:0/3144210936 conn(0x7f750c0d15f0 msgr2=0x7f750c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 -- 172.21.15.67:0/3144210936 shutdown_connections 2024-04-17T08:51:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.053+0000 7f750b200640 1 -- 172.21.15.67:0/3144210936 wait complete. 2024-04-17T08:51:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.055+0000 7f750b200640 1 Processor -- start 2024-04-17T08:51:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.055+0000 7f750b200640 1 -- start start 2024-04-17T08:51:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 0x7f750c154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f750c0cc0c0 0x7f750c160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f750c15f040 con 0x7f750c0cbc40 2024-04-17T08:51:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f750c0b6f70 con 0x7f750c0cc0c0 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f750b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f750c0d00c0 con 0x7f74ec003a20 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.056+0000 7f7509400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38766/0 (socket says 172.21.15.67:38766) 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 -- 172.21.15.67:0/2220000852 learned_addr learned my addr 172.21.15.67:0/2220000852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509e00640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 0x7f750c154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f750a800640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f750c0cc0c0 0x7f750c160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 msgr2=0x7f750c154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 0x7f750c154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f750c0cc0c0 msgr2=0x7f750c160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f750c0cc0c0 0x7f750c160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 -- 172.21.15.67:0/2220000852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750c0cc690 con 0x7f750c0cbc40 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509e00640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 0x7f750c154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.057+0000 7f7509400640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f74f400ca70 tx=0x7f74f400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.058+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4018070 con 0x7f750c0cbc40 2024-04-17T08:51:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.058+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f400a760 con 0x7f750c0cbc40 2024-04-17T08:51:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.058+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750c0cc970 con 0x7f750c0cbc40 2024-04-17T08:51:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.058+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f400ab50 con 0x7f750c0cbc40 2024-04-17T08:51:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.058+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750c0cd6d0 con 0x7f750c0cbc40 2024-04-17T08:51:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.059+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f750c0b71c0 con 0x7f750c0cbc40 2024-04-17T08:51:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.060+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7f74f401b070 con 0x7f750c0cbc40 2024-04-17T08:51:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.062+0000 7f7502000640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 0x7f74e806f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.062+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f74f408bc70 con 0x7f750c0cbc40 2024-04-17T08:51:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.062+0000 7f7509e00640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 msgr2=0x7f74e806f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.062+0000 7f7509e00640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 0x7f74e806f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.066+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f750c0b71c0 con 0x7f750c0cbc40 2024-04-17T08:51:09.082 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.082+0000 7fd63ca48040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:51:09.105 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.106+0000 7f3ff36da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:51:09.168 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.169+0000 7f3ff36da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:51:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.263+0000 7f7509e00640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 msgr2=0x7f74e806f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.263+0000 7f7509e00640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 0x7f74e806f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.273+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f750c068a10 con 0x7f750c0cbc40 2024-04-17T08:51:09.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.276+0000 7f7502000640 1 -- 172.21.15.67:0/2220000852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) ==== 74+0+20973 (secure 0 0 0) 0x7f74f40598c0 con 0x7f750c0cbc40 2024-04-17T08:51:09.275 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:09.276 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":235,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:01.354365+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 msgr2=0x7f74e806f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 0x7f74e806f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 msgr2=0x7f750c160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f74f400ca70 tx=0x7f74f400cf30 comp rx=0 tx=0).stop 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 shutdown_connections 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f74e806cbf0 0x7f74e806f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f750c0cc0c0 0x7f750c160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f750c0cbc40 0x7f750c160210 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 --2- 172.21.15.67:0/2220000852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f74ec003a20 0x7f750c154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.278+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 >> 172.21.15.67:0/2220000852 conn(0x7f750c0d15f0 msgr2=0x7f750c074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.279+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 shutdown_connections 2024-04-17T08:51:09.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.279+0000 7f750b200640 1 -- 172.21.15.67:0/2220000852 wait complete. 2024-04-17T08:51:09.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:50:31.281467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:50:32.343386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:50:46.826027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:50:47.478489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:09.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.507+0000 7f6c92e00640 1 Processor -- start 2024-04-17T08:51:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.507+0000 7f6c92e00640 1 -- start start 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.508+0000 7f6c92e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c8c161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.508+0000 7f6c92e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6c8c1620e0 0x7f6c8c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.508+0000 7f6c92e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c0b4e40 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.508+0000 7f6c92e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6c8c162530 con 0x7f6c8c1620e0 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.508+0000 7f6c92e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c0b5110 con 0x7f6c8c0d33d0 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c91a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6c8c0d2ff0 0x7f6c8c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47174/0 (socket says 172.21.15.67:47174) 2024-04-17T08:51:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c91a00640 1 -- 172.21.15.67:0/1022002958 learned_addr learned my addr 172.21.15.67:0/1022002958 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c92400640 1 --2- 172.21.15.67:0/1022002958 >> v2:172.21.15.187:3300/0 conn(0x7f6c8c1620e0 0x7f6c8c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c91000640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c8c161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568793346 0 0) 0x7f6c8c0b4e40 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c74003520 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.509+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1962436983 0 0) 0x7f6c74003520 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 msgr2=0x7f6c8c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c8c161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 >> v2:172.21.15.187:3300/0 conn(0x7f6c8c1620e0 msgr2=0x7f6c8c0cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 --2- 172.21.15.67:0/1022002958 >> v2:172.21.15.187:3300/0 conn(0x7f6c8c1620e0 0x7f6c8c0cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c0b53f0 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6c80001670 con 0x7f6c8c0d2ff0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c91000640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c8c161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 >> v1:172.21.15.67:6789/0 conn(0x7f6c8c0d2ff0 legacy=0x7f6c8c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74003ef0 0x7f6c74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c74004d40 0x7f6c740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c162530 con 0x7f6c74004d40 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0b5110 con 0x7f6c74003ef0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7400b9f0 con 0x7f6c8c0d33d0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c92400640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c74004d40 0x7f6c740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c91000640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74003ef0 0x7f6c74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.510+0000 7f6c91a00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.511+0000 7f6c92400640 1 -- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 msgr2=0x7f6c740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.511+0000 7f6c92400640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c0d33d0 0x7f6c740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.511+0000 7f6c92400640 1 -- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74003ef0 msgr2=0x7f6c74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.511+0000 7f6c92400640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c74003ef0 0x7f6c74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.511+0000 7f6c92400640 1 -- 172.21.15.67:0/1022002958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c74003520 con 0x7f6c74004d40 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.512+0000 7f6c92400640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c74004d40 0x7f6c740051c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009fc0 tx=0x7f6c74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52cca36ea6fb1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.512+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84039070 con 0x7f6c74004d40 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.512+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8402cc60 con 0x7f6c74004d40 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.512+0000 7f6c8be00640 1 -- 172.21.15.67:0/1022002958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84009c10 con 0x7f6c74004d40 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 -- 172.21.15.67:0/1022002958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c74004d40 msgr2=0x7f6c740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 --2- 172.21.15.67:0/1022002958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c74004d40 0x7f6c740051c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009fc0 tx=0x7f6c74005830 comp rx=0 tx=0).stop 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92400640 1 -- 172.21.15.67:0/1022002958 reap_dead start 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 -- 172.21.15.67:0/1022002958 shutdown_connections 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 -- 172.21.15.67:0/1022002958 >> 172.21.15.67:0/1022002958 conn(0x7f6c8c09da80 msgr2=0x7f6c8c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 -- 172.21.15.67:0/1022002958 shutdown_connections 2024-04-17T08:51:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.513+0000 7f6c92e00640 1 -- 172.21.15.67:0/1022002958 wait complete. 2024-04-17T08:51:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.515+0000 7f6c92e00640 1 Processor -- start 2024-04-17T08:51:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.515+0000 7f6c92e00640 1 -- start start 2024-04-17T08:51:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c152ed0 0x7f6c8c0b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c8c0d4ab0 0x7f6c8c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0cf4c0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0cf340 con 0x7f6c8c0d4ab0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c0d26d0 con 0x7f6c8c152ed0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c91000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c91000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38780/0 (socket says 172.21.15.67:38780) 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c91000640 1 -- 172.21.15.67:0/112116835 learned_addr learned my addr 172.21.15.67:0/112116835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c91a00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c152ed0 0x7f6c8c0b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.516+0000 7f6c92400640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c8c0d4ab0 0x7f6c8c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c152ed0 msgr2=0x7f6c8c0b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c152ed0 0x7f6c8c0b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c8c0d4ab0 msgr2=0x7f6c8c155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c8c0d4ab0 0x7f6c8c155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 -- 172.21.15.67:0/112116835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c0c8600 con 0x7f6c8c0b7900 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c91000640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f6c780108c0 tx=0x7f6c78010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7801a070 con 0x7f6c8c0b7900 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c7800cb40 con 0x7f6c8c0b7900 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800f040 con 0x7f6c8c0b7900 2024-04-17T08:51:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c0c88e0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.517+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c0c8ea0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.518+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c0755d0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.520+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7f6c78015470 con 0x7f6c8c0b7900 2024-04-17T08:51:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.521+0000 7f6c89600640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 0x7f6c6c06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.521+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f6c7808cbb0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.521+0000 7f6c91a00640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 msgr2=0x7f6c6c06f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.521+0000 7f6c91a00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 0x7f6c6c06f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:09.524 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.524+0000 7fd63ca48040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:51:09.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.530+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6c7805a6f0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.576 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.577+0000 7fd63ca48040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:51:09.626 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.627+0000 7fd63ca48040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:51:09.652 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.653+0000 7f3ff36da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:51:09.686 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.687+0000 7fd63ca48040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:51:09.708 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.708+0000 7f3ff36da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:51:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.722+0000 7f6c91a00640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 msgr2=0x7f6c6c06f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:09.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.722+0000 7f6c91a00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 0x7f6c6c06f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:09.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.731+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c8c0d4e90 con 0x7f6c8c0b7900 2024-04-17T08:51:09.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.732+0000 7f6c89600640 1 -- 172.21.15.67:0/112116835 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) ==== 93+0+10 (secure 0 0 0) 0x7f6c78059ff0 con 0x7f6c8c0b7900 2024-04-17T08:51:09.731 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:09.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 msgr2=0x7f6c6c06f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:09.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 0x7f6c6c06f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 msgr2=0x7f6c8c0d4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f6c780108c0 tx=0x7f6c78010d80 comp rx=0 tx=0).stop 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 shutdown_connections 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f6c6c06cbf0 0x7f6c6c06f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6c8c0d4ab0 0x7f6c8c155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c8c0b7900 0x7f6c8c0d4570 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 --2- 172.21.15.67:0/112116835 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6c8c152ed0 0x7f6c8c0b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 >> 172.21.15.67:0/112116835 conn(0x7f6c8c09da80 msgr2=0x7f6c8c0d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 shutdown_connections 2024-04-17T08:51:09.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.735+0000 7f6c92e00640 1 -- 172.21.15.67:0/112116835 wait complete. 2024-04-17T08:51:09.735 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.736+0000 7fd63ca48040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:51:09.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:51:09.763 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.764+0000 7f3ff36da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:51:09.829 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.830+0000 7f3ff36da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:51:09.864 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.865+0000 7fd63ca48040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:51:09.883 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:09.883+0000 7f3ff36da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:51:09.926 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.927+0000 7fd63ca48040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.968+0000 7fd36ca00640 1 Processor -- start 2024-04-17T08:51:09.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.969+0000 7fd36ca00640 1 -- start start 2024-04-17T08:51:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.970+0000 7fd36ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd36806c2d0 0x7fd3680b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.970+0000 7fd36ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3680ae2c0 0x7fd3680ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.970+0000 7fd36ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3680cbb50 con 0x7fd36815c800 2024-04-17T08:51:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.970+0000 7fd36ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd3680cd2c0 con 0x7fd3680ae2c0 2024-04-17T08:51:09.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.970+0000 7fd36ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd3680aec30 con 0x7fd36806c2d0 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3680ae2c0 0x7fd3680ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd366800640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd36815c800 0x7fd3680ccbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:37138/0 (socket says 172.21.15.67:37138) 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3680ae2c0 0x7fd3680ae6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38786/0 (socket says 172.21.15.67:38786) 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd35fe00640 1 --2- 172.21.15.67:0/2761333031 >> v2:172.21.15.187:3300/0 conn(0x7fd36806c2d0 0x7fd3680b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd366800640 1 -- 172.21.15.67:0/2761333031 learned_addr learned my addr 172.21.15.67:0/2761333031 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 -- 172.21.15.67:0/2761333031 >> v2:172.21.15.187:3300/0 conn(0x7fd36806c2d0 msgr2=0x7fd3680b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 --2- 172.21.15.67:0/2761333031 >> v2:172.21.15.187:3300/0 conn(0x7fd36806c2d0 0x7fd3680b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 -- 172.21.15.67:0/2761333031 >> v1:172.21.15.187:6789/0 conn(0x7fd36815c800 legacy=0x7fd3680ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.971+0000 7fd367200640 1 -- 172.21.15.67:0/2761333031 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3680cbdf0 con 0x7fd3680ae2c0 2024-04-17T08:51:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.973+0000 7fd367200640 1 --2- 172.21.15.67:0/2761333031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3680ae2c0 0x7fd3680ae6f0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7fd3680ceef0 tx=0x7fd36002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad82ee7627f6933 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.973+0000 7fd365e00640 1 -- 172.21.15.67:0/2761333031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360039070 con 0x7fd3680ae2c0 2024-04-17T08:51:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.973+0000 7fd365e00640 1 -- 172.21.15.67:0/2761333031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd36002cb60 con 0x7fd3680ae2c0 2024-04-17T08:51:09.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.973+0000 7fd365e00640 1 -- 172.21.15.67:0/2761333031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360034440 con 0x7fd3680ae2c0 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 -- 172.21.15.67:0/2761333031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3680ae2c0 msgr2=0x7fd3680ae6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 --2- 172.21.15.67:0/2761333031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3680ae2c0 0x7fd3680ae6f0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7fd3680ceef0 tx=0x7fd36002bdb0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 -- 172.21.15.67:0/2761333031 shutdown_connections 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 --2- 172.21.15.67:0/2761333031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3680ae2c0 0x7fd3680ae6f0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 --2- 172.21.15.67:0/2761333031 >> v2:172.21.15.187:3300/0 conn(0x7fd36806c2d0 0x7fd3680b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 -- 172.21.15.67:0/2761333031 >> 172.21.15.67:0/2761333031 conn(0x7fd3680b2fe0 msgr2=0x7fd3680b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 -- 172.21.15.67:0/2761333031 shutdown_connections 2024-04-17T08:51:09.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.974+0000 7fd36ca00640 1 -- 172.21.15.67:0/2761333031 wait complete. 2024-04-17T08:51:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.975+0000 7fd36ca00640 1 Processor -- start 2024-04-17T08:51:09.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.976+0000 7fd36ca00640 1 -- start start 2024-04-17T08:51:09.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3680ae2c0 0x7fd3680b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36815c800 0x7fd3680b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3680cc690 con 0x7fd36806c2d0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd3680cc3b0 con 0x7fd36815c800 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd36ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd36815cbe0 con 0x7fd3680ae2c0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38794/0 (socket says 172.21.15.67:38794) 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 -- 172.21.15.67:0/1189966620 learned_addr learned my addr 172.21.15.67:0/1189966620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd367200640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36815c800 0x7fd3680b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd35fe00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3680ae2c0 0x7fd3680b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3680ae2c0 msgr2=0x7fd3680b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3680ae2c0 0x7fd3680b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36815c800 msgr2=0x7fd3680b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36815c800 0x7fd3680b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.977+0000 7fd366800640 1 -- 172.21.15.67:0/1189966620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3681598f0 con 0x7fd36806c2d0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd366800640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c480 tx=0x7fd35800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358017070 con 0x7fd36806c2d0 2024-04-17T08:51:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd358005ce0 con 0x7fd36806c2d0 2024-04-17T08:51:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3580072b0 con 0x7fd36806c2d0 2024-04-17T08:51:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3680c8940 con 0x7fd36806c2d0 2024-04-17T08:51:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.978+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd368159330 con 0x7fd36806c2d0 2024-04-17T08:51:09.979 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:09.978+0000 7fd63ca48040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:51:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.979+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3680755d0 con 0x7fd36806c2d0 2024-04-17T08:51:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.980+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7fd35801a070 con 0x7fd36806c2d0 2024-04-17T08:51:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.981+0000 7fd35ea00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 0x7fd34406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.981+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7fd35808b870 con 0x7fd36806c2d0 2024-04-17T08:51:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.981+0000 7fd35fe00640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 msgr2=0x7fd34406f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.981+0000 7fd35fe00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 0x7fd34406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:09.990+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd3681598f0 con 0x7fd36806c2d0 2024-04-17T08:51:10.028 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.028+0000 7f3ff36da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:51:10.034 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.035+0000 7fd63ca48040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:51:10.085 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.086+0000 7fd63ca48040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:51:10.095 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.096+0000 7f3ff36da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:10.148 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.148+0000 7f3ff36da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.182+0000 7fd35fe00640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 msgr2=0x7fd34406f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.182+0000 7fd35fe00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 0x7fd34406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.195+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3680cec40 con 0x7fd36806c2d0 2024-04-17T08:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.196+0000 7fd35ea00640 1 -- 172.21.15.67:0/1189966620 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v235) ==== 104+0+11 (secure 0 0 0) 0x7fd358059470 con 0x7fd36806c2d0 2024-04-17T08:51:10.195 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 msgr2=0x7fd34406f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 0x7fd34406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 msgr2=0x7fd3681f44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fd35800c480 tx=0x7fd35800c940 comp rx=0 tx=0).stop 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 shutdown_connections 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fd34406cbf0 0x7fd34406f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36815c800 0x7fd3680b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd3680ae2c0 0x7fd3680b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 --2- 172.21.15.67:0/1189966620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36806c2d0 0x7fd3681f44a0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.198+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 >> 172.21.15.67:0/1189966620 conn(0x7fd3680b2fe0 msgr2=0x7fd3680b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.199+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 shutdown_connections 2024-04-17T08:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.199+0000 7fd36ca00640 1 -- 172.21.15.67:0/1189966620 wait complete. 2024-04-17T08:51:10.210 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.211+0000 7f3ff36da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:51:10.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:51:10.268 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.265+0000 7f3ff36da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:51:10.466 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.466+0000 7fd63ca48040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:51:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.469+0000 7f8acae00640 1 Processor -- start 2024-04-17T08:51:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.469+0000 7f8acae00640 1 -- start start 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.470+0000 7f8acae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac415ec00 0x7f8ac40b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.470+0000 7f8acae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.470+0000 7f8acae00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac40d4740 con 0x7f8ac40cf870 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.470+0000 7f8acae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8ac40d3610 con 0x7f8ac40d1a80 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.470+0000 7f8acae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac40d2a00 con 0x7f8ac415ec00 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38798/0 (socket says 172.21.15.67:38798) 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2315065024 learned_addr learned my addr 172.21.15.67:0/2315065024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac3e00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac415ec00 0x7f8ac40b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2315065024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac415ec00 msgr2=0x7f8ac40b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac415ec00 0x7f8ac40b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2315065024 >> v1:172.21.15.187:6789/0 conn(0x7f8ac40cf870 legacy=0x7f8ac40d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.471+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2315065024 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac40d2ce0 con 0x7f8ac40d1a80 2024-04-17T08:51:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.472+0000 7f8ac8e00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009870 tx=0x7f8abc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fc80fb710156c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.473+0000 7f8ac3400640 1 -- 172.21.15.67:0/2315065024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc035020 con 0x7f8ac40d1a80 2024-04-17T08:51:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.473+0000 7f8ac3400640 1 -- 172.21.15.67:0/2315065024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8abc031640 con 0x7f8ac40d1a80 2024-04-17T08:51:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.473+0000 7f8ac3400640 1 -- 172.21.15.67:0/2315065024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02d040 con 0x7f8ac40d1a80 2024-04-17T08:51:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 -- 172.21.15.67:0/2315065024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac40d1a80 msgr2=0x7f8ac40cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009870 tx=0x7f8abc02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 -- 172.21.15.67:0/2315065024 shutdown_connections 2024-04-17T08:51:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac40d1a80 0x7f8ac40cfcd0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 --2- 172.21.15.67:0/2315065024 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac415ec00 0x7f8ac40b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 -- 172.21.15.67:0/2315065024 >> 172.21.15.67:0/2315065024 conn(0x7f8ac406ccb0 msgr2=0x7f8ac406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 -- 172.21.15.67:0/2315065024 shutdown_connections 2024-04-17T08:51:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.474+0000 7f8acae00640 1 -- 172.21.15.67:0/2315065024 wait complete. 2024-04-17T08:51:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.476+0000 7f8acae00640 1 Processor -- start 2024-04-17T08:51:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.476+0000 7f8acae00640 1 -- start start 2024-04-17T08:51:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac40cf870 0x7f8ac40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac40d1a80 0x7f8ac415d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac415efe0 con 0x7f8ac415ec00 2024-04-17T08:51:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac40d32a0 con 0x7f8ac40d1a80 2024-04-17T08:51:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8acae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac415f160 con 0x7f8ac40cf870 2024-04-17T08:51:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8ac8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8ac8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38802/0 (socket says 172.21.15.67:38802) 2024-04-17T08:51:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.477+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2351830393 learned_addr learned my addr 172.21.15.67:0/2351830393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8abb400640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac40d1a80 0x7f8ac415d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac40cf870 msgr2=0x7f8ac40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac40cf870 0x7f8ac40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac40d1a80 msgr2=0x7f8ac415d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac40d1a80 0x7f8ac415d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 -- 172.21.15.67:0/2351830393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac415c600 con 0x7f8ac415ec00 2024-04-17T08:51:10.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac8e00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009870 tx=0x7f8abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc040070 con 0x7f8ac415ec00 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8abc0399f0 con 0x7f8ac415ec00 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac415c8e0 con 0x7f8ac415ec00 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.478+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc035040 con 0x7f8ac415ec00 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.479+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac415cea0 con 0x7f8ac415ec00 2024-04-17T08:51:10.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.480+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac4068b90 con 0x7f8ac415ec00 2024-04-17T08:51:10.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.481+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7f8abc04b060 con 0x7f8ac415ec00 2024-04-17T08:51:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.482+0000 7f8ac0c00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 0x7f8aa006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.482+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f8abc0add80 con 0x7f8ac415ec00 2024-04-17T08:51:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.482+0000 7f8ac3e00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 msgr2=0x7f8aa006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.482+0000 7f8ac3e00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 0x7f8aa006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.486+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8abc07ba90 con 0x7f8ac415ec00 2024-04-17T08:51:10.626 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.626+0000 7fd63ca48040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:51:10.681 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.682+0000 7f3ff36da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:51:10.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.683+0000 7f8ac3e00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 msgr2=0x7f8aa006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:10.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.683+0000 7f8ac3e00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 0x7f8aa006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:10.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.685+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8ac40d26b0 con 0x7f8ac415ec00 2024-04-17T08:51:10.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.686+0000 7f8ac0c00640 1 -- 172.21.15.67:0/2351830393 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v235) ==== 100+0+11 (secure 0 0 0) 0x7f8abc0ad020 con 0x7f8ac415ec00 2024-04-17T08:51:10.685 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 msgr2=0x7f8aa006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 0x7f8aa006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 msgr2=0x7f8ac41612d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009870 tx=0x7f8abc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 shutdown_connections 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8aa006cbc0 0x7f8aa006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ac415ec00 0x7f8ac41612d0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ac40d1a80 0x7f8ac415d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 --2- 172.21.15.67:0/2351830393 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ac40cf870 0x7f8ac40caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.688+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 >> 172.21.15.67:0/2351830393 conn(0x7f8ac406ccb0 msgr2=0x7f8ac40cb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.689+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 shutdown_connections 2024-04-17T08:51:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.689+0000 7f8acae00640 1 -- 172.21.15.67:0/2351830393 wait complete. 2024-04-17T08:51:10.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:51:10.734 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.734+0000 7fd63ca48040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:51:10.783 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.784+0000 7fd63ca48040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:51:10.854 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.854+0000 7f3ff36da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:51:10.909 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:10.910+0000 7fd63ca48040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:51:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.915+0000 7fce53200640 1 Processor -- start 2024-04-17T08:51:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.915+0000 7fce53200640 1 -- start start 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.916+0000 7fce53200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fce5415ec20 0x7fce540d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.916+0000 7fce53200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce540b3b50 0x7fce5406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.916+0000 7fce53200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce540d4f70 con 0x7fce5406cf80 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.916+0000 7fce53200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fce540d5250 con 0x7fce540b3b50 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.916+0000 7fce53200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fce540cf4d0 con 0x7fce5415ec20 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce540b3b50 0x7fce5406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce540b3b50 0x7fce5406ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38810/0 (socket says 172.21.15.67:38810) 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 -- 172.21.15.67:0/2922615436 learned_addr learned my addr 172.21.15.67:0/2922615436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce51e00640 1 --2- 172.21.15.67:0/2922615436 >> v2:172.21.15.187:3300/0 conn(0x7fce5415ec20 0x7fce540d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 -- 172.21.15.67:0/2922615436 >> v2:172.21.15.187:3300/0 conn(0x7fce5415ec20 msgr2=0x7fce540d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 --2- 172.21.15.67:0/2922615436 >> v2:172.21.15.187:3300/0 conn(0x7fce5415ec20 0x7fce540d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 -- 172.21.15.67:0/2922615436 >> v1:172.21.15.187:6789/0 conn(0x7fce5406cf80 legacy=0x7fce540cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.917+0000 7fce49400640 1 -- 172.21.15.67:0/2922615436 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce540cf7b0 con 0x7fce540b3b50 2024-04-17T08:51:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.918+0000 7fce49400640 1 --2- 172.21.15.67:0/2922615436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce540b3b50 0x7fce5406ca40 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fce5408ca90 tx=0x7fce3c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceaeb4e32513d71c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.919+0000 7fce51400640 1 -- 172.21.15.67:0/2922615436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0317d0 con 0x7fce540b3b50 2024-04-17T08:51:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.919+0000 7fce51400640 1 -- 172.21.15.67:0/2922615436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c037070 con 0x7fce540b3b50 2024-04-17T08:51:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.919+0000 7fce51400640 1 -- 172.21.15.67:0/2922615436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c031b90 con 0x7fce540b3b50 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.919+0000 7fce53200640 1 -- 172.21.15.67:0/2922615436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce540b3b50 msgr2=0x7fce5406ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.919+0000 7fce53200640 1 --2- 172.21.15.67:0/2922615436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce540b3b50 0x7fce5406ca40 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fce5408ca90 tx=0x7fce3c02b870 comp rx=0 tx=0).stop 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 -- 172.21.15.67:0/2922615436 shutdown_connections 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 --2- 172.21.15.67:0/2922615436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce540b3b50 0x7fce5406ca40 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 --2- 172.21.15.67:0/2922615436 >> v2:172.21.15.187:3300/0 conn(0x7fce5415ec20 0x7fce540d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 -- 172.21.15.67:0/2922615436 >> 172.21.15.67:0/2922615436 conn(0x7fce5408bff0 msgr2=0x7fce540d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 -- 172.21.15.67:0/2922615436 shutdown_connections 2024-04-17T08:51:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.920+0000 7fce53200640 1 -- 172.21.15.67:0/2922615436 wait complete. 2024-04-17T08:51:10.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.921+0000 7fce53200640 1 Processor -- start 2024-04-17T08:51:10.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.921+0000 7fce53200640 1 -- start start 2024-04-17T08:51:10.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce540b3b50 0x7fce540b9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce5415ec20 0x7fce540cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce540d0560 con 0x7fce5406cf80 2024-04-17T08:51:10.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fce540d03e0 con 0x7fce5415ec20 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce53200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fce540d10d0 con 0x7fce540b3b50 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.922+0000 7fce51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38820/0 (socket says 172.21.15.67:38820) 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 -- 172.21.15.67:0/2322014707 learned_addr learned my addr 172.21.15.67:0/2322014707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce540b3b50 msgr2=0x7fce540b9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce540b3b50 0x7fce540b9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce5415ec20 msgr2=0x7fce540cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce5415ec20 0x7fce540cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 -- 172.21.15.67:0/2322014707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce540ba5f0 con 0x7fce5406cf80 2024-04-17T08:51:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce51e00640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fce44007e40 tx=0x7fce4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.923+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44014620 con 0x7fce5406cf80 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.924+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce44019070 con 0x7fce5406cf80 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.924+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44014a00 con 0x7fce5406cf80 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.924+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce540cdf70 con 0x7fce5406cf80 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.924+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce540ce530 con 0x7fce5406cf80 2024-04-17T08:51:10.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.925+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce54068a10 con 0x7fce5406cf80 2024-04-17T08:51:10.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.926+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7fce44014ba0 con 0x7fce5406cf80 2024-04-17T08:51:10.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.927+0000 7fce4aa00640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 0x7fce3006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:10.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.927+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7fce4408cd70 con 0x7fce5406cf80 2024-04-17T08:51:10.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.927+0000 7fce49400640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 msgr2=0x7fce3006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:10.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.927+0000 7fce49400640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 0x7fce3006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:10.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:10.932+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fce4405a930 con 0x7fce5406cf80 2024-04-17T08:51:10.972 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:10.972+0000 7f3ff36da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:51:11.015 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:11.016+0000 7fd63ca48040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:51:11.026 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.026+0000 7f3ff36da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:51:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.128+0000 7fce49400640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 msgr2=0x7fce3006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.128+0000 7fce49400640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 0x7fce3006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:11.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.135+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7fce540ca980 con 0x7fce5406cf80 2024-04-17T08:51:11.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.136+0000 7fce4aa00640 1 -- 172.21.15.67:0/2322014707 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v235) ==== 107+0+11 (secure 0 0 0) 0x7fce4405a230 con 0x7fce5406cf80 2024-04-17T08:51:11.135 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 msgr2=0x7fce3006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 0x7fce3006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 msgr2=0x7fce541534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fce44007e40 tx=0x7fce4400e4b0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 shutdown_connections 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fce3006cbc0 0x7fce3006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce5415ec20 0x7fce540cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce540b3b50 0x7fce540b9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 --2- 172.21.15.67:0/2322014707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5406cf80 0x7fce541534f0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.138+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 >> 172.21.15.67:0/2322014707 conn(0x7fce5408bff0 msgr2=0x7fce54074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.139+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 shutdown_connections 2024-04-17T08:51:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.139+0000 7fce53200640 1 -- 172.21.15.67:0/2322014707 wait complete. 2024-04-17T08:51:11.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:51:11.163 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.164+0000 7f3ff36da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:51:11.190 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:11.191+0000 7fd63ca48040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:51:11.239 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:11.240+0000 7fd63ca48040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:11.280 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.280+0000 7f3ff36da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:51:11.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.357+0000 7f162fe00640 1 Processor -- start 2024-04-17T08:51:11.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.358+0000 7f162fe00640 1 -- start start 2024-04-17T08:51:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f163015ecb0 0x7f16300d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16300d00b0 con 0x7f16300d3210 2024-04-17T08:51:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f16300d0c30 con 0x7f163015ecb0 2024-04-17T08:51:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16300d0390 con 0x7f16300cfab0 2024-04-17T08:51:11.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f163015ecb0 0x7f16300d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f163015ecb0 0x7f16300d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38836/0 (socket says 172.21.15.67:38836) 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162ea00640 1 -- 172.21.15.67:0/547821069 learned_addr learned my addr 172.21.15.67:0/547821069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.359+0000 7f162f400640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.360+0000 7f162ea00640 1 -- 172.21.15.67:0/547821069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 msgr2=0x7f16300b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.360+0000 7f162ea00640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.360+0000 7f162ea00640 1 -- 172.21.15.67:0/547821069 >> v1:172.21.15.187:6789/0 conn(0x7f16300d3210 legacy=0x7f16300cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.360+0000 7f162ea00640 1 -- 172.21.15.67:0/547821069 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16300d0670 con 0x7f163015ecb0 2024-04-17T08:51:11.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.360+0000 7f162f400640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:11.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.361+0000 7f162ea00640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163015ecb0 0x7f16300d2cd0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f1620009e50 tx=0x7f162002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cefa4337e84dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:11.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.361+0000 7f162d600640 1 -- 172.21.15.67:0/547821069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162002ca70 con 0x7f163015ecb0 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.361+0000 7f162d600640 1 -- 172.21.15.67:0/547821069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f162002cc10 con 0x7f163015ecb0 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.361+0000 7f162d600640 1 -- 172.21.15.67:0/547821069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16200343f0 con 0x7f163015ecb0 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 -- 172.21.15.67:0/547821069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163015ecb0 msgr2=0x7f16300d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163015ecb0 0x7f16300d2cd0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f1620009e50 tx=0x7f162002bed0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 -- 172.21.15.67:0/547821069 shutdown_connections 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 --2- 172.21.15.67:0/547821069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163015ecb0 0x7f16300d2cd0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 -- 172.21.15.67:0/547821069 >> 172.21.15.67:0/547821069 conn(0x7f163008bff0 msgr2=0x7f16300d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 -- 172.21.15.67:0/547821069 shutdown_connections 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.362+0000 7f162fe00640 1 -- 172.21.15.67:0/547821069 wait complete. 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.364+0000 7f162fe00640 1 Processor -- start 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.364+0000 7f162fe00640 1 -- start start 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f163015ecb0 0x7f1630160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f163015f210 con 0x7f16300d3210 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f163015f090 con 0x7f163015ecb0 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16300b4500 con 0x7f16300cfab0 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38848/0 (socket says 172.21.15.67:38848) 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 -- 172.21.15.67:0/3942268638 learned_addr learned my addr 172.21.15.67:0/3942268638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162f400640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f163015ecb0 0x7f1630160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162ea00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 msgr2=0x7f16300b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f163015ecb0 msgr2=0x7f1630160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f163015ecb0 0x7f1630160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.365+0000 7f162e000640 1 -- 172.21.15.67:0/3942268638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16300b7810 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f162e000640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c400 tx=0x7f161c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0052d0 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c005490 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c007260 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16301f8900 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.366+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1630160120 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.367+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1630068a10 con 0x7f16300d3210 2024-04-17T08:51:11.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.368+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7f161c007400 con 0x7f16300d3210 2024-04-17T08:51:11.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.369+0000 7f1626a00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 0x7f161006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.369+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f161c08ba80 con 0x7f16300d3210 2024-04-17T08:51:11.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.369+0000 7f162ea00640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 msgr2=0x7f161006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:11.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.369+0000 7f162ea00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 0x7f161006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:11.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.378+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f161c08de20 con 0x7f16300d3210 2024-04-17T08:51:11.378 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:11.378+0000 7fd63ca48040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:51:11.475 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.475+0000 7f3ff36da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:51:11.528 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.529+0000 7f3ff36da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:51:11.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.570+0000 7f162ea00640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 msgr2=0x7f161006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:11.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.570+0000 7f162ea00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 0x7f161006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:11.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.583+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f16300b5a50 con 0x7f16300d3210 2024-04-17T08:51:11.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.584+0000 7f1626a00640 1 -- 172.21.15.67:0/3942268638 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v235) ==== 103+0+11 (secure 0 0 0) 0x7f161c059640 con 0x7f16300d3210 2024-04-17T08:51:11.583 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 msgr2=0x7f161006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 0x7f161006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 msgr2=0x7f16301f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c400 tx=0x7f161c00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 shutdown_connections 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f161006cbc0 0x7f161006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f163015ecb0 0x7f1630160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16300d3210 0x7f16301f3d60 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 --2- 172.21.15.67:0/3942268638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16300cfab0 0x7f16300b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 >> 172.21.15.67:0/3942268638 conn(0x7f163008bff0 msgr2=0x7f163015df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 shutdown_connections 2024-04-17T08:51:11.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.587+0000 7f162fe00640 1 -- 172.21.15.67:0/3942268638 wait complete. 2024-04-17T08:51:11.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:51:11.676 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:11.676+0000 7f3ff36da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:51:11.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.865+0000 7fb268a00640 1 Processor -- start 2024-04-17T08:51:11.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.865+0000 7fb268a00640 1 -- start start 2024-04-17T08:51:11.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.866+0000 7fb268a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2640ba400 0x7fb2640d4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.866+0000 7fb268a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2640cf560 0x7fb2640cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.866+0000 7fb268a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2640d30b0 con 0x7fb26415eba0 2024-04-17T08:51:11.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb268a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2640d3390 con 0x7fb2640ba400 2024-04-17T08:51:11.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb268a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb2640d3670 con 0x7fb2640cf560 2024-04-17T08:51:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb262800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb26415eba0 0x7fb2640d28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47188/0 (socket says 172.21.15.67:47188) 2024-04-17T08:51:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb262800640 1 -- 172.21.15.67:0/2551859829 learned_addr learned my addr 172.21.15.67:0/2551859829 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261e00640 1 --2- 172.21.15.67:0/2551859829 >> v2:172.21.15.67:3300/0 conn(0x7fb2640ba400 0x7fb2640d4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb263200640 1 --2- 172.21.15.67:0/2551859829 >> v2:172.21.15.187:3300/0 conn(0x7fb2640cf560 0x7fb2640cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261400640 1 -- 172.21.15.67:0/2551859829 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873694277 0 0) 0x7fb2640d30b0 con 0x7fb26415eba0 2024-04-17T08:51:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261400640 1 -- 172.21.15.67:0/2551859829 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb248003520 con 0x7fb26415eba0 2024-04-17T08:51:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261e00640 1 -- 172.21.15.67:0/2551859829 >> v2:172.21.15.187:3300/0 conn(0x7fb2640cf560 msgr2=0x7fb2640cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261e00640 1 --2- 172.21.15.67:0/2551859829 >> v2:172.21.15.187:3300/0 conn(0x7fb2640cf560 0x7fb2640cf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.867+0000 7fb261e00640 1 -- 172.21.15.67:0/2551859829 >> v1:172.21.15.67:6789/0 conn(0x7fb26415eba0 legacy=0x7fb2640d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.868+0000 7fb261e00640 1 -- 172.21.15.67:0/2551859829 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2640d3950 con 0x7fb2640ba400 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.869+0000 7fb261e00640 1 --2- 172.21.15.67:0/2551859829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2640ba400 0x7fb2640d4150 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fb2640b7700 tx=0x7fb24c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac7ea2e8aa09f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.869+0000 7fb261400640 1 -- 172.21.15.67:0/2551859829 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0315a0 con 0x7fb2640ba400 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.869+0000 7fb261400640 1 -- 172.21.15.67:0/2551859829 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb24c037070 con 0x7fb2640ba400 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.869+0000 7fb261400640 1 -- 172.21.15.67:0/2551859829 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c031940 con 0x7fb2640ba400 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 -- 172.21.15.67:0/2551859829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2640ba400 msgr2=0x7fb2640d4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 --2- 172.21.15.67:0/2551859829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2640ba400 0x7fb2640d4150 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fb2640b7700 tx=0x7fb24c02b840 comp rx=0 tx=0).stop 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 -- 172.21.15.67:0/2551859829 shutdown_connections 2024-04-17T08:51:11.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 --2- 172.21.15.67:0/2551859829 >> v2:172.21.15.187:3300/0 conn(0x7fb2640cf560 0x7fb2640cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 --2- 172.21.15.67:0/2551859829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2640ba400 0x7fb2640d4150 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 -- 172.21.15.67:0/2551859829 >> 172.21.15.67:0/2551859829 conn(0x7fb26406cac0 msgr2=0x7fb26406cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 -- 172.21.15.67:0/2551859829 shutdown_connections 2024-04-17T08:51:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.870+0000 7fb268a00640 1 -- 172.21.15.67:0/2551859829 wait complete. 2024-04-17T08:51:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.871+0000 7fb268a00640 1 Processor -- start 2024-04-17T08:51:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.872+0000 7fb268a00640 1 -- start start 2024-04-17T08:51:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.872+0000 7fb268a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2640ba400 0x7fb26414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.872+0000 7fb268a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2640cf560 0x7fb2641505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb268a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb268a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2640b7ad0 con 0x7fb26415eba0 2024-04-17T08:51:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb268a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb2640b7950 con 0x7fb2640ba400 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb268a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2640b7c50 con 0x7fb2640cf560 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb262800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2640ba400 0x7fb26414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38878/0 (socket says 172.21.15.67:38878) 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 -- 172.21.15.67:0/3671408667 learned_addr learned my addr 172.21.15.67:0/3671408667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:11.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb261e00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2640cf560 0x7fb2641505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2640cf560 msgr2=0x7fb2641505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2640cf560 0x7fb2641505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2640ba400 msgr2=0x7fb26414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2640ba400 0x7fb26414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.873+0000 7fb263200640 1 -- 172.21.15.67:0/3671408667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2640cb8f0 con 0x7fb26415eba0 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb263200640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00e930 tx=0x7fb25c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c017070 con 0x7fb26415eba0 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb25c00c980 con 0x7fb26415eba0 2024-04-17T08:51:11.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c00cd70 con 0x7fb26415eba0 2024-04-17T08:51:11.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2640cbbd0 con 0x7fb26415eba0 2024-04-17T08:51:11.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.874+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2640cc190 con 0x7fb26415eba0 2024-04-17T08:51:11.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.875+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2640755d0 con 0x7fb26415eba0 2024-04-17T08:51:11.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.877+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7fb25c01a070 con 0x7fb26415eba0 2024-04-17T08:51:11.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.878+0000 7fb25aa00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 0x7fb24006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.878+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7fb25c08c3f0 con 0x7fb26415eba0 2024-04-17T08:51:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.878+0000 7fb262800640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 msgr2=0x7fb24006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.878+0000 7fb262800640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 0x7fb24006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:11.883+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb25c059a60 con 0x7fb26415eba0 2024-04-17T08:51:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.079+0000 7fb262800640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 msgr2=0x7fb24006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.079+0000 7fb262800640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 0x7fb24006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:51:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.103+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb2640caec0 con 0x7fb26415eba0 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.104+0000 7fb25aa00640 1 -- 172.21.15.67:0/3671408667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v235) ==== 80+0+473 (secure 0 0 0) 0x7fb25c05dd60 con 0x7fb26415eba0 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1585815760 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/629863885 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/150239855 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/150239855 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/665836648 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.103 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3650355481 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.104 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3246022161 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.104 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/291755012 2024-04-18T08:51:01.354257+0000 2024-04-17T08:51:12.104 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T08:51:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 msgr2=0x7fb24006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 0x7fb24006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 msgr2=0x7fb2640cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00e930 tx=0x7fb25c00edf0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 shutdown_connections 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7fb24006cbc0 0x7fb24006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb26415eba0 0x7fb2640cc990 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2640cf560 0x7fb2641505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 --2- 172.21.15.67:0/3671408667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2640ba400 0x7fb26414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 >> 172.21.15.67:0/3671408667 conn(0x7fb26406cac0 msgr2=0x7fb264160f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 shutdown_connections 2024-04-17T08:51:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.107+0000 7fb268a00640 1 -- 172.21.15.67:0/3671408667 wait complete. 2024-04-17T08:51:12.124 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:51:12.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:12.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:51:12.214 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.215+0000 7fd63ca48040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:51:12.257 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:51:12.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:12.258 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:51:12.298 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:51:12.299 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:51:12.299 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:12.352 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:51:12.353 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:51:12.353 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:12.353 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:12.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:51:12.401 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:51:12.401 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:51:12.401 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:12.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:51:12.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:12.526 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.526+0000 7fd63ca48040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:51:12.540 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:12.541+0000 7f3ff36da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:51:12.583 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.584+0000 7fd63ca48040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:51:12.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.668+0000 7f8a49c00640 1 Processor -- start 2024-04-17T08:51:12.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.668+0000 7f8a49c00640 1 -- start start 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8a44163060 0x7f8a440cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a440cf450 0x7f8a440b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4406e3e0 con 0x7f8a44161f40 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8a440cf8a0 con 0x7f8a440cf450 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4406cf50 con 0x7f8a44163060 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a49200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a440cf450 0x7f8a440b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a43e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a44161f40 0x7f8a4406dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47194/0 (socket says 172.21.15.67:47194) 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a43e00640 1 -- 172.21.15.67:0/3689270684 learned_addr learned my addr 172.21.15.67:0/3689270684 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.670+0000 7f8a3be00640 1 --2- 172.21.15.67:0/3689270684 >> v2:172.21.15.187:3300/0 conn(0x7f8a44163060 0x7f8a440cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.671+0000 7f8a49200640 1 -- 172.21.15.67:0/3689270684 >> v2:172.21.15.187:3300/0 conn(0x7f8a44163060 msgr2=0x7f8a440cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.671+0000 7f8a49200640 1 --2- 172.21.15.67:0/3689270684 >> v2:172.21.15.187:3300/0 conn(0x7f8a44163060 0x7f8a440cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.671+0000 7f8a49200640 1 -- 172.21.15.67:0/3689270684 >> v1:172.21.15.67:6789/0 conn(0x7f8a44161f40 legacy=0x7f8a4406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.671+0000 7f8a49200640 1 -- 172.21.15.67:0/3689270684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a440b3cf0 con 0x7f8a440cf450 2024-04-17T08:51:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.672+0000 7f8a49200640 1 --2- 172.21.15.67:0/3689270684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a440cf450 0x7f8a440b1ba0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f8a4409bbb0 tx=0x7f8a3c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e149c283ddf444f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.672+0000 7f8a43400640 1 -- 172.21.15.67:0/3689270684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c039070 con 0x7f8a440cf450 2024-04-17T08:51:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.672+0000 7f8a43400640 1 -- 172.21.15.67:0/3689270684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c02cc70 con 0x7f8a440cf450 2024-04-17T08:51:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.672+0000 7f8a43400640 1 -- 172.21.15.67:0/3689270684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c034660 con 0x7f8a440cf450 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 -- 172.21.15.67:0/3689270684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a440cf450 msgr2=0x7f8a440b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3689270684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a440cf450 0x7f8a440b1ba0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f8a4409bbb0 tx=0x7f8a3c02bec0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 -- 172.21.15.67:0/3689270684 shutdown_connections 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3689270684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a440cf450 0x7f8a440b1ba0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3689270684 >> v2:172.21.15.187:3300/0 conn(0x7f8a44163060 0x7f8a440cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 -- 172.21.15.67:0/3689270684 >> 172.21.15.67:0/3689270684 conn(0x7f8a440d2600 msgr2=0x7f8a440d2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:12.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 -- 172.21.15.67:0/3689270684 shutdown_connections 2024-04-17T08:51:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.673+0000 7f8a49c00640 1 -- 172.21.15.67:0/3689270684 wait complete. 2024-04-17T08:51:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.674+0000 7f8a49c00640 1 Processor -- start 2024-04-17T08:51:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.675+0000 7f8a49c00640 1 -- start start 2024-04-17T08:51:12.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.675+0000 7f8a49c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a440cf450 0x7f8a44159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a44163060 0x7f8a4415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4406d4d0 con 0x7f8a44161f40 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4406d1f0 con 0x7f8a44163060 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4406d7b0 con 0x7f8a440cf450 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38894/0 (socket says 172.21.15.67:38894) 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 -- 172.21.15.67:0/3825198277 learned_addr learned my addr 172.21.15.67:0/3825198277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a49200640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a44163060 0x7f8a4415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a43e00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a440cf450 0x7f8a44159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 -- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a440cf450 msgr2=0x7f8a44159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a440cf450 0x7f8a44159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 -- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a44163060 msgr2=0x7f8a4415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a44163060 0x7f8a4415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.676+0000 7f8a3be00640 1 -- 172.21.15.67:0/3825198277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a44159940 con 0x7f8a44161f40 2024-04-17T08:51:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a3be00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00d830 tx=0x7f8a2c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c00b6c0 con 0x7f8a44161f40 2024-04-17T08:51:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2c019070 con 0x7f8a44161f40 2024-04-17T08:51:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c00ba80 con 0x7f8a44161f40 2024-04-17T08:51:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a441606f0 con 0x7f8a44161f40 2024-04-17T08:51:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.677+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a44160cb0 con 0x7f8a44161f40 2024-04-17T08:51:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.678+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a440755d0 con 0x7f8a44161f40 2024-04-17T08:51:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.680+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 135) ==== 91301+0+0 (secure 0 0 0) 0x7f8a2c01e460 con 0x7f8a44161f40 2024-04-17T08:51:12.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.681+0000 7f8a40c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 0x7f8a2006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.681+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6945+0+0 (secure 0 0 0) 0x7f8a2c08cb60 con 0x7f8a44161f40 2024-04-17T08:51:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.681+0000 7f8a43e00640 1 -- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 msgr2=0x7f8a2006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/3872755981 2024-04-17T08:51:12.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.681+0000 7f8a43e00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 0x7f8a2006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:51:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.686+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8a2c05a760 con 0x7f8a44161f40 2024-04-17T08:51:12.714 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.714+0000 7fd63ca48040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:51:12.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.737+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 136) ==== 90751+0+0 (secure 0 0 0) 0x7f8a2c0566d0 con 0x7f8a44161f40 2024-04-17T08:51:12.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.737+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 msgr2=0x7f8a2006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:51:12.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.737+0000 7f8a40c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 0x7f8a2006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.817 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.818+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.869 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.818+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.870 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.818+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.870 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.818+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.870 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.818+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.819+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.819+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.819+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.819+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.819+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.833+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.833+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.833+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.833+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.833+0000 7fd5b6800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.834+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.834+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.834+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.834+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.872 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:51:12.834+0000 7fd5ba400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:51:12.884 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:12.885+0000 7f3ff36da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:51:12.948 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:12.948+0000 7f3ff36da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:51:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.949+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a4409bea0 con 0x7f8a44161f40 2024-04-17T08:51:12.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.951+0000 7f8a40c00640 1 -- 172.21.15.67:0/3825198277 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7f8a2c076960 con 0x7f8a44161f40 2024-04-17T08:51:12.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:12.950 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 msgr2=0x7f8a440cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00d830 tx=0x7f8a2c00dcf0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 shutdown_connections 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:6832/3872755981,v1:172.21.15.187:6833/3872755981] conn(0x7f8a2006cbc0 0x7f8a2006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a44163060 0x7f8a4415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a44161f40 0x7f8a440cdd20 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 --2- 172.21.15.67:0/3825198277 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8a440cf450 0x7f8a44159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.953+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 >> 172.21.15.67:0/3825198277 conn(0x7f8a440d2600 msgr2=0x7f8a4409b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.954+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 shutdown_connections 2024-04-17T08:51:12.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:12.954+0000 7f8a49c00640 1 -- 172.21.15.67:0/3825198277 wait complete. 2024-04-17T08:51:12.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:13.099 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:51:13.099+0000 7f3ff36da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:51:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.164+0000 7f4b22000640 1 Processor -- start 2024-04-17T08:51:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.164+0000 7f4b22000640 1 -- start start 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b22000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b1c0d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b22000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b1c0d3020 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b22000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b1c0d3300 con 0x7f4b1c0cf640 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b22000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1c0d3c30 con 0x7f4b1c15ed10 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b1be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4b1c0d2ab0 0x7f4b1c0b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47202/0 (socket says 172.21.15.67:47202) 2024-04-17T08:51:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.166+0000 7f4b1be00640 1 -- 172.21.15.67:0/3652907741 learned_addr learned my addr 172.21.15.67:0/3652907741 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b12a00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b1c0d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78312624 0 0) 0x7f4b1c0d3020 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b00003520 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 406466415 0 0) 0x7f4b00003520 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 msgr2=0x7f4b1c0d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b1c0d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 >> v1:172.21.15.187:6789/0 conn(0x7f4b1c0cf640 legacy=0x7f4b1c0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c0d35e0 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4b14001670 con 0x7f4b1c0d2ab0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 >> v1:172.21.15.67:6789/0 conn(0x7f4b1c0d2ab0 legacy=0x7f4b1c0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b000039e0 0x7f4b00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b00004d40 0x7f4b000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c0d3c30 con 0x7f4b000039e0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0000bc30 con 0x7f4b00004d40 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0000be30 con 0x7f4b1c15ed10 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.167+0000 7f4b1b400640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b000039e0 0x7f4b00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1be00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b00004d40 0x7f4b000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b12a00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1b400640 1 -- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 msgr2=0x7f4b00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1b400640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b1c15ed10 0x7f4b00004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1b400640 1 -- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b00004d40 msgr2=0x7f4b000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1b400640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b00004d40 0x7f4b000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1b400640 1 -- 172.21.15.67:0/3652907741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b00003520 con 0x7f4b000039e0 2024-04-17T08:51:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.168+0000 7f4b1be00640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b00004d40 0x7f4b000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.169+0000 7f4b1b400640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b000039e0 0x7f4b00003e00 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c001e50 tx=0x7f4b00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3734d7fc002f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.169+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c001a70 con 0x7f4b000039e0 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.169+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0c003b70 con 0x7f4b000039e0 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.169+0000 7f4b1aa00640 1 -- 172.21.15.67:0/3652907741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0050d0 con 0x7f4b000039e0 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 -- 172.21.15.67:0/3652907741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b000039e0 msgr2=0x7f4b00003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 --2- 172.21.15.67:0/3652907741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b000039e0 0x7f4b00003e00 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c001e50 tx=0x7f4b00005830 comp rx=0 tx=0).stop 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b1be00640 1 -- 172.21.15.67:0/3652907741 reap_dead start 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 -- 172.21.15.67:0/3652907741 shutdown_connections 2024-04-17T08:51:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 -- 172.21.15.67:0/3652907741 >> 172.21.15.67:0/3652907741 conn(0x7f4b1c06cbd0 msgr2=0x7f4b1c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 -- 172.21.15.67:0/3652907741 shutdown_connections 2024-04-17T08:51:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.170+0000 7f4b22000640 1 -- 172.21.15.67:0/3652907741 wait complete. 2024-04-17T08:51:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.172+0000 7f4b22000640 1 Processor -- start 2024-04-17T08:51:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.172+0000 7f4b22000640 1 -- start start 2024-04-17T08:51:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b000039e0 0x7f4b1c14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b1c14def0 0x7f4b1c0ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c0ba300 con 0x7f4b1c0ca840 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1c0d25b0 con 0x7f4b1c14def0 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b22000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1c0ba480 con 0x7f4b000039e0 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38924/0 (socket says 172.21.15.67:38924) 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.173+0000 7f4b1be00640 1 -- 172.21.15.67:0/2214018644 learned_addr learned my addr 172.21.15.67:0/2214018644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 -- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b000039e0 msgr2=0x7f4b1c14a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b000039e0 0x7f4b1c14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 -- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b1c14def0 msgr2=0x7f4b1c0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b1c14def0 0x7f4b1c0ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 -- 172.21.15.67:0/2214018644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1c14aef0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b1be00640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0cb9d0 tx=0x7f4b1400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14007230 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b140073f0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1c1f8bc0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b140047e0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.174+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1c14d6f0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.175+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1c068b90 con 0x7f4b1c0ca840 2024-04-17T08:51:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.176+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 136) ==== 90751+0+0 (secure 0 0 0) 0x7f4b1401b020 con 0x7f4b1c0ca840 2024-04-17T08:51:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.177+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f4b1408b870 con 0x7f4b1c0ca840 2024-04-17T08:51:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.182+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4b14059950 con 0x7f4b1c0ca840 2024-04-17T08:51:13.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.386+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b1c0b40b0 con 0x7f4b1c0ca840 2024-04-17T08:51:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.390+0000 7f4b13e00640 1 -- 172.21.15.67:0/2214018644 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) ==== 74+0+21671 (secure 0 0 0) 0x7f4b14059250 con 0x7f4b1c0ca840 2024-04-17T08:51:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:13.391 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":236,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:12.720937+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:13.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.392+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 msgr2=0x7f4b1c1f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.392+0000 7f4b22000640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0cb9d0 tx=0x7f4b1400dc00 comp rx=0 tx=0).stop 2024-04-17T08:51:13.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 shutdown_connections 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c0ca840 0x7f4b1c1f7f00 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4b1c14def0 0x7f4b1c0ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 --2- 172.21.15.67:0/2214018644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4b000039e0 0x7f4b1c14a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 >> 172.21.15.67:0/2214018644 conn(0x7f4b1c06cbd0 msgr2=0x7f4b1c160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 shutdown_connections 2024-04-17T08:51:13.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.393+0000 7f4b22000640 1 -- 172.21.15.67:0/2214018644 wait complete. 2024-04-17T08:51:13.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:50:31.281467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:50:32.343386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:50:46.826027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:50:47.478489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:13.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:13.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.637+0000 7f23e7200640 1 Processor -- start 2024-04-17T08:51:13.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.637+0000 7f23e7200640 1 -- start start 2024-04-17T08:51:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e80d2ff0 0x7f23e80b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f23e81622b0 0x7f23e80cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23e80b5110 con 0x7f23e80b4c70 2024-04-17T08:51:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f23e80b53f0 con 0x7f23e80d2ff0 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f23e80b56d0 con 0x7f23e81622b0 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e80d2ff0 0x7f23e80b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e80d2ff0 0x7f23e80b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37638/0 (socket says 172.21.15.67:37638) 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e5e00640 1 -- 172.21.15.67:0/3137624962 learned_addr learned my addr 172.21.15.67:0/3137624962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e5400640 1 --1- 172.21.15.67:0/3137624962 >> v1:172.21.15.67:6789/0 conn(0x7f23e80b4c70 0x7f23e8161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52304/0 (socket says 172.21.15.67:52304) 2024-04-17T08:51:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.639+0000 7f23e6800640 1 --2- 172.21.15.67:0/3137624962 >> v2:172.21.15.187:3300/0 conn(0x7f23e81622b0 0x7f23e80cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.640+0000 7f23e5e00640 1 -- 172.21.15.67:0/3137624962 >> v2:172.21.15.187:3300/0 conn(0x7f23e81622b0 msgr2=0x7f23e80cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.640+0000 7f23e5e00640 1 --2- 172.21.15.67:0/3137624962 >> v2:172.21.15.187:3300/0 conn(0x7f23e81622b0 0x7f23e80cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.640+0000 7f23e5e00640 1 -- 172.21.15.67:0/3137624962 >> v1:172.21.15.67:6789/0 conn(0x7f23e80b4c70 legacy=0x7f23e8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.640+0000 7f23e5e00640 1 -- 172.21.15.67:0/3137624962 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e80cf340 con 0x7f23e80d2ff0 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.640+0000 7f23e4a00640 1 -- 172.21.15.67:0/3137624962 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057248100 0 0) 0x7f23e80b5110 con 0x7f23e80b4c70 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.641+0000 7f23e5e00640 1 --2- 172.21.15.67:0/3137624962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80d2ff0 0x7f23e80b4730 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f23e80d5850 tx=0x7f23d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c549aed13bc4d645 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.641+0000 7f23e4a00640 1 -- 172.21.15.67:0/3137624962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8034070 con 0x7f23e80d2ff0 2024-04-17T08:51:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.641+0000 7f23e4a00640 1 -- 172.21.15.67:0/3137624962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d8030860 con 0x7f23e80d2ff0 2024-04-17T08:51:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.641+0000 7f23e4a00640 1 -- 172.21.15.67:0/3137624962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8030c30 con 0x7f23e80d2ff0 2024-04-17T08:51:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 -- 172.21.15.67:0/3137624962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80d2ff0 msgr2=0x7f23e80b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 --2- 172.21.15.67:0/3137624962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80d2ff0 0x7f23e80b4730 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f23e80d5850 tx=0x7f23d802b9a0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 -- 172.21.15.67:0/3137624962 shutdown_connections 2024-04-17T08:51:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 --2- 172.21.15.67:0/3137624962 >> v2:172.21.15.187:3300/0 conn(0x7f23e81622b0 0x7f23e80cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 --2- 172.21.15.67:0/3137624962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80d2ff0 0x7f23e80b4730 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 -- 172.21.15.67:0/3137624962 >> 172.21.15.67:0/3137624962 conn(0x7f23e809da80 msgr2=0x7f23e808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 -- 172.21.15.67:0/3137624962 shutdown_connections 2024-04-17T08:51:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.642+0000 7f23e7200640 1 -- 172.21.15.67:0/3137624962 wait complete. 2024-04-17T08:51:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.644+0000 7f23e7200640 1 Processor -- start 2024-04-17T08:51:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.644+0000 7f23e7200640 1 -- start start 2024-04-17T08:51:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23e80d2ff0 0x7f23e8152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23e81622b0 0x7f23e80d4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23e80d3550 con 0x7f23e80b4c70 2024-04-17T08:51:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f23e80d33d0 con 0x7f23e80d2ff0 2024-04-17T08:51:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23e80d36d0 con 0x7f23e81622b0 2024-04-17T08:51:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37648/0 (socket says 172.21.15.67:37648) 2024-04-17T08:51:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.645+0000 7f23e5e00640 1 -- 172.21.15.67:0/1729920394 learned_addr learned my addr 172.21.15.67:0/1729920394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 -- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23e81622b0 msgr2=0x7f23e80d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23e81622b0 0x7f23e80d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 -- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23e80d2ff0 msgr2=0x7f23e8152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23e80d2ff0 0x7f23e8152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 -- 172.21.15.67:0/1729920394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e81f7a80 con 0x7f23e80b4c70 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e5e00640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f23d802b9e0 tx=0x7f23d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8034040 con 0x7f23e80b4c70 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d8039d90 con 0x7f23e80b4c70 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8043630 con 0x7f23e80b4c70 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e81f7d60 con 0x7f23e80b4c70 2024-04-17T08:51:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.646+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e8153380 con 0x7f23e80b4c70 2024-04-17T08:51:13.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.647+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e80755d0 con 0x7f23e80b4c70 2024-04-17T08:51:13.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.648+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 136) ==== 90751+0+0 (secure 0 0 0) 0x7f23d8037070 con 0x7f23e80b4c70 2024-04-17T08:51:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.650+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f23d80ade10 con 0x7f23e80b4c70 2024-04-17T08:51:13.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.655+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f23d803e070 con 0x7f23e80b4c70 2024-04-17T08:51:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.746+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 137) ==== 90878+0+0 (secure 0 0 0) 0x7f23d80b1030 con 0x7f23e80b4c70 2024-04-17T08:51:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.747+0000 7f23de000640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 0x7f23c4076c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:13.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.749+0000 7f23e5400640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 0x7f23c4076c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:13.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.754+0000 7f23e5400640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 0x7f23c4076c00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f23d0007fd0 tx=0x7f23d0007980 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:13.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.863+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f23e80b3f50 con 0x7f23e80b4c70 2024-04-17T08:51:13.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.865+0000 7f23de000640 1 -- 172.21.15.67:0/1729920394 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v236) ==== 93+0+10 (secure 0 0 0) 0x7f23d807c100 con 0x7f23e80b4c70 2024-04-17T08:51:13.864 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 msgr2=0x7f23c4076c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 0x7f23c4076c00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f23d0007fd0 tx=0x7f23d0007980 comp rx=0 tx=0).stop 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 msgr2=0x7f23e80cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f23d802b9e0 tx=0x7f23d80046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 shutdown_connections 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f23c4074810 0x7f23c4076c00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f23e81622b0 0x7f23e80d4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23e80d2ff0 0x7f23e8152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 --2- 172.21.15.67:0/1729920394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e80b4c70 0x7f23e80cad60 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.867+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 >> 172.21.15.67:0/1729920394 conn(0x7f23e809da80 msgr2=0x7f23e81610f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.868+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 shutdown_connections 2024-04-17T08:51:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:13.868+0000 7f23e7200640 1 -- 172.21.15.67:0/1729920394 wait complete. 2024-04-17T08:51:13.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:51:14.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.111+0000 7fc689e00640 1 Processor -- start 2024-04-17T08:51:14.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.112+0000 7fc689e00640 1 -- start start 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.113+0000 7fc689e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc6840cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.113+0000 7fc689e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6840d4100 con 0x7fc6840cfa10 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.113+0000 7fc689e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc684163860 con 0x7fc68415eca0 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.113+0000 7fc689e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc6840d03b0 con 0x7fc6840d17a0 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.114+0000 7fc67a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6840cfa10 0x7fc6840d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52320/0 (socket says 172.21.15.67:52320) 2024-04-17T08:51:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.114+0000 7fc67a000640 1 -- 172.21.15.67:0/2473780785 learned_addr learned my addr 172.21.15.67:0/2473780785 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.114+0000 7fc683400640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc6840cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.114+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671065025 0 0) 0x7fc6840d4100 con 0x7fc6840cfa10 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.114+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc668003520 con 0x7fc6840cfa10 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4103242811 0 0) 0x7fc668003520 con 0x7fc6840cfa10 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 msgr2=0x7fc6840cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc6840cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 >> v1:172.21.15.187:6789/0 conn(0x7fc68415eca0 legacy=0x7fc6840d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6840d1d40 con 0x7fc6840cfa10 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc66c001690 con 0x7fc6840cfa10 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 >> v1:172.21.15.67:6789/0 conn(0x7fc6840cfa10 legacy=0x7fc6840d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6680039e0 0x7fc668003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc668004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc668004d40 0x7fc6680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc66800bb50 con 0x7fc6680039e0 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc66800bcd0 con 0x7fc668004d40 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc66800bed0 con 0x7fc6840d17a0 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc682a00640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6680039e0 0x7fc668003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc67a000640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc668004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.115+0000 7fc683400640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc668004d40 0x7fc6680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.116+0000 7fc682a00640 1 -- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 msgr2=0x7fc668004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.116+0000 7fc682a00640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc6840d17a0 0x7fc668004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.116+0000 7fc682a00640 1 -- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc668004d40 msgr2=0x7fc6680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:14.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.116+0000 7fc682a00640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc668004d40 0x7fc6680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.116+0000 7fc682a00640 1 -- 172.21.15.67:0/2473780785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc668003520 con 0x7fc6680039e0 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc682a00640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6680039e0 0x7fc668003e00 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fc674001750 tx=0x7fc668005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b5b80c2bb6579c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674001370 con 0x7fc6680039e0 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6740052d0 con 0x7fc6680039e0 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc682000640 1 -- 172.21.15.67:0/2473780785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc674003830 con 0x7fc6680039e0 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc689e00640 1 -- 172.21.15.67:0/2473780785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6680039e0 msgr2=0x7fc668003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.117+0000 7fc689e00640 1 --2- 172.21.15.67:0/2473780785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6680039e0 0x7fc668003e00 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fc674001750 tx=0x7fc668005830 comp rx=0 tx=0).stop 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.118+0000 7fc683400640 1 -- 172.21.15.67:0/2473780785 reap_dead start 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.118+0000 7fc689e00640 1 -- 172.21.15.67:0/2473780785 shutdown_connections 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.118+0000 7fc689e00640 1 -- 172.21.15.67:0/2473780785 >> 172.21.15.67:0/2473780785 conn(0x7fc684067600 msgr2=0x7fc6840679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.118+0000 7fc689e00640 1 -- 172.21.15.67:0/2473780785 shutdown_connections 2024-04-17T08:51:14.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.118+0000 7fc689e00640 1 -- 172.21.15.67:0/2473780785 wait complete. 2024-04-17T08:51:14.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.119+0000 7fc689e00640 1 Processor -- start 2024-04-17T08:51:14.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.120+0000 7fc689e00640 1 -- start start 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6680039e0 0x7fc6840c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc684149d30 0x7fc6840b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6840d31f0 con 0x7fc68415f110 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc6840d2f10 con 0x7fc6680039e0 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc689e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc6840d34d0 con 0x7fc684149d30 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37672/0 (socket says 172.21.15.67:37672) 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 -- 172.21.15.67:0/1767360273 learned_addr learned my addr 172.21.15.67:0/1767360273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc67a000640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc684149d30 0x7fc6840b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc682a00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6680039e0 0x7fc6840c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 -- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc684149d30 msgr2=0x7fc6840b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc684149d30 0x7fc6840b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 -- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6680039e0 msgr2=0x7fc6840c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6680039e0 0x7fc6840c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.121+0000 7fc683400640 1 -- 172.21.15.67:0/1767360273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6840c9b50 con 0x7fc68415f110 2024-04-17T08:51:14.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc683400640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fc6840d0d50 tx=0x7fc67c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c011e10 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c01b070 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6840c9e30 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c003990 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.122+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6840ca3f0 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.124+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc684068ca0 con 0x7fc68415f110 2024-04-17T08:51:14.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.124+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 137) ==== 90878+0+0 (secure 0 0 0) 0x7fc67c005d10 con 0x7fc68415f110 2024-04-17T08:51:14.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.125+0000 7fc67be00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 0x7fc66006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.125+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7fc67c08c370 con 0x7fc68415f110 2024-04-17T08:51:14.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.126+0000 7fc682a00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 0x7fc66006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.127+0000 7fc682a00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 0x7fc66006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc674033000 tx=0x7fc674009030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.131+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc67c025020 con 0x7fc68415f110 2024-04-17T08:51:14.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.336+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6840d2970 con 0x7fc68415f110 2024-04-17T08:51:14.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.337+0000 7fc67be00640 1 -- 172.21.15.67:0/1767360273 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v236) ==== 104+0+11 (secure 0 0 0) 0x7fc67c014350 con 0x7fc68415f110 2024-04-17T08:51:14.336 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 msgr2=0x7fc66006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 0x7fc66006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc674033000 tx=0x7fc674009030 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 msgr2=0x7fc68415f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fc6840d0d50 tx=0x7fc67c0115a0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 shutdown_connections 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc66006c740 0x7fc66006ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc68415f110 0x7fc68415f590 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc684149d30 0x7fc6840b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 --2- 172.21.15.67:0/1767360273 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6680039e0 0x7fc6840c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 >> 172.21.15.67:0/1767360273 conn(0x7fc684067600 msgr2=0x7fc6840b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 shutdown_connections 2024-04-17T08:51:14.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.340+0000 7fc689e00640 1 -- 172.21.15.67:0/1767360273 wait complete. 2024-04-17T08:51:14.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:51:14.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.584+0000 7f5473c00640 1 Processor -- start 2024-04-17T08:51:14.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.584+0000 7f5473c00640 1 -- start start 2024-04-17T08:51:14.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5473c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c15ea20 0x7f546c0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5473c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f546c0cf370 0x7f546c0b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5473c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f546c0cf800 con 0x7f546c0d3ea0 2024-04-17T08:51:14.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5473c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f546c0d2590 con 0x7f546c15ea20 2024-04-17T08:51:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5473c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f546c0d0060 con 0x7f546c0cf370 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.585+0000 7f5471200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c15ea20 0x7f546c0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f546c15ea20 0x7f546c0b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37686/0 (socket says 172.21.15.67:37686) 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 -- 172.21.15.67:0/300112956 learned_addr learned my addr 172.21.15.67:0/300112956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471c00640 1 --1- 172.21.15.67:0/300112956 >> v1:172.21.15.67:6789/0 conn(0x7f546c0d3ea0 0x7f546c0b7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52330/0 (socket says 172.21.15.67:52330) 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f546be00640 1 --2- 172.21.15.67:0/300112956 >> v2:172.21.15.187:3300/0 conn(0x7f546c0cf370 0x7f546c0b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 -- 172.21.15.67:0/300112956 >> v2:172.21.15.187:3300/0 conn(0x7f546c0cf370 msgr2=0x7f546c0b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 --2- 172.21.15.67:0/300112956 >> v2:172.21.15.187:3300/0 conn(0x7f546c0cf370 0x7f546c0b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 -- 172.21.15.67:0/300112956 >> v1:172.21.15.67:6789/0 conn(0x7f546c0d3ea0 legacy=0x7f546c0b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f5471200640 1 -- 172.21.15.67:0/300112956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c0cfaa0 con 0x7f546c15ea20 2024-04-17T08:51:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.586+0000 7f546b400640 1 -- 172.21.15.67:0/300112956 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499753866 0 0) 0x7f546c0cf800 con 0x7f546c0d3ea0 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.587+0000 7f5471200640 1 --2- 172.21.15.67:0/300112956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15ea20 0x7f546c0b4000 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f546c0b7720 tx=0x7f545c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36945915b079410 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f546b400640 1 -- 172.21.15.67:0/300112956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c034070 con 0x7f546c15ea20 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f546b400640 1 -- 172.21.15.67:0/300112956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545c030860 con 0x7f546c15ea20 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f546b400640 1 -- 172.21.15.67:0/300112956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c030c30 con 0x7f546c15ea20 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 -- 172.21.15.67:0/300112956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15ea20 msgr2=0x7f546c0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 --2- 172.21.15.67:0/300112956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15ea20 0x7f546c0b4000 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f546c0b7720 tx=0x7f545c02b9a0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 -- 172.21.15.67:0/300112956 shutdown_connections 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 --2- 172.21.15.67:0/300112956 >> v2:172.21.15.187:3300/0 conn(0x7f546c0cf370 0x7f546c0b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 --2- 172.21.15.67:0/300112956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c15ea20 0x7f546c0b4000 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 -- 172.21.15.67:0/300112956 >> 172.21.15.67:0/300112956 conn(0x7f546c0679f0 msgr2=0x7f546c067e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 -- 172.21.15.67:0/300112956 shutdown_connections 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.588+0000 7f5473c00640 1 -- 172.21.15.67:0/300112956 wait complete. 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.590+0000 7f5473c00640 1 Processor -- start 2024-04-17T08:51:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.590+0000 7f5473c00640 1 -- start start 2024-04-17T08:51:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f546c0d3ea0 0x7f546c0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f546c15ea20 0x7f546c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f546c0b6760 con 0x7f546c0cf370 2024-04-17T08:51:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f546c0b74a0 con 0x7f546c15ea20 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5473c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f546c0b6a40 con 0x7f546c0d3ea0 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5471200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5471200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37698/0 (socket says 172.21.15.67:37698) 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.591+0000 7f5471200640 1 -- 172.21.15.67:0/1611362714 learned_addr learned my addr 172.21.15.67:0/1611362714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 -- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f546c0d3ea0 msgr2=0x7f546c0ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f546c0d3ea0 0x7f546c0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 -- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f546c15ea20 msgr2=0x7f546c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f546c15ea20 0x7f546c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 -- 172.21.15.67:0/1611362714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c1f7e70 con 0x7f546c0cf370 2024-04-17T08:51:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5471200640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f545c000c00 tx=0x7f545c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.592+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c034040 con 0x7f546c0cf370 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.593+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545c03ab90 con 0x7f546c0cf370 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.593+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c004030 con 0x7f546c0cf370 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.593+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546c1f8150 con 0x7f546c0cf370 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.593+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546c1f8710 con 0x7f546c0cf370 2024-04-17T08:51:14.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.594+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f546c068b90 con 0x7f546c0cf370 2024-04-17T08:51:14.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.595+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 137) ==== 90878+0+0 (secure 0 0 0) 0x7f545c04b060 con 0x7f546c0cf370 2024-04-17T08:51:14.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.596+0000 7f5468c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 0x7f544806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:14.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.596+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f545c0ade00 con 0x7f546c0cf370 2024-04-17T08:51:14.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.596+0000 7f546be00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 0x7f544806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:14.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.598+0000 7f546be00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 0x7f544806ec30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54540070b0 tx=0x7f5454007040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:14.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.601+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f545c03f070 con 0x7f546c0cf370 2024-04-17T08:51:14.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.759+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 138) ==== 91188+0+0 (secure 0 0 0) 0x7f545c037070 con 0x7f546c0cf370 2024-04-17T08:51:14.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.811+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f546c0d1f10 con 0x7f546c0cf370 2024-04-17T08:51:14.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.813+0000 7f5468c00640 1 -- 172.21.15.67:0/1611362714 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v236) ==== 100+0+11 (secure 0 0 0) 0x7f545c038e90 con 0x7f546c0cf370 2024-04-17T08:51:14.812 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 msgr2=0x7f544806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 0x7f544806ec30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54540070b0 tx=0x7f5454007040 comp rx=0 tx=0).stop 2024-04-17T08:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 msgr2=0x7f546c1f4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f545c000c00 tx=0x7f545c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 shutdown_connections 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f544806c770 0x7f544806ec30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f546c15ea20 0x7f546c0b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f546c0d3ea0 0x7f546c0ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 --2- 172.21.15.67:0/1611362714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f546c0cf370 0x7f546c1f4400 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 >> 172.21.15.67:0/1611362714 conn(0x7f546c0679f0 msgr2=0x7f546c161940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 shutdown_connections 2024-04-17T08:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:14.816+0000 7f5473c00640 1 -- 172.21.15.67:0/1611362714 wait complete. 2024-04-17T08:51:14.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:51:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.058+0000 7f1698c00640 1 Processor -- start 2024-04-17T08:51:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.059+0000 7f1698c00640 1 -- start start 2024-04-17T08:51:15.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1698c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1694160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1698c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1694160920 con 0x7f16940b4ac0 2024-04-17T08:51:15.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1698c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1694160c00 con 0x7f1694145b00 2024-04-17T08:51:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1698c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1694160ee0 con 0x7f1694146060 2024-04-17T08:51:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1693e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f16940b4ac0 0x7f16940b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52332/0 (socket says 172.21.15.67:52332) 2024-04-17T08:51:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.060+0000 7f1693e00640 1 -- 172.21.15.67:0/2896466581 learned_addr learned my addr 172.21.15.67:0/2896466581 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692a00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1694160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721565344 0 0) 0x7f1694160920 con 0x7f16940b4ac0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1678003520 con 0x7f16940b4ac0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2673011166 0 0) 0x7f1678003520 con 0x7f16940b4ac0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 msgr2=0x7f1694160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1694160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 >> v1:172.21.15.187:6789/0 conn(0x7f1694145b00 legacy=0x7f169415cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16941611c0 con 0x7f16940b4ac0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f168c0019f0 con 0x7f16940b4ac0 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 >> v1:172.21.15.67:6789/0 conn(0x7f16940b4ac0 legacy=0x7f16940b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16780039e0 0x7f1678003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.061+0000 7f1692000640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f1678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692000640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1678005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1694160ee0 con 0x7f1678004800 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f167800bb20 con 0x7f16780039e0 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f167800bd20 con 0x7f1694146060 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f1678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1693400640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16780039e0 0x7f1678003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1693e00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1678005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 -- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 msgr2=0x7f1678005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1678005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 -- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16780039e0 msgr2=0x7f1678003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16780039e0 0x7f1678003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1692a00640 1 -- 172.21.15.67:0/2896466581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1678003520 con 0x7f1678004800 2024-04-17T08:51:15.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.062+0000 7f1693e00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1694146060 0x7f1678005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:15.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.063+0000 7f1692a00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f1678004c80 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f167c009ac0 tx=0x7f1678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202229bdc53cad89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c039070 con 0x7f1678004800 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f167c02c6b0 con 0x7f1678004800 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1692000640 1 -- 172.21.15.67:0/2896466581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167c009690 con 0x7f1678004800 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 -- 172.21.15.67:0/2896466581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 msgr2=0x7f1678004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 --2- 172.21.15.67:0/2896466581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f1678004c80 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f167c009ac0 tx=0x7f1678005830 comp rx=0 tx=0).stop 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1693e00640 1 -- 172.21.15.67:0/2896466581 reap_dead start 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 -- 172.21.15.67:0/2896466581 shutdown_connections 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 -- 172.21.15.67:0/2896466581 >> 172.21.15.67:0/2896466581 conn(0x7f16940b23c0 msgr2=0x7f16940b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 -- 172.21.15.67:0/2896466581 shutdown_connections 2024-04-17T08:51:15.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.064+0000 7f1698c00640 1 -- 172.21.15.67:0/2896466581 wait complete. 2024-04-17T08:51:15.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.066+0000 7f1698c00640 1 Processor -- start 2024-04-17T08:51:15.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.066+0000 7f1698c00640 1 -- start start 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 0x7f169415d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 0x7f169413f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1694160340 con 0x7f1678004800 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f16940cfba0 con 0x7f169413e3d0 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.067+0000 7f1698c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16941604c0 con 0x7f169415dca0 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37708/0 (socket says 172.21.15.67:37708) 2024-04-17T08:51:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 -- 172.21.15.67:0/2354474106 learned_addr learned my addr 172.21.15.67:0/2354474106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:15.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1692a00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 0x7f169415d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693e00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 0x7f169413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 -- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 msgr2=0x7f169413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 0x7f169413f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 -- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 msgr2=0x7f169415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 0x7f169415d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 -- 172.21.15.67:0/2354474106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169413e8b0 con 0x7f1678004800 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1692a00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 0x7f169415d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693e00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 0x7f169413f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.068+0000 7f1693400640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f168400e550 tx=0x7f168400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.069+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16840052e0 con 0x7f1678004800 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.069+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1684018070 con 0x7f1678004800 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.069+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684003810 con 0x7f1678004800 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.069+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169413eb90 con 0x7f1678004800 2024-04-17T08:51:15.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.069+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169413f150 con 0x7f1678004800 2024-04-17T08:51:15.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.070+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1694068a10 con 0x7f1678004800 2024-04-17T08:51:15.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.071+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 138) ==== 91188+0+0 (secure 0 0 0) 0x7f16840039b0 con 0x7f1678004800 2024-04-17T08:51:15.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.072+0000 7f168b400640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 0x7f167006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.072+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f168408c230 con 0x7f1678004800 2024-04-17T08:51:15.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.073+0000 7f1692a00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 0x7f167006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.074+0000 7f1692a00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 0x7f167006ef90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f167c036000 tx=0x7f167c030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.078+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1684059d20 con 0x7f1678004800 2024-04-17T08:51:15.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.280+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f1694068610 con 0x7f1678004800 2024-04-17T08:51:15.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.281+0000 7f168b400640 1 -- 172.21.15.67:0/2354474106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v236) ==== 107+0+11 (secure 0 0 0) 0x7f1684059620 con 0x7f1678004800 2024-04-17T08:51:15.280 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:15.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.283+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 msgr2=0x7f167006ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.283+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 0x7f167006ef90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f167c036000 tx=0x7f167c030000 comp rx=0 tx=0).stop 2024-04-17T08:51:15.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.283+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 msgr2=0x7f169413cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.283+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f168400e550 tx=0x7f168400ea10 comp rx=0 tx=0).stop 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 shutdown_connections 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f167006cad0 0x7f167006ef90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f169415dca0 0x7f169413f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f169413e3d0 0x7f169415d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 --2- 172.21.15.67:0/2354474106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678004800 0x7f169413cf90 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 >> 172.21.15.67:0/2354474106 conn(0x7f16940b23c0 msgr2=0x7f1694143da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 shutdown_connections 2024-04-17T08:51:15.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.284+0000 7f1698c00640 1 -- 172.21.15.67:0/2354474106 wait complete. 2024-04-17T08:51:15.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.529+0000 7f368da00640 1 Processor -- start 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.529+0000 7f368da00640 1 -- start start 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f368da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36880cd920 0x7f368806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f368da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3688146360 con 0x7f3688145f80 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f368da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3688071e70 con 0x7f368806c0a0 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f368da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f36880727a0 con 0x7f36880cd920 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f3687400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36880cd920 0x7f368806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f3687e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3688145f80 0x7f36880cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52348/0 (socket says 172.21.15.67:52348) 2024-04-17T08:51:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f3687400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36880cd920 0x7f368806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37724/0 (socket says 172.21.15.67:37724) 2024-04-17T08:51:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.531+0000 7f3687e00640 1 -- 172.21.15.67:0/3372614142 learned_addr learned my addr 172.21.15.67:0/3372614142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.532+0000 7f3687400640 1 -- 172.21.15.67:0/3372614142 >> v1:172.21.15.187:6789/0 conn(0x7f368806c0a0 legacy=0x7f368806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.532+0000 7f3687400640 1 -- 172.21.15.67:0/3372614142 >> v1:172.21.15.67:6789/0 conn(0x7f3688145f80 legacy=0x7f36880cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.532+0000 7f3687400640 1 -- 172.21.15.67:0/3372614142 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3688072150 con 0x7f36880cd920 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.533+0000 7f3687400640 1 --2- 172.21.15.67:0/3372614142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368806bb60 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f36740097a0 tx=0x7f367402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e5ad78062ae29a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.533+0000 7f3686a00640 1 -- 172.21.15.67:0/3372614142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36740315a0 con 0x7f36880cd920 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.533+0000 7f3686a00640 1 -- 172.21.15.67:0/3372614142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3674037070 con 0x7f36880cd920 2024-04-17T08:51:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.533+0000 7f3686a00640 1 -- 172.21.15.67:0/3372614142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674031960 con 0x7f36880cd920 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 -- 172.21.15.67:0/3372614142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 msgr2=0x7f368806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 --2- 172.21.15.67:0/3372614142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368806bb60 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f36740097a0 tx=0x7f367402b840 comp rx=0 tx=0).stop 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 -- 172.21.15.67:0/3372614142 shutdown_connections 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 --2- 172.21.15.67:0/3372614142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368806bb60 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 -- 172.21.15.67:0/3372614142 >> 172.21.15.67:0/3372614142 conn(0x7f36880b2190 msgr2=0x7f36880b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 -- 172.21.15.67:0/3372614142 shutdown_connections 2024-04-17T08:51:15.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.534+0000 7f368da00640 1 -- 172.21.15.67:0/3372614142 wait complete. 2024-04-17T08:51:15.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.536+0000 7f368da00640 1 Processor -- start 2024-04-17T08:51:15.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.536+0000 7f368da00640 1 -- start start 2024-04-17T08:51:15.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 0x7f368815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3688145f80 0x7f3688136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3688072c30 con 0x7f36880cd920 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f368806c530 con 0x7f368806c0a0 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f368da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3688072db0 con 0x7f3688145f80 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f3687400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f3687400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37728/0 (socket says 172.21.15.67:37728) 2024-04-17T08:51:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.537+0000 7f3687400640 1 -- 172.21.15.67:0/2873701809 learned_addr learned my addr 172.21.15.67:0/2873701809 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687e00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 0x7f368815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 -- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3688145f80 msgr2=0x7f3688136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3688145f80 0x7f3688136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 -- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 msgr2=0x7f368815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 0x7f368815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 -- 172.21.15.67:0/2873701809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368815d3c0 con 0x7f36880cd920 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687e00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 0x7f368815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f3687400640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f3674034040 tx=0x7f36740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674032840 con 0x7f36880cd920 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3674032a00 con 0x7f36880cd920 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674032df0 con 0x7f36880cd920 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.538+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368815d6a0 con 0x7f36880cd920 2024-04-17T08:51:15.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.539+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f368815dc60 con 0x7f36880cd920 2024-04-17T08:51:15.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.540+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3688068a10 con 0x7f36880cd920 2024-04-17T08:51:15.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.541+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 138) ==== 91188+0+0 (secure 0 0 0) 0x7f367403b3f0 con 0x7f36880cd920 2024-04-17T08:51:15.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.542+0000 7f367fe00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 0x7f366806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.542+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f367402f030 con 0x7f36880cd920 2024-04-17T08:51:15.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.542+0000 7f3687e00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 0x7f366806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.543+0000 7f3687e00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 0x7f366806ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3678000f60 tx=0x7f3678009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.546+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3674040070 con 0x7f36880cd920 2024-04-17T08:51:15.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.741+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f36880b1d70 con 0x7f36880cd920 2024-04-17T08:51:15.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.742+0000 7f367fe00640 1 -- 172.21.15.67:0/2873701809 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v236) ==== 103+0+11 (secure 0 0 0) 0x7f367407bd40 con 0x7f36880cd920 2024-04-17T08:51:15.741 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 msgr2=0x7f366806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 0x7f366806ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3678000f60 tx=0x7f3678009040 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 msgr2=0x7f368813f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f3674034040 tx=0x7f36740046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 shutdown_connections 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f366806caa0 0x7f366806ef60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3688145f80 0x7f3688136bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36880cd920 0x7f368813f0a0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 --2- 172.21.15.67:0/2873701809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368806c0a0 0x7f368815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 >> 172.21.15.67:0/2873701809 conn(0x7f36880b2190 msgr2=0x7f36880b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:15.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 shutdown_connections 2024-04-17T08:51:15.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.746+0000 7f368da00640 1 -- 172.21.15.67:0/2873701809 wait complete. 2024-04-17T08:51:15.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:15.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.973+0000 7f7c0fe00640 1 Processor -- start 2024-04-17T08:51:15.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.973+0000 7f7c0fe00640 1 -- start start 2024-04-17T08:51:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c1015b570 0x7f7c1006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c1015bc80 con 0x7f7c1006fce0 2024-04-17T08:51:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7c1015bf40 con 0x7f7c1015b570 2024-04-17T08:51:15.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1015c220 con 0x7f7c1006c040 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.975+0000 7f7c0f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:56350/0 (socket says 172.21.15.67:56350) 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 -- 172.21.15.67:0/3877185081 learned_addr learned my addr 172.21.15.67:0/3877185081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0ea00640 1 --2- 172.21.15.67:0/3877185081 >> v2:172.21.15.187:3300/0 conn(0x7f7c1015b570 0x7f7c1006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 -- 172.21.15.67:0/3877185081 >> v2:172.21.15.187:3300/0 conn(0x7f7c1015b570 msgr2=0x7f7c1006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 --2- 172.21.15.67:0/3877185081 >> v2:172.21.15.187:3300/0 conn(0x7f7c1015b570 0x7f7c1006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 -- 172.21.15.67:0/3877185081 >> v1:172.21.15.187:6789/0 conn(0x7f7c1006fce0 legacy=0x7f7c1006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.976+0000 7f7c0f400640 1 -- 172.21.15.67:0/3877185081 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1015c500 con 0x7f7c1006c040 2024-04-17T08:51:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.978+0000 7f7c0f400640 1 --2- 172.21.15.67:0/3877185081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e50 tx=0x7f7c0802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b67af5f5ace76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.978+0000 7f7c0d600640 1 -- 172.21.15.67:0/3877185081 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0802ca80 con 0x7f7c1006c040 2024-04-17T08:51:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0d600640 1 -- 172.21.15.67:0/3877185081 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0802cc20 con 0x7f7c1006c040 2024-04-17T08:51:15.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0d600640 1 -- 172.21.15.67:0/3877185081 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c080343f0 con 0x7f7c1006c040 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 -- 172.21.15.67:0/3877185081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 msgr2=0x7f7c1015f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/3877185081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7c08009e50 tx=0x7f7c0802bee0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 -- 172.21.15.67:0/3877185081 shutdown_connections 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/3877185081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1006c040 0x7f7c1015f140 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/3877185081 >> v2:172.21.15.187:3300/0 conn(0x7f7c1015b570 0x7f7c1006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 -- 172.21.15.67:0/3877185081 >> 172.21.15.67:0/3877185081 conn(0x7f7c10067d10 msgr2=0x7f7c100b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 -- 172.21.15.67:0/3877185081 shutdown_connections 2024-04-17T08:51:15.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.979+0000 7f7c0fe00640 1 -- 172.21.15.67:0/3877185081 wait complete. 2024-04-17T08:51:15.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.981+0000 7f7c0fe00640 1 Processor -- start 2024-04-17T08:51:15.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.981+0000 7f7c0fe00640 1 -- start start 2024-04-17T08:51:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c1006c040 0x7f7c10147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1015b570 0x7f7c1015e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c100b4930 con 0x7f7c1006fce0 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7c100b47b0 con 0x7f7c1006c040 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.982+0000 7f7c0fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1015f680 con 0x7f7c1015b570 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37740/0 (socket says 172.21.15.67:37740) 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 -- 172.21.15.67:0/2778711616 learned_addr learned my addr 172.21.15.67:0/2778711616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0ea00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c1006c040 0x7f7c10147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0f400640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1015b570 0x7f7c1015e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 -- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1015b570 msgr2=0x7f7c1015e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1015b570 0x7f7c1015e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 -- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c1006c040 msgr2=0x7f7c10147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c1006c040 0x7f7c10147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.983+0000 7f7c0e000640 1 -- 172.21.15.67:0/2778711616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1015cff0 con 0x7f7c1006fce0 2024-04-17T08:51:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c0e000640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00c430 tx=0x7f7bfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0052d0 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bfc005490 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc007260 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1015d2d0 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.984+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1015d890 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.985+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c10068a10 con 0x7f7c1006fce0 2024-04-17T08:51:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.986+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 138) ==== 91188+0+0 (secure 0 0 0) 0x7f7bfc007400 con 0x7f7c1006fce0 2024-04-17T08:51:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.987+0000 7f7c06a00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 0x7f7bf006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.987+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f7bfc08ba90 con 0x7f7c1006fce0 2024-04-17T08:51:15.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.988+0000 7f7c0ea00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 0x7f7bf006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:15.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.989+0000 7f7c0ea00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 0x7f7bf006ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009a30 tx=0x7f7c00008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:15.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:15.992+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7bfc059580 con 0x7f7c1006fce0 2024-04-17T08:51:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.237+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c10158720 con 0x7f7c1006fce0 2024-04-17T08:51:16.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.239+0000 7f7c06a00640 1 -- 172.21.15.67:0/2778711616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7f7bfc058e80 con 0x7f7c1006fce0 2024-04-17T08:51:16.238 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:16.238 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.241+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 msgr2=0x7f7bf006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.241+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 0x7f7bf006ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009a30 tx=0x7f7c00008040 comp rx=0 tx=0).stop 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.241+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 msgr2=0x7f7c101512c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.241+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00c430 tx=0x7f7bfc00c8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 shutdown_connections 2024-04-17T08:51:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f7bf006caa0 0x7f7bf006ef60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c1015b570 0x7f7c1015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c1006fce0 0x7f7c101512c0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 --2- 172.21.15.67:0/2778711616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c1006c040 0x7f7c10147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 >> 172.21.15.67:0/2778711616 conn(0x7f7c10067d10 msgr2=0x7f7c10150020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 shutdown_connections 2024-04-17T08:51:16.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.242+0000 7f7c0fe00640 1 -- 172.21.15.67:0/2778711616 wait complete. 2024-04-17T08:51:16.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.480+0000 7f015f200640 1 Processor -- start 2024-04-17T08:51:16.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.480+0000 7f015f200640 1 -- start start 2024-04-17T08:51:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f015815ec60 0x7f01580b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 0x7f01580b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01580d2d30 con 0x7f01580b6b30 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f01580d49c0 con 0x7f015815ec60 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01580d3660 con 0x7f01580cfae0 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f015815ec60 0x7f01580b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f01580b6b30 0x7f01580cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52360/0 (socket says 172.21.15.67:52360) 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015d400640 1 -- 172.21.15.67:0/3464796111 learned_addr learned my addr 172.21.15.67:0/3464796111 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.482+0000 7f015de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f015815ec60 0x7f01580b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37752/0 (socket says 172.21.15.67:37752) 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015e800640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 0x7f01580b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015de00640 1 -- 172.21.15.67:0/3464796111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 msgr2=0x7f01580b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015de00640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 0x7f01580b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015de00640 1 -- 172.21.15.67:0/3464796111 >> v1:172.21.15.67:6789/0 conn(0x7f01580b6b30 legacy=0x7f01580cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015de00640 1 -- 172.21.15.67:0/3464796111 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01580d3010 con 0x7f015815ec60 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015ca00640 1 -- 172.21.15.67:0/3464796111 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236734701 0 0) 0x7f01580d2d30 con 0x7f01580b6b30 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.483+0000 7f015e800640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 0x7f01580b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.484+0000 7f015de00640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015815ec60 0x7f01580b65f0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f014c0097a0 tx=0x7f014c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5b43a910025fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.484+0000 7f015ca00640 1 -- 172.21.15.67:0/3464796111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c035020 con 0x7f015815ec60 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.484+0000 7f015ca00640 1 -- 172.21.15.67:0/3464796111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014c031830 con 0x7f015815ec60 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.484+0000 7f015ca00640 1 -- 172.21.15.67:0/3464796111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c031c00 con 0x7f015815ec60 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 -- 172.21.15.67:0/3464796111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015815ec60 msgr2=0x7f01580b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015815ec60 0x7f01580b65f0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f014c0097a0 tx=0x7f014c02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 -- 172.21.15.67:0/3464796111 shutdown_connections 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01580cfae0 0x7f01580b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 --2- 172.21.15.67:0/3464796111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015815ec60 0x7f01580b65f0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 -- 172.21.15.67:0/3464796111 >> 172.21.15.67:0/3464796111 conn(0x7f01580d15f0 msgr2=0x7f01580d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 -- 172.21.15.67:0/3464796111 shutdown_connections 2024-04-17T08:51:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.485+0000 7f015f200640 1 -- 172.21.15.67:0/3464796111 wait complete. 2024-04-17T08:51:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.487+0000 7f015f200640 1 Processor -- start 2024-04-17T08:51:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.487+0000 7f015f200640 1 -- start start 2024-04-17T08:51:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01580cfae0 0x7f0158161790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f015815ec60 0x7f01581f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f015815f040 con 0x7f01580b6b30 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f01580b6f70 con 0x7f01580cfae0 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01580d00c0 con 0x7f015815ec60 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37756/0 (socket says 172.21.15.67:37756) 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 -- 172.21.15.67:0/1032068976 learned_addr learned my addr 172.21.15.67:0/1032068976 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 -- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f015815ec60 msgr2=0x7f01581f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f015815ec60 0x7f01581f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015d400640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01580cfae0 0x7f0158161790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 -- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01580cfae0 msgr2=0x7f0158161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01580cfae0 0x7f0158161790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.488+0000 7f015de00640 1 -- 172.21.15.67:0/1032068976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01581f4830 con 0x7f01580b6b30 2024-04-17T08:51:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f015de00640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f014c000c00 tx=0x7f014c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c035050 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014c004230 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c03b800 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015815f830 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.489+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015815fdf0 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.490+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0158068a10 con 0x7f01580b6b30 2024-04-17T08:51:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.491+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 138) ==== 91188+0+0 (secure 0 0 0) 0x7f014c038020 con 0x7f01580b6b30 2024-04-17T08:51:16.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.492+0000 7f0152000640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 0x7f013806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.492+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f014c0ae460 con 0x7f01580b6b30 2024-04-17T08:51:16.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.493+0000 7f015d400640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 0x7f013806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.494+0000 7f015d400640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 0x7f013806ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01580b89a0 tx=0x7f0144009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.497+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f014c040070 con 0x7f01580b6b30 2024-04-17T08:51:16.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.694+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01581ea700 con 0x7f01580b6b30 2024-04-17T08:51:16.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.698+0000 7f0152000640 1 -- 172.21.15.67:0/1032068976 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) ==== 74+0+21671 (secure 0 0 0) 0x7f014c07be40 con 0x7f01580b6b30 2024-04-17T08:51:16.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:16.698 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":236,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:12.720937+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 msgr2=0x7f013806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 0x7f013806ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f01580b89a0 tx=0x7f0144009040 comp rx=0 tx=0).stop 2024-04-17T08:51:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 msgr2=0x7f015815f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f014c000c00 tx=0x7f014c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 shutdown_connections 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f013806caa0 0x7f013806ef60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f015815ec60 0x7f01581f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01580cfae0 0x7f0158161790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 --2- 172.21.15.67:0/1032068976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01580b6b30 0x7f015815f250 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 >> 172.21.15.67:0/1032068976 conn(0x7f01580d15f0 msgr2=0x7f01581e9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 shutdown_connections 2024-04-17T08:51:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.701+0000 7f015f200640 1 -- 172.21.15.67:0/1032068976 wait complete. 2024-04-17T08:51:16.720 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:51:16.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.942+0000 7f6246600640 1 Processor -- start 2024-04-17T08:51:16.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.942+0000 7f6246600640 1 -- start start 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.943+0000 7f6246600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f62400b6d00 0x7f62400cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.943+0000 7f6246600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62400d2d30 con 0x7f624015ec60 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.943+0000 7f6246600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62400d3010 con 0x7f62400cf910 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.943+0000 7f6246600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f62400d49c0 con 0x7f62400b6d00 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f623fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f62400b6d00 0x7f62400cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f6245200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f624015ec60 0x7f62400b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52372/0 (socket says 172.21.15.67:52372) 2024-04-17T08:51:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f6245200640 1 -- 172.21.15.67:0/3659722770 learned_addr learned my addr 172.21.15.67:0/3659722770 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f623fe00640 1 -- 172.21.15.67:0/3659722770 >> v1:172.21.15.187:6789/0 conn(0x7f62400cf910 legacy=0x7f62400b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f623fe00640 1 -- 172.21.15.67:0/3659722770 >> v1:172.21.15.67:6789/0 conn(0x7f624015ec60 legacy=0x7f62400b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.944+0000 7f623fe00640 1 -- 172.21.15.67:0/3659722770 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62400d32f0 con 0x7f62400b6d00 2024-04-17T08:51:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.945+0000 7f623fe00640 1 --2- 172.21.15.67:0/3659722770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400b6d00 0x7f62400cf3d0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3b3183b3739d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f623f400640 1 -- 172.21.15.67:0/3659722770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c039070 con 0x7f62400b6d00 2024-04-17T08:51:16.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f623f400640 1 -- 172.21.15.67:0/3659722770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622c02cc60 con 0x7f62400b6d00 2024-04-17T08:51:16.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f623f400640 1 -- 172.21.15.67:0/3659722770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c034660 con 0x7f62400b6d00 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 -- 172.21.15.67:0/3659722770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400b6d00 msgr2=0x7f62400cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 --2- 172.21.15.67:0/3659722770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400b6d00 0x7f62400cf3d0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f622c009d90 tx=0x7f622c02beb0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 -- 172.21.15.67:0/3659722770 shutdown_connections 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 --2- 172.21.15.67:0/3659722770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400b6d00 0x7f62400cf3d0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 -- 172.21.15.67:0/3659722770 >> 172.21.15.67:0/3659722770 conn(0x7f62400d15f0 msgr2=0x7f62400d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.946+0000 7f6246600640 1 -- 172.21.15.67:0/3659722770 shutdown_connections 2024-04-17T08:51:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.947+0000 7f6246600640 1 -- 172.21.15.67:0/3659722770 wait complete. 2024-04-17T08:51:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.948+0000 7f6246600640 1 Processor -- start 2024-04-17T08:51:16.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.948+0000 7f6246600640 1 -- start start 2024-04-17T08:51:16.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62400b6d00 0x7f6240151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f624015ec60 0x7f62400ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f624015f040 con 0x7f62400cf910 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f62400d3660 con 0x7f62400b6d00 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f6246600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f62400d00c0 con 0x7f624015ec60 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f623fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f623fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37766/0 (socket says 172.21.15.67:37766) 2024-04-17T08:51:16.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.949+0000 7f623fe00640 1 -- 172.21.15.67:0/3657348517 learned_addr learned my addr 172.21.15.67:0/3657348517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 -- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f624015ec60 msgr2=0x7f62400ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f6245200640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62400b6d00 0x7f6240151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f624015ec60 0x7f62400ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 -- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62400b6d00 msgr2=0x7f6240151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62400b6d00 0x7f6240151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 -- 172.21.15.67:0/3657348517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62400cc000 con 0x7f62400cf910 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623fe00640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f622c037040 tx=0x7f622c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.950+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c039050 con 0x7f62400cf910 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.951+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622c045070 con 0x7f62400cf910 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.951+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c03ea50 con 0x7f62400cf910 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.951+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62400cc2e0 con 0x7f62400cf910 2024-04-17T08:51:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.951+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62400cc8a0 con 0x7f62400cf910 2024-04-17T08:51:16.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.952+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6240068a10 con 0x7f62400cf910 2024-04-17T08:51:16.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.953+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f622c02cb20 con 0x7f62400cf910 2024-04-17T08:51:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.954+0000 7f623cc00640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 0x7f622006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.954+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f622c0af4f0 con 0x7f62400cf910 2024-04-17T08:51:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.955+0000 7f6245200640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 0x7f622006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:16.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.956+0000 7f6245200640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 0x7f622006eff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6230000f60 tx=0x7f6230009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:16.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:16.959+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f622c07c8b0 con 0x7f62400cf910 2024-04-17T08:51:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.204+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62400b3950 con 0x7f62400cf910 2024-04-17T08:51:17.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.205+0000 7f623cc00640 1 -- 172.21.15.67:0/3657348517 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7f622c0830d0 con 0x7f62400cf910 2024-04-17T08:51:17.205 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:17.205 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:17.205 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 msgr2=0x7f622006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 0x7f622006eff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6230000f60 tx=0x7f6230009040 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 msgr2=0x7f62400abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f622c037040 tx=0x7f622c004a40 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 shutdown_connections 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f622006cb30 0x7f622006eff0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f624015ec60 0x7f62400ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62400cf910 0x7f62400abe70 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 --2- 172.21.15.67:0/3657348517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62400b6d00 0x7f6240151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 >> 172.21.15.67:0/3657348517 conn(0x7f62400d15f0 msgr2=0x7f62400b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.208+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 shutdown_connections 2024-04-17T08:51:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.209+0000 7f6246600640 1 -- 172.21.15.67:0/3657348517 wait complete. 2024-04-17T08:51:17.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.440+0000 7f9082e00640 1 Processor -- start 2024-04-17T08:51:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.440+0000 7f9082e00640 1 -- start start 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.442+0000 7f9082e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c072420 0x7f907c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.442+0000 7f9082e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907c15f3c0 con 0x7f907c0c68c0 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.442+0000 7f9082e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907c15f660 con 0x7f907c15b530 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.442+0000 7f9082e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f907c15f940 con 0x7f907c072420 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.442+0000 7f9073e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c072420 0x7f907c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f9073e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f907c072420 0x7f907c06dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37768/0 (socket says 172.21.15.67:37768) 2024-04-17T08:51:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f9073e00640 1 -- 172.21.15.67:0/4220536551 learned_addr learned my addr 172.21.15.67:0/4220536551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f9073e00640 1 -- 172.21.15.67:0/4220536551 >> v1:172.21.15.187:6789/0 conn(0x7f907c15b530 legacy=0x7f907c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f9073e00640 1 -- 172.21.15.67:0/4220536551 >> v1:172.21.15.67:6789/0 conn(0x7f907c0c68c0 legacy=0x7f907c06e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f907b400640 1 -- 172.21.15.67:0/4220536551 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1080140041 0 0) 0x7f907c15f3c0 con 0x7f907c0c68c0 2024-04-17T08:51:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.443+0000 7f9073e00640 1 -- 172.21.15.67:0/4220536551 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c15fc20 con 0x7f907c072420 2024-04-17T08:51:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.445+0000 7f9073e00640 1 --2- 172.21.15.67:0/4220536551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c06dd50 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f906400bb10 tx=0x7f906402e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf93599c4775ce70 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.445+0000 7f907b400640 1 -- 172.21.15.67:0/4220536551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906403b070 con 0x7f907c072420 2024-04-17T08:51:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.445+0000 7f907b400640 1 -- 172.21.15.67:0/4220536551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9064036440 con 0x7f907c072420 2024-04-17T08:51:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.445+0000 7f907b400640 1 -- 172.21.15.67:0/4220536551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064036810 con 0x7f907c072420 2024-04-17T08:51:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 -- 172.21.15.67:0/4220536551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 msgr2=0x7f907c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 --2- 172.21.15.67:0/4220536551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c06dd50 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f906400bb10 tx=0x7f906402e200 comp rx=0 tx=0).stop 2024-04-17T08:51:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 -- 172.21.15.67:0/4220536551 shutdown_connections 2024-04-17T08:51:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 --2- 172.21.15.67:0/4220536551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c06dd50 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 -- 172.21.15.67:0/4220536551 >> 172.21.15.67:0/4220536551 conn(0x7f907c06bf50 msgr2=0x7f907c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 -- 172.21.15.67:0/4220536551 shutdown_connections 2024-04-17T08:51:17.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.446+0000 7f9082e00640 1 -- 172.21.15.67:0/4220536551 wait complete. 2024-04-17T08:51:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.448+0000 7f9082e00640 1 Processor -- start 2024-04-17T08:51:17.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.448+0000 7f9082e00640 1 -- start start 2024-04-17T08:51:17.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c0c68c0 0x7f907c13f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c15b530 0x7f907c13fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f907c160140 con 0x7f907c072420 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f907c15b910 con 0x7f907c15b530 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f9082e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f907c14b280 con 0x7f907c0c68c0 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f907be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.449+0000 7f907be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37770/0 (socket says 172.21.15.67:37770) 2024-04-17T08:51:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 -- 172.21.15.67:0/2550036600 learned_addr learned my addr 172.21.15.67:0/2550036600 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 -- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c0c68c0 msgr2=0x7f907c13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f9073e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c0c68c0 0x7f907c13f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c0c68c0 0x7f907c13f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 -- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c15b530 msgr2=0x7f907c13fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c15b530 0x7f907c13fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 -- 172.21.15.67:0/2550036600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c0c9550 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.450+0000 7f907be00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c400 tx=0x7f906c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.451+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c017070 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.451+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f906c0052d0 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.451+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c003830 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.451+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907c0c9830 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.451+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907c0cd300 con 0x7f907c072420 2024-04-17T08:51:17.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.452+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907c0c9a00 con 0x7f907c072420 2024-04-17T08:51:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.453+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f906c01a070 con 0x7f907c072420 2024-04-17T08:51:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.454+0000 7f9078c00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 0x7f905806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.454+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f906c08b9b0 con 0x7f907c072420 2024-04-17T08:51:17.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.455+0000 7f9073e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 0x7f905806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.456+0000 7f9073e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 0x7f905806eff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9064038000 tx=0x7f9064032000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.459+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f906c059370 con 0x7f907c072420 2024-04-17T08:51:17.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.698+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f907c068b90 con 0x7f907c072420 2024-04-17T08:51:17.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.699+0000 7f9078c00640 1 -- 172.21.15.67:0/2550036600 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+6564 (secure 0 0 0) 0x7f906c058c70 con 0x7f907c072420 2024-04-17T08:51:17.699 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:17.699 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":115,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:33.371129+0000","modified":"2024-04-17T08:50:36.401776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":5,"addr":"172.21.15.67:6835/3533295594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3533295594},{"type":"v1","addr":"172.21.15.67:6835","nonce":3533295594}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":13},{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:17.699 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 115 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 msgr2=0x7f905806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 0x7f905806eff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9064038000 tx=0x7f9064032000 comp rx=0 tx=0).stop 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 msgr2=0x7f907c14bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f906c00c400 tx=0x7f906c00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 shutdown_connections 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f905806cb30 0x7f905806eff0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f907c15b530 0x7f907c13fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f907c0c68c0 0x7f907c13f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 --2- 172.21.15.67:0/2550036600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f907c072420 0x7f907c14bca0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 >> 172.21.15.67:0/2550036600 conn(0x7f907c06bf50 msgr2=0x7f907c0d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:17.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 shutdown_connections 2024-04-17T08:51:17.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.702+0000 7f9082e00640 1 -- 172.21.15.67:0/2550036600 wait complete. 2024-04-17T08:51:17.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:17.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.927+0000 7f70ed600640 1 Processor -- start 2024-04-17T08:51:17.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.927+0000 7f70ed600640 1 -- start start 2024-04-17T08:51:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.929+0000 7f70ed600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f70e815ee60 0x7f70e80af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.929+0000 7f70ed600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.929+0000 7f70ed600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e815f440 con 0x7f70e815c5a0 2024-04-17T08:51:17.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.929+0000 7f70ed600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f70e80b4920 con 0x7f70e815ee60 2024-04-17T08:51:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.929+0000 7f70ed600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70e80afdd0 con 0x7f70e80af980 2024-04-17T08:51:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.930+0000 7f70ecc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.930+0000 7f70ecc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:56412/0 (socket says 172.21.15.67:56412) 2024-04-17T08:51:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.930+0000 7f70ecc00640 1 -- 172.21.15.67:0/472996843 learned_addr learned my addr 172.21.15.67:0/472996843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:17.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.930+0000 7f70e7400640 1 --2- 172.21.15.67:0/472996843 >> v2:172.21.15.187:3300/0 conn(0x7f70e815ee60 0x7f70e80af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.931+0000 7f70ecc00640 1 -- 172.21.15.67:0/472996843 >> v2:172.21.15.187:3300/0 conn(0x7f70e815ee60 msgr2=0x7f70e80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.931+0000 7f70ecc00640 1 --2- 172.21.15.67:0/472996843 >> v2:172.21.15.187:3300/0 conn(0x7f70e815ee60 0x7f70e80af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.931+0000 7f70ecc00640 1 -- 172.21.15.67:0/472996843 >> v1:172.21.15.187:6789/0 conn(0x7f70e815c5a0 legacy=0x7f70e80b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.931+0000 7f70ecc00640 1 -- 172.21.15.67:0/472996843 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e815f720 con 0x7f70e80af980 2024-04-17T08:51:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.932+0000 7f70ecc00640 1 --2- 172.21.15.67:0/472996843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009e50 tx=0x7f70e002bc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83dfed2035143e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.933+0000 7f70e6a00640 1 -- 172.21.15.67:0/472996843 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e002c7b0 con 0x7f70e80af980 2024-04-17T08:51:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.933+0000 7f70e6a00640 1 -- 172.21.15.67:0/472996843 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e002c950 con 0x7f70e80af980 2024-04-17T08:51:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.933+0000 7f70e6a00640 1 -- 172.21.15.67:0/472996843 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e002cd20 con 0x7f70e80af980 2024-04-17T08:51:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 -- 172.21.15.67:0/472996843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 msgr2=0x7f70e80ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 --2- 172.21.15.67:0/472996843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009e50 tx=0x7f70e002bc90 comp rx=0 tx=0).stop 2024-04-17T08:51:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 -- 172.21.15.67:0/472996843 shutdown_connections 2024-04-17T08:51:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 --2- 172.21.15.67:0/472996843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e80af980 0x7f70e80ae090 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 --2- 172.21.15.67:0/472996843 >> v2:172.21.15.187:3300/0 conn(0x7f70e815ee60 0x7f70e80af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 -- 172.21.15.67:0/472996843 >> 172.21.15.67:0/472996843 conn(0x7f70e8067d10 msgr2=0x7f70e80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 -- 172.21.15.67:0/472996843 shutdown_connections 2024-04-17T08:51:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.934+0000 7f70ed600640 1 -- 172.21.15.67:0/472996843 wait complete. 2024-04-17T08:51:17.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.936+0000 7f70ed600640 1 Processor -- start 2024-04-17T08:51:17.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.936+0000 7f70ed600640 1 -- start start 2024-04-17T08:51:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70e80af980 0x7f70e8142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e815ee60 0x7f70e815da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70e80adab0 con 0x7f70e815c5a0 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70e815fce0 con 0x7f70e80af980 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70ed600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70e80adc30 con 0x7f70e815ee60 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37782/0 (socket says 172.21.15.67:37782) 2024-04-17T08:51:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.937+0000 7f70e7400640 1 -- 172.21.15.67:0/277840058 learned_addr learned my addr 172.21.15.67:0/277840058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7e00640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70e80af980 0x7f70e8142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70ecc00640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e815ee60 0x7f70e815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 -- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e815ee60 msgr2=0x7f70e815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e815ee60 0x7f70e815da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 -- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70e80af980 msgr2=0x7f70e8142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70e80af980 0x7f70e8142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 -- 172.21.15.67:0/277840058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e81567e0 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70e7400640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f70d400e9b0 tx=0x7f70d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d4017070 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d400c9a0 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d400cd90 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.938+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e8156ac0 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.939+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e8152960 con 0x7f70e815c5a0 2024-04-17T08:51:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.940+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e8068a10 con 0x7f70e815c5a0 2024-04-17T08:51:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.941+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f70d401a070 con 0x7f70e815c5a0 2024-04-17T08:51:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.942+0000 7f70dfe00640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 0x7f70c806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.942+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7f70d408c660 con 0x7f70e815c5a0 2024-04-17T08:51:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.942+0000 7f70e7e00640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 0x7f70c806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.943+0000 7f70e7e00640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 0x7f70c806eff0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f70d8001860 tx=0x7f70d8009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:17.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:17.946+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f70d405a080 con 0x7f70e815c5a0 2024-04-17T08:51:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.152+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70e80ae6d0 con 0x7f70e815c5a0 2024-04-17T08:51:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.154+0000 7f70dfe00640 1 -- 172.21.15.67:0/277840058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) ==== 74+0+21671 (secure 0 0 0) 0x7f70d4059980 con 0x7f70e815c5a0 2024-04-17T08:51:18.154 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:18.154 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":236,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:12.720937+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:50:31.281467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-04-17T08:50:32.343386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"230","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 msgr2=0x7f70c806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 0x7f70c806eff0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f70d8001860 tx=0x7f70d8009040 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 msgr2=0x7f70e8155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f70d400e9b0 tx=0x7f70d400ee70 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 shutdown_connections 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f70c806cb30 0x7f70c806eff0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70e815ee60 0x7f70e815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70e815c5a0 0x7f70e8155ee0 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 --2- 172.21.15.67:0/277840058 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70e80af980 0x7f70e8142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 >> 172.21.15.67:0/277840058 conn(0x7f70e8067d10 msgr2=0x7f70e815b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 shutdown_connections 2024-04-17T08:51:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.157+0000 7f70ed600640 1 -- 172.21.15.67:0/277840058 wait complete. 2024-04-17T08:51:18.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:51:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.393+0000 7efd87200640 1 Processor -- start 2024-04-17T08:51:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.394+0000 7efd87200640 1 -- start start 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efd880cbc00 0x7efd880cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd880b63c0 0x7efd88071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd88160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efd880cfe90 con 0x7efd880b63c0 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efd8815dc70 con 0x7efd880cbc00 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.395+0000 7efd87200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd8815d060 con 0x7efd880723a0 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd880b63c0 0x7efd88071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd880b63c0 0x7efd88071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37798/0 (socket says 172.21.15.67:37798) 2024-04-17T08:51:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 -- 172.21.15.67:0/2329590242 learned_addr learned my addr 172.21.15.67:0/2329590242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd85e00640 1 --2- 172.21.15.67:0/2329590242 >> v2:172.21.15.187:3300/0 conn(0x7efd880cbc00 0x7efd880cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd86800640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd88160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 -- 172.21.15.67:0/2329590242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 msgr2=0x7efd88160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd88160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 -- 172.21.15.67:0/2329590242 >> v2:172.21.15.187:3300/0 conn(0x7efd880cbc00 msgr2=0x7efd880cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 --2- 172.21.15.67:0/2329590242 >> v2:172.21.15.187:3300/0 conn(0x7efd880cbc00 0x7efd880cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd7d400640 1 -- 172.21.15.67:0/2329590242 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8815d340 con 0x7efd880b63c0 2024-04-17T08:51:18.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.396+0000 7efd86800640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd88160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:18.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.397+0000 7efd7d400640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88071e60 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7efd880702d0 tx=0x7efd7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c88c0a01a1be9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd85400640 1 -- 172.21.15.67:0/2329590242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70035020 con 0x7efd880b63c0 2024-04-17T08:51:18.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd85400640 1 -- 172.21.15.67:0/2329590242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd70031870 con 0x7efd880b63c0 2024-04-17T08:51:18.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd85400640 1 -- 172.21.15.67:0/2329590242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70031c40 con 0x7efd880b63c0 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd87200640 1 -- 172.21.15.67:0/2329590242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 msgr2=0x7efd88071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd87200640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88071e60 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7efd880702d0 tx=0x7efd7002b910 comp rx=0 tx=0).stop 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.398+0000 7efd87200640 1 -- 172.21.15.67:0/2329590242 shutdown_connections 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd88160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 --2- 172.21.15.67:0/2329590242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88071e60 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 --2- 172.21.15.67:0/2329590242 >> v2:172.21.15.187:3300/0 conn(0x7efd880cbc00 0x7efd880cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 -- 172.21.15.67:0/2329590242 >> 172.21.15.67:0/2329590242 conn(0x7efd8806bef0 msgr2=0x7efd880b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:18.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 -- 172.21.15.67:0/2329590242 shutdown_connections 2024-04-17T08:51:18.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.399+0000 7efd87200640 1 -- 172.21.15.67:0/2329590242 wait complete. 2024-04-17T08:51:18.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.400+0000 7efd87200640 1 Processor -- start 2024-04-17T08:51:18.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.401+0000 7efd87200640 1 -- start start 2024-04-17T08:51:18.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.401+0000 7efd87200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd8815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.401+0000 7efd87200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd87200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd880cbc00 0x7efd8806f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd87200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efd880d02d0 con 0x7efd880b63c0 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd87200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efd8815d900 con 0x7efd880cbc00 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd87200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd880cbfe0 con 0x7efd880723a0 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37800/0 (socket says 172.21.15.67:37800) 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 -- 172.21.15.67:0/213842508 learned_addr learned my addr 172.21.15.67:0/213842508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd85e00640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd8815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 -- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 msgr2=0x7efd8815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd8815a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 -- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd880cbc00 msgr2=0x7efd8806f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd880cbc00 0x7efd8806f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.402+0000 7efd7d400640 1 -- 172.21.15.67:0/213842508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd880b5010 con 0x7efd880b63c0 2024-04-17T08:51:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd7d400640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7efd70009960 tx=0x7efd700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70035040 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd70030d00 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd880b52f0 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd700456c0 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.403+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd880b58b0 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.404+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd88068a10 con 0x7efd880b63c0 2024-04-17T08:51:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.405+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7efd70038020 con 0x7efd880b63c0 2024-04-17T08:51:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.406+0000 7efd7ea00640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 0x7efd6406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.406+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 7418+0+0 (secure 0 0 0) 0x7efd700ae840 con 0x7efd880b63c0 2024-04-17T08:51:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.407+0000 7efd85e00640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 0x7efd6406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.408+0000 7efd85e00640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 0x7efd6406f040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd78006fd0 tx=0x7efd78008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.411+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efd7007c2d0 con 0x7efd880b63c0 2024-04-17T08:51:18.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.654+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efd880b3020 con 0x7efd880b63c0 2024-04-17T08:51:18.654 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:51:18.655+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:51:18.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.658+0000 7efd7ea00640 1 -- 172.21.15.67:0/213842508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v116) ==== 161+0+0 (secure 0 0 0) 0x7efd7007bbd0 con 0x7efd880b63c0 2024-04-17T08:51:18.658 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:51:18.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 msgr2=0x7efd6406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 0x7efd6406f040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efd78006fd0 tx=0x7efd78008040 comp rx=0 tx=0).stop 2024-04-17T08:51:18.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 msgr2=0x7efd88146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7efd70009960 tx=0x7efd700046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 shutdown_connections 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7efd6406cb80 0x7efd6406f040 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd880cbc00 0x7efd8806f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd880b63c0 0x7efd88146ed0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 --2- 172.21.15.67:0/213842508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd880723a0 0x7efd8815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 >> 172.21.15.67:0/213842508 conn(0x7efd8806bef0 msgr2=0x7efd881592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.662+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 shutdown_connections 2024-04-17T08:51:18.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.663+0000 7efd87200640 1 -- 172.21.15.67:0/213842508 wait complete. 2024-04-17T08:51:18.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:51:18.681 INFO:tasks.ceph.mds.a.smithi067.stderr: -5205> 2024-04-17T08:50:37.279+0000 7fb05a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:51:18.743 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:51:18.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.878+0000 7fd77f200640 1 Processor -- start 2024-04-17T08:51:18.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.879+0000 7fd77f200640 1 -- start start 2024-04-17T08:51:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.880+0000 7fd77f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.880+0000 7fd77f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd7800cf8e0 0x7fd7800b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.880+0000 7fd77f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7800d00b0 con 0x7fd78015ecb0 2024-04-17T08:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.880+0000 7fd77f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd7800d0c30 con 0x7fd7800cf8e0 2024-04-17T08:51:18.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.880+0000 7fd77f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd7800d0390 con 0x7fd7800d33e0 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.881+0000 7fd777e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.881+0000 7fd777e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:56460/0 (socket says 172.21.15.67:56460) 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.881+0000 7fd777e00640 1 -- 172.21.15.67:0/1570285940 learned_addr learned my addr 172.21.15.67:0/1570285940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.881+0000 7fd77e800640 1 --2- 172.21.15.67:0/1570285940 >> v2:172.21.15.187:3300/0 conn(0x7fd7800cf8e0 0x7fd7800b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.882+0000 7fd777e00640 1 -- 172.21.15.67:0/1570285940 >> v2:172.21.15.187:3300/0 conn(0x7fd7800cf8e0 msgr2=0x7fd7800b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.882+0000 7fd777e00640 1 --2- 172.21.15.67:0/1570285940 >> v2:172.21.15.187:3300/0 conn(0x7fd7800cf8e0 0x7fd7800b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.882+0000 7fd777e00640 1 -- 172.21.15.67:0/1570285940 >> v1:172.21.15.187:6789/0 conn(0x7fd78015ecb0 legacy=0x7fd7800d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.882+0000 7fd777e00640 1 -- 172.21.15.67:0/1570285940 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7800d0670 con 0x7fd7800d33e0 2024-04-17T08:51:18.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.882+0000 7fd77d400640 1 -- 172.21.15.67:0/1570285940 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329192057 0 0) 0x7fd7800d00b0 con 0x7fd78015ecb0 2024-04-17T08:51:18.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.883+0000 7fd777e00640 1 --2- 172.21.15.67:0/1570285940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd76802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ec5099af903c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.883+0000 7fd77d400640 1 -- 172.21.15.67:0/1570285940 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768035020 con 0x7fd7800d33e0 2024-04-17T08:51:18.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.883+0000 7fd77d400640 1 -- 172.21.15.67:0/1570285940 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd768031640 con 0x7fd7800d33e0 2024-04-17T08:51:18.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.883+0000 7fd77d400640 1 -- 172.21.15.67:0/1570285940 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76802d040 con 0x7fd7800d33e0 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 -- 172.21.15.67:0/1570285940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 msgr2=0x7fd7800cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 --2- 172.21.15.67:0/1570285940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd768009870 tx=0x7fd76802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 -- 172.21.15.67:0/1570285940 shutdown_connections 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 --2- 172.21.15.67:0/1570285940 >> v2:172.21.15.187:3300/0 conn(0x7fd7800cf8e0 0x7fd7800b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 --2- 172.21.15.67:0/1570285940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7800cf3a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 -- 172.21.15.67:0/1570285940 >> 172.21.15.67:0/1570285940 conn(0x7fd78008bff0 msgr2=0x7fd7800d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 -- 172.21.15.67:0/1570285940 shutdown_connections 2024-04-17T08:51:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.884+0000 7fd77f200640 1 -- 172.21.15.67:0/1570285940 wait complete. 2024-04-17T08:51:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.886+0000 7fd77f200640 1 Processor -- start 2024-04-17T08:51:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.886+0000 7fd77f200640 1 -- start start 2024-04-17T08:51:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7801f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78015ecb0 0x7fd7801f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd78015f090 con 0x7fd7800cf8e0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd7800cfd30 con 0x7fd78015ecb0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd78015f210 con 0x7fd7800d33e0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.887+0000 7fd77de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37822/0 (socket says 172.21.15.67:37822) 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 -- 172.21.15.67:0/2262494510 learned_addr learned my addr 172.21.15.67:0/2262494510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77e800640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78015ecb0 0x7fd7801f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd777e00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7801f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 -- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 msgr2=0x7fd7801f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7801f4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 -- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78015ecb0 msgr2=0x7fd7801f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78015ecb0 0x7fd7801f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 -- 172.21.15.67:0/2262494510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7800b3ec0 con 0x7fd7800cf8e0 2024-04-17T08:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd777e00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7801f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.888+0000 7fd77de00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fd77000c3f0 tx=0x7fd77000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.889+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd770017070 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.889+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7700035f0 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.889+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7801f5490 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.889+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7700039e0 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.889+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7800b4460 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.890+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd780068a10 con 0x7fd7800cf8e0 2024-04-17T08:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.891+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fd77001a070 con 0x7fd7800cf8e0 2024-04-17T08:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.892+0000 7fd776000640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 0x7fd75c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.893+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 7504+0+0 (secure 0 0 0) 0x7fd77008bac0 con 0x7fd7800cf8e0 2024-04-17T08:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.893+0000 7fd777e00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 0x7fd75c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.894+0000 7fd777e00640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 0x7fd75c06eff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd768036000 tx=0x7fd768008030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:18.897+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd770059410 con 0x7fd7800cf8e0 2024-04-17T08:51:19.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.142+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd7800c7fc0 con 0x7fd7800cf8e0 2024-04-17T08:51:19.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.662+0000 7fd776000640 1 -- 172.21.15.67:0/2262494510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v117) ==== 104+0+0 (secure 0 0 0) 0x7fd770058d10 con 0x7fd7800cf8e0 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.666+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 msgr2=0x7fd75c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.666+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 0x7fd75c06eff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd768036000 tx=0x7fd768008030 comp rx=0 tx=0).stop 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.666+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 msgr2=0x7fd7801f8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.666+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fd77000c3f0 tx=0x7fd77000c8b0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 shutdown_connections 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd75c06cb30 0x7fd75c06eff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd78015ecb0 0x7fd7801f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7800d33e0 0x7fd7801f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 --2- 172.21.15.67:0/2262494510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7800cf8e0 0x7fd7801f8a90 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 >> 172.21.15.67:0/2262494510 conn(0x7fd78008bff0 msgr2=0x7fd78015ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:19.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 shutdown_connections 2024-04-17T08:51:19.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.667+0000 7fd77f200640 1 -- 172.21.15.67:0/2262494510 wait complete. 2024-04-17T08:51:19.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:51:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.921+0000 7f7345600640 1 Processor -- start 2024-04-17T08:51:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.922+0000 7f7345600640 1 -- start start 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.923+0000 7f7345600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7340162110 0x7f73400ba2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.923+0000 7f7345600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73400b3920 con 0x7f73400cfd30 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.923+0000 7f7345600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73400d2610 con 0x7f73400d55e0 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.923+0000 7f7345600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f73400d2940 con 0x7f7340162110 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7340162110 0x7f73400ba2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7340162110 0x7f73400ba2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37838/0 (socket says 172.21.15.67:37838) 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 -- 172.21.15.67:0/3534926220 learned_addr learned my addr 172.21.15.67:0/3534926220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733f400640 1 --1- 172.21.15.67:0/3534926220 >> v1:172.21.15.67:6789/0 conn(0x7f73400cfd30 0x7f73400d2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52408/0 (socket says 172.21.15.67:52408) 2024-04-17T08:51:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 -- 172.21.15.67:0/3534926220 >> v1:172.21.15.187:6789/0 conn(0x7f73400d55e0 legacy=0x7f73400cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 -- 172.21.15.67:0/3534926220 >> v1:172.21.15.67:6789/0 conn(0x7f73400cfd30 legacy=0x7f73400d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.924+0000 7f733ea00640 1 -- 172.21.15.67:0/3534926220 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73400b3bc0 con 0x7f7340162110 2024-04-17T08:51:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.925+0000 7f733ea00640 1 --2- 172.21.15.67:0/3534926220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7340162110 0x7f73400ba2c0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f7330009870 tx=0x7f733002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bb020a481bb26a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f733d600640 1 -- 172.21.15.67:0/3534926220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330035020 con 0x7f7340162110 2024-04-17T08:51:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f733d600640 1 -- 172.21.15.67:0/3534926220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733002d040 con 0x7f7340162110 2024-04-17T08:51:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f733d600640 1 -- 172.21.15.67:0/3534926220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330031a80 con 0x7f7340162110 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 -- 172.21.15.67:0/3534926220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7340162110 msgr2=0x7f73400ba2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 --2- 172.21.15.67:0/3534926220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7340162110 0x7f73400ba2c0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f7330009870 tx=0x7f733002b900 comp rx=0 tx=0).stop 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 -- 172.21.15.67:0/3534926220 shutdown_connections 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 --2- 172.21.15.67:0/3534926220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7340162110 0x7f73400ba2c0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 -- 172.21.15.67:0/3534926220 >> 172.21.15.67:0/3534926220 conn(0x7f73400d1a50 msgr2=0x7f73400d49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.926+0000 7f7345600640 1 -- 172.21.15.67:0/3534926220 shutdown_connections 2024-04-17T08:51:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.927+0000 7f7345600640 1 -- 172.21.15.67:0/3534926220 wait complete. 2024-04-17T08:51:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.928+0000 7f7345600640 1 Processor -- start 2024-04-17T08:51:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.928+0000 7f7345600640 1 -- start start 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.929+0000 7f7345600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.929+0000 7f7345600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73400d55e0 0x7f73401f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.929+0000 7f7345600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7340162110 0x7f7340146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f7345600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f73400ba980 con 0x7f73400cfd30 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f7345600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f73400ba800 con 0x7f73400d55e0 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f7345600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73401624f0 con 0x7f7340162110 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37850/0 (socket says 172.21.15.67:37850) 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 -- 172.21.15.67:0/2633889995 learned_addr learned my addr 172.21.15.67:0/2633889995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 -- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7340162110 msgr2=0x7f7340146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7340162110 0x7f7340146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 -- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73400d55e0 msgr2=0x7f73401f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73400d55e0 0x7f73401f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.930+0000 7f733ea00640 1 -- 172.21.15.67:0/2633889995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73401478b0 con 0x7f73400cfd30 2024-04-17T08:51:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f733ea00640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f7330009870 tx=0x7f73300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733003ac60 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733003ae00 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330044630 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73401472f0 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.931+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7340155a40 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.932+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7340155830 con 0x7f73400cfd30 2024-04-17T08:51:19.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.933+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f733004b060 con 0x7f73400cfd30 2024-04-17T08:51:19.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.934+0000 7f7336a00640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 0x7f732006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:19.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.934+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 7504+0+0 (secure 0 0 0) 0x7f73300ae7e0 con 0x7f73400cfd30 2024-04-17T08:51:19.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.935+0000 7f733e000640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 0x7f732006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:19.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.936+0000 7f733e000640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 0x7f732006eff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73400c89a0 tx=0x7f732c0079b0 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:19.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:19.940+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f733003f070 con 0x7f73400cfd30 2024-04-17T08:51:20.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.147+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f73400cb730 con 0x7f73400cfd30 2024-04-17T08:51:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.800+0000 7f7336a00640 1 -- 172.21.15.67:0/2633889995 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v238) ==== 181+0+0 (secure 0 0 0) 0x7f733007c130 con 0x7f73400cfd30 2024-04-17T08:51:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:51:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 msgr2=0x7f732006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 0x7f732006eff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73400c89a0 tx=0x7f732c0079b0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 msgr2=0x7f73400cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f7330009870 tx=0x7f73300046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 shutdown_connections 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f732006cb30 0x7f732006eff0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7340162110 0x7f7340146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73400d55e0 0x7f73401f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 --2- 172.21.15.67:0/2633889995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73400cfd30 0x7f73400cc1e0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.803+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 >> 172.21.15.67:0/2633889995 conn(0x7f73400d1a50 msgr2=0x7f73400b7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.804+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 shutdown_connections 2024-04-17T08:51:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:20.804+0000 7f7345600640 1 -- 172.21.15.67:0/2633889995 wait complete. 2024-04-17T08:51:20.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:51:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.070+0000 7fc012200640 1 Processor -- start 2024-04-17T08:51:21.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.070+0000 7fc012200640 1 -- start start 2024-04-17T08:51:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc012200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc00c072930 0x7fc00c0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc012200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c06de70 con 0x7fc00c0b3e30 2024-04-17T08:51:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc012200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c15f850 con 0x7fc00c15c670 2024-04-17T08:51:21.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc012200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc00c15fb30 con 0x7fc00c072930 2024-04-17T08:51:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc011800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc00c0b3e30 0x7fc00c06e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52412/0 (socket says 172.21.15.67:52412) 2024-04-17T08:51:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.072+0000 7fc011800640 1 -- 172.21.15.67:0/2421640965 learned_addr learned my addr 172.21.15.67:0/2421640965 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.073+0000 7fc00be00640 1 --2- 172.21.15.67:0/2421640965 >> v2:172.21.15.187:3300/0 conn(0x7fc00c072930 0x7fc00c0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.073+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869363462 0 0) 0x7fc00c06de70 con 0x7fc00c0b3e30 2024-04-17T08:51:21.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.073+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff4003520 con 0x7fc00c0b3e30 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1421553122 0 0) 0x7fbff4003520 con 0x7fc00c0b3e30 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 >> v2:172.21.15.187:3300/0 conn(0x7fc00c072930 msgr2=0x7fc00c0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 --2- 172.21.15.67:0/2421640965 >> v2:172.21.15.187:3300/0 conn(0x7fc00c072930 0x7fc00c0cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 >> v1:172.21.15.187:6789/0 conn(0x7fc00c15c670 legacy=0x7fc00c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c15fe10 con 0x7fc00c0b3e30 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc004001a10 con 0x7fc00c0b3e30 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 >> v1:172.21.15.67:6789/0 conn(0x7fc00c0b3e30 legacy=0x7fc00c06e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 0x7fbff4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c15fb30 con 0x7fbff4003a00 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbff400c0c0 con 0x7fbff4004800 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbff400c2c0 con 0x7fbff40051c0 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.074+0000 7fc010e00640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 0x7fbff4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc00be00640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc011800640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc010e00640 1 -- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff40051c0 msgr2=0x7fbff400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc010e00640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff40051c0 0x7fbff400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc010e00640 1 -- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbff4004800 msgr2=0x7fbff4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc010e00640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbff4004800 0x7fbff4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc010e00640 1 -- 172.21.15.67:0/2421640965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff4003520 con 0x7fbff4003a00 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.075+0000 7fc011800640 1 -- 172.21.15.67:0/2421640965 reap_dead start 2024-04-17T08:51:21.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.076+0000 7fc010e00640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 0x7fbff4003e20 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc000001750 tx=0x7fbff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbacd1d3af56f0fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:21.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.077+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000001370 con 0x7fbff4003a00 2024-04-17T08:51:21.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.077+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0000052d0 con 0x7fbff4003a00 2024-04-17T08:51:21.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.077+0000 7fc00b400640 1 -- 172.21.15.67:0/2421640965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc000003830 con 0x7fbff4003a00 2024-04-17T08:51:21.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.077+0000 7fc012200640 1 -- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 msgr2=0x7fbff4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 0x7fbff4003e20 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc000001750 tx=0x7fbff4005830 comp rx=0 tx=0).stop 2024-04-17T08:51:21.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 -- 172.21.15.67:0/2421640965 shutdown_connections 2024-04-17T08:51:21.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 --2- 172.21.15.67:0/2421640965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbff4003a00 0x7fbff4003e20 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 -- 172.21.15.67:0/2421640965 >> 172.21.15.67:0/2421640965 conn(0x7fc00c06b8c0 msgr2=0x7fc00c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:21.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 -- 172.21.15.67:0/2421640965 shutdown_connections 2024-04-17T08:51:21.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.078+0000 7fc012200640 1 -- 172.21.15.67:0/2421640965 wait complete. 2024-04-17T08:51:21.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.080+0000 7fc012200640 1 Processor -- start 2024-04-17T08:51:21.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.080+0000 7fc012200640 1 -- start start 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 0x7fc00c0af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc00c14a220 0x7fc00c0b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c15f110 con 0x7fc00c0b05b0 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c15ef90 con 0x7fc00c14a220 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc012200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc00c15f290 con 0x7fbff4003a00 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37864/0 (socket says 172.21.15.67:37864) 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 -- 172.21.15.67:0/706281310 learned_addr learned my addr 172.21.15.67:0/706281310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc010e00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 0x7fc00c0af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc00be00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc00c14a220 0x7fc00c0b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 -- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 msgr2=0x7fc00c0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 0x7fc00c0af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 -- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc00c14a220 msgr2=0x7fc00c0b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc00c14a220 0x7fc00c0b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.082+0000 7fc011800640 1 -- 172.21.15.67:0/706281310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c0afac0 con 0x7fc00c0b05b0 2024-04-17T08:51:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc010e00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 0x7fc00c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc011800640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fc00c073240 tx=0x7fc004007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc004017020 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00400ab50 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c0ae4e0 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0040133f0 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.083+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c0aeaa0 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.085+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c068a10 con 0x7fc00c0b05b0 2024-04-17T08:51:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.086+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fc00401a020 con 0x7fc00c0b05b0 2024-04-17T08:51:21.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.086+0000 7fc008c00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 0x7fbfec06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:21.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.087+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7066+0+0 (secure 0 0 0) 0x7fc00408cc20 con 0x7fc00c0b05b0 2024-04-17T08:51:21.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.087+0000 7fc010e00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 0x7fbfec06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:21.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.089+0000 7fc010e00640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 0x7fbfec06eff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc000033000 tx=0x7fc000009030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.092+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc004021020 con 0x7fc00c0b05b0 2024-04-17T08:51:21.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.309+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc00c068610 con 0x7fc00c0b05b0 2024-04-17T08:51:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.803+0000 7fc008c00640 1 -- 172.21.15.67:0/706281310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v239) ==== 169+0+0 (secure 0 0 0) 0x7fc00405a830 con 0x7fc00c0b05b0 2024-04-17T08:51:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 msgr2=0x7fbfec06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 0x7fbfec06eff0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc000033000 tx=0x7fc000009030 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 msgr2=0x7fc00c0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fc00c073240 tx=0x7fc004007bd0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 shutdown_connections 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbfec06cb30 0x7fbfec06eff0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc00c0b05b0 0x7fc00c0ad8d0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc00c14a220 0x7fc00c0b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 --2- 172.21.15.67:0/706281310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbff4003a00 0x7fc00c0af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.807+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 >> 172.21.15.67:0/706281310 conn(0x7fc00c06b8c0 msgr2=0x7fc00c15d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.808+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 shutdown_connections 2024-04-17T08:51:21.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:21.808+0000 7fc012200640 1 -- 172.21.15.67:0/706281310 wait complete. 2024-04-17T08:51:21.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:22.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.055+0000 7f0649a00640 1 Processor -- start 2024-04-17T08:51:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.055+0000 7f0649a00640 1 -- start start 2024-04-17T08:51:22.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0649a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06440cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0649a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f064406e020 0x7f064406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0649a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06440ceb00 con 0x7f064406f670 2024-04-17T08:51:22.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0649a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f06440cede0 con 0x7f064406e020 2024-04-17T08:51:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0649a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06440cf0c0 con 0x7f06440cd020 2024-04-17T08:51:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0642a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f064406f670 0x7f064406fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52420/0 (socket says 172.21.15.67:52420) 2024-04-17T08:51:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.057+0000 7f0642a00640 1 -- 172.21.15.67:0/1204665989 learned_addr learned my addr 172.21.15.67:0/1204665989 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0642000640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06440cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0643400640 1 --2- 172.21.15.67:0/1204665989 >> v2:172.21.15.187:3300/0 conn(0x7f064406e020 0x7f064406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731903007 0 0) 0x7f06440ceb00 con 0x7f064406f670 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0628003520 con 0x7f064406f670 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 425397488 0 0) 0x7f0628003520 con 0x7f064406f670 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 msgr2=0x7f06440cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06440cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 >> v2:172.21.15.187:3300/0 conn(0x7f064406e020 msgr2=0x7f064406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 --2- 172.21.15.67:0/1204665989 >> v2:172.21.15.187:3300/0 conn(0x7f064406e020 0x7f064406e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06440cf3a0 con 0x7f064406f670 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0634001670 con 0x7f064406f670 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 >> v1:172.21.15.67:6789/0 conn(0x7f064406f670 legacy=0x7f064406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f0628004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628004d40 0x7f06280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06440cede0 con 0x7f0628003f80 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06440cf0c0 con 0x7f0628004d40 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.058+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f062800b9c0 con 0x7f06440cd020 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f0628004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642a00640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 -- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 msgr2=0x7f06280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06440cd020 0x7f06280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 -- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628004d40 msgr2=0x7f06280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628004d40 0x7f06280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.059+0000 7f0642000640 1 -- 172.21.15.67:0/1204665989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0628003520 con 0x7f0628003f80 2024-04-17T08:51:22.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.060+0000 7f0642000640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f0628004800 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f0630009af0 tx=0x7f0628005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff20caa0072e8bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630039070 con 0x7f0628003f80 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063002c6b0 con 0x7f0628003f80 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0641600640 1 -- 172.21.15.67:0/1204665989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630009690 con 0x7f0628003f80 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 -- 172.21.15.67:0/1204665989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 msgr2=0x7f0628004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 --2- 172.21.15.67:0/1204665989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f0628004800 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f0630009af0 tx=0x7f0628005830 comp rx=0 tx=0).stop 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0643400640 1 -- 172.21.15.67:0/1204665989 reap_dead start 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 -- 172.21.15.67:0/1204665989 shutdown_connections 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 -- 172.21.15.67:0/1204665989 >> 172.21.15.67:0/1204665989 conn(0x7f064406bf50 msgr2=0x7f06440b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 -- 172.21.15.67:0/1204665989 shutdown_connections 2024-04-17T08:51:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.061+0000 7f0649a00640 1 -- 172.21.15.67:0/1204665989 wait complete. 2024-04-17T08:51:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.063+0000 7f0649a00640 1 Processor -- start 2024-04-17T08:51:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.063+0000 7f0649a00640 1 -- start start 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 0x7f06440b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0644155e40 0x7f064414db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06440ce8a0 con 0x7f0628003f80 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06440ce720 con 0x7f06441559c0 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0649a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f064415cb00 con 0x7f0644155e40 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.064+0000 7f0642a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37872/0 (socket says 172.21.15.67:37872) 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 -- 172.21.15.67:0/4150815676 learned_addr learned my addr 172.21.15.67:0/4150815676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642000640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 0x7f06440b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 -- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0644155e40 msgr2=0x7f064414db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0644155e40 0x7f064414db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 -- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 msgr2=0x7f06440b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 0x7f06440b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 -- 172.21.15.67:0/4150815676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06440b3f20 con 0x7f0628003f80 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642000640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 0x7f06440b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f0642a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f0634001050 tx=0x7f063400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.065+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063400c640 con 0x7f0628003f80 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.066+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0634018070 con 0x7f0628003f80 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.066+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063400ca40 con 0x7f0628003f80 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.066+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06440b3c40 con 0x7f0628003f80 2024-04-17T08:51:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.066+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06440c89e0 con 0x7f0628003f80 2024-04-17T08:51:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.067+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0644068b90 con 0x7f0628003f80 2024-04-17T08:51:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.068+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f063400cbe0 con 0x7f0628003f80 2024-04-17T08:51:22.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.069+0000 7f063aa00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 0x7f062406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.069+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f063408c000 con 0x7f0628003f80 2024-04-17T08:51:22.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.070+0000 7f0642000640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 0x7f062406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.071+0000 7f0642000640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 0x7f062406eff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0630036000 tx=0x7f0630030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.073+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0634059e80 con 0x7f0628003f80 2024-04-17T08:51:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.299+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f064406e880 con 0x7f0628003f80 2024-04-17T08:51:22.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.302+0000 7f063aa00640 1 -- 172.21.15.67:0/4150815676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) ==== 74+0+18019 (secure 0 0 0) 0x7f0634059780 con 0x7f0628003f80 2024-04-17T08:51:22.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:22.302 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":239,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:21.800765+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.67:6835/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.67:6834/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:22.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 msgr2=0x7f062406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 0x7f062406eff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0630036000 tx=0x7f0630030000 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 msgr2=0x7f064415aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f0634001050 tx=0x7f063400eb90 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 shutdown_connections 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f062406cb30 0x7f062406eff0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0644155e40 0x7f064414db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06441559c0 0x7f06440b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 --2- 172.21.15.67:0/4150815676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628003f80 0x7f064415aca0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.304+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 >> 172.21.15.67:0/4150815676 conn(0x7f064406bf50 msgr2=0x7f06440b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:22.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.305+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 shutdown_connections 2024-04-17T08:51:22.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.305+0000 7f0649a00640 1 -- 172.21.15.67:0/4150815676 wait complete. 2024-04-17T08:51:22.324 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:50:46.826027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:50:47.478489+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '234', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:22.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.531+0000 7f04bea00640 1 Processor -- start 2024-04-17T08:51:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.531+0000 7f04bea00640 1 -- start start 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b8160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04b8146060 0x7f04b815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 0x7f04b80b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f04b80b5470 con 0x7f04b8146060 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f04b8160920 con 0x7f04b80b4ac0 2024-04-17T08:51:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f04b8160c00 con 0x7f04b8145b00 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04b8146060 0x7f04b815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04b8146060 0x7f04b815cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37884/0 (socket says 172.21.15.67:37884) 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 -- 172.21.15.67:0/2911704622 learned_addr learned my addr 172.21.15.67:0/2911704622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bd600640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b8160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04be000640 1 --2- 172.21.15.67:0/2911704622 >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 0x7f04b80b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 -- 172.21.15.67:0/2911704622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 msgr2=0x7f04b8160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b8160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 -- 172.21.15.67:0/2911704622 >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 msgr2=0x7f04b80b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 --2- 172.21.15.67:0/2911704622 >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 0x7f04b80b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04bcc00640 1 -- 172.21.15.67:0/2911704622 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b8160ee0 con 0x7f04b8146060 2024-04-17T08:51:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.533+0000 7f04be000640 1 --2- 172.21.15.67:0/2911704622 >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 0x7f04b80b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.534+0000 7f04bcc00640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815cd80 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f04b80b6190 tx=0x7f04a402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434a6d980999c18f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.534+0000 7f04b7e00640 1 -- 172.21.15.67:0/2911704622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40317d0 con 0x7f04b8146060 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.534+0000 7f04b7e00640 1 -- 172.21.15.67:0/2911704622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4037070 con 0x7f04b8146060 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.534+0000 7f04b7e00640 1 -- 172.21.15.67:0/2911704622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031b90 con 0x7f04b8146060 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 -- 172.21.15.67:0/2911704622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 msgr2=0x7f04b815cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815cd80 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f04b80b6190 tx=0x7f04a402b870 comp rx=0 tx=0).stop 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 -- 172.21.15.67:0/2911704622 shutdown_connections 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 --2- 172.21.15.67:0/2911704622 >> v2:172.21.15.187:3300/0 conn(0x7f04b80b4ac0 0x7f04b80b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815cd80 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 --2- 172.21.15.67:0/2911704622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b8160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 -- 172.21.15.67:0/2911704622 >> 172.21.15.67:0/2911704622 conn(0x7f04b80b23c0 msgr2=0x7f04b80b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 -- 172.21.15.67:0/2911704622 shutdown_connections 2024-04-17T08:51:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.535+0000 7f04bea00640 1 -- 172.21.15.67:0/2911704622 wait complete. 2024-04-17T08:51:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.538+0000 7f04bea00640 1 Processor -- start 2024-04-17T08:51:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.538+0000 7f04bea00640 1 -- start start 2024-04-17T08:51:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04b80b4ac0 0x7f04b813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b80c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04b80cfba0 con 0x7f04b8146060 2024-04-17T08:51:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f04b80b3760 con 0x7f04b80b4ac0 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f04b815d2c0 con 0x7f04b8145b00 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37900/0 (socket says 172.21.15.67:37900) 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 -- 172.21.15.67:0/2500622558 learned_addr learned my addr 172.21.15.67:0/2500622558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bd600640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04b80b4ac0 0x7f04b813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04bcc00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b80c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 -- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 msgr2=0x7f04b80c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b80c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 -- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04b80b4ac0 msgr2=0x7f04b813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04b80b4ac0 0x7f04b813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 -- 172.21.15.67:0/2500622558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b80ca680 con 0x7f04b8146060 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.539+0000 7f04be000640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f04b000cd80 tx=0x7f04b000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.540+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b000f680 con 0x7f04b8146060 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.540+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b000f840 con 0x7f04b8146060 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.540+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b000fc30 con 0x7f04b8146060 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.540+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b80ca960 con 0x7f04b8146060 2024-04-17T08:51:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.540+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b80caf20 con 0x7f04b8146060 2024-04-17T08:51:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.541+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b8068a10 con 0x7f04b8146060 2024-04-17T08:51:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.542+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f04b000e3e0 con 0x7f04b8146060 2024-04-17T08:51:22.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.543+0000 7f04b5600640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 0x7f049806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:22.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.544+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f04b008d210 con 0x7f04b8146060 2024-04-17T08:51:22.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.544+0000 7f04bd600640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 0x7f049806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.545+0000 7f04bd600640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 0x7f049806eff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006fd0 tx=0x7f04ac008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:22.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.548+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f04b0014030 con 0x7f04b8146060 2024-04-17T08:51:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.756+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04b8068610 con 0x7f04b8146060 2024-04-17T08:51:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.757+0000 7f04b5600640 1 -- 172.21.15.67:0/2500622558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v239) ==== 93+0+10 (secure 0 0 0) 0x7f04b005ae20 con 0x7f04b8146060 2024-04-17T08:51:22.756 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 msgr2=0x7f049806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 0x7f049806eff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f04ac006fd0 tx=0x7f04ac008040 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 msgr2=0x7f04b815fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f04b000cd80 tx=0x7f04b000a650 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 shutdown_connections 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f049806cb30 0x7f049806eff0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04b8146060 0x7f04b815fc30 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04b8145b00 0x7f04b80c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 --2- 172.21.15.67:0/2500622558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04b80b4ac0 0x7f04b813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 >> 172.21.15.67:0/2500622558 conn(0x7f04b80b23c0 msgr2=0x7f04b8071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 shutdown_connections 2024-04-17T08:51:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:22.760+0000 7f04bea00640 1 -- 172.21.15.67:0/2500622558 wait complete. 2024-04-17T08:51:22.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:51:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.003+0000 7f6d45e00640 1 Processor -- start 2024-04-17T08:51:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.003+0000 7f6d45e00640 1 -- start start 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.004+0000 7f6d45e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d40146060 0x7f6d40160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.004+0000 7f6d45e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d40160920 con 0x7f6d40145b00 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.004+0000 7f6d45e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d40160c00 con 0x7f6d400b4ac0 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.004+0000 7f6d45e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6d40160ee0 con 0x7f6d40146060 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d44a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d40145b00 0x7f6d4015cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52424/0 (socket says 172.21.15.67:52424) 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d44a00640 1 -- 172.21.15.67:0/1829973406 learned_addr learned my addr 172.21.15.67:0/1829973406 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3fe00640 1 --2- 172.21.15.67:0/1829973406 >> v2:172.21.15.67:3300/0 conn(0x7f6d40146060 0x7f6d40160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3f400640 1 -- 172.21.15.67:0/1829973406 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472425141 0 0) 0x7f6d40160920 con 0x7f6d40145b00 2024-04-17T08:51:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3f400640 1 -- 172.21.15.67:0/1829973406 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d28003520 con 0x7f6d40145b00 2024-04-17T08:51:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3fe00640 1 -- 172.21.15.67:0/1829973406 >> v1:172.21.15.187:6789/0 conn(0x7f6d400b4ac0 legacy=0x7f6d400b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3fe00640 1 -- 172.21.15.67:0/1829973406 >> v1:172.21.15.67:6789/0 conn(0x7f6d40145b00 legacy=0x7f6d4015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.005+0000 7f6d3fe00640 1 -- 172.21.15.67:0/1829973406 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d401611c0 con 0x7f6d40146060 2024-04-17T08:51:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.006+0000 7f6d3fe00640 1 --2- 172.21.15.67:0/1829973406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40146060 0x7f6d40160340 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0097a0 tx=0x7f6d2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1602b45e3f14d70c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.007+0000 7f6d3f400640 1 -- 172.21.15.67:0/1829973406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c035020 con 0x7f6d40146060 2024-04-17T08:51:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.007+0000 7f6d3f400640 1 -- 172.21.15.67:0/1829973406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2c031830 con 0x7f6d40146060 2024-04-17T08:51:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.007+0000 7f6d3f400640 1 -- 172.21.15.67:0/1829973406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c031c00 con 0x7f6d40146060 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.007+0000 7f6d45e00640 1 -- 172.21.15.67:0/1829973406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40146060 msgr2=0x7f6d40160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.007+0000 7f6d45e00640 1 --2- 172.21.15.67:0/1829973406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40146060 0x7f6d40160340 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0097a0 tx=0x7f6d2c02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.008+0000 7f6d45e00640 1 -- 172.21.15.67:0/1829973406 shutdown_connections 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.008+0000 7f6d45e00640 1 --2- 172.21.15.67:0/1829973406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40146060 0x7f6d40160340 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.008+0000 7f6d45e00640 1 -- 172.21.15.67:0/1829973406 >> 172.21.15.67:0/1829973406 conn(0x7f6d400b23c0 msgr2=0x7f6d400b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.008+0000 7f6d45e00640 1 -- 172.21.15.67:0/1829973406 shutdown_connections 2024-04-17T08:51:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.008+0000 7f6d45e00640 1 -- 172.21.15.67:0/1829973406 wait complete. 2024-04-17T08:51:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.009+0000 7f6d45e00640 1 Processor -- start 2024-04-17T08:51:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.009+0000 7f6d45e00640 1 -- start start 2024-04-17T08:51:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d400b4ac0 0x7f6d401385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d40146060 0x7f6d401440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d400b4f10 con 0x7f6d40145b00 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d400cfba0 con 0x7f6d400b4ac0 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.010+0000 7f6d45e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d400b5090 con 0x7f6d40146060 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37916/0 (socket says 172.21.15.67:37916) 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 -- 172.21.15.67:0/3779002349 learned_addr learned my addr 172.21.15.67:0/3779002349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d44a00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d400b4ac0 0x7f6d401385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 -- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d40146060 msgr2=0x7f6d401440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d40146060 0x7f6d401440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 -- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d400b4ac0 msgr2=0x7f6d401385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d400b4ac0 0x7f6d401385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.011+0000 7f6d3fe00640 1 -- 172.21.15.67:0/3779002349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4006e390 con 0x7f6d40145b00 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d3fe00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c034040 tx=0x7f6d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c040070 con 0x7f6d40145b00 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2c03a910 con 0x7f6d40145b00 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c035040 con 0x7f6d40145b00 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d400aa790 con 0x7f6d40145b00 2024-04-17T08:51:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.012+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4006e600 con 0x7f6d40145b00 2024-04-17T08:51:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.014+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d40068a10 con 0x7f6d40145b00 2024-04-17T08:51:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.014+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6d2c038020 con 0x7f6d40145b00 2024-04-17T08:51:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.015+0000 7f6d3cc00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 0x7f6d2006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.015+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f6d2c0adb60 con 0x7f6d40145b00 2024-04-17T08:51:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.016+0000 7f6d44a00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 0x7f6d2006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.017+0000 7f6d44a00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 0x7f6d2006eff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d340011e0 tx=0x7f6d34009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.021+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d2c031800 con 0x7f6d40145b00 2024-04-17T08:51:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.225+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f6d40068610 con 0x7f6d40145b00 2024-04-17T08:51:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.225+0000 7f6d3cc00640 1 -- 172.21.15.67:0/3779002349 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v239) ==== 107+0+11 (secure 0 0 0) 0x7f6d2c07b910 con 0x7f6d40145b00 2024-04-17T08:51:23.225 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 msgr2=0x7f6d2006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 0x7f6d2006eff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d340011e0 tx=0x7f6d34009040 comp rx=0 tx=0).stop 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 msgr2=0x7f6d40143b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c034040 tx=0x7f6d2c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 shutdown_connections 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d2006cb30 0x7f6d2006eff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d40146060 0x7f6d401440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d40145b00 0x7f6d40143b70 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.228+0000 7f6d45e00640 1 --2- 172.21.15.67:0/3779002349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d400b4ac0 0x7f6d401385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.229+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 >> 172.21.15.67:0/3779002349 conn(0x7f6d400b23c0 msgr2=0x7f6d400b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.229+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 shutdown_connections 2024-04-17T08:51:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.229+0000 7f6d45e00640 1 -- 172.21.15.67:0/3779002349 wait complete. 2024-04-17T08:51:23.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:51:23.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.470+0000 7f8f24400640 1 Processor -- start 2024-04-17T08:51:23.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.470+0000 7f8f24400640 1 -- start start 2024-04-17T08:51:23.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f24400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f1c145ac0 0x7f8f1c071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f24400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1c0cea70 con 0x7f8f1c0ce3b0 2024-04-17T08:51:23.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f24400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1c0ced50 con 0x7f8f1c072d80 2024-04-17T08:51:23.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f24400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1c0ce840 con 0x7f8f1c145ac0 2024-04-17T08:51:23.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f21a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f1c145ac0 0x7f8f1c071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f21a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f1c145ac0 0x7f8f1c071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54258/0 (socket says 172.21.15.67:54258) 2024-04-17T08:51:23.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f21a00640 1 -- 172.21.15.67:0/2456497930 learned_addr learned my addr 172.21.15.67:0/2456497930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:23.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.472+0000 7f8f21000640 1 --1- 172.21.15.67:0/2456497930 >> v1:172.21.15.67:6789/0 conn(0x7f8f1c0ce3b0 0x7f8f1c072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39724/0 (socket says 172.21.15.67:39724) 2024-04-17T08:51:23.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.473+0000 7f8f21a00640 1 -- 172.21.15.67:0/2456497930 >> v1:172.21.15.187:6789/0 conn(0x7f8f1c072d80 legacy=0x7f8f1c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.473+0000 7f8f21a00640 1 -- 172.21.15.67:0/2456497930 >> v1:172.21.15.67:6789/0 conn(0x7f8f1c0ce3b0 legacy=0x7f8f1c072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.473+0000 7f8f21a00640 1 -- 172.21.15.67:0/2456497930 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c0cf030 con 0x7f8f1c145ac0 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.473+0000 7f8f1be00640 1 -- 172.21.15.67:0/2456497930 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421009435 0 0) 0x7f8f1c0cea70 con 0x7f8f1c0ce3b0 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.474+0000 7f8f21a00640 1 --2- 172.21.15.67:0/2456497930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c145ac0 0x7f8f1c071ea0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f8f100098a0 tx=0x7f8f1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83af2f91b5fd872d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.474+0000 7f8f1be00640 1 -- 172.21.15.67:0/2456497930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10034070 con 0x7f8f1c145ac0 2024-04-17T08:51:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.474+0000 7f8f1be00640 1 -- 172.21.15.67:0/2456497930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10030840 con 0x7f8f1c145ac0 2024-04-17T08:51:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.474+0000 7f8f1be00640 1 -- 172.21.15.67:0/2456497930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10030c10 con 0x7f8f1c145ac0 2024-04-17T08:51:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 -- 172.21.15.67:0/2456497930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c145ac0 msgr2=0x7f8f1c071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 --2- 172.21.15.67:0/2456497930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c145ac0 0x7f8f1c071ea0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f8f100098a0 tx=0x7f8f1002b980 comp rx=0 tx=0).stop 2024-04-17T08:51:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 -- 172.21.15.67:0/2456497930 shutdown_connections 2024-04-17T08:51:23.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 --2- 172.21.15.67:0/2456497930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c145ac0 0x7f8f1c071ea0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 -- 172.21.15.67:0/2456497930 >> 172.21.15.67:0/2456497930 conn(0x7f8f1c067ee0 msgr2=0x7f8f1c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:23.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 -- 172.21.15.67:0/2456497930 shutdown_connections 2024-04-17T08:51:23.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.475+0000 7f8f24400640 1 -- 172.21.15.67:0/2456497930 wait complete. 2024-04-17T08:51:23.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.477+0000 7f8f24400640 1 Processor -- start 2024-04-17T08:51:23.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.477+0000 7f8f24400640 1 -- start start 2024-04-17T08:51:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 0x7f8f1c1e5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f1c145ac0 0x7f8f1c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c145ea0 con 0x7f8f1c072d80 2024-04-17T08:51:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c0cf5f0 con 0x7f8f1c0ce3b0 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f24400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1c0d0220 con 0x7f8f1c145ac0 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54274/0 (socket says 172.21.15.67:54274) 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.478+0000 7f8f21a00640 1 -- 172.21.15.67:0/1148908803 learned_addr learned my addr 172.21.15.67:0/1148908803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 -- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f1c145ac0 msgr2=0x7f8f1c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f1c145ac0 0x7f8f1c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 -- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 msgr2=0x7f8f1c1e5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21000640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 0x7f8f1c1e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 0x7f8f1c1e5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 -- 172.21.15.67:0/1148908803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c1e5f00 con 0x7f8f1c072d80 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21000640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 0x7f8f1c1e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f21a00640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f8f100098a0 tx=0x7f8f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10034040 con 0x7f8f1c072d80 2024-04-17T08:51:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10039d90 con 0x7f8f1c072d80 2024-04-17T08:51:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10043610 con 0x7f8f1c072d80 2024-04-17T08:51:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.479+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1c1e61e0 con 0x7f8f1c072d80 2024-04-17T08:51:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.480+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f1c1e67a0 con 0x7f8f1c072d80 2024-04-17T08:51:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.481+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f1c0cc6c0 con 0x7f8f1c072d80 2024-04-17T08:51:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.482+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f8f1004a060 con 0x7f8f1c072d80 2024-04-17T08:51:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.483+0000 7f8f19600640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 0x7f8efc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.483+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f8f100ae660 con 0x7f8f1c072d80 2024-04-17T08:51:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.483+0000 7f8f21000640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 0x7f8efc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.484+0000 7f8f21000640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 0x7f8efc06eff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c06e3f0 tx=0x7f8f08009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.487+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8f1003e070 con 0x7f8f1c072d80 2024-04-17T08:51:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.691+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f8f1c0685d0 con 0x7f8f1c072d80 2024-04-17T08:51:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.692+0000 7f8f19600640 1 -- 172.21.15.67:0/1148908803 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v239) ==== 103+0+11 (secure 0 0 0) 0x7f8f1002d330 con 0x7f8f1c072d80 2024-04-17T08:51:23.691 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 msgr2=0x7f8efc06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 0x7f8efc06eff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c06e3f0 tx=0x7f8f08009040 comp rx=0 tx=0).stop 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 msgr2=0x7f8f1c1e6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f8f100098a0 tx=0x7f8f100046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 shutdown_connections 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f8efc06cb30 0x7f8efc06eff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f1c145ac0 0x7f8f1c15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f1c0ce3b0 0x7f8f1c1e5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 --2- 172.21.15.67:0/1148908803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f1c072d80 0x7f8f1c1e6fa0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.695+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 >> 172.21.15.67:0/1148908803 conn(0x7f8f1c067ee0 msgr2=0x7f8f1c0cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:23.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.696+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 shutdown_connections 2024-04-17T08:51:23.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.696+0000 7f8f24400640 1 -- 172.21.15.67:0/1148908803 wait complete. 2024-04-17T08:51:23.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:23.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.943+0000 7f5346a00640 1 Processor -- start 2024-04-17T08:51:23.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.943+0000 7f5346a00640 1 -- start start 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f5346a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5340072420 0x7f534006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f5346a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534015f3c0 con 0x7f534015b530 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f5346a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534015f660 con 0x7f53400c68c0 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f5346a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f534015f940 con 0x7f5340072420 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f533fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f534015b530 0x7f5340071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39740/0 (socket says 172.21.15.67:39740) 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f533fe00640 1 -- 172.21.15.67:0/797804147 learned_addr learned my addr 172.21.15.67:0/797804147 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:23.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f533f400640 1 --2- 172.21.15.67:0/797804147 >> v2:172.21.15.67:3300/0 conn(0x7f5340072420 0x7f534006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f533ea00640 1 -- 172.21.15.67:0/797804147 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519256360 0 0) 0x7f534015f3c0 con 0x7f534015b530 2024-04-17T08:51:23.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.945+0000 7f533ea00640 1 -- 172.21.15.67:0/797804147 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5324003520 con 0x7f534015b530 2024-04-17T08:51:23.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.946+0000 7f533f400640 1 -- 172.21.15.67:0/797804147 >> v1:172.21.15.187:6789/0 conn(0x7f53400c68c0 legacy=0x7f534006e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:23.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.946+0000 7f533f400640 1 -- 172.21.15.67:0/797804147 >> v1:172.21.15.67:6789/0 conn(0x7f534015b530 legacy=0x7f5340071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.946+0000 7f533f400640 1 -- 172.21.15.67:0/797804147 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534015fc20 con 0x7f5340072420 2024-04-17T08:51:23.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.947+0000 7f533f400640 1 --2- 172.21.15.67:0/797804147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534006dd50 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f532c0097a0 tx=0x7f532c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda3958d66d4694c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.947+0000 7f533ea00640 1 -- 172.21.15.67:0/797804147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c035020 con 0x7f5340072420 2024-04-17T08:51:23.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.947+0000 7f533ea00640 1 -- 172.21.15.67:0/797804147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f532c031830 con 0x7f5340072420 2024-04-17T08:51:23.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.947+0000 7f533ea00640 1 -- 172.21.15.67:0/797804147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c031be0 con 0x7f5340072420 2024-04-17T08:51:23.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 -- 172.21.15.67:0/797804147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 msgr2=0x7f534006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 --2- 172.21.15.67:0/797804147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534006dd50 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f532c0097a0 tx=0x7f532c02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 -- 172.21.15.67:0/797804147 shutdown_connections 2024-04-17T08:51:23.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 --2- 172.21.15.67:0/797804147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534006dd50 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 -- 172.21.15.67:0/797804147 >> 172.21.15.67:0/797804147 conn(0x7f534006bf50 msgr2=0x7f53400b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:23.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 -- 172.21.15.67:0/797804147 shutdown_connections 2024-04-17T08:51:23.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.948+0000 7f5346a00640 1 -- 172.21.15.67:0/797804147 wait complete. 2024-04-17T08:51:23.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.950+0000 7f5346a00640 1 Processor -- start 2024-04-17T08:51:23.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.950+0000 7f5346a00640 1 -- start start 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53400c68c0 0x7f53400ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f534015b530 0x7f53400c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5340160140 con 0x7f5340072420 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53400cc950 con 0x7f534015b530 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f5346a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f534015b9a0 con 0x7f53400c68c0 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54296/0 (socket says 172.21.15.67:54296) 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 -- 172.21.15.67:0/3692811742 learned_addr learned my addr 172.21.15.67:0/3692811742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 -- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53400c68c0 msgr2=0x7f53400ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53400c68c0 0x7f53400ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 -- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f534015b530 msgr2=0x7f53400c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:23.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f534015b530 0x7f53400c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:23.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.951+0000 7f533fe00640 1 -- 172.21.15.67:0/3692811742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53400cf8c0 con 0x7f5340072420 2024-04-17T08:51:23.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f533fe00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f533000c430 tx=0x7f533000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330017070 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5330005ce0 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53400cfba0 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53300072f0 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.952+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5340158bc0 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.953+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5340069260 con 0x7f5340072420 2024-04-17T08:51:23.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.954+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f533001a070 con 0x7f5340072420 2024-04-17T08:51:23.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.955+0000 7f5337e00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 0x7f532006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:23.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.955+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f533008b720 con 0x7f5340072420 2024-04-17T08:51:23.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.956+0000 7f533f400640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 0x7f532006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:23.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.957+0000 7f533f400640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 0x7f532006f020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f532c036000 tx=0x7f532c030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:23.960+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5340069260 con 0x7f5340072420 2024-04-17T08:51:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.203+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5340068b90 con 0x7f5340072420 2024-04-17T08:51:24.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.205+0000 7f5337e00640 1 -- 172.21.15.67:0/3692811742 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5224 (secure 0 0 0) 0x7f5330059590 con 0x7f5340072420 2024-04-17T08:51:24.203 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:24.203 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":117,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2627239033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2627239033},{"type":"v1","addr":"172.21.15.67:6835","nonce":2627239033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:24.204 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 117 2024-04-17T08:51:24.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 msgr2=0x7f532006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 0x7f532006f020 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f532c036000 tx=0x7f532c030000 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 msgr2=0x7f534014d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f533000c430 tx=0x7f533000c8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 shutdown_connections 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f532006cb60 0x7f532006f020 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f534015b530 0x7f53400c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53400c68c0 0x7f53400ceeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 --2- 172.21.15.67:0/3692811742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5340072420 0x7f534014d200 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.207+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 >> 172.21.15.67:0/3692811742 conn(0x7f534006bf50 msgr2=0x7f534006e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.208+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 shutdown_connections 2024-04-17T08:51:24.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.208+0000 7f5346a00640 1 -- 172.21.15.67:0/3692811742 wait complete. 2024-04-17T08:51:24.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.455+0000 7fe969600640 1 Processor -- start 2024-04-17T08:51:24.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.455+0000 7fe969600640 1 -- start start 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe969600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe96406b8c0 0x7fe9640cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe969600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe96406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe969600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9640cd4a0 con 0x7fe964141ed0 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe969600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe96415ce30 con 0x7fe96406b8c0 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe969600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe9641607f0 con 0x7fe964160280 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe968c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe964141ed0 0x7fe96415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39756/0 (socket says 172.21.15.67:39756) 2024-04-17T08:51:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe968c00640 1 -- 172.21.15.67:0/281323886 learned_addr learned my addr 172.21.15.67:0/281323886 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe963e00640 1 --2- 172.21.15.67:0/281323886 >> v2:172.21.15.187:3300/0 conn(0x7fe96406b8c0 0x7fe9640cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe963400640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe96406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.457+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832883827 0 0) 0x7fe9640cd4a0 con 0x7fe964141ed0 2024-04-17T08:51:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe948003520 con 0x7fe964141ed0 2024-04-17T08:51:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 760525841 0 0) 0x7fe948003520 con 0x7fe964141ed0 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 msgr2=0x7fe96406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe96406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 >> v2:172.21.15.187:3300/0 conn(0x7fe96406b8c0 msgr2=0x7fe9640cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 --2- 172.21.15.67:0/281323886 >> v2:172.21.15.187:3300/0 conn(0x7fe96406b8c0 0x7fe9640cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe964160ad0 con 0x7fe964141ed0 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe95c001670 con 0x7fe964141ed0 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe963400640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe96406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 >> v1:172.21.15.67:6789/0 conn(0x7fe964141ed0 legacy=0x7fe96415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe948003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe948003fb0 0x7fe948004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe96415ce30 con 0x7fe948004d40 2024-04-17T08:51:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe9641607f0 con 0x7fe948003fb0 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.458+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe94800baa0 con 0x7fe964160280 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe963400640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe948003fb0 0x7fe948004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe963e00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe948003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 -- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 msgr2=0x7fe948003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe948003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 -- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe948003fb0 msgr2=0x7fe948004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe948003fb0 0x7fe948004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe968c00640 1 -- 172.21.15.67:0/281323886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe948003520 con 0x7fe948004d40 2024-04-17T08:51:24.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.459+0000 7fe963e00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe964160280 0x7fe948003a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.460+0000 7fe968c00640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9480051c0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c190 tx=0x7fe948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1921b7f204e05d62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.460+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c007c80 con 0x7fe948004d40 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.460+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe95c030070 con 0x7fe948004d40 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.460+0000 7fe962a00640 1 -- 172.21.15.67:0/281323886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95c005d70 con 0x7fe948004d40 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 -- 172.21.15.67:0/281323886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 msgr2=0x7fe9480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 --2- 172.21.15.67:0/281323886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9480051c0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c190 tx=0x7fe948005830 comp rx=0 tx=0).stop 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe968c00640 1 -- 172.21.15.67:0/281323886 reap_dead start 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 -- 172.21.15.67:0/281323886 shutdown_connections 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 -- 172.21.15.67:0/281323886 >> 172.21.15.67:0/281323886 conn(0x7fe964067d10 msgr2=0x7fe9640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 -- 172.21.15.67:0/281323886 shutdown_connections 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.461+0000 7fe969600640 1 -- 172.21.15.67:0/281323886 wait complete. 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.463+0000 7fe969600640 1 Processor -- start 2024-04-17T08:51:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.463+0000 7fe969600640 1 -- start start 2024-04-17T08:51:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe96406f770 0x7fe96415e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9640b5220 0x7fe9641f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe964161090 con 0x7fe948004d40 2024-04-17T08:51:24.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe9640cddf0 con 0x7fe9640b5220 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe969600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe96413f9b0 con 0x7fe96406f770 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.464+0000 7fe963e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54308/0 (socket says 172.21.15.67:54308) 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 -- 172.21.15.67:0/29237495 learned_addr learned my addr 172.21.15.67:0/29237495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 -- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe96406f770 msgr2=0x7fe96415e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe96406f770 0x7fe96415e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 -- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9640b5220 msgr2=0x7fe9641f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe968c00640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9640b5220 0x7fe9641f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9640b5220 0x7fe9641f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 -- 172.21.15.67:0/29237495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9640b4460 con 0x7fe948004d40 2024-04-17T08:51:24.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.465+0000 7fe963e00640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fe95400f3d0 tx=0x7fe95400f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.466+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954003990 con 0x7fe948004d40 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.466+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe954003b30 con 0x7fe948004d40 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.466+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954004020 con 0x7fe948004d40 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.466+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9640b4740 con 0x7fe948004d40 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.466+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9640b4d00 con 0x7fe948004d40 2024-04-17T08:51:24.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.467+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe964141cc0 con 0x7fe948004d40 2024-04-17T08:51:24.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.468+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe954005d10 con 0x7fe948004d40 2024-04-17T08:51:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.469+0000 7fe95be00640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 0x7fe94006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.469+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7fe95408cfe0 con 0x7fe948004d40 2024-04-17T08:51:24.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.470+0000 7fe963400640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 0x7fe94006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.471+0000 7fe963400640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 0x7fe94006f020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe94c006f80 tx=0x7fe94c006e30 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.474+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe964141cc0 con 0x7fe948004d40 2024-04-17T08:51:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.679+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe964068a10 con 0x7fe948004d40 2024-04-17T08:51:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.682+0000 7fe95be00640 1 -- 172.21.15.67:0/29237495 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) ==== 74+0+18019 (secure 0 0 0) 0x7fe954024020 con 0x7fe948004d40 2024-04-17T08:51:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:24.681 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":239,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:21.800765+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.67:6835/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.67:6834/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 msgr2=0x7fe94006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 0x7fe94006f020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe94c006f80 tx=0x7fe94c006e30 comp rx=0 tx=0).stop 2024-04-17T08:51:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 msgr2=0x7fe9641457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fe95400f3d0 tx=0x7fe95400f890 comp rx=0 tx=0).stop 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 shutdown_connections 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe94006cb60 0x7fe94006f020 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9640b5220 0x7fe9641f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe96406f770 0x7fe96415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 --2- 172.21.15.67:0/29237495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe948004d40 0x7fe9641457d0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 >> 172.21.15.67:0/29237495 conn(0x7fe964067d10 msgr2=0x7fe96413fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 shutdown_connections 2024-04-17T08:51:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.685+0000 7fe969600640 1 -- 172.21.15.67:0/29237495 wait complete. 2024-04-17T08:51:24.703 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-17T08:51:24.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.919+0000 7fbdf8c00640 1 Processor -- start 2024-04-17T08:51:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.919+0000 7fbdf8c00640 1 -- start start 2024-04-17T08:51:24.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.920+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf4160280 0x7fbdf406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbdf4141ed0 0x7fbdf4142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf8c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdf40cd4a0 con 0x7fbdf406b8c0 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf8c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbdf415ce30 con 0x7fbdf4141ed0 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf8c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf41607f0 con 0x7fbdf4160280 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbdf4141ed0 0x7fbdf4142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbdf4141ed0 0x7fbdf4142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54322/0 (socket says 172.21.15.67:54322) 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 -- 172.21.15.67:0/3137171678 learned_addr learned my addr 172.21.15.67:0/3137171678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf2a00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf4160280 0x7fbdf406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 -- 172.21.15.67:0/3137171678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf4160280 msgr2=0x7fbdf406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf4160280 0x7fbdf406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 -- 172.21.15.67:0/3137171678 >> v1:172.21.15.187:6789/0 conn(0x7fbdf406b8c0 legacy=0x7fbdf40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.921+0000 7fbdf3e00640 1 -- 172.21.15.67:0/3137171678 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf4160ad0 con 0x7fbdf4141ed0 2024-04-17T08:51:24.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf3e00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4141ed0 0x7fbdf4142360 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009870 tx=0x7fbdec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c9381dabe56507 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf2000640 1 -- 172.21.15.67:0/3137171678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec035020 con 0x7fbdf4141ed0 2024-04-17T08:51:24.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf2000640 1 -- 172.21.15.67:0/3137171678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdec031640 con 0x7fbdf4141ed0 2024-04-17T08:51:24.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf2000640 1 -- 172.21.15.67:0/3137171678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec02d040 con 0x7fbdf4141ed0 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf8c00640 1 -- 172.21.15.67:0/3137171678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4141ed0 msgr2=0x7fbdf4142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.923+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4141ed0 0x7fbdf4142360 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009870 tx=0x7fbdec02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 -- 172.21.15.67:0/3137171678 shutdown_connections 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4141ed0 0x7fbdf4142360 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/3137171678 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf4160280 0x7fbdf406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 -- 172.21.15.67:0/3137171678 >> 172.21.15.67:0/3137171678 conn(0x7fbdf4067d10 msgr2=0x7fbdf40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 -- 172.21.15.67:0/3137171678 shutdown_connections 2024-04-17T08:51:24.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.924+0000 7fbdf8c00640 1 -- 172.21.15.67:0/3137171678 wait complete. 2024-04-17T08:51:24.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.925+0000 7fbdf8c00640 1 Processor -- start 2024-04-17T08:51:24.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.925+0000 7fbdf8c00640 1 -- start start 2024-04-17T08:51:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf406b8c0 0x7fbdf4143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf4141ed0 0x7fbdf413e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf4161090 con 0x7fbdf4160280 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf406c1e0 con 0x7fbdf4141ed0 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.926+0000 7fbdf8c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf413f9b0 con 0x7fbdf406b8c0 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54332/0 (socket says 172.21.15.67:54332) 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 -- 172.21.15.67:0/482463329 learned_addr learned my addr 172.21.15.67:0/482463329 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3400640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf406b8c0 0x7fbdf4143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 -- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf406b8c0 msgr2=0x7fbdf4143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf2a00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf4141ed0 0x7fbdf413e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf406b8c0 0x7fbdf4143050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 -- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf4141ed0 msgr2=0x7fbdf413e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf4141ed0 0x7fbdf413e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.927+0000 7fbdf3e00640 1 -- 172.21.15.67:0/482463329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf41443e0 con 0x7fbdf4160280 2024-04-17T08:51:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdf3e00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009870 tx=0x7fbdec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec03ac60 con 0x7fbdf4160280 2024-04-17T08:51:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdec03ae20 con 0x7fbdf4160280 2024-04-17T08:51:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf415dbc0 con 0x7fbdf4160280 2024-04-17T08:51:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec044630 con 0x7fbdf4160280 2024-04-17T08:51:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.928+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf41440d0 con 0x7fbdf4160280 2024-04-17T08:51:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.929+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf4068a10 con 0x7fbdf4160280 2024-04-17T08:51:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.930+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fbdec038020 con 0x7fbdf4160280 2024-04-17T08:51:24.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.931+0000 7fbdeb400640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 0x7fbdd006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:24.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.932+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7fbdec0ae2c0 con 0x7fbdf4160280 2024-04-17T08:51:24.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.932+0000 7fbdf3400640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 0x7fbdd006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:24.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.933+0000 7fbdf3400640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 0x7fbdd006eff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007fd0 tx=0x7fbde4009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:24.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:24.936+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbdec03f070 con 0x7fbdf4160280 2024-04-17T08:51:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.183+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdf4145ff0 con 0x7fbdf4160280 2024-04-17T08:51:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.185+0000 7fbdeb400640 1 -- 172.21.15.67:0/482463329 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5224 (secure 0 0 0) 0x7fbdec07be30 con 0x7fbdf4160280 2024-04-17T08:51:25.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:25.183 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":117,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2627239033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2627239033},{"type":"v1","addr":"172.21.15.67:6835","nonce":2627239033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:25.184 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 117 2024-04-17T08:51:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 msgr2=0x7fbdd006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 0x7fbdd006eff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007fd0 tx=0x7fbde4009040 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 msgr2=0x7fbdf4143b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009870 tx=0x7fbdec0046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 shutdown_connections 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbdd006cb30 0x7fbdd006eff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdf4160280 0x7fbdf4143b90 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdf4141ed0 0x7fbdf413e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 --2- 172.21.15.67:0/482463329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdf406b8c0 0x7fbdf4143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 >> 172.21.15.67:0/482463329 conn(0x7fbdf4067d10 msgr2=0x7fbdf40cfc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 shutdown_connections 2024-04-17T08:51:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.188+0000 7fbdf8c00640 1 -- 172.21.15.67:0/482463329 wait complete. 2024-04-17T08:51:25.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:25.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.421+0000 7f894d000640 1 Processor -- start 2024-04-17T08:51:25.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.421+0000 7f894d000640 1 -- start start 2024-04-17T08:51:25.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.423+0000 7f894d000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f894806f670 0x7f894806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.423+0000 7f894d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f89480cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.423+0000 7f894d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894806ff90 con 0x7f89480cd020 2024-04-17T08:51:25.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.423+0000 7f894d000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89480ceb00 con 0x7f894806f670 2024-04-17T08:51:25.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.423+0000 7f894d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89480cede0 con 0x7f894806f030 2024-04-17T08:51:25.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8945400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89480cd020 0x7f8948071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39758/0 (socket says 172.21.15.67:39758) 2024-04-17T08:51:25.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8945400640 1 -- 172.21.15.67:0/4117322609 learned_addr learned my addr 172.21.15.67:0/4117322609 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:25.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8946800640 1 --2- 172.21.15.67:0/4117322609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f89480cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8945e00640 1 --2- 172.21.15.67:0/4117322609 >> v2:172.21.15.187:3300/0 conn(0x7f894806f670 0x7f894806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440904881 0 0) 0x7f894806ff90 con 0x7f89480cd020 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.424+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f892c003520 con 0x7f89480cd020 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.425+0000 7f8946800640 1 -- 172.21.15.67:0/4117322609 >> v2:172.21.15.187:3300/0 conn(0x7f894806f670 msgr2=0x7f894806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.425+0000 7f8946800640 1 --2- 172.21.15.67:0/4117322609 >> v2:172.21.15.187:3300/0 conn(0x7f894806f670 0x7f894806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.425+0000 7f8946800640 1 -- 172.21.15.67:0/4117322609 >> v1:172.21.15.67:6789/0 conn(0x7f89480cd020 legacy=0x7f8948071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.425+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4056662321 0 0) 0x7f892c003520 con 0x7f89480cd020 2024-04-17T08:51:25.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.425+0000 7f8946800640 1 -- 172.21.15.67:0/4117322609 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480cf0c0 con 0x7f894806f030 2024-04-17T08:51:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.426+0000 7f8946800640 1 --2- 172.21.15.67:0/4117322609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f89480cd480 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f89400097a0 tx=0x7f894002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c8e875830a1e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940037070 con 0x7f894806f030 2024-04-17T08:51:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8940030510 con 0x7f894806f030 2024-04-17T08:51:25.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f8944a00640 1 -- 172.21.15.67:0/4117322609 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930001670 con 0x7f894806f030 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 -- 172.21.15.67:0/4117322609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 msgr2=0x7f89480cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 --2- 172.21.15.67:0/4117322609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f89480cd480 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f89400097a0 tx=0x7f894002b790 comp rx=0 tx=0).stop 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 -- 172.21.15.67:0/4117322609 shutdown_connections 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 --2- 172.21.15.67:0/4117322609 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f89480cd480 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 --2- 172.21.15.67:0/4117322609 >> v2:172.21.15.187:3300/0 conn(0x7f894806f670 0x7f894806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.427+0000 7f894d000640 1 -- 172.21.15.67:0/4117322609 >> 172.21.15.67:0/4117322609 conn(0x7f894806bf50 msgr2=0x7f89480b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.428+0000 7f894d000640 1 -- 172.21.15.67:0/4117322609 shutdown_connections 2024-04-17T08:51:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.428+0000 7f894d000640 1 -- 172.21.15.67:0/4117322609 wait complete. 2024-04-17T08:51:25.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.430+0000 7f894d000640 1 Processor -- start 2024-04-17T08:51:25.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.430+0000 7f894d000640 1 -- start start 2024-04-17T08:51:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f894814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89480cd020 0x7f89480c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89480ce720 con 0x7f894806f670 2024-04-17T08:51:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89480cf680 con 0x7f89480cd020 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.431+0000 7f894d000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89480ce8a0 con 0x7f894806f030 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54336/0 (socket says 172.21.15.67:54336) 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 -- 172.21.15.67:0/3289769984 learned_addr learned my addr 172.21.15.67:0/3289769984 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8946800640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89480cd020 0x7f89480c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945e00640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f894814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 -- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 msgr2=0x7f894814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f894814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.432+0000 7f8945400640 1 -- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89480cd020 msgr2=0x7f89480c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f8945400640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89480cd020 0x7f89480c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f8945400640 1 -- 172.21.15.67:0/3289769984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480afc60 con 0x7f894806f670 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f8945e00640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f894814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f8945400640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f893000c780 tx=0x7f893000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930005ad0 con 0x7f894806f670 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8930005c90 con 0x7f894806f670 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300072b0 con 0x7f894806f670 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89480af6a0 con 0x7f894806f670 2024-04-17T08:51:25.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.433+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89480c5450 con 0x7f894806f670 2024-04-17T08:51:25.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.435+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8948068b90 con 0x7f894806f670 2024-04-17T08:51:25.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.435+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f8930007450 con 0x7f894806f670 2024-04-17T08:51:25.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.436+0000 7f893e000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 0x7f892406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.436+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f893008c0e0 con 0x7f894806f670 2024-04-17T08:51:25.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.437+0000 7f8945e00640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 0x7f892406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.438+0000 7f8945e00640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 0x7f892406eff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f89380074c0 tx=0x7f893800b040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.441+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8930059e90 con 0x7f894806f670 2024-04-17T08:51:25.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.684+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8948075210 con 0x7f894806f670 2024-04-17T08:51:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.685+0000 7f893e000640 1 -- 172.21.15.67:0/3289769984 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5224 (secure 0 0 0) 0x7f8930059790 con 0x7f894806f670 2024-04-17T08:51:25.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:25.684 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":117,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2627239033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2627239033},{"type":"v1","addr":"172.21.15.67:6835","nonce":2627239033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:50:48.498917+0000","modified":"2024-04-17T08:50:51.569805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11025},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11025":{"gid":11025,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":8,"addr":"172.21.15.67:6833/4206695657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4206695657},{"type":"v1","addr":"172.21.15.67:6833","nonce":4206695657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11025,"qdb_cluster":[11025]},"id":14}]} 2024-04-17T08:51:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 117 2024-04-17T08:51:25.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 msgr2=0x7f892406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 0x7f892406eff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f89380074c0 tx=0x7f893800b040 comp rx=0 tx=0).stop 2024-04-17T08:51:25.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 msgr2=0x7f89480c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f893000c780 tx=0x7f893000cc40 comp rx=0 tx=0).stop 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 shutdown_connections 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f892406cb30 0x7f892406eff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89480cd020 0x7f89480c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f894806f670 0x7f89480c8060 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 --2- 172.21.15.67:0/3289769984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f894806f030 0x7f894814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 >> 172.21.15.67:0/3289769984 conn(0x7f894806bf50 msgr2=0x7f89480aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 shutdown_connections 2024-04-17T08:51:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.688+0000 7f894d000640 1 -- 172.21.15.67:0/3289769984 wait complete. 2024-04-17T08:51:25.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.936+0000 7ff59fe00640 1 Processor -- start 2024-04-17T08:51:25.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.936+0000 7ff59fe00640 1 -- start start 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff5a00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a00cd4a0 con 0x7ff5a0160280 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a01607f0 con 0x7ff5a0141ed0 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a015ce30 con 0x7ff5a006b8c0 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5a0160280 0x7ff5a006bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39762/0 (socket says 172.21.15.67:39762) 2024-04-17T08:51:25.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59e000640 1 -- 172.21.15.67:0/251673983 learned_addr learned my addr 172.21.15.67:0/251673983 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.938+0000 7ff59ea00640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff5a00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521341414 0 0) 0x7ff5a00cd4a0 con 0x7ff5a0160280 2024-04-17T08:51:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff584003520 con 0x7ff5a0160280 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1940407032 0 0) 0x7ff584003520 con 0x7ff5a0160280 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 msgr2=0x7ff5a00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff5a00cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 >> v1:172.21.15.187:6789/0 conn(0x7ff5a0141ed0 legacy=0x7ff5a015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a0160ad0 con 0x7ff5a0160280 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7ff58c001670 con 0x7ff5a0160280 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 >> v1:172.21.15.67:6789/0 conn(0x7ff5a0160280 legacy=0x7ff5a006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584003a50 0x7ff584003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff584004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff584004d40 0x7ff5840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a015ce30 con 0x7ff584004d40 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff58400bad0 con 0x7ff584003a50 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.939+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff58400bcd0 con 0x7ff5a006b8c0 2024-04-17T08:51:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff584004d40 0x7ff5840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59e000640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff584004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59ea00640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584003a50 0x7ff584003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 -- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 msgr2=0x7ff584004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a006b8c0 0x7ff584004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 -- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584003a50 msgr2=0x7ff584003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584003a50 0x7ff584003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.940+0000 7ff59f400640 1 -- 172.21.15.67:0/251673983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff584003520 con 0x7ff584004d40 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.941+0000 7ff59f400640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff584004d40 0x7ff5840051c0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7ff598001e20 tx=0x7ff584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f0d951d6d712e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598001a70 con 0x7ff584004d40 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff598003b70 con 0x7ff584004d40 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59d600640 1 -- 172.21.15.67:0/251673983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5980050d0 con 0x7ff584004d40 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 -- 172.21.15.67:0/251673983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff584004d40 msgr2=0x7ff5840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 --2- 172.21.15.67:0/251673983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff584004d40 0x7ff5840051c0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7ff598001e20 tx=0x7ff584005830 comp rx=0 tx=0).stop 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59f400640 1 -- 172.21.15.67:0/251673983 reap_dead start 2024-04-17T08:51:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 -- 172.21.15.67:0/251673983 shutdown_connections 2024-04-17T08:51:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 -- 172.21.15.67:0/251673983 >> 172.21.15.67:0/251673983 conn(0x7ff5a0067d10 msgr2=0x7ff5a00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 -- 172.21.15.67:0/251673983 shutdown_connections 2024-04-17T08:51:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.942+0000 7ff59fe00640 1 -- 172.21.15.67:0/251673983 wait complete. 2024-04-17T08:51:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.944+0000 7ff59fe00640 1 Processor -- start 2024-04-17T08:51:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.944+0000 7ff59fe00640 1 -- start start 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 0x7ff5a0155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a0144da0 0x7ff5a0145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a013f9b0 con 0x7ff5a0141300 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a0161090 con 0x7ff584004d40 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.945+0000 7ff59fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a013fb30 con 0x7ff5a0144da0 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54362/0 (socket says 172.21.15.67:54362) 2024-04-17T08:51:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 -- 172.21.15.67:0/1077202863 learned_addr learned my addr 172.21.15.67:0/1077202863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59ea00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 0x7ff5a0155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 -- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a0144da0 msgr2=0x7ff5a0145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a0144da0 0x7ff5a0145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 -- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 msgr2=0x7ff5a0155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 0x7ff5a0155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 -- 172.21.15.67:0/1077202863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a0145e90 con 0x7ff5a0141300 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59ea00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 0x7ff5a0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.946+0000 7ff59e000640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00e830 tx=0x7ff58c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.947+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c00c850 con 0x7ff5a0141300 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.947+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58c018070 con 0x7ff5a0141300 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.947+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c00cc30 con 0x7ff5a0141300 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.947+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a015d870 con 0x7ff5a0141300 2024-04-17T08:51:25.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.947+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a0145bb0 con 0x7ff5a0141300 2024-04-17T08:51:25.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.948+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a0068a10 con 0x7ff5a0141300 2024-04-17T08:51:25.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.949+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7ff58c01d460 con 0x7ff5a0141300 2024-04-17T08:51:25.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.950+0000 7ff596a00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 0x7ff58006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:25.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.950+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7ff58c08c200 con 0x7ff5a0141300 2024-04-17T08:51:25.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.951+0000 7ff59ea00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 0x7ff58006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:25.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.952+0000 7ff59ea00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 0x7ff58006eff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff590009980 tx=0x7ff590008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:25.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:25.955+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff58c059fe0 con 0x7ff5a0141300 2024-04-17T08:51:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.160+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5a00cc630 con 0x7ff5a0141300 2024-04-17T08:51:26.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.162+0000 7ff596a00640 1 -- 172.21.15.67:0/1077202863 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) ==== 74+0+18019 (secure 0 0 0) 0x7ff58c05ee20 con 0x7ff5a0141300 2024-04-17T08:51:26.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":239,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:21.800765+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:50:46.826027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:50:47.478489+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"234","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.67:6835/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.67:6834/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 msgr2=0x7ff58006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 0x7ff58006eff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff590009980 tx=0x7ff590008040 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 msgr2=0x7ff5a0141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00e830 tx=0x7ff58c00ecf0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 shutdown_connections 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff58006cb30 0x7ff58006eff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff5a0144da0 0x7ff5a0145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a0141300 0x7ff5a0141780 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 --2- 172.21.15.67:0/1077202863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff584004d40 0x7ff5a0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.165+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 >> 172.21.15.67:0/1077202863 conn(0x7ff5a0067d10 msgr2=0x7ff5a00b53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.166+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 shutdown_connections 2024-04-17T08:51:26.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.166+0000 7ff59fe00640 1 -- 172.21.15.67:0/1077202863 wait complete. 2024-04-17T08:51:26.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-17T08:51:26.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.410+0000 7f6ce1c00640 1 Processor -- start 2024-04-17T08:51:26.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.410+0000 7f6ce1c00640 1 -- start start 2024-04-17T08:51:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.411+0000 7f6ce1c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 0x7f6cdc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.411+0000 7f6ce1c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.411+0000 7f6ce1c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cdc15d060 con 0x7f6cdc0723a0 2024-04-17T08:51:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.411+0000 7f6ce1c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc0cfe90 con 0x7f6cdc0cbc00 2024-04-17T08:51:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.411+0000 7f6ce1c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc15dc70 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6cdb400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6cdbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 0x7f6cdc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6ce1200640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f6cdc0723a0 0x7f6cdc160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:57472/0 (socket says 172.21.15.67:57472) 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6cdbe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 0x7f6cdc0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:41644/0 (socket says 172.21.15.67:41644) 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6cdbe00640 1 -- 172.21.15.67:0/1306115119 learned_addr learned my addr 172.21.15.67:0/1306115119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.412+0000 7f6cdb400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:53192/0 (socket says 172.21.15.67:53192) 2024-04-17T08:51:26.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.413+0000 7f6cdb400640 1 -- 172.21.15.67:0/1306115119 >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 msgr2=0x7f6cdc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.413+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1306115119 >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 0x7f6cdc0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.413+0000 7f6cdb400640 1 -- 172.21.15.67:0/1306115119 >> v1:172.21.15.187:6789/0 conn(0x7f6cdc0723a0 legacy=0x7f6cdc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.413+0000 7f6cdb400640 1 -- 172.21.15.67:0/1306115119 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc15d340 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.415+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1306115119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40097a0 tx=0x7f6cc402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211392f613dd0256 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.415+0000 7f6cdaa00640 1 -- 172.21.15.67:0/1306115119 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4035020 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.415+0000 7f6cdaa00640 1 -- 172.21.15.67:0/1306115119 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc4031830 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.415+0000 7f6cdaa00640 1 -- 172.21.15.67:0/1306115119 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4031c00 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1306115119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 msgr2=0x7f6cdc071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1306115119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40097a0 tx=0x7f6cc402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1306115119 shutdown_connections 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1306115119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0b63c0 0x7f6cdc071e60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1306115119 >> v2:172.21.15.187:3300/0 conn(0x7f6cdc0cbc00 0x7f6cdc0cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1306115119 >> 172.21.15.67:0/1306115119 conn(0x7f6cdc06bef0 msgr2=0x7f6cdc0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1306115119 shutdown_connections 2024-04-17T08:51:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.416+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1306115119 wait complete. 2024-04-17T08:51:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.418+0000 7f6ce1c00640 1 Processor -- start 2024-04-17T08:51:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.418+0000 7f6ce1c00640 1 -- start start 2024-04-17T08:51:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 0x7f6cdc157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0cbc00 0x7f6cdc144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc0d02d0 con 0x7f6cdc0723a0 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc15d900 con 0x7f6cdc0b63c0 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6ce1c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc0cbfe0 con 0x7f6cdc0cbc00 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54374/0 (socket says 172.21.15.67:54374) 2024-04-17T08:51:26.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.419+0000 7f6cdbe00640 1 -- 172.21.15.67:0/1679691341 learned_addr learned my addr 172.21.15.67:0/1679691341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6ce1200640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0cbc00 0x7f6cdc144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 0x7f6cdc157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 -- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0cbc00 msgr2=0x7f6cdc144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0cbc00 0x7f6cdc144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 -- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 msgr2=0x7f6cdc157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 0x7f6cdc157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 -- 172.21.15.67:0/1679691341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc1f38d0 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 0x7f6cdc157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cdbe00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00efc0 tx=0x7f6ccc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc00cd40 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ccc019070 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0146a0 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.420+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc158080 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.421+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc156d80 con 0x7f6cdc0723a0 2024-04-17T08:51:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.422+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc068a10 con 0x7f6cdc0723a0 2024-04-17T08:51:26.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.423+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6ccc005d10 con 0x7f6cdc0723a0 2024-04-17T08:51:26.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.424+0000 7f6cd3e00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 0x7f6cb806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.424+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 6652+0+0 (secure 0 0 0) 0x7f6ccc08c870 con 0x7f6cdc0723a0 2024-04-17T08:51:26.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.424+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 0x7f6cb806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.426+0000 7f6cdb400640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 0x7f6cb806eff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4036000 tx=0x7f6cc4008030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.428+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6cdc1f38d0 con 0x7f6cdc0723a0 2024-04-17T08:51:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.674+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7f6cdc0b3020 con 0x7f6cdc0723a0 2024-04-17T08:51:26.674 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:51:26.676+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:51:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.679+0000 7f6cd3e00640 1 -- 172.21.15.67:0/1679691341 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) ==== 167+0+0 (secure 0 0 0) 0x7f6ccc010040 con 0x7f6cdc0723a0 2024-04-17T08:51:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:51:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.683+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 msgr2=0x7f6cb806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.683+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 0x7f6cb806eff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4036000 tx=0x7f6cc4008030 comp rx=0 tx=0).stop 2024-04-17T08:51:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.683+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 msgr2=0x7f6cdc1f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.683+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00efc0 tx=0x7f6ccc00c490 comp rx=0 tx=0).stop 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 shutdown_connections 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6cb806cb30 0x7f6cb806eff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6cdc0cbc00 0x7f6cdc144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6cdc0b63c0 0x7f6cdc157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 --2- 172.21.15.67:0/1679691341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0723a0 0x7f6cdc1f3390 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 >> 172.21.15.67:0/1679691341 conn(0x7f6cdc06bef0 msgr2=0x7f6cdc0cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 shutdown_connections 2024-04-17T08:51:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.684+0000 7f6ce1c00640 1 -- 172.21.15.67:0/1679691341 wait complete. 2024-04-17T08:51:26.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-17T08:51:26.707 INFO:tasks.ceph.mds.c.smithi067.stderr: -2259> 2024-04-17T08:50:52.281+0000 7f4790e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:51:26.769 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:51:26.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.907+0000 7f828a200640 1 Processor -- start 2024-04-17T08:51:26.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.907+0000 7f828a200640 1 -- start start 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f828a200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8284159020 0x7f828415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f828a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 0x7f8284162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f828a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828415f780 con 0x7f828406b8c0 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f828a200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f828415fa60 con 0x7f8284159020 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f828a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82841624a0 con 0x7f82840af400 2024-04-17T08:51:26.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f8289800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f828406b8c0 0x7f82840adb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39766/0 (socket says 172.21.15.67:39766) 2024-04-17T08:51:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f8289800640 1 -- 172.21.15.67:0/2992499122 learned_addr learned my addr 172.21.15.67:0/2992499122 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f8283e00640 1 --2- 172.21.15.67:0/2992499122 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 0x7f8284162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.909+0000 7f8288e00640 1 --2- 172.21.15.67:0/2992499122 >> v2:172.21.15.187:3300/0 conn(0x7f8284159020 0x7f828415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1892179762 0 0) 0x7f828415f780 con 0x7f828406b8c0 2024-04-17T08:51:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c003520 con 0x7f828406b8c0 2024-04-17T08:51:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1538261923 0 0) 0x7f826c003520 con 0x7f828406b8c0 2024-04-17T08:51:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283e00640 1 -- 172.21.15.67:0/2992499122 >> v2:172.21.15.187:3300/0 conn(0x7f8284159020 msgr2=0x7f828415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283e00640 1 --2- 172.21.15.67:0/2992499122 >> v2:172.21.15.187:3300/0 conn(0x7f8284159020 0x7f828415ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283e00640 1 -- 172.21.15.67:0/2992499122 >> v1:172.21.15.67:6789/0 conn(0x7f828406b8c0 legacy=0x7f82840adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.910+0000 7f8283e00640 1 -- 172.21.15.67:0/2992499122 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8284162780 con 0x7f82840af400 2024-04-17T08:51:26.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.912+0000 7f8283e00640 1 --2- 172.21.15.67:0/2992499122 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 0x7f8284162ca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f82700097a0 tx=0x7f827002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c643ccb40177d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.912+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270037070 con 0x7f82840af400 2024-04-17T08:51:26.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.912+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8270030510 con 0x7f82840af400 2024-04-17T08:51:26.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.912+0000 7f8283400640 1 -- 172.21.15.67:0/2992499122 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c0019f0 con 0x7f82840af400 2024-04-17T08:51:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 -- 172.21.15.67:0/2992499122 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 msgr2=0x7f8284162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 --2- 172.21.15.67:0/2992499122 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 0x7f8284162ca0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f82700097a0 tx=0x7f827002b710 comp rx=0 tx=0).stop 2024-04-17T08:51:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 -- 172.21.15.67:0/2992499122 shutdown_connections 2024-04-17T08:51:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 --2- 172.21.15.67:0/2992499122 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82840af400 0x7f8284162ca0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 --2- 172.21.15.67:0/2992499122 >> v2:172.21.15.187:3300/0 conn(0x7f8284159020 0x7f828415ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 -- 172.21.15.67:0/2992499122 >> 172.21.15.67:0/2992499122 conn(0x7f82840b2190 msgr2=0x7f82840b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 -- 172.21.15.67:0/2992499122 shutdown_connections 2024-04-17T08:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.913+0000 7f828a200640 1 -- 172.21.15.67:0/2992499122 wait complete. 2024-04-17T08:51:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.915+0000 7f828a200640 1 Processor -- start 2024-04-17T08:51:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.915+0000 7f828a200640 1 -- start start 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828406b8c0 0x7f8284154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 0x7f82841f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f828415f4f0 con 0x7f8284159020 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f828415f370 con 0x7f82840af400 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f828a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82841631e0 con 0x7f828406b8c0 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f8289800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f8289800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54406/0 (socket says 172.21.15.67:54406) 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f8289800640 1 -- 172.21.15.67:0/4108317065 learned_addr learned my addr 172.21.15.67:0/4108317065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f8283e00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 0x7f82841f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.916+0000 7f8288e00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828406b8c0 0x7f8284154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 -- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828406b8c0 msgr2=0x7f8284154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828406b8c0 0x7f8284154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 -- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 msgr2=0x7f82841f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 0x7f82841f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 -- 172.21.15.67:0/4108317065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82841f5190 con 0x7f8284159020 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8283e00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 0x7f82841f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8289800640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f827c00cc90 tx=0x7f827c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c00ce40 con 0x7f8284159020 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827c012400 con 0x7f8284159020 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828415b5c0 con 0x7f8284159020 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.917+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c0127f0 con 0x7f8284159020 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.918+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828415bb80 con 0x7f8284159020 2024-04-17T08:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.919+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8284068a10 con 0x7f8284159020 2024-04-17T08:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.920+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f827c012990 con 0x7f8284159020 2024-04-17T08:51:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.921+0000 7f8280c00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 0x7f826406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:26.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.921+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 6738+0+0 (secure 0 0 0) 0x7f827c08ccd0 con 0x7f8284159020 2024-04-17T08:51:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.922+0000 7f8288e00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 0x7f826406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.923+0000 7f8288e00640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 0x7f826406eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82841f8880 tx=0x7f8278008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:26.926+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f82841f5190 con 0x7f8284159020 2024-04-17T08:51:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.169+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7f828406c7b0 con 0x7f8284159020 2024-04-17T08:51:27.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:51:27.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:51:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.693+0000 7f8280c00640 1 -- 172.21.15.67:0/4108317065 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v119) ==== 107+0+0 (secure 0 0 0) 0x7f827c05a870 con 0x7f8284159020 2024-04-17T08:51:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 msgr2=0x7f826406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 0x7f826406eff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82841f8880 tx=0x7f8278008040 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 msgr2=0x7f8284159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f827c00cc90 tx=0x7f827c00a430 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 shutdown_connections 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f826406cb30 0x7f826406eff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8284159020 0x7f8284159400 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82840af400 0x7f82841f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 --2- 172.21.15.67:0/4108317065 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f828406b8c0 0x7f8284154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 >> 172.21.15.67:0/4108317065 conn(0x7f82840b2190 msgr2=0x7f8284156bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 shutdown_connections 2024-04-17T08:51:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.696+0000 7f828a200640 1 -- 172.21.15.67:0/4108317065 wait complete. 2024-04-17T08:51:27.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-17T08:51:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.923+0000 7f00e3e00640 1 Processor -- start 2024-04-17T08:51:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.923+0000 7f00e3e00640 1 -- start start 2024-04-17T08:51:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00e4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e4146510 con 0x7f00e40cd2f0 2024-04-17T08:51:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e4161000 con 0x7f00e4146080 2024-04-17T08:51:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00e40cfa60 con 0x7f00e40cd6d0 2024-04-17T08:51:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00e40cd2f0 0x7f00e40cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39776/0 (socket says 172.21.15.67:39776) 2024-04-17T08:51:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e2a00640 1 -- 172.21.15.67:0/2896668048 learned_addr learned my addr 172.21.15.67:0/2896668048 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:27.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.925+0000 7f00e2000640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00e4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133508204 0 0) 0x7f00e4146510 con 0x7f00e40cd2f0 2024-04-17T08:51:27.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00c8003520 con 0x7f00e40cd2f0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2162592308 0 0) 0x7f00c8003520 con 0x7f00e40cd2f0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 msgr2=0x7f00e4145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00e4145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 >> v1:172.21.15.187:6789/0 conn(0x7f00e4146080 legacy=0x7f00e4160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e40ce870 con 0x7f00e40cd2f0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f00d4001670 con 0x7f00e40cd2f0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 >> v1:172.21.15.67:6789/0 conn(0x7f00e40cd2f0 legacy=0x7f00e40cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00c8004800 0x7f00c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40cfa60 con 0x7f00c80039e0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00c800bad0 con 0x7f00c8004800 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00c800bcd0 con 0x7f00e40cd6d0 2024-04-17T08:51:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.926+0000 7f00e2a00640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e3400640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e2a00640 1 -- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 msgr2=0x7f00c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e2a00640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e2a00640 1 -- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00c8004800 msgr2=0x7f00c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e2a00640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00c8004800 0x7f00c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e2a00640 1 -- 172.21.15.67:0/2896668048 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c8003520 con 0x7f00c80039e0 2024-04-17T08:51:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.927+0000 7f00e3400640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40cd6d0 0x7f00c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.928+0000 7f00e2a00640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00c8003e00 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f00d4007fd0 tx=0x7f00c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3d5078ffe9fb3a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.928+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4007c20 con 0x7f00c80039e0 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.928+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d4030070 con 0x7f00c80039e0 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.928+0000 7f00e1600640 1 -- 172.21.15.67:0/2896668048 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4005d30 con 0x7f00c80039e0 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 -- 172.21.15.67:0/2896668048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 msgr2=0x7f00c8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 --2- 172.21.15.67:0/2896668048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00c8003e00 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f00d4007fd0 tx=0x7f00c8005830 comp rx=0 tx=0).stop 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3400640 1 -- 172.21.15.67:0/2896668048 reap_dead start 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 -- 172.21.15.67:0/2896668048 shutdown_connections 2024-04-17T08:51:27.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 -- 172.21.15.67:0/2896668048 >> 172.21.15.67:0/2896668048 conn(0x7f00e406b8c0 msgr2=0x7f00e406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 -- 172.21.15.67:0/2896668048 shutdown_connections 2024-04-17T08:51:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.929+0000 7f00e3e00640 1 -- 172.21.15.67:0/2896668048 wait complete. 2024-04-17T08:51:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.931+0000 7f00e3e00640 1 Processor -- start 2024-04-17T08:51:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.931+0000 7f00e3e00640 1 -- start start 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 0x7f00e4141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40b1870 0x7f00e40b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00e4160a70 con 0x7f00c80039e0 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00e40cee30 con 0x7f00e4141840 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00e4160bf0 con 0x7f00e40b1870 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54436/0 (socket says 172.21.15.67:54436) 2024-04-17T08:51:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.932+0000 7f00e2a00640 1 -- 172.21.15.67:0/3080097159 learned_addr learned my addr 172.21.15.67:0/3080097159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2000640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 0x7f00e4141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 -- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40b1870 msgr2=0x7f00e40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40b1870 0x7f00e40b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 -- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 msgr2=0x7f00e4141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 0x7f00e4141cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 -- 172.21.15.67:0/3080097159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e40b2e40 con 0x7f00c80039e0 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2000640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 0x7f00e4141cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e2a00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f00d400c1a0 tx=0x7f00e41611d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4001b60 con 0x7f00c80039e0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d402cbb0 con 0x7f00c80039e0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4007070 con 0x7f00c80039e0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.933+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e40b25a0 con 0x7f00c80039e0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.934+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e40b2b60 con 0x7f00c80039e0 2024-04-17T08:51:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.935+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e4142200 con 0x7f00c80039e0 2024-04-17T08:51:27.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.936+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f00d4048020 con 0x7f00c80039e0 2024-04-17T08:51:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.937+0000 7f00db400640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 0x7f00c406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.937+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 6738+0+0 (secure 0 0 0) 0x7f00d40ad770 con 0x7f00c80039e0 2024-04-17T08:51:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.937+0000 7f00e2000640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 0x7f00c406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.938+0000 7f00e2000640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 0x7f00c406f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00e40b1510 tx=0x7f00d0008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:27.941+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f00d407b420 con 0x7f00c80039e0 2024-04-17T08:51:28.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.140+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f00e4068a10 con 0x7f00c80039e0 2024-04-17T08:51:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.863+0000 7f00db400640 1 -- 172.21.15.67:0/3080097159 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v241) ==== 190+0+0 (secure 0 0 0) 0x7f00d407ad20 con 0x7f00c80039e0 2024-04-17T08:51:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_metadata' removed 2024-04-17T08:51:28.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 msgr2=0x7f00c406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:28.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 0x7f00c406f020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00e40b1510 tx=0x7f00d0008040 comp rx=0 tx=0).stop 2024-04-17T08:51:28.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 msgr2=0x7f00e413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f00d400c1a0 tx=0x7f00e41611d0 comp rx=0 tx=0).stop 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 shutdown_connections 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f00c406cb60 0x7f00c406f020 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00e40b1510 tx=0x7f00d0008040 comp rx=0 tx=0).stop 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00e40b1870 0x7f00e40b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00e4141840 0x7f00e4141cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 --2- 172.21.15.67:0/3080097159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c80039e0 0x7f00e413ee70 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:28.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.868+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 >> 172.21.15.67:0/3080097159 conn(0x7f00e406b8c0 msgr2=0x7f00e415ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:28.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.869+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 shutdown_connections 2024-04-17T08:51:28.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:28.870+0000 7f00e3e00640 1 -- 172.21.15.67:0/3080097159 wait complete. 2024-04-17T08:51:28.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-17T08:51:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.113+0000 7fa73fe00640 1 Processor -- start 2024-04-17T08:51:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.114+0000 7fa73fe00640 1 -- start start 2024-04-17T08:51:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.115+0000 7fa73fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa74015e840 0x7fa7400cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.115+0000 7fa73fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7400d1710 0x7fa7400d3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.115+0000 7fa73fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7400cf850 con 0x7fa7400b7980 2024-04-17T08:51:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.115+0000 7fa73fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa7400cfb30 con 0x7fa7400d1710 2024-04-17T08:51:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.115+0000 7fa73fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa7400cfe10 con 0x7fa74015e840 2024-04-17T08:51:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.116+0000 7fa73f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7400d1710 0x7fa7400d3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.116+0000 7fa73f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7400d1710 0x7fa7400d3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54450/0 (socket says 172.21.15.67:54450) 2024-04-17T08:51:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.116+0000 7fa73f400640 1 -- 172.21.15.67:0/1804532276 learned_addr learned my addr 172.21.15.67:0/1804532276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.116+0000 7fa736000640 1 --1- 172.21.15.67:0/1804532276 >> v1:172.21.15.67:6789/0 conn(0x7fa7400b7980 0x7fa7400d1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39778/0 (socket says 172.21.15.67:39778) 2024-04-17T08:51:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.116+0000 7fa73ea00640 1 --2- 172.21.15.67:0/1804532276 >> v2:172.21.15.187:3300/0 conn(0x7fa74015e840 0x7fa7400cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.117+0000 7fa73e000640 1 -- 172.21.15.67:0/1804532276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934310233 0 0) 0x7fa7400cf850 con 0x7fa7400b7980 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.117+0000 7fa73e000640 1 -- 172.21.15.67:0/1804532276 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa724003520 con 0x7fa7400b7980 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.118+0000 7fa73f400640 1 -- 172.21.15.67:0/1804532276 >> v2:172.21.15.187:3300/0 conn(0x7fa74015e840 msgr2=0x7fa7400cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.118+0000 7fa73f400640 1 --2- 172.21.15.67:0/1804532276 >> v2:172.21.15.187:3300/0 conn(0x7fa74015e840 0x7fa7400cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.118+0000 7fa73f400640 1 -- 172.21.15.67:0/1804532276 >> v1:172.21.15.67:6789/0 conn(0x7fa7400b7980 legacy=0x7fa7400d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.118+0000 7fa73f400640 1 -- 172.21.15.67:0/1804532276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7400d2870 con 0x7fa7400d1710 2024-04-17T08:51:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.119+0000 7fa73f400640 1 --2- 172.21.15.67:0/1804532276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400d1710 0x7fa7400d3070 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fa74008ce90 tx=0x7fa73802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4241d930d958b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.120+0000 7fa73e000640 1 -- 172.21.15.67:0/1804532276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73802ca80 con 0x7fa7400d1710 2024-04-17T08:51:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.120+0000 7fa73e000640 1 -- 172.21.15.67:0/1804532276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73802cc20 con 0x7fa7400d1710 2024-04-17T08:51:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.120+0000 7fa73e000640 1 -- 172.21.15.67:0/1804532276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7380343f0 con 0x7fa7400d1710 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 -- 172.21.15.67:0/1804532276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400d1710 msgr2=0x7fa7400d3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1804532276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400d1710 0x7fa7400d3070 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fa74008ce90 tx=0x7fa73802bee0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 -- 172.21.15.67:0/1804532276 shutdown_connections 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1804532276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400d1710 0x7fa7400d3070 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1804532276 >> v2:172.21.15.187:3300/0 conn(0x7fa74015e840 0x7fa7400cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 -- 172.21.15.67:0/1804532276 >> 172.21.15.67:0/1804532276 conn(0x7fa74006cae0 msgr2=0x7fa74006cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 -- 172.21.15.67:0/1804532276 shutdown_connections 2024-04-17T08:51:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.121+0000 7fa73fe00640 1 -- 172.21.15.67:0/1804532276 wait complete. 2024-04-17T08:51:29.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.123+0000 7fa73fe00640 1 Processor -- start 2024-04-17T08:51:29.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.123+0000 7fa73fe00640 1 -- start start 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 0x7fa7400cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa74015e840 0x7fa7400cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa74015ec20 con 0x7fa7400b7980 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa7400cf4f0 con 0x7fa7400d1710 2024-04-17T08:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa74015eda0 con 0x7fa74015e840 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54454/0 (socket says 172.21.15.67:54454) 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.124+0000 7fa73ea00640 1 -- 172.21.15.67:0/1136189297 learned_addr learned my addr 172.21.15.67:0/1136189297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa736000640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 0x7fa7400cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 -- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa74015e840 msgr2=0x7fa7400cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa74015e840 0x7fa7400cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 -- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 msgr2=0x7fa7400cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 0x7fa7400cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 -- 172.21.15.67:0/1136189297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7400cd640 con 0x7fa7400b7980 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa736000640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 0x7fa7400cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73ea00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e870 tx=0x7fa73000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:29.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730017070 con 0x7fa7400b7980 2024-04-17T08:51:29.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73000c8c0 con 0x7fa7400b7980 2024-04-17T08:51:29.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000ccb0 con 0x7fa7400b7980 2024-04-17T08:51:29.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.125+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa740160c40 con 0x7fa7400b7980 2024-04-17T08:51:29.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.126+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa740160ee0 con 0x7fa7400b7980 2024-04-17T08:51:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.127+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa740068a10 con 0x7fa7400b7980 2024-04-17T08:51:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.128+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fa73001a070 con 0x7fa7400b7980 2024-04-17T08:51:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.129+0000 7fa737400640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 0x7fa71c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.129+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(241..241 src has 1..241) ==== 6294+0+0 (secure 0 0 0) 0x7fa73008c120 con 0x7fa7400b7980 2024-04-17T08:51:29.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.129+0000 7fa736000640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 0x7fa71c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.131+0000 7fa736000640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 0x7fa71c06eff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa7400cea50 tx=0x7fa728009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.133+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa7400cd640 con 0x7fa7400b7980 2024-04-17T08:51:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.341+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa7400d2420 con 0x7fa7400b7980 2024-04-17T08:51:29.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.867+0000 7fa737400640 1 -- 172.21.15.67:0/1136189297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v242) ==== 178+0+0 (secure 0 0 0) 0x7fa730059f00 con 0x7fa7400b7980 2024-04-17T08:51:29.866 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_data' removed 2024-04-17T08:51:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.871+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 msgr2=0x7fa71c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.871+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 0x7fa71c06eff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa7400cea50 tx=0x7fa728009040 comp rx=0 tx=0).stop 2024-04-17T08:51:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.871+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 msgr2=0x7fa7400ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.871+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fa73000e870 tx=0x7fa73000ed30 comp rx=0 tx=0).stop 2024-04-17T08:51:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 shutdown_connections 2024-04-17T08:51:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa71c06cb30 0x7fa71c06eff0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa74015e840 0x7fa7400cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7400d1710 0x7fa7400cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 --2- 172.21.15.67:0/1136189297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7400b7980 0x7fa7400ce210 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.873+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 >> 172.21.15.67:0/1136189297 conn(0x7fa74006cae0 msgr2=0x7fa7400d4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.874+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 shutdown_connections 2024-04-17T08:51:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:29.874+0000 7fa73fe00640 1 -- 172.21.15.67:0/1136189297 wait complete. 2024-04-17T08:51:29.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:51:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.106+0000 7f2caa200640 1 Processor -- start 2024-04-17T08:51:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.106+0000 7f2caa200640 1 -- start start 2024-04-17T08:51:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.107+0000 7f2caa200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ca415b6b0 0x7f2ca406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2caa200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca415ff40 con 0x7f2ca406f800 2024-04-17T08:51:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2caa200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca406df40 con 0x7f2ca40cfc10 2024-04-17T08:51:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2caa200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca406ed00 con 0x7f2ca415b6b0 2024-04-17T08:51:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2ca8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ca415b6b0 0x7f2ca406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2ca8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ca415b6b0 0x7f2ca406b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54464/0 (socket says 172.21.15.67:54464) 2024-04-17T08:51:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2ca8e00640 1 -- 172.21.15.67:0/3046873334 learned_addr learned my addr 172.21.15.67:0/3046873334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2ca3e00640 1 --1- 172.21.15.67:0/3046873334 >> v1:172.21.15.67:6789/0 conn(0x7f2ca406f800 0x7f2ca415f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39780/0 (socket says 172.21.15.67:39780) 2024-04-17T08:51:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.108+0000 7f2ca8e00640 1 -- 172.21.15.67:0/3046873334 >> v1:172.21.15.187:6789/0 conn(0x7f2ca40cfc10 legacy=0x7f2ca415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.109+0000 7f2ca8e00640 1 -- 172.21.15.67:0/3046873334 >> v1:172.21.15.67:6789/0 conn(0x7f2ca406f800 legacy=0x7f2ca415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.109+0000 7f2ca8e00640 1 -- 172.21.15.67:0/3046873334 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca406e220 con 0x7f2ca415b6b0 2024-04-17T08:51:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.109+0000 7f2ca3400640 1 -- 172.21.15.67:0/3046873334 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135103035 0 0) 0x7f2ca415ff40 con 0x7f2ca406f800 2024-04-17T08:51:30.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.110+0000 7f2ca8e00640 1 --2- 172.21.15.67:0/3046873334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca415b6b0 0x7f2ca406b8c0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009870 tx=0x7f2c9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6eed61378999b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.110+0000 7f2ca3400640 1 -- 172.21.15.67:0/3046873334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98035020 con 0x7f2ca415b6b0 2024-04-17T08:51:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.110+0000 7f2ca3400640 1 -- 172.21.15.67:0/3046873334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98031870 con 0x7f2ca415b6b0 2024-04-17T08:51:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.110+0000 7f2ca3400640 1 -- 172.21.15.67:0/3046873334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98031c40 con 0x7f2ca415b6b0 2024-04-17T08:51:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 -- 172.21.15.67:0/3046873334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca415b6b0 msgr2=0x7f2ca406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 --2- 172.21.15.67:0/3046873334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca415b6b0 0x7f2ca406b8c0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009870 tx=0x7f2c9802b910 comp rx=0 tx=0).stop 2024-04-17T08:51:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 -- 172.21.15.67:0/3046873334 shutdown_connections 2024-04-17T08:51:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 --2- 172.21.15.67:0/3046873334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca415b6b0 0x7f2ca406b8c0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 -- 172.21.15.67:0/3046873334 >> 172.21.15.67:0/3046873334 conn(0x7f2ca40b2190 msgr2=0x7f2ca40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 -- 172.21.15.67:0/3046873334 shutdown_connections 2024-04-17T08:51:30.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.111+0000 7f2caa200640 1 -- 172.21.15.67:0/3046873334 wait complete. 2024-04-17T08:51:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.113+0000 7f2caa200640 1 Processor -- start 2024-04-17T08:51:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.113+0000 7f2caa200640 1 -- start start 2024-04-17T08:51:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 0x7f2ca415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ca415b6b0 0x7f2ca415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca40afd60 con 0x7f2ca40cfc10 2024-04-17T08:51:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca406e870 con 0x7f2ca406f800 2024-04-17T08:51:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2caa200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca40afee0 con 0x7f2ca415b6b0 2024-04-17T08:51:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54466/0 (socket says 172.21.15.67:54466) 2024-04-17T08:51:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca3e00640 1 -- 172.21.15.67:0/2008515331 learned_addr learned my addr 172.21.15.67:0/2008515331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca8e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 0x7f2ca415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca3e00640 1 -- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ca415b6b0 msgr2=0x7f2ca415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca9800640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ca415b6b0 0x7f2ca415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.114+0000 7f2ca3e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ca415b6b0 0x7f2ca415a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca3e00640 1 -- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 msgr2=0x7f2ca415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca3e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 0x7f2ca415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca3e00640 1 -- 172.21.15.67:0/2008515331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca415ace0 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca8e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 0x7f2ca415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca3e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000c4d0 tx=0x7f2c9000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90018070 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c90007500 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90004a20 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca415c100 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.115+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca415c6c0 con 0x7f2ca40cfc10 2024-04-17T08:51:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.117+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca4068a10 con 0x7f2ca40cfc10 2024-04-17T08:51:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.118+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f2c9001b070 con 0x7f2ca40cfc10 2024-04-17T08:51:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.119+0000 7f2ca0c00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 0x7f2c8406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.119+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f2c9008b2c0 con 0x7f2ca40cfc10 2024-04-17T08:51:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.120+0000 7f2ca8e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 0x7f2c8406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.121+0000 7f2ca8e00640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 0x7f2c8406eff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c98036000 tx=0x7f2c9803c040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.124+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c90059370 con 0x7f2ca40cfc10 2024-04-17T08:51:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.331+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f2ca406ee80 con 0x7f2ca40cfc10 2024-04-17T08:51:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.338+0000 7f2ca0c00640 1 -- 172.21.15.67:0/2008515331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) ==== 115+0+0 (secure 0 0 0) 0x7f2c9005f020 con 0x7f2ca40cfc10 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.340+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 msgr2=0x7f2c8406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 0x7f2c8406eff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c98036000 tx=0x7f2c9803c040 comp rx=0 tx=0).stop 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 msgr2=0x7f2ca41491d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000c4d0 tx=0x7f2c9000c990 comp rx=0 tx=0).stop 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 shutdown_connections 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2c8406cb30 0x7f2c8406eff0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2ca415b6b0 0x7f2ca415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ca40cfc10 0x7f2ca41491d0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 --2- 172.21.15.67:0/2008515331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ca406f800 0x7f2ca415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 >> 172.21.15.67:0/2008515331 conn(0x7f2ca40b2190 msgr2=0x7f2ca40ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 shutdown_connections 2024-04-17T08:51:30.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.341+0000 7f2caa200640 1 -- 172.21.15.67:0/2008515331 wait complete. 2024-04-17T08:51:30.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:51:30.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.584+0000 7f0b18e00640 1 Processor -- start 2024-04-17T08:51:30.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.584+0000 7f0b18e00640 1 -- start start 2024-04-17T08:51:30.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.586+0000 7f0b18e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b14071ea0 0x7f0b140ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.586+0000 7f0b18e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b140cea70 con 0x7f0b14145ac0 2024-04-17T08:51:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.586+0000 7f0b18e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b140ced50 con 0x7f0b14072d80 2024-04-17T08:51:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.586+0000 7f0b18e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0b140cf030 con 0x7f0b14071ea0 2024-04-17T08:51:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.589+0000 7f0b11e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0b14145ac0 0x7f0b14072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39792/0 (socket says 172.21.15.67:39792) 2024-04-17T08:51:30.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.589+0000 7f0b11e00640 1 -- 172.21.15.67:0/2906696050 learned_addr learned my addr 172.21.15.67:0/2906696050 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.589+0000 7f0b12800640 1 --2- 172.21.15.67:0/2906696050 >> v2:172.21.15.67:3300/0 conn(0x7f0b14071ea0 0x7f0b140ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3770428299 0 0) 0x7f0b140cea70 con 0x7f0b14145ac0 2024-04-17T08:51:30.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af8003520 con 0x7f0b14145ac0 2024-04-17T08:51:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b12800640 1 -- 172.21.15.67:0/2906696050 >> v1:172.21.15.187:6789/0 conn(0x7f0b14072d80 legacy=0x7f0b140cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b12800640 1 -- 172.21.15.67:0/2906696050 >> v1:172.21.15.67:6789/0 conn(0x7f0b14145ac0 legacy=0x7f0b14072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b12800640 1 -- 172.21.15.67:0/2906696050 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b140cf310 con 0x7f0b14071ea0 2024-04-17T08:51:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.590+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982536886 0 0) 0x7f0af8003520 con 0x7f0b14145ac0 2024-04-17T08:51:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.592+0000 7f0b12800640 1 --2- 172.21.15.67:0/2906696050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14071ea0 0x7f0b140ce3b0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009d90 tx=0x7f0b0c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c660067f7b187b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.592+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c039070 con 0x7f0b14071ea0 2024-04-17T08:51:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.592+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0c02c8b0 con 0x7f0b14071ea0 2024-04-17T08:51:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.592+0000 7f0b10a00640 1 -- 172.21.15.67:0/2906696050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04001670 con 0x7f0b14071ea0 2024-04-17T08:51:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 -- 172.21.15.67:0/2906696050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14071ea0 msgr2=0x7f0b140ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 --2- 172.21.15.67:0/2906696050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14071ea0 0x7f0b140ce3b0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009d90 tx=0x7f0b0c02bd80 comp rx=0 tx=0).stop 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 -- 172.21.15.67:0/2906696050 shutdown_connections 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 --2- 172.21.15.67:0/2906696050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14071ea0 0x7f0b140ce3b0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 -- 172.21.15.67:0/2906696050 >> 172.21.15.67:0/2906696050 conn(0x7f0b14067ee0 msgr2=0x7f0b1406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 -- 172.21.15.67:0/2906696050 shutdown_connections 2024-04-17T08:51:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.593+0000 7f0b18e00640 1 -- 172.21.15.67:0/2906696050 wait complete. 2024-04-17T08:51:30.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.596+0000 7f0b18e00640 1 Processor -- start 2024-04-17T08:51:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.597+0000 7f0b18e00640 1 -- start start 2024-04-17T08:51:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b14071ea0 0x7f0b14139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b14145ac0 0x7f0b1406f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b140d0220 con 0x7f0b14072d80 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14145ea0 con 0x7f0b14071ea0 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b18e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b140d03a0 con 0x7f0b14145ac0 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b11400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b11400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54496/0 (socket says 172.21.15.67:54496) 2024-04-17T08:51:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.598+0000 7f0b11400640 1 -- 172.21.15.67:0/153806790 learned_addr learned my addr 172.21.15.67:0/153806790 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b14071ea0 0x7f0b14139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 -- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b14145ac0 msgr2=0x7f0b1406f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b14145ac0 0x7f0b1406f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 -- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b14071ea0 msgr2=0x7f0b14139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b14071ea0 0x7f0b14139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 -- 172.21.15.67:0/153806790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1415dd90 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b11400640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c460 tx=0x7f0b0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.599+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00016020 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.600+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b00012870 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.600+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1415e070 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.600+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00012c60 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.600+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1415e630 con 0x7f0b14072d80 2024-04-17T08:51:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.601+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b14068ca0 con 0x7f0b14072d80 2024-04-17T08:51:30.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.602+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f0b00019020 con 0x7f0b14072d80 2024-04-17T08:51:30.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.603+0000 7f0b0a000640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 0x7f0af406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:30.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.603+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f0b0008c0d0 con 0x7f0b14072d80 2024-04-17T08:51:30.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.604+0000 7f0b11e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 0x7f0af406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:30.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.606+0000 7f0b11e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 0x7f0af406eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b040011e0 tx=0x7f0b04009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:30.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.607+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0b00020020 con 0x7f0b14072d80 2024-04-17T08:51:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.812+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0b140cc6c0 con 0x7f0b14072d80 2024-04-17T08:51:30.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.820+0000 7f0b0a000640 1 -- 172.21.15.67:0/153806790 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v80) ==== 104+0+0 (secure 0 0 0) 0x7f0b000182c0 con 0x7f0b14072d80 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 msgr2=0x7f0af406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 0x7f0af406eff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b040011e0 tx=0x7f0b04009040 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 msgr2=0x7f0b1406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c460 tx=0x7f0b0000c920 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 shutdown_connections 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f0af406cb30 0x7f0af406eff0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b14145ac0 0x7f0b1406f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b14072d80 0x7f0b1406ee90 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 --2- 172.21.15.67:0/153806790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b14071ea0 0x7f0b14139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.822+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 >> 172.21.15.67:0/153806790 conn(0x7f0b14067ee0 msgr2=0x7f0b1406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.823+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 shutdown_connections 2024-04-17T08:51:30.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:30.823+0000 7f0b18e00640 1 -- 172.21.15.67:0/153806790 wait complete. 2024-04-17T08:51:30.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:51:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.045+0000 7f4308600640 1 Processor -- start 2024-04-17T08:51:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.045+0000 7f4308600640 1 -- start start 2024-04-17T08:51:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4308600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f430009b170 0x7f4300169bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4308600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43000d1520 con 0x7f430006ce00 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4308600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43000d1800 con 0x7f430006daf0 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4308600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f43000d1ae0 con 0x7f430009b170 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4306600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f430009b170 0x7f4300169bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.047+0000 7f4305200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f430006ce00 0x7f430009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39806/0 (socket says 172.21.15.67:39806) 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.048+0000 7f4305200640 1 -- 172.21.15.67:0/1575664509 learned_addr learned my addr 172.21.15.67:0/1575664509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.048+0000 7f4306600640 1 -- 172.21.15.67:0/1575664509 >> v1:172.21.15.187:6789/0 conn(0x7f430006daf0 legacy=0x7f43000d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.048+0000 7f4306600640 1 -- 172.21.15.67:0/1575664509 >> v1:172.21.15.67:6789/0 conn(0x7f430006ce00 legacy=0x7f430009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.048+0000 7f4306600640 1 -- 172.21.15.67:0/1575664509 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43000d1dc0 con 0x7f430009b170 2024-04-17T08:51:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.049+0000 7f4306600640 1 --2- 172.21.15.67:0/1575664509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430009b170 0x7f4300169bd0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097a0 tx=0x7f42f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b445235b7d8e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.049+0000 7f42ffe00640 1 -- 172.21.15.67:0/1575664509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8035020 con 0x7f430009b170 2024-04-17T08:51:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.049+0000 7f42ffe00640 1 -- 172.21.15.67:0/1575664509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f8031830 con 0x7f430009b170 2024-04-17T08:51:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.049+0000 7f42ffe00640 1 -- 172.21.15.67:0/1575664509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8031c00 con 0x7f430009b170 2024-04-17T08:51:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 -- 172.21.15.67:0/1575664509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430009b170 msgr2=0x7f4300169bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 --2- 172.21.15.67:0/1575664509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430009b170 0x7f4300169bd0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f42f80097a0 tx=0x7f42f802b8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 -- 172.21.15.67:0/1575664509 shutdown_connections 2024-04-17T08:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 --2- 172.21.15.67:0/1575664509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430009b170 0x7f4300169bd0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 -- 172.21.15.67:0/1575664509 >> 172.21.15.67:0/1575664509 conn(0x7f43000b41a0 msgr2=0x7f43000b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 -- 172.21.15.67:0/1575664509 shutdown_connections 2024-04-17T08:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.050+0000 7f4308600640 1 -- 172.21.15.67:0/1575664509 wait complete. 2024-04-17T08:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.052+0000 7f4308600640 1 Processor -- start 2024-04-17T08:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.052+0000 7f4308600640 1 -- start start 2024-04-17T08:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f430006ce00 0x7f43000b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f430009b170 0x7f43000cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f430006ded0 con 0x7f430006daf0 2024-04-17T08:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43000d1230 con 0x7f430006ce00 2024-04-17T08:51:31.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4308600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f430006e050 con 0x7f430009b170 2024-04-17T08:51:31.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4305200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4305200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54508/0 (socket says 172.21.15.67:54508) 2024-04-17T08:51:31.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4305200640 1 -- 172.21.15.67:0/3136597232 learned_addr learned my addr 172.21.15.67:0/3136597232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.053+0000 7f4305c00640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f430006ce00 0x7f43000b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 -- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f430009b170 msgr2=0x7f43000cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f430009b170 0x7f43000cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 -- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f430006ce00 msgr2=0x7f43000b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f430006ce00 0x7f43000b67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 -- 172.21.15.67:0/3136597232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43001588f0 con 0x7f430006daf0 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4305200640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c440 tx=0x7f42ec001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec017070 con 0x7f430006daf0 2024-04-17T08:51:31.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec005ce0 con 0x7f430006daf0 2024-04-17T08:51:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0072f0 con 0x7f430006daf0 2024-04-17T08:51:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43000ce330 con 0x7f430006daf0 2024-04-17T08:51:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.054+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43000ce8f0 con 0x7f430006daf0 2024-04-17T08:51:31.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.055+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4300068ca0 con 0x7f430006daf0 2024-04-17T08:51:31.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.056+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f42ec01a070 con 0x7f430006daf0 2024-04-17T08:51:31.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.057+0000 7f42fd600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 0x7f42e006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.057+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f42ec08b480 con 0x7f430006daf0 2024-04-17T08:51:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.058+0000 7f4305c00640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 0x7f42e006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.059+0000 7f4305c00640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 0x7f42e006f040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42f4007fd0 tx=0x7f42f4009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.062+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f43001588f0 con 0x7f430006daf0 2024-04-17T08:51:31.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.268+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f43000b5040 con 0x7f430006daf0 2024-04-17T08:51:31.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.275+0000 7f42fd600640 1 -- 172.21.15.67:0/3136597232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v81) ==== 120+0+0 (secure 0 0 0) 0x7f42ec059480 con 0x7f430006daf0 2024-04-17T08:51:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 msgr2=0x7f42e006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 0x7f42e006f040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f42f4007fd0 tx=0x7f42f4009040 comp rx=0 tx=0).stop 2024-04-17T08:51:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 msgr2=0x7f43000ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c440 tx=0x7f42ec001e70 comp rx=0 tx=0).stop 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 shutdown_connections 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f42e006cb80 0x7f42e006f040 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f430009b170 0x7f43000cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f430006daf0 0x7f43000ccce0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 --2- 172.21.15.67:0/3136597232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f430006ce00 0x7f43000b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 >> 172.21.15.67:0/3136597232 conn(0x7f43000b41a0 msgr2=0x7f43000b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 shutdown_connections 2024-04-17T08:51:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.278+0000 7f4308600640 1 -- 172.21.15.67:0/3136597232 wait complete. 2024-04-17T08:51:31.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-04-17T08:51:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.517+0000 7fbb32a00640 1 Processor -- start 2024-04-17T08:51:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.517+0000 7fbb32a00640 1 -- start start 2024-04-17T08:51:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.518+0000 7fbb32a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb2c0cd080 0x7fbb2c0cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.518+0000 7fbb32a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c06f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.518+0000 7fbb32a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb2c072cc0 con 0x7fbb2c144da0 2024-04-17T08:51:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.518+0000 7fbb32a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbb2c0cda20 con 0x7fbb2c0cd080 2024-04-17T08:51:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.518+0000 7fbb32a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb2c0cdc40 con 0x7fbb2c06ee90 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.519+0000 7fbb31600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb2c144da0 0x7fbb2c0b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39810/0 (socket says 172.21.15.67:39810) 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.519+0000 7fbb31600640 1 -- 172.21.15.67:0/2873051142 learned_addr learned my addr 172.21.15.67:0/2873051142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.519+0000 7fbb30c00640 1 --2- 172.21.15.67:0/2873051142 >> v2:172.21.15.67:3300/0 conn(0x7fbb2c0cd080 0x7fbb2c0cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.519+0000 7fbb32000640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c06f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb2be00640 1 -- 172.21.15.67:0/2873051142 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346952182 0 0) 0x7fbb2c072cc0 con 0x7fbb2c144da0 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb2be00640 1 -- 172.21.15.67:0/2873051142 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb14003520 con 0x7fbb2c144da0 2024-04-17T08:51:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb30c00640 1 -- 172.21.15.67:0/2873051142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 msgr2=0x7fbb2c06f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb30c00640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c06f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb30c00640 1 -- 172.21.15.67:0/2873051142 >> v1:172.21.15.67:6789/0 conn(0x7fbb2c144da0 legacy=0x7fbb2c0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.520+0000 7fbb30c00640 1 -- 172.21.15.67:0/2873051142 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c0b3b50 con 0x7fbb2c0cd080 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.521+0000 7fbb30c00640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c0cd4e0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fbb180097a0 tx=0x7fbb1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa1dcba46bf0372 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.521+0000 7fbb2be00640 1 -- 172.21.15.67:0/2873051142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb180315a0 con 0x7fbb2c0cd080 2024-04-17T08:51:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.521+0000 7fbb2be00640 1 -- 172.21.15.67:0/2873051142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb18037070 con 0x7fbb2c0cd080 2024-04-17T08:51:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.521+0000 7fbb2be00640 1 -- 172.21.15.67:0/2873051142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18031960 con 0x7fbb2c0cd080 2024-04-17T08:51:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 -- 172.21.15.67:0/2873051142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c0cd080 msgr2=0x7fbb2c0cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c0cd4e0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fbb180097a0 tx=0x7fbb1802b840 comp rx=0 tx=0).stop 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 -- 172.21.15.67:0/2873051142 shutdown_connections 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c06f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 --2- 172.21.15.67:0/2873051142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c0cd4e0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 -- 172.21.15.67:0/2873051142 >> 172.21.15.67:0/2873051142 conn(0x7fbb2c0b5f70 msgr2=0x7fbb2c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 -- 172.21.15.67:0/2873051142 shutdown_connections 2024-04-17T08:51:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.522+0000 7fbb32a00640 1 -- 172.21.15.67:0/2873051142 wait complete. 2024-04-17T08:51:31.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.524+0000 7fbb32a00640 1 Processor -- start 2024-04-17T08:51:31.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.524+0000 7fbb32a00640 1 -- start start 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c1378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c13cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c0b43f0 con 0x7fbb2c144da0 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c0b4110 con 0x7fbb2c0cd080 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb2c15baf0 con 0x7fbb2c06ee90 2024-04-17T08:51:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54524/0 (socket says 172.21.15.67:54524) 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.525+0000 7fbb32000640 1 -- 172.21.15.67:0/644430271 learned_addr learned my addr 172.21.15.67:0/644430271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb31600640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c1378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 -- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 msgr2=0x7fbb2c1378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c1378a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 -- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb2c0cd080 msgr2=0x7fbb2c13cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c13cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 -- 172.21.15.67:0/644430271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c137de0 con 0x7fbb2c144da0 2024-04-17T08:51:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32000640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fbb24007e40 tx=0x7fbb2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24014620 con 0x7fbb2c144da0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb24019070 con 0x7fbb2c144da0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24014a00 con 0x7fbb2c144da0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2c138c50 con 0x7fbb2c144da0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.526+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2c139210 con 0x7fbb2c144da0 2024-04-17T08:51:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.527+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2c0b32d0 con 0x7fbb2c144da0 2024-04-17T08:51:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.528+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fbb24014ba0 con 0x7fbb2c144da0 2024-04-17T08:51:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.529+0000 7fbb29600640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 0x7fbb0c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:31.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.529+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7fbb2408c830 con 0x7fbb2c144da0 2024-04-17T08:51:31.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.530+0000 7fbb31600640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 0x7fbb0c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:31.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.531+0000 7fbb31600640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 0x7fbb0c06eff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb20000f60 tx=0x7fbb20009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.534+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb2c0b32d0 con 0x7fbb2c144da0 2024-04-17T08:51:31.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.811+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7fbb2c068a10 con 0x7fbb2c144da0 2024-04-17T08:51:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.890+0000 7fbb29600640 1 -- 172.21.15.67:0/644430271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3226) ==== 186+0+0 (secure 0 0 0) 0x7fbb2405a990 con 0x7fbb2c144da0 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.893+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 msgr2=0x7fbb0c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.893+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 0x7fbb0c06eff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbb20000f60 tx=0x7fbb20009040 comp rx=0 tx=0).stop 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.893+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 msgr2=0x7fbb2c157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.893+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fbb24007e40 tx=0x7fbb2400e4b0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 shutdown_connections 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbb0c06cb30 0x7fbb0c06eff0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2c144da0 0x7fbb2c157780 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb2c0cd080 0x7fbb2c13cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 --2- 172.21.15.67:0/644430271 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb2c06ee90 0x7fbb2c1378a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 >> 172.21.15.67:0/644430271 conn(0x7fbb2c0b5f70 msgr2=0x7fbb2c0cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 shutdown_connections 2024-04-17T08:51:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:31.894+0000 7fbb32a00640 1 -- 172.21.15.67:0/644430271 wait complete. 2024-04-17T08:51:31.916 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-04-17T08:51:31.916 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-04-17T08:51:31.916 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-04-17T08:51:31.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.146+0000 7f8782000640 1 Processor -- start 2024-04-17T08:51:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.146+0000 7f8782000640 1 -- start start 2024-04-17T08:51:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.148+0000 7f8782000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.148+0000 7f8782000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f877c06bfd0 0x7f877c1634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.148+0000 7f8782000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f877c06ead0 con 0x7f877c0b3490 2024-04-17T08:51:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.148+0000 7f8782000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f877c06f6e0 con 0x7f877c06bfd0 2024-04-17T08:51:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.148+0000 7f8782000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f877c06edb0 con 0x7f877c15d530 2024-04-17T08:51:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.149+0000 7f8780c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.149+0000 7f8780c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:53324/0 (socket says 172.21.15.67:53324) 2024-04-17T08:51:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.149+0000 7f8780c00640 1 -- 172.21.15.67:0/802370717 learned_addr learned my addr 172.21.15.67:0/802370717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.149+0000 7f8781600640 1 --2- 172.21.15.67:0/802370717 >> v2:172.21.15.187:3300/0 conn(0x7f877c06bfd0 0x7f877c1634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.150+0000 7f8780c00640 1 -- 172.21.15.67:0/802370717 >> v2:172.21.15.187:3300/0 conn(0x7f877c06bfd0 msgr2=0x7f877c1634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.150+0000 7f8780c00640 1 --2- 172.21.15.67:0/802370717 >> v2:172.21.15.187:3300/0 conn(0x7f877c06bfd0 0x7f877c1634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.150+0000 7f8780c00640 1 -- 172.21.15.67:0/802370717 >> v1:172.21.15.187:6789/0 conn(0x7f877c0b3490 legacy=0x7f877c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.150+0000 7f8780c00640 1 -- 172.21.15.67:0/802370717 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877c06f090 con 0x7f877c15d530 2024-04-17T08:51:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.151+0000 7f8780c00640 1 --2- 172.21.15.67:0/802370717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8770009870 tx=0x7f877002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0f363e2327b905 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.152+0000 7f877b400640 1 -- 172.21.15.67:0/802370717 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770034070 con 0x7f877c15d530 2024-04-17T08:51:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.152+0000 7f877b400640 1 -- 172.21.15.67:0/802370717 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8770030840 con 0x7f877c15d530 2024-04-17T08:51:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.152+0000 7f877b400640 1 -- 172.21.15.67:0/802370717 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770030c10 con 0x7f877c15d530 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 -- 172.21.15.67:0/802370717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 msgr2=0x7f877c164480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 --2- 172.21.15.67:0/802370717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8770009870 tx=0x7f877002b980 comp rx=0 tx=0).stop 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 -- 172.21.15.67:0/802370717 shutdown_connections 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 --2- 172.21.15.67:0/802370717 >> v2:172.21.15.187:3300/0 conn(0x7f877c06bfd0 0x7f877c1634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 --2- 172.21.15.67:0/802370717 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c164480 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 -- 172.21.15.67:0/802370717 >> 172.21.15.67:0/802370717 conn(0x7f877c067d10 msgr2=0x7f877c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 -- 172.21.15.67:0/802370717 shutdown_connections 2024-04-17T08:51:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.153+0000 7f8782000640 1 -- 172.21.15.67:0/802370717 wait complete. 2024-04-17T08:51:32.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.155+0000 7f8782000640 1 Processor -- start 2024-04-17T08:51:32.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.156+0000 7f8782000640 1 -- start start 2024-04-17T08:51:32.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f877c06bfd0 0x7f877c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f877c06fee0 con 0x7f877c0b3490 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f877c06fd60 con 0x7f877c06bfd0 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f8782000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f877c0b3910 con 0x7f877c15d530 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f877be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f877be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54536/0 (socket says 172.21.15.67:54536) 2024-04-17T08:51:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.157+0000 7f877be00640 1 -- 172.21.15.67:0/1315710102 learned_addr learned my addr 172.21.15.67:0/1315710102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f8781600640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f8780c00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f877c06bfd0 0x7f877c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 -- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 msgr2=0x7f877c0cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c0cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 -- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f877c06bfd0 msgr2=0x7f877c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f877c06bfd0 0x7f877c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 -- 172.21.15.67:0/1315710102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877c0d03c0 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.158+0000 7f877be00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f876800c430 tx=0x7f876800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.159+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768017070 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.159+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87680052d0 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.159+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768003850 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.159+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877c1fc5e0 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.159+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877c1fcba0 con 0x7f877c0b3490 2024-04-17T08:51:32.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.160+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877c068a10 con 0x7f877c0b3490 2024-04-17T08:51:32.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.161+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f876801a070 con 0x7f877c0b3490 2024-04-17T08:51:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.162+0000 7f8778c00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 0x7f875c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.162+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f876808b3a0 con 0x7f877c0b3490 2024-04-17T08:51:32.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.163+0000 7f8780c00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 0x7f875c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.164+0000 7f8780c00640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 0x7f875c06f020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8770036000 tx=0x7f877003b040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.167+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f87680593f0 con 0x7f877c0b3490 2024-04-17T08:51:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.377+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f877c162040 con 0x7f877c0b3490 2024-04-17T08:51:32.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.379+0000 7f8778c00640 1 -- 172.21.15.67:0/1315710102 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+14359 (secure 0 0 0) 0x7f8768058cf0 con 0x7f877c0b3490 2024-04-17T08:51:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:32.378 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":242,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:29.863512+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6832/4206695657":"2024-04-18T08:51:26.675869+0000","172.21.15.187:0/3624623080":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/694267287":"2024-04-18T08:51:12.720872+0000","172.21.15.67:6835/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/2725922967":"2024-04-18T08:51:12.720872+0000","172.21.15.67:6833/4206695657":"2024-04-18T08:51:26.675869+0000","172.21.15.187:0/4122261893":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3592903800":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1026294602":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3517976885":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1963425026":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/1585815760":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/629863885":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.187:6833/3872755981":"2024-04-18T08:51:12.720872+0000","172.21.15.187:6832/150239855":"2024-04-18T08:51:01.354257+0000","172.21.15.67:6834/3533295594":"2024-04-18T08:51:18.656338+0000","172.21.15.187:0/665836648":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/3650355481":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/2617628331":"2024-04-18T08:51:12.720872+0000","172.21.15.187:0/3246022161":"2024-04-18T08:51:01.354257+0000","172.21.15.187:0/291755012":"2024-04-18T08:51:01.354257+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 msgr2=0x7f875c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 0x7f875c06f020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8770036000 tx=0x7f877003b040 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 msgr2=0x7f877c0d0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f876800c430 tx=0x7f876800c8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 shutdown_connections 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f875c06cb60 0x7f875c06f020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f877c15d530 0x7f877c0cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c0b3490 0x7f877c0d0ca0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 --2- 172.21.15.67:0/1315710102 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f877c06bfd0 0x7f877c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 >> 172.21.15.67:0/1315710102 conn(0x7f877c067d10 msgr2=0x7f877c161070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 shutdown_connections 2024-04-17T08:51:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.382+0000 7f8782000640 1 -- 172.21.15.67:0/1315710102 wait complete. 2024-04-17T08:51:32.400 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:32.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:32.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.626+0000 7f58a6200640 1 Processor -- start 2024-04-17T08:51:32.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.627+0000 7f58a6200640 1 -- start start 2024-04-17T08:51:32.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.628+0000 7f58a6200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 0x7f58a0160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.628+0000 7f58a6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58a00b4ac0 0x7f58a00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.628+0000 7f58a6200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a0160920 con 0x7f58a0146060 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.628+0000 7f58a6200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f58a00b5450 con 0x7f58a00b4ac0 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.628+0000 7f58a6200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f58a0160c00 con 0x7f58a0145b00 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58a00b4ac0 0x7f58a00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58a00b4ac0 0x7f58a00b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54548/0 (socket says 172.21.15.67:54548) 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 -- 172.21.15.67:0/808497945 learned_addr learned my addr 172.21.15.67:0/808497945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a4e00640 1 --2- 172.21.15.67:0/808497945 >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 0x7f58a0160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 -- 172.21.15.67:0/808497945 >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 msgr2=0x7f58a0160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 --2- 172.21.15.67:0/808497945 >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 0x7f58a0160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 -- 172.21.15.67:0/808497945 >> v1:172.21.15.187:6789/0 conn(0x7f58a0146060 legacy=0x7f58a015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a5800640 1 -- 172.21.15.67:0/808497945 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a0160ee0 con 0x7f58a00b4ac0 2024-04-17T08:51:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.629+0000 7f58a4e00640 1 --2- 172.21.15.67:0/808497945 >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 0x7f58a0160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:32.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.630+0000 7f58a5800640 1 --2- 172.21.15.67:0/808497945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00b4f10 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f58a0073580 tx=0x7f589802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14199e55eb6e1d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.631+0000 7f589f400640 1 -- 172.21.15.67:0/808497945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589802e070 con 0x7f58a00b4ac0 2024-04-17T08:51:32.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.631+0000 7f589f400640 1 -- 172.21.15.67:0/808497945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5898037070 con 0x7f58a00b4ac0 2024-04-17T08:51:32.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.631+0000 7f589f400640 1 -- 172.21.15.67:0/808497945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898031a00 con 0x7f58a00b4ac0 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 -- 172.21.15.67:0/808497945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 msgr2=0x7f58a00b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 --2- 172.21.15.67:0/808497945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00b4f10 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f58a0073580 tx=0x7f589802b880 comp rx=0 tx=0).stop 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 -- 172.21.15.67:0/808497945 shutdown_connections 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 --2- 172.21.15.67:0/808497945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00b4f10 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 --2- 172.21.15.67:0/808497945 >> v2:172.21.15.187:3300/0 conn(0x7f58a0145b00 0x7f58a0160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 -- 172.21.15.67:0/808497945 >> 172.21.15.67:0/808497945 conn(0x7f58a00b23c0 msgr2=0x7f58a00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 -- 172.21.15.67:0/808497945 shutdown_connections 2024-04-17T08:51:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.632+0000 7f58a6200640 1 -- 172.21.15.67:0/808497945 wait complete. 2024-04-17T08:51:32.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.634+0000 7f58a6200640 1 Processor -- start 2024-04-17T08:51:32.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.634+0000 7f58a6200640 1 -- start start 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58a0145b00 0x7f58a00cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58a0146060 0x7f58a00cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58a00cfba0 con 0x7f58a00b4ac0 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f58a00b3760 con 0x7f58a0146060 2024-04-17T08:51:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.635+0000 7f58a6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f58a00b3c80 con 0x7f58a0145b00 2024-04-17T08:51:32.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54562/0 (socket says 172.21.15.67:54562) 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 -- 172.21.15.67:0/913323409 learned_addr learned my addr 172.21.15.67:0/913323409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f589fe00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58a0145b00 0x7f58a00cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a5800640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58a0146060 0x7f58a00cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 -- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58a0145b00 msgr2=0x7f58a00cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58a0145b00 0x7f58a00cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 -- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58a0146060 msgr2=0x7f58a00cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58a0146060 0x7f58a00cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.636+0000 7f58a4e00640 1 -- 172.21.15.67:0/913323409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a00c5e50 con 0x7f58a00b4ac0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f58a4e00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f589000cf60 tx=0x7f589000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589001a070 con 0x7f58a00b4ac0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589000d840 con 0x7f58a00b4ac0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a00cbe70 con 0x7f58a00b4ac0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589000dc10 con 0x7f58a00b4ac0 2024-04-17T08:51:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.637+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a00c6110 con 0x7f58a00b4ac0 2024-04-17T08:51:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.639+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a00c6310 con 0x7f58a00b4ac0 2024-04-17T08:51:32.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.640+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f589001d070 con 0x7f58a00b4ac0 2024-04-17T08:51:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.641+0000 7f589cc00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 0x7f587c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.641+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f589008c7f0 con 0x7f58a00b4ac0 2024-04-17T08:51:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.641+0000 7f589fe00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 0x7f587c06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:32.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.643+0000 7f589fe00640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 0x7f587c06f040 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5888006fd0 tx=0x7f5888008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.645+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f58a00c6310 con 0x7f58a00b4ac0 2024-04-17T08:51:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.850+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f58a0068a10 con 0x7f58a00b4ac0 2024-04-17T08:51:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.850+0000 7f589cc00640 1 -- 172.21.15.67:0/913323409 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) ==== 93+0+10 (secure 0 0 0) 0x7f589005a2a0 con 0x7f58a00b4ac0 2024-04-17T08:51:32.849 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 msgr2=0x7f587c06f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 0x7f587c06f040 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5888006fd0 tx=0x7f5888008040 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 msgr2=0x7f58a00c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f589000cf60 tx=0x7f589000cf90 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 shutdown_connections 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f587c06cb80 0x7f587c06f040 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58a0146060 0x7f58a00cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58a0145b00 0x7f58a00cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 --2- 172.21.15.67:0/913323409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a00b4ac0 0x7f58a00c8790 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 >> 172.21.15.67:0/913323409 conn(0x7f58a00b23c0 msgr2=0x7f58a015e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 shutdown_connections 2024-04-17T08:51:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:32.853+0000 7f58a6200640 1 -- 172.21.15.67:0/913323409 wait complete. 2024-04-17T08:51:32.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-17T08:51:33.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.096+0000 7f0243000640 1 Processor -- start 2024-04-17T08:51:33.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.096+0000 7f0243000640 1 -- start start 2024-04-17T08:51:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.097+0000 7f0243000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f023c144da0 0x7f023c0b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0243000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c072cc0 con 0x7f023c06f5a0 2024-04-17T08:51:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0243000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c0cdc40 con 0x7f023c0cd080 2024-04-17T08:51:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0243000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f023c0b3b50 con 0x7f023c144da0 2024-04-17T08:51:33.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0242600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f023c06f5a0 0x7f023c15baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39818/0 (socket says 172.21.15.67:39818) 2024-04-17T08:51:33.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0242600640 1 -- 172.21.15.67:0/748049202 learned_addr learned my addr 172.21.15.67:0/748049202 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:33.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.098+0000 7f0241c00640 1 --2- 172.21.15.67:0/748049202 >> v2:172.21.15.187:3300/0 conn(0x7f023c144da0 0x7f023c0b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846209267 0 0) 0x7f023c072cc0 con 0x7f023c06f5a0 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0224003520 con 0x7f023c06f5a0 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2004863336 0 0) 0x7f0224003520 con 0x7f023c06f5a0 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 >> v2:172.21.15.187:3300/0 conn(0x7f023c144da0 msgr2=0x7f023c0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 --2- 172.21.15.67:0/748049202 >> v2:172.21.15.187:3300/0 conn(0x7f023c144da0 0x7f023c0b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 >> v1:172.21.15.187:6789/0 conn(0x7f023c0cd080 legacy=0x7f023c06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c0b3e30 con 0x7f023c06f5a0 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f02340019f0 con 0x7f023c06f5a0 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 >> v1:172.21.15.67:6789/0 conn(0x7f023c06f5a0 legacy=0x7f023c15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02240039b0 0x7f0224003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f0224004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02240051c0 0x7f022400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f023c0b3b50 con 0x7f0224004800 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f022400c060 con 0x7f02240039b0 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f022400c260 con 0x7f02240051c0 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.099+0000 7f0241200640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f0224004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241c00640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02240039b0 0x7f0224003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0242600640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02240051c0 0x7f022400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241200640 1 -- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02240051c0 msgr2=0x7f022400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241200640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02240051c0 0x7f022400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241200640 1 -- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02240039b0 msgr2=0x7f0224003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241200640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02240039b0 0x7f0224003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241200640 1 -- 172.21.15.67:0/748049202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0224003520 con 0x7f0224004800 2024-04-17T08:51:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0242600640 1 -- 172.21.15.67:0/748049202 reap_dead start 2024-04-17T08:51:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.100+0000 7f0241c00640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02240039b0 0x7f0224003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.101+0000 7f0241200640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f0224004c80 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f0228001750 tx=0x7f0224005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ec04e2debd0bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.101+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228001370 con 0x7f0224004800 2024-04-17T08:51:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.101+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02280052d0 con 0x7f0224004800 2024-04-17T08:51:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.101+0000 7f023be00640 1 -- 172.21.15.67:0/748049202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228003830 con 0x7f0224004800 2024-04-17T08:51:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 -- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 msgr2=0x7f0224004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f0224004c80 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f0228001750 tx=0x7f0224005830 comp rx=0 tx=0).stop 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 -- 172.21.15.67:0/748049202 shutdown_connections 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 --2- 172.21.15.67:0/748049202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f0224004c80 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 -- 172.21.15.67:0/748049202 >> 172.21.15.67:0/748049202 conn(0x7f023c0b5f70 msgr2=0x7f023c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 -- 172.21.15.67:0/748049202 shutdown_connections 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.102+0000 7f0243000640 1 -- 172.21.15.67:0/748049202 wait complete. 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.104+0000 7f0243000640 1 Processor -- start 2024-04-17T08:51:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.104+0000 7f0243000640 1 -- start start 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023c1f0390 0x7f023c15abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023c13f1b0 0x7f023c13f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f023c0cd4e0 con 0x7f0224004800 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f023c0b43f0 con 0x7f023c1f0390 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0243000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f023c0cd660 con 0x7f023c13f1b0 2024-04-17T08:51:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0241c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0241c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54582/0 (socket says 172.21.15.67:54582) 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.105+0000 7f0241c00640 1 -- 172.21.15.67:0/1683062075 learned_addr learned my addr 172.21.15.67:0/1683062075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0242600640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023c13f1b0 0x7f023c13f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241200640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023c1f0390 0x7f023c15abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 -- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023c13f1b0 msgr2=0x7f023c13f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023c13f1b0 0x7f023c13f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 -- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023c1f0390 msgr2=0x7f023c15abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023c1f0390 0x7f023c15abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 -- 172.21.15.67:0/1683062075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c1f0b70 con 0x7f0224004800 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0241c00640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f02300114e0 tx=0x7f02300119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230015870 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0230015a10 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230015e00 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.106+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c1f0e50 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.107+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c1f1410 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.108+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c068a10 con 0x7f0224004800 2024-04-17T08:51:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.109+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f023001d400 con 0x7f0224004800 2024-04-17T08:51:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.110+0000 7f0239600640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 0x7f021c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.110+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f023008c9b0 con 0x7f0224004800 2024-04-17T08:51:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.110+0000 7f0241200640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 0x7f021c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.112+0000 7f0241200640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 0x7f021c06f020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0228033000 tx=0x7f0228009030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.115+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0230019030 con 0x7f0224004800 2024-04-17T08:51:33.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.396+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7f023c0b3090 con 0x7f0224004800 2024-04-17T08:51:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.907+0000 7f0239600640 1 -- 172.21.15.67:0/1683062075 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3228) ==== 190+0+0 (secure 0 0 0) 0x7f023008c020 con 0x7f0224004800 2024-04-17T08:51:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.910+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 msgr2=0x7f021c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.910+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 0x7f021c06f020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0228033000 tx=0x7f0228009030 comp rx=0 tx=0).stop 2024-04-17T08:51:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.910+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 msgr2=0x7f023c0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.910+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f02300114e0 tx=0x7f02300119a0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 shutdown_connections 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f021c06cb60 0x7f021c06f020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023c13f1b0 0x7f023c13f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023c1f0390 0x7f023c15abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 --2- 172.21.15.67:0/1683062075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224004800 0x7f023c0b2060 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 >> 172.21.15.67:0/1683062075 conn(0x7f023c0b5f70 msgr2=0x7f023c142ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 shutdown_connections 2024-04-17T08:51:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:33.911+0000 7f0243000640 1 -- 172.21.15.67:0/1683062075 wait complete. 2024-04-17T08:51:33.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:51:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.164+0000 7f809a400640 1 Processor -- start 2024-04-17T08:51:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.165+0000 7f809a400640 1 -- start start 2024-04-17T08:51:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f809a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80940cd920 0x7f809406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f809a400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 0x7f809406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f809a400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8094146360 con 0x7f8094145f80 2024-04-17T08:51:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f809a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f80940727a0 con 0x7f80940cd920 2024-04-17T08:51:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f809a400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8094071e70 con 0x7f809406c0a0 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f8099000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8094145f80 0x7f80940cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40150/0 (socket says 172.21.15.67:40150) 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f8099000640 1 -- 172.21.15.67:0/903142834 learned_addr learned my addr 172.21.15.67:0/903142834 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.166+0000 7f8093e00640 1 --2- 172.21.15.67:0/903142834 >> v2:172.21.15.67:3300/0 conn(0x7f80940cd920 0x7f809406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8099a00640 1 --2- 172.21.15.67:0/903142834 >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 0x7f809406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8093e00640 1 -- 172.21.15.67:0/903142834 >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 msgr2=0x7f809406e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:34.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8093e00640 1 --2- 172.21.15.67:0/903142834 >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 0x7f809406e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8093e00640 1 -- 172.21.15.67:0/903142834 >> v1:172.21.15.67:6789/0 conn(0x7f8094145f80 legacy=0x7f80940cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8093e00640 1 -- 172.21.15.67:0/903142834 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8094072150 con 0x7f80940cd920 2024-04-17T08:51:34.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.167+0000 7f8099a00640 1 --2- 172.21.15.67:0/903142834 >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 0x7f809406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:34.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.168+0000 7f8093e00640 1 --2- 172.21.15.67:0/903142834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f809406bb60 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f809415cc90 tx=0x7f808002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c5c337d4139157 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.168+0000 7f8093400640 1 -- 172.21.15.67:0/903142834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800315a0 con 0x7f80940cd920 2024-04-17T08:51:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.168+0000 7f8093400640 1 -- 172.21.15.67:0/903142834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8080037070 con 0x7f80940cd920 2024-04-17T08:51:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.168+0000 7f8093400640 1 -- 172.21.15.67:0/903142834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080031960 con 0x7f80940cd920 2024-04-17T08:51:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 -- 172.21.15.67:0/903142834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 msgr2=0x7f809406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 --2- 172.21.15.67:0/903142834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f809406bb60 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f809415cc90 tx=0x7f808002b840 comp rx=0 tx=0).stop 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 -- 172.21.15.67:0/903142834 shutdown_connections 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 --2- 172.21.15.67:0/903142834 >> v2:172.21.15.187:3300/0 conn(0x7f809406c0a0 0x7f809406e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 --2- 172.21.15.67:0/903142834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f809406bb60 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 -- 172.21.15.67:0/903142834 >> 172.21.15.67:0/903142834 conn(0x7f80940b2190 msgr2=0x7f80940b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 -- 172.21.15.67:0/903142834 shutdown_connections 2024-04-17T08:51:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.169+0000 7f809a400640 1 -- 172.21.15.67:0/903142834 wait complete. 2024-04-17T08:51:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.171+0000 7f809a400640 1 Processor -- start 2024-04-17T08:51:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.171+0000 7f809a400640 1 -- start start 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f809406c0a0 0x7f809415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8094145f80 0x7f80940a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8094072c30 con 0x7f80940cd920 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f809406c530 con 0x7f809406c0a0 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f809a400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8094072db0 con 0x7f8094145f80 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f8093e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f8093e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55856/0 (socket says 172.21.15.67:55856) 2024-04-17T08:51:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.172+0000 7f8093e00640 1 -- 172.21.15.67:0/3878118050 learned_addr learned my addr 172.21.15.67:0/3878118050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8099000640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f809406c0a0 0x7f809415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8099a00640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8094145f80 0x7f80940a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 -- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8094145f80 msgr2=0x7f80940a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8094145f80 0x7f80940a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 -- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f809406c0a0 msgr2=0x7f809415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f809406c0a0 0x7f809415c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 -- 172.21.15.67:0/3878118050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80941f4df0 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.173+0000 7f8093e00640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f8080009890 tx=0x7f80800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.174+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080032840 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.174+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80800329e0 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.174+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080032dd0 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.174+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80941f50d0 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.174+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80941f5690 con 0x7f80940cd920 2024-04-17T08:51:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.175+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8094068a10 con 0x7f80940cd920 2024-04-17T08:51:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.176+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f808004c020 con 0x7f80940cd920 2024-04-17T08:51:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.177+0000 7f8090c00640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 0x7f807406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.177+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f808002f020 con 0x7f80940cd920 2024-04-17T08:51:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.178+0000 7f8099000640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 0x7f807406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.179+0000 7f8099000640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 0x7f807406eff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8088000f60 tx=0x7f8088009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.182+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8080040070 con 0x7f80940cd920 2024-04-17T08:51:34.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.391+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f809415f680 con 0x7f80940cd920 2024-04-17T08:51:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.397+0000 7f8090c00640 1 -- 172.21.15.67:0/3878118050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) ==== 122+0+0 (secure 0 0 0) 0x7f808007be40 con 0x7f80940cd920 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 msgr2=0x7f807406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 0x7f807406eff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8088000f60 tx=0x7f8088009040 comp rx=0 tx=0).stop 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 msgr2=0x7f80940a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f8080009890 tx=0x7f80800046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 shutdown_connections 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f807406cb30 0x7f807406eff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8094145f80 0x7f80940a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80940cd920 0x7f80940a94c0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 --2- 172.21.15.67:0/3878118050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f809406c0a0 0x7f809415c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.400+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 >> 172.21.15.67:0/3878118050 conn(0x7f80940b2190 msgr2=0x7f809415ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.401+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 shutdown_connections 2024-04-17T08:51:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:34.401+0000 7f809a400640 1 -- 172.21.15.67:0/3878118050 wait complete. 2024-04-17T08:51:34.419 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:51:34.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:51:34.532 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:51:34.673 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:51:34.673 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:34.673 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:34.735 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:51:34.736 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:51:34.736 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:34.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:51:34.795 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:34.795 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:34.855 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:51:34.855 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:51:34.855 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:34.913 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:51:34.913 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:51:34.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:35.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.141+0000 7f07f3a00640 1 Processor -- start 2024-04-17T08:51:35.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.141+0000 7f07f3a00640 1 -- start start 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cce90 0x7f07ec0cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07ec0cd270 0x7f07ec15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07ec15bbd0 0x7f07ec071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f07ec15c460 con 0x7f07ec15bbd0 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f07ec155960 con 0x7f07ec0cd270 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.143+0000 7f07f3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec0ce950 con 0x7f07ec0cce90 2024-04-17T08:51:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07ec15bbd0 0x7f07ec071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07ec15bbd0 0x7f07ec071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55870/0 (socket says 172.21.15.67:55870) 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 -- 172.21.15.67:0/2223194847 learned_addr learned my addr 172.21.15.67:0/2223194847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1000640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cce90 0x7f07ec0cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07ebe00640 1 --2- 172.21.15.67:0/2223194847 >> v2:172.21.15.187:3300/0 conn(0x7f07ec0cd270 0x7f07ec15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 -- 172.21.15.67:0/2223194847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cce90 msgr2=0x7f07ec0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cce90 0x7f07ec0cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 -- 172.21.15.67:0/2223194847 >> v2:172.21.15.187:3300/0 conn(0x7f07ec0cd270 msgr2=0x7f07ec15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 --2- 172.21.15.67:0/2223194847 >> v2:172.21.15.187:3300/0 conn(0x7f07ec0cd270 0x7f07ec15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.144+0000 7f07f1a00640 1 -- 172.21.15.67:0/2223194847 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec0cec30 con 0x7f07ec15bbd0 2024-04-17T08:51:35.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.146+0000 7f07f1a00640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec071d10 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0b3100 tx=0x7f07e402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358926962864a392 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.146+0000 7f07eb400640 1 -- 172.21.15.67:0/2223194847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e402ea80 con 0x7f07ec15bbd0 2024-04-17T08:51:35.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.146+0000 7f07eb400640 1 -- 172.21.15.67:0/2223194847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e402ec20 con 0x7f07ec15bbd0 2024-04-17T08:51:35.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.146+0000 7f07eb400640 1 -- 172.21.15.67:0/2223194847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4009600 con 0x7f07ec15bbd0 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 -- 172.21.15.67:0/2223194847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 msgr2=0x7f07ec071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec071d10 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0b3100 tx=0x7f07e402e0e0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 -- 172.21.15.67:0/2223194847 shutdown_connections 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec071d10 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 --2- 172.21.15.67:0/2223194847 >> v2:172.21.15.187:3300/0 conn(0x7f07ec0cd270 0x7f07ec15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 --2- 172.21.15.67:0/2223194847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cce90 0x7f07ec0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 -- 172.21.15.67:0/2223194847 >> 172.21.15.67:0/2223194847 conn(0x7f07ec06bf50 msgr2=0x7f07ec0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 -- 172.21.15.67:0/2223194847 shutdown_connections 2024-04-17T08:51:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.147+0000 7f07f3a00640 1 -- 172.21.15.67:0/2223194847 wait complete. 2024-04-17T08:51:35.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.149+0000 7f07f3a00640 1 Processor -- start 2024-04-17T08:51:35.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.149+0000 7f07f3a00640 1 -- start start 2024-04-17T08:51:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07ec0cce90 0x7f07ec06efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cd270 0x7f07ec0c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0cd6f0 con 0x7f07ec15bbd0 2024-04-17T08:51:35.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec0cf1f0 con 0x7f07ec0cce90 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.150+0000 7f07f3a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07ec15c060 con 0x7f07ec0cd270 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55878/0 (socket says 172.21.15.67:55878) 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 -- 172.21.15.67:0/615258525 learned_addr learned my addr 172.21.15.67:0/615258525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1000640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07ec0cce90 0x7f07ec06efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07ebe00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cd270 0x7f07ec0c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 -- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cd270 msgr2=0x7f07ec0c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cd270 0x7f07ec0c5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 -- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07ec0cce90 msgr2=0x7f07ec06efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07ec0cce90 0x7f07ec06efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.151+0000 7f07f1a00640 1 -- 172.21.15.67:0/615258525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec06fb20 con 0x7f07ec15bbd0 2024-04-17T08:51:35.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07f1a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f07e4038040 tx=0x7f07e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4046020 con 0x7f07ec15bbd0 2024-04-17T08:51:35.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e40428a0 con 0x7f07ec15bbd0 2024-04-17T08:51:35.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e402ea80 con 0x7f07ec15bbd0 2024-04-17T08:51:35.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec06f840 con 0x7f07ec15bbd0 2024-04-17T08:51:35.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.152+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec1e4c00 con 0x7f07ec15bbd0 2024-04-17T08:51:35.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.153+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ec068b90 con 0x7f07ec15bbd0 2024-04-17T08:51:35.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.154+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f07e4049020 con 0x7f07ec15bbd0 2024-04-17T08:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.155+0000 7f07e8c00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 0x7f07cc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.155+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7f07e40b0f20 con 0x7f07ec15bbd0 2024-04-17T08:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.156+0000 7f07f1000640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 0x7f07cc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.157+0000 7f07f1000640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 0x7f07cc06eff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07dc006fd0 tx=0x7f07dc008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.160+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f07e4050030 con 0x7f07ec15bbd0 2024-04-17T08:51:35.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.410+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07ec0ccc70 con 0x7f07ec15bbd0 2024-04-17T08:51:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.411+0000 7f07e8c00640 1 -- 172.21.15.67:0/615258525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+3882 (secure 0 0 0) 0x7f07e407efd0 con 0x7f07ec15bbd0 2024-04-17T08:51:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:35.411 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":119,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7796,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/2470075071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":2470075071},{"type":"v1","addr":"172.21.15.187:6835","nonce":2470075071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":8698,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3207375620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":3207375620},{"type":"v1","addr":"172.21.15.187:6837","nonce":3207375620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":11652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2627239033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2627239033},{"type":"v1","addr":"172.21.15.67:6835","nonce":2627239033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117},{"gid":11742,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/4172193963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4172193963},{"type":"v1","addr":"172.21.15.67:6833","nonce":4172193963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":119}],"filesystems":[]} 2024-04-17T08:51:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 119 2024-04-17T08:51:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 msgr2=0x7f07cc06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 0x7f07cc06eff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07dc006fd0 tx=0x7f07dc008040 comp rx=0 tx=0).stop 2024-04-17T08:51:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 msgr2=0x7f07ec143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f07e4038040 tx=0x7f07e4004a40 comp rx=0 tx=0).stop 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 shutdown_connections 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f07cc06cb30 0x7f07cc06eff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ec15bbd0 0x7f07ec143b40 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07ec0cd270 0x7f07ec0c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 --2- 172.21.15.67:0/615258525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07ec0cce90 0x7f07ec06efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 >> 172.21.15.67:0/615258525 conn(0x7f07ec06bf50 msgr2=0x7f07ec151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.414+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 shutdown_connections 2024-04-17T08:51:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.415+0000 7f07f3a00640 1 -- 172.21.15.67:0/615258525 wait complete. 2024-04-17T08:51:35.433 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:51:35.433 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:51:35.434 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:51:35.434 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:51:35.434 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:51:35.434 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:51:35.434 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:51:35.434 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:51:35.434 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:51:35.435 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:51:35.435 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:51:35.435 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:51:35.435 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:51:35.435 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:51:35.435 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:51:35.435 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:51:35.451 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:51:35.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:51:35.454 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:51:35.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:51:35.457 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:51:35.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:51:35.465 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:51:35.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:51:35.499 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:51:35.499 DEBUG:teuthology.parallel:result is None 2024-04-17T08:51:35.503 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:51:35.503 DEBUG:teuthology.parallel:result is None 2024-04-17T08:51:35.512 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:51:35.513 DEBUG:teuthology.parallel:result is None 2024-04-17T08:51:35.519 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:51:35.519 DEBUG:teuthology.parallel:result is None 2024-04-17T08:51:35.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:51:35.660 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:51:35.663 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:51:35.675 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:51:35.676 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:51:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.709+0000 7fbdbec00640 1 Processor -- start 2024-04-17T08:51:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.709+0000 7fbdbec00640 1 -- start start 2024-04-17T08:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.710+0000 7fbdbec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb80d0b30 0x7fbdb80b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbec00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbdb80ce4d0 0x7fbdb80b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbec00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdb8071f60 con 0x7fbdb815b860 2024-04-17T08:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbec00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbdb8071d10 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb8072240 con 0x7fbdb80d0b30 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdb7e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fbdb815b860 0x7fbdb80d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:38732/0 (socket says 172.21.15.67:38732) 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdb7e00640 1 -- 172.21.15.67:0/1238669807 learned_addr learned my addr 172.21.15.67:0/1238669807 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdaf400640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb80d0b30 0x7fbdb80b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbcc00640 1 --2- 172.21.15.67:0/1238669807 >> v2:172.21.15.187:3300/0 conn(0x7fbdb80ce4d0 0x7fbdb80b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdb7400640 1 -- 172.21.15.67:0/1238669807 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1489222256 0 0) 0x7fbdb8071f60 con 0x7fbdb815b860 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdb7400640 1 -- 172.21.15.67:0/1238669807 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd9c003520 con 0x7fbdb815b860 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbcc00640 1 -- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb80d0b30 msgr2=0x7fbdb80b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbcc00640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb80d0b30 0x7fbdb80b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.711+0000 7fbdbcc00640 1 -- 172.21.15.67:0/1238669807 >> v1:172.21.15.187:6789/0 conn(0x7fbdb815b860 legacy=0x7fbdb80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.712+0000 7fbdbcc00640 1 -- 172.21.15.67:0/1238669807 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb8072520 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.713+0000 7fbdbcc00640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb80b44b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009d90 tx=0x7fbdb002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=253ac9ef58545592 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.713+0000 7fbdb7400640 1 -- 172.21.15.67:0/1238669807 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0039070 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.713+0000 7fbdb7400640 1 -- 172.21.15.67:0/1238669807 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb002cc50 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.713+0000 7fbdb7400640 1 -- 172.21.15.67:0/1238669807 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0034440 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 -- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80ce4d0 msgr2=0x7fbdb80b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb80b44b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009d90 tx=0x7fbdb002bea0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 -- 172.21.15.67:0/1238669807 shutdown_connections 2024-04-17T08:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb80b44b0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 --2- 172.21.15.67:0/1238669807 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb80d0b30 0x7fbdb80b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 -- 172.21.15.67:0/1238669807 >> 172.21.15.67:0/1238669807 conn(0x7fbdb8067ee0 msgr2=0x7fbdb806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:35.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 -- 172.21.15.67:0/1238669807 shutdown_connections 2024-04-17T08:51:35.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.714+0000 7fbdbec00640 1 -- 172.21.15.67:0/1238669807 wait complete. 2024-04-17T08:51:35.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.716+0000 7fbdbec00640 1 Processor -- start 2024-04-17T08:51:35.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.716+0000 7fbdbec00640 1 -- start start 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80d0b30 0x7fbdb81e3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb815b860 0x7fbdb81e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb815bc40 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8072ae0 con 0x7fbdb80d0b30 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbec00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb806f380 con 0x7fbdb815b860 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdb7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdb7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55906/0 (socket says 172.21.15.67:55906) 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdb7e00640 1 -- 172.21.15.67:0/565615111 learned_addr learned my addr 172.21.15.67:0/565615111 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdaf400640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80d0b30 0x7fbdb81e3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.717+0000 7fbdbcc00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb815b860 0x7fbdb81e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 -- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb815b860 msgr2=0x7fbdb81e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb815b860 0x7fbdb81e4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 -- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80d0b30 msgr2=0x7fbdb81e3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80d0b30 0x7fbdb81e3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 -- 172.21.15.67:0/565615111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb81e5c90 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb7e00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbda800c390 tx=0x7fbda800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8017070 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda8005ce0 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda80072f0 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.718+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb81e4bf0 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.719+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb81e51b0 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.720+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb8068ca0 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.721+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fbda801a070 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.722+0000 7fbdb4c00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 0x7fbd9406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.722+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 5874+0+0 (secure 0 0 0) 0x7fbda808b330 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.722+0000 7fbdaf400640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 0x7fbd9406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:35.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.723+0000 7fbdaf400640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 0x7fbd9406eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80b09f0 tx=0x7fbda0009190 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:35.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.726+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbda8058d90 con 0x7fbdb80ce4d0 2024-04-17T08:51:35.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:35.943+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fbdb8159750 con 0x7fbdb80ce4d0 2024-04-17T08:51:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.422+0000 7fbdb4c00640 1 -- 172.21.15.67:0/565615111 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v244) ==== 97+0+0 (secure 0 0 0) 0x7fbda805d090 con 0x7fbdb80ce4d0 2024-04-17T08:51:36.422 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:51:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.426+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 msgr2=0x7fbd9406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.426+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 0x7fbd9406eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80b09f0 tx=0x7fbda0009190 comp rx=0 tx=0).stop 2024-04-17T08:51:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.426+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 msgr2=0x7fbdb8155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.426+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbda800c390 tx=0x7fbda800c850 comp rx=0 tx=0).stop 2024-04-17T08:51:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.426+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 shutdown_connections 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbd9406cb30 0x7fbd9406eff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdb815b860 0x7fbdb81e4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdb80d0b30 0x7fbdb81e3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 --2- 172.21.15.67:0/565615111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdb80ce4d0 0x7fbdb8155df0 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 >> 172.21.15.67:0/565615111 conn(0x7fbdb8067ee0 msgr2=0x7fbdb80b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 shutdown_connections 2024-04-17T08:51:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.427+0000 7fbdbec00640 1 -- 172.21.15.67:0/565615111 wait complete. 2024-04-17T08:51:36.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:51:36.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.649+0000 7f853ca00640 1 Processor -- start 2024-04-17T08:51:36.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.650+0000 7f853ca00640 1 -- start start 2024-04-17T08:51:36.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.651+0000 7f853ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8538072470 0x7f853806f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:36.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.651+0000 7f853ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85380cd510 con 0x7f853815cb50 2024-04-17T08:51:36.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.651+0000 7f853ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f853815d1b0 con 0x7f8538145980 2024-04-17T08:51:36.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.651+0000 7f853ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f853806f590 con 0x7f8538072470 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8536a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f853815cb50 0x7f8538071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40180/0 (socket says 172.21.15.67:40180) 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8536a00640 1 -- 172.21.15.67:0/123071881 learned_addr learned my addr 172.21.15.67:0/123071881 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8537e00640 1 --2- 172.21.15.67:0/123071881 >> v2:172.21.15.67:3300/0 conn(0x7f8538072470 0x7f853806f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8537e00640 1 -- 172.21.15.67:0/123071881 >> v1:172.21.15.187:6789/0 conn(0x7f8538145980 legacy=0x7f85380cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8537e00640 1 -- 172.21.15.67:0/123071881 >> v1:172.21.15.67:6789/0 conn(0x7f853815cb50 legacy=0x7f8538071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8536000640 1 -- 172.21.15.67:0/123071881 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 37950200 0 0) 0x7f85380cd510 con 0x7f853815cb50 2024-04-17T08:51:36.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.652+0000 7f8537e00640 1 -- 172.21.15.67:0/123071881 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853815d490 con 0x7f8538072470 2024-04-17T08:51:36.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.653+0000 7f8537e00640 1 --2- 172.21.15.67:0/123071881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538072470 0x7f853806f050 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f85300097a0 tx=0x7f853002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c3336303ba3338 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:36.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f8536000640 1 -- 172.21.15.67:0/123071881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300315a0 con 0x7f8538072470 2024-04-17T08:51:36.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f8536000640 1 -- 172.21.15.67:0/123071881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8530037070 con 0x7f8538072470 2024-04-17T08:51:36.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f8536000640 1 -- 172.21.15.67:0/123071881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530031960 con 0x7f8538072470 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 -- 172.21.15.67:0/123071881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538072470 msgr2=0x7f853806f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 --2- 172.21.15.67:0/123071881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538072470 0x7f853806f050 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f85300097a0 tx=0x7f853002b840 comp rx=0 tx=0).stop 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 -- 172.21.15.67:0/123071881 shutdown_connections 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 --2- 172.21.15.67:0/123071881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8538072470 0x7f853806f050 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 -- 172.21.15.67:0/123071881 >> 172.21.15.67:0/123071881 conn(0x7f85380b5f70 msgr2=0x7f853806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 -- 172.21.15.67:0/123071881 shutdown_connections 2024-04-17T08:51:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.654+0000 7f853ca00640 1 -- 172.21.15.67:0/123071881 wait complete. 2024-04-17T08:51:36.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.656+0000 7f853ca00640 1 Processor -- start 2024-04-17T08:51:36.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.656+0000 7f853ca00640 1 -- start start 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8538072470 0x7f85381414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8538145980 0x7f85380b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f853815da50 con 0x7f853815cb50 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8538160210 con 0x7f8538145980 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.657+0000 7f853ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8538160730 con 0x7f8538072470 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55932/0 (socket says 172.21.15.67:55932) 2024-04-17T08:51:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 -- 172.21.15.67:0/3212003938 learned_addr learned my addr 172.21.15.67:0/3212003938 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:36.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8536a00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8538145980 0x7f85380b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:36.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537400640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8538072470 0x7f85381414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 -- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8538072470 msgr2=0x7f85381414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8538072470 0x7f85381414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 -- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8538145980 msgr2=0x7f85380b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8538145980 0x7f85380b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.658+0000 7f8537e00640 1 -- 172.21.15.67:0/3212003938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85381f4940 con 0x7f853815cb50 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f8537e00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f85300097a0 tx=0x7f85300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530032840 con 0x7f853815cb50 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85300329e0 con 0x7f853815cb50 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530032db0 con 0x7f853815cb50 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85381f4c20 con 0x7f853815cb50 2024-04-17T08:51:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.659+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85381f51e0 con 0x7f853815cb50 2024-04-17T08:51:36.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.660+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8538068a10 con 0x7f853815cb50 2024-04-17T08:51:36.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.661+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f853003b3f0 con 0x7f853815cb50 2024-04-17T08:51:36.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.662+0000 7f852f400640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 0x7f851806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:36.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.662+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 4885+0+0 (secure 0 0 0) 0x7f853002f030 con 0x7f853815cb50 2024-04-17T08:51:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.663+0000 7f8537400640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 0x7f851806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:36.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.664+0000 7f8537400640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 0x7f851806f020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.666+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8530040070 con 0x7f853815cb50 2024-04-17T08:51:36.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.946+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8538072e00 con 0x7f853815cb50 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.947+0000 7f852f400640 1 -- 172.21.15.67:0/3212003938 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) ==== 80+0+5304 (secure 0 0 0) 0x7f853007bff0 con 0x7f853815cb50 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:51:36.947 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:51:36.948 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:51:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:51:36.955 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:51:36.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.958 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:36.959 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:51:36.960 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:51:36.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 msgr2=0x7f851806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 0x7f851806f020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528009040 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 msgr2=0x7f85380b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f85300097a0 tx=0x7f85300046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 shutdown_connections 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f851806cb60 0x7f851806f020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853815cb50 0x7f85380b2640 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8538145980 0x7f85380b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 --2- 172.21.15.67:0/3212003938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8538072470 0x7f85381414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.950+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 >> 172.21.15.67:0/3212003938 conn(0x7f85380b5f70 msgr2=0x7f8538144100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.951+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 shutdown_connections 2024-04-17T08:51:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:36.951+0000 7f853ca00640 1 -- 172.21.15.67:0/3212003938 wait complete. 2024-04-17T08:51:36.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:37.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.197+0000 7f09d1000640 1 Processor -- start 2024-04-17T08:51:37.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.198+0000 7f09d1000640 1 -- start start 2024-04-17T08:51:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09d1000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f09cc0cfae0 0x7f09cc169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09d1000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc0d2d30 con 0x7f09cc0b6d00 2024-04-17T08:51:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09d1000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc0d3010 con 0x7f09cc15ec60 2024-04-17T08:51:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09d1000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f09cc0d49c0 con 0x7f09cc0cfae0 2024-04-17T08:51:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09caa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09cc0b6d00 0x7f09cc0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40184/0 (socket says 172.21.15.67:40184) 2024-04-17T08:51:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.199+0000 7f09caa00640 1 -- 172.21.15.67:0/3155260022 learned_addr learned my addr 172.21.15.67:0/3155260022 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09cbe00640 1 --2- 172.21.15.67:0/3155260022 >> v2:172.21.15.187:3300/0 conn(0x7f09cc0cfae0 0x7f09cc169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317273473 0 0) 0x7f09cc0d2d30 con 0x7f09cc0b6d00 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09ac003520 con 0x7f09cc0b6d00 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 191089553 0 0) 0x7f09ac003520 con 0x7f09cc0b6d00 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 >> v2:172.21.15.187:3300/0 conn(0x7f09cc0cfae0 msgr2=0x7f09cc169810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 --2- 172.21.15.67:0/3155260022 >> v2:172.21.15.187:3300/0 conn(0x7f09cc0cfae0 0x7f09cc169810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 >> v1:172.21.15.187:6789/0 conn(0x7f09cc15ec60 legacy=0x7f09cc0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc0d32f0 con 0x7f09cc0b6d00 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.200+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f09b40019f0 con 0x7f09cc0b6d00 2024-04-17T08:51:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 >> v1:172.21.15.67:6789/0 conn(0x7f09cc0b6d00 legacy=0x7f09cc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac003a20 0x7f09ac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09ac004800 0x7f09ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 0x7f09ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0d49c0 con 0x7f09ac0051c0 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac00c0f0 con 0x7f09ac003a20 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac00c2f0 con 0x7f09ac004800 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 0x7f09ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 -- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09ac004800 msgr2=0x7f09ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09ac004800 0x7f09ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 -- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac003a20 msgr2=0x7f09ac003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac003a20 0x7f09ac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.201+0000 7f09cbe00640 1 -- 172.21.15.67:0/3155260022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac003520 con 0x7f09ac0051c0 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.202+0000 7f09cbe00640 1 -- 172.21.15.67:0/3155260022 reap_dead start 2024-04-17T08:51:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.202+0000 7f09cbe00640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 0x7f09ac00b9c0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009ac0 tx=0x7f09ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f090aa6f910c1ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4009690 con 0x7f09ac0051c0 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c4004020 con 0x7f09ac0051c0 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09ca000640 1 -- 172.21.15.67:0/3155260022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4004410 con 0x7f09ac0051c0 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 -- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 msgr2=0x7f09ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 0x7f09ac00b9c0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009ac0 tx=0x7f09ac005830 comp rx=0 tx=0).stop 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 -- 172.21.15.67:0/3155260022 shutdown_connections 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 --2- 172.21.15.67:0/3155260022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ac0051c0 0x7f09ac00b9c0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 -- 172.21.15.67:0/3155260022 >> 172.21.15.67:0/3155260022 conn(0x7f09cc0d15f0 msgr2=0x7f09cc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 -- 172.21.15.67:0/3155260022 shutdown_connections 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.203+0000 7f09d1000640 1 -- 172.21.15.67:0/3155260022 wait complete. 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.205+0000 7f09d1000640 1 Processor -- start 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.205+0000 7f09d1000640 1 -- start start 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac0051c0 0x7f09cc160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09cc15a9f0 0x7f09cc161b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0d00c0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0d3660 con 0x7f09ac0051c0 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.206+0000 7f09d1000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09cc0d3af0 con 0x7f09cc15a9f0 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55954/0 (socket says 172.21.15.67:55954) 2024-04-17T08:51:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 -- 172.21.15.67:0/1095208803 learned_addr learned my addr 172.21.15.67:0/1095208803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:37.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cb400640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac0051c0 0x7f09cc160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09caa00640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09cc15a9f0 0x7f09cc161b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 -- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09cc15a9f0 msgr2=0x7f09cc161b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09cc15a9f0 0x7f09cc161b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 -- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac0051c0 msgr2=0x7f09cc160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac0051c0 0x7f09cc160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.207+0000 7f09cbe00640 1 -- 172.21.15.67:0/1095208803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc0b3330 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09cbe00640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f09c402c380 tx=0x7f09c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c402cc50 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c402cdf0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4003850 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc15ae70 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.208+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1601a0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.209+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc08c650 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.211+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f09c40039f0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.212+0000 7f09c3400640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 0x7f09a806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.212+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 4885+0+0 (secure 0 0 0) 0x7f09c40ac9a0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.213+0000 7f09cb400640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 0x7f09a806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.214+0000 7f09cb400640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 0x7f09a806eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09bc001680 tx=0x7f09bc009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.217+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f09cc08c650 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.429+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09cc068a10 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.430+0000 7f09c3400640 1 -- 172.21.15.67:0/1095208803 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) ==== 74+0+12897 (secure 0 0 0) 0x7f09c407ada0 con 0x7f09cc0ab5a0 2024-04-17T08:51:37.429 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:37.429 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":246,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:36.437220+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.432+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 msgr2=0x7f09a806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.432+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 0x7f09a806eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09bc001680 tx=0x7f09bc009040 comp rx=0 tx=0).stop 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.432+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 msgr2=0x7f09cc0aba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.432+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f09c402c380 tx=0x7f09c4004730 comp rx=0 tx=0).stop 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 shutdown_connections 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f09a806cb30 0x7f09a806eff0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09cc0ab5a0 0x7f09cc0aba50 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09cc15a9f0 0x7f09cc161b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 --2- 172.21.15.67:0/1095208803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09ac0051c0 0x7f09cc160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 >> 172.21.15.67:0/1095208803 conn(0x7f09cc0d15f0 msgr2=0x7f09cc0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 shutdown_connections 2024-04-17T08:51:37.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.433+0000 7f09d1000640 1 -- 172.21.15.67:0/1095208803 wait complete. 2024-04-17T08:51:37.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:37.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:37.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.664+0000 7f129da00640 1 Processor -- start 2024-04-17T08:51:37.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.665+0000 7f129da00640 1 -- start start 2024-04-17T08:51:37.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f129da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f129806d220 0x7f129806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f129da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12980cf670 con 0x7f12980d4390 2024-04-17T08:51:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f129da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12980d3680 con 0x7f12980cf1f0 2024-04-17T08:51:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f129da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f12980d2de0 con 0x7f129806d220 2024-04-17T08:51:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f1297e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f12980d4390 0x7f12980d4770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40190/0 (socket says 172.21.15.67:40190) 2024-04-17T08:51:37.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.666+0000 7f1297e00640 1 -- 172.21.15.67:0/1220095535 learned_addr learned my addr 172.21.15.67:0/1220095535 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f129d000640 1 --2- 172.21.15.67:0/1220095535 >> v2:172.21.15.187:3300/0 conn(0x7f129806d220 0x7f129806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663281755 0 0) 0x7f12980cf670 con 0x7f12980d4390 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f127c003520 con 0x7f12980d4390 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2131565703 0 0) 0x7f127c003520 con 0x7f12980d4390 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 >> v2:172.21.15.187:3300/0 conn(0x7f129806d220 msgr2=0x7f129806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 --2- 172.21.15.67:0/1220095535 >> v2:172.21.15.187:3300/0 conn(0x7f129806d220 0x7f129806d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 >> v1:172.21.15.187:6789/0 conn(0x7f12980cf1f0 legacy=0x7f12980d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12980d30c0 con 0x7f12980d4390 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.667+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f12880019f0 con 0x7f12980d4390 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 >> v1:172.21.15.67:6789/0 conn(0x7f12980d4390 legacy=0x7f12980d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f127c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c004800 0x7f127c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c0051c0 0x7f127c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12980d2de0 con 0x7f127c0039b0 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f127c00c060 con 0x7f127c0051c0 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f127c00c260 con 0x7f127c004800 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f127c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 -- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c004800 msgr2=0x7f127c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f128f400640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c004800 0x7f127c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c004800 0x7f127c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 -- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c0051c0 msgr2=0x7f127c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c0051c0 0x7f127c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f1297e00640 1 -- 172.21.15.67:0/1220095535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c003520 con 0x7f127c0039b0 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f129d000640 1 -- 172.21.15.67:0/1220095535 reap_dead start 2024-04-17T08:51:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.668+0000 7f128f400640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c004800 0x7f127c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f1297e00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f127c003dd0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f128800a850 tx=0x7f127c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19448d3a62161dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128800a4a0 con 0x7f127c0039b0 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288034650 con 0x7f127c0039b0 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f1297400640 1 -- 172.21.15.67:0/1220095535 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288034a80 con 0x7f127c0039b0 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f129da00640 1 -- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 msgr2=0x7f127c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.670+0000 7f129da00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f127c003dd0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f128800a850 tx=0x7f127c005830 comp rx=0 tx=0).stop 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.671+0000 7f129da00640 1 -- 172.21.15.67:0/1220095535 shutdown_connections 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.671+0000 7f129da00640 1 --2- 172.21.15.67:0/1220095535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f127c003dd0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.671+0000 7f129da00640 1 -- 172.21.15.67:0/1220095535 >> 172.21.15.67:0/1220095535 conn(0x7f129809da80 msgr2=0x7f129808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.671+0000 7f129da00640 1 -- 172.21.15.67:0/1220095535 shutdown_connections 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.671+0000 7f129da00640 1 -- 172.21.15.67:0/1220095535 wait complete. 2024-04-17T08:51:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.672+0000 7f129da00640 1 Processor -- start 2024-04-17T08:51:37.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.673+0000 7f129da00640 1 -- start start 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.673+0000 7f129da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1298160d10 0x7f1298161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12980c8970 0x7f12980c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1298162050 con 0x7f127c0039b0 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f129806cd00 con 0x7f12980c8970 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12980d1270 con 0x7f1298160d10 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55980/0 (socket says 172.21.15.67:55980) 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 -- 172.21.15.67:0/3791772332 learned_addr learned my addr 172.21.15.67:0/3791772332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f128f400640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1298160d10 0x7f1298161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f129d000640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12980c8970 0x7f12980c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 -- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1298160d10 msgr2=0x7f1298161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1298160d10 0x7f1298161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 -- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12980c8970 msgr2=0x7f12980c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12980c8970 0x7f12980c8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.674+0000 7f1297e00640 1 -- 172.21.15.67:0/3791772332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12980c9330 con 0x7f127c0039b0 2024-04-17T08:51:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f1297e00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f128800a970 tx=0x7f1288001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288039040 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288043070 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288004d60 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12981f4950 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.675+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12981f4f10 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.676+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1298068a10 con 0x7f127c0039b0 2024-04-17T08:51:37.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.677+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f1288034650 con 0x7f127c0039b0 2024-04-17T08:51:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.678+0000 7f1294c00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 0x7f127406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:37.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.678+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 4885+0+0 (secure 0 0 0) 0x7f12880adca0 con 0x7f127c0039b0 2024-04-17T08:51:37.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.679+0000 7f128f400640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 0x7f127406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:37.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.680+0000 7f128f400640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 0x7f127406eff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1280000c00 tx=0x7f1280009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:37.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.683+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f12880019b0 con 0x7f127c0039b0 2024-04-17T08:51:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.886+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12980b6ab0 con 0x7f127c0039b0 2024-04-17T08:51:37.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.887+0000 7f1294c00640 1 -- 172.21.15.67:0/3791772332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v246) ==== 93+0+10 (secure 0 0 0) 0x7f128807c000 con 0x7f127c0039b0 2024-04-17T08:51:37.886 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 msgr2=0x7f127406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 0x7f127406eff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1280000c00 tx=0x7f1280009040 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 msgr2=0x7f129815a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f128800a970 tx=0x7f1288001dc0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 shutdown_connections 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f127406cb30 0x7f127406eff0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12980c8970 0x7f12980c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1298160d10 0x7f1298161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 --2- 172.21.15.67:0/3791772332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f127c0039b0 0x7f129815a5d0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 >> 172.21.15.67:0/3791772332 conn(0x7f129809da80 msgr2=0x7f12980b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 shutdown_connections 2024-04-17T08:51:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:37.889+0000 7f129da00640 1 -- 172.21.15.67:0/3791772332 wait complete. 2024-04-17T08:51:37.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:51:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.101+0000 7f7539000640 1 Processor -- start 2024-04-17T08:51:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.102+0000 7f7539000640 1 -- start start 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.103+0000 7f7539000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753415ec60 0x7f75340b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.103+0000 7f7539000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75340b6b30 0x7f75340cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.103+0000 7f7539000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75340d2d30 con 0x7f75340cf910 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.103+0000 7f7539000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75340d49c0 con 0x7f753415ec60 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.103+0000 7f7539000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f75340d3660 con 0x7f75340b6b30 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753415ec60 0x7f75340b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753415ec60 0x7f75340b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55982/0 (socket says 172.21.15.67:55982) 2024-04-17T08:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 -- 172.21.15.67:0/2622484285 learned_addr learned my addr 172.21.15.67:0/2622484285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7532a00640 1 --2- 172.21.15.67:0/2622484285 >> v2:172.21.15.187:3300/0 conn(0x7f75340b6b30 0x7f75340cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 -- 172.21.15.67:0/2622484285 >> v2:172.21.15.187:3300/0 conn(0x7f75340b6b30 msgr2=0x7f75340cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 --2- 172.21.15.67:0/2622484285 >> v2:172.21.15.187:3300/0 conn(0x7f75340b6b30 0x7f75340cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 -- 172.21.15.67:0/2622484285 >> v1:172.21.15.187:6789/0 conn(0x7f75340cf910 legacy=0x7f753406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.104+0000 7f7533400640 1 -- 172.21.15.67:0/2622484285 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75340d3010 con 0x7f753415ec60 2024-04-17T08:51:38.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.106+0000 7f7533400640 1 --2- 172.21.15.67:0/2622484285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753415ec60 0x7f75340b65f0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f75340ba590 tx=0x7f752c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff2076db5c4e939 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.106+0000 7f7532000640 1 -- 172.21.15.67:0/2622484285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c02ca80 con 0x7f753415ec60 2024-04-17T08:51:38.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.106+0000 7f7532000640 1 -- 172.21.15.67:0/2622484285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f752c02cc20 con 0x7f753415ec60 2024-04-17T08:51:38.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.106+0000 7f7532000640 1 -- 172.21.15.67:0/2622484285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c0343f0 con 0x7f753415ec60 2024-04-17T08:51:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 -- 172.21.15.67:0/2622484285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753415ec60 msgr2=0x7f75340b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 --2- 172.21.15.67:0/2622484285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753415ec60 0x7f75340b65f0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f75340ba590 tx=0x7f752c02bee0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 -- 172.21.15.67:0/2622484285 shutdown_connections 2024-04-17T08:51:38.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 --2- 172.21.15.67:0/2622484285 >> v2:172.21.15.187:3300/0 conn(0x7f75340b6b30 0x7f75340cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 --2- 172.21.15.67:0/2622484285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753415ec60 0x7f75340b65f0 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 -- 172.21.15.67:0/2622484285 >> 172.21.15.67:0/2622484285 conn(0x7f75340d15f0 msgr2=0x7f75340d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 -- 172.21.15.67:0/2622484285 shutdown_connections 2024-04-17T08:51:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.107+0000 7f7539000640 1 -- 172.21.15.67:0/2622484285 wait complete. 2024-04-17T08:51:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.109+0000 7f7539000640 1 Processor -- start 2024-04-17T08:51:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.109+0000 7f7539000640 1 -- start start 2024-04-17T08:51:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 0x7f75340ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f753415ec60 0x7f75340b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75340cfda0 con 0x7f75340cf910 2024-04-17T08:51:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75340b6f70 con 0x7f75340b6b30 2024-04-17T08:51:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7539000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f753415f040 con 0x7f753415ec60 2024-04-17T08:51:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7532a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7532a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55984/0 (socket says 172.21.15.67:55984) 2024-04-17T08:51:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7532a00640 1 -- 172.21.15.67:0/1781550485 learned_addr learned my addr 172.21.15.67:0/1781550485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7532a00640 1 -- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f753415ec60 msgr2=0x7f75340b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7533400640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 0x7f75340ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.110+0000 7f7533e00640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f753415ec60 0x7f75340b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7532a00640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f753415ec60 0x7f75340b5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7532a00640 1 -- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 msgr2=0x7f75340ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7532a00640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 0x7f75340ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7532a00640 1 -- 172.21.15.67:0/1781550485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75341f55c0 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7533400640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 0x7f75340ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7532a00640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f751c00d800 tx=0x7f751c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c00b6c0 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c019070 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c00ba80 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75341f4800 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.111+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75341f4dc0 con 0x7f75340cf910 2024-04-17T08:51:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.113+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7534068a10 con 0x7f75340cf910 2024-04-17T08:51:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.114+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f751c00bc20 con 0x7f75340cf910 2024-04-17T08:51:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.115+0000 7f752b400640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 0x7f751006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.115+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 4885+0+0 (secure 0 0 0) 0x7f751c08c0c0 con 0x7f75340cf910 2024-04-17T08:51:38.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.115+0000 7f7533400640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 0x7f751006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.116+0000 7f7533400640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 0x7f751006eff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f752c035000 tx=0x7f752c030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.119+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f751c05a550 con 0x7f75340cf910 2024-04-17T08:51:38.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.325+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f75340b3a50 con 0x7f75340cf910 2024-04-17T08:51:38.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.326+0000 7f752b400640 1 -- 172.21.15.67:0/1781550485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v246) ==== 84+0+65 (secure 0 0 0) 0x7f751c059e50 con 0x7f75340cf910 2024-04-17T08:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:51:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.328+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 msgr2=0x7f751006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 0x7f751006eff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f752c035000 tx=0x7f752c030000 comp rx=0 tx=0).stop 2024-04-17T08:51:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 msgr2=0x7f75340b5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f751c00d800 tx=0x7f751c00dcc0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 shutdown_connections 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f751006cb30 0x7f751006eff0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f753415ec60 0x7f75340b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75340cf910 0x7f75340b5590 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 --2- 172.21.15.67:0/1781550485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75340b6b30 0x7f75340ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 >> 172.21.15.67:0/1781550485 conn(0x7f75340d15f0 msgr2=0x7f7534074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 shutdown_connections 2024-04-17T08:51:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.329+0000 7f7539000640 1 -- 172.21.15.67:0/1781550485 wait complete. 2024-04-17T08:51:38.348 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:51:38.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:51:38.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.574+0000 7f3804e00640 1 Processor -- start 2024-04-17T08:51:38.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.574+0000 7f3804e00640 1 -- start start 2024-04-17T08:51:38.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f3804e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 0x7f3800145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f3804e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3800146080 0x7f3800160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f3804e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3800146510 con 0x7f38000cd2f0 2024-04-17T08:51:38.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f3804e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f38000cfa60 con 0x7f3800146080 2024-04-17T08:51:38.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f3804e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3800161000 con 0x7f38000cd6d0 2024-04-17T08:51:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f37ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3800146080 0x7f3800160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f37ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3800146080 0x7f3800160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55986/0 (socket says 172.21.15.67:55986) 2024-04-17T08:51:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.576+0000 7f37ffe00640 1 -- 172.21.15.67:0/2899481579 learned_addr learned my addr 172.21.15.67:0/2899481579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37fea00640 1 --2- 172.21.15.67:0/2899481579 >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 0x7f3800145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37ffe00640 1 -- 172.21.15.67:0/2899481579 >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 msgr2=0x7f3800145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37ffe00640 1 --2- 172.21.15.67:0/2899481579 >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 0x7f3800145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37ffe00640 1 -- 172.21.15.67:0/2899481579 >> v1:172.21.15.187:6789/0 conn(0x7f38000cd2f0 legacy=0x7f38000cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37ffe00640 1 -- 172.21.15.67:0/2899481579 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38000ce870 con 0x7f3800146080 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.577+0000 7f37fea00640 1 --2- 172.21.15.67:0/2899481579 >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 0x7f3800145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.578+0000 7f37ffe00640 1 --2- 172.21.15.67:0/2899481579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800146080 0x7f3800160360 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c10e1f8a085d68c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f37fe000640 1 -- 172.21.15.67:0/2899481579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f802ca80 con 0x7f3800146080 2024-04-17T08:51:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f37fe000640 1 -- 172.21.15.67:0/2899481579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f802cc20 con 0x7f3800146080 2024-04-17T08:51:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f37fe000640 1 -- 172.21.15.67:0/2899481579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f80343f0 con 0x7f3800146080 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 -- 172.21.15.67:0/2899481579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800146080 msgr2=0x7f3800160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 --2- 172.21.15.67:0/2899481579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800146080 0x7f3800160360 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009e50 tx=0x7f37f802bee0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 -- 172.21.15.67:0/2899481579 shutdown_connections 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 --2- 172.21.15.67:0/2899481579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800146080 0x7f3800160360 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 --2- 172.21.15.67:0/2899481579 >> v2:172.21.15.187:3300/0 conn(0x7f38000cd6d0 0x7f3800145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 -- 172.21.15.67:0/2899481579 >> 172.21.15.67:0/2899481579 conn(0x7f380006b8c0 msgr2=0x7f380006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 -- 172.21.15.67:0/2899481579 shutdown_connections 2024-04-17T08:51:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.579+0000 7f3804e00640 1 -- 172.21.15.67:0/2899481579 wait complete. 2024-04-17T08:51:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.581+0000 7f3804e00640 1 Processor -- start 2024-04-17T08:51:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.581+0000 7f3804e00640 1 -- start start 2024-04-17T08:51:38.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000cd2f0 0x7f380015c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3800146080 0x7f38000c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38001608a0 con 0x7f38000cd6d0 2024-04-17T08:51:38.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f38000cee30 con 0x7f38000cd2f0 2024-04-17T08:51:38.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.582+0000 7f3804e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3800160a20 con 0x7f3800146080 2024-04-17T08:51:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55996/0 (socket says 172.21.15.67:55996) 2024-04-17T08:51:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 -- 172.21.15.67:0/927804688 learned_addr learned my addr 172.21.15.67:0/927804688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37ffe00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3800146080 0x7f38000c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37ff400640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000cd2f0 0x7f380015c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 -- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3800146080 msgr2=0x7f38000c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3800146080 0x7f38000c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 -- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000cd2f0 msgr2=0x7f380015c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000cd2f0 0x7f380015c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.583+0000 7f37fea00640 1 -- 172.21.15.67:0/927804688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38000c56d0 con 0x7f38000cd6d0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f37fea00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c820 tx=0x7f37ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec004230 con 0x7f38000cd6d0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec0043d0 con 0x7f38000cd6d0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec007a70 con 0x7f38000cd6d0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38000c5970 con 0x7f38000cd6d0 2024-04-17T08:51:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.584+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38000c5f30 con 0x7f38000cd6d0 2024-04-17T08:51:38.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.585+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3800068a10 con 0x7f38000cd6d0 2024-04-17T08:51:38.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.586+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f37ec007c10 con 0x7f38000cd6d0 2024-04-17T08:51:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.587+0000 7f37f7400640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 0x7f37e006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.587+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 4885+0+0 (secure 0 0 0) 0x7f37ec08bdf0 con 0x7f38000cd6d0 2024-04-17T08:51:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.588+0000 7f37ff400640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 0x7f37e006f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:38.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.589+0000 7f37ff400640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 0x7f37e006f070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37f0004ea0 tx=0x7f37f0009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.592+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f37ec015020 con 0x7f38000cd6d0 2024-04-17T08:51:38.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:38.796+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3800068610 con 0x7f38000cd6d0 2024-04-17T08:51:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.435+0000 7f37f7400640 1 -- 172.21.15.67:0/927804688 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v247) ==== 138+0+0 (secure 0 0 0) 0x7f37ec05a0c0 con 0x7f38000cd6d0 2024-04-17T08:51:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.438+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 msgr2=0x7f37e006f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.438+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 0x7f37e006f070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37f0004ea0 tx=0x7f37f0009040 comp rx=0 tx=0).stop 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.438+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 msgr2=0x7f38001378b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.438+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c820 tx=0x7f37ec00cce0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 shutdown_connections 2024-04-17T08:51:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f37e006cbb0 0x7f37e006f070 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3800146080 0x7f38000c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38000cd6d0 0x7f38001378b0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 --2- 172.21.15.67:0/927804688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f38000cd2f0 0x7f380015c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 >> 172.21.15.67:0/927804688 conn(0x7f380006b8c0 msgr2=0x7f380006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 shutdown_connections 2024-04-17T08:51:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.439+0000 7f3804e00640 1 -- 172.21.15.67:0/927804688 wait complete. 2024-04-17T08:51:39.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:51:39.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.699+0000 7f689a200640 1 Processor -- start 2024-04-17T08:51:39.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.699+0000 7f689a200640 1 -- start start 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f689413f6d0 0x7f6894071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 0x7f68940cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 0x7f68940b2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f689415cbe0 con 0x7f689413f6d0 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f689415cea0 con 0x7f68940b20b0 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f689a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f689415d180 con 0x7f68940cd5d0 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f6898e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f689413f6d0 0x7f6894071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f6898e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f689413f6d0 0x7f6894071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56010/0 (socket says 172.21.15.67:56010) 2024-04-17T08:51:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.701+0000 7f6898e00640 1 -- 172.21.15.67:0/979656710 learned_addr learned my addr 172.21.15.67:0/979656710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6893e00640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 0x7f68940cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6898e00640 1 -- 172.21.15.67:0/979656710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 msgr2=0x7f68940cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6899800640 1 --2- 172.21.15.67:0/979656710 >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 0x7f68940b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6898e00640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 0x7f68940cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6898e00640 1 -- 172.21.15.67:0/979656710 >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 msgr2=0x7f68940b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6898e00640 1 --2- 172.21.15.67:0/979656710 >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 0x7f68940b2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6898e00640 1 -- 172.21.15.67:0/979656710 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689415d460 con 0x7f689413f6d0 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6893e00640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 0x7f68940cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.702+0000 7f6899800640 1 --2- 172.21.15.67:0/979656710 >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 0x7f68940b2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:39.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.703+0000 7f6898e00640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689413f6d0 0x7f6894071d00 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f68940b31b0 tx=0x7f688402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87aa1cf744011b19 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.703+0000 7f6893400640 1 -- 172.21.15.67:0/979656710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688402c9e0 con 0x7f689413f6d0 2024-04-17T08:51:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.703+0000 7f6893400640 1 -- 172.21.15.67:0/979656710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688402cb80 con 0x7f689413f6d0 2024-04-17T08:51:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.703+0000 7f6893400640 1 -- 172.21.15.67:0/979656710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68840343f0 con 0x7f689413f6d0 2024-04-17T08:51:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 -- 172.21.15.67:0/979656710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689413f6d0 msgr2=0x7f6894071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689413f6d0 0x7f6894071d00 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f68940b31b0 tx=0x7f688402be40 comp rx=0 tx=0).stop 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 -- 172.21.15.67:0/979656710 shutdown_connections 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 --2- 172.21.15.67:0/979656710 >> v2:172.21.15.187:3300/0 conn(0x7f68940b20b0 0x7f68940b2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68940cd5d0 0x7f68940cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 --2- 172.21.15.67:0/979656710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689413f6d0 0x7f6894071d00 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 -- 172.21.15.67:0/979656710 >> 172.21.15.67:0/979656710 conn(0x7f6894067d10 msgr2=0x7f68940d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 -- 172.21.15.67:0/979656710 shutdown_connections 2024-04-17T08:51:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.704+0000 7f689a200640 1 -- 172.21.15.67:0/979656710 wait complete. 2024-04-17T08:51:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.707+0000 7f689a200640 1 Processor -- start 2024-04-17T08:51:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.707+0000 7f689a200640 1 -- start start 2024-04-17T08:51:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68940b20b0 0x7f6894142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 0x7f689413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f689415da20 con 0x7f68940cd5d0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f68940d1120 con 0x7f68940b20b0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f689a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6894142020 con 0x7f689413f6d0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.708+0000 7f6893e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56014/0 (socket says 172.21.15.67:56014) 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 -- 172.21.15.67:0/3722148077 learned_addr learned my addr 172.21.15.67:0/3722148077 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6898e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68940b20b0 0x7f6894142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6899800640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 0x7f689413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 -- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 msgr2=0x7f689413cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 0x7f689413cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 -- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68940b20b0 msgr2=0x7f6894142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68940b20b0 0x7f6894142660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 -- 172.21.15.67:0/3722148077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6894142ba0 con 0x7f68940cd5d0 2024-04-17T08:51:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6899800640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 0x7f689413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.709+0000 7f6893e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c7f0 tx=0x7f687c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.710+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c015070 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.710+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687c007a30 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.710+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6894144af0 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.710+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c007e00 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.710+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68940b38a0 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.711+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6894068a10 con 0x7f68940cd5d0 2024-04-17T08:51:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.712+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f687c018070 con 0x7f68940cd5d0 2024-04-17T08:51:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.713+0000 7f6890c00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 0x7f687006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.714+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5255+0+0 (secure 0 0 0) 0x7f687c08c380 con 0x7f68940cd5d0 2024-04-17T08:51:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.714+0000 7f6898e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 0x7f687006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.715+0000 7f6898e00640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 0x7f687006eff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6884035000 tx=0x7f6884030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:39.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.718+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6894144af0 con 0x7f68940cd5d0 2024-04-17T08:51:39.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:39.923+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f68940cc9b0 con 0x7f68940cd5d0 2024-04-17T08:51:40.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.442+0000 7f6890c00640 1 -- 172.21.15.67:0/3722148077 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v248) ==== 170+0+0 (secure 0 0 0) 0x7f687c05a450 con 0x7f68940cd5d0 2024-04-17T08:51:40.442 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:51:40.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.445+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 msgr2=0x7f687006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.445+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 0x7f687006eff0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6884035000 tx=0x7f6884030000 comp rx=0 tx=0).stop 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.445+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 msgr2=0x7f68941445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.445+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c7f0 tx=0x7f687c00cbf0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.445+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 shutdown_connections 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.446+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f687006cb30 0x7f687006eff0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.446+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f689413f6d0 0x7f689413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.446+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68940cd5d0 0x7f68941445b0 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.446+0000 7f689a200640 1 --2- 172.21.15.67:0/3722148077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68940b20b0 0x7f6894142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.446+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 >> 172.21.15.67:0/3722148077 conn(0x7f6894067d10 msgr2=0x7f6894145b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.447+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 shutdown_connections 2024-04-17T08:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.447+0000 7f689a200640 1 -- 172.21.15.67:0/3722148077 wait complete. 2024-04-17T08:51:40.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:51:40.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.669+0000 7fa10b200640 1 Processor -- start 2024-04-17T08:51:40.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.669+0000 7fa10b200640 1 -- start start 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c15b6b0 0x7fa10c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa10c0cfc10 0x7fa10c06be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c06df40 con 0x7fa10c06f800 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa10c06ed00 con 0x7fa10c0cfc10 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa10c06e870 con 0x7fa10c15b6b0 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa10c0cfc10 0x7fa10c06be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa10c0cfc10 0x7fa10c06be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56024/0 (socket says 172.21.15.67:56024) 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa103e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa10c06f800 0x7fa10c15f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40206/0 (socket says 172.21.15.67:40206) 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa10a800640 1 -- 172.21.15.67:0/1290438238 learned_addr learned my addr 172.21.15.67:0/1290438238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.671+0000 7fa109e00640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c15b6b0 0x7fa10c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.672+0000 7fa10a800640 1 -- 172.21.15.67:0/1290438238 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c15b6b0 msgr2=0x7fa10c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.672+0000 7fa10a800640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c15b6b0 0x7fa10c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.672+0000 7fa10a800640 1 -- 172.21.15.67:0/1290438238 >> v1:172.21.15.67:6789/0 conn(0x7fa10c06f800 legacy=0x7fa10c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.672+0000 7fa10a800640 1 -- 172.21.15.67:0/1290438238 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c06e220 con 0x7fa10c0cfc10 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.672+0000 7fa109400640 1 -- 172.21.15.67:0/1290438238 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873374072 0 0) 0x7fa10c06df40 con 0x7fa10c06f800 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.673+0000 7fa10a800640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c0cfc10 0x7fa10c06be00 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fa104009e50 tx=0x7fa10402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6b5ef2dea33ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.673+0000 7fa109400640 1 -- 172.21.15.67:0/1290438238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10402ca70 con 0x7fa10c0cfc10 2024-04-17T08:51:40.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.673+0000 7fa109400640 1 -- 172.21.15.67:0/1290438238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10402cc10 con 0x7fa10c0cfc10 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.673+0000 7fa109400640 1 -- 172.21.15.67:0/1290438238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1040343f0 con 0x7fa10c0cfc10 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 -- 172.21.15.67:0/1290438238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c0cfc10 msgr2=0x7fa10c06be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c0cfc10 0x7fa10c06be00 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fa104009e50 tx=0x7fa10402bed0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 -- 172.21.15.67:0/1290438238 shutdown_connections 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c0cfc10 0x7fa10c06be00 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 --2- 172.21.15.67:0/1290438238 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c15b6b0 0x7fa10c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 -- 172.21.15.67:0/1290438238 >> 172.21.15.67:0/1290438238 conn(0x7fa10c0b2190 msgr2=0x7fa10c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 -- 172.21.15.67:0/1290438238 shutdown_connections 2024-04-17T08:51:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.674+0000 7fa10b200640 1 -- 172.21.15.67:0/1290438238 wait complete. 2024-04-17T08:51:40.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.676+0000 7fa10b200640 1 Processor -- start 2024-04-17T08:51:40.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.676+0000 7fa10b200640 1 -- start start 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa10c06f800 0x7fa10c147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c0cfc10 0x7fa10c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c0afee0 con 0x7fa10c15b6b0 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c0afd60 con 0x7fa10c06f800 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa10c0b44a0 con 0x7fa10c0cfc10 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56040/0 (socket says 172.21.15.67:56040) 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa10a800640 1 -- 172.21.15.67:0/1710112253 learned_addr learned my addr 172.21.15.67:0/1710112253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa109e00640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa10c06f800 0x7fa10c147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.677+0000 7fa103e00640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c0cfc10 0x7fa10c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 -- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c0cfc10 msgr2=0x7fa10c15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c0cfc10 0x7fa10c15a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 -- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa10c06f800 msgr2=0x7fa10c147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa10c06f800 0x7fa10c147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 -- 172.21.15.67:0/1710112253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c151a20 con 0x7fa10c15b6b0 2024-04-17T08:51:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10a800640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fa104036040 tx=0x7fa104004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104043020 con 0x7fa10c15b6b0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10403e450 con 0x7fa10c15b6b0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa104009510 con 0x7fa10c15b6b0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c15e7e0 con 0x7fa10c15b6b0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.678+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10c151740 con 0x7fa10c15b6b0 2024-04-17T08:51:40.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.680+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa10c068a10 con 0x7fa10c15b6b0 2024-04-17T08:51:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.682+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fa104046020 con 0x7fa10c15b6b0 2024-04-17T08:51:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.683+0000 7fa102000640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 0x7fa0e806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.683+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5637+0+0 (secure 0 0 0) 0x7fa1040aeaa0 con 0x7fa10c15b6b0 2024-04-17T08:51:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.684+0000 7fa109e00640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 0x7fa0e806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:40.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.685+0000 7fa109e00640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 0x7fa0e806eff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00a970 tx=0x7fa0fc008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:40.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.687+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa10404d030 con 0x7fa10c15b6b0 2024-04-17T08:51:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:40.925+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa10c06ee80 con 0x7fa10c15b6b0 2024-04-17T08:51:41.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.471+0000 7fa102000640 1 -- 172.21.15.67:0/1710112253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v122) ==== 174+0+0 (secure 0 0 0) 0x7fa10407ce40 con 0x7fa10c15b6b0 2024-04-17T08:51:41.471 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 30 and data pool 31 2024-04-17T08:51:41.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.476+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 msgr2=0x7fa0e806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.476+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 0x7fa0e806eff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00a970 tx=0x7fa0fc008040 comp rx=0 tx=0).stop 2024-04-17T08:51:41.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.476+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 msgr2=0x7fa10c151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.476+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fa104036040 tx=0x7fa104004a40 comp rx=0 tx=0).stop 2024-04-17T08:51:41.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 shutdown_connections 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa0e806cb30 0x7fa0e806eff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa10c15b6b0 0x7fa10c151c60 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa10c0cfc10 0x7fa10c15a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 --2- 172.21.15.67:0/1710112253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa10c06f800 0x7fa10c147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.477+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 >> 172.21.15.67:0/1710112253 conn(0x7fa10c0b2190 msgr2=0x7fa10c153ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.478+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 shutdown_connections 2024-04-17T08:51:41.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.478+0000 7fa10b200640 1 -- 172.21.15.67:0/1710112253 wait complete. 2024-04-17T08:51:41.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:41.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.702+0000 7fe1c0200640 1 Processor -- start 2024-04-17T08:51:41.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.702+0000 7fe1c0200640 1 -- start start 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.703+0000 7fe1c0200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1b806f670 0x7fe1b806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.703+0000 7fe1c0200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b8072b30 con 0x7fe1b806f030 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.703+0000 7fe1c0200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b806ff90 con 0x7fe1b80cd020 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.703+0000 7fe1c0200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b80ceb00 con 0x7fe1b806f670 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.704+0000 7fe1be200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe1b806f030 0x7fe1b8072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40214/0 (socket says 172.21.15.67:40214) 2024-04-17T08:51:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.704+0000 7fe1be200640 1 -- 172.21.15.67:0/3237097691 learned_addr learned my addr 172.21.15.67:0/3237097691 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.704+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3237097691 >> v2:172.21.15.187:3300/0 conn(0x7fe1b806f670 0x7fe1b806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.704+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694578125 0 0) 0x7fe1b8072b30 con 0x7fe1b806f030 2024-04-17T08:51:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.704+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a0003520 con 0x7fe1b806f030 2024-04-17T08:51:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3572345242 0 0) 0x7fe1a0003520 con 0x7fe1b806f030 2024-04-17T08:51:41.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 >> v2:172.21.15.187:3300/0 conn(0x7fe1b806f670 msgr2=0x7fe1b806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 --2- 172.21.15.67:0/3237097691 >> v2:172.21.15.187:3300/0 conn(0x7fe1b806f670 0x7fe1b806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 >> v1:172.21.15.187:6789/0 conn(0x7fe1b80cd020 legacy=0x7fe1b8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b80cede0 con 0x7fe1b806f030 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe1b00019f0 con 0x7fe1b806f030 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 >> v1:172.21.15.67:6789/0 conn(0x7fe1b806f030 legacy=0x7fe1b8072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00039b0 0x7fe1a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1a0004800 0x7fe1a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 0x7fe1a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b80ceb00 con 0x7fe1a00051c0 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a000c060 con 0x7fe1a00039b0 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a000c260 con 0x7fe1a0004800 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.705+0000 7fe1be200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 0x7fe1a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00039b0 0x7fe1a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1bce00640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1a0004800 0x7fe1a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1be200640 1 -- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1a0004800 msgr2=0x7fe1a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1be200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1a0004800 0x7fe1a0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1be200640 1 -- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00039b0 msgr2=0x7fe1a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1be200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00039b0 0x7fe1a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1be200640 1 -- 172.21.15.67:0/3237097691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a0003520 con 0x7fe1a00051c0 2024-04-17T08:51:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.706+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00039b0 0x7fe1a0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.707+0000 7fe1be200640 1 -- 172.21.15.67:0/3237097691 reap_dead start 2024-04-17T08:51:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.707+0000 7fe1be200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 0x7fe1a000b9c0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000a850 tx=0x7fe1a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91862089b36e250f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.707+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000a4a0 con 0x7fe1a00051c0 2024-04-17T08:51:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.707+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b0034650 con 0x7fe1a00051c0 2024-04-17T08:51:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.707+0000 7fe1b7e00640 1 -- 172.21.15.67:0/3237097691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0034a40 con 0x7fe1a00051c0 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 -- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 msgr2=0x7fe1a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 0x7fe1a000b9c0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000a850 tx=0x7fe1a0005830 comp rx=0 tx=0).stop 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 -- 172.21.15.67:0/3237097691 shutdown_connections 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3237097691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a00051c0 0x7fe1a000b9c0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 -- 172.21.15.67:0/3237097691 >> 172.21.15.67:0/3237097691 conn(0x7fe1b806bf50 msgr2=0x7fe1b80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 -- 172.21.15.67:0/3237097691 shutdown_connections 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.708+0000 7fe1c0200640 1 -- 172.21.15.67:0/3237097691 wait complete. 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.710+0000 7fe1c0200640 1 Processor -- start 2024-04-17T08:51:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.710+0000 7fe1c0200640 1 -- start start 2024-04-17T08:51:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00051c0 0x7fe1b81ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 0x7fe1b81f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b80cf680 con 0x7fe1b815eba0 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b80cf3a0 con 0x7fe1a00051c0 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1c0200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b80ce720 con 0x7fe1b8158920 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1be200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1be200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56058/0 (socket says 172.21.15.67:56058) 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1be200640 1 -- 172.21.15.67:0/3211810596 learned_addr learned my addr 172.21.15.67:0/3211810596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00051c0 0x7fe1b81ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.711+0000 7fe1bce00640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 0x7fe1b81f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 -- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 msgr2=0x7fe1b81f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 0x7fe1b81f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 -- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00051c0 msgr2=0x7fe1b81ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00051c0 0x7fe1b81ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 -- 172.21.15.67:0/3211810596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b81f33f0 con 0x7fe1b815eba0 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1bce00640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 0x7fe1b81f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1be200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000f80 tx=0x7fe1b0001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0044070 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b003a040 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b80b5720 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0034800 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.712+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b80b4960 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.713+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b8068b90 con 0x7fe1b815eba0 2024-04-17T08:51:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.715+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe1b0047070 con 0x7fe1b815eba0 2024-04-17T08:51:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.715+0000 7fe1b5600640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 0x7fe19806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.716+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 5737+0+0 (secure 0 0 0) 0x7fe1b00ad7d0 con 0x7fe1b815eba0 2024-04-17T08:51:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.716+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 0x7fe19806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.717+0000 7fe1bd800640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 0x7fe19806eff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80cff60 tx=0x7fe1ac008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.720+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1b81f33f0 con 0x7fe1b815eba0 2024-04-17T08:51:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.928+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1b80b1810 con 0x7fe1b815eba0 2024-04-17T08:51:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.929+0000 7fe1b5600640 1 -- 172.21.15.67:0/3211810596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) ==== 74+0+16628 (secure 0 0 0) 0x7fe1b000a540 con 0x7fe1b815eba0 2024-04-17T08:51:41.929 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:41.929 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":249,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:40.927660+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:41.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 msgr2=0x7fe19806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 0x7fe19806eff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80cff60 tx=0x7fe1ac008040 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 msgr2=0x7fe1b815f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0000f80 tx=0x7fe1b0001dc0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 shutdown_connections 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe19806cb30 0x7fe19806eff0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b815eba0 0x7fe1b815f030 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b8158920 0x7fe1b81f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 --2- 172.21.15.67:0/3211810596 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1a00051c0 0x7fe1b81ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 >> 172.21.15.67:0/3211810596 conn(0x7fe1b806bf50 msgr2=0x7fe1b80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 shutdown_connections 2024-04-17T08:51:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:41.932+0000 7fe1c0200640 1 -- 172.21.15.67:0/3211810596 wait complete. 2024-04-17T08:51:41.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.149+0000 7fc863200640 1 Processor -- start 2024-04-17T08:51:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.149+0000 7fc863200640 1 -- start start 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc86415ac80 0x7fc86406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc864071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc864072330 0x7fc86406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc864072790 con 0x7fc864072330 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc8640cdc30 con 0x7fc86415ac80 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc863200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8640cd920 con 0x7fc864072eb0 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc862800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc864072330 0x7fc86406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc862800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc864072330 0x7fc86406ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56062/0 (socket says 172.21.15.67:56062) 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc862800640 1 -- 172.21.15.67:0/2098149699 learned_addr learned my addr 172.21.15.67:0/2098149699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc861e00640 1 --2- 172.21.15.67:0/2098149699 >> v2:172.21.15.187:3300/0 conn(0x7fc86415ac80 0x7fc86406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.151+0000 7fc85be00640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc864071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.152+0000 7fc862800640 1 -- 172.21.15.67:0/2098149699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 msgr2=0x7fc864071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.152+0000 7fc862800640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc864071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.152+0000 7fc862800640 1 -- 172.21.15.67:0/2098149699 >> v2:172.21.15.187:3300/0 conn(0x7fc86415ac80 msgr2=0x7fc86406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.152+0000 7fc862800640 1 --2- 172.21.15.67:0/2098149699 >> v2:172.21.15.187:3300/0 conn(0x7fc86415ac80 0x7fc86406f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.152+0000 7fc862800640 1 -- 172.21.15.67:0/2098149699 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86406f980 con 0x7fc864072330 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.153+0000 7fc862800640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc864072330 0x7fc86406ddf0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fc86406c9f0 tx=0x7fc85c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d44d4b3ced59fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.153+0000 7fc861400640 1 -- 172.21.15.67:0/2098149699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c02c9e0 con 0x7fc864072330 2024-04-17T08:51:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.153+0000 7fc861400640 1 -- 172.21.15.67:0/2098149699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c02cb80 con 0x7fc864072330 2024-04-17T08:51:42.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.153+0000 7fc861400640 1 -- 172.21.15.67:0/2098149699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0343f0 con 0x7fc864072330 2024-04-17T08:51:42.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 -- 172.21.15.67:0/2098149699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc864072330 msgr2=0x7fc86406ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc864072330 0x7fc86406ddf0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fc86406c9f0 tx=0x7fc85c02be40 comp rx=0 tx=0).stop 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 -- 172.21.15.67:0/2098149699 shutdown_connections 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc864072330 0x7fc86406ddf0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 --2- 172.21.15.67:0/2098149699 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc864071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 --2- 172.21.15.67:0/2098149699 >> v2:172.21.15.187:3300/0 conn(0x7fc86415ac80 0x7fc86406f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 -- 172.21.15.67:0/2098149699 >> 172.21.15.67:0/2098149699 conn(0x7fc864067d10 msgr2=0x7fc8640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 -- 172.21.15.67:0/2098149699 shutdown_connections 2024-04-17T08:51:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.154+0000 7fc863200640 1 -- 172.21.15.67:0/2098149699 wait complete. 2024-04-17T08:51:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.156+0000 7fc863200640 1 Processor -- start 2024-04-17T08:51:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.156+0000 7fc863200640 1 -- start start 2024-04-17T08:51:42.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.157+0000 7fc863200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc864072330 0x7fc86414a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.157+0000 7fc863200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc8641546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc863200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc863200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc86415b060 con 0x7fc86415ac80 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc863200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc86406ff40 con 0x7fc864072330 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc863200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8641f82e0 con 0x7fc864072eb0 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56078/0 (socket says 172.21.15.67:56078) 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 -- 172.21.15.67:0/2275239638 learned_addr learned my addr 172.21.15.67:0/2275239638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc85be00640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc8641546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc861e00640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc864072330 0x7fc86414a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 -- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 msgr2=0x7fc8641546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc8641546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 -- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc864072330 msgr2=0x7fc86414a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc864072330 0x7fc86414a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.158+0000 7fc862800640 1 -- 172.21.15.67:0/2275239638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8641f85c0 con 0x7fc86415ac80 2024-04-17T08:51:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc862800640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009f40 tx=0x7fc85c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c043020 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c03e450 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c009510 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86415d140 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.159+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc864154c10 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.160+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc864068a10 con 0x7fc86415ac80 2024-04-17T08:51:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.161+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fc85c046020 con 0x7fc86415ac80 2024-04-17T08:51:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.163+0000 7fc85a000640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 0x7fc84006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.163+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 5737+0+0 (secure 0 0 0) 0x7fc85c0ae500 con 0x7fc86415ac80 2024-04-17T08:51:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.163+0000 7fc861e00640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 0x7fc84006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.164+0000 7fc861e00640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 0x7fc84006eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc854006fd0 tx=0x7fc854008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.167+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc85c04d070 con 0x7fc86415ac80 2024-04-17T08:51:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.371+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8641f8790 con 0x7fc86415ac80 2024-04-17T08:51:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.372+0000 7fc85a000640 1 -- 172.21.15.67:0/2275239638 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) ==== 74+0+16628 (secure 0 0 0) 0x7fc85c07c5a0 con 0x7fc86415ac80 2024-04-17T08:51:42.372 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:42.372 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":249,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:40.927660+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.375+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 msgr2=0x7fc84006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.375+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 0x7fc84006eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc854006fd0 tx=0x7fc854008040 comp rx=0 tx=0).stop 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.375+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 msgr2=0x7fc8641f7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.375+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009f40 tx=0x7fc85c004a40 comp rx=0 tx=0).stop 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 shutdown_connections 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc84006cb30 0x7fc84006eff0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc86415ac80 0x7fc8641f7890 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc864072eb0 0x7fc8641546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 --2- 172.21.15.67:0/2275239638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc864072330 0x7fc86414a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 >> 172.21.15.67:0/2275239638 conn(0x7fc864067d10 msgr2=0x7fc864159770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 shutdown_connections 2024-04-17T08:51:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.376+0000 7fc863200640 1 -- 172.21.15.67:0/2275239638 wait complete. 2024-04-17T08:51:42.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:51:42.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.622+0000 7f3a41c00640 1 Processor -- start 2024-04-17T08:51:42.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.622+0000 7f3a41c00640 1 -- start start 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.623+0000 7f3a41c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a3c0cd920 0x7f3a3c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.623+0000 7f3a41c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3a3c06c0a0 0x7f3a3c06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a41c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a3c146360 con 0x7f3a3c145f80 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a41c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3c0727a0 con 0x7f3a3c0cd920 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a41c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3c071e70 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a33400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a3c0cd920 0x7f3a3c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a3be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a3c145f80 0x7f3a3c0cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40230/0 (socket says 172.21.15.67:40230) 2024-04-17T08:51:42.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a3be00640 1 -- 172.21.15.67:0/102687731 learned_addr learned my addr 172.21.15.67:0/102687731 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:42.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a41200640 1 --2- 172.21.15.67:0/102687731 >> v2:172.21.15.187:3300/0 conn(0x7f3a3c06c0a0 0x7f3a3c06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a33400640 1 -- 172.21.15.67:0/102687731 >> v2:172.21.15.187:3300/0 conn(0x7f3a3c06c0a0 msgr2=0x7f3a3c06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:42.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a33400640 1 --2- 172.21.15.67:0/102687731 >> v2:172.21.15.187:3300/0 conn(0x7f3a3c06c0a0 0x7f3a3c06e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a33400640 1 -- 172.21.15.67:0/102687731 >> v1:172.21.15.67:6789/0 conn(0x7f3a3c145f80 legacy=0x7f3a3c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a3b400640 1 -- 172.21.15.67:0/102687731 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292949692 0 0) 0x7f3a3c146360 con 0x7f3a3c145f80 2024-04-17T08:51:42.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.624+0000 7f3a33400640 1 -- 172.21.15.67:0/102687731 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c072150 con 0x7f3a3c0cd920 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a33400640 1 --2- 172.21.15.67:0/102687731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c06bb60 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c15cc90 tx=0x7f3a2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5286bd7ed05938 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a3b400640 1 -- 172.21.15.67:0/102687731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240315a0 con 0x7f3a3c0cd920 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a3b400640 1 -- 172.21.15.67:0/102687731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a24037070 con 0x7f3a3c0cd920 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a3b400640 1 -- 172.21.15.67:0/102687731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24031960 con 0x7f3a3c0cd920 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a41c00640 1 -- 172.21.15.67:0/102687731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c0cd920 msgr2=0x7f3a3c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.626+0000 7f3a41c00640 1 --2- 172.21.15.67:0/102687731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c06bb60 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c15cc90 tx=0x7f3a2402b840 comp rx=0 tx=0).stop 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 -- 172.21.15.67:0/102687731 shutdown_connections 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 --2- 172.21.15.67:0/102687731 >> v2:172.21.15.187:3300/0 conn(0x7f3a3c06c0a0 0x7f3a3c06e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 --2- 172.21.15.67:0/102687731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c06bb60 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 -- 172.21.15.67:0/102687731 >> 172.21.15.67:0/102687731 conn(0x7f3a3c0b2190 msgr2=0x7f3a3c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 -- 172.21.15.67:0/102687731 shutdown_connections 2024-04-17T08:51:42.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.627+0000 7f3a41c00640 1 -- 172.21.15.67:0/102687731 wait complete. 2024-04-17T08:51:42.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.628+0000 7f3a41c00640 1 Processor -- start 2024-04-17T08:51:42.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.628+0000 7f3a41c00640 1 -- start start 2024-04-17T08:51:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a3c145f80 0x7f3a3c0ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c072c30 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c06c530 con 0x7f3a3c0cd920 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.629+0000 7f3a41c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3c072db0 con 0x7f3a3c145f80 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56102/0 (socket says 172.21.15.67:56102) 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 -- 172.21.15.67:0/622137379 learned_addr learned my addr 172.21.15.67:0/622137379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 -- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a3c145f80 msgr2=0x7f3a3c0ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a41200640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a3c145f80 0x7f3a3c0ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a33400640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a3c145f80 0x7f3a3c0ae510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 -- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0cd920 msgr2=0x7f3a3c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c0b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 -- 172.21.15.67:0/622137379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c0adb20 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.630+0000 7f3a3be00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c460 tx=0x7f3a2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.631+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c018070 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.631+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a2c007500 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.631+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c004a20 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.631+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3c0ade00 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.631+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3c140d20 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.632+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c069120 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.633+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f3a2c01b070 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.634+0000 7f3a38c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 0x7f3a1806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.634+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f3a2c08b290 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.635+0000 7f3a33400640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 0x7f3a1806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:42.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.636+0000 7f3a33400640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 0x7f3a1806eff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a24036000 tx=0x7f3a24008030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.639+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3a3c069120 con 0x7f3a3c06c0a0 2024-04-17T08:51:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:42.887+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3a3c068a10 con 0x7f3a3c06c0a0 2024-04-17T08:51:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.520+0000 7f3a38c00640 1 -- 172.21.15.67:0/622137379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v125) ==== 118+0+0 (secure 0 0 0) 0x7f3a2c059330 con 0x7f3a3c06c0a0 2024-04-17T08:51:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 msgr2=0x7f3a1806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 0x7f3a1806eff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a24036000 tx=0x7f3a24008030 comp rx=0 tx=0).stop 2024-04-17T08:51:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 msgr2=0x7f3a3c13fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00c460 tx=0x7f3a2c00c920 comp rx=0 tx=0).stop 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 shutdown_connections 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f3a1806cb30 0x7f3a1806eff0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a3c145f80 0x7f3a3c0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a3c0cd920 0x7f3a3c0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 --2- 172.21.15.67:0/622137379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a3c06c0a0 0x7f3a3c13fbf0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 >> 172.21.15.67:0/622137379 conn(0x7f3a3c0b2190 msgr2=0x7f3a3c13d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 shutdown_connections 2024-04-17T08:51:43.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.524+0000 7f3a41c00640 1 -- 172.21.15.67:0/622137379 wait complete. 2024-04-17T08:51:43.547 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:51:43.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:51:43.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.892+0000 7f18aa800640 1 Processor -- start 2024-04-17T08:51:43.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.892+0000 7f18aa800640 1 -- start start 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18aa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18a415b860 0x7f18a40d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18aa800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f18a40ce4d0 0x7f18a40d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18aa800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a4071f60 con 0x7f18a406dd10 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18aa800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f18a4071d10 con 0x7f18a415b860 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18aa800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f18a4072240 con 0x7f18a40ce4d0 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.893+0000 7f18a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18a415b860 0x7f18a40d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18a415b860 0x7f18a40d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33206/0 (socket says 172.21.15.67:33206) 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 -- 172.21.15.67:0/3023567717 learned_addr learned my addr 172.21.15.67:0/3023567717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a2a00640 1 --2- 172.21.15.67:0/3023567717 >> v2:172.21.15.187:3300/0 conn(0x7f18a40ce4d0 0x7f18a40d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 -- 172.21.15.67:0/3023567717 >> v2:172.21.15.187:3300/0 conn(0x7f18a40ce4d0 msgr2=0x7f18a40d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 --2- 172.21.15.67:0/3023567717 >> v2:172.21.15.187:3300/0 conn(0x7f18a40ce4d0 0x7f18a40d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 -- 172.21.15.67:0/3023567717 >> v1:172.21.15.187:6789/0 conn(0x7f18a406dd10 legacy=0x7f18a40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.894+0000 7f18a3400640 1 -- 172.21.15.67:0/3023567717 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a4072520 con 0x7f18a415b860 2024-04-17T08:51:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.896+0000 7f18a3400640 1 --2- 172.21.15.67:0/3023567717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a40d0420 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f18a415f340 tx=0x7f189402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be5287f33f90cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:43.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.896+0000 7f18a2000640 1 -- 172.21.15.67:0/3023567717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894035020 con 0x7f18a415b860 2024-04-17T08:51:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.896+0000 7f18a2000640 1 -- 172.21.15.67:0/3023567717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1894031870 con 0x7f18a415b860 2024-04-17T08:51:43.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.896+0000 7f18a2000640 1 -- 172.21.15.67:0/3023567717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894031c40 con 0x7f18a415b860 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 -- 172.21.15.67:0/3023567717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 msgr2=0x7f18a40d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 --2- 172.21.15.67:0/3023567717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a40d0420 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f18a415f340 tx=0x7f189402b910 comp rx=0 tx=0).stop 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 -- 172.21.15.67:0/3023567717 shutdown_connections 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 --2- 172.21.15.67:0/3023567717 >> v2:172.21.15.187:3300/0 conn(0x7f18a40ce4d0 0x7f18a40d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 --2- 172.21.15.67:0/3023567717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a40d0420 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 -- 172.21.15.67:0/3023567717 >> 172.21.15.67:0/3023567717 conn(0x7f18a4067ee0 msgr2=0x7f18a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 -- 172.21.15.67:0/3023567717 shutdown_connections 2024-04-17T08:51:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.897+0000 7f18aa800640 1 -- 172.21.15.67:0/3023567717 wait complete. 2024-04-17T08:51:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.899+0000 7f18aa800640 1 Processor -- start 2024-04-17T08:51:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.899+0000 7f18aa800640 1 -- start start 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a406dd10 0x7f18a415ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18a40ce4d0 0x7f18a415bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18a415bc40 con 0x7f18a415b860 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f18a4072ae0 con 0x7f18a406dd10 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18aa800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f18a406f380 con 0x7f18a40ce4d0 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33216/0 (socket says 172.21.15.67:33216) 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.900+0000 7f18a3e00640 1 -- 172.21.15.67:0/1011626731 learned_addr learned my addr 172.21.15.67:0/1011626731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3400640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a406dd10 0x7f18a415ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a2a00640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18a40ce4d0 0x7f18a415bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 -- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18a40ce4d0 msgr2=0x7f18a415bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18a40ce4d0 0x7f18a415bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 -- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a406dd10 msgr2=0x7f18a415ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a406dd10 0x7f18a415ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 -- 172.21.15.67:0/1011626731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a415d000 con 0x7f18a415b860 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f18a3e00640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c780 tx=0x7f189c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c004b60 con 0x7f18a415b860 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189c004d00 con 0x7f18a415b860 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.901+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c00aa90 con 0x7f18a415b860 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.902+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a415d2e0 con 0x7f18a415b860 2024-04-17T08:51:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.902+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a414e3c0 con 0x7f18a415b860 2024-04-17T08:51:43.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.903+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a4068ca0 con 0x7f18a415b860 2024-04-17T08:51:43.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.904+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f189c00ac30 con 0x7f18a415b860 2024-04-17T08:51:43.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.905+0000 7f189b400640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 0x7f188406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:43.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.905+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f189c08c350 con 0x7f18a415b860 2024-04-17T08:51:43.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.906+0000 7f18a3400640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 0x7f188406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.907+0000 7f18a3400640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 0x7f188406eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1894036000 tx=0x7f189403c040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:43.909+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f189c020070 con 0x7f18a415b860 2024-04-17T08:51:44.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.163+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f18a406ed00 con 0x7f18a415b860 2024-04-17T08:51:44.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.533+0000 7f189b400640 1 -- 172.21.15.67:0/1011626731 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v126) ==== 122+0+0 (secure 0 0 0) 0x7f189c05a360 con 0x7f18a415b860 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.537+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 msgr2=0x7f188406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 0x7f188406eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1894036000 tx=0x7f189403c040 comp rx=0 tx=0).stop 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 msgr2=0x7f18a414de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c780 tx=0x7f189c00cc40 comp rx=0 tx=0).stop 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 shutdown_connections 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f188406cb30 0x7f188406eff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18a415b860 0x7f18a414de80 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f18a40ce4d0 0x7f18a415bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 --2- 172.21.15.67:0/1011626731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18a406dd10 0x7f18a415ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 >> 172.21.15.67:0/1011626731 conn(0x7f18a4067ee0 msgr2=0x7f18a40b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:44.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 shutdown_connections 2024-04-17T08:51:44.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.538+0000 7f18aa800640 1 -- 172.21.15.67:0/1011626731 wait complete. 2024-04-17T08:51:44.563 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:51:44.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.786+0000 7f249b200640 1 Processor -- start 2024-04-17T08:51:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.786+0000 7f249b200640 1 -- start start 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f249c141ed0 0x7f249c142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c0cd4a0 con 0x7f249c160280 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c1607f0 con 0x7f249c06b8c0 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f249c15ce30 con 0x7f249c141ed0 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f249c141ed0 0x7f249c142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f2493e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f249c160280 0x7f249c06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59460/0 (socket says 172.21.15.67:59460) 2024-04-17T08:51:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f2493e00640 1 -- 172.21.15.67:0/1545529181 learned_addr learned my addr 172.21.15.67:0/1545529181 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.788+0000 7f249a800640 1 -- 172.21.15.67:0/1545529181 >> v1:172.21.15.187:6789/0 conn(0x7f249c06b8c0 legacy=0x7f249c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.789+0000 7f249a800640 1 -- 172.21.15.67:0/1545529181 >> v1:172.21.15.67:6789/0 conn(0x7f249c160280 legacy=0x7f249c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.789+0000 7f2499400640 1 -- 172.21.15.67:0/1545529181 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581373841 0 0) 0x7f249c0cd4a0 con 0x7f249c160280 2024-04-17T08:51:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.789+0000 7f249a800640 1 -- 172.21.15.67:0/1545529181 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c160ad0 con 0x7f249c141ed0 2024-04-17T08:51:44.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.790+0000 7f249a800640 1 --2- 172.21.15.67:0/1545529181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c141ed0 0x7f249c142360 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f2494009d90 tx=0x7f249402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2e7f8290b25a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:44.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.790+0000 7f2499400640 1 -- 172.21.15.67:0/1545529181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494039070 con 0x7f249c141ed0 2024-04-17T08:51:44.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.790+0000 7f2499400640 1 -- 172.21.15.67:0/1545529181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249402cc60 con 0x7f249c141ed0 2024-04-17T08:51:44.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.790+0000 7f2499400640 1 -- 172.21.15.67:0/1545529181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494034660 con 0x7f249c141ed0 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.790+0000 7f249b200640 1 -- 172.21.15.67:0/1545529181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c141ed0 msgr2=0x7f249c142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 --2- 172.21.15.67:0/1545529181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c141ed0 0x7f249c142360 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f2494009d90 tx=0x7f249402beb0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 -- 172.21.15.67:0/1545529181 shutdown_connections 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 --2- 172.21.15.67:0/1545529181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c141ed0 0x7f249c142360 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 -- 172.21.15.67:0/1545529181 >> 172.21.15.67:0/1545529181 conn(0x7f249c067d10 msgr2=0x7f249c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:44.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 -- 172.21.15.67:0/1545529181 shutdown_connections 2024-04-17T08:51:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.791+0000 7f249b200640 1 -- 172.21.15.67:0/1545529181 wait complete. 2024-04-17T08:51:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.792+0000 7f249b200640 1 Processor -- start 2024-04-17T08:51:44.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.793+0000 7f249b200640 1 -- start start 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.793+0000 7f249b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.793+0000 7f249b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 0x7f249c142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f249b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249c160280 0x7f249c143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f249b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f249c13f9b0 con 0x7f249c06b8c0 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f249b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f249c161090 con 0x7f249c141ed0 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f249b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f249c13fb30 con 0x7f249c160280 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33232/0 (socket says 172.21.15.67:33232) 2024-04-17T08:51:44.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 -- 172.21.15.67:0/2706087494 learned_addr learned my addr 172.21.15.67:0/2706087494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2493e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 0x7f249c142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 -- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249c160280 msgr2=0x7f249c143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f249a800640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249c160280 0x7f249c143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249c160280 0x7f249c143d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 -- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 msgr2=0x7f249c142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 0x7f249c142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2499e00640 1 -- 172.21.15.67:0/2706087494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c0b0d60 con 0x7f249c06b8c0 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.794+0000 7f2493e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 0x7f249c142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f2499e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c430 tx=0x7f248c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c011790 con 0x7f249c06b8c0 2024-04-17T08:51:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c017070 con 0x7f249c06b8c0 2024-04-17T08:51:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c011b70 con 0x7f249c06b8c0 2024-04-17T08:51:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c0b1040 con 0x7f249c06b8c0 2024-04-17T08:51:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.795+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c0b1600 con 0x7f249c06b8c0 2024-04-17T08:51:44.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.796+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249c1418e0 con 0x7f249c06b8c0 2024-04-17T08:51:44.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.798+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f248c011d10 con 0x7f249c06b8c0 2024-04-17T08:51:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.798+0000 7f2492000640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 0x7f247806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.799+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f248c08bd30 con 0x7f249c06b8c0 2024-04-17T08:51:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.799+0000 7f2493e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 0x7f247806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:44.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.800+0000 7f2493e00640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 0x7f247806eff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f249c0b3b60 tx=0x7f2484009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:44.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:44.803+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f249c1418e0 con 0x7f249c06b8c0 2024-04-17T08:51:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.053+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f249c15cac0 con 0x7f249c06b8c0 2024-04-17T08:51:45.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.054+0000 7f2492000640 1 -- 172.21.15.67:0/2706087494 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5221 (secure 0 0 0) 0x7f249c15cac0 con 0x7f249c06b8c0 2024-04-17T08:51:45.054 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:45.054 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":11841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15}]} 2024-04-17T08:51:45.054 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 126 2024-04-17T08:51:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 msgr2=0x7f247806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 0x7f247806eff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f249c0b3b60 tx=0x7f2484009040 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 msgr2=0x7f249c155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c430 tx=0x7f248c00c8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 shutdown_connections 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f247806cb30 0x7f247806eff0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f249c160280 0x7f249c143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f249c141ed0 0x7f249c142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 --2- 172.21.15.67:0/2706087494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f249c06b8c0 0x7f249c155e30 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.057+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 >> 172.21.15.67:0/2706087494 conn(0x7f249c067d10 msgr2=0x7f249c13fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.058+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 shutdown_connections 2024-04-17T08:51:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.058+0000 7f249b200640 1 -- 172.21.15.67:0/2706087494 wait complete. 2024-04-17T08:51:45.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:45.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.300+0000 7f889fe00640 1 Processor -- start 2024-04-17T08:51:45.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.300+0000 7f889fe00640 1 -- start start 2024-04-17T08:51:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a015ec60 0x7f88a00b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f88a00cfae0 0x7f88a00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a00d2d30 con 0x7f88a00b6b30 2024-04-17T08:51:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f88a00d49c0 con 0x7f88a00cfae0 2024-04-17T08:51:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00d3660 con 0x7f88a015ec60 2024-04-17T08:51:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88a00b6b30 0x7f88a00cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59476/0 (socket says 172.21.15.67:59476) 2024-04-17T08:51:45.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.301+0000 7f889e000640 1 -- 172.21.15.67:0/186173673 learned_addr learned my addr 172.21.15.67:0/186173673 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889ea00640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a015ec60 0x7f88a00b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3240590414 0 0) 0x7f88a00d2d30 con 0x7f88a00b6b30 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889f400640 1 --2- 172.21.15.67:0/186173673 >> v2:172.21.15.187:3300/0 conn(0x7f88a00cfae0 0x7f88a00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8884003520 con 0x7f88a00b6b30 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3117545467 0 0) 0x7f8884003520 con 0x7f88a00b6b30 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a015ec60 msgr2=0x7f88a00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a015ec60 0x7f88a00b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 >> v2:172.21.15.187:3300/0 conn(0x7f88a00cfae0 msgr2=0x7f88a00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 --2- 172.21.15.67:0/186173673 >> v2:172.21.15.187:3300/0 conn(0x7f88a00cfae0 0x7f88a00b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a00d3010 con 0x7f88a00b6b30 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f88900019f0 con 0x7f88a00b6b30 2024-04-17T08:51:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 >> v1:172.21.15.67:6789/0 conn(0x7f88a00b6b30 legacy=0x7f88a00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8884003a20 0x7f8884003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8884004800 0x7f8884004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88840051c0 0x7f888400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00d49c0 con 0x7f88840051c0 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00d3660 con 0x7f8884003a20 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f888400c000 con 0x7f8884004800 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.302+0000 7f889ea00640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a015ec60 0x7f88a00b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88840051c0 0x7f888400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889e000640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8884004800 0x7f8884004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889ea00640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8884003a20 0x7f8884003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 -- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8884004800 msgr2=0x7f8884004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8884004800 0x7f8884004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 -- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8884003a20 msgr2=0x7f8884003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8884003a20 0x7f8884003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.303+0000 7f889f400640 1 -- 172.21.15.67:0/186173673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8884003520 con 0x7f88840051c0 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889f400640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88840051c0 0x7f888400b9c0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f8898009ac0 tx=0x7f8884005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8b7ee896460677 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898039070 con 0x7f88840051c0 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889802c6b0 con 0x7f88840051c0 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889d600640 1 -- 172.21.15.67:0/186173673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898009690 con 0x7f88840051c0 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889fe00640 1 -- 172.21.15.67:0/186173673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88840051c0 msgr2=0x7f888400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889fe00640 1 --2- 172.21.15.67:0/186173673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88840051c0 0x7f888400b9c0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f8898009ac0 tx=0x7f8884005830 comp rx=0 tx=0).stop 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889f400640 1 -- 172.21.15.67:0/186173673 reap_dead start 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889fe00640 1 -- 172.21.15.67:0/186173673 shutdown_connections 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.305+0000 7f889fe00640 1 -- 172.21.15.67:0/186173673 >> 172.21.15.67:0/186173673 conn(0x7f88a00d15f0 msgr2=0x7f88a00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.306+0000 7f889fe00640 1 -- 172.21.15.67:0/186173673 shutdown_connections 2024-04-17T08:51:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.306+0000 7f889fe00640 1 -- 172.21.15.67:0/186173673 wait complete. 2024-04-17T08:51:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.307+0000 7f889fe00640 1 Processor -- start 2024-04-17T08:51:45.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.308+0000 7f889fe00640 1 -- start start 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.308+0000 7f889fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 0x7f88a0158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a00ce300 0x7f88a014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88a015f040 con 0x7f88a0158ca0 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00b6f70 con 0x7f88840051c0 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00d00c0 con 0x7f88a00ce300 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33238/0 (socket says 172.21.15.67:33238) 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 -- 172.21.15.67:0/3517075330 learned_addr learned my addr 172.21.15.67:0/3517075330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889ea00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 0x7f88a0158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889f400640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a00ce300 0x7f88a014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 -- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a00ce300 msgr2=0x7f88a014aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a00ce300 0x7f88a014aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 -- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 msgr2=0x7f88a0158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.309+0000 7f889e000640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 0x7f88a0158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f889e000640 1 -- 172.21.15.67:0/3517075330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a0157810 con 0x7f88a0158ca0 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f889ea00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 0x7f88a0158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f889e000640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f8890007b00 tx=0x7f889000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889000ee50 con 0x7f88a0158ca0 2024-04-17T08:51:45.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8890018070 con 0x7f88a0158ca0 2024-04-17T08:51:45.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890013660 con 0x7f88a0158ca0 2024-04-17T08:51:45.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a01591a0 con 0x7f88a0158ca0 2024-04-17T08:51:45.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.310+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a00cdb00 con 0x7f88a0158ca0 2024-04-17T08:51:45.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.312+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a0068a10 con 0x7f88a0158ca0 2024-04-17T08:51:45.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.313+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f8890013800 con 0x7f88a0158ca0 2024-04-17T08:51:45.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.314+0000 7f8896a00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 0x7f887c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.314+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f889008c680 con 0x7f88a0158ca0 2024-04-17T08:51:45.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.314+0000 7f889ea00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 0x7f887c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.315+0000 7f889ea00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 0x7f887c06eff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.319+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f889005a760 con 0x7f88a0158ca0 2024-04-17T08:51:45.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.527+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88a015be70 con 0x7f88a0158ca0 2024-04-17T08:51:45.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.529+0000 7f8896a00640 1 -- 172.21.15.67:0/3517075330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) ==== 74+0+16615 (secure 0 0 0) 0x7f889005a060 con 0x7f88a0158ca0 2024-04-17T08:51:45.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":250,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:42.471871+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 msgr2=0x7f887c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 0x7f887c06eff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 msgr2=0x7f88a015ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f8890007b00 tx=0x7f889000e530 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 shutdown_connections 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f887c06cb30 0x7f887c06eff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f88a00ce300 0x7f88a014aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88a0158ca0 0x7f88a015ac20 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 --2- 172.21.15.67:0/3517075330 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88840051c0 0x7f88a0158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 >> 172.21.15.67:0/3517075330 conn(0x7f88a00d15f0 msgr2=0x7f88a0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 shutdown_connections 2024-04-17T08:51:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.532+0000 7f889fe00640 1 -- 172.21.15.67:0/3517075330 wait complete. 2024-04-17T08:51:45.551 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:51:45.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:51:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.768+0000 7f6d2d800640 1 Processor -- start 2024-04-17T08:51:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.769+0000 7f6d2d800640 1 -- start start 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.770+0000 7f6d2d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d280c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.770+0000 7f6d2d800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d280cc500 con 0x7f6d28161e70 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.770+0000 7f6d2d800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d280c71e0 con 0x7f6d280ce4a0 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.770+0000 7f6d2d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d280cc7a0 con 0x7f6d280cd5a0 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.771+0000 7f6d26a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d28161e70 0x7f6d280cce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59488/0 (socket says 172.21.15.67:59488) 2024-04-17T08:51:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.771+0000 7f6d26a00640 1 -- 172.21.15.67:0/2779544964 learned_addr learned my addr 172.21.15.67:0/2779544964 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.771+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d280c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.771+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3424249983 0 0) 0x7f6d280cc500 con 0x7f6d28161e70 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.771+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d0c003520 con 0x7f6d28161e70 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4258840364 0 0) 0x7f6d0c003520 con 0x7f6d28161e70 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 msgr2=0x7f6d280c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d280c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 >> v1:172.21.15.187:6789/0 conn(0x7f6d280ce4a0 legacy=0x7f6d280c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d280cca80 con 0x7f6d28161e70 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f6d18001670 con 0x7f6d28161e70 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 >> v1:172.21.15.67:6789/0 conn(0x7f6d28161e70 legacy=0x7f6d280cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d0c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c004d40 0x7f6d0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d280cc7a0 con 0x7f6d0c003fb0 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c00bb00 con 0x7f6d0c004d40 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0c00bd00 con 0x7f6d280cd5a0 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.772+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d26a00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d0c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d27400640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c004d40 0x7f6d0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d1fe00640 1 -- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 msgr2=0x7f6d0c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280cd5a0 0x7f6d0c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d1fe00640 1 -- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c004d40 msgr2=0x7f6d0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d0c004d40 0x7f6d0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.773+0000 7f6d1fe00640 1 -- 172.21.15.67:0/2779544964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c003520 con 0x7f6d0c003fb0 2024-04-17T08:51:45.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.774+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d0c004800 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009ac0 tx=0x7f6d0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d075b5507dd20116 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.774+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10039070 con 0x7f6d0c003fb0 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.774+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1002c600 con 0x7f6d0c003fb0 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.774+0000 7f6d26000640 1 -- 172.21.15.67:0/2779544964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10009690 con 0x7f6d0c003fb0 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 -- 172.21.15.67:0/2779544964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 msgr2=0x7f6d0c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 --2- 172.21.15.67:0/2779544964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d0c004800 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009ac0 tx=0x7f6d0c005830 comp rx=0 tx=0).stop 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d27400640 1 -- 172.21.15.67:0/2779544964 reap_dead start 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 -- 172.21.15.67:0/2779544964 shutdown_connections 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 -- 172.21.15.67:0/2779544964 >> 172.21.15.67:0/2779544964 conn(0x7f6d2806bf50 msgr2=0x7f6d280b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 -- 172.21.15.67:0/2779544964 shutdown_connections 2024-04-17T08:51:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.775+0000 7f6d2d800640 1 -- 172.21.15.67:0/2779544964 wait complete. 2024-04-17T08:51:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.777+0000 7f6d2d800640 1 Processor -- start 2024-04-17T08:51:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.777+0000 7f6d2d800640 1 -- start start 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d280c77f0 0x7f6d281550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280c7c70 0x7f6d28155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d280c6d80 con 0x7f6d0c003fb0 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d280c6c00 con 0x7f6d280c77f0 2024-04-17T08:51:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.778+0000 7f6d2d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d280c6f00 con 0x7f6d280c7c70 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33258/0 (socket says 172.21.15.67:33258) 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 -- 172.21.15.67:0/3746762394 learned_addr learned my addr 172.21.15.67:0/3746762394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d27400640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280c7c70 0x7f6d28155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d280c77f0 0x7f6d281550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 -- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280c7c70 msgr2=0x7f6d28155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280c7c70 0x7f6d28155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 -- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d280c77f0 msgr2=0x7f6d281550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d280c77f0 0x7f6d281550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.779+0000 7f6d26a00640 1 -- 172.21.15.67:0/3746762394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d280b3920 con 0x7f6d0c003fb0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d26a00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800e790 tx=0x7f6d1800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1800eea0 con 0x7f6d0c003fb0 2024-04-17T08:51:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d18018070 con 0x7f6d0c003fb0 2024-04-17T08:51:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1800ca90 con 0x7f6d0c003fb0 2024-04-17T08:51:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d280b3360 con 0x7f6d0c003fb0 2024-04-17T08:51:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.780+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d28150370 con 0x7f6d0c003fb0 2024-04-17T08:51:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.781+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d28068b90 con 0x7f6d0c003fb0 2024-04-17T08:51:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.782+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6d1800cc30 con 0x7f6d0c003fb0 2024-04-17T08:51:45.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.784+0000 7f6d1ea00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 0x7f6d0406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:45.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.784+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f6d1808be10 con 0x7f6d0c003fb0 2024-04-17T08:51:45.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.784+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 0x7f6d0406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:45.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.786+0000 7f6d1fe00640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 0x7f6d0406eff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6d10036000 tx=0x7f6d10030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:45.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.789+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d1808f030 con 0x7f6d0c003fb0 2024-04-17T08:51:45.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:45.990+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d280b3110 con 0x7f6d0406cb30 2024-04-17T08:51:46.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.013+0000 7f6d1ea00640 1 -- 172.21.15.67:0/3746762394 <== mgr.8926 v2:172.21.15.187:6832/2123602634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252482 (secure 0 0 0) 0x7f6d10082a60 con 0x7f6d0406cb30 2024-04-17T08:51:46.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:46.018 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 msgr2=0x7f6d0406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 0x7f6d0406eff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6d10036000 tx=0x7f6d10030000 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 msgr2=0x7f6d28145dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800e790 tx=0x7f6d1800ec50 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 shutdown_connections 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6d0406cb30 0x7f6d0406eff0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d280c7c70 0x7f6d28155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d280c77f0 0x7f6d281550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 --2- 172.21.15.67:0/3746762394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d0c003fb0 0x7f6d28145dd0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 >> 172.21.15.67:0/3746762394 conn(0x7f6d2806bf50 msgr2=0x7f6d280cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 shutdown_connections 2024-04-17T08:51:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.020+0000 7f6d2d800640 1 -- 172.21.15.67:0/3746762394 wait complete. 2024-04-17T08:51:46.039 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-17T08:51:44.773239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":672,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444484,"kb_used_data":25732,"kb_used_omap":12,"kb_used_meta":418611,"kb_avail":754530236,"statfs":{"total":773094113280,"available":772638961664,"internally_reserved":0,"allocated":26349568,"data_stored":13505356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428658224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.334314"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987293+0000","last_change":"2024-04-17T08:51:41.515286+0000","last_active":"2024-04-17T08:51:42.987293+0000","last_peered":"2024-04-17T08:51:42.987293+0000","last_clean":"2024-04-17T08:51:42.987293+0000","last_became_active":"2024-04-17T08:51:41.515011+0000","last_became_peered":"2024-04-17T08:51:41.515011+0000","last_unstale":"2024-04-17T08:51:42.987293+0000","last_undegraded":"2024-04-17T08:51:42.987293+0000","last_fullsized":"2024-04-17T08:51:42.987293+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:27.187699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987373+0000","last_change":"2024-04-17T08:51:41.505707+0000","last_active":"2024-04-17T08:51:42.987373+0000","last_peered":"2024-04-17T08:51:42.987373+0000","last_clean":"2024-04-17T08:51:42.987373+0000","last_became_active":"2024-04-17T08:51:41.505278+0000","last_became_peered":"2024-04-17T08:51:41.505278+0000","last_unstale":"2024-04-17T08:51:42.987373+0000","last_undegraded":"2024-04-17T08:51:42.987373+0000","last_fullsized":"2024-04-17T08:51:42.987373+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:17.019190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.511053+0000","last_change":"2024-04-17T08:51:41.506466+0000","last_active":"2024-04-17T08:51:41.511053+0000","last_peered":"2024-04-17T08:51:41.511053+0000","last_clean":"2024-04-17T08:51:41.511053+0000","last_became_active":"2024-04-17T08:51:41.506181+0000","last_became_peered":"2024-04-17T08:51:41.506181+0000","last_unstale":"2024-04-17T08:51:41.511053+0000","last_undegraded":"2024-04-17T08:51:41.511053+0000","last_fullsized":"2024-04-17T08:51:41.511053+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:33.370084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515311+0000","last_change":"2024-04-17T08:51:41.503459+0000","last_active":"2024-04-17T08:51:41.515311+0000","last_peered":"2024-04-17T08:51:41.515311+0000","last_clean":"2024-04-17T08:51:41.515311+0000","last_became_active":"2024-04-17T08:51:41.503139+0000","last_became_peered":"2024-04-17T08:51:41.503139+0000","last_unstale":"2024-04-17T08:51:41.515311+0000","last_undegraded":"2024-04-17T08:51:41.515311+0000","last_fullsized":"2024-04-17T08:51:41.515311+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:14.891561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.490971+0000","last_change":"2024-04-17T08:51:41.515918+0000","last_active":"2024-04-17T08:51:43.490971+0000","last_peered":"2024-04-17T08:51:43.490971+0000","last_clean":"2024-04-17T08:51:43.490971+0000","last_became_active":"2024-04-17T08:51:41.515377+0000","last_became_peered":"2024-04-17T08:51:41.515377+0000","last_unstale":"2024-04-17T08:51:43.490971+0000","last_undegraded":"2024-04-17T08:51:43.490971+0000","last_fullsized":"2024-04-17T08:51:43.490971+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:51:05.859833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.503789+0000","last_change":"2024-04-17T08:51:41.497092+0000","last_active":"2024-04-17T08:51:41.503789+0000","last_peered":"2024-04-17T08:51:41.503789+0000","last_clean":"2024-04-17T08:51:41.503789+0000","last_became_active":"2024-04-17T08:51:41.496813+0000","last_became_peered":"2024-04-17T08:51:41.496813+0000","last_unstale":"2024-04-17T08:51:41.503789+0000","last_undegraded":"2024-04-17T08:51:41.503789+0000","last_fullsized":"2024-04-17T08:51:41.503789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:19:43.614868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486423+0000","last_change":"2024-04-17T08:51:41.512828+0000","last_active":"2024-04-17T08:51:42.486423+0000","last_peered":"2024-04-17T08:51:42.486423+0000","last_clean":"2024-04-17T08:51:42.486423+0000","last_became_active":"2024-04-17T08:51:41.512495+0000","last_became_peered":"2024-04-17T08:51:41.512495+0000","last_unstale":"2024-04-17T08:51:42.486423+0000","last_undegraded":"2024-04-17T08:51:42.486423+0000","last_fullsized":"2024-04-17T08:51:42.486423+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:59.808221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482730+0000","last_change":"2024-04-17T08:51:41.505463+0000","last_active":"2024-04-17T08:51:42.482730+0000","last_peered":"2024-04-17T08:51:42.482730+0000","last_clean":"2024-04-17T08:51:42.482730+0000","last_became_active":"2024-04-17T08:51:41.505186+0000","last_became_peered":"2024-04-17T08:51:41.505186+0000","last_unstale":"2024-04-17T08:51:42.482730+0000","last_undegraded":"2024-04-17T08:51:42.482730+0000","last_fullsized":"2024-04-17T08:51:42.482730+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.505449+0000","last_change":"2024-04-17T08:51:41.503091+0000","last_active":"2024-04-17T08:51:41.505449+0000","last_peered":"2024-04-17T08:51:41.505449+0000","last_clean":"2024-04-17T08:51:41.505449+0000","last_became_active":"2024-04-17T08:51:41.502796+0000","last_became_peered":"2024-04-17T08:51:41.502796+0000","last_unstale":"2024-04-17T08:51:41.505449+0000","last_undegraded":"2024-04-17T08:51:41.505449+0000","last_fullsized":"2024-04-17T08:51:41.505449+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:47.830902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488230+0000","last_change":"2024-04-17T08:51:41.517488+0000","last_active":"2024-04-17T08:51:42.488230+0000","last_peered":"2024-04-17T08:51:42.488230+0000","last_clean":"2024-04-17T08:51:42.488230+0000","last_became_active":"2024-04-17T08:51:41.517224+0000","last_became_peered":"2024-04-17T08:51:41.517224+0000","last_unstale":"2024-04-17T08:51:42.488230+0000","last_undegraded":"2024-04-17T08:51:42.488230+0000","last_fullsized":"2024-04-17T08:51:42.488230+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481969+0000","last_change":"2024-04-17T08:51:41.520584+0000","last_active":"2024-04-17T08:51:42.481969+0000","last_peered":"2024-04-17T08:51:42.481969+0000","last_clean":"2024-04-17T08:51:42.481969+0000","last_became_active":"2024-04-17T08:51:41.520134+0000","last_became_peered":"2024-04-17T08:51:41.520134+0000","last_unstale":"2024-04-17T08:51:42.481969+0000","last_undegraded":"2024-04-17T08:51:42.481969+0000","last_fullsized":"2024-04-17T08:51:42.481969+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:25.042374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484401+0000","last_change":"2024-04-17T08:51:41.518003+0000","last_active":"2024-04-17T08:51:42.484401+0000","last_peered":"2024-04-17T08:51:42.484401+0000","last_clean":"2024-04-17T08:51:42.484401+0000","last_became_active":"2024-04-17T08:51:41.517732+0000","last_became_peered":"2024-04-17T08:51:41.517732+0000","last_unstale":"2024-04-17T08:51:42.484401+0000","last_undegraded":"2024-04-17T08:51:42.484401+0000","last_fullsized":"2024-04-17T08:51:42.484401+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:55.524149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487551+0000","last_change":"2024-04-17T08:51:41.514813+0000","last_active":"2024-04-17T08:51:42.487551+0000","last_peered":"2024-04-17T08:51:42.487551+0000","last_clean":"2024-04-17T08:51:42.487551+0000","last_became_active":"2024-04-17T08:51:41.512779+0000","last_became_peered":"2024-04-17T08:51:41.512779+0000","last_unstale":"2024-04-17T08:51:42.487551+0000","last_undegraded":"2024-04-17T08:51:42.487551+0000","last_fullsized":"2024-04-17T08:51:42.487551+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986299+0000","last_change":"2024-04-17T08:51:41.507865+0000","last_active":"2024-04-17T08:51:42.986299+0000","last_peered":"2024-04-17T08:51:42.986299+0000","last_clean":"2024-04-17T08:51:42.986299+0000","last_became_active":"2024-04-17T08:51:41.507622+0000","last_became_peered":"2024-04-17T08:51:41.507622+0000","last_unstale":"2024-04-17T08:51:42.986299+0000","last_undegraded":"2024-04-17T08:51:42.986299+0000","last_fullsized":"2024-04-17T08:51:42.986299+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.174140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.504424+0000","last_change":"2024-04-17T08:51:41.501853+0000","last_active":"2024-04-17T08:51:41.504424+0000","last_peered":"2024-04-17T08:51:41.504424+0000","last_clean":"2024-04-17T08:51:41.504424+0000","last_became_active":"2024-04-17T08:51:41.501575+0000","last_became_peered":"2024-04-17T08:51:41.501575+0000","last_unstale":"2024-04-17T08:51:41.504424+0000","last_undegraded":"2024-04-17T08:51:41.504424+0000","last_fullsized":"2024-04-17T08:51:41.504424+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:13.393507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516444+0000","last_change":"2024-04-17T08:51:41.513490+0000","last_active":"2024-04-17T08:51:41.516444+0000","last_peered":"2024-04-17T08:51:41.516444+0000","last_clean":"2024-04-17T08:51:41.516444+0000","last_became_active":"2024-04-17T08:51:41.513192+0000","last_became_peered":"2024-04-17T08:51:41.513192+0000","last_unstale":"2024-04-17T08:51:41.516444+0000","last_undegraded":"2024-04-17T08:51:41.516444+0000","last_fullsized":"2024-04-17T08:51:41.516444+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:36.367018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484601+0000","last_change":"2024-04-17T08:51:41.516221+0000","last_active":"2024-04-17T08:51:42.484601+0000","last_peered":"2024-04-17T08:51:42.484601+0000","last_clean":"2024-04-17T08:51:42.484601+0000","last_became_active":"2024-04-17T08:51:41.515896+0000","last_became_peered":"2024-04-17T08:51:41.515896+0000","last_unstale":"2024-04-17T08:51:42.484601+0000","last_undegraded":"2024-04-17T08:51:42.484601+0000","last_fullsized":"2024-04-17T08:51:42.484601+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:22.484668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987414+0000","last_change":"2024-04-17T08:51:41.501545+0000","last_active":"2024-04-17T08:51:42.987414+0000","last_peered":"2024-04-17T08:51:42.987414+0000","last_clean":"2024-04-17T08:51:42.987414+0000","last_became_active":"2024-04-17T08:51:41.500258+0000","last_became_peered":"2024-04-17T08:51:41.500258+0000","last_unstale":"2024-04-17T08:51:42.987414+0000","last_undegraded":"2024-04-17T08:51:42.987414+0000","last_fullsized":"2024-04-17T08:51:42.987414+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:24.809010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484390+0000","last_change":"2024-04-17T08:51:41.505156+0000","last_active":"2024-04-17T08:51:42.484390+0000","last_peered":"2024-04-17T08:51:42.484390+0000","last_clean":"2024-04-17T08:51:42.484390+0000","last_became_active":"2024-04-17T08:51:41.503363+0000","last_became_peered":"2024-04-17T08:51:41.503363+0000","last_unstale":"2024-04-17T08:51:42.484390+0000","last_undegraded":"2024-04-17T08:51:42.484390+0000","last_fullsized":"2024-04-17T08:51:42.484390+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:28.800799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.507117+0000","last_change":"2024-04-17T08:51:41.500145+0000","last_active":"2024-04-17T08:51:41.507117+0000","last_peered":"2024-04-17T08:51:41.507117+0000","last_clean":"2024-04-17T08:51:41.507117+0000","last_became_active":"2024-04-17T08:51:41.499316+0000","last_became_peered":"2024-04-17T08:51:41.499316+0000","last_unstale":"2024-04-17T08:51:41.507117+0000","last_undegraded":"2024-04-17T08:51:41.507117+0000","last_fullsized":"2024-04-17T08:51:41.507117+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:07.828822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483883+0000","last_change":"2024-04-17T08:51:41.508121+0000","last_active":"2024-04-17T08:51:42.483883+0000","last_peered":"2024-04-17T08:51:42.483883+0000","last_clean":"2024-04-17T08:51:42.483883+0000","last_became_active":"2024-04-17T08:51:41.506717+0000","last_became_peered":"2024-04-17T08:51:41.506717+0000","last_unstale":"2024-04-17T08:51:42.483883+0000","last_undegraded":"2024-04-17T08:51:42.483883+0000","last_fullsized":"2024-04-17T08:51:42.483883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:45.296249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486789+0000","last_change":"2024-04-17T08:51:41.517429+0000","last_active":"2024-04-17T08:51:42.486789+0000","last_peered":"2024-04-17T08:51:42.486789+0000","last_clean":"2024-04-17T08:51:42.486789+0000","last_became_active":"2024-04-17T08:51:41.517130+0000","last_became_peered":"2024-04-17T08:51:41.517130+0000","last_unstale":"2024-04-17T08:51:42.486789+0000","last_undegraded":"2024-04-17T08:51:42.486789+0000","last_fullsized":"2024-04-17T08:51:42.486789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:14.831099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483852+0000","last_change":"2024-04-17T08:51:41.507626+0000","last_active":"2024-04-17T08:51:42.483852+0000","last_peered":"2024-04-17T08:51:42.483852+0000","last_clean":"2024-04-17T08:51:42.483852+0000","last_became_active":"2024-04-17T08:51:41.507278+0000","last_became_peered":"2024-04-17T08:51:41.507278+0000","last_unstale":"2024-04-17T08:51:42.483852+0000","last_undegraded":"2024-04-17T08:51:42.483852+0000","last_fullsized":"2024-04-17T08:51:42.483852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986567+0000","last_change":"2024-04-17T08:51:41.502141+0000","last_active":"2024-04-17T08:51:42.986567+0000","last_peered":"2024-04-17T08:51:42.986567+0000","last_clean":"2024-04-17T08:51:42.986567+0000","last_became_active":"2024-04-17T08:51:41.501895+0000","last_became_peered":"2024-04-17T08:51:41.501895+0000","last_unstale":"2024-04-17T08:51:42.986567+0000","last_undegraded":"2024-04-17T08:51:42.986567+0000","last_fullsized":"2024-04-17T08:51:42.986567+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:45.139092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515944+0000","last_change":"2024-04-17T08:51:41.504007+0000","last_active":"2024-04-17T08:51:41.515944+0000","last_peered":"2024-04-17T08:51:41.515944+0000","last_clean":"2024-04-17T08:51:41.515944+0000","last_became_active":"2024-04-17T08:51:41.503745+0000","last_became_peered":"2024-04-17T08:51:41.503745+0000","last_unstale":"2024-04-17T08:51:41.515944+0000","last_undegraded":"2024-04-17T08:51:41.515944+0000","last_fullsized":"2024-04-17T08:51:41.515944+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:45.006490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488100+0000","last_change":"2024-04-17T08:51:41.513058+0000","last_active":"2024-04-17T08:51:42.488100+0000","last_peered":"2024-04-17T08:51:42.488100+0000","last_clean":"2024-04-17T08:51:42.488100+0000","last_became_active":"2024-04-17T08:51:41.512444+0000","last_became_peered":"2024-04-17T08:51:41.512444+0000","last_unstale":"2024-04-17T08:51:42.488100+0000","last_undegraded":"2024-04-17T08:51:42.488100+0000","last_fullsized":"2024-04-17T08:51:42.488100+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482212+0000","last_change":"2024-04-17T08:51:41.519929+0000","last_active":"2024-04-17T08:51:42.482212+0000","last_peered":"2024-04-17T08:51:42.482212+0000","last_clean":"2024-04-17T08:51:42.482212+0000","last_became_active":"2024-04-17T08:51:41.519591+0000","last_became_peered":"2024-04-17T08:51:41.519591+0000","last_unstale":"2024-04-17T08:51:42.482212+0000","last_undegraded":"2024-04-17T08:51:42.482212+0000","last_fullsized":"2024-04-17T08:51:42.482212+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:03.965519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986948+0000","last_change":"2024-04-17T08:51:41.506863+0000","last_active":"2024-04-17T08:51:42.986948+0000","last_peered":"2024-04-17T08:51:42.986948+0000","last_clean":"2024-04-17T08:51:42.986948+0000","last_became_active":"2024-04-17T08:51:41.506606+0000","last_became_peered":"2024-04-17T08:51:41.506606+0000","last_unstale":"2024-04-17T08:51:42.986948+0000","last_undegraded":"2024-04-17T08:51:42.986948+0000","last_fullsized":"2024-04-17T08:51:42.986948+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:42.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481172+0000","last_change":"2024-04-17T08:51:41.505583+0000","last_active":"2024-04-17T08:51:42.481172+0000","last_peered":"2024-04-17T08:51:42.481172+0000","last_clean":"2024-04-17T08:51:42.481172+0000","last_became_active":"2024-04-17T08:51:41.501478+0000","last_became_peered":"2024-04-17T08:51:41.501478+0000","last_unstale":"2024-04-17T08:51:42.481172+0000","last_undegraded":"2024-04-17T08:51:42.481172+0000","last_fullsized":"2024-04-17T08:51:42.481172+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:01.976342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.490832+0000","last_change":"2024-04-17T08:51:41.500660+0000","last_active":"2024-04-17T08:51:43.490832+0000","last_peered":"2024-04-17T08:51:43.490832+0000","last_clean":"2024-04-17T08:51:43.490832+0000","last_became_active":"2024-04-17T08:51:41.500031+0000","last_became_peered":"2024-04-17T08:51:41.500031+0000","last_unstale":"2024-04-17T08:51:43.490832+0000","last_undegraded":"2024-04-17T08:51:43.490832+0000","last_fullsized":"2024-04-17T08:51:43.490832+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:10.441817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515315+0000","last_change":"2024-04-17T08:51:41.501146+0000","last_active":"2024-04-17T08:51:41.515315+0000","last_peered":"2024-04-17T08:51:41.515315+0000","last_clean":"2024-04-17T08:51:41.515315+0000","last_became_active":"2024-04-17T08:51:41.500704+0000","last_became_peered":"2024-04-17T08:51:41.500704+0000","last_unstale":"2024-04-17T08:51:41.515315+0000","last_undegraded":"2024-04-17T08:51:41.515315+0000","last_fullsized":"2024-04-17T08:51:41.515315+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:39.854967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986854+0000","last_change":"2024-04-17T08:51:41.517358+0000","last_active":"2024-04-17T08:51:42.986854+0000","last_peered":"2024-04-17T08:51:42.986854+0000","last_clean":"2024-04-17T08:51:42.986854+0000","last_became_active":"2024-04-17T08:51:41.517090+0000","last_became_peered":"2024-04-17T08:51:41.517090+0000","last_unstale":"2024-04-17T08:51:42.986854+0000","last_undegraded":"2024-04-17T08:51:42.986854+0000","last_fullsized":"2024-04-17T08:51:42.986854+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:48.083225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510130+0000","last_change":"2024-04-17T08:51:41.503931+0000","last_active":"2024-04-17T08:51:41.510130+0000","last_peered":"2024-04-17T08:51:41.510130+0000","last_clean":"2024-04-17T08:51:41.510130+0000","last_became_active":"2024-04-17T08:51:41.502502+0000","last_became_peered":"2024-04-17T08:51:41.502502+0000","last_unstale":"2024-04-17T08:51:41.510130+0000","last_undegraded":"2024-04-17T08:51:41.510130+0000","last_fullsized":"2024-04-17T08:51:41.510130+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:24.179966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488522+0000","last_change":"2024-04-17T08:51:40.478560+0000","last_active":"2024-04-17T08:51:42.488522+0000","last_peered":"2024-04-17T08:51:42.488522+0000","last_clean":"2024-04-17T08:51:42.488522+0000","last_became_active":"2024-04-17T08:51:40.478144+0000","last_became_peered":"2024-04-17T08:51:40.478144+0000","last_unstale":"2024-04-17T08:51:42.488522+0000","last_undegraded":"2024-04-17T08:51:42.488522+0000","last_fullsized":"2024-04-17T08:51:42.488522+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:37.672975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481410+0000","last_change":"2024-04-17T08:51:41.505188+0000","last_active":"2024-04-17T08:51:42.481410+0000","last_peered":"2024-04-17T08:51:42.481410+0000","last_clean":"2024-04-17T08:51:42.481410+0000","last_became_active":"2024-04-17T08:51:41.504905+0000","last_became_peered":"2024-04-17T08:51:41.504905+0000","last_unstale":"2024-04-17T08:51:42.481410+0000","last_undegraded":"2024-04-17T08:51:42.481410+0000","last_fullsized":"2024-04-17T08:51:42.481410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:11.712753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525859+0000","last_change":"2024-04-17T08:51:40.469433+0000","last_active":"2024-04-17T08:51:41.525859+0000","last_peered":"2024-04-17T08:51:41.525859+0000","last_clean":"2024-04-17T08:51:41.525859+0000","last_became_active":"2024-04-17T08:51:40.469071+0000","last_became_peered":"2024-04-17T08:51:40.469071+0000","last_unstale":"2024-04-17T08:51:41.525859+0000","last_undegraded":"2024-04-17T08:51:41.525859+0000","last_fullsized":"2024-04-17T08:51:41.525859+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:21.208592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484789+0000","last_change":"2024-04-17T08:51:41.516473+0000","last_active":"2024-04-17T08:51:42.484789+0000","last_peered":"2024-04-17T08:51:42.484789+0000","last_clean":"2024-04-17T08:51:42.484789+0000","last_became_active":"2024-04-17T08:51:41.515870+0000","last_became_peered":"2024-04-17T08:51:41.515870+0000","last_unstale":"2024-04-17T08:51:42.484789+0000","last_undegraded":"2024-04-17T08:51:42.484789+0000","last_fullsized":"2024-04-17T08:51:42.484789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:09.372045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.524466+0000","last_change":"2024-04-17T08:51:40.472815+0000","last_active":"2024-04-17T08:51:41.524466+0000","last_peered":"2024-04-17T08:51:41.524466+0000","last_clean":"2024-04-17T08:51:41.524466+0000","last_became_active":"2024-04-17T08:51:40.470696+0000","last_became_peered":"2024-04-17T08:51:40.470696+0000","last_unstale":"2024-04-17T08:51:41.524466+0000","last_undegraded":"2024-04-17T08:51:41.524466+0000","last_fullsized":"2024-04-17T08:51:41.524466+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:32.004523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480808+0000","last_change":"2024-04-17T08:51:41.497630+0000","last_active":"2024-04-17T08:51:42.480808+0000","last_peered":"2024-04-17T08:51:42.480808+0000","last_clean":"2024-04-17T08:51:42.480808+0000","last_became_active":"2024-04-17T08:51:41.497323+0000","last_became_peered":"2024-04-17T08:51:41.497323+0000","last_unstale":"2024-04-17T08:51:42.480808+0000","last_undegraded":"2024-04-17T08:51:42.480808+0000","last_fullsized":"2024-04-17T08:51:42.480808+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:45.677621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.511179+0000","last_change":"2024-04-17T08:51:40.475306+0000","last_active":"2024-04-17T08:51:41.511179+0000","last_peered":"2024-04-17T08:51:41.511179+0000","last_clean":"2024-04-17T08:51:41.511179+0000","last_became_active":"2024-04-17T08:51:40.474409+0000","last_became_peered":"2024-04-17T08:51:40.474409+0000","last_unstale":"2024-04-17T08:51:41.511179+0000","last_undegraded":"2024-04-17T08:51:41.511179+0000","last_fullsized":"2024-04-17T08:51:41.511179+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:17.301101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.512187+0000","last_change":"2024-04-17T08:51:41.511131+0000","last_active":"2024-04-17T08:51:41.512187+0000","last_peered":"2024-04-17T08:51:41.512187+0000","last_clean":"2024-04-17T08:51:41.512187+0000","last_became_active":"2024-04-17T08:51:41.510481+0000","last_became_peered":"2024-04-17T08:51:41.510481+0000","last_unstale":"2024-04-17T08:51:41.512187+0000","last_undegraded":"2024-04-17T08:51:41.512187+0000","last_fullsized":"2024-04-17T08:51:41.512187+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:49.058157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986237+0000","last_change":"2024-04-17T08:51:40.476463+0000","last_active":"2024-04-17T08:51:42.986237+0000","last_peered":"2024-04-17T08:51:42.986237+0000","last_clean":"2024-04-17T08:51:42.986237+0000","last_became_active":"2024-04-17T08:51:40.476164+0000","last_became_peered":"2024-04-17T08:51:40.476164+0000","last_unstale":"2024-04-17T08:51:42.986237+0000","last_undegraded":"2024-04-17T08:51:42.986237+0000","last_fullsized":"2024-04-17T08:51:42.986237+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:15.997620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.508331+0000","last_change":"2024-04-17T08:51:41.507219+0000","last_active":"2024-04-17T08:51:41.508331+0000","last_peered":"2024-04-17T08:51:41.508331+0000","last_clean":"2024-04-17T08:51:41.508331+0000","last_became_active":"2024-04-17T08:51:41.504481+0000","last_became_peered":"2024-04-17T08:51:41.504481+0000","last_unstale":"2024-04-17T08:51:41.508331+0000","last_undegraded":"2024-04-17T08:51:41.508331+0000","last_fullsized":"2024-04-17T08:51:41.508331+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:41.482889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486732+0000","last_change":"2024-04-17T08:51:40.485442+0000","last_active":"2024-04-17T08:51:42.486732+0000","last_peered":"2024-04-17T08:51:42.486732+0000","last_clean":"2024-04-17T08:51:42.486732+0000","last_became_active":"2024-04-17T08:51:40.485100+0000","last_became_peered":"2024-04-17T08:51:40.485100+0000","last_unstale":"2024-04-17T08:51:42.486732+0000","last_undegraded":"2024-04-17T08:51:42.486732+0000","last_fullsized":"2024-04-17T08:51:42.486732+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:18.376309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986392+0000","last_change":"2024-04-17T08:51:41.504552+0000","last_active":"2024-04-17T08:51:42.986392+0000","last_peered":"2024-04-17T08:51:42.986392+0000","last_clean":"2024-04-17T08:51:42.986392+0000","last_became_active":"2024-04-17T08:51:41.504177+0000","last_became_peered":"2024-04-17T08:51:41.504177+0000","last_unstale":"2024-04-17T08:51:42.986392+0000","last_undegraded":"2024-04-17T08:51:42.986392+0000","last_fullsized":"2024-04-17T08:51:42.986392+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:31.634770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477026+0000","last_change":"2024-04-17T08:51:40.470752+0000","last_active":"2024-04-17T08:51:41.477026+0000","last_peered":"2024-04-17T08:51:41.477026+0000","last_clean":"2024-04-17T08:51:41.477026+0000","last_became_active":"2024-04-17T08:51:40.470440+0000","last_became_peered":"2024-04-17T08:51:40.470440+0000","last_unstale":"2024-04-17T08:51:41.477026+0000","last_undegraded":"2024-04-17T08:51:41.477026+0000","last_fullsized":"2024-04-17T08:51:41.477026+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.094101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987750+0000","last_change":"2024-04-17T08:51:41.505515+0000","last_active":"2024-04-17T08:51:42.987750+0000","last_peered":"2024-04-17T08:51:42.987750+0000","last_clean":"2024-04-17T08:51:42.987750+0000","last_became_active":"2024-04-17T08:51:41.504948+0000","last_became_peered":"2024-04-17T08:51:41.504948+0000","last_unstale":"2024-04-17T08:51:42.987750+0000","last_undegraded":"2024-04-17T08:51:42.987750+0000","last_fullsized":"2024-04-17T08:51:42.987750+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:44.205613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484572+0000","last_change":"2024-04-17T08:51:40.472194+0000","last_active":"2024-04-17T08:51:42.484572+0000","last_peered":"2024-04-17T08:51:42.484572+0000","last_clean":"2024-04-17T08:51:42.484572+0000","last_became_active":"2024-04-17T08:51:40.471867+0000","last_became_peered":"2024-04-17T08:51:40.471867+0000","last_unstale":"2024-04-17T08:51:42.484572+0000","last_undegraded":"2024-04-17T08:51:42.484572+0000","last_fullsized":"2024-04-17T08:51:42.484572+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:57.953636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482990+0000","last_change":"2024-04-17T08:51:41.505182+0000","last_active":"2024-04-17T08:51:42.482990+0000","last_peered":"2024-04-17T08:51:42.482990+0000","last_clean":"2024-04-17T08:51:42.482990+0000","last_became_active":"2024-04-17T08:51:41.503770+0000","last_became_peered":"2024-04-17T08:51:41.503770+0000","last_unstale":"2024-04-17T08:51:42.482990+0000","last_undegraded":"2024-04-17T08:51:42.482990+0000","last_fullsized":"2024-04-17T08:51:42.482990+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.476808+0000","last_change":"2024-04-17T08:51:40.478982+0000","last_active":"2024-04-17T08:51:41.476808+0000","last_peered":"2024-04-17T08:51:41.476808+0000","last_clean":"2024-04-17T08:51:41.476808+0000","last_became_active":"2024-04-17T08:51:40.478218+0000","last_became_peered":"2024-04-17T08:51:40.478218+0000","last_unstale":"2024-04-17T08:51:41.476808+0000","last_undegraded":"2024-04-17T08:51:41.476808+0000","last_fullsized":"2024-04-17T08:51:41.476808+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:48:34.166708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.499486+0000","last_change":"2024-04-17T08:51:41.488821+0000","last_active":"2024-04-17T08:51:41.499486+0000","last_peered":"2024-04-17T08:51:41.499486+0000","last_clean":"2024-04-17T08:51:41.499486+0000","last_became_active":"2024-04-17T08:51:41.488545+0000","last_became_peered":"2024-04-17T08:51:41.488545+0000","last_unstale":"2024-04-17T08:51:41.499486+0000","last_undegraded":"2024-04-17T08:51:41.499486+0000","last_fullsized":"2024-04-17T08:51:41.499486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:13.455149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":17,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987036+0000","last_change":"2024-04-17T08:51:40.484183+0000","last_active":"2024-04-17T08:51:42.987036+0000","last_peered":"2024-04-17T08:51:42.987036+0000","last_clean":"2024-04-17T08:51:42.987036+0000","last_became_active":"2024-04-17T08:51:40.483891+0000","last_became_peered":"2024-04-17T08:51:40.483891+0000","last_unstale":"2024-04-17T08:51:42.987036+0000","last_undegraded":"2024-04-17T08:51:42.987036+0000","last_fullsized":"2024-04-17T08:51:42.987036+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:41.230843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487873+0000","last_change":"2024-04-17T08:51:41.515320+0000","last_active":"2024-04-17T08:51:42.487873+0000","last_peered":"2024-04-17T08:51:42.487873+0000","last_clean":"2024-04-17T08:51:42.487873+0000","last_became_active":"2024-04-17T08:51:41.513254+0000","last_became_peered":"2024-04-17T08:51:41.513254+0000","last_unstale":"2024-04-17T08:51:42.487873+0000","last_undegraded":"2024-04-17T08:51:42.487873+0000","last_fullsized":"2024-04-17T08:51:42.487873+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"249'6","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986879+0000","last_change":"2024-04-17T08:51:40.479746+0000","last_active":"2024-04-17T08:51:42.986879+0000","last_peered":"2024-04-17T08:51:42.986879+0000","last_clean":"2024-04-17T08:51:42.986879+0000","last_became_active":"2024-04-17T08:51:40.478655+0000","last_became_peered":"2024-04-17T08:51:40.478655+0000","last_unstale":"2024-04-17T08:51:42.986879+0000","last_undegraded":"2024-04-17T08:51:42.986879+0000","last_fullsized":"2024-04-17T08:51:42.986879+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:03.205317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486935+0000","last_change":"2024-04-17T08:51:41.518673+0000","last_active":"2024-04-17T08:51:42.486935+0000","last_peered":"2024-04-17T08:51:42.486935+0000","last_clean":"2024-04-17T08:51:42.486935+0000","last_became_active":"2024-04-17T08:51:41.517797+0000","last_became_peered":"2024-04-17T08:51:41.517797+0000","last_unstale":"2024-04-17T08:51:42.486935+0000","last_undegraded":"2024-04-17T08:51:42.486935+0000","last_fullsized":"2024-04-17T08:51:42.486935+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:09.689564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986885+0000","last_change":"2024-04-17T08:51:40.481925+0000","last_active":"2024-04-17T08:51:42.986885+0000","last_peered":"2024-04-17T08:51:42.986885+0000","last_clean":"2024-04-17T08:51:42.986885+0000","last_became_active":"2024-04-17T08:51:40.481436+0000","last_became_peered":"2024-04-17T08:51:40.481436+0000","last_unstale":"2024-04-17T08:51:42.986885+0000","last_undegraded":"2024-04-17T08:51:42.986885+0000","last_fullsized":"2024-04-17T08:51:42.986885+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:04.739470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510736+0000","last_change":"2024-04-17T08:51:41.508860+0000","last_active":"2024-04-17T08:51:41.510736+0000","last_peered":"2024-04-17T08:51:41.510736+0000","last_clean":"2024-04-17T08:51:41.510736+0000","last_became_active":"2024-04-17T08:51:41.508524+0000","last_became_peered":"2024-04-17T08:51:41.508524+0000","last_unstale":"2024-04-17T08:51:41.510736+0000","last_undegraded":"2024-04-17T08:51:41.510736+0000","last_fullsized":"2024-04-17T08:51:41.510736+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:41.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.476705+0000","last_change":"2024-04-17T08:51:40.479291+0000","last_active":"2024-04-17T08:51:41.476705+0000","last_peered":"2024-04-17T08:51:41.476705+0000","last_clean":"2024-04-17T08:51:41.476705+0000","last_became_active":"2024-04-17T08:51:40.476885+0000","last_became_peered":"2024-04-17T08:51:40.476885+0000","last_unstale":"2024-04-17T08:51:41.476705+0000","last_undegraded":"2024-04-17T08:51:41.476705+0000","last_fullsized":"2024-04-17T08:51:41.476705+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:53.966461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516541+0000","last_change":"2024-04-17T08:51:41.509196+0000","last_active":"2024-04-17T08:51:41.516541+0000","last_peered":"2024-04-17T08:51:41.516541+0000","last_clean":"2024-04-17T08:51:41.516541+0000","last_became_active":"2024-04-17T08:51:41.508917+0000","last_became_peered":"2024-04-17T08:51:41.508917+0000","last_unstale":"2024-04-17T08:51:41.516541+0000","last_undegraded":"2024-04-17T08:51:41.516541+0000","last_fullsized":"2024-04-17T08:51:41.516541+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:11:24.461261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486989+0000","last_change":"2024-04-17T08:51:40.483627+0000","last_active":"2024-04-17T08:51:42.486989+0000","last_peered":"2024-04-17T08:51:42.486989+0000","last_clean":"2024-04-17T08:51:42.486989+0000","last_became_active":"2024-04-17T08:51:40.482318+0000","last_became_peered":"2024-04-17T08:51:40.482318+0000","last_unstale":"2024-04-17T08:51:42.486989+0000","last_undegraded":"2024-04-17T08:51:42.486989+0000","last_fullsized":"2024-04-17T08:51:42.486989+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.761124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486412+0000","last_change":"2024-04-17T08:51:41.517149+0000","last_active":"2024-04-17T08:51:42.486412+0000","last_peered":"2024-04-17T08:51:42.486412+0000","last_clean":"2024-04-17T08:51:42.486412+0000","last_became_active":"2024-04-17T08:51:41.516844+0000","last_became_peered":"2024-04-17T08:51:41.516844+0000","last_unstale":"2024-04-17T08:51:42.486412+0000","last_undegraded":"2024-04-17T08:51:42.486412+0000","last_fullsized":"2024-04-17T08:51:42.486412+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:37.293958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":22,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480358+0000","last_change":"2024-04-17T08:51:40.479653+0000","last_active":"2024-04-17T08:51:42.480358+0000","last_peered":"2024-04-17T08:51:42.480358+0000","last_clean":"2024-04-17T08:51:42.480358+0000","last_became_active":"2024-04-17T08:51:40.479107+0000","last_became_peered":"2024-04-17T08:51:40.479107+0000","last_unstale":"2024-04-17T08:51:42.480358+0000","last_undegraded":"2024-04-17T08:51:42.480358+0000","last_fullsized":"2024-04-17T08:51:42.480358+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:16.542860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516904+0000","last_change":"2024-04-17T08:51:41.510891+0000","last_active":"2024-04-17T08:51:41.516904+0000","last_peered":"2024-04-17T08:51:41.516904+0000","last_clean":"2024-04-17T08:51:41.516904+0000","last_became_active":"2024-04-17T08:51:41.510538+0000","last_became_peered":"2024-04-17T08:51:41.510538+0000","last_unstale":"2024-04-17T08:51:41.516904+0000","last_undegraded":"2024-04-17T08:51:41.516904+0000","last_fullsized":"2024-04-17T08:51:41.516904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:38.420980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487930+0000","last_change":"2024-04-17T08:51:40.484060+0000","last_active":"2024-04-17T08:51:42.487930+0000","last_peered":"2024-04-17T08:51:42.487930+0000","last_clean":"2024-04-17T08:51:42.487930+0000","last_became_active":"2024-04-17T08:51:40.483692+0000","last_became_peered":"2024-04-17T08:51:40.483692+0000","last_unstale":"2024-04-17T08:51:42.487930+0000","last_undegraded":"2024-04-17T08:51:42.487930+0000","last_fullsized":"2024-04-17T08:51:42.487930+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:53.270393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.517677+0000","last_change":"2024-04-17T08:51:41.516369+0000","last_active":"2024-04-17T08:51:41.517677+0000","last_peered":"2024-04-17T08:51:41.517677+0000","last_clean":"2024-04-17T08:51:41.517677+0000","last_became_active":"2024-04-17T08:51:41.515919+0000","last_became_peered":"2024-04-17T08:51:41.515919+0000","last_unstale":"2024-04-17T08:51:41.517677+0000","last_undegraded":"2024-04-17T08:51:41.517677+0000","last_fullsized":"2024-04-17T08:51:41.517677+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:38.001673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525899+0000","last_change":"2024-04-17T08:51:40.466869+0000","last_active":"2024-04-17T08:51:41.525899+0000","last_peered":"2024-04-17T08:51:41.525899+0000","last_clean":"2024-04-17T08:51:41.525899+0000","last_became_active":"2024-04-17T08:51:40.465973+0000","last_became_peered":"2024-04-17T08:51:40.465973+0000","last_unstale":"2024-04-17T08:51:41.525899+0000","last_undegraded":"2024-04-17T08:51:41.525899+0000","last_fullsized":"2024-04-17T08:51:41.525899+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:57.022722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510339+0000","last_change":"2024-04-17T08:51:41.507082+0000","last_active":"2024-04-17T08:51:41.510339+0000","last_peered":"2024-04-17T08:51:41.510339+0000","last_clean":"2024-04-17T08:51:41.510339+0000","last_became_active":"2024-04-17T08:51:41.506812+0000","last_became_peered":"2024-04-17T08:51:41.506812+0000","last_unstale":"2024-04-17T08:51:41.510339+0000","last_undegraded":"2024-04-17T08:51:41.510339+0000","last_fullsized":"2024-04-17T08:51:41.510339+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:26.370488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.512385+0000","last_change":"2024-04-17T08:51:40.481626+0000","last_active":"2024-04-17T08:51:41.512385+0000","last_peered":"2024-04-17T08:51:41.512385+0000","last_clean":"2024-04-17T08:51:41.512385+0000","last_became_active":"2024-04-17T08:51:40.481268+0000","last_became_peered":"2024-04-17T08:51:40.481268+0000","last_unstale":"2024-04-17T08:51:41.512385+0000","last_undegraded":"2024-04-17T08:51:41.512385+0000","last_fullsized":"2024-04-17T08:51:41.512385+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:35.353434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.485159+0000","last_change":"2024-04-17T08:51:41.503769+0000","last_active":"2024-04-17T08:51:42.485159+0000","last_peered":"2024-04-17T08:51:42.485159+0000","last_clean":"2024-04-17T08:51:42.485159+0000","last_became_active":"2024-04-17T08:51:41.503051+0000","last_became_peered":"2024-04-17T08:51:41.503051+0000","last_unstale":"2024-04-17T08:51:42.485159+0000","last_undegraded":"2024-04-17T08:51:42.485159+0000","last_fullsized":"2024-04-17T08:51:42.485159+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:40.349134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481672+0000","last_change":"2024-04-17T08:51:40.482915+0000","last_active":"2024-04-17T08:51:42.481672+0000","last_peered":"2024-04-17T08:51:42.481672+0000","last_clean":"2024-04-17T08:51:42.481672+0000","last_became_active":"2024-04-17T08:51:40.482629+0000","last_became_peered":"2024-04-17T08:51:40.482629+0000","last_unstale":"2024-04-17T08:51:42.481672+0000","last_undegraded":"2024-04-17T08:51:42.481672+0000","last_fullsized":"2024-04-17T08:51:42.481672+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.115195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483345+0000","last_change":"2024-04-17T08:51:41.518701+0000","last_active":"2024-04-17T08:51:42.483345+0000","last_peered":"2024-04-17T08:51:42.483345+0000","last_clean":"2024-04-17T08:51:42.483345+0000","last_became_active":"2024-04-17T08:51:41.516124+0000","last_became_peered":"2024-04-17T08:51:41.516124+0000","last_unstale":"2024-04-17T08:51:42.483345+0000","last_undegraded":"2024-04-17T08:51:42.483345+0000","last_fullsized":"2024-04-17T08:51:42.483345+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.524888+0000","last_change":"2024-04-17T08:51:40.483946+0000","last_active":"2024-04-17T08:51:41.524888+0000","last_peered":"2024-04-17T08:51:41.524888+0000","last_clean":"2024-04-17T08:51:41.524888+0000","last_became_active":"2024-04-17T08:51:40.483664+0000","last_became_peered":"2024-04-17T08:51:40.483664+0000","last_unstale":"2024-04-17T08:51:41.524888+0000","last_undegraded":"2024-04-17T08:51:41.524888+0000","last_fullsized":"2024-04-17T08:51:41.524888+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:55.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986451+0000","last_change":"2024-04-17T08:51:41.519618+0000","last_active":"2024-04-17T08:51:42.986451+0000","last_peered":"2024-04-17T08:51:42.986451+0000","last_clean":"2024-04-17T08:51:42.986451+0000","last_became_active":"2024-04-17T08:51:41.519247+0000","last_became_peered":"2024-04-17T08:51:41.519247+0000","last_unstale":"2024-04-17T08:51:42.986451+0000","last_undegraded":"2024-04-17T08:51:42.986451+0000","last_fullsized":"2024-04-17T08:51:42.986451+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:32.995157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480822+0000","last_change":"2024-04-17T08:51:40.486177+0000","last_active":"2024-04-17T08:51:42.480822+0000","last_peered":"2024-04-17T08:51:42.480822+0000","last_clean":"2024-04-17T08:51:42.480822+0000","last_became_active":"2024-04-17T08:51:40.484934+0000","last_became_peered":"2024-04-17T08:51:40.484934+0000","last_unstale":"2024-04-17T08:51:42.480822+0000","last_undegraded":"2024-04-17T08:51:42.480822+0000","last_fullsized":"2024-04-17T08:51:42.480822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:23.956487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480339+0000","last_change":"2024-04-17T08:51:41.517111+0000","last_active":"2024-04-17T08:51:42.480339+0000","last_peered":"2024-04-17T08:51:42.480339+0000","last_clean":"2024-04-17T08:51:42.480339+0000","last_became_active":"2024-04-17T08:51:41.516756+0000","last_became_peered":"2024-04-17T08:51:41.516756+0000","last_unstale":"2024-04-17T08:51:42.480339+0000","last_undegraded":"2024-04-17T08:51:42.480339+0000","last_fullsized":"2024-04-17T08:51:42.480339+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:17:58.048828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.490994+0000","last_change":"2024-04-17T08:51:40.479934+0000","last_active":"2024-04-17T08:51:41.490994+0000","last_peered":"2024-04-17T08:51:41.490994+0000","last_clean":"2024-04-17T08:51:41.490994+0000","last_became_active":"2024-04-17T08:51:40.479295+0000","last_became_peered":"2024-04-17T08:51:40.479295+0000","last_unstale":"2024-04-17T08:51:41.490994+0000","last_undegraded":"2024-04-17T08:51:41.490994+0000","last_fullsized":"2024-04-17T08:51:41.490994+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:40.469305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483359+0000","last_change":"2024-04-17T08:51:41.500141+0000","last_active":"2024-04-17T08:51:42.483359+0000","last_peered":"2024-04-17T08:51:42.483359+0000","last_clean":"2024-04-17T08:51:42.483359+0000","last_became_active":"2024-04-17T08:51:41.499513+0000","last_became_peered":"2024-04-17T08:51:41.499513+0000","last_unstale":"2024-04-17T08:51:42.483359+0000","last_undegraded":"2024-04-17T08:51:42.483359+0000","last_fullsized":"2024-04-17T08:51:42.483359+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486787+0000","last_change":"2024-04-17T08:51:40.478664+0000","last_active":"2024-04-17T08:51:42.486787+0000","last_peered":"2024-04-17T08:51:42.486787+0000","last_clean":"2024-04-17T08:51:42.486787+0000","last_became_active":"2024-04-17T08:51:40.477953+0000","last_became_peered":"2024-04-17T08:51:40.477953+0000","last_unstale":"2024-04-17T08:51:42.486787+0000","last_undegraded":"2024-04-17T08:51:42.486787+0000","last_fullsized":"2024-04-17T08:51:42.486787+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:54.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488347+0000","last_change":"2024-04-17T08:51:41.515604+0000","last_active":"2024-04-17T08:51:42.488347+0000","last_peered":"2024-04-17T08:51:42.488347+0000","last_clean":"2024-04-17T08:51:42.488347+0000","last_became_active":"2024-04-17T08:51:41.514707+0000","last_became_peered":"2024-04-17T08:51:41.514707+0000","last_unstale":"2024-04-17T08:51:42.488347+0000","last_undegraded":"2024-04-17T08:51:42.488347+0000","last_fullsized":"2024-04-17T08:51:42.488347+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.518687+0000","last_change":"2024-04-17T08:51:40.481823+0000","last_active":"2024-04-17T08:51:41.518687+0000","last_peered":"2024-04-17T08:51:41.518687+0000","last_clean":"2024-04-17T08:51:41.518687+0000","last_became_active":"2024-04-17T08:51:40.481302+0000","last_became_peered":"2024-04-17T08:51:40.481302+0000","last_unstale":"2024-04-17T08:51:41.518687+0000","last_undegraded":"2024-04-17T08:51:41.518687+0000","last_fullsized":"2024-04-17T08:51:41.518687+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:56.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987380+0000","last_change":"2024-04-17T08:51:41.498889+0000","last_active":"2024-04-17T08:51:42.987380+0000","last_peered":"2024-04-17T08:51:42.987380+0000","last_clean":"2024-04-17T08:51:42.987380+0000","last_became_active":"2024-04-17T08:51:41.498622+0000","last_became_peered":"2024-04-17T08:51:41.498622+0000","last_unstale":"2024-04-17T08:51:42.987380+0000","last_undegraded":"2024-04-17T08:51:42.987380+0000","last_fullsized":"2024-04-17T08:51:42.987380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:42.312039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.494313+0000","last_change":"2024-04-17T08:51:40.482579+0000","last_active":"2024-04-17T08:51:43.494313+0000","last_peered":"2024-04-17T08:51:43.494313+0000","last_clean":"2024-04-17T08:51:43.494313+0000","last_became_active":"2024-04-17T08:51:40.482255+0000","last_became_peered":"2024-04-17T08:51:40.482255+0000","last_unstale":"2024-04-17T08:51:43.494313+0000","last_undegraded":"2024-04-17T08:51:43.494313+0000","last_fullsized":"2024-04-17T08:51:43.494313+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:37.553047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481422+0000","last_change":"2024-04-17T08:51:41.514216+0000","last_active":"2024-04-17T08:51:42.481422+0000","last_peered":"2024-04-17T08:51:42.481422+0000","last_clean":"2024-04-17T08:51:42.481422+0000","last_became_active":"2024-04-17T08:51:41.513937+0000","last_became_peered":"2024-04-17T08:51:41.513937+0000","last_unstale":"2024-04-17T08:51:42.481422+0000","last_undegraded":"2024-04-17T08:51:42.481422+0000","last_fullsized":"2024-04-17T08:51:42.481422+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:27.955059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.493238+0000","last_change":"2024-04-17T08:51:40.482279+0000","last_active":"2024-04-17T08:51:41.493238+0000","last_peered":"2024-04-17T08:51:41.493238+0000","last_clean":"2024-04-17T08:51:41.493238+0000","last_became_active":"2024-04-17T08:51:40.482002+0000","last_became_peered":"2024-04-17T08:51:40.482002+0000","last_unstale":"2024-04-17T08:51:41.493238+0000","last_undegraded":"2024-04-17T08:51:41.493238+0000","last_fullsized":"2024-04-17T08:51:41.493238+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:36.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482199+0000","last_change":"2024-04-17T08:51:41.505389+0000","last_active":"2024-04-17T08:51:42.482199+0000","last_peered":"2024-04-17T08:51:42.482199+0000","last_clean":"2024-04-17T08:51:42.482199+0000","last_became_active":"2024-04-17T08:51:41.503336+0000","last_became_peered":"2024-04-17T08:51:41.503336+0000","last_unstale":"2024-04-17T08:51:42.482199+0000","last_undegraded":"2024-04-17T08:51:42.482199+0000","last_fullsized":"2024-04-17T08:51:42.482199+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:16.345152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481774+0000","last_change":"2024-04-17T08:51:40.480656+0000","last_active":"2024-04-17T08:51:42.481774+0000","last_peered":"2024-04-17T08:51:42.481774+0000","last_clean":"2024-04-17T08:51:42.481774+0000","last_became_active":"2024-04-17T08:51:40.480386+0000","last_became_peered":"2024-04-17T08:51:40.480386+0000","last_unstale":"2024-04-17T08:51:42.481774+0000","last_undegraded":"2024-04-17T08:51:42.481774+0000","last_fullsized":"2024-04-17T08:51:42.481774+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.113174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487258+0000","last_change":"2024-04-17T08:51:41.513350+0000","last_active":"2024-04-17T08:51:42.487258+0000","last_peered":"2024-04-17T08:51:42.487258+0000","last_clean":"2024-04-17T08:51:42.487258+0000","last_became_active":"2024-04-17T08:51:41.512524+0000","last_became_peered":"2024-04-17T08:51:41.512524+0000","last_unstale":"2024-04-17T08:51:42.487258+0000","last_undegraded":"2024-04-17T08:51:42.487258+0000","last_fullsized":"2024-04-17T08:51:42.487258+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:39.456694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487972+0000","last_change":"2024-04-17T08:51:40.480678+0000","last_active":"2024-04-17T08:51:42.487972+0000","last_peered":"2024-04-17T08:51:42.487972+0000","last_clean":"2024-04-17T08:51:42.487972+0000","last_became_active":"2024-04-17T08:51:40.480360+0000","last_became_peered":"2024-04-17T08:51:40.480360+0000","last_unstale":"2024-04-17T08:51:42.487972+0000","last_undegraded":"2024-04-17T08:51:42.487972+0000","last_fullsized":"2024-04-17T08:51:42.487972+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:50.970831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484365+0000","last_change":"2024-04-17T08:51:41.501267+0000","last_active":"2024-04-17T08:51:42.484365+0000","last_peered":"2024-04-17T08:51:42.484365+0000","last_clean":"2024-04-17T08:51:42.484365+0000","last_became_active":"2024-04-17T08:51:41.500983+0000","last_became_peered":"2024-04-17T08:51:41.500983+0000","last_unstale":"2024-04-17T08:51:42.484365+0000","last_undegraded":"2024-04-17T08:51:42.484365+0000","last_fullsized":"2024-04-17T08:51:42.484365+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:53.940422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.494453+0000","last_change":"2024-04-17T08:51:40.477982+0000","last_active":"2024-04-17T08:51:41.494453+0000","last_peered":"2024-04-17T08:51:41.494453+0000","last_clean":"2024-04-17T08:51:41.494453+0000","last_became_active":"2024-04-17T08:51:40.477717+0000","last_became_peered":"2024-04-17T08:51:40.477717+0000","last_unstale":"2024-04-17T08:51:41.494453+0000","last_undegraded":"2024-04-17T08:51:41.494453+0000","last_fullsized":"2024-04-17T08:51:41.494453+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:51.232388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482878+0000","last_change":"2024-04-17T08:51:41.504227+0000","last_active":"2024-04-17T08:51:42.482878+0000","last_peered":"2024-04-17T08:51:42.482878+0000","last_clean":"2024-04-17T08:51:42.482878+0000","last_became_active":"2024-04-17T08:51:41.503919+0000","last_became_peered":"2024-04-17T08:51:41.503919+0000","last_unstale":"2024-04-17T08:51:42.482878+0000","last_undegraded":"2024-04-17T08:51:42.482878+0000","last_fullsized":"2024-04-17T08:51:42.482878+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477745+0000","last_change":"2024-04-17T08:51:40.466461+0000","last_active":"2024-04-17T08:51:41.477745+0000","last_peered":"2024-04-17T08:51:41.477745+0000","last_clean":"2024-04-17T08:51:41.477745+0000","last_became_active":"2024-04-17T08:51:40.466050+0000","last_became_peered":"2024-04-17T08:51:40.466050+0000","last_unstale":"2024-04-17T08:51:41.477745+0000","last_undegraded":"2024-04-17T08:51:41.477745+0000","last_fullsized":"2024-04-17T08:51:41.477745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:17.871122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986557+0000","last_change":"2024-04-17T08:51:41.509452+0000","last_active":"2024-04-17T08:51:42.986557+0000","last_peered":"2024-04-17T08:51:42.986557+0000","last_clean":"2024-04-17T08:51:42.986557+0000","last_became_active":"2024-04-17T08:51:41.509160+0000","last_became_peered":"2024-04-17T08:51:41.509160+0000","last_unstale":"2024-04-17T08:51:42.986557+0000","last_undegraded":"2024-04-17T08:51:42.986557+0000","last_fullsized":"2024-04-17T08:51:42.986557+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:54.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477208+0000","last_change":"2024-04-17T08:51:40.481750+0000","last_active":"2024-04-17T08:51:41.477208+0000","last_peered":"2024-04-17T08:51:41.477208+0000","last_clean":"2024-04-17T08:51:41.477208+0000","last_became_active":"2024-04-17T08:51:40.481311+0000","last_became_peered":"2024-04-17T08:51:40.481311+0000","last_unstale":"2024-04-17T08:51:41.477208+0000","last_undegraded":"2024-04-17T08:51:41.477208+0000","last_fullsized":"2024-04-17T08:51:41.477208+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:50.295078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.491070+0000","last_change":"2024-04-17T08:51:41.490585+0000","last_active":"2024-04-17T08:51:43.491070+0000","last_peered":"2024-04-17T08:51:43.491070+0000","last_clean":"2024-04-17T08:51:43.491070+0000","last_became_active":"2024-04-17T08:51:41.489827+0000","last_became_peered":"2024-04-17T08:51:41.489827+0000","last_unstale":"2024-04-17T08:51:43.491070+0000","last_undegraded":"2024-04-17T08:51:43.491070+0000","last_fullsized":"2024-04-17T08:51:43.491070+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:35.513878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525671+0000","last_change":"2024-04-17T08:51:40.482218+0000","last_active":"2024-04-17T08:51:41.525671+0000","last_peered":"2024-04-17T08:51:41.525671+0000","last_clean":"2024-04-17T08:51:41.525671+0000","last_became_active":"2024-04-17T08:51:40.481390+0000","last_became_peered":"2024-04-17T08:51:40.481390+0000","last_unstale":"2024-04-17T08:51:41.525671+0000","last_undegraded":"2024-04-17T08:51:41.525671+0000","last_fullsized":"2024-04-17T08:51:41.525671+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:41.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"236'376","reported_seq":1563,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.498615+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:51:41.498615+0000","last_peered":"2024-04-17T08:51:41.498615+0000","last_clean":"2024-04-17T08:51:41.498615+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:51:41.498615+0000","last_undegraded":"2024-04-17T08:51:41.498615+0000","last_fullsized":"2024-04-17T08:51:41.498615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":376,"ondisk_log_size":376,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543306,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57032,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314808,"statfs":{"total":96636764160,"available":96578363392,"internally_reserved":0,"allocated":3727360,"data_stored":2126458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543306,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60632,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311208,"statfs":{"total":96636764160,"available":96574676992,"internally_reserved":0,"allocated":3149824,"data_stored":1542952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543305,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30092,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341748,"statfs":{"total":96636764160,"available":96605949952,"internally_reserved":0,"allocated":3141632,"data_stored":1542348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66144,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305696,"statfs":{"total":96636764160,"available":96569032704,"internally_reserved":0,"allocated":3162112,"data_stored":1544052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543307,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52292,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319548,"statfs":{"total":96636764160,"available":96583217152,"internally_reserved":0,"allocated":3133440,"data_stored":1536602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543306,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52292,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319548,"statfs":{"total":96636764160,"available":96583217152,"internally_reserved":0,"allocated":3133440,"data_stored":1536110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":3162112,"data_stored":1544084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543305,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":3739648,"data_stored":2132750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:51:46.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:51:46.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.312+0000 7f685b200640 1 Processor -- start 2024-04-17T08:51:46.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.312+0000 7f685b200640 1 -- start start 2024-04-17T08:51:46.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f685b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f685c0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f685b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685c15ce00 con 0x7f685c15ca20 2024-04-17T08:51:46.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f685b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685c15d090 con 0x7f685c09aa60 2024-04-17T08:51:46.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f685b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f685c15d330 con 0x7f685c09aee0 2024-04-17T08:51:46.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f6859e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f685c15ca20 0x7f685c0d01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59490/0 (socket says 172.21.15.67:59490) 2024-04-17T08:51:46.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.314+0000 7f6859e00640 1 -- 172.21.15.67:0/3919851709 learned_addr learned my addr 172.21.15.67:0/3919851709 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f685a800640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f685c0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569722171 0 0) 0x7f685c15ce00 con 0x7f685c15ca20 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6840003520 con 0x7f685c15ca20 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2866738642 0 0) 0x7f6840003520 con 0x7f685c15ca20 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 msgr2=0x7f685c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f685c0cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 >> v1:172.21.15.187:6789/0 conn(0x7f685c09aa60 legacy=0x7f685c0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c15d610 con 0x7f685c15ca20 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f684c001670 con 0x7f685c15ca20 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 >> v1:172.21.15.67:6789/0 conn(0x7f685c15ca20 legacy=0x7f685c0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f6840003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840004020 0x7f6840004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004d40 0x7f68400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f685c15d330 con 0x7f6840004020 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f684000bb80 con 0x7f6840004d40 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.315+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f684000bd80 con 0x7f685c09aee0 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840004020 0x7f6840004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859e00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f6840003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f685a800640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004d40 0x7f68400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 -- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 msgr2=0x7f6840003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f6840003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 -- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004d40 msgr2=0x7f68400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004d40 0x7f68400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859400640 1 -- 172.21.15.67:0/3919851709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6840003520 con 0x7f6840004020 2024-04-17T08:51:46.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.316+0000 7f6859e00640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c09aee0 0x7f6840003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.317+0000 7f6859400640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840004020 0x7f6840004800 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f6844001720 tx=0x7f6840005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4dae03e6807e655 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844001370 con 0x7f6840004020 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68440052d0 con 0x7f6840004020 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f6858a00640 1 -- 172.21.15.67:0/3919851709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844003830 con 0x7f6840004020 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 -- 172.21.15.67:0/3919851709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840004020 msgr2=0x7f6840004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 --2- 172.21.15.67:0/3919851709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840004020 0x7f6840004800 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f6844001720 tx=0x7f6840005830 comp rx=0 tx=0).stop 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685a800640 1 -- 172.21.15.67:0/3919851709 reap_dead start 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 -- 172.21.15.67:0/3919851709 shutdown_connections 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 -- 172.21.15.67:0/3919851709 >> 172.21.15.67:0/3919851709 conn(0x7f685c0b4100 msgr2=0x7f685c0b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 -- 172.21.15.67:0/3919851709 shutdown_connections 2024-04-17T08:51:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.318+0000 7f685b200640 1 -- 172.21.15.67:0/3919851709 wait complete. 2024-04-17T08:51:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.320+0000 7f685b200640 1 Processor -- start 2024-04-17T08:51:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.320+0000 7f685b200640 1 -- start start 2024-04-17T08:51:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 0x7f685c0b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c0c6120 0x7f685c1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f685c0d08f0 con 0x7f685c06c9d0 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f685c09b330 con 0x7f6840004020 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.321+0000 7f685b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f685c09bd20 con 0x7f685c0c6120 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33268/0 (socket says 172.21.15.67:33268) 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 -- 172.21.15.67:0/1005951707 learned_addr learned my addr 172.21.15.67:0/1005951707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f6859e00640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 0x7f685c0b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f6859400640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c0c6120 0x7f685c1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 -- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c0c6120 msgr2=0x7f685c1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c0c6120 0x7f685c1506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 -- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 msgr2=0x7f685c0b7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 0x7f685c0b7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f685a800640 1 -- 172.21.15.67:0/1005951707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c0ce370 con 0x7f685c06c9d0 2024-04-17T08:51:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.322+0000 7f6859e00640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 0x7f685c0b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f685a800640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f685c0733a0 tx=0x7f68540115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854011e10 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685401b070 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c0ce650 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854003990 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.323+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c0cec10 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.324+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f685c0755d0 con 0x7f685c06c9d0 2024-04-17T08:51:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.325+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6854005d10 con 0x7f685c06c9d0 2024-04-17T08:51:46.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.326+0000 7f6852000640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 0x7f683806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.326+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f685408c420 con 0x7f685c06c9d0 2024-04-17T08:51:46.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.327+0000 7f6859e00640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 0x7f683806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.328+0000 7f6859e00640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 0x7f683806f020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f684c005a90 tx=0x7f684c009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.331+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6854014350 con 0x7f685c06c9d0 2024-04-17T08:51:46.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.534+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f685c09b590 con 0x7f683806cb60 2024-04-17T08:51:46.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.555+0000 7f6852000640 1 -- 172.21.15.67:0/1005951707 <== mgr.8926 v2:172.21.15.187:6832/2123602634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252482 (secure 0 0 0) 0x7f685c09b590 con 0x7f683806cb60 2024-04-17T08:51:46.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 msgr2=0x7f683806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 0x7f683806f020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f684c005a90 tx=0x7f684c009040 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 msgr2=0x7f685c06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f685c0733a0 tx=0x7f68540115a0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 shutdown_connections 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f683806cb60 0x7f683806f020 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f685c06c9d0 0x7f685c06ce80 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f685c0c6120 0x7f685c1506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 --2- 172.21.15.67:0/1005951707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6840004020 0x7f685c0b7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 >> 172.21.15.67:0/1005951707 conn(0x7f685c0b4100 msgr2=0x7f685c15f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 shutdown_connections 2024-04-17T08:51:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.561+0000 7f685b200640 1 -- 172.21.15.67:0/1005951707 wait complete. 2024-04-17T08:51:46.583 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-17T08:51:44.773239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":672,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":415,"ondisk_log_size":415,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444484,"kb_used_data":25732,"kb_used_omap":12,"kb_used_meta":418611,"kb_avail":754530236,"statfs":{"total":773094113280,"available":772638961664,"internally_reserved":0,"allocated":26349568,"data_stored":13505356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428658224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.334314"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987293+0000","last_change":"2024-04-17T08:51:41.515286+0000","last_active":"2024-04-17T08:51:42.987293+0000","last_peered":"2024-04-17T08:51:42.987293+0000","last_clean":"2024-04-17T08:51:42.987293+0000","last_became_active":"2024-04-17T08:51:41.515011+0000","last_became_peered":"2024-04-17T08:51:41.515011+0000","last_unstale":"2024-04-17T08:51:42.987293+0000","last_undegraded":"2024-04-17T08:51:42.987293+0000","last_fullsized":"2024-04-17T08:51:42.987293+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:27.187699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987373+0000","last_change":"2024-04-17T08:51:41.505707+0000","last_active":"2024-04-17T08:51:42.987373+0000","last_peered":"2024-04-17T08:51:42.987373+0000","last_clean":"2024-04-17T08:51:42.987373+0000","last_became_active":"2024-04-17T08:51:41.505278+0000","last_became_peered":"2024-04-17T08:51:41.505278+0000","last_unstale":"2024-04-17T08:51:42.987373+0000","last_undegraded":"2024-04-17T08:51:42.987373+0000","last_fullsized":"2024-04-17T08:51:42.987373+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:17.019190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.511053+0000","last_change":"2024-04-17T08:51:41.506466+0000","last_active":"2024-04-17T08:51:41.511053+0000","last_peered":"2024-04-17T08:51:41.511053+0000","last_clean":"2024-04-17T08:51:41.511053+0000","last_became_active":"2024-04-17T08:51:41.506181+0000","last_became_peered":"2024-04-17T08:51:41.506181+0000","last_unstale":"2024-04-17T08:51:41.511053+0000","last_undegraded":"2024-04-17T08:51:41.511053+0000","last_fullsized":"2024-04-17T08:51:41.511053+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:33.370084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515311+0000","last_change":"2024-04-17T08:51:41.503459+0000","last_active":"2024-04-17T08:51:41.515311+0000","last_peered":"2024-04-17T08:51:41.515311+0000","last_clean":"2024-04-17T08:51:41.515311+0000","last_became_active":"2024-04-17T08:51:41.503139+0000","last_became_peered":"2024-04-17T08:51:41.503139+0000","last_unstale":"2024-04-17T08:51:41.515311+0000","last_undegraded":"2024-04-17T08:51:41.515311+0000","last_fullsized":"2024-04-17T08:51:41.515311+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:14.891561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.490971+0000","last_change":"2024-04-17T08:51:41.515918+0000","last_active":"2024-04-17T08:51:43.490971+0000","last_peered":"2024-04-17T08:51:43.490971+0000","last_clean":"2024-04-17T08:51:43.490971+0000","last_became_active":"2024-04-17T08:51:41.515377+0000","last_became_peered":"2024-04-17T08:51:41.515377+0000","last_unstale":"2024-04-17T08:51:43.490971+0000","last_undegraded":"2024-04-17T08:51:43.490971+0000","last_fullsized":"2024-04-17T08:51:43.490971+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:51:05.859833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.503789+0000","last_change":"2024-04-17T08:51:41.497092+0000","last_active":"2024-04-17T08:51:41.503789+0000","last_peered":"2024-04-17T08:51:41.503789+0000","last_clean":"2024-04-17T08:51:41.503789+0000","last_became_active":"2024-04-17T08:51:41.496813+0000","last_became_peered":"2024-04-17T08:51:41.496813+0000","last_unstale":"2024-04-17T08:51:41.503789+0000","last_undegraded":"2024-04-17T08:51:41.503789+0000","last_fullsized":"2024-04-17T08:51:41.503789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:19:43.614868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486423+0000","last_change":"2024-04-17T08:51:41.512828+0000","last_active":"2024-04-17T08:51:42.486423+0000","last_peered":"2024-04-17T08:51:42.486423+0000","last_clean":"2024-04-17T08:51:42.486423+0000","last_became_active":"2024-04-17T08:51:41.512495+0000","last_became_peered":"2024-04-17T08:51:41.512495+0000","last_unstale":"2024-04-17T08:51:42.486423+0000","last_undegraded":"2024-04-17T08:51:42.486423+0000","last_fullsized":"2024-04-17T08:51:42.486423+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:59.808221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482730+0000","last_change":"2024-04-17T08:51:41.505463+0000","last_active":"2024-04-17T08:51:42.482730+0000","last_peered":"2024-04-17T08:51:42.482730+0000","last_clean":"2024-04-17T08:51:42.482730+0000","last_became_active":"2024-04-17T08:51:41.505186+0000","last_became_peered":"2024-04-17T08:51:41.505186+0000","last_unstale":"2024-04-17T08:51:42.482730+0000","last_undegraded":"2024-04-17T08:51:42.482730+0000","last_fullsized":"2024-04-17T08:51:42.482730+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.505449+0000","last_change":"2024-04-17T08:51:41.503091+0000","last_active":"2024-04-17T08:51:41.505449+0000","last_peered":"2024-04-17T08:51:41.505449+0000","last_clean":"2024-04-17T08:51:41.505449+0000","last_became_active":"2024-04-17T08:51:41.502796+0000","last_became_peered":"2024-04-17T08:51:41.502796+0000","last_unstale":"2024-04-17T08:51:41.505449+0000","last_undegraded":"2024-04-17T08:51:41.505449+0000","last_fullsized":"2024-04-17T08:51:41.505449+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:47.830902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488230+0000","last_change":"2024-04-17T08:51:41.517488+0000","last_active":"2024-04-17T08:51:42.488230+0000","last_peered":"2024-04-17T08:51:42.488230+0000","last_clean":"2024-04-17T08:51:42.488230+0000","last_became_active":"2024-04-17T08:51:41.517224+0000","last_became_peered":"2024-04-17T08:51:41.517224+0000","last_unstale":"2024-04-17T08:51:42.488230+0000","last_undegraded":"2024-04-17T08:51:42.488230+0000","last_fullsized":"2024-04-17T08:51:42.488230+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481969+0000","last_change":"2024-04-17T08:51:41.520584+0000","last_active":"2024-04-17T08:51:42.481969+0000","last_peered":"2024-04-17T08:51:42.481969+0000","last_clean":"2024-04-17T08:51:42.481969+0000","last_became_active":"2024-04-17T08:51:41.520134+0000","last_became_peered":"2024-04-17T08:51:41.520134+0000","last_unstale":"2024-04-17T08:51:42.481969+0000","last_undegraded":"2024-04-17T08:51:42.481969+0000","last_fullsized":"2024-04-17T08:51:42.481969+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:25.042374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484401+0000","last_change":"2024-04-17T08:51:41.518003+0000","last_active":"2024-04-17T08:51:42.484401+0000","last_peered":"2024-04-17T08:51:42.484401+0000","last_clean":"2024-04-17T08:51:42.484401+0000","last_became_active":"2024-04-17T08:51:41.517732+0000","last_became_peered":"2024-04-17T08:51:41.517732+0000","last_unstale":"2024-04-17T08:51:42.484401+0000","last_undegraded":"2024-04-17T08:51:42.484401+0000","last_fullsized":"2024-04-17T08:51:42.484401+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:55.524149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487551+0000","last_change":"2024-04-17T08:51:41.514813+0000","last_active":"2024-04-17T08:51:42.487551+0000","last_peered":"2024-04-17T08:51:42.487551+0000","last_clean":"2024-04-17T08:51:42.487551+0000","last_became_active":"2024-04-17T08:51:41.512779+0000","last_became_peered":"2024-04-17T08:51:41.512779+0000","last_unstale":"2024-04-17T08:51:42.487551+0000","last_undegraded":"2024-04-17T08:51:42.487551+0000","last_fullsized":"2024-04-17T08:51:42.487551+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986299+0000","last_change":"2024-04-17T08:51:41.507865+0000","last_active":"2024-04-17T08:51:42.986299+0000","last_peered":"2024-04-17T08:51:42.986299+0000","last_clean":"2024-04-17T08:51:42.986299+0000","last_became_active":"2024-04-17T08:51:41.507622+0000","last_became_peered":"2024-04-17T08:51:41.507622+0000","last_unstale":"2024-04-17T08:51:42.986299+0000","last_undegraded":"2024-04-17T08:51:42.986299+0000","last_fullsized":"2024-04-17T08:51:42.986299+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.174140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.504424+0000","last_change":"2024-04-17T08:51:41.501853+0000","last_active":"2024-04-17T08:51:41.504424+0000","last_peered":"2024-04-17T08:51:41.504424+0000","last_clean":"2024-04-17T08:51:41.504424+0000","last_became_active":"2024-04-17T08:51:41.501575+0000","last_became_peered":"2024-04-17T08:51:41.501575+0000","last_unstale":"2024-04-17T08:51:41.504424+0000","last_undegraded":"2024-04-17T08:51:41.504424+0000","last_fullsized":"2024-04-17T08:51:41.504424+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:13.393507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516444+0000","last_change":"2024-04-17T08:51:41.513490+0000","last_active":"2024-04-17T08:51:41.516444+0000","last_peered":"2024-04-17T08:51:41.516444+0000","last_clean":"2024-04-17T08:51:41.516444+0000","last_became_active":"2024-04-17T08:51:41.513192+0000","last_became_peered":"2024-04-17T08:51:41.513192+0000","last_unstale":"2024-04-17T08:51:41.516444+0000","last_undegraded":"2024-04-17T08:51:41.516444+0000","last_fullsized":"2024-04-17T08:51:41.516444+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:36.367018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484601+0000","last_change":"2024-04-17T08:51:41.516221+0000","last_active":"2024-04-17T08:51:42.484601+0000","last_peered":"2024-04-17T08:51:42.484601+0000","last_clean":"2024-04-17T08:51:42.484601+0000","last_became_active":"2024-04-17T08:51:41.515896+0000","last_became_peered":"2024-04-17T08:51:41.515896+0000","last_unstale":"2024-04-17T08:51:42.484601+0000","last_undegraded":"2024-04-17T08:51:42.484601+0000","last_fullsized":"2024-04-17T08:51:42.484601+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:22.484668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987414+0000","last_change":"2024-04-17T08:51:41.501545+0000","last_active":"2024-04-17T08:51:42.987414+0000","last_peered":"2024-04-17T08:51:42.987414+0000","last_clean":"2024-04-17T08:51:42.987414+0000","last_became_active":"2024-04-17T08:51:41.500258+0000","last_became_peered":"2024-04-17T08:51:41.500258+0000","last_unstale":"2024-04-17T08:51:42.987414+0000","last_undegraded":"2024-04-17T08:51:42.987414+0000","last_fullsized":"2024-04-17T08:51:42.987414+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:24.809010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484390+0000","last_change":"2024-04-17T08:51:41.505156+0000","last_active":"2024-04-17T08:51:42.484390+0000","last_peered":"2024-04-17T08:51:42.484390+0000","last_clean":"2024-04-17T08:51:42.484390+0000","last_became_active":"2024-04-17T08:51:41.503363+0000","last_became_peered":"2024-04-17T08:51:41.503363+0000","last_unstale":"2024-04-17T08:51:42.484390+0000","last_undegraded":"2024-04-17T08:51:42.484390+0000","last_fullsized":"2024-04-17T08:51:42.484390+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:28.800799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.507117+0000","last_change":"2024-04-17T08:51:41.500145+0000","last_active":"2024-04-17T08:51:41.507117+0000","last_peered":"2024-04-17T08:51:41.507117+0000","last_clean":"2024-04-17T08:51:41.507117+0000","last_became_active":"2024-04-17T08:51:41.499316+0000","last_became_peered":"2024-04-17T08:51:41.499316+0000","last_unstale":"2024-04-17T08:51:41.507117+0000","last_undegraded":"2024-04-17T08:51:41.507117+0000","last_fullsized":"2024-04-17T08:51:41.507117+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:47:07.828822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483883+0000","last_change":"2024-04-17T08:51:41.508121+0000","last_active":"2024-04-17T08:51:42.483883+0000","last_peered":"2024-04-17T08:51:42.483883+0000","last_clean":"2024-04-17T08:51:42.483883+0000","last_became_active":"2024-04-17T08:51:41.506717+0000","last_became_peered":"2024-04-17T08:51:41.506717+0000","last_unstale":"2024-04-17T08:51:42.483883+0000","last_undegraded":"2024-04-17T08:51:42.483883+0000","last_fullsized":"2024-04-17T08:51:42.483883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:45.296249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486789+0000","last_change":"2024-04-17T08:51:41.517429+0000","last_active":"2024-04-17T08:51:42.486789+0000","last_peered":"2024-04-17T08:51:42.486789+0000","last_clean":"2024-04-17T08:51:42.486789+0000","last_became_active":"2024-04-17T08:51:41.517130+0000","last_became_peered":"2024-04-17T08:51:41.517130+0000","last_unstale":"2024-04-17T08:51:42.486789+0000","last_undegraded":"2024-04-17T08:51:42.486789+0000","last_fullsized":"2024-04-17T08:51:42.486789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:14.831099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483852+0000","last_change":"2024-04-17T08:51:41.507626+0000","last_active":"2024-04-17T08:51:42.483852+0000","last_peered":"2024-04-17T08:51:42.483852+0000","last_clean":"2024-04-17T08:51:42.483852+0000","last_became_active":"2024-04-17T08:51:41.507278+0000","last_became_peered":"2024-04-17T08:51:41.507278+0000","last_unstale":"2024-04-17T08:51:42.483852+0000","last_undegraded":"2024-04-17T08:51:42.483852+0000","last_fullsized":"2024-04-17T08:51:42.483852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986567+0000","last_change":"2024-04-17T08:51:41.502141+0000","last_active":"2024-04-17T08:51:42.986567+0000","last_peered":"2024-04-17T08:51:42.986567+0000","last_clean":"2024-04-17T08:51:42.986567+0000","last_became_active":"2024-04-17T08:51:41.501895+0000","last_became_peered":"2024-04-17T08:51:41.501895+0000","last_unstale":"2024-04-17T08:51:42.986567+0000","last_undegraded":"2024-04-17T08:51:42.986567+0000","last_fullsized":"2024-04-17T08:51:42.986567+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:45.139092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515944+0000","last_change":"2024-04-17T08:51:41.504007+0000","last_active":"2024-04-17T08:51:41.515944+0000","last_peered":"2024-04-17T08:51:41.515944+0000","last_clean":"2024-04-17T08:51:41.515944+0000","last_became_active":"2024-04-17T08:51:41.503745+0000","last_became_peered":"2024-04-17T08:51:41.503745+0000","last_unstale":"2024-04-17T08:51:41.515944+0000","last_undegraded":"2024-04-17T08:51:41.515944+0000","last_fullsized":"2024-04-17T08:51:41.515944+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:45.006490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488100+0000","last_change":"2024-04-17T08:51:41.513058+0000","last_active":"2024-04-17T08:51:42.488100+0000","last_peered":"2024-04-17T08:51:42.488100+0000","last_clean":"2024-04-17T08:51:42.488100+0000","last_became_active":"2024-04-17T08:51:41.512444+0000","last_became_peered":"2024-04-17T08:51:41.512444+0000","last_unstale":"2024-04-17T08:51:42.488100+0000","last_undegraded":"2024-04-17T08:51:42.488100+0000","last_fullsized":"2024-04-17T08:51:42.488100+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482212+0000","last_change":"2024-04-17T08:51:41.519929+0000","last_active":"2024-04-17T08:51:42.482212+0000","last_peered":"2024-04-17T08:51:42.482212+0000","last_clean":"2024-04-17T08:51:42.482212+0000","last_became_active":"2024-04-17T08:51:41.519591+0000","last_became_peered":"2024-04-17T08:51:41.519591+0000","last_unstale":"2024-04-17T08:51:42.482212+0000","last_undegraded":"2024-04-17T08:51:42.482212+0000","last_fullsized":"2024-04-17T08:51:42.482212+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:03.965519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986948+0000","last_change":"2024-04-17T08:51:41.506863+0000","last_active":"2024-04-17T08:51:42.986948+0000","last_peered":"2024-04-17T08:51:42.986948+0000","last_clean":"2024-04-17T08:51:42.986948+0000","last_became_active":"2024-04-17T08:51:41.506606+0000","last_became_peered":"2024-04-17T08:51:41.506606+0000","last_unstale":"2024-04-17T08:51:42.986948+0000","last_undegraded":"2024-04-17T08:51:42.986948+0000","last_fullsized":"2024-04-17T08:51:42.986948+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:42.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481172+0000","last_change":"2024-04-17T08:51:41.505583+0000","last_active":"2024-04-17T08:51:42.481172+0000","last_peered":"2024-04-17T08:51:42.481172+0000","last_clean":"2024-04-17T08:51:42.481172+0000","last_became_active":"2024-04-17T08:51:41.501478+0000","last_became_peered":"2024-04-17T08:51:41.501478+0000","last_unstale":"2024-04-17T08:51:42.481172+0000","last_undegraded":"2024-04-17T08:51:42.481172+0000","last_fullsized":"2024-04-17T08:51:42.481172+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:01.976342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.490832+0000","last_change":"2024-04-17T08:51:41.500660+0000","last_active":"2024-04-17T08:51:43.490832+0000","last_peered":"2024-04-17T08:51:43.490832+0000","last_clean":"2024-04-17T08:51:43.490832+0000","last_became_active":"2024-04-17T08:51:41.500031+0000","last_became_peered":"2024-04-17T08:51:41.500031+0000","last_unstale":"2024-04-17T08:51:43.490832+0000","last_undegraded":"2024-04-17T08:51:43.490832+0000","last_fullsized":"2024-04-17T08:51:43.490832+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:10.441817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.515315+0000","last_change":"2024-04-17T08:51:41.501146+0000","last_active":"2024-04-17T08:51:41.515315+0000","last_peered":"2024-04-17T08:51:41.515315+0000","last_clean":"2024-04-17T08:51:41.515315+0000","last_became_active":"2024-04-17T08:51:41.500704+0000","last_became_peered":"2024-04-17T08:51:41.500704+0000","last_unstale":"2024-04-17T08:51:41.515315+0000","last_undegraded":"2024-04-17T08:51:41.515315+0000","last_fullsized":"2024-04-17T08:51:41.515315+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:39.854967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986854+0000","last_change":"2024-04-17T08:51:41.517358+0000","last_active":"2024-04-17T08:51:42.986854+0000","last_peered":"2024-04-17T08:51:42.986854+0000","last_clean":"2024-04-17T08:51:42.986854+0000","last_became_active":"2024-04-17T08:51:41.517090+0000","last_became_peered":"2024-04-17T08:51:41.517090+0000","last_unstale":"2024-04-17T08:51:42.986854+0000","last_undegraded":"2024-04-17T08:51:42.986854+0000","last_fullsized":"2024-04-17T08:51:42.986854+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:48.083225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510130+0000","last_change":"2024-04-17T08:51:41.503931+0000","last_active":"2024-04-17T08:51:41.510130+0000","last_peered":"2024-04-17T08:51:41.510130+0000","last_clean":"2024-04-17T08:51:41.510130+0000","last_became_active":"2024-04-17T08:51:41.502502+0000","last_became_peered":"2024-04-17T08:51:41.502502+0000","last_unstale":"2024-04-17T08:51:41.510130+0000","last_undegraded":"2024-04-17T08:51:41.510130+0000","last_fullsized":"2024-04-17T08:51:41.510130+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:43:24.179966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488522+0000","last_change":"2024-04-17T08:51:40.478560+0000","last_active":"2024-04-17T08:51:42.488522+0000","last_peered":"2024-04-17T08:51:42.488522+0000","last_clean":"2024-04-17T08:51:42.488522+0000","last_became_active":"2024-04-17T08:51:40.478144+0000","last_became_peered":"2024-04-17T08:51:40.478144+0000","last_unstale":"2024-04-17T08:51:42.488522+0000","last_undegraded":"2024-04-17T08:51:42.488522+0000","last_fullsized":"2024-04-17T08:51:42.488522+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:37.672975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481410+0000","last_change":"2024-04-17T08:51:41.505188+0000","last_active":"2024-04-17T08:51:42.481410+0000","last_peered":"2024-04-17T08:51:42.481410+0000","last_clean":"2024-04-17T08:51:42.481410+0000","last_became_active":"2024-04-17T08:51:41.504905+0000","last_became_peered":"2024-04-17T08:51:41.504905+0000","last_unstale":"2024-04-17T08:51:42.481410+0000","last_undegraded":"2024-04-17T08:51:42.481410+0000","last_fullsized":"2024-04-17T08:51:42.481410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:11.712753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525859+0000","last_change":"2024-04-17T08:51:40.469433+0000","last_active":"2024-04-17T08:51:41.525859+0000","last_peered":"2024-04-17T08:51:41.525859+0000","last_clean":"2024-04-17T08:51:41.525859+0000","last_became_active":"2024-04-17T08:51:40.469071+0000","last_became_peered":"2024-04-17T08:51:40.469071+0000","last_unstale":"2024-04-17T08:51:41.525859+0000","last_undegraded":"2024-04-17T08:51:41.525859+0000","last_fullsized":"2024-04-17T08:51:41.525859+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:21.208592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484789+0000","last_change":"2024-04-17T08:51:41.516473+0000","last_active":"2024-04-17T08:51:42.484789+0000","last_peered":"2024-04-17T08:51:42.484789+0000","last_clean":"2024-04-17T08:51:42.484789+0000","last_became_active":"2024-04-17T08:51:41.515870+0000","last_became_peered":"2024-04-17T08:51:41.515870+0000","last_unstale":"2024-04-17T08:51:42.484789+0000","last_undegraded":"2024-04-17T08:51:42.484789+0000","last_fullsized":"2024-04-17T08:51:42.484789+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:09.372045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.524466+0000","last_change":"2024-04-17T08:51:40.472815+0000","last_active":"2024-04-17T08:51:41.524466+0000","last_peered":"2024-04-17T08:51:41.524466+0000","last_clean":"2024-04-17T08:51:41.524466+0000","last_became_active":"2024-04-17T08:51:40.470696+0000","last_became_peered":"2024-04-17T08:51:40.470696+0000","last_unstale":"2024-04-17T08:51:41.524466+0000","last_undegraded":"2024-04-17T08:51:41.524466+0000","last_fullsized":"2024-04-17T08:51:41.524466+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:32.004523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480808+0000","last_change":"2024-04-17T08:51:41.497630+0000","last_active":"2024-04-17T08:51:42.480808+0000","last_peered":"2024-04-17T08:51:42.480808+0000","last_clean":"2024-04-17T08:51:42.480808+0000","last_became_active":"2024-04-17T08:51:41.497323+0000","last_became_peered":"2024-04-17T08:51:41.497323+0000","last_unstale":"2024-04-17T08:51:42.480808+0000","last_undegraded":"2024-04-17T08:51:42.480808+0000","last_fullsized":"2024-04-17T08:51:42.480808+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:45.677621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.511179+0000","last_change":"2024-04-17T08:51:40.475306+0000","last_active":"2024-04-17T08:51:41.511179+0000","last_peered":"2024-04-17T08:51:41.511179+0000","last_clean":"2024-04-17T08:51:41.511179+0000","last_became_active":"2024-04-17T08:51:40.474409+0000","last_became_peered":"2024-04-17T08:51:40.474409+0000","last_unstale":"2024-04-17T08:51:41.511179+0000","last_undegraded":"2024-04-17T08:51:41.511179+0000","last_fullsized":"2024-04-17T08:51:41.511179+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:17.301101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.512187+0000","last_change":"2024-04-17T08:51:41.511131+0000","last_active":"2024-04-17T08:51:41.512187+0000","last_peered":"2024-04-17T08:51:41.512187+0000","last_clean":"2024-04-17T08:51:41.512187+0000","last_became_active":"2024-04-17T08:51:41.510481+0000","last_became_peered":"2024-04-17T08:51:41.510481+0000","last_unstale":"2024-04-17T08:51:41.512187+0000","last_undegraded":"2024-04-17T08:51:41.512187+0000","last_fullsized":"2024-04-17T08:51:41.512187+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:49.058157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986237+0000","last_change":"2024-04-17T08:51:40.476463+0000","last_active":"2024-04-17T08:51:42.986237+0000","last_peered":"2024-04-17T08:51:42.986237+0000","last_clean":"2024-04-17T08:51:42.986237+0000","last_became_active":"2024-04-17T08:51:40.476164+0000","last_became_peered":"2024-04-17T08:51:40.476164+0000","last_unstale":"2024-04-17T08:51:42.986237+0000","last_undegraded":"2024-04-17T08:51:42.986237+0000","last_fullsized":"2024-04-17T08:51:42.986237+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:15.997620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.508331+0000","last_change":"2024-04-17T08:51:41.507219+0000","last_active":"2024-04-17T08:51:41.508331+0000","last_peered":"2024-04-17T08:51:41.508331+0000","last_clean":"2024-04-17T08:51:41.508331+0000","last_became_active":"2024-04-17T08:51:41.504481+0000","last_became_peered":"2024-04-17T08:51:41.504481+0000","last_unstale":"2024-04-17T08:51:41.508331+0000","last_undegraded":"2024-04-17T08:51:41.508331+0000","last_fullsized":"2024-04-17T08:51:41.508331+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:41.482889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486732+0000","last_change":"2024-04-17T08:51:40.485442+0000","last_active":"2024-04-17T08:51:42.486732+0000","last_peered":"2024-04-17T08:51:42.486732+0000","last_clean":"2024-04-17T08:51:42.486732+0000","last_became_active":"2024-04-17T08:51:40.485100+0000","last_became_peered":"2024-04-17T08:51:40.485100+0000","last_unstale":"2024-04-17T08:51:42.486732+0000","last_undegraded":"2024-04-17T08:51:42.486732+0000","last_fullsized":"2024-04-17T08:51:42.486732+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:18.376309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986392+0000","last_change":"2024-04-17T08:51:41.504552+0000","last_active":"2024-04-17T08:51:42.986392+0000","last_peered":"2024-04-17T08:51:42.986392+0000","last_clean":"2024-04-17T08:51:42.986392+0000","last_became_active":"2024-04-17T08:51:41.504177+0000","last_became_peered":"2024-04-17T08:51:41.504177+0000","last_unstale":"2024-04-17T08:51:42.986392+0000","last_undegraded":"2024-04-17T08:51:42.986392+0000","last_fullsized":"2024-04-17T08:51:42.986392+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:31.634770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477026+0000","last_change":"2024-04-17T08:51:40.470752+0000","last_active":"2024-04-17T08:51:41.477026+0000","last_peered":"2024-04-17T08:51:41.477026+0000","last_clean":"2024-04-17T08:51:41.477026+0000","last_became_active":"2024-04-17T08:51:40.470440+0000","last_became_peered":"2024-04-17T08:51:40.470440+0000","last_unstale":"2024-04-17T08:51:41.477026+0000","last_undegraded":"2024-04-17T08:51:41.477026+0000","last_fullsized":"2024-04-17T08:51:41.477026+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.094101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987750+0000","last_change":"2024-04-17T08:51:41.505515+0000","last_active":"2024-04-17T08:51:42.987750+0000","last_peered":"2024-04-17T08:51:42.987750+0000","last_clean":"2024-04-17T08:51:42.987750+0000","last_became_active":"2024-04-17T08:51:41.504948+0000","last_became_peered":"2024-04-17T08:51:41.504948+0000","last_unstale":"2024-04-17T08:51:42.987750+0000","last_undegraded":"2024-04-17T08:51:42.987750+0000","last_fullsized":"2024-04-17T08:51:42.987750+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:44.205613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484572+0000","last_change":"2024-04-17T08:51:40.472194+0000","last_active":"2024-04-17T08:51:42.484572+0000","last_peered":"2024-04-17T08:51:42.484572+0000","last_clean":"2024-04-17T08:51:42.484572+0000","last_became_active":"2024-04-17T08:51:40.471867+0000","last_became_peered":"2024-04-17T08:51:40.471867+0000","last_unstale":"2024-04-17T08:51:42.484572+0000","last_undegraded":"2024-04-17T08:51:42.484572+0000","last_fullsized":"2024-04-17T08:51:42.484572+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:57.953636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482990+0000","last_change":"2024-04-17T08:51:41.505182+0000","last_active":"2024-04-17T08:51:42.482990+0000","last_peered":"2024-04-17T08:51:42.482990+0000","last_clean":"2024-04-17T08:51:42.482990+0000","last_became_active":"2024-04-17T08:51:41.503770+0000","last_became_peered":"2024-04-17T08:51:41.503770+0000","last_unstale":"2024-04-17T08:51:42.482990+0000","last_undegraded":"2024-04-17T08:51:42.482990+0000","last_fullsized":"2024-04-17T08:51:42.482990+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.476808+0000","last_change":"2024-04-17T08:51:40.478982+0000","last_active":"2024-04-17T08:51:41.476808+0000","last_peered":"2024-04-17T08:51:41.476808+0000","last_clean":"2024-04-17T08:51:41.476808+0000","last_became_active":"2024-04-17T08:51:40.478218+0000","last_became_peered":"2024-04-17T08:51:40.478218+0000","last_unstale":"2024-04-17T08:51:41.476808+0000","last_undegraded":"2024-04-17T08:51:41.476808+0000","last_fullsized":"2024-04-17T08:51:41.476808+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:48:34.166708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.499486+0000","last_change":"2024-04-17T08:51:41.488821+0000","last_active":"2024-04-17T08:51:41.499486+0000","last_peered":"2024-04-17T08:51:41.499486+0000","last_clean":"2024-04-17T08:51:41.499486+0000","last_became_active":"2024-04-17T08:51:41.488545+0000","last_became_peered":"2024-04-17T08:51:41.488545+0000","last_unstale":"2024-04-17T08:51:41.499486+0000","last_undegraded":"2024-04-17T08:51:41.499486+0000","last_fullsized":"2024-04-17T08:51:41.499486+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:13.455149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":17,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987036+0000","last_change":"2024-04-17T08:51:40.484183+0000","last_active":"2024-04-17T08:51:42.987036+0000","last_peered":"2024-04-17T08:51:42.987036+0000","last_clean":"2024-04-17T08:51:42.987036+0000","last_became_active":"2024-04-17T08:51:40.483891+0000","last_became_peered":"2024-04-17T08:51:40.483891+0000","last_unstale":"2024-04-17T08:51:42.987036+0000","last_undegraded":"2024-04-17T08:51:42.987036+0000","last_fullsized":"2024-04-17T08:51:42.987036+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:41.230843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487873+0000","last_change":"2024-04-17T08:51:41.515320+0000","last_active":"2024-04-17T08:51:42.487873+0000","last_peered":"2024-04-17T08:51:42.487873+0000","last_clean":"2024-04-17T08:51:42.487873+0000","last_became_active":"2024-04-17T08:51:41.513254+0000","last_became_peered":"2024-04-17T08:51:41.513254+0000","last_unstale":"2024-04-17T08:51:42.487873+0000","last_undegraded":"2024-04-17T08:51:42.487873+0000","last_fullsized":"2024-04-17T08:51:42.487873+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"249'6","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986879+0000","last_change":"2024-04-17T08:51:40.479746+0000","last_active":"2024-04-17T08:51:42.986879+0000","last_peered":"2024-04-17T08:51:42.986879+0000","last_clean":"2024-04-17T08:51:42.986879+0000","last_became_active":"2024-04-17T08:51:40.478655+0000","last_became_peered":"2024-04-17T08:51:40.478655+0000","last_unstale":"2024-04-17T08:51:42.986879+0000","last_undegraded":"2024-04-17T08:51:42.986879+0000","last_fullsized":"2024-04-17T08:51:42.986879+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:03.205317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486935+0000","last_change":"2024-04-17T08:51:41.518673+0000","last_active":"2024-04-17T08:51:42.486935+0000","last_peered":"2024-04-17T08:51:42.486935+0000","last_clean":"2024-04-17T08:51:42.486935+0000","last_became_active":"2024-04-17T08:51:41.517797+0000","last_became_peered":"2024-04-17T08:51:41.517797+0000","last_unstale":"2024-04-17T08:51:42.486935+0000","last_undegraded":"2024-04-17T08:51:42.486935+0000","last_fullsized":"2024-04-17T08:51:42.486935+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:09.689564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986885+0000","last_change":"2024-04-17T08:51:40.481925+0000","last_active":"2024-04-17T08:51:42.986885+0000","last_peered":"2024-04-17T08:51:42.986885+0000","last_clean":"2024-04-17T08:51:42.986885+0000","last_became_active":"2024-04-17T08:51:40.481436+0000","last_became_peered":"2024-04-17T08:51:40.481436+0000","last_unstale":"2024-04-17T08:51:42.986885+0000","last_undegraded":"2024-04-17T08:51:42.986885+0000","last_fullsized":"2024-04-17T08:51:42.986885+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:04.739470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510736+0000","last_change":"2024-04-17T08:51:41.508860+0000","last_active":"2024-04-17T08:51:41.510736+0000","last_peered":"2024-04-17T08:51:41.510736+0000","last_clean":"2024-04-17T08:51:41.510736+0000","last_became_active":"2024-04-17T08:51:41.508524+0000","last_became_peered":"2024-04-17T08:51:41.508524+0000","last_unstale":"2024-04-17T08:51:41.510736+0000","last_undegraded":"2024-04-17T08:51:41.510736+0000","last_fullsized":"2024-04-17T08:51:41.510736+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:41.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.476705+0000","last_change":"2024-04-17T08:51:40.479291+0000","last_active":"2024-04-17T08:51:41.476705+0000","last_peered":"2024-04-17T08:51:41.476705+0000","last_clean":"2024-04-17T08:51:41.476705+0000","last_became_active":"2024-04-17T08:51:40.476885+0000","last_became_peered":"2024-04-17T08:51:40.476885+0000","last_unstale":"2024-04-17T08:51:41.476705+0000","last_undegraded":"2024-04-17T08:51:41.476705+0000","last_fullsized":"2024-04-17T08:51:41.476705+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:53.966461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516541+0000","last_change":"2024-04-17T08:51:41.509196+0000","last_active":"2024-04-17T08:51:41.516541+0000","last_peered":"2024-04-17T08:51:41.516541+0000","last_clean":"2024-04-17T08:51:41.516541+0000","last_became_active":"2024-04-17T08:51:41.508917+0000","last_became_peered":"2024-04-17T08:51:41.508917+0000","last_unstale":"2024-04-17T08:51:41.516541+0000","last_undegraded":"2024-04-17T08:51:41.516541+0000","last_fullsized":"2024-04-17T08:51:41.516541+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:11:24.461261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486989+0000","last_change":"2024-04-17T08:51:40.483627+0000","last_active":"2024-04-17T08:51:42.486989+0000","last_peered":"2024-04-17T08:51:42.486989+0000","last_clean":"2024-04-17T08:51:42.486989+0000","last_became_active":"2024-04-17T08:51:40.482318+0000","last_became_peered":"2024-04-17T08:51:40.482318+0000","last_unstale":"2024-04-17T08:51:42.486989+0000","last_undegraded":"2024-04-17T08:51:42.486989+0000","last_fullsized":"2024-04-17T08:51:42.486989+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.761124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486412+0000","last_change":"2024-04-17T08:51:41.517149+0000","last_active":"2024-04-17T08:51:42.486412+0000","last_peered":"2024-04-17T08:51:42.486412+0000","last_clean":"2024-04-17T08:51:42.486412+0000","last_became_active":"2024-04-17T08:51:41.516844+0000","last_became_peered":"2024-04-17T08:51:41.516844+0000","last_unstale":"2024-04-17T08:51:42.486412+0000","last_undegraded":"2024-04-17T08:51:42.486412+0000","last_fullsized":"2024-04-17T08:51:42.486412+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:37.293958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":22,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480358+0000","last_change":"2024-04-17T08:51:40.479653+0000","last_active":"2024-04-17T08:51:42.480358+0000","last_peered":"2024-04-17T08:51:42.480358+0000","last_clean":"2024-04-17T08:51:42.480358+0000","last_became_active":"2024-04-17T08:51:40.479107+0000","last_became_peered":"2024-04-17T08:51:40.479107+0000","last_unstale":"2024-04-17T08:51:42.480358+0000","last_undegraded":"2024-04-17T08:51:42.480358+0000","last_fullsized":"2024-04-17T08:51:42.480358+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:16.542860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.516904+0000","last_change":"2024-04-17T08:51:41.510891+0000","last_active":"2024-04-17T08:51:41.516904+0000","last_peered":"2024-04-17T08:51:41.516904+0000","last_clean":"2024-04-17T08:51:41.516904+0000","last_became_active":"2024-04-17T08:51:41.510538+0000","last_became_peered":"2024-04-17T08:51:41.510538+0000","last_unstale":"2024-04-17T08:51:41.516904+0000","last_undegraded":"2024-04-17T08:51:41.516904+0000","last_fullsized":"2024-04-17T08:51:41.516904+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:38.420980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487930+0000","last_change":"2024-04-17T08:51:40.484060+0000","last_active":"2024-04-17T08:51:42.487930+0000","last_peered":"2024-04-17T08:51:42.487930+0000","last_clean":"2024-04-17T08:51:42.487930+0000","last_became_active":"2024-04-17T08:51:40.483692+0000","last_became_peered":"2024-04-17T08:51:40.483692+0000","last_unstale":"2024-04-17T08:51:42.487930+0000","last_undegraded":"2024-04-17T08:51:42.487930+0000","last_fullsized":"2024-04-17T08:51:42.487930+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:53.270393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.517677+0000","last_change":"2024-04-17T08:51:41.516369+0000","last_active":"2024-04-17T08:51:41.517677+0000","last_peered":"2024-04-17T08:51:41.517677+0000","last_clean":"2024-04-17T08:51:41.517677+0000","last_became_active":"2024-04-17T08:51:41.515919+0000","last_became_peered":"2024-04-17T08:51:41.515919+0000","last_unstale":"2024-04-17T08:51:41.517677+0000","last_undegraded":"2024-04-17T08:51:41.517677+0000","last_fullsized":"2024-04-17T08:51:41.517677+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:38.001673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525899+0000","last_change":"2024-04-17T08:51:40.466869+0000","last_active":"2024-04-17T08:51:41.525899+0000","last_peered":"2024-04-17T08:51:41.525899+0000","last_clean":"2024-04-17T08:51:41.525899+0000","last_became_active":"2024-04-17T08:51:40.465973+0000","last_became_peered":"2024-04-17T08:51:40.465973+0000","last_unstale":"2024-04-17T08:51:41.525899+0000","last_undegraded":"2024-04-17T08:51:41.525899+0000","last_fullsized":"2024-04-17T08:51:41.525899+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:57.022722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":9,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.510339+0000","last_change":"2024-04-17T08:51:41.507082+0000","last_active":"2024-04-17T08:51:41.510339+0000","last_peered":"2024-04-17T08:51:41.510339+0000","last_clean":"2024-04-17T08:51:41.510339+0000","last_became_active":"2024-04-17T08:51:41.506812+0000","last_became_peered":"2024-04-17T08:51:41.506812+0000","last_unstale":"2024-04-17T08:51:41.510339+0000","last_undegraded":"2024-04-17T08:51:41.510339+0000","last_fullsized":"2024-04-17T08:51:41.510339+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:26.370488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":13,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.512385+0000","last_change":"2024-04-17T08:51:40.481626+0000","last_active":"2024-04-17T08:51:41.512385+0000","last_peered":"2024-04-17T08:51:41.512385+0000","last_clean":"2024-04-17T08:51:41.512385+0000","last_became_active":"2024-04-17T08:51:40.481268+0000","last_became_peered":"2024-04-17T08:51:40.481268+0000","last_unstale":"2024-04-17T08:51:41.512385+0000","last_undegraded":"2024-04-17T08:51:41.512385+0000","last_fullsized":"2024-04-17T08:51:41.512385+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:35.353434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.485159+0000","last_change":"2024-04-17T08:51:41.503769+0000","last_active":"2024-04-17T08:51:42.485159+0000","last_peered":"2024-04-17T08:51:42.485159+0000","last_clean":"2024-04-17T08:51:42.485159+0000","last_became_active":"2024-04-17T08:51:41.503051+0000","last_became_peered":"2024-04-17T08:51:41.503051+0000","last_unstale":"2024-04-17T08:51:42.485159+0000","last_undegraded":"2024-04-17T08:51:42.485159+0000","last_fullsized":"2024-04-17T08:51:42.485159+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:40.349134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481672+0000","last_change":"2024-04-17T08:51:40.482915+0000","last_active":"2024-04-17T08:51:42.481672+0000","last_peered":"2024-04-17T08:51:42.481672+0000","last_clean":"2024-04-17T08:51:42.481672+0000","last_became_active":"2024-04-17T08:51:40.482629+0000","last_became_peered":"2024-04-17T08:51:40.482629+0000","last_unstale":"2024-04-17T08:51:42.481672+0000","last_undegraded":"2024-04-17T08:51:42.481672+0000","last_fullsized":"2024-04-17T08:51:42.481672+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.115195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483345+0000","last_change":"2024-04-17T08:51:41.518701+0000","last_active":"2024-04-17T08:51:42.483345+0000","last_peered":"2024-04-17T08:51:42.483345+0000","last_clean":"2024-04-17T08:51:42.483345+0000","last_became_active":"2024-04-17T08:51:41.516124+0000","last_became_peered":"2024-04-17T08:51:41.516124+0000","last_unstale":"2024-04-17T08:51:42.483345+0000","last_undegraded":"2024-04-17T08:51:42.483345+0000","last_fullsized":"2024-04-17T08:51:42.483345+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.524888+0000","last_change":"2024-04-17T08:51:40.483946+0000","last_active":"2024-04-17T08:51:41.524888+0000","last_peered":"2024-04-17T08:51:41.524888+0000","last_clean":"2024-04-17T08:51:41.524888+0000","last_became_active":"2024-04-17T08:51:40.483664+0000","last_became_peered":"2024-04-17T08:51:40.483664+0000","last_unstale":"2024-04-17T08:51:41.524888+0000","last_undegraded":"2024-04-17T08:51:41.524888+0000","last_fullsized":"2024-04-17T08:51:41.524888+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:55.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986451+0000","last_change":"2024-04-17T08:51:41.519618+0000","last_active":"2024-04-17T08:51:42.986451+0000","last_peered":"2024-04-17T08:51:42.986451+0000","last_clean":"2024-04-17T08:51:42.986451+0000","last_became_active":"2024-04-17T08:51:41.519247+0000","last_became_peered":"2024-04-17T08:51:41.519247+0000","last_unstale":"2024-04-17T08:51:42.986451+0000","last_undegraded":"2024-04-17T08:51:42.986451+0000","last_fullsized":"2024-04-17T08:51:42.986451+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:32.995157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480822+0000","last_change":"2024-04-17T08:51:40.486177+0000","last_active":"2024-04-17T08:51:42.480822+0000","last_peered":"2024-04-17T08:51:42.480822+0000","last_clean":"2024-04-17T08:51:42.480822+0000","last_became_active":"2024-04-17T08:51:40.484934+0000","last_became_peered":"2024-04-17T08:51:40.484934+0000","last_unstale":"2024-04-17T08:51:42.480822+0000","last_undegraded":"2024-04-17T08:51:42.480822+0000","last_fullsized":"2024-04-17T08:51:42.480822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:23.956487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.480339+0000","last_change":"2024-04-17T08:51:41.517111+0000","last_active":"2024-04-17T08:51:42.480339+0000","last_peered":"2024-04-17T08:51:42.480339+0000","last_clean":"2024-04-17T08:51:42.480339+0000","last_became_active":"2024-04-17T08:51:41.516756+0000","last_became_peered":"2024-04-17T08:51:41.516756+0000","last_unstale":"2024-04-17T08:51:42.480339+0000","last_undegraded":"2024-04-17T08:51:42.480339+0000","last_fullsized":"2024-04-17T08:51:42.480339+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:17:58.048828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.490994+0000","last_change":"2024-04-17T08:51:40.479934+0000","last_active":"2024-04-17T08:51:41.490994+0000","last_peered":"2024-04-17T08:51:41.490994+0000","last_clean":"2024-04-17T08:51:41.490994+0000","last_became_active":"2024-04-17T08:51:40.479295+0000","last_became_peered":"2024-04-17T08:51:40.479295+0000","last_unstale":"2024-04-17T08:51:41.490994+0000","last_undegraded":"2024-04-17T08:51:41.490994+0000","last_fullsized":"2024-04-17T08:51:41.490994+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:40.469305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.483359+0000","last_change":"2024-04-17T08:51:41.500141+0000","last_active":"2024-04-17T08:51:42.483359+0000","last_peered":"2024-04-17T08:51:42.483359+0000","last_clean":"2024-04-17T08:51:42.483359+0000","last_became_active":"2024-04-17T08:51:41.499513+0000","last_became_peered":"2024-04-17T08:51:41.499513+0000","last_unstale":"2024-04-17T08:51:42.483359+0000","last_undegraded":"2024-04-17T08:51:42.483359+0000","last_fullsized":"2024-04-17T08:51:42.483359+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.486787+0000","last_change":"2024-04-17T08:51:40.478664+0000","last_active":"2024-04-17T08:51:42.486787+0000","last_peered":"2024-04-17T08:51:42.486787+0000","last_clean":"2024-04-17T08:51:42.486787+0000","last_became_active":"2024-04-17T08:51:40.477953+0000","last_became_peered":"2024-04-17T08:51:40.477953+0000","last_unstale":"2024-04-17T08:51:42.486787+0000","last_undegraded":"2024-04-17T08:51:42.486787+0000","last_fullsized":"2024-04-17T08:51:42.486787+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:54.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.488347+0000","last_change":"2024-04-17T08:51:41.515604+0000","last_active":"2024-04-17T08:51:42.488347+0000","last_peered":"2024-04-17T08:51:42.488347+0000","last_clean":"2024-04-17T08:51:42.488347+0000","last_became_active":"2024-04-17T08:51:41.514707+0000","last_became_peered":"2024-04-17T08:51:41.514707+0000","last_unstale":"2024-04-17T08:51:42.488347+0000","last_undegraded":"2024-04-17T08:51:42.488347+0000","last_fullsized":"2024-04-17T08:51:42.488347+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":14,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.518687+0000","last_change":"2024-04-17T08:51:40.481823+0000","last_active":"2024-04-17T08:51:41.518687+0000","last_peered":"2024-04-17T08:51:41.518687+0000","last_clean":"2024-04-17T08:51:41.518687+0000","last_became_active":"2024-04-17T08:51:40.481302+0000","last_became_peered":"2024-04-17T08:51:40.481302+0000","last_unstale":"2024-04-17T08:51:41.518687+0000","last_undegraded":"2024-04-17T08:51:41.518687+0000","last_fullsized":"2024-04-17T08:51:41.518687+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:56.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.987380+0000","last_change":"2024-04-17T08:51:41.498889+0000","last_active":"2024-04-17T08:51:42.987380+0000","last_peered":"2024-04-17T08:51:42.987380+0000","last_clean":"2024-04-17T08:51:42.987380+0000","last_became_active":"2024-04-17T08:51:41.498622+0000","last_became_peered":"2024-04-17T08:51:41.498622+0000","last_unstale":"2024-04-17T08:51:42.987380+0000","last_undegraded":"2024-04-17T08:51:42.987380+0000","last_fullsized":"2024-04-17T08:51:42.987380+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:42.312039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.494313+0000","last_change":"2024-04-17T08:51:40.482579+0000","last_active":"2024-04-17T08:51:43.494313+0000","last_peered":"2024-04-17T08:51:43.494313+0000","last_clean":"2024-04-17T08:51:43.494313+0000","last_became_active":"2024-04-17T08:51:40.482255+0000","last_became_peered":"2024-04-17T08:51:40.482255+0000","last_unstale":"2024-04-17T08:51:43.494313+0000","last_undegraded":"2024-04-17T08:51:43.494313+0000","last_fullsized":"2024-04-17T08:51:43.494313+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:37.553047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481422+0000","last_change":"2024-04-17T08:51:41.514216+0000","last_active":"2024-04-17T08:51:42.481422+0000","last_peered":"2024-04-17T08:51:42.481422+0000","last_clean":"2024-04-17T08:51:42.481422+0000","last_became_active":"2024-04-17T08:51:41.513937+0000","last_became_peered":"2024-04-17T08:51:41.513937+0000","last_unstale":"2024-04-17T08:51:42.481422+0000","last_undegraded":"2024-04-17T08:51:42.481422+0000","last_fullsized":"2024-04-17T08:51:42.481422+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:27.955059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.493238+0000","last_change":"2024-04-17T08:51:40.482279+0000","last_active":"2024-04-17T08:51:41.493238+0000","last_peered":"2024-04-17T08:51:41.493238+0000","last_clean":"2024-04-17T08:51:41.493238+0000","last_became_active":"2024-04-17T08:51:40.482002+0000","last_became_peered":"2024-04-17T08:51:40.482002+0000","last_unstale":"2024-04-17T08:51:41.493238+0000","last_undegraded":"2024-04-17T08:51:41.493238+0000","last_fullsized":"2024-04-17T08:51:41.493238+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:36.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482199+0000","last_change":"2024-04-17T08:51:41.505389+0000","last_active":"2024-04-17T08:51:42.482199+0000","last_peered":"2024-04-17T08:51:42.482199+0000","last_clean":"2024-04-17T08:51:42.482199+0000","last_became_active":"2024-04-17T08:51:41.503336+0000","last_became_peered":"2024-04-17T08:51:41.503336+0000","last_unstale":"2024-04-17T08:51:42.482199+0000","last_undegraded":"2024-04-17T08:51:42.482199+0000","last_fullsized":"2024-04-17T08:51:42.482199+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:16.345152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.481774+0000","last_change":"2024-04-17T08:51:40.480656+0000","last_active":"2024-04-17T08:51:42.481774+0000","last_peered":"2024-04-17T08:51:42.481774+0000","last_clean":"2024-04-17T08:51:42.481774+0000","last_became_active":"2024-04-17T08:51:40.480386+0000","last_became_peered":"2024-04-17T08:51:40.480386+0000","last_unstale":"2024-04-17T08:51:42.481774+0000","last_undegraded":"2024-04-17T08:51:42.481774+0000","last_fullsized":"2024-04-17T08:51:42.481774+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.113174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487258+0000","last_change":"2024-04-17T08:51:41.513350+0000","last_active":"2024-04-17T08:51:42.487258+0000","last_peered":"2024-04-17T08:51:42.487258+0000","last_clean":"2024-04-17T08:51:42.487258+0000","last_became_active":"2024-04-17T08:51:41.512524+0000","last_became_peered":"2024-04-17T08:51:41.512524+0000","last_unstale":"2024-04-17T08:51:42.487258+0000","last_undegraded":"2024-04-17T08:51:42.487258+0000","last_fullsized":"2024-04-17T08:51:42.487258+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:39.456694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.487972+0000","last_change":"2024-04-17T08:51:40.480678+0000","last_active":"2024-04-17T08:51:42.487972+0000","last_peered":"2024-04-17T08:51:42.487972+0000","last_clean":"2024-04-17T08:51:42.487972+0000","last_became_active":"2024-04-17T08:51:40.480360+0000","last_became_peered":"2024-04-17T08:51:40.480360+0000","last_unstale":"2024-04-17T08:51:42.487972+0000","last_undegraded":"2024-04-17T08:51:42.487972+0000","last_fullsized":"2024-04-17T08:51:42.487972+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:50.970831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.484365+0000","last_change":"2024-04-17T08:51:41.501267+0000","last_active":"2024-04-17T08:51:42.484365+0000","last_peered":"2024-04-17T08:51:42.484365+0000","last_clean":"2024-04-17T08:51:42.484365+0000","last_became_active":"2024-04-17T08:51:41.500983+0000","last_became_peered":"2024-04-17T08:51:41.500983+0000","last_unstale":"2024-04-17T08:51:42.484365+0000","last_undegraded":"2024-04-17T08:51:42.484365+0000","last_fullsized":"2024-04-17T08:51:42.484365+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:53.940422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.494453+0000","last_change":"2024-04-17T08:51:40.477982+0000","last_active":"2024-04-17T08:51:41.494453+0000","last_peered":"2024-04-17T08:51:41.494453+0000","last_clean":"2024-04-17T08:51:41.494453+0000","last_became_active":"2024-04-17T08:51:40.477717+0000","last_became_peered":"2024-04-17T08:51:40.477717+0000","last_unstale":"2024-04-17T08:51:41.494453+0000","last_undegraded":"2024-04-17T08:51:41.494453+0000","last_fullsized":"2024-04-17T08:51:41.494453+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:51.232388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.482878+0000","last_change":"2024-04-17T08:51:41.504227+0000","last_active":"2024-04-17T08:51:42.482878+0000","last_peered":"2024-04-17T08:51:42.482878+0000","last_clean":"2024-04-17T08:51:42.482878+0000","last_became_active":"2024-04-17T08:51:41.503919+0000","last_became_peered":"2024-04-17T08:51:41.503919+0000","last_unstale":"2024-04-17T08:51:42.482878+0000","last_undegraded":"2024-04-17T08:51:42.482878+0000","last_fullsized":"2024-04-17T08:51:42.482878+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477745+0000","last_change":"2024-04-17T08:51:40.466461+0000","last_active":"2024-04-17T08:51:41.477745+0000","last_peered":"2024-04-17T08:51:41.477745+0000","last_clean":"2024-04-17T08:51:41.477745+0000","last_became_active":"2024-04-17T08:51:40.466050+0000","last_became_peered":"2024-04-17T08:51:40.466050+0000","last_unstale":"2024-04-17T08:51:41.477745+0000","last_undegraded":"2024-04-17T08:51:41.477745+0000","last_fullsized":"2024-04-17T08:51:41.477745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:17.871122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:42.986557+0000","last_change":"2024-04-17T08:51:41.509452+0000","last_active":"2024-04-17T08:51:42.986557+0000","last_peered":"2024-04-17T08:51:42.986557+0000","last_clean":"2024-04-17T08:51:42.986557+0000","last_became_active":"2024-04-17T08:51:41.509160+0000","last_became_peered":"2024-04-17T08:51:41.509160+0000","last_unstale":"2024-04-17T08:51:42.986557+0000","last_undegraded":"2024-04-17T08:51:42.986557+0000","last_fullsized":"2024-04-17T08:51:42.986557+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:54.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.477208+0000","last_change":"2024-04-17T08:51:40.481750+0000","last_active":"2024-04-17T08:51:41.477208+0000","last_peered":"2024-04-17T08:51:41.477208+0000","last_clean":"2024-04-17T08:51:41.477208+0000","last_became_active":"2024-04-17T08:51:40.481311+0000","last_became_peered":"2024-04-17T08:51:40.481311+0000","last_unstale":"2024-04-17T08:51:41.477208+0000","last_undegraded":"2024-04-17T08:51:41.477208+0000","last_fullsized":"2024-04-17T08:51:41.477208+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:50.295078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-04-17T08:51:43.491070+0000","last_change":"2024-04-17T08:51:41.490585+0000","last_active":"2024-04-17T08:51:43.491070+0000","last_peered":"2024-04-17T08:51:43.491070+0000","last_clean":"2024-04-17T08:51:43.491070+0000","last_became_active":"2024-04-17T08:51:41.489827+0000","last_became_peered":"2024-04-17T08:51:41.489827+0000","last_unstale":"2024-04-17T08:51:43.491070+0000","last_undegraded":"2024-04-17T08:51:43.491070+0000","last_fullsized":"2024-04-17T08:51:43.491070+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:35.513878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":16,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.525671+0000","last_change":"2024-04-17T08:51:40.482218+0000","last_active":"2024-04-17T08:51:41.525671+0000","last_peered":"2024-04-17T08:51:41.525671+0000","last_clean":"2024-04-17T08:51:41.525671+0000","last_became_active":"2024-04-17T08:51:40.481390+0000","last_became_peered":"2024-04-17T08:51:40.481390+0000","last_unstale":"2024-04-17T08:51:41.525671+0000","last_undegraded":"2024-04-17T08:51:41.525671+0000","last_fullsized":"2024-04-17T08:51:41.525671+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:41.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"236'376","reported_seq":1563,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-04-17T08:51:41.498615+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:51:41.498615+0000","last_peered":"2024-04-17T08:51:41.498615+0000","last_clean":"2024-04-17T08:51:41.498615+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:51:41.498615+0000","last_undegraded":"2024-04-17T08:51:41.498615+0000","last_fullsized":"2024-04-17T08:51:41.498615+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":376,"ondisk_log_size":376,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543306,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57032,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314808,"statfs":{"total":96636764160,"available":96578363392,"internally_reserved":0,"allocated":3727360,"data_stored":2126458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543306,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60632,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311208,"statfs":{"total":96636764160,"available":96574676992,"internally_reserved":0,"allocated":3149824,"data_stored":1542952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543305,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30092,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341748,"statfs":{"total":96636764160,"available":96605949952,"internally_reserved":0,"allocated":3141632,"data_stored":1542348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66144,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305696,"statfs":{"total":96636764160,"available":96569032704,"internally_reserved":0,"allocated":3162112,"data_stored":1544052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543307,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52292,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319548,"statfs":{"total":96636764160,"available":96583217152,"internally_reserved":0,"allocated":3133440,"data_stored":1536602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543306,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52292,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319548,"statfs":{"total":96636764160,"available":96583217152,"internally_reserved":0,"allocated":3133440,"data_stored":1536110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":3162112,"data_stored":1544084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543305,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":3739648,"data_stored":2132750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:51:46.587 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:51:46.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:51:46.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.864+0000 7fe74a200640 1 Processor -- start 2024-04-17T08:51:46.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.864+0000 7fe74a200640 1 -- start start 2024-04-17T08:51:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.866+0000 7fe74a200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 0x7fe74415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.866+0000 7fe74a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7441601d0 0x7fe744163010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.866+0000 7fe74a200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe744163550 con 0x7fe744072300 2024-04-17T08:51:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.866+0000 7fe74a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe74415f600 con 0x7fe7441601d0 2024-04-17T08:51:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.866+0000 7fe74a200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe74415f8e0 con 0x7fe744071f20 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7441601d0 0x7fe744163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7441601d0 0x7fe744163010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33272/0 (socket says 172.21.15.67:33272) 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 -- 172.21.15.67:0/1188441051 learned_addr learned my addr 172.21.15.67:0/1188441051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe743400640 1 --2- 172.21.15.67:0/1188441051 >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 0x7fe74415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 -- 172.21.15.67:0/1188441051 >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 msgr2=0x7fe74415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 --2- 172.21.15.67:0/1188441051 >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 0x7fe74415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 -- 172.21.15.67:0/1188441051 >> v1:172.21.15.187:6789/0 conn(0x7fe744072300 legacy=0x7fe7440af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe742a00640 1 -- 172.21.15.67:0/1188441051 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74415fbc0 con 0x7fe7441601d0 2024-04-17T08:51:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.867+0000 7fe743400640 1 --2- 172.21.15.67:0/1188441051 >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 0x7fe74415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.868+0000 7fe742a00640 1 --2- 172.21.15.67:0/1188441051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe744163010 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fe730009870 tx=0x7fe73002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e764fe3d93d260 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe742000640 1 -- 172.21.15.67:0/1188441051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730035020 con 0x7fe7441601d0 2024-04-17T08:51:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe742000640 1 -- 172.21.15.67:0/1188441051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe730031870 con 0x7fe7441601d0 2024-04-17T08:51:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe742000640 1 -- 172.21.15.67:0/1188441051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730031c40 con 0x7fe7441601d0 2024-04-17T08:51:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 -- 172.21.15.67:0/1188441051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 msgr2=0x7fe744163010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 --2- 172.21.15.67:0/1188441051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe744163010 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fe730009870 tx=0x7fe73002b910 comp rx=0 tx=0).stop 2024-04-17T08:51:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 -- 172.21.15.67:0/1188441051 shutdown_connections 2024-04-17T08:51:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 --2- 172.21.15.67:0/1188441051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe744163010 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 --2- 172.21.15.67:0/1188441051 >> v2:172.21.15.187:3300/0 conn(0x7fe744071f20 0x7fe74415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.869+0000 7fe74a200640 1 -- 172.21.15.67:0/1188441051 >> 172.21.15.67:0/1188441051 conn(0x7fe744067c30 msgr2=0x7fe74406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.870+0000 7fe74a200640 1 -- 172.21.15.67:0/1188441051 shutdown_connections 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.870+0000 7fe74a200640 1 -- 172.21.15.67:0/1188441051 wait complete. 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.871+0000 7fe74a200640 1 Processor -- start 2024-04-17T08:51:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.871+0000 7fe74a200640 1 -- start start 2024-04-17T08:51:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe744071f20 0x7fe7440b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 0x7fe7440b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe744145340 con 0x7fe7441601d0 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe744160680 con 0x7fe744072300 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.872+0000 7fe74a200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe7441454c0 con 0x7fe744071f20 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33284/0 (socket says 172.21.15.67:33284) 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 -- 172.21.15.67:0/1443505019 learned_addr learned my addr 172.21.15.67:0/1443505019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe742a00640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 0x7fe7440b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743400640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe744071f20 0x7fe7440b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 -- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe744071f20 msgr2=0x7fe7440b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe744071f20 0x7fe7440b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 -- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 msgr2=0x7fe7440b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 0x7fe7440b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe743e00640 1 -- 172.21.15.67:0/1443505019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7440b1e40 con 0x7fe7441601d0 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.873+0000 7fe742a00640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 0x7fe7440b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:46.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe743e00640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b990 tx=0x7fe73c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c019070 con 0x7fe7441601d0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73c00cba0 con 0x7fe7441601d0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7440b2120 con 0x7fe7441601d0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c014440 con 0x7fe7441601d0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.874+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7440b26e0 con 0x7fe7441601d0 2024-04-17T08:51:46.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.875+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe744068ca0 con 0x7fe7441601d0 2024-04-17T08:51:46.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.876+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe73c01c070 con 0x7fe7441601d0 2024-04-17T08:51:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.878+0000 7fe73b400640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 0x7fe72406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.878+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fe73c08bd80 con 0x7fe7441601d0 2024-04-17T08:51:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.878+0000 7fe743400640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 0x7fe72406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:46.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.879+0000 7fe743400640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 0x7fe72406f020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe73400b840 tx=0x7fe734008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:46.882+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe73c059d90 con 0x7fe7441601d0 2024-04-17T08:51:47.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.173+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe74406f6a0 con 0x7fe7441601d0 2024-04-17T08:51:47.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.177+0000 7fe73b400640 1 -- 172.21.15.67:0/1443505019 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) ==== 190+0+147 (secure 0 0 0) 0x7fe73c059690 con 0x7fe7441601d0 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:51:47.177 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:51:47.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 msgr2=0x7fe72406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 0x7fe72406f020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe73400b840 tx=0x7fe734008040 comp rx=0 tx=0).stop 2024-04-17T08:51:47.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 msgr2=0x7fe74415d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b990 tx=0x7fe73c00be50 comp rx=0 tx=0).stop 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 shutdown_connections 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe72406cb60 0x7fe72406f020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7441601d0 0x7fe74415d4a0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe744072300 0x7fe7440b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 --2- 172.21.15.67:0/1443505019 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe744071f20 0x7fe7440b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 >> 172.21.15.67:0/1443505019 conn(0x7fe744067c30 msgr2=0x7fe744161530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.180+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 shutdown_connections 2024-04-17T08:51:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.181+0000 7fe74a200640 1 -- 172.21.15.67:0/1443505019 wait complete. 2024-04-17T08:51:47.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.405+0000 7f6ec0a00640 1 Processor -- start 2024-04-17T08:51:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.405+0000 7f6ec0a00640 1 -- start start 2024-04-17T08:51:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6ec0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ebc160280 0x7f6ebc06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6ec0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc0cd4a0 con 0x7f6ebc141ed0 2024-04-17T08:51:47.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6ec0a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc1607f0 con 0x7f6ebc06b8c0 2024-04-17T08:51:47.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6ec0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6ebc15ce30 con 0x7f6ebc160280 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6eb9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ebc160280 0x7f6ebc06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.407+0000 7f6ebb200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ebc141ed0 0x7f6ebc15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59498/0 (socket says 172.21.15.67:59498) 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6ebb200640 1 -- 172.21.15.67:0/2310093430 learned_addr learned my addr 172.21.15.67:0/2310093430 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6eb9400640 1 -- 172.21.15.67:0/2310093430 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3637893803 0 0) 0x7f6ebc0cd4a0 con 0x7f6ebc141ed0 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6eb9400640 1 -- 172.21.15.67:0/2310093430 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea0003520 con 0x7f6ebc141ed0 2024-04-17T08:51:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6eb9e00640 1 -- 172.21.15.67:0/2310093430 >> v1:172.21.15.187:6789/0 conn(0x7f6ebc06b8c0 legacy=0x7f6ebc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6eb9e00640 1 -- 172.21.15.67:0/2310093430 >> v1:172.21.15.67:6789/0 conn(0x7f6ebc141ed0 legacy=0x7f6ebc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.408+0000 7f6eb9e00640 1 -- 172.21.15.67:0/2310093430 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc160ad0 con 0x7f6ebc160280 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.410+0000 7f6eb9e00640 1 --2- 172.21.15.67:0/2310093430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06bca0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009d90 tx=0x7f6ea802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4936631f7e73174d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.410+0000 7f6eb9400640 1 -- 172.21.15.67:0/2310093430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8039070 con 0x7f6ebc160280 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.410+0000 7f6eb9400640 1 -- 172.21.15.67:0/2310093430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea802cd00 con 0x7f6ebc160280 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.410+0000 7f6eb9400640 1 -- 172.21.15.67:0/2310093430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8034660 con 0x7f6ebc160280 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2310093430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 msgr2=0x7f6ebc06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2310093430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06bca0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009d90 tx=0x7f6ea802bf50 comp rx=0 tx=0).stop 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2310093430 shutdown_connections 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2310093430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06bca0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2310093430 >> 172.21.15.67:0/2310093430 conn(0x7f6ebc067d10 msgr2=0x7f6ebc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2310093430 shutdown_connections 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.411+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2310093430 wait complete. 2024-04-17T08:51:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.413+0000 7f6ec0a00640 1 Processor -- start 2024-04-17T08:51:47.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.413+0000 7f6ec0a00640 1 -- start start 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ebc06b8c0 0x7f6ebc155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ebc141ed0 0x7f6ebc133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc161090 con 0x7f6ebc160280 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc06c1e0 con 0x7f6ebc06b8c0 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ec0a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6ebc13f9b0 con 0x7f6ebc141ed0 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ebb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ebb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33306/0 (socket says 172.21.15.67:33306) 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6ebb200640 1 -- 172.21.15.67:0/2524004632 learned_addr learned my addr 172.21.15.67:0/2524004632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6eba800640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ebc06b8c0 0x7f6ebc155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.414+0000 7f6eb9e00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ebc141ed0 0x7f6ebc133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 -- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ebc141ed0 msgr2=0x7f6ebc133230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ebc141ed0 0x7f6ebc133230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 -- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ebc06b8c0 msgr2=0x7f6ebc155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ebc06b8c0 0x7f6ebc155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 -- 172.21.15.67:0/2524004632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc156450 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ebb200640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c460 tx=0x7f6eb400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4017070 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb4005ce0 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc133770 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40072b0 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.415+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ebc06e5c0 con 0x7f6ebc160280 2024-04-17T08:51:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.416+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ebc068a10 con 0x7f6ebc160280 2024-04-17T08:51:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.418+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6eb401a070 con 0x7f6ebc160280 2024-04-17T08:51:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.418+0000 7f6eb2a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 0x7f6e9c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.419+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f6eb408b400 con 0x7f6ebc160280 2024-04-17T08:51:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.419+0000 7f6eba800640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 0x7f6e9c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.420+0000 7f6eba800640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 0x7f6e9c06eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6eac001640 tx=0x7f6eac009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:47.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.423+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6ebc156450 con 0x7f6ebc160280 2024-04-17T08:51:47.428 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T08:51:47.429+0000 7f50fb000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:51:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.671+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ebc06e000 con 0x7f6ebc160280 2024-04-17T08:51:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.672+0000 7f6eb2a00640 1 -- 172.21.15.67:0/2524004632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5221 (secure 0 0 0) 0x7f6eb4059410 con 0x7f6ebc160280 2024-04-17T08:51:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:47.672 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":11841,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15}]} 2024-04-17T08:51:47.672 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 126 2024-04-17T08:51:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.675+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 msgr2=0x7f6e9c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.675+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 0x7f6e9c06eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6eac001640 tx=0x7f6eac009040 comp rx=0 tx=0).stop 2024-04-17T08:51:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.675+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 msgr2=0x7f6ebc06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.675+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c460 tx=0x7f6eb400c920 comp rx=0 tx=0).stop 2024-04-17T08:51:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 shutdown_connections 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6e9c06cb30 0x7f6e9c06eff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc160280 0x7f6ebc06f0a0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ebc141ed0 0x7f6ebc133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/2524004632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ebc06b8c0 0x7f6ebc155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 >> 172.21.15.67:0/2524004632 conn(0x7f6ebc067d10 msgr2=0x7f6ebc142360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 shutdown_connections 2024-04-17T08:51:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:47.676+0000 7f6ec0a00640 1 -- 172.21.15.67:0/2524004632 wait complete. 2024-04-17T08:51:48.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:51:40.927618+0000', 'modified': '2024-04-17T08:51:44.165413+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11844}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11844': {'gid': 11844, 'name': 'a', 'rank': 0, 'incarnation': 122, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/3084367922', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 3084367922}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3084367922}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11844, 'qdb_cluster': [11844]} 2024-04-17T08:51:48.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:51:48.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T08:51:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.335+0000 7f14f5200640 1 Processor -- start 2024-04-17T08:51:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.336+0000 7f14f5200640 1 -- start start 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.337+0000 7f14f5200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f14f00cbc00 0x7f14f00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.337+0000 7f14f5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f00b63c0 0x7f14f0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14f5200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f015d060 con 0x7f14f00723a0 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14f5200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f14f00cfe90 con 0x7f14f00b63c0 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14f5200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f14f015dc70 con 0x7f14f00cbc00 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f00b63c0 0x7f14f0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14f00b63c0 0x7f14f0071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33312/0 (socket says 172.21.15.67:33312) 2024-04-17T08:51:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14e7e00640 1 -- 172.21.15.67:0/317838879 learned_addr learned my addr 172.21.15.67:0/317838879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.338+0000 7f14ef400640 1 --2- 172.21.15.67:0/317838879 >> v2:172.21.15.187:3300/0 conn(0x7f14f00cbc00 0x7f14f00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.339+0000 7f14e7e00640 1 -- 172.21.15.67:0/317838879 >> v2:172.21.15.187:3300/0 conn(0x7f14f00cbc00 msgr2=0x7f14f00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.339+0000 7f14e7e00640 1 --2- 172.21.15.67:0/317838879 >> v2:172.21.15.187:3300/0 conn(0x7f14f00cbc00 0x7f14f00cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.339+0000 7f14e7e00640 1 -- 172.21.15.67:0/317838879 >> v1:172.21.15.187:6789/0 conn(0x7f14f00723a0 legacy=0x7f14f0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.339+0000 7f14e7e00640 1 -- 172.21.15.67:0/317838879 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f015d340 con 0x7f14f00b63c0 2024-04-17T08:51:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.340+0000 7f14e7e00640 1 --2- 172.21.15.67:0/317838879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00b63c0 0x7f14f0071e60 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f14f00702d0 tx=0x7f14d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316c53d6b837b57f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.340+0000 7f14eea00640 1 -- 172.21.15.67:0/317838879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8035020 con 0x7f14f00b63c0 2024-04-17T08:51:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.340+0000 7f14eea00640 1 -- 172.21.15.67:0/317838879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d8031870 con 0x7f14f00b63c0 2024-04-17T08:51:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.340+0000 7f14eea00640 1 -- 172.21.15.67:0/317838879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8031c40 con 0x7f14f00b63c0 2024-04-17T08:51:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 -- 172.21.15.67:0/317838879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00b63c0 msgr2=0x7f14f0071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 --2- 172.21.15.67:0/317838879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00b63c0 0x7f14f0071e60 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f14f00702d0 tx=0x7f14d802b910 comp rx=0 tx=0).stop 2024-04-17T08:51:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 -- 172.21.15.67:0/317838879 shutdown_connections 2024-04-17T08:51:48.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 --2- 172.21.15.67:0/317838879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00b63c0 0x7f14f0071e60 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 --2- 172.21.15.67:0/317838879 >> v2:172.21.15.187:3300/0 conn(0x7f14f00cbc00 0x7f14f00cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 -- 172.21.15.67:0/317838879 >> 172.21.15.67:0/317838879 conn(0x7f14f006bef0 msgr2=0x7f14f00b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 -- 172.21.15.67:0/317838879 shutdown_connections 2024-04-17T08:51:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.341+0000 7f14f5200640 1 -- 172.21.15.67:0/317838879 wait complete. 2024-04-17T08:51:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.343+0000 7f14f5200640 1 Processor -- start 2024-04-17T08:51:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.343+0000 7f14f5200640 1 -- start start 2024-04-17T08:51:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 0x7f14f0155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14f00cbc00 0x7f14f01537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14f00d02d0 con 0x7f14f00723a0 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14f015d900 con 0x7f14f00b63c0 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14f5200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14f00cbfe0 con 0x7f14f00cbc00 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33314/0 (socket says 172.21.15.67:33314) 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.344+0000 7f14ef400640 1 -- 172.21.15.67:0/4283002964 learned_addr learned my addr 172.21.15.67:0/4283002964 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14efe00640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14f00cbc00 0x7f14f01537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14e7e00640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 0x7f14f0155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 -- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14f00cbc00 msgr2=0x7f14f01537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14f00cbc00 0x7f14f01537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 -- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 msgr2=0x7f14f0155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 0x7f14f0155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 -- 172.21.15.67:0/4283002964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f00b0430 con 0x7f14f00723a0 2024-04-17T08:51:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14e7e00640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 0x7f14f0155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.345+0000 7f14ef400640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d5f0 tx=0x7f14e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.346+0000 7f14e7400640 1 -- 172.21.15.67:0/4283002964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00125f0 con 0x7f14f00723a0 2024-04-17T08:51:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.346+0000 7f14e7400640 1 -- 172.21.15.67:0/4283002964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e0019070 con 0x7f14f00723a0 2024-04-17T08:51:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.346+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f00b0710 con 0x7f14f00723a0 2024-04-17T08:51:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.346+0000 7f14e7400640 1 -- 172.21.15.67:0/4283002964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e00129d0 con 0x7f14f00723a0 2024-04-17T08:51:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.346+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f00b0cd0 con 0x7f14f00723a0 2024-04-17T08:51:48.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.348+0000 7f14e7400640 1 -- 172.21.15.67:0/4283002964 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f14e0012b70 con 0x7f14f00723a0 2024-04-17T08:51:48.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.349+0000 7f14e7400640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 0x7f14cc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.349+0000 7f14e7400640 1 -- 172.21.15.67:0/4283002964 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f14e008c3f0 con 0x7f14f00723a0 2024-04-17T08:51:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.350+0000 7f14e7e00640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 0x7f14cc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.350+0000 7f14f9441740 1 Processor -- start 2024-04-17T08:51:48.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.350+0000 7f14f9441740 1 -- start start 2024-04-17T08:51:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14e7e00640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 0x7f14cc06eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f14d8036000 tx=0x7f14d8008030 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff1d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff1bab0 0x557bbff1d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557bbfe7f980 con 0x557bbff17f50 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557bbfe26220 con 0x557bbff65a30 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.351+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557bbfe7b940 con 0x557bbff1bab0 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33322/0 (socket says 172.21.15.67:33322) 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 -- 172.21.15.67:0/698378850 learned_addr learned my addr 172.21.15.67:0/698378850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14ef400640 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff1d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14e7e00640 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff1bab0 0x557bbff1d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 -- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff1bab0 msgr2=0x557bbff1d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff1bab0 0x557bbff1d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 -- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 msgr2=0x557bbff1d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff1d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.352+0000 7f14efe00640 1 -- 172.21.15.67:0/698378850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbfb922d0 con 0x557bbff17f50 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.353+0000 7f14efe00640 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f14e8005400 tx=0x7f14e80058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19323a017c33e46e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.353+0000 7f14dfe00640 1 -- 172.21.15.67:0/698378850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8006d60 con 0x557bbff17f50 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.353+0000 7f14dfe00640 1 -- 172.21.15.67:0/698378850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e8018070 con 0x557bbff17f50 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.353+0000 7f14dfe00640 1 -- 172.21.15.67:0/698378850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e800eb00 con 0x557bbff17f50 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 -- 172.21.15.67:0/698378850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 msgr2=0x557bbff47d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f14e8005400 tx=0x7f14e80058c0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 -- 172.21.15.67:0/698378850 shutdown_connections 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff17f50 0x557bbff47d40 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff1bab0 0x557bbff1d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 --2- 172.21.15.67:0/698378850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff1d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 -- 172.21.15.67:0/698378850 >> 172.21.15.67:0/698378850 conn(0x557bbff1a1c0 msgr2=0x557bbff62d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 -- 172.21.15.67:0/698378850 shutdown_connections 2024-04-17T08:51:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.354+0000 7f14f9441740 1 -- 172.21.15.67:0/698378850 wait complete. 2024-04-17T08:51:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.355+0000 7f14f9441740 1 -- start start 2024-04-17T08:51:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.355+0000 7f14f9441740 1 Processor -- start 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff65a30 0x557bbff67450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff6a730 0x557bbff3e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557bbff1bfc0 con 0x557bbff1bab0 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557bbfe39590 con 0x557bbff6a730 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557bbff1c2a0 con 0x557bbff65a30 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14f9441740 10 client.0 _ll_register_callbacks cb 0x557bbfb91ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33328/0 (socket says 172.21.15.67:33328) 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.356+0000 7f14ef400640 1 -- 172.21.15.67:0/2460109160 learned_addr learned my addr 172.21.15.67:0/2460109160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14efe00640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff6a730 0x557bbff3e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14e7e00640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff65a30 0x557bbff67450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 -- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff65a30 msgr2=0x557bbff67450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff65a30 0x557bbff67450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 -- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff6a730 msgr2=0x557bbff3e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff6a730 0x557bbff3e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 -- 172.21.15.67:0/2460109160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbfc5c6b0 con 0x557bbff1bab0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14ef400640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f14e0006fa0 tx=0x7f14e0059e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbf075d79ffb8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0088060 con 0x557bbff1bab0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f14e0056410 con 0x557bbff1bab0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.357+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14e8018070 con 0x557bbff1bab0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f14e0077050 con 0x557bbff1bab0 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 10 client.11988 fetch_fsmap learned FSMap version 126 2024-04-17T08:51:48.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) -- 0x557bbff74190 con 0x557bbff1bab0 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e8018070 con 0x557bbff1bab0 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 126) ==== 3495+0+0 (secure 0 0 0) 0x7f14e0013070 con 0x557bbff1bab0 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 10 client.11988 fetch_fsmap finished waiting for FSMap version 126 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 10 client.11988 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 20 client.11988 populate_metadata read hostname 'smithi067' 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff45270 0x557bbff82100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 4 client.11988 mds_command: new command op to 11844 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 --> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bbfe8d3a0 con 0x557bbff45270 2024-04-17T08:51:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.358+0000 7f14e7e00640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff45270 0x557bbff82100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.359+0000 7f14e7e00640 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff45270 0x557bbff82100 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18c8623fb4699d7 server_cookie=2f30a5a408d3811c in_seq=0 out_seq=0 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.359+0000 7f14dcc00640 10 client.11988 ms_handle_connect on v2:172.21.15.67:6834/3084367922 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14dcc00640 1 -- 172.21.15.67:0/2460109160 <== mds.0 v2:172.21.15.67:6834/3084367922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bbfe8d3a0 con 0x557bbff45270 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14dcc00640 10 client.11988 handle_command_reply: tid=0 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 client.11988 shutdown 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 msgr2=0x557bbff3dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f14e0006fa0 tx=0x7f14e0059e60 comp rx=0 tx=0).stop 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 shutdown_connections 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff45270 0x557bbff82100 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff6a730 0x557bbff3e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff65a30 0x557bbff67450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 --2- 172.21.15.67:0/2460109160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff3dd00 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14dcc00640 0 client.11988 ms_handle_reset on v2:172.21.15.67:6834/3084367922 2024-04-17T08:51:48.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.360+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 >> 172.21.15.67:0/2460109160 conn(0x557bbff17f50 msgr2=0x557bbff1db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.361+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 shutdown_connections 2024-04-17T08:51:48.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.361+0000 7f14f9441740 1 -- 172.21.15.67:0/2460109160 wait complete. 2024-04-17T08:51:48.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.361+0000 7f14f9441740 20 client.11988 trim_cache size 0 max 16384 2024-04-17T08:51:48.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.381+0000 7f14f9441740 1 Processor -- start 2024-04-17T08:51:48.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.381+0000 7f14f9441740 1 -- start start 2024-04-17T08:51:48.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff64a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff3b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14f006f8a0 con 0x557bbff1bab0 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01570e0 con 0x557bbff65a30 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14f006f250 con 0x557bbff6a730 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33338/0 (socket says 172.21.15.67:33338) 2024-04-17T08:51:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.382+0000 7f14ef400640 1 -- 172.21.15.67:0/1929037403 learned_addr learned my addr 172.21.15.67:0/1929037403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14e7e00640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff64a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14efe00640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff3b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 -- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 msgr2=0x557bbff3b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff3b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 -- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 msgr2=0x557bbff64a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff64a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 -- 172.21.15.67:0/1929037403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbff3be40 con 0x557bbff1bab0 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14e7e00640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff64a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.383+0000 7f14ef400640 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d5c0 tx=0x7f14e005c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf67e8557ff937c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14df400640 1 -- 172.21.15.67:0/1929037403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0072070 con 0x557bbff1bab0 2024-04-17T08:51:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14df400640 1 -- 172.21.15.67:0/1929037403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e00792e0 con 0x557bbff1bab0 2024-04-17T08:51:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14df400640 1 -- 172.21.15.67:0/1929037403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e005d030 con 0x557bbff1bab0 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 -- 172.21.15.67:0/1929037403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 msgr2=0x557bbff4beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d5c0 tx=0x7f14e005c000 comp rx=0 tx=0).stop 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 -- 172.21.15.67:0/1929037403 shutdown_connections 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff3b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff64a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 --2- 172.21.15.67:0/1929037403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbff4beb0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.384+0000 7f14f9441740 1 -- 172.21.15.67:0/1929037403 >> 172.21.15.67:0/1929037403 conn(0x557bbff17f50 msgr2=0x557bbff3f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.385+0000 7f14f9441740 1 -- 172.21.15.67:0/1929037403 shutdown_connections 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.385+0000 7f14f9441740 1 -- 172.21.15.67:0/1929037403 wait complete. 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.385+0000 7f14f9441740 1 -- start start 2024-04-17T08:51:48.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.385+0000 7f14f9441740 1 Processor -- start 2024-04-17T08:51:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff660b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff665f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557bbff450f0 con 0x557bbff1bab0 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557bbff44ca0 con 0x557bbff65a30 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557bbff67e70 con 0x557bbff6a730 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.386+0000 7f14f9441740 10 client.0 _ll_register_callbacks cb 0x7f14e0056040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33342/0 (socket says 172.21.15.67:33342) 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 -- 172.21.15.67:0/1903643810 learned_addr learned my addr 172.21.15.67:0/1903643810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 -- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 msgr2=0x557bbff665f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff665f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 -- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 msgr2=0x557bbff660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14e7e00640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 -- 172.21.15.67:0/1903643810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbff44170 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14ef400640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f14e000dfb0 tx=0x7f14e0006e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7718a8708b54046d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.387+0000 7f14dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e005d070 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f14e008c0b0 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14e00792e0 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) ==== 24+0+0 (secure 0 0 0) 0x7f14e0072040 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 10 client.11994 fetch_fsmap learned FSMap version 126 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) -- 0x557bbff64b60 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e00792e0 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 126) ==== 3495+0+0 (secure 0 0 0) 0x7f14e00624f0 con 0x557bbff1bab0 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 10 client.11994 fetch_fsmap finished waiting for FSMap version 126 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 10 client.11994 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 20 client.11994 populate_metadata read hostname 'smithi067' 2024-04-17T08:51:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff61710 0x557bbff69060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 4 client.11994 mds_command: new command op to 11844 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:51:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.388+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 --> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] -- command(tid 0: {"prefix": "status"}) -- 0x557bbff138e0 con 0x557bbff61710 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14e7e00640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff61710 0x557bbff69060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14e7e00640 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff61710 0x557bbff69060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe65b39b9489355 server_cookie=1c22785e458f8554 in_seq=0 out_seq=0 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14dcc00640 10 client.11994 ms_handle_connect on v2:172.21.15.67:6834/3084367922 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14dcc00640 1 -- 172.21.15.67:0/1903643810 <== mds.0 v2:172.21.15.67:6834/3084367922 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x557bbff138e0 con 0x557bbff61710 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14dcc00640 10 client.11994 handle_command_reply: tid=0 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.389+0000 7f14f9441740 1 client.11994 shutdown 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 msgr2=0x557bbffe2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f14e000dfb0 tx=0x7f14e0006e90 comp rx=0 tx=0).stop 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 shutdown_connections 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:6834/3084367922,v1:172.21.15.67:6835/3084367922] conn(0x557bbff61710 0x557bbff69060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557bbff6a730 0x557bbff665f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557bbff65a30 0x557bbff660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 --2- 172.21.15.67:0/1903643810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557bbff1bab0 0x557bbffe2100 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 >> 172.21.15.67:0/1903643810 conn(0x557bbff17f50 msgr2=0x557bbff18430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14dcc00640 0 client.11994 ms_handle_reset on v2:172.21.15.67:6834/3084367922 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 shutdown_connections 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 1 -- 172.21.15.67:0/1903643810 wait complete. 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.390+0000 7f14f9441740 20 client.11994 trim_cache size 0 max 16384 2024-04-17T08:51:48.392 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 11844, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.9170327020000002, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 126, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 250, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 250, 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 12.721024458 2024-04-17T08:51:48.393 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 msgr2=0x7f14cc06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 0x7f14cc06eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f14d8036000 tx=0x7f14d8008030 comp rx=0 tx=0).stop 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 msgr2=0x7f14f0157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d5f0 tx=0x7f14e000dab0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 shutdown_connections 2024-04-17T08:51:48.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f14cc06cb30 0x7f14cc06eff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14f00cbc00 0x7f14f01537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14f00b63c0 0x7f14f0155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 --2- 172.21.15.67:0/4283002964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f00723a0 0x7f14f0157f40 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 >> 172.21.15.67:0/4283002964 conn(0x7f14f006bef0 msgr2=0x7f14f00cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 shutdown_connections 2024-04-17T08:51:48.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.391+0000 7f14f5200640 1 -- 172.21.15.67:0/4283002964 wait complete. 2024-04-17T08:51:48.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:51:48.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.632+0000 7fc3fa000640 1 Processor -- start 2024-04-17T08:51:48.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.632+0000 7fc3fa000640 1 -- start start 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc3f406b8c0 0x7fc3f40cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3f4160280 0x7fc3f406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f4141ed0 0x7fc3f4142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f415ce30 con 0x7fc3f4160280 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f40cd4a0 con 0x7fc3f406b8c0 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.634+0000 7fc3fa000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f41607f0 con 0x7fc3f4141ed0 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3f4160280 0x7fc3f406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3f4160280 0x7fc3f406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33354/0 (socket says 172.21.15.67:33354) 2024-04-17T08:51:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 -- 172.21.15.67:0/2239882274 learned_addr learned my addr 172.21.15.67:0/2239882274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 -- 172.21.15.67:0/2239882274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f4141ed0 msgr2=0x7fc3f4142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 --2- 172.21.15.67:0/2239882274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f4141ed0 0x7fc3f4142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 -- 172.21.15.67:0/2239882274 >> v2:172.21.15.187:3300/0 conn(0x7fc3f406b8c0 msgr2=0x7fc3f40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 --2- 172.21.15.67:0/2239882274 >> v2:172.21.15.187:3300/0 conn(0x7fc3f406b8c0 0x7fc3f40cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.635+0000 7fc3ebe00640 1 -- 172.21.15.67:0/2239882274 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f4160ad0 con 0x7fc3f4160280 2024-04-17T08:51:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.636+0000 7fc3ebe00640 1 --2- 172.21.15.67:0/2239882274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f406bca0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40b3210 tx=0x7fc3dc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b8ea83d577e77 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.636+0000 7fc3f3e00640 1 -- 172.21.15.67:0/2239882274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc035020 con 0x7fc3f4160280 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3f3e00640 1 -- 172.21.15.67:0/2239882274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3dc031880 con 0x7fc3f4160280 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3f3e00640 1 -- 172.21.15.67:0/2239882274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc031c50 con 0x7fc3f4160280 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 -- 172.21.15.67:0/2239882274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 msgr2=0x7fc3f406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 --2- 172.21.15.67:0/2239882274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f406bca0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40b3210 tx=0x7fc3dc02b920 comp rx=0 tx=0).stop 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 -- 172.21.15.67:0/2239882274 shutdown_connections 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 --2- 172.21.15.67:0/2239882274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f4141ed0 0x7fc3f4142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 --2- 172.21.15.67:0/2239882274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f406bca0 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 --2- 172.21.15.67:0/2239882274 >> v2:172.21.15.187:3300/0 conn(0x7fc3f406b8c0 0x7fc3f40cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 -- 172.21.15.67:0/2239882274 >> 172.21.15.67:0/2239882274 conn(0x7fc3f4067d10 msgr2=0x7fc3f40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 -- 172.21.15.67:0/2239882274 shutdown_connections 2024-04-17T08:51:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.637+0000 7fc3fa000640 1 -- 172.21.15.67:0/2239882274 wait complete. 2024-04-17T08:51:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.639+0000 7fc3fa000640 1 Processor -- start 2024-04-17T08:51:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.639+0000 7fc3fa000640 1 -- start start 2024-04-17T08:51:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 0x7fc3f415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 0x7fc3f4143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f406c1e0 con 0x7fc3f4160280 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f40cddf0 con 0x7fc3f4141ed0 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3fa000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f4161090 con 0x7fc3f406b8c0 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3f9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3f9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33368/0 (socket says 172.21.15.67:33368) 2024-04-17T08:51:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.640+0000 7fc3f9600640 1 -- 172.21.15.67:0/1861320599 learned_addr learned my addr 172.21.15.67:0/1861320599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3ebe00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 0x7fc3f4143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f8c00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 0x7fc3f415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 -- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 msgr2=0x7fc3f415b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 0x7fc3f415b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 -- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 msgr2=0x7fc3f4143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 0x7fc3f4143e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 -- 172.21.15.67:0/1861320599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f406eef0 con 0x7fc3f4160280 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f8c00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 0x7fc3f415b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3ebe00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 0x7fc3f4143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f9600640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b9e0 tx=0x7fc3ec00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec019070 con 0x7fc3f4160280 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ec00cb20 con 0x7fc3f4160280 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.641+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec014440 con 0x7fc3f4160280 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.642+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f406f1d0 con 0x7fc3f4160280 2024-04-17T08:51:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.642+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f406fa70 con 0x7fc3f4160280 2024-04-17T08:51:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.643+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f406f3a0 con 0x7fc3f4160280 2024-04-17T08:51:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.644+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fc3ec01c070 con 0x7fc3f4160280 2024-04-17T08:51:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.645+0000 7fc3f1600640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 0x7fc3d006f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.645+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fc3ec08bd40 con 0x7fc3f4160280 2024-04-17T08:51:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.645+0000 7fc3f8c00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 0x7fc3d006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:48.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.647+0000 7fc3f8c00640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 0x7fc3d006f040 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4006fd0 tx=0x7fc3e4008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.649+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc3ec059dd0 con 0x7fc3f4160280 2024-04-17T08:51:48.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.894+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc3f4068a10 con 0x7fc3f4160280 2024-04-17T08:51:48.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.895+0000 7fc3f1600640 1 -- 172.21.15.67:0/1861320599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v126) ==== 53+0+73 (secure 0 0 0) 0x7fc3ec0596d0 con 0x7fc3f4160280 2024-04-17T08:51:48.895 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:51:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 msgr2=0x7fc3d006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 0x7fc3d006f040 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4006fd0 tx=0x7fc3e4008040 comp rx=0 tx=0).stop 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 msgr2=0x7fc3f4155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b9e0 tx=0x7fc3ec00bea0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 shutdown_connections 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fc3d006cb80 0x7fc3d006f040 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f4160280 0x7fc3f4155bc0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3f4141ed0 0x7fc3f4143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 --2- 172.21.15.67:0/1861320599 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3f406b8c0 0x7fc3f415b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:48.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.897+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 >> 172.21.15.67:0/1861320599 conn(0x7fc3f4067d10 msgr2=0x7fc3f40d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:48.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.898+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 shutdown_connections 2024-04-17T08:51:48.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:48.898+0000 7fc3fa000640 1 -- 172.21.15.67:0/1861320599 wait complete. 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:51:48.918 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:51:48.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:48.919 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 36735sec preferred_lft 36735sec 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:51:48.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:48.977 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:51:49.075 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:51:49.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:49.081 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:51:49.140 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:51:49.140 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:51:49.140 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:51:49.141 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:51:49.141 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:51:49.141 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:51:49.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:51:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:51:49.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:49.409 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:51:49.465 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:51:49.465 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:51:49.466 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:51:49.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:49.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:51:49.572 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:51:49.572 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:51:49.630 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:51:49.630 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:51:49.630 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:51:49.683 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:51:49.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:49.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:51:49.830 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:51:49.830 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:51:49.866 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:51:49.866 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:51:49.866 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:51:49.866 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:51:49.866 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:51:49.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:51:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:51:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:51:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:51:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:51:49.876 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:51:49.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:51:49.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:51:49.966 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:51:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:51:50.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:51:50.113 INFO:teuthology.orchestra.run.smithi067.stdout:client9277 2024-04-17T08:51:50.113 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:51:50.113 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:51:50.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-04-17T08:51:50.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.400+0000 7fd793e00640 1 Processor -- start 2024-04-17T08:51:50.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.400+0000 7fd793e00640 1 -- start start 2024-04-17T08:51:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd793e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd794142e10 0x7fd794071d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd793e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79406f220 0x7fd794072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd793e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7941612d0 con 0x7fd7941460a0 2024-04-17T08:51:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd793e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd7941431f0 con 0x7fd794142e10 2024-04-17T08:51:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd793e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd794160e40 con 0x7fd79406f220 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd78a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79406f220 0x7fd794072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd792a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd794142e10 0x7fd794071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd792a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd794142e10 0x7fd794071d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:59432/0 (socket says 172.21.15.67:59432) 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.402+0000 7fd792a00640 1 -- 172.21.15.67:0/3467805945 learned_addr learned my addr 172.21.15.67:0/3467805945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.403+0000 7fd792a00640 1 -- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79406f220 msgr2=0x7fd794072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.403+0000 7fd792a00640 1 --2- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79406f220 0x7fd794072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.403+0000 7fd792a00640 1 -- 172.21.15.67:0/3467805945 >> v1:172.21.15.187:6789/0 conn(0x7fd7941460a0 legacy=0x7fd79406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.403+0000 7fd792a00640 1 -- 172.21.15.67:0/3467805945 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd794160510 con 0x7fd794142e10 2024-04-17T08:51:50.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.405+0000 7fd792a00640 1 --2- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd794142e10 0x7fd794071d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd784009870 tx=0x7fd78402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87da84cf9722e960 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd792000640 1 -- 172.21.15.67:0/3467805945 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784035020 con 0x7fd794142e10 2024-04-17T08:51:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd792000640 1 -- 172.21.15.67:0/3467805945 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd784031640 con 0x7fd794142e10 2024-04-17T08:51:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd792000640 1 -- 172.21.15.67:0/3467805945 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78402d040 con 0x7fd794142e10 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 -- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd794142e10 msgr2=0x7fd794071d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 --2- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd794142e10 0x7fd794071d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd784009870 tx=0x7fd78402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 -- 172.21.15.67:0/3467805945 shutdown_connections 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 --2- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79406f220 0x7fd794072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 --2- 172.21.15.67:0/3467805945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd794142e10 0x7fd794071d20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 -- 172.21.15.67:0/3467805945 >> 172.21.15.67:0/3467805945 conn(0x7fd79406b8c0 msgr2=0x7fd79406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 -- 172.21.15.67:0/3467805945 shutdown_connections 2024-04-17T08:51:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.406+0000 7fd793e00640 1 -- 172.21.15.67:0/3467805945 wait complete. 2024-04-17T08:51:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.408+0000 7fd793e00640 1 Processor -- start 2024-04-17T08:51:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.408+0000 7fd793e00640 1 -- start start 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 0x7fd7941f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7941f8ae0 0x7fd79415bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd7941444b0 con 0x7fd794142e10 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd794160ad0 con 0x7fd7941460a0 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd793e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd794144630 con 0x7fd7941f8ae0 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.409+0000 7fd792a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33370/0 (socket says 172.21.15.67:33370) 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 -- 172.21.15.67:0/3872132414 learned_addr learned my addr 172.21.15.67:0/3872132414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd793400640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7941f8ae0 0x7fd79415bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd78a000640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 0x7fd7941f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 -- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7941f8ae0 msgr2=0x7fd79415bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7941f8ae0 0x7fd79415bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 -- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 msgr2=0x7fd7941f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 0x7fd7941f8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 -- 172.21.15.67:0/3872132414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79415d930 con 0x7fd794142e10 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd78a000640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 0x7fd7941f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.410+0000 7fd792a00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fd78402b930 tx=0x7fd7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.411+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784040070 con 0x7fd794142e10 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.411+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd78403e070 con 0x7fd794142e10 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.411+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784035040 con 0x7fd794142e10 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.411+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79415dc10 con 0x7fd794142e10 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.411+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79415e1d0 con 0x7fd794142e10 2024-04-17T08:51:50.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.412+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd794068a10 con 0x7fd794142e10 2024-04-17T08:51:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.413+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fd784038020 con 0x7fd794142e10 2024-04-17T08:51:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.414+0000 7fd78b400640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 0x7fd77006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.414+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fd7840ad940 con 0x7fd794142e10 2024-04-17T08:51:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.415+0000 7fd78a000640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 0x7fd77006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.416+0000 7fd78a000640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 0x7fd77006eff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd7941f7e10 tx=0x7fd77c008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:50.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.419+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd78407b9e0 con 0x7fd794142e10 2024-04-17T08:51:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:50.668+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fd794072e30 con 0x7fd794142e10 2024-04-17T08:51:51.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.588+0000 7fd78b400640 1 -- 172.21.15.67:0/3872132414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v127) ==== 136+0+0 (secure 0 0 0) 0x7fd784043e40 con 0x7fd794142e10 2024-04-17T08:51:51.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 msgr2=0x7fd77006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 0x7fd77006eff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd7941f7e10 tx=0x7fd77c008040 comp rx=0 tx=0).stop 2024-04-17T08:51:51.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 msgr2=0x7fd7941e61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7fd78402b930 tx=0x7fd7840046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 shutdown_connections 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd77006cb30 0x7fd77006eff0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd7941f8ae0 0x7fd79415bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7941460a0 0x7fd7941f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 --2- 172.21.15.67:0/3872132414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd794142e10 0x7fd7941e61b0 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 >> 172.21.15.67:0/3872132414 conn(0x7fd79406b8c0 msgr2=0x7fd7940b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 shutdown_connections 2024-04-17T08:51:51.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.592+0000 7fd793e00640 1 -- 172.21.15.67:0/3872132414 wait complete. 2024-04-17T08:51:51.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.840+0000 7fe147c00640 1 Processor -- start 2024-04-17T08:51:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.840+0000 7fe147c00640 1 -- start start 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe147c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe14006f670 0x7fe14006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe147c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1400cd020 0x7fe1400cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe147c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe14006ff90 con 0x7fe140071d10 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe147c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1400ceb00 con 0x7fe1400cd020 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe147c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe1400cede0 con 0x7fe14006f670 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe13fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1400cd020 0x7fe1400cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe13fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1400cd020 0x7fe1400cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33372/0 (socket says 172.21.15.67:33372) 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe13fe00640 1 -- 172.21.15.67:0/810007349 learned_addr learned my addr 172.21.15.67:0/810007349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.842+0000 7fe145c00640 1 --1- 172.21.15.67:0/810007349 >> v1:172.21.15.67:6789/0 conn(0x7fe140071d10 0x7fe140072160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59506/0 (socket says 172.21.15.67:59506) 2024-04-17T08:51:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.843+0000 7fe145200640 1 --2- 172.21.15.67:0/810007349 >> v2:172.21.15.187:3300/0 conn(0x7fe14006f670 0x7fe14006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.843+0000 7fe13fe00640 1 -- 172.21.15.67:0/810007349 >> v2:172.21.15.187:3300/0 conn(0x7fe14006f670 msgr2=0x7fe14006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.843+0000 7fe13fe00640 1 --2- 172.21.15.67:0/810007349 >> v2:172.21.15.187:3300/0 conn(0x7fe14006f670 0x7fe14006fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.843+0000 7fe13fe00640 1 -- 172.21.15.67:0/810007349 >> v1:172.21.15.67:6789/0 conn(0x7fe140071d10 legacy=0x7fe140072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.843+0000 7fe13fe00640 1 -- 172.21.15.67:0/810007349 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1400cf0c0 con 0x7fe1400cd020 2024-04-17T08:51:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.844+0000 7fe13fe00640 1 --2- 172.21.15.67:0/810007349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cd020 0x7fe1400cd480 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fe14006c910 tx=0x7fe12c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110fb734f94b6779 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.844+0000 7fe13f400640 1 -- 172.21.15.67:0/810007349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c034070 con 0x7fe1400cd020 2024-04-17T08:51:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.844+0000 7fe13f400640 1 -- 172.21.15.67:0/810007349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c030860 con 0x7fe1400cd020 2024-04-17T08:51:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.844+0000 7fe13f400640 1 -- 172.21.15.67:0/810007349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c030c30 con 0x7fe1400cd020 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 -- 172.21.15.67:0/810007349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cd020 msgr2=0x7fe1400cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 --2- 172.21.15.67:0/810007349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cd020 0x7fe1400cd480 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fe14006c910 tx=0x7fe12c02b9a0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 -- 172.21.15.67:0/810007349 shutdown_connections 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 --2- 172.21.15.67:0/810007349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cd020 0x7fe1400cd480 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 --2- 172.21.15.67:0/810007349 >> v2:172.21.15.187:3300/0 conn(0x7fe14006f670 0x7fe14006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 -- 172.21.15.67:0/810007349 >> 172.21.15.67:0/810007349 conn(0x7fe14006bf50 msgr2=0x7fe1400b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 -- 172.21.15.67:0/810007349 shutdown_connections 2024-04-17T08:51:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.845+0000 7fe147c00640 1 -- 172.21.15.67:0/810007349 wait complete. 2024-04-17T08:51:51.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.847+0000 7fe147c00640 1 Processor -- start 2024-04-17T08:51:51.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.847+0000 7fe147c00640 1 -- start start 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe14006f670 0x7fe14014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 0x7fe1401588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1400ce720 con 0x7fe140071d10 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1400cf680 con 0x7fe1400cd020 2024-04-17T08:51:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe147c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1400ce8a0 con 0x7fe14006f670 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe13fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33378/0 (socket says 172.21.15.67:33378) 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.848+0000 7fe13fe00640 1 -- 172.21.15.67:0/3553384852 learned_addr learned my addr 172.21.15.67:0/3553384852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe145200640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe14006f670 0x7fe14014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe145c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 0x7fe1401588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 -- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe14006f670 msgr2=0x7fe14014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe14006f670 0x7fe14014afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 -- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 msgr2=0x7fe1401588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 0x7fe1401588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 -- 172.21.15.67:0/3553384852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1400c57e0 con 0x7fe140071d10 2024-04-17T08:51:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe145c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 0x7fe1401588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:51.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.849+0000 7fe13fe00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02b9e0 tx=0x7fe12c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:51.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.850+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c048070 con 0x7fe140071d10 2024-04-17T08:51:51.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.850+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c030660 con 0x7fe140071d10 2024-04-17T08:51:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.850+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c034040 con 0x7fe140071d10 2024-04-17T08:51:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.850+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1400c5ac0 con 0x7fe140071d10 2024-04-17T08:51:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.850+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1400c6080 con 0x7fe140071d10 2024-04-17T08:51:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.851+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1400cbf80 con 0x7fe140071d10 2024-04-17T08:51:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.852+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe12c037070 con 0x7fe140071d10 2024-04-17T08:51:51.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.853+0000 7fe13cc00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 0x7fe12006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:51.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.853+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fe12c0ae260 con 0x7fe140071d10 2024-04-17T08:51:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.854+0000 7fe145200640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 0x7fe12006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:51.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.855+0000 7fe145200640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 0x7fe12006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe134006fd0 tx=0x7fe134008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:51.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:51.858+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1400cbf80 con 0x7fe140071d10 2024-04-17T08:51:52.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.063+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe140158390 con 0x7fe140071d10 2024-04-17T08:51:52.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.065+0000 7fe13cc00640 1 -- 172.21.15.67:0/3553384852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) ==== 74+0+16615 (secure 0 0 0) 0x7fe12c07c280 con 0x7fe140071d10 2024-04-17T08:51:52.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:52.065 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":250,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:42.471871+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 msgr2=0x7fe12006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 0x7fe12006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe134006fd0 tx=0x7fe134008040 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 msgr2=0x7fe14014baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02b9e0 tx=0x7fe12c004730 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 shutdown_connections 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe12006cb30 0x7fe12006eff0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1400cd020 0x7fe1401588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe140071d10 0x7fe14014baa0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 --2- 172.21.15.67:0/3553384852 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe14006f670 0x7fe14014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.068+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 >> 172.21.15.67:0/3553384852 conn(0x7fe14006bf50 msgr2=0x7fe1400b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.069+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 shutdown_connections 2024-04-17T08:51:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.069+0000 7fe147c00640 1 -- 172.21.15.67:0/3553384852 wait complete. 2024-04-17T08:51:52.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:51:38.821605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:51:39.950246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:51:52.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:51:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.312+0000 7f5b3ae00640 1 Processor -- start 2024-04-17T08:51:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.312+0000 7f5b3ae00640 1 -- start start 2024-04-17T08:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b3ae00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5b34145b00 0x7f5b34160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b3ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b34160920 con 0x7f5b34146060 2024-04-17T08:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b3ae00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b34160c00 con 0x7f5b340b4ac0 2024-04-17T08:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b3ae00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5b34160ee0 con 0x7f5b34145b00 2024-04-17T08:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b39000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b34146060 0x7f5b3415cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59520/0 (socket says 172.21.15.67:59520) 2024-04-17T08:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.314+0000 7f5b39000640 1 -- 172.21.15.67:0/2082907257 learned_addr learned my addr 172.21.15.67:0/2082907257 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b39a00640 1 --2- 172.21.15.67:0/2082907257 >> v2:172.21.15.187:3300/0 conn(0x7f5b34145b00 0x7f5b34160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860318347 0 0) 0x7f5b34160920 con 0x7f5b34146060 2024-04-17T08:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b1c003520 con 0x7f5b34146060 2024-04-17T08:51:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1872478894 0 0) 0x7f5b1c003520 con 0x7f5b34146060 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 >> v2:172.21.15.187:3300/0 conn(0x7f5b34145b00 msgr2=0x7f5b34160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 --2- 172.21.15.67:0/2082907257 >> v2:172.21.15.187:3300/0 conn(0x7f5b34145b00 0x7f5b34160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 >> v1:172.21.15.187:6789/0 conn(0x7f5b340b4ac0 legacy=0x7f5b340b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b341611c0 con 0x7f5b34146060 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f5b20001670 con 0x7f5b34146060 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 >> v1:172.21.15.67:6789/0 conn(0x7f5b34146060 legacy=0x7f5b3415cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c003a00 0x7f5b1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c004800 0x7f5b1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.315+0000 7f5b33e00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34160ee0 con 0x7f5b1c005210 2024-04-17T08:51:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1c00c140 con 0x7f5b1c003a00 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1c00c340 con 0x7f5b1c004800 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b39a00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c003a00 0x7f5b1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 -- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c004800 msgr2=0x7f5b1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c004800 0x7f5b1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 -- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c003a00 msgr2=0x7f5b1c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b1c003a00 0x7f5b1c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.316+0000 7f5b3a400640 1 -- 172.21.15.67:0/2082907257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1c003520 con 0x7f5b1c005210 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.317+0000 7f5b3a400640 1 -- 172.21.15.67:0/2082907257 reap_dead start 2024-04-17T08:51:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3a400640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0044b0 tx=0x7f5b1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df505f78ae40f42 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c034070 con 0x7f5b1c005210 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b2c006d70 con 0x7f5b1c005210 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b33e00640 1 -- 172.21.15.67:0/2082907257 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c004080 con 0x7f5b1c005210 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3ae00640 1 -- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 msgr2=0x7f5b1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0044b0 tx=0x7f5b1c005830 comp rx=0 tx=0).stop 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3ae00640 1 -- 172.21.15.67:0/2082907257 shutdown_connections 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/2082907257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c005210 0x7f5b1c00b9c0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.318+0000 7f5b3ae00640 1 -- 172.21.15.67:0/2082907257 >> 172.21.15.67:0/2082907257 conn(0x7f5b340b23c0 msgr2=0x7f5b340b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.319+0000 7f5b3ae00640 1 -- 172.21.15.67:0/2082907257 shutdown_connections 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.319+0000 7f5b3ae00640 1 -- 172.21.15.67:0/2082907257 wait complete. 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.320+0000 7f5b3ae00640 1 Processor -- start 2024-04-17T08:51:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.320+0000 7f5b3ae00640 1 -- start start 2024-04-17T08:51:52.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b34143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 0x7f5b340b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34160340 con 0x7f5b341f4570 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b340cfba0 con 0x7f5b340b4f10 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.321+0000 7f5b3ae00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b341604c0 con 0x7f5b1c005210 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33392/0 (socket says 172.21.15.67:33392) 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 -- 172.21.15.67:0/3637360657 learned_addr learned my addr 172.21.15.67:0/3637360657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b39000640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 0x7f5b340b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b39a00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b34143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 -- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 msgr2=0x7f5b34143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b34143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 -- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 msgr2=0x7f5b340b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 0x7f5b340b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b3a400640 1 -- 172.21.15.67:0/3637360657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b341444c0 con 0x7f5b341f4570 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.322+0000 7f5b39000640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 0x7f5b340b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b3a400640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006fd0 tx=0x7f5b2c003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c034040 con 0x7f5b341f4570 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b2c031c70 con 0x7f5b341f4570 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b340aa8e0 con 0x7f5b341f4570 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c044650 con 0x7f5b341f4570 2024-04-17T08:51:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.323+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b340aaea0 con 0x7f5b341f4570 2024-04-17T08:51:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.324+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b34068a10 con 0x7f5b341f4570 2024-04-17T08:51:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.325+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f5b2c007000 con 0x7f5b341f4570 2024-04-17T08:51:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.326+0000 7f5b31600640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 0x7f5b1406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.326+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f5b2c0adb60 con 0x7f5b341f4570 2024-04-17T08:51:52.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.327+0000 7f5b39a00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 0x7f5b1406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.328+0000 7f5b39a00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 0x7f5b1406f020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b340ab960 tx=0x7f5b28008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.331+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5b2c031e20 con 0x7f5b341f4570 2024-04-17T08:51:52.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.533+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b34068610 con 0x7f5b341f4570 2024-04-17T08:51:52.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.534+0000 7f5b31600640 1 -- 172.21.15.67:0/3637360657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v250) ==== 93+0+10 (secure 0 0 0) 0x7f5b2c07ba60 con 0x7f5b341f4570 2024-04-17T08:51:52.534 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 msgr2=0x7f5b1406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 0x7f5b1406f020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b340ab960 tx=0x7f5b28008040 comp rx=0 tx=0).stop 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 msgr2=0x7f5b341f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006fd0 tx=0x7f5b2c003a70 comp rx=0 tx=0).stop 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 shutdown_connections 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5b1406cb60 0x7f5b1406f020 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b341f4570 0x7f5b341f4a20 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b340b4f10 0x7f5b340b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 --2- 172.21.15.67:0/3637360657 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b1c005210 0x7f5b34143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 >> 172.21.15.67:0/3637360657 conn(0x7f5b340b23c0 msgr2=0x7f5b3406f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.537+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 shutdown_connections 2024-04-17T08:51:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.538+0000 7f5b3ae00640 1 -- 172.21.15.67:0/3637360657 wait complete. 2024-04-17T08:51:52.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:51:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.779+0000 7f45cb200640 1 Processor -- start 2024-04-17T08:51:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.780+0000 7f45cb200640 1 -- start start 2024-04-17T08:51:52.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.781+0000 7f45cb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc145f80 0x7f45cc0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.781+0000 7f45cb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc146360 con 0x7f45cc06c270 2024-04-17T08:51:52.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.781+0000 7f45cb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc071e70 con 0x7f45cc0cd750 2024-04-17T08:51:52.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.781+0000 7f45cb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc0cdbb0 con 0x7f45cc145f80 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.782+0000 7f45ca800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45cc06c270 0x7f45cc06e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59536/0 (socket says 172.21.15.67:59536) 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.782+0000 7f45ca800640 1 -- 172.21.15.67:0/2517162952 learned_addr learned my addr 172.21.15.67:0/2517162952 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.782+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc145f80 0x7f45cc0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.782+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997510064 0 0) 0x7f45cc146360 con 0x7f45cc06c270 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.782+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45b0003520 con 0x7f45cc06c270 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2804035514 0 0) 0x7f45b0003520 con 0x7f45cc06c270 2024-04-17T08:51:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc145f80 msgr2=0x7f45cc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc145f80 0x7f45cc0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 >> v1:172.21.15.187:6789/0 conn(0x7f45cc0cd750 legacy=0x7f45cc06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc072150 con 0x7f45cc06c270 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f45c40019f0 con 0x7f45cc06c270 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 >> v1:172.21.15.67:6789/0 conn(0x7f45cc06c270 legacy=0x7f45cc06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45b00039e0 0x7f45b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45b0004800 0x7f45b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00052e0 0x7f45b000fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45b0010540 con 0x7f45b00052e0 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45b00106f0 con 0x7f45b0004800 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45b00108f0 con 0x7f45b00039e0 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc145f80 0x7f45cc0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:52.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.783+0000 7f45ca800640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00052e0 0x7f45b000fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45c3e00640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45b0004800 0x7f45b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45b00039e0 0x7f45b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45ca800640 1 -- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45b00039e0 msgr2=0x7f45b0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45ca800640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45b00039e0 0x7f45b0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45ca800640 1 -- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45b0004800 msgr2=0x7f45b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45ca800640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45b0004800 0x7f45b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45ca800640 1 -- 172.21.15.67:0/2517162952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b0003520 con 0x7f45b00052e0 2024-04-17T08:51:52.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.784+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45b00039e0 0x7f45b0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45ca800640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00052e0 0x7f45b000fdd0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f45c400a7b0 tx=0x7f45b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7cdb11f4335331 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400a3d0 con 0x7f45b00052e0 2024-04-17T08:51:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c4034630 con 0x7f45b00052e0 2024-04-17T08:51:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45c9400640 1 -- 172.21.15.67:0/2517162952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4034aa0 con 0x7f45b00052e0 2024-04-17T08:51:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45cb200640 1 -- 172.21.15.67:0/2517162952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00052e0 msgr2=0x7f45b000fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45cb200640 1 --2- 172.21.15.67:0/2517162952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00052e0 0x7f45b000fdd0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f45c400a7b0 tx=0x7f45b0005830 comp rx=0 tx=0).stop 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.786+0000 7f45ca800640 1 -- 172.21.15.67:0/2517162952 reap_dead start 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.787+0000 7f45cb200640 1 -- 172.21.15.67:0/2517162952 shutdown_connections 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.787+0000 7f45cb200640 1 -- 172.21.15.67:0/2517162952 >> 172.21.15.67:0/2517162952 conn(0x7f45cc0b2190 msgr2=0x7f45cc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.787+0000 7f45cb200640 1 -- 172.21.15.67:0/2517162952 shutdown_connections 2024-04-17T08:51:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.787+0000 7f45cb200640 1 -- 172.21.15.67:0/2517162952 wait complete. 2024-04-17T08:51:52.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.788+0000 7f45cb200640 1 Processor -- start 2024-04-17T08:51:52.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.789+0000 7f45cb200640 1 -- start start 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.789+0000 7f45cb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 0x7f45cc13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45cb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 0x7f45cc141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45cb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45cb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc072c30 con 0x7f45cc141580 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45cb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc0727a0 con 0x7f45cc13df00 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45cb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc072db0 con 0x7f45cc13e6d0 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33406/0 (socket says 172.21.15.67:33406) 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 -- 172.21.15.67:0/2879646022 learned_addr learned my addr 172.21.15.67:0/2879646022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 0x7f45cc13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45c3e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 0x7f45cc141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 -- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 msgr2=0x7f45cc141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 0x7f45cc141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 -- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 msgr2=0x7f45cc13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 0x7f45cc13fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45ca800640 1 -- 172.21.15.67:0/2879646022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc140230 con 0x7f45cc141580 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45c3e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 0x7f45cc141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.790+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 0x7f45cc13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:52.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45ca800640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f45c402cbd0 tx=0x7f45c4001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4044070 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c403a040 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400a3d0 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc13ec50 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.791+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc13f210 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.792+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc068a10 con 0x7f45cc141580 2024-04-17T08:51:52.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.793+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f45c4047070 con 0x7f45cc141580 2024-04-17T08:51:52.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.794+0000 7f45c2000640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 0x7f45a806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:52.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.794+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7f45c40ad6d0 con 0x7f45cc141580 2024-04-17T08:51:52.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.795+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 0x7f45a806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:52.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.796+0000 7f45c9e00640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 0x7f45a806eff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45bc006e80 tx=0x7f45bc008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:52.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:52.798+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f45c407b050 con 0x7f45cc141580 2024-04-17T08:51:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.001+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45cc0b4110 con 0x7f45cc141580 2024-04-17T08:51:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.002+0000 7f45c2000640 1 -- 172.21.15.67:0/2879646022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v250) ==== 104+0+11 (secure 0 0 0) 0x7f45c407f350 con 0x7f45cc141580 2024-04-17T08:51:53.002 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 msgr2=0x7f45a806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 0x7f45a806eff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45bc006e80 tx=0x7f45bc008040 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 msgr2=0x7f45cc14e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f45c402cbd0 tx=0x7f45c4001cf0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 shutdown_connections 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f45a806cb30 0x7f45a806eff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc141580 0x7f45cc14e4e0 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45cc13e6d0 0x7f45cc141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 --2- 172.21.15.67:0/2879646022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45cc13df00 0x7f45cc13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.005+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 >> 172.21.15.67:0/2879646022 conn(0x7f45cc0b2190 msgr2=0x7f45cc15ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.006+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 shutdown_connections 2024-04-17T08:51:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.006+0000 7f45cb200640 1 -- 172.21.15.67:0/2879646022 wait complete. 2024-04-17T08:51:53.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:51:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.254+0000 7fbee6200640 1 Processor -- start 2024-04-17T08:51:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.255+0000 7fbee6200640 1 -- start start 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.256+0000 7fbee6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbee00cd040 0x7fbee00cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.256+0000 7fbee6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee015ed00 0x7fbee00b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.256+0000 7fbee6200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee00cd960 con 0x7fbee00ce610 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.256+0000 7fbee6200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbee0071f70 con 0x7fbee00cd040 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.256+0000 7fbee6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbee0072250 con 0x7fbee015ed00 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbee00cd040 0x7fbee00cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbee00cd040 0x7fbee00cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33410/0 (socket says 172.21.15.67:33410) 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 -- 172.21.15.67:0/548973001 learned_addr learned my addr 172.21.15.67:0/548973001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedfe00640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee015ed00 0x7fbee00b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 -- 172.21.15.67:0/548973001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee015ed00 msgr2=0x7fbee00b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee015ed00 0x7fbee00b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 -- 172.21.15.67:0/548973001 >> v1:172.21.15.187:6789/0 conn(0x7fbee00ce610 legacy=0x7fbee00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.257+0000 7fbedf400640 1 -- 172.21.15.67:0/548973001 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee0072530 con 0x7fbee00cd040 2024-04-17T08:51:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.258+0000 7fbedf400640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00cd040 0x7fbee00cd420 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fbee006c660 tx=0x7fbed002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736d41e84650c55a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.259+0000 7fbede000640 1 -- 172.21.15.67:0/548973001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed002ca50 con 0x7fbee00cd040 2024-04-17T08:51:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.259+0000 7fbede000640 1 -- 172.21.15.67:0/548973001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed002cbf0 con 0x7fbee00cd040 2024-04-17T08:51:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.259+0000 7fbede000640 1 -- 172.21.15.67:0/548973001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00343f0 con 0x7fbee00cd040 2024-04-17T08:51:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 -- 172.21.15.67:0/548973001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00cd040 msgr2=0x7fbee00cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00cd040 0x7fbee00cd420 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fbee006c660 tx=0x7fbed002beb0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 -- 172.21.15.67:0/548973001 shutdown_connections 2024-04-17T08:51:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee015ed00 0x7fbee00b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 --2- 172.21.15.67:0/548973001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00cd040 0x7fbee00cd420 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 -- 172.21.15.67:0/548973001 >> 172.21.15.67:0/548973001 conn(0x7fbee006b8c0 msgr2=0x7fbee006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 -- 172.21.15.67:0/548973001 shutdown_connections 2024-04-17T08:51:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.260+0000 7fbee6200640 1 -- 172.21.15.67:0/548973001 wait complete. 2024-04-17T08:51:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.262+0000 7fbee6200640 1 Processor -- start 2024-04-17T08:51:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.262+0000 7fbee6200640 1 -- start start 2024-04-17T08:51:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee00cd040 0x7fbee00cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee015ed00 0x7fbee0161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbee00b4600 con 0x7fbee00ce610 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0072af0 con 0x7fbee015ed00 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbee6200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbee00b4780 con 0x7fbee00cd040 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbedea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbedea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33420/0 (socket says 172.21.15.67:33420) 2024-04-17T08:51:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.263+0000 7fbedea00640 1 -- 172.21.15.67:0/1687425335 learned_addr learned my addr 172.21.15.67:0/1687425335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 -- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee00cd040 msgr2=0x7fbee00cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee00cd040 0x7fbee00cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 -- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee015ed00 msgr2=0x7fbee0161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee015ed00 0x7fbee0161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 -- 172.21.15.67:0/1687425335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee015ff90 con 0x7fbee00ce610 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbedea00640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c420 tx=0x7fbec800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8007020 con 0x7fbee00ce610 2024-04-17T08:51:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec80071c0 con 0x7fbee00ce610 2024-04-17T08:51:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee0160270 con 0x7fbee00ce610 2024-04-17T08:51:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8007590 con 0x7fbee00ce610 2024-04-17T08:51:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.264+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee0160830 con 0x7fbee00ce610 2024-04-17T08:51:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.266+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee0068b90 con 0x7fbee00ce610 2024-04-17T08:51:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.266+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fbec8019020 con 0x7fbee00ce610 2024-04-17T08:51:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.268+0000 7fbed7400640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 0x7fbebc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.268+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fbec808b710 con 0x7fbee00ce610 2024-04-17T08:51:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.269+0000 7fbedf400640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 0x7fbebc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.270+0000 7fbedf400640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 0x7fbebc06eff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbed0035000 tx=0x7fbed002e000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.272+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbec80597f0 con 0x7fbee00ce610 2024-04-17T08:51:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.476+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbee00b2b80 con 0x7fbee00ce610 2024-04-17T08:51:53.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.477+0000 7fbed7400640 1 -- 172.21.15.67:0/1687425335 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v250) ==== 100+0+11 (secure 0 0 0) 0x7fbec80590f0 con 0x7fbee00ce610 2024-04-17T08:51:53.476 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:51:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 msgr2=0x7fbebc06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 0x7fbebc06eff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbed0035000 tx=0x7fbed002e000 comp rx=0 tx=0).stop 2024-04-17T08:51:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 msgr2=0x7fbee00c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fbec800c420 tx=0x7fbec800c8e0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 shutdown_connections 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbebc06cb30 0x7fbebc06eff0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbee015ed00 0x7fbee0161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbee00ce610 0x7fbee00c5630 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 --2- 172.21.15.67:0/1687425335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbee00cd040 0x7fbee00cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 >> 172.21.15.67:0/1687425335 conn(0x7fbee006b8c0 msgr2=0x7fbee006e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 shutdown_connections 2024-04-17T08:51:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.480+0000 7fbee6200640 1 -- 172.21.15.67:0/1687425335 wait complete. 2024-04-17T08:51:53.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:51:53.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.722+0000 7fd41b000640 1 Processor -- start 2024-04-17T08:51:53.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.722+0000 7fd41b000640 1 -- start start 2024-04-17T08:51:53.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 0x7fd4140cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd414160280 0x7fd41406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 0x7fd414142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd41415ce30 con 0x7fd414160280 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd4140cd4a0 con 0x7fd414141ed0 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd41b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd4141607f0 con 0x7fd41406b8c0 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd419200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd414160280 0x7fd41406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd419200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd414160280 0x7fd41406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36804/0 (socket says 172.21.15.67:36804) 2024-04-17T08:51:53.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.724+0000 7fd419200640 1 -- 172.21.15.67:0/1680174475 learned_addr learned my addr 172.21.15.67:0/1680174475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd41a600640 1 --2- 172.21.15.67:0/1680174475 >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 0x7fd414142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419c00640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 0x7fd4140cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419200640 1 -- 172.21.15.67:0/1680174475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 msgr2=0x7fd4140cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419200640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 0x7fd4140cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419200640 1 -- 172.21.15.67:0/1680174475 >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 msgr2=0x7fd414142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419200640 1 --2- 172.21.15.67:0/1680174475 >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 0x7fd414142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419200640 1 -- 172.21.15.67:0/1680174475 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd414160ad0 con 0x7fd414160280 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd41a600640 1 --2- 172.21.15.67:0/1680174475 >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 0x7fd414142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.725+0000 7fd419c00640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 0x7fd4140cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.726+0000 7fd419200640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd414160280 0x7fd41406bca0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fd414071d80 tx=0x7fd40002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db95a3048bafde87 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.726+0000 7fd413e00640 1 -- 172.21.15.67:0/1680174475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400035020 con 0x7fd414160280 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.726+0000 7fd413e00640 1 -- 172.21.15.67:0/1680174475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd400031880 con 0x7fd414160280 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.726+0000 7fd413e00640 1 -- 172.21.15.67:0/1680174475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400031c50 con 0x7fd414160280 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 -- 172.21.15.67:0/1680174475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd414160280 msgr2=0x7fd41406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd414160280 0x7fd41406bca0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fd414071d80 tx=0x7fd40002b920 comp rx=0 tx=0).stop 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 -- 172.21.15.67:0/1680174475 shutdown_connections 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 --2- 172.21.15.67:0/1680174475 >> v2:172.21.15.187:3300/0 conn(0x7fd414141ed0 0x7fd414142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd414160280 0x7fd41406bca0 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 --2- 172.21.15.67:0/1680174475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd41406b8c0 0x7fd4140cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 -- 172.21.15.67:0/1680174475 >> 172.21.15.67:0/1680174475 conn(0x7fd414067d10 msgr2=0x7fd4140b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 -- 172.21.15.67:0/1680174475 shutdown_connections 2024-04-17T08:51:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.727+0000 7fd41b000640 1 -- 172.21.15.67:0/1680174475 wait complete. 2024-04-17T08:51:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.729+0000 7fd41b000640 1 Processor -- start 2024-04-17T08:51:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.729+0000 7fd41b000640 1 -- start start 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd414141ed0 0x7fd414155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd414160280 0x7fd4140b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd41406c1e0 con 0x7fd41406b8c0 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd4140cddf0 con 0x7fd414160280 2024-04-17T08:51:53.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd41b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd414161090 con 0x7fd414141ed0 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd419c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd419c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36808/0 (socket says 172.21.15.67:36808) 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.730+0000 7fd419c00640 1 -- 172.21.15.67:0/2035323355 learned_addr learned my addr 172.21.15.67:0/2035323355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd41a600640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd414160280 0x7fd4140b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419200640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd414141ed0 0x7fd414155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 -- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd414141ed0 msgr2=0x7fd414155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd414141ed0 0x7fd414155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 -- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd414160280 msgr2=0x7fd4140b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd414160280 0x7fd4140b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 -- 172.21.15.67:0/2035323355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4141563e0 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd419c00640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fd40800c7c0 tx=0x7fd40800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408015070 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd408007a30 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408007e00 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.731+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4140b4090 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.732+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4140b4610 con 0x7fd41406b8c0 2024-04-17T08:51:53.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.733+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd414068a10 con 0x7fd41406b8c0 2024-04-17T08:51:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.734+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fd408018070 con 0x7fd41406b8c0 2024-04-17T08:51:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.735+0000 7fd411600640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 0x7fd3f406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.735+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fd40808bf10 con 0x7fd41406b8c0 2024-04-17T08:51:53.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.736+0000 7fd419200640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 0x7fd3f406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.737+0000 7fd419200640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 0x7fd3f406eff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd40003c040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:53.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.740+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd4141563e0 con 0x7fd41406b8c0 2024-04-17T08:51:53.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.949+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd414146220 con 0x7fd41406b8c0 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.950+0000 7fd411600640 1 -- 172.21.15.67:0/2035323355 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v250) ==== 84+0+209 (secure 0 0 0) 0x7fd408059f40 con 0x7fd41406b8c0 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:53.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 30, 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": "cephfs_metadata" 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 31, 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": "cephfs_data" 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:51:53.950 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 msgr2=0x7fd3f406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 0x7fd3f406eff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd400036000 tx=0x7fd40003c040 comp rx=0 tx=0).stop 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 msgr2=0x7fd414145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fd40800c7c0 tx=0x7fd40800cc80 comp rx=0 tx=0).stop 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 shutdown_connections 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fd3f406cb30 0x7fd3f406eff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd414160280 0x7fd4140b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd414141ed0 0x7fd414155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 --2- 172.21.15.67:0/2035323355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd41406b8c0 0x7fd414145910 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.952+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 >> 172.21.15.67:0/2035323355 conn(0x7fd414067d10 msgr2=0x7fd41415b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.953+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 shutdown_connections 2024-04-17T08:51:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:53.953+0000 7fd41b000640 1 -- 172.21.15.67:0/2035323355 wait complete. 2024-04-17T08:51:53.973 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-04-17T08:51:53.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-04-17T08:51:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.203+0000 7fe061c00640 1 Processor -- start 2024-04-17T08:51:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.203+0000 7fe061c00640 1 -- start start 2024-04-17T08:51:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe05c0b3be0 0x7fe05c155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c0b4070 con 0x7fe05c0cfc80 2024-04-17T08:51:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe05c15f890 con 0x7fe05c0b3800 2024-04-17T08:51:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe05c1629b0 con 0x7fe05c0b3be0 2024-04-17T08:51:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe05c0b3be0 0x7fe05c155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe05c0b3be0 0x7fe05c155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36824/0 (socket says 172.21.15.67:36824) 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061200640 1 -- 172.21.15.67:0/2316811543 learned_addr learned my addr 172.21.15.67:0/2316811543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe053e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe05c0cfc80 0x7fe05c162e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33308/0 (socket says 172.21.15.67:33308) 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.205+0000 7fe061200640 1 -- 172.21.15.67:0/2316811543 >> v1:172.21.15.187:6789/0 conn(0x7fe05c0b3800 legacy=0x7fe05c15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.206+0000 7fe061200640 1 -- 172.21.15.67:0/2316811543 >> v1:172.21.15.67:6789/0 conn(0x7fe05c0cfc80 legacy=0x7fe05c162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.206+0000 7fe061200640 1 -- 172.21.15.67:0/2316811543 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c15fb70 con 0x7fe05c0b3be0 2024-04-17T08:51:54.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.207+0000 7fe061200640 1 --2- 172.21.15.67:0/2316811543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c155560 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fe054009e50 tx=0x7fe05402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2e7ca2cd37d1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.207+0000 7fe05b400640 1 -- 172.21.15.67:0/2316811543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05402cbb0 con 0x7fe05c0b3be0 2024-04-17T08:51:54.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.207+0000 7fe05b400640 1 -- 172.21.15.67:0/2316811543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe05402cd50 con 0x7fe05c0b3be0 2024-04-17T08:51:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.207+0000 7fe05b400640 1 -- 172.21.15.67:0/2316811543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054034610 con 0x7fe05c0b3be0 2024-04-17T08:51:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 -- 172.21.15.67:0/2316811543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 msgr2=0x7fe05c155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 --2- 172.21.15.67:0/2316811543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c155560 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fe054009e50 tx=0x7fe05402c010 comp rx=0 tx=0).stop 2024-04-17T08:51:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 -- 172.21.15.67:0/2316811543 shutdown_connections 2024-04-17T08:51:54.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 --2- 172.21.15.67:0/2316811543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c155560 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 -- 172.21.15.67:0/2316811543 >> 172.21.15.67:0/2316811543 conn(0x7fe05c067600 msgr2=0x7fe05c0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 -- 172.21.15.67:0/2316811543 shutdown_connections 2024-04-17T08:51:54.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.208+0000 7fe061c00640 1 -- 172.21.15.67:0/2316811543 wait complete. 2024-04-17T08:51:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.210+0000 7fe061c00640 1 Processor -- start 2024-04-17T08:51:54.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.210+0000 7fe061c00640 1 -- start start 2024-04-17T08:51:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 0x7fe05c14eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe05c0cfc80 0x7fe05c0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c162640 con 0x7fe05c0b3be0 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe05c163550 con 0x7fe05c0cfc80 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe061c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe05c155aa0 con 0x7fe05c0b3800 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe053e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe053e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36836/0 (socket says 172.21.15.67:36836) 2024-04-17T08:51:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.211+0000 7fe053e00640 1 -- 172.21.15.67:0/4037387690 learned_addr learned my addr 172.21.15.67:0/4037387690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe05be00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 0x7fe05c14eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe061200640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe05c0cfc80 0x7fe05c0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 -- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 msgr2=0x7fe05c14eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 0x7fe05c14eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 -- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe05c0cfc80 msgr2=0x7fe05c0cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe05c0cfc80 0x7fe05c0cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 -- 172.21.15.67:0/4037387690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05c0b69d0 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe05be00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 0x7fe05c14eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe053e00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fe04400c430 tx=0x7fe04400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0440074e0 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe044007680 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044004a00 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.212+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05c1e8600 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.213+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe05c14f420 con 0x7fe05c0b3be0 2024-04-17T08:51:54.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.214+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05c068a10 con 0x7fe05c0b3be0 2024-04-17T08:51:54.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.215+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe04401a020 con 0x7fe05c0b3be0 2024-04-17T08:51:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.216+0000 7fe058c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 0x7fe03806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.216+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5737+0+0 (secure 0 0 0) 0x7fe04408b800 con 0x7fe05c0b3be0 2024-04-17T08:51:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.217+0000 7fe05be00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 0x7fe03806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.218+0000 7fe05be00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 0x7fe03806f020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00a830 tx=0x7fe04c008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.220+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe044059890 con 0x7fe05c0b3be0 2024-04-17T08:51:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.426+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7fe05c06da00 con 0x7fe05c0b3be0 2024-04-17T08:51:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.619+0000 7fe058c00640 1 -- 172.21.15.67:0/4037387690 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v251) ==== 144+0+0 (secure 0 0 0) 0x7fe044059190 con 0x7fe05c0b3be0 2024-04-17T08:51:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_metadata' created 2024-04-17T08:51:54.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.622+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 msgr2=0x7fe03806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.622+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 0x7fe03806f020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00a830 tx=0x7fe04c008040 comp rx=0 tx=0).stop 2024-04-17T08:51:54.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.622+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 msgr2=0x7fe05c0b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.622+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fe04400c430 tx=0x7fe04400c8f0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 shutdown_connections 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe03806cb60 0x7fe03806f020 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe05c0cfc80 0x7fe05c0cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe05c0b3be0 0x7fe05c0b5f90 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 --2- 172.21.15.67:0/4037387690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe05c0b3800 0x7fe05c14eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.623+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 >> 172.21.15.67:0/4037387690 conn(0x7fe05c067600 msgr2=0x7fe05c1617e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.624+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 shutdown_connections 2024-04-17T08:51:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.624+0000 7fe061c00640 1 -- 172.21.15.67:0/4037387690 wait complete. 2024-04-17T08:51:54.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:51:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.855+0000 7f902da00640 1 Processor -- start 2024-04-17T08:51:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.856+0000 7f902da00640 1 -- start start 2024-04-17T08:51:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.857+0000 7f902da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9028161eb0 0x7f90280b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.857+0000 7f902da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9028162a10 con 0x7f9028162290 2024-04-17T08:51:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.857+0000 7f902da00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9028162cf0 con 0x7f90280d2da0 2024-04-17T08:51:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.857+0000 7f902da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f90281627c0 con 0x7f9028161eb0 2024-04-17T08:51:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.857+0000 7f901e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9028162290 0x7f90280d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33324/0 (socket says 172.21.15.67:33324) 2024-04-17T08:51:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.858+0000 7f901e000640 1 -- 172.21.15.67:0/2301839407 learned_addr learned my addr 172.21.15.67:0/2301839407 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.858+0000 7f9026a00640 1 --2- 172.21.15.67:0/2301839407 >> v2:172.21.15.187:3300/0 conn(0x7f9028161eb0 0x7f90280b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.858+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2313975374 0 0) 0x7f9028162a10 con 0x7f9028162290 2024-04-17T08:51:54.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.858+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f900c003520 con 0x7f9028162290 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136380673 0 0) 0x7f9028162cf0 con 0x7f90280d2da0 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9028162a10 con 0x7f90280d2da0 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1007730508 0 0) 0x7f900c003520 con 0x7f9028162290 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 >> v2:172.21.15.187:3300/0 conn(0x7f9028161eb0 msgr2=0x7f90280b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 --2- 172.21.15.67:0/2301839407 >> v2:172.21.15.187:3300/0 conn(0x7f9028161eb0 0x7f90280b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 >> v1:172.21.15.187:6789/0 conn(0x7f90280d2da0 legacy=0x7f90280718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9028162fd0 con 0x7f9028162290 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9010001670 con 0x7f9028162290 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 >> v1:172.21.15.67:6789/0 conn(0x7f9028162290 legacy=0x7f90280d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c003b90 0x7f900c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 0x7f900c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f900c0051c0 0x7f900c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f90281627c0 con 0x7f900c004800 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f900c00c0c0 con 0x7f900c0051c0 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.859+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f900c00c2c0 con 0x7f900c003b90 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 0x7f900c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f9026a00640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c003b90 0x7f900c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f9027400640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f900c0051c0 0x7f900c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 -- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c003b90 msgr2=0x7f900c003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c003b90 0x7f900c003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 -- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f900c0051c0 msgr2=0x7f900c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f900c0051c0 0x7f900c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f901e000640 1 -- 172.21.15.67:0/2301839407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c003520 con 0x7f900c004800 2024-04-17T08:51:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f9026a00640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c003b90 0x7f900c003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f9027400640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f900c0051c0 0x7f900c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.860+0000 7f9027400640 1 -- 172.21.15.67:0/2301839407 reap_dead start 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.861+0000 7f901e000640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 0x7f900c004c80 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fc0 tx=0x7f900c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855e85b0ebbe8c83 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9010007be0 con 0x7f900c004800 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9010030070 con 0x7f900c004800 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f9026000640 1 -- 172.21.15.67:0/2301839407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9010005d30 con 0x7f900c004800 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 -- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 msgr2=0x7f900c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 0x7f900c004c80 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fc0 tx=0x7f900c005830 comp rx=0 tx=0).stop 2024-04-17T08:51:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 -- 172.21.15.67:0/2301839407 shutdown_connections 2024-04-17T08:51:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 --2- 172.21.15.67:0/2301839407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c004800 0x7f900c004c80 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 -- 172.21.15.67:0/2301839407 >> 172.21.15.67:0/2301839407 conn(0x7f9028070260 msgr2=0x7f9028070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 -- 172.21.15.67:0/2301839407 shutdown_connections 2024-04-17T08:51:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.862+0000 7f902da00640 1 -- 172.21.15.67:0/2301839407 wait complete. 2024-04-17T08:51:54.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.864+0000 7f902da00640 1 Processor -- start 2024-04-17T08:51:54.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.864+0000 7f902da00640 1 -- start start 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c004800 0x7f9028156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902814cb50 0x7f90280ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f902806e820 con 0x7f9028148e00 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f902806e6a0 con 0x7f902814cb50 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.865+0000 7f902da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f902806e9a0 con 0x7f900c004800 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36856/0 (socket says 172.21.15.67:36856) 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 -- 172.21.15.67:0/930675153 learned_addr learned my addr 172.21.15.67:0/930675153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f901e000640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902814cb50 0x7f90280ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9026a00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c004800 0x7f9028156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 -- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c004800 msgr2=0x7f9028156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c004800 0x7f9028156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 -- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902814cb50 msgr2=0x7f90280ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902814cb50 0x7f90280ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.866+0000 7f9027400640 1 -- 172.21.15.67:0/930675153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90281574b0 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f9027400640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9028162700 tx=0x7f902000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020018070 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9020006d70 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020003bb0 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9028157790 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.867+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9028157d50 con 0x7f9028148e00 2024-04-17T08:51:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.869+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902815bf60 con 0x7f9028148e00 2024-04-17T08:51:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.870+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f902001b070 con 0x7f9028148e00 2024-04-17T08:51:54.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.871+0000 7f901f400640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 0x7f900406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:54.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.871+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 6110+0+0 (secure 0 0 0) 0x7f902008b5f0 con 0x7f9028148e00 2024-04-17T08:51:54.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.872+0000 7f9026a00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 0x7f900406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:54.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.873+0000 7f9026a00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 0x7f900406eff0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f902806a830 tx=0x7f9018008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:54.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:54.877+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f902815bf60 con 0x7f9028148e00 2024-04-17T08:51:55.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.080+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f90281616d0 con 0x7f9028148e00 2024-04-17T08:51:55.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.626+0000 7f901f400640 1 -- 172.21.15.67:0/930675153 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v252) ==== 176+0+0 (secure 0 0 0) 0x7f9020059410 con 0x7f9028148e00 2024-04-17T08:51:55.626 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_data' created 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 msgr2=0x7f900406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 0x7f900406eff0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f902806a830 tx=0x7f9018008040 comp rx=0 tx=0).stop 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 msgr2=0x7f9028149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f9028162700 tx=0x7f902000db90 comp rx=0 tx=0).stop 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 shutdown_connections 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f900406cb30 0x7f900406eff0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028148e00 0x7f9028149290 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f902814cb50 0x7f90280ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 --2- 172.21.15.67:0/930675153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f900c004800 0x7f9028156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.630+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 >> 172.21.15.67:0/930675153 conn(0x7f9028070260 msgr2=0x7f90280d2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.631+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 shutdown_connections 2024-04-17T08:51:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.631+0000 7f902da00640 1 -- 172.21.15.67:0/930675153 wait complete. 2024-04-17T08:51:55.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-04-17T08:51:55.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.857+0000 7f1e91400640 1 Processor -- start 2024-04-17T08:51:55.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.857+0000 7f1e91400640 1 -- start start 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e91400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1e8c0cfe00 0x7f1e8c06e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e91400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c0d04b0 0x7f1e8c0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e91400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e8c15beb0 con 0x7f1e8c15f480 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e91400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1e8c15c190 con 0x7f1e8c0cfe00 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e91400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e8c15ec80 con 0x7f1e8c0d04b0 2024-04-17T08:51:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e89e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e8c15f480 0x7f1e8c15b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33336/0 (socket says 172.21.15.67:33336) 2024-04-17T08:51:55.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e89e00640 1 -- 172.21.15.67:0/273449737 learned_addr learned my addr 172.21.15.67:0/273449737 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:55.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e8a800640 1 --2- 172.21.15.67:0/273449737 >> v2:172.21.15.187:3300/0 conn(0x7f1e8c0cfe00 0x7f1e8c06e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c0d04b0 0x7f1e8c0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.859+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439311809 0 0) 0x7f1e8c15beb0 con 0x7f1e8c15f480 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e70003520 con 0x7f1e8c15f480 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 89671595 0 0) 0x7f1e70003520 con 0x7f1e8c15f480 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c0d04b0 msgr2=0x7f1e8c0cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c0d04b0 0x7f1e8c0cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 >> v2:172.21.15.187:3300/0 conn(0x7f1e8c0cfe00 msgr2=0x7f1e8c06e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 --2- 172.21.15.67:0/273449737 >> v2:172.21.15.187:3300/0 conn(0x7f1e8c0cfe00 0x7f1e8c06e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8c15ef60 con 0x7f1e8c15f480 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1e78001690 con 0x7f1e8c15f480 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e8a800640 1 --2- 172.21.15.67:0/273449737 >> v2:172.21.15.187:3300/0 conn(0x7f1e8c0cfe00 0x7f1e8c06e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 >> v1:172.21.15.67:6789/0 conn(0x7f1e8c15f480 legacy=0x7f1e8c15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700039b0 0x7f1e70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e70004800 0x7f1e70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e700051c0 0x7f1e7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c15c190 con 0x7f1e700051c0 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c15ec80 con 0x7f1e700039b0 2024-04-17T08:51:55.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7000c000 con 0x7f1e70004800 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c0d04b0 0x7f1e8c0cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.860+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e700051c0 0x7f1e7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e89e00640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e70004800 0x7f1e70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8a800640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700039b0 0x7f1e70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8b200640 1 -- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e70004800 msgr2=0x7f1e70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e70004800 0x7f1e70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8b200640 1 -- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700039b0 msgr2=0x7f1e70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700039b0 0x7f1e70003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.861+0000 7f1e8b200640 1 -- 172.21.15.67:0/273449737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e70003520 con 0x7f1e700051c0 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.862+0000 7f1e8b200640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e700051c0 0x7f1e7000b9c0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009f30 tx=0x7f1e70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da63545f74e6b29b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.862+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84032070 con 0x7f1e700051c0 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.862+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e84030070 con 0x7f1e700051c0 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.862+0000 7f1e89400640 1 -- 172.21.15.67:0/273449737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84009b50 con 0x7f1e700051c0 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 -- 172.21.15.67:0/273449737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e700051c0 msgr2=0x7f1e7000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 --2- 172.21.15.67:0/273449737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e700051c0 0x7f1e7000b9c0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009f30 tx=0x7f1e70005830 comp rx=0 tx=0).stop 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e8b200640 1 -- 172.21.15.67:0/273449737 reap_dead start 2024-04-17T08:51:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 -- 172.21.15.67:0/273449737 shutdown_connections 2024-04-17T08:51:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 -- 172.21.15.67:0/273449737 >> 172.21.15.67:0/273449737 conn(0x7f1e8c067ee0 msgr2=0x7f1e8c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 -- 172.21.15.67:0/273449737 shutdown_connections 2024-04-17T08:51:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.863+0000 7f1e91400640 1 -- 172.21.15.67:0/273449737 wait complete. 2024-04-17T08:51:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.865+0000 7f1e91400640 1 Processor -- start 2024-04-17T08:51:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.865+0000 7f1e91400640 1 -- start start 2024-04-17T08:51:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700051c0 0x7f1e8c152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c151b20 0x7f1e8c14aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c15fa60 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c15f8e0 con 0x7f1e700051c0 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.866+0000 7f1e91400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e8c0d0900 con 0x7f1e8c151b20 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36872/0 (socket says 172.21.15.67:36872) 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 -- 172.21.15.67:0/620060240 learned_addr learned my addr 172.21.15.67:0/620060240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e8a800640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700051c0 0x7f1e8c152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e8b200640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c151b20 0x7f1e8c14aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 -- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c151b20 msgr2=0x7f1e8c14aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c151b20 0x7f1e8c14aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 -- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700051c0 msgr2=0x7f1e8c152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700051c0 0x7f1e8c152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.867+0000 7f1e89e00640 1 -- 172.21.15.67:0/620060240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8c0a99f0 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e89e00640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800e7b0 tx=0x7f1e7800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800c870 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e78018070 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800cc70 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8c0a9c90 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.868+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e8c0aa250 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.869+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e8c068ca0 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.870+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f1e7801d460 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.871+0000 7f1e82a00640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 0x7f1e6c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.872+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 6495+0+0 (secure 0 0 0) 0x7f1e7808c160 con 0x7f1e8c0aaa50 2024-04-17T08:51:55.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.872+0000 7f1e8a800640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 0x7f1e6c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:55.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.873+0000 7f1e8a800640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 0x7f1e6c06eff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c006fd0 tx=0x7f1e7c008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:55.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:55.877+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1e78059f50 con 0x7f1e8c0aaa50 2024-04-17T08:51:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.126+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f1e8c0afd70 con 0x7f1e8c0aaa50 2024-04-17T08:51:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.676+0000 7f1e82a00640 1 -- 172.21.15.67:0/620060240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v128) ==== 183+0+0 (secure 0 0 0) 0x7f1e78059850 con 0x7f1e8c0aaa50 2024-04-17T08:51:56.680 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 32 and data pool 33 2024-04-17T08:51:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 msgr2=0x7f1e6c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 0x7f1e6c06eff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c006fd0 tx=0x7f1e7c008040 comp rx=0 tx=0).stop 2024-04-17T08:51:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 msgr2=0x7f1e8c0abf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800e7b0 tx=0x7f1e7800ec70 comp rx=0 tx=0).stop 2024-04-17T08:51:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 shutdown_connections 2024-04-17T08:51:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f1e6c06cb30 0x7f1e6c06eff0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e8c151b20 0x7f1e8c14aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8c0aaa50 0x7f1e8c0abf20 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 --2- 172.21.15.67:0/620060240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e700051c0 0x7f1e8c152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.687+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 >> 172.21.15.67:0/620060240 conn(0x7f1e8c067ee0 msgr2=0x7f1e8c15a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.690+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 shutdown_connections 2024-04-17T08:51:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.691+0000 7f1e91400640 1 -- 172.21.15.67:0/620060240 wait complete. 2024-04-17T08:51:56.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.904+0000 7f2e25e00640 1 Processor -- start 2024-04-17T08:51:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.904+0000 7f2e25e00640 1 -- start start 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e200b3d10 0x7f2e2006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200d09f0 0x7f2e2009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2e2006d700 0x7f2e2009afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2006dc30 con 0x7f2e200b3d10 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2006df10 con 0x7f2e2006d700 2024-04-17T08:51:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e25e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2e2006e1f0 con 0x7f2e200d09f0 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e200b3d10 0x7f2e2006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e200b3d10 0x7f2e2006d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36876/0 (socket says 172.21.15.67:36876) 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 -- 172.21.15.67:0/1274080971 learned_addr learned my addr 172.21.15.67:0/1274080971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 -- 172.21.15.67:0/1274080971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200d09f0 msgr2=0x7f2e2009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/1274080971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200d09f0 0x7f2e2009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 -- 172.21.15.67:0/1274080971 >> v2:172.21.15.187:3300/0 conn(0x7f2e2006d700 msgr2=0x7f2e2009afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/1274080971 >> v2:172.21.15.187:3300/0 conn(0x7f2e2006d700 0x7f2e2009afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.906+0000 7f2e1ea00640 1 -- 172.21.15.67:0/1274080971 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2006e4d0 con 0x7f2e200b3d10 2024-04-17T08:51:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/1274080971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e200b3d10 0x7f2e2006d1c0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f2e2006eb90 tx=0x7f2e1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615c66ea6d6129c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e1d600640 1 -- 172.21.15.67:0/1274080971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10035020 con 0x7f2e200b3d10 2024-04-17T08:51:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e1d600640 1 -- 172.21.15.67:0/1274080971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10031870 con 0x7f2e200b3d10 2024-04-17T08:51:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e1d600640 1 -- 172.21.15.67:0/1274080971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10031c40 con 0x7f2e200b3d10 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e25e00640 1 -- 172.21.15.67:0/1274080971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e200b3d10 msgr2=0x7f2e2006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.908+0000 7f2e25e00640 1 --2- 172.21.15.67:0/1274080971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e200b3d10 0x7f2e2006d1c0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f2e2006eb90 tx=0x7f2e1002b910 comp rx=0 tx=0).stop 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 -- 172.21.15.67:0/1274080971 shutdown_connections 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 --2- 172.21.15.67:0/1274080971 >> v2:172.21.15.187:3300/0 conn(0x7f2e2006d700 0x7f2e2009afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 --2- 172.21.15.67:0/1274080971 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200d09f0 0x7f2e2009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 --2- 172.21.15.67:0/1274080971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e200b3d10 0x7f2e2006d1c0 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 -- 172.21.15.67:0/1274080971 >> 172.21.15.67:0/1274080971 conn(0x7f2e200b41b0 msgr2=0x7f2e200b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 -- 172.21.15.67:0/1274080971 shutdown_connections 2024-04-17T08:51:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.909+0000 7f2e25e00640 1 -- 172.21.15.67:0/1274080971 wait complete. 2024-04-17T08:51:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.910+0000 7f2e25e00640 1 Processor -- start 2024-04-17T08:51:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.910+0000 7f2e25e00640 1 -- start start 2024-04-17T08:51:56.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200b3d10 0x7f2e200c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e200d09f0 0x7f2e200c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e20162a00 con 0x7f2e2006d700 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2e20162880 con 0x7f2e200d09f0 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.911+0000 7f2e25e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2e20162b80 con 0x7f2e200b3d10 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36878/0 (socket says 172.21.15.67:36878) 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 -- 172.21.15.67:0/83671473 learned_addr learned my addr 172.21.15.67:0/83671473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 -- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200b3d10 msgr2=0x7f2e200c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200b3d10 0x7f2e200c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 -- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e200d09f0 msgr2=0x7f2e200c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e200d09f0 0x7f2e200c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 -- 172.21.15.67:0/83671473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2014fd80 con 0x7f2e2006d700 2024-04-17T08:51:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e1ea00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.912+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1003ad60 con 0x7f2e2006d700 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.913+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e100443f0 con 0x7f2e2006d700 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.913+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100447e0 con 0x7f2e2006d700 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.913+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e20150060 con 0x7f2e2006d700 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.913+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e20150620 con 0x7f2e2006d700 2024-04-17T08:51:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.914+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e20068b90 con 0x7f2e2006d700 2024-04-17T08:51:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.915+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f2e1004b060 con 0x7f2e2006d700 2024-04-17T08:51:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.916+0000 7f2e17400640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 0x7f2e0006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.916+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 6601+0+0 (secure 0 0 0) 0x7f2e100ae780 con 0x7f2e2006d700 2024-04-17T08:51:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.916+0000 7f2e1e000640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 0x7f2e0006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.918+0000 7f2e1e000640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 0x7f2e0006f020 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e200b8110 tx=0x7f2e0c006bf0 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:56.920+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2e1003f070 con 0x7f2e2006d700 2024-04-17T08:51:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.127+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e200b3420 con 0x7f2e2006d700 2024-04-17T08:51:57.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.130+0000 7f2e17400640 1 -- 172.21.15.67:0/83671473 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) ==== 74+0+20362 (secure 0 0 0) 0x7f2e1002f320 con 0x7f2e2006d700 2024-04-17T08:51:57.130 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:57.130 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":253,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:56.129048+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.132+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 msgr2=0x7f2e0006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.132+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 0x7f2e0006f020 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e200b8110 tx=0x7f2e0c006bf0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.132+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 msgr2=0x7f2e2014f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.132+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c00 tx=0x7f2e100046c0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 shutdown_connections 2024-04-17T08:51:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f2e0006cb60 0x7f2e0006f020 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e200d09f0 0x7f2e200c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2e200b3d10 0x7f2e200c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 --2- 172.21.15.67:0/83671473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e2006d700 0x7f2e2014f4b0 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 >> 172.21.15.67:0/83671473 conn(0x7f2e200b41b0 msgr2=0x7f2e200b73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 shutdown_connections 2024-04-17T08:51:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.133+0000 7f2e25e00640 1 -- 172.21.15.67:0/83671473 wait complete. 2024-04-17T08:51:57.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:51:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.361+0000 7f6f91a00640 1 Processor -- start 2024-04-17T08:51:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.361+0000 7f6f91a00640 1 -- start start 2024-04-17T08:51:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.362+0000 7f6f91a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0b9f00 0x7f6f8c0b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.362+0000 7f6f91a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.362+0000 7f6f91a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8c163a20 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.362+0000 7f6f91a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8c0cf640 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.362+0000 7f6f91a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8c0b84c0 con 0x7f6f8c0b9f00 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36886/0 (socket says 172.21.15.67:36886) 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 -- 172.21.15.67:0/2970898552 learned_addr learned my addr 172.21.15.67:0/2970898552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8be00640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0b9f00 0x7f6f8c0b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 -- 172.21.15.67:0/2970898552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0b9f00 msgr2=0x7f6f8c0b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0b9f00 0x7f6f8c0b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 -- 172.21.15.67:0/2970898552 >> v1:172.21.15.187:6789/0 conn(0x7f6f8c0d12a0 legacy=0x7f6f8c0d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.363+0000 7f6f8b400640 1 -- 172.21.15.67:0/2970898552 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c0cfe20 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.364+0000 7f6f8b400640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009870 tx=0x7f6f7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d72c4cf1de52d78 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.365+0000 7f6f8aa00640 1 -- 172.21.15.67:0/2970898552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f740317c0 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.365+0000 7f6f8aa00640 1 -- 172.21.15.67:0/2970898552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f74037070 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.365+0000 7f6f8aa00640 1 -- 172.21.15.67:0/2970898552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74031b80 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.365+0000 7f6f91a00640 1 -- 172.21.15.67:0/2970898552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cf8f0 msgr2=0x7f6f8c1634e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.365+0000 7f6f91a00640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009870 tx=0x7f6f7402b860 comp rx=0 tx=0).stop 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 -- 172.21.15.67:0/2970898552 shutdown_connections 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1634e0 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 --2- 172.21.15.67:0/2970898552 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0b9f00 0x7f6f8c0b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 -- 172.21.15.67:0/2970898552 >> 172.21.15.67:0/2970898552 conn(0x7f6f8c06cb60 msgr2=0x7f6f8c06cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:57.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 -- 172.21.15.67:0/2970898552 shutdown_connections 2024-04-17T08:51:57.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.366+0000 7f6f91a00640 1 -- 172.21.15.67:0/2970898552 wait complete. 2024-04-17T08:51:57.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.367+0000 7f6f91a00640 1 Processor -- start 2024-04-17T08:51:57.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.367+0000 7f6f91a00640 1 -- start start 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 0x7f6f8c15f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1ead50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c0d3420 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c0d3140 con 0x7f6f8c0b9f00 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.368+0000 7f6f91a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8c0ba2e0 con 0x7f6f8c0cf8f0 2024-04-17T08:51:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36902/0 (socket says 172.21.15.67:36902) 2024-04-17T08:51:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 -- 172.21.15.67:0/3979676739 learned_addr learned my addr 172.21.15.67:0/3979676739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f8be00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 0x7f6f8c15f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f8b400640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 -- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0cf8f0 msgr2=0x7f6f8c1ead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1ead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 -- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 msgr2=0x7f6f8c15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 0x7f6f8c15f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f91000640 1 -- 172.21.15.67:0/3979676739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c0c9300 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.369+0000 7f6f8be00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 0x7f6f8c15f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f91000640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c410 tx=0x7f6f7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c011790 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c017070 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c011bb0 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8c0c86f0 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.370+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c0c8cb0 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.371+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f8c0755d0 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.373+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f6f7c01c460 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.373+0000 7f6f83e00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 0x7f6f6806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.374+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 6601+0+0 (secure 0 0 0) 0x7f6f7c08be90 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.374+0000 7f6f8be00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 0x7f6f6806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.375+0000 7f6f8be00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 0x7f6f6806f020 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f84007fc0 tx=0x7f6f840075b0 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.378+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6f7c021020 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.589+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f8c162e70 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.593+0000 7f6f83e00640 1 -- 172.21.15.67:0/3979676739 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) ==== 74+0+20362 (secure 0 0 0) 0x7f6f7c08d090 con 0x7f6f8c0d12a0 2024-04-17T08:51:57.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:51:57.593 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":253,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:56.129048+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:51:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.595+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 msgr2=0x7f6f6806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.595+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 0x7f6f6806f020 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f84007fc0 tx=0x7f6f840075b0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.595+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 msgr2=0x7f6f8c1f84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.595+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c00c410 tx=0x7f6f7c00c8d0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 shutdown_connections 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f6f6806cb60 0x7f6f6806f020 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0d12a0 0x7f6f8c1f84a0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6f8c0cf8f0 0x7f6f8c1ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 --2- 172.21.15.67:0/3979676739 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6f8c0b9f00 0x7f6f8c15f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 >> 172.21.15.67:0/3979676739 conn(0x7f6f8c06cb60 msgr2=0x7f6f8c0b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 shutdown_connections 2024-04-17T08:51:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.596+0000 7f6f91a00640 1 -- 172.21.15.67:0/3979676739 wait complete. 2024-04-17T08:51:57.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-04-17T08:51:57.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:51:57.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:51:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.838+0000 7fa2c8a00640 1 Processor -- start 2024-04-17T08:51:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.838+0000 7fa2c8a00640 1 -- start start 2024-04-17T08:51:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2c8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2c406b8c0 0x7fa2c406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2c8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2c4163710 con 0x7fa2c40d0b50 2024-04-17T08:51:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2c8a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2c415f360 con 0x7fa2c41627b0 2024-04-17T08:51:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2c8a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa2c406c1e0 con 0x7fa2c406b8c0 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2c406b8c0 0x7fa2c406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2bbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2c40d0b50 0x7fa2c415fb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33350/0 (socket says 172.21.15.67:33350) 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.840+0000 7fa2bbe00640 1 -- 172.21.15.67:0/1895019346 learned_addr learned my addr 172.21.15.67:0/1895019346 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.841+0000 7fa2c1e00640 1 -- 172.21.15.67:0/1895019346 >> v1:172.21.15.187:6789/0 conn(0x7fa2c41627b0 legacy=0x7fa2c40cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.841+0000 7fa2c1e00640 1 -- 172.21.15.67:0/1895019346 >> v1:172.21.15.67:6789/0 conn(0x7fa2c40d0b50 legacy=0x7fa2c415fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.841+0000 7fa2c1400640 1 -- 172.21.15.67:0/1895019346 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320865394 0 0) 0x7fa2c4163710 con 0x7fa2c40d0b50 2024-04-17T08:51:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.841+0000 7fa2c1e00640 1 -- 172.21.15.67:0/1895019346 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c4163150 con 0x7fa2c406b8c0 2024-04-17T08:51:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.842+0000 7fa2c1e00640 1 --2- 172.21.15.67:0/1895019346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c406b8c0 0x7fa2c406bca0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40097a0 tx=0x7fa2b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5570056105238f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.842+0000 7fa2c1400640 1 -- 172.21.15.67:0/1895019346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4034070 con 0x7fa2c406b8c0 2024-04-17T08:51:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.842+0000 7fa2c1400640 1 -- 172.21.15.67:0/1895019346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b4030610 con 0x7fa2c406b8c0 2024-04-17T08:51:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.842+0000 7fa2c1400640 1 -- 172.21.15.67:0/1895019346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b40309e0 con 0x7fa2c406b8c0 2024-04-17T08:51:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 -- 172.21.15.67:0/1895019346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c406b8c0 msgr2=0x7fa2c406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/1895019346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c406b8c0 0x7fa2c406bca0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40097a0 tx=0x7fa2b402b960 comp rx=0 tx=0).stop 2024-04-17T08:51:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 -- 172.21.15.67:0/1895019346 shutdown_connections 2024-04-17T08:51:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/1895019346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c406b8c0 0x7fa2c406bca0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 -- 172.21.15.67:0/1895019346 >> 172.21.15.67:0/1895019346 conn(0x7fa2c4073360 msgr2=0x7fa2c409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 -- 172.21.15.67:0/1895019346 shutdown_connections 2024-04-17T08:51:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.843+0000 7fa2c8a00640 1 -- 172.21.15.67:0/1895019346 wait complete. 2024-04-17T08:51:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.845+0000 7fa2c8a00640 1 Processor -- start 2024-04-17T08:51:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.845+0000 7fa2c8a00640 1 -- start start 2024-04-17T08:51:57.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2c406b8c0 0x7fa2c4153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2c40d0b50 0x7fa2c41f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c415eef0 con 0x7fa2c41627b0 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c415f600 con 0x7fa2c40d0b50 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c8a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2c415f070 con 0x7fa2c406b8c0 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36918/0 (socket says 172.21.15.67:36918) 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c2800640 1 -- 172.21.15.67:0/2027434865 learned_addr learned my addr 172.21.15.67:0/2027434865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2c1e00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2c406b8c0 0x7fa2c4153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.846+0000 7fa2bbe00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2c40d0b50 0x7fa2c41f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 -- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2c406b8c0 msgr2=0x7fa2c4153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2c406b8c0 0x7fa2c4153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 -- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2c40d0b50 msgr2=0x7fa2c41f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2c40d0b50 0x7fa2c41f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 -- 172.21.15.67:0/2027434865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c40c7a90 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.847+0000 7fa2c2800640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c6e0 tx=0x7fa2bc00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.848+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc003b70 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.848+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2bc003d30 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.848+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc005310 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.848+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c40c7d70 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.848+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c41f4c50 con 0x7fa2c41627b0 2024-04-17T08:51:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.849+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c4068ca0 con 0x7fa2c41627b0 2024-04-17T08:51:57.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.850+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fa2bc0054b0 con 0x7fa2c41627b0 2024-04-17T08:51:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.851+0000 7fa2ba000640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 0x7fa2a006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.851+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fa2bc08c0c0 con 0x7fa2c41627b0 2024-04-17T08:51:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.851+0000 7fa2c1e00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 0x7fa2a006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.853+0000 7fa2c1e00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 0x7fa2a006eff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4036000 tx=0x7fa2b403b040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:57.855+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa2bc059720 con 0x7fa2c41627b0 2024-04-17T08:51:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.105+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa2c40b07a0 con 0x7fa2c41627b0 2024-04-17T08:51:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.698+0000 7fa2ba000640 1 -- 172.21.15.67:0/2027434865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v131) ==== 121+0+0 (secure 0 0 0) 0x7fa2bc05da20 con 0x7fa2c41627b0 2024-04-17T08:51:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 msgr2=0x7fa2a006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 0x7fa2a006eff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4036000 tx=0x7fa2b403b040 comp rx=0 tx=0).stop 2024-04-17T08:51:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 msgr2=0x7fa2c4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c6e0 tx=0x7fa2bc00cba0 comp rx=0 tx=0).stop 2024-04-17T08:51:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 shutdown_connections 2024-04-17T08:51:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fa2a006cb30 0x7fa2a006eff0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c41627b0 0x7fa2c4156430 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2c40d0b50 0x7fa2c41f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:58.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 --2- 172.21.15.67:0/2027434865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2c406b8c0 0x7fa2c4153ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:58.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.702+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 >> 172.21.15.67:0/2027434865 conn(0x7fa2c4073360 msgr2=0x7fa2c40ae800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:58.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.703+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 shutdown_connections 2024-04-17T08:51:58.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.703+0000 7fa2c8a00640 1 -- 172.21.15.67:0/2027434865 wait complete. 2024-04-17T08:51:58.725 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:51:58.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-04-17T08:51:58.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.998+0000 7fdc01600640 1 Processor -- start 2024-04-17T08:51:58.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:58.999+0000 7fdc01600640 1 -- start start 2024-04-17T08:51:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc01600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdbfc0d33e0 0x7fdbfc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc01600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc01600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbfc0d00b0 con 0x7fdbfc15ecb0 2024-04-17T08:51:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc01600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdbfc0d0c30 con 0x7fdbfc0cf8e0 2024-04-17T08:51:58.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc01600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdbfc0d0390 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc00c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc00c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36932/0 (socket says 172.21.15.67:36932) 2024-04-17T08:51:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.000+0000 7fdc00c00640 1 -- 172.21.15.67:0/2460204131 learned_addr learned my addr 172.21.15.67:0/2460204131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:59.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.001+0000 7fdbfb400640 1 --2- 172.21.15.67:0/2460204131 >> v2:172.21.15.187:3300/0 conn(0x7fdbfc0d33e0 0x7fdbfc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.001+0000 7fdc00c00640 1 -- 172.21.15.67:0/2460204131 >> v2:172.21.15.187:3300/0 conn(0x7fdbfc0d33e0 msgr2=0x7fdbfc0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.001+0000 7fdc00c00640 1 --2- 172.21.15.67:0/2460204131 >> v2:172.21.15.187:3300/0 conn(0x7fdbfc0d33e0 0x7fdbfc0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.001+0000 7fdc00c00640 1 -- 172.21.15.67:0/2460204131 >> v1:172.21.15.187:6789/0 conn(0x7fdbfc15ecb0 legacy=0x7fdbfc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.001+0000 7fdc00c00640 1 -- 172.21.15.67:0/2460204131 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc0d0670 con 0x7fdbfc0cf8e0 2024-04-17T08:51:59.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.002+0000 7fdc00c00640 1 --2- 172.21.15.67:0/2460204131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0b61a0 tx=0x7fdbf402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3805f557be45b034 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.002+0000 7fdbfaa00640 1 -- 172.21.15.67:0/2460204131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf402e070 con 0x7fdbfc0cf8e0 2024-04-17T08:51:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdbfaa00640 1 -- 172.21.15.67:0/2460204131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf4037070 con 0x7fdbfc0cf8e0 2024-04-17T08:51:59.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdbfaa00640 1 -- 172.21.15.67:0/2460204131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4031a00 con 0x7fdbfc0cf8e0 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 -- 172.21.15.67:0/2460204131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0cf8e0 msgr2=0x7fdbfc0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 --2- 172.21.15.67:0/2460204131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0b61a0 tx=0x7fdbf402b880 comp rx=0 tx=0).stop 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 -- 172.21.15.67:0/2460204131 shutdown_connections 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 --2- 172.21.15.67:0/2460204131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc0b3fc0 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 --2- 172.21.15.67:0/2460204131 >> v2:172.21.15.187:3300/0 conn(0x7fdbfc0d33e0 0x7fdbfc0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 -- 172.21.15.67:0/2460204131 >> 172.21.15.67:0/2460204131 conn(0x7fdbfc08bff0 msgr2=0x7fdbfc0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 -- 172.21.15.67:0/2460204131 shutdown_connections 2024-04-17T08:51:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.003+0000 7fdc01600640 1 -- 172.21.15.67:0/2460204131 wait complete. 2024-04-17T08:51:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.005+0000 7fdc01600640 1 Processor -- start 2024-04-17T08:51:59.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.005+0000 7fdc01600640 1 -- start start 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc1f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc15ecb0 0x7fdbfc1f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc15f090 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc0cfd30 con 0x7fdbfc0cf8e0 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdc01600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdbfc15f210 con 0x7fdbfc15ecb0 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.006+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36936/0 (socket says 172.21.15.67:36936) 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 -- 172.21.15.67:0/3347059617 learned_addr learned my addr 172.21.15.67:0/3347059617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfbe00640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc1f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdc00c00640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc15ecb0 0x7fdbfc1f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 -- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc15ecb0 msgr2=0x7fdbfc1f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc15ecb0 0x7fdbfc1f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 -- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0cf8e0 msgr2=0x7fdbfc1f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc1f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 -- 172.21.15.67:0/3347059617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc0b8bf0 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.007+0000 7fdbfb400640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400d800 tx=0x7fdbe400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.008+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400b6c0 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.008+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe4019070 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.008+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400ba80 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.008+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc0b8ed0 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.008+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc0cce70 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.009+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbfc068a10 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.010+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fdbe400bc20 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.011+0000 7fdbf3e00640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 0x7fdbd806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.012+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fdbe408c800 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.012+0000 7fdbfbe00640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 0x7fdbd806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.013+0000 7fdbfbe00640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 0x7fdbd806eff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009f20 tx=0x7fdbec0094d0 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.016+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdbe405a550 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.264+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdbfc0d1ba0 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.709+0000 7fdbf3e00640 1 -- 172.21.15.67:0/3347059617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v132) ==== 125+0+0 (secure 0 0 0) 0x7fdbe4059e50 con 0x7fdbfc0d33e0 2024-04-17T08:51:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 msgr2=0x7fdbd806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 0x7fdbd806eff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009f20 tx=0x7fdbec0094d0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 msgr2=0x7fdbfc0b8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400d800 tx=0x7fdbe400dcc0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 shutdown_connections 2024-04-17T08:51:59.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fdbd806cb30 0x7fdbd806eff0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbfc15ecb0 0x7fdbfc1f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc0d33e0 0x7fdbfc0b8450 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 --2- 172.21.15.67:0/3347059617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbfc0cf8e0 0x7fdbfc1f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.714+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 >> 172.21.15.67:0/3347059617 conn(0x7fdbfc08bff0 msgr2=0x7fdbfc0d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.715+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 shutdown_connections 2024-04-17T08:51:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.715+0000 7fdc01600640 1 -- 172.21.15.67:0/3347059617 wait complete. 2024-04-17T08:51:59.739 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-04-17T08:51:59.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:51:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.967+0000 7fba43e00640 1 Processor -- start 2024-04-17T08:51:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.967+0000 7fba43e00640 1 -- start start 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.968+0000 7fba43e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba4406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.968+0000 7fba43e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba4406fce0 0x7fba4406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.968+0000 7fba43e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4415bc80 con 0x7fba4406be70 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.968+0000 7fba43e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fba4415bf40 con 0x7fba4406fce0 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.968+0000 7fba43e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba4415c220 con 0x7fba4415b570 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba43400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba4406be70 0x7fba4406ddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33364/0 (socket says 172.21.15.67:33364) 2024-04-17T08:51:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba43400640 1 -- 172.21.15.67:0/1327182324 learned_addr learned my addr 172.21.15.67:0/1327182324 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba42a00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba4406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba3be00640 1 --2- 172.21.15.67:0/1327182324 >> v2:172.21.15.187:3300/0 conn(0x7fba4406fce0 0x7fba4406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830973162 0 0) 0x7fba4415bc80 con 0x7fba4406be70 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.969+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba24003520 con 0x7fba4406be70 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1020890982 0 0) 0x7fba24003520 con 0x7fba4406be70 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 msgr2=0x7fba4406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba4406f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 >> v2:172.21.15.187:3300/0 conn(0x7fba4406fce0 msgr2=0x7fba4406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 --2- 172.21.15.67:0/1327182324 >> v2:172.21.15.187:3300/0 conn(0x7fba4406fce0 0x7fba4406b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4415c500 con 0x7fba4406be70 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fba3c0019f0 con 0x7fba4406be70 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 >> v1:172.21.15.67:6789/0 conn(0x7fba4406be70 legacy=0x7fba4406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba24003a20 0x7fba24003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24004d40 0x7fba240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba4415bf40 con 0x7fba24003a20 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba4415c220 con 0x7fba24004d40 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba2400dc80 con 0x7fba4415b570 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.970+0000 7fba42a00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba24003a20 0x7fba24003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba43400640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24004d40 0x7fba240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba3be00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba42a00640 1 -- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 msgr2=0x7fba24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba42a00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba4415b570 0x7fba24004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba42a00640 1 -- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24004d40 msgr2=0x7fba240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:51:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba42a00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24004d40 0x7fba240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba42a00640 1 -- 172.21.15.67:0/1327182324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba24003520 con 0x7fba24003a20 2024-04-17T08:51:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.971+0000 7fba43400640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24004d40 0x7fba240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.972+0000 7fba42a00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba24003a20 0x7fba24003e40 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fba34009a90 tx=0x7fba24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f068f93336017be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.972+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34039070 con 0x7fba24003a20 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.972+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba3402c680 con 0x7fba24003a20 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.972+0000 7fba42000640 1 -- 172.21.15.67:0/1327182324 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34009690 con 0x7fba24003a20 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 -- 172.21.15.67:0/1327182324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba24003a20 msgr2=0x7fba24003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 --2- 172.21.15.67:0/1327182324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba24003a20 0x7fba24003e40 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fba34009a90 tx=0x7fba24005830 comp rx=0 tx=0).stop 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43400640 1 -- 172.21.15.67:0/1327182324 reap_dead start 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 -- 172.21.15.67:0/1327182324 shutdown_connections 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 -- 172.21.15.67:0/1327182324 >> 172.21.15.67:0/1327182324 conn(0x7fba44067d10 msgr2=0x7fba440b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:51:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 -- 172.21.15.67:0/1327182324 shutdown_connections 2024-04-17T08:51:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.973+0000 7fba43e00640 1 -- 172.21.15.67:0/1327182324 wait complete. 2024-04-17T08:51:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.975+0000 7fba43e00640 1 Processor -- start 2024-04-17T08:51:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.975+0000 7fba43e00640 1 -- start start 2024-04-17T08:51:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 0x7fba4414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 0x7fba4414df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba4406c460 con 0x7fba4414ea00 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba4406c2e0 con 0x7fba24003a20 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba440b47b0 con 0x7fba44150010 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36942/0 (socket says 172.21.15.67:36942) 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43400640 1 -- 172.21.15.67:0/2680780557 learned_addr learned my addr 172.21.15.67:0/2680780557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba42a00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 0x7fba4414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba3be00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 0x7fba4414df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.976+0000 7fba43400640 1 -- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 msgr2=0x7fba4414df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43400640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 0x7fba4414df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43400640 1 -- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 msgr2=0x7fba4414bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43400640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 0x7fba4414bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43400640 1 -- 172.21.15.67:0/2680780557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4414ef70 con 0x7fba4414ea00 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba42a00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 0x7fba4414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba3be00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 0x7fba4414df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:51:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43400640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fba440cdf30 tx=0x7fba3c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c004d60 con 0x7fba4414ea00 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba3c00bb20 con 0x7fba4414ea00 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c007070 con 0x7fba4414ea00 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4414f250 con 0x7fba4414ea00 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.977+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4414f810 con 0x7fba4414ea00 2024-04-17T08:51:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.979+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba44068a10 con 0x7fba4414ea00 2024-04-17T08:51:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.980+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fba3c01b020 con 0x7fba4414ea00 2024-04-17T08:51:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.981+0000 7fba3aa00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 0x7fba2006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:51:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.981+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fba3c08c5d0 con 0x7fba4414ea00 2024-04-17T08:51:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.981+0000 7fba42a00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 0x7fba2006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:51:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.983+0000 7fba42a00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 0x7fba2006eff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fba34036000 tx=0x7fba34030000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:51:59.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:51:59.986+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba3c05a170 con 0x7fba4414ea00 2024-04-17T08:52:00.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.231+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba440cfd50 con 0x7fba4414ea00 2024-04-17T08:52:00.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.232+0000 7fba3aa00640 1 -- 172.21.15.67:0/2680780557 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7fba3c059a70 con 0x7fba4414ea00 2024-04-17T08:52:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:00.232 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:00.232 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 msgr2=0x7fba2006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 0x7fba2006eff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fba34036000 tx=0x7fba34030000 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 msgr2=0x7fba44154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fba440cdf30 tx=0x7fba3c00b4a0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 shutdown_connections 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fba2006cb30 0x7fba2006eff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4414ea00 0x7fba44154c00 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba44150010 0x7fba4414df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 --2- 172.21.15.67:0/2680780557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba24003a20 0x7fba4414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 >> 172.21.15.67:0/2680780557 conn(0x7fba44067d10 msgr2=0x7fba4414c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.235+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 shutdown_connections 2024-04-17T08:52:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.236+0000 7fba43e00640 1 -- 172.21.15.67:0/2680780557 wait complete. 2024-04-17T08:52:00.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.479+0000 7fe618a00640 1 Processor -- start 2024-04-17T08:52:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.480+0000 7fe618a00640 1 -- start start 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.481+0000 7fe618a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe61415b570 0x7fe61406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.481+0000 7fe618a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61415bc80 con 0x7fe61406c040 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe618a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61415bf40 con 0x7fe61406fce0 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe618a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe61406c470 con 0x7fe61415b570 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe61415b570 0x7fe61406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe61415b570 0x7fe61406f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36952/0 (socket says 172.21.15.67:36952) 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 -- 172.21.15.67:0/3909495373 learned_addr learned my addr 172.21.15.67:0/3909495373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe613200640 1 --1- 172.21.15.67:0/3909495373 >> v1:172.21.15.67:6789/0 conn(0x7fe61406c040 0x7fe61415f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33366/0 (socket says 172.21.15.67:33366) 2024-04-17T08:52:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 -- 172.21.15.67:0/3909495373 >> v1:172.21.15.187:6789/0 conn(0x7fe61406fce0 legacy=0x7fe61406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 -- 172.21.15.67:0/3909495373 >> v1:172.21.15.67:6789/0 conn(0x7fe61406c040 legacy=0x7fe61415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe612800640 1 -- 172.21.15.67:0/3909495373 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61415c220 con 0x7fe61415b570 2024-04-17T08:52:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.482+0000 7fe611e00640 1 -- 172.21.15.67:0/3909495373 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455817990 0 0) 0x7fe61415bc80 con 0x7fe61406c040 2024-04-17T08:52:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe612800640 1 --2- 172.21.15.67:0/3909495373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61415b570 0x7fe61406f7a0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fe61406c5f0 tx=0x7fe60402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebfab1117f3e65a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe611e00640 1 -- 172.21.15.67:0/3909495373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604034070 con 0x7fe61415b570 2024-04-17T08:52:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe611e00640 1 -- 172.21.15.67:0/3909495373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe604030840 con 0x7fe61415b570 2024-04-17T08:52:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe611e00640 1 -- 172.21.15.67:0/3909495373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604030c10 con 0x7fe61415b570 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe618a00640 1 -- 172.21.15.67:0/3909495373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61415b570 msgr2=0x7fe61406f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.484+0000 7fe618a00640 1 --2- 172.21.15.67:0/3909495373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61415b570 0x7fe61406f7a0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fe61406c5f0 tx=0x7fe60402b980 comp rx=0 tx=0).stop 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.485+0000 7fe618a00640 1 -- 172.21.15.67:0/3909495373 shutdown_connections 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.485+0000 7fe618a00640 1 --2- 172.21.15.67:0/3909495373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61415b570 0x7fe61406f7a0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.485+0000 7fe618a00640 1 -- 172.21.15.67:0/3909495373 >> 172.21.15.67:0/3909495373 conn(0x7fe614067d10 msgr2=0x7fe6140b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.485+0000 7fe618a00640 1 -- 172.21.15.67:0/3909495373 shutdown_connections 2024-04-17T08:52:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.485+0000 7fe618a00640 1 -- 172.21.15.67:0/3909495373 wait complete. 2024-04-17T08:52:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.487+0000 7fe618a00640 1 Processor -- start 2024-04-17T08:52:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.487+0000 7fe618a00640 1 -- start start 2024-04-17T08:52:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe61406fce0 0x7fe614149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe61415b570 0x7fe614157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6140b47b0 con 0x7fe61406c040 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe61415b950 con 0x7fe61415b570 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe618a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe6140b4930 con 0x7fe61406fce0 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe612800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe612800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36954/0 (socket says 172.21.15.67:36954) 2024-04-17T08:52:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.488+0000 7fe612800640 1 -- 172.21.15.67:0/52811497 learned_addr learned my addr 172.21.15.67:0/52811497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe60be00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe61406fce0 0x7fe614149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe613200640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe61415b570 0x7fe614157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 -- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe61406fce0 msgr2=0x7fe614149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe61406fce0 0x7fe614149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 -- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe61415b570 msgr2=0x7fe614157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe61415b570 0x7fe614157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 -- 172.21.15.67:0/52811497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe614153470 con 0x7fe61406c040 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe612800640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7fe6040090b0 tx=0x7fe6040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.489+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604034040 con 0x7fe61406c040 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.490+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60403ab10 con 0x7fe61406c040 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.490+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604004030 con 0x7fe61406c040 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.490+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61414a3c0 con 0x7fe61406c040 2024-04-17T08:52:00.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.490+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6141539a0 con 0x7fe61406c040 2024-04-17T08:52:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.491+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6140cdcf0 con 0x7fe61406c040 2024-04-17T08:52:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.492+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe604037070 con 0x7fe61406c040 2024-04-17T08:52:00.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.493+0000 7fe60aa00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 0x7fe5f006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.493+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fe6040ae450 con 0x7fe61406c040 2024-04-17T08:52:00.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.493+0000 7fe60be00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 0x7fe5f006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.494+0000 7fe60be00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 0x7fe5f006eff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6140afde0 tx=0x7fe5fc009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.498+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe6140cdcf0 con 0x7fe61406c040 2024-04-17T08:52:00.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.704+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe614068a10 con 0x7fe61406c040 2024-04-17T08:52:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.706+0000 7fe60aa00640 1 -- 172.21.15.67:0/52811497 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) ==== 74+0+20349 (secure 0 0 0) 0x7fe60403f070 con 0x7fe61406c040 2024-04-17T08:52:00.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:00.706 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":254,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:51:57.673948+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 msgr2=0x7fe5f006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 0x7fe5f006eff0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6140afde0 tx=0x7fe5fc009040 comp rx=0 tx=0).stop 2024-04-17T08:52:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 msgr2=0x7fe614159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7fe6040090b0 tx=0x7fe6040046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 shutdown_connections 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe5f006cb30 0x7fe5f006eff0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe61415b570 0x7fe614157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe61406fce0 0x7fe614149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 --2- 172.21.15.67:0/52811497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61406c040 0x7fe614159cf0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 >> 172.21.15.67:0/52811497 conn(0x7fe614067d10 msgr2=0x7fe61406e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 shutdown_connections 2024-04-17T08:52:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.709+0000 7fe618a00640 1 -- 172.21.15.67:0/52811497 wait complete. 2024-04-17T08:52:00.728 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:52:00.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:52:00.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.947+0000 7f1652600640 1 Processor -- start 2024-04-17T08:52:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.947+0000 7f1652600640 1 -- start start 2024-04-17T08:52:00.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.948+0000 7f1652600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.948+0000 7f1652600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f164c06e020 0x7f164c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.948+0000 7f1652600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164c0ceb00 con 0x7f164c06f670 2024-04-17T08:52:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.948+0000 7f1652600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f164c0cede0 con 0x7f164c06e020 2024-04-17T08:52:00.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.948+0000 7f1652600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f164c0cf0c0 con 0x7f164c0cd020 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f164c06e020 0x7f164c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f164c06e020 0x7f164c06e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36956/0 (socket says 172.21.15.67:36956) 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 -- 172.21.15.67:0/3591492705 learned_addr learned my addr 172.21.15.67:0/3591492705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164aa00640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 -- 172.21.15.67:0/3591492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 msgr2=0x7f164c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 -- 172.21.15.67:0/3591492705 >> v1:172.21.15.187:6789/0 conn(0x7f164c06f670 legacy=0x7f164c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:00.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.949+0000 7f164be00640 1 -- 172.21.15.67:0/3591492705 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164c0cf3a0 con 0x7f164c06e020 2024-04-17T08:52:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f164be00640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c06e4d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f1644009870 tx=0x7f164402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff826685162bcb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f164a000640 1 -- 172.21.15.67:0/3591492705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644034070 con 0x7f164c06e020 2024-04-17T08:52:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f164a000640 1 -- 172.21.15.67:0/3591492705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1644030840 con 0x7f164c06e020 2024-04-17T08:52:00.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f164a000640 1 -- 172.21.15.67:0/3591492705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1644030c10 con 0x7f164c06e020 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f1652600640 1 -- 172.21.15.67:0/3591492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 msgr2=0x7f164c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.951+0000 7f1652600640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c06e4d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f1644009870 tx=0x7f164402b980 comp rx=0 tx=0).stop 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 -- 172.21.15.67:0/3591492705 shutdown_connections 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c06e4d0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 --2- 172.21.15.67:0/3591492705 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 -- 172.21.15.67:0/3591492705 >> 172.21.15.67:0/3591492705 conn(0x7f164c06bf50 msgr2=0x7f164c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 -- 172.21.15.67:0/3591492705 shutdown_connections 2024-04-17T08:52:00.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.952+0000 7f1652600640 1 -- 172.21.15.67:0/3591492705 wait complete. 2024-04-17T08:52:00.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.953+0000 7f1652600640 1 Processor -- start 2024-04-17T08:52:00.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 -- start start 2024-04-17T08:52:00.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f164c06f670 0x7f164c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f164c0ce8a0 con 0x7f164c06e020 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f164c0ce720 con 0x7f164c06f670 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.954+0000 7f1652600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f164c15cb00 con 0x7f164c0cd020 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36966/0 (socket says 172.21.15.67:36966) 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 -- 172.21.15.67:0/4271354622 learned_addr learned my addr 172.21.15.67:0/4271354622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164aa00640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f164c06f670 0x7f164c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164be00640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 -- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 msgr2=0x7f164c0c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 -- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f164c06f670 msgr2=0x7f164c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f164c06f670 0x7f164c0c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 -- 172.21.15.67:0/4271354622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164c0c85b0 con 0x7f164c06e020 2024-04-17T08:52:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.955+0000 7f164b400640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c450 tx=0x7f163c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.956+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c0052d0 con 0x7f164c06e020 2024-04-17T08:52:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.956+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163c005470 con 0x7f164c06e020 2024-04-17T08:52:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.956+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c012870 con 0x7f164c06e020 2024-04-17T08:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.956+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f164c0c7930 con 0x7f164c06e020 2024-04-17T08:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.956+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164c0c81d0 con 0x7f164c06e020 2024-04-17T08:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.957+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f164c068b90 con 0x7f164c06e020 2024-04-17T08:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.958+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f163c012a10 con 0x7f164c06e020 2024-04-17T08:52:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.959+0000 7f1643400640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 0x7f162806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.959+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f163c08bfd0 con 0x7f164c06e020 2024-04-17T08:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.960+0000 7f164aa00640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 0x7f162806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.961+0000 7f164aa00640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 0x7f162806eff0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f164c156dd0 tx=0x7f163400b040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:00.963+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f163c059c80 con 0x7f164c06e020 2024-04-17T08:52:01.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.164+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f164c0c7d30 con 0x7f162806cb30 2024-04-17T08:52:01.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.205+0000 7f1643400640 1 -- 172.21.15.67:0/4271354622 <== mgr.8926 v2:172.21.15.187:6832/2123602634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491461 (secure 0 0 0) 0x7f1634009b20 con 0x7f162806cb30 2024-04-17T08:52:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:01.214 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.216+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 msgr2=0x7f162806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.216+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 0x7f162806eff0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f164c156dd0 tx=0x7f163400b040 comp rx=0 tx=0).stop 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.216+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 msgr2=0x7f164c14fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.216+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c450 tx=0x7f163c00c910 comp rx=0 tx=0).stop 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 shutdown_connections 2024-04-17T08:52:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f162806cb30 0x7f162806eff0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f164c0cd020 0x7f164c0c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f164c06f670 0x7f164c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 --2- 172.21.15.67:0/4271354622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f164c06e020 0x7f164c14fb30 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 >> 172.21.15.67:0/4271354622 conn(0x7f164c06bf50 msgr2=0x7f164c071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 shutdown_connections 2024-04-17T08:52:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.217+0000 7f1652600640 1 -- 172.21.15.67:0/4271354622 wait complete. 2024-04-17T08:52:01.243 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-04-17T08:52:00.779953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":719,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444948,"kb_used_data":26180,"kb_used_omap":12,"kb_used_meta":418611,"kb_avail":754529772,"statfs":{"total":773094113280,"available":772638486528,"internally_reserved":0,"allocated":26808320,"data_stored":13739945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428658224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002883"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691213+0000","last_change":"2024-04-17T08:51:56.690582+0000","last_active":"2024-04-17T08:51:57.691213+0000","last_peered":"2024-04-17T08:51:57.691213+0000","last_clean":"2024-04-17T08:51:57.691213+0000","last_became_active":"2024-04-17T08:51:56.685200+0000","last_became_peered":"2024-04-17T08:51:56.685200+0000","last_unstale":"2024-04-17T08:51:57.691213+0000","last_undegraded":"2024-04-17T08:51:57.691213+0000","last_fullsized":"2024-04-17T08:51:57.691213+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:55.532817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948905+0000","last_change":"2024-04-17T08:51:55.657713+0000","last_active":"2024-04-17T08:51:57.948905+0000","last_peered":"2024-04-17T08:51:57.948905+0000","last_clean":"2024-04-17T08:51:57.948905+0000","last_became_active":"2024-04-17T08:51:55.656393+0000","last_became_peered":"2024-04-17T08:51:55.656393+0000","last_unstale":"2024-04-17T08:51:57.948905+0000","last_undegraded":"2024-04-17T08:51:57.948905+0000","last_fullsized":"2024-04-17T08:51:57.948905+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:29.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667483+0000","last_change":"2024-04-17T08:51:41.515286+0000","last_active":"2024-04-17T08:51:56.667483+0000","last_peered":"2024-04-17T08:51:56.667483+0000","last_clean":"2024-04-17T08:51:56.667483+0000","last_became_active":"2024-04-17T08:51:41.515011+0000","last_became_peered":"2024-04-17T08:51:41.515011+0000","last_unstale":"2024-04-17T08:51:56.667483+0000","last_undegraded":"2024-04-17T08:51:56.667483+0000","last_fullsized":"2024-04-17T08:51:56.667483+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:27.187699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706752+0000","last_change":"2024-04-17T08:51:56.696931+0000","last_active":"2024-04-17T08:51:57.706752+0000","last_peered":"2024-04-17T08:51:57.706752+0000","last_clean":"2024-04-17T08:51:57.706752+0000","last_became_active":"2024-04-17T08:51:56.695570+0000","last_became_peered":"2024-04-17T08:51:56.695570+0000","last_unstale":"2024-04-17T08:51:57.706752+0000","last_undegraded":"2024-04-17T08:51:57.706752+0000","last_fullsized":"2024-04-17T08:51:57.706752+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:55:06.177080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706894+0000","last_change":"2024-04-17T08:51:55.666054+0000","last_active":"2024-04-17T08:51:57.706894+0000","last_peered":"2024-04-17T08:51:57.706894+0000","last_clean":"2024-04-17T08:51:57.706894+0000","last_became_active":"2024-04-17T08:51:55.665763+0000","last_became_peered":"2024-04-17T08:51:55.665763+0000","last_unstale":"2024-04-17T08:51:57.706894+0000","last_undegraded":"2024-04-17T08:51:57.706894+0000","last_fullsized":"2024-04-17T08:51:57.706894+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:30.014342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.669531+0000","last_change":"2024-04-17T08:51:41.505707+0000","last_active":"2024-04-17T08:51:56.669531+0000","last_peered":"2024-04-17T08:51:56.669531+0000","last_clean":"2024-04-17T08:51:56.669531+0000","last_became_active":"2024-04-17T08:51:41.505278+0000","last_became_peered":"2024-04-17T08:51:41.505278+0000","last_unstale":"2024-04-17T08:51:56.669531+0000","last_undegraded":"2024-04-17T08:51:56.669531+0000","last_fullsized":"2024-04-17T08:51:56.669531+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:17.019190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706947+0000","last_change":"2024-04-17T08:51:56.675494+0000","last_active":"2024-04-17T08:51:57.706947+0000","last_peered":"2024-04-17T08:51:57.706947+0000","last_clean":"2024-04-17T08:51:57.706947+0000","last_became_active":"2024-04-17T08:51:56.674933+0000","last_became_peered":"2024-04-17T08:51:56.674933+0000","last_unstale":"2024-04-17T08:51:57.706947+0000","last_undegraded":"2024-04-17T08:51:57.706947+0000","last_fullsized":"2024-04-17T08:51:57.706947+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:47:35.987183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688765+0000","last_change":"2024-04-17T08:51:55.662864+0000","last_active":"2024-04-17T08:51:57.688765+0000","last_peered":"2024-04-17T08:51:57.688765+0000","last_clean":"2024-04-17T08:51:57.688765+0000","last_became_active":"2024-04-17T08:51:55.662596+0000","last_became_peered":"2024-04-17T08:51:55.662596+0000","last_unstale":"2024-04-17T08:51:57.688765+0000","last_undegraded":"2024-04-17T08:51:57.688765+0000","last_fullsized":"2024-04-17T08:51:57.688765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:21.284333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950211+0000","last_change":"2024-04-17T08:51:41.506466+0000","last_active":"2024-04-17T08:51:57.950211+0000","last_peered":"2024-04-17T08:51:57.950211+0000","last_clean":"2024-04-17T08:51:57.950211+0000","last_became_active":"2024-04-17T08:51:41.506181+0000","last_became_peered":"2024-04-17T08:51:41.506181+0000","last_unstale":"2024-04-17T08:51:57.950211+0000","last_undegraded":"2024-04-17T08:51:57.950211+0000","last_fullsized":"2024-04-17T08:51:57.950211+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:41.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707235+0000","last_change":"2024-04-17T08:51:56.706215+0000","last_active":"2024-04-17T08:51:57.707235+0000","last_peered":"2024-04-17T08:51:57.707235+0000","last_clean":"2024-04-17T08:51:57.707235+0000","last_became_active":"2024-04-17T08:51:56.705962+0000","last_became_peered":"2024-04-17T08:51:56.705962+0000","last_unstale":"2024-04-17T08:51:57.707235+0000","last_undegraded":"2024-04-17T08:51:57.707235+0000","last_fullsized":"2024-04-17T08:51:57.707235+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:58.280367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688965+0000","last_change":"2024-04-17T08:51:55.658166+0000","last_active":"2024-04-17T08:51:57.688965+0000","last_peered":"2024-04-17T08:51:57.688965+0000","last_clean":"2024-04-17T08:51:57.688965+0000","last_became_active":"2024-04-17T08:51:55.657884+0000","last_became_peered":"2024-04-17T08:51:55.657884+0000","last_unstale":"2024-04-17T08:51:57.688965+0000","last_undegraded":"2024-04-17T08:51:57.688965+0000","last_fullsized":"2024-04-17T08:51:57.688965+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:44.277240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707823+0000","last_change":"2024-04-17T08:51:41.503459+0000","last_active":"2024-04-17T08:51:57.707823+0000","last_peered":"2024-04-17T08:51:57.707823+0000","last_clean":"2024-04-17T08:51:57.707823+0000","last_became_active":"2024-04-17T08:51:41.503139+0000","last_became_peered":"2024-04-17T08:51:41.503139+0000","last_unstale":"2024-04-17T08:51:57.707823+0000","last_undegraded":"2024-04-17T08:51:57.707823+0000","last_fullsized":"2024-04-17T08:51:57.707823+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:58.046980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695632+0000","last_change":"2024-04-17T08:51:56.692609+0000","last_active":"2024-04-17T08:51:57.695632+0000","last_peered":"2024-04-17T08:51:57.695632+0000","last_clean":"2024-04-17T08:51:57.695632+0000","last_became_active":"2024-04-17T08:51:56.692224+0000","last_became_peered":"2024-04-17T08:51:56.692224+0000","last_unstale":"2024-04-17T08:51:57.695632+0000","last_undegraded":"2024-04-17T08:51:57.695632+0000","last_fullsized":"2024-04-17T08:51:57.695632+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:57.428110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949954+0000","last_change":"2024-04-17T08:51:55.661940+0000","last_active":"2024-04-17T08:51:57.949954+0000","last_peered":"2024-04-17T08:51:57.949954+0000","last_clean":"2024-04-17T08:51:57.949954+0000","last_became_active":"2024-04-17T08:51:55.661441+0000","last_became_peered":"2024-04-17T08:51:55.661441+0000","last_unstale":"2024-04-17T08:51:57.949954+0000","last_undegraded":"2024-04-17T08:51:57.949954+0000","last_fullsized":"2024-04-17T08:51:57.949954+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:46.720133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694103+0000","last_change":"2024-04-17T08:51:41.515918+0000","last_active":"2024-04-17T08:51:57.694103+0000","last_peered":"2024-04-17T08:51:57.694103+0000","last_clean":"2024-04-17T08:51:57.694103+0000","last_became_active":"2024-04-17T08:51:41.515377+0000","last_became_peered":"2024-04-17T08:51:41.515377+0000","last_unstale":"2024-04-17T08:51:57.694103+0000","last_undegraded":"2024-04-17T08:51:57.694103+0000","last_fullsized":"2024-04-17T08:51:57.694103+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:51:05.859833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693100+0000","last_change":"2024-04-17T08:51:56.688318+0000","last_active":"2024-04-17T08:51:57.693100+0000","last_peered":"2024-04-17T08:51:57.693100+0000","last_clean":"2024-04-17T08:51:57.693100+0000","last_became_active":"2024-04-17T08:51:56.687757+0000","last_became_peered":"2024-04-17T08:51:56.687757+0000","last_unstale":"2024-04-17T08:51:57.693100+0000","last_undegraded":"2024-04-17T08:51:57.693100+0000","last_fullsized":"2024-04-17T08:51:57.693100+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:22.806186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337849+0000","last_change":"2024-04-17T08:51:55.667305+0000","last_active":"2024-04-17T08:51:58.337849+0000","last_peered":"2024-04-17T08:51:58.337849+0000","last_clean":"2024-04-17T08:51:58.337849+0000","last_became_active":"2024-04-17T08:51:55.666504+0000","last_became_peered":"2024-04-17T08:51:55.666504+0000","last_unstale":"2024-04-17T08:51:58.337849+0000","last_undegraded":"2024-04-17T08:51:58.337849+0000","last_fullsized":"2024-04-17T08:51:58.337849+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:59.854465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707307+0000","last_change":"2024-04-17T08:51:41.497092+0000","last_active":"2024-04-17T08:51:57.707307+0000","last_peered":"2024-04-17T08:51:57.707307+0000","last_clean":"2024-04-17T08:51:57.707307+0000","last_became_active":"2024-04-17T08:51:41.496813+0000","last_became_peered":"2024-04-17T08:51:41.496813+0000","last_unstale":"2024-04-17T08:51:57.707307+0000","last_undegraded":"2024-04-17T08:51:57.707307+0000","last_fullsized":"2024-04-17T08:51:57.707307+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:34.747527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950901+0000","last_change":"2024-04-17T08:51:56.701098+0000","last_active":"2024-04-17T08:51:57.950901+0000","last_peered":"2024-04-17T08:51:57.950901+0000","last_clean":"2024-04-17T08:51:57.950901+0000","last_became_active":"2024-04-17T08:51:56.700484+0000","last_became_peered":"2024-04-17T08:51:56.700484+0000","last_unstale":"2024-04-17T08:51:57.950901+0000","last_undegraded":"2024-04-17T08:51:57.950901+0000","last_fullsized":"2024-04-17T08:51:57.950901+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:07:30.742939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"254'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.705865+0000","last_change":"2024-04-17T08:51:55.657402+0000","last_active":"2024-04-17T08:51:57.705865+0000","last_peered":"2024-04-17T08:51:57.705865+0000","last_clean":"2024-04-17T08:51:57.705865+0000","last_became_active":"2024-04-17T08:51:55.657047+0000","last_became_peered":"2024-04-17T08:51:55.657047+0000","last_unstale":"2024-04-17T08:51:57.705865+0000","last_undegraded":"2024-04-17T08:51:57.705865+0000","last_fullsized":"2024-04-17T08:51:57.705865+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:45.089304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336367+0000","last_change":"2024-04-17T08:51:41.512828+0000","last_active":"2024-04-17T08:51:58.336367+0000","last_peered":"2024-04-17T08:51:58.336367+0000","last_clean":"2024-04-17T08:51:58.336367+0000","last_became_active":"2024-04-17T08:51:41.512495+0000","last_became_peered":"2024-04-17T08:51:41.512495+0000","last_unstale":"2024-04-17T08:51:58.336367+0000","last_undegraded":"2024-04-17T08:51:58.336367+0000","last_fullsized":"2024-04-17T08:51:58.336367+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:59.808221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707743+0000","last_change":"2024-04-17T08:51:56.677033+0000","last_active":"2024-04-17T08:51:57.707743+0000","last_peered":"2024-04-17T08:51:57.707743+0000","last_clean":"2024-04-17T08:51:57.707743+0000","last_became_active":"2024-04-17T08:51:56.675988+0000","last_became_peered":"2024-04-17T08:51:56.675988+0000","last_unstale":"2024-04-17T08:51:57.707743+0000","last_undegraded":"2024-04-17T08:51:57.707743+0000","last_fullsized":"2024-04-17T08:51:57.707743+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:09:56.307968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708209+0000","last_change":"2024-04-17T08:51:55.669776+0000","last_active":"2024-04-17T08:51:57.708209+0000","last_peered":"2024-04-17T08:51:57.708209+0000","last_clean":"2024-04-17T08:51:57.708209+0000","last_became_active":"2024-04-17T08:51:55.669487+0000","last_became_peered":"2024-04-17T08:51:55.669487+0000","last_unstale":"2024-04-17T08:51:57.708209+0000","last_undegraded":"2024-04-17T08:51:57.708209+0000","last_fullsized":"2024-04-17T08:51:57.708209+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:42:25.375909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692839+0000","last_change":"2024-04-17T08:51:41.505463+0000","last_active":"2024-04-17T08:51:57.692839+0000","last_peered":"2024-04-17T08:51:57.692839+0000","last_clean":"2024-04-17T08:51:57.692839+0000","last_became_active":"2024-04-17T08:51:41.505186+0000","last_became_peered":"2024-04-17T08:51:41.505186+0000","last_unstale":"2024-04-17T08:51:57.692839+0000","last_undegraded":"2024-04-17T08:51:57.692839+0000","last_fullsized":"2024-04-17T08:51:57.692839+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688806+0000","last_change":"2024-04-17T08:51:56.683133+0000","last_active":"2024-04-17T08:51:57.688806+0000","last_peered":"2024-04-17T08:51:57.688806+0000","last_clean":"2024-04-17T08:51:57.688806+0000","last_became_active":"2024-04-17T08:51:56.682850+0000","last_became_peered":"2024-04-17T08:51:56.682850+0000","last_unstale":"2024-04-17T08:51:57.688806+0000","last_undegraded":"2024-04-17T08:51:57.688806+0000","last_fullsized":"2024-04-17T08:51:57.688806+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:44.685939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692459+0000","last_change":"2024-04-17T08:51:55.656373+0000","last_active":"2024-04-17T08:51:57.692459+0000","last_peered":"2024-04-17T08:51:57.692459+0000","last_clean":"2024-04-17T08:51:57.692459+0000","last_became_active":"2024-04-17T08:51:55.655998+0000","last_became_peered":"2024-04-17T08:51:55.655998+0000","last_unstale":"2024-04-17T08:51:57.692459+0000","last_undegraded":"2024-04-17T08:51:57.692459+0000","last_fullsized":"2024-04-17T08:51:57.692459+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.690537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667478+0000","last_change":"2024-04-17T08:51:41.503091+0000","last_active":"2024-04-17T08:51:56.667478+0000","last_peered":"2024-04-17T08:51:56.667478+0000","last_clean":"2024-04-17T08:51:56.667478+0000","last_became_active":"2024-04-17T08:51:41.502796+0000","last_became_peered":"2024-04-17T08:51:41.502796+0000","last_unstale":"2024-04-17T08:51:56.667478+0000","last_undegraded":"2024-04-17T08:51:56.667478+0000","last_fullsized":"2024-04-17T08:51:56.667478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:44.277963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948657+0000","last_change":"2024-04-17T08:51:56.696366+0000","last_active":"2024-04-17T08:51:57.948657+0000","last_peered":"2024-04-17T08:51:57.948657+0000","last_clean":"2024-04-17T08:51:57.948657+0000","last_became_active":"2024-04-17T08:51:56.681654+0000","last_became_peered":"2024-04-17T08:51:56.681654+0000","last_unstale":"2024-04-17T08:51:57.948657+0000","last_undegraded":"2024-04-17T08:51:57.948657+0000","last_fullsized":"2024-04-17T08:51:57.948657+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:05:30.582209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.664275+0000","last_change":"2024-04-17T08:51:55.668838+0000","last_active":"2024-04-17T08:51:56.664275+0000","last_peered":"2024-04-17T08:51:56.664275+0000","last_clean":"2024-04-17T08:51:56.664275+0000","last_became_active":"2024-04-17T08:51:55.668484+0000","last_became_peered":"2024-04-17T08:51:55.668484+0000","last_unstale":"2024-04-17T08:51:56.664275+0000","last_undegraded":"2024-04-17T08:51:56.664275+0000","last_fullsized":"2024-04-17T08:51:56.664275+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:58.622251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338671+0000","last_change":"2024-04-17T08:51:41.517488+0000","last_active":"2024-04-17T08:51:58.338671+0000","last_peered":"2024-04-17T08:51:58.338671+0000","last_clean":"2024-04-17T08:51:58.338671+0000","last_became_active":"2024-04-17T08:51:41.517224+0000","last_became_peered":"2024-04-17T08:51:41.517224+0000","last_unstale":"2024-04-17T08:51:58.338671+0000","last_undegraded":"2024-04-17T08:51:58.338671+0000","last_fullsized":"2024-04-17T08:51:58.338671+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.702094+0000","last_change":"2024-04-17T08:51:56.698483+0000","last_active":"2024-04-17T08:51:56.702094+0000","last_peered":"2024-04-17T08:51:56.702094+0000","last_clean":"2024-04-17T08:51:56.702094+0000","last_became_active":"2024-04-17T08:51:56.698188+0000","last_became_peered":"2024-04-17T08:51:56.698188+0000","last_unstale":"2024-04-17T08:51:56.702094+0000","last_undegraded":"2024-04-17T08:51:56.702094+0000","last_fullsized":"2024-04-17T08:51:56.702094+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:19.239448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691954+0000","last_change":"2024-04-17T08:51:55.649899+0000","last_active":"2024-04-17T08:51:57.691954+0000","last_peered":"2024-04-17T08:51:57.691954+0000","last_clean":"2024-04-17T08:51:57.691954+0000","last_became_active":"2024-04-17T08:51:55.649579+0000","last_became_peered":"2024-04-17T08:51:55.649579+0000","last_unstale":"2024-04-17T08:51:57.691954+0000","last_undegraded":"2024-04-17T08:51:57.691954+0000","last_fullsized":"2024-04-17T08:51:57.691954+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:26.416866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689480+0000","last_change":"2024-04-17T08:51:41.520584+0000","last_active":"2024-04-17T08:51:57.689480+0000","last_peered":"2024-04-17T08:51:57.689480+0000","last_clean":"2024-04-17T08:51:57.689480+0000","last_became_active":"2024-04-17T08:51:41.520134+0000","last_became_peered":"2024-04-17T08:51:41.520134+0000","last_unstale":"2024-04-17T08:51:57.689480+0000","last_undegraded":"2024-04-17T08:51:57.689480+0000","last_fullsized":"2024-04-17T08:51:57.689480+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:25.042374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699772+0000","last_change":"2024-04-17T08:51:56.694171+0000","last_active":"2024-04-17T08:51:56.699772+0000","last_peered":"2024-04-17T08:51:56.699772+0000","last_clean":"2024-04-17T08:51:56.699772+0000","last_became_active":"2024-04-17T08:51:56.693872+0000","last_became_peered":"2024-04-17T08:51:56.693872+0000","last_unstale":"2024-04-17T08:51:56.699772+0000","last_undegraded":"2024-04-17T08:51:56.699772+0000","last_fullsized":"2024-04-17T08:51:56.699772+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:49.406094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693922+0000","last_change":"2024-04-17T08:51:55.659521+0000","last_active":"2024-04-17T08:51:57.693922+0000","last_peered":"2024-04-17T08:51:57.693922+0000","last_clean":"2024-04-17T08:51:57.693922+0000","last_became_active":"2024-04-17T08:51:55.658645+0000","last_became_peered":"2024-04-17T08:51:55.658645+0000","last_unstale":"2024-04-17T08:51:57.693922+0000","last_undegraded":"2024-04-17T08:51:57.693922+0000","last_fullsized":"2024-04-17T08:51:57.693922+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:52.153579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695970+0000","last_change":"2024-04-17T08:51:41.518003+0000","last_active":"2024-04-17T08:51:57.695970+0000","last_peered":"2024-04-17T08:51:57.695970+0000","last_clean":"2024-04-17T08:51:57.695970+0000","last_became_active":"2024-04-17T08:51:41.517732+0000","last_became_peered":"2024-04-17T08:51:41.517732+0000","last_unstale":"2024-04-17T08:51:57.695970+0000","last_undegraded":"2024-04-17T08:51:57.695970+0000","last_fullsized":"2024-04-17T08:51:57.695970+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:55.524149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695361+0000","last_change":"2024-04-17T08:51:56.692234+0000","last_active":"2024-04-17T08:51:57.695361+0000","last_peered":"2024-04-17T08:51:57.695361+0000","last_clean":"2024-04-17T08:51:57.695361+0000","last_became_active":"2024-04-17T08:51:56.691508+0000","last_became_peered":"2024-04-17T08:51:56.691508+0000","last_unstale":"2024-04-17T08:51:57.695361+0000","last_undegraded":"2024-04-17T08:51:57.695361+0000","last_fullsized":"2024-04-17T08:51:57.695361+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:19.402584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707656+0000","last_change":"2024-04-17T08:51:55.670726+0000","last_active":"2024-04-17T08:51:57.707656+0000","last_peered":"2024-04-17T08:51:57.707656+0000","last_clean":"2024-04-17T08:51:57.707656+0000","last_became_active":"2024-04-17T08:51:55.668578+0000","last_became_peered":"2024-04-17T08:51:55.668578+0000","last_unstale":"2024-04-17T08:51:57.707656+0000","last_undegraded":"2024-04-17T08:51:57.707656+0000","last_fullsized":"2024-04-17T08:51:57.707656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:32:29.845006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337245+0000","last_change":"2024-04-17T08:51:41.514813+0000","last_active":"2024-04-17T08:51:58.337245+0000","last_peered":"2024-04-17T08:51:58.337245+0000","last_clean":"2024-04-17T08:51:58.337245+0000","last_became_active":"2024-04-17T08:51:41.512779+0000","last_became_peered":"2024-04-17T08:51:41.512779+0000","last_unstale":"2024-04-17T08:51:58.337245+0000","last_undegraded":"2024-04-17T08:51:58.337245+0000","last_fullsized":"2024-04-17T08:51:58.337245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707510+0000","last_change":"2024-04-17T08:51:56.674702+0000","last_active":"2024-04-17T08:51:57.707510+0000","last_peered":"2024-04-17T08:51:57.707510+0000","last_clean":"2024-04-17T08:51:57.707510+0000","last_became_active":"2024-04-17T08:51:56.673432+0000","last_became_peered":"2024-04-17T08:51:56.673432+0000","last_unstale":"2024-04-17T08:51:57.707510+0000","last_undegraded":"2024-04-17T08:51:57.707510+0000","last_fullsized":"2024-04-17T08:51:57.707510+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:10.247987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"253'1","reported_seq":13,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.711806+0000","last_change":"2024-04-17T08:51:55.671864+0000","last_active":"2024-04-17T08:51:56.711806+0000","last_peered":"2024-04-17T08:51:56.711806+0000","last_clean":"2024-04-17T08:51:56.711806+0000","last_became_active":"2024-04-17T08:51:55.671587+0000","last_became_peered":"2024-04-17T08:51:55.671587+0000","last_unstale":"2024-04-17T08:51:56.711806+0000","last_undegraded":"2024-04-17T08:51:56.711806+0000","last_fullsized":"2024-04-17T08:51:56.711806+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:25:29.119928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668474+0000","last_change":"2024-04-17T08:51:41.507865+0000","last_active":"2024-04-17T08:51:56.668474+0000","last_peered":"2024-04-17T08:51:56.668474+0000","last_clean":"2024-04-17T08:51:56.668474+0000","last_became_active":"2024-04-17T08:51:41.507622+0000","last_became_peered":"2024-04-17T08:51:41.507622+0000","last_unstale":"2024-04-17T08:51:56.668474+0000","last_undegraded":"2024-04-17T08:51:56.668474+0000","last_fullsized":"2024-04-17T08:51:56.668474+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.174140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.687058+0000","last_change":"2024-04-17T08:51:56.680740+0000","last_active":"2024-04-17T08:51:56.687058+0000","last_peered":"2024-04-17T08:51:56.687058+0000","last_clean":"2024-04-17T08:51:56.687058+0000","last_became_active":"2024-04-17T08:51:56.680396+0000","last_became_peered":"2024-04-17T08:51:56.680396+0000","last_unstale":"2024-04-17T08:51:56.687058+0000","last_undegraded":"2024-04-17T08:51:56.687058+0000","last_fullsized":"2024-04-17T08:51:56.687058+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:38.087816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.713513+0000","last_change":"2024-04-17T08:51:55.661494+0000","last_active":"2024-04-17T08:51:56.713513+0000","last_peered":"2024-04-17T08:51:56.713513+0000","last_clean":"2024-04-17T08:51:56.713513+0000","last_became_active":"2024-04-17T08:51:55.660821+0000","last_became_peered":"2024-04-17T08:51:55.660821+0000","last_unstale":"2024-04-17T08:51:56.713513+0000","last_undegraded":"2024-04-17T08:51:56.713513+0000","last_fullsized":"2024-04-17T08:51:56.713513+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.789856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666243+0000","last_change":"2024-04-17T08:51:41.501853+0000","last_active":"2024-04-17T08:51:56.666243+0000","last_peered":"2024-04-17T08:51:56.666243+0000","last_clean":"2024-04-17T08:51:56.666243+0000","last_became_active":"2024-04-17T08:51:41.501575+0000","last_became_peered":"2024-04-17T08:51:41.501575+0000","last_unstale":"2024-04-17T08:51:56.666243+0000","last_undegraded":"2024-04-17T08:51:56.666243+0000","last_fullsized":"2024-04-17T08:51:56.666243+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:00.028124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708068+0000","last_change":"2024-04-17T08:51:56.679445+0000","last_active":"2024-04-17T08:51:57.708068+0000","last_peered":"2024-04-17T08:51:57.708068+0000","last_clean":"2024-04-17T08:51:57.708068+0000","last_became_active":"2024-04-17T08:51:56.676586+0000","last_became_peered":"2024-04-17T08:51:56.676586+0000","last_unstale":"2024-04-17T08:51:57.708068+0000","last_undegraded":"2024-04-17T08:51:57.708068+0000","last_fullsized":"2024-04-17T08:51:57.708068+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:45:31.719505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692290+0000","last_change":"2024-04-17T08:51:55.664551+0000","last_active":"2024-04-17T08:51:57.692290+0000","last_peered":"2024-04-17T08:51:57.692290+0000","last_clean":"2024-04-17T08:51:57.692290+0000","last_became_active":"2024-04-17T08:51:55.664267+0000","last_became_peered":"2024-04-17T08:51:55.664267+0000","last_unstale":"2024-04-17T08:51:57.692290+0000","last_undegraded":"2024-04-17T08:51:57.692290+0000","last_fullsized":"2024-04-17T08:51:57.692290+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:21:27.768335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948708+0000","last_change":"2024-04-17T08:51:41.513490+0000","last_active":"2024-04-17T08:51:57.948708+0000","last_peered":"2024-04-17T08:51:57.948708+0000","last_clean":"2024-04-17T08:51:57.948708+0000","last_became_active":"2024-04-17T08:51:41.513192+0000","last_became_peered":"2024-04-17T08:51:41.513192+0000","last_unstale":"2024-04-17T08:51:57.948708+0000","last_undegraded":"2024-04-17T08:51:57.948708+0000","last_fullsized":"2024-04-17T08:51:57.948708+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:47.830902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948935+0000","last_change":"2024-04-17T08:51:56.702728+0000","last_active":"2024-04-17T08:51:57.948935+0000","last_peered":"2024-04-17T08:51:57.948935+0000","last_clean":"2024-04-17T08:51:57.948935+0000","last_became_active":"2024-04-17T08:51:56.689917+0000","last_became_peered":"2024-04-17T08:51:56.689917+0000","last_unstale":"2024-04-17T08:51:57.948935+0000","last_undegraded":"2024-04-17T08:51:57.948935+0000","last_fullsized":"2024-04-17T08:51:57.948935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:09.401482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"253'8","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.712919+0000","last_change":"2024-04-17T08:51:55.661658+0000","last_active":"2024-04-17T08:51:56.712919+0000","last_peered":"2024-04-17T08:51:56.712919+0000","last_clean":"2024-04-17T08:51:56.712919+0000","last_became_active":"2024-04-17T08:51:55.661387+0000","last_became_peered":"2024-04-17T08:51:55.661387+0000","last_unstale":"2024-04-17T08:51:56.712919+0000","last_undegraded":"2024-04-17T08:51:56.712919+0000","last_fullsized":"2024-04-17T08:51:56.712919+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:59.234393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695578+0000","last_change":"2024-04-17T08:51:41.516221+0000","last_active":"2024-04-17T08:51:57.695578+0000","last_peered":"2024-04-17T08:51:57.695578+0000","last_clean":"2024-04-17T08:51:57.695578+0000","last_became_active":"2024-04-17T08:51:41.515896+0000","last_became_peered":"2024-04-17T08:51:41.515896+0000","last_unstale":"2024-04-17T08:51:57.695578+0000","last_undegraded":"2024-04-17T08:51:57.695578+0000","last_fullsized":"2024-04-17T08:51:57.695578+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:22.484668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948492+0000","last_change":"2024-04-17T08:51:56.701571+0000","last_active":"2024-04-17T08:51:57.948492+0000","last_peered":"2024-04-17T08:51:57.948492+0000","last_clean":"2024-04-17T08:51:57.948492+0000","last_became_active":"2024-04-17T08:51:56.698942+0000","last_became_peered":"2024-04-17T08:51:56.698942+0000","last_unstale":"2024-04-17T08:51:57.948492+0000","last_undegraded":"2024-04-17T08:51:57.948492+0000","last_fullsized":"2024-04-17T08:51:57.948492+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:49.809439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666798+0000","last_change":"2024-04-17T08:51:55.667334+0000","last_active":"2024-04-17T08:51:56.666798+0000","last_peered":"2024-04-17T08:51:56.666798+0000","last_clean":"2024-04-17T08:51:56.666798+0000","last_became_active":"2024-04-17T08:51:55.667038+0000","last_became_peered":"2024-04-17T08:51:55.667038+0000","last_unstale":"2024-04-17T08:51:56.666798+0000","last_undegraded":"2024-04-17T08:51:56.666798+0000","last_fullsized":"2024-04-17T08:51:56.666798+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:29.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.669781+0000","last_change":"2024-04-17T08:51:41.501545+0000","last_active":"2024-04-17T08:51:56.669781+0000","last_peered":"2024-04-17T08:51:56.669781+0000","last_clean":"2024-04-17T08:51:56.669781+0000","last_became_active":"2024-04-17T08:51:41.500258+0000","last_became_peered":"2024-04-17T08:51:41.500258+0000","last_unstale":"2024-04-17T08:51:56.669781+0000","last_undegraded":"2024-04-17T08:51:56.669781+0000","last_fullsized":"2024-04-17T08:51:56.669781+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:24.809010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699045+0000","last_change":"2024-04-17T08:51:56.691565+0000","last_active":"2024-04-17T08:51:56.699045+0000","last_peered":"2024-04-17T08:51:56.699045+0000","last_clean":"2024-04-17T08:51:56.699045+0000","last_became_active":"2024-04-17T08:51:56.686842+0000","last_became_peered":"2024-04-17T08:51:56.686842+0000","last_unstale":"2024-04-17T08:51:56.699045+0000","last_undegraded":"2024-04-17T08:51:56.699045+0000","last_fullsized":"2024-04-17T08:51:56.699045+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:43.988578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689124+0000","last_change":"2024-04-17T08:51:55.662235+0000","last_active":"2024-04-17T08:51:57.689124+0000","last_peered":"2024-04-17T08:51:57.689124+0000","last_clean":"2024-04-17T08:51:57.689124+0000","last_became_active":"2024-04-17T08:51:55.660563+0000","last_became_peered":"2024-04-17T08:51:55.660563+0000","last_unstale":"2024-04-17T08:51:57.689124+0000","last_undegraded":"2024-04-17T08:51:57.689124+0000","last_fullsized":"2024-04-17T08:51:57.689124+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:57.995854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693586+0000","last_change":"2024-04-17T08:51:41.505156+0000","last_active":"2024-04-17T08:51:57.693586+0000","last_peered":"2024-04-17T08:51:57.693586+0000","last_clean":"2024-04-17T08:51:57.693586+0000","last_became_active":"2024-04-17T08:51:41.503363+0000","last_became_peered":"2024-04-17T08:51:41.503363+0000","last_unstale":"2024-04-17T08:51:57.693586+0000","last_undegraded":"2024-04-17T08:51:57.693586+0000","last_fullsized":"2024-04-17T08:51:57.693586+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:28.800799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950630+0000","last_change":"2024-04-17T08:51:56.710667+0000","last_active":"2024-04-17T08:51:57.950630+0000","last_peered":"2024-04-17T08:51:57.950630+0000","last_clean":"2024-04-17T08:51:57.950630+0000","last_became_active":"2024-04-17T08:51:56.703014+0000","last_became_peered":"2024-04-17T08:51:56.703014+0000","last_unstale":"2024-04-17T08:51:57.950630+0000","last_undegraded":"2024-04-17T08:51:57.950630+0000","last_fullsized":"2024-04-17T08:51:57.950630+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:52.613169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693148+0000","last_change":"2024-04-17T08:51:55.661978+0000","last_active":"2024-04-17T08:51:57.693148+0000","last_peered":"2024-04-17T08:51:57.693148+0000","last_clean":"2024-04-17T08:51:57.693148+0000","last_became_active":"2024-04-17T08:51:55.661298+0000","last_became_peered":"2024-04-17T08:51:55.661298+0000","last_unstale":"2024-04-17T08:51:57.693148+0000","last_undegraded":"2024-04-17T08:51:57.693148+0000","last_fullsized":"2024-04-17T08:51:57.693148+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:57.667695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949971+0000","last_change":"2024-04-17T08:51:41.500145+0000","last_active":"2024-04-17T08:51:57.949971+0000","last_peered":"2024-04-17T08:51:57.949971+0000","last_clean":"2024-04-17T08:51:57.949971+0000","last_became_active":"2024-04-17T08:51:41.499316+0000","last_became_peered":"2024-04-17T08:51:41.499316+0000","last_unstale":"2024-04-17T08:51:57.949971+0000","last_undegraded":"2024-04-17T08:51:57.949971+0000","last_fullsized":"2024-04-17T08:51:57.949971+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:49.058157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950485+0000","last_change":"2024-04-17T08:51:56.698232+0000","last_active":"2024-04-17T08:51:57.950485+0000","last_peered":"2024-04-17T08:51:57.950485+0000","last_clean":"2024-04-17T08:51:57.950485+0000","last_became_active":"2024-04-17T08:51:56.697860+0000","last_became_peered":"2024-04-17T08:51:56.697860+0000","last_unstale":"2024-04-17T08:51:57.950485+0000","last_undegraded":"2024-04-17T08:51:57.950485+0000","last_fullsized":"2024-04-17T08:51:57.950485+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:24.328571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"253'6","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949718+0000","last_change":"2024-04-17T08:51:55.652190+0000","last_active":"2024-04-17T08:51:57.949718+0000","last_peered":"2024-04-17T08:51:57.949718+0000","last_clean":"2024-04-17T08:51:57.949718+0000","last_became_active":"2024-04-17T08:51:55.651916+0000","last_became_peered":"2024-04-17T08:51:55.651916+0000","last_unstale":"2024-04-17T08:51:57.949718+0000","last_undegraded":"2024-04-17T08:51:57.949718+0000","last_fullsized":"2024-04-17T08:51:57.949718+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:25:29.119928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693446+0000","last_change":"2024-04-17T08:51:41.508121+0000","last_active":"2024-04-17T08:51:57.693446+0000","last_peered":"2024-04-17T08:51:57.693446+0000","last_clean":"2024-04-17T08:51:57.693446+0000","last_became_active":"2024-04-17T08:51:41.506717+0000","last_became_peered":"2024-04-17T08:51:41.506717+0000","last_unstale":"2024-04-17T08:51:57.693446+0000","last_undegraded":"2024-04-17T08:51:57.693446+0000","last_fullsized":"2024-04-17T08:51:57.693446+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:45.296249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707485+0000","last_change":"2024-04-17T08:51:56.697344+0000","last_active":"2024-04-17T08:51:57.707485+0000","last_peered":"2024-04-17T08:51:57.707485+0000","last_clean":"2024-04-17T08:51:57.707485+0000","last_became_active":"2024-04-17T08:51:56.694288+0000","last_became_peered":"2024-04-17T08:51:56.694288+0000","last_unstale":"2024-04-17T08:51:57.707485+0000","last_undegraded":"2024-04-17T08:51:57.707485+0000","last_fullsized":"2024-04-17T08:51:57.707485+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:57.840224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"253'3","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.716009+0000","last_change":"2024-04-17T08:51:55.670632+0000","last_active":"2024-04-17T08:51:56.716009+0000","last_peered":"2024-04-17T08:51:56.716009+0000","last_clean":"2024-04-17T08:51:56.716009+0000","last_became_active":"2024-04-17T08:51:55.670207+0000","last_became_peered":"2024-04-17T08:51:55.670207+0000","last_unstale":"2024-04-17T08:51:56.716009+0000","last_undegraded":"2024-04-17T08:51:56.716009+0000","last_fullsized":"2024-04-17T08:51:56.716009+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:06.066708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336354+0000","last_change":"2024-04-17T08:51:41.517429+0000","last_active":"2024-04-17T08:51:58.336354+0000","last_peered":"2024-04-17T08:51:58.336354+0000","last_clean":"2024-04-17T08:51:58.336354+0000","last_became_active":"2024-04-17T08:51:41.517130+0000","last_became_peered":"2024-04-17T08:51:41.517130+0000","last_unstale":"2024-04-17T08:51:58.336354+0000","last_undegraded":"2024-04-17T08:51:58.336354+0000","last_fullsized":"2024-04-17T08:51:58.336354+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:14.831099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950763+0000","last_change":"2024-04-17T08:51:56.709710+0000","last_active":"2024-04-17T08:51:57.950763+0000","last_peered":"2024-04-17T08:51:57.950763+0000","last_clean":"2024-04-17T08:51:57.950763+0000","last_became_active":"2024-04-17T08:51:56.704006+0000","last_became_peered":"2024-04-17T08:51:56.704006+0000","last_unstale":"2024-04-17T08:51:57.950763+0000","last_undegraded":"2024-04-17T08:51:57.950763+0000","last_fullsized":"2024-04-17T08:51:57.950763+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:51:34.353837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668959+0000","last_change":"2024-04-17T08:51:55.669734+0000","last_active":"2024-04-17T08:51:56.668959+0000","last_peered":"2024-04-17T08:51:56.668959+0000","last_clean":"2024-04-17T08:51:56.668959+0000","last_became_active":"2024-04-17T08:51:55.669456+0000","last_became_peered":"2024-04-17T08:51:55.669456+0000","last_unstale":"2024-04-17T08:51:56.668959+0000","last_undegraded":"2024-04-17T08:51:56.668959+0000","last_fullsized":"2024-04-17T08:51:56.668959+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:28.256872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691710+0000","last_change":"2024-04-17T08:51:41.507626+0000","last_active":"2024-04-17T08:51:57.691710+0000","last_peered":"2024-04-17T08:51:57.691710+0000","last_clean":"2024-04-17T08:51:57.691710+0000","last_became_active":"2024-04-17T08:51:41.507278+0000","last_became_peered":"2024-04-17T08:51:41.507278+0000","last_unstale":"2024-04-17T08:51:57.691710+0000","last_undegraded":"2024-04-17T08:51:57.691710+0000","last_fullsized":"2024-04-17T08:51:57.691710+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338380+0000","last_change":"2024-04-17T08:51:56.689527+0000","last_active":"2024-04-17T08:51:58.338380+0000","last_peered":"2024-04-17T08:51:58.338380+0000","last_clean":"2024-04-17T08:51:58.338380+0000","last_became_active":"2024-04-17T08:51:56.688886+0000","last_became_peered":"2024-04-17T08:51:56.688886+0000","last_unstale":"2024-04-17T08:51:58.338380+0000","last_undegraded":"2024-04-17T08:51:58.338380+0000","last_fullsized":"2024-04-17T08:51:58.338380+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:03.266907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951174+0000","last_change":"2024-04-17T08:51:55.663292+0000","last_active":"2024-04-17T08:51:57.951174+0000","last_peered":"2024-04-17T08:51:57.951174+0000","last_clean":"2024-04-17T08:51:57.951174+0000","last_became_active":"2024-04-17T08:51:55.661670+0000","last_became_peered":"2024-04-17T08:51:55.661670+0000","last_unstale":"2024-04-17T08:51:57.951174+0000","last_undegraded":"2024-04-17T08:51:57.951174+0000","last_fullsized":"2024-04-17T08:51:57.951174+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:37.070857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.670662+0000","last_change":"2024-04-17T08:51:41.502141+0000","last_active":"2024-04-17T08:51:56.670662+0000","last_peered":"2024-04-17T08:51:56.670662+0000","last_clean":"2024-04-17T08:51:56.670662+0000","last_became_active":"2024-04-17T08:51:41.501895+0000","last_became_peered":"2024-04-17T08:51:41.501895+0000","last_unstale":"2024-04-17T08:51:56.670662+0000","last_undegraded":"2024-04-17T08:51:56.670662+0000","last_fullsized":"2024-04-17T08:51:56.670662+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:45.139092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337013+0000","last_change":"2024-04-17T08:51:56.689396+0000","last_active":"2024-04-17T08:51:58.337013+0000","last_peered":"2024-04-17T08:51:58.337013+0000","last_clean":"2024-04-17T08:51:58.337013+0000","last_became_active":"2024-04-17T08:51:56.688849+0000","last_became_peered":"2024-04-17T08:51:56.688849+0000","last_unstale":"2024-04-17T08:51:58.337013+0000","last_undegraded":"2024-04-17T08:51:58.337013+0000","last_fullsized":"2024-04-17T08:51:58.337013+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:51.427089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336888+0000","last_change":"2024-04-17T08:51:55.665394+0000","last_active":"2024-04-17T08:51:58.336888+0000","last_peered":"2024-04-17T08:51:58.336888+0000","last_clean":"2024-04-17T08:51:58.336888+0000","last_became_active":"2024-04-17T08:51:55.664784+0000","last_became_peered":"2024-04-17T08:51:55.664784+0000","last_unstale":"2024-04-17T08:51:58.336888+0000","last_undegraded":"2024-04-17T08:51:58.336888+0000","last_fullsized":"2024-04-17T08:51:58.336888+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:21.789227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708177+0000","last_change":"2024-04-17T08:51:41.504007+0000","last_active":"2024-04-17T08:51:57.708177+0000","last_peered":"2024-04-17T08:51:57.708177+0000","last_clean":"2024-04-17T08:51:57.708177+0000","last_became_active":"2024-04-17T08:51:41.503745+0000","last_became_peered":"2024-04-17T08:51:41.503745+0000","last_unstale":"2024-04-17T08:51:57.708177+0000","last_undegraded":"2024-04-17T08:51:57.708177+0000","last_fullsized":"2024-04-17T08:51:57.708177+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:30.992713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699556+0000","last_change":"2024-04-17T08:51:56.694165+0000","last_active":"2024-04-17T08:51:56.699556+0000","last_peered":"2024-04-17T08:51:56.699556+0000","last_clean":"2024-04-17T08:51:56.699556+0000","last_became_active":"2024-04-17T08:51:56.687018+0000","last_became_peered":"2024-04-17T08:51:56.687018+0000","last_unstale":"2024-04-17T08:51:56.699556+0000","last_undegraded":"2024-04-17T08:51:56.699556+0000","last_fullsized":"2024-04-17T08:51:56.699556+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:52.726637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692793+0000","last_change":"2024-04-17T08:51:55.653389+0000","last_active":"2024-04-17T08:51:57.692793+0000","last_peered":"2024-04-17T08:51:57.692793+0000","last_clean":"2024-04-17T08:51:57.692793+0000","last_became_active":"2024-04-17T08:51:55.650896+0000","last_became_peered":"2024-04-17T08:51:55.650896+0000","last_unstale":"2024-04-17T08:51:57.692793+0000","last_undegraded":"2024-04-17T08:51:57.692793+0000","last_fullsized":"2024-04-17T08:51:57.692793+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:19.603617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338873+0000","last_change":"2024-04-17T08:51:41.513058+0000","last_active":"2024-04-17T08:51:58.338873+0000","last_peered":"2024-04-17T08:51:58.338873+0000","last_clean":"2024-04-17T08:51:58.338873+0000","last_became_active":"2024-04-17T08:51:41.512444+0000","last_became_peered":"2024-04-17T08:51:41.512444+0000","last_unstale":"2024-04-17T08:51:58.338873+0000","last_undegraded":"2024-04-17T08:51:58.338873+0000","last_fullsized":"2024-04-17T08:51:58.338873+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699072+0000","last_change":"2024-04-17T08:51:56.693163+0000","last_active":"2024-04-17T08:51:56.699072+0000","last_peered":"2024-04-17T08:51:56.699072+0000","last_clean":"2024-04-17T08:51:56.699072+0000","last_became_active":"2024-04-17T08:51:56.692676+0000","last_became_peered":"2024-04-17T08:51:56.692676+0000","last_unstale":"2024-04-17T08:51:56.699072+0000","last_undegraded":"2024-04-17T08:51:56.699072+0000","last_fullsized":"2024-04-17T08:51:56.699072+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:56.847018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949550+0000","last_change":"2024-04-17T08:51:55.661438+0000","last_active":"2024-04-17T08:51:57.949550+0000","last_peered":"2024-04-17T08:51:57.949550+0000","last_clean":"2024-04-17T08:51:57.949550+0000","last_became_active":"2024-04-17T08:51:55.661097+0000","last_became_peered":"2024-04-17T08:51:55.661097+0000","last_unstale":"2024-04-17T08:51:57.949550+0000","last_undegraded":"2024-04-17T08:51:57.949550+0000","last_fullsized":"2024-04-17T08:51:57.949550+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:58.622251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689593+0000","last_change":"2024-04-17T08:51:41.519929+0000","last_active":"2024-04-17T08:51:57.689593+0000","last_peered":"2024-04-17T08:51:57.689593+0000","last_clean":"2024-04-17T08:51:57.689593+0000","last_became_active":"2024-04-17T08:51:41.519591+0000","last_became_peered":"2024-04-17T08:51:41.519591+0000","last_unstale":"2024-04-17T08:51:57.689593+0000","last_undegraded":"2024-04-17T08:51:57.689593+0000","last_fullsized":"2024-04-17T08:51:57.689593+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:03.965519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689567+0000","last_change":"2024-04-17T08:51:56.705675+0000","last_active":"2024-04-17T08:51:57.689567+0000","last_peered":"2024-04-17T08:51:57.689567+0000","last_clean":"2024-04-17T08:51:57.689567+0000","last_became_active":"2024-04-17T08:51:56.705389+0000","last_became_peered":"2024-04-17T08:51:56.705389+0000","last_unstale":"2024-04-17T08:51:57.689567+0000","last_undegraded":"2024-04-17T08:51:57.689567+0000","last_fullsized":"2024-04-17T08:51:57.689567+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:43.281173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337062+0000","last_change":"2024-04-17T08:51:55.664957+0000","last_active":"2024-04-17T08:51:58.337062+0000","last_peered":"2024-04-17T08:51:58.337062+0000","last_clean":"2024-04-17T08:51:58.337062+0000","last_became_active":"2024-04-17T08:51:55.664547+0000","last_became_peered":"2024-04-17T08:51:55.664547+0000","last_unstale":"2024-04-17T08:51:58.337062+0000","last_undegraded":"2024-04-17T08:51:58.337062+0000","last_fullsized":"2024-04-17T08:51:58.337062+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:52.224136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667876+0000","last_change":"2024-04-17T08:51:41.506863+0000","last_active":"2024-04-17T08:51:56.667876+0000","last_peered":"2024-04-17T08:51:56.667876+0000","last_clean":"2024-04-17T08:51:56.667876+0000","last_became_active":"2024-04-17T08:51:41.506606+0000","last_became_peered":"2024-04-17T08:51:41.506606+0000","last_unstale":"2024-04-17T08:51:56.667876+0000","last_undegraded":"2024-04-17T08:51:56.667876+0000","last_fullsized":"2024-04-17T08:51:56.667876+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:42.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693654+0000","last_change":"2024-04-17T08:51:56.690518+0000","last_active":"2024-04-17T08:51:57.693654+0000","last_peered":"2024-04-17T08:51:57.693654+0000","last_clean":"2024-04-17T08:51:57.693654+0000","last_became_active":"2024-04-17T08:51:56.688133+0000","last_became_peered":"2024-04-17T08:51:56.688133+0000","last_unstale":"2024-04-17T08:51:57.693654+0000","last_undegraded":"2024-04-17T08:51:57.693654+0000","last_fullsized":"2024-04-17T08:51:57.693654+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:45.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.710606+0000","last_change":"2024-04-17T08:51:55.658488+0000","last_active":"2024-04-17T08:51:56.710606+0000","last_peered":"2024-04-17T08:51:56.710606+0000","last_clean":"2024-04-17T08:51:56.710606+0000","last_became_active":"2024-04-17T08:51:55.658178+0000","last_became_peered":"2024-04-17T08:51:55.658178+0000","last_unstale":"2024-04-17T08:51:56.710606+0000","last_undegraded":"2024-04-17T08:51:56.710606+0000","last_fullsized":"2024-04-17T08:51:56.710606+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:14.517760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689072+0000","last_change":"2024-04-17T08:51:41.505583+0000","last_active":"2024-04-17T08:51:57.689072+0000","last_peered":"2024-04-17T08:51:57.689072+0000","last_clean":"2024-04-17T08:51:57.689072+0000","last_became_active":"2024-04-17T08:51:41.501478+0000","last_became_peered":"2024-04-17T08:51:41.501478+0000","last_unstale":"2024-04-17T08:51:57.689072+0000","last_undegraded":"2024-04-17T08:51:57.689072+0000","last_fullsized":"2024-04-17T08:51:57.689072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:01.976342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337430+0000","last_change":"2024-04-17T08:51:56.697296+0000","last_active":"2024-04-17T08:51:58.337430+0000","last_peered":"2024-04-17T08:51:58.337430+0000","last_clean":"2024-04-17T08:51:58.337430+0000","last_became_active":"2024-04-17T08:51:56.696997+0000","last_became_peered":"2024-04-17T08:51:56.696997+0000","last_unstale":"2024-04-17T08:51:58.337430+0000","last_undegraded":"2024-04-17T08:51:58.337430+0000","last_fullsized":"2024-04-17T08:51:58.337430+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:04.357822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.715404+0000","last_change":"2024-04-17T08:51:55.661037+0000","last_active":"2024-04-17T08:51:56.715404+0000","last_peered":"2024-04-17T08:51:56.715404+0000","last_clean":"2024-04-17T08:51:56.715404+0000","last_became_active":"2024-04-17T08:51:55.660680+0000","last_became_peered":"2024-04-17T08:51:55.660680+0000","last_unstale":"2024-04-17T08:51:56.715404+0000","last_undegraded":"2024-04-17T08:51:56.715404+0000","last_fullsized":"2024-04-17T08:51:56.715404+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:26.513422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693507+0000","last_change":"2024-04-17T08:51:41.500660+0000","last_active":"2024-04-17T08:51:57.693507+0000","last_peered":"2024-04-17T08:51:57.693507+0000","last_clean":"2024-04-17T08:51:57.693507+0000","last_became_active":"2024-04-17T08:51:41.500031+0000","last_became_peered":"2024-04-17T08:51:41.500031+0000","last_unstale":"2024-04-17T08:51:57.693507+0000","last_undegraded":"2024-04-17T08:51:57.693507+0000","last_fullsized":"2024-04-17T08:51:57.693507+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:10.441817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708007+0000","last_change":"2024-04-17T08:51:56.677002+0000","last_active":"2024-04-17T08:51:57.708007+0000","last_peered":"2024-04-17T08:51:57.708007+0000","last_clean":"2024-04-17T08:51:57.708007+0000","last_became_active":"2024-04-17T08:51:56.675209+0000","last_became_peered":"2024-04-17T08:51:56.675209+0000","last_unstale":"2024-04-17T08:51:57.708007+0000","last_undegraded":"2024-04-17T08:51:57.708007+0000","last_fullsized":"2024-04-17T08:51:57.708007+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:38:56.424013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707787+0000","last_change":"2024-04-17T08:51:55.666023+0000","last_active":"2024-04-17T08:51:57.707787+0000","last_peered":"2024-04-17T08:51:57.707787+0000","last_clean":"2024-04-17T08:51:57.707787+0000","last_became_active":"2024-04-17T08:51:55.665089+0000","last_became_peered":"2024-04-17T08:51:55.665089+0000","last_unstale":"2024-04-17T08:51:57.707787+0000","last_undegraded":"2024-04-17T08:51:57.707787+0000","last_fullsized":"2024-04-17T08:51:57.707787+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:59.267963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706929+0000","last_change":"2024-04-17T08:51:41.501146+0000","last_active":"2024-04-17T08:51:57.706929+0000","last_peered":"2024-04-17T08:51:57.706929+0000","last_clean":"2024-04-17T08:51:57.706929+0000","last_became_active":"2024-04-17T08:51:41.500704+0000","last_became_peered":"2024-04-17T08:51:41.500704+0000","last_unstale":"2024-04-17T08:51:57.706929+0000","last_undegraded":"2024-04-17T08:51:57.706929+0000","last_fullsized":"2024-04-17T08:51:57.706929+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:01.914122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338245+0000","last_change":"2024-04-17T08:51:56.689773+0000","last_active":"2024-04-17T08:51:58.338245+0000","last_peered":"2024-04-17T08:51:58.338245+0000","last_clean":"2024-04-17T08:51:58.338245+0000","last_became_active":"2024-04-17T08:51:56.689226+0000","last_became_peered":"2024-04-17T08:51:56.689226+0000","last_unstale":"2024-04-17T08:51:58.338245+0000","last_undegraded":"2024-04-17T08:51:58.338245+0000","last_fullsized":"2024-04-17T08:51:58.338245+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:32:00.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"253'4","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951046+0000","last_change":"2024-04-17T08:51:55.661967+0000","last_active":"2024-04-17T08:51:57.951046+0000","last_peered":"2024-04-17T08:51:57.951046+0000","last_clean":"2024-04-17T08:51:57.951046+0000","last_became_active":"2024-04-17T08:51:55.661233+0000","last_became_peered":"2024-04-17T08:51:55.661233+0000","last_unstale":"2024-04-17T08:51:57.951046+0000","last_undegraded":"2024-04-17T08:51:57.951046+0000","last_fullsized":"2024-04-17T08:51:57.951046+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:05:37.232722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672080+0000","last_change":"2024-04-17T08:51:41.517358+0000","last_active":"2024-04-17T08:51:56.672080+0000","last_peered":"2024-04-17T08:51:56.672080+0000","last_clean":"2024-04-17T08:51:56.672080+0000","last_became_active":"2024-04-17T08:51:41.517090+0000","last_became_peered":"2024-04-17T08:51:41.517090+0000","last_unstale":"2024-04-17T08:51:56.672080+0000","last_undegraded":"2024-04-17T08:51:56.672080+0000","last_fullsized":"2024-04-17T08:51:56.672080+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:48.083225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691415+0000","last_change":"2024-04-17T08:51:56.687011+0000","last_active":"2024-04-17T08:51:57.691415+0000","last_peered":"2024-04-17T08:51:57.691415+0000","last_clean":"2024-04-17T08:51:57.691415+0000","last_became_active":"2024-04-17T08:51:56.686740+0000","last_became_peered":"2024-04-17T08:51:56.686740+0000","last_unstale":"2024-04-17T08:51:57.691415+0000","last_undegraded":"2024-04-17T08:51:57.691415+0000","last_fullsized":"2024-04-17T08:51:57.691415+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:54.896615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950978+0000","last_change":"2024-04-17T08:51:41.503931+0000","last_active":"2024-04-17T08:51:57.950978+0000","last_peered":"2024-04-17T08:51:57.950978+0000","last_clean":"2024-04-17T08:51:57.950978+0000","last_became_active":"2024-04-17T08:51:41.502502+0000","last_became_peered":"2024-04-17T08:51:41.502502+0000","last_unstale":"2024-04-17T08:51:57.950978+0000","last_undegraded":"2024-04-17T08:51:57.950978+0000","last_fullsized":"2024-04-17T08:51:57.950978+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:26.370488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.339006+0000","last_change":"2024-04-17T08:51:40.478560+0000","last_active":"2024-04-17T08:51:58.339006+0000","last_peered":"2024-04-17T08:51:58.339006+0000","last_clean":"2024-04-17T08:51:58.339006+0000","last_became_active":"2024-04-17T08:51:40.478144+0000","last_became_peered":"2024-04-17T08:51:40.478144+0000","last_unstale":"2024-04-17T08:51:58.339006+0000","last_undegraded":"2024-04-17T08:51:58.339006+0000","last_fullsized":"2024-04-17T08:51:58.339006+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:37.672975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706991+0000","last_change":"2024-04-17T08:51:56.687973+0000","last_active":"2024-04-17T08:51:57.706991+0000","last_peered":"2024-04-17T08:51:57.706991+0000","last_clean":"2024-04-17T08:51:57.706991+0000","last_became_active":"2024-04-17T08:51:56.687606+0000","last_became_peered":"2024-04-17T08:51:56.687606+0000","last_unstale":"2024-04-17T08:51:57.706991+0000","last_undegraded":"2024-04-17T08:51:57.706991+0000","last_fullsized":"2024-04-17T08:51:57.706991+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:40.602103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688975+0000","last_change":"2024-04-17T08:51:41.505188+0000","last_active":"2024-04-17T08:51:57.688975+0000","last_peered":"2024-04-17T08:51:57.688975+0000","last_clean":"2024-04-17T08:51:57.688975+0000","last_became_active":"2024-04-17T08:51:41.504905+0000","last_became_peered":"2024-04-17T08:51:41.504905+0000","last_unstale":"2024-04-17T08:51:57.688975+0000","last_undegraded":"2024-04-17T08:51:57.688975+0000","last_fullsized":"2024-04-17T08:51:57.688975+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:11.712753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950539+0000","last_change":"2024-04-17T08:51:40.469433+0000","last_active":"2024-04-17T08:51:57.950539+0000","last_peered":"2024-04-17T08:51:57.950539+0000","last_clean":"2024-04-17T08:51:57.950539+0000","last_became_active":"2024-04-17T08:51:40.469071+0000","last_became_peered":"2024-04-17T08:51:40.469071+0000","last_unstale":"2024-04-17T08:51:57.950539+0000","last_undegraded":"2024-04-17T08:51:57.950539+0000","last_fullsized":"2024-04-17T08:51:57.950539+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:21.208592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950303+0000","last_change":"2024-04-17T08:51:56.700563+0000","last_active":"2024-04-17T08:51:57.950303+0000","last_peered":"2024-04-17T08:51:57.950303+0000","last_clean":"2024-04-17T08:51:57.950303+0000","last_became_active":"2024-04-17T08:51:56.690951+0000","last_became_peered":"2024-04-17T08:51:56.690951+0000","last_unstale":"2024-04-17T08:51:57.950303+0000","last_undegraded":"2024-04-17T08:51:57.950303+0000","last_fullsized":"2024-04-17T08:51:57.950303+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:30.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693787+0000","last_change":"2024-04-17T08:51:41.516473+0000","last_active":"2024-04-17T08:51:57.693787+0000","last_peered":"2024-04-17T08:51:57.693787+0000","last_clean":"2024-04-17T08:51:57.693787+0000","last_became_active":"2024-04-17T08:51:41.515870+0000","last_became_peered":"2024-04-17T08:51:41.515870+0000","last_unstale":"2024-04-17T08:51:57.693787+0000","last_undegraded":"2024-04-17T08:51:57.693787+0000","last_fullsized":"2024-04-17T08:51:57.693787+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:09.372045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950086+0000","last_change":"2024-04-17T08:51:40.472815+0000","last_active":"2024-04-17T08:51:57.950086+0000","last_peered":"2024-04-17T08:51:57.950086+0000","last_clean":"2024-04-17T08:51:57.950086+0000","last_became_active":"2024-04-17T08:51:40.470696+0000","last_became_peered":"2024-04-17T08:51:40.470696+0000","last_unstale":"2024-04-17T08:51:57.950086+0000","last_undegraded":"2024-04-17T08:51:57.950086+0000","last_fullsized":"2024-04-17T08:51:57.950086+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:32.004523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707742+0000","last_change":"2024-04-17T08:51:56.697594+0000","last_active":"2024-04-17T08:51:57.707742+0000","last_peered":"2024-04-17T08:51:57.707742+0000","last_clean":"2024-04-17T08:51:57.707742+0000","last_became_active":"2024-04-17T08:51:56.695529+0000","last_became_peered":"2024-04-17T08:51:56.695529+0000","last_unstale":"2024-04-17T08:51:57.707742+0000","last_undegraded":"2024-04-17T08:51:57.707742+0000","last_fullsized":"2024-04-17T08:51:57.707742+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:18.880218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689293+0000","last_change":"2024-04-17T08:51:41.497630+0000","last_active":"2024-04-17T08:51:57.689293+0000","last_peered":"2024-04-17T08:51:57.689293+0000","last_clean":"2024-04-17T08:51:57.689293+0000","last_became_active":"2024-04-17T08:51:41.497323+0000","last_became_peered":"2024-04-17T08:51:41.497323+0000","last_unstale":"2024-04-17T08:51:57.689293+0000","last_undegraded":"2024-04-17T08:51:57.689293+0000","last_fullsized":"2024-04-17T08:51:57.689293+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:45.677621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707147+0000","last_change":"2024-04-17T08:51:40.475306+0000","last_active":"2024-04-17T08:51:57.707147+0000","last_peered":"2024-04-17T08:51:57.707147+0000","last_clean":"2024-04-17T08:51:57.707147+0000","last_became_active":"2024-04-17T08:51:40.474409+0000","last_became_peered":"2024-04-17T08:51:40.474409+0000","last_unstale":"2024-04-17T08:51:57.707147+0000","last_undegraded":"2024-04-17T08:51:57.707147+0000","last_fullsized":"2024-04-17T08:51:57.707147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:17.301101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337934+0000","last_change":"2024-04-17T08:51:56.689149+0000","last_active":"2024-04-17T08:51:58.337934+0000","last_peered":"2024-04-17T08:51:58.337934+0000","last_clean":"2024-04-17T08:51:58.337934+0000","last_became_active":"2024-04-17T08:51:56.688411+0000","last_became_peered":"2024-04-17T08:51:56.688411+0000","last_unstale":"2024-04-17T08:51:58.337934+0000","last_undegraded":"2024-04-17T08:51:58.337934+0000","last_fullsized":"2024-04-17T08:51:58.337934+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:09.217990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.665848+0000","last_change":"2024-04-17T08:51:41.511131+0000","last_active":"2024-04-17T08:51:56.665848+0000","last_peered":"2024-04-17T08:51:56.665848+0000","last_clean":"2024-04-17T08:51:56.665848+0000","last_became_active":"2024-04-17T08:51:41.510481+0000","last_became_peered":"2024-04-17T08:51:41.510481+0000","last_unstale":"2024-04-17T08:51:56.665848+0000","last_undegraded":"2024-04-17T08:51:56.665848+0000","last_fullsized":"2024-04-17T08:51:56.665848+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:42.844488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":21,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668803+0000","last_change":"2024-04-17T08:51:40.476463+0000","last_active":"2024-04-17T08:51:56.668803+0000","last_peered":"2024-04-17T08:51:56.668803+0000","last_clean":"2024-04-17T08:51:56.668803+0000","last_became_active":"2024-04-17T08:51:40.476164+0000","last_became_peered":"2024-04-17T08:51:40.476164+0000","last_unstale":"2024-04-17T08:51:56.668803+0000","last_undegraded":"2024-04-17T08:51:56.668803+0000","last_fullsized":"2024-04-17T08:51:56.668803+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:15.997620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693773+0000","last_change":"2024-04-17T08:51:56.688954+0000","last_active":"2024-04-17T08:51:57.693773+0000","last_peered":"2024-04-17T08:51:57.693773+0000","last_clean":"2024-04-17T08:51:57.693773+0000","last_became_active":"2024-04-17T08:51:56.688684+0000","last_became_peered":"2024-04-17T08:51:56.688684+0000","last_unstale":"2024-04-17T08:51:57.693773+0000","last_undegraded":"2024-04-17T08:51:57.693773+0000","last_fullsized":"2024-04-17T08:51:57.693773+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:00.871425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948724+0000","last_change":"2024-04-17T08:51:41.507219+0000","last_active":"2024-04-17T08:51:57.948724+0000","last_peered":"2024-04-17T08:51:57.948724+0000","last_clean":"2024-04-17T08:51:57.948724+0000","last_became_active":"2024-04-17T08:51:41.504481+0000","last_became_peered":"2024-04-17T08:51:41.504481+0000","last_unstale":"2024-04-17T08:51:57.948724+0000","last_undegraded":"2024-04-17T08:51:57.948724+0000","last_fullsized":"2024-04-17T08:51:57.948724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:00.028124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336986+0000","last_change":"2024-04-17T08:51:40.485442+0000","last_active":"2024-04-17T08:51:58.336986+0000","last_peered":"2024-04-17T08:51:58.336986+0000","last_clean":"2024-04-17T08:51:58.336986+0000","last_became_active":"2024-04-17T08:51:40.485100+0000","last_became_peered":"2024-04-17T08:51:40.485100+0000","last_unstale":"2024-04-17T08:51:58.336986+0000","last_undegraded":"2024-04-17T08:51:58.336986+0000","last_fullsized":"2024-04-17T08:51:58.336986+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:18.376309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708066+0000","last_change":"2024-04-17T08:51:56.694841+0000","last_active":"2024-04-17T08:51:57.708066+0000","last_peered":"2024-04-17T08:51:57.708066+0000","last_clean":"2024-04-17T08:51:57.708066+0000","last_became_active":"2024-04-17T08:51:56.693519+0000","last_became_peered":"2024-04-17T08:51:56.693519+0000","last_unstale":"2024-04-17T08:51:57.708066+0000","last_undegraded":"2024-04-17T08:51:57.708066+0000","last_fullsized":"2024-04-17T08:51:57.708066+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:23.545377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668684+0000","last_change":"2024-04-17T08:51:41.504552+0000","last_active":"2024-04-17T08:51:56.668684+0000","last_peered":"2024-04-17T08:51:56.668684+0000","last_clean":"2024-04-17T08:51:56.668684+0000","last_became_active":"2024-04-17T08:51:41.504177+0000","last_became_peered":"2024-04-17T08:51:41.504177+0000","last_unstale":"2024-04-17T08:51:56.668684+0000","last_undegraded":"2024-04-17T08:51:56.668684+0000","last_fullsized":"2024-04-17T08:51:56.668684+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:31.634770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948969+0000","last_change":"2024-04-17T08:51:40.470752+0000","last_active":"2024-04-17T08:51:57.948969+0000","last_peered":"2024-04-17T08:51:57.948969+0000","last_clean":"2024-04-17T08:51:57.948969+0000","last_became_active":"2024-04-17T08:51:40.470440+0000","last_became_peered":"2024-04-17T08:51:40.470440+0000","last_unstale":"2024-04-17T08:51:57.948969+0000","last_undegraded":"2024-04-17T08:51:57.948969+0000","last_fullsized":"2024-04-17T08:51:57.948969+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.094101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948552+0000","last_change":"2024-04-17T08:51:56.702337+0000","last_active":"2024-04-17T08:51:57.948552+0000","last_peered":"2024-04-17T08:51:57.948552+0000","last_clean":"2024-04-17T08:51:57.948552+0000","last_became_active":"2024-04-17T08:51:56.701926+0000","last_became_peered":"2024-04-17T08:51:56.701926+0000","last_unstale":"2024-04-17T08:51:57.948552+0000","last_undegraded":"2024-04-17T08:51:57.948552+0000","last_fullsized":"2024-04-17T08:51:57.948552+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:28.103568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.670025+0000","last_change":"2024-04-17T08:51:41.505515+0000","last_active":"2024-04-17T08:51:56.670025+0000","last_peered":"2024-04-17T08:51:56.670025+0000","last_clean":"2024-04-17T08:51:56.670025+0000","last_became_active":"2024-04-17T08:51:41.504948+0000","last_became_peered":"2024-04-17T08:51:41.504948+0000","last_unstale":"2024-04-17T08:51:56.670025+0000","last_undegraded":"2024-04-17T08:51:56.670025+0000","last_fullsized":"2024-04-17T08:51:56.670025+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:44.205613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695652+0000","last_change":"2024-04-17T08:51:40.472194+0000","last_active":"2024-04-17T08:51:57.695652+0000","last_peered":"2024-04-17T08:51:57.695652+0000","last_clean":"2024-04-17T08:51:57.695652+0000","last_became_active":"2024-04-17T08:51:40.471867+0000","last_became_peered":"2024-04-17T08:51:40.471867+0000","last_unstale":"2024-04-17T08:51:57.695652+0000","last_undegraded":"2024-04-17T08:51:57.695652+0000","last_fullsized":"2024-04-17T08:51:57.695652+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:57.953636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336197+0000","last_change":"2024-04-17T08:51:56.691578+0000","last_active":"2024-04-17T08:51:58.336197+0000","last_peered":"2024-04-17T08:51:58.336197+0000","last_clean":"2024-04-17T08:51:58.336197+0000","last_became_active":"2024-04-17T08:51:56.691141+0000","last_became_peered":"2024-04-17T08:51:56.691141+0000","last_unstale":"2024-04-17T08:51:58.336197+0000","last_undegraded":"2024-04-17T08:51:58.336197+0000","last_fullsized":"2024-04-17T08:51:58.336197+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:54.896615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689835+0000","last_change":"2024-04-17T08:51:41.505182+0000","last_active":"2024-04-17T08:51:57.689835+0000","last_peered":"2024-04-17T08:51:57.689835+0000","last_clean":"2024-04-17T08:51:57.689835+0000","last_became_active":"2024-04-17T08:51:41.503770+0000","last_became_peered":"2024-04-17T08:51:41.503770+0000","last_unstale":"2024-04-17T08:51:57.689835+0000","last_undegraded":"2024-04-17T08:51:57.689835+0000","last_fullsized":"2024-04-17T08:51:57.689835+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950406+0000","last_change":"2024-04-17T08:51:40.478982+0000","last_active":"2024-04-17T08:51:57.950406+0000","last_peered":"2024-04-17T08:51:57.950406+0000","last_clean":"2024-04-17T08:51:57.950406+0000","last_became_active":"2024-04-17T08:51:40.478218+0000","last_became_peered":"2024-04-17T08:51:40.478218+0000","last_unstale":"2024-04-17T08:51:57.950406+0000","last_undegraded":"2024-04-17T08:51:57.950406+0000","last_fullsized":"2024-04-17T08:51:57.950406+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:48:34.166708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688968+0000","last_change":"2024-04-17T08:51:56.700642+0000","last_active":"2024-04-17T08:51:57.688968+0000","last_peered":"2024-04-17T08:51:57.688968+0000","last_clean":"2024-04-17T08:51:57.688968+0000","last_became_active":"2024-04-17T08:51:56.700352+0000","last_became_peered":"2024-04-17T08:51:56.700352+0000","last_unstale":"2024-04-17T08:51:57.688968+0000","last_undegraded":"2024-04-17T08:51:57.688968+0000","last_fullsized":"2024-04-17T08:51:57.688968+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:41.048823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950101+0000","last_change":"2024-04-17T08:51:41.488821+0000","last_active":"2024-04-17T08:51:57.950101+0000","last_peered":"2024-04-17T08:51:57.950101+0000","last_clean":"2024-04-17T08:51:57.950101+0000","last_became_active":"2024-04-17T08:51:41.488545+0000","last_became_peered":"2024-04-17T08:51:41.488545+0000","last_unstale":"2024-04-17T08:51:57.950101+0000","last_undegraded":"2024-04-17T08:51:57.950101+0000","last_fullsized":"2024-04-17T08:51:57.950101+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:42.844488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":23,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.671600+0000","last_change":"2024-04-17T08:51:40.484183+0000","last_active":"2024-04-17T08:51:56.671600+0000","last_peered":"2024-04-17T08:51:56.671600+0000","last_clean":"2024-04-17T08:51:56.671600+0000","last_became_active":"2024-04-17T08:51:40.483891+0000","last_became_peered":"2024-04-17T08:51:40.483891+0000","last_unstale":"2024-04-17T08:51:56.671600+0000","last_undegraded":"2024-04-17T08:51:56.671600+0000","last_fullsized":"2024-04-17T08:51:56.671600+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:41.230843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688876+0000","last_change":"2024-04-17T08:51:56.703392+0000","last_active":"2024-04-17T08:51:57.688876+0000","last_peered":"2024-04-17T08:51:57.688876+0000","last_clean":"2024-04-17T08:51:57.688876+0000","last_became_active":"2024-04-17T08:51:56.701149+0000","last_became_peered":"2024-04-17T08:51:56.701149+0000","last_unstale":"2024-04-17T08:51:57.688876+0000","last_undegraded":"2024-04-17T08:51:57.688876+0000","last_fullsized":"2024-04-17T08:51:57.688876+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:50.981331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338221+0000","last_change":"2024-04-17T08:51:41.515320+0000","last_active":"2024-04-17T08:51:58.338221+0000","last_peered":"2024-04-17T08:51:58.338221+0000","last_clean":"2024-04-17T08:51:58.338221+0000","last_became_active":"2024-04-17T08:51:41.513254+0000","last_became_peered":"2024-04-17T08:51:41.513254+0000","last_unstale":"2024-04-17T08:51:58.338221+0000","last_undegraded":"2024-04-17T08:51:58.338221+0000","last_fullsized":"2024-04-17T08:51:58.338221+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"250'8","reported_seq":28,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668176+0000","last_change":"2024-04-17T08:51:40.479746+0000","last_active":"2024-04-17T08:51:56.668176+0000","last_peered":"2024-04-17T08:51:56.668176+0000","last_clean":"2024-04-17T08:51:56.668176+0000","last_became_active":"2024-04-17T08:51:40.478655+0000","last_became_peered":"2024-04-17T08:51:40.478655+0000","last_unstale":"2024-04-17T08:51:56.668176+0000","last_undegraded":"2024-04-17T08:51:56.668176+0000","last_fullsized":"2024-04-17T08:51:56.668176+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:03.205317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699626+0000","last_change":"2024-04-17T08:51:56.694278+0000","last_active":"2024-04-17T08:51:56.699626+0000","last_peered":"2024-04-17T08:51:56.699626+0000","last_clean":"2024-04-17T08:51:56.699626+0000","last_became_active":"2024-04-17T08:51:56.693942+0000","last_became_peered":"2024-04-17T08:51:56.693942+0000","last_unstale":"2024-04-17T08:51:56.699626+0000","last_undegraded":"2024-04-17T08:51:56.699626+0000","last_fullsized":"2024-04-17T08:51:56.699626+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:08:14.536416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336523+0000","last_change":"2024-04-17T08:51:41.518673+0000","last_active":"2024-04-17T08:51:58.336523+0000","last_peered":"2024-04-17T08:51:58.336523+0000","last_clean":"2024-04-17T08:51:58.336523+0000","last_became_active":"2024-04-17T08:51:41.517797+0000","last_became_peered":"2024-04-17T08:51:41.517797+0000","last_unstale":"2024-04-17T08:51:58.336523+0000","last_undegraded":"2024-04-17T08:51:58.336523+0000","last_fullsized":"2024-04-17T08:51:58.336523+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:09.689564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":22,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672535+0000","last_change":"2024-04-17T08:51:40.481925+0000","last_active":"2024-04-17T08:51:56.672535+0000","last_peered":"2024-04-17T08:51:56.672535+0000","last_clean":"2024-04-17T08:51:56.672535+0000","last_became_active":"2024-04-17T08:51:40.481436+0000","last_became_peered":"2024-04-17T08:51:40.481436+0000","last_unstale":"2024-04-17T08:51:56.672535+0000","last_undegraded":"2024-04-17T08:51:56.672535+0000","last_fullsized":"2024-04-17T08:51:56.672535+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:04.739470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337570+0000","last_change":"2024-04-17T08:51:56.698176+0000","last_active":"2024-04-17T08:51:58.337570+0000","last_peered":"2024-04-17T08:51:58.337570+0000","last_clean":"2024-04-17T08:51:58.337570+0000","last_became_active":"2024-04-17T08:51:56.697866+0000","last_became_peered":"2024-04-17T08:51:56.697866+0000","last_unstale":"2024-04-17T08:51:58.337570+0000","last_undegraded":"2024-04-17T08:51:58.337570+0000","last_fullsized":"2024-04-17T08:51:58.337570+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:46.818453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666095+0000","last_change":"2024-04-17T08:51:41.508860+0000","last_active":"2024-04-17T08:51:56.666095+0000","last_peered":"2024-04-17T08:51:56.666095+0000","last_clean":"2024-04-17T08:51:56.666095+0000","last_became_active":"2024-04-17T08:51:41.508524+0000","last_became_peered":"2024-04-17T08:51:41.508524+0000","last_unstale":"2024-04-17T08:51:56.666095+0000","last_undegraded":"2024-04-17T08:51:56.666095+0000","last_fullsized":"2024-04-17T08:51:56.666095+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:23.647788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950300+0000","last_change":"2024-04-17T08:51:40.479291+0000","last_active":"2024-04-17T08:51:57.950300+0000","last_peered":"2024-04-17T08:51:57.950300+0000","last_clean":"2024-04-17T08:51:57.950300+0000","last_became_active":"2024-04-17T08:51:40.476885+0000","last_became_peered":"2024-04-17T08:51:40.476885+0000","last_unstale":"2024-04-17T08:51:57.950300+0000","last_undegraded":"2024-04-17T08:51:57.950300+0000","last_fullsized":"2024-04-17T08:51:57.950300+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:53.966461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338691+0000","last_change":"2024-04-17T08:51:56.690300+0000","last_active":"2024-04-17T08:51:58.338691+0000","last_peered":"2024-04-17T08:51:58.338691+0000","last_clean":"2024-04-17T08:51:58.338691+0000","last_became_active":"2024-04-17T08:51:56.690019+0000","last_became_peered":"2024-04-17T08:51:56.690019+0000","last_unstale":"2024-04-17T08:51:58.338691+0000","last_undegraded":"2024-04-17T08:51:58.338691+0000","last_fullsized":"2024-04-17T08:51:58.338691+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:57.582689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706642+0000","last_change":"2024-04-17T08:51:41.509196+0000","last_active":"2024-04-17T08:51:57.706642+0000","last_peered":"2024-04-17T08:51:57.706642+0000","last_clean":"2024-04-17T08:51:57.706642+0000","last_became_active":"2024-04-17T08:51:41.508917+0000","last_became_peered":"2024-04-17T08:51:41.508917+0000","last_unstale":"2024-04-17T08:51:57.706642+0000","last_undegraded":"2024-04-17T08:51:57.706642+0000","last_fullsized":"2024-04-17T08:51:57.706642+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:39.714011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337208+0000","last_change":"2024-04-17T08:51:40.483627+0000","last_active":"2024-04-17T08:51:58.337208+0000","last_peered":"2024-04-17T08:51:58.337208+0000","last_clean":"2024-04-17T08:51:58.337208+0000","last_became_active":"2024-04-17T08:51:40.482318+0000","last_became_peered":"2024-04-17T08:51:40.482318+0000","last_unstale":"2024-04-17T08:51:58.337208+0000","last_undegraded":"2024-04-17T08:51:58.337208+0000","last_fullsized":"2024-04-17T08:51:58.337208+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.761124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951410+0000","last_change":"2024-04-17T08:51:56.704670+0000","last_active":"2024-04-17T08:51:57.951410+0000","last_peered":"2024-04-17T08:51:57.951410+0000","last_clean":"2024-04-17T08:51:57.951410+0000","last_became_active":"2024-04-17T08:51:56.700800+0000","last_became_peered":"2024-04-17T08:51:56.700800+0000","last_unstale":"2024-04-17T08:51:57.951410+0000","last_undegraded":"2024-04-17T08:51:57.951410+0000","last_fullsized":"2024-04-17T08:51:57.951410+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:46.196158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336698+0000","last_change":"2024-04-17T08:51:41.517149+0000","last_active":"2024-04-17T08:51:58.336698+0000","last_peered":"2024-04-17T08:51:58.336698+0000","last_clean":"2024-04-17T08:51:58.336698+0000","last_became_active":"2024-04-17T08:51:41.516844+0000","last_became_peered":"2024-04-17T08:51:41.516844+0000","last_unstale":"2024-04-17T08:51:58.336698+0000","last_undegraded":"2024-04-17T08:51:58.336698+0000","last_fullsized":"2024-04-17T08:51:58.336698+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:37.293958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":30,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688523+0000","last_change":"2024-04-17T08:51:40.479653+0000","last_active":"2024-04-17T08:51:57.688523+0000","last_peered":"2024-04-17T08:51:57.688523+0000","last_clean":"2024-04-17T08:51:57.688523+0000","last_became_active":"2024-04-17T08:51:40.479107+0000","last_became_peered":"2024-04-17T08:51:40.479107+0000","last_unstale":"2024-04-17T08:51:57.688523+0000","last_undegraded":"2024-04-17T08:51:57.688523+0000","last_fullsized":"2024-04-17T08:51:57.688523+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:16.542860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.701673+0000","last_change":"2024-04-17T08:51:56.694906+0000","last_active":"2024-04-17T08:51:56.701673+0000","last_peered":"2024-04-17T08:51:56.701673+0000","last_clean":"2024-04-17T08:51:56.701673+0000","last_became_active":"2024-04-17T08:51:56.694487+0000","last_became_peered":"2024-04-17T08:51:56.694487+0000","last_unstale":"2024-04-17T08:51:56.701673+0000","last_undegraded":"2024-04-17T08:51:56.701673+0000","last_fullsized":"2024-04-17T08:51:56.701673+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:06.324844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707893+0000","last_change":"2024-04-17T08:51:41.510891+0000","last_active":"2024-04-17T08:51:57.707893+0000","last_peered":"2024-04-17T08:51:57.707893+0000","last_clean":"2024-04-17T08:51:57.707893+0000","last_became_active":"2024-04-17T08:51:41.510538+0000","last_became_peered":"2024-04-17T08:51:41.510538+0000","last_unstale":"2024-04-17T08:51:57.707893+0000","last_undegraded":"2024-04-17T08:51:57.707893+0000","last_fullsized":"2024-04-17T08:51:57.707893+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:00.189780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338428+0000","last_change":"2024-04-17T08:51:40.484060+0000","last_active":"2024-04-17T08:51:58.338428+0000","last_peered":"2024-04-17T08:51:58.338428+0000","last_clean":"2024-04-17T08:51:58.338428+0000","last_became_active":"2024-04-17T08:51:40.483692+0000","last_became_peered":"2024-04-17T08:51:40.483692+0000","last_unstale":"2024-04-17T08:51:58.338428+0000","last_undegraded":"2024-04-17T08:51:58.338428+0000","last_fullsized":"2024-04-17T08:51:58.338428+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:53.270393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706841+0000","last_change":"2024-04-17T08:51:56.673169+0000","last_active":"2024-04-17T08:51:57.706841+0000","last_peered":"2024-04-17T08:51:57.706841+0000","last_clean":"2024-04-17T08:51:57.706841+0000","last_became_active":"2024-04-17T08:51:56.671504+0000","last_became_peered":"2024-04-17T08:51:56.671504+0000","last_unstale":"2024-04-17T08:51:57.706841+0000","last_undegraded":"2024-04-17T08:51:57.706841+0000","last_fullsized":"2024-04-17T08:51:57.706841+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:50.045470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951158+0000","last_change":"2024-04-17T08:51:41.516369+0000","last_active":"2024-04-17T08:51:57.951158+0000","last_peered":"2024-04-17T08:51:57.951158+0000","last_clean":"2024-04-17T08:51:57.951158+0000","last_became_active":"2024-04-17T08:51:41.515919+0000","last_became_peered":"2024-04-17T08:51:41.515919+0000","last_unstale":"2024-04-17T08:51:57.951158+0000","last_undegraded":"2024-04-17T08:51:57.951158+0000","last_fullsized":"2024-04-17T08:51:57.951158+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:23.477382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950526+0000","last_change":"2024-04-17T08:51:40.466869+0000","last_active":"2024-04-17T08:51:57.950526+0000","last_peered":"2024-04-17T08:51:57.950526+0000","last_clean":"2024-04-17T08:51:57.950526+0000","last_became_active":"2024-04-17T08:51:40.465973+0000","last_became_peered":"2024-04-17T08:51:40.465973+0000","last_unstale":"2024-04-17T08:51:57.950526+0000","last_undegraded":"2024-04-17T08:51:57.950526+0000","last_fullsized":"2024-04-17T08:51:57.950526+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:57.022722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706996+0000","last_change":"2024-04-17T08:51:56.691468+0000","last_active":"2024-04-17T08:51:57.706996+0000","last_peered":"2024-04-17T08:51:57.706996+0000","last_clean":"2024-04-17T08:51:57.706996+0000","last_became_active":"2024-04-17T08:51:56.691059+0000","last_became_peered":"2024-04-17T08:51:56.691059+0000","last_unstale":"2024-04-17T08:51:57.706996+0000","last_undegraded":"2024-04-17T08:51:57.706996+0000","last_fullsized":"2024-04-17T08:51:57.706996+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:04:25.930760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.665883+0000","last_change":"2024-04-17T08:51:41.507082+0000","last_active":"2024-04-17T08:51:56.665883+0000","last_peered":"2024-04-17T08:51:56.665883+0000","last_clean":"2024-04-17T08:51:56.665883+0000","last_became_active":"2024-04-17T08:51:41.506812+0000","last_became_peered":"2024-04-17T08:51:41.506812+0000","last_unstale":"2024-04-17T08:51:56.665883+0000","last_undegraded":"2024-04-17T08:51:56.665883+0000","last_fullsized":"2024-04-17T08:51:56.665883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:34.182442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":21,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666753+0000","last_change":"2024-04-17T08:51:40.481626+0000","last_active":"2024-04-17T08:51:56.666753+0000","last_peered":"2024-04-17T08:51:56.666753+0000","last_clean":"2024-04-17T08:51:56.666753+0000","last_became_active":"2024-04-17T08:51:40.481268+0000","last_became_peered":"2024-04-17T08:51:40.481268+0000","last_unstale":"2024-04-17T08:51:56.666753+0000","last_undegraded":"2024-04-17T08:51:56.666753+0000","last_fullsized":"2024-04-17T08:51:56.666753+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:35.353434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707244+0000","last_change":"2024-04-17T08:51:56.700702+0000","last_active":"2024-04-17T08:51:57.707244+0000","last_peered":"2024-04-17T08:51:57.707244+0000","last_clean":"2024-04-17T08:51:57.707244+0000","last_became_active":"2024-04-17T08:51:56.700414+0000","last_became_peered":"2024-04-17T08:51:56.700414+0000","last_unstale":"2024-04-17T08:51:57.707244+0000","last_undegraded":"2024-04-17T08:51:57.707244+0000","last_fullsized":"2024-04-17T08:51:57.707244+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:49.077510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693843+0000","last_change":"2024-04-17T08:51:41.503769+0000","last_active":"2024-04-17T08:51:57.693843+0000","last_peered":"2024-04-17T08:51:57.693843+0000","last_clean":"2024-04-17T08:51:57.693843+0000","last_became_active":"2024-04-17T08:51:41.503051+0000","last_became_peered":"2024-04-17T08:51:41.503051+0000","last_unstale":"2024-04-17T08:51:57.693843+0000","last_undegraded":"2024-04-17T08:51:57.693843+0000","last_fullsized":"2024-04-17T08:51:57.693843+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:40.349134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689547+0000","last_change":"2024-04-17T08:51:40.482915+0000","last_active":"2024-04-17T08:51:57.689547+0000","last_peered":"2024-04-17T08:51:57.689547+0000","last_clean":"2024-04-17T08:51:57.689547+0000","last_became_active":"2024-04-17T08:51:40.482629+0000","last_became_peered":"2024-04-17T08:51:40.482629+0000","last_unstale":"2024-04-17T08:51:57.689547+0000","last_undegraded":"2024-04-17T08:51:57.689547+0000","last_fullsized":"2024-04-17T08:51:57.689547+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.115195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695935+0000","last_change":"2024-04-17T08:51:56.686110+0000","last_active":"2024-04-17T08:51:57.695935+0000","last_peered":"2024-04-17T08:51:57.695935+0000","last_clean":"2024-04-17T08:51:57.695935+0000","last_became_active":"2024-04-17T08:51:56.685801+0000","last_became_peered":"2024-04-17T08:51:56.685801+0000","last_unstale":"2024-04-17T08:51:57.695935+0000","last_undegraded":"2024-04-17T08:51:57.695935+0000","last_fullsized":"2024-04-17T08:51:57.695935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:04.357822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692445+0000","last_change":"2024-04-17T08:51:41.518701+0000","last_active":"2024-04-17T08:51:57.692445+0000","last_peered":"2024-04-17T08:51:57.692445+0000","last_clean":"2024-04-17T08:51:57.692445+0000","last_became_active":"2024-04-17T08:51:41.516124+0000","last_became_peered":"2024-04-17T08:51:41.516124+0000","last_unstale":"2024-04-17T08:51:57.692445+0000","last_undegraded":"2024-04-17T08:51:57.692445+0000","last_fullsized":"2024-04-17T08:51:57.692445+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950789+0000","last_change":"2024-04-17T08:51:40.483946+0000","last_active":"2024-04-17T08:51:57.950789+0000","last_peered":"2024-04-17T08:51:57.950789+0000","last_clean":"2024-04-17T08:51:57.950789+0000","last_became_active":"2024-04-17T08:51:40.483664+0000","last_became_peered":"2024-04-17T08:51:40.483664+0000","last_unstale":"2024-04-17T08:51:57.950789+0000","last_undegraded":"2024-04-17T08:51:57.950789+0000","last_fullsized":"2024-04-17T08:51:57.950789+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:55.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337273+0000","last_change":"2024-04-17T08:51:56.701992+0000","last_active":"2024-04-17T08:51:58.337273+0000","last_peered":"2024-04-17T08:51:58.337273+0000","last_clean":"2024-04-17T08:51:58.337273+0000","last_became_active":"2024-04-17T08:51:56.701643+0000","last_became_peered":"2024-04-17T08:51:56.701643+0000","last_unstale":"2024-04-17T08:51:58.337273+0000","last_undegraded":"2024-04-17T08:51:58.337273+0000","last_fullsized":"2024-04-17T08:51:58.337273+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:45.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672229+0000","last_change":"2024-04-17T08:51:41.519618+0000","last_active":"2024-04-17T08:51:56.672229+0000","last_peered":"2024-04-17T08:51:56.672229+0000","last_clean":"2024-04-17T08:51:56.672229+0000","last_became_active":"2024-04-17T08:51:41.519247+0000","last_became_peered":"2024-04-17T08:51:41.519247+0000","last_unstale":"2024-04-17T08:51:56.672229+0000","last_undegraded":"2024-04-17T08:51:56.672229+0000","last_fullsized":"2024-04-17T08:51:56.672229+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:32.995157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689635+0000","last_change":"2024-04-17T08:51:40.486177+0000","last_active":"2024-04-17T08:51:57.689635+0000","last_peered":"2024-04-17T08:51:57.689635+0000","last_clean":"2024-04-17T08:51:57.689635+0000","last_became_active":"2024-04-17T08:51:40.484934+0000","last_became_peered":"2024-04-17T08:51:40.484934+0000","last_unstale":"2024-04-17T08:51:57.689635+0000","last_undegraded":"2024-04-17T08:51:57.689635+0000","last_fullsized":"2024-04-17T08:51:57.689635+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:23.956487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337705+0000","last_change":"2024-04-17T08:51:56.698271+0000","last_active":"2024-04-17T08:51:58.337705+0000","last_peered":"2024-04-17T08:51:58.337705+0000","last_clean":"2024-04-17T08:51:58.337705+0000","last_became_active":"2024-04-17T08:51:56.697957+0000","last_became_peered":"2024-04-17T08:51:56.697957+0000","last_unstale":"2024-04-17T08:51:58.337705+0000","last_undegraded":"2024-04-17T08:51:58.337705+0000","last_fullsized":"2024-04-17T08:51:58.337705+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:48.178839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688452+0000","last_change":"2024-04-17T08:51:41.517111+0000","last_active":"2024-04-17T08:51:57.688452+0000","last_peered":"2024-04-17T08:51:57.688452+0000","last_clean":"2024-04-17T08:51:57.688452+0000","last_became_active":"2024-04-17T08:51:41.516756+0000","last_became_peered":"2024-04-17T08:51:41.516756+0000","last_unstale":"2024-04-17T08:51:57.688452+0000","last_undegraded":"2024-04-17T08:51:57.688452+0000","last_fullsized":"2024-04-17T08:51:57.688452+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:17:58.048828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666536+0000","last_change":"2024-04-17T08:51:40.479934+0000","last_active":"2024-04-17T08:51:56.666536+0000","last_peered":"2024-04-17T08:51:56.666536+0000","last_clean":"2024-04-17T08:51:56.666536+0000","last_became_active":"2024-04-17T08:51:40.479295+0000","last_became_peered":"2024-04-17T08:51:40.479295+0000","last_unstale":"2024-04-17T08:51:56.666536+0000","last_undegraded":"2024-04-17T08:51:56.666536+0000","last_fullsized":"2024-04-17T08:51:56.666536+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:40.469305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689935+0000","last_change":"2024-04-17T08:51:56.687789+0000","last_active":"2024-04-17T08:51:57.689935+0000","last_peered":"2024-04-17T08:51:57.689935+0000","last_clean":"2024-04-17T08:51:57.689935+0000","last_became_active":"2024-04-17T08:51:56.686913+0000","last_became_peered":"2024-04-17T08:51:56.686913+0000","last_unstale":"2024-04-17T08:51:57.689935+0000","last_undegraded":"2024-04-17T08:51:57.689935+0000","last_fullsized":"2024-04-17T08:51:57.689935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:28.101979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693898+0000","last_change":"2024-04-17T08:51:41.500141+0000","last_active":"2024-04-17T08:51:57.693898+0000","last_peered":"2024-04-17T08:51:57.693898+0000","last_clean":"2024-04-17T08:51:57.693898+0000","last_became_active":"2024-04-17T08:51:41.499513+0000","last_became_peered":"2024-04-17T08:51:41.499513+0000","last_unstale":"2024-04-17T08:51:57.693898+0000","last_undegraded":"2024-04-17T08:51:57.693898+0000","last_fullsized":"2024-04-17T08:51:57.693898+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336601+0000","last_change":"2024-04-17T08:51:40.478664+0000","last_active":"2024-04-17T08:51:58.336601+0000","last_peered":"2024-04-17T08:51:58.336601+0000","last_clean":"2024-04-17T08:51:58.336601+0000","last_became_active":"2024-04-17T08:51:40.477953+0000","last_became_peered":"2024-04-17T08:51:40.477953+0000","last_unstale":"2024-04-17T08:51:58.336601+0000","last_undegraded":"2024-04-17T08:51:58.336601+0000","last_fullsized":"2024-04-17T08:51:58.336601+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:54.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691971+0000","last_change":"2024-04-17T08:51:56.703540+0000","last_active":"2024-04-17T08:51:57.691971+0000","last_peered":"2024-04-17T08:51:57.691971+0000","last_clean":"2024-04-17T08:51:57.691971+0000","last_became_active":"2024-04-17T08:51:56.703086+0000","last_became_peered":"2024-04-17T08:51:56.703086+0000","last_unstale":"2024-04-17T08:51:57.691971+0000","last_undegraded":"2024-04-17T08:51:57.691971+0000","last_fullsized":"2024-04-17T08:51:57.691971+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:39.357009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338933+0000","last_change":"2024-04-17T08:51:41.515604+0000","last_active":"2024-04-17T08:51:58.338933+0000","last_peered":"2024-04-17T08:51:58.338933+0000","last_clean":"2024-04-17T08:51:58.338933+0000","last_became_active":"2024-04-17T08:51:41.514707+0000","last_became_peered":"2024-04-17T08:51:41.514707+0000","last_unstale":"2024-04-17T08:51:58.338933+0000","last_undegraded":"2024-04-17T08:51:58.338933+0000","last_fullsized":"2024-04-17T08:51:58.338933+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708325+0000","last_change":"2024-04-17T08:51:40.481823+0000","last_active":"2024-04-17T08:51:57.708325+0000","last_peered":"2024-04-17T08:51:57.708325+0000","last_clean":"2024-04-17T08:51:57.708325+0000","last_became_active":"2024-04-17T08:51:40.481302+0000","last_became_peered":"2024-04-17T08:51:40.481302+0000","last_unstale":"2024-04-17T08:51:57.708325+0000","last_undegraded":"2024-04-17T08:51:57.708325+0000","last_fullsized":"2024-04-17T08:51:57.708325+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:56.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336214+0000","last_change":"2024-04-17T08:51:56.689127+0000","last_active":"2024-04-17T08:51:58.336214+0000","last_peered":"2024-04-17T08:51:58.336214+0000","last_clean":"2024-04-17T08:51:58.336214+0000","last_became_active":"2024-04-17T08:51:56.687803+0000","last_became_peered":"2024-04-17T08:51:56.687803+0000","last_unstale":"2024-04-17T08:51:58.336214+0000","last_undegraded":"2024-04-17T08:51:58.336214+0000","last_fullsized":"2024-04-17T08:51:58.336214+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:39.357009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672007+0000","last_change":"2024-04-17T08:51:41.498889+0000","last_active":"2024-04-17T08:51:56.672007+0000","last_peered":"2024-04-17T08:51:56.672007+0000","last_clean":"2024-04-17T08:51:56.672007+0000","last_became_active":"2024-04-17T08:51:41.498622+0000","last_became_peered":"2024-04-17T08:51:41.498622+0000","last_unstale":"2024-04-17T08:51:56.672007+0000","last_undegraded":"2024-04-17T08:51:56.672007+0000","last_fullsized":"2024-04-17T08:51:56.672007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:42.312039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695001+0000","last_change":"2024-04-17T08:51:40.482579+0000","last_active":"2024-04-17T08:51:57.695001+0000","last_peered":"2024-04-17T08:51:57.695001+0000","last_clean":"2024-04-17T08:51:57.695001+0000","last_became_active":"2024-04-17T08:51:40.482255+0000","last_became_peered":"2024-04-17T08:51:40.482255+0000","last_unstale":"2024-04-17T08:51:57.695001+0000","last_undegraded":"2024-04-17T08:51:57.695001+0000","last_fullsized":"2024-04-17T08:51:57.695001+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:37.553047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699071+0000","last_change":"2024-04-17T08:51:56.693002+0000","last_active":"2024-04-17T08:51:56.699071+0000","last_peered":"2024-04-17T08:51:56.699071+0000","last_clean":"2024-04-17T08:51:56.699071+0000","last_became_active":"2024-04-17T08:51:56.692691+0000","last_became_peered":"2024-04-17T08:51:56.692691+0000","last_unstale":"2024-04-17T08:51:56.699071+0000","last_undegraded":"2024-04-17T08:51:56.699071+0000","last_fullsized":"2024-04-17T08:51:56.699071+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:23.337727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689212+0000","last_change":"2024-04-17T08:51:41.514216+0000","last_active":"2024-04-17T08:51:57.689212+0000","last_peered":"2024-04-17T08:51:57.689212+0000","last_clean":"2024-04-17T08:51:57.689212+0000","last_became_active":"2024-04-17T08:51:41.513937+0000","last_became_peered":"2024-04-17T08:51:41.513937+0000","last_unstale":"2024-04-17T08:51:57.689212+0000","last_undegraded":"2024-04-17T08:51:57.689212+0000","last_fullsized":"2024-04-17T08:51:57.689212+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:27.955059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667033+0000","last_change":"2024-04-17T08:51:40.482279+0000","last_active":"2024-04-17T08:51:56.667033+0000","last_peered":"2024-04-17T08:51:56.667033+0000","last_clean":"2024-04-17T08:51:56.667033+0000","last_became_active":"2024-04-17T08:51:40.482002+0000","last_became_peered":"2024-04-17T08:51:40.482002+0000","last_unstale":"2024-04-17T08:51:56.667033+0000","last_undegraded":"2024-04-17T08:51:56.667033+0000","last_fullsized":"2024-04-17T08:51:56.667033+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:36.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693959+0000","last_change":"2024-04-17T08:51:56.689516+0000","last_active":"2024-04-17T08:51:57.693959+0000","last_peered":"2024-04-17T08:51:57.693959+0000","last_clean":"2024-04-17T08:51:57.693959+0000","last_became_active":"2024-04-17T08:51:56.689237+0000","last_became_peered":"2024-04-17T08:51:56.689237+0000","last_unstale":"2024-04-17T08:51:57.693959+0000","last_undegraded":"2024-04-17T08:51:57.693959+0000","last_fullsized":"2024-04-17T08:51:57.693959+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:32.194887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689689+0000","last_change":"2024-04-17T08:51:41.505389+0000","last_active":"2024-04-17T08:51:57.689689+0000","last_peered":"2024-04-17T08:51:57.689689+0000","last_clean":"2024-04-17T08:51:57.689689+0000","last_became_active":"2024-04-17T08:51:41.503336+0000","last_became_peered":"2024-04-17T08:51:41.503336+0000","last_unstale":"2024-04-17T08:51:57.689689+0000","last_undegraded":"2024-04-17T08:51:57.689689+0000","last_fullsized":"2024-04-17T08:51:57.689689+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:16.345152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689671+0000","last_change":"2024-04-17T08:51:40.480656+0000","last_active":"2024-04-17T08:51:57.689671+0000","last_peered":"2024-04-17T08:51:57.689671+0000","last_clean":"2024-04-17T08:51:57.689671+0000","last_became_active":"2024-04-17T08:51:40.480386+0000","last_became_peered":"2024-04-17T08:51:40.480386+0000","last_unstale":"2024-04-17T08:51:57.689671+0000","last_undegraded":"2024-04-17T08:51:57.689671+0000","last_fullsized":"2024-04-17T08:51:57.689671+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.113174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692690+0000","last_change":"2024-04-17T08:51:56.684435+0000","last_active":"2024-04-17T08:51:57.692690+0000","last_peered":"2024-04-17T08:51:57.692690+0000","last_clean":"2024-04-17T08:51:57.692690+0000","last_became_active":"2024-04-17T08:51:56.684092+0000","last_became_peered":"2024-04-17T08:51:56.684092+0000","last_unstale":"2024-04-17T08:51:57.692690+0000","last_undegraded":"2024-04-17T08:51:57.692690+0000","last_fullsized":"2024-04-17T08:51:57.692690+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:53.241095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336729+0000","last_change":"2024-04-17T08:51:41.513350+0000","last_active":"2024-04-17T08:51:58.336729+0000","last_peered":"2024-04-17T08:51:58.336729+0000","last_clean":"2024-04-17T08:51:58.336729+0000","last_became_active":"2024-04-17T08:51:41.512524+0000","last_became_peered":"2024-04-17T08:51:41.512524+0000","last_unstale":"2024-04-17T08:51:58.336729+0000","last_undegraded":"2024-04-17T08:51:58.336729+0000","last_fullsized":"2024-04-17T08:51:58.336729+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:39.456694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337701+0000","last_change":"2024-04-17T08:51:40.480678+0000","last_active":"2024-04-17T08:51:58.337701+0000","last_peered":"2024-04-17T08:51:58.337701+0000","last_clean":"2024-04-17T08:51:58.337701+0000","last_became_active":"2024-04-17T08:51:40.480360+0000","last_became_peered":"2024-04-17T08:51:40.480360+0000","last_unstale":"2024-04-17T08:51:58.337701+0000","last_undegraded":"2024-04-17T08:51:58.337701+0000","last_fullsized":"2024-04-17T08:51:58.337701+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:50.970831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694977+0000","last_change":"2024-04-17T08:51:56.691780+0000","last_active":"2024-04-17T08:51:57.694977+0000","last_peered":"2024-04-17T08:51:57.694977+0000","last_clean":"2024-04-17T08:51:57.694977+0000","last_became_active":"2024-04-17T08:51:56.690914+0000","last_became_peered":"2024-04-17T08:51:56.690914+0000","last_unstale":"2024-04-17T08:51:57.694977+0000","last_undegraded":"2024-04-17T08:51:57.694977+0000","last_fullsized":"2024-04-17T08:51:57.694977+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.936737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693668+0000","last_change":"2024-04-17T08:51:41.501267+0000","last_active":"2024-04-17T08:51:57.693668+0000","last_peered":"2024-04-17T08:51:57.693668+0000","last_clean":"2024-04-17T08:51:57.693668+0000","last_became_active":"2024-04-17T08:51:41.500983+0000","last_became_peered":"2024-04-17T08:51:41.500983+0000","last_unstale":"2024-04-17T08:51:57.693668+0000","last_undegraded":"2024-04-17T08:51:57.693668+0000","last_fullsized":"2024-04-17T08:51:57.693668+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:53.940422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707306+0000","last_change":"2024-04-17T08:51:40.477982+0000","last_active":"2024-04-17T08:51:57.707306+0000","last_peered":"2024-04-17T08:51:57.707306+0000","last_clean":"2024-04-17T08:51:57.707306+0000","last_became_active":"2024-04-17T08:51:40.477717+0000","last_became_peered":"2024-04-17T08:51:40.477717+0000","last_unstale":"2024-04-17T08:51:57.707306+0000","last_undegraded":"2024-04-17T08:51:57.707306+0000","last_fullsized":"2024-04-17T08:51:57.707306+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:51.232388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.689690+0000","last_change":"2024-04-17T08:51:56.689119+0000","last_active":"2024-04-17T08:51:56.689690+0000","last_peered":"2024-04-17T08:51:56.689690+0000","last_clean":"2024-04-17T08:51:56.689690+0000","last_became_active":"2024-04-17T08:51:56.688451+0000","last_became_peered":"2024-04-17T08:51:56.688451+0000","last_unstale":"2024-04-17T08:51:56.689690+0000","last_undegraded":"2024-04-17T08:51:56.689690+0000","last_fullsized":"2024-04-17T08:51:56.689690+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:47.737093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693122+0000","last_change":"2024-04-17T08:51:41.504227+0000","last_active":"2024-04-17T08:51:57.693122+0000","last_peered":"2024-04-17T08:51:57.693122+0000","last_clean":"2024-04-17T08:51:57.693122+0000","last_became_active":"2024-04-17T08:51:41.503919+0000","last_became_peered":"2024-04-17T08:51:41.503919+0000","last_unstale":"2024-04-17T08:51:57.693122+0000","last_undegraded":"2024-04-17T08:51:57.693122+0000","last_fullsized":"2024-04-17T08:51:57.693122+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949128+0000","last_change":"2024-04-17T08:51:40.466461+0000","last_active":"2024-04-17T08:51:57.949128+0000","last_peered":"2024-04-17T08:51:57.949128+0000","last_clean":"2024-04-17T08:51:57.949128+0000","last_became_active":"2024-04-17T08:51:40.466050+0000","last_became_peered":"2024-04-17T08:51:40.466050+0000","last_unstale":"2024-04-17T08:51:57.949128+0000","last_undegraded":"2024-04-17T08:51:57.949128+0000","last_fullsized":"2024-04-17T08:51:57.949128+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:17.871122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692724+0000","last_change":"2024-04-17T08:51:56.688449+0000","last_active":"2024-04-17T08:51:57.692724+0000","last_peered":"2024-04-17T08:51:57.692724+0000","last_clean":"2024-04-17T08:51:57.692724+0000","last_became_active":"2024-04-17T08:51:56.687978+0000","last_became_peered":"2024-04-17T08:51:56.687978+0000","last_unstale":"2024-04-17T08:51:57.692724+0000","last_undegraded":"2024-04-17T08:51:57.692724+0000","last_fullsized":"2024-04-17T08:51:57.692724+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:51.427089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668942+0000","last_change":"2024-04-17T08:51:41.509452+0000","last_active":"2024-04-17T08:51:56.668942+0000","last_peered":"2024-04-17T08:51:56.668942+0000","last_clean":"2024-04-17T08:51:56.668942+0000","last_became_active":"2024-04-17T08:51:41.509160+0000","last_became_peered":"2024-04-17T08:51:41.509160+0000","last_unstale":"2024-04-17T08:51:56.668942+0000","last_undegraded":"2024-04-17T08:51:56.668942+0000","last_fullsized":"2024-04-17T08:51:56.668942+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:54.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949147+0000","last_change":"2024-04-17T08:51:40.481750+0000","last_active":"2024-04-17T08:51:57.949147+0000","last_peered":"2024-04-17T08:51:57.949147+0000","last_clean":"2024-04-17T08:51:57.949147+0000","last_became_active":"2024-04-17T08:51:40.481311+0000","last_became_peered":"2024-04-17T08:51:40.481311+0000","last_unstale":"2024-04-17T08:51:57.949147+0000","last_undegraded":"2024-04-17T08:51:57.949147+0000","last_fullsized":"2024-04-17T08:51:57.949147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:50.295078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336053+0000","last_change":"2024-04-17T08:51:56.696688+0000","last_active":"2024-04-17T08:51:58.336053+0000","last_peered":"2024-04-17T08:51:58.336053+0000","last_clean":"2024-04-17T08:51:58.336053+0000","last_became_active":"2024-04-17T08:51:56.696415+0000","last_became_peered":"2024-04-17T08:51:56.696415+0000","last_unstale":"2024-04-17T08:51:58.336053+0000","last_undegraded":"2024-04-17T08:51:58.336053+0000","last_fullsized":"2024-04-17T08:51:58.336053+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:53.876334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694536+0000","last_change":"2024-04-17T08:51:41.490585+0000","last_active":"2024-04-17T08:51:57.694536+0000","last_peered":"2024-04-17T08:51:57.694536+0000","last_clean":"2024-04-17T08:51:57.694536+0000","last_became_active":"2024-04-17T08:51:41.489827+0000","last_became_peered":"2024-04-17T08:51:41.489827+0000","last_unstale":"2024-04-17T08:51:57.694536+0000","last_undegraded":"2024-04-17T08:51:57.694536+0000","last_fullsized":"2024-04-17T08:51:57.694536+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:35.513878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951326+0000","last_change":"2024-04-17T08:51:40.482218+0000","last_active":"2024-04-17T08:51:57.951326+0000","last_peered":"2024-04-17T08:51:57.951326+0000","last_clean":"2024-04-17T08:51:57.951326+0000","last_became_active":"2024-04-17T08:51:40.481390+0000","last_became_peered":"2024-04-17T08:51:40.481390+0000","last_unstale":"2024-04-17T08:51:57.951326+0000","last_undegraded":"2024-04-17T08:51:57.951326+0000","last_fullsized":"2024-04-17T08:51:57.951326+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:41.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"236'376","reported_seq":1571,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.664689+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:51:56.664689+0000","last_peered":"2024-04-17T08:51:56.664689+0000","last_clean":"2024-04-17T08:51:56.664689+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:51:56.664689+0000","last_undegraded":"2024-04-17T08:51:56.664689+0000","last_fullsized":"2024-04-17T08:51:56.664689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":376,"ondisk_log_size":376,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543309,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57084,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314756,"statfs":{"total":96636764160,"available":96578310144,"internally_reserved":0,"allocated":3780608,"data_stored":2154396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543309,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60688,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311152,"statfs":{"total":96636764160,"available":96574619648,"internally_reserved":0,"allocated":3194880,"data_stored":1564604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543308,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30140,"kb_used_data":3116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341700,"statfs":{"total":96636764160,"available":96605900800,"internally_reserved":0,"allocated":3190784,"data_stored":1571083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543308,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3231744,"data_stored":1575700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543310,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52372,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319468,"statfs":{"total":96636764160,"available":96583135232,"internally_reserved":0,"allocated":3211264,"data_stored":1573859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543309,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52356,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319484,"statfs":{"total":96636764160,"available":96583151616,"internally_reserved":0,"allocated":3194880,"data_stored":1571173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543310,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64848,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306992,"statfs":{"total":96636764160,"available":96570359808,"internally_reserved":0,"allocated":3215360,"data_stored":1567645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543308,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61248,"kb_used_data":3700,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310592,"statfs":{"total":96636764160,"available":96574046208,"internally_reserved":0,"allocated":3788800,"data_stored":2161485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:52:01.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:52:01.431 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T08:52:01.432+0000 7fcf37800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:52:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.514+0000 7fbb0ca00640 1 Processor -- start 2024-04-17T08:52:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.515+0000 7fbb0ca00640 1 -- start start 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.516+0000 7fbb0ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb0806b8c0 0x7fbb080cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.516+0000 7fbb0ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb080cd4a0 con 0x7fbb08160280 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.516+0000 7fbb0ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb081607f0 con 0x7fbb08141ed0 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.516+0000 7fbb0ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb0815ce30 con 0x7fbb0806b8c0 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.517+0000 7fbaffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb08160280 0x7fbb0806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33380/0 (socket says 172.21.15.67:33380) 2024-04-17T08:52:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.517+0000 7fbaffe00640 1 -- 172.21.15.67:0/611434066 learned_addr learned my addr 172.21.15.67:0/611434066 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.517+0000 7fbb06800640 1 --2- 172.21.15.67:0/611434066 >> v2:172.21.15.187:3300/0 conn(0x7fbb0806b8c0 0x7fbb080cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.517+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151426266 0 0) 0x7fbb080cd4a0 con 0x7fbb08160280 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.517+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaec003520 con 0x7fbb08160280 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2547712818 0 0) 0x7fbaec003520 con 0x7fbb08160280 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 >> v2:172.21.15.187:3300/0 conn(0x7fbb0806b8c0 msgr2=0x7fbb080cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 --2- 172.21.15.67:0/611434066 >> v2:172.21.15.187:3300/0 conn(0x7fbb0806b8c0 0x7fbb080cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 >> v1:172.21.15.187:6789/0 conn(0x7fbb08141ed0 legacy=0x7fbb0815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb08160ad0 con 0x7fbb08160280 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbaf0001670 con 0x7fbb08160280 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 >> v1:172.21.15.67:6789/0 conn(0x7fbb08160280 legacy=0x7fbb0806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec003a70 0x7fbaec003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbaec004800 0x7fbaec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 0x7fbaec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0815ce30 con 0x7fbaec005210 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec00c140 con 0x7fbaec003a70 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbaec00c340 con 0x7fbaec004800 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb07200640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 0x7fbaec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.518+0000 7fbb06800640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec003a70 0x7fbaec003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbaffe00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbaec004800 0x7fbaec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbb07200640 1 -- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbaec004800 msgr2=0x7fbaec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbb07200640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbaec004800 0x7fbaec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbb07200640 1 -- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec003a70 msgr2=0x7fbaec003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbb07200640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec003a70 0x7fbaec003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbb07200640 1 -- 172.21.15.67:0/611434066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec003520 con 0x7fbaec005210 2024-04-17T08:52:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.519+0000 7fbaffe00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbaec004800 0x7fbaec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.520+0000 7fbb07200640 1 -- 172.21.15.67:0/611434066 reap_dead start 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.520+0000 7fbb07200640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 0x7fbaec00b9c0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fbb00001e50 tx=0x7fbaec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e917a837fbef096b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.520+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00001a70 con 0x7fbaec005210 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.520+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb00003bb0 con 0x7fbaec005210 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.520+0000 7fbb05e00640 1 -- 172.21.15.67:0/611434066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb000050d0 con 0x7fbaec005210 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 -- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 msgr2=0x7fbaec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 0x7fbaec00b9c0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fbb00001e50 tx=0x7fbaec005830 comp rx=0 tx=0).stop 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 -- 172.21.15.67:0/611434066 shutdown_connections 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/611434066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaec005210 0x7fbaec00b9c0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 -- 172.21.15.67:0/611434066 >> 172.21.15.67:0/611434066 conn(0x7fbb08067d10 msgr2=0x7fbb080b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:01.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 -- 172.21.15.67:0/611434066 shutdown_connections 2024-04-17T08:52:01.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.521+0000 7fbb0ca00640 1 -- 172.21.15.67:0/611434066 wait complete. 2024-04-17T08:52:01.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.523+0000 7fbb0ca00640 1 Processor -- start 2024-04-17T08:52:01.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.523+0000 7fbb0ca00640 1 -- start start 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec005210 0x7fbb0815f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb08136bc0 0x7fbb08140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0813f9b0 con 0x7fbb08141230 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb08161090 con 0x7fbaec005210 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0813fb30 con 0x7fbb08136bc0 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb07200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb07200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36980/0 (socket says 172.21.15.67:36980) 2024-04-17T08:52:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb07200640 1 -- 172.21.15.67:0/3999633965 learned_addr learned my addr 172.21.15.67:0/3999633965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbb06800640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec005210 0x7fbb0815f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.524+0000 7fbaffe00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb08136bc0 0x7fbb08140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 -- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb08136bc0 msgr2=0x7fbb08140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb08136bc0 0x7fbb08140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 -- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec005210 msgr2=0x7fbb0815f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec005210 0x7fbb0815f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 -- 172.21.15.67:0/3999633965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0815e290 con 0x7fbb08141230 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb07200640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fbb0002c060 tx=0x7fbb000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00007db0 con 0x7fbb08141230 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb00006d60 con 0x7fbb08141230 2024-04-17T08:52:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb000396a0 con 0x7fbb08141230 2024-04-17T08:52:01.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0815e570 con 0x7fbb08141230 2024-04-17T08:52:01.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.525+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0815eb30 con 0x7fbb08141230 2024-04-17T08:52:01.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.527+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb080692b0 con 0x7fbb08141230 2024-04-17T08:52:01.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.528+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fbb00039840 con 0x7fbb08141230 2024-04-17T08:52:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.529+0000 7fbafea00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 0x7fbae406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.529+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fbb000ae3c0 con 0x7fbb08141230 2024-04-17T08:52:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.529+0000 7fbb06800640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 0x7fbae406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:01.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.530+0000 7fbb06800640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 0x7fbae406f020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009990 tx=0x7fbaf8008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.533+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbb080692b0 con 0x7fbb08141230 2024-04-17T08:52:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.738+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb08142360 con 0x7fbae406cb60 2024-04-17T08:52:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.773+0000 7fbafea00640 1 -- 172.21.15.67:0/3999633965 <== mgr.8926 v2:172.21.15.187:6832/2123602634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491461 (secure 0 0 0) 0x7fbb08142360 con 0x7fbae406cb60 2024-04-17T08:52:01.775 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 msgr2=0x7fbae406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 0x7fbae406f020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009990 tx=0x7fbaf8008040 comp rx=0 tx=0).stop 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 msgr2=0x7fbb08143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fbb0002c060 tx=0x7fbb000014a0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 shutdown_connections 2024-04-17T08:52:01.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fbae406cb60 0x7fbae406f020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb08141230 0x7fbb08143960 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb08136bc0 0x7fbb08140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/3999633965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbaec005210 0x7fbb0815f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.784+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 >> 172.21.15.67:0/3999633965 conn(0x7fbb08067d10 msgr2=0x7fbb08140510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.785+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 shutdown_connections 2024-04-17T08:52:01.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:01.785+0000 7fbb0ca00640 1 -- 172.21.15.67:0/3999633965 wait complete. 2024-04-17T08:52:01.811 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-04-17T08:52:00.779953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":719,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444948,"kb_used_data":26180,"kb_used_omap":12,"kb_used_meta":418611,"kb_avail":754529772,"statfs":{"total":773094113280,"available":772638486528,"internally_reserved":0,"allocated":26808320,"data_stored":13739945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":428658224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002883"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691213+0000","last_change":"2024-04-17T08:51:56.690582+0000","last_active":"2024-04-17T08:51:57.691213+0000","last_peered":"2024-04-17T08:51:57.691213+0000","last_clean":"2024-04-17T08:51:57.691213+0000","last_became_active":"2024-04-17T08:51:56.685200+0000","last_became_peered":"2024-04-17T08:51:56.685200+0000","last_unstale":"2024-04-17T08:51:57.691213+0000","last_undegraded":"2024-04-17T08:51:57.691213+0000","last_fullsized":"2024-04-17T08:51:57.691213+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:55.532817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948905+0000","last_change":"2024-04-17T08:51:55.657713+0000","last_active":"2024-04-17T08:51:57.948905+0000","last_peered":"2024-04-17T08:51:57.948905+0000","last_clean":"2024-04-17T08:51:57.948905+0000","last_became_active":"2024-04-17T08:51:55.656393+0000","last_became_peered":"2024-04-17T08:51:55.656393+0000","last_unstale":"2024-04-17T08:51:57.948905+0000","last_undegraded":"2024-04-17T08:51:57.948905+0000","last_fullsized":"2024-04-17T08:51:57.948905+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:29.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667483+0000","last_change":"2024-04-17T08:51:41.515286+0000","last_active":"2024-04-17T08:51:56.667483+0000","last_peered":"2024-04-17T08:51:56.667483+0000","last_clean":"2024-04-17T08:51:56.667483+0000","last_became_active":"2024-04-17T08:51:41.515011+0000","last_became_peered":"2024-04-17T08:51:41.515011+0000","last_unstale":"2024-04-17T08:51:56.667483+0000","last_undegraded":"2024-04-17T08:51:56.667483+0000","last_fullsized":"2024-04-17T08:51:56.667483+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:27.187699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706752+0000","last_change":"2024-04-17T08:51:56.696931+0000","last_active":"2024-04-17T08:51:57.706752+0000","last_peered":"2024-04-17T08:51:57.706752+0000","last_clean":"2024-04-17T08:51:57.706752+0000","last_became_active":"2024-04-17T08:51:56.695570+0000","last_became_peered":"2024-04-17T08:51:56.695570+0000","last_unstale":"2024-04-17T08:51:57.706752+0000","last_undegraded":"2024-04-17T08:51:57.706752+0000","last_fullsized":"2024-04-17T08:51:57.706752+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:55:06.177080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706894+0000","last_change":"2024-04-17T08:51:55.666054+0000","last_active":"2024-04-17T08:51:57.706894+0000","last_peered":"2024-04-17T08:51:57.706894+0000","last_clean":"2024-04-17T08:51:57.706894+0000","last_became_active":"2024-04-17T08:51:55.665763+0000","last_became_peered":"2024-04-17T08:51:55.665763+0000","last_unstale":"2024-04-17T08:51:57.706894+0000","last_undegraded":"2024-04-17T08:51:57.706894+0000","last_fullsized":"2024-04-17T08:51:57.706894+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:30.014342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.669531+0000","last_change":"2024-04-17T08:51:41.505707+0000","last_active":"2024-04-17T08:51:56.669531+0000","last_peered":"2024-04-17T08:51:56.669531+0000","last_clean":"2024-04-17T08:51:56.669531+0000","last_became_active":"2024-04-17T08:51:41.505278+0000","last_became_peered":"2024-04-17T08:51:41.505278+0000","last_unstale":"2024-04-17T08:51:56.669531+0000","last_undegraded":"2024-04-17T08:51:56.669531+0000","last_fullsized":"2024-04-17T08:51:56.669531+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:17.019190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706947+0000","last_change":"2024-04-17T08:51:56.675494+0000","last_active":"2024-04-17T08:51:57.706947+0000","last_peered":"2024-04-17T08:51:57.706947+0000","last_clean":"2024-04-17T08:51:57.706947+0000","last_became_active":"2024-04-17T08:51:56.674933+0000","last_became_peered":"2024-04-17T08:51:56.674933+0000","last_unstale":"2024-04-17T08:51:57.706947+0000","last_undegraded":"2024-04-17T08:51:57.706947+0000","last_fullsized":"2024-04-17T08:51:57.706947+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:47:35.987183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688765+0000","last_change":"2024-04-17T08:51:55.662864+0000","last_active":"2024-04-17T08:51:57.688765+0000","last_peered":"2024-04-17T08:51:57.688765+0000","last_clean":"2024-04-17T08:51:57.688765+0000","last_became_active":"2024-04-17T08:51:55.662596+0000","last_became_peered":"2024-04-17T08:51:55.662596+0000","last_unstale":"2024-04-17T08:51:57.688765+0000","last_undegraded":"2024-04-17T08:51:57.688765+0000","last_fullsized":"2024-04-17T08:51:57.688765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:21.284333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950211+0000","last_change":"2024-04-17T08:51:41.506466+0000","last_active":"2024-04-17T08:51:57.950211+0000","last_peered":"2024-04-17T08:51:57.950211+0000","last_clean":"2024-04-17T08:51:57.950211+0000","last_became_active":"2024-04-17T08:51:41.506181+0000","last_became_peered":"2024-04-17T08:51:41.506181+0000","last_unstale":"2024-04-17T08:51:57.950211+0000","last_undegraded":"2024-04-17T08:51:57.950211+0000","last_fullsized":"2024-04-17T08:51:57.950211+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:41.111470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707235+0000","last_change":"2024-04-17T08:51:56.706215+0000","last_active":"2024-04-17T08:51:57.707235+0000","last_peered":"2024-04-17T08:51:57.707235+0000","last_clean":"2024-04-17T08:51:57.707235+0000","last_became_active":"2024-04-17T08:51:56.705962+0000","last_became_peered":"2024-04-17T08:51:56.705962+0000","last_unstale":"2024-04-17T08:51:57.707235+0000","last_undegraded":"2024-04-17T08:51:57.707235+0000","last_fullsized":"2024-04-17T08:51:57.707235+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:58.280367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688965+0000","last_change":"2024-04-17T08:51:55.658166+0000","last_active":"2024-04-17T08:51:57.688965+0000","last_peered":"2024-04-17T08:51:57.688965+0000","last_clean":"2024-04-17T08:51:57.688965+0000","last_became_active":"2024-04-17T08:51:55.657884+0000","last_became_peered":"2024-04-17T08:51:55.657884+0000","last_unstale":"2024-04-17T08:51:57.688965+0000","last_undegraded":"2024-04-17T08:51:57.688965+0000","last_fullsized":"2024-04-17T08:51:57.688965+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:08:44.277240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707823+0000","last_change":"2024-04-17T08:51:41.503459+0000","last_active":"2024-04-17T08:51:57.707823+0000","last_peered":"2024-04-17T08:51:57.707823+0000","last_clean":"2024-04-17T08:51:57.707823+0000","last_became_active":"2024-04-17T08:51:41.503139+0000","last_became_peered":"2024-04-17T08:51:41.503139+0000","last_unstale":"2024-04-17T08:51:57.707823+0000","last_undegraded":"2024-04-17T08:51:57.707823+0000","last_fullsized":"2024-04-17T08:51:57.707823+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:58.046980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695632+0000","last_change":"2024-04-17T08:51:56.692609+0000","last_active":"2024-04-17T08:51:57.695632+0000","last_peered":"2024-04-17T08:51:57.695632+0000","last_clean":"2024-04-17T08:51:57.695632+0000","last_became_active":"2024-04-17T08:51:56.692224+0000","last_became_peered":"2024-04-17T08:51:56.692224+0000","last_unstale":"2024-04-17T08:51:57.695632+0000","last_undegraded":"2024-04-17T08:51:57.695632+0000","last_fullsized":"2024-04-17T08:51:57.695632+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:57.428110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949954+0000","last_change":"2024-04-17T08:51:55.661940+0000","last_active":"2024-04-17T08:51:57.949954+0000","last_peered":"2024-04-17T08:51:57.949954+0000","last_clean":"2024-04-17T08:51:57.949954+0000","last_became_active":"2024-04-17T08:51:55.661441+0000","last_became_peered":"2024-04-17T08:51:55.661441+0000","last_unstale":"2024-04-17T08:51:57.949954+0000","last_undegraded":"2024-04-17T08:51:57.949954+0000","last_fullsized":"2024-04-17T08:51:57.949954+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:46.720133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694103+0000","last_change":"2024-04-17T08:51:41.515918+0000","last_active":"2024-04-17T08:51:57.694103+0000","last_peered":"2024-04-17T08:51:57.694103+0000","last_clean":"2024-04-17T08:51:57.694103+0000","last_became_active":"2024-04-17T08:51:41.515377+0000","last_became_peered":"2024-04-17T08:51:41.515377+0000","last_unstale":"2024-04-17T08:51:57.694103+0000","last_undegraded":"2024-04-17T08:51:57.694103+0000","last_fullsized":"2024-04-17T08:51:57.694103+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:51:05.859833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693100+0000","last_change":"2024-04-17T08:51:56.688318+0000","last_active":"2024-04-17T08:51:57.693100+0000","last_peered":"2024-04-17T08:51:57.693100+0000","last_clean":"2024-04-17T08:51:57.693100+0000","last_became_active":"2024-04-17T08:51:56.687757+0000","last_became_peered":"2024-04-17T08:51:56.687757+0000","last_unstale":"2024-04-17T08:51:57.693100+0000","last_undegraded":"2024-04-17T08:51:57.693100+0000","last_fullsized":"2024-04-17T08:51:57.693100+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:22.806186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337849+0000","last_change":"2024-04-17T08:51:55.667305+0000","last_active":"2024-04-17T08:51:58.337849+0000","last_peered":"2024-04-17T08:51:58.337849+0000","last_clean":"2024-04-17T08:51:58.337849+0000","last_became_active":"2024-04-17T08:51:55.666504+0000","last_became_peered":"2024-04-17T08:51:55.666504+0000","last_unstale":"2024-04-17T08:51:58.337849+0000","last_undegraded":"2024-04-17T08:51:58.337849+0000","last_fullsized":"2024-04-17T08:51:58.337849+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:59.854465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707307+0000","last_change":"2024-04-17T08:51:41.497092+0000","last_active":"2024-04-17T08:51:57.707307+0000","last_peered":"2024-04-17T08:51:57.707307+0000","last_clean":"2024-04-17T08:51:57.707307+0000","last_became_active":"2024-04-17T08:51:41.496813+0000","last_became_peered":"2024-04-17T08:51:41.496813+0000","last_unstale":"2024-04-17T08:51:57.707307+0000","last_undegraded":"2024-04-17T08:51:57.707307+0000","last_fullsized":"2024-04-17T08:51:57.707307+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:18:34.747527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950901+0000","last_change":"2024-04-17T08:51:56.701098+0000","last_active":"2024-04-17T08:51:57.950901+0000","last_peered":"2024-04-17T08:51:57.950901+0000","last_clean":"2024-04-17T08:51:57.950901+0000","last_became_active":"2024-04-17T08:51:56.700484+0000","last_became_peered":"2024-04-17T08:51:56.700484+0000","last_unstale":"2024-04-17T08:51:57.950901+0000","last_undegraded":"2024-04-17T08:51:57.950901+0000","last_fullsized":"2024-04-17T08:51:57.950901+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:07:30.742939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"254'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.705865+0000","last_change":"2024-04-17T08:51:55.657402+0000","last_active":"2024-04-17T08:51:57.705865+0000","last_peered":"2024-04-17T08:51:57.705865+0000","last_clean":"2024-04-17T08:51:57.705865+0000","last_became_active":"2024-04-17T08:51:55.657047+0000","last_became_peered":"2024-04-17T08:51:55.657047+0000","last_unstale":"2024-04-17T08:51:57.705865+0000","last_undegraded":"2024-04-17T08:51:57.705865+0000","last_fullsized":"2024-04-17T08:51:57.705865+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:45.089304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336367+0000","last_change":"2024-04-17T08:51:41.512828+0000","last_active":"2024-04-17T08:51:58.336367+0000","last_peered":"2024-04-17T08:51:58.336367+0000","last_clean":"2024-04-17T08:51:58.336367+0000","last_became_active":"2024-04-17T08:51:41.512495+0000","last_became_peered":"2024-04-17T08:51:41.512495+0000","last_unstale":"2024-04-17T08:51:58.336367+0000","last_undegraded":"2024-04-17T08:51:58.336367+0000","last_fullsized":"2024-04-17T08:51:58.336367+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:20:59.808221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707743+0000","last_change":"2024-04-17T08:51:56.677033+0000","last_active":"2024-04-17T08:51:57.707743+0000","last_peered":"2024-04-17T08:51:57.707743+0000","last_clean":"2024-04-17T08:51:57.707743+0000","last_became_active":"2024-04-17T08:51:56.675988+0000","last_became_peered":"2024-04-17T08:51:56.675988+0000","last_unstale":"2024-04-17T08:51:57.707743+0000","last_undegraded":"2024-04-17T08:51:57.707743+0000","last_fullsized":"2024-04-17T08:51:57.707743+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:09:56.307968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708209+0000","last_change":"2024-04-17T08:51:55.669776+0000","last_active":"2024-04-17T08:51:57.708209+0000","last_peered":"2024-04-17T08:51:57.708209+0000","last_clean":"2024-04-17T08:51:57.708209+0000","last_became_active":"2024-04-17T08:51:55.669487+0000","last_became_peered":"2024-04-17T08:51:55.669487+0000","last_unstale":"2024-04-17T08:51:57.708209+0000","last_undegraded":"2024-04-17T08:51:57.708209+0000","last_fullsized":"2024-04-17T08:51:57.708209+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:42:25.375909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692839+0000","last_change":"2024-04-17T08:51:41.505463+0000","last_active":"2024-04-17T08:51:57.692839+0000","last_peered":"2024-04-17T08:51:57.692839+0000","last_clean":"2024-04-17T08:51:57.692839+0000","last_became_active":"2024-04-17T08:51:41.505186+0000","last_became_peered":"2024-04-17T08:51:41.505186+0000","last_unstale":"2024-04-17T08:51:57.692839+0000","last_undegraded":"2024-04-17T08:51:57.692839+0000","last_fullsized":"2024-04-17T08:51:57.692839+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688806+0000","last_change":"2024-04-17T08:51:56.683133+0000","last_active":"2024-04-17T08:51:57.688806+0000","last_peered":"2024-04-17T08:51:57.688806+0000","last_clean":"2024-04-17T08:51:57.688806+0000","last_became_active":"2024-04-17T08:51:56.682850+0000","last_became_peered":"2024-04-17T08:51:56.682850+0000","last_unstale":"2024-04-17T08:51:57.688806+0000","last_undegraded":"2024-04-17T08:51:57.688806+0000","last_fullsized":"2024-04-17T08:51:57.688806+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:10:44.685939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692459+0000","last_change":"2024-04-17T08:51:55.656373+0000","last_active":"2024-04-17T08:51:57.692459+0000","last_peered":"2024-04-17T08:51:57.692459+0000","last_clean":"2024-04-17T08:51:57.692459+0000","last_became_active":"2024-04-17T08:51:55.655998+0000","last_became_peered":"2024-04-17T08:51:55.655998+0000","last_unstale":"2024-04-17T08:51:57.692459+0000","last_undegraded":"2024-04-17T08:51:57.692459+0000","last_fullsized":"2024-04-17T08:51:57.692459+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:55:32.690537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667478+0000","last_change":"2024-04-17T08:51:41.503091+0000","last_active":"2024-04-17T08:51:56.667478+0000","last_peered":"2024-04-17T08:51:56.667478+0000","last_clean":"2024-04-17T08:51:56.667478+0000","last_became_active":"2024-04-17T08:51:41.502796+0000","last_became_peered":"2024-04-17T08:51:41.502796+0000","last_unstale":"2024-04-17T08:51:56.667478+0000","last_undegraded":"2024-04-17T08:51:56.667478+0000","last_fullsized":"2024-04-17T08:51:56.667478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:44.277963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948657+0000","last_change":"2024-04-17T08:51:56.696366+0000","last_active":"2024-04-17T08:51:57.948657+0000","last_peered":"2024-04-17T08:51:57.948657+0000","last_clean":"2024-04-17T08:51:57.948657+0000","last_became_active":"2024-04-17T08:51:56.681654+0000","last_became_peered":"2024-04-17T08:51:56.681654+0000","last_unstale":"2024-04-17T08:51:57.948657+0000","last_undegraded":"2024-04-17T08:51:57.948657+0000","last_fullsized":"2024-04-17T08:51:57.948657+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:05:30.582209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.664275+0000","last_change":"2024-04-17T08:51:55.668838+0000","last_active":"2024-04-17T08:51:56.664275+0000","last_peered":"2024-04-17T08:51:56.664275+0000","last_clean":"2024-04-17T08:51:56.664275+0000","last_became_active":"2024-04-17T08:51:55.668484+0000","last_became_peered":"2024-04-17T08:51:55.668484+0000","last_unstale":"2024-04-17T08:51:56.664275+0000","last_undegraded":"2024-04-17T08:51:56.664275+0000","last_fullsized":"2024-04-17T08:51:56.664275+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:58.622251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338671+0000","last_change":"2024-04-17T08:51:41.517488+0000","last_active":"2024-04-17T08:51:58.338671+0000","last_peered":"2024-04-17T08:51:58.338671+0000","last_clean":"2024-04-17T08:51:58.338671+0000","last_became_active":"2024-04-17T08:51:41.517224+0000","last_became_peered":"2024-04-17T08:51:41.517224+0000","last_unstale":"2024-04-17T08:51:58.338671+0000","last_undegraded":"2024-04-17T08:51:58.338671+0000","last_fullsized":"2024-04-17T08:51:58.338671+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.702094+0000","last_change":"2024-04-17T08:51:56.698483+0000","last_active":"2024-04-17T08:51:56.702094+0000","last_peered":"2024-04-17T08:51:56.702094+0000","last_clean":"2024-04-17T08:51:56.702094+0000","last_became_active":"2024-04-17T08:51:56.698188+0000","last_became_peered":"2024-04-17T08:51:56.698188+0000","last_unstale":"2024-04-17T08:51:56.702094+0000","last_undegraded":"2024-04-17T08:51:56.702094+0000","last_fullsized":"2024-04-17T08:51:56.702094+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:19.239448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691954+0000","last_change":"2024-04-17T08:51:55.649899+0000","last_active":"2024-04-17T08:51:57.691954+0000","last_peered":"2024-04-17T08:51:57.691954+0000","last_clean":"2024-04-17T08:51:57.691954+0000","last_became_active":"2024-04-17T08:51:55.649579+0000","last_became_peered":"2024-04-17T08:51:55.649579+0000","last_unstale":"2024-04-17T08:51:57.691954+0000","last_undegraded":"2024-04-17T08:51:57.691954+0000","last_fullsized":"2024-04-17T08:51:57.691954+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:49:26.416866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689480+0000","last_change":"2024-04-17T08:51:41.520584+0000","last_active":"2024-04-17T08:51:57.689480+0000","last_peered":"2024-04-17T08:51:57.689480+0000","last_clean":"2024-04-17T08:51:57.689480+0000","last_became_active":"2024-04-17T08:51:41.520134+0000","last_became_peered":"2024-04-17T08:51:41.520134+0000","last_unstale":"2024-04-17T08:51:57.689480+0000","last_undegraded":"2024-04-17T08:51:57.689480+0000","last_fullsized":"2024-04-17T08:51:57.689480+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:55:25.042374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699772+0000","last_change":"2024-04-17T08:51:56.694171+0000","last_active":"2024-04-17T08:51:56.699772+0000","last_peered":"2024-04-17T08:51:56.699772+0000","last_clean":"2024-04-17T08:51:56.699772+0000","last_became_active":"2024-04-17T08:51:56.693872+0000","last_became_peered":"2024-04-17T08:51:56.693872+0000","last_unstale":"2024-04-17T08:51:56.699772+0000","last_undegraded":"2024-04-17T08:51:56.699772+0000","last_fullsized":"2024-04-17T08:51:56.699772+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:49.406094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693922+0000","last_change":"2024-04-17T08:51:55.659521+0000","last_active":"2024-04-17T08:51:57.693922+0000","last_peered":"2024-04-17T08:51:57.693922+0000","last_clean":"2024-04-17T08:51:57.693922+0000","last_became_active":"2024-04-17T08:51:55.658645+0000","last_became_peered":"2024-04-17T08:51:55.658645+0000","last_unstale":"2024-04-17T08:51:57.693922+0000","last_undegraded":"2024-04-17T08:51:57.693922+0000","last_fullsized":"2024-04-17T08:51:57.693922+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:52.153579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695970+0000","last_change":"2024-04-17T08:51:41.518003+0000","last_active":"2024-04-17T08:51:57.695970+0000","last_peered":"2024-04-17T08:51:57.695970+0000","last_clean":"2024-04-17T08:51:57.695970+0000","last_became_active":"2024-04-17T08:51:41.517732+0000","last_became_peered":"2024-04-17T08:51:41.517732+0000","last_unstale":"2024-04-17T08:51:57.695970+0000","last_undegraded":"2024-04-17T08:51:57.695970+0000","last_fullsized":"2024-04-17T08:51:57.695970+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:55.524149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695361+0000","last_change":"2024-04-17T08:51:56.692234+0000","last_active":"2024-04-17T08:51:57.695361+0000","last_peered":"2024-04-17T08:51:57.695361+0000","last_clean":"2024-04-17T08:51:57.695361+0000","last_became_active":"2024-04-17T08:51:56.691508+0000","last_became_peered":"2024-04-17T08:51:56.691508+0000","last_unstale":"2024-04-17T08:51:57.695361+0000","last_undegraded":"2024-04-17T08:51:57.695361+0000","last_fullsized":"2024-04-17T08:51:57.695361+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:19.402584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707656+0000","last_change":"2024-04-17T08:51:55.670726+0000","last_active":"2024-04-17T08:51:57.707656+0000","last_peered":"2024-04-17T08:51:57.707656+0000","last_clean":"2024-04-17T08:51:57.707656+0000","last_became_active":"2024-04-17T08:51:55.668578+0000","last_became_peered":"2024-04-17T08:51:55.668578+0000","last_unstale":"2024-04-17T08:51:57.707656+0000","last_undegraded":"2024-04-17T08:51:57.707656+0000","last_fullsized":"2024-04-17T08:51:57.707656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:32:29.845006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337245+0000","last_change":"2024-04-17T08:51:41.514813+0000","last_active":"2024-04-17T08:51:58.337245+0000","last_peered":"2024-04-17T08:51:58.337245+0000","last_clean":"2024-04-17T08:51:58.337245+0000","last_became_active":"2024-04-17T08:51:41.512779+0000","last_became_peered":"2024-04-17T08:51:41.512779+0000","last_unstale":"2024-04-17T08:51:58.337245+0000","last_undegraded":"2024-04-17T08:51:58.337245+0000","last_fullsized":"2024-04-17T08:51:58.337245+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:55:37.986856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707510+0000","last_change":"2024-04-17T08:51:56.674702+0000","last_active":"2024-04-17T08:51:57.707510+0000","last_peered":"2024-04-17T08:51:57.707510+0000","last_clean":"2024-04-17T08:51:57.707510+0000","last_became_active":"2024-04-17T08:51:56.673432+0000","last_became_peered":"2024-04-17T08:51:56.673432+0000","last_unstale":"2024-04-17T08:51:57.707510+0000","last_undegraded":"2024-04-17T08:51:57.707510+0000","last_fullsized":"2024-04-17T08:51:57.707510+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:17:10.247987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"253'1","reported_seq":13,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.711806+0000","last_change":"2024-04-17T08:51:55.671864+0000","last_active":"2024-04-17T08:51:56.711806+0000","last_peered":"2024-04-17T08:51:56.711806+0000","last_clean":"2024-04-17T08:51:56.711806+0000","last_became_active":"2024-04-17T08:51:55.671587+0000","last_became_peered":"2024-04-17T08:51:55.671587+0000","last_unstale":"2024-04-17T08:51:56.711806+0000","last_undegraded":"2024-04-17T08:51:56.711806+0000","last_fullsized":"2024-04-17T08:51:56.711806+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:25:29.119928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668474+0000","last_change":"2024-04-17T08:51:41.507865+0000","last_active":"2024-04-17T08:51:56.668474+0000","last_peered":"2024-04-17T08:51:56.668474+0000","last_clean":"2024-04-17T08:51:56.668474+0000","last_became_active":"2024-04-17T08:51:41.507622+0000","last_became_peered":"2024-04-17T08:51:41.507622+0000","last_unstale":"2024-04-17T08:51:56.668474+0000","last_undegraded":"2024-04-17T08:51:56.668474+0000","last_fullsized":"2024-04-17T08:51:56.668474+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.174140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.687058+0000","last_change":"2024-04-17T08:51:56.680740+0000","last_active":"2024-04-17T08:51:56.687058+0000","last_peered":"2024-04-17T08:51:56.687058+0000","last_clean":"2024-04-17T08:51:56.687058+0000","last_became_active":"2024-04-17T08:51:56.680396+0000","last_became_peered":"2024-04-17T08:51:56.680396+0000","last_unstale":"2024-04-17T08:51:56.687058+0000","last_undegraded":"2024-04-17T08:51:56.687058+0000","last_fullsized":"2024-04-17T08:51:56.687058+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:38.087816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.713513+0000","last_change":"2024-04-17T08:51:55.661494+0000","last_active":"2024-04-17T08:51:56.713513+0000","last_peered":"2024-04-17T08:51:56.713513+0000","last_clean":"2024-04-17T08:51:56.713513+0000","last_became_active":"2024-04-17T08:51:55.660821+0000","last_became_peered":"2024-04-17T08:51:55.660821+0000","last_unstale":"2024-04-17T08:51:56.713513+0000","last_undegraded":"2024-04-17T08:51:56.713513+0000","last_fullsized":"2024-04-17T08:51:56.713513+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.789856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666243+0000","last_change":"2024-04-17T08:51:41.501853+0000","last_active":"2024-04-17T08:51:56.666243+0000","last_peered":"2024-04-17T08:51:56.666243+0000","last_clean":"2024-04-17T08:51:56.666243+0000","last_became_active":"2024-04-17T08:51:41.501575+0000","last_became_peered":"2024-04-17T08:51:41.501575+0000","last_unstale":"2024-04-17T08:51:56.666243+0000","last_undegraded":"2024-04-17T08:51:56.666243+0000","last_fullsized":"2024-04-17T08:51:56.666243+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:00.028124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708068+0000","last_change":"2024-04-17T08:51:56.679445+0000","last_active":"2024-04-17T08:51:57.708068+0000","last_peered":"2024-04-17T08:51:57.708068+0000","last_clean":"2024-04-17T08:51:57.708068+0000","last_became_active":"2024-04-17T08:51:56.676586+0000","last_became_peered":"2024-04-17T08:51:56.676586+0000","last_unstale":"2024-04-17T08:51:57.708068+0000","last_undegraded":"2024-04-17T08:51:57.708068+0000","last_fullsized":"2024-04-17T08:51:57.708068+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:45:31.719505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692290+0000","last_change":"2024-04-17T08:51:55.664551+0000","last_active":"2024-04-17T08:51:57.692290+0000","last_peered":"2024-04-17T08:51:57.692290+0000","last_clean":"2024-04-17T08:51:57.692290+0000","last_became_active":"2024-04-17T08:51:55.664267+0000","last_became_peered":"2024-04-17T08:51:55.664267+0000","last_unstale":"2024-04-17T08:51:57.692290+0000","last_undegraded":"2024-04-17T08:51:57.692290+0000","last_fullsized":"2024-04-17T08:51:57.692290+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:21:27.768335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948708+0000","last_change":"2024-04-17T08:51:41.513490+0000","last_active":"2024-04-17T08:51:57.948708+0000","last_peered":"2024-04-17T08:51:57.948708+0000","last_clean":"2024-04-17T08:51:57.948708+0000","last_became_active":"2024-04-17T08:51:41.513192+0000","last_became_peered":"2024-04-17T08:51:41.513192+0000","last_unstale":"2024-04-17T08:51:57.948708+0000","last_undegraded":"2024-04-17T08:51:57.948708+0000","last_fullsized":"2024-04-17T08:51:57.948708+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:47.830902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948935+0000","last_change":"2024-04-17T08:51:56.702728+0000","last_active":"2024-04-17T08:51:57.948935+0000","last_peered":"2024-04-17T08:51:57.948935+0000","last_clean":"2024-04-17T08:51:57.948935+0000","last_became_active":"2024-04-17T08:51:56.689917+0000","last_became_peered":"2024-04-17T08:51:56.689917+0000","last_unstale":"2024-04-17T08:51:57.948935+0000","last_undegraded":"2024-04-17T08:51:57.948935+0000","last_fullsized":"2024-04-17T08:51:57.948935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:09.401482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"253'8","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.712919+0000","last_change":"2024-04-17T08:51:55.661658+0000","last_active":"2024-04-17T08:51:56.712919+0000","last_peered":"2024-04-17T08:51:56.712919+0000","last_clean":"2024-04-17T08:51:56.712919+0000","last_became_active":"2024-04-17T08:51:55.661387+0000","last_became_peered":"2024-04-17T08:51:55.661387+0000","last_unstale":"2024-04-17T08:51:56.712919+0000","last_undegraded":"2024-04-17T08:51:56.712919+0000","last_fullsized":"2024-04-17T08:51:56.712919+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:59.234393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695578+0000","last_change":"2024-04-17T08:51:41.516221+0000","last_active":"2024-04-17T08:51:57.695578+0000","last_peered":"2024-04-17T08:51:57.695578+0000","last_clean":"2024-04-17T08:51:57.695578+0000","last_became_active":"2024-04-17T08:51:41.515896+0000","last_became_peered":"2024-04-17T08:51:41.515896+0000","last_unstale":"2024-04-17T08:51:57.695578+0000","last_undegraded":"2024-04-17T08:51:57.695578+0000","last_fullsized":"2024-04-17T08:51:57.695578+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:22.484668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948492+0000","last_change":"2024-04-17T08:51:56.701571+0000","last_active":"2024-04-17T08:51:57.948492+0000","last_peered":"2024-04-17T08:51:57.948492+0000","last_clean":"2024-04-17T08:51:57.948492+0000","last_became_active":"2024-04-17T08:51:56.698942+0000","last_became_peered":"2024-04-17T08:51:56.698942+0000","last_unstale":"2024-04-17T08:51:57.948492+0000","last_undegraded":"2024-04-17T08:51:57.948492+0000","last_fullsized":"2024-04-17T08:51:57.948492+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:49.809439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666798+0000","last_change":"2024-04-17T08:51:55.667334+0000","last_active":"2024-04-17T08:51:56.666798+0000","last_peered":"2024-04-17T08:51:56.666798+0000","last_clean":"2024-04-17T08:51:56.666798+0000","last_became_active":"2024-04-17T08:51:55.667038+0000","last_became_peered":"2024-04-17T08:51:55.667038+0000","last_unstale":"2024-04-17T08:51:56.666798+0000","last_undegraded":"2024-04-17T08:51:56.666798+0000","last_fullsized":"2024-04-17T08:51:56.666798+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:27:29.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.669781+0000","last_change":"2024-04-17T08:51:41.501545+0000","last_active":"2024-04-17T08:51:56.669781+0000","last_peered":"2024-04-17T08:51:56.669781+0000","last_clean":"2024-04-17T08:51:56.669781+0000","last_became_active":"2024-04-17T08:51:41.500258+0000","last_became_peered":"2024-04-17T08:51:41.500258+0000","last_unstale":"2024-04-17T08:51:56.669781+0000","last_undegraded":"2024-04-17T08:51:56.669781+0000","last_fullsized":"2024-04-17T08:51:56.669781+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:21:24.809010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699045+0000","last_change":"2024-04-17T08:51:56.691565+0000","last_active":"2024-04-17T08:51:56.699045+0000","last_peered":"2024-04-17T08:51:56.699045+0000","last_clean":"2024-04-17T08:51:56.699045+0000","last_became_active":"2024-04-17T08:51:56.686842+0000","last_became_peered":"2024-04-17T08:51:56.686842+0000","last_unstale":"2024-04-17T08:51:56.699045+0000","last_undegraded":"2024-04-17T08:51:56.699045+0000","last_fullsized":"2024-04-17T08:51:56.699045+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:53:43.988578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689124+0000","last_change":"2024-04-17T08:51:55.662235+0000","last_active":"2024-04-17T08:51:57.689124+0000","last_peered":"2024-04-17T08:51:57.689124+0000","last_clean":"2024-04-17T08:51:57.689124+0000","last_became_active":"2024-04-17T08:51:55.660563+0000","last_became_peered":"2024-04-17T08:51:55.660563+0000","last_unstale":"2024-04-17T08:51:57.689124+0000","last_undegraded":"2024-04-17T08:51:57.689124+0000","last_fullsized":"2024-04-17T08:51:57.689124+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:57.995854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693586+0000","last_change":"2024-04-17T08:51:41.505156+0000","last_active":"2024-04-17T08:51:57.693586+0000","last_peered":"2024-04-17T08:51:57.693586+0000","last_clean":"2024-04-17T08:51:57.693586+0000","last_became_active":"2024-04-17T08:51:41.503363+0000","last_became_peered":"2024-04-17T08:51:41.503363+0000","last_unstale":"2024-04-17T08:51:57.693586+0000","last_undegraded":"2024-04-17T08:51:57.693586+0000","last_fullsized":"2024-04-17T08:51:57.693586+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:31:28.800799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950630+0000","last_change":"2024-04-17T08:51:56.710667+0000","last_active":"2024-04-17T08:51:57.950630+0000","last_peered":"2024-04-17T08:51:57.950630+0000","last_clean":"2024-04-17T08:51:57.950630+0000","last_became_active":"2024-04-17T08:51:56.703014+0000","last_became_peered":"2024-04-17T08:51:56.703014+0000","last_unstale":"2024-04-17T08:51:57.950630+0000","last_undegraded":"2024-04-17T08:51:57.950630+0000","last_fullsized":"2024-04-17T08:51:57.950630+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:52.613169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"253'2","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693148+0000","last_change":"2024-04-17T08:51:55.661978+0000","last_active":"2024-04-17T08:51:57.693148+0000","last_peered":"2024-04-17T08:51:57.693148+0000","last_clean":"2024-04-17T08:51:57.693148+0000","last_became_active":"2024-04-17T08:51:55.661298+0000","last_became_peered":"2024-04-17T08:51:55.661298+0000","last_unstale":"2024-04-17T08:51:57.693148+0000","last_undegraded":"2024-04-17T08:51:57.693148+0000","last_fullsized":"2024-04-17T08:51:57.693148+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:57.667695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949971+0000","last_change":"2024-04-17T08:51:41.500145+0000","last_active":"2024-04-17T08:51:57.949971+0000","last_peered":"2024-04-17T08:51:57.949971+0000","last_clean":"2024-04-17T08:51:57.949971+0000","last_became_active":"2024-04-17T08:51:41.499316+0000","last_became_peered":"2024-04-17T08:51:41.499316+0000","last_unstale":"2024-04-17T08:51:57.949971+0000","last_undegraded":"2024-04-17T08:51:57.949971+0000","last_fullsized":"2024-04-17T08:51:57.949971+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:49.058157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950485+0000","last_change":"2024-04-17T08:51:56.698232+0000","last_active":"2024-04-17T08:51:57.950485+0000","last_peered":"2024-04-17T08:51:57.950485+0000","last_clean":"2024-04-17T08:51:57.950485+0000","last_became_active":"2024-04-17T08:51:56.697860+0000","last_became_peered":"2024-04-17T08:51:56.697860+0000","last_unstale":"2024-04-17T08:51:57.950485+0000","last_undegraded":"2024-04-17T08:51:57.950485+0000","last_fullsized":"2024-04-17T08:51:57.950485+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:24.328571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"253'6","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949718+0000","last_change":"2024-04-17T08:51:55.652190+0000","last_active":"2024-04-17T08:51:57.949718+0000","last_peered":"2024-04-17T08:51:57.949718+0000","last_clean":"2024-04-17T08:51:57.949718+0000","last_became_active":"2024-04-17T08:51:55.651916+0000","last_became_peered":"2024-04-17T08:51:55.651916+0000","last_unstale":"2024-04-17T08:51:57.949718+0000","last_undegraded":"2024-04-17T08:51:57.949718+0000","last_fullsized":"2024-04-17T08:51:57.949718+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:25:29.119928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693446+0000","last_change":"2024-04-17T08:51:41.508121+0000","last_active":"2024-04-17T08:51:57.693446+0000","last_peered":"2024-04-17T08:51:57.693446+0000","last_clean":"2024-04-17T08:51:57.693446+0000","last_became_active":"2024-04-17T08:51:41.506717+0000","last_became_peered":"2024-04-17T08:51:41.506717+0000","last_unstale":"2024-04-17T08:51:57.693446+0000","last_undegraded":"2024-04-17T08:51:57.693446+0000","last_fullsized":"2024-04-17T08:51:57.693446+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:45.296249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707485+0000","last_change":"2024-04-17T08:51:56.697344+0000","last_active":"2024-04-17T08:51:57.707485+0000","last_peered":"2024-04-17T08:51:57.707485+0000","last_clean":"2024-04-17T08:51:57.707485+0000","last_became_active":"2024-04-17T08:51:56.694288+0000","last_became_peered":"2024-04-17T08:51:56.694288+0000","last_unstale":"2024-04-17T08:51:57.707485+0000","last_undegraded":"2024-04-17T08:51:57.707485+0000","last_fullsized":"2024-04-17T08:51:57.707485+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:57.840224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"253'3","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.716009+0000","last_change":"2024-04-17T08:51:55.670632+0000","last_active":"2024-04-17T08:51:56.716009+0000","last_peered":"2024-04-17T08:51:56.716009+0000","last_clean":"2024-04-17T08:51:56.716009+0000","last_became_active":"2024-04-17T08:51:55.670207+0000","last_became_peered":"2024-04-17T08:51:55.670207+0000","last_unstale":"2024-04-17T08:51:56.716009+0000","last_undegraded":"2024-04-17T08:51:56.716009+0000","last_fullsized":"2024-04-17T08:51:56.716009+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:06.066708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336354+0000","last_change":"2024-04-17T08:51:41.517429+0000","last_active":"2024-04-17T08:51:58.336354+0000","last_peered":"2024-04-17T08:51:58.336354+0000","last_clean":"2024-04-17T08:51:58.336354+0000","last_became_active":"2024-04-17T08:51:41.517130+0000","last_became_peered":"2024-04-17T08:51:41.517130+0000","last_unstale":"2024-04-17T08:51:58.336354+0000","last_undegraded":"2024-04-17T08:51:58.336354+0000","last_fullsized":"2024-04-17T08:51:58.336354+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:14.831099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950763+0000","last_change":"2024-04-17T08:51:56.709710+0000","last_active":"2024-04-17T08:51:57.950763+0000","last_peered":"2024-04-17T08:51:57.950763+0000","last_clean":"2024-04-17T08:51:57.950763+0000","last_became_active":"2024-04-17T08:51:56.704006+0000","last_became_peered":"2024-04-17T08:51:56.704006+0000","last_unstale":"2024-04-17T08:51:57.950763+0000","last_undegraded":"2024-04-17T08:51:57.950763+0000","last_fullsized":"2024-04-17T08:51:57.950763+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:51:34.353837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668959+0000","last_change":"2024-04-17T08:51:55.669734+0000","last_active":"2024-04-17T08:51:56.668959+0000","last_peered":"2024-04-17T08:51:56.668959+0000","last_clean":"2024-04-17T08:51:56.668959+0000","last_became_active":"2024-04-17T08:51:55.669456+0000","last_became_peered":"2024-04-17T08:51:55.669456+0000","last_unstale":"2024-04-17T08:51:56.668959+0000","last_undegraded":"2024-04-17T08:51:56.668959+0000","last_fullsized":"2024-04-17T08:51:56.668959+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:28.256872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691710+0000","last_change":"2024-04-17T08:51:41.507626+0000","last_active":"2024-04-17T08:51:57.691710+0000","last_peered":"2024-04-17T08:51:57.691710+0000","last_clean":"2024-04-17T08:51:57.691710+0000","last_became_active":"2024-04-17T08:51:41.507278+0000","last_became_peered":"2024-04-17T08:51:41.507278+0000","last_unstale":"2024-04-17T08:51:57.691710+0000","last_undegraded":"2024-04-17T08:51:57.691710+0000","last_fullsized":"2024-04-17T08:51:57.691710+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338380+0000","last_change":"2024-04-17T08:51:56.689527+0000","last_active":"2024-04-17T08:51:58.338380+0000","last_peered":"2024-04-17T08:51:58.338380+0000","last_clean":"2024-04-17T08:51:58.338380+0000","last_became_active":"2024-04-17T08:51:56.688886+0000","last_became_peered":"2024-04-17T08:51:56.688886+0000","last_unstale":"2024-04-17T08:51:58.338380+0000","last_undegraded":"2024-04-17T08:51:58.338380+0000","last_fullsized":"2024-04-17T08:51:58.338380+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:03.266907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951174+0000","last_change":"2024-04-17T08:51:55.663292+0000","last_active":"2024-04-17T08:51:57.951174+0000","last_peered":"2024-04-17T08:51:57.951174+0000","last_clean":"2024-04-17T08:51:57.951174+0000","last_became_active":"2024-04-17T08:51:55.661670+0000","last_became_peered":"2024-04-17T08:51:55.661670+0000","last_unstale":"2024-04-17T08:51:57.951174+0000","last_undegraded":"2024-04-17T08:51:57.951174+0000","last_fullsized":"2024-04-17T08:51:57.951174+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:58:37.070857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.670662+0000","last_change":"2024-04-17T08:51:41.502141+0000","last_active":"2024-04-17T08:51:56.670662+0000","last_peered":"2024-04-17T08:51:56.670662+0000","last_clean":"2024-04-17T08:51:56.670662+0000","last_became_active":"2024-04-17T08:51:41.501895+0000","last_became_peered":"2024-04-17T08:51:41.501895+0000","last_unstale":"2024-04-17T08:51:56.670662+0000","last_undegraded":"2024-04-17T08:51:56.670662+0000","last_fullsized":"2024-04-17T08:51:56.670662+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:31:45.139092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337013+0000","last_change":"2024-04-17T08:51:56.689396+0000","last_active":"2024-04-17T08:51:58.337013+0000","last_peered":"2024-04-17T08:51:58.337013+0000","last_clean":"2024-04-17T08:51:58.337013+0000","last_became_active":"2024-04-17T08:51:56.688849+0000","last_became_peered":"2024-04-17T08:51:56.688849+0000","last_unstale":"2024-04-17T08:51:58.337013+0000","last_undegraded":"2024-04-17T08:51:58.337013+0000","last_fullsized":"2024-04-17T08:51:58.337013+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:51.427089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336888+0000","last_change":"2024-04-17T08:51:55.665394+0000","last_active":"2024-04-17T08:51:58.336888+0000","last_peered":"2024-04-17T08:51:58.336888+0000","last_clean":"2024-04-17T08:51:58.336888+0000","last_became_active":"2024-04-17T08:51:55.664784+0000","last_became_peered":"2024-04-17T08:51:55.664784+0000","last_unstale":"2024-04-17T08:51:58.336888+0000","last_undegraded":"2024-04-17T08:51:58.336888+0000","last_fullsized":"2024-04-17T08:51:58.336888+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:21.789227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708177+0000","last_change":"2024-04-17T08:51:41.504007+0000","last_active":"2024-04-17T08:51:57.708177+0000","last_peered":"2024-04-17T08:51:57.708177+0000","last_clean":"2024-04-17T08:51:57.708177+0000","last_became_active":"2024-04-17T08:51:41.503745+0000","last_became_peered":"2024-04-17T08:51:41.503745+0000","last_unstale":"2024-04-17T08:51:57.708177+0000","last_undegraded":"2024-04-17T08:51:57.708177+0000","last_fullsized":"2024-04-17T08:51:57.708177+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:42:30.992713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699556+0000","last_change":"2024-04-17T08:51:56.694165+0000","last_active":"2024-04-17T08:51:56.699556+0000","last_peered":"2024-04-17T08:51:56.699556+0000","last_clean":"2024-04-17T08:51:56.699556+0000","last_became_active":"2024-04-17T08:51:56.687018+0000","last_became_peered":"2024-04-17T08:51:56.687018+0000","last_unstale":"2024-04-17T08:51:56.699556+0000","last_undegraded":"2024-04-17T08:51:56.699556+0000","last_fullsized":"2024-04-17T08:51:56.699556+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:52.726637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692793+0000","last_change":"2024-04-17T08:51:55.653389+0000","last_active":"2024-04-17T08:51:57.692793+0000","last_peered":"2024-04-17T08:51:57.692793+0000","last_clean":"2024-04-17T08:51:57.692793+0000","last_became_active":"2024-04-17T08:51:55.650896+0000","last_became_peered":"2024-04-17T08:51:55.650896+0000","last_unstale":"2024-04-17T08:51:57.692793+0000","last_undegraded":"2024-04-17T08:51:57.692793+0000","last_fullsized":"2024-04-17T08:51:57.692793+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:19.603617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338873+0000","last_change":"2024-04-17T08:51:41.513058+0000","last_active":"2024-04-17T08:51:58.338873+0000","last_peered":"2024-04-17T08:51:58.338873+0000","last_clean":"2024-04-17T08:51:58.338873+0000","last_became_active":"2024-04-17T08:51:41.512444+0000","last_became_peered":"2024-04-17T08:51:41.512444+0000","last_unstale":"2024-04-17T08:51:58.338873+0000","last_undegraded":"2024-04-17T08:51:58.338873+0000","last_fullsized":"2024-04-17T08:51:58.338873+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:37:05.436893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699072+0000","last_change":"2024-04-17T08:51:56.693163+0000","last_active":"2024-04-17T08:51:56.699072+0000","last_peered":"2024-04-17T08:51:56.699072+0000","last_clean":"2024-04-17T08:51:56.699072+0000","last_became_active":"2024-04-17T08:51:56.692676+0000","last_became_peered":"2024-04-17T08:51:56.692676+0000","last_unstale":"2024-04-17T08:51:56.699072+0000","last_undegraded":"2024-04-17T08:51:56.699072+0000","last_fullsized":"2024-04-17T08:51:56.699072+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:51:56.847018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949550+0000","last_change":"2024-04-17T08:51:55.661438+0000","last_active":"2024-04-17T08:51:57.949550+0000","last_peered":"2024-04-17T08:51:57.949550+0000","last_clean":"2024-04-17T08:51:57.949550+0000","last_became_active":"2024-04-17T08:51:55.661097+0000","last_became_peered":"2024-04-17T08:51:55.661097+0000","last_unstale":"2024-04-17T08:51:57.949550+0000","last_undegraded":"2024-04-17T08:51:57.949550+0000","last_fullsized":"2024-04-17T08:51:57.949550+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:58.622251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689593+0000","last_change":"2024-04-17T08:51:41.519929+0000","last_active":"2024-04-17T08:51:57.689593+0000","last_peered":"2024-04-17T08:51:57.689593+0000","last_clean":"2024-04-17T08:51:57.689593+0000","last_became_active":"2024-04-17T08:51:41.519591+0000","last_became_peered":"2024-04-17T08:51:41.519591+0000","last_unstale":"2024-04-17T08:51:57.689593+0000","last_undegraded":"2024-04-17T08:51:57.689593+0000","last_fullsized":"2024-04-17T08:51:57.689593+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:03.965519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689567+0000","last_change":"2024-04-17T08:51:56.705675+0000","last_active":"2024-04-17T08:51:57.689567+0000","last_peered":"2024-04-17T08:51:57.689567+0000","last_clean":"2024-04-17T08:51:57.689567+0000","last_became_active":"2024-04-17T08:51:56.705389+0000","last_became_peered":"2024-04-17T08:51:56.705389+0000","last_unstale":"2024-04-17T08:51:57.689567+0000","last_undegraded":"2024-04-17T08:51:57.689567+0000","last_fullsized":"2024-04-17T08:51:57.689567+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:43.281173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337062+0000","last_change":"2024-04-17T08:51:55.664957+0000","last_active":"2024-04-17T08:51:58.337062+0000","last_peered":"2024-04-17T08:51:58.337062+0000","last_clean":"2024-04-17T08:51:58.337062+0000","last_became_active":"2024-04-17T08:51:55.664547+0000","last_became_peered":"2024-04-17T08:51:55.664547+0000","last_unstale":"2024-04-17T08:51:58.337062+0000","last_undegraded":"2024-04-17T08:51:58.337062+0000","last_fullsized":"2024-04-17T08:51:58.337062+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:52.224136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667876+0000","last_change":"2024-04-17T08:51:41.506863+0000","last_active":"2024-04-17T08:51:56.667876+0000","last_peered":"2024-04-17T08:51:56.667876+0000","last_clean":"2024-04-17T08:51:56.667876+0000","last_became_active":"2024-04-17T08:51:41.506606+0000","last_became_peered":"2024-04-17T08:51:41.506606+0000","last_unstale":"2024-04-17T08:51:56.667876+0000","last_undegraded":"2024-04-17T08:51:56.667876+0000","last_fullsized":"2024-04-17T08:51:56.667876+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:42.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693654+0000","last_change":"2024-04-17T08:51:56.690518+0000","last_active":"2024-04-17T08:51:57.693654+0000","last_peered":"2024-04-17T08:51:57.693654+0000","last_clean":"2024-04-17T08:51:57.693654+0000","last_became_active":"2024-04-17T08:51:56.688133+0000","last_became_peered":"2024-04-17T08:51:56.688133+0000","last_unstale":"2024-04-17T08:51:57.693654+0000","last_undegraded":"2024-04-17T08:51:57.693654+0000","last_fullsized":"2024-04-17T08:51:57.693654+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:45.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.710606+0000","last_change":"2024-04-17T08:51:55.658488+0000","last_active":"2024-04-17T08:51:56.710606+0000","last_peered":"2024-04-17T08:51:56.710606+0000","last_clean":"2024-04-17T08:51:56.710606+0000","last_became_active":"2024-04-17T08:51:55.658178+0000","last_became_peered":"2024-04-17T08:51:55.658178+0000","last_unstale":"2024-04-17T08:51:56.710606+0000","last_undegraded":"2024-04-17T08:51:56.710606+0000","last_fullsized":"2024-04-17T08:51:56.710606+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:14.517760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689072+0000","last_change":"2024-04-17T08:51:41.505583+0000","last_active":"2024-04-17T08:51:57.689072+0000","last_peered":"2024-04-17T08:51:57.689072+0000","last_clean":"2024-04-17T08:51:57.689072+0000","last_became_active":"2024-04-17T08:51:41.501478+0000","last_became_peered":"2024-04-17T08:51:41.501478+0000","last_unstale":"2024-04-17T08:51:57.689072+0000","last_undegraded":"2024-04-17T08:51:57.689072+0000","last_fullsized":"2024-04-17T08:51:57.689072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:01.976342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337430+0000","last_change":"2024-04-17T08:51:56.697296+0000","last_active":"2024-04-17T08:51:58.337430+0000","last_peered":"2024-04-17T08:51:58.337430+0000","last_clean":"2024-04-17T08:51:58.337430+0000","last_became_active":"2024-04-17T08:51:56.696997+0000","last_became_peered":"2024-04-17T08:51:56.696997+0000","last_unstale":"2024-04-17T08:51:58.337430+0000","last_undegraded":"2024-04-17T08:51:58.337430+0000","last_fullsized":"2024-04-17T08:51:58.337430+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:04.357822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"253'2","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.715404+0000","last_change":"2024-04-17T08:51:55.661037+0000","last_active":"2024-04-17T08:51:56.715404+0000","last_peered":"2024-04-17T08:51:56.715404+0000","last_clean":"2024-04-17T08:51:56.715404+0000","last_became_active":"2024-04-17T08:51:55.660680+0000","last_became_peered":"2024-04-17T08:51:55.660680+0000","last_unstale":"2024-04-17T08:51:56.715404+0000","last_undegraded":"2024-04-17T08:51:56.715404+0000","last_fullsized":"2024-04-17T08:51:56.715404+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:01:26.513422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693507+0000","last_change":"2024-04-17T08:51:41.500660+0000","last_active":"2024-04-17T08:51:57.693507+0000","last_peered":"2024-04-17T08:51:57.693507+0000","last_clean":"2024-04-17T08:51:57.693507+0000","last_became_active":"2024-04-17T08:51:41.500031+0000","last_became_peered":"2024-04-17T08:51:41.500031+0000","last_unstale":"2024-04-17T08:51:57.693507+0000","last_undegraded":"2024-04-17T08:51:57.693507+0000","last_fullsized":"2024-04-17T08:51:57.693507+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:10.441817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708007+0000","last_change":"2024-04-17T08:51:56.677002+0000","last_active":"2024-04-17T08:51:57.708007+0000","last_peered":"2024-04-17T08:51:57.708007+0000","last_clean":"2024-04-17T08:51:57.708007+0000","last_became_active":"2024-04-17T08:51:56.675209+0000","last_became_peered":"2024-04-17T08:51:56.675209+0000","last_unstale":"2024-04-17T08:51:57.708007+0000","last_undegraded":"2024-04-17T08:51:57.708007+0000","last_fullsized":"2024-04-17T08:51:57.708007+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:38:56.424013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"253'1","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707787+0000","last_change":"2024-04-17T08:51:55.666023+0000","last_active":"2024-04-17T08:51:57.707787+0000","last_peered":"2024-04-17T08:51:57.707787+0000","last_clean":"2024-04-17T08:51:57.707787+0000","last_became_active":"2024-04-17T08:51:55.665089+0000","last_became_peered":"2024-04-17T08:51:55.665089+0000","last_unstale":"2024-04-17T08:51:57.707787+0000","last_undegraded":"2024-04-17T08:51:57.707787+0000","last_fullsized":"2024-04-17T08:51:57.707787+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:12:59.267963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706929+0000","last_change":"2024-04-17T08:51:41.501146+0000","last_active":"2024-04-17T08:51:57.706929+0000","last_peered":"2024-04-17T08:51:57.706929+0000","last_clean":"2024-04-17T08:51:57.706929+0000","last_became_active":"2024-04-17T08:51:41.500704+0000","last_became_peered":"2024-04-17T08:51:41.500704+0000","last_unstale":"2024-04-17T08:51:57.706929+0000","last_undegraded":"2024-04-17T08:51:57.706929+0000","last_fullsized":"2024-04-17T08:51:57.706929+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:01.914122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338245+0000","last_change":"2024-04-17T08:51:56.689773+0000","last_active":"2024-04-17T08:51:58.338245+0000","last_peered":"2024-04-17T08:51:58.338245+0000","last_clean":"2024-04-17T08:51:58.338245+0000","last_became_active":"2024-04-17T08:51:56.689226+0000","last_became_peered":"2024-04-17T08:51:56.689226+0000","last_unstale":"2024-04-17T08:51:58.338245+0000","last_undegraded":"2024-04-17T08:51:58.338245+0000","last_fullsized":"2024-04-17T08:51:58.338245+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:32:00.322774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"253'4","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951046+0000","last_change":"2024-04-17T08:51:55.661967+0000","last_active":"2024-04-17T08:51:57.951046+0000","last_peered":"2024-04-17T08:51:57.951046+0000","last_clean":"2024-04-17T08:51:57.951046+0000","last_became_active":"2024-04-17T08:51:55.661233+0000","last_became_peered":"2024-04-17T08:51:55.661233+0000","last_unstale":"2024-04-17T08:51:57.951046+0000","last_undegraded":"2024-04-17T08:51:57.951046+0000","last_fullsized":"2024-04-17T08:51:57.951046+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:54.602804+0000","last_clean_scrub_stamp":"2024-04-17T08:51:54.602804+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:05:37.232722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672080+0000","last_change":"2024-04-17T08:51:41.517358+0000","last_active":"2024-04-17T08:51:56.672080+0000","last_peered":"2024-04-17T08:51:56.672080+0000","last_clean":"2024-04-17T08:51:56.672080+0000","last_became_active":"2024-04-17T08:51:41.517090+0000","last_became_peered":"2024-04-17T08:51:41.517090+0000","last_unstale":"2024-04-17T08:51:56.672080+0000","last_undegraded":"2024-04-17T08:51:56.672080+0000","last_fullsized":"2024-04-17T08:51:56.672080+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:24:48.083225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691415+0000","last_change":"2024-04-17T08:51:56.687011+0000","last_active":"2024-04-17T08:51:57.691415+0000","last_peered":"2024-04-17T08:51:57.691415+0000","last_clean":"2024-04-17T08:51:57.691415+0000","last_became_active":"2024-04-17T08:51:56.686740+0000","last_became_peered":"2024-04-17T08:51:56.686740+0000","last_unstale":"2024-04-17T08:51:57.691415+0000","last_undegraded":"2024-04-17T08:51:57.691415+0000","last_fullsized":"2024-04-17T08:51:57.691415+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:54.896615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950978+0000","last_change":"2024-04-17T08:51:41.503931+0000","last_active":"2024-04-17T08:51:57.950978+0000","last_peered":"2024-04-17T08:51:57.950978+0000","last_clean":"2024-04-17T08:51:57.950978+0000","last_became_active":"2024-04-17T08:51:41.502502+0000","last_became_peered":"2024-04-17T08:51:41.502502+0000","last_unstale":"2024-04-17T08:51:57.950978+0000","last_undegraded":"2024-04-17T08:51:57.950978+0000","last_fullsized":"2024-04-17T08:51:57.950978+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:58:26.370488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.339006+0000","last_change":"2024-04-17T08:51:40.478560+0000","last_active":"2024-04-17T08:51:58.339006+0000","last_peered":"2024-04-17T08:51:58.339006+0000","last_clean":"2024-04-17T08:51:58.339006+0000","last_became_active":"2024-04-17T08:51:40.478144+0000","last_became_peered":"2024-04-17T08:51:40.478144+0000","last_unstale":"2024-04-17T08:51:58.339006+0000","last_undegraded":"2024-04-17T08:51:58.339006+0000","last_fullsized":"2024-04-17T08:51:58.339006+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:37.672975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706991+0000","last_change":"2024-04-17T08:51:56.687973+0000","last_active":"2024-04-17T08:51:57.706991+0000","last_peered":"2024-04-17T08:51:57.706991+0000","last_clean":"2024-04-17T08:51:57.706991+0000","last_became_active":"2024-04-17T08:51:56.687606+0000","last_became_peered":"2024-04-17T08:51:56.687606+0000","last_unstale":"2024-04-17T08:51:57.706991+0000","last_undegraded":"2024-04-17T08:51:57.706991+0000","last_fullsized":"2024-04-17T08:51:57.706991+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:28:40.602103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688975+0000","last_change":"2024-04-17T08:51:41.505188+0000","last_active":"2024-04-17T08:51:57.688975+0000","last_peered":"2024-04-17T08:51:57.688975+0000","last_clean":"2024-04-17T08:51:57.688975+0000","last_became_active":"2024-04-17T08:51:41.504905+0000","last_became_peered":"2024-04-17T08:51:41.504905+0000","last_unstale":"2024-04-17T08:51:57.688975+0000","last_undegraded":"2024-04-17T08:51:57.688975+0000","last_fullsized":"2024-04-17T08:51:57.688975+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:43:11.712753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950539+0000","last_change":"2024-04-17T08:51:40.469433+0000","last_active":"2024-04-17T08:51:57.950539+0000","last_peered":"2024-04-17T08:51:57.950539+0000","last_clean":"2024-04-17T08:51:57.950539+0000","last_became_active":"2024-04-17T08:51:40.469071+0000","last_became_peered":"2024-04-17T08:51:40.469071+0000","last_unstale":"2024-04-17T08:51:57.950539+0000","last_undegraded":"2024-04-17T08:51:57.950539+0000","last_fullsized":"2024-04-17T08:51:57.950539+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:21.208592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950303+0000","last_change":"2024-04-17T08:51:56.700563+0000","last_active":"2024-04-17T08:51:57.950303+0000","last_peered":"2024-04-17T08:51:57.950303+0000","last_clean":"2024-04-17T08:51:57.950303+0000","last_became_active":"2024-04-17T08:51:56.690951+0000","last_became_peered":"2024-04-17T08:51:56.690951+0000","last_unstale":"2024-04-17T08:51:57.950303+0000","last_undegraded":"2024-04-17T08:51:57.950303+0000","last_fullsized":"2024-04-17T08:51:57.950303+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:30.511975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693787+0000","last_change":"2024-04-17T08:51:41.516473+0000","last_active":"2024-04-17T08:51:57.693787+0000","last_peered":"2024-04-17T08:51:57.693787+0000","last_clean":"2024-04-17T08:51:57.693787+0000","last_became_active":"2024-04-17T08:51:41.515870+0000","last_became_peered":"2024-04-17T08:51:41.515870+0000","last_unstale":"2024-04-17T08:51:57.693787+0000","last_undegraded":"2024-04-17T08:51:57.693787+0000","last_fullsized":"2024-04-17T08:51:57.693787+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:09.372045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950086+0000","last_change":"2024-04-17T08:51:40.472815+0000","last_active":"2024-04-17T08:51:57.950086+0000","last_peered":"2024-04-17T08:51:57.950086+0000","last_clean":"2024-04-17T08:51:57.950086+0000","last_became_active":"2024-04-17T08:51:40.470696+0000","last_became_peered":"2024-04-17T08:51:40.470696+0000","last_unstale":"2024-04-17T08:51:57.950086+0000","last_undegraded":"2024-04-17T08:51:57.950086+0000","last_fullsized":"2024-04-17T08:51:57.950086+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:32.004523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707742+0000","last_change":"2024-04-17T08:51:56.697594+0000","last_active":"2024-04-17T08:51:57.707742+0000","last_peered":"2024-04-17T08:51:57.707742+0000","last_clean":"2024-04-17T08:51:57.707742+0000","last_became_active":"2024-04-17T08:51:56.695529+0000","last_became_peered":"2024-04-17T08:51:56.695529+0000","last_unstale":"2024-04-17T08:51:57.707742+0000","last_undegraded":"2024-04-17T08:51:57.707742+0000","last_fullsized":"2024-04-17T08:51:57.707742+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:18.880218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689293+0000","last_change":"2024-04-17T08:51:41.497630+0000","last_active":"2024-04-17T08:51:57.689293+0000","last_peered":"2024-04-17T08:51:57.689293+0000","last_clean":"2024-04-17T08:51:57.689293+0000","last_became_active":"2024-04-17T08:51:41.497323+0000","last_became_peered":"2024-04-17T08:51:41.497323+0000","last_unstale":"2024-04-17T08:51:57.689293+0000","last_undegraded":"2024-04-17T08:51:57.689293+0000","last_fullsized":"2024-04-17T08:51:57.689293+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:56:45.677621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707147+0000","last_change":"2024-04-17T08:51:40.475306+0000","last_active":"2024-04-17T08:51:57.707147+0000","last_peered":"2024-04-17T08:51:57.707147+0000","last_clean":"2024-04-17T08:51:57.707147+0000","last_became_active":"2024-04-17T08:51:40.474409+0000","last_became_peered":"2024-04-17T08:51:40.474409+0000","last_unstale":"2024-04-17T08:51:57.707147+0000","last_undegraded":"2024-04-17T08:51:57.707147+0000","last_fullsized":"2024-04-17T08:51:57.707147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:17.301101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337934+0000","last_change":"2024-04-17T08:51:56.689149+0000","last_active":"2024-04-17T08:51:58.337934+0000","last_peered":"2024-04-17T08:51:58.337934+0000","last_clean":"2024-04-17T08:51:58.337934+0000","last_became_active":"2024-04-17T08:51:56.688411+0000","last_became_peered":"2024-04-17T08:51:56.688411+0000","last_unstale":"2024-04-17T08:51:58.337934+0000","last_undegraded":"2024-04-17T08:51:58.337934+0000","last_fullsized":"2024-04-17T08:51:58.337934+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:09.217990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.665848+0000","last_change":"2024-04-17T08:51:41.511131+0000","last_active":"2024-04-17T08:51:56.665848+0000","last_peered":"2024-04-17T08:51:56.665848+0000","last_clean":"2024-04-17T08:51:56.665848+0000","last_became_active":"2024-04-17T08:51:41.510481+0000","last_became_peered":"2024-04-17T08:51:41.510481+0000","last_unstale":"2024-04-17T08:51:56.665848+0000","last_undegraded":"2024-04-17T08:51:56.665848+0000","last_fullsized":"2024-04-17T08:51:56.665848+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:42.844488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":21,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668803+0000","last_change":"2024-04-17T08:51:40.476463+0000","last_active":"2024-04-17T08:51:56.668803+0000","last_peered":"2024-04-17T08:51:56.668803+0000","last_clean":"2024-04-17T08:51:56.668803+0000","last_became_active":"2024-04-17T08:51:40.476164+0000","last_became_peered":"2024-04-17T08:51:40.476164+0000","last_unstale":"2024-04-17T08:51:56.668803+0000","last_undegraded":"2024-04-17T08:51:56.668803+0000","last_fullsized":"2024-04-17T08:51:56.668803+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:15.997620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693773+0000","last_change":"2024-04-17T08:51:56.688954+0000","last_active":"2024-04-17T08:51:57.693773+0000","last_peered":"2024-04-17T08:51:57.693773+0000","last_clean":"2024-04-17T08:51:57.693773+0000","last_became_active":"2024-04-17T08:51:56.688684+0000","last_became_peered":"2024-04-17T08:51:56.688684+0000","last_unstale":"2024-04-17T08:51:57.693773+0000","last_undegraded":"2024-04-17T08:51:57.693773+0000","last_fullsized":"2024-04-17T08:51:57.693773+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:00.871425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948724+0000","last_change":"2024-04-17T08:51:41.507219+0000","last_active":"2024-04-17T08:51:57.948724+0000","last_peered":"2024-04-17T08:51:57.948724+0000","last_clean":"2024-04-17T08:51:57.948724+0000","last_became_active":"2024-04-17T08:51:41.504481+0000","last_became_peered":"2024-04-17T08:51:41.504481+0000","last_unstale":"2024-04-17T08:51:57.948724+0000","last_undegraded":"2024-04-17T08:51:57.948724+0000","last_fullsized":"2024-04-17T08:51:57.948724+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:00.028124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336986+0000","last_change":"2024-04-17T08:51:40.485442+0000","last_active":"2024-04-17T08:51:58.336986+0000","last_peered":"2024-04-17T08:51:58.336986+0000","last_clean":"2024-04-17T08:51:58.336986+0000","last_became_active":"2024-04-17T08:51:40.485100+0000","last_became_peered":"2024-04-17T08:51:40.485100+0000","last_unstale":"2024-04-17T08:51:58.336986+0000","last_undegraded":"2024-04-17T08:51:58.336986+0000","last_fullsized":"2024-04-17T08:51:58.336986+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:50:18.376309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708066+0000","last_change":"2024-04-17T08:51:56.694841+0000","last_active":"2024-04-17T08:51:57.708066+0000","last_peered":"2024-04-17T08:51:57.708066+0000","last_clean":"2024-04-17T08:51:57.708066+0000","last_became_active":"2024-04-17T08:51:56.693519+0000","last_became_peered":"2024-04-17T08:51:56.693519+0000","last_unstale":"2024-04-17T08:51:57.708066+0000","last_undegraded":"2024-04-17T08:51:57.708066+0000","last_fullsized":"2024-04-17T08:51:57.708066+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:23.545377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668684+0000","last_change":"2024-04-17T08:51:41.504552+0000","last_active":"2024-04-17T08:51:56.668684+0000","last_peered":"2024-04-17T08:51:56.668684+0000","last_clean":"2024-04-17T08:51:56.668684+0000","last_became_active":"2024-04-17T08:51:41.504177+0000","last_became_peered":"2024-04-17T08:51:41.504177+0000","last_unstale":"2024-04-17T08:51:56.668684+0000","last_undegraded":"2024-04-17T08:51:56.668684+0000","last_fullsized":"2024-04-17T08:51:56.668684+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:31.634770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948969+0000","last_change":"2024-04-17T08:51:40.470752+0000","last_active":"2024-04-17T08:51:57.948969+0000","last_peered":"2024-04-17T08:51:57.948969+0000","last_clean":"2024-04-17T08:51:57.948969+0000","last_became_active":"2024-04-17T08:51:40.470440+0000","last_became_peered":"2024-04-17T08:51:40.470440+0000","last_unstale":"2024-04-17T08:51:57.948969+0000","last_undegraded":"2024-04-17T08:51:57.948969+0000","last_fullsized":"2024-04-17T08:51:57.948969+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.094101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.948552+0000","last_change":"2024-04-17T08:51:56.702337+0000","last_active":"2024-04-17T08:51:57.948552+0000","last_peered":"2024-04-17T08:51:57.948552+0000","last_clean":"2024-04-17T08:51:57.948552+0000","last_became_active":"2024-04-17T08:51:56.701926+0000","last_became_peered":"2024-04-17T08:51:56.701926+0000","last_unstale":"2024-04-17T08:51:57.948552+0000","last_undegraded":"2024-04-17T08:51:57.948552+0000","last_fullsized":"2024-04-17T08:51:57.948552+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:13:28.103568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.670025+0000","last_change":"2024-04-17T08:51:41.505515+0000","last_active":"2024-04-17T08:51:56.670025+0000","last_peered":"2024-04-17T08:51:56.670025+0000","last_clean":"2024-04-17T08:51:56.670025+0000","last_became_active":"2024-04-17T08:51:41.504948+0000","last_became_peered":"2024-04-17T08:51:41.504948+0000","last_unstale":"2024-04-17T08:51:56.670025+0000","last_undegraded":"2024-04-17T08:51:56.670025+0000","last_fullsized":"2024-04-17T08:51:56.670025+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:55:44.205613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695652+0000","last_change":"2024-04-17T08:51:40.472194+0000","last_active":"2024-04-17T08:51:57.695652+0000","last_peered":"2024-04-17T08:51:57.695652+0000","last_clean":"2024-04-17T08:51:57.695652+0000","last_became_active":"2024-04-17T08:51:40.471867+0000","last_became_peered":"2024-04-17T08:51:40.471867+0000","last_unstale":"2024-04-17T08:51:57.695652+0000","last_undegraded":"2024-04-17T08:51:57.695652+0000","last_fullsized":"2024-04-17T08:51:57.695652+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:57.953636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336197+0000","last_change":"2024-04-17T08:51:56.691578+0000","last_active":"2024-04-17T08:51:58.336197+0000","last_peered":"2024-04-17T08:51:58.336197+0000","last_clean":"2024-04-17T08:51:58.336197+0000","last_became_active":"2024-04-17T08:51:56.691141+0000","last_became_peered":"2024-04-17T08:51:56.691141+0000","last_unstale":"2024-04-17T08:51:58.336197+0000","last_undegraded":"2024-04-17T08:51:58.336197+0000","last_fullsized":"2024-04-17T08:51:58.336197+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:54.896615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689835+0000","last_change":"2024-04-17T08:51:41.505182+0000","last_active":"2024-04-17T08:51:57.689835+0000","last_peered":"2024-04-17T08:51:57.689835+0000","last_clean":"2024-04-17T08:51:57.689835+0000","last_became_active":"2024-04-17T08:51:41.503770+0000","last_became_peered":"2024-04-17T08:51:41.503770+0000","last_unstale":"2024-04-17T08:51:57.689835+0000","last_undegraded":"2024-04-17T08:51:57.689835+0000","last_fullsized":"2024-04-17T08:51:57.689835+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:17.888942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950406+0000","last_change":"2024-04-17T08:51:40.478982+0000","last_active":"2024-04-17T08:51:57.950406+0000","last_peered":"2024-04-17T08:51:57.950406+0000","last_clean":"2024-04-17T08:51:57.950406+0000","last_became_active":"2024-04-17T08:51:40.478218+0000","last_became_peered":"2024-04-17T08:51:40.478218+0000","last_unstale":"2024-04-17T08:51:57.950406+0000","last_undegraded":"2024-04-17T08:51:57.950406+0000","last_fullsized":"2024-04-17T08:51:57.950406+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:48:34.166708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688968+0000","last_change":"2024-04-17T08:51:56.700642+0000","last_active":"2024-04-17T08:51:57.688968+0000","last_peered":"2024-04-17T08:51:57.688968+0000","last_clean":"2024-04-17T08:51:57.688968+0000","last_became_active":"2024-04-17T08:51:56.700352+0000","last_became_peered":"2024-04-17T08:51:56.700352+0000","last_unstale":"2024-04-17T08:51:57.688968+0000","last_undegraded":"2024-04-17T08:51:57.688968+0000","last_fullsized":"2024-04-17T08:51:57.688968+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:41.048823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950101+0000","last_change":"2024-04-17T08:51:41.488821+0000","last_active":"2024-04-17T08:51:57.950101+0000","last_peered":"2024-04-17T08:51:57.950101+0000","last_clean":"2024-04-17T08:51:57.950101+0000","last_became_active":"2024-04-17T08:51:41.488545+0000","last_became_peered":"2024-04-17T08:51:41.488545+0000","last_unstale":"2024-04-17T08:51:57.950101+0000","last_undegraded":"2024-04-17T08:51:57.950101+0000","last_fullsized":"2024-04-17T08:51:57.950101+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:42.844488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":23,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.671600+0000","last_change":"2024-04-17T08:51:40.484183+0000","last_active":"2024-04-17T08:51:56.671600+0000","last_peered":"2024-04-17T08:51:56.671600+0000","last_clean":"2024-04-17T08:51:56.671600+0000","last_became_active":"2024-04-17T08:51:40.483891+0000","last_became_peered":"2024-04-17T08:51:40.483891+0000","last_unstale":"2024-04-17T08:51:56.671600+0000","last_undegraded":"2024-04-17T08:51:56.671600+0000","last_fullsized":"2024-04-17T08:51:56.671600+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:41.230843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688876+0000","last_change":"2024-04-17T08:51:56.703392+0000","last_active":"2024-04-17T08:51:57.688876+0000","last_peered":"2024-04-17T08:51:57.688876+0000","last_clean":"2024-04-17T08:51:57.688876+0000","last_became_active":"2024-04-17T08:51:56.701149+0000","last_became_peered":"2024-04-17T08:51:56.701149+0000","last_unstale":"2024-04-17T08:51:57.688876+0000","last_undegraded":"2024-04-17T08:51:57.688876+0000","last_fullsized":"2024-04-17T08:51:57.688876+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:06:50.981331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338221+0000","last_change":"2024-04-17T08:51:41.515320+0000","last_active":"2024-04-17T08:51:58.338221+0000","last_peered":"2024-04-17T08:51:58.338221+0000","last_clean":"2024-04-17T08:51:58.338221+0000","last_became_active":"2024-04-17T08:51:41.513254+0000","last_became_peered":"2024-04-17T08:51:41.513254+0000","last_unstale":"2024-04-17T08:51:58.338221+0000","last_undegraded":"2024-04-17T08:51:58.338221+0000","last_fullsized":"2024-04-17T08:51:58.338221+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"250'8","reported_seq":28,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668176+0000","last_change":"2024-04-17T08:51:40.479746+0000","last_active":"2024-04-17T08:51:56.668176+0000","last_peered":"2024-04-17T08:51:56.668176+0000","last_clean":"2024-04-17T08:51:56.668176+0000","last_became_active":"2024-04-17T08:51:40.478655+0000","last_became_peered":"2024-04-17T08:51:40.478655+0000","last_unstale":"2024-04-17T08:51:56.668176+0000","last_undegraded":"2024-04-17T08:51:56.668176+0000","last_fullsized":"2024-04-17T08:51:56.668176+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:03.205317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699626+0000","last_change":"2024-04-17T08:51:56.694278+0000","last_active":"2024-04-17T08:51:56.699626+0000","last_peered":"2024-04-17T08:51:56.699626+0000","last_clean":"2024-04-17T08:51:56.699626+0000","last_became_active":"2024-04-17T08:51:56.693942+0000","last_became_peered":"2024-04-17T08:51:56.693942+0000","last_unstale":"2024-04-17T08:51:56.699626+0000","last_undegraded":"2024-04-17T08:51:56.699626+0000","last_fullsized":"2024-04-17T08:51:56.699626+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:08:14.536416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336523+0000","last_change":"2024-04-17T08:51:41.518673+0000","last_active":"2024-04-17T08:51:58.336523+0000","last_peered":"2024-04-17T08:51:58.336523+0000","last_clean":"2024-04-17T08:51:58.336523+0000","last_became_active":"2024-04-17T08:51:41.517797+0000","last_became_peered":"2024-04-17T08:51:41.517797+0000","last_unstale":"2024-04-17T08:51:58.336523+0000","last_undegraded":"2024-04-17T08:51:58.336523+0000","last_fullsized":"2024-04-17T08:51:58.336523+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:09.689564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":22,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672535+0000","last_change":"2024-04-17T08:51:40.481925+0000","last_active":"2024-04-17T08:51:56.672535+0000","last_peered":"2024-04-17T08:51:56.672535+0000","last_clean":"2024-04-17T08:51:56.672535+0000","last_became_active":"2024-04-17T08:51:40.481436+0000","last_became_peered":"2024-04-17T08:51:40.481436+0000","last_unstale":"2024-04-17T08:51:56.672535+0000","last_undegraded":"2024-04-17T08:51:56.672535+0000","last_fullsized":"2024-04-17T08:51:56.672535+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:04.739470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337570+0000","last_change":"2024-04-17T08:51:56.698176+0000","last_active":"2024-04-17T08:51:58.337570+0000","last_peered":"2024-04-17T08:51:58.337570+0000","last_clean":"2024-04-17T08:51:58.337570+0000","last_became_active":"2024-04-17T08:51:56.697866+0000","last_became_peered":"2024-04-17T08:51:56.697866+0000","last_unstale":"2024-04-17T08:51:58.337570+0000","last_undegraded":"2024-04-17T08:51:58.337570+0000","last_fullsized":"2024-04-17T08:51:58.337570+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:46.818453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666095+0000","last_change":"2024-04-17T08:51:41.508860+0000","last_active":"2024-04-17T08:51:56.666095+0000","last_peered":"2024-04-17T08:51:56.666095+0000","last_clean":"2024-04-17T08:51:56.666095+0000","last_became_active":"2024-04-17T08:51:41.508524+0000","last_became_peered":"2024-04-17T08:51:41.508524+0000","last_unstale":"2024-04-17T08:51:56.666095+0000","last_undegraded":"2024-04-17T08:51:56.666095+0000","last_fullsized":"2024-04-17T08:51:56.666095+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:23.647788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950300+0000","last_change":"2024-04-17T08:51:40.479291+0000","last_active":"2024-04-17T08:51:57.950300+0000","last_peered":"2024-04-17T08:51:57.950300+0000","last_clean":"2024-04-17T08:51:57.950300+0000","last_became_active":"2024-04-17T08:51:40.476885+0000","last_became_peered":"2024-04-17T08:51:40.476885+0000","last_unstale":"2024-04-17T08:51:57.950300+0000","last_undegraded":"2024-04-17T08:51:57.950300+0000","last_fullsized":"2024-04-17T08:51:57.950300+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:53.966461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338691+0000","last_change":"2024-04-17T08:51:56.690300+0000","last_active":"2024-04-17T08:51:58.338691+0000","last_peered":"2024-04-17T08:51:58.338691+0000","last_clean":"2024-04-17T08:51:58.338691+0000","last_became_active":"2024-04-17T08:51:56.690019+0000","last_became_peered":"2024-04-17T08:51:56.690019+0000","last_unstale":"2024-04-17T08:51:58.338691+0000","last_undegraded":"2024-04-17T08:51:58.338691+0000","last_fullsized":"2024-04-17T08:51:58.338691+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:28:57.582689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706642+0000","last_change":"2024-04-17T08:51:41.509196+0000","last_active":"2024-04-17T08:51:57.706642+0000","last_peered":"2024-04-17T08:51:57.706642+0000","last_clean":"2024-04-17T08:51:57.706642+0000","last_became_active":"2024-04-17T08:51:41.508917+0000","last_became_peered":"2024-04-17T08:51:41.508917+0000","last_unstale":"2024-04-17T08:51:57.706642+0000","last_undegraded":"2024-04-17T08:51:57.706642+0000","last_fullsized":"2024-04-17T08:51:57.706642+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:18:39.714011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337208+0000","last_change":"2024-04-17T08:51:40.483627+0000","last_active":"2024-04-17T08:51:58.337208+0000","last_peered":"2024-04-17T08:51:58.337208+0000","last_clean":"2024-04-17T08:51:58.337208+0000","last_became_active":"2024-04-17T08:51:40.482318+0000","last_became_peered":"2024-04-17T08:51:40.482318+0000","last_unstale":"2024-04-17T08:51:58.337208+0000","last_undegraded":"2024-04-17T08:51:58.337208+0000","last_fullsized":"2024-04-17T08:51:58.337208+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.761124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951410+0000","last_change":"2024-04-17T08:51:56.704670+0000","last_active":"2024-04-17T08:51:57.951410+0000","last_peered":"2024-04-17T08:51:57.951410+0000","last_clean":"2024-04-17T08:51:57.951410+0000","last_became_active":"2024-04-17T08:51:56.700800+0000","last_became_peered":"2024-04-17T08:51:56.700800+0000","last_unstale":"2024-04-17T08:51:57.951410+0000","last_undegraded":"2024-04-17T08:51:57.951410+0000","last_fullsized":"2024-04-17T08:51:57.951410+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:46.196158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336698+0000","last_change":"2024-04-17T08:51:41.517149+0000","last_active":"2024-04-17T08:51:58.336698+0000","last_peered":"2024-04-17T08:51:58.336698+0000","last_clean":"2024-04-17T08:51:58.336698+0000","last_became_active":"2024-04-17T08:51:41.516844+0000","last_became_peered":"2024-04-17T08:51:41.516844+0000","last_unstale":"2024-04-17T08:51:58.336698+0000","last_undegraded":"2024-04-17T08:51:58.336698+0000","last_fullsized":"2024-04-17T08:51:58.336698+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:37.293958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":30,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688523+0000","last_change":"2024-04-17T08:51:40.479653+0000","last_active":"2024-04-17T08:51:57.688523+0000","last_peered":"2024-04-17T08:51:57.688523+0000","last_clean":"2024-04-17T08:51:57.688523+0000","last_became_active":"2024-04-17T08:51:40.479107+0000","last_became_peered":"2024-04-17T08:51:40.479107+0000","last_unstale":"2024-04-17T08:51:57.688523+0000","last_undegraded":"2024-04-17T08:51:57.688523+0000","last_fullsized":"2024-04-17T08:51:57.688523+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:16.542860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.701673+0000","last_change":"2024-04-17T08:51:56.694906+0000","last_active":"2024-04-17T08:51:56.701673+0000","last_peered":"2024-04-17T08:51:56.701673+0000","last_clean":"2024-04-17T08:51:56.701673+0000","last_became_active":"2024-04-17T08:51:56.694487+0000","last_became_peered":"2024-04-17T08:51:56.694487+0000","last_unstale":"2024-04-17T08:51:56.701673+0000","last_undegraded":"2024-04-17T08:51:56.701673+0000","last_fullsized":"2024-04-17T08:51:56.701673+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:06.324844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707893+0000","last_change":"2024-04-17T08:51:41.510891+0000","last_active":"2024-04-17T08:51:57.707893+0000","last_peered":"2024-04-17T08:51:57.707893+0000","last_clean":"2024-04-17T08:51:57.707893+0000","last_became_active":"2024-04-17T08:51:41.510538+0000","last_became_peered":"2024-04-17T08:51:41.510538+0000","last_unstale":"2024-04-17T08:51:57.707893+0000","last_undegraded":"2024-04-17T08:51:57.707893+0000","last_fullsized":"2024-04-17T08:51:57.707893+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:00.189780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338428+0000","last_change":"2024-04-17T08:51:40.484060+0000","last_active":"2024-04-17T08:51:58.338428+0000","last_peered":"2024-04-17T08:51:58.338428+0000","last_clean":"2024-04-17T08:51:58.338428+0000","last_became_active":"2024-04-17T08:51:40.483692+0000","last_became_peered":"2024-04-17T08:51:40.483692+0000","last_unstale":"2024-04-17T08:51:58.338428+0000","last_undegraded":"2024-04-17T08:51:58.338428+0000","last_fullsized":"2024-04-17T08:51:58.338428+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:53.270393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706841+0000","last_change":"2024-04-17T08:51:56.673169+0000","last_active":"2024-04-17T08:51:57.706841+0000","last_peered":"2024-04-17T08:51:57.706841+0000","last_clean":"2024-04-17T08:51:57.706841+0000","last_became_active":"2024-04-17T08:51:56.671504+0000","last_became_peered":"2024-04-17T08:51:56.671504+0000","last_unstale":"2024-04-17T08:51:57.706841+0000","last_undegraded":"2024-04-17T08:51:57.706841+0000","last_fullsized":"2024-04-17T08:51:57.706841+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:50.045470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951158+0000","last_change":"2024-04-17T08:51:41.516369+0000","last_active":"2024-04-17T08:51:57.951158+0000","last_peered":"2024-04-17T08:51:57.951158+0000","last_clean":"2024-04-17T08:51:57.951158+0000","last_became_active":"2024-04-17T08:51:41.515919+0000","last_became_peered":"2024-04-17T08:51:41.515919+0000","last_unstale":"2024-04-17T08:51:57.951158+0000","last_undegraded":"2024-04-17T08:51:57.951158+0000","last_fullsized":"2024-04-17T08:51:57.951158+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:50:23.477382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950526+0000","last_change":"2024-04-17T08:51:40.466869+0000","last_active":"2024-04-17T08:51:57.950526+0000","last_peered":"2024-04-17T08:51:57.950526+0000","last_clean":"2024-04-17T08:51:57.950526+0000","last_became_active":"2024-04-17T08:51:40.465973+0000","last_became_peered":"2024-04-17T08:51:40.465973+0000","last_unstale":"2024-04-17T08:51:57.950526+0000","last_undegraded":"2024-04-17T08:51:57.950526+0000","last_fullsized":"2024-04-17T08:51:57.950526+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:57.022722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.706996+0000","last_change":"2024-04-17T08:51:56.691468+0000","last_active":"2024-04-17T08:51:57.706996+0000","last_peered":"2024-04-17T08:51:57.706996+0000","last_clean":"2024-04-17T08:51:57.706996+0000","last_became_active":"2024-04-17T08:51:56.691059+0000","last_became_peered":"2024-04-17T08:51:56.691059+0000","last_unstale":"2024-04-17T08:51:57.706996+0000","last_undegraded":"2024-04-17T08:51:57.706996+0000","last_fullsized":"2024-04-17T08:51:57.706996+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:04:25.930760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.665883+0000","last_change":"2024-04-17T08:51:41.507082+0000","last_active":"2024-04-17T08:51:56.665883+0000","last_peered":"2024-04-17T08:51:56.665883+0000","last_clean":"2024-04-17T08:51:56.665883+0000","last_became_active":"2024-04-17T08:51:41.506812+0000","last_became_peered":"2024-04-17T08:51:41.506812+0000","last_unstale":"2024-04-17T08:51:56.665883+0000","last_undegraded":"2024-04-17T08:51:56.665883+0000","last_fullsized":"2024-04-17T08:51:56.665883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:39:34.182442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":21,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666753+0000","last_change":"2024-04-17T08:51:40.481626+0000","last_active":"2024-04-17T08:51:56.666753+0000","last_peered":"2024-04-17T08:51:56.666753+0000","last_clean":"2024-04-17T08:51:56.666753+0000","last_became_active":"2024-04-17T08:51:40.481268+0000","last_became_peered":"2024-04-17T08:51:40.481268+0000","last_unstale":"2024-04-17T08:51:56.666753+0000","last_undegraded":"2024-04-17T08:51:56.666753+0000","last_fullsized":"2024-04-17T08:51:56.666753+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:35.353434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707244+0000","last_change":"2024-04-17T08:51:56.700702+0000","last_active":"2024-04-17T08:51:57.707244+0000","last_peered":"2024-04-17T08:51:57.707244+0000","last_clean":"2024-04-17T08:51:57.707244+0000","last_became_active":"2024-04-17T08:51:56.700414+0000","last_became_peered":"2024-04-17T08:51:56.700414+0000","last_unstale":"2024-04-17T08:51:57.707244+0000","last_undegraded":"2024-04-17T08:51:57.707244+0000","last_fullsized":"2024-04-17T08:51:57.707244+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:49.077510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693843+0000","last_change":"2024-04-17T08:51:41.503769+0000","last_active":"2024-04-17T08:51:57.693843+0000","last_peered":"2024-04-17T08:51:57.693843+0000","last_clean":"2024-04-17T08:51:57.693843+0000","last_became_active":"2024-04-17T08:51:41.503051+0000","last_became_peered":"2024-04-17T08:51:41.503051+0000","last_unstale":"2024-04-17T08:51:57.693843+0000","last_undegraded":"2024-04-17T08:51:57.693843+0000","last_fullsized":"2024-04-17T08:51:57.693843+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:22:40.349134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689547+0000","last_change":"2024-04-17T08:51:40.482915+0000","last_active":"2024-04-17T08:51:57.689547+0000","last_peered":"2024-04-17T08:51:57.689547+0000","last_clean":"2024-04-17T08:51:57.689547+0000","last_became_active":"2024-04-17T08:51:40.482629+0000","last_became_peered":"2024-04-17T08:51:40.482629+0000","last_unstale":"2024-04-17T08:51:57.689547+0000","last_undegraded":"2024-04-17T08:51:57.689547+0000","last_fullsized":"2024-04-17T08:51:57.689547+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.115195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695935+0000","last_change":"2024-04-17T08:51:56.686110+0000","last_active":"2024-04-17T08:51:57.695935+0000","last_peered":"2024-04-17T08:51:57.695935+0000","last_clean":"2024-04-17T08:51:57.695935+0000","last_became_active":"2024-04-17T08:51:56.685801+0000","last_became_peered":"2024-04-17T08:51:56.685801+0000","last_unstale":"2024-04-17T08:51:57.695935+0000","last_undegraded":"2024-04-17T08:51:57.695935+0000","last_fullsized":"2024-04-17T08:51:57.695935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:04.357822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692445+0000","last_change":"2024-04-17T08:51:41.518701+0000","last_active":"2024-04-17T08:51:57.692445+0000","last_peered":"2024-04-17T08:51:57.692445+0000","last_clean":"2024-04-17T08:51:57.692445+0000","last_became_active":"2024-04-17T08:51:41.516124+0000","last_became_peered":"2024-04-17T08:51:41.516124+0000","last_unstale":"2024-04-17T08:51:57.692445+0000","last_undegraded":"2024-04-17T08:51:57.692445+0000","last_fullsized":"2024-04-17T08:51:57.692445+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:35.827445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.950789+0000","last_change":"2024-04-17T08:51:40.483946+0000","last_active":"2024-04-17T08:51:57.950789+0000","last_peered":"2024-04-17T08:51:57.950789+0000","last_clean":"2024-04-17T08:51:57.950789+0000","last_became_active":"2024-04-17T08:51:40.483664+0000","last_became_peered":"2024-04-17T08:51:40.483664+0000","last_unstale":"2024-04-17T08:51:57.950789+0000","last_undegraded":"2024-04-17T08:51:57.950789+0000","last_fullsized":"2024-04-17T08:51:57.950789+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:15:55.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337273+0000","last_change":"2024-04-17T08:51:56.701992+0000","last_active":"2024-04-17T08:51:58.337273+0000","last_peered":"2024-04-17T08:51:58.337273+0000","last_clean":"2024-04-17T08:51:58.337273+0000","last_became_active":"2024-04-17T08:51:56.701643+0000","last_became_peered":"2024-04-17T08:51:56.701643+0000","last_unstale":"2024-04-17T08:51:58.337273+0000","last_undegraded":"2024-04-17T08:51:58.337273+0000","last_fullsized":"2024-04-17T08:51:58.337273+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:45.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672229+0000","last_change":"2024-04-17T08:51:41.519618+0000","last_active":"2024-04-17T08:51:56.672229+0000","last_peered":"2024-04-17T08:51:56.672229+0000","last_clean":"2024-04-17T08:51:56.672229+0000","last_became_active":"2024-04-17T08:51:41.519247+0000","last_became_peered":"2024-04-17T08:51:41.519247+0000","last_unstale":"2024-04-17T08:51:56.672229+0000","last_undegraded":"2024-04-17T08:51:56.672229+0000","last_fullsized":"2024-04-17T08:51:56.672229+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:32.995157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689635+0000","last_change":"2024-04-17T08:51:40.486177+0000","last_active":"2024-04-17T08:51:57.689635+0000","last_peered":"2024-04-17T08:51:57.689635+0000","last_clean":"2024-04-17T08:51:57.689635+0000","last_became_active":"2024-04-17T08:51:40.484934+0000","last_became_peered":"2024-04-17T08:51:40.484934+0000","last_unstale":"2024-04-17T08:51:57.689635+0000","last_undegraded":"2024-04-17T08:51:57.689635+0000","last_fullsized":"2024-04-17T08:51:57.689635+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:23.956487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337705+0000","last_change":"2024-04-17T08:51:56.698271+0000","last_active":"2024-04-17T08:51:58.337705+0000","last_peered":"2024-04-17T08:51:58.337705+0000","last_clean":"2024-04-17T08:51:58.337705+0000","last_became_active":"2024-04-17T08:51:56.697957+0000","last_became_peered":"2024-04-17T08:51:56.697957+0000","last_unstale":"2024-04-17T08:51:58.337705+0000","last_undegraded":"2024-04-17T08:51:58.337705+0000","last_fullsized":"2024-04-17T08:51:58.337705+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:34:48.178839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.688452+0000","last_change":"2024-04-17T08:51:41.517111+0000","last_active":"2024-04-17T08:51:57.688452+0000","last_peered":"2024-04-17T08:51:57.688452+0000","last_clean":"2024-04-17T08:51:57.688452+0000","last_became_active":"2024-04-17T08:51:41.516756+0000","last_became_peered":"2024-04-17T08:51:41.516756+0000","last_unstale":"2024-04-17T08:51:57.688452+0000","last_undegraded":"2024-04-17T08:51:57.688452+0000","last_fullsized":"2024-04-17T08:51:57.688452+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:17:58.048828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.666536+0000","last_change":"2024-04-17T08:51:40.479934+0000","last_active":"2024-04-17T08:51:56.666536+0000","last_peered":"2024-04-17T08:51:56.666536+0000","last_clean":"2024-04-17T08:51:56.666536+0000","last_became_active":"2024-04-17T08:51:40.479295+0000","last_became_peered":"2024-04-17T08:51:40.479295+0000","last_unstale":"2024-04-17T08:51:56.666536+0000","last_undegraded":"2024-04-17T08:51:56.666536+0000","last_fullsized":"2024-04-17T08:51:56.666536+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:40.469305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689935+0000","last_change":"2024-04-17T08:51:56.687789+0000","last_active":"2024-04-17T08:51:57.689935+0000","last_peered":"2024-04-17T08:51:57.689935+0000","last_clean":"2024-04-17T08:51:57.689935+0000","last_became_active":"2024-04-17T08:51:56.686913+0000","last_became_peered":"2024-04-17T08:51:56.686913+0000","last_unstale":"2024-04-17T08:51:57.689935+0000","last_undegraded":"2024-04-17T08:51:57.689935+0000","last_fullsized":"2024-04-17T08:51:57.689935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:28.101979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693898+0000","last_change":"2024-04-17T08:51:41.500141+0000","last_active":"2024-04-17T08:51:57.693898+0000","last_peered":"2024-04-17T08:51:57.693898+0000","last_clean":"2024-04-17T08:51:57.693898+0000","last_became_active":"2024-04-17T08:51:41.499513+0000","last_became_peered":"2024-04-17T08:51:41.499513+0000","last_unstale":"2024-04-17T08:51:57.693898+0000","last_undegraded":"2024-04-17T08:51:57.693898+0000","last_fullsized":"2024-04-17T08:51:57.693898+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336601+0000","last_change":"2024-04-17T08:51:40.478664+0000","last_active":"2024-04-17T08:51:58.336601+0000","last_peered":"2024-04-17T08:51:58.336601+0000","last_clean":"2024-04-17T08:51:58.336601+0000","last_became_active":"2024-04-17T08:51:40.477953+0000","last_became_peered":"2024-04-17T08:51:40.477953+0000","last_unstale":"2024-04-17T08:51:58.336601+0000","last_undegraded":"2024-04-17T08:51:58.336601+0000","last_fullsized":"2024-04-17T08:51:58.336601+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:59:54.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.691971+0000","last_change":"2024-04-17T08:51:56.703540+0000","last_active":"2024-04-17T08:51:57.691971+0000","last_peered":"2024-04-17T08:51:57.691971+0000","last_clean":"2024-04-17T08:51:57.691971+0000","last_became_active":"2024-04-17T08:51:56.703086+0000","last_became_peered":"2024-04-17T08:51:56.703086+0000","last_unstale":"2024-04-17T08:51:57.691971+0000","last_undegraded":"2024-04-17T08:51:57.691971+0000","last_fullsized":"2024-04-17T08:51:57.691971+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:39.357009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.338933+0000","last_change":"2024-04-17T08:51:41.515604+0000","last_active":"2024-04-17T08:51:58.338933+0000","last_peered":"2024-04-17T08:51:58.338933+0000","last_clean":"2024-04-17T08:51:58.338933+0000","last_became_active":"2024-04-17T08:51:41.514707+0000","last_became_peered":"2024-04-17T08:51:41.514707+0000","last_unstale":"2024-04-17T08:51:58.338933+0000","last_undegraded":"2024-04-17T08:51:58.338933+0000","last_fullsized":"2024-04-17T08:51:58.338933+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:26:43.231806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":24,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.708325+0000","last_change":"2024-04-17T08:51:40.481823+0000","last_active":"2024-04-17T08:51:57.708325+0000","last_peered":"2024-04-17T08:51:57.708325+0000","last_clean":"2024-04-17T08:51:57.708325+0000","last_became_active":"2024-04-17T08:51:40.481302+0000","last_became_peered":"2024-04-17T08:51:40.481302+0000","last_unstale":"2024-04-17T08:51:57.708325+0000","last_undegraded":"2024-04-17T08:51:57.708325+0000","last_fullsized":"2024-04-17T08:51:57.708325+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:56.473928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336214+0000","last_change":"2024-04-17T08:51:56.689127+0000","last_active":"2024-04-17T08:51:58.336214+0000","last_peered":"2024-04-17T08:51:58.336214+0000","last_clean":"2024-04-17T08:51:58.336214+0000","last_became_active":"2024-04-17T08:51:56.687803+0000","last_became_peered":"2024-04-17T08:51:56.687803+0000","last_unstale":"2024-04-17T08:51:58.336214+0000","last_undegraded":"2024-04-17T08:51:58.336214+0000","last_fullsized":"2024-04-17T08:51:58.336214+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:56:39.357009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.672007+0000","last_change":"2024-04-17T08:51:41.498889+0000","last_active":"2024-04-17T08:51:56.672007+0000","last_peered":"2024-04-17T08:51:56.672007+0000","last_clean":"2024-04-17T08:51:56.672007+0000","last_became_active":"2024-04-17T08:51:41.498622+0000","last_became_peered":"2024-04-17T08:51:41.498622+0000","last_unstale":"2024-04-17T08:51:56.672007+0000","last_undegraded":"2024-04-17T08:51:56.672007+0000","last_fullsized":"2024-04-17T08:51:56.672007+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:59:42.312039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":23,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.695001+0000","last_change":"2024-04-17T08:51:40.482579+0000","last_active":"2024-04-17T08:51:57.695001+0000","last_peered":"2024-04-17T08:51:57.695001+0000","last_clean":"2024-04-17T08:51:57.695001+0000","last_became_active":"2024-04-17T08:51:40.482255+0000","last_became_peered":"2024-04-17T08:51:40.482255+0000","last_unstale":"2024-04-17T08:51:57.695001+0000","last_undegraded":"2024-04-17T08:51:57.695001+0000","last_fullsized":"2024-04-17T08:51:57.695001+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:37.553047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.699071+0000","last_change":"2024-04-17T08:51:56.693002+0000","last_active":"2024-04-17T08:51:56.699071+0000","last_peered":"2024-04-17T08:51:56.699071+0000","last_clean":"2024-04-17T08:51:56.699071+0000","last_became_active":"2024-04-17T08:51:56.692691+0000","last_became_peered":"2024-04-17T08:51:56.692691+0000","last_unstale":"2024-04-17T08:51:56.699071+0000","last_undegraded":"2024-04-17T08:51:56.699071+0000","last_fullsized":"2024-04-17T08:51:56.699071+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:23.337727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689212+0000","last_change":"2024-04-17T08:51:41.514216+0000","last_active":"2024-04-17T08:51:57.689212+0000","last_peered":"2024-04-17T08:51:57.689212+0000","last_clean":"2024-04-17T08:51:57.689212+0000","last_became_active":"2024-04-17T08:51:41.513937+0000","last_became_peered":"2024-04-17T08:51:41.513937+0000","last_unstale":"2024-04-17T08:51:57.689212+0000","last_undegraded":"2024-04-17T08:51:57.689212+0000","last_fullsized":"2024-04-17T08:51:57.689212+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:27.955059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.667033+0000","last_change":"2024-04-17T08:51:40.482279+0000","last_active":"2024-04-17T08:51:56.667033+0000","last_peered":"2024-04-17T08:51:56.667033+0000","last_clean":"2024-04-17T08:51:56.667033+0000","last_became_active":"2024-04-17T08:51:40.482002+0000","last_became_peered":"2024-04-17T08:51:40.482002+0000","last_unstale":"2024-04-17T08:51:56.667033+0000","last_undegraded":"2024-04-17T08:51:56.667033+0000","last_fullsized":"2024-04-17T08:51:56.667033+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:56:36.988089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693959+0000","last_change":"2024-04-17T08:51:56.689516+0000","last_active":"2024-04-17T08:51:57.693959+0000","last_peered":"2024-04-17T08:51:57.693959+0000","last_clean":"2024-04-17T08:51:57.693959+0000","last_became_active":"2024-04-17T08:51:56.689237+0000","last_became_peered":"2024-04-17T08:51:56.689237+0000","last_unstale":"2024-04-17T08:51:57.693959+0000","last_undegraded":"2024-04-17T08:51:57.693959+0000","last_fullsized":"2024-04-17T08:51:57.693959+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:32.194887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689689+0000","last_change":"2024-04-17T08:51:41.505389+0000","last_active":"2024-04-17T08:51:57.689689+0000","last_peered":"2024-04-17T08:51:57.689689+0000","last_clean":"2024-04-17T08:51:57.689689+0000","last_became_active":"2024-04-17T08:51:41.503336+0000","last_became_peered":"2024-04-17T08:51:41.503336+0000","last_unstale":"2024-04-17T08:51:57.689689+0000","last_undegraded":"2024-04-17T08:51:57.689689+0000","last_fullsized":"2024-04-17T08:51:57.689689+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:16.345152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.689671+0000","last_change":"2024-04-17T08:51:40.480656+0000","last_active":"2024-04-17T08:51:57.689671+0000","last_peered":"2024-04-17T08:51:57.689671+0000","last_clean":"2024-04-17T08:51:57.689671+0000","last_became_active":"2024-04-17T08:51:40.480386+0000","last_became_peered":"2024-04-17T08:51:40.480386+0000","last_unstale":"2024-04-17T08:51:57.689671+0000","last_undegraded":"2024-04-17T08:51:57.689671+0000","last_fullsized":"2024-04-17T08:51:57.689671+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.113174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692690+0000","last_change":"2024-04-17T08:51:56.684435+0000","last_active":"2024-04-17T08:51:57.692690+0000","last_peered":"2024-04-17T08:51:57.692690+0000","last_clean":"2024-04-17T08:51:57.692690+0000","last_became_active":"2024-04-17T08:51:56.684092+0000","last_became_peered":"2024-04-17T08:51:56.684092+0000","last_unstale":"2024-04-17T08:51:57.692690+0000","last_undegraded":"2024-04-17T08:51:57.692690+0000","last_fullsized":"2024-04-17T08:51:57.692690+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:53.241095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336729+0000","last_change":"2024-04-17T08:51:41.513350+0000","last_active":"2024-04-17T08:51:58.336729+0000","last_peered":"2024-04-17T08:51:58.336729+0000","last_clean":"2024-04-17T08:51:58.336729+0000","last_became_active":"2024-04-17T08:51:41.512524+0000","last_became_peered":"2024-04-17T08:51:41.512524+0000","last_unstale":"2024-04-17T08:51:58.336729+0000","last_undegraded":"2024-04-17T08:51:58.336729+0000","last_fullsized":"2024-04-17T08:51:58.336729+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:39.456694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.337701+0000","last_change":"2024-04-17T08:51:40.480678+0000","last_active":"2024-04-17T08:51:58.337701+0000","last_peered":"2024-04-17T08:51:58.337701+0000","last_clean":"2024-04-17T08:51:58.337701+0000","last_became_active":"2024-04-17T08:51:40.480360+0000","last_became_peered":"2024-04-17T08:51:40.480360+0000","last_unstale":"2024-04-17T08:51:58.337701+0000","last_undegraded":"2024-04-17T08:51:58.337701+0000","last_fullsized":"2024-04-17T08:51:58.337701+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:42:50.970831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694977+0000","last_change":"2024-04-17T08:51:56.691780+0000","last_active":"2024-04-17T08:51:57.694977+0000","last_peered":"2024-04-17T08:51:57.694977+0000","last_clean":"2024-04-17T08:51:57.694977+0000","last_became_active":"2024-04-17T08:51:56.690914+0000","last_became_peered":"2024-04-17T08:51:56.690914+0000","last_unstale":"2024-04-17T08:51:57.694977+0000","last_undegraded":"2024-04-17T08:51:57.694977+0000","last_fullsized":"2024-04-17T08:51:57.694977+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:35:20.936737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693668+0000","last_change":"2024-04-17T08:51:41.501267+0000","last_active":"2024-04-17T08:51:57.693668+0000","last_peered":"2024-04-17T08:51:57.693668+0000","last_clean":"2024-04-17T08:51:57.693668+0000","last_became_active":"2024-04-17T08:51:41.500983+0000","last_became_peered":"2024-04-17T08:51:41.500983+0000","last_unstale":"2024-04-17T08:51:57.693668+0000","last_undegraded":"2024-04-17T08:51:57.693668+0000","last_fullsized":"2024-04-17T08:51:57.693668+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:48:53.940422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.707306+0000","last_change":"2024-04-17T08:51:40.477982+0000","last_active":"2024-04-17T08:51:57.707306+0000","last_peered":"2024-04-17T08:51:57.707306+0000","last_clean":"2024-04-17T08:51:57.707306+0000","last_became_active":"2024-04-17T08:51:40.477717+0000","last_became_peered":"2024-04-17T08:51:40.477717+0000","last_unstale":"2024-04-17T08:51:57.707306+0000","last_undegraded":"2024-04-17T08:51:57.707306+0000","last_fullsized":"2024-04-17T08:51:57.707306+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:51.232388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":9,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.689690+0000","last_change":"2024-04-17T08:51:56.689119+0000","last_active":"2024-04-17T08:51:56.689690+0000","last_peered":"2024-04-17T08:51:56.689690+0000","last_clean":"2024-04-17T08:51:56.689690+0000","last_became_active":"2024-04-17T08:51:56.688451+0000","last_became_peered":"2024-04-17T08:51:56.688451+0000","last_unstale":"2024-04-17T08:51:56.689690+0000","last_undegraded":"2024-04-17T08:51:56.689690+0000","last_fullsized":"2024-04-17T08:51:56.689690+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:47.737093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.693122+0000","last_change":"2024-04-17T08:51:41.504227+0000","last_active":"2024-04-17T08:51:57.693122+0000","last_peered":"2024-04-17T08:51:57.693122+0000","last_clean":"2024-04-17T08:51:57.693122+0000","last_became_active":"2024-04-17T08:51:41.503919+0000","last_became_peered":"2024-04-17T08:51:41.503919+0000","last_unstale":"2024-04-17T08:51:57.693122+0000","last_undegraded":"2024-04-17T08:51:57.693122+0000","last_fullsized":"2024-04-17T08:51:57.693122+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:30.922580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949128+0000","last_change":"2024-04-17T08:51:40.466461+0000","last_active":"2024-04-17T08:51:57.949128+0000","last_peered":"2024-04-17T08:51:57.949128+0000","last_clean":"2024-04-17T08:51:57.949128+0000","last_became_active":"2024-04-17T08:51:40.466050+0000","last_became_peered":"2024-04-17T08:51:40.466050+0000","last_unstale":"2024-04-17T08:51:57.949128+0000","last_undegraded":"2024-04-17T08:51:57.949128+0000","last_fullsized":"2024-04-17T08:51:57.949128+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:17.871122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.692724+0000","last_change":"2024-04-17T08:51:56.688449+0000","last_active":"2024-04-17T08:51:57.692724+0000","last_peered":"2024-04-17T08:51:57.692724+0000","last_clean":"2024-04-17T08:51:57.692724+0000","last_became_active":"2024-04-17T08:51:56.687978+0000","last_became_peered":"2024-04-17T08:51:56.687978+0000","last_unstale":"2024-04-17T08:51:57.692724+0000","last_undegraded":"2024-04-17T08:51:57.692724+0000","last_fullsized":"2024-04-17T08:51:57.692724+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:51.427089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.668942+0000","last_change":"2024-04-17T08:51:41.509452+0000","last_active":"2024-04-17T08:51:56.668942+0000","last_peered":"2024-04-17T08:51:56.668942+0000","last_clean":"2024-04-17T08:51:56.668942+0000","last_became_active":"2024-04-17T08:51:41.509160+0000","last_became_peered":"2024-04-17T08:51:41.509160+0000","last_unstale":"2024-04-17T08:51:56.668942+0000","last_undegraded":"2024-04-17T08:51:56.668942+0000","last_fullsized":"2024-04-17T08:51:56.668942+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:54.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":22,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.949147+0000","last_change":"2024-04-17T08:51:40.481750+0000","last_active":"2024-04-17T08:51:57.949147+0000","last_peered":"2024-04-17T08:51:57.949147+0000","last_clean":"2024-04-17T08:51:57.949147+0000","last_became_active":"2024-04-17T08:51:40.481311+0000","last_became_peered":"2024-04-17T08:51:40.481311+0000","last_unstale":"2024-04-17T08:51:57.949147+0000","last_undegraded":"2024-04-17T08:51:57.949147+0000","last_fullsized":"2024-04-17T08:51:57.949147+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:46:50.295078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:58.336053+0000","last_change":"2024-04-17T08:51:56.696688+0000","last_active":"2024-04-17T08:51:58.336053+0000","last_peered":"2024-04-17T08:51:58.336053+0000","last_clean":"2024-04-17T08:51:58.336053+0000","last_became_active":"2024-04-17T08:51:56.696415+0000","last_became_peered":"2024-04-17T08:51:56.696415+0000","last_unstale":"2024-04-17T08:51:58.336053+0000","last_undegraded":"2024-04-17T08:51:58.336053+0000","last_fullsized":"2024-04-17T08:51:58.336053+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:55.619763+0000","last_clean_scrub_stamp":"2024-04-17T08:51:55.619763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:53.876334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.694536+0000","last_change":"2024-04-17T08:51:41.490585+0000","last_active":"2024-04-17T08:51:57.694536+0000","last_peered":"2024-04-17T08:51:57.694536+0000","last_clean":"2024-04-17T08:51:57.694536+0000","last_became_active":"2024-04-17T08:51:41.489827+0000","last_became_peered":"2024-04-17T08:51:41.489827+0000","last_unstale":"2024-04-17T08:51:57.694536+0000","last_undegraded":"2024-04-17T08:51:57.694536+0000","last_fullsized":"2024-04-17T08:51:57.694536+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:40.436081+0000","last_clean_scrub_stamp":"2024-04-17T08:51:40.436081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:17:35.513878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-04-17T08:51:57.951326+0000","last_change":"2024-04-17T08:51:40.482218+0000","last_active":"2024-04-17T08:51:57.951326+0000","last_peered":"2024-04-17T08:51:57.951326+0000","last_clean":"2024-04-17T08:51:57.951326+0000","last_became_active":"2024-04-17T08:51:40.481390+0000","last_became_peered":"2024-04-17T08:51:40.481390+0000","last_unstale":"2024-04-17T08:51:57.951326+0000","last_undegraded":"2024-04-17T08:51:57.951326+0000","last_fullsized":"2024-04-17T08:51:57.951326+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:51:39.422496+0000","last_clean_scrub_stamp":"2024-04-17T08:51:39.422496+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:41.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"236'376","reported_seq":1571,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-04-17T08:51:56.664689+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:51:56.664689+0000","last_peered":"2024-04-17T08:51:56.664689+0000","last_clean":"2024-04-17T08:51:56.664689+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:51:56.664689+0000","last_undegraded":"2024-04-17T08:51:56.664689+0000","last_fullsized":"2024-04-17T08:51:56.664689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2370,"num_read_kb":5002,"num_write":627,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":376,"ondisk_log_size":376,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543309,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57084,"kb_used_data":3692,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314756,"statfs":{"total":96636764160,"available":96578310144,"internally_reserved":0,"allocated":3780608,"data_stored":2154396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543309,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60688,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311152,"statfs":{"total":96636764160,"available":96574619648,"internally_reserved":0,"allocated":3194880,"data_stored":1564604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543308,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30140,"kb_used_data":3116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341700,"statfs":{"total":96636764160,"available":96605900800,"internally_reserved":0,"allocated":3190784,"data_stored":1571083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543308,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3231744,"data_stored":1575700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543310,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52372,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319468,"statfs":{"total":96636764160,"available":96583135232,"internally_reserved":0,"allocated":3211264,"data_stored":1573859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543309,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52356,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319484,"statfs":{"total":96636764160,"available":96583151616,"internally_reserved":0,"allocated":3194880,"data_stored":1571173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543310,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64848,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306992,"statfs":{"total":96636764160,"available":96570359808,"internally_reserved":0,"allocated":3215360,"data_stored":1567645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543308,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61248,"kb_used_data":3700,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310592,"statfs":{"total":96636764160,"available":96574046208,"internally_reserved":0,"allocated":3788800,"data_stored":2161485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:52:01.818 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:52:01.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:02.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.084+0000 7f4802200640 1 Processor -- start 2024-04-17T08:52:02.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.085+0000 7f4802200640 1 -- start start 2024-04-17T08:52:02.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.086+0000 7f4802200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47fc0b6d00 0x7f47fc0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.086+0000 7f4802200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 0x7f47fc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.086+0000 7f4802200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47fc0d2d30 con 0x7f47fc15ec60 2024-04-17T08:52:02.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.086+0000 7f4802200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f47fc0d49c0 con 0x7f47fc0b6d00 2024-04-17T08:52:02.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.086+0000 7f4802200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc0d3660 con 0x7f47fc0cf910 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f47fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47fc0b6d00 0x7f47fc0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f4800e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47fc15ec60 0x7f47fc0b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33386/0 (socket says 172.21.15.67:33386) 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f4800e00640 1 -- 172.21.15.67:0/2195071756 learned_addr learned my addr 172.21.15.67:0/2195071756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f4801800640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 0x7f47fc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f47fbe00640 1 -- 172.21.15.67:0/2195071756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 msgr2=0x7f47fc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.087+0000 7f47fbe00640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 0x7f47fc0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.088+0000 7f47fbe00640 1 -- 172.21.15.67:0/2195071756 >> v1:172.21.15.67:6789/0 conn(0x7f47fc15ec60 legacy=0x7f47fc0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.088+0000 7f47fbe00640 1 -- 172.21.15.67:0/2195071756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc0d3010 con 0x7f47fc0b6d00 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.088+0000 7f47fb400640 1 -- 172.21.15.67:0/2195071756 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186263658 0 0) 0x7f47fc0d2d30 con 0x7f47fc15ec60 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.088+0000 7f4801800640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 0x7f47fc0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.089+0000 7f47fbe00640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0b6d00 0x7f47fc0cf3d0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009d90 tx=0x7f47e802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283602e8c31a7d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.089+0000 7f47fb400640 1 -- 172.21.15.67:0/2195071756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8039070 con 0x7f47fc0b6d00 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.089+0000 7f47fb400640 1 -- 172.21.15.67:0/2195071756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e802cc60 con 0x7f47fc0b6d00 2024-04-17T08:52:02.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.089+0000 7f47fb400640 1 -- 172.21.15.67:0/2195071756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8034660 con 0x7f47fc0b6d00 2024-04-17T08:52:02.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 -- 172.21.15.67:0/2195071756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0b6d00 msgr2=0x7f47fc0cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0b6d00 0x7f47fc0cf3d0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009d90 tx=0x7f47e802beb0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 -- 172.21.15.67:0/2195071756 shutdown_connections 2024-04-17T08:52:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0cf910 0x7f47fc0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 --2- 172.21.15.67:0/2195071756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0b6d00 0x7f47fc0cf3d0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 -- 172.21.15.67:0/2195071756 >> 172.21.15.67:0/2195071756 conn(0x7f47fc0d15f0 msgr2=0x7f47fc0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 -- 172.21.15.67:0/2195071756 shutdown_connections 2024-04-17T08:52:02.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.090+0000 7f4802200640 1 -- 172.21.15.67:0/2195071756 wait complete. 2024-04-17T08:52:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.092+0000 7f4802200640 1 Processor -- start 2024-04-17T08:52:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.092+0000 7f4802200640 1 -- start start 2024-04-17T08:52:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0b6d00 0x7f47fc0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 0x7f47fc14adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc15f040 con 0x7f47fc15ec60 2024-04-17T08:52:02.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0cfdc0 con 0x7f47fc0cf910 2024-04-17T08:52:02.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4802200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc0d00c0 con 0x7f47fc0b6d00 2024-04-17T08:52:02.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4801800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4801800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37004/0 (socket says 172.21.15.67:37004) 2024-04-17T08:52:02.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.093+0000 7f4801800640 1 -- 172.21.15.67:0/1884511702 learned_addr learned my addr 172.21.15.67:0/1884511702 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:02.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4800e00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0b6d00 0x7f47fc0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f47fbe00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 0x7f47fc14adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 -- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0b6d00 msgr2=0x7f47fc0cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0b6d00 0x7f47fc0cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 -- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 msgr2=0x7f47fc14adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 0x7f47fc14adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 -- 172.21.15.67:0/1884511702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc0cdda0 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f47fbe00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 0x7f47fc14adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.094+0000 7f4801800640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f47f400c7f0 tx=0x7f47f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.095+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4015070 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.095+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f40077c0 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.095+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007bf0 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.095+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc0ce080 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.095+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc15fac0 con 0x7f47fc15ec60 2024-04-17T08:52:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.096+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc068a10 con 0x7f47fc15ec60 2024-04-17T08:52:02.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.097+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f47f4018070 con 0x7f47fc15ec60 2024-04-17T08:52:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.098+0000 7f47f8c00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 0x7f47dc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:02.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.098+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f47f408c6a0 con 0x7f47fc15ec60 2024-04-17T08:52:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.099+0000 7f4800e00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 0x7f47dc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.100+0000 7f4800e00640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 0x7f47dc06eff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000f60 tx=0x7f47ec007c00 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:02.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.102+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f47f405a1b0 con 0x7f47fc15ec60 2024-04-17T08:52:02.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.351+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47fc15b650 con 0x7f47fc15ec60 2024-04-17T08:52:02.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.352+0000 7f47f8c00640 1 -- 172.21.15.67:0/1884511702 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7f47f4059ab0 con 0x7f47fc15ec60 2024-04-17T08:52:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:02.352 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:02.352 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.355+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 msgr2=0x7f47dc06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.355+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 0x7f47dc06eff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000f60 tx=0x7f47ec007c00 comp rx=0 tx=0).stop 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.355+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 msgr2=0x7f47fc154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.355+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f47f400c7f0 tx=0x7f47f400ccb0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 shutdown_connections 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f47dc06cb30 0x7f47dc06eff0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc15ec60 0x7f47fc154ab0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47fc0cf910 0x7f47fc14adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 --2- 172.21.15.67:0/1884511702 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47fc0b6d00 0x7f47fc0cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:02.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 >> 172.21.15.67:0/1884511702 conn(0x7f47fc0d15f0 msgr2=0x7f47fc0b86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:02.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 shutdown_connections 2024-04-17T08:52:02.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:02.356+0000 7f4802200640 1 -- 172.21.15.67:0/1884511702 wait complete. 2024-04-17T08:52:02.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:51:56.128948+0000', 'modified': '2024-04-17T08:51:59.267006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11841}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11841': {'gid': 11841, 'name': 'c', 'rank': 0, 'incarnation': 128, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.67:6833/4106648933', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 4106648933}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 4106648933}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11841, 'qdb_cluster': [11841]} 2024-04-17T08:52:02.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:52:02.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T08:52:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.010+0000 7f20ddc00640 1 Processor -- start 2024-04-17T08:52:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.010+0000 7f20ddc00640 1 -- start start 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20ddc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20d806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20ddc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20d815fad0 0x7f20d80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20ddc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80cb420 con 0x7f20d806eec0 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20ddc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20d80cc030 con 0x7f20d815fad0 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20ddc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80cb700 con 0x7f20d80afbf0 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20cf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20d806eec0 0x7f20d815f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33396/0 (socket says 172.21.15.67:33396) 2024-04-17T08:52:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20cf400640 1 -- 172.21.15.67:0/178058493 learned_addr learned my addr 172.21.15.67:0/178058493 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20d7e00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20d806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.012+0000 7f20dd200640 1 --2- 172.21.15.67:0/178058493 >> v2:172.21.15.187:3300/0 conn(0x7f20d815fad0 0x7f20d80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83120380 0 0) 0x7f20d80cb420 con 0x7f20d806eec0 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc003520 con 0x7f20d806eec0 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 591490328 0 0) 0x7f20bc003520 con 0x7f20d806eec0 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 msgr2=0x7f20d806e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20d806e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 >> v2:172.21.15.187:3300/0 conn(0x7f20d815fad0 msgr2=0x7f20d80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 --2- 172.21.15.67:0/178058493 >> v2:172.21.15.187:3300/0 conn(0x7f20d815fad0 0x7f20d80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80cb9e0 con 0x7f20d806eec0 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f20c0001670 con 0x7f20d806eec0 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 >> v1:172.21.15.67:6789/0 conn(0x7f20d806eec0 legacy=0x7f20d815f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc003a20 0x7f20bc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80cc030 con 0x7f20bc003a20 2024-04-17T08:52:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80cb700 con 0x7f20bc004800 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.013+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20bc00dc00 con 0x7f20d80afbf0 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc003a20 0x7f20bc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20cf400640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20dd200640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 -- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 msgr2=0x7f20bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d80afbf0 0x7f20bc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 -- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc004800 msgr2=0x7f20bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.014+0000 7f20d7e00640 1 -- 172.21.15.67:0/178058493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc003520 con 0x7f20bc003a20 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.015+0000 7f20d7e00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc003a20 0x7f20bc003e40 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009fc0 tx=0x7f20bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c82526c8dda09d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.015+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8039070 con 0x7f20bc003a20 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.015+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c802cc20 con 0x7f20bc003a20 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20d7400640 1 -- 172.21.15.67:0/178058493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8009c10 con 0x7f20bc003a20 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 -- 172.21.15.67:0/178058493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc003a20 msgr2=0x7f20bc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 --2- 172.21.15.67:0/178058493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20bc003a20 0x7f20bc003e40 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009fc0 tx=0x7f20bc005830 comp rx=0 tx=0).stop 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20dd200640 1 -- 172.21.15.67:0/178058493 reap_dead start 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 -- 172.21.15.67:0/178058493 shutdown_connections 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 -- 172.21.15.67:0/178058493 >> 172.21.15.67:0/178058493 conn(0x7f20d806bef0 msgr2=0x7f20d80b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 -- 172.21.15.67:0/178058493 shutdown_connections 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.016+0000 7f20ddc00640 1 -- 172.21.15.67:0/178058493 wait complete. 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.018+0000 7f20ddc00640 1 Processor -- start 2024-04-17T08:52:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.018+0000 7f20ddc00640 1 -- start start 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc003a20 0x7f20d81eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d80b47d0 0x7f20d81f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80affd0 con 0x7f20d8154cd0 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20d8160110 con 0x7f20d80b47d0 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.019+0000 7f20ddc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80b0150 con 0x7f20bc003a20 2024-04-17T08:52:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37022/0 (socket says 172.21.15.67:37022) 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 -- 172.21.15.67:0/2517665103 learned_addr learned my addr 172.21.15.67:0/2517665103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 -- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc003a20 msgr2=0x7f20d81eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20d7e00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc003a20 0x7f20d81eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc003a20 0x7f20d81eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 -- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d80b47d0 msgr2=0x7f20d81f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d80b47d0 0x7f20d81f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 -- 172.21.15.67:0/2517665103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80b3a30 con 0x7f20d8154cd0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.020+0000 7f20dd200640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f20d80cd0e0 tx=0x7f20d0010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.021+0000 7f20d4c00640 1 -- 172.21.15.67:0/2517665103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d000f040 con 0x7f20d8154cd0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.021+0000 7f20d4c00640 1 -- 172.21.15.67:0/2517665103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d001c070 con 0x7f20d8154cd0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.021+0000 7f20d4c00640 1 -- 172.21.15.67:0/2517665103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d000eb00 con 0x7f20d8154cd0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.021+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d80b57d0 con 0x7f20d8154cd0 2024-04-17T08:52:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.021+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d80b3fd0 con 0x7f20d8154cd0 2024-04-17T08:52:03.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.023+0000 7f20d4c00640 1 -- 172.21.15.67:0/2517665103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7f20d000eca0 con 0x7f20d8154cd0 2024-04-17T08:52:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.024+0000 7f20d4c00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 0x7f20b406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.024+0000 7f20d4c00640 1 -- 172.21.15.67:0/2517665103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f20d008bb00 con 0x7f20d8154cd0 2024-04-17T08:52:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.025+0000 7f20d7e00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 0x7f20b406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.025+0000 7f20e1fc3740 1 Processor -- start 2024-04-17T08:52:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.025+0000 7f20e1fc3740 1 -- start start 2024-04-17T08:52:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20d7e00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 0x7f20b406f020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f20c8036000 tx=0x7f20c802e000 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca75f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aca7559980 con 0x55aca75edb40 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55aca7500220 con 0x55aca75f16a0 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55aca7555940 con 0x55aca763b620 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37038/0 (socket says 172.21.15.67:37038) 2024-04-17T08:52:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.026+0000 7f20dd200640 1 -- 172.21.15.67:0/4284016732 learned_addr learned my addr 172.21.15.67:0/4284016732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20cf400640 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 -- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 msgr2=0x55aca75f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca75f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 -- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 msgr2=0x55aca75f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 -- 172.21.15.67:0/4284016732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca726c2d0 con 0x55aca75edb40 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20dd200640 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f20d0001eb0 tx=0x7f20d00590b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c8074d71902484 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20ccc00640 1 -- 172.21.15.67:0/4284016732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0061c90 con 0x55aca75edb40 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20ccc00640 1 -- 172.21.15.67:0/4284016732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0055720 con 0x55aca75edb40 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.027+0000 7f20ccc00640 1 -- 172.21.15.67:0/4284016732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d005d250 con 0x55aca75edb40 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4284016732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 msgr2=0x55aca761d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f20d0001eb0 tx=0x7f20d00590b0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4284016732 shutdown_connections 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca75edb40 0x55aca761d930 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4284016732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca75f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4284016732 >> 172.21.15.67:0/4284016732 conn(0x55aca75efdb0 msgr2=0x55aca7638930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4284016732 shutdown_connections 2024-04-17T08:52:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.028+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4284016732 wait complete. 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.029+0000 7f20e1fc3740 1 -- start start 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.029+0000 7f20e1fc3740 1 Processor -- start 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca76101c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca7618c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aca75f1bb0 con 0x55aca76142d0 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55aca7513590 con 0x55aca763b620 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55aca75f1e90 con 0x55aca75f16a0 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.030+0000 7f20e1fc3740 10 client.0 _ll_register_callbacks cb 0x55aca726bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37040/0 (socket says 172.21.15.67:37040) 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 -- 172.21.15.67:0/105982665 learned_addr learned my addr 172.21.15.67:0/105982665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20cf400640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca7618c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20d7e00640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca76101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 -- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 msgr2=0x55aca76101c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca76101c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 -- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 msgr2=0x55aca7618c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca7618c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 -- 172.21.15.67:0/105982665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca73366b0 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20cf400640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca7618c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20dd200640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f20d0055510 tx=0x7f20d000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135cd069977a9082 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.031+0000 7f20c5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0089a50 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f20d0089bf0 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20d0055720 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f20d0054e00 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 10 client.12132 fetch_fsmap learned FSMap version 132 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) -- 0x55aca764aeb0 con 0x55aca76142d0 2024-04-17T08:52:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0055720 con 0x55aca76142d0 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 132) ==== 4143+0+0 (secure 0 0 0) 0x7f20d0054fa0 con 0x55aca76142d0 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 10 client.12132 fetch_fsmap finished waiting for FSMap version 132 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 10 client.12132 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 20 client.12132 populate_metadata read hostname 'smithi067' 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.032+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca7649ad0 0x55aca7649eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.033+0000 7f20e1fc3740 4 client.12132 mds_command: new command op to 11841 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.033+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 --> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aca75673a0 con 0x55aca7649ad0 2024-04-17T08:52:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.033+0000 7f20d7e00640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca7649ad0 0x55aca7649eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.033+0000 7f20d7e00640 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca7649ad0 0x55aca7649eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c18d0909221f76 server_cookie=c3d5f1ba168d6722 in_seq=0 out_seq=0 2024-04-17T08:52:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.033+0000 7f20c5600640 10 client.12132 ms_handle_connect on v2:172.21.15.67:6832/4106648933 2024-04-17T08:52:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.034+0000 7f20c5600640 1 -- 172.21.15.67:0/105982665 <== mds.0 v2:172.21.15.67:6832/4106648933 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aca75673a0 con 0x55aca7649ad0 2024-04-17T08:52:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.034+0000 7f20c5600640 10 client.12132 handle_command_reply: tid=0 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.034+0000 7f20e1fc3740 1 client.12132 shutdown 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 msgr2=0x55aca7619170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f20d0055510 tx=0x7f20d000ef90 comp rx=0 tx=0).stop 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 shutdown_connections 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca7649ad0 0x55aca7649eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca7619170 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca7618c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/105982665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca76101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 >> 172.21.15.67:0/105982665 conn(0x55aca75edb40 msgr2=0x55aca7638730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20c5600640 0 client.12132 ms_handle_reset on v2:172.21.15.67:6832/4106648933 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 shutdown_connections 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 1 -- 172.21.15.67:0/105982665 wait complete. 2024-04-17T08:52:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.035+0000 7f20e1fc3740 20 client.12132 trim_cache size 0 max 16384 2024-04-17T08:52:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.055+0000 7f20e1fc3740 1 Processor -- start 2024-04-17T08:52:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.055+0000 7f20e1fc3740 1 -- start start 2024-04-17T08:52:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca763ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d8204620 con 0x55aca76142d0 2024-04-17T08:52:03.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20d8202040 con 0x55aca75f16a0 2024-04-17T08:52:03.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.056+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20d8202690 con 0x55aca763b620 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37050/0 (socket says 172.21.15.67:37050) 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 -- 172.21.15.67:0/2101974074 learned_addr learned my addr 172.21.15.67:0/2101974074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20d7e00640 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20dd200640 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca763ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 -- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 msgr2=0x55aca763ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca763ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 -- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 msgr2=0x55aca75f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.057+0000 7f20cf400640 1 -- 172.21.15.67:0/2101974074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca764c370 con 0x55aca76142d0 2024-04-17T08:52:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.058+0000 7f20cf400640 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f20c0001840 tx=0x7f20c0007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cadd8fd205129b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.058+0000 7f20c7e00640 1 -- 172.21.15.67:0/2101974074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c00072e0 con 0x55aca76142d0 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.058+0000 7f20c7e00640 1 -- 172.21.15.67:0/2101974074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c0007480 con 0x55aca76142d0 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.058+0000 7f20c7e00640 1 -- 172.21.15.67:0/2101974074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0004a00 con 0x55aca76142d0 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- 172.21.15.67:0/2101974074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 msgr2=0x55aca75f3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f20c0001840 tx=0x7f20c0007ca0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- 172.21.15.67:0/2101974074 shutdown_connections 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca763b620 0x55aca763ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca75f3e90 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/2101974074 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca75f16a0 0x55aca75f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- 172.21.15.67:0/2101974074 >> 172.21.15.67:0/2101974074 conn(0x55aca75edb40 msgr2=0x55aca75edff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- 172.21.15.67:0/2101974074 shutdown_connections 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- 172.21.15.67:0/2101974074 wait complete. 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.059+0000 7f20e1fc3740 1 -- start start 2024-04-17T08:52:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.060+0000 7f20e1fc3740 1 Processor -- start 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca75e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca763ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aca75f1bb0 con 0x55aca76142d0 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55aca763d000 con 0x55aca763b620 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55aca75f1e90 con 0x55aca75f16a0 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20e1fc3740 10 client.0 _ll_register_callbacks cb 0x7f20d0071040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37064/0 (socket says 172.21.15.67:37064) 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20cf400640 1 -- 172.21.15.67:0/4025078839 learned_addr learned my addr 172.21.15.67:0/4025078839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20d7e00640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca75e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.061+0000 7f20dd200640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca763ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 -- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 msgr2=0x55aca75e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca75e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 -- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 msgr2=0x55aca763ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca763ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 -- 172.21.15.67:0/4025078839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca7640ae0 con 0x55aca76142d0 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20cf400640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f20c0013000 tx=0x7f20c0000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31aa3618751774d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20c5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c00072e0 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f20c0019030 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.062+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20c0007480 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f20c0007630 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 10 client.12138 fetch_fsmap learned FSMap version 132 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) -- 0x55aca7618df0 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c0007480 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 132) ==== 4143+0+0 (secure 0 0 0) 0x7f20c000f7b0 con 0x55aca76142d0 2024-04-17T08:52:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 10 client.12138 fetch_fsmap finished waiting for FSMap version 132 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 10 client.12138 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 20 client.12138 populate_metadata read hostname 'smithi067' 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca763c190 0x55aca76146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 4 client.12138 mds_command: new command op to 11841 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 --> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] -- command(tid 0: {"prefix": "status"}) -- 0x55aca7561210 con 0x55aca763c190 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.063+0000 7f20d7e00640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca763c190 0x55aca76146b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20d7e00640 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca763c190 0x55aca76146b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a4cb4067a4c0c11 server_cookie=e48d735cf2efac75 in_seq=0 out_seq=0 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20c5600640 10 client.12138 ms_handle_connect on v2:172.21.15.67:6832/4106648933 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20c5600640 1 -- 172.21.15.67:0/4025078839 <== mds.0 v2:172.21.15.67:6832/4106648933 1 ==== command_reply(tid 0: 0 ) ==== 8+0+338 (crc 0 0 0) 0x55aca7561210 con 0x55aca763c190 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20c5600640 10 client.12138 handle_command_reply: tid=0 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 client.12138 shutdown 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 msgr2=0x55aca765d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f20c0013000 tx=0x7f20c0000f50 comp rx=0 tx=0).stop 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 shutdown_connections 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55aca763b620 0x55aca763ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:6832/4106648933,v1:172.21.15.67:6833/4106648933] conn(0x55aca763c190 0x55aca76146b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aca76142d0 0x55aca765d2e0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.064+0000 7f20e1fc3740 1 --2- 172.21.15.67:0/4025078839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55aca75f16a0 0x55aca75e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 >> 172.21.15.67:0/4025078839 conn(0x55aca75edb40 msgr2=0x55aca75f3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20c5600640 0 client.12138 ms_handle_reset on v2:172.21.15.67:6832/4106648933 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 shutdown_connections 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20e1fc3740 1 -- 172.21.15.67:0/4025078839 wait complete. 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20e1fc3740 20 client.12138 trim_cache size 0 max 16384 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:52:03.066 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 11841, 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "backup_fs", 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.3850319329999996, 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 132, 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 254, 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 254, 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 27.399084805000001 2024-04-17T08:52:03.067 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 msgr2=0x7f20b406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 0x7f20b406f020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f20c8036000 tx=0x7f20c802e000 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 msgr2=0x7f20d8155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.065+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f20d80cd0e0 tx=0x7f20d0010c20 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 shutdown_connections 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f20b406cb60 0x7f20b406f020 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d8154cd0 0x7f20d8155150 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d80b47d0 0x7f20d81f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 --2- 172.21.15.67:0/2517665103 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20bc003a20 0x7f20d81eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 >> 172.21.15.67:0/2517665103 conn(0x7f20d806bef0 msgr2=0x7f20d80cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 shutdown_connections 2024-04-17T08:52:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.066+0000 7f20ddc00640 1 -- 172.21.15.67:0/2517665103 wait complete. 2024-04-17T08:52:03.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:52:03.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.306+0000 7ff10a400640 1 Processor -- start 2024-04-17T08:52:03.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.307+0000 7ff10a400640 1 -- start start 2024-04-17T08:52:03.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.308+0000 7ff10a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff10415ee20 0x7ff1040d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.308+0000 7ff10a400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1040d4140 con 0x7ff1040d3520 2024-04-17T08:52:03.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.308+0000 7ff10a400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1040d3b80 con 0x7ff1040d18e0 2024-04-17T08:52:03.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.308+0000 7ff10a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff1040b83b0 con 0x7ff10415ee20 2024-04-17T08:52:03.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff10415ee20 0x7ff1040d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff10415ee20 0x7ff1040d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48222/0 (socket says 172.21.15.67:48222) 2024-04-17T08:52:03.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 -- 172.21.15.67:0/4255945384 learned_addr learned my addr 172.21.15.67:0/4255945384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 -- 172.21.15.67:0/4255945384 >> v1:172.21.15.187:6789/0 conn(0x7ff1040d18e0 legacy=0x7ff1040d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 -- 172.21.15.67:0/4255945384 >> v1:172.21.15.67:6789/0 conn(0x7ff1040d3520 legacy=0x7ff1040b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff109000640 1 -- 172.21.15.67:0/4255945384 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1040d3e60 con 0x7ff10415ee20 2024-04-17T08:52:03.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.309+0000 7ff103400640 1 -- 172.21.15.67:0/4255945384 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186194440 0 0) 0x7ff1040d4140 con 0x7ff1040d3520 2024-04-17T08:52:03.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff109000640 1 --2- 172.21.15.67:0/4255945384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10415ee20 0x7ff1040d13a0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbf2552555c6ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff103400640 1 -- 172.21.15.67:0/4255945384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f402f840 con 0x7ff10415ee20 2024-04-17T08:52:03.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff103400640 1 -- 172.21.15.67:0/4255945384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f402f9e0 con 0x7ff10415ee20 2024-04-17T08:52:03.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff103400640 1 -- 172.21.15.67:0/4255945384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f402fdb0 con 0x7ff10415ee20 2024-04-17T08:52:03.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff10a400640 1 -- 172.21.15.67:0/4255945384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10415ee20 msgr2=0x7ff1040d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.311+0000 7ff10a400640 1 --2- 172.21.15.67:0/4255945384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10415ee20 0x7ff1040d13a0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f402ba20 comp rx=0 tx=0).stop 2024-04-17T08:52:03.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.312+0000 7ff10a400640 1 -- 172.21.15.67:0/4255945384 shutdown_connections 2024-04-17T08:52:03.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.312+0000 7ff10a400640 1 --2- 172.21.15.67:0/4255945384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10415ee20 0x7ff1040d13a0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.312+0000 7ff10a400640 1 -- 172.21.15.67:0/4255945384 >> 172.21.15.67:0/4255945384 conn(0x7ff10406d3c0 msgr2=0x7ff1040b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.312+0000 7ff10a400640 1 -- 172.21.15.67:0/4255945384 shutdown_connections 2024-04-17T08:52:03.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.312+0000 7ff10a400640 1 -- 172.21.15.67:0/4255945384 wait complete. 2024-04-17T08:52:03.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.313+0000 7ff10a400640 1 Processor -- start 2024-04-17T08:52:03.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.313+0000 7ff10a400640 1 -- start start 2024-04-17T08:52:03.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 0x7ff1041f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff10415ee20 0x7ff1041f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff10415f200 con 0x7ff1040d18e0 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff1040d1d20 con 0x7ff1040d3520 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.314+0000 7ff10a400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1040b8f50 con 0x7ff10415ee20 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48236/0 (socket says 172.21.15.67:48236) 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 -- 172.21.15.67:0/1178143462 learned_addr learned my addr 172.21.15.67:0/1178143462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff103e00640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 0x7ff1041f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 -- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff10415ee20 msgr2=0x7ff1041f7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff10415ee20 0x7ff1041f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 -- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 msgr2=0x7ff1041f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 0x7ff1041f7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff109000640 1 -- 172.21.15.67:0/1178143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1040cb350 con 0x7ff1040d18e0 2024-04-17T08:52:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.315+0000 7ff103e00640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 0x7ff1041f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff109000640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4041400 con 0x7ff1040d18e0 2024-04-17T08:52:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4046070 con 0x7ff1040d18e0 2024-04-17T08:52:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4041800 con 0x7ff1040d18e0 2024-04-17T08:52:03.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1040c9d70 con 0x7ff1040d18e0 2024-04-17T08:52:03.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.316+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1040ca330 con 0x7ff1040d18e0 2024-04-17T08:52:03.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.317+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff104068a10 con 0x7ff1040d18e0 2024-04-17T08:52:03.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.318+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7ff0f4040070 con 0x7ff1040d18e0 2024-04-17T08:52:03.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.319+0000 7ff100c00640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 0x7ff0e406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.319+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7ff0f40aec50 con 0x7ff1040d18e0 2024-04-17T08:52:03.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.320+0000 7ff103e00640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 0x7ff0e406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.321+0000 7ff103e00640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 0x7ff0e406eff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff1040b47f0 tx=0x7ff0f0009040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.324+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff0f4033030 con 0x7ff1040d18e0 2024-04-17T08:52:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.557+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7ff1040ce3b0 con 0x7ff0e406cb30 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.561+0000 7ff100c00640 1 -- 172.21.15.67:0/1178143462 <== mgr.8926 v2:172.21.15.187:6832/2123602634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7ff1040ce3b0 con 0x7ff0e406cb30 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "backup_fs" 2024-04-17T08:52:03.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:03.561 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 msgr2=0x7ff0e406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 0x7ff0e406eff0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff1040b47f0 tx=0x7ff0f0009040 comp rx=0 tx=0).stop 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 msgr2=0x7ff1040cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f4004730 comp rx=0 tx=0).stop 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 shutdown_connections 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7ff0e406cb30 0x7ff0e406eff0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff10415ee20 0x7ff1041f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1040d3520 0x7ff1041f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 --2- 172.21.15.67:0/1178143462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1040d18e0 0x7ff1040cae10 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.563+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 >> 172.21.15.67:0/1178143462 conn(0x7ff10406d3c0 msgr2=0x7ff1040b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.564+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 shutdown_connections 2024-04-17T08:52:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.564+0000 7ff10a400640 1 -- 172.21.15.67:0/1178143462 wait complete. 2024-04-17T08:52:03.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:52:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.792+0000 7fe32f800640 1 Processor -- start 2024-04-17T08:52:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.792+0000 7fe32f800640 1 -- start start 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe328145ac0 0x7fe328071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3280ce3b0 0x7fe328072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3280ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe3280ce840 con 0x7fe3280ce3b0 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe3280cea70 con 0x7fe328145ac0 2024-04-17T08:52:03.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe32f800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe3280ced50 con 0x7fe328072bb0 2024-04-17T08:52:03.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe327e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3280ce3b0 0x7fe328072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe327e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3280ce3b0 0x7fe328072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48250/0 (socket says 172.21.15.67:48250) 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.794+0000 7fe327e00640 1 -- 172.21.15.67:0/3234254560 learned_addr learned my addr 172.21.15.67:0/3234254560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe32ce00640 1 --2- 172.21.15.67:0/3234254560 >> v2:172.21.15.187:3300/0 conn(0x7fe328145ac0 0x7fe328071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe32d800640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3280ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe327e00640 1 -- 172.21.15.67:0/3234254560 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 msgr2=0x7fe3280ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe327e00640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3280ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe327e00640 1 -- 172.21.15.67:0/3234254560 >> v2:172.21.15.187:3300/0 conn(0x7fe328145ac0 msgr2=0x7fe328071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe327e00640 1 --2- 172.21.15.67:0/3234254560 >> v2:172.21.15.187:3300/0 conn(0x7fe328145ac0 0x7fe328071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.795+0000 7fe327e00640 1 -- 172.21.15.67:0/3234254560 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3280cf030 con 0x7fe3280ce3b0 2024-04-17T08:52:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.796+0000 7fe327e00640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280ce3b0 0x7fe328072670 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dbf209a07080e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.796+0000 7fe327400640 1 -- 172.21.15.67:0/3234254560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314035020 con 0x7fe3280ce3b0 2024-04-17T08:52:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe327400640 1 -- 172.21.15.67:0/3234254560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe314031640 con 0x7fe3280ce3b0 2024-04-17T08:52:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe327400640 1 -- 172.21.15.67:0/3234254560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31402d040 con 0x7fe3280ce3b0 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 -- 172.21.15.67:0/3234254560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280ce3b0 msgr2=0x7fe328072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280ce3b0 0x7fe328072670 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 -- 172.21.15.67:0/3234254560 shutdown_connections 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3280ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 --2- 172.21.15.67:0/3234254560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3280ce3b0 0x7fe328072670 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 --2- 172.21.15.67:0/3234254560 >> v2:172.21.15.187:3300/0 conn(0x7fe328145ac0 0x7fe328071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 -- 172.21.15.67:0/3234254560 >> 172.21.15.67:0/3234254560 conn(0x7fe328067ee0 msgr2=0x7fe32806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 -- 172.21.15.67:0/3234254560 shutdown_connections 2024-04-17T08:52:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.797+0000 7fe32f800640 1 -- 172.21.15.67:0/3234254560 wait complete. 2024-04-17T08:52:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.799+0000 7fe32f800640 1 Processor -- start 2024-04-17T08:52:03.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.799+0000 7fe32f800640 1 -- start start 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3281e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280ce3b0 0x7fe3281e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe328145ea0 con 0x7fe328145ac0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe3280cf5f0 con 0x7fe3280ce3b0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.800+0000 7fe32f800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe3280cd3e0 con 0x7fe328072bb0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48252/0 (socket says 172.21.15.67:48252) 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 -- 172.21.15.67:0/2917023212 learned_addr learned my addr 172.21.15.67:0/2917023212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32ce00640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3281e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe327e00640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280ce3b0 0x7fe3281e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 -- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 msgr2=0x7fe3281e4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3281e4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 -- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280ce3b0 msgr2=0x7fe3281e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280ce3b0 0x7fe3281e5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.801+0000 7fe32d800640 1 -- 172.21.15.67:0/2917023212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32815ed70 con 0x7fe328145ac0 2024-04-17T08:52:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe32d800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe320007e40 tx=0x7fe32000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320014620 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe320019070 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320014a20 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32815f010 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.802+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32815f5d0 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.803+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe328068ca0 con 0x7fe328145ac0 2024-04-17T08:52:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.804+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 139) ==== 91244+0+0 (secure 0 0 0) 0x7fe320014bc0 con 0x7fe328145ac0 2024-04-17T08:52:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.805+0000 7fe324c00640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 0x7fe30806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.805+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fe32008cc10 con 0x7fe328145ac0 2024-04-17T08:52:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.806+0000 7fe32ce00640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 0x7fe30806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.807+0000 7fe32ce00640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 0x7fe30806eff0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006fd0 tx=0x7fe31c008040 comp rx=0 tx=0).ready entity=mgr.8926 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:03.810+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe32005a990 con 0x7fe328145ac0 2024-04-17T08:52:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.063+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fe3280cc6c0 con 0x7fe328145ac0 2024-04-17T08:52:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.754+0000 7fe324c00640 1 -- 172.21.15.67:0/2917023212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v140) ==== 92+0+0 (secure 0 0 0) 0x7fe32005a290 con 0x7fe328145ac0 2024-04-17T08:52:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 msgr2=0x7fe30806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 0x7fe30806eff0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006fd0 tx=0x7fe31c008040 comp rx=0 tx=0).stop 2024-04-17T08:52:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 msgr2=0x7fe3280cb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe320007e40 tx=0x7fe32000e4b0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 shutdown_connections 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fe30806cb30 0x7fe30806eff0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe328145ac0 0x7fe3280cb3a0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3280ce3b0 0x7fe3281e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 --2- 172.21.15.67:0/2917023212 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe328072bb0 0x7fe3281e4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.758+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 >> 172.21.15.67:0/2917023212 conn(0x7fe328067ee0 msgr2=0x7fe32806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.759+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 shutdown_connections 2024-04-17T08:52:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.759+0000 7fe32f800640 1 -- 172.21.15.67:0/2917023212 wait complete. 2024-04-17T08:52:04.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:52:04.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.980+0000 7fb46f200640 1 Processor -- start 2024-04-17T08:52:04.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.981+0000 7fb46f200640 1 -- start start 2024-04-17T08:52:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb46815e9b0 0x7fb4680cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4680b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4680b51d0 con 0x7fb4680b4c30 2024-04-17T08:52:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb4680b5790 con 0x7fb46815e9b0 2024-04-17T08:52:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4680d2830 con 0x7fb4680cf910 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4680b4c30 0x7fb4680d4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52170/0 (socket says 172.21.15.67:52170) 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46e800640 1 -- 172.21.15.67:0/4192830839 learned_addr learned my addr 172.21.15.67:0/4192830839 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46de00640 1 --2- 172.21.15.67:0/4192830839 >> v2:172.21.15.187:3300/0 conn(0x7fb46815e9b0 0x7fb4680cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.982+0000 7fb46d400640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4680b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46ca00640 1 -- 172.21.15.67:0/4192830839 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223975855 0 0) 0x7fb4680b51d0 con 0x7fb4680b4c30 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46ca00640 1 -- 172.21.15.67:0/4192830839 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb450003520 con 0x7fb4680b4c30 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46de00640 1 -- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 msgr2=0x7fb4680b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46de00640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4680b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46de00640 1 -- 172.21.15.67:0/4192830839 >> v1:172.21.15.67:6789/0 conn(0x7fb4680b4c30 legacy=0x7fb4680d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46de00640 1 -- 172.21.15.67:0/4192830839 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4680b54b0 con 0x7fb46815e9b0 2024-04-17T08:52:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.983+0000 7fb46d400640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4680b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.984+0000 7fb46de00640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb46815e9b0 0x7fb4680cf3d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0097a0 tx=0x7fb45c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f10f9792a9159140 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46ca00640 1 -- 172.21.15.67:0/4192830839 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c031780 con 0x7fb46815e9b0 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46ca00640 1 -- 172.21.15.67:0/4192830839 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45c037070 con 0x7fb46815e9b0 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46ca00640 1 -- 172.21.15.67:0/4192830839 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c031b20 con 0x7fb46815e9b0 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 -- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb46815e9b0 msgr2=0x7fb4680cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb46815e9b0 0x7fb4680cf3d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0097a0 tx=0x7fb45c02b820 comp rx=0 tx=0).stop 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 -- 172.21.15.67:0/4192830839 shutdown_connections 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4680b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 --2- 172.21.15.67:0/4192830839 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb46815e9b0 0x7fb4680cf3d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 -- 172.21.15.67:0/4192830839 >> 172.21.15.67:0/4192830839 conn(0x7fb46809daa0 msgr2=0x7fb46808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.985+0000 7fb46f200640 1 -- 172.21.15.67:0/4192830839 shutdown_connections 2024-04-17T08:52:04.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.986+0000 7fb46f200640 1 -- 172.21.15.67:0/4192830839 wait complete. 2024-04-17T08:52:04.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.987+0000 7fb46f200640 1 Processor -- start 2024-04-17T08:52:04.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.987+0000 7fb46f200640 1 -- start start 2024-04-17T08:52:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4680b4c30 0x7fb468156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4681616a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb46815ed90 con 0x7fb46815e9b0 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb468162830 con 0x7fb4680b4c30 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb46815ef10 con 0x7fb4680cf910 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.988+0000 7fb46e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48280/0 (socket says 172.21.15.67:48280) 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 -- 172.21.15.67:0/1980369344 learned_addr learned my addr 172.21.15.67:0/1980369344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46de00640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4680b4c30 0x7fb468156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46d400640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4681616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 msgr2=0x7fb4681616a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4681616a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4680b4c30 msgr2=0x7fb468156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4680b4c30 0x7fb468156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 -- 172.21.15.67:0/1980369344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb468157000 con 0x7fb46815e9b0 2024-04-17T08:52:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.989+0000 7fb46e800640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fb46400cb40 tx=0x7fb46400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.990+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464017070 con 0x7fb46815e9b0 2024-04-17T08:52:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.990+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46400aca0 con 0x7fb46815e9b0 2024-04-17T08:52:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.990+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464012620 con 0x7fb46815e9b0 2024-04-17T08:52:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.990+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4680b7750 con 0x7fb46815e9b0 2024-04-17T08:52:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.990+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4680b7d10 con 0x7fb46815e9b0 2024-04-17T08:52:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.991+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4680d56c0 con 0x7fb46815e9b0 2024-04-17T08:52:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.992+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 140) ==== 91261+0+0 (secure 0 0 0) 0x7fb46401a070 con 0x7fb46815e9b0 2024-04-17T08:52:04.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.993+0000 7fb462000640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 0x7fb44c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:04.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.993+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7fb46408c140 con 0x7fb46815e9b0 2024-04-17T08:52:04.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.993+0000 7fb46de00640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 msgr2=0x7fb44c06f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.993+0000 7fb46de00640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 0x7fb44c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:04.998+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb4680d56c0 con 0x7fb46815e9b0 2024-04-17T08:52:05.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.194+0000 7fb46de00640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 msgr2=0x7fb44c06f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:05.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.194+0000 7fb46de00640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 0x7fb44c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.219+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fb4680b5e70 con 0x7fb46815e9b0 2024-04-17T08:52:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.226+0000 7fb462000640 1 -- 172.21.15.67:0/1980369344 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) ==== 138+0+0 (secure 0 0 0) 0x7fb4640597e0 con 0x7fb46815e9b0 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 msgr2=0x7fb44c06f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 0x7fb44c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 msgr2=0x7fb46806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fb46400cb40 tx=0x7fb46400a060 comp rx=0 tx=0).stop 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 shutdown_connections 2024-04-17T08:52:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7fb44c06cb80 0x7fb44c06f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46815e9b0 0x7fb46806d3b0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4680cf910 0x7fb4681616a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 --2- 172.21.15.67:0/1980369344 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4680b4c30 0x7fb468156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 >> 172.21.15.67:0/1980369344 conn(0x7fb46809daa0 msgr2=0x7fb4680d2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 shutdown_connections 2024-04-17T08:52:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.230+0000 7fb46f200640 1 -- 172.21.15.67:0/1980369344 wait complete. 2024-04-17T08:52:05.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:52:05.338 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:05.339+0000 7fe765641040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:05.346 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:05.347+0000 7f2afc7bf040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:05.396 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:05.397+0000 7fe765641040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:05.400 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:05.401+0000 7f2afc7bf040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:05.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.456+0000 7f9aca000640 1 Processor -- start 2024-04-17T08:52:05.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.457+0000 7f9aca000640 1 -- start start 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.458+0000 7f9aca000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9ac415c800 0x7f9ac406c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.458+0000 7f9aca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.458+0000 7f9aca000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40cbb50 con 0x7f9ac40cd0f0 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.458+0000 7f9aca000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac40cbdf0 con 0x7f9ac40b3b00 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.458+0000 7f9aca000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac40cc0d0 con 0x7f9ac415c800 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48282/0 (socket says 172.21.15.67:48282) 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 -- 172.21.15.67:0/3594907866 learned_addr learned my addr 172.21.15.67:0/3594907866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac8c00640 1 --2- 172.21.15.67:0/3594907866 >> v2:172.21.15.187:3300/0 conn(0x7f9ac415c800 0x7f9ac406c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 -- 172.21.15.67:0/3594907866 >> v2:172.21.15.187:3300/0 conn(0x7f9ac415c800 msgr2=0x7f9ac406c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 --2- 172.21.15.67:0/3594907866 >> v2:172.21.15.187:3300/0 conn(0x7f9ac415c800 0x7f9ac406c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 -- 172.21.15.67:0/3594907866 >> v1:172.21.15.187:6789/0 conn(0x7f9ac40cd0f0 legacy=0x7f9ac40ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.459+0000 7f9ac3e00640 1 -- 172.21.15.67:0/3594907866 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40cc3b0 con 0x7f9ac40b3b00 2024-04-17T08:52:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.460+0000 7f9ac3e00640 1 --2- 172.21.15.67:0/3594907866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f9ac40ceef0 tx=0x7f9ab402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c56986d01093bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.460+0000 7f9ac3400640 1 -- 172.21.15.67:0/3594907866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4038070 con 0x7f9ac40b3b00 2024-04-17T08:52:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.460+0000 7f9ac3400640 1 -- 172.21.15.67:0/3594907866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab4031740 con 0x7f9ac40b3b00 2024-04-17T08:52:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9ac3400640 1 -- 172.21.15.67:0/3594907866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab402f050 con 0x7f9ac40b3b00 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 -- 172.21.15.67:0/3594907866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac40b3b00 msgr2=0x7f9ac40ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 --2- 172.21.15.67:0/3594907866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f9ac40ceef0 tx=0x7f9ab402b7e0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 -- 172.21.15.67:0/3594907866 shutdown_connections 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 --2- 172.21.15.67:0/3594907866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac40b3b00 0x7f9ac40ccbb0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 --2- 172.21.15.67:0/3594907866 >> v2:172.21.15.187:3300/0 conn(0x7f9ac415c800 0x7f9ac406c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 -- 172.21.15.67:0/3594907866 >> 172.21.15.67:0/3594907866 conn(0x7f9ac40b2fe0 msgr2=0x7f9ac40b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 -- 172.21.15.67:0/3594907866 shutdown_connections 2024-04-17T08:52:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.461+0000 7f9aca000640 1 -- 172.21.15.67:0/3594907866 wait complete. 2024-04-17T08:52:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.463+0000 7f9aca000640 1 Processor -- start 2024-04-17T08:52:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.463+0000 7f9aca000640 1 -- start start 2024-04-17T08:52:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ac40b3b00 0x7f9ac415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 0x7f9ac40c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac415cd60 con 0x7f9ac415c800 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac415cbe0 con 0x7f9ac40cd750 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9aca000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40ae9d0 con 0x7f9ac40b3b00 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48284/0 (socket says 172.21.15.67:48284) 2024-04-17T08:52:05.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.464+0000 7f9ac3e00640 1 -- 172.21.15.67:0/4285992578 learned_addr learned my addr 172.21.15.67:0/4285992578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ac40b3b00 msgr2=0x7f9ac415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ac40b3b00 0x7f9ac415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 msgr2=0x7f9ac40c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac9600640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 0x7f9ac40c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 0x7f9ac40c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 -- 172.21.15.67:0/4285992578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40c8540 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac9600640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 0x7f9ac40c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac3e00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40090b0 tx=0x7f9ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4042070 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab4031e80 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac41f8cf0 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.465+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4038040 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.466+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41f8700 con 0x7f9ac415c800 2024-04-17T08:52:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.467+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac415c060 con 0x7f9ac415c800 2024-04-17T08:52:05.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.468+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 140) ==== 91261+0+0 (secure 0 0 0) 0x7f9ab4045070 con 0x7f9ac415c800 2024-04-17T08:52:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.468+0000 7f9ac0c00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 0x7f9aa006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.469+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f9ab40ae140 con 0x7f9ac415c800 2024-04-17T08:52:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.469+0000 7f9ac8c00640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 msgr2=0x7f9aa006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.469+0000 7f9ac8c00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 0x7f9aa006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.473+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9ac415c060 con 0x7f9ac415c800 2024-04-17T08:52:05.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.670+0000 7f9ac8c00640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 msgr2=0x7f9aa006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:05.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.670+0000 7f9ac8c00640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 0x7f9aa006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.683+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9ac41f9090 con 0x7f9ac415c800 2024-04-17T08:52:05.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.689+0000 7f9ac0c00640 1 -- 172.21.15.67:0/4285992578 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) ==== 133+0+0 (secure 0 0 0) 0x7f9ab404c030 con 0x7f9ac415c800 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 msgr2=0x7f9aa006f070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 0x7f9aa006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 msgr2=0x7f9ac41f81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40090b0 tx=0x7f9ab40046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 shutdown_connections 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f9aa006cbb0 0x7f9aa006f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ac40cd750 0x7f9ac40c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ac415c800 0x7f9ac41f81c0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 --2- 172.21.15.67:0/4285992578 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ac40b3b00 0x7f9ac415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 >> 172.21.15.67:0/4285992578 conn(0x7f9ac40b2fe0 msgr2=0x7f9ac40b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 shutdown_connections 2024-04-17T08:52:05.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.691+0000 7f9aca000640 1 -- 172.21.15.67:0/4285992578 wait complete. 2024-04-17T08:52:05.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:05.708 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-04-17T08:52:05.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-04-17T08:52:05.854 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:05.855+0000 7f2afc7bf040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:05.881 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:05.882+0000 7fe765641040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:05.906 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:05.907+0000 7f2afc7bf040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:05.937 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:05.938+0000 7fe765641040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:05.956 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:05.957+0000 7f2afc7bf040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.970+0000 7f5d1e400640 1 Processor -- start 2024-04-17T08:52:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.971+0000 7f5d1e400640 1 -- start start 2024-04-17T08:52:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d1e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d1815e9b0 0x7f5d180cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d1e400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d180b51d0 con 0x7f5d180cf910 2024-04-17T08:52:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d1e400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d180b54b0 con 0x7f5d180d4510 2024-04-17T08:52:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d1e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5d180b5790 con 0x7f5d1815e9b0 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d1d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d1815e9b0 0x7f5d180cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d17e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5d180cf910 0x7f5d180b46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52182/0 (socket says 172.21.15.67:52182) 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.972+0000 7f5d17e00640 1 -- 172.21.15.67:0/787871626 learned_addr learned my addr 172.21.15.67:0/787871626 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.973+0000 7f5d1d000640 1 -- 172.21.15.67:0/787871626 >> v1:172.21.15.187:6789/0 conn(0x7f5d180d4510 legacy=0x7f5d180d4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.973+0000 7f5d1d000640 1 -- 172.21.15.67:0/787871626 >> v1:172.21.15.67:6789/0 conn(0x7f5d180cf910 legacy=0x7f5d180b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.973+0000 7f5d1d000640 1 -- 172.21.15.67:0/787871626 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d180d24c0 con 0x7f5d1815e9b0 2024-04-17T08:52:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.974+0000 7f5d1d000640 1 --2- 172.21.15.67:0/787871626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d1815e9b0 0x7f5d180cf3d0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f5d180b5910 tx=0x7f5d0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298e1260b6795f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.974+0000 7f5d17400640 1 -- 172.21.15.67:0/787871626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d080315a0 con 0x7f5d1815e9b0 2024-04-17T08:52:05.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.974+0000 7f5d17400640 1 -- 172.21.15.67:0/787871626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d08037070 con 0x7f5d1815e9b0 2024-04-17T08:52:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.974+0000 7f5d17400640 1 -- 172.21.15.67:0/787871626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08031960 con 0x7f5d1815e9b0 2024-04-17T08:52:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 -- 172.21.15.67:0/787871626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d1815e9b0 msgr2=0x7f5d180cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 --2- 172.21.15.67:0/787871626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d1815e9b0 0x7f5d180cf3d0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f5d180b5910 tx=0x7f5d0802b840 comp rx=0 tx=0).stop 2024-04-17T08:52:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 -- 172.21.15.67:0/787871626 shutdown_connections 2024-04-17T08:52:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 --2- 172.21.15.67:0/787871626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d1815e9b0 0x7f5d180cf3d0 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 -- 172.21.15.67:0/787871626 >> 172.21.15.67:0/787871626 conn(0x7f5d1809daa0 msgr2=0x7f5d1808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 -- 172.21.15.67:0/787871626 shutdown_connections 2024-04-17T08:52:05.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.975+0000 7f5d1e400640 1 -- 172.21.15.67:0/787871626 wait complete. 2024-04-17T08:52:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.977+0000 7f5d1e400640 1 Processor -- start 2024-04-17T08:52:05.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.977+0000 7f5d1e400640 1 -- start start 2024-04-17T08:52:05.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d180cf910 0x7f5d180b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d180d4510 0x7f5d180c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d1815e9b0 0x7f5d180c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d180b4e00 con 0x7f5d180cf910 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d180d2830 con 0x7f5d180d4510 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d180b4f80 con 0x7f5d1815e9b0 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d180cf910 0x7f5d180b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d180cf910 0x7f5d180b9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48292/0 (socket says 172.21.15.67:48292) 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 -- 172.21.15.67:0/3055547916 learned_addr learned my addr 172.21.15.67:0/3055547916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d180d4510 0x7f5d180c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d1815e9b0 msgr2=0x7f5d180c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d1815e9b0 0x7f5d180c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d180d4510 msgr2=0x7f5d180c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:05.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d180d4510 0x7f5d180c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.979+0000 7f5d1d000640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d180ca550 con 0x7f5d180cf910 2024-04-17T08:52:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d1d000640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d180cf910 0x7f5d180b9870 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f5d08000c00 tx=0x7f5d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08032840 con 0x7f5d180cf910 2024-04-17T08:52:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d08032a00 con 0x7f5d180cf910 2024-04-17T08:52:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08032df0 con 0x7f5d180cf910 2024-04-17T08:52:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d1e400640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d180b8000 con 0x7f5d180cf910 2024-04-17T08:52:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.980+0000 7f5d1e400640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d180b85c0 con 0x7f5d180cf910 2024-04-17T08:52:05.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.982+0000 7f5d1e400640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d180755d0 con 0x7f5d180cf910 2024-04-17T08:52:05.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.983+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 140) ==== 91261+0+0 (secure 0 0 0) 0x7f5d0804d3f0 con 0x7f5d180cf910 2024-04-17T08:52:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.984+0000 7f5d14c00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.984+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6601+0+0 (secure 0 0 0) 0x7f5d0802f030 con 0x7f5d180cf910 2024-04-17T08:52:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.984+0000 7f5d17e00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:05.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.984+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:05.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:05.992+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5d180ca550 con 0x7f5d180cf910 2024-04-17T08:52:05.997 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:05.997+0000 7fe765641040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:06.018 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.019+0000 7f2afc7bf040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:06.063 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.063+0000 7fe765641040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:06.067 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.068+0000 7f2afc7bf040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:06.115 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.116+0000 7fe765641040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:06.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:06.185+0000 7f5d17e00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:06.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:06.185+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:06.196 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.197+0000 7f2afc7bf040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:06.222+0000 7f5d1e400640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f5d180b7d40 con 0x7f5cf406cb80 2024-04-17T08:52:06.253 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.253+0000 7fe765641040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:06.258 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.259+0000 7f2afc7bf040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:06.310 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.311+0000 7f2afc7bf040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:06.320 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.321+0000 7fe765641040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:06.367 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.368+0000 7f2afc7bf040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:06.373 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.374+0000 7fe765641040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:06.418 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.419+0000 7f2afc7bf040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:06.433 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.434+0000 7fe765641040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:06.486 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.487+0000 7fe765641040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:06.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:06.586+0000 7f5d17e00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:06.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:06.586+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:52:06.799 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.800+0000 7f2afc7bf040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:06.895 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:06.895+0000 7fe765641040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:06.958 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:06.959+0000 7f2afc7bf040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:07.064 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.064+0000 7fe765641040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:07.066 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.066+0000 7f2afc7bf040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:07.116 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.117+0000 7f2afc7bf040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:07.179 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.179+0000 7fe765641040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:07.233 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.233+0000 7fe765641040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:07.241 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.242+0000 7f2afc7bf040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:07.350 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.351+0000 7f2afc7bf040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:07.366 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.367+0000 7fe765641040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:07.387+0000 7f5d17e00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:07.387+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:52:07.480 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.480+0000 7fe765641040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:07.525 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.526+0000 7f2afc7bf040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:07.575 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.576+0000 7f2afc7bf040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:07.666 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.666+0000 7fe765641040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:07.711 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:07.712+0000 7f2afc7bf040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:07.718 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.719+0000 7fe765641040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:07.863 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:07.864+0000 7fe765641040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:08.551 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:08.551+0000 7f2afc7bf040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:08.743 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:08.744+0000 7fe765641040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:08.854 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:08.854+0000 7f2afc7bf040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:52:08.911 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:08.911+0000 7f2afc7bf040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:52:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:08.990+0000 7f5d17e00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/2123602634 2024-04-17T08:52:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:08.990+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T08:52:09.042 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.042+0000 7f2afc7bf040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:52:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:09.067+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 141) ==== 90768+0+0 (secure 0 0 0) 0x7f5d08077c80 con 0x7f5d180cf910 2024-04-17T08:52:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:09.067+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 msgr2=0x7f5cf406f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:09.067+0000 7f5d14c00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/2123602634,v1:172.21.15.187:6833/2123602634] conn(0x7f5cf406cb80 0x7f5cf406f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:09.070 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:09.072+0000 7fe765641040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:52:09.133 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:09.135+0000 7fe765641040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:52:09.279 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:09.280+0000 7fe765641040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:52:09.491 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.492 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.492+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.513+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.513+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.513+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.513+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.512 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.513+0000 7f2a70a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.514+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.514+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.514+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.514+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:09.513 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:09.514+0000 7f2a6d800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:10.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.076+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 142) ==== 90895+0+0 (secure 0 0 0) 0x7f5d08005990 con 0x7f5d180cf910 2024-04-17T08:52:10.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.077+0000 7f5d14c00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f5cf40770c0 0x7f5cf40794b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.077+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 --> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f5d180b7d40 con 0x7f5cf40770c0 2024-04-17T08:52:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.079+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f5cf40770c0 0x7f5cf40794b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.083+0000 7f5d17e00640 1 --2- 172.21.15.67:0/3055547916 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f5cf40770c0 0x7f5cf40794b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5d180ba4a0 tx=0x7f5d00001b40 comp rx=0 tx=0).ready entity=mgr.8276 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:10.085 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:10.086+0000 7f2a7a000640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-04-17T08:52:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.088+0000 7f5d14c00640 1 -- 172.21.15.67:0/3055547916 <== mgr.8276 v2:172.21.15.187:6832/986946406 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) ==== 79+0+0 (secure 0 0 0) 0x7f5d180b7d40 con 0x7f5cf40770c0 2024-04-17T08:52:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-04-17T08:52:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-17T08:52:10.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:10.102 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-04-17T08:52:10.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:52:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.382+0000 7fb2cca00640 1 Processor -- start 2024-04-17T08:52:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.382+0000 7fb2cca00640 1 -- start start 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.383+0000 7fb2cca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c815b6b0 0x7fb2c806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2cca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb2c80cfc10 0x7fb2c806be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2cca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c806df40 con 0x7fb2c806f800 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2cca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c806ed00 con 0x7fb2c815b6b0 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2cca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c806e870 con 0x7fb2c80cfc10 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c815b6b0 0x7fb2c806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c815b6b0 0x7fb2c806b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48294/0 (socket says 172.21.15.67:48294) 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2c7400640 1 -- 172.21.15.67:0/720816029 learned_addr learned my addr 172.21.15.67:0/720816029 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2c7400640 1 -- 172.21.15.67:0/720816029 >> v2:172.21.15.187:3300/0 conn(0x7fb2c80cfc10 msgr2=0x7fb2c806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.384+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/720816029 >> v2:172.21.15.187:3300/0 conn(0x7fb2c80cfc10 0x7fb2c806be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.385+0000 7fb2c7400640 1 --2- 172.21.15.67:0/720816029 >> v2:172.21.15.187:3300/0 conn(0x7fb2c80cfc10 0x7fb2c806be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.385+0000 7fb2c7400640 1 -- 172.21.15.67:0/720816029 >> v1:172.21.15.187:6789/0 conn(0x7fb2c806f800 legacy=0x7fb2c815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:10.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.385+0000 7fb2c7400640 1 -- 172.21.15.67:0/720816029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c806e220 con 0x7fb2c815b6b0 2024-04-17T08:52:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.386+0000 7fb2c7400640 1 --2- 172.21.15.67:0/720816029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c815b6b0 0x7fb2c806b8c0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fb2c815f050 tx=0x7fb2b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a6f93d35d19ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.386+0000 7fb2c6a00640 1 -- 172.21.15.67:0/720816029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8035020 con 0x7fb2c815b6b0 2024-04-17T08:52:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.386+0000 7fb2c6a00640 1 -- 172.21.15.67:0/720816029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b8031870 con 0x7fb2c815b6b0 2024-04-17T08:52:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.386+0000 7fb2c6a00640 1 -- 172.21.15.67:0/720816029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8031c40 con 0x7fb2c815b6b0 2024-04-17T08:52:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 -- 172.21.15.67:0/720816029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c815b6b0 msgr2=0x7fb2c806b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 --2- 172.21.15.67:0/720816029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c815b6b0 0x7fb2c806b8c0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fb2c815f050 tx=0x7fb2b802b910 comp rx=0 tx=0).stop 2024-04-17T08:52:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 -- 172.21.15.67:0/720816029 shutdown_connections 2024-04-17T08:52:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 --2- 172.21.15.67:0/720816029 >> v2:172.21.15.187:3300/0 conn(0x7fb2c80cfc10 0x7fb2c806be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 --2- 172.21.15.67:0/720816029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c815b6b0 0x7fb2c806b8c0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 -- 172.21.15.67:0/720816029 >> 172.21.15.67:0/720816029 conn(0x7fb2c80b2190 msgr2=0x7fb2c80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 -- 172.21.15.67:0/720816029 shutdown_connections 2024-04-17T08:52:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.387+0000 7fb2cca00640 1 -- 172.21.15.67:0/720816029 wait complete. 2024-04-17T08:52:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.389+0000 7fb2cca00640 1 Processor -- start 2024-04-17T08:52:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.389+0000 7fb2cca00640 1 -- start start 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 0x7fb2c8168e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c815b6b0 0x7fb2c8151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80afee0 con 0x7fb2c806f800 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80afd60 con 0x7fb2c815b6b0 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2cca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c80b44a0 con 0x7fb2c80cfc10 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2c7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2c7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48302/0 (socket says 172.21.15.67:48302) 2024-04-17T08:52:10.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2c7400640 1 -- 172.21.15.67:0/3273260879 learned_addr learned my addr 172.21.15.67:0/3273260879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2c7e00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c815b6b0 0x7fb2c8151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2bfe00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 0x7fb2c8168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.390+0000 7fb2c7400640 1 -- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 msgr2=0x7fb2c8168e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2c7400640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 0x7fb2c8168e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2c7400640 1 -- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c815b6b0 msgr2=0x7fb2c8151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2c7400640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c815b6b0 0x7fb2c8151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2c7400640 1 -- 172.21.15.67:0/3273260879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c80ad2e0 con 0x7fb2c806f800 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2bfe00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 0x7fb2c8168e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2c7400640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fb2b802b950 tx=0x7fb2b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:10.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8040070 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b803a420 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c80abe20 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8035040 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.391+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c8167d10 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.392+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c8068a10 con 0x7fb2c806f800 2024-04-17T08:52:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.393+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 142) ==== 90895+0+0 (secure 0 0 0) 0x7fb2b804b060 con 0x7fb2c806f800 2024-04-17T08:52:10.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.394+0000 7fb2bf400640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 0x7fb2a406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:10.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.394+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7fb2b80add20 con 0x7fb2c806f800 2024-04-17T08:52:10.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.395+0000 7fb2bfe00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 0x7fb2a406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:10.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.396+0000 7fb2bfe00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 0x7fb2a406ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb2c81696a0 tx=0x7fb2b0009040 comp rx=0 tx=0).ready entity=mgr.8276 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:10.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.399+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb2b807be90 con 0x7fb2c806f800 2024-04-17T08:52:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:10.661+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fb2c806ee80 con 0x7fb2c806f800 2024-04-17T08:52:11.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.120+0000 7fb2bf400640 1 -- 172.21.15.67:0/3273260879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v143) ==== 93+0+0 (secure 0 0 0) 0x7fb2b807b790 con 0x7fb2c806f800 2024-04-17T08:52:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.125+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 msgr2=0x7fb2a406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.125+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 0x7fb2a406ec80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb2c81696a0 tx=0x7fb2b0009040 comp rx=0 tx=0).stop 2024-04-17T08:52:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.125+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 msgr2=0x7fb2c814ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.125+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fb2b802b950 tx=0x7fb2b80046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 shutdown_connections 2024-04-17T08:52:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fb2a406c7c0 0x7fb2a406ec80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb2c815b6b0 0x7fb2c8151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb2c80cfc10 0x7fb2c8168e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 --2- 172.21.15.67:0/3273260879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c806f800 0x7fb2c814ba90 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 >> 172.21.15.67:0/3273260879 conn(0x7fb2c80b2190 msgr2=0x7fb2c80ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 shutdown_connections 2024-04-17T08:52:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.128+0000 7fb2cca00640 1 -- 172.21.15.67:0/3273260879 wait complete. 2024-04-17T08:52:11.148 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:52:11.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:52:11.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:52:11.343 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:52:11.343 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:52:11.344 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:11.344 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:11.395 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:11.396 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:11.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:11.688 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:11.689+0000 7f32ef016040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.699+0000 7fd60e800640 1 Processor -- start 2024-04-17T08:52:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.699+0000 7fd60e800640 1 -- start start 2024-04-17T08:52:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd60e800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd6080d2ab0 0x7fd6080b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd60e800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6080d3020 con 0x7fd60815ed10 2024-04-17T08:52:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd60e800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6080d3300 con 0x7fd6080cf640 2024-04-17T08:52:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd60e800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd6080d3c30 con 0x7fd6080d2ab0 2024-04-17T08:52:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd5ffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd60815ed10 0x7fd6080d40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52184/0 (socket says 172.21.15.67:52184) 2024-04-17T08:52:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd5ffe00640 1 -- 172.21.15.67:0/2688903813 learned_addr learned my addr 172.21.15.67:0/2688903813 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd607e00640 1 --2- 172.21.15.67:0/2688903813 >> v2:172.21.15.187:3300/0 conn(0x7fd6080d2ab0 0x7fd6080b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.701+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119016880 0 0) 0x7fd6080d3020 con 0x7fd60815ed10 2024-04-17T08:52:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5ec003520 con 0x7fd60815ed10 2024-04-17T08:52:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1029593943 0 0) 0x7fd5ec003520 con 0x7fd60815ed10 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 >> v2:172.21.15.187:3300/0 conn(0x7fd6080d2ab0 msgr2=0x7fd6080b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 --2- 172.21.15.67:0/2688903813 >> v2:172.21.15.187:3300/0 conn(0x7fd6080d2ab0 0x7fd6080b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 >> v1:172.21.15.187:6789/0 conn(0x7fd6080cf640 legacy=0x7fd6080cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080d35e0 con 0x7fd60815ed10 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd5f0001670 con 0x7fd60815ed10 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 >> v1:172.21.15.67:6789/0 conn(0x7fd60815ed10 legacy=0x7fd6080d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec003a00 0x7fd5ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec004800 0x7fd5ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 0x7fd5ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080d3c30 con 0x7fd5ec0051c0 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec00c0a0 con 0x7fd5ec004800 2024-04-17T08:52:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.702+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ec00c2a0 con 0x7fd5ec003a00 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 0x7fd5ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd5ffe00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec004800 0x7fd5ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607400640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec003a00 0x7fd5ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 -- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec003a00 msgr2=0x7fd5ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec003a00 0x7fd5ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 -- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec004800 msgr2=0x7fd5ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec004800 0x7fd5ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607e00640 1 -- 172.21.15.67:0/2688903813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec003520 con 0x7fd5ec0051c0 2024-04-17T08:52:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.703+0000 7fd607400640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5ec003a00 0x7fd5ec003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.704+0000 7fd607e00640 1 -- 172.21.15.67:0/2688903813 reap_dead start 2024-04-17T08:52:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.704+0000 7fd607e00640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 0x7fd5ec00b9c0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fd600009fc0 tx=0x7fd5ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f8df424831f827 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.704+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600009c10 con 0x7fd5ec0051c0 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.704+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60002cca0 con 0x7fd5ec0051c0 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.704+0000 7fd606a00640 1 -- 172.21.15.67:0/2688903813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600003bd0 con 0x7fd5ec0051c0 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 -- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 msgr2=0x7fd5ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 0x7fd5ec00b9c0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fd600009fc0 tx=0x7fd5ec005830 comp rx=0 tx=0).stop 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 -- 172.21.15.67:0/2688903813 shutdown_connections 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 --2- 172.21.15.67:0/2688903813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5ec0051c0 0x7fd5ec00b9c0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 -- 172.21.15.67:0/2688903813 >> 172.21.15.67:0/2688903813 conn(0x7fd60806cbd0 msgr2=0x7fd60806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 -- 172.21.15.67:0/2688903813 shutdown_connections 2024-04-17T08:52:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.705+0000 7fd60e800640 1 -- 172.21.15.67:0/2688903813 wait complete. 2024-04-17T08:52:11.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.707+0000 7fd60e800640 1 Processor -- start 2024-04-17T08:52:11.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.707+0000 7fd60e800640 1 -- start start 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 0x7fd60814dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd60814e7f0 0x7fd60814eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080ba300 con 0x7fd6080b5390 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080d25b0 con 0x7fd5ec0051c0 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd60e800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6080ba480 con 0x7fd60814e7f0 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd5ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd5ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48328/0 (socket says 172.21.15.67:48328) 2024-04-17T08:52:11.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.708+0000 7fd5ffe00640 1 -- 172.21.15.67:0/3664686026 learned_addr learned my addr 172.21.15.67:0/3664686026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:11.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd607400640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 0x7fd60814dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd60814e7f0 msgr2=0x7fd60814eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd607e00640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd60814e7f0 0x7fd60814eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd60814e7f0 0x7fd60814eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 msgr2=0x7fd60814dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 0x7fd60814dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 -- 172.21.15.67:0/3664686026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080b58c0 con 0x7fd6080b5390 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd607400640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 0x7fd60814dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.709+0000 7fd5ffe00640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0001200 tx=0x7fd5f000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.710+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f000c660 con 0x7fd6080b5390 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.710+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f0018070 con 0x7fd6080b5390 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.710+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f000ca40 con 0x7fd6080b5390 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.710+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6080b5ba0 con 0x7fd6080b5390 2024-04-17T08:52:11.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.710+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6080b6160 con 0x7fd6080b5390 2024-04-17T08:52:11.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.711+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd608068b90 con 0x7fd6080b5390 2024-04-17T08:52:11.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.713+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7fd5f000cbe0 con 0x7fd6080b5390 2024-04-17T08:52:11.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.714+0000 7fd5ff400640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 0x7fd5e406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:11.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.714+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7fd5f008c150 con 0x7fd6080b5390 2024-04-17T08:52:11.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.714+0000 7fd607400640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 msgr2=0x7fd5e406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:11.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.714+0000 7fd607400640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 0x7fd5e406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.722+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd5f0059e30 con 0x7fd6080b5390 2024-04-17T08:52:11.724 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:11.726+0000 7f9de2293040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:11.744 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:11.745+0000 7f32ef016040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:11.783 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:11.784+0000 7f9de2293040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:11.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.915+0000 7fd607400640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 msgr2=0x7fd5e406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:11.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.915+0000 7fd607400640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 0x7fd5e406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:11.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.931+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd60815e2d0 con 0x7fd6080b5390 2024-04-17T08:52:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.934+0000 7fd5ff400640 1 -- 172.21.15.67:0/3664686026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) ==== 74+0+20981 (secure 0 0 0) 0x7fd5f0059730 con 0x7fd6080b5390 2024-04-17T08:52:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:11.933 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":255,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:09.047948+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.936+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 msgr2=0x7fd5e406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.936+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 0x7fd5e406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.936+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 msgr2=0x7fd6080ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.936+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0001200 tx=0x7fd5f000ec10 comp rx=0 tx=0).stop 2024-04-17T08:52:11.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 shutdown_connections 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fd5e406cb30 0x7fd5e406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd60814e7f0 0x7fd60814eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6080b5390 0x7fd6080ce360 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 --2- 172.21.15.67:0/3664686026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5ec0051c0 0x7fd60814dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 >> 172.21.15.67:0/3664686026 conn(0x7fd60806cbd0 msgr2=0x7fd6080c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 shutdown_connections 2024-04-17T08:52:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:11.937+0000 7fd60e800640 1 -- 172.21.15.67:0/3664686026 wait complete. 2024-04-17T08:52:11.953 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:51:38.821605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:51:39.950246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:51:54.448741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:51:55.106540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:52:11.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:52:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.171+0000 7f924be00640 1 Processor -- start 2024-04-17T08:52:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.171+0000 7f924be00640 1 -- start start 2024-04-17T08:52:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f924c0b3c80 0x7f924c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c09b170 0x7f924c15f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c0d0350 con 0x7f924c0ce160 2024-04-17T08:52:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f924c0b2ee0 con 0x7f924c0b3c80 2024-04-17T08:52:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f924c0d0630 con 0x7f924c09b170 2024-04-17T08:52:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f924c0b3c80 0x7f924c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f924c0b3c80 0x7f924c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:59764/0 (socket says 172.21.15.67:59764) 2024-04-17T08:52:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924aa00640 1 -- 172.21.15.67:0/481862905 learned_addr learned my addr 172.21.15.67:0/481862905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.173+0000 7f924b400640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c09b170 0x7f924c15f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.174+0000 7f924aa00640 1 -- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c09b170 msgr2=0x7f924c15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.174+0000 7f924aa00640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c09b170 0x7f924c15f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.174+0000 7f924aa00640 1 -- 172.21.15.67:0/481862905 >> v1:172.21.15.187:6789/0 conn(0x7f924c0ce160 legacy=0x7f924c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.174+0000 7f924aa00640 1 -- 172.21.15.67:0/481862905 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0d0910 con 0x7f924c0b3c80 2024-04-17T08:52:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.175+0000 7f924aa00640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c0cdc20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f923c009870 tx=0x7f923c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda14e80dbcf83fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.176+0000 7f924a000640 1 -- 172.21.15.67:0/481862905 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c034070 con 0x7f924c0b3c80 2024-04-17T08:52:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.176+0000 7f924a000640 1 -- 172.21.15.67:0/481862905 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c030840 con 0x7f924c0b3c80 2024-04-17T08:52:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.176+0000 7f924a000640 1 -- 172.21.15.67:0/481862905 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c030c10 con 0x7f924c0b3c80 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.176+0000 7f924be00640 1 -- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 msgr2=0x7f924c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.176+0000 7f924be00640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c0cdc20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f923c009870 tx=0x7f923c02b980 comp rx=0 tx=0).stop 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 -- 172.21.15.67:0/481862905 shutdown_connections 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c09b170 0x7f924c15f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 --2- 172.21.15.67:0/481862905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c0cdc20 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 -- 172.21.15.67:0/481862905 >> 172.21.15.67:0/481862905 conn(0x7f924c0b4060 msgr2=0x7f924c0b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 -- 172.21.15.67:0/481862905 shutdown_connections 2024-04-17T08:52:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.177+0000 7f924be00640 1 -- 172.21.15.67:0/481862905 wait complete. 2024-04-17T08:52:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.178+0000 7f924be00640 1 Processor -- start 2024-04-17T08:52:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.179+0000 7f924be00640 1 -- start start 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c1f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 0x7f924c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f924c0d13f0 con 0x7f924c09b170 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f924c0d0ed0 con 0x7f924c0b3c80 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f924c0d1570 con 0x7f924c0ce160 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48334/0 (socket says 172.21.15.67:48334) 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 -- 172.21.15.67:0/65082648 learned_addr learned my addr 172.21.15.67:0/65082648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f9243e00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c1f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924b400640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 0x7f924c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 msgr2=0x7f924c15c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 0x7f924c15c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 msgr2=0x7f924c1f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c1f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924aa00640 1 -- 172.21.15.67:0/65082648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0c5200 con 0x7f924c09b170 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f924b400640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 0x7f924c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.180+0000 7f9243e00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c1f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f924aa00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f923c000c00 tx=0x7f923c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c048070 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c030640 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c034040 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c0c54e0 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.181+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c15cc10 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.182+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c0755d0 con 0x7f924c09b170 2024-04-17T08:52:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.183+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f923c037070 con 0x7f924c09b170 2024-04-17T08:52:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.184+0000 7f9242a00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 0x7f922806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.184+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f923c0ae400 con 0x7f924c09b170 2024-04-17T08:52:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.185+0000 7f9243e00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 msgr2=0x7f922806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.185+0000 7f9243e00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 0x7f922806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.189+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f923c07bf10 con 0x7f924c09b170 2024-04-17T08:52:12.188 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.189+0000 7f32ef016040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:12.240 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.241+0000 7f32ef016040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:12.260 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.261+0000 7f9de2293040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:12.291 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.291+0000 7f32ef016040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:12.315 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.317+0000 7f9de2293040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:12.352 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.352+0000 7f32ef016040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:12.369 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.371+0000 7f9de2293040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:12.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.386+0000 7f9243e00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 msgr2=0x7f922806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:12.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.386+0000 7f9243e00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 0x7f922806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:12.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.390+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f924c0b3100 con 0x7f924c09b170 2024-04-17T08:52:12.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.391+0000 7f9242a00640 1 -- 172.21.15.67:0/65082648 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v255) ==== 93+0+10 (secure 0 0 0) 0x7f923c07b810 con 0x7f924c09b170 2024-04-17T08:52:12.389 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.393+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 msgr2=0x7f922806eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.393+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 0x7f922806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.393+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 msgr2=0x7f924c06cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.393+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f923c000c00 tx=0x7f923c004730 comp rx=0 tx=0).stop 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 shutdown_connections 2024-04-17T08:52:12.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f922806cb30 0x7f922806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f924c0ce160 0x7f924c15c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f924c0b3c80 0x7f924c1f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 --2- 172.21.15.67:0/65082648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f924c09b170 0x7f924c06cb60 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 >> 172.21.15.67:0/65082648 conn(0x7f924c0b4060 msgr2=0x7f924c0b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 shutdown_connections 2024-04-17T08:52:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.394+0000 7f924be00640 1 -- 172.21.15.67:0/65082648 wait complete. 2024-04-17T08:52:12.402 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.403+0000 7f32ef016040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:12.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:52:12.436 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.438+0000 7f9de2293040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:12.491 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.493+0000 7f9de2293040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:12.536 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.537+0000 7f32ef016040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:12.601 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.602+0000 7f32ef016040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.630+0000 7f2616600640 1 Processor -- start 2024-04-17T08:52:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.630+0000 7f2616600640 1 -- start start 2024-04-17T08:52:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f2616600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f26100b4fd0 0x7f26100cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f2616600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26100d00b0 con 0x7f26100d33e0 2024-04-17T08:52:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f2616600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26100d0390 con 0x7f261015ecb0 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f2616600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f26100d0c30 con 0x7f26100b4fd0 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f260fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26100d33e0 0x7f26100cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52186/0 (socket says 172.21.15.67:52186) 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f260fe00640 1 -- 172.21.15.67:0/1951631635 learned_addr learned my addr 172.21.15.67:0/1951631635 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.632+0000 7f2615c00640 1 --2- 172.21.15.67:0/1951631635 >> v2:172.21.15.187:3300/0 conn(0x7f26100b4fd0 0x7f26100cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789266423 0 0) 0x7f26100d00b0 con 0x7f26100d33e0 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25f8003520 con 0x7f26100d33e0 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f2615c00640 1 -- 172.21.15.67:0/1951631635 >> v1:172.21.15.187:6789/0 conn(0x7f261015ecb0 legacy=0x7f26100d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f2615c00640 1 -- 172.21.15.67:0/1951631635 >> v1:172.21.15.67:6789/0 conn(0x7f26100d33e0 legacy=0x7f26100cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2418024923 0 0) 0x7f25f8003520 con 0x7f26100d33e0 2024-04-17T08:52:12.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.633+0000 7f2615c00640 1 -- 172.21.15.67:0/1951631635 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26100d0670 con 0x7f26100b4fd0 2024-04-17T08:52:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.634+0000 7f2615c00640 1 --2- 172.21.15.67:0/1951631635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100b4fd0 0x7f26100cfab0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f26100d0db0 tx=0x7f260802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=948af1e709f585af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.635+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608039070 con 0x7f26100b4fd0 2024-04-17T08:52:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.635+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f260802cc50 con 0x7f26100b4fd0 2024-04-17T08:52:12.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.635+0000 7f260f400640 1 -- 172.21.15.67:0/1951631635 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608034440 con 0x7f26100b4fd0 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 -- 172.21.15.67:0/1951631635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100b4fd0 msgr2=0x7f26100cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 --2- 172.21.15.67:0/1951631635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100b4fd0 0x7f26100cfab0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f26100d0db0 tx=0x7f260802bea0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 -- 172.21.15.67:0/1951631635 shutdown_connections 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 --2- 172.21.15.67:0/1951631635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100b4fd0 0x7f26100cfab0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 -- 172.21.15.67:0/1951631635 >> 172.21.15.67:0/1951631635 conn(0x7f261008bff0 msgr2=0x7f26100d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 -- 172.21.15.67:0/1951631635 shutdown_connections 2024-04-17T08:52:12.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.636+0000 7f2616600640 1 -- 172.21.15.67:0/1951631635 wait complete. 2024-04-17T08:52:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.638+0000 7f2616600640 1 Processor -- start 2024-04-17T08:52:12.636 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.638+0000 7f9de2293040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.638+0000 7f2616600640 1 -- start start 2024-04-17T08:52:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26100b4fd0 0x7f261014cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100d33e0 0x7f2610159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f261015f210 con 0x7f261015ecb0 2024-04-17T08:52:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f261015f090 con 0x7f26100d33e0 2024-04-17T08:52:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2616600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26101698f0 con 0x7f26100b4fd0 2024-04-17T08:52:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.639+0000 7f2615c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48350/0 (socket says 172.21.15.67:48350) 2024-04-17T08:52:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 -- 172.21.15.67:0/77555629 learned_addr learned my addr 172.21.15.67:0/77555629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615200640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26100b4fd0 0x7f261014cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f260fe00640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100d33e0 0x7f2610159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26100b4fd0 msgr2=0x7f261014cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26100b4fd0 0x7f261014cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100d33e0 msgr2=0x7f2610159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100d33e0 0x7f2610159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.640+0000 7f2615c00640 1 -- 172.21.15.67:0/77555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26100b73a0 con 0x7f261015ecb0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f2615c00640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f2608009e80 tx=0x7f2608004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608039050 con 0x7f261015ecb0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2608045070 con 0x7f261015ecb0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261015c9c0 con 0x7f261015ecb0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608009450 con 0x7f261015ecb0 2024-04-17T08:52:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.641+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26100b7640 con 0x7f261015ecb0 2024-04-17T08:52:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.642+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2610068a10 con 0x7f261015ecb0 2024-04-17T08:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.643+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f260803e610 con 0x7f261015ecb0 2024-04-17T08:52:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.644+0000 7f260cc00640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 0x7f25f006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.644+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f26080aefe0 con 0x7f261015ecb0 2024-04-17T08:52:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.645+0000 7f2615200640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 msgr2=0x7f25f006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.645+0000 7f2615200640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 0x7f25f006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:12.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.649+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f260807cbd0 con 0x7f261015ecb0 2024-04-17T08:52:12.654 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.655+0000 7f32ef016040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:12.710 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.711+0000 7f9de2293040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:12.715 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.716+0000 7f32ef016040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:12.764 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.765+0000 7f9de2293040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:12.767 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:12.768+0000 7f32ef016040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:12.824 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.825+0000 7f9de2293040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.846+0000 7f2615200640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 msgr2=0x7f25f006eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.846+0000 7f2615200640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 0x7f25f006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:12.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.858+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2610156700 con 0x7f261015ecb0 2024-04-17T08:52:12.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.859+0000 7f260cc00640 1 -- 172.21.15.67:0/77555629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v255) ==== 104+0+11 (secure 0 0 0) 0x7f260807c4d0 con 0x7f261015ecb0 2024-04-17T08:52:12.857 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:52:12.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 msgr2=0x7f25f006eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 0x7f25f006eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 msgr2=0x7f261015bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f2608009e80 tx=0x7f2608004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 shutdown_connections 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f25f006cb30 0x7f25f006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f261015ecb0 0x7f261015bc10 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26100d33e0 0x7f2610159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 --2- 172.21.15.67:0/77555629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26100b4fd0 0x7f261014cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 >> 172.21.15.67:0/77555629 conn(0x7f261008bff0 msgr2=0x7f26100d4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 shutdown_connections 2024-04-17T08:52:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:12.862+0000 7f2616600640 1 -- 172.21.15.67:0/77555629 wait complete. 2024-04-17T08:52:12.878 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:12.880+0000 7f9de2293040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:12.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:52:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.136+0000 7f19d3200640 1 Processor -- start 2024-04-17T08:52:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.136+0000 7f19d3200640 1 -- start start 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19d40d0690 0x7f19d4072440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d415cfd0 con 0x7f19d406cda0 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d3200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d4072bb0 con 0x7f19d415a380 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f19d4072980 con 0x7f19d40d0690 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19d40d0690 0x7f19d4072440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19d406cda0 0x7f19d40cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52194/0 (socket says 172.21.15.67:52194) 2024-04-17T08:52:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d1400640 1 -- 172.21.15.67:0/3797818768 learned_addr learned my addr 172.21.15.67:0/3797818768 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d2800640 1 -- 172.21.15.67:0/3797818768 >> v1:172.21.15.187:6789/0 conn(0x7f19d415a380 legacy=0x7f19d415d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d2800640 1 -- 172.21.15.67:0/3797818768 >> v1:172.21.15.67:6789/0 conn(0x7f19d406cda0 legacy=0x7f19d40cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.138+0000 7f19d2800640 1 -- 172.21.15.67:0/3797818768 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d4072e90 con 0x7f19d40d0690 2024-04-17T08:52:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.140+0000 7f19d2800640 1 --2- 172.21.15.67:0/3797818768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40d0690 0x7f19d4072440 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51851f442f7c2fdf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.140+0000 7f19d0a00640 1 -- 172.21.15.67:0/3797818768 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc039070 con 0x7f19d40d0690 2024-04-17T08:52:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.140+0000 7f19d0a00640 1 -- 172.21.15.67:0/3797818768 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19cc02cbd0 con 0x7f19d40d0690 2024-04-17T08:52:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.140+0000 7f19d0a00640 1 -- 172.21.15.67:0/3797818768 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc034440 con 0x7f19d40d0690 2024-04-17T08:52:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 -- 172.21.15.67:0/3797818768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40d0690 msgr2=0x7f19d4072440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 --2- 172.21.15.67:0/3797818768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40d0690 0x7f19d4072440 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009d90 tx=0x7f19cc02be20 comp rx=0 tx=0).stop 2024-04-17T08:52:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 -- 172.21.15.67:0/3797818768 shutdown_connections 2024-04-17T08:52:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 --2- 172.21.15.67:0/3797818768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d40d0690 0x7f19d4072440 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 -- 172.21.15.67:0/3797818768 >> 172.21.15.67:0/3797818768 conn(0x7f19d40d0b80 msgr2=0x7f19d40d0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 -- 172.21.15.67:0/3797818768 shutdown_connections 2024-04-17T08:52:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.141+0000 7f19d3200640 1 -- 172.21.15.67:0/3797818768 wait complete. 2024-04-17T08:52:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.144+0000 7f19d3200640 1 Processor -- start 2024-04-17T08:52:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.144+0000 7f19d3200640 1 -- start start 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19d40d0690 0x7f19d40aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d415a380 0x7f19d4160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19d415a760 con 0x7f19d406cda0 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19d415dea0 con 0x7f19d415a380 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.145+0000 7f19d3200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19d415a8e0 con 0x7f19d40d0690 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48380/0 (socket says 172.21.15.67:48380) 2024-04-17T08:52:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 -- 172.21.15.67:0/1215836915 learned_addr learned my addr 172.21.15.67:0/1215836915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1400640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19d40d0690 0x7f19d40aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d2800640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d415a380 0x7f19d4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19d40d0690 msgr2=0x7f19d40aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19d40d0690 0x7f19d40aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d415a380 msgr2=0x7f19d4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d415a380 0x7f19d4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.146+0000 7f19d1e00640 1 -- 172.21.15.67:0/1215836915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d40af500 con 0x7f19d406cda0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19d1e00640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c430 tx=0x7f19c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4011790 con 0x7f19d406cda0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c4017070 con 0x7f19d406cda0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d40af7e0 con 0x7f19d406cda0 2024-04-17T08:52:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4011b70 con 0x7f19d406cda0 2024-04-17T08:52:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.147+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d41f5df0 con 0x7f19d406cda0 2024-04-17T08:52:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.148+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d40755d0 con 0x7f19d406cda0 2024-04-17T08:52:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.149+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f19c4011d10 con 0x7f19d406cda0 2024-04-17T08:52:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.150+0000 7f19ca000640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 0x7f19b406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.150+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f19c408c2c0 con 0x7f19d406cda0 2024-04-17T08:52:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.150+0000 7f19d1400640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 msgr2=0x7f19b406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.151+0000 7f19d1400640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 0x7f19b406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:13.150 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.151+0000 7f32ef016040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.155+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f19c4059f20 con 0x7f19d406cda0 2024-04-17T08:52:13.297 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.299+0000 7f9de2293040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:13.310 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.311+0000 7f32ef016040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.352+0000 7f19d1400640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 msgr2=0x7f19b406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.352+0000 7f19d1400640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 0x7f19b406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.360+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19d415d830 con 0x7f19d406cda0 2024-04-17T08:52:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.361+0000 7f19ca000640 1 -- 172.21.15.67:0/1215836915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v255) ==== 100+0+11 (secure 0 0 0) 0x7f19c4059820 con 0x7f19d406cda0 2024-04-17T08:52:13.359 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:52:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.363+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 msgr2=0x7f19b406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.363+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 0x7f19b406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.363+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 msgr2=0x7f19d41f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.363+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c430 tx=0x7f19c400c8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 shutdown_connections 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f19b406cb30 0x7f19b406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d415a380 0x7f19d4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19d40d0690 0x7f19d40aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 --2- 172.21.15.67:0/1215836915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d406cda0 0x7f19d41f6310 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 >> 172.21.15.67:0/1215836915 conn(0x7f19d40d0b80 msgr2=0x7f19d4158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 shutdown_connections 2024-04-17T08:52:13.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.364+0000 7f19d3200640 1 -- 172.21.15.67:0/1215836915 wait complete. 2024-04-17T08:52:13.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:52:13.421 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.421+0000 7f32ef016040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:13.471 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.472+0000 7f32ef016040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:13.471 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.473+0000 7f9de2293040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:13.594 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.595+0000 7f9de2293040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.598+0000 7f3fc7e00640 1 Processor -- start 2024-04-17T08:52:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.598+0000 7f3fc7e00640 1 -- start start 2024-04-17T08:52:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80b3b50 0x7f3fc80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3fc80cfcd0 0x7f3fc80d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fc80d4f70 con 0x7f3fc815ec20 2024-04-17T08:52:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3fc80d5250 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc80cf4d0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3fc815ec20 0x7f3fc806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46940/0 (socket says 172.21.15.67:46940) 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc6a00640 1 -- 172.21.15.67:0/182622036 learned_addr learned my addr 172.21.15.67:0/182622036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.600+0000 7f3fc7400640 1 --2- 172.21.15.67:0/182622036 >> v2:172.21.15.67:3300/0 conn(0x7f3fc80cfcd0 0x7f3fc80d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525645212 0 0) 0x7f3fc80d4f70 con 0x7f3fc815ec20 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fbfe00640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80b3b50 0x7f3fc80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fac003520 con 0x7f3fc815ec20 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc7400640 1 -- 172.21.15.67:0/182622036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80b3b50 msgr2=0x7f3fc80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc7400640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80b3b50 0x7f3fc80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3283617194 0 0) 0x7f3fac003520 con 0x7f3fc815ec20 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc7400640 1 -- 172.21.15.67:0/182622036 >> v1:172.21.15.67:6789/0 conn(0x7f3fc815ec20 legacy=0x7f3fc806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.601+0000 7f3fc7400640 1 -- 172.21.15.67:0/182622036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc80cf7b0 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.603+0000 7f3fc7400640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80cfcd0 0x7f3fc80d0180 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009d90 tx=0x7f3fc002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c57aa8b2d9e248e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.603+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0039070 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.603+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc002c780 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.603+0000 7f3fc6000640 1 -- 172.21.15.67:0/182622036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8001670 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.602 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.603+0000 7f32ef016040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 -- 172.21.15.67:0/182622036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80cfcd0 msgr2=0x7f3fc80d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80cfcd0 0x7f3fc80d0180 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009d90 tx=0x7f3fc002bc50 comp rx=0 tx=0).stop 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 -- 172.21.15.67:0/182622036 shutdown_connections 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80cfcd0 0x7f3fc80d0180 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/182622036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80b3b50 0x7f3fc80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 -- 172.21.15.67:0/182622036 >> 172.21.15.67:0/182622036 conn(0x7f3fc808bff0 msgr2=0x7f3fc80d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 -- 172.21.15.67:0/182622036 shutdown_connections 2024-04-17T08:52:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.604+0000 7f3fc7e00640 1 -- 172.21.15.67:0/182622036 wait complete. 2024-04-17T08:52:13.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.606+0000 7f3fc7e00640 1 Processor -- start 2024-04-17T08:52:13.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.606+0000 7f3fc7e00640 1 -- start start 2024-04-17T08:52:13.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80cfcd0 0x7f3fc80b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fc815ec20 0x7f3fc80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc806d2d0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc806d150 con 0x7f3fc815ec20 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc80d10d0 con 0x7f3fc80cfcd0 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57928/0 (socket says 172.21.15.67:57928) 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 -- 172.21.15.67:0/1804922027 learned_addr learned my addr 172.21.15.67:0/1804922027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fbfe00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80cfcd0 0x7f3fc80b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc7400640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fc815ec20 0x7f3fc80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80cfcd0 msgr2=0x7f3fc80b9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80cfcd0 0x7f3fc80b9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fc815ec20 msgr2=0x7f3fc80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fc815ec20 0x7f3fc80b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.607+0000 7f3fc6a00640 1 -- 172.21.15.67:0/1804922027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc80cab30 con 0x7f3fc80b3b50 2024-04-17T08:52:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fc6a00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80017e0 tx=0x7f3fb800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8005ad0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fb8005c90 con 0x7f3fc80b3b50 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80072b0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc80cbcb0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.608+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc80cc550 con 0x7f3fc80b3b50 2024-04-17T08:52:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.609+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc8068a10 con 0x7f3fc80b3b50 2024-04-17T08:52:13.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.611+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f3fb8007450 con 0x7f3fc80b3b50 2024-04-17T08:52:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.612+0000 7f3fbea00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 0x7f3fa406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.612+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f3fb808c2c0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.612+0000 7f3fbfe00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 msgr2=0x7f3fa406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.612+0000 7f3fbfe00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 0x7f3fa406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.617+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3fb8010300 con 0x7f3fc80b3b50 2024-04-17T08:52:13.652 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.653+0000 7f9de2293040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:13.709 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.710+0000 7f32ef016040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:13.786 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.788+0000 7f9de2293040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:13.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.813+0000 7f3fbfe00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 msgr2=0x7f3fa406eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:13.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.813+0000 7f3fbfe00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 0x7f3fa406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:13.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.816+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f3fc80d55f0 con 0x7f3fc80b3b50 2024-04-17T08:52:13.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.816+0000 7f3fbea00640 1 -- 172.21.15.67:0/1804922027 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v255) ==== 107+0+11 (secure 0 0 0) 0x7f3fb8059e40 con 0x7f3fc80b3b50 2024-04-17T08:52:13.815 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 msgr2=0x7f3fa406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 0x7f3fa406eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 msgr2=0x7f3fc80ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80017e0 tx=0x7f3fb800cbe0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 shutdown_connections 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f3fa406cb30 0x7f3fa406eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3fc815ec20 0x7f3fc80b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3fc80cfcd0 0x7f3fc80b9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 --2- 172.21.15.67:0/1804922027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fc80b3b50 0x7f3fc80ca5f0 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 >> 172.21.15.67:0/1804922027 conn(0x7f3fc808bff0 msgr2=0x7f3fc80d3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:13.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 shutdown_connections 2024-04-17T08:52:13.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:13.819+0000 7f3fc7e00640 1 -- 172.21.15.67:0/1804922027 wait complete. 2024-04-17T08:52:13.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:52:13.883 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.884+0000 7f32ef016040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:13.903 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:13.904+0000 7f9de2293040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:13.932 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:13.933+0000 7f32ef016040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.043+0000 7f736a600640 1 Processor -- start 2024-04-17T08:52:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.044+0000 7f736a600640 1 -- start start 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.045+0000 7f736a600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f73640cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.045+0000 7f736a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73640b51d0 con 0x7f73640cf910 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.045+0000 7f736a600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73640b54b0 con 0x7f73640d4510 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.045+0000 7f736a600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73640b5790 con 0x7f736415e9b0 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.045+0000 7f7363e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f73640cf910 0x7f73640b46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46942/0 (socket says 172.21.15.67:46942) 2024-04-17T08:52:14.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363e00640 1 -- 172.21.15.67:0/3305330929 learned_addr learned my addr 172.21.15.67:0/3305330929 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7369200640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f73640cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3481766585 0 0) 0x7f73640b51d0 con 0x7f73640cf910 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f734c003520 con 0x7f73640cf910 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 744018449 0 0) 0x7f734c003520 con 0x7f73640cf910 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 msgr2=0x7f73640cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f73640cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 >> v1:172.21.15.187:6789/0 conn(0x7f73640d4510 legacy=0x7f73640d4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73640d24c0 con 0x7f73640cf910 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7350001670 con 0x7f73640cf910 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.046+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 >> v1:172.21.15.67:6789/0 conn(0x7f73640cf910 legacy=0x7f73640b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f734c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f734c003f80 0x7f734c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f734c004d40 0x7f734c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f73640b5790 con 0x7f734c004d40 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f734c00bc00 con 0x7f734c003f80 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f734c00be00 con 0x7f736415e9b0 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f734c004d40 0x7f734c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7363e00640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f734c003f80 0x7f734c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369200640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f734c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 -- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 msgr2=0x7f734c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415e9b0 0x7f734c003a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 -- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f734c003f80 msgr2=0x7f734c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f734c003f80 0x7f734c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.047+0000 7f7369c00640 1 -- 172.21.15.67:0/3305330929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c003520 con 0x7f734c004d40 2024-04-17T08:52:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.048+0000 7f7369c00640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f734c004d40 0x7f734c0051c0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f735c001e50 tx=0x7f734c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694ad06f384ac155 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c001a70 con 0x7f734c004d40 2024-04-17T08:52:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f735c003b70 con 0x7f734c004d40 2024-04-17T08:52:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f7363400640 1 -- 172.21.15.67:0/3305330929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0050d0 con 0x7f734c004d40 2024-04-17T08:52:14.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f736a600640 1 -- 172.21.15.67:0/3305330929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f734c004d40 msgr2=0x7f734c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f736a600640 1 --2- 172.21.15.67:0/3305330929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f734c004d40 0x7f734c0051c0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f735c001e50 tx=0x7f734c005830 comp rx=0 tx=0).stop 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.049+0000 7f7369c00640 1 -- 172.21.15.67:0/3305330929 reap_dead start 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.050+0000 7f736a600640 1 -- 172.21.15.67:0/3305330929 shutdown_connections 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.050+0000 7f736a600640 1 -- 172.21.15.67:0/3305330929 >> 172.21.15.67:0/3305330929 conn(0x7f736409daa0 msgr2=0x7f736408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.050+0000 7f736a600640 1 -- 172.21.15.67:0/3305330929 shutdown_connections 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.050+0000 7f736a600640 1 -- 172.21.15.67:0/3305330929 wait complete. 2024-04-17T08:52:14.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.052+0000 7f736a600640 1 Processor -- start 2024-04-17T08:52:14.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.053+0000 7f736a600640 1 -- start start 2024-04-17T08:52:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7364151b00 0x7f7364160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415aaa0 0x7f73640cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f73640b4e00 con 0x7f736415ed90 2024-04-17T08:52:14.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f73640d2830 con 0x7f7364151b00 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f736a600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73640b4f80 con 0x7f736415aaa0 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f7363e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f7363e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57944/0 (socket says 172.21.15.67:57944) 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.054+0000 7f7363e00640 1 -- 172.21.15.67:0/2794815819 learned_addr learned my addr 172.21.15.67:0/2794815819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7369200640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7364151b00 0x7f7364160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7369c00640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415aaa0 0x7f73640cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415aaa0 msgr2=0x7f73640cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415aaa0 0x7f73640cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7364151b00 msgr2=0x7f7364160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7364151b00 0x7f7364160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 -- 172.21.15.67:0/2794815819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736415f290 con 0x7f736415ed90 2024-04-17T08:52:14.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.055+0000 7f7363e00640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f735000e730 tx=0x7f735000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.056+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735000ee40 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.056+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7350018070 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.056+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735000cab0 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.056+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7364161760 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.056+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7364161d20 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.057+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7364161f20 con 0x7f736415ed90 2024-04-17T08:52:14.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.058+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f735000cc50 con 0x7f736415ed90 2024-04-17T08:52:14.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.059+0000 7f7360c00640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 0x7f734406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.059+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f735008bfb0 con 0x7f736415ed90 2024-04-17T08:52:14.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.059+0000 7f7369200640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 msgr2=0x7f734406f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:14.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.059+0000 7f7369200640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 0x7f734406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:14.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.063+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7364161f20 con 0x7f736415ed90 2024-04-17T08:52:14.070 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:14.070+0000 7f32ef016040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:14.095 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:14.097+0000 7f9de2293040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:14.148 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:14.150+0000 7f9de2293040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:14.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.260+0000 7f7369200640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 msgr2=0x7f734406f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:14.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.260+0000 7f7369200640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 0x7f734406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.266+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f73640755d0 con 0x7f736415ed90 2024-04-17T08:52:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.266+0000 7f7360c00640 1 -- 172.21.15.67:0/2794815819 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v255) ==== 103+0+11 (secure 0 0 0) 0x7f7350059b80 con 0x7f736415ed90 2024-04-17T08:52:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:52:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 msgr2=0x7f734406f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 0x7f734406f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 msgr2=0x7f7364160ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f735000e730 tx=0x7f735000ebf0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 shutdown_connections 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f734406cb60 0x7f734406f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f736415aaa0 0x7f73640cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f736415ed90 0x7f7364160ed0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 --2- 172.21.15.67:0/2794815819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7364151b00 0x7f7364160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 >> 172.21.15.67:0/2794815819 conn(0x7f736409daa0 msgr2=0x7f736406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 shutdown_connections 2024-04-17T08:52:14.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.269+0000 7f736a600640 1 -- 172.21.15.67:0/2794815819 wait complete. 2024-04-17T08:52:14.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:52:14.296 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:14.297+0000 7f9de2293040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:14.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.495+0000 7f6062600640 1 Processor -- start 2024-04-17T08:52:14.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.495+0000 7f6062600640 1 -- start start 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6062600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f605c0d1ab0 0x7f605c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6062600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6062600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c0d4140 con 0x7f605c15ee20 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6062600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f605c0b83b0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6062600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f605c0d3b80 con 0x7f605c0d3350 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f605be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f605c0d1ab0 0x7f605c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6061200640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f605c15ee20 0x7f605c0d13a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:35562/0 (socket says 172.21.15.67:35562) 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6061c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.497+0000 7f6061200640 1 -- 172.21.15.67:0/3238888848 learned_addr learned my addr 172.21.15.67:0/3238888848 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605b400640 1 -- 172.21.15.67:0/3238888848 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879384196 0 0) 0x7f605c0d4140 con 0x7f605c15ee20 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605b400640 1 -- 172.21.15.67:0/3238888848 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6044003520 con 0x7f605c15ee20 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605be00640 1 -- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 msgr2=0x7f605c0b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605be00640 1 --2- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605be00640 1 -- 172.21.15.67:0/3238888848 >> v1:172.21.15.187:6789/0 conn(0x7f605c15ee20 legacy=0x7f605c0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.498+0000 7f605be00640 1 -- 172.21.15.67:0/3238888848 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c0d3e60 con 0x7f605c0d1ab0 2024-04-17T08:52:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.500+0000 7f605be00640 1 --2- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c0d1ab0 0x7f605c0d2e10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f60480097a0 tx=0x7f604802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e83507b8424b2382 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.500+0000 7f605b400640 1 -- 172.21.15.67:0/3238888848 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048035020 con 0x7f605c0d1ab0 2024-04-17T08:52:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.500+0000 7f605b400640 1 -- 172.21.15.67:0/3238888848 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6048031810 con 0x7f605c0d1ab0 2024-04-17T08:52:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.500+0000 7f605b400640 1 -- 172.21.15.67:0/3238888848 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048031bc0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 -- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c0d1ab0 msgr2=0x7f605c0d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 --2- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c0d1ab0 0x7f605c0d2e10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f60480097a0 tx=0x7f604802b8b0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 -- 172.21.15.67:0/3238888848 shutdown_connections 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 --2- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 --2- 172.21.15.67:0/3238888848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c0d1ab0 0x7f605c0d2e10 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 -- 172.21.15.67:0/3238888848 >> 172.21.15.67:0/3238888848 conn(0x7f605c06d3c0 msgr2=0x7f605c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 -- 172.21.15.67:0/3238888848 shutdown_connections 2024-04-17T08:52:14.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.501+0000 7f6062600640 1 -- 172.21.15.67:0/3238888848 wait complete. 2024-04-17T08:52:14.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.502+0000 7f6062600640 1 Processor -- start 2024-04-17T08:52:14.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.503+0000 7f6062600640 1 -- start start 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c15ee20 0x7f605c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0d3800 con 0x7f605c0d1ab0 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f605c15f200 con 0x7f605c15ee20 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6062600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f605c0b8d80 con 0x7f605c0d3350 2024-04-17T08:52:14.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57950/0 (socket says 172.21.15.67:57950) 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 -- 172.21.15.67:0/457113738 learned_addr learned my addr 172.21.15.67:0/457113738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f605be00640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061c00640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c15ee20 0x7f605c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 msgr2=0x7f605c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c15ee20 msgr2=0x7f605c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c15ee20 0x7f605c0cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.504+0000 7f6061200640 1 -- 172.21.15.67:0/457113738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c0ccbe0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6061200640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f605000c440 tx=0x7f605000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050017070 con 0x7f605c0d1ab0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6050005ce0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500072b0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c0ccec0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.505+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c0cd480 con 0x7f605c0d1ab0 2024-04-17T08:52:14.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.506+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c068a10 con 0x7f605c0d1ab0 2024-04-17T08:52:14.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.507+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7f605001a070 con 0x7f605c0d1ab0 2024-04-17T08:52:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.508+0000 7f6058c00640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 0x7f603c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.508+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7f605008b7e0 con 0x7f605c0d1ab0 2024-04-17T08:52:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.509+0000 7f605be00640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 msgr2=0x7f603c06f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.509+0000 7f605be00640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 0x7f603c06f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:14.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.517+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6050059430 con 0x7f605c0d1ab0 2024-04-17T08:52:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.710+0000 7f605be00640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 msgr2=0x7f603c06f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.710+0000 7f605be00640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 0x7f603c06f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:14.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.728+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f605c15e460 con 0x7f605c0d1ab0 2024-04-17T08:52:14.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.729+0000 7f6058c00640 1 -- 172.21.15.67:0/457113738 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v255) ==== 81+0+593 (secure 0 0 0) 0x7f6050058d30 con 0x7f605c0d1ab0 2024-04-17T08:52:14.727 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/2123602634 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.727 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/2123602634 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3628769219 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/1806888124 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2102410647 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2774796164 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/609167221 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/259125585 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/545321551 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2338521910 2024-04-18T08:52:09.047891+0000 2024-04-17T08:52:14.728 INFO:teuthology.orchestra.run.smithi067.stderr:listed 10 entries 2024-04-17T08:52:14.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 msgr2=0x7f603c06f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:14.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 0x7f603c06f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 msgr2=0x7f605c0caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:14.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f605000c440 tx=0x7f605000c900 comp rx=0 tx=0).stop 2024-04-17T08:52:14.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 shutdown_connections 2024-04-17T08:52:14.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7f603c06cb60 0x7f603c06f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f605c15ee20 0x7f605c0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605c0d3350 0x7f605c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 --2- 172.21.15.67:0/457113738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f605c0d1ab0 0x7f605c0caef0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:14.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.731+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 >> 172.21.15.67:0/457113738 conn(0x7f605c06d3c0 msgr2=0x7f605c15ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:14.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.732+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 shutdown_connections 2024-04-17T08:52:14.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:14.732+0000 7f6062600640 1 -- 172.21.15.67:0/457113738 wait complete. 2024-04-17T08:52:14.749 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:52:14.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:14.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:52:14.873 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:52:14.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:14.875 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:52:14.888 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:14.889+0000 7f32ef016040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:14.915 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:52:14.916 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:52:14.916 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:14.969 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:52:14.969 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:52:14.969 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:14.969 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:15.023 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:52:15.023 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:52:15.024 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:52:15.024 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:15.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:52:15.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:15.177 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:15.179+0000 7f9de2293040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:15.201 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.202+0000 7f32ef016040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:52:15.258 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.259+0000 7f32ef016040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:52:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.295+0000 7fce6e400640 1 Processor -- start 2024-04-17T08:52:15.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.295+0000 7fce6e400640 1 -- start start 2024-04-17T08:52:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce680ce770 0x7fce681625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680d1f00 0x7fce68162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6e400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce680d0e60 con 0x7fce68163020 2024-04-17T08:52:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fce680ceb50 con 0x7fce680ce770 2024-04-17T08:52:15.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fce680d1a70 con 0x7fce680d1f00 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce680ce770 0x7fce681625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce680ce770 0x7fce681625a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57960/0 (socket says 172.21.15.67:57960) 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.297+0000 7fce6d000640 1 -- 172.21.15.67:0/2477208135 learned_addr learned my addr 172.21.15.67:0/2477208135 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.298+0000 7fce67e00640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680d1f00 0x7fce68162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.298+0000 7fce6d000640 1 -- 172.21.15.67:0/2477208135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680d1f00 msgr2=0x7fce68162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:15.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.298+0000 7fce6d000640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680d1f00 0x7fce68162ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.298+0000 7fce6d000640 1 -- 172.21.15.67:0/2477208135 >> v1:172.21.15.187:6789/0 conn(0x7fce68163020 legacy=0x7fce680cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.298+0000 7fce6d000640 1 -- 172.21.15.67:0/2477208135 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce680d1140 con 0x7fce680ce770 2024-04-17T08:52:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.299+0000 7fce6d000640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680ce770 0x7fce681625a0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009870 tx=0x7fce5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc9be0e5893c27e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.299+0000 7fce67400640 1 -- 172.21.15.67:0/2477208135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c035020 con 0x7fce680ce770 2024-04-17T08:52:15.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.299+0000 7fce67400640 1 -- 172.21.15.67:0/2477208135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5c031640 con 0x7fce680ce770 2024-04-17T08:52:15.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.299+0000 7fce67400640 1 -- 172.21.15.67:0/2477208135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c02d040 con 0x7fce680ce770 2024-04-17T08:52:15.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 -- 172.21.15.67:0/2477208135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680ce770 msgr2=0x7fce681625a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680ce770 0x7fce681625a0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009870 tx=0x7fce5c02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 -- 172.21.15.67:0/2477208135 shutdown_connections 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680d1f00 0x7fce68162ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 --2- 172.21.15.67:0/2477208135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680ce770 0x7fce681625a0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 -- 172.21.15.67:0/2477208135 >> 172.21.15.67:0/2477208135 conn(0x7fce680b4170 msgr2=0x7fce680b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 -- 172.21.15.67:0/2477208135 shutdown_connections 2024-04-17T08:52:15.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.300+0000 7fce6e400640 1 -- 172.21.15.67:0/2477208135 wait complete. 2024-04-17T08:52:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.302+0000 7fce6e400640 1 Processor -- start 2024-04-17T08:52:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.302+0000 7fce6e400640 1 -- start start 2024-04-17T08:52:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680ce770 0x7fce681f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 0x7fce680b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce680d09f0 con 0x7fce680d1f00 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fce680d1700 con 0x7fce68163020 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce6e400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fce680d0b70 con 0x7fce680ce770 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57968/0 (socket says 172.21.15.67:57968) 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.303+0000 7fce67e00640 1 -- 172.21.15.67:0/1554813385 learned_addr learned my addr 172.21.15.67:0/1554813385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce6d000640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680ce770 0x7fce681f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 -- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680ce770 msgr2=0x7fce681f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680ce770 0x7fce681f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 -- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 msgr2=0x7fce680b71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce6da00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 0x7fce680b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 0x7fce680b71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 -- 172.21.15.67:0/1554813385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce681f5850 con 0x7fce680d1f00 2024-04-17T08:52:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce6da00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 0x7fce680b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce67e00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fce5400c7f0 tx=0x7fce5400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.304+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54015070 con 0x7fce680d1f00 2024-04-17T08:52:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.305+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce54004220 con 0x7fce680d1f00 2024-04-17T08:52:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.305+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54007620 con 0x7fce680d1f00 2024-04-17T08:52:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.305+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce681f60b0 con 0x7fce680d1f00 2024-04-17T08:52:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.305+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681f6670 con 0x7fce680d1f00 2024-04-17T08:52:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.306+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce680755d0 con 0x7fce680d1f00 2024-04-17T08:52:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.307+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 143) ==== 91245+0+0 (secure 0 0 0) 0x7fce540079d0 con 0x7fce680d1f00 2024-04-17T08:52:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.308+0000 7fce64c00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 0x7fce4806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.308+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 7031+0+0 (secure 0 0 0) 0x7fce5408c9f0 con 0x7fce680d1f00 2024-04-17T08:52:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.308+0000 7fce6d000640 1 -- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 msgr2=0x7fce4806eff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/986946406 2024-04-17T08:52:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.309+0000 7fce6d000640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 0x7fce4806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:15.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.313+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fce5405a470 con 0x7fce680d1f00 2024-04-17T08:52:15.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.390+0000 7f32ef016040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:52:15.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.413+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 144) ==== 90751+0+0 (secure 0 0 0) 0x7fce540563e0 con 0x7fce680d1f00 2024-04-17T08:52:15.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.413+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 msgr2=0x7fce4806eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:15.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.413+0000 7fce64c00640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 0x7fce4806eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.489 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.489 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.490 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.490+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.505 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.505 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.505 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.505 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.505 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f3268800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.506 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.506 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.506 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.506 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.506 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:15.506+0000 7f326c400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:52:15.518 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:15.520+0000 7f9de2293040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:52:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.573+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce68161680 con 0x7fce680d1f00 2024-04-17T08:52:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.575+0000 7fce64c00640 1 -- 172.21.15.67:0/1554813385 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7fce54060880 con 0x7fce680d1f00 2024-04-17T08:52:15.573 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:15.573 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.577+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 msgr2=0x7fce681f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.577+0000 7fce6e400640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fce5400c7f0 tx=0x7fce5400ccb0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 shutdown_connections 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:6832/986946406,v1:172.21.15.187:6833/986946406] conn(0x7fce4806cb30 0x7fce4806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce68163020 0x7fce680b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce680d1f00 0x7fce681f5310 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 --2- 172.21.15.67:0/1554813385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce680ce770 0x7fce681f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 >> 172.21.15.67:0/1554813385 conn(0x7fce680b4170 msgr2=0x7fce6809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 shutdown_connections 2024-04-17T08:52:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.578+0000 7fce6e400640 1 -- 172.21.15.67:0/1554813385 wait complete. 2024-04-17T08:52:15.579 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:15.581+0000 7f9de2293040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:52:15.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:15.724 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:15.725+0000 7f9de2293040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:52:15.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.802+0000 7f64b7200640 1 Processor -- start 2024-04-17T08:52:15.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.803+0000 7f64b7200640 1 -- start start 2024-04-17T08:52:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 0x7f64b015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64b00b3c50 0x7f64b006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f64b00705f0 0x7f64b0070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f64b015f620 con 0x7f64b00b3c50 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f64b00b39d0 con 0x7f64b00705f0 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.804+0000 7f64b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64b006bfa0 con 0x7f64b00d1080 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64b00b3c50 0x7f64b006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64b00b3c50 0x7f64b006b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57978/0 (socket says 172.21.15.67:57978) 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 -- 172.21.15.67:0/310197232 learned_addr learned my addr 172.21.15.67:0/310197232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b6800640 1 --2- 172.21.15.67:0/310197232 >> v2:172.21.15.187:3300/0 conn(0x7f64b00705f0 0x7f64b0070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5e00640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 0x7f64b015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 -- 172.21.15.67:0/310197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 msgr2=0x7f64b015fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 0x7f64b015fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 -- 172.21.15.67:0/310197232 >> v2:172.21.15.187:3300/0 conn(0x7f64b00705f0 msgr2=0x7f64b0070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 --2- 172.21.15.67:0/310197232 >> v2:172.21.15.187:3300/0 conn(0x7f64b00705f0 0x7f64b0070aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5400640 1 -- 172.21.15.67:0/310197232 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b006c280 con 0x7f64b00b3c50 2024-04-17T08:52:15.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.805+0000 7f64b5e00640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 0x7f64b015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.807+0000 7f64b5400640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00b3c50 0x7f64b006b8c0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f64b009b250 tx=0x7f64a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281493af9d862ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.807+0000 7f64b4a00640 1 -- 172.21.15.67:0/310197232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0035020 con 0x7f64b00b3c50 2024-04-17T08:52:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.807+0000 7f64b4a00640 1 -- 172.21.15.67:0/310197232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a0031870 con 0x7f64b00b3c50 2024-04-17T08:52:15.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.807+0000 7f64b4a00640 1 -- 172.21.15.67:0/310197232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0031c40 con 0x7f64b00b3c50 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 -- 172.21.15.67:0/310197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00b3c50 msgr2=0x7f64b006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00b3c50 0x7f64b006b8c0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f64b009b250 tx=0x7f64a002b910 comp rx=0 tx=0).stop 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 -- 172.21.15.67:0/310197232 shutdown_connections 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 --2- 172.21.15.67:0/310197232 >> v2:172.21.15.187:3300/0 conn(0x7f64b00705f0 0x7f64b0070aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00b3c50 0x7f64b006b8c0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 --2- 172.21.15.67:0/310197232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00d1080 0x7f64b015fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 -- 172.21.15.67:0/310197232 >> 172.21.15.67:0/310197232 conn(0x7f64b009aa60 msgr2=0x7f64b00b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:15.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 -- 172.21.15.67:0/310197232 shutdown_connections 2024-04-17T08:52:15.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.808+0000 7f64b7200640 1 -- 172.21.15.67:0/310197232 wait complete. 2024-04-17T08:52:15.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.809+0000 7f64b7200640 1 Processor -- start 2024-04-17T08:52:15.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.809+0000 7f64b7200640 1 -- start start 2024-04-17T08:52:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00b3c50 0x7f64b015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64b00d1080 0x7f64b00b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64b0162550 con 0x7f64b00705f0 2024-04-17T08:52:15.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f64b015ef60 con 0x7f64b00d1080 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.810+0000 7f64b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64b01626d0 con 0x7f64b00b3c50 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57990/0 (socket says 172.21.15.67:57990) 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 -- 172.21.15.67:0/2496560347 learned_addr learned my addr 172.21.15.67:0/2496560347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5400640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00b3c50 0x7f64b015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 -- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00b3c50 msgr2=0x7f64b015a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00b3c50 0x7f64b015a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 -- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64b00d1080 msgr2=0x7f64b00b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64b00d1080 0x7f64b00b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:15.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 -- 172.21.15.67:0/2496560347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b00ca460 con 0x7f64b00705f0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.811+0000 7f64b5e00640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f64a800ce00 tx=0x7f64a800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.812+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a800f890 con 0x7f64b00705f0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.812+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a800fa30 con 0x7f64b00705f0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.812+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a800fe00 con 0x7f64b00705f0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.812+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b00ca740 con 0x7f64b00705f0 2024-04-17T08:52:15.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.812+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b0150380 con 0x7f64b00705f0 2024-04-17T08:52:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.813+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b00755d0 con 0x7f64b00705f0 2024-04-17T08:52:15.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.814+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 144) ==== 90751+0+0 (secure 0 0 0) 0x7f64a8018400 con 0x7f64b00705f0 2024-04-17T08:52:15.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.815+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7f64a808cb30 con 0x7f64b00705f0 2024-04-17T08:52:15.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:15.819+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f64a8014030 con 0x7f64b00705f0 2024-04-17T08:52:16.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.021+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64b00ca910 con 0x7f64b00705f0 2024-04-17T08:52:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.025+0000 7f64a6000640 1 -- 172.21.15.67:0/2496560347 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) ==== 74+0+21613 (secure 0 0 0) 0x7f64a800ae20 con 0x7f64b00705f0 2024-04-17T08:52:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:16.024 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":256,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:15.395631+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.027+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 msgr2=0x7f64b014fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.027+0000 7f64b7200640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f64a800ce00 tx=0x7f64a800a650 comp rx=0 tx=0).stop 2024-04-17T08:52:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 shutdown_connections 2024-04-17T08:52:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64b00d1080 0x7f64b00b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64b00b3c50 0x7f64b015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 --2- 172.21.15.67:0/2496560347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64b00705f0 0x7f64b014fe40 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 >> 172.21.15.67:0/2496560347 conn(0x7f64b009aa60 msgr2=0x7f64b0161e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 shutdown_connections 2024-04-17T08:52:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.028+0000 7f64b7200640 1 -- 172.21.15.67:0/2496560347 wait complete. 2024-04-17T08:52:16.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:51:38.821605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:51:39.950246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:51:54.448741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:51:55.106540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:52:16.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:52:16.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.272+0000 7fd302000640 1 Processor -- start 2024-04-17T08:52:16.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.272+0000 7fd302000640 1 -- start start 2024-04-17T08:52:16.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.273+0000 7fd302000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd2fc145b00 0x7fd2fc160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.273+0000 7fd302000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.273+0000 7fd302000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2fc160920 con 0x7fd2fc146060 2024-04-17T08:52:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.273+0000 7fd302000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd2fc0b5450 con 0x7fd2fc145b00 2024-04-17T08:52:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.273+0000 7fd302000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2fc160c00 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.274+0000 7fd301600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.274+0000 7fd301600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47672/0 (socket says 172.21.15.67:47672) 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.274+0000 7fd301600640 1 -- 172.21.15.67:0/3303411035 learned_addr learned my addr 172.21.15.67:0/3303411035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.274+0000 7fd300c00640 1 --2- 172.21.15.67:0/3303411035 >> v2:172.21.15.187:3300/0 conn(0x7fd2fc145b00 0x7fd2fc160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.275+0000 7fd301600640 1 -- 172.21.15.67:0/3303411035 >> v2:172.21.15.187:3300/0 conn(0x7fd2fc145b00 msgr2=0x7fd2fc160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.275+0000 7fd301600640 1 --2- 172.21.15.67:0/3303411035 >> v2:172.21.15.187:3300/0 conn(0x7fd2fc145b00 0x7fd2fc160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.275+0000 7fd301600640 1 -- 172.21.15.67:0/3303411035 >> v1:172.21.15.187:6789/0 conn(0x7fd2fc146060 legacy=0x7fd2fc15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.275+0000 7fd301600640 1 -- 172.21.15.67:0/3303411035 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc160ee0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.276+0000 7fd301600640 1 --2- 172.21.15.67:0/3303411035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009e50 tx=0x7fd2f402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd207b344aae8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.277+0000 7fd2fb400640 1 -- 172.21.15.67:0/3303411035 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f402ca70 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.277+0000 7fd2fb400640 1 -- 172.21.15.67:0/3303411035 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f402cc10 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.277+0000 7fd2fb400640 1 -- 172.21.15.67:0/3303411035 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40343f0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 -- 172.21.15.67:0/3303411035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 msgr2=0x7fd2fc0b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 --2- 172.21.15.67:0/3303411035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009e50 tx=0x7fd2f402bed0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 -- 172.21.15.67:0/3303411035 shutdown_connections 2024-04-17T08:52:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 --2- 172.21.15.67:0/3303411035 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc0b4ac0 0x7fd2fc0b4f10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 --2- 172.21.15.67:0/3303411035 >> v2:172.21.15.187:3300/0 conn(0x7fd2fc145b00 0x7fd2fc160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 -- 172.21.15.67:0/3303411035 >> 172.21.15.67:0/3303411035 conn(0x7fd2fc0b23c0 msgr2=0x7fd2fc0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 -- 172.21.15.67:0/3303411035 shutdown_connections 2024-04-17T08:52:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.278+0000 7fd302000640 1 -- 172.21.15.67:0/3303411035 wait complete. 2024-04-17T08:52:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.280+0000 7fd302000640 1 Processor -- start 2024-04-17T08:52:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.280+0000 7fd302000640 1 -- start start 2024-04-17T08:52:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc145b00 0x7fd2fc13acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2fc146060 0x7fd2fc13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc0cfba0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc0b3760 con 0x7fd2fc146060 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd302000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2fc0b3c80 con 0x7fd2fc145b00 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd300c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd300c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58004/0 (socket says 172.21.15.67:58004) 2024-04-17T08:52:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.281+0000 7fd300c00640 1 -- 172.21.15.67:0/4005762826 learned_addr learned my addr 172.21.15.67:0/4005762826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd2fbe00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc145b00 0x7fd2fc13acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd301600640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2fc146060 0x7fd2fc13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 -- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc145b00 msgr2=0x7fd2fc13acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc145b00 0x7fd2fc13acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 -- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2fc146060 msgr2=0x7fd2fc13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2fc146060 0x7fd2fc13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 -- 172.21.15.67:0/4005762826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc0cb1f0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd300c00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00e930 tx=0x7fd2ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec017070 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ec00ca00 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00cdf0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.282+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc0cb4d0 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.283+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc0cba90 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.284+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc068a10 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.285+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 144) ==== 90751+0+0 (secure 0 0 0) 0x7fd2ec01a070 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.286+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fd2ec08c150 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.290+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd2ec05a320 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.419+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 145) ==== 90878+0+0 (secure 0 0 0) 0x7fd2ec055970 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.420+0000 7fd2f8c00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 0x7fd2dc076c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.430+0000 7fd2fbe00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 0x7fd2dc076c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.445+0000 7fd2fbe00640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 0x7fd2dc076c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8007fd0 tx=0x7fd2e8009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.495+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2fc068610 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.496+0000 7fd2f8c00640 1 -- 172.21.15.67:0/4005762826 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v256) ==== 93+0+10 (secure 0 0 0) 0x7fd2ec03bb90 con 0x7fd2fc0b4ac0 2024-04-17T08:52:16.494 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 msgr2=0x7fd2dc076c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 0x7fd2dc076c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8007fd0 tx=0x7fd2e8009040 comp rx=0 tx=0).stop 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 msgr2=0x7fd2fc074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00e930 tx=0x7fd2ec00edf0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 shutdown_connections 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd2dc074850 0x7fd2dc076c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2fc146060 0x7fd2fc13b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2fc145b00 0x7fd2fc13acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 --2- 172.21.15.67:0/4005762826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc0b4ac0 0x7fd2fc074fb0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.498+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 >> 172.21.15.67:0/4005762826 conn(0x7fd2fc0b23c0 msgr2=0x7fd2fc15dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:16.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.499+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 shutdown_connections 2024-04-17T08:52:16.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.499+0000 7fd302000640 1 -- 172.21.15.67:0/4005762826 wait complete. 2024-04-17T08:52:16.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:52:16.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.739+0000 7fe238000640 1 Processor -- start 2024-04-17T08:52:16.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.740+0000 7fe238000640 1 -- start start 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.741+0000 7fe238000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.741+0000 7fe238000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe23006c8d0 0x7fe23006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.741+0000 7fe238000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2300cb960 con 0x7fe2300cb3f0 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.741+0000 7fe238000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe2300cbc40 con 0x7fe23006c8d0 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.741+0000 7fe238000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe2300cbf20 con 0x7fe23015ec60 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe23006c8d0 0x7fe23006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe23006c8d0 0x7fe23006cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58008/0 (socket says 172.21.15.67:58008) 2024-04-17T08:52:16.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 -- 172.21.15.67:0/3982323397 learned_addr learned my addr 172.21.15.67:0/3982323397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe235600640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 -- 172.21.15.67:0/3982323397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 msgr2=0x7fe2300b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 -- 172.21.15.67:0/3982323397 >> v1:172.21.15.187:6789/0 conn(0x7fe2300cb3f0 legacy=0x7fe23006edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.742+0000 7fe236000640 1 -- 172.21.15.67:0/3982323397 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2300cc200 con 0x7fe23006c8d0 2024-04-17T08:52:16.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.743+0000 7fe236000640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe23006c8d0 0x7fe23006cd60 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fe228009e50 tx=0x7fe22802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22200867418fc913 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe22fe00640 1 -- 172.21.15.67:0/3982323397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22802cb00 con 0x7fe23006c8d0 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe22fe00640 1 -- 172.21.15.67:0/3982323397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe22802cca0 con 0x7fe23006c8d0 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe22fe00640 1 -- 172.21.15.67:0/3982323397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228034610 con 0x7fe23006c8d0 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 -- 172.21.15.67:0/3982323397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe23006c8d0 msgr2=0x7fe23006cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe23006c8d0 0x7fe23006cd60 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fe228009e50 tx=0x7fe22802bf60 comp rx=0 tx=0).stop 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 -- 172.21.15.67:0/3982323397 shutdown_connections 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe23006c8d0 0x7fe23006cd60 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 --2- 172.21.15.67:0/3982323397 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 -- 172.21.15.67:0/3982323397 >> 172.21.15.67:0/3982323397 conn(0x7fe2300725d0 msgr2=0x7fe2300729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 -- 172.21.15.67:0/3982323397 shutdown_connections 2024-04-17T08:52:16.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.744+0000 7fe238000640 1 -- 172.21.15.67:0/3982323397 wait complete. 2024-04-17T08:52:16.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.746+0000 7fe238000640 1 Processor -- start 2024-04-17T08:52:16.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.746+0000 7fe238000640 1 -- start start 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 0x7fe2300ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe23006f4e0 con 0x7fe2300cb3f0 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe23015f040 con 0x7fe23006c8d0 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe238000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe23006f660 con 0x7fe23015ec60 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.747+0000 7fe234c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58020/0 (socket says 172.21.15.67:58020) 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 -- 172.21.15.67:0/398931523 learned_addr learned my addr 172.21.15.67:0/398931523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe235600640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 0x7fe2300ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe236000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 -- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 msgr2=0x7fe2300b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 -- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 msgr2=0x7fe2300ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 0x7fe2300ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 -- 172.21.15.67:0/398931523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2300b3e90 con 0x7fe2300cb3f0 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe235600640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 0x7fe2300ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.748+0000 7fe234c00640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c430 tx=0x7fe21c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.749+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0052d0 con 0x7fe2300cb3f0 2024-04-17T08:52:16.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.749+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe21c005490 con 0x7fe2300cb3f0 2024-04-17T08:52:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.749+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c007260 con 0x7fe2300cb3f0 2024-04-17T08:52:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.749+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2300b6600 con 0x7fe2300cb3f0 2024-04-17T08:52:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.749+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2300c8220 con 0x7fe2300cb3f0 2024-04-17T08:52:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.750+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2300c8040 con 0x7fe2300cb3f0 2024-04-17T08:52:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.751+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 145) ==== 90878+0+0 (secure 0 0 0) 0x7fe21c007400 con 0x7fe2300cb3f0 2024-04-17T08:52:16.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.752+0000 7fe22d600640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 0x7fe21006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:16.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.752+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fe21c08b480 con 0x7fe2300cb3f0 2024-04-17T08:52:16.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.752+0000 7fe235600640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 0x7fe21006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:16.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.754+0000 7fe235600640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 0x7fe21006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe224006fd0 tx=0x7fe224008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.756+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe2300c8040 con 0x7fe2300cb3f0 2024-04-17T08:52:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.952+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe230068ca0 con 0x7fe2300cb3f0 2024-04-17T08:52:16.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.952+0000 7fe22d600640 1 -- 172.21.15.67:0/398931523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v256) ==== 104+0+11 (secure 0 0 0) 0x7fe21c059440 con 0x7fe2300cb3f0 2024-04-17T08:52:16.951 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:52:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 msgr2=0x7fe21006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 0x7fe21006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe224006fd0 tx=0x7fe224008040 comp rx=0 tx=0).stop 2024-04-17T08:52:16.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 msgr2=0x7fe2300b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c430 tx=0x7fe21c00c8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 shutdown_connections 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe21006c740 0x7fe21006ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe23015ec60 0x7fe2300b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2300cb3f0 0x7fe2300b5a70 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 --2- 172.21.15.67:0/398931523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe23006c8d0 0x7fe2300ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 >> 172.21.15.67:0/398931523 conn(0x7fe2300725d0 msgr2=0x7fe230162e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 shutdown_connections 2024-04-17T08:52:16.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:16.955+0000 7fe238000640 1 -- 172.21.15.67:0/398931523 wait complete. 2024-04-17T08:52:16.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:52:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.201+0000 7ff1bea00640 1 Processor -- start 2024-04-17T08:52:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.201+0000 7ff1bea00640 1 -- start start 2024-04-17T08:52:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.203+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.203+0000 7ff1bea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 0x7ff1b815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.203+0000 7ff1bea00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b8160920 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.203+0000 7ff1bea00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b8160c00 con 0x7ff1b8146060 2024-04-17T08:52:17.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.203+0000 7ff1bea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b8160ee0 con 0x7ff1b8145b00 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47718/0 (socket says 172.21.15.67:47718) 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 -- 172.21.15.67:0/1817919585 learned_addr learned my addr 172.21.15.67:0/1817919585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bcc00640 1 --2- 172.21.15.67:0/1817919585 >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 0x7ff1b815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 -- 172.21.15.67:0/1817919585 >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 msgr2=0x7ff1b815cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 --2- 172.21.15.67:0/1817919585 >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 0x7ff1b815cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 -- 172.21.15.67:0/1817919585 >> v1:172.21.15.187:6789/0 conn(0x7ff1b80b4ac0 legacy=0x7ff1b80b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bd600640 1 -- 172.21.15.67:0/1817919585 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b81611c0 con 0x7ff1b8145b00 2024-04-17T08:52:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.204+0000 7ff1bcc00640 1 --2- 172.21.15.67:0/1817919585 >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 0x7ff1b815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:17.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.206+0000 7ff1bd600640 1 --2- 172.21.15.67:0/1817919585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009870 tx=0x7ff1ac02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55c8aab67b8f15c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1b7e00640 1 -- 172.21.15.67:0/1817919585 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac035020 con 0x7ff1b8145b00 2024-04-17T08:52:17.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1b7e00640 1 -- 172.21.15.67:0/1817919585 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac02d040 con 0x7ff1b8145b00 2024-04-17T08:52:17.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1b7e00640 1 -- 172.21.15.67:0/1817919585 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac031a80 con 0x7ff1b8145b00 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 -- 172.21.15.67:0/1817919585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 msgr2=0x7ff1b8160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 --2- 172.21.15.67:0/1817919585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009870 tx=0x7ff1ac02b900 comp rx=0 tx=0).stop 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 -- 172.21.15.67:0/1817919585 shutdown_connections 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 --2- 172.21.15.67:0/1817919585 >> v2:172.21.15.187:3300/0 conn(0x7ff1b8146060 0x7ff1b815cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 --2- 172.21.15.67:0/1817919585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8145b00 0x7ff1b8160340 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.207+0000 7ff1bea00640 1 -- 172.21.15.67:0/1817919585 >> 172.21.15.67:0/1817919585 conn(0x7ff1b80b23c0 msgr2=0x7ff1b80b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.208+0000 7ff1bea00640 1 -- 172.21.15.67:0/1817919585 shutdown_connections 2024-04-17T08:52:17.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.208+0000 7ff1bea00640 1 -- 172.21.15.67:0/1817919585 wait complete. 2024-04-17T08:52:17.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.209+0000 7ff1bea00640 1 Processor -- start 2024-04-17T08:52:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 -- start start 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b8145b00 0x7ff1b813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8146060 0x7ff1b813c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b815d2c0 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b80cfba0 con 0x7ff1b8145b00 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.210+0000 7ff1bea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b80b4f30 con 0x7ff1b8146060 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58026/0 (socket says 172.21.15.67:58026) 2024-04-17T08:52:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 -- 172.21.15.67:0/461610558 learned_addr learned my addr 172.21.15.67:0/461610558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1be000640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8146060 0x7ff1b813c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bcc00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b8145b00 0x7ff1b813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 -- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8146060 msgr2=0x7ff1b813c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8146060 0x7ff1b813c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 -- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b8145b00 msgr2=0x7ff1b813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b8145b00 0x7ff1b813adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.211+0000 7ff1bd600640 1 -- 172.21.15.67:0/461610558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b813b790 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1bd600640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac000c00 tx=0x7ff1ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac040070 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac039bf0 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac035040 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b813ba70 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.212+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b813c030 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.213+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b8068a10 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.214+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 145) ==== 90878+0+0 (secure 0 0 0) 0x7ff1ac04b060 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.215+0000 7ff1b5600640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 0x7ff19806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.215+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7ff1ac0adb10 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.216+0000 7ff1bcc00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 0x7ff19806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.217+0000 7ff1bcc00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 0x7ff19806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006fd0 tx=0x7ff1a4008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.220+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff1ac07bad0 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.430+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7ff1ac038020 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.433+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff1b8068610 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.434+0000 7ff1b5600640 1 -- 172.21.15.67:0/461610558 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v256) ==== 100+0+11 (secure 0 0 0) 0x7ff1ac0412b0 con 0x7ff1b80b4ac0 2024-04-17T08:52:17.432 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:52:17.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 msgr2=0x7ff19806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 0x7ff19806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4006fd0 tx=0x7ff1a4008040 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 msgr2=0x7ff1b815c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac000c00 tx=0x7ff1ac0046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 shutdown_connections 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff19806c740 0x7ff19806ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff1b8146060 0x7ff1b813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff1b8145b00 0x7ff1b813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 --2- 172.21.15.67:0/461610558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1b80b4ac0 0x7ff1b815c060 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 >> 172.21.15.67:0/461610558 conn(0x7ff1b80b23c0 msgr2=0x7ff1b80b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 shutdown_connections 2024-04-17T08:52:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.437+0000 7ff1bea00640 1 -- 172.21.15.67:0/461610558 wait complete. 2024-04-17T08:52:17.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:52:17.459 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:17.458+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:17.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.683+0000 7f023f000640 1 Processor -- start 2024-04-17T08:52:17.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.683+0000 7f023f000640 1 -- start start 2024-04-17T08:52:17.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.684+0000 7f023f000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0238145b00 0x7f0238160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023f000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0238160920 con 0x7f0238146060 2024-04-17T08:52:17.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023f000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0238160c00 con 0x7f02380b4ac0 2024-04-17T08:52:17.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023f000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0238160ee0 con 0x7f0238145b00 2024-04-17T08:52:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023d200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0238146060 0x7f023815cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46950/0 (socket says 172.21.15.67:46950) 2024-04-17T08:52:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023d200640 1 -- 172.21.15.67:0/2913686206 learned_addr learned my addr 172.21.15.67:0/2913686206 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.685+0000 7f023dc00640 1 --2- 172.21.15.67:0/2913686206 >> v2:172.21.15.187:3300/0 conn(0x7f0238145b00 0x7f0238160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601153544 0 0) 0x7f0238160920 con 0x7f0238146060 2024-04-17T08:52:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0220003520 con 0x7f0238146060 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2118474581 0 0) 0x7f0220003520 con 0x7f0238146060 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 >> v2:172.21.15.187:3300/0 conn(0x7f0238145b00 msgr2=0x7f0238160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 --2- 172.21.15.67:0/2913686206 >> v2:172.21.15.187:3300/0 conn(0x7f0238145b00 0x7f0238160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 >> v1:172.21.15.187:6789/0 conn(0x7f02380b4ac0 legacy=0x7f02380b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02381611c0 con 0x7f0238146060 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0224001670 con 0x7f0238146060 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 >> v1:172.21.15.67:6789/0 conn(0x7f0238146060 legacy=0x7f023815cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02200039b0 0x7f0220003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0220004800 0x7f0220004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f022000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0238160ee0 con 0x7f02200051c0 2024-04-17T08:52:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f022000c160 con 0x7f0220004800 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.686+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f022000c360 con 0x7f02200039b0 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f022000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023dc00640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02200039b0 0x7f0220003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023d200640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0220004800 0x7f0220004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 -- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02200039b0 msgr2=0x7f0220003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02200039b0 0x7f0220003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 -- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0220004800 msgr2=0x7f0220004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0220004800 0x7f0220004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023e600640 1 -- 172.21.15.67:0/2913686206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0220003520 con 0x7f02200051c0 2024-04-17T08:52:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.687+0000 7f023d200640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0220004800 0x7f0220004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.688+0000 7f023e600640 1 -- 172.21.15.67:0/2913686206 reap_dead start 2024-04-17T08:52:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.688+0000 7f023e600640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f022000b9c0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f0230001e50 tx=0x7f0220005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522b54d0eceed24c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.688+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230001a70 con 0x7f02200051c0 2024-04-17T08:52:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.688+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0230003bb0 con 0x7f02200051c0 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.688+0000 7f0237e00640 1 -- 172.21.15.67:0/2913686206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02300050d0 con 0x7f02200051c0 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 -- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 msgr2=0x7f022000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f022000b9c0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f0230001e50 tx=0x7f0220005830 comp rx=0 tx=0).stop 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 -- 172.21.15.67:0/2913686206 shutdown_connections 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 --2- 172.21.15.67:0/2913686206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f022000b9c0 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 -- 172.21.15.67:0/2913686206 >> 172.21.15.67:0/2913686206 conn(0x7f02380b23c0 msgr2=0x7f02380b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 -- 172.21.15.67:0/2913686206 shutdown_connections 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.689+0000 7f023f000640 1 -- 172.21.15.67:0/2913686206 wait complete. 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.691+0000 7f023f000640 1 Processor -- start 2024-04-17T08:52:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.691+0000 7f023f000640 1 -- start start 2024-04-17T08:52:17.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023813ac20 0x7f023813d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023813d850 0x7f023806f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0238160340 con 0x7f02200051c0 2024-04-17T08:52:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f02380cfba0 con 0x7f023813d850 2024-04-17T08:52:17.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.692+0000 7f023f000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02381604c0 con 0x7f023813ac20 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58044/0 (socket says 172.21.15.67:58044) 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 -- 172.21.15.67:0/847024892 learned_addr learned my addr 172.21.15.67:0/847024892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023e600640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023813d850 0x7f023806f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023d200640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023813ac20 0x7f023813d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 -- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023813ac20 msgr2=0x7f023813d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023813ac20 0x7f023813d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 -- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023813d850 msgr2=0x7f023806f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023813d850 0x7f023806f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 -- 172.21.15.67:0/847024892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023806e340 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.693+0000 7f023dc00640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f022c0108c0 tx=0x7f022c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.694+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c00f040 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.694+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022c00eb60 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.694+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023806e620 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.694+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c01c400 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.694+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023806ebe0 con 0x7f02200051c0 2024-04-17T08:52:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.695+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0238068a10 con 0x7f02200051c0 2024-04-17T08:52:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.696+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f022c01c5a0 con 0x7f02200051c0 2024-04-17T08:52:17.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.697+0000 7f0235600640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 0x7f021806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:17.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.697+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7f022c08cc30 con 0x7f02200051c0 2024-04-17T08:52:17.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.698+0000 7f023d200640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 0x7f021806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:17.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.699+0000 7f023d200640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 0x7f021806ef60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0224001920 tx=0x7f0224009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:17.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.702+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f022c018030 con 0x7f02200051c0 2024-04-17T08:52:17.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.911+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f0238068610 con 0x7f02200051c0 2024-04-17T08:52:17.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.912+0000 7f0235600640 1 -- 172.21.15.67:0/847024892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v256) ==== 107+0+11 (secure 0 0 0) 0x7f022c05a700 con 0x7f02200051c0 2024-04-17T08:52:17.910 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:52:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.914+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 msgr2=0x7f021806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.914+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 0x7f021806ef60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0224001920 tx=0x7f0224009040 comp rx=0 tx=0).stop 2024-04-17T08:52:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 msgr2=0x7f023815b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f022c0108c0 tx=0x7f022c010d80 comp rx=0 tx=0).stop 2024-04-17T08:52:17.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 shutdown_connections 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f021806caa0 0x7f021806ef60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f023813d850 0x7f023806f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f023813ac20 0x7f023813d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 --2- 172.21.15.67:0/847024892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02200051c0 0x7f023815b970 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 >> 172.21.15.67:0/847024892 conn(0x7f02380b23c0 msgr2=0x7f0238144570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 shutdown_connections 2024-04-17T08:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:17.915+0000 7f023f000640 1 -- 172.21.15.67:0/847024892 wait complete. 2024-04-17T08:52:17.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:52:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.160+0000 7fc770a00640 1 Processor -- start 2024-04-17T08:52:18.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.161+0000 7fc770a00640 1 -- start start 2024-04-17T08:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.162+0000 7fc770a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.162+0000 7fc770a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc76c15fd30 0x7fc76c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.162+0000 7fc770a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0c8110 con 0x7fc76c15ed10 2024-04-17T08:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.162+0000 7fc770a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c15f250 con 0x7fc76c15fd30 2024-04-17T08:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.162+0000 7fc770a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c15f530 con 0x7fc76c06b8c0 2024-04-17T08:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.163+0000 7fc76a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.163+0000 7fc76a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47748/0 (socket says 172.21.15.67:47748) 2024-04-17T08:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.163+0000 7fc76a800640 1 -- 172.21.15.67:0/2873148268 learned_addr learned my addr 172.21.15.67:0/2873148268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.163+0000 7fc769e00640 1 --2- 172.21.15.67:0/2873148268 >> v2:172.21.15.187:3300/0 conn(0x7fc76c15fd30 0x7fc76c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.164+0000 7fc76a800640 1 -- 172.21.15.67:0/2873148268 >> v2:172.21.15.187:3300/0 conn(0x7fc76c15fd30 msgr2=0x7fc76c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.164+0000 7fc76a800640 1 --2- 172.21.15.67:0/2873148268 >> v2:172.21.15.187:3300/0 conn(0x7fc76c15fd30 0x7fc76c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.164+0000 7fc76a800640 1 -- 172.21.15.67:0/2873148268 >> v1:172.21.15.187:6789/0 conn(0x7fc76c15ed10 legacy=0x7fc76c0cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.164+0000 7fc76a800640 1 -- 172.21.15.67:0/2873148268 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c15f810 con 0x7fc76c06b8c0 2024-04-17T08:52:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.166+0000 7fc76a800640 1 --2- 172.21.15.67:0/2873148268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009e50 tx=0x7fc75c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e407b609dc65baa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc769400640 1 -- 172.21.15.67:0/2873148268 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c02c9e0 con 0x7fc76c06b8c0 2024-04-17T08:52:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc769400640 1 -- 172.21.15.67:0/2873148268 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75c02cb80 con 0x7fc76c06b8c0 2024-04-17T08:52:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc769400640 1 -- 172.21.15.67:0/2873148268 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0343f0 con 0x7fc76c06b8c0 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 -- 172.21.15.67:0/2873148268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 msgr2=0x7fc76c0c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 --2- 172.21.15.67:0/2873148268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009e50 tx=0x7fc75c02be40 comp rx=0 tx=0).stop 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 -- 172.21.15.67:0/2873148268 shutdown_connections 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 --2- 172.21.15.67:0/2873148268 >> v2:172.21.15.187:3300/0 conn(0x7fc76c15fd30 0x7fc76c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 --2- 172.21.15.67:0/2873148268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c06b8c0 0x7fc76c0c79c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.167+0000 7fc770a00640 1 -- 172.21.15.67:0/2873148268 >> 172.21.15.67:0/2873148268 conn(0x7fc76c0b2fe0 msgr2=0x7fc76c0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.168+0000 7fc770a00640 1 -- 172.21.15.67:0/2873148268 shutdown_connections 2024-04-17T08:52:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.168+0000 7fc770a00640 1 -- 172.21.15.67:0/2873148268 wait complete. 2024-04-17T08:52:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.169+0000 7fc770a00640 1 Processor -- start 2024-04-17T08:52:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.169+0000 7fc770a00640 1 -- start start 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 0x7fc76c157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c15fd30 0x7fc76c0c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c06c360 con 0x7fc76c15ed10 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c06c1e0 con 0x7fc76c06b8c0 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.170+0000 7fc770a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c06c4e0 con 0x7fc76c15fd30 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58058/0 (socket says 172.21.15.67:58058) 2024-04-17T08:52:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 -- 172.21.15.67:0/3150368016 learned_addr learned my addr 172.21.15.67:0/3150368016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc76b200640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c15fd30 0x7fc76c0c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc76a800640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 0x7fc76c157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 -- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c15fd30 msgr2=0x7fc76c0c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c15fd30 0x7fc76c0c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 -- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 msgr2=0x7fc76c157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 0x7fc76c157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc769e00640 1 -- 172.21.15.67:0/3150368016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c1f54c0 con 0x7fc76c15ed10 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.171+0000 7fc76a800640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 0x7fc76c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc769e00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fc75800e8a0 tx=0x7fc75800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758017070 con 0x7fc76c15ed10 2024-04-17T08:52:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75800c940 con 0x7fc76c15ed10 2024-04-17T08:52:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75800cd30 con 0x7fc76c15ed10 2024-04-17T08:52:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c1f51e0 con 0x7fc76c15ed10 2024-04-17T08:52:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.172+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c1593a0 con 0x7fc76c15ed10 2024-04-17T08:52:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.173+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c0755d0 con 0x7fc76c15ed10 2024-04-17T08:52:18.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.174+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fc75801a070 con 0x7fc76c15ed10 2024-04-17T08:52:18.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.175+0000 7fc762a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 0x7fc74c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.176+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fc75808c4c0 con 0x7fc76c15ed10 2024-04-17T08:52:18.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.176+0000 7fc76a800640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 0x7fc74c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.177+0000 7fc76a800640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 0x7fc74c06ef60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc75c035000 tx=0x7fc75c030000 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.180+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc758059ef0 con 0x7fc76c15ed10 2024-04-17T08:52:18.371 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:18.371+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.391+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7fc76c158e90 con 0x7fc76c15ed10 2024-04-17T08:52:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.392+0000 7fc762a00640 1 -- 172.21.15.67:0/3150368016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v256) ==== 103+0+11 (secure 0 0 0) 0x7fc7580597f0 con 0x7fc76c15ed10 2024-04-17T08:52:18.390 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:52:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.394+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 msgr2=0x7fc74c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.394+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 0x7fc74c06ef60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc75c035000 tx=0x7fc75c030000 comp rx=0 tx=0).stop 2024-04-17T08:52:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.394+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 msgr2=0x7fc76c159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.394+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fc75800e8a0 tx=0x7fc75800ed60 comp rx=0 tx=0).stop 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 shutdown_connections 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74c06caa0 0x7fc74c06ef60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76c15fd30 0x7fc76c0c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76c15ed10 0x7fc76c159ba0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 --2- 172.21.15.67:0/3150368016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76c06b8c0 0x7fc76c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 >> 172.21.15.67:0/3150368016 conn(0x7fc76c0b2fe0 msgr2=0x7fc76c0b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 shutdown_connections 2024-04-17T08:52:18.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.395+0000 7fc770a00640 1 -- 172.21.15.67:0/3150368016 wait complete. 2024-04-17T08:52:18.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.634+0000 7fb692c00640 1 Processor -- start 2024-04-17T08:52:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.635+0000 7fb692c00640 1 -- start start 2024-04-17T08:52:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.636+0000 7fb692c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb68c161ec0 0x7fb68c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.636+0000 7fb692c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c0ae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb692c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c0d1310 con 0x7fb68c06b8c0 2024-04-17T08:52:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb692c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb68c06f060 con 0x7fb68c161ec0 2024-04-17T08:52:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb692c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb68c0d0d50 con 0x7fb68c0b0800 2024-04-17T08:52:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb690e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c0ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb692200640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb68c06b8c0 0x7fb68c06f820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:35628/0 (socket says 172.21.15.67:35628) 2024-04-17T08:52:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb692200640 1 -- 172.21.15.67:0/3950735997 learned_addr learned my addr 172.21.15.67:0/3950735997 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.637+0000 7fb691800640 1 --2- 172.21.15.67:0/3950735997 >> v2:172.21.15.187:3300/0 conn(0x7fb68c161ec0 0x7fb68c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.638+0000 7fb690e00640 1 -- 172.21.15.67:0/3950735997 >> v2:172.21.15.187:3300/0 conn(0x7fb68c161ec0 msgr2=0x7fb68c0d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.638+0000 7fb690e00640 1 --2- 172.21.15.67:0/3950735997 >> v2:172.21.15.187:3300/0 conn(0x7fb68c161ec0 0x7fb68c0d06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.638+0000 7fb690e00640 1 -- 172.21.15.67:0/3950735997 >> v1:172.21.15.187:6789/0 conn(0x7fb68c06b8c0 legacy=0x7fb68c06f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.638+0000 7fb690e00640 1 -- 172.21.15.67:0/3950735997 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c0d1030 con 0x7fb68c0b0800 2024-04-17T08:52:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.640+0000 7fb690e00640 1 --2- 172.21.15.67:0/3950735997 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c0ae330 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6780097a0 tx=0x7fb67802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3c545fa69f4e82 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.641+0000 7fb68be00640 1 -- 172.21.15.67:0/3950735997 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6780315a0 con 0x7fb68c0b0800 2024-04-17T08:52:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.641+0000 7fb68be00640 1 -- 172.21.15.67:0/3950735997 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb678037070 con 0x7fb68c0b0800 2024-04-17T08:52:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.641+0000 7fb68be00640 1 -- 172.21.15.67:0/3950735997 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb678031960 con 0x7fb68c0b0800 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.641+0000 7fb692c00640 1 -- 172.21.15.67:0/3950735997 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 msgr2=0x7fb68c0ae330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.641+0000 7fb692c00640 1 --2- 172.21.15.67:0/3950735997 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c0ae330 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6780097a0 tx=0x7fb67802b840 comp rx=0 tx=0).stop 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 -- 172.21.15.67:0/3950735997 shutdown_connections 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 --2- 172.21.15.67:0/3950735997 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c0ae330 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 --2- 172.21.15.67:0/3950735997 >> v2:172.21.15.187:3300/0 conn(0x7fb68c161ec0 0x7fb68c0d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 -- 172.21.15.67:0/3950735997 >> 172.21.15.67:0/3950735997 conn(0x7fb68c09aa60 msgr2=0x7fb68c0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 -- 172.21.15.67:0/3950735997 shutdown_connections 2024-04-17T08:52:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.642+0000 7fb692c00640 1 -- 172.21.15.67:0/3950735997 wait complete. 2024-04-17T08:52:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.644+0000 7fb692c00640 1 Processor -- start 2024-04-17T08:52:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.644+0000 7fb692c00640 1 -- start start 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb68c06b8c0 0x7fb68c159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c1f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c0ae870 con 0x7fb68c161ec0 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c1622a0 con 0x7fb68c06b8c0 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb68c0b0c60 con 0x7fb68c0b0800 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58068/0 (socket says 172.21.15.67:58068) 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb692200640 1 -- 172.21.15.67:0/4076524265 learned_addr learned my addr 172.21.15.67:0/4076524265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb690e00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c1f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.645+0000 7fb691800640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb68c06b8c0 0x7fb68c159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 -- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 msgr2=0x7fb68c1f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c1f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 -- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb68c06b8c0 msgr2=0x7fb68c159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb68c06b8c0 0x7fb68c159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 -- 172.21.15.67:0/4076524265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c1f4ef0 con 0x7fb68c161ec0 2024-04-17T08:52:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692200640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c3c0 tx=0x7fb68400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684017070 con 0x7fb68c161ec0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb684005ce0 con 0x7fb68c161ec0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6840072d0 con 0x7fb68c161ec0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68c1f51d0 con 0x7fb68c161ec0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.646+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb68c1ea660 con 0x7fb68c161ec0 2024-04-17T08:52:18.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.648+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb68c0755d0 con 0x7fb68c161ec0 2024-04-17T08:52:18.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.649+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fb68401a070 con 0x7fb68c161ec0 2024-04-17T08:52:18.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.650+0000 7fb689600640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 0x7fb66c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:18.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.650+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fb68408b810 con 0x7fb68c161ec0 2024-04-17T08:52:18.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.651+0000 7fb691800640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 0x7fb66c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:18.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.652+0000 7fb691800640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 0x7fb66c06ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb67c007380 tx=0x7fb67c00b040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:18.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.655+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb684058bf0 con 0x7fb68c161ec0 2024-04-17T08:52:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.908+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb68c0afeb0 con 0x7fb68c161ec0 2024-04-17T08:52:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.909+0000 7fb689600640 1 -- 172.21.15.67:0/4076524265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7fb68405cef0 con 0x7fb68c161ec0 2024-04-17T08:52:18.908 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:18.908 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:18.908 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 msgr2=0x7fb66c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 0x7fb66c06ef60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb67c007380 tx=0x7fb67c00b040 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 msgr2=0x7fb68c1f5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c3c0 tx=0x7fb68400c880 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 shutdown_connections 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fb66c06caa0 0x7fb66c06ef60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb68c161ec0 0x7fb68c1f5620 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb68c0b0800 0x7fb68c1f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 --2- 172.21.15.67:0/4076524265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb68c06b8c0 0x7fb68c159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 >> 172.21.15.67:0/4076524265 conn(0x7fb68c09aa60 msgr2=0x7fb68c06bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.912+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 shutdown_connections 2024-04-17T08:52:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:18.913+0000 7fb692c00640 1 -- 172.21.15.67:0/4076524265 wait complete. 2024-04-17T08:52:18.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:19.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.160+0000 7fd263200640 1 Processor -- start 2024-04-17T08:52:19.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.160+0000 7fd263200640 1 -- start start 2024-04-17T08:52:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.162+0000 7fd263200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd26415c5a0 0x7fd26415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.162+0000 7fd263200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2640b46c0 0x7fd2640ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.162+0000 7fd263200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26415f440 con 0x7fd2640b4210 2024-04-17T08:52:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.162+0000 7fd263200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd26415f720 con 0x7fd26415c5a0 2024-04-17T08:52:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.162+0000 7fd263200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd26415fa00 con 0x7fd2640b46c0 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd26415c5a0 0x7fd26415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd25be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2640b4210 0x7fd2640af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46960/0 (socket says 172.21.15.67:46960) 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd25be00640 1 -- 172.21.15.67:0/2265099472 learned_addr learned my addr 172.21.15.67:0/2265099472 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd262800640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2640b46c0 0x7fd2640ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261e00640 1 -- 172.21.15.67:0/2265099472 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2640b46c0 msgr2=0x7fd2640ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261e00640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2640b46c0 0x7fd2640ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261e00640 1 -- 172.21.15.67:0/2265099472 >> v1:172.21.15.67:6789/0 conn(0x7fd2640b4210 legacy=0x7fd2640af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261e00640 1 -- 172.21.15.67:0/2265099472 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26415fce0 con 0x7fd26415c5a0 2024-04-17T08:52:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.163+0000 7fd261400640 1 -- 172.21.15.67:0/2265099472 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420904013 0 0) 0x7fd26415f440 con 0x7fd2640b4210 2024-04-17T08:52:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.165+0000 7fd261e00640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26415c5a0 0x7fd26415ee60 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fd2540097a0 tx=0x7fd25402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc3672bf84aba45 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.165+0000 7fd261400640 1 -- 172.21.15.67:0/2265099472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254038070 con 0x7fd26415c5a0 2024-04-17T08:52:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.165+0000 7fd261400640 1 -- 172.21.15.67:0/2265099472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd254031700 con 0x7fd26415c5a0 2024-04-17T08:52:19.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.165+0000 7fd261400640 1 -- 172.21.15.67:0/2265099472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254031ad0 con 0x7fd26415c5a0 2024-04-17T08:52:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 -- 172.21.15.67:0/2265099472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26415c5a0 msgr2=0x7fd26415ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26415c5a0 0x7fd26415ee60 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fd2540097a0 tx=0x7fd25402b7a0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 -- 172.21.15.67:0/2265099472 shutdown_connections 2024-04-17T08:52:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2640b46c0 0x7fd2640ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 --2- 172.21.15.67:0/2265099472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26415c5a0 0x7fd26415ee60 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 -- 172.21.15.67:0/2265099472 >> 172.21.15.67:0/2265099472 conn(0x7fd264067d10 msgr2=0x7fd2640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 -- 172.21.15.67:0/2265099472 shutdown_connections 2024-04-17T08:52:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.166+0000 7fd263200640 1 -- 172.21.15.67:0/2265099472 wait complete. 2024-04-17T08:52:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.168+0000 7fd263200640 1 Processor -- start 2024-04-17T08:52:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.168+0000 7fd263200640 1 -- start start 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd26415c5a0 0x7fd26415a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd26415bb50 0x7fd2641e6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd26406e7f0 con 0x7fd2640b4210 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd26406e670 con 0x7fd26415bb50 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd263200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2640afb50 con 0x7fd26415c5a0 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd261e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd261e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58076/0 (socket says 172.21.15.67:58076) 2024-04-17T08:52:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.169+0000 7fd261e00640 1 -- 172.21.15.67:0/927353204 learned_addr learned my addr 172.21.15.67:0/927353204 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 -- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd26415c5a0 msgr2=0x7fd26415a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd26415c5a0 0x7fd26415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 -- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd26415bb50 msgr2=0x7fd2641e6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd26415bb50 0x7fd2641e6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 -- 172.21.15.67:0/927353204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26415aa50 con 0x7fd2640b4210 2024-04-17T08:52:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd261e00640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fd254000c00 tx=0x7fd2540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254042070 con 0x7fd2640b4210 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd254033a30 con 0x7fd2640b4210 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254038040 con 0x7fd2640b4210 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26415ad30 con 0x7fd2640b4210 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.170+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2641e4d70 con 0x7fd2640b4210 2024-04-17T08:52:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.172+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd264068a10 con 0x7fd2640b4210 2024-04-17T08:52:19.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.173+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fd254045070 con 0x7fd2640b4210 2024-04-17T08:52:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.174+0000 7fd25a000640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 0x7fd24006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.174+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fd2540ae1f0 con 0x7fd2640b4210 2024-04-17T08:52:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.174+0000 7fd25be00640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 0x7fd24006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.176+0000 7fd25be00640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 0x7fd24006ef90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007fd0 tx=0x7fd24c007920 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.179+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd254030070 con 0x7fd2640b4210 2024-04-17T08:52:19.292 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:19.291+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:19.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.396+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd26406cdb0 con 0x7fd2640b4210 2024-04-17T08:52:19.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.400+0000 7fd25a000640 1 -- 172.21.15.67:0/927353204 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) ==== 74+0+21613 (secure 0 0 0) 0x7fd25407bcd0 con 0x7fd2640b4210 2024-04-17T08:52:19.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:19.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":256,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:15.395631+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.403+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 msgr2=0x7fd24006ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.403+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 0x7fd24006ef90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007fd0 tx=0x7fd24c007920 comp rx=0 tx=0).stop 2024-04-17T08:52:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.403+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 msgr2=0x7fd264142f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.403+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7fd254000c00 tx=0x7fd2540046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 shutdown_connections 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd24006cad0 0x7fd24006ef90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd26415bb50 0x7fd2641e6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd26415c5a0 0x7fd26415a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 --2- 172.21.15.67:0/927353204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2640b4210 0x7fd264142f30 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 >> 172.21.15.67:0/927353204 conn(0x7fd264067d10 msgr2=0x7fd2640adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 shutdown_connections 2024-04-17T08:52:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.404+0000 7fd263200640 1 -- 172.21.15.67:0/927353204 wait complete. 2024-04-17T08:52:19.424 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:52:19.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.647+0000 7f211a800640 1 Processor -- start 2024-04-17T08:52:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.647+0000 7f211a800640 1 -- start start 2024-04-17T08:52:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f211415ec60 0x7f21140b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21140b6b30 0x7f21140cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 0x7f211406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21140d49c0 con 0x7f211415ec60 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f21140d3660 con 0x7f21140cf910 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f211a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21140b6f70 con 0x7f21140b6b30 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f2119400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f211415ec60 0x7f21140b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f2119400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f211415ec60 0x7f21140b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58086/0 (socket says 172.21.15.67:58086) 2024-04-17T08:52:19.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.649+0000 7f2119400640 1 -- 172.21.15.67:0/122647496 learned_addr learned my addr 172.21.15.67:0/122647496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2118a00640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21140b6b30 0x7f21140cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119e00640 1 --2- 172.21.15.67:0/122647496 >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 0x7f211406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119400640 1 -- 172.21.15.67:0/122647496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21140b6b30 msgr2=0x7f21140cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119400640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21140b6b30 0x7f21140cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119400640 1 -- 172.21.15.67:0/122647496 >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 msgr2=0x7f211406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119400640 1 --2- 172.21.15.67:0/122647496 >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 0x7f211406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119400640 1 -- 172.21.15.67:0/122647496 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21140d2d30 con 0x7f211415ec60 2024-04-17T08:52:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.650+0000 7f2119e00640 1 --2- 172.21.15.67:0/122647496 >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 0x7f211406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:19.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.651+0000 7f2119400640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f211415ec60 0x7f21140b65f0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f21140c8670 tx=0x7f210402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f729819baf5a76d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f2113e00640 1 -- 172.21.15.67:0/122647496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104035020 con 0x7f211415ec60 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f2113e00640 1 -- 172.21.15.67:0/122647496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210402d040 con 0x7f211415ec60 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f2113e00640 1 -- 172.21.15.67:0/122647496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104031a80 con 0x7f211415ec60 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 -- 172.21.15.67:0/122647496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f211415ec60 msgr2=0x7f21140b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f211415ec60 0x7f21140b65f0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f21140c8670 tx=0x7f210402b900 comp rx=0 tx=0).stop 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 -- 172.21.15.67:0/122647496 shutdown_connections 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 --2- 172.21.15.67:0/122647496 >> v2:172.21.15.187:3300/0 conn(0x7f21140cf910 0x7f211406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21140b6b30 0x7f21140cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 --2- 172.21.15.67:0/122647496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f211415ec60 0x7f21140b65f0 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 -- 172.21.15.67:0/122647496 >> 172.21.15.67:0/122647496 conn(0x7f21140d15f0 msgr2=0x7f21140d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.652+0000 7f211a800640 1 -- 172.21.15.67:0/122647496 shutdown_connections 2024-04-17T08:52:19.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.653+0000 7f211a800640 1 -- 172.21.15.67:0/122647496 wait complete. 2024-04-17T08:52:19.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.654+0000 7f211a800640 1 Processor -- start 2024-04-17T08:52:19.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.654+0000 7f211a800640 1 -- start start 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21140cf910 0x7f21141f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f211415ec60 0x7f211415d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21140cfda0 con 0x7f21140b6b30 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f21140d32f0 con 0x7f21140cf910 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.655+0000 7f211a800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f211415f040 con 0x7f211415ec60 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58092/0 (socket says 172.21.15.67:58092) 2024-04-17T08:52:19.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 -- 172.21.15.67:0/2333612154 learned_addr learned my addr 172.21.15.67:0/2333612154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:19.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 -- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f211415ec60 msgr2=0x7f211415d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f211415ec60 0x7f211415d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 -- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21140cf910 msgr2=0x7f21141f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21140cf910 0x7f21141f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 -- 172.21.15.67:0/2333612154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21141f4860 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.656+0000 7f2119400640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f210402b940 tx=0x7f21040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.657+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104035040 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.657+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2104039c10 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.657+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104045480 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.657+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21141f4b40 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.657+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21140cce10 con 0x7f21140b6b30 2024-04-17T08:52:19.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.658+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21141f5060 con 0x7f21140b6b30 2024-04-17T08:52:19.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.659+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f2104038020 con 0x7f21140b6b30 2024-04-17T08:52:19.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.661+0000 7f2111600640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 0x7f20f806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:19.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.661+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7f21040ae770 con 0x7f21140b6b30 2024-04-17T08:52:19.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.662+0000 7f2118a00640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 0x7f20f806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:19.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.663+0000 7f2118a00640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 0x7f20f806ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2108009540 tx=0x7f2108009450 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:19.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.666+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f21141f5060 con 0x7f21140b6b30 2024-04-17T08:52:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.920+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2114068a10 con 0x7f21140b6b30 2024-04-17T08:52:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.922+0000 7f2111600640 1 -- 172.21.15.67:0/2333612154 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7f210407c1a0 con 0x7f21140b6b30 2024-04-17T08:52:19.921 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:19.921 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:19.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 msgr2=0x7f20f806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 0x7f20f806ef60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2108009540 tx=0x7f2108009450 comp rx=0 tx=0).stop 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 msgr2=0x7f21140ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f210402b940 tx=0x7f21040046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 shutdown_connections 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.924+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f20f806caa0 0x7f20f806ef60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f211415ec60 0x7f211415d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21140cf910 0x7f21141f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 --2- 172.21.15.67:0/2333612154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21140b6b30 0x7f21140ce720 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 >> 172.21.15.67:0/2333612154 conn(0x7f21140d15f0 msgr2=0x7f21140b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 shutdown_connections 2024-04-17T08:52:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:19.925+0000 7f211a800640 1 -- 172.21.15.67:0/2333612154 wait complete. 2024-04-17T08:52:19.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:20.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.167+0000 7fa3b9600640 1 Processor -- start 2024-04-17T08:52:20.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.167+0000 7fa3b9600640 1 -- start start 2024-04-17T08:52:20.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.168+0000 7fa3b9600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 0x7fa3b406f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b9600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b415bc80 con 0x7fa3b406fce0 2024-04-17T08:52:20.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b9600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b415bf40 con 0x7fa3b415b570 2024-04-17T08:52:20.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b415c220 con 0x7fa3b406c040 2024-04-17T08:52:20.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b8c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47804/0 (socket says 172.21.15.67:47804) 2024-04-17T08:52:20.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b8c00640 1 -- 172.21.15.67:0/3111162982 learned_addr learned my addr 172.21.15.67:0/3111162982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:20.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.169+0000 7fa3b3e00640 1 --2- 172.21.15.67:0/3111162982 >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 0x7fa3b406f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.170+0000 7fa3b8c00640 1 -- 172.21.15.67:0/3111162982 >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 msgr2=0x7fa3b406f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.170+0000 7fa3b8c00640 1 --2- 172.21.15.67:0/3111162982 >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 0x7fa3b406f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.170+0000 7fa3b8c00640 1 -- 172.21.15.67:0/3111162982 >> v1:172.21.15.187:6789/0 conn(0x7fa3b406fce0 legacy=0x7fa3b406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.170+0000 7fa3b8c00640 1 -- 172.21.15.67:0/3111162982 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b415c500 con 0x7fa3b406c040 2024-04-17T08:52:20.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.170+0000 7fa3b3e00640 1 --2- 172.21.15.67:0/3111162982 >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 0x7fa3b406f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:52:20.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.172+0000 7fa3b8c00640 1 --2- 172.21.15.67:0/3111162982 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac009e50 tx=0x7fa3ac02e0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422cbff4872ae4a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.172+0000 7fa3b3400640 1 -- 172.21.15.67:0/3111162982 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac02ec90 con 0x7fa3b406c040 2024-04-17T08:52:20.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.172+0000 7fa3b3400640 1 -- 172.21.15.67:0/3111162982 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3ac02ee30 con 0x7fa3b406c040 2024-04-17T08:52:20.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.172+0000 7fa3b3400640 1 -- 172.21.15.67:0/3111162982 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0365f0 con 0x7fa3b406c040 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 -- 172.21.15.67:0/3111162982 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 msgr2=0x7fa3b415f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 --2- 172.21.15.67:0/3111162982 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac009e50 tx=0x7fa3ac02e0f0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 -- 172.21.15.67:0/3111162982 shutdown_connections 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 --2- 172.21.15.67:0/3111162982 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b406c040 0x7fa3b415f140 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 --2- 172.21.15.67:0/3111162982 >> v2:172.21.15.187:3300/0 conn(0x7fa3b415b570 0x7fa3b406f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 -- 172.21.15.67:0/3111162982 >> 172.21.15.67:0/3111162982 conn(0x7fa3b4067d10 msgr2=0x7fa3b40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 -- 172.21.15.67:0/3111162982 shutdown_connections 2024-04-17T08:52:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.173+0000 7fa3b9600640 1 -- 172.21.15.67:0/3111162982 wait complete. 2024-04-17T08:52:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.175+0000 7fa3b9600640 1 Processor -- start 2024-04-17T08:52:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.175+0000 7fa3b9600640 1 -- start start 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b406c040 0x7fa3b41f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b415b570 0x7fa3b40b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b40b4930 con 0x7fa3b406fce0 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b40b47b0 con 0x7fa3b406c040 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.176+0000 7fa3b9600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b415f680 con 0x7fa3b415b570 2024-04-17T08:52:20.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58108/0 (socket says 172.21.15.67:58108) 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 -- 172.21.15.67:0/2943186010 learned_addr learned my addr 172.21.15.67:0/2943186010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3b3e00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b406c040 0x7fa3b41f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3b8c00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b415b570 0x7fa3b40b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 -- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b415b570 msgr2=0x7fa3b40b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b415b570 0x7fa3b40b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 -- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b406c040 msgr2=0x7fa3b41f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b406c040 0x7fa3b41f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.177+0000 7fa3abe00640 1 -- 172.21.15.67:0/2943186010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b41f4640 con 0x7fa3b406fce0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3abe00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c400 tx=0x7fa39c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0052d0 con 0x7fa3b406fce0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39c005490 con 0x7fa3b406fce0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c007260 con 0x7fa3b406fce0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b4159820 con 0x7fa3b406fce0 2024-04-17T08:52:20.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.178+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b4159de0 con 0x7fa3b406fce0 2024-04-17T08:52:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.180+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b4068a10 con 0x7fa3b406fce0 2024-04-17T08:52:20.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.180+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fa39c007400 con 0x7fa3b406fce0 2024-04-17T08:52:20.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.181+0000 7fa3b0c00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 0x7fa39006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.181+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fa39c08ba40 con 0x7fa3b406fce0 2024-04-17T08:52:20.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.182+0000 7fa3b3e00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 0x7fa39006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.183+0000 7fa3b3e00640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 0x7fa39006ef60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009560 tx=0x7fa3a4009190 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.187+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa39c058ec0 con 0x7fa3b406fce0 2024-04-17T08:52:20.212 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:20.212+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.444+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3b406f350 con 0x7fa3b406fce0 2024-04-17T08:52:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.445+0000 7fa3b0c00640 1 -- 172.21.15.67:0/2943186010 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6564 (secure 0 0 0) 0x7fa39c05d1c0 con 0x7fa3b406fce0 2024-04-17T08:52:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:20.445 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":132,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:40.927618+0000","modified":"2024-04-17T08:51:44.165413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11844":{"gid":11844,"name":"a","rank":0,"incarnation":122,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3084367922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3084367922},{"type":"v1","addr":"172.21.15.67:6835","nonce":3084367922}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11844,"qdb_cluster":[11844]},"id":15},{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:20.445 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 132 2024-04-17T08:52:20.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 msgr2=0x7fa39006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 0x7fa39006ef60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009560 tx=0x7fa3a4009190 comp rx=0 tx=0).stop 2024-04-17T08:52:20.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 msgr2=0x7fa3b40b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fa39c00c400 tx=0x7fa39c00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 shutdown_connections 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa39006caa0 0x7fa39006ef60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3b415b570 0x7fa3b40b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3b406fce0 0x7fa3b40b0980 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 --2- 172.21.15.67:0/2943186010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3b406c040 0x7fa3b41f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 >> 172.21.15.67:0/2943186010 conn(0x7fa3b4067d10 msgr2=0x7fa3b4071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.448+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 shutdown_connections 2024-04-17T08:52:20.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.449+0000 7fa3b9600640 1 -- 172.21.15.67:0/2943186010 wait complete. 2024-04-17T08:52:20.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.694+0000 7fc1d0a00640 1 Processor -- start 2024-04-17T08:52:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.694+0000 7fc1d0a00640 1 -- start start 2024-04-17T08:52:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.696+0000 7fc1d0a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.696+0000 7fc1d0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1cc09b170 0x7fc1cc0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.696+0000 7fc1d0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1cc15ce00 con 0x7fc1cc0d0720 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.696+0000 7fc1d0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc1cc15d090 con 0x7fc1cc09b170 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.696+0000 7fc1d0a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc15d330 con 0x7fc1cc15ca20 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1cc09b170 0x7fc1cc0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1cc09b170 0x7fc1cc0cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58122/0 (socket says 172.21.15.67:58122) 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 -- 172.21.15.67:0/2611779741 learned_addr learned my addr 172.21.15.67:0/2611779741 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1caa00640 1 --1- 172.21.15.67:0/2611779741 >> v1:172.21.15.67:6789/0 conn(0x7fc1cc0d0720 0x7fc1cc09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46964/0 (socket says 172.21.15.67:46964) 2024-04-17T08:52:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cb400640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 -- 172.21.15.67:0/2611779741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 msgr2=0x7fc1cc0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 -- 172.21.15.67:0/2611779741 >> v1:172.21.15.67:6789/0 conn(0x7fc1cc0d0720 legacy=0x7fc1cc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1cbe00640 1 -- 172.21.15.67:0/2611779741 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc15d610 con 0x7fc1cc09b170 2024-04-17T08:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.697+0000 7fc1ca000640 1 -- 172.21.15.67:0/2611779741 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687541474 0 0) 0x7fc1cc15ce00 con 0x7fc1cc0d0720 2024-04-17T08:52:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.699+0000 7fc1cbe00640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc0cd190 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009e80 tx=0x7fc1c402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1242762ac9ae7dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.699+0000 7fc1ca000640 1 -- 172.21.15.67:0/2611779741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c402cae0 con 0x7fc1cc09b170 2024-04-17T08:52:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.699+0000 7fc1ca000640 1 -- 172.21.15.67:0/2611779741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c402cc80 con 0x7fc1cc09b170 2024-04-17T08:52:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.699+0000 7fc1ca000640 1 -- 172.21.15.67:0/2611779741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4034610 con 0x7fc1cc09b170 2024-04-17T08:52:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 -- 172.21.15.67:0/2611779741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 msgr2=0x7fc1cc0cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc0cd190 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009e80 tx=0x7fc1c402bf40 comp rx=0 tx=0).stop 2024-04-17T08:52:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 -- 172.21.15.67:0/2611779741 shutdown_connections 2024-04-17T08:52:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc0cd190 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/2611779741 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 -- 172.21.15.67:0/2611779741 >> 172.21.15.67:0/2611779741 conn(0x7fc1cc0b4100 msgr2=0x7fc1cc0b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 -- 172.21.15.67:0/2611779741 shutdown_connections 2024-04-17T08:52:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.700+0000 7fc1d0a00640 1 -- 172.21.15.67:0/2611779741 wait complete. 2024-04-17T08:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.702+0000 7fc1d0a00640 1 Processor -- start 2024-04-17T08:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.702+0000 7fc1d0a00640 1 -- start start 2024-04-17T08:52:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1cc0d0720 0x7fc1cc0c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc09bea0 con 0x7fc1cc09b170 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc09bd20 con 0x7fc1cc0d0720 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.703+0000 7fc1d0a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc0cd6d0 con 0x7fc1cc15ca20 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58134/0 (socket says 172.21.15.67:58134) 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 -- 172.21.15.67:0/311465478 learned_addr learned my addr 172.21.15.67:0/311465478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 -- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 msgr2=0x7fc1cc0c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 -- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1cc0d0720 msgr2=0x7fc1cc0c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1cc0d0720 0x7fc1cc0c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.704+0000 7fc1cb400640 1 -- 172.21.15.67:0/311465478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc09ff90 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1cb400640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00d5f0 tx=0x7fc1bc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0125f0 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc019070 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0129b0 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc06e590 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.705+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc1e7420 con 0x7fc1cc09b170 2024-04-17T08:52:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.706+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1cc0755d0 con 0x7fc1cc09b170 2024-04-17T08:52:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.707+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fc1bc012b50 con 0x7fc1cc09b170 2024-04-17T08:52:20.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.709+0000 7fc1c3400640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 0x7fc1ac06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:20.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.709+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7fc1bc08c920 con 0x7fc1cc09b170 2024-04-17T08:52:20.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.710+0000 7fc1caa00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 0x7fc1ac06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.711+0000 7fc1caa00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 0x7fc1ac06ef60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8007fd0 tx=0x7fc1b8007920 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:20.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.714+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc1bc05a480 con 0x7fc1cc09b170 2024-04-17T08:52:20.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.928+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1cc0b3600 con 0x7fc1cc09b170 2024-04-17T08:52:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.932+0000 7fc1c3400640 1 -- 172.21.15.67:0/311465478 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) ==== 74+0+21613 (secure 0 0 0) 0x7fc1bc059d80 con 0x7fc1cc09b170 2024-04-17T08:52:20.932 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:20.932 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":256,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:15.395631+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:51:38.821605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-04-17T08:51:39.950246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 msgr2=0x7fc1ac06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 0x7fc1ac06ef60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8007fd0 tx=0x7fc1b8007920 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 msgr2=0x7fc1cc09fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00d5f0 tx=0x7fc1bc00dab0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 shutdown_connections 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc1ac06caa0 0x7fc1ac06ef60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1cc15ca20 0x7fc1cc0c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1cc0d0720 0x7fc1cc0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 --2- 172.21.15.67:0/311465478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc09b170 0x7fc1cc09fa50 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 >> 172.21.15.67:0/311465478 conn(0x7fc1cc0b4100 msgr2=0x7fc1cc0cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 shutdown_connections 2024-04-17T08:52:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:20.935+0000 7fc1d0a00640 1 -- 172.21.15.67:0/311465478 wait complete. 2024-04-17T08:52:20.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:52:21.141 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:21.140+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:21.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.171+0000 7f484cc00640 1 Processor -- start 2024-04-17T08:52:21.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.171+0000 7f484cc00640 1 -- start start 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.172+0000 7f484cc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 0x7f48480d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.172+0000 7f484cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48480d18e0 0x7f48480d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.172+0000 7f484cc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48480d4140 con 0x7f48480d3350 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.172+0000 7f484cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f48480b83b0 con 0x7f48480d18e0 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.172+0000 7f484cc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f48480d1d20 con 0x7f484815ee20 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48480d18e0 0x7f48480d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48480d18e0 0x7f48480d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58144/0 (socket says 172.21.15.67:58144) 2024-04-17T08:52:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 -- 172.21.15.67:0/1779941123 learned_addr learned my addr 172.21.15.67:0/1779941123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4847400640 1 --2- 172.21.15.67:0/1779941123 >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 0x7f48480d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 -- 172.21.15.67:0/1779941123 >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 msgr2=0x7f48480d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 --2- 172.21.15.67:0/1779941123 >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 0x7f48480d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 -- 172.21.15.67:0/1779941123 >> v1:172.21.15.187:6789/0 conn(0x7f48480d3350 legacy=0x7f4848067600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4846a00640 1 -- 172.21.15.67:0/1779941123 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48480d3b80 con 0x7f48480d18e0 2024-04-17T08:52:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.173+0000 7f4847400640 1 --2- 172.21.15.67:0/1779941123 >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 0x7f48480d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f4846a00640 1 --2- 172.21.15.67:0/1779941123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d18e0 0x7f48480d2e10 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f48480d1370 tx=0x7f483002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6379a2c17f73beb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f4846000640 1 -- 172.21.15.67:0/1779941123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830035020 con 0x7f48480d18e0 2024-04-17T08:52:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f4846000640 1 -- 172.21.15.67:0/1779941123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4830031870 con 0x7f48480d18e0 2024-04-17T08:52:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f4846000640 1 -- 172.21.15.67:0/1779941123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830031c40 con 0x7f48480d18e0 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f484cc00640 1 -- 172.21.15.67:0/1779941123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d18e0 msgr2=0x7f48480d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.175+0000 7f484cc00640 1 --2- 172.21.15.67:0/1779941123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d18e0 0x7f48480d2e10 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f48480d1370 tx=0x7f483002b910 comp rx=0 tx=0).stop 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 -- 172.21.15.67:0/1779941123 shutdown_connections 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 --2- 172.21.15.67:0/1779941123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d18e0 0x7f48480d2e10 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 --2- 172.21.15.67:0/1779941123 >> v2:172.21.15.187:3300/0 conn(0x7f484815ee20 0x7f48480d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 -- 172.21.15.67:0/1779941123 >> 172.21.15.67:0/1779941123 conn(0x7f484806d3c0 msgr2=0x7f48480b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 -- 172.21.15.67:0/1779941123 shutdown_connections 2024-04-17T08:52:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.176+0000 7f484cc00640 1 -- 172.21.15.67:0/1779941123 wait complete. 2024-04-17T08:52:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.177+0000 7f484cc00640 1 Processor -- start 2024-04-17T08:52:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.178+0000 7f484cc00640 1 -- start start 2024-04-17T08:52:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.178+0000 7f484cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48480d18e0 0x7f4848151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f484cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f484cc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484815ee20 0x7f4848153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f484cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f484815f200 con 0x7f48480d3350 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f484cc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f48480b8040 con 0x7f484815ee20 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f484cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f48480d37c0 con 0x7f48480d18e0 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58148/0 (socket says 172.21.15.67:58148) 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 -- 172.21.15.67:0/2526624145 learned_addr learned my addr 172.21.15.67:0/2526624145 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4847e00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484815ee20 0x7f4848153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 -- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48480d18e0 msgr2=0x7f4848151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48480d18e0 0x7f4848151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 -- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484815ee20 msgr2=0x7f4848153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484815ee20 0x7f4848153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.179+0000 7f4846a00640 1 -- 172.21.15.67:0/2526624145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48481521d0 con 0x7f48480d3350 2024-04-17T08:52:21.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f4846a00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f483002b950 tx=0x7f48300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830040070 con 0x7f48480d3350 2024-04-17T08:52:21.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483003a420 con 0x7f48480d3350 2024-04-17T08:52:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830035040 con 0x7f48480d3350 2024-04-17T08:52:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4848154460 con 0x7f48480d3350 2024-04-17T08:52:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.180+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4848154a20 con 0x7f48480d3350 2024-04-17T08:52:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.181+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48480c9460 con 0x7f48480d3350 2024-04-17T08:52:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.182+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f4830038020 con 0x7f48480d3350 2024-04-17T08:52:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.183+0000 7f483f400640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 0x7f482406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.183+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 7461+0+0 (secure 0 0 0) 0x7f48300ae590 con 0x7f48480d3350 2024-04-17T08:52:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.184+0000 7f4847400640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 0x7f482406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.185+0000 7f4847400640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 0x7f482406ef60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4838008e00 tx=0x7f4838008330 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.188+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f483007c050 con 0x7f48480d3350 2024-04-17T08:52:21.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.449+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4848068a10 con 0x7f48480d3350 2024-04-17T08:52:21.449 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:52:21.450+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:52:21.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.454+0000 7f483f400640 1 -- 172.21.15.67:0/2526624145 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v133) ==== 161+0+0 (secure 0 0 0) 0x7f483007b950 con 0x7f48480d3350 2024-04-17T08:52:21.453 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:52:21.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 msgr2=0x7f482406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 0x7f482406ef60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4838008e00 tx=0x7f4838008330 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 msgr2=0x7f48481538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f483002b950 tx=0x7f48300046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 shutdown_connections 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f482406caa0 0x7f482406ef60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f484815ee20 0x7f4848153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48480d3350 0x7f48481538f0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 --2- 172.21.15.67:0/2526624145 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f48480d18e0 0x7f4848151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.457+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 >> 172.21.15.67:0/2526624145 conn(0x7f484806d3c0 msgr2=0x7f48480b90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:21.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.458+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 shutdown_connections 2024-04-17T08:52:21.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.459+0000 7f484cc00640 1 -- 172.21.15.67:0/2526624145 wait complete. 2024-04-17T08:52:21.473 INFO:tasks.ceph.mds.a.smithi067.stderr: -4794> 2024-04-17T08:51:47.429+0000 7f50fb000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:52:21.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:52:21.548 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:52:21.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.682+0000 7f6439400640 1 Processor -- start 2024-04-17T08:52:21.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.683+0000 7f6439400640 1 -- start start 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6439400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 0x7f64340b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6439400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64340ce4d0 0x7f64340b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6439400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6434071f60 con 0x7f643415b860 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6439400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6434071d10 con 0x7f64340ce4d0 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6439400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6434072240 con 0x7f64340d0b30 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6432800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f643415b860 0x7f64340d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46978/0 (socket says 172.21.15.67:46978) 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6432800640 1 -- 172.21.15.67:0/994557357 learned_addr learned my addr 172.21.15.67:0/994557357 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:21.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.684+0000 7f6433200640 1 --2- 172.21.15.67:0/994557357 >> v2:172.21.15.67:3300/0 conn(0x7f64340ce4d0 0x7f64340b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6431e00640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 0x7f64340b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6433200640 1 -- 172.21.15.67:0/994557357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 msgr2=0x7f64340b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6433200640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 0x7f64340b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6433200640 1 -- 172.21.15.67:0/994557357 >> v1:172.21.15.67:6789/0 conn(0x7f643415b860 legacy=0x7f64340d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6433200640 1 -- 172.21.15.67:0/994557357 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434072520 con 0x7f64340ce4d0 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6431e00640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 0x7f64340b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.685+0000 7f6431400640 1 -- 172.21.15.67:0/994557357 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1335246578 0 0) 0x7f6434071f60 con 0x7f643415b860 2024-04-17T08:52:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.686+0000 7f6433200640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340ce4d0 0x7f64340b44b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afbc20601107b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6431400640 1 -- 172.21.15.67:0/994557357 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c039070 con 0x7f64340ce4d0 2024-04-17T08:52:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6431400640 1 -- 172.21.15.67:0/994557357 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642c02cc50 con 0x7f64340ce4d0 2024-04-17T08:52:21.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6431400640 1 -- 172.21.15.67:0/994557357 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c034440 con 0x7f64340ce4d0 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 -- 172.21.15.67:0/994557357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340ce4d0 msgr2=0x7f64340b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340ce4d0 0x7f64340b44b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c02bea0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 -- 172.21.15.67:0/994557357 shutdown_connections 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340ce4d0 0x7f64340b44b0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 --2- 172.21.15.67:0/994557357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340d0b30 0x7f64340b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 -- 172.21.15.67:0/994557357 >> 172.21.15.67:0/994557357 conn(0x7f6434067ee0 msgr2=0x7f643406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.687+0000 7f6439400640 1 -- 172.21.15.67:0/994557357 shutdown_connections 2024-04-17T08:52:21.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.688+0000 7f6439400640 1 -- 172.21.15.67:0/994557357 wait complete. 2024-04-17T08:52:21.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.689+0000 7f6439400640 1 Processor -- start 2024-04-17T08:52:21.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.689+0000 7f6439400640 1 -- start start 2024-04-17T08:52:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.690+0000 7f6439400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340ce4d0 0x7f64341534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.690+0000 7f6439400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 0x7f64341f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.690+0000 7f6439400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6439400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f643415bc40 con 0x7f643415b860 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6439400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6434072ae0 con 0x7f64340d0b30 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6439400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f643406f380 con 0x7f64340ce4d0 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58188/0 (socket says 172.21.15.67:58188) 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 -- 172.21.15.67:0/3481870857 learned_addr learned my addr 172.21.15.67:0/3481870857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6432800640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340ce4d0 0x7f64341534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6431e00640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 0x7f64341f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 -- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340ce4d0 msgr2=0x7f64341534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340ce4d0 0x7f64341534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 -- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 msgr2=0x7f64341f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 0x7f64341f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6433200640 1 -- 172.21.15.67:0/3481870857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6434157500 con 0x7f643415b860 2024-04-17T08:52:21.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.691+0000 7f6431e00640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 0x7f64341f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f6433200640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c044020 con 0x7f643415b860 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642c03a040 con 0x7f643415b860 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c039050 con 0x7f643415b860 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64341577e0 con 0x7f643415b860 2024-04-17T08:52:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.692+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64341e4470 con 0x7f643415b860 2024-04-17T08:52:21.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.694+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6434068ca0 con 0x7f643415b860 2024-04-17T08:52:21.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.694+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f642c003990 con 0x7f643415b860 2024-04-17T08:52:21.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.695+0000 7f642aa00640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 0x7f641406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:21.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.696+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 1..257) ==== 7547+0+0 (secure 0 0 0) 0x7f642c0aea10 con 0x7f643415b860 2024-04-17T08:52:21.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.696+0000 7f6432800640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 0x7f641406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:21.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.697+0000 7f6432800640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 0x7f641406ef90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6424000f60 tx=0x7f6424007c00 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:21.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.700+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f642c04e060 con 0x7f643415b860 2024-04-17T08:52:21.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:21.954+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f64340b2d10 con 0x7f643415b860 2024-04-17T08:52:22.084 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:22.084+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:22.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.503+0000 7f642aa00640 1 -- 172.21.15.67:0/3481870857 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v134) ==== 104+0+0 (secure 0 0 0) 0x7f642c03d360 con 0x7f643415b860 2024-04-17T08:52:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 msgr2=0x7f641406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 0x7f641406ef90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6424000f60 tx=0x7f6424007c00 comp rx=0 tx=0).stop 2024-04-17T08:52:22.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 msgr2=0x7f64341e3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f642c009d90 tx=0x7f642c004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:22.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 shutdown_connections 2024-04-17T08:52:22.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f641406cad0 0x7f641406ef90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f643415b860 0x7f64341e3f30 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f64340d0b30 0x7f64341f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 --2- 172.21.15.67:0/3481870857 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64340ce4d0 0x7f64341534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.506+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 >> 172.21.15.67:0/3481870857 conn(0x7f6434067ee0 msgr2=0x7f643415a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.507+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 shutdown_connections 2024-04-17T08:52:22.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.507+0000 7f6439400640 1 -- 172.21.15.67:0/3481870857 wait complete. 2024-04-17T08:52:22.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:52:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.746+0000 7f80aea00640 1 Processor -- start 2024-04-17T08:52:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.746+0000 7f80aea00640 1 -- start start 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.747+0000 7f80aea00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f80a80cd6d0 0x7f80a8145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.747+0000 7f80aea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80a8146510 con 0x7f80a8146080 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.747+0000 7f80aea00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80a8161000 con 0x7f80a80cd2f0 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.747+0000 7f80aea00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f80a80cfa60 con 0x7f80a80cd6d0 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.748+0000 7f80ae000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80a8146080 0x7f80a8160360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46990/0 (socket says 172.21.15.67:46990) 2024-04-17T08:52:22.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.748+0000 7f80ae000640 1 -- 172.21.15.67:0/4225918501 learned_addr learned my addr 172.21.15.67:0/4225918501 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:22.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.748+0000 7f80acc00640 1 --2- 172.21.15.67:0/4225918501 >> v2:172.21.15.187:3300/0 conn(0x7f80a80cd6d0 0x7f80a8145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.748+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339031980 0 0) 0x7f80a8146510 con 0x7f80a8146080 2024-04-17T08:52:22.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.748+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8090003520 con 0x7f80a8146080 2024-04-17T08:52:22.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 213040830 0 0) 0x7f8090003520 con 0x7f80a8146080 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 >> v2:172.21.15.187:3300/0 conn(0x7f80a80cd6d0 msgr2=0x7f80a8145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 --2- 172.21.15.67:0/4225918501 >> v2:172.21.15.187:3300/0 conn(0x7f80a80cd6d0 0x7f80a8145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 >> v1:172.21.15.187:6789/0 conn(0x7f80a80cd2f0 legacy=0x7f80a80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a80ce870 con 0x7f80a8146080 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f80a0001670 con 0x7f80a8146080 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 >> v1:172.21.15.67:6789/0 conn(0x7f80a8146080 legacy=0x7f80a8160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8090003a30 0x7f8090003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8090004800 0x7f8090004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 0x7f809000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80a80cfa60 con 0x7f80900051c0 2024-04-17T08:52:22.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f809000c0f0 con 0x7f8090004800 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f809000c2f0 con 0x7f8090003a30 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.749+0000 7f80ae000640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 0x7f809000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ad600640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8090003a30 0x7f8090003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80acc00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8090004800 0x7f8090004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ae000640 1 -- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8090003a30 msgr2=0x7f8090003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ae000640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8090003a30 0x7f8090003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ae000640 1 -- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8090004800 msgr2=0x7f8090004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ae000640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8090004800 0x7f8090004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.750+0000 7f80ae000640 1 -- 172.21.15.67:0/4225918501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8090003520 con 0x7f80900051c0 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.751+0000 7f80ae000640 1 -- 172.21.15.67:0/4225918501 reap_dead start 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.751+0000 7f80ae000640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 0x7f809000b9c0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f80a000c0e0 tx=0x7f8090005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52adf7dbec209500 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.751+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0007c20 con 0x7f80900051c0 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.751+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a0030070 con 0x7f80900051c0 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.751+0000 7f80a7e00640 1 -- 172.21.15.67:0/4225918501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0005d30 con 0x7f80900051c0 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 -- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 msgr2=0x7f809000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 0x7f809000b9c0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f80a000c0e0 tx=0x7f8090005830 comp rx=0 tx=0).stop 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 -- 172.21.15.67:0/4225918501 shutdown_connections 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 --2- 172.21.15.67:0/4225918501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80900051c0 0x7f809000b9c0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 -- 172.21.15.67:0/4225918501 >> 172.21.15.67:0/4225918501 conn(0x7f80a806b8c0 msgr2=0x7f80a806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 -- 172.21.15.67:0/4225918501 shutdown_connections 2024-04-17T08:52:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.752+0000 7f80aea00640 1 -- 172.21.15.67:0/4225918501 wait complete. 2024-04-17T08:52:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.754+0000 7f80aea00640 1 Processor -- start 2024-04-17T08:52:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.754+0000 7f80aea00640 1 -- start start 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80900051c0 0x7f80a813a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a815bb50 0x7f80a813d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80a8145b40 con 0x7f80a813d620 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f80a80cee30 con 0x7f80a815bb50 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80aea00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f80a8145cc0 con 0x7f80900051c0 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58204/0 (socket says 172.21.15.67:58204) 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.755+0000 7f80ae000640 1 -- 172.21.15.67:0/2638869990 learned_addr learned my addr 172.21.15.67:0/2638869990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ad600640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80900051c0 0x7f80a813a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80acc00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a815bb50 0x7f80a813d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 -- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80900051c0 msgr2=0x7f80a813a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80900051c0 0x7f80a813a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 -- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a815bb50 msgr2=0x7f80a813d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a815bb50 0x7f80a813d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 -- 172.21.15.67:0/2638869990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a813aab0 con 0x7f80a813d620 2024-04-17T08:52:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.756+0000 7f80ae000640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009000 tx=0x7f80a80cea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:22.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.757+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a00047e0 con 0x7f80a813d620 2024-04-17T08:52:22.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.757+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a00049a0 con 0x7f80a813d620 2024-04-17T08:52:22.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.757+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0004d90 con 0x7f80a813d620 2024-04-17T08:52:22.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.757+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a813ad90 con 0x7f80a813d620 2024-04-17T08:52:22.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.757+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a813b350 con 0x7f80a813d620 2024-04-17T08:52:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.758+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a8068a10 con 0x7f80a813d620 2024-04-17T08:52:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.759+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f80a0007070 con 0x7f80a813d620 2024-04-17T08:52:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.760+0000 7f80a5600640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 0x7f808806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.760+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(257..257 src has 1..257) ==== 7547+0+0 (secure 0 0 0) 0x7f80a00ad0b0 con 0x7f80a813d620 2024-04-17T08:52:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.760+0000 7f80ad600640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 0x7f808806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.762+0000 7f80ad600640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 0x7f808806ef90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f809c007fd0 tx=0x7f809c009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.764+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f80a0047030 con 0x7f80a813d620 2024-04-17T08:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:22.970+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f80a8068610 con 0x7f80a813d620 2024-04-17T08:52:23.056 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:23.056+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:23.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.524+0000 7f80a5600640 1 -- 172.21.15.67:0/2638869990 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v258) ==== 181+0+0 (secure 0 0 0) 0x7f80a007aad0 con 0x7f80a813d620 2024-04-17T08:52:23.523 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.527+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 msgr2=0x7f808806ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.527+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 0x7f808806ef90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f809c007fd0 tx=0x7f809c009040 comp rx=0 tx=0).stop 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.527+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 msgr2=0x7f80a815d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.527+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009000 tx=0x7f80a80cea40 comp rx=0 tx=0).stop 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 shutdown_connections 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f808806cad0 0x7f808806ef90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80a813d620 0x7f80a815d030 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80a815bb50 0x7f80a813d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 --2- 172.21.15.67:0/2638869990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80900051c0 0x7f80a813a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.528+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 >> 172.21.15.67:0/2638869990 conn(0x7f80a806b8c0 msgr2=0x7f80a806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.530+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 shutdown_connections 2024-04-17T08:52:23.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.531+0000 7f80aea00640 1 -- 172.21.15.67:0/2638869990 wait complete. 2024-04-17T08:52:23.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:52:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.769+0000 7fe1b5600640 1 Processor -- start 2024-04-17T08:52:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.770+0000 7fe1b5600640 1 -- start start 2024-04-17T08:52:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe1b00cd2f0 0x7fe1b00cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1b00cd6d0 0x7fe1b0145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b0146080 0x7fe1b0160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b0146530 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b0161000 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1b5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b00ce870 con 0x7fe1b0146080 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.771+0000 7fe1af400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1b00cd6d0 0x7fe1b0145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1b00cd6d0 0x7fe1b0145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52094/0 (socket says 172.21.15.67:52094) 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 -- 172.21.15.67:0/1981563149 learned_addr learned my addr 172.21.15.67:0/1981563149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1afe00640 1 --2- 172.21.15.67:0/1981563149 >> v2:172.21.15.187:3300/0 conn(0x7fe1b00cd2f0 0x7fe1b00cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1b4c00640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b0146080 0x7fe1b0160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 -- 172.21.15.67:0/1981563149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b0146080 msgr2=0x7fe1b0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b0146080 0x7fe1b0160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 -- 172.21.15.67:0/1981563149 >> v2:172.21.15.187:3300/0 conn(0x7fe1b00cd2f0 msgr2=0x7fe1b00cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 --2- 172.21.15.67:0/1981563149 >> v2:172.21.15.187:3300/0 conn(0x7fe1b00cd2f0 0x7fe1b00cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.772+0000 7fe1af400640 1 -- 172.21.15.67:0/1981563149 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b00ceb50 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.773+0000 7fe1af400640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd6d0 0x7fe1b0145b40 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009870 tx=0x7fe19c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104c1b04a7d36f2b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1aea00640 1 -- 172.21.15.67:0/1981563149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c035020 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1aea00640 1 -- 172.21.15.67:0/1981563149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19c031640 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1aea00640 1 -- 172.21.15.67:0/1981563149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c02d040 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 -- 172.21.15.67:0/1981563149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd6d0 msgr2=0x7fe1b0145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd6d0 0x7fe1b0145b40 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009870 tx=0x7fe19c02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 -- 172.21.15.67:0/1981563149 shutdown_connections 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b0146080 0x7fe1b0160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 --2- 172.21.15.67:0/1981563149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd6d0 0x7fe1b0145b40 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 --2- 172.21.15.67:0/1981563149 >> v2:172.21.15.187:3300/0 conn(0x7fe1b00cd2f0 0x7fe1b00cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 -- 172.21.15.67:0/1981563149 >> 172.21.15.67:0/1981563149 conn(0x7fe1b006b8c0 msgr2=0x7fe1b006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 -- 172.21.15.67:0/1981563149 shutdown_connections 2024-04-17T08:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.774+0000 7fe1b5600640 1 -- 172.21.15.67:0/1981563149 wait complete. 2024-04-17T08:52:23.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.776+0000 7fe1b5600640 1 Processor -- start 2024-04-17T08:52:23.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.776+0000 7fe1b5600640 1 -- start start 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b00cd6d0 0x7fe1b01396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1b0146080 0x7fe1b0139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b00cfa10 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b00cf890 con 0x7fe1b0146080 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.777+0000 7fe1b5600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b01608a0 con 0x7fe1b00cd6d0 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52098/0 (socket says 172.21.15.67:52098) 2024-04-17T08:52:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 -- 172.21.15.67:0/2300975319 learned_addr learned my addr 172.21.15.67:0/2300975319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1af400640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b00cd6d0 0x7fe1b01396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 -- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b00cd6d0 msgr2=0x7fe1b01396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b00cd6d0 0x7fe1b01396a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 -- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1b0146080 msgr2=0x7fe1b0139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1b0146080 0x7fe1b0139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 -- 172.21.15.67:0/2300975319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b013b4e0 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.778+0000 7fe1afe00640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000d800 tx=0x7fe1a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.779+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a000b6c0 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.779+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a0019070 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.779+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a000ba80 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.779+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b01388e0 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.779+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b0138ea0 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.780+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b0068a10 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.781+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fe1a000bc20 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.783+0000 7fe1a7e00640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 0x7fe19006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.783+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 7109+0+0 (secure 0 0 0) 0x7fe1a008c950 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.783+0000 7fe1af400640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 0x7fe19006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.785+0000 7fe1af400640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 0x7fe19006ef60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe19c036000 tx=0x7fe19c008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:23.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.788+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1a005a4e0 con 0x7fe1b00cd2f0 2024-04-17T08:52:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:23.993+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe1b0068610 con 0x7fe1b00cd2f0 2024-04-17T08:52:24.050 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:24.050+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:24.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.528+0000 7fe1a7e00640 1 -- 172.21.15.67:0/2300975319 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v259) ==== 169+0+0 (secure 0 0 0) 0x7fe1a0059de0 con 0x7fe1b00cd2f0 2024-04-17T08:52:24.527 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:52:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.532+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 msgr2=0x7fe19006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.532+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 0x7fe19006ef60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe19c036000 tx=0x7fe19c008030 comp rx=0 tx=0).stop 2024-04-17T08:52:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.532+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 msgr2=0x7fe1b0138000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.532+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000d800 tx=0x7fe1a000dcc0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 shutdown_connections 2024-04-17T08:52:24.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe19006caa0 0x7fe19006ef60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1b0146080 0x7fe1b0139be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1b00cd6d0 0x7fe1b01396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 --2- 172.21.15.67:0/2300975319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b00cd2f0 0x7fe1b0138000 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.533+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 >> 172.21.15.67:0/2300975319 conn(0x7fe1b006b8c0 msgr2=0x7fe1b00b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.534+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 shutdown_connections 2024-04-17T08:52:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.534+0000 7fe1b5600640 1 -- 172.21.15.67:0/2300975319 wait complete. 2024-04-17T08:52:24.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.779+0000 7f7dd9a00640 1 Processor -- start 2024-04-17T08:52:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.780+0000 7f7dd9a00640 1 -- start start 2024-04-17T08:52:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.781+0000 7f7dd9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.781+0000 7f7dd9a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7dd40afb50 0x7f7dd40ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.781+0000 7f7dd9a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dd415f440 con 0x7f7dd40b4210 2024-04-17T08:52:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.781+0000 7f7dd9a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd415f720 con 0x7f7dd40afb50 2024-04-17T08:52:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.781+0000 7f7dd9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd415fa00 con 0x7f7dd415c5a0 2024-04-17T08:52:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.782+0000 7f7dd3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.782+0000 7f7dd3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:60676/0 (socket says 172.21.15.67:60676) 2024-04-17T08:52:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.782+0000 7f7dd3e00640 1 -- 172.21.15.67:0/3582757474 learned_addr learned my addr 172.21.15.67:0/3582757474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.782+0000 7f7dd9000640 1 --2- 172.21.15.67:0/3582757474 >> v2:172.21.15.187:3300/0 conn(0x7f7dd40afb50 0x7f7dd40ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.783+0000 7f7dd3e00640 1 -- 172.21.15.67:0/3582757474 >> v2:172.21.15.187:3300/0 conn(0x7f7dd40afb50 msgr2=0x7f7dd40ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.783+0000 7f7dd3e00640 1 --2- 172.21.15.67:0/3582757474 >> v2:172.21.15.187:3300/0 conn(0x7f7dd40afb50 0x7f7dd40ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.783+0000 7f7dd3e00640 1 -- 172.21.15.67:0/3582757474 >> v1:172.21.15.187:6789/0 conn(0x7f7dd40b4210 legacy=0x7f7dd40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.783+0000 7f7dd3e00640 1 -- 172.21.15.67:0/3582757474 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd415fce0 con 0x7f7dd415c5a0 2024-04-17T08:52:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.785+0000 7f7dd3e00640 1 --2- 172.21.15.67:0/3582757474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40098a0 tx=0x7f7dc402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98a4833c4edf47b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.785+0000 7f7dd2a00640 1 -- 172.21.15.67:0/3582757474 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40317c0 con 0x7f7dd415c5a0 2024-04-17T08:52:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd2a00640 1 -- 172.21.15.67:0/3582757474 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc4037070 con 0x7f7dd415c5a0 2024-04-17T08:52:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd2a00640 1 -- 172.21.15.67:0/3582757474 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4031b80 con 0x7f7dd415c5a0 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 -- 172.21.15.67:0/3582757474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 msgr2=0x7f7dd415ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/3582757474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40098a0 tx=0x7f7dc402b860 comp rx=0 tx=0).stop 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 -- 172.21.15.67:0/3582757474 shutdown_connections 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/3582757474 >> v2:172.21.15.187:3300/0 conn(0x7f7dd40afb50 0x7f7dd40ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/3582757474 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd415c5a0 0x7f7dd415ee60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 -- 172.21.15.67:0/3582757474 >> 172.21.15.67:0/3582757474 conn(0x7f7dd4067d10 msgr2=0x7f7dd40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 -- 172.21.15.67:0/3582757474 shutdown_connections 2024-04-17T08:52:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.786+0000 7f7dd9a00640 1 -- 172.21.15.67:0/3582757474 wait complete. 2024-04-17T08:52:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.788+0000 7f7dd9a00640 1 Processor -- start 2024-04-17T08:52:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.788+0000 7f7dd9a00640 1 -- start start 2024-04-17T08:52:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd40afb50 0x7f7dd4156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 0x7f7dd4157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd406e7f0 con 0x7f7dd415c5a0 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd406e670 con 0x7f7dd40b4210 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.789+0000 7f7dd9a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd40b4690 con 0x7f7dd40afb50 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52102/0 (socket says 172.21.15.67:52102) 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 -- 172.21.15.67:0/1027879723 learned_addr learned my addr 172.21.15.67:0/1027879723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd3e00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd40afb50 0x7f7dd4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd3400640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 0x7f7dd4157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 -- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd40afb50 msgr2=0x7f7dd4156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd40afb50 0x7f7dd4156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 -- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 msgr2=0x7f7dd4157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 0x7f7dd4157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd9000640 1 -- 172.21.15.67:0/1027879723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd4157890 con 0x7f7dd415c5a0 2024-04-17T08:52:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.790+0000 7f7dd3400640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 0x7f7dd4157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dd9000640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc004650 tx=0x7f7dcc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc017070 con 0x7f7dd415c5a0 2024-04-17T08:52:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dcc00c940 con 0x7f7dd415c5a0 2024-04-17T08:52:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd4157b30 con 0x7f7dd415c5a0 2024-04-17T08:52:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc00cd30 con 0x7f7dd415c5a0 2024-04-17T08:52:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.791+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd41583d0 con 0x7f7dd415c5a0 2024-04-17T08:52:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.793+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd4068a10 con 0x7f7dd415c5a0 2024-04-17T08:52:24.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.794+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f7dcc01a070 con 0x7f7dd415c5a0 2024-04-17T08:52:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.794+0000 7f7dcbe00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 0x7f7db006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.795+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f7dcc08c350 con 0x7f7dd415c5a0 2024-04-17T08:52:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.795+0000 7f7dd3e00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 0x7f7db006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.796+0000 7f7dd3e00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 0x7f7db006ef60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4036000 tx=0x7f7dc4008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:24.799+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7dcc05a000 con 0x7f7dd415c5a0 2024-04-17T08:52:25.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.011+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7dd40b1b50 con 0x7f7dd415c5a0 2024-04-17T08:52:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.014+0000 7f7dcbe00640 1 -- 172.21.15.67:0/1027879723 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+18025 (secure 0 0 0) 0x7f7dcc059900 con 0x7f7dd415c5a0 2024-04-17T08:52:25.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:25.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":259,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:24.524735+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.67:6835/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.67:6834/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:25.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 msgr2=0x7f7db006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 0x7f7db006ef60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4036000 tx=0x7f7dc4008030 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 msgr2=0x7f7dd40aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc004650 tx=0x7f7dcc00ece0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 shutdown_connections 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7db006caa0 0x7f7db006ef60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd415c5a0 0x7f7dd40aeb80 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dd40b4210 0x7f7dd4157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 --2- 172.21.15.67:0/1027879723 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dd40afb50 0x7f7dd4156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 >> 172.21.15.67:0/1027879723 conn(0x7f7dd4067d10 msgr2=0x7f7dd415b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 shutdown_connections 2024-04-17T08:52:25.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.017+0000 7f7dd9a00640 1 -- 172.21.15.67:0/1027879723 wait complete. 2024-04-17T08:52:25.034 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:25.033+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:25.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-04-17T08:51:54.448741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-04-17T08:51:55.106540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:52:25.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:52:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.271+0000 7f1153c00640 1 Processor -- start 2024-04-17T08:52:25.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.272+0000 7f1153c00640 1 -- start start 2024-04-17T08:52:25.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.273+0000 7f1153c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f114c142e20 0x7f114c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.273+0000 7f1153c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.273+0000 7f1153c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f114c1607a0 con 0x7f114c143200 2024-04-17T08:52:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.273+0000 7f1153c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f114c160a60 con 0x7f114c142e20 2024-04-17T08:52:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.273+0000 7f1153c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f114c160d40 con 0x7f114c0cf160 2024-04-17T08:52:25.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.274+0000 7f1143e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.274+0000 7f1143e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:60690/0 (socket says 172.21.15.67:60690) 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.274+0000 7f1143e00640 1 -- 172.21.15.67:0/2832345863 learned_addr learned my addr 172.21.15.67:0/2832345863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.274+0000 7f1151200640 1 --2- 172.21.15.67:0/2832345863 >> v2:172.21.15.187:3300/0 conn(0x7f114c142e20 0x7f114c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.275+0000 7f1143e00640 1 -- 172.21.15.67:0/2832345863 >> v2:172.21.15.187:3300/0 conn(0x7f114c142e20 msgr2=0x7f114c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.275+0000 7f1143e00640 1 --2- 172.21.15.67:0/2832345863 >> v2:172.21.15.187:3300/0 conn(0x7f114c142e20 0x7f114c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.275+0000 7f1143e00640 1 -- 172.21.15.67:0/2832345863 >> v1:172.21.15.187:6789/0 conn(0x7f114c143200 legacy=0x7f114c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.275+0000 7f1143e00640 1 -- 172.21.15.67:0/2832345863 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c161020 con 0x7f114c0cf160 2024-04-17T08:52:25.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.276+0000 7f1143e00640 1 --2- 172.21.15.67:0/2832345863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1134009870 tx=0x7f113402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20d772f3be1a79df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.277+0000 7f114be00640 1 -- 172.21.15.67:0/2832345863 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134035020 con 0x7f114c0cf160 2024-04-17T08:52:25.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.277+0000 7f114be00640 1 -- 172.21.15.67:0/2832345863 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1134031640 con 0x7f114c0cf160 2024-04-17T08:52:25.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.277+0000 7f114be00640 1 -- 172.21.15.67:0/2832345863 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113402d040 con 0x7f114c0cf160 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 -- 172.21.15.67:0/2832345863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 msgr2=0x7f114c0cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 --2- 172.21.15.67:0/2832345863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1134009870 tx=0x7f113402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 -- 172.21.15.67:0/2832345863 shutdown_connections 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 --2- 172.21.15.67:0/2832345863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c0cf160 0x7f114c0cd040 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 --2- 172.21.15.67:0/2832345863 >> v2:172.21.15.187:3300/0 conn(0x7f114c142e20 0x7f114c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 -- 172.21.15.67:0/2832345863 >> 172.21.15.67:0/2832345863 conn(0x7f114c06b8c0 msgr2=0x7f114c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 -- 172.21.15.67:0/2832345863 shutdown_connections 2024-04-17T08:52:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.278+0000 7f1153c00640 1 -- 172.21.15.67:0/2832345863 wait complete. 2024-04-17T08:52:25.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.280+0000 7f1153c00640 1 Processor -- start 2024-04-17T08:52:25.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.280+0000 7f1153c00640 1 -- start start 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f114c0cf160 0x7f114c0ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c143200 0x7f114c0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f114c0b18f0 con 0x7f114c142e20 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f114c0ce730 con 0x7f114c0cf160 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1153c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f114c0b1a70 con 0x7f114c143200 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1143e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1143e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52114/0 (socket says 172.21.15.67:52114) 2024-04-17T08:52:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.281+0000 7f1143e00640 1 -- 172.21.15.67:0/2342147001 learned_addr learned my addr 172.21.15.67:0/2342147001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 -- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c143200 msgr2=0x7f114c0aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c143200 0x7f114c0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 -- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f114c0cf160 msgr2=0x7f114c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f114c0cf160 0x7f114c0ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 -- 172.21.15.67:0/2342147001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114c0af2e0 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.282+0000 7f1143e00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f1134000c00 tx=0x7f11340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.283+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134040070 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.283+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11340399f0 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.283+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134035040 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.283+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114c0af5c0 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.283+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f114c0afe60 con 0x7f114c142e20 2024-04-17T08:52:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.284+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f114c068b90 con 0x7f114c142e20 2024-04-17T08:52:25.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.285+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f113404b060 con 0x7f114c142e20 2024-04-17T08:52:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.286+0000 7f1149600640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 0x7f112806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.286+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f11340ada90 con 0x7f114c142e20 2024-04-17T08:52:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.287+0000 7f1151200640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 0x7f112806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.288+0000 7f1151200640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 0x7f112806ef60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f113c006a90 tx=0x7f113c006a20 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.291+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f113407b8f0 con 0x7f114c142e20 2024-04-17T08:52:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.508+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f114c0b16d0 con 0x7f114c142e20 2024-04-17T08:52:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.508+0000 7f1149600640 1 -- 172.21.15.67:0/2342147001 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v259) ==== 93+0+10 (secure 0 0 0) 0x7f113407b1f0 con 0x7f114c142e20 2024-04-17T08:52:25.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:52:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 msgr2=0x7f112806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 0x7f112806ef60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f113c006a90 tx=0x7f113c006a20 comp rx=0 tx=0).stop 2024-04-17T08:52:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 msgr2=0x7f114c0ae670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f1134000c00 tx=0x7f11340046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 shutdown_connections 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f112806caa0 0x7f112806ef60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f114c143200 0x7f114c0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f114c142e20 0x7f114c0ae670 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 --2- 172.21.15.67:0/2342147001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f114c0cf160 0x7f114c0ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 >> 172.21.15.67:0/2342147001 conn(0x7f114c06b8c0 msgr2=0x7f114c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 shutdown_connections 2024-04-17T08:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.511+0000 7f1153c00640 1 -- 172.21.15.67:0/2342147001 wait complete. 2024-04-17T08:52:25.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-04-17T08:52:25.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.766+0000 7f37e7000640 1 Processor -- start 2024-04-17T08:52:25.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.766+0000 7f37e7000640 1 -- start start 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e7000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37e00cd0c0 0x7f37e0163360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e7000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e0160100 con 0x7f37e006b8c0 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e7000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f37e006c260 con 0x7f37e00cd0c0 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f37e015fe20 con 0x7f37e015f870 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37e00cd0c0 0x7f37e0163360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e5200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f37e006b8c0 0x7f37e015f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48054/0 (socket says 172.21.15.67:48054) 2024-04-17T08:52:25.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.768+0000 7f37e5200640 1 -- 172.21.15.67:0/938632124 learned_addr learned my addr 172.21.15.67:0/938632124 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e6600640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e5c00640 1 -- 172.21.15.67:0/938632124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 msgr2=0x7f37e006bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e5c00640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e006bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e5c00640 1 -- 172.21.15.67:0/938632124 >> v1:172.21.15.67:6789/0 conn(0x7f37e006b8c0 legacy=0x7f37e015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e5c00640 1 -- 172.21.15.67:0/938632124 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e0162880 con 0x7f37e00cd0c0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37dfe00640 1 -- 172.21.15.67:0/938632124 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040657102 0 0) 0x7f37e0160100 con 0x7f37e006b8c0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.769+0000 7f37e6600640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.770+0000 7f37e5c00640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e0163360 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f37d40097a0 tx=0x7f37d402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae2c9dfb2e0620 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37dfe00640 1 -- 172.21.15.67:0/938632124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4031780 con 0x7f37e00cd0c0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37dfe00640 1 -- 172.21.15.67:0/938632124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d4037070 con 0x7f37e00cd0c0 2024-04-17T08:52:25.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37dfe00640 1 -- 172.21.15.67:0/938632124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4031b40 con 0x7f37e00cd0c0 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 -- 172.21.15.67:0/938632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 msgr2=0x7f37e0163360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e0163360 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f37d40097a0 tx=0x7f37d402b820 comp rx=0 tx=0).stop 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 -- 172.21.15.67:0/938632124 shutdown_connections 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 --2- 172.21.15.67:0/938632124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e0163360 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 -- 172.21.15.67:0/938632124 >> 172.21.15.67:0/938632124 conn(0x7f37e009aa60 msgr2=0x7f37e00b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 -- 172.21.15.67:0/938632124 shutdown_connections 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.771+0000 7f37e7000640 1 -- 172.21.15.67:0/938632124 wait complete. 2024-04-17T08:52:25.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.773+0000 7f37e7000640 1 Processor -- start 2024-04-17T08:52:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.773+0000 7f37e7000640 1 -- start start 2024-04-17T08:52:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 0x7f37e015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e00b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f37e00cd4a0 con 0x7f37e00cd0c0 2024-04-17T08:52:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f37e0162e40 con 0x7f37e006b8c0 2024-04-17T08:52:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f37e00cd620 con 0x7f37e015f870 2024-04-17T08:52:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.774+0000 7f37e5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52134/0 (socket says 172.21.15.67:52134) 2024-04-17T08:52:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 -- 172.21.15.67:0/3698016368 learned_addr learned my addr 172.21.15.67:0/3698016368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e6600640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e00b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5c00640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 0x7f37e015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 -- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 msgr2=0x7f37e00b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e00b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 -- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 msgr2=0x7f37e015c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 0x7f37e015c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 -- 172.21.15.67:0/3698016368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e00b6c10 con 0x7f37e00cd0c0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e6600640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e00b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5c00640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 0x7f37e015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.775+0000 7f37e5200640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c400 tx=0x7f37cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.776+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc017070 con 0x7f37e00cd0c0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.776+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37cc005ce0 con 0x7f37e00cd0c0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.776+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37cc007310 con 0x7f37e00cd0c0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.776+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e00b6ef0 con 0x7f37e00cd0c0 2024-04-17T08:52:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.776+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e00c5230 con 0x7f37e00cd0c0 2024-04-17T08:52:25.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.777+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e00755d0 con 0x7f37e00cd0c0 2024-04-17T08:52:25.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.778+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f37cc01a070 con 0x7f37e00cd0c0 2024-04-17T08:52:25.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.779+0000 7f37dd600640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 0x7f37c006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:25.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.779+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f37cc08b650 con 0x7f37e00cd0c0 2024-04-17T08:52:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.780+0000 7f37e5c00640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 0x7f37c006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.781+0000 7f37e5c00640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 0x7f37c006ef60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37d4036000 tx=0x7f37d4008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:25.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:25.784+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f37cc059410 con 0x7f37e00cd0c0 2024-04-17T08:52:26.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.005+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f37e00cecb0 con 0x7f37e00cd0c0 2024-04-17T08:52:26.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.006+0000 7f37dd600640 1 -- 172.21.15.67:0/3698016368 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v259) ==== 107+0+11 (secure 0 0 0) 0x7f37cc058d10 con 0x7f37e00cd0c0 2024-04-17T08:52:26.006 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 msgr2=0x7f37c006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 0x7f37c006ef60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37d4036000 tx=0x7f37d4008030 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 msgr2=0x7f37e00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f37cc00c400 tx=0x7f37cc00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 shutdown_connections 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f37c006caa0 0x7f37c006ef60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f37e015f870 0x7f37e00b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e00cd0c0 0x7f37e00b60a0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 --2- 172.21.15.67:0/3698016368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37e006b8c0 0x7f37e015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.009+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 >> 172.21.15.67:0/3698016368 conn(0x7f37e009aa60 msgr2=0x7f37e00cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.010+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 shutdown_connections 2024-04-17T08:52:26.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.010+0000 7f37e7000640 1 -- 172.21.15.67:0/3698016368 wait complete. 2024-04-17T08:52:26.013 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:26.012+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:26.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-04-17T08:52:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.251+0000 7f3905400640 1 Processor -- start 2024-04-17T08:52:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.251+0000 7f3905400640 1 -- start start 2024-04-17T08:52:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3905400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3900159020 0x7f390015ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3905400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 0x7f3900162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3905400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390015f780 con 0x7f390006b8c0 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3905400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f390015fa60 con 0x7f3900159020 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3905400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f39001624a0 con 0x7f39000af400 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f38ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3900159020 0x7f390015ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f38ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3900159020 0x7f390015ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52144/0 (socket says 172.21.15.67:52144) 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f3904a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f390006b8c0 0x7f39000adb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48070/0 (socket says 172.21.15.67:48070) 2024-04-17T08:52:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.253+0000 7f38ffe00640 1 -- 172.21.15.67:0/2271917097 learned_addr learned my addr 172.21.15.67:0/2271917097 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ff400640 1 --2- 172.21.15.67:0/2271917097 >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 0x7f3900162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ffe00640 1 -- 172.21.15.67:0/2271917097 >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 msgr2=0x7f3900162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ffe00640 1 --2- 172.21.15.67:0/2271917097 >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 0x7f3900162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ffe00640 1 -- 172.21.15.67:0/2271917097 >> v1:172.21.15.67:6789/0 conn(0x7f390006b8c0 legacy=0x7f39000adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ffe00640 1 -- 172.21.15.67:0/2271917097 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3900162780 con 0x7f3900159020 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.254+0000 7f38ff400640 1 --2- 172.21.15.67:0/2271917097 >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 0x7f3900162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.255+0000 7f38ffe00640 1 --2- 172.21.15.67:0/2271917097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3900159020 0x7f390015ee30 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f39000b58a0 tx=0x7f38f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b217a3cf947fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.255+0000 7f38fea00640 1 -- 172.21.15.67:0/2271917097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0035020 con 0x7f3900159020 2024-04-17T08:52:26.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.255+0000 7f38fea00640 1 -- 172.21.15.67:0/2271917097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f0031870 con 0x7f3900159020 2024-04-17T08:52:26.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f38fea00640 1 -- 172.21.15.67:0/2271917097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0031c40 con 0x7f3900159020 2024-04-17T08:52:26.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 -- 172.21.15.67:0/2271917097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3900159020 msgr2=0x7f390015ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 --2- 172.21.15.67:0/2271917097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3900159020 0x7f390015ee30 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f39000b58a0 tx=0x7f38f002b910 comp rx=0 tx=0).stop 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 -- 172.21.15.67:0/2271917097 shutdown_connections 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 --2- 172.21.15.67:0/2271917097 >> v2:172.21.15.187:3300/0 conn(0x7f39000af400 0x7f3900162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 --2- 172.21.15.67:0/2271917097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3900159020 0x7f390015ee30 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 -- 172.21.15.67:0/2271917097 >> 172.21.15.67:0/2271917097 conn(0x7f39000b2190 msgr2=0x7f39000b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 -- 172.21.15.67:0/2271917097 shutdown_connections 2024-04-17T08:52:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.256+0000 7f3905400640 1 -- 172.21.15.67:0/2271917097 wait complete. 2024-04-17T08:52:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.258+0000 7f3905400640 1 Processor -- start 2024-04-17T08:52:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.258+0000 7f3905400640 1 -- start start 2024-04-17T08:52:26.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 0x7f390006f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3900159020 0x7f3900159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f390015f4f0 con 0x7f390006b8c0 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f390015f370 con 0x7f39000af400 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f3905400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39001631e0 con 0x7f3900159020 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.259+0000 7f38ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52160/0 (socket says 172.21.15.67:52160) 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 -- 172.21.15.67:0/3843678553 learned_addr learned my addr 172.21.15.67:0/3843678553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ff400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 0x7f390006f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 -- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3900159020 msgr2=0x7f3900159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f3904a00640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3900159020 0x7f3900159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3900159020 0x7f3900159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 -- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 msgr2=0x7f390006f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 0x7f390006f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ffe00640 1 -- 172.21.15.67:0/3843678553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39001613c0 con 0x7f390006b8c0 2024-04-17T08:52:26.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.260+0000 7f38ff400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 0x7f390006f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:26.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f38ffe00640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f38f002b950 tx=0x7f38f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0040070 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f003a420 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3900159950 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0035040 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.261+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3900161940 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.262+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3900068a10 con 0x7f390006b8c0 2024-04-17T08:52:26.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.263+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f38f004b060 con 0x7f390006b8c0 2024-04-17T08:52:26.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.264+0000 7f38f7e00640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 0x7f38dc06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.265+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f38f00ae2e0 con 0x7f390006b8c0 2024-04-17T08:52:26.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.265+0000 7f38ff400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 0x7f38dc06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.266+0000 7f38ff400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 0x7f38dc06ef60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38e8006fd0 tx=0x7f38e8008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.269+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f38f007c0b0 con 0x7f390006b8c0 2024-04-17T08:52:26.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.477+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f390006c7b0 con 0x7f390006b8c0 2024-04-17T08:52:26.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.478+0000 7f38f7e00640 1 -- 172.21.15.67:0/3843678553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v259) ==== 103+0+11 (secure 0 0 0) 0x7f38f007b9b0 con 0x7f390006b8c0 2024-04-17T08:52:26.477 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.480+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 msgr2=0x7f38dc06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.480+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 0x7f38dc06ef60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38e8006fd0 tx=0x7f38e8008040 comp rx=0 tx=0).stop 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 msgr2=0x7f39000ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f38f002b950 tx=0x7f38f00046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 shutdown_connections 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38dc06caa0 0x7f38dc06ef60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3900159020 0x7f3900159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39000af400 0x7f390006f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 --2- 172.21.15.67:0/3843678553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f390006b8c0 0x7f39000ced70 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 >> 172.21.15.67:0/3843678553 conn(0x7f39000b2190 msgr2=0x7f3900071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 shutdown_connections 2024-04-17T08:52:26.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.481+0000 7f3905400640 1 -- 172.21.15.67:0/3843678553 wait complete. 2024-04-17T08:52:26.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.731+0000 7f618d800640 1 Processor -- start 2024-04-17T08:52:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.731+0000 7f618d800640 1 -- start start 2024-04-17T08:52:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f618806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61880cd020 0x7f61880cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6188071d10 0x7f61880721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f618806ff90 con 0x7f61880cd020 2024-04-17T08:52:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f61880ceb00 con 0x7f6188071d10 2024-04-17T08:52:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.732+0000 7f618d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61880cede0 con 0x7f618806f670 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61880cd020 0x7f61880cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f618806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61880cd020 0x7f61880cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52168/0 (socket says 172.21.15.67:52168) 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 -- 172.21.15.67:0/2020080851 learned_addr learned my addr 172.21.15.67:0/2020080851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6187400640 1 --2- 172.21.15.67:0/2020080851 >> v2:172.21.15.187:3300/0 conn(0x7f6188071d10 0x7f61880721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 -- 172.21.15.67:0/2020080851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 msgr2=0x7f618806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 --2- 172.21.15.67:0/2020080851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f618806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 -- 172.21.15.67:0/2020080851 >> v2:172.21.15.187:3300/0 conn(0x7f6188071d10 msgr2=0x7f61880721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 --2- 172.21.15.67:0/2020080851 >> v2:172.21.15.187:3300/0 conn(0x7f6188071d10 0x7f61880721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.733+0000 7f6186000640 1 -- 172.21.15.67:0/2020080851 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61880cf0c0 con 0x7f61880cd020 2024-04-17T08:52:26.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.735+0000 7f6186000640 1 --2- 172.21.15.67:0/2020080851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cd020 0x7f61880cd480 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f61880cdf80 tx=0x7f617402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad660c30d7bf5838 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.735+0000 7f6185600640 1 -- 172.21.15.67:0/2020080851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174035020 con 0x7f61880cd020 2024-04-17T08:52:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f6185600640 1 -- 172.21.15.67:0/2020080851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174031870 con 0x7f61880cd020 2024-04-17T08:52:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f6185600640 1 -- 172.21.15.67:0/2020080851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174031c40 con 0x7f61880cd020 2024-04-17T08:52:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 -- 172.21.15.67:0/2020080851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cd020 msgr2=0x7f61880cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 --2- 172.21.15.67:0/2020080851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cd020 0x7f61880cd480 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f61880cdf80 tx=0x7f617402b910 comp rx=0 tx=0).stop 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 -- 172.21.15.67:0/2020080851 shutdown_connections 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 --2- 172.21.15.67:0/2020080851 >> v2:172.21.15.187:3300/0 conn(0x7f6188071d10 0x7f61880721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 --2- 172.21.15.67:0/2020080851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cd020 0x7f61880cd480 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 --2- 172.21.15.67:0/2020080851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f618806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 -- 172.21.15.67:0/2020080851 >> 172.21.15.67:0/2020080851 conn(0x7f618806bf50 msgr2=0x7f61880b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 -- 172.21.15.67:0/2020080851 shutdown_connections 2024-04-17T08:52:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.736+0000 7f618d800640 1 -- 172.21.15.67:0/2020080851 wait complete. 2024-04-17T08:52:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.738+0000 7f618d800640 1 Processor -- start 2024-04-17T08:52:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.738+0000 7f618d800640 1 -- start start 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f61880c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61880cd020 0x7f618814f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61880ce720 con 0x7f6188071d10 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61880cf680 con 0x7f61880cd020 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f618d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61880ce8a0 con 0x7f618806f670 2024-04-17T08:52:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.739+0000 7f6186000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52174/0 (socket says 172.21.15.67:52174) 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 -- 172.21.15.67:0/2031073028 learned_addr learned my addr 172.21.15.67:0/2031073028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6187400640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61880cd020 0x7f618814f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186a00640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f61880c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 -- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 msgr2=0x7f61880c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f61880c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 -- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61880cd020 msgr2=0x7f618814f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61880cd020 0x7f618814f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:26.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.740+0000 7f6186000640 1 -- 172.21.15.67:0/2031073028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61880c77a0 con 0x7f6188071d10 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f6186000640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f617402b950 tx=0x7f61740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174040070 con 0x7f6188071d10 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174030d00 con 0x7f6188071d10 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174035040 con 0x7f6188071d10 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61880c7a80 con 0x7f6188071d10 2024-04-17T08:52:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.741+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61880c8040 con 0x7f6188071d10 2024-04-17T08:52:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.742+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61880c7530 con 0x7f6188071d10 2024-04-17T08:52:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.743+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f6174038020 con 0x7f6188071d10 2024-04-17T08:52:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.745+0000 7f617ea00640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 0x7f616806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.745+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f61740ae0c0 con 0x7f6188071d10 2024-04-17T08:52:26.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.746+0000 7f6186a00640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 0x7f616806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:26.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.747+0000 7f6186a00640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 0x7f616806ef60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61780099b0 tx=0x7f6178008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:26.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:26.749+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f617407bdf0 con 0x7f6188071d10 2024-04-17T08:52:26.986 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:26.985+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.007+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f618806ccd0 con 0x7f6188071d10 2024-04-17T08:52:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.008+0000 7f617ea00640 1 -- 172.21.15.67:0/2031073028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5221 (secure 0 0 0) 0x7f617407b6f0 con 0x7f6188071d10 2024-04-17T08:52:27.008 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:27.008 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":12360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/991489299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":991489299},{"type":"v1","addr":"172.21.15.67:6835","nonce":991489299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 134 2024-04-17T08:52:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.011+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 msgr2=0x7f616806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.011+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 0x7f616806ef60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61780099b0 tx=0x7f6178008040 comp rx=0 tx=0).stop 2024-04-17T08:52:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.011+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 msgr2=0x7f618814eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.011+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7f617402b950 tx=0x7f61740046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 shutdown_connections 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f616806caa0 0x7f616806ef60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61880cd020 0x7f618814f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188071d10 0x7f618814eb60 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 --2- 172.21.15.67:0/2031073028 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f618806f670 0x7f61880c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 >> 172.21.15.67:0/2031073028 conn(0x7f618806bf50 msgr2=0x7f618806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 shutdown_connections 2024-04-17T08:52:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.012+0000 7f618d800640 1 -- 172.21.15.67:0/2031073028 wait complete. 2024-04-17T08:52:27.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:27.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.258+0000 7f6171a00640 1 Processor -- start 2024-04-17T08:52:27.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.258+0000 7f6171a00640 1 -- start start 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.259+0000 7f6171a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.259+0000 7f6171a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c06f030 0x7f616c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f6171a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f616c06ff90 con 0x7f616c0cd020 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f6171a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f616c0ceb00 con 0x7f616c06f030 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f6171a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f616c0cede0 con 0x7f616c06f670 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c06f030 0x7f616c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c06f030 0x7f616c0cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52178/0 (socket says 172.21.15.67:52178) 2024-04-17T08:52:27.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 -- 172.21.15.67:0/1829940081 learned_addr learned my addr 172.21.15.67:0/1829940081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616aa00640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 -- 172.21.15.67:0/1829940081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 msgr2=0x7f616c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 -- 172.21.15.67:0/1829940081 >> v1:172.21.15.187:6789/0 conn(0x7f616c0cd020 legacy=0x7f616c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.260+0000 7f616b400640 1 -- 172.21.15.67:0/1829940081 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616c0cf0c0 con 0x7f616c06f030 2024-04-17T08:52:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.262+0000 7f616b400640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c06f030 0x7f616c0cd480 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f6164009e50 tx=0x7f616402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a447e04fc4923b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.262+0000 7f6169600640 1 -- 172.21.15.67:0/1829940081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616402ca70 con 0x7f616c06f030 2024-04-17T08:52:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.262+0000 7f6169600640 1 -- 172.21.15.67:0/1829940081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616402cc10 con 0x7f616c06f030 2024-04-17T08:52:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.262+0000 7f6169600640 1 -- 172.21.15.67:0/1829940081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61640343f0 con 0x7f616c06f030 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 -- 172.21.15.67:0/1829940081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c06f030 msgr2=0x7f616c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c06f030 0x7f616c0cd480 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f6164009e50 tx=0x7f616402bed0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 -- 172.21.15.67:0/1829940081 shutdown_connections 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c06f030 0x7f616c0cd480 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 --2- 172.21.15.67:0/1829940081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 -- 172.21.15.67:0/1829940081 >> 172.21.15.67:0/1829940081 conn(0x7f616c06bf50 msgr2=0x7f616c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 -- 172.21.15.67:0/1829940081 shutdown_connections 2024-04-17T08:52:27.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.263+0000 7f6171a00640 1 -- 172.21.15.67:0/1829940081 wait complete. 2024-04-17T08:52:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.264+0000 7f6171a00640 1 Processor -- start 2024-04-17T08:52:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.265+0000 7f6171a00640 1 -- start start 2024-04-17T08:52:27.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f616c06f030 0x7f616c1560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c14cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f616c0ce720 con 0x7f616c0cd020 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f616c0cf680 con 0x7f616c06f030 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f6171a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f616c0ce8a0 con 0x7f616c06f670 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52184/0 (socket says 172.21.15.67:52184) 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 -- 172.21.15.67:0/4274350516 learned_addr learned my addr 172.21.15.67:0/4274350516 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616aa00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f616c06f030 0x7f616c1560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 -- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 msgr2=0x7f616c14cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616a000640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c14cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c14cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 -- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f616c06f030 msgr2=0x7f616c1560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f616c06f030 0x7f616c1560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616b400640 1 -- 172.21.15.67:0/4274350516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616c156630 con 0x7f616c0cd020 2024-04-17T08:52:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.266+0000 7f616a000640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c14cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f616b400640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f6164036040 tx=0x7f6164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164043020 con 0x7f616c0cd020 2024-04-17T08:52:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616403e450 con 0x7f616c0cd020 2024-04-17T08:52:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164009510 con 0x7f616c0cd020 2024-04-17T08:52:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616c155050 con 0x7f616c0cd020 2024-04-17T08:52:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.267+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616c155610 con 0x7f616c0cd020 2024-04-17T08:52:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.269+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616c068b90 con 0x7f616c0cd020 2024-04-17T08:52:27.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.270+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f6164046020 con 0x7f616c0cd020 2024-04-17T08:52:27.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.271+0000 7f6162a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 0x7f614c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.271+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f61640aebd0 con 0x7f616c0cd020 2024-04-17T08:52:27.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.271+0000 7f616aa00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 0x7f614c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.273+0000 7f616aa00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 0x7f614c06ef60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f615c006fd0 tx=0x7f615c008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.275+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f616404d030 con 0x7f616c0cd020 2024-04-17T08:52:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.492+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f616c06ccd0 con 0x7f616c0cd020 2024-04-17T08:52:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.495+0000 7f6162a00640 1 -- 172.21.15.67:0/4274350516 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+18025 (secure 0 0 0) 0x7f616407c870 con 0x7f616c0cd020 2024-04-17T08:52:27.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:27.495 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":259,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:24.524735+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.67:6835/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.67:6834/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 msgr2=0x7f614c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 0x7f614c06ef60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f615c006fd0 tx=0x7f615c008040 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 msgr2=0x7f616c1568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f6164036040 tx=0x7f6164004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 shutdown_connections 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f614c06caa0 0x7f614c06ef60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c0cd020 0x7f616c1568b0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f616c06f670 0x7f616c14cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 --2- 172.21.15.67:0/4274350516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f616c06f030 0x7f616c1560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 >> 172.21.15.67:0/4274350516 conn(0x7f616c06bf50 msgr2=0x7f616c0c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 shutdown_connections 2024-04-17T08:52:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.498+0000 7f6171a00640 1 -- 172.21.15.67:0/4274350516 wait complete. 2024-04-17T08:52:27.519 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-04-17T08:52:27.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.736+0000 7fc38de00640 1 Processor -- start 2024-04-17T08:52:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.736+0000 7fc38de00640 1 -- start start 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.737+0000 7fc38de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc3880b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38de00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc38806bfd0 0x7fc3880b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3880d0890 con 0x7fc3880b7080 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38de00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc38815f0b0 con 0x7fc38806bfd0 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3880d0640 con 0x7fc38815ecd0 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc387e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3880b7080 0x7fc38806b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48084/0 (socket says 172.21.15.67:48084) 2024-04-17T08:52:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc387e00640 1 -- 172.21.15.67:0/945526953 learned_addr learned my addr 172.21.15.67:0/945526953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38ca00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc3880b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc38d400640 1 --2- 172.21.15.67:0/945526953 >> v2:172.21.15.187:3300/0 conn(0x7fc38806bfd0 0x7fc3880b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.738+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659175972 0 0) 0x7fc3880d0890 con 0x7fc3880b7080 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc370003520 con 0x7fc3880b7080 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2892788321 0 0) 0x7fc370003520 con 0x7fc3880b7080 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 msgr2=0x7fc3880b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc3880b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 >> v2:172.21.15.187:3300/0 conn(0x7fc38806bfd0 msgr2=0x7fc3880b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 --2- 172.21.15.67:0/945526953 >> v2:172.21.15.187:3300/0 conn(0x7fc38806bfd0 0x7fc3880b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3880d0b70 con 0x7fc3880b7080 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc374001670 con 0x7fc3880b7080 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 >> v1:172.21.15.67:6789/0 conn(0x7fc3880b7080 legacy=0x7fc38806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc370003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc370004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc370004d40 0x7fc3700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3880d0640 con 0x7fc370003a20 2024-04-17T08:52:27.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc37000ba70 con 0x7fc370004d40 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc37000bc40 con 0x7fc38815ecd0 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.739+0000 7fc38ca00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc370003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc387e00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc370004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38ca00640 1 -- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 msgr2=0x7fc370004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38d400640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc370004d40 0x7fc3700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38ca00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc370004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38ca00640 1 -- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc370004d40 msgr2=0x7fc3700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38ca00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc370004d40 0x7fc3700051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc38ca00640 1 -- 172.21.15.67:0/945526953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc370003520 con 0x7fc370003a20 2024-04-17T08:52:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.740+0000 7fc387e00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc38815ecd0 0x7fc370004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.741+0000 7fc38ca00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc370003e40 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc370005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4c63040ec259b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c039070 con 0x7fc370003a20 2024-04-17T08:52:27.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c02c6b0 con 0x7fc370003a20 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc387400640 1 -- 172.21.15.67:0/945526953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c009690 con 0x7fc370003a20 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc38de00640 1 -- 172.21.15.67:0/945526953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 msgr2=0x7fc370003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc38de00640 1 --2- 172.21.15.67:0/945526953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc370003e40 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc370005830 comp rx=0 tx=0).stop 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.742+0000 7fc38d400640 1 -- 172.21.15.67:0/945526953 reap_dead start 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.743+0000 7fc38de00640 1 -- 172.21.15.67:0/945526953 shutdown_connections 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.743+0000 7fc38de00640 1 -- 172.21.15.67:0/945526953 >> 172.21.15.67:0/945526953 conn(0x7fc38809aa60 msgr2=0x7fc3880b6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.743+0000 7fc38de00640 1 -- 172.21.15.67:0/945526953 shutdown_connections 2024-04-17T08:52:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.743+0000 7fc38de00640 1 -- 172.21.15.67:0/945526953 wait complete. 2024-04-17T08:52:27.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.744+0000 7fc38de00640 1 Processor -- start 2024-04-17T08:52:27.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.745+0000 7fc38de00640 1 -- start start 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.745+0000 7fc38de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 0x7fc388154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 0x7fc3881600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3880b1a40 con 0x7fc370003a20 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3880b44a0 con 0x7fc3881529e0 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3880b1bc0 con 0x7fc388154590 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52204/0 (socket says 172.21.15.67:52204) 2024-04-17T08:52:27.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 -- 172.21.15.67:0/831459572 learned_addr learned my addr 172.21.15.67:0/831459572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc387e00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 0x7fc388154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 -- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 msgr2=0x7fc3881600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38d400640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 0x7fc3881600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 0x7fc3881600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 -- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 msgr2=0x7fc388154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 0x7fc388154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38ca00640 1 -- 172.21.15.67:0/831459572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38815f300 con 0x7fc370003a20 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc38d400640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 0x7fc3881600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.746+0000 7fc387e00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 0x7fc388154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc38ca00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc37c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c042020 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37c03a040 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c039070 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38815f5e0 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.747+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc38815fba0 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.749+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3880755d0 con 0x7fc370003a20 2024-04-17T08:52:27.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.749+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fc37c045020 con 0x7fc370003a20 2024-04-17T08:52:27.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.750+0000 7fc384c00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 0x7fc36806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.750+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7fc37c0ad770 con 0x7fc370003a20 2024-04-17T08:52:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.751+0000 7fc387e00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 0x7fc36806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:27.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.752+0000 7fc387e00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 0x7fc36806ef60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc374001200 tx=0x7fc374009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:27.755+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc37c04c070 con 0x7fc370003a20 2024-04-17T08:52:27.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:52:27.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:52:27.959 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:27.959+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.010+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc388070af0 con 0x7fc370003a20 2024-04-17T08:52:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.011+0000 7fc384c00640 1 -- 172.21.15.67:0/831459572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5221 (secure 0 0 0) 0x7fc37c07b530 con 0x7fc370003a20 2024-04-17T08:52:28.011 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:28.011 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":12360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/991489299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":991489299},{"type":"v1","addr":"172.21.15.67:6835","nonce":991489299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:28.011 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 134 2024-04-17T08:52:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 msgr2=0x7fc36806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 0x7fc36806ef60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc374001200 tx=0x7fc374009040 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 msgr2=0x7fc3881524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009af0 tx=0x7fc37c004730 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 shutdown_connections 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc36806caa0 0x7fc36806ef60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc388154590 0x7fc3881600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3881529e0 0x7fc388154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 --2- 172.21.15.67:0/831459572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc370003a20 0x7fc3881524a0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 >> 172.21.15.67:0/831459572 conn(0x7fc38809aa60 msgr2=0x7fc3880cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 shutdown_connections 2024-04-17T08:52:28.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.014+0000 7fc38de00640 1 -- 172.21.15.67:0/831459572 wait complete. 2024-04-17T08:52:28.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.243+0000 7f5ada800640 1 Processor -- start 2024-04-17T08:52:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.243+0000 7f5ada800640 1 -- start start 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.244+0000 7f5ada800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ad4160280 0x7f5ad406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.244+0000 7f5ada800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad40cd4a0 con 0x7f5ad406b8c0 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.244+0000 7f5ada800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad41607f0 con 0x7f5ad4141ed0 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.244+0000 7f5ada800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5ad415ce30 con 0x7f5ad4160280 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ad4160280 0x7f5ad406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ad406b8c0 0x7f5ad40cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48088/0 (socket says 172.21.15.67:48088) 2024-04-17T08:52:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad9400640 1 -- 172.21.15.67:0/2731486650 learned_addr learned my addr 172.21.15.67:0/2731486650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad8a00640 1 -- 172.21.15.67:0/2731486650 >> v1:172.21.15.187:6789/0 conn(0x7f5ad4141ed0 legacy=0x7f5ad415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad8a00640 1 -- 172.21.15.67:0/2731486650 >> v1:172.21.15.67:6789/0 conn(0x7f5ad406b8c0 legacy=0x7f5ad40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.245+0000 7f5ad8a00640 1 -- 172.21.15.67:0/2731486650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad4160ad0 con 0x7f5ad4160280 2024-04-17T08:52:28.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ad8a00640 1 --2- 172.21.15.67:0/2731486650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad4160280 0x7f5ad406bca0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00097a0 tx=0x7f5ac002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9fb5cc4032f419 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ad3e00640 1 -- 172.21.15.67:0/2731486650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0035020 con 0x7f5ad4160280 2024-04-17T08:52:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ad3e00640 1 -- 172.21.15.67:0/2731486650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac0031830 con 0x7f5ad4160280 2024-04-17T08:52:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ad3e00640 1 -- 172.21.15.67:0/2731486650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0031c00 con 0x7f5ad4160280 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ada800640 1 -- 172.21.15.67:0/2731486650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad4160280 msgr2=0x7f5ad406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.247+0000 7f5ada800640 1 --2- 172.21.15.67:0/2731486650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad4160280 0x7f5ad406bca0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00097a0 tx=0x7f5ac002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.248+0000 7f5ada800640 1 -- 172.21.15.67:0/2731486650 shutdown_connections 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.248+0000 7f5ada800640 1 --2- 172.21.15.67:0/2731486650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad4160280 0x7f5ad406bca0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.248+0000 7f5ada800640 1 -- 172.21.15.67:0/2731486650 >> 172.21.15.67:0/2731486650 conn(0x7f5ad4067d10 msgr2=0x7f5ad40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.248+0000 7f5ada800640 1 -- 172.21.15.67:0/2731486650 shutdown_connections 2024-04-17T08:52:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.248+0000 7f5ada800640 1 -- 172.21.15.67:0/2731486650 wait complete. 2024-04-17T08:52:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.249+0000 7f5ada800640 1 Processor -- start 2024-04-17T08:52:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.250+0000 7f5ada800640 1 -- start start 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.250+0000 7f5ada800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ada800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ad4141ed0 0x7f5ad415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ada800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ad4160280 0x7f5ad406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ada800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4161090 con 0x7f5ad406b8c0 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ada800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad406c1e0 con 0x7f5ad4141ed0 2024-04-17T08:52:28.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ada800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad413f9b0 con 0x7f5ad4160280 2024-04-17T08:52:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52222/0 (socket says 172.21.15.67:52222) 2024-04-17T08:52:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 -- 172.21.15.67:0/2003652523 learned_addr learned my addr 172.21.15.67:0/2003652523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad8a00640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ad4141ed0 0x7f5ad415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 -- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ad4160280 msgr2=0x7f5ad406ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ad4160280 0x7f5ad406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 -- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ad4141ed0 msgr2=0x7f5ad415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ad4141ed0 0x7f5ad415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.251+0000 7f5ad9400640 1 -- 172.21.15.67:0/2003652523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad415e790 con 0x7f5ad406b8c0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ad9400640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c400 tx=0x7f5ac800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8017070 con 0x7f5ad406b8c0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac8005ce0 con 0x7f5ad406b8c0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80072f0 con 0x7f5ad406b8c0 2024-04-17T08:52:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad415ea70 con 0x7f5ad406b8c0 2024-04-17T08:52:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.252+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad415f030 con 0x7f5ad406b8c0 2024-04-17T08:52:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.253+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad4068a10 con 0x7f5ad406b8c0 2024-04-17T08:52:28.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.254+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f5ac801a070 con 0x7f5ad406b8c0 2024-04-17T08:52:28.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.255+0000 7f5ad1600640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 0x7f5ab406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.255+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f5ac808b670 con 0x7f5ad406b8c0 2024-04-17T08:52:28.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.256+0000 7f5ad8a00640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 0x7f5ab406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.257+0000 7f5ad8a00640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 0x7f5ab406ef60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0036000 tx=0x7f5ac0008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.260+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5ac8059440 con 0x7f5ad406b8c0 2024-04-17T08:52:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.518+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad406cdb0 con 0x7f5ad406b8c0 2024-04-17T08:52:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.519+0000 7f5ad1600640 1 -- 172.21.15.67:0/2003652523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5221 (secure 0 0 0) 0x7f5ac8058d40 con 0x7f5ad406b8c0 2024-04-17T08:52:28.519 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:28.519 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":134,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":12360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/991489299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":991489299},{"type":"v1","addr":"172.21.15.67:6835","nonce":991489299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:51:56.128948+0000","modified":"2024-04-17T08:51:59.267006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11841":{"gid":11841,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":7,"addr":"172.21.15.67:6833/4106648933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4106648933},{"type":"v1","addr":"172.21.15.67:6833","nonce":4106648933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11841,"qdb_cluster":[11841]},"id":16}]} 2024-04-17T08:52:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 134 2024-04-17T08:52:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 msgr2=0x7f5ab406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 0x7f5ab406ef60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0036000 tx=0x7f5ac0008030 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 msgr2=0x7f5ad413cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c400 tx=0x7f5ac800c8c0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 shutdown_connections 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5ab406caa0 0x7f5ab406ef60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5ad4160280 0x7f5ad406ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5ad4141ed0 0x7f5ad415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 --2- 172.21.15.67:0/2003652523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ad406b8c0 0x7f5ad413cdd0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 >> 172.21.15.67:0/2003652523 conn(0x7f5ad4067d10 msgr2=0x7f5ad415b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 shutdown_connections 2024-04-17T08:52:28.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.522+0000 7f5ada800640 1 -- 172.21.15.67:0/2003652523 wait complete. 2024-04-17T08:52:28.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.769+0000 7f391b200640 1 Processor -- start 2024-04-17T08:52:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.770+0000 7f391b200640 1 -- start start 2024-04-17T08:52:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.771+0000 7f391b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f391c15ec60 0x7f391c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.771+0000 7f391b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f391c0d2d30 con 0x7f391c0b6b30 2024-04-17T08:52:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.771+0000 7f391b200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f391c0d3010 con 0x7f391c0cfae0 2024-04-17T08:52:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.771+0000 7f391b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f391c0d49c0 con 0x7f391c15ec60 2024-04-17T08:52:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f391c15ec60 0x7f391c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f391c15ec60 0x7f391c0b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52226/0 (socket says 172.21.15.67:52226) 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 -- 172.21.15.67:0/1629153860 learned_addr learned my addr 172.21.15.67:0/1629153860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f391c0b6b30 0x7f391c0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48102/0 (socket says 172.21.15.67:48102) 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 -- 172.21.15.67:0/1629153860 >> v1:172.21.15.187:6789/0 conn(0x7f391c0cfae0 legacy=0x7f391c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 -- 172.21.15.67:0/1629153860 >> v1:172.21.15.67:6789/0 conn(0x7f391c0b6b30 legacy=0x7f391c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.772+0000 7f3919e00640 1 -- 172.21.15.67:0/1629153860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c0d32f0 con 0x7f391c15ec60 2024-04-17T08:52:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.773+0000 7f3919e00640 1 --2- 172.21.15.67:0/1629153860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c15ec60 0x7f391c0b65f0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f3904009870 tx=0x7f390402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e811556df72b04 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.774+0000 7f3918a00640 1 -- 172.21.15.67:0/1629153860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904034070 con 0x7f391c15ec60 2024-04-17T08:52:28.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.774+0000 7f3918a00640 1 -- 172.21.15.67:0/1629153860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3904030860 con 0x7f391c15ec60 2024-04-17T08:52:28.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.774+0000 7f3918a00640 1 -- 172.21.15.67:0/1629153860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904030c30 con 0x7f391c15ec60 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.774+0000 7f391b200640 1 -- 172.21.15.67:0/1629153860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c15ec60 msgr2=0x7f391c0b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.774+0000 7f391b200640 1 --2- 172.21.15.67:0/1629153860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c15ec60 0x7f391c0b65f0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f3904009870 tx=0x7f390402b9a0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.775+0000 7f391b200640 1 -- 172.21.15.67:0/1629153860 shutdown_connections 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.775+0000 7f391b200640 1 --2- 172.21.15.67:0/1629153860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c15ec60 0x7f391c0b65f0 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.775+0000 7f391b200640 1 -- 172.21.15.67:0/1629153860 >> 172.21.15.67:0/1629153860 conn(0x7f391c0d15f0 msgr2=0x7f391c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.775+0000 7f391b200640 1 -- 172.21.15.67:0/1629153860 shutdown_connections 2024-04-17T08:52:28.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.775+0000 7f391b200640 1 -- 172.21.15.67:0/1629153860 wait complete. 2024-04-17T08:52:28.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.776+0000 7f391b200640 1 Processor -- start 2024-04-17T08:52:28.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.777+0000 7f391b200640 1 -- start start 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f391c0b6b30 0x7f391c0b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f391c15ec60 0x7f391c1602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0b6f70 con 0x7f391c0cfae0 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0d3660 con 0x7f391c0b6b30 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f391c0d00c0 con 0x7f391c15ec60 2024-04-17T08:52:28.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52236/0 (socket says 172.21.15.67:52236) 2024-04-17T08:52:28.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 -- 172.21.15.67:0/3773021662 learned_addr learned my addr 172.21.15.67:0/3773021662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:28.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919e00640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f391c0b6b30 0x7f391c0b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f391a800640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f391c15ec60 0x7f391c1602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 -- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f391c15ec60 msgr2=0x7f391c1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f391c15ec60 0x7f391c1602a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 -- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f391c0b6b30 msgr2=0x7f391c0b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f391c0b6b30 0x7f391c0b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.778+0000 7f3919400640 1 -- 172.21.15.67:0/3773021662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c0ac3c0 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f3919400640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f390c00c790 tx=0x7f390c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c017070 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f390c00a640 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c00a9f0 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c0ac660 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.779+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c0acc20 con 0x7f391c0cfae0 2024-04-17T08:52:28.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.780+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391c068a10 con 0x7f391c0cfae0 2024-04-17T08:52:28.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.781+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f390c01a070 con 0x7f391c0cfae0 2024-04-17T08:52:28.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.782+0000 7f3912000640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 0x7f38f806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:28.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.783+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f390c08b7e0 con 0x7f391c0cfae0 2024-04-17T08:52:28.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.783+0000 7f3919e00640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 0x7f38f806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:28.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.785+0000 7f3919e00640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 0x7f38f806ef60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3904036000 tx=0x7f390403b040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:28.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.787+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f390c058f50 con 0x7f391c0cfae0 2024-04-17T08:52:28.925 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:28.925+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:28.999+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f391c0b3590 con 0x7f391c0cfae0 2024-04-17T08:52:29.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.002+0000 7f3912000640 1 -- 172.21.15.67:0/3773021662 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+18025 (secure 0 0 0) 0x7f390c05d250 con 0x7f391c0cfae0 2024-04-17T08:52:29.002 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:29.002 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":259,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:24.524735+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-04-17T08:51:54.448741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-04-17T08:51:55.106540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.67:6835/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.67:6834/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:29.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.005+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 msgr2=0x7f38f806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.005+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 0x7f38f806ef60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3904036000 tx=0x7f390403b040 comp rx=0 tx=0).stop 2024-04-17T08:52:29.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 msgr2=0x7f391c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f390c00c790 tx=0x7f390c00cc50 comp rx=0 tx=0).stop 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 shutdown_connections 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f38f806caa0 0x7f38f806ef60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f391c15ec60 0x7f391c1602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f391c0cfae0 0x7f391c0abe80 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 --2- 172.21.15.67:0/3773021662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f391c0b6b30 0x7f391c0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 >> 172.21.15.67:0/3773021662 conn(0x7f391c0d15f0 msgr2=0x7f391c0b60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 shutdown_connections 2024-04-17T08:52:29.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.006+0000 7f391b200640 1 -- 172.21.15.67:0/3773021662 wait complete. 2024-04-17T08:52:29.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-04-17T08:52:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.241+0000 7f07e0c00640 1 Processor -- start 2024-04-17T08:52:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.242+0000 7f07e0c00640 1 -- start start 2024-04-17T08:52:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07e0c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07dc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07e0c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07dc0d3520 0x7f07dc0b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07e0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07dc0d4140 con 0x7f07dc0d18e0 2024-04-17T08:52:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07e0c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f07dc0b83b0 con 0x7f07dc0d3520 2024-04-17T08:52:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07e0c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc0d1d20 con 0x7f07dc15ee20 2024-04-17T08:52:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07dc0d18e0 0x7f07dc0d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48108/0 (socket says 172.21.15.67:48108) 2024-04-17T08:52:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.243+0000 7f07daa00640 1 -- 172.21.15.67:0/200693286 learned_addr learned my addr 172.21.15.67:0/200693286 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07dbe00640 1 --2- 172.21.15.67:0/200693286 >> v2:172.21.15.187:3300/0 conn(0x7f07dc0d3520 0x7f07dc0b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07db400640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07dc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518272037 0 0) 0x7f07dc0d4140 con 0x7f07dc0d18e0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07c0003520 con 0x7f07dc0d18e0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3398870591 0 0) 0x7f07c0003520 con 0x7f07dc0d18e0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 msgr2=0x7f07dc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07dc0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 >> v2:172.21.15.187:3300/0 conn(0x7f07dc0d3520 msgr2=0x7f07dc0b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 --2- 172.21.15.67:0/200693286 >> v2:172.21.15.187:3300/0 conn(0x7f07dc0d3520 0x7f07dc0b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc0d3b80 con 0x7f07dc0d18e0 2024-04-17T08:52:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f07c8001670 con 0x7f07dc0d18e0 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.244+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 >> v1:172.21.15.67:6789/0 conn(0x7f07dc0d18e0 legacy=0x7f07dc0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07c0003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c0003ff0 0x7f07c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c0004d40 0x7f07c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b83b0 con 0x7f07c0004d40 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0d1d20 con 0x7f07c0003ff0 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07c000ba40 con 0x7f07dc15ee20 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c0004d40 0x7f07c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07db400640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07c0003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 -- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 msgr2=0x7f07c0003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc15ee20 0x7f07c0003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 -- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c0003ff0 msgr2=0x7f07c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07c0003ff0 0x7f07c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.245+0000 7f07dbe00640 1 -- 172.21.15.67:0/200693286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c0003520 con 0x7f07c0004d40 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07dbe00640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c0004d40 0x7f07c00051c0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009fc0 tx=0x7f07c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998cd52ff3cf98ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4039070 con 0x7f07c0004d40 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d402cc60 con 0x7f07c0004d40 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07da000640 1 -- 172.21.15.67:0/200693286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4009c10 con 0x7f07c0004d40 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07e0c00640 1 -- 172.21.15.67:0/200693286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c0004d40 msgr2=0x7f07c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07e0c00640 1 --2- 172.21.15.67:0/200693286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c0004d40 0x7f07c00051c0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009fc0 tx=0x7f07c0005830 comp rx=0 tx=0).stop 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.247+0000 7f07dbe00640 1 -- 172.21.15.67:0/200693286 reap_dead start 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.248+0000 7f07e0c00640 1 -- 172.21.15.67:0/200693286 shutdown_connections 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.248+0000 7f07e0c00640 1 -- 172.21.15.67:0/200693286 >> 172.21.15.67:0/200693286 conn(0x7f07dc06d3c0 msgr2=0x7f07dc0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.248+0000 7f07e0c00640 1 -- 172.21.15.67:0/200693286 shutdown_connections 2024-04-17T08:52:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.248+0000 7f07e0c00640 1 -- 172.21.15.67:0/200693286 wait complete. 2024-04-17T08:52:29.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.250+0000 7f07e0c00640 1 Processor -- start 2024-04-17T08:52:29.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.250+0000 7f07e0c00640 1 -- start start 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07c0004d40 0x7f07dc0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc156440 0x7f07dc154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc15f200 con 0x7f07dc0b9af0 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b8040 con 0x7f07dc156440 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07e0c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc0d13a0 con 0x7f07c0004d40 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52260/0 (socket says 172.21.15.67:52260) 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.251+0000 7f07dbe00640 1 -- 172.21.15.67:0/541989110 learned_addr learned my addr 172.21.15.67:0/541989110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07daa00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc156440 0x7f07dc154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07db400640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07c0004d40 0x7f07dc0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 -- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07c0004d40 msgr2=0x7f07dc0b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07c0004d40 0x7f07dc0b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 -- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc156440 msgr2=0x7f07dc154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc156440 0x7f07dc154d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 -- 172.21.15.67:0/541989110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc0cddf0 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07dbe00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f07d402c6d0 tx=0x7f07d4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4039030 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d403a040 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc156a00 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.252+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d4003990 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.253+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc156fc0 con 0x7f07dc0b9af0 2024-04-17T08:52:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.254+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc0ba4b0 con 0x7f07dc0b9af0 2024-04-17T08:52:29.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.255+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f07d4045030 con 0x7f07dc0b9af0 2024-04-17T08:52:29.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.256+0000 7f07d3400640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 0x7f07bc06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.256+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 6695+0+0 (secure 0 0 0) 0x7f07d40ae1d0 con 0x7f07dc0b9af0 2024-04-17T08:52:29.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.257+0000 7f07db400640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 0x7f07bc06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.258+0000 7f07db400640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 0x7f07bc06efb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009950 tx=0x7f07cc008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.261+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f07d404c020 con 0x7f07dc0b9af0 2024-04-17T08:52:29.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.512+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7f07dc068a10 con 0x7f07dc0b9af0 2024-04-17T08:52:29.514 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:52:29.514+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:52:29.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.518+0000 7f07d3400640 1 -- 172.21.15.67:0/541989110 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v135) ==== 167+0+0 (secure 0 0 0) 0x7f07dc0cddf0 con 0x7f07dc0b9af0 2024-04-17T08:52:29.517 INFO:teuthology.orchestra.run.smithi067.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:52:29.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.521+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 msgr2=0x7f07bc06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.521+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 0x7f07bc06efb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009950 tx=0x7f07cc008040 comp rx=0 tx=0).stop 2024-04-17T08:52:29.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.521+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 msgr2=0x7f07dc0b9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.521+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f07d402c6d0 tx=0x7f07d4009750 comp rx=0 tx=0).stop 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 shutdown_connections 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f07bc06caf0 0x7f07bc06efb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc0b9af0 0x7f07dc0b9f70 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc156440 0x7f07dc154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 --2- 172.21.15.67:0/541989110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07c0004d40 0x7f07dc0b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 >> 172.21.15.67:0/541989110 conn(0x7f07dc06d3c0 msgr2=0x7f07dc0d4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.522+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 shutdown_connections 2024-04-17T08:52:29.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.523+0000 7f07e0c00640 1 -- 172.21.15.67:0/541989110 wait complete. 2024-04-17T08:52:29.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-04-17T08:52:29.544 INFO:tasks.ceph.mds.c.smithi067.stderr: -1948> 2024-04-17T08:52:01.432+0000 7fcf37800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:52:29.626 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:52:29.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.746+0000 7f955ca00640 1 Processor -- start 2024-04-17T08:52:29.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.746+0000 7f955ca00640 1 -- start start 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.747+0000 7f955ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95580af400 0x7f9558162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f955ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955815f780 con 0x7f955806b8c0 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f955ca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955815fa60 con 0x7f9558159020 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f955ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f955815f540 con 0x7f95580af400 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9556a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95580af400 0x7f9558162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9557e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f955806b8c0 0x7f95580adb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48122/0 (socket says 172.21.15.67:48122) 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9557e00640 1 -- 172.21.15.67:0/687479658 learned_addr learned my addr 172.21.15.67:0/687479658 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9556a00640 1 -- 172.21.15.67:0/687479658 >> v1:172.21.15.187:6789/0 conn(0x7f9558159020 legacy=0x7f955815ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9556a00640 1 -- 172.21.15.67:0/687479658 >> v1:172.21.15.67:6789/0 conn(0x7f955806b8c0 legacy=0x7f95580adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.748+0000 7f9556a00640 1 -- 172.21.15.67:0/687479658 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95581624a0 con 0x7f95580af400 2024-04-17T08:52:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.750+0000 7f9556a00640 1 --2- 172.21.15.67:0/687479658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95580af400 0x7f9558162ca0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f95440097a0 tx=0x7f954402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de035f26e6adea7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.750+0000 7f9556000640 1 -- 172.21.15.67:0/687479658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544035020 con 0x7f95580af400 2024-04-17T08:52:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.750+0000 7f9556000640 1 -- 172.21.15.67:0/687479658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9544031830 con 0x7f95580af400 2024-04-17T08:52:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.750+0000 7f9556000640 1 -- 172.21.15.67:0/687479658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544031c00 con 0x7f95580af400 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 -- 172.21.15.67:0/687479658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95580af400 msgr2=0x7f9558162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 --2- 172.21.15.67:0/687479658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95580af400 0x7f9558162ca0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f95440097a0 tx=0x7f954402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 -- 172.21.15.67:0/687479658 shutdown_connections 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 --2- 172.21.15.67:0/687479658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95580af400 0x7f9558162ca0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 -- 172.21.15.67:0/687479658 >> 172.21.15.67:0/687479658 conn(0x7f95580b2190 msgr2=0x7f95580b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 -- 172.21.15.67:0/687479658 shutdown_connections 2024-04-17T08:52:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.751+0000 7f955ca00640 1 -- 172.21.15.67:0/687479658 wait complete. 2024-04-17T08:52:29.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.753+0000 7f955ca00640 1 Processor -- start 2024-04-17T08:52:29.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.753+0000 7f955ca00640 1 -- start start 2024-04-17T08:52:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955806b8c0 0x7f9558153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95580af400 0x7f9558160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f95581631e0 con 0x7f9558159020 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f955815fd40 con 0x7f955806b8c0 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f955ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9558163360 con 0x7f95580af400 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52294/0 (socket says 172.21.15.67:52294) 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 -- 172.21.15.67:0/373582085 learned_addr learned my addr 172.21.15.67:0/373582085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557400640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955806b8c0 0x7f9558153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9556a00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95580af400 0x7f9558160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 -- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95580af400 msgr2=0x7f9558160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95580af400 0x7f9558160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 -- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955806b8c0 msgr2=0x7f9558153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.754+0000 7f9557e00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955806b8c0 0x7f9558153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f9557e00640 1 -- 172.21.15.67:0/373582085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9558160e70 con 0x7f9558159020 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f9557e00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f955000cbf0 tx=0x7f955000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550017070 con 0x7f9558159020 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955000ad40 con 0x7f9558159020 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9558161150 con 0x7f9558159020 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550012600 con 0x7f9558159020 2024-04-17T08:52:29.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.755+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9558161710 con 0x7f9558159020 2024-04-17T08:52:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.757+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95580b41d0 con 0x7f9558159020 2024-04-17T08:52:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.757+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f955001a070 con 0x7f9558159020 2024-04-17T08:52:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.758+0000 7f954f400640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 0x7f953806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:29.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.758+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 6781+0+0 (secure 0 0 0) 0x7f955008bfc0 con 0x7f9558159020 2024-04-17T08:52:29.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.759+0000 7f9557400640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 0x7f953806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:29.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.760+0000 7f9557400640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 0x7f953806ef60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9548007fd0 tx=0x7f9548009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:29.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:29.763+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f95580b41d0 con 0x7f9558159020 2024-04-17T08:52:29.922 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:29.922+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:30.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.019+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7f9558068a10 con 0x7f9558159020 2024-04-17T08:52:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.586+0000 7f954f400640 1 -- 172.21.15.67:0/373582085 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v136) ==== 107+0+0 (secure 0 0 0) 0x7f95500596e0 con 0x7f9558159020 2024-04-17T08:52:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 msgr2=0x7f953806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 0x7f953806ef60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9548007fd0 tx=0x7f9548009040 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 msgr2=0x7f95581e8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f955000cbf0 tx=0x7f955000a430 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 shutdown_connections 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f953806caa0 0x7f953806ef60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9558159020 0x7f95581e8f90 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95580af400 0x7f9558160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 --2- 172.21.15.67:0/373582085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f955806b8c0 0x7f9558153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.590+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 >> 172.21.15.67:0/373582085 conn(0x7f95580b2190 msgr2=0x7f95580d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.591+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 shutdown_connections 2024-04-17T08:52:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.591+0000 7f955ca00640 1 -- 172.21.15.67:0/373582085 wait complete. 2024-04-17T08:52:30.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-04-17T08:52:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.846+0000 7fc898000640 1 Processor -- start 2024-04-17T08:52:30.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.846+0000 7fc898000640 1 -- start start 2024-04-17T08:52:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.848+0000 7fc898000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8900b18d0 0x7fc8900cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.848+0000 7fc898000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8900b1db0 0x7fc8900cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.848+0000 7fc898000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc890167a40 con 0x7fc890162f40 2024-04-17T08:52:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.848+0000 7fc898000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc890166ec0 con 0x7fc8900b18d0 2024-04-17T08:52:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.848+0000 7fc898000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8901671a0 con 0x7fc8900b1db0 2024-04-17T08:52:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8900b18d0 0x7fc8900cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8900b18d0 0x7fc8900cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52296/0 (socket says 172.21.15.67:52296) 2024-04-17T08:52:30.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 -- 172.21.15.67:0/3866733941 learned_addr learned my addr 172.21.15.67:0/3866733941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc895600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc890162f40 0x7fc89006cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48136/0 (socket says 172.21.15.67:48136) 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc896000640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8900b1db0 0x7fc8900cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 -- 172.21.15.67:0/3866733941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8900b1db0 msgr2=0x7fc8900cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8900b1db0 0x7fc8900cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 -- 172.21.15.67:0/3866733941 >> v1:172.21.15.67:6789/0 conn(0x7fc890162f40 legacy=0x7fc89006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.849+0000 7fc894c00640 1 -- 172.21.15.67:0/3866733941 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc890167480 con 0x7fc8900b18d0 2024-04-17T08:52:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.851+0000 7fc894c00640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc8900cf150 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009870 tx=0x7fc87c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69b86c27eedd7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.851+0000 7fc88fe00640 1 -- 172.21.15.67:0/3866733941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0317c0 con 0x7fc8900b18d0 2024-04-17T08:52:30.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.851+0000 7fc88fe00640 1 -- 172.21.15.67:0/3866733941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87c037070 con 0x7fc8900b18d0 2024-04-17T08:52:30.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.851+0000 7fc88fe00640 1 -- 172.21.15.67:0/3866733941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c031b80 con 0x7fc8900b18d0 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 -- 172.21.15.67:0/3866733941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 msgr2=0x7fc8900cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc8900cf150 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009870 tx=0x7fc87c02b860 comp rx=0 tx=0).stop 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 -- 172.21.15.67:0/3866733941 shutdown_connections 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8900b1db0 0x7fc8900cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 --2- 172.21.15.67:0/3866733941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc8900cf150 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 -- 172.21.15.67:0/3866733941 >> 172.21.15.67:0/3866733941 conn(0x7fc8900b4390 msgr2=0x7fc8900b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 -- 172.21.15.67:0/3866733941 shutdown_connections 2024-04-17T08:52:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.852+0000 7fc898000640 1 -- 172.21.15.67:0/3866733941 wait complete. 2024-04-17T08:52:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.855+0000 7fc898000640 1 Processor -- start 2024-04-17T08:52:30.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.855+0000 7fc898000640 1 -- start start 2024-04-17T08:52:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8900b1db0 0x7fc89014b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc890162f40 0x7fc89015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc890166ae0 con 0x7fc8900b18d0 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc890163320 con 0x7fc8900b1db0 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.856+0000 7fc898000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc890166c60 con 0x7fc890162f40 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52300/0 (socket says 172.21.15.67:52300) 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 -- 172.21.15.67:0/1208885497 learned_addr learned my addr 172.21.15.67:0/1208885497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc896000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc890162f40 0x7fc89015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 -- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc890162f40 msgr2=0x7fc89015ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc890162f40 0x7fc89015ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 -- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8900b1db0 msgr2=0x7fc89014b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8900b1db0 0x7fc89014b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.857+0000 7fc895600640 1 -- 172.21.15.67:0/1208885497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8901643d0 con 0x7fc8900b18d0 2024-04-17T08:52:30.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc895600640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc88400c730 tx=0x7fc88400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:30.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884017070 con 0x7fc8900b18d0 2024-04-17T08:52:30.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88400ce40 con 0x7fc8900b18d0 2024-04-17T08:52:30.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88400a860 con 0x7fc8900b18d0 2024-04-17T08:52:30.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8901640f0 con 0x7fc8900b18d0 2024-04-17T08:52:30.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.858+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc890165070 con 0x7fc8900b18d0 2024-04-17T08:52:30.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.859+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc890071750 con 0x7fc8900b18d0 2024-04-17T08:52:30.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.860+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fc88401a070 con 0x7fc8900b18d0 2024-04-17T08:52:30.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.861+0000 7fc88d600640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 0x7fc87006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:30.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.861+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 6781+0+0 (secure 0 0 0) 0x7fc88408b8d0 con 0x7fc8900b18d0 2024-04-17T08:52:30.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.862+0000 7fc894c00640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 0x7fc87006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:30.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.863+0000 7fc894c00640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 0x7fc87006ef60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc87c036000 tx=0x7fc87c030000 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:30.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:30.866+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc884059640 con 0x7fc8900b18d0 2024-04-17T08:52:30.950 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:30.950+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.073+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc890068d60 con 0x7fc8900b18d0 2024-04-17T08:52:31.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.598+0000 7fc88d600640 1 -- 172.21.15.67:0/1208885497 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v261) ==== 190+0+0 (secure 0 0 0) 0x7fc884058f40 con 0x7fc8900b18d0 2024-04-17T08:52:31.598 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_metadata' removed 2024-04-17T08:52:31.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 msgr2=0x7fc87006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:31.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 0x7fc87006ef60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc87c036000 tx=0x7fc87c030000 comp rx=0 tx=0).stop 2024-04-17T08:52:31.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 msgr2=0x7fc890165870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:31.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc88400c730 tx=0x7fc88400cbf0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 shutdown_connections 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc87006caa0 0x7fc87006ef60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc890162f40 0x7fc89015ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8900b1db0 0x7fc89014b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 --2- 172.21.15.67:0/1208885497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900b18d0 0x7fc890165870 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 >> 172.21.15.67:0/1208885497 conn(0x7fc8900b4390 msgr2=0x7fc8900b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:31.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.603+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 shutdown_connections 2024-04-17T08:52:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.605+0000 7fc898000640 1 -- 172.21.15.67:0/1208885497 wait complete. 2024-04-17T08:52:31.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-04-17T08:52:31.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.842+0000 7fabafe00640 1 Processor -- start 2024-04-17T08:52:31.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.842+0000 7fabafe00640 1 -- start start 2024-04-17T08:52:31.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faba80cfe90 0x7faba806e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba80cd940 0x7faba80cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faba806b8c0 0x7faba806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faba815d210 con 0x7faba806b8c0 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faba815d4f0 con 0x7faba80cfe90 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.844+0000 7fabafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faba815d7d0 con 0x7faba80cd940 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faba806b8c0 0x7faba806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faba806b8c0 0x7faba806bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52304/0 (socket says 172.21.15.67:52304) 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 -- 172.21.15.67:0/3106541727 learned_addr learned my addr 172.21.15.67:0/3106541727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabaca00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba80cd940 0x7faba80cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabad400640 1 --2- 172.21.15.67:0/3106541727 >> v2:172.21.15.187:3300/0 conn(0x7faba80cfe90 0x7faba806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 -- 172.21.15.67:0/3106541727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba80cd940 msgr2=0x7faba80cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba80cd940 0x7faba80cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 -- 172.21.15.67:0/3106541727 >> v2:172.21.15.187:3300/0 conn(0x7faba80cfe90 msgr2=0x7faba806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 --2- 172.21.15.67:0/3106541727 >> v2:172.21.15.187:3300/0 conn(0x7faba80cfe90 0x7faba806e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.845+0000 7fabade00640 1 -- 172.21.15.67:0/3106541727 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba815dab0 con 0x7faba806b8c0 2024-04-17T08:52:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.846+0000 7fabade00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba806b8c0 0x7faba806bd50 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7faba80d0520 tx=0x7faba002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca849a8ae56d2d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.847+0000 7faba7e00640 1 -- 172.21.15.67:0/3106541727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba002ca70 con 0x7faba806b8c0 2024-04-17T08:52:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.847+0000 7faba7e00640 1 -- 172.21.15.67:0/3106541727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba002cc10 con 0x7faba806b8c0 2024-04-17T08:52:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.847+0000 7faba7e00640 1 -- 172.21.15.67:0/3106541727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00343f0 con 0x7faba806b8c0 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 -- 172.21.15.67:0/3106541727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba806b8c0 msgr2=0x7faba806bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba806b8c0 0x7faba806bd50 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7faba80d0520 tx=0x7faba002bed0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 -- 172.21.15.67:0/3106541727 shutdown_connections 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba806b8c0 0x7faba806bd50 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 --2- 172.21.15.67:0/3106541727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba80cd940 0x7faba80cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 --2- 172.21.15.67:0/3106541727 >> v2:172.21.15.187:3300/0 conn(0x7faba80cfe90 0x7faba806e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 -- 172.21.15.67:0/3106541727 >> 172.21.15.67:0/3106541727 conn(0x7faba80b2370 msgr2=0x7faba80b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 -- 172.21.15.67:0/3106541727 shutdown_connections 2024-04-17T08:52:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.848+0000 7fabafe00640 1 -- 172.21.15.67:0/3106541727 wait complete. 2024-04-17T08:52:31.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.850+0000 7fabafe00640 1 Processor -- start 2024-04-17T08:52:31.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.850+0000 7fabafe00640 1 -- start start 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba806b8c0 0x7faba815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faba80cd940 0x7faba81f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faba815ce90 con 0x7faba80cfe90 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faba80d0270 con 0x7faba80cd940 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabafe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faba80722d0 con 0x7faba806b8c0 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52306/0 (socket says 172.21.15.67:52306) 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabade00640 1 -- 172.21.15.67:0/1013068559 learned_addr learned my addr 172.21.15.67:0/1013068559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabad400640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba806b8c0 0x7faba815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.851+0000 7fabaca00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faba80cd940 0x7faba81f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 -- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba806b8c0 msgr2=0x7faba815c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba806b8c0 0x7faba815c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 -- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faba80cd940 msgr2=0x7faba81f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faba80cd940 0x7faba81f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 -- 172.21.15.67:0/1013068559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba80ac640 con 0x7faba80cfe90 2024-04-17T08:52:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabade00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7faba0036040 tx=0x7faba0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0043020 con 0x7faba80cfe90 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba003e450 con 0x7faba80cfe90 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0009510 con 0x7faba80cfe90 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba80ac920 con 0x7faba80cfe90 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.852+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba80acee0 con 0x7faba80cfe90 2024-04-17T08:52:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.854+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba8073980 con 0x7faba80cfe90 2024-04-17T08:52:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.855+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7faba0046020 con 0x7faba80cfe90 2024-04-17T08:52:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.856+0000 7faba5600640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 0x7fab8c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.856+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(261..261 src has 1..261) ==== 6337+0+0 (secure 0 0 0) 0x7faba00aea30 con 0x7faba80cfe90 2024-04-17T08:52:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.857+0000 7fabad400640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 0x7fab8c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.858+0000 7fabad400640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 0x7fab8c06ef60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab9c006fd0 tx=0x7fab9c008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:31.860+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7faba004d030 con 0x7faba80cfe90 2024-04-17T08:52:32.001 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:32.000+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.068+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7faba8068d60 con 0x7faba80cfe90 2024-04-17T08:52:32.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.627+0000 7faba5600640 1 -- 172.21.15.67:0/1013068559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v262) ==== 178+0+0 (secure 0 0 0) 0x7faba007c8c0 con 0x7faba80cfe90 2024-04-17T08:52:32.627 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'backup_fs_data' removed 2024-04-17T08:52:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.632+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 msgr2=0x7fab8c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.632+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 0x7fab8c06ef60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab9c006fd0 tx=0x7fab9c008040 comp rx=0 tx=0).stop 2024-04-17T08:52:32.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.632+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 msgr2=0x7faba80ab9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.632+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7faba0036040 tx=0x7faba0004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 shutdown_connections 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fab8c06caa0 0x7fab8c06ef60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faba80cfe90 0x7faba80ab9f0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faba80cd940 0x7faba81f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 --2- 172.21.15.67:0/1013068559 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faba806b8c0 0x7faba815c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.634+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 >> 172.21.15.67:0/1013068559 conn(0x7faba80b2370 msgr2=0x7faba8144ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:32.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.636+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 shutdown_connections 2024-04-17T08:52:32.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.636+0000 7fabafe00640 1 -- 172.21.15.67:0/1013068559 wait complete. 2024-04-17T08:52:32.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:52:32.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.877+0000 7f9fd1200640 1 Processor -- start 2024-04-17T08:52:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.877+0000 7f9fd1200640 1 -- start start 2024-04-17T08:52:32.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fd1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fcc0cd920 0x7f9fcc06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc06c0a0 0x7f9fcc06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fd1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fcc146360 con 0x7f9fcc145f80 2024-04-17T08:52:32.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fd1200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9fcc0727a0 con 0x7f9fcc0cd920 2024-04-17T08:52:32.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fd1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fcc071e70 con 0x7f9fcc06c0a0 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.879+0000 7f9fc3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fcc0cd920 0x7f9fcc06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fcc0cd920 0x7f9fcc06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52308/0 (socket says 172.21.15.67:52308) 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 -- 172.21.15.67:0/1920679588 learned_addr learned my addr 172.21.15.67:0/1920679588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fcbe00640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc06c0a0 0x7f9fcc06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 -- 172.21.15.67:0/1920679588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc06c0a0 msgr2=0x7f9fcc06e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc06c0a0 0x7f9fcc06e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 -- 172.21.15.67:0/1920679588 >> v1:172.21.15.67:6789/0 conn(0x7f9fcc145f80 legacy=0x7f9fcc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.880+0000 7f9fc3e00640 1 -- 172.21.15.67:0/1920679588 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fcc072150 con 0x7f9fcc0cd920 2024-04-17T08:52:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.881+0000 7f9fc3e00640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc06bb60 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009870 tx=0x7f9fb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1906dac1becf4d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fcaa00640 1 -- 172.21.15.67:0/1920679588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4035020 con 0x7f9fcc0cd920 2024-04-17T08:52:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fcaa00640 1 -- 172.21.15.67:0/1920679588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb4031640 con 0x7f9fcc0cd920 2024-04-17T08:52:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fcaa00640 1 -- 172.21.15.67:0/1920679588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb402d040 con 0x7f9fcc0cd920 2024-04-17T08:52:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 -- 172.21.15.67:0/1920679588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 msgr2=0x7f9fcc06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc06bb60 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009870 tx=0x7f9fb402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 -- 172.21.15.67:0/1920679588 shutdown_connections 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc06c0a0 0x7f9fcc06e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 --2- 172.21.15.67:0/1920679588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc06bb60 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 -- 172.21.15.67:0/1920679588 >> 172.21.15.67:0/1920679588 conn(0x7f9fcc0b2190 msgr2=0x7f9fcc0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.882+0000 7f9fd1200640 1 -- 172.21.15.67:0/1920679588 shutdown_connections 2024-04-17T08:52:32.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.883+0000 7f9fd1200640 1 -- 172.21.15.67:0/1920679588 wait complete. 2024-04-17T08:52:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.884+0000 7f9fd1200640 1 Processor -- start 2024-04-17T08:52:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.884+0000 7f9fd1200640 1 -- start start 2024-04-17T08:52:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fcc06c0a0 0x7f9fcc1f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc145f80 0x7f9fcc1f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9fcc072c30 con 0x7f9fcc0cd920 2024-04-17T08:52:32.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9fcc06c530 con 0x7f9fcc06c0a0 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.885+0000 7f9fd1200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fcc072db0 con 0x7f9fcc145f80 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52318/0 (socket says 172.21.15.67:52318) 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 -- 172.21.15.67:0/742361840 learned_addr learned my addr 172.21.15.67:0/742361840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fcbe00640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc145f80 0x7f9fcc1f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 -- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc145f80 msgr2=0x7f9fcc1f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc145f80 0x7f9fcc1f4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 -- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fcc06c0a0 msgr2=0x7f9fcc1f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fcc06c0a0 0x7f9fcc1f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.886+0000 7f9fc3e00640 1 -- 172.21.15.67:0/742361840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fcc1f5140 con 0x7f9fcc0cd920 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fc3e00640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4000c00 tx=0x7f9fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4040070 con 0x7f9fcc0cd920 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb40399f0 con 0x7f9fcc0cd920 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4035040 con 0x7f9fcc0cd920 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fcc1f5420 con 0x7f9fcc0cd920 2024-04-17T08:52:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.887+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fcc1f81a0 con 0x7f9fcc0cd920 2024-04-17T08:52:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.888+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fcc068a10 con 0x7f9fcc0cd920 2024-04-17T08:52:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.889+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f9fb404b060 con 0x7f9fcc0cd920 2024-04-17T08:52:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.890+0000 7f9fc3400640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 0x7f9fa806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:32.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.890+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f9fb40ad7f0 con 0x7f9fcc0cd920 2024-04-17T08:52:32.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.891+0000 7f9fcb400640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 0x7f9fa806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.892+0000 7f9fcb400640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 0x7f9fa806ef90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000f60 tx=0x7f9fbc0079d0 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:32.894+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9fb407b960 con 0x7f9fcc0cd920 2024-04-17T08:52:33.064 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:33.064+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.121+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f9fcc0b1f70 con 0x7f9fcc0cd920 2024-04-17T08:52:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.129+0000 7f9fc3400640 1 -- 172.21.15.67:0/742361840 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) ==== 115+0+0 (secure 0 0 0) 0x7f9fb407b260 con 0x7f9fcc0cd920 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 msgr2=0x7f9fa806ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 0x7f9fa806ef90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000f60 tx=0x7f9fbc0079d0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 msgr2=0x7f9fcc0aec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4000c00 tx=0x7f9fb40046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 shutdown_connections 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9fa806cad0 0x7f9fa806ef90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fcc145f80 0x7f9fcc1f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fcc0cd920 0x7f9fcc0aec00 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 --2- 172.21.15.67:0/742361840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fcc06c0a0 0x7f9fcc1f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 >> 172.21.15.67:0/742361840 conn(0x7f9fcc0b2190 msgr2=0x7f9fcc0b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 shutdown_connections 2024-04-17T08:52:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.132+0000 7f9fd1200640 1 -- 172.21.15.67:0/742361840 wait complete. 2024-04-17T08:52:33.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:52:33.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.379+0000 7f2b16a00640 1 Processor -- start 2024-04-17T08:52:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.379+0000 7f2b16a00640 1 -- start start 2024-04-17T08:52:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b10145f80 0x7f2b100cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b100cd750 0x7f2b1006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2b1006c0a0 0x7f2b10160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2b100cdbb0 con 0x7f2b100cd750 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2b100727a0 con 0x7f2b1006c0a0 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b16a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b10146360 con 0x7f2b10145f80 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b100cd750 0x7f2b1006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b100cd750 0x7f2b1006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51076/0 (socket says 172.21.15.67:51076) 2024-04-17T08:52:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.381+0000 7f2b14c00640 1 -- 172.21.15.67:0/1837818890 learned_addr learned my addr 172.21.15.67:0/1837818890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b15600640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b10145f80 0x7f2b100cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b16000640 1 --2- 172.21.15.67:0/1837818890 >> v2:172.21.15.187:3300/0 conn(0x7f2b1006c0a0 0x7f2b10160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b14c00640 1 -- 172.21.15.67:0/1837818890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b10145f80 msgr2=0x7f2b100cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b14c00640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b10145f80 0x7f2b100cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b14c00640 1 -- 172.21.15.67:0/1837818890 >> v2:172.21.15.187:3300/0 conn(0x7f2b1006c0a0 msgr2=0x7f2b10160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b14c00640 1 --2- 172.21.15.67:0/1837818890 >> v2:172.21.15.187:3300/0 conn(0x7f2b1006c0a0 0x7f2b10160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.382+0000 7f2b14c00640 1 -- 172.21.15.67:0/1837818890 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b10071e70 con 0x7f2b100cd750 2024-04-17T08:52:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.383+0000 7f2b14c00640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006bb60 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f2b100d03d0 tx=0x7f2afc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fb44000c6e65bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b0fe00640 1 -- 172.21.15.67:0/1837818890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc02e070 con 0x7f2b100cd750 2024-04-17T08:52:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b0fe00640 1 -- 172.21.15.67:0/1837818890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc037070 con 0x7f2b100cd750 2024-04-17T08:52:33.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b0fe00640 1 -- 172.21.15.67:0/1837818890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc031a00 con 0x7f2b100cd750 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 -- 172.21.15.67:0/1837818890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 msgr2=0x7f2b1006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006bb60 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f2b100d03d0 tx=0x7f2afc02b880 comp rx=0 tx=0).stop 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 -- 172.21.15.67:0/1837818890 shutdown_connections 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 --2- 172.21.15.67:0/1837818890 >> v2:172.21.15.187:3300/0 conn(0x7f2b1006c0a0 0x7f2b10160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006bb60 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 --2- 172.21.15.67:0/1837818890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b10145f80 0x7f2b100cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.384+0000 7f2b16a00640 1 -- 172.21.15.67:0/1837818890 >> 172.21.15.67:0/1837818890 conn(0x7f2b100b2190 msgr2=0x7f2b100b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.385+0000 7f2b16a00640 1 -- 172.21.15.67:0/1837818890 shutdown_connections 2024-04-17T08:52:33.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.385+0000 7f2b16a00640 1 -- 172.21.15.67:0/1837818890 wait complete. 2024-04-17T08:52:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.386+0000 7f2b16a00640 1 Processor -- start 2024-04-17T08:52:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.386+0000 7f2b16a00640 1 -- start start 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b1006c0a0 0x7f2b1006e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b10145f80 0x7f2b10144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b10072c10 con 0x7f2b100cd750 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b10072430 con 0x7f2b10145f80 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.387+0000 7f2b16a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b10072d90 con 0x7f2b1006c0a0 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51084/0 (socket says 172.21.15.67:51084) 2024-04-17T08:52:33.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 -- 172.21.15.67:0/624697847 learned_addr learned my addr 172.21.15.67:0/624697847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b16000640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b10145f80 0x7f2b10144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b15600640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b1006c0a0 0x7f2b1006e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 -- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b1006c0a0 msgr2=0x7f2b1006e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b1006c0a0 0x7f2b1006e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 -- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b10145f80 msgr2=0x7f2b10144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b10145f80 0x7f2b10144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.388+0000 7f2b14c00640 1 -- 172.21.15.67:0/624697847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b10140870 con 0x7f2b100cd750 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b14c00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc032670 con 0x7f2b100cd750 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2afc032850 con 0x7f2b100cd750 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc032c40 con 0x7f2b100cd750 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1015eb60 con 0x7f2b100cd750 2024-04-17T08:52:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.389+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b10140590 con 0x7f2b100cd750 2024-04-17T08:52:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.390+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b10068a10 con 0x7f2b100cd750 2024-04-17T08:52:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.392+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f2afc0453f0 con 0x7f2b100cd750 2024-04-17T08:52:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.393+0000 7f2b0d600640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 0x7f2af006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.393+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f2afc0adeb0 con 0x7f2b100cd750 2024-04-17T08:52:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.393+0000 7f2b15600640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 0x7f2af006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.394+0000 7f2b15600640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 0x7f2af006ef60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b04006fd0 tx=0x7f2b04008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.397+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2b10140870 con 0x7f2b100cd750 2024-04-17T08:52:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.606+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f2b1006be60 con 0x7f2b100cd750 2024-04-17T08:52:33.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.611+0000 7f2b0d600640 1 -- 172.21.15.67:0/624697847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v86) ==== 104+0+0 (secure 0 0 0) 0x7f2afc040070 con 0x7f2b100cd750 2024-04-17T08:52:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.614+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 msgr2=0x7f2af006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.614+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 0x7f2af006ef60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b04006fd0 tx=0x7f2b04008040 comp rx=0 tx=0).stop 2024-04-17T08:52:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.614+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 msgr2=0x7f2b1006ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.614+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 shutdown_connections 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2af006caa0 0x7f2af006ef60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b10145f80 0x7f2b10144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b100cd750 0x7f2b1006ec40 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 --2- 172.21.15.67:0/624697847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b1006c0a0 0x7f2b1006e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 >> 172.21.15.67:0/624697847 conn(0x7f2b100b2190 msgr2=0x7f2b100b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 shutdown_connections 2024-04-17T08:52:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.615+0000 7f2b16a00640 1 -- 172.21.15.67:0/624697847 wait complete. 2024-04-17T08:52:33.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:52:33.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.861+0000 7f26e7200640 1 Processor -- start 2024-04-17T08:52:33.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.861+0000 7f26e7200640 1 -- start start 2024-04-17T08:52:33.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.862+0000 7f26e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.862+0000 7f26e7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 0x7f26e806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26e8146360 con 0x7f26e8145f80 2024-04-17T08:52:33.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f26e80727a0 con 0x7f26e806c0a0 2024-04-17T08:52:33.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26e8071e70 con 0x7f26e80cd920 2024-04-17T08:52:33.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e5400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:56456/0 (socket says 172.21.15.67:56456) 2024-04-17T08:52:33.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.863+0000 7f26e5e00640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f26e8145f80 0x7f26e80cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:53686/0 (socket says 172.21.15.67:53686) 2024-04-17T08:52:33.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.864+0000 7f26e5400640 1 -- 172.21.15.67:0/1868685720 learned_addr learned my addr 172.21.15.67:0/1868685720 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:33.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.864+0000 7f26e6800640 1 --2- 172.21.15.67:0/1868685720 >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 0x7f26e806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.864+0000 7f26e4a00640 1 -- 172.21.15.67:0/1868685720 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253009695 0 0) 0x7f26e8146360 con 0x7f26e8145f80 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.865+0000 7f26e5400640 1 -- 172.21.15.67:0/1868685720 >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 msgr2=0x7f26e806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.865+0000 7f26e5400640 1 --2- 172.21.15.67:0/1868685720 >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 0x7f26e806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.865+0000 7f26e5400640 1 -- 172.21.15.67:0/1868685720 >> v1:172.21.15.187:6789/0 conn(0x7f26e8145f80 legacy=0x7f26e80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.865+0000 7f26e5400640 1 -- 172.21.15.67:0/1868685720 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e8072150 con 0x7f26e80cd920 2024-04-17T08:52:33.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.865+0000 7f26e6800640 1 --2- 172.21.15.67:0/1868685720 >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 0x7f26e806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:52:33.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.866+0000 7f26e5400640 1 --2- 172.21.15.67:0/1868685720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009e50 tx=0x7f26d402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce306db36225f460 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.867+0000 7f26e4a00640 1 -- 172.21.15.67:0/1868685720 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d402ca70 con 0x7f26e80cd920 2024-04-17T08:52:33.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.867+0000 7f26e4a00640 1 -- 172.21.15.67:0/1868685720 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d402cc10 con 0x7f26e80cd920 2024-04-17T08:52:33.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.867+0000 7f26e4a00640 1 -- 172.21.15.67:0/1868685720 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40343f0 con 0x7f26e80cd920 2024-04-17T08:52:33.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 -- 172.21.15.67:0/1868685720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 msgr2=0x7f26e806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 --2- 172.21.15.67:0/1868685720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009e50 tx=0x7f26d402bed0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 -- 172.21.15.67:0/1868685720 shutdown_connections 2024-04-17T08:52:33.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 --2- 172.21.15.67:0/1868685720 >> v2:172.21.15.187:3300/0 conn(0x7f26e806c0a0 0x7f26e806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 --2- 172.21.15.67:0/1868685720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e80cd920 0x7f26e806bb60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 -- 172.21.15.67:0/1868685720 >> 172.21.15.67:0/1868685720 conn(0x7f26e80b2190 msgr2=0x7f26e80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:33.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 -- 172.21.15.67:0/1868685720 shutdown_connections 2024-04-17T08:52:33.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.868+0000 7f26e7200640 1 -- 172.21.15.67:0/1868685720 wait complete. 2024-04-17T08:52:33.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.870+0000 7f26e7200640 1 Processor -- start 2024-04-17T08:52:33.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.870+0000 7f26e7200640 1 -- start start 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e806c0a0 0x7f26e8141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 0x7f26e813fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f26e8072db0 con 0x7f26e8145f80 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f26e8072c30 con 0x7f26e80cd920 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26e806c4a0 con 0x7f26e806c0a0 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51098/0 (socket says 172.21.15.67:51098) 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e6800640 1 -- 172.21.15.67:0/2593683107 learned_addr learned my addr 172.21.15.67:0/2593683107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.871+0000 7f26e5e00640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e806c0a0 0x7f26e8141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e5400640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 0x7f26e813fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 -- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e806c0a0 msgr2=0x7f26e8141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e806c0a0 0x7f26e8141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 -- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 msgr2=0x7f26e813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 0x7f26e813fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 -- 172.21.15.67:0/2593683107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e8140e00 con 0x7f26e8145f80 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e5400640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 0x7f26e813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e6800640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e930 tx=0x7f26e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0017070 con 0x7f26e8145f80 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e000ca00 con 0x7f26e8145f80 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e000cdd0 con 0x7f26e8145f80 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.872+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e8140360 con 0x7f26e8145f80 2024-04-17T08:52:33.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.873+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e8142fc0 con 0x7f26e8145f80 2024-04-17T08:52:33.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.874+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e8068a10 con 0x7f26e8145f80 2024-04-17T08:52:33.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.875+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f26e0013470 con 0x7f26e8145f80 2024-04-17T08:52:33.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.876+0000 7f26de000640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 0x7f26c806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:33.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.876+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f26e008bb40 con 0x7f26e8145f80 2024-04-17T08:52:33.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.876+0000 7f26e5e00640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 0x7f26c806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.878+0000 7f26e5e00640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 0x7f26c806ef60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f60 tx=0x7f26d8007d00 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:33.880+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f26e0059c40 con 0x7f26e8145f80 2024-04-17T08:52:34.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.100+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f26e80cf5a0 con 0x7f26e8145f80 2024-04-17T08:52:34.102 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:34.101+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:34.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.105+0000 7f26de000640 1 -- 172.21.15.67:0/2593683107 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v87) ==== 120+0+0 (secure 0 0 0) 0x7f26e0059540 con 0x7f26e8145f80 2024-04-17T08:52:34.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 msgr2=0x7f26c806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 0x7f26c806ef60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f60 tx=0x7f26d8007d00 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 msgr2=0x7f26e8143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e930 tx=0x7f26e000edf0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 shutdown_connections 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f26c806caa0 0x7f26c806ef60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8145f80 0x7f26e8143d80 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26e80cd920 0x7f26e813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 --2- 172.21.15.67:0/2593683107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26e806c0a0 0x7f26e8141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 >> 172.21.15.67:0/2593683107 conn(0x7f26e80b2190 msgr2=0x7f26e80b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.108+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 shutdown_connections 2024-04-17T08:52:34.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.109+0000 7f26e7200640 1 -- 172.21.15.67:0/2593683107 wait complete. 2024-04-17T08:52:34.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-04-17T08:52:34.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.353+0000 7ff46fe00640 1 Processor -- start 2024-04-17T08:52:34.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.353+0000 7ff46fe00640 1 -- start start 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470155030 0x7ff4700725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 0x7ff470072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 0x7ff4700b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff4700b4110 con 0x7ff470155030 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff47015e8e0 con 0x7ff470155410 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.354+0000 7ff46fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff47015ebc0 con 0x7ff47015f3c0 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470155030 0x7ff4700725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff470155030 0x7ff4700725f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51108/0 (socket says 172.21.15.67:51108) 2024-04-17T08:52:34.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 -- 172.21.15.67:0/3546930470 learned_addr learned my addr 172.21.15.67:0/3546930470 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:34.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff467e00640 1 --2- 172.21.15.67:0/3546930470 >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 0x7ff470072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46f400640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 0x7ff4700b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 -- 172.21.15.67:0/3546930470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 msgr2=0x7ff4700b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 0x7ff4700b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 -- 172.21.15.67:0/3546930470 >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 msgr2=0x7ff470072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 --2- 172.21.15.67:0/3546930470 >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 0x7ff470072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46ea00640 1 -- 172.21.15.67:0/3546930470 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47015eea0 con 0x7ff470155030 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff467e00640 1 --2- 172.21.15.67:0/3546930470 >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 0x7ff470072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.355+0000 7ff46f400640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 0x7ff4700b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:34.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.357+0000 7ff46ea00640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470155030 0x7ff4700725f0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7ff47006c980 tx=0x7ff46002ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136fc844d1cb8889 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.357+0000 7ff46e000640 1 -- 172.21.15.67:0/3546930470 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460030850 con 0x7ff470155030 2024-04-17T08:52:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.357+0000 7ff46e000640 1 -- 172.21.15.67:0/3546930470 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4600309f0 con 0x7ff470155030 2024-04-17T08:52:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.357+0000 7ff46e000640 1 -- 172.21.15.67:0/3546930470 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460030dc0 con 0x7ff470155030 2024-04-17T08:52:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 -- 172.21.15.67:0/3546930470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470155030 msgr2=0x7ff4700725f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470155030 0x7ff4700725f0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7ff47006c980 tx=0x7ff46002ca30 comp rx=0 tx=0).stop 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 -- 172.21.15.67:0/3546930470 shutdown_connections 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff47015f3c0 0x7ff4700b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 --2- 172.21.15.67:0/3546930470 >> v2:172.21.15.187:3300/0 conn(0x7ff470155410 0x7ff470072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 --2- 172.21.15.67:0/3546930470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff470155030 0x7ff4700725f0 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 -- 172.21.15.67:0/3546930470 >> 172.21.15.67:0/3546930470 conn(0x7ff4700b5f70 msgr2=0x7ff47006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 -- 172.21.15.67:0/3546930470 shutdown_connections 2024-04-17T08:52:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.358+0000 7ff46fe00640 1 -- 172.21.15.67:0/3546930470 wait complete. 2024-04-17T08:52:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.360+0000 7ff46fe00640 1 Processor -- start 2024-04-17T08:52:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.360+0000 7ff46fe00640 1 -- start start 2024-04-17T08:52:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.361+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff470155030 0x7ff47014ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.361+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff470155410 0x7ff47013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4700b3d30 con 0x7ff47015f3c0 2024-04-17T08:52:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff4700b3bb0 con 0x7ff470155410 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff4700b3eb0 con 0x7ff470155030 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51114/0 (socket says 172.21.15.67:51114) 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 -- 172.21.15.67:0/19877383 learned_addr learned my addr 172.21.15.67:0/19877383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46ea00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff470155030 0x7ff47014ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff467e00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff470155410 0x7ff47013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 -- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff470155030 msgr2=0x7ff47014ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff470155030 0x7ff47014ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 -- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff470155410 msgr2=0x7ff47013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff470155410 0x7ff47013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.362+0000 7ff46f400640 1 -- 172.21.15.67:0/19877383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47014d500 con 0x7ff47015f3c0 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff46f400640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7ff46800cd80 tx=0x7ff46800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46800f680 con 0x7ff47015f3c0 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46800f840 con 0x7ff47015f3c0 2024-04-17T08:52:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46800fc30 con 0x7ff47015f3c0 2024-04-17T08:52:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47014d7e0 con 0x7ff47015f3c0 2024-04-17T08:52:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.363+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47014dda0 con 0x7ff47015f3c0 2024-04-17T08:52:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.365+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff47014dfa0 con 0x7ff47015f3c0 2024-04-17T08:52:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.366+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7ff46800e3e0 con 0x7ff47015f3c0 2024-04-17T08:52:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.367+0000 7ff466a00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 0x7ff44c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.367+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7ff46808cc80 con 0x7ff47015f3c0 2024-04-17T08:52:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.368+0000 7ff46ea00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 0x7ff44c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:34.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.369+0000 7ff46ea00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 0x7ff44c06ef60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff460037000 tx=0x7ff460008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:34.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.372+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff47014dfa0 con 0x7ff47015f3c0 2024-04-17T08:52:34.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:34.654+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7ff470068a10 con 0x7ff47015f3c0 2024-04-17T08:52:35.165 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:35.166+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.636+0000 7ff466a00640 1 -- 172.21.15.67:0/19877383 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3286) ==== 187+0+0 (secure 0 0 0) 0x7ff468014030 con 0x7ff47015f3c0 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.639+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 msgr2=0x7ff44c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.639+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 0x7ff44c06ef60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff460037000 tx=0x7ff460008030 comp rx=0 tx=0).stop 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 msgr2=0x7ff47014f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7ff46800cd80 tx=0x7ff46800a650 comp rx=0 tx=0).stop 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 shutdown_connections 2024-04-17T08:52:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff44c06caa0 0x7ff44c06ef60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff47015f3c0 0x7ff47014f4d0 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff470155410 0x7ff47013fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 --2- 172.21.15.67:0/19877383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff470155030 0x7ff47014ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 >> 172.21.15.67:0/19877383 conn(0x7ff4700b5f70 msgr2=0x7ff470154300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 shutdown_connections 2024-04-17T08:52:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.640+0000 7ff46fe00640 1 -- 172.21.15.67:0/19877383 wait complete. 2024-04-17T08:52:35.664 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-04-17T08:52:35.664 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-04-17T08:52:35.664 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-04-17T08:52:35.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:35.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.894+0000 7fa536a00640 1 Processor -- start 2024-04-17T08:52:35.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.894+0000 7fa536a00640 1 -- start start 2024-04-17T08:52:35.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa536a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa530160280 0x7fa53006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa536a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 0x7fa530142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa536a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5300cd4a0 con 0x7fa53006b8c0 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa536a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa53015ce30 con 0x7fa530160280 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa536a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa5301607f0 con 0x7fa530141ed0 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa534c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa530160280 0x7fa53006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa534c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa530160280 0x7fa53006bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51120/0 (socket says 172.21.15.67:51120) 2024-04-17T08:52:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.896+0000 7fa534c00640 1 -- 172.21.15.67:0/1007202900 learned_addr learned my addr 172.21.15.67:0/1007202900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa534c00640 1 -- 172.21.15.67:0/1007202900 >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 msgr2=0x7fa530142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa536000640 1 --2- 172.21.15.67:0/1007202900 >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 0x7fa530142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa534c00640 1 --2- 172.21.15.67:0/1007202900 >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 0x7fa530142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa534c00640 1 -- 172.21.15.67:0/1007202900 >> v1:172.21.15.187:6789/0 conn(0x7fa53006b8c0 legacy=0x7fa5300cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:35.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa534c00640 1 -- 172.21.15.67:0/1007202900 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa530160ad0 con 0x7fa530160280 2024-04-17T08:52:35.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.897+0000 7fa536000640 1 --2- 172.21.15.67:0/1007202900 >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 0x7fa530142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:35.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.898+0000 7fa534c00640 1 --2- 172.21.15.67:0/1007202900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa53006bca0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fa5300b2120 tx=0x7fa51c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ccd338674b50df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.898+0000 7fa52fe00640 1 -- 172.21.15.67:0/1007202900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c035020 con 0x7fa530160280 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa52fe00640 1 -- 172.21.15.67:0/1007202900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa51c031870 con 0x7fa530160280 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa52fe00640 1 -- 172.21.15.67:0/1007202900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c031c40 con 0x7fa530160280 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 -- 172.21.15.67:0/1007202900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 msgr2=0x7fa53006bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 --2- 172.21.15.67:0/1007202900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa53006bca0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fa5300b2120 tx=0x7fa51c02b910 comp rx=0 tx=0).stop 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 -- 172.21.15.67:0/1007202900 shutdown_connections 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 --2- 172.21.15.67:0/1007202900 >> v2:172.21.15.187:3300/0 conn(0x7fa530141ed0 0x7fa530142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 --2- 172.21.15.67:0/1007202900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa53006bca0 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 -- 172.21.15.67:0/1007202900 >> 172.21.15.67:0/1007202900 conn(0x7fa530067d10 msgr2=0x7fa5300b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.899+0000 7fa536a00640 1 -- 172.21.15.67:0/1007202900 shutdown_connections 2024-04-17T08:52:35.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.900+0000 7fa536a00640 1 -- 172.21.15.67:0/1007202900 wait complete. 2024-04-17T08:52:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.902+0000 7fa536a00640 1 Processor -- start 2024-04-17T08:52:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.902+0000 7fa536a00640 1 -- start start 2024-04-17T08:52:35.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa53006b8c0 0x7fa53015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa530141ed0 0x7fa53013f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa530161090 con 0x7fa530160280 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa53006c1e0 con 0x7fa53006b8c0 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.903+0000 7fa536a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa53013f9b0 con 0x7fa530141ed0 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51128/0 (socket says 172.21.15.67:51128) 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 -- 172.21.15.67:0/4182015334 learned_addr learned my addr 172.21.15.67:0/4182015334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa535600640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa53006b8c0 0x7fa53015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa534c00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa530141ed0 0x7fa53013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 -- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa530141ed0 msgr2=0x7fa53013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:35.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa530141ed0 0x7fa53013f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 -- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa53006b8c0 msgr2=0x7fa53015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:35.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa53006b8c0 0x7fa53015f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:35.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.904+0000 7fa536000640 1 -- 172.21.15.67:0/4182015334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53015d2c0 con 0x7fa530160280 2024-04-17T08:52:35.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa536000640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa52800cf20 tx=0x7fa52800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52801a070 con 0x7fa530160280 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52800d840 con 0x7fa530160280 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52800dc30 con 0x7fa530160280 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5301f7bf0 con 0x7fa530160280 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.905+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5301f81b0 con 0x7fa530160280 2024-04-17T08:52:35.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.906+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa530068a10 con 0x7fa530160280 2024-04-17T08:52:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.907+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fa52801d070 con 0x7fa530160280 2024-04-17T08:52:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.908+0000 7fa52d600640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 0x7fa51006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.908+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7fa52808c840 con 0x7fa530160280 2024-04-17T08:52:35.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.909+0000 7fa535600640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 0x7fa51006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:35.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.910+0000 7fa535600640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 0x7fa51006ef60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa524004ea0 tx=0x7fa524009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:35.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:35.914+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa52805a940 con 0x7fa530160280 2024-04-17T08:52:36.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.134+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5300b5890 con 0x7fa530160280 2024-04-17T08:52:36.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.137+0000 7fa52d600640 1 -- 172.21.15.67:0/4182015334 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) ==== 74+0+14421 (secure 0 0 0) 0x7fa52805a240 con 0x7fa530160280 2024-04-17T08:52:36.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:36.136 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":262,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:32.613818+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6832/4106648933":"2024-04-18T08:52:29.514995+0000","172.21.15.187:0/1858361599":"2024-04-18T08:52:15.395572+0000","172.21.15.67:6833/4106648933":"2024-04-18T08:52:29.514995+0000","172.21.15.187:0/4229015777":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/4125581727":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/2806017671":"2024-04-18T08:52:15.395572+0000","172.21.15.67:6835/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:6833/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6832/986946406":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/895672932":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/1404113161":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/625860360":"2024-04-18T08:52:15.395572+0000","172.21.15.187:0/3128349368":"2024-04-18T08:52:15.395572+0000","172.21.15.187:6833/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:6832/2123602634":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/3628769219":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/1806888124":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2102410647":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2774796164":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/609167221":"2024-04-18T08:52:09.047891+0000","172.21.15.67:6834/3084367922":"2024-04-18T08:52:21.451169+0000","172.21.15.187:0/259125585":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/545321551":"2024-04-18T08:52:09.047891+0000","172.21.15.187:0/2338521910":"2024-04-18T08:52:09.047891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.139+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 msgr2=0x7fa51006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.139+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 0x7fa51006ef60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa524004ea0 tx=0x7fa524009040 comp rx=0 tx=0).stop 2024-04-17T08:52:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.139+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 msgr2=0x7fa5300b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.139+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa52800cf20 tx=0x7fa52800a7d0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 shutdown_connections 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa51006caa0 0x7fa51006ef60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa530160280 0x7fa5300b4290 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa530141ed0 0x7fa53013f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 --2- 172.21.15.67:0/4182015334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa53006b8c0 0x7fa53015f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 >> 172.21.15.67:0/4182015334 conn(0x7fa530067d10 msgr2=0x7fa5300cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 shutdown_connections 2024-04-17T08:52:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.140+0000 7fa536a00640 1 -- 172.21.15.67:0/4182015334 wait complete. 2024-04-17T08:52:36.158 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:52:36.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:52:36.238 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:36.239+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:36.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.389+0000 7fd023200640 1 Processor -- start 2024-04-17T08:52:36.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.389+0000 7fd023200640 1 -- start start 2024-04-17T08:52:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.390+0000 7fd023200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd01c146060 0x7fd01c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.390+0000 7fd023200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c160920 con 0x7fd01c0b4ac0 2024-04-17T08:52:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.390+0000 7fd023200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c160c00 con 0x7fd01c145b00 2024-04-17T08:52:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.390+0000 7fd023200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd01c160ee0 con 0x7fd01c146060 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd021400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd01c146060 0x7fd01c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd022800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd01c0b4ac0 0x7fd01c0b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52666/0 (socket says 172.21.15.67:52666) 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd022800640 1 -- 172.21.15.67:0/3026787650 learned_addr learned my addr 172.21.15.67:0/3026787650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd021400640 1 -- 172.21.15.67:0/3026787650 >> v1:172.21.15.187:6789/0 conn(0x7fd01c145b00 legacy=0x7fd01c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd021400640 1 -- 172.21.15.67:0/3026787650 >> v1:172.21.15.67:6789/0 conn(0x7fd01c0b4ac0 legacy=0x7fd01c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd020a00640 1 -- 172.21.15.67:0/3026787650 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686687723 0 0) 0x7fd01c160920 con 0x7fd01c0b4ac0 2024-04-17T08:52:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.391+0000 7fd021400640 1 -- 172.21.15.67:0/3026787650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c1611c0 con 0x7fd01c146060 2024-04-17T08:52:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.393+0000 7fd021400640 1 --2- 172.21.15.67:0/3026787650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c146060 0x7fd01c160340 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0097a0 tx=0x7fd00c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32b4eb413ce34b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.393+0000 7fd020a00640 1 -- 172.21.15.67:0/3026787650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c035020 con 0x7fd01c146060 2024-04-17T08:52:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.393+0000 7fd020a00640 1 -- 172.21.15.67:0/3026787650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00c031830 con 0x7fd01c146060 2024-04-17T08:52:36.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.393+0000 7fd020a00640 1 -- 172.21.15.67:0/3026787650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c031c00 con 0x7fd01c146060 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 -- 172.21.15.67:0/3026787650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c146060 msgr2=0x7fd01c160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 --2- 172.21.15.67:0/3026787650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c146060 0x7fd01c160340 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0097a0 tx=0x7fd00c02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 -- 172.21.15.67:0/3026787650 shutdown_connections 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 --2- 172.21.15.67:0/3026787650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c146060 0x7fd01c160340 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 -- 172.21.15.67:0/3026787650 >> 172.21.15.67:0/3026787650 conn(0x7fd01c0b23c0 msgr2=0x7fd01c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 -- 172.21.15.67:0/3026787650 shutdown_connections 2024-04-17T08:52:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.394+0000 7fd023200640 1 -- 172.21.15.67:0/3026787650 wait complete. 2024-04-17T08:52:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.396+0000 7fd023200640 1 Processor -- start 2024-04-17T08:52:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.396+0000 7fd023200640 1 -- start start 2024-04-17T08:52:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd01c0b4ac0 0x7fd01c141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd01c146060 0x7fd01c1e7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c0b4f10 con 0x7fd01c145b00 2024-04-17T08:52:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c0cfba0 con 0x7fd01c0b4ac0 2024-04-17T08:52:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd023200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd01c0b5090 con 0x7fd01c146060 2024-04-17T08:52:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd021400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd021400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51152/0 (socket says 172.21.15.67:51152) 2024-04-17T08:52:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.397+0000 7fd021400640 1 -- 172.21.15.67:0/2587632356 learned_addr learned my addr 172.21.15.67:0/2587632356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 -- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd01c146060 msgr2=0x7fd01c1e7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd01c146060 0x7fd01c1e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 -- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd01c0b4ac0 msgr2=0x7fd01c141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd01c0b4ac0 0x7fd01c141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 -- 172.21.15.67:0/2587632356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c1f51d0 con 0x7fd01c145b00 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd021400640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c040070 con 0x7fd01c145b00 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00c039a00 con 0x7fd01c145b00 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01c1f54b0 con 0x7fd01c145b00 2024-04-17T08:52:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.398+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c035040 con 0x7fd01c145b00 2024-04-17T08:52:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.399+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd01c1e62f0 con 0x7fd01c145b00 2024-04-17T08:52:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.400+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01c068a10 con 0x7fd01c145b00 2024-04-17T08:52:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.401+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fd00c04b060 con 0x7fd01c145b00 2024-04-17T08:52:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.402+0000 7fd016000640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 0x7fd00006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.402+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7fd00c0ad940 con 0x7fd01c145b00 2024-04-17T08:52:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.402+0000 7fd021e00640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 0x7fd00006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.404+0000 7fd021e00640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 0x7fd00006ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd010000f60 tx=0x7fd0100079b0 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.406+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd00c07ba20 con 0x7fd01c145b00 2024-04-17T08:52:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.618+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd01c068610 con 0x7fd01c145b00 2024-04-17T08:52:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.619+0000 7fd016000640 1 -- 172.21.15.67:0/2587632356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) ==== 93+0+10 (secure 0 0 0) 0x7fd00c07b320 con 0x7fd01c145b00 2024-04-17T08:52:36.619 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:52:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 msgr2=0x7fd00006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 0x7fd00006ef60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd010000f60 tx=0x7fd0100079b0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 msgr2=0x7fd01c1f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000c00 tx=0x7fd00c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 shutdown_connections 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fd00006caa0 0x7fd00006ef60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd01c146060 0x7fd01c1e7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd01c145b00 0x7fd01c1f47f0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 --2- 172.21.15.67:0/2587632356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd01c0b4ac0 0x7fd01c141940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 >> 172.21.15.67:0/2587632356 conn(0x7fd01c0b23c0 msgr2=0x7fd01c0b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 shutdown_connections 2024-04-17T08:52:36.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.622+0000 7fd023200640 1 -- 172.21.15.67:0/2587632356 wait complete. 2024-04-17T08:52:36.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-17T08:52:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.863+0000 7f06f7000640 1 Processor -- start 2024-04-17T08:52:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.864+0000 7f06f7000640 1 -- start start 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.865+0000 7f06f7000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f06f015ec60 0x7f06f00b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06f00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f7000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f00d2d30 con 0x7f06f00b6b30 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f7000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f06f00d49c0 con 0x7f06f015ec60 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06f00d3660 con 0x7f06f00cfae0 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f5200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06f00b6b30 0x7f06f00cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52674/0 (socket says 172.21.15.67:52674) 2024-04-17T08:52:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f5200640 1 -- 172.21.15.67:0/1662456891 learned_addr learned my addr 172.21.15.67:0/1662456891 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f6600640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06f00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> v2:172.21.15.187:3300/0 conn(0x7f06f015ec60 0x7f06f00b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.866+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831583430 0 0) 0x7f06f00d2d30 con 0x7f06f00b6b30 2024-04-17T08:52:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06d8003520 con 0x7f06f00b6b30 2024-04-17T08:52:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 969964696 0 0) 0x7f06d8003520 con 0x7f06f00b6b30 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 msgr2=0x7f06f00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06f00b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 >> v2:172.21.15.187:3300/0 conn(0x7f06f015ec60 msgr2=0x7f06f00b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 --2- 172.21.15.67:0/1662456891 >> v2:172.21.15.187:3300/0 conn(0x7f06f015ec60 0x7f06f00b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f00d3010 con 0x7f06f00b6b30 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f06dc001670 con 0x7f06f00b6b30 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06f6600640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06f00b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 >> v1:172.21.15.67:6789/0 conn(0x7f06f00b6b30 legacy=0x7f06f00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06d8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d8004d40 0x7f06d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06f00d49c0 con 0x7f06d8003a20 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06f00d3660 con 0x7f06d8004d40 2024-04-17T08:52:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06d800bb20 con 0x7f06f00cfae0 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.867+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> v2:172.21.15.187:3300/0 conn(0x7f06f015ec60 0x7f06f00b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06d8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 -- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 msgr2=0x7f06d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f6600640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d8004d40 0x7f06d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f00cfae0 0x7f06d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 -- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d8004d40 msgr2=0x7f06d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06d8004d40 0x7f06d80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.868+0000 7f06f5c00640 1 -- 172.21.15.67:0/1662456891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d8003520 con 0x7f06d8003a20 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.869+0000 7f06f5c00640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06d8003e40 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f06e40073b0 tx=0x7f06d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a200bc62349f4db1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.869+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4039070 con 0x7f06d8003a20 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.869+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e402cd90 con 0x7f06d8003a20 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.869+0000 7f06efe00640 1 -- 172.21.15.67:0/1662456891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4009c70 con 0x7f06d8003a20 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 -- 172.21.15.67:0/1662456891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 msgr2=0x7f06d8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 --2- 172.21.15.67:0/1662456891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06d8003e40 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f06e40073b0 tx=0x7f06d8005830 comp rx=0 tx=0).stop 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f6600640 1 -- 172.21.15.67:0/1662456891 reap_dead start 2024-04-17T08:52:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 -- 172.21.15.67:0/1662456891 shutdown_connections 2024-04-17T08:52:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 -- 172.21.15.67:0/1662456891 >> 172.21.15.67:0/1662456891 conn(0x7f06f00d15f0 msgr2=0x7f06f00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 -- 172.21.15.67:0/1662456891 shutdown_connections 2024-04-17T08:52:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.870+0000 7f06f7000640 1 -- 172.21.15.67:0/1662456891 wait complete. 2024-04-17T08:52:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.872+0000 7f06f7000640 1 Processor -- start 2024-04-17T08:52:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.872+0000 7f06f7000640 1 -- start start 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 0x7f06f0160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f0154d30 0x7f06f01551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06f015f040 con 0x7f06d8003a20 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06f00b6f70 con 0x7f06f0154030 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.873+0000 7f06f7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06f00d00c0 con 0x7f06f0154d30 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51158/0 (socket says 172.21.15.67:51158) 2024-04-17T08:52:36.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 -- 172.21.15.67:0/3205317499 learned_addr learned my addr 172.21.15.67:0/3205317499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 -- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f0154d30 msgr2=0x7f06f01551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f6600640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f0154d30 0x7f06f01551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5200640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 0x7f06f0160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f0154d30 0x7f06f01551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 -- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 msgr2=0x7f06f0160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 0x7f06f0160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5c00640 1 -- 172.21.15.67:0/3205317499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f0161450 con 0x7f06d8003a20 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.874+0000 7f06f5200640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 0x7f06f0160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06f5c00640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f06e40074d0 tx=0x7f06e4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4042020 con 0x7f06d8003a20 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e403a040 con 0x7f06d8003a20 2024-04-17T08:52:36.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4039070 con 0x7f06d8003a20 2024-04-17T08:52:36.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f0161730 con 0x7f06d8003a20 2024-04-17T08:52:36.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.875+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f0161160 con 0x7f06d8003a20 2024-04-17T08:52:36.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.876+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f00cc760 con 0x7f06d8003a20 2024-04-17T08:52:36.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.877+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f06e4045020 con 0x7f06d8003a20 2024-04-17T08:52:36.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.878+0000 7f06ed600640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 0x7f06d006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:36.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.878+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f06e40ade60 con 0x7f06d8003a20 2024-04-17T08:52:36.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.879+0000 7f06f5200640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 0x7f06d006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:36.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.880+0000 7f06f5200640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 0x7f06d006ef60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0017e0 tx=0x7f06dc009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:36.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:36.883+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f06e404c030 con 0x7f06d8003a20 2024-04-17T08:52:37.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.164+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7f06f0068a10 con 0x7f06d8003a20 2024-04-17T08:52:37.306 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:37.306+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:37.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.650+0000 7f06ed600640 1 -- 172.21.15.67:0/3205317499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3288) ==== 158+0+0 (secure 0 0 0) 0x7f06e407bf60 con 0x7f06d8003a20 2024-04-17T08:52:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 msgr2=0x7f06d006ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 0x7f06d006ef60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0017e0 tx=0x7f06dc009040 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 msgr2=0x7f06f014af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f06e40074d0 tx=0x7f06e4004ab0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 shutdown_connections 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f06d006caa0 0x7f06d006ef60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06f0154d30 0x7f06f01551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06f0154030 0x7f06f0160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 --2- 172.21.15.67:0/3205317499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d8003a20 0x7f06f014af70 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.654+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 >> 172.21.15.67:0/3205317499 conn(0x7f06f00d15f0 msgr2=0x7f06f0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.655+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 shutdown_connections 2024-04-17T08:52:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.655+0000 7f06f7000640 1 -- 172.21.15.67:0/3205317499 wait complete. 2024-04-17T08:52:37.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:52:37.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.910+0000 7fe38dc00640 1 Processor -- start 2024-04-17T08:52:37.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.910+0000 7fe38dc00640 1 -- start start 2024-04-17T08:52:37.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe38dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 0x7fe3880b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe38dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe388166a60 con 0x7fe3880b4580 2024-04-17T08:52:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe38dc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe388166d40 con 0x7fe388162da0 2024-04-17T08:52:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe38dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe388166810 con 0x7fe3880b41a0 2024-04-17T08:52:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe387400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3880b4580 0x7fe388162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52678/0 (socket says 172.21.15.67:52678) 2024-04-17T08:52:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.912+0000 7fe387400640 1 -- 172.21.15.67:0/3153627512 learned_addr learned my addr 172.21.15.67:0/3153627512 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe387e00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 0x7fe3880b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941193598 0 0) 0x7fe388166a60 con 0x7fe3880b4580 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe36c003520 con 0x7fe3880b4580 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3498260668 0 0) 0x7fe36c003520 con 0x7fe3880b4580 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 msgr2=0x7fe3880b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 0x7fe3880b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 >> v1:172.21.15.187:6789/0 conn(0x7fe388162da0 legacy=0x7fe38806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe388167020 con 0x7fe3880b4580 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe374001670 con 0x7fe3880b4580 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 >> v1:172.21.15.67:6789/0 conn(0x7fe3880b4580 legacy=0x7fe388162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.913+0000 7fe386a00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe36c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe386a00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 0x7fe36c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe388166810 con 0x7fe36c004800 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c00ba70 con 0x7fe36c0039e0 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe36c00bc70 con 0x7fe3880b41a0 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe36c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387e00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 -- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 msgr2=0x7fe36c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe3880b41a0 0x7fe36c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 -- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe36c0039e0 msgr2=0x7fe36c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe36c0039e0 0x7fe36c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.914+0000 7fe387400640 1 -- 172.21.15.67:0/3153627512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c003520 con 0x7fe36c004800 2024-04-17T08:52:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.915+0000 7fe387400640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe36c004c80 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c210 tx=0x7fe36c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3172dd5812b3b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:37.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.916+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374007d10 con 0x7fe36c004800 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.916+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe374030070 con 0x7fe36c004800 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.916+0000 7fe386a00640 1 -- 172.21.15.67:0/3153627512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374005d30 con 0x7fe36c004800 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 -- 172.21.15.67:0/3153627512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 msgr2=0x7fe36c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 --2- 172.21.15.67:0/3153627512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe36c004c80 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c210 tx=0x7fe36c005830 comp rx=0 tx=0).stop 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38d200640 1 -- 172.21.15.67:0/3153627512 reap_dead start 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 -- 172.21.15.67:0/3153627512 shutdown_connections 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 -- 172.21.15.67:0/3153627512 >> 172.21.15.67:0/3153627512 conn(0x7fe3880b2190 msgr2=0x7fe3880b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 -- 172.21.15.67:0/3153627512 shutdown_connections 2024-04-17T08:52:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.917+0000 7fe38dc00640 1 -- 172.21.15.67:0/3153627512 wait complete. 2024-04-17T08:52:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.919+0000 7fe38dc00640 1 Processor -- start 2024-04-17T08:52:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.919+0000 7fe38dc00640 1 -- start start 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe388156b30 0x7fe38815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe388158fa0 0x7fe388159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe38806f980 con 0x7fe36c004800 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe38806f800 con 0x7fe388158fa0 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe38dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe38806fb00 con 0x7fe388156b30 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe387e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe387e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51164/0 (socket says 172.21.15.67:51164) 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.920+0000 7fe387e00640 1 -- 172.21.15.67:0/2096421799 learned_addr learned my addr 172.21.15.67:0/2096421799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 -- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe388156b30 msgr2=0x7fe38815dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe388156b30 0x7fe38815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 -- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe388158fa0 msgr2=0x7fe388159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe388158fa0 0x7fe388159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 -- 172.21.15.67:0/2096421799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe388157f00 con 0x7fe36c004800 2024-04-17T08:52:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe387e00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fe378011610 tx=0x7fe378011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378004220 con 0x7fe36c004800 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3780043c0 con 0x7fe36c004800 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378015870 con 0x7fe36c004800 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.921+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3881581e0 con 0x7fe36c004800 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.922+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3881587a0 con 0x7fe36c004800 2024-04-17T08:52:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.923+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe388068a10 con 0x7fe36c004800 2024-04-17T08:52:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.924+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fe378015a10 con 0x7fe36c004800 2024-04-17T08:52:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.925+0000 7fe37fe00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 0x7fe36806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.925+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7fe37808c230 con 0x7fe36c004800 2024-04-17T08:52:37.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.926+0000 7fe387400640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 0x7fe36806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:37.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.927+0000 7fe387400640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 0x7fe36806ef60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe374034000 tx=0x7fe374039040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:37.930+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe378019030 con 0x7fe36c004800 2024-04-17T08:52:38.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.139+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe38806bde0 con 0x7fe36c004800 2024-04-17T08:52:38.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.145+0000 7fe37fe00640 1 -- 172.21.15.67:0/2096421799 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) ==== 122+0+0 (secure 0 0 0) 0x7fe37805a270 con 0x7fe36c004800 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 msgr2=0x7fe36806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 0x7fe36806ef60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe374034000 tx=0x7fe374039040 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 msgr2=0x7fe388157860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fe378011610 tx=0x7fe378011ad0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 shutdown_connections 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fe36806caa0 0x7fe36806ef60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe388158fa0 0x7fe388159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe388156b30 0x7fe38815dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 --2- 172.21.15.67:0/2096421799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe36c004800 0x7fe388157860 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 >> 172.21.15.67:0/2096421799 conn(0x7fe3880b2190 msgr2=0x7fe3880b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 shutdown_connections 2024-04-17T08:52:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.148+0000 7fe38dc00640 1 -- 172.21.15.67:0/2096421799 wait complete. 2024-04-17T08:52:38.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:52:38.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:52:38.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:52:38.379 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:38.380+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:38.412 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:52:38.412 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:38.412 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:38.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:52:38.474 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:52:38.474 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:38.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:52:38.530 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:38.530 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:38.591 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:52:38.591 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:52:38.591 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:38.649 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:52:38.650 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:52:38.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:38.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.879+0000 7f40fce00640 1 Processor -- start 2024-04-17T08:52:38.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.879+0000 7f40fce00640 1 -- start start 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40fce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40f80b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40fce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f80ce1e0 con 0x7f40f80b2bd0 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40fce00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f80ce4c0 con 0x7f40f806bfd0 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40fce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40f80ce7a0 con 0x7f40f815eb80 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40efe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f40f80b2bd0 0x7f40f806b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52692/0 (socket says 172.21.15.67:52692) 2024-04-17T08:52:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.881+0000 7f40efe00640 1 -- 172.21.15.67:0/8455192 learned_addr learned my addr 172.21.15.67:0/8455192 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:38.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164201959 0 0) 0x7f40f80ce1e0 con 0x7f40f80b2bd0 2024-04-17T08:52:38.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40dc003520 con 0x7f40f80b2bd0 2024-04-17T08:52:38.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5e00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40f80b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3070036213 0 0) 0x7f40dc003520 con 0x7f40f80b2bd0 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 msgr2=0x7f40f80b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40f80b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 >> v1:172.21.15.187:6789/0 conn(0x7f40f806bfd0 legacy=0x7f40f80b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f80cea80 con 0x7f40f80b2bd0 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f40e0001670 con 0x7f40f80b2bd0 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 >> v1:172.21.15.67:6789/0 conn(0x7f40f80b2bd0 legacy=0x7f40f806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40dc0039e0 0x7f40dc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc004800 0x7f40dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40f80ce7a0 con 0x7f40dc004800 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc00baf0 con 0x7f40dc0039e0 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.882+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40dc00bcf0 con 0x7f40f815eb80 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc004800 0x7f40dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40f5e00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40dc0039e0 0x7f40dc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40f6800640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 -- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 msgr2=0x7f40dc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40f815eb80 0x7f40dc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 -- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40dc0039e0 msgr2=0x7f40dc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40dc0039e0 0x7f40dc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.883+0000 7f40efe00640 1 -- 172.21.15.67:0/8455192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc003520 con 0x7f40dc004800 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.884+0000 7f40efe00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc004800 0x7f40dc004c80 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f40e000c210 tx=0x7f40dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6722349850ceadf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0007d30 con 0x7f40dc004800 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e0030070 con 0x7f40dc004800 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40f5400640 1 -- 172.21.15.67:0/8455192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e002cd60 con 0x7f40dc004800 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40fce00640 1 -- 172.21.15.67:0/8455192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc004800 msgr2=0x7f40dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40fce00640 1 --2- 172.21.15.67:0/8455192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc004800 0x7f40dc004c80 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f40e000c210 tx=0x7f40dc005830 comp rx=0 tx=0).stop 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40f6800640 1 -- 172.21.15.67:0/8455192 reap_dead start 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40fce00640 1 -- 172.21.15.67:0/8455192 shutdown_connections 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.885+0000 7f40fce00640 1 -- 172.21.15.67:0/8455192 >> 172.21.15.67:0/8455192 conn(0x7f40f8073360 msgr2=0x7f40f809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.886+0000 7f40fce00640 1 -- 172.21.15.67:0/8455192 shutdown_connections 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.886+0000 7f40fce00640 1 -- 172.21.15.67:0/8455192 wait complete. 2024-04-17T08:52:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.887+0000 7f40fce00640 1 Processor -- start 2024-04-17T08:52:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 -- start start 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40dc004800 0x7f40f80cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40f80b6260 0x7f40f80b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40f80b2810 con 0x7f40f814c360 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f40f80b2690 con 0x7f40f80b6260 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.888+0000 7f40fce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f40f80b2990 con 0x7f40dc004800 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51186/0 (socket says 172.21.15.67:51186) 2024-04-17T08:52:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 -- 172.21.15.67:0/1115247629 learned_addr learned my addr 172.21.15.67:0/1115247629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40f5e00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40dc004800 0x7f40f80cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40f6800640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40f80b6260 0x7f40f80b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 -- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40dc004800 msgr2=0x7f40f80cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40dc004800 0x7f40f80cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 -- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40f80b6260 msgr2=0x7f40f80b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40f80b6260 0x7f40f80b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.889+0000 7f40efe00640 1 -- 172.21.15.67:0/1115247629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f80b4260 con 0x7f40f814c360 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40efe00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f40e0001970 tx=0x7f40e0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0004bf0 con 0x7f40f814c360 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e0004d90 con 0x7f40f814c360 2024-04-17T08:52:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0007290 con 0x7f40f814c360 2024-04-17T08:52:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f80c8e00 con 0x7f40f814c360 2024-04-17T08:52:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.890+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f80c88e0 con 0x7f40f814c360 2024-04-17T08:52:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.891+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f8068b90 con 0x7f40f814c360 2024-04-17T08:52:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.892+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f40e0048020 con 0x7f40f814c360 2024-04-17T08:52:38.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.893+0000 7f40ee000640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 0x7f40d406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.894+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f40e00ad490 con 0x7f40f814c360 2024-04-17T08:52:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.894+0000 7f40f5e00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 0x7f40d406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:38.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.895+0000 7f40f5e00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 0x7f40d406ef90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40e8006fd0 tx=0x7f40e8008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:38.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:38.902+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f40e007b560 con 0x7f40f814c360 2024-04-17T08:52:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.161+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40f80cda00 con 0x7f40f814c360 2024-04-17T08:52:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.162+0000 7f40ee000640 1 -- 172.21.15.67:0/1115247629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+3879 (secure 0 0 0) 0x7f40e007ae60 con 0x7f40f814c360 2024-04-17T08:52:39.162 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:39.162 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":136,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9181,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3834744177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3834744177},{"type":"v1","addr":"172.21.15.187:6836","nonce":3834744177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":9184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3641856351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3641856351},{"type":"v1","addr":"172.21.15.187:6837","nonce":3641856351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121},{"gid":12360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/991489299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":991489299},{"type":"v1","addr":"172.21.15.67:6835","nonce":991489299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":134},{"gid":12450,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2319727491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2319727491},{"type":"v1","addr":"172.21.15.67:6833","nonce":2319727491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[]} 2024-04-17T08:52:39.162 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 136 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 msgr2=0x7f40d406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 0x7f40d406ef90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40e8006fd0 tx=0x7f40e8008040 comp rx=0 tx=0).stop 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 msgr2=0x7f40f80b5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f40e0001970 tx=0x7f40e0001a40 comp rx=0 tx=0).stop 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 shutdown_connections 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f40d406cad0 0x7f40d406ef90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f40f80b6260 0x7f40f80b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f814c360 0x7f40f80b5d20 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 --2- 172.21.15.67:0/1115247629 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f40dc004800 0x7f40f80cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 >> 172.21.15.67:0/1115247629 conn(0x7f40f8073360 msgr2=0x7f40f80cbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:39.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 shutdown_connections 2024-04-17T08:52:39.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.165+0000 7f40fce00640 1 -- 172.21.15.67:0/1115247629 wait complete. 2024-04-17T08:52:39.184 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:52:39.184 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:52:39.184 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:52:39.184 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:52:39.184 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:52:39.184 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:52:39.184 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:52:39.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:52:39.185 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:52:39.185 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:52:39.185 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:52:39.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:52:39.185 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:52:39.185 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:52:39.186 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:52:39.186 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:52:39.206 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:52:39.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:52:39.207 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:52:39.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:52:39.209 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:52:39.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:52:39.212 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:52:39.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:52:39.257 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:52:39.257 DEBUG:teuthology.parallel:result is None 2024-04-17T08:52:39.259 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:52:39.259 DEBUG:teuthology.parallel:result is None 2024-04-17T08:52:39.263 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:52:39.263 DEBUG:teuthology.parallel:result is None 2024-04-17T08:52:39.265 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:52:39.266 DEBUG:teuthology.parallel:result is None 2024-04-17T08:52:39.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:52:39.404 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:39.405+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:39.421 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:52:39.421 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:52:39.426 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:52:39.427 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:52:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.463+0000 7f6e72000640 1 Processor -- start 2024-04-17T08:52:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.463+0000 7f6e72000640 1 -- start start 2024-04-17T08:52:39.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e72000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e72000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c0d3520 0x7f6e6c0b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e72000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e6c0d4140 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e72000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6e6c0b83b0 con 0x7f6e6c15ee20 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e72000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6c0d1d20 con 0x7f6e6c0d3520 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.464+0000 7f6e70c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51226/0 (socket says 172.21.15.67:51226) 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 -- 172.21.15.67:0/495808117 learned_addr learned my addr 172.21.15.67:0/495808117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e71600640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c0d3520 0x7f6e6c0b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 -- 172.21.15.67:0/495808117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c0d3520 msgr2=0x7f6e6c0b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:39.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c0d3520 0x7f6e6c0b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 -- 172.21.15.67:0/495808117 >> v1:172.21.15.187:6789/0 conn(0x7f6e6c0d18e0 legacy=0x7f6e6c0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.465+0000 7f6e70c00640 1 -- 172.21.15.67:0/495808117 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6c0d3b80 con 0x7f6e6c15ee20 2024-04-17T08:52:39.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.466+0000 7f6e70c00640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009870 tx=0x7f6e6002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193d9e2bd3c09843 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:39.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e6b400640 1 -- 172.21.15.67:0/495808117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60037070 con 0x7f6e6c15ee20 2024-04-17T08:52:39.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e6b400640 1 -- 172.21.15.67:0/495808117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e60004230 con 0x7f6e6c15ee20 2024-04-17T08:52:39.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e6b400640 1 -- 172.21.15.67:0/495808117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60003610 con 0x7f6e6c15ee20 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 -- 172.21.15.67:0/495808117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c15ee20 msgr2=0x7f6e6c0d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009870 tx=0x7f6e6002b7e0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 -- 172.21.15.67:0/495808117 shutdown_connections 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c0d3520 0x7f6e6c0b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 --2- 172.21.15.67:0/495808117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c15ee20 0x7f6e6c0d13a0 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 -- 172.21.15.67:0/495808117 >> 172.21.15.67:0/495808117 conn(0x7f6e6c06d3c0 msgr2=0x7f6e6c0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 -- 172.21.15.67:0/495808117 shutdown_connections 2024-04-17T08:52:39.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.467+0000 7f6e72000640 1 -- 172.21.15.67:0/495808117 wait complete. 2024-04-17T08:52:39.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.469+0000 7f6e72000640 1 Processor -- start 2024-04-17T08:52:39.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.469+0000 7f6e72000640 1 -- start start 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e6c0d3520 0x7f6e6c0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c15ee20 0x7f6e6c15f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c15f200 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c0b8040 con 0x7f6e6c0d3520 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.470+0000 7f6e72000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6c0b8d80 con 0x7f6e6c15ee20 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51232/0 (socket says 172.21.15.67:51232) 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 -- 172.21.15.67:0/3181469294 learned_addr learned my addr 172.21.15.67:0/3181469294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e6be00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e6c0d3520 0x7f6e6c0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e71600640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c15ee20 0x7f6e6c15f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 -- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c15ee20 msgr2=0x7f6e6c15f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c15ee20 0x7f6e6c15f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 -- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e6c0d3520 msgr2=0x7f6e6c0c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:39.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e6c0d3520 0x7f6e6c0c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:39.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.471+0000 7f6e70c00640 1 -- 172.21.15.67:0/3181469294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6c0cb4a0 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e70c00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f6e60000c00 tx=0x7f6e600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:39.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60041070 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e60030510 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60037040 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6c0ca920 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.472+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6c0caee0 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.473+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e6c068a10 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.474+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f6e60044070 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.475+0000 7f6e68c00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 0x7f6e4c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:39.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.475+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5917+0+0 (secure 0 0 0) 0x7f6e600ada50 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.476+0000 7f6e6be00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 0x7f6e4c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:39.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.477+0000 7f6e6be00640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 0x7f6e4c06ef60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0d4af0 tx=0x7f6e58009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.481+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6e6007b470 con 0x7f6e6c0d18e0 2024-04-17T08:52:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:39.716+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6e6c0d4b20 con 0x7f6e6c0d18e0 2024-04-17T08:52:40.487 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:40.487+0000 7f3285e00640 -1 client.8375: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.9379:8ff5c105-a514-4710-b740-6876b0ff7e05:v1:172.21.15.187:0/4229015777] 2024-04-17T08:52:40.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.709+0000 7f6e68c00640 1 -- 172.21.15.67:0/3181469294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v267) ==== 97+0+0 (secure 0 0 0) 0x7f6e6007f770 con 0x7f6e6c0d18e0 2024-04-17T08:52:40.709 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:52:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 msgr2=0x7f6e4c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 0x7f6e4c06ef60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0d4af0 tx=0x7f6e58009040 comp rx=0 tx=0).stop 2024-04-17T08:52:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 msgr2=0x7f6e6c146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f6e60000c00 tx=0x7f6e600046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 shutdown_connections 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f6e4c06caa0 0x7f6e4c06ef60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6e6c15ee20 0x7f6e6c15f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e6c0d3520 0x7f6e6c0c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 --2- 172.21.15.67:0/3181469294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0d18e0 0x7f6e6c146b90 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 >> 172.21.15.67:0/3181469294 conn(0x7f6e6c06d3c0 msgr2=0x7f6e6c0d5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.714+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 shutdown_connections 2024-04-17T08:52:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.715+0000 7f6e72000640 1 -- 172.21.15.67:0/3181469294 wait complete. 2024-04-17T08:52:40.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:52:40.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.951+0000 7fa63b200640 1 Processor -- start 2024-04-17T08:52:40.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.951+0000 7fa63b200640 1 -- start start 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa63c15e9a0 0x7fa63c0d4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 0x7fa63c0d1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c15fb20 0x7fa63c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa63c15ed80 con 0x7fa63c15e9a0 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa63c0cf730 con 0x7fa63c0d04f0 2024-04-17T08:52:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa63b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c0cfa10 con 0x7fa63c15fb20 2024-04-17T08:52:40.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa639e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa63c15e9a0 0x7fa63c0d4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa639e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa63c15e9a0 0x7fa63c0d4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51242/0 (socket says 172.21.15.67:51242) 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.953+0000 7fa639e00640 1 -- 172.21.15.67:0/3317176823 learned_addr learned my addr 172.21.15.67:0/3317176823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa631400640 1 --2- 172.21.15.67:0/3317176823 >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 0x7fa63c0d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa63a800640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c15fb20 0x7fa63c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa639e00640 1 -- 172.21.15.67:0/3317176823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c15fb20 msgr2=0x7fa63c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa639e00640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c15fb20 0x7fa63c0d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa639e00640 1 -- 172.21.15.67:0/3317176823 >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 msgr2=0x7fa63c0d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa639e00640 1 --2- 172.21.15.67:0/3317176823 >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 0x7fa63c0d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa639e00640 1 -- 172.21.15.67:0/3317176823 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c0cfcf0 con 0x7fa63c15e9a0 2024-04-17T08:52:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.954+0000 7fa631400640 1 --2- 172.21.15.67:0/3317176823 >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 0x7fa63c0d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:40.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.955+0000 7fa639e00640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0d4e30 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fa63c08cfc0 tx=0x7fa62c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c457d9cbb703a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:40.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa639400640 1 -- 172.21.15.67:0/3317176823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c035020 con 0x7fa63c15e9a0 2024-04-17T08:52:40.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa639400640 1 -- 172.21.15.67:0/3317176823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c031870 con 0x7fa63c15e9a0 2024-04-17T08:52:40.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa639400640 1 -- 172.21.15.67:0/3317176823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c031c40 con 0x7fa63c15e9a0 2024-04-17T08:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 -- 172.21.15.67:0/3317176823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15e9a0 msgr2=0x7fa63c0d4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0d4e30 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fa63c08cfc0 tx=0x7fa62c02b910 comp rx=0 tx=0).stop 2024-04-17T08:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 -- 172.21.15.67:0/3317176823 shutdown_connections 2024-04-17T08:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c15fb20 0x7fa63c0d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 --2- 172.21.15.67:0/3317176823 >> v2:172.21.15.187:3300/0 conn(0x7fa63c0d04f0 0x7fa63c0d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 --2- 172.21.15.67:0/3317176823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0d4e30 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.956+0000 7fa63b200640 1 -- 172.21.15.67:0/3317176823 >> 172.21.15.67:0/3317176823 conn(0x7fa63c08bff0 msgr2=0x7fa63c0d4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.957+0000 7fa63b200640 1 -- 172.21.15.67:0/3317176823 shutdown_connections 2024-04-17T08:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.957+0000 7fa63b200640 1 -- 172.21.15.67:0/3317176823 wait complete. 2024-04-17T08:52:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.959+0000 7fa63b200640 1 Processor -- start 2024-04-17T08:52:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.959+0000 7fa63b200640 1 -- start start 2024-04-17T08:52:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c0d04f0 0x7fa63c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c0cf3e0 con 0x7fa63c15fb20 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c0d5370 con 0x7fa63c15e9a0 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa63c162490 con 0x7fa63c0d04f0 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51258/0 (socket says 172.21.15.67:51258) 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa63a800640 1 -- 172.21.15.67:0/3408442624 learned_addr learned my addr 172.21.15.67:0/3408442624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.960+0000 7fa639e00640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c0d04f0 0x7fa63c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa631400640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 -- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c0d04f0 msgr2=0x7fa63c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c0d04f0 0x7fa63c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 -- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa63c15e9a0 msgr2=0x7fa63c0c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 -- 172.21.15.67:0/3408442624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c0cd0e0 con 0x7fa63c15fb20 2024-04-17T08:52:40.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa63a800640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fa634007e60 tx=0x7fa63400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.961+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634014420 con 0x7fa63c15fb20 2024-04-17T08:52:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.962+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa634019070 con 0x7fa63c15fb20 2024-04-17T08:52:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.962+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634014820 con 0x7fa63c15fb20 2024-04-17T08:52:40.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.962+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c0cd3c0 con 0x7fa63c15fb20 2024-04-17T08:52:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.962+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c0cd980 con 0x7fa63c15fb20 2024-04-17T08:52:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.963+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c068a10 con 0x7fa63c15fb20 2024-04-17T08:52:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.964+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7fa6340149c0 con 0x7fa63c15fb20 2024-04-17T08:52:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.965+0000 7fa632a00640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 0x7fa61806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.965+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 4885+0+0 (secure 0 0 0) 0x7fa63408c3a0 con 0x7fa63c15fb20 2024-04-17T08:52:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.966+0000 7fa639e00640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 0x7fa61806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.967+0000 7fa639e00640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 0x7fa61806ef60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03c040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:40.971+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa63405a880 con 0x7fa63c15fb20 2024-04-17T08:52:41.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.261+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa63c0cea30 con 0x7fa63c15fb20 2024-04-17T08:52:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.262+0000 7fa632a00640 1 -- 172.21.15.67:0/3408442624 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) ==== 80+0+5304 (secure 0 0 0) 0x7fa63405a180 con 0x7fa63c15fb20 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:52:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:52:41.264 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.266 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:52:41.267 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:52:41.269 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:52:41.271 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:52:41.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:52:41.274 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.264+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 msgr2=0x7fa61806ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.264+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 0x7fa61806ef60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa62c036000 tx=0x7fa62c03c040 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 msgr2=0x7fa63c0b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fa634007e60 tx=0x7fa63400e4b0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 shutdown_connections 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa61806caa0 0x7fa61806ef60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa63c15fb20 0x7fa63c0b97f0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa63c15e9a0 0x7fa63c0c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 --2- 172.21.15.67:0/3408442624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa63c0d04f0 0x7fa63c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 >> 172.21.15.67:0/3408442624 conn(0x7fa63c08bff0 msgr2=0x7fa63c1603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 shutdown_connections 2024-04-17T08:52:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.265+0000 7fa63b200640 1 -- 172.21.15.67:0/3408442624 wait complete. 2024-04-17T08:52:41.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.508+0000 7f9533200640 1 Processor -- start 2024-04-17T08:52:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.508+0000 7f9533200640 1 -- start start 2024-04-17T08:52:41.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9533200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953406b8c0 0x7f95340cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9533200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 0x7f9534142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9533200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95340cd4a0 con 0x7f9534160280 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9533200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f953415ce30 con 0x7f953406b8c0 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9533200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95341607f0 con 0x7f9534141ed0 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9531e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953406b8c0 0x7f95340cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9531e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953406b8c0 0x7f95340cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51272/0 (socket says 172.21.15.67:51272) 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9531400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9534160280 0x7f953406bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52700/0 (socket says 172.21.15.67:52700) 2024-04-17T08:52:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.510+0000 7f9531e00640 1 -- 172.21.15.67:0/2019777244 learned_addr learned my addr 172.21.15.67:0/2019777244 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9532800640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 0x7f9534142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9531e00640 1 -- 172.21.15.67:0/2019777244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 msgr2=0x7f9534142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9531e00640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 0x7f9534142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9531e00640 1 -- 172.21.15.67:0/2019777244 >> v1:172.21.15.67:6789/0 conn(0x7f9534160280 legacy=0x7f953406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9531e00640 1 -- 172.21.15.67:0/2019777244 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9534160ad0 con 0x7f953406b8c0 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.511+0000 7f9532800640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 0x7f9534142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.512+0000 7f9531e00640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953406b8c0 0x7f95340cd8b0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f9524009870 tx=0x7f952402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c128f759cc6da2fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.512+0000 7f9530a00640 1 -- 172.21.15.67:0/2019777244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524034070 con 0x7f953406b8c0 2024-04-17T08:52:41.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.512+0000 7f9530a00640 1 -- 172.21.15.67:0/2019777244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9524030840 con 0x7f953406b8c0 2024-04-17T08:52:41.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.512+0000 7f9530a00640 1 -- 172.21.15.67:0/2019777244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524030c10 con 0x7f953406b8c0 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 -- 172.21.15.67:0/2019777244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953406b8c0 msgr2=0x7f95340cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953406b8c0 0x7f95340cd8b0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f9524009870 tx=0x7f952402b980 comp rx=0 tx=0).stop 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 -- 172.21.15.67:0/2019777244 shutdown_connections 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9534141ed0 0x7f9534142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 --2- 172.21.15.67:0/2019777244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953406b8c0 0x7f95340cd8b0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 -- 172.21.15.67:0/2019777244 >> 172.21.15.67:0/2019777244 conn(0x7f9534067d10 msgr2=0x7f95340b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 -- 172.21.15.67:0/2019777244 shutdown_connections 2024-04-17T08:52:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.513+0000 7f9533200640 1 -- 172.21.15.67:0/2019777244 wait complete. 2024-04-17T08:52:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.515+0000 7f9533200640 1 Processor -- start 2024-04-17T08:52:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.515+0000 7f9533200640 1 -- start start 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 0x7f953406ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9534160280 0x7f95340b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9534161090 con 0x7f9534141ed0 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f95340cddf0 con 0x7f9534160280 2024-04-17T08:52:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9533200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f953413f9b0 con 0x7f953406b8c0 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9531400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9531400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51276/0 (socket says 172.21.15.67:51276) 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.516+0000 7f9531400640 1 -- 172.21.15.67:0/4183298751 learned_addr learned my addr 172.21.15.67:0/4183298751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531e00640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 0x7f953406ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 -- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 msgr2=0x7f953406ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 0x7f953406ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 -- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9534160280 msgr2=0x7f95340b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9534160280 0x7f95340b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 -- 172.21.15.67:0/4183298751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953406e3d0 con 0x7f9534141ed0 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531e00640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 0x7f953406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9531400640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f952000c3f0 tx=0x7f952000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95200074e0 con 0x7f9534141ed0 2024-04-17T08:52:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9520007680 con 0x7f9534141ed0 2024-04-17T08:52:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520004a00 con 0x7f9534141ed0 2024-04-17T08:52:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.517+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f953406e6b0 con 0x7f9534141ed0 2024-04-17T08:52:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.518+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f953406ec70 con 0x7f9534141ed0 2024-04-17T08:52:41.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.519+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9534068a10 con 0x7f9534141ed0 2024-04-17T08:52:41.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.520+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f952001a020 con 0x7f9534141ed0 2024-04-17T08:52:41.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.521+0000 7f952a000640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 0x7f951406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.521+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 4885+0+0 (secure 0 0 0) 0x7f952008b360 con 0x7f9534141ed0 2024-04-17T08:52:41.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.521+0000 7f9531e00640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 0x7f951406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.523+0000 7f9531e00640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 0x7f951406ef60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9524036000 tx=0x7f952403b040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.525+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9520010300 con 0x7f9534141ed0 2024-04-17T08:52:41.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.740+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f953406c0b0 con 0x7f9534141ed0 2024-04-17T08:52:41.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.742+0000 7f952a000640 1 -- 172.21.15.67:0/4183298751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) ==== 74+0+12959 (secure 0 0 0) 0x7f9520059840 con 0x7f9534141ed0 2024-04-17T08:52:41.741 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:41.742 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":268,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:41.710247+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 msgr2=0x7f951406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 0x7f951406ef60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9524036000 tx=0x7f952403b040 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 msgr2=0x7f95340b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7f952000c3f0 tx=0x7f952000c8b0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 shutdown_connections 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951406caa0 0x7f951406ef60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9534160280 0x7f95340b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9534141ed0 0x7f95340b4de0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 --2- 172.21.15.67:0/4183298751 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953406b8c0 0x7f953406ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 >> 172.21.15.67:0/4183298751 conn(0x7f9534067d10 msgr2=0x7f95340d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 shutdown_connections 2024-04-17T08:52:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.745+0000 7f9533200640 1 -- 172.21.15.67:0/4183298751 wait complete. 2024-04-17T08:52:41.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:52:41.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:52:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.974+0000 7f7c07e00640 1 Processor -- start 2024-04-17T08:52:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.974+0000 7f7c07e00640 1 -- start start 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.975+0000 7f7c07e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c0815ec20 0x7f7c080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.975+0000 7f7c07e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7c080b3b50 0x7f7c0806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.975+0000 7f7c07e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c080d4f70 con 0x7f7c0806cf80 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.975+0000 7f7c07e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7c080d5250 con 0x7f7c0815ec20 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.975+0000 7f7c07e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7c080cf4d0 con 0x7f7c080b3b50 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c0815ec20 0x7f7c080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c0815ec20 0x7f7c080d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51290/0 (socket says 172.21.15.67:51290) 2024-04-17T08:52:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 -- 172.21.15.67:0/801269134 learned_addr learned my addr 172.21.15.67:0/801269134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 -- 172.21.15.67:0/801269134 >> v2:172.21.15.187:3300/0 conn(0x7f7c080b3b50 msgr2=0x7f7c0806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 --2- 172.21.15.67:0/801269134 >> v2:172.21.15.187:3300/0 conn(0x7f7c080b3b50 0x7f7c0806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 -- 172.21.15.67:0/801269134 >> v1:172.21.15.187:6789/0 conn(0x7f7c0806cf80 legacy=0x7f7c080cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.976+0000 7f7c06a00640 1 -- 172.21.15.67:0/801269134 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c080cf7b0 con 0x7f7c0815ec20 2024-04-17T08:52:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.978+0000 7f7c06a00640 1 --2- 172.21.15.67:0/801269134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c080d22b0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f7c080b71e0 tx=0x7f7bf002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b649353a2a4f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.978+0000 7f7c05600640 1 -- 172.21.15.67:0/801269134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0038070 con 0x7f7c0815ec20 2024-04-17T08:52:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.978+0000 7f7c05600640 1 -- 172.21.15.67:0/801269134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf002f050 con 0x7f7c0815ec20 2024-04-17T08:52:41.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.978+0000 7f7c05600640 1 -- 172.21.15.67:0/801269134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0031970 con 0x7f7c0815ec20 2024-04-17T08:52:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 -- 172.21.15.67:0/801269134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 msgr2=0x7f7c080d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 --2- 172.21.15.67:0/801269134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c080d22b0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f7c080b71e0 tx=0x7f7bf002b7f0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 -- 172.21.15.67:0/801269134 shutdown_connections 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 --2- 172.21.15.67:0/801269134 >> v2:172.21.15.187:3300/0 conn(0x7f7c080b3b50 0x7f7c0806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 --2- 172.21.15.67:0/801269134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c080d22b0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 -- 172.21.15.67:0/801269134 >> 172.21.15.67:0/801269134 conn(0x7f7c0808bff0 msgr2=0x7f7c080d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 -- 172.21.15.67:0/801269134 shutdown_connections 2024-04-17T08:52:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.979+0000 7f7c07e00640 1 -- 172.21.15.67:0/801269134 wait complete. 2024-04-17T08:52:41.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.981+0000 7f7c07e00640 1 Processor -- start 2024-04-17T08:52:41.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.981+0000 7f7c07e00640 1 -- start start 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c0806cf80 0x7f7c08160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c080b3b50 0x7f7c0815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c080d0560 con 0x7f7c0815ec20 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7c080d03e0 con 0x7f7c080b3b50 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c080d10d0 con 0x7f7c0806cf80 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51296/0 (socket says 172.21.15.67:51296) 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c07400640 1 -- 172.21.15.67:0/1920256872 learned_addr learned my addr 172.21.15.67:0/1920256872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.983+0000 7f7c06000640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c080b3b50 0x7f7c0815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c06a00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c0806cf80 0x7f7c08160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 -- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c0806cf80 msgr2=0x7f7c08160800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c0806cf80 0x7f7c08160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 -- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c080b3b50 msgr2=0x7f7c0815d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c080b3b50 0x7f7c0815d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 -- 172.21.15.67:0/1920256872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c081f57c0 con 0x7f7c0815ec20 2024-04-17T08:52:41.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07400640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c410 tx=0x7f7c0000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00017070 con 0x7f7c0815ec20 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c000035f0 con 0x7f7c0815ec20 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00003a20 con 0x7f7c0815ec20 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.984+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0815f760 con 0x7f7c0815ec20 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.985+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0815fd20 con 0x7f7c0815ec20 2024-04-17T08:52:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.986+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c08068a10 con 0x7f7c0815ec20 2024-04-17T08:52:41.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.987+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f7c0001a070 con 0x7f7c0815ec20 2024-04-17T08:52:41.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.988+0000 7f7bfea00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 0x7f7be406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:41.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.988+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4928+0+0 (secure 0 0 0) 0x7f7c0008aed0 con 0x7f7c0815ec20 2024-04-17T08:52:41.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.988+0000 7f7c06a00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 0x7f7be406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:41.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.990+0000 7f7c06a00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 0x7f7be406ef60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0035000 tx=0x7f7bf0008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:41.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:41.994+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7c081f57c0 con 0x7f7c0815ec20 2024-04-17T08:52:42.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.197+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c080d55f0 con 0x7f7c0815ec20 2024-04-17T08:52:42.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.198+0000 7f7bfea00640 1 -- 172.21.15.67:0/1920256872 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v268) ==== 93+0+10 (secure 0 0 0) 0x7f7c00059390 con 0x7f7c0815ec20 2024-04-17T08:52:42.197 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 msgr2=0x7f7be406ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 0x7f7be406ef60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0035000 tx=0x7f7bf0008030 comp rx=0 tx=0).stop 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 msgr2=0x7f7c081f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c410 tx=0x7f7c0000c8d0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 shutdown_connections 2024-04-17T08:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f7be406caa0 0x7f7be406ef60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0815ec20 0x7f7c081f5280 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c080b3b50 0x7f7c0815d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 --2- 172.21.15.67:0/1920256872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c0806cf80 0x7f7c08160800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 >> 172.21.15.67:0/1920256872 conn(0x7f7c0808bff0 msgr2=0x7f7c080b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 shutdown_connections 2024-04-17T08:52:42.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.201+0000 7f7c07e00640 1 -- 172.21.15.67:0/1920256872 wait complete. 2024-04-17T08:52:42.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:52:42.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.436+0000 7ff781200640 1 Processor -- start 2024-04-17T08:52:42.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.437+0000 7ff781200640 1 -- start start 2024-04-17T08:52:42.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06d8a0 0x7ff77c09b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff77c06ff60 0x7ff77c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c06ca70 0x7ff77c09afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff77c0d1cd0 con 0x7ff77c06ca70 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff77c09bd40 con 0x7ff77c06ff60 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.438+0000 7ff781200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c0cd770 con 0x7ff77c06d8a0 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c06ca70 0x7ff77c09afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff77c06ca70 0x7ff77c09afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51306/0 (socket says 172.21.15.67:51306) 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 -- 172.21.15.67:0/1080667632 learned_addr learned my addr 172.21.15.67:0/1080667632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77aa00640 1 --2- 172.21.15.67:0/1080667632 >> v2:172.21.15.187:3300/0 conn(0x7ff77c06ff60 0x7ff77c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77b400640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06d8a0 0x7ff77c09b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 -- 172.21.15.67:0/1080667632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06d8a0 msgr2=0x7ff77c09b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06d8a0 0x7ff77c09b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 -- 172.21.15.67:0/1080667632 >> v2:172.21.15.187:3300/0 conn(0x7ff77c06ff60 msgr2=0x7ff77c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 --2- 172.21.15.67:0/1080667632 >> v2:172.21.15.187:3300/0 conn(0x7ff77c06ff60 0x7ff77c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.439+0000 7ff77be00640 1 -- 172.21.15.67:0/1080667632 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c0cce40 con 0x7ff77c06ca70 2024-04-17T08:52:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.440+0000 7ff77be00640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ca70 0x7ff77c09afa0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7ff77c073ad0 tx=0x7ff77402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea84572682dda752 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.441+0000 7ff77a000640 1 -- 172.21.15.67:0/1080667632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774034070 con 0x7ff77c06ca70 2024-04-17T08:52:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.441+0000 7ff77a000640 1 -- 172.21.15.67:0/1080667632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff774030860 con 0x7ff77c06ca70 2024-04-17T08:52:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.441+0000 7ff77a000640 1 -- 172.21.15.67:0/1080667632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774030c30 con 0x7ff77c06ca70 2024-04-17T08:52:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.441+0000 7ff781200640 1 -- 172.21.15.67:0/1080667632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ca70 msgr2=0x7ff77c09afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.441+0000 7ff781200640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ca70 0x7ff77c09afa0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7ff77c073ad0 tx=0x7ff77402b9a0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 -- 172.21.15.67:0/1080667632 shutdown_connections 2024-04-17T08:52:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ca70 0x7ff77c09afa0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 --2- 172.21.15.67:0/1080667632 >> v2:172.21.15.187:3300/0 conn(0x7ff77c06ff60 0x7ff77c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 --2- 172.21.15.67:0/1080667632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06d8a0 0x7ff77c09b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 -- 172.21.15.67:0/1080667632 >> 172.21.15.67:0/1080667632 conn(0x7ff77c0b4140 msgr2=0x7ff77c0b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:42.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 -- 172.21.15.67:0/1080667632 shutdown_connections 2024-04-17T08:52:42.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.442+0000 7ff781200640 1 -- 172.21.15.67:0/1080667632 wait complete. 2024-04-17T08:52:42.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.443+0000 7ff781200640 1 Processor -- start 2024-04-17T08:52:42.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.444+0000 7ff781200640 1 -- start start 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 0x7ff77c0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c06d8a0 0x7ff77c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c06dc80 con 0x7ff77c06ff60 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff77c0cd400 con 0x7ff77c06d8a0 2024-04-17T08:52:42.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff781200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff77c06de00 con 0x7ff77c06ca70 2024-04-17T08:52:42.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51318/0 (socket says 172.21.15.67:51318) 2024-04-17T08:52:42.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 -- 172.21.15.67:0/283718178 learned_addr learned my addr 172.21.15.67:0/283718178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:42.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77aa00640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c06d8a0 0x7ff77c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77b400640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 0x7ff77c0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 -- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 msgr2=0x7ff77c0c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 0x7ff77c0c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 -- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c06d8a0 msgr2=0x7ff77c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.445+0000 7ff77be00640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c06d8a0 0x7ff77c15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff77be00640 1 -- 172.21.15.67:0/283718178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77c0c8430 con 0x7ff77c06ff60 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff77b400640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 0x7ff77c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff77be00640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7ff77402b9e0 tx=0x7ff774004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774048070 con 0x7ff77c06ff60 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff774030660 con 0x7ff77c06ff60 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774034040 con 0x7ff77c06ff60 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.446+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77c0c8710 con 0x7ff77c06ff60 2024-04-17T08:52:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.447+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77c0b6510 con 0x7ff77c06ff60 2024-04-17T08:52:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.448+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff77c0c88e0 con 0x7ff77c06ff60 2024-04-17T08:52:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.449+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7ff77404b070 con 0x7ff77c06ff60 2024-04-17T08:52:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.449+0000 7ff773400640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 0x7ff75c06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.450+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4928+0+0 (secure 0 0 0) 0x7ff7740ade00 con 0x7ff77c06ff60 2024-04-17T08:52:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.450+0000 7ff77b400640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 0x7ff75c06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.451+0000 7ff77b400640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 0x7ff75c06ef60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009f70 tx=0x7ff76c00b040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.454+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff77407c1e0 con 0x7ff77c06ff60 2024-04-17T08:52:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.661+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff77c0755d0 con 0x7ff77c06ff60 2024-04-17T08:52:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.662+0000 7ff773400640 1 -- 172.21.15.67:0/283718178 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v268) ==== 84+0+65 (secure 0 0 0) 0x7ff77407bae0 con 0x7ff77c06ff60 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:42.662 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:52:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 msgr2=0x7ff75c06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 0x7ff75c06ef60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009f70 tx=0x7ff76c00b040 comp rx=0 tx=0).stop 2024-04-17T08:52:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 msgr2=0x7ff77c0cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7ff77402b9e0 tx=0x7ff774004730 comp rx=0 tx=0).stop 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 shutdown_connections 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7ff75c06caa0 0x7ff75c06ef60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff77c06ff60 0x7ff77c0cbad0 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff77c06d8a0 0x7ff77c15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 --2- 172.21.15.67:0/283718178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff77c06ca70 0x7ff77c0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 >> 172.21.15.67:0/283718178 conn(0x7ff77c0b4140 msgr2=0x7ff77c0cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 shutdown_connections 2024-04-17T08:52:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.665+0000 7ff781200640 1 -- 172.21.15.67:0/283718178 wait complete. 2024-04-17T08:52:42.683 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:52:42.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:52:42.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.914+0000 7f95aca00640 1 Processor -- start 2024-04-17T08:52:42.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.914+0000 7f95aca00640 1 -- start start 2024-04-17T08:52:42.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f95aca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95a8146250 0x7f95a8160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f95aca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a8161000 con 0x7f95a80cd6d0 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f95aca00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a80ce870 con 0x7f95a80cd2f0 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f95aca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f95a80cfa60 con 0x7f95a8146250 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f959de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95a80cd6d0 0x7f95a8145b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52716/0 (socket says 172.21.15.67:52716) 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f959de00640 1 -- 172.21.15.67:0/1382255253 learned_addr learned my addr 172.21.15.67:0/1382255253 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.916+0000 7f95a7200640 1 --2- 172.21.15.67:0/1382255253 >> v2:172.21.15.67:3300/0 conn(0x7f95a8146250 0x7f95a8160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.917+0000 7f95a5e00640 1 -- 172.21.15.67:0/1382255253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3763566548 0 0) 0x7f95a8161000 con 0x7f95a80cd6d0 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.917+0000 7f95a7200640 1 -- 172.21.15.67:0/1382255253 >> v1:172.21.15.187:6789/0 conn(0x7f95a80cd2f0 legacy=0x7f95a80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.917+0000 7f95a7200640 1 -- 172.21.15.67:0/1382255253 >> v1:172.21.15.67:6789/0 conn(0x7f95a80cd6d0 legacy=0x7f95a8145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.917+0000 7f95a7200640 1 -- 172.21.15.67:0/1382255253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a80ceb50 con 0x7f95a8146250 2024-04-17T08:52:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.918+0000 7f95a7200640 1 --2- 172.21.15.67:0/1382255253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a8146250 0x7f95a8160360 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f95a00097d0 tx=0x7f95a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c34c87509f87bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.918+0000 7f95a5e00640 1 -- 172.21.15.67:0/1382255253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0035020 con 0x7f95a8146250 2024-04-17T08:52:42.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.918+0000 7f95a5e00640 1 -- 172.21.15.67:0/1382255253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a0031810 con 0x7f95a8146250 2024-04-17T08:52:42.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.918+0000 7f95a5e00640 1 -- 172.21.15.67:0/1382255253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0031be0 con 0x7f95a8146250 2024-04-17T08:52:42.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 -- 172.21.15.67:0/1382255253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a8146250 msgr2=0x7f95a8160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 --2- 172.21.15.67:0/1382255253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a8146250 0x7f95a8160360 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f95a00097d0 tx=0x7f95a002b8b0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 -- 172.21.15.67:0/1382255253 shutdown_connections 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 --2- 172.21.15.67:0/1382255253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a8146250 0x7f95a8160360 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 -- 172.21.15.67:0/1382255253 >> 172.21.15.67:0/1382255253 conn(0x7f95a806b8c0 msgr2=0x7f95a806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 -- 172.21.15.67:0/1382255253 shutdown_connections 2024-04-17T08:52:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.919+0000 7f95aca00640 1 -- 172.21.15.67:0/1382255253 wait complete. 2024-04-17T08:52:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.921+0000 7f95aca00640 1 Processor -- start 2024-04-17T08:52:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.921+0000 7f95aca00640 1 -- start start 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95a80cd2f0 0x7f95a813dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95a8146250 0x7f95a813f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f95a8160a20 con 0x7f95a80cd6d0 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f95a81608a0 con 0x7f95a80cd2f0 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f95aca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95a8160ba0 con 0x7f95a8146250 2024-04-17T08:52:42.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f959de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f959de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51330/0 (socket says 172.21.15.67:51330) 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.922+0000 7f959de00640 1 -- 172.21.15.67:0/4291519025 learned_addr learned my addr 172.21.15.67:0/4291519025 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f95a6800640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95a80cd2f0 0x7f95a813dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f95a7200640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95a8146250 0x7f95a813f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 -- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95a8146250 msgr2=0x7f95a813f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95a8146250 0x7f95a813f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 -- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95a80cd2f0 msgr2=0x7f95a813dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95a80cd2f0 0x7f95a813dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 -- 172.21.15.67:0/4291519025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a815d100 con 0x7f95a80cd6d0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.923+0000 7f959de00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f959000c4e0 tx=0x7f959000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.924+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590018070 con 0x7f95a80cd6d0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.924+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9590007500 con 0x7f95a80cd6d0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.924+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a815d3e0 con 0x7f95a80cd6d0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.924+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590004a20 con 0x7f95a80cd6d0 2024-04-17T08:52:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.924+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a815d9a0 con 0x7f95a80cd6d0 2024-04-17T08:52:42.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.925+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a8068a10 con 0x7f95a80cd6d0 2024-04-17T08:52:42.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.926+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 146) ==== 91188+0+0 (secure 0 0 0) 0x7f959001b070 con 0x7f95a80cd6d0 2024-04-17T08:52:42.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.927+0000 7f959f400640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 0x7f958406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:42.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.927+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4928+0+0 (secure 0 0 0) 0x7f959008ae50 con 0x7f95a80cd6d0 2024-04-17T08:52:42.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.928+0000 7f95a6800640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 0x7f958406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.929+0000 7f95a6800640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 0x7f958406ef90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9598009f20 tx=0x7f95980094d0 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:42.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:42.931+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f95900593e0 con 0x7f95a80cd6d0 2024-04-17T08:52:43.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.136+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f95a8068610 con 0x7f95a80cd6d0 2024-04-17T08:52:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.738+0000 7f959f400640 1 -- 172.21.15.67:0/4291519025 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v269) ==== 138+0+0 (secure 0 0 0) 0x7f9590058ce0 con 0x7f95a80cd6d0 2024-04-17T08:52:43.738 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:52:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.742+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 msgr2=0x7f958406ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 0x7f958406ef90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9598009f20 tx=0x7f95980094d0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 msgr2=0x7f95a813e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f959000c4e0 tx=0x7f959000c9a0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 shutdown_connections 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f958406cad0 0x7f958406ef90 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9598009f20 tx=0x7f95980094d0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95a8146250 0x7f95a813f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95a80cd6d0 0x7f95a813e7f0 secure :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f959000c4e0 tx=0x7f959000c9a0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 --2- 172.21.15.67:0/4291519025 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95a80cd2f0 0x7f95a813dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.743+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 >> 172.21.15.67:0/4291519025 conn(0x7f95a806b8c0 msgr2=0x7f95a80cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:43.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.744+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 shutdown_connections 2024-04-17T08:52:43.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.744+0000 7f95aca00640 1 -- 172.21.15.67:0/4291519025 wait complete. 2024-04-17T08:52:43.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:52:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.989+0000 7f078d800640 1 Processor -- start 2024-04-17T08:52:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.989+0000 7f078d800640 1 -- start start 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.991+0000 7f078d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f078806c8d0 0x7f078806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.991+0000 7f078d800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f07880af6c0 0x7f07880afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.991+0000 7f078d800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078806e4f0 con 0x7f07881448b0 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.991+0000 7f078d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f078806d270 con 0x7f078806c8d0 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.991+0000 7f078d800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f07880b62f0 con 0x7f07880af6c0 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f077f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f078806c8d0 0x7f078806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f0787e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07881448b0 0x7f07880b5be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45376/0 (socket says 172.21.15.67:45376) 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f0787e00640 1 -- 172.21.15.67:0/2648100204 learned_addr learned my addr 172.21.15.67:0/2648100204 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f078ce00640 1 --2- 172.21.15.67:0/2648100204 >> v2:172.21.15.187:3300/0 conn(0x7f07880af6c0 0x7f07880afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f077f400640 1 -- 172.21.15.67:0/2648100204 >> v2:172.21.15.187:3300/0 conn(0x7f07880af6c0 msgr2=0x7f07880afb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f077f400640 1 --2- 172.21.15.67:0/2648100204 >> v2:172.21.15.187:3300/0 conn(0x7f07880af6c0 0x7f07880afb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f077f400640 1 -- 172.21.15.67:0/2648100204 >> v1:172.21.15.67:6789/0 conn(0x7f07881448b0 legacy=0x7f07880b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f077f400640 1 -- 172.21.15.67:0/2648100204 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078806e7d0 con 0x7f078806c8d0 2024-04-17T08:52:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.992+0000 7f0787400640 1 -- 172.21.15.67:0/2648100204 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065157735 0 0) 0x7f078806e4f0 con 0x7f07881448b0 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.994+0000 7f077f400640 1 --2- 172.21.15.67:0/2648100204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f078806c8d0 0x7f078806cd30 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f078815f8d0 tx=0x7f077002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9963226511440913 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.994+0000 7f0787400640 1 -- 172.21.15.67:0/2648100204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770035020 con 0x7f078806c8d0 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.994+0000 7f0787400640 1 -- 172.21.15.67:0/2648100204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0770031830 con 0x7f078806c8d0 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.994+0000 7f0787400640 1 -- 172.21.15.67:0/2648100204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770031c00 con 0x7f078806c8d0 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.994+0000 7f078d800640 1 -- 172.21.15.67:0/2648100204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f078806c8d0 msgr2=0x7f078806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 --2- 172.21.15.67:0/2648100204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f078806c8d0 0x7f078806cd30 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f078815f8d0 tx=0x7f077002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 -- 172.21.15.67:0/2648100204 shutdown_connections 2024-04-17T08:52:43.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 --2- 172.21.15.67:0/2648100204 >> v2:172.21.15.187:3300/0 conn(0x7f07880af6c0 0x7f07880afb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 --2- 172.21.15.67:0/2648100204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f078806c8d0 0x7f078806cd30 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 -- 172.21.15.67:0/2648100204 >> 172.21.15.67:0/2648100204 conn(0x7f07880725b0 msgr2=0x7f07880729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:43.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 -- 172.21.15.67:0/2648100204 shutdown_connections 2024-04-17T08:52:43.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.995+0000 7f078d800640 1 -- 172.21.15.67:0/2648100204 wait complete. 2024-04-17T08:52:43.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.997+0000 7f078d800640 1 Processor -- start 2024-04-17T08:52:43.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.997+0000 7f078d800640 1 -- start start 2024-04-17T08:52:43.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 0x7f07880b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:43.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:43.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07881448b0 0x7f078815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:43.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f078806f070 con 0x7f07880af6c0 2024-04-17T08:52:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f078806ed90 con 0x7f07881448b0 2024-04-17T08:52:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f078d800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07880b3dc0 con 0x7f078806c8d0 2024-04-17T08:52:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f077f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f077f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55720/0 (socket says 172.21.15.67:55720) 2024-04-17T08:52:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.998+0000 7f077f400640 1 -- 172.21.15.67:0/2822912974 learned_addr learned my addr 172.21.15.67:0/2822912974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f0787e00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 0x7f07880b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f078ce00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07881448b0 0x7f078815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 -- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 msgr2=0x7f07880b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 0x7f07880b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 -- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07881448b0 msgr2=0x7f078815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07881448b0 0x7f078815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 -- 172.21.15.67:0/2822912974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0788139c90 con 0x7f07880af6c0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f0787e00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 0x7f07880b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f077f400640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f0770009890 tx=0x7f07700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770041070 con 0x7f07880af6c0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077003b920 con 0x7f07880af6c0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0788139f70 con 0x7f07880af6c0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:43.999+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770035040 con 0x7f07880af6c0 2024-04-17T08:52:44.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.000+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0788141920 con 0x7f07880af6c0 2024-04-17T08:52:44.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.001+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0788068a10 con 0x7f07880af6c0 2024-04-17T08:52:44.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.002+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f077004c060 con 0x7f07880af6c0 2024-04-17T08:52:44.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.003+0000 7f0784c00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 0x7f076406f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.003+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 5298+0+0 (secure 0 0 0) 0x7f07700ad7b0 con 0x7f07880af6c0 2024-04-17T08:52:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.004+0000 7f0787e00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 0x7f076406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:44.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.005+0000 7f0787e00640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 0x7f076406f020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0778007fd0 tx=0x7f0778009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.009+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0788068a10 con 0x7f07880af6c0 2024-04-17T08:52:44.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.215+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0788141b80 con 0x7f07880af6c0 2024-04-17T08:52:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.759+0000 7f0784c00640 1 -- 172.21.15.67:0/2822912974 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v270) ==== 170+0+0 (secure 0 0 0) 0x7f077007b980 con 0x7f07880af6c0 2024-04-17T08:52:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:52:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.762+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 msgr2=0x7f076406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.762+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 0x7f076406f020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0778007fd0 tx=0x7f0778009040 comp rx=0 tx=0).stop 2024-04-17T08:52:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.762+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 msgr2=0x7f078815b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.762+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f0770009890 tx=0x7f07700046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 shutdown_connections 2024-04-17T08:52:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f076406cb60 0x7f076406f020 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07881448b0 0x7f078815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07880af6c0 0x7f078815b240 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 --2- 172.21.15.67:0/2822912974 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f078806c8d0 0x7f07880b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.763+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 >> 172.21.15.67:0/2822912974 conn(0x7f07880725b0 msgr2=0x7f0788067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:44.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.766+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 shutdown_connections 2024-04-17T08:52:44.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.768+0000 7f078d800640 1 -- 172.21.15.67:0/2822912974 wait complete. 2024-04-17T08:52:44.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:52:44.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.995+0000 7f5579200640 1 Processor -- start 2024-04-17T08:52:44.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.995+0000 7f5579200640 1 -- start start 2024-04-17T08:52:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5579200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557415b6b0 0x7f557406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5579200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557415ff40 con 0x7f557406f800 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5579200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557406df40 con 0x7f55740cfc10 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5579200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f557406ed00 con 0x7f557415b6b0 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5573400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557415b6b0 0x7f557406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5573400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557415b6b0 0x7f557406b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55728/0 (socket says 172.21.15.67:55728) 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.997+0000 7f5573400640 1 -- 172.21.15.67:0/528876845 learned_addr learned my addr 172.21.15.67:0/528876845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.998+0000 7f5573400640 1 -- 172.21.15.67:0/528876845 >> v1:172.21.15.187:6789/0 conn(0x7f55740cfc10 legacy=0x7f557415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.998+0000 7f5573400640 1 -- 172.21.15.67:0/528876845 >> v1:172.21.15.67:6789/0 conn(0x7f557406f800 legacy=0x7f557415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.998+0000 7f5573400640 1 -- 172.21.15.67:0/528876845 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557406e220 con 0x7f557415b6b0 2024-04-17T08:52:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.998+0000 7f5572a00640 1 -- 172.21.15.67:0/528876845 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148588208 0 0) 0x7f557415ff40 con 0x7f557406f800 2024-04-17T08:52:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:44.999+0000 7f5573400640 1 --2- 172.21.15.67:0/528876845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557415b6b0 0x7f557406b8c0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f5564009870 tx=0x7f556402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68558347eae77e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5572a00640 1 -- 172.21.15.67:0/528876845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5564035020 con 0x7f557415b6b0 2024-04-17T08:52:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5572a00640 1 -- 172.21.15.67:0/528876845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5564031870 con 0x7f557415b6b0 2024-04-17T08:52:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5572a00640 1 -- 172.21.15.67:0/528876845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5564031c40 con 0x7f557415b6b0 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 -- 172.21.15.67:0/528876845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557415b6b0 msgr2=0x7f557406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 --2- 172.21.15.67:0/528876845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557415b6b0 0x7f557406b8c0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f5564009870 tx=0x7f556402b910 comp rx=0 tx=0).stop 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 -- 172.21.15.67:0/528876845 shutdown_connections 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 --2- 172.21.15.67:0/528876845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557415b6b0 0x7f557406b8c0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 -- 172.21.15.67:0/528876845 >> 172.21.15.67:0/528876845 conn(0x7f55740b2190 msgr2=0x7f55740b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 -- 172.21.15.67:0/528876845 shutdown_connections 2024-04-17T08:52:45.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.000+0000 7f5579200640 1 -- 172.21.15.67:0/528876845 wait complete. 2024-04-17T08:52:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.002+0000 7f5579200640 1 Processor -- start 2024-04-17T08:52:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.002+0000 7f5579200640 1 -- start start 2024-04-17T08:52:45.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55740cfc10 0x7f557415a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 0x7f5574153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55740afd60 con 0x7f557406f800 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f557406e870 con 0x7f557415b6b0 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.003+0000 7f5579200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55740afee0 con 0x7f55740cfc10 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55744/0 (socket says 172.21.15.67:55744) 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 -- 172.21.15.67:0/3389760836 learned_addr learned my addr 172.21.15.67:0/3389760836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 -- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55740cfc10 msgr2=0x7f557415a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55740cfc10 0x7f557415a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 -- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 msgr2=0x7f5574153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573e00640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 0x7f5574153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 0x7f5574153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 -- 172.21.15.67:0/3389760836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557415a620 con 0x7f557406f800 2024-04-17T08:52:45.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573e00640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 0x7f5574153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:45.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.004+0000 7f5573400640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f5564009870 tx=0x7f55640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.005+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556403ad60 con 0x7f557406f800 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.005+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55640443f0 con 0x7f557406f800 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.005+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55640447e0 con 0x7f557406f800 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.005+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5574151910 con 0x7f557406f800 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.005+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5574151ed0 con 0x7f557406f800 2024-04-17T08:52:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.006+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5574068a10 con 0x7f557406f800 2024-04-17T08:52:45.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.008+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f556404b060 con 0x7f557406f800 2024-04-17T08:52:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.009+0000 7f556be00640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 0x7f555006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.009+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(270..270 src has 1..270) ==== 5680+0+0 (secure 0 0 0) 0x7f55640ae660 con 0x7f557406f800 2024-04-17T08:52:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.010+0000 7f556aa00640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 0x7f555006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.011+0000 7f556aa00640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 0x7f555006eff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f557414dca0 tx=0x7f555c007910 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:45.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.014+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f556403f070 con 0x7f557406f800 2024-04-17T08:52:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.262+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f557406ee80 con 0x7f557406f800 2024-04-17T08:52:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.764+0000 7f556be00640 1 -- 172.21.15.67:0/3389760836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v139) ==== 174+0+0 (secure 0 0 0) 0x7f556402f320 con 0x7f557406f800 2024-04-17T08:52:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 34 and data pool 35 2024-04-17T08:52:45.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 msgr2=0x7f555006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 0x7f555006eff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f557414dca0 tx=0x7f555c007910 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 msgr2=0x7f557414d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f5564009870 tx=0x7f55640046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 shutdown_connections 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f555006cb30 0x7f555006eff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f557415b6b0 0x7f5574153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55740cfc10 0x7f557415a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 --2- 172.21.15.67:0/3389760836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557406f800 0x7f557414d080 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.768+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 >> 172.21.15.67:0/3389760836 conn(0x7f55740b2190 msgr2=0x7f55740b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.769+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 shutdown_connections 2024-04-17T08:52:45.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.770+0000 7f5579200640 1 -- 172.21.15.67:0/3389760836 wait complete. 2024-04-17T08:52:45.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.994+0000 7f8197200640 1 Processor -- start 2024-04-17T08:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.994+0000 7f8197200640 1 -- start start 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 0x7f81980cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8198160280 0x7f819806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 0x7f8198142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f819815ce30 con 0x7f8198160280 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f81980cd4a0 con 0x7f819806b8c0 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.995+0000 7f8197200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81981607f0 con 0x7f8198141ed0 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8198160280 0x7f819806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8198160280 0x7f819806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55752/0 (socket says 172.21.15.67:55752) 2024-04-17T08:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 -- 172.21.15.67:0/3822888927 learned_addr learned my addr 172.21.15.67:0/3822888927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f8196800640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 0x7f8198142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f8195e00640 1 --2- 172.21.15.67:0/3822888927 >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 0x7f81980cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 -- 172.21.15.67:0/3822888927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 msgr2=0x7f8198142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 0x7f8198142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 -- 172.21.15.67:0/3822888927 >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 msgr2=0x7f81980cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 --2- 172.21.15.67:0/3822888927 >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 0x7f81980cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f818d400640 1 -- 172.21.15.67:0/3822888927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8198160ad0 con 0x7f8198160280 2024-04-17T08:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f8196800640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 0x7f8198142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.996+0000 7f8195e00640 1 --2- 172.21.15.67:0/3822888927 >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 0x7f81980cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.998+0000 7f818d400640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8198160280 0x7f819806bca0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f81980b3210 tx=0x7f818002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebaab234789fa1e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.998+0000 7f8195400640 1 -- 172.21.15.67:0/3822888927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180035020 con 0x7f8198160280 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.998+0000 7f8195400640 1 -- 172.21.15.67:0/3822888927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8180031880 con 0x7f8198160280 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.998+0000 7f8195400640 1 -- 172.21.15.67:0/3822888927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180031c50 con 0x7f8198160280 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 -- 172.21.15.67:0/3822888927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8198160280 msgr2=0x7f819806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8198160280 0x7f819806bca0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f81980b3210 tx=0x7f818002b920 comp rx=0 tx=0).stop 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 -- 172.21.15.67:0/3822888927 shutdown_connections 2024-04-17T08:52:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198141ed0 0x7f8198142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 --2- 172.21.15.67:0/3822888927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8198160280 0x7f819806bca0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 --2- 172.21.15.67:0/3822888927 >> v2:172.21.15.187:3300/0 conn(0x7f819806b8c0 0x7f81980cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 -- 172.21.15.67:0/3822888927 >> 172.21.15.67:0/3822888927 conn(0x7f8198067d10 msgr2=0x7f81980b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 -- 172.21.15.67:0/3822888927 shutdown_connections 2024-04-17T08:52:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:45.999+0000 7f8197200640 1 -- 172.21.15.67:0/3822888927 wait complete. 2024-04-17T08:52:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.001+0000 7f8197200640 1 Processor -- start 2024-04-17T08:52:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.001+0000 7f8197200640 1 -- start start 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8198141ed0 0x7f81980d0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198160280 0x7f81980d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f819806c1e0 con 0x7f819806b8c0 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81980cddf0 con 0x7f8198141ed0 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8197200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8198161090 con 0x7f8198160280 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8195e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8195e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55766/0 (socket says 172.21.15.67:55766) 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f8195e00640 1 -- 172.21.15.67:0/193276357 learned_addr learned my addr 172.21.15.67:0/193276357 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.002+0000 7f818d400640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8198141ed0 0x7f81980d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 -- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198160280 msgr2=0x7f81980d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8196800640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198160280 0x7f81980d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198160280 0x7f81980d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 -- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8198141ed0 msgr2=0x7f81980d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8198141ed0 0x7f81980d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 -- 172.21.15.67:0/193276357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81981f2720 con 0x7f819806b8c0 2024-04-17T08:52:46.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8195e00640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f818800cee0 tx=0x7f818800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818801a070 con 0x7f819806b8c0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818800d840 con 0x7f819806b8c0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818800dc10 con 0x7f819806b8c0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81981f2a00 con 0x7f819806b8c0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.003+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81981f32a0 con 0x7f819806b8c0 2024-04-17T08:52:46.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.005+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81981f3090 con 0x7f819806b8c0 2024-04-17T08:52:46.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.006+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f818801d070 con 0x7f819806b8c0 2024-04-17T08:52:46.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.006+0000 7f818ea00640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 0x7f817406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.007+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5780+0+0 (secure 0 0 0) 0x7f818808c7b0 con 0x7f819806b8c0 2024-04-17T08:52:46.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.007+0000 7f818d400640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 0x7f817406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.009+0000 7f818d400640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 0x7f817406eff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8180036000 tx=0x7f8180008030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.011+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f818805a140 con 0x7f819806b8c0 2024-04-17T08:52:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.210+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8198142b20 con 0x7f819806b8c0 2024-04-17T08:52:46.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.212+0000 7f818ea00640 1 -- 172.21.15.67:0/193276357 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+16698 (secure 0 0 0) 0x7f818805e440 con 0x7f819806b8c0 2024-04-17T08:52:46.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:46.213 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":271,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:45.263994+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:46.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 msgr2=0x7f817406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 0x7f817406eff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8180036000 tx=0x7f8180008030 comp rx=0 tx=0).stop 2024-04-17T08:52:46.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 msgr2=0x7f81981f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f818800cee0 tx=0x7f818800a710 comp rx=0 tx=0).stop 2024-04-17T08:52:46.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 shutdown_connections 2024-04-17T08:52:46.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f817406cb30 0x7f817406eff0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8198160280 0x7f81980d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8198141ed0 0x7f81980d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 --2- 172.21.15.67:0/193276357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f819806b8c0 0x7f81981f1ed0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.215+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 >> 172.21.15.67:0/193276357 conn(0x7f8198067d10 msgr2=0x7f81981412b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.216+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 shutdown_connections 2024-04-17T08:52:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.216+0000 7f8197200640 1 -- 172.21.15.67:0/193276357 wait complete. 2024-04-17T08:52:46.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.451+0000 7f12b7200640 1 Processor -- start 2024-04-17T08:52:46.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.451+0000 7f12b7200640 1 -- start start 2024-04-17T08:52:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12b8141ed0 0x7f12b8142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b7200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b80cd4a0 con 0x7f12b8160280 2024-04-17T08:52:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f12b815ce30 con 0x7f12b8141ed0 2024-04-17T08:52:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12b81607f0 con 0x7f12b806b8c0 2024-04-17T08:52:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b6800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12b8141ed0 0x7f12b8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b6800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f12b8141ed0 0x7f12b8142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:42694/0 (socket says 172.21.15.67:42694) 2024-04-17T08:52:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.453+0000 7f12b5e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.454+0000 7f12b6800640 1 -- 172.21.15.67:0/273335855 learned_addr learned my addr 172.21.15.67:0/273335855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.454+0000 7f12b6800640 1 -- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 msgr2=0x7f12b80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.454+0000 7f12b6800640 1 --2- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.454+0000 7f12b6800640 1 -- 172.21.15.67:0/273335855 >> v1:172.21.15.187:6789/0 conn(0x7f12b8160280 legacy=0x7f12b806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.454+0000 7f12b6800640 1 -- 172.21.15.67:0/273335855 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b8160ad0 con 0x7f12b8141ed0 2024-04-17T08:52:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.456+0000 7f12b6800640 1 --2- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b8142360 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009e50 tx=0x7f12b002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46329b49b018ce59 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.456+0000 7f12b4a00640 1 -- 172.21.15.67:0/273335855 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b002cb00 con 0x7f12b8141ed0 2024-04-17T08:52:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.456+0000 7f12b4a00640 1 -- 172.21.15.67:0/273335855 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b002cca0 con 0x7f12b8141ed0 2024-04-17T08:52:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.456+0000 7f12b4a00640 1 -- 172.21.15.67:0/273335855 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0034610 con 0x7f12b8141ed0 2024-04-17T08:52:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 -- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 msgr2=0x7f12b8142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 --2- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b8142360 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009e50 tx=0x7f12b002bf60 comp rx=0 tx=0).stop 2024-04-17T08:52:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 -- 172.21.15.67:0/273335855 shutdown_connections 2024-04-17T08:52:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 --2- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b8142360 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 --2- 172.21.15.67:0/273335855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 -- 172.21.15.67:0/273335855 >> 172.21.15.67:0/273335855 conn(0x7f12b8067d10 msgr2=0x7f12b80b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 -- 172.21.15.67:0/273335855 shutdown_connections 2024-04-17T08:52:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.457+0000 7f12b7200640 1 -- 172.21.15.67:0/273335855 wait complete. 2024-04-17T08:52:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.459+0000 7f12b7200640 1 Processor -- start 2024-04-17T08:52:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.459+0000 7f12b7200640 1 -- start start 2024-04-17T08:52:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b80b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8161090 con 0x7f12b8160280 2024-04-17T08:52:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f12b80cddf0 con 0x7f12b8141ed0 2024-04-17T08:52:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f12b813f9b0 con 0x7f12b806b8c0 2024-04-17T08:52:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55768/0 (socket says 172.21.15.67:55768) 2024-04-17T08:52:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b6800640 1 -- 172.21.15.67:0/2161878823 learned_addr learned my addr 172.21.15.67:0/2161878823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b5400640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b80b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.460+0000 7f12b5e00640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 -- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 msgr2=0x7f12b80b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 -- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 msgr2=0x7f12b80b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b80b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 -- 172.21.15.67:0/2161878823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b81f4ab0 con 0x7f12b8160280 2024-04-17T08:52:46.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b6800640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f12b0036040 tx=0x7f12b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0043020 con 0x7f12b8160280 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b002ce50 con 0x7f12b8160280 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0009510 con 0x7f12b8160280 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.461+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b81f4d90 con 0x7f12b8160280 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.462+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b81f8da0 con 0x7f12b8160280 2024-04-17T08:52:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.463+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b8068a10 con 0x7f12b8160280 2024-04-17T08:52:46.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.464+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f12b0046020 con 0x7f12b8160280 2024-04-17T08:52:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.465+0000 7f12ae000640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 0x7f129806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.465+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5780+0+0 (secure 0 0 0) 0x7f12b00ae620 con 0x7f12b8160280 2024-04-17T08:52:46.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.465+0000 7f12b5e00640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 0x7f129806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.466+0000 7f12b5e00640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 0x7f129806eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009ad0 tx=0x7f12a8008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.469+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f12b004d030 con 0x7f12b8160280 2024-04-17T08:52:46.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.665+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12b8140980 con 0x7f12b8160280 2024-04-17T08:52:46.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.667+0000 7f12ae000640 1 -- 172.21.15.67:0/2161878823 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+16698 (secure 0 0 0) 0x7f12b003c350 con 0x7f12b8160280 2024-04-17T08:52:46.667 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:46.667 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":271,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:45.263994+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.669+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 msgr2=0x7f129806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.669+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 0x7f129806eff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009ad0 tx=0x7f12a8008040 comp rx=0 tx=0).stop 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.669+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 msgr2=0x7f12b80b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.669+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f12b0036040 tx=0x7f12b0004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 shutdown_connections 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f129806cb30 0x7f129806eff0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12b8160280 0x7f12b80b17c0 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12b8141ed0 0x7f12b80b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 --2- 172.21.15.67:0/2161878823 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f12b806b8c0 0x7f12b80b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 >> 172.21.15.67:0/2161878823 conn(0x7f12b8067d10 msgr2=0x7f12b80cf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:46.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 shutdown_connections 2024-04-17T08:52:46.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.670+0000 7f12b7200640 1 -- 172.21.15.67:0/2161878823 wait complete. 2024-04-17T08:52:46.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:52:46.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.897+0000 7f8dfca00640 1 Processor -- start 2024-04-17T08:52:46.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.897+0000 7f8dfca00640 1 -- start start 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8dfca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8df8145f80 0x7f8df80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8dfca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8df806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8dfca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df806f1d0 con 0x7f8df806c0a0 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8dfca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8df80727a0 con 0x7f8df8145f80 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8dfca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8df8146360 con 0x7f8df80cd750 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8df7200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8df806c0a0 0x7f8df8160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45396/0 (socket says 172.21.15.67:45396) 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.899+0000 7f8df7200640 1 -- 172.21.15.67:0/1212642978 learned_addr learned my addr 172.21.15.67:0/1212642978 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df6800640 1 --2- 172.21.15.67:0/1212642978 >> v2:172.21.15.187:3300/0 conn(0x7f8df8145f80 0x7f8df80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248829854 0 0) 0x7f8df0001490 con 0x7f8df806c0a0 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc003520 con 0x7f8df806c0a0 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8df806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 563809640 0 0) 0x7f8ddc003520 con 0x7f8df806c0a0 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 msgr2=0x7f8df806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8df806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 >> v2:172.21.15.187:3300/0 conn(0x7f8df8145f80 msgr2=0x7f8df80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 --2- 172.21.15.67:0/1212642978 >> v2:172.21.15.187:3300/0 conn(0x7f8df8145f80 0x7f8df80cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df8071e70 con 0x7f8df806c0a0 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f8df0001bc0 con 0x7f8df806c0a0 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 >> v1:172.21.15.67:6789/0 conn(0x7f8df806c0a0 legacy=0x7f8df8160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ddc0039b0 0x7f8ddc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8ddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8ddc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8df80727a0 con 0x7f8ddc004800 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8146360 con 0x7f8ddc0039b0 2024-04-17T08:52:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.900+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ddc00baf0 con 0x7f8df80cd750 2024-04-17T08:52:46.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8ddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df7200640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8ddc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df6800640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ddc0039b0 0x7f8ddc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 -- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 msgr2=0x7f8ddc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:46.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df80cd750 0x7f8ddc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 -- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ddc0039b0 msgr2=0x7f8ddc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ddc0039b0 0x7f8ddc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.901+0000 7f8df5e00640 1 -- 172.21.15.67:0/1212642978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc003520 con 0x7f8ddc004800 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.902+0000 7f8df5e00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8ddc004c80 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f8de400ba00 tx=0x7f8ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b144f6fb8ab97d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.902+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4036020 con 0x7f8ddc004800 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.902+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de40325f0 con 0x7f8ddc004800 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.902+0000 7f8df5400640 1 -- 172.21.15.67:0/1212642978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de400b600 con 0x7f8ddc004800 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 -- 172.21.15.67:0/1212642978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 msgr2=0x7f8ddc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 --2- 172.21.15.67:0/1212642978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8ddc004c80 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f8de400ba00 tx=0x7f8ddc005830 comp rx=0 tx=0).stop 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8df7200640 1 -- 172.21.15.67:0/1212642978 reap_dead start 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 -- 172.21.15.67:0/1212642978 shutdown_connections 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 -- 172.21.15.67:0/1212642978 >> 172.21.15.67:0/1212642978 conn(0x7f8df80b2190 msgr2=0x7f8df80b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 -- 172.21.15.67:0/1212642978 shutdown_connections 2024-04-17T08:52:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.903+0000 7f8dfca00640 1 -- 172.21.15.67:0/1212642978 wait complete. 2024-04-17T08:52:46.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.905+0000 7f8dfca00640 1 Processor -- start 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.905+0000 7f8dfca00640 1 -- start start 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df813e920 0x7f8df813eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df80cc280 0x7f8df80cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8072c30 con 0x7f8ddc004800 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8df8072430 con 0x7f8df80cc280 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8dfca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8df8072db0 con 0x7f8df813e920 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8df6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8df6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55782/0 (socket says 172.21.15.67:55782) 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.906+0000 7f8df6800640 1 -- 172.21.15.67:0/641639544 learned_addr learned my addr 172.21.15.67:0/641639544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 -- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df813e920 msgr2=0x7f8df813eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df813e920 0x7f8df813eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 -- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df80cc280 msgr2=0x7f8df80cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df80cc280 0x7f8df80cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 -- 172.21.15.67:0/641639544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df81f8140 con 0x7f8ddc004800 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8df6800640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f8de8011700 tx=0x7f8de8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de800f9c0 con 0x7f8ddc004800 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de800fb60 con 0x7f8ddc004800 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8015460 con 0x7f8ddc004800 2024-04-17T08:52:46.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df81f8420 con 0x7f8ddc004800 2024-04-17T08:52:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.907+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df81f89e0 con 0x7f8ddc004800 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.908+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df8068a10 con 0x7f8ddc004800 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.910+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f8de8015600 con 0x7f8ddc004800 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.910+0000 7f8deea00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 0x7f8dd806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.910+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f8de808c9d0 con 0x7f8ddc004800 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.911+0000 7f8df5e00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 0x7f8dd806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.912+0000 7f8df5e00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 0x7f8dd806eff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8de4037000 tx=0x7f8de4032000 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:46.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:46.915+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8de8019030 con 0x7f8ddc004800 2024-04-17T08:52:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.159+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8df815e0e0 con 0x7f8ddc004800 2024-04-17T08:52:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.816+0000 7f8deea00640 1 -- 172.21.15.67:0/641639544 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v142) ==== 118+0+0 (secure 0 0 0) 0x7f8de805aae0 con 0x7f8ddc004800 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 msgr2=0x7f8dd806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 0x7f8dd806eff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8de4037000 tx=0x7f8de4032000 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 msgr2=0x7f8df813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f8de8011700 tx=0x7f8de8011bc0 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 shutdown_connections 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f8dd806cb30 0x7f8dd806eff0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8df80cc280 0x7f8df80cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8df813e920 0x7f8df813eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 --2- 172.21.15.67:0/641639544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ddc004800 0x7f8df813b080 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 >> 172.21.15.67:0/641639544 conn(0x7f8df80b2190 msgr2=0x7f8df80cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.820+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 shutdown_connections 2024-04-17T08:52:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:47.821+0000 7f8dfca00640 1 -- 172.21.15.67:0/641639544 wait complete. 2024-04-17T08:52:47.843 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:52:47.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:52:48.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.070+0000 7f820b000640 1 Processor -- start 2024-04-17T08:52:48.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.071+0000 7f820b000640 1 -- start start 2024-04-17T08:52:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.072+0000 7f820b000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8204159020 0x7f820415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.072+0000 7f820b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82040af400 0x7f8204162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.072+0000 7f820b000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820415f780 con 0x7f820406b8c0 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.072+0000 7f820b000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f820415fa60 con 0x7f8204159020 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.072+0000 7f820b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82041624a0 con 0x7f82040af400 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.073+0000 7f8209c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8204159020 0x7f820415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.073+0000 7f8209200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82040af400 0x7f8204162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.073+0000 7f8209c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8204159020 0x7f820415ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:42752/0 (socket says 172.21.15.67:42752) 2024-04-17T08:52:48.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.073+0000 7f8209c00640 1 -- 172.21.15.67:0/320617353 learned_addr learned my addr 172.21.15.67:0/320617353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:48.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.074+0000 7f8209c00640 1 -- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82040af400 msgr2=0x7f8204162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:48.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.074+0000 7f8209c00640 1 --2- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82040af400 0x7f8204162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.074+0000 7f8209c00640 1 -- 172.21.15.67:0/320617353 >> v1:172.21.15.187:6789/0 conn(0x7f820406b8c0 legacy=0x7f82040adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.074+0000 7f8209c00640 1 -- 172.21.15.67:0/320617353 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8204162780 con 0x7f8204159020 2024-04-17T08:52:48.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.075+0000 7f8209c00640 1 --2- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8204159020 0x7f820415ee30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009e50 tx=0x7f81f802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2968d1d89bd96fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.076+0000 7f8203e00640 1 -- 172.21.15.67:0/320617353 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f802ca80 con 0x7f8204159020 2024-04-17T08:52:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.076+0000 7f8203e00640 1 -- 172.21.15.67:0/320617353 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f802cc20 con 0x7f8204159020 2024-04-17T08:52:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.076+0000 7f8203e00640 1 -- 172.21.15.67:0/320617353 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f80343f0 con 0x7f8204159020 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 -- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8204159020 msgr2=0x7f820415ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 --2- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8204159020 0x7f820415ee30 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009e50 tx=0x7f81f802bee0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 -- 172.21.15.67:0/320617353 shutdown_connections 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 --2- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f82040af400 0x7f8204162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 --2- 172.21.15.67:0/320617353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8204159020 0x7f820415ee30 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 -- 172.21.15.67:0/320617353 >> 172.21.15.67:0/320617353 conn(0x7f82040b2190 msgr2=0x7f82040b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 -- 172.21.15.67:0/320617353 shutdown_connections 2024-04-17T08:52:48.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.077+0000 7f820b000640 1 -- 172.21.15.67:0/320617353 wait complete. 2024-04-17T08:52:48.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.078+0000 7f820b000640 1 Processor -- start 2024-04-17T08:52:48.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.079+0000 7f820b000640 1 -- start start 2024-04-17T08:52:48.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.079+0000 7f820b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f820406b8c0 0x7f82041600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820b000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82040af400 0x7f8204155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820b000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820b000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f820415f4f0 con 0x7f8204159020 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820b000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f820415f370 con 0x7f82040af400 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f82041631e0 con 0x7f820406b8c0 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55790/0 (socket says 172.21.15.67:55790) 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 -- 172.21.15.67:0/1056593127 learned_addr learned my addr 172.21.15.67:0/1056593127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f8209200640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82040af400 0x7f8204155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f8209c00640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f820406b8c0 0x7f82041600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 -- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f820406b8c0 msgr2=0x7f82041600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f820406b8c0 0x7f82041600b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 -- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82040af400 msgr2=0x7f8204155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82040af400 0x7f8204155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.080+0000 7f820a600640 1 -- 172.21.15.67:0/1056593127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82041607b0 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f820a600640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c3c0 tx=0x7f81fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc017070 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81fc0052d0 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8204160a90 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc003810 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.081+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8204161050 con 0x7f8204159020 2024-04-17T08:52:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.083+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8204068a10 con 0x7f8204159020 2024-04-17T08:52:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.084+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f81fc01a070 con 0x7f8204159020 2024-04-17T08:52:48.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.085+0000 7f8201600640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 0x7f81e406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:48.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.085+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f81fc08b2b0 con 0x7f8204159020 2024-04-17T08:52:48.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.086+0000 7f8209c00640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 0x7f81e406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:48.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.087+0000 7f8209c00640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 0x7f81e406eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f81f8035000 tx=0x7f81f8030000 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:48.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.090+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f81fc059370 con 0x7f8204159020 2024-04-17T08:52:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.336+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f8204072de0 con 0x7f8204159020 2024-04-17T08:52:48.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.820+0000 7f8201600640 1 -- 172.21.15.67:0/1056593127 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v143) ==== 122+0+0 (secure 0 0 0) 0x7f81fc058c70 con 0x7f8204159020 2024-04-17T08:52:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 msgr2=0x7f81e406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 0x7f81e406eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f81f8035000 tx=0x7f81f8030000 comp rx=0 tx=0).stop 2024-04-17T08:52:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 msgr2=0x7f820415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c3c0 tx=0x7f81fc00c880 comp rx=0 tx=0).stop 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 shutdown_connections 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f81e406cb30 0x7f81e406eff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8204159020 0x7f820415c7c0 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82040af400 0x7f8204155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 --2- 172.21.15.67:0/1056593127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f820406b8c0 0x7f82041600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.823+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 >> 172.21.15.67:0/1056593127 conn(0x7f82040b2190 msgr2=0x7f82040b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.824+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 shutdown_connections 2024-04-17T08:52:48.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:48.824+0000 7f820b000640 1 -- 172.21.15.67:0/1056593127 wait complete. 2024-04-17T08:52:48.845 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:52:48.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:49.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.065+0000 7f2f3f200640 1 Processor -- start 2024-04-17T08:52:49.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.066+0000 7f2f3f200640 1 -- start start 2024-04-17T08:52:49.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.067+0000 7f2f3f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.067+0000 7f2f3f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2f4006fce0 0x7f2f4006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.067+0000 7f2f3f200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f4015bc80 con 0x7f2f4006be70 2024-04-17T08:52:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.067+0000 7f2f3f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4015bf40 con 0x7f2f4006fce0 2024-04-17T08:52:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.067+0000 7f2f3f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4015c220 con 0x7f2f4015b570 2024-04-17T08:52:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.068+0000 7f2f3de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.068+0000 7f2f3d400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2f4006fce0 0x7f2f4006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.068+0000 7f2f3de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4006f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:52656/0 (socket says 172.21.15.67:52656) 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.068+0000 7f2f3de00640 1 -- 172.21.15.67:0/9412174 learned_addr learned my addr 172.21.15.67:0/9412174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.069+0000 7f2f3d400640 1 -- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 msgr2=0x7f2f4006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.069+0000 7f2f3d400640 1 --2- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4006f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.069+0000 7f2f3d400640 1 -- 172.21.15.67:0/9412174 >> v1:172.21.15.187:6789/0 conn(0x7f2f4006be70 legacy=0x7f2f4006ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.069+0000 7f2f3d400640 1 -- 172.21.15.67:0/9412174 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4015c500 con 0x7f2f4006fce0 2024-04-17T08:52:49.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.070+0000 7f2f3d400640 1 --2- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f4006b930 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0097a0 tx=0x7f2f2c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684b29de9c28a90a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.070+0000 7f2f3ca00640 1 -- 172.21.15.67:0/9412174 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031780 con 0x7f2f4006fce0 2024-04-17T08:52:49.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.070+0000 7f2f3ca00640 1 -- 172.21.15.67:0/9412174 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2c037070 con 0x7f2f4006fce0 2024-04-17T08:52:49.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.070+0000 7f2f3ca00640 1 -- 172.21.15.67:0/9412174 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031b40 con 0x7f2f4006fce0 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 -- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 msgr2=0x7f2f4006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 --2- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f4006b930 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0097a0 tx=0x7f2f2c02b820 comp rx=0 tx=0).stop 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 -- 172.21.15.67:0/9412174 shutdown_connections 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 --2- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f4006b930 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 --2- 172.21.15.67:0/9412174 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 -- 172.21.15.67:0/9412174 >> 172.21.15.67:0/9412174 conn(0x7f2f40067d10 msgr2=0x7f2f400b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 -- 172.21.15.67:0/9412174 shutdown_connections 2024-04-17T08:52:49.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.071+0000 7f2f3f200640 1 -- 172.21.15.67:0/9412174 wait complete. 2024-04-17T08:52:49.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.073+0000 7f2f3f200640 1 Processor -- start 2024-04-17T08:52:49.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.073+0000 7f2f3f200640 1 -- start start 2024-04-17T08:52:49.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f40154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4006c460 con 0x7f2f4006be70 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4006c2e0 con 0x7f2f4006fce0 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f400b47b0 con 0x7f2f4015b570 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.074+0000 7f2f3de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55800/0 (socket says 172.21.15.67:55800) 2024-04-17T08:52:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 -- 172.21.15.67:0/4176762346 learned_addr learned my addr 172.21.15.67:0/4176762346 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:49.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3d400640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f40154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3e800640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 -- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 msgr2=0x7f2f4014df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4014df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 -- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 msgr2=0x7f2f40154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f40154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 -- 172.21.15.67:0/4176762346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f401f8320 con 0x7f2f4006be70 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.075+0000 7f2f3de00640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009330 tx=0x7f2f3000e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.076+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3000b0d0 con 0x7f2f4006be70 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.076+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f30019070 con 0x7f2f4006be70 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.076+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3000b470 con 0x7f2f4006be70 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.076+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f401f8600 con 0x7f2f4006be70 2024-04-17T08:52:49.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.076+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f401f8bc0 con 0x7f2f4006be70 2024-04-17T08:52:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.077+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f40068a10 con 0x7f2f4006be70 2024-04-17T08:52:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.078+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f2f30005d10 con 0x7f2f4006be70 2024-04-17T08:52:49.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.079+0000 7f2f36000640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 0x7f2f2006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.079+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f2f3008c330 con 0x7f2f4006be70 2024-04-17T08:52:49.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.080+0000 7f2f3d400640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 0x7f2f2006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.081+0000 7f2f3d400640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 0x7f2f2006f1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c036000 tx=0x7f2f2c03d040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.084+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2f3005a330 con 0x7f2f4006be70 2024-04-17T08:52:49.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.329+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f4006f350 con 0x7f2f4006be70 2024-04-17T08:52:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.330+0000 7f2f36000640 1 -- 172.21.15.67:0/4176762346 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7f2f30059c30 con 0x7f2f4006be70 2024-04-17T08:52:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:49.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:52:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:52:49.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 msgr2=0x7f2f2006f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 0x7f2f2006f1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c036000 tx=0x7f2f2c03d040 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 msgr2=0x7f2f40155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009330 tx=0x7f2f3000e490 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 shutdown_connections 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f2006cd00 0x7f2f2006f1c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f4015b570 0x7f2f4014df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f4006fce0 0x7f2f40154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 --2- 172.21.15.67:0/4176762346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006be70 0x7f2f40155890 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 >> 172.21.15.67:0/4176762346 conn(0x7f2f40067d10 msgr2=0x7f2f400cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 shutdown_connections 2024-04-17T08:52:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.333+0000 7f2f3f200640 1 -- 172.21.15.67:0/4176762346 wait complete. 2024-04-17T08:52:49.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:52:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.575+0000 7f105a600640 1 Processor -- start 2024-04-17T08:52:49.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.575+0000 7f105a600640 1 -- start start 2024-04-17T08:52:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f105a600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f10540cd5a0 0x7f10540c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f105a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10540cc500 con 0x7f10540ce4a0 2024-04-17T08:52:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f105a600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10540c71e0 con 0x7f1054161e70 2024-04-17T08:52:49.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f105a600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10540cc7a0 con 0x7f10540cd5a0 2024-04-17T08:52:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f1053e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f10540ce4a0 0x7f10540c54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45404/0 (socket says 172.21.15.67:45404) 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.577+0000 7f1053e00640 1 -- 172.21.15.67:0/942045869 learned_addr learned my addr 172.21.15.67:0/942045869 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052a00640 1 --2- 172.21.15.67:0/942045869 >> v2:172.21.15.187:3300/0 conn(0x7f10540cd5a0 0x7f10540c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3125815458 0 0) 0x7f10540cc500 con 0x7f10540ce4a0 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1038003520 con 0x7f10540ce4a0 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1792559082 0 0) 0x7f1038003520 con 0x7f10540ce4a0 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 >> v2:172.21.15.187:3300/0 conn(0x7f10540cd5a0 msgr2=0x7f10540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 --2- 172.21.15.67:0/942045869 >> v2:172.21.15.187:3300/0 conn(0x7f10540cd5a0 0x7f10540c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 >> v1:172.21.15.187:6789/0 conn(0x7f1054161e70 legacy=0x7f10540cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10540cca80 con 0x7f10540ce4a0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f104c001a10 con 0x7f10540ce4a0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 >> v1:172.21.15.67:6789/0 conn(0x7f10540ce4a0 legacy=0x7f10540c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10380039b0 0x7f1038003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038004800 0x7f1038004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.578+0000 7f1052000640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f103800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10540cc7a0 con 0x7f10380051c0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f103800c060 con 0x7f1038004800 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f103800c260 con 0x7f10380039b0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f103800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1052a00640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038004800 0x7f1038004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053400640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10380039b0 0x7f1038003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 -- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10380039b0 msgr2=0x7f1038003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10380039b0 0x7f1038003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 -- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038004800 msgr2=0x7f1038004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:49.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1038004800 0x7f1038004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.579+0000 7f1053e00640 1 -- 172.21.15.67:0/942045869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1038003520 con 0x7f10380051c0 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.580+0000 7f1053e00640 1 -- 172.21.15.67:0/942045869 reap_dead start 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.580+0000 7f1053e00640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f103800b9c0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f104c00a850 tx=0x7f1038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874b59f42ad232da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00a4a0 con 0x7f10380051c0 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c034630 con 0x7f10380051c0 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f1052000640 1 -- 172.21.15.67:0/942045869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c034a00 con 0x7f10380051c0 2024-04-17T08:52:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f105a600640 1 -- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 msgr2=0x7f103800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f105a600640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f103800b9c0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f104c00a850 tx=0x7f1038005830 comp rx=0 tx=0).stop 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f105a600640 1 -- 172.21.15.67:0/942045869 shutdown_connections 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f105a600640 1 --2- 172.21.15.67:0/942045869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f103800b9c0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.581+0000 7f105a600640 1 -- 172.21.15.67:0/942045869 >> 172.21.15.67:0/942045869 conn(0x7f105406bf50 msgr2=0x7f10540b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.582+0000 7f105a600640 1 -- 172.21.15.67:0/942045869 shutdown_connections 2024-04-17T08:52:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.582+0000 7f105a600640 1 -- 172.21.15.67:0/942045869 wait complete. 2024-04-17T08:52:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.583+0000 7f105a600640 1 Processor -- start 2024-04-17T08:52:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.584+0000 7f105a600640 1 -- start start 2024-04-17T08:52:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10540b3110 0x7f105406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10541f36e0 0x7f10541f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10540c6d80 con 0x7f10380051c0 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10540c6c00 con 0x7f10540b3110 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f105a600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10540c6f00 con 0x7f10541f36e0 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55816/0 (socket says 172.21.15.67:55816) 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 -- 172.21.15.67:0/1315633833 learned_addr learned my addr 172.21.15.67:0/1315633833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 -- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10541f36e0 msgr2=0x7f10541f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10541f36e0 0x7f10541f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 -- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10540b3110 msgr2=0x7f105406ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10540b3110 0x7f105406ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.585+0000 7f1053400640 1 -- 172.21.15.67:0/1315633833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10541f5280 con 0x7f10380051c0 2024-04-17T08:52:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f1053400640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f104400e480 tx=0x7f104400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10440052e0 con 0x7f10380051c0 2024-04-17T08:52:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1044018070 con 0x7f10380051c0 2024-04-17T08:52:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044003830 con 0x7f10380051c0 2024-04-17T08:52:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10540b36b0 con 0x7f10380051c0 2024-04-17T08:52:49.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.586+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10540b3c70 con 0x7f10380051c0 2024-04-17T08:52:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.587+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1054068b90 con 0x7f10380051c0 2024-04-17T08:52:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.588+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f10440039d0 con 0x7f10380051c0 2024-04-17T08:52:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.589+0000 7f104b400640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 0x7f103406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:49.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.589+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f104408bd40 con 0x7f10380051c0 2024-04-17T08:52:49.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.590+0000 7f1052a00640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 0x7f103406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:49.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.591+0000 7f1052a00640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 0x7f103406eff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f10540ced30 tx=0x7f1040008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:49.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.594+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f10541f5280 con 0x7f10380051c0 2024-04-17T08:52:49.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.806+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1054157750 con 0x7f10380051c0 2024-04-17T08:52:49.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.808+0000 7f104b400640 1 -- 172.21.15.67:0/1315633833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) ==== 74+0+16685 (secure 0 0 0) 0x7f1044059db0 con 0x7f10380051c0 2024-04-17T08:52:49.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:49.809 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":272,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:52:46.765039+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 msgr2=0x7f103406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 0x7f103406eff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f10540ced30 tx=0x7f1040008040 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 msgr2=0x7f10541f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f104400e480 tx=0x7f104400e940 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 shutdown_connections 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f103406cb30 0x7f103406eff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10541f36e0 0x7f10541f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10540b3110 0x7f105406ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.811+0000 7f105a600640 1 --2- 172.21.15.67:0/1315633833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10380051c0 0x7f10541f4d40 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.812+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 >> 172.21.15.67:0/1315633833 conn(0x7f105406bf50 msgr2=0x7f10541569c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:49.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.812+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 shutdown_connections 2024-04-17T08:52:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:49.812+0000 7f105a600640 1 -- 172.21.15.67:0/1315633833 wait complete. 2024-04-17T08:52:49.830 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:52:49.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:52:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.051+0000 7fc766c00640 1 Processor -- start 2024-04-17T08:52:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.051+0000 7fc766c00640 1 -- start start 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.052+0000 7fc766c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc76006feb0 0x7fc76006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.052+0000 7fc766c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc76006be70 0x7fc76015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.052+0000 7fc766c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76015bc80 con 0x7fc76015b570 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.052+0000 7fc766c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc76015bf40 con 0x7fc76006be70 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.052+0000 7fc766c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc76015c220 con 0x7fc76006feb0 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc76006be70 0x7fc76015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc76006be70 0x7fc76015f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55832/0 (socket says 172.21.15.67:55832) 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 -- 172.21.15.67:0/3536357834 learned_addr learned my addr 172.21.15.67:0/3536357834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:50.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc765800640 1 --1- 172.21.15.67:0/3536357834 >> v1:172.21.15.67:6789/0 conn(0x7fc76015b570 0x7fc76006f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45416/0 (socket says 172.21.15.67:45416) 2024-04-17T08:52:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc764e00640 1 --2- 172.21.15.67:0/3536357834 >> v2:172.21.15.187:3300/0 conn(0x7fc76006feb0 0x7fc76006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 -- 172.21.15.67:0/3536357834 >> v2:172.21.15.187:3300/0 conn(0x7fc76006feb0 msgr2=0x7fc76006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 --2- 172.21.15.67:0/3536357834 >> v2:172.21.15.187:3300/0 conn(0x7fc76006feb0 0x7fc76006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 -- 172.21.15.67:0/3536357834 >> v1:172.21.15.67:6789/0 conn(0x7fc76015b570 legacy=0x7fc76006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc766200640 1 -- 172.21.15.67:0/3536357834 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76015c500 con 0x7fc76006be70 2024-04-17T08:52:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.053+0000 7fc75fe00640 1 -- 172.21.15.67:0/3536357834 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2483952833 0 0) 0x7fc76015bc80 con 0x7fc76015b570 2024-04-17T08:52:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.055+0000 7fc766200640 1 --2- 172.21.15.67:0/3536357834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76006be70 0x7fc76015f140 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fc760073040 tx=0x7fc75802e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ab5a6040c8369e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.055+0000 7fc75fe00640 1 -- 172.21.15.67:0/3536357834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75802ea90 con 0x7fc76006be70 2024-04-17T08:52:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.055+0000 7fc75fe00640 1 -- 172.21.15.67:0/3536357834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75802ec30 con 0x7fc76006be70 2024-04-17T08:52:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.055+0000 7fc75fe00640 1 -- 172.21.15.67:0/3536357834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758009600 con 0x7fc76006be70 2024-04-17T08:52:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 -- 172.21.15.67:0/3536357834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76006be70 msgr2=0x7fc76015f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 --2- 172.21.15.67:0/3536357834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76006be70 0x7fc76015f140 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fc760073040 tx=0x7fc75802e0f0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 -- 172.21.15.67:0/3536357834 shutdown_connections 2024-04-17T08:52:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 --2- 172.21.15.67:0/3536357834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76006be70 0x7fc76015f140 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 --2- 172.21.15.67:0/3536357834 >> v2:172.21.15.187:3300/0 conn(0x7fc76006feb0 0x7fc76006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 -- 172.21.15.67:0/3536357834 >> 172.21.15.67:0/3536357834 conn(0x7fc760067d10 msgr2=0x7fc7600b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 -- 172.21.15.67:0/3536357834 shutdown_connections 2024-04-17T08:52:50.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.056+0000 7fc766c00640 1 -- 172.21.15.67:0/3536357834 wait complete. 2024-04-17T08:52:50.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.058+0000 7fc766c00640 1 Processor -- start 2024-04-17T08:52:50.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.058+0000 7fc766c00640 1 -- start start 2024-04-17T08:52:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76006be70 0x7fc760146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 0x7fc7601f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7600b4930 con 0x7fc76015b570 2024-04-17T08:52:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc7600b47b0 con 0x7fc76006be70 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc76006c280 con 0x7fc76006feb0 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55838/0 (socket says 172.21.15.67:55838) 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.059+0000 7fc766200640 1 -- 172.21.15.67:0/2673276312 learned_addr learned my addr 172.21.15.67:0/2673276312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc765800640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76006be70 0x7fc760146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc764e00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 0x7fc7601f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 -- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 msgr2=0x7fc7601f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 0x7fc7601f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 -- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76006be70 msgr2=0x7fc760146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76006be70 0x7fc760146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 -- 172.21.15.67:0/2673276312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7601473b0 con 0x7fc76015b570 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc764e00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 0x7fc7601f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:50.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.060+0000 7fc766200640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fc758038040 tx=0x7fc758004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.061+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75802ea90 con 0x7fc76015b570 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.061+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc758045070 con 0x7fc76015b570 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.061+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7601f4f80 con 0x7fc76015b570 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.061+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75800b720 con 0x7fc76015b570 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.061+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc760158c90 con 0x7fc76015b570 2024-04-17T08:52:50.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.062+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc760068a10 con 0x7fc76015b570 2024-04-17T08:52:50.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.063+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7fc758036a60 con 0x7fc76015b570 2024-04-17T08:52:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.064+0000 7fc75d600640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 0x7fc74006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.065+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fc7580aef40 con 0x7fc76015b570 2024-04-17T08:52:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.065+0000 7fc765800640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 0x7fc74006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.066+0000 7fc765800640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 0x7fc74006eff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc754000f60 tx=0x7fc754007c00 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.070+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc7601473b0 con 0x7fc76015b570 2024-04-17T08:52:50.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.270+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 --> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc76006f350 con 0x7fc74006cb30 2024-04-17T08:52:50.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.293+0000 7fc75d600640 1 -- 172.21.15.67:0/2673276312 <== mgr.8339 v2:172.21.15.187:6832/1143620423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252439 (secure 0 0 0) 0x7fc76006f350 con 0x7fc74006cb30 2024-04-17T08:52:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 msgr2=0x7fc74006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 0x7fc74006eff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc754000f60 tx=0x7fc754007c00 comp rx=0 tx=0).stop 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 msgr2=0x7fc7600aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fc758038040 tx=0x7fc758004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 shutdown_connections 2024-04-17T08:52:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc74006cb30 0x7fc74006eff0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc76015b570 0x7fc7600aebc0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc76006feb0 0x7fc7601f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 --2- 172.21.15.67:0/2673276312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc76006be70 0x7fc760146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 >> 172.21.15.67:0/2673276312 conn(0x7fc760067d10 msgr2=0x7fc76006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 shutdown_connections 2024-04-17T08:52:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.299+0000 7fc766c00640 1 -- 172.21.15.67:0/2673276312 wait complete. 2024-04-17T08:52:50.319 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-17T08:52:49.439704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":693,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435532,"kb_used_data":27840,"kb_used_omap":12,"kb_used_meta":407539,"kb_avail":754539188,"statfs":{"total":773094113280,"available":772648128512,"internally_reserved":0,"allocated":28508160,"data_stored":14637288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":417320496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.692488"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809653+0000","last_change":"2024-04-17T08:52:45.806244+0000","last_active":"2024-04-17T08:52:45.809653+0000","last_peered":"2024-04-17T08:52:45.809653+0000","last_clean":"2024-04-17T08:52:45.809653+0000","last_became_active":"2024-04-17T08:52:45.805938+0000","last_became_peered":"2024-04-17T08:52:45.805938+0000","last_unstale":"2024-04-17T08:52:45.809653+0000","last_undegraded":"2024-04-17T08:52:45.809653+0000","last_fullsized":"2024-04-17T08:52:45.809653+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:36.504830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808504+0000","last_change":"2024-04-17T08:52:45.807742+0000","last_active":"2024-04-17T08:52:45.808504+0000","last_peered":"2024-04-17T08:52:45.808504+0000","last_clean":"2024-04-17T08:52:45.808504+0000","last_became_active":"2024-04-17T08:52:45.807288+0000","last_became_peered":"2024-04-17T08:52:45.807288+0000","last_unstale":"2024-04-17T08:52:45.808504+0000","last_undegraded":"2024-04-17T08:52:45.808504+0000","last_fullsized":"2024-04-17T08:52:45.808504+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:45.534422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791527+0000","last_change":"2024-04-17T08:52:45.789926+0000","last_active":"2024-04-17T08:52:45.791527+0000","last_peered":"2024-04-17T08:52:45.791527+0000","last_clean":"2024-04-17T08:52:45.791527+0000","last_became_active":"2024-04-17T08:52:45.789563+0000","last_became_peered":"2024-04-17T08:52:45.789563+0000","last_unstale":"2024-04-17T08:52:45.791527+0000","last_undegraded":"2024-04-17T08:52:45.791527+0000","last_fullsized":"2024-04-17T08:52:45.791527+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:43.696094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815633+0000","last_change":"2024-04-17T08:52:45.806281+0000","last_active":"2024-04-17T08:52:45.815633+0000","last_peered":"2024-04-17T08:52:45.815633+0000","last_clean":"2024-04-17T08:52:45.815633+0000","last_became_active":"2024-04-17T08:52:45.805443+0000","last_became_peered":"2024-04-17T08:52:45.805443+0000","last_unstale":"2024-04-17T08:52:45.815633+0000","last_undegraded":"2024-04-17T08:52:45.815633+0000","last_fullsized":"2024-04-17T08:52:45.815633+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:55.444975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805586+0000","last_change":"2024-04-17T08:52:45.803305+0000","last_active":"2024-04-17T08:52:45.805586+0000","last_peered":"2024-04-17T08:52:45.805586+0000","last_clean":"2024-04-17T08:52:45.805586+0000","last_became_active":"2024-04-17T08:52:45.802973+0000","last_became_peered":"2024-04-17T08:52:45.802973+0000","last_unstale":"2024-04-17T08:52:45.805586+0000","last_undegraded":"2024-04-17T08:52:45.805586+0000","last_fullsized":"2024-04-17T08:52:45.805586+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:50:06.823016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805879+0000","last_change":"2024-04-17T08:52:45.803387+0000","last_active":"2024-04-17T08:52:45.805879+0000","last_peered":"2024-04-17T08:52:45.805879+0000","last_clean":"2024-04-17T08:52:45.805879+0000","last_became_active":"2024-04-17T08:52:45.803064+0000","last_became_peered":"2024-04-17T08:52:45.803064+0000","last_unstale":"2024-04-17T08:52:45.805879+0000","last_undegraded":"2024-04-17T08:52:45.805879+0000","last_fullsized":"2024-04-17T08:52:45.805879+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:05:03.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809245+0000","last_change":"2024-04-17T08:52:45.804073+0000","last_active":"2024-04-17T08:52:45.809245+0000","last_peered":"2024-04-17T08:52:45.809245+0000","last_clean":"2024-04-17T08:52:45.809245+0000","last_became_active":"2024-04-17T08:52:45.803705+0000","last_became_peered":"2024-04-17T08:52:45.803705+0000","last_unstale":"2024-04-17T08:52:45.809245+0000","last_undegraded":"2024-04-17T08:52:45.809245+0000","last_fullsized":"2024-04-17T08:52:45.809245+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:16.650512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815084+0000","last_change":"2024-04-17T08:52:45.813614+0000","last_active":"2024-04-17T08:52:45.815084+0000","last_peered":"2024-04-17T08:52:45.815084+0000","last_clean":"2024-04-17T08:52:45.815084+0000","last_became_active":"2024-04-17T08:52:45.813138+0000","last_became_peered":"2024-04-17T08:52:45.813138+0000","last_unstale":"2024-04-17T08:52:45.815084+0000","last_undegraded":"2024-04-17T08:52:45.815084+0000","last_fullsized":"2024-04-17T08:52:45.815084+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:19:55.111177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.801370+0000","last_change":"2024-04-17T08:52:45.798804+0000","last_active":"2024-04-17T08:52:45.801370+0000","last_peered":"2024-04-17T08:52:45.801370+0000","last_clean":"2024-04-17T08:52:45.801370+0000","last_became_active":"2024-04-17T08:52:45.798107+0000","last_became_peered":"2024-04-17T08:52:45.798107+0000","last_unstale":"2024-04-17T08:52:45.801370+0000","last_undegraded":"2024-04-17T08:52:45.801370+0000","last_fullsized":"2024-04-17T08:52:45.801370+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:10.216665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795833+0000","last_change":"2024-04-17T08:52:45.789435+0000","last_active":"2024-04-17T08:52:45.795833+0000","last_peered":"2024-04-17T08:52:45.795833+0000","last_clean":"2024-04-17T08:52:45.795833+0000","last_became_active":"2024-04-17T08:52:45.789070+0000","last_became_peered":"2024-04-17T08:52:45.789070+0000","last_unstale":"2024-04-17T08:52:45.795833+0000","last_undegraded":"2024-04-17T08:52:45.795833+0000","last_fullsized":"2024-04-17T08:52:45.795833+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:12.457858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812920+0000","last_change":"2024-04-17T08:52:45.809361+0000","last_active":"2024-04-17T08:52:45.812920+0000","last_peered":"2024-04-17T08:52:45.812920+0000","last_clean":"2024-04-17T08:52:45.812920+0000","last_became_active":"2024-04-17T08:52:45.809059+0000","last_became_peered":"2024-04-17T08:52:45.809059+0000","last_unstale":"2024-04-17T08:52:45.812920+0000","last_undegraded":"2024-04-17T08:52:45.812920+0000","last_fullsized":"2024-04-17T08:52:45.812920+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:35:03.494039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.788631+0000","last_change":"2024-04-17T08:52:45.784885+0000","last_active":"2024-04-17T08:52:45.788631+0000","last_peered":"2024-04-17T08:52:45.788631+0000","last_clean":"2024-04-17T08:52:45.788631+0000","last_became_active":"2024-04-17T08:52:45.784588+0000","last_became_peered":"2024-04-17T08:52:45.784588+0000","last_unstale":"2024-04-17T08:52:45.788631+0000","last_undegraded":"2024-04-17T08:52:45.788631+0000","last_fullsized":"2024-04-17T08:52:45.788631+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:47.995195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.787222+0000","last_change":"2024-04-17T08:52:45.781912+0000","last_active":"2024-04-17T08:52:45.787222+0000","last_peered":"2024-04-17T08:52:45.787222+0000","last_clean":"2024-04-17T08:52:45.787222+0000","last_became_active":"2024-04-17T08:52:45.781586+0000","last_became_peered":"2024-04-17T08:52:45.781586+0000","last_unstale":"2024-04-17T08:52:45.787222+0000","last_undegraded":"2024-04-17T08:52:45.787222+0000","last_fullsized":"2024-04-17T08:52:45.787222+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:07.643506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813961+0000","last_change":"2024-04-17T08:52:45.802842+0000","last_active":"2024-04-17T08:52:45.813961+0000","last_peered":"2024-04-17T08:52:45.813961+0000","last_clean":"2024-04-17T08:52:45.813961+0000","last_became_active":"2024-04-17T08:52:45.802529+0000","last_became_peered":"2024-04-17T08:52:45.802529+0000","last_unstale":"2024-04-17T08:52:45.813961+0000","last_undegraded":"2024-04-17T08:52:45.813961+0000","last_fullsized":"2024-04-17T08:52:45.813961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:03.656548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809302+0000","last_change":"2024-04-17T08:52:45.806197+0000","last_active":"2024-04-17T08:52:45.809302+0000","last_peered":"2024-04-17T08:52:45.809302+0000","last_clean":"2024-04-17T08:52:45.809302+0000","last_became_active":"2024-04-17T08:52:45.805572+0000","last_became_peered":"2024-04-17T08:52:45.805572+0000","last_unstale":"2024-04-17T08:52:45.809302+0000","last_undegraded":"2024-04-17T08:52:45.809302+0000","last_fullsized":"2024-04-17T08:52:45.809302+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:45.967149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813767+0000","last_change":"2024-04-17T08:52:45.812640+0000","last_active":"2024-04-17T08:52:45.813767+0000","last_peered":"2024-04-17T08:52:45.813767+0000","last_clean":"2024-04-17T08:52:45.813767+0000","last_became_active":"2024-04-17T08:52:45.812350+0000","last_became_peered":"2024-04-17T08:52:45.812350+0000","last_unstale":"2024-04-17T08:52:45.813767+0000","last_undegraded":"2024-04-17T08:52:45.813767+0000","last_fullsized":"2024-04-17T08:52:45.813767+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:46.173369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.802420+0000","last_change":"2024-04-17T08:52:45.801355+0000","last_active":"2024-04-17T08:52:45.802420+0000","last_peered":"2024-04-17T08:52:45.802420+0000","last_clean":"2024-04-17T08:52:45.802420+0000","last_became_active":"2024-04-17T08:52:45.798491+0000","last_became_peered":"2024-04-17T08:52:45.798491+0000","last_unstale":"2024-04-17T08:52:45.802420+0000","last_undegraded":"2024-04-17T08:52:45.802420+0000","last_fullsized":"2024-04-17T08:52:45.802420+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:24.947037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807327+0000","last_change":"2024-04-17T08:52:45.803226+0000","last_active":"2024-04-17T08:52:45.807327+0000","last_peered":"2024-04-17T08:52:45.807327+0000","last_clean":"2024-04-17T08:52:45.807327+0000","last_became_active":"2024-04-17T08:52:45.802932+0000","last_became_peered":"2024-04-17T08:52:45.802932+0000","last_unstale":"2024-04-17T08:52:45.807327+0000","last_undegraded":"2024-04-17T08:52:45.807327+0000","last_fullsized":"2024-04-17T08:52:45.807327+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:15.650336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813574+0000","last_change":"2024-04-17T08:52:45.811041+0000","last_active":"2024-04-17T08:52:45.813574+0000","last_peered":"2024-04-17T08:52:45.813574+0000","last_clean":"2024-04-17T08:52:45.813574+0000","last_became_active":"2024-04-17T08:52:45.810742+0000","last_became_peered":"2024-04-17T08:52:45.810742+0000","last_unstale":"2024-04-17T08:52:45.813574+0000","last_undegraded":"2024-04-17T08:52:45.813574+0000","last_fullsized":"2024-04-17T08:52:45.813574+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:38.526225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813732+0000","last_change":"2024-04-17T08:52:45.811410+0000","last_active":"2024-04-17T08:52:45.813732+0000","last_peered":"2024-04-17T08:52:45.813732+0000","last_clean":"2024-04-17T08:52:45.813732+0000","last_became_active":"2024-04-17T08:52:45.810039+0000","last_became_peered":"2024-04-17T08:52:45.810039+0000","last_unstale":"2024-04-17T08:52:45.813732+0000","last_undegraded":"2024-04-17T08:52:45.813732+0000","last_fullsized":"2024-04-17T08:52:45.813732+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:08.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.788636+0000","last_change":"2024-04-17T08:52:45.785070+0000","last_active":"2024-04-17T08:52:45.788636+0000","last_peered":"2024-04-17T08:52:45.788636+0000","last_clean":"2024-04-17T08:52:45.788636+0000","last_became_active":"2024-04-17T08:52:45.784740+0000","last_became_peered":"2024-04-17T08:52:45.784740+0000","last_unstale":"2024-04-17T08:52:45.788636+0000","last_undegraded":"2024-04-17T08:52:45.788636+0000","last_fullsized":"2024-04-17T08:52:45.788636+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:20.400133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805901+0000","last_change":"2024-04-17T08:52:45.804234+0000","last_active":"2024-04-17T08:52:45.805901+0000","last_peered":"2024-04-17T08:52:45.805901+0000","last_clean":"2024-04-17T08:52:45.805901+0000","last_became_active":"2024-04-17T08:52:45.803866+0000","last_became_peered":"2024-04-17T08:52:45.803866+0000","last_unstale":"2024-04-17T08:52:45.805901+0000","last_undegraded":"2024-04-17T08:52:45.805901+0000","last_fullsized":"2024-04-17T08:52:45.805901+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:27.259709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809520+0000","last_change":"2024-04-17T08:52:45.808357+0000","last_active":"2024-04-17T08:52:45.809520+0000","last_peered":"2024-04-17T08:52:45.809520+0000","last_clean":"2024-04-17T08:52:45.809520+0000","last_became_active":"2024-04-17T08:52:45.807482+0000","last_became_peered":"2024-04-17T08:52:45.807482+0000","last_unstale":"2024-04-17T08:52:45.809520+0000","last_undegraded":"2024-04-17T08:52:45.809520+0000","last_fullsized":"2024-04-17T08:52:45.809520+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.581776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805545+0000","last_change":"2024-04-17T08:52:45.803241+0000","last_active":"2024-04-17T08:52:45.805545+0000","last_peered":"2024-04-17T08:52:45.805545+0000","last_clean":"2024-04-17T08:52:45.805545+0000","last_became_active":"2024-04-17T08:52:45.799072+0000","last_became_peered":"2024-04-17T08:52:45.799072+0000","last_unstale":"2024-04-17T08:52:45.805545+0000","last_undegraded":"2024-04-17T08:52:45.805545+0000","last_fullsized":"2024-04-17T08:52:45.805545+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:43.022668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.802920+0000","last_change":"2024-04-17T08:52:45.801863+0000","last_active":"2024-04-17T08:52:45.802920+0000","last_peered":"2024-04-17T08:52:45.802920+0000","last_clean":"2024-04-17T08:52:45.802920+0000","last_became_active":"2024-04-17T08:52:45.801346+0000","last_became_peered":"2024-04-17T08:52:45.801346+0000","last_unstale":"2024-04-17T08:52:45.802920+0000","last_undegraded":"2024-04-17T08:52:45.802920+0000","last_fullsized":"2024-04-17T08:52:45.802920+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:27.951601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.790122+0000","last_change":"2024-04-17T08:52:45.787077+0000","last_active":"2024-04-17T08:52:45.790122+0000","last_peered":"2024-04-17T08:52:45.790122+0000","last_clean":"2024-04-17T08:52:45.790122+0000","last_became_active":"2024-04-17T08:52:45.785130+0000","last_became_peered":"2024-04-17T08:52:45.785130+0000","last_unstale":"2024-04-17T08:52:45.790122+0000","last_undegraded":"2024-04-17T08:52:45.790122+0000","last_fullsized":"2024-04-17T08:52:45.790122+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:14:19.453418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814369+0000","last_change":"2024-04-17T08:52:45.811133+0000","last_active":"2024-04-17T08:52:45.814369+0000","last_peered":"2024-04-17T08:52:45.814369+0000","last_clean":"2024-04-17T08:52:45.814369+0000","last_became_active":"2024-04-17T08:52:45.807009+0000","last_became_peered":"2024-04-17T08:52:45.807009+0000","last_unstale":"2024-04-17T08:52:45.814369+0000","last_undegraded":"2024-04-17T08:52:45.814369+0000","last_fullsized":"2024-04-17T08:52:45.814369+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:55.444975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795814+0000","last_change":"2024-04-17T08:52:45.790478+0000","last_active":"2024-04-17T08:52:45.795814+0000","last_peered":"2024-04-17T08:52:45.795814+0000","last_clean":"2024-04-17T08:52:45.795814+0000","last_became_active":"2024-04-17T08:52:45.789914+0000","last_became_peered":"2024-04-17T08:52:45.789914+0000","last_unstale":"2024-04-17T08:52:45.795814+0000","last_undegraded":"2024-04-17T08:52:45.795814+0000","last_fullsized":"2024-04-17T08:52:45.795814+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:45.967149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812688+0000","last_change":"2024-04-17T08:52:45.805647+0000","last_active":"2024-04-17T08:52:45.812688+0000","last_peered":"2024-04-17T08:52:45.812688+0000","last_clean":"2024-04-17T08:52:45.812688+0000","last_became_active":"2024-04-17T08:52:45.805335+0000","last_became_peered":"2024-04-17T08:52:45.805335+0000","last_unstale":"2024-04-17T08:52:45.812688+0000","last_undegraded":"2024-04-17T08:52:45.812688+0000","last_fullsized":"2024-04-17T08:52:45.812688+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:33.108709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809267+0000","last_change":"2024-04-17T08:52:45.803839+0000","last_active":"2024-04-17T08:52:45.809267+0000","last_peered":"2024-04-17T08:52:45.809267+0000","last_clean":"2024-04-17T08:52:45.809267+0000","last_became_active":"2024-04-17T08:52:45.803535+0000","last_became_peered":"2024-04-17T08:52:45.803535+0000","last_unstale":"2024-04-17T08:52:45.809267+0000","last_undegraded":"2024-04-17T08:52:45.809267+0000","last_fullsized":"2024-04-17T08:52:45.809267+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:56.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.800175+0000","last_change":"2024-04-17T08:52:45.798116+0000","last_active":"2024-04-17T08:52:45.800175+0000","last_peered":"2024-04-17T08:52:45.800175+0000","last_clean":"2024-04-17T08:52:45.800175+0000","last_became_active":"2024-04-17T08:52:45.797818+0000","last_became_peered":"2024-04-17T08:52:45.797818+0000","last_unstale":"2024-04-17T08:52:45.800175+0000","last_undegraded":"2024-04-17T08:52:45.800175+0000","last_fullsized":"2024-04-17T08:52:45.800175+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:22.726576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807183+0000","last_change":"2024-04-17T08:52:45.796201+0000","last_active":"2024-04-17T08:52:45.807183+0000","last_peered":"2024-04-17T08:52:45.807183+0000","last_clean":"2024-04-17T08:52:45.807183+0000","last_became_active":"2024-04-17T08:52:45.794965+0000","last_became_peered":"2024-04-17T08:52:45.794965+0000","last_unstale":"2024-04-17T08:52:45.807183+0000","last_undegraded":"2024-04-17T08:52:45.807183+0000","last_fullsized":"2024-04-17T08:52:45.807183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:25.873820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"271'4","reported_seq":16,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.810089+0000","last_change":"2024-04-17T08:52:44.786017+0000","last_active":"2024-04-17T08:52:45.810089+0000","last_peered":"2024-04-17T08:52:45.810089+0000","last_clean":"2024-04-17T08:52:45.810089+0000","last_became_active":"2024-04-17T08:52:44.785722+0000","last_became_peered":"2024-04-17T08:52:44.785722+0000","last_unstale":"2024-04-17T08:52:45.810089+0000","last_undegraded":"2024-04-17T08:52:45.810089+0000","last_fullsized":"2024-04-17T08:52:45.810089+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:33.812816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814599+0000","last_change":"2024-04-17T08:52:45.813087+0000","last_active":"2024-04-17T08:52:45.814599+0000","last_peered":"2024-04-17T08:52:45.814599+0000","last_clean":"2024-04-17T08:52:45.814599+0000","last_became_active":"2024-04-17T08:52:45.810176+0000","last_became_peered":"2024-04-17T08:52:45.810176+0000","last_unstale":"2024-04-17T08:52:45.814599+0000","last_undegraded":"2024-04-17T08:52:45.814599+0000","last_fullsized":"2024-04-17T08:52:45.814599+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:10.130950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806893+0000","last_change":"2024-04-17T08:52:44.778826+0000","last_active":"2024-04-17T08:52:45.806893+0000","last_peered":"2024-04-17T08:52:45.806893+0000","last_clean":"2024-04-17T08:52:45.806893+0000","last_became_active":"2024-04-17T08:52:44.778501+0000","last_became_peered":"2024-04-17T08:52:44.778501+0000","last_unstale":"2024-04-17T08:52:45.806893+0000","last_undegraded":"2024-04-17T08:52:45.806893+0000","last_fullsized":"2024-04-17T08:52:45.806893+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:14:26.167346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806509+0000","last_change":"2024-04-17T08:52:45.801588+0000","last_active":"2024-04-17T08:52:45.806509+0000","last_peered":"2024-04-17T08:52:45.806509+0000","last_clean":"2024-04-17T08:52:45.806509+0000","last_became_active":"2024-04-17T08:52:45.799430+0000","last_became_peered":"2024-04-17T08:52:45.799430+0000","last_unstale":"2024-04-17T08:52:45.806509+0000","last_undegraded":"2024-04-17T08:52:45.806509+0000","last_fullsized":"2024-04-17T08:52:45.806509+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:22.430148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806020+0000","last_change":"2024-04-17T08:52:44.787868+0000","last_active":"2024-04-17T08:52:45.806020+0000","last_peered":"2024-04-17T08:52:45.806020+0000","last_clean":"2024-04-17T08:52:45.806020+0000","last_became_active":"2024-04-17T08:52:44.786958+0000","last_became_peered":"2024-04-17T08:52:44.786958+0000","last_unstale":"2024-04-17T08:52:45.806020+0000","last_undegraded":"2024-04-17T08:52:45.806020+0000","last_fullsized":"2024-04-17T08:52:45.806020+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:51.368864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804411+0000","last_change":"2024-04-17T08:52:45.799638+0000","last_active":"2024-04-17T08:52:45.804411+0000","last_peered":"2024-04-17T08:52:45.804411+0000","last_clean":"2024-04-17T08:52:45.804411+0000","last_became_active":"2024-04-17T08:52:45.799330+0000","last_became_peered":"2024-04-17T08:52:45.799330+0000","last_unstale":"2024-04-17T08:52:45.804411+0000","last_undegraded":"2024-04-17T08:52:45.804411+0000","last_fullsized":"2024-04-17T08:52:45.804411+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:34.157322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814703+0000","last_change":"2024-04-17T08:52:44.794136+0000","last_active":"2024-04-17T08:52:45.814703+0000","last_peered":"2024-04-17T08:52:45.814703+0000","last_clean":"2024-04-17T08:52:45.814703+0000","last_became_active":"2024-04-17T08:52:44.793742+0000","last_became_peered":"2024-04-17T08:52:44.793742+0000","last_unstale":"2024-04-17T08:52:45.814703+0000","last_undegraded":"2024-04-17T08:52:45.814703+0000","last_fullsized":"2024-04-17T08:52:45.814703+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:48.350187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.800987+0000","last_change":"2024-04-17T08:52:45.798881+0000","last_active":"2024-04-17T08:52:45.800987+0000","last_peered":"2024-04-17T08:52:45.800987+0000","last_clean":"2024-04-17T08:52:45.800987+0000","last_became_active":"2024-04-17T08:52:45.797000+0000","last_became_peered":"2024-04-17T08:52:45.797000+0000","last_unstale":"2024-04-17T08:52:45.800987+0000","last_undegraded":"2024-04-17T08:52:45.800987+0000","last_fullsized":"2024-04-17T08:52:45.800987+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:40:38.486256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813114+0000","last_change":"2024-04-17T08:52:44.794571+0000","last_active":"2024-04-17T08:52:45.813114+0000","last_peered":"2024-04-17T08:52:45.813114+0000","last_clean":"2024-04-17T08:52:45.813114+0000","last_became_active":"2024-04-17T08:52:44.794234+0000","last_became_peered":"2024-04-17T08:52:44.794234+0000","last_unstale":"2024-04-17T08:52:45.813114+0000","last_undegraded":"2024-04-17T08:52:45.813114+0000","last_fullsized":"2024-04-17T08:52:45.813114+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:22.360922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813852+0000","last_change":"2024-04-17T08:52:45.809998+0000","last_active":"2024-04-17T08:52:45.813852+0000","last_peered":"2024-04-17T08:52:45.813852+0000","last_clean":"2024-04-17T08:52:45.813852+0000","last_became_active":"2024-04-17T08:52:45.809720+0000","last_became_peered":"2024-04-17T08:52:45.809720+0000","last_unstale":"2024-04-17T08:52:45.813852+0000","last_undegraded":"2024-04-17T08:52:45.813852+0000","last_fullsized":"2024-04-17T08:52:45.813852+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:03.656548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.811963+0000","last_change":"2024-04-17T08:52:44.783789+0000","last_active":"2024-04-17T08:52:45.811963+0000","last_peered":"2024-04-17T08:52:45.811963+0000","last_clean":"2024-04-17T08:52:45.811963+0000","last_became_active":"2024-04-17T08:52:44.783293+0000","last_became_peered":"2024-04-17T08:52:44.783293+0000","last_unstale":"2024-04-17T08:52:45.811963+0000","last_undegraded":"2024-04-17T08:52:45.811963+0000","last_fullsized":"2024-04-17T08:52:45.811963+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:51.365742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.801326+0000","last_change":"2024-04-17T08:52:45.799397+0000","last_active":"2024-04-17T08:52:45.801326+0000","last_peered":"2024-04-17T08:52:45.801326+0000","last_clean":"2024-04-17T08:52:45.801326+0000","last_became_active":"2024-04-17T08:52:45.798121+0000","last_became_peered":"2024-04-17T08:52:45.798121+0000","last_unstale":"2024-04-17T08:52:45.801326+0000","last_undegraded":"2024-04-17T08:52:45.801326+0000","last_fullsized":"2024-04-17T08:52:45.801326+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:16.158903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770556+0000","last_change":"2024-04-17T08:52:44.790792+0000","last_active":"2024-04-17T08:52:45.770556+0000","last_peered":"2024-04-17T08:52:45.770556+0000","last_clean":"2024-04-17T08:52:45.770556+0000","last_became_active":"2024-04-17T08:52:44.790116+0000","last_became_peered":"2024-04-17T08:52:44.790116+0000","last_unstale":"2024-04-17T08:52:45.770556+0000","last_undegraded":"2024-04-17T08:52:45.770556+0000","last_fullsized":"2024-04-17T08:52:45.770556+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:19.130136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791864+0000","last_change":"2024-04-17T08:52:45.783329+0000","last_active":"2024-04-17T08:52:45.791864+0000","last_peered":"2024-04-17T08:52:45.791864+0000","last_clean":"2024-04-17T08:52:45.791864+0000","last_became_active":"2024-04-17T08:52:45.782842+0000","last_became_peered":"2024-04-17T08:52:45.782842+0000","last_unstale":"2024-04-17T08:52:45.791864+0000","last_undegraded":"2024-04-17T08:52:45.791864+0000","last_fullsized":"2024-04-17T08:52:45.791864+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:31.491513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770524+0000","last_change":"2024-04-17T08:52:44.784249+0000","last_active":"2024-04-17T08:52:45.770524+0000","last_peered":"2024-04-17T08:52:45.770524+0000","last_clean":"2024-04-17T08:52:45.770524+0000","last_became_active":"2024-04-17T08:52:44.783749+0000","last_became_peered":"2024-04-17T08:52:44.783749+0000","last_unstale":"2024-04-17T08:52:45.770524+0000","last_undegraded":"2024-04-17T08:52:45.770524+0000","last_fullsized":"2024-04-17T08:52:45.770524+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:04.758592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.797512+0000","last_change":"2024-04-17T08:52:45.793087+0000","last_active":"2024-04-17T08:52:45.797512+0000","last_peered":"2024-04-17T08:52:45.797512+0000","last_clean":"2024-04-17T08:52:45.797512+0000","last_became_active":"2024-04-17T08:52:45.792836+0000","last_became_peered":"2024-04-17T08:52:45.792836+0000","last_unstale":"2024-04-17T08:52:45.797512+0000","last_undegraded":"2024-04-17T08:52:45.797512+0000","last_fullsized":"2024-04-17T08:52:45.797512+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:13.826400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.778980+0000","last_change":"2024-04-17T08:52:44.788711+0000","last_active":"2024-04-17T08:52:45.778980+0000","last_peered":"2024-04-17T08:52:45.778980+0000","last_clean":"2024-04-17T08:52:45.778980+0000","last_became_active":"2024-04-17T08:52:44.788367+0000","last_became_peered":"2024-04-17T08:52:44.788367+0000","last_unstale":"2024-04-17T08:52:45.778980+0000","last_undegraded":"2024-04-17T08:52:45.778980+0000","last_fullsized":"2024-04-17T08:52:45.778980+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:48.350187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815680+0000","last_change":"2024-04-17T08:52:45.807559+0000","last_active":"2024-04-17T08:52:45.815680+0000","last_peered":"2024-04-17T08:52:45.815680+0000","last_clean":"2024-04-17T08:52:45.815680+0000","last_became_active":"2024-04-17T08:52:45.802969+0000","last_became_peered":"2024-04-17T08:52:45.802969+0000","last_unstale":"2024-04-17T08:52:45.815680+0000","last_undegraded":"2024-04-17T08:52:45.815680+0000","last_fullsized":"2024-04-17T08:52:45.815680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:08.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.792407+0000","last_change":"2024-04-17T08:52:44.791840+0000","last_active":"2024-04-17T08:52:45.792407+0000","last_peered":"2024-04-17T08:52:45.792407+0000","last_clean":"2024-04-17T08:52:45.792407+0000","last_became_active":"2024-04-17T08:52:44.790900+0000","last_became_peered":"2024-04-17T08:52:44.790900+0000","last_unstale":"2024-04-17T08:52:45.792407+0000","last_undegraded":"2024-04-17T08:52:45.792407+0000","last_fullsized":"2024-04-17T08:52:45.792407+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:23:04.311098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813973+0000","last_change":"2024-04-17T08:52:45.810697+0000","last_active":"2024-04-17T08:52:45.813973+0000","last_peered":"2024-04-17T08:52:45.813973+0000","last_clean":"2024-04-17T08:52:45.813973+0000","last_became_active":"2024-04-17T08:52:45.809697+0000","last_became_peered":"2024-04-17T08:52:45.809697+0000","last_unstale":"2024-04-17T08:52:45.813973+0000","last_undegraded":"2024-04-17T08:52:45.813973+0000","last_fullsized":"2024-04-17T08:52:45.813973+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:58.663244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"271'3","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809389+0000","last_change":"2024-04-17T08:52:44.795395+0000","last_active":"2024-04-17T08:52:45.809389+0000","last_peered":"2024-04-17T08:52:45.809389+0000","last_clean":"2024-04-17T08:52:45.809389+0000","last_became_active":"2024-04-17T08:52:44.795081+0000","last_became_peered":"2024-04-17T08:52:44.795081+0000","last_unstale":"2024-04-17T08:52:45.809389+0000","last_undegraded":"2024-04-17T08:52:45.809389+0000","last_fullsized":"2024-04-17T08:52:45.809389+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:31:36.722960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795773+0000","last_change":"2024-04-17T08:52:45.789450+0000","last_active":"2024-04-17T08:52:45.795773+0000","last_peered":"2024-04-17T08:52:45.795773+0000","last_clean":"2024-04-17T08:52:45.795773+0000","last_became_active":"2024-04-17T08:52:45.788971+0000","last_became_peered":"2024-04-17T08:52:45.788971+0000","last_unstale":"2024-04-17T08:52:45.795773+0000","last_undegraded":"2024-04-17T08:52:45.795773+0000","last_fullsized":"2024-04-17T08:52:45.795773+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:41.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"271'6","reported_seq":18,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.817006+0000","last_change":"2024-04-17T08:52:44.792222+0000","last_active":"2024-04-17T08:52:45.817006+0000","last_peered":"2024-04-17T08:52:45.817006+0000","last_clean":"2024-04-17T08:52:45.817006+0000","last_became_active":"2024-04-17T08:52:44.792006+0000","last_became_peered":"2024-04-17T08:52:44.792006+0000","last_unstale":"2024-04-17T08:52:45.817006+0000","last_undegraded":"2024-04-17T08:52:45.817006+0000","last_fullsized":"2024-04-17T08:52:45.817006+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:36.142621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813058+0000","last_change":"2024-04-17T08:52:45.809705+0000","last_active":"2024-04-17T08:52:45.813058+0000","last_peered":"2024-04-17T08:52:45.813058+0000","last_clean":"2024-04-17T08:52:45.813058+0000","last_became_active":"2024-04-17T08:52:45.809405+0000","last_became_peered":"2024-04-17T08:52:45.809405+0000","last_unstale":"2024-04-17T08:52:45.813058+0000","last_undegraded":"2024-04-17T08:52:45.813058+0000","last_fullsized":"2024-04-17T08:52:45.813058+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:41.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809094+0000","last_change":"2024-04-17T08:52:44.781025+0000","last_active":"2024-04-17T08:52:45.809094+0000","last_peered":"2024-04-17T08:52:45.809094+0000","last_clean":"2024-04-17T08:52:45.809094+0000","last_became_active":"2024-04-17T08:52:44.780345+0000","last_became_peered":"2024-04-17T08:52:44.780345+0000","last_unstale":"2024-04-17T08:52:45.809094+0000","last_undegraded":"2024-04-17T08:52:45.809094+0000","last_fullsized":"2024-04-17T08:52:45.809094+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:52.340674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798184+0000","last_change":"2024-04-17T08:52:45.792085+0000","last_active":"2024-04-17T08:52:45.798184+0000","last_peered":"2024-04-17T08:52:45.798184+0000","last_clean":"2024-04-17T08:52:45.798184+0000","last_became_active":"2024-04-17T08:52:45.791812+0000","last_became_peered":"2024-04-17T08:52:45.791812+0000","last_unstale":"2024-04-17T08:52:45.798184+0000","last_undegraded":"2024-04-17T08:52:45.798184+0000","last_fullsized":"2024-04-17T08:52:45.798184+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:48.509427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.782295+0000","last_change":"2024-04-17T08:52:44.789701+0000","last_active":"2024-04-17T08:52:45.782295+0000","last_peered":"2024-04-17T08:52:45.782295+0000","last_clean":"2024-04-17T08:52:45.782295+0000","last_became_active":"2024-04-17T08:52:44.789335+0000","last_became_peered":"2024-04-17T08:52:44.789335+0000","last_unstale":"2024-04-17T08:52:45.782295+0000","last_undegraded":"2024-04-17T08:52:45.782295+0000","last_fullsized":"2024-04-17T08:52:45.782295+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:03.310641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.803442+0000","last_change":"2024-04-17T08:52:45.802122+0000","last_active":"2024-04-17T08:52:45.803442+0000","last_peered":"2024-04-17T08:52:45.803442+0000","last_clean":"2024-04-17T08:52:45.803442+0000","last_became_active":"2024-04-17T08:52:45.801833+0000","last_became_peered":"2024-04-17T08:52:45.801833+0000","last_unstale":"2024-04-17T08:52:45.803442+0000","last_undegraded":"2024-04-17T08:52:45.803442+0000","last_fullsized":"2024-04-17T08:52:45.803442+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:10:23.451300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.775983+0000","last_change":"2024-04-17T08:52:44.788118+0000","last_active":"2024-04-17T08:52:45.775983+0000","last_peered":"2024-04-17T08:52:45.775983+0000","last_clean":"2024-04-17T08:52:45.775983+0000","last_became_active":"2024-04-17T08:52:44.787120+0000","last_became_peered":"2024-04-17T08:52:44.787120+0000","last_unstale":"2024-04-17T08:52:45.775983+0000","last_undegraded":"2024-04-17T08:52:45.775983+0000","last_fullsized":"2024-04-17T08:52:45.775983+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:17.372666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809284+0000","last_change":"2024-04-17T08:52:45.806000+0000","last_active":"2024-04-17T08:52:45.809284+0000","last_peered":"2024-04-17T08:52:45.809284+0000","last_clean":"2024-04-17T08:52:45.809284+0000","last_became_active":"2024-04-17T08:52:45.805651+0000","last_became_peered":"2024-04-17T08:52:45.805651+0000","last_unstale":"2024-04-17T08:52:45.809284+0000","last_undegraded":"2024-04-17T08:52:45.809284+0000","last_fullsized":"2024-04-17T08:52:45.809284+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:12.457858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"271'8","reported_seq":20,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812526+0000","last_change":"2024-04-17T08:52:44.796437+0000","last_active":"2024-04-17T08:52:45.812526+0000","last_peered":"2024-04-17T08:52:45.812526+0000","last_clean":"2024-04-17T08:52:45.812526+0000","last_became_active":"2024-04-17T08:52:44.796075+0000","last_became_peered":"2024-04-17T08:52:44.796075+0000","last_unstale":"2024-04-17T08:52:45.812526+0000","last_undegraded":"2024-04-17T08:52:45.812526+0000","last_fullsized":"2024-04-17T08:52:45.812526+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:26.759899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791147+0000","last_change":"2024-04-17T08:52:45.788018+0000","last_active":"2024-04-17T08:52:45.791147+0000","last_peered":"2024-04-17T08:52:45.791147+0000","last_clean":"2024-04-17T08:52:45.791147+0000","last_became_active":"2024-04-17T08:52:45.787761+0000","last_became_peered":"2024-04-17T08:52:45.787761+0000","last_unstale":"2024-04-17T08:52:45.791147+0000","last_undegraded":"2024-04-17T08:52:45.791147+0000","last_fullsized":"2024-04-17T08:52:45.791147+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:47.148301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814151+0000","last_change":"2024-04-17T08:52:44.788336+0000","last_active":"2024-04-17T08:52:45.814151+0000","last_peered":"2024-04-17T08:52:45.814151+0000","last_clean":"2024-04-17T08:52:45.814151+0000","last_became_active":"2024-04-17T08:52:44.786747+0000","last_became_peered":"2024-04-17T08:52:44.786747+0000","last_unstale":"2024-04-17T08:52:45.814151+0000","last_undegraded":"2024-04-17T08:52:45.814151+0000","last_fullsized":"2024-04-17T08:52:45.814151+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:32.087413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804477+0000","last_change":"2024-04-17T08:52:45.800440+0000","last_active":"2024-04-17T08:52:45.804477+0000","last_peered":"2024-04-17T08:52:45.804477+0000","last_clean":"2024-04-17T08:52:45.804477+0000","last_became_active":"2024-04-17T08:52:45.800139+0000","last_became_peered":"2024-04-17T08:52:45.800139+0000","last_unstale":"2024-04-17T08:52:45.804477+0000","last_undegraded":"2024-04-17T08:52:45.804477+0000","last_fullsized":"2024-04-17T08:52:45.804477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:46:10.741528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815918+0000","last_change":"2024-04-17T08:52:44.791544+0000","last_active":"2024-04-17T08:52:45.815918+0000","last_peered":"2024-04-17T08:52:45.815918+0000","last_clean":"2024-04-17T08:52:45.815918+0000","last_became_active":"2024-04-17T08:52:44.791189+0000","last_became_peered":"2024-04-17T08:52:44.791189+0000","last_unstale":"2024-04-17T08:52:45.815918+0000","last_undegraded":"2024-04-17T08:52:45.815918+0000","last_fullsized":"2024-04-17T08:52:45.815918+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:07.782565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.799738+0000","last_change":"2024-04-17T08:52:45.792797+0000","last_active":"2024-04-17T08:52:45.799738+0000","last_peered":"2024-04-17T08:52:45.799738+0000","last_clean":"2024-04-17T08:52:45.799738+0000","last_became_active":"2024-04-17T08:52:45.792492+0000","last_became_peered":"2024-04-17T08:52:45.792492+0000","last_unstale":"2024-04-17T08:52:45.799738+0000","last_undegraded":"2024-04-17T08:52:45.799738+0000","last_fullsized":"2024-04-17T08:52:45.799738+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:20:05.687948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815698+0000","last_change":"2024-04-17T08:52:44.793351+0000","last_active":"2024-04-17T08:52:45.815698+0000","last_peered":"2024-04-17T08:52:45.815698+0000","last_clean":"2024-04-17T08:52:45.815698+0000","last_became_active":"2024-04-17T08:52:44.791062+0000","last_became_peered":"2024-04-17T08:52:44.791062+0000","last_unstale":"2024-04-17T08:52:45.815698+0000","last_undegraded":"2024-04-17T08:52:45.815698+0000","last_fullsized":"2024-04-17T08:52:45.815698+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:55.182502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805454+0000","last_change":"2024-04-17T08:52:45.803230+0000","last_active":"2024-04-17T08:52:45.805454+0000","last_peered":"2024-04-17T08:52:45.805454+0000","last_clean":"2024-04-17T08:52:45.805454+0000","last_became_active":"2024-04-17T08:52:45.802914+0000","last_became_peered":"2024-04-17T08:52:45.802914+0000","last_unstale":"2024-04-17T08:52:45.805454+0000","last_undegraded":"2024-04-17T08:52:45.805454+0000","last_fullsized":"2024-04-17T08:52:45.805454+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:24.733697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.776793+0000","last_change":"2024-04-17T08:52:44.796757+0000","last_active":"2024-04-17T08:52:45.776793+0000","last_peered":"2024-04-17T08:52:45.776793+0000","last_clean":"2024-04-17T08:52:45.776793+0000","last_became_active":"2024-04-17T08:52:44.796461+0000","last_became_peered":"2024-04-17T08:52:44.796461+0000","last_unstale":"2024-04-17T08:52:45.776793+0000","last_undegraded":"2024-04-17T08:52:45.776793+0000","last_fullsized":"2024-04-17T08:52:45.776793+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:55.905650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793297+0000","last_change":"2024-04-17T08:52:45.792588+0000","last_active":"2024-04-17T08:52:45.793297+0000","last_peered":"2024-04-17T08:52:45.793297+0000","last_clean":"2024-04-17T08:52:45.793297+0000","last_became_active":"2024-04-17T08:52:45.792296+0000","last_became_peered":"2024-04-17T08:52:45.792296+0000","last_unstale":"2024-04-17T08:52:45.793297+0000","last_undegraded":"2024-04-17T08:52:45.793297+0000","last_fullsized":"2024-04-17T08:52:45.793297+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:22.701177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808439+0000","last_change":"2024-04-17T08:52:44.774370+0000","last_active":"2024-04-17T08:52:45.808439+0000","last_peered":"2024-04-17T08:52:45.808439+0000","last_clean":"2024-04-17T08:52:45.808439+0000","last_became_active":"2024-04-17T08:52:44.773885+0000","last_became_peered":"2024-04-17T08:52:44.773885+0000","last_unstale":"2024-04-17T08:52:45.808439+0000","last_undegraded":"2024-04-17T08:52:45.808439+0000","last_fullsized":"2024-04-17T08:52:45.808439+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:43:14.491703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798133+0000","last_change":"2024-04-17T08:52:45.787050+0000","last_active":"2024-04-17T08:52:45.798133+0000","last_peered":"2024-04-17T08:52:45.798133+0000","last_clean":"2024-04-17T08:52:45.798133+0000","last_became_active":"2024-04-17T08:52:45.786621+0000","last_became_peered":"2024-04-17T08:52:45.786621+0000","last_unstale":"2024-04-17T08:52:45.798133+0000","last_undegraded":"2024-04-17T08:52:45.798133+0000","last_fullsized":"2024-04-17T08:52:45.798133+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:40:21.323003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.779053+0000","last_change":"2024-04-17T08:52:44.792524+0000","last_active":"2024-04-17T08:52:45.779053+0000","last_peered":"2024-04-17T08:52:45.779053+0000","last_clean":"2024-04-17T08:52:45.779053+0000","last_became_active":"2024-04-17T08:52:44.792167+0000","last_became_peered":"2024-04-17T08:52:44.792167+0000","last_unstale":"2024-04-17T08:52:45.779053+0000","last_undegraded":"2024-04-17T08:52:45.779053+0000","last_fullsized":"2024-04-17T08:52:45.779053+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:05.888249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796591+0000","last_change":"2024-04-17T08:52:45.785382+0000","last_active":"2024-04-17T08:52:45.796591+0000","last_peered":"2024-04-17T08:52:45.796591+0000","last_clean":"2024-04-17T08:52:45.796591+0000","last_became_active":"2024-04-17T08:52:45.785026+0000","last_became_peered":"2024-04-17T08:52:45.785026+0000","last_unstale":"2024-04-17T08:52:45.796591+0000","last_undegraded":"2024-04-17T08:52:45.796591+0000","last_fullsized":"2024-04-17T08:52:45.796591+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:46.615853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.779524+0000","last_change":"2024-04-17T08:52:44.786171+0000","last_active":"2024-04-17T08:52:45.779524+0000","last_peered":"2024-04-17T08:52:45.779524+0000","last_clean":"2024-04-17T08:52:45.779524+0000","last_became_active":"2024-04-17T08:52:44.785710+0000","last_became_peered":"2024-04-17T08:52:44.785710+0000","last_unstale":"2024-04-17T08:52:45.779524+0000","last_undegraded":"2024-04-17T08:52:45.779524+0000","last_fullsized":"2024-04-17T08:52:45.779524+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:15.629216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807288+0000","last_change":"2024-04-17T08:52:45.797482+0000","last_active":"2024-04-17T08:52:45.807288+0000","last_peered":"2024-04-17T08:52:45.807288+0000","last_clean":"2024-04-17T08:52:45.807288+0000","last_became_active":"2024-04-17T08:52:45.795959+0000","last_became_peered":"2024-04-17T08:52:45.795959+0000","last_unstale":"2024-04-17T08:52:45.807288+0000","last_undegraded":"2024-04-17T08:52:45.807288+0000","last_fullsized":"2024-04-17T08:52:45.807288+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:22:23.578941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770708+0000","last_change":"2024-04-17T08:52:44.783643+0000","last_active":"2024-04-17T08:52:45.770708+0000","last_peered":"2024-04-17T08:52:45.770708+0000","last_clean":"2024-04-17T08:52:45.770708+0000","last_became_active":"2024-04-17T08:52:44.783303+0000","last_became_peered":"2024-04-17T08:52:44.783303+0000","last_unstale":"2024-04-17T08:52:45.770708+0000","last_undegraded":"2024-04-17T08:52:45.770708+0000","last_fullsized":"2024-04-17T08:52:45.770708+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:48.383757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815160+0000","last_change":"2024-04-17T08:52:45.814332+0000","last_active":"2024-04-17T08:52:45.815160+0000","last_peered":"2024-04-17T08:52:45.815160+0000","last_clean":"2024-04-17T08:52:45.815160+0000","last_became_active":"2024-04-17T08:52:45.812211+0000","last_became_peered":"2024-04-17T08:52:45.812211+0000","last_unstale":"2024-04-17T08:52:45.815160+0000","last_undegraded":"2024-04-17T08:52:45.815160+0000","last_fullsized":"2024-04-17T08:52:45.815160+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:22.756478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.816217+0000","last_change":"2024-04-17T08:52:44.799305+0000","last_active":"2024-04-17T08:52:45.816217+0000","last_peered":"2024-04-17T08:52:45.816217+0000","last_clean":"2024-04-17T08:52:45.816217+0000","last_became_active":"2024-04-17T08:52:44.798855+0000","last_became_peered":"2024-04-17T08:52:44.798855+0000","last_unstale":"2024-04-17T08:52:45.816217+0000","last_undegraded":"2024-04-17T08:52:45.816217+0000","last_fullsized":"2024-04-17T08:52:45.816217+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:27.955640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814785+0000","last_change":"2024-04-17T08:52:45.811183+0000","last_active":"2024-04-17T08:52:45.814785+0000","last_peered":"2024-04-17T08:52:45.814785+0000","last_clean":"2024-04-17T08:52:45.814785+0000","last_became_active":"2024-04-17T08:52:45.809747+0000","last_became_peered":"2024-04-17T08:52:45.809747+0000","last_unstale":"2024-04-17T08:52:45.814785+0000","last_undegraded":"2024-04-17T08:52:45.814785+0000","last_fullsized":"2024-04-17T08:52:45.814785+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:07.496274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.782644+0000","last_change":"2024-04-17T08:52:44.783014+0000","last_active":"2024-04-17T08:52:45.782644+0000","last_peered":"2024-04-17T08:52:45.782644+0000","last_clean":"2024-04-17T08:52:45.782644+0000","last_became_active":"2024-04-17T08:52:44.782651+0000","last_became_peered":"2024-04-17T08:52:44.782651+0000","last_unstale":"2024-04-17T08:52:45.782644+0000","last_undegraded":"2024-04-17T08:52:45.782644+0000","last_fullsized":"2024-04-17T08:52:45.782644+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:45.681524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814842+0000","last_change":"2024-04-17T08:52:45.812637+0000","last_active":"2024-04-17T08:52:45.814842+0000","last_peered":"2024-04-17T08:52:45.814842+0000","last_clean":"2024-04-17T08:52:45.814842+0000","last_became_active":"2024-04-17T08:52:45.810031+0000","last_became_peered":"2024-04-17T08:52:45.810031+0000","last_unstale":"2024-04-17T08:52:45.814842+0000","last_undegraded":"2024-04-17T08:52:45.814842+0000","last_fullsized":"2024-04-17T08:52:45.814842+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:57.407900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798994+0000","last_change":"2024-04-17T08:52:44.792514+0000","last_active":"2024-04-17T08:52:45.798994+0000","last_peered":"2024-04-17T08:52:45.798994+0000","last_clean":"2024-04-17T08:52:45.798994+0000","last_became_active":"2024-04-17T08:52:44.791211+0000","last_became_peered":"2024-04-17T08:52:44.791211+0000","last_unstale":"2024-04-17T08:52:45.798994+0000","last_undegraded":"2024-04-17T08:52:45.798994+0000","last_fullsized":"2024-04-17T08:52:45.798994+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:18:22.622174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804388+0000","last_change":"2024-04-17T08:52:45.793297+0000","last_active":"2024-04-17T08:52:45.804388+0000","last_peered":"2024-04-17T08:52:45.804388+0000","last_clean":"2024-04-17T08:52:45.804388+0000","last_became_active":"2024-04-17T08:52:45.793003+0000","last_became_peered":"2024-04-17T08:52:45.793003+0000","last_unstale":"2024-04-17T08:52:45.804388+0000","last_undegraded":"2024-04-17T08:52:45.804388+0000","last_fullsized":"2024-04-17T08:52:45.804388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:35.618353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795081+0000","last_change":"2024-04-17T08:52:44.795429+0000","last_active":"2024-04-17T08:52:45.795081+0000","last_peered":"2024-04-17T08:52:45.795081+0000","last_clean":"2024-04-17T08:52:45.795081+0000","last_became_active":"2024-04-17T08:52:44.795050+0000","last_became_peered":"2024-04-17T08:52:44.795050+0000","last_unstale":"2024-04-17T08:52:45.795081+0000","last_undegraded":"2024-04-17T08:52:45.795081+0000","last_fullsized":"2024-04-17T08:52:45.795081+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:56:39.800086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793257+0000","last_change":"2024-04-17T08:52:45.792489+0000","last_active":"2024-04-17T08:52:45.793257+0000","last_peered":"2024-04-17T08:52:45.793257+0000","last_clean":"2024-04-17T08:52:45.793257+0000","last_became_active":"2024-04-17T08:52:45.792119+0000","last_became_peered":"2024-04-17T08:52:45.792119+0000","last_unstale":"2024-04-17T08:52:45.793257+0000","last_undegraded":"2024-04-17T08:52:45.793257+0000","last_fullsized":"2024-04-17T08:52:45.793257+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:07.556062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796137+0000","last_change":"2024-04-17T08:52:44.795387+0000","last_active":"2024-04-17T08:52:45.796137+0000","last_peered":"2024-04-17T08:52:45.796137+0000","last_clean":"2024-04-17T08:52:45.796137+0000","last_became_active":"2024-04-17T08:52:44.793794+0000","last_became_peered":"2024-04-17T08:52:44.793794+0000","last_unstale":"2024-04-17T08:52:45.796137+0000","last_undegraded":"2024-04-17T08:52:45.796137+0000","last_fullsized":"2024-04-17T08:52:45.796137+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:55.905650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798204+0000","last_change":"2024-04-17T08:52:45.792651+0000","last_active":"2024-04-17T08:52:45.798204+0000","last_peered":"2024-04-17T08:52:45.798204+0000","last_clean":"2024-04-17T08:52:45.798204+0000","last_became_active":"2024-04-17T08:52:45.790732+0000","last_became_peered":"2024-04-17T08:52:45.790732+0000","last_unstale":"2024-04-17T08:52:45.798204+0000","last_undegraded":"2024-04-17T08:52:45.798204+0000","last_fullsized":"2024-04-17T08:52:45.798204+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:43:51.867535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770470+0000","last_change":"2024-04-17T08:52:44.779751+0000","last_active":"2024-04-17T08:52:45.770470+0000","last_peered":"2024-04-17T08:52:45.770470+0000","last_clean":"2024-04-17T08:52:45.770470+0000","last_became_active":"2024-04-17T08:52:44.777362+0000","last_became_peered":"2024-04-17T08:52:44.777362+0000","last_unstale":"2024-04-17T08:52:45.770470+0000","last_undegraded":"2024-04-17T08:52:45.770470+0000","last_fullsized":"2024-04-17T08:52:45.770470+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:14:29.894624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796615+0000","last_change":"2024-04-17T08:52:45.789551+0000","last_active":"2024-04-17T08:52:45.796615+0000","last_peered":"2024-04-17T08:52:45.796615+0000","last_clean":"2024-04-17T08:52:45.796615+0000","last_became_active":"2024-04-17T08:52:45.787902+0000","last_became_peered":"2024-04-17T08:52:45.787902+0000","last_unstale":"2024-04-17T08:52:45.796615+0000","last_undegraded":"2024-04-17T08:52:45.796615+0000","last_fullsized":"2024-04-17T08:52:45.796615+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:29.112824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770738+0000","last_change":"2024-04-17T08:52:44.777566+0000","last_active":"2024-04-17T08:52:45.770738+0000","last_peered":"2024-04-17T08:52:45.770738+0000","last_clean":"2024-04-17T08:52:45.770738+0000","last_became_active":"2024-04-17T08:52:44.776658+0000","last_became_peered":"2024-04-17T08:52:44.776658+0000","last_unstale":"2024-04-17T08:52:45.770738+0000","last_undegraded":"2024-04-17T08:52:45.770738+0000","last_fullsized":"2024-04-17T08:52:45.770738+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:18.960800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814782+0000","last_change":"2024-04-17T08:52:45.803953+0000","last_active":"2024-04-17T08:52:45.814782+0000","last_peered":"2024-04-17T08:52:45.814782+0000","last_clean":"2024-04-17T08:52:45.814782+0000","last_became_active":"2024-04-17T08:52:45.803662+0000","last_became_peered":"2024-04-17T08:52:45.803662+0000","last_unstale":"2024-04-17T08:52:45.814782+0000","last_undegraded":"2024-04-17T08:52:45.814782+0000","last_fullsized":"2024-04-17T08:52:45.814782+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:38.526225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.775762+0000","last_change":"2024-04-17T08:52:44.780482+0000","last_active":"2024-04-17T08:52:45.775762+0000","last_peered":"2024-04-17T08:52:45.775762+0000","last_clean":"2024-04-17T08:52:45.775762+0000","last_became_active":"2024-04-17T08:52:44.779902+0000","last_became_peered":"2024-04-17T08:52:44.779902+0000","last_unstale":"2024-04-17T08:52:45.775762+0000","last_undegraded":"2024-04-17T08:52:45.775762+0000","last_fullsized":"2024-04-17T08:52:45.775762+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:30.925823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808896+0000","last_change":"2024-04-17T08:52:45.807050+0000","last_active":"2024-04-17T08:52:45.808896+0000","last_peered":"2024-04-17T08:52:45.808896+0000","last_clean":"2024-04-17T08:52:45.808896+0000","last_became_active":"2024-04-17T08:52:45.806768+0000","last_became_peered":"2024-04-17T08:52:45.806768+0000","last_unstale":"2024-04-17T08:52:45.808896+0000","last_undegraded":"2024-04-17T08:52:45.808896+0000","last_fullsized":"2024-04-17T08:52:45.808896+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:47:27.662918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"268'2749","reported_seq":4037,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793765+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:52:45.793765+0000","last_peered":"2024-04-17T08:52:45.793765+0000","last_clean":"2024-04-17T08:52:45.793765+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:52:45.793765+0000","last_undegraded":"2024-04-17T08:52:45.793765+0000","last_fullsized":"2024-04-17T08:52:45.793765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2749,"ondisk_log_size":2749,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48976,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322864,"statfs":{"total":96636764160,"available":96586612736,"internally_reserved":0,"allocated":3997696,"data_stored":2268507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543318,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60896,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310944,"statfs":{"total":96636764160,"available":96574406656,"internally_reserved":0,"allocated":3420160,"data_stored":1685075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543317,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48428,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323412,"statfs":{"total":96636764160,"available":96587173888,"internally_reserved":0,"allocated":3436544,"data_stored":1687249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66380,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305460,"statfs":{"total":96636764160,"available":96568791040,"internally_reserved":0,"allocated":3403776,"data_stored":1678139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3428352,"data_stored":1686597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543318,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52576,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3420160,"data_stored":1685075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48396,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323444,"statfs":{"total":96636764160,"available":96587206656,"internally_reserved":0,"allocated":3403776,"data_stored":1678139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543317,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57296,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314544,"statfs":{"total":96636764160,"available":96578093056,"internally_reserved":0,"allocated":3997696,"data_stored":2268507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:52:50.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:52:50.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.588+0000 7f5c7b000640 1 Processor -- start 2024-04-17T08:52:50.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.589+0000 7f5c7b000640 1 -- start start 2024-04-17T08:52:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 0x7f5c7406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 0x7f5c740cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c74071d10 0x7f5c740721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5c7406ff90 con 0x7f5c74071d10 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5c740ceb00 con 0x7f5c740cd020 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.590+0000 7f5c7b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5c740cede0 con 0x7f5c7406f670 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c74071d10 0x7f5c740721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c74071d10 0x7f5c740721c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55842/0 (socket says 172.21.15.67:55842) 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 -- 172.21.15.67:0/3565123512 learned_addr learned my addr 172.21.15.67:0/3565123512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c73e00640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 0x7f5c7406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c73400640 1 --2- 172.21.15.67:0/3565123512 >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 0x7f5c740cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 -- 172.21.15.67:0/3565123512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 msgr2=0x7f5c7406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 0x7f5c7406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 -- 172.21.15.67:0/3565123512 >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 msgr2=0x7f5c740cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 --2- 172.21.15.67:0/3565123512 >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 0x7f5c740cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c79000640 1 -- 172.21.15.67:0/3565123512 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c740cf0c0 con 0x7f5c74071d10 2024-04-17T08:52:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c73e00640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 0x7f5c7406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:50.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.591+0000 7f5c73400640 1 --2- 172.21.15.67:0/3565123512 >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 0x7f5c740cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.592+0000 7f5c79000640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c74071d10 0x7f5c740721c0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f5c74072de0 tx=0x7f5c6c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603f8ea2d85e0d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.593+0000 7f5c72a00640 1 -- 172.21.15.67:0/3565123512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c035020 con 0x7f5c74071d10 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.593+0000 7f5c72a00640 1 -- 172.21.15.67:0/3565123512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c6c02d040 con 0x7f5c74071d10 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.593+0000 7f5c72a00640 1 -- 172.21.15.67:0/3565123512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c031a80 con 0x7f5c74071d10 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 -- 172.21.15.67:0/3565123512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c74071d10 msgr2=0x7f5c740721c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c74071d10 0x7f5c740721c0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f5c74072de0 tx=0x7f5c6c02b900 comp rx=0 tx=0).stop 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 -- 172.21.15.67:0/3565123512 shutdown_connections 2024-04-17T08:52:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c74071d10 0x7f5c740721c0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 --2- 172.21.15.67:0/3565123512 >> v2:172.21.15.187:3300/0 conn(0x7f5c740cd020 0x7f5c740cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 --2- 172.21.15.67:0/3565123512 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c7406f670 0x7f5c7406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 -- 172.21.15.67:0/3565123512 >> 172.21.15.67:0/3565123512 conn(0x7f5c7406bf50 msgr2=0x7f5c740b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 -- 172.21.15.67:0/3565123512 shutdown_connections 2024-04-17T08:52:50.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.594+0000 7f5c7b000640 1 -- 172.21.15.67:0/3565123512 wait complete. 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.596+0000 7f5c7b000640 1 Processor -- start 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.596+0000 7f5c7b000640 1 -- start start 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 0x7f5c7414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c74071d10 0x7f5c7415b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c740ce720 con 0x7f5c740cd020 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5c740cf680 con 0x7f5c7406f670 2024-04-17T08:52:50.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c7b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5c740ce8a0 con 0x7f5c74071d10 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55856/0 (socket says 172.21.15.67:55856) 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 -- 172.21.15.67:0/1370502310 learned_addr learned my addr 172.21.15.67:0/1370502310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c73e00640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 0x7f5c7414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 -- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c74071d10 msgr2=0x7f5c7415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c73400640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c74071d10 0x7f5c7415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c74071d10 0x7f5c7415b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 -- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 msgr2=0x7f5c7414f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 0x7f5c7414f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c79000640 1 -- 172.21.15.67:0/1370502310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c740ca240 con 0x7f5c740cd020 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.597+0000 7f5c73e00640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 0x7f5c7414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c79000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0090b0 tx=0x7f5c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c040070 con 0x7f5c740cd020 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c6c039b90 con 0x7f5c740cd020 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c035040 con 0x7f5c740cd020 2024-04-17T08:52:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c7415d6e0 con 0x7f5c740cd020 2024-04-17T08:52:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.598+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c7415dca0 con 0x7f5c740cd020 2024-04-17T08:52:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.599+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c74068b90 con 0x7f5c740cd020 2024-04-17T08:52:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.601+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f5c6c04b060 con 0x7f5c740cd020 2024-04-17T08:52:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.601+0000 7f5c6be00640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 0x7f5c5006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.601+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f5c6c0adcb0 con 0x7f5c740cd020 2024-04-17T08:52:50.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.602+0000 7f5c73e00640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 0x7f5c5006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:50.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.603+0000 7f5c73e00640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 0x7f5c5006eff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c741f5570 tx=0x7f5c64008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.606+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5c6c07bc90 con 0x7f5c740cd020 2024-04-17T08:52:50.701 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T08:52:50.701+0000 7f9d33a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:52:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.806+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 --> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c7406ccd0 con 0x7f5c5006cb30 2024-04-17T08:52:50.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.827+0000 7f5c6be00640 1 -- 172.21.15.67:0/1370502310 <== mgr.8339 v2:172.21.15.187:6832/1143620423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252439 (secure 0 0 0) 0x7f5c7406ccd0 con 0x7f5c5006cb30 2024-04-17T08:52:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:50.831 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 msgr2=0x7f5c5006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 0x7f5c5006eff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c741f5570 tx=0x7f5c64008040 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 msgr2=0x7f5c7415bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0090b0 tx=0x7f5c6c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 shutdown_connections 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f5c5006cb30 0x7f5c5006eff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c740cd020 0x7f5c7415bb50 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c74071d10 0x7f5c7415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 --2- 172.21.15.67:0/1370502310 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c7406f670 0x7f5c7414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:50.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 >> 172.21.15.67:0/1370502310 conn(0x7f5c7406bf50 msgr2=0x7f5c740aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:50.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 shutdown_connections 2024-04-17T08:52:50.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:50.833+0000 7f5c7b000640 1 -- 172.21.15.67:0/1370502310 wait complete. 2024-04-17T08:52:50.852 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-17T08:52:49.439704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":693,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435532,"kb_used_data":27840,"kb_used_omap":12,"kb_used_meta":407539,"kb_avail":754539188,"statfs":{"total":773094113280,"available":772648128512,"internally_reserved":0,"allocated":28508160,"data_stored":14637288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":417320496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":201,"num_write":63,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.692488"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809653+0000","last_change":"2024-04-17T08:52:45.806244+0000","last_active":"2024-04-17T08:52:45.809653+0000","last_peered":"2024-04-17T08:52:45.809653+0000","last_clean":"2024-04-17T08:52:45.809653+0000","last_became_active":"2024-04-17T08:52:45.805938+0000","last_became_peered":"2024-04-17T08:52:45.805938+0000","last_unstale":"2024-04-17T08:52:45.809653+0000","last_undegraded":"2024-04-17T08:52:45.809653+0000","last_fullsized":"2024-04-17T08:52:45.809653+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:54:36.504830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808504+0000","last_change":"2024-04-17T08:52:45.807742+0000","last_active":"2024-04-17T08:52:45.808504+0000","last_peered":"2024-04-17T08:52:45.808504+0000","last_clean":"2024-04-17T08:52:45.808504+0000","last_became_active":"2024-04-17T08:52:45.807288+0000","last_became_peered":"2024-04-17T08:52:45.807288+0000","last_unstale":"2024-04-17T08:52:45.808504+0000","last_undegraded":"2024-04-17T08:52:45.808504+0000","last_fullsized":"2024-04-17T08:52:45.808504+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:45.534422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791527+0000","last_change":"2024-04-17T08:52:45.789926+0000","last_active":"2024-04-17T08:52:45.791527+0000","last_peered":"2024-04-17T08:52:45.791527+0000","last_clean":"2024-04-17T08:52:45.791527+0000","last_became_active":"2024-04-17T08:52:45.789563+0000","last_became_peered":"2024-04-17T08:52:45.789563+0000","last_unstale":"2024-04-17T08:52:45.791527+0000","last_undegraded":"2024-04-17T08:52:45.791527+0000","last_fullsized":"2024-04-17T08:52:45.791527+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:43.696094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815633+0000","last_change":"2024-04-17T08:52:45.806281+0000","last_active":"2024-04-17T08:52:45.815633+0000","last_peered":"2024-04-17T08:52:45.815633+0000","last_clean":"2024-04-17T08:52:45.815633+0000","last_became_active":"2024-04-17T08:52:45.805443+0000","last_became_peered":"2024-04-17T08:52:45.805443+0000","last_unstale":"2024-04-17T08:52:45.815633+0000","last_undegraded":"2024-04-17T08:52:45.815633+0000","last_fullsized":"2024-04-17T08:52:45.815633+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:55.444975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805586+0000","last_change":"2024-04-17T08:52:45.803305+0000","last_active":"2024-04-17T08:52:45.805586+0000","last_peered":"2024-04-17T08:52:45.805586+0000","last_clean":"2024-04-17T08:52:45.805586+0000","last_became_active":"2024-04-17T08:52:45.802973+0000","last_became_peered":"2024-04-17T08:52:45.802973+0000","last_unstale":"2024-04-17T08:52:45.805586+0000","last_undegraded":"2024-04-17T08:52:45.805586+0000","last_fullsized":"2024-04-17T08:52:45.805586+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:50:06.823016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805879+0000","last_change":"2024-04-17T08:52:45.803387+0000","last_active":"2024-04-17T08:52:45.805879+0000","last_peered":"2024-04-17T08:52:45.805879+0000","last_clean":"2024-04-17T08:52:45.805879+0000","last_became_active":"2024-04-17T08:52:45.803064+0000","last_became_peered":"2024-04-17T08:52:45.803064+0000","last_unstale":"2024-04-17T08:52:45.805879+0000","last_undegraded":"2024-04-17T08:52:45.805879+0000","last_fullsized":"2024-04-17T08:52:45.805879+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:05:03.637273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809245+0000","last_change":"2024-04-17T08:52:45.804073+0000","last_active":"2024-04-17T08:52:45.809245+0000","last_peered":"2024-04-17T08:52:45.809245+0000","last_clean":"2024-04-17T08:52:45.809245+0000","last_became_active":"2024-04-17T08:52:45.803705+0000","last_became_peered":"2024-04-17T08:52:45.803705+0000","last_unstale":"2024-04-17T08:52:45.809245+0000","last_undegraded":"2024-04-17T08:52:45.809245+0000","last_fullsized":"2024-04-17T08:52:45.809245+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:16.650512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815084+0000","last_change":"2024-04-17T08:52:45.813614+0000","last_active":"2024-04-17T08:52:45.815084+0000","last_peered":"2024-04-17T08:52:45.815084+0000","last_clean":"2024-04-17T08:52:45.815084+0000","last_became_active":"2024-04-17T08:52:45.813138+0000","last_became_peered":"2024-04-17T08:52:45.813138+0000","last_unstale":"2024-04-17T08:52:45.815084+0000","last_undegraded":"2024-04-17T08:52:45.815084+0000","last_fullsized":"2024-04-17T08:52:45.815084+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:19:55.111177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.801370+0000","last_change":"2024-04-17T08:52:45.798804+0000","last_active":"2024-04-17T08:52:45.801370+0000","last_peered":"2024-04-17T08:52:45.801370+0000","last_clean":"2024-04-17T08:52:45.801370+0000","last_became_active":"2024-04-17T08:52:45.798107+0000","last_became_peered":"2024-04-17T08:52:45.798107+0000","last_unstale":"2024-04-17T08:52:45.801370+0000","last_undegraded":"2024-04-17T08:52:45.801370+0000","last_fullsized":"2024-04-17T08:52:45.801370+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:10.216665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795833+0000","last_change":"2024-04-17T08:52:45.789435+0000","last_active":"2024-04-17T08:52:45.795833+0000","last_peered":"2024-04-17T08:52:45.795833+0000","last_clean":"2024-04-17T08:52:45.795833+0000","last_became_active":"2024-04-17T08:52:45.789070+0000","last_became_peered":"2024-04-17T08:52:45.789070+0000","last_unstale":"2024-04-17T08:52:45.795833+0000","last_undegraded":"2024-04-17T08:52:45.795833+0000","last_fullsized":"2024-04-17T08:52:45.795833+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:12.457858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812920+0000","last_change":"2024-04-17T08:52:45.809361+0000","last_active":"2024-04-17T08:52:45.812920+0000","last_peered":"2024-04-17T08:52:45.812920+0000","last_clean":"2024-04-17T08:52:45.812920+0000","last_became_active":"2024-04-17T08:52:45.809059+0000","last_became_peered":"2024-04-17T08:52:45.809059+0000","last_unstale":"2024-04-17T08:52:45.812920+0000","last_undegraded":"2024-04-17T08:52:45.812920+0000","last_fullsized":"2024-04-17T08:52:45.812920+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:35:03.494039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.788631+0000","last_change":"2024-04-17T08:52:45.784885+0000","last_active":"2024-04-17T08:52:45.788631+0000","last_peered":"2024-04-17T08:52:45.788631+0000","last_clean":"2024-04-17T08:52:45.788631+0000","last_became_active":"2024-04-17T08:52:45.784588+0000","last_became_peered":"2024-04-17T08:52:45.784588+0000","last_unstale":"2024-04-17T08:52:45.788631+0000","last_undegraded":"2024-04-17T08:52:45.788631+0000","last_fullsized":"2024-04-17T08:52:45.788631+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:47.995195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.787222+0000","last_change":"2024-04-17T08:52:45.781912+0000","last_active":"2024-04-17T08:52:45.787222+0000","last_peered":"2024-04-17T08:52:45.787222+0000","last_clean":"2024-04-17T08:52:45.787222+0000","last_became_active":"2024-04-17T08:52:45.781586+0000","last_became_peered":"2024-04-17T08:52:45.781586+0000","last_unstale":"2024-04-17T08:52:45.787222+0000","last_undegraded":"2024-04-17T08:52:45.787222+0000","last_fullsized":"2024-04-17T08:52:45.787222+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:07.643506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813961+0000","last_change":"2024-04-17T08:52:45.802842+0000","last_active":"2024-04-17T08:52:45.813961+0000","last_peered":"2024-04-17T08:52:45.813961+0000","last_clean":"2024-04-17T08:52:45.813961+0000","last_became_active":"2024-04-17T08:52:45.802529+0000","last_became_peered":"2024-04-17T08:52:45.802529+0000","last_unstale":"2024-04-17T08:52:45.813961+0000","last_undegraded":"2024-04-17T08:52:45.813961+0000","last_fullsized":"2024-04-17T08:52:45.813961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:03.656548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809302+0000","last_change":"2024-04-17T08:52:45.806197+0000","last_active":"2024-04-17T08:52:45.809302+0000","last_peered":"2024-04-17T08:52:45.809302+0000","last_clean":"2024-04-17T08:52:45.809302+0000","last_became_active":"2024-04-17T08:52:45.805572+0000","last_became_peered":"2024-04-17T08:52:45.805572+0000","last_unstale":"2024-04-17T08:52:45.809302+0000","last_undegraded":"2024-04-17T08:52:45.809302+0000","last_fullsized":"2024-04-17T08:52:45.809302+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:45.967149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813767+0000","last_change":"2024-04-17T08:52:45.812640+0000","last_active":"2024-04-17T08:52:45.813767+0000","last_peered":"2024-04-17T08:52:45.813767+0000","last_clean":"2024-04-17T08:52:45.813767+0000","last_became_active":"2024-04-17T08:52:45.812350+0000","last_became_peered":"2024-04-17T08:52:45.812350+0000","last_unstale":"2024-04-17T08:52:45.813767+0000","last_undegraded":"2024-04-17T08:52:45.813767+0000","last_fullsized":"2024-04-17T08:52:45.813767+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:46.173369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.802420+0000","last_change":"2024-04-17T08:52:45.801355+0000","last_active":"2024-04-17T08:52:45.802420+0000","last_peered":"2024-04-17T08:52:45.802420+0000","last_clean":"2024-04-17T08:52:45.802420+0000","last_became_active":"2024-04-17T08:52:45.798491+0000","last_became_peered":"2024-04-17T08:52:45.798491+0000","last_unstale":"2024-04-17T08:52:45.802420+0000","last_undegraded":"2024-04-17T08:52:45.802420+0000","last_fullsized":"2024-04-17T08:52:45.802420+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:24.947037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807327+0000","last_change":"2024-04-17T08:52:45.803226+0000","last_active":"2024-04-17T08:52:45.807327+0000","last_peered":"2024-04-17T08:52:45.807327+0000","last_clean":"2024-04-17T08:52:45.807327+0000","last_became_active":"2024-04-17T08:52:45.802932+0000","last_became_peered":"2024-04-17T08:52:45.802932+0000","last_unstale":"2024-04-17T08:52:45.807327+0000","last_undegraded":"2024-04-17T08:52:45.807327+0000","last_fullsized":"2024-04-17T08:52:45.807327+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:15.650336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813574+0000","last_change":"2024-04-17T08:52:45.811041+0000","last_active":"2024-04-17T08:52:45.813574+0000","last_peered":"2024-04-17T08:52:45.813574+0000","last_clean":"2024-04-17T08:52:45.813574+0000","last_became_active":"2024-04-17T08:52:45.810742+0000","last_became_peered":"2024-04-17T08:52:45.810742+0000","last_unstale":"2024-04-17T08:52:45.813574+0000","last_undegraded":"2024-04-17T08:52:45.813574+0000","last_fullsized":"2024-04-17T08:52:45.813574+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:38.526225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813732+0000","last_change":"2024-04-17T08:52:45.811410+0000","last_active":"2024-04-17T08:52:45.813732+0000","last_peered":"2024-04-17T08:52:45.813732+0000","last_clean":"2024-04-17T08:52:45.813732+0000","last_became_active":"2024-04-17T08:52:45.810039+0000","last_became_peered":"2024-04-17T08:52:45.810039+0000","last_unstale":"2024-04-17T08:52:45.813732+0000","last_undegraded":"2024-04-17T08:52:45.813732+0000","last_fullsized":"2024-04-17T08:52:45.813732+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:08.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.788636+0000","last_change":"2024-04-17T08:52:45.785070+0000","last_active":"2024-04-17T08:52:45.788636+0000","last_peered":"2024-04-17T08:52:45.788636+0000","last_clean":"2024-04-17T08:52:45.788636+0000","last_became_active":"2024-04-17T08:52:45.784740+0000","last_became_peered":"2024-04-17T08:52:45.784740+0000","last_unstale":"2024-04-17T08:52:45.788636+0000","last_undegraded":"2024-04-17T08:52:45.788636+0000","last_fullsized":"2024-04-17T08:52:45.788636+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:20.400133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805901+0000","last_change":"2024-04-17T08:52:45.804234+0000","last_active":"2024-04-17T08:52:45.805901+0000","last_peered":"2024-04-17T08:52:45.805901+0000","last_clean":"2024-04-17T08:52:45.805901+0000","last_became_active":"2024-04-17T08:52:45.803866+0000","last_became_peered":"2024-04-17T08:52:45.803866+0000","last_unstale":"2024-04-17T08:52:45.805901+0000","last_undegraded":"2024-04-17T08:52:45.805901+0000","last_fullsized":"2024-04-17T08:52:45.805901+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:24:27.259709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809520+0000","last_change":"2024-04-17T08:52:45.808357+0000","last_active":"2024-04-17T08:52:45.809520+0000","last_peered":"2024-04-17T08:52:45.809520+0000","last_clean":"2024-04-17T08:52:45.809520+0000","last_became_active":"2024-04-17T08:52:45.807482+0000","last_became_peered":"2024-04-17T08:52:45.807482+0000","last_unstale":"2024-04-17T08:52:45.809520+0000","last_undegraded":"2024-04-17T08:52:45.809520+0000","last_fullsized":"2024-04-17T08:52:45.809520+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:48.581776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805545+0000","last_change":"2024-04-17T08:52:45.803241+0000","last_active":"2024-04-17T08:52:45.805545+0000","last_peered":"2024-04-17T08:52:45.805545+0000","last_clean":"2024-04-17T08:52:45.805545+0000","last_became_active":"2024-04-17T08:52:45.799072+0000","last_became_peered":"2024-04-17T08:52:45.799072+0000","last_unstale":"2024-04-17T08:52:45.805545+0000","last_undegraded":"2024-04-17T08:52:45.805545+0000","last_fullsized":"2024-04-17T08:52:45.805545+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:43.022668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.802920+0000","last_change":"2024-04-17T08:52:45.801863+0000","last_active":"2024-04-17T08:52:45.802920+0000","last_peered":"2024-04-17T08:52:45.802920+0000","last_clean":"2024-04-17T08:52:45.802920+0000","last_became_active":"2024-04-17T08:52:45.801346+0000","last_became_peered":"2024-04-17T08:52:45.801346+0000","last_unstale":"2024-04-17T08:52:45.802920+0000","last_undegraded":"2024-04-17T08:52:45.802920+0000","last_fullsized":"2024-04-17T08:52:45.802920+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:27.951601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.790122+0000","last_change":"2024-04-17T08:52:45.787077+0000","last_active":"2024-04-17T08:52:45.790122+0000","last_peered":"2024-04-17T08:52:45.790122+0000","last_clean":"2024-04-17T08:52:45.790122+0000","last_became_active":"2024-04-17T08:52:45.785130+0000","last_became_peered":"2024-04-17T08:52:45.785130+0000","last_unstale":"2024-04-17T08:52:45.790122+0000","last_undegraded":"2024-04-17T08:52:45.790122+0000","last_fullsized":"2024-04-17T08:52:45.790122+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:14:19.453418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814369+0000","last_change":"2024-04-17T08:52:45.811133+0000","last_active":"2024-04-17T08:52:45.814369+0000","last_peered":"2024-04-17T08:52:45.814369+0000","last_clean":"2024-04-17T08:52:45.814369+0000","last_became_active":"2024-04-17T08:52:45.807009+0000","last_became_peered":"2024-04-17T08:52:45.807009+0000","last_unstale":"2024-04-17T08:52:45.814369+0000","last_undegraded":"2024-04-17T08:52:45.814369+0000","last_fullsized":"2024-04-17T08:52:45.814369+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:55.444975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795814+0000","last_change":"2024-04-17T08:52:45.790478+0000","last_active":"2024-04-17T08:52:45.795814+0000","last_peered":"2024-04-17T08:52:45.795814+0000","last_clean":"2024-04-17T08:52:45.795814+0000","last_became_active":"2024-04-17T08:52:45.789914+0000","last_became_peered":"2024-04-17T08:52:45.789914+0000","last_unstale":"2024-04-17T08:52:45.795814+0000","last_undegraded":"2024-04-17T08:52:45.795814+0000","last_fullsized":"2024-04-17T08:52:45.795814+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:52:45.967149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812688+0000","last_change":"2024-04-17T08:52:45.805647+0000","last_active":"2024-04-17T08:52:45.812688+0000","last_peered":"2024-04-17T08:52:45.812688+0000","last_clean":"2024-04-17T08:52:45.812688+0000","last_became_active":"2024-04-17T08:52:45.805335+0000","last_became_peered":"2024-04-17T08:52:45.805335+0000","last_unstale":"2024-04-17T08:52:45.812688+0000","last_undegraded":"2024-04-17T08:52:45.812688+0000","last_fullsized":"2024-04-17T08:52:45.812688+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:33.108709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809267+0000","last_change":"2024-04-17T08:52:45.803839+0000","last_active":"2024-04-17T08:52:45.809267+0000","last_peered":"2024-04-17T08:52:45.809267+0000","last_clean":"2024-04-17T08:52:45.809267+0000","last_became_active":"2024-04-17T08:52:45.803535+0000","last_became_peered":"2024-04-17T08:52:45.803535+0000","last_unstale":"2024-04-17T08:52:45.809267+0000","last_undegraded":"2024-04-17T08:52:45.809267+0000","last_fullsized":"2024-04-17T08:52:45.809267+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:56.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.800175+0000","last_change":"2024-04-17T08:52:45.798116+0000","last_active":"2024-04-17T08:52:45.800175+0000","last_peered":"2024-04-17T08:52:45.800175+0000","last_clean":"2024-04-17T08:52:45.800175+0000","last_became_active":"2024-04-17T08:52:45.797818+0000","last_became_peered":"2024-04-17T08:52:45.797818+0000","last_unstale":"2024-04-17T08:52:45.800175+0000","last_undegraded":"2024-04-17T08:52:45.800175+0000","last_fullsized":"2024-04-17T08:52:45.800175+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:22.726576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807183+0000","last_change":"2024-04-17T08:52:45.796201+0000","last_active":"2024-04-17T08:52:45.807183+0000","last_peered":"2024-04-17T08:52:45.807183+0000","last_clean":"2024-04-17T08:52:45.807183+0000","last_became_active":"2024-04-17T08:52:45.794965+0000","last_became_peered":"2024-04-17T08:52:45.794965+0000","last_unstale":"2024-04-17T08:52:45.807183+0000","last_undegraded":"2024-04-17T08:52:45.807183+0000","last_fullsized":"2024-04-17T08:52:45.807183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:25.873820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"271'4","reported_seq":16,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.810089+0000","last_change":"2024-04-17T08:52:44.786017+0000","last_active":"2024-04-17T08:52:45.810089+0000","last_peered":"2024-04-17T08:52:45.810089+0000","last_clean":"2024-04-17T08:52:45.810089+0000","last_became_active":"2024-04-17T08:52:44.785722+0000","last_became_peered":"2024-04-17T08:52:44.785722+0000","last_unstale":"2024-04-17T08:52:45.810089+0000","last_undegraded":"2024-04-17T08:52:45.810089+0000","last_fullsized":"2024-04-17T08:52:45.810089+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:33.812816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814599+0000","last_change":"2024-04-17T08:52:45.813087+0000","last_active":"2024-04-17T08:52:45.814599+0000","last_peered":"2024-04-17T08:52:45.814599+0000","last_clean":"2024-04-17T08:52:45.814599+0000","last_became_active":"2024-04-17T08:52:45.810176+0000","last_became_peered":"2024-04-17T08:52:45.810176+0000","last_unstale":"2024-04-17T08:52:45.814599+0000","last_undegraded":"2024-04-17T08:52:45.814599+0000","last_fullsized":"2024-04-17T08:52:45.814599+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:10.130950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806893+0000","last_change":"2024-04-17T08:52:44.778826+0000","last_active":"2024-04-17T08:52:45.806893+0000","last_peered":"2024-04-17T08:52:45.806893+0000","last_clean":"2024-04-17T08:52:45.806893+0000","last_became_active":"2024-04-17T08:52:44.778501+0000","last_became_peered":"2024-04-17T08:52:44.778501+0000","last_unstale":"2024-04-17T08:52:45.806893+0000","last_undegraded":"2024-04-17T08:52:45.806893+0000","last_fullsized":"2024-04-17T08:52:45.806893+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:14:26.167346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806509+0000","last_change":"2024-04-17T08:52:45.801588+0000","last_active":"2024-04-17T08:52:45.806509+0000","last_peered":"2024-04-17T08:52:45.806509+0000","last_clean":"2024-04-17T08:52:45.806509+0000","last_became_active":"2024-04-17T08:52:45.799430+0000","last_became_peered":"2024-04-17T08:52:45.799430+0000","last_unstale":"2024-04-17T08:52:45.806509+0000","last_undegraded":"2024-04-17T08:52:45.806509+0000","last_fullsized":"2024-04-17T08:52:45.806509+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:22.430148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.806020+0000","last_change":"2024-04-17T08:52:44.787868+0000","last_active":"2024-04-17T08:52:45.806020+0000","last_peered":"2024-04-17T08:52:45.806020+0000","last_clean":"2024-04-17T08:52:45.806020+0000","last_became_active":"2024-04-17T08:52:44.786958+0000","last_became_peered":"2024-04-17T08:52:44.786958+0000","last_unstale":"2024-04-17T08:52:45.806020+0000","last_undegraded":"2024-04-17T08:52:45.806020+0000","last_fullsized":"2024-04-17T08:52:45.806020+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:57:51.368864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804411+0000","last_change":"2024-04-17T08:52:45.799638+0000","last_active":"2024-04-17T08:52:45.804411+0000","last_peered":"2024-04-17T08:52:45.804411+0000","last_clean":"2024-04-17T08:52:45.804411+0000","last_became_active":"2024-04-17T08:52:45.799330+0000","last_became_peered":"2024-04-17T08:52:45.799330+0000","last_unstale":"2024-04-17T08:52:45.804411+0000","last_undegraded":"2024-04-17T08:52:45.804411+0000","last_fullsized":"2024-04-17T08:52:45.804411+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:34.157322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814703+0000","last_change":"2024-04-17T08:52:44.794136+0000","last_active":"2024-04-17T08:52:45.814703+0000","last_peered":"2024-04-17T08:52:45.814703+0000","last_clean":"2024-04-17T08:52:45.814703+0000","last_became_active":"2024-04-17T08:52:44.793742+0000","last_became_peered":"2024-04-17T08:52:44.793742+0000","last_unstale":"2024-04-17T08:52:45.814703+0000","last_undegraded":"2024-04-17T08:52:45.814703+0000","last_fullsized":"2024-04-17T08:52:45.814703+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:48.350187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.800987+0000","last_change":"2024-04-17T08:52:45.798881+0000","last_active":"2024-04-17T08:52:45.800987+0000","last_peered":"2024-04-17T08:52:45.800987+0000","last_clean":"2024-04-17T08:52:45.800987+0000","last_became_active":"2024-04-17T08:52:45.797000+0000","last_became_peered":"2024-04-17T08:52:45.797000+0000","last_unstale":"2024-04-17T08:52:45.800987+0000","last_undegraded":"2024-04-17T08:52:45.800987+0000","last_fullsized":"2024-04-17T08:52:45.800987+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:40:38.486256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813114+0000","last_change":"2024-04-17T08:52:44.794571+0000","last_active":"2024-04-17T08:52:45.813114+0000","last_peered":"2024-04-17T08:52:45.813114+0000","last_clean":"2024-04-17T08:52:45.813114+0000","last_became_active":"2024-04-17T08:52:44.794234+0000","last_became_peered":"2024-04-17T08:52:44.794234+0000","last_unstale":"2024-04-17T08:52:45.813114+0000","last_undegraded":"2024-04-17T08:52:45.813114+0000","last_fullsized":"2024-04-17T08:52:45.813114+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:22.360922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813852+0000","last_change":"2024-04-17T08:52:45.809998+0000","last_active":"2024-04-17T08:52:45.813852+0000","last_peered":"2024-04-17T08:52:45.813852+0000","last_clean":"2024-04-17T08:52:45.813852+0000","last_became_active":"2024-04-17T08:52:45.809720+0000","last_became_peered":"2024-04-17T08:52:45.809720+0000","last_unstale":"2024-04-17T08:52:45.813852+0000","last_undegraded":"2024-04-17T08:52:45.813852+0000","last_fullsized":"2024-04-17T08:52:45.813852+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:09:03.656548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.811963+0000","last_change":"2024-04-17T08:52:44.783789+0000","last_active":"2024-04-17T08:52:45.811963+0000","last_peered":"2024-04-17T08:52:45.811963+0000","last_clean":"2024-04-17T08:52:45.811963+0000","last_became_active":"2024-04-17T08:52:44.783293+0000","last_became_peered":"2024-04-17T08:52:44.783293+0000","last_unstale":"2024-04-17T08:52:45.811963+0000","last_undegraded":"2024-04-17T08:52:45.811963+0000","last_fullsized":"2024-04-17T08:52:45.811963+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:25:51.365742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.801326+0000","last_change":"2024-04-17T08:52:45.799397+0000","last_active":"2024-04-17T08:52:45.801326+0000","last_peered":"2024-04-17T08:52:45.801326+0000","last_clean":"2024-04-17T08:52:45.801326+0000","last_became_active":"2024-04-17T08:52:45.798121+0000","last_became_peered":"2024-04-17T08:52:45.798121+0000","last_unstale":"2024-04-17T08:52:45.801326+0000","last_undegraded":"2024-04-17T08:52:45.801326+0000","last_fullsized":"2024-04-17T08:52:45.801326+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:16.158903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770556+0000","last_change":"2024-04-17T08:52:44.790792+0000","last_active":"2024-04-17T08:52:45.770556+0000","last_peered":"2024-04-17T08:52:45.770556+0000","last_clean":"2024-04-17T08:52:45.770556+0000","last_became_active":"2024-04-17T08:52:44.790116+0000","last_became_peered":"2024-04-17T08:52:44.790116+0000","last_unstale":"2024-04-17T08:52:45.770556+0000","last_undegraded":"2024-04-17T08:52:45.770556+0000","last_fullsized":"2024-04-17T08:52:45.770556+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:19.130136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791864+0000","last_change":"2024-04-17T08:52:45.783329+0000","last_active":"2024-04-17T08:52:45.791864+0000","last_peered":"2024-04-17T08:52:45.791864+0000","last_clean":"2024-04-17T08:52:45.791864+0000","last_became_active":"2024-04-17T08:52:45.782842+0000","last_became_peered":"2024-04-17T08:52:45.782842+0000","last_unstale":"2024-04-17T08:52:45.791864+0000","last_undegraded":"2024-04-17T08:52:45.791864+0000","last_fullsized":"2024-04-17T08:52:45.791864+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:31.491513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770524+0000","last_change":"2024-04-17T08:52:44.784249+0000","last_active":"2024-04-17T08:52:45.770524+0000","last_peered":"2024-04-17T08:52:45.770524+0000","last_clean":"2024-04-17T08:52:45.770524+0000","last_became_active":"2024-04-17T08:52:44.783749+0000","last_became_peered":"2024-04-17T08:52:44.783749+0000","last_unstale":"2024-04-17T08:52:45.770524+0000","last_undegraded":"2024-04-17T08:52:45.770524+0000","last_fullsized":"2024-04-17T08:52:45.770524+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:04.758592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.797512+0000","last_change":"2024-04-17T08:52:45.793087+0000","last_active":"2024-04-17T08:52:45.797512+0000","last_peered":"2024-04-17T08:52:45.797512+0000","last_clean":"2024-04-17T08:52:45.797512+0000","last_became_active":"2024-04-17T08:52:45.792836+0000","last_became_peered":"2024-04-17T08:52:45.792836+0000","last_unstale":"2024-04-17T08:52:45.797512+0000","last_undegraded":"2024-04-17T08:52:45.797512+0000","last_fullsized":"2024-04-17T08:52:45.797512+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:13.826400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.778980+0000","last_change":"2024-04-17T08:52:44.788711+0000","last_active":"2024-04-17T08:52:45.778980+0000","last_peered":"2024-04-17T08:52:45.778980+0000","last_clean":"2024-04-17T08:52:45.778980+0000","last_became_active":"2024-04-17T08:52:44.788367+0000","last_became_peered":"2024-04-17T08:52:44.788367+0000","last_unstale":"2024-04-17T08:52:45.778980+0000","last_undegraded":"2024-04-17T08:52:45.778980+0000","last_fullsized":"2024-04-17T08:52:45.778980+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:49:48.350187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815680+0000","last_change":"2024-04-17T08:52:45.807559+0000","last_active":"2024-04-17T08:52:45.815680+0000","last_peered":"2024-04-17T08:52:45.815680+0000","last_clean":"2024-04-17T08:52:45.815680+0000","last_became_active":"2024-04-17T08:52:45.802969+0000","last_became_peered":"2024-04-17T08:52:45.802969+0000","last_unstale":"2024-04-17T08:52:45.815680+0000","last_undegraded":"2024-04-17T08:52:45.815680+0000","last_fullsized":"2024-04-17T08:52:45.815680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:08.359579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.792407+0000","last_change":"2024-04-17T08:52:44.791840+0000","last_active":"2024-04-17T08:52:45.792407+0000","last_peered":"2024-04-17T08:52:45.792407+0000","last_clean":"2024-04-17T08:52:45.792407+0000","last_became_active":"2024-04-17T08:52:44.790900+0000","last_became_peered":"2024-04-17T08:52:44.790900+0000","last_unstale":"2024-04-17T08:52:45.792407+0000","last_undegraded":"2024-04-17T08:52:45.792407+0000","last_fullsized":"2024-04-17T08:52:45.792407+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:23:04.311098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813973+0000","last_change":"2024-04-17T08:52:45.810697+0000","last_active":"2024-04-17T08:52:45.813973+0000","last_peered":"2024-04-17T08:52:45.813973+0000","last_clean":"2024-04-17T08:52:45.813973+0000","last_became_active":"2024-04-17T08:52:45.809697+0000","last_became_peered":"2024-04-17T08:52:45.809697+0000","last_unstale":"2024-04-17T08:52:45.813973+0000","last_undegraded":"2024-04-17T08:52:45.813973+0000","last_fullsized":"2024-04-17T08:52:45.813973+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:58.663244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"271'3","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809389+0000","last_change":"2024-04-17T08:52:44.795395+0000","last_active":"2024-04-17T08:52:45.809389+0000","last_peered":"2024-04-17T08:52:45.809389+0000","last_clean":"2024-04-17T08:52:45.809389+0000","last_became_active":"2024-04-17T08:52:44.795081+0000","last_became_peered":"2024-04-17T08:52:44.795081+0000","last_unstale":"2024-04-17T08:52:45.809389+0000","last_undegraded":"2024-04-17T08:52:45.809389+0000","last_fullsized":"2024-04-17T08:52:45.809389+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:31:36.722960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795773+0000","last_change":"2024-04-17T08:52:45.789450+0000","last_active":"2024-04-17T08:52:45.795773+0000","last_peered":"2024-04-17T08:52:45.795773+0000","last_clean":"2024-04-17T08:52:45.795773+0000","last_became_active":"2024-04-17T08:52:45.788971+0000","last_became_peered":"2024-04-17T08:52:45.788971+0000","last_unstale":"2024-04-17T08:52:45.795773+0000","last_undegraded":"2024-04-17T08:52:45.795773+0000","last_fullsized":"2024-04-17T08:52:45.795773+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:41.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"271'6","reported_seq":18,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.817006+0000","last_change":"2024-04-17T08:52:44.792222+0000","last_active":"2024-04-17T08:52:45.817006+0000","last_peered":"2024-04-17T08:52:45.817006+0000","last_clean":"2024-04-17T08:52:45.817006+0000","last_became_active":"2024-04-17T08:52:44.792006+0000","last_became_peered":"2024-04-17T08:52:44.792006+0000","last_unstale":"2024-04-17T08:52:45.817006+0000","last_undegraded":"2024-04-17T08:52:45.817006+0000","last_fullsized":"2024-04-17T08:52:45.817006+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:05:36.142621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.813058+0000","last_change":"2024-04-17T08:52:45.809705+0000","last_active":"2024-04-17T08:52:45.813058+0000","last_peered":"2024-04-17T08:52:45.813058+0000","last_clean":"2024-04-17T08:52:45.813058+0000","last_became_active":"2024-04-17T08:52:45.809405+0000","last_became_peered":"2024-04-17T08:52:45.809405+0000","last_unstale":"2024-04-17T08:52:45.813058+0000","last_undegraded":"2024-04-17T08:52:45.813058+0000","last_fullsized":"2024-04-17T08:52:45.813058+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:41.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809094+0000","last_change":"2024-04-17T08:52:44.781025+0000","last_active":"2024-04-17T08:52:45.809094+0000","last_peered":"2024-04-17T08:52:45.809094+0000","last_clean":"2024-04-17T08:52:45.809094+0000","last_became_active":"2024-04-17T08:52:44.780345+0000","last_became_peered":"2024-04-17T08:52:44.780345+0000","last_unstale":"2024-04-17T08:52:45.809094+0000","last_undegraded":"2024-04-17T08:52:45.809094+0000","last_fullsized":"2024-04-17T08:52:45.809094+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:52.340674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798184+0000","last_change":"2024-04-17T08:52:45.792085+0000","last_active":"2024-04-17T08:52:45.798184+0000","last_peered":"2024-04-17T08:52:45.798184+0000","last_clean":"2024-04-17T08:52:45.798184+0000","last_became_active":"2024-04-17T08:52:45.791812+0000","last_became_peered":"2024-04-17T08:52:45.791812+0000","last_unstale":"2024-04-17T08:52:45.798184+0000","last_undegraded":"2024-04-17T08:52:45.798184+0000","last_fullsized":"2024-04-17T08:52:45.798184+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:48.509427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.782295+0000","last_change":"2024-04-17T08:52:44.789701+0000","last_active":"2024-04-17T08:52:45.782295+0000","last_peered":"2024-04-17T08:52:45.782295+0000","last_clean":"2024-04-17T08:52:45.782295+0000","last_became_active":"2024-04-17T08:52:44.789335+0000","last_became_peered":"2024-04-17T08:52:44.789335+0000","last_unstale":"2024-04-17T08:52:45.782295+0000","last_undegraded":"2024-04-17T08:52:45.782295+0000","last_fullsized":"2024-04-17T08:52:45.782295+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:03.310641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.803442+0000","last_change":"2024-04-17T08:52:45.802122+0000","last_active":"2024-04-17T08:52:45.803442+0000","last_peered":"2024-04-17T08:52:45.803442+0000","last_clean":"2024-04-17T08:52:45.803442+0000","last_became_active":"2024-04-17T08:52:45.801833+0000","last_became_peered":"2024-04-17T08:52:45.801833+0000","last_unstale":"2024-04-17T08:52:45.803442+0000","last_undegraded":"2024-04-17T08:52:45.803442+0000","last_fullsized":"2024-04-17T08:52:45.803442+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:10:23.451300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.775983+0000","last_change":"2024-04-17T08:52:44.788118+0000","last_active":"2024-04-17T08:52:45.775983+0000","last_peered":"2024-04-17T08:52:45.775983+0000","last_clean":"2024-04-17T08:52:45.775983+0000","last_became_active":"2024-04-17T08:52:44.787120+0000","last_became_peered":"2024-04-17T08:52:44.787120+0000","last_unstale":"2024-04-17T08:52:45.775983+0000","last_undegraded":"2024-04-17T08:52:45.775983+0000","last_fullsized":"2024-04-17T08:52:45.775983+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:17.372666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.809284+0000","last_change":"2024-04-17T08:52:45.806000+0000","last_active":"2024-04-17T08:52:45.809284+0000","last_peered":"2024-04-17T08:52:45.809284+0000","last_clean":"2024-04-17T08:52:45.809284+0000","last_became_active":"2024-04-17T08:52:45.805651+0000","last_became_peered":"2024-04-17T08:52:45.805651+0000","last_unstale":"2024-04-17T08:52:45.809284+0000","last_undegraded":"2024-04-17T08:52:45.809284+0000","last_fullsized":"2024-04-17T08:52:45.809284+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:12.457858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"271'8","reported_seq":20,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.812526+0000","last_change":"2024-04-17T08:52:44.796437+0000","last_active":"2024-04-17T08:52:45.812526+0000","last_peered":"2024-04-17T08:52:45.812526+0000","last_clean":"2024-04-17T08:52:45.812526+0000","last_became_active":"2024-04-17T08:52:44.796075+0000","last_became_peered":"2024-04-17T08:52:44.796075+0000","last_unstale":"2024-04-17T08:52:45.812526+0000","last_undegraded":"2024-04-17T08:52:45.812526+0000","last_fullsized":"2024-04-17T08:52:45.812526+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:26.759899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.791147+0000","last_change":"2024-04-17T08:52:45.788018+0000","last_active":"2024-04-17T08:52:45.791147+0000","last_peered":"2024-04-17T08:52:45.791147+0000","last_clean":"2024-04-17T08:52:45.791147+0000","last_became_active":"2024-04-17T08:52:45.787761+0000","last_became_peered":"2024-04-17T08:52:45.787761+0000","last_unstale":"2024-04-17T08:52:45.791147+0000","last_undegraded":"2024-04-17T08:52:45.791147+0000","last_fullsized":"2024-04-17T08:52:45.791147+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:47.148301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814151+0000","last_change":"2024-04-17T08:52:44.788336+0000","last_active":"2024-04-17T08:52:45.814151+0000","last_peered":"2024-04-17T08:52:45.814151+0000","last_clean":"2024-04-17T08:52:45.814151+0000","last_became_active":"2024-04-17T08:52:44.786747+0000","last_became_peered":"2024-04-17T08:52:44.786747+0000","last_unstale":"2024-04-17T08:52:45.814151+0000","last_undegraded":"2024-04-17T08:52:45.814151+0000","last_fullsized":"2024-04-17T08:52:45.814151+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T08:54:32.087413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804477+0000","last_change":"2024-04-17T08:52:45.800440+0000","last_active":"2024-04-17T08:52:45.804477+0000","last_peered":"2024-04-17T08:52:45.804477+0000","last_clean":"2024-04-17T08:52:45.804477+0000","last_became_active":"2024-04-17T08:52:45.800139+0000","last_became_peered":"2024-04-17T08:52:45.800139+0000","last_unstale":"2024-04-17T08:52:45.804477+0000","last_undegraded":"2024-04-17T08:52:45.804477+0000","last_fullsized":"2024-04-17T08:52:45.804477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:46:10.741528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815918+0000","last_change":"2024-04-17T08:52:44.791544+0000","last_active":"2024-04-17T08:52:45.815918+0000","last_peered":"2024-04-17T08:52:45.815918+0000","last_clean":"2024-04-17T08:52:45.815918+0000","last_became_active":"2024-04-17T08:52:44.791189+0000","last_became_peered":"2024-04-17T08:52:44.791189+0000","last_unstale":"2024-04-17T08:52:45.815918+0000","last_undegraded":"2024-04-17T08:52:45.815918+0000","last_fullsized":"2024-04-17T08:52:45.815918+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:07.782565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.799738+0000","last_change":"2024-04-17T08:52:45.792797+0000","last_active":"2024-04-17T08:52:45.799738+0000","last_peered":"2024-04-17T08:52:45.799738+0000","last_clean":"2024-04-17T08:52:45.799738+0000","last_became_active":"2024-04-17T08:52:45.792492+0000","last_became_peered":"2024-04-17T08:52:45.792492+0000","last_unstale":"2024-04-17T08:52:45.799738+0000","last_undegraded":"2024-04-17T08:52:45.799738+0000","last_fullsized":"2024-04-17T08:52:45.799738+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:20:05.687948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"271'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815698+0000","last_change":"2024-04-17T08:52:44.793351+0000","last_active":"2024-04-17T08:52:45.815698+0000","last_peered":"2024-04-17T08:52:45.815698+0000","last_clean":"2024-04-17T08:52:45.815698+0000","last_became_active":"2024-04-17T08:52:44.791062+0000","last_became_peered":"2024-04-17T08:52:44.791062+0000","last_unstale":"2024-04-17T08:52:45.815698+0000","last_undegraded":"2024-04-17T08:52:45.815698+0000","last_fullsized":"2024-04-17T08:52:45.815698+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:55.182502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.805454+0000","last_change":"2024-04-17T08:52:45.803230+0000","last_active":"2024-04-17T08:52:45.805454+0000","last_peered":"2024-04-17T08:52:45.805454+0000","last_clean":"2024-04-17T08:52:45.805454+0000","last_became_active":"2024-04-17T08:52:45.802914+0000","last_became_peered":"2024-04-17T08:52:45.802914+0000","last_unstale":"2024-04-17T08:52:45.805454+0000","last_undegraded":"2024-04-17T08:52:45.805454+0000","last_fullsized":"2024-04-17T08:52:45.805454+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:39:24.733697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.776793+0000","last_change":"2024-04-17T08:52:44.796757+0000","last_active":"2024-04-17T08:52:45.776793+0000","last_peered":"2024-04-17T08:52:45.776793+0000","last_clean":"2024-04-17T08:52:45.776793+0000","last_became_active":"2024-04-17T08:52:44.796461+0000","last_became_peered":"2024-04-17T08:52:44.796461+0000","last_unstale":"2024-04-17T08:52:45.776793+0000","last_undegraded":"2024-04-17T08:52:45.776793+0000","last_fullsized":"2024-04-17T08:52:45.776793+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:55.905650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793297+0000","last_change":"2024-04-17T08:52:45.792588+0000","last_active":"2024-04-17T08:52:45.793297+0000","last_peered":"2024-04-17T08:52:45.793297+0000","last_clean":"2024-04-17T08:52:45.793297+0000","last_became_active":"2024-04-17T08:52:45.792296+0000","last_became_peered":"2024-04-17T08:52:45.792296+0000","last_unstale":"2024-04-17T08:52:45.793297+0000","last_undegraded":"2024-04-17T08:52:45.793297+0000","last_fullsized":"2024-04-17T08:52:45.793297+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:22.701177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808439+0000","last_change":"2024-04-17T08:52:44.774370+0000","last_active":"2024-04-17T08:52:45.808439+0000","last_peered":"2024-04-17T08:52:45.808439+0000","last_clean":"2024-04-17T08:52:45.808439+0000","last_became_active":"2024-04-17T08:52:44.773885+0000","last_became_peered":"2024-04-17T08:52:44.773885+0000","last_unstale":"2024-04-17T08:52:45.808439+0000","last_undegraded":"2024-04-17T08:52:45.808439+0000","last_fullsized":"2024-04-17T08:52:45.808439+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:43:14.491703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798133+0000","last_change":"2024-04-17T08:52:45.787050+0000","last_active":"2024-04-17T08:52:45.798133+0000","last_peered":"2024-04-17T08:52:45.798133+0000","last_clean":"2024-04-17T08:52:45.798133+0000","last_became_active":"2024-04-17T08:52:45.786621+0000","last_became_peered":"2024-04-17T08:52:45.786621+0000","last_unstale":"2024-04-17T08:52:45.798133+0000","last_undegraded":"2024-04-17T08:52:45.798133+0000","last_fullsized":"2024-04-17T08:52:45.798133+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:40:21.323003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.779053+0000","last_change":"2024-04-17T08:52:44.792524+0000","last_active":"2024-04-17T08:52:45.779053+0000","last_peered":"2024-04-17T08:52:45.779053+0000","last_clean":"2024-04-17T08:52:45.779053+0000","last_became_active":"2024-04-17T08:52:44.792167+0000","last_became_peered":"2024-04-17T08:52:44.792167+0000","last_unstale":"2024-04-17T08:52:45.779053+0000","last_undegraded":"2024-04-17T08:52:45.779053+0000","last_fullsized":"2024-04-17T08:52:45.779053+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:05.888249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796591+0000","last_change":"2024-04-17T08:52:45.785382+0000","last_active":"2024-04-17T08:52:45.796591+0000","last_peered":"2024-04-17T08:52:45.796591+0000","last_clean":"2024-04-17T08:52:45.796591+0000","last_became_active":"2024-04-17T08:52:45.785026+0000","last_became_peered":"2024-04-17T08:52:45.785026+0000","last_unstale":"2024-04-17T08:52:45.796591+0000","last_undegraded":"2024-04-17T08:52:45.796591+0000","last_fullsized":"2024-04-17T08:52:45.796591+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:46.615853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.779524+0000","last_change":"2024-04-17T08:52:44.786171+0000","last_active":"2024-04-17T08:52:45.779524+0000","last_peered":"2024-04-17T08:52:45.779524+0000","last_clean":"2024-04-17T08:52:45.779524+0000","last_became_active":"2024-04-17T08:52:44.785710+0000","last_became_peered":"2024-04-17T08:52:44.785710+0000","last_unstale":"2024-04-17T08:52:45.779524+0000","last_undegraded":"2024-04-17T08:52:45.779524+0000","last_fullsized":"2024-04-17T08:52:45.779524+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:02:15.629216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.807288+0000","last_change":"2024-04-17T08:52:45.797482+0000","last_active":"2024-04-17T08:52:45.807288+0000","last_peered":"2024-04-17T08:52:45.807288+0000","last_clean":"2024-04-17T08:52:45.807288+0000","last_became_active":"2024-04-17T08:52:45.795959+0000","last_became_peered":"2024-04-17T08:52:45.795959+0000","last_unstale":"2024-04-17T08:52:45.807288+0000","last_undegraded":"2024-04-17T08:52:45.807288+0000","last_fullsized":"2024-04-17T08:52:45.807288+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:22:23.578941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770708+0000","last_change":"2024-04-17T08:52:44.783643+0000","last_active":"2024-04-17T08:52:45.770708+0000","last_peered":"2024-04-17T08:52:45.770708+0000","last_clean":"2024-04-17T08:52:45.770708+0000","last_became_active":"2024-04-17T08:52:44.783303+0000","last_became_peered":"2024-04-17T08:52:44.783303+0000","last_unstale":"2024-04-17T08:52:45.770708+0000","last_undegraded":"2024-04-17T08:52:45.770708+0000","last_fullsized":"2024-04-17T08:52:45.770708+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:13:48.383757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.815160+0000","last_change":"2024-04-17T08:52:45.814332+0000","last_active":"2024-04-17T08:52:45.815160+0000","last_peered":"2024-04-17T08:52:45.815160+0000","last_clean":"2024-04-17T08:52:45.815160+0000","last_became_active":"2024-04-17T08:52:45.812211+0000","last_became_peered":"2024-04-17T08:52:45.812211+0000","last_unstale":"2024-04-17T08:52:45.815160+0000","last_undegraded":"2024-04-17T08:52:45.815160+0000","last_fullsized":"2024-04-17T08:52:45.815160+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:56:22.756478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"271'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.816217+0000","last_change":"2024-04-17T08:52:44.799305+0000","last_active":"2024-04-17T08:52:45.816217+0000","last_peered":"2024-04-17T08:52:45.816217+0000","last_clean":"2024-04-17T08:52:45.816217+0000","last_became_active":"2024-04-17T08:52:44.798855+0000","last_became_peered":"2024-04-17T08:52:44.798855+0000","last_unstale":"2024-04-17T08:52:45.816217+0000","last_undegraded":"2024-04-17T08:52:45.816217+0000","last_fullsized":"2024-04-17T08:52:45.816217+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:27.955640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814785+0000","last_change":"2024-04-17T08:52:45.811183+0000","last_active":"2024-04-17T08:52:45.814785+0000","last_peered":"2024-04-17T08:52:45.814785+0000","last_clean":"2024-04-17T08:52:45.814785+0000","last_became_active":"2024-04-17T08:52:45.809747+0000","last_became_peered":"2024-04-17T08:52:45.809747+0000","last_unstale":"2024-04-17T08:52:45.814785+0000","last_undegraded":"2024-04-17T08:52:45.814785+0000","last_fullsized":"2024-04-17T08:52:45.814785+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:07.496274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.782644+0000","last_change":"2024-04-17T08:52:44.783014+0000","last_active":"2024-04-17T08:52:45.782644+0000","last_peered":"2024-04-17T08:52:45.782644+0000","last_clean":"2024-04-17T08:52:45.782644+0000","last_became_active":"2024-04-17T08:52:44.782651+0000","last_became_peered":"2024-04-17T08:52:44.782651+0000","last_unstale":"2024-04-17T08:52:45.782644+0000","last_undegraded":"2024-04-17T08:52:45.782644+0000","last_fullsized":"2024-04-17T08:52:45.782644+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:29:45.681524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814842+0000","last_change":"2024-04-17T08:52:45.812637+0000","last_active":"2024-04-17T08:52:45.814842+0000","last_peered":"2024-04-17T08:52:45.814842+0000","last_clean":"2024-04-17T08:52:45.814842+0000","last_became_active":"2024-04-17T08:52:45.810031+0000","last_became_peered":"2024-04-17T08:52:45.810031+0000","last_unstale":"2024-04-17T08:52:45.814842+0000","last_undegraded":"2024-04-17T08:52:45.814842+0000","last_fullsized":"2024-04-17T08:52:45.814842+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:57.407900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798994+0000","last_change":"2024-04-17T08:52:44.792514+0000","last_active":"2024-04-17T08:52:45.798994+0000","last_peered":"2024-04-17T08:52:45.798994+0000","last_clean":"2024-04-17T08:52:45.798994+0000","last_became_active":"2024-04-17T08:52:44.791211+0000","last_became_peered":"2024-04-17T08:52:44.791211+0000","last_unstale":"2024-04-17T08:52:45.798994+0000","last_undegraded":"2024-04-17T08:52:45.798994+0000","last_fullsized":"2024-04-17T08:52:45.798994+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:18:22.622174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.804388+0000","last_change":"2024-04-17T08:52:45.793297+0000","last_active":"2024-04-17T08:52:45.804388+0000","last_peered":"2024-04-17T08:52:45.804388+0000","last_clean":"2024-04-17T08:52:45.804388+0000","last_became_active":"2024-04-17T08:52:45.793003+0000","last_became_peered":"2024-04-17T08:52:45.793003+0000","last_unstale":"2024-04-17T08:52:45.804388+0000","last_undegraded":"2024-04-17T08:52:45.804388+0000","last_fullsized":"2024-04-17T08:52:45.804388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:30:35.618353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.795081+0000","last_change":"2024-04-17T08:52:44.795429+0000","last_active":"2024-04-17T08:52:45.795081+0000","last_peered":"2024-04-17T08:52:45.795081+0000","last_clean":"2024-04-17T08:52:45.795081+0000","last_became_active":"2024-04-17T08:52:44.795050+0000","last_became_peered":"2024-04-17T08:52:44.795050+0000","last_unstale":"2024-04-17T08:52:45.795081+0000","last_undegraded":"2024-04-17T08:52:45.795081+0000","last_fullsized":"2024-04-17T08:52:45.795081+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:56:39.800086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793257+0000","last_change":"2024-04-17T08:52:45.792489+0000","last_active":"2024-04-17T08:52:45.793257+0000","last_peered":"2024-04-17T08:52:45.793257+0000","last_clean":"2024-04-17T08:52:45.793257+0000","last_became_active":"2024-04-17T08:52:45.792119+0000","last_became_peered":"2024-04-17T08:52:45.792119+0000","last_unstale":"2024-04-17T08:52:45.793257+0000","last_undegraded":"2024-04-17T08:52:45.793257+0000","last_fullsized":"2024-04-17T08:52:45.793257+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:10:07.556062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796137+0000","last_change":"2024-04-17T08:52:44.795387+0000","last_active":"2024-04-17T08:52:45.796137+0000","last_peered":"2024-04-17T08:52:45.796137+0000","last_clean":"2024-04-17T08:52:45.796137+0000","last_became_active":"2024-04-17T08:52:44.793794+0000","last_became_peered":"2024-04-17T08:52:44.793794+0000","last_unstale":"2024-04-17T08:52:45.796137+0000","last_undegraded":"2024-04-17T08:52:45.796137+0000","last_fullsized":"2024-04-17T08:52:45.796137+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:55.905650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.798204+0000","last_change":"2024-04-17T08:52:45.792651+0000","last_active":"2024-04-17T08:52:45.798204+0000","last_peered":"2024-04-17T08:52:45.798204+0000","last_clean":"2024-04-17T08:52:45.798204+0000","last_became_active":"2024-04-17T08:52:45.790732+0000","last_became_peered":"2024-04-17T08:52:45.790732+0000","last_unstale":"2024-04-17T08:52:45.798204+0000","last_undegraded":"2024-04-17T08:52:45.798204+0000","last_fullsized":"2024-04-17T08:52:45.798204+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:43:51.867535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770470+0000","last_change":"2024-04-17T08:52:44.779751+0000","last_active":"2024-04-17T08:52:45.770470+0000","last_peered":"2024-04-17T08:52:45.770470+0000","last_clean":"2024-04-17T08:52:45.770470+0000","last_became_active":"2024-04-17T08:52:44.777362+0000","last_became_peered":"2024-04-17T08:52:44.777362+0000","last_unstale":"2024-04-17T08:52:45.770470+0000","last_undegraded":"2024-04-17T08:52:45.770470+0000","last_fullsized":"2024-04-17T08:52:45.770470+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:14:29.894624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.796615+0000","last_change":"2024-04-17T08:52:45.789551+0000","last_active":"2024-04-17T08:52:45.796615+0000","last_peered":"2024-04-17T08:52:45.796615+0000","last_clean":"2024-04-17T08:52:45.796615+0000","last_became_active":"2024-04-17T08:52:45.787902+0000","last_became_peered":"2024-04-17T08:52:45.787902+0000","last_unstale":"2024-04-17T08:52:45.796615+0000","last_undegraded":"2024-04-17T08:52:45.796615+0000","last_fullsized":"2024-04-17T08:52:45.796615+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:22:29.112824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.770738+0000","last_change":"2024-04-17T08:52:44.777566+0000","last_active":"2024-04-17T08:52:45.770738+0000","last_peered":"2024-04-17T08:52:45.770738+0000","last_clean":"2024-04-17T08:52:45.770738+0000","last_became_active":"2024-04-17T08:52:44.776658+0000","last_became_peered":"2024-04-17T08:52:44.776658+0000","last_unstale":"2024-04-17T08:52:45.770738+0000","last_undegraded":"2024-04-17T08:52:45.770738+0000","last_fullsized":"2024-04-17T08:52:45.770738+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:18.960800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.814782+0000","last_change":"2024-04-17T08:52:45.803953+0000","last_active":"2024-04-17T08:52:45.814782+0000","last_peered":"2024-04-17T08:52:45.814782+0000","last_clean":"2024-04-17T08:52:45.814782+0000","last_became_active":"2024-04-17T08:52:45.803662+0000","last_became_peered":"2024-04-17T08:52:45.803662+0000","last_unstale":"2024-04-17T08:52:45.814782+0000","last_undegraded":"2024-04-17T08:52:45.814782+0000","last_fullsized":"2024-04-17T08:52:45.814782+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:35:38.526225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.775762+0000","last_change":"2024-04-17T08:52:44.780482+0000","last_active":"2024-04-17T08:52:45.775762+0000","last_peered":"2024-04-17T08:52:45.775762+0000","last_clean":"2024-04-17T08:52:45.775762+0000","last_became_active":"2024-04-17T08:52:44.779902+0000","last_became_peered":"2024-04-17T08:52:44.779902+0000","last_unstale":"2024-04-17T08:52:45.775762+0000","last_undegraded":"2024-04-17T08:52:45.775762+0000","last_fullsized":"2024-04-17T08:52:45.775762+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:43.718598+0000","last_clean_scrub_stamp":"2024-04-17T08:52:43.718598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:15:30.925823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":9,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.808896+0000","last_change":"2024-04-17T08:52:45.807050+0000","last_active":"2024-04-17T08:52:45.808896+0000","last_peered":"2024-04-17T08:52:45.808896+0000","last_clean":"2024-04-17T08:52:45.808896+0000","last_became_active":"2024-04-17T08:52:45.806768+0000","last_became_peered":"2024-04-17T08:52:45.806768+0000","last_unstale":"2024-04-17T08:52:45.808896+0000","last_undegraded":"2024-04-17T08:52:45.808896+0000","last_fullsized":"2024-04-17T08:52:45.808896+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:52:44.739894+0000","last_clean_scrub_stamp":"2024-04-17T08:52:44.739894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:47:27.662918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"268'2749","reported_seq":4037,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-04-17T08:52:45.793765+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:52:45.793765+0000","last_peered":"2024-04-17T08:52:45.793765+0000","last_clean":"2024-04-17T08:52:45.793765+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:52:45.793765+0000","last_undegraded":"2024-04-17T08:52:45.793765+0000","last_fullsized":"2024-04-17T08:52:45.793765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2505,"num_read_kb":5246,"num_write":649,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2749,"ondisk_log_size":2749,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543318,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48976,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322864,"statfs":{"total":96636764160,"available":96586612736,"internally_reserved":0,"allocated":3997696,"data_stored":2268507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543318,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60896,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310944,"statfs":{"total":96636764160,"available":96574406656,"internally_reserved":0,"allocated":3420160,"data_stored":1685075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543317,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48428,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323412,"statfs":{"total":96636764160,"available":96587173888,"internally_reserved":0,"allocated":3436544,"data_stored":1687249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543317,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66380,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305460,"statfs":{"total":96636764160,"available":96568791040,"internally_reserved":0,"allocated":3403776,"data_stored":1678139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52584,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319256,"statfs":{"total":96636764160,"available":96582918144,"internally_reserved":0,"allocated":3428352,"data_stored":1686597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543318,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52576,"kb_used_data":3340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3420160,"data_stored":1685075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543319,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48396,"kb_used_data":3324,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323444,"statfs":{"total":96636764160,"available":96587206656,"internally_reserved":0,"allocated":3403776,"data_stored":1678139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543317,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57296,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314544,"statfs":{"total":96636764160,"available":96578093056,"internally_reserved":0,"allocated":3997696,"data_stored":2268507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:52:50.856 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:52:50.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:52:51.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.133+0000 7f58ca000640 1 Processor -- start 2024-04-17T08:52:51.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.133+0000 7f58ca000640 1 -- start start 2024-04-17T08:52:51.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58ca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58c4072630 0x7f58c415cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58ca000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c40cda40 con 0x7f58c40cd060 2024-04-17T08:52:51.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58ca000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c40cdd20 con 0x7f58c4146150 2024-04-17T08:52:51.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58ca000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f58c4146530 con 0x7f58c4072630 2024-04-17T08:52:51.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58c3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58c4072630 0x7f58c415cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58c3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58c4072630 0x7f58c415cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55858/0 (socket says 172.21.15.67:55858) 2024-04-17T08:52:51.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.135+0000 7f58c3e00640 1 -- 172.21.15.67:0/635015282 learned_addr learned my addr 172.21.15.67:0/635015282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.136+0000 7f58c3e00640 1 -- 172.21.15.67:0/635015282 >> v1:172.21.15.187:6789/0 conn(0x7f58c4146150 legacy=0x7f58c4160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.136+0000 7f58c3e00640 1 -- 172.21.15.67:0/635015282 >> v1:172.21.15.67:6789/0 conn(0x7f58c40cd060 legacy=0x7f58c4071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.136+0000 7f58c3e00640 1 -- 172.21.15.67:0/635015282 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c4160500 con 0x7f58c4072630 2024-04-17T08:52:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.137+0000 7f58c3e00640 1 --2- 172.21.15.67:0/635015282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c4072630 0x7f58c415cd60 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009870 tx=0x7f58bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fa415237c7a105 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.137+0000 7f58c2a00640 1 -- 172.21.15.67:0/635015282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc034070 con 0x7f58c4072630 2024-04-17T08:52:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.138+0000 7f58c2a00640 1 -- 172.21.15.67:0/635015282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58bc030860 con 0x7f58c4072630 2024-04-17T08:52:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.138+0000 7f58c2a00640 1 -- 172.21.15.67:0/635015282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc030c30 con 0x7f58c4072630 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.138+0000 7f58ca000640 1 -- 172.21.15.67:0/635015282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c4072630 msgr2=0x7f58c415cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.138+0000 7f58ca000640 1 --2- 172.21.15.67:0/635015282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c4072630 0x7f58c415cd60 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009870 tx=0x7f58bc02b9a0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.138+0000 7f58ca000640 1 -- 172.21.15.67:0/635015282 shutdown_connections 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.139+0000 7f58ca000640 1 --2- 172.21.15.67:0/635015282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c4072630 0x7f58c415cd60 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.139+0000 7f58ca000640 1 -- 172.21.15.67:0/635015282 >> 172.21.15.67:0/635015282 conn(0x7f58c4067c30 msgr2=0x7f58c406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.139+0000 7f58ca000640 1 -- 172.21.15.67:0/635015282 shutdown_connections 2024-04-17T08:52:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.139+0000 7f58ca000640 1 -- 172.21.15.67:0/635015282 wait complete. 2024-04-17T08:52:51.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.141+0000 7f58ca000640 1 Processor -- start 2024-04-17T08:52:51.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.141+0000 7f58ca000640 1 -- start start 2024-04-17T08:52:51.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c4072630 0x7f58c41423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 0x7f58c41e8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58c406f490 con 0x7f58c40cd060 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f58c406f310 con 0x7f58c4072630 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.142+0000 7f58ca000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f58c40cd4d0 con 0x7f58c4146150 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55864/0 (socket says 172.21.15.67:55864) 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 -- 172.21.15.67:0/2144852052 learned_addr learned my addr 172.21.15.67:0/2144852052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58c3400640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c4072630 0x7f58c41423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58c3e00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 0x7f58c41e8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 -- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 msgr2=0x7f58c41e8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 0x7f58c41e8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 -- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c4072630 msgr2=0x7f58c41423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c4072630 0x7f58c41423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58baa00640 1 -- 172.21.15.67:0/2144852052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c41e6cc0 con 0x7f58c40cd060 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.143+0000 7f58c3e00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 0x7f58c41e8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58baa00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00c400 tx=0x7f58ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0052d0 con 0x7f58c40cd060 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58ac005490 con 0x7f58c40cd060 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac007260 con 0x7f58c40cd060 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c41e6fa0 con 0x7f58c40cd060 2024-04-17T08:52:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.144+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c41e7560 con 0x7f58c40cd060 2024-04-17T08:52:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.145+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c4068ca0 con 0x7f58c40cd060 2024-04-17T08:52:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.146+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f58ac007400 con 0x7f58c40cd060 2024-04-17T08:52:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.148+0000 7f58bbe00640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 0x7f58a006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.148+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f58ac08b540 con 0x7f58c40cd060 2024-04-17T08:52:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.149+0000 7f58c3400640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 0x7f58a006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.150+0000 7f58c3400640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 0x7f58a006eff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009f20 tx=0x7f58b40094d0 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.155+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f58ac0595f0 con 0x7f58c40cd060 2024-04-17T08:52:51.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.437+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f58c40731d0 con 0x7f58c40cd060 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.442+0000 7f58bbe00640 1 -- 172.21.15.67:0/2144852052 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) ==== 190+0+147 (secure 0 0 0) 0x7f58ac058ef0 con 0x7f58c40cd060 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:52:51.442 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:52:51.443 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 msgr2=0x7f58a006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 0x7f58a006eff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009f20 tx=0x7f58b40094d0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 msgr2=0x7f58c41e62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00c400 tx=0x7f58ac00c8c0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 shutdown_connections 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f58a006cb30 0x7f58a006eff0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f58c4146150 0x7f58c41e8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c40cd060 0x7f58c41e62e0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 --2- 172.21.15.67:0/2144852052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f58c4072630 0x7f58c41423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.445+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 >> 172.21.15.67:0/2144852052 conn(0x7f58c4067c30 msgr2=0x7f58c406e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.446+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 shutdown_connections 2024-04-17T08:52:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.446+0000 7f58ca000640 1 -- 172.21.15.67:0/2144852052 wait complete. 2024-04-17T08:52:51.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:52:51.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.694+0000 7f9a5ca00640 1 Processor -- start 2024-04-17T08:52:51.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.695+0000 7f9a5ca00640 1 -- start start 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a5ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a580cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a5ca00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9a58160280 0x7f9a5806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a5ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a580cd4a0 con 0x7f9a58141ed0 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a5ca00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9a5815ce30 con 0x7f9a58160280 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a5ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9a581607f0 con 0x7f9a5806b8c0 2024-04-17T08:52:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a57200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a58141ed0 0x7f9a5815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45428/0 (socket says 172.21.15.67:45428) 2024-04-17T08:52:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a57200640 1 -- 172.21.15.67:0/211141166 learned_addr learned my addr 172.21.15.67:0/211141166 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a56800640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a580cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a55e00640 1 --2- 172.21.15.67:0/211141166 >> v2:172.21.15.187:3300/0 conn(0x7f9a58160280 0x7f9a5806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.697+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594555363 0 0) 0x7f9a580cd4a0 con 0x7f9a58141ed0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a3c003520 con 0x7f9a58141ed0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3739404334 0 0) 0x7f9a3c003520 con 0x7f9a58141ed0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 msgr2=0x7f9a580cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a580cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 >> v2:172.21.15.187:3300/0 conn(0x7f9a58160280 msgr2=0x7f9a5806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 --2- 172.21.15.67:0/211141166 >> v2:172.21.15.187:3300/0 conn(0x7f9a58160280 0x7f9a5806bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a58160ad0 con 0x7f9a58141ed0 2024-04-17T08:52:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9a50001670 con 0x7f9a58141ed0 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 >> v1:172.21.15.67:6789/0 conn(0x7f9a58141ed0 legacy=0x7f9a5815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a3c003a20 0x7f9a3c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a3c004d40 0x7f9a3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5815ce30 con 0x7f9a3c004d40 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9a581607f0 con 0x7f9a3c003a20 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9a3c00b9c0 con 0x7f9a5806b8c0 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.698+0000 7f9a57200640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a3c004d40 0x7f9a3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a56800640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a3c003a20 0x7f9a3c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a55e00640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a57200640 1 -- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 msgr2=0x7f9a3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a57200640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a5806b8c0 0x7f9a3c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a57200640 1 -- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a3c003a20 msgr2=0x7f9a3c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a57200640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a3c003a20 0x7f9a3c003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.699+0000 7f9a57200640 1 -- 172.21.15.67:0/211141166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3c003520 con 0x7f9a3c004d40 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.700+0000 7f9a57200640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a3c004d40 0x7f9a3c0051c0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c210 tx=0x7f9a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab92e74744d2ecb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.700+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50007d50 con 0x7f9a3c004d40 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.700+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a50030070 con 0x7f9a3c004d40 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.700+0000 7f9a55400640 1 -- 172.21.15.67:0/211141166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5002cd80 con 0x7f9a3c004d40 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 -- 172.21.15.67:0/211141166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a3c004d40 msgr2=0x7f9a3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/211141166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a3c004d40 0x7f9a3c0051c0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c210 tx=0x7f9a3c005830 comp rx=0 tx=0).stop 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a57200640 1 -- 172.21.15.67:0/211141166 reap_dead start 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 -- 172.21.15.67:0/211141166 shutdown_connections 2024-04-17T08:52:51.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 -- 172.21.15.67:0/211141166 >> 172.21.15.67:0/211141166 conn(0x7f9a58067d10 msgr2=0x7f9a580b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 -- 172.21.15.67:0/211141166 shutdown_connections 2024-04-17T08:52:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.701+0000 7f9a5ca00640 1 -- 172.21.15.67:0/211141166 wait complete. 2024-04-17T08:52:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.703+0000 7f9a5ca00640 1 Processor -- start 2024-04-17T08:52:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.703+0000 7f9a5ca00640 1 -- start start 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a3c004d40 0x7f9a58144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 0x7f9a5806e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a58161090 con 0x7f9a5815b5f0 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5806c1e0 con 0x7f9a5806e3c0 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a5ca00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5813f9b0 con 0x7f9a3c004d40 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55886/0 (socket says 172.21.15.67:55886) 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.704+0000 7f9a55e00640 1 -- 172.21.15.67:0/2546036189 learned_addr learned my addr 172.21.15.67:0/2546036189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:51.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a56800640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a3c004d40 0x7f9a58144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a57200640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 0x7f9a5806e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 -- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a3c004d40 msgr2=0x7f9a58144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a3c004d40 0x7f9a58144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 -- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 msgr2=0x7f9a5806e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 0x7f9a5806e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 -- 172.21.15.67:0/2546036189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5806efd0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a57200640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 0x7f9a5806e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a55e00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f9a400108e0 tx=0x7f9a40010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4000f040 con 0x7f9a5815b5f0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4000eb60 con 0x7f9a5815b5f0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5806f2b0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.705+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4001c400 con 0x7f9a5815b5f0 2024-04-17T08:52:51.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.706+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5806f870 con 0x7f9a5815b5f0 2024-04-17T08:52:51.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.707+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5815bfb0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.708+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f9a4001c5a0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.709+0000 7f9a4ea00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 0x7f9a3406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:51.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.709+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f9a4008c7f0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.709+0000 7f9a56800640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 0x7f9a3406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.710+0000 7f9a56800640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 0x7f9a3406eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a5815f290 tx=0x7f9a4800b040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:51.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.713+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9a40018030 con 0x7f9a5815b5f0 2024-04-17T08:52:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.961+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a58068a10 con 0x7f9a5815b5f0 2024-04-17T08:52:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.963+0000 7f9a4ea00640 1 -- 172.21.15.67:0/2546036189 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7f9a4005a8a0 con 0x7f9a5815b5f0 2024-04-17T08:52:51.962 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:52:51.962 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:52:51.963 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.965+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 msgr2=0x7f9a3406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.965+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 0x7f9a3406eff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a5815f290 tx=0x7f9a4800b040 comp rx=0 tx=0).stop 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 msgr2=0x7f9a5815ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f9a400108e0 tx=0x7f9a40010da0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 shutdown_connections 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9a3406cb30 0x7f9a3406eff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a5806e3c0 0x7f9a5806e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a5815b5f0 0x7f9a5815ba70 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 --2- 172.21.15.67:0/2546036189 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a3c004d40 0x7f9a58144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:51.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 >> 172.21.15.67:0/2546036189 conn(0x7f9a58067d10 msgr2=0x7f9a580b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:51.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 shutdown_connections 2024-04-17T08:52:51.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:51.966+0000 7f9a5ca00640 1 -- 172.21.15.67:0/2546036189 wait complete. 2024-04-17T08:52:52.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 143, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:52:45.263955+0000', 'modified': '2024-04-17T08:52:48.338438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12552}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12552': {'gid': 12552, 'name': 'a', 'rank': 0, 'incarnation': 139, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1995843637', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 1995843637}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1995843637}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12552, 'qdb_cluster': [12552]} 2024-04-17T08:52:52.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:52:52.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T08:52:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.621+0000 7fa44fe00640 1 Processor -- start 2024-04-17T08:52:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.622+0000 7fa44fe00640 1 -- start start 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.623+0000 7fa44fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa4500d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.623+0000 7fa44fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4500d3350 0x7fa4500b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.623+0000 7fa44fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4500d4140 con 0x7fa45015ee20 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.623+0000 7fa44fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa4500b83b0 con 0x7fa4500d3350 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.623+0000 7fa44fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500d3b80 con 0x7fa4500d1ab0 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4500d3350 0x7fa4500b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4500d3350 0x7fa4500b8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55898/0 (socket says 172.21.15.67:55898) 2024-04-17T08:52:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 -- 172.21.15.67:0/1628847758 learned_addr learned my addr 172.21.15.67:0/1628847758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa447e00640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa4500d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 -- 172.21.15.67:0/1628847758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 msgr2=0x7fa4500d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa4500d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 -- 172.21.15.67:0/1628847758 >> v1:172.21.15.187:6789/0 conn(0x7fa45015ee20 legacy=0x7fa4500d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa44f400640 1 -- 172.21.15.67:0/1628847758 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4500d3e60 con 0x7fa4500d3350 2024-04-17T08:52:52.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.624+0000 7fa447e00640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa4500d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.625+0000 7fa44f400640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa4500b8840 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fa448009e50 tx=0x7fa44802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8821886a82b9a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.626+0000 7fa44e000640 1 -- 172.21.15.67:0/1628847758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44802ca70 con 0x7fa4500d3350 2024-04-17T08:52:52.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.626+0000 7fa44e000640 1 -- 172.21.15.67:0/1628847758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44802cc10 con 0x7fa4500d3350 2024-04-17T08:52:52.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.626+0000 7fa44e000640 1 -- 172.21.15.67:0/1628847758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4480343f0 con 0x7fa4500d3350 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.626+0000 7fa44fe00640 1 -- 172.21.15.67:0/1628847758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 msgr2=0x7fa4500b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.626+0000 7fa44fe00640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa4500b8840 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fa448009e50 tx=0x7fa44802bed0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 -- 172.21.15.67:0/1628847758 shutdown_connections 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa4500b8840 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 --2- 172.21.15.67:0/1628847758 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa4500d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 -- 172.21.15.67:0/1628847758 >> 172.21.15.67:0/1628847758 conn(0x7fa45006d3c0 msgr2=0x7fa4500b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 -- 172.21.15.67:0/1628847758 shutdown_connections 2024-04-17T08:52:52.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.627+0000 7fa44fe00640 1 -- 172.21.15.67:0/1628847758 wait complete. 2024-04-17T08:52:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.629+0000 7fa44fe00640 1 Processor -- start 2024-04-17T08:52:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.629+0000 7fa44fe00640 1 -- start start 2024-04-17T08:52:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa450158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 0x7fa4500cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500d3800 con 0x7fa4500d3350 2024-04-17T08:52:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa45015f200 con 0x7fa45015ee20 2024-04-17T08:52:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500b8d80 con 0x7fa4500d1ab0 2024-04-17T08:52:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa447e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa447e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55908/0 (socket says 172.21.15.67:55908) 2024-04-17T08:52:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa447e00640 1 -- 172.21.15.67:0/932425137 learned_addr learned my addr 172.21.15.67:0/932425137 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.630+0000 7fa44f400640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 0x7fa4500cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa44ea00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa450158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 -- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 msgr2=0x7fa450158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa450158ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 -- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 msgr2=0x7fa4500cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 0x7fa4500cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 -- 172.21.15.67:0/932425137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4500c9de0 con 0x7fa4500d3350 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa44f400640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 0x7fa4500cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa447e00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7fa43800c7f0 tx=0x7fa43800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa446a00640 1 -- 172.21.15.67:0/932425137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438015070 con 0x7fa4500d3350 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa446a00640 1 -- 172.21.15.67:0/932425137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa438007a30 con 0x7fa4500d3350 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa446a00640 1 -- 172.21.15.67:0/932425137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438007e00 con 0x7fa4500d3350 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.631+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4500ca0c0 con 0x7fa4500d3350 2024-04-17T08:52:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.632+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4500ca960 con 0x7fa4500d3350 2024-04-17T08:52:52.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.634+0000 7fa446a00640 1 -- 172.21.15.67:0/932425137 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7fa438018070 con 0x7fa4500d3350 2024-04-17T08:52:52.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.635+0000 7fa446a00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 0x7fa42c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.636+0000 7fa446a00640 1 -- 172.21.15.67:0/932425137 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fa43808c190 con 0x7fa4500d3350 2024-04-17T08:52:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.636+0000 7fa44ea00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 0x7fa42c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.636+0000 7fa458626740 1 Processor -- start 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.637+0000 7fa458626740 1 -- start start 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.637+0000 7fa44ea00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 0x7fa42c06eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa440006fd0 tx=0x7fa440008040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a38d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 0x561b9a3b7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b9a2f3980 con 0x561b9a3d5620 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x561b9a29a220 con 0x561b9a387b40 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x561b9a2ef940 con 0x561b9a38b6a0 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55924/0 (socket says 172.21.15.67:55924) 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 -- 172.21.15.67:0/1593066789 learned_addr learned my addr 172.21.15.67:0/1593066789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa447e00640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a38d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44f400640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 0x561b9a3b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 -- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 msgr2=0x561b9a38d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a38d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 -- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 msgr2=0x561b9a3b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 0x561b9a3b7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.638+0000 7fa44ea00640 1 -- 172.21.15.67:0/1593066789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b9a0062d0 con 0x561b9a3d5620 2024-04-17T08:52:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa447e00640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a38d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa44f400640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 0x561b9a3b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa44ea00640 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c920 tx=0x7fa44000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523b6c0dad797f59 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa43f400640 1 -- 172.21.15.67:0/1593066789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440019040 con 0x561b9a3d5620 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa43f400640 1 -- 172.21.15.67:0/1593066789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440004020 con 0x561b9a3d5620 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.639+0000 7fa43f400640 1 -- 172.21.15.67:0/1593066789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44000cea0 con 0x561b9a3d5620 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 -- 172.21.15.67:0/1593066789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 msgr2=0x561b9a38cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c920 tx=0x7fa44000f000 comp rx=0 tx=0).stop 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 -- 172.21.15.67:0/1593066789 shutdown_connections 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a387b40 0x561b9a3b7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a38d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 --2- 172.21.15.67:0/1593066789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a38cc50 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 -- 172.21.15.67:0/1593066789 >> 172.21.15.67:0/1593066789 conn(0x561b9a389db0 msgr2=0x561b9a3d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 -- 172.21.15.67:0/1593066789 shutdown_connections 2024-04-17T08:52:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.640+0000 7fa458626740 1 -- 172.21.15.67:0/1593066789 wait complete. 2024-04-17T08:52:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.641+0000 7fa458626740 1 -- start start 2024-04-17T08:52:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.641+0000 7fa458626740 1 Processor -- start 2024-04-17T08:52:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a3ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b9a38bbb0 con 0x561b9a3d5620 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x561b9a2ad590 con 0x561b9a3da6b0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x561b9a38be90 con 0x561b9a38b6a0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa458626740 10 client.0 _ll_register_callbacks cb 0x561b9a005ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.642+0000 7fa447e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55926/0 (socket says 172.21.15.67:55926) 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 -- 172.21.15.67:0/3869931978 learned_addr learned my addr 172.21.15.67:0/3869931978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa44f400640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a3ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 -- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 msgr2=0x561b9a3ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a3ad640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 -- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 msgr2=0x561b9a3b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 -- 172.21.15.67:0/3869931978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b9a0d06b0 con 0x561b9a3d5620 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa44f400640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa447e00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fa438059660 tx=0x7fa43805a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c02a5609364dfd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.643+0000 7fa433e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43801f040 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fa43808a6a0 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa440004020 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) ==== 24+0+0 (secure 0 0 0) 0x7fa438058db0 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 10 client.12690 fetch_fsmap learned FSMap version 143 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=2+}) -- 0x561b9a3d8290 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440004020 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 143) ==== 3495+0+0 (secure 0 0 0) 0x7fa438058f50 con 0x561b9a3d5620 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 10 client.12690 fetch_fsmap finished waiting for FSMap version 143 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 10 client.12690 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 20 client.12690 populate_metadata read hostname 'smithi067' 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3b3320 0x561b9a3b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 4 client.12690 mds_command: new command op to 12552 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:52:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.644+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 --> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b9a3013a0 con 0x561b9a3b3320 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.645+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3b3320 0x561b9a3b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.645+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3b3320 0x561b9a3b3700 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7532f667f4d43286 server_cookie=2480d006f7b05fc4 in_seq=0 out_seq=0 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.645+0000 7fa433e00640 10 client.12690 ms_handle_connect on v2:172.21.15.67:6833/1995843637 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.646+0000 7fa433e00640 1 -- 172.21.15.67:0/3869931978 <== mds.0 v2:172.21.15.67:6833/1995843637 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b9a3013a0 con 0x561b9a3b3320 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.646+0000 7fa433e00640 10 client.12690 handle_command_reply: tid=0 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.646+0000 7fa458626740 1 client.12690 shutdown 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 msgr2=0x561b9a3d8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fa438059660 tx=0x7fa43805a1a0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 shutdown_connections 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3b3320 0x561b9a3b3700 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3d5620 0x561b9a3d8d70 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 --2- 172.21.15.67:0/3869931978 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a38b6a0 0x561b9a3ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa433e00640 0 client.12690 ms_handle_reset on v2:172.21.15.67:6833/1995843637 2024-04-17T08:52:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 >> 172.21.15.67:0/3869931978 conn(0x561b9a387b40 msgr2=0x561b9a38d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 shutdown_connections 2024-04-17T08:52:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 1 -- 172.21.15.67:0/3869931978 wait complete. 2024-04-17T08:52:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.647+0000 7fa458626740 20 client.12690 trim_cache size 0 max 16384 2024-04-17T08:52:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.668+0000 7fa458626740 1 Processor -- start 2024-04-17T08:52:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.668+0000 7fa458626740 1 -- start start 2024-04-17T08:52:52.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500b5270 con 0x561b9a38b6a0 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500b45d0 con 0x561b9a3da6b0 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500b4c20 con 0x561b9a3d5620 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55934/0 (socket says 172.21.15.67:55934) 2024-04-17T08:52:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.669+0000 7fa44ea00640 1 -- 172.21.15.67:0/3909096670 learned_addr learned my addr 172.21.15.67:0/3909096670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 -- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 msgr2=0x561b9a3b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 -- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 msgr2=0x561b9a3b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 -- 172.21.15.67:0/3909096670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b9a3d7a60 con 0x561b9a38b6a0 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x561b9a387ff0 tx=0x7fa440011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcfa0cb894f6b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa43ea00640 1 -- 172.21.15.67:0/3909096670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440004250 con 0x561b9a38b6a0 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa43ea00640 1 -- 172.21.15.67:0/3909096670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4400043f0 con 0x561b9a38b6a0 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.670+0000 7fa43ea00640 1 -- 172.21.15.67:0/3909096670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440013b40 con 0x561b9a38b6a0 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 -- 172.21.15.67:0/3909096670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 msgr2=0x561b9a3d2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x561b9a387ff0 tx=0x7fa440011000 comp rx=0 tx=0).stop 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 -- 172.21.15.67:0/3909096670 shutdown_connections 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a3da6b0 0x561b9a3b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 --2- 172.21.15.67:0/3909096670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a38b6a0 0x561b9a3d2ff0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 -- 172.21.15.67:0/3909096670 >> 172.21.15.67:0/3909096670 conn(0x561b9a387b40 msgr2=0x561b9a38c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 -- 172.21.15.67:0/3909096670 shutdown_connections 2024-04-17T08:52:52.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.671+0000 7fa458626740 1 -- 172.21.15.67:0/3909096670 wait complete. 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.672+0000 7fa458626740 1 -- start start 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.672+0000 7fa458626740 1 Processor -- start 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a38b6a0 0x561b9a3d96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b9a3dabd0 con 0x561b9a3da6b0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x561b9a3d8630 con 0x561b9a38b6a0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x561b9a3aed30 con 0x561b9a3d5620 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.673+0000 7fa458626740 10 client.0 _ll_register_callbacks cb 0x7fa4380597b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55938/0 (socket says 172.21.15.67:55938) 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 -- 172.21.15.67:0/3849515863 learned_addr learned my addr 172.21.15.67:0/3849515863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a38b6a0 0x561b9a3d96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa447e00640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 -- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 msgr2=0x561b9a3b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 -- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a38b6a0 msgr2=0x561b9a3d96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a38b6a0 0x561b9a3d96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa44f400640 1 -- 172.21.15.67:0/3849515863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b9a3c8500 con 0x561b9a3da6b0 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.674+0000 7fa447e00640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa44f400640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fa448009f70 tx=0x7fa448004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a72e7c3b1f054 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa433e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa448009510 con 0x561b9a3da6b0 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fa448034b30 con 0x561b9a3da6b0 2024-04-17T08:52:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4400043f0 con 0x561b9a3da6b0 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) ==== 24+0+0 (secure 0 0 0) 0x7fa448034d30 con 0x561b9a3da6b0 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa458626740 10 client.12696 fetch_fsmap learned FSMap version 143 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.675+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=2+}) -- 0x561b9a3b4260 con 0x561b9a3da6b0 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4400043f0 con 0x561b9a3da6b0 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 143) ==== 3495+0+0 (secure 0 0 0) 0x7fa448008040 con 0x561b9a3da6b0 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 10 client.12696 fetch_fsmap finished waiting for FSMap version 143 2024-04-17T08:52:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 10 client.12696 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 20 client.12696 populate_metadata read hostname 'smithi067' 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3af790 0x561b9a3afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 4 client.12696 mds_command: new command op to 12552 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 --> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] -- command(tid 0: {"prefix": "status"}) -- 0x561b9a2fb210 con 0x561b9a3af790 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.676+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3af790 0x561b9a3afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa44ea00640 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3af790 0x561b9a3afb70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454872f319c4e0e8 server_cookie=b44bcfdd694b34a8 in_seq=0 out_seq=0 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa433e00640 10 client.12696 ms_handle_connect on v2:172.21.15.67:6833/1995843637 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa433e00640 1 -- 172.21.15.67:0/3849515863 <== mds.0 v2:172.21.15.67:6833/1995843637 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x561b9a2fb210 con 0x561b9a3af790 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa433e00640 10 client.12696 handle_command_reply: tid=0 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 client.12696 shutdown 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 msgr2=0x561b9a3b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7fa448009f70 tx=0x7fa448004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 shutdown_connections 2024-04-17T08:52:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b9a3da6b0 0x561b9a3b3c40 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x561b9a3d5620 0x561b9a3b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.67:6833/1995843637,v1:172.21.15.67:6835/1995843637] conn(0x561b9a3af790 0x561b9a3afb70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 --2- 172.21.15.67:0/3849515863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x561b9a38b6a0 0x561b9a3d96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 >> 172.21.15.67:0/3849515863 conn(0x561b9a387b40 msgr2=0x561b9a3bb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.677+0000 7fa433e00640 0 client.12696 ms_handle_reset on v2:172.21.15.67:6833/1995843637 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 shutdown_connections 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa458626740 1 -- 172.21.15.67:0/3849515863 wait complete. 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa458626740 20 client.12696 trim_cache size 0 max 16384 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:52:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 12552, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.9095539800000001, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 143, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 272, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 272, 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.243382036 2024-04-17T08:52:52.680 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 msgr2=0x7fa42c06eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 0x7fa42c06eff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa440006fd0 tx=0x7fa440008040 comp rx=0 tx=0).stop 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 msgr2=0x7fa45014fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7fa43800c7f0 tx=0x7fa43800cbf0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 shutdown_connections 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fa42c06cb30 0x7fa42c06eff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa45015ee20 0x7fa4500cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4500d3350 0x7fa45014fd70 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.678+0000 7fa44fe00640 1 --2- 172.21.15.67:0/932425137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4500d1ab0 0x7fa450158ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.679+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 >> 172.21.15.67:0/932425137 conn(0x7fa45006d3c0 msgr2=0x7fa45015dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.679+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 shutdown_connections 2024-04-17T08:52:52.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.679+0000 7fa44fe00640 1 -- 172.21.15.67:0/932425137 wait complete. 2024-04-17T08:52:52.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:52:52.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.912+0000 7f552be00640 1 Processor -- start 2024-04-17T08:52:52.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.912+0000 7f552be00640 1 -- start start 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f552c160280 0x7f552c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f552c142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552c0cd4a0 con 0x7f552c06b8c0 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f552c15ce30 con 0x7f552c160280 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f552c1607f0 con 0x7f552c141ed0 2024-04-17T08:52:52.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f552c06b8c0 0x7f552c0cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45440/0 (socket says 172.21.15.67:45440) 2024-04-17T08:52:52.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552aa00640 1 -- 172.21.15.67:0/2025441077 learned_addr learned my addr 172.21.15.67:0/2025441077 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552b400640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f552c142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959702129 0 0) 0x7f552c0cd4a0 con 0x7f552c06b8c0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f552a000640 1 --2- 172.21.15.67:0/2025441077 >> v2:172.21.15.187:3300/0 conn(0x7f552c160280 0x7f552c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.914+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5510003520 con 0x7f552c06b8c0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1589402017 0 0) 0x7f5510003520 con 0x7f552c06b8c0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 msgr2=0x7f552c142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f552c142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 >> v2:172.21.15.187:3300/0 conn(0x7f552c160280 msgr2=0x7f552c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 --2- 172.21.15.67:0/2025441077 >> v2:172.21.15.187:3300/0 conn(0x7f552c160280 0x7f552c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c160ad0 con 0x7f552c06b8c0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f551c001670 con 0x7f552c06b8c0 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 >> v1:172.21.15.67:6789/0 conn(0x7f552c06b8c0 legacy=0x7f552c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f5510003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5510003ff0 0x7f5510004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5510004d40 0x7f55100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f552c15ce30 con 0x7f5510004d40 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1607f0 con 0x7f5510003ff0 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f551000ba40 con 0x7f552c141ed0 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.915+0000 7f552b400640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5510004d40 0x7f55100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552a000640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5510003ff0 0x7f5510004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552aa00640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f5510003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552b400640 1 -- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 msgr2=0x7f5510003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552b400640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f552c141ed0 0x7f5510003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552b400640 1 -- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5510003ff0 msgr2=0x7f5510004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552b400640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5510003ff0 0x7f5510004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.916+0000 7f552b400640 1 -- 172.21.15.67:0/2025441077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5510003520 con 0x7f5510004d40 2024-04-17T08:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.917+0000 7f552b400640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5510004d40 0x7f55100051c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f5524009a90 tx=0x7f5510005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dddf13a949b0779 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.917+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524039070 con 0x7f5510004d40 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.917+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552402c680 con 0x7f5510004d40 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.917+0000 7f5529600640 1 -- 172.21.15.67:0/2025441077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524009690 con 0x7f5510004d40 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 -- 172.21.15.67:0/2025441077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5510004d40 msgr2=0x7f55100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 --2- 172.21.15.67:0/2025441077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5510004d40 0x7f55100051c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f5524009a90 tx=0x7f5510005830 comp rx=0 tx=0).stop 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552b400640 1 -- 172.21.15.67:0/2025441077 reap_dead start 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 -- 172.21.15.67:0/2025441077 shutdown_connections 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 -- 172.21.15.67:0/2025441077 >> 172.21.15.67:0/2025441077 conn(0x7f552c067d10 msgr2=0x7f552c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 -- 172.21.15.67:0/2025441077 shutdown_connections 2024-04-17T08:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.918+0000 7f552be00640 1 -- 172.21.15.67:0/2025441077 wait complete. 2024-04-17T08:52:52.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.920+0000 7f552be00640 1 Processor -- start 2024-04-17T08:52:52.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.921+0000 7f552be00640 1 -- start start 2024-04-17T08:52:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5510004d40 0x7f552c13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 0x7f552c145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f552c161090 con 0x7f552c0b5150 2024-04-17T08:52:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f552c06c1e0 con 0x7f552c15bb30 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f552c13f9b0 con 0x7f5510004d40 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55964/0 (socket says 172.21.15.67:55964) 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.922+0000 7f552a000640 1 -- 172.21.15.67:0/1732474598 learned_addr learned my addr 172.21.15.67:0/1732474598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552b400640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 0x7f552c145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552aa00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5510004d40 0x7f552c13ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 -- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5510004d40 msgr2=0x7f552c13ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5510004d40 0x7f552c13ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 -- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 msgr2=0x7f552c145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 0x7f552c145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 -- 172.21.15.67:0/1732474598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c13f3a0 con 0x7f552c0b5150 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552b400640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 0x7f552c145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f552a000640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f5518007f00 tx=0x7f55180115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.923+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551801a020 con 0x7f552c0b5150 2024-04-17T08:52:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.924+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5518016470 con 0x7f552c0b5150 2024-04-17T08:52:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.924+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552c0b4390 con 0x7f552c0b5150 2024-04-17T08:52:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.924+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5518016860 con 0x7f552c0b5150 2024-04-17T08:52:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.924+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f552c0b4950 con 0x7f552c0b5150 2024-04-17T08:52:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.925+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552c068a10 con 0x7f552c0b5150 2024-04-17T08:52:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.926+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f5518005d10 con 0x7f552c0b5150 2024-04-17T08:52:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.927+0000 7f5522a00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 0x7f550c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.927+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f551808cfd0 con 0x7f552c0b5150 2024-04-17T08:52:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.928+0000 7f552aa00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 0x7f550c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.929+0000 7f552aa00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 0x7f550c06f020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f551c0058b0 tx=0x7f551c000f50 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:52.932+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5518024030 con 0x7f552c0b5150 2024-04-17T08:52:53.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.178+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f552c15bf10 con 0x7f552c0b5150 2024-04-17T08:52:53.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.179+0000 7f5522a00640 1 -- 172.21.15.67:0/1732474598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v143) ==== 53+0+73 (secure 0 0 0) 0x7f552c13f3a0 con 0x7f552c0b5150 2024-04-17T08:52:53.177 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 msgr2=0x7f550c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 0x7f550c06f020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f551c0058b0 tx=0x7f551c000f50 comp rx=0 tx=0).stop 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 msgr2=0x7f552c15b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f5518007f00 tx=0x7f55180115a0 comp rx=0 tx=0).stop 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 shutdown_connections 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f550c06cb60 0x7f550c06f020 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f552c15bb30 0x7f552c145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552c0b5150 0x7f552c15b5f0 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 --2- 172.21.15.67:0/1732474598 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5510004d40 0x7f552c13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 >> 172.21.15.67:0/1732474598 conn(0x7f552c067d10 msgr2=0x7f552c074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 shutdown_connections 2024-04-17T08:52:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53.182+0000 7f552be00640 1 -- 172.21.15.67:0/1732474598 wait complete. 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:52:53.200 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:52:53.201 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:52:53.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.201 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.255 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 36671sec preferred_lft 36671sec 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:52:53.256 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:52:53.257 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:52:53.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.258 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:52:53.350 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:52:53.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.354 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:52:53.413 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:52:53.413 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:52:53.413 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:52:53.414 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:52:53.414 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:52:53.414 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:52:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:52:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:52:53.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.679 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:52:53.735 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:52:53.735 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:52:53.736 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:52:53.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:52:53.845 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:52:53.846 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:52:53.904 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:52:53.904 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:52:53.905 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:52:53.952 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:52:53.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:53.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:52:54.091 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:52:54.091 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:52:54.127 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:52:54.127 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:52:54.127 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:52:54.128 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:52:54.128 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:52:54.130 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:52:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:52:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:52:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:52:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:52:54.135 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:52:54.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:54.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:52:54.226 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:52:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:52:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:52:54.361 INFO:teuthology.orchestra.run.smithi067.stdout:client8594 2024-04-17T08:52:54.361 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:52:54.361 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:52:54.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:52:54.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.638+0000 7f2f8fa00640 1 Processor -- start 2024-04-17T08:52:54.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.639+0000 7f2f8fa00640 1 -- start start 2024-04-17T08:52:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f8fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f88169bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f8fa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f880d1520 con 0x7f2f8806ce00 2024-04-17T08:52:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f8fa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f880d1800 con 0x7f2f8806daf0 2024-04-17T08:52:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f8fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f880d1ae0 con 0x7f2f8809b170 2024-04-17T08:52:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f87e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f8806ce00 0x7f2f8809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60480/0 (socket says 172.21.15.67:60480) 2024-04-17T08:52:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.640+0000 7f2f87e00640 1 -- 172.21.15.67:0/1186679198 learned_addr learned my addr 172.21.15.67:0/1186679198 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:54.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f8da00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f88169bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195564732 0 0) 0x7f2f880d1520 con 0x7f2f8806ce00 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f70003520 con 0x7f2f8806ce00 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 842343402 0 0) 0x7f2f70003520 con 0x7f2f8806ce00 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 msgr2=0x7f2f88169bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f88169bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 >> v1:172.21.15.187:6789/0 conn(0x7f2f8806daf0 legacy=0x7f2f880d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f880d1dc0 con 0x7f2f8806ce00 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f2f74001670 con 0x7f2f8806ce00 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 >> v1:172.21.15.67:6789/0 conn(0x7f2f8806ce00 legacy=0x7f2f8809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f70004d40 0x7f2f700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f880d1ae0 con 0x7f2f70003fb0 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7000bb00 con 0x7f2f70004d40 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.641+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7000bd00 con 0x7f2f8809b170 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f8d000640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f8da00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f70004d40 0x7f2f700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 -- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 msgr2=0x7f2f700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 -- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f70004d40 msgr2=0x7f2f700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f70004d40 0x7f2f700051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f87e00640 1 -- 172.21.15.67:0/1186679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f70003520 con 0x7f2f70003fb0 2024-04-17T08:52:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.642+0000 7f2f8d000640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f8809b170 0x7f2f700039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.643+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f70004800 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c0e0 tx=0x7f2f70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f8bbd06e0513a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.643+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74007c40 con 0x7f2f70003fb0 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.643+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f74030070 con 0x7f2f70003fb0 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.643+0000 7f2f87400640 1 -- 172.21.15.67:0/1186679198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74005d70 con 0x7f2f70003fb0 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1186679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 msgr2=0x7f2f70004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1186679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f70004800 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c0e0 tx=0x7f2f70005830 comp rx=0 tx=0).stop 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8da00640 1 -- 172.21.15.67:0/1186679198 reap_dead start 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1186679198 shutdown_connections 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1186679198 >> 172.21.15.67:0/1186679198 conn(0x7f2f880b41a0 msgr2=0x7f2f880b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1186679198 shutdown_connections 2024-04-17T08:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.644+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1186679198 wait complete. 2024-04-17T08:52:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.646+0000 7f2f8fa00640 1 Processor -- start 2024-04-17T08:52:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.646+0000 7f2f8fa00640 1 -- start start 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 0x7f2f880b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f880b6ce0 0x7f2f880b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8806ded0 con 0x7f2f70003fb0 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2f880d1230 con 0x7f2f880b6ce0 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8806e050 con 0x7f2f88154e40 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52248/0 (socket says 172.21.15.67:52248) 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.647+0000 7f2f8d000640 1 -- 172.21.15.67:0/1471619418 learned_addr learned my addr 172.21.15.67:0/1471619418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8da00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f880b6ce0 0x7f2f880b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 0x7f2f880b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 -- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 msgr2=0x7f2f880b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 0x7f2f880b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 -- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f880b6ce0 msgr2=0x7f2f880b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f880b6ce0 0x7f2f880b7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 -- 172.21.15.67:0/1471619418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f880b5c40 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 0x7f2f880b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f8d000640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00d7a0 tx=0x7f2f7c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.648+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c005070 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.649+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c005210 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.649+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c007030 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.649+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f880b5f20 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.649+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f880b64e0 con 0x7f2f70003fb0 2024-04-17T08:52:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.650+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f88068ca0 con 0x7f2f70003fb0 2024-04-17T08:52:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.651+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f2f7c01a020 con 0x7f2f70003fb0 2024-04-17T08:52:54.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.652+0000 7f2f84c00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 0x7f2f6806f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:54.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.652+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f2f7c08b9a0 con 0x7f2f70003fb0 2024-04-17T08:52:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.653+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 0x7f2f6806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.654+0000 7f2f87e00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 0x7f2f6806f020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f74034000 tx=0x7f2f74009030 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.657+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2f7c059870 con 0x7f2f70003fb0 2024-04-17T08:52:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.885+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 --> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f2f880b5040 con 0x7f2f6806cb60 2024-04-17T08:52:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.889+0000 7f2f84c00640 1 -- 172.21.15.67:0/1471619418 <== mgr.8339 v2:172.21.15.187:6832/1143620423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2f880b5040 con 0x7f2f6806cb60 2024-04-17T08:52:54.887 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:52:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:52:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:52:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:52:54.888 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 msgr2=0x7f2f6806f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 0x7f2f6806f020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f74034000 tx=0x7f2f74009030 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 msgr2=0x7f2f88154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00d7a0 tx=0x7f2f7c00dc60 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 shutdown_connections 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f2f6806cb60 0x7f2f6806f020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2f880b6ce0 0x7f2f880b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2f88154e40 0x7f2f880b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 --2- 172.21.15.67:0/1471619418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f70003fb0 0x7f2f88154720 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:54.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.891+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 >> 172.21.15.67:0/1471619418 conn(0x7f2f880b41a0 msgr2=0x7f2f880d0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:54.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.892+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 shutdown_connections 2024-04-17T08:52:54.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:54.892+0000 7f2f8fa00640 1 -- 172.21.15.67:0/1471619418 wait complete. 2024-04-17T08:52:54.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:52:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.131+0000 7f9538a00640 1 Processor -- start 2024-04-17T08:52:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.131+0000 7f9538a00640 1 -- start start 2024-04-17T08:52:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.132+0000 7f9538a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953415ecb0 0x7f95340d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9538a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 0x7f95340b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9538a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95340d00b0 con 0x7f95340d3210 2024-04-17T08:52:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9538a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f95340d0c30 con 0x7f953415ecb0 2024-04-17T08:52:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9538a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f95340d0390 con 0x7f95340cfab0 2024-04-17T08:52:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9532800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953415ecb0 0x7f95340d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9531e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95340d3210 0x7f95340cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60484/0 (socket says 172.21.15.67:60484) 2024-04-17T08:52:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9532800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f953415ecb0 0x7f95340d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52264/0 (socket says 172.21.15.67:52264) 2024-04-17T08:52:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9531e00640 1 -- 172.21.15.67:0/435263555 learned_addr learned my addr 172.21.15.67:0/435263555 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:52:55.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9533200640 1 --2- 172.21.15.67:0/435263555 >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 0x7f95340b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9532800640 1 -- 172.21.15.67:0/435263555 >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 msgr2=0x7f95340b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9532800640 1 --2- 172.21.15.67:0/435263555 >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 0x7f95340b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.133+0000 7f9532800640 1 -- 172.21.15.67:0/435263555 >> v1:172.21.15.67:6789/0 conn(0x7f95340d3210 legacy=0x7f95340cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.134+0000 7f9532800640 1 -- 172.21.15.67:0/435263555 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95340d0670 con 0x7f953415ecb0 2024-04-17T08:52:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.134+0000 7f9533200640 1 --2- 172.21.15.67:0/435263555 >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 0x7f95340b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.135+0000 7f9532800640 1 --2- 172.21.15.67:0/435263555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953415ecb0 0x7f95340d2cd0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f95340c81b0 tx=0x7f952402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c61c47a3bcdd9ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.135+0000 7f9531400640 1 -- 172.21.15.67:0/435263555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524039070 con 0x7f953415ecb0 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.135+0000 7f9531400640 1 -- 172.21.15.67:0/435263555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f952402cbe0 con 0x7f953415ecb0 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.135+0000 7f9531400640 1 -- 172.21.15.67:0/435263555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524034440 con 0x7f953415ecb0 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 -- 172.21.15.67:0/435263555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953415ecb0 msgr2=0x7f95340d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 --2- 172.21.15.67:0/435263555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953415ecb0 0x7f95340d2cd0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f95340c81b0 tx=0x7f952402be30 comp rx=0 tx=0).stop 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 -- 172.21.15.67:0/435263555 shutdown_connections 2024-04-17T08:52:55.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 --2- 172.21.15.67:0/435263555 >> v2:172.21.15.187:3300/0 conn(0x7f95340cfab0 0x7f95340b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 --2- 172.21.15.67:0/435263555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f953415ecb0 0x7f95340d2cd0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 -- 172.21.15.67:0/435263555 >> 172.21.15.67:0/435263555 conn(0x7f953408bff0 msgr2=0x7f95340d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 -- 172.21.15.67:0/435263555 shutdown_connections 2024-04-17T08:52:55.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.136+0000 7f9538a00640 1 -- 172.21.15.67:0/435263555 wait complete. 2024-04-17T08:52:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.138+0000 7f9538a00640 1 Processor -- start 2024-04-17T08:52:55.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.138+0000 7f9538a00640 1 -- start start 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 0x7f95340cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953415ecb0 0x7f95340b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f953415f210 con 0x7f95340cfab0 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f953415f090 con 0x7f95340d3210 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9538a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95340b4500 con 0x7f953415ecb0 2024-04-17T08:52:55.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9532800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9532800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52266/0 (socket says 172.21.15.67:52266) 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9532800640 1 -- 172.21.15.67:0/1965125063 learned_addr learned my addr 172.21.15.67:0/1965125063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9531e00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 0x7f95340cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9532800640 1 -- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953415ecb0 msgr2=0x7f95340b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.139+0000 7f9532800640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953415ecb0 0x7f95340b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9532800640 1 -- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 msgr2=0x7f95340cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9532800640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 0x7f95340cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9532800640 1 -- 172.21.15.67:0/1965125063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95340cc120 con 0x7f95340cfab0 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9531e00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 0x7f95340cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:52:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9532800640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f9524009e80 tx=0x7f9524004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9524039050 con 0x7f95340cfab0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9524045070 con 0x7f95340cfab0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f952403ea50 con 0x7f95340cfab0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.140+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95340cc400 con 0x7f95340cfab0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.141+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95340cc9c0 con 0x7f95340cfab0 2024-04-17T08:52:55.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.142+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9534068a10 con 0x7f95340cfab0 2024-04-17T08:52:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.143+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 147) ==== 91244+0+0 (secure 0 0 0) 0x7f952402ca60 con 0x7f95340cfab0 2024-04-17T08:52:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.144+0000 7f952aa00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 0x7f951006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.144+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f95240aef30 con 0x7f95340cfab0 2024-04-17T08:52:55.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.144+0000 7f9531e00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 0x7f951006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.146+0000 7f9531e00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 0x7f951006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f95340b85a0 tx=0x7f951c009040 comp rx=0 tx=0).ready entity=mgr.8339 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:55.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.149+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f952407d030 con 0x7f95340cfab0 2024-04-17T08:52:55.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.406+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f95340b6fe0 con 0x7f95340cfab0 2024-04-17T08:52:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.864+0000 7f952aa00640 1 -- 172.21.15.67:0/1965125063 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v148) ==== 92+0+0 (secure 0 0 0) 0x7f952407c930 con 0x7f95340cfab0 2024-04-17T08:52:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.869+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 msgr2=0x7f951006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.869+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 0x7f951006eff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f95340b85a0 tx=0x7f951c009040 comp rx=0 tx=0).stop 2024-04-17T08:52:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.869+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 msgr2=0x7f95341539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:55.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.869+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f9524009e80 tx=0x7f9524004a40 comp rx=0 tx=0).stop 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 shutdown_connections 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f951006cb30 0x7f951006eff0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f953415ecb0 0x7f95340b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95340d3210 0x7f95340cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 --2- 172.21.15.67:0/1965125063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95340cfab0 0x7f95341539b0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 >> 172.21.15.67:0/1965125063 conn(0x7f953408bff0 msgr2=0x7f95340b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 shutdown_connections 2024-04-17T08:52:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:55.870+0000 7f9538a00640 1 -- 172.21.15.67:0/1965125063 wait complete. 2024-04-17T08:52:55.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:52:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.106+0000 7fdf5be00640 1 Processor -- start 2024-04-17T08:52:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.107+0000 7fdf5be00640 1 -- start start 2024-04-17T08:52:56.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.108+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 0x7fdf5c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.108+0000 7fdf5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.108+0000 7fdf5be00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf5c0b51d0 con 0x7fdf5c0b4c30 2024-04-17T08:52:56.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.108+0000 7fdf5be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdf5c0b5790 con 0x7fdf5c0cf910 2024-04-17T08:52:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.108+0000 7fdf5be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5c0d2830 con 0x7fdf5c15e9b0 2024-04-17T08:52:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52288/0 (socket says 172.21.15.67:52288) 2024-04-17T08:52:56.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 -- 172.21.15.67:0/3043873420 learned_addr learned my addr 172.21.15.67:0/3043873420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5aa00640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 0x7fdf5c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 -- 172.21.15.67:0/3043873420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 msgr2=0x7fdf5c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 0x7fdf5c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 -- 172.21.15.67:0/3043873420 >> v1:172.21.15.187:6789/0 conn(0x7fdf5c0b4c30 legacy=0x7fdf5c0d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5a000640 1 -- 172.21.15.67:0/3043873420 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c0b54b0 con 0x7fdf5c0cf910 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.109+0000 7fdf5aa00640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 0x7fdf5c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.111+0000 7fdf5a000640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009870 tx=0x7fdf4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5daa9b876124727 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf59600640 1 -- 172.21.15.67:0/3043873420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035020 con 0x7fdf5c0cf910 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf59600640 1 -- 172.21.15.67:0/3043873420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf48031640 con 0x7fdf5c0cf910 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf59600640 1 -- 172.21.15.67:0/3043873420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802d040 con 0x7fdf5c0cf910 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 -- 172.21.15.67:0/3043873420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 msgr2=0x7fdf5c0b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009870 tx=0x7fdf4802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 -- 172.21.15.67:0/3043873420 shutdown_connections 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b46f0 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 --2- 172.21.15.67:0/3043873420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c15e9b0 0x7fdf5c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.112+0000 7fdf5be00640 1 -- 172.21.15.67:0/3043873420 >> 172.21.15.67:0/3043873420 conn(0x7fdf5c09daa0 msgr2=0x7fdf5c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.113+0000 7fdf5be00640 1 -- 172.21.15.67:0/3043873420 shutdown_connections 2024-04-17T08:52:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.113+0000 7fdf5be00640 1 -- 172.21.15.67:0/3043873420 wait complete. 2024-04-17T08:52:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.114+0000 7fdf5be00640 1 Processor -- start 2024-04-17T08:52:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.115+0000 7fdf5be00640 1 -- start start 2024-04-17T08:52:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c0b4c30 0x7fdf5c0c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5c15e9b0 0x7fdf5c0b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c15ed90 con 0x7fdf5c0cf910 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c162830 con 0x7fdf5c15e9b0 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5c15ef10 con 0x7fdf5c0b4c30 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52302/0 (socket says 172.21.15.67:52302) 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5a000640 1 -- 172.21.15.67:0/1516908927 learned_addr learned my addr 172.21.15.67:0/1516908927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5aa00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c0b4c30 0x7fdf5c0c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.116+0000 7fdf5b400640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5c15e9b0 0x7fdf5c0b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c0b4c30 msgr2=0x7fdf5c0c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c0b4c30 0x7fdf5c0c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5c15e9b0 msgr2=0x7fdf5c0b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5c15e9b0 0x7fdf5c0b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 -- 172.21.15.67:0/1516908927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c15f160 con 0x7fdf5c0cf910 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5a000640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48040070 con 0x7fdf5c0cf910 2024-04-17T08:52:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf480399f0 con 0x7fdf5c0cf910 2024-04-17T08:52:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035040 con 0x7fdf5c0cf910 2024-04-17T08:52:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.117+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c0c9550 con 0x7fdf5c0cf910 2024-04-17T08:52:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.118+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c0c9b10 con 0x7fdf5c0cf910 2024-04-17T08:52:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.119+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c0755d0 con 0x7fdf5c0cf910 2024-04-17T08:52:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.120+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 148) ==== 91261+0+0 (secure 0 0 0) 0x7fdf4804b060 con 0x7fdf5c0cf910 2024-04-17T08:52:56.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.121+0000 7fdf52a00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 0x7fdf3c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.121+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fdf480ad9c0 con 0x7fdf5c0cf910 2024-04-17T08:52:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.121+0000 7fdf5aa00640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 msgr2=0x7fdf3c06f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.121+0000 7fdf5aa00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 0x7fdf3c06f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:56.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.126+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdf4807ba30 con 0x7fdf5c0cf910 2024-04-17T08:52:56.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.322+0000 7fdf5aa00640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 msgr2=0x7fdf3c06f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:56.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.322+0000 7fdf5aa00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 0x7fdf3c06f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.333+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fdf5c0b64c0 con 0x7fdf5c0cf910 2024-04-17T08:52:56.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.338+0000 7fdf52a00640 1 -- 172.21.15.67:0/1516908927 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) ==== 138+0+0 (secure 0 0 0) 0x7fdf4807b330 con 0x7fdf5c0cf910 2024-04-17T08:52:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 msgr2=0x7fdf3c06f070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 0x7fdf3c06f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 msgr2=0x7fdf5c0b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 shutdown_connections 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fdf3c06cbb0 0x7fdf3c06f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdf5c15e9b0 0x7fdf5c0b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cf910 0x7fdf5c0b88e0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1516908927 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdf5c0b4c30 0x7fdf5c0c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 >> 172.21.15.67:0/1516908927 conn(0x7fdf5c09daa0 msgr2=0x7fdf5c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 shutdown_connections 2024-04-17T08:52:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.342+0000 7fdf5be00640 1 -- 172.21.15.67:0/1516908927 wait complete. 2024-04-17T08:52:56.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:52:56.423 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:56.424+0000 7fa85823c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:56.469 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:56.471+0000 7fb929445040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:52:56.478 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:56.478+0000 7fa85823c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:56.533 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:56.534+0000 7fb929445040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:52:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.572+0000 7f9f54400640 1 Processor -- start 2024-04-17T08:52:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.572+0000 7f9f54400640 1 -- start start 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9f4c06b8c0 0x7f9f4c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c0d0b50 0x7f9f4c15fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f4c163710 con 0x7f9f4c1627b0 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9f4c15f360 con 0x7f9f4c06b8c0 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f54400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c06c1e0 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f52400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f52400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52316/0 (socket says 172.21.15.67:52316) 2024-04-17T08:52:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.574+0000 7f9f52400640 1 -- 172.21.15.67:0/1391010139 learned_addr learned my addr 172.21.15.67:0/1391010139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f51a00640 1 --2- 172.21.15.67:0/1391010139 >> v2:172.21.15.187:3300/0 conn(0x7f9f4c06b8c0 0x7f9f4c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f51000640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c0d0b50 0x7f9f4c15fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f52400640 1 -- 172.21.15.67:0/1391010139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c0d0b50 msgr2=0x7f9f4c15fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f52400640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c0d0b50 0x7f9f4c15fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f52400640 1 -- 172.21.15.67:0/1391010139 >> v2:172.21.15.187:3300/0 conn(0x7f9f4c06b8c0 msgr2=0x7f9f4c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f52400640 1 --2- 172.21.15.67:0/1391010139 >> v2:172.21.15.187:3300/0 conn(0x7f9f4c06b8c0 0x7f9f4c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.575+0000 7f9f52400640 1 -- 172.21.15.67:0/1391010139 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c163150 con 0x7f9f4c1627b0 2024-04-17T08:52:56.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.576+0000 7f9f52400640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0d2660 tx=0x7f9f4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa4ca8907470395 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:56.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.576+0000 7f9f4be00640 1 -- 172.21.15.67:0/1391010139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44035020 con 0x7f9f4c1627b0 2024-04-17T08:52:56.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.576+0000 7f9f4be00640 1 -- 172.21.15.67:0/1391010139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f44031870 con 0x7f9f4c1627b0 2024-04-17T08:52:56.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.576+0000 7f9f4be00640 1 -- 172.21.15.67:0/1391010139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44031c40 con 0x7f9f4c1627b0 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 -- 172.21.15.67:0/1391010139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c1627b0 msgr2=0x7f9f4c0cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0d2660 tx=0x7f9f4402b910 comp rx=0 tx=0).stop 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 -- 172.21.15.67:0/1391010139 shutdown_connections 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c0cdf10 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 --2- 172.21.15.67:0/1391010139 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c0d0b50 0x7f9f4c15fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 --2- 172.21.15.67:0/1391010139 >> v2:172.21.15.187:3300/0 conn(0x7f9f4c06b8c0 0x7f9f4c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 -- 172.21.15.67:0/1391010139 >> 172.21.15.67:0/1391010139 conn(0x7f9f4c073360 msgr2=0x7f9f4c09aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.577+0000 7f9f54400640 1 -- 172.21.15.67:0/1391010139 shutdown_connections 2024-04-17T08:52:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.578+0000 7f9f54400640 1 -- 172.21.15.67:0/1391010139 wait complete. 2024-04-17T08:52:56.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.580+0000 7f9f54400640 1 Processor -- start 2024-04-17T08:52:56.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.580+0000 7f9f54400640 1 -- start start 2024-04-17T08:52:56.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.581+0000 7f9f54400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 0x7f9f4c1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.581+0000 7f9f54400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f54400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f54400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c160060 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f54400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c15f600 con 0x7f9f4c1627b0 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f54400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c1601e0 con 0x7f9f4c06b8c0 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52328/0 (socket says 172.21.15.67:52328) 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 -- 172.21.15.67:0/1025694771 learned_addr learned my addr 172.21.15.67:0/1025694771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51a00640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 0x7f9f4c1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f52400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 msgr2=0x7f9f4c1527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 0x7f9f4c1527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f4c1627b0 msgr2=0x7f9f4c142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51000640 1 -- 172.21.15.67:0/1025694771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c158f90 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.582+0000 7f9f51a00640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 0x7f9f4c1527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f51000640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800d830 tx=0x7f9f3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800b660 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f38019070 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4c0c6310 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800ba20 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.583+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4c0c6bb0 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.585+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f4c068ca0 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.586+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 148) ==== 91261+0+0 (secure 0 0 0) 0x7f9f3801e460 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.587+0000 7f9f49600640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 0x7f9f2c06f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:56.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.587+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7f9f3808c6e0 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.587+0000 7f9f51a00640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 msgr2=0x7f9f2c06f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:56.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.587+0000 7f9f51a00640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 0x7f9f2c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.591+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9f4c158f90 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.788+0000 7f9f51a00640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 msgr2=0x7f9f2c06f040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.788+0000 7f9f51a00640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 0x7f9f2c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:56.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.795+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9f4c0b07a0 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.800+0000 7f9f49600640 1 -- 172.21.15.67:0/1025694771 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) ==== 133+0+0 (secure 0 0 0) 0x7f9f3805a760 con 0x7f9f4c0d0b50 2024-04-17T08:52:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 msgr2=0x7f9f2c06f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 0x7f9f2c06f040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 msgr2=0x7f9f4c158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800d830 tx=0x7f9f3800dcf0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 shutdown_connections 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7f9f2c06cb80 0x7f9f2c06f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f4c1627b0 0x7f9f4c142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f4c0d0b50 0x7f9f4c158a50 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 --2- 172.21.15.67:0/1025694771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f4c06b8c0 0x7f9f4c1527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 >> 172.21.15.67:0/1025694771 conn(0x7f9f4c073360 msgr2=0x7f9f4c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 shutdown_connections 2024-04-17T08:52:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:56.803+0000 7f9f54400640 1 -- 172.21.15.67:0/1025694771 wait complete. 2024-04-17T08:52:56.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:52:56.821 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-04-17T08:52:56.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-04-17T08:52:56.925 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:56.925+0000 7fa85823c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:56.976 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:56.977+0000 7fa85823c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:57.015 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.016+0000 7fb929445040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:52:57.027 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.028+0000 7fa85823c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:57.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.068+0000 7fc453e00640 1 Processor -- start 2024-04-17T08:52:57.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.068+0000 7fc453e00640 1 -- start start 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.070+0000 7fc453e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc45406b8c0 0x7fc4540c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.070+0000 7fc453e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.070+0000 7fc453e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4540c8110 con 0x7fc45415ed10 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.070+0000 7fc453e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc45415f250 con 0x7fc45406b8c0 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.070+0000 7fc453e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc45415f530 con 0x7fc45415fd30 2024-04-17T08:52:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.071+0000 7fc44be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.071+0000 7fc452a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc45406b8c0 0x7fc4540c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.071+0000 7fc44be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:50712/0 (socket says 172.21.15.67:50712) 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.071+0000 7fc44be00640 1 -- 172.21.15.67:0/905939570 learned_addr learned my addr 172.21.15.67:0/905939570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.072+0000 7fc44be00640 1 -- 172.21.15.67:0/905939570 >> v2:172.21.15.187:3300/0 conn(0x7fc45406b8c0 msgr2=0x7fc4540c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.072+0000 7fc44be00640 1 --2- 172.21.15.67:0/905939570 >> v2:172.21.15.187:3300/0 conn(0x7fc45406b8c0 0x7fc4540c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.072+0000 7fc44be00640 1 -- 172.21.15.67:0/905939570 >> v1:172.21.15.187:6789/0 conn(0x7fc45415ed10 legacy=0x7fc4540cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:57.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.072+0000 7fc44be00640 1 -- 172.21.15.67:0/905939570 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45415f810 con 0x7fc45415fd30 2024-04-17T08:52:57.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.073+0000 7fc44be00640 1 --2- 172.21.15.67:0/905939570 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009870 tx=0x7fc43c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9cffe9a656c1812 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.073+0000 7fc452000640 1 -- 172.21.15.67:0/905939570 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0317c0 con 0x7fc45415fd30 2024-04-17T08:52:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.073+0000 7fc452000640 1 -- 172.21.15.67:0/905939570 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c037070 con 0x7fc45415fd30 2024-04-17T08:52:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc452000640 1 -- 172.21.15.67:0/905939570 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c031b80 con 0x7fc45415fd30 2024-04-17T08:52:57.072 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fb929445040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 -- 172.21.15.67:0/905939570 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 msgr2=0x7fc45406bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 --2- 172.21.15.67:0/905939570 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009870 tx=0x7fc43c02b860 comp rx=0 tx=0).stop 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 -- 172.21.15.67:0/905939570 shutdown_connections 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 --2- 172.21.15.67:0/905939570 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45415fd30 0x7fc45406bca0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 --2- 172.21.15.67:0/905939570 >> v2:172.21.15.187:3300/0 conn(0x7fc45406b8c0 0x7fc4540c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.074+0000 7fc453e00640 1 -- 172.21.15.67:0/905939570 >> 172.21.15.67:0/905939570 conn(0x7fc4540b2fe0 msgr2=0x7fc4540b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.075+0000 7fc453e00640 1 -- 172.21.15.67:0/905939570 shutdown_connections 2024-04-17T08:52:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.075+0000 7fc453e00640 1 -- 172.21.15.67:0/905939570 wait complete. 2024-04-17T08:52:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.077+0000 7fc453e00640 1 Processor -- start 2024-04-17T08:52:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.077+0000 7fc453e00640 1 -- start start 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45406b8c0 0x7fc454149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc45415ed10 0x7fc4540ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 0x7fc4540b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc45406c360 con 0x7fc45415fd30 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc45406c1e0 con 0x7fc45415ed10 2024-04-17T08:52:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.078+0000 7fc453e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc45406c4e0 con 0x7fc45406b8c0 2024-04-17T08:52:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 0x7fc4540b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 0x7fc4540b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52330/0 (socket says 172.21.15.67:52330) 2024-04-17T08:52:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 -- 172.21.15.67:0/519471650 learned_addr learned my addr 172.21.15.67:0/519471650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:52:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45406b8c0 0x7fc454149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc44be00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc45415ed10 0x7fc4540ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45406b8c0 msgr2=0x7fc454149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc45406b8c0 0x7fc454149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc45415ed10 msgr2=0x7fc4540ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc45415ed10 0x7fc4540ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.079+0000 7fc453400640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540b75b0 con 0x7fc45415fd30 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc453400640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 0x7fc4540b3fd0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c7d0 tx=0x7fc44c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c003b70 con 0x7fc45415fd30 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44c003d30 con 0x7fc45415fd30 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4540b6ff0 con 0x7fc45415fd30 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c00aa30 con 0x7fc45415fd30 2024-04-17T08:52:57.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.080+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4540c9fc0 con 0x7fc45415fd30 2024-04-17T08:52:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.082+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc454073c50 con 0x7fc45415fd30 2024-04-17T08:52:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.084+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 148) ==== 91261+0+0 (secure 0 0 0) 0x7fc44c00abd0 con 0x7fc45415fd30 2024-04-17T08:52:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.085+0000 7fc44aa00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:52:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.085+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5780+0+0 (secure 0 0 0) 0x7fc44c08c850 con 0x7fc45415fd30 2024-04-17T08:52:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.085+0000 7fc452a00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.085+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:52:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.089+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc454073c50 con 0x7fc45415fd30 2024-04-17T08:52:57.090 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.090+0000 7fa85823c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:57.128 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.130+0000 7fb929445040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:52:57.139 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.140+0000 7fa85823c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:57.194 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.195+0000 7fb929445040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:52:57.246 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.248+0000 7fb929445040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:52:57.267 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.268+0000 7fa85823c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:57.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.286+0000 7fc452a00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:57.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.286+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:52:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.317+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc4540755d0 con 0x7fc43006cbb0 2024-04-17T08:52:57.330 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.331+0000 7fa85823c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:57.381 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.381+0000 7fa85823c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:57.384 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.385+0000 7fb929445040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:52:57.438 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.438+0000 7fa85823c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:57.450 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.452+0000 7fb929445040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:57.489 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.489+0000 7fa85823c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:57.503 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.504+0000 7fb929445040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:52:57.563 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.565+0000 7fb929445040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:52:57.617 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:57.618+0000 7fb929445040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:52:57.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.687+0000 7fc452a00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:57.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:57.687+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:52:57.875 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:57.876+0000 7fa85823c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:58.022 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.023+0000 7fb929445040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:52:58.034 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.035+0000 7fa85823c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:58.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:52:58.143 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.144+0000 7fa85823c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:58.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:52:58.193 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.194+0000 7fb929445040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:52:58.197 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.198+0000 7fa85823c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:58.309 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.311+0000 7fb929445040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:52:58.327 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.328+0000 7fa85823c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:58.363 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.364+0000 7fb929445040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:52:58.436 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.437+0000 7fa85823c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:58.488+0000 7fc452a00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:52:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:52:58.489+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:52:58.495 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.497+0000 7fb929445040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:58.608 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.610+0000 7fb929445040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:52:58.610 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.610+0000 7fa85823c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:58.661 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.662+0000 7fa85823c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:58.792 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.793+0000 7fb929445040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:52:58.797 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:58.797+0000 7fa85823c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:58.844 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.846+0000 7fb929445040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:52:58.989 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:58.990+0000 7fb929445040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:52:59.622 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:59.622+0000 7fa85823c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:59.875 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:52:59.877+0000 7fb929445040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:52:59.924 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:59.924+0000 7fa85823c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:52:59.981 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:52:59.981+0000 7fa85823c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:00.091+0000 7fc452a00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1143620423 2024-04-17T08:53:00.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:00.091+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T08:53:00.111 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.111+0000 7fa85823c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:00.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:00.136+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 149) ==== 90768+0+0 (secure 0 0 0) 0x7fc44c056940 con 0x7fc45415fd30 2024-04-17T08:53:00.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:00.136+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 msgr2=0x7fc43006f070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:00.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:00.136+0000 7fc44aa00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1143620423,v1:172.21.15.187:6833/1143620423] conn(0x7fc43006cbb0 0x7fc43006f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:00.204 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:00.205+0000 7fb929445040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:53:00.269 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:00.270+0000 7fb929445040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:00.374 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.374 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cee00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cb200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cb200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cb200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cb200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.375 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:00.375+0000 7fa7cb200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:00.415 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:00.417+0000 7fb929445040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.147+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 150) ==== 90895+0+0 (secure 0 0 0) 0x7fc44c03da30 con 0x7fc45415fd30 2024-04-17T08:53:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.149+0000 7fc44aa00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fc430074b80 0x7fc430076f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.149+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 --> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc44c0079b0 con 0x7fc430074b80 2024-04-17T08:53:01.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.150+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fc430074b80 0x7fc430076f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.154+0000 7fc452a00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fc430074b80 0x7fc430076f70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc444009590 tx=0x7fc444009190 comp rx=0 tx=0).ready entity=mgr.8600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.326+0000 7fc44aa00640 1 -- 172.21.15.67:0/519471650 <== mgr.8600 v2:172.21.15.187:6832/1608360619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc44c0079b0 con 0x7fc430074b80 2024-04-17T08:53:01.325 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path snap_test_dir1 2024-04-17T08:53:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fc430074b80 msgr2=0x7fc430076f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453e00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fc430074b80 0x7fc430076f70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc444009590 tx=0x7fc444009190 comp rx=0 tx=0).stop 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 msgr2=0x7fc4540b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453e00640 1 --2- 172.21.15.67:0/519471650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45415fd30 0x7fc4540b3fd0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c7d0 tx=0x7fc44c00cc90 comp rx=0 tx=0).stop 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453400640 1 -- 172.21.15.67:0/519471650 reap_dead start 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.329+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 shutdown_connections 2024-04-17T08:53:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.330+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 >> 172.21.15.67:0/519471650 conn(0x7fc4540b2fe0 msgr2=0x7fc4540cdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.330+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 shutdown_connections 2024-04-17T08:53:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.330+0000 7fc453e00640 1 -- 172.21.15.67:0/519471650 wait complete. 2024-04-17T08:53:01.353 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-04-17T08:53:01.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-04-17T08:53:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.628+0000 7fb15f200640 1 Processor -- start 2024-04-17T08:53:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.628+0000 7fb15f200640 1 -- start start 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.629+0000 7fb15f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb15815c670 0x7fb15815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.629+0000 7fb15f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb158072930 0x7fb1580cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15f200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb1580b3e30 0x7fb1580cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb15806de70 con 0x7fb15815c670 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15f200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb15815f850 con 0x7fb1580b3e30 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb15815fb30 con 0x7fb158072930 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb15815c670 0x7fb15815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb15815c670 0x7fb15815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52332/0 (socket says 172.21.15.67:52332) 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15de00640 1 -- 172.21.15.67:0/395994018 learned_addr learned my addr 172.21.15.67:0/395994018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15d400640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb158072930 0x7fb1580cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.630+0000 7fb15e800640 1 --2- 172.21.15.67:0/395994018 >> v2:172.21.15.187:3300/0 conn(0x7fb1580b3e30 0x7fb1580cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.631+0000 7fb15de00640 1 -- 172.21.15.67:0/395994018 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb158072930 msgr2=0x7fb1580cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.631+0000 7fb15de00640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb158072930 0x7fb1580cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.631+0000 7fb15de00640 1 -- 172.21.15.67:0/395994018 >> v2:172.21.15.187:3300/0 conn(0x7fb1580b3e30 msgr2=0x7fb1580cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.631+0000 7fb15de00640 1 --2- 172.21.15.67:0/395994018 >> v2:172.21.15.187:3300/0 conn(0x7fb1580b3e30 0x7fb1580cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.631+0000 7fb15de00640 1 -- 172.21.15.67:0/395994018 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15815fe10 con 0x7fb15815c670 2024-04-17T08:53:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.632+0000 7fb15de00640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15815c670 0x7fb15815ef90 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7fb1580b5b40 tx=0x7fb14c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d6e6e0e647b16c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.632+0000 7fb15ca00640 1 -- 172.21.15.67:0/395994018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c035020 con 0x7fb15815c670 2024-04-17T08:53:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.632+0000 7fb15ca00640 1 -- 172.21.15.67:0/395994018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb14c031870 con 0x7fb15815c670 2024-04-17T08:53:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.632+0000 7fb15ca00640 1 -- 172.21.15.67:0/395994018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c031c40 con 0x7fb15815c670 2024-04-17T08:53:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 -- 172.21.15.67:0/395994018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15815c670 msgr2=0x7fb15815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15815c670 0x7fb15815ef90 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7fb1580b5b40 tx=0x7fb14c02b910 comp rx=0 tx=0).stop 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 -- 172.21.15.67:0/395994018 shutdown_connections 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 --2- 172.21.15.67:0/395994018 >> v2:172.21.15.187:3300/0 conn(0x7fb1580b3e30 0x7fb1580cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb158072930 0x7fb1580cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 --2- 172.21.15.67:0/395994018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb15815c670 0x7fb15815ef90 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 -- 172.21.15.67:0/395994018 >> 172.21.15.67:0/395994018 conn(0x7fb15806b8c0 msgr2=0x7fb15806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 -- 172.21.15.67:0/395994018 shutdown_connections 2024-04-17T08:53:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.633+0000 7fb15f200640 1 -- 172.21.15.67:0/395994018 wait complete. 2024-04-17T08:53:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.635+0000 7fb15f200640 1 Processor -- start 2024-04-17T08:53:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.635+0000 7fb15f200640 1 -- start start 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1580b3e30 0x7fb15815adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb15815c670 0x7fb15815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1580ceec0 con 0x7fb158072930 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb15815f4d0 con 0x7fb1580b3e30 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb1580cf040 con 0x7fb15815c670 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52336/0 (socket says 172.21.15.67:52336) 2024-04-17T08:53:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.636+0000 7fb15de00640 1 -- 172.21.15.67:0/3871033561 learned_addr learned my addr 172.21.15.67:0/3871033561 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 -- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb15815c670 msgr2=0x7fb15815b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb15815c670 0x7fb15815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 -- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1580b3e30 msgr2=0x7fb15815adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1580b3e30 0x7fb15815adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 -- 172.21.15.67:0/3871033561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15814a180 con 0x7fb158072930 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb15de00640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fb14c02b950 tx=0x7fb14c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c040070 con 0x7fb158072930 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb14c030d00 con 0x7fb158072930 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.637+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c035040 con 0x7fb158072930 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.638+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15815b920 con 0x7fb158072930 2024-04-17T08:53:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.638+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb158147330 con 0x7fb158072930 2024-04-17T08:53:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.639+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb158068a10 con 0x7fb158072930 2024-04-17T08:53:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.640+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 150) ==== 90895+0+0 (secure 0 0 0) 0x7fb14c038020 con 0x7fb158072930 2024-04-17T08:53:01.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.641+0000 7fb152000640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 0x7fb13c06ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:01.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.641+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fb14c0ad790 con 0x7fb158072930 2024-04-17T08:53:01.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.642+0000 7fb15d400640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 0x7fb13c06ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.643+0000 7fb15d400640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 0x7fb13c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1580ccd30 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.8600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.646+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb14c07bd00 con 0x7fb158072930 2024-04-17T08:53:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:01.885+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 --> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb158068610 con 0x7fb13c06c790 2024-04-17T08:53:02.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.051+0000 7fb152000640 1 -- 172.21.15.67:0/3871033561 <== mgr.8600 v2:172.21.15.187:6832/1608360619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7fb158068610 con 0x7fb13c06c790 2024-04-17T08:53:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:Retention added to path snap_test_dir1 2024-04-17T08:53:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 msgr2=0x7fb13c06ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 0x7fb13c06ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1580ccd30 tx=0x7fb148008040 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 msgr2=0x7fb158146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fb14c02b950 tx=0x7fb14c0046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 shutdown_connections 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fb13c06c790 0x7fb13c06ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb15815c670 0x7fb15815b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1580b3e30 0x7fb15815adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 --2- 172.21.15.67:0/3871033561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb158072930 0x7fb158146ba0 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 >> 172.21.15.67:0/3871033561 conn(0x7fb15806b8c0 msgr2=0x7fb15806f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 shutdown_connections 2024-04-17T08:53:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.055+0000 7fb15f200640 1 -- 172.21.15.67:0/3871033561 wait complete. 2024-04-17T08:53:02.079 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-04-17T08:53:02.084 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-04-17-08_54_00 in ~58s... 2024-04-17T08:53:02.084 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-04-17T08:53:02.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-04-17T08:53:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.326+0000 7f977be00640 1 Processor -- start 2024-04-17T08:53:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.327+0000 7f977be00640 1 -- start start 2024-04-17T08:53:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f977be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f977c0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f977be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f977c0cfab0 0x7f977c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f977be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977c0d00b0 con 0x7f977c0d3210 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f977be00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f977c0d3670 con 0x7f977c0cfab0 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f977be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f977c0d0c30 con 0x7f977c15ecb0 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f9772000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f977c0d3210 0x7f977c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60500/0 (socket says 172.21.15.67:60500) 2024-04-17T08:53:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.328+0000 7f9772000640 1 -- 172.21.15.67:0/720703138 learned_addr learned my addr 172.21.15.67:0/720703138 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977aa00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f977c0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977b400640 1 --2- 172.21.15.67:0/720703138 >> v2:172.21.15.187:3300/0 conn(0x7f977c0cfab0 0x7f977c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279182521 0 0) 0x7f977c0d00b0 con 0x7f977c0d3210 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9760003520 con 0x7f977c0d3210 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2558805795 0 0) 0x7f9760003520 con 0x7f977c0d3210 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 msgr2=0x7f977c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f977c0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 >> v2:172.21.15.187:3300/0 conn(0x7f977c0cfab0 msgr2=0x7f977c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 --2- 172.21.15.67:0/720703138 >> v2:172.21.15.187:3300/0 conn(0x7f977c0cfab0 0x7f977c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c0d0390 con 0x7f977c0d3210 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9764001690 con 0x7f977c0d3210 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 >> v1:172.21.15.67:6789/0 conn(0x7f977c0d3210 legacy=0x7f977c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.329+0000 7f977a000640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f9760004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977a000640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f977c0d0c30 con 0x7f97600039b0 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f976000b9c0 con 0x7f9760004d40 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f976000bb70 con 0x7f977c15ecb0 2024-04-17T08:53:02.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f9772000640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f9760004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977b400640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 -- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 msgr2=0x7f9760004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c15ecb0 0x7f9760004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 -- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9760004d40 msgr2=0x7f97600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.330+0000 7f977aa00640 1 -- 172.21.15.67:0/720703138 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9760003520 con 0x7f97600039b0 2024-04-17T08:53:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977aa00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f976c009ac0 tx=0x7f9760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08a61d9d8169fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c039070 con 0x7f97600039b0 2024-04-17T08:53:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f976c02c6b0 con 0x7f97600039b0 2024-04-17T08:53:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977a000640 1 -- 172.21.15.67:0/720703138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c009690 con 0x7f97600039b0 2024-04-17T08:53:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977be00640 1 -- 172.21.15.67:0/720703138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600039b0 msgr2=0x7f9760003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977be00640 1 --2- 172.21.15.67:0/720703138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f976c009ac0 tx=0x7f9760005830 comp rx=0 tx=0).stop 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.332+0000 7f977b400640 1 -- 172.21.15.67:0/720703138 reap_dead start 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.333+0000 7f977be00640 1 -- 172.21.15.67:0/720703138 shutdown_connections 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.333+0000 7f977be00640 1 -- 172.21.15.67:0/720703138 >> 172.21.15.67:0/720703138 conn(0x7f977c08bff0 msgr2=0x7f977c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.333+0000 7f977be00640 1 -- 172.21.15.67:0/720703138 shutdown_connections 2024-04-17T08:53:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.333+0000 7f977be00640 1 -- 172.21.15.67:0/720703138 wait complete. 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.335+0000 7f977be00640 1 Processor -- start 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.335+0000 7f977be00640 1 -- start start 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97600039b0 0x7f977c150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c0b5600 0x7f977c0b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f977c15f090 con 0x7f977c0b5180 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f977c0d0950 con 0x7f97600039b0 2024-04-17T08:53:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977be00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f977c15f210 con 0x7f977c0b5600 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f9772000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f9772000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52354/0 (socket says 172.21.15.67:52354) 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f9772000640 1 -- 172.21.15.67:0/990809584 learned_addr learned my addr 172.21.15.67:0/990809584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977aa00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97600039b0 0x7f977c150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.336+0000 7f977b400640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c0b5600 0x7f977c0b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 -- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c0b5600 msgr2=0x7f977c0b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c0b5600 0x7f977c0b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 -- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97600039b0 msgr2=0x7f977c150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97600039b0 0x7f977c150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 -- 172.21.15.67:0/990809584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c0ccf50 con 0x7f977c0b5180 2024-04-17T08:53:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9772000640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f976400e680 tx=0x7f976400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976400c6c0 con 0x7f977c0b5180 2024-04-17T08:53:02.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764018070 con 0x7f977c0b5180 2024-04-17T08:53:02.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976400cac0 con 0x7f977c0b5180 2024-04-17T08:53:02.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977c0b87f0 con 0x7f977c0b5180 2024-04-17T08:53:02.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.337+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977c0b8db0 con 0x7f977c0b5180 2024-04-17T08:53:02.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.339+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977c068a10 con 0x7f977c0b5180 2024-04-17T08:53:02.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.340+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 151) ==== 91160+0+0 (secure 0 0 0) 0x7f976400cc60 con 0x7f977c0b5180 2024-04-17T08:53:02.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.341+0000 7f9773400640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 0x7f975806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.341+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7f976408bda0 con 0x7f977c0b5180 2024-04-17T08:53:02.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.342+0000 7f977aa00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 0x7f975806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.343+0000 7f977aa00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 0x7f975806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f976c036000 tx=0x7f976c030000 comp rx=0 tx=0).ready entity=mgr.8600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.346+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9764059eb0 con 0x7f977c0b5180 2024-04-17T08:53:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.576+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 --> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f977c0b5a70 con 0x7f975806ca60 2024-04-17T08:53:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.593+0000 7f9773400640 1 -- 172.21.15.67:0/990809584 <== mgr.8600 v2:172.21.15.187:6832/1608360619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7f977c0b5a70 con 0x7f975806ca60 2024-04-17T08:53:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:02.591 INFO:teuthology.orchestra.run.smithi067.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-17T08:53:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 msgr2=0x7f975806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 0x7f975806ef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f976c036000 tx=0x7f976c030000 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 msgr2=0x7f977c0cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f976400e680 tx=0x7f976400eb40 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 shutdown_connections 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f975806ca60 0x7f975806ef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f977c0b5600 0x7f977c0b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f977c0b5180 0x7f977c0cc490 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 --2- 172.21.15.67:0/990809584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97600039b0 0x7f977c150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 >> 172.21.15.67:0/990809584 conn(0x7f977c08bff0 msgr2=0x7f977c06ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 shutdown_connections 2024-04-17T08:53:02.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.596+0000 7f977be00640 1 -- 172.21.15.67:0/990809584 wait complete. 2024-04-17T08:53:02.616 DEBUG:tasks.cephfs.test_snap_schedules:res={"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-04-17T08:53:02.617 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-04-17T08:53:02.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-04-17T08:53:02.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.847+0000 7f49a5a00640 1 Processor -- start 2024-04-17T08:53:02.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.847+0000 7f49a5a00640 1 -- start start 2024-04-17T08:53:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f49a5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 0x7f49a00cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f49a5a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f49a00cf8e0 0x7f49a00b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f49a5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a00d00b0 con 0x7f49a015ecb0 2024-04-17T08:53:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f49a5a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f49a00d0c30 con 0x7f49a00cf8e0 2024-04-17T08:53:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f49a5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49a00d0390 con 0x7f49a00d33e0 2024-04-17T08:53:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f499fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49a015ecb0 0x7f49a00d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60502/0 (socket says 172.21.15.67:60502) 2024-04-17T08:53:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.848+0000 7f499fe00640 1 -- 172.21.15.67:0/2001251228 learned_addr learned my addr 172.21.15.67:0/2001251228 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594332531 0 0) 0x7f49a00d00b0 con 0x7f49a015ecb0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499f400640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 0x7f49a00cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4984003520 con 0x7f49a015ecb0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f49a5000640 1 --2- 172.21.15.67:0/2001251228 >> v2:172.21.15.187:3300/0 conn(0x7f49a00cf8e0 0x7f49a00b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2772712891 0 0) 0x7f4984003520 con 0x7f49a015ecb0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 msgr2=0x7f49a00cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 0x7f49a00cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 >> v2:172.21.15.187:3300/0 conn(0x7f49a00cf8e0 msgr2=0x7f49a00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 --2- 172.21.15.67:0/2001251228 >> v2:172.21.15.187:3300/0 conn(0x7f49a00cf8e0 0x7f49a00b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a00d0670 con 0x7f49a015ecb0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4990001670 con 0x7f49a015ecb0 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 >> v1:172.21.15.67:6789/0 conn(0x7f49a015ecb0 legacy=0x7f49a00d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49840039b0 0x7f4984003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 0x7f4984004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.849+0000 7f499ea00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4984004d40 0x7f49840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49a00d0c30 con 0x7f49840039b0 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49a00d0390 con 0x7f4984004d40 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f498400bac0 con 0x7f49a00d33e0 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49840039b0 0x7f4984003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f49a5000640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4984004d40 0x7f49840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 -- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 msgr2=0x7f4984004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49a00d33e0 0x7f4984004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 -- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4984004d40 msgr2=0x7f49840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4984004d40 0x7f49840051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.850+0000 7f499fe00640 1 -- 172.21.15.67:0/2001251228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4984003520 con 0x7f49840039b0 2024-04-17T08:53:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.851+0000 7f499fe00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49840039b0 0x7f4984003dd0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f499000c270 tx=0x7f4984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a76007e085cfb37 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990007d30 con 0x7f49840039b0 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4990030070 con 0x7f49840039b0 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f499ea00640 1 -- 172.21.15.67:0/2001251228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499002cd60 con 0x7f49840039b0 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 -- 172.21.15.67:0/2001251228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49840039b0 msgr2=0x7f4984003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 --2- 172.21.15.67:0/2001251228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49840039b0 0x7f4984003dd0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f499000c270 tx=0x7f4984005830 comp rx=0 tx=0).stop 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5000640 1 -- 172.21.15.67:0/2001251228 reap_dead start 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 -- 172.21.15.67:0/2001251228 shutdown_connections 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 -- 172.21.15.67:0/2001251228 >> 172.21.15.67:0/2001251228 conn(0x7f49a008bff0 msgr2=0x7f49a00d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 -- 172.21.15.67:0/2001251228 shutdown_connections 2024-04-17T08:53:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.852+0000 7f49a5a00640 1 -- 172.21.15.67:0/2001251228 wait complete. 2024-04-17T08:53:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.854+0000 7f49a5a00640 1 Processor -- start 2024-04-17T08:53:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.854+0000 7f49a5a00640 1 -- start start 2024-04-17T08:53:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840039b0 0x7f49a00b9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49a00b8aa0 0x7f49a00cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49a015f090 con 0x7f49a00b8f00 2024-04-17T08:53:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f49a00cfd30 con 0x7f49a00b8aa0 2024-04-17T08:53:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.855+0000 7f49a5a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f49a015f210 con 0x7f49840039b0 2024-04-17T08:53:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52372/0 (socket says 172.21.15.67:52372) 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 -- 172.21.15.67:0/1959939119 learned_addr learned my addr 172.21.15.67:0/1959939119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f499fe00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840039b0 0x7f49a00b9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f499f400640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49a00b8aa0 0x7f49a00cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 -- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840039b0 msgr2=0x7f49a00b9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840039b0 0x7f49a00b9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 -- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49a00b8aa0 msgr2=0x7f49a00cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49a00b8aa0 0x7f49a00cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.856+0000 7f49a5000640 1 -- 172.21.15.67:0/1959939119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a00cd280 con 0x7f49a00b8f00 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f49a5000640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f49a00c81e0 tx=0x7f4998009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998019070 con 0x7f49a00b8f00 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4998007da0 con 0x7f49a00b8f00 2024-04-17T08:53:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998014640 con 0x7f49a00b8f00 2024-04-17T08:53:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a00cd560 con 0x7f49a00b8f00 2024-04-17T08:53:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.857+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a00b7710 con 0x7f49a00b8f00 2024-04-17T08:53:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.858+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a00b7910 con 0x7f49a00b8f00 2024-04-17T08:53:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.859+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 151) ==== 91160+0+0 (secure 0 0 0) 0x7f499801c070 con 0x7f49a00b8f00 2024-04-17T08:53:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.860+0000 7f4997e00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 0x7f497c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:02.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.860+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7f499808bb80 con 0x7f49a00b8f00 2024-04-17T08:53:02.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.861+0000 7f499fe00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 0x7f497c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:02.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.862+0000 7f499fe00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 0x7f497c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4990034000 tx=0x7f4990007000 comp rx=0 tx=0).ready entity=mgr.8600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:02.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:02.865+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f49a00b7910 con 0x7f49a00b8f00 2024-04-17T08:53:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.095+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 --> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f49a0068a10 con 0x7f497c06ca60 2024-04-17T08:53:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.288+0000 7f4997e00640 1 -- 172.21.15.67:0/1959939119 <== mgr.8600 v2:172.21.15.187:6832/1608360619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f49a0068a10 con 0x7f497c06ca60 2024-04-17T08:53:03.287 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path snap_test_dir1 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 msgr2=0x7f497c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 0x7f497c06ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4990034000 tx=0x7f4990007000 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 msgr2=0x7f49a00ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f49a00c81e0 tx=0x7f4998009da0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 shutdown_connections 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f497c06ca60 0x7f497c06ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49a00b8f00 0x7f49a00ca220 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49a00b8aa0 0x7f49a00cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 --2- 172.21.15.67:0/1959939119 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f49840039b0 0x7f49a00b9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 >> 172.21.15.67:0/1959939119 conn(0x7f49a008bff0 msgr2=0x7f49a006d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 shutdown_connections 2024-04-17T08:53:03.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.292+0000 7f49a5a00640 1 -- 172.21.15.67:0/1959939119 wait complete. 2024-04-17T08:53:03.313 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-04-17T08:53:03.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:03.313 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-04-17T08:53:03.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:03.370 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-04-17T08:53:03.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:53:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.652+0000 7f1077e00640 1 Processor -- start 2024-04-17T08:53:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.652+0000 7f1077e00640 1 -- start start 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10780ce480 0x7f107806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f107806db30 0x7f107809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107809afa0 0x7f1078162850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10780d1650 con 0x7f107809afa0 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f10780d07a0 con 0x7f107806db30 2024-04-17T08:53:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.653+0000 7f1077e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10780d1ae0 con 0x7f10780ce480 2024-04-17T08:53:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107809afa0 0x7f1078162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107809afa0 0x7f1078162850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47240/0 (socket says 172.21.15.67:47240) 2024-04-17T08:53:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 -- 172.21.15.67:0/438003171 learned_addr learned my addr 172.21.15.67:0/438003171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 -- 172.21.15.67:0/438003171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10780ce480 msgr2=0x7f107806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 --2- 172.21.15.67:0/438003171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10780ce480 0x7f107806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 -- 172.21.15.67:0/438003171 >> v2:172.21.15.187:3300/0 conn(0x7f107806db30 msgr2=0x7f107809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1076000640 1 --2- 172.21.15.67:0/438003171 >> v2:172.21.15.187:3300/0 conn(0x7f107806db30 0x7f107809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 --2- 172.21.15.67:0/438003171 >> v2:172.21.15.187:3300/0 conn(0x7f107806db30 0x7f107809aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.654+0000 7f1077400640 1 -- 172.21.15.67:0/438003171 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107809ba50 con 0x7f107809afa0 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1077400640 1 --2- 172.21.15.67:0/438003171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107809afa0 0x7f1078162850 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f107806c580 tx=0x7f107002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f5754dddfc42c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1075600640 1 -- 172.21.15.67:0/438003171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070035020 con 0x7f107809afa0 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1075600640 1 -- 172.21.15.67:0/438003171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1070031870 con 0x7f107809afa0 2024-04-17T08:53:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1075600640 1 -- 172.21.15.67:0/438003171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070031c40 con 0x7f107809afa0 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1077e00640 1 -- 172.21.15.67:0/438003171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107809afa0 msgr2=0x7f1078162850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.656+0000 7f1077e00640 1 --2- 172.21.15.67:0/438003171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107809afa0 0x7f1078162850 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f107806c580 tx=0x7f107002b910 comp rx=0 tx=0).stop 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 -- 172.21.15.67:0/438003171 shutdown_connections 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 --2- 172.21.15.67:0/438003171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107809afa0 0x7f1078162850 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 --2- 172.21.15.67:0/438003171 >> v2:172.21.15.187:3300/0 conn(0x7f107806db30 0x7f107809aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 --2- 172.21.15.67:0/438003171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10780ce480 0x7f107806d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 -- 172.21.15.67:0/438003171 >> 172.21.15.67:0/438003171 conn(0x7f10780b4050 msgr2=0x7f10780b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 -- 172.21.15.67:0/438003171 shutdown_connections 2024-04-17T08:53:03.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.657+0000 7f1077e00640 1 -- 172.21.15.67:0/438003171 wait complete. 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.658+0000 7f1077e00640 1 Processor -- start 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.659+0000 7f1077e00640 1 -- start start 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107806db30 0x7f10780701e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107809afa0 0x7f107809e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10780d1000 con 0x7f10780ce480 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10780d0d20 con 0x7f107809afa0 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10780d12e0 con 0x7f107806db30 2024-04-17T08:53:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47242/0 (socket says 172.21.15.67:47242) 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077400640 1 -- 172.21.15.67:0/1266688962 learned_addr learned my addr 172.21.15.67:0/1266688962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1076a00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107806db30 0x7f10780701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1076000640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107809afa0 0x7f107809e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077400640 1 -- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107806db30 msgr2=0x7f10780701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.660+0000 7f1077400640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107806db30 0x7f10780701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077400640 1 -- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107809afa0 msgr2=0x7f107809e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077400640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107809afa0 0x7f107809e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077400640 1 -- 172.21.15.67:0/1266688962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107809d1b0 con 0x7f10780ce480 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077400640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f1070009960 tx=0x7f10700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070035040 con 0x7f10780ce480 2024-04-17T08:53:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1070045480 con 0x7f10780ce480 2024-04-17T08:53:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070045870 con 0x7f10780ce480 2024-04-17T08:53:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107809d490 con 0x7f10780ce480 2024-04-17T08:53:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.661+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f107809da50 con 0x7f10780ce480 2024-04-17T08:53:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.662+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10780cfdf0 con 0x7f10780ce480 2024-04-17T08:53:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.664+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 151) ==== 91160+0+0 (secure 0 0 0) 0x7f1070038020 con 0x7f10780ce480 2024-04-17T08:53:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.664+0000 7f106ea00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 0x7f105406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.664+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7f10700ae430 con 0x7f10780ce480 2024-04-17T08:53:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.665+0000 7f1076a00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 0x7f105406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:03.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.666+0000 7f1076a00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 0x7f105406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1068009890 tx=0x7f1068008040 comp rx=0 tx=0).ready entity=mgr.8600 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:03.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.669+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f10780cfdf0 con 0x7f10780ce480 2024-04-17T08:53:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:03.926+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f10780755d0 con 0x7f10780ce480 2024-04-17T08:53:04.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.209+0000 7f106ea00640 1 -- 172.21.15.67:0/1266688962 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v152) ==== 93+0+0 (secure 0 0 0) 0x7f1070039e60 con 0x7f10780ce480 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.212+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 msgr2=0x7f105406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.212+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 0x7f105406ef20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1068009890 tx=0x7f1068008040 comp rx=0 tx=0).stop 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.212+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 msgr2=0x7f107809eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.212+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f1070009960 tx=0x7f10700046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 shutdown_connections 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f105406ca60 0x7f105406ef20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10780ce480 0x7f107809eec0 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107809afa0 0x7f107809e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 --2- 172.21.15.67:0/1266688962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f107806db30 0x7f10780701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 >> 172.21.15.67:0/1266688962 conn(0x7f10780b4050 msgr2=0x7f10780cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.213+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 shutdown_connections 2024-04-17T08:53:04.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.214+0000 7f1077e00640 1 -- 172.21.15.67:0/1266688962 wait complete. 2024-04-17T08:53:04.238 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:53:04.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:53:04.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:53:04.450 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:53:04.450 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:53:04.451 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:04.451 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:04.501 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:04.502 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:04.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:04.775 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:04.775+0000 7f0194e49040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:53:04.784 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:04.785+0000 7f751fc49040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:53:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.797+0000 7fdbeee00640 1 Processor -- start 2024-04-17T08:53:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.797+0000 7fdbeee00640 1 -- start start 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbeee00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 0x7fdbe80b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbeee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbe80b6b30 0x7fdbe80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbeee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe80d2d30 con 0x7fdbe80cf910 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbeee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe80d49c0 con 0x7fdbe80b6b30 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbeee00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe80d3660 con 0x7fdbe815ec60 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbed000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbe80b6b30 0x7fdbe80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbee400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdbe80cf910 0x7fdbe806cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42518/0 (socket says 172.21.15.67:42518) 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.799+0000 7fdbee400640 1 -- 172.21.15.67:0/2473547923 learned_addr learned my addr 172.21.15.67:0/2473547923 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbeda00640 1 --2- 172.21.15.67:0/2473547923 >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 0x7fdbe80b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbed000640 1 -- 172.21.15.67:0/2473547923 >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 msgr2=0x7fdbe80b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbed000640 1 --2- 172.21.15.67:0/2473547923 >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 0x7fdbe80b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbed000640 1 -- 172.21.15.67:0/2473547923 >> v1:172.21.15.67:6789/0 conn(0x7fdbe80cf910 legacy=0x7fdbe806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbed000640 1 -- 172.21.15.67:0/2473547923 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe80d3010 con 0x7fdbe80b6b30 2024-04-17T08:53:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.800+0000 7fdbeda00640 1 --2- 172.21.15.67:0/2473547923 >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 0x7fdbe80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.801+0000 7fdbed000640 1 --2- 172.21.15.67:0/2473547923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe80cf3d0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80d1550 tx=0x7fdbd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0627a76a158fb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbe7e00640 1 -- 172.21.15.67:0/2473547923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4035020 con 0x7fdbe80b6b30 2024-04-17T08:53:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbe7e00640 1 -- 172.21.15.67:0/2473547923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd4031830 con 0x7fdbe80b6b30 2024-04-17T08:53:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbe7e00640 1 -- 172.21.15.67:0/2473547923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4031c00 con 0x7fdbe80b6b30 2024-04-17T08:53:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 -- 172.21.15.67:0/2473547923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 msgr2=0x7fdbe80cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 --2- 172.21.15.67:0/2473547923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe80cf3d0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80d1550 tx=0x7fdbd402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 -- 172.21.15.67:0/2473547923 shutdown_connections 2024-04-17T08:53:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 --2- 172.21.15.67:0/2473547923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe80cf3d0 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 --2- 172.21.15.67:0/2473547923 >> v2:172.21.15.187:3300/0 conn(0x7fdbe815ec60 0x7fdbe80b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.802+0000 7fdbeee00640 1 -- 172.21.15.67:0/2473547923 >> 172.21.15.67:0/2473547923 conn(0x7fdbe80d15f0 msgr2=0x7fdbe80d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.803+0000 7fdbeee00640 1 -- 172.21.15.67:0/2473547923 shutdown_connections 2024-04-17T08:53:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.803+0000 7fdbeee00640 1 -- 172.21.15.67:0/2473547923 wait complete. 2024-04-17T08:53:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.804+0000 7fdbeee00640 1 Processor -- start 2024-04-17T08:53:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.804+0000 7fdbeee00640 1 -- start start 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbe80cf910 0x7fdbe814a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 0x7fdbe814af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe80cfda0 con 0x7fdbe80b6b30 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe80b6f70 con 0x7fdbe815ec60 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.805+0000 7fdbeee00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe815f040 con 0x7fdbe80cf910 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47292/0 (socket says 172.21.15.67:47292) 2024-04-17T08:53:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 -- 172.21.15.67:0/1599436655 learned_addr learned my addr 172.21.15.67:0/1599436655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbee400640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 0x7fdbe814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbed000640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbe80cf910 0x7fdbe814a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbe80cf910 msgr2=0x7fdbe814a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbe80cf910 0x7fdbe814a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 msgr2=0x7fdbe814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 0x7fdbe814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 -- 172.21.15.67:0/1599436655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe8157810 con 0x7fdbe80b6b30 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbee400640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 0x7fdbe814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.806+0000 7fdbeda00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00d800 tx=0x7fdbdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.807+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc004030 con 0x7fdbe80b6b30 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.807+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbdc0041d0 con 0x7fdbe80b6b30 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.807+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc009640 con 0x7fdbe80b6b30 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.807+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe8157af0 con 0x7fdbe80b6b30 2024-04-17T08:53:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.807+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe80cc340 con 0x7fdbe80b6b30 2024-04-17T08:53:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.808+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbe8068a10 con 0x7fdbe80b6b30 2024-04-17T08:53:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.809+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7fdbdc0097e0 con 0x7fdbe80b6b30 2024-04-17T08:53:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.810+0000 7fdbe5600640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 0x7fdbc806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.810+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fdbdc08c410 con 0x7fdbe80b6b30 2024-04-17T08:53:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.810+0000 7fdbed000640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 msgr2=0x7fdbc806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:04.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.810+0000 7fdbed000640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 0x7fdbc806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:04.814+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdbdc05a320 con 0x7fdbe80b6b30 2024-04-17T08:53:04.835 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:04.836+0000 7f0194e49040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:53:04.845 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:04.846+0000 7f751fc49040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:53:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.011+0000 7fdbed000640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 msgr2=0x7fdbc806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.011+0000 7fdbed000640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 0x7fdbc806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:05.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.018+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbe815b8c0 con 0x7fdbe80b6b30 2024-04-17T08:53:05.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.019+0000 7fdbe5600640 1 -- 172.21.15.67:0/1599436655 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) ==== 74+0+17319 (secure 0 0 0) 0x7fdbdc059c20 con 0x7fdbe80b6b30 2024-04-17T08:53:05.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:05.019 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":273,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:00.117836+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3094572004":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2893316565":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2486101166":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3800011905":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3721653371":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/227074939":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/456496732":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6832/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6833/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3973592351":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:05.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 msgr2=0x7fdbc806ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:05.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 0x7fdbc806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 msgr2=0x7fdbe8154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00d800 tx=0x7fdbdc00dcc0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 shutdown_connections 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fdbc806caa0 0x7fdbc806ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbe815ec60 0x7fdbe814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdbe80cf910 0x7fdbe814a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 --2- 172.21.15.67:0/1599436655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbe80b6b30 0x7fdbe8154b50 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 >> 172.21.15.67:0/1599436655 conn(0x7fdbe80d15f0 msgr2=0x7fdbe80d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 shutdown_connections 2024-04-17T08:53:05.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.022+0000 7fdbeee00640 1 -- 172.21.15.67:0/1599436655 wait complete. 2024-04-17T08:53:05.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:52:43.162800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:52:44.240860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:53:05.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:53:05.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.249+0000 7f951fa00640 1 Processor -- start 2024-04-17T08:53:05.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.249+0000 7f951fa00640 1 -- start start 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f951fa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95180b0860 0x7f95180b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f951fa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9518072f70 con 0x7f951806bfd0 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f951fa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9518162630 con 0x7f9518162250 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f951fa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f95180b52e0 con 0x7f95180b0860 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f9517e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f951806bfd0 0x7f95180b4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42522/0 (socket says 172.21.15.67:42522) 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.251+0000 7f9517e00640 1 -- 172.21.15.67:0/1806143015 learned_addr learned my addr 172.21.15.67:0/1806143015 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:05.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f951da00640 1 --2- 172.21.15.67:0/1806143015 >> v2:172.21.15.187:3300/0 conn(0x7f95180b0860 0x7f95180b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841068205 0 0) 0x7f9518072f70 con 0x7f951806bfd0 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9500003520 con 0x7f951806bfd0 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2217957077 0 0) 0x7f9500003520 con 0x7f951806bfd0 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 >> v2:172.21.15.187:3300/0 conn(0x7f95180b0860 msgr2=0x7f95180b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 --2- 172.21.15.67:0/1806143015 >> v2:172.21.15.187:3300/0 conn(0x7f95180b0860 0x7f95180b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 >> v1:172.21.15.187:6789/0 conn(0x7f9518162250 legacy=0x7f951806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9518071570 con 0x7f951806bfd0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f9504001670 con 0x7f951806bfd0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 >> v1:172.21.15.67:6789/0 conn(0x7f951806bfd0 legacy=0x7f95180b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.252+0000 7f9517400640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9500003a00 0x7f9500003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517400640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f9500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517400640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95000052e0 0x7f950000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f95180b52e0 con 0x7f9500004800 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f950000c1e0 con 0x7f95000052e0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f950000c3e0 con 0x7f9500003a00 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f9500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f951da00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95000052e0 0x7f950000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f951d000640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9500003a00 0x7f9500003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 -- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9500003a00 msgr2=0x7f9500003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9500003a00 0x7f9500003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 -- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95000052e0 msgr2=0x7f950000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95000052e0 0x7f950000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f9517e00640 1 -- 172.21.15.67:0/1806143015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9500003520 con 0x7f9500004800 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.253+0000 7f951da00640 1 -- 172.21.15.67:0/1806143015 reap_dead start 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.254+0000 7f9517e00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f9500004c80 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f950400c0e0 tx=0x7f9500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1be1ae3a8bf5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.255+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504007c20 con 0x7f9500004800 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.255+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9504030070 con 0x7f9500004800 2024-04-17T08:53:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.255+0000 7f9517400640 1 -- 172.21.15.67:0/1806143015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504005db0 con 0x7f9500004800 2024-04-17T08:53:05.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.255+0000 7f951fa00640 1 -- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 msgr2=0x7f9500004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.255+0000 7f951fa00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f9500004c80 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f950400c0e0 tx=0x7f9500005830 comp rx=0 tx=0).stop 2024-04-17T08:53:05.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.256+0000 7f951fa00640 1 -- 172.21.15.67:0/1806143015 shutdown_connections 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.256+0000 7f951fa00640 1 --2- 172.21.15.67:0/1806143015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f9500004c80 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.256+0000 7f951fa00640 1 -- 172.21.15.67:0/1806143015 >> 172.21.15.67:0/1806143015 conn(0x7f95180b3390 msgr2=0x7f95180b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.256+0000 7f951fa00640 1 -- 172.21.15.67:0/1806143015 shutdown_connections 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.256+0000 7f951fa00640 1 -- 172.21.15.67:0/1806143015 wait complete. 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.257+0000 7f951fa00640 1 Processor -- start 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 -- start start 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9518147f30 0x7f9518148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 0x7f95181614c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9518073230 con 0x7f9500004800 2024-04-17T08:53:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f95180735a0 con 0x7f9518147f30 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.258+0000 7f951fa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95180b0f40 con 0x7f9518161040 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47314/0 (socket says 172.21.15.67:47314) 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 -- 172.21.15.67:0/4249732347 learned_addr learned my addr 172.21.15.67:0/4249732347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f9517e00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9518147f30 0x7f9518148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951da00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 0x7f95181614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 msgr2=0x7f95181614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 0x7f95181614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9518147f30 msgr2=0x7f9518148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9518147f30 0x7f9518148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 -- 172.21.15.67:0/4249732347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95181489f0 con 0x7f9500004800 2024-04-17T08:53:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951da00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 0x7f95181614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.259+0000 7f951d000640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f950c00e4b0 tx=0x7f950c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.260+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c0052e0 con 0x7f9500004800 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.260+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950c018070 con 0x7f9500004800 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.260+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c003810 con 0x7f9500004800 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.260+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9518148cd0 con 0x7f9500004800 2024-04-17T08:53:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.260+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9518146390 con 0x7f9500004800 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.261+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9518145ea0 con 0x7f9500004800 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.262+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7f950c0039b0 con 0x7f9500004800 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.263+0000 7f9514c00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 0x7f94f806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.263+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7f950c08bcb0 con 0x7f9500004800 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.263+0000 7f9517e00640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 msgr2=0x7f94f806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.263+0000 7f9517e00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 0x7f94f806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.267+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f950c059c50 con 0x7f9500004800 2024-04-17T08:53:05.300 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.300+0000 7f751fc49040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:53:05.308 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.309+0000 7f0194e49040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:53:05.352 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.353+0000 7f751fc49040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:53:05.364 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.365+0000 7f0194e49040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:53:05.403 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.404+0000 7f751fc49040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:53:05.418 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.419+0000 7f0194e49040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:53:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.464+0000 7f9517e00640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 msgr2=0x7f94f806ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.464+0000 7f9517e00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 0x7f94f806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:05.465 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.465+0000 7f751fc49040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:53:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.474+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9518068b90 con 0x7f9500004800 2024-04-17T08:53:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.475+0000 7f9514c00640 1 -- 172.21.15.67:0/4249732347 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v273) ==== 93+0+10 (secure 0 0 0) 0x7f950c059550 con 0x7f9500004800 2024-04-17T08:53:05.474 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 msgr2=0x7f94f806ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 0x7f94f806ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 msgr2=0x7f95180ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f950c00e4b0 tx=0x7f950c00e970 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 shutdown_connections 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f94f806caa0 0x7f94f806ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9518161040 0x7f95181614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9518147f30 0x7f9518148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 --2- 172.21.15.67:0/4249732347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9500004800 0x7f95180ce610 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.477+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 >> 172.21.15.67:0/4249732347 conn(0x7f95180b3390 msgr2=0x7f95180b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.478+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 shutdown_connections 2024-04-17T08:53:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.478+0000 7f951fa00640 1 -- 172.21.15.67:0/4249732347 wait complete. 2024-04-17T08:53:05.483 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.484+0000 7f0194e49040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:53:05.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:53:05.514 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.514+0000 7f751fc49040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:53:05.538 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.539+0000 7f0194e49040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:53:05.642 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.642+0000 7f751fc49040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:53:05.675 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.676+0000 7f0194e49040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:53:05.704 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.705+0000 7f751fc49040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:05.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.707+0000 7f9590a00640 1 Processor -- start 2024-04-17T08:53:05.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.707+0000 7f9590a00640 1 -- start start 2024-04-17T08:53:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f9590a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f958c0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f9590a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0b51d0 con 0x7f958c0d4510 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f9590a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0b54b0 con 0x7f958c15e9b0 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f9590a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0b5790 con 0x7f958c0b46f0 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f958b200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f958c0d4510 0x7f958c0d4960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42536/0 (socket says 172.21.15.67:42536) 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.709+0000 7f958b200640 1 -- 172.21.15.67:0/673801622 learned_addr learned my addr 172.21.15.67:0/673801622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589e00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f958c0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960627663 0 0) 0x7f958c0b51d0 con 0x7f958c0d4510 2024-04-17T08:53:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9570003520 con 0x7f958c0d4510 2024-04-17T08:53:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2372696949 0 0) 0x7f9570003520 con 0x7f958c0d4510 2024-04-17T08:53:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 msgr2=0x7f958c0b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f958c0b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 >> v1:172.21.15.187:6789/0 conn(0x7f958c15e9b0 legacy=0x7f958c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0d24c0 con 0x7f958c0d4510 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f95840019f0 con 0x7f958c0d4510 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 >> v1:172.21.15.67:6789/0 conn(0x7f958c0d4510 legacy=0x7f958c0d4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f95700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f9570004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9570004d40 0x7f95700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0b5790 con 0x7f9570003f20 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f957000bad0 con 0x7f9570004d40 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.710+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f957000bcd0 con 0x7f958c0b46f0 2024-04-17T08:53:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f9570004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f958a800640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f95700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f958b200640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9570004d40 0x7f95700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 -- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 msgr2=0x7f95700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c0b46f0 0x7f95700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 -- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9570004d40 msgr2=0x7f95700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9570004d40 0x7f95700051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.711+0000 7f9589e00640 1 -- 172.21.15.67:0/673801622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9570003520 con 0x7f9570003f20 2024-04-17T08:53:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.712+0000 7f9589e00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f9570004800 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f9578009ac0 tx=0x7f9570005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d6f9bf49f4a800 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.712+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578039070 con 0x7f9570003f20 2024-04-17T08:53:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957802c6b0 con 0x7f9570003f20 2024-04-17T08:53:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9589400640 1 -- 172.21.15.67:0/673801622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578009690 con 0x7f9570003f20 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 -- 172.21.15.67:0/673801622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 msgr2=0x7f9570004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 --2- 172.21.15.67:0/673801622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f9570004800 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f9578009ac0 tx=0x7f9570005830 comp rx=0 tx=0).stop 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f958b200640 1 -- 172.21.15.67:0/673801622 reap_dead start 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 -- 172.21.15.67:0/673801622 shutdown_connections 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 -- 172.21.15.67:0/673801622 >> 172.21.15.67:0/673801622 conn(0x7f958c09daa0 msgr2=0x7f958c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 -- 172.21.15.67:0/673801622 shutdown_connections 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.713+0000 7f9590a00640 1 -- 172.21.15.67:0/673801622 wait complete. 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.715+0000 7f9590a00640 1 Processor -- start 2024-04-17T08:53:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.715+0000 7f9590a00640 1 -- start start 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c156bf0 0x7f958c15d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 0x7f958c0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f958c162830 con 0x7f9570003f20 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0d2830 con 0x7f958c15db40 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f9590a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f958c15ed90 con 0x7f958c156bf0 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f958a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f958a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47332/0 (socket says 172.21.15.67:47332) 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.716+0000 7f958a800640 1 -- 172.21.15.67:0/280875341 learned_addr learned my addr 172.21.15.67:0/280875341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958b200640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 0x7f958c0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c156bf0 msgr2=0x7f958c15d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f9589e00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c156bf0 0x7f958c15d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c156bf0 0x7f958c15d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 msgr2=0x7f958c0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 0x7f958c0c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 -- 172.21.15.67:0/280875341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0c9cb0 con 0x7f9570003f20 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958b200640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 0x7f958c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f958a800640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e7d0 tx=0x7f957c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c00ee80 con 0x7f9570003f20 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f957c018070 con 0x7f9570003f20 2024-04-17T08:53:05.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.717+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c0c9f90 con 0x7f9570003f20 2024-04-17T08:53:05.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.718+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c00c6c0 con 0x7f9570003f20 2024-04-17T08:53:05.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.718+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c0ca550 con 0x7f9570003f20 2024-04-17T08:53:05.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.719+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c0755d0 con 0x7f9570003f20 2024-04-17T08:53:05.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.720+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7f957c00c860 con 0x7f9570003f20 2024-04-17T08:53:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.721+0000 7f9582a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 0x7f956c06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.721+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7f957c08bfe0 con 0x7f9570003f20 2024-04-17T08:53:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.721+0000 7f9589e00640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 msgr2=0x7f956c06ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.721+0000 7f9589e00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 0x7f956c06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:05.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.725+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f957c059f50 con 0x7f9570003f20 2024-04-17T08:53:05.745 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.746+0000 7f0194e49040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:05.755 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.756+0000 7f751fc49040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:53:05.798 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.799+0000 7f0194e49040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:53:05.813 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.814+0000 7f751fc49040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:53:05.862 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.863+0000 7f0194e49040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:53:05.866 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:05.867+0000 7f751fc49040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:53:05.916 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:05.916+0000 7f0194e49040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:53:05.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.922+0000 7f9589e00640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 msgr2=0x7f956c06ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:05.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.922+0000 7f9589e00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 0x7f956c06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:05.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.933+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f958c0b64c0 con 0x7f9570003f20 2024-04-17T08:53:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.934+0000 7f9582a00640 1 -- 172.21.15.67:0/280875341 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v273) ==== 104+0+11 (secure 0 0 0) 0x7f957c059850 con 0x7f9570003f20 2024-04-17T08:53:05.934 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:53:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 msgr2=0x7f956c06ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 0x7f956c06ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 msgr2=0x7f958c0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e7d0 tx=0x7f957c00ec90 comp rx=0 tx=0).stop 2024-04-17T08:53:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 shutdown_connections 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7f956c06cad0 0x7f956c06ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f958c15db40 0x7f958c0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.937+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f958c156bf0 0x7f958c15d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.938+0000 7f9590a00640 1 --2- 172.21.15.67:0/280875341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9570003f20 0x7f958c0ce040 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.938+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 >> 172.21.15.67:0/280875341 conn(0x7f958c09daa0 msgr2=0x7f958c0b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.938+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 shutdown_connections 2024-04-17T08:53:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:05.938+0000 7f9590a00640 1 -- 172.21.15.67:0/280875341 wait complete. 2024-04-17T08:53:05.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:53:06.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.215+0000 7efc98e00640 1 Processor -- start 2024-04-17T08:53:06.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.216+0000 7efc98e00640 1 -- start start 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.217+0000 7efc98e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.217+0000 7efc98e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc940b4c70 0x7efc94161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.217+0000 7efc98e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc940b5110 con 0x7efc940d33d0 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.217+0000 7efc98e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efc940b53f0 con 0x7efc940b4c70 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.217+0000 7efc98e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efc940b56d0 con 0x7efc940d2ff0 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc940b4c70 0x7efc94161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc940b4c70 0x7efc94161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47334/0 (socket says 172.21.15.67:47334) 2024-04-17T08:53:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 -- 172.21.15.67:0/1567527859 learned_addr learned my addr 172.21.15.67:0/1567527859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc93400640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 -- 172.21.15.67:0/1567527859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 msgr2=0x7efc940b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.218+0000 7efc8aa00640 1 -- 172.21.15.67:0/1567527859 >> v1:172.21.15.67:6789/0 conn(0x7efc940d33d0 legacy=0x7efc940d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:06.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.219+0000 7efc8aa00640 1 -- 172.21.15.67:0/1567527859 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc940cf340 con 0x7efc940b4c70 2024-04-17T08:53:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.220+0000 7efc8aa00640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94161ba0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009870 tx=0x7efc7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b3224c9057c6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.220+0000 7efc92a00640 1 -- 172.21.15.67:0/1567527859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c034070 con 0x7efc940b4c70 2024-04-17T08:53:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.220+0000 7efc92a00640 1 -- 172.21.15.67:0/1567527859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc7c030840 con 0x7efc940b4c70 2024-04-17T08:53:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.220+0000 7efc92a00640 1 -- 172.21.15.67:0/1567527859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c030c10 con 0x7efc940b4c70 2024-04-17T08:53:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 -- 172.21.15.67:0/1567527859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 msgr2=0x7efc94161ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94161ba0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009870 tx=0x7efc7c02b980 comp rx=0 tx=0).stop 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 -- 172.21.15.67:0/1567527859 shutdown_connections 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94161ba0 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 --2- 172.21.15.67:0/1567527859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 -- 172.21.15.67:0/1567527859 >> 172.21.15.67:0/1567527859 conn(0x7efc9409da80 msgr2=0x7efc9408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 -- 172.21.15.67:0/1567527859 shutdown_connections 2024-04-17T08:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.221+0000 7efc98e00640 1 -- 172.21.15.67:0/1567527859 wait complete. 2024-04-17T08:53:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.223+0000 7efc98e00640 1 Processor -- start 2024-04-17T08:53:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.223+0000 7efc98e00640 1 -- start start 2024-04-17T08:53:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc940d33d0 0x7efc940cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc940d0dd0 con 0x7efc940b4c70 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efc940d0c50 con 0x7efc940d33d0 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc98e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efc940d0f50 con 0x7efc940d2ff0 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47338/0 (socket says 172.21.15.67:47338) 2024-04-17T08:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.224+0000 7efc93400640 1 -- 172.21.15.67:0/2992158532 learned_addr learned my addr 172.21.15.67:0/2992158532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc940d33d0 0x7efc940cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc8aa00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 msgr2=0x7efc940ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc940d33d0 msgr2=0x7efc940cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc940d33d0 0x7efc940cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 -- 172.21.15.67:0/2992158532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc941f49e0 con 0x7efc940b4c70 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.225+0000 7efc93400640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7efc84007f20 tx=0x7efc8400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.226+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84014620 con 0x7efc940b4c70 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.226+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc84019070 con 0x7efc940b4c70 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.226+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84014a00 con 0x7efc940b4c70 2024-04-17T08:53:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.226+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc941f4cc0 con 0x7efc940b4c70 2024-04-17T08:53:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.226+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc941f5280 con 0x7efc940b4c70 2024-04-17T08:53:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.227+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc940755d0 con 0x7efc940b4c70 2024-04-17T08:53:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.228+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7efc84014ba0 con 0x7efc940b4c70 2024-04-17T08:53:06.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.229+0000 7efc8be00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 0x7efc7006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.229+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7efc8408ca20 con 0x7efc940b4c70 2024-04-17T08:53:06.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.229+0000 7efc8aa00640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 msgr2=0x7efc7006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.229+0000 7efc8aa00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 0x7efc7006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:06.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.234+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efc8405a930 con 0x7efc940b4c70 2024-04-17T08:53:06.252 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.253+0000 7f751fc49040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:53:06.331 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.332+0000 7f0194e49040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:53:06.412 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.412+0000 7f751fc49040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:53:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.430+0000 7efc8aa00640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 msgr2=0x7efc7006ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:06.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.430+0000 7efc8aa00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 0x7efc7006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:06.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.435+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc9406d730 con 0x7efc940b4c70 2024-04-17T08:53:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.436+0000 7efc8be00640 1 -- 172.21.15.67:0/2992158532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v273) ==== 100+0+11 (secure 0 0 0) 0x7efc8405a230 con 0x7efc940b4c70 2024-04-17T08:53:06.435 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 msgr2=0x7efc7006ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 0x7efc7006ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 msgr2=0x7efc94155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7efc84007f20 tx=0x7efc8400e570 comp rx=0 tx=0).stop 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 shutdown_connections 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7efc7006cad0 0x7efc7006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc940d33d0 0x7efc940cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc940d2ff0 0x7efc940ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 --2- 172.21.15.67:0/2992158532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc940b4c70 0x7efc94155d30 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 >> 172.21.15.67:0/2992158532 conn(0x7efc9409da80 msgr2=0x7efc9406cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.438+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 shutdown_connections 2024-04-17T08:53:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.439+0000 7efc98e00640 1 -- 172.21.15.67:0/2992158532 wait complete. 2024-04-17T08:53:06.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:53:06.504 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.505+0000 7f0194e49040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:53:06.521 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.522+0000 7f751fc49040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:53:06.571 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.572+0000 7f751fc49040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:53:06.620 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.621+0000 7f0194e49040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:53:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.664+0000 7fd38dc00640 1 Processor -- start 2024-04-17T08:53:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.664+0000 7fd38dc00640 1 -- start start 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.665+0000 7fd38dc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd3880b9850 0x7fd3880d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.665+0000 7fd38dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3880d4140 con 0x7fd3880d1ab0 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.665+0000 7fd38dc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3880d3b80 con 0x7fd38815ee20 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.665+0000 7fd38dc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd3880b83b0 con 0x7fd3880b9850 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd37fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd3880d1ab0 0x7fd3880d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42566/0 (socket says 172.21.15.67:42566) 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd37fe00640 1 -- 172.21.15.67:0/1316960664 learned_addr learned my addr 172.21.15.67:0/1316960664 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd38d200640 1 --2- 172.21.15.67:0/1316960664 >> v2:172.21.15.187:3300/0 conn(0x7fd3880b9850 0x7fd3880d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224857556 0 0) 0x7fd3880d4140 con 0x7fd3880d1ab0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd36c003520 con 0x7fd3880d1ab0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2852551545 0 0) 0x7fd36c003520 con 0x7fd3880d1ab0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 >> v2:172.21.15.187:3300/0 conn(0x7fd3880b9850 msgr2=0x7fd3880d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 --2- 172.21.15.67:0/1316960664 >> v2:172.21.15.187:3300/0 conn(0x7fd3880b9850 0x7fd3880d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 >> v1:172.21.15.187:6789/0 conn(0x7fd38815ee20 legacy=0x7fd3880d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3880d3e60 con 0x7fd3880d1ab0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.666+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd370001670 con 0x7fd3880d1ab0 2024-04-17T08:53:06.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 >> v1:172.21.15.67:6789/0 conn(0x7fd3880d1ab0 legacy=0x7fd3880d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 0x7fd36c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36c004800 0x7fd36c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c0051c0 0x7fd36c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3880b83b0 con 0x7fd36c003a20 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c00c0f0 con 0x7fd36c004800 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd36c00c2f0 con 0x7fd36c0051c0 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 0x7fd36c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd37fe00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36c004800 0x7fd36c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd38d200640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c0051c0 0x7fd36c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 -- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c0051c0 msgr2=0x7fd36c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c0051c0 0x7fd36c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 -- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36c004800 msgr2=0x7fd36c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd36c004800 0x7fd36c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd387e00640 1 -- 172.21.15.67:0/1316960664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c003520 con 0x7fd36c003a20 2024-04-17T08:53:06.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.667+0000 7fd38d200640 1 -- 172.21.15.67:0/1316960664 reap_dead start 2024-04-17T08:53:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.668+0000 7fd387e00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 0x7fd36c003e40 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fd378001750 tx=0x7fd36c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6149682b2f87c977 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.669+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378001370 con 0x7fd36c003a20 2024-04-17T08:53:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.669+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3780052d0 con 0x7fd36c003a20 2024-04-17T08:53:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.669+0000 7fd387400640 1 -- 172.21.15.67:0/1316960664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378003830 con 0x7fd36c003a20 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 -- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 msgr2=0x7fd36c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 0x7fd36c003e40 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fd378001750 tx=0x7fd36c005830 comp rx=0 tx=0).stop 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 -- 172.21.15.67:0/1316960664 shutdown_connections 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 --2- 172.21.15.67:0/1316960664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd36c003a20 0x7fd36c003e40 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 -- 172.21.15.67:0/1316960664 >> 172.21.15.67:0/1316960664 conn(0x7fd38806d3c0 msgr2=0x7fd3880b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 -- 172.21.15.67:0/1316960664 shutdown_connections 2024-04-17T08:53:06.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.670+0000 7fd38dc00640 1 -- 172.21.15.67:0/1316960664 wait complete. 2024-04-17T08:53:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.672+0000 7fd38dc00640 1 Processor -- start 2024-04-17T08:53:06.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.672+0000 7fd38dc00640 1 -- start start 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c003a20 0x7fd3881f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3881525f0 0x7fd3880c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3880d13a0 con 0x7fd3880b5700 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd38815f200 con 0x7fd3881525f0 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd3880d1520 con 0x7fd36c003a20 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47352/0 (socket says 172.21.15.67:47352) 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 -- 172.21.15.67:0/36583913 learned_addr learned my addr 172.21.15.67:0/36583913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd387e00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c003a20 0x7fd3881f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd37fe00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3881525f0 0x7fd3880c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c003a20 msgr2=0x7fd3881f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c003a20 0x7fd3881f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3881525f0 msgr2=0x7fd3880c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3881525f0 0x7fd3880c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.673+0000 7fd38d200640 1 -- 172.21.15.67:0/36583913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3880ca390 con 0x7fd3880b5700 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd38d200640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fd3880b9ce0 tx=0x7fd38000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380016620 con 0x7fd3880b5700 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd38001b070 con 0x7fd3880b5700 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3880ca670 con 0x7fd3880b5700 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380016a00 con 0x7fd3880b5700 2024-04-17T08:53:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.674+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3880cac30 con 0x7fd3880b5700 2024-04-17T08:53:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.676+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd388068a10 con 0x7fd3880b5700 2024-04-17T08:53:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.676+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7fd380016ba0 con 0x7fd3880b5700 2024-04-17T08:53:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.677+0000 7fd384c00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 0x7fd36406ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.677+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fd38008d0a0 con 0x7fd3880b5700 2024-04-17T08:53:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.677+0000 7fd387e00640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 msgr2=0x7fd36406ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.677+0000 7fd387e00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 0x7fd36406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:06.677 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.678+0000 7f0194e49040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:53:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.681+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd380025020 con 0x7fd3880b5700 2024-04-17T08:53:06.699 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.700+0000 7f751fc49040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:06.807 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.807+0000 7f751fc49040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:53:06.812 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.813+0000 7f0194e49040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:06.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.878+0000 7fd387e00640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 msgr2=0x7fd36406ef90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:06.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.878+0000 7fd387e00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 0x7fd36406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.897+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd3880d4140 con 0x7fd3880b5700 2024-04-17T08:53:06.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.897+0000 7fd384c00640 1 -- 172.21.15.67:0/36583913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 11 entries v273) ==== 81+0+653 (secure 0 0 0) 0x7fd38005af30 con 0x7fd3880b5700 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3094572004 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2893316565 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2486101166 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3800011905 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3721653371 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/227074939 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/456496732 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/1143620423 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/1143620423 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3973592351 2024-04-18T08:53:00.117721+0000 2024-04-17T08:53:06.898 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/4229015777 2024-04-17T09:52:41.140106+0000 2024-04-17T08:53:06.898 INFO:teuthology.orchestra.run.smithi067.stderr:listed 11 entries 2024-04-17T08:53:06.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 msgr2=0x7fd36406ef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:06.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 0x7fd36406ef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 msgr2=0x7fd3880b5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:06.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7fd3880b9ce0 tx=0x7fd38000f520 comp rx=0 tx=0).stop 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 shutdown_connections 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd36406cad0 0x7fd36406ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3880b5700 0x7fd3880b5b90 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3881525f0 0x7fd3880c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 --2- 172.21.15.67:0/36583913 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd36c003a20 0x7fd3881f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 >> 172.21.15.67:0/36583913 conn(0x7fd38806d3c0 msgr2=0x7fd3880cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 shutdown_connections 2024-04-17T08:53:06.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:06.900+0000 7fd38dc00640 1 -- 172.21.15.67:0/36583913 wait complete. 2024-04-17T08:53:06.924 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:53:06.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:06.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:53:06.934 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:06.935+0000 7f0194e49040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:53:06.982 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:06.982+0000 7f751fc49040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:07.030 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:53:07.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:07.031 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:53:07.032 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:07.033+0000 7f751fc49040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:07.121 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:53:07.121 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:53:07.121 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:07.128 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:07.128+0000 7f0194e49040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:07.172 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:07.172+0000 7f751fc49040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:53:07.174 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:53:07.174 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:53:07.175 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:07.175 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:07.183 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:07.181+0000 7f0194e49040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:07.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:53:07.227 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:53:07.227 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:53:07.227 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:07.278 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:53:07.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:07.328 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:07.328+0000 7f0194e49040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:53:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.491+0000 7fa4e7e00640 1 Processor -- start 2024-04-17T08:53:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.492+0000 7fa4e7e00640 1 -- start start 2024-04-17T08:53:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4e7e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4e815ca20 0x7fa4e80d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4e7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e815ce00 con 0x7fa4e80d0720 2024-04-17T08:53:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4e7e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e815d090 con 0x7fa4e809b170 2024-04-17T08:53:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4e7e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa4e815d330 con 0x7fa4e815ca20 2024-04-17T08:53:07.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4dfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa4e80d0720 0x7fa4e809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42582/0 (socket says 172.21.15.67:42582) 2024-04-17T08:53:07.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4dfe00640 1 -- 172.21.15.67:0/1777488955 learned_addr learned my addr 172.21.15.67:0/1777488955 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:07.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.493+0000 7fa4e6a00640 1 --2- 172.21.15.67:0/1777488955 >> v2:172.21.15.187:3300/0 conn(0x7fa4e815ca20 0x7fa4e80d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 878819923 0 0) 0x7fa4e815ce00 con 0x7fa4e80d0720 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4cc003520 con 0x7fa4e80d0720 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2294914458 0 0) 0x7fa4cc003520 con 0x7fa4e80d0720 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 >> v2:172.21.15.187:3300/0 conn(0x7fa4e815ca20 msgr2=0x7fa4e80d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 --2- 172.21.15.67:0/1777488955 >> v2:172.21.15.187:3300/0 conn(0x7fa4e815ca20 0x7fa4e80d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 >> v1:172.21.15.187:6789/0 conn(0x7fa4e809b170 legacy=0x7fa4e80cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e815d610 con 0x7fa4e80d0720 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa4d0001670 con 0x7fa4e80d0720 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 >> v1:172.21.15.67:6789/0 conn(0x7fa4e80d0720 legacy=0x7fa4e809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4cc003a70 0x7fa4cc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4cc004800 0x7fa4cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e815d330 con 0x7fa4cc005210 2024-04-17T08:53:07.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4cc00c140 con 0x7fa4cc004800 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.494+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4cc00c340 con 0x7fa4cc003a70 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4dfe00640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4cc004800 0x7fa4cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e6a00640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4cc003a70 0x7fa4cc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 -- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4cc003a70 msgr2=0x7fa4cc003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4cc003a70 0x7fa4cc003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 -- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4cc004800 msgr2=0x7fa4cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4cc004800 0x7fa4cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.495+0000 7fa4e7400640 1 -- 172.21.15.67:0/1777488955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4cc003520 con 0x7fa4cc005210 2024-04-17T08:53:07.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.496+0000 7fa4e7400640 1 -- 172.21.15.67:0/1777488955 reap_dead start 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.496+0000 7fa4e7400640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4cc00b9c0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0001e50 tx=0x7fa4cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec8bd3d3edd9deb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.496+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0001a70 con 0x7fa4cc005210 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.496+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e0003bb0 con 0x7fa4cc005210 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.496+0000 7fa4e6000640 1 -- 172.21.15.67:0/1777488955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00050d0 con 0x7fa4cc005210 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 -- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 msgr2=0x7fa4cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4cc00b9c0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0001e50 tx=0x7fa4cc005830 comp rx=0 tx=0).stop 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 -- 172.21.15.67:0/1777488955 shutdown_connections 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/1777488955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4cc00b9c0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 -- 172.21.15.67:0/1777488955 >> 172.21.15.67:0/1777488955 conn(0x7fa4e80b4100 msgr2=0x7fa4e80b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 -- 172.21.15.67:0/1777488955 shutdown_connections 2024-04-17T08:53:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.497+0000 7fa4e7e00640 1 -- 172.21.15.67:0/1777488955 wait complete. 2024-04-17T08:53:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.499+0000 7fa4e7e00640 1 Processor -- start 2024-04-17T08:53:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.499+0000 7fa4e7e00640 1 -- start start 2024-04-17T08:53:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4e809ce70 0x7fa4e809d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 0x7fa4e80ce660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e809bea0 con 0x7fa4cc005210 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e809bd20 con 0x7fa4e809ce70 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e7e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e80d01e0 con 0x7fa4e80ce1e0 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47364/0 (socket says 172.21.15.67:47364) 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.500+0000 7fa4e6a00640 1 -- 172.21.15.67:0/444116989 learned_addr learned my addr 172.21.15.67:0/444116989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e7400640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 0x7fa4e80ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4dfe00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4e809ce70 0x7fa4e809d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 msgr2=0x7fa4e80ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 0x7fa4e80ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4e809ce70 msgr2=0x7fa4e809d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4e809ce70 0x7fa4e809d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 -- 172.21.15.67:0/444116989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e809fa40 con 0x7fa4cc005210 2024-04-17T08:53:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e7400640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 0x7fa4e80ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.501+0000 7fa4e6a00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8011540 tx=0x7fa4d8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:07.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.502+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8015930 con 0x7fa4cc005210 2024-04-17T08:53:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.502+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d8015ad0 con 0x7fa4cc005210 2024-04-17T08:53:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.502+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800f460 con 0x7fa4cc005210 2024-04-17T08:53:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.502+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e809fd20 con 0x7fa4cc005210 2024-04-17T08:53:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.502+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e80b6320 con 0x7fa4cc005210 2024-04-17T08:53:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.503+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e80b6140 con 0x7fa4cc005210 2024-04-17T08:53:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.505+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7fa4d800f600 con 0x7fa4cc005210 2024-04-17T08:53:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.506+0000 7fa4df400640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 0x7fa4c406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.506+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fa4d808c920 con 0x7fa4cc005210 2024-04-17T08:53:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.507+0000 7fa4dfe00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 msgr2=0x7fa4c406efb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:07.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.507+0000 7fa4dfe00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 0x7fa4c406efb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.511+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa4e80b6140 con 0x7fa4cc005210 2024-04-17T08:53:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.708+0000 7fa4dfe00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 msgr2=0x7fa4c406efb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.708+0000 7fa4dfe00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 0x7fa4c406efb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.755+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4e80755d0 con 0x7fa4cc005210 2024-04-17T08:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.756+0000 7fa4df400640 1 -- 172.21.15.67:0/444116989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7fa4d8019030 con 0x7fa4cc005210 2024-04-17T08:53:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:07.759 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 msgr2=0x7fa4c406efb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 0x7fa4c406efb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 msgr2=0x7fa4e814cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8011540 tx=0x7fa4d8011a00 comp rx=0 tx=0).stop 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 shutdown_connections 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fa4c406caf0 0x7fa4c406efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4e80ce1e0 0x7fa4e80ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4e809ce70 0x7fa4e809d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 --2- 172.21.15.67:0/444116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4cc005210 0x7fa4e814cb50 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 >> 172.21.15.67:0/444116989 conn(0x7fa4e80b4100 msgr2=0x7fa4e80ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 shutdown_connections 2024-04-17T08:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.759+0000 7fa4e7e00640 1 -- 172.21.15.67:0/444116989 wait complete. 2024-04-17T08:53:07.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:07.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.964+0000 7fec12000640 1 Processor -- start 2024-04-17T08:53:07.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.964+0000 7fec12000640 1 -- start start 2024-04-17T08:53:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.966+0000 7fec12000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.966+0000 7fec12000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c160280 0x7fec0c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.966+0000 7fec12000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0c0cd4a0 con 0x7fec0c141ed0 2024-04-17T08:53:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.966+0000 7fec12000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fec0c15ce30 con 0x7fec0c06b8c0 2024-04-17T08:53:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.966+0000 7fec12000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec0c1607f0 con 0x7fec0c160280 2024-04-17T08:53:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec0be00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c160280 0x7fec0c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:51278/0 (socket says 172.21.15.67:51278) 2024-04-17T08:53:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 -- 172.21.15.67:0/1604172995 learned_addr learned my addr 172.21.15.67:0/1604172995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 -- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c160280 msgr2=0x7fec0c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 --2- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c160280 0x7fec0c06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 -- 172.21.15.67:0/1604172995 >> v1:172.21.15.187:6789/0 conn(0x7fec0c141ed0 legacy=0x7fec0c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.967+0000 7fec10c00640 1 -- 172.21.15.67:0/1604172995 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0c160ad0 con 0x7fec0c06b8c0 2024-04-17T08:53:07.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.969+0000 7fec10c00640 1 --2- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fec00009870 tx=0x7fec0002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21465011994cbefd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:07.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.969+0000 7fec0b400640 1 -- 172.21.15.67:0/1604172995 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00034070 con 0x7fec0c06b8c0 2024-04-17T08:53:07.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.969+0000 7fec0b400640 1 -- 172.21.15.67:0/1604172995 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec00030840 con 0x7fec0c06b8c0 2024-04-17T08:53:07.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.969+0000 7fec0b400640 1 -- 172.21.15.67:0/1604172995 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00030c10 con 0x7fec0c06b8c0 2024-04-17T08:53:07.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 -- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c06b8c0 msgr2=0x7fec0c0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 --2- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fec00009870 tx=0x7fec0002b980 comp rx=0 tx=0).stop 2024-04-17T08:53:07.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 -- 172.21.15.67:0/1604172995 shutdown_connections 2024-04-17T08:53:07.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 --2- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c160280 0x7fec0c06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 --2- 172.21.15.67:0/1604172995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c06b8c0 0x7fec0c0cd8b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 -- 172.21.15.67:0/1604172995 >> 172.21.15.67:0/1604172995 conn(0x7fec0c067d10 msgr2=0x7fec0c0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 -- 172.21.15.67:0/1604172995 shutdown_connections 2024-04-17T08:53:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.970+0000 7fec12000640 1 -- 172.21.15.67:0/1604172995 wait complete. 2024-04-17T08:53:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.972+0000 7fec12000640 1 Processor -- start 2024-04-17T08:53:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.972+0000 7fec12000640 1 -- start start 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c141ed0 0x7fec0c155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c160280 0x7fec0c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec0c06c1e0 con 0x7fec0c06b8c0 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec0c0cddf0 con 0x7fec0c160280 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec12000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec0c161090 con 0x7fec0c141ed0 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47376/0 (socket says 172.21.15.67:47376) 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 -- 172.21.15.67:0/2550122240 learned_addr learned my addr 172.21.15.67:0/2550122240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c141ed0 msgr2=0x7fec0c155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c141ed0 0x7fec0c155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c160280 msgr2=0x7fec0c156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c160280 0x7fec0c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.974+0000 7fec10c00640 1 -- 172.21.15.67:0/2550122240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0c13c220 con 0x7fec0c06b8c0 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec10c00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fec00009870 tx=0x7fec00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00048070 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec00030640 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0c13c500 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00034040 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.975+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec0c13cac0 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.977+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0c068a10 con 0x7fec0c06b8c0 2024-04-17T08:53:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.978+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7fec00037070 con 0x7fec0c06b8c0 2024-04-17T08:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.978+0000 7fec08c00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 0x7febec06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.978+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fec000adfa0 con 0x7fec0c06b8c0 2024-04-17T08:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.979+0000 7fec0be00640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 msgr2=0x7febec06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.979+0000 7fec0be00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 0x7febec06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:07.985+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fec0007bed0 con 0x7fec0c06b8c0 2024-04-17T08:53:08.003 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.004+0000 7f751fc49040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:53:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.180+0000 7fec0be00640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 msgr2=0x7febec06ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.180+0000 7fec0be00640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 0x7febec06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.191+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec0c06cdb0 con 0x7fec0c06b8c0 2024-04-17T08:53:08.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.193+0000 7fec08c00640 1 -- 172.21.15.67:0/2550122240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) ==== 74+0+17319 (secure 0 0 0) 0x7fec0007b7d0 con 0x7fec0c06b8c0 2024-04-17T08:53:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:08.193 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":273,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:00.117836+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/3094572004":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2893316565":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2486101166":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3800011905":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3721653371":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/227074939":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/456496732":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6832/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6833/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3973592351":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:08.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.195+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 msgr2=0x7febec06ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:08.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.195+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 0x7febec06ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.195+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 msgr2=0x7fec0c06f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.195+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fec00009870 tx=0x7fec00004730 comp rx=0 tx=0).stop 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 shutdown_connections 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7febec06caa0 0x7febec06ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec0c160280 0x7fec0c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec0c141ed0 0x7fec0c155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 --2- 172.21.15.67:0/2550122240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec0c06b8c0 0x7fec0c06f0b0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 >> 172.21.15.67:0/2550122240 conn(0x7fec0c067d10 msgr2=0x7fec0c0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 shutdown_connections 2024-04-17T08:53:08.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.196+0000 7fec12000640 1 -- 172.21.15.67:0/2550122240 wait complete. 2024-04-17T08:53:08.196 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:08.197+0000 7f0194e49040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:53:08.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:52:43.162800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:52:44.240860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:53:08.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:53:08.308 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.309+0000 7f751fc49040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:53:08.367 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.367+0000 7f751fc49040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.431+0000 7fd133200640 1 Processor -- start 2024-04-17T08:53:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.431+0000 7fd133200640 1 -- start start 2024-04-17T08:53:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd133200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd12c0d22b0 0x7fd12c0d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd133200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12c0d4f70 con 0x7fd12c15ec20 2024-04-17T08:53:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd133200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12c0d5250 con 0x7fd12c0b3b50 2024-04-17T08:53:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd133200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd12c0d2760 con 0x7fd12c0d22b0 2024-04-17T08:53:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd131e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd12c15ec20 0x7fd12c06ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42594/0 (socket says 172.21.15.67:42594) 2024-04-17T08:53:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.433+0000 7fd131e00640 1 -- 172.21.15.67:0/1668142533 learned_addr learned my addr 172.21.15.67:0/1668142533 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd132800640 1 --2- 172.21.15.67:0/1668142533 >> v2:172.21.15.187:3300/0 conn(0x7fd12c0d22b0 0x7fd12c0d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133997680 0 0) 0x7fd12c0d4f70 con 0x7fd12c15ec20 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd114003520 con 0x7fd12c15ec20 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4208558637 0 0) 0x7fd114003520 con 0x7fd12c15ec20 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 >> v2:172.21.15.187:3300/0 conn(0x7fd12c0d22b0 msgr2=0x7fd12c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 --2- 172.21.15.67:0/1668142533 >> v2:172.21.15.187:3300/0 conn(0x7fd12c0d22b0 0x7fd12c0d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 >> v1:172.21.15.187:6789/0 conn(0x7fd12c0b3b50 legacy=0x7fd12c0cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c0cf4d0 con 0x7fd12c15ec20 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd120001670 con 0x7fd12c15ec20 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 >> v1:172.21.15.67:6789/0 conn(0x7fd12c15ec20 legacy=0x7fd12c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 0x7fd114003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114004800 0x7fd114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1140052e0 0x7fd11400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c0d2760 con 0x7fd114003a00 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd11400c1e0 con 0x7fd114004800 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.434+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd11400c3e0 con 0x7fd1140052e0 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 0x7fd114003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131400640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114004800 0x7fd114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd132800640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1140052e0 0x7fd11400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 -- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1140052e0 msgr2=0x7fd11400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1140052e0 0x7fd11400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 -- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114004800 msgr2=0x7fd114004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114004800 0x7fd114004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131e00640 1 -- 172.21.15.67:0/1668142533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd114003520 con 0x7fd114003a00 2024-04-17T08:53:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd132800640 1 -- 172.21.15.67:0/1668142533 reap_dead start 2024-04-17T08:53:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.435+0000 7fd131400640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114004800 0x7fd114004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.437+0000 7fd131e00640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 0x7fd114003e20 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7fd12000c110 tx=0x7fd114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f96b81601c68520 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.437+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120007c40 con 0x7fd114003a00 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.437+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd120030070 con 0x7fd114003a00 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.437+0000 7fd130a00640 1 -- 172.21.15.67:0/1668142533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120005d10 con 0x7fd114003a00 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 -- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 msgr2=0x7fd114003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 0x7fd114003e20 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7fd12000c110 tx=0x7fd114005830 comp rx=0 tx=0).stop 2024-04-17T08:53:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 -- 172.21.15.67:0/1668142533 shutdown_connections 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 --2- 172.21.15.67:0/1668142533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd114003a00 0x7fd114003e20 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 -- 172.21.15.67:0/1668142533 >> 172.21.15.67:0/1668142533 conn(0x7fd12c08bff0 msgr2=0x7fd12c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 -- 172.21.15.67:0/1668142533 shutdown_connections 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.438+0000 7fd133200640 1 -- 172.21.15.67:0/1668142533 wait complete. 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.440+0000 7fd133200640 1 Processor -- start 2024-04-17T08:53:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.440+0000 7fd133200640 1 -- start start 2024-04-17T08:53:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114003a00 0x7fd12c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd12c1448a0 0x7fd12c144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c06ca40 con 0x7fd12c15f000 2024-04-17T08:53:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c0cf180 con 0x7fd114003a00 2024-04-17T08:53:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd133200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd12c06cbc0 con 0x7fd12c1448a0 2024-04-17T08:53:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd131400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd131400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47388/0 (socket says 172.21.15.67:47388) 2024-04-17T08:53:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.441+0000 7fd131400640 1 -- 172.21.15.67:0/3395326084 learned_addr learned my addr 172.21.15.67:0/3395326084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd132800640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd12c1448a0 0x7fd12c144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131e00640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114003a00 0x7fd12c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 -- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd12c1448a0 msgr2=0x7fd12c144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd12c1448a0 0x7fd12c144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 -- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114003a00 msgr2=0x7fd12c0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114003a00 0x7fd12c0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 -- 172.21.15.67:0/3395326084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c0ca8a0 con 0x7fd12c15f000 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd131400640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e470 tx=0x7fd11c00e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c019070 con 0x7fd12c15f000 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd11c0052e0 con 0x7fd12c15f000 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c003880 con 0x7fd12c15f000 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.442+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12c145260 con 0x7fd12c15f000 2024-04-17T08:53:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.443+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12c1457d0 con 0x7fd12c15f000 2024-04-17T08:53:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.444+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd12c068a10 con 0x7fd12c15f000 2024-04-17T08:53:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.445+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 152) ==== 91199+0+0 (secure 0 0 0) 0x7fd11c01c070 con 0x7fd12c15f000 2024-04-17T08:53:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.446+0000 7fd126000640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 0x7fd11006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.446+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6210+0+0 (secure 0 0 0) 0x7fd11c08b6f0 con 0x7fd12c15f000 2024-04-17T08:53:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.446+0000 7fd131e00640 1 -- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 msgr2=0x7fd11006ef60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1608360619 2024-04-17T08:53:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.446+0000 7fd131e00640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 0x7fd11006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.450+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd12c0ca8a0 con 0x7fd12c15f000 2024-04-17T08:53:08.499 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.499+0000 7f751fc49040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:08.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.520+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 153) ==== 90751+0+0 (secure 0 0 0) 0x7fd11c055590 con 0x7fd12c15f000 2024-04-17T08:53:08.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.521+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 msgr2=0x7fd11006ef60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:08.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.521+0000 7fd126000640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 0x7fd11006ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.535 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:08.536+0000 7f0194e49040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:53:08.594 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f749d400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.594 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f749d400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f749d400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f749d400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f749d400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f7499800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f7499800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f7499800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f7499800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.595 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:08.595+0000 7f7499800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:08.600 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:08.601+0000 7f0194e49040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.666+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd12c0d55f0 con 0x7fd12c15f000 2024-04-17T08:53:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.667+0000 7fd126000640 1 -- 172.21.15.67:0/3395326084 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v274) ==== 93+0+10 (secure 0 0 0) 0x7fd11c077850 con 0x7fd12c15f000 2024-04-17T08:53:08.667 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:53:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 msgr2=0x7fd12c15f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e470 tx=0x7fd11c00e930 comp rx=0 tx=0).stop 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 shutdown_connections 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:6832/1608360619,v1:172.21.15.187:6833/1608360619] conn(0x7fd11006caa0 0x7fd11006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd12c1448a0 0x7fd12c144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd12c15f000 0x7fd12c15f460 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 --2- 172.21.15.67:0/3395326084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd114003a00 0x7fd12c0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 >> 172.21.15.67:0/3395326084 conn(0x7fd12c08bff0 msgr2=0x7fd12c0b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 shutdown_connections 2024-04-17T08:53:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.670+0000 7fd133200640 1 -- 172.21.15.67:0/3395326084 wait complete. 2024-04-17T08:53:08.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:53:08.741 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:08.742+0000 7f0194e49040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:08.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.896+0000 7f0847200640 1 Processor -- start 2024-04-17T08:53:08.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.896+0000 7f0847200640 1 -- start start 2024-04-17T08:53:08.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f084015ecb0 0x7f08400d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08400d3210 0x7f08400cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08400cf8e0 0x7f08400d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f08400d0c30 con 0x7f08400d3210 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f08400d00b0 con 0x7f084015ecb0 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0847200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08400d0390 con 0x7f08400cf8e0 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0845400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08400d3210 0x7f08400cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0845400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08400d3210 0x7f08400cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47396/0 (socket says 172.21.15.67:47396) 2024-04-17T08:53:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.898+0000 7f0845400640 1 -- 172.21.15.67:0/520676515 learned_addr learned my addr 172.21.15.67:0/520676515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845400640 1 -- 172.21.15.67:0/520676515 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08400cf8e0 msgr2=0x7f08400d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845e00640 1 --2- 172.21.15.67:0/520676515 >> v2:172.21.15.187:3300/0 conn(0x7f084015ecb0 0x7f08400d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845400640 1 --2- 172.21.15.67:0/520676515 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08400cf8e0 0x7f08400d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845400640 1 -- 172.21.15.67:0/520676515 >> v2:172.21.15.187:3300/0 conn(0x7f084015ecb0 msgr2=0x7f08400d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845400640 1 --2- 172.21.15.67:0/520676515 >> v2:172.21.15.187:3300/0 conn(0x7f084015ecb0 0x7f08400d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.899+0000 7f0845400640 1 -- 172.21.15.67:0/520676515 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08400d0670 con 0x7f08400d3210 2024-04-17T08:53:08.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.900+0000 7f0845400640 1 --2- 172.21.15.67:0/520676515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400d3210 0x7f08400cf3a0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f08400d05a0 tx=0x7f083002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ad1efb18e7bd4b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.900+0000 7f0844a00640 1 -- 172.21.15.67:0/520676515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08300317c0 con 0x7f08400d3210 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.900+0000 7f0844a00640 1 -- 172.21.15.67:0/520676515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0830037070 con 0x7f08400d3210 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.900+0000 7f0844a00640 1 -- 172.21.15.67:0/520676515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830031b80 con 0x7f08400d3210 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 -- 172.21.15.67:0/520676515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400d3210 msgr2=0x7f08400cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 --2- 172.21.15.67:0/520676515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400d3210 0x7f08400cf3a0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f08400d05a0 tx=0x7f083002b860 comp rx=0 tx=0).stop 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 -- 172.21.15.67:0/520676515 shutdown_connections 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 --2- 172.21.15.67:0/520676515 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f08400cf8e0 0x7f08400d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 --2- 172.21.15.67:0/520676515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400d3210 0x7f08400cf3a0 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 --2- 172.21.15.67:0/520676515 >> v2:172.21.15.187:3300/0 conn(0x7f084015ecb0 0x7f08400d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 -- 172.21.15.67:0/520676515 >> 172.21.15.67:0/520676515 conn(0x7f084008bff0 msgr2=0x7f08400d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 -- 172.21.15.67:0/520676515 shutdown_connections 2024-04-17T08:53:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.901+0000 7f0847200640 1 -- 172.21.15.67:0/520676515 wait complete. 2024-04-17T08:53:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.903+0000 7f0847200640 1 Processor -- start 2024-04-17T08:53:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.903+0000 7f0847200640 1 -- start start 2024-04-17T08:53:08.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08400d3210 0x7f08401f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f084015ecb0 0x7f08400b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08400cfd00 con 0x7f08400cf8e0 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f084015f090 con 0x7f08400d3210 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0847200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f08401f55f0 con 0x7f084015ecb0 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0845e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0845e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47400/0 (socket says 172.21.15.67:47400) 2024-04-17T08:53:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.904+0000 7f0845e00640 1 -- 172.21.15.67:0/2882631993 learned_addr learned my addr 172.21.15.67:0/2882631993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 -- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f084015ecb0 msgr2=0x7f08400b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f084015ecb0 0x7f08400b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 -- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08400d3210 msgr2=0x7f08401f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08400d3210 0x7f08401f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 -- 172.21.15.67:0/2882631993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08401f5030 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0845e00640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f083400c7f0 tx=0x7f083400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834015070 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834004220 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834007620 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.905+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08401f5310 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.906+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08401621e0 con 0x7f08400cf8e0 2024-04-17T08:53:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.907+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0840068a10 con 0x7f08400cf8e0 2024-04-17T08:53:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.907+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 153) ==== 90751+0+0 (secure 0 0 0) 0x7f08340079d0 con 0x7f08400cf8e0 2024-04-17T08:53:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.908+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f083408ba50 con 0x7f08400cf8e0 2024-04-17T08:53:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:08.913+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0834059e60 con 0x7f08400cf8e0 2024-04-17T08:53:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.116+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f08400b5a70 con 0x7f08400cf8e0 2024-04-17T08:53:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.117+0000 7f083a000640 1 -- 172.21.15.67:0/2882631993 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v274) ==== 104+0+11 (secure 0 0 0) 0x7f0834059760 con 0x7f08400cf8e0 2024-04-17T08:53:09.116 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:53:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 msgr2=0x7f08400cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f083400c7f0 tx=0x7f083400ccb0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 shutdown_connections 2024-04-17T08:53:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f084015ecb0 0x7f08400b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08400d3210 0x7f08401f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 --2- 172.21.15.67:0/2882631993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08400cf8e0 0x7f08400cabe0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 >> 172.21.15.67:0/2882631993 conn(0x7f084008bff0 msgr2=0x7f084015dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 shutdown_connections 2024-04-17T08:53:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.120+0000 7f0847200640 1 -- 172.21.15.67:0/2882631993 wait complete. 2024-04-17T08:53:09.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:53:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.367+0000 7f0c13000640 1 Processor -- start 2024-04-17T08:53:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.367+0000 7f0c13000640 1 -- start start 2024-04-17T08:53:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.369+0000 7f0c13000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.369+0000 7f0c13000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0c0c146060 0x7f0c0c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.369+0000 7f0c13000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0c160920 con 0x7f0c0c0b4ac0 2024-04-17T08:53:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.369+0000 7f0c13000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0c0c160c00 con 0x7f0c0c146060 2024-04-17T08:53:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.369+0000 7f0c13000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0c0c160ee0 con 0x7f0c0c145b00 2024-04-17T08:53:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:42766/0 (socket says 172.21.15.67:42766) 2024-04-17T08:53:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11c00640 1 -- 172.21.15.67:0/1749371446 learned_addr learned my addr 172.21.15.67:0/1749371446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c12600640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0c0c0b4ac0 0x7f0c0c0b3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:53848/0 (socket says 172.21.15.67:53848) 2024-04-17T08:53:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11200640 1 --2- 172.21.15.67:0/1749371446 >> v2:172.21.15.187:3300/0 conn(0x7f0c0c146060 0x7f0c0c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11c00640 1 -- 172.21.15.67:0/1749371446 >> v2:172.21.15.187:3300/0 conn(0x7f0c0c146060 msgr2=0x7f0c0c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.370+0000 7f0c11c00640 1 --2- 172.21.15.67:0/1749371446 >> v2:172.21.15.187:3300/0 conn(0x7f0c0c146060 0x7f0c0c15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.371+0000 7f0c11c00640 1 -- 172.21.15.67:0/1749371446 >> v1:172.21.15.187:6789/0 conn(0x7f0c0c0b4ac0 legacy=0x7f0c0c0b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.371+0000 7f0c11c00640 1 -- 172.21.15.67:0/1749371446 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c1611c0 con 0x7f0c0c145b00 2024-04-17T08:53:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.373+0000 7f0c11c00640 1 --2- 172.21.15.67:0/1749371446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009870 tx=0x7f0c0002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b52bc7a5c5ddaa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.373+0000 7f0c0be00640 1 -- 172.21.15.67:0/1749371446 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00035020 con 0x7f0c0c145b00 2024-04-17T08:53:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.373+0000 7f0c0be00640 1 -- 172.21.15.67:0/1749371446 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c00031640 con 0x7f0c0c145b00 2024-04-17T08:53:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.373+0000 7f0c0be00640 1 -- 172.21.15.67:0/1749371446 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0002d040 con 0x7f0c0c145b00 2024-04-17T08:53:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 -- 172.21.15.67:0/1749371446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 msgr2=0x7f0c0c160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 --2- 172.21.15.67:0/1749371446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009870 tx=0x7f0c0002b8f0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 -- 172.21.15.67:0/1749371446 shutdown_connections 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 --2- 172.21.15.67:0/1749371446 >> v2:172.21.15.187:3300/0 conn(0x7f0c0c146060 0x7f0c0c15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 --2- 172.21.15.67:0/1749371446 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c145b00 0x7f0c0c160340 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 -- 172.21.15.67:0/1749371446 >> 172.21.15.67:0/1749371446 conn(0x7f0c0c0b23c0 msgr2=0x7f0c0c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 -- 172.21.15.67:0/1749371446 shutdown_connections 2024-04-17T08:53:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.374+0000 7f0c13000640 1 -- 172.21.15.67:0/1749371446 wait complete. 2024-04-17T08:53:09.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.376+0000 7f0c13000640 1 Processor -- start 2024-04-17T08:53:09.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.376+0000 7f0c13000640 1 -- start start 2024-04-17T08:53:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c0b4ac0 0x7f0c0c13fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c146060 0x7f0c0c1e6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c15d2c0 con 0x7f0c0c145b00 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c0cfba0 con 0x7f0c0c0b4ac0 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c13000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0c0c0b4f30 con 0x7f0c0c146060 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c11200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c11200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47402/0 (socket says 172.21.15.67:47402) 2024-04-17T08:53:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.377+0000 7f0c11200640 1 -- 172.21.15.67:0/1202567740 learned_addr learned my addr 172.21.15.67:0/1202567740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 -- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c146060 msgr2=0x7f0c0c1e6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c146060 0x7f0c0c1e6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 -- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c0b4ac0 msgr2=0x7f0c0c13fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11c00640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c0b4ac0 0x7f0c0c13fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c0b4ac0 0x7f0c0c13fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 -- 172.21.15.67:0/1202567740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c1406e0 con 0x7f0c0c145b00 2024-04-17T08:53:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c11200640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e8a0 tx=0x7f0bf800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8017070 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf800c940 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf800cd30 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.378+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0c1409c0 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.379+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0c13b750 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.380+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0c068a10 con 0x7f0c0c145b00 2024-04-17T08:53:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.381+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 153) ==== 90751+0+0 (secure 0 0 0) 0x7f0bf801a070 con 0x7f0c0c145b00 2024-04-17T08:53:09.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.381+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f0bf808bc70 con 0x7f0c0c145b00 2024-04-17T08:53:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.386+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0bf805a080 con 0x7f0c0c145b00 2024-04-17T08:53:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.531+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 154) ==== 90878+0+0 (secure 0 0 0) 0x7f0bf80556d0 con 0x7f0c0c145b00 2024-04-17T08:53:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.532+0000 7f0c09600640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 0x7f0bec0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.536+0000 7f0c11c00640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 0x7f0bec0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.542+0000 7f0c11c00640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 0x7f0bec0745a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c00036000 tx=0x7f0c00008030 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.589+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0c0c068610 con 0x7f0c0c145b00 2024-04-17T08:53:09.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.590+0000 7f0c09600640 1 -- 172.21.15.67:0/1202567740 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v274) ==== 100+0+11 (secure 0 0 0) 0x7f0bf8075b80 con 0x7f0c0c145b00 2024-04-17T08:53:09.589 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.592+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 msgr2=0x7f0bec0745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.592+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 0x7f0bec0745a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c00036000 tx=0x7f0c00008030 comp rx=0 tx=0).stop 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.592+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 msgr2=0x7f0c0c1e6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.592+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800e8a0 tx=0x7f0bf800ed60 comp rx=0 tx=0).stop 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 shutdown_connections 2024-04-17T08:53:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0bec0721b0 0x7f0bec0745a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0c0c146060 0x7f0c0c1e6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c0c145b00 0x7f0c0c1e6110 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 --2- 172.21.15.67:0/1202567740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0c0c0b4ac0 0x7f0c0c13fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 >> 172.21.15.67:0/1202567740 conn(0x7f0c0c0b23c0 msgr2=0x7f0c0c074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 shutdown_connections 2024-04-17T08:53:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.593+0000 7f0c13000640 1 -- 172.21.15.67:0/1202567740 wait complete. 2024-04-17T08:53:09.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.825+0000 7fafcfa00640 1 Processor -- start 2024-04-17T08:53:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.825+0000 7fafcfa00640 1 -- start start 2024-04-17T08:53:09.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcfa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafc806f670 0x7fafc806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcfa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fafc806f030 0x7fafc80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcfa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc806ff90 con 0x7fafc80cd020 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcfa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fafc80ceb00 con 0x7fafc806f670 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcfa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fafc80cede0 con 0x7fafc806f030 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafc806f670 0x7fafc806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafc806f670 0x7fafc806fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47406/0 (socket says 172.21.15.67:47406) 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcd000640 1 -- 172.21.15.67:0/2239264888 learned_addr learned my addr 172.21.15.67:0/2239264888 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.827+0000 7fafcda00640 1 --2- 172.21.15.67:0/2239264888 >> v2:172.21.15.187:3300/0 conn(0x7fafc806f030 0x7fafc80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.828+0000 7fafcd000640 1 -- 172.21.15.67:0/2239264888 >> v2:172.21.15.187:3300/0 conn(0x7fafc806f030 msgr2=0x7fafc80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:09.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.828+0000 7fafcd000640 1 --2- 172.21.15.67:0/2239264888 >> v2:172.21.15.187:3300/0 conn(0x7fafc806f030 0x7fafc80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.828+0000 7fafcd000640 1 -- 172.21.15.67:0/2239264888 >> v1:172.21.15.187:6789/0 conn(0x7fafc80cd020 legacy=0x7fafc8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:09.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.828+0000 7fafcd000640 1 -- 172.21.15.67:0/2239264888 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc80cf0c0 con 0x7fafc806f670 2024-04-17T08:53:09.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.829+0000 7fafcd000640 1 --2- 172.21.15.67:0/2239264888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f670 0x7fafc806fa50 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fafc80cdf80 tx=0x7fafbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07365548e384a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.830+0000 7fafc7400640 1 -- 172.21.15.67:0/2239264888 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc035020 con 0x7fafc806f670 2024-04-17T08:53:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.830+0000 7fafc7400640 1 -- 172.21.15.67:0/2239264888 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafbc031870 con 0x7fafc806f670 2024-04-17T08:53:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.830+0000 7fafc7400640 1 -- 172.21.15.67:0/2239264888 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc031c40 con 0x7fafc806f670 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.830+0000 7fafcfa00640 1 -- 172.21.15.67:0/2239264888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f670 msgr2=0x7fafc806fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.830+0000 7fafcfa00640 1 --2- 172.21.15.67:0/2239264888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f670 0x7fafc806fa50 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fafc80cdf80 tx=0x7fafbc02b910 comp rx=0 tx=0).stop 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 -- 172.21.15.67:0/2239264888 shutdown_connections 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 --2- 172.21.15.67:0/2239264888 >> v2:172.21.15.187:3300/0 conn(0x7fafc806f030 0x7fafc80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 --2- 172.21.15.67:0/2239264888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f670 0x7fafc806fa50 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 -- 172.21.15.67:0/2239264888 >> 172.21.15.67:0/2239264888 conn(0x7fafc806bf50 msgr2=0x7fafc80b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 -- 172.21.15.67:0/2239264888 shutdown_connections 2024-04-17T08:53:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.831+0000 7fafcfa00640 1 -- 172.21.15.67:0/2239264888 wait complete. 2024-04-17T08:53:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.832+0000 7fafcfa00640 1 Processor -- start 2024-04-17T08:53:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.832+0000 7fafcfa00640 1 -- start start 2024-04-17T08:53:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafc806f670 0x7fafc81f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fafc80cd020 0x7fafc80b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fafc80ce720 con 0x7fafc806f030 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fafc80cf680 con 0x7fafc806f670 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.833+0000 7fafcfa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fafc80ce8a0 con 0x7fafc80cd020 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47414/0 (socket says 172.21.15.67:47414) 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 -- 172.21.15.67:0/188076276 learned_addr learned my addr 172.21.15.67:0/188076276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 -- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fafc80cd020 msgr2=0x7fafc80b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fafc80cd020 0x7fafc80b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 -- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafc806f670 msgr2=0x7fafc81f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafc806f670 0x7fafc81f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 -- 172.21.15.67:0/188076276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc81f3420 con 0x7fafc806f030 2024-04-17T08:53:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.834+0000 7fafcd000640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000c00 tx=0x7fafbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.835+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc040070 con 0x7fafc806f030 2024-04-17T08:53:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.835+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafbc030d00 con 0x7fafc806f030 2024-04-17T08:53:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.835+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc035040 con 0x7fafc806f030 2024-04-17T08:53:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.835+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc80b4a90 con 0x7fafc806f030 2024-04-17T08:53:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.835+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc80b3790 con 0x7fafc806f030 2024-04-17T08:53:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.836+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc8068b90 con 0x7fafc806f030 2024-04-17T08:53:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.837+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 154) ==== 90878+0+0 (secure 0 0 0) 0x7fafbc04b060 con 0x7fafc806f030 2024-04-17T08:53:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.838+0000 7fafc4c00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 0x7fafa806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.838+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7fafbc0ad980 con 0x7fafc806f030 2024-04-17T08:53:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.839+0000 7fafc7e00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 0x7fafa806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:09.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.840+0000 7fafc7e00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 0x7fafa806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006fd0 tx=0x7fafb4006bc0 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:09.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:09.843+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fafc81f3420 con 0x7fafc806f030 2024-04-17T08:53:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.095+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafc80af980 con 0x7fafc806f030 2024-04-17T08:53:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.096+0000 7fafc4c00640 1 -- 172.21.15.67:0/188076276 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7fafbc07bd70 con 0x7fafc806f030 2024-04-17T08:53:10.096 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:10.096 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:53:10.097 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.099+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 msgr2=0x7fafa806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.099+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 0x7fafa806ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006fd0 tx=0x7fafb4006bc0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.099+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 msgr2=0x7fafc81ef980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.099+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000c00 tx=0x7fafbc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 shutdown_connections 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fafa806c740 0x7fafa806ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fafc80cd020 0x7fafc80b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fafc806f670 0x7fafc81f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 --2- 172.21.15.67:0/188076276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc806f030 0x7fafc81ef980 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 >> 172.21.15.67:0/188076276 conn(0x7fafc806bf50 msgr2=0x7fafc806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 shutdown_connections 2024-04-17T08:53:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.100+0000 7fafcfa00640 1 -- 172.21.15.67:0/188076276 wait complete. 2024-04-17T08:53:10.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:10.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.336+0000 7f982ce00640 1 Processor -- start 2024-04-17T08:53:10.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.336+0000 7f982ce00640 1 -- start start 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f982ce00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f98280b6d00 0x7f98280cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f982ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98280cf910 0x7f98280b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f982ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98280d2d30 con 0x7f982815ec60 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f982ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f98280d49c0 con 0x7f98280cf910 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f982ce00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f98280d3660 con 0x7f98280b6d00 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f9827400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f982815ec60 0x7f98280b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42600/0 (socket says 172.21.15.67:42600) 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.337+0000 7f9827400640 1 -- 172.21.15.67:0/2100537194 learned_addr learned my addr 172.21.15.67:0/2100537194 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9827e00640 1 --2- 172.21.15.67:0/2100537194 >> v2:172.21.15.67:3300/0 conn(0x7f98280cf910 0x7f98280b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9826a00640 1 --2- 172.21.15.67:0/2100537194 >> v2:172.21.15.187:3300/0 conn(0x7f98280b6d00 0x7f98280cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9827e00640 1 -- 172.21.15.67:0/2100537194 >> v2:172.21.15.187:3300/0 conn(0x7f98280b6d00 msgr2=0x7f98280cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:10.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9827e00640 1 --2- 172.21.15.67:0/2100537194 >> v2:172.21.15.187:3300/0 conn(0x7f98280b6d00 0x7f98280cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9827e00640 1 -- 172.21.15.67:0/2100537194 >> v1:172.21.15.67:6789/0 conn(0x7f982815ec60 legacy=0x7f98280b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.338+0000 7f9827e00640 1 -- 172.21.15.67:0/2100537194 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98280d3010 con 0x7f98280cf910 2024-04-17T08:53:10.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.339+0000 7f9827e00640 1 --2- 172.21.15.67:0/2100537194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280cf910 0x7f98280b7b70 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f98280ba590 tx=0x7f982002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5ba2966cb3c68f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.339+0000 7f9826000640 1 -- 172.21.15.67:0/2100537194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820039070 con 0x7f98280cf910 2024-04-17T08:53:10.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f9826000640 1 -- 172.21.15.67:0/2100537194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982002cc80 con 0x7f98280cf910 2024-04-17T08:53:10.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f9826000640 1 -- 172.21.15.67:0/2100537194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820034660 con 0x7f98280cf910 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 -- 172.21.15.67:0/2100537194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280cf910 msgr2=0x7f98280b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 --2- 172.21.15.67:0/2100537194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280cf910 0x7f98280b7b70 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f98280ba590 tx=0x7f982002bed0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 -- 172.21.15.67:0/2100537194 shutdown_connections 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 --2- 172.21.15.67:0/2100537194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280cf910 0x7f98280b7b70 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 --2- 172.21.15.67:0/2100537194 >> v2:172.21.15.187:3300/0 conn(0x7f98280b6d00 0x7f98280cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 -- 172.21.15.67:0/2100537194 >> 172.21.15.67:0/2100537194 conn(0x7f98280d15f0 msgr2=0x7f98280d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 -- 172.21.15.67:0/2100537194 shutdown_connections 2024-04-17T08:53:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.340+0000 7f982ce00640 1 -- 172.21.15.67:0/2100537194 wait complete. 2024-04-17T08:53:10.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.342+0000 7f982ce00640 1 Processor -- start 2024-04-17T08:53:10.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.342+0000 7f982ce00640 1 -- start start 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98280cf910 0x7f98281556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f982815ec60 0x7f982815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f982815f040 con 0x7f98280b6d00 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f98280cfdc0 con 0x7f98280cf910 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.343+0000 7f982ce00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f98280d00c0 con 0x7f982815ec60 2024-04-17T08:53:10.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47432/0 (socket says 172.21.15.67:47432) 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 -- 172.21.15.67:0/146995572 learned_addr learned my addr 172.21.15.67:0/146995572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9826a00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98280cf910 0x7f98281556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827e00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f982815ec60 0x7f982815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 -- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f982815ec60 msgr2=0x7f982815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f982815ec60 0x7f982815aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 -- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98280cf910 msgr2=0x7f98281556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98280cf910 0x7f98281556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 -- 172.21.15.67:0/146995572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9828156060 con 0x7f98280b6d00 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.344+0000 7f9827400640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f981800c3c0 tx=0x7f981800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.345+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818017070 con 0x7f98280b6d00 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.345+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9818005ce0 con 0x7f98280b6d00 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.345+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9828156340 con 0x7f98280b6d00 2024-04-17T08:53:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.345+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180072f0 con 0x7f98280b6d00 2024-04-17T08:53:10.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.345+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9828156900 con 0x7f98280b6d00 2024-04-17T08:53:10.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.346+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9828068a10 con 0x7f98280b6d00 2024-04-17T08:53:10.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.347+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 154) ==== 90878+0+0 (secure 0 0 0) 0x7f981801a070 con 0x7f98280b6d00 2024-04-17T08:53:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.348+0000 7f981f400640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 0x7f980406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.348+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f981808ae80 con 0x7f98280b6d00 2024-04-17T08:53:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.349+0000 7f9826a00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 0x7f980406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.350+0000 7f9826a00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 0x7f980406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9810006fd0 tx=0x7f9810008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.352+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9818059270 con 0x7f98280b6d00 2024-04-17T08:53:10.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.545+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 155) ==== 91086+0+0 (secure 0 0 0) 0x7f98180551e0 con 0x7f98280b6d00 2024-04-17T08:53:10.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.565+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98280ba360 con 0x7f98280b6d00 2024-04-17T08:53:10.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.569+0000 7f981f400640 1 -- 172.21.15.67:0/146995572 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+17892 (secure 0 0 0) 0x7f9818079050 con 0x7f98280b6d00 2024-04-17T08:53:10.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:10.569 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":274,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:08.504926+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/2991348352":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3653376109":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3444724328":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3175082800":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/1234602879":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3978763551":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3094572004":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2893316565":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2486101166":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3800011905":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6832/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3721653371":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/227074939":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/456496732":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/1663113056":"2024-04-18T08:53:08.504849+0000","172.21.15.187:6832/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6833/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3973592351":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:10.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 msgr2=0x7f980406ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 0x7f980406ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9810006fd0 tx=0x7f9810008040 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 msgr2=0x7f9828157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f981800c3c0 tx=0x7f981800c880 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 shutdown_connections 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f980406c740 0x7f980406ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f982815ec60 0x7f982815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98280cf910 0x7f98281556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 --2- 172.21.15.67:0/146995572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98280b6d00 0x7f9828157100 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 >> 172.21.15.67:0/146995572 conn(0x7f98280d15f0 msgr2=0x7f982815e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 shutdown_connections 2024-04-17T08:53:10.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.572+0000 7f982ce00640 1 -- 172.21.15.67:0/146995572 wait complete. 2024-04-17T08:53:10.590 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:53:10.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.810+0000 7ff8c6400640 1 Processor -- start 2024-04-17T08:53:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.810+0000 7ff8c6400640 1 -- start start 2024-04-17T08:53:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c6400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8c0160280 0x7ff8c006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c6400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8c0141ed0 0x7ff8c0142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c6400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8c00cd4a0 con 0x7ff8c006b8c0 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c6400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8c015ce30 con 0x7ff8c0141ed0 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c6400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff8c01607f0 con 0x7ff8c0160280 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c5000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8c006b8c0 0x7ff8c00cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42604/0 (socket says 172.21.15.67:42604) 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c5000640 1 -- 172.21.15.67:0/2235817629 learned_addr learned my addr 172.21.15.67:0/2235817629 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.812+0000 7ff8c5a00640 1 --2- 172.21.15.67:0/2235817629 >> v2:172.21.15.67:3300/0 conn(0x7ff8c0141ed0 0x7ff8c0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8bfe00640 1 --2- 172.21.15.67:0/2235817629 >> v2:172.21.15.187:3300/0 conn(0x7ff8c0160280 0x7ff8c006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8bf400640 1 -- 172.21.15.67:0/2235817629 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593071328 0 0) 0x7ff8c00cd4a0 con 0x7ff8c006b8c0 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8bf400640 1 -- 172.21.15.67:0/2235817629 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8a8003520 con 0x7ff8c006b8c0 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8c5a00640 1 -- 172.21.15.67:0/2235817629 >> v2:172.21.15.187:3300/0 conn(0x7ff8c0160280 msgr2=0x7ff8c006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8c5a00640 1 --2- 172.21.15.67:0/2235817629 >> v2:172.21.15.187:3300/0 conn(0x7ff8c0160280 0x7ff8c006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8c5a00640 1 -- 172.21.15.67:0/2235817629 >> v1:172.21.15.67:6789/0 conn(0x7ff8c006b8c0 legacy=0x7ff8c00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.813+0000 7ff8c5a00640 1 -- 172.21.15.67:0/2235817629 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c0160ad0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.814+0000 7ff8c5a00640 1 --2- 172.21.15.67:0/2235817629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c0142360 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7ff8c00aff50 tx=0x7ff8b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeccfdf56d85fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.814+0000 7ff8bf400640 1 -- 172.21.15.67:0/2235817629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8039070 con 0x7ff8c0141ed0 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.814+0000 7ff8bf400640 1 -- 172.21.15.67:0/2235817629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b802cc80 con 0x7ff8c0141ed0 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.814+0000 7ff8bf400640 1 -- 172.21.15.67:0/2235817629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8034640 con 0x7ff8c0141ed0 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 -- 172.21.15.67:0/2235817629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 msgr2=0x7ff8c0142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2235817629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c0142360 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7ff8c00aff50 tx=0x7ff8b802bed0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 -- 172.21.15.67:0/2235817629 shutdown_connections 2024-04-17T08:53:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2235817629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c0142360 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2235817629 >> v2:172.21.15.187:3300/0 conn(0x7ff8c0160280 0x7ff8c006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 -- 172.21.15.67:0/2235817629 >> 172.21.15.67:0/2235817629 conn(0x7ff8c0067d10 msgr2=0x7ff8c00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 -- 172.21.15.67:0/2235817629 shutdown_connections 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.815+0000 7ff8c6400640 1 -- 172.21.15.67:0/2235817629 wait complete. 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.817+0000 7ff8c6400640 1 Processor -- start 2024-04-17T08:53:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.817+0000 7ff8c6400640 1 -- start start 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8c006b8c0 0x7ff8c013ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8c0160280 0x7ff8c0142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0161090 con 0x7ff8c0141ed0 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c006c1e0 con 0x7ff8c006b8c0 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8c6400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8c013f9b0 con 0x7ff8c0160280 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47438/0 (socket says 172.21.15.67:47438) 2024-04-17T08:53:10.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.818+0000 7ff8bfe00640 1 -- 172.21.15.67:0/2950271113 learned_addr learned my addr 172.21.15.67:0/2950271113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8c5000640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8c006b8c0 0x7ff8c013ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 -- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8c0160280 msgr2=0x7ff8c0142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8c0160280 0x7ff8c0142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 -- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8c006b8c0 msgr2=0x7ff8c013ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8c006b8c0 0x7ff8c013ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 -- 172.21.15.67:0/2950271113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c01f7cf0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bfe00640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d5f0 tx=0x7ff8ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0125f0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac019070 con 0x7ff8c0141ed0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0129d0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c01f7fd0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.819+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c01f8590 con 0x7ff8c0141ed0 2024-04-17T08:53:10.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.821+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8c0160040 con 0x7ff8c0141ed0 2024-04-17T08:53:10.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.822+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 155) ==== 91086+0+0 (secure 0 0 0) 0x7ff8ac012b70 con 0x7ff8c0141ed0 2024-04-17T08:53:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.823+0000 7ff8bcc00640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 0x7ff8a006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.823+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7ff8ac08c3b0 con 0x7ff8c0141ed0 2024-04-17T08:53:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.823+0000 7ff8c5000640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 0x7ff8a006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:10.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.824+0000 7ff8c5000640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 0x7ff8a006ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4000f60 tx=0x7ff8b4009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:10.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:10.827+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff8c0160040 con 0x7ff8c0141ed0 2024-04-17T08:53:11.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.071+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8c00b59b0 con 0x7ff8c0141ed0 2024-04-17T08:53:11.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.072+0000 7ff8bcc00640 1 -- 172.21.15.67:0/2950271113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7ff8ac05a400 con 0x7ff8c0141ed0 2024-04-17T08:53:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:11.072 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:53:11.072 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:53:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.074+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 msgr2=0x7ff8a006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 0x7ff8a006ee40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4000f60 tx=0x7ff8b4009040 comp rx=0 tx=0).stop 2024-04-17T08:53:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 msgr2=0x7ff8c01f5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d5f0 tx=0x7ff8ac00dab0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 shutdown_connections 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff8a006c980 0x7ff8a006ee40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8c0160280 0x7ff8c0142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8c0141ed0 0x7ff8c01f5450 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 --2- 172.21.15.67:0/2950271113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8c006b8c0 0x7ff8c013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 >> 172.21.15.67:0/2950271113 conn(0x7ff8c0067d10 msgr2=0x7ff8c0074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 shutdown_connections 2024-04-17T08:53:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.075+0000 7ff8c6400640 1 -- 172.21.15.67:0/2950271113 wait complete. 2024-04-17T08:53:11.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:11.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.309+0000 7f260a000640 1 Processor -- start 2024-04-17T08:53:11.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.310+0000 7f260a000640 1 -- start start 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.311+0000 7f260a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26040cfae0 0x7f2604169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.311+0000 7f260a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26040d2d30 con 0x7f260415ec60 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.311+0000 7f260a000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26040d3010 con 0x7f26040b6d00 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.311+0000 7f260a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f26040d49c0 con 0x7f26040cfae0 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.311+0000 7f2609600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26040cfae0 0x7f2604169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.312+0000 7f2609600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26040cfae0 0x7f2604169810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47442/0 (socket says 172.21.15.67:47442) 2024-04-17T08:53:11.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.312+0000 7f2609600640 1 -- 172.21.15.67:0/1335480287 learned_addr learned my addr 172.21.15.67:0/1335480287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.312+0000 7f2609600640 1 -- 172.21.15.67:0/1335480287 >> v1:172.21.15.187:6789/0 conn(0x7f26040b6d00 legacy=0x7f26040cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.312+0000 7f2609600640 1 -- 172.21.15.67:0/1335480287 >> v1:172.21.15.67:6789/0 conn(0x7f260415ec60 legacy=0x7f26040b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.312+0000 7f2609600640 1 -- 172.21.15.67:0/1335480287 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26040d32f0 con 0x7f26040cfae0 2024-04-17T08:53:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.313+0000 7f2609600640 1 --2- 172.21.15.67:0/1335480287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26040cfae0 0x7f2604169810 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009e50 tx=0x7f25fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b940efdfc790fd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.314+0000 7f2603e00640 1 -- 172.21.15.67:0/1335480287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc02cb00 con 0x7f26040cfae0 2024-04-17T08:53:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.314+0000 7f2603e00640 1 -- 172.21.15.67:0/1335480287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25fc02cca0 con 0x7f26040cfae0 2024-04-17T08:53:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.314+0000 7f2603e00640 1 -- 172.21.15.67:0/1335480287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc034610 con 0x7f26040cfae0 2024-04-17T08:53:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.314+0000 7f260a000640 1 -- 172.21.15.67:0/1335480287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26040cfae0 msgr2=0x7f2604169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.314+0000 7f260a000640 1 --2- 172.21.15.67:0/1335480287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26040cfae0 0x7f2604169810 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009e50 tx=0x7f25fc02bf60 comp rx=0 tx=0).stop 2024-04-17T08:53:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.315+0000 7f260a000640 1 -- 172.21.15.67:0/1335480287 shutdown_connections 2024-04-17T08:53:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.315+0000 7f260a000640 1 --2- 172.21.15.67:0/1335480287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26040cfae0 0x7f2604169810 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.315+0000 7f260a000640 1 -- 172.21.15.67:0/1335480287 >> 172.21.15.67:0/1335480287 conn(0x7f26040d15f0 msgr2=0x7f26040d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.315+0000 7f260a000640 1 -- 172.21.15.67:0/1335480287 shutdown_connections 2024-04-17T08:53:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.315+0000 7f260a000640 1 -- 172.21.15.67:0/1335480287 wait complete. 2024-04-17T08:53:11.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.316+0000 7f260a000640 1 Processor -- start 2024-04-17T08:53:11.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.317+0000 7f260a000640 1 -- start start 2024-04-17T08:53:11.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26040b6d00 0x7f260415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26040cfae0 0x7f26040ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f26040b8b80 con 0x7f260415ec60 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f26040d3660 con 0x7f26040cfae0 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f260a000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26040d00c0 con 0x7f26040b6d00 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47450/0 (socket says 172.21.15.67:47450) 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 -- 172.21.15.67:0/4220397937 learned_addr learned my addr 172.21.15.67:0/4220397937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2608c00640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26040b6d00 0x7f260415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f25fbe00640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26040cfae0 0x7f26040ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 -- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26040b6d00 msgr2=0x7f260415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26040b6d00 0x7f260415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 -- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26040cfae0 msgr2=0x7f26040ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26040cfae0 0x7f26040ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.318+0000 7f2609600640 1 -- 172.21.15.67:0/4220397937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26040abf40 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f2609600640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009f70 tx=0x7f25fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc043020 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25fc02ce50 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc009510 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26040ac220 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.319+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26040ac7e0 con 0x7f260415ec60 2024-04-17T08:53:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.320+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2604068a10 con 0x7f260415ec60 2024-04-17T08:53:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.321+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 155) ==== 91086+0+0 (secure 0 0 0) 0x7f25fc046020 con 0x7f260415ec60 2024-04-17T08:53:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.322+0000 7f2601600640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 0x7f25e006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.322+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f25fc0ae730 con 0x7f260415ec60 2024-04-17T08:53:11.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.323+0000 7f2608c00640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 0x7f25e006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.324+0000 7f2608c00640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 0x7f25e006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f4007fd0 tx=0x7f25f4009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.327+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f25fc04d030 con 0x7f260415ec60 2024-04-17T08:53:11.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.576+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26040d5670 con 0x7f260415ec60 2024-04-17T08:53:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.577+0000 7f2601600640 1 -- 172.21.15.67:0/4220397937 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5220 (secure 0 0 0) 0x7f25fc03c350 con 0x7f260415ec60 2024-04-17T08:53:11.576 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:11.576 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:52:45.263955+0000","modified":"2024-04-17T08:52:48.338438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12552},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12552":{"gid":12552,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1995843637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1995843637},{"type":"v1","addr":"172.21.15.67:6835","nonce":1995843637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12552,"qdb_cluster":[12552]},"id":17}]} 2024-04-17T08:53:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 143 2024-04-17T08:53:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 msgr2=0x7f25e006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 0x7f25e006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f4007fd0 tx=0x7f25f4009040 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 msgr2=0x7f26040cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009f70 tx=0x7f25fc004a40 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 shutdown_connections 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f25e006c980 0x7f25e006ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f260415ec60 0x7f26040cd230 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26040cfae0 0x7f26040ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 --2- 172.21.15.67:0/4220397937 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26040b6d00 0x7f260415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 >> 172.21.15.67:0/4220397937 conn(0x7f26040d15f0 msgr2=0x7f260415e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 shutdown_connections 2024-04-17T08:53:11.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.580+0000 7f260a000640 1 -- 172.21.15.67:0/4220397937 wait complete. 2024-04-17T08:53:11.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:11.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.812+0000 7f4a74a00640 1 Processor -- start 2024-04-17T08:53:11.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.812+0000 7f4a74a00640 1 -- start start 2024-04-17T08:53:11.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a74a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a7015c5a0 0x7f4a7015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a74a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7015f440 con 0x7f4a700b46c0 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a74a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7015f720 con 0x7f4a700b4210 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a74a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a7015fa00 con 0x7f4a7015c5a0 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a7015c5a0 0x7f4a7015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a7015c5a0 0x7f4a7015ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47460/0 (socket says 172.21.15.67:47460) 2024-04-17T08:53:11.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.814+0000 7f4a6e800640 1 -- 172.21.15.67:0/3633530312 learned_addr learned my addr 172.21.15.67:0/3633530312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.815+0000 7f4a6e800640 1 -- 172.21.15.67:0/3633530312 >> v1:172.21.15.187:6789/0 conn(0x7f4a700b4210 legacy=0x7f4a700af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.815+0000 7f4a6e800640 1 -- 172.21.15.67:0/3633530312 >> v1:172.21.15.67:6789/0 conn(0x7f4a700b46c0 legacy=0x7f4a700ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.815+0000 7f4a6e800640 1 -- 172.21.15.67:0/3633530312 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7015fce0 con 0x7f4a7015c5a0 2024-04-17T08:53:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.815+0000 7f4a6de00640 1 -- 172.21.15.67:0/3633530312 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456333480 0 0) 0x7f4a7015f440 con 0x7f4a700b46c0 2024-04-17T08:53:11.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.816+0000 7f4a6e800640 1 --2- 172.21.15.67:0/3633530312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015c5a0 0x7f4a7015ee60 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009870 tx=0x7f4a6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9392439e81185a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.817+0000 7f4a6de00640 1 -- 172.21.15.67:0/3633530312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a60035020 con 0x7f4a7015c5a0 2024-04-17T08:53:11.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.817+0000 7f4a6de00640 1 -- 172.21.15.67:0/3633530312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a60031870 con 0x7f4a7015c5a0 2024-04-17T08:53:11.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.817+0000 7f4a6de00640 1 -- 172.21.15.67:0/3633530312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a60031c40 con 0x7f4a7015c5a0 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.817+0000 7f4a74a00640 1 -- 172.21.15.67:0/3633530312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015c5a0 msgr2=0x7f4a7015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.817+0000 7f4a74a00640 1 --2- 172.21.15.67:0/3633530312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015c5a0 0x7f4a7015ee60 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009870 tx=0x7f4a6002b910 comp rx=0 tx=0).stop 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.818+0000 7f4a74a00640 1 -- 172.21.15.67:0/3633530312 shutdown_connections 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.818+0000 7f4a74a00640 1 --2- 172.21.15.67:0/3633530312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015c5a0 0x7f4a7015ee60 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.818+0000 7f4a74a00640 1 -- 172.21.15.67:0/3633530312 >> 172.21.15.67:0/3633530312 conn(0x7f4a70067d10 msgr2=0x7f4a700b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.818+0000 7f4a74a00640 1 -- 172.21.15.67:0/3633530312 shutdown_connections 2024-04-17T08:53:11.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.818+0000 7f4a74a00640 1 -- 172.21.15.67:0/3633530312 wait complete. 2024-04-17T08:53:11.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.819+0000 7f4a74a00640 1 Processor -- start 2024-04-17T08:53:11.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.820+0000 7f4a74a00640 1 -- start start 2024-04-17T08:53:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.820+0000 7f4a74a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 0x7f4a701e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.820+0000 7f4a74a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a7015c5a0 0x7f4a70145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a74a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a74a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7006e7f0 con 0x7f4a7015a820 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a74a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7006e670 con 0x7f4a700b4210 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a74a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a700afb50 con 0x7f4a7015c5a0 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47464/0 (socket says 172.21.15.67:47464) 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 -- 172.21.15.67:0/1653776681 learned_addr learned my addr 172.21.15.67:0/1653776681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6e800640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 0x7f4a701e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a67e00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a7015c5a0 0x7f4a70145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 -- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a7015c5a0 msgr2=0x7f4a70145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a7015c5a0 0x7f4a70145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 -- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 msgr2=0x7f4a701e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 0x7f4a701e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6f200640 1 -- 172.21.15.67:0/1653776681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a701f5b70 con 0x7f4a7015a820 2024-04-17T08:53:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.821+0000 7f4a6e800640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 0x7f4a701e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a6f200640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800c730 tx=0x7f4a6800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68003da0 con 0x7f4a7015a820 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6800ce40 con 0x7f4a7015a820 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a680052b0 con 0x7f4a7015a820 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a701f5e50 con 0x7f4a7015a820 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.822+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a701f6410 con 0x7f4a7015a820 2024-04-17T08:53:11.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.823+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a700b5630 con 0x7f4a7015a820 2024-04-17T08:53:11.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.825+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 155) ==== 91086+0+0 (secure 0 0 0) 0x7f4a68005450 con 0x7f4a7015a820 2024-04-17T08:53:11.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.826+0000 7f4a66a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 0x7f4a4c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:11.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.826+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f4a6808bea0 con 0x7f4a7015a820 2024-04-17T08:53:11.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.826+0000 7f4a6e800640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 0x7f4a4c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:11.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.828+0000 7f4a6e800640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 0x7f4a4c06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a60008030 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:11.830+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4a700b5630 con 0x7f4a7015a820 2024-04-17T08:53:12.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.035+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a70068a10 con 0x7f4a7015a820 2024-04-17T08:53:12.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.036+0000 7f4a66a00640 1 -- 172.21.15.67:0/1653776681 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+17892 (secure 0 0 0) 0x7f4a68059ef0 con 0x7f4a7015a820 2024-04-17T08:53:12.036 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:12.036 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":274,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:08.504926+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:52:43.162800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-04-17T08:52:44.240860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6833/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/2991348352":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3653376109":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3444724328":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3175082800":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/1234602879":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3978763551":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3094572004":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2893316565":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2486101166":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3800011905":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6832/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3721653371":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/227074939":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/456496732":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/1663113056":"2024-04-18T08:53:08.504849+0000","172.21.15.187:6832/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6833/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3973592351":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:12.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 msgr2=0x7f4a4c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 0x7f4a4c06ee70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a60008030 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 msgr2=0x7f4a7015acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800c730 tx=0x7f4a6800cbf0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 shutdown_connections 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4a4c06c9b0 0x7f4a4c06ee70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7015a820 0x7f4a7015acb0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a7015c5a0 0x7f4a70145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 --2- 172.21.15.67:0/1653776681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a700b4210 0x7f4a701e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 >> 172.21.15.67:0/1653776681 conn(0x7f4a70067d10 msgr2=0x7f4a700ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 shutdown_connections 2024-04-17T08:53:12.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.039+0000 7f4a74a00640 1 -- 172.21.15.67:0/1653776681 wait complete. 2024-04-17T08:53:12.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:53:12.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.276+0000 7f5601600640 1 Processor -- start 2024-04-17T08:53:12.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.277+0000 7f5601600640 1 -- start start 2024-04-17T08:53:12.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.278+0000 7f5601600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc15ee20 0x7f55fc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.278+0000 7f5601600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55fc0d3520 0x7f55fc0b8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.278+0000 7f5601600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55fc0d4140 con 0x7f55fc0d18e0 2024-04-17T08:53:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.278+0000 7f5601600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f55fc0b83b0 con 0x7f55fc0d3520 2024-04-17T08:53:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.278+0000 7f5601600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55fc0d1d20 con 0x7f55fc15ee20 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55fc0d3520 0x7f55fc0b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55fc0d3520 0x7f55fc0b8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47480/0 (socket says 172.21.15.67:47480) 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 -- 172.21.15.67:0/3524855106 learned_addr learned my addr 172.21.15.67:0/3524855106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f55fbe00640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc15ee20 0x7f55fc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 -- 172.21.15.67:0/3524855106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc15ee20 msgr2=0x7f55fc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc15ee20 0x7f55fc0d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 -- 172.21.15.67:0/3524855106 >> v1:172.21.15.187:6789/0 conn(0x7f55fc0d18e0 legacy=0x7f55fc0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:12.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.279+0000 7f5600c00640 1 -- 172.21.15.67:0/3524855106 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55fc0d3b80 con 0x7f55fc0d3520 2024-04-17T08:53:12.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.280+0000 7f5600c00640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc0b8840 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009870 tx=0x7f55f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90578bcf05dbb670 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.281+0000 7f55faa00640 1 -- 172.21.15.67:0/3524855106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4035020 con 0x7f55fc0d3520 2024-04-17T08:53:12.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.281+0000 7f55faa00640 1 -- 172.21.15.67:0/3524855106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f4031640 con 0x7f55fc0d3520 2024-04-17T08:53:12.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.281+0000 7f55faa00640 1 -- 172.21.15.67:0/3524855106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f402d040 con 0x7f55fc0d3520 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.281+0000 7f5601600640 1 -- 172.21.15.67:0/3524855106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 msgr2=0x7f55fc0b8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.281+0000 7f5601600640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc0b8840 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009870 tx=0x7f55f402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 -- 172.21.15.67:0/3524855106 shutdown_connections 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc0b8840 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 --2- 172.21.15.67:0/3524855106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc15ee20 0x7f55fc0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 -- 172.21.15.67:0/3524855106 >> 172.21.15.67:0/3524855106 conn(0x7f55fc06d3c0 msgr2=0x7f55fc0b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 -- 172.21.15.67:0/3524855106 shutdown_connections 2024-04-17T08:53:12.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.282+0000 7f5601600640 1 -- 172.21.15.67:0/3524855106 wait complete. 2024-04-17T08:53:12.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.283+0000 7f5601600640 1 Processor -- start 2024-04-17T08:53:12.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.283+0000 7f5601600640 1 -- start start 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc0d18e0 0x7f55fc1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55fc15ee20 0x7f55fc0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55fc15f200 con 0x7f55fc0d3520 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f55fc0b8040 con 0x7f55fc15ee20 2024-04-17T08:53:12.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.284+0000 7f5601600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55fc0b8d80 con 0x7f55fc0d18e0 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47494/0 (socket says 172.21.15.67:47494) 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 -- 172.21.15.67:0/32417171 learned_addr learned my addr 172.21.15.67:0/32417171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 -- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc0d18e0 msgr2=0x7f55fc1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fbe00640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc0d18e0 0x7f55fc1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f5600c00640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55fc15ee20 0x7f55fc0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc0d18e0 0x7f55fc1610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 -- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55fc15ee20 msgr2=0x7f55fc0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55fc15ee20 0x7f55fc0b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 -- 172.21.15.67:0/32417171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55fc0b4180 con 0x7f55fc0d3520 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.285+0000 7f55fb400640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00c7c0 tx=0x7f55ec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.286+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec016020 con 0x7f55fc0d3520 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.286+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55ec00aa50 con 0x7f55fc0d3520 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.286+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec012420 con 0x7f55fc0d3520 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.286+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55fc160490 con 0x7f55fc0d3520 2024-04-17T08:53:12.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.286+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55fc160a50 con 0x7f55fc0d3520 2024-04-17T08:53:12.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.287+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55fc0b8840 con 0x7f55fc0d3520 2024-04-17T08:53:12.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.288+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 155) ==== 91086+0+0 (secure 0 0 0) 0x7f55ec019020 con 0x7f55fc0d3520 2024-04-17T08:53:12.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.289+0000 7f55f3e00640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 0x7f55d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.289+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6597+0+0 (secure 0 0 0) 0x7f55ec08bff0 con 0x7f55fc0d3520 2024-04-17T08:53:12.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.290+0000 7f55fbe00640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 0x7f55d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.291+0000 7f55fbe00640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 0x7f55d806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.293+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f55fc0b8840 con 0x7f55fc0d3520 2024-04-17T08:53:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.528+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f55fc068a10 con 0x7f55fc0d3520 2024-04-17T08:53:12.528 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:53:12.529+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:53:12.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.532+0000 7f55f3e00640 1 -- 172.21.15.67:0/32417171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v144) ==== 161+0+0 (secure 0 0 0) 0x7f55ec020030 con 0x7f55fc0d3520 2024-04-17T08:53:12.532 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.536+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 msgr2=0x7f55d806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.536+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 0x7f55d806ee70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006fd0 tx=0x7f55e4008040 comp rx=0 tx=0).stop 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.536+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 msgr2=0x7f55fc15fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.536+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00c7c0 tx=0x7f55ec00cc80 comp rx=0 tx=0).stop 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 shutdown_connections 2024-04-17T08:53:12.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f55d806c9b0 0x7f55d806ee70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55fc15ee20 0x7f55fc0b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55fc0d3520 0x7f55fc15fd40 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 --2- 172.21.15.67:0/32417171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55fc0d18e0 0x7f55fc1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.537+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 >> 172.21.15.67:0/32417171 conn(0x7f55fc06d3c0 msgr2=0x7f55fc0b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.538+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 shutdown_connections 2024-04-17T08:53:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.538+0000 7f5601600640 1 -- 172.21.15.67:0/32417171 wait complete. 2024-04-17T08:53:12.548 INFO:tasks.ceph.mds.a.smithi067.stderr: -4326> 2024-04-17T08:52:50.701+0000 7f9d33a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:53:12.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:53:12.612 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:53:12.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.760+0000 7fba77a00640 1 Processor -- start 2024-04-17T08:53:12.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.761+0000 7fba77a00640 1 -- start start 2024-04-17T08:53:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.762+0000 7fba77a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba7006daf0 0x7fba700d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.762+0000 7fba77a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba700d1520 con 0x7fba700d1060 2024-04-17T08:53:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.762+0000 7fba77a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba700d1800 con 0x7fba7009b170 2024-04-17T08:53:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.762+0000 7fba77a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fba700d1ae0 con 0x7fba7006daf0 2024-04-17T08:53:12.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.762+0000 7fba75000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba7006daf0 0x7fba700d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.763+0000 7fba75000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba7006daf0 0x7fba700d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47520/0 (socket says 172.21.15.67:47520) 2024-04-17T08:53:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.763+0000 7fba75000640 1 -- 172.21.15.67:0/1301053590 learned_addr learned my addr 172.21.15.67:0/1301053590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.763+0000 7fba75000640 1 -- 172.21.15.67:0/1301053590 >> v1:172.21.15.187:6789/0 conn(0x7fba7009b170 legacy=0x7fba70070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.763+0000 7fba75000640 1 -- 172.21.15.67:0/1301053590 >> v1:172.21.15.67:6789/0 conn(0x7fba700d1060 legacy=0x7fba7009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.763+0000 7fba75000640 1 -- 172.21.15.67:0/1301053590 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba700d1dc0 con 0x7fba7006daf0 2024-04-17T08:53:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.764+0000 7fba75000640 1 --2- 172.21.15.67:0/1301053590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7006daf0 0x7fba700d0b20 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e1adda5b4d5da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.764+0000 7fba6f400640 1 -- 172.21.15.67:0/1301053590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60034070 con 0x7fba7006daf0 2024-04-17T08:53:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba6f400640 1 -- 172.21.15.67:0/1301053590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba60030860 con 0x7fba7006daf0 2024-04-17T08:53:12.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba6f400640 1 -- 172.21.15.67:0/1301053590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60030c30 con 0x7fba7006daf0 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 -- 172.21.15.67:0/1301053590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7006daf0 msgr2=0x7fba700d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 --2- 172.21.15.67:0/1301053590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7006daf0 0x7fba700d0b20 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6002b9a0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 -- 172.21.15.67:0/1301053590 shutdown_connections 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 --2- 172.21.15.67:0/1301053590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba7006daf0 0x7fba700d0b20 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 -- 172.21.15.67:0/1301053590 >> 172.21.15.67:0/1301053590 conn(0x7fba700b41a0 msgr2=0x7fba700b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 -- 172.21.15.67:0/1301053590 shutdown_connections 2024-04-17T08:53:12.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.765+0000 7fba77a00640 1 -- 172.21.15.67:0/1301053590 wait complete. 2024-04-17T08:53:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.767+0000 7fba77a00640 1 Processor -- start 2024-04-17T08:53:12.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.767+0000 7fba77a00640 1 -- start start 2024-04-17T08:53:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba7006daf0 0x7fba700b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 0x7fba701f8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba7006e050 con 0x7fba700d1060 2024-04-17T08:53:12.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba7006ded0 con 0x7fba7009b170 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba77a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba7006ce00 con 0x7fba7006daf0 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba75a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba75a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47530/0 (socket says 172.21.15.67:47530) 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.768+0000 7fba75a00640 1 -- 172.21.15.67:0/1044708172 learned_addr learned my addr 172.21.15.67:0/1044708172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 -- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba7006daf0 msgr2=0x7fba700b7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba6fe00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 0x7fba701f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba7006daf0 0x7fba700b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 -- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 msgr2=0x7fba701f8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 0x7fba701f8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 -- 172.21.15.67:0/1044708172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba700c6750 con 0x7fba700d1060 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba6fe00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 0x7fba701f8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:12.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba75a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c6d0 tx=0x7fba6800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.769+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68016020 con 0x7fba700d1060 2024-04-17T08:53:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.770+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba6800a840 con 0x7fba700d1060 2024-04-17T08:53:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.770+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6800ac70 con 0x7fba700d1060 2024-04-17T08:53:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.770+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba700c8860 con 0x7fba700d1060 2024-04-17T08:53:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.770+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba700c8e20 con 0x7fba700d1060 2024-04-17T08:53:12.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.771+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba70068ca0 con 0x7fba700d1060 2024-04-17T08:53:12.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.772+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fba68019020 con 0x7fba700d1060 2024-04-17T08:53:12.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.773+0000 7fba6cc00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 0x7fba4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:12.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.773+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6683+0+0 (secure 0 0 0) 0x7fba6808c5f0 con 0x7fba700d1060 2024-04-17T08:53:12.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.774+0000 7fba75000640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 0x7fba4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:12.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.775+0000 7fba75000640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 0x7fba4c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba6003b040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:12.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:12.778+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fba68020020 con 0x7fba700d1060 2024-04-17T08:53:13.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.023+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fba700b11e0 con 0x7fba700d1060 2024-04-17T08:53:13.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.561+0000 7fba6cc00640 1 -- 172.21.15.67:0/1044708172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v145) ==== 104+0+0 (secure 0 0 0) 0x7fba700c6750 con 0x7fba700d1060 2024-04-17T08:53:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 msgr2=0x7fba4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 0x7fba4c06eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba6003b040 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 msgr2=0x7fba701f8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c6d0 tx=0x7fba6800cb90 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 shutdown_connections 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fba4c06ca10 0x7fba4c06eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba700d1060 0x7fba701f8bd0 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba7009b170 0x7fba701f8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 --2- 172.21.15.67:0/1044708172 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba7006daf0 0x7fba700b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.565+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 >> 172.21.15.67:0/1044708172 conn(0x7fba700b41a0 msgr2=0x7fba700b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.566+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 shutdown_connections 2024-04-17T08:53:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.566+0000 7fba77a00640 1 -- 172.21.15.67:0/1044708172 wait complete. 2024-04-17T08:53:13.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:53:13.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.865+0000 7f247b000640 1 Processor -- start 2024-04-17T08:53:13.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.865+0000 7f247b000640 1 -- start start 2024-04-17T08:53:13.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f247b000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2474163d10 0x7f2474162e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f247b000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2474071d20 0x7f2474072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f247b000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24740b3f20 con 0x7f2474162a20 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f247b000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f24740726d0 con 0x7f2474163d10 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f247b000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f24741641a0 con 0x7f2474071d20 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f2479c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2474162a20 0x7f24740b3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43124/0 (socket says 172.21.15.67:43124) 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.867+0000 7f2479c00640 1 -- 172.21.15.67:0/360216201 learned_addr learned my addr 172.21.15.67:0/360216201 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:13.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2479200640 1 --2- 172.21.15.67:0/360216201 >> v2:172.21.15.67:3300/0 conn(0x7f2474163d10 0x7f2474162e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f247a600640 1 --2- 172.21.15.67:0/360216201 >> v2:172.21.15.187:3300/0 conn(0x7f2474071d20 0x7f2474072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2473e00640 1 -- 172.21.15.67:0/360216201 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984873192 0 0) 0x7f24740b3f20 con 0x7f2474162a20 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2473e00640 1 -- 172.21.15.67:0/360216201 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f245c003520 con 0x7f2474162a20 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2479200640 1 -- 172.21.15.67:0/360216201 >> v2:172.21.15.187:3300/0 conn(0x7f2474071d20 msgr2=0x7f2474072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2479200640 1 --2- 172.21.15.67:0/360216201 >> v2:172.21.15.187:3300/0 conn(0x7f2474071d20 0x7f2474072190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2479200640 1 -- 172.21.15.67:0/360216201 >> v1:172.21.15.67:6789/0 conn(0x7f2474162a20 legacy=0x7f24740b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.868+0000 7f2479200640 1 -- 172.21.15.67:0/360216201 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24740b4200 con 0x7f2474163d10 2024-04-17T08:53:13.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.870+0000 7f2479200640 1 --2- 172.21.15.67:0/360216201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f2474162e00 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f24741666c0 tx=0x7f246002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cdf7f9737228e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f2473e00640 1 -- 172.21.15.67:0/360216201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24600315a0 con 0x7f2474163d10 2024-04-17T08:53:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f2473e00640 1 -- 172.21.15.67:0/360216201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2460037070 con 0x7f2474163d10 2024-04-17T08:53:13.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f2473e00640 1 -- 172.21.15.67:0/360216201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460031960 con 0x7f2474163d10 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 -- 172.21.15.67:0/360216201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 msgr2=0x7f2474162e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 --2- 172.21.15.67:0/360216201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f2474162e00 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f24741666c0 tx=0x7f246002b840 comp rx=0 tx=0).stop 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 -- 172.21.15.67:0/360216201 shutdown_connections 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 --2- 172.21.15.67:0/360216201 >> v2:172.21.15.187:3300/0 conn(0x7f2474071d20 0x7f2474072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 --2- 172.21.15.67:0/360216201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f2474162e00 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.871+0000 7f247b000640 1 -- 172.21.15.67:0/360216201 >> 172.21.15.67:0/360216201 conn(0x7f247406b8c0 msgr2=0x7f247406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.872+0000 7f247b000640 1 -- 172.21.15.67:0/360216201 shutdown_connections 2024-04-17T08:53:13.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.872+0000 7f247b000640 1 -- 172.21.15.67:0/360216201 wait complete. 2024-04-17T08:53:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.873+0000 7f247b000640 1 Processor -- start 2024-04-17T08:53:13.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.873+0000 7f247b000640 1 -- start start 2024-04-17T08:53:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 0x7f2474160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 0x7f24741f9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24740b3b30 con 0x7f2474163d10 2024-04-17T08:53:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f247406dfb0 con 0x7f2474162a20 2024-04-17T08:53:13.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.874+0000 7f247b000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f24740b3cb0 con 0x7f2474071d20 2024-04-17T08:53:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54358/0 (socket says 172.21.15.67:54358) 2024-04-17T08:53:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 -- 172.21.15.67:0/1704257339 learned_addr learned my addr 172.21.15.67:0/1704257339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f2479c00640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 0x7f2474160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f2479200640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 0x7f24741f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 -- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 msgr2=0x7f2474160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 0x7f2474160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 -- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 msgr2=0x7f24741f9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 0x7f24741f9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 -- 172.21.15.67:0/1704257339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24741fbf80 con 0x7f2474163d10 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f2479c00640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 0x7f2474160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f2479200640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 0x7f24741f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.875+0000 7f247a600640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f246c007e40 tx=0x7f246c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.876+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c014620 con 0x7f2474163d10 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.876+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c019070 con 0x7f2474163d10 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.876+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c014a20 con 0x7f2474163d10 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.876+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24741fc220 con 0x7f2474163d10 2024-04-17T08:53:13.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.876+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24741fc7e0 con 0x7f2474163d10 2024-04-17T08:53:13.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.877+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2474068a10 con 0x7f2474163d10 2024-04-17T08:53:13.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.878+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f246c014bc0 con 0x7f2474163d10 2024-04-17T08:53:13.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.880+0000 7f2471600640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 0x7f245406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:13.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.880+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6683+0+0 (secure 0 0 0) 0x7f246c08c9b0 con 0x7f2474163d10 2024-04-17T08:53:13.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.880+0000 7f2479c00640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 0x7f245406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:13.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.881+0000 7f2479c00640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 0x7f245406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2468005880 tx=0x7f2468009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:13.884+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f246c05a8f0 con 0x7f2474163d10 2024-04-17T08:53:14.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.089+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2474072fd0 con 0x7f2474163d10 2024-04-17T08:53:14.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.581+0000 7f2471600640 1 -- 172.21.15.67:0/1704257339 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v276) ==== 181+0+0 (secure 0 0 0) 0x7f246c05a1f0 con 0x7f2474163d10 2024-04-17T08:53:14.580 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:53:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.586+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 msgr2=0x7f245406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.586+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 0x7f245406eed0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2468005880 tx=0x7f2468009040 comp rx=0 tx=0).stop 2024-04-17T08:53:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.586+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 msgr2=0x7f247415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.586+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7f246c007e40 tx=0x7f246c00e4b0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 shutdown_connections 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f245406ca10 0x7f245406eed0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2474163d10 0x7f247415d280 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2474162a20 0x7f24741f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 --2- 172.21.15.67:0/1704257339 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2474071d20 0x7f2474160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.588+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 >> 172.21.15.67:0/1704257339 conn(0x7f247406b8c0 msgr2=0x7f247406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.589+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 shutdown_connections 2024-04-17T08:53:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.589+0000 7f247b000640 1 -- 172.21.15.67:0/1704257339 wait complete. 2024-04-17T08:53:14.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:53:14.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.832+0000 7fbd68000640 1 Processor -- start 2024-04-17T08:53:14.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.832+0000 7fbd68000640 1 -- start start 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.833+0000 7fbd68000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd600b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd68000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd6006edb0 con 0x7fbd60148210 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd68000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd600b4260 con 0x7fbd60071f80 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd68000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd600b4540 con 0x7fbd600b3780 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd65600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbd60148210 0x7fbd6006e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43138/0 (socket says 172.21.15.67:43138) 2024-04-17T08:53:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd65600640 1 -- 172.21.15.67:0/3162440939 learned_addr learned my addr 172.21.15.67:0/3162440939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:14.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635715344 0 0) 0x7fbd6006edb0 con 0x7fbd60148210 2024-04-17T08:53:14.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd64c00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd600b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.834+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd48003520 con 0x7fbd60148210 2024-04-17T08:53:14.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3476723318 0 0) 0x7fbd48003520 con 0x7fbd60148210 2024-04-17T08:53:14.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 msgr2=0x7fbd600b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd600b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 >> v1:172.21.15.187:6789/0 conn(0x7fbd60071f80 legacy=0x7fbd600b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6006dea0 con 0x7fbd60148210 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbd54001670 con 0x7fbd60148210 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 >> v1:172.21.15.67:6789/0 conn(0x7fbd60148210 legacy=0x7fbd6006e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd48004060 0x7fbd48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004d40 0x7fbd480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd600b4540 con 0x7fbd48004060 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4800bca0 con 0x7fbd48004d40 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4800bea0 con 0x7fbd600b3780 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.835+0000 7fbd64c00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd48004060 0x7fbd48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd65600640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd66000640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004d40 0x7fbd480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd64c00640 1 -- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 msgr2=0x7fbd480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd64c00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd64c00640 1 -- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004d40 msgr2=0x7fbd480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd64c00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004d40 0x7fbd480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd64c00640 1 -- 172.21.15.67:0/3162440939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd48003520 con 0x7fbd48004060 2024-04-17T08:53:14.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.836+0000 7fbd65600640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd600b3780 0x7fbd480039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.837+0000 7fbd64c00640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd48004060 0x7fbd48004800 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00ba30 tx=0x7fbd48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feaf071c2f3f7b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.838+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c036020 con 0x7fbd48004060 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.838+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4c032630 con 0x7fbd48004060 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.838+0000 7fbd5fe00640 1 -- 172.21.15.67:0/3162440939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c00b600 con 0x7fbd48004060 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.838+0000 7fbd68000640 1 -- 172.21.15.67:0/3162440939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd48004060 msgr2=0x7fbd48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.838+0000 7fbd68000640 1 --2- 172.21.15.67:0/3162440939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd48004060 0x7fbd48004800 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00ba30 tx=0x7fbd48005830 comp rx=0 tx=0).stop 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.839+0000 7fbd66000640 1 -- 172.21.15.67:0/3162440939 reap_dead start 2024-04-17T08:53:14.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.839+0000 7fbd68000640 1 -- 172.21.15.67:0/3162440939 shutdown_connections 2024-04-17T08:53:14.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.839+0000 7fbd68000640 1 -- 172.21.15.67:0/3162440939 >> 172.21.15.67:0/3162440939 conn(0x7fbd6006b8c0 msgr2=0x7fbd6006bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:14.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.839+0000 7fbd68000640 1 -- 172.21.15.67:0/3162440939 shutdown_connections 2024-04-17T08:53:14.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.839+0000 7fbd68000640 1 -- 172.21.15.67:0/3162440939 wait complete. 2024-04-17T08:53:14.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.841+0000 7fbd68000640 1 Processor -- start 2024-04-17T08:53:14.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.841+0000 7fbd68000640 1 -- start start 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004060 0x7fbd60162ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd60153cd0 0x7fbd6015ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd601485f0 con 0x7fbd601604c0 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6006f680 con 0x7fbd48004060 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd68000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd60148770 con 0x7fbd60153cd0 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54370/0 (socket says 172.21.15.67:54370) 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 -- 172.21.15.67:0/2939996287 learned_addr learned my addr 172.21.15.67:0/2939996287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:14.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 -- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd60153cd0 msgr2=0x7fbd6015ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd60153cd0 0x7fbd6015ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 -- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004060 msgr2=0x7fbd60162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.842+0000 7fbd66000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004060 0x7fbd60162ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd66000640 1 -- 172.21.15.67:0/2939996287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd60162130 con 0x7fbd601604c0 2024-04-17T08:53:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd66000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007fd0 tx=0x7fbd5800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58003970 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd58018070 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58003d50 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd60162410 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.843+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd601629d0 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.844+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd60073a50 con 0x7fbd601604c0 2024-04-17T08:53:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.845+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fbd580050d0 con 0x7fbd601604c0 2024-04-17T08:53:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.846+0000 7fbd5d600640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 0x7fbd4006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.847+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(276..276 src has 1..276) ==== 6245+0+0 (secure 0 0 0) 0x7fbd5808bc00 con 0x7fbd601604c0 2024-04-17T08:53:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.847+0000 7fbd65600640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 0x7fbd4006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:14.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.848+0000 7fbd65600640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 0x7fbd4006eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbd54000c00 tx=0x7fbd54001a00 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:14.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:14.851+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbd58059c60 con 0x7fbd601604c0 2024-04-17T08:53:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.049+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbd60068d60 con 0x7fbd601604c0 2024-04-17T08:53:15.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.584+0000 7fbd5d600640 1 -- 172.21.15.67:0/2939996287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v277) ==== 169+0+0 (secure 0 0 0) 0x7fbd58059560 con 0x7fbd601604c0 2024-04-17T08:53:15.584 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:53:15.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 msgr2=0x7fbd4006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 0x7fbd4006eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbd54000c00 tx=0x7fbd54001a00 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 msgr2=0x7fbd60146850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007fd0 tx=0x7fbd5800eb90 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 shutdown_connections 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbd4006ca10 0x7fbd4006eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd601604c0 0x7fbd60146850 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd60153cd0 0x7fbd6015ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 --2- 172.21.15.67:0/2939996287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd48004060 0x7fbd60162ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 >> 172.21.15.67:0/2939996287 conn(0x7fbd6006b8c0 msgr2=0x7fbd6013d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.588+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 shutdown_connections 2024-04-17T08:53:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.589+0000 7fbd68000640 1 -- 172.21.15.67:0/2939996287 wait complete. 2024-04-17T08:53:15.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:53:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.831+0000 7f8b9fe00640 1 Processor -- start 2024-04-17T08:53:15.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.831+0000 7f8b9fe00640 1 -- start start 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.832+0000 7f8b9fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ba006bfd0 0x7f8ba00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.832+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba006f800 0x7f8ba00b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba006df40 con 0x7f8ba015b6b0 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba006ed00 con 0x7f8ba006bfd0 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba006e870 con 0x7f8ba006f800 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8ba015b6b0 0x7f8ba006b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43154/0 (socket says 172.21.15.67:43154) 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9ea00640 1 -- 172.21.15.67:0/3981885173 learned_addr learned my addr 172.21.15.67:0/3981885173 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:15.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b97e00640 1 --2- 172.21.15.67:0/3981885173 >> v2:172.21.15.67:3300/0 conn(0x7f8ba006bfd0 0x7f8ba00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9f400640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba006f800 0x7f8ba00b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9e000640 1 -- 172.21.15.67:0/3981885173 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558936439 0 0) 0x7f8ba006df40 con 0x7f8ba015b6b0 2024-04-17T08:53:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.833+0000 7f8b9e000640 1 -- 172.21.15.67:0/3981885173 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b84003520 con 0x7f8ba015b6b0 2024-04-17T08:53:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.834+0000 7f8b97e00640 1 -- 172.21.15.67:0/3981885173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba006f800 msgr2=0x7f8ba00b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.834+0000 7f8b97e00640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba006f800 0x7f8ba00b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.834+0000 7f8b97e00640 1 -- 172.21.15.67:0/3981885173 >> v1:172.21.15.67:6789/0 conn(0x7f8ba015b6b0 legacy=0x7f8ba006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.834+0000 7f8b97e00640 1 -- 172.21.15.67:0/3981885173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba006e220 con 0x7f8ba006bfd0 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.835+0000 7f8b97e00640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00b3490 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097a0 tx=0x7f8b8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04ea68615f66cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.835+0000 7f8b9e000640 1 -- 172.21.15.67:0/3981885173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b880315a0 con 0x7f8ba006bfd0 2024-04-17T08:53:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.835+0000 7f8b9e000640 1 -- 172.21.15.67:0/3981885173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b88037070 con 0x7f8ba006bfd0 2024-04-17T08:53:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.835+0000 7f8b9e000640 1 -- 172.21.15.67:0/3981885173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031960 con 0x7f8ba006bfd0 2024-04-17T08:53:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 -- 172.21.15.67:0/3981885173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006bfd0 msgr2=0x7f8ba00b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00b3490 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097a0 tx=0x7f8b8802b840 comp rx=0 tx=0).stop 2024-04-17T08:53:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 -- 172.21.15.67:0/3981885173 shutdown_connections 2024-04-17T08:53:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba006f800 0x7f8ba00b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/3981885173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00b3490 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 -- 172.21.15.67:0/3981885173 >> 172.21.15.67:0/3981885173 conn(0x7f8ba00b2190 msgr2=0x7f8ba00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 -- 172.21.15.67:0/3981885173 shutdown_connections 2024-04-17T08:53:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.836+0000 7f8b9fe00640 1 -- 172.21.15.67:0/3981885173 wait complete. 2024-04-17T08:53:15.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.838+0000 7f8b9fe00640 1 Processor -- start 2024-04-17T08:53:15.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.838+0000 7f8b9fe00640 1 -- start start 2024-04-17T08:53:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba015b6b0 0x7f8ba0144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba00afee0 con 0x7f8ba006f800 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba00afd60 con 0x7f8ba006bfd0 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba00cfc10 con 0x7f8ba015b6b0 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54394/0 (socket says 172.21.15.67:54394) 2024-04-17T08:53:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.839+0000 7f8b97e00640 1 -- 172.21.15.67:0/1715954047 learned_addr learned my addr 172.21.15.67:0/1715954047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:15.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b9ea00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 -- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba015b6b0 msgr2=0x7f8ba0144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b9f400640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba015b6b0 0x7f8ba0144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba015b6b0 0x7f8ba0144790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 -- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba006bfd0 msgr2=0x7f8ba00aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 -- 172.21.15.67:0/1715954047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba015a510 con 0x7f8ba006f800 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b97e00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f8b88034040 tx=0x7f8b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.840+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88032630 con 0x7f8ba006f800 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.841+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b880327f0 con 0x7f8ba006f800 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.841+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba015a7f0 con 0x7f8ba006f800 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.841+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88032be0 con 0x7f8ba006f800 2024-04-17T08:53:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.841+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba01573d0 con 0x7f8ba006f800 2024-04-17T08:53:15.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.842+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba0068a10 con 0x7f8ba006f800 2024-04-17T08:53:15.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.843+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f8b8804e430 con 0x7f8ba006f800 2024-04-17T08:53:15.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.844+0000 7f8b96a00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 0x7f8b7c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:15.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.844+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f8b8802f030 con 0x7f8ba006f800 2024-04-17T08:53:15.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.844+0000 7f8b9ea00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 0x7f8b7c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.845+0000 7f8b9ea00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 0x7f8b7c06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b90007fd0 tx=0x7f8b90009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:15.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:15.848+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8b88040070 con 0x7f8ba006f800 2024-04-17T08:53:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.051+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f8ba00b28f0 con 0x7f8ba006f800 2024-04-17T08:53:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.058+0000 7f8b96a00640 1 -- 172.21.15.67:0/1715954047 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) ==== 115+0+0 (secure 0 0 0) 0x7f8b8807bed0 con 0x7f8ba006f800 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 msgr2=0x7f8b7c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 0x7f8b7c06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8b90007fd0 tx=0x7f8b90009040 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 msgr2=0x7f8ba0144250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f8b88034040 tx=0x7f8b880046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 shutdown_connections 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f8b7c06ca10 0x7f8b7c06eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8ba015b6b0 0x7f8ba0144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba006f800 0x7f8ba0144250 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1715954047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8ba006bfd0 0x7f8ba00aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 >> 172.21.15.67:0/1715954047 conn(0x7f8ba00b2190 msgr2=0x7f8ba00ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 shutdown_connections 2024-04-17T08:53:16.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.061+0000 7f8b9fe00640 1 -- 172.21.15.67:0/1715954047 wait complete. 2024-04-17T08:53:16.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:53:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.300+0000 7fc277400640 1 Processor -- start 2024-04-17T08:53:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.300+0000 7fc277400640 1 -- start start 2024-04-17T08:53:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc277400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc27006ce00 0x7fc27009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc277400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc27009afa0 0x7fc270070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc277400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2700d1520 con 0x7fc27006daf0 2024-04-17T08:53:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc277400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc2700d1800 con 0x7fc27006ce00 2024-04-17T08:53:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc277400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc2700d1ae0 con 0x7fc27009afa0 2024-04-17T08:53:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc274a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc27006daf0 0x7fc2700d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43170/0 (socket says 172.21.15.67:43170) 2024-04-17T08:53:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc274a00640 1 -- 172.21.15.67:0/1223519501 learned_addr learned my addr 172.21.15.67:0/1223519501 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:16.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc27009afa0 0x7fc270070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.302+0000 7fc26fe00640 1 --2- 172.21.15.67:0/1223519501 >> v2:172.21.15.187:3300/0 conn(0x7fc27006ce00 0x7fc27009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2206905482 0 0) 0x7fc2700d1520 con 0x7fc27006daf0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc254003520 con 0x7fc27006daf0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1152636100 0 0) 0x7fc254003520 con 0x7fc27006daf0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc27009afa0 msgr2=0x7fc270070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc27009afa0 0x7fc270070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 >> v2:172.21.15.187:3300/0 conn(0x7fc27006ce00 msgr2=0x7fc27009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 --2- 172.21.15.67:0/1223519501 >> v2:172.21.15.187:3300/0 conn(0x7fc27006ce00 0x7fc27009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2700d1dc0 con 0x7fc27006daf0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fc260001670 con 0x7fc27006daf0 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 >> v1:172.21.15.67:6789/0 conn(0x7fc27006daf0 legacy=0x7fc2700d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2540039b0 0x7fc254003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2540043a0 0x7fc254004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc254004fa0 0x7fc25400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc2700d1800 con 0x7fc254004fa0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc2700d1ae0 con 0x7fc2540043a0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc25400bf30 con 0x7fc2540039b0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.303+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc27009afa0 0x7fc270070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc254004fa0 0x7fc25400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc274a00640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2540039b0 0x7fc254003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc26fe00640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2540043a0 0x7fc254004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 -- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2540039b0 msgr2=0x7fc254003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc2540039b0 0x7fc254003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 -- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2540043a0 msgr2=0x7fc254004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2540043a0 0x7fc254004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.304+0000 7fc275400640 1 -- 172.21.15.67:0/1223519501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc254003520 con 0x7fc254004fa0 2024-04-17T08:53:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.305+0000 7fc275400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc254004fa0 0x7fc25400b9c0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fc268006fd0 tx=0x7fc254005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a57fdd60475c62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26800f040 con 0x7fc254004fa0 2024-04-17T08:53:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc268033600 con 0x7fc254004fa0 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc26f400640 1 -- 172.21.15.67:0/1223519501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2680339d0 con 0x7fc254004fa0 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 -- 172.21.15.67:0/1223519501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc254004fa0 msgr2=0x7fc25400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 --2- 172.21.15.67:0/1223519501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc254004fa0 0x7fc25400b9c0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fc268006fd0 tx=0x7fc254005830 comp rx=0 tx=0).stop 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc275400640 1 -- 172.21.15.67:0/1223519501 reap_dead start 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 -- 172.21.15.67:0/1223519501 shutdown_connections 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 -- 172.21.15.67:0/1223519501 >> 172.21.15.67:0/1223519501 conn(0x7fc2700b41a0 msgr2=0x7fc2700b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 -- 172.21.15.67:0/1223519501 shutdown_connections 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.306+0000 7fc277400640 1 -- 172.21.15.67:0/1223519501 wait complete. 2024-04-17T08:53:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.308+0000 7fc277400640 1 Processor -- start 2024-04-17T08:53:16.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.308+0000 7fc277400640 1 -- start start 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc254004fa0 0x7fc2701537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc270154640 0x7fc2700c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc27006ded0 con 0x7fc270155c50 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc2700d1230 con 0x7fc270154640 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc277400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc27006e050 con 0x7fc254004fa0 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc275400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc275400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54412/0 (socket says 172.21.15.67:54412) 2024-04-17T08:53:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.309+0000 7fc275400640 1 -- 172.21.15.67:0/3169200597 learned_addr learned my addr 172.21.15.67:0/3169200597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:16.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc274a00640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc254004fa0 0x7fc2701537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc26fe00640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc270154640 0x7fc2700c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 -- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc254004fa0 msgr2=0x7fc2701537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc254004fa0 0x7fc2701537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 -- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc270154640 msgr2=0x7fc2700c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc270154640 0x7fc2700c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 -- 172.21.15.67:0/3169200597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2700c9300 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc275400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fc26802bdc0 tx=0x7fc2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.310+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268041030 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.311+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc268033400 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.311+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc270154bb0 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.311+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268032a00 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.311+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc270155170 con 0x7fc270155c50 2024-04-17T08:53:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.312+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc270068ca0 con 0x7fc270155c50 2024-04-17T08:53:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.313+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fc268044030 con 0x7fc270155c50 2024-04-17T08:53:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.314+0000 7fc26cc00640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 0x7fc24c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.314+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7fc2680ad6d0 con 0x7fc270155c50 2024-04-17T08:53:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.314+0000 7fc274a00640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 0x7fc24c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.316+0000 7fc274a00640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 0x7fc24c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc270154350 tx=0x7fc260000f30 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.319+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc2680335d0 con 0x7fc270155c50 2024-04-17T08:53:16.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.524+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fc2700b5040 con 0x7fc270155c50 2024-04-17T08:53:16.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.532+0000 7fc26cc00640 1 -- 172.21.15.67:0/3169200597 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v92) ==== 104+0+0 (secure 0 0 0) 0x7fc26807b730 con 0x7fc270155c50 2024-04-17T08:53:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 msgr2=0x7fc24c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 0x7fc24c06eed0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc270154350 tx=0x7fc260000f30 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 msgr2=0x7fc27015b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fc26802bdc0 tx=0x7fc2680046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 shutdown_connections 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc24c06ca10 0x7fc24c06eed0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc270155c50 0x7fc27015b890 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc270154640 0x7fc2700c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 --2- 172.21.15.67:0/3169200597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc254004fa0 0x7fc2701537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 >> 172.21.15.67:0/3169200597 conn(0x7fc2700b41a0 msgr2=0x7fc2700b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 shutdown_connections 2024-04-17T08:53:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.535+0000 7fc277400640 1 -- 172.21.15.67:0/3169200597 wait complete. 2024-04-17T08:53:16.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:53:16.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.762+0000 7f9c35800640 1 Processor -- start 2024-04-17T08:53:16.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.763+0000 7f9c35800640 1 -- start start 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.764+0000 7f9c35800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c30145ac0 0x7f9c30071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.764+0000 7f9c35800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c300cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.764+0000 7f9c35800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c300cea70 con 0x7f9c300ce3b0 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.764+0000 7f9c35800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9c300ce840 con 0x7f9c30145ac0 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.764+0000 7f9c35800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300ced50 con 0x7f9c30072d80 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c30145ac0 0x7f9c30071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c30145ac0 0x7f9c30071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54426/0 (socket says 172.21.15.67:54426) 2024-04-17T08:53:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 -- 172.21.15.67:0/1793223012 learned_addr learned my addr 172.21.15.67:0/1793223012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2f400640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c300cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 -- 172.21.15.67:0/1793223012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 msgr2=0x7f9c300cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c300cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 -- 172.21.15.67:0/1793223012 >> v1:172.21.15.187:6789/0 conn(0x7f9c300ce3b0 legacy=0x7f9c30072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2ea00640 1 -- 172.21.15.67:0/1793223012 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c300cf030 con 0x7f9c30145ac0 2024-04-17T08:53:16.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.765+0000 7f9c2f400640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c300cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.766+0000 7f9c2ea00640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c30071ea0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009e80 tx=0x7f9c2002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ef114c4cf79b5d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c2d600640 1 -- 172.21.15.67:0/1793223012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2002ca50 con 0x7f9c30145ac0 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c2d600640 1 -- 172.21.15.67:0/1793223012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2002cbf0 con 0x7f9c30145ac0 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c2d600640 1 -- 172.21.15.67:0/1793223012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c200343f0 con 0x7f9c30145ac0 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 -- 172.21.15.67:0/1793223012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 msgr2=0x7f9c30071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c30071ea0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009e80 tx=0x7f9c2002beb0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 -- 172.21.15.67:0/1793223012 shutdown_connections 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c300cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 --2- 172.21.15.67:0/1793223012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c30071ea0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.767+0000 7f9c35800640 1 -- 172.21.15.67:0/1793223012 >> 172.21.15.67:0/1793223012 conn(0x7f9c30067ee0 msgr2=0x7f9c3006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.768+0000 7f9c35800640 1 -- 172.21.15.67:0/1793223012 shutdown_connections 2024-04-17T08:53:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.768+0000 7f9c35800640 1 -- 172.21.15.67:0/1793223012 wait complete. 2024-04-17T08:53:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.769+0000 7f9c35800640 1 Processor -- start 2024-04-17T08:53:16.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.769+0000 7f9c35800640 1 -- start start 2024-04-17T08:53:16.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c3006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 0x7f9c301ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30145ea0 con 0x7f9c30145ac0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300cf5f0 con 0x7f9c300ce3b0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.770+0000 7f9c35800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300d0050 con 0x7f9c30072d80 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54434/0 (socket says 172.21.15.67:54434) 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 -- 172.21.15.67:0/2135033856 learned_addr learned my addr 172.21.15.67:0/2135033856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2ea00640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c3006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 -- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 msgr2=0x7f9c3006e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c3006e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2e000640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 0x7f9c301ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 -- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 msgr2=0x7f9c301ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 0x7f9c301ee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2f400640 1 -- 172.21.15.67:0/2135033856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3006ec10 con 0x7f9c30145ac0 2024-04-17T08:53:16.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.771+0000 7f9c2e000640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 0x7f9c301ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:16.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c2f400640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c7f0 tx=0x7f9c2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28015070 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c28007820 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3006eef0 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28007c50 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.772+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3006f4b0 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.773+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c30068ca0 con 0x7f9c30145ac0 2024-04-17T08:53:16.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.774+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f9c28018070 con 0x7f9c30145ac0 2024-04-17T08:53:16.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.775+0000 7f9c26a00640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 0x7f9c0c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:16.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.775+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f9c2808c000 con 0x7f9c30145ac0 2024-04-17T08:53:16.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.776+0000 7f9c2ea00640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 0x7f9c0c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.777+0000 7f9c2ea00640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 0x7f9c0c06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c20035000 tx=0x7f9c20030000 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:16.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.780+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f9c2805a190 con 0x7f9c30145ac0 2024-04-17T08:53:16.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.983+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f9c3006b9d0 con 0x7f9c30145ac0 2024-04-17T08:53:16.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.990+0000 7f9c26a00640 1 -- 172.21.15.67:0/2135033856 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v93) ==== 120+0+0 (secure 0 0 0) 0x7f9c28059a90 con 0x7f9c30145ac0 2024-04-17T08:53:16.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 msgr2=0x7f9c0c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 0x7f9c0c06eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c20035000 tx=0x7f9c20030000 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 msgr2=0x7f9c300ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c7f0 tx=0x7f9c2800ccb0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 shutdown_connections 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f9c0c06ca10 0x7f9c0c06eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30145ac0 0x7f9c300ca2c0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c300ce3b0 0x7f9c301ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 --2- 172.21.15.67:0/2135033856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c30072d80 0x7f9c3006e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.993+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 >> 172.21.15.67:0/2135033856 conn(0x7f9c30067ee0 msgr2=0x7f9c30143a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:16.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.994+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 shutdown_connections 2024-04-17T08:53:16.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:16.994+0000 7f9c35800640 1 -- 172.21.15.67:0/2135033856 wait complete. 2024-04-17T08:53:17.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-04-17T08:53:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.227+0000 7fdfff200640 1 Processor -- start 2024-04-17T08:53:17.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.227+0000 7fdfff200640 1 -- start start 2024-04-17T08:53:17.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdfff200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000161e40 0x7fe0000ba3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdfff200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0000cfd60 0x7fe0001626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdfff200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe000162e60 con 0x7fe000162220 2024-04-17T08:53:17.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdfff200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe000162c30 con 0x7fe0000cfd60 2024-04-17T08:53:17.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdfff200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe000163140 con 0x7fe000161e40 2024-04-17T08:53:17.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.229+0000 7fdffe800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0000cfd60 0x7fe0001626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0000cfd60 0x7fe0001626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54438/0 (socket says 172.21.15.67:54438) 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 -- 172.21.15.67:0/478363079 learned_addr learned my addr 172.21.15.67:0/478363079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffde00640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000161e40 0x7fe0000ba3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 -- 172.21.15.67:0/478363079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000161e40 msgr2=0x7fe0000ba3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000161e40 0x7fe0000ba3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 -- 172.21.15.67:0/478363079 >> v1:172.21.15.187:6789/0 conn(0x7fe000162220 legacy=0x7fe0000cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.230+0000 7fdffe800640 1 -- 172.21.15.67:0/478363079 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe000163420 con 0x7fe0000cfd60 2024-04-17T08:53:17.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.231+0000 7fdffe800640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0000cfd60 0x7fe0001626f0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009e50 tx=0x7fdff802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2953ac3a43522474 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdffd400640 1 -- 172.21.15.67:0/478363079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff802cb00 con 0x7fe0000cfd60 2024-04-17T08:53:17.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdffd400640 1 -- 172.21.15.67:0/478363079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff802cca0 con 0x7fe0000cfd60 2024-04-17T08:53:17.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdffd400640 1 -- 172.21.15.67:0/478363079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8034610 con 0x7fe0000cfd60 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 -- 172.21.15.67:0/478363079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0000cfd60 msgr2=0x7fe0001626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0000cfd60 0x7fe0001626f0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009e50 tx=0x7fdff802bf60 comp rx=0 tx=0).stop 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 -- 172.21.15.67:0/478363079 shutdown_connections 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0000cfd60 0x7fe0001626f0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 --2- 172.21.15.67:0/478363079 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000161e40 0x7fe0000ba3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 -- 172.21.15.67:0/478363079 >> 172.21.15.67:0/478363079 conn(0x7fe00006cbf0 msgr2=0x7fe00006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.232+0000 7fdfff200640 1 -- 172.21.15.67:0/478363079 shutdown_connections 2024-04-17T08:53:17.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.233+0000 7fdfff200640 1 -- 172.21.15.67:0/478363079 wait complete. 2024-04-17T08:53:17.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.234+0000 7fdfff200640 1 Processor -- start 2024-04-17T08:53:17.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.234+0000 7fdfff200640 1 -- start start 2024-04-17T08:53:17.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0000cfd60 0x7fe0000cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000162220 0x7fe0000cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0000d34e0 con 0x7fe000161e40 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0000d3360 con 0x7fe0000cfd60 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.235+0000 7fdfff200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0000d3660 con 0x7fe000162220 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54444/0 (socket says 172.21.15.67:54444) 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 -- 172.21.15.67:0/4129165437 learned_addr learned my addr 172.21.15.67:0/4129165437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdffe800640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000162220 0x7fe0000cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdffde00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0000cfd60 0x7fe0000cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 -- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000162220 msgr2=0x7fe0000cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000162220 0x7fe0000cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 -- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0000cfd60 msgr2=0x7fe0000cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0000cfd60 0x7fe0000cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 -- 172.21.15.67:0/4129165437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0001f4540 con 0x7fe000161e40 2024-04-17T08:53:17.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.236+0000 7fdff7e00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c3c0 tx=0x7fdfe800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.237+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe80052d0 con 0x7fe000161e40 2024-04-17T08:53:17.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.237+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe8005490 con 0x7fe000161e40 2024-04-17T08:53:17.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.237+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8007260 con 0x7fe000161e40 2024-04-17T08:53:17.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.237+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0001f4820 con 0x7fe000161e40 2024-04-17T08:53:17.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.237+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0001f4de0 con 0x7fe000161e40 2024-04-17T08:53:17.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.238+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe000068a10 con 0x7fe000161e40 2024-04-17T08:53:17.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.239+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fdfe8007400 con 0x7fe000161e40 2024-04-17T08:53:17.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.240+0000 7fdff6000640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 0x7fdfdc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.240+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7fdfe808b210 con 0x7fe000161e40 2024-04-17T08:53:17.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.241+0000 7fdffde00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 0x7fdfdc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.242+0000 7fdffde00640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 0x7fdfdc06ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.244+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdfe80594a0 con 0x7fe000161e40 2024-04-17T08:53:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.521+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7fe0000d2590 con 0x7fe000161e40 2024-04-17T08:53:17.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.608+0000 7fdff6000640 1 -- 172.21.15.67:0/4129165437 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3325) ==== 155+0+0 (secure 0 0 0) 0x7fdfe8058da0 con 0x7fe000161e40 2024-04-17T08:53:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.610+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 msgr2=0x7fdfdc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.610+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 0x7fdfdc06ef00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdff0006fd0 tx=0x7fdff0008040 comp rx=0 tx=0).stop 2024-04-17T08:53:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.610+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 msgr2=0x7fe0000c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.610+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fdfe800c3c0 tx=0x7fdfe800c880 comp rx=0 tx=0).stop 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 shutdown_connections 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdfdc06ca40 0x7fdfdc06ef00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe000162220 0x7fe0000cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe000161e40 0x7fe0000c9700 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 --2- 172.21.15.67:0/4129165437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0000cfd60 0x7fe0000cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 >> 172.21.15.67:0/4129165437 conn(0x7fe00006cbf0 msgr2=0x7fe00015c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 shutdown_connections 2024-04-17T08:53:17.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.611+0000 7fdfff200640 1 -- 172.21.15.67:0/4129165437 wait complete. 2024-04-17T08:53:17.628 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-04-17T08:53:17.628 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-04-17T08:53:17.629 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-17T08:53:17.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.847+0000 7fcb6fe00640 1 Processor -- start 2024-04-17T08:53:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.847+0000 7fcb6fe00640 1 -- start start 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6fe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcb7006feb0 0x7fcb7006b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb7006be70 0x7fcb7015f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb7015bc80 con 0x7fcb7015b570 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7015bf40 con 0x7fcb7006be70 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6fe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7015c220 con 0x7fcb7006feb0 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb7006be70 0x7fcb7015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb7006be70 0x7fcb7015f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54458/0 (socket says 172.21.15.67:54458) 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb6f400640 1 -- 172.21.15.67:0/89963678 learned_addr learned my addr 172.21.15.67:0/89963678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.849+0000 7fcb66000640 1 --2- 172.21.15.67:0/89963678 >> v2:172.21.15.187:3300/0 conn(0x7fcb7006feb0 0x7fcb7006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.850+0000 7fcb6f400640 1 -- 172.21.15.67:0/89963678 >> v2:172.21.15.187:3300/0 conn(0x7fcb7006feb0 msgr2=0x7fcb7006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.850+0000 7fcb6f400640 1 --2- 172.21.15.67:0/89963678 >> v2:172.21.15.187:3300/0 conn(0x7fcb7006feb0 0x7fcb7006b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.850+0000 7fcb6f400640 1 -- 172.21.15.67:0/89963678 >> v1:172.21.15.187:6789/0 conn(0x7fcb7015b570 legacy=0x7fcb7006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.850+0000 7fcb6f400640 1 -- 172.21.15.67:0/89963678 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7015c500 con 0x7fcb7006be70 2024-04-17T08:53:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.851+0000 7fcb6f400640 1 --2- 172.21.15.67:0/89963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb7015f140 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fcb70073040 tx=0x7fcb6802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06aa1318f51aef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.851+0000 7fcb6e000640 1 -- 172.21.15.67:0/89963678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6802c9f0 con 0x7fcb7006be70 2024-04-17T08:53:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.851+0000 7fcb6e000640 1 -- 172.21.15.67:0/89963678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb6802cb90 con 0x7fcb7006be70 2024-04-17T08:53:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.851+0000 7fcb6e000640 1 -- 172.21.15.67:0/89963678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb680343f0 con 0x7fcb7006be70 2024-04-17T08:53:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 -- 172.21.15.67:0/89963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 msgr2=0x7fcb7015f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/89963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb7015f140 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fcb70073040 tx=0x7fcb6802be50 comp rx=0 tx=0).stop 2024-04-17T08:53:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 -- 172.21.15.67:0/89963678 shutdown_connections 2024-04-17T08:53:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/89963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb7015f140 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/89963678 >> v2:172.21.15.187:3300/0 conn(0x7fcb7006feb0 0x7fcb7006b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 -- 172.21.15.67:0/89963678 >> 172.21.15.67:0/89963678 conn(0x7fcb70067d10 msgr2=0x7fcb700b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 -- 172.21.15.67:0/89963678 shutdown_connections 2024-04-17T08:53:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.852+0000 7fcb6fe00640 1 -- 172.21.15.67:0/89963678 wait complete. 2024-04-17T08:53:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.854+0000 7fcb6fe00640 1 Processor -- start 2024-04-17T08:53:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.854+0000 7fcb6fe00640 1 -- start start 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb7006feb0 0x7fcb700cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb7015b570 0x7fcb700b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb700b4930 con 0x7fcb7006be70 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcb700b47b0 con 0x7fcb7015b570 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7006c280 con 0x7fcb7006feb0 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54462/0 (socket says 172.21.15.67:54462) 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 -- 172.21.15.67:0/2315354672 learned_addr learned my addr 172.21.15.67:0/2315354672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 -- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb7006feb0 msgr2=0x7fcb700cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb7006feb0 0x7fcb700cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 -- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb7015b570 msgr2=0x7fcb700b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb7015b570 0x7fcb700b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.855+0000 7fcb6ea00640 1 -- 172.21.15.67:0/2315354672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb70159920 con 0x7fcb7006be70 2024-04-17T08:53:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb6ea00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b3f0 tx=0x7fcb6000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60015020 con 0x7fcb7006be70 2024-04-17T08:53:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb60011830 con 0x7fcb7006be70 2024-04-17T08:53:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60011c20 con 0x7fcb7006be70 2024-04-17T08:53:17.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb700cf580 con 0x7fcb7006be70 2024-04-17T08:53:17.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.856+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb700ce560 con 0x7fcb7006be70 2024-04-17T08:53:17.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.857+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb70068a10 con 0x7fcb7006be70 2024-04-17T08:53:17.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.858+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fcb60018020 con 0x7fcb7006be70 2024-04-17T08:53:17.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.859+0000 7fcb67400640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 0x7fcb4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:17.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.859+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7fcb6008ba20 con 0x7fcb7006be70 2024-04-17T08:53:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.860+0000 7fcb66000640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 0x7fcb4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.861+0000 7fcb66000640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 0x7fcb4c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb701696a0 tx=0x7fcb58008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:17.864+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcb6001f030 con 0x7fcb7006be70 2024-04-17T08:53:18.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.067+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb700ccbf0 con 0x7fcb7006be70 2024-04-17T08:53:18.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.068+0000 7fcb67400640 1 -- 172.21.15.67:0/2315354672 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v277) ==== 74+0+14296 (secure 0 0 0) 0x7fcb700cf580 con 0x7fcb7006be70 2024-04-17T08:53:18.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:18.068 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":277,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:15.581774+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/1995843637":"2024-04-18T08:53:12.530404+0000","172.21.15.67:6833/1995843637":"2024-04-18T08:53:12.530404+0000","172.21.15.187:6833/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/2991348352":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3653376109":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3444724328":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3175082800":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/1234602879":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3978763551":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3094572004":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2893316565":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/2486101166":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3800011905":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6832/1608360619":"2024-04-18T08:53:08.504849+0000","172.21.15.187:0/3721653371":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/227074939":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/456496732":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/1663113056":"2024-04-18T08:53:08.504849+0000","172.21.15.187:6832/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:6833/1143620423":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/3973592351":"2024-04-18T08:53:00.117721+0000","172.21.15.187:0/4229015777":"2024-04-17T09:52:41.140106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 msgr2=0x7fcb4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 0x7fcb4c06eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb701696a0 tx=0x7fcb58008040 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 msgr2=0x7fcb700cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000b3f0 tx=0x7fcb6000b8b0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 shutdown_connections 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fcb4c06ca10 0x7fcb4c06eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb7015b570 0x7fcb700b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcb7006feb0 0x7fcb700cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 --2- 172.21.15.67:0/2315354672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb7006be70 0x7fcb700cf970 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 >> 172.21.15.67:0/2315354672 conn(0x7fcb70067d10 msgr2=0x7fcb700b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.071+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 shutdown_connections 2024-04-17T08:53:18.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.072+0000 7fcb6fe00640 1 -- 172.21.15.67:0/2315354672 wait complete. 2024-04-17T08:53:18.091 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:53:18.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:53:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.307+0000 7f0285a00640 1 Processor -- start 2024-04-17T08:53:18.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.308+0000 7f0285a00640 1 -- start start 2024-04-17T08:53:18.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.309+0000 7f0285a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f02800d33d0 0x7f0280161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.309+0000 7f0285a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02801620e0 0x7f02800cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.309+0000 7f0285a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02801624f0 con 0x7f02800d2ff0 2024-04-17T08:53:18.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.309+0000 7f0285a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02800b4e40 con 0x7f02801620e0 2024-04-17T08:53:18.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.309+0000 7f0285a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f02800b5110 con 0x7f02800d33d0 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02801620e0 0x7f02800cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02801620e0 0x7f02800cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54474/0 (socket says 172.21.15.67:54474) 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 -- 172.21.15.67:0/2567198256 learned_addr learned my addr 172.21.15.67:0/2567198256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f027f400640 1 --2- 172.21.15.67:0/2567198256 >> v2:172.21.15.187:3300/0 conn(0x7f02800d33d0 0x7f0280161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 -- 172.21.15.67:0/2567198256 >> v2:172.21.15.187:3300/0 conn(0x7f02800d33d0 msgr2=0x7f0280161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 --2- 172.21.15.67:0/2567198256 >> v2:172.21.15.187:3300/0 conn(0x7f02800d33d0 0x7f0280161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 -- 172.21.15.67:0/2567198256 >> v1:172.21.15.187:6789/0 conn(0x7f02800d2ff0 legacy=0x7f02800b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:18.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.310+0000 7f0285000640 1 -- 172.21.15.67:0/2567198256 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800b53f0 con 0x7f02801620e0 2024-04-17T08:53:18.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.311+0000 7f0285000640 1 --2- 172.21.15.67:0/2567198256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02801620e0 0x7f02800cfc40 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f0280068500 tx=0x7f027802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7da1cafa1bf632 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f027ea00640 1 -- 172.21.15.67:0/2567198256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278035020 con 0x7f02801620e0 2024-04-17T08:53:18.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f027ea00640 1 -- 172.21.15.67:0/2567198256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0278031640 con 0x7f02801620e0 2024-04-17T08:53:18.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f027ea00640 1 -- 172.21.15.67:0/2567198256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027802d040 con 0x7f02801620e0 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 -- 172.21.15.67:0/2567198256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02801620e0 msgr2=0x7f02800cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 --2- 172.21.15.67:0/2567198256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02801620e0 0x7f02800cfc40 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f0280068500 tx=0x7f027802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 -- 172.21.15.67:0/2567198256 shutdown_connections 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 --2- 172.21.15.67:0/2567198256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02801620e0 0x7f02800cfc40 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 --2- 172.21.15.67:0/2567198256 >> v2:172.21.15.187:3300/0 conn(0x7f02800d33d0 0x7f0280161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 -- 172.21.15.67:0/2567198256 >> 172.21.15.67:0/2567198256 conn(0x7f028009da80 msgr2=0x7f028008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.312+0000 7f0285a00640 1 -- 172.21.15.67:0/2567198256 shutdown_connections 2024-04-17T08:53:18.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.313+0000 7f0285a00640 1 -- 172.21.15.67:0/2567198256 wait complete. 2024-04-17T08:53:18.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.314+0000 7f0285a00640 1 Processor -- start 2024-04-17T08:53:18.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.315+0000 7f0285a00640 1 -- start start 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.315+0000 7f0285a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02800d2ff0 0x7f02800b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02801620e0 0x7f02800b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02800cf4c0 con 0x7f02800d33d0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f02800cf340 con 0x7f02801620e0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f02800d26d0 con 0x7f02800d2ff0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54488/0 (socket says 172.21.15.67:54488) 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 -- 172.21.15.67:0/3977562240 learned_addr learned my addr 172.21.15.67:0/3977562240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027fe00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02800d2ff0 0x7f02800b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f0285000640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02801620e0 0x7f02800b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 -- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02800d2ff0 msgr2=0x7f02800b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02800d2ff0 0x7f02800b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 -- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02801620e0 msgr2=0x7f02800b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02801620e0 0x7f02800b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.316+0000 7f027f400640 1 -- 172.21.15.67:0/3977562240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800b8db0 con 0x7f02800d33d0 2024-04-17T08:53:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f027f400640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f0270007ef0 tx=0x7f027000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270014620 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0270019070 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02800ca3a0 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270014a00 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.317+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02800b7d10 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.318+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02800b7ee0 con 0x7f02800d33d0 2024-04-17T08:53:18.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.319+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f0270014ba0 con 0x7f02800d33d0 2024-04-17T08:53:18.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.321+0000 7f0277e00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 0x7f025c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.321+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f027008c6b0 con 0x7f02800d33d0 2024-04-17T08:53:18.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.321+0000 7f027fe00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 0x7f025c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.322+0000 7f027fe00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 0x7f025c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0268004ea0 tx=0x7f0268009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.325+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f02800b8db0 con 0x7f02800d33d0 2024-04-17T08:53:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.529+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02800755d0 con 0x7f02800d33d0 2024-04-17T08:53:18.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.530+0000 7f0277e00640 1 -- 172.21.15.67:0/3977562240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v277) ==== 93+0+10 (secure 0 0 0) 0x7f027005a940 con 0x7f02800d33d0 2024-04-17T08:53:18.530 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 msgr2=0x7f025c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 0x7f025c06eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0268004ea0 tx=0x7f0268009040 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 msgr2=0x7f02800a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f0270007ef0 tx=0x7f027000e4b0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 shutdown_connections 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f025c06ca10 0x7f025c06eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02801620e0 0x7f02800b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02800d33d0 0x7f02800a1fb0 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 --2- 172.21.15.67:0/3977562240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f02800d2ff0 0x7f02800b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 >> 172.21.15.67:0/3977562240 conn(0x7f028009da80 msgr2=0x7f0280160eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 shutdown_connections 2024-04-17T08:53:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.533+0000 7f0285a00640 1 -- 172.21.15.67:0/3977562240 wait complete. 2024-04-17T08:53:18.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-17T08:53:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.773+0000 7f532b200640 1 Processor -- start 2024-04-17T08:53:18.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.774+0000 7f532b200640 1 -- start start 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532b200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f53240b6d00 0x7f53240cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53240cf910 0x7f53240b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53240d2d30 con 0x7f532415ec60 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f53240d49c0 con 0x7f53240cf910 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532b200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f53240d3660 con 0x7f53240b6d00 2024-04-17T08:53:18.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f5329e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f532415ec60 0x7f53240b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43184/0 (socket says 172.21.15.67:43184) 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f5329e00640 1 -- 172.21.15.67:0/481017521 learned_addr learned my addr 172.21.15.67:0/481017521 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.775+0000 7f532a800640 1 --2- 172.21.15.67:0/481017521 >> v2:172.21.15.67:3300/0 conn(0x7f53240cf910 0x7f53240b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.776+0000 7f5329400640 1 --2- 172.21.15.67:0/481017521 >> v2:172.21.15.187:3300/0 conn(0x7f53240b6d00 0x7f53240cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.776+0000 7f532a800640 1 -- 172.21.15.67:0/481017521 >> v2:172.21.15.187:3300/0 conn(0x7f53240b6d00 msgr2=0x7f53240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.776+0000 7f532a800640 1 --2- 172.21.15.67:0/481017521 >> v2:172.21.15.187:3300/0 conn(0x7f53240b6d00 0x7f53240cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.776+0000 7f532a800640 1 -- 172.21.15.67:0/481017521 >> v1:172.21.15.67:6789/0 conn(0x7f532415ec60 legacy=0x7f53240b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.776+0000 7f532a800640 1 -- 172.21.15.67:0/481017521 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53240d3010 con 0x7f53240cf910 2024-04-17T08:53:18.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.777+0000 7f532a800640 1 --2- 172.21.15.67:0/481017521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53240b7b70 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f53240ba590 tx=0x7f532002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db69a0ba79cb56f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f5328a00640 1 -- 172.21.15.67:0/481017521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320035020 con 0x7f53240cf910 2024-04-17T08:53:18.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f5328a00640 1 -- 172.21.15.67:0/481017521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5320031830 con 0x7f53240cf910 2024-04-17T08:53:18.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f5328a00640 1 -- 172.21.15.67:0/481017521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320031c00 con 0x7f53240cf910 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 -- 172.21.15.67:0/481017521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 msgr2=0x7f53240b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 --2- 172.21.15.67:0/481017521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53240b7b70 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f53240ba590 tx=0x7f532002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 -- 172.21.15.67:0/481017521 shutdown_connections 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 --2- 172.21.15.67:0/481017521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53240b7b70 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 --2- 172.21.15.67:0/481017521 >> v2:172.21.15.187:3300/0 conn(0x7f53240b6d00 0x7f53240cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 -- 172.21.15.67:0/481017521 >> 172.21.15.67:0/481017521 conn(0x7f53240d15f0 msgr2=0x7f53240d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 -- 172.21.15.67:0/481017521 shutdown_connections 2024-04-17T08:53:18.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.778+0000 7f532b200640 1 -- 172.21.15.67:0/481017521 wait complete. 2024-04-17T08:53:18.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.780+0000 7f532b200640 1 Processor -- start 2024-04-17T08:53:18.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.780+0000 7f532b200640 1 -- start start 2024-04-17T08:53:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 0x7f53241e97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 0x7f53241f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f532415f040 con 0x7f53240cf910 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53240cfdc0 con 0x7f532415ec60 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.781+0000 7f532b200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53240d00c0 con 0x7f53240b6d00 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54502/0 (socket says 172.21.15.67:54502) 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 -- 172.21.15.67:0/3464785917 learned_addr learned my addr 172.21.15.67:0/3464785917 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329e00640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 0x7f53241e97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f532a800640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 0x7f53241f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 -- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 msgr2=0x7f53241e97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 0x7f53241e97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 -- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 msgr2=0x7f53241f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 0x7f53241f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329400640 1 -- 172.21.15.67:0/3464785917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53241f5400 con 0x7f53240cf910 2024-04-17T08:53:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f5329e00640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 0x7f53241e97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.782+0000 7f532a800640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 0x7f53241f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f5329400640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f531400d830 tx=0x7f531400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400b6c0 con 0x7f53240cf910 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5314019070 con 0x7f53240cf910 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400ba80 con 0x7f53240cf910 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241f56e0 con 0x7f53240cf910 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.783+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53240cc490 con 0x7f53240cf910 2024-04-17T08:53:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.784+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5324068a10 con 0x7f53240cf910 2024-04-17T08:53:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.785+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f531401e460 con 0x7f53240cf910 2024-04-17T08:53:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.786+0000 7f531a000640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 0x7f530806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.786+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f531408c4d0 con 0x7f53240cf910 2024-04-17T08:53:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.787+0000 7f5329e00640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 0x7f530806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:18.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.788+0000 7f5329e00640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 0x7f530806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f531c0012e0 tx=0x7f531c009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:18.791+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f531405a6d0 con 0x7f53240cf910 2024-04-17T08:53:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.071+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7f53240b5450 con 0x7f53240cf910 2024-04-17T08:53:19.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.620+0000 7f531a000640 1 -- 172.21.15.67:0/3464785917 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3327) ==== 204+0+0 (secure 0 0 0) 0x7f5314059fd0 con 0x7f53240cf910 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 msgr2=0x7f530806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 0x7f530806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f531c0012e0 tx=0x7f531c009040 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 msgr2=0x7f53241e9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f531400d830 tx=0x7f531400dcf0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 shutdown_connections 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f530806ca10 0x7f530806eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f532415ec60 0x7f53241f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53240cf910 0x7f53241e9d00 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 --2- 172.21.15.67:0/3464785917 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53240b6d00 0x7f53241e97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.624+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 >> 172.21.15.67:0/3464785917 conn(0x7f53240d15f0 msgr2=0x7f532415d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.625+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 shutdown_connections 2024-04-17T08:53:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.625+0000 7f532b200640 1 -- 172.21.15.67:0/3464785917 wait complete. 2024-04-17T08:53:19.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T08:53:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.868+0000 7f66ab200640 1 Processor -- start 2024-04-17T08:53:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.868+0000 7f66ab200640 1 -- start start 2024-04-17T08:53:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.869+0000 7f66ab200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac0d33e0 0x7f66ac0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.869+0000 7f66ab200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.869+0000 7f66ab200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ac0d00b0 con 0x7f66ac15ecb0 2024-04-17T08:53:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.869+0000 7f66ab200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f66ac0d0c30 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.869+0000 7f66ab200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66ac0d0390 con 0x7f66ac0d33e0 2024-04-17T08:53:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.870+0000 7f66aa800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.870+0000 7f66a9400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac0d33e0 0x7f66ac0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.870+0000 7f66aa800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:46534/0 (socket says 172.21.15.67:46534) 2024-04-17T08:53:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.870+0000 7f66aa800640 1 -- 172.21.15.67:0/2676735903 learned_addr learned my addr 172.21.15.67:0/2676735903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.870+0000 7f66a9e00640 1 --1- 172.21.15.67:0/2676735903 >> v1:172.21.15.187:6789/0 conn(0x7f66ac15ecb0 0x7f66ac0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:59052/0 (socket says 172.21.15.67:59052) 2024-04-17T08:53:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.871+0000 7f66aa800640 1 -- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac0d33e0 msgr2=0x7f66ac0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.871+0000 7f66aa800640 1 --2- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac0d33e0 0x7f66ac0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.871+0000 7f66aa800640 1 -- 172.21.15.67:0/2676735903 >> v1:172.21.15.187:6789/0 conn(0x7f66ac15ecb0 legacy=0x7f66ac0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.871+0000 7f66aa800640 1 -- 172.21.15.67:0/2676735903 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac0d0670 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.871+0000 7f66a8a00640 1 -- 172.21.15.67:0/2676735903 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031415310 0 0) 0x7f66ac0d00b0 con 0x7f66ac15ecb0 2024-04-17T08:53:19.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.873+0000 7f66aa800640 1 --2- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009e50 tx=0x7f66a402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327dbac4eeec89b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.873+0000 7f66a8a00640 1 -- 172.21.15.67:0/2676735903 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a402ca70 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.873+0000 7f66a8a00640 1 -- 172.21.15.67:0/2676735903 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a402cc10 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.873+0000 7f66a8a00640 1 -- 172.21.15.67:0/2676735903 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a40343f0 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 -- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 msgr2=0x7f66ac0b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 --2- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009e50 tx=0x7f66a402bed0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 -- 172.21.15.67:0/2676735903 shutdown_connections 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 --2- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b3fc0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 --2- 172.21.15.67:0/2676735903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac0d33e0 0x7f66ac0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 -- 172.21.15.67:0/2676735903 >> 172.21.15.67:0/2676735903 conn(0x7f66ac08bff0 msgr2=0x7f66ac0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 -- 172.21.15.67:0/2676735903 shutdown_connections 2024-04-17T08:53:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.874+0000 7f66ab200640 1 -- 172.21.15.67:0/2676735903 wait complete. 2024-04-17T08:53:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.876+0000 7f66ab200640 1 Processor -- start 2024-04-17T08:53:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.876+0000 7f66ab200640 1 -- start start 2024-04-17T08:53:19.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 0x7f66ac0c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac15f090 con 0x7f66ac0d33e0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac0cfd30 con 0x7f66ac0cf8e0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.877+0000 7f66ab200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f66ac15f210 con 0x7f66ac15ecb0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54504/0 (socket says 172.21.15.67:54504) 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 -- 172.21.15.67:0/2374407532 learned_addr learned my addr 172.21.15.67:0/2374407532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9e00640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66aa800640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 0x7f66ac0c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 -- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 msgr2=0x7f66ac0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 0x7f66ac0c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 -- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 msgr2=0x7f66ac0b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66a9400640 1 -- 172.21.15.67:0/2374407532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac0c9e40 con 0x7f66ac0d33e0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.878+0000 7f66aa800640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 0x7f66ac0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66a9400640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9b0 tx=0x7f669800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698017070 con 0x7f66ac0d33e0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f669800c9a0 con 0x7f66ac0d33e0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ac0ca120 con 0x7f66ac0d33e0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669800cd90 con 0x7f66ac0d33e0 2024-04-17T08:53:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.879+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ac0ca6e0 con 0x7f66ac0d33e0 2024-04-17T08:53:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.880+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ac068a10 con 0x7f66ac0d33e0 2024-04-17T08:53:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.881+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f669801a070 con 0x7f66ac0d33e0 2024-04-17T08:53:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.883+0000 7f66a2000640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 0x7f668c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.883+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f669808c2a0 con 0x7f66ac0d33e0 2024-04-17T08:53:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.883+0000 7f66a9e00640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 0x7f668c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.884+0000 7f66a9e00640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 0x7f668c06eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f669c000f60 tx=0x7f669c007d00 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:19.887+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f669805a420 con 0x7f66ac0d33e0 2024-04-17T08:53:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.092+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f66ac0b5a70 con 0x7f66ac0d33e0 2024-04-17T08:53:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.100+0000 7f66a2000640 1 -- 172.21.15.67:0/2374407532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) ==== 122+0+0 (secure 0 0 0) 0x7f6698059d20 con 0x7f66ac0d33e0 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 msgr2=0x7f668c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 0x7f668c06eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f669c000f60 tx=0x7f669c007d00 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 msgr2=0x7f66ac0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9b0 tx=0x7f669800ee70 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 shutdown_connections 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f668c06ca10 0x7f668c06eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f66ac15ecb0 0x7f66ac0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0d33e0 0x7f66ac0c9260 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 --2- 172.21.15.67:0/2374407532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66ac0cf8e0 0x7f66ac0b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 >> 172.21.15.67:0/2374407532 conn(0x7f66ac08bff0 msgr2=0x7f66ac06daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 shutdown_connections 2024-04-17T08:53:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.103+0000 7f66ab200640 1 -- 172.21.15.67:0/2374407532 wait complete. 2024-04-17T08:53:20.121 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:53:20.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:53:20.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:53:20.354 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:53:20.355 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:20.355 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:20.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:53:20.417 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:53:20.417 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:20.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:53:20.474 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:20.474 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:20.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:53:20.535 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:53:20.535 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:20.593 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:53:20.594 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T08:53:20.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:20.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.823+0000 7f5f8f200640 1 Processor -- start 2024-04-17T08:53:20.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.824+0000 7f5f8f200640 1 -- start start 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.825+0000 7f5f8f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f880cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.825+0000 7f5f8f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f88160280 0x7f5f8806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.825+0000 7f5f8f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f880cd4a0 con 0x7f5f88141ed0 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.825+0000 7f5f8f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5f8815ce30 con 0x7f5f88160280 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.825+0000 7f5f8f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5f881607f0 con 0x7f5f8806b8c0 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f88160280 0x7f5f8806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f88160280 0x7f5f8806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54512/0 (socket says 172.21.15.67:54512) 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 -- 172.21.15.67:0/1162016453 learned_addr learned my addr 172.21.15.67:0/1162016453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8de00640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f880cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 -- 172.21.15.67:0/1162016453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 msgr2=0x7f5f880cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f880cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 -- 172.21.15.67:0/1162016453 >> v1:172.21.15.67:6789/0 conn(0x7f5f88141ed0 legacy=0x7f5f8815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.826+0000 7f5f8d400640 1 -- 172.21.15.67:0/1162016453 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f88160ad0 con 0x7f5f88160280 2024-04-17T08:53:20.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.827+0000 7f5f8d400640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f8806bca0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009870 tx=0x7f5f7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e02be0b740d2f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:20.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.828+0000 7f5f8ca00640 1 -- 172.21.15.67:0/1162016453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034070 con 0x7f5f88160280 2024-04-17T08:53:20.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.828+0000 7f5f8ca00640 1 -- 172.21.15.67:0/1162016453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f78030840 con 0x7f5f88160280 2024-04-17T08:53:20.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.828+0000 7f5f8ca00640 1 -- 172.21.15.67:0/1162016453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78030c10 con 0x7f5f88160280 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 -- 172.21.15.67:0/1162016453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 msgr2=0x7f5f8806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f8806bca0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009870 tx=0x7f5f7802b980 comp rx=0 tx=0).stop 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 -- 172.21.15.67:0/1162016453 shutdown_connections 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f8806bca0 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 --2- 172.21.15.67:0/1162016453 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f880cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 -- 172.21.15.67:0/1162016453 >> 172.21.15.67:0/1162016453 conn(0x7f5f88067d10 msgr2=0x7f5f880b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 -- 172.21.15.67:0/1162016453 shutdown_connections 2024-04-17T08:53:20.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.829+0000 7f5f8f200640 1 -- 172.21.15.67:0/1162016453 wait complete. 2024-04-17T08:53:20.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.831+0000 7f5f8f200640 1 Processor -- start 2024-04-17T08:53:20.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.831+0000 7f5f8f200640 1 -- start start 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.832+0000 7f5f8f200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f88134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8f200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 0x7f5f88145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8806c1e0 con 0x7f5f88160280 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8f200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5f880cddf0 con 0x7f5f88141ed0 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8f200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5f88161090 con 0x7f5f8806b8c0 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54514/0 (socket says 172.21.15.67:54514) 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 -- 172.21.15.67:0/3645122953 learned_addr learned my addr 172.21.15.67:0/3645122953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8d400640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 0x7f5f88145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 -- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 msgr2=0x7f5f88134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8de00640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f88134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f88134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 -- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 msgr2=0x7f5f88145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 0x7f5f88145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8e800640 1 -- 172.21.15.67:0/3645122953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f88143c20 con 0x7f5f88160280 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.833+0000 7f5f8d400640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 0x7f5f88145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:20.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f8e800640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c710 tx=0x7f5f8400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f84016020 con 0x7f5f88160280 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f84006d50 con 0x7f5f88160280 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f84003bb0 con 0x7f5f88160280 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f88143f00 con 0x7f5f88160280 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.834+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f881444c0 con 0x7f5f88160280 2024-04-17T08:53:20.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.836+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f88068a10 con 0x7f5f88160280 2024-04-17T08:53:20.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.837+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f5f8400a850 con 0x7f5f88160280 2024-04-17T08:53:20.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.838+0000 7f5f82000640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 0x7f5f6c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:20.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.838+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7f5f8408b9e0 con 0x7f5f88160280 2024-04-17T08:53:20.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.838+0000 7f5f8de00640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 0x7f5f6c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:20.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.840+0000 7f5f8de00640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 0x7f5f6c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c006fd0 tx=0x7f5f7c008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:20.842+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5f84020020 con 0x7f5f88160280 2024-04-17T08:53:21.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.086+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f880b1900 con 0x7f5f88160280 2024-04-17T08:53:21.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.087+0000 7f5f82000640 1 -- 172.21.15.67:0/3645122953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+3881 (secure 0 0 0) 0x7f5f84019310 con 0x7f5f88160280 2024-04-17T08:53:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:21.087 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":145,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/1461771453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":1461771453},{"type":"v1","addr":"172.21.15.187:6836","nonce":1461771453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8540,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/3851061252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":3851061252},{"type":"v1","addr":"172.21.15.187:6837","nonce":3851061252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":9676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3831795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3831795314},{"type":"v1","addr":"172.21.15.67:6834","nonce":3831795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":12954,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3195352396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3195352396},{"type":"v1","addr":"172.21.15.67:6835","nonce":3195352396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[]} 2024-04-17T08:53:21.087 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 145 2024-04-17T08:53:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 msgr2=0x7f5f6c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 0x7f5f6c06eed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c006fd0 tx=0x7f5f7c008040 comp rx=0 tx=0).stop 2024-04-17T08:53:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 msgr2=0x7f5f88137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c710 tx=0x7f5f8400cbd0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 shutdown_connections 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5f6c06ca10 0x7f5f6c06eed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f88160280 0x7f5f88137110 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f88141ed0 0x7f5f88145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 --2- 172.21.15.67:0/3645122953 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f8806b8c0 0x7f5f88134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 >> 172.21.15.67:0/3645122953 conn(0x7f5f88067d10 msgr2=0x7f5f880b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 shutdown_connections 2024-04-17T08:53:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.090+0000 7f5f8f200640 1 -- 172.21.15.67:0/3645122953 wait complete. 2024-04-17T08:53:21.112 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T08:53:21.112 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T08:53:21.112 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:53:21.112 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:53:21.113 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T08:53:21.113 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T08:53:21.113 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:53:21.113 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:53:21.113 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T08:53:21.113 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T08:53:21.114 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:53:21.114 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:53:21.114 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T08:53:21.114 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T08:53:21.114 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:53:21.114 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:53:21.130 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:53:21.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T08:53:21.133 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:53:21.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T08:53:21.135 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:53:21.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T08:53:21.141 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:53:21.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T08:53:21.182 INFO:tasks.ceph.mds.c:Started 2024-04-17T08:53:21.182 DEBUG:teuthology.parallel:result is None 2024-04-17T08:53:21.188 INFO:tasks.ceph.mds.d:Started 2024-04-17T08:53:21.189 DEBUG:teuthology.parallel:result is None 2024-04-17T08:53:21.189 INFO:tasks.ceph.mds.a:Started 2024-04-17T08:53:21.189 DEBUG:teuthology.parallel:result is None 2024-04-17T08:53:21.195 INFO:tasks.ceph.mds.b:Started 2024-04-17T08:53:21.195 DEBUG:teuthology.parallel:result is None 2024-04-17T08:53:21.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T08:53:21.346 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T08:53:21.348 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T08:53:21.351 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:53:21.352 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-17T08:53:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.393+0000 7fc1dfe00640 1 Processor -- start 2024-04-17T08:53:21.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.393+0000 7fc1dfe00640 1 -- start start 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1dfe00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 0x7fc1e00710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1dfe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e00cb940 con 0x7fc1e006be00 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1dfe00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e01624a0 con 0x7fc1e01620c0 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e00b4650 con 0x7fc1e0071610 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.394+0000 7fc1df400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1e006be00 0x7fc1e0162690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43206/0 (socket says 172.21.15.67:43206) 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1df400640 1 -- 172.21.15.67:0/1490672951 learned_addr learned my addr 172.21.15.67:0/1490672951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1dea00640 1 --2- 172.21.15.67:0/1490672951 >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 0x7fc1e00710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1d6000640 1 --2- 172.21.15.67:0/1490672951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1d6000640 1 -- 172.21.15.67:0/1490672951 >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 msgr2=0x7fc1e00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1d6000640 1 --2- 172.21.15.67:0/1490672951 >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 0x7fc1e00710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1d6000640 1 -- 172.21.15.67:0/1490672951 >> v1:172.21.15.67:6789/0 conn(0x7fc1e006be00 legacy=0x7fc1e0162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1d6000640 1 -- 172.21.15.67:0/1490672951 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e00b4930 con 0x7fc1e0071610 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1dea00640 1 --2- 172.21.15.67:0/1490672951 >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 0x7fc1e00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:53:21.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.395+0000 7fc1de000640 1 -- 172.21.15.67:0/1490672951 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11432885 0 0) 0x7fc1e00cb940 con 0x7fc1e006be00 2024-04-17T08:53:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.396+0000 7fc1d6000640 1 --2- 172.21.15.67:0/1490672951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e006b8c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80097a0 tx=0x7fc1c802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7c57c0b0d923576 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.397+0000 7fc1de000640 1 -- 172.21.15.67:0/1490672951 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8031780 con 0x7fc1e0071610 2024-04-17T08:53:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.397+0000 7fc1de000640 1 -- 172.21.15.67:0/1490672951 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c8037070 con 0x7fc1e0071610 2024-04-17T08:53:21.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.397+0000 7fc1de000640 1 -- 172.21.15.67:0/1490672951 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8031b40 con 0x7fc1e0071610 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.397+0000 7fc1dfe00640 1 -- 172.21.15.67:0/1490672951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 msgr2=0x7fc1e006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.397+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/1490672951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e006b8c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80097a0 tx=0x7fc1c802b820 comp rx=0 tx=0).stop 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 -- 172.21.15.67:0/1490672951 shutdown_connections 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/1490672951 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e006b8c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/1490672951 >> v2:172.21.15.187:3300/0 conn(0x7fc1e01620c0 0x7fc1e00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 -- 172.21.15.67:0/1490672951 >> 172.21.15.67:0/1490672951 conn(0x7fc1e006f060 msgr2=0x7fc1e006f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 -- 172.21.15.67:0/1490672951 shutdown_connections 2024-04-17T08:53:21.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.398+0000 7fc1dfe00640 1 -- 172.21.15.67:0/1490672951 wait complete. 2024-04-17T08:53:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.399+0000 7fc1dfe00640 1 Processor -- start 2024-04-17T08:53:21.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.400+0000 7fc1dfe00640 1 -- start start 2024-04-17T08:53:21.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.400+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.400+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e015df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.400+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1e01620c0 0x7fc1e0156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e00b51d0 con 0x7fc1e006be00 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e00b4ef0 con 0x7fc1e01620c0 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e00739f0 con 0x7fc1e0071610 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54582/0 (socket says 172.21.15.67:54582) 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 -- 172.21.15.67:0/4115792725 learned_addr learned my addr 172.21.15.67:0/4115792725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1df400640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1e01620c0 0x7fc1e0156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1d6000640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e015df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 -- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 msgr2=0x7fc1e015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e015df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 -- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1e01620c0 msgr2=0x7fc1e0156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1e01620c0 0x7fc1e0156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.401+0000 7fc1dea00640 1 -- 172.21.15.67:0/4115792725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e0158ce0 con 0x7fc1e006be00 2024-04-17T08:53:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1dea00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000ebe0 tx=0x7fc1d000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0017070 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d000cc30 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0013470 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e0158fc0 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.402+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e0159580 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.403+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e0068a10 con 0x7fc1e006be00 2024-04-17T08:53:21.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.404+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fc1d001a070 con 0x7fc1e006be00 2024-04-17T08:53:21.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.405+0000 7fc1d7400640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 0x7fc1bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:21.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.405+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 5831+0+0 (secure 0 0 0) 0x7fc1d008bf10 con 0x7fc1e006be00 2024-04-17T08:53:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.406+0000 7fc1d6000640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 0x7fc1bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.407+0000 7fc1d6000640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 0x7fc1bc06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8036000 tx=0x7fc1c803d040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.410+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc1d0059bf0 con 0x7fc1e006be00 2024-04-17T08:53:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:21.627+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc1e00cc940 con 0x7fc1e006be00 2024-04-17T08:53:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.663+0000 7fc1d7400640 1 -- 172.21.15.67:0/4115792725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v282) ==== 97+0+0 (secure 0 0 0) 0x7fc1d005def0 con 0x7fc1e006be00 2024-04-17T08:53:22.662 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T08:53:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.666+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 msgr2=0x7fc1bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.666+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 0x7fc1bc06eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8036000 tx=0x7fc1c803d040 comp rx=0 tx=0).stop 2024-04-17T08:53:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.666+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 msgr2=0x7fc1e014e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.666+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000ebe0 tx=0x7fc1d000c520 comp rx=0 tx=0).stop 2024-04-17T08:53:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 shutdown_connections 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fc1bc06ca10 0x7fc1bc06eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1e01620c0 0x7fc1e0156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1e0071610 0x7fc1e015df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 --2- 172.21.15.67:0/4115792725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e006be00 0x7fc1e014e8a0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 >> 172.21.15.67:0/4115792725 conn(0x7fc1e006f060 msgr2=0x7fc1e00ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 shutdown_connections 2024-04-17T08:53:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.667+0000 7fc1dfe00640 1 -- 172.21.15.67:0/4115792725 wait complete. 2024-04-17T08:53:22.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T08:53:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.914+0000 7f06ab800640 1 Processor -- start 2024-04-17T08:53:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.914+0000 7f06ab800640 1 -- start start 2024-04-17T08:53:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40cd040 0x7f06a40cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f06a40ce610 0x7f06a40b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06a415ed00 0x7f06a40b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06a40cd960 con 0x7f06a415ed00 2024-04-17T08:53:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f06a4071f70 con 0x7f06a40ce610 2024-04-17T08:53:22.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06ab800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06a4072250 con 0x7f06a40cd040 2024-04-17T08:53:22.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06a9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06a415ed00 0x7f06a40b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.916+0000 7f06a9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06a415ed00 0x7f06a40b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54590/0 (socket says 172.21.15.67:54590) 2024-04-17T08:53:22.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 -- 172.21.15.67:0/476478813 learned_addr learned my addr 172.21.15.67:0/476478813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a3e00640 1 --2- 172.21.15.67:0/476478813 >> v2:172.21.15.187:3300/0 conn(0x7f06a40ce610 0x7f06a40b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a8e00640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40cd040 0x7f06a40cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 -- 172.21.15.67:0/476478813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40cd040 msgr2=0x7f06a40cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40cd040 0x7f06a40cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 -- 172.21.15.67:0/476478813 >> v2:172.21.15.187:3300/0 conn(0x7f06a40ce610 msgr2=0x7f06a40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 --2- 172.21.15.67:0/476478813 >> v2:172.21.15.187:3300/0 conn(0x7f06a40ce610 0x7f06a40b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.917+0000 7f06a9800640 1 -- 172.21.15.67:0/476478813 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a4072530 con 0x7f06a415ed00 2024-04-17T08:53:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.918+0000 7f06a9800640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a40b0960 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f06a4073680 tx=0x7f069c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fa3a68e63b02bc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.918+0000 7f06a3400640 1 -- 172.21.15.67:0/476478813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c02ca80 con 0x7f06a415ed00 2024-04-17T08:53:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.918+0000 7f06a3400640 1 -- 172.21.15.67:0/476478813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c02cc20 con 0x7f06a415ed00 2024-04-17T08:53:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.918+0000 7f06a3400640 1 -- 172.21.15.67:0/476478813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0343f0 con 0x7f06a415ed00 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 -- 172.21.15.67:0/476478813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 msgr2=0x7f06a40b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a40b0960 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f06a4073680 tx=0x7f069c02bee0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 -- 172.21.15.67:0/476478813 shutdown_connections 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a40b0960 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 --2- 172.21.15.67:0/476478813 >> v2:172.21.15.187:3300/0 conn(0x7f06a40ce610 0x7f06a40b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 --2- 172.21.15.67:0/476478813 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40cd040 0x7f06a40cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 -- 172.21.15.67:0/476478813 >> 172.21.15.67:0/476478813 conn(0x7f06a406b8c0 msgr2=0x7f06a406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 -- 172.21.15.67:0/476478813 shutdown_connections 2024-04-17T08:53:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.919+0000 7f06ab800640 1 -- 172.21.15.67:0/476478813 wait complete. 2024-04-17T08:53:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.921+0000 7f06ab800640 1 Processor -- start 2024-04-17T08:53:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.921+0000 7f06ab800640 1 -- start start 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 0x7f06a4145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40ce610 0x7f06a415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06a40b2100 con 0x7f06a415ed00 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06a4072af0 con 0x7f06a40cd040 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06ab800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06a40b0ea0 con 0x7f06a40ce610 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.922+0000 7f06a9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54592/0 (socket says 172.21.15.67:54592) 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 -- 172.21.15.67:0/2610318111 learned_addr learned my addr 172.21.15.67:0/2610318111 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a8e00640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 0x7f06a4145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 -- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40ce610 msgr2=0x7f06a415fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40ce610 0x7f06a415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 -- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 msgr2=0x7f06a4145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 0x7f06a4145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 -- 172.21.15.67:0/2610318111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a41603d0 con 0x7f06a415ed00 2024-04-17T08:53:22.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a8e00640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 0x7f06a4145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a9800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f069c009f40 tx=0x7f069c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.923+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c043020 con 0x7f06a415ed00 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.924+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c03e450 con 0x7f06a415ed00 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.924+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c009510 con 0x7f06a415ed00 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.924+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a41606b0 con 0x7f06a415ed00 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.924+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a4160c70 con 0x7f06a415ed00 2024-04-17T08:53:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.925+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a4068b90 con 0x7f06a415ed00 2024-04-17T08:53:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.926+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f069c046020 con 0x7f06a415ed00 2024-04-17T08:53:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.927+0000 7f06a0c00640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 0x7f068406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.927+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4885+0+0 (secure 0 0 0) 0x7f069c0ae2a0 con 0x7f06a415ed00 2024-04-17T08:53:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.928+0000 7f06a8e00640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 0x7f068406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:22.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.929+0000 7f06a8e00640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 0x7f068406eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0698006fd0 tx=0x7f0698008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:22.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:22.932+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f069c04d030 con 0x7f06a415ed00 2024-04-17T08:53:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.216+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f06a406bd40 con 0x7f06a415ed00 2024-04-17T08:53:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.217+0000 7f06a0c00640 1 -- 172.21.15.67:0/2610318111 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) ==== 80+0+5304 (secure 0 0 0) 0x7f069c07c850 con 0x7f06a415ed00 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzCPVGxAAgBAExhnyHpD9N/hDEVpuEQ==", 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mMdaEAhAAvQ8t4vVv6fzrJ7JWeHgGEQ==", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mqICULRAAnbHz3or5s8oYRi955yGfig==", 2024-04-17T08:53:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mILyZExAA0vW0Jk2jFvfv6INyK4mpog==", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB+dh9mXZ3YJBAAFkc8y+rJZpk3dEQn3FI4uA==", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCBdh9mkYChMRAAntp0a/MZG15DY4nm3pprVQ==", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCFdh9mLtnrEhAAEAWX2NKKTC45cOa/vypWkA==", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T08:53:23.221 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCIdh9m8nu7LhAAID3fWT7rkzuoRjmkBoji0A==", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCPdh9m8weRCxAAmMLOam28HcK9FGTT3LcMNw==", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCSdh9mHEYpFxAA4Oek2Ksjpkd2v914gZyhow==", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCVdh9mbSMsIhAAF6FJOx1BQgihrpvX9BDu7w==", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCZdh9m577cDxAAjFlxRKV/sk/SH2XW0KgGEQ==", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw==", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T08:53:23.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9mm+8PORAAi2z95fAcBU0tA6b8/foe6g==", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB6dh9m9NU2LRAAGHlXoIfW9Qg1tWmCNGIpNg==", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB7dh9mlXE4BxAAZR8rueVQ0JE10aALCS0ceA==", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T08:53:23.226 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB4dh9mMHwbDxAArYphrIWLzCd3XZdFGS2qjg==", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mg+goEBAAX9eL/W/y3C19pSA8gORCGw==", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQB5dh9mzWHQAxAAZOErUiYpCxjmvTmO5KLi2w==", 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T08:53:23.228 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 msgr2=0x7f068406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 0x7f068406eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0698006fd0 tx=0x7f0698008040 comp rx=0 tx=0).stop 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 msgr2=0x7f06a4145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f069c009f40 tx=0x7f069c004a40 comp rx=0 tx=0).stop 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 shutdown_connections 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f068406ca10 0x7f068406eed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06a415ed00 0x7f06a4145760 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06a40ce610 0x7f06a415fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 --2- 172.21.15.67:0/2610318111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40cd040 0x7f06a4145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 >> 172.21.15.67:0/2610318111 conn(0x7f06a406b8c0 msgr2=0x7f06a40ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 shutdown_connections 2024-04-17T08:53:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.220+0000 7f06ab800640 1 -- 172.21.15.67:0/2610318111 wait complete. 2024-04-17T08:53:23.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:23.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.447+0000 7f21f3e00640 1 Processor -- start 2024-04-17T08:53:23.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.447+0000 7f21f3e00640 1 -- start start 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.448+0000 7f21f3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21f415ee60 0x7f21f40af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.448+0000 7f21f3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f415f440 con 0x7f21f415c5a0 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f415f720 con 0x7f21f40af980 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21f40b4920 con 0x7f21f415ee60 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21f415c5a0 0x7f21f40b4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58234/0 (socket says 172.21.15.67:58234) 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f2a00640 1 -- 172.21.15.67:0/2547817655 learned_addr learned my addr 172.21.15.67:0/2547817655 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:23.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f2000640 1 --2- 172.21.15.67:0/2547817655 >> v2:172.21.15.67:3300/0 conn(0x7f21f415ee60 0x7f21f40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f1600640 1 -- 172.21.15.67:0/2547817655 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388000210 0 0) 0x7f21f415f440 con 0x7f21f415c5a0 2024-04-17T08:53:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.449+0000 7f21f1600640 1 -- 172.21.15.67:0/2547817655 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21d8003520 con 0x7f21f415c5a0 2024-04-17T08:53:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.450+0000 7f21f2000640 1 -- 172.21.15.67:0/2547817655 >> v1:172.21.15.187:6789/0 conn(0x7f21f40af980 legacy=0x7f21f40ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.450+0000 7f21f2000640 1 -- 172.21.15.67:0/2547817655 >> v1:172.21.15.67:6789/0 conn(0x7f21f415c5a0 legacy=0x7f21f40b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.450+0000 7f21f2000640 1 -- 172.21.15.67:0/2547817655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f415fa00 con 0x7f21f415ee60 2024-04-17T08:53:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.451+0000 7f21f2000640 1 --2- 172.21.15.67:0/2547817655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415ee60 0x7f21f40af440 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097a0 tx=0x7f21e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da63768b066cd5ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.451+0000 7f21f1600640 1 -- 172.21.15.67:0/2547817655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e00315a0 con 0x7f21f415ee60 2024-04-17T08:53:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.451+0000 7f21f1600640 1 -- 172.21.15.67:0/2547817655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e0037070 con 0x7f21f415ee60 2024-04-17T08:53:23.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.451+0000 7f21f1600640 1 -- 172.21.15.67:0/2547817655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0031960 con 0x7f21f415ee60 2024-04-17T08:53:23.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 -- 172.21.15.67:0/2547817655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415ee60 msgr2=0x7f21f40af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2547817655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415ee60 0x7f21f40af440 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097a0 tx=0x7f21e002b840 comp rx=0 tx=0).stop 2024-04-17T08:53:23.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 -- 172.21.15.67:0/2547817655 shutdown_connections 2024-04-17T08:53:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 --2- 172.21.15.67:0/2547817655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415ee60 0x7f21f40af440 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 -- 172.21.15.67:0/2547817655 >> 172.21.15.67:0/2547817655 conn(0x7f21f4067d10 msgr2=0x7f21f40b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 -- 172.21.15.67:0/2547817655 shutdown_connections 2024-04-17T08:53:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.452+0000 7f21f3e00640 1 -- 172.21.15.67:0/2547817655 wait complete. 2024-04-17T08:53:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.454+0000 7f21f3e00640 1 Processor -- start 2024-04-17T08:53:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.454+0000 7f21f3e00640 1 -- start start 2024-04-17T08:53:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 0x7f21f4150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21f415ee60 0x7f21f4156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40adab0 con 0x7f21f415c5a0 2024-04-17T08:53:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40afdd0 con 0x7f21f415ee60 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f21f40adc30 con 0x7f21f40af980 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46508/0 (socket says 172.21.15.67:46508) 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.455+0000 7f21f2000640 1 -- 172.21.15.67:0/1243649566 learned_addr learned my addr 172.21.15.67:0/1243649566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2a00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 0x7f21f4150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f3400640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21f415ee60 0x7f21f4156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 -- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 msgr2=0x7f21f4150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 0x7f21f4150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 -- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21f415ee60 msgr2=0x7f21f4156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21f415ee60 0x7f21f4156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 -- 172.21.15.67:0/1243649566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f4158380 con 0x7f21f415c5a0 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2a00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 0x7f21f4150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21f2000640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f21e0000c00 tx=0x7f21e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.456+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0032630 con 0x7f21f415c5a0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.457+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e00327f0 con 0x7f21f415c5a0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.457+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f4158660 con 0x7f21f415c5a0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.457+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0032be0 con 0x7f21f415c5a0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.457+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f41514a0 con 0x7f21f415c5a0 2024-04-17T08:53:23.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.458+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f4068a10 con 0x7f21f415c5a0 2024-04-17T08:53:23.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.459+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f21e004d3f0 con 0x7f21f415c5a0 2024-04-17T08:53:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.460+0000 7f21eaa00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 0x7f21d406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.460+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4885+0+0 (secure 0 0 0) 0x7f21e002f030 con 0x7f21f415c5a0 2024-04-17T08:53:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.461+0000 7f21f2a00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 0x7f21d406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.462+0000 7f21f2a00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 0x7f21d406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f21e40011e0 tx=0x7f21e4009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.465+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f21e0040070 con 0x7f21f415c5a0 2024-04-17T08:53:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.672+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21f41589f0 con 0x7f21f415c5a0 2024-04-17T08:53:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.673+0000 7f21eaa00640 1 -- 172.21.15.67:0/1243649566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v282) ==== 74+0+12897 (secure 0 0 0) 0x7f21e007bd30 con 0x7f21f415c5a0 2024-04-17T08:53:23.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:23.673 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":282,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:22.660572+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.676+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 msgr2=0x7f21d406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.676+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 0x7f21d406eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f21e40011e0 tx=0x7f21e4009040 comp rx=0 tx=0).stop 2024-04-17T08:53:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 msgr2=0x7f21f415e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f21e0000c00 tx=0x7f21e00046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 shutdown_connections 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f21d406ca10 0x7f21d406eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f21f415ee60 0x7f21f4156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21f415c5a0 0x7f21f415e2c0 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 --2- 172.21.15.67:0/1243649566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f21f40af980 0x7f21f4150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 >> 172.21.15.67:0/1243649566 conn(0x7f21f4067d10 msgr2=0x7f21f406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 shutdown_connections 2024-04-17T08:53:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.677+0000 7f21f3e00640 1 -- 172.21.15.67:0/1243649566 wait complete. 2024-04-17T08:53:23.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:53:23.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:53:23.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.923+0000 7f9993e00640 1 Processor -- start 2024-04-17T08:53:23.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.923+0000 7f9993e00640 1 -- start start 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9994144fb0 0x7f99940b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f999415fcb0 0x7f99940b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99940cf620 0x7f99940cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9994160110 con 0x7f99940cf620 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f999415f240 con 0x7f9994144fb0 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f99940cff80 con 0x7f999415fcb0 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.924+0000 7f9993400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99940cf620 0x7f99940cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99940cf620 0x7f99940cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46518/0 (socket says 172.21.15.67:46518) 2024-04-17T08:53:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 -- 172.21.15.67:0/1450320194 learned_addr learned my addr 172.21.15.67:0/1450320194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f998a000640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f999415fcb0 0x7f99940b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9992a00640 1 --2- 172.21.15.67:0/1450320194 >> v2:172.21.15.187:3300/0 conn(0x7f9994144fb0 0x7f99940b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 -- 172.21.15.67:0/1450320194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f999415fcb0 msgr2=0x7f99940b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f999415fcb0 0x7f99940b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 -- 172.21.15.67:0/1450320194 >> v2:172.21.15.187:3300/0 conn(0x7f9994144fb0 msgr2=0x7f99940b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 --2- 172.21.15.67:0/1450320194 >> v2:172.21.15.187:3300/0 conn(0x7f9994144fb0 0x7f99940b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.925+0000 7f9993400640 1 -- 172.21.15.67:0/1450320194 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99940b6420 con 0x7f99940cf620 2024-04-17T08:53:23.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.926+0000 7f9993400640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f99940cfa40 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f9994073520 tx=0x7f998c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a123db11ed96f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.926+0000 7f9992000640 1 -- 172.21.15.67:0/1450320194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c0317c0 con 0x7f99940cf620 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.926+0000 7f9992000640 1 -- 172.21.15.67:0/1450320194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f998c037070 con 0x7f99940cf620 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.926+0000 7f9992000640 1 -- 172.21.15.67:0/1450320194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c031b80 con 0x7f99940cf620 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 -- 172.21.15.67:0/1450320194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 msgr2=0x7f99940cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f99940cfa40 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f9994073520 tx=0x7f998c02b860 comp rx=0 tx=0).stop 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 -- 172.21.15.67:0/1450320194 shutdown_connections 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f99940cfa40 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 --2- 172.21.15.67:0/1450320194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f999415fcb0 0x7f99940b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 --2- 172.21.15.67:0/1450320194 >> v2:172.21.15.187:3300/0 conn(0x7f9994144fb0 0x7f99940b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 -- 172.21.15.67:0/1450320194 >> 172.21.15.67:0/1450320194 conn(0x7f99940b23c0 msgr2=0x7f99940b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 -- 172.21.15.67:0/1450320194 shutdown_connections 2024-04-17T08:53:23.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.927+0000 7f9993e00640 1 -- 172.21.15.67:0/1450320194 wait complete. 2024-04-17T08:53:23.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.929+0000 7f9993e00640 1 Processor -- start 2024-04-17T08:53:23.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.929+0000 7f9993e00640 1 -- start start 2024-04-17T08:53:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9994144fb0 0x7f9994143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f999415fcb0 0x7f9994143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99940b6700 con 0x7f99940cf620 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f999415f790 con 0x7f999415fcb0 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9993e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9994145390 con 0x7f9994144fb0 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9992a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9992a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46528/0 (socket says 172.21.15.67:46528) 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.930+0000 7f9992a00640 1 -- 172.21.15.67:0/1983969730 learned_addr learned my addr 172.21.15.67:0/1983969730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 -- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9994144fb0 msgr2=0x7f9994143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9994144fb0 0x7f9994143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9993400640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f999415fcb0 0x7f9994143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 -- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f999415fcb0 msgr2=0x7f9994143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f999415fcb0 0x7f9994143e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 -- 172.21.15.67:0/1983969730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999413b9f0 con 0x7f99940cf620 2024-04-17T08:53:23.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f9992a00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f998400ce10 tx=0x7f998400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.931+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9984003b90 con 0x7f99940cf620 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.932+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9984003d30 con 0x7f99940cf620 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.932+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9984004240 con 0x7f99940cf620 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.932+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999413bcd0 con 0x7f99940cf620 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.932+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999413c290 con 0x7f99940cf620 2024-04-17T08:53:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.933+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9994068a10 con 0x7f99940cf620 2024-04-17T08:53:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.934+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f99840043e0 con 0x7f99940cf620 2024-04-17T08:53:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.935+0000 7f998b400640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 0x7f997006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.935+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4885+0+0 (secure 0 0 0) 0x7f998408c690 con 0x7f99940cf620 2024-04-17T08:53:23.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.936+0000 7f998a000640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 0x7f997006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:23.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.937+0000 7f998a000640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 0x7f997006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99940b3dc0 tx=0x7f997c008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:23.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:23.939+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f998401d340 con 0x7f99940cf620 2024-04-17T08:53:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.144+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9994068610 con 0x7f99940cf620 2024-04-17T08:53:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.145+0000 7f998b400640 1 -- 172.21.15.67:0/1983969730 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v282) ==== 93+0+10 (secure 0 0 0) 0x7f998405abc0 con 0x7f99940cf620 2024-04-17T08:53:24.144 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.147+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 msgr2=0x7f997006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.147+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 0x7f997006eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99940b3dc0 tx=0x7f997c008040 comp rx=0 tx=0).stop 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.147+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 msgr2=0x7f999415aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.147+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f998400ce10 tx=0x7f998400a660 comp rx=0 tx=0).stop 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 shutdown_connections 2024-04-17T08:53:24.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f997006ca10 0x7f997006eed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f999415fcb0 0x7f9994143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9994144fb0 0x7f9994143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 --2- 172.21.15.67:0/1983969730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99940cf620 0x7f999415aec0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 >> 172.21.15.67:0/1983969730 conn(0x7f99940b23c0 msgr2=0x7f99940b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 shutdown_connections 2024-04-17T08:53:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.148+0000 7f9993e00640 1 -- 172.21.15.67:0/1983969730 wait complete. 2024-04-17T08:53:24.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T08:53:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.394+0000 7f7775000640 1 Processor -- start 2024-04-17T08:53:24.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.394+0000 7f7775000640 1 -- start start 2024-04-17T08:53:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.395+0000 7f7775000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f777009b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.395+0000 7f7775000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77700b3180 0x7f777009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.395+0000 7f7775000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777009bd80 con 0x7f777009afa0 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.395+0000 7f7775000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f77700d1640 con 0x7f77700b3180 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.395+0000 7f7775000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f777006c810 con 0x7f777006e320 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f777009afa0 0x7f7770162800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58250/0 (socket says 172.21.15.67:58250) 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776fe00640 1 -- 172.21.15.67:0/3733419724 learned_addr learned my addr 172.21.15.67:0/3733419724 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776f400640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f777009b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776ea00640 1 --2- 172.21.15.67:0/3733419724 >> v2:172.21.15.187:3300/0 conn(0x7f77700b3180 0x7f777009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764602240 0 0) 0x7f777009bd80 con 0x7f777009afa0 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.396+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7754003520 con 0x7f777009afa0 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1095568753 0 0) 0x7f7754003520 con 0x7f777009afa0 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 msgr2=0x7f777009b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f777009b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 >> v2:172.21.15.187:3300/0 conn(0x7f77700b3180 msgr2=0x7f777009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 --2- 172.21.15.67:0/3733419724 >> v2:172.21.15.187:3300/0 conn(0x7f77700b3180 0x7f777009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77700cdeb0 con 0x7f777009afa0 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7768001670 con 0x7f777009afa0 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 >> v1:172.21.15.67:6789/0 conn(0x7f777009afa0 legacy=0x7f7770162800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7754003a20 0x7f7754003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7754004800 0x7f7754004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f77540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77700d1640 con 0x7f7754003a20 2024-04-17T08:53:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f777006c810 con 0x7f7754004800 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f775400ba40 con 0x7f777006e320 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.397+0000 7f776f400640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7754003a20 0x7f7754003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776ea00640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7754004800 0x7f7754004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776fe00640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f77540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776f400640 1 -- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 msgr2=0x7f77540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776f400640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f777006e320 0x7f77540051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776f400640 1 -- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7754004800 msgr2=0x7f7754004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776f400640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7754004800 0x7f7754004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.398+0000 7f776f400640 1 -- 172.21.15.67:0/3733419724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7754003520 con 0x7f7754003a20 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.399+0000 7f776f400640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7754003a20 0x7f7754003e40 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f7760009fc0 tx=0x7f7754005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a16d2b1186accc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.399+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760039070 con 0x7f7754003a20 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.399+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776002cbf0 con 0x7f7754003a20 2024-04-17T08:53:24.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.399+0000 7f776e000640 1 -- 172.21.15.67:0/3733419724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760009c10 con 0x7f7754003a20 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 -- 172.21.15.67:0/3733419724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7754003a20 msgr2=0x7f7754003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 --2- 172.21.15.67:0/3733419724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7754003a20 0x7f7754003e40 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f7760009fc0 tx=0x7f7754005830 comp rx=0 tx=0).stop 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f776fe00640 1 -- 172.21.15.67:0/3733419724 reap_dead start 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 -- 172.21.15.67:0/3733419724 shutdown_connections 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 -- 172.21.15.67:0/3733419724 >> 172.21.15.67:0/3733419724 conn(0x7f77700b4170 msgr2=0x7f77700b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 -- 172.21.15.67:0/3733419724 shutdown_connections 2024-04-17T08:53:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.400+0000 7f7775000640 1 -- 172.21.15.67:0/3733419724 wait complete. 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.402+0000 7f7775000640 1 Processor -- start 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.402+0000 7f7775000640 1 -- start start 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7754003a20 0x7f7770160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77700b7ff0 0x7f77701f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77700b3880 con 0x7f77700cb390 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f77700ce470 con 0x7f77700b7ff0 2024-04-17T08:53:24.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f7775000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f77700b3a00 con 0x7f7754003a20 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f776fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f776fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46546/0 (socket says 172.21.15.67:46546) 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f776fe00640 1 -- 172.21.15.67:0/2952893438 learned_addr learned my addr 172.21.15.67:0/2952893438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.403+0000 7f776f400640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7754003a20 0x7f7770160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776ea00640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77700b7ff0 0x7f77701f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 -- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7754003a20 msgr2=0x7f7770160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7754003a20 0x7f7770160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 -- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77700b7ff0 msgr2=0x7f77701f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77700b7ff0 0x7f77701f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 -- 172.21.15.67:0/2952893438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77700b04b0 con 0x7f77700cb390 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f776fe00640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7768000c00 tx=0x7f776800d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.404+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768007440 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.405+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7768007600 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.405+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768004a20 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.405+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77700b6b50 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.405+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77700b6dd0 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.406+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7770068a10 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.407+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f776801b060 con 0x7f77700cb390 2024-04-17T08:53:24.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.408+0000 7f7767400640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 0x7f775006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.408+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4885+0+0 (secure 0 0 0) 0x7f776808b380 con 0x7f77700cb390 2024-04-17T08:53:24.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.409+0000 7f776f400640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 0x7f775006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.410+0000 7f776f400640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 0x7f775006ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7760036000 tx=0x7f776002e000 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.412+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f77700b6b50 con 0x7f77700cb390 2024-04-17T08:53:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.619+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f77700b4fa0 con 0x7f77700cb390 2024-04-17T08:53:24.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.620+0000 7f7767400640 1 -- 172.21.15.67:0/2952893438 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v282) ==== 84+0+65 (secure 0 0 0) 0x7f7768059820 con 0x7f77700cb390 2024-04-17T08:53:24.620 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:24.620 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:53:24.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:24.620 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T08:53:24.621 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T08:53:24.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:24.621 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 msgr2=0x7f775006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 0x7f775006ef00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7760036000 tx=0x7f776002e000 comp rx=0 tx=0).stop 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 msgr2=0x7f77700cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f7768000c00 tx=0x7f776800d9b0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 shutdown_connections 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f775006ca40 0x7f775006ef00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77700cb390 0x7f77700cb820 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77700b7ff0 0x7f77701f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 --2- 172.21.15.67:0/2952893438 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7754003a20 0x7f7770160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 >> 172.21.15.67:0/2952893438 conn(0x7f77700b4170 msgr2=0x7f7770070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 shutdown_connections 2024-04-17T08:53:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.623+0000 7f7775000640 1 -- 172.21.15.67:0/2952893438 wait complete. 2024-04-17T08:53:24.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T08:53:24.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T08:53:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.865+0000 7fb664a00640 1 Processor -- start 2024-04-17T08:53:24.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.865+0000 7fb664a00640 1 -- start start 2024-04-17T08:53:24.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.867+0000 7fb664a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb66015ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.867+0000 7fb664a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb66006de70 con 0x7fb660072930 2024-04-17T08:53:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.867+0000 7fb664a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb66015f850 con 0x7fb6600cd770 2024-04-17T08:53:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.867+0000 7fb664a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb66015fb30 con 0x7fb66015c670 2024-04-17T08:53:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.868+0000 7fb65de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb660072930 0x7fb6600cd060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58252/0 (socket says 172.21.15.67:58252) 2024-04-17T08:53:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.868+0000 7fb65de00640 1 -- 172.21.15.67:0/2879609634 learned_addr learned my addr 172.21.15.67:0/2879609634 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.868+0000 7fb65e800640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb66015ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.868+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425594696 0 0) 0x7fb66006de70 con 0x7fb660072930 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.868+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb644003520 con 0x7fb660072930 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3172231093 0 0) 0x7fb644003520 con 0x7fb660072930 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 msgr2=0x7fb66015ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb66015ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 >> v1:172.21.15.187:6789/0 conn(0x7fb6600cd770 legacy=0x7fb66006e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66015fe10 con 0x7fb660072930 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb64c001690 con 0x7fb660072930 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 >> v1:172.21.15.67:6789/0 conn(0x7fb660072930 legacy=0x7fb6600cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb644003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb644004800 0x7fb644004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb6440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb66015fb30 con 0x7fb6440039e0 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb64400ba70 con 0x7fb644004800 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb64400bc70 con 0x7fb66015c670 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65e800640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb644003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.869+0000 7fb65f200640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb6440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65e800640 1 -- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 msgr2=0x7fb6440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65e800640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015c670 0x7fb6440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65e800640 1 -- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb644004800 msgr2=0x7fb644004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65de00640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb644004800 0x7fb644004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65e800640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb644004800 0x7fb644004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.870+0000 7fb65e800640 1 -- 172.21.15.67:0/2879609634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb644003520 con 0x7fb6440039e0 2024-04-17T08:53:24.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.871+0000 7fb65e800640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb644003e00 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fb650009a90 tx=0x7fb644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e37e8dc98543099 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.871+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650039070 con 0x7fb6440039e0 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.871+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65002c680 con 0x7fb6440039e0 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.871+0000 7fb65d400640 1 -- 172.21.15.67:0/2879609634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650009690 con 0x7fb6440039e0 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 -- 172.21.15.67:0/2879609634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 msgr2=0x7fb644003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 --2- 172.21.15.67:0/2879609634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb644003e00 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fb650009a90 tx=0x7fb644005830 comp rx=0 tx=0).stop 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb65f200640 1 -- 172.21.15.67:0/2879609634 reap_dead start 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 -- 172.21.15.67:0/2879609634 shutdown_connections 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 -- 172.21.15.67:0/2879609634 >> 172.21.15.67:0/2879609634 conn(0x7fb66006b8c0 msgr2=0x7fb66006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 -- 172.21.15.67:0/2879609634 shutdown_connections 2024-04-17T08:53:24.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.872+0000 7fb664a00640 1 -- 172.21.15.67:0/2879609634 wait complete. 2024-04-17T08:53:24.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.874+0000 7fb664a00640 1 Processor -- start 2024-04-17T08:53:24.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.874+0000 7fb664a00640 1 -- start start 2024-04-17T08:53:24.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6600ad8d0 0x7fb6600add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015a6d0 0x7fb66015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6600b3fb0 con 0x7fb6440039e0 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb6600b3e30 con 0x7fb6600ad8d0 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb664a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb6600b4130 con 0x7fb66015a6d0 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46554/0 (socket says 172.21.15.67:46554) 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 -- 172.21.15.67:0/1682083281 learned_addr learned my addr 172.21.15.67:0/1682083281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 -- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015a6d0 msgr2=0x7fb66015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015a6d0 0x7fb66015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 -- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6600ad8d0 msgr2=0x7fb6600add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6600ad8d0 0x7fb6600add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:24.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.875+0000 7fb65e800640 1 -- 172.21.15.67:0/1682083281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66014d500 con 0x7fb6440039e0 2024-04-17T08:53:24.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb65e800640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fb650009a90 tx=0x7fb650004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650039050 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb650045070 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65002de60 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6600ae390 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.876+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6600ae950 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.877+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb660068a10 con 0x7fb6440039e0 2024-04-17T08:53:24.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.878+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fb650003a50 con 0x7fb6440039e0 2024-04-17T08:53:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.879+0000 7fb656a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 0x7fb64006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.879+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(282..282 src has 1..282) ==== 4885+0+0 (secure 0 0 0) 0x7fb6500ad450 con 0x7fb6440039e0 2024-04-17T08:53:24.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.880+0000 7fb65de00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 0x7fb64006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:24.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.881+0000 7fb65de00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 0x7fb64006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000f60 tx=0x7fb64c009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:24.884+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb65004e020 con 0x7fb6440039e0 2024-04-17T08:53:25.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.092+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb660068610 con 0x7fb6440039e0 2024-04-17T08:53:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.695+0000 7fb656a00640 1 -- 172.21.15.67:0/1682083281 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v283) ==== 138+0+0 (secure 0 0 0) 0x7fb66014d500 con 0x7fb6440039e0 2024-04-17T08:53:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T08:53:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 msgr2=0x7fb64006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 0x7fb64006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000f60 tx=0x7fb64c009040 comp rx=0 tx=0).stop 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 msgr2=0x7fb66014cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fb650009a90 tx=0x7fb650004730 comp rx=0 tx=0).stop 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 shutdown_connections 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fb64006ca10 0x7fb64006eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb66015a6d0 0x7fb66015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6600ad8d0 0x7fb6600add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 --2- 172.21.15.67:0/1682083281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6440039e0 0x7fb66014cfc0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.699+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 >> 172.21.15.67:0/1682083281 conn(0x7fb66006b8c0 msgr2=0x7fb660161750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.700+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 shutdown_connections 2024-04-17T08:53:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.700+0000 7fb664a00640 1 -- 172.21.15.67:0/1682083281 wait complete. 2024-04-17T08:53:25.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T08:53:25.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.951+0000 7f6786000640 1 Processor -- start 2024-04-17T08:53:25.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.951+0000 7f6786000640 1 -- start start 2024-04-17T08:53:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f6786000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f678006f7a0 0x7f678015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f6786000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67800d0030 con 0x7f67800b58d0 2024-04-17T08:53:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f6786000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67800d0310 con 0x7f6780144f90 2024-04-17T08:53:25.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f6786000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67800cfe10 con 0x7f678006f7a0 2024-04-17T08:53:25.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f677ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67800b58d0 0x7f678006f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58262/0 (socket says 172.21.15.67:58262) 2024-04-17T08:53:25.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.953+0000 7f677ea00640 1 -- 172.21.15.67:0/3183270226 learned_addr learned my addr 172.21.15.67:0/3183270226 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1281332415 0 0) 0x7f67800d0030 con 0x7f67800b58d0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6764005730 con 0x7f67800b58d0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677fe00640 1 --2- 172.21.15.67:0/3183270226 >> v2:172.21.15.187:3300/0 conn(0x7f678006f7a0 0x7f678015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1030749375 0 0) 0x7f6764005730 con 0x7f67800b58d0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 >> v2:172.21.15.187:3300/0 conn(0x7f678006f7a0 msgr2=0x7f678015fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 --2- 172.21.15.67:0/3183270226 >> v2:172.21.15.187:3300/0 conn(0x7f678006f7a0 0x7f678015fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 >> v1:172.21.15.187:6789/0 conn(0x7f6780144f90 legacy=0x7f67800cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678015f2e0 con 0x7f67800b58d0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f676c001670 con 0x7f67800b58d0 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 >> v1:172.21.15.67:6789/0 conn(0x7f67800b58d0 legacy=0x7f678006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 0x7f6764005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.954+0000 7f677e000640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6764006a10 0x7f6764006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677e000640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67640073d0 0x7f676400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67800cfe10 con 0x7f6764005bc0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f676400e320 con 0x7f67640073d0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f676400e520 con 0x7f6764006a10 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677fe00640 1 --2- 172.21.15.67:0/3183270226 >> v2:172.21.15.187:3300/0 conn(0x7f678006f7a0 0x7f678015fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 0x7f6764005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677ea00640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6764006a10 0x7f6764006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677fe00640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67640073d0 0x7f676400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 -- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6764006a10 msgr2=0x7f6764006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6764006a10 0x7f6764006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 -- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67640073d0 msgr2=0x7f676400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67640073d0 0x7f676400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677f400640 1 -- 172.21.15.67:0/3183270226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6764005730 con 0x7f6764005bc0 2024-04-17T08:53:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677fe00640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67640073d0 0x7f676400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:25.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677ea00640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6764006a10 0x7f6764006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:25.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.955+0000 7f677fe00640 1 -- 172.21.15.67:0/3183270226 reap_dead start 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f677f400640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 0x7f6764005fe0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f6770001750 tx=0x7f6764007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627f8dcbbe70d34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770001370 con 0x7f6764005bc0 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67700052d0 con 0x7f6764005bc0 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f677e000640 1 -- 172.21.15.67:0/3183270226 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770003830 con 0x7f6764005bc0 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f6786000640 1 -- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 msgr2=0x7f6764005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f6786000640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 0x7f6764005fe0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f6770001750 tx=0x7f6764007a40 comp rx=0 tx=0).stop 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f6786000640 1 -- 172.21.15.67:0/3183270226 shutdown_connections 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f6786000640 1 --2- 172.21.15.67:0/3183270226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764005bc0 0x7f6764005fe0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.957+0000 7f6786000640 1 -- 172.21.15.67:0/3183270226 >> 172.21.15.67:0/3183270226 conn(0x7f6780067c30 msgr2=0x7f678006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:25.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.958+0000 7f6786000640 1 -- 172.21.15.67:0/3183270226 shutdown_connections 2024-04-17T08:53:25.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.958+0000 7f6786000640 1 -- 172.21.15.67:0/3183270226 wait complete. 2024-04-17T08:53:25.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.959+0000 7f6786000640 1 Processor -- start 2024-04-17T08:53:25.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.960+0000 7f6786000640 1 -- start start 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.960+0000 7f6786000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6764005bc0 0x7f678015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.960+0000 7f6786000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67800b2a50 0x7f67800cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f6786000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f6786000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f678015fdc0 con 0x7f67800b2ed0 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f6786000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f678015f8a0 con 0x7f6764005bc0 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f6786000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f678015ff40 con 0x7f67800b2a50 2024-04-17T08:53:25.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46568/0 (socket says 172.21.15.67:46568) 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 -- 172.21.15.67:0/2661538820 learned_addr learned my addr 172.21.15.67:0/2661538820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677f400640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6764005bc0 0x7f678015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677ea00640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67800b2a50 0x7f67800cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 -- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67800b2a50 msgr2=0x7f67800cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67800b2a50 0x7f67800cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 -- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6764005bc0 msgr2=0x7f678015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6764005bc0 0x7f678015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.961+0000 7f677fe00640 1 -- 172.21.15.67:0/2661538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678015aff0 con 0x7f67800b2ed0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f677fe00640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f67800b3a50 tx=0x7f6778004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778019070 con 0x7f67800b2ed0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6778007cb0 con 0x7f67800b2ed0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778007e60 con 0x7f67800b2ed0 2024-04-17T08:53:25.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67800b1570 con 0x7f67800b2ed0 2024-04-17T08:53:25.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.962+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67800b1b30 con 0x7f67800b2ed0 2024-04-17T08:53:25.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.963+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67800b2140 con 0x7f67800b2ed0 2024-04-17T08:53:25.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.965+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f6778005d10 con 0x7f67800b2ed0 2024-04-17T08:53:25.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.966+0000 7f6777400640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 0x7f676006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:25.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.966+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(283..283 src has 1..283) ==== 5255+0+0 (secure 0 0 0) 0x7f677808baa0 con 0x7f67800b2ed0 2024-04-17T08:53:25.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.967+0000 7f677f400640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 0x7f676006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:25.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.968+0000 7f677f400640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 0x7f676006eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6770033000 tx=0x7f6770009030 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:25.970+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6778023080 con 0x7f67800b2ed0 2024-04-17T08:53:26.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.181+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f678006c850 con 0x7f67800b2ed0 2024-04-17T08:53:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.713+0000 7f6777400640 1 -- 172.21.15.67:0/2661538820 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v284) ==== 170+0+0 (secure 0 0 0) 0x7f677805a060 con 0x7f67800b2ed0 2024-04-17T08:53:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.716+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 msgr2=0x7f676006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.716+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 0x7f676006eed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6770033000 tx=0x7f6770009030 comp rx=0 tx=0).stop 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 msgr2=0x7f67800ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f67800b3a50 tx=0x7f6778004700 comp rx=0 tx=0).stop 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 shutdown_connections 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f676006ca10 0x7f676006eed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67800b2ed0 0x7f67800ce3c0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67800b2a50 0x7f67800cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 --2- 172.21.15.67:0/2661538820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6764005bc0 0x7f678015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 >> 172.21.15.67:0/2661538820 conn(0x7f6780067c30 msgr2=0x7f678006df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 shutdown_connections 2024-04-17T08:53:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.717+0000 7f6786000640 1 -- 172.21.15.67:0/2661538820 wait complete. 2024-04-17T08:53:26.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T08:53:26.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.942+0000 7f5b59600640 1 Processor -- start 2024-04-17T08:53:26.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.942+0000 7f5b59600640 1 -- start start 2024-04-17T08:53:26.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.943+0000 7f5b59600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b5415eca0 0x7f5b540d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.943+0000 7f5b59600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b540d17a0 0x7f5b540cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.943+0000 7f5b59600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b54163860 con 0x7f5b540cfa10 2024-04-17T08:53:26.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.943+0000 7f5b59600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5b540d03b0 con 0x7f5b540d17a0 2024-04-17T08:53:26.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b59600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b540d1d40 con 0x7f5b5415eca0 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b540d17a0 0x7f5b540cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b540d17a0 0x7f5b540cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46578/0 (socket says 172.21.15.67:46578) 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 -- 172.21.15.67:0/2283464127 learned_addr learned my addr 172.21.15.67:0/2283464127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b52a00640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b5415eca0 0x7f5b540d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 -- 172.21.15.67:0/2283464127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b5415eca0 msgr2=0x7f5b540d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b5415eca0 0x7f5b540d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.944+0000 7f5b53400640 1 -- 172.21.15.67:0/2283464127 >> v1:172.21.15.67:6789/0 conn(0x7f5b540cfa10 legacy=0x7f5b540d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.945+0000 7f5b53400640 1 -- 172.21.15.67:0/2283464127 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b540d2c30 con 0x7f5b540d17a0 2024-04-17T08:53:26.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.945+0000 7f5b51600640 1 -- 172.21.15.67:0/2283464127 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239140428 0 0) 0x7f5b54163860 con 0x7f5b540cfa10 2024-04-17T08:53:26.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.946+0000 7f5b53400640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540d17a0 0x7f5b540cfe70 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009e50 tx=0x7f5b4c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbccd9a01c98a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:26.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.946+0000 7f5b51600640 1 -- 172.21.15.67:0/2283464127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c02cb00 con 0x7f5b540d17a0 2024-04-17T08:53:26.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.946+0000 7f5b51600640 1 -- 172.21.15.67:0/2283464127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4c02cca0 con 0x7f5b540d17a0 2024-04-17T08:53:26.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.946+0000 7f5b51600640 1 -- 172.21.15.67:0/2283464127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c034610 con 0x7f5b540d17a0 2024-04-17T08:53:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 -- 172.21.15.67:0/2283464127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540d17a0 msgr2=0x7f5b540cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540d17a0 0x7f5b540cfe70 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009e50 tx=0x7f5b4c02bf60 comp rx=0 tx=0).stop 2024-04-17T08:53:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 -- 172.21.15.67:0/2283464127 shutdown_connections 2024-04-17T08:53:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540d17a0 0x7f5b540cfe70 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 --2- 172.21.15.67:0/2283464127 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b5415eca0 0x7f5b540d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 -- 172.21.15.67:0/2283464127 >> 172.21.15.67:0/2283464127 conn(0x7f5b54067600 msgr2=0x7f5b540679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:26.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 -- 172.21.15.67:0/2283464127 shutdown_connections 2024-04-17T08:53:26.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.947+0000 7f5b59600640 1 -- 172.21.15.67:0/2283464127 wait complete. 2024-04-17T08:53:26.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.949+0000 7f5b59600640 1 Processor -- start 2024-04-17T08:53:26.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.949+0000 7f5b59600640 1 -- start start 2024-04-17T08:53:26.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b540d17a0 0x7f5b541f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b5415eca0 0x7f5b541f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b540d34d0 con 0x7f5b540cfa10 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5b540d31f0 con 0x7f5b5415eca0 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b59600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5b5415f080 con 0x7f5b540d17a0 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b52a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b52a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46582/0 (socket says 172.21.15.67:46582) 2024-04-17T08:53:26.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.950+0000 7f5b52a00640 1 -- 172.21.15.67:0/2023380712 learned_addr learned my addr 172.21.15.67:0/2023380712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:26.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52000640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b540d17a0 0x7f5b541f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b53400640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b5415eca0 0x7f5b541f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 -- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b540d17a0 msgr2=0x7f5b541f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b540d17a0 0x7f5b541f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 -- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b5415eca0 msgr2=0x7f5b541f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b5415eca0 0x7f5b541f1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 -- 172.21.15.67:0/2023380712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b540cd410 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b52a00640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400d830 tx=0x7f5b4400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4400b6c0 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b44019070 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4400ba80 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.951+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b541f4f60 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.952+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b541f5520 con 0x7f5b540cfa10 2024-04-17T08:53:26.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.953+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b54068ca0 con 0x7f5b540cfa10 2024-04-17T08:53:26.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.954+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f5b4401e460 con 0x7f5b540cfa10 2024-04-17T08:53:26.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.955+0000 7f5b4aa00640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 0x7f5b3006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:26.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.955+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(284..284 src has 1..284) ==== 5637+0+0 (secure 0 0 0) 0x7f5b4408ce10 con 0x7f5b540cfa10 2024-04-17T08:53:26.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.956+0000 7f5b52000640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 0x7f5b3006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:26.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.957+0000 7f5b52000640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 0x7f5b3006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c007fd0 tx=0x7f5b3c009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:26.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:26.959+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f5b4405b1e0 con 0x7f5b540cfa10 2024-04-17T08:53:27.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.205+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5b540d2970 con 0x7f5b540cfa10 2024-04-17T08:53:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.741+0000 7f5b4aa00640 1 -- 172.21.15.67:0/2023380712 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v147) ==== 174+0+0 (secure 0 0 0) 0x7f5b4405aae0 con 0x7f5b540cfa10 2024-04-17T08:53:27.743 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 36 and data pool 37 2024-04-17T08:53:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.750+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 msgr2=0x7f5b3006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.750+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 0x7f5b3006eed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c007fd0 tx=0x7f5b3c009040 comp rx=0 tx=0).stop 2024-04-17T08:53:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.750+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 msgr2=0x7f5b540ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.750+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400d830 tx=0x7f5b4400dcf0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 shutdown_connections 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f5b3006ca10 0x7f5b3006eed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5b5415eca0 0x7f5b541f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5b540d17a0 0x7f5b541f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 --2- 172.21.15.67:0/2023380712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b540cfa10 0x7f5b540ce510 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.754+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 >> 172.21.15.67:0/2023380712 conn(0x7f5b54067600 msgr2=0x7f5b5415d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:27.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.755+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 shutdown_connections 2024-04-17T08:53:27.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.757+0000 7f5b59600640 1 -- 172.21.15.67:0/2023380712 wait complete. 2024-04-17T08:53:27.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.972+0000 7fd869c00640 1 Processor -- start 2024-04-17T08:53:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.972+0000 7fd869c00640 1 -- start start 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd86406b8c0 0x7fd8640cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd864160280 0x7fd86406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8640cd4a0 con 0x7fd864141ed0 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd86415ce30 con 0x7fd86406b8c0 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd8641607f0 con 0x7fd864160280 2024-04-17T08:53:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd863e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd86406b8c0 0x7fd8640cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd864141ed0 0x7fd86415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58272/0 (socket says 172.21.15.67:58272) 2024-04-17T08:53:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd869200640 1 -- 172.21.15.67:0/619137109 learned_addr learned my addr 172.21.15.67:0/619137109 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd863e00640 1 -- 172.21.15.67:0/619137109 >> v2:172.21.15.187:3300/0 conn(0x7fd864160280 msgr2=0x7fd86406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd863e00640 1 --2- 172.21.15.67:0/619137109 >> v2:172.21.15.187:3300/0 conn(0x7fd864160280 0x7fd86406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd863e00640 1 -- 172.21.15.67:0/619137109 >> v1:172.21.15.67:6789/0 conn(0x7fd864141ed0 legacy=0x7fd86415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.974+0000 7fd863e00640 1 -- 172.21.15.67:0/619137109 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd864160ad0 con 0x7fd86406b8c0 2024-04-17T08:53:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd863e00640 1 --2- 172.21.15.67:0/619137109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86406b8c0 0x7fd8640cd8b0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7fd8640aff50 tx=0x7fd85402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e130fbcf291aaba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd863400640 1 -- 172.21.15.67:0/619137109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854035020 con 0x7fd86406b8c0 2024-04-17T08:53:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd863400640 1 -- 172.21.15.67:0/619137109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd854031830 con 0x7fd86406b8c0 2024-04-17T08:53:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd863400640 1 -- 172.21.15.67:0/619137109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854031c00 con 0x7fd86406b8c0 2024-04-17T08:53:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd869c00640 1 -- 172.21.15.67:0/619137109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86406b8c0 msgr2=0x7fd8640cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.976+0000 7fd869c00640 1 --2- 172.21.15.67:0/619137109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86406b8c0 0x7fd8640cd8b0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7fd8640aff50 tx=0x7fd85402b8d0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 -- 172.21.15.67:0/619137109 shutdown_connections 2024-04-17T08:53:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 --2- 172.21.15.67:0/619137109 >> v2:172.21.15.187:3300/0 conn(0x7fd864160280 0x7fd86406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 --2- 172.21.15.67:0/619137109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86406b8c0 0x7fd8640cd8b0 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 -- 172.21.15.67:0/619137109 >> 172.21.15.67:0/619137109 conn(0x7fd864067d10 msgr2=0x7fd8640b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 -- 172.21.15.67:0/619137109 shutdown_connections 2024-04-17T08:53:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.977+0000 7fd869c00640 1 -- 172.21.15.67:0/619137109 wait complete. 2024-04-17T08:53:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.978+0000 7fd869c00640 1 Processor -- start 2024-04-17T08:53:27.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.979+0000 7fd869c00640 1 -- start start 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd86406b8c0 0x7fd86415ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd864141ed0 0x7fd8641f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd86406c1e0 con 0x7fd864160280 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd8640cddf0 con 0x7fd86406b8c0 2024-04-17T08:53:27.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd864161090 con 0x7fd864141ed0 2024-04-17T08:53:27.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:27.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46602/0 (socket says 172.21.15.67:46602) 2024-04-17T08:53:27.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 -- 172.21.15.67:0/2470404848 learned_addr learned my addr 172.21.15.67:0/2470404848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:27.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd85b400640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd864141ed0 0x7fd8641f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 -- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd864141ed0 msgr2=0x7fd8641f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd864141ed0 0x7fd8641f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 -- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd86406b8c0 msgr2=0x7fd86415ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd86406b8c0 0x7fd86415ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.980+0000 7fd869200640 1 -- 172.21.15.67:0/2470404848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8641f3390 con 0x7fd864160280 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd869200640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00cbc0 tx=0x7fd85c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c017070 con 0x7fd864160280 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85c00ad40 con 0x7fd864160280 2024-04-17T08:53:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c012600 con 0x7fd864160280 2024-04-17T08:53:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd864144500 con 0x7fd864160280 2024-04-17T08:53:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.981+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd864144ac0 con 0x7fd864160280 2024-04-17T08:53:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.982+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd864068a10 con 0x7fd864160280 2024-04-17T08:53:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.983+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fd85c01a070 con 0x7fd864160280 2024-04-17T08:53:27.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.984+0000 7fd860c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 0x7fd84006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:27.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.984+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(285..285 src has 1..285) ==== 5737+0+0 (secure 0 0 0) 0x7fd85c08b9b0 con 0x7fd864160280 2024-04-17T08:53:27.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.985+0000 7fd863e00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 0x7fd84006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:27.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.986+0000 7fd863e00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 0x7fd84006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd854036000 tx=0x7fd854030000 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:27.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:27.989+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd8641f3390 con 0x7fd864160280 2024-04-17T08:53:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.196+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8640ce480 con 0x7fd864160280 2024-04-17T08:53:28.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.198+0000 7fd860c00640 1 -- 172.21.15.67:0/2470404848 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v285) ==== 74+0+16636 (secure 0 0 0) 0x7fd85c059650 con 0x7fd864160280 2024-04-17T08:53:28.198 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:28.198 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":285,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:27.207163+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 msgr2=0x7fd84006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 0x7fd84006eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd854036000 tx=0x7fd854030000 comp rx=0 tx=0).stop 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 msgr2=0x7fd8641f3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00cbc0 tx=0x7fd85c00a430 comp rx=0 tx=0).stop 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 shutdown_connections 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd84006ca10 0x7fd84006eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd864160280 0x7fd8641f3590 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd864141ed0 0x7fd8641f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 --2- 172.21.15.67:0/2470404848 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd86406b8c0 0x7fd86415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.200+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 >> 172.21.15.67:0/2470404848 conn(0x7fd864067d10 msgr2=0x7fd8640b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.201+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 shutdown_connections 2024-04-17T08:53:28.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.201+0000 7fd869c00640 1 -- 172.21.15.67:0/2470404848 wait complete. 2024-04-17T08:53:28.218 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:28.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:53:28.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:53:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.429+0000 7fd0b5800640 1 Processor -- start 2024-04-17T08:53:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.430+0000 7fd0b5800640 1 -- start start 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.431+0000 7fd0b5800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.431+0000 7fd0b5800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0b00d12a0 0x7fd0b00d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.431+0000 7fd0b5800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b00cfe20 con 0x7fd0b00b9f00 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.431+0000 7fd0b5800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b00d1c50 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.431+0000 7fd0b5800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b00cf640 con 0x7fd0b00d12a0 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46606/0 (socket says 172.21.15.67:46606) 2024-04-17T08:53:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 -- 172.21.15.67:0/1794321721 learned_addr learned my addr 172.21.15.67:0/1794321721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0b4e00640 1 --2- 172.21.15.67:0/1794321721 >> v2:172.21.15.187:3300/0 conn(0x7fd0b00d12a0 0x7fd0b00d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 -- 172.21.15.67:0/1794321721 >> v2:172.21.15.187:3300/0 conn(0x7fd0b00d12a0 msgr2=0x7fd0b00d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 --2- 172.21.15.67:0/1794321721 >> v2:172.21.15.187:3300/0 conn(0x7fd0b00d12a0 0x7fd0b00d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 -- 172.21.15.67:0/1794321721 >> v1:172.21.15.187:6789/0 conn(0x7fd0b00b9f00 legacy=0x7fd0b01634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.432+0000 7fd0a7e00640 1 -- 172.21.15.67:0/1794321721 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b00d0100 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.433+0000 7fd0a7e00640 1 --2- 172.21.15.67:0/1794321721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00b7810 tx=0x7fd09802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edd13ce4d0c2f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.434+0000 7fd0af400640 1 -- 172.21.15.67:0/1794321721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09802e070 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.434+0000 7fd0af400640 1 -- 172.21.15.67:0/1794321721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd098037070 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.434+0000 7fd0af400640 1 -- 172.21.15.67:0/1794321721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098031a00 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.434+0000 7fd0b5800640 1 -- 172.21.15.67:0/1794321721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00cf8f0 msgr2=0x7fd0b00b7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.434+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1794321721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00b7810 tx=0x7fd09802b880 comp rx=0 tx=0).stop 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 -- 172.21.15.67:0/1794321721 shutdown_connections 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1794321721 >> v2:172.21.15.187:3300/0 conn(0x7fd0b00d12a0 0x7fd0b00d1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1794321721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b00b7f80 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 -- 172.21.15.67:0/1794321721 >> 172.21.15.67:0/1794321721 conn(0x7fd0b006cb60 msgr2=0x7fd0b006cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 -- 172.21.15.67:0/1794321721 shutdown_connections 2024-04-17T08:53:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.435+0000 7fd0b5800640 1 -- 172.21.15.67:0/1794321721 wait complete. 2024-04-17T08:53:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.437+0000 7fd0b5800640 1 Processor -- start 2024-04-17T08:53:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.437+0000 7fd0b5800640 1 -- start start 2024-04-17T08:53:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0b00b9f00 0x7fd0b00c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b01f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00ba2e0 con 0x7fd0b00d12a0 2024-04-17T08:53:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00d3420 con 0x7fd0b00cf8f0 2024-04-17T08:53:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.438+0000 7fd0b5800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b00d03e0 con 0x7fd0b00b9f00 2024-04-17T08:53:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46618/0 (socket says 172.21.15.67:46618) 2024-04-17T08:53:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 -- 172.21.15.67:0/1206237911 learned_addr learned my addr 172.21.15.67:0/1206237911 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0a7e00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b01f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0afe00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0b00b9f00 0x7fd0b00c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 -- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0b00b9f00 msgr2=0x7fd0b00c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0b00b9f00 0x7fd0b00c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 -- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 msgr2=0x7fd0b01f58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b01f58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0b4e00640 1 -- 172.21.15.67:0/1206237911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b01f5e20 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.439+0000 7fd0a7e00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b01f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0b4e00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800cc30 tx=0x7fd0a800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800ce40 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a8003990 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b00cb280 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003d40 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.440+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b00c8b60 con 0x7fd0b00d12a0 2024-04-17T08:53:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.441+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b00755d0 con 0x7fd0b00d12a0 2024-04-17T08:53:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.442+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fd0a8004000 con 0x7fd0b00d12a0 2024-04-17T08:53:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.444+0000 7fd0acc00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 0x7fd08c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.444+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(285..285 src has 1..285) ==== 5737+0+0 (secure 0 0 0) 0x7fd0a808c2c0 con 0x7fd0b00d12a0 2024-04-17T08:53:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.444+0000 7fd0afe00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 0x7fd08c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.445+0000 7fd0afe00640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 0x7fd08c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0162d80 tx=0x7fd0a0009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.448+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd0b01f5e20 con 0x7fd0b00d12a0 2024-04-17T08:53:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.657+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0b00b5d60 con 0x7fd0b00d12a0 2024-04-17T08:53:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.660+0000 7fd0acc00640 1 -- 172.21.15.67:0/1206237911 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v285) ==== 74+0+16636 (secure 0 0 0) 0x7fd0a8014030 con 0x7fd0b00d12a0 2024-04-17T08:53:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:28.660 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":285,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:27.207163+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 msgr2=0x7fd08c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 0x7fd08c06eed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0162d80 tx=0x7fd0a0009040 comp rx=0 tx=0).stop 2024-04-17T08:53:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 msgr2=0x7fd0b0161810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800cc30 tx=0x7fd0a800a5c0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 shutdown_connections 2024-04-17T08:53:28.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08c06ca10 0x7fd08c06eed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0b00d12a0 0x7fd0b0161810 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0b00cf8f0 0x7fd0b01f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 --2- 172.21.15.67:0/1206237911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0b00b9f00 0x7fd0b00c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 >> 172.21.15.67:0/1206237911 conn(0x7fd0b006cb60 msgr2=0x7fd0b015ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 shutdown_connections 2024-04-17T08:53:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.663+0000 7fd0b5800640 1 -- 172.21.15.67:0/1206237911 wait complete. 2024-04-17T08:53:28.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T08:53:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.942+0000 7fb013200640 1 Processor -- start 2024-04-17T08:53:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.942+0000 7fb013200640 1 -- start start 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb013200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb01406bfd0 0x7fb0140b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb013200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01406df40 con 0x7fb01406f800 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb013200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01406e220 con 0x7fb01415b6b0 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb013200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb01406ed00 con 0x7fb01406bfd0 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb012800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb01406f800 0x7fb0140b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58282/0 (socket says 172.21.15.67:58282) 2024-04-17T08:53:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.944+0000 7fb012800640 1 -- 172.21.15.67:0/807483523 learned_addr learned my addr 172.21.15.67:0/807483523 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb011400640 1 --2- 172.21.15.67:0/807483523 >> v2:172.21.15.187:3300/0 conn(0x7fb01406bfd0 0x7fb0140b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488227699 0 0) 0x7fb01406df40 con 0x7fb01406f800 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faff8003520 con 0x7fb01406f800 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1813509321 0 0) 0x7faff8003520 con 0x7fb01406f800 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 >> v2:172.21.15.187:3300/0 conn(0x7fb01406bfd0 msgr2=0x7fb0140b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 --2- 172.21.15.67:0/807483523 >> v2:172.21.15.187:3300/0 conn(0x7fb01406bfd0 0x7fb0140b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 >> v1:172.21.15.187:6789/0 conn(0x7fb01415b6b0 legacy=0x7fb01406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01406e500 con 0x7fb01406f800 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb00c0019f0 con 0x7fb01406f800 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 >> v1:172.21.15.67:6789/0 conn(0x7fb01406f800 legacy=0x7fb0140b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faff8003a00 0x7faff8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 0x7faff8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8005210 0x7faff800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb01406ed00 con 0x7faff8004800 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faff800c160 con 0x7faff8003a00 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.945+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faff800e570 con 0x7faff8005210 2024-04-17T08:53:28.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 0x7faff8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb012800640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8005210 0x7faff800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 -- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8005210 msgr2=0x7faff800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8005210 0x7faff800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 -- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faff8003a00 msgr2=0x7faff8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011e00640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faff8003a00 0x7faff8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faff8003a00 0x7faff8003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb011400640 1 -- 172.21.15.67:0/807483523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff8003520 con 0x7faff8004800 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb012800640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8005210 0x7faff800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.946+0000 7fb012800640 1 -- 172.21.15.67:0/807483523 reap_dead start 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.947+0000 7fb011400640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 0x7faff8004c80 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7fb000009a90 tx=0x7faff8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7214a9e58d1244b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000039070 con 0x7faff8004800 2024-04-17T08:53:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00002c680 con 0x7faff8004800 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb010a00640 1 -- 172.21.15.67:0/807483523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000009690 con 0x7faff8004800 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 -- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 msgr2=0x7faff8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 0x7faff8004c80 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7fb000009a90 tx=0x7faff8005830 comp rx=0 tx=0).stop 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 -- 172.21.15.67:0/807483523 shutdown_connections 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 --2- 172.21.15.67:0/807483523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faff8004800 0x7faff8004c80 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 -- 172.21.15.67:0/807483523 >> 172.21.15.67:0/807483523 conn(0x7fb0140b2190 msgr2=0x7fb0140b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 -- 172.21.15.67:0/807483523 shutdown_connections 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.948+0000 7fb013200640 1 -- 172.21.15.67:0/807483523 wait complete. 2024-04-17T08:53:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.950+0000 7fb013200640 1 Processor -- start 2024-04-17T08:53:28.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.950+0000 7fb013200640 1 -- start start 2024-04-17T08:53:28.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 0x7fb01415d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014157eb0 0x7fb014158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0140cfc10 con 0x7fb01414a9a0 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb01406e870 con 0x7fb014157eb0 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.951+0000 7fb013200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb0140cfd90 con 0x7faff8004800 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46634/0 (socket says 172.21.15.67:46634) 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 -- 172.21.15.67:0/64346322 learned_addr learned my addr 172.21.15.67:0/64346322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb011e00640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 0x7fb01415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb011400640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014157eb0 0x7fb014158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 -- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 msgr2=0x7fb01415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 0x7fb01415d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 -- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014157eb0 msgr2=0x7fb014158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014157eb0 0x7fb014158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb012800640 1 -- 172.21.15.67:0/64346322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01415d940 con 0x7fb01414a9a0 2024-04-17T08:53:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.952+0000 7fb011e00640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 0x7fb01415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb012800640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fb0140ccb30 tx=0x7fb00c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c00de90 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c004b60 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c007070 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb014158c90 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.953+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb014159250 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.954+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb014068a10 con 0x7fb01414a9a0 2024-04-17T08:53:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.955+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fb00c01b020 con 0x7fb01414a9a0 2024-04-17T08:53:28.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.956+0000 7fb00a000640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 0x7faff406f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:28.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.956+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fb00c08c070 con 0x7fb01414a9a0 2024-04-17T08:53:28.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.957+0000 7fb011e00640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 0x7faff406f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:28.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.959+0000 7fb011e00640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 0x7faff406f0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0140690c0 tx=0x7fb004009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:28.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:28.961+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb00c05a1c0 con 0x7fb01414a9a0 2024-04-17T08:53:29.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.206+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb01406ee80 con 0x7fb01414a9a0 2024-04-17T08:53:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.760+0000 7fb00a000640 1 -- 172.21.15.67:0/64346322 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v150) ==== 118+0+0 (secure 0 0 0) 0x7fb00c059ac0 con 0x7fb01414a9a0 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 msgr2=0x7faff406f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 0x7faff406f0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0140690c0 tx=0x7fb004009040 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 msgr2=0x7fb01414ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7fb0140ccb30 tx=0x7fb00c00b4a0 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 shutdown_connections 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7faff406cbe0 0x7faff406f0a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01414a9a0 0x7fb01414ae20 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014157eb0 0x7fb014158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 --2- 172.21.15.67:0/64346322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faff8004800 0x7fb01415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 >> 172.21.15.67:0/64346322 conn(0x7fb0140b2190 msgr2=0x7fb0140ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 shutdown_connections 2024-04-17T08:53:29.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:29.764+0000 7fb013200640 1 -- 172.21.15.67:0/64346322 wait complete. 2024-04-17T08:53:29.789 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T08:53:29.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T08:53:30.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.014+0000 7f6227e00640 1 Processor -- start 2024-04-17T08:53:30.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.014+0000 7f6227e00640 1 -- start start 2024-04-17T08:53:30.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6227e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6228145f80 0x7f62280cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6227e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f622806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6227e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6228146360 con 0x7f62280cd750 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6227e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f62280727a0 con 0x7f6228145f80 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6227e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6228071e70 con 0x7f622806c270 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6226a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6228145f80 0x7f62280cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6226a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6228145f80 0x7f62280cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46648/0 (socket says 172.21.15.67:46648) 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.016+0000 7f6226a00640 1 -- 172.21.15.67:0/3607797805 learned_addr learned my addr 172.21.15.67:0/3607797805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.017+0000 7f6226a00640 1 -- 172.21.15.67:0/3607797805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 msgr2=0x7f622806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.017+0000 7f6226a00640 1 --2- 172.21.15.67:0/3607797805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f622806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.017+0000 7f6226a00640 1 -- 172.21.15.67:0/3607797805 >> v1:172.21.15.187:6789/0 conn(0x7f62280cd750 legacy=0x7f622806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:30.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.017+0000 7f6226a00640 1 -- 172.21.15.67:0/3607797805 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6228072150 con 0x7f6228145f80 2024-04-17T08:53:30.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.018+0000 7f6226a00640 1 --2- 172.21.15.67:0/3607797805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280cd210 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f6218009e50 tx=0x7f621802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49522da03f46770 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:30.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.018+0000 7f6226000640 1 -- 172.21.15.67:0/3607797805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621802c9e0 con 0x7f6228145f80 2024-04-17T08:53:30.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.018+0000 7f6226000640 1 -- 172.21.15.67:0/3607797805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621802cb80 con 0x7f6228145f80 2024-04-17T08:53:30.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.018+0000 7f6226000640 1 -- 172.21.15.67:0/3607797805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62180343f0 con 0x7f6228145f80 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 -- 172.21.15.67:0/3607797805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 msgr2=0x7f62280cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 --2- 172.21.15.67:0/3607797805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280cd210 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f6218009e50 tx=0x7f621802be40 comp rx=0 tx=0).stop 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 -- 172.21.15.67:0/3607797805 shutdown_connections 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 --2- 172.21.15.67:0/3607797805 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f622806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 --2- 172.21.15.67:0/3607797805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280cd210 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 -- 172.21.15.67:0/3607797805 >> 172.21.15.67:0/3607797805 conn(0x7f62280b2190 msgr2=0x7f62280b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:30.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 -- 172.21.15.67:0/3607797805 shutdown_connections 2024-04-17T08:53:30.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.019+0000 7f6227e00640 1 -- 172.21.15.67:0/3607797805 wait complete. 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.021+0000 7f6227e00640 1 Processor -- start 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.022+0000 7f6227e00640 1 -- start start 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f6228134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 0x7f62280b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6228072db0 con 0x7f6228145f80 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6228072c30 con 0x7f62280cd750 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f622806e700 con 0x7f622806c270 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46654/0 (socket says 172.21.15.67:46654) 2024-04-17T08:53:30.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6227400640 1 -- 172.21.15.67:0/1705735092 learned_addr learned my addr 172.21.15.67:0/1705735092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:30.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f6226a00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f6228134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:30.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.023+0000 7f621e000640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 0x7f62280b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:30.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 -- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 msgr2=0x7f6228134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:30.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f6228134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 -- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 msgr2=0x7f62280b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 0x7f62280b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 -- 172.21.15.67:0/1705735092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6228140150 con 0x7f6228145f80 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f621e000640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 0x7f62280b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227400640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f6220009fd0 tx=0x7f622000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220011830 con 0x7f6228145f80 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6220016070 con 0x7f6228145f80 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220011bf0 con 0x7f6228145f80 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622815f2c0 con 0x7f6228145f80 2024-04-17T08:53:30.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.024+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62280a99f0 con 0x7f6228145f80 2024-04-17T08:53:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.026+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62280a9520 con 0x7f6228145f80 2024-04-17T08:53:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.027+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f622001b460 con 0x7f6228145f80 2024-04-17T08:53:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.028+0000 7f621f400640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 0x7f620406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.028+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f622008ba00 con 0x7f6228145f80 2024-04-17T08:53:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.029+0000 7f6226a00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 0x7f620406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:30.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.030+0000 7f6226a00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 0x7f620406ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6218035000 tx=0x7f6218030000 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:30.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.033+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6220059cf0 con 0x7f6228145f80 2024-04-17T08:53:30.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.272+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f62280b5430 con 0x7f6228145f80 2024-04-17T08:53:30.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.774+0000 7f621f400640 1 -- 172.21.15.67:0/1705735092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v151) ==== 122+0+0 (secure 0 0 0) 0x7f62200595f0 con 0x7f6228145f80 2024-04-17T08:53:30.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.777+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 msgr2=0x7f620406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:30.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.777+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 0x7f620406ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6218035000 tx=0x7f6218030000 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.777+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 msgr2=0x7f62280b1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.777+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f6220009fd0 tx=0x7f622000b8e0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 shutdown_connections 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f620406ca40 0x7f620406ef00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6228145f80 0x7f62280b1690 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62280cd750 0x7f62280b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 --2- 172.21.15.67:0/1705735092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f622806c270 0x7f6228134bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 >> 172.21.15.67:0/1705735092 conn(0x7f62280b2190 msgr2=0x7f62280b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 shutdown_connections 2024-04-17T08:53:30.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:30.778+0000 7f6227e00640 1 -- 172.21.15.67:0/1705735092 wait complete. 2024-04-17T08:53:30.802 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T08:53:30.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:31.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.026+0000 7fbde5800640 1 Processor -- start 2024-04-17T08:53:31.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.026+0000 7fbde5800640 1 -- start start 2024-04-17T08:53:31.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbde5800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbde0160280 0x7fbde006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbde5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde00cd4a0 con 0x7fbde006b8c0 2024-04-17T08:53:31.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbde5800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde01607f0 con 0x7fbde0141ed0 2024-04-17T08:53:31.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbde5800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbde015ce30 con 0x7fbde0160280 2024-04-17T08:53:31.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbddfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbde006b8c0 0x7fbde00cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58296/0 (socket says 172.21.15.67:58296) 2024-04-17T08:53:31.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.028+0000 7fbddfe00640 1 -- 172.21.15.67:0/2064897973 learned_addr learned my addr 172.21.15.67:0/2064897973 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddf400640 1 --2- 172.21.15.67:0/2064897973 >> v2:172.21.15.187:3300/0 conn(0x7fbde0160280 0x7fbde006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041488687 0 0) 0x7fbde00cd4a0 con 0x7fbde006b8c0 2024-04-17T08:53:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc4003520 con 0x7fbde006b8c0 2024-04-17T08:53:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2424576159 0 0) 0x7fbdc4003520 con 0x7fbde006b8c0 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 >> v2:172.21.15.187:3300/0 conn(0x7fbde0160280 msgr2=0x7fbde006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 --2- 172.21.15.67:0/2064897973 >> v2:172.21.15.187:3300/0 conn(0x7fbde0160280 0x7fbde006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 >> v1:172.21.15.187:6789/0 conn(0x7fbde0141ed0 legacy=0x7fbde015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde0160ad0 con 0x7fbde006b8c0 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.029+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fbdd0001670 con 0x7fbde006b8c0 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 >> v1:172.21.15.67:6789/0 conn(0x7fbde006b8c0 legacy=0x7fbde00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc4003a20 0x7fbdc4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdc4004800 0x7fbdc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 0x7fbdc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbde015ce30 con 0x7fbdc40051c0 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc400c0f0 con 0x7fbdc4003a20 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc400c2f0 con 0x7fbdc4004800 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbde4e00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 0x7fbdc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddf400640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdc4004800 0x7fbdc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbddfe00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc4003a20 0x7fbdc4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbde4e00640 1 -- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdc4004800 msgr2=0x7fbdc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.030+0000 7fbde4e00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbdc4004800 0x7fbdc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.031+0000 7fbde4e00640 1 -- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc4003a20 msgr2=0x7fbdc4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.031+0000 7fbde4e00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc4003a20 0x7fbdc4003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.031+0000 7fbde4e00640 1 -- 172.21.15.67:0/2064897973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc4003520 con 0x7fbdc40051c0 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.031+0000 7fbde4e00640 1 -- 172.21.15.67:0/2064897973 reap_dead start 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbde4e00640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 0x7fbdc400b9c0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8001e50 tx=0x7fbdc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa17efff6af78cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8001a70 con 0x7fbdc40051c0 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdd8003bb0 con 0x7fbdc40051c0 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbddea00640 1 -- 172.21.15.67:0/2064897973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80050d0 con 0x7fbdc40051c0 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbde5800640 1 -- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 msgr2=0x7fbdc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.032+0000 7fbde5800640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 0x7fbdc400b9c0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8001e50 tx=0x7fbdc4005830 comp rx=0 tx=0).stop 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.033+0000 7fbde5800640 1 -- 172.21.15.67:0/2064897973 shutdown_connections 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.033+0000 7fbde5800640 1 --2- 172.21.15.67:0/2064897973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc40051c0 0x7fbdc400b9c0 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.033+0000 7fbde5800640 1 -- 172.21.15.67:0/2064897973 >> 172.21.15.67:0/2064897973 conn(0x7fbde0067d10 msgr2=0x7fbde00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.033+0000 7fbde5800640 1 -- 172.21.15.67:0/2064897973 shutdown_connections 2024-04-17T08:53:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.033+0000 7fbde5800640 1 -- 172.21.15.67:0/2064897973 wait complete. 2024-04-17T08:53:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.034+0000 7fbde5800640 1 Processor -- start 2024-04-17T08:53:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.034+0000 7fbde5800640 1 -- start start 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc40051c0 0x7fbde013d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 0x7fbde015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbde013f9b0 con 0x7fbde01458a0 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbde0161090 con 0x7fbdc40051c0 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.035+0000 7fbde5800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbde013fb30 con 0x7fbde0145d20 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46670/0 (socket says 172.21.15.67:46670) 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 -- 172.21.15.67:0/1834930694 learned_addr learned my addr 172.21.15.67:0/1834930694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbde4e00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 0x7fbde015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddfe00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc40051c0 0x7fbde013d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 -- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 msgr2=0x7fbde015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 0x7fbde015bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 -- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc40051c0 msgr2=0x7fbde013d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc40051c0 0x7fbde013d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 -- 172.21.15.67:0/1834930694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde01f7bf0 con 0x7fbde01458a0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbde4e00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 0x7fbde015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.036+0000 7fbddf400640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8011570 tx=0x7fbdc8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.037+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8015930 con 0x7fbde01458a0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.037+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc8015ad0 con 0x7fbde01458a0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.037+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800f460 con 0x7fbde01458a0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.037+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde01f7ed0 con 0x7fbde01458a0 2024-04-17T08:53:31.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.037+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde01f8490 con 0x7fbde01458a0 2024-04-17T08:53:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.038+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde0068a10 con 0x7fbde01458a0 2024-04-17T08:53:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.039+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fbdc800f600 con 0x7fbde01458a0 2024-04-17T08:53:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.040+0000 7fbdd7e00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 0x7fbdbc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.040+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fbdc808c620 con 0x7fbde01458a0 2024-04-17T08:53:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.041+0000 7fbddfe00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 0x7fbdbc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.042+0000 7fbddfe00640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 0x7fbdbc06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00017e0 tx=0x7fbdd0009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.045+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fbdc8019030 con 0x7fbde01458a0 2024-04-17T08:53:31.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.281+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbde01416a0 con 0x7fbde01458a0 2024-04-17T08:53:31.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.283+0000 7fbdd7e00640 1 -- 172.21.15.67:0/1834930694 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7fbdc805a910 con 0x7fbde01458a0 2024-04-17T08:53:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:31.282 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:53:31.283 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:53:31.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.285+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 msgr2=0x7fbdbc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.285+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 0x7fbdbc06ef00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00017e0 tx=0x7fbdd0009040 comp rx=0 tx=0).stop 2024-04-17T08:53:31.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.285+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 msgr2=0x7fbde015b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.285+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8011570 tx=0x7fbdc8011a30 comp rx=0 tx=0).stop 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 shutdown_connections 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fbdbc06ca40 0x7fbdbc06ef00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbde0145d20 0x7fbde015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde01458a0 0x7fbde015b680 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 --2- 172.21.15.67:0/1834930694 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbdc40051c0 0x7fbde013d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 >> 172.21.15.67:0/1834930694 conn(0x7fbde0067d10 msgr2=0x7fbde00cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 shutdown_connections 2024-04-17T08:53:31.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.286+0000 7fbde5800640 1 -- 172.21.15.67:0/1834930694 wait complete. 2024-04-17T08:53:31.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:53:31.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.508+0000 7fd22dc00640 1 Processor -- start 2024-04-17T08:53:31.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.508+0000 7fd22dc00640 1 -- start start 2024-04-17T08:53:31.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd22dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd228072420 0x7fd22806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd22dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22815f3c0 con 0x7fd22815b530 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd22dc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22815f660 con 0x7fd2280c68c0 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd22dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd22815f940 con 0x7fd228072420 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd226a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd22815b530 0x7fd228071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58310/0 (socket says 172.21.15.67:58310) 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd226a00640 1 -- 172.21.15.67:0/3366032632 learned_addr learned my addr 172.21.15.67:0/3366032632 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:31.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.510+0000 7fd21fe00640 1 --2- 172.21.15.67:0/3366032632 >> v2:172.21.15.67:3300/0 conn(0x7fd228072420 0x7fd22806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.511+0000 7fd21fe00640 1 -- 172.21.15.67:0/3366032632 >> v1:172.21.15.187:6789/0 conn(0x7fd2280c68c0 legacy=0x7fd22806e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:31.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.511+0000 7fd21fe00640 1 -- 172.21.15.67:0/3366032632 >> v1:172.21.15.67:6789/0 conn(0x7fd22815b530 legacy=0x7fd228071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.511+0000 7fd21fe00640 1 -- 172.21.15.67:0/3366032632 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22815fc20 con 0x7fd228072420 2024-04-17T08:53:31.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.511+0000 7fd226000640 1 -- 172.21.15.67:0/3366032632 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444831179 0 0) 0x7fd22815f3c0 con 0x7fd22815b530 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.512+0000 7fd21fe00640 1 --2- 172.21.15.67:0/3366032632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22806dd50 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7fd2100097a0 tx=0x7fd21002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1173094ae902643b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.512+0000 7fd226000640 1 -- 172.21.15.67:0/3366032632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210035020 con 0x7fd228072420 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.512+0000 7fd226000640 1 -- 172.21.15.67:0/3366032632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd210031830 con 0x7fd228072420 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd226000640 1 -- 172.21.15.67:0/3366032632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210031c00 con 0x7fd228072420 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 -- 172.21.15.67:0/3366032632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 msgr2=0x7fd22806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 --2- 172.21.15.67:0/3366032632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22806dd50 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7fd2100097a0 tx=0x7fd21002b8d0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 -- 172.21.15.67:0/3366032632 shutdown_connections 2024-04-17T08:53:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 --2- 172.21.15.67:0/3366032632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22806dd50 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 -- 172.21.15.67:0/3366032632 >> 172.21.15.67:0/3366032632 conn(0x7fd22806bf50 msgr2=0x7fd2280b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 -- 172.21.15.67:0/3366032632 shutdown_connections 2024-04-17T08:53:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.513+0000 7fd22dc00640 1 -- 172.21.15.67:0/3366032632 wait complete. 2024-04-17T08:53:31.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.515+0000 7fd22dc00640 1 Processor -- start 2024-04-17T08:53:31.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.515+0000 7fd22dc00640 1 -- start start 2024-04-17T08:53:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2280c68c0 0x7fd2280c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd22815b530 0x7fd2280cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd228160140 con 0x7fd228072420 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd22815b910 con 0x7fd22815b530 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd22dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd2280b2040 con 0x7fd2280c68c0 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd226a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd226a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46690/0 (socket says 172.21.15.67:46690) 2024-04-17T08:53:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.516+0000 7fd226a00640 1 -- 172.21.15.67:0/414999952 learned_addr learned my addr 172.21.15.67:0/414999952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd21fe00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2280c68c0 0x7fd2280c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 -- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2280c68c0 msgr2=0x7fd2280c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2280c68c0 0x7fd2280c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 -- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd22815b530 msgr2=0x7fd2280cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd22815b530 0x7fd2280cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 -- 172.21.15.67:0/414999952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2280c8640 con 0x7fd228072420 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd226a00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c430 tx=0x7fd21800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2180074e0 con 0x7fd228072420 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2180047e0 con 0x7fd228072420 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2280c8920 con 0x7fd228072420 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.517+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218004bd0 con 0x7fd228072420 2024-04-17T08:53:31.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.518+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2280c8ee0 con 0x7fd228072420 2024-04-17T08:53:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.519+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd228075990 con 0x7fd228072420 2024-04-17T08:53:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.520+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fd218011070 con 0x7fd228072420 2024-04-17T08:53:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.520+0000 7fd21ea00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 0x7fd20406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.521+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fd21808b900 con 0x7fd228072420 2024-04-17T08:53:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.521+0000 7fd21fe00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 0x7fd20406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.522+0000 7fd21fe00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 0x7fd20406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd210036000 tx=0x7fd210008030 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.525+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd228075990 con 0x7fd228072420 2024-04-17T08:53:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.729+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd228068b90 con 0x7fd228072420 2024-04-17T08:53:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.731+0000 7fd21ea00640 1 -- 172.21.15.67:0/414999952 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v286) ==== 74+0+16623 (secure 0 0 0) 0x7fd218059bf0 con 0x7fd228072420 2024-04-17T08:53:31.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:31.731 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":286,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:53:28.718196+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 msgr2=0x7fd20406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 0x7fd20406ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd210036000 tx=0x7fd210008030 comp rx=0 tx=0).stop 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 msgr2=0x7fd22814b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c430 tx=0x7fd21800c8f0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 shutdown_connections 2024-04-17T08:53:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd20406ca40 0x7fd20406ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd22815b530 0x7fd2280cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd2280c68c0 0x7fd2280c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 --2- 172.21.15.67:0/414999952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd228072420 0x7fd22814b170 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 >> 172.21.15.67:0/414999952 conn(0x7fd22806bf50 msgr2=0x7fd228159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.733+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 shutdown_connections 2024-04-17T08:53:31.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.734+0000 7fd22dc00640 1 -- 172.21.15.67:0/414999952 wait complete. 2024-04-17T08:53:31.752 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T08:53:31.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:53:31.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.955+0000 7fdc34e00640 1 Processor -- start 2024-04-17T08:53:31.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.955+0000 7fdc34e00640 1 -- start start 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc34e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc3015ec60 0x7fdc300b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc34e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 0x7fdc300b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc34e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc300d2d30 con 0x7fdc300b6b30 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc34e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdc300d49c0 con 0x7fdc3015ec60 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc34e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc300d3660 con 0x7fdc300cfae0 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc2f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc3015ec60 0x7fdc300b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc2f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc3015ec60 0x7fdc300b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46696/0 (socket says 172.21.15.67:46696) 2024-04-17T08:53:31.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.957+0000 7fdc2f400640 1 -- 172.21.15.67:0/177922845 learned_addr learned my addr 172.21.15.67:0/177922845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2f400640 1 -- 172.21.15.67:0/177922845 >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 msgr2=0x7fdc300b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2fe00640 1 --2- 172.21.15.67:0/177922845 >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 0x7fdc300b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2f400640 1 --2- 172.21.15.67:0/177922845 >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 0x7fdc300b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2f400640 1 -- 172.21.15.67:0/177922845 >> v1:172.21.15.67:6789/0 conn(0x7fdc300b6b30 legacy=0x7fdc300cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2f400640 1 -- 172.21.15.67:0/177922845 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc300d3010 con 0x7fdc3015ec60 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2fe00640 1 --2- 172.21.15.67:0/177922845 >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 0x7fdc300b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:31.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.958+0000 7fdc2e000640 1 -- 172.21.15.67:0/177922845 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800301253 0 0) 0x7fdc300d2d30 con 0x7fdc300b6b30 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.959+0000 7fdc2f400640 1 --2- 172.21.15.67:0/177922845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3015ec60 0x7fdc300b65f0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cf170 tx=0x7fdc2002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8babfca7622dab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.959+0000 7fdc2e000640 1 -- 172.21.15.67:0/177922845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20034070 con 0x7fdc3015ec60 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.959+0000 7fdc2e000640 1 -- 172.21.15.67:0/177922845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc20030860 con 0x7fdc3015ec60 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.959+0000 7fdc2e000640 1 -- 172.21.15.67:0/177922845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20030c30 con 0x7fdc3015ec60 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 -- 172.21.15.67:0/177922845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3015ec60 msgr2=0x7fdc300b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 --2- 172.21.15.67:0/177922845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3015ec60 0x7fdc300b65f0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7fdc300cf170 tx=0x7fdc2002b9a0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 -- 172.21.15.67:0/177922845 shutdown_connections 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 --2- 172.21.15.67:0/177922845 >> v2:172.21.15.187:3300/0 conn(0x7fdc300cfae0 0x7fdc300b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 --2- 172.21.15.67:0/177922845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3015ec60 0x7fdc300b65f0 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 -- 172.21.15.67:0/177922845 >> 172.21.15.67:0/177922845 conn(0x7fdc300d15f0 msgr2=0x7fdc300d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 -- 172.21.15.67:0/177922845 shutdown_connections 2024-04-17T08:53:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.960+0000 7fdc34e00640 1 -- 172.21.15.67:0/177922845 wait complete. 2024-04-17T08:53:31.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.962+0000 7fdc34e00640 1 Processor -- start 2024-04-17T08:53:31.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.962+0000 7fdc34e00640 1 -- start start 2024-04-17T08:53:31.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc300cfae0 0x7fdc300cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc3015ec60 0x7fdc300cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3015f040 con 0x7fdc300b6b30 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc300b6f70 con 0x7fdc3015ec60 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc34e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc300d00c0 con 0x7fdc300cfae0 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46712/0 (socket says 172.21.15.67:46712) 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 -- 172.21.15.67:0/2241129938 learned_addr learned my addr 172.21.15.67:0/2241129938 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 -- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc300cfae0 msgr2=0x7fdc300cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc300cfae0 0x7fdc300cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 -- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc3015ec60 msgr2=0x7fdc300cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc3015ec60 0x7fdc300cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:31.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.963+0000 7fdc2f400640 1 -- 172.21.15.67:0/2241129938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc300b05f0 con 0x7fdc300b6b30 2024-04-17T08:53:31.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc2f400640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20034040 con 0x7fdc300b6b30 2024-04-17T08:53:31.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc200433f0 con 0x7fdc300b6b30 2024-04-17T08:53:31.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200437e0 con 0x7fdc300b6b30 2024-04-17T08:53:31.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc300ce360 con 0x7fdc300b6b30 2024-04-17T08:53:31.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.964+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc300b0b70 con 0x7fdc300b6b30 2024-04-17T08:53:31.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.965+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc30068a10 con 0x7fdc300b6b30 2024-04-17T08:53:31.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.966+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fdc20037070 con 0x7fdc300b6b30 2024-04-17T08:53:31.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.967+0000 7fdc27400640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 0x7fdc1006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:31.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.967+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fdc200ae1d0 con 0x7fdc300b6b30 2024-04-17T08:53:31.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.968+0000 7fdc2ea00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 0x7fdc1006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:31.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.969+0000 7fdc2ea00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 0x7fdc1006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc300756e0 tx=0x7fdc1c007910 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:31.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:31.971+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdc2003e070 con 0x7fdc300b6b30 2024-04-17T08:53:32.119 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-17T08:53:32.120+0000 7ff87ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T08:53:32.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.174+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 --> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc300ac3c0 con 0x7fdc1006ca10 2024-04-17T08:53:32.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.197+0000 7fdc27400640 1 -- 172.21.15.67:0/2241129938 <== mgr.9871 v2:172.21.15.187:6832/46399553 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252506 (secure 0 0 0) 0x7fdc300ac3c0 con 0x7fdc1006ca10 2024-04-17T08:53:32.201 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 msgr2=0x7fdc1006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 0x7fdc1006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc300756e0 tx=0x7fdc1c007910 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 msgr2=0x7fdc300cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc200046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 shutdown_connections 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdc1006ca10 0x7fdc1006eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc3015ec60 0x7fdc300cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc300cfae0 0x7fdc300cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 --2- 172.21.15.67:0/2241129938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc300b6b30 0x7fdc300cafa0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 >> 172.21.15.67:0/2241129938 conn(0x7fdc300d15f0 msgr2=0x7fdc3015e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 shutdown_connections 2024-04-17T08:53:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.203+0000 7fdc34e00640 1 -- 172.21.15.67:0/2241129938 wait complete. 2024-04-17T08:53:32.222 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-17T08:53:30.546312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":734,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463436,"kb_used_data":29404,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754511284,"statfs":{"total":773094113280,"available":772619554816,"internally_reserved":0,"allocated":30109696,"data_stored":15426708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.848282"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732426+0000","last_change":"2024-04-17T08:53:27.760015+0000","last_active":"2024-04-17T08:53:28.732426+0000","last_peered":"2024-04-17T08:53:28.732426+0000","last_clean":"2024-04-17T08:53:28.732426+0000","last_became_active":"2024-04-17T08:53:27.755424+0000","last_became_peered":"2024-04-17T08:53:27.755424+0000","last_unstale":"2024-04-17T08:53:28.732426+0000","last_undegraded":"2024-04-17T08:53:28.732426+0000","last_fullsized":"2024-04-17T08:53:28.732426+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:54.090607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.726945+0000","last_change":"2024-04-17T08:53:27.761717+0000","last_active":"2024-04-17T08:53:28.726945+0000","last_peered":"2024-04-17T08:53:28.726945+0000","last_clean":"2024-04-17T08:53:28.726945+0000","last_became_active":"2024-04-17T08:53:27.761172+0000","last_became_peered":"2024-04-17T08:53:27.761172+0000","last_unstale":"2024-04-17T08:53:28.726945+0000","last_undegraded":"2024-04-17T08:53:28.726945+0000","last_fullsized":"2024-04-17T08:53:28.726945+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:03.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161780+0000","last_change":"2024-04-17T08:53:27.765941+0000","last_active":"2024-04-17T08:53:29.161780+0000","last_peered":"2024-04-17T08:53:29.161780+0000","last_clean":"2024-04-17T08:53:29.161780+0000","last_became_active":"2024-04-17T08:53:27.763199+0000","last_became_peered":"2024-04-17T08:53:27.763199+0000","last_unstale":"2024-04-17T08:53:29.161780+0000","last_undegraded":"2024-04-17T08:53:29.161780+0000","last_fullsized":"2024-04-17T08:53:29.161780+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:27.337553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161320+0000","last_change":"2024-04-17T08:53:27.766908+0000","last_active":"2024-04-17T08:53:29.161320+0000","last_peered":"2024-04-17T08:53:29.161320+0000","last_clean":"2024-04-17T08:53:29.161320+0000","last_became_active":"2024-04-17T08:53:27.765440+0000","last_became_peered":"2024-04-17T08:53:27.765440+0000","last_unstale":"2024-04-17T08:53:29.161320+0000","last_undegraded":"2024-04-17T08:53:29.161320+0000","last_fullsized":"2024-04-17T08:53:29.161320+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:42.329109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160797+0000","last_change":"2024-04-17T08:53:27.755767+0000","last_active":"2024-04-17T08:53:29.160797+0000","last_peered":"2024-04-17T08:53:29.160797+0000","last_clean":"2024-04-17T08:53:29.160797+0000","last_became_active":"2024-04-17T08:53:27.754616+0000","last_became_peered":"2024-04-17T08:53:27.754616+0000","last_unstale":"2024-04-17T08:53:29.160797+0000","last_undegraded":"2024-04-17T08:53:29.160797+0000","last_fullsized":"2024-04-17T08:53:29.160797+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:52.661584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050561+0000","last_change":"2024-04-17T08:53:27.757627+0000","last_active":"2024-04-17T08:53:29.050561+0000","last_peered":"2024-04-17T08:53:29.050561+0000","last_clean":"2024-04-17T08:53:29.050561+0000","last_became_active":"2024-04-17T08:53:27.756356+0000","last_became_peered":"2024-04-17T08:53:27.756356+0000","last_unstale":"2024-04-17T08:53:29.050561+0000","last_undegraded":"2024-04-17T08:53:29.050561+0000","last_fullsized":"2024-04-17T08:53:29.050561+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:04.684548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.730725+0000","last_change":"2024-04-17T08:53:27.755416+0000","last_active":"2024-04-17T08:53:28.730725+0000","last_peered":"2024-04-17T08:53:28.730725+0000","last_clean":"2024-04-17T08:53:28.730725+0000","last_became_active":"2024-04-17T08:53:27.755106+0000","last_became_peered":"2024-04-17T08:53:27.755106+0000","last_unstale":"2024-04-17T08:53:28.730725+0000","last_undegraded":"2024-04-17T08:53:28.730725+0000","last_fullsized":"2024-04-17T08:53:28.730725+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:04:44.298092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763747+0000","last_change":"2024-04-17T08:53:27.768291+0000","last_active":"2024-04-17T08:53:28.763747+0000","last_peered":"2024-04-17T08:53:28.763747+0000","last_clean":"2024-04-17T08:53:28.763747+0000","last_became_active":"2024-04-17T08:53:27.767995+0000","last_became_peered":"2024-04-17T08:53:27.767995+0000","last_unstale":"2024-04-17T08:53:28.763747+0000","last_undegraded":"2024-04-17T08:53:28.763747+0000","last_fullsized":"2024-04-17T08:53:28.763747+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:40.621216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.727789+0000","last_change":"2024-04-17T08:53:27.764198+0000","last_active":"2024-04-17T08:53:28.727789+0000","last_peered":"2024-04-17T08:53:28.727789+0000","last_clean":"2024-04-17T08:53:28.727789+0000","last_became_active":"2024-04-17T08:53:27.762546+0000","last_became_peered":"2024-04-17T08:53:27.762546+0000","last_unstale":"2024-04-17T08:53:28.727789+0000","last_undegraded":"2024-04-17T08:53:28.727789+0000","last_fullsized":"2024-04-17T08:53:28.727789+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:33:05.811195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732750+0000","last_change":"2024-04-17T08:53:27.758757+0000","last_active":"2024-04-17T08:53:28.732750+0000","last_peered":"2024-04-17T08:53:28.732750+0000","last_clean":"2024-04-17T08:53:28.732750+0000","last_became_active":"2024-04-17T08:53:27.757344+0000","last_became_peered":"2024-04-17T08:53:27.757344+0000","last_unstale":"2024-04-17T08:53:28.732750+0000","last_undegraded":"2024-04-17T08:53:28.732750+0000","last_fullsized":"2024-04-17T08:53:28.732750+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:19.631870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764198+0000","last_change":"2024-04-17T08:53:27.765421+0000","last_active":"2024-04-17T08:53:28.764198+0000","last_peered":"2024-04-17T08:53:28.764198+0000","last_clean":"2024-04-17T08:53:28.764198+0000","last_became_active":"2024-04-17T08:53:27.765135+0000","last_became_peered":"2024-04-17T08:53:27.765135+0000","last_unstale":"2024-04-17T08:53:28.764198+0000","last_undegraded":"2024-04-17T08:53:28.764198+0000","last_fullsized":"2024-04-17T08:53:28.764198+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:57.079390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767909+0000","last_change":"2024-04-17T08:53:27.766495+0000","last_active":"2024-04-17T08:53:28.767909+0000","last_peered":"2024-04-17T08:53:28.767909+0000","last_clean":"2024-04-17T08:53:28.767909+0000","last_became_active":"2024-04-17T08:53:27.766200+0000","last_became_peered":"2024-04-17T08:53:27.766200+0000","last_unstale":"2024-04-17T08:53:28.767909+0000","last_undegraded":"2024-04-17T08:53:28.767909+0000","last_fullsized":"2024-04-17T08:53:28.767909+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:06:19.372923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962633+0000","last_change":"2024-04-17T08:53:27.752287+0000","last_active":"2024-04-17T08:53:28.962633+0000","last_peered":"2024-04-17T08:53:28.962633+0000","last_clean":"2024-04-17T08:53:28.962633+0000","last_became_active":"2024-04-17T08:53:27.751988+0000","last_became_peered":"2024-04-17T08:53:27.751988+0000","last_unstale":"2024-04-17T08:53:28.962633+0000","last_undegraded":"2024-04-17T08:53:28.962633+0000","last_fullsized":"2024-04-17T08:53:28.962633+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:35.119011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733445+0000","last_change":"2024-04-17T08:53:27.754990+0000","last_active":"2024-04-17T08:53:28.733445+0000","last_peered":"2024-04-17T08:53:28.733445+0000","last_clean":"2024-04-17T08:53:28.733445+0000","last_became_active":"2024-04-17T08:53:27.754636+0000","last_became_peered":"2024-04-17T08:53:27.754636+0000","last_unstale":"2024-04-17T08:53:28.733445+0000","last_undegraded":"2024-04-17T08:53:28.733445+0000","last_fullsized":"2024-04-17T08:53:28.733445+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:00:09.165919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161307+0000","last_change":"2024-04-17T08:53:27.761755+0000","last_active":"2024-04-17T08:53:29.161307+0000","last_peered":"2024-04-17T08:53:29.161307+0000","last_clean":"2024-04-17T08:53:29.161307+0000","last_became_active":"2024-04-17T08:53:27.757829+0000","last_became_peered":"2024-04-17T08:53:27.757829+0000","last_unstale":"2024-04-17T08:53:29.161307+0000","last_undegraded":"2024-04-17T08:53:29.161307+0000","last_fullsized":"2024-04-17T08:53:29.161307+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:19.407703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767433+0000","last_change":"2024-04-17T08:53:27.752976+0000","last_active":"2024-04-17T08:53:28.767433+0000","last_peered":"2024-04-17T08:53:28.767433+0000","last_clean":"2024-04-17T08:53:28.767433+0000","last_became_active":"2024-04-17T08:53:27.752670+0000","last_became_peered":"2024-04-17T08:53:27.752670+0000","last_unstale":"2024-04-17T08:53:28.767433+0000","last_undegraded":"2024-04-17T08:53:28.767433+0000","last_fullsized":"2024-04-17T08:53:28.767433+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:15:00.163187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963707+0000","last_change":"2024-04-17T08:53:27.767409+0000","last_active":"2024-04-17T08:53:28.963707+0000","last_peered":"2024-04-17T08:53:28.963707+0000","last_clean":"2024-04-17T08:53:28.963707+0000","last_became_active":"2024-04-17T08:53:27.767094+0000","last_became_peered":"2024-04-17T08:53:27.767094+0000","last_unstale":"2024-04-17T08:53:28.963707+0000","last_undegraded":"2024-04-17T08:53:28.963707+0000","last_fullsized":"2024-04-17T08:53:28.963707+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:20.212152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963314+0000","last_change":"2024-04-17T08:53:27.755357+0000","last_active":"2024-04-17T08:53:28.963314+0000","last_peered":"2024-04-17T08:53:28.963314+0000","last_clean":"2024-04-17T08:53:28.963314+0000","last_became_active":"2024-04-17T08:53:27.755055+0000","last_became_peered":"2024-04-17T08:53:27.755055+0000","last_unstale":"2024-04-17T08:53:28.963314+0000","last_undegraded":"2024-04-17T08:53:28.963314+0000","last_fullsized":"2024-04-17T08:53:28.963314+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:43.164794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734096+0000","last_change":"2024-04-17T08:53:27.735746+0000","last_active":"2024-04-17T08:53:28.734096+0000","last_peered":"2024-04-17T08:53:28.734096+0000","last_clean":"2024-04-17T08:53:28.734096+0000","last_became_active":"2024-04-17T08:53:27.734958+0000","last_became_peered":"2024-04-17T08:53:27.734958+0000","last_unstale":"2024-04-17T08:53:28.734096+0000","last_undegraded":"2024-04-17T08:53:28.734096+0000","last_fullsized":"2024-04-17T08:53:28.734096+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:46.207937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.728041+0000","last_change":"2024-04-17T08:53:27.768242+0000","last_active":"2024-04-17T08:53:28.728041+0000","last_peered":"2024-04-17T08:53:28.728041+0000","last_clean":"2024-04-17T08:53:28.728041+0000","last_became_active":"2024-04-17T08:53:27.767817+0000","last_became_peered":"2024-04-17T08:53:27.767817+0000","last_unstale":"2024-04-17T08:53:28.728041+0000","last_undegraded":"2024-04-17T08:53:28.728041+0000","last_fullsized":"2024-04-17T08:53:28.728041+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:45.468247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764179+0000","last_change":"2024-04-17T08:53:27.762550+0000","last_active":"2024-04-17T08:53:28.764179+0000","last_peered":"2024-04-17T08:53:28.764179+0000","last_clean":"2024-04-17T08:53:28.764179+0000","last_became_active":"2024-04-17T08:53:27.762273+0000","last_became_peered":"2024-04-17T08:53:27.762273+0000","last_unstale":"2024-04-17T08:53:28.764179+0000","last_undegraded":"2024-04-17T08:53:28.764179+0000","last_fullsized":"2024-04-17T08:53:28.764179+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:07.576063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160591+0000","last_change":"2024-04-17T08:53:27.757654+0000","last_active":"2024-04-17T08:53:29.160591+0000","last_peered":"2024-04-17T08:53:29.160591+0000","last_clean":"2024-04-17T08:53:29.160591+0000","last_became_active":"2024-04-17T08:53:27.756517+0000","last_became_peered":"2024-04-17T08:53:27.756517+0000","last_unstale":"2024-04-17T08:53:29.160591+0000","last_undegraded":"2024-04-17T08:53:29.160591+0000","last_fullsized":"2024-04-17T08:53:29.160591+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:52.281394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962678+0000","last_change":"2024-04-17T08:53:27.751778+0000","last_active":"2024-04-17T08:53:28.962678+0000","last_peered":"2024-04-17T08:53:28.962678+0000","last_clean":"2024-04-17T08:53:28.962678+0000","last_became_active":"2024-04-17T08:53:27.751488+0000","last_became_peered":"2024-04-17T08:53:27.751488+0000","last_unstale":"2024-04-17T08:53:28.962678+0000","last_undegraded":"2024-04-17T08:53:28.962678+0000","last_fullsized":"2024-04-17T08:53:28.962678+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:47.254725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733082+0000","last_change":"2024-04-17T08:53:27.760846+0000","last_active":"2024-04-17T08:53:28.733082+0000","last_peered":"2024-04-17T08:53:28.733082+0000","last_clean":"2024-04-17T08:53:28.733082+0000","last_became_active":"2024-04-17T08:53:27.760568+0000","last_became_peered":"2024-04-17T08:53:27.760568+0000","last_unstale":"2024-04-17T08:53:28.733082+0000","last_undegraded":"2024-04-17T08:53:28.733082+0000","last_fullsized":"2024-04-17T08:53:28.733082+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:01.214990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963718+0000","last_change":"2024-04-17T08:53:27.763257+0000","last_active":"2024-04-17T08:53:28.963718+0000","last_peered":"2024-04-17T08:53:28.963718+0000","last_clean":"2024-04-17T08:53:28.963718+0000","last_became_active":"2024-04-17T08:53:27.762591+0000","last_became_peered":"2024-04-17T08:53:27.762591+0000","last_unstale":"2024-04-17T08:53:28.963718+0000","last_undegraded":"2024-04-17T08:53:28.963718+0000","last_fullsized":"2024-04-17T08:53:28.963718+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:50.559615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733226+0000","last_change":"2024-04-17T08:53:27.739805+0000","last_active":"2024-04-17T08:53:28.733226+0000","last_peered":"2024-04-17T08:53:28.733226+0000","last_clean":"2024-04-17T08:53:28.733226+0000","last_became_active":"2024-04-17T08:53:27.739454+0000","last_became_peered":"2024-04-17T08:53:27.739454+0000","last_unstale":"2024-04-17T08:53:28.733226+0000","last_undegraded":"2024-04-17T08:53:28.733226+0000","last_fullsized":"2024-04-17T08:53:28.733226+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:27.744675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767985+0000","last_change":"2024-04-17T08:53:27.761924+0000","last_active":"2024-04-17T08:53:28.767985+0000","last_peered":"2024-04-17T08:53:28.767985+0000","last_clean":"2024-04-17T08:53:28.767985+0000","last_became_active":"2024-04-17T08:53:27.761646+0000","last_became_peered":"2024-04-17T08:53:27.761646+0000","last_unstale":"2024-04-17T08:53:28.767985+0000","last_undegraded":"2024-04-17T08:53:28.767985+0000","last_fullsized":"2024-04-17T08:53:28.767985+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:02.681319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050988+0000","last_change":"2024-04-17T08:53:27.753405+0000","last_active":"2024-04-17T08:53:29.050988+0000","last_peered":"2024-04-17T08:53:29.050988+0000","last_clean":"2024-04-17T08:53:29.050988+0000","last_became_active":"2024-04-17T08:53:27.753037+0000","last_became_peered":"2024-04-17T08:53:27.753037+0000","last_unstale":"2024-04-17T08:53:29.050988+0000","last_undegraded":"2024-04-17T08:53:29.050988+0000","last_fullsized":"2024-04-17T08:53:29.050988+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:06.905009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963406+0000","last_change":"2024-04-17T08:53:27.762072+0000","last_active":"2024-04-17T08:53:28.963406+0000","last_peered":"2024-04-17T08:53:28.963406+0000","last_clean":"2024-04-17T08:53:28.963406+0000","last_became_active":"2024-04-17T08:53:27.761767+0000","last_became_peered":"2024-04-17T08:53:27.761767+0000","last_unstale":"2024-04-17T08:53:28.963406+0000","last_undegraded":"2024-04-17T08:53:28.963406+0000","last_fullsized":"2024-04-17T08:53:28.963406+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:32.684604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734026+0000","last_change":"2024-04-17T08:53:27.754362+0000","last_active":"2024-04-17T08:53:28.734026+0000","last_peered":"2024-04-17T08:53:28.734026+0000","last_clean":"2024-04-17T08:53:28.734026+0000","last_became_active":"2024-04-17T08:53:27.754037+0000","last_became_peered":"2024-04-17T08:53:27.754037+0000","last_unstale":"2024-04-17T08:53:28.734026+0000","last_undegraded":"2024-04-17T08:53:28.734026+0000","last_fullsized":"2024-04-17T08:53:28.734026+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:42.185007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160183+0000","last_change":"2024-04-17T08:53:27.763986+0000","last_active":"2024-04-17T08:53:29.160183+0000","last_peered":"2024-04-17T08:53:29.160183+0000","last_clean":"2024-04-17T08:53:29.160183+0000","last_became_active":"2024-04-17T08:53:27.761675+0000","last_became_peered":"2024-04-17T08:53:27.761675+0000","last_unstale":"2024-04-17T08:53:29.160183+0000","last_undegraded":"2024-04-17T08:53:29.160183+0000","last_fullsized":"2024-04-17T08:53:29.160183+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:17.321293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767991+0000","last_change":"2024-04-17T08:53:27.760269+0000","last_active":"2024-04-17T08:53:28.767991+0000","last_peered":"2024-04-17T08:53:28.767991+0000","last_clean":"2024-04-17T08:53:28.767991+0000","last_became_active":"2024-04-17T08:53:27.757069+0000","last_became_peered":"2024-04-17T08:53:27.757069+0000","last_unstale":"2024-04-17T08:53:28.767991+0000","last_undegraded":"2024-04-17T08:53:28.767991+0000","last_fullsized":"2024-04-17T08:53:28.767991+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:44.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"285'4","reported_seq":18,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734418+0000","last_change":"2024-04-17T08:53:26.743874+0000","last_active":"2024-04-17T08:53:28.734418+0000","last_peered":"2024-04-17T08:53:28.734418+0000","last_clean":"2024-04-17T08:53:28.734418+0000","last_became_active":"2024-04-17T08:53:26.740497+0000","last_became_peered":"2024-04-17T08:53:26.740497+0000","last_unstale":"2024-04-17T08:53:28.734418+0000","last_undegraded":"2024-04-17T08:53:28.734418+0000","last_fullsized":"2024-04-17T08:53:28.734418+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:34:08.891894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733938+0000","last_change":"2024-04-17T08:53:27.751391+0000","last_active":"2024-04-17T08:53:28.733938+0000","last_peered":"2024-04-17T08:53:28.733938+0000","last_clean":"2024-04-17T08:53:28.733938+0000","last_became_active":"2024-04-17T08:53:27.750872+0000","last_became_peered":"2024-04-17T08:53:27.750872+0000","last_unstale":"2024-04-17T08:53:28.733938+0000","last_undegraded":"2024-04-17T08:53:28.733938+0000","last_fullsized":"2024-04-17T08:53:28.733938+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:03.490011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051407+0000","last_change":"2024-04-17T08:53:26.743260+0000","last_active":"2024-04-17T08:53:29.051407+0000","last_peered":"2024-04-17T08:53:29.051407+0000","last_clean":"2024-04-17T08:53:29.051407+0000","last_became_active":"2024-04-17T08:53:26.742962+0000","last_became_peered":"2024-04-17T08:53:26.742962+0000","last_unstale":"2024-04-17T08:53:29.051407+0000","last_undegraded":"2024-04-17T08:53:29.051407+0000","last_fullsized":"2024-04-17T08:53:29.051407+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:57.495011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051102+0000","last_change":"2024-04-17T08:53:27.757214+0000","last_active":"2024-04-17T08:53:29.051102+0000","last_peered":"2024-04-17T08:53:29.051102+0000","last_clean":"2024-04-17T08:53:29.051102+0000","last_became_active":"2024-04-17T08:53:27.755914+0000","last_became_peered":"2024-04-17T08:53:27.755914+0000","last_unstale":"2024-04-17T08:53:29.051102+0000","last_undegraded":"2024-04-17T08:53:29.051102+0000","last_fullsized":"2024-04-17T08:53:29.051102+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:38:06.802840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050963+0000","last_change":"2024-04-17T08:53:26.751887+0000","last_active":"2024-04-17T08:53:29.050963+0000","last_peered":"2024-04-17T08:53:29.050963+0000","last_clean":"2024-04-17T08:53:29.050963+0000","last_became_active":"2024-04-17T08:53:26.751580+0000","last_became_peered":"2024-04-17T08:53:26.751580+0000","last_unstale":"2024-04-17T08:53:29.050963+0000","last_undegraded":"2024-04-17T08:53:29.050963+0000","last_fullsized":"2024-04-17T08:53:29.050963+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:58.199283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764429+0000","last_change":"2024-04-17T08:53:27.753505+0000","last_active":"2024-04-17T08:53:28.764429+0000","last_peered":"2024-04-17T08:53:28.764429+0000","last_clean":"2024-04-17T08:53:28.764429+0000","last_became_active":"2024-04-17T08:53:27.753069+0000","last_became_peered":"2024-04-17T08:53:27.753069+0000","last_unstale":"2024-04-17T08:53:28.764429+0000","last_undegraded":"2024-04-17T08:53:28.764429+0000","last_fullsized":"2024-04-17T08:53:28.764429+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:15.997208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161887+0000","last_change":"2024-04-17T08:53:26.749774+0000","last_active":"2024-04-17T08:53:29.161887+0000","last_peered":"2024-04-17T08:53:29.161887+0000","last_clean":"2024-04-17T08:53:29.161887+0000","last_became_active":"2024-04-17T08:53:26.747101+0000","last_became_peered":"2024-04-17T08:53:26.747101+0000","last_unstale":"2024-04-17T08:53:29.161887+0000","last_undegraded":"2024-04-17T08:53:29.161887+0000","last_fullsized":"2024-04-17T08:53:29.161887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:24.544210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764055+0000","last_change":"2024-04-17T08:53:27.771261+0000","last_active":"2024-04-17T08:53:28.764055+0000","last_peered":"2024-04-17T08:53:28.764055+0000","last_clean":"2024-04-17T08:53:28.764055+0000","last_became_active":"2024-04-17T08:53:27.768177+0000","last_became_peered":"2024-04-17T08:53:27.768177+0000","last_unstale":"2024-04-17T08:53:28.764055+0000","last_undegraded":"2024-04-17T08:53:28.764055+0000","last_fullsized":"2024-04-17T08:53:28.764055+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:44:17.413361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767627+0000","last_change":"2024-04-17T08:53:26.745660+0000","last_active":"2024-04-17T08:53:28.767627+0000","last_peered":"2024-04-17T08:53:28.767627+0000","last_clean":"2024-04-17T08:53:28.767627+0000","last_became_active":"2024-04-17T08:53:26.744798+0000","last_became_peered":"2024-04-17T08:53:26.744798+0000","last_unstale":"2024-04-17T08:53:28.767627+0000","last_undegraded":"2024-04-17T08:53:28.767627+0000","last_fullsized":"2024-04-17T08:53:28.767627+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:26.474714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.726956+0000","last_change":"2024-04-17T08:53:27.771755+0000","last_active":"2024-04-17T08:53:28.726956+0000","last_peered":"2024-04-17T08:53:28.726956+0000","last_clean":"2024-04-17T08:53:28.726956+0000","last_became_active":"2024-04-17T08:53:27.771483+0000","last_became_peered":"2024-04-17T08:53:27.771483+0000","last_unstale":"2024-04-17T08:53:28.726956+0000","last_undegraded":"2024-04-17T08:53:28.726956+0000","last_fullsized":"2024-04-17T08:53:28.726956+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:44.194308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963364+0000","last_change":"2024-04-17T08:53:26.747648+0000","last_active":"2024-04-17T08:53:28.963364+0000","last_peered":"2024-04-17T08:53:28.963364+0000","last_clean":"2024-04-17T08:53:28.963364+0000","last_became_active":"2024-04-17T08:53:26.746769+0000","last_became_peered":"2024-04-17T08:53:26.746769+0000","last_unstale":"2024-04-17T08:53:28.963364+0000","last_undegraded":"2024-04-17T08:53:28.963364+0000","last_fullsized":"2024-04-17T08:53:28.963364+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:14.007388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962571+0000","last_change":"2024-04-17T08:53:27.750595+0000","last_active":"2024-04-17T08:53:28.962571+0000","last_peered":"2024-04-17T08:53:28.962571+0000","last_clean":"2024-04-17T08:53:28.962571+0000","last_became_active":"2024-04-17T08:53:27.750239+0000","last_became_peered":"2024-04-17T08:53:27.750239+0000","last_unstale":"2024-04-17T08:53:28.962571+0000","last_undegraded":"2024-04-17T08:53:28.962571+0000","last_fullsized":"2024-04-17T08:53:28.962571+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:47.233876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767424+0000","last_change":"2024-04-17T08:53:26.751295+0000","last_active":"2024-04-17T08:53:28.767424+0000","last_peered":"2024-04-17T08:53:28.767424+0000","last_clean":"2024-04-17T08:53:28.767424+0000","last_became_active":"2024-04-17T08:53:26.750802+0000","last_became_peered":"2024-04-17T08:53:26.750802+0000","last_unstale":"2024-04-17T08:53:28.767424+0000","last_undegraded":"2024-04-17T08:53:28.767424+0000","last_fullsized":"2024-04-17T08:53:28.767424+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:08.200001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050389+0000","last_change":"2024-04-17T08:53:27.761622+0000","last_active":"2024-04-17T08:53:29.050389+0000","last_peered":"2024-04-17T08:53:29.050389+0000","last_clean":"2024-04-17T08:53:29.050389+0000","last_became_active":"2024-04-17T08:53:27.760836+0000","last_became_peered":"2024-04-17T08:53:27.760836+0000","last_unstale":"2024-04-17T08:53:29.050389+0000","last_undegraded":"2024-04-17T08:53:29.050389+0000","last_fullsized":"2024-04-17T08:53:29.050389+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:04.659149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.049928+0000","last_change":"2024-04-17T08:53:26.748546+0000","last_active":"2024-04-17T08:53:29.049928+0000","last_peered":"2024-04-17T08:53:29.049928+0000","last_clean":"2024-04-17T08:53:29.049928+0000","last_became_active":"2024-04-17T08:53:26.748238+0000","last_became_peered":"2024-04-17T08:53:26.748238+0000","last_unstale":"2024-04-17T08:53:29.049928+0000","last_undegraded":"2024-04-17T08:53:29.049928+0000","last_fullsized":"2024-04-17T08:53:29.049928+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:19.360257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051533+0000","last_change":"2024-04-17T08:53:27.740625+0000","last_active":"2024-04-17T08:53:29.051533+0000","last_peered":"2024-04-17T08:53:29.051533+0000","last_clean":"2024-04-17T08:53:29.051533+0000","last_became_active":"2024-04-17T08:53:27.739857+0000","last_became_peered":"2024-04-17T08:53:27.739857+0000","last_unstale":"2024-04-17T08:53:29.051533+0000","last_undegraded":"2024-04-17T08:53:29.051533+0000","last_fullsized":"2024-04-17T08:53:29.051533+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:30.165464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734696+0000","last_change":"2024-04-17T08:53:26.743930+0000","last_active":"2024-04-17T08:53:28.734696+0000","last_peered":"2024-04-17T08:53:28.734696+0000","last_clean":"2024-04-17T08:53:28.734696+0000","last_became_active":"2024-04-17T08:53:26.743566+0000","last_became_peered":"2024-04-17T08:53:26.743566+0000","last_unstale":"2024-04-17T08:53:28.734696+0000","last_undegraded":"2024-04-17T08:53:28.734696+0000","last_fullsized":"2024-04-17T08:53:28.734696+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:19.426548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734400+0000","last_change":"2024-04-17T08:53:27.737808+0000","last_active":"2024-04-17T08:53:28.734400+0000","last_peered":"2024-04-17T08:53:28.734400+0000","last_clean":"2024-04-17T08:53:28.734400+0000","last_became_active":"2024-04-17T08:53:27.735367+0000","last_became_peered":"2024-04-17T08:53:27.735367+0000","last_unstale":"2024-04-17T08:53:28.734400+0000","last_undegraded":"2024-04-17T08:53:28.734400+0000","last_fullsized":"2024-04-17T08:53:28.734400+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:47.233876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733463+0000","last_change":"2024-04-17T08:53:26.743084+0000","last_active":"2024-04-17T08:53:28.733463+0000","last_peered":"2024-04-17T08:53:28.733463+0000","last_clean":"2024-04-17T08:53:28.733463+0000","last_became_active":"2024-04-17T08:53:26.742793+0000","last_became_peered":"2024-04-17T08:53:26.742793+0000","last_unstale":"2024-04-17T08:53:28.733463+0000","last_undegraded":"2024-04-17T08:53:28.733463+0000","last_fullsized":"2024-04-17T08:53:28.733463+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:08.721488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161736+0000","last_change":"2024-04-17T08:53:27.764587+0000","last_active":"2024-04-17T08:53:29.161736+0000","last_peered":"2024-04-17T08:53:29.161736+0000","last_clean":"2024-04-17T08:53:29.161736+0000","last_became_active":"2024-04-17T08:53:27.763767+0000","last_became_peered":"2024-04-17T08:53:27.763767+0000","last_unstale":"2024-04-17T08:53:29.161736+0000","last_undegraded":"2024-04-17T08:53:29.161736+0000","last_fullsized":"2024-04-17T08:53:29.161736+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:11:26.459401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"285'3","reported_seq":17,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161260+0000","last_change":"2024-04-17T08:53:26.747359+0000","last_active":"2024-04-17T08:53:29.161260+0000","last_peered":"2024-04-17T08:53:29.161260+0000","last_clean":"2024-04-17T08:53:29.161260+0000","last_became_active":"2024-04-17T08:53:26.746728+0000","last_became_peered":"2024-04-17T08:53:26.746728+0000","last_unstale":"2024-04-17T08:53:29.161260+0000","last_undegraded":"2024-04-17T08:53:29.161260+0000","last_fullsized":"2024-04-17T08:53:29.161260+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:19.585571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050925+0000","last_change":"2024-04-17T08:53:27.743702+0000","last_active":"2024-04-17T08:53:29.050925+0000","last_peered":"2024-04-17T08:53:29.050925+0000","last_clean":"2024-04-17T08:53:29.050925+0000","last_became_active":"2024-04-17T08:53:27.743012+0000","last_became_peered":"2024-04-17T08:53:27.743012+0000","last_unstale":"2024-04-17T08:53:29.050925+0000","last_undegraded":"2024-04-17T08:53:29.050925+0000","last_fullsized":"2024-04-17T08:53:29.050925+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:52.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"285'6","reported_seq":20,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962902+0000","last_change":"2024-04-17T08:53:26.747673+0000","last_active":"2024-04-17T08:53:28.962902+0000","last_peered":"2024-04-17T08:53:28.962902+0000","last_clean":"2024-04-17T08:53:28.962902+0000","last_became_active":"2024-04-17T08:53:26.747314+0000","last_became_peered":"2024-04-17T08:53:26.747314+0000","last_unstale":"2024-04-17T08:53:28.962902+0000","last_undegraded":"2024-04-17T08:53:28.962902+0000","last_fullsized":"2024-04-17T08:53:28.962902+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:46.638519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733441+0000","last_change":"2024-04-17T08:53:27.738946+0000","last_active":"2024-04-17T08:53:28.733441+0000","last_peered":"2024-04-17T08:53:28.733441+0000","last_clean":"2024-04-17T08:53:28.733441+0000","last_became_active":"2024-04-17T08:53:27.735949+0000","last_became_peered":"2024-04-17T08:53:27.735949+0000","last_unstale":"2024-04-17T08:53:28.733441+0000","last_undegraded":"2024-04-17T08:53:28.733441+0000","last_fullsized":"2024-04-17T08:53:28.733441+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:38:18.815195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764084+0000","last_change":"2024-04-17T08:53:26.742464+0000","last_active":"2024-04-17T08:53:28.764084+0000","last_peered":"2024-04-17T08:53:28.764084+0000","last_clean":"2024-04-17T08:53:28.764084+0000","last_became_active":"2024-04-17T08:53:26.742089+0000","last_became_peered":"2024-04-17T08:53:26.742089+0000","last_unstale":"2024-04-17T08:53:28.764084+0000","last_undegraded":"2024-04-17T08:53:28.764084+0000","last_fullsized":"2024-04-17T08:53:28.764084+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:09:00.803362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733394+0000","last_change":"2024-04-17T08:53:27.755458+0000","last_active":"2024-04-17T08:53:28.733394+0000","last_peered":"2024-04-17T08:53:28.733394+0000","last_clean":"2024-04-17T08:53:28.733394+0000","last_became_active":"2024-04-17T08:53:27.755173+0000","last_became_peered":"2024-04-17T08:53:27.755173+0000","last_unstale":"2024-04-17T08:53:28.733394+0000","last_undegraded":"2024-04-17T08:53:28.733394+0000","last_fullsized":"2024-04-17T08:53:28.733394+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:22.628804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160521+0000","last_change":"2024-04-17T08:53:26.746821+0000","last_active":"2024-04-17T08:53:29.160521+0000","last_peered":"2024-04-17T08:53:29.160521+0000","last_clean":"2024-04-17T08:53:29.160521+0000","last_became_active":"2024-04-17T08:53:26.746486+0000","last_became_peered":"2024-04-17T08:53:26.746486+0000","last_unstale":"2024-04-17T08:53:29.160521+0000","last_undegraded":"2024-04-17T08:53:29.160521+0000","last_fullsized":"2024-04-17T08:53:29.160521+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:15.686242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767389+0000","last_change":"2024-04-17T08:53:27.767572+0000","last_active":"2024-04-17T08:53:28.767389+0000","last_peered":"2024-04-17T08:53:28.767389+0000","last_clean":"2024-04-17T08:53:28.767389+0000","last_became_active":"2024-04-17T08:53:27.767304+0000","last_became_peered":"2024-04-17T08:53:27.767304+0000","last_unstale":"2024-04-17T08:53:28.767389+0000","last_undegraded":"2024-04-17T08:53:28.767389+0000","last_fullsized":"2024-04-17T08:53:28.767389+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:24.086844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963112+0000","last_change":"2024-04-17T08:53:26.751868+0000","last_active":"2024-04-17T08:53:28.963112+0000","last_peered":"2024-04-17T08:53:28.963112+0000","last_clean":"2024-04-17T08:53:28.963112+0000","last_became_active":"2024-04-17T08:53:26.751504+0000","last_became_peered":"2024-04-17T08:53:26.751504+0000","last_unstale":"2024-04-17T08:53:28.963112+0000","last_undegraded":"2024-04-17T08:53:28.963112+0000","last_fullsized":"2024-04-17T08:53:28.963112+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:31:49.324708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.731326+0000","last_change":"2024-04-17T08:53:27.734856+0000","last_active":"2024-04-17T08:53:28.731326+0000","last_peered":"2024-04-17T08:53:28.731326+0000","last_clean":"2024-04-17T08:53:28.731326+0000","last_became_active":"2024-04-17T08:53:27.734211+0000","last_became_peered":"2024-04-17T08:53:27.734211+0000","last_unstale":"2024-04-17T08:53:28.731326+0000","last_undegraded":"2024-04-17T08:53:28.731326+0000","last_fullsized":"2024-04-17T08:53:28.731326+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:28.278067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"285'8","reported_seq":22,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160350+0000","last_change":"2024-04-17T08:53:26.751597+0000","last_active":"2024-04-17T08:53:29.160350+0000","last_peered":"2024-04-17T08:53:29.160350+0000","last_clean":"2024-04-17T08:53:29.160350+0000","last_became_active":"2024-04-17T08:53:26.751259+0000","last_became_peered":"2024-04-17T08:53:26.751259+0000","last_unstale":"2024-04-17T08:53:29.160350+0000","last_undegraded":"2024-04-17T08:53:29.160350+0000","last_fullsized":"2024-04-17T08:53:29.160350+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:47.079433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161628+0000","last_change":"2024-04-17T08:53:27.759892+0000","last_active":"2024-04-17T08:53:29.161628+0000","last_peered":"2024-04-17T08:53:29.161628+0000","last_clean":"2024-04-17T08:53:29.161628+0000","last_became_active":"2024-04-17T08:53:27.759613+0000","last_became_peered":"2024-04-17T08:53:27.759613+0000","last_unstale":"2024-04-17T08:53:29.161628+0000","last_undegraded":"2024-04-17T08:53:29.161628+0000","last_fullsized":"2024-04-17T08:53:29.161628+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:09.350160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764496+0000","last_change":"2024-04-17T08:53:26.733363+0000","last_active":"2024-04-17T08:53:28.764496+0000","last_peered":"2024-04-17T08:53:28.764496+0000","last_clean":"2024-04-17T08:53:28.764496+0000","last_became_active":"2024-04-17T08:53:26.732933+0000","last_became_peered":"2024-04-17T08:53:26.732933+0000","last_unstale":"2024-04-17T08:53:28.764496+0000","last_undegraded":"2024-04-17T08:53:28.764496+0000","last_fullsized":"2024-04-17T08:53:28.764496+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:04.414260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.768301+0000","last_change":"2024-04-17T08:53:27.760742+0000","last_active":"2024-04-17T08:53:28.768301+0000","last_peered":"2024-04-17T08:53:28.768301+0000","last_clean":"2024-04-17T08:53:28.768301+0000","last_became_active":"2024-04-17T08:53:27.759388+0000","last_became_peered":"2024-04-17T08:53:27.759388+0000","last_unstale":"2024-04-17T08:53:28.768301+0000","last_undegraded":"2024-04-17T08:53:28.768301+0000","last_fullsized":"2024-04-17T08:53:28.768301+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:06.580148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733833+0000","last_change":"2024-04-17T08:53:26.741308+0000","last_active":"2024-04-17T08:53:28.733833+0000","last_peered":"2024-04-17T08:53:28.733833+0000","last_clean":"2024-04-17T08:53:28.733833+0000","last_became_active":"2024-04-17T08:53:26.740903+0000","last_became_peered":"2024-04-17T08:53:26.740903+0000","last_unstale":"2024-04-17T08:53:28.733833+0000","last_undegraded":"2024-04-17T08:53:28.733833+0000","last_fullsized":"2024-04-17T08:53:28.733833+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:45.272229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051449+0000","last_change":"2024-04-17T08:53:27.766005+0000","last_active":"2024-04-17T08:53:29.051449+0000","last_peered":"2024-04-17T08:53:29.051449+0000","last_clean":"2024-04-17T08:53:29.051449+0000","last_became_active":"2024-04-17T08:53:27.765741+0000","last_became_peered":"2024-04-17T08:53:27.765741+0000","last_unstale":"2024-04-17T08:53:29.051449+0000","last_undegraded":"2024-04-17T08:53:29.051449+0000","last_fullsized":"2024-04-17T08:53:29.051449+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:31:58.805099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051261+0000","last_change":"2024-04-17T08:53:26.744543+0000","last_active":"2024-04-17T08:53:29.051261+0000","last_peered":"2024-04-17T08:53:29.051261+0000","last_clean":"2024-04-17T08:53:29.051261+0000","last_became_active":"2024-04-17T08:53:26.741353+0000","last_became_peered":"2024-04-17T08:53:26.741353+0000","last_unstale":"2024-04-17T08:53:29.051261+0000","last_undegraded":"2024-04-17T08:53:29.051261+0000","last_fullsized":"2024-04-17T08:53:29.051261+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:51.108689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734681+0000","last_change":"2024-04-17T08:53:27.764495+0000","last_active":"2024-04-17T08:53:28.734681+0000","last_peered":"2024-04-17T08:53:28.734681+0000","last_clean":"2024-04-17T08:53:28.734681+0000","last_became_active":"2024-04-17T08:53:27.764119+0000","last_became_peered":"2024-04-17T08:53:27.764119+0000","last_unstale":"2024-04-17T08:53:28.734681+0000","last_undegraded":"2024-04-17T08:53:28.734681+0000","last_fullsized":"2024-04-17T08:53:28.734681+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:35.119011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051348+0000","last_change":"2024-04-17T08:53:26.749738+0000","last_active":"2024-04-17T08:53:29.051348+0000","last_peered":"2024-04-17T08:53:29.051348+0000","last_clean":"2024-04-17T08:53:29.051348+0000","last_became_active":"2024-04-17T08:53:26.749434+0000","last_became_peered":"2024-04-17T08:53:26.749434+0000","last_unstale":"2024-04-17T08:53:29.051348+0000","last_undegraded":"2024-04-17T08:53:29.051348+0000","last_fullsized":"2024-04-17T08:53:29.051348+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:16.942239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767624+0000","last_change":"2024-04-17T08:53:27.759602+0000","last_active":"2024-04-17T08:53:28.767624+0000","last_peered":"2024-04-17T08:53:28.767624+0000","last_clean":"2024-04-17T08:53:28.767624+0000","last_became_active":"2024-04-17T08:53:27.759246+0000","last_became_peered":"2024-04-17T08:53:27.759246+0000","last_unstale":"2024-04-17T08:53:28.767624+0000","last_undegraded":"2024-04-17T08:53:28.767624+0000","last_fullsized":"2024-04-17T08:53:28.767624+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:11.101157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160782+0000","last_change":"2024-04-17T08:53:26.750785+0000","last_active":"2024-04-17T08:53:29.160782+0000","last_peered":"2024-04-17T08:53:29.160782+0000","last_clean":"2024-04-17T08:53:29.160782+0000","last_became_active":"2024-04-17T08:53:26.746628+0000","last_became_peered":"2024-04-17T08:53:26.746628+0000","last_unstale":"2024-04-17T08:53:29.160782+0000","last_undegraded":"2024-04-17T08:53:29.160782+0000","last_fullsized":"2024-04-17T08:53:29.160782+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:36.492477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.727771+0000","last_change":"2024-04-17T08:53:27.759085+0000","last_active":"2024-04-17T08:53:28.727771+0000","last_peered":"2024-04-17T08:53:28.727771+0000","last_clean":"2024-04-17T08:53:28.727771+0000","last_became_active":"2024-04-17T08:53:27.758789+0000","last_became_peered":"2024-04-17T08:53:27.758789+0000","last_unstale":"2024-04-17T08:53:28.727771+0000","last_undegraded":"2024-04-17T08:53:28.727771+0000","last_fullsized":"2024-04-17T08:53:28.727771+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:08.507541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764070+0000","last_change":"2024-04-17T08:53:26.737860+0000","last_active":"2024-04-17T08:53:28.764070+0000","last_peered":"2024-04-17T08:53:28.764070+0000","last_clean":"2024-04-17T08:53:28.764070+0000","last_became_active":"2024-04-17T08:53:26.737488+0000","last_became_peered":"2024-04-17T08:53:26.737488+0000","last_unstale":"2024-04-17T08:53:28.764070+0000","last_undegraded":"2024-04-17T08:53:28.764070+0000","last_fullsized":"2024-04-17T08:53:28.764070+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:54.388994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763883+0000","last_change":"2024-04-17T08:53:27.758116+0000","last_active":"2024-04-17T08:53:28.763883+0000","last_peered":"2024-04-17T08:53:28.763883+0000","last_clean":"2024-04-17T08:53:28.763883+0000","last_became_active":"2024-04-17T08:53:27.757774+0000","last_became_peered":"2024-04-17T08:53:27.757774+0000","last_unstale":"2024-04-17T08:53:28.763883+0000","last_undegraded":"2024-04-17T08:53:28.763883+0000","last_fullsized":"2024-04-17T08:53:28.763883+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:52.088922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050375+0000","last_change":"2024-04-17T08:53:26.746525+0000","last_active":"2024-04-17T08:53:29.050375+0000","last_peered":"2024-04-17T08:53:29.050375+0000","last_clean":"2024-04-17T08:53:29.050375+0000","last_became_active":"2024-04-17T08:53:26.746243+0000","last_became_peered":"2024-04-17T08:53:26.746243+0000","last_unstale":"2024-04-17T08:53:29.050375+0000","last_undegraded":"2024-04-17T08:53:29.050375+0000","last_fullsized":"2024-04-17T08:53:29.050375+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:59.343774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764322+0000","last_change":"2024-04-17T08:53:27.759138+0000","last_active":"2024-04-17T08:53:28.764322+0000","last_peered":"2024-04-17T08:53:28.764322+0000","last_clean":"2024-04-17T08:53:28.764322+0000","last_became_active":"2024-04-17T08:53:27.758840+0000","last_became_peered":"2024-04-17T08:53:27.758840+0000","last_unstale":"2024-04-17T08:53:28.764322+0000","last_undegraded":"2024-04-17T08:53:28.764322+0000","last_fullsized":"2024-04-17T08:53:28.764322+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:37.949351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763998+0000","last_change":"2024-04-17T08:53:26.738392+0000","last_active":"2024-04-17T08:53:28.763998+0000","last_peered":"2024-04-17T08:53:28.763998+0000","last_clean":"2024-04-17T08:53:28.763998+0000","last_became_active":"2024-04-17T08:53:26.738088+0000","last_became_peered":"2024-04-17T08:53:26.738088+0000","last_unstale":"2024-04-17T08:53:28.763998+0000","last_undegraded":"2024-04-17T08:53:28.763998+0000","last_fullsized":"2024-04-17T08:53:28.763998+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:00.572399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.728121+0000","last_change":"2024-04-17T08:53:27.765625+0000","last_active":"2024-04-17T08:53:28.728121+0000","last_peered":"2024-04-17T08:53:28.728121+0000","last_clean":"2024-04-17T08:53:28.728121+0000","last_became_active":"2024-04-17T08:53:27.765345+0000","last_became_peered":"2024-04-17T08:53:27.765345+0000","last_unstale":"2024-04-17T08:53:28.728121+0000","last_undegraded":"2024-04-17T08:53:28.728121+0000","last_fullsized":"2024-04-17T08:53:28.728121+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:23.225419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161826+0000","last_change":"2024-04-17T08:53:26.746062+0000","last_active":"2024-04-17T08:53:29.161826+0000","last_peered":"2024-04-17T08:53:29.161826+0000","last_clean":"2024-04-17T08:53:29.161826+0000","last_became_active":"2024-04-17T08:53:26.745779+0000","last_became_peered":"2024-04-17T08:53:26.745779+0000","last_unstale":"2024-04-17T08:53:29.161826+0000","last_undegraded":"2024-04-17T08:53:29.161826+0000","last_fullsized":"2024-04-17T08:53:29.161826+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:35.197781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963920+0000","last_change":"2024-04-17T08:53:27.762182+0000","last_active":"2024-04-17T08:53:28.963920+0000","last_peered":"2024-04-17T08:53:28.963920+0000","last_clean":"2024-04-17T08:53:28.963920+0000","last_became_active":"2024-04-17T08:53:27.761799+0000","last_became_peered":"2024-04-17T08:53:27.761799+0000","last_unstale":"2024-04-17T08:53:28.963920+0000","last_undegraded":"2024-04-17T08:53:28.963920+0000","last_fullsized":"2024-04-17T08:53:28.963920+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:18.619031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"286'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.766518+0000","last_change":"2024-04-17T08:53:26.749321+0000","last_active":"2024-04-17T08:53:28.766518+0000","last_peered":"2024-04-17T08:53:28.766518+0000","last_clean":"2024-04-17T08:53:28.766518+0000","last_became_active":"2024-04-17T08:53:26.748989+0000","last_became_peered":"2024-04-17T08:53:26.748989+0000","last_unstale":"2024-04-17T08:53:28.766518+0000","last_undegraded":"2024-04-17T08:53:28.766518+0000","last_fullsized":"2024-04-17T08:53:28.766518+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:06:22.673592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733395+0000","last_change":"2024-04-17T08:53:27.765517+0000","last_active":"2024-04-17T08:53:28.733395+0000","last_peered":"2024-04-17T08:53:28.733395+0000","last_clean":"2024-04-17T08:53:28.733395+0000","last_became_active":"2024-04-17T08:53:27.765133+0000","last_became_peered":"2024-04-17T08:53:27.765133+0000","last_unstale":"2024-04-17T08:53:28.733395+0000","last_undegraded":"2024-04-17T08:53:28.733395+0000","last_fullsized":"2024-04-17T08:53:28.733395+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.909810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161390+0000","last_change":"2024-04-17T08:53:26.750892+0000","last_active":"2024-04-17T08:53:29.161390+0000","last_peered":"2024-04-17T08:53:29.161390+0000","last_clean":"2024-04-17T08:53:29.161390+0000","last_became_active":"2024-04-17T08:53:26.747783+0000","last_became_peered":"2024-04-17T08:53:26.747783+0000","last_unstale":"2024-04-17T08:53:29.161390+0000","last_undegraded":"2024-04-17T08:53:29.161390+0000","last_fullsized":"2024-04-17T08:53:29.161390+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:18.301387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732666+0000","last_change":"2024-04-17T08:53:27.747949+0000","last_active":"2024-04-17T08:53:28.732666+0000","last_peered":"2024-04-17T08:53:28.732666+0000","last_clean":"2024-04-17T08:53:28.732666+0000","last_became_active":"2024-04-17T08:53:27.747593+0000","last_became_peered":"2024-04-17T08:53:27.747593+0000","last_unstale":"2024-04-17T08:53:28.732666+0000","last_undegraded":"2024-04-17T08:53:28.732666+0000","last_fullsized":"2024-04-17T08:53:28.732666+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:10.441752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767958+0000","last_change":"2024-04-17T08:53:26.744431+0000","last_active":"2024-04-17T08:53:28.767958+0000","last_peered":"2024-04-17T08:53:28.767958+0000","last_clean":"2024-04-17T08:53:28.767958+0000","last_became_active":"2024-04-17T08:53:26.744114+0000","last_became_peered":"2024-04-17T08:53:26.744114+0000","last_unstale":"2024-04-17T08:53:28.767958+0000","last_undegraded":"2024-04-17T08:53:28.767958+0000","last_fullsized":"2024-04-17T08:53:28.767958+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:26.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050979+0000","last_change":"2024-04-17T08:53:27.755114+0000","last_active":"2024-04-17T08:53:29.050979+0000","last_peered":"2024-04-17T08:53:29.050979+0000","last_clean":"2024-04-17T08:53:29.050979+0000","last_became_active":"2024-04-17T08:53:27.754780+0000","last_became_peered":"2024-04-17T08:53:27.754780+0000","last_unstale":"2024-04-17T08:53:29.050979+0000","last_undegraded":"2024-04-17T08:53:29.050979+0000","last_fullsized":"2024-04-17T08:53:29.050979+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:58.116875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767780+0000","last_change":"2024-04-17T08:53:26.745411+0000","last_active":"2024-04-17T08:53:28.767780+0000","last_peered":"2024-04-17T08:53:28.767780+0000","last_clean":"2024-04-17T08:53:28.767780+0000","last_became_active":"2024-04-17T08:53:26.744057+0000","last_became_peered":"2024-04-17T08:53:26.744057+0000","last_unstale":"2024-04-17T08:53:28.767780+0000","last_undegraded":"2024-04-17T08:53:28.767780+0000","last_fullsized":"2024-04-17T08:53:28.767780+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:00:27.500568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160658+0000","last_change":"2024-04-17T08:53:27.754343+0000","last_active":"2024-04-17T08:53:29.160658+0000","last_peered":"2024-04-17T08:53:29.160658+0000","last_clean":"2024-04-17T08:53:29.160658+0000","last_became_active":"2024-04-17T08:53:27.753840+0000","last_became_peered":"2024-04-17T08:53:27.753840+0000","last_unstale":"2024-04-17T08:53:29.160658+0000","last_undegraded":"2024-04-17T08:53:29.160658+0000","last_fullsized":"2024-04-17T08:53:29.160658+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:36.160059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160717+0000","last_change":"2024-04-17T08:53:26.750396+0000","last_active":"2024-04-17T08:53:29.160717+0000","last_peered":"2024-04-17T08:53:29.160717+0000","last_clean":"2024-04-17T08:53:29.160717+0000","last_became_active":"2024-04-17T08:53:26.750083+0000","last_became_peered":"2024-04-17T08:53:26.750083+0000","last_unstale":"2024-04-17T08:53:29.160717+0000","last_undegraded":"2024-04-17T08:53:29.160717+0000","last_fullsized":"2024-04-17T08:53:29.160717+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:02.113591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763882+0000","last_change":"2024-04-17T08:53:27.761040+0000","last_active":"2024-04-17T08:53:28.763882+0000","last_peered":"2024-04-17T08:53:28.763882+0000","last_clean":"2024-04-17T08:53:28.763882+0000","last_became_active":"2024-04-17T08:53:27.760475+0000","last_became_peered":"2024-04-17T08:53:27.760475+0000","last_unstale":"2024-04-17T08:53:28.763882+0000","last_undegraded":"2024-04-17T08:53:28.763882+0000","last_fullsized":"2024-04-17T08:53:28.763882+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:37.569340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734110+0000","last_change":"2024-04-17T08:53:26.744129+0000","last_active":"2024-04-17T08:53:28.734110+0000","last_peered":"2024-04-17T08:53:28.734110+0000","last_clean":"2024-04-17T08:53:28.734110+0000","last_became_active":"2024-04-17T08:53:26.743810+0000","last_became_peered":"2024-04-17T08:53:26.743810+0000","last_unstale":"2024-04-17T08:53:28.734110+0000","last_undegraded":"2024-04-17T08:53:28.734110+0000","last_fullsized":"2024-04-17T08:53:28.734110+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:28.088593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.731485+0000","last_change":"2024-04-17T08:53:27.740704+0000","last_active":"2024-04-17T08:53:28.731485+0000","last_peered":"2024-04-17T08:53:28.731485+0000","last_clean":"2024-04-17T08:53:28.731485+0000","last_became_active":"2024-04-17T08:53:27.739818+0000","last_became_peered":"2024-04-17T08:53:27.739818+0000","last_unstale":"2024-04-17T08:53:28.731485+0000","last_undegraded":"2024-04-17T08:53:28.731485+0000","last_fullsized":"2024-04-17T08:53:28.731485+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:59.716934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160318+0000","last_change":"2024-04-17T08:53:26.751325+0000","last_active":"2024-04-17T08:53:29.160318+0000","last_peered":"2024-04-17T08:53:29.160318+0000","last_clean":"2024-04-17T08:53:29.160318+0000","last_became_active":"2024-04-17T08:53:26.750739+0000","last_became_peered":"2024-04-17T08:53:26.750739+0000","last_unstale":"2024-04-17T08:53:29.160318+0000","last_undegraded":"2024-04-17T08:53:29.160318+0000","last_fullsized":"2024-04-17T08:53:29.160318+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:19.069226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161417+0000","last_change":"2024-04-17T08:53:27.765857+0000","last_active":"2024-04-17T08:53:29.161417+0000","last_peered":"2024-04-17T08:53:29.161417+0000","last_clean":"2024-04-17T08:53:29.161417+0000","last_became_active":"2024-04-17T08:53:27.763911+0000","last_became_peered":"2024-04-17T08:53:27.763911+0000","last_unstale":"2024-04-17T08:53:29.161417+0000","last_undegraded":"2024-04-17T08:53:29.161417+0000","last_fullsized":"2024-04-17T08:53:29.161417+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:21.006238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"274'2771","reported_seq":4143,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767467+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:53:28.767467+0000","last_peered":"2024-04-17T08:53:28.767467+0000","last_clean":"2024-04-17T08:53:28.767467+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:53:28.767467+0000","last_undegraded":"2024-04-17T08:53:28.767467+0000","last_fullsized":"2024-04-17T08:53:28.767467+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2771,"ondisk_log_size":2771,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543325,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53332,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318508,"statfs":{"total":96636764160,"available":96582152192,"internally_reserved":0,"allocated":4202496,"data_stored":2371303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543325,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61080,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310760,"statfs":{"total":96636764160,"available":96574218240,"internally_reserved":0,"allocated":3608576,"data_stored":1776219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543324,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52772,"kb_used_data":3540,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319068,"statfs":{"total":96636764160,"available":96582725632,"internally_reserved":0,"allocated":3624960,"data_stored":1782637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543324,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84636,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287204,"statfs":{"total":96636764160,"available":96550096896,"internally_reserved":0,"allocated":3616768,"data_stored":1781587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543326,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52756,"kb_used_data":3528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3612672,"data_stored":1781517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543325,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3616768,"data_stored":1781059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543326,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48600,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323240,"statfs":{"total":96636764160,"available":96586997760,"internally_reserved":0,"allocated":3616768,"data_stored":1780991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543324,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57500,"kb_used_data":4112,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314340,"statfs":{"total":96636764160,"available":96577884160,"internally_reserved":0,"allocated":4210688,"data_stored":2371395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:53:32.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:53:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.499+0000 7f0e0fe00640 1 Processor -- start 2024-04-17T08:53:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.500+0000 7f0e0fe00640 1 -- start start 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.501+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 0x7f0e100b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.501+0000 7f0e0fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e100d2d30 con 0x7f0e100b6b30 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.501+0000 7f0e0fe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e100d3010 con 0x7f0e100cfae0 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.501+0000 7f0e0fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e100d49c0 con 0x7f0e1015ec60 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.502+0000 7f0e07e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e100b6b30 0x7f0e100cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58326/0 (socket says 172.21.15.67:58326) 2024-04-17T08:53:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.502+0000 7f0e07e00640 1 -- 172.21.15.67:0/3932599311 learned_addr learned my addr 172.21.15.67:0/3932599311 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.502+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 0x7f0e100b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.502+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730491320 0 0) 0x7f0e100d2d30 con 0x7f0e100b6b30 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.502+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0df4003520 con 0x7f0e100b6b30 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3159569358 0 0) 0x7f0df4003520 con 0x7f0e100b6b30 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 msgr2=0x7f0e100b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 0x7f0e100b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 >> v1:172.21.15.187:6789/0 conn(0x7f0e100cfae0 legacy=0x7f0e100b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e100d32f0 con 0x7f0e100b6b30 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0df8001670 con 0x7f0e100b6b30 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 >> v1:172.21.15.67:6789/0 conn(0x7f0e100b6b30 legacy=0x7f0e100cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4003a50 0x7f0df4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0df4004800 0x7f0df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 0x7f0df40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e100d49c0 con 0x7f0df4004800 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0df400bac0 con 0x7f0df4003a50 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0df400bcc0 con 0x7f0e1015ec60 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.503+0000 7f0e07e00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0df4004800 0x7f0df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4003a50 0x7f0df4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e07e00640 1 -- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 msgr2=0x7f0df40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e07e00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e1015ec60 0x7f0df40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e07e00640 1 -- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4003a50 msgr2=0x7f0df4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e07e00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4003a50 0x7f0df4003e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.504+0000 7f0e07e00640 1 -- 172.21.15.67:0/3932599311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df4003520 con 0x7f0df4004800 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.505+0000 7f0e07e00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0df4004800 0x7f0df4004c80 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c240 tx=0x7f0df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515c5e734909c1f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.505+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8007d10 con 0x7f0df4004800 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.505+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df8030070 con 0x7f0df4004800 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.505+0000 7f0e0e000640 1 -- 172.21.15.67:0/3932599311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df802cd60 con 0x7f0df4004800 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 -- 172.21.15.67:0/3932599311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0df4004800 msgr2=0x7f0df4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/3932599311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0df4004800 0x7f0df4004c80 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f0df800c240 tx=0x7f0df4005830 comp rx=0 tx=0).stop 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0f400640 1 -- 172.21.15.67:0/3932599311 reap_dead start 2024-04-17T08:53:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 -- 172.21.15.67:0/3932599311 shutdown_connections 2024-04-17T08:53:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 -- 172.21.15.67:0/3932599311 >> 172.21.15.67:0/3932599311 conn(0x7f0e100d15f0 msgr2=0x7f0e100d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 -- 172.21.15.67:0/3932599311 shutdown_connections 2024-04-17T08:53:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.506+0000 7f0e0fe00640 1 -- 172.21.15.67:0/3932599311 wait complete. 2024-04-17T08:53:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.508+0000 7f0e0fe00640 1 Processor -- start 2024-04-17T08:53:32.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.508+0000 7f0e0fe00640 1 -- start start 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 0x7f0e10154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e10157b20 0x7f0e10155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e100b6f70 con 0x7f0e10156fb0 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0e100d3660 con 0x7f0df4004800 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0fe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e100d00c0 con 0x7f0e10157b20 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46732/0 (socket says 172.21.15.67:46732) 2024-04-17T08:53:32.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0f400640 1 -- 172.21.15.67:0/2545422496 learned_addr learned my addr 172.21.15.67:0/2545422496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 0x7f0e10154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.509+0000 7f0e07e00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e10157b20 0x7f0e10155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 -- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e10157b20 msgr2=0x7f0e10155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e10157b20 0x7f0e10155a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 -- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 msgr2=0x7f0e10154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 0x7f0e10154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 -- 172.21.15.67:0/2545422496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e10155fa0 con 0x7f0e10156fb0 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 0x7f0e10154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0f400640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f0e1008cd30 tx=0x7f0e0800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08017070 con 0x7f0e10156fb0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e08005d60 con 0x7f0e10156fb0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0800a850 con 0x7f0e10156fb0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1015b2f0 con 0x7f0e10156fb0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.510+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e100cc7f0 con 0x7f0e10156fb0 2024-04-17T08:53:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.511+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e10068a10 con 0x7f0e10156fb0 2024-04-17T08:53:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.513+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f0e0801a070 con 0x7f0e10156fb0 2024-04-17T08:53:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.513+0000 7f0e06a00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 0x7f0dec06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.514+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f0e0808aff0 con 0x7f0e10156fb0 2024-04-17T08:53:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.514+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 0x7f0dec06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:32.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.515+0000 7f0e0ea00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 0x7f0dec06ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009a00 tx=0x7f0e00008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:32.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.518+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0e080592e0 con 0x7f0e10156fb0 2024-04-17T08:53:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.710+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 --> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e1015dbe0 con 0x7f0dec06ca40 2024-04-17T08:53:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.730+0000 7f0e06a00640 1 -- 172.21.15.67:0/2545422496 <== mgr.9871 v2:172.21.15.187:6832/46399553 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252506 (secure 0 0 0) 0x7f0e1015dbe0 con 0x7f0dec06ca40 2024-04-17T08:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:32.734 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:53:32.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 msgr2=0x7f0dec06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 0x7f0dec06ef00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009a00 tx=0x7f0e00008040 comp rx=0 tx=0).stop 2024-04-17T08:53:32.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 msgr2=0x7f0e10157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:32.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f0e1008cd30 tx=0x7f0e0800cb90 comp rx=0 tx=0).stop 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 shutdown_connections 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0dec06ca40 0x7f0dec06ef00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10156fb0 0x7f0e10157430 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e10157b20 0x7f0e10155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2545422496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0df4004800 0x7f0e10154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 >> 172.21.15.67:0/2545422496 conn(0x7f0e100d15f0 msgr2=0x7f0e10074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 shutdown_connections 2024-04-17T08:53:32.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:32.736+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2545422496 wait complete. 2024-04-17T08:53:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-17T08:53:32.547257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":734,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":463436,"kb_used_data":29404,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754511284,"statfs":{"total":773094113280,"available":772619554816,"internally_reserved":0,"allocated":30109696,"data_stored":15426708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002189"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732426+0000","last_change":"2024-04-17T08:53:27.760015+0000","last_active":"2024-04-17T08:53:28.732426+0000","last_peered":"2024-04-17T08:53:28.732426+0000","last_clean":"2024-04-17T08:53:28.732426+0000","last_became_active":"2024-04-17T08:53:27.755424+0000","last_became_peered":"2024-04-17T08:53:27.755424+0000","last_unstale":"2024-04-17T08:53:28.732426+0000","last_undegraded":"2024-04-17T08:53:28.732426+0000","last_fullsized":"2024-04-17T08:53:28.732426+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:54.090607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.726945+0000","last_change":"2024-04-17T08:53:27.761717+0000","last_active":"2024-04-17T08:53:28.726945+0000","last_peered":"2024-04-17T08:53:28.726945+0000","last_clean":"2024-04-17T08:53:28.726945+0000","last_became_active":"2024-04-17T08:53:27.761172+0000","last_became_peered":"2024-04-17T08:53:27.761172+0000","last_unstale":"2024-04-17T08:53:28.726945+0000","last_undegraded":"2024-04-17T08:53:28.726945+0000","last_fullsized":"2024-04-17T08:53:28.726945+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:03.990833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161780+0000","last_change":"2024-04-17T08:53:27.765941+0000","last_active":"2024-04-17T08:53:29.161780+0000","last_peered":"2024-04-17T08:53:29.161780+0000","last_clean":"2024-04-17T08:53:29.161780+0000","last_became_active":"2024-04-17T08:53:27.763199+0000","last_became_peered":"2024-04-17T08:53:27.763199+0000","last_unstale":"2024-04-17T08:53:29.161780+0000","last_undegraded":"2024-04-17T08:53:29.161780+0000","last_fullsized":"2024-04-17T08:53:29.161780+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:27:27.337553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161320+0000","last_change":"2024-04-17T08:53:27.766908+0000","last_active":"2024-04-17T08:53:29.161320+0000","last_peered":"2024-04-17T08:53:29.161320+0000","last_clean":"2024-04-17T08:53:29.161320+0000","last_became_active":"2024-04-17T08:53:27.765440+0000","last_became_peered":"2024-04-17T08:53:27.765440+0000","last_unstale":"2024-04-17T08:53:29.161320+0000","last_undegraded":"2024-04-17T08:53:29.161320+0000","last_fullsized":"2024-04-17T08:53:29.161320+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:42.329109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160797+0000","last_change":"2024-04-17T08:53:27.755767+0000","last_active":"2024-04-17T08:53:29.160797+0000","last_peered":"2024-04-17T08:53:29.160797+0000","last_clean":"2024-04-17T08:53:29.160797+0000","last_became_active":"2024-04-17T08:53:27.754616+0000","last_became_peered":"2024-04-17T08:53:27.754616+0000","last_unstale":"2024-04-17T08:53:29.160797+0000","last_undegraded":"2024-04-17T08:53:29.160797+0000","last_fullsized":"2024-04-17T08:53:29.160797+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:52.661584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050561+0000","last_change":"2024-04-17T08:53:27.757627+0000","last_active":"2024-04-17T08:53:29.050561+0000","last_peered":"2024-04-17T08:53:29.050561+0000","last_clean":"2024-04-17T08:53:29.050561+0000","last_became_active":"2024-04-17T08:53:27.756356+0000","last_became_peered":"2024-04-17T08:53:27.756356+0000","last_unstale":"2024-04-17T08:53:29.050561+0000","last_undegraded":"2024-04-17T08:53:29.050561+0000","last_fullsized":"2024-04-17T08:53:29.050561+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:09:04.684548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.730725+0000","last_change":"2024-04-17T08:53:27.755416+0000","last_active":"2024-04-17T08:53:28.730725+0000","last_peered":"2024-04-17T08:53:28.730725+0000","last_clean":"2024-04-17T08:53:28.730725+0000","last_became_active":"2024-04-17T08:53:27.755106+0000","last_became_peered":"2024-04-17T08:53:27.755106+0000","last_unstale":"2024-04-17T08:53:28.730725+0000","last_undegraded":"2024-04-17T08:53:28.730725+0000","last_fullsized":"2024-04-17T08:53:28.730725+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:04:44.298092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763747+0000","last_change":"2024-04-17T08:53:27.768291+0000","last_active":"2024-04-17T08:53:28.763747+0000","last_peered":"2024-04-17T08:53:28.763747+0000","last_clean":"2024-04-17T08:53:28.763747+0000","last_became_active":"2024-04-17T08:53:27.767995+0000","last_became_peered":"2024-04-17T08:53:27.767995+0000","last_unstale":"2024-04-17T08:53:28.763747+0000","last_undegraded":"2024-04-17T08:53:28.763747+0000","last_fullsized":"2024-04-17T08:53:28.763747+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:46:40.621216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.727789+0000","last_change":"2024-04-17T08:53:27.764198+0000","last_active":"2024-04-17T08:53:28.727789+0000","last_peered":"2024-04-17T08:53:28.727789+0000","last_clean":"2024-04-17T08:53:28.727789+0000","last_became_active":"2024-04-17T08:53:27.762546+0000","last_became_peered":"2024-04-17T08:53:27.762546+0000","last_unstale":"2024-04-17T08:53:28.727789+0000","last_undegraded":"2024-04-17T08:53:28.727789+0000","last_fullsized":"2024-04-17T08:53:28.727789+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:33:05.811195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732750+0000","last_change":"2024-04-17T08:53:27.758757+0000","last_active":"2024-04-17T08:53:28.732750+0000","last_peered":"2024-04-17T08:53:28.732750+0000","last_clean":"2024-04-17T08:53:28.732750+0000","last_became_active":"2024-04-17T08:53:27.757344+0000","last_became_peered":"2024-04-17T08:53:27.757344+0000","last_unstale":"2024-04-17T08:53:28.732750+0000","last_undegraded":"2024-04-17T08:53:28.732750+0000","last_fullsized":"2024-04-17T08:53:28.732750+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:19.631870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764198+0000","last_change":"2024-04-17T08:53:27.765421+0000","last_active":"2024-04-17T08:53:28.764198+0000","last_peered":"2024-04-17T08:53:28.764198+0000","last_clean":"2024-04-17T08:53:28.764198+0000","last_became_active":"2024-04-17T08:53:27.765135+0000","last_became_peered":"2024-04-17T08:53:27.765135+0000","last_unstale":"2024-04-17T08:53:28.764198+0000","last_undegraded":"2024-04-17T08:53:28.764198+0000","last_fullsized":"2024-04-17T08:53:28.764198+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:00:57.079390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767909+0000","last_change":"2024-04-17T08:53:27.766495+0000","last_active":"2024-04-17T08:53:28.767909+0000","last_peered":"2024-04-17T08:53:28.767909+0000","last_clean":"2024-04-17T08:53:28.767909+0000","last_became_active":"2024-04-17T08:53:27.766200+0000","last_became_peered":"2024-04-17T08:53:27.766200+0000","last_unstale":"2024-04-17T08:53:28.767909+0000","last_undegraded":"2024-04-17T08:53:28.767909+0000","last_fullsized":"2024-04-17T08:53:28.767909+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:06:19.372923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962633+0000","last_change":"2024-04-17T08:53:27.752287+0000","last_active":"2024-04-17T08:53:28.962633+0000","last_peered":"2024-04-17T08:53:28.962633+0000","last_clean":"2024-04-17T08:53:28.962633+0000","last_became_active":"2024-04-17T08:53:27.751988+0000","last_became_peered":"2024-04-17T08:53:27.751988+0000","last_unstale":"2024-04-17T08:53:28.962633+0000","last_undegraded":"2024-04-17T08:53:28.962633+0000","last_fullsized":"2024-04-17T08:53:28.962633+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:35.119011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733445+0000","last_change":"2024-04-17T08:53:27.754990+0000","last_active":"2024-04-17T08:53:28.733445+0000","last_peered":"2024-04-17T08:53:28.733445+0000","last_clean":"2024-04-17T08:53:28.733445+0000","last_became_active":"2024-04-17T08:53:27.754636+0000","last_became_peered":"2024-04-17T08:53:27.754636+0000","last_unstale":"2024-04-17T08:53:28.733445+0000","last_undegraded":"2024-04-17T08:53:28.733445+0000","last_fullsized":"2024-04-17T08:53:28.733445+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:00:09.165919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161307+0000","last_change":"2024-04-17T08:53:27.761755+0000","last_active":"2024-04-17T08:53:29.161307+0000","last_peered":"2024-04-17T08:53:29.161307+0000","last_clean":"2024-04-17T08:53:29.161307+0000","last_became_active":"2024-04-17T08:53:27.757829+0000","last_became_peered":"2024-04-17T08:53:27.757829+0000","last_unstale":"2024-04-17T08:53:29.161307+0000","last_undegraded":"2024-04-17T08:53:29.161307+0000","last_fullsized":"2024-04-17T08:53:29.161307+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:19.407703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767433+0000","last_change":"2024-04-17T08:53:27.752976+0000","last_active":"2024-04-17T08:53:28.767433+0000","last_peered":"2024-04-17T08:53:28.767433+0000","last_clean":"2024-04-17T08:53:28.767433+0000","last_became_active":"2024-04-17T08:53:27.752670+0000","last_became_peered":"2024-04-17T08:53:27.752670+0000","last_unstale":"2024-04-17T08:53:28.767433+0000","last_undegraded":"2024-04-17T08:53:28.767433+0000","last_fullsized":"2024-04-17T08:53:28.767433+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:15:00.163187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963707+0000","last_change":"2024-04-17T08:53:27.767409+0000","last_active":"2024-04-17T08:53:28.963707+0000","last_peered":"2024-04-17T08:53:28.963707+0000","last_clean":"2024-04-17T08:53:28.963707+0000","last_became_active":"2024-04-17T08:53:27.767094+0000","last_became_peered":"2024-04-17T08:53:27.767094+0000","last_unstale":"2024-04-17T08:53:28.963707+0000","last_undegraded":"2024-04-17T08:53:28.963707+0000","last_fullsized":"2024-04-17T08:53:28.963707+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:20.212152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963314+0000","last_change":"2024-04-17T08:53:27.755357+0000","last_active":"2024-04-17T08:53:28.963314+0000","last_peered":"2024-04-17T08:53:28.963314+0000","last_clean":"2024-04-17T08:53:28.963314+0000","last_became_active":"2024-04-17T08:53:27.755055+0000","last_became_peered":"2024-04-17T08:53:27.755055+0000","last_unstale":"2024-04-17T08:53:28.963314+0000","last_undegraded":"2024-04-17T08:53:28.963314+0000","last_fullsized":"2024-04-17T08:53:28.963314+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:43.164794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734096+0000","last_change":"2024-04-17T08:53:27.735746+0000","last_active":"2024-04-17T08:53:28.734096+0000","last_peered":"2024-04-17T08:53:28.734096+0000","last_clean":"2024-04-17T08:53:28.734096+0000","last_became_active":"2024-04-17T08:53:27.734958+0000","last_became_peered":"2024-04-17T08:53:27.734958+0000","last_unstale":"2024-04-17T08:53:28.734096+0000","last_undegraded":"2024-04-17T08:53:28.734096+0000","last_fullsized":"2024-04-17T08:53:28.734096+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:33:46.207937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.728041+0000","last_change":"2024-04-17T08:53:27.768242+0000","last_active":"2024-04-17T08:53:28.728041+0000","last_peered":"2024-04-17T08:53:28.728041+0000","last_clean":"2024-04-17T08:53:28.728041+0000","last_became_active":"2024-04-17T08:53:27.767817+0000","last_became_peered":"2024-04-17T08:53:27.767817+0000","last_unstale":"2024-04-17T08:53:28.728041+0000","last_undegraded":"2024-04-17T08:53:28.728041+0000","last_fullsized":"2024-04-17T08:53:28.728041+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:45.468247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764179+0000","last_change":"2024-04-17T08:53:27.762550+0000","last_active":"2024-04-17T08:53:28.764179+0000","last_peered":"2024-04-17T08:53:28.764179+0000","last_clean":"2024-04-17T08:53:28.764179+0000","last_became_active":"2024-04-17T08:53:27.762273+0000","last_became_peered":"2024-04-17T08:53:27.762273+0000","last_unstale":"2024-04-17T08:53:28.764179+0000","last_undegraded":"2024-04-17T08:53:28.764179+0000","last_fullsized":"2024-04-17T08:53:28.764179+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:07.576063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160591+0000","last_change":"2024-04-17T08:53:27.757654+0000","last_active":"2024-04-17T08:53:29.160591+0000","last_peered":"2024-04-17T08:53:29.160591+0000","last_clean":"2024-04-17T08:53:29.160591+0000","last_became_active":"2024-04-17T08:53:27.756517+0000","last_became_peered":"2024-04-17T08:53:27.756517+0000","last_unstale":"2024-04-17T08:53:29.160591+0000","last_undegraded":"2024-04-17T08:53:29.160591+0000","last_fullsized":"2024-04-17T08:53:29.160591+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:52.281394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962678+0000","last_change":"2024-04-17T08:53:27.751778+0000","last_active":"2024-04-17T08:53:28.962678+0000","last_peered":"2024-04-17T08:53:28.962678+0000","last_clean":"2024-04-17T08:53:28.962678+0000","last_became_active":"2024-04-17T08:53:27.751488+0000","last_became_peered":"2024-04-17T08:53:27.751488+0000","last_unstale":"2024-04-17T08:53:28.962678+0000","last_undegraded":"2024-04-17T08:53:28.962678+0000","last_fullsized":"2024-04-17T08:53:28.962678+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:47.254725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733082+0000","last_change":"2024-04-17T08:53:27.760846+0000","last_active":"2024-04-17T08:53:28.733082+0000","last_peered":"2024-04-17T08:53:28.733082+0000","last_clean":"2024-04-17T08:53:28.733082+0000","last_became_active":"2024-04-17T08:53:27.760568+0000","last_became_peered":"2024-04-17T08:53:27.760568+0000","last_unstale":"2024-04-17T08:53:28.733082+0000","last_undegraded":"2024-04-17T08:53:28.733082+0000","last_fullsized":"2024-04-17T08:53:28.733082+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:01.214990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963718+0000","last_change":"2024-04-17T08:53:27.763257+0000","last_active":"2024-04-17T08:53:28.963718+0000","last_peered":"2024-04-17T08:53:28.963718+0000","last_clean":"2024-04-17T08:53:28.963718+0000","last_became_active":"2024-04-17T08:53:27.762591+0000","last_became_peered":"2024-04-17T08:53:27.762591+0000","last_unstale":"2024-04-17T08:53:28.963718+0000","last_undegraded":"2024-04-17T08:53:28.963718+0000","last_fullsized":"2024-04-17T08:53:28.963718+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:50.559615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733226+0000","last_change":"2024-04-17T08:53:27.739805+0000","last_active":"2024-04-17T08:53:28.733226+0000","last_peered":"2024-04-17T08:53:28.733226+0000","last_clean":"2024-04-17T08:53:28.733226+0000","last_became_active":"2024-04-17T08:53:27.739454+0000","last_became_peered":"2024-04-17T08:53:27.739454+0000","last_unstale":"2024-04-17T08:53:28.733226+0000","last_undegraded":"2024-04-17T08:53:28.733226+0000","last_fullsized":"2024-04-17T08:53:28.733226+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:38:27.744675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767985+0000","last_change":"2024-04-17T08:53:27.761924+0000","last_active":"2024-04-17T08:53:28.767985+0000","last_peered":"2024-04-17T08:53:28.767985+0000","last_clean":"2024-04-17T08:53:28.767985+0000","last_became_active":"2024-04-17T08:53:27.761646+0000","last_became_peered":"2024-04-17T08:53:27.761646+0000","last_unstale":"2024-04-17T08:53:28.767985+0000","last_undegraded":"2024-04-17T08:53:28.767985+0000","last_fullsized":"2024-04-17T08:53:28.767985+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:37:02.681319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050988+0000","last_change":"2024-04-17T08:53:27.753405+0000","last_active":"2024-04-17T08:53:29.050988+0000","last_peered":"2024-04-17T08:53:29.050988+0000","last_clean":"2024-04-17T08:53:29.050988+0000","last_became_active":"2024-04-17T08:53:27.753037+0000","last_became_peered":"2024-04-17T08:53:27.753037+0000","last_unstale":"2024-04-17T08:53:29.050988+0000","last_undegraded":"2024-04-17T08:53:29.050988+0000","last_fullsized":"2024-04-17T08:53:29.050988+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:06.905009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963406+0000","last_change":"2024-04-17T08:53:27.762072+0000","last_active":"2024-04-17T08:53:28.963406+0000","last_peered":"2024-04-17T08:53:28.963406+0000","last_clean":"2024-04-17T08:53:28.963406+0000","last_became_active":"2024-04-17T08:53:27.761767+0000","last_became_peered":"2024-04-17T08:53:27.761767+0000","last_unstale":"2024-04-17T08:53:28.963406+0000","last_undegraded":"2024-04-17T08:53:28.963406+0000","last_fullsized":"2024-04-17T08:53:28.963406+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:01:32.684604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734026+0000","last_change":"2024-04-17T08:53:27.754362+0000","last_active":"2024-04-17T08:53:28.734026+0000","last_peered":"2024-04-17T08:53:28.734026+0000","last_clean":"2024-04-17T08:53:28.734026+0000","last_became_active":"2024-04-17T08:53:27.754037+0000","last_became_peered":"2024-04-17T08:53:27.754037+0000","last_unstale":"2024-04-17T08:53:28.734026+0000","last_undegraded":"2024-04-17T08:53:28.734026+0000","last_fullsized":"2024-04-17T08:53:28.734026+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:48:42.185007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160183+0000","last_change":"2024-04-17T08:53:27.763986+0000","last_active":"2024-04-17T08:53:29.160183+0000","last_peered":"2024-04-17T08:53:29.160183+0000","last_clean":"2024-04-17T08:53:29.160183+0000","last_became_active":"2024-04-17T08:53:27.761675+0000","last_became_peered":"2024-04-17T08:53:27.761675+0000","last_unstale":"2024-04-17T08:53:29.160183+0000","last_undegraded":"2024-04-17T08:53:29.160183+0000","last_fullsized":"2024-04-17T08:53:29.160183+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:17.321293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767991+0000","last_change":"2024-04-17T08:53:27.760269+0000","last_active":"2024-04-17T08:53:28.767991+0000","last_peered":"2024-04-17T08:53:28.767991+0000","last_clean":"2024-04-17T08:53:28.767991+0000","last_became_active":"2024-04-17T08:53:27.757069+0000","last_became_peered":"2024-04-17T08:53:27.757069+0000","last_unstale":"2024-04-17T08:53:28.767991+0000","last_undegraded":"2024-04-17T08:53:28.767991+0000","last_fullsized":"2024-04-17T08:53:28.767991+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:44.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"285'4","reported_seq":18,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734418+0000","last_change":"2024-04-17T08:53:26.743874+0000","last_active":"2024-04-17T08:53:28.734418+0000","last_peered":"2024-04-17T08:53:28.734418+0000","last_clean":"2024-04-17T08:53:28.734418+0000","last_became_active":"2024-04-17T08:53:26.740497+0000","last_became_peered":"2024-04-17T08:53:26.740497+0000","last_unstale":"2024-04-17T08:53:28.734418+0000","last_undegraded":"2024-04-17T08:53:28.734418+0000","last_fullsized":"2024-04-17T08:53:28.734418+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:34:08.891894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733938+0000","last_change":"2024-04-17T08:53:27.751391+0000","last_active":"2024-04-17T08:53:28.733938+0000","last_peered":"2024-04-17T08:53:28.733938+0000","last_clean":"2024-04-17T08:53:28.733938+0000","last_became_active":"2024-04-17T08:53:27.750872+0000","last_became_peered":"2024-04-17T08:53:27.750872+0000","last_unstale":"2024-04-17T08:53:28.733938+0000","last_undegraded":"2024-04-17T08:53:28.733938+0000","last_fullsized":"2024-04-17T08:53:28.733938+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:03.490011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051407+0000","last_change":"2024-04-17T08:53:26.743260+0000","last_active":"2024-04-17T08:53:29.051407+0000","last_peered":"2024-04-17T08:53:29.051407+0000","last_clean":"2024-04-17T08:53:29.051407+0000","last_became_active":"2024-04-17T08:53:26.742962+0000","last_became_peered":"2024-04-17T08:53:26.742962+0000","last_unstale":"2024-04-17T08:53:29.051407+0000","last_undegraded":"2024-04-17T08:53:29.051407+0000","last_fullsized":"2024-04-17T08:53:29.051407+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:03:57.495011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051102+0000","last_change":"2024-04-17T08:53:27.757214+0000","last_active":"2024-04-17T08:53:29.051102+0000","last_peered":"2024-04-17T08:53:29.051102+0000","last_clean":"2024-04-17T08:53:29.051102+0000","last_became_active":"2024-04-17T08:53:27.755914+0000","last_became_peered":"2024-04-17T08:53:27.755914+0000","last_unstale":"2024-04-17T08:53:29.051102+0000","last_undegraded":"2024-04-17T08:53:29.051102+0000","last_fullsized":"2024-04-17T08:53:29.051102+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:38:06.802840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050963+0000","last_change":"2024-04-17T08:53:26.751887+0000","last_active":"2024-04-17T08:53:29.050963+0000","last_peered":"2024-04-17T08:53:29.050963+0000","last_clean":"2024-04-17T08:53:29.050963+0000","last_became_active":"2024-04-17T08:53:26.751580+0000","last_became_peered":"2024-04-17T08:53:26.751580+0000","last_unstale":"2024-04-17T08:53:29.050963+0000","last_undegraded":"2024-04-17T08:53:29.050963+0000","last_fullsized":"2024-04-17T08:53:29.050963+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:58.199283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764429+0000","last_change":"2024-04-17T08:53:27.753505+0000","last_active":"2024-04-17T08:53:28.764429+0000","last_peered":"2024-04-17T08:53:28.764429+0000","last_clean":"2024-04-17T08:53:28.764429+0000","last_became_active":"2024-04-17T08:53:27.753069+0000","last_became_peered":"2024-04-17T08:53:27.753069+0000","last_unstale":"2024-04-17T08:53:28.764429+0000","last_undegraded":"2024-04-17T08:53:28.764429+0000","last_fullsized":"2024-04-17T08:53:28.764429+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:53:15.997208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161887+0000","last_change":"2024-04-17T08:53:26.749774+0000","last_active":"2024-04-17T08:53:29.161887+0000","last_peered":"2024-04-17T08:53:29.161887+0000","last_clean":"2024-04-17T08:53:29.161887+0000","last_became_active":"2024-04-17T08:53:26.747101+0000","last_became_peered":"2024-04-17T08:53:26.747101+0000","last_unstale":"2024-04-17T08:53:29.161887+0000","last_undegraded":"2024-04-17T08:53:29.161887+0000","last_fullsized":"2024-04-17T08:53:29.161887+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:58:24.544210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764055+0000","last_change":"2024-04-17T08:53:27.771261+0000","last_active":"2024-04-17T08:53:28.764055+0000","last_peered":"2024-04-17T08:53:28.764055+0000","last_clean":"2024-04-17T08:53:28.764055+0000","last_became_active":"2024-04-17T08:53:27.768177+0000","last_became_peered":"2024-04-17T08:53:27.768177+0000","last_unstale":"2024-04-17T08:53:28.764055+0000","last_undegraded":"2024-04-17T08:53:28.764055+0000","last_fullsized":"2024-04-17T08:53:28.764055+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:44:17.413361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767627+0000","last_change":"2024-04-17T08:53:26.745660+0000","last_active":"2024-04-17T08:53:28.767627+0000","last_peered":"2024-04-17T08:53:28.767627+0000","last_clean":"2024-04-17T08:53:28.767627+0000","last_became_active":"2024-04-17T08:53:26.744798+0000","last_became_peered":"2024-04-17T08:53:26.744798+0000","last_unstale":"2024-04-17T08:53:28.767627+0000","last_undegraded":"2024-04-17T08:53:28.767627+0000","last_fullsized":"2024-04-17T08:53:28.767627+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:26.474714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.726956+0000","last_change":"2024-04-17T08:53:27.771755+0000","last_active":"2024-04-17T08:53:28.726956+0000","last_peered":"2024-04-17T08:53:28.726956+0000","last_clean":"2024-04-17T08:53:28.726956+0000","last_became_active":"2024-04-17T08:53:27.771483+0000","last_became_peered":"2024-04-17T08:53:27.771483+0000","last_unstale":"2024-04-17T08:53:28.726956+0000","last_undegraded":"2024-04-17T08:53:28.726956+0000","last_fullsized":"2024-04-17T08:53:28.726956+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:44.194308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963364+0000","last_change":"2024-04-17T08:53:26.747648+0000","last_active":"2024-04-17T08:53:28.963364+0000","last_peered":"2024-04-17T08:53:28.963364+0000","last_clean":"2024-04-17T08:53:28.963364+0000","last_became_active":"2024-04-17T08:53:26.746769+0000","last_became_peered":"2024-04-17T08:53:26.746769+0000","last_unstale":"2024-04-17T08:53:28.963364+0000","last_undegraded":"2024-04-17T08:53:28.963364+0000","last_fullsized":"2024-04-17T08:53:28.963364+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:58:14.007388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962571+0000","last_change":"2024-04-17T08:53:27.750595+0000","last_active":"2024-04-17T08:53:28.962571+0000","last_peered":"2024-04-17T08:53:28.962571+0000","last_clean":"2024-04-17T08:53:28.962571+0000","last_became_active":"2024-04-17T08:53:27.750239+0000","last_became_peered":"2024-04-17T08:53:27.750239+0000","last_unstale":"2024-04-17T08:53:28.962571+0000","last_undegraded":"2024-04-17T08:53:28.962571+0000","last_fullsized":"2024-04-17T08:53:28.962571+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:47.233876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767424+0000","last_change":"2024-04-17T08:53:26.751295+0000","last_active":"2024-04-17T08:53:28.767424+0000","last_peered":"2024-04-17T08:53:28.767424+0000","last_clean":"2024-04-17T08:53:28.767424+0000","last_became_active":"2024-04-17T08:53:26.750802+0000","last_became_peered":"2024-04-17T08:53:26.750802+0000","last_unstale":"2024-04-17T08:53:28.767424+0000","last_undegraded":"2024-04-17T08:53:28.767424+0000","last_fullsized":"2024-04-17T08:53:28.767424+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:08.200001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050389+0000","last_change":"2024-04-17T08:53:27.761622+0000","last_active":"2024-04-17T08:53:29.050389+0000","last_peered":"2024-04-17T08:53:29.050389+0000","last_clean":"2024-04-17T08:53:29.050389+0000","last_became_active":"2024-04-17T08:53:27.760836+0000","last_became_peered":"2024-04-17T08:53:27.760836+0000","last_unstale":"2024-04-17T08:53:29.050389+0000","last_undegraded":"2024-04-17T08:53:29.050389+0000","last_fullsized":"2024-04-17T08:53:29.050389+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:04.659149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.049928+0000","last_change":"2024-04-17T08:53:26.748546+0000","last_active":"2024-04-17T08:53:29.049928+0000","last_peered":"2024-04-17T08:53:29.049928+0000","last_clean":"2024-04-17T08:53:29.049928+0000","last_became_active":"2024-04-17T08:53:26.748238+0000","last_became_peered":"2024-04-17T08:53:26.748238+0000","last_unstale":"2024-04-17T08:53:29.049928+0000","last_undegraded":"2024-04-17T08:53:29.049928+0000","last_fullsized":"2024-04-17T08:53:29.049928+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:19.360257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051533+0000","last_change":"2024-04-17T08:53:27.740625+0000","last_active":"2024-04-17T08:53:29.051533+0000","last_peered":"2024-04-17T08:53:29.051533+0000","last_clean":"2024-04-17T08:53:29.051533+0000","last_became_active":"2024-04-17T08:53:27.739857+0000","last_became_peered":"2024-04-17T08:53:27.739857+0000","last_unstale":"2024-04-17T08:53:29.051533+0000","last_undegraded":"2024-04-17T08:53:29.051533+0000","last_fullsized":"2024-04-17T08:53:29.051533+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:30.165464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734696+0000","last_change":"2024-04-17T08:53:26.743930+0000","last_active":"2024-04-17T08:53:28.734696+0000","last_peered":"2024-04-17T08:53:28.734696+0000","last_clean":"2024-04-17T08:53:28.734696+0000","last_became_active":"2024-04-17T08:53:26.743566+0000","last_became_peered":"2024-04-17T08:53:26.743566+0000","last_unstale":"2024-04-17T08:53:28.734696+0000","last_undegraded":"2024-04-17T08:53:28.734696+0000","last_fullsized":"2024-04-17T08:53:28.734696+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:41:19.426548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734400+0000","last_change":"2024-04-17T08:53:27.737808+0000","last_active":"2024-04-17T08:53:28.734400+0000","last_peered":"2024-04-17T08:53:28.734400+0000","last_clean":"2024-04-17T08:53:28.734400+0000","last_became_active":"2024-04-17T08:53:27.735367+0000","last_became_peered":"2024-04-17T08:53:27.735367+0000","last_unstale":"2024-04-17T08:53:28.734400+0000","last_undegraded":"2024-04-17T08:53:28.734400+0000","last_fullsized":"2024-04-17T08:53:28.734400+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:42:47.233876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733463+0000","last_change":"2024-04-17T08:53:26.743084+0000","last_active":"2024-04-17T08:53:28.733463+0000","last_peered":"2024-04-17T08:53:28.733463+0000","last_clean":"2024-04-17T08:53:28.733463+0000","last_became_active":"2024-04-17T08:53:26.742793+0000","last_became_peered":"2024-04-17T08:53:26.742793+0000","last_unstale":"2024-04-17T08:53:28.733463+0000","last_undegraded":"2024-04-17T08:53:28.733463+0000","last_fullsized":"2024-04-17T08:53:28.733463+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:08.721488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161736+0000","last_change":"2024-04-17T08:53:27.764587+0000","last_active":"2024-04-17T08:53:29.161736+0000","last_peered":"2024-04-17T08:53:29.161736+0000","last_clean":"2024-04-17T08:53:29.161736+0000","last_became_active":"2024-04-17T08:53:27.763767+0000","last_became_peered":"2024-04-17T08:53:27.763767+0000","last_unstale":"2024-04-17T08:53:29.161736+0000","last_undegraded":"2024-04-17T08:53:29.161736+0000","last_fullsized":"2024-04-17T08:53:29.161736+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:11:26.459401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"285'3","reported_seq":17,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161260+0000","last_change":"2024-04-17T08:53:26.747359+0000","last_active":"2024-04-17T08:53:29.161260+0000","last_peered":"2024-04-17T08:53:29.161260+0000","last_clean":"2024-04-17T08:53:29.161260+0000","last_became_active":"2024-04-17T08:53:26.746728+0000","last_became_peered":"2024-04-17T08:53:26.746728+0000","last_unstale":"2024-04-17T08:53:29.161260+0000","last_undegraded":"2024-04-17T08:53:29.161260+0000","last_fullsized":"2024-04-17T08:53:29.161260+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:18:19.585571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050925+0000","last_change":"2024-04-17T08:53:27.743702+0000","last_active":"2024-04-17T08:53:29.050925+0000","last_peered":"2024-04-17T08:53:29.050925+0000","last_clean":"2024-04-17T08:53:29.050925+0000","last_became_active":"2024-04-17T08:53:27.743012+0000","last_became_peered":"2024-04-17T08:53:27.743012+0000","last_unstale":"2024-04-17T08:53:29.050925+0000","last_undegraded":"2024-04-17T08:53:29.050925+0000","last_fullsized":"2024-04-17T08:53:29.050925+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:52.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"285'6","reported_seq":20,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.962902+0000","last_change":"2024-04-17T08:53:26.747673+0000","last_active":"2024-04-17T08:53:28.962902+0000","last_peered":"2024-04-17T08:53:28.962902+0000","last_clean":"2024-04-17T08:53:28.962902+0000","last_became_active":"2024-04-17T08:53:26.747314+0000","last_became_peered":"2024-04-17T08:53:26.747314+0000","last_unstale":"2024-04-17T08:53:28.962902+0000","last_undegraded":"2024-04-17T08:53:28.962902+0000","last_fullsized":"2024-04-17T08:53:28.962902+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:46.638519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733441+0000","last_change":"2024-04-17T08:53:27.738946+0000","last_active":"2024-04-17T08:53:28.733441+0000","last_peered":"2024-04-17T08:53:28.733441+0000","last_clean":"2024-04-17T08:53:28.733441+0000","last_became_active":"2024-04-17T08:53:27.735949+0000","last_became_peered":"2024-04-17T08:53:27.735949+0000","last_unstale":"2024-04-17T08:53:28.733441+0000","last_undegraded":"2024-04-17T08:53:28.733441+0000","last_fullsized":"2024-04-17T08:53:28.733441+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:38:18.815195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764084+0000","last_change":"2024-04-17T08:53:26.742464+0000","last_active":"2024-04-17T08:53:28.764084+0000","last_peered":"2024-04-17T08:53:28.764084+0000","last_clean":"2024-04-17T08:53:28.764084+0000","last_became_active":"2024-04-17T08:53:26.742089+0000","last_became_peered":"2024-04-17T08:53:26.742089+0000","last_unstale":"2024-04-17T08:53:28.764084+0000","last_undegraded":"2024-04-17T08:53:28.764084+0000","last_fullsized":"2024-04-17T08:53:28.764084+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:09:00.803362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733394+0000","last_change":"2024-04-17T08:53:27.755458+0000","last_active":"2024-04-17T08:53:28.733394+0000","last_peered":"2024-04-17T08:53:28.733394+0000","last_clean":"2024-04-17T08:53:28.733394+0000","last_became_active":"2024-04-17T08:53:27.755173+0000","last_became_peered":"2024-04-17T08:53:27.755173+0000","last_unstale":"2024-04-17T08:53:28.733394+0000","last_undegraded":"2024-04-17T08:53:28.733394+0000","last_fullsized":"2024-04-17T08:53:28.733394+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:22.628804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160521+0000","last_change":"2024-04-17T08:53:26.746821+0000","last_active":"2024-04-17T08:53:29.160521+0000","last_peered":"2024-04-17T08:53:29.160521+0000","last_clean":"2024-04-17T08:53:29.160521+0000","last_became_active":"2024-04-17T08:53:26.746486+0000","last_became_peered":"2024-04-17T08:53:26.746486+0000","last_unstale":"2024-04-17T08:53:29.160521+0000","last_undegraded":"2024-04-17T08:53:29.160521+0000","last_fullsized":"2024-04-17T08:53:29.160521+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:53:15.686242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767389+0000","last_change":"2024-04-17T08:53:27.767572+0000","last_active":"2024-04-17T08:53:28.767389+0000","last_peered":"2024-04-17T08:53:28.767389+0000","last_clean":"2024-04-17T08:53:28.767389+0000","last_became_active":"2024-04-17T08:53:27.767304+0000","last_became_peered":"2024-04-17T08:53:27.767304+0000","last_unstale":"2024-04-17T08:53:28.767389+0000","last_undegraded":"2024-04-17T08:53:28.767389+0000","last_fullsized":"2024-04-17T08:53:28.767389+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:58:24.086844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963112+0000","last_change":"2024-04-17T08:53:26.751868+0000","last_active":"2024-04-17T08:53:28.963112+0000","last_peered":"2024-04-17T08:53:28.963112+0000","last_clean":"2024-04-17T08:53:28.963112+0000","last_became_active":"2024-04-17T08:53:26.751504+0000","last_became_peered":"2024-04-17T08:53:26.751504+0000","last_unstale":"2024-04-17T08:53:28.963112+0000","last_undegraded":"2024-04-17T08:53:28.963112+0000","last_fullsized":"2024-04-17T08:53:28.963112+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:31:49.324708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.731326+0000","last_change":"2024-04-17T08:53:27.734856+0000","last_active":"2024-04-17T08:53:28.731326+0000","last_peered":"2024-04-17T08:53:28.731326+0000","last_clean":"2024-04-17T08:53:28.731326+0000","last_became_active":"2024-04-17T08:53:27.734211+0000","last_became_peered":"2024-04-17T08:53:27.734211+0000","last_unstale":"2024-04-17T08:53:28.731326+0000","last_undegraded":"2024-04-17T08:53:28.731326+0000","last_fullsized":"2024-04-17T08:53:28.731326+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:15:28.278067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"285'8","reported_seq":22,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160350+0000","last_change":"2024-04-17T08:53:26.751597+0000","last_active":"2024-04-17T08:53:29.160350+0000","last_peered":"2024-04-17T08:53:29.160350+0000","last_clean":"2024-04-17T08:53:29.160350+0000","last_became_active":"2024-04-17T08:53:26.751259+0000","last_became_peered":"2024-04-17T08:53:26.751259+0000","last_unstale":"2024-04-17T08:53:29.160350+0000","last_undegraded":"2024-04-17T08:53:29.160350+0000","last_fullsized":"2024-04-17T08:53:29.160350+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:35:47.079433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161628+0000","last_change":"2024-04-17T08:53:27.759892+0000","last_active":"2024-04-17T08:53:29.161628+0000","last_peered":"2024-04-17T08:53:29.161628+0000","last_clean":"2024-04-17T08:53:29.161628+0000","last_became_active":"2024-04-17T08:53:27.759613+0000","last_became_peered":"2024-04-17T08:53:27.759613+0000","last_unstale":"2024-04-17T08:53:29.161628+0000","last_undegraded":"2024-04-17T08:53:29.161628+0000","last_fullsized":"2024-04-17T08:53:29.161628+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:09.350160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764496+0000","last_change":"2024-04-17T08:53:26.733363+0000","last_active":"2024-04-17T08:53:28.764496+0000","last_peered":"2024-04-17T08:53:28.764496+0000","last_clean":"2024-04-17T08:53:28.764496+0000","last_became_active":"2024-04-17T08:53:26.732933+0000","last_became_peered":"2024-04-17T08:53:26.732933+0000","last_unstale":"2024-04-17T08:53:28.764496+0000","last_undegraded":"2024-04-17T08:53:28.764496+0000","last_fullsized":"2024-04-17T08:53:28.764496+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:53:04.414260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.768301+0000","last_change":"2024-04-17T08:53:27.760742+0000","last_active":"2024-04-17T08:53:28.768301+0000","last_peered":"2024-04-17T08:53:28.768301+0000","last_clean":"2024-04-17T08:53:28.768301+0000","last_became_active":"2024-04-17T08:53:27.759388+0000","last_became_peered":"2024-04-17T08:53:27.759388+0000","last_unstale":"2024-04-17T08:53:28.768301+0000","last_undegraded":"2024-04-17T08:53:28.768301+0000","last_fullsized":"2024-04-17T08:53:28.768301+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:06.580148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733833+0000","last_change":"2024-04-17T08:53:26.741308+0000","last_active":"2024-04-17T08:53:28.733833+0000","last_peered":"2024-04-17T08:53:28.733833+0000","last_clean":"2024-04-17T08:53:28.733833+0000","last_became_active":"2024-04-17T08:53:26.740903+0000","last_became_peered":"2024-04-17T08:53:26.740903+0000","last_unstale":"2024-04-17T08:53:28.733833+0000","last_undegraded":"2024-04-17T08:53:28.733833+0000","last_fullsized":"2024-04-17T08:53:28.733833+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:37:45.272229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051449+0000","last_change":"2024-04-17T08:53:27.766005+0000","last_active":"2024-04-17T08:53:29.051449+0000","last_peered":"2024-04-17T08:53:29.051449+0000","last_clean":"2024-04-17T08:53:29.051449+0000","last_became_active":"2024-04-17T08:53:27.765741+0000","last_became_peered":"2024-04-17T08:53:27.765741+0000","last_unstale":"2024-04-17T08:53:29.051449+0000","last_undegraded":"2024-04-17T08:53:29.051449+0000","last_fullsized":"2024-04-17T08:53:29.051449+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:31:58.805099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"285'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051261+0000","last_change":"2024-04-17T08:53:26.744543+0000","last_active":"2024-04-17T08:53:29.051261+0000","last_peered":"2024-04-17T08:53:29.051261+0000","last_clean":"2024-04-17T08:53:29.051261+0000","last_became_active":"2024-04-17T08:53:26.741353+0000","last_became_peered":"2024-04-17T08:53:26.741353+0000","last_unstale":"2024-04-17T08:53:29.051261+0000","last_undegraded":"2024-04-17T08:53:29.051261+0000","last_fullsized":"2024-04-17T08:53:29.051261+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:30:51.108689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734681+0000","last_change":"2024-04-17T08:53:27.764495+0000","last_active":"2024-04-17T08:53:28.734681+0000","last_peered":"2024-04-17T08:53:28.734681+0000","last_clean":"2024-04-17T08:53:28.734681+0000","last_became_active":"2024-04-17T08:53:27.764119+0000","last_became_peered":"2024-04-17T08:53:27.764119+0000","last_unstale":"2024-04-17T08:53:28.734681+0000","last_undegraded":"2024-04-17T08:53:28.734681+0000","last_fullsized":"2024-04-17T08:53:28.734681+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:35.119011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.051348+0000","last_change":"2024-04-17T08:53:26.749738+0000","last_active":"2024-04-17T08:53:29.051348+0000","last_peered":"2024-04-17T08:53:29.051348+0000","last_clean":"2024-04-17T08:53:29.051348+0000","last_became_active":"2024-04-17T08:53:26.749434+0000","last_became_peered":"2024-04-17T08:53:26.749434+0000","last_unstale":"2024-04-17T08:53:29.051348+0000","last_undegraded":"2024-04-17T08:53:29.051348+0000","last_fullsized":"2024-04-17T08:53:29.051348+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:41:16.942239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767624+0000","last_change":"2024-04-17T08:53:27.759602+0000","last_active":"2024-04-17T08:53:28.767624+0000","last_peered":"2024-04-17T08:53:28.767624+0000","last_clean":"2024-04-17T08:53:28.767624+0000","last_became_active":"2024-04-17T08:53:27.759246+0000","last_became_peered":"2024-04-17T08:53:27.759246+0000","last_unstale":"2024-04-17T08:53:28.767624+0000","last_undegraded":"2024-04-17T08:53:28.767624+0000","last_fullsized":"2024-04-17T08:53:28.767624+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:11.101157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160782+0000","last_change":"2024-04-17T08:53:26.750785+0000","last_active":"2024-04-17T08:53:29.160782+0000","last_peered":"2024-04-17T08:53:29.160782+0000","last_clean":"2024-04-17T08:53:29.160782+0000","last_became_active":"2024-04-17T08:53:26.746628+0000","last_became_peered":"2024-04-17T08:53:26.746628+0000","last_unstale":"2024-04-17T08:53:29.160782+0000","last_undegraded":"2024-04-17T08:53:29.160782+0000","last_fullsized":"2024-04-17T08:53:29.160782+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:36.492477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.727771+0000","last_change":"2024-04-17T08:53:27.759085+0000","last_active":"2024-04-17T08:53:28.727771+0000","last_peered":"2024-04-17T08:53:28.727771+0000","last_clean":"2024-04-17T08:53:28.727771+0000","last_became_active":"2024-04-17T08:53:27.758789+0000","last_became_peered":"2024-04-17T08:53:27.758789+0000","last_unstale":"2024-04-17T08:53:28.727771+0000","last_undegraded":"2024-04-17T08:53:28.727771+0000","last_fullsized":"2024-04-17T08:53:28.727771+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:49:08.507541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764070+0000","last_change":"2024-04-17T08:53:26.737860+0000","last_active":"2024-04-17T08:53:28.764070+0000","last_peered":"2024-04-17T08:53:28.764070+0000","last_clean":"2024-04-17T08:53:28.764070+0000","last_became_active":"2024-04-17T08:53:26.737488+0000","last_became_peered":"2024-04-17T08:53:26.737488+0000","last_unstale":"2024-04-17T08:53:28.764070+0000","last_undegraded":"2024-04-17T08:53:28.764070+0000","last_fullsized":"2024-04-17T08:53:28.764070+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:54.388994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763883+0000","last_change":"2024-04-17T08:53:27.758116+0000","last_active":"2024-04-17T08:53:28.763883+0000","last_peered":"2024-04-17T08:53:28.763883+0000","last_clean":"2024-04-17T08:53:28.763883+0000","last_became_active":"2024-04-17T08:53:27.757774+0000","last_became_peered":"2024-04-17T08:53:27.757774+0000","last_unstale":"2024-04-17T08:53:28.763883+0000","last_undegraded":"2024-04-17T08:53:28.763883+0000","last_fullsized":"2024-04-17T08:53:28.763883+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:54:52.088922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050375+0000","last_change":"2024-04-17T08:53:26.746525+0000","last_active":"2024-04-17T08:53:29.050375+0000","last_peered":"2024-04-17T08:53:29.050375+0000","last_clean":"2024-04-17T08:53:29.050375+0000","last_became_active":"2024-04-17T08:53:26.746243+0000","last_became_peered":"2024-04-17T08:53:26.746243+0000","last_unstale":"2024-04-17T08:53:29.050375+0000","last_undegraded":"2024-04-17T08:53:29.050375+0000","last_fullsized":"2024-04-17T08:53:29.050375+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:28:59.343774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.764322+0000","last_change":"2024-04-17T08:53:27.759138+0000","last_active":"2024-04-17T08:53:28.764322+0000","last_peered":"2024-04-17T08:53:28.764322+0000","last_clean":"2024-04-17T08:53:28.764322+0000","last_became_active":"2024-04-17T08:53:27.758840+0000","last_became_peered":"2024-04-17T08:53:27.758840+0000","last_unstale":"2024-04-17T08:53:28.764322+0000","last_undegraded":"2024-04-17T08:53:28.764322+0000","last_fullsized":"2024-04-17T08:53:28.764322+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:37.949351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763998+0000","last_change":"2024-04-17T08:53:26.738392+0000","last_active":"2024-04-17T08:53:28.763998+0000","last_peered":"2024-04-17T08:53:28.763998+0000","last_clean":"2024-04-17T08:53:28.763998+0000","last_became_active":"2024-04-17T08:53:26.738088+0000","last_became_peered":"2024-04-17T08:53:26.738088+0000","last_unstale":"2024-04-17T08:53:28.763998+0000","last_undegraded":"2024-04-17T08:53:28.763998+0000","last_fullsized":"2024-04-17T08:53:28.763998+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:00.572399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.728121+0000","last_change":"2024-04-17T08:53:27.765625+0000","last_active":"2024-04-17T08:53:28.728121+0000","last_peered":"2024-04-17T08:53:28.728121+0000","last_clean":"2024-04-17T08:53:28.728121+0000","last_became_active":"2024-04-17T08:53:27.765345+0000","last_became_peered":"2024-04-17T08:53:27.765345+0000","last_unstale":"2024-04-17T08:53:28.728121+0000","last_undegraded":"2024-04-17T08:53:28.728121+0000","last_fullsized":"2024-04-17T08:53:28.728121+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T09:57:23.225419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"285'2","reported_seq":16,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161826+0000","last_change":"2024-04-17T08:53:26.746062+0000","last_active":"2024-04-17T08:53:29.161826+0000","last_peered":"2024-04-17T08:53:29.161826+0000","last_clean":"2024-04-17T08:53:29.161826+0000","last_became_active":"2024-04-17T08:53:26.745779+0000","last_became_peered":"2024-04-17T08:53:26.745779+0000","last_unstale":"2024-04-17T08:53:29.161826+0000","last_undegraded":"2024-04-17T08:53:29.161826+0000","last_fullsized":"2024-04-17T08:53:29.161826+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:35.197781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.963920+0000","last_change":"2024-04-17T08:53:27.762182+0000","last_active":"2024-04-17T08:53:28.963920+0000","last_peered":"2024-04-17T08:53:28.963920+0000","last_clean":"2024-04-17T08:53:28.963920+0000","last_became_active":"2024-04-17T08:53:27.761799+0000","last_became_peered":"2024-04-17T08:53:27.761799+0000","last_unstale":"2024-04-17T08:53:28.963920+0000","last_undegraded":"2024-04-17T08:53:28.963920+0000","last_fullsized":"2024-04-17T08:53:28.963920+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:18.619031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"286'1","reported_seq":15,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.766518+0000","last_change":"2024-04-17T08:53:26.749321+0000","last_active":"2024-04-17T08:53:28.766518+0000","last_peered":"2024-04-17T08:53:28.766518+0000","last_clean":"2024-04-17T08:53:28.766518+0000","last_became_active":"2024-04-17T08:53:26.748989+0000","last_became_peered":"2024-04-17T08:53:26.748989+0000","last_unstale":"2024-04-17T08:53:28.766518+0000","last_undegraded":"2024-04-17T08:53:28.766518+0000","last_fullsized":"2024-04-17T08:53:28.766518+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:06:22.673592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.733395+0000","last_change":"2024-04-17T08:53:27.765517+0000","last_active":"2024-04-17T08:53:28.733395+0000","last_peered":"2024-04-17T08:53:28.733395+0000","last_clean":"2024-04-17T08:53:28.733395+0000","last_became_active":"2024-04-17T08:53:27.765133+0000","last_became_peered":"2024-04-17T08:53:27.765133+0000","last_unstale":"2024-04-17T08:53:28.733395+0000","last_undegraded":"2024-04-17T08:53:28.733395+0000","last_fullsized":"2024-04-17T08:53:28.733395+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:29:01.909810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161390+0000","last_change":"2024-04-17T08:53:26.750892+0000","last_active":"2024-04-17T08:53:29.161390+0000","last_peered":"2024-04-17T08:53:29.161390+0000","last_clean":"2024-04-17T08:53:29.161390+0000","last_became_active":"2024-04-17T08:53:26.747783+0000","last_became_peered":"2024-04-17T08:53:26.747783+0000","last_unstale":"2024-04-17T08:53:29.161390+0000","last_undegraded":"2024-04-17T08:53:29.161390+0000","last_fullsized":"2024-04-17T08:53:29.161390+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:45:18.301387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.732666+0000","last_change":"2024-04-17T08:53:27.747949+0000","last_active":"2024-04-17T08:53:28.732666+0000","last_peered":"2024-04-17T08:53:28.732666+0000","last_clean":"2024-04-17T08:53:28.732666+0000","last_became_active":"2024-04-17T08:53:27.747593+0000","last_became_peered":"2024-04-17T08:53:27.747593+0000","last_unstale":"2024-04-17T08:53:28.732666+0000","last_undegraded":"2024-04-17T08:53:28.732666+0000","last_fullsized":"2024-04-17T08:53:28.732666+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:10.441752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767958+0000","last_change":"2024-04-17T08:53:26.744431+0000","last_active":"2024-04-17T08:53:28.767958+0000","last_peered":"2024-04-17T08:53:28.767958+0000","last_clean":"2024-04-17T08:53:28.767958+0000","last_became_active":"2024-04-17T08:53:26.744114+0000","last_became_peered":"2024-04-17T08:53:26.744114+0000","last_unstale":"2024-04-17T08:53:28.767958+0000","last_undegraded":"2024-04-17T08:53:28.767958+0000","last_fullsized":"2024-04-17T08:53:28.767958+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:26.954146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.050979+0000","last_change":"2024-04-17T08:53:27.755114+0000","last_active":"2024-04-17T08:53:29.050979+0000","last_peered":"2024-04-17T08:53:29.050979+0000","last_clean":"2024-04-17T08:53:29.050979+0000","last_became_active":"2024-04-17T08:53:27.754780+0000","last_became_peered":"2024-04-17T08:53:27.754780+0000","last_unstale":"2024-04-17T08:53:29.050979+0000","last_undegraded":"2024-04-17T08:53:29.050979+0000","last_fullsized":"2024-04-17T08:53:29.050979+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:13:58.116875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767780+0000","last_change":"2024-04-17T08:53:26.745411+0000","last_active":"2024-04-17T08:53:28.767780+0000","last_peered":"2024-04-17T08:53:28.767780+0000","last_clean":"2024-04-17T08:53:28.767780+0000","last_became_active":"2024-04-17T08:53:26.744057+0000","last_became_peered":"2024-04-17T08:53:26.744057+0000","last_unstale":"2024-04-17T08:53:28.767780+0000","last_undegraded":"2024-04-17T08:53:28.767780+0000","last_fullsized":"2024-04-17T08:53:28.767780+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:00:27.500568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160658+0000","last_change":"2024-04-17T08:53:27.754343+0000","last_active":"2024-04-17T08:53:29.160658+0000","last_peered":"2024-04-17T08:53:29.160658+0000","last_clean":"2024-04-17T08:53:29.160658+0000","last_became_active":"2024-04-17T08:53:27.753840+0000","last_became_peered":"2024-04-17T08:53:27.753840+0000","last_unstale":"2024-04-17T08:53:29.160658+0000","last_undegraded":"2024-04-17T08:53:29.160658+0000","last_fullsized":"2024-04-17T08:53:29.160658+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:36.160059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160717+0000","last_change":"2024-04-17T08:53:26.750396+0000","last_active":"2024-04-17T08:53:29.160717+0000","last_peered":"2024-04-17T08:53:29.160717+0000","last_clean":"2024-04-17T08:53:29.160717+0000","last_became_active":"2024-04-17T08:53:26.750083+0000","last_became_peered":"2024-04-17T08:53:26.750083+0000","last_unstale":"2024-04-17T08:53:29.160717+0000","last_undegraded":"2024-04-17T08:53:29.160717+0000","last_fullsized":"2024-04-17T08:53:29.160717+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:02.113591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.763882+0000","last_change":"2024-04-17T08:53:27.761040+0000","last_active":"2024-04-17T08:53:28.763882+0000","last_peered":"2024-04-17T08:53:28.763882+0000","last_clean":"2024-04-17T08:53:28.763882+0000","last_became_active":"2024-04-17T08:53:27.760475+0000","last_became_peered":"2024-04-17T08:53:27.760475+0000","last_unstale":"2024-04-17T08:53:28.763882+0000","last_undegraded":"2024-04-17T08:53:28.763882+0000","last_fullsized":"2024-04-17T08:53:28.763882+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:20:37.569340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.734110+0000","last_change":"2024-04-17T08:53:26.744129+0000","last_active":"2024-04-17T08:53:28.734110+0000","last_peered":"2024-04-17T08:53:28.734110+0000","last_clean":"2024-04-17T08:53:28.734110+0000","last_became_active":"2024-04-17T08:53:26.743810+0000","last_became_peered":"2024-04-17T08:53:26.743810+0000","last_unstale":"2024-04-17T08:53:28.734110+0000","last_undegraded":"2024-04-17T08:53:28.734110+0000","last_fullsized":"2024-04-17T08:53:28.734110+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:28.088593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.731485+0000","last_change":"2024-04-17T08:53:27.740704+0000","last_active":"2024-04-17T08:53:28.731485+0000","last_peered":"2024-04-17T08:53:28.731485+0000","last_clean":"2024-04-17T08:53:28.731485+0000","last_became_active":"2024-04-17T08:53:27.739818+0000","last_became_peered":"2024-04-17T08:53:27.739818+0000","last_unstale":"2024-04-17T08:53:28.731485+0000","last_undegraded":"2024-04-17T08:53:28.731485+0000","last_fullsized":"2024-04-17T08:53:28.731485+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:59.716934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.160318+0000","last_change":"2024-04-17T08:53:26.751325+0000","last_active":"2024-04-17T08:53:29.160318+0000","last_peered":"2024-04-17T08:53:29.160318+0000","last_clean":"2024-04-17T08:53:29.160318+0000","last_became_active":"2024-04-17T08:53:26.750739+0000","last_became_peered":"2024-04-17T08:53:26.750739+0000","last_unstale":"2024-04-17T08:53:29.160318+0000","last_undegraded":"2024-04-17T08:53:29.160318+0000","last_fullsized":"2024-04-17T08:53:29.160318+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:25.680186+0000","last_clean_scrub_stamp":"2024-04-17T08:53:25.680186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:19.069226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":12,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:29.161417+0000","last_change":"2024-04-17T08:53:27.765857+0000","last_active":"2024-04-17T08:53:29.161417+0000","last_peered":"2024-04-17T08:53:29.161417+0000","last_clean":"2024-04-17T08:53:29.161417+0000","last_became_active":"2024-04-17T08:53:27.763911+0000","last_became_peered":"2024-04-17T08:53:27.763911+0000","last_unstale":"2024-04-17T08:53:29.161417+0000","last_undegraded":"2024-04-17T08:53:29.161417+0000","last_fullsized":"2024-04-17T08:53:29.161417+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T08:53:26.697866+0000","last_clean_scrub_stamp":"2024-04-17T08:53:26.697866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:21.006238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"274'2771","reported_seq":4143,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-17T08:53:28.767467+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:53:28.767467+0000","last_peered":"2024-04-17T08:53:28.767467+0000","last_clean":"2024-04-17T08:53:28.767467+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:53:28.767467+0000","last_undegraded":"2024-04-17T08:53:28.767467+0000","last_fullsized":"2024-04-17T08:53:28.767467+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2697,"num_read_kb":5658,"num_write":689,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2771,"ondisk_log_size":2771,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543325,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53332,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318508,"statfs":{"total":96636764160,"available":96582152192,"internally_reserved":0,"allocated":4202496,"data_stored":2371303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543325,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61080,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310760,"statfs":{"total":96636764160,"available":96574218240,"internally_reserved":0,"allocated":3608576,"data_stored":1776219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543324,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52772,"kb_used_data":3540,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319068,"statfs":{"total":96636764160,"available":96582725632,"internally_reserved":0,"allocated":3624960,"data_stored":1782637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543324,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84636,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287204,"statfs":{"total":96636764160,"available":96550096896,"internally_reserved":0,"allocated":3616768,"data_stored":1781587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543326,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52756,"kb_used_data":3528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319084,"statfs":{"total":96636764160,"available":96582742016,"internally_reserved":0,"allocated":3612672,"data_stored":1781517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543325,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":3616768,"data_stored":1781059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543326,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48600,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323240,"statfs":{"total":96636764160,"available":96586997760,"internally_reserved":0,"allocated":3616768,"data_stored":1780991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543324,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57500,"kb_used_data":4112,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314340,"statfs":{"total":96636764160,"available":96577884160,"internally_reserved":0,"allocated":4210688,"data_stored":2371395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:53:32.756 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T08:53:32.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T08:53:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.029+0000 7fd0a3e00640 1 Processor -- start 2024-04-17T08:53:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.029+0000 7fd0a3e00640 1 -- start start 2024-04-17T08:53:33.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 0x7fd0a40ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0a40b20b0 0x7fd0a40b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a40b29f0 con 0x7fd0a415c580 2024-04-17T08:53:33.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd0a40b3770 con 0x7fd0a40b20b0 2024-04-17T08:53:33.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a415f050 con 0x7fd0a415c1a0 2024-04-17T08:53:33.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0a415c580 0x7fd0a40af470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58328/0 (socket says 172.21.15.67:58328) 2024-04-17T08:53:33.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a2000640 1 -- 172.21.15.67:0/584897299 learned_addr learned my addr 172.21.15.67:0/584897299 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:33.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 0x7fd0a40ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.031+0000 7fd0a3400640 1 --2- 172.21.15.67:0/584897299 >> v2:172.21.15.187:3300/0 conn(0x7fd0a40b20b0 0x7fd0a40b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491954777 0 0) 0x7fd0a40b29f0 con 0x7fd0a415c580 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd088003520 con 0x7fd0a415c580 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2151809764 0 0) 0x7fd088003520 con 0x7fd0a415c580 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 msgr2=0x7fd0a40ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 0x7fd0a40ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 >> v2:172.21.15.187:3300/0 conn(0x7fd0a40b20b0 msgr2=0x7fd0a40b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 --2- 172.21.15.67:0/584897299 >> v2:172.21.15.187:3300/0 conn(0x7fd0a40b20b0 0x7fd0a40b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a415f330 con 0x7fd0a415c580 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd08c001670 con 0x7fd0a415c580 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 >> v1:172.21.15.67:6789/0 conn(0x7fd0a415c580 legacy=0x7fd0a40af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0880039b0 0x7fd088003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 0x7fd0880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40b3770 con 0x7fd0880039b0 2024-04-17T08:53:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a415f050 con 0x7fd088004800 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd08800b9c0 con 0x7fd0a415c1a0 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.032+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0880039b0 0x7fd088003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.033+0000 7fd0a2a00640 1 -- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 msgr2=0x7fd0880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.033+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0a415c1a0 0x7fd0880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.033+0000 7fd0a2a00640 1 -- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd088004800 msgr2=0x7fd088004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.033+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd088004800 0x7fd088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.033+0000 7fd0a2a00640 1 -- 172.21.15.67:0/584897299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd088003520 con 0x7fd0880039b0 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.034+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0880039b0 0x7fd088003dd0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fc0 tx=0x7fd088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51a2bc91dfc403d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.034+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094039070 con 0x7fd0880039b0 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.034+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09402cc60 con 0x7fd0880039b0 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.034+0000 7fd0a1600640 1 -- 172.21.15.67:0/584897299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094009c10 con 0x7fd0880039b0 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 -- 172.21.15.67:0/584897299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0880039b0 msgr2=0x7fd088003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/584897299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0880039b0 0x7fd088003dd0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fc0 tx=0x7fd088005830 comp rx=0 tx=0).stop 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3400640 1 -- 172.21.15.67:0/584897299 reap_dead start 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 -- 172.21.15.67:0/584897299 shutdown_connections 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 -- 172.21.15.67:0/584897299 >> 172.21.15.67:0/584897299 conn(0x7fd0a4067d10 msgr2=0x7fd0a40d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 -- 172.21.15.67:0/584897299 shutdown_connections 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.035+0000 7fd0a3e00640 1 -- 172.21.15.67:0/584897299 wait complete. 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.037+0000 7fd0a3e00640 1 Processor -- start 2024-04-17T08:53:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.037+0000 7fd0a3e00640 1 -- start start 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0880039b0 0x7fd0a4161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0a4158840 0x7fd0a4158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40afb80 con 0x7fd0a4150960 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a415f8f0 con 0x7fd0a4158840 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a40afd00 con 0x7fd0880039b0 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46750/0 (socket says 172.21.15.67:46750) 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.038+0000 7fd0a2000640 1 -- 172.21.15.67:0/3490310098 learned_addr learned my addr 172.21.15.67:0/3490310098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0880039b0 0x7fd0a4161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 -- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0880039b0 msgr2=0x7fd0a4161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0880039b0 0x7fd0a4161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 -- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0a4158840 msgr2=0x7fd0a4158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a3400640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0a4158840 0x7fd0a4158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0a4158840 0x7fd0a4158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 -- 172.21.15.67:0/3490310098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a40b0900 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a2000640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00c7f0 tx=0x7fd08c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c018070 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08c0074e0 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c004a00 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.039+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a40b0be0 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.040+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a40b11a0 con 0x7fd0a4150960 2024-04-17T08:53:33.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.041+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a4068a10 con 0x7fd0a4150960 2024-04-17T08:53:33.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.042+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7fd08c01b070 con 0x7fd0a4150960 2024-04-17T08:53:33.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.043+0000 7fd09aa00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 0x7fd08006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.043+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fd08c08b420 con 0x7fd0a4150960 2024-04-17T08:53:33.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.043+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 0x7fd08006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.045+0000 7fd0a2a00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 0x7fd08006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd094036000 tx=0x7fd09402e000 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.047+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd08c059680 con 0x7fd0a4150960 2024-04-17T08:53:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.322+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd0a406e5f0 con 0x7fd0a4150960 2024-04-17T08:53:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.326+0000 7fd09aa00640 1 -- 172.21.15.67:0/3490310098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) ==== 190+0+147 (secure 0 0 0) 0x7fd08c058f80 con 0x7fd0a4150960 2024-04-17T08:53:33.325 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T08:53:33.326 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB6dh9mLgE4IhAA6r3RLU38yi6y+U1NQqdEbw== 2024-04-17T08:53:33.326 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T08:53:33.326 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T08:53:33.326 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T08:53:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 msgr2=0x7fd08006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 0x7fd08006eed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd094036000 tx=0x7fd09402e000 comp rx=0 tx=0).stop 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 msgr2=0x7fd0a40aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00c7f0 tx=0x7fd08c00ccb0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 shutdown_connections 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fd08006ca10 0x7fd08006eed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0a4158840 0x7fd0a4158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0a4150960 0x7fd0a40aff90 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 --2- 172.21.15.67:0/3490310098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0880039b0 0x7fd0a4161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 >> 172.21.15.67:0/3490310098 conn(0x7fd0a4067d10 msgr2=0x7fd0a406edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 shutdown_connections 2024-04-17T08:53:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.329+0000 7fd0a3e00640 1 -- 172.21.15.67:0/3490310098 wait complete. 2024-04-17T08:53:33.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:53:33.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.565+0000 7f2b77200640 1 Processor -- start 2024-04-17T08:53:33.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.565+0000 7f2b77200640 1 -- start start 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.566+0000 7f2b77200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b7015ed10 0x7f2b700d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.566+0000 7f2b77200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b700d3020 con 0x7f2b700cf640 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.566+0000 7f2b77200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b700d3300 con 0x7f2b700d2ab0 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.566+0000 7f2b77200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2b700d3c30 con 0x7f2b7015ed10 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.567+0000 7f2b6fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b700cf640 0x7f2b700cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54402/0 (socket says 172.21.15.67:54402) 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.567+0000 7f2b6fe00640 1 -- 172.21.15.67:0/259771873 learned_addr learned my addr 172.21.15.67:0/259771873 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:33.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.567+0000 7f2b6f400640 1 --2- 172.21.15.67:0/259771873 >> v2:172.21.15.67:3300/0 conn(0x7f2b7015ed10 0x7f2b700d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.567+0000 7f2b6ea00640 1 -- 172.21.15.67:0/259771873 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1341907921 0 0) 0x7f2b700d3020 con 0x7f2b700cf640 2024-04-17T08:53:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.568+0000 7f2b6ea00640 1 -- 172.21.15.67:0/259771873 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b54003520 con 0x7f2b700cf640 2024-04-17T08:53:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.568+0000 7f2b6f400640 1 -- 172.21.15.67:0/259771873 >> v1:172.21.15.187:6789/0 conn(0x7f2b700d2ab0 legacy=0x7f2b700b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.568+0000 7f2b6f400640 1 -- 172.21.15.67:0/259771873 >> v1:172.21.15.67:6789/0 conn(0x7f2b700cf640 legacy=0x7f2b700cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.568+0000 7f2b6f400640 1 -- 172.21.15.67:0/259771873 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b700d35e0 con 0x7f2b7015ed10 2024-04-17T08:53:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.569+0000 7f2b6f400640 1 --2- 172.21.15.67:0/259771873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700d40c0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097a0 tx=0x7f2b5802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb415cdf33a2a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.569+0000 7f2b6ea00640 1 -- 172.21.15.67:0/259771873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5802f810 con 0x7f2b7015ed10 2024-04-17T08:53:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.569+0000 7f2b6ea00640 1 -- 172.21.15.67:0/259771873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5802f9b0 con 0x7f2b7015ed10 2024-04-17T08:53:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.569+0000 7f2b6ea00640 1 -- 172.21.15.67:0/259771873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5802fd80 con 0x7f2b7015ed10 2024-04-17T08:53:33.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 -- 172.21.15.67:0/259771873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 msgr2=0x7f2b700d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 --2- 172.21.15.67:0/259771873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700d40c0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097a0 tx=0x7f2b5802b9f0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 -- 172.21.15.67:0/259771873 shutdown_connections 2024-04-17T08:53:33.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 --2- 172.21.15.67:0/259771873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700d40c0 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 -- 172.21.15.67:0/259771873 >> 172.21.15.67:0/259771873 conn(0x7f2b7006cbd0 msgr2=0x7f2b7006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 -- 172.21.15.67:0/259771873 shutdown_connections 2024-04-17T08:53:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.570+0000 7f2b77200640 1 -- 172.21.15.67:0/259771873 wait complete. 2024-04-17T08:53:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.572+0000 7f2b77200640 1 Processor -- start 2024-04-17T08:53:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.572+0000 7f2b77200640 1 -- start start 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b700cf640 0x7f2b700b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 0x7f2b700cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700d4600 con 0x7f2b7015ed10 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700d25b0 con 0x7f2b700d2ab0 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b77200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2b700d4780 con 0x7f2b700cf640 2024-04-17T08:53:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b75200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b75200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60408/0 (socket says 172.21.15.67:60408) 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.573+0000 7f2b75200640 1 -- 172.21.15.67:0/2613268012 learned_addr learned my addr 172.21.15.67:0/2613268012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b6fe00640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b700cf640 0x7f2b700b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b6f400640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 0x7f2b700cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 -- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b700cf640 msgr2=0x7f2b700b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b700cf640 0x7f2b700b5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 -- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 msgr2=0x7f2b700cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 0x7f2b700cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 -- 172.21.15.67:0/2613268012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b700cc190 con 0x7f2b7015ed10 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b6f400640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 0x7f2b700cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b75200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800c7a0 tx=0x7f2b6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b68003b70 con 0x7f2b7015ed10 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b68003d30 con 0x7f2b7015ed10 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6800aa90 con 0x7f2b7015ed10 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.574+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b700cc470 con 0x7f2b7015ed10 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.575+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b700cca30 con 0x7f2b7015ed10 2024-04-17T08:53:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.576+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b70068b90 con 0x7f2b7015ed10 2024-04-17T08:53:33.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.577+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f2b6800ac30 con 0x7f2b7015ed10 2024-04-17T08:53:33.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.577+0000 7f2b67e00640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 0x7f2b4c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:33.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.578+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f2b6808bcb0 con 0x7f2b7015ed10 2024-04-17T08:53:33.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.578+0000 7f2b6fe00640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 0x7f2b4c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:33.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.579+0000 7f2b6fe00640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 0x7f2b4c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b600011e0 tx=0x7f2b60009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:33.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.582+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2b68059fa0 con 0x7f2b7015ed10 2024-04-17T08:53:33.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.827+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b700b3640 con 0x7f2b7015ed10 2024-04-17T08:53:33.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.828+0000 7f2b67e00640 1 -- 172.21.15.67:0/2613268012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7f2b680598a0 con 0x7f2b7015ed10 2024-04-17T08:53:33.827 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:53:33.828 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:53:33.828 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:53:33.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.830+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 msgr2=0x7f2b4c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.830+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 0x7f2b4c06eed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b600011e0 tx=0x7f2b60009040 comp rx=0 tx=0).stop 2024-04-17T08:53:33.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.830+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 msgr2=0x7f2b700cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:33.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.830+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800c7a0 tx=0x7f2b6800cc60 comp rx=0 tx=0).stop 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 shutdown_connections 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f2b4c06ca10 0x7f2b4c06eed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b7015ed10 0x7f2b700cba40 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b700d2ab0 0x7f2b700cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 --2- 172.21.15.67:0/2613268012 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2b700cf640 0x7f2b700b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 >> 172.21.15.67:0/2613268012 conn(0x7f2b7006cbd0 msgr2=0x7f2b700ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 shutdown_connections 2024-04-17T08:53:33.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:33.831+0000 7f2b77200640 1 -- 172.21.15.67:0/2613268012 wait complete. 2024-04-17T08:53:34.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 151, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T08:53:27.207090+0000', 'modified': '2024-04-17T08:53:30.274229+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'snap_rstat': True, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13062}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13062': {'gid': 13062, 'name': 'd', 'rank': 0, 'incarnation': 147, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6837/2174028812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6835', 'nonce': 2174028812}, {'type': 'v1', 'addr': '172.21.15.187:6837', 'nonce': 2174028812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 13062, 'qdb_cluster': [13062]} 2024-04-17T08:53:34.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T08:53:34.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T08:53:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.491+0000 7f7e1cc00640 1 Processor -- start 2024-04-17T08:53:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.491+0000 7f7e1cc00640 1 -- start start 2024-04-17T08:53:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e1cc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7e180cbc00 0x7f7e180cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e1cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 0x7f7e18071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e1cc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1815d060 con 0x7f7e180723a0 2024-04-17T08:53:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e1cc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7e180cfe90 con 0x7f7e180cbc00 2024-04-17T08:53:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e1cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1815dc70 con 0x7f7e180b63c0 2024-04-17T08:53:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e17e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e180723a0 0x7f7e18160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54406/0 (socket says 172.21.15.67:54406) 2024-04-17T08:53:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.493+0000 7f7e17e00640 1 -- 172.21.15.67:0/4171562143 learned_addr learned my addr 172.21.15.67:0/4171562143 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e17400640 1 --2- 172.21.15.67:0/4171562143 >> v2:172.21.15.187:3300/0 conn(0x7f7e180cbc00 0x7f7e180cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16a00640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 0x7f7e18071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946860695 0 0) 0x7f7e1815d060 con 0x7f7e180723a0 2024-04-17T08:53:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dfc003520 con 0x7f7e180723a0 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 14623457 0 0) 0x7f7dfc003520 con 0x7f7e180723a0 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 msgr2=0x7f7e18071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 0x7f7e18071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 >> v2:172.21.15.187:3300/0 conn(0x7f7e180cbc00 msgr2=0x7f7e180cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 --2- 172.21.15.67:0/4171562143 >> v2:172.21.15.187:3300/0 conn(0x7f7e180cbc00 0x7f7e180cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1815d340 con 0x7f7e180723a0 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f7e10001670 con 0x7f7e180723a0 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 >> v1:172.21.15.67:6789/0 conn(0x7f7e180723a0 legacy=0x7f7e18160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc003a20 0x7f7dfc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 0x7f7dfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dfc004d40 0x7f7dfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e180cfe90 con 0x7f7dfc003a20 2024-04-17T08:53:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1815dc70 con 0x7f7dfc004d40 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.494+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7dfc00b9c0 con 0x7f7e180b63c0 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc003a20 0x7f7dfc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 -- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 msgr2=0x7f7dfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7e180b63c0 0x7f7dfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 -- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dfc004d40 msgr2=0x7f7dfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dfc004d40 0x7f7dfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.495+0000 7f7e17400640 1 -- 172.21.15.67:0/4171562143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc003520 con 0x7f7dfc003a20 2024-04-17T08:53:34.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.496+0000 7f7e17400640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc003a20 0x7f7dfc003e40 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009fc0 tx=0x7f7dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9457ae57bf91d125 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08039070 con 0x7f7dfc003a20 2024-04-17T08:53:34.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0802cc60 con 0x7f7dfc003a20 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e16000640 1 -- 172.21.15.67:0/4171562143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08009c10 con 0x7f7dfc003a20 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 -- 172.21.15.67:0/4171562143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc003a20 msgr2=0x7f7dfc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/4171562143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc003a20 0x7f7dfc003e40 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009fc0 tx=0x7f7dfc005830 comp rx=0 tx=0).stop 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e17e00640 1 -- 172.21.15.67:0/4171562143 reap_dead start 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 -- 172.21.15.67:0/4171562143 shutdown_connections 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 -- 172.21.15.67:0/4171562143 >> 172.21.15.67:0/4171562143 conn(0x7f7e1806bef0 msgr2=0x7f7e180b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 -- 172.21.15.67:0/4171562143 shutdown_connections 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.497+0000 7f7e1cc00640 1 -- 172.21.15.67:0/4171562143 wait complete. 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.499+0000 7f7e1cc00640 1 Processor -- start 2024-04-17T08:53:34.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.499+0000 7f7e1cc00640 1 -- start start 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dfc003a20 0x7f7e18150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e1806e810 0x7f7e18158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e180d02d0 con 0x7f7e181559c0 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1815d900 con 0x7f7e1806e810 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e1cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e180cbfe0 con 0x7f7dfc003a20 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.500+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60436/0 (socket says 172.21.15.67:60436) 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 -- 172.21.15.67:0/2352991358 learned_addr learned my addr 172.21.15.67:0/2352991358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 -- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dfc003a20 msgr2=0x7f7e18150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dfc003a20 0x7f7e18150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 -- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e1806e810 msgr2=0x7f7e18158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e1806e810 0x7f7e18158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 -- 172.21.15.67:0/2352991358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e181567b0 con 0x7f7e181559c0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.501+0000 7f7e16a00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e820 tx=0x7f7e0400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.502+0000 7f7e0f400640 1 -- 172.21.15.67:0/2352991358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04017070 con 0x7f7e181559c0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.502+0000 7f7e0f400640 1 -- 172.21.15.67:0/2352991358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0400c940 con 0x7f7e181559c0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.502+0000 7f7e0f400640 1 -- 172.21.15.67:0/2352991358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0400cd10 con 0x7f7e181559c0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.502+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1806f3e0 con 0x7f7e181559c0 2024-04-17T08:53:34.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.502+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1806f9a0 con 0x7f7e181559c0 2024-04-17T08:53:34.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.504+0000 7f7e0f400640 1 -- 172.21.15.67:0/2352991358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f7e0401a070 con 0x7f7e181559c0 2024-04-17T08:53:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.505+0000 7f7e0f400640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 0x7f7df806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.505+0000 7f7e0f400640 1 -- 172.21.15.67:0/2352991358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f7e0408bbe0 con 0x7f7e181559c0 2024-04-17T08:53:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.506+0000 7f7e17400640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 0x7f7df806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.506+0000 7f7e20e48740 1 Processor -- start 2024-04-17T08:53:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.506+0000 7f7e20e48740 1 -- start start 2024-04-17T08:53:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e17400640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 0x7f7df806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e08036000 tx=0x7f7e0802e070 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f540e620 0x55e6f53c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f53c6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e6f532c980 con 0x55e6f53c0b40 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e6f52d3220 con 0x55e6f53c46a0 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.507+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e6f5328940 con 0x55e6f540e620 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60442/0 (socket says 172.21.15.67:60442) 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 -- 172.21.15.67:0/675187454 learned_addr learned my addr 172.21.15.67:0/675187454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17400640 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f540e620 0x55e6f53c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e16a00640 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f53c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 -- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f540e620 msgr2=0x55e6f53c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f540e620 0x55e6f53c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 -- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 msgr2=0x55e6f53c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f53c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.508+0000 7f7e17e00640 1 -- 172.21.15.67:0/675187454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f503f2d0 con 0x55e6f53c0b40 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.509+0000 7f7e17e00640 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f7e1806c720 tx=0x7f7e10007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d82899c34605be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.509+0000 7f7e03400640 1 -- 172.21.15.67:0/675187454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100074e0 con 0x55e6f53c0b40 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.509+0000 7f7e03400640 1 -- 172.21.15.67:0/675187454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e100047e0 con 0x55e6f53c0b40 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.509+0000 7f7e03400640 1 -- 172.21.15.67:0/675187454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10004bd0 con 0x55e6f53c0b40 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 -- 172.21.15.67:0/675187454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 msgr2=0x55e6f53f0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f7e1806c720 tx=0x7f7e10007ca0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 -- 172.21.15.67:0/675187454 shutdown_connections 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c0b40 0x55e6f53f0930 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f53c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 --2- 172.21.15.67:0/675187454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f540e620 0x55e6f53c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 -- 172.21.15.67:0/675187454 >> 172.21.15.67:0/675187454 conn(0x55e6f53c2db0 msgr2=0x55e6f540b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 -- 172.21.15.67:0/675187454 shutdown_connections 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.510+0000 7f7e20e48740 1 -- 172.21.15.67:0/675187454 wait complete. 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.511+0000 7f7e20e48740 1 -- start start 2024-04-17T08:53:34.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.511+0000 7f7e20e48740 1 Processor -- start 2024-04-17T08:53:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f5412ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f540d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e6f53c4bb0 con 0x55e6f540e620 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e6f52e6590 con 0x55e6f53c46a0 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e6f53c4e90 con 0x55e6f53e8190 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e20e48740 10 client.0 _ll_register_callbacks cb 0x55e6f503eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60454/0 (socket says 172.21.15.67:60454) 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.512+0000 7f7e16a00640 1 -- 172.21.15.67:0/3028071426 learned_addr learned my addr 172.21.15.67:0/3028071426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 -- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 msgr2=0x55e6f540d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f540d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 -- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 msgr2=0x55e6f5412ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f5412ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 -- 172.21.15.67:0/3028071426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f51096b0 con 0x55e6f540e620 2024-04-17T08:53:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e16a00640 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e7f0 tx=0x7f7e04059f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed46435b8b01f14 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7df7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04021040 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e100047e0 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f7e04058e20 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f7e040623b0 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e20e48740 10 client.13200 fetch_fsmap learned FSMap version 151 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.513+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=2+}) -- 0x55e6f54125a0 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e100047e0 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 151) ==== 3495+0+0 (secure 0 0 0) 0x7f7e04024060 con 0x55e6f540e620 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 10 client.13200 fetch_fsmap finished waiting for FSMap version 151 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 10 client.13200 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 20 client.13200 populate_metadata read hostname 'smithi067' 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f541cc40 0x55e6f542af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 4 client.13200 mds_command: new command op to 13062 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T08:53:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.514+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 --> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6f533a3a0 con 0x55e6f541cc40 2024-04-17T08:53:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.515+0000 7f7e17400640 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f541cc40 0x55e6f542af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.516+0000 7f7e17400640 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f541cc40 0x55e6f542af10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4af765ff44b64c34 server_cookie=a729f3d4c9e8f195 in_seq=0 out_seq=0 2024-04-17T08:53:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.516+0000 7f7df7e00640 10 client.13200 ms_handle_connect on v2:172.21.15.187:6835/2174028812 2024-04-17T08:53:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.518+0000 7f7df7e00640 1 -- 172.21.15.67:0/3028071426 <== mds.0 v2:172.21.15.187:6835/2174028812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6f533a3a0 con 0x55e6f541cc40 2024-04-17T08:53:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.518+0000 7f7df7e00640 10 client.13200 handle_command_reply: tid=0 2024-04-17T08:53:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 client.13200 shutdown 2024-04-17T08:53:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 msgr2=0x55e6f540cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e7f0 tx=0x7f7e04059f90 comp rx=0 tx=0).stop 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 shutdown_connections 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f541cc40 0x55e6f542af10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f540d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f540cba0 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 --2- 172.21.15.67:0/3028071426 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53c46a0 0x55e6f5412ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 >> 172.21.15.67:0/3028071426 conn(0x55e6f53c0b40 msgr2=0x55e6f53c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7df7e00640 0 client.13200 ms_handle_reset on v2:172.21.15.187:6835/2174028812 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 shutdown_connections 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 1 -- 172.21.15.67:0/3028071426 wait complete. 2024-04-17T08:53:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.519+0000 7f7e20e48740 20 client.13200 trim_cache size 0 max 16384 2024-04-17T08:53:34.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.539+0000 7f7e20e48740 1 Processor -- start 2024-04-17T08:53:34.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.539+0000 7f7e20e48740 1 -- start start 2024-04-17T08:53:34.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f5430260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f540e620 0x55e6f54307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e181f5230 con 0x55e6f53c46a0 2024-04-17T08:53:34.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7e181f4590 con 0x55e6f540e620 2024-04-17T08:53:34.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.540+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7e181f4be0 con 0x55e6f53e8190 2024-04-17T08:53:34.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60462/0 (socket says 172.21.15.67:60462) 2024-04-17T08:53:34.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 -- 172.21.15.67:0/296833530 learned_addr learned my addr 172.21.15.67:0/296833530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e16a00640 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f5430260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17e00640 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f540e620 0x55e6f54307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 -- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 msgr2=0x55e6f5430260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f5430260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 -- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f540e620 msgr2=0x55e6f54307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f540e620 0x55e6f54307a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.541+0000 7f7e17400640 1 -- 172.21.15.67:0/296833530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f53c2510 con 0x55e6f53c46a0 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.542+0000 7f7e17400640 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007120 tx=0x7f7e08030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd69afcdd58a257 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.542+0000 7f7e02a00640 1 -- 172.21.15.67:0/296833530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08054020 con 0x55e6f53c46a0 2024-04-17T08:53:34.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.542+0000 7f7e02a00640 1 -- 172.21.15.67:0/296833530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0803a040 con 0x55e6f53c46a0 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.542+0000 7f7e02a00640 1 -- 172.21.15.67:0/296833530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08034e40 con 0x55e6f53c46a0 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 -- 172.21.15.67:0/296833530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 msgr2=0x55e6f54136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f7e08007120 tx=0x7f7e08030000 comp rx=0 tx=0).stop 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 -- 172.21.15.67:0/296833530 shutdown_connections 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f540e620 0x55e6f54307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53e8190 0x55e6f5430260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 --2- 172.21.15.67:0/296833530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f53c46a0 0x55e6f54136c0 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 -- 172.21.15.67:0/296833530 >> 172.21.15.67:0/296833530 conn(0x55e6f53c0b40 msgr2=0x55e6f53c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 -- 172.21.15.67:0/296833530 shutdown_connections 2024-04-17T08:53:34.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.543+0000 7f7e20e48740 1 -- 172.21.15.67:0/296833530 wait complete. 2024-04-17T08:53:34.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.544+0000 7f7e20e48740 1 -- start start 2024-04-17T08:53:34.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.544+0000 7f7e20e48740 1 Processor -- start 2024-04-17T08:53:34.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53c46a0 0x55e6f546f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53e8190 0x55e6f5410cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e6f541cac0 con 0x55e6f540e620 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55e6f53ee870 con 0x55e6f53e8190 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55e6f5410000 con 0x55e6f53c46a0 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e20e48740 10 client.0 _ll_register_callbacks cb 0x7f7e04055a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60472/0 (socket says 172.21.15.67:60472) 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.545+0000 7f7e17e00640 1 -- 172.21.15.67:0/635467150 learned_addr learned my addr 172.21.15.67:0/635467150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 -- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53c46a0 msgr2=0x55e6f546f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53c46a0 0x55e6f546f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 -- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53e8190 msgr2=0x55e6f5410cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53e8190 0x55e6f5410cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 -- 172.21.15.67:0/635467150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f5430900 con 0x55e6f540e620 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e17e00640 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7e1815cf90 tx=0x7f7e04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b9ca3c2e336fc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7df7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10017030 con 0x55e6f540e620 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f7e100049f0 con 0x55e6f540e620 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0803a040 con 0x55e6f540e620 2024-04-17T08:53:34.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) ==== 24+0+0 (secure 0 0 0) 0x7f7e10004bf0 con 0x55e6f540e620 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e20e48740 10 client.13206 fetch_fsmap learned FSMap version 151 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.546+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=2+}) -- 0x55e6f546fcb0 con 0x55e6f540e620 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0803a040 con 0x55e6f540e620 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 151) ==== 3495+0+0 (secure 0 0 0) 0x7f7e10029070 con 0x55e6f540e620 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 10 client.13206 fetch_fsmap finished waiting for FSMap version 151 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 10 client.13206 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 20 client.13206 populate_metadata read hostname 'smithi067' 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f540f450 0x55e6f542ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 4 client.13206 mds_command: new command op to 13062 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.547+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 --> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] -- command(tid 0: {"prefix": "status"}) -- 0x55e6f5334210 con 0x55e6f540f450 2024-04-17T08:53:34.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.548+0000 7f7e17400640 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f540f450 0x55e6f542ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.549+0000 7f7e17400640 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f540f450 0x55e6f542ef30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db203d9b56dbd2b5 server_cookie=6077ff1b4a3c3a00 in_seq=0 out_seq=0 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.549+0000 7f7df7e00640 10 client.13206 ms_handle_connect on v2:172.21.15.187:6835/2174028812 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7df7e00640 1 -- 172.21.15.67:0/635467150 <== mds.0 v2:172.21.15.187:6835/2174028812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55e6f5334210 con 0x55e6f540f450 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7df7e00640 10 client.13206 handle_command_reply: tid=0 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 client.13206 shutdown 2024-04-17T08:53:34.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 msgr2=0x55e6f54111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7e1815cf90 tx=0x7f7e04004730 comp rx=0 tx=0).stop 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 shutdown_connections 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:6835/2174028812,v1:172.21.15.187:6837/2174028812] conn(0x55e6f540f450 0x55e6f542ef30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e6f540e620 0x55e6f54111f0 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55e6f53e8190 0x55e6f5410cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 --2- 172.21.15.67:0/635467150 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55e6f53c46a0 0x55e6f546f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7df7e00640 0 client.13206 ms_handle_reset on v2:172.21.15.187:6835/2174028812 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 >> 172.21.15.67:0/635467150 conn(0x55e6f53c0b40 msgr2=0x55e6f53c1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 shutdown_connections 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.550+0000 7f7e20e48740 1 -- 172.21.15.67:0/635467150 wait complete. 2024-04-17T08:53:34.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e20e48740 20 client.13206 trim_cache size 0 max 16384 2024-04-17T08:53:34.554 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:53:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "c9dd6e72-8918-4f7c-8224-a4ea40cec0ec", 2024-04-17T08:53:34.554 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T08:53:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 13062, 2024-04-17T08:53:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T08:53:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T08:53:34.558 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.8069491690000001, 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 151, 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 286, 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 286, 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.192732861 2024-04-17T08:53:34.559 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 msgr2=0x7f7df806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 0x7f7df806eed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e08036000 tx=0x7f7e0802e070 comp rx=0 tx=0).stop 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 msgr2=0x7f7e18156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e820 tx=0x7f7e0400ece0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 shutdown_connections 2024-04-17T08:53:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f7df806ca10 0x7f7df806eed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e1806e810 0x7f7e18158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e181559c0 0x7f7e18156270 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 --2- 172.21.15.67:0/2352991358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7dfc003a20 0x7f7e18150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 >> 172.21.15.67:0/2352991358 conn(0x7f7e1806bef0 msgr2=0x7f7e180cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 shutdown_connections 2024-04-17T08:53:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.551+0000 7f7e1cc00640 1 -- 172.21.15.67:0/2352991358 wait complete. 2024-04-17T08:53:34.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T08:53:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.793+0000 7f6dac200640 1 Processor -- start 2024-04-17T08:53:34.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.793+0000 7f6dac200640 1 -- start start 2024-04-17T08:53:34.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6dac200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da415ed10 0x7f6da40d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6dac200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6da40d2ab0 0x7f6da40b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6dac200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da40d3020 con 0x7f6da40cf640 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6dac200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6da40d3c30 con 0x7f6da40d2ab0 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6dac200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40d3300 con 0x7f6da415ed10 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6daa200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6da40d2ab0 0x7f6da40b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6daa200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6da40d2ab0 0x7f6da40b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60484/0 (socket says 172.21.15.67:60484) 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.795+0000 7f6daa200640 1 -- 172.21.15.67:0/1942767668 learned_addr learned my addr 172.21.15.67:0/1942767668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.796+0000 7f6daa200640 1 -- 172.21.15.67:0/1942767668 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da415ed10 msgr2=0x7f6da40d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.796+0000 7f6daa200640 1 --2- 172.21.15.67:0/1942767668 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da415ed10 0x7f6da40d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.796+0000 7f6daa200640 1 -- 172.21.15.67:0/1942767668 >> v1:172.21.15.187:6789/0 conn(0x7f6da40cf640 legacy=0x7f6da40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.796+0000 7f6daa200640 1 -- 172.21.15.67:0/1942767668 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da40d35e0 con 0x7f6da40d2ab0 2024-04-17T08:53:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.797+0000 7f6daa200640 1 --2- 172.21.15.67:0/1942767668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da40d2ab0 0x7f6da40b7b10 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009e50 tx=0x7f6d9c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7338f990f62f1559 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.797+0000 7f6da3e00640 1 -- 172.21.15.67:0/1942767668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c02cb00 con 0x7f6da40d2ab0 2024-04-17T08:53:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.797+0000 7f6da3e00640 1 -- 172.21.15.67:0/1942767668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9c02cca0 con 0x7f6da40d2ab0 2024-04-17T08:53:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.797+0000 7f6da3e00640 1 -- 172.21.15.67:0/1942767668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c034610 con 0x7f6da40d2ab0 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 -- 172.21.15.67:0/1942767668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da40d2ab0 msgr2=0x7f6da40b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 --2- 172.21.15.67:0/1942767668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da40d2ab0 0x7f6da40b7b10 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009e50 tx=0x7f6d9c02bf60 comp rx=0 tx=0).stop 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 -- 172.21.15.67:0/1942767668 shutdown_connections 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 --2- 172.21.15.67:0/1942767668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da40d2ab0 0x7f6da40b7b10 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 --2- 172.21.15.67:0/1942767668 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da415ed10 0x7f6da40d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 -- 172.21.15.67:0/1942767668 >> 172.21.15.67:0/1942767668 conn(0x7f6da406cbd0 msgr2=0x7f6da406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 -- 172.21.15.67:0/1942767668 shutdown_connections 2024-04-17T08:53:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.798+0000 7f6dac200640 1 -- 172.21.15.67:0/1942767668 wait complete. 2024-04-17T08:53:34.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.800+0000 7f6dac200640 1 Processor -- start 2024-04-17T08:53:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.800+0000 7f6dac200640 1 -- start start 2024-04-17T08:53:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.801+0000 7f6dac200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da40cf640 0x7f6da40b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6dac200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da40d2ab0 0x7f6da40b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6dac200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6dac200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40d4600 con 0x7f6da415ed10 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6dac200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40d25b0 con 0x7f6da40cf640 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6dac200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40d4780 con 0x7f6da40d2ab0 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6daa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6daa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60490/0 (socket says 172.21.15.67:60490) 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6daa200640 1 -- 172.21.15.67:0/371901485 learned_addr learned my addr 172.21.15.67:0/371901485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6da8e00640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da40d2ab0 0x7f6da40b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.802+0000 7f6da9800640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da40cf640 0x7f6da40b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 -- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da40d2ab0 msgr2=0x7f6da40b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da40d2ab0 0x7f6da40b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 -- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da40cf640 msgr2=0x7f6da40b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da40cf640 0x7f6da40b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 -- 172.21.15.67:0/371901485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41592b0 con 0x7f6da415ed10 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.803+0000 7f6daa200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009e50 tx=0x7f6d9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.804+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c043020 con 0x7f6da415ed10 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.804+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9c02ce50 con 0x7f6da415ed10 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.804+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c03e670 con 0x7f6da415ed10 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.804+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da40b8510 con 0x7f6da415ed10 2024-04-17T08:53:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.804+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da40b8ad0 con 0x7f6da415ed10 2024-04-17T08:53:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.806+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da40b92a0 con 0x7f6da415ed10 2024-04-17T08:53:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.807+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f6d9c046020 con 0x7f6da415ed10 2024-04-17T08:53:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.808+0000 7f6da1600640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 0x7f6d8406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.808+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f6d9c0ae1d0 con 0x7f6da415ed10 2024-04-17T08:53:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.808+0000 7f6da9800640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 0x7f6d8406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:34.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.810+0000 7f6da9800640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 0x7f6d8406ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6d98004ea0 tx=0x7f6d98009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:34.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:34.816+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6d9c04d070 con 0x7f6da415ed10 2024-04-17T08:53:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.065+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6da4068b90 con 0x7f6da415ed10 2024-04-17T08:53:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.065+0000 7f6da1600640 1 -- 172.21.15.67:0/371901485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v151) ==== 53+0+73 (secure 0 0 0) 0x7f6da41592b0 con 0x7f6da415ed10 2024-04-17T08:53:35.065 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T08:53:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 msgr2=0x7f6d8406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 0x7f6d8406ef00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6d98004ea0 tx=0x7f6d98009040 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 msgr2=0x7f6da414bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009e50 tx=0x7f6d9c004a40 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 shutdown_connections 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f6d8406ca40 0x7f6d8406ef00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da415ed10 0x7f6da414bdf0 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6da40d2ab0 0x7f6da40b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 --2- 172.21.15.67:0/371901485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6da40cf640 0x7f6da40b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 >> 172.21.15.67:0/371901485 conn(0x7f6da406cbd0 msgr2=0x7f6da4160930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 shutdown_connections 2024-04-17T08:53:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35.069+0000 7f6dac200640 1 -- 172.21.15.67:0/371901485 wait complete. 2024-04-17T08:53:35.089 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T08:53:35.090 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:53:35.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.091 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T08:53:35.145 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T08:53:35.145 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T08:53:35.146 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 36629sec preferred_lft 36629sec 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::878:11ff:fe7e:aac4/64 scope link 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ba:6a:78:10:30:bd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b86a:78ff:fe10:30bd/64 scope link 2024-04-17T08:53:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.148 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T08:53:35.148 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d6:7e:f2:2d:2c:f9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:53:35.148 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d47e:f2ff:fe2d:2cf9/64 scope link 2024-04-17T08:53:35.148 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T08:53:35.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.150 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T08:53:35.244 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T08:53:35.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.248 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T08:53:35.303 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T08:53:35.304 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T08:53:35.304 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T08:53:35.304 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T08:53:35.304 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T08:53:35.304 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T08:53:35.304 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T08:53:35.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T08:53:35.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T08:53:35.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T08:53:35.305 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T08:53:35.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T08:53:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:53:35.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.567 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:53:35.618 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:53:35.618 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:53:35.618 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:53:35.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T08:53:35.721 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:53:35.721 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T08:53:35.781 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T08:53:35.781 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T08:53:35.781 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:53:35.833 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T08:53:35.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:35.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:53:35.973 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:53:35.974 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:53:36.010 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T08:53:36.010 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T08:53:36.010 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-17T08:53:36.011 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@c9dd6e72-8918-4f7c-8224-a4ea40cec0ec.cephfs=/ 2024-04-17T08:53:36.011 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-17T08:53:36.013 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-17T08:53:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-17T08:53:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-17T08:53:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-17T08:53:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-17T08:53:36.019 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T08:53:36.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:53:36.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T08:53:36.117 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T08:53:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T08:53:36.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T08:53:36.269 INFO:teuthology.orchestra.run.smithi067.stdout:client8858 2024-04-17T08:53:36.269 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:53:36.269 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:53:36.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-04-17T08:53:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.552+0000 7f0cdac00640 1 Processor -- start 2024-04-17T08:53:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.552+0000 7f0cdac00640 1 -- start start 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.553+0000 7f0cdac00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0cd40b2b40 0x7f0cd415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.553+0000 7f0cdac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd415d390 con 0x7f0cd406c2a0 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.553+0000 7f0cdac00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd415d670 con 0x7f0cd415a450 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.553+0000 7f0cdac00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0cd415a830 con 0x7f0cd40b2b40 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.554+0000 7f0cda200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cd406c2a0 0x7f0cd40cd710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54412/0 (socket says 172.21.15.67:54412) 2024-04-17T08:53:36.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.554+0000 7f0cda200640 1 -- 172.21.15.67:0/790329000 learned_addr learned my addr 172.21.15.67:0/790329000 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.554+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615374564 0 0) 0x7f0cd415d390 con 0x7f0cd406c2a0 2024-04-17T08:53:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.554+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/790329000 >> v2:172.21.15.187:3300/0 conn(0x7f0cd40b2b40 0x7f0cd415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.554+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cbc003520 con 0x7f0cd406c2a0 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 911162871 0 0) 0x7f0cbc003520 con 0x7f0cd406c2a0 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 >> v2:172.21.15.187:3300/0 conn(0x7f0cd40b2b40 msgr2=0x7f0cd415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 --2- 172.21.15.67:0/790329000 >> v2:172.21.15.187:3300/0 conn(0x7f0cd40b2b40 0x7f0cd415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 >> v1:172.21.15.187:6789/0 conn(0x7f0cd415a450 legacy=0x7f0cd40b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd415d950 con 0x7f0cd406c2a0 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0ccc0019f0 con 0x7f0cd406c2a0 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 >> v1:172.21.15.67:6789/0 conn(0x7f0cd406c2a0 legacy=0x7f0cd40cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cbc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cbc005210 0x7f0cbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd415a830 con 0x7f0cbc003a70 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc00c140 con 0x7f0cbc004800 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc00c340 con 0x7f0cbc005210 2024-04-17T08:53:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.555+0000 7f0cd9800640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cbc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd9800640 1 -- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cbc005210 msgr2=0x7f0cbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cda200640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cbc005210 0x7f0cbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd9800640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cbc005210 0x7f0cbc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd9800640 1 -- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cbc004800 msgr2=0x7f0cbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd9800640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd9800640 1 -- 172.21.15.67:0/790329000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc003520 con 0x7f0cbc003a70 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cda200640 1 -- 172.21.15.67:0/790329000 reap_dead start 2024-04-17T08:53:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.556+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cbc004800 0x7f0cbc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.557+0000 7f0cd9800640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cbc003e90 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8001750 tx=0x7f0cbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c4dc34ff9a800a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.557+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8001370 con 0x7f0cbc003a70 2024-04-17T08:53:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.557+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc80052d0 con 0x7f0cbc003a70 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.557+0000 7f0cd3e00640 1 -- 172.21.15.67:0/790329000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8003830 con 0x7f0cbc003a70 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 -- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 msgr2=0x7f0cbc003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cbc003e90 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8001750 tx=0x7f0cbc005830 comp rx=0 tx=0).stop 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 -- 172.21.15.67:0/790329000 shutdown_connections 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 --2- 172.21.15.67:0/790329000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cbc003e90 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 -- 172.21.15.67:0/790329000 >> 172.21.15.67:0/790329000 conn(0x7f0cd409aa60 msgr2=0x7f0cd40b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 -- 172.21.15.67:0/790329000 shutdown_connections 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.558+0000 7f0cdac00640 1 -- 172.21.15.67:0/790329000 wait complete. 2024-04-17T08:53:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.560+0000 7f0cdac00640 1 Processor -- start 2024-04-17T08:53:36.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.560+0000 7f0cdac00640 1 -- start start 2024-04-17T08:53:36.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 0x7f0cd41567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 0x7f0cd40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd415cdb0 con 0x7f0cbc003a70 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd40b3cb0 con 0x7f0cd40c77d0 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cdac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0cd415cf30 con 0x7f0cd40c7c50 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cd9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cd9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60504/0 (socket says 172.21.15.67:60504) 2024-04-17T08:53:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.561+0000 7f0cd9800640 1 -- 172.21.15.67:0/2376413800 learned_addr learned my addr 172.21.15.67:0/2376413800 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cda200640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 0x7f0cd40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 0x7f0cd41567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 -- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 msgr2=0x7f0cd40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 0x7f0cd40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 -- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 msgr2=0x7f0cd41567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 0x7f0cd41567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 -- 172.21.15.67:0/2376413800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd40c5030 con 0x7f0cbc003a70 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cda200640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 0x7f0cd40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 0x7f0cd41567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.562+0000 7f0cd9800640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8001870 tx=0x7f0cc8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.563+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8041020 con 0x7f0cbc003a70 2024-04-17T08:53:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.563+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc8005480 con 0x7f0cbc003a70 2024-04-17T08:53:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.563+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd40c5310 con 0x7f0cbc003a70 2024-04-17T08:53:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.563+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8001370 con 0x7f0cbc003a70 2024-04-17T08:53:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.563+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd40d03c0 con 0x7f0cbc003a70 2024-04-17T08:53:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.564+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd40755d0 con 0x7f0cbc003a70 2024-04-17T08:53:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.565+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f0cc8044020 con 0x7f0cbc003a70 2024-04-17T08:53:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.567+0000 7f0cd1600640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 0x7f0cb806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.567+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f0cc80ad790 con 0x7f0cbc003a70 2024-04-17T08:53:36.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.567+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 0x7f0cb806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:36.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.568+0000 7f0cd8e00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 0x7f0cb806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0cd41e6fa0 tx=0x7f0cc4008040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:36.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.572+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f0cc804b030 con 0x7f0cbc003a70 2024-04-17T08:53:36.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.802+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 --> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f0cd40b7480 con 0x7f0cb806ca10 2024-04-17T08:53:36.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.807+0000 7f0cd1600640 1 -- 172.21.15.67:0/2376413800 <== mgr.9871 v2:172.21.15.187:6832/46399553 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f0cd40b7480 con 0x7f0cb806ca10 2024-04-17T08:53:36.806 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T08:53:36.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T08:53:36.807 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-04-17T08:53:36.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T08:53:36.807 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.809+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 msgr2=0x7f0cb806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.809+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 0x7f0cb806eed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0cd41e6fa0 tx=0x7f0cc4008040 comp rx=0 tx=0).stop 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.809+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 msgr2=0x7f0cd40c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.809+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8001870 tx=0x7f0cc8001120 comp rx=0 tx=0).stop 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 shutdown_connections 2024-04-17T08:53:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f0cb806ca10 0x7f0cb806eed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0cd40c7c50 0x7f0cd40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0cd40c77d0 0x7f0cd41567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 --2- 172.21.15.67:0/2376413800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc003a70 0x7f0cd40c5ee0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 >> 172.21.15.67:0/2376413800 conn(0x7f0cd409aa60 msgr2=0x7f0cd40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 shutdown_connections 2024-04-17T08:53:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:36.810+0000 7f0cdac00640 1 -- 172.21.15.67:0/2376413800 wait complete. 2024-04-17T08:53:36.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-04-17T08:53:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.063+0000 7f4c7cc00640 1 Processor -- start 2024-04-17T08:53:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.063+0000 7f4c7cc00640 1 -- start start 2024-04-17T08:53:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c7cc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4c780b41a0 0x7f4c780b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c7cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c7cc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c78166a60 con 0x7f4c780b4580 2024-04-17T08:53:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c7cc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4c78166810 con 0x7f4c780b41a0 2024-04-17T08:53:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c7cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c78166d40 con 0x7f4c78162da0 2024-04-17T08:53:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.065+0000 7f4c77e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:36220/0 (socket says 172.21.15.67:36220) 2024-04-17T08:53:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 -- 172.21.15.67:0/937636891 learned_addr learned my addr 172.21.15.67:0/937636891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77400640 1 --2- 172.21.15.67:0/937636891 >> v2:172.21.15.187:3300/0 conn(0x7f4c780b41a0 0x7f4c780b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 -- 172.21.15.67:0/937636891 >> v2:172.21.15.187:3300/0 conn(0x7f4c780b41a0 msgr2=0x7f4c780b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 --2- 172.21.15.67:0/937636891 >> v2:172.21.15.187:3300/0 conn(0x7f4c780b41a0 0x7f4c780b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 -- 172.21.15.67:0/937636891 >> v1:172.21.15.187:6789/0 conn(0x7f4c780b4580 legacy=0x7f4c78162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.066+0000 7f4c77e00640 1 -- 172.21.15.67:0/937636891 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c78167020 con 0x7f4c78162da0 2024-04-17T08:53:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.068+0000 7f4c77e00640 1 --2- 172.21.15.67:0/937636891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4c70009e50 tx=0x7f4c7002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8ab7fd864047bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.068+0000 7f4c76a00640 1 -- 172.21.15.67:0/937636891 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7002cb00 con 0x7f4c78162da0 2024-04-17T08:53:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.068+0000 7f4c76a00640 1 -- 172.21.15.67:0/937636891 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7002cca0 con 0x7f4c78162da0 2024-04-17T08:53:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.068+0000 7f4c76a00640 1 -- 172.21.15.67:0/937636891 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70034610 con 0x7f4c78162da0 2024-04-17T08:53:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 -- 172.21.15.67:0/937636891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 msgr2=0x7f4c7806dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/937636891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4c70009e50 tx=0x7f4c7002bf60 comp rx=0 tx=0).stop 2024-04-17T08:53:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 -- 172.21.15.67:0/937636891 shutdown_connections 2024-04-17T08:53:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/937636891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c7806dd30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/937636891 >> v2:172.21.15.187:3300/0 conn(0x7f4c780b41a0 0x7f4c780b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 -- 172.21.15.67:0/937636891 >> 172.21.15.67:0/937636891 conn(0x7f4c780b2190 msgr2=0x7f4c780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 -- 172.21.15.67:0/937636891 shutdown_connections 2024-04-17T08:53:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.069+0000 7f4c7cc00640 1 -- 172.21.15.67:0/937636891 wait complete. 2024-04-17T08:53:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.071+0000 7f4c7cc00640 1 Processor -- start 2024-04-17T08:53:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.071+0000 7f4c7cc00640 1 -- start start 2024-04-17T08:53:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c780b4580 0x7f4c78165490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c780ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7806f980 con 0x7f4c780b41a0 2024-04-17T08:53:37.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7806f800 con 0x7f4c780b4580 2024-04-17T08:53:37.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c7cc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7806fb00 con 0x7f4c78162da0 2024-04-17T08:53:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.072+0000 7f4c77400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60518/0 (socket says 172.21.15.67:60518) 2024-04-17T08:53:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 -- 172.21.15.67:0/4117159536 learned_addr learned my addr 172.21.15.67:0/4117159536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77e00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c780ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c6fe00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c780b4580 0x7f4c78165490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 -- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 msgr2=0x7f4c780ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c780ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 -- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c780b4580 msgr2=0x7f4c78165490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c780b4580 0x7f4c78165490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 -- 172.21.15.67:0/4117159536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c781643f0 con 0x7f4c780b41a0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77e00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c780ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.073+0000 7f4c77400640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800e9e0 tx=0x7f4c6800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.074+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68017070 con 0x7f4c780b41a0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.074+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6800c9d0 con 0x7f4c780b41a0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.074+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800cdc0 con 0x7f4c780b41a0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.074+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c781646d0 con 0x7f4c780b41a0 2024-04-17T08:53:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.074+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c78164c90 con 0x7f4c780b41a0 2024-04-17T08:53:37.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.075+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c78068a10 con 0x7f4c780b41a0 2024-04-17T08:53:37.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.076+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 156) ==== 91142+0+0 (secure 0 0 0) 0x7f4c6801a070 con 0x7f4c780b41a0 2024-04-17T08:53:37.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.077+0000 7f4c6f400640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 0x7f4c5406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:37.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.077+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f4c6808c0d0 con 0x7f4c780b41a0 2024-04-17T08:53:37.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.078+0000 7f4c6fe00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 0x7f4c5406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:37.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.079+0000 7f4c6fe00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 0x7f4c5406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c781564a0 tx=0x7f4c60009040 comp rx=0 tx=0).ready entity=mgr.9871 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:37.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.082+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4c6805a3c0 con 0x7f4c780b41a0 2024-04-17T08:53:37.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.340+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f4c7806c7b0 con 0x7f4c780b41a0 2024-04-17T08:53:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.824+0000 7f4c6f400640 1 -- 172.21.15.67:0/4117159536 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v157) ==== 92+0+0 (secure 0 0 0) 0x7f4c68059cc0 con 0x7f4c780b41a0 2024-04-17T08:53:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.827+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 msgr2=0x7f4c5406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.827+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 0x7f4c5406eed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c781564a0 tx=0x7f4c60009040 comp rx=0 tx=0).stop 2024-04-17T08:53:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.827+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 msgr2=0x7f4c7814c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.827+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800e9e0 tx=0x7f4c6800eea0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 shutdown_connections 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f4c5406ca10 0x7f4c5406eed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4c78162da0 0x7f4c780ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c780b4580 0x7f4c78165490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 --2- 172.21.15.67:0/4117159536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c780b41a0 0x7f4c7814c600 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 >> 172.21.15.67:0/4117159536 conn(0x7f4c780b2190 msgr2=0x7f4c780b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 shutdown_connections 2024-04-17T08:53:37.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:37.828+0000 7f4c7cc00640 1 -- 172.21.15.67:0/4117159536 wait complete. 2024-04-17T08:53:37.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-04-17T08:53:38.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.059+0000 7f17d1a00640 1 Processor -- start 2024-04-17T08:53:38.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.060+0000 7f17d1a00640 1 -- start start 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.061+0000 7f17d1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17cc15eca0 0x7f17cc0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.061+0000 7f17d1a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc163860 con 0x7f17cc0cfa10 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.061+0000 7f17d1a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc0d1d40 con 0x7f17cc0d17a0 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.061+0000 7f17d1a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17cc0d2c30 con 0x7f17cc15eca0 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17cc15eca0 0x7f17cc0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17c3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17cc0cfa10 0x7f17cc0d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54428/0 (socket says 172.21.15.67:54428) 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17cc15eca0 0x7f17cc0d39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60542/0 (socket says 172.21.15.67:60542) 2024-04-17T08:53:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17c3e00640 1 -- 172.21.15.67:0/717387281 learned_addr learned my addr 172.21.15.67:0/717387281 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17caa00640 1 -- 172.21.15.67:0/717387281 >> v1:172.21.15.187:6789/0 conn(0x7f17cc0d17a0 legacy=0x7f17cc0b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17caa00640 1 -- 172.21.15.67:0/717387281 >> v1:172.21.15.67:6789/0 conn(0x7f17cc0cfa10 legacy=0x7f17cc0d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17caa00640 1 -- 172.21.15.67:0/717387281 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc0d2f10 con 0x7f17cc15eca0 2024-04-17T08:53:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.062+0000 7f17ca000640 1 -- 172.21.15.67:0/717387281 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107284593 0 0) 0x7f17cc163860 con 0x7f17cc0cfa10 2024-04-17T08:53:38.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.063+0000 7f17caa00640 1 --2- 172.21.15.67:0/717387281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc15eca0 0x7f17cc0d39f0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097a0 tx=0x7f17bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf316d882f939ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.064+0000 7f17ca000640 1 -- 172.21.15.67:0/717387281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc035020 con 0x7f17cc15eca0 2024-04-17T08:53:38.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.064+0000 7f17ca000640 1 -- 172.21.15.67:0/717387281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc031830 con 0x7f17cc15eca0 2024-04-17T08:53:38.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.064+0000 7f17ca000640 1 -- 172.21.15.67:0/717387281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc031c00 con 0x7f17cc15eca0 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.064+0000 7f17d1a00640 1 -- 172.21.15.67:0/717387281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc15eca0 msgr2=0x7f17cc0d39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.064+0000 7f17d1a00640 1 --2- 172.21.15.67:0/717387281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc15eca0 0x7f17cc0d39f0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097a0 tx=0x7f17bc02b8d0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.065+0000 7f17d1a00640 1 -- 172.21.15.67:0/717387281 shutdown_connections 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.065+0000 7f17d1a00640 1 --2- 172.21.15.67:0/717387281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc15eca0 0x7f17cc0d39f0 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.065+0000 7f17d1a00640 1 -- 172.21.15.67:0/717387281 >> 172.21.15.67:0/717387281 conn(0x7f17cc067600 msgr2=0x7f17cc0679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:38.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.065+0000 7f17d1a00640 1 -- 172.21.15.67:0/717387281 shutdown_connections 2024-04-17T08:53:38.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.065+0000 7f17d1a00640 1 -- 172.21.15.67:0/717387281 wait complete. 2024-04-17T08:53:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.066+0000 7f17d1a00640 1 Processor -- start 2024-04-17T08:53:38.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.067+0000 7f17d1a00640 1 -- start start 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.067+0000 7f17d1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.067+0000 7f17d1a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 0x7f17cc14c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17d1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 0x7f17cc14cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17d1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc15f080 con 0x7f17cc0cfa10 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17d1a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0d34d0 con 0x7f17cc0d17a0 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17d1a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc15f200 con 0x7f17cc15eca0 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60546/0 (socket says 172.21.15.67:60546) 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 -- 172.21.15.67:0/1762219133 learned_addr learned my addr 172.21.15.67:0/1762219133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17cb400640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 0x7f17cc14cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17c3e00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 0x7f17cc14c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 msgr2=0x7f17cc14cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 0x7f17cc14cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 msgr2=0x7f17cc14c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 0x7f17cc14c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17caa00640 1 -- 172.21.15.67:0/1762219133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc0ba570 con 0x7f17cc0cfa10 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17cb400640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 0x7f17cc14cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.068+0000 7f17c3e00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 0x7f17cc14c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17caa00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097a0 tx=0x7f17bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc004230 con 0x7f17cc0cfa10 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc0043f0 con 0x7f17cc0cfa10 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003850 con 0x7f17cc0cfa10 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc152d40 con 0x7f17cc0cfa10 2024-04-17T08:53:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.069+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc14d380 con 0x7f17cc0cfa10 2024-04-17T08:53:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.070+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17cc068ca0 con 0x7f17cc0cfa10 2024-04-17T08:53:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.072+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 157) ==== 91159+0+0 (secure 0 0 0) 0x7f17bc04c060 con 0x7f17cc0cfa10 2024-04-17T08:53:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.072+0000 7f17c2a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 0x7f17a806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.073+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7f17bc0ad9a0 con 0x7f17cc0cfa10 2024-04-17T08:53:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.073+0000 7f17c3e00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 msgr2=0x7f17a806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.073+0000 7f17c3e00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 0x7f17a806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.077+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f17bc040070 con 0x7f17cc0cfa10 2024-04-17T08:53:38.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.274+0000 7f17c3e00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 msgr2=0x7f17a806ef50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:38.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.274+0000 7f17c3e00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 0x7f17a806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.277+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f17cc0d2970 con 0x7f17cc0cfa10 2024-04-17T08:53:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.283+0000 7f17c2a00640 1 -- 172.21.15.67:0/1762219133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) ==== 138+0+0 (secure 0 0 0) 0x7f17bc07bb80 con 0x7f17cc0cfa10 2024-04-17T08:53:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 msgr2=0x7f17a806ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 0x7f17a806ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 msgr2=0x7f17cc14c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0097a0 tx=0x7f17bc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 shutdown_connections 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7f17a806ca90 0x7f17a806ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17cc15eca0 0x7f17cc14cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17cc0d17a0 0x7f17cc14c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 --2- 172.21.15.67:0/1762219133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0cfa10 0x7f17cc14c0b0 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 >> 172.21.15.67:0/1762219133 conn(0x7f17cc067600 msgr2=0x7f17cc0ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.285+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 shutdown_connections 2024-04-17T08:53:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.286+0000 7f17d1a00640 1 -- 172.21.15.67:0/1762219133 wait complete. 2024-04-17T08:53:38.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-04-17T08:53:38.386 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:38.387+0000 7f230e933040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:53:38.401 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:38.402+0000 7f7e62445040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:53:38.440 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:38.441+0000 7f230e933040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:53:38.459 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:38.461+0000 7f7e62445040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:53:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.501+0000 7fdb67e00640 1 Processor -- start 2024-04-17T08:53:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.502+0000 7fdb67e00640 1 -- start start 2024-04-17T08:53:38.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.503+0000 7fdb67e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdb6815ecb0 0x7fdb680d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.503+0000 7fdb67e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb680d3210 0x7fdb680cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.503+0000 7fdb67e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb680d00b0 con 0x7fdb680cf8e0 2024-04-17T08:53:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.503+0000 7fdb67e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdb680d0c30 con 0x7fdb6815ecb0 2024-04-17T08:53:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.503+0000 7fdb67e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb680d0390 con 0x7fdb680d3210 2024-04-17T08:53:38.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.504+0000 7fdb66a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdb6815ecb0 0x7fdb680d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.504+0000 7fdb5fe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb680d3210 0x7fdb680cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.504+0000 7fdb66a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdb6815ecb0 0x7fdb680d2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:52180/0 (socket says 172.21.15.67:52180) 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.504+0000 7fdb66a00640 1 -- 172.21.15.67:0/2912848834 learned_addr learned my addr 172.21.15.67:0/2912848834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.505+0000 7fdb66a00640 1 -- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb680d3210 msgr2=0x7fdb680cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.505+0000 7fdb66a00640 1 --2- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb680d3210 0x7fdb680cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.505+0000 7fdb66a00640 1 -- 172.21.15.67:0/2912848834 >> v1:172.21.15.187:6789/0 conn(0x7fdb680cf8e0 legacy=0x7fdb680d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.505+0000 7fdb66a00640 1 -- 172.21.15.67:0/2912848834 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb680d0670 con 0x7fdb6815ecb0 2024-04-17T08:53:38.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.506+0000 7fdb66a00640 1 --2- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6815ecb0 0x7fdb680d2cd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdb680d0db0 tx=0x7fdb5802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63a35ea766d5d65 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.506+0000 7fdb66000640 1 -- 172.21.15.67:0/2912848834 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58035020 con 0x7fdb6815ecb0 2024-04-17T08:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.506+0000 7fdb66000640 1 -- 172.21.15.67:0/2912848834 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb58031640 con 0x7fdb6815ecb0 2024-04-17T08:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.506+0000 7fdb66000640 1 -- 172.21.15.67:0/2912848834 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5802d040 con 0x7fdb6815ecb0 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.507+0000 7fdb67e00640 1 -- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6815ecb0 msgr2=0x7fdb680d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.507+0000 7fdb67e00640 1 --2- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6815ecb0 0x7fdb680d2cd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdb680d0db0 tx=0x7fdb5802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 -- 172.21.15.67:0/2912848834 shutdown_connections 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 --2- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb680d3210 0x7fdb680cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 --2- 172.21.15.67:0/2912848834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb6815ecb0 0x7fdb680d2cd0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 -- 172.21.15.67:0/2912848834 >> 172.21.15.67:0/2912848834 conn(0x7fdb6808bff0 msgr2=0x7fdb680d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 -- 172.21.15.67:0/2912848834 shutdown_connections 2024-04-17T08:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.508+0000 7fdb67e00640 1 -- 172.21.15.67:0/2912848834 wait complete. 2024-04-17T08:53:38.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.510+0000 7fdb67e00640 1 Processor -- start 2024-04-17T08:53:38.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.510+0000 7fdb67e00640 1 -- start start 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb680cf8e0 0x7fdb680b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 0x7fdb68157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6815f090 con 0x7fdb680d3210 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdb680cfd70 con 0x7fdb680cf8e0 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb67e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6815f210 con 0x7fdb6815ecb0 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.511+0000 7fdb5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60558/0 (socket says 172.21.15.67:60558) 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 -- 172.21.15.67:0/3501067725 learned_addr learned my addr 172.21.15.67:0/3501067725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:38.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb66a00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb680cf8e0 0x7fdb680b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb67400640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 0x7fdb68157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 msgr2=0x7fdb68157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 0x7fdb68157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb680cf8e0 msgr2=0x7fdb680b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb680cf8e0 0x7fdb680b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 -- 172.21.15.67:0/3501067725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb68160060 con 0x7fdb680d3210 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb67400640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 0x7fdb68157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.512+0000 7fdb5fe00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e9b0 tx=0x7fdb5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.513+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50017070 con 0x7fdb680d3210 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.513+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5000c9a0 con 0x7fdb680d3210 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.513+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb681f8530 con 0x7fdb680d3210 2024-04-17T08:53:38.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.513+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5000cd90 con 0x7fdb680d3210 2024-04-17T08:53:38.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.513+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb680cc480 con 0x7fdb680d3210 2024-04-17T08:53:38.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.514+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb681f8c40 con 0x7fdb680d3210 2024-04-17T08:53:38.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.515+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 157) ==== 91159+0+0 (secure 0 0 0) 0x7fdb5001a070 con 0x7fdb680d3210 2024-04-17T08:53:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.516+0000 7fdb5ea00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 0x7fdb4406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.516+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7fdb5008c0c0 con 0x7fdb680d3210 2024-04-17T08:53:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.516+0000 7fdb66a00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 msgr2=0x7fdb4406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.516+0000 7fdb66a00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 0x7fdb4406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:38.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.520+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fdb50059d80 con 0x7fdb680d3210 2024-04-17T08:53:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.717+0000 7fdb66a00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 msgr2=0x7fdb4406ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.717+0000 7fdb66a00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 0x7fdb4406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:38.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.723+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fdb68068a10 con 0x7fdb680d3210 2024-04-17T08:53:38.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.728+0000 7fdb5ea00640 1 -- 172.21.15.67:0/3501067725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) ==== 133+0+0 (secure 0 0 0) 0x7fdb5005e080 con 0x7fdb680d3210 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 msgr2=0x7fdb4406ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 0x7fdb4406ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 msgr2=0x7fdb6815f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e9b0 tx=0x7fdb5000ee70 comp rx=0 tx=0).stop 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 shutdown_connections 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7fdb4406ca60 0x7fdb4406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdb6815ecb0 0x7fdb68157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb680d3210 0x7fdb6815f5a0 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 --2- 172.21.15.67:0/3501067725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb680cf8e0 0x7fdb680b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.730+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 >> 172.21.15.67:0/3501067725 conn(0x7fdb6808bff0 msgr2=0x7fdb6806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.731+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 shutdown_connections 2024-04-17T08:53:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.731+0000 7fdb67e00640 1 -- 172.21.15.67:0/3501067725 wait complete. 2024-04-17T08:53:38.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-04-17T08:53:38.888 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:38.889+0000 7f230e933040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:53:38.937 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:38.939+0000 7f7e62445040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:53:38.940 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:38.942+0000 7f230e933040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:53:38.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.962+0000 7ff102c00640 1 Processor -- start 2024-04-17T08:53:38.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.963+0000 7ff102c00640 1 -- start start 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.964+0000 7ff102c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff0fc0b46f0 0x7ff0fc0b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.964+0000 7ff102c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0fc0d4510 0x7ff0fc0d4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.964+0000 7ff102c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0b51d0 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.964+0000 7ff102c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0b5790 con 0x7ff0fc0d4510 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.964+0000 7ff102c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0d2830 con 0x7ff0fc0b46f0 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0fc0d4510 0x7ff0fc0d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0fc0d4510 0x7ff0fc0d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60568/0 (socket says 172.21.15.67:60568) 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 -- 172.21.15.67:0/648484235 learned_addr learned my addr 172.21.15.67:0/648484235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:38.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff100e00640 1 --2- 172.21.15.67:0/648484235 >> v2:172.21.15.187:3300/0 conn(0x7ff0fc0b46f0 0x7ff0fc0b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 -- 172.21.15.67:0/648484235 >> v2:172.21.15.187:3300/0 conn(0x7ff0fc0b46f0 msgr2=0x7ff0fc0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 --2- 172.21.15.67:0/648484235 >> v2:172.21.15.187:3300/0 conn(0x7ff0fc0b46f0 0x7ff0fc0b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 -- 172.21.15.67:0/648484235 >> v1:172.21.15.187:6789/0 conn(0x7ff0fc15e9b0 legacy=0x7ff0fc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.965+0000 7ff102200640 1 -- 172.21.15.67:0/648484235 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0b54b0 con 0x7ff0fc0d4510 2024-04-17T08:53:38.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.966+0000 7ff102200640 1 --2- 172.21.15.67:0/648484235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc0d4920 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc08dee0 tx=0x7ff0ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553c05b0da449631 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff0fbe00640 1 -- 172.21.15.67:0/648484235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec02e070 con 0x7ff0fc0d4510 2024-04-17T08:53:38.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff0fbe00640 1 -- 172.21.15.67:0/648484235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0ec037070 con 0x7ff0fc0d4510 2024-04-17T08:53:38.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff0fbe00640 1 -- 172.21.15.67:0/648484235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec031a00 con 0x7ff0fc0d4510 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 -- 172.21.15.67:0/648484235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0d4510 msgr2=0x7ff0fc0d4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 --2- 172.21.15.67:0/648484235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc0d4920 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc08dee0 tx=0x7ff0ec02b880 comp rx=0 tx=0).stop 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 -- 172.21.15.67:0/648484235 shutdown_connections 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 --2- 172.21.15.67:0/648484235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc0d4920 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 --2- 172.21.15.67:0/648484235 >> v2:172.21.15.187:3300/0 conn(0x7ff0fc0b46f0 0x7ff0fc0b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.967+0000 7ff102c00640 1 -- 172.21.15.67:0/648484235 >> 172.21.15.67:0/648484235 conn(0x7ff0fc09daa0 msgr2=0x7ff0fc08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.968+0000 7ff102c00640 1 -- 172.21.15.67:0/648484235 shutdown_connections 2024-04-17T08:53:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.968+0000 7ff102c00640 1 -- 172.21.15.67:0/648484235 wait complete. 2024-04-17T08:53:38.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.969+0000 7ff102c00640 1 Processor -- start 2024-04-17T08:53:38.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.969+0000 7ff102c00640 1 -- start start 2024-04-17T08:53:38.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc0b46f0 0x7ff0fc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 0x7ff0fc0b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d4e60 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0cfae0 con 0x7ff0fc0d4510 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.970+0000 7ff102c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc162830 con 0x7ff0fc0b46f0 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 0x7ff0fc0b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 0x7ff0fc0b9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60582/0 (socket says 172.21.15.67:60582) 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 -- 172.21.15.67:0/126183311 learned_addr learned my addr 172.21.15.67:0/126183311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff100e00640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc0b46f0 0x7ff0fc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc0b46f0 msgr2=0x7ff0fc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff0fc0b46f0 0x7ff0fc15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc0d4510 msgr2=0x7ff0fc152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0fc0d4510 0x7ff0fc152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:38.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc153180 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.971+0000 7ff102200640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 0x7ff0fc0b9c00 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009960 tx=0x7ff0ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:38.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.972+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec032670 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.972+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0ec032810 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.972+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec032c00 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.972+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc0b8b60 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.972+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc0b9120 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.973+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc0755d0 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.974+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 157) ==== 91159+0+0 (secure 0 0 0) 0x7ff0ec04c020 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.975+0000 7ff0f9600640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.975+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(286..286 src has 1..286) ==== 5737+0+0 (secure 0 0 0) 0x7ff0ec0adce0 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.975+0000 7ff101800640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.975+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:53:38.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:38.979+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff0fc153180 con 0x7ff0fc15e9b0 2024-04-17T08:53:38.992 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:38.994+0000 7f230e933040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:53:38.995 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:38.997+0000 7f7e62445040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:53:39.050 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.051+0000 7f7e62445040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:53:39.053 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.055+0000 7f230e933040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:53:39.102 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.104+0000 7f230e933040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:53:39.115 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.117+0000 7f7e62445040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:53:39.168 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.169+0000 7f7e62445040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:53:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:39.176+0000 7ff101800640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:39.176+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:53:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:39.212+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7ff0fc0b3ea0 con 0x7ff0dc06ca60 2024-04-17T08:53:39.232 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.233+0000 7f230e933040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:53:39.294 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.296+0000 7f230e933040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:39.304 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.306+0000 7f7e62445040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:53:39.346 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.347+0000 7f230e933040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:53:39.371 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.373+0000 7f7e62445040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:39.403 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.404+0000 7f230e933040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:53:39.424 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.425+0000 7f7e62445040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:53:39.454 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.455+0000 7f230e933040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:53:39.484 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.486+0000 7f7e62445040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:53:39.538 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.539+0000 7f7e62445040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:53:39.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:39.577+0000 7ff101800640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:39.577+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-17T08:53:39.838 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.839+0000 7f230e933040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:53:39.944 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:39.945+0000 7f7e62445040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:53:39.997 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:39.999+0000 7f230e933040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:53:40.106 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.108+0000 7f230e933040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:53:40.113 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.114+0000 7f7e62445040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:53:40.157 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.158+0000 7f230e933040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:53:40.227 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.229+0000 7f7e62445040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:53:40.280 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.282+0000 7f7e62445040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:53:40.282 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.284+0000 7f230e933040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:40.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:40.378+0000 7ff101800640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:40.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:40.378+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-17T08:53:40.391 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.392+0000 7f230e933040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:53:40.414 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.416+0000 7f7e62445040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:40.528 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.529+0000 7f7e62445040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:53:40.565 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.567+0000 7f230e933040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:40.615 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.616+0000 7f230e933040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:40.713 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.714+0000 7f7e62445040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:53:40.752 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:40.753+0000 7f230e933040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:53:40.764 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.766+0000 7f7e62445040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:53:40.908 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:40.910+0000 7f7e62445040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:53:41.566 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:41.567+0000 7f230e933040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:53:41.782 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:41.784+0000 7f7e62445040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:53:41.869 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:41.871+0000 7f230e933040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:53:41.927 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:41.929+0000 7f230e933040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:41.980+0000 7ff101800640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/46399553 2024-04-17T08:53:41.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:41.980+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-17T08:53:42.058 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.059+0000 7f230e933040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:42.084+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 158) ==== 90768+0+0 (secure 0 0 0) 0x7ff0ec077ec0 con 0x7ff0fc15e9b0 2024-04-17T08:53:42.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:42.085+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 msgr2=0x7ff0dc06ef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:53:42.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:42.085+0000 7ff0f9600640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/46399553,v1:172.21.15.187:6833/46399553] conn(0x7ff0dc06ca60 0x7ff0dc06ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:42.138 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:42.140+0000 7f7e62445040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:53:42.209 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:42.211+0000 7f7e62445040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:53:42.370 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:53:42.372+0000 7f7e62445040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.390+0000 7f2282000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.390+0000 7f2282000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2282000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2282000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2282000640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.389 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2284800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.390 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2284800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.390 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2284800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.390 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2284800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:42.390 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:42.391+0000 7f2284800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.151+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 159) ==== 90895+0+0 (secure 0 0 0) 0x7ff0ec050d20 con 0x7ff0fc15e9b0 2024-04-17T08:53:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.152+0000 7ff0f9600640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7ff0dc074960 0x7ff0dc076d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.152+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7ff0ec04de70 con 0x7ff0dc074960 2024-04-17T08:53:43.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.158+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7ff0dc074960 0x7ff0dc076d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.161+0000 7ff101800640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7ff0dc074960 0x7ff0dc076d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40094d0 tx=0x7ff0f40097c0 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:43.174 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:43.176+0000 7f227e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.175 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:43.176+0000 7f227e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.175 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:43.176+0000 7f227e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.175 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:43.176+0000 7f227e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.175 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:53:43.176+0000 7f227e400640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:53:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.225+0000 7ff0f9600640 1 -- 172.21.15.67:0/126183311 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff0ec04de70 con 0x7ff0dc074960 2024-04-17T08:53:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7ff0dc074960 msgr2=0x7ff0dc076d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7ff0dc074960 0x7ff0dc076d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40094d0 tx=0x7ff0f40097c0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 msgr2=0x7ff0fc0b9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 --2- 172.21.15.67:0/126183311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0fc15e9b0 0x7ff0fc0b9c00 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009960 tx=0x7ff0ec0046c0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102200640 1 -- 172.21.15.67:0/126183311 reap_dead start 2024-04-17T08:53:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 shutdown_connections 2024-04-17T08:53:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 >> 172.21.15.67:0/126183311 conn(0x7ff0fc09daa0 msgr2=0x7ff0fc15dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 shutdown_connections 2024-04-17T08:53:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.229+0000 7ff102c00640 1 -- 172.21.15.67:0/126183311 wait complete. 2024-04-17T08:53:43.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-04-17T08:53:43.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.486+0000 7f8c1de00640 1 Processor -- start 2024-04-17T08:53:43.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.486+0000 7f8c1de00640 1 -- start start 2024-04-17T08:53:43.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1de00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8c1815ca80 0x7f8c18160de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c1809b170 0x7f8c180cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c180b1710 con 0x7f8c1809b820 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1815d940 con 0x7f8c1809b170 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1de00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1815ddd0 con 0x7f8c1815ca80 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c0fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c1809b820 0x7f8c1809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53708/0 (socket says 172.21.15.67:53708) 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c0fe00640 1 -- 172.21.15.67:0/2189829640 learned_addr learned my addr 172.21.15.67:0/2189829640 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.488+0000 7f8c1d400640 1 --2- 172.21.15.67:0/2189829640 >> v2:172.21.15.67:3300/0 conn(0x7f8c1809b170 0x7f8c180cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.489+0000 7f8c1ca00640 1 --2- 172.21.15.67:0/2189829640 >> v2:172.21.15.187:3300/0 conn(0x7f8c1815ca80 0x7f8c18160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.489+0000 7f8c1d400640 1 -- 172.21.15.67:0/2189829640 >> v2:172.21.15.187:3300/0 conn(0x7f8c1815ca80 msgr2=0x7f8c18160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.489+0000 7f8c1d400640 1 --2- 172.21.15.67:0/2189829640 >> v2:172.21.15.187:3300/0 conn(0x7f8c1815ca80 0x7f8c18160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.489+0000 7f8c1d400640 1 -- 172.21.15.67:0/2189829640 >> v1:172.21.15.67:6789/0 conn(0x7f8c1809b820 legacy=0x7f8c1809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.489+0000 7f8c1d400640 1 -- 172.21.15.67:0/2189829640 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1809bd20 con 0x7f8c1809b170 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.490+0000 7f8c1d400640 1 --2- 172.21.15.67:0/2189829640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b170 0x7f8c180cd150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009d90 tx=0x7f8c1002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4744add8c1a441 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.490+0000 7f8c17e00640 1 -- 172.21.15.67:0/2189829640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10039070 con 0x7f8c1809b170 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.490+0000 7f8c17e00640 1 -- 172.21.15.67:0/2189829640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c1002ccf0 con 0x7f8c1809b170 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.490+0000 7f8c17e00640 1 -- 172.21.15.67:0/2189829640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10034660 con 0x7f8c1809b170 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 -- 172.21.15.67:0/2189829640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b170 msgr2=0x7f8c180cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2189829640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b170 0x7f8c180cd150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009d90 tx=0x7f8c1002bf40 comp rx=0 tx=0).stop 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 -- 172.21.15.67:0/2189829640 shutdown_connections 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2189829640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b170 0x7f8c180cd150 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2189829640 >> v2:172.21.15.187:3300/0 conn(0x7f8c1815ca80 0x7f8c18160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 -- 172.21.15.67:0/2189829640 >> 172.21.15.67:0/2189829640 conn(0x7f8c180d24d0 msgr2=0x7f8c180d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 -- 172.21.15.67:0/2189829640 shutdown_connections 2024-04-17T08:53:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.491+0000 7f8c1de00640 1 -- 172.21.15.67:0/2189829640 wait complete. 2024-04-17T08:53:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.493+0000 7f8c1de00640 1 Processor -- start 2024-04-17T08:53:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.493+0000 7f8c1de00640 1 -- start start 2024-04-17T08:53:43.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c1809b170 0x7f8c1806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c1815ca80 0x7f8c18159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1815d050 con 0x7f8c1809b820 2024-04-17T08:53:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8c18161320 con 0x7f8c1809b170 2024-04-17T08:53:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c1de00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1815d5d0 con 0x7f8c1815ca80 2024-04-17T08:53:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33644/0 (socket says 172.21.15.67:33644) 2024-04-17T08:53:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.494+0000 7f8c0fe00640 1 -- 172.21.15.67:0/2519477165 learned_addr learned my addr 172.21.15.67:0/2519477165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c1d400640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c1815ca80 0x7f8c18159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c1ca00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c1809b170 0x7f8c1806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 -- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c1815ca80 msgr2=0x7f8c18159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c1815ca80 0x7f8c18159ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 -- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c1809b170 msgr2=0x7f8c1806e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c1809b170 0x7f8c1806e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 -- 172.21.15.67:0/2519477165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1806cb70 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c0fe00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c470 tx=0x7f8c00001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00017070 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c00005ce0 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c000072f0 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1806ce50 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.495+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c180d0300 con 0x7f8c1809b820 2024-04-17T08:53:43.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.497+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c180755d0 con 0x7f8c1809b820 2024-04-17T08:53:43.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.498+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 159) ==== 90895+0+0 (secure 0 0 0) 0x7f8c0001a070 con 0x7f8c1809b820 2024-04-17T08:53:43.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.499+0000 7f8c15600640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 0x7f8bf406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:53:43.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.499+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(287..287 src has 1..287) ==== 6081+0+0 (secure 0 0 0) 0x7f8c0008ad40 con 0x7f8c1809b820 2024-04-17T08:53:43.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.499+0000 7f8c1ca00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 0x7f8bf406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:53:43.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.501+0000 7f8c1ca00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 0x7f8bf406ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c08006fd0 tx=0x7f8c08008040 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:53:43.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.504+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f8c0008e030 con 0x7f8c1809b820 2024-04-17T08:53:43.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.729+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8c180d1450 con 0x7f8bf406c790 2024-04-17T08:53:43.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.963+0000 7f8c15600640 1 -- 172.21.15.67:0/2519477165 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+85 (secure 0 0 0) 0x7f8c180d1450 con 0x7f8bf406c790 2024-04-17T08:53:43.961 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule set for path /volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/.. 2024-04-17T08:53:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.966+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 msgr2=0x7f8bf406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.966+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 0x7f8bf406ec50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c08006fd0 tx=0x7f8c08008040 comp rx=0 tx=0).stop 2024-04-17T08:53:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.966+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 msgr2=0x7f8c181599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:53:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.966+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c470 tx=0x7f8c00001e70 comp rx=0 tx=0).stop 2024-04-17T08:53:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 shutdown_connections 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f8bf406c790 0x7f8bf406ec50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8c1815ca80 0x7f8c18159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c1809b820 0x7f8c181599a0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 --2- 172.21.15.67:0/2519477165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c1809b170 0x7f8c1806e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 >> 172.21.15.67:0/2519477165 conn(0x7f8c180d24d0 msgr2=0x7f8c180b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 shutdown_connections 2024-04-17T08:53:43.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:53:43.967+0000 7f8c1de00640 1 -- 172.21.15.67:0/2519477165 wait complete. 2024-04-17T08:53:43.987 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/.. 2024-04-17T08:53:58.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:53:58.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:54:00.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:54:00.987+0000 7f226b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:54:00.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:54:00.987+0000 7f226b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:54:00.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:54:00.987+0000 7f226b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:54:00.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:54:00.988+0000 7f226b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:54:00.987 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:54:00.988+0000 7f226b800640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:54:28.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:54:28.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:54:58.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:54:58.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:54:58.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-04-17T08:54:59.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.209+0000 7f416ac00640 1 Processor -- start 2024-04-17T08:54:59.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.209+0000 7f416ac00640 1 -- start start 2024-04-17T08:54:59.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f416ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f416415ecb0 0x7f41640d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f416ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41640cfab0 0x7f41640b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f416ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41640d00b0 con 0x7f41640d3210 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f416ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f41640d0c30 con 0x7f416415ecb0 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f416ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41640d0390 con 0x7f41640cfab0 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f4169800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f416415ecb0 0x7f41640d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f4168e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f41640d3210 0x7f41640cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42298/0 (socket says 172.21.15.67:42298) 2024-04-17T08:54:59.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.211+0000 7f4168e00640 1 -- 172.21.15.67:0/4247370469 learned_addr learned my addr 172.21.15.67:0/4247370469 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.212+0000 7f4169800640 1 -- 172.21.15.67:0/4247370469 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41640cfab0 msgr2=0x7f41640b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.212+0000 7f4169800640 1 --2- 172.21.15.67:0/4247370469 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41640cfab0 0x7f41640b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.212+0000 7f4169800640 1 -- 172.21.15.67:0/4247370469 >> v1:172.21.15.67:6789/0 conn(0x7f41640d3210 legacy=0x7f41640cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.212+0000 7f4169800640 1 -- 172.21.15.67:0/4247370469 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41640d0670 con 0x7f416415ecb0 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.212+0000 7f4163e00640 1 -- 172.21.15.67:0/4247370469 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2607475544 0 0) 0x7f41640d00b0 con 0x7f41640d3210 2024-04-17T08:54:59.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.213+0000 7f4169800640 1 --2- 172.21.15.67:0/4247370469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416415ecb0 0x7f41640d2cd0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f41580097a0 tx=0x7f415802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd591227d04118f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.213+0000 7f4163e00640 1 -- 172.21.15.67:0/4247370469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158031780 con 0x7f416415ecb0 2024-04-17T08:54:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.213+0000 7f4163e00640 1 -- 172.21.15.67:0/4247370469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4158037070 con 0x7f416415ecb0 2024-04-17T08:54:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.213+0000 7f4163e00640 1 -- 172.21.15.67:0/4247370469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158031b40 con 0x7f416415ecb0 2024-04-17T08:54:59.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 -- 172.21.15.67:0/4247370469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416415ecb0 msgr2=0x7f41640d2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 --2- 172.21.15.67:0/4247370469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416415ecb0 0x7f41640d2cd0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f41580097a0 tx=0x7f415802b820 comp rx=0 tx=0).stop 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 -- 172.21.15.67:0/4247370469 shutdown_connections 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 --2- 172.21.15.67:0/4247370469 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41640cfab0 0x7f41640b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 --2- 172.21.15.67:0/4247370469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416415ecb0 0x7f41640d2cd0 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 -- 172.21.15.67:0/4247370469 >> 172.21.15.67:0/4247370469 conn(0x7f416408bff0 msgr2=0x7f41640d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 -- 172.21.15.67:0/4247370469 shutdown_connections 2024-04-17T08:54:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.214+0000 7f416ac00640 1 -- 172.21.15.67:0/4247370469 wait complete. 2024-04-17T08:54:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.217+0000 7f416ac00640 1 Processor -- start 2024-04-17T08:54:59.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.217+0000 7f416ac00640 1 -- start start 2024-04-17T08:54:59.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.218+0000 7f416ac00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41640cfab0 0x7f416415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.218+0000 7f416ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f416ac00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 0x7f41640b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f416ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f416415f210 con 0x7f41640d3210 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f416ac00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f416415f090 con 0x7f41640cfab0 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f416ac00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41640b4500 con 0x7f416415ecb0 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f4168e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f4168e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43214/0 (socket says 172.21.15.67:43214) 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f4168e00640 1 -- 172.21.15.67:0/3100104013 learned_addr learned my addr 172.21.15.67:0/3100104013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f4169800640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41640cfab0 0x7f416415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.219+0000 7f416a200640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 0x7f41640b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 -- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 msgr2=0x7f41640b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 0x7f41640b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 -- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41640cfab0 msgr2=0x7f416415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41640cfab0 0x7f416415f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 -- 172.21.15.67:0/3100104013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416415fe50 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f416a200640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 0x7f41640b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4168e00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f415000c4d0 tx=0x7f415000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150018070 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4150007500 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4164160130 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.220+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150004a20 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.221+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41641606f0 con 0x7f41640d3210 2024-04-17T08:54:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.222+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41640d25e0 con 0x7f41640d3210 2024-04-17T08:54:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.223+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 162) ==== 91324+0+0 (secure 0 0 0) 0x7f415001b070 con 0x7f41640d3210 2024-04-17T08:54:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.224+0000 7f4161600640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 0x7f414406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.225+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f415008b3c0 con 0x7f41640d3210 2024-04-17T08:54:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.225+0000 7f4169800640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 0x7f414406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.226+0000 7f4169800640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 0x7f414406f0d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f4158008030 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.230+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f41640d25e0 con 0x7f41640d3210 2024-04-17T08:54:59.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.466+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f4164068a10 con 0x7f414406cc10 2024-04-17T08:54:59.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.486+0000 7f4161600640 1 -- 172.21.15.67:0/3100104013 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7f4164068a10 con 0x7f414406cc10 2024-04-17T08:54:59.485 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770 2024-04-17T08:54:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 msgr2=0x7f414406f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 0x7f414406f0d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4158036000 tx=0x7f4158008030 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 msgr2=0x7f41640b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f415000c4d0 tx=0x7f415000c990 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 shutdown_connections 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f414406cc10 0x7f414406f0d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f416415ecb0 0x7f41640b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41640d3210 0x7f41640b1290 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 --2- 172.21.15.67:0/3100104013 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41640cfab0 0x7f416415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 >> 172.21.15.67:0/3100104013 conn(0x7f416408bff0 msgr2=0x7f416415de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 shutdown_connections 2024-04-17T08:54:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.488+0000 7f416ac00640 1 -- 172.21.15.67:0/3100104013 wait complete. 2024-04-17T08:54:59.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:54:59.509 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/../.snap) 2024-04-17T08:54:59.567 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_54_00_UTC 2024-04-17T08:54:59.569 DEBUG:tasks.cephfs.test_snap_schedules:snaps:['scheduled-2024-04-17-08_54_00_UTC'] 2024-04-17T08:54:59.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-04-17T08:54:59.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.791+0000 7fcacdc00640 1 Processor -- start 2024-04-17T08:54:59.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.791+0000 7fcacdc00640 1 -- start start 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.792+0000 7fcacdc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcac809aa60 0x7fcac809ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.792+0000 7fcacdc00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcac80d0280 0x7fcac80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.792+0000 7fcacdc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac809b380 con 0x7fcac81605a0 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.792+0000 7fcacdc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcac815d350 con 0x7fcac809aa60 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.792+0000 7fcacdc00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcac80d0c20 con 0x7fcac80d0280 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcac809aa60 0x7fcac809ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcac809aa60 0x7fcac809ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43222/0 (socket says 172.21.15.67:43222) 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 -- 172.21.15.67:0/2051163686 learned_addr learned my addr 172.21.15.67:0/2051163686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:54:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac7400640 1 --1- 172.21.15.67:0/2051163686 >> v1:172.21.15.67:6789/0 conn(0x7fcac81605a0 0x7fcac8160a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42312/0 (socket says 172.21.15.67:42312) 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6000640 1 --2- 172.21.15.67:0/2051163686 >> v2:172.21.15.187:3300/0 conn(0x7fcac80d0280 0x7fcac80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 -- 172.21.15.67:0/2051163686 >> v2:172.21.15.187:3300/0 conn(0x7fcac80d0280 msgr2=0x7fcac80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 --2- 172.21.15.67:0/2051163686 >> v2:172.21.15.187:3300/0 conn(0x7fcac80d0280 0x7fcac80d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 -- 172.21.15.67:0/2051163686 >> v1:172.21.15.67:6789/0 conn(0x7fcac81605a0 legacy=0x7fcac8160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac6a00640 1 -- 172.21.15.67:0/2051163686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac80d0e60 con 0x7fcac809aa60 2024-04-17T08:54:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.793+0000 7fcac5600640 1 -- 172.21.15.67:0/2051163686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 917586172 0 0) 0x7fcac809b380 con 0x7fcac81605a0 2024-04-17T08:54:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.795+0000 7fcac6a00640 1 --2- 172.21.15.67:0/2051163686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac809ae40 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fcac80d01b0 tx=0x7fcab002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3b84fb08419340 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.795+0000 7fcac5600640 1 -- 172.21.15.67:0/2051163686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0034070 con 0x7fcac809aa60 2024-04-17T08:54:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.795+0000 7fcac5600640 1 -- 172.21.15.67:0/2051163686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab0030860 con 0x7fcac809aa60 2024-04-17T08:54:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.795+0000 7fcac5600640 1 -- 172.21.15.67:0/2051163686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0030c30 con 0x7fcac809aa60 2024-04-17T08:54:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 -- 172.21.15.67:0/2051163686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 msgr2=0x7fcac809ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 --2- 172.21.15.67:0/2051163686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac809ae40 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fcac80d01b0 tx=0x7fcab002b9a0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 -- 172.21.15.67:0/2051163686 shutdown_connections 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 --2- 172.21.15.67:0/2051163686 >> v2:172.21.15.187:3300/0 conn(0x7fcac80d0280 0x7fcac80d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 --2- 172.21.15.67:0/2051163686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac809ae40 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 -- 172.21.15.67:0/2051163686 >> 172.21.15.67:0/2051163686 conn(0x7fcac80b41a0 msgr2=0x7fcac80b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 -- 172.21.15.67:0/2051163686 shutdown_connections 2024-04-17T08:54:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.796+0000 7fcacdc00640 1 -- 172.21.15.67:0/2051163686 wait complete. 2024-04-17T08:54:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.798+0000 7fcacdc00640 1 Processor -- start 2024-04-17T08:54:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.798+0000 7fcacdc00640 1 -- start start 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcac80d0280 0x7fcac815eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac81605a0 0x7fcac815f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcac80d1420 con 0x7fcac809aa60 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcac80cd630 con 0x7fcac81605a0 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcacdc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcac80d1700 con 0x7fcac80d0280 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcac6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcac6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43226/0 (socket says 172.21.15.67:43226) 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.799+0000 7fcac6a00640 1 -- 172.21.15.67:0/3319032001 learned_addr learned my addr 172.21.15.67:0/3319032001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 -- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcac80d0280 msgr2=0x7fcac815eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcac80d0280 0x7fcac815eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 -- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac81605a0 msgr2=0x7fcac815f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac81605a0 0x7fcac815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:54:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 -- 172.21.15.67:0/3319032001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac80c7c70 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.800+0000 7fcac6a00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.801+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0034040 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.801+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab0030660 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.801+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0039cc0 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.801+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac80c7f50 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.801+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac80c8510 con 0x7fcac809aa60 2024-04-17T08:54:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.802+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac8068ca0 con 0x7fcac809aa60 2024-04-17T08:54:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.803+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 162) ==== 91324+0+0 (secure 0 0 0) 0x7fcab004b060 con 0x7fcac809aa60 2024-04-17T08:54:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.804+0000 7fcabea00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 0x7fcaa406f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:54:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.804+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7fcab00ae100 con 0x7fcac809aa60 2024-04-17T08:54:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.805+0000 7fcac6000640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 0x7fcaa406f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:54:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.806+0000 7fcac6000640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 0x7fcaa406f100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcab80070b0 tx=0x7fcab8007040 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:54:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:54:59.808+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcab007beb0 con 0x7fcac809aa60 2024-04-17T08:55:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.042+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7fcac809bfa0 con 0x7fcaa406cc40 2024-04-17T08:55:00.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.055+0000 7fcabea00640 1 -- 172.21.15.67:0/3319032001 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7fcac809bfa0 con 0x7fcaa406cc40 2024-04-17T08:55:00.055 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770 2024-04-17T08:55:00.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 msgr2=0x7fcaa406f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 0x7fcaa406f100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcab80070b0 tx=0x7fcab8007040 comp rx=0 tx=0).stop 2024-04-17T08:55:00.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 msgr2=0x7fcac806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab00046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 shutdown_connections 2024-04-17T08:55:00.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fcaa406cc40 0x7fcaa406f100 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcac81605a0 0x7fcac815f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcac80d0280 0x7fcac815eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 --2- 172.21.15.67:0/3319032001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac809aa60 0x7fcac806e3c0 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 >> 172.21.15.67:0/3319032001 conn(0x7fcac80b41a0 msgr2=0x7fcac80cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:00.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 shutdown_connections 2024-04-17T08:55:00.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.057+0000 7fcacdc00640 1 -- 172.21.15.67:0/3319032001 wait complete. 2024-04-17T08:55:00.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:00.074 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/../.snap) 2024-04-17T08:55:00.126 INFO:teuthology.orchestra.run.smithi067.stdout:scheduled-2024-04-17-08_54_00_UTC 2024-04-17T08:55:00.127 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/../.snap/scheduled-2024-04-17-08_54_00_UTC 2024-04-17T08:55:00.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:00.128 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/../.snap/scheduled-2024-04-17-08_54_00_UTC) 2024-04-17T08:55:00.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --subvol sv01 . 1m 2024-04-17T08:55:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.448+0000 7f34b1a00640 1 Processor -- start 2024-04-17T08:55:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.449+0000 7f34b1a00640 1 -- start start 2024-04-17T08:55:00.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34b1a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f34ac15eaf0 0x7f34ac0d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34b1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f34ac0b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34b1a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34ac0d4330 con 0x7f34ac0cfe70 2024-04-17T08:55:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34b1a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f34ac0ba690 con 0x7f34ac15eaf0 2024-04-17T08:55:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34b1a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34ac0d4610 con 0x7f34ac0cfa10 2024-04-17T08:55:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34ab400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34ac0cfe70 0x7f34ac0b7b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42326/0 (socket says 172.21.15.67:42326) 2024-04-17T08:55:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.450+0000 7f34ab400640 1 -- 172.21.15.67:0/1527532465 learned_addr learned my addr 172.21.15.67:0/1527532465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34aaa00640 1 --2- 172.21.15.67:0/1527532465 >> v2:172.21.15.187:3300/0 conn(0x7f34ac15eaf0 0x7f34ac0d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34aa000640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f34ac0b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329384471 0 0) 0x7f34ac0d4330 con 0x7f34ac0cfe70 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f348c003520 con 0x7f34ac0cfe70 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2539804583 0 0) 0x7f348c003520 con 0x7f34ac0cfe70 2024-04-17T08:55:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 msgr2=0x7f34ac0b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.451+0000 7f34a9600640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f34ac0b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 >> v2:172.21.15.187:3300/0 conn(0x7f34ac15eaf0 msgr2=0x7f34ac0d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 --2- 172.21.15.67:0/1527532465 >> v2:172.21.15.187:3300/0 conn(0x7f34ac15eaf0 0x7f34ac0d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac0cf070 con 0x7f34ac0cfe70 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f34a4001670 con 0x7f34ac0cfe70 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 >> v1:172.21.15.67:6789/0 conn(0x7f34ac0cfe70 legacy=0x7f34ac0b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c0039b0 0x7f348c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f348c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f348c004d40 0x7f348c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac0d4610 con 0x7f348c004d40 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f348c00b9c0 con 0x7f348c0039b0 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f348c00bb40 con 0x7f34ac0cfa10 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34ab400640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f348c004d40 0x7f348c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34aaa00640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c0039b0 0x7f348c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34aa000640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f348c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34ab400640 1 -- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 msgr2=0x7f348c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.452+0000 7f34ab400640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac0cfa10 0x7f348c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.453+0000 7f34ab400640 1 -- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c0039b0 msgr2=0x7f348c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.453+0000 7f34ab400640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c0039b0 0x7f348c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.453+0000 7f34ab400640 1 -- 172.21.15.67:0/1527532465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348c003520 con 0x7f348c004d40 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34ab400640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f348c004d40 0x7f348c0051c0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c190 tx=0x7f348c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c75ccf601f0868c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4007c60 con 0x7f348c004d40 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a4030070 con 0x7f348c004d40 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34a9600640 1 -- 172.21.15.67:0/1527532465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4005d30 con 0x7f348c004d40 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34b1a00640 1 -- 172.21.15.67:0/1527532465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f348c004d40 msgr2=0x7f348c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34b1a00640 1 --2- 172.21.15.67:0/1527532465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f348c004d40 0x7f348c0051c0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c190 tx=0x7f348c005830 comp rx=0 tx=0).stop 2024-04-17T08:55:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.454+0000 7f34ab400640 1 -- 172.21.15.67:0/1527532465 reap_dead start 2024-04-17T08:55:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.455+0000 7f34b1a00640 1 -- 172.21.15.67:0/1527532465 shutdown_connections 2024-04-17T08:55:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.455+0000 7f34b1a00640 1 -- 172.21.15.67:0/1527532465 >> 172.21.15.67:0/1527532465 conn(0x7f34ac067d10 msgr2=0x7f34ac0d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.455+0000 7f34b1a00640 1 -- 172.21.15.67:0/1527532465 shutdown_connections 2024-04-17T08:55:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.455+0000 7f34b1a00640 1 -- 172.21.15.67:0/1527532465 wait complete. 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.456+0000 7f34b1a00640 1 Processor -- start 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 -- start start 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c004d40 0x7f34ac1f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac15d770 0x7f34ac15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac0d4000 con 0x7f34ac143fc0 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac0cf630 con 0x7f348c004d40 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.457+0000 7f34b1a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f34ac15eed0 con 0x7f34ac15d770 2024-04-17T08:55:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43246/0 (socket says 172.21.15.67:43246) 2024-04-17T08:55:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 -- 172.21.15.67:0/146114304 learned_addr learned my addr 172.21.15.67:0/146114304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aaa00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c004d40 0x7f34ac1f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34ab400640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac15d770 0x7f34ac15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 -- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac15d770 msgr2=0x7f34ac15dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac15d770 0x7f34ac15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 -- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c004d40 msgr2=0x7f34ac1f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c004d40 0x7f34ac1f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.458+0000 7f34aa000640 1 -- 172.21.15.67:0/146114304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac0c8510 con 0x7f34ac143fc0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34aa000640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f3494007f60 tx=0x7f3494007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349401a020 con 0x7f34ac143fc0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3494016470 con 0x7f34ac143fc0 2024-04-17T08:55:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3494016860 con 0x7f34ac143fc0 2024-04-17T08:55:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac0b4cf0 con 0x7f34ac143fc0 2024-04-17T08:55:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.459+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac0b52b0 con 0x7f34ac143fc0 2024-04-17T08:55:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.460+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac06e090 con 0x7f34ac143fc0 2024-04-17T08:55:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.461+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 162) ==== 91324+0+0 (secure 0 0 0) 0x7f3494005d10 con 0x7f34ac143fc0 2024-04-17T08:55:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.462+0000 7f34a2a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 0x7f348806f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.462+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f349408d1c0 con 0x7f34ac143fc0 2024-04-17T08:55:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.463+0000 7f34aaa00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 0x7f348806f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.464+0000 7f34aaa00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 0x7f348806f100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f349c009a90 tx=0x7f349c008040 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.467+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f3494024030 con 0x7f34ac143fc0 2024-04-17T08:55:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.696+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "subvol": "sv01", "path": ".", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f34ac068d60 con 0x7f348806cc40 2024-04-17T08:55:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.926+0000 7f34a2a00640 1 -- 172.21.15.67:0/146114304 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+89 (secure 0 0 0) 0x7f34ac068d60 con 0x7f348806cc40 2024-04-17T08:55:00.926 INFO:teuthology.orchestra.run.smithi067.stdout:Schedule removed for path /volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/.. 2024-04-17T08:55:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.929+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 msgr2=0x7f348806f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.929+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 0x7f348806f100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f349c009a90 tx=0x7f349c008040 comp rx=0 tx=0).stop 2024-04-17T08:55:00.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.929+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 msgr2=0x7f34ac0b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.929+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f3494007f60 tx=0x7f3494007f90 comp rx=0 tx=0).stop 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 shutdown_connections 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f348806cc40 0x7f348806f100 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f34ac15d770 0x7f34ac15dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac143fc0 0x7f34ac0b4410 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 --2- 172.21.15.67:0/146114304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f348c004d40 0x7f34ac1f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 >> 172.21.15.67:0/146114304 conn(0x7f34ac067d10 msgr2=0x7f34ac0b9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 shutdown_connections 2024-04-17T08:55:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:00.930+0000 7f34b1a00640 1 -- 172.21.15.67:0/146114304 wait complete. 2024-04-17T08:55:00.953 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /volumes/_nogroup/sv01/e1708f23-c981-46c3-bf65-c114e6a69770/.. 2024-04-17T08:55:00.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01 2024-04-17T08:55:01.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.167+0000 7f917dc00640 1 Processor -- start 2024-04-17T08:55:01.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.167+0000 7f917dc00640 1 -- start start 2024-04-17T08:55:01.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.169+0000 7f917dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 0x7f91780cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.169+0000 7f917dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f917806c270 0x7f917806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.169+0000 7f917dc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9178146360 con 0x7f91780cd750 2024-04-17T08:55:01.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.169+0000 7f917dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91780cdbb0 con 0x7f917806c270 2024-04-17T08:55:01.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.169+0000 7f917dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91780727a0 con 0x7f9178145f80 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.170+0000 7f917d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f917806c270 0x7f917806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.170+0000 7f917d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f917806c270 0x7f917806e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43252/0 (socket says 172.21.15.67:43252) 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.170+0000 7f917d200640 1 -- 172.21.15.67:0/840749786 learned_addr learned my addr 172.21.15.67:0/840749786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.170+0000 7f9177e00640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 0x7f91780cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.171+0000 7f917d200640 1 -- 172.21.15.67:0/840749786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 msgr2=0x7f91780cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.171+0000 7f917d200640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 0x7f91780cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.171+0000 7f917d200640 1 -- 172.21.15.67:0/840749786 >> v1:172.21.15.187:6789/0 conn(0x7f91780cd750 legacy=0x7f917806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.171+0000 7f917d200640 1 -- 172.21.15.67:0/840749786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9178071e70 con 0x7f917806c270 2024-04-17T08:55:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.171+0000 7f9177e00640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 0x7f91780cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.172+0000 7f917d200640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f917806c270 0x7f917806e1c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f9170009e50 tx=0x7f917002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb8624a659ae800 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.172+0000 7f9177400640 1 -- 172.21.15.67:0/840749786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917002ca70 con 0x7f917806c270 2024-04-17T08:55:01.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.172+0000 7f9177400640 1 -- 172.21.15.67:0/840749786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917002cc10 con 0x7f917806c270 2024-04-17T08:55:01.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.172+0000 7f9177400640 1 -- 172.21.15.67:0/840749786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91700343f0 con 0x7f917806c270 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 -- 172.21.15.67:0/840749786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f917806c270 msgr2=0x7f917806e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f917806c270 0x7f917806e1c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f9170009e50 tx=0x7f917002bed0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 -- 172.21.15.67:0/840749786 shutdown_connections 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f917806c270 0x7f917806e1c0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 --2- 172.21.15.67:0/840749786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9178145f80 0x7f91780cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 -- 172.21.15.67:0/840749786 >> 172.21.15.67:0/840749786 conn(0x7f91780b2190 msgr2=0x7f91780b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 -- 172.21.15.67:0/840749786 shutdown_connections 2024-04-17T08:55:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.173+0000 7f917dc00640 1 -- 172.21.15.67:0/840749786 wait complete. 2024-04-17T08:55:01.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.176+0000 7f917dc00640 1 Processor -- start 2024-04-17T08:55:01.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.176+0000 7f917dc00640 1 -- start start 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f917806c270 0x7f91780ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91780cd750 0x7f91780a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178145f80 0x7f91780b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9178072c30 con 0x7f9178145f80 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9178072430 con 0x7f917806c270 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9178072db0 con 0x7f91780cd750 2024-04-17T08:55:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.177+0000 7f917d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178145f80 0x7f91780b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178145f80 0x7f91780b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43262/0 (socket says 172.21.15.67:43262) 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 -- 172.21.15.67:0/863939766 learned_addr learned my addr 172.21.15.67:0/863939766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f9177e00640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f917806c270 0x7f91780ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f916fe00640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91780cd750 0x7f91780a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 -- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91780cd750 msgr2=0x7f91780a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91780cd750 0x7f91780a9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 -- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f917806c270 msgr2=0x7f91780ab080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f917806c270 0x7f91780ab080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 -- 172.21.15.67:0/863939766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9178131a40 con 0x7f9178145f80 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.178+0000 7f917d200640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9178145f80 0x7f91780b0b50 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f9170036040 tx=0x7f9170004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.179+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170043020 con 0x7f9178145f80 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.179+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917003e450 con 0x7f9178145f80 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.179+0000 7f917dc00640 1 -- 172.21.15.67:0/863939766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9178131d20 con 0x7f9178145f80 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.179+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9170009510 con 0x7f9178145f80 2024-04-17T08:55:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.179+0000 7f917dc00640 1 -- 172.21.15.67:0/863939766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91781322e0 con 0x7f9178145f80 2024-04-17T08:55:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.180+0000 7f917dc00640 1 -- 172.21.15.67:0/863939766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9178068a10 con 0x7f9178145f80 2024-04-17T08:55:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.181+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 162) ==== 91324+0+0 (secure 0 0 0) 0x7f9170046020 con 0x7f9178145f80 2024-04-17T08:55:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.182+0000 7f9174c00640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f915406cc10 0x7f915406f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.182+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f91700aea10 con 0x7f9178145f80 2024-04-17T08:55:01.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.183+0000 7f9177e00640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f915406cc10 0x7f915406f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.184+0000 7f9177e00640 1 --2- 172.21.15.67:0/863939766 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f915406cc10 0x7f915406f0d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9168006fd0 tx=0x7f9168008040 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.187+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f917004d030 con 0x7f9178145f80 2024-04-17T08:55:01.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.420+0000 7f917dc00640 1 -- 172.21.15.67:0/863939766 --> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f91780cf950 con 0x7f915406cc10 2024-04-17T08:55:01.438 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:01.439+0000 7f227e400640 -1 mgr.server reply reply (39) Directory not empty subvolume 'sv01' has snapshots 2024-04-17T08:55:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.440+0000 7f9174c00640 1 -- 172.21.15.67:0/863939766 <== mgr.10117 v2:172.21.15.187:6832/1576025869 1 ==== mgr_command_reply(tid 0: -39 subvolume 'sv01' has snapshots) ==== 38+0+0 (secure 0 0 0) 0x7f91780cf950 con 0x7f915406cc10 2024-04-17T08:55:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOTEMPTY: subvolume 'sv01' has snapshots 2024-04-17T08:55:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 39 2024-04-17T08:55:01.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-04-17T08:55:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.685+0000 7f5253200640 1 Processor -- start 2024-04-17T08:55:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.685+0000 7f5253200640 1 -- start start 2024-04-17T08:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c145f80 0x7f524c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f524c0cd750 0x7f524c06bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f524c06c0a0 0x7f524c160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0cdbb0 con 0x7f524c06c0a0 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f524c0727a0 con 0x7f524c0cd750 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5253200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f524c146360 con 0x7f524c145f80 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5252800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f524c06c0a0 0x7f524c160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5252800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f524c06c0a0 0x7f524c160200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43264/0 (socket says 172.21.15.67:43264) 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.687+0000 7f5252800640 1 -- 172.21.15.67:0/1603362879 learned_addr learned my addr 172.21.15.67:0/1603362879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5251e00640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c145f80 0x7f524c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5251400640 1 --2- 172.21.15.67:0/1603362879 >> v2:172.21.15.187:3300/0 conn(0x7f524c0cd750 0x7f524c06bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5252800640 1 -- 172.21.15.67:0/1603362879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c145f80 msgr2=0x7f524c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5252800640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c145f80 0x7f524c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5252800640 1 -- 172.21.15.67:0/1603362879 >> v2:172.21.15.187:3300/0 conn(0x7f524c0cd750 msgr2=0x7f524c06bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5252800640 1 --2- 172.21.15.67:0/1603362879 >> v2:172.21.15.187:3300/0 conn(0x7f524c0cd750 0x7f524c06bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.688+0000 7f5252800640 1 -- 172.21.15.67:0/1603362879 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c071e70 con 0x7f524c06c0a0 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.689+0000 7f5252800640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c160200 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f524c0b2f50 tx=0x7f524802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcbce10335d14f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.689+0000 7f5250a00640 1 -- 172.21.15.67:0/1603362879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524802c9e0 con 0x7f524c06c0a0 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.689+0000 7f5250a00640 1 -- 172.21.15.67:0/1603362879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524802cb80 con 0x7f524c06c0a0 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.689+0000 7f5250a00640 1 -- 172.21.15.67:0/1603362879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52480343f0 con 0x7f524c06c0a0 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 -- 172.21.15.67:0/1603362879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 msgr2=0x7f524c160200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c160200 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f524c0b2f50 tx=0x7f524802be40 comp rx=0 tx=0).stop 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 -- 172.21.15.67:0/1603362879 shutdown_connections 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c160200 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 --2- 172.21.15.67:0/1603362879 >> v2:172.21.15.187:3300/0 conn(0x7f524c0cd750 0x7f524c06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 --2- 172.21.15.67:0/1603362879 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c145f80 0x7f524c0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 -- 172.21.15.67:0/1603362879 >> 172.21.15.67:0/1603362879 conn(0x7f524c0b2190 msgr2=0x7f524c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.690+0000 7f5253200640 1 -- 172.21.15.67:0/1603362879 shutdown_connections 2024-04-17T08:55:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.691+0000 7f5253200640 1 -- 172.21.15.67:0/1603362879 wait complete. 2024-04-17T08:55:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.692+0000 7f5253200640 1 Processor -- start 2024-04-17T08:55:01.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.693+0000 7f5253200640 1 -- start start 2024-04-17T08:55:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.693+0000 7f5253200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.693+0000 7f5253200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c0cd750 0x7f524c15bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.693+0000 7f5253200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f524c145f80 0x7f524c1f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5253200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f524c072c30 con 0x7f524c06c0a0 2024-04-17T08:55:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5253200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f524c072430 con 0x7f524c145f80 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5253200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f524c072db0 con 0x7f524c0cd750 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43276/0 (socket says 172.21.15.67:43276) 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 -- 172.21.15.67:0/1336902193 learned_addr learned my addr 172.21.15.67:0/1336902193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251400640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c0cd750 0x7f524c15bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 -- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c0cd750 msgr2=0x7f524c15bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c0cd750 0x7f524c15bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 -- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f524c145f80 msgr2=0x7f524c1f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f524c145f80 0x7f524c1f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.694+0000 7f5251e00640 1 -- 172.21.15.67:0/1336902193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524c1f52a0 con 0x7f524c06c0a0 2024-04-17T08:55:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5251e00640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f524000c7f0 tx=0x7f524000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240015070 con 0x7f524c06c0a0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5240004220 con 0x7f524c06c0a0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524c1f5580 con 0x7f524c06c0a0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240007620 con 0x7f524c06c0a0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.695+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f524c1f8080 con 0x7f524c06c0a0 2024-04-17T08:55:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.696+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f524c0cc6a0 con 0x7f524c06c0a0 2024-04-17T08:55:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.697+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 162) ==== 91324+0+0 (secure 0 0 0) 0x7f52400079d0 con 0x7f524c06c0a0 2024-04-17T08:55:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.698+0000 7f5246000640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 0x7f523006f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.698+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f524008c2b0 con 0x7f524c06c0a0 2024-04-17T08:55:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.699+0000 7f5251400640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 0x7f523006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.700+0000 7f5251400640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 0x7f523006f100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f524c15ded0 tx=0x7f523c008040 comp rx=0 tx=0).ready entity=mgr.10117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.703+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f524c0cc6a0 con 0x7f524c06c0a0 2024-04-17T08:55:01.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:01.963+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f524c0cec90 con 0x7f524c06c0a0 2024-04-17T08:55:02.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.523+0000 7f5246000640 1 -- 172.21.15.67:0/1336902193 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v163) ==== 93+0+0 (secure 0 0 0) 0x7f524c0cec90 con 0x7f524c06c0a0 2024-04-17T08:55:02.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 msgr2=0x7f523006f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:02.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 0x7f523006f100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f524c15ded0 tx=0x7f523c008040 comp rx=0 tx=0).stop 2024-04-17T08:55:02.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 msgr2=0x7f524c1f8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:02.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f524000c7f0 tx=0x7f524000ccb0 comp rx=0 tx=0).stop 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 shutdown_connections 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f523006cc40 0x7f523006f100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f524c145f80 0x7f524c1f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f524c0cd750 0x7f524c15bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 --2- 172.21.15.67:0/1336902193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f524c06c0a0 0x7f524c1f8880 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:02.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.527+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 >> 172.21.15.67:0/1336902193 conn(0x7f524c0b2190 msgr2=0x7f524c15cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:02.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.528+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 shutdown_connections 2024-04-17T08:55:02.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:02.529+0000 7f5253200640 1 -- 172.21.15.67:0/1336902193 wait complete. 2024-04-17T08:55:02.554 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T08:55:02.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:55:02.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T08:55:02.786 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T08:55:02.786 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:55:02.786 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:55:02.786 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:02.839 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:55:02.839 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:02.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:55:03.098 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.098+0000 7f1720a0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:55:03.106 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.107+0000 7f7e9ce32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T08:55:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.124+0000 7f1223a00640 1 Processor -- start 2024-04-17T08:55:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.124+0000 7f1223a00640 1 -- start start 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1223a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f121c15ed10 0x7f121c0d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1223a00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f121c0d2ab0 0x7f121c0b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1223a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121c0d3020 con 0x7f121c0cf640 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1223a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f121c0d3c30 con 0x7f121c15ed10 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1223a00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f121c0d3300 con 0x7f121c0d2ab0 2024-04-17T08:55:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f121c15ed10 0x7f121c0d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f121c15ed10 0x7f121c0d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43304/0 (socket says 172.21.15.67:43304) 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 -- 172.21.15.67:0/2412336785 learned_addr learned my addr 172.21.15.67:0/2412336785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f1221a00640 1 --2- 172.21.15.67:0/2412336785 >> v2:172.21.15.187:3300/0 conn(0x7f121c0d2ab0 0x7f121c0b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 -- 172.21.15.67:0/2412336785 >> v2:172.21.15.187:3300/0 conn(0x7f121c0d2ab0 msgr2=0x7f121c0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 --2- 172.21.15.67:0/2412336785 >> v2:172.21.15.187:3300/0 conn(0x7f121c0d2ab0 0x7f121c0b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 -- 172.21.15.67:0/2412336785 >> v1:172.21.15.67:6789/0 conn(0x7f121c0cf640 legacy=0x7f121c0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.126+0000 7f121be00640 1 -- 172.21.15.67:0/2412336785 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c0d35e0 con 0x7f121c15ed10 2024-04-17T08:55:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.127+0000 7f121be00640 1 --2- 172.21.15.67:0/2412336785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c15ed10 0x7f121c0d40c0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f121c0d12d0 tx=0x7f120802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfb83f9cb3d729f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.128+0000 7f121b400640 1 -- 172.21.15.67:0/2412336785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120802f850 con 0x7f121c15ed10 2024-04-17T08:55:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.128+0000 7f121b400640 1 -- 172.21.15.67:0/2412336785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f120802f9f0 con 0x7f121c15ed10 2024-04-17T08:55:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.128+0000 7f121b400640 1 -- 172.21.15.67:0/2412336785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120802fdc0 con 0x7f121c15ed10 2024-04-17T08:55:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.128+0000 7f1223a00640 1 -- 172.21.15.67:0/2412336785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c15ed10 msgr2=0x7f121c0d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 --2- 172.21.15.67:0/2412336785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c15ed10 0x7f121c0d40c0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f121c0d12d0 tx=0x7f120802ba30 comp rx=0 tx=0).stop 2024-04-17T08:55:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 -- 172.21.15.67:0/2412336785 shutdown_connections 2024-04-17T08:55:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 --2- 172.21.15.67:0/2412336785 >> v2:172.21.15.187:3300/0 conn(0x7f121c0d2ab0 0x7f121c0b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 --2- 172.21.15.67:0/2412336785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c15ed10 0x7f121c0d40c0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 -- 172.21.15.67:0/2412336785 >> 172.21.15.67:0/2412336785 conn(0x7f121c06cbd0 msgr2=0x7f121c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 -- 172.21.15.67:0/2412336785 shutdown_connections 2024-04-17T08:55:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.129+0000 7f1223a00640 1 -- 172.21.15.67:0/2412336785 wait complete. 2024-04-17T08:55:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.130+0000 7f1223a00640 1 Processor -- start 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.131+0000 7f1223a00640 1 -- start start 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.131+0000 7f1223a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 0x7f121c15a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.131+0000 7f1223a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.131+0000 7f1223a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f121c15ed10 0x7f121c15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1223a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f121c0d4600 con 0x7f121c0d2ab0 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1223a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f121c0d25b0 con 0x7f121c0cf640 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1223a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f121c0d4780 con 0x7f121c15ed10 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43312/0 (socket says 172.21.15.67:43312) 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 -- 172.21.15.67:0/2446464194 learned_addr learned my addr 172.21.15.67:0/2446464194 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1221a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f121c15ed10 0x7f121c15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1221000640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 0x7f121c15a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f121c15ed10 msgr2=0x7f121c15c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f121c15ed10 0x7f121c15c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 msgr2=0x7f121c15a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 0x7f121c15a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f121be00640 1 -- 172.21.15.67:0/2446464194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c15ad40 con 0x7f121c0d2ab0 2024-04-17T08:55:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.132+0000 7f1221000640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 0x7f121c15a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f121be00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f120802ba70 tx=0x7f1208004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208041400 con 0x7f121c0d2ab0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1208046070 con 0x7f121c0d2ab0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12080417e0 con 0x7f121c0d2ab0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121c15a380 con 0x7f121c0d2ab0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.133+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121c155c10 con 0x7f121c0d2ab0 2024-04-17T08:55:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.135+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121c068b90 con 0x7f121c0d2ab0 2024-04-17T08:55:03.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.135+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7f1208040070 con 0x7f121c0d2ab0 2024-04-17T08:55:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.136+0000 7f1218c00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 0x7f11fc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.136+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f12080aead0 con 0x7f121c0d2ab0 2024-04-17T08:55:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.137+0000 7f1221000640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 msgr2=0x7f11fc06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.137+0000 7f1221000640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 0x7f11fc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.141+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f1208033030 con 0x7f121c0d2ab0 2024-04-17T08:55:03.154 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.154+0000 7f1720a0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:55:03.166 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.167+0000 7f7e9ce32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T08:55:03.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.338+0000 7f1221000640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 msgr2=0x7f11fc06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:03.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.338+0000 7f1221000640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 0x7f11fc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:03.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.345+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f121c0d42e0 con 0x7f121c0d2ab0 2024-04-17T08:55:03.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.348+0000 7f1218c00640 1 -- 172.21.15.67:0/2446464194 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v289) ==== 74+0+17232 (secure 0 0 0) 0x7f120807c9a0 con 0x7f121c0d2ab0 2024-04-17T08:55:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:03.348 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":289,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:54:43.367150+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.14","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2687160515":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3174473408":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/740627636":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/492269958":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3820292707":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6833/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3797546006":"2024-04-18T08:53:42.064401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 msgr2=0x7f11fc06f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 0x7f11fc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 msgr2=0x7f121c1f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f120802ba70 tx=0x7f1208004730 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 shutdown_connections 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f11fc06cbc0 0x7f11fc06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f121c15ed10 0x7f121c15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f121c0d2ab0 0x7f121c1f48b0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 --2- 172.21.15.67:0/2446464194 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f121c0cf640 0x7f121c15a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 >> 172.21.15.67:0/2446464194 conn(0x7f121c06cbd0 msgr2=0x7f121c15dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 shutdown_connections 2024-04-17T08:55:03.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.351+0000 7f1223a00640 1 -- 172.21.15.67:0/2446464194 wait complete. 2024-04-17T08:55:03.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:53:25.119464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:53:26.202816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:55:03.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:55:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.582+0000 7fac47200640 1 Processor -- start 2024-04-17T08:55:03.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.582+0000 7fac47200640 1 -- start start 2024-04-17T08:55:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.583+0000 7fac47200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac480d2ff0 0x7fac480b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.583+0000 7fac47200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac480b4c70 0x7fac48161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.583+0000 7fac47200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac480b5110 con 0x7fac480d33d0 2024-04-17T08:55:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.583+0000 7fac47200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fac480b53f0 con 0x7fac480b4c70 2024-04-17T08:55:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.583+0000 7fac47200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fac480b56d0 con 0x7fac480d2ff0 2024-04-17T08:55:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac480b4c70 0x7fac48161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac480b4c70 0x7fac48161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51686/0 (socket says 172.21.15.67:51686) 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 -- 172.21.15.67:0/418787368 learned_addr learned my addr 172.21.15.67:0/418787368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac45e00640 1 --2- 172.21.15.67:0/418787368 >> v2:172.21.15.187:3300/0 conn(0x7fac480d2ff0 0x7fac480b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 -- 172.21.15.67:0/418787368 >> v2:172.21.15.187:3300/0 conn(0x7fac480d2ff0 msgr2=0x7fac480b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 --2- 172.21.15.67:0/418787368 >> v2:172.21.15.187:3300/0 conn(0x7fac480d2ff0 0x7fac480b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 -- 172.21.15.67:0/418787368 >> v1:172.21.15.67:6789/0 conn(0x7fac480d33d0 legacy=0x7fac480d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.584+0000 7fac3fe00640 1 -- 172.21.15.67:0/418787368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac480cf340 con 0x7fac480b4c70 2024-04-17T08:55:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.585+0000 7fac3fe00640 1 --2- 172.21.15.67:0/418787368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac48161ba0 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fac480b7370 tx=0x7fac3002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ee5e8739486efc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac45400640 1 -- 172.21.15.67:0/418787368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30034070 con 0x7fac480b4c70 2024-04-17T08:55:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac45400640 1 -- 172.21.15.67:0/418787368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac30030860 con 0x7fac480b4c70 2024-04-17T08:55:03.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac45400640 1 -- 172.21.15.67:0/418787368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30030c30 con 0x7fac480b4c70 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 -- 172.21.15.67:0/418787368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 msgr2=0x7fac48161ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 --2- 172.21.15.67:0/418787368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac48161ba0 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fac480b7370 tx=0x7fac3002b9a0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 -- 172.21.15.67:0/418787368 shutdown_connections 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 --2- 172.21.15.67:0/418787368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac48161ba0 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 --2- 172.21.15.67:0/418787368 >> v2:172.21.15.187:3300/0 conn(0x7fac480d2ff0 0x7fac480b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.586+0000 7fac47200640 1 -- 172.21.15.67:0/418787368 >> 172.21.15.67:0/418787368 conn(0x7fac4809da80 msgr2=0x7fac4808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.587+0000 7fac47200640 1 -- 172.21.15.67:0/418787368 shutdown_connections 2024-04-17T08:55:03.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.587+0000 7fac47200640 1 -- 172.21.15.67:0/418787368 wait complete. 2024-04-17T08:55:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.589+0000 7fac47200640 1 Processor -- start 2024-04-17T08:55:03.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.589+0000 7fac47200640 1 -- start start 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 0x7fac480ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 0x7fac480cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac480d0dd0 con 0x7fac480b4c70 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac480d0c50 con 0x7fac480d33d0 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac47200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac480d0f50 con 0x7fac480d2ff0 2024-04-17T08:55:03.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51694/0 (socket says 172.21.15.67:51694) 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.591+0000 7fac45e00640 1 -- 172.21.15.67:0/2804572331 learned_addr learned my addr 172.21.15.67:0/2804572331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac46800640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 0x7fac480cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 0x7fac480ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 msgr2=0x7fac480ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 0x7fac480ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 msgr2=0x7fac480cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 0x7fac480cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 -- 172.21.15.67:0/2804572331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac480cd8e0 con 0x7fac480b4c70 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac46800640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 0x7fac480cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 0x7fac480ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:03.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac45e00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d5f0 tx=0x7fac3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac380125f0 con 0x7fac480b4c70 2024-04-17T08:55:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac38019070 con 0x7fac480b4c70 2024-04-17T08:55:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac480cdbc0 con 0x7fac480b4c70 2024-04-17T08:55:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.592+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac380129d0 con 0x7fac480b4c70 2024-04-17T08:55:03.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.593+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac480ce180 con 0x7fac480b4c70 2024-04-17T08:55:03.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.594+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac480755d0 con 0x7fac480b4c70 2024-04-17T08:55:03.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.595+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7fac38012b70 con 0x7fac480b4c70 2024-04-17T08:55:03.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.596+0000 7fac3e000640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 0x7fac2406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:03.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.596+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7fac3808c540 con 0x7fac480b4c70 2024-04-17T08:55:03.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.597+0000 7fac3fe00640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 msgr2=0x7fac2406f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:03.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.597+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 0x7fac2406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:03.597 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.598+0000 7f1720a0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:55:03.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.600+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fac3805a4e0 con 0x7fac480b4c70 2024-04-17T08:55:03.642 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.643+0000 7f7e9ce32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T08:55:03.649 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.650+0000 7f1720a0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:55:03.698 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.698+0000 7f7e9ce32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T08:55:03.700 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.701+0000 7f1720a0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:55:03.751 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.751+0000 7f7e9ce32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T08:55:03.765 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.766+0000 7f1720a0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:55:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.798+0000 7fac3fe00640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 msgr2=0x7fac2406f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.798+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 0x7fac2406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.807+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac4808c1d0 con 0x7fac480b4c70 2024-04-17T08:55:03.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.808+0000 7fac3e000640 1 -- 172.21.15.67:0/2804572331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v289) ==== 93+0+10 (secure 0 0 0) 0x7fac38059de0 con 0x7fac480b4c70 2024-04-17T08:55:03.808 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 msgr2=0x7fac2406f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 0x7fac2406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 msgr2=0x7fac481f7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d5f0 tx=0x7fac3800dab0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 shutdown_connections 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fac2406cbc0 0x7fac2406f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480d33d0 0x7fac480cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac480d2ff0 0x7fac480ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 --2- 172.21.15.67:0/2804572331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac480b4c70 0x7fac481f7790 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 >> 172.21.15.67:0/2804572331 conn(0x7fac4809da80 msgr2=0x7fac4815ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.811+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 shutdown_connections 2024-04-17T08:55:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:03.812+0000 7fac47200640 1 -- 172.21.15.67:0/2804572331 wait complete. 2024-04-17T08:55:03.816 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.817+0000 7f1720a0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:55:03.820 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.821+0000 7f7e9ce32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T08:55:03.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:55:03.874 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:03.874+0000 7f7e9ce32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T08:55:03.945 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:03.946+0000 7f1720a0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:55:04.008 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.008+0000 7f1720a0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:55:04.011 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.012+0000 7f7e9ce32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-17T08:55:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.031+0000 7f7ea7200640 1 Processor -- start 2024-04-17T08:55:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.032+0000 7f7ea7200640 1 -- start start 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea806b8c0 0x7f7ea80c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ea815fd30 0x7f7ea806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7ea815ed10 0x7f7ea806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ea80c8110 con 0x7f7ea815fd30 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7ea815f250 con 0x7f7ea815ed10 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea815f530 con 0x7f7ea806b8c0 2024-04-17T08:55:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ea815fd30 0x7f7ea806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ea815fd30 0x7f7ea806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51700/0 (socket says 172.21.15.67:51700) 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.033+0000 7f7ea5400640 1 -- 172.21.15.67:0/4032938792 learned_addr learned my addr 172.21.15.67:0/4032938792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 -- 172.21.15.67:0/4032938792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea806b8c0 msgr2=0x7f7ea80c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 --2- 172.21.15.67:0/4032938792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea806b8c0 0x7f7ea80c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 -- 172.21.15.67:0/4032938792 >> v2:172.21.15.187:3300/0 conn(0x7f7ea815ed10 msgr2=0x7f7ea806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 --2- 172.21.15.67:0/4032938792 >> v2:172.21.15.187:3300/0 conn(0x7f7ea815ed10 0x7f7ea806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 -- 172.21.15.67:0/4032938792 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea815f810 con 0x7f7ea815fd30 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea5400640 1 --2- 172.21.15.67:0/4032938792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea815fd30 0x7f7ea806bca0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8073fb0 tx=0x7f7e9002b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81681669abc9b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea4a00640 1 -- 172.21.15.67:0/4032938792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90031760 con 0x7f7ea815fd30 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea4a00640 1 -- 172.21.15.67:0/4032938792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e90037070 con 0x7f7ea815fd30 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.035+0000 7f7ea4a00640 1 -- 172.21.15.67:0/4032938792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90031b20 con 0x7f7ea815fd30 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 -- 172.21.15.67:0/4032938792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea815fd30 msgr2=0x7f7ea806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 --2- 172.21.15.67:0/4032938792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea815fd30 0x7f7ea806bca0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8073fb0 tx=0x7f7e9002b800 comp rx=0 tx=0).stop 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 -- 172.21.15.67:0/4032938792 shutdown_connections 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 --2- 172.21.15.67:0/4032938792 >> v2:172.21.15.187:3300/0 conn(0x7f7ea815ed10 0x7f7ea806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 --2- 172.21.15.67:0/4032938792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea815fd30 0x7f7ea806bca0 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 --2- 172.21.15.67:0/4032938792 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea806b8c0 0x7f7ea80c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 -- 172.21.15.67:0/4032938792 >> 172.21.15.67:0/4032938792 conn(0x7f7ea80b2fe0 msgr2=0x7f7ea80b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 -- 172.21.15.67:0/4032938792 shutdown_connections 2024-04-17T08:55:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.036+0000 7f7ea7200640 1 -- 172.21.15.67:0/4032938792 wait complete. 2024-04-17T08:55:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.038+0000 7f7ea7200640 1 Processor -- start 2024-04-17T08:55:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.038+0000 7f7ea7200640 1 -- start start 2024-04-17T08:55:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea815ed10 0x7f7ea80b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ea815fd30 0x7f7ea80b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea80cdd90 con 0x7f7ea806b8c0 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea80cdf10 con 0x7f7ea815fd30 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea7200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea80ce090 con 0x7f7ea815ed10 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51704/0 (socket says 172.21.15.67:51704) 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea5e00640 1 -- 172.21.15.67:0/734300143 learned_addr learned my addr 172.21.15.67:0/734300143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.039+0000 7f7ea5400640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea815ed10 0x7f7ea80b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea815ed10 msgr2=0x7f7ea80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea815ed10 0x7f7ea80b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ea815fd30 msgr2=0x7f7ea80b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ea815fd30 0x7f7ea80b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 -- 172.21.15.67:0/734300143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea80b4be0 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea5e00640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800b4a0 tx=0x7f7e9800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98014070 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e98010610 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98010a00 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea80b4ec0 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.040+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea80b5480 con 0x7f7ea806b8c0 2024-04-17T08:55:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.042+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea80755d0 con 0x7f7ea806b8c0 2024-04-17T08:55:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.043+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7f7e98017070 con 0x7f7ea806b8c0 2024-04-17T08:55:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.043+0000 7f7e9e000640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 0x7f7e8406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.044+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f7e9808bc20 con 0x7f7ea806b8c0 2024-04-17T08:55:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.044+0000 7f7ea5400640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 msgr2=0x7f7e8406f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.044+0000 7f7ea5400640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 0x7f7e8406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.052+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f7e98059b30 con 0x7f7ea806b8c0 2024-04-17T08:55:04.059 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.060+0000 7f1720a0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:55:04.084 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.085+0000 7f7e9ce32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T08:55:04.116 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.117+0000 7f1720a0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:55:04.137 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.137+0000 7f7e9ce32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T08:55:04.166 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.167+0000 7f1720a0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:55:04.197 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.197+0000 7f7e9ce32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T08:55:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.245+0000 7f7ea5400640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 msgr2=0x7f7e8406f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:04.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.245+0000 7f7ea5400640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 0x7f7e8406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:04.326 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.250+0000 7f7e9ce32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T08:55:04.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.251+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7ea80b3db0 con 0x7f7ea806b8c0 2024-04-17T08:55:04.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.252+0000 7f7e9e000640 1 -- 172.21.15.67:0/734300143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v289) ==== 104+0+11 (secure 0 0 0) 0x7f7e98059430 con 0x7f7ea806b8c0 2024-04-17T08:55:04.326 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 msgr2=0x7f7e8406f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 0x7f7e8406f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 msgr2=0x7f7ea80b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800b4a0 tx=0x7f7e9800b960 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 shutdown_connections 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f7e8406cbc0 0x7f7e8406f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ea815fd30 0x7f7ea80b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ea815ed10 0x7f7ea80b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 --2- 172.21.15.67:0/734300143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ea806b8c0 0x7f7ea80b65d0 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 >> 172.21.15.67:0/734300143 conn(0x7f7ea80b2fe0 msgr2=0x7f7ea80d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:04.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 shutdown_connections 2024-04-17T08:55:04.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.255+0000 7f7ea7200640 1 -- 172.21.15.67:0/734300143 wait complete. 2024-04-17T08:55:04.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:55:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.548+0000 7fa486600640 1 Processor -- start 2024-04-17T08:55:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.548+0000 7fa486600640 1 -- start start 2024-04-17T08:55:04.548 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.548+0000 7f1720a0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:55:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa48015eb50 0x7fa4800d2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 0x7fa4800d31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa48015ef30 0x7fa4800d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa4800b8160 con 0x7fa48015eb50 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa4800d3880 con 0x7fa4800ba600 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa486600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4800d3b60 con 0x7fa48015ef30 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa47f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa48015eb50 0x7fa4800d2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa47f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa48015eb50 0x7fa4800d2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51718/0 (socket says 172.21.15.67:51718) 2024-04-17T08:55:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.550+0000 7fa47f400640 1 -- 172.21.15.67:0/2176341819 learned_addr learned my addr 172.21.15.67:0/2176341819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47f400640 1 -- 172.21.15.67:0/2176341819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa48015ef30 msgr2=0x7fa4800d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47fe00640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa48015ef30 0x7fa4800d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa477e00640 1 --2- 172.21.15.67:0/2176341819 >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 0x7fa4800d31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47f400640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa48015ef30 0x7fa4800d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47f400640 1 -- 172.21.15.67:0/2176341819 >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 msgr2=0x7fa4800d31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47f400640 1 --2- 172.21.15.67:0/2176341819 >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 0x7fa4800d31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa47f400640 1 -- 172.21.15.67:0/2176341819 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4800d3e40 con 0x7fa48015eb50 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.551+0000 7fa477e00640 1 --2- 172.21.15.67:0/2176341819 >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 0x7fa4800d31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.552+0000 7fa47f400640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015eb50 0x7fa4800d2c70 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa480162860 tx=0x7fa47002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0bec7189737f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.552+0000 7fa47ea00640 1 -- 172.21.15.67:0/2176341819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470035020 con 0x7fa48015eb50 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.552+0000 7fa47ea00640 1 -- 172.21.15.67:0/2176341819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa47002d040 con 0x7fa48015eb50 2024-04-17T08:55:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.552+0000 7fa47ea00640 1 -- 172.21.15.67:0/2176341819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470031a80 con 0x7fa48015eb50 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 -- 172.21.15.67:0/2176341819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015eb50 msgr2=0x7fa4800d2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015eb50 0x7fa4800d2c70 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa480162860 tx=0x7fa47002b900 comp rx=0 tx=0).stop 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 -- 172.21.15.67:0/2176341819 shutdown_connections 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa48015ef30 0x7fa4800d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 --2- 172.21.15.67:0/2176341819 >> v2:172.21.15.187:3300/0 conn(0x7fa4800ba600 0x7fa4800d31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 --2- 172.21.15.67:0/2176341819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015eb50 0x7fa4800d2c70 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 -- 172.21.15.67:0/2176341819 >> 172.21.15.67:0/2176341819 conn(0x7fa48006cc70 msgr2=0x7fa48006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 -- 172.21.15.67:0/2176341819 shutdown_connections 2024-04-17T08:55:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.553+0000 7fa486600640 1 -- 172.21.15.67:0/2176341819 wait complete. 2024-04-17T08:55:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.555+0000 7fa486600640 1 Processor -- start 2024-04-17T08:55:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.555+0000 7fa486600640 1 -- start start 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4800ba600 0x7fa4801f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa48015eb50 0x7fa480156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4800d17d0 con 0x7fa48015ef30 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4800d1650 con 0x7fa48015eb50 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa486600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4800d1950 con 0x7fa4800ba600 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa47fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa47fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51724/0 (socket says 172.21.15.67:51724) 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa47fe00640 1 -- 172.21.15.67:0/2330457253 learned_addr learned my addr 172.21.15.67:0/2330457253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:04.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa47f400640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4800ba600 0x7fa4801f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.556+0000 7fa477e00640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa48015eb50 0x7fa480156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4800ba600 msgr2=0x7fa4801f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4800ba600 0x7fa4801f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa48015eb50 msgr2=0x7fa480156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa48015eb50 0x7fa480156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 -- 172.21.15.67:0/2330457253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4800cbf60 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa47fe00640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa47800cd70 tx=0x7fa47800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478003b90 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa478003d30 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478004240 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.557+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4800cc240 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.558+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4800cc800 con 0x7fa48015ef30 2024-04-17T08:55:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.559+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa480068b90 con 0x7fa48015ef30 2024-04-17T08:55:04.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.560+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7fa4780043e0 con 0x7fa48015ef30 2024-04-17T08:55:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.561+0000 7fa477400640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 0x7fa45c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.561+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7fa47808cbf0 con 0x7fa48015ef30 2024-04-17T08:55:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.562+0000 7fa47f400640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 msgr2=0x7fa45c06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.562+0000 7fa47f400640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 0x7fa45c06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:04.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.566+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa47805aac0 con 0x7fa48015ef30 2024-04-17T08:55:04.660 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.661+0000 7f7e9ce32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-17T08:55:04.709 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.710+0000 7f1720a0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:55:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.763+0000 7fa47f400640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 msgr2=0x7fa45c06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.763+0000 7fa47f400640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 0x7fa45c06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.768+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa4800ced50 con 0x7fa48015ef30 2024-04-17T08:55:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.768+0000 7fa477400640 1 -- 172.21.15.67:0/2330457253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v289) ==== 100+0+11 (secure 0 0 0) 0x7fa47805a3c0 con 0x7fa48015ef30 2024-04-17T08:55:04.768 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 msgr2=0x7fa45c06f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 0x7fa45c06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 msgr2=0x7fa4800cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa47800cd70 tx=0x7fa47800a660 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 shutdown_connections 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa45c06cbc0 0x7fa45c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa48015ef30 0x7fa4800cdbc0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa48015eb50 0x7fa480156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 --2- 172.21.15.67:0/2330457253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4800ba600 0x7fa4801f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 >> 172.21.15.67:0/2330457253 conn(0x7fa48006cc70 msgr2=0x7fa4800d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 shutdown_connections 2024-04-17T08:55:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.771+0000 7fa486600640 1 -- 172.21.15.67:0/2330457253 wait complete. 2024-04-17T08:55:04.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T08:55:04.818 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.818+0000 7f1720a0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:55:04.831 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.832+0000 7f7e9ce32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T08:55:04.868 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.869+0000 7f1720a0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:55:04.947 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:04.947+0000 7f7e9ce32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T08:55:04.994 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:04.994+0000 7f1720a0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:55:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.995+0000 7f6012a00640 1 Processor -- start 2024-04-17T08:55:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.995+0000 7f6012a00640 1 -- start start 2024-04-17T08:55:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c0cea40 0x7f600c166470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c160040 con 0x7f600c0735a0 2024-04-17T08:55:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c15f280 con 0x7f600c155c70 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f600c15f560 con 0x7f600c0cea40 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c0cea40 0x7f600c166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f600c0cea40 0x7f600c166470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51728/0 (socket says 172.21.15.67:51728) 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.997+0000 7f6012000640 1 -- 172.21.15.67:0/1822330235 learned_addr learned my addr 172.21.15.67:0/1822330235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.998+0000 7f6012000640 1 -- 172.21.15.67:0/1822330235 >> v1:172.21.15.187:6789/0 conn(0x7f600c155c70 legacy=0x7f600c072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.998+0000 7f6012000640 1 -- 172.21.15.67:0/1822330235 >> v1:172.21.15.67:6789/0 conn(0x7f600c0735a0 legacy=0x7f600c0ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.998+0000 7f6012000640 1 -- 172.21.15.67:0/1822330235 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c15f840 con 0x7f600c0cea40 2024-04-17T08:55:04.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.999+0000 7f6012000640 1 --2- 172.21.15.67:0/1822330235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cea40 0x7f600c166470 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f6004009e50 tx=0x7f600402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104668f23c0e7e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:04.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.999+0000 7f600be00640 1 -- 172.21.15.67:0/1822330235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600402cb10 con 0x7f600c0cea40 2024-04-17T08:55:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.999+0000 7f600be00640 1 -- 172.21.15.67:0/1822330235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600402ccb0 con 0x7f600c0cea40 2024-04-17T08:55:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:04.999+0000 7f600be00640 1 -- 172.21.15.67:0/1822330235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600402ce60 con 0x7f600c0cea40 2024-04-17T08:55:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 -- 172.21.15.67:0/1822330235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cea40 msgr2=0x7f600c166470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 --2- 172.21.15.67:0/1822330235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cea40 0x7f600c166470 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f6004009e50 tx=0x7f600402bf70 comp rx=0 tx=0).stop 2024-04-17T08:55:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 -- 172.21.15.67:0/1822330235 shutdown_connections 2024-04-17T08:55:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 --2- 172.21.15.67:0/1822330235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0cea40 0x7f600c166470 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 -- 172.21.15.67:0/1822330235 >> 172.21.15.67:0/1822330235 conn(0x7f600c0b33d0 msgr2=0x7f600c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 -- 172.21.15.67:0/1822330235 shutdown_connections 2024-04-17T08:55:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.000+0000 7f6012a00640 1 -- 172.21.15.67:0/1822330235 wait complete. 2024-04-17T08:55:05.001 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.002+0000 7f7e9ce32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T08:55:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.002+0000 7f6012a00640 1 Processor -- start 2024-04-17T08:55:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.002+0000 7f6012a00640 1 -- start start 2024-04-17T08:55:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f600c0cea40 0x7f600c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f600c155c70 0x7f600c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f600c166b30 con 0x7f600c0735a0 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1669b0 con 0x7f600c155c70 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6012a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f600c166cb0 con 0x7f600c0cea40 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6011600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6011600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51744/0 (socket says 172.21.15.67:51744) 2024-04-17T08:55:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.003+0000 7f6011600640 1 -- 172.21.15.67:0/2257578854 learned_addr learned my addr 172.21.15.67:0/2257578854 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6012000640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f600c155c70 0x7f600c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f600c0cea40 msgr2=0x7f600c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f600c0cea40 0x7f600c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f600c155c70 msgr2=0x7f600c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f600c155c70 0x7f600c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 -- 172.21.15.67:0/2257578854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c152260 con 0x7f600c0735a0 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6011600640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f600000c430 tx=0x7f600000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000017070 con 0x7f600c0735a0 2024-04-17T08:55:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60000052d0 con 0x7f600c0735a0 2024-04-17T08:55:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000003830 con 0x7f600c0735a0 2024-04-17T08:55:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600c152540 con 0x7f600c0735a0 2024-04-17T08:55:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.004+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600c152b00 con 0x7f600c0735a0 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.006+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f600c0755d0 con 0x7f600c0735a0 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.007+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7f600001a070 con 0x7f600c0735a0 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.007+0000 7f6009600640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 0x7f5fec06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.008+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f600008b590 con 0x7f600c0735a0 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.008+0000 7f6010c00640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 msgr2=0x7f5fec06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.008+0000 7f6010c00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 0x7f5fec06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.012+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f60000593c0 con 0x7f600c0735a0 2024-04-17T08:55:05.102 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:05.103+0000 7f1720a0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:55:05.136 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.136+0000 7f7e9ce32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:55:05.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.209+0000 7f6010c00640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 msgr2=0x7f5fec06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:05.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.209+0000 7f6010c00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 0x7f5fec06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.228+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f600c1550f0 con 0x7f600c0735a0 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.229+0000 7f6009600640 1 -- 172.21.15.67:0/2257578854 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v289) ==== 80+0+472 (secure 0 0 0) 0x7f6000058cc0 con 0x7f600c0735a0 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/2687160515 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3174473408 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/740627636 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/492269958 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.228 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3820292707 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.229 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6833/46399553 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.229 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:6832/46399553 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.229 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.187:0/3797546006 2024-04-18T08:53:42.064401+0000 2024-04-17T08:55:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:listed 8 entries 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 msgr2=0x7f5fec06f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 0x7f5fec06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 msgr2=0x7f600c148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f600000c430 tx=0x7f600000c8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 shutdown_connections 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f5fec06cbc0 0x7f5fec06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f600c155c70 0x7f600c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f600c0cea40 0x7f600c0b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 --2- 172.21.15.67:0/2257578854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f600c0735a0 0x7f600c148ff0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 >> 172.21.15.67:0/2257578854 conn(0x7f600c0b33d0 msgr2=0x7f600c0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 shutdown_connections 2024-04-17T08:55:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.231+0000 7f6012a00640 1 -- 172.21.15.67:0/2257578854 wait complete. 2024-04-17T08:55:05.248 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T08:55:05.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:05.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T08:55:05.250 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.250+0000 7f7e9ce32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T08:55:05.277 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:05.277+0000 7f1720a0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:55:05.325 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:05.326+0000 7f1720a0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:55:05.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:55:05.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:05.352 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T08:55:05.400 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:55:05.400 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:55:05.400 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:05.440 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.440+0000 7f7e9ce32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T08:55:05.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:55:05.451 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:55:05.452 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:55:05.452 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:05.469 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:05.469+0000 7f1720a0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:55:05.494 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.495+0000 7f7e9ce32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T08:55:05.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:55:05.503 INFO:tasks.cephfs.mount:Terminating background process 2024-04-17T08:55:05.503 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:55:05.503 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:05.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:55:05.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:55:05.643 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:05.643+0000 7f7e9ce32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T08:55:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.766+0000 7fa902e00640 1 Processor -- start 2024-04-17T08:55:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.766+0000 7fa902e00640 1 -- start start 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.767+0000 7fa902e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8fc15b530 0x7fa8fc071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa902e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc072250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa902e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8fc15b910 0x7fa8fc0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa902e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa8fc15f3c0 con 0x7fa8fc15b910 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa902e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa8fc15f660 con 0x7fa8fc15b530 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa902e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8fc15f940 con 0x7fa8fc0c68c0 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8fc15b910 0x7fa8fc0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8fc15b910 0x7fa8fc0c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51758/0 (socket says 172.21.15.67:51758) 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 -- 172.21.15.67:0/2382449785 learned_addr learned my addr 172.21.15.67:0/2382449785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa8fbe00640 1 --2- 172.21.15.67:0/2382449785 >> v2:172.21.15.187:3300/0 conn(0x7fa8fc15b530 0x7fa8fc071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa8fb400640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 -- 172.21.15.67:0/2382449785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 msgr2=0x7fa8fc072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 -- 172.21.15.67:0/2382449785 >> v2:172.21.15.187:3300/0 conn(0x7fa8fc15b530 msgr2=0x7fa8fc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.768+0000 7fa900e00640 1 --2- 172.21.15.67:0/2382449785 >> v2:172.21.15.187:3300/0 conn(0x7fa8fc15b530 0x7fa8fc071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.769+0000 7fa900e00640 1 -- 172.21.15.67:0/2382449785 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc15fc20 con 0x7fa8fc15b910 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.770+0000 7fa900e00640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0c5320 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc06c840 tx=0x7fa8f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70645415d43f67c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.770+0000 7fa8faa00640 1 -- 172.21.15.67:0/2382449785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f402ca70 con 0x7fa8fc15b910 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.770+0000 7fa8faa00640 1 -- 172.21.15.67:0/2382449785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f402cc10 con 0x7fa8fc15b910 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.770+0000 7fa8faa00640 1 -- 172.21.15.67:0/2382449785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f40343f0 con 0x7fa8fc15b910 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 -- 172.21.15.67:0/2382449785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b910 msgr2=0x7fa8fc0c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0c5320 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc06c840 tx=0x7fa8f402bed0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 -- 172.21.15.67:0/2382449785 shutdown_connections 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0c5320 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 --2- 172.21.15.67:0/2382449785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 --2- 172.21.15.67:0/2382449785 >> v2:172.21.15.187:3300/0 conn(0x7fa8fc15b530 0x7fa8fc071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 -- 172.21.15.67:0/2382449785 >> 172.21.15.67:0/2382449785 conn(0x7fa8fc06bf50 msgr2=0x7fa8fc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 -- 172.21.15.67:0/2382449785 shutdown_connections 2024-04-17T08:55:05.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.771+0000 7fa902e00640 1 -- 172.21.15.67:0/2382449785 wait complete. 2024-04-17T08:55:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.773+0000 7fa902e00640 1 Processor -- start 2024-04-17T08:55:05.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.773+0000 7fa902e00640 1 -- start start 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc13ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc0cd030 con 0x7fa8fc15b530 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc0cceb0 con 0x7fa8fc15b910 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa902e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8fc0cd1b0 con 0x7fa8fc0c68c0 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51768/0 (socket says 172.21.15.67:51768) 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fb400640 1 -- 172.21.15.67:0/171671449 learned_addr learned my addr 172.21.15.67:0/171671449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa900e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fbe00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc13ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fb400640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 msgr2=0x7fa8fc13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.774+0000 7fa8fb400640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc13ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8fb400640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8fc15b910 msgr2=0x7fa8fc0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8fb400640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8fb400640 1 -- 172.21.15.67:0/171671449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc0c7550 con 0x7fa8fc15b530 2024-04-17T08:55:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8fb400640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d800 tx=0x7fa8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80096a0 con 0x7fa8fc15b530 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8019070 con 0x7fa8fc15b530 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc0c77f0 con 0x7fa8fc15b530 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8009a60 con 0x7fa8fc15b530 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.775+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8fc0ce250 con 0x7fa8fc15b530 2024-04-17T08:55:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.777+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8fc068b90 con 0x7fa8fc15b530 2024-04-17T08:55:05.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.778+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7fa8e8009c00 con 0x7fa8fc15b530 2024-04-17T08:55:05.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.779+0000 7fa8f3e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 0x7fa8dc06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:05.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.779+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7fa8e808c4e0 con 0x7fa8fc15b530 2024-04-17T08:55:05.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.780+0000 7fa8fbe00640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 msgr2=0x7fa8dc06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:05.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.780+0000 7fa8fbe00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 0x7fa8dc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.784+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa8e805a4c0 con 0x7fa8fc15b530 2024-04-17T08:55:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.981+0000 7fa8fbe00640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 msgr2=0x7fa8dc06f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:05.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:05.981+0000 7fa8fbe00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 0x7fa8dc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.027+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8fc0c6390 con 0x7fa8fc15b530 2024-04-17T08:55:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.028+0000 7fa8f3e00640 1 -- 172.21.15.67:0/171671449 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7fa8e8059dc0 con 0x7fa8fc15b530 2024-04-17T08:55:06.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:06.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:55:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:55:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.030+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 msgr2=0x7fa8dc06f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.030+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 0x7fa8dc06f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.030+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 msgr2=0x7fa8fc15a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.030+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d800 tx=0x7fa8e800dcc0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 shutdown_connections 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7fa8dc06cbc0 0x7fa8dc06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8fc15b910 0x7fa8fc0cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8fc15b530 0x7fa8fc15a910 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 --2- 172.21.15.67:0/171671449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8fc0c68c0 0x7fa8fc13ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 >> 172.21.15.67:0/171671449 conn(0x7fa8fc06bf50 msgr2=0x7fa8fc06dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 shutdown_connections 2024-04-17T08:55:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.031+0000 7fa902e00640 1 -- 172.21.15.67:0/171671449 wait complete. 2024-04-17T08:55:06.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:55:06.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.247+0000 7f4cea200640 1 Processor -- start 2024-04-17T08:55:06.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.247+0000 7f4cea200640 1 -- start start 2024-04-17T08:55:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.248+0000 7f4cea200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ce40cd020 0x7f4ce40cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.248+0000 7f4cea200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 0x7f4ce406e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.248+0000 7f4cea200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce40ceb00 con 0x7f4ce406f670 2024-04-17T08:55:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.248+0000 7f4cea200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce40cede0 con 0x7f4ce40cd020 2024-04-17T08:55:06.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.248+0000 7f4cea200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce40cf0c0 con 0x7f4ce406e020 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ce40cd020 0x7f4ce40cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ce40cd020 0x7f4ce40cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51782/0 (socket says 172.21.15.67:51782) 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 -- 172.21.15.67:0/3149424614 learned_addr learned my addr 172.21.15.67:0/3149424614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce3e00640 1 --2- 172.21.15.67:0/3149424614 >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 0x7f4ce406e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 -- 172.21.15.67:0/3149424614 >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 msgr2=0x7f4ce406e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 --2- 172.21.15.67:0/3149424614 >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 0x7f4ce406e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 -- 172.21.15.67:0/3149424614 >> v1:172.21.15.187:6789/0 conn(0x7f4ce406f670 legacy=0x7f4ce406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce2a00640 1 -- 172.21.15.67:0/3149424614 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce40cf3a0 con 0x7f4ce40cd020 2024-04-17T08:55:06.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.249+0000 7f4ce3e00640 1 --2- 172.21.15.67:0/3149424614 >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 0x7f4ce406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:06.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.250+0000 7f4ce2a00640 1 --2- 172.21.15.67:0/3149424614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce40cd020 0x7f4ce40cd480 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40ce1e0 tx=0x7f4cd002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9094cbcf6f507854 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:06.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.250+0000 7f4ce2000640 1 -- 172.21.15.67:0/3149424614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0035020 con 0x7f4ce40cd020 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4ce2000640 1 -- 172.21.15.67:0/3149424614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd0031870 con 0x7f4ce40cd020 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4ce2000640 1 -- 172.21.15.67:0/3149424614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0031c40 con 0x7f4ce40cd020 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 -- 172.21.15.67:0/3149424614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce40cd020 msgr2=0x7f4ce40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 --2- 172.21.15.67:0/3149424614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce40cd020 0x7f4ce40cd480 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f4ce40ce1e0 tx=0x7f4cd002b910 comp rx=0 tx=0).stop 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 -- 172.21.15.67:0/3149424614 shutdown_connections 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 --2- 172.21.15.67:0/3149424614 >> v2:172.21.15.187:3300/0 conn(0x7f4ce406e020 0x7f4ce406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 --2- 172.21.15.67:0/3149424614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce40cd020 0x7f4ce40cd480 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 -- 172.21.15.67:0/3149424614 >> 172.21.15.67:0/3149424614 conn(0x7f4ce406bf50 msgr2=0x7f4ce40b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.251+0000 7f4cea200640 1 -- 172.21.15.67:0/3149424614 shutdown_connections 2024-04-17T08:55:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.252+0000 7f4cea200640 1 -- 172.21.15.67:0/3149424614 wait complete. 2024-04-17T08:55:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.254+0000 7f4cea200640 1 Processor -- start 2024-04-17T08:55:06.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.254+0000 7f4cea200640 1 -- start start 2024-04-17T08:55:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ce406e020 0x7f4ce4148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ce40cd020 0x7f4ce415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce40ce8a0 con 0x7f4ce406f670 2024-04-17T08:55:06.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce40ce720 con 0x7f4ce40cd020 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4cea200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce415cb00 con 0x7f4ce406e020 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51786/0 (socket says 172.21.15.67:51786) 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 -- 172.21.15.67:0/1410480633 learned_addr learned my addr 172.21.15.67:0/1410480633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce3e00640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ce40cd020 0x7f4ce415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce3400640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ce406e020 0x7f4ce4148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ce406e020 msgr2=0x7f4ce4148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ce406e020 0x7f4ce4148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.256+0000 7f4ce2a00640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ce40cd020 msgr2=0x7f4ce415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4ce2a00640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ce40cd020 0x7f4ce415d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4ce2a00640 1 -- 172.21.15.67:0/1410480633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce4154eb0 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4ce2a00640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f4cd002b950 tx=0x7f4cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0041070 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd003b430 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0035040 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce4155190 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.257+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce4155750 con 0x7f4ce406f670 2024-04-17T08:55:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.258+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce4068b90 con 0x7f4ce406f670 2024-04-17T08:55:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.260+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7f4cd004c060 con 0x7f4ce406f670 2024-04-17T08:55:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.261+0000 7f4cdb400640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 0x7f4cc406f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.261+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(289..289 src has 1..289) ==== 6110+0+0 (secure 0 0 0) 0x7f4cd00ae5b0 con 0x7f4ce406f670 2024-04-17T08:55:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.261+0000 7f4ce3400640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 msgr2=0x7f4cc406f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.261+0000 7f4ce3400640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 0x7f4cc406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.267+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4cd007c540 con 0x7f4ce406f670 2024-04-17T08:55:06.290 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.290+0000 7f1720a0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:55:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.462+0000 7f4ce3400640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 msgr2=0x7f4cc406f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:06.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.462+0000 7f4ce3400640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 0x7f4cc406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T08:55:06.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.465+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ce4075870 con 0x7f4ce406f670 2024-04-17T08:55:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.467+0000 7f4cdb400640 1 -- 172.21.15.67:0/1410480633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v289) ==== 74+0+17232 (secure 0 0 0) 0x7f4cd007be40 con 0x7f4ce406f670 2024-04-17T08:55:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:06.466 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":289,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:54:43.367150+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"286","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.14","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/2687160515":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3174473408":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/740627636":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/492269958":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3820292707":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6833/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3797546006":"2024-04-18T08:53:42.064401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:55:06.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 msgr2=0x7f4cc406f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 0x7f4cc406f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 msgr2=0x7f4ce415f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f4cd002b950 tx=0x7f4cd00046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 shutdown_connections 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f4cc406cbf0 0x7f4cc406f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ce40cd020 0x7f4ce415d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ce406f670 0x7f4ce415f610 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 --2- 172.21.15.67:0/1410480633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ce406e020 0x7f4ce4148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 >> 172.21.15.67:0/1410480633 conn(0x7f4ce406bf50 msgr2=0x7f4ce406fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 shutdown_connections 2024-04-17T08:55:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.470+0000 7f4cea200640 1 -- 172.21.15.67:0/1410480633 wait complete. 2024-04-17T08:55:06.486 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T07:13:51.387984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T08:53:25.119464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T08:53:26.202816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T08:55:06.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T08:55:06.514 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:06.516+0000 7f7e9ce32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T08:55:06.603 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.604+0000 7f1720a0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:55:06.662 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.663+0000 7f1720a0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:55:06.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.719+0000 7f50e3e00640 1 Processor -- start 2024-04-17T08:55:06.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.719+0000 7f50e3e00640 1 -- start start 2024-04-17T08:55:06.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50e415ec20 0x7f50e40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e40b3b50 0x7f50e406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50e40d4f70 con 0x7f50e406cf80 2024-04-17T08:55:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f50e40d5250 con 0x7f50e415ec20 2024-04-17T08:55:06.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50e40cf4d0 con 0x7f50e40b3b50 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50e406cf80 0x7f50e40cfcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59386/0 (socket says 172.21.15.67:59386) 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.720+0000 7f50e3400640 1 -- 172.21.15.67:0/3099210690 learned_addr learned my addr 172.21.15.67:0/3099210690 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2a00640 1 --2- 172.21.15.67:0/3099210690 >> v2:172.21.15.187:3300/0 conn(0x7f50e415ec20 0x7f50e40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50dbe00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e40b3b50 0x7f50e406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2000640 1 -- 172.21.15.67:0/3099210690 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200478919 0 0) 0x7f50e40d4f70 con 0x7f50e406cf80 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2000640 1 -- 172.21.15.67:0/3099210690 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c8003520 con 0x7f50e406cf80 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2a00640 1 -- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e40b3b50 msgr2=0x7f50e406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2a00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e40b3b50 0x7f50e406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2a00640 1 -- 172.21.15.67:0/3099210690 >> v1:172.21.15.67:6789/0 conn(0x7f50e406cf80 legacy=0x7f50e40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.721+0000 7f50e2a00640 1 -- 172.21.15.67:0/3099210690 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e40cf7b0 con 0x7f50e415ec20 2024-04-17T08:55:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.722+0000 7f50e2a00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e415ec20 0x7f50e40d22b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f50d40097a0 tx=0x7f50d402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4125bf0a25aca050 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.723+0000 7f50e2000640 1 -- 172.21.15.67:0/3099210690 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4039070 con 0x7f50e415ec20 2024-04-17T08:55:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.723+0000 7f50e2000640 1 -- 172.21.15.67:0/3099210690 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d40316f0 con 0x7f50e415ec20 2024-04-17T08:55:06.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.723+0000 7f50e2000640 1 -- 172.21.15.67:0/3099210690 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4031ac0 con 0x7f50e415ec20 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.723+0000 7f50e3e00640 1 -- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e415ec20 msgr2=0x7f50e40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.723+0000 7f50e3e00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e415ec20 0x7f50e40d22b0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f50d40097a0 tx=0x7f50d402b790 comp rx=0 tx=0).stop 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 -- 172.21.15.67:0/3099210690 shutdown_connections 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e40b3b50 0x7f50e406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 --2- 172.21.15.67:0/3099210690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e415ec20 0x7f50e40d22b0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 -- 172.21.15.67:0/3099210690 >> 172.21.15.67:0/3099210690 conn(0x7f50e408bff0 msgr2=0x7f50e40d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:06.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 -- 172.21.15.67:0/3099210690 shutdown_connections 2024-04-17T08:55:06.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.724+0000 7f50e3e00640 1 -- 172.21.15.67:0/3099210690 wait complete. 2024-04-17T08:55:06.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.725+0000 7f50e3e00640 1 Processor -- start 2024-04-17T08:55:06.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.725+0000 7f50e3e00640 1 -- start start 2024-04-17T08:55:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e406cf80 0x7f50e4161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e40b3b50 0x7f50e41489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e40d0560 con 0x7f50e415ec20 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f50e40d03e0 con 0x7f50e40b3b50 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.726+0000 7f50e3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50e40d10d0 con 0x7f50e406cf80 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51792/0 (socket says 172.21.15.67:51792) 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 -- 172.21.15.67:0/884093298 learned_addr learned my addr 172.21.15.67:0/884093298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50dbe00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e40b3b50 0x7f50e41489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e2a00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e406cf80 0x7f50e4161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 -- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e406cf80 msgr2=0x7f50e4161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e406cf80 0x7f50e4161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 -- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e40b3b50 msgr2=0x7f50e41489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e40b3b50 0x7f50e41489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 -- 172.21.15.67:0/884093298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e4148f00 con 0x7f50e415ec20 2024-04-17T08:55:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.727+0000 7f50e3400640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00caa0 tx=0x7f50dc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.728+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc017070 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.728+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50dc00ac00 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.728+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e414c1a0 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.728+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc012400 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.728+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e4160560 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.729+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e4068a10 con 0x7f50e415ec20 2024-04-17T08:55:06.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.730+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 163) ==== 91307+0+0 (secure 0 0 0) 0x7f50dc01a070 con 0x7f50e415ec20 2024-04-17T08:55:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.731+0000 7f50daa00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 0x7f50c006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.731+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(290..290 src has 1..290) ==== 6198+0+0 (secure 0 0 0) 0x7f50dc08bfd0 con 0x7f50e415ec20 2024-04-17T08:55:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.731+0000 7f50e2a00640 1 -- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 msgr2=0x7f50c006f080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.187:6832/1576025869 2024-04-17T08:55:06.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.731+0000 7f50e2a00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 0x7f50c006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T08:55:06.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.735+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f50dc0597b0 con 0x7f50e415ec20 2024-04-17T08:55:06.805 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.806+0000 7f1720a0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:55:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.827+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 164) ==== 90751+0+0 (secure 0 0 0) 0x7f50dc055e20 con 0x7f50e415ec20 2024-04-17T08:55:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.827+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 msgr2=0x7f50c006f080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T08:55:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.827+0000 7f50daa00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 0x7f50c006f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.893 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:06.892+0000 7f7e9ce32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.947+0000 7f169a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.947+0000 7f169a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.947+0000 7f169a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.947+0000 7f169a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.947+0000 7f169a200640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.948+0000 7f169de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.948+0000 7f169de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.947 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.948+0000 7f169de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.948 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.948+0000 7f169de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.948 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-17T08:55:06.948+0000 7f169de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-17T08:55:06.960 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:06.962+0000 7f7e9ce32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T08:55:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.981+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f50e40d55f0 con 0x7f50e415ec20 2024-04-17T08:55:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.982+0000 7f50daa00640 1 -- 172.21.15.67:0/884093298 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v291) ==== 93+0+10 (secure 0 0 0) 0x7f50dc059eb0 con 0x7f50e415ec20 2024-04-17T08:55:06.980 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 msgr2=0x7f50e414bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00caa0 tx=0x7f50dc00cf60 comp rx=0 tx=0).stop 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 shutdown_connections 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:6832/1576025869,v1:172.21.15.187:6833/1576025869] conn(0x7f50c006cbc0 0x7f50c006f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e415ec20 0x7f50e414bc60 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e40b3b50 0x7f50e41489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 --2- 172.21.15.67:0/884093298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50e406cf80 0x7f50e4161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 >> 172.21.15.67:0/884093298 conn(0x7f50e408bff0 msgr2=0x7f50e40d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 shutdown_connections 2024-04-17T08:55:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:06.984+0000 7f50e3e00640 1 -- 172.21.15.67:0/884093298 wait complete. 2024-04-17T08:55:07.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T08:55:07.100 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T08:55:07.101+0000 7f7e9ce32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T08:55:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.227+0000 7fa201400640 1 Processor -- start 2024-04-17T08:55:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.227+0000 7fa201400640 1 -- start start 2024-04-17T08:55:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.229+0000 7fa201400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc072d80 0x7fa1fc071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.229+0000 7fa201400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.229+0000 7fa201400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1fc0cea70 con 0x7fa1fc145ac0 2024-04-17T08:55:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.229+0000 7fa201400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa1fc0ce840 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.229+0000 7fa201400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1fc0ced50 con 0x7fa1fc072d80 2024-04-17T08:55:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51804/0 (socket says 172.21.15.67:51804) 2024-04-17T08:55:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 -- 172.21.15.67:0/881118644 learned_addr learned my addr 172.21.15.67:0/881118644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1f3e00640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc072d80 0x7fa1fc071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 -- 172.21.15.67:0/881118644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc072d80 msgr2=0x7fa1fc071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc072d80 0x7fa1fc071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 -- 172.21.15.67:0/881118644 >> v1:172.21.15.187:6789/0 conn(0x7fa1fc145ac0 legacy=0x7fa1fc072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.230+0000 7fa1fb200640 1 -- 172.21.15.67:0/881118644 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1fc0cf030 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.231+0000 7fa1fb200640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e50 tx=0x7fa1f402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ee7088d2861ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa1f9e00640 1 -- 172.21.15.67:0/881118644 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f402cb00 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa1f9e00640 1 -- 172.21.15.67:0/881118644 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f402cca0 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa1f9e00640 1 -- 172.21.15.67:0/881118644 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4034610 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 -- 172.21.15.67:0/881118644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc0ce3b0 msgr2=0x7fa1fc0cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e50 tx=0x7fa1f402bf60 comp rx=0 tx=0).stop 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 -- 172.21.15.67:0/881118644 shutdown_connections 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc0ce3b0 0x7fa1fc0cfb10 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 --2- 172.21.15.67:0/881118644 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc072d80 0x7fa1fc071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.232+0000 7fa201400640 1 -- 172.21.15.67:0/881118644 >> 172.21.15.67:0/881118644 conn(0x7fa1fc067ee0 msgr2=0x7fa1fc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.233+0000 7fa201400640 1 -- 172.21.15.67:0/881118644 shutdown_connections 2024-04-17T08:55:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.233+0000 7fa201400640 1 -- 172.21.15.67:0/881118644 wait complete. 2024-04-17T08:55:07.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.234+0000 7fa201400640 1 Processor -- start 2024-04-17T08:55:07.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.234+0000 7fa201400640 1 -- start start 2024-04-17T08:55:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1fc072d80 0x7fa1fc134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc0ce3b0 0x7fa1fc15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1fc145ea0 con 0x7fa1fc145ac0 2024-04-17T08:55:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa1fc0cf5f0 con 0x7fa1fc072d80 2024-04-17T08:55:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa201400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1fc0d0050 con 0x7fa1fc0ce3b0 2024-04-17T08:55:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa1fb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.235+0000 7fa1fb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51820/0 (socket says 172.21.15.67:51820) 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 -- 172.21.15.67:0/720733221 learned_addr learned my addr 172.21.15.67:0/720733221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 -- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc0ce3b0 msgr2=0x7fa1fc15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc0ce3b0 0x7fa1fc15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 -- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1fc072d80 msgr2=0x7fa1fc134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1fc072d80 0x7fa1fc134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 -- 172.21.15.67:0/720733221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1fc15dd50 con 0x7fa1fc145ac0 2024-04-17T08:55:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.236+0000 7fa1fb200640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e50 tx=0x7fa1f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.237+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4043020 con 0x7fa1fc145ac0 2024-04-17T08:55:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.237+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f402ce50 con 0x7fa1fc145ac0 2024-04-17T08:55:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.237+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4009510 con 0x7fa1fc145ac0 2024-04-17T08:55:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.237+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1fc15d790 con 0x7fa1fc145ac0 2024-04-17T08:55:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.237+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1fc15e4a0 con 0x7fa1fc145ac0 2024-04-17T08:55:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.238+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1fc068ca0 con 0x7fa1fc145ac0 2024-04-17T08:55:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.239+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 164) ==== 90751+0+0 (secure 0 0 0) 0x7fa1f4046020 con 0x7fa1fc145ac0 2024-04-17T08:55:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.240+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7fa1f40adfb0 con 0x7fa1fc145ac0 2024-04-17T08:55:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.244+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa1f404d030 con 0x7fa1fc145ac0 2024-04-17T08:55:07.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.444+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1fc0cd1e0 con 0x7fa1fc145ac0 2024-04-17T08:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.445+0000 7fa1f2a00640 1 -- 172.21.15.67:0/720733221 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v291) ==== 104+0+11 (secure 0 0 0) 0x7fa1f407c490 con 0x7fa1fc145ac0 2024-04-17T08:55:07.444 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.447+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 msgr2=0x7fa1fc0ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.447+0000 7fa201400640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009e50 tx=0x7fa1f4004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 shutdown_connections 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1fc145ac0 0x7fa1fc0ca7f0 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1fc0ce3b0 0x7fa1fc15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 --2- 172.21.15.67:0/720733221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1fc072d80 0x7fa1fc134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 >> 172.21.15.67:0/720733221 conn(0x7fa1fc067ee0 msgr2=0x7fa1fc144950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:07.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 shutdown_connections 2024-04-17T08:55:07.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.448+0000 7fa201400640 1 -- 172.21.15.67:0/720733221 wait complete. 2024-04-17T08:55:07.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T08:55:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.686+0000 7fd937200640 1 Processor -- start 2024-04-17T08:55:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.687+0000 7fd937200640 1 -- start start 2024-04-17T08:55:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.688+0000 7fd937200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.688+0000 7fd937200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9380d4f70 con 0x7fd9380cfcd0 2024-04-17T08:55:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.688+0000 7fd937200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9380d5250 con 0x7fd9380b3b50 2024-04-17T08:55:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.688+0000 7fd937200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9380cf4d0 con 0x7fd93815ec20 2024-04-17T08:55:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.689+0000 7fd936800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd9380cfcd0 0x7fd9380d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59394/0 (socket says 172.21.15.67:59394) 2024-04-17T08:55:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.689+0000 7fd936800640 1 -- 172.21.15.67:0/2093154135 learned_addr learned my addr 172.21.15.67:0/2093154135 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.689+0000 7fd935e00640 1 --2- 172.21.15.67:0/2093154135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.689+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 153260188 0 0) 0x7fd9380d4f70 con 0x7fd9380cfcd0 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.689+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd918003520 con 0x7fd9380cfcd0 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3977061129 0 0) 0x7fd9380d5250 con 0x7fd9380b3b50 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9380d4f70 con 0x7fd9380b3b50 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd935e00640 1 -- 172.21.15.67:0/2093154135 >> v1:172.21.15.187:6789/0 conn(0x7fd9380b3b50 legacy=0x7fd93806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd935e00640 1 -- 172.21.15.67:0/2093154135 >> v1:172.21.15.67:6789/0 conn(0x7fd9380cfcd0 legacy=0x7fd9380d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3608422800 0 0) 0x7fd918003520 con 0x7fd9380cfcd0 2024-04-17T08:55:07.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.690+0000 7fd935e00640 1 -- 172.21.15.67:0/2093154135 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9380cf7b0 con 0x7fd93815ec20 2024-04-17T08:55:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.691+0000 7fd935e00640 1 --2- 172.21.15.67:0/2093154135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380d22b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd9280097d0 tx=0x7fd92802b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cea385cb68d47ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.691+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928037070 con 0x7fd93815ec20 2024-04-17T08:55:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd928030510 con 0x7fd93815ec20 2024-04-17T08:55:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd934a00640 1 -- 172.21.15.67:0/2093154135 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930001670 con 0x7fd93815ec20 2024-04-17T08:55:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd937200640 1 -- 172.21.15.67:0/2093154135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 msgr2=0x7fd9380d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd937200640 1 --2- 172.21.15.67:0/2093154135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380d22b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd9280097d0 tx=0x7fd92802b6e0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd937200640 1 -- 172.21.15.67:0/2093154135 shutdown_connections 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd937200640 1 --2- 172.21.15.67:0/2093154135 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380d22b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.692+0000 7fd937200640 1 -- 172.21.15.67:0/2093154135 >> 172.21.15.67:0/2093154135 conn(0x7fd93808bff0 msgr2=0x7fd9380d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.693+0000 7fd937200640 1 -- 172.21.15.67:0/2093154135 shutdown_connections 2024-04-17T08:55:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.693+0000 7fd937200640 1 -- 172.21.15.67:0/2093154135 wait complete. 2024-04-17T08:55:07.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.694+0000 7fd937200640 1 Processor -- start 2024-04-17T08:55:07.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.694+0000 7fd937200640 1 -- start start 2024-04-17T08:55:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 0x7fd9380b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380b9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd93806d2d0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd93806d150 con 0x7fd9380b3b50 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.695+0000 7fd937200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9380d17e0 con 0x7fd93815ec20 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51830/0 (socket says 172.21.15.67:51830) 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 -- 172.21.15.67:0/260638141 learned_addr learned my addr 172.21.15.67:0/260638141 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935e00640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 0x7fd9380b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd936800640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380b9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 -- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 msgr2=0x7fd9380b9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380b9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 -- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 msgr2=0x7fd9380b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 0x7fd9380b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935400640 1 -- 172.21.15.67:0/260638141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd938161000 con 0x7fd9380cfcd0 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.696+0000 7fd935e00640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 0x7fd9380b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd935400640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fd92000c870 tx=0x7fd92000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920003bc0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd920003d60 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92000aa90 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9381612e0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.697+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9381618a0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.698+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd938068a10 con 0x7fd9380cfcd0 2024-04-17T08:55:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.699+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 164) ==== 90751+0+0 (secure 0 0 0) 0x7fd92000ac30 con 0x7fd9380cfcd0 2024-04-17T08:55:07.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.701+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7fd92008ba50 con 0x7fd9380cfcd0 2024-04-17T08:55:07.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.705+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd920059f30 con 0x7fd9380cfcd0 2024-04-17T08:55:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.838+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 165) ==== 90878+0+0 (secure 0 0 0) 0x7fd920055580 con 0x7fd9380cfcd0 2024-04-17T08:55:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.839+0000 7fd92e000640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 0x7fd914074640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:07.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.843+0000 7fd935e00640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 0x7fd914074640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.851+0000 7fd935e00640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 0x7fd914074640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd928034000 tx=0x7fd928008030 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.911+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd9380d55f0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.912+0000 7fd92e000640 1 -- 172.21.15.67:0/260638141 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v291) ==== 100+0+11 (secure 0 0 0) 0x7fd9200795f0 con 0x7fd9380cfcd0 2024-04-17T08:55:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T08:55:07.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 msgr2=0x7fd914074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 0x7fd914074640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd928034000 tx=0x7fd928008030 comp rx=0 tx=0).stop 2024-04-17T08:55:07.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 msgr2=0x7fd9380b98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:07.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fd92000c870 tx=0x7fd92000cd30 comp rx=0 tx=0).stop 2024-04-17T08:55:07.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 shutdown_connections 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd914072250 0x7fd914074640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd93815ec20 0x7fd9380b9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380cfcd0 0x7fd9380b98c0 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 --2- 172.21.15.67:0/260638141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9380b3b50 0x7fd9380b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.914+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 >> 172.21.15.67:0/260638141 conn(0x7fd93808bff0 msgr2=0x7fd9380d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.915+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 shutdown_connections 2024-04-17T08:55:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:07.915+0000 7fd937200640 1 -- 172.21.15.67:0/260638141 wait complete. 2024-04-17T08:55:07.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:55:08.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.136+0000 7ff746000640 1 Processor -- start 2024-04-17T08:55:08.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.136+0000 7ff746000640 1 -- start start 2024-04-17T08:55:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.138+0000 7ff746000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400ce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.138+0000 7ff746000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7400c6f90 0x7ff74006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.138+0000 7ff746000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff740072b20 con 0x7ff740166440 2024-04-17T08:55:08.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.138+0000 7ff746000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff7400ce960 con 0x7ff7400c6f90 2024-04-17T08:55:08.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.138+0000 7ff746000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7400c5eb0 con 0x7ff740072420 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7400c6f90 0x7ff74006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7400c6f90 0x7ff74006dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51842/0 (socket says 172.21.15.67:51842) 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 -- 172.21.15.67:0/2475062786 learned_addr learned my addr 172.21.15.67:0/2475062786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73ea00640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 -- 172.21.15.67:0/2475062786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 msgr2=0x7ff7400ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 -- 172.21.15.67:0/2475062786 >> v1:172.21.15.187:6789/0 conn(0x7ff740166440 legacy=0x7ff740071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.139+0000 7ff73fe00640 1 -- 172.21.15.67:0/2475062786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7400c6190 con 0x7ff7400c6f90 2024-04-17T08:55:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.140+0000 7ff73fe00640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7400c6f90 0x7ff74006dd50 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff73802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dc491533137ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff73e000640 1 -- 172.21.15.67:0/2475062786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73802c9e0 con 0x7ff7400c6f90 2024-04-17T08:55:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff73e000640 1 -- 172.21.15.67:0/2475062786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff73802cb80 con 0x7ff7400c6f90 2024-04-17T08:55:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff73e000640 1 -- 172.21.15.67:0/2475062786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7380343f0 con 0x7ff7400c6f90 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff746000640 1 -- 172.21.15.67:0/2475062786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7400c6f90 msgr2=0x7ff74006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff746000640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7400c6f90 0x7ff74006dd50 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff73802be40 comp rx=0 tx=0).stop 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.141+0000 7ff746000640 1 -- 172.21.15.67:0/2475062786 shutdown_connections 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.142+0000 7ff746000640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7400c6f90 0x7ff74006dd50 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.142+0000 7ff746000640 1 --2- 172.21.15.67:0/2475062786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.142+0000 7ff746000640 1 -- 172.21.15.67:0/2475062786 >> 172.21.15.67:0/2475062786 conn(0x7ff74006bf50 msgr2=0x7ff7400b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.142+0000 7ff746000640 1 -- 172.21.15.67:0/2475062786 shutdown_connections 2024-04-17T08:55:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.142+0000 7ff746000640 1 -- 172.21.15.67:0/2475062786 wait complete. 2024-04-17T08:55:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.143+0000 7ff746000640 1 Processor -- start 2024-04-17T08:55:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.144+0000 7ff746000640 1 -- start start 2024-04-17T08:55:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.144+0000 7ff746000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.144+0000 7ff746000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7400c6f90 0x7ff74015a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.144+0000 7ff746000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff746000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7400cf0d0 con 0x7ff740166440 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff746000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff74006e290 con 0x7ff7400c6f90 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff746000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff7400b4000 con 0x7ff740072420 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51846/0 (socket says 172.21.15.67:51846) 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 -- 172.21.15.67:0/2818386223 learned_addr learned my addr 172.21.15.67:0/2818386223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73f400640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73ea00640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7400c6f90 0x7ff74015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 -- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 msgr2=0x7ff7400c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400c7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 -- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7400c6f90 msgr2=0x7ff74015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7400c6f90 0x7ff74015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.145+0000 7ff73fe00640 1 -- 172.21.15.67:0/2818386223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7400c7cc0 con 0x7ff740166440 2024-04-17T08:55:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff73fe00640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738043020 con 0x7ff740166440 2024-04-17T08:55:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff73803e450 con 0x7ff740166440 2024-04-17T08:55:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738009510 con 0x7ff740166440 2024-04-17T08:55:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7400c7fa0 con 0x7ff740166440 2024-04-17T08:55:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.146+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7400c8560 con 0x7ff740166440 2024-04-17T08:55:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.147+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff740068b90 con 0x7ff740166440 2024-04-17T08:55:08.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.148+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 165) ==== 90878+0+0 (secure 0 0 0) 0x7ff738046020 con 0x7ff740166440 2024-04-17T08:55:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.149+0000 7ff737400640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 0x7ff72006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.149+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7ff7380ae430 con 0x7ff740166440 2024-04-17T08:55:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.149+0000 7ff73f400640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 0x7ff72006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.151+0000 7ff73f400640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 0x7ff72006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff730006fd0 tx=0x7ff730008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.153+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7ff73804d030 con 0x7ff740166440 2024-04-17T08:55:08.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.400+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7400b6480 con 0x7ff740166440 2024-04-17T08:55:08.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.401+0000 7ff737400640 1 -- 172.21.15.67:0/2818386223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7ff73807c730 con 0x7ff740166440 2024-04-17T08:55:08.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:08.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:55:08.400 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:55:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 msgr2=0x7ff72006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 0x7ff72006ec00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff730006fd0 tx=0x7ff730008040 comp rx=0 tx=0).stop 2024-04-17T08:55:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 msgr2=0x7ff7400b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7ff738009e50 tx=0x7ff738004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 shutdown_connections 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7ff72006c740 0x7ff72006ec00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff740166440 0x7ff7400b31a0 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7400c6f90 0x7ff74015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 --2- 172.21.15.67:0/2818386223 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff740072420 0x7ff7400c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 >> 172.21.15.67:0/2818386223 conn(0x7ff74006bf50 msgr2=0x7ff7400b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 shutdown_connections 2024-04-17T08:55:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.404+0000 7ff746000640 1 -- 172.21.15.67:0/2818386223 wait complete. 2024-04-17T08:55:08.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:55:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.643+0000 7efefb200640 1 Processor -- start 2024-04-17T08:55:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.643+0000 7efefb200640 1 -- start start 2024-04-17T08:55:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef40cf910 0x7efef40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefb200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40d2d30 con 0x7efef415ec60 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efef40d49c0 con 0x7efef40cf910 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efef40d3660 con 0x7efef40b6d00 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef40cf910 0x7efef40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef40cf910 0x7efef40b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51852/0 (socket says 172.21.15.67:51852) 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.645+0000 7efefa800640 1 -- 172.21.15.67:0/3934668646 learned_addr learned my addr 172.21.15.67:0/3934668646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.646+0000 7efefa800640 1 -- 172.21.15.67:0/3934668646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 msgr2=0x7efef40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.646+0000 7efefa800640 1 --2- 172.21.15.67:0/3934668646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.646+0000 7efefa800640 1 -- 172.21.15.67:0/3934668646 >> v1:172.21.15.187:6789/0 conn(0x7efef415ec60 legacy=0x7efef40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.646+0000 7efefa800640 1 -- 172.21.15.67:0/3934668646 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40d3010 con 0x7efef40cf910 2024-04-17T08:55:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.647+0000 7efefa800640 1 --2- 172.21.15.67:0/3934668646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef40cf910 0x7efef40b7b70 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e50 tx=0x7efef002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f4e441d19b3123 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.647+0000 7efef8a00640 1 -- 172.21.15.67:0/3934668646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef002ca70 con 0x7efef40cf910 2024-04-17T08:55:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.647+0000 7efef8a00640 1 -- 172.21.15.67:0/3934668646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef002cc10 con 0x7efef40cf910 2024-04-17T08:55:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.647+0000 7efef8a00640 1 -- 172.21.15.67:0/3934668646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef00343f0 con 0x7efef40cf910 2024-04-17T08:55:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 -- 172.21.15.67:0/3934668646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef40cf910 msgr2=0x7efef40b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 --2- 172.21.15.67:0/3934668646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef40cf910 0x7efef40b7b70 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e50 tx=0x7efef002bed0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 -- 172.21.15.67:0/3934668646 shutdown_connections 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 --2- 172.21.15.67:0/3934668646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef40cf910 0x7efef40b7b70 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 --2- 172.21.15.67:0/3934668646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 -- 172.21.15.67:0/3934668646 >> 172.21.15.67:0/3934668646 conn(0x7efef40d15f0 msgr2=0x7efef40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 -- 172.21.15.67:0/3934668646 shutdown_connections 2024-04-17T08:55:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.648+0000 7efefb200640 1 -- 172.21.15.67:0/3934668646 wait complete. 2024-04-17T08:55:08.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.650+0000 7efefb200640 1 Processor -- start 2024-04-17T08:55:08.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.650+0000 7efefb200640 1 -- start start 2024-04-17T08:55:08.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efef40cf910 0x7efef41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efef415f040 con 0x7efef415ec60 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efef40cfdc0 con 0x7efef40cf910 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefb200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efef40d00c0 con 0x7efef40b6d00 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51864/0 (socket says 172.21.15.67:51864) 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.651+0000 7efefa800640 1 -- 172.21.15.67:0/1984835972 learned_addr learned my addr 172.21.15.67:0/1984835972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efef9e00640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efef9400640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efef40cf910 0x7efef41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 -- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 msgr2=0x7efef40cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 -- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efef40cf910 msgr2=0x7efef41573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efef40cf910 0x7efef41573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 -- 172.21.15.67:0/1984835972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40cdba0 con 0x7efef415ec60 2024-04-17T08:55:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.652+0000 7efefa800640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7efef0036040 tx=0x7efef0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.653+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0043020 con 0x7efef415ec60 2024-04-17T08:55:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.653+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef003e450 con 0x7efef415ec60 2024-04-17T08:55:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.653+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef40cde80 con 0x7efef415ec60 2024-04-17T08:55:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.653+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef003e820 con 0x7efef415ec60 2024-04-17T08:55:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.653+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef40ce440 con 0x7efef415ec60 2024-04-17T08:55:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.654+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef4068a10 con 0x7efef415ec60 2024-04-17T08:55:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.655+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 165) ==== 90878+0+0 (secure 0 0 0) 0x7efef0046020 con 0x7efef415ec60 2024-04-17T08:55:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.656+0000 7efeee000640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 0x7efed806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.656+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7efef00ae460 con 0x7efef415ec60 2024-04-17T08:55:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.657+0000 7efef9e00640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 0x7efed806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.658+0000 7efef9e00640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 0x7efed806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.660+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efef004d030 con 0x7efef415ec60 2024-04-17T08:55:08.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.850+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 166) ==== 91086+0+0 (secure 0 0 0) 0x7efef00787a0 con 0x7efef415ec60 2024-04-17T08:55:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.869+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efef415f820 con 0x7efef415ec60 2024-04-17T08:55:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.871+0000 7efeee000640 1 -- 172.21.15.67:0/1984835972 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v291) ==== 74+0+18019 (secure 0 0 0) 0x7efef007c830 con 0x7efef415ec60 2024-04-17T08:55:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:08.870 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":291,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:55:06.811837+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"290","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":290,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.14","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/839885293":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3001440476":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/150412697":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2744215170":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/1795094394":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2855667520":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/933679678":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3424176410":"2024-04-18T08:55:06.811712+0000","172.21.15.187:6833/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3757546694":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2687160515":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3174473408":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/740627636":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/492269958":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3820292707":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6833/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3797546006":"2024-04-18T08:53:42.064401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":37,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 msgr2=0x7efed806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 0x7efed806ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8008040 comp rx=0 tx=0).stop 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 msgr2=0x7efef40cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7efef0036040 tx=0x7efef0004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 shutdown_connections 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efed806c740 0x7efed806ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef415ec60 0x7efef40cb3d0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efef40cf910 0x7efef41573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 --2- 172.21.15.67:0/1984835972 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efef40b6d00 0x7efef40cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.874+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 >> 172.21.15.67:0/1984835972 conn(0x7efef40d15f0 msgr2=0x7efef40b9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.875+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 shutdown_connections 2024-04-17T08:55:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:08.875+0000 7efefb200640 1 -- 172.21.15.67:0/1984835972 wait complete. 2024-04-17T08:55:08.893 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:55:08.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:55:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.117+0000 7fcbe3e00640 1 Processor -- start 2024-04-17T08:55:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.117+0000 7fcbe3e00640 1 -- start start 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbe3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe40cfae0 0x7fcbe4169810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbe3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe40d2d30 con 0x7fcbe40b6d00 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbe3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe40d3010 con 0x7fcbe415ec60 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbe3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe40d49c0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbe3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe40cfae0 0x7fcbe4169810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbdbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcbe40b6d00 0x7fcbe40cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59410/0 (socket says 172.21.15.67:59410) 2024-04-17T08:55:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.119+0000 7fcbdbe00640 1 -- 172.21.15.67:0/551331980 learned_addr learned my addr 172.21.15.67:0/551331980 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.120+0000 7fcbe3400640 1 -- 172.21.15.67:0/551331980 >> v1:172.21.15.187:6789/0 conn(0x7fcbe415ec60 legacy=0x7fcbe40b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.120+0000 7fcbe3400640 1 -- 172.21.15.67:0/551331980 >> v1:172.21.15.67:6789/0 conn(0x7fcbe40b6d00 legacy=0x7fcbe40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.120+0000 7fcbe3400640 1 -- 172.21.15.67:0/551331980 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe40d32f0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.121+0000 7fcbe3400640 1 --2- 172.21.15.67:0/551331980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe4169810 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009d90 tx=0x7fcbdc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e9d066a73e7d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.121+0000 7fcbe2000640 1 -- 172.21.15.67:0/551331980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc039070 con 0x7fcbe40cfae0 2024-04-17T08:55:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.121+0000 7fcbe2000640 1 -- 172.21.15.67:0/551331980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbdc02cc60 con 0x7fcbe40cfae0 2024-04-17T08:55:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.121+0000 7fcbe2000640 1 -- 172.21.15.67:0/551331980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc034660 con 0x7fcbe40cfae0 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 -- 172.21.15.67:0/551331980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 msgr2=0x7fcbe4169810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/551331980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe4169810 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009d90 tx=0x7fcbdc02beb0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 -- 172.21.15.67:0/551331980 shutdown_connections 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/551331980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe4169810 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 -- 172.21.15.67:0/551331980 >> 172.21.15.67:0/551331980 conn(0x7fcbe40d15f0 msgr2=0x7fcbe40d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 -- 172.21.15.67:0/551331980 shutdown_connections 2024-04-17T08:55:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.122+0000 7fcbe3e00640 1 -- 172.21.15.67:0/551331980 wait complete. 2024-04-17T08:55:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.124+0000 7fcbe3e00640 1 Processor -- start 2024-04-17T08:55:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.124+0000 7fcbe3e00640 1 -- start start 2024-04-17T08:55:09.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe40b6d00 0x7fcbe4153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbe415ec60 0x7fcbe40b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe40b8b80 con 0x7fcbe40cfae0 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe40d3660 con 0x7fcbe40b6d00 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbe3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe40d00c0 con 0x7fcbe415ec60 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51884/0 (socket says 172.21.15.67:51884) 2024-04-17T08:55:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.125+0000 7fcbdbe00640 1 -- 172.21.15.67:0/2333619367 learned_addr learned my addr 172.21.15.67:0/2333619367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 -- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbe415ec60 msgr2=0x7fcbe40b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbe415ec60 0x7fcbe40b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 -- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe40b6d00 msgr2=0x7fcbe4153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe40b6d00 0x7fcbe4153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 -- 172.21.15.67:0/2333619367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe41f89d0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdbe00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c430 tx=0x7fcbcc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc017070 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc005ce0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe41f8cb0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.126+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0072f0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.127+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe41544b0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.128+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe4068a10 con 0x7fcbe40cfae0 2024-04-17T08:55:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.129+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 166) ==== 91086+0+0 (secure 0 0 0) 0x7fcbcc01a070 con 0x7fcbe40cfae0 2024-04-17T08:55:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.130+0000 7fcbdaa00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 0x7fcbc006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.130+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7fcbcc08b390 con 0x7fcbe40cfae0 2024-04-17T08:55:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.130+0000 7fcbe2a00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 0x7fcbc006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.131+0000 7fcbe2a00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 0x7fcbc006ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006fd0 tx=0x7fcbd4006bc0 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.134+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcbcc0593c0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.381+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbe40cef20 con 0x7fcbe40cfae0 2024-04-17T08:55:09.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.382+0000 7fcbdaa00640 1 -- 172.21.15.67:0/2333619367 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7fcbcc058cc0 con 0x7fcbe40cfae0 2024-04-17T08:55:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:09.381 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:55:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:55:09.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.385+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 msgr2=0x7fcbc006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.385+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 0x7fcbc006ee70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4006fd0 tx=0x7fcbd4006bc0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.385+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 msgr2=0x7fcbe41f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.385+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c430 tx=0x7fcbcc00c8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 shutdown_connections 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcbc006c9b0 0x7fcbc006ee70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbe415ec60 0x7fcbe40b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe40cfae0 0x7fcbe41f7f80 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/2333619367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe40b6d00 0x7fcbe4153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 >> 172.21.15.67:0/2333619367 conn(0x7fcbe40d15f0 msgr2=0x7fcbe40d4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 shutdown_connections 2024-04-17T08:55:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.386+0000 7fcbe3e00640 1 -- 172.21.15.67:0/2333619367 wait complete. 2024-04-17T08:55:09.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:55:09.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.631+0000 7fe349400640 1 Processor -- start 2024-04-17T08:55:09.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.631+0000 7fe349400640 1 -- start start 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe349400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe3440cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe349400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3440cd4a0 con 0x7fe344160280 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe349400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3441607f0 con 0x7fe344141ed0 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe349400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe34415ce30 con 0x7fe34406b8c0 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe33be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe344160280 0x7fe34406bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59426/0 (socket says 172.21.15.67:59426) 2024-04-17T08:55:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.633+0000 7fe33be00640 1 -- 172.21.15.67:0/1564798525 learned_addr learned my addr 172.21.15.67:0/1564798525 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:09.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343e00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe3440cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 598045480 0 0) 0x7fe3440cd4a0 con 0x7fe344160280 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe324003520 con 0x7fe344160280 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1333223081 0 0) 0x7fe324003520 con 0x7fe344160280 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 msgr2=0x7fe3440cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe3440cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 >> v1:172.21.15.187:6789/0 conn(0x7fe344141ed0 legacy=0x7fe34415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe344160ad0 con 0x7fe344160280 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe32c001670 con 0x7fe344160280 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343e00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe3440cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 >> v1:172.21.15.67:6789/0 conn(0x7fe344160280 legacy=0x7fe34406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe324003a50 0x7fe324003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe324005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe34415ce30 con 0x7fe324004800 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe32400bb50 con 0x7fe324003a50 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.634+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe32400bd20 con 0x7fe34406b8c0 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe343e00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe324003a50 0x7fe324003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe348a00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe324005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 -- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 msgr2=0x7fe324005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34406b8c0 0x7fe324005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 -- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe324003a50 msgr2=0x7fe324003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe324003a50 0x7fe324003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.635+0000 7fe33be00640 1 -- 172.21.15.67:0/1564798525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe324003520 con 0x7fe324004800 2024-04-17T08:55:09.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.636+0000 7fe33be00640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe324004c80 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c210 tx=0x7fe324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26289eca50ffeed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c007d30 con 0x7fe324004800 2024-04-17T08:55:09.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32c030070 con 0x7fe324004800 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe343400640 1 -- 172.21.15.67:0/1564798525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c02cd60 con 0x7fe324004800 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 -- 172.21.15.67:0/1564798525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 msgr2=0x7fe324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 --2- 172.21.15.67:0/1564798525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe324004c80 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c210 tx=0x7fe324005830 comp rx=0 tx=0).stop 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe348a00640 1 -- 172.21.15.67:0/1564798525 reap_dead start 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 -- 172.21.15.67:0/1564798525 shutdown_connections 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 -- 172.21.15.67:0/1564798525 >> 172.21.15.67:0/1564798525 conn(0x7fe344067d10 msgr2=0x7fe3440b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 -- 172.21.15.67:0/1564798525 shutdown_connections 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.637+0000 7fe349400640 1 -- 172.21.15.67:0/1564798525 wait complete. 2024-04-17T08:55:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.639+0000 7fe349400640 1 Processor -- start 2024-04-17T08:55:09.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.639+0000 7fe349400640 1 -- start start 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34415e060 0x7fe3441f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3440b09f0 0x7fe3440b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe34413f9b0 con 0x7fe324004800 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe344161090 con 0x7fe3440b09f0 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.640+0000 7fe349400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe34413fb30 con 0x7fe34415e060 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51906/0 (socket says 172.21.15.67:51906) 2024-04-17T08:55:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 -- 172.21.15.67:0/2928524527 learned_addr learned my addr 172.21.15.67:0/2928524527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:09.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe33be00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34415e060 0x7fe3441f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe348a00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3440b09f0 0x7fe3440b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 -- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34415e060 msgr2=0x7fe3441f1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34415e060 0x7fe3441f1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 -- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3440b09f0 msgr2=0x7fe3440b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3440b09f0 0x7fe3440b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.641+0000 7fe343e00640 1 -- 172.21.15.67:0/2928524527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3441f1d70 con 0x7fe324004800 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe343e00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7fe334010890 tx=0x7fe334010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33401a070 con 0x7fe324004800 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33400cb40 con 0x7fe324004800 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33400f040 con 0x7fe324004800 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3441f4550 con 0x7fe324004800 2024-04-17T08:55:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.642+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3441f47f0 con 0x7fe324004800 2024-04-17T08:55:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.643+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3440b1390 con 0x7fe324004800 2024-04-17T08:55:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.645+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 166) ==== 91086+0+0 (secure 0 0 0) 0x7fe334015470 con 0x7fe324004800 2024-04-17T08:55:09.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.646+0000 7fe340c00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 0x7fe32006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:09.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.646+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(291..291 src has 1..291) ==== 6627+0+0 (secure 0 0 0) 0x7fe33408c6e0 con 0x7fe324004800 2024-04-17T08:55:09.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.646+0000 7fe33be00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 0x7fe32006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:09.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.647+0000 7fe33be00640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 0x7fe32006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe32c034000 tx=0x7fe32c009030 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.650+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe33405a680 con 0x7fe324004800 2024-04-17T08:55:09.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.908+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe344068a10 con 0x7fe324004800 2024-04-17T08:55:09.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.909+0000 7fe340c00640 1 -- 172.21.15.67:0/2928524527 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5219 (secure 0 0 0) 0x7fe334059f80 con 0x7fe324004800 2024-04-17T08:55:09.908 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:09.908 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T08:53:27.207090+0000","modified":"2024-04-17T08:53:30.274229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"snap_rstat":true,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13062},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13062":{"gid":13062,"name":"d","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6837/2174028812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":2174028812},{"type":"v1","addr":"172.21.15.187:6837","nonce":2174028812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13062,"qdb_cluster":[13062]},"id":18}]} 2024-04-17T08:55:09.908 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 151 2024-04-17T08:55:09.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 msgr2=0x7fe32006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 0x7fe32006ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe32c034000 tx=0x7fe32c009030 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 msgr2=0x7fe3440b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7fe334010890 tx=0x7fe334010d50 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 shutdown_connections 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe32006c980 0x7fe32006ee40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3440b09f0 0x7fe3440b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe34415e060 0x7fe3441f1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 --2- 172.21.15.67:0/2928524527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe324004800 0x7fe3440b4ea0 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 >> 172.21.15.67:0/2928524527 conn(0x7fe344067d10 msgr2=0x7fe3440cf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 shutdown_connections 2024-04-17T08:55:09.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:09.912+0000 7fe349400640 1 -- 172.21.15.67:0/2928524527 wait complete. 2024-04-17T08:55:09.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:55:10.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.137+0000 7f9707000640 1 Processor -- start 2024-04-17T08:55:10.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.138+0000 7f9707000640 1 -- start start 2024-04-17T08:55:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.139+0000 7f9707000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f970015ec60 0x7f97000b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.139+0000 7f9707000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9707000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97000d2d30 con 0x7f97000b6b30 2024-04-17T08:55:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9707000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f97000d49c0 con 0x7f970015ec60 2024-04-17T08:55:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9707000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97000d3660 con 0x7f97000cfae0 2024-04-17T08:55:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9705c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f970015ec60 0x7f97000b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9706600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9705c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f970015ec60 0x7f97000b65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:36272/0 (socket says 172.21.15.67:36272) 2024-04-17T08:55:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9705200640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f97000b6b30 0x7f97000cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.67:60422/0 (socket says 172.21.15.67:60422) 2024-04-17T08:55:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.140+0000 7f9705c00640 1 -- 172.21.15.67:0/908408769 learned_addr learned my addr 172.21.15.67:0/908408769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.141+0000 7f9705c00640 1 -- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 msgr2=0x7f97000b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.141+0000 7f9705c00640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.141+0000 7f9705c00640 1 -- 172.21.15.67:0/908408769 >> v1:172.21.15.187:6789/0 conn(0x7f97000b6b30 legacy=0x7f97000cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.141+0000 7f9705c00640 1 -- 172.21.15.67:0/908408769 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97000d3010 con 0x7f970015ec60 2024-04-17T08:55:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.141+0000 7f9706600640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:55:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.143+0000 7f9705c00640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000b65f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009e50 tx=0x7f96f402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0789278d5d1563f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.143+0000 7f96ffe00640 1 -- 172.21.15.67:0/908408769 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f402cb00 con 0x7f970015ec60 2024-04-17T08:55:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.143+0000 7f96ffe00640 1 -- 172.21.15.67:0/908408769 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f402cca0 con 0x7f970015ec60 2024-04-17T08:55:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.143+0000 7f96ffe00640 1 -- 172.21.15.67:0/908408769 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4034610 con 0x7f970015ec60 2024-04-17T08:55:10.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 -- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 msgr2=0x7f97000b65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000b65f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009e50 tx=0x7f96f402bf60 comp rx=0 tx=0).stop 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 -- 172.21.15.67:0/908408769 shutdown_connections 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 --2- 172.21.15.67:0/908408769 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000b65f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 -- 172.21.15.67:0/908408769 >> 172.21.15.67:0/908408769 conn(0x7f97000d15f0 msgr2=0x7f97000d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 -- 172.21.15.67:0/908408769 shutdown_connections 2024-04-17T08:55:10.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.144+0000 7f9707000640 1 -- 172.21.15.67:0/908408769 wait complete. 2024-04-17T08:55:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.146+0000 7f9707000640 1 Processor -- start 2024-04-17T08:55:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.147+0000 7f9707000640 1 -- start start 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000d4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f970015f040 con 0x7f97000b6b30 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f97000b6f70 con 0x7f970015ec60 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9707000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97000d00c0 con 0x7f97000cfae0 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9705c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9705c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51922/0 (socket says 172.21.15.67:51922) 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.148+0000 7f9705c00640 1 -- 172.21.15.67:0/2830861361 learned_addr learned my addr 172.21.15.67:0/2830861361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705200640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9706600640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000d4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 -- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 msgr2=0x7f97000ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 -- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 msgr2=0x7f97000d4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000d4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 -- 172.21.15.67:0/2830861361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9700154380 con 0x7f97000b6b30 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9706600640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000d4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9705c00640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f96f4000f80 tx=0x7f96f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4043020 con 0x7f97000b6b30 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f402ce50 con 0x7f97000b6b30 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4009510 con 0x7f97000b6b30 2024-04-17T08:55:10.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.149+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97000abed0 con 0x7f97000b6b30 2024-04-17T08:55:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.150+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97000ac490 con 0x7f97000b6b30 2024-04-17T08:55:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.151+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9700068a10 con 0x7f97000b6b30 2024-04-17T08:55:10.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.152+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 166) ==== 91086+0+0 (secure 0 0 0) 0x7f96f4046020 con 0x7f97000b6b30 2024-04-17T08:55:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.153+0000 7f96fd600640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 0x7f96e006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.153+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(292..292 src has 1..292) ==== 6627+0+0 (secure 0 0 0) 0x7f96f40ae550 con 0x7f97000b6b30 2024-04-17T08:55:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.153+0000 7f9705200640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 0x7f96e006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.155+0000 7f9705200640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 0x7f96e006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97000d4290 tx=0x7f96ec009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.158+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f96f404d030 con 0x7f97000b6b30 2024-04-17T08:55:10.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.357+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97000d3d00 con 0x7f97000b6b30 2024-04-17T08:55:10.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.359+0000 7f96fd600640 1 -- 172.21.15.67:0/2830861361 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v292) ==== 74+0+18018 (secure 0 0 0) 0x7f96f403c350 con 0x7f97000b6b30 2024-04-17T08:55:10.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:10.359 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":292,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:55:09.855586+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-04-17T08:53:25.119464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"285","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-04-17T08:53:26.202816+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"290","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":290,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.14","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/839885293":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3001440476":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/150412697":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2744215170":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/1795094394":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2855667520":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/933679678":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3424176410":"2024-04-18T08:55:06.811712+0000","172.21.15.187:6833/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3757546694":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2687160515":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3174473408":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/740627636":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/492269958":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3820292707":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6833/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3797546006":"2024-04-18T08:53:42.064401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":37,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:55:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 msgr2=0x7f96e006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 0x7f96e006ee40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97000d4290 tx=0x7f96ec009040 comp rx=0 tx=0).stop 2024-04-17T08:55:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 msgr2=0x7f9700153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f96f4000f80 tx=0x7f96f4004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 shutdown_connections 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f96e006c980 0x7f96e006ee40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f970015ec60 0x7f97000d4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97000cfae0 0x7f97000ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 --2- 172.21.15.67:0/2830861361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97000b6b30 0x7f9700153e40 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 >> 172.21.15.67:0/2830861361 conn(0x7f97000d15f0 msgr2=0x7f97000b8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 shutdown_connections 2024-04-17T08:55:10.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.362+0000 7f9707000640 1 -- 172.21.15.67:0/2830861361 wait complete. 2024-04-17T08:55:10.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T08:55:10.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.599+0000 7f6313200640 1 Processor -- start 2024-04-17T08:55:10.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.600+0000 7f6313200640 1 -- start start 2024-04-17T08:55:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.601+0000 7f6313200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f631415f3c0 0x7f631415f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.601+0000 7f6313200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.601+0000 7f6313200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63140cb420 con 0x7f63140afbf0 2024-04-17T08:55:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.601+0000 7f6313200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f631415fd60 con 0x7f631415f3c0 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.601+0000 7f6313200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63140cc030 con 0x7f63140ce510 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.602+0000 7f6312800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.602+0000 7f630be00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f631415f3c0 0x7f631415f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.602+0000 7f6312800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:60092/0 (socket says 172.21.15.67:60092) 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.602+0000 7f6312800640 1 -- 172.21.15.67:0/3804134365 learned_addr learned my addr 172.21.15.67:0/3804134365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.603+0000 7f6312800640 1 -- 172.21.15.67:0/3804134365 >> v2:172.21.15.187:3300/0 conn(0x7f631415f3c0 msgr2=0x7f631415f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.603+0000 7f6312800640 1 --2- 172.21.15.67:0/3804134365 >> v2:172.21.15.187:3300/0 conn(0x7f631415f3c0 0x7f631415f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.603+0000 7f6312800640 1 -- 172.21.15.67:0/3804134365 >> v1:172.21.15.187:6789/0 conn(0x7f63140afbf0 legacy=0x7f631406e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.603+0000 7f6312800640 1 -- 172.21.15.67:0/3804134365 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63140cb700 con 0x7f63140ce510 2024-04-17T08:55:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.603+0000 7f6311400640 1 -- 172.21.15.67:0/3804134365 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911491856 0 0) 0x7f63140cb420 con 0x7f63140afbf0 2024-04-17T08:55:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.605+0000 7f6312800640 1 --2- 172.21.15.67:0/3804134365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f630c009e50 tx=0x7f630c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f833289aec94de88 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.605+0000 7f6311400640 1 -- 172.21.15.67:0/3804134365 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c02cb00 con 0x7f63140ce510 2024-04-17T08:55:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.605+0000 7f6311400640 1 -- 172.21.15.67:0/3804134365 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f630c02cca0 con 0x7f63140ce510 2024-04-17T08:55:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.605+0000 7f6311400640 1 -- 172.21.15.67:0/3804134365 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c034610 con 0x7f63140ce510 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 -- 172.21.15.67:0/3804134365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 msgr2=0x7f63140ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 --2- 172.21.15.67:0/3804134365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f630c009e50 tx=0x7f630c02bf60 comp rx=0 tx=0).stop 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 -- 172.21.15.67:0/3804134365 shutdown_connections 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 --2- 172.21.15.67:0/3804134365 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63140ce510 0x7f63140ce980 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 --2- 172.21.15.67:0/3804134365 >> v2:172.21.15.187:3300/0 conn(0x7f631415f3c0 0x7f631415f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 -- 172.21.15.67:0/3804134365 >> 172.21.15.67:0/3804134365 conn(0x7f631406bef0 msgr2=0x7f63140b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 -- 172.21.15.67:0/3804134365 shutdown_connections 2024-04-17T08:55:10.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.606+0000 7f6313200640 1 -- 172.21.15.67:0/3804134365 wait complete. 2024-04-17T08:55:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.608+0000 7f6313200640 1 Processor -- start 2024-04-17T08:55:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.608+0000 7f6313200640 1 -- start start 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63140ce510 0x7f63140b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f631415f3c0 0x7f63140b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f631406f090 con 0x7f63140afbf0 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f63140cbcc0 con 0x7f63140ce510 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6313200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6314160110 con 0x7f631415f3c0 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6311e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6311e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51938/0 (socket says 172.21.15.67:51938) 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6311e00640 1 -- 172.21.15.67:0/11577517 learned_addr learned my addr 172.21.15.67:0/11577517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f6312800640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f631415f3c0 0x7f63140b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.610+0000 7f630be00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63140ce510 0x7f63140b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 -- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f631415f3c0 msgr2=0x7f63140b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f631415f3c0 0x7f63140b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 -- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63140ce510 msgr2=0x7f63140b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63140ce510 0x7f63140b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 -- 172.21.15.67:0/11577517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63140b4ee0 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6311e00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f630400c4b0 tx=0x7f630400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304017070 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6304005480 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304003830 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.611+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63140b51c0 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.612+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63140b5780 con 0x7f63140afbf0 2024-04-17T08:55:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.613+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6314068a10 con 0x7f63140afbf0 2024-04-17T08:55:10.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.614+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 166) ==== 91086+0+0 (secure 0 0 0) 0x7f630401a070 con 0x7f63140afbf0 2024-04-17T08:55:10.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.614+0000 7f630a000640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 0x7f62f006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:10.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.615+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(292..292 src has 1..292) ==== 6627+0+0 (secure 0 0 0) 0x7f630408b330 con 0x7f63140afbf0 2024-04-17T08:55:10.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.615+0000 7f630be00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 0x7f62f006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:10.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.616+0000 7f630be00640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 0x7f62f006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f63140b2140 tx=0x7f62fc008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.620+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f63040593d0 con 0x7f63140afbf0 2024-04-17T08:55:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.871+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6314155b60 con 0x7f63140afbf0 2024-04-17T08:55:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.881+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f6304055340 con 0x7f63140afbf0 2024-04-17T08:55:10.883 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T08:55:10.885+0000 7ff0f4e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T08:55:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.890+0000 7f630a000640 1 -- 172.21.15.67:0/11577517 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v152) ==== 161+0+0 (secure 0 0 0) 0x7f6304079380 con 0x7f63140afbf0 2024-04-17T08:55:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T08:55:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 msgr2=0x7f62f006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 0x7f62f006ee40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f63140b2140 tx=0x7f62fc008040 comp rx=0 tx=0).stop 2024-04-17T08:55:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 msgr2=0x7f631414e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f630400c4b0 tx=0x7f630400c970 comp rx=0 tx=0).stop 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 shutdown_connections 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f62f006c980 0x7f62f006ee40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f631415f3c0 0x7f63140b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63140ce510 0x7f63140b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 --2- 172.21.15.67:0/11577517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63140afbf0 0x7f631414e060 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.893+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 >> 172.21.15.67:0/11577517 conn(0x7f631406bef0 msgr2=0x7f63140ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.894+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 shutdown_connections 2024-04-17T08:55:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:10.894+0000 7f6313200640 1 -- 172.21.15.67:0/11577517 wait complete. 2024-04-17T08:55:10.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T08:55:11.006 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-17T08:55:11.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.126+0000 7fab57e00640 1 Processor -- start 2024-04-17T08:55:11.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.127+0000 7fab57e00640 1 -- start start 2024-04-17T08:55:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab57e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab5806d940 0x7fab580b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab57e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab580cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab57e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab580d1560 con 0x7fab5815ee90 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab57e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fab5809bd00 con 0x7fab5806d940 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab57e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab580d1820 con 0x7fab580d10f0 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab56a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab5815ee90 0x7fab5806d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59442/0 (socket says 172.21.15.67:59442) 2024-04-17T08:55:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.128+0000 7fab56a00640 1 -- 172.21.15.67:0/3874047440 learned_addr learned my addr 172.21.15.67:0/3874047440 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab57400640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab580cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab56000640 1 --2- 172.21.15.67:0/3874047440 >> v2:172.21.15.187:3300/0 conn(0x7fab5806d940 0x7fab580b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987153924 0 0) 0x7fab580d1560 con 0x7fab5815ee90 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab3c003520 con 0x7fab5815ee90 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2725871306 0 0) 0x7fab3c003520 con 0x7fab5815ee90 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 msgr2=0x7fab580cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab580cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 >> v2:172.21.15.187:3300/0 conn(0x7fab5806d940 msgr2=0x7fab580b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 --2- 172.21.15.67:0/3874047440 >> v2:172.21.15.187:3300/0 conn(0x7fab5806d940 0x7fab580b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab580d1b00 con 0x7fab5815ee90 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fab48001670 con 0x7fab5815ee90 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.129+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 >> v1:172.21.15.67:6789/0 conn(0x7fab5815ee90 legacy=0x7fab5806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab3c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab3c003f30 0x7fab3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab5809bd00 con 0x7fab3c004d40 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab580d1820 con 0x7fab3c003f30 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab3c00ba10 con 0x7fab580d10f0 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab56000640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab3c003f30 0x7fab3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab56a00640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab3c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 -- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 msgr2=0x7fab3c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab3c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 -- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab3c003f30 msgr2=0x7fab3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab3c003f30 0x7fab3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab57400640 1 -- 172.21.15.67:0/3874047440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c003520 con 0x7fab3c004d40 2024-04-17T08:55:11.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.130+0000 7fab56a00640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab580d10f0 0x7fab3c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.131+0000 7fab57400640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab3c0051c0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7fab5000e170 tx=0x7fab3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f314f248fc0d2f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50039070 con 0x7fab3c004d40 2024-04-17T08:55:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab5002cbf0 con 0x7fab3c004d40 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab55600640 1 -- 172.21.15.67:0/3874047440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50009c10 con 0x7fab3c004d40 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab57e00640 1 -- 172.21.15.67:0/3874047440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 msgr2=0x7fab3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab57e00640 1 --2- 172.21.15.67:0/3874047440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab3c0051c0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7fab5000e170 tx=0x7fab3c005830 comp rx=0 tx=0).stop 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab57400640 1 -- 172.21.15.67:0/3874047440 reap_dead start 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab57e00640 1 -- 172.21.15.67:0/3874047440 shutdown_connections 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.132+0000 7fab57e00640 1 -- 172.21.15.67:0/3874047440 >> 172.21.15.67:0/3874047440 conn(0x7fab580b4090 msgr2=0x7fab580b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.133+0000 7fab57e00640 1 -- 172.21.15.67:0/3874047440 shutdown_connections 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.133+0000 7fab57e00640 1 -- 172.21.15.67:0/3874047440 wait complete. 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.134+0000 7fab57e00640 1 Processor -- start 2024-04-17T08:55:11.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.134+0000 7fab57e00640 1 -- start start 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 0x7fab580b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 0x7fab580c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab580ccbf0 con 0x7fab3c004d40 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab580d20c0 con 0x7fab580b30e0 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.135+0000 7fab57e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab580cd110 con 0x7fab5814e1a0 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51962/0 (socket says 172.21.15.67:51962) 2024-04-17T08:55:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 -- 172.21.15.67:0/1644894796 learned_addr learned my addr 172.21.15.67:0/1644894796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab57400640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 0x7fab580c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56000640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 0x7fab580b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 -- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 msgr2=0x7fab580b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 0x7fab580b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 -- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 msgr2=0x7fab580c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 0x7fab580c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 -- 172.21.15.67:0/1644894796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5814ce60 con 0x7fab3c004d40 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab57400640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 0x7fab580c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56000640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 0x7fab580b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.136+0000 7fab56a00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fab4800dae0 tx=0x7fab48007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.137+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab48019070 con 0x7fab3c004d40 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.137+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab480074e0 con 0x7fab3c004d40 2024-04-17T08:55:11.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.137+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab480049e0 con 0x7fab3c004d40 2024-04-17T08:55:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.137+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5814d100 con 0x7fab3c004d40 2024-04-17T08:55:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.137+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5814d6c0 con 0x7fab3c004d40 2024-04-17T08:55:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.138+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab580b34f0 con 0x7fab3c004d40 2024-04-17T08:55:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.139+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fab4801c070 con 0x7fab3c004d40 2024-04-17T08:55:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.140+0000 7fab4ea00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 0x7fab3806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.140+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(293..293 src has 1..293) ==== 6669+0+0 (secure 0 0 0) 0x7fab4808bde0 con 0x7fab3c004d40 2024-04-17T08:55:11.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.141+0000 7fab56000640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 0x7fab3806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.142+0000 7fab56000640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 0x7fab3806ef00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab580b3f30 tx=0x7fab44008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:11.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.145+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fab580b34f0 con 0x7fab3c004d40 2024-04-17T08:55:11.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.389+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fab58068a10 con 0x7fab3c004d40 2024-04-17T08:55:11.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.894+0000 7fab4ea00640 1 -- 172.21.15.67:0/1644894796 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v153) ==== 104+0+0 (secure 0 0 0) 0x7fab48059c30 con 0x7fab3c004d40 2024-04-17T08:55:11.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 msgr2=0x7fab3806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 0x7fab3806ef00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab580b3f30 tx=0x7fab44008040 comp rx=0 tx=0).stop 2024-04-17T08:55:11.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 msgr2=0x7fab5814c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fab4800dae0 tx=0x7fab48007eb0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 shutdown_connections 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab3806ca40 0x7fab3806ef00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab580b30e0 0x7fab580c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab5814e1a0 0x7fab580b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 --2- 172.21.15.67:0/1644894796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab3c004d40 0x7fab5814c920 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:11.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 >> 172.21.15.67:0/1644894796 conn(0x7fab580b4090 msgr2=0x7fab5815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:11.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 shutdown_connections 2024-04-17T08:55:11.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:11.897+0000 7fab57e00640 1 -- 172.21.15.67:0/1644894796 wait complete. 2024-04-17T08:55:11.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T08:55:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.142+0000 7fab99200640 1 Processor -- start 2024-04-17T08:55:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.142+0000 7fab99200640 1 -- start start 2024-04-17T08:55:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab99200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab9415e840 0x7fab940cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab99200640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fab940b7980 0x7fab940d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab99200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab940cf850 con 0x7fab9415ec20 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab99200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fab940cfb30 con 0x7fab9415e840 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab99200640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fab940cfe10 con 0x7fab940b7980 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab93400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab9415e840 0x7fab940cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab93400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab9415e840 0x7fab940cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51968/0 (socket says 172.21.15.67:51968) 2024-04-17T08:55:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.144+0000 7fab93400640 1 -- 172.21.15.67:0/529148379 learned_addr learned my addr 172.21.15.67:0/529148379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.145+0000 7fab92a00640 1 --2- 172.21.15.67:0/529148379 >> v2:172.21.15.187:3300/0 conn(0x7fab940b7980 0x7fab940d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.145+0000 7fab93400640 1 -- 172.21.15.67:0/529148379 >> v2:172.21.15.187:3300/0 conn(0x7fab940b7980 msgr2=0x7fab940d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.145+0000 7fab93400640 1 --2- 172.21.15.67:0/529148379 >> v2:172.21.15.187:3300/0 conn(0x7fab940b7980 0x7fab940d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.145+0000 7fab93400640 1 -- 172.21.15.67:0/529148379 >> v1:172.21.15.187:6789/0 conn(0x7fab9415ec20 legacy=0x7fab940b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.145+0000 7fab93400640 1 -- 172.21.15.67:0/529148379 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab940d2870 con 0x7fab9415e840 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.146+0000 7fab93400640 1 --2- 172.21.15.67:0/529148379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab9415e840 0x7fab940cefb0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fab940b7250 tx=0x7fab8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ef737be952e06 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.146+0000 7fab92000640 1 -- 172.21.15.67:0/529148379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8402ca80 con 0x7fab9415e840 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.146+0000 7fab92000640 1 -- 172.21.15.67:0/529148379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab8402cc20 con 0x7fab9415e840 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab92000640 1 -- 172.21.15.67:0/529148379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab840343f0 con 0x7fab9415e840 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 -- 172.21.15.67:0/529148379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab9415e840 msgr2=0x7fab940cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 --2- 172.21.15.67:0/529148379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab9415e840 0x7fab940cefb0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7fab940b7250 tx=0x7fab8402bee0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 -- 172.21.15.67:0/529148379 shutdown_connections 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 --2- 172.21.15.67:0/529148379 >> v2:172.21.15.187:3300/0 conn(0x7fab940b7980 0x7fab940d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 --2- 172.21.15.67:0/529148379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab9415e840 0x7fab940cefb0 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 -- 172.21.15.67:0/529148379 >> 172.21.15.67:0/529148379 conn(0x7fab9406cae0 msgr2=0x7fab9406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:12.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 -- 172.21.15.67:0/529148379 shutdown_connections 2024-04-17T08:55:12.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.147+0000 7fab99200640 1 -- 172.21.15.67:0/529148379 wait complete. 2024-04-17T08:55:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.150+0000 7fab99200640 1 Processor -- start 2024-04-17T08:55:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.150+0000 7fab99200640 1 -- start start 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 0x7fab94159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 0x7fab940b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab94163200 con 0x7fab940b7980 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fab940cf4f0 con 0x7fab9415e840 2024-04-17T08:55:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.151+0000 7fab99200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fab94163380 con 0x7fab9415ec20 2024-04-17T08:55:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51984/0 (socket says 172.21.15.67:51984) 2024-04-17T08:55:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 -- 172.21.15.67:0/3762279316 learned_addr learned my addr 172.21.15.67:0/3762279316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93e00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 0x7fab940b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab92a00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 0x7fab94159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 -- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 msgr2=0x7fab940b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 0x7fab940b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 -- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 msgr2=0x7fab94159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 0x7fab94159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93400640 1 -- 172.21.15.67:0/3762279316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab941f5b20 con 0x7fab940b7980 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab93e00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 0x7fab940b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.152+0000 7fab92a00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 0x7fab94159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab93400640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7fab84036040 tx=0x7fab84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:12.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab84043020 con 0x7fab940b7980 2024-04-17T08:55:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab8403e450 con 0x7fab940b7980 2024-04-17T08:55:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab84009510 con 0x7fab940b7980 2024-04-17T08:55:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9415a1f0 con 0x7fab940b7980 2024-04-17T08:55:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.153+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab940cd410 con 0x7fab940b7980 2024-04-17T08:55:12.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.154+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab94068a10 con 0x7fab940b7980 2024-04-17T08:55:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.155+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fab84046020 con 0x7fab940b7980 2024-04-17T08:55:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.156+0000 7fab8b400640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 0x7fab7406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.156+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(293..293 src has 1..293) ==== 6669+0+0 (secure 0 0 0) 0x7fab840ae6d0 con 0x7fab940b7980 2024-04-17T08:55:12.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.157+0000 7fab92a00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 0x7fab7406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:12.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.158+0000 7fab92a00640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 0x7fab7406ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fab940cb1f0 tx=0x7fab80008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:12.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.161+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fab8404d070 con 0x7fab940b7980 2024-04-17T08:55:12.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.367+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fab9415e0d0 con 0x7fab940b7980 2024-04-17T08:55:12.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.897+0000 7fab8b400640 1 -- 172.21.15.67:0/3762279316 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v294) ==== 181+0+0 (secure 0 0 0) 0x7fab9415a1f0 con 0x7fab940b7980 2024-04-17T08:55:12.896 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T08:55:12.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 msgr2=0x7fab7406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:12.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 0x7fab7406ef00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fab940cb1f0 tx=0x7fab80008040 comp rx=0 tx=0).stop 2024-04-17T08:55:12.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 msgr2=0x7fab941f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:12.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7fab84036040 tx=0x7fab84004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 shutdown_connections 2024-04-17T08:55:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fab7406ca40 0x7fab7406ef00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fab9415ec20 0x7fab940b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab9415e840 0x7fab94159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 --2- 172.21.15.67:0/3762279316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab940b7980 0x7fab941f55e0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:12.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.901+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 >> 172.21.15.67:0/3762279316 conn(0x7fab9406cae0 msgr2=0x7fab9415f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:12.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.902+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 shutdown_connections 2024-04-17T08:55:12.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:12.902+0000 7fab99200640 1 -- 172.21.15.67:0/3762279316 wait complete. 2024-04-17T08:55:12.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T08:55:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.159+0000 7f0620a00640 1 Processor -- start 2024-04-17T08:55:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.159+0000 7f0620a00640 1 -- start start 2024-04-17T08:55:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f0620a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 0x7f061c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f0620a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f061c146080 0x7f061c160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f0620a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c146510 con 0x7f061c0cd2f0 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f0620a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f061c0cfa60 con 0x7f061c146080 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f0620a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f061c161000 con 0x7f061c0cd6d0 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f061a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f061c0cd2f0 0x7f061c0cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59448/0 (socket says 172.21.15.67:59448) 2024-04-17T08:55:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.161+0000 7f061a800640 1 -- 172.21.15.67:0/2639279460 learned_addr learned my addr 172.21.15.67:0/2639279460 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f061b200640 1 --2- 172.21.15.67:0/2639279460 >> v2:172.21.15.67:3300/0 conn(0x7f061c146080 0x7f061c160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f0619400640 1 -- 172.21.15.67:0/2639279460 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589702995 0 0) 0x7f061c146510 con 0x7f061c0cd2f0 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f0619400640 1 -- 172.21.15.67:0/2639279460 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0600003520 con 0x7f061c0cd2f0 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f0619e00640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 0x7f061c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f061b200640 1 -- 172.21.15.67:0/2639279460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 msgr2=0x7f061c145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f061b200640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 0x7f061c145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f061b200640 1 -- 172.21.15.67:0/2639279460 >> v1:172.21.15.67:6789/0 conn(0x7f061c0cd2f0 legacy=0x7f061c0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f061b200640 1 -- 172.21.15.67:0/2639279460 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0ce870 con 0x7f061c146080 2024-04-17T08:55:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.162+0000 7f0619e00640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 0x7f061c145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.163+0000 7f061b200640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c146080 0x7f061c160360 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f0614009d90 tx=0x7f061402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de23837ce2906d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0619400640 1 -- 172.21.15.67:0/2639279460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614039070 con 0x7f061c146080 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0619400640 1 -- 172.21.15.67:0/2639279460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f061402cc50 con 0x7f061c146080 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0619400640 1 -- 172.21.15.67:0/2639279460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614034440 con 0x7f061c146080 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 -- 172.21.15.67:0/2639279460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c146080 msgr2=0x7f061c160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c146080 0x7f061c160360 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f0614009d90 tx=0x7f061402bea0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 -- 172.21.15.67:0/2639279460 shutdown_connections 2024-04-17T08:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c146080 0x7f061c160360 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 --2- 172.21.15.67:0/2639279460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c0cd6d0 0x7f061c145b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.164+0000 7f0620a00640 1 -- 172.21.15.67:0/2639279460 >> 172.21.15.67:0/2639279460 conn(0x7f061c06b8c0 msgr2=0x7f061c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.165+0000 7f0620a00640 1 -- 172.21.15.67:0/2639279460 shutdown_connections 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.165+0000 7f0620a00640 1 -- 172.21.15.67:0/2639279460 wait complete. 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.166+0000 7f0620a00640 1 Processor -- start 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.166+0000 7f0620a00640 1 -- start start 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 0x7f061c0adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c146080 0x7f061c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f061c1608a0 con 0x7f061c0cd2f0 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0cee30 con 0x7f061c0cd6d0 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.167+0000 7f0620a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f061c160a20 con 0x7f061c146080 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52002/0 (socket says 172.21.15.67:52002) 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 -- 172.21.15.67:0/911337873 learned_addr learned my addr 172.21.15.67:0/911337873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 -- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c146080 msgr2=0x7f061c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061b200640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c146080 0x7f061c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f0619e00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 0x7f061c0adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c146080 0x7f061c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 -- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 msgr2=0x7f061c0adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 0x7f061c0adb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f061a800640 1 -- 172.21.15.67:0/911337873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0ae5c0 con 0x7f061c0cd2f0 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.168+0000 7f0619e00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 0x7f061c0adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f061a800640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c460 tx=0x7f060c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c017070 con 0x7f061c0cd2f0 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c005ce0 con 0x7f061c0cd2f0 2024-04-17T08:55:13.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c007290 con 0x7f061c0cd2f0 2024-04-17T08:55:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c0ae8a0 con 0x7f061c0cd2f0 2024-04-17T08:55:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.169+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c0aee60 con 0x7f061c0cd2f0 2024-04-17T08:55:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.170+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c068a10 con 0x7f061c0cd2f0 2024-04-17T08:55:13.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.171+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f060c01a070 con 0x7f061c0cd2f0 2024-04-17T08:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.172+0000 7f0612a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 0x7f05fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.172+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(294..294 src has 1..294) ==== 6231+0+0 (secure 0 0 0) 0x7f060c08b270 con 0x7f061c0cd2f0 2024-04-17T08:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.173+0000 7f0619e00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 0x7f05fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.174+0000 7f0619e00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 0x7f05fc06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0608006fd0 tx=0x7f0608008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.177+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f060c059380 con 0x7f061c0cd2f0 2024-04-17T08:55:13.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.381+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f061c068610 con 0x7f061c0cd2f0 2024-04-17T08:55:13.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.916+0000 7f0612a00640 1 -- 172.21.15.67:0/911337873 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v295) ==== 169+0+0 (secure 0 0 0) 0x7f060c058c80 con 0x7f061c0cd2f0 2024-04-17T08:55:13.915 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 msgr2=0x7f05fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 0x7f05fc06eed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0608006fd0 tx=0x7f0608008040 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 msgr2=0x7f061c0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c460 tx=0x7f060c00c920 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 shutdown_connections 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f05fc06ca10 0x7f05fc06eed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f061c146080 0x7f061c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f061c0cd6d0 0x7f061c0adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 --2- 172.21.15.67:0/911337873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0cd2f0 0x7f061c0af660 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 >> 172.21.15.67:0/911337873 conn(0x7f061c06b8c0 msgr2=0x7f061c0b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:13.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.920+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 shutdown_connections 2024-04-17T08:55:13.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:13.921+0000 7f0620a00640 1 -- 172.21.15.67:0/911337873 wait complete. 2024-04-17T08:55:13.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-04-17T08:55:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.164+0000 7f41c3600640 1 Processor -- start 2024-04-17T08:55:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.165+0000 7f41c3600640 1 -- start start 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 0x7f41bc0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41bc0ce4d0 0x7f41bc0d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc06dd10 0x7f41bc0ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f41bc071d10 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f41bc071f60 con 0x7f41bc15b860 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.166+0000 7f41c3600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc072240 con 0x7f41bc06dd10 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41bc0ce4d0 0x7f41bc0d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41bc0ce4d0 0x7f41bc0d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47252/0 (socket says 172.21.15.67:47252) 2024-04-17T08:55:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 -- 172.21.15.67:0/4246272867 learned_addr learned my addr 172.21.15.67:0/4246272867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41c0c00640 1 --2- 172.21.15.67:0/4246272867 >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 0x7f41bc0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 -- 172.21.15.67:0/4246272867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc06dd10 msgr2=0x7f41bc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 --2- 172.21.15.67:0/4246272867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc06dd10 0x7f41bc0ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 -- 172.21.15.67:0/4246272867 >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 msgr2=0x7f41bc0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 --2- 172.21.15.67:0/4246272867 >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 0x7f41bc0d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41bbe00640 1 -- 172.21.15.67:0/4246272867 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc072520 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.167+0000 7f41c0c00640 1 --2- 172.21.15.67:0/4246272867 >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 0x7f41bc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.168+0000 7f41bbe00640 1 --2- 172.21.15.67:0/4246272867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ce4d0 0x7f41bc0d0960 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f41bc072450 tx=0x7f41a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652dc7af2c6bf59d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.169+0000 7f41bb400640 1 -- 172.21.15.67:0/4246272867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8035020 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.169+0000 7f41bb400640 1 -- 172.21.15.67:0/4246272867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8031640 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.169+0000 7f41bb400640 1 -- 172.21.15.67:0/4246272867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a802d040 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.169+0000 7f41c3600640 1 -- 172.21.15.67:0/4246272867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ce4d0 msgr2=0x7f41bc0d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.169+0000 7f41c3600640 1 --2- 172.21.15.67:0/4246272867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ce4d0 0x7f41bc0d0960 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f41bc072450 tx=0x7f41a802b8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 -- 172.21.15.67:0/4246272867 shutdown_connections 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 --2- 172.21.15.67:0/4246272867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc06dd10 0x7f41bc0ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 --2- 172.21.15.67:0/4246272867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ce4d0 0x7f41bc0d0960 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 --2- 172.21.15.67:0/4246272867 >> v2:172.21.15.187:3300/0 conn(0x7f41bc15b860 0x7f41bc0d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 -- 172.21.15.67:0/4246272867 >> 172.21.15.67:0/4246272867 conn(0x7f41bc067ee0 msgr2=0x7f41bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 -- 172.21.15.67:0/4246272867 shutdown_connections 2024-04-17T08:55:14.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.170+0000 7f41c3600640 1 -- 172.21.15.67:0/4246272867 wait complete. 2024-04-17T08:55:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.172+0000 7f41c3600640 1 Processor -- start 2024-04-17T08:55:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.172+0000 7f41c3600640 1 -- start start 2024-04-17T08:55:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc0ce4d0 0x7f41bc15d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 0x7f41bc157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc15bc40 con 0x7f41bc06dd10 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc072ae0 con 0x7f41bc15b860 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c3600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41bc06f380 con 0x7f41bc0ce4d0 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47268/0 (socket says 172.21.15.67:47268) 2024-04-17T08:55:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.173+0000 7f41c0c00640 1 -- 172.21.15.67:0/551118625 learned_addr learned my addr 172.21.15.67:0/551118625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 -- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc0ce4d0 msgr2=0x7f41bc15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc0ce4d0 0x7f41bc15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 -- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 msgr2=0x7f41bc157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c1600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 0x7f41bc157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 0x7f41bc157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 -- 172.21.15.67:0/551118625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc15d9a0 con 0x7f41bc06dd10 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c1600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 0x7f41bc157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.174+0000 7f41c0c00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c7f0 tx=0x7f41b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.175+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0015070 con 0x7f41bc06dd10 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.175+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b0007820 con 0x7f41bc06dd10 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.175+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0007bd0 con 0x7f41bc06dd10 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.175+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc15dc80 con 0x7f41bc06dd10 2024-04-17T08:55:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.175+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc15e520 con 0x7f41bc06dd10 2024-04-17T08:55:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.176+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41bc15e070 con 0x7f41bc06dd10 2024-04-17T08:55:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.177+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f41b0018070 con 0x7f41bc06dd10 2024-04-17T08:55:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.178+0000 7f41b8c00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 0x7f419c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.178+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f41b008bf10 con 0x7f41bc06dd10 2024-04-17T08:55:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.179+0000 7f41bbe00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 0x7f419c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.180+0000 7f41bbe00640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 0x7f419c06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f41a8036000 tx=0x7f41a8030000 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.183+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f41b005a160 con 0x7f41bc06dd10 2024-04-17T08:55:14.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.388+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) -- 0x7f41bc0b3390 con 0x7f41bc06dd10 2024-04-17T08:55:14.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.395+0000 7f41b8c00640 1 -- 172.21.15.67:0/551118625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) ==== 115+0+0 (secure 0 0 0) 0x7f41b0059a60 con 0x7f41bc06dd10 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 msgr2=0x7f419c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 0x7f419c06eed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f41a8036000 tx=0x7f41a8030000 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 msgr2=0x7f41bc1e52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c7f0 tx=0x7f41b000ccb0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 shutdown_connections 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f419c06ca10 0x7f419c06eed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41bc15b860 0x7f41bc157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41bc0ce4d0 0x7f41bc15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 --2- 172.21.15.67:0/551118625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc06dd10 0x7f41bc1e52e0 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.397+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 >> 172.21.15.67:0/551118625 conn(0x7f41bc067ee0 msgr2=0x7f41bc0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:14.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.398+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 shutdown_connections 2024-04-17T08:55:14.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.398+0000 7f41c3600640 1 -- 172.21.15.67:0/551118625 wait complete. 2024-04-17T08:55:14.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T08:55:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.642+0000 7fd1e5000640 1 Processor -- start 2024-04-17T08:55:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.642+0000 7fd1e5000640 1 -- start start 2024-04-17T08:55:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1e5000640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1e015b860 0x7fd1e00d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1e5000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00cfcc0 con 0x7fd1e00b4f80 2024-04-17T08:55:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1e5000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e0071f60 con 0x7fd1e00ce4d0 2024-04-17T08:55:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1e5000640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e0071d10 con 0x7fd1e015b860 2024-04-17T08:55:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1de800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1e00b4f80 0x7fd1e00cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34170/0 (socket says 172.21.15.67:34170) 2024-04-17T08:55:14.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.644+0000 7fd1de800640 1 -- 172.21.15.67:0/1588910604 learned_addr learned my addr 172.21.15.67:0/1588910604 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dde00640 1 --2- 172.21.15.67:0/1588910604 >> v2:172.21.15.187:3300/0 conn(0x7fd1e015b860 0x7fd1e00d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475694682 0 0) 0x7fd1e00cfcc0 con 0x7fd1e00b4f80 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c4003520 con 0x7fd1e00b4f80 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2414961268 0 0) 0x7fd1c4003520 con 0x7fd1e00b4f80 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 >> v2:172.21.15.187:3300/0 conn(0x7fd1e015b860 msgr2=0x7fd1e00d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 --2- 172.21.15.67:0/1588910604 >> v2:172.21.15.187:3300/0 conn(0x7fd1e015b860 0x7fd1e00d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 >> v1:172.21.15.187:6789/0 conn(0x7fd1e00ce4d0 legacy=0x7fd1e00ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e0072240 con 0x7fd1e00b4f80 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fd1d80019f0 con 0x7fd1e00b4f80 2024-04-17T08:55:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 >> v1:172.21.15.67:6789/0 conn(0x7fd1e00b4f80 legacy=0x7fd1e00cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4004800 0x7fd1c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40051c0 0x7fd1c400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0071d10 con 0x7fd1c4003a00 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c400e2a0 con 0x7fd1c40051c0 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c400e4a0 con 0x7fd1c4004800 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.645+0000 7fd1dde00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1dde00640 1 -- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4004800 msgr2=0x7fd1c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1dde00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4004800 0x7fd1c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1dde00640 1 -- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40051c0 msgr2=0x7fd1c400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:14.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1de800640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40051c0 0x7fd1c400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1dde00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40051c0 0x7fd1c400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1dde00640 1 -- 172.21.15.67:0/1588910604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c4003520 con 0x7fd1c4003a00 2024-04-17T08:55:14.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1de800640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40051c0 0x7fd1c400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:14.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.646+0000 7fd1de800640 1 -- 172.21.15.67:0/1588910604 reap_dead start 2024-04-17T08:55:14.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.647+0000 7fd1dde00640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1c4003e00 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009ac0 tx=0x7fd1c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5551092322af95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.647+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0039070 con 0x7fd1c4003a00 2024-04-17T08:55:14.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.647+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d002c6b0 con 0x7fd1c4003a00 2024-04-17T08:55:14.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.647+0000 7fd1dca00640 1 -- 172.21.15.67:0/1588910604 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0009690 con 0x7fd1c4003a00 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 -- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 msgr2=0x7fd1c4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1c4003e00 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009ac0 tx=0x7fd1c4005830 comp rx=0 tx=0).stop 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 -- 172.21.15.67:0/1588910604 shutdown_connections 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 --2- 172.21.15.67:0/1588910604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1c4003e00 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 -- 172.21.15.67:0/1588910604 >> 172.21.15.67:0/1588910604 conn(0x7fd1e0067ee0 msgr2=0x7fd1e006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 -- 172.21.15.67:0/1588910604 shutdown_connections 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.648+0000 7fd1e5000640 1 -- 172.21.15.67:0/1588910604 wait complete. 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.650+0000 7fd1e5000640 1 Processor -- start 2024-04-17T08:55:14.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.650+0000 7fd1e5000640 1 -- start start 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1e01478d0 0x7fd1e0149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1e0147ee0 0x7fd1e0148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0072ae0 con 0x7fd1c4003a00 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0072800 con 0x7fd1e01478d0 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1e5000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e015bc40 con 0x7fd1e0147ee0 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47282/0 (socket says 172.21.15.67:47282) 2024-04-17T08:55:14.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.651+0000 7fd1dde00640 1 -- 172.21.15.67:0/2783246915 learned_addr learned my addr 172.21.15.67:0/2783246915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dd400640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1e01478d0 0x7fd1e0149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 -- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1e0147ee0 msgr2=0x7fd1e0148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1de800640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1e0147ee0 0x7fd1e0148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1e0147ee0 0x7fd1e0148360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 -- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1e01478d0 msgr2=0x7fd1e0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1e01478d0 0x7fd1e0149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 -- 172.21.15.67:0/2783246915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00ad8d0 con 0x7fd1c4003a00 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1dde00640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0037040 tx=0x7fd1d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.652+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0039050 con 0x7fd1c4003a00 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.653+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d0045070 con 0x7fd1c4003a00 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.653+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00044e0 con 0x7fd1c4003a00 2024-04-17T08:55:14.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.653+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e014a7c0 con 0x7fd1c4003a00 2024-04-17T08:55:14.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.653+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e014ad80 con 0x7fd1c4003a00 2024-04-17T08:55:14.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.654+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e0068ca0 con 0x7fd1c4003a00 2024-04-17T08:55:14.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.655+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fd1d0003a50 con 0x7fd1c4003a00 2024-04-17T08:55:14.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.656+0000 7fd1d6000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 0x7fd1c006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:14.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.656+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fd1d00ad800 con 0x7fd1c4003a00 2024-04-17T08:55:14.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.657+0000 7fd1dd400640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 0x7fd1c006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:14.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.658+0000 7fd1dd400640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 0x7fd1c006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1e015b0b0 tx=0x7fd1cc009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:14.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.661+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fd1d004e020 con 0x7fd1c4003a00 2024-04-17T08:55:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.865+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fd1e006e090 con 0x7fd1c4003a00 2024-04-17T08:55:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.871+0000 7fd1d6000640 1 -- 172.21.15.67:0/2783246915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v98) ==== 104+0+0 (secure 0 0 0) 0x7fd1d007bac0 con 0x7fd1c4003a00 2024-04-17T08:55:14.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 msgr2=0x7fd1c006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 0x7fd1c006ef00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1e015b0b0 tx=0x7fd1cc009040 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 msgr2=0x7fd1e00af300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0037040 tx=0x7fd1d0004730 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 shutdown_connections 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fd1c006ca40 0x7fd1c006ef00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1e0147ee0 0x7fd1e0148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1e01478d0 0x7fd1e0149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 --2- 172.21.15.67:0/2783246915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1c4003a00 0x7fd1e00af300 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 >> 172.21.15.67:0/2783246915 conn(0x7fd1e0067ee0 msgr2=0x7fd1e00b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 shutdown_connections 2024-04-17T08:55:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:14.874+0000 7fd1e5000640 1 -- 172.21.15.67:0/2783246915 wait complete. 2024-04-17T08:55:14.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-04-17T08:55:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.104+0000 7f173dc00640 1 Processor -- start 2024-04-17T08:55:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.104+0000 7f173dc00640 1 -- start start 2024-04-17T08:55:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380d33e0 0x7f17380cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17380cf8e0 0x7f17380b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173dc00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17380d00b0 con 0x7f173815ecb0 2024-04-17T08:55:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17380d0c30 con 0x7f17380cf8e0 2024-04-17T08:55:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f17380d0390 con 0x7f17380d33e0 2024-04-17T08:55:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17380cf8e0 0x7f17380b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17380cf8e0 0x7f17380b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47284/0 (socket says 172.21.15.67:47284) 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.106+0000 7f173d200640 1 -- 172.21.15.67:0/1342216173 learned_addr learned my addr 172.21.15.67:0/1342216173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.107+0000 7f1737400640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380d33e0 0x7f17380cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.107+0000 7f173d200640 1 -- 172.21.15.67:0/1342216173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380d33e0 msgr2=0x7f17380cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.107+0000 7f173d200640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380d33e0 0x7f17380cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.107+0000 7f173d200640 1 -- 172.21.15.67:0/1342216173 >> v1:172.21.15.187:6789/0 conn(0x7f173815ecb0 legacy=0x7f17380d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.107+0000 7f173d200640 1 -- 172.21.15.67:0/1342216173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380d0670 con 0x7f17380cf8e0 2024-04-17T08:55:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.108+0000 7f173d200640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17380cf8e0 0x7f17380b3fc0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f1730009e50 tx=0x7f173002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129b6a32e845207b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.109+0000 7f1736a00640 1 -- 172.21.15.67:0/1342216173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173002ca70 con 0x7f17380cf8e0 2024-04-17T08:55:15.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.109+0000 7f1736a00640 1 -- 172.21.15.67:0/1342216173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f173002cc10 con 0x7f17380cf8e0 2024-04-17T08:55:15.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.109+0000 7f1736a00640 1 -- 172.21.15.67:0/1342216173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300343f0 con 0x7f17380cf8e0 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.109+0000 7f173dc00640 1 -- 172.21.15.67:0/1342216173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17380cf8e0 msgr2=0x7f17380b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.109+0000 7f173dc00640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17380cf8e0 0x7f17380b3fc0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f1730009e50 tx=0x7f173002bed0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 -- 172.21.15.67:0/1342216173 shutdown_connections 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17380cf8e0 0x7f17380b3fc0 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 --2- 172.21.15.67:0/1342216173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380d33e0 0x7f17380cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 -- 172.21.15.67:0/1342216173 >> 172.21.15.67:0/1342216173 conn(0x7f173808bff0 msgr2=0x7f17380d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 -- 172.21.15.67:0/1342216173 shutdown_connections 2024-04-17T08:55:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.110+0000 7f173dc00640 1 -- 172.21.15.67:0/1342216173 wait complete. 2024-04-17T08:55:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.112+0000 7f173dc00640 1 Processor -- start 2024-04-17T08:55:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.112+0000 7f173dc00640 1 -- start start 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 0x7f1738157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 0x7f17380b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f173815f090 con 0x7f173815ecb0 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f17380cfd30 con 0x7f17380d33e0 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173dc00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f173815f210 con 0x7f17380cf8e0 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47288/0 (socket says 172.21.15.67:47288) 2024-04-17T08:55:15.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 -- 172.21.15.67:0/3800559737 learned_addr learned my addr 172.21.15.67:0/3800559737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f1737400640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 0x7f17380b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f1737e00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 0x7f1738157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 -- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 msgr2=0x7f1738157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 0x7f1738157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 -- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 msgr2=0x7f17380b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 0x7f17380b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f173d200640 1 -- 172.21.15.67:0/3800559737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380b7370 con 0x7f173815ecb0 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f1737400640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 0x7f17380b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.113+0000 7f1737e00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 0x7f1738157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f173d200640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f1730009f70 tx=0x7f1730004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730043020 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f173003e450 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17380b7650 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730009510 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.114+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17380b7c10 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.115+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1738068a10 con 0x7f173815ecb0 2024-04-17T08:55:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.116+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f1730046020 con 0x7f173815ecb0 2024-04-17T08:55:15.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.117+0000 7f172fe00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 0x7f171806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.117+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f17300ae530 con 0x7f173815ecb0 2024-04-17T08:55:15.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.118+0000 7f1737e00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 0x7f171806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.119+0000 7f1737e00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 0x7f171806eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17280015e0 tx=0x7f1728009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.122+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f173004d030 con 0x7f173815ecb0 2024-04-17T08:55:15.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.328+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) -- 0x7f17380cf490 con 0x7f173815ecb0 2024-04-17T08:55:15.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.335+0000 7f172fe00640 1 -- 172.21.15.67:0/3800559737 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v99) ==== 120+0+0 (secure 0 0 0) 0x7f173007c800 con 0x7f173815ecb0 2024-04-17T08:55:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.338+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 msgr2=0x7f171806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.338+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 0x7f171806eed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17280015e0 tx=0x7f1728009040 comp rx=0 tx=0).stop 2024-04-17T08:55:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.338+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 msgr2=0x7f17380b9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.338+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f1730009f70 tx=0x7f1730004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 shutdown_connections 2024-04-17T08:55:15.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f171806ca10 0x7f171806eed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f173815ecb0 0x7f17380b9d10 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17380d33e0 0x7f17380b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 --2- 172.21.15.67:0/3800559737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f17380cf8e0 0x7f1738157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 >> 172.21.15.67:0/3800559737 conn(0x7f173808bff0 msgr2=0x7f17380d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 shutdown_connections 2024-04-17T08:55:15.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.339+0000 7f173dc00640 1 -- 172.21.15.67:0/3800559737 wait complete. 2024-04-17T08:55:15.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-04-17T08:55:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.572+0000 7f402d600640 1 Processor -- start 2024-04-17T08:55:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.572+0000 7f402d600640 1 -- start start 2024-04-17T08:55:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f402d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f4028142380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f402d600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40280cd4a0 con 0x7f4028160280 2024-04-17T08:55:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f402d600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40281607f0 con 0x7f402806b8c0 2024-04-17T08:55:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f402d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f402815ce30 con 0x7f4028141ed0 2024-04-17T08:55:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f4027400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4028160280 0x7f402806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34178/0 (socket says 172.21.15.67:34178) 2024-04-17T08:55:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.574+0000 7f4027400640 1 -- 172.21.15.67:0/522714096 learned_addr learned my addr 172.21.15.67:0/522714096 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:15.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f402cc00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f4028142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573168913 0 0) 0x7f40280cd4a0 con 0x7f4028160280 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f400c003520 con 0x7f4028160280 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4083180206 0 0) 0x7f400c003520 con 0x7f4028160280 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 msgr2=0x7f4028142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f4028142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 >> v1:172.21.15.187:6789/0 conn(0x7f402806b8c0 legacy=0x7f40280cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4028160ad0 con 0x7f4028160280 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f4014001670 con 0x7f4028160280 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 >> v1:172.21.15.67:6789/0 conn(0x7f4028160280 legacy=0x7f402806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f400c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c003fd0 0x7f400c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f400c004d40 0x7f400c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f402815ce30 con 0x7f400c003fd0 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.575+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f400c00bb20 con 0x7f400c004d40 2024-04-17T08:55:15.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f400c00bd20 con 0x7f4028141ed0 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c003fd0 0x7f400c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027e00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f400c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f402cc00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f400c004d40 0x7f400c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 -- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 msgr2=0x7f400c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4028141ed0 0x7f400c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 -- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f400c004d40 msgr2=0x7f400c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f400c004d40 0x7f400c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f4027400640 1 -- 172.21.15.67:0/522714096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c003520 con 0x7f400c003fd0 2024-04-17T08:55:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.576+0000 7f402cc00640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f400c004d40 0x7f400c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:15.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.577+0000 7f4027400640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c003fd0 0x7f400c004800 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f401400c210 tx=0x7f400c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9459df5d5f49cd3e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014007d30 con 0x7f400c003fd0 2024-04-17T08:55:15.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4014030070 con 0x7f400c003fd0 2024-04-17T08:55:15.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f4026a00640 1 -- 172.21.15.67:0/522714096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4014005d70 con 0x7f400c003fd0 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 -- 172.21.15.67:0/522714096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c003fd0 msgr2=0x7f400c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 --2- 172.21.15.67:0/522714096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c003fd0 0x7f400c004800 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f401400c210 tx=0x7f400c005830 comp rx=0 tx=0).stop 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402cc00640 1 -- 172.21.15.67:0/522714096 reap_dead start 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 -- 172.21.15.67:0/522714096 shutdown_connections 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 -- 172.21.15.67:0/522714096 >> 172.21.15.67:0/522714096 conn(0x7f4028067d10 msgr2=0x7f40280b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 -- 172.21.15.67:0/522714096 shutdown_connections 2024-04-17T08:55:15.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.578+0000 7f402d600640 1 -- 172.21.15.67:0/522714096 wait complete. 2024-04-17T08:55:15.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.580+0000 7f402d600640 1 Processor -- start 2024-04-17T08:55:15.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.580+0000 7f402d600640 1 -- start start 2024-04-17T08:55:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f400c003fd0 0x7f4028156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 0x7f40281f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f402813f9b0 con 0x7f402806e8d0 2024-04-17T08:55:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4028161090 con 0x7f402806f9d0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.581+0000 7f402d600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f402813fb30 con 0x7f400c003fd0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47306/0 (socket says 172.21.15.67:47306) 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 -- 172.21.15.67:0/886384368 learned_addr learned my addr 172.21.15.67:0/886384368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f4027e00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f400c003fd0 0x7f4028156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f4027400640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 0x7f40281f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 -- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f400c003fd0 msgr2=0x7f4028156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f400c003fd0 0x7f4028156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 -- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 msgr2=0x7f40281f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 0x7f40281f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f402cc00640 1 -- 172.21.15.67:0/886384368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4028142fc0 con 0x7f402806e8d0 2024-04-17T08:55:15.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.582+0000 7f4027400640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 0x7f40281f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f402cc00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f40280b00d0 tx=0x7f402000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402001c070 con 0x7f402806e8d0 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402000fe70 con 0x7f402806e8d0 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4028143260 con 0x7f402806e8d0 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020017620 con 0x7f402806e8d0 2024-04-17T08:55:15.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.583+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402806ddf0 con 0x7f402806e8d0 2024-04-17T08:55:15.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.584+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4028068a10 con 0x7f402806e8d0 2024-04-17T08:55:15.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.586+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f402001f070 con 0x7f402806e8d0 2024-04-17T08:55:15.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.586+0000 7f401fe00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 0x7f400806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.587+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f402008be20 con 0x7f402806e8d0 2024-04-17T08:55:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.587+0000 7f4027e00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 0x7f400806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:15.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.588+0000 7f4027e00640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 0x7f400806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.595+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f4020012070 con 0x7f402806e8d0 2024-04-17T08:55:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.872+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7f402815fa60 con 0x7f402806e8d0 2024-04-17T08:55:15.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.922+0000 7f401fe00640 1 -- 172.21.15.67:0/886384368 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3405) ==== 201+0+0 (secure 0 0 0) 0x7f40200598f0 con 0x7f402806e8d0 2024-04-17T08:55:15.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 msgr2=0x7f400806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 0x7f400806eed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).stop 2024-04-17T08:55:15.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 msgr2=0x7f402815e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:15.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f40280b00d0 tx=0x7f402000f4a0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 shutdown_connections 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f400806ca10 0x7f400806eed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f402806e8d0 0x7f402815e340 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f402806f9d0 0x7f40281f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 --2- 172.21.15.67:0/886384368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f400c003fd0 0x7f4028156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.925+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 >> 172.21.15.67:0/886384368 conn(0x7f4028067d10 msgr2=0x7f40280cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.926+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 shutdown_connections 2024-04-17T08:55:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:15.926+0000 7f402d600640 1 -- 172.21.15.67:0/886384368 wait complete. 2024-04-17T08:55:15.987 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/test_snap_schedules.py", line 683, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-04-17T08:55:15.988 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue().strip() 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: r.wait() 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 39: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner:Ran 15 tests in 6036.119s 2024-04-17T08:55:15.989 INFO:tasks.cephfs_test_runner: 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/test_snap_schedules.py", line 683, in test_snap_schedule_subvol_and_group_arguments_01 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue().strip() 2024-04-17T08:55:15.990 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: r.wait() 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 39: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-04-17T08:55:15.991 INFO:tasks.cephfs_test_runner: 2024-04-17T08:55:15.992 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-17T08:55:16.205 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=feb1efc3644146dab33b42a18eba8462 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a8c84f5997950fd7d008204839253291d52e6eda/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-04-17T08:55:16.207 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-17T08:55:16.218 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-17T08:55:16.289 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-17T08:55:16.290 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:55:16.290 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:16.356 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:55:16.356 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T08:55:16.356 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:16.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052777,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T08:55:16.412 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:55:16.412 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:16.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:55:16.472 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-17T08:55:16.472 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T08:55:16.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052761,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T08:55:16.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.524 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:55:16.583 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T08:55:16.584 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T08:55:16.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T08:55:16.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T08:55:16.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T08:55:16.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.902 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T08:55:16.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:16.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T08:55:17.088 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-17T08:55:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:55:17.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.093 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T08:55:17.153 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-17T08:55:17.153 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-17T08:55:17.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-17T08:55:17.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-17T08:55:17.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-17T08:55:17.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.449 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-17T08:55:17.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T08:55:17.491 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-17T08:55:17.616 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-17T08:55:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:55:17.619 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-17T08:55:17.629 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-17T08:55:17.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.868+0000 7fe1e1800640 1 Processor -- start 2024-04-17T08:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.869+0000 7fe1e1800640 1 -- start start 2024-04-17T08:55:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.870+0000 7fe1e1800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc161e70 0x7fe1dc0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.870+0000 7fe1e1800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.870+0000 7fe1e1800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1dc0cc500 con 0x7fe1dc0cce90 2024-04-17T08:55:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1e1800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1dc0c71e0 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1e1800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1dc0cc7a0 con 0x7fe1dc161e70 2024-04-17T08:55:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1da000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1da000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47316/0 (socket says 172.21.15.67:47316) 2024-04-17T08:55:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1da000640 1 -- 172.21.15.67:0/3072584158 learned_addr learned my addr 172.21.15.67:0/3072584158 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1daa00640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc161e70 0x7fe1dc0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1da000640 1 -- 172.21.15.67:0/3072584158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc161e70 msgr2=0x7fe1dc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.871+0000 7fe1da000640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc161e70 0x7fe1dc0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.872+0000 7fe1da000640 1 -- 172.21.15.67:0/3072584158 >> v1:172.21.15.187:6789/0 conn(0x7fe1dc0cce90 legacy=0x7fe1dc0cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.872+0000 7fe1da000640 1 -- 172.21.15.67:0/3072584158 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc0cca80 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.873+0000 7fe1da000640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009e50 tx=0x7fe1c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4481be2031cca21e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.873+0000 7fe1d9600640 1 -- 172.21.15.67:0/3072584158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c402ca70 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.873+0000 7fe1d9600640 1 -- 172.21.15.67:0/3072584158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c402cc10 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.873+0000 7fe1d9600640 1 -- 172.21.15.67:0/3072584158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40343f0 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 -- 172.21.15.67:0/3072584158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 msgr2=0x7fe1dc162fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009e50 tx=0x7fe1c402bed0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 -- 172.21.15.67:0/3072584158 shutdown_connections 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc162fe0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 --2- 172.21.15.67:0/3072584158 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc161e70 0x7fe1dc0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 -- 172.21.15.67:0/3072584158 >> 172.21.15.67:0/3072584158 conn(0x7fe1dc06bf50 msgr2=0x7fe1dc0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 -- 172.21.15.67:0/3072584158 shutdown_connections 2024-04-17T08:55:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.874+0000 7fe1e1800640 1 -- 172.21.15.67:0/3072584158 wait complete. 2024-04-17T08:55:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.876+0000 7fe1e1800640 1 Processor -- start 2024-04-17T08:55:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.876+0000 7fe1e1800640 1 -- start start 2024-04-17T08:55:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc0cce90 0x7fe1dc150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1dc161e70 0x7fe1dc154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc0c5630 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc0c54b0 con 0x7fe1dc161e70 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.877+0000 7fe1e1800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe1dc0c57b0 con 0x7fe1dc0cce90 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47330/0 (socket says 172.21.15.67:47330) 2024-04-17T08:55:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 -- 172.21.15.67:0/2883780732 learned_addr learned my addr 172.21.15.67:0/2883780732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1daa00640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc0cce90 0x7fe1dc150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 -- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc0cce90 msgr2=0x7fe1dc150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc0cce90 0x7fe1dc150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 -- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1dc161e70 msgr2=0x7fe1dc154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1dc161e70 0x7fe1dc154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.878+0000 7fe1da000640 1 -- 172.21.15.67:0/2883780732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc150870 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1da000640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009e50 tx=0x7fe1c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4043020 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c403e450 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4009510 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc06f120 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.879+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc06f6e0 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.880+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1dc068b90 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.881+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fe1c4046020 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.882+0000 7fe1d2a00640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 0x7fe1b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:17.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.882+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fe1c40ae650 con 0x7fe1dc0ce4a0 2024-04-17T08:55:17.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.883+0000 7fe1daa00640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 0x7fe1b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:17.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.884+0000 7fe1daa00640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 0x7fe1b806eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006fd0 tx=0x7fe1cc008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:17.887+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe1c404d030 con 0x7fe1dc0ce4a0 2024-04-17T08:55:18.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.093+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1dc0b2510 con 0x7fe1b806ca10 2024-04-17T08:55:18.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.095+0000 7fe1d2a00640 1 -- 172.21.15.67:0/2883780732 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13539 (secure 0 0 0) 0x7fe1dc0b2510 con 0x7fe1b806ca10 2024-04-17T08:55:18.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:18.096 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 msgr2=0x7fe1b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 0x7fe1b806eed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc006fd0 tx=0x7fe1cc008040 comp rx=0 tx=0).stop 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 msgr2=0x7fe1dc06e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009e50 tx=0x7fe1c4004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 shutdown_connections 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe1b806ca10 0x7fe1b806eed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1dc161e70 0x7fe1dc154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1dc0ce4a0 0x7fe1dc06e760 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 --2- 172.21.15.67:0/2883780732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe1dc0cce90 0x7fe1dc150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 >> 172.21.15.67:0/2883780732 conn(0x7fe1dc06bf50 msgr2=0x7fe1dc162460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:18.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 shutdown_connections 2024-04-17T08:55:18.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.098+0000 7fe1e1800640 1 -- 172.21.15.67:0/2883780732 wait complete. 2024-04-17T08:55:18.119 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-17T08:55:16.855992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":465592,"kb_used_data":31576,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754509128,"statfs":{"total":773094113280,"available":772617347072,"internally_reserved":0,"allocated":32333824,"data_stored":17342582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-782129,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-430,"num_read_kb":-263,"num_write":-418,"num_write_kb":-2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001771"},"pg_stats":[{"pgid":"1.0","version":"291'2793","reported_seq":4236,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-17T08:55:12.410539+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:55:12.410539+0000","last_peered":"2024-04-17T08:55:12.410539+0000","last_clean":"2024-04-17T08:55:12.410539+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:55:12.410539+0000","last_undegraded":"2024-04-17T08:55:12.410539+0000","last_fullsized":"2024-04-17T08:55:12.410539+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543344,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53792,"kb_used_data":4564,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318048,"statfs":{"total":96636764160,"available":96581681152,"internally_reserved":0,"allocated":4673536,"data_stored":2804928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543344,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61212,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310628,"statfs":{"total":96636764160,"available":96574083072,"internally_reserved":0,"allocated":3751936,"data_stored":1880981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543343,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52912,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318928,"statfs":{"total":96636764160,"available":96582582272,"internally_reserved":0,"allocated":3772416,"data_stored":1888149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543343,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85040,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286800,"statfs":{"total":96636764160,"available":96549683200,"internally_reserved":0,"allocated":4034560,"data_stored":2157060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543345,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52828,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319012,"statfs":{"total":96636764160,"available":96582668288,"internally_reserved":0,"allocated":3686400,"data_stored":1821803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543344,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3702784,"data_stored":1828300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543345,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49068,"kb_used_data":4000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322772,"statfs":{"total":96636764160,"available":96586518528,"internally_reserved":0,"allocated":4096000,"data_stored":2221448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543343,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57896,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94313944,"statfs":{"total":96636764160,"available":96577478656,"internally_reserved":0,"allocated":4616192,"data_stored":2739913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:55:18.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:55:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.342+0000 7fb67d200640 1 Processor -- start 2024-04-17T08:55:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.342+0000 7fb67d200640 1 -- start start 2024-04-17T08:55:18.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.343+0000 7fb67d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6780d2ab0 0x7fb6780b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.343+0000 7fb67d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6780d3020 con 0x7fb6780cf640 2024-04-17T08:55:18.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb67d200640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6780d3300 con 0x7fb67815ed10 2024-04-17T08:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb67d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb6780d3c30 con 0x7fb6780d2ab0 2024-04-17T08:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb676800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6780d2ab0 0x7fb6780b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb676800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6780d2ab0 0x7fb6780b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47342/0 (socket says 172.21.15.67:47342) 2024-04-17T08:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb676800640 1 -- 172.21.15.67:0/1914100894 learned_addr learned my addr 172.21.15.67:0/1914100894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.344+0000 7fb676800640 1 -- 172.21.15.67:0/1914100894 >> v1:172.21.15.187:6789/0 conn(0x7fb67815ed10 legacy=0x7fb6780d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.345+0000 7fb676800640 1 -- 172.21.15.67:0/1914100894 >> v1:172.21.15.67:6789/0 conn(0x7fb6780cf640 legacy=0x7fb6780cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.345+0000 7fb676800640 1 -- 172.21.15.67:0/1914100894 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6780d35e0 con 0x7fb6780d2ab0 2024-04-17T08:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.346+0000 7fb676800640 1 --2- 172.21.15.67:0/1914100894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780d2ab0 0x7fb6780b7b10 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb67002c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f9b64df67c6d0b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.346+0000 7fb674a00640 1 -- 172.21.15.67:0/1914100894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67002cc30 con 0x7fb6780d2ab0 2024-04-17T08:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.346+0000 7fb674a00640 1 -- 172.21.15.67:0/1914100894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb67002cdd0 con 0x7fb6780d2ab0 2024-04-17T08:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.346+0000 7fb674a00640 1 -- 172.21.15.67:0/1914100894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670034610 con 0x7fb6780d2ab0 2024-04-17T08:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 -- 172.21.15.67:0/1914100894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780d2ab0 msgr2=0x7fb6780b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 --2- 172.21.15.67:0/1914100894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780d2ab0 0x7fb6780b7b10 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb67002c090 comp rx=0 tx=0).stop 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 -- 172.21.15.67:0/1914100894 shutdown_connections 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 --2- 172.21.15.67:0/1914100894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780d2ab0 0x7fb6780b7b10 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 -- 172.21.15.67:0/1914100894 >> 172.21.15.67:0/1914100894 conn(0x7fb67806cbd0 msgr2=0x7fb67806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 -- 172.21.15.67:0/1914100894 shutdown_connections 2024-04-17T08:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.347+0000 7fb67d200640 1 -- 172.21.15.67:0/1914100894 wait complete. 2024-04-17T08:55:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.348+0000 7fb67d200640 1 Processor -- start 2024-04-17T08:55:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 -- start start 2024-04-17T08:55:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6780d2ab0 0x7fb6781e75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb67815ed10 0x7fb6780b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780ba300 con 0x7fb6780cf640 2024-04-17T08:55:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb6780d25b0 con 0x7fb6780d2ab0 2024-04-17T08:55:18.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.349+0000 7fb67d200640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb6780ba480 con 0x7fb67815ed10 2024-04-17T08:55:18.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47358/0 (socket says 172.21.15.67:47358) 2024-04-17T08:55:18.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 -- 172.21.15.67:0/983236505 learned_addr learned my addr 172.21.15.67:0/983236505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:18.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 -- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb67815ed10 msgr2=0x7fb6780b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb67815ed10 0x7fb6780b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 -- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6780d2ab0 msgr2=0x7fb6781e75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6780d2ab0 0x7fb6781e75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.350+0000 7fb675e00640 1 -- 172.21.15.67:0/983236505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6780b4c30 con 0x7fb6780cf640 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb675e00640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c450 tx=0x7fb66800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680052d0 con 0x7fb6780cf640 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb668005470 con 0x7fb6780cf640 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668012870 con 0x7fb6780cf640 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6780cdcf0 con 0x7fb6780cf640 2024-04-17T08:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.351+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6780ce2b0 con 0x7fb6780cf640 2024-04-17T08:55:18.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.352+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb678068b90 con 0x7fb6780cf640 2024-04-17T08:55:18.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.353+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fb668012a10 con 0x7fb6780cf640 2024-04-17T08:55:18.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.354+0000 7fb66e000640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 0x7fb65806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.355+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fb66808b950 con 0x7fb6780cf640 2024-04-17T08:55:18.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.355+0000 7fb675400640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 0x7fb65806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.356+0000 7fb675400640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 0x7fb65806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6780c9ce0 tx=0x7fb6640079b0 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.359+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fb668059c20 con 0x7fb6780cf640 2024-04-17T08:55:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.560+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6780b40b0 con 0x7fb65806ca10 2024-04-17T08:55:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.562+0000 7fb66e000640 1 -- 172.21.15.67:0/983236505 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13539 (secure 0 0 0) 0x7fb6780b40b0 con 0x7fb65806ca10 2024-04-17T08:55:18.562 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:18.563 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 msgr2=0x7fb65806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 0x7fb65806eed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6780c9ce0 tx=0x7fb6640079b0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 msgr2=0x7fb6780cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c450 tx=0x7fb66800c910 comp rx=0 tx=0).stop 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 shutdown_connections 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb65806ca10 0x7fb65806eed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb67815ed10 0x7fb6780b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb6780d2ab0 0x7fb6781e75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 --2- 172.21.15.67:0/983236505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6780cf640 0x7fb6780cd4a0 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 >> 172.21.15.67:0/983236505 conn(0x7fb67806cbd0 msgr2=0x7fb678161150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:18.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.565+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 shutdown_connections 2024-04-17T08:55:18.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.566+0000 7fb67d200640 1 -- 172.21.15.67:0/983236505 wait complete. 2024-04-17T08:55:18.584 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-17T08:55:16.855992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":465592,"kb_used_data":31576,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754509128,"statfs":{"total":773094113280,"available":772617347072,"internally_reserved":0,"allocated":32333824,"data_stored":17342582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-782129,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-430,"num_read_kb":-263,"num_write":-418,"num_write_kb":-2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001771"},"pg_stats":[{"pgid":"1.0","version":"291'2793","reported_seq":4236,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-17T08:55:12.410539+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:55:12.410539+0000","last_peered":"2024-04-17T08:55:12.410539+0000","last_clean":"2024-04-17T08:55:12.410539+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:55:12.410539+0000","last_undegraded":"2024-04-17T08:55:12.410539+0000","last_fullsized":"2024-04-17T08:55:12.410539+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543344,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53792,"kb_used_data":4564,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318048,"statfs":{"total":96636764160,"available":96581681152,"internally_reserved":0,"allocated":4673536,"data_stored":2804928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543344,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61212,"kb_used_data":3664,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310628,"statfs":{"total":96636764160,"available":96574083072,"internally_reserved":0,"allocated":3751936,"data_stored":1880981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543343,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52912,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318928,"statfs":{"total":96636764160,"available":96582582272,"internally_reserved":0,"allocated":3772416,"data_stored":1888149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543343,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85040,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286800,"statfs":{"total":96636764160,"available":96549683200,"internally_reserved":0,"allocated":4034560,"data_stored":2157060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543345,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52828,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319012,"statfs":{"total":96636764160,"available":96582668288,"internally_reserved":0,"allocated":3686400,"data_stored":1821803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543344,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3702784,"data_stored":1828300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543345,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49068,"kb_used_data":4000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322772,"statfs":{"total":96636764160,"available":96586518528,"internally_reserved":0,"allocated":4096000,"data_stored":2221448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543343,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57896,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94313944,"statfs":{"total":96636764160,"available":96577478656,"internally_reserved":0,"allocated":4616192,"data_stored":2739913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:55:18.585 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-17T08:55:18.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:55:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.782+0000 7f32abe00640 1 Processor -- start 2024-04-17T08:55:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.783+0000 7f32abe00640 1 -- start start 2024-04-17T08:55:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32abe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32abe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0cf910 0x7f32ac0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32abe00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ac0d2d30 con 0x7f32ac15ec60 2024-04-17T08:55:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32abe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f32ac0d49c0 con 0x7f32ac0b6d00 2024-04-17T08:55:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32abe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac0d3660 con 0x7f32ac0cf910 2024-04-17T08:55:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32a3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32a3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47360/0 (socket says 172.21.15.67:47360) 2024-04-17T08:55:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.784+0000 7f32a3e00640 1 -- 172.21.15.67:0/1505163582 learned_addr learned my addr 172.21.15.67:0/1505163582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.785+0000 7f32ab400640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0cf910 0x7f32ac0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.785+0000 7f32a3e00640 1 -- 172.21.15.67:0/1505163582 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0cf910 msgr2=0x7f32ac0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.785+0000 7f32a3e00640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0cf910 0x7f32ac0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.785+0000 7f32a3e00640 1 -- 172.21.15.67:0/1505163582 >> v1:172.21.15.187:6789/0 conn(0x7f32ac15ec60 legacy=0x7f32ac0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.785+0000 7f32a3e00640 1 -- 172.21.15.67:0/1505163582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac0d3010 con 0x7f32ac0b6d00 2024-04-17T08:55:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.786+0000 7f32a3e00640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f3294009870 tx=0x7f329402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c67a6e6a05f8719 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.786+0000 7f32aa000640 1 -- 172.21.15.67:0/1505163582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294035020 con 0x7f32ac0b6d00 2024-04-17T08:55:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.786+0000 7f32aa000640 1 -- 172.21.15.67:0/1505163582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3294031640 con 0x7f32ac0b6d00 2024-04-17T08:55:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.786+0000 7f32aa000640 1 -- 172.21.15.67:0/1505163582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329402d040 con 0x7f32ac0b6d00 2024-04-17T08:55:18.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 -- 172.21.15.67:0/1505163582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0b6d00 msgr2=0x7f32ac0cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f3294009870 tx=0x7f329402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 -- 172.21.15.67:0/1505163582 shutdown_connections 2024-04-17T08:55:18.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0cf910 0x7f32ac0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 --2- 172.21.15.67:0/1505163582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0b6d00 0x7f32ac0cf3d0 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 -- 172.21.15.67:0/1505163582 >> 172.21.15.67:0/1505163582 conn(0x7f32ac0d15f0 msgr2=0x7f32ac0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 -- 172.21.15.67:0/1505163582 shutdown_connections 2024-04-17T08:55:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.787+0000 7f32abe00640 1 -- 172.21.15.67:0/1505163582 wait complete. 2024-04-17T08:55:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.789+0000 7f32abe00640 1 Processor -- start 2024-04-17T08:55:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.789+0000 7f32abe00640 1 -- start start 2024-04-17T08:55:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0b6d00 0x7f32ac1f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32ac15ec60 0x7f32ac155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac15f040 con 0x7f32ac0cf910 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0cfdc0 con 0x7f32ac15ec60 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32abe00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac0d00c0 con 0x7f32ac0b6d00 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47362/0 (socket says 172.21.15.67:47362) 2024-04-17T08:55:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 -- 172.21.15.67:0/1235029506 learned_addr learned my addr 172.21.15.67:0/1235029506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:18.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 -- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0b6d00 msgr2=0x7f32ac1f56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0b6d00 0x7f32ac1f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 -- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32ac15ec60 msgr2=0x7f32ac155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32ac15ec60 0x7f32ac155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.790+0000 7f32a3e00640 1 -- 172.21.15.67:0/1235029506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac1f5be0 con 0x7f32ac0cf910 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32a3e00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f329402b930 tx=0x7f32940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294040070 con 0x7f32ac0cf910 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3294039c10 con 0x7f32ac0cf910 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ac0cd7a0 con 0x7f32ac0cf910 2024-04-17T08:55:18.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294035040 con 0x7f32ac0cf910 2024-04-17T08:55:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.791+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ac0cdd60 con 0x7f32ac0cf910 2024-04-17T08:55:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.793+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ac0b8ee0 con 0x7f32ac0cf910 2024-04-17T08:55:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.794+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f329404b060 con 0x7f32ac0cf910 2024-04-17T08:55:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.795+0000 7f32a2a00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 0x7f328806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.795+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f32940ad8e0 con 0x7f32ac0cf910 2024-04-17T08:55:18.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.796+0000 7f32aaa00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 0x7f328806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:18.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.797+0000 7f32aaa00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 0x7f328806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f329c006fd0 tx=0x7f329c006bc0 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:18.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:18.799+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f32ac0b8ee0 con 0x7f32ac0cf910 2024-04-17T08:55:18.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.000+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f32ac0cb830 con 0x7f328806ca10 2024-04-17T08:55:19.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.002+0000 7f32a2a00640 1 -- 172.21.15.67:0/1235029506 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13532 (secure 0 0 0) 0x7f32ac0cb830 con 0x7f328806ca10 2024-04-17T08:55:19.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:19.002 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:55:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 msgr2=0x7f328806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 0x7f328806eed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f329c006fd0 tx=0x7f329c006bc0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 msgr2=0x7f32ac154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f329402b930 tx=0x7f32940046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 shutdown_connections 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f328806ca10 0x7f328806eed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32ac15ec60 0x7f32ac155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32ac0cf910 0x7f32ac154ae0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 --2- 172.21.15.67:0/1235029506 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f32ac0b6d00 0x7f32ac1f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 >> 172.21.15.67:0/1235029506 conn(0x7f32ac0d15f0 msgr2=0x7f32ac0b7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 shutdown_connections 2024-04-17T08:55:19.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.005+0000 7f32abe00640 1 -- 172.21.15.67:0/1235029506 wait complete. 2024-04-17T08:55:19.023 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-17T08:55:18.856569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":464312,"kb_used_data":30280,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754510408,"statfs":{"total":773094113280,"available":772618657792,"internally_reserved":0,"allocated":31006720,"data_stored":15912576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-782129,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-430,"num_read_kb":-263,"num_write":-418,"num_write_kb":-2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.963341"},"pg_stats":[{"pgid":"1.0","version":"291'2793","reported_seq":4239,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-17T08:55:13.929134+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:55:13.929134+0000","last_peered":"2024-04-17T08:55:13.929134+0000","last_clean":"2024-04-17T08:55:13.929134+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:55:13.929134+0000","last_undegraded":"2024-04-17T08:55:13.929134+0000","last_fullsized":"2024-04-17T08:55:13.929134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543345,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53444,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318396,"statfs":{"total":96636764160,"available":96582037504,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61192,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310648,"statfs":{"total":96636764160,"available":96574103552,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52868,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318972,"statfs":{"total":96636764160,"available":96582627328,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543344,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84740,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287100,"statfs":{"total":96636764160,"available":96549990400,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52872,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318968,"statfs":{"total":96636764160,"available":96582623232,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52876,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48708,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323132,"statfs":{"total":96636764160,"available":96586887168,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:55:19.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T08:55:19.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.236+0000 7f6b80400640 1 Processor -- start 2024-04-17T08:55:19.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.237+0000 7f6b80400640 1 -- start start 2024-04-17T08:55:19.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.238+0000 7f6b80400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6b780cf640 0x7f6b780cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.238+0000 7f6b80400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 0x7f6b780d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.238+0000 7f6b80400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b780cff60 con 0x7f6b780d2ab0 2024-04-17T08:55:19.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.238+0000 7f6b80400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6b780d3c30 con 0x7f6b780cf640 2024-04-17T08:55:19.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.238+0000 7f6b80400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b780d3020 con 0x7f6b7815ed10 2024-04-17T08:55:19.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.239+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 0x7f6b780d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.239+0000 7f6b7da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6b780cf640 0x7f6b780cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.239+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 0x7f6b780d40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.67:47642/0 (socket says 172.21.15.67:47642) 2024-04-17T08:55:19.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.239+0000 7f6b7d000640 1 -- 172.21.15.67:0/1650245737 learned_addr learned my addr 172.21.15.67:0/1650245737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:19.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.240+0000 7f6b7da00640 1 -- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 msgr2=0x7f6b780d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:19.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.240+0000 7f6b7da00640 1 --2- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 0x7f6b780d40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.240+0000 7f6b7da00640 1 -- 172.21.15.67:0/1650245737 >> v1:172.21.15.187:6789/0 conn(0x7f6b780d2ab0 legacy=0x7f6b780b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.240+0000 7f6b7da00640 1 -- 172.21.15.67:0/1650245737 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b780d3300 con 0x7f6b780cf640 2024-04-17T08:55:19.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.242+0000 7f6b7da00640 1 --2- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780cf640 0x7f6b780cfa20 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0097a0 tx=0x7f6b6c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273e7af050f904ed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:19.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.242+0000 7f6b77e00640 1 -- 172.21.15.67:0/1650245737 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c034070 con 0x7f6b780cf640 2024-04-17T08:55:19.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.242+0000 7f6b77e00640 1 -- 172.21.15.67:0/1650245737 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6c030800 con 0x7f6b780cf640 2024-04-17T08:55:19.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.242+0000 7f6b77e00640 1 -- 172.21.15.67:0/1650245737 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c030bd0 con 0x7f6b780cf640 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 -- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780cf640 msgr2=0x7f6b780cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 --2- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780cf640 0x7f6b780cfa20 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0097a0 tx=0x7f6b6c02b940 comp rx=0 tx=0).stop 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 -- 172.21.15.67:0/1650245737 shutdown_connections 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 --2- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b7815ed10 0x7f6b780d40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 --2- 172.21.15.67:0/1650245737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780cf640 0x7f6b780cfa20 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 -- 172.21.15.67:0/1650245737 >> 172.21.15.67:0/1650245737 conn(0x7f6b7806cbd0 msgr2=0x7f6b7806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:19.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 -- 172.21.15.67:0/1650245737 shutdown_connections 2024-04-17T08:55:19.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.243+0000 7f6b80400640 1 -- 172.21.15.67:0/1650245737 wait complete. 2024-04-17T08:55:19.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.246+0000 7f6b80400640 1 Processor -- start 2024-04-17T08:55:19.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.246+0000 7f6b80400640 1 -- start start 2024-04-17T08:55:19.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780cf640 0x7f6b7814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 0x7f6b780b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b780d25b0 con 0x7f6b7815ed10 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6b780d38c0 con 0x7f6b780d2ab0 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b80400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6b780d4600 con 0x7f6b780cf640 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47366/0 (socket says 172.21.15.67:47366) 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b7e400640 1 -- 172.21.15.67:0/634913036 learned_addr learned my addr 172.21.15.67:0/634913036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:19.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b7d000640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 0x7f6b780b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.248+0000 7f6b7da00640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780cf640 0x7f6b7814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 -- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780cf640 msgr2=0x7f6b7814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780cf640 0x7f6b7814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 -- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 msgr2=0x7f6b780b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 0x7f6b780b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 -- 172.21.15.67:0/634913036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b78159670 con 0x7f6b7815ed10 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7d000640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 0x7f6b780b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b7e400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c7d0 tx=0x7f6b7000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70003b70 con 0x7f6b7815ed10 2024-04-17T08:55:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b70003d30 con 0x7f6b7815ed10 2024-04-17T08:55:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b780b7230 con 0x7f6b7815ed10 2024-04-17T08:55:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7000aa90 con 0x7f6b7815ed10 2024-04-17T08:55:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.249+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7815bf60 con 0x7f6b7815ed10 2024-04-17T08:55:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.251+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b78068b90 con 0x7f6b7815ed10 2024-04-17T08:55:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.252+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f6b7000ac30 con 0x7f6b7815ed10 2024-04-17T08:55:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.253+0000 7f6b75600640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 0x7f6b5806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.253+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f6b7008bc20 con 0x7f6b7815ed10 2024-04-17T08:55:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.254+0000 7f6b7da00640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 0x7f6b5806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.255+0000 7f6b7da00640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 0x7f6b5806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c036000 tx=0x7f6b6c03b040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:19.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.257+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f6b78159670 con 0x7f6b7815ed10 2024-04-17T08:55:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.471+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b780b4a40 con 0x7f6b7815ed10 2024-04-17T08:55:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.473+0000 7f6b75600640 1 -- 172.21.15.67:0/634913036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v295) ==== 74+0+14228 (secure 0 0 0) 0x7f6b70059f80 con 0x7f6b7815ed10 2024-04-17T08:55:19.473 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:19.473 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":295,"fsid":"c9dd6e72-8918-4f7c-8224-a4ea40cec0ec","created":"2024-04-17T07:13:38.096760+0000","modified":"2024-04-17T08:55:13.912571+0000","last_up_change":"2024-04-17T07:13:50.093373+0000","last_in_change":"2024-04-17T07:13:41.675397+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T07:13:51.387984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03fdbf1a-d02e-4746-9057-f28a2d7c5535","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6817","nonce":2121978331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6819","nonce":2121978331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6823","nonce":2121978331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2121978331},{"type":"v1","addr":"172.21.15.67:6821","nonce":2121978331}]},"public_addr":"172.21.15.67:6817/2121978331","cluster_addr":"172.21.15.67:6819/2121978331","heartbeat_back_addr":"172.21.15.67:6823/2121978331","heartbeat_front_addr":"172.21.15.67:6821/2121978331","state":["exists","up"]},{"osd":1,"uuid":"a364ec30-0f7e-4847-99b5-945415b1a816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6801","nonce":2081347185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6803","nonce":2081347185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6812","nonce":2081347185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":2081347185},{"type":"v1","addr":"172.21.15.67:6807","nonce":2081347185}]},"public_addr":"172.21.15.67:6801/2081347185","cluster_addr":"172.21.15.67:6803/2081347185","heartbeat_back_addr":"172.21.15.67:6812/2081347185","heartbeat_front_addr":"172.21.15.67:6807/2081347185","state":["exists","up"]},{"osd":2,"uuid":"18b200d0-3b8a-4c95-9ead-ff065c9bb590","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6825","nonce":873912528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6827","nonce":873912528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6831","nonce":873912528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":873912528},{"type":"v1","addr":"172.21.15.67:6829","nonce":873912528}]},"public_addr":"172.21.15.67:6825/873912528","cluster_addr":"172.21.15.67:6827/873912528","heartbeat_back_addr":"172.21.15.67:6831/873912528","heartbeat_front_addr":"172.21.15.67:6829/873912528","state":["exists","up"]},{"osd":3,"uuid":"77a9ac54-619f-409f-8a76-2438f56f0ec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":288,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6806","nonce":100610741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6809","nonce":100610741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6816","nonce":100610741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":100610741},{"type":"v1","addr":"172.21.15.67:6813","nonce":100610741}]},"public_addr":"172.21.15.67:6806/100610741","cluster_addr":"172.21.15.67:6809/100610741","heartbeat_back_addr":"172.21.15.67:6816/100610741","heartbeat_front_addr":"172.21.15.67:6813/100610741","state":["exists","up"]},{"osd":4,"uuid":"24e32618-be8c-48e7-a978-94e71d7fc380","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6801","nonce":27895010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6803","nonce":27895010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6807","nonce":27895010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":27895010},{"type":"v1","addr":"172.21.15.187:6805","nonce":27895010}]},"public_addr":"172.21.15.187:6801/27895010","cluster_addr":"172.21.15.187:6803/27895010","heartbeat_back_addr":"172.21.15.187:6807/27895010","heartbeat_front_addr":"172.21.15.187:6805/27895010","state":["exists","up"]},{"osd":5,"uuid":"faa69671-29c7-4e2f-8f96-d7a55545d56d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6818","nonce":656328999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6822","nonce":656328999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6830","nonce":656328999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":656328999},{"type":"v1","addr":"172.21.15.187:6826","nonce":656328999}]},"public_addr":"172.21.15.187:6818/656328999","cluster_addr":"172.21.15.187:6822/656328999","heartbeat_back_addr":"172.21.15.187:6830/656328999","heartbeat_front_addr":"172.21.15.187:6826/656328999","state":["exists","up"]},{"osd":6,"uuid":"0d9b56be-9793-41b6-a062-a63e3a0b83e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6817","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6819","nonce":3359222512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6821","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6823","nonce":3359222512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6829","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6831","nonce":3359222512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6825","nonce":3359222512},{"type":"v1","addr":"172.21.15.187:6827","nonce":3359222512}]},"public_addr":"172.21.15.187:6819/3359222512","cluster_addr":"172.21.15.187:6823/3359222512","heartbeat_back_addr":"172.21.15.187:6831/3359222512","heartbeat_front_addr":"172.21.15.187:6827/3359222512","state":["exists","up"]},{"osd":7,"uuid":"161cf315-6bb4-4c45-936c-59ec7d1e4e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6809","nonce":1997375062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6811","nonce":1997375062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6815","nonce":1997375062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1997375062},{"type":"v1","addr":"172.21.15.187:6813","nonce":1997375062}]},"public_addr":"172.21.15.187:6809/1997375062","cluster_addr":"172.21.15.187:6811/1997375062","heartbeat_back_addr":"172.21.15.187:6815/1997375062","heartbeat_front_addr":"172.21.15.187:6813/1997375062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.414443+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.155463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.356318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.033505+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.256821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.100874+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.179552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T07:13:47.039900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:0/839885293":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3001440476":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/150412697":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2744215170":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/1795094394":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2855667520":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/933679678":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3424176410":"2024-04-18T08:55:06.811712+0000","172.21.15.187:6837/2174028812":"2024-04-18T08:55:10.885117+0000","172.21.15.187:6833/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/3757546694":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/2687160515":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6835/2174028812":"2024-04-18T08:55:10.885117+0000","172.21.15.187:0/3174473408":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/740627636":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/1576025869":"2024-04-18T08:55:06.811712+0000","172.21.15.187:0/492269958":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3820292707":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6833/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:6832/46399553":"2024-04-18T08:53:42.064401+0000","172.21.15.187:0/3797546006":"2024-04-18T08:53:42.064401+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 msgr2=0x7f6b5806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 0x7f6b5806eed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c036000 tx=0x7f6b6c03b040 comp rx=0 tx=0).stop 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 msgr2=0x7f6b780b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f6b7000c7d0 tx=0x7f6b7000cc90 comp rx=0 tx=0).stop 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 shutdown_connections 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f6b5806ca10 0x7f6b5806eed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b7815ed10 0x7f6b780b4c60 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b780d2ab0 0x7f6b780b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 --2- 172.21.15.67:0/634913036 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6b780cf640 0x7f6b7814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:19.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 >> 172.21.15.67:0/634913036 conn(0x7f6b7806cbd0 msgr2=0x7f6b780b93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:19.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 shutdown_connections 2024-04-17T08:55:19.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:19.476+0000 7f6b80400640 1 -- 172.21.15.67:0/634913036 wait complete. 2024-04-17T08:55:20.495 INFO:tasks.ceph:Scrubbing osd.0 2024-04-17T08:55:20.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:20.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.729+0000 7f064a600640 1 Processor -- start 2024-04-17T08:55:20.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.729+0000 7f064a600640 1 -- start start 2024-04-17T08:55:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.730+0000 7f064a600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f06440cd770 0x7f06441695c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.730+0000 7f064a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064406de70 con 0x7f064415c670 2024-04-17T08:55:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.730+0000 7f064a600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064415f850 con 0x7f0644072930 2024-04-17T08:55:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.730+0000 7f064a600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f064415fb30 con 0x7f06440cd770 2024-04-17T08:55:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.731+0000 7f0649200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f064415c670 0x7f064415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34200/0 (socket says 172.21.15.67:34200) 2024-04-17T08:55:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.731+0000 7f0649200640 1 -- 172.21.15.67:0/1902742169 learned_addr learned my addr 172.21.15.67:0/1902742169 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.731+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147710798 0 0) 0x7f064406de70 con 0x7f064415c670 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.731+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0628003520 con 0x7f064415c670 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.731+0000 7f0649c00640 1 --2- 172.21.15.67:0/1902742169 >> v2:172.21.15.187:3300/0 conn(0x7f06440cd770 0x7f06441695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4000461054 0 0) 0x7f0628003520 con 0x7f064415c670 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 >> v2:172.21.15.187:3300/0 conn(0x7f06440cd770 msgr2=0x7f06441695c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 --2- 172.21.15.67:0/1902742169 >> v2:172.21.15.187:3300/0 conn(0x7f06440cd770 0x7f06441695c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 >> v1:172.21.15.187:6789/0 conn(0x7f0644072930 legacy=0x7f06440cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064415fe10 con 0x7f064415c670 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f0634001690 con 0x7f064415c670 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 >> v1:172.21.15.67:6789/0 conn(0x7f064415c670 legacy=0x7f064415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628003a00 0x7f0628003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 0x7f0628004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628005210 0x7f062800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f064415fb30 con 0x7f0628004800 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f062800c160 con 0x7f0628003a00 2024-04-17T08:55:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f062800c360 con 0x7f0628005210 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.732+0000 7f0643e00640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 0x7f0628004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0649200640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628003a00 0x7f0628003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0643e00640 1 -- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628005210 msgr2=0x7f062800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0643e00640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628005210 0x7f062800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0643e00640 1 -- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628003a00 msgr2=0x7f0628003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0643e00640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0628003a00 0x7f0628003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0643e00640 1 -- 172.21.15.67:0/1902742169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0628003520 con 0x7f0628004800 2024-04-17T08:55:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.733+0000 7f0649c00640 1 -- 172.21.15.67:0/1902742169 reap_dead start 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.734+0000 7f0643e00640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 0x7f0628004c80 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f0630001e50 tx=0x7f0628005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7b4deb7484b980 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.734+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630001a70 con 0x7f0628004800 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.734+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0630003b70 con 0x7f0628004800 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.734+0000 7f0643400640 1 -- 172.21.15.67:0/1902742169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06300050d0 con 0x7f0628004800 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 -- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 msgr2=0x7f0628004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 0x7f0628004c80 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f0630001e50 tx=0x7f0628005830 comp rx=0 tx=0).stop 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 -- 172.21.15.67:0/1902742169 shutdown_connections 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 --2- 172.21.15.67:0/1902742169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0628004800 0x7f0628004c80 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 -- 172.21.15.67:0/1902742169 >> 172.21.15.67:0/1902742169 conn(0x7f064406b8c0 msgr2=0x7f064406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 -- 172.21.15.67:0/1902742169 shutdown_connections 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.735+0000 7f064a600640 1 -- 172.21.15.67:0/1902742169 wait complete. 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.737+0000 7f064a600640 1 Processor -- start 2024-04-17T08:55:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.737+0000 7f064a600640 1 -- start start 2024-04-17T08:55:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628004800 0x7f06440ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 0x7f06441587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06440c60a0 0x7f06440c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06440b3fb0 con 0x7f06440c5630 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06440b3e30 con 0x7f06440c60a0 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.738+0000 7f064a600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06440b4130 con 0x7f0628004800 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 0x7f06441587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 0x7f06441587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47374/0 (socket says 172.21.15.67:47374) 2024-04-17T08:55:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 -- 172.21.15.67:0/3389229828 learned_addr learned my addr 172.21.15.67:0/3389229828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0649200640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628004800 0x7f06440ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0649c00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06440c60a0 0x7f06440c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 -- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628004800 msgr2=0x7f06440ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0628004800 0x7f06440ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 -- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06440c60a0 msgr2=0x7f06440c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06440c60a0 0x7f06440c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.739+0000 7f0643e00640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06440cc4b0 con 0x7f06440c5630 2024-04-17T08:55:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f0643e00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 0x7f06441587e0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f0630000f80 tx=0x7f06300014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630003970 con 0x7f06440c5630 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0630006b50 con 0x7f06440c5630 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06440ae9e0 con 0x7f06440c5630 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630007020 con 0x7f06440c5630 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.740+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06440aefa0 con 0x7f06440c5630 2024-04-17T08:55:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.741+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06440b0680 con 0x7f06440c5630 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.742+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f0630049060 con 0x7f06440c5630 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.743+0000 7f0640c00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f062406ca40 0x7f062406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.743+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f06300ad210 con 0x7f06440c5630 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.743+0000 7f0640c00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f0624072fe0 0x7f0624075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.743+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0624075ad0 con 0x7f0624072fe0 2024-04-17T08:55:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.744+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7f06300b4050 con 0x7f06440c5630 2024-04-17T08:55:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.744+0000 7f0649c00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f0624072fe0 0x7f0624075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.744+0000 7f0649200640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f062406ca40 0x7f062406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.745+0000 7f0649c00640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f0624072fe0 0x7f0624075400 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.745+0000 7f0649200640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f062406ca40 0x7f062406ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0634001050 tx=0x7f0634009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.745+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== osd.0 v2:172.21.15.67:6815/2121978331 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0624075ad0 con 0x7f0624072fe0 2024-04-17T08:55:20.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.760+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 --> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f064415a940 con 0x7f0624072fe0 2024-04-17T08:55:20.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f0640c00640 1 -- 172.21.15.67:0/3389229828 <== osd.0 v2:172.21.15.67:6815/2121978331 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f064415a940 con 0x7f0624072fe0 2024-04-17T08:55:20.760 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:20.761 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:20.761 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f0624072fe0 msgr2=0x7f0624075400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:6815/2121978331,v1:172.21.15.67:6817/2121978331] conn(0x7f0624072fe0 0x7f0624075400 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f062406ca40 msgr2=0x7f062406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f062406ca40 0x7f062406ef00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0634001050 tx=0x7f0634009040 comp rx=0 tx=0).stop 2024-04-17T08:55:20.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 msgr2=0x7f06441587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 --2- 172.21.15.67:0/3389229828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06440c5630 0x7f06441587e0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f0630000f80 tx=0x7f06300014a0 comp rx=0 tx=0).stop 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f0649c00640 1 -- 172.21.15.67:0/3389229828 reap_dead start 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 shutdown_connections 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.761+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 >> 172.21.15.67:0/3389229828 conn(0x7f064406b8c0 msgr2=0x7f06440b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.762+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 shutdown_connections 2024-04-17T08:55:20.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:20.762+0000 7f064a600640 1 -- 172.21.15.67:0/3389229828 wait complete. 2024-04-17T08:55:20.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-17T08:55:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.001+0000 7fad0e800640 1 Processor -- start 2024-04-17T08:55:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.001+0000 7fad0e800640 1 -- start start 2024-04-17T08:55:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0e800640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 0x7fad080cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad08072570 0x7fad080ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0e800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0815d060 con 0x7fad080b63c0 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fad080cfe90 con 0x7fad08072570 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0e800640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fad0815dc70 con 0x7fad080cbc00 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad08072570 0x7fad080ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad08072570 0x7fad080ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47384/0 (socket says 172.21.15.67:47384) 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad080b63c0 0x7fad08071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34210/0 (socket says 172.21.15.67:34210) 2024-04-17T08:55:21.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.003+0000 7fad0de00640 1 -- 172.21.15.67:0/1606741059 learned_addr learned my addr 172.21.15.67:0/1606741059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:21.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0d400640 1 --2- 172.21.15.67:0/1606741059 >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 0x7fad080cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0de00640 1 -- 172.21.15.67:0/1606741059 >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 msgr2=0x7fad080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0de00640 1 --2- 172.21.15.67:0/1606741059 >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 0x7fad080cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0de00640 1 -- 172.21.15.67:0/1606741059 >> v1:172.21.15.67:6789/0 conn(0x7fad080b63c0 legacy=0x7fad08071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0de00640 1 -- 172.21.15.67:0/1606741059 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0815d340 con 0x7fad08072570 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.004+0000 7fad0d400640 1 --2- 172.21.15.67:0/1606741059 >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 0x7fad080cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.005+0000 7fad0de00640 1 --2- 172.21.15.67:0/1606741059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080ce610 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fad00009e50 tx=0x7fad0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ed47c2164fc0be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.005+0000 7fad07e00640 1 -- 172.21.15.67:0/1606741059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0002cb20 con 0x7fad08072570 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.005+0000 7fad07e00640 1 -- 172.21.15.67:0/1606741059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0002ccc0 con 0x7fad08072570 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.005+0000 7fad07e00640 1 -- 172.21.15.67:0/1606741059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad00034610 con 0x7fad08072570 2024-04-17T08:55:21.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 -- 172.21.15.67:0/1606741059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 msgr2=0x7fad080ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 --2- 172.21.15.67:0/1606741059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080ce610 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fad00009e50 tx=0x7fad0002bf80 comp rx=0 tx=0).stop 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 -- 172.21.15.67:0/1606741059 shutdown_connections 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 --2- 172.21.15.67:0/1606741059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080ce610 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 --2- 172.21.15.67:0/1606741059 >> v2:172.21.15.187:3300/0 conn(0x7fad080cbc00 0x7fad080cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 -- 172.21.15.67:0/1606741059 >> 172.21.15.67:0/1606741059 conn(0x7fad0806bef0 msgr2=0x7fad080b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 -- 172.21.15.67:0/1606741059 shutdown_connections 2024-04-17T08:55:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.006+0000 7fad0e800640 1 -- 172.21.15.67:0/1606741059 wait complete. 2024-04-17T08:55:21.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.008+0000 7fad0e800640 1 Processor -- start 2024-04-17T08:55:21.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.008+0000 7fad0e800640 1 -- start start 2024-04-17T08:55:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad080b63c0 0x7fad081f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad080cbc00 0x7fad080c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad080d02d0 con 0x7fad08072570 2024-04-17T08:55:21.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fad0815d900 con 0x7fad080b63c0 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0e800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fad080cbfe0 con 0x7fad080cbc00 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.009+0000 7fad0d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47398/0 (socket says 172.21.15.67:47398) 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 -- 172.21.15.67:0/374947902 learned_addr learned my addr 172.21.15.67:0/374947902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0ca00640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad080b63c0 0x7fad081f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0de00640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad080cbc00 0x7fad080c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 -- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad080cbc00 msgr2=0x7fad080c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad080cbc00 0x7fad080c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 -- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad080b63c0 msgr2=0x7fad081f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad080b63c0 0x7fad081f6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 -- 172.21.15.67:0/374947902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad080affe0 con 0x7fad08072570 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.010+0000 7fad0d400640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c820 tx=0x7facfc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.011+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc015070 con 0x7fad08072570 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.011+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc007a30 con 0x7fad08072570 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.011+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc007de0 con 0x7fad08072570 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.011+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad080b02c0 con 0x7fad08072570 2024-04-17T08:55:21.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.011+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad080af020 con 0x7fad08072570 2024-04-17T08:55:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.012+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad080cdeb0 con 0x7fad08072570 2024-04-17T08:55:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.013+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7facfc018070 con 0x7fad08072570 2024-04-17T08:55:21.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.014+0000 7fad05600640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 0x7face806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.014+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7facfc08be10 con 0x7fad08072570 2024-04-17T08:55:21.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.015+0000 7fad0ca00640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 0x7face806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.016+0000 7fad0ca00640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 0x7face806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad080b1f00 tx=0x7facf4007cc0 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.019+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fad080cdeb0 con 0x7fad08072570 2024-04-17T08:55:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.227+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fad08068a10 con 0x7face806ca10 2024-04-17T08:55:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.228+0000 7fad05600640 1 -- 172.21.15.67:0/374947902 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fad08068a10 con 0x7face806ca10 2024-04-17T08:55:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.231+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 msgr2=0x7face806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.231+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 0x7face806eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad080b1f00 tx=0x7facf4007cc0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.231+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 msgr2=0x7fad080c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.231+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c820 tx=0x7facfc00cce0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 shutdown_connections 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7face806ca10 0x7face806eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad080cbc00 0x7fad080c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad080b63c0 0x7fad081f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 --2- 172.21.15.67:0/374947902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad08072570 0x7fad080c5f70 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 >> 172.21.15.67:0/374947902 conn(0x7fad0806bef0 msgr2=0x7fad0815a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 shutdown_connections 2024-04-17T08:55:21.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.232+0000 7fad0e800640 1 -- 172.21.15.67:0/374947902 wait complete. 2024-04-17T08:55:21.251 INFO:tasks.ceph:Scrubbing osd.1 2024-04-17T08:55:21.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.483+0000 7f57eb000640 1 Processor -- start 2024-04-17T08:55:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.483+0000 7f57eb000640 1 -- start start 2024-04-17T08:55:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57eb000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57e40cd770 0x7f57e41695c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57eb000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e406de70 con 0x7f57e415c670 2024-04-17T08:55:21.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57eb000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e415f850 con 0x7f57e4072930 2024-04-17T08:55:21.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57eb000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57e415fb30 con 0x7f57e40cd770 2024-04-17T08:55:21.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57e9c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57e415c670 0x7f57e415ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34216/0 (socket says 172.21.15.67:34216) 2024-04-17T08:55:21.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57e9c00640 1 -- 172.21.15.67:0/539955150 learned_addr learned my addr 172.21.15.67:0/539955150 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.485+0000 7f57ea600640 1 --2- 172.21.15.67:0/539955150 >> v2:172.21.15.67:3300/0 conn(0x7f57e40cd770 0x7f57e41695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.486+0000 7f57e3e00640 1 -- 172.21.15.67:0/539955150 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744734735 0 0) 0x7f57e406de70 con 0x7f57e415c670 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.486+0000 7f57e3e00640 1 -- 172.21.15.67:0/539955150 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57cc003520 con 0x7f57e415c670 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.486+0000 7f57ea600640 1 -- 172.21.15.67:0/539955150 >> v1:172.21.15.187:6789/0 conn(0x7f57e4072930 legacy=0x7f57e40cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.486+0000 7f57ea600640 1 -- 172.21.15.67:0/539955150 >> v1:172.21.15.67:6789/0 conn(0x7f57e415c670 legacy=0x7f57e415ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.486+0000 7f57ea600640 1 -- 172.21.15.67:0/539955150 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e415fe10 con 0x7f57e40cd770 2024-04-17T08:55:21.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.487+0000 7f57ea600640 1 --2- 172.21.15.67:0/539955150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e40cd770 0x7f57e41695c0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009d90 tx=0x7f57dc02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e30ce6ed3c0325c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.487+0000 7f57e3e00640 1 -- 172.21.15.67:0/539955150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc039070 con 0x7f57e40cd770 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.487+0000 7f57e3e00640 1 -- 172.21.15.67:0/539955150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57dc02ccf0 con 0x7f57e40cd770 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57e3e00640 1 -- 172.21.15.67:0/539955150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc034660 con 0x7f57e40cd770 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 -- 172.21.15.67:0/539955150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e40cd770 msgr2=0x7f57e41695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 --2- 172.21.15.67:0/539955150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e40cd770 0x7f57e41695c0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009d90 tx=0x7f57dc02bf40 comp rx=0 tx=0).stop 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 -- 172.21.15.67:0/539955150 shutdown_connections 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 --2- 172.21.15.67:0/539955150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e40cd770 0x7f57e41695c0 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 -- 172.21.15.67:0/539955150 >> 172.21.15.67:0/539955150 conn(0x7f57e406b8c0 msgr2=0x7f57e406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 -- 172.21.15.67:0/539955150 shutdown_connections 2024-04-17T08:55:21.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.488+0000 7f57eb000640 1 -- 172.21.15.67:0/539955150 wait complete. 2024-04-17T08:55:21.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.490+0000 7f57eb000640 1 Processor -- start 2024-04-17T08:55:21.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.490+0000 7f57eb000640 1 -- start start 2024-04-17T08:55:21.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 0x7f57e4148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57e40cd770 0x7f57e40b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57e415c670 0x7f57e40b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57e40b3fb0 con 0x7f57e4072930 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f57e40b3e30 con 0x7f57e40cd770 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57eb000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f57e40b4130 con 0x7f57e415c670 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57e9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 0x7f57e4148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57e9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 0x7f57e4148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47406/0 (socket says 172.21.15.67:47406) 2024-04-17T08:55:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.491+0000 7f57e9c00640 1 -- 172.21.15.67:0/3335011109 learned_addr learned my addr 172.21.15.67:0/3335011109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9200640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57e40cd770 0x7f57e40b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57ea600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57e415c670 0x7f57e40b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 -- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57e415c670 msgr2=0x7f57e40b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57e415c670 0x7f57e40b1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 -- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57e40cd770 msgr2=0x7f57e40b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f57e40cd770 0x7f57e40b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e41e9b70 con 0x7f57e4072930 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57ea600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f57e415c670 0x7f57e40b1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.492+0000 7f57e9c00640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 0x7f57e4148f10 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f57d800c430 tx=0x7f57d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.493+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d8017070 con 0x7f57e4072930 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.493+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d8005ce0 con 0x7f57e4072930 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.493+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d80072d0 con 0x7f57e4072930 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.493+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e40ad9e0 con 0x7f57e4072930 2024-04-17T08:55:21.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.493+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e40adfa0 con 0x7f57e4072930 2024-04-17T08:55:21.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.494+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f57e40b1970 con 0x7f57e4072930 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.495+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f57d801a070 con 0x7f57e4072930 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57e1600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f57c406ca10 0x7f57c406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f57d808b0c0 con 0x7f57e4072930 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57e1600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f57c4072fb0 0x7f57c40753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57d8010e50 con 0x7f57c4072fb0 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7f57d8092050 con 0x7f57e4072930 2024-04-17T08:55:21.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.496+0000 7f57ea600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f57c4072fb0 0x7f57c40753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.497+0000 7f57e9200640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f57c406ca10 0x7f57c406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.497+0000 7f57ea600640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f57c4072fb0 0x7f57c40753d0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.498+0000 7f57e9200640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f57c406ca10 0x7f57c406eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f57e40b33d0 tx=0x7f57d0009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.498+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== osd.1 v2:172.21.15.67:6800/2081347185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f57d8010e50 con 0x7f57c4072fb0 2024-04-17T08:55:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.513+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 --> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f57e4068a10 con 0x7f57c4072fb0 2024-04-17T08:55:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.513+0000 7f57e1600640 1 -- 172.21.15.67:0/3335011109 <== osd.1 v2:172.21.15.67:6800/2081347185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f57e4068a10 con 0x7f57c4072fb0 2024-04-17T08:55:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:21.513 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:21.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.513+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f57c4072fb0 msgr2=0x7f57c40753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:6800/2081347185,v1:172.21.15.67:6801/2081347185] conn(0x7f57c4072fb0 0x7f57c40753d0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f57c406ca10 msgr2=0x7f57c406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f57c406ca10 0x7f57c406eed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f57e40b33d0 tx=0x7f57d0009040 comp rx=0 tx=0).stop 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 msgr2=0x7f57e4148f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 --2- 172.21.15.67:0/3335011109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57e4072930 0x7f57e4148f10 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f57d800c430 tx=0x7f57d800c8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57ea600640 1 -- 172.21.15.67:0/3335011109 reap_dead start 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 shutdown_connections 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 >> 172.21.15.67:0/3335011109 conn(0x7f57e406b8c0 msgr2=0x7f57e415e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 shutdown_connections 2024-04-17T08:55:21.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.514+0000 7f57eb000640 1 -- 172.21.15.67:0/3335011109 wait complete. 2024-04-17T08:55:21.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-17T08:55:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.758+0000 7f73c0a00640 1 Processor -- start 2024-04-17T08:55:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.758+0000 7f73c0a00640 1 -- start start 2024-04-17T08:55:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.759+0000 7f73c0a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc145ac0 0x7f73bc071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.759+0000 7f73c0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73bc0ce3b0 0x7f73bc072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73c0a00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73bc0cea70 con 0x7f73bc072bb0 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73c0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f73bc0ce810 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73c0a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc0ced50 con 0x7f73bc145ac0 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73bc0ce3b0 0x7f73bc072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73bc0ce3b0 0x7f73bc072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47416/0 (socket says 172.21.15.67:47416) 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 -- 172.21.15.67:0/2709818639 learned_addr learned my addr 172.21.15.67:0/2709818639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9e00640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc145ac0 0x7f73bc071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 -- 172.21.15.67:0/2709818639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc145ac0 msgr2=0x7f73bc071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc145ac0 0x7f73bc071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 -- 172.21.15.67:0/2709818639 >> v1:172.21.15.187:6789/0 conn(0x7f73bc072bb0 legacy=0x7f73bc0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.760+0000 7f73b9400640 1 -- 172.21.15.67:0/2709818639 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc0cf030 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73b9400640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc072670 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2b0054fb9c5456 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73b8a00640 1 -- 172.21.15.67:0/2709818639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4035020 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73b8a00640 1 -- 172.21.15.67:0/2709818639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a4031640 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73b8a00640 1 -- 172.21.15.67:0/2709818639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a402d040 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73c0a00640 1 -- 172.21.15.67:0/2709818639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 msgr2=0x7f73bc072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.762+0000 7f73c0a00640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc072670 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009870 tx=0x7f73a402b8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 -- 172.21.15.67:0/2709818639 shutdown_connections 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc072670 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 --2- 172.21.15.67:0/2709818639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc145ac0 0x7f73bc071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 -- 172.21.15.67:0/2709818639 >> 172.21.15.67:0/2709818639 conn(0x7f73bc067ee0 msgr2=0x7f73bc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 -- 172.21.15.67:0/2709818639 shutdown_connections 2024-04-17T08:55:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.763+0000 7f73c0a00640 1 -- 172.21.15.67:0/2709818639 wait complete. 2024-04-17T08:55:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.764+0000 7f73c0a00640 1 Processor -- start 2024-04-17T08:55:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 -- start start 2024-04-17T08:55:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 0x7f73bc13ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73bc145ac0 0x7f73bc0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc145ea0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc0cf5f0 con 0x7f73bc145ac0 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.765+0000 7f73c0a00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc0d0b20 con 0x7f73bc072bb0 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47428/0 (socket says 172.21.15.67:47428) 2024-04-17T08:55:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 -- 172.21.15.67:0/984972632 learned_addr learned my addr 172.21.15.67:0/984972632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9e00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 0x7f73bc13ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73ba800640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73bc145ac0 0x7f73bc0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 -- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 msgr2=0x7f73bc13ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 0x7f73bc13ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 -- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73bc145ac0 msgr2=0x7f73bc0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73bc145ac0 0x7f73bc0b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9400640 1 -- 172.21.15.67:0/984972632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc1505d0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.766+0000 7f73b9e00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 0x7f73bc13ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73b9400640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f73a4000c00 tx=0x7f73a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4040070 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a40399f0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4035040 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc1f3ea0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.767+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc1f43f0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.768+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc068ca0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.769+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f73a404b060 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.770+0000 7f73b2000640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 0x7f739806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.770+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f73a40ad670 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.771+0000 7f73b9e00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 0x7f739806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.772+0000 7f73b9e00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 0x7f739806eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009f50 tx=0x7f73ac00b040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:21.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.775+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f73a407b9b0 con 0x7f73bc0ce3b0 2024-04-17T08:55:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.981+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f73bc0cc730 con 0x7f739806ca10 2024-04-17T08:55:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.982+0000 7f73b2000640 1 -- 172.21.15.67:0/984972632 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f73bc0cc730 con 0x7f739806ca10 2024-04-17T08:55:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-04-17T08:55:21.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 msgr2=0x7f739806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 0x7f739806eed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009f50 tx=0x7f73ac00b040 comp rx=0 tx=0).stop 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 msgr2=0x7f73bc0b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f73a4000c00 tx=0x7f73a40046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 shutdown_connections 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f739806ca10 0x7f739806eed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73bc145ac0 0x7f73bc0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73bc0ce3b0 0x7f73bc0b3210 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 --2- 172.21.15.67:0/984972632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f73bc072bb0 0x7f73bc13ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:21.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 >> 172.21.15.67:0/984972632 conn(0x7f73bc067ee0 msgr2=0x7f73bc15c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:21.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 shutdown_connections 2024-04-17T08:55:21.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:21.985+0000 7f73c0a00640 1 -- 172.21.15.67:0/984972632 wait complete. 2024-04-17T08:55:22.003 INFO:tasks.ceph:Scrubbing osd.2 2024-04-17T08:55:22.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.229+0000 7fbb88e00640 1 Processor -- start 2024-04-17T08:55:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.230+0000 7fbb88e00640 1 -- start start 2024-04-17T08:55:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.231+0000 7fbb88e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 0x7fbb840b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.231+0000 7fbb88e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb84162e40 0x7fbb84155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.231+0000 7fbb88e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb8415f890 con 0x7fbb840b3800 2024-04-17T08:55:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.231+0000 7fbb88e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbb841629b0 con 0x7fbb84162e40 2024-04-17T08:55:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.231+0000 7fbb88e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8415fb70 con 0x7fbb840cfc80 2024-04-17T08:55:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb840b3800 0x7fbb8415f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34220/0 (socket says 172.21.15.67:34220) 2024-04-17T08:55:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83400640 1 -- 172.21.15.67:0/1186288638 learned_addr learned my addr 172.21.15.67:0/1186288638 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83e00640 1 --2- 172.21.15.67:0/1186288638 >> v2:172.21.15.67:3300/0 conn(0x7fbb84162e40 0x7fbb84155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb7be00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 0x7fbb840b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83e00640 1 -- 172.21.15.67:0/1186288638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 msgr2=0x7fbb840b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83e00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 0x7fbb840b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83e00640 1 -- 172.21.15.67:0/1186288638 >> v1:172.21.15.67:6789/0 conn(0x7fbb840b3800 legacy=0x7fbb8415f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb83e00640 1 -- 172.21.15.67:0/1186288638 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8415fe50 con 0x7fbb84162e40 2024-04-17T08:55:22.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.232+0000 7fbb7be00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 0x7fbb840b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:22.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.234+0000 7fbb83e00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb84155560 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009d90 tx=0x7fbb7c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f59e2dacf126896 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.234+0000 7fbb82a00640 1 -- 172.21.15.67:0/1186288638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c039070 con 0x7fbb84162e40 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.234+0000 7fbb82a00640 1 -- 172.21.15.67:0/1186288638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7c02cce0 con 0x7fbb84162e40 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.234+0000 7fbb82a00640 1 -- 172.21.15.67:0/1186288638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c034660 con 0x7fbb84162e40 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 -- 172.21.15.67:0/1186288638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 msgr2=0x7fbb84155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb84155560 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009d90 tx=0x7fbb7c02bf30 comp rx=0 tx=0).stop 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 -- 172.21.15.67:0/1186288638 shutdown_connections 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb84155560 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 --2- 172.21.15.67:0/1186288638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840cfc80 0x7fbb840b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 -- 172.21.15.67:0/1186288638 >> 172.21.15.67:0/1186288638 conn(0x7fbb84067600 msgr2=0x7fbb840679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:22.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 -- 172.21.15.67:0/1186288638 shutdown_connections 2024-04-17T08:55:22.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.235+0000 7fbb88e00640 1 -- 172.21.15.67:0/1186288638 wait complete. 2024-04-17T08:55:22.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.237+0000 7fbb88e00640 1 Processor -- start 2024-04-17T08:55:22.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.237+0000 7fbb88e00640 1 -- start start 2024-04-17T08:55:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840b3800 0x7fbb840ab370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840cfc80 0x7fbb841f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb840cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84163450 con 0x7fbb84162e40 2024-04-17T08:55:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841632d0 con 0x7fbb840cfc80 2024-04-17T08:55:22.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb88e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb84155aa0 con 0x7fbb840b3800 2024-04-17T08:55:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb83400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840b3800 0x7fbb840ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb840cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.238+0000 7fbb83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb840cb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47440/0 (socket says 172.21.15.67:47440) 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 -- 172.21.15.67:0/4195802152 learned_addr learned my addr 172.21.15.67:0/4195802152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb7be00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840cfc80 0x7fbb841f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 -- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840b3800 msgr2=0x7fbb840ab370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb840b3800 0x7fbb840ab370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 -- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840cfc80 msgr2=0x7fbb841f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840cfc80 0x7fbb841f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb83e00640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb840b5700 con 0x7fbb84162e40 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.239+0000 7fbb7be00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb840cfc80 0x7fbb841f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb83e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb840cb3a0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009d90 tx=0x7fbb7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c039050 con 0x7fbb84162e40 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7c045070 con 0x7fbb84162e40 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c03e830 con 0x7fbb84162e40 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb840b59e0 con 0x7fbb84162e40 2024-04-17T08:55:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.240+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb84152d90 con 0x7fbb84162e40 2024-04-17T08:55:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.241+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb84068a10 con 0x7fbb84162e40 2024-04-17T08:55:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.242+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fbb7c03e9d0 con 0x7fbb84162e40 2024-04-17T08:55:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.243+0000 7fbb7b400640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fbb6006ca10 0x7fbb6006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.243+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fbb7c0aeab0 con 0x7fbb84162e40 2024-04-17T08:55:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb7b400640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7fbb60072fb0 0x7fbb600753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb7c048e80 con 0x7fbb60072fb0 2024-04-17T08:55:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7fbb7c0b32a0 con 0x7fbb84162e40 2024-04-17T08:55:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb7be00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7fbb60072fb0 0x7fbb600753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb83400640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fbb6006ca10 0x7fbb6006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.244+0000 7fbb7be00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7fbb60072fb0 0x7fbb600753d0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.245+0000 7fbb83400640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fbb6006ca10 0x7fbb6006eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb74000f60 tx=0x7fbb74009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.245+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== osd.2 v2:172.21.15.67:6824/873912528 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb7c048e80 con 0x7fbb60072fb0 2024-04-17T08:55:22.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.260+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 --> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbb8406da00 con 0x7fbb60072fb0 2024-04-17T08:55:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb7b400640 1 -- 172.21.15.67:0/4195802152 <== osd.2 v2:172.21.15.67:6824/873912528 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbb8406da00 con 0x7fbb60072fb0 2024-04-17T08:55:22.261 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:22.261 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:22.261 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7fbb60072fb0 msgr2=0x7fbb600753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:6824/873912528,v1:172.21.15.67:6825/873912528] conn(0x7fbb60072fb0 0x7fbb600753d0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fbb6006ca10 msgr2=0x7fbb6006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fbb6006ca10 0x7fbb6006eed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbb74000f60 tx=0x7fbb74009040 comp rx=0 tx=0).stop 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 msgr2=0x7fbb840cb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 --2- 172.21.15.67:0/4195802152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb84162e40 0x7fbb840cb3a0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009d90 tx=0x7fbb7c004a40 comp rx=0 tx=0).stop 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb83e00640 1 -- 172.21.15.67:0/4195802152 reap_dead start 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 shutdown_connections 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 >> 172.21.15.67:0/4195802152 conn(0x7fbb84067600 msgr2=0x7fbb840b9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.261+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 shutdown_connections 2024-04-17T08:55:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.262+0000 7fbb88e00640 1 -- 172.21.15.67:0/4195802152 wait complete. 2024-04-17T08:55:22.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-17T08:55:22.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.503+0000 7fe54da00640 1 Processor -- start 2024-04-17T08:55:22.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.503+0000 7fe54da00640 1 -- start start 2024-04-17T08:55:22.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.504+0000 7fe54da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe5480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe54da00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe548072570 0x7fe5480ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe54da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54815d060 con 0x7fe5480b63c0 2024-04-17T08:55:22.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe54da00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe5480cfe90 con 0x7fe548072570 2024-04-17T08:55:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe54da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe54815dc70 con 0x7fe5480cbc00 2024-04-17T08:55:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe547400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe5480b63c0 0x7fe548071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34226/0 (socket says 172.21.15.67:34226) 2024-04-17T08:55:22.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe547400640 1 -- 172.21.15.67:0/2386143867 learned_addr learned my addr 172.21.15.67:0/2386143867 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe547e00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe5480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.505+0000 7fe54d000640 1 --2- 172.21.15.67:0/2386143867 >> v2:172.21.15.187:3300/0 conn(0x7fe548072570 0x7fe5480ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 943604077 0 0) 0x7fe54815d060 con 0x7fe5480b63c0 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe52c003520 con 0x7fe5480b63c0 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 402495487 0 0) 0x7fe52c003520 con 0x7fe5480b63c0 2024-04-17T08:55:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 msgr2=0x7fe5480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe5480cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 >> v2:172.21.15.187:3300/0 conn(0x7fe548072570 msgr2=0x7fe5480ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 --2- 172.21.15.67:0/2386143867 >> v2:172.21.15.187:3300/0 conn(0x7fe548072570 0x7fe5480ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54815d340 con 0x7fe5480b63c0 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fe534001670 con 0x7fe5480b63c0 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 >> v1:172.21.15.67:6789/0 conn(0x7fe5480b63c0 legacy=0x7fe548071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52c003a20 0x7fe52c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe52c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe52c004d40 0x7fe52c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480cfe90 con 0x7fe52c003a20 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe54815dc70 con 0x7fe52c004d40 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe52c00b9c0 con 0x7fe5480cbc00 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.506+0000 7fe547e00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52c003a20 0x7fe52c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe54d000640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe52c004d40 0x7fe52c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547400640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe52c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547e00640 1 -- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 msgr2=0x7fe52c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547e00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe52c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547e00640 1 -- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe52c004d40 msgr2=0x7fe52c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547e00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe52c004d40 0x7fe52c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547e00640 1 -- 172.21.15.67:0/2386143867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52c003520 con 0x7fe52c003a20 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.507+0000 7fe547400640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5480cbc00 0x7fe52c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.508+0000 7fe547e00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52c003a20 0x7fe52c003e40 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fe538009ac0 tx=0x7fe52c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650c5e3d09ce2487 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.508+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538039070 con 0x7fe52c003a20 2024-04-17T08:55:22.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53802c6b0 con 0x7fe52c003a20 2024-04-17T08:55:22.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe546a00640 1 -- 172.21.15.67:0/2386143867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538009690 con 0x7fe52c003a20 2024-04-17T08:55:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 -- 172.21.15.67:0/2386143867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52c003a20 msgr2=0x7fe52c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 --2- 172.21.15.67:0/2386143867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe52c003a20 0x7fe52c003e40 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fe538009ac0 tx=0x7fe52c005830 comp rx=0 tx=0).stop 2024-04-17T08:55:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54d000640 1 -- 172.21.15.67:0/2386143867 reap_dead start 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 -- 172.21.15.67:0/2386143867 shutdown_connections 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 -- 172.21.15.67:0/2386143867 >> 172.21.15.67:0/2386143867 conn(0x7fe54806bef0 msgr2=0x7fe5480b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 -- 172.21.15.67:0/2386143867 shutdown_connections 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.509+0000 7fe54da00640 1 -- 172.21.15.67:0/2386143867 wait complete. 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.511+0000 7fe54da00640 1 Processor -- start 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.511+0000 7fe54da00640 1 -- start start 2024-04-17T08:55:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 0x7fe54806f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe548143fa0 0x7fe5480c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480d02d0 con 0x7fe54814f440 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe54815d900 con 0x7fe548143fa0 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe54da00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe5480cbfe0 con 0x7fe52c003a20 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.512+0000 7fe547400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47458/0 (socket says 172.21.15.67:47458) 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 -- 172.21.15.67:0/1055013251 learned_addr learned my addr 172.21.15.67:0/1055013251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:22.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe54d000640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe548143fa0 0x7fe5480c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547e00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 0x7fe54806f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 -- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 msgr2=0x7fe54806f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 0x7fe54806f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 -- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe548143fa0 msgr2=0x7fe5480c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe548143fa0 0x7fe5480c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 -- 172.21.15.67:0/1055013251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe548144430 con 0x7fe54814f440 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547e00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 0x7fe54806f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:22.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.513+0000 7fe547400640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fe53400d840 tx=0x7fe53400dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.514+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534019070 con 0x7fe54814f440 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.514+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5340074e0 con 0x7fe54814f440 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.514+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534004a20 con 0x7fe54814f440 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.514+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54806e400 con 0x7fe54814f440 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.514+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54806e9c0 con 0x7fe54814f440 2024-04-17T08:55:22.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.515+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe548068a10 con 0x7fe54814f440 2024-04-17T08:55:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.516+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fe534004bc0 con 0x7fe54814f440 2024-04-17T08:55:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.517+0000 7fe53fe00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 0x7fe52806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.517+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fe53408b810 con 0x7fe54814f440 2024-04-17T08:55:22.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.518+0000 7fe547e00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 0x7fe52806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.519+0000 7fe547e00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 0x7fe52806ef00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538030000 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.522+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fe534059a00 con 0x7fe54814f440 2024-04-17T08:55:22.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.727+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fe5480afc10 con 0x7fe52806ca40 2024-04-17T08:55:22.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.728+0000 7fe53fe00640 1 -- 172.21.15.67:0/1055013251 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe5480afc10 con 0x7fe52806ca40 2024-04-17T08:55:22.727 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-04-17T08:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.730+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 msgr2=0x7fe52806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.730+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 0x7fe52806ef00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538030000 comp rx=0 tx=0).stop 2024-04-17T08:55:22.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.730+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 msgr2=0x7fe5480b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.730+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7fe53400d840 tx=0x7fe53400dd00 comp rx=0 tx=0).stop 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.730+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 shutdown_connections 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fe52806ca40 0x7fe52806ef00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe548143fa0 0x7fe5480c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe54814f440 0x7fe5480b1880 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 --2- 172.21.15.67:0/1055013251 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe52c003a20 0x7fe54806f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 >> 172.21.15.67:0/1055013251 conn(0x7fe54806bef0 msgr2=0x7fe5480cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 shutdown_connections 2024-04-17T08:55:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.731+0000 7fe54da00640 1 -- 172.21.15.67:0/1055013251 wait complete. 2024-04-17T08:55:22.749 INFO:tasks.ceph:Scrubbing osd.3 2024-04-17T08:55:22.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.978+0000 7fb032600640 1 Processor -- start 2024-04-17T08:55:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.978+0000 7fb032600640 1 -- start start 2024-04-17T08:55:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.979+0000 7fb032600640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb02c0cd6d0 0x7fb02c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.979+0000 7fb032600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02c146510 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.979+0000 7fb032600640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02c161000 con 0x7fb02c146080 2024-04-17T08:55:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.979+0000 7fb032600640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb02c0cfa60 con 0x7fb02c0cd6d0 2024-04-17T08:55:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.980+0000 7fb031200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb02c0cd2f0 0x7fb02c0cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34234/0 (socket says 172.21.15.67:34234) 2024-04-17T08:55:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.980+0000 7fb031200640 1 -- 172.21.15.67:0/3483547096 learned_addr learned my addr 172.21.15.67:0/3483547096 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.980+0000 7fb02be00640 1 --2- 172.21.15.67:0/3483547096 >> v2:172.21.15.187:3300/0 conn(0x7fb02c0cd6d0 0x7fb02c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.980+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164777746 0 0) 0x7fb02c146510 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.980+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb014003520 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3941086021 0 0) 0x7fb014003520 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 >> v2:172.21.15.187:3300/0 conn(0x7fb02c0cd6d0 msgr2=0x7fb02c145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 --2- 172.21.15.67:0/3483547096 >> v2:172.21.15.187:3300/0 conn(0x7fb02c0cd6d0 0x7fb02c145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 >> v1:172.21.15.187:6789/0 conn(0x7fb02c146080 legacy=0x7fb02c160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c0ce870 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fb020001670 con 0x7fb02c0cd2f0 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 >> v1:172.21.15.67:6789/0 conn(0x7fb02c0cd2f0 legacy=0x7fb02c0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 0x7fb014003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014004800 0x7fb014004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140051c0 0x7fb01400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c0cfa60 con 0x7fb0140039e0 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb01400c090 con 0x7fb014004800 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb01400c290 con 0x7fb0140051c0 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb031200640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 0x7fb014003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb031200640 1 -- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140051c0 msgr2=0x7fb01400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb031200640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140051c0 0x7fb01400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb031200640 1 -- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014004800 msgr2=0x7fb014004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.981+0000 7fb02be00640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014004800 0x7fb014004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.982+0000 7fb031200640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb014004800 0x7fb014004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.982+0000 7fb031200640 1 -- 172.21.15.67:0/3483547096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb014003520 con 0x7fb0140039e0 2024-04-17T08:55:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.982+0000 7fb031c00640 1 -- 172.21.15.67:0/3483547096 reap_dead start 2024-04-17T08:55:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb031200640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 0x7fb014003e00 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fb02000c240 tx=0x7fb014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6798151944c396bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020007cf0 con 0x7fb0140039e0 2024-04-17T08:55:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb020030070 con 0x7fb0140039e0 2024-04-17T08:55:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb02b400640 1 -- 172.21.15.67:0/3483547096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020005d70 con 0x7fb0140039e0 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb032600640 1 -- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 msgr2=0x7fb014003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.983+0000 7fb032600640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 0x7fb014003e00 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7fb02000c240 tx=0x7fb014005830 comp rx=0 tx=0).stop 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.984+0000 7fb032600640 1 -- 172.21.15.67:0/3483547096 shutdown_connections 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.984+0000 7fb032600640 1 --2- 172.21.15.67:0/3483547096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0140039e0 0x7fb014003e00 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.984+0000 7fb032600640 1 -- 172.21.15.67:0/3483547096 >> 172.21.15.67:0/3483547096 conn(0x7fb02c06b8c0 msgr2=0x7fb02c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.984+0000 7fb032600640 1 -- 172.21.15.67:0/3483547096 shutdown_connections 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.984+0000 7fb032600640 1 -- 172.21.15.67:0/3483547096 wait complete. 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.985+0000 7fb032600640 1 Processor -- start 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.986+0000 7fb032600640 1 -- start start 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140039e0 0x7fb02c13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 0x7fb02c13e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb02c13ca70 0x7fb02c1448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c145b40 con 0x7fb02c13c5f0 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c0cee30 con 0x7fb02c13ca70 2024-04-17T08:55:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb032600640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb02c145cc0 con 0x7fb0140039e0 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 0x7fb02c13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 0x7fb02c13e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47466/0 (socket says 172.21.15.67:47466) 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 -- 172.21.15.67:0/17672061 learned_addr learned my addr 172.21.15.67:0/17672061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb031200640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140039e0 0x7fb02c13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 -- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140039e0 msgr2=0x7fb02c13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140039e0 0x7fb02c13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb031c00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb02c13ca70 0x7fb02c1448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 -- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb02c13ca70 msgr2=0x7fb02c1448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb02c13ca70 0x7fb02c1448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb02be00640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c140510 con 0x7fb02c13c5f0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.987+0000 7fb031200640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0140039e0 0x7fb02c13ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb02be00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 0x7fb02c13e110 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7fb0180107c0 tx=0x7fb018010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01800f040 con 0x7fb02c13c5f0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb01800eb60 con 0x7fb02c13c5f0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01801c400 con 0x7fb02c13c5f0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02c14bcf0 con 0x7fb02c13c5f0 2024-04-17T08:55:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.988+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02c14c2b0 con 0x7fb02c13c5f0 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.989+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb02c068a10 con 0x7fb02c13c5f0 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.990+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fb01801c5a0 con 0x7fb02c13c5f0 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.991+0000 7fb028c00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb00c06ca10 0x7fb00c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.991+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fb01808c370 con 0x7fb02c13c5f0 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.992+0000 7fb028c00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fb00c072fb0 0x7fb00c0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.992+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb01800ae50 con 0x7fb00c072fb0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.992+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7fb018093050 con 0x7fb02c13c5f0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.992+0000 7fb031c00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fb00c072fb0 0x7fb00c0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.992+0000 7fb031200640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb00c06ca10 0x7fb00c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.993+0000 7fb031c00640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fb00c072fb0 0x7fb00c0753d0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.993+0000 7fb031200640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb00c06ca10 0x7fb00c06eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb020034000 tx=0x7fb020039040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:22.993+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== osd.3 v2:172.21.15.67:6804/100610741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb01800ae50 con 0x7fb00c072fb0 2024-04-17T08:55:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.008+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 --> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb02c068610 con 0x7fb00c072fb0 2024-04-17T08:55:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb028c00640 1 -- 172.21.15.67:0/17672061 <== osd.3 v2:172.21.15.67:6804/100610741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb02c068610 con 0x7fb00c072fb0 2024-04-17T08:55:23.008 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:23.009 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:23.009 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fb00c072fb0 msgr2=0x7fb00c0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:6804/100610741,v1:172.21.15.67:6806/100610741] conn(0x7fb00c072fb0 0x7fb00c0753d0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb00c06ca10 msgr2=0x7fb00c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fb00c06ca10 0x7fb00c06eed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb020034000 tx=0x7fb020039040 comp rx=0 tx=0).stop 2024-04-17T08:55:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 msgr2=0x7fb02c13e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 --2- 172.21.15.67:0/17672061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb02c13c5f0 0x7fb02c13e110 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7fb0180107c0 tx=0x7fb018010c80 comp rx=0 tx=0).stop 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb031c00640 1 -- 172.21.15.67:0/17672061 reap_dead start 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 shutdown_connections 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.009+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 >> 172.21.15.67:0/17672061 conn(0x7fb02c06b8c0 msgr2=0x7fb02c071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.010+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 shutdown_connections 2024-04-17T08:55:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.010+0000 7fb032600640 1 -- 172.21.15.67:0/17672061 wait complete. 2024-04-17T08:55:23.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-17T08:55:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.249+0000 7fc09d400640 1 Processor -- start 2024-04-17T08:55:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.249+0000 7fc09d400640 1 -- start start 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc09d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc0980cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc09d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980b63c0 0x7fc098071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc09d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc09815d060 con 0x7fc0980723a0 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc09d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc0980cfe90 con 0x7fc0980b63c0 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc09d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc09815dc70 con 0x7fc0980cbc00 2024-04-17T08:55:23.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.251+0000 7fc097400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980b63c0 0x7fc098071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980b63c0 0x7fc098071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47476/0 (socket says 172.21.15.67:47476) 2024-04-17T08:55:23.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 -- 172.21.15.67:0/2344476908 learned_addr learned my addr 172.21.15.67:0/2344476908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 -- 172.21.15.67:0/2344476908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 msgr2=0x7fc0980cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097e00640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc0980cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc0980cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 -- 172.21.15.67:0/2344476908 >> v1:172.21.15.67:6789/0 conn(0x7fc0980723a0 legacy=0x7fc098160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097400640 1 -- 172.21.15.67:0/2344476908 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09815d340 con 0x7fc0980b63c0 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.252+0000 7fc097e00640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc0980cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:23.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.253+0000 7fc097400640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980b63c0 0x7fc098071e60 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fc084009870 tx=0x7fc08402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2dc70d8619cfc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.254+0000 7fc096a00640 1 -- 172.21.15.67:0/2344476908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084034070 con 0x7fc0980b63c0 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.254+0000 7fc096a00640 1 -- 172.21.15.67:0/2344476908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc084030840 con 0x7fc0980b63c0 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.254+0000 7fc096a00640 1 -- 172.21.15.67:0/2344476908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084030c10 con 0x7fc0980b63c0 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.254+0000 7fc09d400640 1 -- 172.21.15.67:0/2344476908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980b63c0 msgr2=0x7fc098071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.254+0000 7fc09d400640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980b63c0 0x7fc098071e60 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fc084009870 tx=0x7fc08402b980 comp rx=0 tx=0).stop 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 -- 172.21.15.67:0/2344476908 shutdown_connections 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980b63c0 0x7fc098071e60 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 --2- 172.21.15.67:0/2344476908 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc0980cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 -- 172.21.15.67:0/2344476908 >> 172.21.15.67:0/2344476908 conn(0x7fc09806bef0 msgr2=0x7fc0980b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 -- 172.21.15.67:0/2344476908 shutdown_connections 2024-04-17T08:55:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.255+0000 7fc09d400640 1 -- 172.21.15.67:0/2344476908 wait complete. 2024-04-17T08:55:23.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.257+0000 7fc09d400640 1 Processor -- start 2024-04-17T08:55:23.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.257+0000 7fc09d400640 1 -- start start 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0980b63c0 0x7fc098152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc09815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0980d02d0 con 0x7fc0980723a0 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815d900 con 0x7fc0980b63c0 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc09d400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0980cbfe0 con 0x7fc0980cbc00 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47486/0 (socket says 172.21.15.67:47486) 2024-04-17T08:55:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 -- 172.21.15.67:0/4159556920 learned_addr learned my addr 172.21.15.67:0/4159556920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 -- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 msgr2=0x7fc09815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc09815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 -- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0980b63c0 msgr2=0x7fc098152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0980b63c0 0x7fc098152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.258+0000 7fc097e00640 1 -- 172.21.15.67:0/4159556920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09815aef0 con 0x7fc0980723a0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc097e00640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c820 tx=0x7fc08800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088015070 con 0x7fc0980723a0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc088007a30 con 0x7fc0980723a0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088007de0 con 0x7fc0980723a0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09815b1d0 con 0x7fc0980723a0 2024-04-17T08:55:23.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.259+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09815b790 con 0x7fc0980723a0 2024-04-17T08:55:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.260+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc098068a10 con 0x7fc0980723a0 2024-04-17T08:55:23.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.261+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fc088018070 con 0x7fc0980723a0 2024-04-17T08:55:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.262+0000 7fc08fe00640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 0x7fc07806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.262+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fc08808bea0 con 0x7fc0980723a0 2024-04-17T08:55:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.263+0000 7fc097400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 0x7fc07806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.264+0000 7fc097400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 0x7fc07806eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc084036000 tx=0x7fc08403b040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.267+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc088059f30 con 0x7fc0980723a0 2024-04-17T08:55:23.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.472+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc0980b3020 con 0x7fc07806ca10 2024-04-17T08:55:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.473+0000 7fc08fe00640 1 -- 172.21.15.67:0/4159556920 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc0980b3020 con 0x7fc07806ca10 2024-04-17T08:55:23.473 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-04-17T08:55:23.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 msgr2=0x7fc07806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 0x7fc07806eed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc084036000 tx=0x7fc08403b040 comp rx=0 tx=0).stop 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 msgr2=0x7fc0981f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fc08800c820 tx=0x7fc08800cce0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 shutdown_connections 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc07806ca10 0x7fc07806eed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc0980cbc00 0x7fc09815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0980b63c0 0x7fc098152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 --2- 172.21.15.67:0/4159556920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980723a0 0x7fc0981f2c00 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 >> 172.21.15.67:0/4159556920 conn(0x7fc09806bef0 msgr2=0x7fc0980b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 shutdown_connections 2024-04-17T08:55:23.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.476+0000 7fc09d400640 1 -- 172.21.15.67:0/4159556920 wait complete. 2024-04-17T08:55:23.494 INFO:tasks.ceph:Scrubbing osd.4 2024-04-17T08:55:23.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:23.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.720+0000 7f9c72800640 1 Processor -- start 2024-04-17T08:55:23.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.721+0000 7f9c72800640 1 -- start start 2024-04-17T08:55:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.722+0000 7f9c72800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.722+0000 7f9c72800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6c0d0390 con 0x7f9c6c15c8c0 2024-04-17T08:55:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c72800640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6c0d0670 con 0x7f9c6c0b4390 2024-04-17T08:55:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c72800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9c6c15cca0 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49442/0 (socket says 172.21.15.67:49442) 2024-04-17T08:55:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 -- 172.21.15.67:0/3248467733 learned_addr learned my addr 172.21.15.67:0/3248467733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 -- 172.21.15.67:0/3248467733 >> v1:172.21.15.187:6789/0 conn(0x7f9c6c0b4390 legacy=0x7f9c6c0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:23.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 -- 172.21.15.67:0/3248467733 >> v1:172.21.15.67:6789/0 conn(0x7f9c6c15c8c0 legacy=0x7f9c6c160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.723+0000 7f9c6aa00640 1 -- 172.21.15.67:0/3248467733 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6c1604e0 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.725+0000 7f9c6aa00640 1 --2- 172.21.15.67:0/3248467733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009870 tx=0x7f9c5402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c15ad41f5ed9c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.725+0000 7f9c6a000640 1 -- 172.21.15.67:0/3248467733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c54035020 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.725+0000 7f9c6a000640 1 -- 172.21.15.67:0/3248467733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c5402d040 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.725+0000 7f9c6a000640 1 -- 172.21.15.67:0/3248467733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c54031a80 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 -- 172.21.15.67:0/3248467733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 msgr2=0x7f9c6c0d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 --2- 172.21.15.67:0/3248467733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009870 tx=0x7f9c5402b900 comp rx=0 tx=0).stop 2024-04-17T08:55:23.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 -- 172.21.15.67:0/3248467733 shutdown_connections 2024-04-17T08:55:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 --2- 172.21.15.67:0/3248467733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0d0d00 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 -- 172.21.15.67:0/3248467733 >> 172.21.15.67:0/3248467733 conn(0x7f9c6c0d27b0 msgr2=0x7f9c6c0d2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 -- 172.21.15.67:0/3248467733 shutdown_connections 2024-04-17T08:55:23.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.726+0000 7f9c72800640 1 -- 172.21.15.67:0/3248467733 wait complete. 2024-04-17T08:55:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.728+0000 7f9c72800640 1 Processor -- start 2024-04-17T08:55:23.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.728+0000 7f9c72800640 1 -- start start 2024-04-17T08:55:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c6c0b4390 0x7f9c6c1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c6c15c8c0 0x7f9c6c0c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c0d1240 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6c0d0950 con 0x7f9c6c15c8c0 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c72800640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6c0c7b00 con 0x7f9c6c0b4390 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49448/0 (socket says 172.21.15.67:49448) 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c6be00640 1 -- 172.21.15.67:0/3697914503 learned_addr learned my addr 172.21.15.67:0/3697914503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c6b400640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c6c0b4390 0x7f9c6c1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.729+0000 7f9c6be00640 1 -- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c6c0b4390 msgr2=0x7f9c6c1550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6be00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c6c0b4390 0x7f9c6c1550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6aa00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c6c15c8c0 0x7f9c6c0c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6be00640 1 -- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c6c15c8c0 msgr2=0x7f9c6c0c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6be00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c6c15c8c0 0x7f9c6c0c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6be00640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6c155a60 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c6be00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0cb820 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c7b0 tx=0x7f9c6400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64004b60 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c64004d00 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6400aa90 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c6c155d40 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.730+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c6c156300 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.731+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c6c0717c0 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.733+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f9c6400ac30 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c63400640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f9c4806ca10 0x7f9c4806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f9c6408c140 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c63400640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f9c48072fb0 0x7f9c480753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c64009eb0 con 0x7f9c48072fb0 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7f9c64093050 con 0x7f9c6c15d9d0 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.734+0000 7f9c6b400640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f9c4806ca10 0x7f9c4806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.735+0000 7f9c6aa00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f9c48072fb0 0x7f9c480753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.735+0000 7f9c6b400640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f9c4806ca10 0x7f9c4806eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0015a0 tx=0x7f9c5c009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.736+0000 7f9c6aa00640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f9c48072fb0 0x7f9c480753d0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.738+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== osd.4 v2:172.21.15.187:6800/27895010 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c64009eb0 con 0x7f9c48072fb0 2024-04-17T08:55:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.752+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 --> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9c6c068d60 con 0x7f9c48072fb0 2024-04-17T08:55:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.754+0000 7f9c63400640 1 -- 172.21.15.67:0/3697914503 <== osd.4 v2:172.21.15.187:6800/27895010 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9c6c068d60 con 0x7f9c48072fb0 2024-04-17T08:55:23.753 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:23.754 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.754+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f9c48072fb0 msgr2=0x7f9c480753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.754+0000 7f9c72800640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6800/27895010,v1:172.21.15.187:6801/27895010] conn(0x7f9c48072fb0 0x7f9c480753d0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f9c4806ca10 msgr2=0x7f9c4806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f9c4806ca10 0x7f9c4806eed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0015a0 tx=0x7f9c5c009040 comp rx=0 tx=0).stop 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 msgr2=0x7f9c6c0cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 --2- 172.21.15.67:0/3697914503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c6c15d9d0 0x7f9c6c0cb820 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400c7b0 tx=0x7f9c6400cc70 comp rx=0 tx=0).stop 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c6be00640 1 -- 172.21.15.67:0/3697914503 reap_dead start 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 shutdown_connections 2024-04-17T08:55:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 >> 172.21.15.67:0/3697914503 conn(0x7f9c6c0d27b0 msgr2=0x7f9c6c0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 shutdown_connections 2024-04-17T08:55:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.755+0000 7f9c72800640 1 -- 172.21.15.67:0/3697914503 wait complete. 2024-04-17T08:55:23.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-17T08:55:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.998+0000 7f2c27e00640 1 Processor -- start 2024-04-17T08:55:23.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:23.998+0000 7f2c27e00640 1 -- start start 2024-04-17T08:55:23.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c280afc70 0x7f2c2815fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c27e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2c28162740 0x7f2c280c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:23.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c27e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c280ca390 con 0x7f2c280c9f80 2024-04-17T08:55:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c27e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c2806e1b0 con 0x7f2c280afc70 2024-04-17T08:55:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c27e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2c281600e0 con 0x7f2c28162740 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.000+0000 7f2c26a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c280afc70 0x7f2c2815fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c280afc70 0x7f2c2815fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49452/0 (socket says 172.21.15.67:49452) 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 -- 172.21.15.67:0/2095771975 learned_addr learned my addr 172.21.15.67:0/2095771975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c1e000640 1 --2- 172.21.15.67:0/2095771975 >> v2:172.21.15.187:3300/0 conn(0x7f2c28162740 0x7f2c280c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 -- 172.21.15.67:0/2095771975 >> v2:172.21.15.187:3300/0 conn(0x7f2c28162740 msgr2=0x7f2c280c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:24.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 --2- 172.21.15.67:0/2095771975 >> v2:172.21.15.187:3300/0 conn(0x7f2c28162740 0x7f2c280c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 -- 172.21.15.67:0/2095771975 >> v1:172.21.15.187:6789/0 conn(0x7f2c280c9f80 legacy=0x7f2c28071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.001+0000 7f2c26a00640 1 -- 172.21.15.67:0/2095771975 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c28162ce0 con 0x7f2c280afc70 2024-04-17T08:55:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.003+0000 7f2c26a00640 1 --2- 172.21.15.67:0/2095771975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280afc70 0x7f2c2815fba0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f2c28148890 tx=0x7f2c1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bfed907e0e578e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c26000640 1 -- 172.21.15.67:0/2095771975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18035020 con 0x7f2c280afc70 2024-04-17T08:55:24.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c26000640 1 -- 172.21.15.67:0/2095771975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c18031870 con 0x7f2c280afc70 2024-04-17T08:55:24.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c26000640 1 -- 172.21.15.67:0/2095771975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18031c40 con 0x7f2c280afc70 2024-04-17T08:55:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 -- 172.21.15.67:0/2095771975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280afc70 msgr2=0x7f2c2815fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 --2- 172.21.15.67:0/2095771975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280afc70 0x7f2c2815fba0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f2c28148890 tx=0x7f2c1802b910 comp rx=0 tx=0).stop 2024-04-17T08:55:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 -- 172.21.15.67:0/2095771975 shutdown_connections 2024-04-17T08:55:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 --2- 172.21.15.67:0/2095771975 >> v2:172.21.15.187:3300/0 conn(0x7f2c28162740 0x7f2c280c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 --2- 172.21.15.67:0/2095771975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280afc70 0x7f2c2815fba0 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 -- 172.21.15.67:0/2095771975 >> 172.21.15.67:0/2095771975 conn(0x7f2c2806bef0 msgr2=0x7f2c280b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 -- 172.21.15.67:0/2095771975 shutdown_connections 2024-04-17T08:55:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.004+0000 7f2c27e00640 1 -- 172.21.15.67:0/2095771975 wait complete. 2024-04-17T08:55:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.006+0000 7f2c27e00640 1 Processor -- start 2024-04-17T08:55:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.006+0000 7f2c27e00640 1 -- start start 2024-04-17T08:55:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c280afc70 0x7f2c28143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c28162740 0x7f2c281e8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28163580 con 0x7f2c280c9f80 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2c281632a0 con 0x7f2c28162740 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c27e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2c280b0050 con 0x7f2c280afc70 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.007+0000 7f2c1e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49458/0 (socket says 172.21.15.67:49458) 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 -- 172.21.15.67:0/3996773954 learned_addr learned my addr 172.21.15.67:0/3996773954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c27400640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c28162740 0x7f2c281e8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c26a00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c280afc70 0x7f2c28143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 -- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c280afc70 msgr2=0x7f2c28143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c280afc70 0x7f2c28143e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 -- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c28162740 msgr2=0x7f2c281e8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c28162740 0x7f2c281e8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 -- 172.21.15.67:0/3996773954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c280b4bd0 con 0x7f2c280c9f80 2024-04-17T08:55:24.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.008+0000 7f2c1e000640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000efc0 tx=0x7f2c1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.009+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10018020 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.009+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c10014480 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.009+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c280b4eb0 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.009+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10014850 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.009+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c280b5470 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.010+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c28144340 con 0x7f2c280c9f80 2024-04-17T08:55:24.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.011+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f2c10005d10 con 0x7f2c280c9f80 2024-04-17T08:55:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.012+0000 7f2c1f400640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 0x7f2c0406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.012+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f2c1008cd90 con 0x7f2c280c9f80 2024-04-17T08:55:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.013+0000 7f2c26a00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 0x7f2c0406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.014+0000 7f2c26a00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 0x7f2c0406eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036000 tx=0x7f2c18008030 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.017+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f2c10022030 con 0x7f2c280c9f80 2024-04-17T08:55:24.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.221+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f2c28159390 con 0x7f2c0406ca10 2024-04-17T08:55:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.222+0000 7f2c1f400640 1 -- 172.21.15.67:0/3996773954 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2c28159390 con 0x7f2c0406ca10 2024-04-17T08:55:24.222 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 msgr2=0x7f2c0406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 0x7f2c0406eed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036000 tx=0x7f2c18008030 comp rx=0 tx=0).stop 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 msgr2=0x7f2c281e8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000efc0 tx=0x7f2c1000c490 comp rx=0 tx=0).stop 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 shutdown_connections 2024-04-17T08:55:24.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f2c0406ca10 0x7f2c0406eed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2c28162740 0x7f2c281e8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c280c9f80 0x7f2c281e8810 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 --2- 172.21.15.67:0/3996773954 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2c280afc70 0x7f2c28143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.225+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 >> 172.21.15.67:0/3996773954 conn(0x7f2c2806bef0 msgr2=0x7f2c28161720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.226+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 shutdown_connections 2024-04-17T08:55:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.226+0000 7f2c27e00640 1 -- 172.21.15.67:0/3996773954 wait complete. 2024-04-17T08:55:24.243 INFO:tasks.ceph:Scrubbing osd.5 2024-04-17T08:55:24.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:24.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.471+0000 7f09e7000640 1 Processor -- start 2024-04-17T08:55:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.471+0000 7f09e7000640 1 -- start start 2024-04-17T08:55:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.472+0000 7f09e7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 0x7f09e00ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e7000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09e0145030 0x7f09e006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e7000640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e006df10 con 0x7f09e0144a70 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e7000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f09e00cd460 con 0x7f09e0145030 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09e00cd740 con 0x7f09e00cf650 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09e0145030 0x7f09e006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09e0145030 0x7f09e006f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49464/0 (socket says 172.21.15.67:49464) 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 -- 172.21.15.67:0/540577193 learned_addr learned my addr 172.21.15.67:0/540577193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5c00640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 0x7f09e00ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 -- 172.21.15.67:0/540577193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 msgr2=0x7f09e00ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:24.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 0x7f09e00ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 -- 172.21.15.67:0/540577193 >> v1:172.21.15.187:6789/0 conn(0x7f09e0144a70 legacy=0x7f09e006e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5200640 1 -- 172.21.15.67:0/540577193 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e00cda20 con 0x7f09e0145030 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.473+0000 7f09e5c00640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 0x7f09e00ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.475+0000 7f09e5200640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0145030 0x7f09e006f600 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009870 tx=0x7f09cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d900dcc7ab3d98e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.475+0000 7f09dfe00640 1 -- 172.21.15.67:0/540577193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc035020 con 0x7f09e0145030 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.475+0000 7f09dfe00640 1 -- 172.21.15.67:0/540577193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc031640 con 0x7f09e0145030 2024-04-17T08:55:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.475+0000 7f09dfe00640 1 -- 172.21.15.67:0/540577193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc02d040 con 0x7f09e0145030 2024-04-17T08:55:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 -- 172.21.15.67:0/540577193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0145030 msgr2=0x7f09e006f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0145030 0x7f09e006f600 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009870 tx=0x7f09cc02b8f0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 -- 172.21.15.67:0/540577193 shutdown_connections 2024-04-17T08:55:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0145030 0x7f09e006f600 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 --2- 172.21.15.67:0/540577193 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e00cf650 0x7f09e00ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 -- 172.21.15.67:0/540577193 >> 172.21.15.67:0/540577193 conn(0x7f09e006b8c0 msgr2=0x7f09e006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 -- 172.21.15.67:0/540577193 shutdown_connections 2024-04-17T08:55:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.476+0000 7f09e7000640 1 -- 172.21.15.67:0/540577193 wait complete. 2024-04-17T08:55:24.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.478+0000 7f09e7000640 1 Processor -- start 2024-04-17T08:55:24.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.478+0000 7f09e7000640 1 -- start start 2024-04-17T08:55:24.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09e00cf650 0x7f09e0135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 0x7f09e00b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e0145030 0x7f09e00b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09e00cd070 con 0x7f09e0144a70 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f09e00cea50 con 0x7f09e00cf650 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e7000640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09e00cd1f0 con 0x7f09e0145030 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 0x7f09e00b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 0x7f09e00b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49480/0 (socket says 172.21.15.67:49480) 2024-04-17T08:55:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 -- 172.21.15.67:0/3338090949 learned_addr learned my addr 172.21.15.67:0/3338090949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5c00640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09e00cf650 0x7f09e0135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e6600640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e0145030 0x7f09e00b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 -- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e0145030 msgr2=0x7f09e00b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09e0145030 0x7f09e00b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 -- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09e00cf650 msgr2=0x7f09e0135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09e00cf650 0x7f09e0135470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.479+0000 7f09e5200640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e01359b0 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09e5200640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 0x7f09e00b0310 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f09cc02b930 tx=0x7f09cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc03ac60 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09cc03ae20 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e01343d0 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc044630 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.480+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e0134990 con 0x7f09e0144a70 2024-04-17T08:55:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.481+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09e0068a10 con 0x7f09e0144a70 2024-04-17T08:55:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.483+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f09cc04b060 con 0x7f09e0144a70 2024-04-17T08:55:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09dd600640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f09c006ca10 0x7f09c006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f09cc0ae360 con 0x7f09e0144a70 2024-04-17T08:55:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09dd600640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f09c0072fb0 0x7f09c00753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09cc04ee50 con 0x7f09c0072fb0 2024-04-17T08:55:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7f09cc0ae710 con 0x7f09e0144a70 2024-04-17T08:55:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.484+0000 7f09e5c00640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f09c006ca10 0x7f09c006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.485+0000 7f09e6600640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f09c0072fb0 0x7f09c00753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.486+0000 7f09e6600640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f09c0072fb0 0x7f09c00753d0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.486+0000 7f09e5c00640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f09c006ca10 0x7f09c006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09d400b840 tx=0x7f09d4008040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.488+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== osd.5 v2:172.21.15.187:6816/656328999 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09cc04ee50 con 0x7f09c0072fb0 2024-04-17T08:55:24.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.503+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 --> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f09e0134e20 con 0x7f09c0072fb0 2024-04-17T08:55:24.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09dd600640 1 -- 172.21.15.67:0/3338090949 <== osd.5 v2:172.21.15.187:6816/656328999 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f09e0134e20 con 0x7f09c0072fb0 2024-04-17T08:55:24.507 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:24.507 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:24.507 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f09c0072fb0 msgr2=0x7f09c00753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6816/656328999,v1:172.21.15.187:6818/656328999] conn(0x7f09c0072fb0 0x7f09c00753d0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f09c006ca10 msgr2=0x7f09c006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f09c006ca10 0x7f09c006eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09d400b840 tx=0x7f09d4008040 comp rx=0 tx=0).stop 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 msgr2=0x7f09e00b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e7000640 1 --2- 172.21.15.67:0/3338090949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e0144a70 0x7f09e00b0310 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f09cc02b930 tx=0x7f09cc0046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.504+0000 7f09e6600640 1 -- 172.21.15.67:0/3338090949 reap_dead start 2024-04-17T08:55:24.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.505+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 shutdown_connections 2024-04-17T08:55:24.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.505+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 >> 172.21.15.67:0/3338090949 conn(0x7f09e006b8c0 msgr2=0x7f09e00af740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.505+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 shutdown_connections 2024-04-17T08:55:24.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.505+0000 7f09e7000640 1 -- 172.21.15.67:0/3338090949 wait complete. 2024-04-17T08:55:24.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-17T08:55:24.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.751+0000 7fcbc3e00640 1 Processor -- start 2024-04-17T08:55:24.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.752+0000 7fcbc3e00640 1 -- start start 2024-04-17T08:55:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbc40cbc00 0x7fcbc40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc415d060 con 0x7fcbc4072570 2024-04-17T08:55:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc3e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc415d340 con 0x7fcbc40b63c0 2024-04-17T08:55:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc40cfe90 con 0x7fcbc40cbc00 2024-04-17T08:55:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbc40cbc00 0x7fcbc40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbc40cbc00 0x7fcbc40cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49492/0 (socket says 172.21.15.67:49492) 2024-04-17T08:55:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.753+0000 7fcbc2a00640 1 -- 172.21.15.67:0/348840746 learned_addr learned my addr 172.21.15.67:0/348840746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.754+0000 7fcbc2a00640 1 -- 172.21.15.67:0/348840746 >> v1:172.21.15.187:6789/0 conn(0x7fcbc40b63c0 legacy=0x7fcbc4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.754+0000 7fcbc2a00640 1 -- 172.21.15.67:0/348840746 >> v1:172.21.15.67:6789/0 conn(0x7fcbc4072570 legacy=0x7fcbc40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.754+0000 7fcbc2a00640 1 -- 172.21.15.67:0/348840746 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc415d620 con 0x7fcbc40cbc00 2024-04-17T08:55:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.755+0000 7fcbc2a00640 1 --2- 172.21.15.67:0/348840746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc40cbc00 0x7fcbc40cf950 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40098a0 tx=0x7fcbb402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39ff4bfc67dc4f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.755+0000 7fcbc1600640 1 -- 172.21.15.67:0/348840746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4034070 con 0x7fcbc40cbc00 2024-04-17T08:55:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc1600640 1 -- 172.21.15.67:0/348840746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb4030840 con 0x7fcbc40cbc00 2024-04-17T08:55:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc1600640 1 -- 172.21.15.67:0/348840746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4030c10 con 0x7fcbc40cbc00 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 -- 172.21.15.67:0/348840746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc40cbc00 msgr2=0x7fcbc40cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/348840746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc40cbc00 0x7fcbc40cf950 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40098a0 tx=0x7fcbb402b980 comp rx=0 tx=0).stop 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 -- 172.21.15.67:0/348840746 shutdown_connections 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/348840746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc40cbc00 0x7fcbc40cf950 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 -- 172.21.15.67:0/348840746 >> 172.21.15.67:0/348840746 conn(0x7fcbc406bef0 msgr2=0x7fcbc40b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 -- 172.21.15.67:0/348840746 shutdown_connections 2024-04-17T08:55:24.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.756+0000 7fcbc3e00640 1 -- 172.21.15.67:0/348840746 wait complete. 2024-04-17T08:55:24.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.758+0000 7fcbc3e00640 1 Processor -- start 2024-04-17T08:55:24.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.758+0000 7fcbc3e00640 1 -- start start 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 0x7fcbc4152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 0x7fcbc414fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc40d02d0 con 0x7fcbc4072570 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc415dc70 con 0x7fcbc40cbc00 2024-04-17T08:55:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.759+0000 7fcbc3e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc40cbfe0 con 0x7fcbc40b63c0 2024-04-17T08:55:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49500/0 (socket says 172.21.15.67:49500) 2024-04-17T08:55:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 -- 172.21.15.67:0/1594407434 learned_addr learned my addr 172.21.15.67:0/1594407434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2000640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 0x7fcbc4152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc3400640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 0x7fcbc414fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 -- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 msgr2=0x7fcbc4152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 0x7fcbc4152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 -- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 msgr2=0x7fcbc414fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 0x7fcbc414fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2a00640 1 -- 172.21.15.67:0/1594407434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc4156b60 con 0x7fcbc4072570 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc2000640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 0x7fcbc4152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.760+0000 7fcbc3400640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 0x7fcbc414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:24.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbc2a00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4034040 con 0x7fcbc4072570 2024-04-17T08:55:24.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb4039d90 con 0x7fcbc4072570 2024-04-17T08:55:24.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4043610 con 0x7fcbc4072570 2024-04-17T08:55:24.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc4157290 con 0x7fcbc4072570 2024-04-17T08:55:24.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.761+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc41567c0 con 0x7fcbc4072570 2024-04-17T08:55:24.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.763+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc4068a10 con 0x7fcbc4072570 2024-04-17T08:55:24.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.763+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fcbb404a060 con 0x7fcbc4072570 2024-04-17T08:55:24.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.764+0000 7fcbbaa00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 0x7fcba406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:24.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.764+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fcbb40ae090 con 0x7fcbc4072570 2024-04-17T08:55:24.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.765+0000 7fcbc2000640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 0x7fcba406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:24.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.766+0000 7fcbc2000640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 0x7fcba406ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbc415a350 tx=0x7fcbb0009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:24.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.769+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fcbb403e070 con 0x7fcbc4072570 2024-04-17T08:55:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.977+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fcbc40b3020 con 0x7fcba406ca40 2024-04-17T08:55:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.979+0000 7fcbbaa00640 1 -- 172.21.15.67:0/1594407434 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcbc40b3020 con 0x7fcba406ca40 2024-04-17T08:55:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-04-17T08:55:24.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 msgr2=0x7fcba406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 0x7fcba406ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcbc415a350 tx=0x7fcbb0009040 comp rx=0 tx=0).stop 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 msgr2=0x7fcbc414eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb40046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 shutdown_connections 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fcba406ca40 0x7fcba406ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc40cbc00 0x7fcbc414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcbc40b63c0 0x7fcbc4152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 --2- 172.21.15.67:0/1594407434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbc4072570 0x7fcbc414eb70 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 >> 172.21.15.67:0/1594407434 conn(0x7fcbc406bef0 msgr2=0x7fcbc40cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.981+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 shutdown_connections 2024-04-17T08:55:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:24.982+0000 7fcbc3e00640 1 -- 172.21.15.67:0/1594407434 wait complete. 2024-04-17T08:55:25.001 INFO:tasks.ceph:Scrubbing osd.6 2024-04-17T08:55:25.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.217+0000 7efff0c00640 1 Processor -- start 2024-04-17T08:55:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.217+0000 7efff0c00640 1 -- start start 2024-04-17T08:55:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7efff0c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7effec0cd2f0 0x7effec0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7efff0c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effec145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7efff0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effec146510 con 0x7effec146080 2024-04-17T08:55:25.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7efff0c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7effec161000 con 0x7effec0cd2f0 2024-04-17T08:55:25.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7efff0c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effec0ce870 con 0x7effec0cd6d0 2024-04-17T08:55:25.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7effebe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7effec146080 0x7effec160360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48502/0 (socket says 172.21.15.67:48502) 2024-04-17T08:55:25.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.219+0000 7effebe00640 1 -- 172.21.15.67:0/3662871492 learned_addr learned my addr 172.21.15.67:0/3662871492 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effeb400640 1 --2- 172.21.15.67:0/3662871492 >> v2:172.21.15.187:3300/0 conn(0x7effec0cd2f0 0x7effec0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effeaa00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effec145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264430903 0 0) 0x7effec146510 con 0x7effec146080 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd0003520 con 0x7effec146080 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3847838741 0 0) 0x7effd0003520 con 0x7effec146080 2024-04-17T08:55:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 msgr2=0x7effec145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effec145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 >> v2:172.21.15.187:3300/0 conn(0x7effec0cd2f0 msgr2=0x7effec0cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 --2- 172.21.15.67:0/3662871492 >> v2:172.21.15.187:3300/0 conn(0x7effec0cd2f0 0x7effec0cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec0ceb50 con 0x7effec146080 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7effe40019f0 con 0x7effec146080 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 >> v1:172.21.15.67:6789/0 conn(0x7effec146080 legacy=0x7effec160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effd00039b0 0x7effd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effec161000 con 0x7effd0004800 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7effec0ce870 con 0x7effd00039b0 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.220+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effd000b9c0 con 0x7effec0cd6d0 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effebe00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 -- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 msgr2=0x7effd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effd00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 -- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effd00039b0 msgr2=0x7effd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeb400640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effd00039b0 0x7effd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effd00039b0 0x7effd0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeaa00640 1 -- 172.21.15.67:0/3662871492 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd0003520 con 0x7effd0004800 2024-04-17T08:55:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effeb400640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effd00039b0 0x7effd0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.221+0000 7effebe00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec0cd6d0 0x7effd00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.222+0000 7effeaa00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effd0004c80 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7effd8009ac0 tx=0x7effd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e705a8abacf6432e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.222+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8039070 con 0x7effd0004800 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.222+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd802c6b0 con 0x7effd0004800 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.222+0000 7effea000640 1 -- 172.21.15.67:0/3662871492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8009690 con 0x7effd0004800 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 -- 172.21.15.67:0/3662871492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 msgr2=0x7effd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 --2- 172.21.15.67:0/3662871492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effd0004c80 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7effd8009ac0 tx=0x7effd0005830 comp rx=0 tx=0).stop 2024-04-17T08:55:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7effebe00640 1 -- 172.21.15.67:0/3662871492 reap_dead start 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 -- 172.21.15.67:0/3662871492 shutdown_connections 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 -- 172.21.15.67:0/3662871492 >> 172.21.15.67:0/3662871492 conn(0x7effec06b8c0 msgr2=0x7effec06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 -- 172.21.15.67:0/3662871492 shutdown_connections 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.223+0000 7efff0c00640 1 -- 172.21.15.67:0/3662871492 wait complete. 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.225+0000 7efff0c00640 1 Processor -- start 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.225+0000 7efff0c00640 1 -- start start 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effec151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effec13d8b0 0x7effec13bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec13c300 0x7effec13e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effec0cfa10 con 0x7effd0004800 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7effec0cf890 con 0x7effec13d8b0 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7efff0c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effec160a70 con 0x7effec13c300 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7effeb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effec151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7effeb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effec151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49522/0 (socket says 172.21.15.67:49522) 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.226+0000 7effeb400640 1 -- 172.21.15.67:0/1775581353 learned_addr learned my addr 172.21.15.67:0/1775581353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effebe00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec13c300 0x7effec13e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeaa00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effec13d8b0 0x7effec13bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 -- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec13c300 msgr2=0x7effec13e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec13c300 0x7effec13e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 -- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effec13d8b0 msgr2=0x7effec13bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effec13d8b0 0x7effec13bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec13c810 con 0x7effd0004800 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeaa00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effec13d8b0 0x7effec13bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effebe00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effec13c300 0x7effec13e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effeb400640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effec151ca0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7effdc010a00 tx=0x7effdc010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc00f040 con 0x7effd0004800 2024-04-17T08:55:25.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effdc00ec20 con 0x7effd0004800 2024-04-17T08:55:25.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.227+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effec13caf0 con 0x7effd0004800 2024-04-17T08:55:25.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.228+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc01c400 con 0x7effd0004800 2024-04-17T08:55:25.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.228+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effec13d0b0 con 0x7effd0004800 2024-04-17T08:55:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.229+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7effec0ce170 con 0x7effd0004800 2024-04-17T08:55:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.230+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7effdc01c5a0 con 0x7effd0004800 2024-04-17T08:55:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effe3400640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7effcc06ca40 0x7effcc06ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7effdc08c4c0 con 0x7effd0004800 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effe3400640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7effcc072fe0 0x7effcc075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7effdc007e90 con 0x7effcc072fe0 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7effdc093050 con 0x7effd0004800 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.231+0000 7effeaa00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7effcc06ca40 0x7effcc06ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.232+0000 7effebe00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7effcc072fe0 0x7effcc075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.233+0000 7effeaa00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7effcc06ca40 0x7effcc06ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effd8036000 tx=0x7effd8030000 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.233+0000 7effebe00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7effcc072fe0 0x7effcc075400 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.235+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== osd.6 v2:172.21.15.187:6817/3359222512 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7effdc007e90 con 0x7effcc072fe0 2024-04-17T08:55:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.250+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 --> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7effec068a10 con 0x7effcc072fe0 2024-04-17T08:55:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7effe3400640 1 -- 172.21.15.67:0/1775581353 <== osd.6 v2:172.21.15.187:6817/3359222512 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7effec068a10 con 0x7effcc072fe0 2024-04-17T08:55:25.250 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:25.251 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:25.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7effcc072fe0 msgr2=0x7effcc075400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6817/3359222512,v1:172.21.15.187:6819/3359222512] conn(0x7effcc072fe0 0x7effcc075400 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7effcc06ca40 msgr2=0x7effcc06ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7effcc06ca40 0x7effcc06ef00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effd8036000 tx=0x7effd8030000 comp rx=0 tx=0).stop 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 msgr2=0x7effec151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.251+0000 7efff0c00640 1 --2- 172.21.15.67:0/1775581353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0004800 0x7effec151ca0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7effdc010a00 tx=0x7effdc010ec0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.252+0000 7effebe00640 1 -- 172.21.15.67:0/1775581353 reap_dead start 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.252+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 shutdown_connections 2024-04-17T08:55:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.252+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 >> 172.21.15.67:0/1775581353 conn(0x7effec06b8c0 msgr2=0x7effec15cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.252+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 shutdown_connections 2024-04-17T08:55:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.252+0000 7efff0c00640 1 -- 172.21.15.67:0/1775581353 wait complete. 2024-04-17T08:55:25.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-17T08:55:25.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.493+0000 7fc167e00640 1 Processor -- start 2024-04-17T08:55:25.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.493+0000 7fc167e00640 1 -- start start 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc167e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1680cbc00 0x7fc1680cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc167e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1680ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc167e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc16815d060 con 0x7fc1680b63c0 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc167e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc1680cfe90 con 0x7fc1680cbc00 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc167e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc16815dc70 con 0x7fc168072570 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.495+0000 7fc166a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1680cbc00 0x7fc1680cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1680cbc00 0x7fc1680cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49536/0 (socket says 172.21.15.67:49536) 2024-04-17T08:55:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 -- 172.21.15.67:0/3454296952 learned_addr learned my addr 172.21.15.67:0/3454296952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc167400640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1680ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 -- 172.21.15.67:0/3454296952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 msgr2=0x7fc1680ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1680ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 -- 172.21.15.67:0/3454296952 >> v1:172.21.15.67:6789/0 conn(0x7fc1680b63c0 legacy=0x7fc168071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc166a00640 1 -- 172.21.15.67:0/3454296952 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16815d340 con 0x7fc1680cbc00 2024-04-17T08:55:25.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.496+0000 7fc167400640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1680ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T08:55:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.497+0000 7fc166a00640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc1680cf950 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fc158009870 tx=0x7fc15802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477490c84dc981b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.498+0000 7fc165600640 1 -- 172.21.15.67:0/3454296952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158034070 con 0x7fc1680cbc00 2024-04-17T08:55:25.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.498+0000 7fc165600640 1 -- 172.21.15.67:0/3454296952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc158030840 con 0x7fc1680cbc00 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.498+0000 7fc165600640 1 -- 172.21.15.67:0/3454296952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158030c10 con 0x7fc1680cbc00 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.498+0000 7fc167e00640 1 -- 172.21.15.67:0/3454296952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 msgr2=0x7fc1680cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.498+0000 7fc167e00640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc1680cf950 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7fc158009870 tx=0x7fc15802b980 comp rx=0 tx=0).stop 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 -- 172.21.15.67:0/3454296952 shutdown_connections 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1680ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 --2- 172.21.15.67:0/3454296952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc1680cf950 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 -- 172.21.15.67:0/3454296952 >> 172.21.15.67:0/3454296952 conn(0x7fc16806bef0 msgr2=0x7fc1680b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 -- 172.21.15.67:0/3454296952 shutdown_connections 2024-04-17T08:55:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.499+0000 7fc167e00640 1 -- 172.21.15.67:0/3454296952 wait complete. 2024-04-17T08:55:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.501+0000 7fc167e00640 1 Processor -- start 2024-04-17T08:55:25.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.501+0000 7fc167e00640 1 -- start start 2024-04-17T08:55:25.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1681566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1680b63c0 0x7fc16815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1680d02d0 con 0x7fc1680cbc00 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc16815d900 con 0x7fc1680b63c0 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1680cbfe0 con 0x7fc168072570 2024-04-17T08:55:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49542/0 (socket says 172.21.15.67:49542) 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.502+0000 7fc167400640 1 -- 172.21.15.67:0/2268126294 learned_addr learned my addr 172.21.15.67:0/2268126294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 -- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 msgr2=0x7fc1681566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1681566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 -- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1680b63c0 msgr2=0x7fc16815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1680b63c0 0x7fc16815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 -- 172.21.15.67:0/2268126294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16815aae0 con 0x7fc1680cbc00 2024-04-17T08:55:25.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167400640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fc160007ef0 tx=0x7fc16000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16001a070 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc160003b90 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160004030 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16814f540 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.503+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc168152790 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.504+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1680cd6e0 con 0x7fc1680cbc00 2024-04-17T08:55:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.505+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fc16001d070 con 0x7fc1680cbc00 2024-04-17T08:55:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.507+0000 7fc15f400640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 0x7fc14806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.507+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fc16008c070 con 0x7fc1680cbc00 2024-04-17T08:55:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.508+0000 7fc166a00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 0x7fc14806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.509+0000 7fc166a00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 0x7fc14806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc158036000 tx=0x7fc158030000 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.512+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fc1680cd6e0 con 0x7fc1680cbc00 2024-04-17T08:55:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.716+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fc168068a10 con 0x7fc14806ca10 2024-04-17T08:55:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.717+0000 7fc15f400640 1 -- 172.21.15.67:0/2268126294 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc168068a10 con 0x7fc14806ca10 2024-04-17T08:55:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-04-17T08:55:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 msgr2=0x7fc14806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 0x7fc14806eed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc158036000 tx=0x7fc158030000 comp rx=0 tx=0).stop 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 msgr2=0x7fc168160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7fc160007ef0 tx=0x7fc16000d4a0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 shutdown_connections 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fc14806ca10 0x7fc14806eed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1680cbc00 0x7fc168160610 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1680b63c0 0x7fc16815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 --2- 172.21.15.67:0/2268126294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc168072570 0x7fc1681566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 >> 172.21.15.67:0/2268126294 conn(0x7fc16806bef0 msgr2=0x7fc1680b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 shutdown_connections 2024-04-17T08:55:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.720+0000 7fc167e00640 1 -- 172.21.15.67:0/2268126294 wait complete. 2024-04-17T08:55:25.739 INFO:tasks.ceph:Scrubbing osd.7 2024-04-17T08:55:25.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-17T08:55:25.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.966+0000 7f3643e00640 1 Processor -- start 2024-04-17T08:55:25.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.966+0000 7f3643e00640 1 -- start start 2024-04-17T08:55:25.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.968+0000 7f3643e00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f36440b3be0 0x7f3644155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.968+0000 7f3643e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36440b4070 con 0x7f36440cfc80 2024-04-17T08:55:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.968+0000 7f3643e00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f364415f890 con 0x7f36440b3800 2024-04-17T08:55:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.968+0000 7f3643e00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f36441629b0 con 0x7f36440b3be0 2024-04-17T08:55:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.969+0000 7f363be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f36440cfc80 0x7f3644162e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48518/0 (socket says 172.21.15.67:48518) 2024-04-17T08:55:25.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.969+0000 7f363be00640 1 -- 172.21.15.67:0/2451414509 learned_addr learned my addr 172.21.15.67:0/2451414509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.969+0000 7f3643400640 1 --2- 172.21.15.67:0/2451414509 >> v2:172.21.15.187:3300/0 conn(0x7f36440b3be0 0x7f3644155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.969+0000 7f3642000640 1 -- 172.21.15.67:0/2451414509 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3927034928 0 0) 0x7f36440b4070 con 0x7f36440cfc80 2024-04-17T08:55:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.969+0000 7f3642000640 1 -- 172.21.15.67:0/2451414509 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3628003520 con 0x7f36440cfc80 2024-04-17T08:55:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.970+0000 7f3643400640 1 -- 172.21.15.67:0/2451414509 >> v1:172.21.15.187:6789/0 conn(0x7f36440b3800 legacy=0x7f364415f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.970+0000 7f3643400640 1 -- 172.21.15.67:0/2451414509 >> v1:172.21.15.67:6789/0 conn(0x7f36440cfc80 legacy=0x7f3644162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.970+0000 7f3643400640 1 -- 172.21.15.67:0/2451414509 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364415fb70 con 0x7f36440b3be0 2024-04-17T08:55:25.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.971+0000 7f3643400640 1 --2- 172.21.15.67:0/2451414509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b3be0 0x7f3644155560 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baf3993e31d81486 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.971+0000 7f3642000640 1 -- 172.21.15.67:0/2451414509 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c039070 con 0x7f36440b3be0 2024-04-17T08:55:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.971+0000 7f3642000640 1 -- 172.21.15.67:0/2451414509 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363c02cd80 con 0x7f36440b3be0 2024-04-17T08:55:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.971+0000 7f3642000640 1 -- 172.21.15.67:0/2451414509 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c034640 con 0x7f36440b3be0 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 -- 172.21.15.67:0/2451414509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b3be0 msgr2=0x7f3644155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 --2- 172.21.15.67:0/2451414509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b3be0 0x7f3644155560 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c02bfd0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 -- 172.21.15.67:0/2451414509 shutdown_connections 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 --2- 172.21.15.67:0/2451414509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440b3be0 0x7f3644155560 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 -- 172.21.15.67:0/2451414509 >> 172.21.15.67:0/2451414509 conn(0x7f3644067600 msgr2=0x7f36440679e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 -- 172.21.15.67:0/2451414509 shutdown_connections 2024-04-17T08:55:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.972+0000 7f3643e00640 1 -- 172.21.15.67:0/2451414509 wait complete. 2024-04-17T08:55:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.974+0000 7f3643e00640 1 Processor -- start 2024-04-17T08:55:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.974+0000 7f3643e00640 1 -- start start 2024-04-17T08:55:25.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 0x7f36440b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f36440b3be0 0x7f3644160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440cfc80 0x7f36441613c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3644162640 con 0x7f36440b3800 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3644163550 con 0x7f36440cfc80 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3643e00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3644155aa0 con 0x7f36440b3be0 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3642a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 0x7f36440b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3642a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 0x7f36440b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49552/0 (socket says 172.21.15.67:49552) 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.975+0000 7f3642a00640 1 -- 172.21.15.67:0/416595840 learned_addr learned my addr 172.21.15.67:0/416595840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:25.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3643400640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440cfc80 0x7f36441613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 -- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f36440b3be0 msgr2=0x7f3644160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f36440b3be0 0x7f3644160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 -- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440cfc80 msgr2=0x7f36441613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36440cfc80 0x7f36441613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3644153620 con 0x7f36440b3800 2024-04-17T08:55:25.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3642a00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 0x7f36440b2f50 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f363400c3f0 tx=0x7f363400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36340052d0 con 0x7f36440b3800 2024-04-17T08:55:25.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3634005490 con 0x7f36440b3800 2024-04-17T08:55:25.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634007260 con 0x7f36440b3800 2024-04-17T08:55:25.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.976+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3644145970 con 0x7f36440b3800 2024-04-17T08:55:25.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.977+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3644142d70 con 0x7f36440b3800 2024-04-17T08:55:25.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.977+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3644143430 con 0x7f36440b3800 2024-04-17T08:55:25.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.979+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f3634007400 con 0x7f36440b3800 2024-04-17T08:55:25.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.980+0000 7f363aa00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f362006ca40 0x7f362006ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.980+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f363408b250 con 0x7f36440b3800 2024-04-17T08:55:25.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.980+0000 7f363aa00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f3620072fe0 0x7f3620075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:25.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.980+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3634009e50 con 0x7f3620072fe0 2024-04-17T08:55:25.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.980+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) ==== 24+0+0 (secure 0 0 0) 0x7f3634092050 con 0x7f36440b3800 2024-04-17T08:55:25.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.981+0000 7f363be00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f362006ca40 0x7f362006ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.981+0000 7f3643400640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f3620072fe0 0x7f3620075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:25.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.982+0000 7f363be00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f362006ca40 0x7f362006ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3644067c00 tx=0x7f362c007930 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.982+0000 7f3643400640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f3620072fe0 0x7f3620075400 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:25.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:25.985+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== osd.7 v2:172.21.15.187:6808/1997375062 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3634009e50 con 0x7f3620072fe0 2024-04-17T08:55:26.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.000+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 --> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3644068a10 con 0x7f3620072fe0 2024-04-17T08:55:26.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f363aa00640 1 -- 172.21.15.67:0/416595840 <== osd.7 v2:172.21.15.187:6808/1997375062 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3644068a10 con 0x7f3620072fe0 2024-04-17T08:55:26.001 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T08:55:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-17T08:55:26.002 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f3620072fe0 msgr2=0x7f3620075400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6808/1997375062,v1:172.21.15.187:6809/1997375062] conn(0x7f3620072fe0 0x7f3620075400 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f362006ca40 msgr2=0x7f362006ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f362006ca40 0x7f362006ef00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3644067c00 tx=0x7f362c007930 comp rx=0 tx=0).stop 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 msgr2=0x7f36440b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.001+0000 7f3643e00640 1 --2- 172.21.15.67:0/416595840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36440b3800 0x7f36440b2f50 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f363400c3f0 tx=0x7f363400c8b0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.002+0000 7f3643400640 1 -- 172.21.15.67:0/416595840 reap_dead start 2024-04-17T08:55:26.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.002+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 shutdown_connections 2024-04-17T08:55:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.002+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 >> 172.21.15.67:0/416595840 conn(0x7f3644067600 msgr2=0x7f36440b99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.002+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 shutdown_connections 2024-04-17T08:55:26.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.002+0000 7f3643e00640 1 -- 172.21.15.67:0/416595840 wait complete. 2024-04-17T08:55:26.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-17T08:55:26.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.234+0000 7f1282c00640 1 Processor -- start 2024-04-17T08:55:26.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.234+0000 7f1282c00640 1 -- start start 2024-04-17T08:55:26.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1282c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f127c0afef0 0x7f127c15b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1282c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c0cf470 con 0x7f127c0b2eb0 2024-04-17T08:55:26.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1282c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c0ce6b0 con 0x7f127c0b2ad0 2024-04-17T08:55:26.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1282c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f127c0727e0 con 0x7f127c0afef0 2024-04-17T08:55:26.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1280e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f127c0b2eb0 0x7f127c0af7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48526/0 (socket says 172.21.15.67:48526) 2024-04-17T08:55:26.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1280e00640 1 -- 172.21.15.67:0/4159009638 learned_addr learned my addr 172.21.15.67:0/4159009638 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:26.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f1282200640 1 --2- 172.21.15.67:0/4159009638 >> v2:172.21.15.187:3300/0 conn(0x7f127c0afef0 0x7f127c15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.236+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444320280 0 0) 0x7f127c0cf470 con 0x7f127c0b2eb0 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1260003520 con 0x7f127c0b2eb0 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3195270540 0 0) 0x7f1260003520 con 0x7f127c0b2eb0 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 >> v2:172.21.15.187:3300/0 conn(0x7f127c0afef0 msgr2=0x7f127c15b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 --2- 172.21.15.67:0/4159009638 >> v2:172.21.15.187:3300/0 conn(0x7f127c0afef0 0x7f127c15b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 >> v1:172.21.15.187:6789/0 conn(0x7f127c0b2ad0 legacy=0x7f127c0720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c0ce990 con 0x7f127c0b2eb0 2024-04-17T08:55:26.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7f1268001670 con 0x7f127c0b2eb0 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 >> v1:172.21.15.67:6789/0 conn(0x7f127c0b2eb0 legacy=0x7f127c0af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1260003a70 0x7f1260003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f1260004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1260005210 0x7f126000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f127c0727e0 con 0x7f1260004800 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f126000c140 con 0x7f1260003a70 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f126000c340 con 0x7f1260005210 2024-04-17T08:55:26.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.237+0000 7f1280e00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f1260004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1281800640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1260003a70 0x7f1260003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1282200640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1260005210 0x7f126000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1280e00640 1 -- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1260005210 msgr2=0x7f126000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1280e00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1260005210 0x7f126000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1280e00640 1 -- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1260003a70 msgr2=0x7f1260003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1280e00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1260003a70 0x7f1260003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1280e00640 1 -- 172.21.15.67:0/4159009638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1260003520 con 0x7f1260004800 2024-04-17T08:55:26.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.238+0000 7f1282200640 1 -- 172.21.15.67:0/4159009638 reap_dead start 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.239+0000 7f1280e00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f1260004c80 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f126800c110 tx=0x7f1260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f3cdc57cc85057 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.239+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268007c40 con 0x7f1260004800 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.239+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1268030070 con 0x7f1260004800 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.239+0000 7f127be00640 1 -- 172.21.15.67:0/4159009638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268005d70 con 0x7f1260004800 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 -- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 msgr2=0x7f1260004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f1260004c80 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f126800c110 tx=0x7f1260005830 comp rx=0 tx=0).stop 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 -- 172.21.15.67:0/4159009638 shutdown_connections 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 --2- 172.21.15.67:0/4159009638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f1260004c80 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 -- 172.21.15.67:0/4159009638 >> 172.21.15.67:0/4159009638 conn(0x7f127c06bef0 msgr2=0x7f127c0b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 -- 172.21.15.67:0/4159009638 shutdown_connections 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.240+0000 7f1282c00640 1 -- 172.21.15.67:0/4159009638 wait complete. 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.242+0000 7f1282c00640 1 Processor -- start 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.242+0000 7f1282c00640 1 -- start start 2024-04-17T08:55:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c1f81b0 0x7f127c1f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c1f51d0 0x7f127c1f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f127c0720d0 con 0x7f1260004800 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f127c0cef50 con 0x7f127c1f81b0 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1282c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f127c072250 con 0x7f127c1f51d0 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.243+0000 7f1281800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49572/0 (socket says 172.21.15.67:49572) 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 -- 172.21.15.67:0/2623323010 learned_addr learned my addr 172.21.15.67:0/2623323010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 -- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c1f51d0 msgr2=0x7f127c1f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c1f51d0 0x7f127c1f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1280e00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c1f81b0 0x7f127c1f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 -- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c1f81b0 msgr2=0x7f127c1f8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c1f81b0 0x7f127c1f8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 -- 172.21.15.67:0/2623323010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c0cd5c0 con 0x7f1260004800 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.244+0000 7f1281800640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f126c007b90 tx=0x7f126c00ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.245+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c00ee30 con 0x7f1260004800 2024-04-17T08:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.245+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126c018070 con 0x7f1260004800 2024-04-17T08:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.245+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127c0cd8a0 con 0x7f1260004800 2024-04-17T08:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.245+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c003da0 con 0x7f1260004800 2024-04-17T08:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.245+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f127c1362e0 con 0x7f1260004800 2024-04-17T08:55:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.246+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f127c068a10 con 0x7f1260004800 2024-04-17T08:55:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.247+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f126c0050d0 con 0x7f1260004800 2024-04-17T08:55:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.248+0000 7f1279600640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 0x7f125c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.248+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f126c08bc80 con 0x7f1260004800 2024-04-17T08:55:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.249+0000 7f1280e00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 0x7f125c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.250+0000 7f1280e00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 0x7f125c06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1268034000 tx=0x7f1268039040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.253+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f126c059f40 con 0x7f1260004800 2024-04-17T08:55:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.459+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f127c1f4df0 con 0x7f125c06ca10 2024-04-17T08:55:26.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.460+0000 7f1279600640 1 -- 172.21.15.67:0/2623323010 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f127c1f4df0 con 0x7f125c06ca10 2024-04-17T08:55:26.459 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-04-17T08:55:26.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 msgr2=0x7f125c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 0x7f125c06eed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1268034000 tx=0x7f1268039040 comp rx=0 tx=0).stop 2024-04-17T08:55:26.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 msgr2=0x7f127c135da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f126c007b90 tx=0x7f126c00ebe0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 shutdown_connections 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f125c06ca10 0x7f125c06eed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f127c1f51d0 0x7f127c1f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f127c1f81b0 0x7f127c1f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 --2- 172.21.15.67:0/2623323010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1260004800 0x7f127c135da0 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 >> 172.21.15.67:0/2623323010 conn(0x7f127c06bef0 msgr2=0x7f127c159080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 shutdown_connections 2024-04-17T08:55:26.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.463+0000 7f1282c00640 1 -- 172.21.15.67:0/2623323010 wait complete. 2024-04-17T08:55:26.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:55:26.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.706+0000 7fa50aa00640 1 Processor -- start 2024-04-17T08:55:26.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.706+0000 7fa50aa00640 1 -- start start 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa50aa00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa504141ed0 0x7fa504142360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa50aa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5040cd4a0 con 0x7fa504160280 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa50aa00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5041607f0 con 0x7fa50406b8c0 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa50aa00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa50415ce30 con 0x7fa504141ed0 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa508c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa504160280 0x7fa50406bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48536/0 (socket says 172.21.15.67:48536) 2024-04-17T08:55:26.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.708+0000 7fa508c00640 1 -- 172.21.15.67:0/1687798311 learned_addr learned my addr 172.21.15.67:0/1687798311 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:26.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa50a000640 1 --2- 172.21.15.67:0/1687798311 >> v2:172.21.15.187:3300/0 conn(0x7fa504141ed0 0x7fa504142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259175302 0 0) 0x7fa5040cd4a0 con 0x7fa504160280 2024-04-17T08:55:26.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ec003520 con 0x7fa504160280 2024-04-17T08:55:26.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2398546263 0 0) 0x7fa4ec003520 con 0x7fa504160280 2024-04-17T08:55:26.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 >> v2:172.21.15.187:3300/0 conn(0x7fa504141ed0 msgr2=0x7fa504142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 --2- 172.21.15.67:0/1687798311 >> v2:172.21.15.187:3300/0 conn(0x7fa504141ed0 0x7fa504142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 >> v1:172.21.15.187:6789/0 conn(0x7fa50406b8c0 legacy=0x7fa5040cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa504160ad0 con 0x7fa504160280 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7fa4f0001670 con 0x7fa504160280 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 >> v1:172.21.15.67:6789/0 conn(0x7fa504160280 legacy=0x7fa50406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 0x7fa4ec003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4ec005210 0x7fa4ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa50415ce30 con 0x7fa4ec003a70 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec00c140 con 0x7fa4ec005210 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.709+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ec00c340 con 0x7fa4ec004800 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 0x7fa4ec003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa508c00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa50a000640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4ec005210 0x7fa4ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 -- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec004800 msgr2=0x7fa4ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec004800 0x7fa4ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 -- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4ec005210 msgr2=0x7fa4ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4ec005210 0x7fa4ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa50a000640 1 -- 172.21.15.67:0/1687798311 reap_dead start 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.710+0000 7fa509600640 1 -- 172.21.15.67:0/1687798311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec003520 con 0x7fa4ec003a70 2024-04-17T08:55:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.712+0000 7fa509600640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 0x7fa4ec003e90 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8001750 tx=0x7fa4ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62291d5f69ef1ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.712+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8001370 con 0x7fa4ec003a70 2024-04-17T08:55:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.712+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f80052d0 con 0x7fa4ec003a70 2024-04-17T08:55:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.712+0000 7fa503e00640 1 -- 172.21.15.67:0/1687798311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8003830 con 0x7fa4ec003a70 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 -- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 msgr2=0x7fa4ec003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 0x7fa4ec003e90 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8001750 tx=0x7fa4ec005830 comp rx=0 tx=0).stop 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 -- 172.21.15.67:0/1687798311 shutdown_connections 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 --2- 172.21.15.67:0/1687798311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4ec003a70 0x7fa4ec003e90 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 -- 172.21.15.67:0/1687798311 >> 172.21.15.67:0/1687798311 conn(0x7fa504067d10 msgr2=0x7fa5040b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 -- 172.21.15.67:0/1687798311 shutdown_connections 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.713+0000 7fa50aa00640 1 -- 172.21.15.67:0/1687798311 wait complete. 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.715+0000 7fa50aa00640 1 Processor -- start 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.715+0000 7fa50aa00640 1 -- start start 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec003a70 0x7fa504143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa504155900 0x7fa504155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa50413f9b0 con 0x7fa504144ca0 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa504161090 con 0x7fa504155900 2024-04-17T08:55:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa50aa00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa50413fb30 con 0x7fa4ec003a70 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa508c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa508c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49594/0 (socket says 172.21.15.67:49594) 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.716+0000 7fa508c00640 1 -- 172.21.15.67:0/631683688 learned_addr learned my addr 172.21.15.67:0/631683688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa50a000640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa504155900 0x7fa504155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa509600640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec003a70 0x7fa504143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 -- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec003a70 msgr2=0x7fa504143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec003a70 0x7fa504143380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 -- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa504155900 msgr2=0x7fa504155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa504155900 0x7fa504155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 -- 172.21.15.67:0/631683688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa504145510 con 0x7fa504144ca0 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa508c00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000e580 tx=0x7fa4f000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0012590 con 0x7fa504144ca0 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f0018070 con 0x7fa504144ca0 2024-04-17T08:55:26.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0012990 con 0x7fa504144ca0 2024-04-17T08:55:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.717+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5041457f0 con 0x7fa504144ca0 2024-04-17T08:55:26.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.718+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa504145db0 con 0x7fa504144ca0 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.719+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa504068a10 con 0x7fa504144ca0 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.720+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7fa4f0012b30 con 0x7fa504144ca0 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.721+0000 7fa501600640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 0x7fa4e406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.721+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7fa4f008b780 con 0x7fa504144ca0 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.721+0000 7fa509600640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 0x7fa4e406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:26.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.722+0000 7fa509600640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 0x7fa4e406ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8033000 tx=0x7fa4f8009030 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:26.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.725+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7fa4f00599c0 con 0x7fa504144ca0 2024-04-17T08:55:26.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.927+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5040d10d0 con 0x7fa4e406ca40 2024-04-17T08:55:26.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.929+0000 7fa501600640 1 -- 172.21.15.67:0/631683688 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13391 (secure 0 0 0) 0x7fa5040d10d0 con 0x7fa4e406ca40 2024-04-17T08:55:26.929 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:26.929 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 msgr2=0x7fa4e406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 0x7fa4e406ef00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8033000 tx=0x7fa4f8009030 comp rx=0 tx=0).stop 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 msgr2=0x7fa50415b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000e580 tx=0x7fa4f000ea40 comp rx=0 tx=0).stop 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 shutdown_connections 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7fa4e406ca40 0x7fa4e406ef00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa504155900 0x7fa504155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa504144ca0 0x7fa50415b8f0 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 --2- 172.21.15.67:0/631683688 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4ec003a70 0x7fa504143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:26.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 >> 172.21.15.67:0/631683688 conn(0x7fa504067d10 msgr2=0x7fa504074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:26.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 shutdown_connections 2024-04-17T08:55:26.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:26.932+0000 7fa50aa00640 1 -- 172.21.15.67:0/631683688 wait complete. 2024-04-17T08:55:26.951 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-17T08:55:26.858933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":464312,"kb_used_data":30280,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754510408,"statfs":{"total":773094113280,"available":772618657792,"internally_reserved":0,"allocated":31006720,"data_stored":15912576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003419"},"pg_stats":[{"pgid":"1.0","version":"291'2793","reported_seq":4239,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-17T08:55:13.929134+0000","last_change":"2024-04-17T07:13:53.439370+0000","last_active":"2024-04-17T08:55:13.929134+0000","last_peered":"2024-04-17T08:55:13.929134+0000","last_clean":"2024-04-17T08:55:13.929134+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:55:13.929134+0000","last_undegraded":"2024-04-17T08:55:13.929134+0000","last_fullsized":"2024-04-17T08:55:13.929134+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T07:13:52.101442+0000","last_clean_scrub_stamp":"2024-04-17T07:13:52.101442+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:44:11.676464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2793,"ondisk_log_size":2793,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543346,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53444,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318396,"statfs":{"total":96636764160,"available":96582037504,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61192,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310648,"statfs":{"total":96636764160,"available":96574103552,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52868,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318972,"statfs":{"total":96636764160,"available":96582627328,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543345,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84740,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287100,"statfs":{"total":96636764160,"available":96549990400,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52872,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318968,"statfs":{"total":96636764160,"available":96582623232,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52876,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48708,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323132,"statfs":{"total":96636764160,"available":96586887168,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:55:26.954 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-17T07:13:52.101442+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=17, tm_hour=7, tm_min=13, tm_sec=52, tm_wday=2, tm_yday=108, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=17, tm_hour=8, tm_min=55, tm_sec=19, tm_wday=2, tm_yday=108, tm_isdst=0) 2024-04-17T08:55:26.954 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-17T08:55:29.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:55:29.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:55:46.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T08:55:47.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.187+0000 7f71e2400640 1 Processor -- start 2024-04-17T08:55:47.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.187+0000 7f71e2400640 1 -- start start 2024-04-17T08:55:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71e2400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71dc160280 0x7f71dc06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71e2400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71dc0cd4a0 con 0x7f71dc141ed0 2024-04-17T08:55:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71e2400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71dc1607f0 con 0x7f71dc06b8c0 2024-04-17T08:55:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71e2400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f71dc15ce30 con 0x7f71dc160280 2024-04-17T08:55:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71dc160280 0x7f71dc06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71dc160280 0x7f71dc06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49412/0 (socket says 172.21.15.67:49412) 2024-04-17T08:55:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 -- 172.21.15.67:0/4077015374 learned_addr learned my addr 172.21.15.67:0/4077015374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 -- 172.21.15.67:0/4077015374 >> v1:172.21.15.187:6789/0 conn(0x7f71dc06b8c0 legacy=0x7f71dc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 -- 172.21.15.67:0/4077015374 >> v1:172.21.15.67:6789/0 conn(0x7f71dc141ed0 legacy=0x7f71dc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.189+0000 7f71dbe00640 1 -- 172.21.15.67:0/4077015374 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc160ad0 con 0x7f71dc160280 2024-04-17T08:55:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.191+0000 7f71dbe00640 1 --2- 172.21.15.67:0/4077015374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc160280 0x7f71dc06bca0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009870 tx=0x7f71c802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9866cbb0a7f4dc09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.191+0000 7f71db400640 1 -- 172.21.15.67:0/4077015374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8034070 con 0x7f71dc160280 2024-04-17T08:55:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.191+0000 7f71db400640 1 -- 172.21.15.67:0/4077015374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c8030850 con 0x7f71dc160280 2024-04-17T08:55:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.191+0000 7f71db400640 1 -- 172.21.15.67:0/4077015374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8030c20 con 0x7f71dc160280 2024-04-17T08:55:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 -- 172.21.15.67:0/4077015374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc160280 msgr2=0x7f71dc06bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 --2- 172.21.15.67:0/4077015374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc160280 0x7f71dc06bca0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009870 tx=0x7f71c802b990 comp rx=0 tx=0).stop 2024-04-17T08:55:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 -- 172.21.15.67:0/4077015374 shutdown_connections 2024-04-17T08:55:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 --2- 172.21.15.67:0/4077015374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc160280 0x7f71dc06bca0 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 -- 172.21.15.67:0/4077015374 >> 172.21.15.67:0/4077015374 conn(0x7f71dc067d10 msgr2=0x7f71dc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 -- 172.21.15.67:0/4077015374 shutdown_connections 2024-04-17T08:55:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.192+0000 7f71e2400640 1 -- 172.21.15.67:0/4077015374 wait complete. 2024-04-17T08:55:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.194+0000 7f71e2400640 1 Processor -- start 2024-04-17T08:55:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.194+0000 7f71e2400640 1 -- start start 2024-04-17T08:55:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 0x7f71dc13ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71dc160280 0x7f71dc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc161090 con 0x7f71dc141ed0 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc06c1e0 con 0x7f71dc06b8c0 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71e2400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc13f9b0 con 0x7f71dc160280 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49416/0 (socket says 172.21.15.67:49416) 2024-04-17T08:55:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.195+0000 7f71dbe00640 1 -- 172.21.15.67:0/1787742075 learned_addr learned my addr 172.21.15.67:0/1787742075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 -- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71dc160280 msgr2=0x7f71dc0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71e1000640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 0x7f71dc13ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71e1a00640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71dc160280 0x7f71dc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71dc160280 0x7f71dc0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 -- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 msgr2=0x7f71dc13ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 0x7f71dc13ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 -- 172.21.15.67:0/1787742075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc145270 con 0x7f71dc141ed0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71e1000640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 0x7f71dc13ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71dbe00640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f71c80090b0 tx=0x7f71c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.196+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8034040 con 0x7f71dc141ed0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.197+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c80443f0 con 0x7f71dc141ed0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.197+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c80447e0 con 0x7f71dc141ed0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.197+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71dc145550 con 0x7f71dc141ed0 2024-04-17T08:55:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.197+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71dc145b10 con 0x7f71dc141ed0 2024-04-17T08:55:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.198+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc068a10 con 0x7f71dc141ed0 2024-04-17T08:55:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.199+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f71c804b060 con 0x7f71dc141ed0 2024-04-17T08:55:47.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.200+0000 7f71d8c00640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 0x7f71bc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.200+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f71c80ae090 con 0x7f71dc141ed0 2024-04-17T08:55:47.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.201+0000 7f71e1000640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 0x7f71bc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.202+0000 7f71e1000640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 0x7f71bc06eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f71d0003eb0 tx=0x7f71d0009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.205+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f71c803f070 con 0x7f71dc141ed0 2024-04-17T08:55:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.405+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 --> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f71dc0b5530 con 0x7f71bc06ca10 2024-04-17T08:55:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.409+0000 7f71d8c00640 1 -- 172.21.15.67:0/1787742075 <== mgr.10216 v2:172.21.15.187:6832/3601707721 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13402 (secure 0 0 0) 0x7f71d001b470 con 0x7f71bc06ca10 2024-04-17T08:55:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T08:55:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 msgr2=0x7f71bc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 0x7f71bc06eed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f71d0003eb0 tx=0x7f71d0009040 comp rx=0 tx=0).stop 2024-04-17T08:55:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 msgr2=0x7f71dc0b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f71c80090b0 tx=0x7f71c80046c0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 shutdown_connections 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f71bc06ca10 0x7f71bc06eed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f71dc160280 0x7f71dc0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71dc141ed0 0x7f71dc0b1280 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 --2- 172.21.15.67:0/1787742075 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71dc06b8c0 0x7f71dc13ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 >> 172.21.15.67:0/1787742075 conn(0x7f71dc067d10 msgr2=0x7f71dc0cfbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 shutdown_connections 2024-04-17T08:55:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.411+0000 7f71e2400640 1 -- 172.21.15.67:0/1787742075 wait complete. 2024-04-17T08:55:47.431 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-17T08:55:46.864836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2794,"ondisk_log_size":2794,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":464304,"kb_used_data":30280,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754510416,"statfs":{"total":773094113280,"available":772618665984,"internally_reserved":0,"allocated":31006720,"data_stored":15912576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":444321328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"1.0","version":"295'2794","reported_seq":4249,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-17T08:55:27.293923+0000","last_change":"2024-04-17T08:55:27.293999+0000","last_active":"2024-04-17T08:55:27.293923+0000","last_peered":"2024-04-17T08:55:27.293923+0000","last_clean":"2024-04-17T08:55:27.293923+0000","last_became_active":"2024-04-17T07:13:53.438272+0000","last_became_peered":"2024-04-17T07:13:53.438272+0000","last_unstale":"2024-04-17T08:55:27.293923+0000","last_undegraded":"2024-04-17T08:55:27.293923+0000","last_fullsized":"2024-04-17T08:55:27.293923+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'2794","last_scrub_stamp":"2024-04-17T08:55:27.293735+0000","last_deep_scrub":"295'2794","last_deep_scrub_stamp":"2024-04-17T08:55:27.293735+0000","last_clean_scrub_stamp":"2024-04-17T08:55:27.293735+0000","objects_scrubbed":2,"log_size":2794,"log_dups_size":0,"ondisk_log_size":2794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:53:34.544226+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2889,"num_read_kb":6070,"num_write":729,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2794,"ondisk_log_size":2794,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129543350,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53444,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318396,"statfs":{"total":96636764160,"available":96582037504,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129543350,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61192,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310648,"statfs":{"total":96636764160,"available":96574103552,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129543349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52868,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318972,"statfs":{"total":96636764160,"available":96582627328,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129543349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84740,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287100,"statfs":{"total":96636764160,"available":96549990400,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129543351,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52872,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318968,"statfs":{"total":96636764160,"available":96582623232,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129543350,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52876,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129543351,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48708,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323132,"statfs":{"total":96636764160,"available":96586887168,"internally_reserved":0,"allocated":3727360,"data_stored":1841480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129543349,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57604,"kb_used_data":4220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314236,"statfs":{"total":96636764160,"available":96577777664,"internally_reserved":0,"allocated":4321280,"data_stored":2431848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T08:55:47.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-17T08:55:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.642+0000 7efe42400640 1 Processor -- start 2024-04-17T08:55:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.642+0000 7efe42400640 1 -- start start 2024-04-17T08:55:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe42400640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe3c0b4fd0 0x7efe3c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe42400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c0d00b0 con 0x7efe3c15ecb0 2024-04-17T08:55:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe42400640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c0d0390 con 0x7efe3c0d33e0 2024-04-17T08:55:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe42400640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efe3c0d0c30 con 0x7efe3c0b4fd0 2024-04-17T08:55:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe41000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efe3c15ecb0 0x7efe3c0d2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34690/0 (socket says 172.21.15.67:34690) 2024-04-17T08:55:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.644+0000 7efe41000640 1 -- 172.21.15.67:0/1484370213 learned_addr learned my addr 172.21.15.67:0/1484370213 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T08:55:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe41a00640 1 --2- 172.21.15.67:0/1484370213 >> v2:172.21.15.187:3300/0 conn(0x7efe3c0b4fd0 0x7efe3c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583942842 0 0) 0x7efe3c0d00b0 con 0x7efe3c15ecb0 2024-04-17T08:55:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe24003520 con 0x7efe3c15ecb0 2024-04-17T08:55:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1455394069 0 0) 0x7efe24003520 con 0x7efe3c15ecb0 2024-04-17T08:55:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 >> v2:172.21.15.187:3300/0 conn(0x7efe3c0b4fd0 msgr2=0x7efe3c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 --2- 172.21.15.67:0/1484370213 >> v2:172.21.15.187:3300/0 conn(0x7efe3c0b4fd0 0x7efe3c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 >> v1:172.21.15.187:6789/0 conn(0x7efe3c0d33e0 legacy=0x7efe3c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c0d0670 con 0x7efe3c15ecb0 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 884137876 0 0) 0x7efe30001670 con 0x7efe3c15ecb0 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 >> v1:172.21.15.67:6789/0 conn(0x7efe3c15ecb0 legacy=0x7efe3c0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe24003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe24004800 0x7efe24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24005210 0x7efe2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c0d0c30 con 0x7efe240039b0 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efe2400c140 con 0x7efe24005210 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.645+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efe2400c340 con 0x7efe24004800 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe24003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41a00640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24005210 0x7efe2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe3be00640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe24004800 0x7efe24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 -- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe24004800 msgr2=0x7efe24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe24004800 0x7efe24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 -- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24005210 msgr2=0x7efe2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe24005210 0x7efe2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41a00640 1 -- 172.21.15.67:0/1484370213 reap_dead start 2024-04-17T08:55:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.646+0000 7efe41000640 1 -- 172.21.15.67:0/1484370213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe24003520 con 0x7efe240039b0 2024-04-17T08:55:47.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.648+0000 7efe41000640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe24003df0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c240 tx=0x7efe24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d69aa6c12c128b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.648+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30007cf0 con 0x7efe240039b0 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe30030070 con 0x7efe240039b0 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe3b400640 1 -- 172.21.15.67:0/1484370213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30005d70 con 0x7efe240039b0 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe42400640 1 -- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 msgr2=0x7efe24003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe42400640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe24003df0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c240 tx=0x7efe24005830 comp rx=0 tx=0).stop 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe42400640 1 -- 172.21.15.67:0/1484370213 shutdown_connections 2024-04-17T08:55:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe42400640 1 --2- 172.21.15.67:0/1484370213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe24003df0 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.649+0000 7efe42400640 1 -- 172.21.15.67:0/1484370213 >> 172.21.15.67:0/1484370213 conn(0x7efe3c08bff0 msgr2=0x7efe3c0d46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.650+0000 7efe42400640 1 -- 172.21.15.67:0/1484370213 shutdown_connections 2024-04-17T08:55:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.650+0000 7efe42400640 1 -- 172.21.15.67:0/1484370213 wait complete. 2024-04-17T08:55:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.652+0000 7efe42400640 1 Processor -- start 2024-04-17T08:55:47.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.652+0000 7efe42400640 1 -- start start 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe3c162050 0x7efe3c0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe3c0cb120 0x7efe3c06da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c15f210 con 0x7efe240039b0 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c15f090 con 0x7efe3c162050 2024-04-17T08:55:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe42400640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efe3c0d2cd0 con 0x7efe3c0cb120 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe41000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe41000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49440/0 (socket says 172.21.15.67:49440) 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.654+0000 7efe41000640 1 -- 172.21.15.67:0/3161311955 learned_addr learned my addr 172.21.15.67:0/3161311955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 -- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe3c0cb120 msgr2=0x7efe3c06da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe3c0cb120 0x7efe3c06da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe3be00640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe3c162050 0x7efe3c0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 -- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe3c162050 msgr2=0x7efe3c0cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe3c162050 0x7efe3c0cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 -- 172.21.15.67:0/3161311955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c160fb0 con 0x7efe240039b0 2024-04-17T08:55:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe41000640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c360 tx=0x7efe3c0d0560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30004bf0 con 0x7efe240039b0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe30000ed0 con 0x7efe240039b0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30007070 con 0x7efe240039b0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.655+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3c161290 con 0x7efe240039b0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.656+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe3c161850 con 0x7efe240039b0 2024-04-17T08:55:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.657+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe3c068a10 con 0x7efe240039b0 2024-04-17T08:55:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.658+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7efe30048020 con 0x7efe240039b0 2024-04-17T08:55:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.659+0000 7efe38c00640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 0x7efe1c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:47.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.659+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7efe300acfb0 con 0x7efe240039b0 2024-04-17T08:55:47.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.660+0000 7efe3be00640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 0x7efe1c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:47.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.661+0000 7efe3be00640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 0x7efe1c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe28007fd0 tx=0x7efe28009040 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:47.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.665+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7efe3007b310 con 0x7efe240039b0 2024-04-17T08:55:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.866+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7efe3c06d810 con 0x7efe240039b0 2024-04-17T08:55:47.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.873+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) ==== 123+0+0 (secure 0 0 0) 0x7efe3007ac10 con 0x7efe240039b0 2024-04-17T08:55:47.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.876+0000 7efe38c00640 1 -- 172.21.15.67:0/3161311955 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efe30030020 con 0x7efe240039b0 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 msgr2=0x7efe1c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 0x7efe1c06eed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe28007fd0 tx=0x7efe28009040 comp rx=0 tx=0).stop 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 msgr2=0x7efe3c15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7efe3000c360 tx=0x7efe3c0d0560 comp rx=0 tx=0).stop 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 shutdown_connections 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7efe1c06ca10 0x7efe1c06eed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe3c0cb120 0x7efe3c06da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe3c162050 0x7efe3c0cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 --2- 172.21.15.67:0/3161311955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe240039b0 0x7efe3c15ae00 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 >> 172.21.15.67:0/3161311955 conn(0x7efe3c08bff0 msgr2=0x7efe3c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 shutdown_connections 2024-04-17T08:55:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:47.877+0000 7efe42400640 1 -- 172.21.15.67:0/3161311955 wait complete. 2024-04-17T08:55:47.897 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T08:55:47.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T08:55:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.117+0000 7f99c7c00640 1 Processor -- start 2024-04-17T08:55:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.117+0000 7f99c7c00640 1 -- start start 2024-04-17T08:55:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c7c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c7c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c7c00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c006ff90 con 0x7f99c00cd020 2024-04-17T08:55:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c7c00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f99c00ceb00 con 0x7f99c006f030 2024-04-17T08:55:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c7c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f99c00cede0 con 0x7f99c006f670 2024-04-17T08:55:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.119+0000 7f99c5c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5200640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5c00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.67:40496/0 (socket says 172.21.15.67:40496) 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5c00640 1 -- 172.21.15.67:0/2456079575 learned_addr learned my addr 172.21.15.67:0/2456079575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5200640 1 -- 172.21.15.67:0/2456079575 >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 msgr2=0x7f99c00cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5200640 1 --2- 172.21.15.67:0/2456079575 >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5200640 1 -- 172.21.15.67:0/2456079575 >> v1:172.21.15.187:6789/0 conn(0x7f99c00cd020 legacy=0x7f99c0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5200640 1 -- 172.21.15.67:0/2456079575 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c00cf0c0 con 0x7f99c006f670 2024-04-17T08:55:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.120+0000 7f99c5c00640 1 --2- 172.21.15.67:0/2456079575 >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T08:55:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.122+0000 7f99c5200640 1 --2- 172.21.15.67:0/2456079575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c006fa50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f99b40097a0 tx=0x7f99b402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cc4c5fda232a4ff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:48.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.122+0000 7f99bf400640 1 -- 172.21.15.67:0/2456079575 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4035020 con 0x7f99c006f670 2024-04-17T08:55:48.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.122+0000 7f99bf400640 1 -- 172.21.15.67:0/2456079575 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f99b4031810 con 0x7f99c006f670 2024-04-17T08:55:48.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.122+0000 7f99bf400640 1 -- 172.21.15.67:0/2456079575 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4031c00 con 0x7f99c006f670 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 -- 172.21.15.67:0/2456079575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 msgr2=0x7f99c006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 --2- 172.21.15.67:0/2456079575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c006fa50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f99b40097a0 tx=0x7f99b402b8b0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 -- 172.21.15.67:0/2456079575 shutdown_connections 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 --2- 172.21.15.67:0/2456079575 >> v2:172.21.15.187:3300/0 conn(0x7f99c006f030 0x7f99c00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 --2- 172.21.15.67:0/2456079575 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c006fa50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 -- 172.21.15.67:0/2456079575 >> 172.21.15.67:0/2456079575 conn(0x7f99c006bf50 msgr2=0x7f99c00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 -- 172.21.15.67:0/2456079575 shutdown_connections 2024-04-17T08:55:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.123+0000 7f99c7c00640 1 -- 172.21.15.67:0/2456079575 wait complete. 2024-04-17T08:55:48.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.125+0000 7f99c7c00640 1 Processor -- start 2024-04-17T08:55:48.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.125+0000 7f99c7c00640 1 -- start start 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c006f030 0x7f99c00aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c0154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00ce720 con 0x7f99c00cd020 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00cf680 con 0x7f99c006f030 2024-04-17T08:55:48.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.126+0000 7f99c7c00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f99c00ce8a0 con 0x7f99c006f670 2024-04-17T08:55:48.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49450/0 (socket says 172.21.15.67:49450) 2024-04-17T08:55:48.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 -- 172.21.15.67:0/661068531 learned_addr learned my addr 172.21.15.67:0/661068531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T08:55:48.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99bfe00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c0154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5200640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c006f030 0x7f99c00aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 -- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 msgr2=0x7f99c0154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c0154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 -- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c006f030 msgr2=0x7f99c00aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c006f030 0x7f99c00aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.127+0000 7f99c5c00640 1 -- 172.21.15.67:0/661068531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c00b3540 con 0x7f99c00cd020 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99c5c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f99b800efc0 tx=0x7f99b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b800cbe0 con 0x7f99c00cd020 2024-04-17T08:55:48.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f99b800cd80 con 0x7f99c00cd020 2024-04-17T08:55:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8003bf0 con 0x7f99c00cd020 2024-04-17T08:55:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c00b3820 con 0x7f99c00cd020 2024-04-17T08:55:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.128+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c00b3de0 con 0x7f99c00cd020 2024-04-17T08:55:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.130+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c0068b90 con 0x7f99c00cd020 2024-04-17T08:55:48.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.131+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 167) ==== 91142+0+0 (secure 0 0 0) 0x7f99b801b030 con 0x7f99c00cd020 2024-04-17T08:55:48.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.131+0000 7f99bcc00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 0x7f99a006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T08:55:48.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.131+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(295..295 src has 1..295) ==== 5788+0+0 (secure 0 0 0) 0x7f99b808c390 con 0x7f99c00cd020 2024-04-17T08:55:48.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.132+0000 7f99c5200640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 0x7f99a006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T08:55:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.134+0000 7f99c5200640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 0x7f99a006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99b4036000 tx=0x7f99b4008030 comp rx=0 tx=0).ready entity=mgr.10216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T08:55:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.136+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187257 (secure 0 0 0) 0x7f99b8022020 con 0x7f99c00cd020 2024-04-17T08:55:48.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.404+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99c015b9d0 con 0x7f99c00cd020 2024-04-17T08:55:48.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.405+0000 7f99bcc00640 1 -- 172.21.15.67:0/661068531 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) ==== 95+0+3876 (secure 0 0 0) 0x7f99b8010040 con 0x7f99c00cd020 2024-04-17T08:55:48.405 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T08:55:48.405 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":153,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6836/3168359127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":3168359127},{"type":"v1","addr":"172.21.15.187:6836","nonce":3168359127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":9044,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/142732569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6835","nonce":142732569},{"type":"v1","addr":"172.21.15.187:6837","nonce":142732569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":153},{"gid":13053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/229724785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":229724785},{"type":"v1","addr":"172.21.15.67:6834","nonce":229724785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146},{"gid":13056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1095802563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1095802563},{"type":"v1","addr":"172.21.15.67:6835","nonce":1095802563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[]} 2024-04-17T08:55:48.405 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 153 2024-04-17T08:55:48.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.408+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 msgr2=0x7f99a006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.408+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 0x7f99a006eed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99b4036000 tx=0x7f99b4008030 comp rx=0 tx=0).stop 2024-04-17T08:55:48.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.408+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 msgr2=0x7f99c0151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T08:55:48.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.408+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f99b800efc0 tx=0x7f99b800c490 comp rx=0 tx=0).stop 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 shutdown_connections 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:6832/3601707721,v1:172.21.15.187:6833/3601707721] conn(0x7f99a006ca10 0x7f99a006eed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99c00cd020 0x7f99c0151b00 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c006f670 0x7f99c0154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 --2- 172.21.15.67:0/661068531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c006f030 0x7f99c00aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 >> 172.21.15.67:0/661068531 conn(0x7f99c006bf50 msgr2=0x7f99c006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 shutdown_connections 2024-04-17T08:55:48.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T08:55:48.409+0000 7f99c7c00640 1 -- 172.21.15.67:0/661068531 wait complete. 2024-04-17T08:55:48.429 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-17T08:55:48.430 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-17T08:55:48.430 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T08:55:48.430 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.447 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T08:55:48.447 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T08:55:48.448 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.511 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T08:55:48.511 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T08:55:48.511 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.538 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T08:55:48.539 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T08:55:48.539 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.612 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T08:55:48.612 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-17T08:55:48.613 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-17T08:55:48.613 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.688 INFO:tasks.ceph.osd.0:Stopped 2024-04-17T08:55:48.688 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-17T08:55:48.688 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:48.757 INFO:tasks.ceph.osd.1:Stopped 2024-04-17T08:55:48.757 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-17T08:55:48.757 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:53.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-17T08:55:54.859 INFO:tasks.ceph.osd.2:Stopped 2024-04-17T08:55:54.860 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-17T08:55:54.860 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:54.928 INFO:tasks.ceph.osd.3:Stopped 2024-04-17T08:55:54.929 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-17T08:55:54.929 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:54.997 INFO:tasks.ceph.osd.4:Stopped 2024-04-17T08:55:54.997 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-17T08:55:54.997 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:55.064 INFO:tasks.ceph.osd.5:Stopped 2024-04-17T08:55:55.064 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-17T08:55:55.065 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:55.135 INFO:tasks.ceph.osd.6:Stopped 2024-04-17T08:55:55.135 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-17T08:55:55.135 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:55.203 INFO:tasks.ceph.osd.7:Stopped 2024-04-17T08:55:55.203 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-17T08:55:55.203 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-17T08:55:55.204 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:55:58.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-04-17T08:55:58.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-17T08:55:59.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:55:59.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:56:01.306 INFO:tasks.ceph.mgr.y:Stopped 2024-04-17T08:56:01.306 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-17T08:56:01.306 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:56:01.392 INFO:tasks.ceph.mgr.x:Stopped 2024-04-17T08:56:01.393 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-17T08:56:01.393 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-17T08:56:01.393 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:56:01.481 INFO:tasks.ceph.mon.a:Stopped 2024-04-17T08:56:01.481 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-17T08:56:01.481 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:56:01.546 INFO:tasks.ceph.mon.b:Stopped 2024-04-17T08:56:01.546 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-17T08:56:01.546 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T08:56:01.587 INFO:tasks.ceph.mon.c:Stopped 2024-04-17T08:56:01.587 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-17T08:56:01.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-04-17T08:56:01.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:56:01.712 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-17T08:56:02.117 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:56:02.117 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-17T08:56:02.290 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:56:02.290 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-17T08:56:02.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:56:02.448 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-17T08:56:02.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T08:56:02.631 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-17T08:56:02.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T08:56:02.945 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-17T08:56:03.124 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T08:56:03.124 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-17T08:56:03.262 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T08:56:03.263 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-17T08:56:03.444 INFO:tasks.ceph:Archiving mon data... 2024-04-17T08:56:03.446 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/data/mon.a.tgz 2024-04-17T08:56:03.446 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-17T08:56:03.506 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.8KTJ3fxK6j 2024-04-17T08:56:03.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8KTJ3fxK6j 2024-04-17T08:56:03.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-04-17T08:56:04.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.8KTJ3fxK6j 2024-04-17T08:56:04.433 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.8KTJ3fxK6j is 4MB 2024-04-17T08:56:04.628 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.8KTJ3fxK6j 2024-04-17T08:56:04.691 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/data/mon.b.tgz 2024-04-17T08:56:04.691 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-17T08:56:04.748 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.VFOho8A0vI 2024-04-17T08:56:04.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VFOho8A0vI 2024-04-17T08:56:05.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.VFOho8A0vI 2024-04-17T08:56:05.892 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.VFOho8A0vI is 4MB 2024-04-17T08:56:06.069 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.VFOho8A0vI 2024-04-17T08:56:06.129 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/data/mon.c.tgz 2024-04-17T08:56:06.129 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-17T08:56:06.180 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.50jKy5xYny 2024-04-17T08:56:06.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.50jKy5xYny 2024-04-17T08:56:06.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.50jKy5xYny 2024-04-17T08:56:07.055 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.50jKy5xYny is 4MB 2024-04-17T08:56:07.242 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.50jKy5xYny 2024-04-17T08:56:07.301 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-17T08:56:07.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-17T08:56:07.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-17T08:56:07.465 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-17T08:56:07.521 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-17T08:56:07.521 INFO:tasks.ceph:Archiving crash dumps... 2024-04-17T08:56:07.523 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi067/crash 2024-04-17T08:56:07.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-17T08:56:07.625 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi187/crash 2024-04-17T08:56:07.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-17T08:56:07.726 INFO:tasks.ceph:Shutting down logrotate 2024-04-17T08:56:07.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:56:07.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T08:56:07.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-17T08:56:07.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-17T08:56:08.073 INFO:tasks.ceph:Compressing logs... 2024-04-17T08:56:08.074 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-17T08:56:08.081 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-17T08:56:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72715.log 2024-04-17T08:56:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72122.log 2024-04-17T08:56:08.171 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-17T08:56:08.172 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37429.log 2024-04-17T08:56:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72715.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72715.log.gz 2024-04-17T08:56:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80931.log 2024-04-17T08:56:08.173 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101723.log 2024-04-17T08:56:08.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80931.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72122.log.gz 2024-04-17T08:56:08.175 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80931.log.gz 2024-04-17T08:56:08.175 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65416.log 2024-04-17T08:56:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-04-17T08:56:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37429.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-17T08:56:08.176 INFO:teuthology.orchestra.run.smithi187.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37429.log.gz 2024-04-17T08:56:08.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37229.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-17T08:56:08.176 INFO:teuthology.orchestra.run.smithi187.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-04-17T08:56:08.177 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-17T08:56:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59558.log 2024-04-17T08:56:08.178 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101723.log.gz 2024-04-17T08:56:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-04-17T08:56:08.179 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65416.log.gz 2024-04-17T08:56:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45826.log 2024-04-17T08:56:08.179 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59558.log.gz 2024-04-17T08:56:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106577.log 2024-04-17T08:56:08.181 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37269.log 2024-04-17T08:56:08.182 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42741.log 2024-04-17T08:56:08.182 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-04-17T08:56:08.182 INFO:teuthology.orchestra.run.smithi187.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37269.log.gz 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60123.log 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43430.log 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106577.log.gz 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66490.log 2024-04-17T08:56:08.183 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60123.log.gz 2024-04-17T08:56:08.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72631.log 2024-04-17T08:56:08.184 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43430.log.gz 2024-04-17T08:56:08.185 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.1.42741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43156.log 2024-04-17T08:56:08.185 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42741.log.gz 2024-04-17T08:56:08.186 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37509.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-17T08:56:08.186 INFO:teuthology.orchestra.run.smithi187.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-04-17T08:56:08.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66490.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66490.log.gz 2024-04-17T08:56:08.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46483.log 2024-04-17T08:56:08.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97830.log 2024-04-17T08:56:08.187 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72631.log.gz 2024-04-17T08:56:08.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46483.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46483.log.gz 2024-04-17T08:56:08.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72673.log 2024-04-17T08:56:08.189 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.3.43156.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-17T08:56:08.189 INFO:teuthology.orchestra.run.smithi187.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.43156.log.gz 2024-04-17T08:56:08.189 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37349.log 2024-04-17T08:56:08.189 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-17T08:56:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97830.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97830.log.gz 2024-04-17T08:56:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49612.log 2024-04-17T08:56:08.191 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42643.log 2024-04-17T08:56:08.191 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72673.log.gz 2024-04-17T08:56:08.191 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42981.log 2024-04-17T08:56:08.191 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49612.log.gz 2024-04-17T08:56:08.192 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37349.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-17T08:56:08.192 INFO:teuthology.orchestra.run.smithi187.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37349.log.gz 2024-04-17T08:56:08.192 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37389.log 2024-04-17T08:56:08.193 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51509.log 2024-04-17T08:56:08.193 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42643.log.gz 2024-04-17T08:56:08.193 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49553.log 2024-04-17T08:56:08.194 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42981.log.gz 2024-04-17T08:56:08.194 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51509.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51509.log.gz 2024-04-17T08:56:08.194 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43724.log 2024-04-17T08:56:08.196 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-04-17T08:56:08.196 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37389.log.gz 2024-04-17T08:56:08.196 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37309.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-04-17T08:56:08.196 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49553.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49553.log.gz 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82775.log 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62749.log 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43724.log.gz 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96359.log 2024-04-17T08:56:08.197 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82775.log.gz 2024-04-17T08:56:08.199 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93430.log 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62749.log.gz 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96359.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.96359.log.gz 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106012.log 2024-04-17T08:56:08.200 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93430.log.gz 2024-04-17T08:56:08.201 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-17T08:56:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68820.log 2024-04-17T08:56:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66532.log 2024-04-17T08:56:08.202 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106012.log.gz 2024-04-17T08:56:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103354.log 2024-04-17T08:56:08.203 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68820.log.gz 2024-04-17T08:56:08.204 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37469.log 2024-04-17T08:56:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66532.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66532.log.gz 2024-04-17T08:56:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76869.log 2024-04-17T08:56:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105325.log 2024-04-17T08:56:08.205 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103354.log.gz 2024-04-17T08:56:08.206 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-17T08:56:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97788.log 2024-04-17T08:56:08.206 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76869.log.gz 2024-04-17T08:56:08.207 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.37469.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37469.log.gz 2024-04-17T08:56:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77207.log 2024-04-17T08:56:08.208 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.105325.log.gz 2024-04-17T08:56:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96705.log 2024-04-17T08:56:08.209 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97788.log.gz 2024-04-17T08:56:08.209 INFO:teuthology.orchestra.run.smithi187.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-17T08:56:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.77207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.77207.log.gz 2024-04-17T08:56:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68526.log 2024-04-17T08:56:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70416.log 2024-04-17T08:56:08.211 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96705.log.gz 2024-04-17T08:56:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-04-17T08:56:08.212 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68526.log.gz 2024-04-17T08:56:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.70416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70416.log.gz 2024-04-17T08:56:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85292.log 2024-04-17T08:56:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81372.log 2024-04-17T08:56:08.214 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-04-17T08:56:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106138.log 2024-04-17T08:56:08.216 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85292.log.gz 2024-04-17T08:56:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62665.log 2024-04-17T08:56:08.218 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81372.log.gz 2024-04-17T08:56:08.218 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mon.c.log: 94.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-17T08:56:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65101.log 2024-04-17T08:56:08.219 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106138.log.gz 2024-04-17T08:56:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84282.log 2024-04-17T08:56:08.219 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62665.log.gz 2024-04-17T08:56:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102122.log 2024-04-17T08:56:08.221 INFO:teuthology.orchestra.run.smithi067.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.65101.log.gz 2024-04-17T08:56:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89077.log 2024-04-17T08:56:08.222 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84282.log.gz 2024-04-17T08:56:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76222.log 2024-04-17T08:56:08.223 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102122.log.gz 2024-04-17T08:56:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84823.log 2024-04-17T08:56:08.224 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89077.log.gz 2024-04-17T08:56:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76222.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76222.log.gz 2024-04-17T08:56:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56693.log 2024-04-17T08:56:08.227 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63488.log 2024-04-17T08:56:08.227 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84823.log.gz 2024-04-17T08:56:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43556.log 2024-04-17T08:56:08.228 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56693.log.gz 2024-04-17T08:56:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102248.log 2024-04-17T08:56:08.230 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63488.log.gz 2024-04-17T08:56:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2024-04-17T08:56:08.231 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43556.log.gz 2024-04-17T08:56:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46957.log 2024-04-17T08:56:08.232 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102248.log.gz 2024-04-17T08:56:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57111.log 2024-04-17T08:56:08.233 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2024-04-17T08:56:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63001.log 2024-04-17T08:56:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57111.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46957.log.gz 2024-04-17T08:56:08.235 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57111.log.gz 2024-04-17T08:56:08.235 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109941.log 2024-04-17T08:56:08.236 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61809.log 2024-04-17T08:56:08.236 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63001.log.gz 2024-04-17T08:56:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109941.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109941.log.gz 2024-04-17T08:56:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47724.log 2024-04-17T08:56:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71462.log 2024-04-17T08:56:08.239 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47724.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61809.log.gz 2024-04-17T08:56:08.240 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47724.log.gz 2024-04-17T08:56:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87521.log 2024-04-17T08:56:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-04-17T08:56:08.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71462.log.gz 2024-04-17T08:56:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99404.log 2024-04-17T08:56:08.242 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87521.log.gz 2024-04-17T08:56:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72549.log 2024-04-17T08:56:08.243 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-04-17T08:56:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65584.log 2024-04-17T08:56:08.244 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99404.log.gz 2024-04-17T08:56:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46897.log 2024-04-17T08:56:08.245 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72549.log.gz 2024-04-17T08:56:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59600.log 2024-04-17T08:56:08.247 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65584.log.gz 2024-04-17T08:56:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46897.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.46897.log.gz -- 2024-04-17T08:56:08.248 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.0.57031.log 2024-04-17T08:56:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75587.log 2024-04-17T08:56:08.250 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59600.log.gz 2024-04-17T08:56:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.57031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.57031.log.gz 2024-04-17T08:56:08.251 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107528.log 2024-04-17T08:56:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102855.log 2024-04-17T08:56:08.252 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75587.log.gz 2024-04-17T08:56:08.253 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46525.log 2024-04-17T08:56:08.253 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107528.log.gz 2024-04-17T08:56:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88573.log 2024-04-17T08:56:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46525.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102855.log.gz 2024-04-17T08:56:08.255 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46525.log.gz 2024-04-17T08:56:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74052.log 2024-04-17T08:56:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50386.log 2024-04-17T08:56:08.257 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88573.log.gz 2024-04-17T08:56:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69780.log 2024-04-17T08:56:08.258 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74052.log.gz 2024-04-17T08:56:08.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75214.log 2024-04-17T08:56:08.259 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50386.log.gz 2024-04-17T08:56:08.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69780.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.69780.log.gz -5 2024-04-17T08:56:08.260 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51070.log 2024-04-17T08:56:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75466.log 2024-04-17T08:56:08.262 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75214.log.gz 2024-04-17T08:56:08.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74654.log 2024-04-17T08:56:08.263 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51070.log.gz 2024-04-17T08:56:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68442.log 2024-04-17T08:56:08.264 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75466.log.gz 2024-04-17T08:56:08.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56525.log 2024-04-17T08:56:08.265 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74654.log.gz 2024-04-17T08:56:08.266 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73254.log 2024-04-17T08:56:08.266 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68442.log.gz 2024-04-17T08:56:08.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63043.log 2024-04-17T08:56:08.267 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56525.log.gz 2024-04-17T08:56:08.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108625.log 2024-04-17T08:56:08.269 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73254.log.gz 2024-04-17T08:56:08.269 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63043.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63043.log.gz 2024-04-17T08:56:08.269 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.43598.log 2024-04-17T08:56:08.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74229.log 2024-04-17T08:56:08.271 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108625.log.gz 2024-04-17T08:56:08.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53425.log 2024-04-17T08:56:08.272 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43598.log.gz 2024-04-17T08:56:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93826.log 2024-04-17T08:56:08.273 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.74229.log.gz 2024-04-17T08:56:08.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53425.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53425.log.gz 2024-04-17T08:56:08.274 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88909.log 2024-04-17T08:56:08.275 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70645.log 2024-04-17T08:56:08.275 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93826.log.gz 2024-04-17T08:56:08.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46393.log 2024-04-17T08:56:08.276 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88909.log.gz 2024-04-17T08:56:08.278 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109570.log 2024-04-17T08:56:08.278 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70645.log.gz 2024-04-17T08:56:08.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110717.log 2024-04-17T08:56:08.279 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46393.log.gz 2024-04-17T08:56:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94079.log 2024-04-17T08:56:08.280 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109570.log.gz 2024-04-17T08:56:08.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86509.log 2024-04-17T08:56:08.282 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110717.log.gz 2024-04-17T08:56:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94079.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94079.log.gz 2024-04-17T08:56:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73026.log 2024-04-17T08:56:08.284 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62539.log 2024-04-17T08:56:08.284 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86509.log.gz 2024-04-17T08:56:08.285 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109444.log 2024-04-17T08:56:08.285 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73026.log.gz 2024-04-17T08:56:08.286 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60673.log 2024-04-17T08:56:08.286 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62539.log.gz 2024-04-17T08:56:08.287 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109444.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.109444.log.gz 2024-04-17T08:56:08.287 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69251.log 2024-04-17T08:56:08.288 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43262.log 2024-04-17T08:56:08.288 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60673.log.gz 2024-04-17T08:56:08.289 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102771.log 2024-04-17T08:56:08.290 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69251.log.gz 2024-04-17T08:56:08.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43262.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43262.log.gz 2024-04-17T08:56:08.292 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2024-04-17T08:56:08.292 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97704.log 2024-04-17T08:56:08.292 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102771.log.gz 2024-04-17T08:56:08.293 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76478.log 2024-04-17T08:56:08.293 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2024-04-17T08:56:08.294 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75881.log 2024-04-17T08:56:08.295 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97704.log.gz 2024-04-17T08:56:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76478.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.47293.log 2024-04-17T08:56:08.296 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76478.log.gz 2024-04-17T08:56:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88292.log 2024-04-17T08:56:08.297 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75881.log.gz 2024-04-17T08:56:08.298 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42980.log 2024-04-17T08:56:08.298 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47293.log.gz 2024-04-17T08:56:08.299 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72038.log 2024-04-17T08:56:08.299 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88292.log.gz 2024-04-17T08:56:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42980.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42980.log.gz 2024-04-17T08:56:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100965.log 2024-04-17T08:56:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105094.log 2024-04-17T08:56:08.301 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72038.log.gz 2024-04-17T08:56:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93640.log 2024-04-17T08:56:08.303 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100965.log.gz 2024-04-17T08:56:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-04-17T08:56:08.304 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105094.log.gz 2024-04-17T08:56:08.305 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63614.log 2024-04-17T08:56:08.305 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93640.log.gz 2024-04-17T08:56:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102374.log 2024-04-17T08:56:08.306 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-04-17T08:56:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73296.log 2024-04-17T08:56:08.307 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63614.log.gz 2024-04-17T08:56:08.308 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103186.log 2024-04-17T08:56:08.308 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102374.log.gz 2024-04-17T08:56:08.309 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-04-17T08:56:08.310 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73296.log.gz 2024-04-17T08:56:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81684.log 2024-04-17T08:56:08.311 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103186.log.gz 2024-04-17T08:56:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60291.log 2024-04-17T08:56:08.312 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-04-17T08:56:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61434.log 2024-04-17T08:56:08.313 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81684.log.gz 2024-04-17T08:56:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110278.log 2024-04-17T08:56:08.315 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gz 2024-04-17T08:56:08.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104170.log 2024-04-17T08:56:08.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61434.log.gz 2024-04-17T08:56:08.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106451.log 2024-04-17T08:56:08.317 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110278.log.gz 2024-04-17T08:56:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47167.log 2024-04-17T08:56:08.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104170.log.gz 2024-04-17T08:56:08.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46999.log 2024-04-17T08:56:08.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106451.log.gz 2024-04-17T08:56:08.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69557.log 2024-04-17T08:56:08.321 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47167.log.gz 2024-04-17T08:56:08.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74170.log 2024-04-17T08:56:08.323 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46999.log.gz 2024-04-17T08:56:08.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69557.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69557.log.gz 2024-04-17T08:56:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67040.log 2024-04-17T08:56:08.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74170.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74170.log.gz 2024-04-17T08:56:08.325 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82404.log 2024-04-17T08:56:08.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70228.log 2024-04-17T08:56:08.326 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67040.log.gz 2024-04-17T08:56:08.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-04-17T08:56:08.327 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82404.log.gz 2024-04-17T08:56:08.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73086.log 2024-04-17T08:56:08.329 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70228.log.gz 2024-04-17T08:56:08.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.90404.log 2024-04-17T08:56:08.330 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-04-17T08:56:08.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101091.log 2024-04-17T08:56:08.331 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73086.log.gz 2024-04-17T08:56:08.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.90404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.90404.log.gz 2024-04-17T08:56:08.332 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81193.log 2024-04-17T08:56:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47861.log 2024-04-17T08:56:08.333 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101091.log.gz 2024-04-17T08:56:08.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74773.log 2024-04-17T08:56:08.334 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81193.log.gz 2024-04-17T08:56:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47861.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.56567.log -- replaced with /var/log/ceph/ceph-client.admin.47861.log.gz 2024-04-17T08:56:08.335 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:08.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103905.log 2024-04-17T08:56:08.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74773.log.gz 2024-04-17T08:56:08.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64219.log 2024-04-17T08:56:08.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56567.log.gz 2024-04-17T08:56:08.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41557.log 2024-04-17T08:56:08.339 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103905.log.gz 2024-04-17T08:56:08.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86704.log 2024-04-17T08:56:08.340 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64219.log.gz 2024-04-17T08:56:08.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60165.log 2024-04-17T08:56:08.341 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41557.log.gz 2024-04-17T08:56:08.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71368.log 2024-04-17T08:56:08.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86704.log.gz 2024-04-17T08:56:08.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44293.log 2024-04-17T08:56:08.344 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60165.log.gz 2024-04-17T08:56:08.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71368.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71368.log.gz 2024-04-17T08:56:08.345 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54996.log 2024-04-17T08:56:08.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.44293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44293.log.gz 2024-04-17T08:56:08.346 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71725.log 2024-04-17T08:56:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103041.log 2024-04-17T08:56:08.348 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54996.log.gz 2024-04-17T08:56:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72164.log 2024-04-17T08:56:08.349 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71725.log.gz 2024-04-17T08:56:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103041.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103041.log.gz 2024-04-17T08:56:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97220.log 2024-04-17T08:56:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49494.log 2024-04-17T08:56:08.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72164.log.gz 2024-04-17T08:56:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-04-17T08:56:08.353 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97220.log.gz 2024-04-17T08:56:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49494.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49494.log.gz 2024-04-17T08:56:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-04-17T08:56:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105970.log 2024-04-17T08:56:08.355 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-04-17T08:56:08.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106876.log 2024-04-17T08:56:08.356 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-04-17T08:56:08.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105970.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105970.log.gz 2024-04-17T08:56:08.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109108.log 2024-04-17T08:56:08.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86425.log 2024-04-17T08:56:08.359 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106876.log.gz 2024-04-17T08:56:08.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68001.log 2024-04-17T08:56:08.360 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109108.log.gz 2024-04-17T08:56:08.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89256.log 2024-04-17T08:56:08.361 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86425.log.gz 2024-04-17T08:56:08.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89631.log 2024-04-17T08:56:08.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68001.log.gz 2024-04-17T08:56:08.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69114.log 2024-04-17T08:56:08.364 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89256.log.gz 2024-04-17T08:56:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45658.log 2024-04-17T08:56:08.365 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89631.log.gz 2024-04-17T08:56:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94037.log 2024-04-17T08:56:08.366 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69114.log.gz 2024-04-17T08:56:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47522.log 2024-04-17T08:56:08.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45658.log.gz 2024-04-17T08:56:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102603.log 2024-04-17T08:56:08.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94037.log.gz 2024-04-17T08:56:08.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73864.log 2024-04-17T08:56:08.370 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47522.log.gz 2024-04-17T08:56:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104885.log 2024-04-17T08:56:08.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102603.log.gz 2024-04-17T08:56:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74371.log 2024-04-17T08:56:08.372 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73864.log.gz 2024-04-17T08:56:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88615.log 2024-04-17T08:56:08.373 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104885.log.gz 2024-04-17T08:56:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90764.log 2024-04-17T08:56:08.375 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74371.log.gz 2024-04-17T08:56:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88615.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88615.log.gzgzip 2024-04-17T08:56:08.378 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69599.log 2024-04-17T08:56:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90764.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90764.log.gz 2024-04-17T08:56:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48255.log 2024-04-17T08:56:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59200.log 2024-04-17T08:56:08.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69599.log.gz 2024-04-17T08:56:08.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61851.log 2024-04-17T08:56:08.379 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48255.log.gz 2024-04-17T08:56:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69868.log 2024-04-17T08:56:08.380 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59200.log.gz 2024-04-17T08:56:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61851.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61851.log.gz 2024-04-17T08:56:08.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99532.log 2024-04-17T08:56:08.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109024.log 2024-04-17T08:56:08.382 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69868.log.gz 2024-04-17T08:56:08.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48553.log 2024-04-17T08:56:08.384 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99532.log.gz 2024-04-17T08:56:08.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109024.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109024.log.gz 2024-04-17T08:56:08.385 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54954.log 2024-04-17T08:56:08.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95253.log 2024-04-17T08:56:08.386 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48553.log.gz 2024-04-17T08:56:08.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81811.log 2024-04-17T08:56:08.388 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54954.log.gz 2024-04-17T08:56:08.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108887.log 2024-04-17T08:56:08.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81811.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.95253.log.gz 2024-04-17T08:56:08.395 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75797.log 2024-04-17T08:56:08.395 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81811.log.gz 2024-04-17T08:56:08.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103821.log 2024-04-17T08:56:08.396 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108887.log.gz 2024-04-17T08:56:08.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59463.log 2024-04-17T08:56:08.396 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75797.log.gz 2024-04-17T08:56:08.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106054.log 2024-04-17T08:56:08.397 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103821.log.gz 2024-04-17T08:56:08.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105571.log 2024-04-17T08:56:08.397 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59463.log.gz 2024-04-17T08:56:08.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63921.log 2024-04-17T08:56:08.398 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106054.log.gz 2024-04-17T08:56:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100468.log 2024-04-17T08:56:08.403 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105571.log.gz 2024-04-17T08:56:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63921.log.gz 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57402.log 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100468.log.gz 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88867.log 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-04-17T08:56:08.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72332.log 2024-04-17T08:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57402.log.gz 2024-04-17T08:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76701.log 2024-04-17T08:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88867.log.gz 2024-04-17T08:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100117.log 2024-04-17T08:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72332.log.gz 2024-04-17T08:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76701.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.76701.log.gz 2024-04-17T08:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68610.log 2024-04-17T08:56:08.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66872.log 2024-04-17T08:56:08.412 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.100117.log.gz 2024-04-17T08:56:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45574.log 2024-04-17T08:56:08.413 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68610.log.gz 2024-04-17T08:56:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66872.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.66872.log.gz /var/log/ceph/ceph-client.admin.69826.log 2024-04-17T08:56:08.413 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:08.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69994.log 2024-04-17T08:56:08.414 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45574.log.gz 2024-04-17T08:56:08.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93953.log 2024-04-17T08:56:08.414 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69826.log.gz 2024-04-17T08:56:08.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73806.log 2024-04-17T08:56:08.415 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69994.log.gz 2024-04-17T08:56:08.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62455.log 2024-04-17T08:56:08.415 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93953.log.gz 2024-04-17T08:56:08.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86346.log 2024-04-17T08:56:08.416 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73806.log.gz 2024-04-17T08:56:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107216.log 2024-04-17T08:56:08.422 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62455.log.gz 2024-04-17T08:56:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.86346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86346.log.gz 2024-04-17T08:56:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97914.log 2024-04-17T08:56:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43178.log 2024-04-17T08:56:08.423 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107216.log.gz 2024-04-17T08:56:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74412.log 2024-04-17T08:56:08.424 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97914.log.gz 2024-04-17T08:56:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99158.log 2024-04-17T08:56:08.424 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43178.log.gz 2024-04-17T08:56:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71200.log 2024-04-17T08:56:08.425 INFO:teuthology.orchestra.run.smithi067.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.74412.log.gz 2024-04-17T08:56:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95001.log 2024-04-17T08:56:08.425 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99158.log.gz 2024-04-17T08:56:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75077.log 2024-04-17T08:56:08.426 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71200.log.gz 2024-04-17T08:56:08.426 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95001.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95001.log.gz 2024-04-17T08:56:08.426 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61074.log 2024-04-17T08:56:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59081.log 2024-04-17T08:56:08.432 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75077.log.gz 2024-04-17T08:56:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55579.log 88.4% 2024-04-17T08:56:08.432 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61074.log.gz 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43304.log 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59081.log.gz 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85672.log 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55579.log.gz 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72080.log 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43304.log.gz 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100680.log 2024-04-17T08:56:08.433 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85672.log.gz 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72080.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.72080.log.gz /var/log/ceph/ceph-client.admin.69072.log 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100680.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100680.log.gz 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90024.log 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69072.log.gz 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97304.log 2024-04-17T08:56:08.434 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90024.log.gz 2024-04-17T08:56:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-04-17T08:56:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72248.log 2024-04-17T08:56:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61743.log 2024-04-17T08:56:08.437 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97304.log.gz 2024-04-17T08:56:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72248.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30891.log 2024-04-17T08:56:08.437 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72248.log.gz 2024-04-17T08:56:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50902.log 88.1% 2024-04-17T08:56:08.438 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61743.log.gz 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.30891.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30891.log.gz 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46063.log 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98227.log 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50902.log.gz 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47041.log 2024-04-17T08:56:08.440 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46063.log.gz 2024-04-17T08:56:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98227.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.98227.log.gz 2024-04-17T08:56:08.441 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106367.log 2024-04-17T08:56:08.442 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37286.log 2024-04-17T08:56:08.442 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47041.log.gz 2024-04-17T08:56:08.443 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106283.log 2024-04-17T08:56:08.443 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106367.log.gz 2024-04-17T08:56:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108137.log 2024-04-17T08:56:08.444 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37286.log.gz 2024-04-17T08:56:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87988.log 2024-04-17T08:56:08.445 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106283.log.gz 2024-04-17T08:56:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108137.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108137.log.gz 2024-04-17T08:56:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93377.log 2024-04-17T08:56:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87988.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87988.log.gzgzip 2024-04-17T08:56:08.447 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.94163.log 2024-04-17T08:56:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93377.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.93377.log.gz 2024-04-17T08:56:08.448 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.0.73634.log 2024-04-17T08:56:08.449 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107678.log 2024-04-17T08:56:08.449 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.94163.log.gz 2024-04-17T08:56:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.73634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73634.log.gz 2024-04-17T08:56:08.450 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98269.log 2024-04-17T08:56:08.452 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107678.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107678.log.gz 2024-04-17T08:56:08.452 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64975.log 2024-04-17T08:56:08.452 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75172.log 2024-04-17T08:56:08.453 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98269.log.gz 2024-04-17T08:56:08.453 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76140.log 2024-04-17T08:56:08.454 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64975.log.gz 2024-04-17T08:56:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75172.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.75172.log.gz 2024-04-17T08:56:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95838.log 2024-04-17T08:56:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103863.log 2024-04-17T08:56:08.456 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76140.log.gz 2024-04-17T08:56:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97746.log 2024-04-17T08:56:08.457 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95838.log.gz 2024-04-17T08:56:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103863.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103863.log.gz 2024-04-17T08:56:08.458 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.99883.log 2024-04-17T08:56:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96957.log 2024-04-17T08:56:08.459 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97746.log.gz 2024-04-17T08:56:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82985.log 2024-04-17T08:56:08.460 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99883.log.gz 2024-04-17T08:56:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96957.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.96957.log.gz 2024-04-17T08:56:08.461 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103083.log 2024-04-17T08:56:08.462 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81540.log 2024-04-17T08:56:08.462 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82985.log.gz 2024-04-17T08:56:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65332.log 2024-04-17T08:56:08.463 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.103083.log.gz 2024-04-17T08:56:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46651.log 2024-04-17T08:56:08.464 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81540.log.gz 2024-04-17T08:56:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65332.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65332.log.gz 2024-04-17T08:56:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70495.log 2024-04-17T08:56:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54723.log 2024-04-17T08:56:08.466 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46651.log.gz 2024-04-17T08:56:08.467 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66704.log 2024-04-17T08:56:08.467 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70495.log.gz 2024-04-17T08:56:08.468 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89161.log 2024-04-17T08:56:08.469 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54723.log.gz 2024-04-17T08:56:08.469 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66704.log: gzip 89.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.66704.log.gz -- 2024-04-17T08:56:08.469 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.77286.log 2024-04-17T08:56:08.470 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110362.log 2024-04-17T08:56:08.471 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89161.log.gz 2024-04-17T08:56:08.471 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69952.log 2024-04-17T08:56:08.472 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77286.log.gz 2024-04-17T08:56:08.473 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84324.log 2024-04-17T08:56:08.473 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110362.log.gz 2024-04-17T08:56:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94690.log 2024-04-17T08:56:08.474 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69952.log.gz 2024-04-17T08:56:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87154.log 2024-04-17T08:56:08.475 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.84324.log.gz 2024-04-17T08:56:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98396.log 2024-04-17T08:56:08.476 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94690.log.gz 2024-04-17T08:56:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97136.log 2024-04-17T08:56:08.477 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87154.log.gz 2024-04-17T08:56:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103522.log 2024-04-17T08:56:08.478 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98396.log.gz 2024-04-17T08:56:08.479 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97262.log 2024-04-17T08:56:08.479 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97136.log.gz 2024-04-17T08:56:08.480 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63698.log 2024-04-17T08:56:08.480 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103522.log.gz 2024-04-17T08:56:08.481 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97262.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97262.log.gz 2024-04-17T08:56:08.481 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-04-17T08:56:08.482 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51450.log 2024-04-17T08:56:08.482 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63698.log.gz 2024-04-17T08:56:08.483 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63085.log 2024-04-17T08:56:08.483 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-04-17T08:56:08.484 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77328.log 2024-04-17T08:56:08.484 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51450.log.gz 2024-04-17T08:56:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63085.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63085.log.gz 2024-04-17T08:56:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109486.log 2024-04-17T08:56:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60800.log 2024-04-17T08:56:08.487 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77328.log.gz 2024-04-17T08:56:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41609.log 2024-04-17T08:56:08.488 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109486.log.gz 2024-04-17T08:56:08.489 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60550.log 2024-04-17T08:56:08.489 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60800.log.gz 2024-04-17T08:56:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105802.log 2024-04-17T08:56:08.490 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41609.log.gz 2024-04-17T08:56:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85033.log 2024-04-17T08:56:08.491 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60550.log.gz 2024-04-17T08:56:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88657.log 2024-04-17T08:56:08.492 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105802.log.gz 2024-04-17T08:56:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54597.log 2024-04-17T08:56:08.493 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85033.log.gz 2024-04-17T08:56:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104426.log 2024-04-17T08:56:08.494 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88657.log.gz 2024-04-17T08:56:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54639.log 2024-04-17T08:56:08.495 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54597.log.gz 2024-04-17T08:56:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73993.log 2024-04-17T08:56:08.497 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104426.log.gz 2024-04-17T08:56:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81727.log 2024-04-17T08:56:08.498 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54639.log.gz 2024-04-17T08:56:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45868.log 2024-04-17T08:56:08.499 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73993.log.gz 2024-04-17T08:56:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107444.log 2024-04-17T08:56:08.500 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81727.log.gz 2024-04-17T08:56:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97872.log 2024-04-17T08:56:08.501 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45868.log.gz 2024-04-17T08:56:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107444.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107444.log.gz 2024-04-17T08:56:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98811.log 2024-04-17T08:56:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2024-04-17T08:56:08.503 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97872.log.gz 2024-04-17T08:56:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59684.log 2024-04-17T08:56:08.504 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98811.log.gz 2024-04-17T08:56:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84618.log 2024-04-17T08:56:08.505 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2024-04-17T08:56:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59684.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59684.log.gz 2024-04-17T08:56:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-04-17T08:56:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84231.log 2024-04-17T08:56:08.507 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84618.log.gz 2024-04-17T08:56:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51028.log 2024-04-17T08:56:08.509 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-04-17T08:56:08.509 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84231.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.84231.log.gz 2024-04-17T08:56:08.509 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2024-04-17T08:56:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106493.log 2024-04-17T08:56:08.511 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51028.log.gz 2024-04-17T08:56:08.511 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95796.log 2024-04-17T08:56:08.512 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2024-04-17T08:56:08.513 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108667.log 2024-04-17T08:56:08.513 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106493.log.gz 2024-04-17T08:56:08.514 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2024-04-17T08:56:08.514 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95796.log.gz 2024-04-17T08:56:08.515 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41792.log 2024-04-17T08:56:08.515 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108667.log.gz 2024-04-17T08:56:08.516 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69515.log 2024-04-17T08:56:08.516 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2024-04-17T08:56:08.517 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105754.log 2024-04-17T08:56:08.517 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41792.log.gz 2024-04-17T08:56:08.518 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56302.log 2024-04-17T08:56:08.518 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69515.log.gz 2024-04-17T08:56:08.519 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104468.log 2024-04-17T08:56:08.519 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105754.log.gz 2024-04-17T08:56:08.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98480.log 2024-04-17T08:56:08.521 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56302.log.gz 2024-04-17T08:56:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45700.log 2024-04-17T08:56:08.522 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104468.log.gz 2024-04-17T08:56:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93157.log 2024-04-17T08:56:08.523 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98480.log.gz 2024-04-17T08:56:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61979.log 2024-04-17T08:56:08.524 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2024-04-17T08:56:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107174.log 2024-04-17T08:56:08.525 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93157.log.gz 2024-04-17T08:56:08.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89119.log 2024-04-17T08:56:08.526 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61979.log.gz 2024-04-17T08:56:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90806.log 2024-04-17T08:56:08.527 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107174.log.gz 2024-04-17T08:56:08.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68169.log 2024-04-17T08:56:08.528 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89119.log.gz 2024-04-17T08:56:08.529 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61015.log 2024-04-17T08:56:08.529 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90806.log.gz 2024-04-17T08:56:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70758.log 2024-04-17T08:56:08.531 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68169.log.gz 2024-04-17T08:56:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55453.log 2024-04-17T08:56:08.532 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61015.log.gz 2024-04-17T08:56:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70758.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.70758.log.gz 2024-04-17T08:56:08.533 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98143.log 2024-04-17T08:56:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100775.log 2024-04-17T08:56:08.534 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55453.log.gz 2024-04-17T08:56:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95754.log 2024-04-17T08:56:08.535 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98143.log.gz 2024-04-17T08:56:08.536 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99627.log 2024-04-17T08:56:08.537 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100775.log.gz 2024-04-17T08:56:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95754.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.95754.log.gz 2024-04-17T08:56:08.537 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81769.log 2024-04-17T08:56:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76434.log 2024-04-17T08:56:08.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81769.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99627.log.gz 2024-04-17T08:56:08.540 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81769.log.gz 2024-04-17T08:56:08.540 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109360.log 2024-04-17T08:56:08.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43388.log 2024-04-17T08:56:08.541 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76434.log.gz 2024-04-17T08:56:08.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109360.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.109360.log.gz 2024-04-17T08:56:08.542 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66998.log 2024-04-17T08:56:08.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71284.log 2024-04-17T08:56:08.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66998.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43388.log.gz 2024-04-17T08:56:08.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66998.log.gz 2024-04-17T08:56:08.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106794.log 2024-04-17T08:56:08.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89856.log 2024-04-17T08:56:08.546 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71284.log.gz 2024-04-17T08:56:08.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98058.log 2024-04-17T08:56:08.547 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106794.log.gz 2024-04-17T08:56:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99488.log 2024-04-17T08:56:08.548 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89856.log.gz 2024-04-17T08:56:08.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69331.log 2024-04-17T08:56:08.549 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98058.log.gz 2024-04-17T08:56:08.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99488.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99488.log.gz 2024-04-17T08:56:08.550 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110446.log 2024-04-17T08:56:08.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-04-17T08:56:08.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110446.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69331.log.gz 2024-04-17T08:56:08.553 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110446.log.gz 2024-04-17T08:56:08.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96275.log 2024-04-17T08:56:08.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85798.log 2024-04-17T08:56:08.554 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-04-17T08:56:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86008.log 2024-04-17T08:56:08.554 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96275.log.gz 2024-04-17T08:56:08.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60888.log 2024-04-17T08:56:08.555 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85798.log.gz 2024-04-17T08:56:08.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64177.log 2024-04-17T08:56:08.556 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86008.log.gz 2024-04-17T08:56:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106918.log 2024-04-17T08:56:08.557 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60888.log.gz 2024-04-17T08:56:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109066.log 2024-04-17T08:56:08.570 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64177.log.gz 2024-04-17T08:56:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100594.log 2024-04-17T08:56:08.571 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106918.log.gz 2024-04-17T08:56:08.571 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98685.log 2024-04-17T08:56:08.571 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109066.log.gz 2024-04-17T08:56:08.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104128.log 2024-04-17T08:56:08.572 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.100594.log.gz 2024-04-17T08:56:08.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84051.log 2024-04-17T08:56:08.572 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98685.log.gz 2024-04-17T08:56:08.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56214.log 2024-04-17T08:56:08.573 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.104128.log.gz 2024-04-17T08:56:08.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63837.log 2024-04-17T08:56:08.573 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84051.log.gz 2024-04-17T08:56:08.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98522.log 2024-04-17T08:56:08.574 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56214.log.gz 2024-04-17T08:56:08.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63837.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63837.log.gz 2024-04-17T08:56:08.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96149.log 2024-04-17T08:56:08.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99319.log 2024-04-17T08:56:08.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96149.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98522.log.gz 2024-04-17T08:56:08.575 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96149.log.gz 2024-04-17T08:56:08.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102164.log 2024-04-17T08:56:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106198.log 2024-04-17T08:56:08.580 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99319.log.gz 2024-04-17T08:56:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62875.log 2024-04-17T08:56:08.580 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102164.log.gz 2024-04-17T08:56:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106198.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106198.log.gz 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71081.log 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69431.log 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62875.log.gz 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69156.log 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71081.log.gz 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72980.log 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69431.log.gz 2024-04-17T08:56:08.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107088.log 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69156.log.gz 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42307.log 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72980.log.gz 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82362.log 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107088.log.gz 2024-04-17T08:56:08.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53951.log 2024-04-17T08:56:08.584 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2024-04-17T08:56:08.584 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55747.log 2024-04-17T08:56:08.584 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82362.log.gz 2024-04-17T08:56:08.584 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96107.log 2024-04-17T08:56:08.585 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53951.log.gz 2024-04-17T08:56:08.585 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96425.log 2024-04-17T08:56:08.585 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55747.log.gz 2024-04-17T08:56:08.586 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84366.log 2024-04-17T08:56:08.586 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96107.log.gz 2024-04-17T08:56:08.587 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49789.log 2024-04-17T08:56:08.587 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96425.log.gz 2024-04-17T08:56:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62497.log 2024-04-17T08:56:08.588 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84366.log.gz 2024-04-17T08:56:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102416.log 2024-04-17T08:56:08.588 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49789.log.gz 2024-04-17T08:56:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89675.log 2024-04-17T08:56:08.589 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62497.log.gz 2024-04-17T08:56:08.590 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104618.log 2024-04-17T08:56:08.590 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102416.log.gz 2024-04-17T08:56:08.591 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86651.log 2024-04-17T08:56:08.591 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89675.log.gz 2024-04-17T08:56:08.592 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42974.log 2024-04-17T08:56:08.592 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104618.log.gz 2024-04-17T08:56:08.593 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50092.log 2024-04-17T08:56:08.594 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86651.log.gz 2024-04-17T08:56:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71893.log 2024-04-17T08:56:08.595 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42974.log.gz 2024-04-17T08:56:08.595 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46771.log 2024-04-17T08:56:08.596 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50092.log.gz 2024-04-17T08:56:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52635.log 2024-04-17T08:56:08.597 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71893.log.gz 2024-04-17T08:56:08.598 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64594.log 2024-04-17T08:56:08.598 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46771.log.gz 2024-04-17T08:56:08.599 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110757.log 2024-04-17T08:56:08.599 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52635.log.gz 2024-04-17T08:56:08.600 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56437.log 2024-04-17T08:56:08.600 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64594.log.gz 2024-04-17T08:56:08.601 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85966.log 2024-04-17T08:56:08.601 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110757.log.gz 2024-04-17T08:56:08.602 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89460.log 2024-04-17T08:56:08.602 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56437.log.gz 2024-04-17T08:56:08.603 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96509.log 2024-04-17T08:56:08.603 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85966.log.gz 2024-04-17T08:56:08.604 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102038.log 2024-04-17T08:56:08.604 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89460.log.gz 2024-04-17T08:56:08.606 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85542.log 2024-04-17T08:56:08.606 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96509.log.gz 2024-04-17T08:56:08.606 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66574.log 2024-04-17T08:56:08.607 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102038.log.gz 2024-04-17T08:56:08.608 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66405.log 2024-04-17T08:56:08.608 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85542.log.gz 2024-04-17T08:56:08.609 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66574.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.42982.log 2024-04-17T08:56:08.609 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66574.log.gz 2024-04-17T08:56:08.610 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96747.log 2024-04-17T08:56:08.611 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66405.log.gz 2024-04-17T08:56:08.611 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36918.log 2024-04-17T08:56:08.611 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42982.log.gz 2024-04-17T08:56:08.612 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49966.log 2024-04-17T08:56:08.612 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.96747.log.gz 2024-04-17T08:56:08.613 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36918.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36918.log.gz 2024-04-17T08:56:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43346.log 2024-04-17T08:56:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76390.log 2024-04-17T08:56:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43346.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49966.log.gz 2024-04-17T08:56:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72469.log 2024-04-17T08:56:08.615 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43346.log.gz 2024-04-17T08:56:08.616 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55140.log 2024-04-17T08:56:08.617 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76390.log.gz 2024-04-17T08:56:08.617 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72469.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-17T08:56:08.618 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72469.log.gz 2024-04-17T08:56:08.619 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54477.log 2024-04-17T08:56:08.619 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55140.log.gz 2024-04-17T08:56:08.620 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75671.log 2024-04-17T08:56:08.621 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67517.log 2024-04-17T08:56:08.621 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54477.log.gz 2024-04-17T08:56:08.622 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75671.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75671.log.gz -5 2024-04-17T08:56:08.623 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84865.log 2024-04-17T08:56:08.624 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108751.log 2024-04-17T08:56:08.624 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67517.log.gz 2024-04-17T08:56:08.625 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84865.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84865.log.gz 2024-04-17T08:56:08.625 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74815.log 2024-04-17T08:56:08.626 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96873.log 2024-04-17T08:56:08.627 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108751.log.gz 2024-04-17T08:56:08.627 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74815.log: 90.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74815.log.gz -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-17T08:56:08.627 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:08.628 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74899.log 2024-04-17T08:56:08.628 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63530.log 2024-04-17T08:56:08.629 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96873.log.gz 2024-04-17T08:56:08.629 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95211.log 2024-04-17T08:56:08.629 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74899.log.gz 2024-04-17T08:56:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85415.log 2024-04-17T08:56:08.631 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63530.log.gz 2024-04-17T08:56:08.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65542.log 2024-04-17T08:56:08.632 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95211.log.gz 2024-04-17T08:56:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99238.log 2024-04-17T08:56:08.633 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85415.log.gz 2024-04-17T08:56:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63656.log 2024-04-17T08:56:08.634 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65542.log.gz 2024-04-17T08:56:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86158.log 2024-04-17T08:56:08.635 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99238.log.gz 2024-04-17T08:56:08.636 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82488.log 2024-04-17T08:56:08.636 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63656.log.gz 2024-04-17T08:56:08.637 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109276.log 2024-04-17T08:56:08.637 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86158.log.gz 2024-04-17T08:56:08.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110110.log 2024-04-17T08:56:08.638 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82488.log.gz 2024-04-17T08:56:08.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74111.log 2024-04-17T08:56:08.639 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109276.log.gz 2024-04-17T08:56:08.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71599.log 2024-04-17T08:56:08.640 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110110.log.gz 2024-04-17T08:56:08.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84781.log 2024-04-17T08:56:08.641 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74111.log.gz 2024-04-17T08:56:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71809.log 2024-04-17T08:56:08.642 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71599.log.gz 2024-04-17T08:56:08.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83748.log 2024-04-17T08:56:08.644 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84781.log.gz 2024-04-17T08:56:08.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71809.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71809.log.gz 2024-04-17T08:56:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88825.log 2024-04-17T08:56:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65374.log 2024-04-17T08:56:08.646 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.83748.log.gz 2024-04-17T08:56:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107132.log 2024-04-17T08:56:08.647 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88825.log.gz 2024-04-17T08:56:08.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70162.log 2024-04-17T08:56:08.648 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65374.log.gz 2024-04-17T08:56:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81456.log 2024-04-17T08:56:08.649 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107132.log.gz 2024-04-17T08:56:08.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99841.log 2024-04-17T08:56:08.651 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70162.log.gz 2024-04-17T08:56:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60631.log 2024-04-17T08:56:08.652 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81456.log.gz 2024-04-17T08:56:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-04-17T08:56:08.653 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99841.log.gz 2024-04-17T08:56:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59326.log 2024-04-17T08:56:08.654 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60631.log.gz 2024-04-17T08:56:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65017.log 2024-04-17T08:56:08.655 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-04-17T08:56:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2024-04-17T08:56:08.656 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59326.log.gz 2024-04-17T08:56:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95628.log 2024-04-17T08:56:08.657 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65017.log.gz 2024-04-17T08:56:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-04-17T08:56:08.658 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2024-04-17T08:56:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102813.log 2024-04-17T08:56:08.659 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95628.log.gz 2024-04-17T08:56:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67798.log 2024-04-17T08:56:08.660 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-04-17T08:56:08.661 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82817.log 2024-04-17T08:56:08.661 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102813.log.gz 2024-04-17T08:56:08.662 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55964.log 2024-04-17T08:56:08.663 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67798.log.gz 2024-04-17T08:56:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72887.log 2024-04-17T08:56:08.663 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82817.log.gz 2024-04-17T08:56:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107866.log 2024-04-17T08:56:08.665 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55964.log.gz 2024-04-17T08:56:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68484.log 2024-04-17T08:56:08.666 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72887.log.gz 2024-04-17T08:56:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.107866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107866.log.gz 2024-04-17T08:56:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74494.log 2024-04-17T08:56:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42979.log 2024-04-17T08:56:08.668 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68484.log.gz 2024-04-17T08:56:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109612.log 2024-04-17T08:56:08.669 INFO:teuthology.orchestra.run.smithi067.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.74494.log.gz 2024-04-17T08:56:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89768.log 2024-04-17T08:56:08.670 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42979.log.gz 2024-04-17T08:56:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101765.log 2024-04-17T08:56:08.671 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109612.log.gz 2024-04-17T08:56:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42976.log 2024-04-17T08:56:08.672 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.89768.log.gz 2024-04-17T08:56:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68778.log 2024-04-17T08:56:08.673 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101765.log.gz 2024-04-17T08:56:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68085.log 2024-04-17T08:56:08.674 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42976.log.gz 2024-04-17T08:56:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68400.log 2024-04-17T08:56:08.675 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68778.log.gz 2024-04-17T08:56:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90108.log 2024-04-17T08:56:08.676 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68085.log.gz 2024-04-17T08:56:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96593.log 2024-04-17T08:56:08.678 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68400.log.gz 2024-04-17T08:56:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90108.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90108.log.gz 2024-04-17T08:56:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95319.log 2024-04-17T08:56:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70704.log 2024-04-17T08:56:08.680 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96593.log.gz 2024-04-17T08:56:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89503.log 2024-04-17T08:56:08.681 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95319.log.gz 2024-04-17T08:56:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93073.log 2024-04-17T08:56:08.682 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70704.log.gz 2024-04-17T08:56:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71508.log 2024-04-17T08:56:08.683 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89503.log.gz 2024-04-17T08:56:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49306.log 2024-04-17T08:56:08.684 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93073.log.gz 2024-04-17T08:56:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59954.log 2024-04-17T08:56:08.685 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71508.log.gz 2024-04-17T08:56:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66618.log 2024-04-17T08:56:08.686 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49306.log.gz 2024-04-17T08:56:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93335.log 2024-04-17T08:56:08.687 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59954.log.gz 2024-04-17T08:56:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56609.log 2024-04-17T08:56:08.688 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66618.log.gz 2024-04-17T08:56:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62791.log 2024-04-17T08:56:08.689 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93335.log.gz 2024-04-17T08:56:08.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48637.log 2024-04-17T08:56:08.691 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56609.log.gz 2024-04-17T08:56:08.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66788.log 2024-04-17T08:56:08.692 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62791.log.gz 2024-04-17T08:56:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48427.log 2024-04-17T08:56:08.693 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48637.log.gz 2024-04-17T08:56:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106096.log 2024-04-17T08:56:08.694 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66788.log.gz 2024-04-17T08:56:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43766.log 2024-04-17T08:56:08.695 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48427.log.gz 2024-04-17T08:56:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105249.log 2024-04-17T08:56:08.696 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106096.log.gz 2024-04-17T08:56:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96789.log 2024-04-17T08:56:08.697 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43766.log.gz 2024-04-17T08:56:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66023.log 2024-04-17T08:56:08.698 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105249.log.gz 2024-04-17T08:56:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76306.log 2024-04-17T08:56:08.699 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96789.log.gz 2024-04-17T08:56:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88951.log 2024-04-17T08:56:08.700 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66023.log.gz 2024-04-17T08:56:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82901.log 2024-04-17T08:56:08.701 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76306.log.gz 2024-04-17T08:56:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46729.log 2024-04-17T08:56:08.702 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88951.log.gz 2024-04-17T08:56:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94247.log 2024-04-17T08:56:08.703 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82901.log.gz 2024-04-17T08:56:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101891.log 2024-04-17T08:56:08.704 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46729.log.gz 2024-04-17T08:56:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76571.log 2024-04-17T08:56:08.705 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94247.log.gz 2024-04-17T08:56:08.706 INFO:teuthology.orchestra.run.smithi187.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-17T08:56:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-04-17T08:56:08.707 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101891.log.gz 2024-04-17T08:56:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102458.log 2024-04-17T08:56:08.708 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76571.log.gz 2024-04-17T08:56:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67142.log 2024-04-17T08:56:08.709 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-04-17T08:56:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49730.log 2024-04-17T08:56:08.710 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102458.log.gz 2024-04-17T08:56:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68862.log 2024-04-17T08:56:08.711 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.67142.log.gz 2024-04-17T08:56:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2024-04-17T08:56:08.712 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49730.log.gz 2024-04-17T08:56:08.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50860.log 2024-04-17T08:56:08.713 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68862.log.gz 2024-04-17T08:56:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93869.log 2024-04-17T08:56:08.714 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2024-04-17T08:56:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102729.log 2024-04-17T08:56:08.715 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50860.log.gz 2024-04-17T08:56:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82859.log 2024-04-17T08:56:08.716 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93869.log.gz 2024-04-17T08:56:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96317.log 2024-04-17T08:56:08.717 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102729.log.gz 2024-04-17T08:56:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90066.log 2024-04-17T08:56:08.718 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82859.log.gz 2024-04-17T08:56:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84534.log 2024-04-17T08:56:08.719 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96317.log.gz 2024-04-17T08:56:08.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85756.log 2024-04-17T08:56:08.721 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90066.log.gz 2024-04-17T08:56:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 2024-04-17T08:56:08.722 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84534.log.gz 2024-04-17T08:56:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76785.log 2024-04-17T08:56:08.723 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85756.log.gz 2024-04-17T08:56:08.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56258.log 2024-04-17T08:56:08.724 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz 2024-04-17T08:56:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103228.log 2024-04-17T08:56:08.725 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76785.log.gz 2024-04-17T08:56:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98727.log 2024-04-17T08:56:08.726 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56258.log.gz 2024-04-17T08:56:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47335.log 2024-04-17T08:56:08.727 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103228.log.gz 2024-04-17T08:56:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48343.log 2024-04-17T08:56:08.728 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98727.log.gz 2024-04-17T08:56:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81624.log 2024-04-17T08:56:08.729 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47335.log.gz 2024-04-17T08:56:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97347.log 2024-04-17T08:56:08.730 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48343.log.gz 2024-04-17T08:56:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50344.log 2024-04-17T08:56:08.731 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81624.log.gz 2024-04-17T08:56:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62959.log 2024-04-17T08:56:08.732 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97347.log.gz 2024-04-17T08:56:08.733 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65279.log 2024-04-17T08:56:08.734 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50344.log.gz 2024-04-17T08:56:08.734 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54901.log 2024-04-17T08:56:08.734 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62959.log.gz 2024-04-17T08:56:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65813.log 2024-04-17T08:56:08.736 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65279.log.gz 2024-04-17T08:56:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59768.log 2024-04-17T08:56:08.737 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54901.log.gz 2024-04-17T08:56:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85840.log 2024-04-17T08:56:08.738 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65813.log.gz 2024-04-17T08:56:08.739 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108506.log 2024-04-17T08:56:08.739 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59768.log.gz 2024-04-17T08:56:08.740 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56483.log 2024-04-17T08:56:08.740 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85840.log.gz 2024-04-17T08:56:08.741 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99799.log 2024-04-17T08:56:08.741 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108506.log.gz 2024-04-17T08:56:08.743 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52898.log 2024-04-17T08:56:08.743 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56483.log.gz 2024-04-17T08:56:08.744 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42645.log 2024-04-17T08:56:08.745 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99799.log.gz 2024-04-17T08:56:08.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71326.log 2024-04-17T08:56:08.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52898.log.gz 2024-04-17T08:56:08.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2024-04-17T08:56:08.746 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42645.log.gz 2024-04-17T08:56:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83486.log 2024-04-17T08:56:08.747 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71326.log.gz 2024-04-17T08:56:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83653.log 2024-04-17T08:56:08.748 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2024-04-17T08:56:08.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85373.log 2024-04-17T08:56:08.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83486.log.gz 2024-04-17T08:56:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106409.log 2024-04-17T08:56:08.750 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83653.log.gz 2024-04-17T08:56:08.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71641.log 2024-04-17T08:56:08.751 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85373.log.gz 2024-04-17T08:56:08.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94522.log 2024-04-17T08:56:08.752 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106409.log.gz 2024-04-17T08:56:08.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97956.log 2024-04-17T08:56:08.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71641.log.gz 2024-04-17T08:56:08.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81937.log 2024-04-17T08:56:08.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94522.log.gz 2024-04-17T08:56:08.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45979.log 2024-04-17T08:56:08.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97956.log.gz 2024-04-17T08:56:08.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66447.log 2024-04-17T08:56:08.756 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81937.log.gz 2024-04-17T08:56:08.757 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100863.log 2024-04-17T08:56:08.757 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45979.log.gz 2024-04-17T08:56:08.758 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44768.log 2024-04-17T08:56:08.758 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66447.log.gz 2024-04-17T08:56:08.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89587.log 2024-04-17T08:56:08.759 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100863.log.gz 2024-04-17T08:56:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.2.44768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.44768.log.gz 2024-04-17T08:56:08.760 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47251.log 2024-04-17T08:56:08.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101450.log 2024-04-17T08:56:08.761 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89587.log.gz 2024-04-17T08:56:08.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50218.log 2024-04-17T08:56:08.768 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47251.log.gz 2024-04-17T08:56:08.768 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42641.log 2024-04-17T08:56:08.768 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101450.log.gz 2024-04-17T08:56:08.768 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80973.log 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50218.log.gz 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105165.log 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42641.log.gz 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98564.log 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80973.log.gz 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99711.log 2024-04-17T08:56:08.769 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105165.log.gz 2024-04-17T08:56:08.770 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-04-17T08:56:08.770 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98564.log.gz 2024-04-17T08:56:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100426.log 2024-04-17T08:56:08.771 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99711.log.gz 2024-04-17T08:56:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101283.log 2024-04-17T08:56:08.772 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-04-17T08:56:08.772 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83925.log 2024-04-17T08:56:08.773 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100426.log.gz 2024-04-17T08:56:08.773 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94121.log 2024-04-17T08:56:08.774 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.101283.log.gz 2024-04-17T08:56:08.774 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108095.log 2024-04-17T08:56:08.774 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83925.log.gz 2024-04-17T08:56:08.775 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51745.log 2024-04-17T08:56:08.776 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94121.log.gz 2024-04-17T08:56:08.777 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71557.log 2024-04-17T08:56:08.777 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108095.log.gz 2024-04-17T08:56:08.778 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82592.log 2024-04-17T08:56:08.778 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51745.log.gz 2024-04-17T08:56:08.778 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42309.log 2024-04-17T08:56:08.779 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71557.log.gz 2024-04-17T08:56:08.780 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75130.log 2024-04-17T08:56:08.780 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82592.log.gz 2024-04-17T08:56:08.781 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65771.log 2024-04-17T08:56:08.781 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42309.log.gz 2024-04-17T08:56:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55360.log 2024-04-17T08:56:08.782 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75130.log.gz 2024-04-17T08:56:08.783 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61116.log 2024-04-17T08:56:08.783 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65771.log.gz 2024-04-17T08:56:08.784 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55360.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-17T08:56:08.784 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55360.log.gz 2024-04-17T08:56:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110026.log 2024-04-17T08:56:08.785 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61116.log.gz 2024-04-17T08:56:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63967.log 2024-04-17T08:56:08.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60039.log 2024-04-17T08:56:08.787 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110026.log.gz 2024-04-17T08:56:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47377.log 2024-04-17T08:56:08.789 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63967.log.gz 2024-04-17T08:56:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62917.log 2024-04-17T08:56:08.790 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60039.log.gz 2024-04-17T08:56:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64636.log 2024-04-17T08:56:08.792 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47377.log.gz 2024-04-17T08:56:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110320.log 2024-04-17T08:56:08.793 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62917.log.gz 2024-04-17T08:56:08.794 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63879.log 2024-04-17T08:56:08.794 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64636.log.gz 2024-04-17T08:56:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73923.log 2024-04-17T08:56:08.795 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110320.log.gz 2024-04-17T08:56:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68736.log 2024-04-17T08:56:08.797 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63879.log.gz 2024-04-17T08:56:08.798 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95043.log 2024-04-17T08:56:08.798 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73923.log.gz 2024-04-17T08:56:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94959.log 2024-04-17T08:56:08.799 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68736.log.gz 2024-04-17T08:56:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69738.log 2024-04-17T08:56:08.801 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95043.log.gz 2024-04-17T08:56:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81235.log 2024-04-17T08:56:08.802 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94959.log.gz 2024-04-17T08:56:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56172.log 2024-04-17T08:56:08.803 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69738.log.gz 2024-04-17T08:56:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41294.log 2024-04-17T08:56:08.805 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81235.log.gz 2024-04-17T08:56:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82021.log 2024-04-17T08:56:08.806 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56172.log.gz 2024-04-17T08:56:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66746.log 2024-04-17T08:56:08.807 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41294.log.gz 2024-04-17T08:56:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93995.log 2024-04-17T08:56:08.809 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82021.log.gz 2024-04-17T08:56:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98354.log 2024-04-17T08:56:08.810 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66746.log.gz 2024-04-17T08:56:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75839.log 2024-04-17T08:56:08.811 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93995.log.gz 2024-04-17T08:56:08.812 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62224.log 2024-04-17T08:56:08.813 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98354.log.gz 2024-04-17T08:56:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100384.log 2024-04-17T08:56:08.814 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75839.log.gz 2024-04-17T08:56:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47480.log 2024-04-17T08:56:08.815 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62224.log.gz 2024-04-17T08:56:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88783.log 2024-04-17T08:56:08.817 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100384.log.gz 2024-04-17T08:56:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83706.log 2024-04-17T08:56:08.818 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47480.log.gz 2024-04-17T08:56:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88783.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88783.log.gz 2024-04-17T08:56:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65237.log 2024-04-17T08:56:08.820 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65939.log 2024-04-17T08:56:08.821 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83706.log.gz 2024-04-17T08:56:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103396.log 2024-04-17T08:56:08.821 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65237.log.gz 2024-04-17T08:56:08.822 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41876.log 2024-04-17T08:56:08.823 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65939.log.gz 2024-04-17T08:56:08.823 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103480.log 2024-04-17T08:56:08.824 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103396.log.gz 2024-04-17T08:56:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96065.log 2024-04-17T08:56:08.825 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41876.log.gz 2024-04-17T08:56:08.825 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70120.log 2024-04-17T08:56:08.826 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103480.log.gz 2024-04-17T08:56:08.826 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65059.log 2024-04-17T08:56:08.827 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96065.log.gz 2024-04-17T08:56:08.827 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104258.log 2024-04-17T08:56:08.828 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70120.log.gz 2024-04-17T08:56:08.828 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83528.log 2024-04-17T08:56:08.829 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65059.log.gz 2024-04-17T08:56:08.829 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110404.log 2024-04-17T08:56:08.830 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104258.log.gz 2024-04-17T08:56:08.830 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67720.log 2024-04-17T08:56:08.831 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83528.log.gz 2024-04-17T08:56:08.831 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74857.log 2024-04-17T08:56:08.832 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110404.log.gz 2024-04-17T08:56:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73380.log 2024-04-17T08:56:08.833 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67720.log.gz 2024-04-17T08:56:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68127.log 2024-04-17T08:56:08.834 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74857.log.gz 2024-04-17T08:56:08.834 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75713.log 2024-04-17T08:56:08.835 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.73380.log.gz 2024-04-17T08:56:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43514.log 2024-04-17T08:56:08.836 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68127.log.gz 2024-04-17T08:56:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46351.log 2024-04-17T08:56:08.837 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75713.log.gz 2024-04-17T08:56:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43514.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43514.log.gz 2024-04-17T08:56:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94829.log 2024-04-17T08:56:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95882.log 2024-04-17T08:56:08.840 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46351.log.gz 2024-04-17T08:56:08.840 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65728.log 2024-04-17T08:56:08.840 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94829.log.gz 2024-04-17T08:56:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71767.log 2024-04-17T08:56:08.841 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95882.log.gz 2024-04-17T08:56:08.842 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77495.log 2024-04-17T08:56:08.842 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65728.log.gz 2024-04-17T08:56:08.843 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96019.log 2024-04-17T08:56:08.843 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71767.log.gz 2024-04-17T08:56:08.844 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62266.log 2024-04-17T08:56:08.844 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77495.log.gz 2024-04-17T08:56:08.845 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101807.log 2024-04-17T08:56:08.845 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96019.log.gz 2024-04-17T08:56:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.95507.log 2024-04-17T08:56:08.846 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62266.log.gz 2024-04-17T08:56:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95085.log 2024-04-17T08:56:08.847 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101807.log.gz 2024-04-17T08:56:08.848 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.95507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.95507.log.gz 2024-04-17T08:56:08.848 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2024-04-17T08:56:08.849 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108286.log 2024-04-17T08:56:08.849 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95085.log.gz 2024-04-17T08:56:08.850 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110194.log 2024-04-17T08:56:08.850 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2024-04-17T08:56:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69030.log 2024-04-17T08:56:08.851 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108286.log.gz 2024-04-17T08:56:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-04-17T08:56:08.852 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110194.log.gz 2024-04-17T08:56:08.853 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76529.log 2024-04-17T08:56:08.853 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69030.log.gz 2024-04-17T08:56:08.854 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82943.log 2024-04-17T08:56:08.854 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-04-17T08:56:08.855 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76529.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.96467.log -- replaced with /var/log/ceph/ceph-client.admin.76529.log.gz 2024-04-17T08:56:08.855 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82943.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.82943.log.gz 2024-04-17T08:56:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107002.log 2024-04-17T08:56:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75035.log 2024-04-17T08:56:08.857 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96467.log.gz 2024-04-17T08:56:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.100305.log 2024-04-17T08:56:08.858 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107002.log.gz 2024-04-17T08:56:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108793.log 2024-04-17T08:56:08.859 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75035.log.gz 2024-04-17T08:56:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.100305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.100305.log.gz 2024-04-17T08:56:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 2024-04-17T08:56:08.861 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107044.log 2024-04-17T08:56:08.861 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108793.log.gz 2024-04-17T08:56:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109804.log 2024-04-17T08:56:08.862 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-04-17T08:56:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2024-04-17T08:56:08.863 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107044.log.gz 2024-04-17T08:56:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109804.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109804.log.gz 2024-04-17T08:56:08.864 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82320.log 2024-04-17T08:56:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103947.log 2024-04-17T08:56:08.865 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2024-04-17T08:56:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42629.log 2024-04-17T08:56:08.866 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82320.log.gz 2024-04-17T08:56:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103947.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103947.log.gz 2024-04-17T08:56:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110656.log 2024-04-17T08:56:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103312.log 2024-04-17T08:56:08.869 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42629.log.gz 2024-04-17T08:56:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71996.log 2024-04-17T08:56:08.870 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110656.log.gz 2024-04-17T08:56:08.871 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88531.log 2024-04-17T08:56:08.871 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103312.log.gz 2024-04-17T08:56:08.872 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62140.log 2024-04-17T08:56:08.872 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71996.log.gz 2024-04-17T08:56:08.872 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60930.log 2024-04-17T08:56:08.873 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88531.log.gz 2024-04-17T08:56:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43808.log 2024-04-17T08:56:08.874 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62140.log.gz 2024-04-17T08:56:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66914.log 2024-04-17T08:56:08.875 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60930.log.gz 2024-04-17T08:56:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100051.log 2024-04-17T08:56:08.876 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43808.log.gz 2024-04-17T08:56:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110152.log 2024-04-17T08:56:08.877 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66914.log.gz 2024-04-17T08:56:08.878 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101241.log 2024-04-17T08:56:08.878 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.100051.log.gz 2024-04-17T08:56:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110530.log 2024-04-17T08:56:08.879 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110152.log.gz 2024-04-17T08:56:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82729.log 2024-04-17T08:56:08.880 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101241.log.gz 2024-04-17T08:56:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110530.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.70036.log 2024-04-17T08:56:08.881 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.110530.log.gz 2024-04-17T08:56:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82729.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.82729.log.gz 2024-04-17T08:56:08.882 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.83111.log 2024-04-17T08:56:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88030.log 2024-04-17T08:56:08.883 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70036.log.gz 2024-04-17T08:56:08.884 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45280.log 2024-04-17T08:56:08.885 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.83111.log.gz 2024-04-17T08:56:08.886 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85075.log 2024-04-17T08:56:08.886 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88030.log.gz 2024-04-17T08:56:08.887 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85924.log 2024-04-17T08:56:08.887 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45280.log.gz 2024-04-17T08:56:08.888 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88072.log 2024-04-17T08:56:08.888 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85075.log.gz 2024-04-17T08:56:08.889 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73714.log 2024-04-17T08:56:08.889 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85924.log.gz 2024-04-17T08:56:08.890 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88072.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88072.log.gz 2024-04-17T08:56:08.890 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104510.log 2024-04-17T08:56:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48763.log 2024-04-17T08:56:08.891 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73714.log.gz 2024-04-17T08:56:08.892 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68043.log 2024-04-17T08:56:08.893 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104510.log.gz 2024-04-17T08:56:08.894 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83365.log 2024-04-17T08:56:08.894 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48763.log.gz 2024-04-17T08:56:08.895 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68043.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68043.log.gz 2024-04-17T08:56:08.895 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61622.log 2024-04-17T08:56:08.896 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.83365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.83365.log.gz 2024-04-17T08:56:08.896 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75298.log 2024-04-17T08:56:08.897 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.61622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61622.log.gz 2024-04-17T08:56:08.897 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71242.log 2024-04-17T08:56:08.898 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69910.log 2024-04-17T08:56:08.898 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75298.log.gz 2024-04-17T08:56:08.900 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71242.log.gz 2024-04-17T08:56:08.900 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64009.log 2024-04-17T08:56:08.900 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96831.log 2024-04-17T08:56:08.901 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69910.log.gz 2024-04-17T08:56:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64009.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64009.log.gz 2024-04-17T08:56:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99361.log 2024-04-17T08:56:08.903 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61284.log 2024-04-17T08:56:08.903 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96831.log.gz 2024-04-17T08:56:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93556.log 2024-04-17T08:56:08.904 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99361.log.gz 2024-04-17T08:56:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107318.log 2024-04-17T08:56:08.905 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61284.log.gz 2024-04-17T08:56:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59642.log 2024-04-17T08:56:08.907 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93556.log.gz 2024-04-17T08:56:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68358.log 2024-04-17T08:56:08.908 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107318.log.gz 2024-04-17T08:56:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63786.log 2024-04-17T08:56:08.909 INFO:teuthology.orchestra.run.smithi067.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.59642.log.gz 2024-04-17T08:56:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59997.log 2024-04-17T08:56:08.911 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68358.log.gz 2024-04-17T08:56:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104342.log 2024-04-17T08:56:08.912 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63786.log.gz 2024-04-17T08:56:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51391.log 2024-04-17T08:56:08.913 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59997.log.gz 2024-04-17T08:56:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92954.log 2024-04-17T08:56:08.915 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104342.log.gz 2024-04-17T08:56:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81099.log 2024-04-17T08:56:08.916 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51391.log.gz 2024-04-17T08:56:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60375.log 2024-04-17T08:56:08.917 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92954.log.gz 2024-04-17T08:56:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81099.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.0.104806.log 2024-04-17T08:56:08.919 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81099.log.gz 2024-04-17T08:56:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73212.log 2024-04-17T08:56:08.920 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60375.log.gz 2024-04-17T08:56:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.104806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.104806.log.gz 2024-04-17T08:56:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107945.log 2024-04-17T08:56:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55182.log 2024-04-17T08:56:08.922 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73212.log.gz 2024-04-17T08:56:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103564.log 2024-04-17T08:56:08.923 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107945.log.gz 2024-04-17T08:56:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55182.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55182.log.gz 2024-04-17T08:56:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94205.log 2024-04-17T08:56:08.925 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51568.log 2024-04-17T08:56:08.926 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103564.log.gz 2024-04-17T08:56:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66107.log 2024-04-17T08:56:08.927 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94205.log.gz 2024-04-17T08:56:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74330.log 2024-04-17T08:56:08.928 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-04-17T08:56:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42044.log 2024-04-17T08:56:08.930 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66107.log.gz 2024-04-17T08:56:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107987.log 2024-04-17T08:56:08.931 INFO:teuthology.orchestra.run.smithi067.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.74330.log.gz 2024-04-17T08:56:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108053.log 2024-04-17T08:56:08.932 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2024-04-17T08:56:08.933 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89417.log 2024-04-17T08:56:08.933 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107987.log.gz 2024-04-17T08:56:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65897.log 2024-04-17T08:56:08.934 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108053.log.gz 2024-04-17T08:56:08.935 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95712.log 2024-04-17T08:56:08.936 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.89417.log.gz 2024-04-17T08:56:08.936 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98312.log 2024-04-17T08:56:08.937 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65897.log.gz 2024-04-17T08:56:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77436.log 2024-04-17T08:56:08.938 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95712.log.gz 2024-04-17T08:56:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62360.log 2024-04-17T08:56:08.939 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98312.log.gz 2024-04-17T08:56:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97016.log 2024-04-17T08:56:08.940 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77436.log.gz 2024-04-17T08:56:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73170.log 2024-04-17T08:56:08.942 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62360.log.gz 2024-04-17T08:56:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60470.log 2024-04-17T08:56:08.943 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97016.log.gz 2024-04-17T08:56:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69643.log 2024-04-17T08:56:08.945 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73170.log.gz 2024-04-17T08:56:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101175.log 2024-04-17T08:56:08.946 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60470.log.gz 2024-04-17T08:56:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-04-17T08:56:08.947 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69643.log.gz 2024-04-17T08:56:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93115.log 2024-04-17T08:56:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46855.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.101175.log.gz 2024-04-17T08:56:08.950 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-04-17T08:56:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-17T08:56:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-04-17T08:56:08.951 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93115.log.gz 2024-04-17T08:56:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59726.log 2024-04-17T08:56:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97494.log 2024-04-17T08:56:08.954 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-04-17T08:56:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106714.log 2024-04-17T08:56:08.956 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59726.log.gz 2024-04-17T08:56:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101681.log 2024-04-17T08:56:08.958 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97494.log.gz 2024-04-17T08:56:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-04-17T08:56:08.959 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106714.log.gz 2024-04-17T08:56:08.960 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93724.log 2024-04-17T08:56:08.960 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101681.log.gz 2024-04-17T08:56:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56353.log 2024-04-17T08:56:08.962 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-04-17T08:56:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103989.log 2024-04-17T08:56:08.963 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93724.log.gz 2024-04-17T08:56:08.964 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104033.log 2024-04-17T08:56:08.964 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56353.log.gz 2024-04-17T08:56:08.965 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98624.log 2024-04-17T08:56:08.966 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103989.log.gz 2024-04-17T08:56:08.967 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49240.log 2024-04-17T08:56:08.968 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104033.log.gz 2024-04-17T08:56:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109528.log 2024-04-17T08:56:08.969 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98624.log.gz 2024-04-17T08:56:08.970 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55308.log 2024-04-17T08:56:08.970 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49240.log.gz 2024-04-17T08:56:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85882.log 2024-04-17T08:56:08.972 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109528.log.gz 2024-04-17T08:56:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105207.log 2024-04-17T08:56:08.973 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55308.log.gz 2024-04-17T08:56:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63169.log 2024-04-17T08:56:08.974 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85882.log.gz 2024-04-17T08:56:08.976 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42630.log 2024-04-17T08:56:08.976 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105207.log.gz 2024-04-17T08:56:08.977 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51804.log 2024-04-17T08:56:08.977 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63169.log.gz 2024-04-17T08:56:08.978 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65855.log 2024-04-17T08:56:08.979 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42630.log.gz 2024-04-17T08:56:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66662.log 2024-04-17T08:56:08.980 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51804.log.gz 2024-04-17T08:56:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98101.log 2024-04-17T08:56:08.982 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65855.log.gz 2024-04-17T08:56:08.983 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85714.log 2024-04-17T08:56:08.983 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66662.log.gz 2024-04-17T08:56:08.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57444.log 2024-04-17T08:56:08.984 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98101.log.gz 2024-04-17T08:56:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102206.log 2024-04-17T08:56:08.986 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85714.log.gz 2024-04-17T08:56:08.987 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46105.log 2024-04-17T08:56:08.987 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57444.log.gz 2024-04-17T08:56:08.988 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110572.log 2024-04-17T08:56:08.988 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102206.log.gz 2024-04-17T08:56:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2024-04-17T08:56:08.990 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46105.log.gz 2024-04-17T08:56:08.991 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81288.log 2024-04-17T08:56:08.991 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110572.log.gz 2024-04-17T08:56:08.992 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77566.log 2024-04-17T08:56:08.992 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2024-04-17T08:56:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107486.log 2024-04-17T08:56:08.994 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81288.log.gz 2024-04-17T08:56:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102687.log 2024-04-17T08:56:08.995 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77566.log.gz 2024-04-17T08:56:08.996 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101391.log 2024-04-17T08:56:08.996 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107486.log.gz 2024-04-17T08:56:08.997 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89035.log 2024-04-17T08:56:08.997 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102687.log.gz 2024-04-17T08:56:08.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95977.log 2024-04-17T08:56:08.999 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101391.log.gz 2024-04-17T08:56:08.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94917.log 2024-04-17T08:56:09.000 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.89035.log.gz 2024-04-17T08:56:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64261.log 2024-04-17T08:56:09.001 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95977.log.gz 2024-04-17T08:56:09.002 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89336.log 2024-04-17T08:56:09.002 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94917.log.gz 2024-04-17T08:56:09.003 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106619.log 2024-04-17T08:56:09.004 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89336.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.64261.log.gz 2024-04-17T08:56:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100636.log 2024-04-17T08:56:09.005 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89336.log.gz 2024-04-17T08:56:09.006 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76348.log 2024-04-17T08:56:09.006 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106619.log.gz 2024-04-17T08:56:09.007 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45742.log 2024-04-17T08:56:09.008 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100636.log.gz 2024-04-17T08:56:09.008 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76348.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76348.log.gz 2024-04-17T08:56:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93766.log 2024-04-17T08:56:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64327.log 2024-04-17T08:56:09.010 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45742.log.gz 2024-04-17T08:56:09.011 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109192.log 2024-04-17T08:56:09.011 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93766.log.gz 2024-04-17T08:56:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109150.log 2024-04-17T08:56:09.013 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64327.log.gz 2024-04-17T08:56:09.014 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62623.log 2024-04-17T08:56:09.014 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109192.log.gz 2024-04-17T08:56:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101520.log 2024-04-17T08:56:09.015 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109150.log.gz 2024-04-17T08:56:09.016 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-04-17T08:56:09.016 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62623.log.gz 2024-04-17T08:56:09.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66244.log 2024-04-17T08:56:09.018 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101520.log.gz 2024-04-17T08:56:09.018 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76953.log 2024-04-17T08:56:09.019 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-04-17T08:56:09.020 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83027.log 2024-04-17T08:56:09.020 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66244.log.gz 2024-04-17T08:56:09.021 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81498.log 2024-04-17T08:56:09.022 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.76953.log.gz 2024-04-17T08:56:09.022 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56130.log 2024-04-17T08:56:09.023 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83027.log.gz 2024-04-17T08:56:09.024 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84093.log 2024-04-17T08:56:09.024 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81498.log.gz 2024-04-17T08:56:09.025 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-04-17T08:56:09.025 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56130.log.gz 2024-04-17T08:56:09.026 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41960.log 2024-04-17T08:56:09.027 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84093.log.gz 2024-04-17T08:56:09.028 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101849.log 2024-04-17T08:56:09.029 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-04-17T08:56:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41960.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/ceph-client.admin.41960.log.gz /var/log/ceph/ceph-client.admin.51627.log 2024-04-17T08:56:09.029 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:09.030 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68263.log 2024-04-17T08:56:09.030 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101849.log.gz 2024-04-17T08:56:09.031 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77019.log 2024-04-17T08:56:09.032 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51627.log.gz 2024-04-17T08:56:09.033 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58971.log 2024-04-17T08:56:09.033 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68263.log.gz 2024-04-17T08:56:09.034 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83967.log 2024-04-17T08:56:09.034 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77019.log.gz 2024-04-17T08:56:09.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51686.log 2024-04-17T08:56:09.035 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58971.log.gz 2024-04-17T08:56:09.036 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45616.log 2024-04-17T08:56:09.036 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83967.log.gz 2024-04-17T08:56:09.037 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102332.log 2024-04-17T08:56:09.038 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51686.log.gz 2024-04-17T08:56:09.039 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76911.log 2024-04-17T08:56:09.040 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45616.log.gz 2024-04-17T08:56:09.040 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99669.log 2024-04-17T08:56:09.040 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.102332.log.gz 2024-04-17T08:56:09.042 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62581.log 2024-04-17T08:56:09.042 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76911.log.gz 2024-04-17T08:56:09.043 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99669.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99669.log.gz 2024-04-17T08:56:09.043 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103438.log 2024-04-17T08:56:09.044 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47125.log 2024-04-17T08:56:09.044 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62581.log.gz 2024-04-17T08:56:09.045 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105613.log 2024-04-17T08:56:09.045 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103438.log.gz 2024-04-17T08:56:09.046 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109234.log 2024-04-17T08:56:09.047 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47125.log.gz 2024-04-17T08:56:09.048 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82687.log 2024-04-17T08:56:09.048 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105613.log.gz 2024-04-17T08:56:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72799.log 2024-04-17T08:56:09.049 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109234.log.gz 2024-04-17T08:56:09.050 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50818.log 2024-04-17T08:56:09.050 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82687.log.gz 2024-04-17T08:56:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-04-17T08:56:09.052 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72799.log.gz 2024-04-17T08:56:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108345.log 2024-04-17T08:56:09.053 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50818.log.gz 2024-04-17T08:56:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99446.log 2024-04-17T08:56:09.054 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-04-17T08:56:09.055 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76743.log 2024-04-17T08:56:09.055 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108345.log.gz 2024-04-17T08:56:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90634.log 2024-04-17T08:56:09.057 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99446.log.gz 2024-04-17T08:56:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49119.log 2024-04-17T08:56:09.058 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76743.log.gz 2024-04-17T08:56:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87196.log 2024-04-17T08:56:09.059 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90634.log.gz 2024-04-17T08:56:09.060 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.49119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62098.log 2024-04-17T08:56:09.060 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49119.log.gz 2024-04-17T08:56:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94648.log 2024-04-17T08:56:09.062 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87196.log.gz 2024-04-17T08:56:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50260.log 2024-04-17T08:56:09.063 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62098.log.gz 2024-04-17T08:56:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47437.log 2024-04-17T08:56:09.064 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94648.log.gz 2024-04-17T08:56:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82446.log 2024-04-17T08:56:09.065 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50260.log.gz 2024-04-17T08:56:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47437.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.47437.log.gz 2024-04-17T08:56:09.067 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.106325.log 2024-04-17T08:56:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76060.log 2024-04-17T08:56:09.068 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82446.log.gz 2024-04-17T08:56:09.069 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100552.log 2024-04-17T08:56:09.069 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106325.log.gz 2024-04-17T08:56:09.070 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89982.log 2024-04-17T08:56:09.070 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76060.log.gz 2024-04-17T08:56:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105886.log 2024-04-17T08:56:09.072 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100552.log.gz 2024-04-17T08:56:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89982.log: gzip -5 --verbose 87.5% -- /var/log/ceph/ceph-client.admin.90525.log -- replaced with /var/log/ceph/ceph-client.admin.89982.log.gz 2024-04-17T08:56:09.073 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:09.074 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60972.log 2024-04-17T08:56:09.074 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105886.log.gz 2024-04-17T08:56:09.075 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60758.log 2024-04-17T08:56:09.075 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90525.log.gz 2024-04-17T08:56:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100817.log 2024-04-17T08:56:09.077 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60972.log.gz 2024-04-17T08:56:09.078 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60758.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60758.log.gz 2024-04-17T08:56:09.078 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47811.log 2024-04-17T08:56:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69473.log 2024-04-17T08:56:09.079 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100817.log.gz 2024-04-17T08:56:09.080 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105035.log 2024-04-17T08:56:09.081 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47811.log.gz 2024-04-17T08:56:09.082 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95926.log 2024-04-17T08:56:09.082 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69473.log.gz 2024-04-17T08:56:09.083 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45322.log 2024-04-17T08:56:09.083 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105035.log.gz 2024-04-17T08:56:09.084 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57214.log 2024-04-17T08:56:09.084 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95926.log.gz 2024-04-17T08:56:09.085 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68904.log 2024-04-17T08:56:09.086 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45322.log.gz 2024-04-17T08:56:09.087 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62406.log 2024-04-17T08:56:09.087 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57214.log.gz 2024-04-17T08:56:09.088 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75382.log 2024-04-17T08:56:09.088 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68904.log.gz 2024-04-17T08:56:09.089 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84009.log 2024-04-17T08:56:09.089 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62406.log.gz 2024-04-17T08:56:09.090 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97536.log 2024-04-17T08:56:09.091 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75382.log.gz 2024-04-17T08:56:09.092 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75965.log 2024-04-17T08:56:09.092 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84009.log.gz 2024-04-17T08:56:09.093 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2024-04-17T08:56:09.093 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97536.log.gz 2024-04-17T08:56:09.094 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81015.log 2024-04-17T08:56:09.095 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75965.log.gz 2024-04-17T08:56:09.096 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94480.log 2024-04-17T08:56:09.096 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2024-04-17T08:56:09.097 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75629.log 2024-04-17T08:56:09.097 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81015.log.gz 2024-04-17T08:56:09.098 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95670.log 2024-04-17T08:56:09.098 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94480.log.gz 2024-04-17T08:56:09.099 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59810.log 2024-04-17T08:56:09.099 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75629.log.gz 2024-04-17T08:56:09.100 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86467.log 2024-04-17T08:56:09.101 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95670.log.gz 2024-04-17T08:56:09.101 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101007.log 2024-04-17T08:56:09.102 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59810.log.gz 2024-04-17T08:56:09.103 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101325.log 2024-04-17T08:56:09.103 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86467.log.gz 2024-04-17T08:56:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105445.log 2024-04-17T08:56:09.105 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101007.log.gz 2024-04-17T08:56:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74453.log 2024-04-17T08:56:09.106 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101325.log.gz 2024-04-17T08:56:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94734.log 2024-04-17T08:56:09.107 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105445.log.gz 2024-04-17T08:56:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74453.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.74453.log.gz 2024-04-17T08:56:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52372.log 2024-04-17T08:56:09.109 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94734.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.42635.log -- replaced with /var/log/ceph/ceph-client.admin.94734.log.gz 2024-04-17T08:56:09.110 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99063.log 2024-04-17T08:56:09.111 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52372.log.gz 2024-04-17T08:56:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48679.log 2024-04-17T08:56:09.112 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-04-17T08:56:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80812.log 2024-04-17T08:56:09.114 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99063.log.gz 2024-04-17T08:56:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49424.log 2024-04-17T08:56:09.115 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48679.log.gz 2024-04-17T08:56:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.80812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97620.log 2024-04-17T08:56:09.116 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80812.log.gz 2024-04-17T08:56:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98769.log 2024-04-17T08:56:09.118 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49424.log.gz 2024-04-17T08:56:09.118 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87869.log 2024-04-17T08:56:09.119 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97620.log.gz 2024-04-17T08:56:09.119 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83177.log 2024-04-17T08:56:09.120 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98769.log.gz 2024-04-17T08:56:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65668.log 2024-04-17T08:56:09.121 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87869.log.gz 2024-04-17T08:56:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61326.log 2024-04-17T08:56:09.122 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83177.log.gz 2024-04-17T08:56:09.123 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110799.log 2024-04-17T08:56:09.124 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65668.log.gz 2024-04-17T08:56:09.124 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61326.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61326.log.gz 2024-04-17T08:56:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104927.log 2024-04-17T08:56:09.125 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63742.log 2024-04-17T08:56:09.126 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110799.log.gz 2024-04-17T08:56:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104977.log 2024-04-17T08:56:09.127 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104927.log.gz 2024-04-17T08:56:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63742.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.63742.log.gz --verbose 2024-04-17T08:56:09.128 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.93472.log 2024-04-17T08:56:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108190.log 2024-04-17T08:56:09.130 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104977.log.gz 2024-04-17T08:56:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93472.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.93472.log.gz 2024-04-17T08:56:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95127.log 2024-04-17T08:56:09.132 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108190.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108190.log.gz 2024-04-17T08:56:09.132 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50944.log 2024-04-17T08:56:09.133 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95127.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-17T08:56:09.133 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95127.log.gz 2024-04-17T08:56:09.135 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2024-04-17T08:56:09.135 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50944.log.gz 2024-04-17T08:56:09.135 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93514.log 2024-04-17T08:56:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68946.log 2024-04-17T08:56:09.137 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2024-04-17T08:56:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50986.log 2024-04-17T08:56:09.138 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93514.log.gz 2024-04-17T08:56:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68946.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68946.log.gz 2024-04-17T08:56:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61910.log 2024-04-17T08:56:09.141 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50986.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50986.log.gz 2024-04-17T08:56:09.141 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83594.log 2024-04-17T08:56:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72374.log 2024-04-17T08:56:09.143 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61910.log.gz 2024-04-17T08:56:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59284.log 2024-04-17T08:56:09.144 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83594.log.gz 2024-04-17T08:56:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57331.log 2024-04-17T08:56:09.146 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72374.log.gz 2024-04-17T08:56:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65626.log 2024-04-17T08:56:09.147 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59284.log.gz 2024-04-17T08:56:09.148 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67567.log 2024-04-17T08:56:09.149 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57331.log.gz 2024-04-17T08:56:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59505.log 2024-04-17T08:56:09.150 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65626.log.gz 2024-04-17T08:56:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52109.log 2024-04-17T08:56:09.151 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67567.log.gz 2024-04-17T08:56:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59505.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59505.log.gz 2024-04-17T08:56:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97662.log 2024-04-17T08:56:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72843.log 2024-04-17T08:56:09.161 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52109.log.gz 2024-04-17T08:56:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76617.log 2024-04-17T08:56:09.162 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97662.log.gz 2024-04-17T08:56:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102999.log 2024-04-17T08:56:09.162 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72843.log.gz 2024-04-17T08:56:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93199.log 2024-04-17T08:56:09.163 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76617.log.gz 2024-04-17T08:56:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48297.log 2024-04-17T08:56:09.163 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102999.log.gz 2024-04-17T08:56:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66956.log 2024-04-17T08:56:09.164 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93199.log.gz 2024-04-17T08:56:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54214.log 2024-04-17T08:56:09.167 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48297.log.gz 2024-04-17T08:56:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84949.log 2024-04-17T08:56:09.167 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66956.log.gz 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63271.log 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54214.log.gz 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79741.log 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84949.log.gz 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65981.log 2024-04-17T08:56:09.168 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63271.log.gz 2024-04-17T08:56:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67100.log 2024-04-17T08:56:09.170 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79741.log.gz 2024-04-17T08:56:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82063.log 2024-04-17T08:56:09.170 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65981.log.gz 2024-04-17T08:56:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64051.log 2024-04-17T08:56:09.171 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67100.log.gz 2024-04-17T08:56:09.172 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73756.log 2024-04-17T08:56:09.172 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82063.log.gz 2024-04-17T08:56:09.173 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70988.log 2024-04-17T08:56:09.173 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64051.log.gz 2024-04-17T08:56:09.174 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97578.log 2024-04-17T08:56:09.177 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73756.log.gz 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51214.log 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70988.log.gz 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56777.log 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97578.log.gz 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66065.log 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51214.log.gz 2024-04-17T08:56:09.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82548.log 2024-04-17T08:56:09.181 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.56777.log.gz 2024-04-17T08:56:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89545.log 2024-04-17T08:56:09.182 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66065.log.gz 2024-04-17T08:56:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85586.log 2024-04-17T08:56:09.182 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82548.log.gz 2024-04-17T08:56:09.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105487.log 2024-04-17T08:56:09.183 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.89545.log.gz 2024-04-17T08:56:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90693.log 2024-04-17T08:56:09.185 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85586.log.gz 2024-04-17T08:56:09.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90150.log 2024-04-17T08:56:09.186 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105487.log.gz 2024-04-17T08:56:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55663.log 2024-04-17T08:56:09.187 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90693.log.gz 2024-04-17T08:56:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74289.log 2024-04-17T08:56:09.189 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.90150.log.gz 2024-04-17T08:56:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64856.log 2024-04-17T08:56:09.190 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55663.log.gz 2024-04-17T08:56:09.191 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85630.log 2024-04-17T08:56:09.191 INFO:teuthology.orchestra.run.smithi067.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.74289.log.gz 2024-04-17T08:56:09.192 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68694.log 2024-04-17T08:56:09.192 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64856.log.gz 2024-04-17T08:56:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60249.log 2024-04-17T08:56:09.193 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85630.log.gz 2024-04-17T08:56:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79783.log 2024-04-17T08:56:09.195 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68694.log.gz 2024-04-17T08:56:09.195 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87479.log 2024-04-17T08:56:09.196 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60249.log.gz 2024-04-17T08:56:09.197 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.79783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55537.log 2024-04-17T08:56:09.197 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79783.log.gz 2024-04-17T08:56:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108465.log 2024-04-17T08:56:09.198 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87479.log.gz 2024-04-17T08:56:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88429.log 2024-04-17T08:56:09.200 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55537.log.gz 2024-04-17T08:56:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64093.log 2024-04-17T08:56:09.201 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108465.log.gz 2024-04-17T08:56:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68652.log 2024-04-17T08:56:09.202 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88429.log.gz 2024-04-17T08:56:09.203 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51112.log 2024-04-17T08:56:09.203 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64093.log.gz 2024-04-17T08:56:09.205 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82105.log 2024-04-17T08:56:09.205 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68652.log.gz 2024-04-17T08:56:09.206 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107276.log 2024-04-17T08:56:09.206 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.51112.log.gz 2024-04-17T08:56:09.207 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2024-04-17T08:56:09.207 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82105.log.gz 2024-04-17T08:56:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84450.log 2024-04-17T08:56:09.209 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107276.log.gz 2024-04-17T08:56:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98853.log 2024-04-17T08:56:09.210 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2024-04-17T08:56:09.211 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81853.log 2024-04-17T08:56:09.211 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84450.log.gz 2024-04-17T08:56:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100724.log 2024-04-17T08:56:09.213 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98853.log.gz 2024-04-17T08:56:09.213 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81853.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81853.log.gz 2024-04-17T08:56:09.214 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64135.log 2024-04-17T08:56:09.215 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100724.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100724.log.gz 2024-04-17T08:56:09.215 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107612.log 2024-04-17T08:56:09.217 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48469.log 2024-04-17T08:56:09.217 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64135.log.gz 2024-04-17T08:56:09.218 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105529.log 2024-04-17T08:56:09.219 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.107612.log.gz 2024-04-17T08:56:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2024-04-17T08:56:09.219 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48469.log.gz 2024-04-17T08:56:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61242.log 2024-04-17T08:56:09.221 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105529.log.gz 2024-04-17T08:56:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82200.log 2024-04-17T08:56:09.222 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2024-04-17T08:56:09.223 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41409.log 2024-04-17T08:56:09.223 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61242.log.gz 2024-04-17T08:56:09.224 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48913.log 2024-04-17T08:56:09.225 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82200.log.gz 2024-04-17T08:56:09.226 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50008.log 2024-04-17T08:56:09.226 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41409.log.gz 2024-04-17T08:56:09.227 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42971.log 2024-04-17T08:56:09.227 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48913.log.gz 2024-04-17T08:56:09.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84678.log 2024-04-17T08:56:09.228 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50008.log.gz 2024-04-17T08:56:09.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96915.log 2024-04-17T08:56:09.230 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42971.log.gz 2024-04-17T08:56:09.231 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75755.log 2024-04-17T08:56:09.231 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84678.log.gz 2024-04-17T08:56:09.232 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110068.log 2024-04-17T08:56:09.232 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96915.log.gz 2024-04-17T08:56:09.233 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69687.log 2024-04-17T08:56:09.233 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75755.log.gz 2024-04-17T08:56:09.235 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108982.log 2024-04-17T08:56:09.235 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110068.log.gz 2024-04-17T08:56:09.236 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.69687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48160.log 2024-04-17T08:56:09.236 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69687.log.gz 2024-04-17T08:56:09.237 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64786.log 2024-04-17T08:56:09.237 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108982.log.gz 2024-04-17T08:56:09.238 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59242.log 2024-04-17T08:56:09.239 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48160.log.gz 2024-04-17T08:56:09.240 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65500.log 2024-04-17T08:56:09.240 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64786.log.gz 2024-04-17T08:56:09.241 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56651.log 2024-04-17T08:56:09.241 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59242.log.gz 2024-04-17T08:56:09.242 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94342.log 2024-04-17T08:56:09.242 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65500.log.gz 2024-04-17T08:56:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81330.log 2024-04-17T08:56:09.244 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56651.log.gz 2024-04-17T08:56:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105707.log 2024-04-17T08:56:09.245 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94342.log.gz 2024-04-17T08:56:09.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70537.log 2024-04-17T08:56:09.246 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81330.log.gz 2024-04-17T08:56:09.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73128.log 2024-04-17T08:56:09.248 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105707.log.gz 2024-04-17T08:56:09.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55098.log 2024-04-17T08:56:09.249 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70537.log.gz 2024-04-17T08:56:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67882.log 2024-04-17T08:56:09.250 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73128.log.gz 2024-04-17T08:56:09.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-04-17T08:56:09.251 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55098.log.gz 2024-04-17T08:56:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104384.log 2024-04-17T08:56:09.253 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67882.log.gz 2024-04-17T08:56:09.254 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44944.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-17T08:56:09.254 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-04-17T08:56:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54859.log 2024-04-17T08:56:09.255 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104384.log.gz 2024-04-17T08:56:09.257 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50134.log 2024-04-17T08:56:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110614.log 2024-04-17T08:56:09.258 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54859.log.gz 2024-04-17T08:56:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50134.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50134.log.gz 2024-04-17T08:56:09.260 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45238.log 2024-04-17T08:56:09.260 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110614.log.gz 2024-04-17T08:56:09.261 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90483.log 2024-04-17T08:56:09.262 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72757.log 2024-04-17T08:56:09.263 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45238.log.gz 2024-04-17T08:56:09.264 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110236.log 2024-04-17T08:56:09.264 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90483.log.gz 2024-04-17T08:56:09.265 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72757.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72757.log.gz 2024-04-17T08:56:09.266 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94564.log 2024-04-17T08:56:09.267 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110236.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110236.log.gz 2024-04-17T08:56:09.267 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68306.log 2024-04-17T08:56:09.268 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43220.log 2024-04-17T08:56:09.268 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94564.log.gz 2024-04-17T08:56:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68306.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68306.log.gz 2024-04-17T08:56:09.270 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81895.log 2024-04-17T08:56:09.272 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43220.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2024-04-17T08:56:09.272 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43220.log.gz 2024-04-17T08:56:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81895.log: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-17T08:56:09.273 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81895.log.gz 2024-04-17T08:56:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72206.log 2024-04-17T08:56:09.274 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41746.log 2024-04-17T08:56:09.274 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2024-04-17T08:56:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73446.log 2024-04-17T08:56:09.276 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72206.log.gz 2024-04-17T08:56:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107570.log 2024-04-17T08:56:09.277 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41746.log.gz 2024-04-17T08:56:09.277 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101639.log 2024-04-17T08:56:09.278 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73446.log.gz 2024-04-17T08:56:09.279 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66149.log 2024-04-17T08:56:09.279 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107570.log.gz 2024-04-17T08:56:09.280 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88345.log 2024-04-17T08:56:09.280 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101639.log.gz 2024-04-17T08:56:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85117.log 2024-04-17T08:56:09.282 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66149.log.gz 2024-04-17T08:56:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56735.log 2024-04-17T08:56:09.283 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88345.log.gz 2024-04-17T08:56:09.283 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48385.log 2024-04-17T08:56:09.284 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85117.log.gz 2024-04-17T08:56:09.284 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43682.log 2024-04-17T08:56:09.285 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56735.log.gz 2024-04-17T08:56:09.286 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97998.log 2024-04-17T08:56:09.286 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48385.log.gz 2024-04-17T08:56:09.287 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43682.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43682.log.gz 2024-04-17T08:56:09.287 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102290.log 2024-04-17T08:56:09.288 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86575.log 2024-04-17T08:56:09.289 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97998.log.gz 2024-04-17T08:56:09.289 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73338.log 2024-04-17T08:56:09.290 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102290.log.gz 2024-04-17T08:56:09.290 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49348.log 2024-04-17T08:56:09.291 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86575.log.gz 2024-04-17T08:56:09.291 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.73338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83444.log 2024-04-17T08:56:09.292 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73338.log.gz 2024-04-17T08:56:09.293 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81582.log 2024-04-17T08:56:09.293 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49348.log.gz 2024-04-17T08:56:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83444.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83444.log.gz 2024-04-17T08:56:09.294 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2024-04-17T08:56:09.295 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88114.log 2024-04-17T08:56:09.295 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81582.log.gz 2024-04-17T08:56:09.297 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60333.log 2024-04-17T08:56:09.297 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2024-04-17T08:56:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88114.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88114.log.gz 2024-04-17T08:56:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99925.log 2024-04-17T08:56:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67396.log 2024-04-17T08:56:09.299 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60333.log.gz 2024-04-17T08:56:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50050.log 2024-04-17T08:56:09.300 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99925.log.gz 2024-04-17T08:56:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.67396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67396.log.gz 2024-04-17T08:56:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89814.log 2024-04-17T08:56:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98895.log 2024-04-17T08:56:09.304 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50050.log.gz 2024-04-17T08:56:09.304 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89814.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89814.log.gz 2024-04-17T08:56:09.304 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51172.log 2024-04-17T08:56:09.305 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48595.log 2024-04-17T08:56:09.305 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98895.log.gz 2024-04-17T08:56:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103135.log 2024-04-17T08:56:09.306 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51172.log.gz 2024-04-17T08:56:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67625.log 2024-04-17T08:56:09.308 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48595.log.gz 2024-04-17T08:56:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47083.log 2024-04-17T08:56:09.308 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103135.log.gz 2024-04-17T08:56:09.309 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107402.log 2024-04-17T08:56:09.309 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67625.log.gz 2024-04-17T08:56:09.311 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42299.log 2024-04-17T08:56:09.311 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47083.log.gz 2024-04-17T08:56:09.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109892.log 2024-04-17T08:56:09.312 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107402.log.gz 2024-04-17T08:56:09.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84907.log 2024-04-17T08:56:09.313 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42299.log.gz 2024-04-17T08:56:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42310.log 2024-04-17T08:56:09.315 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109892.log.gz 2024-04-17T08:56:09.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97389.log 2024-04-17T08:56:09.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84907.log.gz 2024-04-17T08:56:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-04-17T08:56:09.317 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42310.log.gz 2024-04-17T08:56:09.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.97389.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-17T08:56:09.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97389.log.gz 2024-04-17T08:56:09.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53688.log 2024-04-17T08:56:09.319 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-04-17T08:56:09.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66324.log 2024-04-17T08:56:09.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75340.log 2024-04-17T08:56:09.322 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53688.log.gz 2024-04-17T08:56:09.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66324.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66324.log.gz 2024-04-17T08:56:09.323 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67475.log 2024-04-17T08:56:09.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108933.log 2024-04-17T08:56:09.324 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75340.log.gz 2024-04-17T08:56:09.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61200.log 2024-04-17T08:56:09.326 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67475.log.gz 2024-04-17T08:56:09.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108933.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108933.log.gz 2024-04-17T08:56:09.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89940.log 2024-04-17T08:56:09.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84408.log 2024-04-17T08:56:09.328 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61200.log.gz 2024-04-17T08:56:09.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100510.log 2024-04-17T08:56:09.330 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89940.log.gz 2024-04-17T08:56:09.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84408.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84408.log.gz 2024-04-17T08:56:09.331 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84730.log 2024-04-17T08:56:09.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101049.log 2024-04-17T08:56:09.332 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.100510.log.gz 2024-04-17T08:56:09.332 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-17T08:56:09.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72290.log 2024-04-17T08:56:09.333 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84730.log.gz 2024-04-17T08:56:09.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102080.log 2024-04-17T08:56:09.334 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101049.log.gz 2024-04-17T08:56:09.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61158.log 2024-04-17T08:56:09.336 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72290.log.gz 2024-04-17T08:56:09.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102080.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-17T08:56:09.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102080.log.gz 2024-04-17T08:56:09.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59368.log 2024-04-17T08:56:09.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61158.log.gz 2024-04-17T08:56:09.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103270.log 2024-04-17T08:56:09.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87420.log 2024-04-17T08:56:09.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59368.log.gz 2024-04-17T08:56:09.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75256.log 2024-04-17T08:56:09.342 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103270.log.gz 2024-04-17T08:56:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87420.log.gz 2024-04-17T08:56:09.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41704.log 2024-04-17T08:56:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81414.log 2024-04-17T08:56:09.344 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75256.log.gz 2024-04-17T08:56:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41704.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-17T08:56:09.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41704.log.gz 2024-04-17T08:56:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81414.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81414.log.gz 2024-04-17T08:56:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102645.log 2024-04-17T08:56:09.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47209.log 2024-04-17T08:56:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102645.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102645.log.gz 2024-04-17T08:56:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51846.log 2024-04-17T08:56:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47209.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47209.log.gz 2024-04-17T08:56:09.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60207.log 2024-04-17T08:56:09.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51846.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51846.log.gz 2024-04-17T08:56:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60844.log 2024-04-17T08:56:09.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60207.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60207.log.gz 2024-04-17T08:56:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82636.log 2024-04-17T08:56:09.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60844.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60844.log.gz 2024-04-17T08:56:09.359 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43640.log 2024-04-17T08:56:09.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.82636.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82636.log.gz 2024-04-17T08:56:09.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47682.log 2024-04-17T08:56:09.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43640.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43640.log.gz 2024-04-17T08:56:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74941.log 2024-04-17T08:56:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47682.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47682.log.gz 2024-04-17T08:56:09.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104212.log 2024-04-17T08:56:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.74941.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74941.log.gz 2024-04-17T08:56:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96652.log 2024-04-17T08:56:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104212.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104212.log.gz 2024-04-17T08:56:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76659.log 2024-04-17T08:56:09.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96652.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96652.log.gz 2024-04-17T08:56:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83806.log 2024-04-17T08:56:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76659.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76659.log.gz 2024-04-17T08:56:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71953.log 2024-04-17T08:56:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.83806.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83806.log.gz 2024-04-17T08:56:09.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55884.log 2024-04-17T08:56:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71953.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71953.log.gz 2024-04-17T08:56:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103659.log 2024-04-17T08:56:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55884.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2024-04-17T08:56:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92861.log 2024-04-17T08:56:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103659.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103659.log.gz 2024-04-17T08:56:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2024-04-17T08:56:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92861.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92861.log.gz 2024-04-17T08:56:09.383 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63127.log 2024-04-17T08:56:09.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46567.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2024-04-17T08:56:09.385 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102560.log 2024-04-17T08:56:09.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63127.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63127.log.gz 2024-04-17T08:56:09.387 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85212.log 2024-04-17T08:56:09.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102560.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102560.log.gz 2024-04-17T08:56:09.389 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2024-04-17T08:56:09.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85212.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85212.log.gz 2024-04-17T08:56:09.390 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104300.log 2024-04-17T08:56:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46609.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2024-04-17T08:56:09.393 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.104300.log: -- /var/log/ceph/ceph-client.admin.95586.log 2024-04-17T08:56:09.394 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104300.log.gz 2024-04-17T08:56:09.395 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109847.log 2024-04-17T08:56:09.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.95586.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95586.log.gz 2024-04-17T08:56:09.397 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108423.log 2024-04-17T08:56:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109847.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109847.log.gz 2024-04-17T08:56:09.399 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97178.log 2024-04-17T08:56:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108423.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108423.log.gz 2024-04-17T08:56:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.97178.log: --verbose -- /var/log/ceph/ceph-client.admin.71851.log 2024-04-17T08:56:09.402 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97178.log.gz 2024-04-17T08:56:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64686.log 2024-04-17T08:56:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71851.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71851.log.gz 2024-04-17T08:56:09.405 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86050.log 2024-04-17T08:56:09.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64686.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64686.log.gz 2024-04-17T08:56:09.407 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75526.log 2024-04-17T08:56:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86050.log.gz 2024-04-17T08:56:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59894.log 2024-04-17T08:56:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75526.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75526.log.gz 2024-04-17T08:56:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110488.log 2024-04-17T08:56:09.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59894.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59894.log.gz 2024-04-17T08:56:09.413 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99021.log 2024-04-17T08:56:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.110488.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110488.log.gz 2024-04-17T08:56:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84492.log 2024-04-17T08:56:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99021.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99021.log.gz 2024-04-17T08:56:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55705.log/var/log/ceph/ceph-client.admin.84492.log: 2024-04-17T08:56:09.418 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84492.log.gz 2024-04-17T08:56:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55411.log 2024-04-17T08:56:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55705.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55705.log.gz 2024-04-17T08:56:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106535.log 2024-04-17T08:56:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55411.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55411.log.gz 2024-04-17T08:56:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41470.log 2024-04-17T08:56:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106535.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106535.log.gz 2024-04-17T08:56:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47905.log 2024-04-17T08:56:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41470.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41470.log.gz 2024-04-17T08:56:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68568.log 2024-04-17T08:56:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47905.log.gz 2024-04-17T08:56:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47602.log 2024-04-17T08:56:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68568.log.gz 2024-04-17T08:56:09.431 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99576.log 2024-04-17T08:56:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47602.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47602.log.gz 2024-04-17T08:56:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107360.log 2024-04-17T08:56:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99576.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99576.log.gz 2024-04-17T08:56:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95169.log 2024-04-17T08:56:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.107360.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107360.log.gz 2024-04-17T08:56:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88387.log/var/log/ceph/ceph-client.admin.95169.log: 2024-04-17T08:56:09.439 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95169.log.gz 2024-04-17T08:56:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43472.log 2024-04-17T08:56:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88387.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88387.log.gz 2024-04-17T08:56:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2024-04-17T08:56:09.442 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43472.log.gz 2024-04-17T08:56:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109402.log 2024-04-17T08:56:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57161.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2024-04-17T08:56:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88699.log 2024-04-17T08:56:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109402.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109402.log.gz 2024-04-17T08:56:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70599.log 2024-04-17T08:56:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88699.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88699.log.gz 2024-04-17T08:56:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105928.log 2024-04-17T08:56:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70599.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70599.log.gz 2024-04-17T08:56:09.452 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 2024-04-17T08:56:09.453 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.105928.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105928.log.gz 2024-04-17T08:56:09.454 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70078.log 2024-04-17T08:56:09.454 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42300.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-04-17T08:56:09.455 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60716.log 2024-04-17T08:56:09.457 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.70078.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70078.log.gz 2024-04-17T08:56:09.458 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57272.log 2024-04-17T08:56:09.458 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60716.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60716.log.gz 2024-04-17T08:56:09.459 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55266.log 2024-04-17T08:56:09.461 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57272.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57272.log.gz 2024-04-17T08:56:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93241.log/var/log/ceph/ceph-client.admin.55266.log: 2024-04-17T08:56:09.462 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55266.log.gz 2024-04-17T08:56:09.463 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83069.log 2024-04-17T08:56:09.465 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93241.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93241.log.gz 2024-04-17T08:56:09.466 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47768.log/var/log/ceph/ceph-client.admin.83069.log: 2024-04-17T08:56:09.466 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83069.log.gz 2024-04-17T08:56:09.467 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98185.log 2024-04-17T08:56:09.469 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47768.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47768.log.gz 2024-04-17T08:56:09.470 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-04-17T08:56:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98185.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98185.log.gz 2024-04-17T08:56:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88250.log 2024-04-17T08:56:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42644.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gz 2024-04-17T08:56:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44902.log 2024-04-17T08:56:09.475 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88250.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88250.log.gz 2024-04-17T08:56:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86092.log 2024-04-17T08:56:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44902.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44902.log.gz 2024-04-17T08:56:09.478 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55224.log 2024-04-17T08:56:09.479 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86092.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.86092.log.gz 2024-04-17T08:56:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49198.log 2024-04-17T08:56:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55224.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55224.log.gz 2024-04-17T08:56:09.482 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48805.log 2024-04-17T08:56:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49198.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49198.log.gz 2024-04-17T08:56:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56087.log 2024-04-17T08:56:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48805.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48805.log.gz 2024-04-17T08:56:09.486 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-04-17T08:56:09.487 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56087.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56087.log.gz 2024-04-17T08:56:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64515.log 2024-04-17T08:56:09.489 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42086.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-04-17T08:56:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103739.log 2024-04-17T08:56:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.64515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64515.log.gz 2024-04-17T08:56:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94871.log 2024-04-17T08:56:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.103739.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103739.log.gz 2024-04-17T08:56:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.94871.log: /var/log/ceph/ceph-client.admin.90216.log 2024-04-17T08:56:09.495 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94871.log.gz 2024-04-17T08:56:09.496 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62833.log 2024-04-17T08:56:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90216.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90216.log.gz 2024-04-17T08:56:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62833.log: /var/log/ceph/ceph-client.admin.96191.log 2024-04-17T08:56:09.499 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62833.log.gz 2024-04-17T08:56:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108709.log 2024-04-17T08:56:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96191.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96191.log.gz 2024-04-17T08:56:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.108709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50176.log 2024-04-17T08:56:09.503 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108709.log.gz 2024-04-17T08:56:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93911.log 2024-04-17T08:56:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50176.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50176.log.gz 2024-04-17T08:56:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66830.log 2024-04-17T08:56:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93911.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93911.log.gz 2024-04-17T08:56:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-04-17T08:56:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66830.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66830.log.gz 2024-04-17T08:56:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45490.log: /var/log/ceph/ceph-client.admin.76827.log 2024-04-17T08:56:09.511 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-04-17T08:56:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89898.log 2024-04-17T08:56:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76827.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76827.log.gz 2024-04-17T08:56:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96233.log 2024-04-17T08:56:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89898.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89898.log.gz 2024-04-17T08:56:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55495.log 2024-04-17T08:56:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.96233.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96233.log.gz 2024-04-17T08:56:09.518 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102939.log 2024-04-17T08:56:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55495.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55495.log.gz 2024-04-17T08:56:09.520 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60081.log 2024-04-17T08:56:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102939.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102939.log.gz 2024-04-17T08:56:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.60081.log: -- /var/log/ceph/ceph-client.admin.104552.log 2024-04-17T08:56:09.523 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60081.log.gz 2024-04-17T08:56:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75923.log 2024-04-17T08:56:09.526 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.104552.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.104552.log.gz 2024-04-17T08:56:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.75923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45406.log 2024-04-17T08:56:09.529 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75923.log.gz 2024-04-17T08:56:09.529 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55038.log 2024-04-17T08:56:09.530 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45406.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45406.log.gz 2024-04-17T08:56:09.531 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88156.log 2024-04-17T08:56:09.532 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55038.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55038.log.gz 2024-04-17T08:56:09.533 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84991.log 2024-04-17T08:56:09.534 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88156.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88156.log.gz 2024-04-17T08:56:09.536 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88993.log/var/log/ceph/ceph-client.admin.84991.log: 2024-04-17T08:56:09.536 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84991.log.gz 2024-04-17T08:56:09.537 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62182.log 2024-04-17T08:56:09.538 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88993.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88993.log.gz 2024-04-17T08:56:09.539 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62182.log: /var/log/ceph/ceph-client.admin.81979.log 2024-04-17T08:56:09.540 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62182.log.gz 2024-04-17T08:56:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75424.log 2024-04-17T08:56:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81979.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81979.log.gz 2024-04-17T08:56:09.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.75424.log: --verbose -- /var/log/ceph/ceph-client.admin.61368.log 2024-04-17T08:56:09.545 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75424.log.gz 2024-04-17T08:56:09.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65143.log 2024-04-17T08:56:09.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61368.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.61368.log.gz 2024-04-17T08:56:09.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72938.log 2024-04-17T08:56:09.549 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65143.log.gz 2024-04-17T08:56:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94778.log 2024-04-17T08:56:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.72938.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72938.log.gz 2024-04-17T08:56:09.552 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.94778.log: --verbose -- /var/log/ceph/ceph-client.admin.46813.log 2024-04-17T08:56:09.552 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94778.log.gz 2024-04-17T08:56:09.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71683.log 2024-04-17T08:56:09.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46813.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46813.log.gz 2024-04-17T08:56:09.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37172.log 2024-04-17T08:56:09.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.71683.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71683.log.gz 2024-04-17T08:56:09.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63367.log 2024-04-17T08:56:09.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37172.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37172.log.gz 2024-04-17T08:56:09.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67667.log 2024-04-17T08:56:09.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63367.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63367.log.gz 2024-04-17T08:56:09.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76264.log 2024-04-17T08:56:09.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67667.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67667.log.gz 2024-04-17T08:56:09.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2024-04-17T08:56:09.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.76264.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76264.log.gz 2024-04-17T08:56:09.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44446.log 2024-04-17T08:56:09.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41834.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2024-04-17T08:56:09.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106241.log 2024-04-17T08:56:09.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44446.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44446.log.gz 2024-04-17T08:56:09.570 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61701.log 2024-04-17T08:56:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106241.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106241.log.gz 2024-04-17T08:56:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84576.log 2024-04-17T08:56:09.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61701.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61701.log.gz 2024-04-17T08:56:09.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46021.log 2024-04-17T08:56:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84576.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84576.log.gz 2024-04-17T08:56:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93598.log 2024-04-17T08:56:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46021.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46021.log.gz 2024-04-17T08:56:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88489.log 2024-04-17T08:56:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93598.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93598.log.gz 2024-04-17T08:56:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93682.log 2024-04-17T08:56:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88489.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88489.log.gz 2024-04-17T08:56:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81057.log 2024-04-17T08:56:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93682.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93682.log.gz 2024-04-17T08:56:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109318.log 2024-04-17T08:56:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.81057.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81057.log.gz 2024-04-17T08:56:09.587 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101133.log 2024-04-17T08:56:09.588 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109318.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109318.log.gz 2024-04-17T08:56:09.589 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100923.log 2024-04-17T08:56:09.589 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101133.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101133.log.gz 2024-04-17T08:56:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46147.log 2024-04-17T08:56:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100923.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100923.log.gz 2024-04-17T08:56:09.593 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99967.log 2024-04-17T08:56:09.594 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46147.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46147.log.gz 2024-04-17T08:56:09.594 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44986.log 2024-04-17T08:56:09.596 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99967.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99967.log.gz 2024-04-17T08:56:09.596 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51332.log 2024-04-17T08:56:09.597 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44986.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44986.log.gz 2024-04-17T08:56:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63572.log 2024-04-17T08:56:09.599 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51332.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51332.log.gz 2024-04-17T08:56:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98438.log 2024-04-17T08:56:09.601 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63572.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63572.log.gz 2024-04-17T08:56:09.602 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77372.log 2024-04-17T08:56:09.603 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98438.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98438.log.gz 2024-04-17T08:56:09.604 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.77372.log: /var/log/ceph/ceph-client.admin.100009.log 2024-04-17T08:56:09.605 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77372.log.gz 2024-04-17T08:56:09.606 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65458.log 2024-04-17T08:56:09.608 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.100009.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100009.log.gz 2024-04-17T08:56:09.608 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106960.log 2024-04-17T08:56:09.611 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65458.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65458.log.gz 2024-04-17T08:56:09.611 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104077.log 2024-04-17T08:56:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.106960.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106960.log.gz 2024-04-17T08:56:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85500.log/var/log/ceph/ceph-client.admin.104077.log: 2024-04-17T08:56:09.613 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104077.log.gz 2024-04-17T08:56:09.614 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45154.log 2024-04-17T08:56:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85500.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85500.log.gz 2024-04-17T08:56:09.616 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.45154.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.84187.log 2024-04-17T08:56:09.618 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45154.log.gz 2024-04-17T08:56:09.618 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77608.log 2024-04-17T08:56:09.620 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.84187.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84187.log.gz 2024-04-17T08:56:09.620 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50302.log 2024-04-17T08:56:09.621 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.77608.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77608.log.gz 2024-04-17T08:56:09.622 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63211.log 2024-04-17T08:56:09.624 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50302.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50302.log.gz 2024-04-17T08:56:09.625 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102897.log 2024-04-17T08:56:09.625 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63211.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63211.log.gz 2024-04-17T08:56:09.626 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-17T08:56:09.628 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102897.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102897.log.gz 2024-04-17T08:56:09.629 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59852.log/var/log/ceph/ceph-mds.a.log: 2024-04-17T08:56:09.631 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89726.log 2024-04-17T08:56:09.633 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59852.log.gz 2024-04-17T08:56:09.634 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105844.log 2024-04-17T08:56:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.89726.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89726.log.gz 2024-04-17T08:56:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.105844.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68988.log 2024-04-17T08:56:09.640 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105844.log.gz 2024-04-17T08:56:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109984.log 2024-04-17T08:56:09.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.68988.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68988.log.gz 2024-04-17T08:56:09.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.109984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64744.log 2024-04-17T08:56:09.648 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109984.log.gz 2024-04-17T08:56:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90592.log 2024-04-17T08:56:09.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64744.log: /var/log/ceph/ceph-client.admin.90592.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55621.log 2024-04-17T08:56:09.654 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64744.log.gz 2024-04-17T08:56:09.654 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90592.log.gz 2024-04-17T08:56:09.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98979.log 2024-04-17T08:56:09.658 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55621.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55621.log.gz 2024-04-17T08:56:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67208.log 2024-04-17T08:56:09.662 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98979.log.gz 2024-04-17T08:56:09.663 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67840.log 2024-04-17T08:56:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.67208.log: /var/log/ceph/ceph-client.admin.67840.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101996.log 2024-04-17T08:56:09.668 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67208.log.gz 2024-04-17T08:56:09.669 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67840.log.gz 2024-04-17T08:56:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54765.log 2024-04-17T08:56:09.673 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-17T08:56:09.674 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54765.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54765.log.gz 2024-04-17T08:56:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.101996.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101996.log.gz 2024-04-17T08:56:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94606.log 2024-04-17T08:56:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56843.log 2024-04-17T08:56:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.94606.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94606.log.gz 2024-04-17T08:56:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56843.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56843.log.gz 2024-04-17T08:56:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102500.log 2024-04-17T08:56:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88741.log 2024-04-17T08:56:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.102500.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102500.log.gz 2024-04-17T08:56:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88741.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88741.log.gz 2024-04-17T08:56:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85458.log 2024-04-17T08:56:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62707.log 2024-04-17T08:56:09.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.85458.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85458.log.gz 2024-04-17T08:56:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62707.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62707.log.gz 2024-04-17T08:56:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98937.log 2024-04-17T08:56:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99757.log 2024-04-17T08:56:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.98937.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98937.log.gz 2024-04-17T08:56:09.694 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.99757.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99757.log.gz 2024-04-17T08:56:10.120 INFO:teuthology.orchestra.run.smithi187.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-17T08:56:10.434 INFO:teuthology.orchestra.run.smithi187.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-17T08:56:10.532 INFO:teuthology.orchestra.run.smithi187.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-17T08:56:10.949 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-17T08:56:11.716 INFO:teuthology.orchestra.run.smithi187.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-17T08:56:11.935 INFO:teuthology.orchestra.run.smithi187.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-17T08:56:11.940 INFO:teuthology.orchestra.run.smithi187.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-17T08:56:12.246 INFO:teuthology.orchestra.run.smithi067.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-17T08:56:12.717 INFO:teuthology.orchestra.run.smithi067.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-17T08:56:12.723 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-17T08:56:12.725 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-17T08:56:12.977 INFO:teuthology.orchestra.run.smithi187.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-17T08:56:12.981 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-17T08:56:12.981 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m4.849s 2024-04-17T08:56:12.981 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m21.145s 2024-04-17T08:56:12.981 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m1.853s 2024-04-17T08:56:14.393 INFO:teuthology.orchestra.run.smithi067.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-17T08:56:14.397 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-17T08:56:14.397 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m6.269s 2024-04-17T08:56:14.398 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m20.972s 2024-04-17T08:56:14.398 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m4.601s 2024-04-17T08:56:14.399 INFO:tasks.ceph:Archiving logs... 2024-04-17T08:56:14.399 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi067/log 2024-04-17T08:56:14.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-17T08:56:18.065 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi187/log 2024-04-17T08:56:18.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-17T08:56:20.693 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-17T08:56:20.704 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-17T08:56:20.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-17T08:56:20.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-17T08:56:20.875 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-17T08:56:20.885 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-17T08:56:20.886 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T08:56:20.894 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T08:56:20.945 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-17T08:56:20.948 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 +1205us[+1130us] +/- 66ms 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 68 -1146us[-1220us] +/- 62ms 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +113us[ +39us] +/- 73ms 2024-04-17T08:56:20.956 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv01.front.sepia.ceph.com 3 7 377 127 +1171us[+1143us] +/- 65ms 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 8 -1202us[-1202us] +/- 63ms 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 67 +104us[ +104us] +/- 73ms 2024-04-17T08:56:20.957 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-17T08:56:20.959 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-17T08:56:20.969 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-17T08:56:20.970 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-17T08:56:20.996 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-17T08:56:21.288 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-17T08:56:21.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-17T08:56:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:56:21.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-17T08:56:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:56:21.728 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-17T08:56:21.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-17T08:56:21.776 INFO:teuthology.task.internal:Duration was 6722.889394 seconds 2024-04-17T08:56:21.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-17T08:56:21.810 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-17T08:56:21.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-17T08:56:21.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-17T08:56:21.983 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-17T08:56:21.983 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-17T08:56:22.372 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-17T08:56:22.372 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T08:56:22.373 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-17T08:56:22.443 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-04-17T08:56:22.443 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-17T08:56:22.519 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-17T08:56:22.519 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-17T08:56:22.527 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-17T08:56:22.626 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-17T08:56:22.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-17T08:56:22.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-17T08:56:22.907 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-17T08:56:22.918 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-17T08:56:22.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-17T08:56:23.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-17T08:56:23.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-17T08:56:23.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-17T08:56:23.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-17T08:56:23.241 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-17T08:56:23.252 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-04-17T08:56:23.308 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-17T08:56:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:56:23.365 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-17T08:56:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T08:56:23.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-17T08:56:23.430 INFO:teuthology.task.internal:Transferring archived files... 2024-04-17T08:56:23.431 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi067 2024-04-17T08:56:23.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-17T08:56:23.548 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904/remote/smithi187 2024-04-17T08:56:23.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-17T08:56:23.654 INFO:teuthology.task.internal:Removing archive directory... 2024-04-17T08:56:23.655 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-17T08:56:23.663 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-17T08:56:23.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-17T08:56:23.732 INFO:teuthology.task.internal:Not uploading archives. 2024-04-17T08:56:23.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-17T08:56:23.778 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-17T08:56:23.778 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-17T08:56:23.786 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-17T08:56:23.840 INFO:teuthology.orchestra.run.smithi067.stdout: 393848 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 17 08:56 /home/ubuntu/cephtest 2024-04-17T08:56:23.848 INFO:teuthology.orchestra.run.smithi187.stdout: 524637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 17 08:56 /home/ubuntu/cephtest 2024-04-17T08:56:23.849 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-17T08:56:23.860 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-17T08:56:23.927 INFO:teuthology.nuke:Checking targets against current locks 2024-04-17T08:56:23.952 DEBUG:teuthology.nuke:shortname: smithi067 2024-04-17T08:56:23.952 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-17T08:56:23.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 06:55:03.485080', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjfw6h6JlM/VdJSY9YLTvtZbS8r4uvCOoXXUwb++K3eVRFZ3cvDITS7Ooghp+2zjwYm8uc+/FA5Tmz2lvvAQ9A='} 2024-04-17T08:56:23.983 DEBUG:teuthology.nuke:shortname: smithi187 2024-04-17T08:56:23.983 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-17T08:56:24.010 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-17_04:58:44-fs-wip-rishabh-testing-20240416.193735-testing-default-smithi/7659904', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 06:55:03.486577', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMER5OGO3me8ZozXOURqAX21fPe/kP8GyM6fXYA7gpJD1upTcNvwuBB2LbxwQ9RM10QyUxYXSASm1L2lMQnlalE='} 2024-04-17T08:56:24.040 INFO:teuthology.orchestra.console.smithi067:Power off 2024-04-17T08:56:24.041 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-17T08:56:24.066 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-04-17T08:56:24.067 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:24.084 INFO:teuthology.orchestra.console.smithi187:Power off 2024-04-17T08:56:24.085 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-17T08:56:24.095 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-17T08:56:24.107 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2024-04-17T08:56:24.107 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:24.135 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-17T08:56:28.096 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:28.136 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:28.208 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-17T08:56:28.246 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2024-04-17T08:56:32.210 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:32.248 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:32.323 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-17T08:56:32.359 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2024-04-17T08:56:32.460 INFO:teuthology.orchestra.console.smithi187:Power off completed 2024-04-17T08:56:36.326 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-17T08:56:36.438 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-04-17T08:56:36.539 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-04-17T08:56:36.645 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 client.1-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 6722.889393568039 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=feb1efc3644146dab33b42a18eba8462 status: fail success: false 2024-04-17T08:56:36.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-17T08:56:36.823 INFO:teuthology.run:FAIL